2026-09-30 14:59:55 [scrapy.utils.log] INFO: Scrapy 2.18.0 started (bot: grabbers) 2026-09-30 14:59:55 [scrapy.utils.log] INFO: Versions: {'lxml': '6.1.3', 'libxml2': '2.14.6', 'cssselect': '1.5.0', 'parsel': '1.12.1', 'w3lib': '2.4.1', 'Twisted': '26.4.0', 'Python': '3.14.7 (main, Sep 1 2026, 14:18:09) [Clang 22.1.3 ]', 'pyOpenSSL': '26.4.0 (OpenSSL 4.0.2 25 Aug 2026)', 'cryptography': '50.0.1', 'Platform': 'Linux-6.18.38-73.137.amzn2023.x86_64-x86_64-with-glibc2.35'} 2026-09-30 14:59:55 [scrapy.addons] INFO: Enabled addons: [] 2026-09-30 14:59:55 [py.warnings] WARNING: /layers/paketo-buildpacks_python-package-managers-run/uv-env/venv/lib/python3.14/site-packages/scrapy/extensions/feedexport.py:479: ScrapyDeprecationWarning: The `FEED_URI` and `FEED_FORMAT` settings have been deprecated in favor of the `FEEDS` setting. Please see the `FEEDS` setting docs for more details exporter = cls(crawler) 2026-09-30 14:59:55 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.logcount.LogCount', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.logstats.LogStats', 'grabbers.extensions.custom_fields.ExportCustomFieldsExtension', 'grabbers.extensions.kol_roles.ExportKolRolesExtension', 'grabbers.extensions.exporters.DefaultScrapedItemsExporter'] 2026-09-30 14:59:55 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'grabbers', 'FEED_EXPORT_FIELDS': ['External Id', 'Parent External Id', 'Name', 'Plain text: Parent Name', 'Start date', 'Start time', 'End date', 'End time', 'Location', 'Plain text: Event Link', 'Plain text: Event Link (Excel)', 'Filter: Session Type', 'Plain text: Description', 'Plain text: Session ID', 'Filter: Tag(s)', 'Filter: Network/Committee Affiliation', 'Filter: Ticketed Sessions', 'Plain text: Speaker(s)', 'Plain text: Presenter', 'Plain text: Chair', 'Plain text: Co-Chair', 'Plain text: Faculty', 'Plain text: Moderator'], 'FEED_FORMAT': 'xlsx', 'FEED_URI': 's3://usummit-prod-grabbers-bucket/CHEST/CHEST-2553_2026-09-30_14-59-46.xlsx', 'LOG_FILE': '/etc/scrapyd/dbs/logs/grabbers/CHEST/93ba8a7ebcdf11f1bd97b3dae8fe8c65.log', 'LOG_LEVEL': 'INFO', 'NEWSPIDER_MODULE': 'grabbers.spiders', 'SPIDER_MODULES': ['grabbers.spiders'], 'TELNETCONSOLE_ENABLED': False, 'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 ' 'Firefox/131.0'} 2026-09-30 14:59:56 [scrapy_fake_useragent.middleware] INFO: Using '' as the User-Agent provider 2026-09-30 14:59:56 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'grabbers.middlewares.RandomUserAgentMiddleware', 'grabbers.middlewares.RetryUserAgentMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2026-09-30 14:59:56 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.start.StartSpiderMiddleware', 'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'grabbers.middlewares.SilentValidationMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware', 'scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware'] 2026-09-30 14:59:56 [scrapy.middleware] INFO: Enabled item pipelines: ['grabbers.pipelines.CleanTextPipeline', 'grabbers.pipelines.DateTimeFormatPipeline', 'grabbers.pipelines.BoolFormatPipeline', 'grabbers.pipelines.ValidateEventNamePipeline', 'grabbers.pipelines.CountryCodePipeline', 'grabbers.pipelines.HyperlinkedEventLinkPipeline'] 2026-09-30 14:59:56 [scrapy.core.engine] INFO: Spider opened 2026-09-30 14:59:56 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2026-09-30 14:59:58 [scrapy_fake_useragent.middleware] INFO: Using '' as the User-Agent provider 2026-09-30 15:00:15 [CHEST] WARNING: Some events don't have times. Please set `ENABLE_MISSING_TIME_LOGS` environment variable to `1` to see detailed logs about these events. 2026-09-30 15:00:56 [scrapy.extensions.logstats] INFO: Crawled 587 pages (at 587 pages/min), scraped 584 items (at 584 items/min) 2026-09-30 15:01:56 [scrapy.extensions.logstats] INFO: Crawled 1308 pages (at 721 pages/min), scraped 1305 items (at 721 items/min) 2026-09-30 15:02:56 [scrapy.extensions.logstats] INFO: Crawled 2039 pages (at 731 pages/min), scraped 2033 items (at 728 items/min) 2026-09-30 15:03:56 [scrapy.extensions.logstats] INFO: Crawled 2743 pages (at 704 pages/min), scraped 2737 items (at 704 items/min) 2026-09-30 15:04:56 [scrapy.extensions.logstats] INFO: Crawled 3508 pages (at 765 pages/min), scraped 3502 items (at 765 items/min) 2026-09-30 15:05:56 [scrapy.extensions.logstats] INFO: Crawled 4300 pages (at 792 pages/min), scraped 4294 items (at 792 items/min) 2026-09-30 15:06:56 [scrapy.extensions.logstats] INFO: Crawled 5137 pages (at 837 pages/min), scraped 5131 items (at 837 items/min) 2026-09-30 15:07:56 [scrapy.extensions.logstats] INFO: Crawled 5915 pages (at 778 pages/min), scraped 5909 items (at 778 items/min) 2026-09-30 15:08:56 [scrapy.extensions.logstats] INFO: Crawled 6635 pages (at 720 pages/min), scraped 6629 items (at 720 items/min) 2026-09-30 15:09:56 [scrapy.extensions.logstats] INFO: Crawled 7350 pages (at 715 pages/min), scraped 7344 items (at 715 items/min) 2026-09-30 15:10:39 [scrapy.core.engine] INFO: Closing spider (finished) 2026-09-30 15:10:39 [CHEST] WARNING: Some events have missing dates. They were filled with earliest known date: 2026-10-16