2026-03-03 06:46:23 [scrapy.utils.log] INFO: Scrapy 2.14.1 started (bot: grabbers) 2026-03-03 06:46:23 [scrapy.utils.log] INFO: Versions: {'lxml': '6.0.2', 'libxml2': '2.14.6', 'cssselect': '1.4.0', 'parsel': '1.11.0', 'w3lib': '2.4.0', 'Twisted': '25.5.0', 'Python': '3.14.3 (main, Feb 4 2026, 14:18:08) [GCC 11.4.0]', 'pyOpenSSL': '25.3.0 (OpenSSL 3.5.5 27 Jan 2026)', 'cryptography': '46.0.5', 'Platform': 'Linux-6.12.58-82.121.amzn2023.x86_64-x86_64-with-glibc2.35'} 2026-03-03 06:46:23 [scrapy.addons] INFO: Enabled addons: [] 2026-03-03 06:46:23 [py.warnings] WARNING: /layers/paketo-buildpacks_pip-install/packages/lib/python3.14/site-packages/scrapy/extensions/feedexport.py:436: ScrapyDeprecationWarning: The `FEED_URI` and `FEED_FORMAT` settings have been deprecated in favor of the `FEEDS` setting. Please see the `FEEDS` setting docs for more details exporter = cls(crawler) 2026-03-03 06:46:23 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.logcount.LogCount', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.logstats.LogStats', 'grabbers.extensions.custom_fields.ExportCustomFieldsExtension', 'grabbers.extensions.kol_roles.ExportKolRolesExtension', 'grabbers.extensions.exporters.DefaultScrapedItemsExporter'] 2026-03-03 06:46:23 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'grabbers', 'FEED_EXPORT_FIELDS': ['External Id', 'Parent External Id', 'Name', 'Plain text: Parent Name', 'Start date', 'Start time', 'End date', 'End time', 'Location', 'Plain text: Event Link', 'Plain text: Event Link (Excel)', 'Plain text: Learning Objectives', 'Plain text: Faculty Disclosures', 'Plain text: Description', 'Plain text: Speakers', 'Plain text: Directors', 'Plain text: Moderators', 'Filter: On Demand', 'Filter: Session Format', 'Filter: CME Available', 'Filter: Session Type', 'Filter: Event Type', 'Filter: Topics', 'Filter: Program Level', 'Filter: Program Tracks', 'Filter: Teaching Styles', 'Filter: Recommended audience', 'Filter: Core Competencies'], 'FEED_FORMAT': 'xlsx', 'LOG_FILE': '/etc/scrapyd/dbs/logs/grabbers/AAN ' '(Events)/ac34dd5216cc11f1a30dce1a2a20682d.log', 'LOG_LEVEL': 'INFO', 'NEWSPIDER_MODULE': 'grabbers.spiders', 'SPIDER_MODULES': ['grabbers.spiders'], 'TELNETCONSOLE_ENABLED': False, 'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 ' 'Firefox/131.0'} 2026-03-03 06:46:24 [scrapy_fake_useragent.middleware] INFO: Using '' as the User-Agent provider 2026-03-03 06:46:24 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'grabbers.middlewares.RandomUserAgentMiddleware', 'grabbers.middlewares.RetryUserAgentMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2026-03-03 06:46:24 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.start.StartSpiderMiddleware', 'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2026-03-03 06:46:25 [scrapy.middleware] INFO: Enabled item pipelines: ['grabbers.pipelines.CleanTextPipeline', 'grabbers.pipelines.DateTimeFormatPipeline', 'grabbers.pipelines.BoolFormatPipeline', 'grabbers.pipelines.ValidateEventNamePipeline', 'grabbers.pipelines.CountryCodePipeline', 'grabbers.pipelines.HyperlinkedEventLinkPipeline', 'grabbers.pipelines.ElasticSearchPipeline'] 2026-03-03 06:46:25 [scrapy.core.engine] INFO: Spider opened 2026-03-03 06:46:25 [elastic_transport.transport] INFO: HEAD https://elasticsearch-es-http.elasticsearch.svc:9200/congress-events [status:200 duration:0.027s] 2026-03-03 06:46:25 [AAN (Events)] INFO: Start reindexing existing events to historical index. 2026-03-03 06:46:25 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/congress-events/_settings [status:200 duration:0.123s] 2026-03-03 06:46:25 [elastic_transport.transport] INFO: DELETE https://elasticsearch-es-http.elasticsearch.svc:9200/congress-events-historical?ignore_unavailable=true [status:200 duration:0.057s] 2026-03-03 06:46:26 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/congress-events/_clone/congress-events-historical [status:200 duration:0.712s] 2026-03-03 06:46:26 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/congress-events/_settings [status:200 duration:0.098s] 2026-03-03 06:46:26 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/congress-events-historical/_settings [status:200 duration:0.077s] 2026-03-03 06:46:27 [scrapy.crawler] INFO: Received SIGINT, shutting down gracefully. Send again to force 2026-03-03 06:46:33 [elastic_transport.transport] INFO: POST https://elasticsearch-es-http.elasticsearch.svc:9200/_reindex?wait_for_completion=true [status:200 duration:6.902s] 2026-03-03 06:46:33 [elastic_transport.transport] INFO: POST https://elasticsearch-es-http.elasticsearch.svc:9200/congress-events/_delete_by_query [status:200 duration:0.273s] 2026-03-03 06:46:33 [AAN (Events)] INFO: Events have been reindexed. 2026-03-03 06:46:33 [elastic_transport.transport] INFO: HEAD https://elasticsearch-es-http.elasticsearch.svc:9200/congress-fields [status:200 duration:0.003s] 2026-03-03 06:46:33 [AAN (Events)] INFO: Delete existing congress fields before saving new ones. 2026-03-03 06:46:33 [elastic_transport.transport] INFO: POST https://elasticsearch-es-http.elasticsearch.svc:9200/congress-fields/_delete_by_query [status:200 duration:0.021s] 2026-03-03 06:46:33 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)