2026-09-30 15:07:16 [scrapy.utils.log] INFO: Scrapy 2.18.0 started (bot: grabbers)
2026-09-30 15:07:16 [scrapy.utils.log] INFO: Versions:
{'lxml': '6.1.3',
 'libxml2': '2.14.6',
 'cssselect': '1.5.0',
 'parsel': '1.12.1',
 'w3lib': '2.4.1',
 'Twisted': '26.4.0',
 'Python': '3.14.7 (main, Sep  1 2026, 14:18:09) [Clang 22.1.3 ]',
 'pyOpenSSL': '26.4.0 (OpenSSL 4.0.2 25 Aug 2026)',
 'cryptography': '50.0.1',
 'Platform': 'Linux-6.18.38-73.137.amzn2023.x86_64-x86_64-with-glibc2.35'}
2026-09-30 15:07:16 [scrapy.addons] INFO: Enabled addons:
[]
2026-09-30 15:07:16 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.logcount.LogCount',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats',
 'grabbers.extensions.custom_fields.ExportCustomFieldsExtension',
 'grabbers.extensions.kol_roles.ExportKolRolesExtension']
2026-09-30 15:07:16 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'grabbers',
 'FEED_EXPORT_FIELDS': ['External Id',
                        'Name',
                        'Start date',
                        'Plain text: Parent Name',
                        'Start time',
                        'End date',
                        'End time',
                        'Plain text: Event Link',
                        'Plain text: Event Link (Excel)',
                        'Filter: Session type',
                        'Plain text: Speakers',
                        'Plain text: Presentation number',
                        'Is all day',
                        'Plain text: Author(s)'],
 'LOG_FILE': '/etc/scrapyd/dbs/logs/grabbers/ESMO Congress '
             'Posters/14e23782bce011f1bd97b3dae8fe8c65.log',
 'LOG_LEVEL': 'INFO',
 'NEWSPIDER_MODULE': 'grabbers.spiders',
 'SPIDER_MODULES': ['grabbers.spiders'],
 'STATS_DUMP': 'True',
 'TELNETCONSOLE_ENABLED': False,
 'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
               '(KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36'}
2026-09-30 15:07:17 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'grabbers.middlewares.WafSolverMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2026-09-30 15:07:17 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.start.StartSpiderMiddleware',
 'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'grabbers.middlewares.SilentValidationMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware',
 'scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware']
2026-09-30 15:07:17 [scrapy.middleware] INFO: Enabled item pipelines:
['grabbers.pipelines.CleanTextPipeline',
 'grabbers.pipelines.DateTimeFormatPipeline',
 'grabbers.pipelines.BoolFormatPipeline',
 'grabbers.pipelines.ValidateEventNamePipeline',
 'grabbers.pipelines.CountryCodePipeline',
 'grabbers.pipelines.HyperlinkedEventLinkPipeline',
 'grabbers.pipelines.ElasticSearchPipeline']
2026-09-30 15:07:17 [scrapy.core.engine] INFO: Spider opened
2026-09-30 15:07:17 [elastic_transport.transport] INFO: HEAD https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events [status:200 duration:0.017s]
2026-09-30 15:07:17 [ESMO Congress Posters] INFO: Delete existing congress events before saving new ones.
2026-09-30 15:07:17 [elastic_transport.transport] INFO: POST https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_delete_by_query [status:200 duration:0.196s]
2026-09-30 15:07:17 [ESMO Congress Posters] INFO: Existing congress events have been deleted.
2026-09-30 15:07:17 [elastic_transport.transport] INFO: HEAD https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-congress-fields [status:200 duration:0.003s]
2026-09-30 15:07:17 [ESMO Congress Posters] INFO: Delete existing congress fields before saving new ones.
2026-09-30 15:07:17 [elastic_transport.transport] INFO: POST https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-congress-fields/_delete_by_query [status:200 duration:0.020s]
2026-09-30 15:07:17 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2026-09-30 15:07:18 [grabbers.middlewares.waf_solver] INFO: AWS WAF challenge detected and solved for domain cslide.ctimeetingtech.com.
2026-09-30 15:08:17 [scrapy.extensions.logstats] INFO: Crawled 75 pages (at 75 pages/min), scraped 0 items (at 0 items/min)
2026-09-30 15:09:17 [scrapy.extensions.logstats] INFO: Crawled 149 pages (at 74 pages/min), scraped 0 items (at 0 items/min)
2026-09-30 15:10:17 [scrapy.extensions.logstats] INFO: Crawled 223 pages (at 74 pages/min), scraped 0 items (at 0 items/min)