2026-08-11 00:01:58 [scrapy.utils.log] INFO: Scrapy 2.17.0 started (bot: grabbers) 2026-08-11 00:01:58 [scrapy.utils.log] INFO: Versions: {'lxml': '6.1.1', 'libxml2': '2.14.6', 'cssselect': '1.5.0', 'parsel': '1.11.0', 'w3lib': '2.4.1', 'Twisted': '26.4.0', 'Python': '3.14.6 (main, Jul 18 2026, 17:01:31) [Clang 22.1.3 ]', 'pyOpenSSL': '26.4.0 (OpenSSL 4.0.1 9 Jun 2026)', 'cryptography': '50.0.0', 'Platform': 'Linux-6.12.94-123.190.amzn2023.x86_64-x86_64-with-glibc2.35'} 2026-08-11 00:01:58 [scrapy.addons] INFO: Enabled addons: [] 2026-08-11 00:01:58 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.logcount.LogCount', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.logstats.LogStats', 'grabbers.extensions.custom_fields.ExportCustomFieldsExtension', 'grabbers.extensions.kol_roles.ExportKolRolesExtension'] 2026-08-11 00:01:58 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'grabbers', 'DOWNLOAD_DELAY': 0.3, 'FEED_EXPORT_FIELDS': ['External Id', 'Parent External Id', 'Name', 'Plain text: Parent Name', 'Start date', 'Start time', 'End date', 'End time', 'Location', 'Plain text: Event Link', 'Plain text: Event Link (Excel)', 'Filter: Tag', 'Plain text: Learning Objectives', 'Plain text: Details', 'Filter: Badges', 'Filter: Separate registration required', 'Filter: No CME for this activity', 'Filter: This session will be broadcast live. ' 'Livestreamed presentations will be recorded and added ' 'to the Platform after the meeting.', 'Filter: A video recording will be available during or ' 'after the meeting.', 'Filter: ACG Award-Winning Abstract', 'Plain text: Speaker(s)', 'Plain text: Bonus Session Speaker(s)', 'Plain text: Breakfast Session Moderator(s)', 'Plain text: Breakfast Session Presenter(s)', 'Plain text: Course Director(s)', 'Plain text: Hands-on Faculty Leader(s)', 'Plain text: Hands-on Faculty(s)', 'Plain text: Hands-on Instructor(s)', 'Plain text: Hands-on Workshop Director(s)', 'Plain text: Introduction Presenter(s)', 'Plain text: Invited Lecturer(s)', 'Plain text: Lunch Session Speaker(s)', 'Plain text: Moderator(s)', 'Plain text: Panelist(s)', 'Plain text: Past President(s)', 'Plain text: Presenter(s)', 'Plain text: Presenting Author - Endoscopy Video ' 'Forum(s)', 'Plain text: Presenting Author(s)', 'Plain text: Faculty(s)', 'Plain text: Judge(s)', 'Plain text: Video Presenter(s)'], 'LOG_FILE': '/etc/scrapyd/dbs/logs/grabbers/ACG ' '2025/dac87ee8951711f1a9a57f7371225765.log', 'LOG_LEVEL': 'INFO', 'NEWSPIDER_MODULE': 'grabbers.spiders', 'SPIDER_MODULES': ['grabbers.spiders'], 'TELNETCONSOLE_ENABLED': False, 'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 ' 'Firefox/131.0'} 2026-08-11 00:01:59 [scrapy_fake_useragent.middleware] INFO: Using '' as the User-Agent provider 2026-08-11 00:01:59 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'grabbers.middlewares.RandomUserAgentMiddleware', 'grabbers.middlewares.RetryUserAgentMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2026-08-11 00:01:59 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.start.StartSpiderMiddleware', 'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'grabbers.middlewares.SilentValidationMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2026-08-11 00:01:59 [scrapy.middleware] INFO: Enabled item pipelines: ['grabbers.pipelines.CleanTextPipeline', 'grabbers.pipelines.DateTimeFormatPipeline', 'grabbers.pipelines.BoolFormatPipeline', 'grabbers.pipelines.ValidateEventNamePipeline', 'grabbers.pipelines.CountryCodePipeline', 'grabbers.pipelines.HyperlinkedEventLinkPipeline', 'grabbers.pipelines.ElasticSearchPipeline'] 2026-08-11 00:01:59 [scrapy.core.engine] INFO: Spider opened 2026-08-11 00:01:59 [elastic_transport.transport] INFO: HEAD https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events [status:200 duration:0.117s] 2026-08-11 00:01:59 [ACG 2025] INFO: Delete existing congress events before saving new ones. 2026-08-11 00:01:59 [elastic_transport.transport] INFO: POST https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_delete_by_query [status:200 duration:0.025s] 2026-08-11 00:01:59 [ACG 2025] INFO: Existing congress events have been deleted. 2026-08-11 00:01:59 [elastic_transport.transport] INFO: HEAD https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-congress-fields [status:200 duration:0.005s] 2026-08-11 00:01:59 [ACG 2025] INFO: Delete existing congress fields before saving new ones. 2026-08-11 00:01:59 [elastic_transport.transport] INFO: POST https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-congress-fields/_delete_by_query [status:200 duration:0.015s] 2026-08-11 00:01:59 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2026-08-11 00:02:09 [scrapy_fake_useragent.middleware] INFO: Using '' as the User-Agent provider 2026-08-11 00:02:10 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_The%20American%20Journal%20of%20Gastroenterology%20Lecture_2025-10-27_15%3A05%3A00 [status:201 duration:0.008s] 2026-08-11 00:02:10 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_The%20American%20Journal%20of%20Gastroenterology%20Lecture_2025-10-27_15%3A05%3A00 [status:200 duration:0.006s] 2026-08-11 00:02:12 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Endoscopy%20School%20Hands-On%20Workshop%20Center_2025-10-27_14%3A05%3A00 [status:201 duration:0.008s] 2026-08-11 00:02:12 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Endoscopy%20School%20Hands-On%20Workshop%20Center_2025-10-27_14%3A05%3A00 [status:200 duration:0.006s] 2026-08-11 00:02:12 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Endoscopy%20School%20Hands-On%20Workshop%20Center_2025-10-27_13%3A55%3A00 [status:201 duration:0.007s] 2026-08-11 00:02:12 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Endoscopy%20School%20Hands-On%20Workshop%20Center_2025-10-27_13%3A55%3A00 [status:200 duration:0.006s] 2026-08-11 00:02:36 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_David%20Y.%20Graham%20Lecture_2025-10-29_10%3A20%3A00 [status:201 duration:0.007s] 2026-08-11 00:02:36 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_David%20Y.%20Graham%20Lecture_2025-10-29_10%3A20%3A00 [status:200 duration:0.006s] 2026-08-11 00:02:47 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Emily%20Couric%20Memorial%20Lecture_2025-10-28_16%3A30%3A00 [status:201 duration:0.007s] 2026-08-11 00:02:47 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Emily%20Couric%20Memorial%20Lecture_2025-10-28_16%3A30%3A00 [status:200 duration:0.006s] 2026-08-11 00:02:59 [scrapy.extensions.logstats] INFO: Crawled 70 pages (at 70 pages/min), scraped 5 items (at 5 items/min) 2026-08-11 00:03:12 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Endoscopy%20School%20Hands-On%20Workshop%20Center_2025-10-28_13%3A55%3A00 [status:201 duration:0.007s] 2026-08-11 00:03:12 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Endoscopy%20School%20Hands-On%20Workshop%20Center_2025-10-28_13%3A55%3A00 [status:200 duration:0.015s] 2026-08-11 00:03:49 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Endoscopy%20School%20Hands-On%20Workshop%20Center_2025-10-28_10%3A20%3A00 [status:201 duration:0.007s] 2026-08-11 00:03:49 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Endoscopy%20School%20Hands-On%20Workshop%20Center_2025-10-28_10%3A20%3A00 [status:200 duration:0.006s] 2026-08-11 00:03:50 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Endoscopy%20School%20Hands-On%20Workshop%20Center_2025-10-28_10%3A05%3A00 [status:201 duration:0.007s] 2026-08-11 00:03:50 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Endoscopy%20School%20Hands-On%20Workshop%20Center_2025-10-28_10%3A05%3A00 [status:200 duration:0.006s] 2026-08-11 00:03:50 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_ACG%27s%20J.%20Edward%20Berk%20Distinguished%20Lecture_2025-10-28_10%3A05%3A00 [status:201 duration:0.007s] 2026-08-11 00:03:50 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_ACG%27s%20J.%20Edward%20Berk%20Distinguished%20Lecture_2025-10-28_10%3A05%3A00 [status:200 duration:0.006s] 2026-08-11 00:03:59 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_GI%20Ignite%20Meet%20the%20Experts%20Breakfast%20Sessions_2025-10-28_07%3A15%3A00 [status:201 duration:0.007s] 2026-08-11 00:03:59 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_GI%20Ignite%20Meet%20the%20Experts%20Breakfast%20Sessions_2025-10-28_07%3A15%3A00 [status:200 duration:0.006s] 2026-08-11 00:03:59 [scrapy.extensions.logstats] INFO: Crawled 154 pages (at 84 pages/min), scraped 10 items (at 5 items/min) 2026-08-11 00:04:04 [ACG 2025] WARNING: Some events don't have times. Please set `ENABLE_MISSING_TIME_LOGS` environment variable to `1` to see detailed logs about these events. 2026-08-11 00:04:04 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20P%3A%20Management%20of%20Liver%20Lesions%3A%20What%20to%20Do%20and%20Where%20Are%20We%20Going%3F_2025-10-28_ [status:201 duration:0.009s] 2026-08-11 00:04:04 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20P%3A%20Management%20of%20Liver%20Lesions%3A%20What%20to%20Do%20and%20Where%20Are%20We%20Going%3F_2025-10-28_ [status:200 duration:0.007s] 2026-08-11 00:04:21 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20O%3A%20Hot%20Topics%20in%20Hepatology_2025-10-28_ [status:201 duration:0.007s] 2026-08-11 00:04:21 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20O%3A%20Hot%20Topics%20in%20Hepatology_2025-10-28_ [status:200 duration:0.014s] 2026-08-11 00:04:21 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20N%3A%20Anorectal%20Lowdown_2025-10-28_ [status:201 duration:0.008s] 2026-08-11 00:04:21 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20N%3A%20Anorectal%20Lowdown_2025-10-28_ [status:200 duration:0.006s] 2026-08-11 00:04:21 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20M%3A%20Gastric%20Slowdown_2025-10-28_ [status:201 duration:0.007s] 2026-08-11 00:04:21 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20M%3A%20Gastric%20Slowdown_2025-10-28_ [status:200 duration:0.006s] 2026-08-11 00:04:22 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20L%3A%20Pancreatic%20Cancer%3A%20From%20Detection%20to%20Palliation_2025-10-28_ [status:201 duration:0.007s] 2026-08-11 00:04:22 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20L%3A%20Pancreatic%20Cancer%3A%20From%20Detection%20to%20Palliation_2025-10-28_ [status:200 duration:0.007s] 2026-08-11 00:04:23 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20K%20%3A%20Colon%20Poop-Pourri_2025-10-28_ [status:201 duration:0.007s] 2026-08-11 00:04:23 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20K%20%3A%20Colon%20Poop-Pourri_2025-10-28_ [status:200 duration:0.006s] 2026-08-11 00:04:24 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20J%3A%20Utilization%20of%20Advanced%20Practice%20Providers%20%28APPs%29%20in%20Team-Based%20Care_2025-10-28_ [status:201 duration:0.007s] 2026-08-11 00:04:24 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20J%3A%20Utilization%20of%20Advanced%20Practice%20Providers%20%28APPs%29%20in%20Team-Based%20Care_2025-10-28_ [status:200 duration:0.006s] 2026-08-11 00:04:25 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20I%3A%20Integrating%20Lifestyle%20and%20Patient-Centered%20Care%20in%20IBD_2025-10-28_ [status:201 duration:0.007s] 2026-08-11 00:04:25 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20I%3A%20Integrating%20Lifestyle%20and%20Patient-Centered%20Care%20in%20IBD_2025-10-28_ [status:200 duration:0.007s] 2026-08-11 00:04:42 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Symposium%202C%3A%20Endoscopy%20Video%20Forum_2025-10-27_16%3A45%3A00 [status:201 duration:0.007s] 2026-08-11 00:04:42 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Symposium%202C%3A%20Endoscopy%20Video%20Forum_2025-10-27_16%3A45%3A00 [status:200 duration:0.007s] 2026-08-11 00:04:59 [scrapy.extensions.logstats] INFO: Crawled 241 pages (at 87 pages/min), scraped 19 items (at 9 items/min) 2026-08-11 00:05:05 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Endoscopy%20School%20Hands-On%20Workshop%20Center_2025-10-27_10%3A30%3A00 [status:201 duration:0.007s] 2026-08-11 00:05:05 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Endoscopy%20School%20Hands-On%20Workshop%20Center_2025-10-27_10%3A30%3A00 [status:200 duration:0.006s] 2026-08-11 00:05:18 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Endoscopy%20School%20Hands-On%20Workshop%20Center_2025-10-27_10%3A15%3A00 [status:201 duration:0.007s] 2026-08-11 00:05:18 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Endoscopy%20School%20Hands-On%20Workshop%20Center_2025-10-27_10%3A15%3A00 [status:200 duration:0.006s] 2026-08-11 00:05:19 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_President%27s%20Address_2025-10-27_09%3A00%3A00 [status:201 duration:0.007s] 2026-08-11 00:05:19 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_President%27s%20Address_2025-10-27_09%3A00%3A00 [status:200 duration:0.006s] 2026-08-11 00:05:28 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Endoscopy%20School%20Hands-On%20Workshop%20Center_2025-10-27_07%3A45%3A00 [status:201 duration:0.012s] 2026-08-11 00:05:28 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Endoscopy%20School%20Hands-On%20Workshop%20Center_2025-10-27_07%3A45%3A00 [status:200 duration:0.014s] 2026-08-11 00:05:29 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_GI%20Ignite%20Meet%20the%20Experts%20Breakfast%20Sessions_2025-10-27_06%3A45%3A00 [status:201 duration:0.008s] 2026-08-11 00:05:29 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_GI%20Ignite%20Meet%20the%20Experts%20Breakfast%20Sessions_2025-10-27_06%3A45%3A00 [status:200 duration:0.006s] 2026-08-11 00:05:46 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20H%3A%20Great%20Debates%20in%20Endoscopy_2025-10-27_ [status:201 duration:0.007s] 2026-08-11 00:05:46 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20H%3A%20Great%20Debates%20in%20Endoscopy_2025-10-27_ [status:200 duration:0.006s] 2026-08-11 00:05:46 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20G%3A%20Gastrointestinal%20Endoscopy%20in%202025%20and%20Beyond_2025-10-27_ [status:201 duration:0.008s] 2026-08-11 00:05:46 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20G%3A%20Gastrointestinal%20Endoscopy%20in%202025%20and%20Beyond_2025-10-27_ [status:200 duration:0.006s] 2026-08-11 00:05:46 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20F%3A%20Expanding%20the%20Horizons%20of%20Esophageal%20Endoscopy_2025-10-27_ [status:201 duration:0.006s] 2026-08-11 00:05:46 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20F%3A%20Expanding%20the%20Horizons%20of%20Esophageal%20Endoscopy_2025-10-27_ [status:200 duration:0.006s] 2026-08-11 00:05:47 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20E%3A%20Tools%20and%20Treatments%20for%20Esophageal%20Motility%20Disorders_2025-10-27_ [status:201 duration:0.007s] 2026-08-11 00:05:47 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20E%3A%20Tools%20and%20Treatments%20for%20Esophageal%20Motility%20Disorders_2025-10-27_ [status:200 duration:0.006s] 2026-08-11 00:05:48 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20D%3A%20Incorporating%20Management%20of%20Obesity%20Into%20Your%20Gastroenterology%20Practice_2025-10-27_ [status:201 duration:0.007s] 2026-08-11 00:05:48 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20D%3A%20Incorporating%20Management%20of%20Obesity%20Into%20Your%20Gastroenterology%20Practice_2025-10-27_ [status:200 duration:0.007s] 2026-08-11 00:05:49 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20C%3A%20Building%20Your%20GI%20Team_2025-10-27_ [status:201 duration:0.007s] 2026-08-11 00:05:49 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20C%3A%20Building%20Your%20GI%20Team_2025-10-27_ [status:200 duration:0.006s] 2026-08-11 00:05:49 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20B%3A%20The%20New%20Kids%20on%20the%20IBD%20Block%3A%20Treatment%20in%202025_2025-10-27_ [status:201 duration:0.007s] 2026-08-11 00:05:49 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20B%3A%20The%20New%20Kids%20on%20the%20IBD%20Block%3A%20Treatment%20in%202025_2025-10-27_ [status:200 duration:0.006s] 2026-08-11 00:05:49 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20A%3A%20Staying%20One%20Step%20Ahead%3A%20Monitoring%20in%20IBD_2025-10-27_ [status:201 duration:0.007s] 2026-08-11 00:05:49 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Bonus%20Session%20A%3A%20Staying%20One%20Step%20Ahead%3A%20Monitoring%20in%20IBD_2025-10-27_ [status:200 duration:0.006s] 2026-08-11 00:05:59 [scrapy.extensions.logstats] INFO: Crawled 333 pages (at 92 pages/min), scraped 32 items (at 13 items/min) 2026-08-11 00:06:14 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Endoscopy%20School%20Hands-On%20Workshop%20Center_2025-10-26_16%3A00%3A00 [status:201 duration:0.007s] 2026-08-11 00:06:14 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Endoscopy%20School%20Hands-On%20Workshop%20Center_2025-10-26_16%3A00%3A00 [status:200 duration:0.007s] 2026-08-11 00:06:27 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Optional%20Learning%20Luncheons%20%28registration%20required%20%24%24%24%29_2025-10-26_12%3A20%3A00 [status:201 duration:0.007s] 2026-08-11 00:06:27 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Optional%20Learning%20Luncheons%20%28registration%20required%20%24%24%24%29_2025-10-26_12%3A20%3A00 [status:200 duration:0.006s] 2026-08-11 00:06:59 [scrapy.extensions.logstats] INFO: Crawled 415 pages (at 82 pages/min), scraped 34 items (at 2 items/min) 2026-08-11 00:07:15 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Optional%20Learning%20Luncheons%20%28registration%20required%20%24%24%24%29_2025-10-25_12%3A20%3A00 [status:201 duration:0.007s] 2026-08-11 00:07:15 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Optional%20Learning%20Luncheons%20%28registration%20required%20%24%24%24%29_2025-10-25_12%3A20%3A00 [status:200 duration:0.013s] 2026-08-11 00:07:44 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_ACG%27s%20Trailblazer%20Lecture_2025-10-25_10%3A00%3A00 [status:201 duration:0.007s] 2026-08-11 00:07:44 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_ACG%27s%20Trailblazer%20Lecture_2025-10-25_10%3A00%3A00 [status:200 duration:0.006s] 2026-08-11 00:07:59 [scrapy.extensions.logstats] INFO: Crawled 496 pages (at 81 pages/min), scraped 36 items (at 2 items/min) 2026-08-11 00:08:17 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Break_2025-10-24_15%3A40%3A00 [status:201 duration:0.007s] 2026-08-11 00:08:17 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-events/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0_Break_2025-10-24_15%3A40%3A00 [status:200 duration:0.014s] 2026-08-11 00:08:59 [scrapy.extensions.logstats] INFO: Crawled 578 pages (at 82 pages/min), scraped 37 items (at 1 items/min) 2026-08-11 00:09:32 [scrapy.core.engine] INFO: Closing spider (finished) 2026-08-11 00:09:32 [ACG 2025] ERROR: Pydantic validation errors during parsing: Field 'parent_name': This field is not specified in `ACGEvent`. Consider adding it as `TextField` or `FilterField`. Field 'separate_registration': Input should be a valid string input: Field 'no_cme': Input should be a valid string input: Field 'livestream': Input should be a valid string input: Field 'video_recording': Input should be a valid string input: Field 'acg_award': Input should be a valid string input: 2026-08-11 00:09:32 [ACG 2025] ERROR: ACG 2025: Some fields are missing. Please add next fields to your Event class (Don't forget to update types accordingly): acg_newsworthy_abstract acg_health_equity_research_abstract_of_distinction_award late_breaking_abstract_presenter 2026-08-11 00:09:32 [ACG 2025] INFO: Save the actual congress fields. 2026-08-11 00:09:32 [elastic_transport.transport] INFO: PUT https://elasticsearch-es-http.elasticsearch.svc:9200/scrapyd-congress-fields/_doc/acg-2025-2a84ad0179299986123ff5035b33eddbc0be73e96ad678e17870df8d4887d7e0 [status:201 duration:0.008s] 2026-08-11 00:09:32 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 1371592, 'downloader/request_count': 1220, 'downloader/request_method_count/GET': 1220, 'downloader/response_bytes': 12085203, 'downloader/response_count': 1220, 'downloader/response_status_count/200': 629, 'downloader/response_status_count/302': 591, 'elapsed_time_seconds': 452.87353931809776, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2026, 8, 11, 0, 9, 32, 869918, tzinfo=datetime.timezone.utc), 'httpcompression/response_bytes': 27426022, 'httpcompression/response_count': 629, 'item_scraped_count': 37, 'items_per_minute': 4.911504424778761, 'log_count/ERROR': 2, 'log_count/INFO': 84, 'log_count/WARNING': 1, 'memusage/max': 235687936, 'memusage/startup': 222277632, 'request_depth_max': 2, 'response_received_count': 629, 'responses_per_minute': 83.49557522123894, 'scheduler/dequeued': 1220, 'scheduler/dequeued/memory': 1220, 'scheduler/enqueued': 1220, 'scheduler/enqueued/memory': 1220, 'start_time': datetime.datetime(2026, 8, 11, 0, 1, 59, 996378, tzinfo=datetime.timezone.utc)} 2026-08-11 00:09:32 [scrapy.core.engine] INFO: Spider closed (finished)