{"action":"created","issue":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/1455","labels_url":"https://api.github.com/repos/scrapy/scrapy/issues/1455/labels{/name}","comments_url":"https://api.github.com/repos/scrapy/scrapy/issues/1455/comments","events_url":"https://api.github.com/repos/scrapy/scrapy/issues/1455/events","html_url":"https://github.com/scrapy/scrapy/pull/1455","id":102880781,"number":1455,"title":"[WIP] POC for asyncio+aiohttp","user":{"login":"dangra","id":37369,"avatar_url":"https://avatars.githubusercontent.com/u/37369?v=3","gravatar_id":"","url":"https://api.github.com/users/dangra","html_url":"https://github.com/dangra","followers_url":"https://api.github.com/users/dangra/followers","following_url":"https://api.github.com/users/dangra/following{/other_user}","gists_url":"https://api.github.com/users/dangra/gists{/gist_id}","starred_url":"https://api.github.com/users/dangra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dangra/subscriptions","organizations_url":"https://api.github.com/users/dangra/orgs","repos_url":"https://api.github.com/users/dangra/repos","events_url":"https://api.github.com/users/dangra/events{/privacy}","received_events_url":"https://api.github.com/users/dangra/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-08-24T20:08:33Z","updated_at":"2015-08-26T21:29:33Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/scrapy/scrapy/pulls/1455","html_url":"https://github.com/scrapy/scrapy/pull/1455","diff_url":"https://github.com/scrapy/scrapy/pull/1455.diff","patch_url":"https://github.com/scrapy/scrapy/pull/1455.patch"},"body":"** DO NOT MERGE - Strongly toxic and cool drug here **\r\n\r\nIntegration of a http downloader handler based on [aiohttp](https://github.com/KeepSafe/aiohttp) and asyncio on python3.\r\n\r\nbehind the scenes It hooks asyncio loop as a twisted reactor (provided by [txtulip](https://github.com/itamarst/txtulip)).\r\n\r\nTests are not supposed to pass, this is just a working proof of concept!\r\n\r\nClone [testspiders's scrapy project](https://github.com/scrapinghub/testspiders) and run `scrapy crawl justfollow`, see how smoothly it hits websites :)\r\n\r\n\r\n```\r\n$ scrapy crawl justfollow\r\n2015-08-24 17:10:10 [scrapy] INFO: Scrapy 1.1.0dev1 started (bot: testspiders)\r\n2015-08-24 17:10:10 [scrapy] INFO: Optional features available: ssl, http11\r\n2015-08-24 17:10:10 [scrapy] INFO: Overridden settings: {'CLOSESPIDER_TIMEOUT': 3600, 'RETRY_ENABLED': False, 'NEWSPIDER_MODULE': 'testspiders.spiders', 'SPIDER_MODULES': ['testspiders.spiders'], 'CLOSESPIDER_PAGECOUNT': 1000, 'COOKIES_ENABLED': False, 'BOT_NAME': 'testspiders'}\r\n2015-08-24 17:10:11 [scrapy] INFO: Enabled extensions: SpiderState, CoreStats, CloseSpider, LogStats\r\n2015-08-24 17:10:11 [scrapy] INFO: Enabled downloader middlewares: RandomUserAgent, ErrorMonkeyMiddleware, HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, ChunkedTransferMiddleware, DownloaderStats\r\n2015-08-24 17:10:11 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware\r\n2015-08-24 17:10:11 [scrapy] INFO: Enabled item pipelines: \r\n2015-08-24 17:10:11 [scrapy] INFO: Spider opened\r\n2015-08-24 17:10:11 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)\r\n2015-08-24 17:10:11 [scrapy] DEBUG: Crawled (200) <GET http://scrapinghub.com> (referer: None)\r\n2015-08-24 17:10:11 [scrapy] DEBUG: Filtered duplicate request: <GET http://scrapinghub.com/platform/> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Crawled (200) <GET http://scrapinghub.com/pricing/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Crawled (200) <GET http://scrapinghub.com> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Crawled (200) <GET http://scrapinghub.com/platform/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Redirecting (301) to <GET http://www.parsely.com/> from <GET http://parsely.com/>\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Redirecting (301) to <GET https://eventup.com/> from <GET http://eventup.com/>\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Crawled (200) <GET http://www.swoop.com/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Crawled (200) <GET http://appmonsta.com/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Redirecting (301) to <GET https://p2pi.org/> from <GET http://p2pi.org/>\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Redirecting (301) to <GET https://www.linkedin.com/in/jacobperkins/> from <GET http://www.linkedin.com/in/jacobperkins/>\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Redirecting (301) to <GET http://fabkids.com/wb/> from <GET http://wittlebee.com/>\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Crawled (200) <GET http://scrapinghub.com/abuse-report/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Crawled (200) <GET http://scrapinghub.com/tos/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Crawled (200) <GET http://science-inc.com/> (referer: b'http://scrapinghub.com')\r\n^C2015-08-24 17:10:12 [scrapy] INFO: Received SIG_SETMASK, shutting down gracefully. Send again to force \r\n2015-08-24 17:10:12 [scrapy] INFO: Closing spider (shutdown)\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Redirecting (301) to <GET https://mydeco.com/> from <GET http://mydeco.com/>\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Redirecting (302) to <GET https://dash.scrapinghub.com/account/login/?next=/> from <GET https://dash.scrapinghub.com/>\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Crawled (200) <GET http://www.parsely.com/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:13 [scrapy] DEBUG: Crawled (200) <GET http://appmonsta.com/dashboard/api-documentation/> (referer: b'http://appmonsta.com/')\r\n2015-08-24 17:10:13 [scrapy] DEBUG: Redirecting (301) to <GET https://www.olark.com/?welcome> from <GET http://www.olark.com?welcome>\r\n2015-08-24 17:10:13 [scrapy] DEBUG: Crawled (200) <GET https://eventup.com/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:13 [scrapy] DEBUG: Crawled (200) <GET http://status.scrapinghub.com/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:13 [scrapy] DEBUG: Crawled (200) <GET http://ar.linkedin.com/in/juancatalano> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:13 [scrapy] DEBUG: Redirecting (302) to <GET http://www.fabkids.com/wb/> from <GET http://fabkids.com/wb/>\r\n2015-08-24 17:10:13 [scrapy] DEBUG: Crawled (200) <GET https://www.linkedin.com/pub/terese-herbig/1/137/370> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:13 [scrapy] DEBUG: Crawled (200) <GET http://scrapinghub.com/screenshots/large/forum.png> (referer: b'http://scrapinghub.com/platform/')\r\n2015-08-24 17:10:13 [scrapy] DEBUG: Crawled (200) <GET https://p2pi.org/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:13 [scrapy] DEBUG: Crawled (200) <GET https://www.olark.com/site/2109-478-10-2394/contact> (referer: b'http://appmonsta.com/')\r\n2015-08-24 17:10:14 [scrapy] DEBUG: Crawled (200) <GET https://www.linkedin.com/in/jacobperkins/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:15 [aiohttp.client] WARNING: Can not load response cookies: Illegal key value: wpgb_visit_last_php-http://scrapinghub_com\r\n2015-08-24 17:10:15 [scrapy] DEBUG: Crawled (200) <GET http://streamhacker.com> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:20 [scrapy] DEBUG: Crawled (200) <GET http://scrapinghub.com/screenshots/large/kb.png> (referer: b'http://scrapinghub.com/platform/')\r\n2015-08-24 17:10:21 [scrapy] INFO: Dumping Scrapy stats:\r\n{'downloader/request_bytes': 9925,\r\n 'downloader/request_count': 30,\r\n 'downloader/request_method_count/GET': 30,\r\n 'downloader/response_bytes': 1157700,\r\n 'downloader/response_count': 30,\r\n 'downloader/response_status_count/200': 21,\r\n 'downloader/response_status_count/301': 7,\r\n 'downloader/response_status_count/302': 2,\r\n 'dupefilter/filtered': 667,\r\n 'finish_reason': 'shutdown',\r\n 'finish_time': datetime.datetime(2015, 8, 24, 20, 10, 21, 100933),\r\n 'log_count/DEBUG': 31,\r\n 'log_count/INFO': 8,\r\n 'log_count/WARNING': 1,\r\n 'request_depth_max': 3,\r\n 'response_received_count': 21,\r\n 'scheduler/dequeued': 30,\r\n 'scheduler/dequeued/memory': 30,\r\n 'scheduler/enqueued': 839,\r\n 'scheduler/enqueued/memory': 839,\r\n 'start_time': datetime.datetime(2015, 8, 24, 20, 10, 11, 321827)}\r\n2015-08-24 17:10:21 [scrapy] INFO: Spider closed (shutdown)\r\n```"},"comment":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/comments/135175579","html_url":"https://github.com/scrapy/scrapy/pull/1455#issuecomment-135175579","issue_url":"https://api.github.com/repos/scrapy/scrapy/issues/1455","id":135175579,"user":{"login":"dangra","id":37369,"avatar_url":"https://avatars.githubusercontent.com/u/37369?v=3","gravatar_id":"","url":"https://api.github.com/users/dangra","html_url":"https://github.com/dangra","followers_url":"https://api.github.com/users/dangra/followers","following_url":"https://api.github.com/users/dangra/following{/other_user}","gists_url":"https://api.github.com/users/dangra/gists{/gist_id}","starred_url":"https://api.github.com/users/dangra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dangra/subscriptions","organizations_url":"https://api.github.com/users/dangra/orgs","repos_url":"https://api.github.com/users/dangra/repos","events_url":"https://api.github.com/users/dangra/events{/privacy}","received_events_url":"https://api.github.com/users/dangra/received_events","type":"User","site_admin":false},"created_at":"2015-08-26T21:29:33Z","updated_at":"2015-08-26T21:29:33Z","body":"> we deprecate it\r\n\r\n@nramirezuy jajaja\r\n\r\n@chekunkov I guess asyncio integration will be experimental for a while and only available for py3."}}