{"action":"created","comment":{"url":"https://api.github.com/repos/scrapy/scrapy/pulls/comments/38265800","id":38265800,"diff_hunk":"@@ -0,0 +1,52 @@\n+\"\"\"Download handlers for http and https schemes\"\"\"\n+import logging\n+import asyncio\n+\n+import aiohttp\n+from twisted.internet import defer\n+\n+from scrapy.http import Headers\n+from scrapy.responsetypes import responsetypes\n+\n+logger = logging.getLogger(__name__)\n+\n+\n+class HTTPDownloadHandler(object):\n+\n+    def __init__(self, settings):\n+        self.settings = settings\n+\n+    def download_request(self, request, spider):\n+        return _force_deferred(self._download_request(request, spider))\n+\n+    def _download_request(self, request, spider):\n+        \"\"\"Return a deferred for the HTTP download\"\"\"\n+        headers=list((k.decode('latin1'), v.decode('latin1'))\n+                     for k, vs in request.headers.items()\n+                     for v in vs)\n+\n+        aioresponse = yield from aiohttp.request(","path":"scrapy/core/downloader/handlers/aiohttp.py","position":28,"original_position":28,"commit_id":"55c85dd57b9fbf54c2cbee496500ee11a3822300","original_commit_id":"55c85dd57b9fbf54c2cbee496500ee11a3822300","user":{"login":"dangra","id":37369,"avatar_url":"https://avatars.githubusercontent.com/u/37369?v=3","gravatar_id":"","url":"https://api.github.com/users/dangra","html_url":"https://github.com/dangra","followers_url":"https://api.github.com/users/dangra/followers","following_url":"https://api.github.com/users/dangra/following{/other_user}","gists_url":"https://api.github.com/users/dangra/gists{/gist_id}","starred_url":"https://api.github.com/users/dangra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dangra/subscriptions","organizations_url":"https://api.github.com/users/dangra/orgs","repos_url":"https://api.github.com/users/dangra/repos","events_url":"https://api.github.com/users/dangra/events{/privacy}","received_events_url":"https://api.github.com/users/dangra/received_events","type":"User","site_admin":false},"body":"thanks for pointing at ClientSession, I looked at it while writing this POC, it offers connection pooling and keepalive which is nice, but it also handles cookiejar which is not compatible with how Scrapy manages cookies so I preferred to avoid ClientSession for now.\r\n\r\nInstead of ClientSession, I was more keen to reuse TCPConnector and ProxyConnector instances but creating an ephemeral session on each request. I suppose temporal sessions objects is what `aiohttp.request()` does internally.\r\n\r\nAnother option is to extend ClientSession and overrride cookie handling with noops.\r\n\r\nThe basic feature-set for a successful replacement of Twisted http agents should provide:\r\n\r\n1- HTTP 1.1 support (http and https)\r\n2- Connection pooling\r\n3- Keepalive\r\n4- Proxy support (socks would be nice too, and sending https urls over plain text too)\r\n5- binding outgoing requests to specific source ips\r\n6- Stopping downloads by inspecting response headers before downloading whole body (Content-type and content-length checks usually)\r\n7- Degrade but do not fail to parse invalid responses\r\n8- Whole download timeout, not just connect or read timeouts.\r\n\r\nSome of this features are already implemented in current Scrapy download handler but others are missing and were part of [GSOC ideas](http://gsoc2015.scrapinghub.com/ideas/#download-handler).\r\n\r\nI appreciate the time you took to review the PR and I don't want to nag you with all of this. Personally I found your comment useful to brain dump all this stuff. thanks! ","created_at":"2015-08-29T20:44:03Z","updated_at":"2015-08-29T20:44:03Z","html_url":"https://github.com/scrapy/scrapy/pull/1455#discussion_r38265800","pull_request_url":"https://api.github.com/repos/scrapy/scrapy/pulls/1455","_links":{"self":{"href":"https://api.github.com/repos/scrapy/scrapy/pulls/comments/38265800"},"html":{"href":"https://github.com/scrapy/scrapy/pull/1455#discussion_r38265800"},"pull_request":{"href":"https://api.github.com/repos/scrapy/scrapy/pulls/1455"}}},"pull_request":{"url":"https://api.github.com/repos/scrapy/scrapy/pulls/1455","id":43215863,"html_url":"https://github.com/scrapy/scrapy/pull/1455","diff_url":"https://github.com/scrapy/scrapy/pull/1455.diff","patch_url":"https://github.com/scrapy/scrapy/pull/1455.patch","issue_url":"https://api.github.com/repos/scrapy/scrapy/issues/1455","number":1455,"state":"open","locked":false,"title":"[WIP] POC for asyncio+aiohttp","user":{"login":"dangra","id":37369,"avatar_url":"https://avatars.githubusercontent.com/u/37369?v=3","gravatar_id":"","url":"https://api.github.com/users/dangra","html_url":"https://github.com/dangra","followers_url":"https://api.github.com/users/dangra/followers","following_url":"https://api.github.com/users/dangra/following{/other_user}","gists_url":"https://api.github.com/users/dangra/gists{/gist_id}","starred_url":"https://api.github.com/users/dangra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dangra/subscriptions","organizations_url":"https://api.github.com/users/dangra/orgs","repos_url":"https://api.github.com/users/dangra/repos","events_url":"https://api.github.com/users/dangra/events{/privacy}","received_events_url":"https://api.github.com/users/dangra/received_events","type":"User","site_admin":false},"body":"** DO NOT MERGE - Strongly toxic and cool drug here **\r\n\r\nIntegration of a http downloader handler based on [aiohttp](https://github.com/KeepSafe/aiohttp) and asyncio on python3.\r\n\r\nbehind the scenes It hooks asyncio loop as a twisted reactor (provided by [txtulip](https://github.com/itamarst/txtulip)).\r\n\r\nTests are not supposed to pass, this is just a working proof of concept!\r\n\r\nClone [testspiders's scrapy project](https://github.com/scrapinghub/testspiders) and run `scrapy crawl justfollow`, see how smoothly it hits websites :)\r\n\r\n\r\n```\r\n$ scrapy crawl justfollow\r\n2015-08-24 17:10:10 [scrapy] INFO: Scrapy 1.1.0dev1 started (bot: testspiders)\r\n2015-08-24 17:10:10 [scrapy] INFO: Optional features available: ssl, http11\r\n2015-08-24 17:10:10 [scrapy] INFO: Overridden settings: {'CLOSESPIDER_TIMEOUT': 3600, 'RETRY_ENABLED': False, 'NEWSPIDER_MODULE': 'testspiders.spiders', 'SPIDER_MODULES': ['testspiders.spiders'], 'CLOSESPIDER_PAGECOUNT': 1000, 'COOKIES_ENABLED': False, 'BOT_NAME': 'testspiders'}\r\n2015-08-24 17:10:11 [scrapy] INFO: Enabled extensions: SpiderState, CoreStats, CloseSpider, LogStats\r\n2015-08-24 17:10:11 [scrapy] INFO: Enabled downloader middlewares: RandomUserAgent, ErrorMonkeyMiddleware, HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, ChunkedTransferMiddleware, DownloaderStats\r\n2015-08-24 17:10:11 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware\r\n2015-08-24 17:10:11 [scrapy] INFO: Enabled item pipelines: \r\n2015-08-24 17:10:11 [scrapy] INFO: Spider opened\r\n2015-08-24 17:10:11 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)\r\n2015-08-24 17:10:11 [scrapy] DEBUG: Crawled (200) <GET http://scrapinghub.com> (referer: None)\r\n2015-08-24 17:10:11 [scrapy] DEBUG: Filtered duplicate request: <GET http://scrapinghub.com/platform/> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Crawled (200) <GET http://scrapinghub.com/pricing/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Crawled (200) <GET http://scrapinghub.com> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Crawled (200) <GET http://scrapinghub.com/platform/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Redirecting (301) to <GET http://www.parsely.com/> from <GET http://parsely.com/>\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Redirecting (301) to <GET https://eventup.com/> from <GET http://eventup.com/>\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Crawled (200) <GET http://www.swoop.com/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Crawled (200) <GET http://appmonsta.com/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Redirecting (301) to <GET https://p2pi.org/> from <GET http://p2pi.org/>\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Redirecting (301) to <GET https://www.linkedin.com/in/jacobperkins/> from <GET http://www.linkedin.com/in/jacobperkins/>\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Redirecting (301) to <GET http://fabkids.com/wb/> from <GET http://wittlebee.com/>\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Crawled (200) <GET http://scrapinghub.com/abuse-report/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Crawled (200) <GET http://scrapinghub.com/tos/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Crawled (200) <GET http://science-inc.com/> (referer: b'http://scrapinghub.com')\r\n^C2015-08-24 17:10:12 [scrapy] INFO: Received SIG_SETMASK, shutting down gracefully. Send again to force \r\n2015-08-24 17:10:12 [scrapy] INFO: Closing spider (shutdown)\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Redirecting (301) to <GET https://mydeco.com/> from <GET http://mydeco.com/>\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Redirecting (302) to <GET https://dash.scrapinghub.com/account/login/?next=/> from <GET https://dash.scrapinghub.com/>\r\n2015-08-24 17:10:12 [scrapy] DEBUG: Crawled (200) <GET http://www.parsely.com/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:13 [scrapy] DEBUG: Crawled (200) <GET http://appmonsta.com/dashboard/api-documentation/> (referer: b'http://appmonsta.com/')\r\n2015-08-24 17:10:13 [scrapy] DEBUG: Redirecting (301) to <GET https://www.olark.com/?welcome> from <GET http://www.olark.com?welcome>\r\n2015-08-24 17:10:13 [scrapy] DEBUG: Crawled (200) <GET https://eventup.com/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:13 [scrapy] DEBUG: Crawled (200) <GET http://status.scrapinghub.com/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:13 [scrapy] DEBUG: Crawled (200) <GET http://ar.linkedin.com/in/juancatalano> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:13 [scrapy] DEBUG: Redirecting (302) to <GET http://www.fabkids.com/wb/> from <GET http://fabkids.com/wb/>\r\n2015-08-24 17:10:13 [scrapy] DEBUG: Crawled (200) <GET https://www.linkedin.com/pub/terese-herbig/1/137/370> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:13 [scrapy] DEBUG: Crawled (200) <GET http://scrapinghub.com/screenshots/large/forum.png> (referer: b'http://scrapinghub.com/platform/')\r\n2015-08-24 17:10:13 [scrapy] DEBUG: Crawled (200) <GET https://p2pi.org/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:13 [scrapy] DEBUG: Crawled (200) <GET https://www.olark.com/site/2109-478-10-2394/contact> (referer: b'http://appmonsta.com/')\r\n2015-08-24 17:10:14 [scrapy] DEBUG: Crawled (200) <GET https://www.linkedin.com/in/jacobperkins/> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:15 [aiohttp.client] WARNING: Can not load response cookies: Illegal key value: wpgb_visit_last_php-http://scrapinghub_com\r\n2015-08-24 17:10:15 [scrapy] DEBUG: Crawled (200) <GET http://streamhacker.com> (referer: b'http://scrapinghub.com')\r\n2015-08-24 17:10:20 [scrapy] DEBUG: Crawled (200) <GET http://scrapinghub.com/screenshots/large/kb.png> (referer: b'http://scrapinghub.com/platform/')\r\n2015-08-24 17:10:21 [scrapy] INFO: Dumping Scrapy stats:\r\n{'downloader/request_bytes': 9925,\r\n 'downloader/request_count': 30,\r\n 'downloader/request_method_count/GET': 30,\r\n 'downloader/response_bytes': 1157700,\r\n 'downloader/response_count': 30,\r\n 'downloader/response_status_count/200': 21,\r\n 'downloader/response_status_count/301': 7,\r\n 'downloader/response_status_count/302': 2,\r\n 'dupefilter/filtered': 667,\r\n 'finish_reason': 'shutdown',\r\n 'finish_time': datetime.datetime(2015, 8, 24, 20, 10, 21, 100933),\r\n 'log_count/DEBUG': 31,\r\n 'log_count/INFO': 8,\r\n 'log_count/WARNING': 1,\r\n 'request_depth_max': 3,\r\n 'response_received_count': 21,\r\n 'scheduler/dequeued': 30,\r\n 'scheduler/dequeued/memory': 30,\r\n 'scheduler/enqueued': 839,\r\n 'scheduler/enqueued/memory': 839,\r\n 'start_time': datetime.datetime(2015, 8, 24, 20, 10, 11, 321827)}\r\n2015-08-24 17:10:21 [scrapy] INFO: Spider closed (shutdown)\r\n```","created_at":"2015-08-24T20:08:33Z","updated_at":"2015-08-29T20:44:03Z","closed_at":null,"merged_at":null,"merge_commit_sha":"b77d567fd0de90e033472462d33d6dd927deb203","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/scrapy/scrapy/pulls/1455/commits","review_comments_url":"https://api.github.com/repos/scrapy/scrapy/pulls/1455/comments","review_comment_url":"https://api.github.com/repos/scrapy/scrapy/pulls/comments{/number}","comments_url":"https://api.github.com/repos/scrapy/scrapy/issues/1455/comments","statuses_url":"https://api.github.com/repos/scrapy/scrapy/statuses/55c85dd57b9fbf54c2cbee496500ee11a3822300","head":{"label":"scrapy:asyncio","ref":"asyncio","sha":"55c85dd57b9fbf54c2cbee496500ee11a3822300","user":{"login":"scrapy","id":733635,"avatar_url":"https://avatars.githubusercontent.com/u/733635?v=3","gravatar_id":"","url":"https://api.github.com/users/scrapy","html_url":"https://github.com/scrapy","followers_url":"https://api.github.com/users/scrapy/followers","following_url":"https://api.github.com/users/scrapy/following{/other_user}","gists_url":"https://api.github.com/users/scrapy/gists{/gist_id}","starred_url":"https://api.github.com/users/scrapy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scrapy/subscriptions","organizations_url":"https://api.github.com/users/scrapy/orgs","repos_url":"https://api.github.com/users/scrapy/repos","events_url":"https://api.github.com/users/scrapy/events{/privacy}","received_events_url":"https://api.github.com/users/scrapy/received_events","type":"Organization","site_admin":false},"repo":{"id":529502,"name":"scrapy","full_name":"scrapy/scrapy","owner":{"login":"scrapy","id":733635,"avatar_url":"https://avatars.githubusercontent.com/u/733635?v=3","gravatar_id":"","url":"https://api.github.com/users/scrapy","html_url":"https://github.com/scrapy","followers_url":"https://api.github.com/users/scrapy/followers","following_url":"https://api.github.com/users/scrapy/following{/other_user}","gists_url":"https://api.github.com/users/scrapy/gists{/gist_id}","starred_url":"https://api.github.com/users/scrapy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scrapy/subscriptions","organizations_url":"https://api.github.com/users/scrapy/orgs","repos_url":"https://api.github.com/users/scrapy/repos","events_url":"https://api.github.com/users/scrapy/events{/privacy}","received_events_url":"https://api.github.com/users/scrapy/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/scrapy/scrapy","description":"Scrapy, a fast high-level web crawling & scraping framework for Python.","fork":false,"url":"https://api.github.com/repos/scrapy/scrapy","forks_url":"https://api.github.com/repos/scrapy/scrapy/forks","keys_url":"https://api.github.com/repos/scrapy/scrapy/keys{/key_id}","collaborators_url":"https://api.github.com/repos/scrapy/scrapy/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/scrapy/scrapy/teams","hooks_url":"https://api.github.com/repos/scrapy/scrapy/hooks","issue_events_url":"https://api.github.com/repos/scrapy/scrapy/issues/events{/number}","events_url":"https://api.github.com/repos/scrapy/scrapy/events","assignees_url":"https://api.github.com/repos/scrapy/scrapy/assignees{/user}","branches_url":"https://api.github.com/repos/scrapy/scrapy/branches{/branch}","tags_url":"https://api.github.com/repos/scrapy/scrapy/tags","blobs_url":"https://api.github.com/repos/scrapy/scrapy/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/scrapy/scrapy/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/scrapy/scrapy/git/refs{/sha}","trees_url":"https://api.github.com/repos/scrapy/scrapy/git/trees{/sha}","statuses_url":"https://api.github.com/repos/scrapy/scrapy/statuses/{sha}","languages_url":"https://api.github.com/repos/scrapy/scrapy/languages","stargazers_url":"https://api.github.com/repos/scrapy/scrapy/stargazers","contributors_url":"https://api.github.com/repos/scrapy/scrapy/contributors","subscribers_url":"https://api.github.com/repos/scrapy/scrapy/subscribers","subscription_url":"https://api.github.com/repos/scrapy/scrapy/subscription","commits_url":"https://api.github.com/repos/scrapy/scrapy/commits{/sha}","git_commits_url":"https://api.github.com/repos/scrapy/scrapy/git/commits{/sha}","comments_url":"https://api.github.com/repos/scrapy/scrapy/comments{/number}","issue_comment_url":"https://api.github.com/repos/scrapy/scrapy/issues/comments{/number}","contents_url":"https://api.github.com/repos/scrapy/scrapy/contents/{+path}","compare_url":"https://api.github.com/repos/scrapy/scrapy/compare/{base}...{head}","merges_url":"https://api.github.com/repos/scrapy/scrapy/merges","archive_url":"https://api.github.com/repos/scrapy/scrapy/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/scrapy/scrapy/downloads","issues_url":"https://api.github.com/repos/scrapy/scrapy/issues{/number}","pulls_url":"https://api.github.com/repos/scrapy/scrapy/pulls{/number}","milestones_url":"https://api.github.com/repos/scrapy/scrapy/milestones{/number}","notifications_url":"https://api.github.com/repos/scrapy/scrapy/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/scrapy/scrapy/labels{/name}","releases_url":"https://api.github.com/repos/scrapy/scrapy/releases{/id}","created_at":"2010-02-22T02:01:14Z","updated_at":"2015-08-29T16:20:29Z","pushed_at":"2015-08-29T19:29:10Z","git_url":"git://github.com/scrapy/scrapy.git","ssh_url":"git@github.com:scrapy/scrapy.git","clone_url":"https://github.com/scrapy/scrapy.git","svn_url":"https://github.com/scrapy/scrapy","homepage":"http://scrapy.org","size":28697,"stargazers_count":10046,"watchers_count":10046,"language":"Python","has_issues":true,"has_downloads":false,"has_wiki":true,"has_pages":false,"forks_count":2878,"mirror_url":null,"open_issues_count":283,"forks":2878,"open_issues":283,"watchers":10046,"default_branch":"master"}},"base":{"label":"scrapy:master","ref":"master","sha":"cfae62f9cc115a5f45c638b29bd6bde60f5737a5","user":{"login":"scrapy","id":733635,"avatar_url":"https://avatars.githubusercontent.com/u/733635?v=3","gravatar_id":"","url":"https://api.github.com/users/scrapy","html_url":"https://github.com/scrapy","followers_url":"https://api.github.com/users/scrapy/followers","following_url":"https://api.github.com/users/scrapy/following{/other_user}","gists_url":"https://api.github.com/users/scrapy/gists{/gist_id}","starred_url":"https://api.github.com/users/scrapy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scrapy/subscriptions","organizations_url":"https://api.github.com/users/scrapy/orgs","repos_url":"https://api.github.com/users/scrapy/repos","events_url":"https://api.github.com/users/scrapy/events{/privacy}","received_events_url":"https://api.github.com/users/scrapy/received_events","type":"Organization","site_admin":false},"repo":{"id":529502,"name":"scrapy","full_name":"scrapy/scrapy","owner":{"login":"scrapy","id":733635,"avatar_url":"https://avatars.githubusercontent.com/u/733635?v=3","gravatar_id":"","url":"https://api.github.com/users/scrapy","html_url":"https://github.com/scrapy","followers_url":"https://api.github.com/users/scrapy/followers","following_url":"https://api.github.com/users/scrapy/following{/other_user}","gists_url":"https://api.github.com/users/scrapy/gists{/gist_id}","starred_url":"https://api.github.com/users/scrapy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scrapy/subscriptions","organizations_url":"https://api.github.com/users/scrapy/orgs","repos_url":"https://api.github.com/users/scrapy/repos","events_url":"https://api.github.com/users/scrapy/events{/privacy}","received_events_url":"https://api.github.com/users/scrapy/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/scrapy/scrapy","description":"Scrapy, a fast high-level web crawling & scraping framework for Python.","fork":false,"url":"https://api.github.com/repos/scrapy/scrapy","forks_url":"https://api.github.com/repos/scrapy/scrapy/forks","keys_url":"https://api.github.com/repos/scrapy/scrapy/keys{/key_id}","collaborators_url":"https://api.github.com/repos/scrapy/scrapy/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/scrapy/scrapy/teams","hooks_url":"https://api.github.com/repos/scrapy/scrapy/hooks","issue_events_url":"https://api.github.com/repos/scrapy/scrapy/issues/events{/number}","events_url":"https://api.github.com/repos/scrapy/scrapy/events","assignees_url":"https://api.github.com/repos/scrapy/scrapy/assignees{/user}","branches_url":"https://api.github.com/repos/scrapy/scrapy/branches{/branch}","tags_url":"https://api.github.com/repos/scrapy/scrapy/tags","blobs_url":"https://api.github.com/repos/scrapy/scrapy/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/scrapy/scrapy/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/scrapy/scrapy/git/refs{/sha}","trees_url":"https://api.github.com/repos/scrapy/scrapy/git/trees{/sha}","statuses_url":"https://api.github.com/repos/scrapy/scrapy/statuses/{sha}","languages_url":"https://api.github.com/repos/scrapy/scrapy/languages","stargazers_url":"https://api.github.com/repos/scrapy/scrapy/stargazers","contributors_url":"https://api.github.com/repos/scrapy/scrapy/contributors","subscribers_url":"https://api.github.com/repos/scrapy/scrapy/subscribers","subscription_url":"https://api.github.com/repos/scrapy/scrapy/subscription","commits_url":"https://api.github.com/repos/scrapy/scrapy/commits{/sha}","git_commits_url":"https://api.github.com/repos/scrapy/scrapy/git/commits{/sha}","comments_url":"https://api.github.com/repos/scrapy/scrapy/comments{/number}","issue_comment_url":"https://api.github.com/repos/scrapy/scrapy/issues/comments{/number}","contents_url":"https://api.github.com/repos/scrapy/scrapy/contents/{+path}","compare_url":"https://api.github.com/repos/scrapy/scrapy/compare/{base}...{head}","merges_url":"https://api.github.com/repos/scrapy/scrapy/merges","archive_url":"https://api.github.com/repos/scrapy/scrapy/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/scrapy/scrapy/downloads","issues_url":"https://api.github.com/repos/scrapy/scrapy/issues{/number}","pulls_url":"https://api.github.com/repos/scrapy/scrapy/pulls{/number}","milestones_url":"https://api.github.com/repos/scrapy/scrapy/milestones{/number}","notifications_url":"https://api.github.com/repos/scrapy/scrapy/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/scrapy/scrapy/labels{/name}","releases_url":"https://api.github.com/repos/scrapy/scrapy/releases{/id}","created_at":"2010-02-22T02:01:14Z","updated_at":"2015-08-29T16:20:29Z","pushed_at":"2015-08-29T19:29:10Z","git_url":"git://github.com/scrapy/scrapy.git","ssh_url":"git@github.com:scrapy/scrapy.git","clone_url":"https://github.com/scrapy/scrapy.git","svn_url":"https://github.com/scrapy/scrapy","homepage":"http://scrapy.org","size":28697,"stargazers_count":10046,"watchers_count":10046,"language":"Python","has_issues":true,"has_downloads":false,"has_wiki":true,"has_pages":false,"forks_count":2878,"mirror_url":null,"open_issues_count":283,"forks":2878,"open_issues":283,"watchers":10046,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/scrapy/scrapy/pulls/1455"},"html":{"href":"https://github.com/scrapy/scrapy/pull/1455"},"issue":{"href":"https://api.github.com/repos/scrapy/scrapy/issues/1455"},"comments":{"href":"https://api.github.com/repos/scrapy/scrapy/issues/1455/comments"},"review_comments":{"href":"https://api.github.com/repos/scrapy/scrapy/pulls/1455/comments"},"review_comment":{"href":"https://api.github.com/repos/scrapy/scrapy/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/scrapy/scrapy/pulls/1455/commits"},"statuses":{"href":"https://api.github.com/repos/scrapy/scrapy/statuses/55c85dd57b9fbf54c2cbee496500ee11a3822300"}}}}