{"action":"created","comment":{"url":"https://api.github.com/repos/scrapy/scrapy/pulls/comments/28391956","id":28391956,"diff_hunk":"@@ -1,76 +1,94 @@\n .. _topics-images:\n \n-=======================\n-Downloading Item Images\n-=======================\n+===========================================\n+Downloading and processing files and images\n+===========================================\n \n .. currentmodule:: scrapy.contrib.pipeline.images\n \n-Scrapy provides an :doc:`item pipeline </topics/item-pipeline>` for downloading\n-images attached to a particular item, for example, when you scrape products and\n-also want to download their images locally.\n+Scrapy provides reusable :doc:`item pipelines </topics/item-pipeline>` for\n+downloading fies attached to a particular item (for example, when you scrape\n+products and also want to download their images locally). These pipelines share\n+a bit of functionality and structure (we refer to them as media pipelines), but\n+typically you'll either use the Files Pipeline or the Images Pipeline.\n \n-This pipeline, called the Images Pipeline and implemented in the\n-:class:`ImagesPipeline` class, provides a convenient way for\n-downloading and storing images locally with some additional features:\n+Both pipelines implement these features:\n+\n+* Avoid re-downloading media that was downloaded recently\n+* Specifying where to store the files (filesystem directory, Amazon S3 bucket)\n+\n+The Images Pipeline has a few extra functions for processing images:\n \n * Convert all downloaded images to a common format (JPG) and mode (RGB)\n-* Avoid re-downloading images which were downloaded recently\n * Thumbnail generation\n * Check images width/height to make sure they meet a minimum constraint\n \n-This pipeline also keeps an internal queue of those images which are currently\n-being scheduled for download, and connects those items that arrive containing\n-the same image, to that queue. This avoids downloading the same image more than\n+The pipelines also keep an internal queue of those images which are currently\n+being scheduled for download, and connect those items that arrive containing","path":"docs/topics/images.rst","position":40,"original_position":40,"commit_id":"dce48b8654977d3f7a003addd507b0aa5e71a76c","original_commit_id":"dce48b8654977d3f7a003addd507b0aa5e71a76c","user":{"login":"dangra","id":37369,"avatar_url":"https://avatars.githubusercontent.com/u/37369?v=3","gravatar_id":"","url":"https://api.github.com/users/dangra","html_url":"https://github.com/dangra","followers_url":"https://api.github.com/users/dangra/followers","following_url":"https://api.github.com/users/dangra/following{/other_user}","gists_url":"https://api.github.com/users/dangra/gists{/gist_id}","starred_url":"https://api.github.com/users/dangra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dangra/subscriptions","organizations_url":"https://api.github.com/users/dangra/orgs","repos_url":"https://api.github.com/users/dangra/repos","events_url":"https://api.github.com/users/dangra/events{/privacy}","received_events_url":"https://api.github.com/users/dangra/received_events","type":"User","site_admin":false},"body":"s/items/responses/","created_at":"2015-04-15T03:59:31Z","updated_at":"2015-04-15T03:59:31Z","html_url":"https://github.com/scrapy/scrapy/pull/1150#discussion_r28391956","pull_request_url":"https://api.github.com/repos/scrapy/scrapy/pulls/1150","_links":{"self":{"href":"https://api.github.com/repos/scrapy/scrapy/pulls/comments/28391956"},"html":{"href":"https://github.com/scrapy/scrapy/pull/1150#discussion_r28391956"},"pull_request":{"href":"https://api.github.com/repos/scrapy/scrapy/pulls/1150"}}},"pull_request":{"url":"https://api.github.com/repos/scrapy/scrapy/pulls/1150","id":33106382,"html_url":"https://github.com/scrapy/scrapy/pull/1150","diff_url":"https://github.com/scrapy/scrapy/pull/1150.diff","patch_url":"https://github.com/scrapy/scrapy/pull/1150.patch","issue_url":"https://api.github.com/repos/scrapy/scrapy/issues/1150","number":1150,"state":"open","locked":false,"title":"Documenting Files Pipeline together with Images Pipeline","user":{"login":"eliasdorneles","id":37565,"avatar_url":"https://avatars.githubusercontent.com/u/37565?v=3","gravatar_id":"","url":"https://api.github.com/users/eliasdorneles","html_url":"https://github.com/eliasdorneles","followers_url":"https://api.github.com/users/eliasdorneles/followers","following_url":"https://api.github.com/users/eliasdorneles/following{/other_user}","gists_url":"https://api.github.com/users/eliasdorneles/gists{/gist_id}","starred_url":"https://api.github.com/users/eliasdorneles/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eliasdorneles/subscriptions","organizations_url":"https://api.github.com/users/eliasdorneles/orgs","repos_url":"https://api.github.com/users/eliasdorneles/repos","events_url":"https://api.github.com/users/eliasdorneles/events{/privacy}","received_events_url":"https://api.github.com/users/eliasdorneles/received_events","type":"User","site_admin":false},"body":"Hey folks,\r\n\r\nSo, here is a proposal addressing #599 , documenting both media pipelines together.\r\nDoes this look good?\r\n\r\nThank you!","created_at":"2015-04-11T17:04:25Z","updated_at":"2015-04-15T03:59:32Z","closed_at":null,"merged_at":null,"merge_commit_sha":"b142ffc4329a2dc774650749aad0d9d9b20b866a","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/scrapy/scrapy/pulls/1150/commits","review_comments_url":"https://api.github.com/repos/scrapy/scrapy/pulls/1150/comments","review_comment_url":"https://api.github.com/repos/scrapy/scrapy/pulls/comments{/number}","comments_url":"https://api.github.com/repos/scrapy/scrapy/issues/1150/comments","statuses_url":"https://api.github.com/repos/scrapy/scrapy/statuses/dce48b8654977d3f7a003addd507b0aa5e71a76c","head":{"label":"eliasdorneles:docs-files-pipeline","ref":"docs-files-pipeline","sha":"dce48b8654977d3f7a003addd507b0aa5e71a76c","user":{"login":"eliasdorneles","id":37565,"avatar_url":"https://avatars.githubusercontent.com/u/37565?v=3","gravatar_id":"","url":"https://api.github.com/users/eliasdorneles","html_url":"https://github.com/eliasdorneles","followers_url":"https://api.github.com/users/eliasdorneles/followers","following_url":"https://api.github.com/users/eliasdorneles/following{/other_user}","gists_url":"https://api.github.com/users/eliasdorneles/gists{/gist_id}","starred_url":"https://api.github.com/users/eliasdorneles/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eliasdorneles/subscriptions","organizations_url":"https://api.github.com/users/eliasdorneles/orgs","repos_url":"https://api.github.com/users/eliasdorneles/repos","events_url":"https://api.github.com/users/eliasdorneles/events{/privacy}","received_events_url":"https://api.github.com/users/eliasdorneles/received_events","type":"User","site_admin":false},"repo":{"id":22083994,"name":"scrapy","full_name":"eliasdorneles/scrapy","owner":{"login":"eliasdorneles","id":37565,"avatar_url":"https://avatars.githubusercontent.com/u/37565?v=3","gravatar_id":"","url":"https://api.github.com/users/eliasdorneles","html_url":"https://github.com/eliasdorneles","followers_url":"https://api.github.com/users/eliasdorneles/followers","following_url":"https://api.github.com/users/eliasdorneles/following{/other_user}","gists_url":"https://api.github.com/users/eliasdorneles/gists{/gist_id}","starred_url":"https://api.github.com/users/eliasdorneles/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eliasdorneles/subscriptions","organizations_url":"https://api.github.com/users/eliasdorneles/orgs","repos_url":"https://api.github.com/users/eliasdorneles/repos","events_url":"https://api.github.com/users/eliasdorneles/events{/privacy}","received_events_url":"https://api.github.com/users/eliasdorneles/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/eliasdorneles/scrapy","description":"Scrapy, a fast high-level screen scraping and web crawling framework for Python.","fork":true,"url":"https://api.github.com/repos/eliasdorneles/scrapy","forks_url":"https://api.github.com/repos/eliasdorneles/scrapy/forks","keys_url":"https://api.github.com/repos/eliasdorneles/scrapy/keys{/key_id}","collaborators_url":"https://api.github.com/repos/eliasdorneles/scrapy/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/eliasdorneles/scrapy/teams","hooks_url":"https://api.github.com/repos/eliasdorneles/scrapy/hooks","issue_events_url":"https://api.github.com/repos/eliasdorneles/scrapy/issues/events{/number}","events_url":"https://api.github.com/repos/eliasdorneles/scrapy/events","assignees_url":"https://api.github.com/repos/eliasdorneles/scrapy/assignees{/user}","branches_url":"https://api.github.com/repos/eliasdorneles/scrapy/branches{/branch}","tags_url":"https://api.github.com/repos/eliasdorneles/scrapy/tags","blobs_url":"https://api.github.com/repos/eliasdorneles/scrapy/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/eliasdorneles/scrapy/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/eliasdorneles/scrapy/git/refs{/sha}","trees_url":"https://api.github.com/repos/eliasdorneles/scrapy/git/trees{/sha}","statuses_url":"https://api.github.com/repos/eliasdorneles/scrapy/statuses/{sha}","languages_url":"https://api.github.com/repos/eliasdorneles/scrapy/languages","stargazers_url":"https://api.github.com/repos/eliasdorneles/scrapy/stargazers","contributors_url":"https://api.github.com/repos/eliasdorneles/scrapy/contributors","subscribers_url":"https://api.github.com/repos/eliasdorneles/scrapy/subscribers","subscription_url":"https://api.github.com/repos/eliasdorneles/scrapy/subscription","commits_url":"https://api.github.com/repos/eliasdorneles/scrapy/commits{/sha}","git_commits_url":"https://api.github.com/repos/eliasdorneles/scrapy/git/commits{/sha}","comments_url":"https://api.github.com/repos/eliasdorneles/scrapy/comments{/number}","issue_comment_url":"https://api.github.com/repos/eliasdorneles/scrapy/issues/comments{/number}","contents_url":"https://api.github.com/repos/eliasdorneles/scrapy/contents/{+path}","compare_url":"https://api.github.com/repos/eliasdorneles/scrapy/compare/{base}...{head}","merges_url":"https://api.github.com/repos/eliasdorneles/scrapy/merges","archive_url":"https://api.github.com/repos/eliasdorneles/scrapy/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/eliasdorneles/scrapy/downloads","issues_url":"https://api.github.com/repos/eliasdorneles/scrapy/issues{/number}","pulls_url":"https://api.github.com/repos/eliasdorneles/scrapy/pulls{/number}","milestones_url":"https://api.github.com/repos/eliasdorneles/scrapy/milestones{/number}","notifications_url":"https://api.github.com/repos/eliasdorneles/scrapy/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/eliasdorneles/scrapy/labels{/name}","releases_url":"https://api.github.com/repos/eliasdorneles/scrapy/releases{/id}","created_at":"2014-07-21T23:36:45Z","updated_at":"2015-03-28T00:48:55Z","pushed_at":"2015-04-11T16:58:12Z","git_url":"git://github.com/eliasdorneles/scrapy.git","ssh_url":"git@github.com:eliasdorneles/scrapy.git","clone_url":"https://github.com/eliasdorneles/scrapy.git","svn_url":"https://github.com/eliasdorneles/scrapy","homepage":"http://scrapy.org","size":13759,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":false,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"scrapy:master","ref":"master","sha":"6f000b8cd4fc0598ec410296ad31e6333bd4b5a2","user":{"login":"scrapy","id":733635,"avatar_url":"https://avatars.githubusercontent.com/u/733635?v=3","gravatar_id":"","url":"https://api.github.com/users/scrapy","html_url":"https://github.com/scrapy","followers_url":"https://api.github.com/users/scrapy/followers","following_url":"https://api.github.com/users/scrapy/following{/other_user}","gists_url":"https://api.github.com/users/scrapy/gists{/gist_id}","starred_url":"https://api.github.com/users/scrapy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scrapy/subscriptions","organizations_url":"https://api.github.com/users/scrapy/orgs","repos_url":"https://api.github.com/users/scrapy/repos","events_url":"https://api.github.com/users/scrapy/events{/privacy}","received_events_url":"https://api.github.com/users/scrapy/received_events","type":"Organization","site_admin":false},"repo":{"id":529502,"name":"scrapy","full_name":"scrapy/scrapy","owner":{"login":"scrapy","id":733635,"avatar_url":"https://avatars.githubusercontent.com/u/733635?v=3","gravatar_id":"","url":"https://api.github.com/users/scrapy","html_url":"https://github.com/scrapy","followers_url":"https://api.github.com/users/scrapy/followers","following_url":"https://api.github.com/users/scrapy/following{/other_user}","gists_url":"https://api.github.com/users/scrapy/gists{/gist_id}","starred_url":"https://api.github.com/users/scrapy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scrapy/subscriptions","organizations_url":"https://api.github.com/users/scrapy/orgs","repos_url":"https://api.github.com/users/scrapy/repos","events_url":"https://api.github.com/users/scrapy/events{/privacy}","received_events_url":"https://api.github.com/users/scrapy/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/scrapy/scrapy","description":"Scrapy, a fast high-level web crawling and screen scraping framework for Python.","fork":false,"url":"https://api.github.com/repos/scrapy/scrapy","forks_url":"https://api.github.com/repos/scrapy/scrapy/forks","keys_url":"https://api.github.com/repos/scrapy/scrapy/keys{/key_id}","collaborators_url":"https://api.github.com/repos/scrapy/scrapy/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/scrapy/scrapy/teams","hooks_url":"https://api.github.com/repos/scrapy/scrapy/hooks","issue_events_url":"https://api.github.com/repos/scrapy/scrapy/issues/events{/number}","events_url":"https://api.github.com/repos/scrapy/scrapy/events","assignees_url":"https://api.github.com/repos/scrapy/scrapy/assignees{/user}","branches_url":"https://api.github.com/repos/scrapy/scrapy/branches{/branch}","tags_url":"https://api.github.com/repos/scrapy/scrapy/tags","blobs_url":"https://api.github.com/repos/scrapy/scrapy/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/scrapy/scrapy/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/scrapy/scrapy/git/refs{/sha}","trees_url":"https://api.github.com/repos/scrapy/scrapy/git/trees{/sha}","statuses_url":"https://api.github.com/repos/scrapy/scrapy/statuses/{sha}","languages_url":"https://api.github.com/repos/scrapy/scrapy/languages","stargazers_url":"https://api.github.com/repos/scrapy/scrapy/stargazers","contributors_url":"https://api.github.com/repos/scrapy/scrapy/contributors","subscribers_url":"https://api.github.com/repos/scrapy/scrapy/subscribers","subscription_url":"https://api.github.com/repos/scrapy/scrapy/subscription","commits_url":"https://api.github.com/repos/scrapy/scrapy/commits{/sha}","git_commits_url":"https://api.github.com/repos/scrapy/scrapy/git/commits{/sha}","comments_url":"https://api.github.com/repos/scrapy/scrapy/comments{/number}","issue_comment_url":"https://api.github.com/repos/scrapy/scrapy/issues/comments{/number}","contents_url":"https://api.github.com/repos/scrapy/scrapy/contents/{+path}","compare_url":"https://api.github.com/repos/scrapy/scrapy/compare/{base}...{head}","merges_url":"https://api.github.com/repos/scrapy/scrapy/merges","archive_url":"https://api.github.com/repos/scrapy/scrapy/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/scrapy/scrapy/downloads","issues_url":"https://api.github.com/repos/scrapy/scrapy/issues{/number}","pulls_url":"https://api.github.com/repos/scrapy/scrapy/pulls{/number}","milestones_url":"https://api.github.com/repos/scrapy/scrapy/milestones{/number}","notifications_url":"https://api.github.com/repos/scrapy/scrapy/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/scrapy/scrapy/labels{/name}","releases_url":"https://api.github.com/repos/scrapy/scrapy/releases{/id}","created_at":"2010-02-22T02:01:14Z","updated_at":"2015-04-15T03:32:43Z","pushed_at":"2015-04-15T03:32:45Z","git_url":"git://github.com/scrapy/scrapy.git","ssh_url":"git@github.com:scrapy/scrapy.git","clone_url":"https://github.com/scrapy/scrapy.git","svn_url":"https://github.com/scrapy/scrapy","homepage":"http://scrapy.org","size":26625,"stargazers_count":8262,"watchers_count":8262,"language":"Python","has_issues":true,"has_downloads":false,"has_wiki":true,"has_pages":false,"forks_count":2334,"mirror_url":null,"open_issues_count":245,"forks":2334,"open_issues":245,"watchers":8262,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/scrapy/scrapy/pulls/1150"},"html":{"href":"https://github.com/scrapy/scrapy/pull/1150"},"issue":{"href":"https://api.github.com/repos/scrapy/scrapy/issues/1150"},"comments":{"href":"https://api.github.com/repos/scrapy/scrapy/issues/1150/comments"},"review_comments":{"href":"https://api.github.com/repos/scrapy/scrapy/pulls/1150/comments"},"review_comment":{"href":"https://api.github.com/repos/scrapy/scrapy/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/scrapy/scrapy/pulls/1150/commits"},"statuses":{"href":"https://api.github.com/repos/scrapy/scrapy/statuses/dce48b8654977d3f7a003addd507b0aa5e71a76c"}}}}