{"action":"created","issue":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/1577","labels_url":"https://api.github.com/repos/scrapy/scrapy/issues/1577/labels{/name}","comments_url":"https://api.github.com/repos/scrapy/scrapy/issues/1577/comments","events_url":"https://api.github.com/repos/scrapy/scrapy/issues/1577/events","html_url":"https://github.com/scrapy/scrapy/issues/1577","id":114538107,"number":1577,"title":"Streaming request with subprocess calls in pipeline","user":{"login":"ssbb","id":3309586,"avatar_url":"https://avatars.githubusercontent.com/u/3309586?v=3","gravatar_id":"","url":"https://api.github.com/users/ssbb","html_url":"https://github.com/ssbb","followers_url":"https://api.github.com/users/ssbb/followers","following_url":"https://api.github.com/users/ssbb/following{/other_user}","gists_url":"https://api.github.com/users/ssbb/gists{/gist_id}","starred_url":"https://api.github.com/users/ssbb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ssbb/subscriptions","organizations_url":"https://api.github.com/users/ssbb/orgs","repos_url":"https://api.github.com/users/ssbb/repos","events_url":"https://api.github.com/users/ssbb/events{/privacy}","received_events_url":"https://api.github.com/users/ssbb/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-11-02T07:01:32Z","updated_at":"2015-11-04T19:30:39Z","closed_at":"2015-11-04T19:30:39Z","body":"Hi! \r\n\r\nI want to download first 100-200kb of file for each scraped item (normal size is much bigger) and run subprocess call. Now I have this logic (inside celery): downloading X bytes of file (with requests/urllib), call `ffprobe` via subprocess, stop if file can be parsed. If not - get more X bytes and check again... Loop here.\r\n\r\nSo what you think if I will move this logic to Scrapy pipeline? I will have blocks?"},"comment":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/comments/153839096","html_url":"https://github.com/scrapy/scrapy/issues/1577#issuecomment-153839096","issue_url":"https://api.github.com/repos/scrapy/scrapy/issues/1577","id":153839096,"user":{"login":"nramirezuy","id":1042865,"avatar_url":"https://avatars.githubusercontent.com/u/1042865?v=3","gravatar_id":"","url":"https://api.github.com/users/nramirezuy","html_url":"https://github.com/nramirezuy","followers_url":"https://api.github.com/users/nramirezuy/followers","following_url":"https://api.github.com/users/nramirezuy/following{/other_user}","gists_url":"https://api.github.com/users/nramirezuy/gists{/gist_id}","starred_url":"https://api.github.com/users/nramirezuy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nramirezuy/subscriptions","organizations_url":"https://api.github.com/users/nramirezuy/orgs","repos_url":"https://api.github.com/users/nramirezuy/repos","events_url":"https://api.github.com/users/nramirezuy/events{/privacy}","received_events_url":"https://api.github.com/users/nramirezuy/received_events","type":"User","site_admin":false},"created_at":"2015-11-04T19:30:37Z","updated_at":"2015-11-04T19:30:37Z","body":"@ssbb well it kinda depends on a lot of things. The simplest way would be to try it out based on the items per minute you have with and without the pipeline and seeing if is fast enough.\r\n\r\nSo, these issues are meant for bugs about Scrapy.\r\nPlease use the community channels to ask for help about how to use Scrapy (mailing list: http://groups.google.com/group/scrapy-users Stackoverflow scrapy tag: http://stackoverflow.com/questions/tagged/scrapy)"}}