{"action":"closed","issue":{"url":"https://api.github.com/repos/datalad/datalad/issues/10","labels_url":"https://api.github.com/repos/datalad/datalad/issues/10/labels{/name}","comments_url":"https://api.github.com/repos/datalad/datalad/issues/10/comments","events_url":"https://api.github.com/repos/datalad/datalad/issues/10/events","html_url":"https://github.com/datalad/datalad/issues/10","id":45011759,"number":10,"title":"git-annex: custom downloaders","user":{"login":"yarikoptic","id":39889,"avatar_url":"https://avatars.githubusercontent.com/u/39889?v=3","gravatar_id":"","url":"https://api.github.com/users/yarikoptic","html_url":"https://github.com/yarikoptic","followers_url":"https://api.github.com/users/yarikoptic/followers","following_url":"https://api.github.com/users/yarikoptic/following{/other_user}","gists_url":"https://api.github.com/users/yarikoptic/gists{/gist_id}","starred_url":"https://api.github.com/users/yarikoptic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yarikoptic/subscriptions","organizations_url":"https://api.github.com/users/yarikoptic/orgs","repos_url":"https://api.github.com/users/yarikoptic/repos","events_url":"https://api.github.com/users/yarikoptic/events{/privacy}","received_events_url":"https://api.github.com/users/yarikoptic/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/datalad/datalad/labels/!%20important%20!","name":"! important !","color":"f7c6c7"},{"url":"https://api.github.com/repos/datalad/datalad/labels/enhancement","name":"enhancement","color":"84b6eb"},{"url":"https://api.github.com/repos/datalad/datalad/labels/git-annex","name":"git-annex","color":"eb6420"}],"state":"closed","locked":false,"assignee":{"login":"joeyh","id":16392,"avatar_url":"https://avatars.githubusercontent.com/u/16392?v=3","gravatar_id":"","url":"https://api.github.com/users/joeyh","html_url":"https://github.com/joeyh","followers_url":"https://api.github.com/users/joeyh/followers","following_url":"https://api.github.com/users/joeyh/following{/other_user}","gists_url":"https://api.github.com/users/joeyh/gists{/gist_id}","starred_url":"https://api.github.com/users/joeyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joeyh/subscriptions","organizations_url":"https://api.github.com/users/joeyh/orgs","repos_url":"https://api.github.com/users/joeyh/repos","events_url":"https://api.github.com/users/joeyh/events{/privacy}","received_events_url":"https://api.github.com/users/joeyh/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/datalad/datalad/milestones/8","html_url":"https://github.com/datalad/datalad/milestones/Backend:%20downloaders%20API","labels_url":"https://api.github.com/repos/datalad/datalad/milestones/8/labels","id":928616,"number":8,"title":"Backend: downloaders API","description":"DataLad side of the downloaders API.  Recently introduced in the scope of this project git-annex external special remotes already provide API to enable custom downloaders.  This milestone will be achieved when we have sample custom downloaders for the planned to be supported providers (at least S3 and tarballs)","creator":{"login":"yarikoptic","id":39889,"avatar_url":"https://avatars.githubusercontent.com/u/39889?v=3","gravatar_id":"","url":"https://api.github.com/users/yarikoptic","html_url":"https://github.com/yarikoptic","followers_url":"https://api.github.com/users/yarikoptic/followers","following_url":"https://api.github.com/users/yarikoptic/following{/other_user}","gists_url":"https://api.github.com/users/yarikoptic/gists{/gist_id}","starred_url":"https://api.github.com/users/yarikoptic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yarikoptic/subscriptions","organizations_url":"https://api.github.com/users/yarikoptic/orgs","repos_url":"https://api.github.com/users/yarikoptic/repos","events_url":"https://api.github.com/users/yarikoptic/events{/privacy}","received_events_url":"https://api.github.com/users/yarikoptic/received_events","type":"User","site_admin":false},"open_issues":0,"closed_issues":1,"state":"open","created_at":"2015-01-09T13:56:30Z","updated_at":"2015-05-12T16:25:19Z","due_on":"2015-06-10T04:00:00Z","closed_at":null},"comments":17,"created_at":"2014-10-06T17:40:53Z","updated_at":"2015-05-12T16:25:19Z","closed_at":"2015-05-12T16:25:19Z","body":"As we originally planned, we would need custom helpers to fetch data from e.g. image databases neither exposing their data via HTTP, nor providing a universal \"key store\" like facility, so prohibiting development of dedicated special remote handlers for them.\r\n\r\nOne way to mitigate this would be to have in git-annex some way to specify custom \"downloaders\" given a specific URI pattern.  E.g. for URL regexp \"http.*\\.torrent\" still use built-in wget/curl for checking content presence and ```aria2c %(url)s``` for fetching the content.  Such custom downloaders would also have their registry of authorization credentials for data sources requiring them.  Also we could support extraction from locally present archives, via custom downloaders, see e.g. https://github.com/datalad/datalad/blob/54b71be5d514991aa432931564698744492bcb6c/docs/design.rst#L277 on details.\r\n\r\nIf support of custom downloaders would not be implemented directly in git-annex, it might be implementable by datalad in many cases by \"proxying\" calls to wget and/or curl and then checking/fetching content accordingly.\r\n\r\nCurrently annex only has support for regular wget/curl and quvi (for youtube videos), with hardcoded logic.\r\n\r\nRelated:\r\n- all issues tagged as new-datatransfer-protocol https://github.com/datalad/datalad/issues?q=is%3Aopen+is%3Aissue+label%3Anew-datatransfer-protocol\r\n- Using an external client (addurl torrent support)\r\nhttp://git-annex.branchable.com/todo/Bittorrent-like_features/#index2h1\r\n\r\n"}}