{"action":"created","issue":{"url":"https://api.github.com/repos/galaxyproject/tools-iuc/issues/345","labels_url":"https://api.github.com/repos/galaxyproject/tools-iuc/issues/345/labels{/name}","comments_url":"https://api.github.com/repos/galaxyproject/tools-iuc/issues/345/comments","events_url":"https://api.github.com/repos/galaxyproject/tools-iuc/issues/345/events","html_url":"https://github.com/galaxyproject/tools-iuc/issues/345","id":112414163,"number":345,"title":"RFC: Package Caching Proxy","user":{"login":"erasche","id":458683,"avatar_url":"https://avatars.githubusercontent.com/u/458683?v=3","gravatar_id":"","url":"https://api.github.com/users/erasche","html_url":"https://github.com/erasche","followers_url":"https://api.github.com/users/erasche/followers","following_url":"https://api.github.com/users/erasche/following{/other_user}","gists_url":"https://api.github.com/users/erasche/gists{/gist_id}","starred_url":"https://api.github.com/users/erasche/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/erasche/subscriptions","organizations_url":"https://api.github.com/users/erasche/orgs","repos_url":"https://api.github.com/users/erasche/repos","events_url":"https://api.github.com/users/erasche/events{/privacy}","received_events_url":"https://api.github.com/users/erasche/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/galaxyproject/tools-iuc/labels/planning","name":"planning","color":"FFC0DB"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-10-20T16:58:38Z","updated_at":"2015-10-20T21:09:24Z","closed_at":null,"body":"I'm proposing a package caching proxy that should provide some relief from the bad-url issue. I'd like feedback from other members of the IUC/devteam before actual implementation.\r\n\r\n## The Problem\r\n\r\nDownloads disappear, get renamed, etc. We cannot rely on externally controlled URLs for longevity. [Bad URLs are a continuous issue for us](https://github.com/galaxyproject/tools-iuc/issues?utf8=%E2%9C%93&q=is%3Aissue+url+)\r\n\r\n## The (Interim) Solution\r\n\r\nA proxy which we can route requests through which cache them as long as the service is alive. \r\n\r\nI've proposed similar things in the past such as a download store where the storage burden is shared across multiple servers, but for an initial implementation we can avoid the complexity of a distributed solution, and just opt for the simple caching proxy which is one step above the existing situation. I.e. it fixes one problem, not all of them. Incremental improvements :)\r\n\r\n## Implementation\r\n\r\nImplementation shouldn't be too difficult, the RFC is more concerned with what features will be offered than how to actually write it.\r\n\r\nI imagine the following:\r\n\r\n- simple service where you make requests like:\r\n\r\n  `https://gx-package-proxy/?q=http%3A%2F%2Fbioinf.uni-greifswald.de%2Faugustus%2Fbinaries%2Fold%2Faugustus-3.1.tar.gz`\r\n\r\n  and receive their package downloads.\r\n- registered + approved users of the service can request a URL be archived permanently\r\n\r\nI originally proposed a reputation based system, but that seems like excessive complexity when we're not making the statement \"your download will be available until the end of time\", we're making the statement \"This will likely be much longer lived than your existing URLs\".\r\n\r\n## Alternatives\r\n\r\n### Squid\r\n\r\nI wrote some squid caching rules at one point, but:\r\n\r\n-  those acted as per-backend URL proxies (though they were really quite nice). \r\n- Squid did not guarantee that things would be cached forever and if things went wrong we'd be no better off than where we are now\r\n\r\n### Distributed\r\n\r\nI've dreamt up alternatives that make use of TAHOE-LAFS or bittorrent over http to share the package cache amongst several groups, but that has a huge amount of complexity associated and requires a lot more engineering than I can offer this month/calendar year.\r\n\r\n### Brew\r\n\r\nThat work is complementary to this, with things like CPAN in existence where .tar.gzs may disappear with no warning, something like this will always be required.\r\n\r\ncc @bgruening @natefoo"},"comment":{"url":"https://api.github.com/repos/galaxyproject/tools-iuc/issues/comments/149704216","html_url":"https://github.com/galaxyproject/tools-iuc/issues/345#issuecomment-149704216","issue_url":"https://api.github.com/repos/galaxyproject/tools-iuc/issues/345","id":149704216,"user":{"login":"erasche","id":458683,"avatar_url":"https://avatars.githubusercontent.com/u/458683?v=3","gravatar_id":"","url":"https://api.github.com/users/erasche","html_url":"https://github.com/erasche","followers_url":"https://api.github.com/users/erasche/followers","following_url":"https://api.github.com/users/erasche/following{/other_user}","gists_url":"https://api.github.com/users/erasche/gists{/gist_id}","starred_url":"https://api.github.com/users/erasche/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/erasche/subscriptions","organizations_url":"https://api.github.com/users/erasche/orgs","repos_url":"https://api.github.com/users/erasche/repos","events_url":"https://api.github.com/users/erasche/events{/privacy}","received_events_url":"https://api.github.com/users/erasche/received_events","type":"User","site_admin":false},"created_at":"2015-10-20T21:09:24Z","updated_at":"2015-10-20T21:09:24Z","body":"@natefoo okay, I'll add this as a feature. You'll be able to request two types of proxied packages:\r\n\r\n- served from a local copy\r\n- redirects to existing URLs\r\n\r\nBut I worry that we don't really gain so much with that, and that we're just trading one backend that could disappear for another backend."}}