{"action":"created","issue":{"url":"https://api.github.com/repos/NixOS/nixpkgs/issues/11177","labels_url":"https://api.github.com/repos/NixOS/nixpkgs/issues/11177/labels{/name}","comments_url":"https://api.github.com/repos/NixOS/nixpkgs/issues/11177/comments","events_url":"https://api.github.com/repos/NixOS/nixpkgs/issues/11177/events","html_url":"https://github.com/NixOS/nixpkgs/issues/11177","id":118135974,"number":11177,"title":"nixpkgs Upstream Dependencies are Offline or Their Declared hash Differs from Actual hash","user":{"login":"martinvahi","id":861080,"avatar_url":"https://avatars.githubusercontent.com/u/861080?v=3","gravatar_id":"","url":"https://api.github.com/users/martinvahi","html_url":"https://github.com/martinvahi","followers_url":"https://api.github.com/users/martinvahi/followers","following_url":"https://api.github.com/users/martinvahi/following{/other_user}","gists_url":"https://api.github.com/users/martinvahi/gists{/gist_id}","starred_url":"https://api.github.com/users/martinvahi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martinvahi/subscriptions","organizations_url":"https://api.github.com/users/martinvahi/orgs","repos_url":"https://api.github.com/users/martinvahi/repos","events_url":"https://api.github.com/users/martinvahi/events{/privacy}","received_events_url":"https://api.github.com/users/martinvahi/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/NixOS/nixpkgs/labels/question","name":"question","color":"cc317c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-11-20T22:04:42Z","updated_at":"2015-11-21T15:10:48Z","closed_at":null,"body":"I first tried to write about that to \r\n\r\n    nix-dev@lists.science.uu.nl\r\n\r\nbut my letter got stuck to some spam filter or just did not go through due to some weird reason. So, the following is a Copy/Paste with Markup related modifications.\r\n\r\n    ---citation--start---\r\n\r\nnixpkgs dependencies availability test is about 300MiB in size due to a clone of the\r\n\r\n    https://github.com/NixOS/nixpkgs\r\n\r\nand resides at [a location at my web site](http://technology.softf1.com/nix/2015/2015_11_19_NixOS_nixpkgs_download_test_by_martin_vahi_at_softf1_com.tar.bz2)\r\n\r\nsha256:\r\nf05cf8bb85527a14f22adf0e2b3a7fa56f63461b7580620e4d54204c4d2db1ba\r\n\r\nThe idea is roughly:\r\n\r\n    git clone https://github.com/NixOS/nixpkgs\r\n    cd nixpkgs\r\n    find . -name '*.nix' > list_of_nixfiles.txt\r\n\r\nand then each of the nix-files is searched for pairs of fetch-url's and hash values and a gigantic bash-script is generated that contains lines of calls to the nix-prefetch-url\r\n\r\n    ---a--demo--excerpt--start---\r\n    nix-prefetch-url --type sha256\r\n    http://bits.xensource.com/oss-xen/release/4.5.1/xen-4.5.1.tar.gz\r\n    0w8kbqy7zixacrpbk3yj51xx7b3f6l8ghsg3551w8ym6zka13336 ;\r\n    nix-prefetch-url --type sha256 sourceInfo.url sourceInfo.hash ;\r\n    nix-prefetch-url --type sha256\r\n    http://download.belastingdienst.nl/belastingdienst/apps/linux/ib2007_linux.tar.gz\r\n    13p3gv086jn95wvmfygdmk9qjn0qxqdv7pp0v5pmw6i5hp8rmjxf ;\r\n    ---a--demo--excerpt--end---\r\n\r\nI executed the gigantic bash script and my conclusion is that many up-stream packages, their files, are offline or their hashes do not match, specially those of the [npm](https://www.npmjs.com/).\r\n\r\nIn those circumstances it is absolutely no wonder that there are various build problems. I propose that in addition to \"stable\" and \"unstable\" packages would be classified as \"Nix-contained\" and\r\n\"up-stream-dependent\".\r\n\r\nThe \"Nix-contained\" packages would be buildable without downloading anything from any server other than the Nix project servers and various caches. I believe that package specific, up-stream dependent, package upgrade tools might be bundled with the \"Nix-contained\" packages, so that when the up-stream servers come online, the package upgrade could be _manually_ made and the \"Nix-contained\" set of packages is supplemented with a new version of the package without removing the old version.\r\n\r\nCurrently it seems to me that the up-stream dependent package build scripts try to do 2 in one: build a Nix package and upgrade its dependencies. In the case of the \"Nix-contained\" packages the package building would work without downloading anything from upstream and all testing and experimentation that is related to the upgrade of the dependencies of a \"Nix-contained\" package, would be carried out by the package maintainer at a time, when he/she has the time to work on the package, not whenever the package is being automatically built.\r\n\r\n\r\nThank You for reading my letter and I hope to receive criticism that tells that I'm missing/omitting\r\na lot of important issues or have otherwise come up with a spectacular blunder. \r\n\r\nRegards,\r\nMartin.Vahi@softf1.com\r\n\r\nP.S. The \"manual\" upgrade of dependencies will probably be a requirement even due to computational limitations and security requirements. Formal verification might take a lot of CPU-power and it might not be optimal to run that every time a package is built.\r\n\r\n    ---citation--end-----"},"comment":{"url":"https://api.github.com/repos/NixOS/nixpkgs/issues/comments/158650467","html_url":"https://github.com/NixOS/nixpkgs/issues/11177#issuecomment-158650467","issue_url":"https://api.github.com/repos/NixOS/nixpkgs/issues/11177","id":158650467,"user":{"login":"martinvahi","id":861080,"avatar_url":"https://avatars.githubusercontent.com/u/861080?v=3","gravatar_id":"","url":"https://api.github.com/users/martinvahi","html_url":"https://github.com/martinvahi","followers_url":"https://api.github.com/users/martinvahi/followers","following_url":"https://api.github.com/users/martinvahi/following{/other_user}","gists_url":"https://api.github.com/users/martinvahi/gists{/gist_id}","starred_url":"https://api.github.com/users/martinvahi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martinvahi/subscriptions","organizations_url":"https://api.github.com/users/martinvahi/orgs","repos_url":"https://api.github.com/users/martinvahi/repos","events_url":"https://api.github.com/users/martinvahi/events{/privacy}","received_events_url":"https://api.github.com/users/martinvahi/received_events","type":"User","site_admin":false},"created_at":"2015-11-21T15:10:48Z","updated_at":"2015-11-21T15:10:48Z","body":"May be a dirty workaround might be the use of some proxy, like\r\n\r\n    url = http://www.my_proxy_site.com/nix_proxy.php?hash_algorithm=foo&hash_value=bar&network=tor&hint_url_1=<the first url to try>&hint_url_2=<the second URL to try>&<etc.>\r\n\r\nand for local proxies the .nix file is preprocessed by replacing \r\n\r\n    www.my_proxy_site.com/\r\n    with\r\n    localhost:<some port number>/\r\n\r\nProbing for different, mirroring, URL-s would then be the task of the proxy."}}