{"action":"created","issue":{"url":"https://api.github.com/repos/JuliaLang/METADATA.jl/issues/2777","labels_url":"https://api.github.com/repos/JuliaLang/METADATA.jl/issues/2777/labels{/name}","comments_url":"https://api.github.com/repos/JuliaLang/METADATA.jl/issues/2777/comments","events_url":"https://api.github.com/repos/JuliaLang/METADATA.jl/issues/2777/events","html_url":"https://github.com/JuliaLang/METADATA.jl/issues/2777","id":91100863,"number":2777,"title":"Require a package's own tests to pass before allowing a version bump","user":{"login":"quinnj","id":2896623,"avatar_url":"https://avatars.githubusercontent.com/u/2896623?v=3","gravatar_id":"","url":"https://api.github.com/users/quinnj","html_url":"https://github.com/quinnj","followers_url":"https://api.github.com/users/quinnj/followers","following_url":"https://api.github.com/users/quinnj/following{/other_user}","gists_url":"https://api.github.com/users/quinnj/gists{/gist_id}","starred_url":"https://api.github.com/users/quinnj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/quinnj/subscriptions","organizations_url":"https://api.github.com/users/quinnj/orgs","repos_url":"https://api.github.com/users/quinnj/repos","events_url":"https://api.github.com/users/quinnj/events{/privacy}","received_events_url":"https://api.github.com/users/quinnj/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-06-25T23:15:19Z","updated_at":"2015-06-26T16:12:14Z","closed_at":null,"body":"As discussed at JuliaCon 2015 Thursday evening:\r\n\r\n* To bump a version of a package, a PR would be submitted, a bot/machine would download the proposed version of the package along with the proposed compatible version of julia and run the tests of the package\r\n* On passing tests, the PR would be automatically merged and version bumped\r\n* On failing tests, PR is not merged (locked?) and PR submitter is notified of failing tests or version mismatch\r\n\r\nOn Julia version bumps (minor or major), all packages would be re-run if the package has no upper limit on julia version compatibility to ensure that its tests still pass:\r\n* If tests still pass, a new version is automatically bumped pointing to the same previous version\r\n* On failing, the previous version is amended with an upper limit on the previous working version of julia; trying to `Pkg.add()` that package on the newer julia version would result in a `no version compatible with your julia version exists` or something like that\r\n\r\nObviously this is a lot of buildbot/testing infrastructure, but it seems like this would allow the state of METADATA packages to be 100% passing their own tests, with full transparency if changes to julia occur that break the package. The ideal goal here is that a user would never be able to `Pkg.add()` and get a package that \"doesn't work\" (at least according ot their own tests).\r\n\r\nA related issue to discuss would be:\r\n * minimum package testing standards that would at least be \"eye-balled\""},"comment":{"url":"https://api.github.com/repos/JuliaLang/METADATA.jl/issues/comments/115743563","html_url":"https://github.com/JuliaLang/METADATA.jl/issues/2777#issuecomment-115743563","issue_url":"https://api.github.com/repos/JuliaLang/METADATA.jl/issues/2777","id":115743563,"user":{"login":"timholy","id":1525481,"avatar_url":"https://avatars.githubusercontent.com/u/1525481?v=3","gravatar_id":"","url":"https://api.github.com/users/timholy","html_url":"https://github.com/timholy","followers_url":"https://api.github.com/users/timholy/followers","following_url":"https://api.github.com/users/timholy/following{/other_user}","gists_url":"https://api.github.com/users/timholy/gists{/gist_id}","starred_url":"https://api.github.com/users/timholy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timholy/subscriptions","organizations_url":"https://api.github.com/users/timholy/orgs","repos_url":"https://api.github.com/users/timholy/repos","events_url":"https://api.github.com/users/timholy/events{/privacy}","received_events_url":"https://api.github.com/users/timholy/received_events","type":"User","site_admin":false},"created_at":"2015-06-26T16:12:14Z","updated_at":"2015-06-26T16:12:14Z","body":"@jiahao pointed me to this. There's a lot to like about this proposal. The only problem I see with it is that I occasionally find myself reluctant to comment out tests that are failing for reasons \"beyond my control,\" by which I mean highly-nontrivial upstream bugs that do not affect any of the platforms I depend on ([like this one](https://github.com/timholy/Images.jl/issues/284)). I nevertheless sometimes would prefer to leave the test there to deliberately annoy myself. But maybe I just need to dig into more flexible testing frameworks?"}}