{"action":"created","issue":{"url":"https://api.github.com/repos/hexpm/hex_web/issues/74","labels_url":"https://api.github.com/repos/hexpm/hex_web/issues/74/labels{/name}","comments_url":"https://api.github.com/repos/hexpm/hex_web/issues/74/comments","events_url":"https://api.github.com/repos/hexpm/hex_web/issues/74/events","html_url":"https://github.com/hexpm/hex_web/issues/74","id":49451761,"number":74,"title":"Serve Rebar 3 packages with Hex","user":{"login":"josevalim","id":9582,"avatar_url":"https://avatars.githubusercontent.com/u/9582?v=3","gravatar_id":"","url":"https://api.github.com/users/josevalim","html_url":"https://github.com/josevalim","followers_url":"https://api.github.com/users/josevalim/followers","following_url":"https://api.github.com/users/josevalim/following{/other_user}","gists_url":"https://api.github.com/users/josevalim/gists{/gist_id}","starred_url":"https://api.github.com/users/josevalim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/josevalim/subscriptions","organizations_url":"https://api.github.com/users/josevalim/orgs","repos_url":"https://api.github.com/users/josevalim/repos","events_url":"https://api.github.com/users/josevalim/events{/privacy}","received_events_url":"https://api.github.com/users/josevalim/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":24,"created_at":"2014-11-19T20:53:10Z","updated_at":"2015-02-27T16:16:13Z","closed_at":null,"body":"Based on a discussion on IRC with @tsloughter (and to some extent @ferd), this is a proposal and discussion to have Rebar 3 packages stored and served from Hex.\r\n\r\n## Rationale\r\n\r\nHex already provides the whole structure for publishing and serving packages. Also, since Hex aims to be the package manager for the whole Erlang Ecosystem, with better tooling for Erlang already on the way, it makes sense for Rebar 3 and Hex teams to join efforts in the packages department.\r\n\r\n## Precompiled packages\r\n\r\nToday Hex currently just stores the source code for packages. On the other hand, Rebar packages are actually precompiled packages (with BEAM files) per erts version.\r\n\r\nAlthough those approaches may look conflicting at first, Rebar packages are a natural expansion of the Hex model. Once a package (its source) is published, the author (or a build service) can push precompiled versions of the package based on the erts version. In simpler words, we will extend Hex so a package can have multiple precompiled versions.\r\n\r\nSince the Rebar team is currently only interested in precompiled versions, they can fetch the package registry, filter the registry based on the current erts version, and just download those packages. In the future, the Rebar team may also consider supporting source-based packages, which may be an option for compiling packages with C dependencies.\r\n\r\nWhile the Elixir tooling still requires the source, which will continue to be available in Hex, nothing stops Elixir from also using precompiled packages in the future, saving in compilation time.\r\n\r\n## Hex overview\r\n\r\nThis is an overview of how Hex works today and the changes that will be done to support precompiled packages. This is also the API Rebar will use for fetching precompiled packages.\r\n\r\n### S3\r\n\r\nThe registry and packages are currently stored in S3. The available endpoints are:\r\n\r\n  * `/registry.ets.gz` - gzipped ets file\r\n  * `/tarballs/PACKAGE-VERSION.tar` - package tarball\r\n\r\nWe propose a new end-point, `/tarballs/erts/6.2/cowboy-1.0.0.tar`, to be added for precompiled package tarball per erts as part of this discussion.\r\n\r\nThe tarball contents for precompiled packages is the same as the package tarball except it also contains precompiled `.beam` files.\r\n\r\n### The registry\r\n\r\nThe registry is an ETS table generated with `ets:tab2file/2`. The items are:\r\n\r\n  * `{'$$version$$', Version}` - the registry version\r\n    - Version: integer, incremented on breaking changes\r\n\r\n  * `{Package, [Versions]}` - all releases of a package\r\n    - Package: binary string\r\n    - Versions: list of binary string semver versions\r\n\r\n  * `{{Package, Version}, [Deps, Checksum, Builds | _]}` - a package release's dependencies\r\n    - Package: binary string\r\n    - Version: binary string semver version\r\n    - Deps: List of dependencies\r\n      - Dep: [Name, Requirement, Optional, App]\r\n        - Name: binary package name\r\n        - Requirement: binary Mix.Version requirement\r\n        - Optional: boolean true if it's an optional dependency\r\n        - App: binary OTP application name\r\n    - Checksum: Binary hex encoded sha256 checksum of package, see below\r\n    - Builds: List of builds\r\n      - Build: [Params]\r\n        - Params: List of parameters\r\n          - Param: [{Key, Value}]\r\n            Key: binary\r\n            Value: binary\r\n\r\nThe `Builds` field does not currently exist in the registry and we propose it to be added as part of this discussion. We have designed it as a proplist in case we want to support types of builds in the future.\r\n\r\n### Package tarball\r\n \r\nThe package tarball contains the following files:\r\n\r\n  * VERSION\r\n    - the tarball version as a single ASCII integer\r\n \r\n  * CHECKSUM\r\n    - sha256 hex encoded checksum of tarball\r\n    - `sha256(<<contents(VERSION)/binary, contents(metadata.exs)/binary, contents(contents.tar.gz)/binary)`\r\n \r\n  * metadata.exs\r\n    - Erlang term file\r\n \r\n  * contents.tar.gz\r\n    - gzipped tarball with package contents\r\n\r\n## Next steps\r\n\r\nThis provides an overview of the current API and the work we are going to do for supporting precompiled packages in Hex. It also specifies the API that Rebar 3 will consume.\r\n\r\n@tsloughter and @ferd, please validate if this API is enough for you to get started and if there is anything we should change and/or clarify in the short term. Also let us know if you need help or want to discuss anything on the Rebar 3 side of things."},"comment":{"url":"https://api.github.com/repos/hexpm/hex_web/issues/comments/76420798","html_url":"https://github.com/hexpm/hex_web/issues/74#issuecomment-76420798","issue_url":"https://api.github.com/repos/hexpm/hex_web/issues/74","id":76420798,"user":{"login":"ferd","id":111141,"avatar_url":"https://avatars.githubusercontent.com/u/111141?v=3","gravatar_id":"","url":"https://api.github.com/users/ferd","html_url":"https://github.com/ferd","followers_url":"https://api.github.com/users/ferd/followers","following_url":"https://api.github.com/users/ferd/following{/other_user}","gists_url":"https://api.github.com/users/ferd/gists{/gist_id}","starred_url":"https://api.github.com/users/ferd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ferd/subscriptions","organizations_url":"https://api.github.com/users/ferd/orgs","repos_url":"https://api.github.com/users/ferd/repos","events_url":"https://api.github.com/users/ferd/events{/privacy}","received_events_url":"https://api.github.com/users/ferd/received_events","type":"User","site_admin":false},"created_at":"2015-02-27T16:16:13Z","updated_at":"2015-02-27T16:16:13Z","body":"- Are your endpoints key/values sorted, or do you expect combinatorial explosion of parameters in paths? Either have explicit path hierarchies or an explicit order to them\r\n"}}