{"action":"closed","issue":{"url":"https://api.github.com/repos/stedolan/jq/issues/672","labels_url":"https://api.github.com/repos/stedolan/jq/issues/672/labels{/name}","comments_url":"https://api.github.com/repos/stedolan/jq/issues/672/comments","events_url":"https://api.github.com/repos/stedolan/jq/issues/672/events","html_url":"https://github.com/stedolan/jq/issues/672","id":54819162,"number":672,"title":"Module system is a mess","user":{"login":"nicowilliams","id":604851,"avatar_url":"https://avatars.githubusercontent.com/u/604851?v=3","gravatar_id":"","url":"https://api.github.com/users/nicowilliams","html_url":"https://github.com/nicowilliams","followers_url":"https://api.github.com/users/nicowilliams/followers","following_url":"https://api.github.com/users/nicowilliams/following{/other_user}","gists_url":"https://api.github.com/users/nicowilliams/gists{/gist_id}","starred_url":"https://api.github.com/users/nicowilliams/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nicowilliams/subscriptions","organizations_url":"https://api.github.com/users/nicowilliams/orgs","repos_url":"https://api.github.com/users/nicowilliams/repos","events_url":"https://api.github.com/users/nicowilliams/events{/privacy}","received_events_url":"https://api.github.com/users/nicowilliams/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/stedolan/jq/labels/bug","name":"bug","color":"e11d21"}],"state":"closed","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/stedolan/jq/milestones/3","html_url":"https://github.com/stedolan/jq/milestones/1.5%20release","labels_url":"https://api.github.com/repos/stedolan/jq/milestones/3/labels","id":682789,"number":3,"title":"1.5 release","description":"jq 1.5 should get all the features that won't be going into 1.4:\r\n\r\n - loadable modules\r\n - better jq library support\r\n - regexp/glob support\r\n - I/O primitives\r\n - generator primitives: `while`,\r\n - generator utils: `foreach`, `limit`\r\n - `try EXP catch EXP` (exceptions)\r\n - bignum support\r\n - anything else?","creator":{"login":"nicowilliams","id":604851,"avatar_url":"https://avatars.githubusercontent.com/u/604851?v=3","gravatar_id":"","url":"https://api.github.com/users/nicowilliams","html_url":"https://github.com/nicowilliams","followers_url":"https://api.github.com/users/nicowilliams/followers","following_url":"https://api.github.com/users/nicowilliams/following{/other_user}","gists_url":"https://api.github.com/users/nicowilliams/gists{/gist_id}","starred_url":"https://api.github.com/users/nicowilliams/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nicowilliams/subscriptions","organizations_url":"https://api.github.com/users/nicowilliams/orgs","repos_url":"https://api.github.com/users/nicowilliams/repos","events_url":"https://api.github.com/users/nicowilliams/events{/privacy}","received_events_url":"https://api.github.com/users/nicowilliams/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":96,"state":"open","created_at":"2014-06-05T23:19:44Z","updated_at":"2015-07-26T05:19:12Z","due_on":null,"closed_at":null},"comments":34,"created_at":"2015-01-19T22:16:01Z","updated_at":"2015-07-26T05:19:12Z","closed_at":"2015-07-26T05:19:12Z","body":"@joelpurra @pkoppstein @wtlangford \r\n\r\nIt's been a few weeks since I've given this any thought.  Here's an attempt to look at the whole jq module system from a fresh perspective.\r\n\r\njq has no build artifacts at this time, no `.jqc` files.  This means that .jq files (sources) cannot contain \"absolute\" references to dependencies: relative paths are OK, absolute paths are not OK, minimum version numbers are OK, exact version numbers (and/or commit hashes, etcetera) are not.\r\n\r\nAbsolute locators/bindings have to be in an external location, in a `.jqc` or in a `.json` or some such file.  Or else we must do without absolute locators/bindings, using dependent- and/or universe-relative locators only, leaving absolute binding to an external link-editor of sorts (a/the packaging system) that can use symlinks to resolve relative bindings.\r\n\r\nIn general absolute bindings at run-time is the best answer for semantic reasons (absolute bindings mean repeatable results) and for performance reasons: the work of link-editing can be done at build- (or, libtool-like, at intstall-) time.  We don't have this luxury at this time, but we shouldn't want to close the door on a future `.jqc` world either (nor, in such a future, would we want to force all jq programs and modules to be compiled).\r\n\r\nFor user-friendliness it really helps for a language like jq (which has not build step) to support running \"in place\", meaning: without having to `make install` then run from the install location.\r\n\r\nOn thing that becomes obvious is that we shouldn't really have a \"path\" (relative or otherwise) in `import` directives in `.jq` \"source\" files: it's a locator, even if relative, when all we need is a \"name\" -- an identifier.  Ideally we'd have:\r\n\r\n    import as foo {uri: \"...\", <perhaps other metadata>};\r\n\r\nIf you want to use whatever that module is, your pkg system has to fetch and install it (if it isn't already installed) relative to the install location of your dependent so you can run it.\r\n\r\nBut then we absolutely need a separate file, built by the pkg mgr (really, now functioning as a link-editor) in which to store resolved dependency information that *jq* will read.  And then we need a build step, which I think we don't and shouldn't want to _require_.\r\n\r\nI think that's a fair synthesis of the problem we're facing.  Would you agree?\r\n\r\nSo now what?\r\n\r\nThe problem that arises is that we need to have two somewhat contradictory options work with the same source code for the dependent:\r\n\r\n - external file with absolute locators\r\n - run in-place must work\r\n\r\nWhich means we need to distinguish at run-time (since that's all we have right now) whether a dependency is intended to be run in-place or otherwise, and we need the `import` directive to provide enough information for both: the external link-edit/pkg mgr case, and the run in-place case.\r\n\r\nThe only tool we have for squaring this circle without requiring additional steps to build/install additional metadata files, is installing all dependencies local (relative) to their dependents, possibly with symlinks where available.  But there's no reason that we couldn't support the use of additional metadata files as an option.\r\n\r\nIt could be OK to have `import` directives which, as-written, only support the external link-edit/pkg mgr case:\r\n\r\n    # Locator for \"foobar\" must be in .jqc file (or .json?)\r\n    # (look ma'!  no path here!)\r\n    import as foo {name:\"foobar\", <other metadata>};\r\n\r\nBut it should also be OK to have `import` directives with relative path locators for running in-place without necessarily having to run an external link-edit/pkg mgr tool:\r\n\r\n    # Search [<dependent's $ORIGIN>, <universe $ORIGIN>] for\r\n    # \"some/path/to/mod/src/mod.jq\", \"some/path/to/mod/mod.jq\":\r\n    import some/path/to/mod as sym {dir:[\"src\", null]};\r\n    \r\nwith metadata options for various location algorithms as desired.\r\n\r\n\r\nRelatedly, what would it take to actually add a build step to jq, and a `.jqc` file format?  Ideally we'd write bytecode (to make run-time loading fast), but because of the way jq does link-editing now we'd either have to re-write a lot of `compile.c`, `bytecode.c`, and `linker.c`, or we'd have to instead serialize the intermediate representation of jq programs (\"blocks\").  (Also, bytecode generation depends on knowing whether a function being called is C-coded or not.  Delaying this to run-time loading and linking time would require bytecode fixup anyways...)  Obviously, for a first version we could go for the latter.  But still, it'd be a fair amount of work.  For now all we need is the jq source and the fully-resolved dependency locators, something like: `[\"jqc\",1.5,{<dependency locators>},\"<source code>\"]`.  A future version might look like `[\"jqc\",1.6,<dependency locators>, <binary serialization of block representation of source>` (i.e., no longer JSON).  The file magic would be: it starts with `[\"jqc\",`.\r\n\r\nAn external tool could produce `[\"jqc\",1.5,{<dependency locators>, \"<source code>\"]` today, with no new help from jq (no `--compile` option needed).  I'm liking the `modulemeta` builtin as a helper for such an external tool: it can use it to extract the import directives, add any additional external pkg metadata, resolve universal identifiers/locators for the dependencies, install them, resolve local filesystem locators for them, and write the `.jqc` files.\r\n\r\nThis way we could support all the use-cases that we want:\r\n\r\n - run in-place\r\n - run with dependencies installed in `$HOME/.jq/<universe-name>`\r\n - run with dependencies installed in `$HOME/.jq/default/`\r\n - run with dependencies installed in `/usr/lib/jq/...` (perhaps with Nix-style `<hash>-<name>` directories or symlinks to them)\r\n - run with dependencies installed local to each dependent (except for and/or including programs, which might be in a bindir, but still, a `$ORIGIN/../lib/...`-style path should work).\r\n\r\nOf course, one could write programs/modules that only work with a pkg mgr (because no relative path is given).  That might be seen as unfriendly to users, but I think it's fine: if you're really writing production code in jq (as I have) then you'll want a build/install step anyways.  Even if you're just playing, an \"install dependencies\" step seems reasonable if you're going to use external dependencies you didn't write."}}