{"action":"created","issue":{"url":"https://api.github.com/repos/stedolan/jq/issues/1005","labels_url":"https://api.github.com/repos/stedolan/jq/issues/1005/labels{/name}","comments_url":"https://api.github.com/repos/stedolan/jq/issues/1005/comments","events_url":"https://api.github.com/repos/stedolan/jq/issues/1005/events","html_url":"https://github.com/stedolan/jq/pull/1005","id":113727320,"number":1005,"title":"C coded generators (explodes, readfile, _popenr)","user":{"login":"nicowilliams","id":604851,"avatar_url":"https://avatars.githubusercontent.com/u/604851?v=3","gravatar_id":"","url":"https://api.github.com/users/nicowilliams","html_url":"https://github.com/nicowilliams","followers_url":"https://api.github.com/users/nicowilliams/followers","following_url":"https://api.github.com/users/nicowilliams/following{/other_user}","gists_url":"https://api.github.com/users/nicowilliams/gists{/gist_id}","starred_url":"https://api.github.com/users/nicowilliams/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nicowilliams/subscriptions","organizations_url":"https://api.github.com/users/nicowilliams/orgs","repos_url":"https://api.github.com/users/nicowilliams/repos","events_url":"https://api.github.com/users/nicowilliams/events{/privacy}","received_events_url":"https://api.github.com/users/nicowilliams/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/stedolan/jq/labels/feature%20request","name":"feature request","color":"207de5"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/stedolan/jq/milestones/7","html_url":"https://github.com/stedolan/jq/milestones/1.6%20release","labels_url":"https://api.github.com/repos/stedolan/jq/milestones/7/labels","id":882049,"number":7,"title":"1.6 release","description":"","creator":{"login":"nicowilliams","id":604851,"avatar_url":"https://avatars.githubusercontent.com/u/604851?v=3","gravatar_id":"","url":"https://api.github.com/users/nicowilliams","html_url":"https://github.com/nicowilliams","followers_url":"https://api.github.com/users/nicowilliams/followers","following_url":"https://api.github.com/users/nicowilliams/following{/other_user}","gists_url":"https://api.github.com/users/nicowilliams/gists{/gist_id}","starred_url":"https://api.github.com/users/nicowilliams/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nicowilliams/subscriptions","organizations_url":"https://api.github.com/users/nicowilliams/orgs","repos_url":"https://api.github.com/users/nicowilliams/repos","events_url":"https://api.github.com/users/nicowilliams/events{/privacy}","received_events_url":"https://api.github.com/users/nicowilliams/received_events","type":"User","site_admin":false},"open_issues":29,"closed_issues":8,"state":"open","created_at":"2014-11-26T23:57:47Z","updated_at":"2015-10-28T02:11:40Z","due_on":null,"closed_at":null},"comments":12,"created_at":"2015-10-28T02:05:37Z","updated_at":"2015-11-01T21:53:07Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/stedolan/jq/pulls/1005","html_url":"https://github.com/stedolan/jq/pull/1005","diff_url":"https://github.com/stedolan/jq/pull/1005.diff","patch_url":"https://github.com/stedolan/jq/pull/1005.patch"},"body":"NOTE: NOT DONE YET.\r\n\r\nThis is just a proof of concept.  I'm able to read files, read from commands.\r\n\r\nTODO:\r\n\r\n - DONE: Implement @stedolan's comments regarding C-coded generators.\r\n - Make a nice jq-coded wrapper around `_popenr`\r\n - NOT NOW: Add an argument array input version of `_popenr` (an arg-vector form should use posix_spawn(), or spawn() on Windows, and bypass the shell)\r\n - Figure out how to handle raw vs. parsed vs. streamed, slurped vs. not.\r\n    - A separate builtin for each case?  (that's six cases, so six file reading builtins, six popen reading builtins, ...)\r\n    - Or one that takes an input describing what's desired?\r\n    - Or always read a line at a time, raw, and output strings with the newline (so that a final read without a newline could be denoted)?  The caller could slurp and/or parse as desired.  But what if the file has no newlines and is huge?  Incremental parsing seems like a much better option.\r\n - Implement privilege management design discussed earlier\r\n    - A command-line option for enabling privileges for specific modules\r\n    - Unprivileged versions of `readfile` and `_popenr` that raise an `error`\r\n - And, of course, increase test coverage (tests are missing; it's early days)\r\n\r\nMore importantly, how to handle something like `writefile(EXP)` or `popen(EXP)`?  Even C-coded generators can't call jq expressions.  So that means having file handles.  We can avoid those for `readfile`.  We've discussed how to do filehandles in a way that doesn't leak handles out, doesn't allow callers to slip in an incorrect handle, and doesn't have a dangling reference problem: make the C-coded functions to open/close handles and read from / write to handles be available only to privileged jq-coded functions that will use them in such a way as to never leak them:\r\n\r\n    def writefile(exp):\r\n        openfile_write as $handle | try (exp | filehandle_write) catch ($handle|filehandle_close), error;\r\n\r\n((Food for though: Some day it would be nice to be able to C-coded functions that take closure arguments.  I imagine setting up a trampoline in the VM for a C-coded function to call a closure argument, but to avoid filling the C call stack the C-coded function would have to immediately return and get called again with the closure's output(s); state would be kept in a state structure pointed to by the C-coded function's jq VM frame.  Calling closures in separate VMs could also be done, but arranging for them to be able to access their captured variables would be tricky.  With such an extension there'd be no need for handles for simple I/O builtins.))\r\n\r\nWe'll still need handles, and ones that are exposed to jq callers, for co-routines, so we might want to bite this bullet sooner rather than later.  I have an old branch with file handle support for the jq VM that I could revive.  Still, for simple I/O builtins, we don't need to expose a concept of handles, and that seems appealing."},"comment":{"url":"https://api.github.com/repos/stedolan/jq/issues/comments/152868429","html_url":"https://github.com/stedolan/jq/pull/1005#issuecomment-152868429","issue_url":"https://api.github.com/repos/stedolan/jq/issues/1005","id":152868429,"user":{"login":"nicowilliams","id":604851,"avatar_url":"https://avatars.githubusercontent.com/u/604851?v=3","gravatar_id":"","url":"https://api.github.com/users/nicowilliams","html_url":"https://github.com/nicowilliams","followers_url":"https://api.github.com/users/nicowilliams/followers","following_url":"https://api.github.com/users/nicowilliams/following{/other_user}","gists_url":"https://api.github.com/users/nicowilliams/gists{/gist_id}","starred_url":"https://api.github.com/users/nicowilliams/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nicowilliams/subscriptions","organizations_url":"https://api.github.com/users/nicowilliams/orgs","repos_url":"https://api.github.com/users/nicowilliams/repos","events_url":"https://api.github.com/users/nicowilliams/events{/privacy}","received_events_url":"https://api.github.com/users/nicowilliams/received_events","type":"User","site_admin":false},"created_at":"2015-11-01T21:53:07Z","updated_at":"2015-11-01T21:53:07Z","body":"A trampoline for C-coding jq functions that take and call closure arguments won't be that hard, and could be quite useful.  It will require two new opcodes: one to enter the C-coded function, and one to re-enter when a closure it \"calls\" outputs a value, that way the outputs of the closures can be made available to the C-coded closure.  A start at such a thing can be seen in the `c-coded-generics` branch of my clone.  (Among other things the C-coded generics could and would take arbitrary numbers of closure arguments.)\r\n\r\nThe idea is to make it possible to implement `parse(readfileraw)` with the existing C-coded JSON incremental parser.  A C-coded `parse/1` would invoke its argument closure to get a block of text to parse, update the parser, output any values until more input is needed, and then backtrack to the argument closure to get the next chunk of text to parse.\r\n\r\nThe same approach would allow for a C-coded `writefile(EXP)` that writes the outputs of `EXP` to a file.\r\n\r\nBoth could be implemented with something like file handles, of course.\r\n\r\nNote that we could even implement co-routines with this C-coded generic function scheme -- without handles.  A C-coded `coroutine(EXP; EXP; ..; EXP)` would create a VM for each closure argument, and would accept outputs from the expressions whose form allows the expressions to indicate that a value should be passed as an input to another co-routine (`{call:\"exp0,value:...}`)."}}