{"action":"created","issue":{"url":"https://api.github.com/repos/HazyResearch/deepdive/issues/445","labels_url":"https://api.github.com/repos/HazyResearch/deepdive/issues/445/labels{/name}","comments_url":"https://api.github.com/repos/HazyResearch/deepdive/issues/445/comments","events_url":"https://api.github.com/repos/HazyResearch/deepdive/issues/445/events","html_url":"https://github.com/HazyResearch/deepdive/pull/445","id":122645673,"number":445,"title":"DeepDive execution plan compiler","user":{"login":"netj","id":53634,"avatar_url":"https://avatars.githubusercontent.com/u/53634?v=3","gravatar_id":"","url":"https://api.github.com/users/netj","html_url":"https://github.com/netj","followers_url":"https://api.github.com/users/netj/followers","following_url":"https://api.github.com/users/netj/following{/other_user}","gists_url":"https://api.github.com/users/netj/gists{/gist_id}","starred_url":"https://api.github.com/users/netj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/netj/subscriptions","organizations_url":"https://api.github.com/users/netj/orgs","repos_url":"https://api.github.com/users/netj/repos","events_url":"https://api.github.com/users/netj/events{/privacy}","received_events_url":"https://api.github.com/users/netj/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"feiranwang","id":5935656,"avatar_url":"https://avatars.githubusercontent.com/u/5935656?v=3","gravatar_id":"","url":"https://api.github.com/users/feiranwang","html_url":"https://github.com/feiranwang","followers_url":"https://api.github.com/users/feiranwang/followers","following_url":"https://api.github.com/users/feiranwang/following{/other_user}","gists_url":"https://api.github.com/users/feiranwang/gists{/gist_id}","starred_url":"https://api.github.com/users/feiranwang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/feiranwang/subscriptions","organizations_url":"https://api.github.com/users/feiranwang/orgs","repos_url":"https://api.github.com/users/feiranwang/repos","events_url":"https://api.github.com/users/feiranwang/events{/privacy}","received_events_url":"https://api.github.com/users/feiranwang/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/HazyResearch/deepdive/milestones/5","html_url":"https://github.com/HazyResearch/deepdive/milestones/DeepDive%200.8.0","labels_url":"https://api.github.com/repos/HazyResearch/deepdive/milestones/5/labels","id":1203001,"number":5,"title":"DeepDive 0.8.0","description":"* Refactored internals\r\n* Fine-grain control over DeepDive runs\r\n* Efficient extraction and grounding","creator":{"login":"netj","id":53634,"avatar_url":"https://avatars.githubusercontent.com/u/53634?v=3","gravatar_id":"","url":"https://api.github.com/users/netj","html_url":"https://github.com/netj","followers_url":"https://api.github.com/users/netj/followers","following_url":"https://api.github.com/users/netj/following{/other_user}","gists_url":"https://api.github.com/users/netj/gists{/gist_id}","starred_url":"https://api.github.com/users/netj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/netj/subscriptions","organizations_url":"https://api.github.com/users/netj/orgs","repos_url":"https://api.github.com/users/netj/repos","events_url":"https://api.github.com/users/netj/events{/privacy}","received_events_url":"https://api.github.com/users/netj/received_events","type":"User","site_admin":false},"open_issues":24,"closed_issues":4,"state":"open","created_at":"2015-07-10T20:16:11Z","updated_at":"2015-12-17T21:18:21Z","due_on":"2015-12-21T08:00:00Z","closed_at":null},"comments":19,"created_at":"2015-12-17T02:07:38Z","updated_at":"2015-12-30T21:48:25Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/HazyResearch/deepdive/pulls/445","html_url":"https://github.com/HazyResearch/deepdive/pull/445","diff_url":"https://github.com/HazyResearch/deepdive/pull/445.diff","patch_url":"https://github.com/HazyResearch/deepdive/pull/445.patch"},"body":"This not-so-small PR adds a new implementation of DeepDive that literally compiles an *execution plan* to run the app in a much more efficient way in both human time and machine time/space.  I invite everyone to review the code and give a try on your existing app (especially @feiranwang @SenWu @ajratner @zhangce @raphaelhoffmann @alldefector @zifeishan @xiaoling @Colossus @ThomasPalomares @juhanaka)  The plan is to get feedback over the next few days while I update the documentation and carve out release v0.8.0.\r\n\r\nAn execution plan is basically a set of shell scripts that tells what to run for user-defined extractors as well as the built-in processes for grounding the factor graph and performing learning and inference, and a Makefile and that describes the complete dependencies among them.  These are compiled from the app's deepdive.conf, app.ddlog, and schema.json, mainly by a series of JSON transformations implemented as [jq programs](https://stedolan.github.io/jq/tutorial/) with a little bit of help from bash, which all resides under `compiler/` in the source tree.\r\n\r\nIt implements most of the existing functionality provided by the current Scala implementation, except a few things, e.g., Greenplum parallel unloading/loading, which won't be difficult to add given the much much modular architecture.  On the other hand, exciting new features and improvements are added.  To highlight just a few:\r\n* **Full Dependency Support with Selective Execution.** It's now possible to selectively run, repeat, or skip certain parts of the app's data flow or extraction pipeline without being aware of all the dependencies between them. (fixes #431, closes #427, closes #273)  The user has full control over every step of the execution plan.  Not only that, but grounding is also broken down into smaller processes, so it's possible to just change or add one inference rule and update the grounded factor graph without having to recompute everything from scratch. (fixes #280)\r\n* **Zero Footprint Extraction.** `tsv_extractor`s now have nearly zero footprint on the filesystem.  The data is streamed from the database through the UDFs and back to the database.  [mkmimo](https://github.com/netj/mkmimo) is used with named pipes to make the connection between the database and the parallel UDF processes (cf. `$DEEPDIVE_NUM_PROCESSES`). (fixes #428)  It doesn't support other extractor styles yet, but we can probably drop them unless there's a compelling reason. (closes #384)\r\n* **Compute Drivers.** A new compute driver architecture for executing such extractors is now in place, so it's now clear where to extend to support remote execution or clusters with a job scheduler, such as Hadoop/YARN, SLURM, Torque/GridEngine/PBS. (#426)  The local execution driver is what implements the streaming `tsv_extractor` mentioned above.  Moreover, the grounding processes as well as the user-defined extractors also make use of the compute drivers, so some parts of the grounding will automatically take advantage of such extensions.\r\n* **Zero Footprint Grounding.** The grounding processes also minimizes footprint on the filesystem.  No data for the factor graph are duplicated.  Instead of creating a concatenated copies of factors, weights, and variables, they are merged as the sampler loads them.  Also, the binary format conversion is done on the fly as the grounded rows are unloaded from the database, so no ephemeral textual form is ever stored anywhere.  In fact, only a few line changes to the compiler can compress the binary forms and shrink the factor graphs footprint on the filesystem by an order of magnitude (not included in this PR).\r\n* **More User Commands.** `deepdive.pipeline` config is obsolete as well as the `deepdive run` command, although they'll work the same as before.  Now, the user can simply state what is the goal for execution to the `deepdive do` command, e.g., `deepdive do model/calibration-plot` or `deepdive do data/has_spouse_features`, as many times as necessary once the app is compiled with the `deepdive compile` command.  Supporting commands such as `deepdive plan`, `deepdive mark`, `deepdive redo` are there to speed up typical user workflows with DeepDive apps.  `deepdive initdb` and `deepdive load` have been improved to be more useful, and is used by a few compiled processes. (fixes #351, fixes #357)\r\n* **Error Checking.** Errors in the app (and of course the compiler itself) are checked at compile time, and can also be done by the user with the `deepdive check` command.  The checkers are modular and isolated, so many useful checks can be quickly added, such as test firing UDF commands, etc.  Only basic checks have been implemented so far. (fixes #349, fixes #1)\r\n* **Simpler, Efficient Multinomial Factors.** Multinomial factors are won't materialize unnecessary data and use VIEWs as much as possible, e.g., dd_*_cardinality tables and the dd_graph_weights.  Also, nearly no code has been duplicated to support it, so that's good news for developers.\r\n* **Bundled Runtime Dependencies.** DeepDive now builds essential runtime dependencies, such as bash, coreutils, jq, bc, graphviz, so no more Mac vs. Linux or software installation version issues will pop up. (fixes #441 as documentation also ended up in this PR)\r\n\r\nAlso some good stuffs that comes with a clean rewrite:\r\n* Closes #412 by doing a reasonable job and making the base relations clear to the user, and also closes  #421.\r\n* Closes #110 as no more SQL parsing is done.\r\n* Closes #383 as logging has been completely redone.\r\n* Closes #361, closes #20 as JDBC is no longer used.\r\n* Closes #329 as Scala implementation will be simply dropped in a future release (v0.8.1 or v0.9?)."},"comment":{"url":"https://api.github.com/repos/HazyResearch/deepdive/issues/comments/168079546","html_url":"https://github.com/HazyResearch/deepdive/pull/445#issuecomment-168079546","issue_url":"https://api.github.com/repos/HazyResearch/deepdive/issues/445","id":168079546,"user":{"login":"feiranwang","id":5935656,"avatar_url":"https://avatars.githubusercontent.com/u/5935656?v=3","gravatar_id":"","url":"https://api.github.com/users/feiranwang","html_url":"https://github.com/feiranwang","followers_url":"https://api.github.com/users/feiranwang/followers","following_url":"https://api.github.com/users/feiranwang/following{/other_user}","gists_url":"https://api.github.com/users/feiranwang/gists{/gist_id}","starred_url":"https://api.github.com/users/feiranwang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/feiranwang/subscriptions","organizations_url":"https://api.github.com/users/feiranwang/orgs","repos_url":"https://api.github.com/users/feiranwang/repos","events_url":"https://api.github.com/users/feiranwang/events{/privacy}","received_events_url":"https://api.github.com/users/feiranwang/received_events","type":"User","site_admin":false},"created_at":"2015-12-30T21:48:24Z","updated_at":"2015-12-30T21:48:24Z","body":"@netj I tried this on several examples. `deepdive run` and  `deepdive do all` work great. I have a small problem with `deepdive do`. When I type `deepdive do`, it gives a list of targets. In smoke example there's a `process/grounding/variable/person_has_cancer/assign_id`. When I tried to run with that target, it gives an error\r\n```\r\nprocess/grounding/variable/person_has_cancer/assign_id: Unknown target\r\n```"}}