{"action":"created","issue":{"url":"https://api.github.com/repos/datacleaner/DataCleaner/issues/224","labels_url":"https://api.github.com/repos/datacleaner/DataCleaner/issues/224/labels{/name}","comments_url":"https://api.github.com/repos/datacleaner/DataCleaner/issues/224/comments","events_url":"https://api.github.com/repos/datacleaner/DataCleaner/issues/224/events","html_url":"https://github.com/datacleaner/DataCleaner/issues/224","id":56675564,"number":224,"title":"Output DataSet producers and jobs","user":{"login":"kaspersorensen","id":291450,"avatar_url":"https://avatars.githubusercontent.com/u/291450?v=3","gravatar_id":"","url":"https://api.github.com/users/kaspersorensen","html_url":"https://github.com/kaspersorensen","followers_url":"https://api.github.com/users/kaspersorensen/followers","following_url":"https://api.github.com/users/kaspersorensen/following{/other_user}","gists_url":"https://api.github.com/users/kaspersorensen/gists{/gist_id}","starred_url":"https://api.github.com/users/kaspersorensen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kaspersorensen/subscriptions","organizations_url":"https://api.github.com/users/kaspersorensen/orgs","repos_url":"https://api.github.com/users/kaspersorensen/repos","events_url":"https://api.github.com/users/kaspersorensen/events{/privacy}","received_events_url":"https://api.github.com/users/kaspersorensen/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/datacleaner/DataCleaner/labels/Component:%20Desktop","name":"Component: Desktop","color":"5319e7"},{"url":"https://api.github.com/repos/datacleaner/DataCleaner/labels/Component:%20Engine","name":"Component: Engine","color":"006b75"},{"url":"https://api.github.com/repos/datacleaner/DataCleaner/labels/Type:%20Enhancement","name":"Type: Enhancement","color":"009800"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/datacleaner/DataCleaner/milestones/10","html_url":"https://github.com/datacleaner/DataCleaner/milestones/4.1","labels_url":"https://api.github.com/repos/datacleaner/DataCleaner/milestones/10/labels","id":967128,"number":10,"title":"4.1","description":"","creator":{"login":"kaspersorensen","id":291450,"avatar_url":"https://avatars.githubusercontent.com/u/291450?v=3","gravatar_id":"","url":"https://api.github.com/users/kaspersorensen","html_url":"https://github.com/kaspersorensen","followers_url":"https://api.github.com/users/kaspersorensen/followers","following_url":"https://api.github.com/users/kaspersorensen/following{/other_user}","gists_url":"https://api.github.com/users/kaspersorensen/gists{/gist_id}","starred_url":"https://api.github.com/users/kaspersorensen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kaspersorensen/subscriptions","organizations_url":"https://api.github.com/users/kaspersorensen/orgs","repos_url":"https://api.github.com/users/kaspersorensen/repos","events_url":"https://api.github.com/users/kaspersorensen/events{/privacy}","received_events_url":"https://api.github.com/users/kaspersorensen/received_events","type":"User","site_admin":false},"open_issues":16,"closed_issues":18,"state":"open","created_at":"2015-02-05T14:07:01Z","updated_at":"2015-04-21T20:14:26Z","due_on":null,"closed_at":null},"comments":1,"created_at":"2015-02-05T14:18:37Z","updated_at":"2015-04-21T22:17:37Z","closed_at":null,"body":"Today one of the limitations to the compositional power of DataCleaner is that a job's data stream always ends when an analyzer is used. This is because the analyzer is performing an aggregate function on the dataset and as such it cannot produce records in the same cardinality as the input. We do have a couple of use cases where an aggregate function + further post-processing is necesary though:\r\n\r\n * Sorting data\r\n * Using a batch-based remote/web service for data transformation\r\n  * Will typically require sending off the whole dataset and then wait for a result. Once the result comes back the stream in DC is out of scope.\r\n * Deduplication\r\n  * Typically a duplicate detection component would detect duplicate records / pairs / groups (3 different output datasets).\r\n  * A succeeding merging component would consume those records and produce merged records.\r\n\r\nWe want to make this possible so that we can model larger logical flows in DataCleaner.\r\n\r\nFrom a XML spec standpoint, here's a draft example (using the deduplication scenario because that's the most complex one):\r\n\r\n```\r\n<analysis-job>\r\n<source>\r\n  <column name=\"person_id\" id=\"id\" />\r\n  <column name=\"Full name\" id=\"fullname\" />\r\n  <column name=\"Age\" id=\"age\" />\r\n</source>\r\n<analyzer descriptor=\"Duplicate detection\">\r\n  <input ref=\"fullname\" />\r\n  <input ref=\"age\" />\r\n  <output-dataset id=\"PAIRS\" name=\"Pairs\">\r\n    <analysis-job>\r\n      <source>\r\n         <column name=\"RecordID1\" id=\"id1\" />\r\n         <column name=\"RecordID2\" id=\"id2\" />\r\n         <column name=\"Score\" id=\"score\" />\r\n      </source>\r\n      ...\r\n    </analysis-job>\r\n  </output-dataset>\r\n  <output-dataset id=\"RECORDS\" name=\"Records\">\r\n    ...\r\n  </output-dataset>\r\n</analyzer>\r\n</analysis-job>\r\n```\r\n\r\nThings to notice:\r\n\r\n * The analyzer can have multiple output-datasets\r\n * In the output-dataset we can introduce a completely new analysis job.\r\n * The column definitions and references in the output-dataset's job are completely separate from the outer analysis job. In fact, they should also not have any way to access the outer data stream because the scope and granularity will not have any direct connection."},"comment":{"url":"https://api.github.com/repos/datacleaner/DataCleaner/issues/comments/94958914","html_url":"https://github.com/datacleaner/DataCleaner/issues/224#issuecomment-94958914","issue_url":"https://api.github.com/repos/datacleaner/DataCleaner/issues/224","id":94958914,"user":{"login":"kaspersorensen","id":291450,"avatar_url":"https://avatars.githubusercontent.com/u/291450?v=3","gravatar_id":"","url":"https://api.github.com/users/kaspersorensen","html_url":"https://github.com/kaspersorensen","followers_url":"https://api.github.com/users/kaspersorensen/followers","following_url":"https://api.github.com/users/kaspersorensen/following{/other_user}","gists_url":"https://api.github.com/users/kaspersorensen/gists{/gist_id}","starred_url":"https://api.github.com/users/kaspersorensen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kaspersorensen/subscriptions","organizations_url":"https://api.github.com/users/kaspersorensen/orgs","repos_url":"https://api.github.com/users/kaspersorensen/repos","events_url":"https://api.github.com/users/kaspersorensen/events{/privacy}","received_events_url":"https://api.github.com/users/kaspersorensen/received_events","type":"User","site_admin":false},"created_at":"2015-04-21T22:17:37Z","updated_at":"2015-04-21T22:17:37Z","body":"Some random thoughts on this ...\r\n\r\nDesigning a good interface for components that produce output datasets is a bit of a challenge. I can imagine that they would need at least to implement a method that provides metadata (column names, types etc.) for the OutputDataSet:\r\n\r\n```\r\npublic OutputDataSetMetadata getOutputDataSetMetadata()\r\n```\r\n\r\nthis metadata object would essentially contain one or more metamodel Tables. Why Table? Because if we want to reuse AnalysisJob and all that comes with it (builders and stuff) then we at some point need Column objects and so on too. So I think something like this would make sense (but probably not exhaustive):\r\n\r\n```\r\ninterface OutputDataSetMetadata {\r\n  public Table[] getTables();\r\n}\r\n```\r\n\r\nNow to the part of publishing the records of these tables. The thing that is hard to say here is whether that should be push-based or pull-based. Push based would essentially be easier, it could basically be a simple addition of a table as a \"target qualifier\" to the existing OutputRowCollector interface:\r\n\r\n```\r\n@Provided OutputRowCollector outputRowCollector;\r\n\r\n// in either run/transform/filter/getResult method:\r\noutputRowCollector.putValues(targetTable, new Object[] { ... });\r\n```\r\n\r\nBut pull-based is a lot easier to manage if we want to support optimizations such as:\r\n\r\n * Omitting output datasets that have no consumers in a particular job (probably pretty common, so I think this optimization is pretty important).\r\n * Allowing query optimization (e.g. Filter push down stuff such as a Max rows filter) in the consuming analysis job (probably pretty complex, so I consider it less important but nice to allow at least from an API point of view).\r\n\r\nThat could basically look like a metamodel query-like method on the component's interface:\r\n```\r\npublic DataSet provideOutputDataSet(Query q);\r\n```\r\nTo implement this method we could provide a few simple utilities leveraging MetaModel's query engine to provide the table data in various ways. The component could also have stored the data temporarily and simply serve this query by using a normal MetaModel DataContext.\r\n\r\nIt would be convenient in the pull-based approach also to have some way as a component to know if an output dataset is ever consumed or not. That may save time during the execution of the job if certain steps can be omitted.\r\n\r\nThe downside of pull based is that then it becomes hard to implement anything that is streaming - which would be great for performance when it is possible. Could we do both?"}}