{"action":"created","comment":{"url":"https://api.github.com/repos/ckan/ideas-and-roadmap/pulls/comments/43401889","id":43401889,"diff_hunk":"@@ -0,0 +1,130 @@\n+# CKAN / Data Package integration\n+\n+## Overview\n+[Data Packages](http://data.okfn.org/doc/data-package) are a really simple specification for packaging up data files to allow easy reuse and sharing. They are essentially a descriptor file named `datapackage.json` which contains metadata about the dataset and points towards the data files.   It has a focus on simplicity and extensibility.\n+\n+For instance, there is a special flavour of Data Package aimed at [Tabular data](http://data.okfn.org/doc/tabular-data-package) which enforces the use of the [JSON Table schema](http://dataprotocols.org/json-table-schema/), another spec part of the same family of standards which allows to describe the structure of CSV files.\n+\n+The full Data Package spec can be found here:\n+\n+http://dataprotocols.org/data-packages/\n+\n+There is a growing number of [tools](http://data.okfn.org/tools) being developed around Data Packages. Improving its support within CKAN would allow out of the box integration with this ecosystem of tools.\n+\n+Also besides direct support for handling Data Packages, there is a number of areas where adopting and implementing standards around them could help deal with long standing issues in CKAN. Perhaps the most clear example is the JSON Table Schema, which would allow publishers to properly describe data shown on previews or being imported to the DataStore. Implementing this would be a massive win in order to build better import and ETL tools into CKAN (or even just to get the previews right).\n+\n+## Prior work\n+\n+Open Knowledge developed a while ago http://datapackager.okfn.org, a tool for creating Data Packages from data files.\n+\n+It was implemented in CKAN (as a CKAN [extension](https://github.com/ckan/ckanext-datapackager) that heavily customized some parts of the CKAN UI).\n+\n+It contains functions to translate Data Packages metadata into CKAN datasets and viceversa, but perhaps more interestingly includes logic functions and an interface for modifying the schema for tabular resources:\n+\n+![Schemas](http://i.imgur.com/i5L5oNz.png)\n+\n+\n+\n+## Main goals\n+\n+1. Allow importing Data Packages into CKAN, creating the necessary datasets and resources, both via the UI and the API\n+\n+2. Allow exporting CKAN datasets as Data Packages, regardless of whether they were imported from Data Packages in a first instance or not.\n+\n+3. Support schema definition for CKAN resources, allowing publishers to describe the contents of published files\n+\n+\n+For achieving this it seems sensible to reuse as much stuff as we can from ckanext-datapackager, extending it if necessary\n+\n+The first two points could be implemented either in core or separate extension, but I'd like to propose that the third regarding resource schemas is implemented in core (in a generic form). It seems central enough to all future work around data import, cleaning, etc (eg closer integration with DataStore) to justify it.\n+\n+## Field mapping    \n+\n+Both for transforming Data Packages into CKAN datasets and to generate Data Packages from CKAN datasets we need to have a good mapping between the two.\n+\n+CKAN **dataset** core fields have essentially the same equivalents on Data Packages (`title`, `description` , `keywords`, ...). Potential issues:\n+\n+* `name` - collision with existing CKAN one (we can just add an incremental number)\n+* `license` - mapping to the ones available on the CKAN instance. **Q:** What happens if we can not map it?\n+* `author` / `contributors` - these refer to the datapackage creators not the data ones (`sources` is used for this) **Q:** Should be bother trying to map to CKAN's `author` / `author_email` or `maintainer` /  `maintainer_email`?\n+\n+Data Packages can have extra fields not present in the CKAN schema:\n+* `sources` (name, web, email)\n+* `image`\n+* `base`\n+* `dataDependencies` (ordering of the hash keys is important)\n+* `schemas` \n+* Arbitrary fields added to the Data Package\n+\n+If we don't store the original Data Package somewhere we'll need to store these as extras. \n+\n+As per **resources**,  Data Packages `resources` can point the actual data files using `url` or `path` + `base`, or have the data embedded with a `data` field. (The current spec allows all of them to be present at the same time, which hopefully we can restrict https://github.com/dataprotocols/dataprotocols/issues/223). If Data Packages have `data` embedded we can push it to the DataStore to make it available.\n+\n+Most fields map to CKAN core ones. The ones that are not defined are:\n+\n+* `encoding`\n+* `hash`\n+* `sources`\n+* `license`\n+\n+This is not as problematic as dataset fields, as we can add arbitrary top level fields to resources.\n+\n+## Importing Data Packages\n+\n+### Implementation\n+\n+At a higher level this seems quite straight-forward: a  `datapackage_import` or similar action which inherits the auth from `package_create` and `package_update` (eg you need to belong to an org by default etc). The action will validate the JSON object, transform it to a CKAN dict and create or update the relevant package. A controller will handle requests from the UI (at `/datapackage/import`).\n+In terms of UI there would be a dedicated page for importing Data Packages. This will essentially contain just an upload field to upload the `datapackage.json` or a text file to enter the URL to an accessible one. This page would show validation errors and redirect to the dataset page once it's created.\n+\n+Perhaps to make it more accessible we can link to this page from the dataset creation form (from the top bar or the left hand column).\n+\n+### Potential issues\n+\n+Identifiers: Data Packages are solely identified by the `name` field, which according to the Data Package spec:\n+\n+> [is a] short url-usable (and preferably human-readable) name of the package. This MUST be lower-case and contain only alphanumeric characters along with \u001C., \u001C_ or \u001C-\u001C characters. It will function as a unique identifier and therefore SHOULD be unique in relation to any registry in which this package will be deposited (and preferably globally unique).\n+\n+We can not assume that these identifiers will be unique and that they will not clash with existing CKAN datasets. We can append stuff to the name but then we need to consider what happens when we reupload the same Data Package (ie to update the dataset).\n+\n+**Q:** Do we upload the files described on the Data Package to the CKAN server?\n+I would say not by default. As long as the files are publicly accessible the DataPusher can get them into the DataStore.\n+\n+**Q:** Do we keep a copy of the original `datapackage.json` ?\n+I think yes. Trying to store all fields as extras and replicate them back when generating the datapackage back would be too much hassle. Also we would need to keep adding fields as the spec changes and handle arbitrary fields added by publishers. An easier approach might be outputting back what was uploaded, but with the fields that are actually mapped to CKAN (title, description, keywords etc) updated, as these might have been modified in CKAN.\n+","path":"specs/datapackages/README.md","position":94,"original_position":94,"commit_id":"0f4a73ccd50a8d5412a7bb2e20826153beccc824","original_commit_id":"0f4a73ccd50a8d5412a7bb2e20826153beccc824","user":{"login":"rossjones","id":118063,"avatar_url":"https://avatars.githubusercontent.com/u/118063?v=3","gravatar_id":"","url":"https://api.github.com/users/rossjones","html_url":"https://github.com/rossjones","followers_url":"https://api.github.com/users/rossjones/followers","following_url":"https://api.github.com/users/rossjones/following{/other_user}","gists_url":"https://api.github.com/users/rossjones/gists{/gist_id}","starred_url":"https://api.github.com/users/rossjones/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rossjones/subscriptions","organizations_url":"https://api.github.com/users/rossjones/orgs","repos_url":"https://api.github.com/users/rossjones/repos","events_url":"https://api.github.com/users/rossjones/events{/privacy}","received_events_url":"https://api.github.com/users/rossjones/received_events","type":"User","site_admin":false},"body":"Is there an ETA on when the spec will be 1.0, and no longer a draft? Having the spec shifting under the feet of the developer seems like it will be problematic, and force things like storing an updated datapackage.json unnecessarily?  ","created_at":"2015-10-29T15:41:14Z","updated_at":"2015-10-29T15:41:14Z","html_url":"https://github.com/ckan/ideas-and-roadmap/pull/160#discussion_r43401889","pull_request_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/pulls/160","_links":{"self":{"href":"https://api.github.com/repos/ckan/ideas-and-roadmap/pulls/comments/43401889"},"html":{"href":"https://github.com/ckan/ideas-and-roadmap/pull/160#discussion_r43401889"},"pull_request":{"href":"https://api.github.com/repos/ckan/ideas-and-roadmap/pulls/160"}}},"pull_request":{"url":"https://api.github.com/repos/ckan/ideas-and-roadmap/pulls/160","id":49140476,"html_url":"https://github.com/ckan/ideas-and-roadmap/pull/160","diff_url":"https://github.com/ckan/ideas-and-roadmap/pull/160.diff","patch_url":"https://github.com/ckan/ideas-and-roadmap/pull/160.patch","issue_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/issues/160","number":160,"state":"open","locked":false,"title":"First draft of the CKAN / DataPackages spec","user":{"login":"amercader","id":200230,"avatar_url":"https://avatars.githubusercontent.com/u/200230?v=3","gravatar_id":"","url":"https://api.github.com/users/amercader","html_url":"https://github.com/amercader","followers_url":"https://api.github.com/users/amercader/followers","following_url":"https://api.github.com/users/amercader/following{/other_user}","gists_url":"https://api.github.com/users/amercader/gists{/gist_id}","starred_url":"https://api.github.com/users/amercader/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amercader/subscriptions","organizations_url":"https://api.github.com/users/amercader/orgs","repos_url":"https://api.github.com/users/amercader/repos","events_url":"https://api.github.com/users/amercader/events{/privacy}","received_events_url":"https://api.github.com/users/amercader/received_events","type":"User","site_admin":false},"body":"","created_at":"2015-10-29T13:43:22Z","updated_at":"2015-10-29T15:41:14Z","closed_at":null,"merged_at":null,"merge_commit_sha":"531fc42b3f7d0a5168020f374f5bd3b43e1f3290","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/pulls/160/commits","review_comments_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/pulls/160/comments","review_comment_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/pulls/comments{/number}","comments_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/issues/160/comments","statuses_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/statuses/0f4a73ccd50a8d5412a7bb2e20826153beccc824","head":{"label":"ckan:datapackages-spec","ref":"datapackages-spec","sha":"0f4a73ccd50a8d5412a7bb2e20826153beccc824","user":{"login":"ckan","id":1630326,"avatar_url":"https://avatars.githubusercontent.com/u/1630326?v=3","gravatar_id":"","url":"https://api.github.com/users/ckan","html_url":"https://github.com/ckan","followers_url":"https://api.github.com/users/ckan/followers","following_url":"https://api.github.com/users/ckan/following{/other_user}","gists_url":"https://api.github.com/users/ckan/gists{/gist_id}","starred_url":"https://api.github.com/users/ckan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ckan/subscriptions","organizations_url":"https://api.github.com/users/ckan/orgs","repos_url":"https://api.github.com/users/ckan/repos","events_url":"https://api.github.com/users/ckan/events{/privacy}","received_events_url":"https://api.github.com/users/ckan/received_events","type":"Organization","site_admin":false},"repo":{"id":7557010,"name":"ideas-and-roadmap","full_name":"ckan/ideas-and-roadmap","owner":{"login":"ckan","id":1630326,"avatar_url":"https://avatars.githubusercontent.com/u/1630326?v=3","gravatar_id":"","url":"https://api.github.com/users/ckan","html_url":"https://github.com/ckan","followers_url":"https://api.github.com/users/ckan/followers","following_url":"https://api.github.com/users/ckan/following{/other_user}","gists_url":"https://api.github.com/users/ckan/gists{/gist_id}","starred_url":"https://api.github.com/users/ckan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ckan/subscriptions","organizations_url":"https://api.github.com/users/ckan/orgs","repos_url":"https://api.github.com/users/ckan/repos","events_url":"https://api.github.com/users/ckan/events{/privacy}","received_events_url":"https://api.github.com/users/ckan/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/ckan/ideas-and-roadmap","description":"CKAN Feature Ideas and Roadmap","fork":false,"url":"https://api.github.com/repos/ckan/ideas-and-roadmap","forks_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/forks","keys_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/teams","hooks_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/hooks","issue_events_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/issues/events{/number}","events_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/events","assignees_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/assignees{/user}","branches_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/branches{/branch}","tags_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/tags","blobs_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/git/refs{/sha}","trees_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/statuses/{sha}","languages_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/languages","stargazers_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/stargazers","contributors_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/contributors","subscribers_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/subscribers","subscription_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/subscription","commits_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/commits{/sha}","git_commits_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/git/commits{/sha}","comments_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/comments{/number}","issue_comment_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/issues/comments{/number}","contents_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/contents/{+path}","compare_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/merges","archive_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/downloads","issues_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/issues{/number}","pulls_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/pulls{/number}","milestones_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/milestones{/number}","notifications_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/labels{/name}","releases_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/releases{/id}","created_at":"2013-01-11T09:07:26Z","updated_at":"2015-10-25T14:32:37Z","pushed_at":"2015-10-29T13:43:22Z","git_url":"git://github.com/ckan/ideas-and-roadmap.git","ssh_url":"git@github.com:ckan/ideas-and-roadmap.git","clone_url":"https://github.com/ckan/ideas-and-roadmap.git","svn_url":"https://github.com/ckan/ideas-and-roadmap","homepage":"https://github.com/ckan/ideas/issues","size":264,"stargazers_count":21,"watchers_count":21,"language":null,"has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":141,"forks":0,"open_issues":141,"watchers":21,"default_branch":"master"}},"base":{"label":"ckan:master","ref":"master","sha":"8a5f535a51f0c4d9d9650ca4d9037b0e648c960f","user":{"login":"ckan","id":1630326,"avatar_url":"https://avatars.githubusercontent.com/u/1630326?v=3","gravatar_id":"","url":"https://api.github.com/users/ckan","html_url":"https://github.com/ckan","followers_url":"https://api.github.com/users/ckan/followers","following_url":"https://api.github.com/users/ckan/following{/other_user}","gists_url":"https://api.github.com/users/ckan/gists{/gist_id}","starred_url":"https://api.github.com/users/ckan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ckan/subscriptions","organizations_url":"https://api.github.com/users/ckan/orgs","repos_url":"https://api.github.com/users/ckan/repos","events_url":"https://api.github.com/users/ckan/events{/privacy}","received_events_url":"https://api.github.com/users/ckan/received_events","type":"Organization","site_admin":false},"repo":{"id":7557010,"name":"ideas-and-roadmap","full_name":"ckan/ideas-and-roadmap","owner":{"login":"ckan","id":1630326,"avatar_url":"https://avatars.githubusercontent.com/u/1630326?v=3","gravatar_id":"","url":"https://api.github.com/users/ckan","html_url":"https://github.com/ckan","followers_url":"https://api.github.com/users/ckan/followers","following_url":"https://api.github.com/users/ckan/following{/other_user}","gists_url":"https://api.github.com/users/ckan/gists{/gist_id}","starred_url":"https://api.github.com/users/ckan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ckan/subscriptions","organizations_url":"https://api.github.com/users/ckan/orgs","repos_url":"https://api.github.com/users/ckan/repos","events_url":"https://api.github.com/users/ckan/events{/privacy}","received_events_url":"https://api.github.com/users/ckan/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/ckan/ideas-and-roadmap","description":"CKAN Feature Ideas and Roadmap","fork":false,"url":"https://api.github.com/repos/ckan/ideas-and-roadmap","forks_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/forks","keys_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/teams","hooks_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/hooks","issue_events_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/issues/events{/number}","events_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/events","assignees_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/assignees{/user}","branches_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/branches{/branch}","tags_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/tags","blobs_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/git/refs{/sha}","trees_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/statuses/{sha}","languages_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/languages","stargazers_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/stargazers","contributors_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/contributors","subscribers_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/subscribers","subscription_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/subscription","commits_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/commits{/sha}","git_commits_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/git/commits{/sha}","comments_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/comments{/number}","issue_comment_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/issues/comments{/number}","contents_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/contents/{+path}","compare_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/merges","archive_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/downloads","issues_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/issues{/number}","pulls_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/pulls{/number}","milestones_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/milestones{/number}","notifications_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/labels{/name}","releases_url":"https://api.github.com/repos/ckan/ideas-and-roadmap/releases{/id}","created_at":"2013-01-11T09:07:26Z","updated_at":"2015-10-25T14:32:37Z","pushed_at":"2015-10-29T13:43:22Z","git_url":"git://github.com/ckan/ideas-and-roadmap.git","ssh_url":"git@github.com:ckan/ideas-and-roadmap.git","clone_url":"https://github.com/ckan/ideas-and-roadmap.git","svn_url":"https://github.com/ckan/ideas-and-roadmap","homepage":"https://github.com/ckan/ideas/issues","size":264,"stargazers_count":21,"watchers_count":21,"language":null,"has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":141,"forks":0,"open_issues":141,"watchers":21,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/ckan/ideas-and-roadmap/pulls/160"},"html":{"href":"https://github.com/ckan/ideas-and-roadmap/pull/160"},"issue":{"href":"https://api.github.com/repos/ckan/ideas-and-roadmap/issues/160"},"comments":{"href":"https://api.github.com/repos/ckan/ideas-and-roadmap/issues/160/comments"},"review_comments":{"href":"https://api.github.com/repos/ckan/ideas-and-roadmap/pulls/160/comments"},"review_comment":{"href":"https://api.github.com/repos/ckan/ideas-and-roadmap/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/ckan/ideas-and-roadmap/pulls/160/commits"},"statuses":{"href":"https://api.github.com/repos/ckan/ideas-and-roadmap/statuses/0f4a73ccd50a8d5412a7bb2e20826153beccc824"}}}}