{"action":"created","issue":{"url":"https://api.github.com/repos/18F/data-act-pilot/issues/161","labels_url":"https://api.github.com/repos/18F/data-act-pilot/issues/161/labels{/name}","comments_url":"https://api.github.com/repos/18F/data-act-pilot/issues/161/comments","events_url":"https://api.github.com/repos/18F/data-act-pilot/issues/161/events","html_url":"https://github.com/18F/data-act-pilot/issues/161","id":103551149,"number":161,"title":"Research issue: open source data standards","user":{"login":"bsweger","id":540544,"avatar_url":"https://avatars.githubusercontent.com/u/540544?v=3","gravatar_id":"","url":"https://api.github.com/users/bsweger","html_url":"https://github.com/bsweger","followers_url":"https://api.github.com/users/bsweger/followers","following_url":"https://api.github.com/users/bsweger/following{/other_user}","gists_url":"https://api.github.com/users/bsweger/gists{/gist_id}","starred_url":"https://api.github.com/users/bsweger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bsweger/subscriptions","organizations_url":"https://api.github.com/users/bsweger/orgs","repos_url":"https://api.github.com/users/bsweger/repos","events_url":"https://api.github.com/users/bsweger/events{/privacy}","received_events_url":"https://api.github.com/users/bsweger/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/18F/data-act-pilot/labels/research%20questions","name":"research questions","color":"009800"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":38,"created_at":"2015-08-27T16:52:23Z","updated_at":"2015-09-15T21:35:30Z","closed_at":null,"body":"Opening this issue to track thoughts on open source data schemas/standards/formats.\r\n\r\n**General Questions**\r\n* How are people in the world implementing open-source data standards/schemas and moving data around?\r\n* Do special considerations exist for (non-PII, non-confidential) federal spending govt. data standards that distinguish this space from data standard implementations in other industries?\r\n* How important is it to build validations/metadata into a schema versus having a separate validation process and metadata definitions?\r\n* Related: what are the pros and cons of building a data input/validation process tied to a particular schema syntax vs. de-coupling syntax from validations/metadata?\r\n\r\n**Questions for specific data standard/schema options:**\r\n* How robust is its open-source ecosystem? Has the open source community developed tooling around it? If so, how actively are the tools updated/maintained, and how large is the contributor base? Will users incur licensing fees or get locked into a vendor?\r\n* How widely adopted is the data standard/schema? Is it limited to a particular domain?\r\n* How easy is it for data submitters/producers (both biz and tech) to use?\r\n* How easy is it for data consumers (both biz and tech) to use?\r\n* What resources exist for non-coders to understand the data standard/schema? Is that even relevant if good, robust metadata exists (see above)?\r\n* What's the process for updating and tracking changes to the schema/standard?\r\n\r\n**These are schemas/standards/formats we know about. What are others?**\r\n* XML/[XBRL](https://www.xbrl.org/)\r\n* [JSON](http://json.org/)/[JSON-LD](http://json-ld.org/)/[JSON Schema](http://json-schema.org/)\r\n* [Protocol Buffers](https://developers.google.com/protocol-buffers/?hl=en)\r\n* [Avro](https://avro.apache.org/)\r\n* [Thrift](http://thrift.apache.org/)\r\n* [Message Pack](http://msgpack.org/)\r\n* [ORC](https://orc.apache.org/) --> likely not relevant but here for completeness\r\n* [Parquet](http://parquet.apache.org/) --> likely not relevant but here for completeness\r\n\r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/18F/data-act-pilot/issues/comments/140553923","html_url":"https://github.com/18F/data-act-pilot/issues/161#issuecomment-140553923","issue_url":"https://api.github.com/repos/18F/data-act-pilot/issues/161","id":140553923,"user":{"login":"scopatz","id":320553,"avatar_url":"https://avatars.githubusercontent.com/u/320553?v=3","gravatar_id":"","url":"https://api.github.com/users/scopatz","html_url":"https://github.com/scopatz","followers_url":"https://api.github.com/users/scopatz/followers","following_url":"https://api.github.com/users/scopatz/following{/other_user}","gists_url":"https://api.github.com/users/scopatz/gists{/gist_id}","starred_url":"https://api.github.com/users/scopatz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scopatz/subscriptions","organizations_url":"https://api.github.com/users/scopatz/orgs","repos_url":"https://api.github.com/users/scopatz/repos","events_url":"https://api.github.com/users/scopatz/events{/privacy}","received_events_url":"https://api.github.com/users/scopatz/received_events","type":"User","site_admin":false},"created_at":"2015-09-15T21:35:30Z","updated_at":"2015-09-15T21:35:30Z","body":"@wesm I think the PyTables comments miss the mark a little. We have done a ton of work and a lot of innovative things with PyTables since it went from being Francesc's baby to a community code.  "}}