{"action":"created","issue":{"url":"https://api.github.com/repos/codeforscience/data-packaging/issues/1","labels_url":"https://api.github.com/repos/codeforscience/data-packaging/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/codeforscience/data-packaging/issues/1/comments","events_url":"https://api.github.com/repos/codeforscience/data-packaging/issues/1/events","html_url":"https://github.com/codeforscience/data-packaging/issues/1","id":119845185,"number":1,"title":"Share your domain specific format here","user":{"login":"maxogden","id":39759,"avatar_url":"https://avatars.githubusercontent.com/u/39759?v=3","gravatar_id":"","url":"https://api.github.com/users/maxogden","html_url":"https://github.com/maxogden","followers_url":"https://api.github.com/users/maxogden/followers","following_url":"https://api.github.com/users/maxogden/following{/other_user}","gists_url":"https://api.github.com/users/maxogden/gists{/gist_id}","starred_url":"https://api.github.com/users/maxogden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maxogden/subscriptions","organizations_url":"https://api.github.com/users/maxogden/orgs","repos_url":"https://api.github.com/users/maxogden/repos","events_url":"https://api.github.com/users/maxogden/events{/privacy}","received_events_url":"https://api.github.com/users/maxogden/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-12-02T01:13:47Z","updated_at":"2015-12-04T01:24:07Z","closed_at":null,"body":"Inspired by an unconference session @olgabot convened at CodeNeuro 2015, use this issue to share a data format from your scientific discipline, the goal being that people from outside your domain expertise area can learn about what works and what doesn't.\r\n\r\nFor example, at the unconf session we learned about 4 domain specific formats from different scientific disciplines, here are some rough notes:\r\n\r\n1. @shoyer presented on climate data in [`xray`](https://github.com/shoyer) using [NetCDF](http://www.unidata.ucar.edu/software/thredds/current/netcdf-java/CDM/)\r\n\t- Store data compressed as a small int with a scaling factor\r\n\t- Share dimensions between different matrices\r\n\t- How to share the data? Some centralized datasets\r\n2. @danlurie presented on fMRI data at [Brain Imaging Data Structures](http://bids.neuroimaging.io/)\r\n\t- fMRI data of voxels (volume pixels)\r\n\t- Standardized data format - like a jpeg\r\n\t- Agreed upon headers\r\n\t- DICOM - every volume is a separate image\r\n\t- \"volume\" = one snapshot in time\r\n\t- Coming up with a standard of how subjects and time is named\r\n3. @sofroniewn presented on neuroimaging microscopy data\r\n        - jupyter notebooks here: [`tactile-coding`](https://github.com/sofroniewn/tactile-coding)\r\n\t- Data coming off the microscope every second! (24 MB/s)\r\n\t\t- ~86GB/hour!\r\n\t- No clear standard. Ad-hoc JSON files of metadata\r\n\t- [Binder](http://app.mybinder.org/3165413014/tree) of several Jupyter notebooks that describe the code and data used with his paper\r\n4. @maxogden presented on the new ASDF format for astronomy datasets\r\n\t- Replaces FITS\r\n\t- Raw image of the sky, plus a \"catalog\" which is the interpreted data\r\n\t- Requirements:\r\n\t\t- Hierarchical representation in the same file as the image data\r\n\t\t- Open it in a program and get the metadata AND the raw data\r\n\t- [ASDF paper](http://www.sciencedirect.com/science/article/pii/S2213133715000645)\r\n\t\t- Really good background research (covers NETCDF, csv, etc)\r\n\t\t- Describe their data requirements\r\n\t\t- Use [YAML](http://yaml.org/) as their human readable format\r\n\t\t- [\"Exploded form\"](http://asdf-standard.readthedocs.org/en/latest/file_layout.html#exploded-form)\r\n\t\t\t- Allows random access of large files\r\n\t\t- [Source of the paper](https://github.com/spacetelescope/asdf-standard)\r\n\t\t\t- [Issue Tracker](https://github.com/spacetelescope/asdf-standard/issues)\r\n\t\t- Still 0.1.0\r\n\t- [Python implementation of ASDF](https://github.com/spacetelescope/pyasdf)\r\n\r\nIf you want to share another one, leave an introduction and summary of pros/cons in a comment below, it can be as long or as short as you prefer. Bonus points if you link to some example raw data and some code that processes it."},"comment":{"url":"https://api.github.com/repos/codeforscience/data-packaging/issues/comments/161844858","html_url":"https://github.com/codeforscience/data-packaging/issues/1#issuecomment-161844858","issue_url":"https://api.github.com/repos/codeforscience/data-packaging/issues/1","id":161844858,"user":{"login":"kgryte","id":2643044,"avatar_url":"https://avatars.githubusercontent.com/u/2643044?v=3","gravatar_id":"","url":"https://api.github.com/users/kgryte","html_url":"https://github.com/kgryte","followers_url":"https://api.github.com/users/kgryte/followers","following_url":"https://api.github.com/users/kgryte/following{/other_user}","gists_url":"https://api.github.com/users/kgryte/gists{/gist_id}","starred_url":"https://api.github.com/users/kgryte/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kgryte/subscriptions","organizations_url":"https://api.github.com/users/kgryte/orgs","repos_url":"https://api.github.com/users/kgryte/repos","events_url":"https://api.github.com/users/kgryte/events{/privacy}","received_events_url":"https://api.github.com/users/kgryte/received_events","type":"User","site_admin":false},"created_at":"2015-12-04T01:24:07Z","updated_at":"2015-12-04T01:24:07Z","body":"During my time as a research scientist in single-module biophysics, I encountered the following:\r\n* University of Michigan\r\n  - primarily used MATLAB for image acquisition and analysis\r\n  - files were stored in MATLAB's `*.mat` files\r\n  - data stored on image acquisition machines, personal laptops, and external drives\r\n* University of Oxford\r\n  - used LabView for image acquisition and photon detection\r\n  - used C/C++, Cuda, and MATLAB for analysis\r\n  - raw image data collected from EMCCD cameras was stored as movies in the FITS file format (with the possibility to store each frame as TIFF files)\r\n  - raw photon data was persisted in a domain-specific binary format `*.sm` (single-molecule)\r\n  - analyzed data (single-molecule time series) was stored in MATLAB's `*.mat` files\r\n  - data was stored on image acquisition machines, personal laptops, external drives, and also pushed to centralized terastations/departmental file servers for archival and centralization\r\n  - centralized storage solutions were routinely filled due to the volume of data generated; to free-up space, either more tera stations were bought, data was culled (only analyzed data retained), or archived onto external hard drives and stored in the office\r\n\r\nMost of the field relies on LabView, MATLAB, Origin, and, to some extent, IDL. And thus, formats are often geared toward those software solutions. Recent efforts have been made to move away from `*.sm` to HDF5, but this transition will take considerable time due to legacy software which is not going to be updated anytime soon (software which still requires running on Windows XP).\r\n\r\nToward the end of my stint at Oxford, I moved a considerable amount of the analyzed data to JSON (wrote JSON serialization scripts in MATLAB) and stored on Github.\r\n\r\nThe reality for much of the discipline (and others) is that\r\n\r\n1. formats are either tied to proprietary software, legacy systems, or both\r\n2. data centralization is rare\r\n3. most researchers lack basic skills and/or awareness to think otherwise (e.g., version control is an alien concept outside of tracking changes in Word)"}}