{"action":"created","issue":{"url":"https://api.github.com/repos/emhart/10-simple-rules-data-storage/issues/48","labels_url":"https://api.github.com/repos/emhart/10-simple-rules-data-storage/issues/48/labels{/name}","comments_url":"https://api.github.com/repos/emhart/10-simple-rules-data-storage/issues/48/comments","events_url":"https://api.github.com/repos/emhart/10-simple-rules-data-storage/issues/48/events","html_url":"https://github.com/emhart/10-simple-rules-data-storage/pull/48","id":69678654,"number":48,"title":"Draft for Rule 10","user":{"login":"fmichonneau","id":5502922,"avatar_url":"https://avatars.githubusercontent.com/u/5502922?v=3","gravatar_id":"","url":"https://api.github.com/users/fmichonneau","html_url":"https://github.com/fmichonneau","followers_url":"https://api.github.com/users/fmichonneau/followers","following_url":"https://api.github.com/users/fmichonneau/following{/other_user}","gists_url":"https://api.github.com/users/fmichonneau/gists{/gist_id}","starred_url":"https://api.github.com/users/fmichonneau/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fmichonneau/subscriptions","organizations_url":"https://api.github.com/users/fmichonneau/orgs","repos_url":"https://api.github.com/users/fmichonneau/repos","events_url":"https://api.github.com/users/fmichonneau/events{/privacy}","received_events_url":"https://api.github.com/users/fmichonneau/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-04-20T20:58:03Z","updated_at":"2015-04-23T02:46:14Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/emhart/10-simple-rules-data-storage/pulls/48","html_url":"https://github.com/emhart/10-simple-rules-data-storage/pull/48","diff_url":"https://github.com/emhart/10-simple-rules-data-storage/pull/48.diff","patch_url":"https://github.com/emhart/10-simple-rules-data-storage/pull/48.patch"},"body":"Here is an initial draft for Rule 10.\r\n\r\nThere might be material in it that belong to other rules. Let me know if you think it's the case.\r\n\r\n@naupaka and @emhart let me know if I captured all the concepts you were hoping this rule would cover.\r\n\r\nNote that I didn't commit the manuscript here as the compilation fails on my system (probably issue with version of pandoc) and I haven't had a chance to investigate.\r\n\r\nCheers!"},"comment":{"url":"https://api.github.com/repos/emhart/10-simple-rules-data-storage/issues/comments/95399281","html_url":"https://github.com/emhart/10-simple-rules-data-storage/pull/48#issuecomment-95399281","issue_url":"https://api.github.com/repos/emhart/10-simple-rules-data-storage/issues/48","id":95399281,"user":{"login":"dlebauer","id":464871,"avatar_url":"https://avatars.githubusercontent.com/u/464871?v=3","gravatar_id":"","url":"https://api.github.com/users/dlebauer","html_url":"https://github.com/dlebauer","followers_url":"https://api.github.com/users/dlebauer/followers","following_url":"https://api.github.com/users/dlebauer/following{/other_user}","gists_url":"https://api.github.com/users/dlebauer/gists{/gist_id}","starred_url":"https://api.github.com/users/dlebauer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dlebauer/subscriptions","organizations_url":"https://api.github.com/users/dlebauer/orgs","repos_url":"https://api.github.com/users/dlebauer/repos","events_url":"https://api.github.com/users/dlebauer/events{/privacy}","received_events_url":"https://api.github.com/users/dlebauer/received_events","type":"User","site_admin":false},"created_at":"2015-04-23T02:46:14Z","updated_at":"2015-04-23T02:46:14Z","body":"Perhaps a key point is \"for smaller datasets, use flat files because they are easy and accessible\". I propose a definition and methods for handling bigger data issues in rule 9 (#39)\r\n\r\nAre databases or NetCDF useful for small datasets? I am not sure, except where the small dataset is a subset of a larger one that is already stored in NetCDF or a database.\r\n\r\nTo clarify a few points made by @emhart, though:\r\n\r\n* as of v4, netCDF is a front-end to HDF5, ie. in many ways they are the same, and can use many of the same tools (e.g. NCO) for computing 'on disk' and across similar variables in the data heirarchy (e.g. from different members of a stochastic simulation or model ensemble).\r\n* In principle, i am not sure if there is a difference between what these can do and a RDBMS, except that HDF5/NetCDF4 can be optimized for raster / spatial data\r\n* both of these have arbitrary levels of hierarchy, and both work well with gridded data\r\n* the number of dimensions can be high (climate data provided in this format generally have latitude, longitude, height and time dimensions for each of many variables in a raster)\r\n\r\nIn this example, \r\n\r\n\r\n```\r\nRegion1/\r\n|- SiteA/\r\n|- Tree1, 4.5\r\n|- Tree2, 3.9\r\n|- Tree3, 2.6\r\n```\r\n\r\nThis is the same representation (right?) as can be done in HDF5 or CSV as shown above or in a three-table relational database (tables `regions, sites, trees` with each region having >= 0 sites, and each site having  >= 0 trees).\r\n\r\nAn alternative representation might be spatially explicit: a recursive PostGIS table in which the geometry column can represent regions and sites as vector polygons and trees as points, supporting queries to locate all trees within some site. Not sure how much of this is useful. \r\n\r\nAgain, I suggest that for data that fits in memory, such models are less essential than having tidy, i.e. easily useable, data. "}}