{"action":"created","issue":{"url":"https://api.github.com/repos/choderalab/openpathsampling/issues/300","labels_url":"https://api.github.com/repos/choderalab/openpathsampling/issues/300/labels{/name}","comments_url":"https://api.github.com/repos/choderalab/openpathsampling/issues/300/comments","events_url":"https://api.github.com/repos/choderalab/openpathsampling/issues/300/events","html_url":"https://github.com/choderalab/openpathsampling/pull/300","id":101920547,"number":300,"title":"Moving General Storage Logic into separate class","user":{"login":"jhprinz","id":5091934,"avatar_url":"https://avatars.githubusercontent.com/u/5091934?v=3","gravatar_id":"","url":"https://api.github.com/users/jhprinz","html_url":"https://github.com/jhprinz","followers_url":"https://api.github.com/users/jhprinz/followers","following_url":"https://api.github.com/users/jhprinz/following{/other_user}","gists_url":"https://api.github.com/users/jhprinz/gists{/gist_id}","starred_url":"https://api.github.com/users/jhprinz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jhprinz/subscriptions","organizations_url":"https://api.github.com/users/jhprinz/orgs","repos_url":"https://api.github.com/users/jhprinz/repos","events_url":"https://api.github.com/users/jhprinz/events{/privacy}","received_events_url":"https://api.github.com/users/jhprinz/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/choderalab/openpathsampling/labels/ref%20docs","name":"ref docs","color":"0052cc"}],"state":"open","locked":false,"assignee":{"login":"dwhswenson","id":8178546,"avatar_url":"https://avatars.githubusercontent.com/u/8178546?v=3","gravatar_id":"","url":"https://api.github.com/users/dwhswenson","html_url":"https://github.com/dwhswenson","followers_url":"https://api.github.com/users/dwhswenson/followers","following_url":"https://api.github.com/users/dwhswenson/following{/other_user}","gists_url":"https://api.github.com/users/dwhswenson/gists{/gist_id}","starred_url":"https://api.github.com/users/dwhswenson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dwhswenson/subscriptions","organizations_url":"https://api.github.com/users/dwhswenson/orgs","repos_url":"https://api.github.com/users/dwhswenson/repos","events_url":"https://api.github.com/users/dwhswenson/events{/privacy}","received_events_url":"https://api.github.com/users/dwhswenson/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/choderalab/openpathsampling/milestones/1","html_url":"https://github.com/choderalab/openpathsampling/milestones/1.0","labels_url":"https://api.github.com/repos/choderalab/openpathsampling/milestones/1/labels","id":882847,"number":1,"title":"1.0","description":"Issues that should be solved before first public release.","creator":{"login":"dwhswenson","id":8178546,"avatar_url":"https://avatars.githubusercontent.com/u/8178546?v=3","gravatar_id":"","url":"https://api.github.com/users/dwhswenson","html_url":"https://github.com/dwhswenson","followers_url":"https://api.github.com/users/dwhswenson/followers","following_url":"https://api.github.com/users/dwhswenson/following{/other_user}","gists_url":"https://api.github.com/users/dwhswenson/gists{/gist_id}","starred_url":"https://api.github.com/users/dwhswenson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dwhswenson/subscriptions","organizations_url":"https://api.github.com/users/dwhswenson/orgs","repos_url":"https://api.github.com/users/dwhswenson/repos","events_url":"https://api.github.com/users/dwhswenson/events{/privacy}","received_events_url":"https://api.github.com/users/dwhswenson/received_events","type":"User","site_admin":false},"open_issues":10,"closed_issues":36,"state":"open","created_at":"2014-11-27T17:27:40Z","updated_at":"2015-10-27T21:54:27Z","due_on":null,"closed_at":null},"comments":76,"created_at":"2015-08-19T15:31:12Z","updated_at":"2015-10-28T17:53:21Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/choderalab/openpathsampling/pulls/300","html_url":"https://github.com/choderalab/openpathsampling/pull/300","diff_url":"https://github.com/choderalab/openpathsampling/pull/300.diff","patch_url":"https://github.com/choderalab/openpathsampling/pull/300.patch"},"body":"I thought it would be useful to separate the JSON Pickling and object storage logic from the actual MD/OPS Application. This makes the code more readable and the base file format can be used for other projects. This is what I came up with so far:\r\n\r\n#### netCDF Subclass `NetCDFPlus`\r\n\r\nI made a wrapper class for the usual netCDF which enhances it with all sorts of nice features, but leaves the basic netCDF API intact. So you can safely switch to nCDF+ without problems.\r\n\r\n1. native support for `simtk.unit`\r\n        You can simply create a variable with units and it is automatically stripped off, stored and reattached if loaded. So writing `storage.vars['my_length'][1] = Quantity(1.0, unit.nanometer)` works. The values are converted to the variable units and stored. This way you always know what the numbers in your file mean.\r\n2. Each variable (only numpy and floats) has a unit property\r\n3. Access the enhanced variables using `storage.vars[name]` and the netCDF implementation using `storage.variables[name]`\r\n4. Support for storage of python objects as we do right now. This requires the objects to be subclassed from `OPSObject` (which we should rename). Then you can create a special object variable using\r\n    `storage.add('my_object_var', ObjectStore(options))` and store and access these objects via `storage.my_object_var[index]` or `storage.my_object.save(obj)`. Note that these requires immutable objects to not mess up the complete storage. The ObjectStore can either be specialized to store certain objects effectively or there is a standard version that uses JSON Pickling.\r\n5. More Powerful variable types:\r\n    1. native python types `bool`, `float`, `int`, `str`\r\n        Besides the native implementation of bumpy arrays I fould it useful to move the type casting into the variable itself. So if you create a variable of type `bool` then assigning `storage.vars['my_bool'][10] = True` works and `storage.vars['my_bool'][10]` returns a bool. Similar with other types\r\n    2. There is a variable type that references objects that can be stored. So just create a variable of type `obj.my_object_var` and store or save like any other variable `storage.vars['obj_var'][idx] = obj`. This will  \r\nstore the object in the netCDF file using the `ObjectStore` and store in `obj_var` the index to find the object. The actual netcdf type of these indices are `numpy.int32`\r\n\r\n\r\nAll of this is totally generic and independent of our usecase. We just add special Stores for some objects like snapshots, trajectories, etc. and pre-initiate the storage with store for all of our storable objects.\r\n\r\nWe might at some point move the extended storage into a separate project."},"comment":{"url":"https://api.github.com/repos/choderalab/openpathsampling/issues/comments/151930693","html_url":"https://github.com/choderalab/openpathsampling/pull/300#issuecomment-151930693","issue_url":"https://api.github.com/repos/choderalab/openpathsampling/issues/300","id":151930693,"user":{"login":"jhprinz","id":5091934,"avatar_url":"https://avatars.githubusercontent.com/u/5091934?v=3","gravatar_id":"","url":"https://api.github.com/users/jhprinz","html_url":"https://github.com/jhprinz","followers_url":"https://api.github.com/users/jhprinz/followers","following_url":"https://api.github.com/users/jhprinz/following{/other_user}","gists_url":"https://api.github.com/users/jhprinz/gists{/gist_id}","starred_url":"https://api.github.com/users/jhprinz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jhprinz/subscriptions","organizations_url":"https://api.github.com/users/jhprinz/orgs","repos_url":"https://api.github.com/users/jhprinz/repos","events_url":"https://api.github.com/users/jhprinz/events{/privacy}","received_events_url":"https://api.github.com/users/jhprinz/received_events","type":"User","site_admin":false},"created_at":"2015-10-28T17:53:21Z","updated_at":"2015-10-28T17:53:21Z","body":"I fixed it now that we have actually `f_` for the function, but it does not look too nice..."}}