{"action":"created","issue":{"url":"https://api.github.com/repos/openworm/movement_validation/issues/129","labels_url":"https://api.github.com/repos/openworm/movement_validation/issues/129/labels{/name}","comments_url":"https://api.github.com/repos/openworm/movement_validation/issues/129/comments","events_url":"https://api.github.com/repos/openworm/movement_validation/issues/129/events","html_url":"https://github.com/openworm/movement_validation/issues/129","id":81222736,"number":129,"title":"JSON vs HDF5","user":{"login":"MichaelCurrie","id":5267078,"avatar_url":"https://avatars.githubusercontent.com/u/5267078?v=3","gravatar_id":"","url":"https://api.github.com/users/MichaelCurrie","html_url":"https://github.com/MichaelCurrie","followers_url":"https://api.github.com/users/MichaelCurrie/followers","following_url":"https://api.github.com/users/MichaelCurrie/following{/other_user}","gists_url":"https://api.github.com/users/MichaelCurrie/gists{/gist_id}","starred_url":"https://api.github.com/users/MichaelCurrie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MichaelCurrie/subscriptions","organizations_url":"https://api.github.com/users/MichaelCurrie/orgs","repos_url":"https://api.github.com/users/MichaelCurrie/repos","events_url":"https://api.github.com/users/MichaelCurrie/events{/privacy}","received_events_url":"https://api.github.com/users/MichaelCurrie/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/openworm/movement_validation/labels/question","name":"question","color":"cc317c"},{"url":"https://api.github.com/repos/openworm/movement_validation/labels/ready","name":"ready","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/openworm/movement_validation/milestones/4","html_url":"https://github.com/openworm/movement_validation/milestones/Implement%20service%20on%20the%20cloud","labels_url":"https://api.github.com/repos/openworm/movement_validation/milestones/4/labels","id":980314,"number":4,"title":"Implement service on the cloud","description":"The movement validation service and the underlying worm videos and data should be hosted in the cloud\r\n\r\nA large corpus of worm videos is hosted in the cloud\r\n\r\nMetadata is stored about the videos, such as what lab they were recorded at and when.  This will be useful when videos of varying quality and type start to be uploaded.  So statistics could be run on multi-worm videos, for instance.\r\n\r\nA web-based interface to support the following use cases:\r\n- Add a film to the database of worm videos\r\n- Add a feature to the list of features that can be calculated  (e.g. Balazs’)\r\n- Run a statistical comparison of a worm type to another worm type on all or some\r\n- View a time-series plot of a feature\r\n- View a worm animation alongside time-series and event features as they occur\r\n- View the raw video overlaid with segmentation\r\n- Download results in various formats (excel, CSV, JPEG for plots, etc)\r\n- Relevant for the rest of OpenWorm: Pass in a video or contour file, and obtain a scalar between 0 and 1 (i.e. the “fitness function”)\r\n\r\nImplementation:\r\n- Data hosted on Amazon S3\r\n- Website and web service hosted on Amazon/EC2\r\n\r\n\r\nWe stand to be able to host the Schafer Lab corpus of worm videos, which is over 1 TB of data.  Most researchers wanting to calculate statistics on a newly developed feature type would prefer to not have to download this database and then perform calculations on 1 TB of video data on their desktop computers.\r\n\r\nInstead, the relational database could have a service running in the background that will asynchronously refresh the feature and statistics tables if a new feature type is entered into the codebase.  Once the feature has been calculated across the entire video corpus, the researcher can perform statistical comparisons on any of the hundreds of worm types available, and quickly see if his or her new code is achieving the aim he or she was hoping for.","creator":{"login":"MichaelCurrie","id":5267078,"avatar_url":"https://avatars.githubusercontent.com/u/5267078?v=3","gravatar_id":"","url":"https://api.github.com/users/MichaelCurrie","html_url":"https://github.com/MichaelCurrie","followers_url":"https://api.github.com/users/MichaelCurrie/followers","following_url":"https://api.github.com/users/MichaelCurrie/following{/other_user}","gists_url":"https://api.github.com/users/MichaelCurrie/gists{/gist_id}","starred_url":"https://api.github.com/users/MichaelCurrie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MichaelCurrie/subscriptions","organizations_url":"https://api.github.com/users/MichaelCurrie/orgs","repos_url":"https://api.github.com/users/MichaelCurrie/repos","events_url":"https://api.github.com/users/MichaelCurrie/events{/privacy}","received_events_url":"https://api.github.com/users/MichaelCurrie/received_events","type":"User","site_admin":false},"open_issues":7,"closed_issues":4,"state":"open","created_at":"2015-02-15T20:12:51Z","updated_at":"2015-05-27T01:48:59Z","due_on":null,"closed_at":null},"comments":1,"created_at":"2015-05-27T01:48:47Z","updated_at":"2015-06-03T19:52:57Z","closed_at":null,"body":"Right now as is demonstrated by [JSON demo.py](https://github.com/openworm/movement_validation/blob/master/examples/JSON%20demo.py), we have the ability to serialize NormalizedWorm and some other classes as JSON flat files.  JSON is format familiar to web developers as it is often exchanged via HTTP between the client and server sides of web apps, but it isn't binary and so it might be very slow given the 50+ megabyte size of the arrays of floats we are looking to serialize.\r\n\r\nHDF5 is the obvious alternative, and it's the one used natively by Matlab 12+ and some of the original Schafer Lab code.\r\n\r\nThis is not a decision that has to be made instantly since we can easily refactor at a later stage, since the serialization code is all encapsulated in a single class, [JSONSerializer](https://github.com/openworm/movement_validation/blob/master/movement_validation/basic_worm.py), but @JimHokanson and I were wondering if anyone had any strong thoughts on the matter."},"comment":{"url":"https://api.github.com/repos/openworm/movement_validation/issues/comments/108595393","html_url":"https://github.com/openworm/movement_validation/issues/129#issuecomment-108595393","issue_url":"https://api.github.com/repos/openworm/movement_validation/issues/129","id":108595393,"user":{"login":"MichaelCurrie","id":5267078,"avatar_url":"https://avatars.githubusercontent.com/u/5267078?v=3","gravatar_id":"","url":"https://api.github.com/users/MichaelCurrie","html_url":"https://github.com/MichaelCurrie","followers_url":"https://api.github.com/users/MichaelCurrie/followers","following_url":"https://api.github.com/users/MichaelCurrie/following{/other_user}","gists_url":"https://api.github.com/users/MichaelCurrie/gists{/gist_id}","starred_url":"https://api.github.com/users/MichaelCurrie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MichaelCurrie/subscriptions","organizations_url":"https://api.github.com/users/MichaelCurrie/orgs","repos_url":"https://api.github.com/users/MichaelCurrie/repos","events_url":"https://api.github.com/users/MichaelCurrie/events{/privacy}","received_events_url":"https://api.github.com/users/MichaelCurrie/received_events","type":"User","site_admin":false},"created_at":"2015-06-03T19:52:57Z","updated_at":"2015-06-03T19:52:57Z","body":"@joebowen @clinzy having some web development experience, maybe you two have thoughts on this?  Thanks!"}}