{"action":"created","issue":{"url":"https://api.github.com/repos/thunder-project/thunder/issues/237","labels_url":"https://api.github.com/repos/thunder-project/thunder/issues/237/labels{/name}","comments_url":"https://api.github.com/repos/thunder-project/thunder/issues/237/comments","events_url":"https://api.github.com/repos/thunder-project/thunder/issues/237/events","html_url":"https://github.com/thunder-project/thunder/pull/237","id":117023840,"number":237,"title":"Refactor (WIP)","user":{"login":"freeman-lab","id":3387500,"avatar_url":"https://avatars.githubusercontent.com/u/3387500?v=3","gravatar_id":"","url":"https://api.github.com/users/freeman-lab","html_url":"https://github.com/freeman-lab","followers_url":"https://api.github.com/users/freeman-lab/followers","following_url":"https://api.github.com/users/freeman-lab/following{/other_user}","gists_url":"https://api.github.com/users/freeman-lab/gists{/gist_id}","starred_url":"https://api.github.com/users/freeman-lab/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/freeman-lab/subscriptions","organizations_url":"https://api.github.com/users/freeman-lab/orgs","repos_url":"https://api.github.com/users/freeman-lab/repos","events_url":"https://api.github.com/users/freeman-lab/events{/privacy}","received_events_url":"https://api.github.com/users/freeman-lab/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2015-11-15T22:32:23Z","updated_at":"2015-11-27T20:51:21Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/thunder-project/thunder/pulls/237","html_url":"https://github.com/thunder-project/thunder/pull/237","diff_url":"https://github.com/thunder-project/thunder/pull/237.diff","patch_url":"https://github.com/thunder-project/thunder/pull/237.patch"},"body":"This is a major refactoring of Thunder, and will the basis of an upcoming new release. Work is ongoing; this PR shares the current progress.\r\n\r\nThere are three primary goals, based on community experience and feedback, and consideration of the current ecosystem:\r\n\r\n1. Loosen the dependency on Spark. This is a big one. Many superficial issues, including installation issues, complexity for new users and contributors, etc are due to Thunder's hard dependence on Spark. We will definitely continue to support Spark, we also want to enable work seamlessly across local and distributed environments, and against a variety of execution engines, including Spark but also new libraries like Dask. This PR begins that effort through some fundemental but neccessary refactoring.\r\n\r\n2. Modularize the components. Thunder has started absorbing a wide variety of algorithms / analyses, especially with recent additions to image registration and spatiotemporal source extraction. These components are at different levels of maturity and specificity, and are better off as pluggable, composable pieces living in separate repos.\r\n\r\n3. Modernize the codebase, and make more friendly to the Python ecosystem, in paticular by ensuring Python 3 compatibility, using py.test for unit tests, and using snake case.\r\n\r\n\r\nrefactoring\r\n\r\n- [x] develop global context manager for backend\r\n- [x] refactor data reading / writing\r\n- [x] update reading / writing tests\r\n- [x] remove executables\r\n- [x] remove standalone scripts\r\n- [x] use S3 for external data\r\n- [x] use py.test for unit tests\r\n- [ ] update documentation\r\n- [ ] make python 3 compatible\r\n- [ ] use snakecase (might be controversial!)\r\n\r\nnew packages (inside `thunder-project`)\r\n\r\n- [ ] rime - source extraction\r\n- [ ] sleet - image registration\r\n- [ ] thundercloud - manage cluster on ec2\r\n\r\nnew packages (external)\r\n\r\n- [x] [station](https://github.com/freeman-lab/station) - context manager for distributed backends\r\n- [x] [checkist](https://github.com/freeman-lab/checkist) - minimal argument checking\r\n- [x] [showit](https://github.com/freeman-lab/showit) - simple display of images and tiled images\r\n- [ ] serdeme - custom class serialization/deserialization\r\n"},"comment":{"url":"https://api.github.com/repos/thunder-project/thunder/issues/comments/160202125","html_url":"https://github.com/thunder-project/thunder/pull/237#issuecomment-160202125","issue_url":"https://api.github.com/repos/thunder-project/thunder/issues/237","id":160202125,"user":{"login":"freeman-lab","id":3387500,"avatar_url":"https://avatars.githubusercontent.com/u/3387500?v=3","gravatar_id":"","url":"https://api.github.com/users/freeman-lab","html_url":"https://github.com/freeman-lab","followers_url":"https://api.github.com/users/freeman-lab/followers","following_url":"https://api.github.com/users/freeman-lab/following{/other_user}","gists_url":"https://api.github.com/users/freeman-lab/gists{/gist_id}","starred_url":"https://api.github.com/users/freeman-lab/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/freeman-lab/subscriptions","organizations_url":"https://api.github.com/users/freeman-lab/orgs","repos_url":"https://api.github.com/users/freeman-lab/repos","events_url":"https://api.github.com/users/freeman-lab/events{/privacy}","received_events_url":"https://api.github.com/users/freeman-lab/received_events","type":"User","site_admin":false},"created_at":"2015-11-27T20:51:21Z","updated_at":"2015-11-27T20:51:21Z","body":"Thanks @d-v-b , I was counting on your pedantry :+1: \r\n\r\n- re: RGB, we could for now just add a `rgb=True` flag that treats 3D images as RGB\r\n\r\n- re: bit-depth, i like the `rescale=True` idea, we can put off deciding the default but I'm fine with making it `false` for now (which puts the oness on the user to do the right thing). The only thing will be the `png` handling will be different from `tif`, because as far as I can tell the most sane, dependency-free `png` saving method is `scipy.misc.imsave`, which itself does scaling, so there's no way to skip it.\r\n\r\n\r\n"}}