{"action":"created","issue":{"url":"https://api.github.com/repos/thunder-project/thunder/issues/139","labels_url":"https://api.github.com/repos/thunder-project/thunder/issues/139/labels{/name}","comments_url":"https://api.github.com/repos/thunder-project/thunder/issues/139/comments","events_url":"https://api.github.com/repos/thunder-project/thunder/issues/139/events","html_url":"https://github.com/thunder-project/thunder/pull/139","id":63635245,"number":139,"title":"(WIP) Source extraction","user":{"login":"freeman-lab","id":3387500,"avatar_url":"https://avatars.githubusercontent.com/u/3387500?v=3","gravatar_id":"","url":"https://api.github.com/users/freeman-lab","html_url":"https://github.com/freeman-lab","followers_url":"https://api.github.com/users/freeman-lab/followers","following_url":"https://api.github.com/users/freeman-lab/following{/other_user}","gists_url":"https://api.github.com/users/freeman-lab/gists{/gist_id}","starred_url":"https://api.github.com/users/freeman-lab/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/freeman-lab/subscriptions","organizations_url":"https://api.github.com/users/freeman-lab/orgs","repos_url":"https://api.github.com/users/freeman-lab/repos","events_url":"https://api.github.com/users/freeman-lab/events{/privacy}","received_events_url":"https://api.github.com/users/freeman-lab/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/thunder-project/thunder/labels/in%20progress","name":"in progress","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-03-23T06:10:19Z","updated_at":"2015-04-10T07:12:59Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/thunder-project/thunder/pulls/139","html_url":"https://github.com/thunder-project/thunder/pull/139","diff_url":"https://github.com/thunder-project/thunder/pull/139.diff","patch_url":"https://github.com/thunder-project/thunder/pull/139.patch"},"body":"This PR introduces the \"source extraction\" API for Thunder. Source extraction describes the problem of identifying spatially-structured and temporally-coherent sources in spatio-temopral data (for example, neurons in fluorescence imaging data).\r\n\r\nAn [iPython notebook](http://nbviewer.ipython.org/github/thunder-project/thunder-example-notebooks/blob/master/source-extraction-demo.ipynb) introduces the proposed API and walks through working toy examples.\r\n\r\nThe key design goals were:\r\n- Providing a general, modular framework or \"harness\" that supports plugging in a variety of existing algorithmic approaches while exposing a simple, high-level interface\r\n- Maintaining scalability and performance by distributing core operations where possible using Spark\r\n\r\nThe design is based on extensive discussions with @pkaifosh, @poolio, @broxtronix  , @logang, @speron, @sofroniewn, feedback welcome from all of them (and from others! ccing @jakirkham @tomsains )\r\n\r\nAlthough the API is new, it uses lots of core functionality we have been building into Thunder (for example, the ability to express `Images` as distributed `Blocks`, the ability to extract signals from arbitrary spatial regions in either `Images` or `Series`, and new support for JSON serialization).\r\n\r\nSummary of key additions:\r\n- Source and SourceModel classes for creating, manipulating, saving/loading, and rendering sources\r\n- Block based source extraction methods, including algorithms and mergers\r\n- Feature based source extraction methods, including algorithms and creators\r\n- Two very simple example implementations (a feature method, `localmax`, and a block method, `nmf`).\r\n\r\nNew dependencies:\r\n- scikit-image (image morphology processing)\r\n- zope.cachedescriptors (lazy property handling)\r\n\r\nOutstanding issues:\r\n- The trick used to expose keyword arguments at the component level (e.g. for algorithms and mergers) to the top-level factory method does not trigger ordinary keyword argument validation. Open to alternatives, so long as we can still expose a top-level factory method, which has great benefit for ease-of-use.\r\n- The `BasicBlockMerger` is extremely minimal and should be enhanced.\r\n- More algorithm-specific unit tests.\r\n- More geometric operators and comparators for sources (expand, dilate, overlap).\r\n- At least one or two more example methods (let's add one that imports an algorithm from SIMA?)\r\n"},"comment":{"url":"https://api.github.com/repos/thunder-project/thunder/issues/comments/91460292","html_url":"https://github.com/thunder-project/thunder/pull/139#issuecomment-91460292","issue_url":"https://api.github.com/repos/thunder-project/thunder/issues/139","id":91460292,"user":{"login":"freeman-lab","id":3387500,"avatar_url":"https://avatars.githubusercontent.com/u/3387500?v=3","gravatar_id":"","url":"https://api.github.com/users/freeman-lab","html_url":"https://github.com/freeman-lab","followers_url":"https://api.github.com/users/freeman-lab/followers","following_url":"https://api.github.com/users/freeman-lab/following{/other_user}","gists_url":"https://api.github.com/users/freeman-lab/gists{/gist_id}","starred_url":"https://api.github.com/users/freeman-lab/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/freeman-lab/subscriptions","organizations_url":"https://api.github.com/users/freeman-lab/orgs","repos_url":"https://api.github.com/users/freeman-lab/repos","events_url":"https://api.github.com/users/freeman-lab/events{/privacy}","received_events_url":"https://api.github.com/users/freeman-lab/received_events","type":"User","site_admin":false},"created_at":"2015-04-10T07:12:58Z","updated_at":"2015-04-10T07:12:58Z","body":"Thanks again for the review, @pkaifosh I added support for sequence of `Cleaners`, and more generally refactored the cleaning to be a little more generic (as it applies to all of these algorithms) and flexible (as you suggested). Along with several other enhancements and fixes.\r\n\r\nI am going to merge this version in, but will create a few issues with simple extensions and enhancements that we should explore next."}}