{"action":"created","issue":{"url":"https://api.github.com/repos/NSLS-II/wishlist/issues/61","labels_url":"https://api.github.com/repos/NSLS-II/wishlist/issues/61/labels{/name}","comments_url":"https://api.github.com/repos/NSLS-II/wishlist/issues/61/comments","events_url":"https://api.github.com/repos/NSLS-II/wishlist/issues/61/events","html_url":"https://github.com/NSLS-II/wishlist/issues/61","id":108754540,"number":61,"title":"Notice of new scan and access to dataset during scan ","user":{"login":"xiaojinghuang","id":14880298,"avatar_url":"https://avatars.githubusercontent.com/u/14880298?v=3","gravatar_id":"","url":"https://api.github.com/users/xiaojinghuang","html_url":"https://github.com/xiaojinghuang","followers_url":"https://api.github.com/users/xiaojinghuang/followers","following_url":"https://api.github.com/users/xiaojinghuang/following{/other_user}","gists_url":"https://api.github.com/users/xiaojinghuang/gists{/gist_id}","starred_url":"https://api.github.com/users/xiaojinghuang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xiaojinghuang/subscriptions","organizations_url":"https://api.github.com/users/xiaojinghuang/orgs","repos_url":"https://api.github.com/users/xiaojinghuang/repos","events_url":"https://api.github.com/users/xiaojinghuang/events{/privacy}","received_events_url":"https://api.github.com/users/xiaojinghuang/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-09-28T21:54:33Z","updated_at":"2015-10-01T15:36:13Z","closed_at":null,"body":"When a new scan starts, it will be great to be reflected with a status value (a PV?).\r\n\r\nOn-line data analysis tools need access to the running dataset during scan. Could you please provide a tool that can automatically check and read newly saved files?"},"comment":{"url":"https://api.github.com/repos/NSLS-II/wishlist/issues/comments/144764715","html_url":"https://github.com/NSLS-II/wishlist/issues/61#issuecomment-144764715","issue_url":"https://api.github.com/repos/NSLS-II/wishlist/issues/61","id":144764715,"user":{"login":"cowanml","id":5480419,"avatar_url":"https://avatars.githubusercontent.com/u/5480419?v=3","gravatar_id":"","url":"https://api.github.com/users/cowanml","html_url":"https://github.com/cowanml","followers_url":"https://api.github.com/users/cowanml/followers","following_url":"https://api.github.com/users/cowanml/following{/other_user}","gists_url":"https://api.github.com/users/cowanml/gists{/gist_id}","starred_url":"https://api.github.com/users/cowanml/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cowanml/subscriptions","organizations_url":"https://api.github.com/users/cowanml/orgs","repos_url":"https://api.github.com/users/cowanml/repos","events_url":"https://api.github.com/users/cowanml/events{/privacy}","received_events_url":"https://api.github.com/users/cowanml/received_events","type":"User","site_admin":false},"created_at":"2015-10-01T15:36:13Z","updated_at":"2015-10-01T15:36:13Z","body":"\nhrmmm... a standardized way of handling streaming...?\n\nCould we handle the eiger streaming interface with the same layer?\nWith control and metadata traveling through the regular daq stack but\n\"live\" data potentially multicast to go straight from detector to stream\nprocessing clients, multiple parallel pipelines in some cases, as well\nas to the regular daq stack?  Or distribute stream processing across a\ncluster?\n\nthis stuff keeps coming up for amx/fmx.\n\n\nOn Mon, 28 Sep 2015, K Lauer wrote:\n> Correct me if I'm wrong, but I don't believe we have a standardized way to push data from runs to an\n> external processing pipeline (i.e., one residing in a separate process).\n> \n> @arkilic today mentioned to me about the possibility for a caching layer /\n> monitor-for-new-scans-and-events directly from mongodb using \"capped collections.\" (Admittedly, I\n> don't know too much of the details required to implement the above, perhaps @arkilic can comment\n> himself.) I think such a layer could support either just notifying of when scans have been written\n> to the db or pushing data into a processing pipeline as @danielballan seems to prefer - with no\n> changes necessary to the collection side.\n"}}