{"action":"created","issue":{"url":"https://api.github.com/repos/SciLifeLab/TACA/issues/102","labels_url":"https://api.github.com/repos/SciLifeLab/TACA/issues/102/labels{/name}","comments_url":"https://api.github.com/repos/SciLifeLab/TACA/issues/102/comments","events_url":"https://api.github.com/repos/SciLifeLab/TACA/issues/102/events","html_url":"https://github.com/SciLifeLab/TACA/issues/102","id":71563900,"number":102,"title":"Database for storing run status","user":{"login":"guillermo-carrasco","id":1574147,"avatar_url":"https://avatars.githubusercontent.com/u/1574147?v=3","gravatar_id":"","url":"https://api.github.com/users/guillermo-carrasco","html_url":"https://github.com/guillermo-carrasco","followers_url":"https://api.github.com/users/guillermo-carrasco/followers","following_url":"https://api.github.com/users/guillermo-carrasco/following{/other_user}","gists_url":"https://api.github.com/users/guillermo-carrasco/gists{/gist_id}","starred_url":"https://api.github.com/users/guillermo-carrasco/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guillermo-carrasco/subscriptions","organizations_url":"https://api.github.com/users/guillermo-carrasco/orgs","repos_url":"https://api.github.com/users/guillermo-carrasco/repos","events_url":"https://api.github.com/users/guillermo-carrasco/events{/privacy}","received_events_url":"https://api.github.com/users/guillermo-carrasco/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/SciLifeLab/TACA/labels/enhancement","name":"enhancement","color":"84b6eb"},{"url":"https://api.github.com/repos/SciLifeLab/TACA/labels/help%20wanted","name":"help wanted","color":"159818"}],"state":"open","locked":false,"assignee":{"login":"guillermo-carrasco","id":1574147,"avatar_url":"https://avatars.githubusercontent.com/u/1574147?v=3","gravatar_id":"","url":"https://api.github.com/users/guillermo-carrasco","html_url":"https://github.com/guillermo-carrasco","followers_url":"https://api.github.com/users/guillermo-carrasco/followers","following_url":"https://api.github.com/users/guillermo-carrasco/following{/other_user}","gists_url":"https://api.github.com/users/guillermo-carrasco/gists{/gist_id}","starred_url":"https://api.github.com/users/guillermo-carrasco/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guillermo-carrasco/subscriptions","organizations_url":"https://api.github.com/users/guillermo-carrasco/orgs","repos_url":"https://api.github.com/users/guillermo-carrasco/repos","events_url":"https://api.github.com/users/guillermo-carrasco/events{/privacy}","received_events_url":"https://api.github.com/users/guillermo-carrasco/received_events","type":"User","site_admin":false},"milestone":null,"comments":4,"created_at":"2015-04-28T11:35:06Z","updated_at":"2015-05-05T06:57:50Z","closed_at":null,"body":"Would be nice to have a small **and simple** database just to save the status of the runs, i.e `SEQUENCING`, `ARCHIVING`, `ARHCIVED`, etc.\r\n\r\nThe idea is to implement it in such a way that the database backend should be abstract/plugable. Basically define an API (in the `Run` class probably) with 2 main functions: `get_run_status()` and `set_run_status(status)`\r\n\r\n@vezzi you can use this issue to discuss implementation and/or define status."},"comment":{"url":"https://api.github.com/repos/SciLifeLab/TACA/issues/comments/98974855","html_url":"https://github.com/SciLifeLab/TACA/issues/102#issuecomment-98974855","issue_url":"https://api.github.com/repos/SciLifeLab/TACA/issues/102","id":98974855,"user":{"login":"guillermo-carrasco","id":1574147,"avatar_url":"https://avatars.githubusercontent.com/u/1574147?v=3","gravatar_id":"","url":"https://api.github.com/users/guillermo-carrasco","html_url":"https://github.com/guillermo-carrasco","followers_url":"https://api.github.com/users/guillermo-carrasco/followers","following_url":"https://api.github.com/users/guillermo-carrasco/following{/other_user}","gists_url":"https://api.github.com/users/guillermo-carrasco/gists{/gist_id}","starred_url":"https://api.github.com/users/guillermo-carrasco/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guillermo-carrasco/subscriptions","organizations_url":"https://api.github.com/users/guillermo-carrasco/orgs","repos_url":"https://api.github.com/users/guillermo-carrasco/repos","events_url":"https://api.github.com/users/guillermo-carrasco/events{/privacy}","received_events_url":"https://api.github.com/users/guillermo-carrasco/received_events","type":"User","site_admin":false},"created_at":"2015-05-05T06:57:50Z","updated_at":"2015-05-05T06:57:50Z","body":"Thanks for the comments @robinandeer !\r\n\r\n> I guess you can handle the pluggability also through entry points instead of a custom solution\r\n\r\nI guess its a trade off between CLI cleanness and abstraction, isn't it? For example, if we decide to do this through entry points we still have to implement the different backends, its just that we _also_ have to implement the CLI part, so would be something like `taca storage cleanup --backend couchdb` or similar. Instead deducing the backend from the YAML file frees us from writing that `--backend couchdb`. \r\n\r\nYou may argue that we are doing precisely that for archiving, i.e `taca storage archive --backend swestore`, however I think that its a different case, is not a technology backend that you are choosing, but an end \"physical\" place were to place your data, its good to be explicit in this case.\r\n\r\n> We are thinking about something similar and one problem has been the different levels we are working on; Run/Flowcell, Lane, Sample, Family, Project... This could get complicated fast and perhaps change over time if for example you back up VCFs instead of fastq-files\r\n\r\nI don't see how this could affect the idea behind this issue. What we want (at least by now) its a very simple status, i.e `DEMULTIPLEXING`, `ARCHIVING`, etc. Does it matter the level you're working at?\r\n\r\n> If you want to get real fancy, you could default to a SQLite database/YAML file if a proper backend isn't setup\r\n\r\nWe thought on that as well, the problem is that we _don't_ want a local database because what we want is something that helps the NASs and the processing machines to communicate. For example, remove a run in `preproc1` _only_ if it has been archived in swestore (which is done in the NASs).\r\n\r\n> Is it possible to create this sub-database so general that both NGI and Clinical could make use of it? I'll try to think about what we would need to store to make it work for us.\r\n\r\nThat was the idea, so good you're in for that ^^! We could define a simple API on the `db` package, that would then instantiate the correct submodule depending on the backend. \r\n\r\nFirst set of methods proposed for the API: \r\n\r\n* `get_run_status(run)`: Get the status of a particular run\r\n* `set_run_status(status)`: Set the status of a particular run\r\n* `get_processing_runs()`: Get a list of all currently processing (demultiplexing) runs\r\n* `get_archiving_runs()`: Get a list of all currently archiving runs\r\n* `get_archived_run()`: Get a list of all archived runs\r\n\r\nThanks again! Let's keep discussing this ^^"}}