{"action":"created","issue":{"url":"https://api.github.com/repos/modc08/modc08/issues/24","labels_url":"https://api.github.com/repos/modc08/modc08/issues/24/labels{/name}","comments_url":"https://api.github.com/repos/modc08/modc08/issues/24/comments","events_url":"https://api.github.com/repos/modc08/modc08/issues/24/events","html_url":"https://github.com/modc08/modc08/issues/24","id":53461144,"number":24,"title":"Ingest metadata from Excel source file","user":{"login":"franksc","id":8837821,"avatar_url":"https://avatars.githubusercontent.com/u/8837821?v=3","gravatar_id":"","url":"https://api.github.com/users/franksc","html_url":"https://github.com/franksc","followers_url":"https://api.github.com/users/franksc/followers","following_url":"https://api.github.com/users/franksc/following{/other_user}","gists_url":"https://api.github.com/users/franksc/gists{/gist_id}","starred_url":"https://api.github.com/users/franksc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/franksc/subscriptions","organizations_url":"https://api.github.com/users/franksc/orgs","repos_url":"https://api.github.com/users/franksc/repos","events_url":"https://api.github.com/users/franksc/events{/privacy}","received_events_url":"https://api.github.com/users/franksc/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/modc08/modc08/labels/requirement","name":"requirement","color":"5319e7"}],"state":"open","locked":false,"assignee":{"login":"craigpatten","id":1353030,"avatar_url":"https://avatars.githubusercontent.com/u/1353030?v=3","gravatar_id":"","url":"https://api.github.com/users/craigpatten","html_url":"https://github.com/craigpatten","followers_url":"https://api.github.com/users/craigpatten/followers","following_url":"https://api.github.com/users/craigpatten/following{/other_user}","gists_url":"https://api.github.com/users/craigpatten/gists{/gist_id}","starred_url":"https://api.github.com/users/craigpatten/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/craigpatten/subscriptions","organizations_url":"https://api.github.com/users/craigpatten/orgs","repos_url":"https://api.github.com/users/craigpatten/repos","events_url":"https://api.github.com/users/craigpatten/events{/privacy}","received_events_url":"https://api.github.com/users/craigpatten/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/modc08/modc08/milestones/2","labels_url":"https://api.github.com/repos/modc08/modc08/milestones/2/labels","id":889170,"number":2,"title":"Phase Three: Implement storage for datasets","description":"Encompassing establish storage, workflow and MyTardus & object store interaction...","creator":{"login":"millercathy","id":9669002,"avatar_url":"https://avatars.githubusercontent.com/u/9669002?v=3","gravatar_id":"","url":"https://api.github.com/users/millercathy","html_url":"https://github.com/millercathy","followers_url":"https://api.github.com/users/millercathy/followers","following_url":"https://api.github.com/users/millercathy/following{/other_user}","gists_url":"https://api.github.com/users/millercathy/gists{/gist_id}","starred_url":"https://api.github.com/users/millercathy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/millercathy/subscriptions","organizations_url":"https://api.github.com/users/millercathy/orgs","repos_url":"https://api.github.com/users/millercathy/repos","events_url":"https://api.github.com/users/millercathy/events{/privacy}","received_events_url":"https://api.github.com/users/millercathy/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":3,"state":"open","created_at":"2014-12-02T23:53:35Z","updated_at":"2015-01-29T12:36:55Z","due_on":"2015-02-16T13:30:00Z","closed_at":null},"comments":3,"created_at":"2015-01-05T23:23:01Z","updated_at":"2015-02-02T02:05:36Z","closed_at":null,"body":"We need a means to populate the models implemented in #2 with provenance and pipeline metadata relating to ingested datasets. ACAD have indicated their preferred option would be to load a source/template file such as a CSV or Excel doc containing this metadata.\r\n\r\nAn Excel document would seem preferable to a CSV as it allows for multiple sheets/tabs representing different metadata categories (Source, Sample etc) to be included in the one file. An Excel template file based on the MODC08 metadata schema has been prepared and ACAD have used it to provide some sample metadata.\r\n\r\nIdeally this metadata ingest would be hooked into the data batch ingest workflow cf. #22, so when a metadata source file is encountered this would initiate a process to ingest the metadata it contains into the appropriate models/tables.\r\n\r\nExample contents of a sheet for category Source:\r\n\r\nElement name | Description | Value | ...some more columns...\r\n---- | ---- | ---- | ----\r\nid|Source accession id|Source1234|...\r\ndate|Date source was collected|1990-01-01|...\r\nage_cat|Age category of source|adult|...\r\n\r\nWe are only interested in columns 'Element name' and 'Value', with values in the former matching the names of fields in the Source model.\r\n\r\nIt is quite possible a source file will have multiple sheets of the same type (e.g. Source1, Source2 or Source, Source1), so this will need to be handled."},"comment":{"url":"https://api.github.com/repos/modc08/modc08/issues/comments/72397883","html_url":"https://github.com/modc08/modc08/issues/24#issuecomment-72397883","issue_url":"https://api.github.com/repos/modc08/modc08/issues/24","id":72397883,"user":{"login":"franksc","id":8837821,"avatar_url":"https://avatars.githubusercontent.com/u/8837821?v=3","gravatar_id":"","url":"https://api.github.com/users/franksc","html_url":"https://github.com/franksc","followers_url":"https://api.github.com/users/franksc/followers","following_url":"https://api.github.com/users/franksc/following{/other_user}","gists_url":"https://api.github.com/users/franksc/gists{/gist_id}","starred_url":"https://api.github.com/users/franksc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/franksc/subscriptions","organizations_url":"https://api.github.com/users/franksc/orgs","repos_url":"https://api.github.com/users/franksc/repos","events_url":"https://api.github.com/users/franksc/events{/privacy}","received_events_url":"https://api.github.com/users/franksc/received_events","type":"User","site_admin":false},"created_at":"2015-02-02T02:05:36Z","updated_at":"2015-02-02T02:05:36Z","body":"The state of play:\r\n\r\n- We have (cf. #2) models/tables for ACAD provenance and pipeline metadata records (Source, Sample, Extract etc), integrated into MyTardis and ready to receive data.\r\n\r\n- We have an Excel template that ACAD will be using to record metadata associated with their datasets, with multiple sheets/tabs containing different record types (Source, Sample etc).\r\n\r\n    - There is no fixed set or pattern to what metadata record types will appear in the file, e.g. a Source relating to the dataset could be included, or equally no Source could appear if it has already been loaded as part of a previously uploaded dataset. The only assumption is that each dataset's ACAD-model metadata will include exactly one Analysis record, which has a one-one relationship with a MyTardis dataset.\r\n\r\n- We need functionality for ACAD to drop an Excel template file into each dataset-to-be-uploaded directory and have this metadata be read from the Excel file and loaded into the appropriate ACAD models/tables in MyTardis as part of the batch ingest process (cf. #22). This includes recording the relationship between the dataset's Analysis metadata record and the MyTardis dataset created as part of the ingest process.\r\n\r\n- 283ef25b2d546739f5d1a4314c6111b51f0429e9 implements reading each sheet in the Excel template file and returning a JSON representation suitable for passing to Django's serializers.deserialize() – this should take care of loading ACAD-model metadata into the appropriate tables."}}