{"action":"created","issue":{"url":"https://api.github.com/repos/nilmtk/nilm_metadata/issues/16","labels_url":"https://api.github.com/repos/nilmtk/nilm_metadata/issues/16/labels{/name}","comments_url":"https://api.github.com/repos/nilmtk/nilm_metadata/issues/16/comments","events_url":"https://api.github.com/repos/nilmtk/nilm_metadata/issues/16/events","html_url":"https://github.com/nilmtk/nilm_metadata/issues/16","id":54304452,"number":16,"title":"Proposal for a 'simple' NILM Metadata schema","user":{"login":"JackKelly","id":460756,"avatar_url":"https://avatars.githubusercontent.com/u/460756?v=3","gravatar_id":"","url":"https://api.github.com/users/JackKelly","html_url":"https://github.com/JackKelly","followers_url":"https://api.github.com/users/JackKelly/followers","following_url":"https://api.github.com/users/JackKelly/following{/other_user}","gists_url":"https://api.github.com/users/JackKelly/gists{/gist_id}","starred_url":"https://api.github.com/users/JackKelly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JackKelly/subscriptions","organizations_url":"https://api.github.com/users/JackKelly/orgs","repos_url":"https://api.github.com/users/JackKelly/repos","events_url":"https://api.github.com/users/JackKelly/events{/privacy}","received_events_url":"https://api.github.com/users/JackKelly/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-01-14T10:09:32Z","updated_at":"2015-02-06T15:32:29Z","closed_at":null,"body":"NILM Metadata tries to make it possible to capture pretty much any conceivable scenario. But, as more datasets become available, it appears that a large proportion of datasets could be described using a simpler metadata schema. It would be great to discuss the design of a \"Simple NILM Metadata\" schema which could exist along side \"NILM Metadata\". Perhaps CSV is even easier to read than YAML in Matlab, Java etc so it might be nice if we can use CSV. We'd have a check-list to help people decide whether they require the full expressive power of \"NILM Metadata\" or if they can get by with \"Simple NILM Metadata\".\r\n\r\nThe simple schema could also be used for adding metadata to the *output* of disaggregation algorithms (hence [helping to simplify NILMTK disaggregation algorithm implementation](https://github.com/nilmtk/nilmtk/issues/210)); and for describing the training dataset and the responses for any future NILM competition or validation tool (I'm working with a group of MSc students who aim to produce a proof-of-concept NILM validation tool by the end of this term; [here's the project spec](https://docs.google.com/document/d/14UVX5ZxJHqcM1uRnyxenztotwy0RdNoVkYYTCah4YOA/pub).)\r\n\r\nSo, here's an initial proposal, using REDD as an example:\r\n\r\n### building1_labels.csv\r\n\r\nThis looks a little like `labels.dat` in the REDD format except that:\r\n* we use a comma as a separator (which is standard for CSV, and also allows us to use spaces in strings without using quotes)\r\n* we use the file suffix `csv` not `dat` (so that spreadsheet applications know how to open the file)\r\n* we use our NILM Metadata controlled vocabulary for appliance names\r\n* we give an instance number for each appliance\r\n* if there are multiple appliances measured by a meter then separate them by a semicolon e.g. `6,television#1;light#1`\r\n* we could optionally use a third column to specify the `submeter_of` property.  If this is not specified then we assume that anything that isn't a `site meter` is downstream of all site meters, and that all site meters should be summed to get the total whole-house power demand.  Or maybe we should keep \"Simple NILM Metadata\" as simple as possible and say that any non-standard wiring hierarchy simply cannot be expressed using \"Simple NILM Metadata\"?\r\n\r\n```\r\nmeter instance, label\r\n\r\n1,site meter\r\n2,site meter\r\n3,electric oven#1\r\n4,electric oven#1\r\n5,fridge#1\r\n```\r\n\r\n### meter_devices.csv \r\n\r\nWe also need to specify what is measured in each data file.  In NILM Metadata this is done in `meter_devices.yaml`.  In \"Simple NILM Metadata\" this could be done in a `meter_devices.csv` files.  The file would contain three columns; each row would be a `<meter device name>,<key>,<value>` tuple.  e.g.:\r\n\r\n```\r\nmeter device name,key,value\r\n\r\nsite meters,sample period,1\r\nsite meters,measurements,active power;apparent power\r\nsubmeters,sample period,3\r\nsubmeters,measurements,active power\r\nsubmeters,model,eMonitor\r\nsubmeters,manufacturer,Powerhouse Dynamics\r\n```\r\n\r\nThe assumption would be that all meters with the label `site meter` would take attributes from `site meters` and all other meters would take attributes from `submeters`.  If this is not the case (e.g. if there are several types of submeter) then we could do the following (and we'd only have to specify this for the meters for which the default assumption does not hold).\r\n\r\n```\r\nmeter_devices_mapping.csv\r\n```\r\n```\r\nbuilding instance,meter instance,meter device name\r\n\r\n1,1,Current Cost\r\n1,2,SCPM\r\n```\r\n\r\nAny thoughts?  If you use Matlab / Java / Scala / Julia / C++ etc, would you find it easier to load metadata described using CSV files rather than YAML files?  If you maintain a dataset, is there anything in your own dataset that the proposal above cannot express?\r\n"},"comment":{"url":"https://api.github.com/repos/nilmtk/nilm_metadata/issues/comments/73254329","html_url":"https://github.com/nilmtk/nilm_metadata/issues/16#issuecomment-73254329","issue_url":"https://api.github.com/repos/nilmtk/nilm_metadata/issues/16","id":73254329,"user":{"login":"eleijonmarck","id":7727602,"avatar_url":"https://avatars.githubusercontent.com/u/7727602?v=3","gravatar_id":"","url":"https://api.github.com/users/eleijonmarck","html_url":"https://github.com/eleijonmarck","followers_url":"https://api.github.com/users/eleijonmarck/followers","following_url":"https://api.github.com/users/eleijonmarck/following{/other_user}","gists_url":"https://api.github.com/users/eleijonmarck/gists{/gist_id}","starred_url":"https://api.github.com/users/eleijonmarck/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eleijonmarck/subscriptions","organizations_url":"https://api.github.com/users/eleijonmarck/orgs","repos_url":"https://api.github.com/users/eleijonmarck/repos","events_url":"https://api.github.com/users/eleijonmarck/events{/privacy}","received_events_url":"https://api.github.com/users/eleijonmarck/received_events","type":"User","site_admin":false},"created_at":"2015-02-06T15:32:29Z","updated_at":"2015-02-06T15:32:29Z","body":"Hey, I would also prefer CSV files. It would be great to have a universal way of interacting with NILM data  as it seems to just hit the market and start fresh."}}