{"action":"created","issue":{"url":"https://api.github.com/repos/nilmtk/nilmtk/issues/411","labels_url":"https://api.github.com/repos/nilmtk/nilmtk/issues/411/labels{/name}","comments_url":"https://api.github.com/repos/nilmtk/nilmtk/issues/411/comments","events_url":"https://api.github.com/repos/nilmtk/nilmtk/issues/411/events","html_url":"https://github.com/nilmtk/nilmtk/issues/411","id":89047667,"number":411,"title":"Hart algorithm issue","user":{"login":"gjwo","id":12109990,"avatar_url":"https://avatars.githubusercontent.com/u/12109990?v=3","gravatar_id":"","url":"https://api.github.com/users/gjwo","html_url":"https://github.com/gjwo","followers_url":"https://api.github.com/users/gjwo/followers","following_url":"https://api.github.com/users/gjwo/following{/other_user}","gists_url":"https://api.github.com/users/gjwo/gists{/gist_id}","starred_url":"https://api.github.com/users/gjwo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gjwo/subscriptions","organizations_url":"https://api.github.com/users/gjwo/orgs","repos_url":"https://api.github.com/users/gjwo/repos","events_url":"https://api.github.com/users/gjwo/events{/privacy}","received_events_url":"https://api.github.com/users/gjwo/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-06-17T15:22:02Z","updated_at":"2015-10-04T16:48:10Z","closed_at":"2015-10-04T16:48:10Z","body":"As requested\r\n```python\r\nDisaggregation experiments\r\nCustomary imports\r\nIn [42]:\r\n\r\nimport numpy as np\r\nimport pandas as pd\r\nfrom os.path import join\r\n​\r\nfrom pylab import rcParams\r\nimport matplotlib.pyplot as plt\r\n%matplotlib inline\r\n​\r\n#rcParams['figure.figsize'] = (12, 6)\r\nrcParams['figure.figsize'] = (13, 6)\r\nplt.style.use('ggplot')\r\nimport nilmtk\r\nfrom nilmtk import DataSet, TimeFrame, MeterGroup, HDFDataStore\r\nfrom nilmtk.disaggregate import CombinatorialOptimisation\r\nfrom nilmtk.utils import print_dict, show_versions\r\nfrom nilmtk.metrics import f1_score\r\n#import seaborn as sns\r\n#sns.set_palette(\"Set3\", n_colors=12)\r\n​\r\n​\r\nimport warnings\r\nwarnings.filterwarnings(\"ignore\")\r\nshow versions for any diagnostics\r\nIn [28]:\r\n\r\nshow_versions()\r\n​\r\n\r\nINSTALLED VERSIONS\r\n------------------\r\ncommit: None\r\npython: 2.7.9.final.0\r\npython-bits: 64\r\nOS: Windows\r\nOS-release: 7\r\nmachine: AMD64\r\nprocessor: Intel64 Family 6 Model 37 Stepping 2, GenuineIntel\r\nbyteorder: little\r\nLC_ALL: None\r\nLANG: None\r\n\r\npandas: 0.16.1\r\nnose: 1.3.6\r\nCython: 0.22\r\nnumpy: 1.9.2\r\nscipy: 0.15.1\r\nstatsmodels: 0.6.1\r\nIPython: 3.1.0\r\nsphinx: 1.2.3\r\npatsy: 0.3.0\r\ndateutil: 2.4.2\r\npytz: 2015.4\r\nbottleneck: None\r\ntables: 3.1.1\r\nnumexpr: 2.3.1\r\nmatplotlib: 1.4.3\r\nopenpyxl: 1.8.5\r\nxlrd: 0.9.3\r\nxlwt: 0.7.5\r\nxlsxwriter: 0.6.7\r\nlxml: 3.4.2\r\nbs4: 4.3.2\r\nhtml5lib: None\r\nhttplib2: None\r\napiclient: None\r\nsqlalchemy: 0.9.9\r\npymysql: None\r\npsycopg2: None\r\nNone\r\n\r\nDate: 2015-06-17 10:25:58.140000\r\nPlatform: Windows-7-6.1.7601-SP1\r\nSystem version: 2.7\r\nnilmtk version: 0.2.0.dev-4ae125a\r\nnilm_metadata version: 0.2.0.dev-f4f8eaa\r\nnumpy version: 1.9.2\r\nmatplotlib version: 1.4.3\r\npandas version: 0.16.1\r\nsklearn version: 0.16.1\r\nLoad dataset\r\nIn [43]:\r\n\r\ndata_dir = '/Users/GJWood/nilm_gjw_data/HDF5/'\r\ngjw = DataSet(join(data_dir, 'nilm_gjw_data.hdf5'))\r\nprint('loaded ' + str(len(gjw.buildings)) + ' buildings')\r\nbuilding_number=1\r\nloaded 1 buildings\r\nLet us perform our analysis on selected 2 days\r\nIn [44]:\r\n\r\ngjw.store.window = TimeFrame(start='2013-12-03 00:00:00', end='2013-12-05 00:00:00')\r\nelec = gjw.buildings[building_number].elec\r\nmains = elec.mains()\r\nIn [45]:\r\n\r\nhouse = elec['fridge'] #only one meter so any selection will do\r\ndf = house.load().next() #load the first chunk of data into a dataframe\r\ndf.info() #check that the data is what we want (optional)\r\n#note the data has two columns and a time index\r\n​\r\n<class 'pandas.core.frame.DataFrame'>\r\nDatetimeIndex: 172800 entries, 2013-12-03 00:00:00+00:00 to 2013-12-04 23:59:59+00:00\r\nData columns (total 2 columns):\r\n(power, apparent)    172800 non-null float32\r\n(power, reactive)    172800 non-null float32\r\ndtypes: float32(2)\r\nmemory usage: 2.6 MB\r\nIn [38]:\r\n\r\ndf.head()\r\nOut[38]:\r\nphysical_quantity\tpower\r\ntype\tapparent\treactive\r\n2013-12-03 00:00:00+00:00\t231\t96\r\n2013-12-03 00:00:01+00:00\t231\t96\r\n2013-12-03 00:00:02+00:00\t231\t96\r\n2013-12-03 00:00:03+00:00\t231\t96\r\n2013-12-03 00:00:04+00:00\t231\t96\r\nIn [39]:\r\n\r\ndf.tail()\r\nOut[39]:\r\nphysical_quantity\tpower\r\ntype\tapparent\treactive\r\n2013-12-04 23:59:55+00:00\t271\t0\r\n2013-12-04 23:59:56+00:00\t271\t0\r\n2013-12-04 23:59:57+00:00\t271\t0\r\n2013-12-04 23:59:58+00:00\t271\t0\r\n2013-12-04 23:59:59+00:00\t271\t0\r\nIn [46]:\r\n\r\ndf.plot()\r\nOut[46]:\r\n<matplotlib.axes._subplots.AxesSubplot at 0x27248ba8>\r\n\r\nTraining\r\nWe'll now do the training from the aggregate data. The algorithm segments the time series data into steady and transient states. Thus, we'll first figure out the transient and the steady states. Next, we'll try and pair the on and the off transitions based on their proximity in time and value.\r\nIn [51]:\r\n\r\ndf.ix['2013-12-03 11:00:00':'2013-12-03 12:00:00'].plot()# select a time range and plot it\r\nOut[51]:\r\n<matplotlib.axes._subplots.AxesSubplot at 0x1c0018d0>\r\n\r\nIn [48]:\r\n\r\nfrom nilmtk.disaggregate.hart_85 import Hart85\r\nh = Hart85()\r\nh.train(mains,cols=[('power','apparent'),('power','reactive')])\r\n​\r\nFinding Edges, please wait ...\r\nEdge detection complete.\r\nCreating transition frame ...\r\nTransition frame created.\r\nCreating states frame ...\r\nStates frame created.\r\nFinished.\r\n---------------------------------------------------------------------------\r\nValueError                                Traceback (most recent call last)\r\n<ipython-input-48-e070a6ace91e> in <module>()\r\n      1 from nilmtk.disaggregate.hart_85 import Hart85\r\n      2 h = Hart85()\r\n----> 3 h.train(mains,cols=[('power','apparent'),('power','reactive')])\r\n\r\nc:\\users\\gjwood\\nilmtk\\nilmtk\\disaggregate\\hart_85.pyc in train(self, metergroup, cols, buffer_size, noise_level, state_threshold, min_tolerance, percent_tolerance, large_transition, **kwargs)\r\n    233         [self.steady_states, self.transients] = find_steady_states_transients(metergroup,  cols, noise_level, state_threshold, **kwargs)\r\n    234         self.pair_df = self.pair(buffer_size, min_tolerance, percent_tolerance, large_transition)\r\n--> 235         self.centroids = hart85_means_shift_cluster(self.pair_df, cols)\r\n    236 \r\n    237     def pair(self, buffer_size, min_tolerance, percent_tolerance, large_transition):\r\n\r\nc:\\users\\gjwood\\nilmtk\\nilmtk\\feature_detectors\\cluster.pyc in hart85_means_shift_cluster(pair_buffer_df, cols)\r\n    135                                                                      ((np.fabs(row['T1 Reactive']) + np.fabs(row['T2 Reactive'])) / 2), axis=1), index=pair_buffer_df.index)\r\n    136 \r\n--> 137     X = cluster_df.values.reshape((len(cluster_df.index), len(cols)))\r\n    138     ms = MeanShift(bin_seeding=True)\r\n    139     ms.fit(X)\r\n\r\nValueError: total size of new array must be unchanged\r\n\r\nIn [49]:\r\n\r\nfrom nilmtk.disaggregate.hart_85 import Hart85\r\nh = Hart85()\r\nh.train(mains,cols=[('power','apparent')])\r\nFinding Edges, please wait ...\r\nEdge detection complete.\r\nCreating transition frame ...\r\nTransition frame created.\r\nCreating states frame ...\r\nStates frame created.\r\nFinished.\r\n---------------------------------------------------------------------------\r\nValueError                                Traceback (most recent call last)\r\n<ipython-input-49-0fd1cf0ce185> in <module>()\r\n      1 from nilmtk.disaggregate.hart_85 import Hart85\r\n      2 h = Hart85()\r\n----> 3 h.train(mains,cols=[('power','apparent')])\r\n\r\nc:\\users\\gjwood\\nilmtk\\nilmtk\\disaggregate\\hart_85.pyc in train(self, metergroup, cols, buffer_size, noise_level, state_threshold, min_tolerance, percent_tolerance, large_transition, **kwargs)\r\n    233         [self.steady_states, self.transients] = find_steady_states_transients(metergroup,  cols, noise_level, state_threshold, **kwargs)\r\n    234         self.pair_df = self.pair(buffer_size, min_tolerance, percent_tolerance, large_transition)\r\n--> 235         self.centroids = hart85_means_shift_cluster(self.pair_df, cols)\r\n    236 \r\n    237     def pair(self, buffer_size, min_tolerance, percent_tolerance, large_transition):\r\n\r\nc:\\users\\gjwood\\nilmtk\\nilmtk\\feature_detectors\\cluster.pyc in hart85_means_shift_cluster(pair_buffer_df, cols)\r\n    137     X = cluster_df.values.reshape((len(cluster_df.index), len(cols)))\r\n    138     ms = MeanShift(bin_seeding=True)\r\n--> 139     ms.fit(X)\r\n    140     labels = ms.labels_\r\n    141     cluster_centers = ms.cluster_centers_\r\n\r\nc:\\Users\\GJWood\\Anaconda\\lib\\site-packages\\sklearn\\cluster\\mean_shift_.pyc in fit(self, X, y)\r\n    343             Samples to cluster.\r\n    344         \"\"\"\r\n--> 345         X = check_array(X)\r\n    346         self.cluster_centers_, self.labels_ = \\\r\n    347             mean_shift(X, bandwidth=self.bandwidth, seeds=self.seeds,\r\n\r\nc:\\Users\\GJWood\\Anaconda\\lib\\site-packages\\sklearn\\utils\\validation.pyc in check_array(array, accept_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features)\r\n    358             raise ValueError(\"Found array with %d sample(s) (shape=%s) while a\"\r\n    359                              \" minimum of %d is required.\"\r\n--> 360                              % (n_samples, shape_repr, ensure_min_samples))\r\n    361 \r\n    362     if ensure_min_features > 0 and array.ndim == 2:\r\n\r\nValueError: Found array with 0 sample(s) (shape=(0, 1)) while a minimum of 1 is required.\r\n```"},"comment":{"url":"https://api.github.com/repos/nilmtk/nilmtk/issues/comments/145365713","html_url":"https://github.com/nilmtk/nilmtk/issues/411#issuecomment-145365713","issue_url":"https://api.github.com/repos/nilmtk/nilmtk/issues/411","id":145365713,"user":{"login":"gjwo","id":12109990,"avatar_url":"https://avatars.githubusercontent.com/u/12109990?v=3","gravatar_id":"","url":"https://api.github.com/users/gjwo","html_url":"https://github.com/gjwo","followers_url":"https://api.github.com/users/gjwo/followers","following_url":"https://api.github.com/users/gjwo/following{/other_user}","gists_url":"https://api.github.com/users/gjwo/gists{/gist_id}","starred_url":"https://api.github.com/users/gjwo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gjwo/subscriptions","organizations_url":"https://api.github.com/users/gjwo/orgs","repos_url":"https://api.github.com/users/gjwo/repos","events_url":"https://api.github.com/users/gjwo/events{/privacy}","received_events_url":"https://api.github.com/users/gjwo/received_events","type":"User","site_admin":false},"created_at":"2015-10-04T16:48:10Z","updated_at":"2015-10-04T16:48:10Z","body":"I can now run past the training stage up to the point in disaggregation were I get the error in issue #435 so I think this issue is now superseded."}}