{"action":"created","issue":{"url":"https://api.github.com/repos/lisa-lab/pylearn2/issues/1375","labels_url":"https://api.github.com/repos/lisa-lab/pylearn2/issues/1375/labels{/name}","comments_url":"https://api.github.com/repos/lisa-lab/pylearn2/issues/1375/comments","events_url":"https://api.github.com/repos/lisa-lab/pylearn2/issues/1375/events","html_url":"https://github.com/lisa-lab/pylearn2/pull/1375","id":55939302,"number":1375,"title":"Hdf5 file new handler","user":{"login":"fvisin","id":4639261,"avatar_url":"https://avatars.githubusercontent.com/u/4639261?v=3","gravatar_id":"","url":"https://api.github.com/users/fvisin","html_url":"https://github.com/fvisin","followers_url":"https://api.github.com/users/fvisin/followers","following_url":"https://api.github.com/users/fvisin/following{/other_user}","gists_url":"https://api.github.com/users/fvisin/gists{/gist_id}","starred_url":"https://api.github.com/users/fvisin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fvisin/subscriptions","organizations_url":"https://api.github.com/users/fvisin/orgs","repos_url":"https://api.github.com/users/fvisin/repos","events_url":"https://api.github.com/users/fvisin/events{/privacy}","received_events_url":"https://api.github.com/users/fvisin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-01-29T18:44:32Z","updated_at":"2015-02-03T03:35:27Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/lisa-lab/pylearn2/pulls/1375","html_url":"https://github.com/lisa-lab/pylearn2/pull/1375","diff_url":"https://github.com/lisa-lab/pylearn2/pull/1375.diff","patch_url":"https://github.com/lisa-lab/pylearn2/pull/1375.patch"},"body":""},"comment":{"url":"https://api.github.com/repos/lisa-lab/pylearn2/issues/comments/72587046","html_url":"https://github.com/lisa-lab/pylearn2/pull/1375#issuecomment-72587046","issue_url":"https://api.github.com/repos/lisa-lab/pylearn2/issues/1375","id":72587046,"user":{"login":"fvisin","id":4639261,"avatar_url":"https://avatars.githubusercontent.com/u/4639261?v=3","gravatar_id":"","url":"https://api.github.com/users/fvisin","html_url":"https://github.com/fvisin","followers_url":"https://api.github.com/users/fvisin/followers","following_url":"https://api.github.com/users/fvisin/following{/other_user}","gists_url":"https://api.github.com/users/fvisin/gists{/gist_id}","starred_url":"https://api.github.com/users/fvisin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fvisin/subscriptions","organizations_url":"https://api.github.com/users/fvisin/orgs","repos_url":"https://api.github.com/users/fvisin/repos","events_url":"https://api.github.com/users/fvisin/events{/privacy}","received_events_url":"https://api.github.com/users/fvisin/received_events","type":"User","site_admin":false},"created_at":"2015-02-03T03:35:27Z","updated_at":"2015-02-03T03:35:27Z","body":"> Having an `__iter__` method actually adds something, it enables the\r\n> following syntax:\r\n> ```python\r\n> for data in dataset:\r\n>     ...\r\n> ```\r\n> rather than:\r\n> ```python\r\n> it = dataset.iterator()\r\n> for data in it:\r\n>     ...\r\n> ```\r\n> So I would not get rid of it.\r\n\r\nAgreed.\r\n\r\n> That would not be true of datasets where the data is generated on the fly,\r\n> for instance. For other cases, `index` could be a hash, or a string.\r\n\r\nThe interface seems still good to me, just to say that the `get` method is the one you should refer to when you want to get data from the dataset. The specific implementation will depend on the specific dataset. Index can be set to None by default: the on-the-fly dataset will not use it, the topo-view-like dataset will use a topo-view-like nd index, the design-matrix-like dataset will use an int. The common syntax would be that index is a list of _indices_, i.e., slice/int for dense design matrices, nd-slices/nd-int for topo-view, and so on ...\r\n\r\n> Right now, the convention is that the \"mode\" or iteration scheme in\r\n> FiniteDatasetIterator returns something that is fit to index an ndarray\r\n> or similar object, and is the only place using `get`.\r\n\r\nIf _ndarray_ includes 1d arrays and 2d arrays, then you can have one code that deals with all the possible cases. The modification I was proposing would do exactly this, basically implementing the _list of indices_ idea of before, for the case of ndarrays (e.g. topo view and dense design matrices).\r\n\r\n> If we want to use FiniteDatasetIterator everywhere, then we can\r\n> decide that all Datasets should have a `get` method that takes as\r\n> indices something suitable for it, but then the user has to make sure\r\n> it uses a mode that returns the right kind of indices for the dataset.\r\n> It would make more sense for me to have a different kind of Iterator\r\n> for these datasets, and if we do, we may as well not use `get`.\r\n\r\nIf you prefer to keep them separated we need to think how to deal with topo-view-like datasets, cause ImageNet is in that format, but I don't think it is necessary.\r\n\r\nI am probably loosing the big picture here, but I would say that the DenseDesign vs TopoView distinction is more semantic than topological: they both are nd arrays (the space in between is on purpose), what really matters to the end of training is the space that tells you the shape and the semantic of the data.\r\n\r\nWhat really matters to be able to iterate on the data, instead, is just the shape of the dataset (that we already have). Once you pass a list of one or more indices of the same shape of the dataset, it is easy to check that they have the right shape and to manage the 3 different possible cases (slices, index, indices). The only situation in which this would not work, would be if you have a dataset whose data cannot be accessed specifying an index (whatever the format of the index), but I cannot imagine such a case."}}