{"action":"created","issue":{"url":"https://api.github.com/repos/bartvm/fuel/issues/90","labels_url":"https://api.github.com/repos/bartvm/fuel/issues/90/labels{/name}","comments_url":"https://api.github.com/repos/bartvm/fuel/issues/90/comments","events_url":"https://api.github.com/repos/bartvm/fuel/issues/90/events","html_url":"https://github.com/bartvm/fuel/pull/90","id":71913072,"number":90,"title":"Introduce default batch iterator and default transformers","user":{"login":"vdumoulin","id":2034067,"avatar_url":"https://avatars.githubusercontent.com/u/2034067?v=3","gravatar_id":"","url":"https://api.github.com/users/vdumoulin","html_url":"https://github.com/vdumoulin","followers_url":"https://api.github.com/users/vdumoulin/followers","following_url":"https://api.github.com/users/vdumoulin/following{/other_user}","gists_url":"https://api.github.com/users/vdumoulin/gists{/gist_id}","starred_url":"https://api.github.com/users/vdumoulin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vdumoulin/subscriptions","organizations_url":"https://api.github.com/users/vdumoulin/orgs","repos_url":"https://api.github.com/users/vdumoulin/repos","events_url":"https://api.github.com/users/vdumoulin/events{/privacy}","received_events_url":"https://api.github.com/users/vdumoulin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-04-29T15:48:48Z","updated_at":"2015-05-07T15:42:42Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/bartvm/fuel/pulls/90","html_url":"https://github.com/bartvm/fuel/pull/90","diff_url":"https://github.com/bartvm/fuel/pull/90.diff","patch_url":"https://github.com/bartvm/fuel/pull/90.patch"},"body":"Following an offline discussion with @lamblin, I propose to introduce a notion of \"default transformers\" to `Dataset` subclasses.\r\n\r\nThe problem we're trying to solve is the following: suppose that we have a dataset of 8-bit-per-channel images that we want to add to Fuel's built-in datasets. Here are three approaches for storing this data on disk:\r\n\r\n* Store the data as `float32`, with pixel values between 0.0 and 1.0. Files are four times larger than they could be, which can be problematic for very large datasets, but it avoids complications on the implementation side.\r\n* Store the data as `uint8`. This leads to minimal file sizes, which could be important for very large datasets. However, since each pixel value is represented as an integer between 0 and 255, some preprocessing  (dividing by 255.0, casting as `float32`) is necessary so that the data is ML-friendly. Two approaches can be taken:\r\n    - Document the data format properly, and leave it to the user to do the conversion with `Transformer`s. This is quite simple on the implementation side, but it's not very user-friendly, since most of the time users have more boilerplate code to write.\r\n    - Let the dataset define a default transformation to the data, and define a canonical way for the users to access an iterator that will apply this default transformation.\r\n\r\nWe concluded that the best approach would be the latter one, as it combines compactness and user-friendliness.\r\n\r\nIn order to keep it simple for developers to implement new datasets, we figured that the best place to put this code was in the `Dataset` abstract class. Here's how it works:\r\n* We introduced a `apply_default_transformers` method that takes a data stream as input and applies a series of `Transformer`s to it. ~~By default, it simply returns the data stream passed as argument. Subclasses can override this method to define default transformations to apply, such as dividing by 255.0 and casting to `float32'~~. In addition, there's now a `default_transformers` property which is a tuple of tuples containing\r\n\r\n    - the class to instantiate,\r\n    - the args other than the data stream to pass to the constructor, and\r\n    - the kwargs to pass to the constructor.\r\n\r\nThe method will instantiate each transformer sequentially by passing it the previous transformer as the data stream argument, starting from the data stream received as input.\r\n* ~~In addition to `example_iteration_scheme`, we defined a `batch_iteration_scheme` attribute which subclasses can set to implement a sane iteration scheme default.~~\r\n* ~~We introduced a `get_batch_stream` method which takes an iteration scheme as argument, defaulting to `None`. If no iteration scheme is passed, `self.batch_iteration_scheme` is used. The method then instantiates a `DataStream` using the chosen iteration scheme and calls `self.apply_default_transformers` to apply default transformations.~~\r\n* A `default_stream` class method was added to `DataStream` as the preferred way of instantiating a data stream with the default transformers applied."},"comment":{"url":"https://api.github.com/repos/bartvm/fuel/issues/comments/99915906","html_url":"https://github.com/bartvm/fuel/pull/90#issuecomment-99915906","issue_url":"https://api.github.com/repos/bartvm/fuel/issues/90","id":99915906,"user":{"login":"bartvm","id":2299595,"avatar_url":"https://avatars.githubusercontent.com/u/2299595?v=3","gravatar_id":"","url":"https://api.github.com/users/bartvm","html_url":"https://github.com/bartvm","followers_url":"https://api.github.com/users/bartvm/followers","following_url":"https://api.github.com/users/bartvm/following{/other_user}","gists_url":"https://api.github.com/users/bartvm/gists{/gist_id}","starred_url":"https://api.github.com/users/bartvm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartvm/subscriptions","organizations_url":"https://api.github.com/users/bartvm/orgs","repos_url":"https://api.github.com/users/bartvm/repos","events_url":"https://api.github.com/users/bartvm/events{/privacy}","received_events_url":"https://api.github.com/users/bartvm/received_events","type":"User","site_admin":false},"created_at":"2015-05-07T15:42:41Z","updated_at":"2015-05-07T15:42:41Z","body":"Except for my comment about it being useful to have access to the default transformers of a dataset without the need to instantiate it (so that you can easily apply the same set of transformations to other datasets), the rest of it looks good to me."}}