{"action":"closed","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/672","labels_url":"https://api.github.com/repos/fchollet/keras/issues/672/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/672/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/672/events","html_url":"https://github.com/fchollet/keras/issues/672","id":105917414,"number":672,"title":"Data Loading and Preprocessing","user":{"login":"anjishnu","id":2019824,"avatar_url":"https://avatars.githubusercontent.com/u/2019824?v=3","gravatar_id":"","url":"https://api.github.com/users/anjishnu","html_url":"https://github.com/anjishnu","followers_url":"https://api.github.com/users/anjishnu/followers","following_url":"https://api.github.com/users/anjishnu/following{/other_user}","gists_url":"https://api.github.com/users/anjishnu/gists{/gist_id}","starred_url":"https://api.github.com/users/anjishnu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anjishnu/subscriptions","organizations_url":"https://api.github.com/users/anjishnu/orgs","repos_url":"https://api.github.com/users/anjishnu/repos","events_url":"https://api.github.com/users/anjishnu/events{/privacy}","received_events_url":"https://api.github.com/users/anjishnu/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-09-10T23:17:19Z","updated_at":"2015-09-11T15:35:17Z","closed_at":"2015-09-11T15:35:17Z","body":"I see that a lot of the examples have classes like the one attached below which basically encoder from some data format into a vector representation needed by the model.\r\n\r\nI think it makes sense for us to add these data encoders/decoders into Keras itself. They can provide an abstraction on top of the original data source that models need not be aware of. In terms of DNN training we could do something like. \r\n\r\ndata_provider = FileSystemBackedTextDataProvider(data_encoder=CharacterTable(max_length=20))\r\n\r\nfor i in range(200):\r\n  X_data, y_labels = data_provider.get_train(num_samples=100000)\r\n  model.fit(X_data, y_labels)\r\n\r\nThis way we abstract away the handling of the data from the actual model - within the data_provider we can have support for preprocessing and loading large files in batches. (a.la: https://github.com/karpathy/char-rnn/blob/master/util/CharSplitLMMinibatchLoader.lua)\r\n\r\nExample of a data_encoder:\r\n\r\n```\r\nclass CharacterTable(object):\r\n    \"\"\"\r\n    Given a set of characters:\r\n    + Encode them to a one hot integer representation\r\n    + Decode the one hot integer representation to their character output\r\n    + Decode a vector of probabilties to their character output\r\n    \"\"\"\r\n    def __init__(self, chars, maxlen):\r\n        self.chars = sorted(set(chars))\r\n        self.char_indices = dict((c, i) for i, c in enumerate(self.chars))\r\n        self.indices_char = dict((i, c) for i, c in enumerate(self.chars))\r\n        self.maxlen = maxlen\r\n    def encode(self, C, maxlen=None):\r\n        maxlen = maxlen if maxlen else self.maxlen\r\n        X = np.zeros((maxlen, len(self.chars)))\r\n        for i, c in enumerate(C):\r\n            X[i, self.char_indices[c]] = 1\r\n        return X\r\n    def decode(self, X, calc_argmax=True):\r\n        if calc_argmax:\r\n            X = X.argmax(axis=-1)\r\n        return ''.join(self.indices_char[x] for x in X)\r\n```"}}