{"action":"created","issue":{"url":"https://api.github.com/repos/bartvm/blocks/issues/236","labels_url":"https://api.github.com/repos/bartvm/blocks/issues/236/labels{/name}","comments_url":"https://api.github.com/repos/bartvm/blocks/issues/236/comments","events_url":"https://api.github.com/repos/bartvm/blocks/issues/236/events","html_url":"https://github.com/bartvm/blocks/issues/236","id":57169029,"number":236,"title":"Serialization and reproducibility ","user":{"login":"bartvm","id":2299595,"avatar_url":"https://avatars.githubusercontent.com/u/2299595?v=3","gravatar_id":"","url":"https://api.github.com/users/bartvm","html_url":"https://github.com/bartvm","followers_url":"https://api.github.com/users/bartvm/followers","following_url":"https://api.github.com/users/bartvm/following{/other_user}","gists_url":"https://api.github.com/users/bartvm/gists{/gist_id}","starred_url":"https://api.github.com/users/bartvm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartvm/subscriptions","organizations_url":"https://api.github.com/users/bartvm/orgs","repos_url":"https://api.github.com/users/bartvm/repos","events_url":"https://api.github.com/users/bartvm/events{/privacy}","received_events_url":"https://api.github.com/users/bartvm/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/bartvm/blocks/labels/major","name":"major","color":"eb6420"},{"url":"https://api.github.com/repos/bartvm/blocks/labels/question","name":"question","color":"cc317c"},{"url":"https://api.github.com/repos/bartvm/blocks/labels/refactor","name":"refactor","color":"5319e7"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-02-10T12:36:29Z","updated_at":"2015-02-11T15:55:31Z","closed_at":null,"body":"At the risk of (re-)opening a can of worms, I wanted to bring up the topic of serialization and reproducibility once more. (Mostly following a lengthy discussion on Pylearn2 yesterday where many issues regarding reproducibility reared their head, and my frustrating experience with serialization and `into` yesterday.)\r\n\r\nSo here's the proposition:\r\n\r\n> Should we relinquish complete reproducibility for the sake of clarity of code and ease of development? Instead of aiming for checkpointing at any point we could e.g. aim to support completely transparent checkpointing **only between epochs**.\r\n\r\n## Pros\r\n\r\n* Clarity of code; probably best illustrated by examples (see end).\r\n* Ease of development; fewer peculiarities that new developers need to get familiar with e.g. no generators, no lambda expressions, etc.\r\n* Full reproducibility is only relevant in a limited number of cases; in these cases, you would simply have to make sure that you only restart in between epochs.\r\n* We could possibly switch to a single serialization method instead of having to maintain two.\r\n* Possibility of using `pickle`/`cPickle` instead of `dill`; `dill` tends to crash because it exceeds the default maximum recursion depth.\r\n\r\n## Cons\r\n\r\n* Results won't be entirely reproducible, while reproducibility should be the hallmark of scientific experimentation; this could be especially problematic for experiments where epochs don't exist (infinite datasets) or where epochs are particularly long (order of days).\r\n* Our serialization protocol will be more involved; the current serialization is limited to a single `dill.dump` call.\r\n* Users might not realize the full effects of stopping and restarting their experiments; they could be training only on the first half of a large dataset because they keep restarting, not realizing their epoch iterator won't be restored\r\n\r\n## Proposal\r\n\r\n* If aiming for full reproducibility in between epochs only, we don't need to serialize the dataset iterator (epoch iterator) or the dataset's state. The data stream can simply delete its state when pickled, by default.\r\n* We could let states implement a `__getstate__` method, allowing datasets to support a form of checkpointing nonetheless if they do so explicitly.\r\n* The current secondary serialization method would have to be extended to include the state of training extensions (e.g. momentum, moving averages). This too could be implemented by requiring these extensions to implement `__getstate__` and `__setstate__` methods, or simply by pickling them (assuming that we will still pickle the Theano graph instead of dumping variables).\r\n\r\nI am not entirely convinced that the parameter dump we have implemented right now would be a good replacement for pickling the Theano computation graph though. We might still want to support two separate methods there (as well as push for #2339 to be merged).\r\n\r\n## Examples\r\n\r\nThese are some examples of what code looks like right now in the library, and what it could look like if we no longer had to worry about serializing in the middle of an epoch.\r\n\r\n```python\r\nclass TextFileState(object):\r\n    pass\r\n\r\nclass TextFile(Dataset):\r\n    ...\r\n    def open(self):\r\n        state = TextFileState()\r\n        state.current_index = 0\r\n        state.file = self._open_file(state.current_index)\r\n        return state\r\n\r\n    def get_data(self, state=None, request=None):\r\n        ...\r\n        while True:\r\n            if state.file is None:\r\n                raise StopIteration\r\n            sentence = state.file.readline()\r\n            if not sentence:\r\n                state.file.close()\r\n                state.file = None\r\n                if state.current_index == len(self.files) - 1:\r\n                    raise StopIteration\r\n                else:\r\n                    state.current_index += 1\r\n                    state.file = self._open_file(state.current_index)\r\n            else:\r\n                break\r\n```\r\n```python\r\nclass TextFile(Dataset):\r\n    ...\r\n    def open(self):\r\n        return concat(into(Iterator, TextFile(filename)) for filename in self.files)\r\n\r\n\r\n    def get_data(self, state=None, request=None):\r\n        ...\r\n        sentence = next(state)\r\n```\r\n---\r\n```python\r\nclass LambdaIterator(six.Iterator):\r\n    def __init__(self, next_function):\r\n        self.next_function = next_function\r\n\r\n    def __iter__(self):\r\n        return self\r\n\r\n    def __next__(self):\r\n        return self.next_function()\r\n```\r\n```\r\n```\r\n---\r\n```python\r\nclass ConstantScheme(BatchSizeScheme):\r\n    def __init__(self, batch_size, num_examples=None, times=None):\r\n        if num_examples and times:\r\n            raise ValueError\r\n        self.batch_size = batch_size\r\n        self.num_examples = num_examples\r\n        self.times = times\r\n\r\n    def get_request_iterator(self):\r\n        return ConstantIterator(self.batch_size, self.num_examples, self.times)\r\n\r\n\r\nclass ConstantIterator(six.Iterator):\r\n    def __init__(self, batch_size, num_examples=None, times=None):\r\n        if num_examples is not None and times is not None:\r\n            raise ValueError\r\n        if times is not None or num_examples is not None:\r\n            if not ((times is None or times >= 1) or\r\n                    (num_examples is None or num_examples >= 1)):\r\n                raise ValueError\r\n            self.current = 0\r\n\r\n        self.batch_size = batch_size\r\n        self.num_examples = num_examples\r\n        self.times = times\r\n\r\n    def __iter__(self):\r\n        return self\r\n\r\n    def __next__(self):\r\n        if self.times or self.num_examples:\r\n            if self.current == self.times or self.current == self.num_examples:\r\n                raise StopIteration\r\n            if self.times:\r\n                self.current += 1\r\n            else:\r\n                batch_size = min(self.batch_size,\r\n                                 self.num_examples - self.current)\r\n                self.current += batch_size\r\n                return batch_size\r\n        return self.batch_size\r\n```\r\n```python\r\nclass ConstantScheme(BatchSizeScheme):\r\n    def __init__(self, batch_size, num_examples=None, times=None):\r\n        if num_examples and times:\r\n            raise ValueError\r\n        self.batch_size = batch_size\r\n        self.num_examples = num_examples\r\n        self.times = times\r\n\r\n    def get_request_iterator(self):\r\n        if self.times:\r\n            return repeat(self.batch_size, self.times)\r\n        if self.num_examples:\r\n            d, r = divmod(self.num_examples, self.batch_size)\r\n            return concat(repeat(self.batch_size, d), [r] if r else [])\r\n        return repeat(self.batch_size)\r\n```\r\n\r\n## Thoughts?\r\n\r\nHow important is reproducibility and checkpointing, really? Is it worth the effort we put in right now? Or does the maintainability and clarity of the framework weigh heavier? Considering the amount of changes this would need, I think we should make this decision before the framework grows much larger, because it will only become harder to refactor."},"comment":{"url":"https://api.github.com/repos/bartvm/blocks/issues/comments/73905128","html_url":"https://github.com/bartvm/blocks/issues/236#issuecomment-73905128","issue_url":"https://api.github.com/repos/bartvm/blocks/issues/236","id":73905128,"user":{"login":"dwf","id":60206,"avatar_url":"https://avatars.githubusercontent.com/u/60206?v=3","gravatar_id":"","url":"https://api.github.com/users/dwf","html_url":"https://github.com/dwf","followers_url":"https://api.github.com/users/dwf/followers","following_url":"https://api.github.com/users/dwf/following{/other_user}","gists_url":"https://api.github.com/users/dwf/gists{/gist_id}","starred_url":"https://api.github.com/users/dwf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dwf/subscriptions","organizations_url":"https://api.github.com/users/dwf/orgs","repos_url":"https://api.github.com/users/dwf/repos","events_url":"https://api.github.com/users/dwf/events{/privacy}","received_events_url":"https://api.github.com/users/dwf/received_events","type":"User","site_admin":false},"created_at":"2015-02-11T15:55:31Z","updated_at":"2015-02-11T15:55:31Z","body":"Note that we've had problems in pylearn2 with the recursion limit,\ncompletely dill-free. Normal pickle has this problem too, to a degree.\n"}}