{"action":"created","comment":{"url":"https://api.github.com/repos/mila-udem/fuel/pulls/comments/39675146","id":39675146,"diff_hunk":"@@ -1,11 +1,410 @@\n Overview\n ========\n \n-The data pipeline abstraction that Fuel provides looks complicated at first\n-sight, but can be very powerful.\n+We'll go over a quick example to see what Fuel is capable of.\n+\n+Let's start by creating some random data to act as features and targets. We'll\n+pretend that we have eight 2x2 grayscale images separated into four classes.\n+\n+>>> import numpy\n+>>> seed = 1234\n+>>> rng = numpy.random.RandomState(seed)\n+>>> features = rng.randint(256, size=(8, 2, 2))\n+>>> targets = rng.randint(4, size=(8, 1))\n+\n+Our goal is to use Fuel to interface with this data, iterate over it in various\n+ways and apply transformations to it on the fly.\n+\n+Division of labour\n+------------------\n+\n+There are four basic tasks that Fuel needs to handle:\n+\n+* Interface with the data, be it on disk or in memory.\n+* Decide which data points to visit, and in which order.\n+* Iterate over the selected data points.\n+* At each iteration step, apply some transformation to the selected data points.\n+\n+Each of those four tasks is delegated to a particular class of objects, which\n+we'll be introducing in order.\n+\n+Datasets: interfacing with data\n+-------------------------------\n+\n+.. admonition:: In summary\n+    :class: tip","path":"docs/overview.rst","position":37,"original_position":37,"commit_id":"2f71d5bb44c5288832d7546dfee7ec6754c0e990","original_commit_id":"2f71d5bb44c5288832d7546dfee7ec6754c0e990","user":{"login":"fvisin","id":4639261,"avatar_url":"https://avatars.githubusercontent.com/u/4639261?v=3","gravatar_id":"","url":"https://api.github.com/users/fvisin","html_url":"https://github.com/fvisin","followers_url":"https://api.github.com/users/fvisin/followers","following_url":"https://api.github.com/users/fvisin/following{/other_user}","gists_url":"https://api.github.com/users/fvisin/gists{/gist_id}","starred_url":"https://api.github.com/users/fvisin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fvisin/subscriptions","organizations_url":"https://api.github.com/users/fvisin/orgs","repos_url":"https://api.github.com/users/fvisin/repos","events_url":"https://api.github.com/users/fvisin/events{/privacy}","received_events_url":"https://api.github.com/users/fvisin/received_events","type":"User","site_admin":false},"body":"I would add another level to the list, moving all these items in one of this top level items:\r\n* :class:`Dataset`\r\n* :class:`IterableDataset`\r\n* :class:`IndexableDataset`","created_at":"2015-09-16T19:37:17Z","updated_at":"2015-09-16T19:37:17Z","html_url":"https://github.com/mila-udem/fuel/pull/225#discussion_r39675146","pull_request_url":"https://api.github.com/repos/mila-udem/fuel/pulls/225","_links":{"self":{"href":"https://api.github.com/repos/mila-udem/fuel/pulls/comments/39675146"},"html":{"href":"https://github.com/mila-udem/fuel/pull/225#discussion_r39675146"},"pull_request":{"href":"https://api.github.com/repos/mila-udem/fuel/pulls/225"}}},"pull_request":{"url":"https://api.github.com/repos/mila-udem/fuel/pulls/225","id":44143039,"html_url":"https://github.com/mila-udem/fuel/pull/225","diff_url":"https://github.com/mila-udem/fuel/pull/225.diff","patch_url":"https://github.com/mila-udem/fuel/pull/225.patch","issue_url":"https://api.github.com/repos/mila-udem/fuel/issues/225","number":225,"state":"open","locked":false,"title":"Overhaul documentation","user":{"login":"vdumoulin","id":2034067,"avatar_url":"https://avatars.githubusercontent.com/u/2034067?v=3","gravatar_id":"","url":"https://api.github.com/users/vdumoulin","html_url":"https://github.com/vdumoulin","followers_url":"https://api.github.com/users/vdumoulin/followers","following_url":"https://api.github.com/users/vdumoulin/following{/other_user}","gists_url":"https://api.github.com/users/vdumoulin/gists{/gist_id}","starred_url":"https://api.github.com/users/vdumoulin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vdumoulin/subscriptions","organizations_url":"https://api.github.com/users/vdumoulin/orgs","repos_url":"https://api.github.com/users/vdumoulin/repos","events_url":"https://api.github.com/users/vdumoulin/events{/privacy}","received_events_url":"https://api.github.com/users/vdumoulin/received_events","type":"User","site_admin":false},"body":"Fixes #219, #195.\r\n\r\n@lamblin would you mind having a look and telling me what you think of it?","created_at":"2015-09-03T18:50:57Z","updated_at":"2015-09-16T19:37:17Z","closed_at":null,"merged_at":null,"merge_commit_sha":"2662a6c6f7166a400259a9d3710591067c035476","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/mila-udem/fuel/pulls/225/commits","review_comments_url":"https://api.github.com/repos/mila-udem/fuel/pulls/225/comments","review_comment_url":"https://api.github.com/repos/mila-udem/fuel/pulls/comments{/number}","comments_url":"https://api.github.com/repos/mila-udem/fuel/issues/225/comments","statuses_url":"https://api.github.com/repos/mila-udem/fuel/statuses/2f71d5bb44c5288832d7546dfee7ec6754c0e990","head":{"label":"vdumoulin:documentation","ref":"documentation","sha":"2f71d5bb44c5288832d7546dfee7ec6754c0e990","user":{"login":"vdumoulin","id":2034067,"avatar_url":"https://avatars.githubusercontent.com/u/2034067?v=3","gravatar_id":"","url":"https://api.github.com/users/vdumoulin","html_url":"https://github.com/vdumoulin","followers_url":"https://api.github.com/users/vdumoulin/followers","following_url":"https://api.github.com/users/vdumoulin/following{/other_user}","gists_url":"https://api.github.com/users/vdumoulin/gists{/gist_id}","starred_url":"https://api.github.com/users/vdumoulin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vdumoulin/subscriptions","organizations_url":"https://api.github.com/users/vdumoulin/orgs","repos_url":"https://api.github.com/users/vdumoulin/repos","events_url":"https://api.github.com/users/vdumoulin/events{/privacy}","received_events_url":"https://api.github.com/users/vdumoulin/received_events","type":"User","site_admin":false},"repo":{"id":30834931,"name":"fuel","full_name":"vdumoulin/fuel","owner":{"login":"vdumoulin","id":2034067,"avatar_url":"https://avatars.githubusercontent.com/u/2034067?v=3","gravatar_id":"","url":"https://api.github.com/users/vdumoulin","html_url":"https://github.com/vdumoulin","followers_url":"https://api.github.com/users/vdumoulin/followers","following_url":"https://api.github.com/users/vdumoulin/following{/other_user}","gists_url":"https://api.github.com/users/vdumoulin/gists{/gist_id}","starred_url":"https://api.github.com/users/vdumoulin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vdumoulin/subscriptions","organizations_url":"https://api.github.com/users/vdumoulin/orgs","repos_url":"https://api.github.com/users/vdumoulin/repos","events_url":"https://api.github.com/users/vdumoulin/events{/privacy}","received_events_url":"https://api.github.com/users/vdumoulin/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/vdumoulin/fuel","description":"A data pipeline framework for machine learning","fork":true,"url":"https://api.github.com/repos/vdumoulin/fuel","forks_url":"https://api.github.com/repos/vdumoulin/fuel/forks","keys_url":"https://api.github.com/repos/vdumoulin/fuel/keys{/key_id}","collaborators_url":"https://api.github.com/repos/vdumoulin/fuel/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/vdumoulin/fuel/teams","hooks_url":"https://api.github.com/repos/vdumoulin/fuel/hooks","issue_events_url":"https://api.github.com/repos/vdumoulin/fuel/issues/events{/number}","events_url":"https://api.github.com/repos/vdumoulin/fuel/events","assignees_url":"https://api.github.com/repos/vdumoulin/fuel/assignees{/user}","branches_url":"https://api.github.com/repos/vdumoulin/fuel/branches{/branch}","tags_url":"https://api.github.com/repos/vdumoulin/fuel/tags","blobs_url":"https://api.github.com/repos/vdumoulin/fuel/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/vdumoulin/fuel/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/vdumoulin/fuel/git/refs{/sha}","trees_url":"https://api.github.com/repos/vdumoulin/fuel/git/trees{/sha}","statuses_url":"https://api.github.com/repos/vdumoulin/fuel/statuses/{sha}","languages_url":"https://api.github.com/repos/vdumoulin/fuel/languages","stargazers_url":"https://api.github.com/repos/vdumoulin/fuel/stargazers","contributors_url":"https://api.github.com/repos/vdumoulin/fuel/contributors","subscribers_url":"https://api.github.com/repos/vdumoulin/fuel/subscribers","subscription_url":"https://api.github.com/repos/vdumoulin/fuel/subscription","commits_url":"https://api.github.com/repos/vdumoulin/fuel/commits{/sha}","git_commits_url":"https://api.github.com/repos/vdumoulin/fuel/git/commits{/sha}","comments_url":"https://api.github.com/repos/vdumoulin/fuel/comments{/number}","issue_comment_url":"https://api.github.com/repos/vdumoulin/fuel/issues/comments{/number}","contents_url":"https://api.github.com/repos/vdumoulin/fuel/contents/{+path}","compare_url":"https://api.github.com/repos/vdumoulin/fuel/compare/{base}...{head}","merges_url":"https://api.github.com/repos/vdumoulin/fuel/merges","archive_url":"https://api.github.com/repos/vdumoulin/fuel/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/vdumoulin/fuel/downloads","issues_url":"https://api.github.com/repos/vdumoulin/fuel/issues{/number}","pulls_url":"https://api.github.com/repos/vdumoulin/fuel/pulls{/number}","milestones_url":"https://api.github.com/repos/vdumoulin/fuel/milestones{/number}","notifications_url":"https://api.github.com/repos/vdumoulin/fuel/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/vdumoulin/fuel/labels{/name}","releases_url":"https://api.github.com/repos/vdumoulin/fuel/releases{/id}","created_at":"2015-02-15T17:25:11Z","updated_at":"2015-05-08T14:41:11Z","pushed_at":"2015-09-16T15:24:34Z","git_url":"git://github.com/vdumoulin/fuel.git","ssh_url":"git@github.com:vdumoulin/fuel.git","clone_url":"https://github.com/vdumoulin/fuel.git","svn_url":"https://github.com/vdumoulin/fuel","homepage":null,"size":1596,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"mila-udem:master","ref":"master","sha":"e35d2bb4a475a36d83bce5139557aa68325c91b4","user":{"login":"mila-udem","id":11724251,"avatar_url":"https://avatars.githubusercontent.com/u/11724251?v=3","gravatar_id":"","url":"https://api.github.com/users/mila-udem","html_url":"https://github.com/mila-udem","followers_url":"https://api.github.com/users/mila-udem/followers","following_url":"https://api.github.com/users/mila-udem/following{/other_user}","gists_url":"https://api.github.com/users/mila-udem/gists{/gist_id}","starred_url":"https://api.github.com/users/mila-udem/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mila-udem/subscriptions","organizations_url":"https://api.github.com/users/mila-udem/orgs","repos_url":"https://api.github.com/users/mila-udem/repos","events_url":"https://api.github.com/users/mila-udem/events{/privacy}","received_events_url":"https://api.github.com/users/mila-udem/received_events","type":"Organization","site_admin":false},"repo":{"id":30722866,"name":"fuel","full_name":"mila-udem/fuel","owner":{"login":"mila-udem","id":11724251,"avatar_url":"https://avatars.githubusercontent.com/u/11724251?v=3","gravatar_id":"","url":"https://api.github.com/users/mila-udem","html_url":"https://github.com/mila-udem","followers_url":"https://api.github.com/users/mila-udem/followers","following_url":"https://api.github.com/users/mila-udem/following{/other_user}","gists_url":"https://api.github.com/users/mila-udem/gists{/gist_id}","starred_url":"https://api.github.com/users/mila-udem/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mila-udem/subscriptions","organizations_url":"https://api.github.com/users/mila-udem/orgs","repos_url":"https://api.github.com/users/mila-udem/repos","events_url":"https://api.github.com/users/mila-udem/events{/privacy}","received_events_url":"https://api.github.com/users/mila-udem/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/mila-udem/fuel","description":"A data pipeline framework for machine learning","fork":false,"url":"https://api.github.com/repos/mila-udem/fuel","forks_url":"https://api.github.com/repos/mila-udem/fuel/forks","keys_url":"https://api.github.com/repos/mila-udem/fuel/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mila-udem/fuel/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mila-udem/fuel/teams","hooks_url":"https://api.github.com/repos/mila-udem/fuel/hooks","issue_events_url":"https://api.github.com/repos/mila-udem/fuel/issues/events{/number}","events_url":"https://api.github.com/repos/mila-udem/fuel/events","assignees_url":"https://api.github.com/repos/mila-udem/fuel/assignees{/user}","branches_url":"https://api.github.com/repos/mila-udem/fuel/branches{/branch}","tags_url":"https://api.github.com/repos/mila-udem/fuel/tags","blobs_url":"https://api.github.com/repos/mila-udem/fuel/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mila-udem/fuel/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mila-udem/fuel/git/refs{/sha}","trees_url":"https://api.github.com/repos/mila-udem/fuel/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mila-udem/fuel/statuses/{sha}","languages_url":"https://api.github.com/repos/mila-udem/fuel/languages","stargazers_url":"https://api.github.com/repos/mila-udem/fuel/stargazers","contributors_url":"https://api.github.com/repos/mila-udem/fuel/contributors","subscribers_url":"https://api.github.com/repos/mila-udem/fuel/subscribers","subscription_url":"https://api.github.com/repos/mila-udem/fuel/subscription","commits_url":"https://api.github.com/repos/mila-udem/fuel/commits{/sha}","git_commits_url":"https://api.github.com/repos/mila-udem/fuel/git/commits{/sha}","comments_url":"https://api.github.com/repos/mila-udem/fuel/comments{/number}","issue_comment_url":"https://api.github.com/repos/mila-udem/fuel/issues/comments{/number}","contents_url":"https://api.github.com/repos/mila-udem/fuel/contents/{+path}","compare_url":"https://api.github.com/repos/mila-udem/fuel/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mila-udem/fuel/merges","archive_url":"https://api.github.com/repos/mila-udem/fuel/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mila-udem/fuel/downloads","issues_url":"https://api.github.com/repos/mila-udem/fuel/issues{/number}","pulls_url":"https://api.github.com/repos/mila-udem/fuel/pulls{/number}","milestones_url":"https://api.github.com/repos/mila-udem/fuel/milestones{/number}","notifications_url":"https://api.github.com/repos/mila-udem/fuel/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mila-udem/fuel/labels{/name}","releases_url":"https://api.github.com/repos/mila-udem/fuel/releases{/id}","created_at":"2015-02-12T20:42:44Z","updated_at":"2015-09-14T20:40:35Z","pushed_at":"2015-09-16T15:24:35Z","git_url":"git://github.com/mila-udem/fuel.git","ssh_url":"git@github.com:mila-udem/fuel.git","clone_url":"https://github.com/mila-udem/fuel.git","svn_url":"https://github.com/mila-udem/fuel","homepage":null,"size":2871,"stargazers_count":174,"watchers_count":174,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":81,"mirror_url":null,"open_issues_count":52,"forks":81,"open_issues":52,"watchers":174,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/mila-udem/fuel/pulls/225"},"html":{"href":"https://github.com/mila-udem/fuel/pull/225"},"issue":{"href":"https://api.github.com/repos/mila-udem/fuel/issues/225"},"comments":{"href":"https://api.github.com/repos/mila-udem/fuel/issues/225/comments"},"review_comments":{"href":"https://api.github.com/repos/mila-udem/fuel/pulls/225/comments"},"review_comment":{"href":"https://api.github.com/repos/mila-udem/fuel/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/mila-udem/fuel/pulls/225/commits"},"statuses":{"href":"https://api.github.com/repos/mila-udem/fuel/statuses/2f71d5bb44c5288832d7546dfee7ec6754c0e990"}}}}