{"action":"created","comment":{"url":"https://api.github.com/repos/mila-udem/blocks/pulls/comments/38557310","id":38557310,"diff_hunk":"@@ -0,0 +1,602 @@\n+\n+Bricks design explained\n+=======================\n+\n+This is a runnable tutorial, which means you can copy its fragments \n+to an IPython notebook and run. The import section follows:\n+\n+    >>> import theano\n+    >>> theano.config.compute_test_value = 'raise'\n+    >>> from theano import tensor\n+\n+Please keep in mind that the code in this tutorial was written for illustration\n+purposes only. Often only the common case is handled and corner cases are\n+ignored.\n+\n+Introduction\n+------------\n+\n+Deep learning (DL) models implementing using Theano share numerous\n+building blocks which become more and more standartized over time.\n+Factoring out such components is a very lucrative idea, and many people\n+have fallen victims to the temptation to develop a library of DL\n+building blocks. The list of rather well-known projects includes\n+`PyLearn2`_, `Groundhog`_, `Lasagne`_, `Keras`_ and of course Blocks. Many people and\n+research groups have smaller and more specialized frameworks.\n+\n+This document contains speculations on how such libraries should be\n+designed. We will start with the simplest approach possible and show\n+step by step what benefits more advanced methods bring. The final goal is\n+to explain the ideas at the core of the respective part of Blocks. \n+In Blocks the components assisting to build the computation\n+graph are called \"bricks\". Bricks are a unified representation for\n+components of all levels: basic ones, like activation functions, and\n+complex ones, like attention mechanism and a generic sequence generator.\n+Reading this document should answer many questions that people\n+often ask about bricks, such as \"why so many decorators\", \"why is not my\n+brick initialized in the constructor\" and others, which are asked not so\n+often, but still daunt people' minds.\n+\n+.. _PyLearn2: https://github.com/lisa-lab/pylearn2\n+.. _Groundhog: https://github.com/lisa-groundhog/GroundHog\n+.. _Lasagne: https://github.com/Lasagne/Lasagne\n+.. _Keras: https://github.com/fchollet/keras\n+\n+Design for basic components\n+---------------------------\n+\n+The toy task\n+~~~~~~~~~~~~\n+\n+To make our discussion less abstract, let us consider an example. D,\n+a novice in deep learning, has learnt about fully-connected feed-forward\n+and recurrent networks (FFN and RNN respectively). He plans to use them\n+in a number of projects, so he decides to start with coding a reference\n+implementation first. Few hours of hard work and here it is:\n+\n+    >>> # D's FFN, D2L2v0.1\n+    >>> inputs = tensor.matrix('x')\n+    >>> inputs.tag.test_value = numpy.random.uniform(size=(10, 754))\n+    >>> dims = [754, 100, 100, 10]\n+    >>> activations = [tensor.tanh, tensor.tanh, tensor.nnet.softmax]\n+    >>> weight_matrices = []\n+    >>> bias_vectors = []\n+    >>> for i in range(len(dims) - 1):\n+    >>>     weight_matrices.append(theano.shared(numpy.random.uniform(\n+    >>>         size=(dims[i], dims[i + 1]))))\n+    >>>     bias_vectors.append(theano.shared(numpy.zeros(dims[i + 1])))\n+    >>> outputs = inputs\n+    >>> for dim, activation, weight_matrix, bias_vector in zip(\n+    >>>         dims[1:], activations, weight_matrices, bias_vectors):\n+    >>>     outputs = activation(outputs.dot(weight_matrix) + bias_vector)\n+    >>> \n+    >>> # D's RNN, D2L2v0.1","path":"docs/bricks_design.rst","position":73,"original_position":73,"commit_id":"39cb8153b6833e2d70fc21bd8c303a9644b85691","original_commit_id":"39cb8153b6833e2d70fc21bd8c303a9644b85691","user":{"login":"fvisin","id":4639261,"avatar_url":"https://avatars.githubusercontent.com/u/4639261?v=3","gravatar_id":"","url":"https://api.github.com/users/fvisin","html_url":"https://github.com/fvisin","followers_url":"https://api.github.com/users/fvisin/followers","following_url":"https://api.github.com/users/fvisin/following{/other_user}","gists_url":"https://api.github.com/users/fvisin/gists{/gist_id}","starred_url":"https://api.github.com/users/fvisin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fvisin/subscriptions","organizations_url":"https://api.github.com/users/fvisin/orgs","repos_url":"https://api.github.com/users/fvisin/repos","events_url":"https://api.github.com/users/fvisin/events{/privacy}","received_events_url":"https://api.github.com/users/fvisin/received_events","type":"User","site_admin":false},"body":"Dicto","created_at":"2015-09-02T17:02:32Z","updated_at":"2015-09-02T17:02:32Z","html_url":"https://github.com/mila-udem/blocks/pull/822#discussion_r38557310","pull_request_url":"https://api.github.com/repos/mila-udem/blocks/pulls/822","_links":{"self":{"href":"https://api.github.com/repos/mila-udem/blocks/pulls/comments/38557310"},"html":{"href":"https://github.com/mila-udem/blocks/pull/822#discussion_r38557310"},"pull_request":{"href":"https://api.github.com/repos/mila-udem/blocks/pulls/822"}}},"pull_request":{"url":"https://api.github.com/repos/mila-udem/blocks/pulls/822","id":43893334,"html_url":"https://github.com/mila-udem/blocks/pull/822","diff_url":"https://github.com/mila-udem/blocks/pull/822.diff","patch_url":"https://github.com/mila-udem/blocks/pull/822.patch","issue_url":"https://api.github.com/repos/mila-udem/blocks/issues/822","number":822,"state":"open","locked":false,"title":"Bricks design docs","user":{"login":"rizar","id":654434,"avatar_url":"https://avatars.githubusercontent.com/u/654434?v=3","gravatar_id":"","url":"https://api.github.com/users/rizar","html_url":"https://github.com/rizar","followers_url":"https://api.github.com/users/rizar/followers","following_url":"https://api.github.com/users/rizar/following{/other_user}","gists_url":"https://api.github.com/users/rizar/gists{/gist_id}","starred_url":"https://api.github.com/users/rizar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rizar/subscriptions","organizations_url":"https://api.github.com/users/rizar/orgs","repos_url":"https://api.github.com/users/rizar/repos","events_url":"https://api.github.com/users/rizar/events{/privacy}","received_events_url":"https://api.github.com/users/rizar/received_events","type":"User","site_admin":false},"body":"I am trying to write docs explains the design decision behind bricks (since this seems to be the most controversial part of Blocks so far). The approach I have taken is to develop something like bricks and explain every step on the way. My current progress is in this PR. I am especially interested in general feedback: do you think it is a good way to write docs? Is the tutorial overloaded with copy-pasted code? Do I expect too much from the reader or maybe on the opposite, am I explaining obvious things?\r\n\r\n@bartvm , @dmitriy-serdyuk , @dwf, @vdumoulin , @fvisin , @janchorowski ","created_at":"2015-09-01T15:56:49Z","updated_at":"2015-09-02T17:02:32Z","closed_at":null,"merged_at":null,"merge_commit_sha":"eaccf4202762da9ed2a613954f47f35fbc1a147e","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/mila-udem/blocks/pulls/822/commits","review_comments_url":"https://api.github.com/repos/mila-udem/blocks/pulls/822/comments","review_comment_url":"https://api.github.com/repos/mila-udem/blocks/pulls/comments{/number}","comments_url":"https://api.github.com/repos/mila-udem/blocks/issues/822/comments","statuses_url":"https://api.github.com/repos/mila-udem/blocks/statuses/39cb8153b6833e2d70fc21bd8c303a9644b85691","head":{"label":"rizar:bricks_design_docs","ref":"bricks_design_docs","sha":"39cb8153b6833e2d70fc21bd8c303a9644b85691","user":{"login":"rizar","id":654434,"avatar_url":"https://avatars.githubusercontent.com/u/654434?v=3","gravatar_id":"","url":"https://api.github.com/users/rizar","html_url":"https://github.com/rizar","followers_url":"https://api.github.com/users/rizar/followers","following_url":"https://api.github.com/users/rizar/following{/other_user}","gists_url":"https://api.github.com/users/rizar/gists{/gist_id}","starred_url":"https://api.github.com/users/rizar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rizar/subscriptions","organizations_url":"https://api.github.com/users/rizar/orgs","repos_url":"https://api.github.com/users/rizar/repos","events_url":"https://api.github.com/users/rizar/events{/privacy}","received_events_url":"https://api.github.com/users/rizar/received_events","type":"User","site_admin":false},"repo":{"id":24884672,"name":"blocks","full_name":"rizar/blocks","owner":{"login":"rizar","id":654434,"avatar_url":"https://avatars.githubusercontent.com/u/654434?v=3","gravatar_id":"","url":"https://api.github.com/users/rizar","html_url":"https://github.com/rizar","followers_url":"https://api.github.com/users/rizar/followers","following_url":"https://api.github.com/users/rizar/following{/other_user}","gists_url":"https://api.github.com/users/rizar/gists{/gist_id}","starred_url":"https://api.github.com/users/rizar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rizar/subscriptions","organizations_url":"https://api.github.com/users/rizar/orgs","repos_url":"https://api.github.com/users/rizar/repos","events_url":"https://api.github.com/users/rizar/events{/privacy}","received_events_url":"https://api.github.com/users/rizar/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/rizar/blocks","description":"","fork":true,"url":"https://api.github.com/repos/rizar/blocks","forks_url":"https://api.github.com/repos/rizar/blocks/forks","keys_url":"https://api.github.com/repos/rizar/blocks/keys{/key_id}","collaborators_url":"https://api.github.com/repos/rizar/blocks/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/rizar/blocks/teams","hooks_url":"https://api.github.com/repos/rizar/blocks/hooks","issue_events_url":"https://api.github.com/repos/rizar/blocks/issues/events{/number}","events_url":"https://api.github.com/repos/rizar/blocks/events","assignees_url":"https://api.github.com/repos/rizar/blocks/assignees{/user}","branches_url":"https://api.github.com/repos/rizar/blocks/branches{/branch}","tags_url":"https://api.github.com/repos/rizar/blocks/tags","blobs_url":"https://api.github.com/repos/rizar/blocks/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/rizar/blocks/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/rizar/blocks/git/refs{/sha}","trees_url":"https://api.github.com/repos/rizar/blocks/git/trees{/sha}","statuses_url":"https://api.github.com/repos/rizar/blocks/statuses/{sha}","languages_url":"https://api.github.com/repos/rizar/blocks/languages","stargazers_url":"https://api.github.com/repos/rizar/blocks/stargazers","contributors_url":"https://api.github.com/repos/rizar/blocks/contributors","subscribers_url":"https://api.github.com/repos/rizar/blocks/subscribers","subscription_url":"https://api.github.com/repos/rizar/blocks/subscription","commits_url":"https://api.github.com/repos/rizar/blocks/commits{/sha}","git_commits_url":"https://api.github.com/repos/rizar/blocks/git/commits{/sha}","comments_url":"https://api.github.com/repos/rizar/blocks/comments{/number}","issue_comment_url":"https://api.github.com/repos/rizar/blocks/issues/comments{/number}","contents_url":"https://api.github.com/repos/rizar/blocks/contents/{+path}","compare_url":"https://api.github.com/repos/rizar/blocks/compare/{base}...{head}","merges_url":"https://api.github.com/repos/rizar/blocks/merges","archive_url":"https://api.github.com/repos/rizar/blocks/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/rizar/blocks/downloads","issues_url":"https://api.github.com/repos/rizar/blocks/issues{/number}","pulls_url":"https://api.github.com/repos/rizar/blocks/pulls{/number}","milestones_url":"https://api.github.com/repos/rizar/blocks/milestones{/number}","notifications_url":"https://api.github.com/repos/rizar/blocks/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/rizar/blocks/labels{/name}","releases_url":"https://api.github.com/repos/rizar/blocks/releases{/id}","created_at":"2014-10-07T09:45:14Z","updated_at":"2015-04-16T13:34:58Z","pushed_at":"2015-09-01T17:25:15Z","git_url":"git://github.com/rizar/blocks.git","ssh_url":"git@github.com:rizar/blocks.git","clone_url":"https://github.com/rizar/blocks.git","svn_url":"https://github.com/rizar/blocks","homepage":null,"size":4591,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"mila-udem:master","ref":"master","sha":"0f6b4e09ee602e2b0ee8504bbdd6bbffdb749410","user":{"login":"mila-udem","id":11724251,"avatar_url":"https://avatars.githubusercontent.com/u/11724251?v=3","gravatar_id":"","url":"https://api.github.com/users/mila-udem","html_url":"https://github.com/mila-udem","followers_url":"https://api.github.com/users/mila-udem/followers","following_url":"https://api.github.com/users/mila-udem/following{/other_user}","gists_url":"https://api.github.com/users/mila-udem/gists{/gist_id}","starred_url":"https://api.github.com/users/mila-udem/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mila-udem/subscriptions","organizations_url":"https://api.github.com/users/mila-udem/orgs","repos_url":"https://api.github.com/users/mila-udem/repos","events_url":"https://api.github.com/users/mila-udem/events{/privacy}","received_events_url":"https://api.github.com/users/mila-udem/received_events","type":"Organization","site_admin":false},"repo":{"id":24829371,"name":"blocks","full_name":"mila-udem/blocks","owner":{"login":"mila-udem","id":11724251,"avatar_url":"https://avatars.githubusercontent.com/u/11724251?v=3","gravatar_id":"","url":"https://api.github.com/users/mila-udem","html_url":"https://github.com/mila-udem","followers_url":"https://api.github.com/users/mila-udem/followers","following_url":"https://api.github.com/users/mila-udem/following{/other_user}","gists_url":"https://api.github.com/users/mila-udem/gists{/gist_id}","starred_url":"https://api.github.com/users/mila-udem/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mila-udem/subscriptions","organizations_url":"https://api.github.com/users/mila-udem/orgs","repos_url":"https://api.github.com/users/mila-udem/repos","events_url":"https://api.github.com/users/mila-udem/events{/privacy}","received_events_url":"https://api.github.com/users/mila-udem/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/mila-udem/blocks","description":"A Theano framework for building and training neural networks","fork":false,"url":"https://api.github.com/repos/mila-udem/blocks","forks_url":"https://api.github.com/repos/mila-udem/blocks/forks","keys_url":"https://api.github.com/repos/mila-udem/blocks/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mila-udem/blocks/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mila-udem/blocks/teams","hooks_url":"https://api.github.com/repos/mila-udem/blocks/hooks","issue_events_url":"https://api.github.com/repos/mila-udem/blocks/issues/events{/number}","events_url":"https://api.github.com/repos/mila-udem/blocks/events","assignees_url":"https://api.github.com/repos/mila-udem/blocks/assignees{/user}","branches_url":"https://api.github.com/repos/mila-udem/blocks/branches{/branch}","tags_url":"https://api.github.com/repos/mila-udem/blocks/tags","blobs_url":"https://api.github.com/repos/mila-udem/blocks/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mila-udem/blocks/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mila-udem/blocks/git/refs{/sha}","trees_url":"https://api.github.com/repos/mila-udem/blocks/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mila-udem/blocks/statuses/{sha}","languages_url":"https://api.github.com/repos/mila-udem/blocks/languages","stargazers_url":"https://api.github.com/repos/mila-udem/blocks/stargazers","contributors_url":"https://api.github.com/repos/mila-udem/blocks/contributors","subscribers_url":"https://api.github.com/repos/mila-udem/blocks/subscribers","subscription_url":"https://api.github.com/repos/mila-udem/blocks/subscription","commits_url":"https://api.github.com/repos/mila-udem/blocks/commits{/sha}","git_commits_url":"https://api.github.com/repos/mila-udem/blocks/git/commits{/sha}","comments_url":"https://api.github.com/repos/mila-udem/blocks/comments{/number}","issue_comment_url":"https://api.github.com/repos/mila-udem/blocks/issues/comments{/number}","contents_url":"https://api.github.com/repos/mila-udem/blocks/contents/{+path}","compare_url":"https://api.github.com/repos/mila-udem/blocks/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mila-udem/blocks/merges","archive_url":"https://api.github.com/repos/mila-udem/blocks/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mila-udem/blocks/downloads","issues_url":"https://api.github.com/repos/mila-udem/blocks/issues{/number}","pulls_url":"https://api.github.com/repos/mila-udem/blocks/pulls{/number}","milestones_url":"https://api.github.com/repos/mila-udem/blocks/milestones{/number}","notifications_url":"https://api.github.com/repos/mila-udem/blocks/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mila-udem/blocks/labels{/name}","releases_url":"https://api.github.com/repos/mila-udem/blocks/releases{/id}","created_at":"2014-10-06T00:08:32Z","updated_at":"2015-09-02T16:28:55Z","pushed_at":"2015-09-01T23:59:58Z","git_url":"git://github.com/mila-udem/blocks.git","ssh_url":"git@github.com:mila-udem/blocks.git","clone_url":"https://github.com/mila-udem/blocks.git","svn_url":"https://github.com/mila-udem/blocks","homepage":"","size":29634,"stargazers_count":439,"watchers_count":439,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":129,"mirror_url":null,"open_issues_count":126,"forks":129,"open_issues":126,"watchers":439,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/mila-udem/blocks/pulls/822"},"html":{"href":"https://github.com/mila-udem/blocks/pull/822"},"issue":{"href":"https://api.github.com/repos/mila-udem/blocks/issues/822"},"comments":{"href":"https://api.github.com/repos/mila-udem/blocks/issues/822/comments"},"review_comments":{"href":"https://api.github.com/repos/mila-udem/blocks/pulls/822/comments"},"review_comment":{"href":"https://api.github.com/repos/mila-udem/blocks/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/mila-udem/blocks/pulls/822/commits"},"statuses":{"href":"https://api.github.com/repos/mila-udem/blocks/statuses/39cb8153b6833e2d70fc21bd8c303a9644b85691"}}}}