{"action":"created","comment":{"url":"https://api.github.com/repos/Theano/Theano/pulls/comments/27448172","id":27448172,"diff_hunk":"@@ -721,3 +725,120 @@ def c_support_code_apply(self, node, nodename):\n         return ret1 + \"\\n\" + ret2\n \n gpu_softmax_with_bias = GpuSoftmaxWithBias()\n+\n+\n+def hierarchical_softmax(W1, b1, W2, b2, x, n_outputs, target=None):\n+    \"\"\"\n+    GPU-only function that returns the outputs of a two-level hierarchical\n+    softmax.\n+    In the two-level hierarchical softmax architecture, outputs are grouped\n+    in sqrt(n_outputs) classes.\n+    There are two softmax layers. The first predicts the class of the input x\n+    while the second predicts the output of the input x in the predicted class.\n+    More explanations can be found in the original paper\n+    http://arxiv.org/abs/cs/0108006.\n+    If target is specified, it will only compute the outputs of the\n+    corresponding targets. Otherwise, if target is None, it will compute all\n+    the outputs.\n+\n+    :type W1: a Theano shared variable of shape (number of features of the\n+        input x, number of classes)\n+    :param W1: the weight matrix of the first softmax, which maps the\n+        input x to the probabilities of the classes.\n+\n+    :type b1: a Theano shared variable of shape (number of classes,)\n+    :param b1: the bias vector of the first softmax layer.\n+\n+    :type W2: a Theano shared variable of shape (number of classes, number of\n+        features of the input x, number of outputs per class)\n+    :param W2: the weight matrix of the second softmax, which maps the input\n+        x to the probabilities of the outputs.\n+\n+    :type b2: a Theano shared variable of shape (number of classes, number of\n+        outputs per class)\n+    :param b2: the bias vector of the second softmax layer.\n+\n+    :type x: Theano tensor variable of shape (batch_size, number of features)\n+    :param x: the minibatch input of the multi-class softmax.\n+\n+    :type n_outputs: int\n+    :param n_outputs: the number of outputs.\n+\n+    :type target: a Theano variable of shape either (batch_size,) or\n+        (batch_size, 1)\n+    :param target: Contains the indices of the targets for the minibatch\n+        input x. For each input, the function computes the output for its\n+        corresponding target. If target is None, then all the outputs are\n+        computed for each input.\n+\n+    :rtype: Theano tensor variable of the following shape:\n+        - (batch_size, 1) if target is not None\n+        - (batch_size, n_out) if target is None\n+    :returns: the minibatch output of the 2-layer hierarchical softmax.\n+\n+    :note: If n_outputs is not a square number, it will still work but the\n+        user has to specify the following shapes of its weight/bias matrices:\n+        - number of outputs per class: n_outputs_per_class = ceil(sqrt(\n+          n_outputs)\n+        - number of classes: n_classes = ceil(n_outputs / n_outputs_per_class)\n+\n+    \"\"\"\n+\n+    if theano.config.device != 'gpu':\n+        raise Exception('The multiclass softmax only works on gpu.')\n+\n+    batch_size = x.shape[0]\n+\n+    n_classes, n_outputs_per_class = b2.get_value(True, True).shape\n+\n+    # First softmax which computes the probabilities of belonging to each class\n+    class_probs = softmax(T.dot(x, W1) + b1)\n+\n+    if target is None:\n+        # Computes the probabilites of all the outputs\n+\n+        class_ids = T.arange(n_classes)\n+\n+        def _compute_output_probs(class_id):\n+            # Second softmax that computes the probabilities of the outputs\n+            # in a given class indexed by class_id\n+            output_prob = softmax(\n+                T.dot(x, W2[class_id, :, :]) + b2[class_id, :])","path":"theano/sandbox/cuda/nnet.py","position":92,"original_position":92,"commit_id":"092da29c9237d1e6b416b478f15809d47fd4814b","original_commit_id":"092da29c9237d1e6b416b478f15809d47fd4814b","user":{"login":"bouthilx","id":2099418,"avatar_url":"https://avatars.githubusercontent.com/u/2099418?v=3","gravatar_id":"","url":"https://api.github.com/users/bouthilx","html_url":"https://github.com/bouthilx","followers_url":"https://api.github.com/users/bouthilx/followers","following_url":"https://api.github.com/users/bouthilx/following{/other_user}","gists_url":"https://api.github.com/users/bouthilx/gists{/gist_id}","starred_url":"https://api.github.com/users/bouthilx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bouthilx/subscriptions","organizations_url":"https://api.github.com/users/bouthilx/orgs","repos_url":"https://api.github.com/users/bouthilx/repos","events_url":"https://api.github.com/users/bouthilx/events{/privacy}","received_events_url":"https://api.github.com/users/bouthilx/received_events","type":"User","site_admin":false},"body":"Looks more like a tensordot. The scan seems to do something like \r\n\r\n```python \r\noutput_probs = (T.dot(x, W2) + b2) * class_probs[:,:,None]\r\noutput_probs = output_probs.reshape((batch_size, n_classes*n_outputs_per_class))[:,:n_outputs]\r\n```","created_at":"2015-03-31T02:05:28Z","updated_at":"2015-03-31T02:05:28Z","html_url":"https://github.com/Theano/Theano/pull/2693#discussion_r27448172","pull_request_url":"https://api.github.com/repos/Theano/Theano/pulls/2693","_links":{"self":{"href":"https://api.github.com/repos/Theano/Theano/pulls/comments/27448172"},"html":{"href":"https://github.com/Theano/Theano/pull/2693#discussion_r27448172"},"pull_request":{"href":"https://api.github.com/repos/Theano/Theano/pulls/2693"}}},"pull_request":{"url":"https://api.github.com/repos/Theano/Theano/pulls/2693","id":32134223,"html_url":"https://github.com/Theano/Theano/pull/2693","diff_url":"https://github.com/Theano/Theano/pull/2693.diff","patch_url":"https://github.com/Theano/Theano/pull/2693.patch","issue_url":"https://api.github.com/repos/Theano/Theano/issues/2693","number":2693,"state":"open","locked":false,"title":"GPU version of a 2-layer hierarchical softmax (ccw)","user":{"login":"adbrebs","id":6939554,"avatar_url":"https://avatars.githubusercontent.com/u/6939554?v=3","gravatar_id":"","url":"https://api.github.com/users/adbrebs","html_url":"https://github.com/adbrebs","followers_url":"https://api.github.com/users/adbrebs/followers","following_url":"https://api.github.com/users/adbrebs/following{/other_user}","gists_url":"https://api.github.com/users/adbrebs/gists{/gist_id}","starred_url":"https://api.github.com/users/adbrebs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adbrebs/subscriptions","organizations_url":"https://api.github.com/users/adbrebs/orgs","repos_url":"https://api.github.com/users/adbrebs/repos","events_url":"https://api.github.com/users/adbrebs/events{/privacy}","received_events_url":"https://api.github.com/users/adbrebs/received_events","type":"User","site_admin":false},"body":"","created_at":"2015-03-27T18:44:22Z","updated_at":"2015-03-31T02:05:28Z","closed_at":null,"merged_at":null,"merge_commit_sha":"8708aea573684b657d8daf9f91040c01ce9e8214","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/Theano/Theano/pulls/2693/commits","review_comments_url":"https://api.github.com/repos/Theano/Theano/pulls/2693/comments","review_comment_url":"https://api.github.com/repos/Theano/Theano/pulls/comments{/number}","comments_url":"https://api.github.com/repos/Theano/Theano/issues/2693/comments","statuses_url":"https://api.github.com/repos/Theano/Theano/statuses/092da29c9237d1e6b416b478f15809d47fd4814b","head":{"label":"adbrebs:h_softmax","ref":"h_softmax","sha":"092da29c9237d1e6b416b478f15809d47fd4814b","user":{"login":"adbrebs","id":6939554,"avatar_url":"https://avatars.githubusercontent.com/u/6939554?v=3","gravatar_id":"","url":"https://api.github.com/users/adbrebs","html_url":"https://github.com/adbrebs","followers_url":"https://api.github.com/users/adbrebs/followers","following_url":"https://api.github.com/users/adbrebs/following{/other_user}","gists_url":"https://api.github.com/users/adbrebs/gists{/gist_id}","starred_url":"https://api.github.com/users/adbrebs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adbrebs/subscriptions","organizations_url":"https://api.github.com/users/adbrebs/orgs","repos_url":"https://api.github.com/users/adbrebs/repos","events_url":"https://api.github.com/users/adbrebs/events{/privacy}","received_events_url":"https://api.github.com/users/adbrebs/received_events","type":"User","site_admin":false},"repo":{"id":29032332,"name":"Theano","full_name":"adbrebs/Theano","owner":{"login":"adbrebs","id":6939554,"avatar_url":"https://avatars.githubusercontent.com/u/6939554?v=3","gravatar_id":"","url":"https://api.github.com/users/adbrebs","html_url":"https://github.com/adbrebs","followers_url":"https://api.github.com/users/adbrebs/followers","following_url":"https://api.github.com/users/adbrebs/following{/other_user}","gists_url":"https://api.github.com/users/adbrebs/gists{/gist_id}","starred_url":"https://api.github.com/users/adbrebs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adbrebs/subscriptions","organizations_url":"https://api.github.com/users/adbrebs/orgs","repos_url":"https://api.github.com/users/adbrebs/repos","events_url":"https://api.github.com/users/adbrebs/events{/privacy}","received_events_url":"https://api.github.com/users/adbrebs/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/adbrebs/Theano","description":"Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. It can use GPUs and perform efficient symbolic differentiation.","fork":true,"url":"https://api.github.com/repos/adbrebs/Theano","forks_url":"https://api.github.com/repos/adbrebs/Theano/forks","keys_url":"https://api.github.com/repos/adbrebs/Theano/keys{/key_id}","collaborators_url":"https://api.github.com/repos/adbrebs/Theano/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/adbrebs/Theano/teams","hooks_url":"https://api.github.com/repos/adbrebs/Theano/hooks","issue_events_url":"https://api.github.com/repos/adbrebs/Theano/issues/events{/number}","events_url":"https://api.github.com/repos/adbrebs/Theano/events","assignees_url":"https://api.github.com/repos/adbrebs/Theano/assignees{/user}","branches_url":"https://api.github.com/repos/adbrebs/Theano/branches{/branch}","tags_url":"https://api.github.com/repos/adbrebs/Theano/tags","blobs_url":"https://api.github.com/repos/adbrebs/Theano/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/adbrebs/Theano/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/adbrebs/Theano/git/refs{/sha}","trees_url":"https://api.github.com/repos/adbrebs/Theano/git/trees{/sha}","statuses_url":"https://api.github.com/repos/adbrebs/Theano/statuses/{sha}","languages_url":"https://api.github.com/repos/adbrebs/Theano/languages","stargazers_url":"https://api.github.com/repos/adbrebs/Theano/stargazers","contributors_url":"https://api.github.com/repos/adbrebs/Theano/contributors","subscribers_url":"https://api.github.com/repos/adbrebs/Theano/subscribers","subscription_url":"https://api.github.com/repos/adbrebs/Theano/subscription","commits_url":"https://api.github.com/repos/adbrebs/Theano/commits{/sha}","git_commits_url":"https://api.github.com/repos/adbrebs/Theano/git/commits{/sha}","comments_url":"https://api.github.com/repos/adbrebs/Theano/comments{/number}","issue_comment_url":"https://api.github.com/repos/adbrebs/Theano/issues/comments{/number}","contents_url":"https://api.github.com/repos/adbrebs/Theano/contents/{+path}","compare_url":"https://api.github.com/repos/adbrebs/Theano/compare/{base}...{head}","merges_url":"https://api.github.com/repos/adbrebs/Theano/merges","archive_url":"https://api.github.com/repos/adbrebs/Theano/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/adbrebs/Theano/downloads","issues_url":"https://api.github.com/repos/adbrebs/Theano/issues{/number}","pulls_url":"https://api.github.com/repos/adbrebs/Theano/pulls{/number}","milestones_url":"https://api.github.com/repos/adbrebs/Theano/milestones{/number}","notifications_url":"https://api.github.com/repos/adbrebs/Theano/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/adbrebs/Theano/labels{/name}","releases_url":"https://api.github.com/repos/adbrebs/Theano/releases{/id}","created_at":"2015-01-09T19:45:48Z","updated_at":"2015-03-27T18:40:11Z","pushed_at":"2015-03-27T18:50:06Z","git_url":"git://github.com/adbrebs/Theano.git","ssh_url":"git@github.com:adbrebs/Theano.git","clone_url":"https://github.com/adbrebs/Theano.git","svn_url":"https://github.com/adbrebs/Theano","homepage":"http://www.deeplearning.net/software/theano","size":27554,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"Theano:master","ref":"master","sha":"c2efab91dbceddf05e1d113539fdc260a24185c7","user":{"login":"Theano","id":970517,"avatar_url":"https://avatars.githubusercontent.com/u/970517?v=3","gravatar_id":"","url":"https://api.github.com/users/Theano","html_url":"https://github.com/Theano","followers_url":"https://api.github.com/users/Theano/followers","following_url":"https://api.github.com/users/Theano/following{/other_user}","gists_url":"https://api.github.com/users/Theano/gists{/gist_id}","starred_url":"https://api.github.com/users/Theano/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Theano/subscriptions","organizations_url":"https://api.github.com/users/Theano/orgs","repos_url":"https://api.github.com/users/Theano/repos","events_url":"https://api.github.com/users/Theano/events{/privacy}","received_events_url":"https://api.github.com/users/Theano/received_events","type":"Organization","site_admin":false},"repo":{"id":2183193,"name":"Theano","full_name":"Theano/Theano","owner":{"login":"Theano","id":970517,"avatar_url":"https://avatars.githubusercontent.com/u/970517?v=3","gravatar_id":"","url":"https://api.github.com/users/Theano","html_url":"https://github.com/Theano","followers_url":"https://api.github.com/users/Theano/followers","following_url":"https://api.github.com/users/Theano/following{/other_user}","gists_url":"https://api.github.com/users/Theano/gists{/gist_id}","starred_url":"https://api.github.com/users/Theano/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Theano/subscriptions","organizations_url":"https://api.github.com/users/Theano/orgs","repos_url":"https://api.github.com/users/Theano/repos","events_url":"https://api.github.com/users/Theano/events{/privacy}","received_events_url":"https://api.github.com/users/Theano/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Theano/Theano","description":"Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. It can use GPUs and perform efficient symbolic differentiation.","fork":false,"url":"https://api.github.com/repos/Theano/Theano","forks_url":"https://api.github.com/repos/Theano/Theano/forks","keys_url":"https://api.github.com/repos/Theano/Theano/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Theano/Theano/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Theano/Theano/teams","hooks_url":"https://api.github.com/repos/Theano/Theano/hooks","issue_events_url":"https://api.github.com/repos/Theano/Theano/issues/events{/number}","events_url":"https://api.github.com/repos/Theano/Theano/events","assignees_url":"https://api.github.com/repos/Theano/Theano/assignees{/user}","branches_url":"https://api.github.com/repos/Theano/Theano/branches{/branch}","tags_url":"https://api.github.com/repos/Theano/Theano/tags","blobs_url":"https://api.github.com/repos/Theano/Theano/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Theano/Theano/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Theano/Theano/git/refs{/sha}","trees_url":"https://api.github.com/repos/Theano/Theano/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Theano/Theano/statuses/{sha}","languages_url":"https://api.github.com/repos/Theano/Theano/languages","stargazers_url":"https://api.github.com/repos/Theano/Theano/stargazers","contributors_url":"https://api.github.com/repos/Theano/Theano/contributors","subscribers_url":"https://api.github.com/repos/Theano/Theano/subscribers","subscription_url":"https://api.github.com/repos/Theano/Theano/subscription","commits_url":"https://api.github.com/repos/Theano/Theano/commits{/sha}","git_commits_url":"https://api.github.com/repos/Theano/Theano/git/commits{/sha}","comments_url":"https://api.github.com/repos/Theano/Theano/comments{/number}","issue_comment_url":"https://api.github.com/repos/Theano/Theano/issues/comments{/number}","contents_url":"https://api.github.com/repos/Theano/Theano/contents/{+path}","compare_url":"https://api.github.com/repos/Theano/Theano/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Theano/Theano/merges","archive_url":"https://api.github.com/repos/Theano/Theano/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Theano/Theano/downloads","issues_url":"https://api.github.com/repos/Theano/Theano/issues{/number}","pulls_url":"https://api.github.com/repos/Theano/Theano/pulls{/number}","milestones_url":"https://api.github.com/repos/Theano/Theano/milestones{/number}","notifications_url":"https://api.github.com/repos/Theano/Theano/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Theano/Theano/labels{/name}","releases_url":"https://api.github.com/repos/Theano/Theano/releases{/id}","created_at":"2011-08-10T03:48:06Z","updated_at":"2015-03-30T23:10:34Z","pushed_at":"2015-03-31T01:23:59Z","git_url":"git://github.com/Theano/Theano.git","ssh_url":"git@github.com:Theano/Theano.git","clone_url":"https://github.com/Theano/Theano.git","svn_url":"https://github.com/Theano/Theano","homepage":"http://www.deeplearning.net/software/theano","size":41547,"stargazers_count":1325,"watchers_count":1325,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":487,"mirror_url":null,"open_issues_count":333,"forks":487,"open_issues":333,"watchers":1325,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/Theano/Theano/pulls/2693"},"html":{"href":"https://github.com/Theano/Theano/pull/2693"},"issue":{"href":"https://api.github.com/repos/Theano/Theano/issues/2693"},"comments":{"href":"https://api.github.com/repos/Theano/Theano/issues/2693/comments"},"review_comments":{"href":"https://api.github.com/repos/Theano/Theano/pulls/2693/comments"},"review_comment":{"href":"https://api.github.com/repos/Theano/Theano/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/Theano/Theano/pulls/2693/commits"},"statuses":{"href":"https://api.github.com/repos/Theano/Theano/statuses/092da29c9237d1e6b416b478f15809d47fd4814b"}}}}