{"action":"created","comment":{"url":"https://api.github.com/repos/stan-dev/stan/pulls/comments/37995372","id":37995372,"diff_hunk":"@@ -150,18 +150,143 @@ namespace stan {\n       }\n \n       /**\n-       * Runs stochastic gradient ascent with Adagrad.\n+       * Adaptively set hyperparameters for ADVI.\n+       *\n+       * @param variational variational distribution\n+       * @param tuning_iter number of tuning iterations\n+       * @return optimal value of eta in a coarse grid\n+       */\n+      double tune(Q& variational, int tuning_iter) const {\n+        static const char* function =\n+          \"stan::variational::advi::tune\";\n+\n+        // Gradient parameters\n+        Q elbo_grad = Q(model_.num_params_r());\n+\n+        // Learning rate parameters\n+        Q params_prop = Q(model_.num_params_r());\n+        double tau = 1.0;\n+        double pre_factor  = 0.9;\n+        double post_factor = 0.1;\n+        double eta_scaled;\n+\n+        // Sequence of eta values to try during tuning\n+        std::queue<double> eta_sequence;\n+        eta_sequence.push(1.00);\n+        eta_sequence.push(0.50);\n+        eta_sequence.push(0.10);\n+        eta_sequence.push(0.05);\n+        eta_sequence.push(0.01);\n+\n+        // Print progress\n+        int eta_sequence_size = eta_sequence.size();\n+        int m;\n+        if (print_stream_) {\n+          *print_stream_ << \"Begin hyperparameter tuning.\" << std::endl;\n+        }\n+\n+        // Initialize ELBO and convergence tracking variables\n+        double elbo(0.0);\n+        double elbo_best = -std::numeric_limits<double>::max();\n+        double elbo_init = calc_ELBO(variational);\n+        double eta;\n+        double eta_best;\n+\n+        int iter_tune;\n+        bool do_more_tuning = true;\n+        while (do_more_tuning) {\n+          // Try next eta\n+          eta = eta_sequence.front();\n+          eta_sequence.pop();\n+\n+          for (iter_tune = 1; iter_tune <= tuning_iter; ++iter_tune) {\n+            m = (eta_sequence_size - eta_sequence.size() - 1) *\n+              tuning_iter + iter_tune; // # of total tuning iterations\n+            stan::services::variational::print_progress(\n+              m, 0, tuning_iter*eta_sequence_size,\n+              tuning_iter, true, \"\", \"\", *print_stream_);\n+\n+            // Compute gradient of ELBO\n+            calc_ELBO_grad(variational, elbo_grad);","path":"src/stan/variational/advi.hpp","position":132,"original_position":132,"commit_id":"b678c066cb579af32a688f9f566c6e50b9835567","original_commit_id":"b678c066cb579af32a688f9f566c6e50b9835567","user":{"login":"akucukelbir","id":329960,"avatar_url":"https://avatars.githubusercontent.com/u/329960?v=3","gravatar_id":"","url":"https://api.github.com/users/akucukelbir","html_url":"https://github.com/akucukelbir","followers_url":"https://api.github.com/users/akucukelbir/followers","following_url":"https://api.github.com/users/akucukelbir/following{/other_user}","gists_url":"https://api.github.com/users/akucukelbir/gists{/gist_id}","starred_url":"https://api.github.com/users/akucukelbir/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/akucukelbir/subscriptions","organizations_url":"https://api.github.com/users/akucukelbir/orgs","repos_url":"https://api.github.com/users/akucukelbir/repos","events_url":"https://api.github.com/users/akucukelbir/events{/privacy}","received_events_url":"https://api.github.com/users/akucukelbir/received_events","type":"User","site_admin":false},"body":"roger.","created_at":"2015-08-26T15:23:52Z","updated_at":"2015-08-26T15:23:52Z","html_url":"https://github.com/stan-dev/stan/pull/1590#discussion_r37995372","pull_request_url":"https://api.github.com/repos/stan-dev/stan/pulls/1590","_links":{"self":{"href":"https://api.github.com/repos/stan-dev/stan/pulls/comments/37995372"},"html":{"href":"https://github.com/stan-dev/stan/pull/1590#discussion_r37995372"},"pull_request":{"href":"https://api.github.com/repos/stan-dev/stan/pulls/1590"}}},"pull_request":{"url":"https://api.github.com/repos/stan-dev/stan/pulls/1590","id":42615647,"html_url":"https://github.com/stan-dev/stan/pull/1590","diff_url":"https://github.com/stan-dev/stan/pull/1590.diff","patch_url":"https://github.com/stan-dev/stan/pull/1590.patch","issue_url":"https://api.github.com/repos/stan-dev/stan/issues/1590","number":1590,"state":"open","locked":false,"title":"feature/advi-adapt-eta","user":{"login":"akucukelbir","id":329960,"avatar_url":"https://avatars.githubusercontent.com/u/329960?v=3","gravatar_id":"","url":"https://api.github.com/users/akucukelbir","html_url":"https://github.com/akucukelbir","followers_url":"https://api.github.com/users/akucukelbir/followers","following_url":"https://api.github.com/users/akucukelbir/following{/other_user}","gists_url":"https://api.github.com/users/akucukelbir/gists{/gist_id}","starred_url":"https://api.github.com/users/akucukelbir/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/akucukelbir/subscriptions","organizations_url":"https://api.github.com/users/akucukelbir/orgs","repos_url":"https://api.github.com/users/akucukelbir/repos","events_url":"https://api.github.com/users/akucukelbir/events{/privacy}","received_events_url":"https://api.github.com/users/akucukelbir/received_events","type":"User","site_admin":false},"body":"#### Summary:\r\nImplements a \"warmup\" phase for ADVI that autotunes the `eta_adagrad` parameter. If the user defines this parameter, the \"warmup\" phase is skipped. To avoid confusion with the warmup phase with sampling, we call it the tuning phase.\r\n\r\nThe default parameter for `eta` is now equal to `0.0`, which means autotune. It uses a sequence of `eta` values to try: 1.0, 0.5, 0.01, 0.05, and 0.01.\r\n\r\nAlso implements the following changes:\r\n\r\n- [x] rename `eta_adagrad` argument to `eta`\r\n- [x] rename a lot of internal variables in `advi.hpp`.\r\n- [x] move `eta` from being a protected member of `advi.hpp` into a parameter of `run(...)`\r\n- [x] move helper functions in `advi.hpp` to be protected members\r\n- [x] scale `eta` by 1/sqrt(iteration_counter), which makes convergence more reliable.\r\n- [x] increase minimum size of `elbo_diff` to `2`.\r\n- [x] fix issue #1588: output a (more verbose) warning if max iterations is reached; and if ELBO is higher at a previous iteration than the ELBO at what ADVI converged to\r\n- [x] print a friendly message to `print_stream` while drawing samples from final approximation\r\n\r\nOther things on todolist (that won't be explicitly part of the above list once they're completed):\r\n- [x] have `eta` default to a certain value (not zero) if warmup is done, similar to how MCMC does it\r\n- [x] figure out how to report the tuned `eta`\r\n- [x] allow the # of tuning iterations to be specified by the user\r\n\r\n#### Intended Effect:\r\nAutomatic variational inference is now more automatic.\r\n\r\n#### How to Verify:\r\nRun any complicated model.\r\n- [x] unit tests\r\n- [x] new unit test for tuning\r\n\r\n#### Side Effects:\r\nNone\r\n\r\n#### Documentation:\r\n- [x] update main README.md in both CmdStan and Stan to include ADVI description\r\n- [ ] CmdStan\r\n- [ ] Stan\r\n\r\n#### Reviewer Suggestions:\r\n@dustinvtran and @syclik ","created_at":"2015-08-17T17:58:29Z","updated_at":"2015-08-26T15:23:52Z","closed_at":null,"merged_at":null,"merge_commit_sha":"7ca78cb61979981ec94f297b396c3a616c853422","assignee":{"login":"syclik","id":425751,"avatar_url":"https://avatars.githubusercontent.com/u/425751?v=3","gravatar_id":"","url":"https://api.github.com/users/syclik","html_url":"https://github.com/syclik","followers_url":"https://api.github.com/users/syclik/followers","following_url":"https://api.github.com/users/syclik/following{/other_user}","gists_url":"https://api.github.com/users/syclik/gists{/gist_id}","starred_url":"https://api.github.com/users/syclik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/syclik/subscriptions","organizations_url":"https://api.github.com/users/syclik/orgs","repos_url":"https://api.github.com/users/syclik/repos","events_url":"https://api.github.com/users/syclik/events{/privacy}","received_events_url":"https://api.github.com/users/syclik/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/stan-dev/stan/milestones/17","html_url":"https://github.com/stan-dev/stan/milestones/v2.7.1","labels_url":"https://api.github.com/repos/stan-dev/stan/milestones/17/labels","id":1143351,"number":17,"title":"v2.7.1","description":"Next version after 2.7.0.","creator":{"login":"syclik","id":425751,"avatar_url":"https://avatars.githubusercontent.com/u/425751?v=3","gravatar_id":"","url":"https://api.github.com/users/syclik","html_url":"https://github.com/syclik","followers_url":"https://api.github.com/users/syclik/followers","following_url":"https://api.github.com/users/syclik/following{/other_user}","gists_url":"https://api.github.com/users/syclik/gists{/gist_id}","starred_url":"https://api.github.com/users/syclik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/syclik/subscriptions","organizations_url":"https://api.github.com/users/syclik/orgs","repos_url":"https://api.github.com/users/syclik/repos","events_url":"https://api.github.com/users/syclik/events{/privacy}","received_events_url":"https://api.github.com/users/syclik/received_events","type":"User","site_admin":false},"open_issues":15,"closed_issues":29,"state":"open","created_at":"2015-06-01T16:58:07Z","updated_at":"2015-08-18T18:49:45Z","due_on":null,"closed_at":null},"commits_url":"https://api.github.com/repos/stan-dev/stan/pulls/1590/commits","review_comments_url":"https://api.github.com/repos/stan-dev/stan/pulls/1590/comments","review_comment_url":"https://api.github.com/repos/stan-dev/stan/pulls/comments{/number}","comments_url":"https://api.github.com/repos/stan-dev/stan/issues/1590/comments","statuses_url":"https://api.github.com/repos/stan-dev/stan/statuses/b678c066cb579af32a688f9f566c6e50b9835567","head":{"label":"stan-dev:feature/advi-adapt-eta","ref":"feature/advi-adapt-eta","sha":"b678c066cb579af32a688f9f566c6e50b9835567","user":{"login":"stan-dev","id":3374820,"avatar_url":"https://avatars.githubusercontent.com/u/3374820?v=3","gravatar_id":"","url":"https://api.github.com/users/stan-dev","html_url":"https://github.com/stan-dev","followers_url":"https://api.github.com/users/stan-dev/followers","following_url":"https://api.github.com/users/stan-dev/following{/other_user}","gists_url":"https://api.github.com/users/stan-dev/gists{/gist_id}","starred_url":"https://api.github.com/users/stan-dev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stan-dev/subscriptions","organizations_url":"https://api.github.com/users/stan-dev/orgs","repos_url":"https://api.github.com/users/stan-dev/repos","events_url":"https://api.github.com/users/stan-dev/events{/privacy}","received_events_url":"https://api.github.com/users/stan-dev/received_events","type":"Organization","site_admin":false},"repo":{"id":8043756,"name":"stan","full_name":"stan-dev/stan","owner":{"login":"stan-dev","id":3374820,"avatar_url":"https://avatars.githubusercontent.com/u/3374820?v=3","gravatar_id":"","url":"https://api.github.com/users/stan-dev","html_url":"https://github.com/stan-dev","followers_url":"https://api.github.com/users/stan-dev/followers","following_url":"https://api.github.com/users/stan-dev/following{/other_user}","gists_url":"https://api.github.com/users/stan-dev/gists{/gist_id}","starred_url":"https://api.github.com/users/stan-dev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stan-dev/subscriptions","organizations_url":"https://api.github.com/users/stan-dev/orgs","repos_url":"https://api.github.com/users/stan-dev/repos","events_url":"https://api.github.com/users/stan-dev/events{/privacy}","received_events_url":"https://api.github.com/users/stan-dev/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/stan-dev/stan","description":"Stan development repository (home page is linked below). The master branch contains the current release. The develop branch contains the latest stable development.  See the Developer Process Wiki for details.  ","fork":false,"url":"https://api.github.com/repos/stan-dev/stan","forks_url":"https://api.github.com/repos/stan-dev/stan/forks","keys_url":"https://api.github.com/repos/stan-dev/stan/keys{/key_id}","collaborators_url":"https://api.github.com/repos/stan-dev/stan/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/stan-dev/stan/teams","hooks_url":"https://api.github.com/repos/stan-dev/stan/hooks","issue_events_url":"https://api.github.com/repos/stan-dev/stan/issues/events{/number}","events_url":"https://api.github.com/repos/stan-dev/stan/events","assignees_url":"https://api.github.com/repos/stan-dev/stan/assignees{/user}","branches_url":"https://api.github.com/repos/stan-dev/stan/branches{/branch}","tags_url":"https://api.github.com/repos/stan-dev/stan/tags","blobs_url":"https://api.github.com/repos/stan-dev/stan/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/stan-dev/stan/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/stan-dev/stan/git/refs{/sha}","trees_url":"https://api.github.com/repos/stan-dev/stan/git/trees{/sha}","statuses_url":"https://api.github.com/repos/stan-dev/stan/statuses/{sha}","languages_url":"https://api.github.com/repos/stan-dev/stan/languages","stargazers_url":"https://api.github.com/repos/stan-dev/stan/stargazers","contributors_url":"https://api.github.com/repos/stan-dev/stan/contributors","subscribers_url":"https://api.github.com/repos/stan-dev/stan/subscribers","subscription_url":"https://api.github.com/repos/stan-dev/stan/subscription","commits_url":"https://api.github.com/repos/stan-dev/stan/commits{/sha}","git_commits_url":"https://api.github.com/repos/stan-dev/stan/git/commits{/sha}","comments_url":"https://api.github.com/repos/stan-dev/stan/comments{/number}","issue_comment_url":"https://api.github.com/repos/stan-dev/stan/issues/comments{/number}","contents_url":"https://api.github.com/repos/stan-dev/stan/contents/{+path}","compare_url":"https://api.github.com/repos/stan-dev/stan/compare/{base}...{head}","merges_url":"https://api.github.com/repos/stan-dev/stan/merges","archive_url":"https://api.github.com/repos/stan-dev/stan/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/stan-dev/stan/downloads","issues_url":"https://api.github.com/repos/stan-dev/stan/issues{/number}","pulls_url":"https://api.github.com/repos/stan-dev/stan/pulls{/number}","milestones_url":"https://api.github.com/repos/stan-dev/stan/milestones{/number}","notifications_url":"https://api.github.com/repos/stan-dev/stan/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/stan-dev/stan/labels{/name}","releases_url":"https://api.github.com/repos/stan-dev/stan/releases{/id}","created_at":"2013-02-06T03:25:04Z","updated_at":"2015-08-26T06:50:37Z","pushed_at":"2015-08-25T21:53:40Z","git_url":"git://github.com/stan-dev/stan.git","ssh_url":"git@github.com:stan-dev/stan.git","clone_url":"https://github.com/stan-dev/stan.git","svn_url":"https://github.com/stan-dev/stan","homepage":"http://mc-stan.org","size":356935,"stargazers_count":370,"watchers_count":370,"language":"C++","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":99,"mirror_url":null,"open_issues_count":197,"forks":99,"open_issues":197,"watchers":370,"default_branch":"develop"}},"base":{"label":"stan-dev:develop","ref":"develop","sha":"9f9ca95e31adb86180777104051ae61186636907","user":{"login":"stan-dev","id":3374820,"avatar_url":"https://avatars.githubusercontent.com/u/3374820?v=3","gravatar_id":"","url":"https://api.github.com/users/stan-dev","html_url":"https://github.com/stan-dev","followers_url":"https://api.github.com/users/stan-dev/followers","following_url":"https://api.github.com/users/stan-dev/following{/other_user}","gists_url":"https://api.github.com/users/stan-dev/gists{/gist_id}","starred_url":"https://api.github.com/users/stan-dev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stan-dev/subscriptions","organizations_url":"https://api.github.com/users/stan-dev/orgs","repos_url":"https://api.github.com/users/stan-dev/repos","events_url":"https://api.github.com/users/stan-dev/events{/privacy}","received_events_url":"https://api.github.com/users/stan-dev/received_events","type":"Organization","site_admin":false},"repo":{"id":8043756,"name":"stan","full_name":"stan-dev/stan","owner":{"login":"stan-dev","id":3374820,"avatar_url":"https://avatars.githubusercontent.com/u/3374820?v=3","gravatar_id":"","url":"https://api.github.com/users/stan-dev","html_url":"https://github.com/stan-dev","followers_url":"https://api.github.com/users/stan-dev/followers","following_url":"https://api.github.com/users/stan-dev/following{/other_user}","gists_url":"https://api.github.com/users/stan-dev/gists{/gist_id}","starred_url":"https://api.github.com/users/stan-dev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stan-dev/subscriptions","organizations_url":"https://api.github.com/users/stan-dev/orgs","repos_url":"https://api.github.com/users/stan-dev/repos","events_url":"https://api.github.com/users/stan-dev/events{/privacy}","received_events_url":"https://api.github.com/users/stan-dev/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/stan-dev/stan","description":"Stan development repository (home page is linked below). The master branch contains the current release. The develop branch contains the latest stable development.  See the Developer Process Wiki for details.  ","fork":false,"url":"https://api.github.com/repos/stan-dev/stan","forks_url":"https://api.github.com/repos/stan-dev/stan/forks","keys_url":"https://api.github.com/repos/stan-dev/stan/keys{/key_id}","collaborators_url":"https://api.github.com/repos/stan-dev/stan/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/stan-dev/stan/teams","hooks_url":"https://api.github.com/repos/stan-dev/stan/hooks","issue_events_url":"https://api.github.com/repos/stan-dev/stan/issues/events{/number}","events_url":"https://api.github.com/repos/stan-dev/stan/events","assignees_url":"https://api.github.com/repos/stan-dev/stan/assignees{/user}","branches_url":"https://api.github.com/repos/stan-dev/stan/branches{/branch}","tags_url":"https://api.github.com/repos/stan-dev/stan/tags","blobs_url":"https://api.github.com/repos/stan-dev/stan/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/stan-dev/stan/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/stan-dev/stan/git/refs{/sha}","trees_url":"https://api.github.com/repos/stan-dev/stan/git/trees{/sha}","statuses_url":"https://api.github.com/repos/stan-dev/stan/statuses/{sha}","languages_url":"https://api.github.com/repos/stan-dev/stan/languages","stargazers_url":"https://api.github.com/repos/stan-dev/stan/stargazers","contributors_url":"https://api.github.com/repos/stan-dev/stan/contributors","subscribers_url":"https://api.github.com/repos/stan-dev/stan/subscribers","subscription_url":"https://api.github.com/repos/stan-dev/stan/subscription","commits_url":"https://api.github.com/repos/stan-dev/stan/commits{/sha}","git_commits_url":"https://api.github.com/repos/stan-dev/stan/git/commits{/sha}","comments_url":"https://api.github.com/repos/stan-dev/stan/comments{/number}","issue_comment_url":"https://api.github.com/repos/stan-dev/stan/issues/comments{/number}","contents_url":"https://api.github.com/repos/stan-dev/stan/contents/{+path}","compare_url":"https://api.github.com/repos/stan-dev/stan/compare/{base}...{head}","merges_url":"https://api.github.com/repos/stan-dev/stan/merges","archive_url":"https://api.github.com/repos/stan-dev/stan/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/stan-dev/stan/downloads","issues_url":"https://api.github.com/repos/stan-dev/stan/issues{/number}","pulls_url":"https://api.github.com/repos/stan-dev/stan/pulls{/number}","milestones_url":"https://api.github.com/repos/stan-dev/stan/milestones{/number}","notifications_url":"https://api.github.com/repos/stan-dev/stan/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/stan-dev/stan/labels{/name}","releases_url":"https://api.github.com/repos/stan-dev/stan/releases{/id}","created_at":"2013-02-06T03:25:04Z","updated_at":"2015-08-26T06:50:37Z","pushed_at":"2015-08-25T21:53:40Z","git_url":"git://github.com/stan-dev/stan.git","ssh_url":"git@github.com:stan-dev/stan.git","clone_url":"https://github.com/stan-dev/stan.git","svn_url":"https://github.com/stan-dev/stan","homepage":"http://mc-stan.org","size":356935,"stargazers_count":370,"watchers_count":370,"language":"C++","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":99,"mirror_url":null,"open_issues_count":197,"forks":99,"open_issues":197,"watchers":370,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/stan-dev/stan/pulls/1590"},"html":{"href":"https://github.com/stan-dev/stan/pull/1590"},"issue":{"href":"https://api.github.com/repos/stan-dev/stan/issues/1590"},"comments":{"href":"https://api.github.com/repos/stan-dev/stan/issues/1590/comments"},"review_comments":{"href":"https://api.github.com/repos/stan-dev/stan/pulls/1590/comments"},"review_comment":{"href":"https://api.github.com/repos/stan-dev/stan/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/stan-dev/stan/pulls/1590/commits"},"statuses":{"href":"https://api.github.com/repos/stan-dev/stan/statuses/b678c066cb579af32a688f9f566c6e50b9835567"}}}}