{"action":"created","issue":{"url":"https://api.github.com/repos/freedomofpress/securedrop/issues/1008","labels_url":"https://api.github.com/repos/freedomofpress/securedrop/issues/1008/labels{/name}","comments_url":"https://api.github.com/repos/freedomofpress/securedrop/issues/1008/comments","events_url":"https://api.github.com/repos/freedomofpress/securedrop/issues/1008/events","html_url":"https://github.com/freedomofpress/securedrop/issues/1008","id":71384001,"number":1008,"title":"ansible playbook over tor with reboots","user":{"login":"dolanjs","id":1495205,"avatar_url":"https://avatars.githubusercontent.com/u/1495205?v=3","gravatar_id":"","url":"https://api.github.com/users/dolanjs","html_url":"https://github.com/dolanjs","followers_url":"https://api.github.com/users/dolanjs/followers","following_url":"https://api.github.com/users/dolanjs/following{/other_user}","gists_url":"https://api.github.com/users/dolanjs/gists{/gist_id}","starred_url":"https://api.github.com/users/dolanjs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dolanjs/subscriptions","organizations_url":"https://api.github.com/users/dolanjs/orgs","repos_url":"https://api.github.com/users/dolanjs/repos","events_url":"https://api.github.com/users/dolanjs/events{/privacy}","received_events_url":"https://api.github.com/users/dolanjs/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-04-27T19:51:18Z","updated_at":"2015-05-07T22:43:19Z","closed_at":null,"body":"Experiencing intermittent timeout errors after the ansible playbook reboots the servers.\r\n\r\n1) The playbook is being ran over tor\r\n2) A reboot is required during the anisble playbook\r\n3) The [task](https://github.com/freedomofpress/securedrop/blob/develop/install_files/ansible-base/roles/common/tasks/apply_grsec_lock.yml#L27) to wait for the ssh port to become available again completes successfully\r\n4) The [task after](https://github.com/freedomofpress/securedrop/blob/develop/install_files/ansible-base/roles/common/tasks/apply_grsec_lock.yml#L48) waiting for the ssh port to be available timeouts\r\n\r\n* This issue only applies when the ansible playbook is being ran over tor so does not affect initial installs where the playbook is not being ran over tor.\r\n* The issue only applies if the conditions requiring a reboot (new grsec kernel or other security updates that require a reboot) are met\r\n* This issue only happens intermittently. To test you could create the reboot-required file on the app/mon servers that the playbook looks for to require a reboot."},"comment":{"url":"https://api.github.com/repos/freedomofpress/securedrop/issues/comments/100040446","html_url":"https://github.com/freedomofpress/securedrop/issues/1008#issuecomment-100040446","issue_url":"https://api.github.com/repos/freedomofpress/securedrop/issues/1008","id":100040446,"user":{"login":"garrettr","id":407302,"avatar_url":"https://avatars.githubusercontent.com/u/407302?v=3","gravatar_id":"","url":"https://api.github.com/users/garrettr","html_url":"https://github.com/garrettr","followers_url":"https://api.github.com/users/garrettr/followers","following_url":"https://api.github.com/users/garrettr/following{/other_user}","gists_url":"https://api.github.com/users/garrettr/gists{/gist_id}","starred_url":"https://api.github.com/users/garrettr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/garrettr/subscriptions","organizations_url":"https://api.github.com/users/garrettr/orgs","repos_url":"https://api.github.com/users/garrettr/repos","events_url":"https://api.github.com/users/garrettr/events{/privacy}","received_events_url":"https://api.github.com/users/garrettr/received_events","type":"User","site_admin":false},"created_at":"2015-05-07T22:43:19Z","updated_at":"2015-05-07T22:43:19Z","body":"I think it is interesting that there are at least 3 separate tasks that implement \"waiting for the server to come back\". These tasks are similar, *but different*. Is it possible that these differences are the cause of the problem? I'm not sure which tasks is used when (for example, I don't know which specific task was used in the traceback @dolanjs posted in the above comment), but it seems like these differences could be involved.\r\n\r\nGenerally, we should:\r\n\r\n1. DRY out the playbooks. I understand this is a long term project. However having 3 different (and *differing*) implementations of the same functionality is a recipe for confusion and bugs.\r\n2. Look into whether some difference between these tasks if the a cause of the problem. For example, the significantly shorter delay on the version in `cron_apt.yml` seems suspicious to me.\r\n\r\n```sh\r\n➜  securedrop git:(0.3.3-release-testing-fixes) ag -A 10 \"waiting for server to come back\"\r\ninstall_files/ansible-base/roles/common/tasks/apply_grsec_lock.yml\r\n23:- name: waiting for server to come back\r\n24-  local_action:\r\n25-    module: wait_for\r\n26-      host={{ ansible_ssh_host }}\r\n27-      port={{ ansible_ssh_port }}\r\n28-      delay=45\r\n29-      state=started\r\n30-  when: not grsec_lock.stat.exists\r\n31-  sudo: false\r\n32-\r\n33-- sysctl: name=\"{{ item.name }}\" value=\"{{ item.value }}\" sysctl_set=yes state=present reload=yes\r\n\r\ninstall_files/ansible-base/roles/common/tasks/cron_apt.yml\r\n38:- name: waiting for server to come back\r\n39-  local_action:\r\n40-    module: wait_for\r\n41-      host={{ ansible_ssh_host }}\r\n42-      port={{ ansible_ssh_port }}\r\n43-      delay=20\r\n44-      state=started\r\n45-  when: reboot_required.stat.exists\r\n46-  sudo: false\r\n47-\r\n\r\n\r\ninstall_files/ansible-base/roles/common/tasks/from_fpf_repo_install_grsec.yml\r\n43:- name: waiting for server to come back\r\n44-  local_action:\r\n45-    module: wait_for\r\n46-      host={{ ansible_ssh_host }}\r\n47-      port={{ ansible_ssh_port }}\r\n48-      delay=45\r\n49-      state=started\r\n50-  sudo: false\r\n51-  when: not running_grsec.stat.exists\r\n52-  tags:\r\n53-    - reboot\r\n```"}}