{"action":"created","issue":{"url":"https://api.github.com/repos/crowbar/crowbar-ha/issues/13","labels_url":"https://api.github.com/repos/crowbar/crowbar-ha/issues/13/labels{/name}","comments_url":"https://api.github.com/repos/crowbar/crowbar-ha/issues/13/comments","events_url":"https://api.github.com/repos/crowbar/crowbar-ha/issues/13/events","html_url":"https://github.com/crowbar/crowbar-ha/pull/13","id":104948401,"number":13,"title":"[UNTESTED] immediately abort chef-client if maintenance mode is set","user":{"login":"aspiers","id":100738,"avatar_url":"https://avatars.githubusercontent.com/u/100738?v=3","gravatar_id":"","url":"https://api.github.com/users/aspiers","html_url":"https://github.com/aspiers","followers_url":"https://api.github.com/users/aspiers/followers","following_url":"https://api.github.com/users/aspiers/following{/other_user}","gists_url":"https://api.github.com/users/aspiers/gists{/gist_id}","starred_url":"https://api.github.com/users/aspiers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aspiers/subscriptions","organizations_url":"https://api.github.com/users/aspiers/orgs","repos_url":"https://api.github.com/users/aspiers/repos","events_url":"https://api.github.com/users/aspiers/events{/privacy}","received_events_url":"https://api.github.com/users/aspiers/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-09-04T18:55:59Z","updated_at":"2015-09-11T13:12:38Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/crowbar/crowbar-ha/pulls/13","html_url":"https://github.com/crowbar/crowbar-ha/pull/13","diff_url":"https://github.com/crowbar/crowbar-ha/pull/13.diff","patch_url":"https://github.com/crowbar/crowbar-ha/pull/13.patch"},"body":"The reasoning is explained in the comments.  I'm not sure why I ever thought it was a good idea to allow a chef-client run to proceed if maintenance mode is set.  It's not even as if that approach could ever restore an ill node to full health, because we were deliberately checking whether the maintenance mode was set prior to the chef-client run, and if so, leaving it set.  This way, if a node is left in maintenance mode, it will be discovered sooner, resulting in the cloud operator being alerted to a degraded cluster sooner.\r\n\r\nAlso ported https://github.com/crowbar/barclamp-pacemaker/issues/149 on top of this."},"comment":{"url":"https://api.github.com/repos/crowbar/crowbar-ha/issues/comments/139541964","html_url":"https://github.com/crowbar/crowbar-ha/pull/13#issuecomment-139541964","issue_url":"https://api.github.com/repos/crowbar/crowbar-ha/issues/13","id":139541964,"user":{"login":"aspiers","id":100738,"avatar_url":"https://avatars.githubusercontent.com/u/100738?v=3","gravatar_id":"","url":"https://api.github.com/users/aspiers","html_url":"https://github.com/aspiers","followers_url":"https://api.github.com/users/aspiers/followers","following_url":"https://api.github.com/users/aspiers/following{/other_user}","gists_url":"https://api.github.com/users/aspiers/gists{/gist_id}","starred_url":"https://api.github.com/users/aspiers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aspiers/subscriptions","organizations_url":"https://api.github.com/users/aspiers/orgs","repos_url":"https://api.github.com/users/aspiers/repos","events_url":"https://api.github.com/users/aspiers/events{/privacy}","received_events_url":"https://api.github.com/users/aspiers/received_events","type":"User","site_admin":false},"created_at":"2015-09-11T13:12:38Z","updated_at":"2015-09-11T13:12:38Z","body":"Crap, raising an exception in the start handler doesn't abort the run :-(\r\n\r\n```\r\n[2015-09-11T13:09:48+00:00] ERROR: Report handler Chef::Pacemaker::StartHandler raised #<RuntimeError: Pacemaker maintenance mode was already set on d52-54-02-77-77-02; aborting! Please diagnose why this was the case, \r\nfix the root cause, and then unset maintenance mode via HAWK or by running 'crm node ready' on the node.>\r\n[2015-09-11T13:09:48+00:00] ERROR: /var/chef/handlers/pacemaker_maintenance_handlers.rb:36:in `report'\r\n[2015-09-11T13:09:48+00:00] ERROR: /usr/lib64/ruby/gems/2.1.0/gems/chef-10.32.2/lib/chef/handler.rb:226:in `run_report_unsafe'\r\n[2015-09-11T13:09:48+00:00] ERROR: /usr/lib64/ruby/gems/2.1.0/gems/chef-10.32.2/lib/chef/handler.rb:214:in `run_report_safely'\r\n[2015-09-11T13:09:48+00:00] ERROR: /usr/lib64/ruby/gems/2.1.0/gems/chef-10.32.2/lib/chef/handler.rb:70:in `block in run_start_handlers'\r\n[2015-09-11T13:09:48+00:00] ERROR: /usr/lib64/ruby/gems/2.1.0/gems/chef-10.32.2/lib/chef/handler.rb:69:in `each'\r\n[2015-09-11T13:09:48+00:00] ERROR: /usr/lib64/ruby/gems/2.1.0/gems/chef-10.32.2/lib/chef/handler.rb:69:in `run_start_handlers'\r\n[2015-09-11T13:09:48+00:00] ERROR: /usr/lib64/ruby/gems/2.1.0/gems/chef-10.32.2/lib/chef/handler.rb:78:in `block in <class:Handler>'\r\n[2015-09-11T13:09:48+00:00] ERROR: /usr/lib64/ruby/gems/2.1.0/gems/chef-10.32.2/lib/chef/client.rb:103:in `call'\r\n[2015-09-11T13:09:48+00:00] ERROR: /usr/lib64/ruby/gems/2.1.0/gems/chef-10.32.2/lib/chef/client.rb:103:in `block in run_started'\r\n[2015-09-11T13:09:48+00:00] ERROR: /usr/lib64/ruby/gems/2.1.0/gems/chef-10.32.2/lib/chef/client.rb:102:in `each'\r\n[2015-09-11T13:09:48+00:00] ERROR: /usr/lib64/ruby/gems/2.1.0/gems/chef-10.32.2/lib/chef/client.rb:102:in `run_started'\r\n[2015-09-11T13:09:48+00:00] ERROR: /usr/lib64/ruby/gems/2.1.0/gems/chef-10.32.2/lib/chef/client.rb:426:in `do_run'\r\n[2015-09-11T13:09:48+00:00] ERROR: /usr/lib64/ruby/gems/2.1.0/gems/chef-10.32.2/lib/chef/client.rb:184:in `run'\r\n[2015-09-11T13:09:48+00:00] ERROR: /usr/lib64/ruby/gems/2.1.0/gems/chef-10.32.2/lib/chef/application.rb:133:in `run_chef_client'\r\n[2015-09-11T13:09:48+00:00] ERROR: /usr/lib64/ruby/gems/2.1.0/gems/chef-10.32.2/lib/chef/application/client.rb:306:in `block in run_application'\r\n[2015-09-11T13:09:48+00:00] ERROR: /usr/lib64/ruby/gems/2.1.0/gems/chef-10.32.2/lib/chef/application/client.rb:294:in `loop'\r\n[2015-09-11T13:09:48+00:00] ERROR: /usr/lib64/ruby/gems/2.1.0/gems/chef-10.32.2/lib/chef/application/client.rb:294:in `run_application'\r\n[2015-09-11T13:09:48+00:00] ERROR: /usr/lib64/ruby/gems/2.1.0/gems/chef-10.32.2/lib/chef/application.rb:65:in `run'\r\n[2015-09-11T13:09:48+00:00] ERROR: /usr/lib64/ruby/gems/2.1.0/gems/chef-10.32.2/bin/chef-client:26:in `<top (required)>'\r\n[2015-09-11T13:09:48+00:00] ERROR: /usr/bin/chef-client:23:in `load'\r\n[2015-09-11T13:09:48+00:00] ERROR: /usr/bin/chef-client:23:in `<main>'\r\n[2015-09-11T13:09:48+00:00] INFO: Start handlers complete.\r\n[2015-09-11T13:09:48+00:00] INFO: Loading cookbooks [apache2, barclamp, bluepill, bmc-nat, ceph, cinder, corosync, crowbar-hacks, crowbar-openstack, crowbar-pacemaker, database, drbd, glance, haproxy, hawk, kernel-pani\r\nc, keystone, logging, lvm, mysql, nagios, network, neutron, nsx, ntp, ohai, openssl, pacemaker, postgresql, provisioner, rabbitmq, repos, utils, uwsgi, xfs]\r\n[2015-09-11T13:09:48+00:00] INFO: Storing updated cookbooks/crowbar-pacemaker/libraries/maintenance_mode_helpers.rb in the cache.\r\n```\r\nNeed to find a way to do this."}}