{"action":"created","issue":{"url":"https://api.github.com/repos/chef/chef-server/issues/63","labels_url":"https://api.github.com/repos/chef/chef-server/issues/63/labels{/name}","comments_url":"https://api.github.com/repos/chef/chef-server/issues/63/comments","events_url":"https://api.github.com/repos/chef/chef-server/issues/63/events","html_url":"https://github.com/chef/chef-server/issues/63","id":54546502,"number":63,"title":"chef FATAL: Net::HTTPServerException: 403 \"Forbidden\"","user":{"login":"gokulnath","id":650295,"avatar_url":"https://avatars.githubusercontent.com/u/650295?v=3","gravatar_id":"","url":"https://api.github.com/users/gokulnath","html_url":"https://github.com/gokulnath","followers_url":"https://api.github.com/users/gokulnath/followers","following_url":"https://api.github.com/users/gokulnath/following{/other_user}","gists_url":"https://api.github.com/users/gokulnath/gists{/gist_id}","starred_url":"https://api.github.com/users/gokulnath/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gokulnath/subscriptions","organizations_url":"https://api.github.com/users/gokulnath/orgs","repos_url":"https://api.github.com/users/gokulnath/repos","events_url":"https://api.github.com/users/gokulnath/events{/privacy}","received_events_url":"https://api.github.com/users/gokulnath/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-01-16T07:12:09Z","updated_at":"2015-02-12T13:26:29Z","closed_at":null,"body":"Towards the end of my chef-client run, the following error appears (sudo chef-client -l debug)\r\n```\r\n[2015-01-16T06:56:37+00:00] DEBUG: Saving the current state of node vas-netmagic-cloud-git-deploy\r\n[2015-01-16T06:56:37+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::JSONInput#handle_request\r\n[2015-01-16T06:56:37+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::JSONToModelOutput#handle_request\r\n[2015-01-16T06:56:37+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::CookieManager#handle_request\r\n[2015-01-16T06:56:37+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::Decompressor#handle_request\r\n[2015-01-16T06:56:37+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::Authenticator#handle_request\r\n[2015-01-16T06:56:37+00:00] DEBUG: Signing the request as vas-netmagic-cloud-git-deploy\r\n[2015-01-16T06:56:37+00:00] DEBUG: String to sign: 'Method:PUT\r\nHashed Path:O9nOlTTP4Om5QeHdkq8Iu0qTqF8=\r\nX-Ops-Content-Hash:1Nt/7y3OeEbJRPxqc2E/xz+XcvY=\r\nX-Ops-Timestamp:2015-01-16T06:56:37Z\r\nX-Ops-UserId:vas-netmagic-cloud-git-deploy'\r\nHeader hash: {\"X-Ops-Sign\"=>\"algorithm=sha1;version=1.0;\", \"X-Ops-Userid\"=>\"vas-netmagic-cloud-git-deploy\", \"X-Ops-Timestamp\"=>\"2015-01-16T06:56:37Z\", \"X-Ops-Content-Hash\"=>\"contenthash\", \"X-Ops-Authorization-1\"=>\"authorization-hash\", \"X-Ops-Authorization-2\"=>\"authorization-hash\", \"X-Ops-Authorization-3\"=>\"authorization-hash\", \"X-Ops-Authorization-4\"=>\"aurhorization-hash\", \"X-Ops-Authorization-5\"=>\"anotherhash\", \"X-Ops-Authorization-6\"=>\"last hash :)\"}\r\n[2015-01-16T06:56:37+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::RemoteRequestID#handle_request\r\n[2015-01-16T06:56:37+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::ValidateContentLength#handle_request\r\n[2015-01-16T06:56:37+00:00] DEBUG: Initiating PUT to https://chef.mobme.in/nodes/vas-netmagic-cloud-git-deploy\r\n[2015-01-16T06:56:37+00:00] DEBUG: ---- HTTP Request Header Data: ----\r\n[2015-01-16T06:56:37+00:00] DEBUG: Content-Type: application/json\r\n[2015-01-16T06:56:37+00:00] DEBUG: Accept: application/json\r\n[2015-01-16T06:56:37+00:00] DEBUG: Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\n[2015-01-16T06:56:37+00:00] DEBUG: X-OPS-SIGN: algorithm=sha1;version=1.0;\r\n[2015-01-16T06:56:37+00:00] DEBUG: X-OPS-USERID: vas-netmagic-cloud-git-deploy\r\n[2015-01-16T06:56:37+00:00] DEBUG: X-OPS-TIMESTAMP: 2015-01-16T06:56:37Z\r\n[2015-01-16T06:56:37+00:00] DEBUG: X-OPS-CONTENT-HASH: 1Nt/7y3OeEbJRPxqc2E/xz+XcvY=\r\n[2015-01-16T06:56:37+00:00] DEBUG: X-OPS-AUTHORIZATION-1: \r\nauthorizationhash\r\n[2015-01-16T06:56:37+00:00] DEBUG: X-OPS-AUTHORIZATION-2: \r\nauthorizationhash\r\n[2015-01-16T06:56:37+00:00] DEBUG: X-OPS-AUTHORIZATION-3: \r\nauthorizationhash\r\n[2015-01-16T06:56:37+00:00] DEBUG: X-OPS-AUTHORIZATION-4: authorizationhash\r\n[2015-01-16T06:56:37+00:00] DEBUG: X-OPS-AUTHORIZATION-5: authorizationhash\r\n[2015-01-16T06:56:37+00:00] DEBUG: X-OPS-AUTHORIZATION-6: authorizationhash\r\n[2015-01-16T06:56:37+00:00] DEBUG: HOST: chef.mobme.in:443\r\n[2015-01-16T06:56:37+00:00] DEBUG: X-REMOTE-REQUEST-ID: bd99e189-21c3-4ee0-a563-ba2f4b5c9136\r\n[2015-01-16T06:56:37+00:00] DEBUG: Content-Length: 71537\r\n[2015-01-16T06:56:37+00:00] DEBUG: ---- End HTTP Request Header Data ----\r\n[2015-01-16T06:56:37+00:00] DEBUG: ---- HTTP Status and Header Data: ----\r\n[2015-01-16T06:56:37+00:00] DEBUG: HTTP 1.1 403 Forbidden\r\n[2015-01-16T06:56:37+00:00] DEBUG: server: ngx_openresty/1.4.3.6\r\n[2015-01-16T06:56:37+00:00] DEBUG: date: Fri, 16 Jan 2015 06:56:37 GMT\r\n[2015-01-16T06:56:37+00:00] DEBUG: content-length: 39\r\n[2015-01-16T06:56:37+00:00] DEBUG: connection: close\r\n[2015-01-16T06:56:37+00:00] DEBUG: x-ops-api-info: flavor=cs;version=12.0.0;oc_erchef=0.29.4\r\n[2015-01-16T06:56:37+00:00] DEBUG: ---- End HTTP Status/Header Data ----\r\n[2015-01-16T06:56:37+00:00] DEBUG: ---- HTTP Response Body ----\r\n[2015-01-16T06:56:37+00:00] DEBUG: {\"error\":[\"missing update permission\"]}\r\n[2015-01-16T06:56:37+00:00] DEBUG: ---- End HTTP Response Body -----\r\n[2015-01-16T06:56:37+00:00] DEBUG: ---- HTTP Request Body ----\r\n[2015-01-16T06:56:37+00:00] DEBUG: { \"content\" :\r\n  \"A big JSON }\r\n[2015-01-16T06:56:37+00:00] DEBUG: ---- End HTTP Request Body ----\r\n[2015-01-16T06:56:37+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::ValidateContentLength#handle_response\r\n[2015-01-16T06:56:37+00:00] DEBUG: Content-Length validated correctly.\r\n[2015-01-16T06:56:37+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::RemoteRequestID#handle_response\r\n[2015-01-16T06:56:37+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::Authenticator#handle_response\r\n[2015-01-16T06:56:37+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::Decompressor#handle_response\r\n[2015-01-16T06:56:37+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::CookieManager#handle_response\r\n[2015-01-16T06:56:37+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::JSONToModelOutput#handle_response\r\n[2015-01-16T06:56:37+00:00] DEBUG: Expected JSON response, but got content-type ''\r\n[2015-01-16T06:56:37+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::JSONInput#handle_response\r\n[2015-01-16T06:56:37+00:00] INFO: HTTP Request Returned 403 Forbidden: error\r\n[2015-01-16T06:56:37+00:00] DEBUG: Re-raising exception: Net::HTTPServerException - 403 \"Forbidden\"\r\n/opt/chef/embedded/lib/ruby/2.1.0/net/http/response.rb:119:in `error!'\r\n  /opt/chef/embedded/apps/chef/lib/chef/http.rb:145:in `request'\r\n  /opt/chef/embedded/apps/chef/lib/chef/http.rb:118:in `put'\r\n  /opt/chef/embedded/apps/chef/lib/chef/node.rb:512:in `save'\r\n  /opt/chef/embedded/apps/chef/lib/chef/client.rb:257:in `save_updated_node'\r\n  /opt/chef/embedded/apps/chef/lib/chef/client.rb:403:in `run'\r\n  /opt/chef/embedded/apps/chef/lib/chef/application.rb:261:in `block in fork_chef_client'\r\n  /opt/chef/embedded/apps/chef/lib/chef/application.rb:249:in `fork'\r\n  /opt/chef/embedded/apps/chef/lib/chef/application.rb:249:in `fork_chef_client'\r\n  /opt/chef/embedded/apps/chef/lib/chef/application.rb:215:in `block in run_chef_client'\r\n  /opt/chef/embedded/apps/chef/lib/chef/local_mode.rb:38:in `with_server_connectivity'\r\n  /opt/chef/embedded/apps/chef/lib/chef/application.rb:201:in `run_chef_client'\r\n  /opt/chef/embedded/apps/chef/lib/chef/application/client.rb:355:in `block in interval_run_chef_client'\r\n  /opt/chef/embedded/apps/chef/lib/chef/application/client.rb:345:in `loop'\r\n  /opt/chef/embedded/apps/chef/lib/chef/application/client.rb:345:in `interval_run_chef_client'\r\n  /opt/chef/embedded/apps/chef/lib/chef/application/client.rb:335:in `run_application'\r\n  /opt/chef/embedded/apps/chef/lib/chef/application.rb:58:in `run'\r\n  /opt/chef/embedded/apps/chef/bin/chef-client:26:in `<top (required)>'\r\n  /usr/bin/chef-client:40:in `load'\r\n  /usr/bin/chef-client:40:in `<main>'\r\n\r\nRunning handlers:\r\n[2015-01-16T06:56:37+00:00] ERROR: Running exception handlers\r\nRunning handlers complete\r\n[2015-01-16T06:56:37+00:00] ERROR: Exception handlers complete\r\n[2015-01-16T06:56:37+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out\r\n[2015-01-16T06:56:37+00:00] DEBUG: Net::HTTPServerException: 403 \"Forbidden\"\r\n/opt/chef/embedded/lib/ruby/2.1.0/net/http/response.rb:119:in `error!'\r\n/opt/chef/embedded/apps/chef/lib/chef/http.rb:145:in `request'\r\n/opt/chef/embedded/apps/chef/lib/chef/http.rb:118:in `put'\r\n/opt/chef/embedded/apps/chef/lib/chef/node.rb:512:in `save'\r\n/opt/chef/embedded/apps/chef/lib/chef/client.rb:257:in `save_updated_node'\r\n/opt/chef/embedded/apps/chef/lib/chef/client.rb:403:in `run'\r\n/opt/chef/embedded/apps/chef/lib/chef/application.rb:261:in `block in fork_chef_client'\r\n/opt/chef/embedded/apps/chef/lib/chef/application.rb:249:in `fork'\r\n/opt/chef/embedded/apps/chef/lib/chef/application.rb:249:in `fork_chef_client'\r\n/opt/chef/embedded/apps/chef/lib/chef/application.rb:215:in `block in run_chef_client'\r\n/opt/chef/embedded/apps/chef/lib/chef/local_mode.rb:38:in `with_server_connectivity'\r\n/opt/chef/embedded/apps/chef/lib/chef/application.rb:201:in `run_chef_client'\r\n/opt/chef/embedded/apps/chef/lib/chef/application/client.rb:355:in `block in interval_run_chef_client'\r\n/opt/chef/embedded/apps/chef/lib/chef/application/client.rb:345:in `loop'\r\n/opt/chef/embedded/apps/chef/lib/chef/application/client.rb:345:in `interval_run_chef_client'\r\n/opt/chef/embedded/apps/chef/lib/chef/application/client.rb:335:in `run_application'\r\n/opt/chef/embedded/apps/chef/lib/chef/application.rb:58:in `run'\r\n/opt/chef/embedded/apps/chef/bin/chef-client:26:in `<top (required)>'\r\n/usr/bin/chef-client:40:in `load'\r\n/usr/bin/chef-client:40:in `<main>'\r\nChef Client failed. 6 resources updated in 68.290737107 seconds\r\n[2015-01-16T06:56:37+00:00] DEBUG: Server doesn't support resource history, skipping resource report.\r\n[2015-01-16T06:56:38+00:00] ERROR: 403 \"Forbidden\"\r\n[2015-01-16T06:56:38+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)\r\n```\r\nI suspect this can be solved with https://docs.chef.io/errors.html#forbidden. But I don't have chef management console. Can I fix this with knife or by any other way.\r\n\r\nBTW, this issue started appearing after my upgrade from chef 11.0.8 to 12.0.1"},"comment":{"url":"https://api.github.com/repos/chef/chef-server/issues/comments/74069692","html_url":"https://github.com/chef/chef-server/issues/63#issuecomment-74069692","issue_url":"https://api.github.com/repos/chef/chef-server/issues/63","id":74069692,"user":{"login":"gokulnath","id":650295,"avatar_url":"https://avatars.githubusercontent.com/u/650295?v=3","gravatar_id":"","url":"https://api.github.com/users/gokulnath","html_url":"https://github.com/gokulnath","followers_url":"https://api.github.com/users/gokulnath/followers","following_url":"https://api.github.com/users/gokulnath/following{/other_user}","gists_url":"https://api.github.com/users/gokulnath/gists{/gist_id}","starred_url":"https://api.github.com/users/gokulnath/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gokulnath/subscriptions","organizations_url":"https://api.github.com/users/gokulnath/orgs","repos_url":"https://api.github.com/users/gokulnath/repos","events_url":"https://api.github.com/users/gokulnath/events{/privacy}","received_events_url":"https://api.github.com/users/gokulnath/received_events","type":"User","site_admin":false},"created_at":"2015-02-12T13:26:29Z","updated_at":"2015-02-12T13:26:29Z","body":"This is what I get.\r\n% knife acl add nodes gokul-vagrant update client gokul-vagrant\r\nffi-yajl/json_gem is deprecated, these monkeypatches will be dropped shortly\r\n\r\n```\r\nERROR: The data in your request was invalid\r\nResponse: <html><head><title>400 Bad Request</title></head><body><h1>Bad Request</h1>Bad Request<p><hr><address>mochiweb+webmachine web server</address></body></html>\r\n```"}}