{"action":"closed","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/7517","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/7517/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/7517/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/7517/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/7517","id":71963777,"number":7517,"title":"apiserver incorrectly returns 405 when 404 expected during a HEAD request","user":{"login":"satnam6502","id":6164339,"avatar_url":"https://avatars.githubusercontent.com/u/6164339?v=3","gravatar_id":"","url":"https://api.github.com/users/satnam6502","html_url":"https://github.com/satnam6502","followers_url":"https://api.github.com/users/satnam6502/followers","following_url":"https://api.github.com/users/satnam6502/following{/other_user}","gists_url":"https://api.github.com/users/satnam6502/gists{/gist_id}","starred_url":"https://api.github.com/users/satnam6502/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/satnam6502/subscriptions","organizations_url":"https://api.github.com/users/satnam6502/orgs","repos_url":"https://api.github.com/users/satnam6502/repos","events_url":"https://api.github.com/users/satnam6502/events{/privacy}","received_events_url":"https://api.github.com/users/satnam6502/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/component/apiserver","name":"component/apiserver","color":"0052cc"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/kind/bug","name":"kind/bug","color":"ff0000"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/master","name":"team/master","color":"d2b48c"}],"state":"closed","locked":false,"assignee":{"login":"satnam6502","id":6164339,"avatar_url":"https://avatars.githubusercontent.com/u/6164339?v=3","gravatar_id":"","url":"https://api.github.com/users/satnam6502","html_url":"https://github.com/satnam6502","followers_url":"https://api.github.com/users/satnam6502/followers","following_url":"https://api.github.com/users/satnam6502/following{/other_user}","gists_url":"https://api.github.com/users/satnam6502/gists{/gist_id}","starred_url":"https://api.github.com/users/satnam6502/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/satnam6502/subscriptions","organizations_url":"https://api.github.com/users/satnam6502/orgs","repos_url":"https://api.github.com/users/satnam6502/repos","events_url":"https://api.github.com/users/satnam6502/events{/privacy}","received_events_url":"https://api.github.com/users/satnam6502/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones/6","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/milestones/v1.0","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones/6/labels","id":969317,"number":6,"title":"v1.0","description":"Issues that need to be resolved for v1.0 of Kubernetes.","creator":{"login":"goltermann","id":9358478,"avatar_url":"https://avatars.githubusercontent.com/u/9358478?v=3","gravatar_id":"","url":"https://api.github.com/users/goltermann","html_url":"https://github.com/goltermann","followers_url":"https://api.github.com/users/goltermann/followers","following_url":"https://api.github.com/users/goltermann/following{/other_user}","gists_url":"https://api.github.com/users/goltermann/gists{/gist_id}","starred_url":"https://api.github.com/users/goltermann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/goltermann/subscriptions","organizations_url":"https://api.github.com/users/goltermann/orgs","repos_url":"https://api.github.com/users/goltermann/repos","events_url":"https://api.github.com/users/goltermann/events{/privacy}","received_events_url":"https://api.github.com/users/goltermann/received_events","type":"User","site_admin":false},"open_issues":41,"closed_issues":259,"state":"open","created_at":"2015-02-06T20:27:16Z","updated_at":"2015-05-08T21:19:42Z","due_on":null,"closed_at":null},"comments":10,"created_at":"2015-04-29T19:20:13Z","updated_at":"2015-05-08T21:19:42Z","closed_at":"2015-05-08T21:19:42Z","body":"I think this is the root of the reason why the new Kibana to Elasticsearch setup failed to work (even with basic auth). I think a 404 response is incorrectly being turned into a 405 response.\r\n\r\nIn these example I am running a Kibana service which maps to a pod running a Kibana server.\r\n```\r\nPOD                    IP           CONTAINER(S)     IMAGE(S)                              HOST                                    LABELS                                                   STATUS    CREATED\r\nkibana-logging-88fi0   10.244.0.6   kibana-logging   gcr.io/google_containers/kibana:1.3   kubernetes-minion-02dr/104.197.15.218   kubernetes.io/cluster-service=true,name=kibana-logging   Running   44 minutes\r\nCONTROLLER       CONTAINER(S)     IMAGE(S)                              SELECTOR              REPLICAS\r\nkibana-logging   kibana-logging   gcr.io/google_containers/kibana:1.3   name=kibana-logging   1\r\nNAME             LABELS                                                   SELECTOR              IP            PORT(S)\r\nkibana-logging   kubernetes.io/cluster-service=true,name=kibana-logging   name=kibana-logging   10.0.32.105   5601/TCP\r\n```\r\n\r\nNote that the Kibana service has the IP 10.0.32.105. If we ssh into a minion and curl against this we get the expected response i.e. \"a .kibana index does not exist\". This is the correct expected behavior.\r\n\r\n```\r\n$ curl -X HEAD -i  10.0.32.105:5601/elasticsearch/.kibana\r\nHTTP/1.1 404 Not Found\r\nX-App-Name: kibana\r\ncontent-type: text/plain; charset=UTF-8\r\ncontent-length: 0\r\nDate: Wed, 29 Apr 2015 18:57:21 GMT\r\nConnection: keep-alive\r\n```\r\n\r\nHowever, the Kibana dashboard reports an error when it tries to connect to Elasticsearch (which it does by going through its own URL with the subpath `elasticsearch`) saying the method is not allowed:\r\n\r\n```\r\nhttps://104.197.26.147/api/v1beta3/proxy/namespaces/default/services/kibana-logging/elasticsearch/.kibana Failed to load resource: the server responded with a status of 405 (Method Not Allowed)\r\n```\r\n\r\nIf we ssh to the master and perform the same HEAD request against the apiserver we get a 404 Forbidden:\r\n```\r\n$ curl -X HEAD -i  http://localhost:7080/api/v1beta3/proxy/namespaces/default/services/kibana-logging/.kibana\r\nHTTP/1.1 403 Forbidden\r\nDate: Wed, 29 Apr 2015 19:07:37 GMT\r\nContent-Length: 29\r\nContent-Type: text/plain; charset=utf-8\r\n```\r\n\r\nAdding the basic auth we get a 405 Method Not Allowed:\r\n\r\n```\r\n$ curl -X HEAD -i -k -u admin:REDACTED  https://localhost/api/v1beta3/proxy/namespaces/default/services/kibana-logging/.kibana\r\nHTTP/1.1 405 Method Not Allowed\r\nContent-Type: application/json\r\nDate: Wed, 29 Apr 2015 19:08:36 GMT\r\nContent-Length: 234\r\n```\r\n\r\nHowever, we should have got the same answer as doing a HEAD request on `10.0.32.105:5601/elasticsearch/.kibana` i.e. a 404 Not Found.\r\n\r\nI think this is a bug and this is what is preventing Kibana from speaking to Elasticsearch.\r\n\r\n@thockin @roberthbailey @lavalamp \r\nThanks to who @ArtfulCoder who helped diagnose what was going on."}}