{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16384","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16384/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16384/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16384/events","html_url":"https://github.com/kubernetes/kubernetes/pull/16384","id":113684158,"number":16384,"title":"Large amount of memory churn in etcd helper prefixing keys","user":{"login":"derekwaynecarr","id":6233452,"avatar_url":"https://avatars.githubusercontent.com/u/6233452?v=3","gravatar_id":"","url":"https://api.github.com/users/derekwaynecarr","html_url":"https://github.com/derekwaynecarr","followers_url":"https://api.github.com/users/derekwaynecarr/followers","following_url":"https://api.github.com/users/derekwaynecarr/following{/other_user}","gists_url":"https://api.github.com/users/derekwaynecarr/gists{/gist_id}","starred_url":"https://api.github.com/users/derekwaynecarr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekwaynecarr/subscriptions","organizations_url":"https://api.github.com/users/derekwaynecarr/orgs","repos_url":"https://api.github.com/users/derekwaynecarr/repos","events_url":"https://api.github.com/users/derekwaynecarr/events{/privacy}","received_events_url":"https://api.github.com/users/derekwaynecarr/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/cla:%20yes","name":"cla: yes","color":"bfe5bf"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/lgtm","name":"lgtm","color":"15dd18"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/size/XS","name":"size/XS","color":"009900"}],"state":"open","locked":false,"assignee":{"login":"thockin","id":5595220,"avatar_url":"https://avatars.githubusercontent.com/u/5595220?v=3","gravatar_id":"","url":"https://api.github.com/users/thockin","html_url":"https://github.com/thockin","followers_url":"https://api.github.com/users/thockin/followers","following_url":"https://api.github.com/users/thockin/following{/other_user}","gists_url":"https://api.github.com/users/thockin/gists{/gist_id}","starred_url":"https://api.github.com/users/thockin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thockin/subscriptions","organizations_url":"https://api.github.com/users/thockin/orgs","repos_url":"https://api.github.com/users/thockin/repos","events_url":"https://api.github.com/users/thockin/events{/privacy}","received_events_url":"https://api.github.com/users/thockin/received_events","type":"User","site_admin":false},"milestone":null,"comments":4,"created_at":"2015-10-27T20:40:04Z","updated_at":"2015-10-28T08:44:27Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/16384","html_url":"https://github.com/kubernetes/kubernetes/pull/16384","diff_url":"https://github.com/kubernetes/kubernetes/pull/16384.diff","patch_url":"https://github.com/kubernetes/kubernetes/pull/16384.patch"},"body":"```shell\r\n$ go tool pprof --alloc_space ./_output/local/go/bin/openshift /home/decarr/Downloads/meminfo/mem.pprof \r\nEntering interactive mode (type \"help\" for commands)\r\n(pprof) top\r\n189.15GB of 360.87GB total (52.41%)\r\nDropped 2118 nodes (cum <= 1.80GB)\r\nShowing top 10 nodes out of 280 (cum >= 6.68GB)\r\n      flat  flat%   sum%        cum   cum%\r\n   54.21GB 15.02% 15.02%    54.21GB 15.02%  bytes.(*Buffer).ReadRune\r\n   33.39GB  9.25% 24.27%    33.39GB  9.25%  github.com/ugorji/go/codec.(*encFnInfo).kInt\r\n   23.55GB  6.52% 30.80%    23.55GB  6.52%  reflect.methodReceiver\r\n   19.33GB  5.36% 36.15%    25.40GB  7.04%  math/big.divisors\r\n   11.77GB  3.26% 39.42%    11.77GB  3.26%  math/big.nat.mul\r\n   11.21GB  3.11% 42.52%    11.21GB  3.11%  reflect.(*structType).FieldByNameFunc\r\n   11.04GB  3.06% 45.58%    21.03GB  5.83%  encoding/json.(*decodeState).literalStore\r\n    9.73GB  2.70% 48.28%    44.67GB 12.38%  type..hash.github.com/ugorji/go/codec.encFnInfo\r\n    8.25GB  2.29% 50.56%    42.76GB 11.85%  k8s.io/kubernetes/pkg/storage/etcd.(*etcdHelper).prefixEtcdKey\r\n    6.68GB  1.85% 52.41%     6.68GB  1.85%  github.com/openshift/origin/pkg/authorization/cache.(*readOnlyAuthorizationCache).ListClusterPolicyBindings\r\n(pprof) list prefixEtcdKey\r\nTotal: 360.87GB\r\nROUTINE ======================== k8s.io/kubernetes/pkg/storage/etcd.(*etcdHelper).prefixEtcdKey in /home/decarr/go/src/github.com/openshift/ose/Godeps/_workspace/src/k8s.io/kubernetes/pkg/storage/etcd/etcd_helper.go\r\n    8.25GB       51GB (flat, cum) 14.13% of Total\r\n         .          .    446:\t\t_, _, err = h.extractObj(response, err, ptrToType, false, false)\r\n         .          .    447:\t\treturn err\r\n         .          .    448:\t}\r\n         .          .    449:}\r\n         .          .    450:\r\n         .     1.41MB    451:func (h *etcdHelper) prefixEtcdKey(key string) string {\r\n         .    42.75GB    452:\tif strings.HasPrefix(key, path.Join(\"/\", h.pathPrefix)) {\r\n         .          .    453:\t\treturn key\r\n         .          .    454:\t}\r\n    8.25GB     8.25GB    455:\treturn path.Join(\"/\", h.pathPrefix, key)\r\n         .          .    456:}\r\n         .          .    457:\r\n         .          .    458:// etcdCache defines interface used for caching objects stored in etcd. Objects are keyed by\r\n         .          .    459:// their Node.ModifiedIndex, which is unique across all types.\r\n         .          .    460:// All implementations must be thread-safe.\r\n```\r\n\r\nFix is to only join with / with pathPrefix one time."},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/151767478","html_url":"https://github.com/kubernetes/kubernetes/pull/16384#issuecomment-151767478","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16384","id":151767478,"user":{"login":"wojtek-t","id":10743879,"avatar_url":"https://avatars.githubusercontent.com/u/10743879?v=3","gravatar_id":"","url":"https://api.github.com/users/wojtek-t","html_url":"https://github.com/wojtek-t","followers_url":"https://api.github.com/users/wojtek-t/followers","following_url":"https://api.github.com/users/wojtek-t/following{/other_user}","gists_url":"https://api.github.com/users/wojtek-t/gists{/gist_id}","starred_url":"https://api.github.com/users/wojtek-t/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wojtek-t/subscriptions","organizations_url":"https://api.github.com/users/wojtek-t/orgs","repos_url":"https://api.github.com/users/wojtek-t/repos","events_url":"https://api.github.com/users/wojtek-t/events{/privacy}","received_events_url":"https://api.github.com/users/wojtek-t/received_events","type":"User","site_admin":false},"created_at":"2015-10-28T08:44:26Z","updated_at":"2015-10-28T08:44:26Z","body":"@derekwaynecarr - thanks a lot for the fix!\r\n\r\n@k8s-bot unit test this please"}}