{"action":"created","issue":{"url":"https://api.github.com/repos/openshift/origin/issues/5355","labels_url":"https://api.github.com/repos/openshift/origin/issues/5355/labels{/name}","comments_url":"https://api.github.com/repos/openshift/origin/issues/5355/comments","events_url":"https://api.github.com/repos/openshift/origin/issues/5355/events","html_url":"https://github.com/openshift/origin/issues/5355","id":112992105,"number":5355,"title":"OpenShift eating my CPU","user":{"login":"mfojtik","id":44136,"avatar_url":"https://avatars.githubusercontent.com/u/44136?v=3","gravatar_id":"","url":"https://api.github.com/users/mfojtik","html_url":"https://github.com/mfojtik","followers_url":"https://api.github.com/users/mfojtik/followers","following_url":"https://api.github.com/users/mfojtik/following{/other_user}","gists_url":"https://api.github.com/users/mfojtik/gists{/gist_id}","starred_url":"https://api.github.com/users/mfojtik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mfojtik/subscriptions","organizations_url":"https://api.github.com/users/mfojtik/orgs","repos_url":"https://api.github.com/users/mfojtik/repos","events_url":"https://api.github.com/users/mfojtik/events{/privacy}","received_events_url":"https://api.github.com/users/mfojtik/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/openshift/origin/labels/area/performance","name":"area/performance","color":"006b75"},{"url":"https://api.github.com/repos/openshift/origin/labels/kind/bug","name":"kind/bug","color":"fc2929"},{"url":"https://api.github.com/repos/openshift/origin/labels/priority/P1","name":"priority/P1","color":"eb6420"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-10-23T10:14:31Z","updated_at":"2015-10-26T16:45:20Z","closed_at":null,"body":"When I run the latest OSE image in Docker container, everything works normally but after some time and after I start some builds and deploy stuff the \"openshift\" process start eating >100% CPU and it keeps doing that until I stop the container. This sometimes takes couple minutes to happen, sometimes it happens in few seconds...\r\n\r\nHere is heap dump: http://omicron.mifo.sk/heap.txt\r\nHere is goroutine dump: http://omicron.mifo.sk/go.txt\r\nAnd here is loglevel=5 openshift log: http://omicron.mifo.sk/ose.txt\r\n\r\n```console\r\n[vagrant@localhost ~]$ docker run openshift3/ose version\r\nopenshift v3.0.2.902\r\nkubernetes v1.2.0-alpha.1-1107-g4c8e6f4\r\netcd 2.1.2\r\n```\r\n\r\nThe container is started like this (on the RHEL 7.1  VM with docker 1.7.1:\r\n\r\n```console\r\n$ docker run -d --name \"ose\" --privileged --net=host --pid=host \\\r\n  -v /:/rootfs:ro \\\r\n  -v /var/run:/var/run:rw \\\r\n  -v /sys:/sys:ro \\\r\n  -v /var/lib/docker:/var/lib/docker:rw \\\r\n  -v #{ORIGIN_DIR}/openshift.local.volumes:#{ORIGIN_DIR}/openshift.local.volumes:z \\\r\n  -v #{ORIGIN_DIR}/openshift.local.config:#{ORIGIN_DIR}/openshift.local.config:z \\\r\n  -v #{ORIGIN_DIR}/openshift.local.etcd:#{ORIGIN_DIR}/openshift.local.etcd:z \\\r\n  openshift3/ose start \\\r\n   --master=\"https://#{PUBLIC_ADDRESS}:8443\" \\\r\n   --etcd-dir=\"#{ORIGIN_DIR}/openshift.local.etcd\"\r\n```\r\n\r\n```console\r\nPID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND\r\n24806 root      20   0 1456368 240616  19092 S 193.4 12.8   2:46.28 openshift \r\n```\r\nThe registry and router are also installed and running properly.\r\n\r\nIt seems like something got stuck on decoding/encoding JSON, but I can't see anything in the container logs...\r\n\r\n@smarterclayton @deads2k FYI\r\n\r\n(since this is very reproducible and it is happening with the OSE Docker image, this should get P1?)"},"comment":{"url":"https://api.github.com/repos/openshift/origin/issues/comments/151204821","html_url":"https://github.com/openshift/origin/issues/5355#issuecomment-151204821","issue_url":"https://api.github.com/repos/openshift/origin/issues/5355","id":151204821,"user":{"login":"smarterclayton","id":1163175,"avatar_url":"https://avatars.githubusercontent.com/u/1163175?v=3","gravatar_id":"","url":"https://api.github.com/users/smarterclayton","html_url":"https://github.com/smarterclayton","followers_url":"https://api.github.com/users/smarterclayton/followers","following_url":"https://api.github.com/users/smarterclayton/following{/other_user}","gists_url":"https://api.github.com/users/smarterclayton/gists{/gist_id}","starred_url":"https://api.github.com/users/smarterclayton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarterclayton/subscriptions","organizations_url":"https://api.github.com/users/smarterclayton/orgs","repos_url":"https://api.github.com/users/smarterclayton/repos","events_url":"https://api.github.com/users/smarterclayton/events{/privacy}","received_events_url":"https://api.github.com/users/smarterclayton/received_events","type":"User","site_admin":false},"created_at":"2015-10-26T16:45:19Z","updated_at":"2015-10-26T16:45:19Z","body":"Need to isolate where this is caused by a websocket client (browser vs not)\nor the server.\n\nOn Mon, Oct 26, 2015 at 12:33 PM, Clayton Coleman <ccoleman@redhat.com>\nwrote:\n\n> Suspect this is a hot read loop for zero byte reads in the websocket code.\n>\n> On Mon, Oct 26, 2015 at 12:30 PM, Jordan Liggitt <notifications@github.com\n> > wrote:\n>\n>> Captured profile by starting with OPENSHIFT_PROFILE=web, and grabbing\n>> with go tool pprof http://localhost:6060/debug/pprof/profile when at\n>> 250% CPU:\n>>\n>> [image: profile]\n>> <https://cloud.githubusercontent.com/assets/980082/10735223/5cf77118-7bdd-11e5-889c-0727910bec5f.png>\n>>\n>> —\n>> Reply to this email directly or view it on GitHub\n>> <https://github.com/openshift/origin/issues/5355#issuecomment-151198038>.\n>>\n>\n>\n"}}