{"action":"created","issue":{"url":"https://api.github.com/repos/openshift/origin/issues/5106","labels_url":"https://api.github.com/repos/openshift/origin/issues/5106/labels{/name}","comments_url":"https://api.github.com/repos/openshift/origin/issues/5106/comments","events_url":"https://api.github.com/repos/openshift/origin/issues/5106/events","html_url":"https://github.com/openshift/origin/issues/5106","id":111235498,"number":5106,"title":"tcp resets initiated by openshift-node","user":{"login":"jeremyeder","id":1919640,"avatar_url":"https://avatars.githubusercontent.com/u/1919640?v=3","gravatar_id":"","url":"https://api.github.com/users/jeremyeder","html_url":"https://github.com/jeremyeder","followers_url":"https://api.github.com/users/jeremyeder/followers","following_url":"https://api.github.com/users/jeremyeder/following{/other_user}","gists_url":"https://api.github.com/users/jeremyeder/gists{/gist_id}","starred_url":"https://api.github.com/users/jeremyeder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeremyeder/subscriptions","organizations_url":"https://api.github.com/users/jeremyeder/orgs","repos_url":"https://api.github.com/users/jeremyeder/repos","events_url":"https://api.github.com/users/jeremyeder/events{/privacy}","received_events_url":"https://api.github.com/users/jeremyeder/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":22,"created_at":"2015-10-13T18:02:22Z","updated_at":"2015-10-15T01:42:46Z","closed_at":null,"body":"We have noted an excessive number of tcp connection RST packets when using openshift 3.0 to port 8443 on the master.  On a 100 node cluster, ~20 RST's per second (according to tcpdump on the master).\r\n\r\nIf I stop the openshift-node processes, the resets go away.  There are connection resets even within a single VM that has both openshift-master and openshift-node running (just wanted to verify it wasn't load-related, EC2 related or sdn).  Wasn't able to find any reference to this in google-containers mailing list or kubernetes github issues.\r\n\r\nTo reproduce this, I'm using RHEL7.2 in EC2 and OpenShift 3.0.900, installed with openshift-ansible.\r\nYou can either profile using pprof or just look at ```netstat -s|grep reset```\r\n\r\nThis profile was taken on the master of a 100 node cluster.  I let it run for about 25 minutes.\r\n```\r\n(pprof) top10000 -cum\r\n1201.47s of 1487.59s total (80.77%)\r\nDropped 1947 nodes (cum <= 7.44s)\r\n      flat  flat%   sum%        cum   cum%\r\n     0.01s 0.00067% 0.00067%   1238.11s 83.23%  runtime.goexit\r\n     0.07s 0.0047% 0.0054%    976.03s 65.61%  crypto/tls.(*Conn).Handshake\r\n     0.10s 0.0067% 0.012%    755.21s 50.77%  net/http.(*conn).serve\r\n     0.04s 0.0027% 0.015%    693.91s 46.65%  crypto/rsa.SignPKCS1v15\r\n     0.07s 0.0047% 0.019%    693.71s 46.63%  crypto/rsa.decrypt\r\n     0.25s 0.017% 0.036%    640.78s 43.08%  math/big.nat.expNN\r\n     0.05s 0.0034%  0.04%    639.68s 43.00%  math/big.(*Int).Exp\r\n     0.01s 0.00067%  0.04%    618.90s 41.60%  crypto/tls.(*Conn).serverHandshake\r\n     0.02s 0.0013% 0.042%    614.45s 41.31%  crypto/tls.(*serverHandshakeState).doFullHandshake\r\n     4.21s  0.28%  0.32%    596.06s 40.07%  math/big.nat.expNNWindowed\r\n     4.70s  0.32%  0.64%    501.48s 33.71%  math/big.nat.div\r\n   120.26s  8.08%  8.72%    494.64s 33.25%  math/big.nat.divLarge\r\n     0.03s 0.002%  8.73%    487.08s 32.74%  crypto/tls.(*ecdheKeyAgreement).generateServerKeyExchange\r\n     0.01s 0.00067%  8.73%    356.75s 23.98%  net/http.func·022\r\n     0.03s 0.002%  8.73%    356.71s 23.98%  crypto/tls.(*Conn).clientHandshake\r\n     0.09s 0.0061%  8.74%    354.18s 23.81%  crypto/tls.(*clientHandshakeState).doFullHandshake\r\n         0     0%  8.74%    244.64s 16.45%  crypto/rsa.(*PrivateKey).Sign\r\n```"},"comment":{"url":"https://api.github.com/repos/openshift/origin/issues/comments/148251791","html_url":"https://github.com/openshift/origin/issues/5106#issuecomment-148251791","issue_url":"https://api.github.com/repos/openshift/origin/issues/5106","id":148251791,"user":{"login":"smarterclayton","id":1163175,"avatar_url":"https://avatars.githubusercontent.com/u/1163175?v=3","gravatar_id":"","url":"https://api.github.com/users/smarterclayton","html_url":"https://github.com/smarterclayton","followers_url":"https://api.github.com/users/smarterclayton/followers","following_url":"https://api.github.com/users/smarterclayton/following{/other_user}","gists_url":"https://api.github.com/users/smarterclayton/gists{/gist_id}","starred_url":"https://api.github.com/users/smarterclayton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarterclayton/subscriptions","organizations_url":"https://api.github.com/users/smarterclayton/orgs","repos_url":"https://api.github.com/users/smarterclayton/repos","events_url":"https://api.github.com/users/smarterclayton/events{/privacy}","received_events_url":"https://api.github.com/users/smarterclayton/received_events","type":"User","site_admin":false},"created_at":"2015-10-15T01:42:46Z","updated_at":"2015-10-15T01:42:46Z","body":"That's getting fixed now...\n\nOn Wed, Oct 14, 2015 at 9:41 PM, Clayton Coleman <ccoleman@redhat.com>\nwrote:\n\n> Oh wow - the proxy has a resync interval of 30 seconds.\n>\n> On Wed, Oct 14, 2015 at 8:04 PM, Jeremy Eder <notifications@github.com>\n> wrote:\n>\n>> Seconds -- the graph is 300 seconds long in total, with a sampling\n>> interval of every 3-seconds.\n>>\n>> —\n>> Reply to this email directly or view it on GitHub\n>> <https://github.com/openshift/origin/issues/5106#issuecomment-148237343>.\n>>\n>\n>\n"}}