{"action":"created","issue":{"url":"https://api.github.com/repos/chihaya/chihaya/issues/65","labels_url":"https://api.github.com/repos/chihaya/chihaya/issues/65/labels{/name}","comments_url":"https://api.github.com/repos/chihaya/chihaya/issues/65/comments","events_url":"https://api.github.com/repos/chihaya/chihaya/issues/65/events","html_url":"https://github.com/chihaya/chihaya/issues/65","id":89425154,"number":65,"title":"Potential Memory Leak","user":{"login":"dtrackd","id":11036365,"avatar_url":"https://avatars.githubusercontent.com/u/11036365?v=3","gravatar_id":"","url":"https://api.github.com/users/dtrackd","html_url":"https://github.com/dtrackd","followers_url":"https://api.github.com/users/dtrackd/followers","following_url":"https://api.github.com/users/dtrackd/following{/other_user}","gists_url":"https://api.github.com/users/dtrackd/gists{/gist_id}","starred_url":"https://api.github.com/users/dtrackd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dtrackd/subscriptions","organizations_url":"https://api.github.com/users/dtrackd/orgs","repos_url":"https://api.github.com/users/dtrackd/repos","events_url":"https://api.github.com/users/dtrackd/events{/privacy}","received_events_url":"https://api.github.com/users/dtrackd/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/chihaya/chihaya/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2015-06-18T23:43:01Z","updated_at":"2015-06-19T09:14:56Z","closed_at":null,"body":"Working from develop brach on a debain 8 64bit system\r\ngo version go1.4.2 linux/amd64\r\n\r\nconfig: \r\n```\r\n{\r\n  \"create_on_announce\": true,\r\n  \"private_enabled\": false,\r\n  \"freeleech_enabled\": false,\r\n  \"purge_inactive_torrents\": false,\r\n  \"announce\": \"30m\",\r\n  \"min_announce\": \"15m\",\r\n  \"default_num_want\": 50,\r\n  \"torrent_map_shards\": 1,\r\n  \"allow_ip_spoofing\": true,\r\n  \"dual_stacked_peers\": true,\r\n  \"real_ip_header\": \"\",\r\n  \"respect_af\": false,\r\n  \"client_whitelist_enabled\": false,\r\n  \"client_whitelist\": [\"OP1011\"],\r\n  \"udp_listen_addr\": \":80\",\r\n  \"http_listen_addr\": \":8080\",\r\n  \"http_request_timeout\": \"4s\",\r\n  \"http_read_timeout\": \"4s\",\r\n  \"http_write_timeout\": \"4s\",\r\n  \"http_listen_limit\": 0,\r\n  \"driver\": \"noop\",\r\n  \"stats_buffer_size\": 0,\r\n  \"include_mem_stats\": true,\r\n  \"verbose_mem_stats\": false,\r\n  \"mem_stats_interval\": \"5s\"\r\n}\r\n```\r\n\r\nRunning the tracker seems to make the memeroy usage continue to grow at a pretty constant rate of ~3MB every 5 seconds or so under real world traffic.\r\n\r\nthe server has 1GB of memory and no other daemons running, its only used for chihaya.\r\nSo in reality we are able to play with ~850MB of memory.\r\n\r\nHere you can also see the growth in ram over periods of time (`4156` is the PID of the tracker)\r\n\r\n```\r\nroot@box:~/chihaya/bin# date; cat /proc/4156/status | grep VmSize\r\nFri Jun 19 01:30:43 CEST 2015\r\nVmSize:\t  668200 kB\r\n```\r\n\r\n(~8mins later...)\r\n```\r\nroot@box:~/chihaya/bin# date; cat /proc/4156/status | grep VmSize\r\nFri Jun 19 01:38:32 CEST 2015\r\nVmSize:\t  708752 kB\r\n```\r\n\r\n(~4mins mins after that... )\r\n```\r\nroot@box:~/chihaya/bin# date; cat /proc/4156/status | grep VmSize\r\nFri Jun 19 01:42:40 CEST 2015\r\nVmSize:\t  737320 kB\r\n```\r\n\r\nAt the point of when the tracker was using `737320 kB` of memory the stats showed as follows:\r\n\r\n```json\r\nStarted: \"2015-06-19T01:09:20.30660432+02:00\",\r\nConnections.Open: 1,\r\nConnections.Accepted: 66,\r\nBytesTransmitted: 0,\r\nRuntime.GoRoutines: 17,\r\nRequests.Handled: 1,\r\nRequests.Errored: 0,\r\nRequests.Bad: 2625430,\r\nResponseTime: {\r\nP50: 0.236624,\r\nP90: 0.236624,\r\nP95: 0.236624\r\n},\r\nTracker.Announces: 471471,\r\nTracker.Scrapes: 355241,\r\nTorrents.Size: 63044,\r\nTorrents.Added: 63044,\r\nTorrents.Removed: 0,\r\nTorrents.Reaped: 0,\r\nPeers.IPv4: {\r\nPeers: {\r\nCurrent: 246036,\r\nJoined: 298737,\r\nLeft: 52701,\r\nReaped: 0\r\n},\r\nSeeds: {\r\nCurrent: 178876,\r\nJoined: 217853,\r\nLeft: 42606,\r\nReaped: 0\r\n},\r\nCompleted: 3629\r\n},\r\nPeers.IPv6: {\r\nPeers: {\r\nCurrent: 0,\r\nJoined: 0,\r\nLeft: 0,\r\nReaped: 0\r\n},\r\nSeeds: {\r\nCurrent: 0,\r\nJoined: 0,\r\nLeft: 0,\r\nReaped: 0\r\n},\r\nCompleted: 0\r\n},\r\nMemory: {\r\nAlloc: 383058872,\r\nTotalAlloc: 18189594344,\r\nSys: 440996432,\r\nLookups: 142,\r\nMallocs: 322075692,\r\nFrees: 319396919,\r\nHeapAlloc: 383058872,\r\nHeapSys: 408485888,\r\nHeapIdle: 10510336,\r\nHeapInuse: 397975552,\r\nHeapReleased: 0,\r\nHeapObjects: 2678773,\r\nPauseTotalNs: 21756544417\r\n}\r\n}\r\n```\r\n\r\nAfter it runs out of memory obviously give the following errors (which is expected):\r\n\r\n\r\n```\r\n\r\nroot@box:~/chihaya/bin# ./chihaya -v=0 -logtostderr=true -config=config.json\r\nI0619 00:08:25.996675    4144 http.go:125] Starting HTTP on :8080\r\nI0619 00:08:25.996831    4144 udp.go:93] Starting UDP on :80\r\nI0619 00:38:25.996926    4144 tracker.go:99] Purging peers with no announces since 2015-06-18 23:38:25.996921616 +0200 CEST\r\nfatal error: runtime: out of memory\r\n\r\nruntime stack:\r\nruntime.SysMap(0xc23a9d0000, 0x100000, 0x428900, 0x8ea9f8)\r\n/usr/local/go/src/runtime/mem_linux.c:149 +0x98\r\nruntime.MHeap_SysAlloc(0x8f00e0, 0x100000, 0x430792)\r\n/usr/local/go/src/runtime/malloc.c:284 +0x124\r\nruntime.MHeap_Alloc(0x8f00e0, 0x26, 0x10100000000, 0x8)\r\n/usr/local/go/src/runtime/mheap.c:240 +0x66\r\n\r\ngoroutine 6927789 [running]:\r\nruntime.switchtoM()\r\n/usr/local/go/src/runtime/asm_amd64.s:198 fp=0xc237de56e0 sp=0xc237de56d8\r\nruntime.mallocgc(0x4bfc0, 0x718c60, 0x0, 0xc23a95c000)\r\n/usr/local/go/src/runtime/malloc.go:199 +0x9f3 fp=0xc237de5790 sp=0xc237de56e0\r\nruntime.newarray(0x718c60, 0xcaa, 0x30000)\r\n/usr/local/go/src/runtime/malloc.go:365 +0xc1 fp=0xc237de57c8 sp=0xc237de5790\r\nruntime.growslice(0x662b40, 0xc23a95c000, 0xa00, 0xa00, 0x1, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/runtime/slice.go:87 +0x2bb fp=0xc237de5828 sp=0xc237de57c8\r\ngithub.com/chihaya/chihaya/tracker/models.AppendPeer(0xc209463180, 0xc209463220, 0xc237aa8c40, 0xc219b5faa0, 0x0)\r\n/root/chihaya/src/github.com/chihaya/chihaya/tracker/models/peermap.go:195 +0x407 fp=0xc237de5900 sp=0xc237de5828\r\ngithub.com/chihaya/chihaya/tracker/models.(*PeerMap).AppendPeers(0xc208038870, 0xc23a95c000, 0x8ea, 0xa00, 0x8e8e58, 0x0, 0x0, 0xc237aa8c40, 0xfffff715, 0x0, ...)\r\n/root/chihaya/src/github.com/chihaya/chihaya/tracker/models/peermap.go:162 +0x707 fp=0xc237de5b40 sp=0xc237de5900\r\ngithub.com/chihaya/chihaya/tracker.getPeers(0xc237aa8c40, 0xc23a95c000, 0x8ea, 0xa00, 0x8e8e58, 0x0, 0x0)\r\n/root/chihaya/src/github.com/chihaya/chihaya/tracker/announce.go:298 +0x2e6 fp=0xc237de5bf0 sp=0xc237de5b40\r\ngithub.com/chihaya/chihaya/tracker.newAnnounceResponse(0xc237aa8c40, 0xc237aa8c40)\r\n/root/chihaya/src/github.com/chihaya/chihaya/tracker/announce.go:276 +0x1d5 fp=0xc237de5c90 sp=0xc237de5bf0\r\ngithub.com/chihaya/chihaya/tracker.(*Tracker).HandleAnnounce(0xc20801d260, 0xc237aa8c40, 0x7f84d79b6548, 0xc20b511fc0, 0x0, 0x0)\r\n/root/chihaya/src/github.com/chihaya/chihaya/tracker/announce.go:74 +0x76b fp=0xc237de5d78 sp=0xc237de5c90\r\ngithub.com/chihaya/chihaya/udp.(*Server).handlePacket(0xc20803ae40, 0xc208267000, 0x6d, 0x800, 0xc20990b530, 0x0, 0x0, 0x0, 0x7592d0, 0x8)\r\n/root/chihaya/src/github.com/chihaya/chihaya/udp/protocol.go:102 +0x630 fp=0xc237de5ea8 sp=0xc237de5d78\r\ngithub.com/chihaya/chihaya/udp.func·002()\r\n/root/chihaya/src/github.com/chihaya/chihaya/udp/udp.go:74 +0xab fp=0xc237de5fe0 sp=0xc237de5ea8\r\nruntime.goexit()\r\n/usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc237de5fe8 sp=0xc237de5fe0\r\ncreated by github.com/chihaya/chihaya/udp.(*Server).serve\r\n/root/chihaya/src/github.com/chihaya/chihaya/udp/udp.go:85 +0x895\r\n\r\ngoroutine 1 [chan receive, 42 minutes]:\r\ngithub.com/chihaya/chihaya.Boot()\r\n/root/chihaya/src/github.com/chihaya/chihaya/chihaya.go:118 +0x1090\r\nmain.main()\r\n/root/chihaya/src/github.com/chihaya/chihaya/cmd/chihaya/main.go:10 +0x1b\r\n\r\ngoroutine 5 [chan receive]:\r\ngithub.com/golang/glog.(*loggingT).flushDaemon(0x8e06a0)\r\n/root/chihaya/src/github.com/chihaya/chihaya/Godeps/_workspace/src/github.com/golang/glog/glog.go:879 +0x78\r\ncreated by github.com/golang/glog.init·1\r\n/root/chihaya/src/github.com/chihaya/chihaya/Godeps/_workspace/src/github.com/golang/glog/glog.go:410 +0x2a7\r\n\r\ngoroutine 17 [syscall, 42 minutes, locked to thread]:\r\nruntime.goexit()\r\n/usr/local/go/src/runtime/asm_amd64.s:2232 +0x1\r\n\r\ngoroutine 6 [syscall, 42 minutes]:\r\nos/signal.loop()\r\n/usr/local/go/src/os/signal/signal_unix.go:21 +0x1f\r\ncreated by os/signal.init·1\r\n/usr/local/go/src/os/signal/signal_unix.go:27 +0x35\r\n\r\ngoroutine 8 [runnable]:\r\ngithub.com/chihaya/chihaya/stats.(*Stats).handleEvents(0xc208082160)\r\n/root/chihaya/src/github.com/chihaya/chihaya/stats/stats.go:168 +0x2de\r\ncreated by github.com/chihaya/chihaya/stats.New\r\n/root/chihaya/src/github.com/chihaya/chihaya/stats/stats.go:129 +0x721\r\n\r\ngoroutine 9 [chan receive, 12 minutes]:\r\ngithub.com/chihaya/chihaya/tracker.(*Tracker).purgeInactivePeers(0xc20801d260, 0xc208050200, 0x34630b8a000, 0x1a3185c5000)\r\n/root/chihaya/src/github.com/chihaya/chihaya/tracker/tracker.go:97 +0x8a\r\ncreated by github.com/chihaya/chihaya/tracker.New\r\n/root/chihaya/src/github.com/chihaya/chihaya/tracker/tracker.go:54 +0x183\r\n\r\ngoroutine 10 [IO wait, 10 minutes]:\r\nnet.(*pollDesc).Wait(0xc20800e290, 0x72, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47\r\nnet.(*pollDesc).WaitRead(0xc20800e290, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43\r\nnet.(*netFD).accept(0xc20800e230, 0x0, 0x7f84d79b4b60, 0xc217d86550)\r\n/usr/local/go/src/net/fd_unix.go:419 +0x40b\r\nnet.(*TCPListener).AcceptTCP(0xc2080360c0, 0xc208057c08, 0x0, 0x0)\r\n/usr/local/go/src/net/tcpsock_posix.go:234 +0x4e\r\nnet.(*TCPListener).Accept(0xc2080360c0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/tcpsock_posix.go:244 +0x4c\r\nnet/http.(*Server).Serve(0xc2080505a0, 0x7f84d79b53b0, 0xc2080360c0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/server.go:1728 +0x92\r\ngithub.com/tylerb/graceful.(*Server).Serve(0xc208050540, 0x7f84d79b53b0, 0xc2080360c0, 0x0, 0x0)\r\n/root/chihaya/src/github.com/chihaya/chihaya/Godeps/_workspace/src/github.com/tylerb/graceful/graceful.go:197 +0x446\r\ngithub.com/tylerb/graceful.(*Server).ListenAndServe(0xc208050540, 0x0, 0x0)\r\n/root/chihaya/src/github.com/chihaya/chihaya/Godeps/_workspace/src/github.com/tylerb/graceful/graceful.go:110 +0x211\r\ngithub.com/chihaya/chihaya/http.(*Server).Serve(0xc20801d2a0, 0xc208028d03, 0x5)\r\n/root/chihaya/src/github.com/chihaya/chihaya/http/http.go:149 +0x612\r\ngithub.com/chihaya/chihaya.func·002()\r\n/root/chihaya/src/github.com/chihaya/chihaya/chihaya.go:94 +0x79\r\ncreated by github.com/chihaya/chihaya.Boot\r\n/root/chihaya/src/github.com/chihaya/chihaya/chihaya.go:95 +0xbac\r\n\r\ngoroutine 11 [IO wait]:\r\nnet.(*pollDesc).Wait(0xc20800e300, 0x72, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47\r\nnet.(*pollDesc).WaitRead(0xc20800e300, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43\r\nnet.(*netFD).readFrom(0xc20800e2a0, 0xc20844c800, 0x800, 0x800, 0x0, 0x0, 0x0, 0x7f84d79b4b60, 0xc23910a310)\r\n/usr/local/go/src/net/fd_unix.go:269 +0x4a1\r\nnet.(*UDPConn).ReadFromUDP(0xc2080360f8, 0xc20844c800, 0x800, 0x800, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/udpsock_posix.go:67 +0x124\r\ngithub.com/chihaya/chihaya/udp.(*Server).serve(0xc20803ae40, 0xc208028d00, 0x3, 0x0, 0x0)\r\n/root/chihaya/src/github.com/chihaya/chihaya/udp/udp.go:61 +0x5d7\r\ngithub.com/chihaya/chihaya/udp.(*Server).Serve(0xc20803ae40, 0xc208028d00, 0x3)\r\n/root/chihaya/src/github.com/chihaya/chihaya/udp/udp.go:102 +0x229\r\ngithub.com/chihaya/chihaya.func·003()\r\n/root/chihaya/src/github.com/chihaya/chihaya/chihaya.go:105 +0x79\r\ncreated by github.com/chihaya/chihaya.Boot\r\n/root/chihaya/src/github.com/chihaya/chihaya/chihaya.go:106 +0xe21\r\n\r\ngoroutine 12 [semacquire, 42 minutes]:\r\nsync.(*WaitGroup).Wait(0xc20801d280)\r\n/usr/local/go/src/sync/waitgroup.go:132 +0x169\r\ngithub.com/chihaya/chihaya.func·004()\r\n/root/chihaya/src/github.com/chihaya/chihaya/chihaya.go:113 +0x30\r\ncreated by github.com/chihaya/chihaya.Boot\r\n/root/chihaya/src/github.com/chihaya/chihaya/chihaya.go:116 +0x1065\r\n\r\ngoroutine 13 [select, 10 minutes]:\r\ngithub.com/tylerb/graceful.(*Server).manageConnections(0xc208050540, 0xc208050600, 0xc208050660, 0xc2080506c0, 0xc208050720)\r\n/root/chihaya/src/github.com/chihaya/chihaya/Godeps/_workspace/src/github.com/tylerb/graceful/graceful.go:234 +0x3da\r\ncreated by github.com/tylerb/graceful.(*Server).Serve\r\n/root/chihaya/src/github.com/chihaya/chihaya/Godeps/_workspace/src/github.com/tylerb/graceful/graceful.go:184 +0x244\r\n\r\ngoroutine 14 [chan receive, 42 minutes]:\r\ngithub.com/tylerb/graceful.(*Server).handleInterrupt(0xc208050540, 0xc208050780, 0x7f84d79b53b0, 0xc2080360c0)\r\n/root/chihaya/src/github.com/chihaya/chihaya/Godeps/_workspace/src/github.com/tylerb/graceful/graceful.go:269 +0x3d\r\ncreated by github.com/tylerb/graceful.(*Server).Serve\r\n/root/chihaya/src/github.com/chihaya/chihaya/Godeps/_workspace/src/github.com/tylerb/graceful/graceful.go:193 +0x416\r\n\r\ngoroutine 15 [chan receive, 42 minutes]:\r\ngithub.com/chihaya/chihaya/udp.func·003()\r\n/root/chihaya/src/github.com/chihaya/chihaya/udp/udp.go:97 +0x7e\r\ncreated by github.com/chihaya/chihaya/udp.(*Server).Serve\r\n/root/chihaya/src/github.com/chihaya/chihaya/udp/udp.go:100 +0x1fc\r\n```"},"comment":{"url":"https://api.github.com/repos/chihaya/chihaya/issues/comments/113442321","html_url":"https://github.com/chihaya/chihaya/issues/65#issuecomment-113442321","issue_url":"https://api.github.com/repos/chihaya/chihaya/issues/65","id":113442321,"user":{"login":"cooperlees","id":3005596,"avatar_url":"https://avatars.githubusercontent.com/u/3005596?v=3","gravatar_id":"","url":"https://api.github.com/users/cooperlees","html_url":"https://github.com/cooperlees","followers_url":"https://api.github.com/users/cooperlees/followers","following_url":"https://api.github.com/users/cooperlees/following{/other_user}","gists_url":"https://api.github.com/users/cooperlees/gists{/gist_id}","starred_url":"https://api.github.com/users/cooperlees/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cooperlees/subscriptions","organizations_url":"https://api.github.com/users/cooperlees/orgs","repos_url":"https://api.github.com/users/cooperlees/repos","events_url":"https://api.github.com/users/cooperlees/events{/privacy}","received_events_url":"https://api.github.com/users/cooperlees/received_events","type":"User","site_admin":false},"created_at":"2015-06-19T09:14:56Z","updated_at":"2015-06-19T09:14:56Z","body":"I'm going to try and get this code in front of some real Traffic @ FB this week and will try help debug any issues I find :)\n\nCooper\n\nSent from my iPhone\n\n> On Jun 19, 2015, at 3:13 AM, dtrackd <notifications@github.com> wrote:\n> \n> @jzelinskie is there more info I can maybe provide in order to help detect where this issue is coming from? I am not entirely sure how to execute the internal test so I am just putting this in front off real tracker traffic.\n> \n> —\n> Reply to this email directly or view it on GitHub.\n> \n"}}