{"action":"created","issue":{"url":"https://api.github.com/repos/golang/go/issues/13287","labels_url":"https://api.github.com/repos/golang/go/issues/13287/labels{/name}","comments_url":"https://api.github.com/repos/golang/go/issues/13287/comments","events_url":"https://api.github.com/repos/golang/go/issues/13287/events","html_url":"https://github.com/golang/go/issues/13287","id":117306541,"number":13287,"title":"runtime: fatal error: free list corrupted (3)","user":{"login":"msackman","id":389910,"avatar_url":"https://avatars.githubusercontent.com/u/389910?v=3","gravatar_id":"","url":"https://api.github.com/users/msackman","html_url":"https://github.com/msackman","followers_url":"https://api.github.com/users/msackman/followers","following_url":"https://api.github.com/users/msackman/following{/other_user}","gists_url":"https://api.github.com/users/msackman/gists{/gist_id}","starred_url":"https://api.github.com/users/msackman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/msackman/subscriptions","organizations_url":"https://api.github.com/users/msackman/orgs","repos_url":"https://api.github.com/users/msackman/repos","events_url":"https://api.github.com/users/msackman/events{/privacy}","received_events_url":"https://api.github.com/users/msackman/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/golang/go/milestones/27","html_url":"https://github.com/golang/go/milestones/Go1.5.2","labels_url":"https://api.github.com/repos/golang/go/milestones/27/labels","id":1295987,"number":27,"title":"Go1.5.2","description":"Go 1.5.2 release.","creator":{"login":"ianlancetaylor","id":3194333,"avatar_url":"https://avatars.githubusercontent.com/u/3194333?v=3","gravatar_id":"","url":"https://api.github.com/users/ianlancetaylor","html_url":"https://github.com/ianlancetaylor","followers_url":"https://api.github.com/users/ianlancetaylor/followers","following_url":"https://api.github.com/users/ianlancetaylor/following{/other_user}","gists_url":"https://api.github.com/users/ianlancetaylor/gists{/gist_id}","starred_url":"https://api.github.com/users/ianlancetaylor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ianlancetaylor/subscriptions","organizations_url":"https://api.github.com/users/ianlancetaylor/orgs","repos_url":"https://api.github.com/users/ianlancetaylor/repos","events_url":"https://api.github.com/users/ianlancetaylor/events{/privacy}","received_events_url":"https://api.github.com/users/ianlancetaylor/received_events","type":"User","site_admin":false},"open_issues":12,"closed_issues":16,"state":"open","created_at":"2015-09-08T17:31:04Z","updated_at":"2015-11-17T13:28:37Z","due_on":"2015-10-31T07:00:00Z","closed_at":null},"comments":2,"created_at":"2015-11-17T08:25:59Z","updated_at":"2015-11-17T13:28:54Z","closed_at":null,"body":"Possible dup of #11411 and #12879\r\n\r\n    runtime: free list of span 0x7ea0832a2d40:\r\n    0xc87618ea00 -> 0x80c87618fa40 (BAD)\r\n    fatal error: free list corrupted\r\n    \r\n    runtime stack:\r\n    runtime.throw(0x85ecc0, 0x13)\r\n        /home/matthew/src/golang/go1.5.1/src/runtime/panic.go:527 +0x90\r\n    runtime.mSpan_Sweep(0x7ea0832a2d40, 0x18100000100, 0xc80002a801)\r\n        /home/matthew/src/golang/go1.5.1/src/runtime/mgcsweep.go:186 +0x800\r\n    runtime.sweepone(0x439b12)\r\n        /home/matthew/src/golang/go1.5.1/src/runtime/mgcsweep.go:97 +0x154\r\n    runtime.gosweepone.func1()\r\n        /home/matthew/src/golang/go1.5.1/src/runtime/mgcsweep.go:109 +0x21\r\n    runtime.systemstack(0xc820023500)\r\n        /home/matthew/src/golang/go1.5.1/src/runtime/asm_amd64.s:262 +0x79\r\n    runtime.mstart()\r\n        /home/matthew/src/golang/go1.5.1/src/runtime/proc1.go:674\r\n\r\n    > go version\r\n    go version go1.5.1 linux/amd64\r\n\r\nThe software is a distributed database server. At the time, there were 3 servers running, all connected to each other (and all running on the same machine). All the servers are running the exact same binary. Clients would connect, run some tests, disconnect. I was asleep.\r\n\r\nFrom the rest of the stack traces, it looks as though the panic happened 22 minutes after the server was started and 14 minutes after the last client disconnect (test had passed). All 3 connected servers would have been idle at this point.\r\n\r\nOf the 3 servers, one (server1) failed with the above, one survived (server2) until the morning when I found it, and the other (server3) appears to have failed at exactly the same time with:\r\n\r\n    fatal error: C malloc failed\r\n    \r\n    goroutine 77 [running]:\r\n    runtime.throw(0x8241f0, 0xf)\r\n        /home/matthew/src/golang/go1.5.1/src/runtime/panic.go:527 +0x90 fp=0xc8e6765408 sp=0xc8e67653f0\r\n    runtime.cmalloc(0xa, 0x409617)\r\n        /home/matthew/src/golang/go1.5.1/src/runtime/cgocall.go:148 +0x68 fp=0xc8e6765438 sp=0xc8e6765408\r\n    net._Cfunc_CString(0xc8202cb800, 0x9, 0xc8e67654e8)\r\n        ??:0 +0x28 fp=0xc8e67654a8 sp=0xc8e6765438\r\n    net.cgoLookupIPCNAME(0xc8202cb800, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n        /home/matthew/src/golang/go1.5.1/src/net/cgo_unix.go:108 +0x13c fp=0xc8e67655d0 sp=0xc8e67654a8\r\n    net.cgoLookupIP(0xc8202cb800, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n        /home/matthew/src/golang/go1.5.1/src/net/cgo_unix.go:163 +0x56 fp=0xc8e6765628 sp=0xc8e67655d0\r\n    net.lookupIP(0xc8202cb800, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n        /home/matthew/src/golang/go1.5.1/src/net/lookup_unix.go:67 +0x94 fp=0xc8e6765698 sp=0xc8e6765628\r\n    net.glob.func15(0x8d0300, 0xc8202cb800, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n        /home/matthew/src/golang/go1.5.1/src/net/hook.go:10 +0x4d fp=0xc8e67656d8 sp=0xc8e6765698\r\n    net.lookupIPMerge.func1(0x0, 0x0, 0x0, 0x0)\r\n        /home/matthew/src/golang/go1.5.1/src/net/lookup.go:68 +0x71 fp=0xc8e6765758 sp=0xc8e67656d8\r\n    internal/singleflight.(*Group).doCall(0xc2c570, 0xc8764ae230, 0xc8202cb800, 0x9, 0xc8e6765950)\r\n        /home/matthew/src/golang/go1.5.1/src/internal/singleflight/singleflight.go:93 +0x2c fp=0xc8e6765808 sp=0xc8e6765758\r\n    internal/singleflight.(*Group).Do(0xc2c570, 0xc8202cb800, 0x9, 0xc8e6765950, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n        /home/matthew/src/golang/go1.5.1/src/internal/singleflight/singleflight.go:63 +0x284 fp=0xc8e6765878 sp=0xc8e6765808\r\n    net.lookupIPMerge(0xc8202cb800, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n        /home/matthew/src/golang/go1.5.1/src/net/lookup.go:69 +0x9b fp=0xc8e6765988 sp=0xc8e6765878\r\n    net.lookupIPDeadline(0xc8202cb800, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n        /home/matthew/src/golang/go1.5.1/src/net/lookup.go:91 +0xde fp=0xc8e6765bc0 sp=0xc8e6765988\r\n    net.internetAddrList(0x821bd8, 0x3, 0xc8202cb800, 0xf, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n        /home/matthew/src/golang/go1.5.1/src/net/ipsock.go:252 +0x6ee fp=0xc8e6765d28 sp=0xc8e6765bc0\r\n    net.ResolveTCPAddr(0x821bd8, 0x3, 0xc8202cb800, 0xf, 0x5d8abc, 0x0, 0x0)\r\n        /home/matthew/src/golang/go1.5.1/src/net/tcpsock.go:56 +0x11b fp=0xc8e6765de8 sp=0xc8e6765d28\r\n    ....my code.\r\n\r\nThere could have been memory pressure at the time, but I find it unlikely give indications are it's some 14 mins after the last test finished (and passed). Syslog does not show any activity by the kernel OOM process killer.\r\n\r\nThis server (server3) would only have been in this code because it was trying to reconnect to server1 after server1 had failed. So this would have been exactly 5 seconds after server1 had failed. With server1 having failed, I can't believe there really could have been any memory pressure in the system.\r\n\r\nThus I think this could be the same issue as #12879 in that the server is idle at the time. I do not know if the two crashes are or could be related at all. I shall attempt to see how reproducible this is."},"comment":{"url":"https://api.github.com/repos/golang/go/issues/comments/157370519","html_url":"https://github.com/golang/go/issues/13287#issuecomment-157370519","issue_url":"https://api.github.com/repos/golang/go/issues/13287","id":157370519,"user":{"login":"ianlancetaylor","id":3194333,"avatar_url":"https://avatars.githubusercontent.com/u/3194333?v=3","gravatar_id":"","url":"https://api.github.com/users/ianlancetaylor","html_url":"https://github.com/ianlancetaylor","followers_url":"https://api.github.com/users/ianlancetaylor/followers","following_url":"https://api.github.com/users/ianlancetaylor/following{/other_user}","gists_url":"https://api.github.com/users/ianlancetaylor/gists{/gist_id}","starred_url":"https://api.github.com/users/ianlancetaylor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ianlancetaylor/subscriptions","organizations_url":"https://api.github.com/users/ianlancetaylor/orgs","repos_url":"https://api.github.com/users/ianlancetaylor/repos","events_url":"https://api.github.com/users/ianlancetaylor/events{/privacy}","received_events_url":"https://api.github.com/users/ianlancetaylor/received_events","type":"User","site_admin":false},"created_at":"2015-11-17T13:28:54Z","updated_at":"2015-11-17T13:28:54Z","body":"CC @aclements @RLH "}}