{"action":"opened","issue":{"url":"https://api.github.com/repos/coreos/bugs/issues/334","labels_url":"https://api.github.com/repos/coreos/bugs/issues/334/labels{/name}","comments_url":"https://api.github.com/repos/coreos/bugs/issues/334/comments","events_url":"https://api.github.com/repos/coreos/bugs/issues/334/events","html_url":"https://github.com/coreos/bugs/issues/334","id":70124160,"number":334,"title":"systemd-journald crash with a coredump","user":{"login":"cgeoffroy","id":1462293,"avatar_url":"https://avatars.githubusercontent.com/u/1462293?v=3","gravatar_id":"","url":"https://api.github.com/users/cgeoffroy","html_url":"https://github.com/cgeoffroy","followers_url":"https://api.github.com/users/cgeoffroy/followers","following_url":"https://api.github.com/users/cgeoffroy/following{/other_user}","gists_url":"https://api.github.com/users/cgeoffroy/gists{/gist_id}","starred_url":"https://api.github.com/users/cgeoffroy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cgeoffroy/subscriptions","organizations_url":"https://api.github.com/users/cgeoffroy/orgs","repos_url":"https://api.github.com/users/cgeoffroy/repos","events_url":"https://api.github.com/users/cgeoffroy/events{/privacy}","received_events_url":"https://api.github.com/users/cgeoffroy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-04-22T13:21:37Z","updated_at":"2015-04-22T13:21:37Z","closed_at":null,"body":"I got an unexpected crash with the systemd-journal service on CoreOs version 607.0.0 .\r\nThis abrupt crash leads to a restart of the docker service (and all my containers).\r\n\r\nThe only piece of information related to this crash is:\r\n```\r\nApr 22 09:11:51 dhub-generated-panamax.novalocal systemd-coredump[1208]: Detected coredump of the journal daemon itself, diverted to /var/lib/systemd/coredump/core.systemd-journal.0.764a20295f824802bb8bf929af8987da.393.142969\r\n```\r\nI uploaded the corresponding dump there: [(dropbox) core.systemd-journal.0.764a20295f824802bb8bf929af8987da.393.1429693860000000.xz]( https://www.dropbox.com/s/4wtrxx9ez01z685/core.systemd-journal.0.764a20295f824802bb8bf929af8987da.393.1429693860000000.xz?dl=0). But the 'systemd-journald' executable was compiled without debug symbols so i wasn't able to retrieve a human readable stack trace.\r\n\r\n---\r\n\r\nHere are the journalctl logs:\r\n```\r\nApr 22 09:09:24 dhub-generated-panamax.novalocal dockerd[563]: time=\"2015-04-22T09:09:24Z\" level=\"info\" msg=\"+job container_inspect(fraunhoferiais_se_3d-map-tiles_latest)\"\r\nApr 22 09:09:24 dhub-generated-panamax.novalocal dockerd[563]: No such container: fraunhoferiais_se_3d-map-tiles_latest\r\nApr 22 09:09:24 dhub-generated-panamax.novalocal dockerd[563]: time=\"2015-04-22T09:09:24Z\" level=\"info\" msg=\"-job container_inspect(fraunhoferiais_se_3d-map-tiles_latest) = ERR (1)\"\r\nApr 22 09:09:24 dhub-generated-panamax.novalocal dockerd[563]: time=\"2015-04-22T09:09:24Z\" level=\"error\" msg=\"Handler for GET /containers/{name:.*}/json returned error: No such container: fraunhoferiais_se_3d-map-tiles_latest\r\nApr 22 09:09:24 dhub-generated-panamax.novalocal dockerd[563]: time=\"2015-04-22T09:09:24Z\" level=\"error\" msg=\"HTTP Error: statusCode=404 No such container: fraunhoferiais_se_3d-map-tiles_latest\"\r\nApr 22 09:09:24 dhub-generated-panamax.novalocal docker[677]: I, [2015-04-22T09:09:24.874998 #40]  INFO -- : Completed 200 OK in 18ms (Views: 16.9ms | ActiveRecord: 0.4ms)\r\nApr 22 09:09:24 dhub-generated-panamax.novalocal docker[703]: I, [2015-04-22T09:09:24.877602 #10]  INFO -- : Completed 200 OK in 29ms (Views: 0.4ms)\r\nApr 22 09:09:29 dhub-generated-panamax.novalocal docker[1044]: 2103b00b3fdf: Download complete\r\nApr 22 09:09:29 dhub-generated-panamax.novalocal docker[1044]: 323a3326d61a: Pulling metadata\r\nApr 22 09:09:29 dhub-generated-panamax.novalocal dockerd[563]: time=\"2015-04-22T09:09:29Z\" level=\"debug\" msg=\"http://193.52.45.131:8080/v1/images/323a3326d61a304c849b95259ea8aeacf47aef7a4c42ebdb5b1212ce33e141e3/json -- HEADER\r\nApr 22 09:09:30 dhub-generated-panamax.novalocal docker[1044]: 323a3326d61a: Pulling fs layer\r\nApr 22 09:09:30 dhub-generated-panamax.novalocal dockerd[563]: time=\"2015-04-22T09:09:30Z\" level=\"debug\" msg=\"http://193.52.45.131:8080/v1/images/323a3326d61a304c849b95259ea8aeacf47aef7a4c42ebdb5b1212ce33e141e3/layer -- HEADE\r\nApr 22 09:11:51 dhub-generated-panamax.novalocal systemd-journal[1216]: Permanent journal is using 24.0M (max allowed 500.0M, trying to leave 4.0G free of 32.4G available → current limit 500.0M).\r\nApr 22 09:11:51 dhub-generated-panamax.novalocal systemd-coredump[1208]: Detected coredump of the journal daemon itself, diverted to /var/lib/systemd/coredump/core.systemd-journal.0.764a20295f824802bb8bf929af8987da.393.142969\r\nApr 22 09:11:51 dhub-generated-panamax.novalocal systemd-journal[1216]: Journal started\r\nApr 22 09:09:44 dhub-generated-panamax.novalocal systemd[1]: Starting Generate /run/coreos/motd...\r\nApr 22 09:09:44 dhub-generated-panamax.novalocal systemd[1]: Started Generate /run/coreos/motd.\r\nApr 22 09:10:31 dhub-generated-panamax.novalocal systemd[1]: systemd-journald.service watchdog timeout (limit 1min)!\r\nApr 22 09:11:51 dhub-generated-panamax.novalocal systemd[1]: Starting Flush Journal to Persistent Storage...\r\nApr 22 09:11:51 dhub-generated-panamax.novalocal systemd[1]: Started Flush Journal to Persistent Storage.\r\nApr 22 09:11:51 dhub-generated-panamax.novalocal dockerd[1223]: time=\"2015-04-22T09:11:09Z\" level=\"info\" msg=\"+job serveapi(fd://)\"\r\nApr 22 09:11:51 dhub-generated-panamax.novalocal dockerd[1223]: time=\"2015-04-22T09:11:09Z\" level=\"debug\" msg=\"Using graph driver overlay\"\r\nApr 22 09:11:51 dhub-generated-panamax.novalocal dockerd[1223]: time=\"2015-04-22T09:11:09Z\" level=\"debug\" msg=\"Creating images graph\"\r\nApr 22 09:11:51 dhub-generated-panamax.novalocal dockerd[1223]: time=\"2015-04-22T09:11:09Z\" level=\"debug\" msg=\"Restored 122 elements\"\r\nApr 22 09:11:51 dhub-generated-panamax.novalocal dockerd[1223]: time=\"2015-04-22T09:11:09Z\" level=\"debug\" msg=\"Creating repository list\"\r\nApr 22 09:11:51 dhub-generated-panamax.novalocal dockerd[1223]: time=\"2015-04-22T09:11:09Z\" level=\"debug\" msg=\"Reloaded graph with 3 grants expiring at 2017-03-22 19:04:46.713978458 +0000 UTC\"\r\nApr 22 09:11:51 dhub-generated-panamax.novalocal dockerd[1223]: time=\"2015-04-22T09:11:09Z\" level=\"info\" msg=\"+job init_networkdriver()\"\r\nApr 22 09:11:51 dhub-generated-panamax.novalocal dockerd[1223]: time=\"2015-04-22T09:11:09Z\" level=\"debug\" msg=\"/usr/sbin/iptables, [--wait -C POSTROUTING -t nat -s 172.17.42.1/16 ! -o docker0 -j MASQUERADE]\"\r\nApr 22 09:11:51 dhub-generated-panamax.novalocal dockerd[1223]: time=\"2015-04-22T09:11:09Z\" level=\"info\" msg=\"Listening for HTTP on fd ()\"\r\nApr 22 09:11:51 dhub-generated-panamax.novalocal dockerd[1223]: time=\"2015-04-22T09:11:09Z\" level=\"debug\" msg=\"Registering GET, /images/json\"\r\nApr 22 09:11:51 dhub-generated-panamax.novalocal dockerd[1223]: time=\"2015-04-22T09:11:09Z\" level=\"debug\" msg=\"Registering GET, /images/{name:.*}/json\"\r\nApr 22 09:11:51 dhub-generated-panamax.novalocal dockerd[1223]: time=\"2015-04-22T09:11:09Z\" level=\"debug\" msg=\"Registering GET, /containers/ps\"\r\n```\r\n\r\nHere is my drop-ins configuration for the systemd-journald.service unit.\r\n```\r\ncore@dhub-generated-panamax ~ $ cat /etc/systemd/journald.conf.d/50-limit-journal-size.conf \r\n[Journal]\r\nCompress=yes\r\nSystemMaxUse=500M\r\nSystemMaxFileSize=10M\r\n```"}}