{"action":"created","issue":{"url":"https://api.github.com/repos/systemd/systemd/issues/2155","labels_url":"https://api.github.com/repos/systemd/systemd/issues/2155/labels{/name}","comments_url":"https://api.github.com/repos/systemd/systemd/issues/2155/comments","events_url":"https://api.github.com/repos/systemd/systemd/issues/2155/events","html_url":"https://github.com/systemd/systemd/issues/2155","id":121925040,"number":2155,"title":"Repeated journald crashes","user":{"login":"goeranu","id":1842587,"avatar_url":"https://avatars.githubusercontent.com/u/1842587?v=3","gravatar_id":"","url":"https://api.github.com/users/goeranu","html_url":"https://github.com/goeranu","followers_url":"https://api.github.com/users/goeranu/followers","following_url":"https://api.github.com/users/goeranu/following{/other_user}","gists_url":"https://api.github.com/users/goeranu/gists{/gist_id}","starred_url":"https://api.github.com/users/goeranu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/goeranu/subscriptions","organizations_url":"https://api.github.com/users/goeranu/orgs","repos_url":"https://api.github.com/users/goeranu/repos","events_url":"https://api.github.com/users/goeranu/events{/privacy}","received_events_url":"https://api.github.com/users/goeranu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-12-13T15:24:44Z","updated_at":"2015-12-14T07:37:37Z","closed_at":null,"body":"During the night, the journald process crashed several times.  27 core files lay in my /var/lib/systemd/coredump now, the first one from 01:30 and the last one 08:03.\r\n\r\nWhen I investigated at the end of the period, a mdraid resynchronization was also running, making I/O very sluggish.  This may or may not have been related.  It is now around 8 hours since the resync ended, and there hasn't been any further crashes.\r\n\r\nLooking at the backtraces, many end in various file system operations, like ftruncate (e.g. pid 8974), or getdents (e.g. pid 10809).  That might indicate a connection to the slow response during the resync.  But there were also several that crashed in memset (e.g pid 610, the first crash), which would not appear to be related to a slow disk response.\r\n\r\nThere were no explanations in the journal.  That may not be surprising since it was journald itself crashing.  But I still run syslog too, and in /var/log/messages I found the pattern below repeated over and over.\r\n\r\nThis happens on a Fedora system with systemd-222-8.fc23.x86_64.\r\n\r\nAs there were different backtraces, I can't say for sure all these are because of the same problem.  But the proximity in time makes me belive there is **some** kind of connection.  I attach a tar file with all the cores, so you have whatever might be needed.\r\n\r\n    Dec 13 05:28:48 mimmi journal: Permanent journal is using 3.2G (max allowed 4.0G, trying to leave 4.0G free of 170.1G available → current limit 4.0G).\r\n    Dec 13 05:28:48 mimmi systemd-coredump[8970]: Detected coredump of the journal daemon itself, diverted to /var/lib/systemd/coredump/core.systemd-journal.0.352695e5a66440faad208d9f8b0c5fd3.8584.1449980924000000.xz.\r\n    Dec 13 05:28:48 mimmi journal: Journal started\r\n    Dec 13 05:28:48 mimmi systemd: systemd-journald.service: Watchdog timeout (limit 1min)!\r\n    Dec 13 05:28:48 mimmi audit: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:syslogd_t:s0 pid=8584 comm=\"systemd-journal\" exe=\"/usr/lib/systemd/systemd-journald\" sig=6\r\n    Dec 13 05:28:48 mimmi audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-journald comm=\"systemd\" exe=\"/usr/lib/systemd/systemd\" hostname=? addr=? terminal=? res=failed'\r\n    Dec 13 05:28:48 mimmi audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-journald comm=\"systemd\" exe=\"/usr/lib/systemd/systemd\" hostname=? addr=? terminal=? res=success'\r\n    Dec 13 05:28:48 mimmi audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-journald comm=\"systemd\" exe=\"/usr/lib/systemd/systemd\" hostname=? addr=? terminal=? res=success'\r\n    Dec 13 05:28:48 mimmi audit: CONFIG_CHANGE audit_enabled=1 old=1 auid=4294967295 ses=4294967295 subj=system_u:system_r:syslogd_t:s0 res=1\r\n    Dec 13 05:28:48 mimmi audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=mrtg comm=\"systemd\" exe=\"/usr/lib/systemd/systemd\" hostname=? addr=? terminal=? res=failed'\r\n    Dec 13 05:28:48 mimmi audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-journald comm=\"systemd\" exe=\"/usr/lib/systemd/systemd\" hostname=? addr=? terminal=? res=success'\r\n\r\n\r\n[coredumps.zip](https://github.com/systemd/systemd/files/60486/coredumps.zip)\r\n"},"comment":{"url":"https://api.github.com/repos/systemd/systemd/issues/comments/164369273","html_url":"https://github.com/systemd/systemd/issues/2155#issuecomment-164369273","issue_url":"https://api.github.com/repos/systemd/systemd/issues/2155","id":164369273,"user":{"login":"goeranu","id":1842587,"avatar_url":"https://avatars.githubusercontent.com/u/1842587?v=3","gravatar_id":"","url":"https://api.github.com/users/goeranu","html_url":"https://github.com/goeranu","followers_url":"https://api.github.com/users/goeranu/followers","following_url":"https://api.github.com/users/goeranu/following{/other_user}","gists_url":"https://api.github.com/users/goeranu/gists{/gist_id}","starred_url":"https://api.github.com/users/goeranu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/goeranu/subscriptions","organizations_url":"https://api.github.com/users/goeranu/orgs","repos_url":"https://api.github.com/users/goeranu/repos","events_url":"https://api.github.com/users/goeranu/events{/privacy}","received_events_url":"https://api.github.com/users/goeranu/received_events","type":"User","site_admin":false},"created_at":"2015-12-14T07:37:37Z","updated_at":"2015-12-14T07:37:37Z","body":"Just so I understand what you are saying: you believe all crashes, regardless of traceback, are because of the timeout, right?\r\n\r\nLooking further back in my logs, I see occasional occurrences of the *Watchdog timeout* messages, almost (but not quite) always accompanied by the *Detected coredump* message.  I take it the watchdog intentionally makes the service crash.\r\n\r\nI've tweaked my local configuration to increase the timeout.  Thanks for the pointer!"}}