{"action":"created","issue":{"url":"https://api.github.com/repos/mozilla/rr/issues/1524","labels_url":"https://api.github.com/repos/mozilla/rr/issues/1524/labels{/name}","comments_url":"https://api.github.com/repos/mozilla/rr/issues/1524/comments","events_url":"https://api.github.com/repos/mozilla/rr/issues/1524/events","html_url":"https://github.com/mozilla/rr/issues/1524","id":95520725,"number":1524,"title":"Process exhaustion","user":{"login":"hotsphink","id":78649,"avatar_url":"https://avatars.githubusercontent.com/u/78649?v=3","gravatar_id":"","url":"https://api.github.com/users/hotsphink","html_url":"https://github.com/hotsphink","followers_url":"https://api.github.com/users/hotsphink/followers","following_url":"https://api.github.com/users/hotsphink/following{/other_user}","gists_url":"https://api.github.com/users/hotsphink/gists{/gist_id}","starred_url":"https://api.github.com/users/hotsphink/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hotsphink/subscriptions","organizations_url":"https://api.github.com/users/hotsphink/orgs","repos_url":"https://api.github.com/users/hotsphink/repos","events_url":"https://api.github.com/users/hotsphink/events{/privacy}","received_events_url":"https://api.github.com/users/hotsphink/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-07-16T20:29:35Z","updated_at":"2015-07-16T22:03:07Z","closed_at":null,"body":"Sometimes while replaying, it hangs and everything on my system starts failing to fork any new processes. It was hard to debug what was happening when I wasn't allowed to fork. I finally had the brilliant idea to log in as root (duh!). It's a simple case of nprocs exhaustion -- my nproc limit is 1024, and I just had too many threads.\r\n\r\n(Helpful note to anyone who runs across this same issue: I un-wedged myself without killing off the replay by running as root: «ps aux  | egrep '^sfink' | perl -lane 'print [1]' | xargs -n 1 prlimit -u4096 -p». That will bump the limit for all process owned by 'sfink' to 4096.)\r\n\r\nI wonder if part of the problem is that rr is leaving behind processes? When I run with rr replay -V, I get a *lot* of messages like:\r\n\r\n[WARN ~Task() errno: 3 'No such process'] 10860 is unstable; not blocking on its termination\r\n\r\nThen again, I don't see any zombie processes in my ps output. You can see the output of ps ax -L at http://people.mozilla.org/~sfink/data/psax-L.txt or to better see what was started by rr, there's ps axf output at http://people.mozilla.org/~sfink/data/fullpsf.txt . Oddly, it seems to show a whole bunch of \"python2.7 ./mach test ...\" processes parented to pid 1 instead of rr or whatever? Those *are* from rr; lsof shows /home/sfink/.rr/mach-34/mmap_26_hardlink_librrpreload.so loaded in, as well as lots of other rr goop.\r\n"},"comment":{"url":"https://api.github.com/repos/mozilla/rr/issues/comments/122116890","html_url":"https://github.com/mozilla/rr/issues/1524#issuecomment-122116890","issue_url":"https://api.github.com/repos/mozilla/rr/issues/1524","id":122116890,"user":{"login":"rocallahan","id":296135,"avatar_url":"https://avatars.githubusercontent.com/u/296135?v=3","gravatar_id":"","url":"https://api.github.com/users/rocallahan","html_url":"https://github.com/rocallahan","followers_url":"https://api.github.com/users/rocallahan/followers","following_url":"https://api.github.com/users/rocallahan/following{/other_user}","gists_url":"https://api.github.com/users/rocallahan/gists{/gist_id}","starred_url":"https://api.github.com/users/rocallahan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rocallahan/subscriptions","organizations_url":"https://api.github.com/users/rocallahan/orgs","repos_url":"https://api.github.com/users/rocallahan/repos","events_url":"https://api.github.com/users/rocallahan/events{/privacy}","received_events_url":"https://api.github.com/users/rocallahan/received_events","type":"User","site_admin":false},"created_at":"2015-07-16T22:03:07Z","updated_at":"2015-07-16T22:03:07Z","body":"It's certainly possible to have processes that are under rr's control and not leaked but owned by pid 1. This happens whenever a tracee process exits while it has children that should not exit. And that can happen without any explicit tracee activity, e.g. when we fork process A1 to create a checkpoint in process A2, and then decide to drop A1 and resume A2, A1 will exit and A2 will be owned by pid 1. rr is still its ptracer parent of course.\r\n\r\nI think maybe your limit of 1024 is just too low. In Fedora 21 and up it's set to 4096 (I'm on Fedora 22). See https://bugzilla.redhat.com/show_bug.cgi?id=432903#c25. I'd suggest just changing your limit to 4096 and see if you still have problems."}}