{"action":"created","issue":{"url":"https://api.github.com/repos/SeattleTestbed/nodemanager/issues/115","labels_url":"https://api.github.com/repos/SeattleTestbed/nodemanager/issues/115/labels{/name}","comments_url":"https://api.github.com/repos/SeattleTestbed/nodemanager/issues/115/comments","events_url":"https://api.github.com/repos/SeattleTestbed/nodemanager/issues/115/events","html_url":"https://github.com/SeattleTestbed/nodemanager/issues/115","id":55550106,"number":115,"title":"`daemon.py` hangs if `init` process has PID other than 1","user":{"login":"aaaaalbert","id":3806695,"avatar_url":"https://avatars.githubusercontent.com/u/3806695?v=3","gravatar_id":"","url":"https://api.github.com/users/aaaaalbert","html_url":"https://github.com/aaaaalbert","followers_url":"https://api.github.com/users/aaaaalbert/followers","following_url":"https://api.github.com/users/aaaaalbert/following{/other_user}","gists_url":"https://api.github.com/users/aaaaalbert/gists{/gist_id}","starred_url":"https://api.github.com/users/aaaaalbert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaaaalbert/subscriptions","organizations_url":"https://api.github.com/users/aaaaalbert/orgs","repos_url":"https://api.github.com/users/aaaaalbert/repos","events_url":"https://api.github.com/users/aaaaalbert/events{/privacy}","received_events_url":"https://api.github.com/users/aaaaalbert/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-01-26T22:57:24Z","updated_at":"2015-01-27T15:08:53Z","closed_at":null,"body":"Seattle has a `daemon` library that helps daemonize the nodemanager process on Unix-like systems. It does this using a variant of the usual \"double-fork\" approach, which can be summarized as follows (with some terminology borrowed from [here](http://www.win.tue.nl/~aeb/linux/lk/lk-10.html):\r\n\r\n* The parent process `fork()`s off the first child process and exits.\r\n* The first child\r\n * Creates a new session\r\n * Closes the standard file descriptors inherited from the parent\r\n * `fork()`s off the second child process and exits\r\n* The second child waits for child 1, its parent process, to exit and can then start its actual work after the `daemon.daemonize()` function returns. (*Note:* Waiting here for child 1 to exit appears to be non-standard, but should not hurt either if done correctly.)\r\n\r\nOur implementation of the last bit is what gets us in trouble: [The code](https://github.com/SeattleTestbed/nodemanager/blob/master/daemon.py#L46-L49) waits for the `init` process to adopt child 2 after child 1 exited, and tries to detect this by waiting for its parent process ID (`ppid`) to become 1, `init`'s usual PID.\r\n\r\nHowever, [Upstart](https://help.ubuntu.com/community/UbuntuBootupHowto), Ubuntu Linux's current `init` replacement, has a *User Session Mode* that results in multiple `init` processes running on the system, one \"classical\" (PID 1) and another one that spawns processes from interactions with the GUI (such as a terminal window), `init --user`, with a different PID. This makes our code loop indefinitely.\r\n\r\nWhat we should do instead can be derived from the table below, showing an instrumented version of `daemon.py` going through the different phases of forking:\r\n\r\n\r\nRef | What? | PID | PPID | PGRP | SID | TTY\r\n---|---|---|---|---|---|---|\r\n1 | Parent process | 26105 | 3643 | 26105 | 3643 | /dev/pts/132\r\n2 | Child1, forked, but before `setsid` | **26106** | **26105** | 26105 | 3643 | /dev/pts/13\r\n3 | Child1, before closing stdstreams | 26106 | 26105 | **26106** | **26106** | /dev/pts/13\r\n4 | Child1, afterwards | 26106 | 26105 | 26106 | 26106 | **EBADF**\r\n5 | Child2, forked but waiting for Child1 to exit | **26107** | **26106** | 26106 | 26106 | EBADF\r\n6 | Child2, ready | 26107 | **2957** | 26106 | 26106 | EBADF\r\n\r\n\r\n\r\n1. shows the parent process, as started by `bash` which also is the session leader (i.e. the session `ID` is its `PID`). The parent started the process group, `PGRP`.\r\n2. shows Child 1, with the Parent process as its `PPID`, still within the same session and process group. The child calls `setsid()` now, resulting in...\r\n3. Child 1 is the process group leader, and also owns this new session.\r\n4. Child 1 has closed the standard file descriptors it inherited from Parent. Accessing them results in a \"Bad File descriptor\" error.\r\n5. shows Child 2 forked off. Its parent, session leader, and process group owner is Child 1.\r\n6. Child 1 has exited, leaving Child 2 to be adopted by `init --user`.\r\n\r\n-----\r\n\r\n**Fix for `daemon.py`:**\r\nIn order to wait for Child 1 to exit, Child 2 can check whether its `PGRP` or `SID` equals the `PPID` it sees. If not, Child 1 has exited, and Child 2 can continue.\r\n\r\n```diff\r\n46,47c46\r\n<       ppid = os.getppid()\r\n<       while ppid != 1:\r\n---\r\n>       while os.getppid() == os.getpgrp():\r\n49d47\r\n<         ppid = os.getppid()\r\n```\r\n\r\n------\r\n\r\nI'm currently testing this patch on Mac OS X 10.6.8 with Python 2.7.8, and Ubuntu 14.04.1 with Python 2.7.6 running inside VirtualBox 4.3.20.\r\n"},"comment":{"url":"https://api.github.com/repos/SeattleTestbed/nodemanager/issues/comments/71662825","html_url":"https://github.com/SeattleTestbed/nodemanager/issues/115#issuecomment-71662825","issue_url":"https://api.github.com/repos/SeattleTestbed/nodemanager/issues/115","id":71662825,"user":{"login":"aaaaalbert","id":3806695,"avatar_url":"https://avatars.githubusercontent.com/u/3806695?v=3","gravatar_id":"","url":"https://api.github.com/users/aaaaalbert","html_url":"https://github.com/aaaaalbert","followers_url":"https://api.github.com/users/aaaaalbert/followers","following_url":"https://api.github.com/users/aaaaalbert/following{/other_user}","gists_url":"https://api.github.com/users/aaaaalbert/gists{/gist_id}","starred_url":"https://api.github.com/users/aaaaalbert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaaaalbert/subscriptions","organizations_url":"https://api.github.com/users/aaaaalbert/orgs","repos_url":"https://api.github.com/users/aaaaalbert/repos","events_url":"https://api.github.com/users/aaaaalbert/events{/privacy}","received_events_url":"https://api.github.com/users/aaaaalbert/received_events","type":"User","site_admin":false},"created_at":"2015-01-27T15:08:53Z","updated_at":"2015-01-27T15:08:53Z","body":"(I decided to write up what I learned over the last few days so I have a single reference in the future. Let me know what parts of it you think should go into the docstring, if any :-) )\r\n\r\n--------\r\n\r\nCreating a daemon process\r\n----\r\nor: Why are we doing this convoluted thing again?\r\n\r\n\r\nGoals of a daemon process:\r\n\r\n* Detaches from its ancestor processes so that it doesn't block the launching terminal / script, and is not affected by their fate (e.g. termination)\r\n* Detaches from any controlling TTY (interactive terminal or similar) so it cannot be stopped or otherwise interacted with inadvertently, and drops privileges that would allow it to reclaim a TTY\r\n* Can be created from any starting point -- interactive shell, Python script, `cron`, ...\r\n* If done very correctly, also detaches from the creator's environment such as mount points to minimize interference\r\n\r\n\r\nNotes:\r\n* `fork()` copies the existing parent process and creates a new child process. Execution in both processes continues where the `fork` function call returns: In the parent, it returns the new child's process ID (PID); in the child, it returns 0. The child process inherits the parent's file descriptors, including `stdin`, `stdout`, and `stderr`.\r\n* `wait()` makes a parent process block until one of its immediate children (but not grandchildren etc.) exits.\r\n* Processes are members of *process groups*. The first process in a process group is the *process group leader*. Child processes inherit their process group from their parents, but can create new ones when they create a new session.\r\n* A *session* usually is a login session. It contains a process group; the first member of the session is the *session leader*, its PID is also the session ID. A child can create a new session at will (thus also creating a new process group). It won't have a controlling TTY when it does.\r\n* The *controlling TTY* sends any input to the members of the foreground process group in the session (if any). This could be keyboard input, signals (`KILL`), etc.\r\n* Session leaders and process group leaders can get a new controlling TTY programmatically; other session and process group members can not.\r\n\r\n--------\r\n\r\n\r\n1. We start with the parent process that wants to create a daemonized copy of itself. Let's assume we started Parent in an interactive shell.\r\n * Parent process: PID=parentID, PPID=shellIP, PGRP=parentID, SID=shellID\r\n1. Parent calls `fork()` to fork off Child 1. The parent process `wait()`s for Child 1 to exit.\r\n * Child 1: PID=child1ID, PPID=parentID, PGRP=parentID, SID=shellID\r\n * (Child 1 has the parent process as its parent, shares its process group, and is in the shell's session.)\r\n1. We are not done yet , because Parent is `wait()`ing for Child 1 to terminate.\r\n1. Child 1 now calls `setsid()`, creating a new session, becoming its leader, and also becoming the process group leader. (Its leadership will become important only after the next `fork()`, see below).\r\n * Child 1: PID=child1ID, PPID=parentID, PGRP=child1ID, SID=child1ID\r\n1. (Depending on the requirements of the implementation, Child 1 should also close all of the file descriptors it inherited, `chdir` into `/`, and set its `umask` to 0. Alternatively, this might be done in Child 2 instead.)\r\n1. We are not done yet because\r\n * Parent is still `wait()`ing for Child 1, so if Child 1 would continue to run, this would keep Parent alive too.\r\n * Child1 is the leader of the new session, and can thus reacquire the controlling terminal even if it closed the file descriptors that it inherited from Parent. We specifically wanted to make this impossible for the daemon process.\r\n1. Thus, Child 1 calls `fork()` itself, creating Child 2 which is neither the process group nor session leader, and therefore cannot reacquire the controlling terminal. Note that Parent does *not* `wait()` for Child 2, as this is a grand-child.\r\n * Child 2: PID=child2ID, PPID=child1ID, PGRP=child1ID, SID=child1ID\r\n1. Following the fork, Child 1 exits. This leaves Child 2 without a parent for a moment, but an `init` process will adopt it soon. The consequence of Child 1's exit is that Parent can exit now, too. Eventually, we are left with only Child 2 which is now a daemon:\r\n* Child 2: PID=child2ID, PPID=initID, PGRP=child1ID, SID=child1ID\r\n\r\nNote that in contrast to traditional lore, the process ID of the `init` process (`initID` above) is not necessarily `1`. [Upstart](http://upstart.ubuntu.com/) (and possibly other `init` replacements) has `init --user` processes with different PIDs for graphical sessions aka \"User Session Mode\".\r\n\r\n\r\nFurther reading:\r\n* \"Unix FAQ\" with a section on how to properly daemonize a process: http://www.faqs.org/faqs/unix-faq/programmer/faq/ (Section 1.7)\r\n* Process, process groups, sessions, etc.: http://www.win.tue.nl/~aeb/linux/lk/lk-10.html\r\n* Wikipedia: https://en.wikipedia.org/wiki/Process_group\r\n* POSIX documentation:\r\n * http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html\r\n * http://pubs.opengroup.org/onlinepubs/009695399/functions/setsid.html\r\n* SCO on the controlling TTY: http://uw714doc.sco.com/en/SDK_sysprog/_The_Controlling-Terminal_and_Pr.html\r\n\r\nCode samples:\r\n* One with very detailed explanations which also mentions variants for the various steps: http://code.activestate.com/recipes/278731/\r\n* http://code.activestate.com/recipes/66012/\r\n* The thing our current code is based on, with its seemingly unique waiting for being adopted by PID 1 (`init`): http://code.activestate.com/lists/python-list/209948/\r\n* PEP3143, proposed (but deferred) Python standard daemon library, mentioning the correct default behavior and a few implementations: https://www.python.org/dev/peps/pep-3143/#correct-daemon-behaviour\r\n* An implementation of PEP3143, https://pypi.python.org/pypi/python-daemon/\r\n* http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/\r\n"}}