{"action":"created","issue":{"url":"https://api.github.com/repos/blaze/distributed/issues/53","labels_url":"https://api.github.com/repos/blaze/distributed/issues/53/labels{/name}","comments_url":"https://api.github.com/repos/blaze/distributed/issues/53/comments","events_url":"https://api.github.com/repos/blaze/distributed/issues/53/events","html_url":"https://github.com/blaze/distributed/pull/53","id":122404527,"number":53,"title":"Cluster management","user":{"login":"broxtronix","id":63793,"avatar_url":"https://avatars.githubusercontent.com/u/63793?v=3","gravatar_id":"","url":"https://api.github.com/users/broxtronix","html_url":"https://github.com/broxtronix","followers_url":"https://api.github.com/users/broxtronix/followers","following_url":"https://api.github.com/users/broxtronix/following{/other_user}","gists_url":"https://api.github.com/users/broxtronix/gists{/gist_id}","starred_url":"https://api.github.com/users/broxtronix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/broxtronix/subscriptions","organizations_url":"https://api.github.com/users/broxtronix/orgs","repos_url":"https://api.github.com/users/broxtronix/repos","events_url":"https://api.github.com/users/broxtronix/events{/privacy}","received_events_url":"https://api.github.com/users/broxtronix/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-12-16T01:11:11Z","updated_at":"2015-12-16T19:59:10Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/blaze/distributed/pulls/53","html_url":"https://github.com/blaze/distributed/pull/53","diff_url":"https://github.com/blaze/distributed/pull/53.diff","patch_url":"https://github.com/blaze/distributed/pull/53.patch"},"body":"This is a first attempt to improve the `dcluster` command line tool for `distributed`.  This PR includes the following improvements:\r\n\r\n 1. `dcluster` now has command line arguments for specifying the number of cores/worker (`--ncores`) and the number of workers/host (`--nworkers`).  This makes it easy to move adjust the blend of SMP vs. multi-process parallelism on hosts in the cluster.  Some workloads benefit from SMP, while others benefit from multi-processing.\r\n\r\n 2. The `--center-port` can now be specified in `dcluster`.\r\n\r\n 3. The `dworker` port is now selected at random.  However, I see that you just checked in another PR #52  that addresses this issue, so possibly this should be removed here.\r\n\r\n 4. A new `--log-directory` argument allows the user to specify a directory to use on each host in the cluster for streaming the `stdout` and `stderr` of all `dcenter` and `dworker` commands to disk.  This has the benefit of creating useful logfiles, and it also reduces the amount of network chatter over the `dcluster` ssh connections, which may be an important optimization for very large clusters with many workers.\r\n\r\n 5. If no `--log-directory` is specified (the default), then the output of all `dworker` and `dcenter` processes is streamed to the console running `dworker`.  This output has been cleaned up to make this stream of information more readable and useful.  There is also light use of terminal colors that, if you find it superfluous, would be easy to remove.  (I like the enhanced readability!) :)\r\n\r\n  6. The `dworker` command now captures any `KeyboardInterrupt` event, and shuts itself and all remote `dworker` and `dcenter` processes down cleanly.  Note that this means that the cluster lifetime is currently tied to the lifetime of the `dcluster` process.  This is convenient and easy to use, but does mean that `dcluster` must remain running, and any interruptions to ssh connections between the master and worker nodes may cause remote processes to terminate.\r\n\r\nHaving spent some time today hacking and thinking on this today, I have a few thoughts related to this current tool:\r\n\r\n* On a cluster with 32 nodes and 4 hosts, I can potentially run up to 128 `dworker` processes.  Right now, each one requires its own SSH connection, and these take just over a second to establish.  Thus, it can already take a minute to start up a small cluster with many CPUs if I want to use \"multiprocessing\" mode.  Establishing these connection concurrently (via threading these ssh connections) will probably be necessary to reduce startup time.  For me, this is the next logical addition on top of what I've done here. I'd be happy to work on that if you think it would be a good idea.  \r\n\r\n* That said, it would be nice if the `dworker` command could could have a command line argument to switch it from SMP mode to multi-processing mode.  That way, we would still only need one ssh connection per host, and could switch between SMP vs multi-processing workers with a quick command-line switch.\r\n\r\n* Although I am actually sort of liking this ability to start and stop a cluster simply by starting and stopping `dcluster`, I think a more robust long-term solution would be to [daemonize](http://www.jejik.com/files/examples/daemon3x.py) the `dcenter` and `dworker` processes so that they can start and run in the background.  This way their lifetime does not depend on the persistence of SSH connections or a single remote command on the master that the user might carelessly terminate.  At that point, the user might start and stop the cluster using `dcluster start` and `dcluster stop`.  \r\n\r\nIn any case, this is all just spit-balling.  Let me know what you think of the improved `dcluster`, and in particular what you think of these last two larger questions.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/blaze/distributed/issues/comments/165228871","html_url":"https://github.com/blaze/distributed/pull/53#issuecomment-165228871","issue_url":"https://api.github.com/repos/blaze/distributed/issues/53","id":165228871,"user":{"login":"mrocklin","id":306380,"avatar_url":"https://avatars.githubusercontent.com/u/306380?v=3","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"created_at":"2015-12-16T19:59:10Z","updated_at":"2015-12-16T19:59:10Z","body":"Basic local use is pretty slick.  Way nicer than before.\r\n\r\nSome additional usability comments.\r\n\r\n1.  We could consider extending `dcluster` to include a `dscheduler` process on the same node provided for the center\r\n2.  It would be nice to have a highly visible line that gives the address of the center or scheduler.  This would need to be visibly marked so that it didn't fall into the mess of logs\r\n\r\n    ```\r\n    ==============\r\n    Center Address:\r\n    Scheduler Address: \r\n    ==============\r\n    ```\r\n\r\n    I often search for such an address and then copy-paste it into a call to `Executor`\r\n3.  How should we specify options to SSH?  For example I often use a special ssh key when interacting with nodes on particular clusters and use a special username.  I might do something like `ssh -i ~/.ssh/my-special-key.pem ubuntu@hostname`."}}