{"action":"created","issue":{"url":"https://api.github.com/repos/datacratic/StarCluster/issues/23","labels_url":"https://api.github.com/repos/datacratic/StarCluster/issues/23/labels{/name}","comments_url":"https://api.github.com/repos/datacratic/StarCluster/issues/23/comments","events_url":"https://api.github.com/repos/datacratic/StarCluster/issues/23/events","html_url":"https://github.com/datacratic/StarCluster/issues/23","id":107872433,"number":23,"title":"add_node does not run plugins in vanilla_improvements, but does in 0.95.6","user":{"login":"kamalic","id":4681485,"avatar_url":"https://avatars.githubusercontent.com/u/4681485?v=3","gravatar_id":"","url":"https://api.github.com/users/kamalic","html_url":"https://github.com/kamalic","followers_url":"https://api.github.com/users/kamalic/followers","following_url":"https://api.github.com/users/kamalic/following{/other_user}","gists_url":"https://api.github.com/users/kamalic/gists{/gist_id}","starred_url":"https://api.github.com/users/kamalic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kamalic/subscriptions","organizations_url":"https://api.github.com/users/kamalic/orgs","repos_url":"https://api.github.com/users/kamalic/repos","events_url":"https://api.github.com/users/kamalic/events{/privacy}","received_events_url":"https://api.github.com/users/kamalic/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":15,"created_at":"2015-09-23T08:32:53Z","updated_at":"2015-10-30T21:27:04Z","closed_at":"2015-10-30T21:27:04Z","body":"In vanilla_improvements, when I spin up a multi-node cluster from scratch, it runs all plugins I have configured in the the config file.  However, when I add a node with \"starcluster addnode -n # clustername\" or via the load balancer, it runs the default SGEplugin, but none of my configured plugins.\r\n\r\nWhat is happening here?  Below is the cluster, node, volume, and one of the plugins I wrote (the ops_scratchunrestrict, which indeed does things on_add_node):\r\n\r\n[cluster platform]\r\nKEYNAME = starclusterbot\r\nCLUSTER_SIZE = 1\r\nCLUSTER_USER = sgeadmin\r\nCLUSTER_SHELL = bash\r\n#NODE_INSTANCE_TYPE = c3.8xlarge\r\nNODE_INSTANCE_ARRAY = c38,r38,g28\r\nMASTER_INSTANCE_TYPE = c3.xlarge\r\nPLUGINS = pg_platform_paths, pg_platform_update_alternatives, pg_memreq, ops_pecreate, ops_scratchunrestrict\r\nCONFIG_ON_MASTER=True\r\nVOLUMES=vol1\r\n\r\n\r\n[node c38]\r\nINSTANCE_TYPE = c3.8xlarge\r\nSPOT_BID = 3.00\r\nSELECTION_FACTOR = 0.5\r\nIMAGE_ID = ami-9d61d7f6\r\nPUBLIC_IPS = False\r\nPLUGINS = pg_platform_paths, pg_platform_update_alternatives, pg_memreq, ops_pec\r\nreate, ops_scratchunrestrict\r\n\r\n[node g28]\r\nINSTANCE_TYPE = g2.8xlarge\r\nSPOT_BID = 3.00\r\nSELECTION_FACTOR = 0.8\r\nIMAGE_ID = ami-9d61d7f6\r\nPUBLIC_IPS = False\r\nPLUGINS = pg_platform_paths, pg_platform_update_alternatives, pg_memreq, ops_pec\r\nreate, ops_scratchunrestrict\r\n\r\n[node r38]\r\nINSTANCE_TYPE = r3.8xlarge\r\nSPOT_BID = 3.00\r\nSELECTION_FACTOR = 1.0\r\nIMAGE_ID = ami-9d61d7f6\r\nPUBLIC_IPS = False\r\nPLUGINS = pg_platform_paths, pg_platform_update_alternatives, pg_memreq, ops_pecreate, ops_scratchunrestrict\r\n\r\n#############################\r\n## Configuring EBS Volumes ##\r\n#############################\r\n[volume vol1]\r\nVOLUME_ID=vol-XXXXXXX\r\nmount_path=/home\r\npartition=0\r\n\r\n\r\n###############################\r\n## Configuring other plugins ##\r\n###############################\r\n[plugin ops_scratchunrestrict]\r\nsetup_class = ScratchUnrestrict.ScratchUnrestrict\r\n\r\n\r\nAnd here is ScratchUnrestrict.py :\r\n\r\nfrom starcluster.clustersetup import ClusterSetup\r\nfrom starcluster.logger import log\r\n\r\nclass ScratchUnrestrict(ClusterSetup):\r\n     def run(self, nodes, master, user, user_shell, volumes):\r\n          for node in nodes:\r\n               log.info(\"Unrestricting scratch on %s\" % (node.alias))\r\n               node.ssh.execute('rm -rf /scratch; ln -s /mnt /scratch; chmod 1777 /mnt; mount /dev/xvdaa /tmp')\r\n     def on_add_node(self, node, nodes, master, user, user_shell, volumes):\r\n          log.info(\"Unrestricting scratch on %s\" % (node.alias))\r\n          node.ssh.execute('rm -rf /scratch; ln -s /mnt /scratch; chmod 1777 /mnt; mount /dev/xvdaa /tmp')\r\n\r\nOutput when I launch the initial cluster with \"starcluster start clustername\", I can see it run the various other plugins (ops_scratchunrestrict for example):\r\n>>> Adding parallel environment 'orte' to queue 'all.q'\r\n>>> Running plugin pg_platform_paths\r\ngns.conf 100% ||||||||||||||||||||||||||||||||||||||| Time: 00:00:00 394.49 K/s\r\ngns.sh 100% ||||||||||||||||||||||||||||||||||||||||| Time: 00:00:00 406.79 K/s\r\nthreaded.pe 100% |||||||||||||||||||||||||||||||||||| Time: 00:00:00 571.58 K/s\r\n>>> Running plugin pg_platform_update_alternatives\r\nupdate-alternatives [a bunch of output for all the alternatives being set up]\r\n>>> Running plugin pg_memreq\r\n>>> Adding SGE environment to SSH paths\r\nmem_requested.txt 100% |||||||||||||||||||||||||||||| Time: 00:00:00 744.91 K/s\r\n>>> Configuring resources and other cluster details\r\nroot@master added \"mem_requested\" to complex entry list\r\nchanged scheduler configuration\r\n>>> setting up mem_requested to 50G for master\r\nroot@master modified \"master\" in exechost list\r\n>>> Running plugin ops_pecreate\r\n>>> Creating the threaded parallel environment\r\n>>> Running plugin ops_scratchunrestrict\r\n>>> Unrestricting scratch on master\r\n>>> Configuring cluster took 0.994 mins\r\n>>> Starting cluster took 3.134 mins\r\n\r\nThe cluster is now ready to use.\r\n\r\n...but when I add a new node with \"starcluster addnode -n 1 clustername\", you can see it runs th estarcluster.plugins.sge.SGEPlugin but that's all!\r\n\r\n>>> Launching node(s): node001\r\nSpotInstanceRequest:sir-02446xcr\r\n>>> Waiting for one of the new nodes to be up (updating every 30s)\r\n>>> Still waiting for spots: [SpotInstanceRequest:sir-02446xcr]\r\n>>> 2015-09-23 08:27:10.227239+00:00 Sleeping for 30 seconds\r\n>>> Still waiting for spots: [SpotInstanceRequest:sir-02446xcr]\r\n>>> 2015-09-23 08:27:40.376914+00:00 Sleeping for 30 seconds\r\n>>> Instance ids:[u'i-5f6038fc']\r\n>>> node001 ssh is not up                                              |   0%  \r\n1/1 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%  \r\n>>> Still waiting for instances: [<Node: node001 ([NODEID])>]\r\n>>> 2015-09-23 08:28:43.958372+00:00 Sleeping for 30 seconds\r\n>>> node001 ssh is not up                                              |   0%  \r\n1/1 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%  \r\n>>> Still waiting for instances: [<Node: node001 ([NODEID])>]\r\n>>> 2015-09-23 08:29:15.103292+00:00 Sleeping for 30 seconds\r\n1/1 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%  \r\n>>> Adding node: node001\r\n>>> Running plugin starcluster.clustersetup.DefaultClusterSetup\r\n>>> Configuring hostnames...\r\n1/1 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%  \r\n>>> Configuring /etc/hosts on each node\r\n2/2 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%  \r\n/tmp/exports_qGNMsW 100% |||||||||||||||||||||||||||| Time: 00:00:00 264.27 K/s\r\nexports_GEsuDE 100% ||||||||||||||||||||||||||||||||| Time: 00:00:00 876.25 K/s\r\n>>> Configuring NFS exports path(s):\r\n/home\r\n>>> Mounting all NFS export path(s) on 1 worker node(s)\r\n/tmp/fstab_Si5CBz 100% |||||||||||||||||||||||||||||| Time: 00:00:00 288.61 K/s\r\nfstab_pMFbQB 100% ||||||||||||||||||||||||||||||||||| Time: 00:00:00   1.04 M/s\r\n1/1 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%  \r\n>>> Setting up NFS took 0.021 mins\r\n1/1 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%  \r\n>>> Configuring scratch space for user(s): sgeadmin\r\n1/1 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%  \r\n>>> Configuring passwordless ssh for root\r\n/tmp/known_hosts_5Bvrno 100% |||||||||||||||||||||||| Time: 00:00:00 439.64 K/s\r\nknown_hosts_vpMu5S 100% ||||||||||||||||||||||||||||| Time: 00:00:00 466.03 K/s\r\n/tmp/id_rsa_6mRyZ1 100% ||||||||||||||||||||||||||||| Time: 00:00:00 218.46 K/s\r\nid_rsa_6mRyZ1 100% |||||||||||||||||||||||||||||||||| Time: 00:00:00   4.01 M/s\r\n/tmp/id_rsa.pub_8QC9iU 100% ||||||||||||||||||||||||| Time: 00:00:00 234.73 K/s\r\nid_rsa.pub_8QC9iU 100% |||||||||||||||||||||||||||||| Time: 00:00:00 697.22 K/s\r\n/tmp/authorized_keys_HvL6Hm 100% |||||||||||||||||||| Time: 00:00:00 859.34 K/s\r\nauthorized_keys_HvL6Hm 100% ||||||||||||||||||||||||| Time: 00:00:00   2.41 M/s\r\n/tmp/known_hosts_f3syMl 100% |||||||||||||||||||||||| Time: 00:00:00 896.82 K/s\r\nknown_hosts_f3syMl 100% ||||||||||||||||||||||||||||| Time: 00:00:00   7.35 M/s\r\n>>> Configuring passwordless ssh for sgeadmin\r\n/tmp/known_hosts_E6WYe3 100% |||||||||||||||||||||||| Time: 00:00:00 445.57 K/s\r\nknown_hosts_fOFkZs 100% ||||||||||||||||||||||||||||| Time: 00:00:00 349.53 K/s\r\n>>> Running plugin starcluster.plugins.sge.SGEPlugin\r\n>>> Adding node001 to SGE\r\n/tmp/exports_w61SUi 100% |||||||||||||||||||||||||||| Time: 00:00:00 301.48 K/s\r\nexports_fi3ZoF 100% ||||||||||||||||||||||||||||||||| Time: 00:00:00 845.70 K/s\r\n>>> Configuring NFS exports path(s):\r\n/opt/sge6\r\n>>> Mounting all NFS export path(s) on 1 worker node(s)\r\n/tmp/fstab_Xp34tN 100% |||||||||||||||||||||||||||||| Time: 00:00:00 280.41 K/s\r\nfstab_Gq5qqH 100% ||||||||||||||||||||||||||||||||||| Time: 00:00:00 961.68 K/s\r\n1/1 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%  \r\n>>> Setting up NFS took 0.021 mins\r\n>>> Updating SGE parallel environment 'orte'\r\n2/2 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%  \r\n>>> Adding parallel environment 'orte' to queue 'all.q'\r\n>>> Shutting down threads...\r\n20/20 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%  \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/datacratic/StarCluster/issues/comments/152655302","html_url":"https://github.com/datacratic/StarCluster/issues/23#issuecomment-152655302","issue_url":"https://api.github.com/repos/datacratic/StarCluster/issues/23","id":152655302,"user":{"login":"FinchPowers","id":607566,"avatar_url":"https://avatars.githubusercontent.com/u/607566?v=3","gravatar_id":"","url":"https://api.github.com/users/FinchPowers","html_url":"https://github.com/FinchPowers","followers_url":"https://api.github.com/users/FinchPowers/followers","following_url":"https://api.github.com/users/FinchPowers/following{/other_user}","gists_url":"https://api.github.com/users/FinchPowers/gists{/gist_id}","starred_url":"https://api.github.com/users/FinchPowers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/FinchPowers/subscriptions","organizations_url":"https://api.github.com/users/FinchPowers/orgs","repos_url":"https://api.github.com/users/FinchPowers/repos","events_url":"https://api.github.com/users/FinchPowers/events{/privacy}","received_events_url":"https://api.github.com/users/FinchPowers/received_events","type":"User","site_admin":false},"created_at":"2015-10-30T21:27:04Z","updated_at":"2015-10-30T21:27:04Z","body":"Should be fixed by PR #28. Reopen if it did not fix the issue."}}