{"push_id":595334428,"size":1,"distinct_size":1,"ref":"refs/heads/master","head":"52ed7da12e26c45734ce53a1be19ef148b2b953e","before":"334c5bd1ae50ac76770e545cab302361673f45de","commits":[{"sha":"52ed7da12e26c45734ce53a1be19ef148b2b953e","author":{"email":"fbfd45890515f7eeeeabf87d7a80d3449c3bce83@gmail.com","name":"Nicholas Chammas"},"message":"[SPARK-6193] [EC2] Push group filter up to EC2\n\nWhen looking for a cluster, spark-ec2 currently pulls down [info for all instances](https://github.com/apache/spark/blob/eb48fd6e9d55fb034c00e61374bb9c2a86a82fb8/ec2/spark_ec2.py#L620) and filters locally. When working on an AWS account with hundreds of active instances, this step alone can take over 10 seconds.\n\nThis PR improves how spark-ec2 searches for clusters by pushing the filter up to EC2.\n\nBasically, the problem (and solution) look like this:\n\n```python\n>>> timeit.timeit('blah = conn.get_all_reservations()', setup='from __main__ import conn', number=10)\n116.96390509605408\n>>> timeit.timeit('blah = conn.get_all_reservations(filters={\"instance.group-name\": [\"my-cluster-master\"]})', setup='from __main__ import conn', number=10)\n4.629754066467285\n```\n\nTranslated to a user-visible action, this looks like (against an AWS account with ~200 active instances):\n\n```shell\n# master\n$ python -m timeit -n 3 --setup 'import subprocess' 'subprocess.call(\"./spark-ec2 get-master my-cluster --region us-west-2\", shell=True)'\n...\n3 loops, best of 3: 9.83 sec per loop\n\n# this PR\n$ python -m timeit -n 3 --setup 'import subprocess' 'subprocess.call(\"./spark-ec2 get-master my-cluster --region us-west-2\", shell=True)'\n...\n3 loops, best of 3: 1.47 sec per loop\n```\n\nThis PR also refactors `get_existing_cluster()` to make it, I hope, simpler.\n\nFinally, this PR fixes some minor grammar issues related to printing status to the user. :tophat: :clap:\n\nAuthor: Nicholas Chammas <nicholas.chammas@gmail.com>\n\nCloses #4922 from nchammas/get-existing-cluster-faster and squashes the following commits:\n\n18802f1 [Nicholas Chammas] ignore shutting-down\nf2a5b9f [Nicholas Chammas] fix grammar\nd96a489 [Nicholas Chammas] push group filter up to EC2","distinct":true,"url":"https://api.github.com/repos/apache/spark/commits/52ed7da12e26c45734ce53a1be19ef148b2b953e"}]}