{"push_id":596043566,"size":3,"distinct_size":3,"ref":"refs/heads/master","head":"55b1b32dc8b9b25deea8e5864b53fe802bb92741","before":"334c5bd1ae50ac76770e545cab302361673f45de","commits":[{"sha":"52ed7da12e26c45734ce53a1be19ef148b2b953e","author":{"email":"fbfd45890515f7eeeeabf87d7a80d3449c3bce83@gmail.com","name":"Nicholas Chammas"},"message":"[SPARK-6193] [EC2] Push group filter up to EC2\n\nWhen looking for a cluster, spark-ec2 currently pulls down [info for all instances](https://github.com/apache/spark/blob/eb48fd6e9d55fb034c00e61374bb9c2a86a82fb8/ec2/spark_ec2.py#L620) and filters locally. When working on an AWS account with hundreds of active instances, this step alone can take over 10 seconds.\n\nThis PR improves how spark-ec2 searches for clusters by pushing the filter up to EC2.\n\nBasically, the problem (and solution) look like this:\n\n```python\n>>> timeit.timeit('blah = conn.get_all_reservations()', setup='from __main__ import conn', number=10)\n116.96390509605408\n>>> timeit.timeit('blah = conn.get_all_reservations(filters={\"instance.group-name\": [\"my-cluster-master\"]})', setup='from __main__ import conn', number=10)\n4.629754066467285\n```\n\nTranslated to a user-visible action, this looks like (against an AWS account with ~200 active instances):\n\n```shell\n# master\n$ python -m timeit -n 3 --setup 'import subprocess' 'subprocess.call(\"./spark-ec2 get-master my-cluster --region us-west-2\", shell=True)'\n...\n3 loops, best of 3: 9.83 sec per loop\n\n# this PR\n$ python -m timeit -n 3 --setup 'import subprocess' 'subprocess.call(\"./spark-ec2 get-master my-cluster --region us-west-2\", shell=True)'\n...\n3 loops, best of 3: 1.47 sec per loop\n```\n\nThis PR also refactors `get_existing_cluster()` to make it, I hope, simpler.\n\nFinally, this PR fixes some minor grammar issues related to printing status to the user. :tophat: :clap:\n\nAuthor: Nicholas Chammas <nicholas.chammas@gmail.com>\n\nCloses #4922 from nchammas/get-existing-cluster-faster and squashes the following commits:\n\n18802f1 [Nicholas Chammas] ignore shutting-down\nf2a5b9f [Nicholas Chammas] fix grammar\nd96a489 [Nicholas Chammas] push group filter up to EC2","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/52ed7da12e26c45734ce53a1be19ef148b2b953e"},{"sha":"f16b7b031feeb13ec9c17608bd99566f56431869","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"SPARK-6205 [CORE] UISeleniumSuite fails for Hadoop 2.x test with NoClassDefFoundError\n\nAdd xml-apis to core test deps to work aroudn UISeleniumSuite classpath issue\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #4933 from srowen/SPARK-6205 and squashes the following commits:\n\nddd4d32 [Sean Owen] Add xml-apis to core test deps to work aroudn UISeleniumSuite classpath issue","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/f16b7b031feeb13ec9c17608bd99566f56431869"},{"sha":"55b1b32dc8b9b25deea8e5864b53fe802bb92741","author":{"email":"5d1ed0070fb6b6d5f66c1cf6a3d0bb37cf6f15d1@huawei.com","name":"Jacky Li"},"message":"[GraphX] Improve LiveJournalPageRank example\n\n1. Removed unnecessary import\n2. Modified usage print since user must specify the --numEPart parameter as it is required in Analytics.main\n\nAuthor: Jacky Li <jacky.likun@huawei.com>\n\nCloses #4917 from jackylk/import and squashes the following commits:\n\n6c07682 [Jacky Li] fix comment\nc0df8f2 [Jacky Li] fix scalastyle\nb6235e6 [Jacky Li] fix for comment\n87be83b [Jacky Li] remove default value description\n5caae76 [Jacky Li] remove import and modify usage","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/55b1b32dc8b9b25deea8e5864b53fe802bb92741"}]}