{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/5427","labels_url":"https://api.github.com/repos/apache/spark/issues/5427/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/5427/comments","events_url":"https://api.github.com/repos/apache/spark/issues/5427/events","html_url":"https://github.com/apache/spark/pull/5427","id":67204627,"number":5427,"title":"[SPARK-6953] [PySpark] speed up python tests","user":{"login":"davies","id":40902,"avatar_url":"https://avatars.githubusercontent.com/u/40902?v=3","gravatar_id":"","url":"https://api.github.com/users/davies","html_url":"https://github.com/davies","followers_url":"https://api.github.com/users/davies/followers","following_url":"https://api.github.com/users/davies/following{/other_user}","gists_url":"https://api.github.com/users/davies/gists{/gist_id}","starred_url":"https://api.github.com/users/davies/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davies/subscriptions","organizations_url":"https://api.github.com/users/davies/orgs","repos_url":"https://api.github.com/users/davies/repos","events_url":"https://api.github.com/users/davies/events{/privacy}","received_events_url":"https://api.github.com/users/davies/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":26,"created_at":"2015-04-08T19:46:54Z","updated_at":"2015-04-18T09:15:40Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5427","html_url":"https://github.com/apache/spark/pull/5427","diff_url":"https://github.com/apache/spark/pull/5427.diff","patch_url":"https://github.com/apache/spark/pull/5427.patch"},"body":"This PR try to speed up some python tests:\r\n```\r\ntests.py                        144s -> 103s      -41s\r\nmllib/classification.py   24s -> 17s          -7s\r\nmllib/regression.py       27s -> 15s          -12s\r\nmllib/tree.py                 27s  -> 13s         -14s\r\nmllib/tests.py                64s -> 31s         -33s\r\nstreaming/tests.py       185s -> 84s        -101s\r\n```\r\nConsidering python3, the total saving will be 558s (almost 10 minutes) (core, and streaming run three times, mllib runs twice).\r\n\r\nDuring testing, it will show used time for each test file:\r\n```\r\nRun core tests ...\r\nRunning test: pyspark/rdd.py ... ok (22s)\r\nRunning test: pyspark/context.py ... ok (16s)\r\nRunning test: pyspark/conf.py ... ok (4s)\r\nRunning test: pyspark/broadcast.py ... ok (4s)\r\nRunning test: pyspark/accumulators.py ... ok (4s)\r\nRunning test: pyspark/serializers.py ... ok (6s)\r\nRunning test: pyspark/profiler.py ... ok (5s)\r\nRunning test: pyspark/shuffle.py ... ok (1s)\r\nRunning test: pyspark/tests.py ... ok (103s)   144s\r\n```"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/94146269","html_url":"https://github.com/apache/spark/pull/5427#issuecomment-94146269","issue_url":"https://api.github.com/repos/apache/spark/issues/5427","id":94146269,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-04-18T09:15:40Z","updated_at":"2015-04-18T09:15:40Z","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30516/\nTest PASSed."}}