{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/5801","labels_url":"https://api.github.com/repos/apache/spark/issues/5801/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/5801/comments","events_url":"https://api.github.com/repos/apache/spark/issues/5801/events","html_url":"https://github.com/apache/spark/pull/5801","id":72086555,"number":5801,"title":"[SPARK-5938][SPARK-5443][SQL] Improve JsonRDD performance","user":{"login":"NathanHowell","id":170829,"avatar_url":"https://avatars.githubusercontent.com/u/170829?v=3","gravatar_id":"","url":"https://api.github.com/users/NathanHowell","html_url":"https://github.com/NathanHowell","followers_url":"https://api.github.com/users/NathanHowell/followers","following_url":"https://api.github.com/users/NathanHowell/following{/other_user}","gists_url":"https://api.github.com/users/NathanHowell/gists{/gist_id}","starred_url":"https://api.github.com/users/NathanHowell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/NathanHowell/subscriptions","organizations_url":"https://api.github.com/users/NathanHowell/orgs","repos_url":"https://api.github.com/users/NathanHowell/repos","events_url":"https://api.github.com/users/NathanHowell/events{/privacy}","received_events_url":"https://api.github.com/users/NathanHowell/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":95,"created_at":"2015-04-30T07:24:11Z","updated_at":"2015-05-06T16:31:51Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5801","html_url":"https://github.com/apache/spark/pull/5801","diff_url":"https://github.com/apache/spark/pull/5801.diff","patch_url":"https://github.com/apache/spark/pull/5801.patch"},"body":"This patch comprises of a few related pieces of work:\r\n\r\n* Schema inference is performed directly on the JSON token stream\r\n* `String => Row` conversion populate Spark SQL structures without intermediate types\r\n* Projection pushdown is implemented via CatalystScan for DataFrame queries\r\n* Support for the legacy parser by setting `spark.sql.json.useJacksonStreamingAPI` to `false`\r\n\r\nPerformance improvements depend on the schema and queries being executed, but it should be faster across the board. Below are benchmarks using the last.fm Million Song dataset:\r\n\r\n```\r\nCommand                                            | Baseline | Patched\r\n---------------------------------------------------|----------|--------\r\nimport sqlContext.implicits._                      |          |\r\nval df = sqlContext.jsonFile(\"/tmp/lastfm.json\")   |    70.0s |   14.6s\r\ndf.count()                                         |    28.8s |    6.2s\r\ndf.rdd.count()                                     |    35.3s |   21.5s\r\ndf.where($\"artist\" === \"Robert Hood\").collect()    |    28.3s |   16.9s\r\n```\r\n\r\nTo prepare this dataset for benchmarking, follow these steps:\r\n\r\n```\r\n# Fetch the datasets from http://labrosa.ee.columbia.edu/millionsong/lastfm\r\nwget http://labrosa.ee.columbia.edu/millionsong/sites/default/files/lastfm/lastfm_test.zip \\\r\n     http://labrosa.ee.columbia.edu/millionsong/sites/default/files/lastfm/lastfm_train.zip\r\n\r\n# Decompress and combine, pipe through `jq -c` to ensure there is one record per line\r\nunzip -p lastfm_test.zip lastfm_train.zip  | jq -c . > lastfm.json\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/99530593","html_url":"https://github.com/apache/spark/pull/5801#issuecomment-99530593","issue_url":"https://api.github.com/repos/apache/spark/issues/5801","id":99530593,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-05-06T16:31:51Z","updated_at":"2015-05-06T16:31:51Z","body":"Merged build finished. Test PASSed."}}