{"action":"created","issue":{"url":"https://api.github.com/repos/databricks/spark-csv/issues/49","labels_url":"https://api.github.com/repos/databricks/spark-csv/issues/49/labels{/name}","comments_url":"https://api.github.com/repos/databricks/spark-csv/issues/49/comments","events_url":"https://api.github.com/repos/databricks/spark-csv/issues/49/events","html_url":"https://github.com/databricks/spark-csv/pull/49","id":68186747,"number":49,"title":"more efficient, robust and flexible csv parsing","user":{"login":"mohitjaggi","id":3218743,"avatar_url":"https://avatars.githubusercontent.com/u/3218743?v=3","gravatar_id":"","url":"https://api.github.com/users/mohitjaggi","html_url":"https://github.com/mohitjaggi","followers_url":"https://api.github.com/users/mohitjaggi/followers","following_url":"https://api.github.com/users/mohitjaggi/following{/other_user}","gists_url":"https://api.github.com/users/mohitjaggi/gists{/gist_id}","starred_url":"https://api.github.com/users/mohitjaggi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mohitjaggi/subscriptions","organizations_url":"https://api.github.com/users/mohitjaggi/orgs","repos_url":"https://api.github.com/users/mohitjaggi/repos","events_url":"https://api.github.com/users/mohitjaggi/events{/privacy}","received_events_url":"https://api.github.com/users/mohitjaggi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-04-13T20:26:48Z","updated_at":"2015-04-22T00:19:40Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/databricks/spark-csv/pulls/49","html_url":"https://github.com/databricks/spark-csv/pull/49","diff_url":"https://github.com/databricks/spark-csv/pull/49.diff","patch_url":"https://github.com/databricks/spark-csv/pull/49.patch"},"body":"Right now one unit test is broken. \r\nThe string\r\n|\"thing \r\nis parsed as \r\n|\"thing\r\nby univocity parser\r\nbut as \r\n\"thing \r\n\r\nby common csv in the last unit test.\r\n\r\nThey are both right in their own way.\r\n\r\nReason for speedup --\r\n1. Calling CSVParser.parse() creates a new CSVParser object and a new StringReader object for every line in the input CSV file. This is what I was initially doing in bigdf. So I wrote CsvReader which includes a way to parse a whole partition of data with a single parser. This required re-inserting the newline characters in the data which sc.textFile() removes. I think I do that quite efficiently, but there might be more optimizations that others can think of.\r\n2. I also switched to using Univocity parser which is more efficient. I did some tests and found it twice as fast as Apace Commons CSV.\r\n3. Univocity parser also supports more options for flexibility in parsing as well as performance tuning\r\n4. Instead of \"filtering\" the header line (which bigdf started with as well), I now use iter.drop(1) on the first partition to skip the first line.\r\n\r\nA functional change --\r\nI changed the CSV writing code (which still uses common csv) to write the header to only the first partition. I believe this is correct as although the partitions may show up as different files in the filesystem they are logically a single file. We may coalesce or repartition them and in that case the headers will no longer be on the first line. \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/databricks/spark-csv/issues/comments/94979995","html_url":"https://github.com/databricks/spark-csv/pull/49#issuecomment-94979995","issue_url":"https://api.github.com/repos/databricks/spark-csv/issues/49","id":94979995,"user":{"login":"falaki","id":512364,"avatar_url":"https://avatars.githubusercontent.com/u/512364?v=3","gravatar_id":"","url":"https://api.github.com/users/falaki","html_url":"https://github.com/falaki","followers_url":"https://api.github.com/users/falaki/followers","following_url":"https://api.github.com/users/falaki/following{/other_user}","gists_url":"https://api.github.com/users/falaki/gists{/gist_id}","starred_url":"https://api.github.com/users/falaki/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/falaki/subscriptions","organizations_url":"https://api.github.com/users/falaki/orgs","repos_url":"https://api.github.com/users/falaki/repos","events_url":"https://api.github.com/users/falaki/events{/privacy}","received_events_url":"https://api.github.com/users/falaki/received_events","type":"User","site_admin":false},"created_at":"2015-04-22T00:19:40Z","updated_at":"2015-04-22T00:19:40Z","body":"@mohitjaggi you are failing scala style check because of the different copyright message in header."}}