{"action":"created","issue":{"url":"https://api.github.com/repos/JuliaLang/julia/issues/10428","labels_url":"https://api.github.com/repos/JuliaLang/julia/issues/10428/labels{/name}","comments_url":"https://api.github.com/repos/JuliaLang/julia/issues/10428/comments","events_url":"https://api.github.com/repos/JuliaLang/julia/issues/10428/events","html_url":"https://github.com/JuliaLang/julia/issues/10428","id":60192966,"number":10428,"title":"Poor performance of readdlm on master","user":{"login":"jiahao","id":1732,"avatar_url":"https://avatars.githubusercontent.com/u/1732?v=3","gravatar_id":"","url":"https://api.github.com/users/jiahao","html_url":"https://github.com/jiahao","followers_url":"https://api.github.com/users/jiahao/followers","following_url":"https://api.github.com/users/jiahao/following{/other_user}","gists_url":"https://api.github.com/users/jiahao/gists{/gist_id}","starred_url":"https://api.github.com/users/jiahao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jiahao/subscriptions","organizations_url":"https://api.github.com/users/jiahao/orgs","repos_url":"https://api.github.com/users/jiahao/repos","events_url":"https://api.github.com/users/jiahao/events{/privacy}","received_events_url":"https://api.github.com/users/jiahao/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/JuliaLang/julia/labels/performance","name":"performance","color":"d7e102"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/JuliaLang/julia/milestones/11","html_url":"https://github.com/JuliaLang/julia/milestones/0.4.0","labels_url":"https://api.github.com/repos/JuliaLang/julia/milestones/11/labels","id":748886,"number":11,"title":"0.4.0","description":"For tracking release-defining items for 0.4: important new features, breaking changes, and blocking bugs.\r\n\r\nAfter a brief period for deciding scope, adding issues to this milestone is not allowed. Once the items in this milestone are done, 0.4 will enter its stabilizing period and a release candidate can be made soon after.","creator":{"login":"JeffBezanson","id":744556,"avatar_url":"https://avatars.githubusercontent.com/u/744556?v=3","gravatar_id":"","url":"https://api.github.com/users/JeffBezanson","html_url":"https://github.com/JeffBezanson","followers_url":"https://api.github.com/users/JeffBezanson/followers","following_url":"https://api.github.com/users/JeffBezanson/following{/other_user}","gists_url":"https://api.github.com/users/JeffBezanson/gists{/gist_id}","starred_url":"https://api.github.com/users/JeffBezanson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JeffBezanson/subscriptions","organizations_url":"https://api.github.com/users/JeffBezanson/orgs","repos_url":"https://api.github.com/users/JeffBezanson/repos","events_url":"https://api.github.com/users/JeffBezanson/events{/privacy}","received_events_url":"https://api.github.com/users/JeffBezanson/received_events","type":"User","site_admin":false},"open_issues":27,"closed_issues":37,"state":"open","created_at":"2014-08-10T18:13:46Z","updated_at":"2015-04-22T04:40:00Z","due_on":"2015-06-20T18:30:00Z","closed_at":null},"comments":26,"created_at":"2015-03-07T06:06:48Z","updated_at":"2015-04-22T18:58:57Z","closed_at":null,"body":"Here is a simple benchmark for reading in real-world data in TSV (tab-separated values) format. The data are financial time series from a proprietary source with 47 fields. The smaller samples are constructed from taking the first `n` rows of the original file, which has 100,654,985 rows and is 21.879 GiB in size.\r\n\r\nTimings reported in seconds. Timings under 60s are best of 5. Timings over 60s were run once.\r\n\r\nnumber of rows | `wc -l` | `grep -c` | `data.table fread` | `R read.table` | `pandas.read_csv` | `readdlm` | `DataFrames.readtable`  | `timeit pandas.read_csv` | `gc_disabled readdlm` | `gc_disabled DataFrames.readtable`\r\n----------- | ------ | ------ | -------- | -------- | ------ | --------- | ------- | ------ | ------- | -------\r\n      1,000 |  0.003 |  0.003 |   0.013* |    0.035 |  0.023 |     0.028 |   0.033 |  0.013 |   0.029 |   0.020\r\n     10,000 |  0.004 |  0.007 |   0.052* |    0.316 |  0.100 |     0.353 |   0.357 |  0.080 |   0.316 |   0.202\r\n    100,000 |  0.017 |  0.035 |   0.461* |    5.190 |  0.666 |     6.177 |  20.090 |  0.749 |   3.424 |   2.643\r\n  1,000,000 |  0.132 |  0.278 |   4.604* |   48.302 |  7.846 |   268.673 |  47.400 |  6.704 |  35.731 |  24.813\r\n 10,000,000 |  1.292 |  2.732 |  47.503* |  549.017 | 93.413 | 25298.222 | 916.866 | 71.286 | 348.457 | 264.402\r\n100,654,985 | 12.691 | 25.779 | 725.966* | 5147.951 | 799.608* |           |         | 825.816* |         |\r\n\r\n*timings obtained from files with NUL characters stripped.\r\n\r\nNote that the pandas timings obtained with `timeit` have the [Python garbage collector disabled](https://docs.python.org/2/library/timeit.html#timeit.Timer.timeit), and so are fair comparisons with the `gc_disable`d numbers in Julia. The first pandas column is timed with simple `time.time()` wrappers and are analogous to Julia `@time` reports.\r\n\r\nIt's quite clear from the timings that garbage collection causes the scaling to become superlinear, with the result that `readdlm` and `readtable` become unusable for large data, even on a machine large enough to read all the data into memory. However, the baseline performance of pandas is still far superior to what we have to offer in Julia, even with garbage collection turned off. This is perhaps unsurprising, since the main pandas parser is written in Cython and not pure Python.\r\n\r\nWe could probably do much better if `readdlm` generated less garbage.\r\n\r\n---\r\n\r\nActual commands run:\r\n\r\n```\r\nwc -l datafile.tsv\r\ngrep -c AAPL datafile.tsv\r\n```\r\n```py\r\n#Python 2.7.9 |Anaconda 2.1.0\r\nt=time(); A=pandas.read_csv(\"datafile.tsv\", sep=\"\\t\", header=None); time()-t\r\ntimeit.timeit('pandas.read_csv(\"datafile.tsv\", sep=\"\\t\", header=None)', setup='import pandas', number=5)/5\r\n```\r\n```jl\r\n#Julia 0.4-dev\r\ngc(); readdlm(\"datafile.tsv\", '\\t');\r\ngc(); DataFrames.readtable(\"datafile.tsv\", separator = '\\t', header = false);\r\ngc_disable(); #repeat readdlm, readtable, etc.\r\n```\r\n\r\n```r\r\n#R 3.1.2\r\n> system.time(read.table(\"datafile.tsv\", sep=\"\\t\", header=FALSE, skipNul=TRUE))                                                                                                   \r\n   user  system elapsed \r\n  0.034   0.000   0.035\r\n... similarly\r\n  0.316   0.000   0.316 \r\n  5.148   0.040   5.190 \r\n 47.191   0.802  48.302 \r\n535.361  10.463 549.017\r\n4867.606  229.902 5147.951\r\n\r\n> system.time(fread(\"datafile.tsv\", sep=\"\\t\", header=FALSE)) #Chokes on NUL characters\r\n   user  system elapsed \r\n  0.008   0.004   0.013 \r\n... similarly\r\n  0.052   0.000   0.052 \r\n  0.457   0.004   0.461 \r\n  4.430   0.173   4.604 \r\n 46.230   1.051  47.503 \r\n703.847  18.296 725.966 \r\n\r\n```\r\nJulia `@time` info\r\n\r\n```\r\nreaddlm (returns Array{Any,2})\r\n10^3 elapsed time: 0.027763177 seconds (7 MB allocated)\r\n10^4 elapsed time: 0.353573064 seconds (80 MB allocated, 29.06% gc time in 3 pauses with 0 full sweep)\r\n10^5 elapsed time: 6.177088453 seconds (809 MB allocated, 52.92% gc time in 19 pauses with 4 full sweep)\r\n10^6 elapsed time: 268.672912983 seconds (8097 MB allocated, 86.89% gc time in 160 pauses with 17 full sweep\r\n10^7 elapsed time: 25298.222635691 seconds (80990 MB allocated, 98.63% gc time in 2424 pauses with 177 full sweep)\r\n\r\nDataFrames.readtable\r\n10^3 elapsed time: 0.033111172 seconds (4 MB allocated)\r\n10^4 elapsed time: 0.356714838 seconds (41 MB allocated, 11.54% gc time in 1 pauses with 0 full sweep)\r\n10^5 elapsed time: 20.089927057 seconds (433 MB allocated, 79.65% gc time in 5 pauses with 3 full sweep)\r\n10^6 elapsed time: 47.399711163 seconds (4176 MB allocated, 13.33% gc time in 40 pauses with 8 full sweep)\r\n10^7 elapsed time: 916.86617322 seconds (40755 MB allocated, 52.70% gc time in 523 pauses with 39 full sweep)\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/JuliaLang/julia/issues/comments/95303572","html_url":"https://github.com/JuliaLang/julia/issues/10428#issuecomment-95303572","issue_url":"https://api.github.com/repos/JuliaLang/julia/issues/10428","id":95303572,"user":{"login":"JeffBezanson","id":744556,"avatar_url":"https://avatars.githubusercontent.com/u/744556?v=3","gravatar_id":"","url":"https://api.github.com/users/JeffBezanson","html_url":"https://github.com/JeffBezanson","followers_url":"https://api.github.com/users/JeffBezanson/followers","following_url":"https://api.github.com/users/JeffBezanson/following{/other_user}","gists_url":"https://api.github.com/users/JeffBezanson/gists{/gist_id}","starred_url":"https://api.github.com/users/JeffBezanson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JeffBezanson/subscriptions","organizations_url":"https://api.github.com/users/JeffBezanson/orgs","repos_url":"https://api.github.com/users/JeffBezanson/repos","events_url":"https://api.github.com/users/JeffBezanson/events{/privacy}","received_events_url":"https://api.github.com/users/JeffBezanson/received_events","type":"User","site_admin":false},"created_at":"2015-04-22T18:58:57Z","updated_at":"2015-04-22T18:58:57Z","body":"If people want to disable gc to tweak performance, that's fine. But there is a massive difference. 2x slower is acceptable, 100x is not."}}