{"action":"created","issue":{"url":"https://api.github.com/repos/embulk/embulk/issues/350","labels_url":"https://api.github.com/repos/embulk/embulk/issues/350/labels{/name}","comments_url":"https://api.github.com/repos/embulk/embulk/issues/350/comments","events_url":"https://api.github.com/repos/embulk/embulk/issues/350/events","html_url":"https://github.com/embulk/embulk/issues/350","id":122702890,"number":350,"title":"How to improve embulk-input-filesplit performance","user":{"login":"benbenja","id":16332863,"avatar_url":"https://avatars.githubusercontent.com/u/16332863?v=3","gravatar_id":"","url":"https://api.github.com/users/benbenja","html_url":"https://github.com/benbenja","followers_url":"https://api.github.com/users/benbenja/followers","following_url":"https://api.github.com/users/benbenja/following{/other_user}","gists_url":"https://api.github.com/users/benbenja/gists{/gist_id}","starred_url":"https://api.github.com/users/benbenja/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benbenja/subscriptions","organizations_url":"https://api.github.com/users/benbenja/orgs","repos_url":"https://api.github.com/users/benbenja/repos","events_url":"https://api.github.com/users/benbenja/events{/privacy}","received_events_url":"https://api.github.com/users/benbenja/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-12-17T10:11:04Z","updated_at":"2015-12-18T02:54:17Z","closed_at":null,"body":"Hi,\r\n\r\n         We have lots of 100 GB files in a directory. We need to basically read those files line by line and mask couple of columns using hash function and write it to another file. For following configuration it takes 9 hrs 10 mins to process single 100 GB file (1360,000,000 rows). Our hardware configuration 8 core, 65 GB RAM and 1 TB hard disk. Is there is any way to improve the performance?\r\n\r\nin:\r\n  type: filesplit\r\n  path: {{ env.input_fpath }}\r\n  header_line: true\r\n  tasks: 24\r\n  parser:\r\n    charset: UTF-8\r\n    newline: CRLF\r\n    type: csv\r\n    delimiter: ','\r\n    quote: '\"'\r\n    escape: '\"'\r\n    trim_if_not_quoted: false\r\n    skip_header_lines: 0\r\n    allow_extra_columns: false\r\n    allow_optional_columns: false\r\n    columns:\r\n    - {name: timestamp, type: string}\r\n    - {name: src_ip, type: string}\r\n    - {name: src_port, type: long}\r\n    - {name: dest_ip, type: string}\r\n    - {name: dest_port, type: long}\r\n    - {name: user_id, type: string}\r\n    - {name: in_bytes, type: long}\r\n    - {name: out_bytes, type: long}\r\nfilters:\r\n  - type: hash\r\n    columns:\r\n    - {name: src_ip, algorithm: SHA-512}\r\n    - {name: user_id, algorithm: SHA-512}\r\nout:\r\n  type: file\r\n  path_prefix: {{ env.output_dpath }}/{{ env.fname }}_\r\n  sequence_format: '%03d.'\r\n  file_ext: csv\r\n  formatter: {type: csv}\r\n\r\n\r\nThanks,\r\nBen"},"comment":{"url":"https://api.github.com/repos/embulk/embulk/issues/comments/165655319","html_url":"https://github.com/embulk/embulk/issues/350#issuecomment-165655319","issue_url":"https://api.github.com/repos/embulk/embulk/issues/350","id":165655319,"user":{"login":"benbenja","id":16332863,"avatar_url":"https://avatars.githubusercontent.com/u/16332863?v=3","gravatar_id":"","url":"https://api.github.com/users/benbenja","html_url":"https://github.com/benbenja","followers_url":"https://api.github.com/users/benbenja/followers","following_url":"https://api.github.com/users/benbenja/following{/other_user}","gists_url":"https://api.github.com/users/benbenja/gists{/gist_id}","starred_url":"https://api.github.com/users/benbenja/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benbenja/subscriptions","organizations_url":"https://api.github.com/users/benbenja/orgs","repos_url":"https://api.github.com/users/benbenja/repos","events_url":"https://api.github.com/users/benbenja/events{/privacy}","received_events_url":"https://api.github.com/users/benbenja/received_events","type":"User","site_admin":false},"created_at":"2015-12-18T02:54:17Z","updated_at":"2015-12-18T02:54:17Z","body":"Thanks for the suggestion. Will try the same.\r\n\r\nRegards,\r\nBen"}}