{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4821","labels_url":"https://api.github.com/repos/apache/spark/issues/4821/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4821/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4821/events","html_url":"https://github.com/apache/spark/pull/4821","id":59310501,"number":4821,"title":"[SPARK-6066] Make event log format easier to parse","user":{"login":"andrewor14","id":2133137,"avatar_url":"https://avatars.githubusercontent.com/u/2133137?v=3","gravatar_id":"","url":"https://api.github.com/users/andrewor14","html_url":"https://github.com/andrewor14","followers_url":"https://api.github.com/users/andrewor14/followers","following_url":"https://api.github.com/users/andrewor14/following{/other_user}","gists_url":"https://api.github.com/users/andrewor14/gists{/gist_id}","starred_url":"https://api.github.com/users/andrewor14/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andrewor14/subscriptions","organizations_url":"https://api.github.com/users/andrewor14/orgs","repos_url":"https://api.github.com/users/andrewor14/repos","events_url":"https://api.github.com/users/andrewor14/events{/privacy}","received_events_url":"https://api.github.com/users/andrewor14/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":35,"created_at":"2015-02-27T23:38:26Z","updated_at":"2015-03-02T23:02:35Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4821","html_url":"https://github.com/apache/spark/pull/4821","diff_url":"https://github.com/apache/spark/pull/4821.diff","patch_url":"https://github.com/apache/spark/pull/4821.patch"},"body":"Some users have reported difficulty in parsing the new event log format. Since we embed the metadata in the beginning of the file, when we compress the event log we need to skip the metadata because we need that information to parse the log later. This means we'll end up with a partially compressed file if event logging compression is turned on. The old format looks like:\r\n```\r\nsparkVersion = 1.3.0\r\ncompressionCodec = org.apache.spark.io.LZFCompressionCodec\r\n=== LOG_HEADER_END ===\r\n// actual events, could be compressed bytes\r\n```\r\nThe new format in this patch puts the compression codec in the log file name instead. It also removes the metadata header altogether along with the Spark version, which was not needed. The new file name looks something like:\r\n```\r\napp_without_compression\r\napp_123.lzf\r\napp_456.snappy\r\n```\r\n\r\nI tested this with and without compression, using different compression codecs and event logging directories. I verified that both the `Master` and the `HistoryServer` can render both compressed and uncompressed logs as before."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/76846594","html_url":"https://github.com/apache/spark/pull/4821#issuecomment-76846594","issue_url":"https://api.github.com/repos/apache/spark/issues/4821","id":76846594,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-03-02T23:02:35Z","updated_at":"2015-03-02T23:02:35Z","body":"  [Test build #28198 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28198/consoleFull) for   PR 4821 at commit [`fdae14c`](https://github.com/apache/spark/commit/fdae14c86f660047b50020b40a782b842182336c).\n * This patch merges cleanly."}}