{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4821","labels_url":"https://api.github.com/repos/apache/spark/issues/4821/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4821/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4821/events","html_url":"https://github.com/apache/spark/pull/4821","id":59310501,"number":4821,"title":"[SPARK-6066] Make event log format easier to parse","user":{"login":"andrewor14","id":2133137,"avatar_url":"https://avatars.githubusercontent.com/u/2133137?v=3","gravatar_id":"","url":"https://api.github.com/users/andrewor14","html_url":"https://github.com/andrewor14","followers_url":"https://api.github.com/users/andrewor14/followers","following_url":"https://api.github.com/users/andrewor14/following{/other_user}","gists_url":"https://api.github.com/users/andrewor14/gists{/gist_id}","starred_url":"https://api.github.com/users/andrewor14/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andrewor14/subscriptions","organizations_url":"https://api.github.com/users/andrewor14/orgs","repos_url":"https://api.github.com/users/andrewor14/repos","events_url":"https://api.github.com/users/andrewor14/events{/privacy}","received_events_url":"https://api.github.com/users/andrewor14/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-02-27T23:38:26Z","updated_at":"2015-02-27T23:42:41Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4821","html_url":"https://github.com/apache/spark/pull/4821","diff_url":"https://github.com/apache/spark/pull/4821.diff","patch_url":"https://github.com/apache/spark/pull/4821.patch"},"body":"The event log format before was incredibly difficult to parse. Since we embed the metadata in the beginning of the file, when we compress the event log we need to skip the metadata because we need that information to parse the log later. This means we'll end up with a partially compressed file if event logging compression is turned on. The old format looks like:\r\n```\r\nsparkVersion = 1.3.0\r\ncompressionCodec = org.apache.spark.io.LZFCompressionCodec\r\n=== LOG_HEADER_END ===\r\n// actual events, could be compressed bytes\r\n```\r\nThe new format that this patch adds puts the compression codec and the Spark version in the log file name instead. It also represents the metadata in the first line of the event log as JSON, which is easy for 3rd party applications to parse:\r\n```\r\n{\"Event\": \"SparkListenerMetadataIdentifier\", \"SPARK_VERSION\":\"1.3.0\", \"COMPRESSION_CODEC\":\"...\"}\r\n// actual events\r\n```\r\nand the file name looks something like:\r\n```\r\nEVENT_LOG_app_123_SPARK_VERSION_1.3.1\r\nEVENT_LOG_app_123_SPARK_VERSION_1.3.1_COMPRESSION_CODEC_{...}\r\n```\r\n\r\nI tested this with and without compression, using different compression codecs and event logging directories. I verified that both the `Master` and the `HistoryServer` can render both compressed and uncompressed logs as before."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/76493745","html_url":"https://github.com/apache/spark/pull/4821#issuecomment-76493745","issue_url":"https://api.github.com/repos/apache/spark/issues/4821","id":76493745,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-02-27T23:42:41Z","updated_at":"2015-02-27T23:42:41Z","body":"  [Test build #28098 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28098/consoleFull) for   PR 4821 at commit [`8db5a06`](https://github.com/apache/spark/commit/8db5a06d108d8a2ddb8460e48e3509f46cc4fc2f).\n * This patch merges cleanly."}}