{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4067","labels_url":"https://api.github.com/repos/apache/spark/issues/4067/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4067/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4067/events","html_url":"https://github.com/apache/spark/pull/4067","id":54532482,"number":4067,"title":"[SPARK-4874] [CORE] Collect record count metrics","user":{"login":"ksakellis","id":6590087,"avatar_url":"https://avatars.githubusercontent.com/u/6590087?v=3","gravatar_id":"","url":"https://api.github.com/users/ksakellis","html_url":"https://github.com/ksakellis","followers_url":"https://api.github.com/users/ksakellis/followers","following_url":"https://api.github.com/users/ksakellis/following{/other_user}","gists_url":"https://api.github.com/users/ksakellis/gists{/gist_id}","starred_url":"https://api.github.com/users/ksakellis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ksakellis/subscriptions","organizations_url":"https://api.github.com/users/ksakellis/orgs","repos_url":"https://api.github.com/users/ksakellis/repos","events_url":"https://api.github.com/users/ksakellis/events{/privacy}","received_events_url":"https://api.github.com/users/ksakellis/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-01-16T02:18:58Z","updated_at":"2015-01-16T03:49:11Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4067","html_url":"https://github.com/apache/spark/pull/4067","diff_url":"https://github.com/apache/spark/pull/4067.diff","patch_url":"https://github.com/apache/spark/pull/4067.patch"},"body":"Collects record counts for both Input/Output and Shuffle Metrics. For the input/output metrics, it just appends the counter every time the iterators get accessed.\r\n    \r\nFor shuffle on the write side, we count the metrics post aggregation (after a map side combine) and on the read side we count the metrics pre aggregation. This allows both the bytes read/written metrics and the records read/written to line up.\r\n\r\nFor backwards compatibility, if we deserialize an older event that doesn't have record metrics, we set the metric to -1."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/70204733","html_url":"https://github.com/apache/spark/pull/4067#issuecomment-70204733","issue_url":"https://api.github.com/repos/apache/spark/issues/4067","id":70204733,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-01-16T03:49:11Z","updated_at":"2015-01-16T03:49:11Z","body":"  [Test build #25628 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25628/consoleFull) for   PR 4067 at commit [`571cb69`](https://github.com/apache/spark/commit/571cb69f694b7f05e829c7237a07f68fa058e118).\n * This patch **passes all tests**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}