{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/3638","labels_url":"https://api.github.com/repos/apache/spark/issues/3638/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/3638/comments","events_url":"https://api.github.com/repos/apache/spark/issues/3638/events","html_url":"https://github.com/apache/spark/pull/3638","id":51349729,"number":3638,"title":"[SPARK-4737] Task set manager properly handles serialization errors","user":{"login":"mccheah","id":8975385,"avatar_url":"https://avatars.githubusercontent.com/u/8975385?v=3","gravatar_id":"","url":"https://api.github.com/users/mccheah","html_url":"https://github.com/mccheah","followers_url":"https://api.github.com/users/mccheah/followers","following_url":"https://api.github.com/users/mccheah/following{/other_user}","gists_url":"https://api.github.com/users/mccheah/gists{/gist_id}","starred_url":"https://api.github.com/users/mccheah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mccheah/subscriptions","organizations_url":"https://api.github.com/users/mccheah/orgs","repos_url":"https://api.github.com/users/mccheah/repos","events_url":"https://api.github.com/users/mccheah/events{/privacy}","received_events_url":"https://api.github.com/users/mccheah/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":30,"created_at":"2014-12-08T20:44:14Z","updated_at":"2015-01-09T01:06:19Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/3638","html_url":"https://github.com/apache/spark/pull/3638","diff_url":"https://github.com/apache/spark/pull/3638.diff","patch_url":"https://github.com/apache/spark/pull/3638.patch"},"body":"Dealing with [SPARK-4737], the handling of serialization errors should not be the DAGScheduler's responsibility. The task set manager now catches the error and aborts the stage.\r\n\r\nIf the TaskSetManager throws a TaskNotSerializableException, the TaskSchedulerImpl will return an empty list of task descriptions, because no tasks were started. The scheduler should abort the stage gracefully.\r\n\r\nNote that I'm not too familiar with this part of the codebase and its place in the overall architecture of the Spark stack. If implementing it this way will have any averse side effects please voice that loudly."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/69278693","html_url":"https://github.com/apache/spark/pull/3638#issuecomment-69278693","issue_url":"https://api.github.com/repos/apache/spark/issues/3638","id":69278693,"user":{"login":"andrewor14","id":2133137,"avatar_url":"https://avatars.githubusercontent.com/u/2133137?v=3","gravatar_id":"","url":"https://api.github.com/users/andrewor14","html_url":"https://github.com/andrewor14","followers_url":"https://api.github.com/users/andrewor14/followers","following_url":"https://api.github.com/users/andrewor14/following{/other_user}","gists_url":"https://api.github.com/users/andrewor14/gists{/gist_id}","starred_url":"https://api.github.com/users/andrewor14/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andrewor14/subscriptions","organizations_url":"https://api.github.com/users/andrewor14/orgs","repos_url":"https://api.github.com/users/andrewor14/repos","events_url":"https://api.github.com/users/andrewor14/events{/privacy}","received_events_url":"https://api.github.com/users/andrewor14/received_events","type":"User","site_admin":false},"created_at":"2015-01-09T01:06:19Z","updated_at":"2015-01-09T01:06:19Z","body":"@mccheah @JoshRosen high level question. So what happens now when a task is not serializable? Before it would throw a loud exception and fail the task, but now we catch the task not serializable exception and silently not schedule it. I may be missing something, but do we ever abort the stage or fail the task?"}}