{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/7710","labels_url":"https://api.github.com/repos/apache/spark/issues/7710/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/7710/comments","events_url":"https://api.github.com/repos/apache/spark/issues/7710/events","html_url":"https://github.com/apache/spark/pull/7710","id":97598444,"number":7710,"title":"[SPARK-9393] [SQL] Fix several error-handling bugs in ScriptTransform operator","user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":18,"created_at":"2015-07-28T02:23:22Z","updated_at":"2015-07-28T17:29:13Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/7710","html_url":"https://github.com/apache/spark/pull/7710","diff_url":"https://github.com/apache/spark/pull/7710.diff","patch_url":"https://github.com/apache/spark/pull/7710.patch"},"body":"SparkSQL's ScriptTransform operator has several serious bugs which make debugging fairly difficult:\r\n\r\n- If exceptions are thrown in the writing thread then the child process will not be killed, leading to a deadlock because the reader thread will block while waiting for input that will never arrive.\r\n- TaskContext is not propagated to the writer thread, which may cause errors in upstream pipelined operators.\r\n- Exceptions which occur in the writer thread are not propagated to the main reader thread, which may cause upstream errors to be silently ignored instead of killing the job.  This can lead to silently incorrect query results.\r\n- The writer thread is not a daemon thread, but it should be.\r\n\r\nIn addition, the code in this file is extremely messy:\r\n\r\n- Lots of fields are nullable but the nullability isn't clearly explained.\r\n- Many confusing variable names: for instance, there are variables named `ite` and `iterator` that are defined in the same scope.\r\n- Some code was misindented.\r\n- The `*serdeClass` variables are actually expected to be single-quoted strings, which is really confusing: I feel that this parsing / extraction should be performed in the analyzer, not in the operator itself.\r\n- There were no unit tests for the operator itself, only end-to-end tests.\r\n\r\nThis pull request addresses these issues, borrowing some error-handling techniques from PySpark's PythonRDD."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/125685607","html_url":"https://github.com/apache/spark/pull/7710#issuecomment-125685607","issue_url":"https://api.github.com/repos/apache/spark/issues/7710","id":125685607,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-07-28T17:29:13Z","updated_at":"2015-07-28T17:29:13Z","body":"  [Test build #38714 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38714/console) for   PR 7710 at commit [`983f200`](https://github.com/apache/spark/commit/983f200b73b9c04432c3cf086586e5bcaef0658f).\n * This patch **passes all tests**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}