{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/8257","labels_url":"https://api.github.com/repos/apache/spark/issues/8257/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/8257/comments","events_url":"https://api.github.com/repos/apache/spark/issues/8257/events","html_url":"https://github.com/apache/spark/pull/8257","id":101530291,"number":8257,"title":"[SPARK-10072][STREAMING] BlockGenerator can deadlock when the queue block queue of generate blocks fills up to capacity","user":{"login":"tdas","id":663212,"avatar_url":"https://avatars.githubusercontent.com/u/663212?v=3","gravatar_id":"","url":"https://api.github.com/users/tdas","html_url":"https://github.com/tdas","followers_url":"https://api.github.com/users/tdas/followers","following_url":"https://api.github.com/users/tdas/following{/other_user}","gists_url":"https://api.github.com/users/tdas/gists{/gist_id}","starred_url":"https://api.github.com/users/tdas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdas/subscriptions","organizations_url":"https://api.github.com/users/tdas/orgs","repos_url":"https://api.github.com/users/tdas/repos","events_url":"https://api.github.com/users/tdas/events{/privacy}","received_events_url":"https://api.github.com/users/tdas/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-08-17T23:27:47Z","updated_at":"2015-08-18T01:58:16Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8257","html_url":"https://github.com/apache/spark/pull/8257","diff_url":"https://github.com/apache/spark/pull/8257.diff","patch_url":"https://github.com/apache/spark/pull/8257.patch"},"body":"Generated blocks are inserted into an ArrayBlockingQueue, and another thread pulls stuff from the ArrayBlockingQueue and pushes it into BlockManager. Now if that queue fills up to capacity (default is 10 blocks), then the inserting into queue (done in the function updateCurrentBuffer) get blocked inside a synchronized block. However, the thread that is pulling blocks from the queue uses the same lock to check the current (active or stopped) while pulling from the queue. Since the block generating threads is blocked (as the queue is full) on the lock, this thread that is supposed to drain the queue gets blocked. Ergo, deadlock.\r\n\r\nSolution: Moved blocking call to ArrayBlockingQueue outside the synchronized to prevent deadlock."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/132029418","html_url":"https://github.com/apache/spark/pull/8257#issuecomment-132029418","issue_url":"https://api.github.com/repos/apache/spark/issues/8257","id":132029418,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-08-18T01:58:16Z","updated_at":"2015-08-18T01:58:16Z","body":"Merged build finished. Test FAILed."}}