{"action":"created","issue":{"url":"https://api.github.com/repos/spotify/cassandra-reaper/issues/89","labels_url":"https://api.github.com/repos/spotify/cassandra-reaper/issues/89/labels{/name}","comments_url":"https://api.github.com/repos/spotify/cassandra-reaper/issues/89/comments","events_url":"https://api.github.com/repos/spotify/cassandra-reaper/issues/89/events","html_url":"https://github.com/spotify/cassandra-reaper/issues/89","id":67462790,"number":89,"title":"Entire repair run moves to ERROR state on exceptions in JMX methods","user":{"login":"ahenry","id":620115,"avatar_url":"https://avatars.githubusercontent.com/u/620115?v=3","gravatar_id":"","url":"https://api.github.com/users/ahenry","html_url":"https://github.com/ahenry","followers_url":"https://api.github.com/users/ahenry/followers","following_url":"https://api.github.com/users/ahenry/following{/other_user}","gists_url":"https://api.github.com/users/ahenry/gists{/gist_id}","starred_url":"https://api.github.com/users/ahenry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ahenry/subscriptions","organizations_url":"https://api.github.com/users/ahenry/orgs","repos_url":"https://api.github.com/users/ahenry/repos","events_url":"https://api.github.com/users/ahenry/events{/privacy}","received_events_url":"https://api.github.com/users/ahenry/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-04-09T22:19:56Z","updated_at":"2015-04-10T08:27:01Z","closed_at":null,"body":"I've had several repair runs fail due to transient errors (usually involving a node going up or down, but now always).  The exceptions are thrown by JMX remote calls.  For example:\r\n\r\n    ERROR [2015-03-30 22:37:34,539] com.spotify.reaper.service.RepairRunner: RepairRun FAILURE\r\n    ERROR [2015-03-30 22:37:34,540] com.spotify.reaper.service.RepairRunner: java.lang.reflect.UndeclaredThrowableException\r\n    ERROR [2015-03-30 22:37:34,540] com.spotify.reaper.service.RepairRunner: [com.sun.proxy.$Proxy59.forceTerminateAllRepairSessions(Unknown Source), com.spotify.reaper.cassandra.JmxProxy.cancelAllRepairs(JmxProxy.java:265), com.spotify.reaper.service.SegmentRunner.abort(SegmentRunner.java:89), com.spotify.reaper.service.SegmentRunner.abort(SegmentRunner.java:212), com.spotify.reaper.service.SegmentRunner.runRepair(SegmentRunner.java:150), com.spotify.reaper.service.SegmentRunner.triggerRepair(SegmentRunner.java:70), com.spotify.reaper.service.RepairRunner.repairSegment(RepairRunner.java:202), com.spotify.reaper.service.RepairRunner.startNextSegment(RepairRunner.java:156), com.spotify.reaper.service.RepairRunner.run(RepairRunner.java:89), java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471), java.util.concurrent.FutureTask.run(FutureTask.java:262), java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178), java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292), java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145), java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615), java.lang.Thread.run(Thread.java:745)]\r\n\r\nand\r\n\r\n    ERROR [2015-04-08 21:19:01,950] com.spotify.reaper.service.RepairRunner: RepairRun FAILURE\r\n    ERROR [2015-04-08 21:19:01,950] com.spotify.reaper.service.RepairRunner: java.lang.reflect.UndeclaredThrowableException\r\n    ERROR [2015-04-08 21:19:01,950] com.spotify.reaper.service.RepairRunner: [com.sun.proxy.$Proxy60.getPendingTasks(Unknown Source), com.spotify.reaper.cassandra.JmxProxy.getPendingCompactions(JmxProxy.java:232), com.spotify.reaper.service.SegmentRunner.canRepair(SegmentRunner.java:177), com.spotify.reaper.service.SegmentRunner.runRepair(SegmentRunner.java:105), com.spotify.reaper.service.SegmentRunner.triggerRepair(SegmentRunner.java:70), com.spotify.reaper.service.RepairRunner.repairSegment(RepairRunner.java:202), com.spotify.reaper.service.RepairRunner.startNextSegment(RepairRunner.java:156), com.spotify.reaper.service.RepairRunner.run(RepairRunner.java:89), java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471), java.util.concurrent.FutureTask.run(FutureTask.java:262), java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178), java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292), java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145), java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615), java.lang.Thread.run(Thread.java:745)]\r\n\r\nI've also seen an exception (don't have the logs, sorry) thrown by tokenRangeToEndpoint.\r\n\r\nFollowing the style of ed21152e20e3c8a2bd7923b8b7ebaeb9c73755cd, I wrote a patch that catches RuntimeException in a couple of places in SegmentRunner.canRepair()\r\n\r\nAre you interested in a pull request, a patch file, or something else?"},"comment":{"url":"https://api.github.com/repos/spotify/cassandra-reaper/issues/comments/91476804","html_url":"https://github.com/spotify/cassandra-reaper/issues/89#issuecomment-91476804","issue_url":"https://api.github.com/repos/spotify/cassandra-reaper/issues/89","id":91476804,"user":{"login":"Yarin78","id":849617,"avatar_url":"https://avatars.githubusercontent.com/u/849617?v=3","gravatar_id":"","url":"https://api.github.com/users/Yarin78","html_url":"https://github.com/Yarin78","followers_url":"https://api.github.com/users/Yarin78/followers","following_url":"https://api.github.com/users/Yarin78/following{/other_user}","gists_url":"https://api.github.com/users/Yarin78/gists{/gist_id}","starred_url":"https://api.github.com/users/Yarin78/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yarin78/subscriptions","organizations_url":"https://api.github.com/users/Yarin78/orgs","repos_url":"https://api.github.com/users/Yarin78/repos","events_url":"https://api.github.com/users/Yarin78/events{/privacy}","received_events_url":"https://api.github.com/users/Yarin78/received_events","type":"User","site_admin":false},"created_at":"2015-04-10T08:27:01Z","updated_at":"2015-04-10T08:27:01Z","body":"Hi,\r\n\r\nA PR is fine!\r\n\r\nI think I fixed this one place recently (see https://github.com/spotify/cassandra-reaper/commit/ed21152e20e3c8a2bd7923b8b7ebaeb9c73755cd)."}}