{"action":"created","issue":{"url":"https://api.github.com/repos/thinkaurelius/titan/issues/908","labels_url":"https://api.github.com/repos/thinkaurelius/titan/issues/908/labels{/name}","comments_url":"https://api.github.com/repos/thinkaurelius/titan/issues/908/comments","events_url":"https://api.github.com/repos/thinkaurelius/titan/issues/908/events","html_url":"https://github.com/thinkaurelius/titan/issues/908","id":53366389,"number":908,"title":"Titan server 0.5.2 does not recover from backend failure","user":{"login":"smalyshev","id":155000,"avatar_url":"https://avatars.githubusercontent.com/u/155000?v=3","gravatar_id":"","url":"https://api.github.com/users/smalyshev","html_url":"https://github.com/smalyshev","followers_url":"https://api.github.com/users/smalyshev/followers","following_url":"https://api.github.com/users/smalyshev/following{/other_user}","gists_url":"https://api.github.com/users/smalyshev/gists{/gist_id}","starred_url":"https://api.github.com/users/smalyshev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smalyshev/subscriptions","organizations_url":"https://api.github.com/users/smalyshev/orgs","repos_url":"https://api.github.com/users/smalyshev/repos","events_url":"https://api.github.com/users/smalyshev/events{/privacy}","received_events_url":"https://api.github.com/users/smalyshev/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-01-05T05:50:50Z","updated_at":"2015-01-05T12:08:42Z","closed_at":null,"body":"I'm running Titan server 0.5.2 on top of Cassandra and if Cassandra goes down, I get exceptions like this:\r\n\r\n```\r\n23126 [pool-3-thread-1] WARN  com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog  - Could not read messages for timestamp [Timepoint[1419916997444000 μs]] (this read will be retried)\r\ncom.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend exception\r\nat com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44)\r\nat com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:144)\r\nat com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:706)\r\nat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\r\n...\r\n\r\nCaused by: java.net.SocketException: Broken pipe\r\nat java.net.SocketOutputStream.socketWrite0(Native Method)\r\nat java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)\r\nat java.net.SocketOutputStream.write(SocketOutputStream.java:153)\r\nat java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)\r\nat java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)\r\nat org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)\r\n\r\n... 22 more\r\n```\r\n(full text here: http://pastebin.com/1nMgZ6eN) and the connection never recovers. Worse yet, connecting to the server with rexster console starts to give this error on any query:\r\n\r\n```\r\n==>An error occurred while processing the script for language [groovy]. All transactions across all graphs in the session have been concluded with failure: java.util.concurrent.ExecutionException: javax.script.ScriptException: javax.script.ScriptException: com.thinkaurelius.titan.core.TitanException: Could not process individual retrieval call \r\n```\r\n\r\nSo far I have found no way to recover from it except shutting down the Titan server and restarting it again. I do not think it is good to run it in production this way - so that any problem on the storage backend would make the server permanently unusable even after the storage backend came up and will require manual restart. "},"comment":{"url":"https://api.github.com/repos/thinkaurelius/titan/issues/comments/68699976","html_url":"https://github.com/thinkaurelius/titan/issues/908#issuecomment-68699976","issue_url":"https://api.github.com/repos/thinkaurelius/titan/issues/908","id":68699976,"user":{"login":"spmallette","id":384249,"avatar_url":"https://avatars.githubusercontent.com/u/384249?v=3","gravatar_id":"","url":"https://api.github.com/users/spmallette","html_url":"https://github.com/spmallette","followers_url":"https://api.github.com/users/spmallette/followers","following_url":"https://api.github.com/users/spmallette/following{/other_user}","gists_url":"https://api.github.com/users/spmallette/gists{/gist_id}","starred_url":"https://api.github.com/users/spmallette/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spmallette/subscriptions","organizations_url":"https://api.github.com/users/spmallette/orgs","repos_url":"https://api.github.com/users/spmallette/repos","events_url":"https://api.github.com/users/spmallette/events{/privacy}","received_events_url":"https://api.github.com/users/spmallette/received_events","type":"User","site_admin":false},"created_at":"2015-01-05T12:08:42Z","updated_at":"2015-01-05T12:08:42Z","body":"Are you seeing this when explicitly using Titan Server (i.e. using `titan.sh`)?  I don't see this problem when running a cassandra instance and rexster instance separately.  I can kill cassandra, rexster will throw some errors if it tries to connect, then if i restart cassandra, rexster is happy to start working again. I did configure rexster to work with `storage.backend=cassandra` for what that is worth (i.e astyanax).\r\n\r\nI didn't try Rexster Console at all....but generally speaking, Rexster Console isn't always good at reconnecting to a Rexster instance that has gone down.  A restart is sometimes necessary.  In this case though, Rexster has stayed up and cassandra has gone down, so I would expect that once Rexster recovers its connection to cassandra, Rexster Console should start working again."}}