{"action":"closed","issue":{"url":"https://api.github.com/repos/hazelcast/hazelcast/issues/6168","labels_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/6168/labels{/name}","comments_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/6168/comments","events_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/6168/events","html_url":"https://github.com/hazelcast/hazelcast/issues/6168","id":105776237,"number":6168,"title":"Client fails to proeperly reconnect to single node cluster after hazelcast server is restarted","user":{"login":"kedar-joshi","id":13585883,"avatar_url":"https://avatars.githubusercontent.com/u/13585883?v=3","gravatar_id":"","url":"https://api.github.com/users/kedar-joshi","html_url":"https://github.com/kedar-joshi","followers_url":"https://api.github.com/users/kedar-joshi/followers","following_url":"https://api.github.com/users/kedar-joshi/following{/other_user}","gists_url":"https://api.github.com/users/kedar-joshi/gists{/gist_id}","starred_url":"https://api.github.com/users/kedar-joshi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kedar-joshi/subscriptions","organizations_url":"https://api.github.com/users/kedar-joshi/orgs","repos_url":"https://api.github.com/users/kedar-joshi/repos","events_url":"https://api.github.com/users/kedar-joshi/events{/privacy}","received_events_url":"https://api.github.com/users/kedar-joshi/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/hazelcast/hazelcast/labels/Team:%20Client","name":"Team: Client","color":"99ccff"},{"url":"https://api.github.com/repos/hazelcast/hazelcast/labels/Type:%20Defect","name":"Type: Defect","color":"fc2929"}],"state":"closed","locked":false,"assignee":{"login":"sancar","id":570462,"avatar_url":"https://avatars.githubusercontent.com/u/570462?v=3","gravatar_id":"","url":"https://api.github.com/users/sancar","html_url":"https://github.com/sancar","followers_url":"https://api.github.com/users/sancar/followers","following_url":"https://api.github.com/users/sancar/following{/other_user}","gists_url":"https://api.github.com/users/sancar/gists{/gist_id}","starred_url":"https://api.github.com/users/sancar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sancar/subscriptions","organizations_url":"https://api.github.com/users/sancar/orgs","repos_url":"https://api.github.com/users/sancar/repos","events_url":"https://api.github.com/users/sancar/events{/privacy}","received_events_url":"https://api.github.com/users/sancar/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/hazelcast/hazelcast/milestones/68","html_url":"https://github.com/hazelcast/hazelcast/milestones/3.5.3","labels_url":"https://api.github.com/repos/hazelcast/hazelcast/milestones/68/labels","id":1277459,"number":68,"title":"3.5.3","description":null,"creator":{"login":"sancar","id":570462,"avatar_url":"https://avatars.githubusercontent.com/u/570462?v=3","gravatar_id":"","url":"https://api.github.com/users/sancar","html_url":"https://github.com/sancar","followers_url":"https://api.github.com/users/sancar/followers","following_url":"https://api.github.com/users/sancar/following{/other_user}","gists_url":"https://api.github.com/users/sancar/gists{/gist_id}","starred_url":"https://api.github.com/users/sancar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sancar/subscriptions","organizations_url":"https://api.github.com/users/sancar/orgs","repos_url":"https://api.github.com/users/sancar/repos","events_url":"https://api.github.com/users/sancar/events{/privacy}","received_events_url":"https://api.github.com/users/sancar/received_events","type":"User","site_admin":false},"open_issues":35,"closed_issues":15,"state":"open","created_at":"2015-08-28T12:07:27Z","updated_at":"2015-09-17T09:34:16Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2015-09-10T10:08:59Z","updated_at":"2015-09-17T09:34:16Z","closed_at":"2015-09-17T09:34:16Z","body":"Hi,\r\nI ran into following exception while upgrading to version 3.5.2 from version 3.2.3. The scenario is as follows -\r\n\r\n1. Use JAVA client programmatic configuration to connect to single node Hazelcast instance. This step work as expected and client is able to perform get and put operations on the distributed maps\r\n2. Shutdown the Hazelcast server and restart it after couple of seconds.\r\n3. The client is shown to have connected to the cluster, but all the map operations take very long to return. After a long time these map operations return with following exceptions, and the client never recovers.\r\n\r\nI am using following code to configure the client -\r\n\r\n```\r\n// Preparing client configuration\r\nClientConfig clientConfig = new ClientConfig();\r\nclientConfig.getGroupConfig().setName(SystemConfigUtil.getCacheGroup()).setPassword(SystemConfigUtil.getCachePassword());\r\n\r\nClientNetworkConfig networkConfig = clientConfig.getNetworkConfig();\r\n\r\nnetworkConfig.addAddress(\"127.0.0.1\");\r\nnetworkConfig.setConnectionAttemptLimit(Integer.MAX_VALUE);\r\nnetworkConfig.setConnectionAttemptPeriod(10000);\r\nnetworkConfig.setConnectionTimeout(5000);\r\n\r\n// Creating client instance\r\nHAZELCAST_CLIENT = HazelcastClient.newHazelcastClient(clientConfig);\r\n\r\n```\r\n\r\nFollowing exception is thrown for every map operation after Hazelcast server is restarted -\r\n\r\n```\r\ncom.hazelcast.core.HazelcastException: java.io.IOException: Not able to setup owner connection!\r\n at com.hazelcast.util.ExceptionUtil.rethrow(ExceptionUtil.java:67) ~[ExceptionUtil.class:3.5.2]\r\n at com.hazelcast.util.ExceptionUtil.rethrow(ExceptionUtil.java:62) ~[ExceptionUtil.class:3.5.2]\r\n at com.hazelcast.client.spi.ClientProxy.invoke(ClientProxy.java:133) ~[ClientProxy.class:3.5.2]\r\n at com.hazelcast.client.proxy.ClientMapProxy.put(ClientMapProxy.java:364) ~[ClientMapProxy.class:3.5.2]\r\n at com.hazelcast.client.proxy.ClientMapProxy.put(ClientMapProxy.java:206) ~[ClientMapProxy.class:3.5.2]\r\n at com.locationguru.CSF.cache.manager.CacheManagerImpl.setUserInfo(CacheManagerImpl.java:826) ~[CacheManagerImpl.class:na]\r\n at com.locationguru.llp.user.controller.UserController.doAuthenticate(UserController.java:266) ~[UserController.class:na]\r\n at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_71]\r\n at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_71]\r\n at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_71]\r\n at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_71]\r\n at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:213) [InvocableHandlerMethod.class:3.1.1.RELEASE]\r\n at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126) [InvocableHandlerMethod.class:3.1.1.RELEASE]\r\n at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96) [ServletInvocableHandlerMethod.class:3.1.1.RELEASE]\r\n at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617) [RequestMappingHandlerAdapter.class:3.1.1.RELEASE]\r\n at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578) [RequestMappingHandlerAdapter.class:3.1.1.RELEASE]\r\n at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80) [AbstractHandlerMethodAdapter.class:3.1.1.RELEASE]\r\n at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923) [DispatcherServlet.class:3.1.1.RELEASE]\r\n at com.locationguru.CSF.base.BaseDispatcherServlet.doDispatch(BaseDispatcherServlet.java:37) [BaseDispatcherServlet.class:na]\r\n at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852) [DispatcherServlet.class:3.1.1.RELEASE]\r\n at com.locationguru.CSF.base.BaseDispatcherServlet.doService(BaseDispatcherServlet.java:32) [BaseDispatcherServlet.class:na]\r\n at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882) [FrameworkServlet.class:3.1.1.RELEASE]\r\n at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789) [FrameworkServlet.class:3.1.1.RELEASE]\r\n at javax.servlet.http.HttpServlet.service(HttpServlet.java:644) [servlet-api.jar:na]\r\n at com.locationguru.CSF.base.BaseDispatcherServlet.service(BaseDispatcherServlet.java:42) [BaseDispatcherServlet.class:na]\r\n at javax.servlet.http.HttpServlet.service(HttpServlet.java:725) [servlet-api.jar:na]\r\n at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291) [catalina.jar:8.0.12]\r\n at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.12]\r\n at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.0.12]\r\n at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.12]\r\n at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.12]\r\n at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) [catalina.jar:8.0.12]\r\n at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) [catalina.jar:8.0.12]\r\n at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) [catalina.jar:8.0.12]\r\n at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142) [catalina.jar:8.0.12]\r\n at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [catalina.jar:8.0.12]\r\n at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610) [catalina.jar:8.0.12]\r\n at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) [catalina.jar:8.0.12]\r\n at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:534) [catalina.jar:8.0.12]\r\n at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1081) [tomcat-coyote.jar:8.0.12]\r\n at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:658) [tomcat-coyote.jar:8.0.12]\r\n at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:222) [tomcat-coyote.jar:8.0.12]\r\n at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1566) [tomcat-coyote.jar:8.0.12]\r\n at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1523) [tomcat-coyote.jar:8.0.12]\r\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_71]\r\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_71]\r\n at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.0.12]\r\n at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]\r\n```\r\n\r\nAs per my analysis, following method in com.hazelcast.client.spi.impl.ClientSmartInvocationServiceImpl.java checks for owner connections but **ownerConnectionAddress** is always returned as NULL by **clientClusterService.getOwnerConnectionAddress();** method, causing almost infinite checks for valid connections.\r\n\r\n    private void ensureOwnerConnectionAvailable() throws IOException {\r\n        ClientClusterService clientClusterService = client.getClientClusterService();\r\n        Address ownerConnectionAddress = clientClusterService.getOwnerConnectionAddress();\r\n\r\n        boolean isOwnerConnectionAvailable = ownerConnectionAddress != null\r\n                && connectionManager.getConnection(ownerConnectionAddress) != null;\r\n\r\n        if (!isOwnerConnectionAvailable) {\r\n            if (isShutdown()) {\r\n                throw new HazelcastException(\"ConnectionManager is not active!\");\r\n            }\r\n            throw new IOException(\"Not able to setup owner connection!\");\r\n        }\r\n    }\r\n\r\nIf you check the following method in com.hazelcast.client.spi.impl.ClusterListenerSupport.java, **ownerConnectionAddress** is set to NULL in the first line but is never set to a proper value again. This causes getOwnerConnectionAddress() to always return NULL when called from **ensureOwnerConnectionAvailable()**\r\n\r\n    private void connectToOne() throws Exception {\r\n        ownerConnectionAddress = null;\r\n        final ClientNetworkConfig networkConfig = client.getClientConfig().getNetworkConfig();\r\n        final int connAttemptLimit = networkConfig.getConnectionAttemptLimit();\r\n        final int connectionAttemptPeriod = networkConfig.getConnectionAttemptPeriod();\r\n\r\n        final int connectionAttemptLimit = connAttemptLimit == 0 ? Integer.MAX_VALUE : connAttemptLimit;\r\n\r\n        int attempt = 0;\r\n        Set<InetSocketAddress> triedAddresses = new HashSet<InetSocketAddress>();\r\n        while (attempt < connectionAttemptLimit) {\r\n            if (!client.getLifecycleService().isRunning()) {\r\n                if (LOGGER.isFinestEnabled()) {\r\n                    LOGGER.finest(\"Giving up on retrying to connect to cluster since client is shutdown\");\r\n                }\r\n                break;\r\n            }\r\n            attempt++;\r\n            final long nextTry = Clock.currentTimeMillis() + connectionAttemptPeriod;\r\n\r\n            boolean isConnected = connect(triedAddresses);\r\n\r\n            if (isConnected) {\r\n                return;\r\n            }\r\n\r\n            final long remainingTime = nextTry - Clock.currentTimeMillis();\r\n            LOGGER.warning(\r\n                    String.format(\"Unable to get alive cluster connection, try in %d ms later, attempt %d of %d.\",\r\n                            Math.max(0, remainingTime), attempt, connectionAttemptLimit));\r\n\r\n            if (remainingTime > 0) {\r\n                try {\r\n                    Thread.sleep(remainingTime);\r\n                } catch (InterruptedException e) {\r\n                    break;\r\n                }\r\n            }\r\n        }\r\n        throw new IllegalStateException(\"Unable to connect to any address in the config! \"\r\n                + \"The following addresses were tried:\" + triedAddresses);\r\n    }\r\n\r\n\r\n"}}