WSO2 API Manager Store - Access token key generation failure
<p>I have setup an API Manager 1.10 cluster, with each component in a separate VM, fronted by an nginx proxy.</p>

<p>Whenever I try to generate a key I get the following error in store (logs from store):</p>

<pre><code>ERROR - AMDefaultKeyManagerImpl Error while creating tokens - Connection timed out
java.net.ConnectException: Connection timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
   ...
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1739)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1698)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

[2016-12-14 16:25:03,376] ERROR - APIConsumerImpl Error while re-generating AccessToken
org.wso2.carbon.apimgt.api.APIManagementException: Error while creating tokens - Connection timed out
    at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.handleException(AMDefaultKeyManagerImpl.java:592)
    at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.getNewApplicationAccessToken(AMDefaultKeyManagerImpl.java:369)
    at org.wso2.carbon.apimgt.impl.APIConsumerImpl.renewAccessToken(APIConsumerImpl.java:659)
    at org.wso2.carbon.apimgt.impl.UserAwareAPIConsumer.renewAccessToken(UserAwareAPIConsumer.java:36)
    at org.wso2.carbon.apimgt.hostobjects.APIStoreHostObject.jsFunction_refreshToken(APIStoreHostObject.java:3800)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    .....
    at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
    at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1739)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1698)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
    at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:522)
    at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:401)
    at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:178)
    at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
    at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131)
    at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
    at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
    at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.getNewApplicationAccessToken(AMDefaultKeyManagerImpl.java:303)
    ... 70 more
[2016-12-14 16:25:03,377] ERROR - APIStoreHostObject Error while renewing Access Token for Consumer Key qZf8VvT4ts7a0TT6nE6ZHfY9t5sa and user admin
[2016-12-14 16:25:03,377] ERROR - subscription-add:jag org.wso2.carbon.apimgt.api.APIManagementException: Error while renewing Access Token for Consumer Key 88 and user admin
</code></pre>

<p>I have configured connections among the components by following this <a href="https://docs.wso2.com/display/CLUSTER44x/Clustering+API+Manager+1.10.0#ClusteringAPIManager1.10.0-Configuringtheconnectionsamongthecomponents" rel="noreferrer">article</a> The logs above are only showing up in the API store console. </p>

<p>I have also configured the Keymanager host and port in API Gateway in the following locations:</p>

<pre><code>/repository/deployment/server/synapse-configs/default/api
AuthorizeAPI.xml
RevokeAPI.xml
TokenAPI.xml
</code></pre>

<p>What do I need to do to fix this?</p>