{"action":"created","issue":{"url":"https://api.github.com/repos/JioCloud/puppet-rjil/issues/517","labels_url":"https://api.github.com/repos/JioCloud/puppet-rjil/issues/517/labels{/name}","comments_url":"https://api.github.com/repos/JioCloud/puppet-rjil/issues/517/comments","events_url":"https://api.github.com/repos/JioCloud/puppet-rjil/issues/517/events","html_url":"https://github.com/JioCloud/puppet-rjil/issues/517","id":66135491,"number":517,"title":"rpc timeout issues on various openstack components ","user":{"login":"hkumarmk","id":6316902,"avatar_url":"https://avatars.githubusercontent.com/u/6316902?v=3","gravatar_id":"","url":"https://api.github.com/users/hkumarmk","html_url":"https://github.com/hkumarmk","followers_url":"https://api.github.com/users/hkumarmk/followers","following_url":"https://api.github.com/users/hkumarmk/following{/other_user}","gists_url":"https://api.github.com/users/hkumarmk/gists{/gist_id}","starred_url":"https://api.github.com/users/hkumarmk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hkumarmk/subscriptions","organizations_url":"https://api.github.com/users/hkumarmk/orgs","repos_url":"https://api.github.com/users/hkumarmk/repos","events_url":"https://api.github.com/users/hkumarmk/events{/privacy}","received_events_url":"https://api.github.com/users/hkumarmk/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-04-03T12:17:09Z","updated_at":"2015-04-03T14:34:31Z","closed_at":null,"body":"I am not sure if this is deadly or just kind of errors, but sometimes we see missing messages and other problems and we see these messages that time also.\r\n\r\nWe (Soren and me) did some detailed investigation previously, but in the middle of the investigation, the issue jst fixed automatically (within few mins ).\r\n\r\nHEre is the error we see on other component's logs also\r\n\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler     instance_id, address)\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler   File \"/usr/lib/python2.7/dist-packages/nova/api/metadata/base.py\", line 509, in get_metadata_by_instance_id\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler     return InstanceMetadata(instance, address)\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler   File \"/usr/lib/python2.7/dist-packages/nova/api/metadata/base.py\", line 148, in __init__\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler     obj_base.obj_to_primitive(instance))\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler   File \"/usr/lib/python2.7/dist-packages/nova/conductor/api.py\", line 274, in get_ec2_ids\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler     return self._manager.get_ec2_ids(context, instance)\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler   File \"/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py\", line 413, in get_ec2_ids\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler     instance=instance_p)\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler   File \"/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py\", line 150, in call\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler     wait_for_reply=True, timeout=timeout)\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler   File \"/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py\", line 90, in _send\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler     timeout=timeout)\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler   File \"/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_zmq.py\", line 927, in send\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler     return self._send(target, ctxt, message, wait_for_reply, timeout)\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler   File \"/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_zmq.py\", line 921, in _send\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler     allowed_remote_exmods=self._allowed_remote_exmods)\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler   File \"/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_zmq.py\", line 756, in _multi_send\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler     envelope, allowed_remote_exmods)\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler   File \"/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_zmq.py\", line 691, in _call\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler     msg = msg_waiter.recv()\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler   File \"/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_zmq.py\", line 212, in recv\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler     return self.sock.recv_multipart(**kwargs)\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler   File \"/usr/lib/python2.7/dist-packages/eventlet/green/zmq.py\", line 345, in recv_multipart\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler     return _Socket_recv_multipart(self, flags, copy, track)\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler   File \"/usr/lib/python2.7/dist-packages/zmq/sugar/socket.py\", line 265, in recv_multipart\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler     parts = [self.recv(flags, copy=copy, track=track)]\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler   File \"/usr/lib/python2.7/dist-packages/eventlet/green/zmq.py\", line 324, in recv\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler     self._eventlet_recv_event.block()\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler   File \"/usr/lib/python2.7/dist-packages/eventlet/green/zmq.py\", line 97, in block\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler     self._hub.switch()\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler   File \"/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py\", line 187, in switch\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler     return self.greenlet.switch()\r\n2015-04-03 12:12:21.327 17876 TRACE nova.api.metadata.handler Timeout: Timeout while waiting on RPC response - topic: \"<unknown>\", RPC method: \"<unknown>\" info: \"<unknown>\"\r\n"},"comment":{"url":"https://api.github.com/repos/JioCloud/puppet-rjil/issues/comments/89305500","html_url":"https://github.com/JioCloud/puppet-rjil/issues/517#issuecomment-89305500","issue_url":"https://api.github.com/repos/JioCloud/puppet-rjil/issues/517","id":89305500,"user":{"login":"hkumarmk","id":6316902,"avatar_url":"https://avatars.githubusercontent.com/u/6316902?v=3","gravatar_id":"","url":"https://api.github.com/users/hkumarmk","html_url":"https://github.com/hkumarmk","followers_url":"https://api.github.com/users/hkumarmk/followers","following_url":"https://api.github.com/users/hkumarmk/following{/other_user}","gists_url":"https://api.github.com/users/hkumarmk/gists{/gist_id}","starred_url":"https://api.github.com/users/hkumarmk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hkumarmk/subscriptions","organizations_url":"https://api.github.com/users/hkumarmk/orgs","repos_url":"https://api.github.com/users/hkumarmk/repos","events_url":"https://api.github.com/users/hkumarmk/events{/privacy}","received_events_url":"https://api.github.com/users/hkumarmk/received_events","type":"User","site_admin":false},"created_at":"2015-04-03T14:34:31Z","updated_at":"2015-04-03T14:34:31Z","body":"Oh yeah, I tried to reduce the threads for nova compute, metadata, ec2, conductor from 32 each to 4 each to see if large number of threads accessing the rpc is causing this issue, but it did not fix it."}}