{"action":"opened","issue":{"url":"https://api.github.com/repos/rsyslog/librelp/issues/10","labels_url":"https://api.github.com/repos/rsyslog/librelp/issues/10/labels{/name}","comments_url":"https://api.github.com/repos/rsyslog/librelp/issues/10/comments","events_url":"https://api.github.com/repos/rsyslog/librelp/issues/10/events","html_url":"https://github.com/rsyslog/librelp/issues/10","id":55520221,"number":10,"title":"Infinite loop when the tcp send buffer fills","user":{"login":"dpkirchner","id":165134,"avatar_url":"https://avatars.githubusercontent.com/u/165134?v=3","gravatar_id":"","url":"https://api.github.com/users/dpkirchner","html_url":"https://github.com/dpkirchner","followers_url":"https://api.github.com/users/dpkirchner/followers","following_url":"https://api.github.com/users/dpkirchner/following{/other_user}","gists_url":"https://api.github.com/users/dpkirchner/gists{/gist_id}","starred_url":"https://api.github.com/users/dpkirchner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dpkirchner/subscriptions","organizations_url":"https://api.github.com/users/dpkirchner/orgs","repos_url":"https://api.github.com/users/dpkirchner/repos","events_url":"https://api.github.com/users/dpkirchner/events{/privacy}","received_events_url":"https://api.github.com/users/dpkirchner/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-01-26T18:42:00Z","updated_at":"2015-01-26T18:42:00Z","closed_at":null,"body":"I believe this is the cause of the problem reported here:\r\n\r\nhttp://lists.adiscon.net/pipermail/rsyslog/2013-November/034869.html\r\n\r\nwhere rsyslogd, acting as a client, tries to write a log message to the server and another rsyslogd, as a server, tries to write an acknowledgement to a client and fails. Both sides can end up with a full write buffer and thus end up deadlocked.\r\n\r\nI think the problem starts in `relpSendqSend`:\r\n\r\nhttps://github.com/rsyslog/librelp/blob/master/src/sendq.c#L247\r\n\r\nAn infinite loop occurs when `localRet` is `RELP_RET_PARTIAL_WRITE`. As the name suggests that will be the case if `relpSendbufSend`, via `relpTcpSend`, does not write as many bytes as requested. In this case, the `send` call is writing zero bytes, so `relpSendbufSend` returns that partial constant and the loop starts over having done effectively nothing.\r\n\r\nIn `relpTcpSend` (https://github.com/rsyslog/librelp/blob/master/src/tcp.c#L1547) the a `send` call that can fail due to having no more buffer space to write data to the socket. I am not sure what leads to this scenario. I suspect that it's so easy to reproduce once the server first runs out of buffer space because the client has a large queue of messages to send to the server due to blocking that occurred in the past.\r\n\r\nIn any case, the relp thread blocks forever, or at least as long as the TCP connection remains alive (which can be forever). I figure a fix might be to have the server close the socket if the server fails to send any data for N seconds."}}