Surviving icinga2 restart in a python requests stream
<p>I have been working on a chatbot interface to icinga2, and have not found a persistent way to survive the restart/reload of the icinga2 server. After a week of moving try/except blocks, using requests sessions, et al, it's time to reach out to the community.</p>

<p>Here is the current iteration of the request function:</p>

<pre><code>    def i2api_request(url, headers={}, data={}, stream=False, *, auth=api_auth, ca=api_ca):
    ''' Do not call this function directly; it's a helper for the i2* command functions '''
# Adapted from http://docs.icinga.org/icinga2/latest/doc/module/icinga2/chapter/icinga2-api
# Section 11.10.3.1

    try:
        r = requests.post(url,
            headers=headers,
            auth=auth,
            data=json.dumps(data),
            verify=ca,
            stream=stream
            )
    except (requests.exceptions.ChunkedEncodingError,requests.packages.urllib3.exceptions.ProtocolError, http.client.IncompleteRead,ValueError) as drop:
        return("No connection to Icinga API")

    if r.status_code == 200:
        for line in r.iter_lines():
            try:
                if stream == True:
                    yield(json.loads(line.decode('utf-8')))
                else:
                    return(json.loads(line.decode('utf-8')))
            except:
                debug("Could not produce JSON from "+line)
                continue
    else:
        #r.raise_for_status()
        debug('Received a bad response from Icinga API: '+str(r.status_code))
        print('Icinga2 API connection lost.')
</code></pre>

<p>(The debug function just flags and prints the indicated error to the console.)</p>

<p>This code works fine handling events from the API and sending them to the chatbot, but if the icinga server is reloaded, as would be needed after adding a new server definition in /etc/icinga2..., the listener crashes.</p>

<p>Here is the error response I get when the server is restarted:</p>

<pre><code>    Exception in thread Thread-11:
Traceback (most recent call last):
  File "/home/errbot/err3/lib/python3.4/site-packages/requests/packages/urllib3/response.py", line 447, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/errbot/err3/lib/python3.4/site-packages/requests/packages/urllib3/response.py", line 228, in _error_catcher
    yield
  File "/home/errbot/err3/lib/python3.4/site-packages/requests/packages/urllib3/response.py", line 498, in read_chunked
    self._update_chunk_length()
  File "/home/errbot/err3/lib/python3.4/site-packages/requests/packages/urllib3/response.py", line 451, in _update_chunk_length
    raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/errbot/err3/lib/python3.4/site-packages/requests/models.py", line 664, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/home/errbot/err3/lib/python3.4/site-packages/requests/packages/urllib3/response.py", line 349, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/home/errbot/err3/lib/python3.4/site-packages/requests/packages/urllib3/response.py", line 526, in read_chunked
    self._original_response.close()
  File "/usr/lib64/python3.4/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/errbot/err3/lib/python3.4/site-packages/requests/packages/urllib3/response.py", line 246, in _error_catcher
    raise ProtocolError('Connection broken: %r' % e, e)
requests.packages.urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.4/threading.py", line 920, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.4/threading.py", line 868, in run
    self._target(*self._args, **self._kwargs)
  File "/home/errbot/plugins/icinga2bot.py", line 186, in report_events
    for line in queue:
  File "/home/errbot/plugins/icinga2bot.py", line 158, in i2events
    for line in queue:
  File "/home/errbot/plugins/icinga2bot.py", line 98, in i2api_request
    for line in r.iter_lines():
  File "/home/errbot/err3/lib/python3.4/site-packages/requests/models.py", line 706, in iter_lines
    for chunk in self.iter_content(chunk_size=chunk_size, decode_unicode=decode_unicode):
  File "/home/errbot/err3/lib/python3.4/site-packages/requests/models.py", line 667, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
</code></pre>

<p>With Icinga2.4, this crash happened every time the server was restarted. I thought the problem had gone away after we upgraded to 2.5, but it now appears to have turned into a heisenbug. </p>