Py 2.7 arch: How to persistent HTTP/S with multiple servers and not gather data to send multiple times?
<p>This is kind of a complicated question, so I'll do my best to be plain in my explanation and not give too many unnecessary details.</p>

<p>I developed a python script for work last year. It grabs basic system data and sends it to an HTTP/S server, which can send commands back if a users so chooses. It's kind of been a big experiment for the last year, seeing what works and what doesn't. Testing different needs within the company, etc. But now I have a pretty solid understanding of what we need. So I am starting my journey on version 2.</p>

<p>The aim of this new version is to maintain functionality while reducing system/CPU load and bandwidth. After developing this Python script out, the rest of the work will be done on the HTTP/S server. My question is specifically about the Client side, the Python script. I am using Python 2.7.x, most often on Debian based systems.</p>

<p>The v1 script grabs system data, reads a config file that contains servers to send the data to, uses threads to send to each server. (Still in those threads) each server can return 1 or more commands, which are then also processed through their own threads. The script is run once a minute via crontab. You can have 5 or more servers send 10 commands each and the script still executes everything smoothly, effectively and without taking a long time to finish commands issued by the servers.</p>

<p>In the v2 script, I am seeking to make the following required changes:</p>

<ul>
<li><p>Will be run as a system service. So instead of the code being run by cron every minute, the script will loop every few seconds.</p></li>
<li><p>The loop needs to gather data once each time through the loop, then send it to each web server (As defined in the configuration file)</p></li>
<li><p>I want persistent HTTP/S connections for performance and bandwidth optimization.</p></li>
<li><p>I don't want to gather data each time through the loop for each HTTP/S server. I want to only gather data once per iteration through the main loop which drives the service, and then send that data to the threads that are managing the established HTTP/S persistent connections.</p></li>
</ul>

<p>Here in lies my problem. How do I get persistent connections inside in their respective threads AND get data to those threads while only collecting the data once?</p>

<p>From <a href="https://stackoverflow.com/questions/21054316/does-httplib-reuse-tcp-connections">does httplib reuse TCP connections?</a> I see that persistent connections can be done in such a manner (Thank you <a href="https://stackoverflow.com/users/16148/corey-goldberg">Corey Goldberg</a>):</p>

<pre><code>con = httplib.HTTPConnection("myweb.com")
while True:
    con.request("GET", "/x.css", headers={"Connection":" keep-alive"})
    result = con.getresponse()
    result.read()
    print result.reason, result.getheaders()
</code></pre>

<p>Data gathering needs to happen inside this loop. But I need this to happen in multiple threads talking to various servers at the same time, and don't want to waste the resources to go and fetch the data more than once. I just don't see how it is possible, given my relatively limited knowledge of Python.</p>

<p>Basically as I see it right now, there needs to be a loop that drives the HTTP/S inside of their threads. Then I need some kind of loop to gather my data and prepare it to go to the HTTP/S connections. But how do I get the first loops inside of the second loops in such a way? It's like I need the HTTP/S persistent connection loop inside the data gathering loop, but I also need the data gathering loop inside the HTTP/S loop.</p>

<p>I would like to explore any pure 2.7.x pythonic ways this might be accomplished. Depending on outside utilities may be problematic for various reasons. This script, when finished, will be deployed to 150+ linux systems and the less that can go wrong, the better.</p>

<p>Thank you for your help and consideration!</p>