ThreadPoolExecutor - how to return arguments
<p>I need to parse around 1000 URLs. So far I have a function that returns a pandas dataframe after parsing the URL. How should I best structure the program so I can add all the dataframes together? I'm also unsure how to return arguments into 'futures'. In the below example, how can I eventually merge all temp dataframes into a single dataframe (i.e. finalDF=finalDF.append(temp)</p>

<pre><code>import concurrent.futures

def Parser(ptf):
    temp=pd.DataFrame()
    URL="http://"+str(URL)
    #..some complex operations, including a requests.get(URL) which returns eventually a temp: a pandas dataframe
    return temp #returns a pandas dataframe

def conc_caller(ptf):
    temp=Parser(ptf)

    #this won't work because finalDF is not defined, unclear how to handle this
    finalDF= finalDF.append(temp)
    return df

booklist=['a','b','c']
finalDF=pd.DataFrame()        
executor = concurrent.futures.ProcessPoolExecutor(3)
futures = [executor.submit(conc_caller, item) for item in booklist]
concurrent.futures.wait(futures)
</code></pre>

<p>Another problem is that I get the error message:</p>

<pre><code> An attempt has been made to start a new process before the
 current process has finished its bootstrapping phase.
</code></pre>

<p>Any suggestion how to fix the code are appreciated.</p>