Pickling error in passing Pandas DataFrame to method call in multi processing
<p>I was scratching my head over past few days why my method call in multiprocessing was failing, and after drilling down further I realized that it was due to some DataFrames being passed as arguments to method which wasn't getting pickled. So as alternative to that, I parsed the DataFrame to csv in my parent method, passed the name of csv file to child method where I read csv to get the data. But this is pretty inefficient method, so let me know where I did wrong. The following are my codes.</p>

<p>DataFrames: 2 No.</p>

<p>DataFrame: a</p>

<pre><code>FIELD_NAME                    object
DEST_LOC_NBR                 float64
APPT_NBR                     float64
SEQ_NBR                      float64
FIELD_NBR                    float64
CREATE_TS             datetime64[ns]
CREATE_USERID_V               object
BEFORE_FIELD_VALUE            object
AFTER_FIELD_VALUE             object
CHNG_ORIGIN_I                float64
APPT_ STAT_ NBR              float64
DESCRPTN                      object
LOC_NBR_x                    float64
LOCN_ABBR_x                   object
LOC_NBR_y                     object
LOCN_ABBR_y                   object
FRGT_TYP_NBR_x               float64
FRGT_TYP_DESC_x               object
FRGT_TYP_NBR_y                object
FRGT_TYP_DESC_y               object
</code></pre>

<p>DataFrame: d1</p>

<pre><code>APPT_NBR                             float64
APPT_MERGE_F                          object
APPT_STAT_NBR                         object
DEST_LOC_NBR                         float64
APPT_TYP_NBR                         float64
LOC_NBR                              float64
FRGT_TYP_NBR                         float64
APPT_FRT_TYP_AFTER_ARVL_TS            object
APPT_EMPTY_F                          object
APPT_ACTL_ARVL_TS             datetime64[ns]
APPT_SCHD_ARVL_TS             datetime64[ns]
APPT_NBR_OF_CNTRS                    float64
APPT_NBR_OF_GOHS                     float64
CREATE_TS                     datetime64[ns]
LAST_UPD_TS                           object
VERSION_NBR                          float64
APPT_CLOSE_TS                         object
APPT_OPEN_TS                          object
APPT_UNCNFRMD_TS                      object
APPT_ORG_SCH_AR_TS                    object
APPT_CNCL_TS                          object
APPT_SUSP_TS                          object
APPT_ARCHV_TS                         object
CARRIER                               object
VND_NBR                              float64
YARD_AREA_NBR                        float64
CREATE_USERID_V                       object
APPT_ACTL_ARVL_D                      object
APPT_SCHD_ARVL_D                      object
dtype: object
</code></pre>

<p>I tried even manually pickling on parent method and unpickle in child like I do for csv files, but the same error raises.</p>

<p>I first create random file names as this is Multi-threading process, hence file name can't be same:</p>

<pre><code>fname = ''.join(random.SystemRandom().choice(string.ascii_uppercase + string.digits) for _ in range(5))
fnames = dict()
fnames['a'] = '{}_a.pkl'.format(fname)
fnames['d1'] = '{}_d1.pkl'.format(fname)
fnames['file'] = fname
</code></pre>

<p>And, I pickle dataframes using the following:</p>

<pre><code>d1.to_pickle(fnames['d1'])
a.to_pickle(fnames['a'])
</code></pre>

<p>And, the following is my method call:</p>

<pre><code>p = multiprocessing.Process(target=ParallelLoopTest, args=(dd, fnames, return_list, final_col_dates, d, final_col_dates_mod, iter, self.DC, self.start_hour_of_day))
</code></pre>

<p>While my method ParallelLoopTest definition is as follows:</p>

<pre><code>def ParallelLoopTest(dd, fnames, days_out_vars, final_col_dates, d, final_col_dates_mod, iter, DC, start_hour_of_day, store):

    a = pd.read_pickle(fnames['a'])
    d1 = pd.read_pickle(fnames['d1'])

    df = pd.read_pickle('{}_df.pkl'.format(fnames['file']))
</code></pre>

<p>And, I face the following error:</p>

<pre><code>Traceback (most recent call last):
  File "C:\Users\dkanhar\Anaconda3\lib\multiprocessing\process.py", line 249, in _bootstrap
    self.run()
  File "C:\Users\dkanhar\Anaconda3\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "E:\Projects\Predictive Inbound Cartoon Estimation-MLO\Python\dataprep\DataPrep.py", line 497, in ParallelLoopTest
    a = pd.read_pickle(fnames['a'])
  File "C:\Users\dkanhar\Anaconda3\lib\site-packages\pandas\io\pickle.py", line 63, in read_pickle
    return try_read(path, encoding='latin1')
  File "C:\Users\dkanhar\Anaconda3\lib\site-packages\pandas\io\pickle.py", line 57, in try_read
    return pc.load(fh, encoding=encoding, compat=True)
  File "C:\Users\dkanhar\Anaconda3\lib\site-packages\pandas\compat\pickle_compat.py", line 118, in load
    return up.load()
  File "C:\Users\dkanhar\Anaconda3\lib\pickle.py", line 1039, in load
    dispatch[key[0]](self)
  File "C:\Users\dkanhar\Anaconda3\lib\site-packages\pandas\compat\pickle_compat.py", line 73, in load_newobj
    obj = cls.__new__(cls, *args)
TypeError: function takes at most 0 arguments (1 given)
</code></pre>

<p>So, as you can see, Pandas fail to unpickle the DatFrame, no matter what I try. I tried picking manually using <strong>pickly.dump() and pickle.load()</strong> but even that failed with similar error. (<em>TypeError, Function takes 0 arguments (1 given)</em>)</p>

<p>I also feel this is due to DataFrame problem, as I created a random DataFrame using:</p>

<pre><code>df = pd.DataFrame(np.random.randint(0, 100, size=(100000, 4)), columns=list('ABCD'))
</code></pre>

<p>Pickling it, and unpickling on other end, and that worked.</p>

<p>So, what must be issue with DataFrame? Possible scenarios where DataFrame can't be pickled? </p>

<p>Note that its around 200mb pickle file for Dataframe <strong>a</strong> and 25mb for pickled DataFrame <strong>d1</strong>.</p>

<p>Also, is there way to post head of whole DataFrame? It has around 50 columns which wont be printed, and hence cant be added here, but in meanwhile I'm adding dtypes of both DataFrames.</p>

<p>Any help would be really helpful.</p>

<p>PS: The following link is for my previous post which described the error while Passed DataFRame directly to Method call instead of picking and unpickling:
<a href="https://stackoverflow.com/questions/39993436/similar-errors-in-multiprocessing-mismatch-number-of-arguments-to-function?noredirect=1#comment67449576_39993436">Similar errors in MultiProcessing. Mismatch number of arguments to function</a></p>

<p><strong>Update</strong>:
The following is link to Google spreedsheet with head n = 50 for both the DataFrames in concern.</p>

<p><a href="https://docs.google.com/spreadsheets/d/1bGkpmV0__aPVUtc0HSeuRQpufu1T4pmKB4YxsQsPO50/edit?usp=sharing" rel="nofollow noreferrer">https://docs.google.com/spreadsheets/d/1bGkpmV0__aPVUtc0HSeuRQpufu1T4pmKB4YxsQsPO50/edit?usp=sharing</a></p>