Pandas DataFrame with date and time from JSON format
<p>I'm importing data from <code>.json</code> file with pandas <code>DataFrame</code> and the result is a bit broken: </p>

<pre><code>              &gt;&gt; print df
              summary                                response_date
                  8.0  {u'$date': u'2009-02-19T10:54:00.000+0000'}
                 11.0  {u'$date': u'2009-02-24T11:23:45.000+0000'}
                 14.0  {u'$date': u'2009-03-03T17:55:07.000+0000'}
                 16.0  {u'$date': u'2009-03-10T12:23:04.000+0000'}
                 19.0  {u'$date': u'2009-03-17T17:19:55.000+0000'}
                 13.0  {u'$date': u'2009-03-25T15:10:52.000+0000'}
                 22.0  {u'$date': u'2009-04-02T16:57:31.000+0100'}
                 15.0  {u'$date': u'2009-04-08T22:29:09.000+0100'}
                 20.0  {u'$date': u'2009-04-16T18:14:20.000+0100'}
                 13.0  {u'$date': u'2009-04-29T10:47:06.000+0100'}
                 15.0  {u'$date': u'2009-05-06T13:45:45.000+0100'}
                 20.0  {u'$date': u'2009-05-26T10:41:52.000+0100'}
</code></pre>

<p>How to get rid of 'date' and other mess to create a normal column with date and time. To convert from ISO8601 format I normally use:</p>

<pre><code>df.response_date = pd.to_datetime(df.response_date)
</code></pre>

<p><strong>UPDATE 1</strong></p>

<pre><code>       summary                 response_date                                  closed_date                                    open_date
          24.0  2011-10-15T00:00:00.000+0100                                          NaN                                          NaN
          24.0  2011-11-24T09:00:00.000+0000                                          NaN                                          NaN
          19.0  2011-10-01T09:00:00.000+0100                                          NaN                                          NaN
          25.0  2011-10-29T09:00:00.000+0100                                          NaN                                          NaN
          19.0  2011-10-08T09:00:00.000+0100                                          NaN                                          NaN
          -1.0  2011-11-09T17:20:00.000+0000  {u'$date': u'2011-11-16T15:20:00.000+0000'}  {u'$date': u'2011-11-09T15:20:00.000+0000'}
          -1.0  2011-11-16T17:20:00.000+0000  {u'$date': u'2011-11-23T15:20:00.000+0000'}  {u'$date': u'2011-11-16T15:20:00.000+0000'}
          -1.0  2011-11-23T17:20:00.000+0000  {u'$date': u'2011-11-30T15:20:00.000+0000'}  {u'$date': u'2011-11-23T15:20:00.000+0000'}
          -1.0  2011-11-30T17:20:00.000+0000  {u'$date': u'2011-12-07T15:20:00.000+0000'}  {u'$date': u'2011-11-30T15:20:00.000+0000'}
</code></pre>

<p>So, the </p>

<pre><code>&gt;&gt; df.response_date = pd.DataFrame(df.response_date.values.tolist())
</code></pre>

<p>worked perfectly, but other columns contain NaN values, and imputing with "-1" doesn't help. </p>

<pre><code>&gt;&gt; print type(df.ix[0,'scheduleClosedAt'])
&lt;type 'int'&gt;
</code></pre>

<p><strong>UPDATED 2</strong></p>

<p>Why this (masking) method does not work?</p>

<pre><code>&gt;&gt; df.reset_index(inplace=True)
&gt;&gt; indx_nan_closed = df.closed_date.isnull()
&gt;&gt; df[~indx_nan_closed].closed_date = pd.DataFrame(df[~indx_nan_closed].closed_date.values.tolist())
</code></pre>

<p>This line is equivalent to the one in above, but with masking array, so I want to apply this method only to non-NaN values, but the result is that my data frame "df" remains unchanged. This is quite strange.</p>

<p>Any thoughts?</p>