Optimization of date subtraction on large dataframe - Pandas
<p>I'm a beginner learning Python. I have a very large dataset - I'm having trouble optimizing my code to make this run faster. </p>

<p>My goal is to optimize all of this (my current code works but slow):</p>

<ol>
<li><p>Subtract two date columns</p></li>
<li><p>Create new column with the result of that subtraction</p></li>
<li><p>Remove original two columns</p></li>
<li><p>Do all of this in a fast manner</p></li>
</ol>

<p>Random finds:
Thinking about changing the initial file read method...
<a href="https://softwarerecs.stackexchange.com/questions/7463/fastest-python-library-to-read-a-csv-file">https://softwarerecs.stackexchange.com/questions/7463/fastest-python-library-to-read-a-csv-file</a></p>

<p>I have parse_dates=True when reading the CSV file - so could this be a slowdown? I have 50+ columns but only 1 timestamp column and 1 year column. </p>

<p>This column:</p>

<pre><code>          saledate
1          3/26/2004 0:00
2          2/26/2004 0:00
3          5/19/2011 0:00
4          7/23/2009 0:00
5         12/18/2008 0:00
</code></pre>

<p>Subtracted by (Should this be converted to a format like 1/1/1996?):</p>

<pre><code>          YearMade
1         1996
2         2001
3         2001
4         2007
5         2004
</code></pre>

<p>Current code:</p>

<pre><code>mean_YearMade = dfx[dfx['YearMade'] &gt; 1000]['YearMade'].mean()

def age_at_sale(df, mean_YearMade):
    '''
    INPUT: Dateframe
    OUTPUT: Dataframe
    Add a column called AgeSale
    '''
    df.loc[:, 'YearMade'][df['YearMade'] == 1000] = mean_YearMade
    # Column has tons of erroneous years with 1000
    df['saledate'] = pd.to_datetime(df['saledate'])
    df['saleyear'] = df['saledate'].dt.year
    df['Age_at_Sale'] = df['saleyear'] - df['YearMade']
    df = df.drop('saledate', axis=1)
    df = df.drop('YearMade', axis=1)
    df = df.drop('saleyear', axis=1)
    return df
</code></pre>

<p>Any optimization tricks would be much appreciated...</p>