<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Recap
In order of priority/time taken
basalareaincremementnonspatialaw
this is actually slow because of the number of times the BAFromZeroToDataAw function is called as shown above
relaxing the tolerance may help
indeed the tolerance is 0.01 * some value while the other factor finder functions have 0.1 tolerance i think
can also use cython for the increment functions
vectorize merch and gross volume functions
they require a lot of getting scalars off data frame, which is quite slow. faster to get an array
do a profiling run with IO (of reading input data and writing the plot curves to files) in next run
Decide on the action
speed up increment functions
use cython for increment functions
it turns out this may not help that much. the function is pretty fast, it's called almost 500,000 times on the sample of 300 plots
reduce the number of times its called maybe by using gradient descent for the optimization?
relax the tolerance
gives a context to refactor them as well (into their own module) which would be a welcome change
the increment functions use numpy functions but operate on scalars, there is no benefit to using numpy functions there
performance-wise, it is not clear that this will pay off so much. vectorizing the volume functions is probably wiser
Characterize what is happening
Step1:  The original gross volume function checks that top height is greater than 0
``` python
def GrossTotalVolume_Pl(BA_Pl, topHeight_Pl)
Step2:  MWEs
If we can rewrite it to handle 0s properly, i.e. to return 0 where an input is 0, then it is trivial to vectorize
Step3:  Timings
Step4:  The array method is 20x faster. This is worth implementing. We should also add tests to help be explicity about the behaviour of these volume functions.
Revise the code
Go on. Do it.
Tests
Yes, though data was changed as the new implementations yield NaN where input is NaN, instead of yielding 0.
Review code changes
Step5:  Run timings
From last time
Step6:  It yielded a 13% reduction in the time.
Run profiling
Step7:  Compare performance visualizations
Now use either of these commands to visualize the profiling
```
pyprof2calltree -k -i forward-sim-1.prof forward-sim-3.txt
or
dc run --service-ports snakeviz notebooks/forward-sim-3.prof
```
Old
New
Summary of performance improvements
The calculation of gross and merchantable volume is drastically faster now; under profiling it decrease to 1 second from 22 seconds.
A lot of that seems to be profiler overhead, as when using gypsy simulate CLI it only got 15% faster; however I expect i/o is obfuscating the outcome there.
Profile with I/O
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import numpy as np
Explanation: Recap
In order of priority/time taken
basalareaincremementnonspatialaw
this is actually slow because of the number of times the BAFromZeroToDataAw function is called as shown above
relaxing the tolerance may help
indeed the tolerance is 0.01 * some value while the other factor finder functions have 0.1 tolerance i think
can also use cython for the increment functions
vectorize merch and gross volume functions
they require a lot of getting scalars off data frame, which is quite slow. faster to get an array
do a profiling run with IO (of reading input data and writing the plot curves to files) in next run
Decide on the action
speed up increment functions
use cython for increment functions
it turns out this may not help that much. the function is pretty fast, it's called almost 500,000 times on the sample of 300 plots
reduce the number of times its called maybe by using gradient descent for the optimization?
relax the tolerance
gives a context to refactor them as well (into their own module) which would be a welcome change
the increment functions use numpy functions but operate on scalars, there is no benefit to using numpy functions there
performance-wise, it is not clear that this will pay off so much. vectorizing the volume functions is probably wiser
Characterize what is happening
End of explanation
from gypsy.GYPSYNonSpatial import GrossTotalVolume_Pl
GrossTotalVolume_Pl(np.random.random(10) * 100, np.random.random(10) * 100)
Explanation: The original gross volume function checks that top height is greater than 0
``` python
def GrossTotalVolume_Pl(BA_Pl, topHeight_Pl):
    Tvol_Pl = 0
if topHeight_Pl &gt; 0:
    a1 = 0.194086
    a2 = 0.988276
    a3 = 0.949346
    a4 = -3.39036
    Tvol_Pl = a1* (BA_Pl**a2) * (topHeight_Pl **a3) * numpy.exp(1+(a4/((topHeight_Pl**2)+1)))
return Tvol_Pl
```
This makes it fail if trying to use it on an array:
End of explanation
def GrossTotalVolume_Pl_arr(BA_Pl, topHeight_Pl):
    a1 = 0.194086
    a2 = 0.988276
    a3 = 0.949346
    a4 = -3.39036
    Tvol_Pl = a1* (BA_Pl**a2) * (topHeight_Pl **a3) * np.exp(1+(a4/((topHeight_Pl**2)+1)))
    return Tvol_Pl
print(GrossTotalVolume_Pl_arr(10, 10))
print(GrossTotalVolume_Pl_arr(0, 10))
print(GrossTotalVolume_Pl_arr(10, 0))
print(GrossTotalVolume_Pl_arr(np.random.random(10) * 100, np.random.random(10) * 100))
print(GrossTotalVolume_Pl_arr(np.zeros(10) * 100, np.random.random(10) * 100))
Explanation: MWEs
If we can rewrite it to handle 0s properly, i.e. to return 0 where an input is 0, then it is trivial to vectorize
End of explanation
ba = np.random.random(1000) * 100
top_height = np.random.random(1000) * 100
d = pd.DataFrame({'ba': ba, 'th': top_height})
%%timeit
d.apply(
    lambda x: GrossTotalVolume_Pl(
        x.at['ba'],
        x.at['th']
    ),
    axis=1
)
%%timeit
GrossTotalVolume_Pl_arr(ba, top_height)
Explanation: Timings
End of explanation
%%bash
git log --since "2016-11-14 19:30" --oneline # 19:30 GMT/UTC
! git diff "HEAD~$(git log --since "2016-11-14 19:30" --oneline | wc -l)" ../gypsy
Explanation: The array method is 20x faster. This is worth implementing. We should also add tests to help be explicity about the behaviour of these volume functions.
Revise the code
Go on. Do it.
Tests
Yes, though data was changed as the new implementations yield NaN where input is NaN, instead of yielding 0.
Review code changes
End of explanation
%%bash
# git checkout 36941343aca2df763f93192abef461093918fff4 -b vectorize-volume-functions
# time gypsy simulate ../private-data/prepped_random_sample_300.csv --output-dir tmp
# rm -rfd tmp
# real	4m51.287s
# user	4m41.770s
# sys	0m1.070s
45/336.
Explanation: Run timings
From last time:
real    5m36.407s
user    5m25.740s
sys     0m2.140s
After cython'ing iter functions:
End of explanation
from gypsy.forward_simulation import simulate_forwards_df
data = pd.read_csv('../private-data/prepped_random_sample_300.csv', index_col=0, nrows=10)
%%prun -D forward-sim-3.prof -T forward-sim-3.txt -q
result = simulate_forwards_df(data)
!head forward-sim-3.txt
Explanation: It yielded a 13% reduction in the time.
Run profiling
End of explanation
! rm -rfd gypsy-output
output_dir = 'gypsy-output'
%%prun -D forward-sim-2.prof -T forward-sim-2.txt -q
# restart the kernel first
data = pd.read_csv('../private-data/prepped_random_sample_300.csv', index_col=0, nrows=10)
result = simulate_forwards_df(data)
os.makedirs(output_dir)
for plot_id, df in result.items():
    filename = '%s.csv' % plot_id
    output_path = os.path.join(output_dir, filename)
    df.to_csv(output_path)
Explanation: Compare performance visualizations
Now use either of these commands to visualize the profiling
```
pyprof2calltree -k -i forward-sim-1.prof forward-sim-3.txt
or
dc run --service-ports snakeviz notebooks/forward-sim-3.prof
```
Old
New
Summary of performance improvements
The calculation of gross and merchantable volume is drastically faster now; under profiling it decrease to 1 second from 22 seconds.
A lot of that seems to be profiler overhead, as when using gypsy simulate CLI it only got 15% faster; however I expect i/o is obfuscating the outcome there.
Profile with I/O
End of explanation
<END_TASK>