<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <!--BOOK_INFORMATION-->
<img align="left" style="padding-right
Step1:  As discussed in Computation on NumPy Arrays
Step2:  But this abstraction can become less efficient when computing compound expressions.
For example, consider the following expression
Step3:  Because NumPy evaluates each subexpression, this is roughly equivalent to the following
Step4:  In other words, every intermediate step is explicitly allocated in memory. If the x and y arrays are very large, this can lead to significant memory and computational overhead.
The Numexpr library gives you the ability to compute this type of compound expression element by element, without the need to allocate full intermediate arrays.
The Numexpr documentation has more details, but for the time being it is sufficient to say that the library accepts a string giving the NumPy-style expression you'd like to compute
Step5:  The benefit here is that Numexpr evaluates the expression in a way that does not use full-sized temporary arrays, and thus can be much more efficient than NumPy, especially for large arrays.
The Pandas eval() and query() tools that we will discuss here are conceptually similar, and depend on the Numexpr package.
pandas.eval() for Efficient Operations
The eval() function in Pandas uses string expressions to efficiently compute operations using DataFrames.
For example, consider the following DataFrames
Step6:  To compute the sum of all four DataFrames using the typical Pandas approach, we can just write the sum
Step7:  The same result can be computed via pd.eval by constructing the expression as a string
Step8:  The eval() version of this expression is about 50% faster (and uses much less memory), while giving the same result
Step9:  Operations supported by pd.eval()
As of Pandas v0.16, pd.eval() supports a wide range of operations.
To demonstrate these, we'll use the following integer DataFrames
Step10:  Arithmetic operators
pd.eval() supports all arithmetic operators. For example
Step11:  Comparison operators
pd.eval() supports all comparison operators, including chained expressions
Step12:  Bitwise operators
pd.eval() supports the &amp; and | bitwise operators
Step13:  In addition, it supports the use of the literal and and or in Boolean expressions
Step14:  Object attributes and indices
pd.eval() supports access to object attributes via the obj.attr syntax, and indexes via the obj[index] syntax
Step15:  Other operations
Other operations such as function calls, conditional statements, loops, and other more involved constructs are currently not implemented in pd.eval().
If you'd like to execute these more complicated types of expressions, you can use the Numexpr library itself.
DataFrame.eval() for Column-Wise Operations
Just as Pandas has a top-level pd.eval() function, DataFrames have an eval() method that works in similar ways.
The benefit of the eval() method is that columns can be referred to by name.
We'll use this labeled array as an example
Step16:  Using pd.eval() as above, we can compute expressions with the three columns like this
Step17:  The DataFrame.eval() method allows much more succinct evaluation of expressions with the columns
Step18:  Notice here that we treat column names as variables within the evaluated expression, and the result is what we would wish.
Assignment in DataFrame.eval()
In addition to the options just discussed, DataFrame.eval()  also allows assignment to any column.
Let's use the DataFrame from before, which has columns 'A', 'B', and 'C'
Step19:  We can use df.eval() to create a new column 'D' and assign to it a value computed from the other columns
Step20:  In the same way, any existing column can be modified
Step21:  Local variables in DataFrame.eval()
The DataFrame.eval() method supports an additional syntax that lets it work with local Python variables.
Consider the following
Step22:  The @ character here marks a variable name rather than a column name, and lets you efficiently evaluate expressions involving the two "namespaces"
Step23:  As with the example used in our discussion of DataFrame.eval(), this is an expression involving columns of the DataFrame.
It cannot be expressed using the DataFrame.eval() syntax, however!
Instead, for this type of filtering operation, you can use the query() method
Step24:  In addition to being a more efficient computation, compared to the masking expression this is much easier to read and understand.
Note that the query() method also accepts the @ flag to mark local variables
Step25:  Performance
Step26:  Is roughly equivalent to this
Step27:  If the size of the temporary DataFrames is significant compared to your available system memory (typically several gigabytes) then it's a good idea to use an eval() or query() expression.
You can check the approximate size of your array in bytes using this
<ASSISTANT_TASK:>
Python Code:
import numpy as np
rng = np.random.RandomState(42)
x = rng.rand(1000000)
y = rng.rand(1000000)
%timeit x + y
Explanation: <!--BOOK_INFORMATION-->
<img align="left" style="padding-right:10px;" src="figures/PDSH-cover-small.png">
This notebook contains an excerpt from the Python Data Science Handbook by Jake VanderPlas; the content is available on GitHub.
The text is released under the CC-BY-NC-ND license, and code is released under the MIT license. If you find this content useful, please consider supporting the work by buying the book!
No changes were made to the contents of this notebook from the original.
<!--NAVIGATION-->
< Working with Time Series | Contents | Further Resources >
High-Performance Pandas: eval() and query()
As we've already seen in previous sections, the power of the PyData stack is built upon the ability of NumPy and Pandas to push basic operations into C via an intuitive syntax: examples are vectorized/broadcasted operations in NumPy, and grouping-type operations in Pandas.
While these abstractions are efficient and effective for many common use cases, they often rely on the creation of temporary intermediate objects, which can cause undue overhead in computational time and memory use.
As of version 0.13 (released January 2014), Pandas includes some experimental tools that allow you to directly access C-speed operations without costly allocation of intermediate arrays.
These are the eval() and query() functions, which rely on the Numexpr package.
In this notebook we will walk through their use and give some rules-of-thumb about when you might think about using them.
Motivating query() and eval(): Compound Expressions
We've seen previously that NumPy and Pandas support fast vectorized operations; for example, when adding the elements of two arrays:
End of explanation
%timeit np.fromiter((xi + yi for xi, yi in zip(x, y)), dtype=x.dtype, count=len(x))
Explanation: As discussed in Computation on NumPy Arrays: Universal Functions, this is much faster than doing the addition via a Python loop or comprehension:
End of explanation
mask = (x > 0.5) & (y < 0.5)
Explanation: But this abstraction can become less efficient when computing compound expressions.
For example, consider the following expression:
End of explanation
tmp1 = (x > 0.5)
tmp2 = (y < 0.5)
mask = tmp1 & tmp2
Explanation: Because NumPy evaluates each subexpression, this is roughly equivalent to the following:
End of explanation
import numexpr
mask_numexpr = numexpr.evaluate('(x > 0.5) & (y < 0.5)')
np.allclose(mask, mask_numexpr)
Explanation: In other words, every intermediate step is explicitly allocated in memory. If the x and y arrays are very large, this can lead to significant memory and computational overhead.
The Numexpr library gives you the ability to compute this type of compound expression element by element, without the need to allocate full intermediate arrays.
The Numexpr documentation has more details, but for the time being it is sufficient to say that the library accepts a string giving the NumPy-style expression you'd like to compute:
End of explanation
import pandas as pd
nrows, ncols = 100000, 100
rng = np.random.RandomState(42)
df1, df2, df3, df4 = (pd.DataFrame(rng.rand(nrows, ncols))
                      for i in range(4))
Explanation: The benefit here is that Numexpr evaluates the expression in a way that does not use full-sized temporary arrays, and thus can be much more efficient than NumPy, especially for large arrays.
The Pandas eval() and query() tools that we will discuss here are conceptually similar, and depend on the Numexpr package.
pandas.eval() for Efficient Operations
The eval() function in Pandas uses string expressions to efficiently compute operations using DataFrames.
For example, consider the following DataFrames:
End of explanation
%timeit df1 + df2 + df3 + df4
Explanation: To compute the sum of all four DataFrames using the typical Pandas approach, we can just write the sum:
End of explanation
%timeit pd.eval('df1 + df2 + df3 + df4')
Explanation: The same result can be computed via pd.eval by constructing the expression as a string:
End of explanation
np.allclose(df1 + df2 + df3 + df4,
            pd.eval('df1 + df2 + df3 + df4'))
Explanation: The eval() version of this expression is about 50% faster (and uses much less memory), while giving the same result:
End of explanation
df1, df2, df3, df4, df5 = (pd.DataFrame(rng.randint(0, 1000, (100, 3)))
                           for i in range(5))
Explanation: Operations supported by pd.eval()
As of Pandas v0.16, pd.eval() supports a wide range of operations.
To demonstrate these, we'll use the following integer DataFrames:
End of explanation
result1 = -df1 * df2 / (df3 + df4) - df5
result2 = pd.eval('-df1 * df2 / (df3 + df4) - df5')
np.allclose(result1, result2)
Explanation: Arithmetic operators
pd.eval() supports all arithmetic operators. For example:
End of explanation
result1 = (df1 < df2) & (df2 <= df3) & (df3 != df4)
result2 = pd.eval('df1 < df2 <= df3 != df4')
np.allclose(result1, result2)
Explanation: Comparison operators
pd.eval() supports all comparison operators, including chained expressions:
End of explanation
result1 = (df1 < 0.5) & (df2 < 0.5) | (df3 < df4)
result2 = pd.eval('(df1 < 0.5) & (df2 < 0.5) | (df3 < df4)')
np.allclose(result1, result2)
Explanation: Bitwise operators
pd.eval() supports the &amp; and | bitwise operators:
End of explanation
result3 = pd.eval('(df1 < 0.5) and (df2 < 0.5) or (df3 < df4)')
np.allclose(result1, result3)
Explanation: In addition, it supports the use of the literal and and or in Boolean expressions:
End of explanation
result1 = df2.T[0] + df3.iloc[1]
result2 = pd.eval('df2.T[0] + df3.iloc[1]')
np.allclose(result1, result2)
Explanation: Object attributes and indices
pd.eval() supports access to object attributes via the obj.attr syntax, and indexes via the obj[index] syntax:
End of explanation
df = pd.DataFrame(rng.rand(1000, 3), columns=['A', 'B', 'C'])
df.head()
Explanation: Other operations
Other operations such as function calls, conditional statements, loops, and other more involved constructs are currently not implemented in pd.eval().
If you'd like to execute these more complicated types of expressions, you can use the Numexpr library itself.
DataFrame.eval() for Column-Wise Operations
Just as Pandas has a top-level pd.eval() function, DataFrames have an eval() method that works in similar ways.
The benefit of the eval() method is that columns can be referred to by name.
We'll use this labeled array as an example:
End of explanation
result1 = (df['A'] + df['B']) / (df['C'] - 1)
result2 = pd.eval("(df.A + df.B) / (df.C - 1)")
np.allclose(result1, result2)
Explanation: Using pd.eval() as above, we can compute expressions with the three columns like this:
End of explanation
result3 = df.eval('(A + B) / (C - 1)')
np.allclose(result1, result3)
Explanation: The DataFrame.eval() method allows much more succinct evaluation of expressions with the columns:
End of explanation
df.head()
Explanation: Notice here that we treat column names as variables within the evaluated expression, and the result is what we would wish.
Assignment in DataFrame.eval()
In addition to the options just discussed, DataFrame.eval()  also allows assignment to any column.
Let's use the DataFrame from before, which has columns 'A', 'B', and 'C':
End of explanation
df.eval('D = (A + B) / C', inplace=True)
df.head()
Explanation: We can use df.eval() to create a new column 'D' and assign to it a value computed from the other columns:
End of explanation
df.eval('D = (A - B) / C', inplace=True)
df.head()
Explanation: In the same way, any existing column can be modified:
End of explanation
column_mean = df.mean(1)
result1 = df['A'] + column_mean
result2 = df.eval('A + @column_mean')
np.allclose(result1, result2)
Explanation: Local variables in DataFrame.eval()
The DataFrame.eval() method supports an additional syntax that lets it work with local Python variables.
Consider the following:
End of explanation
result1 = df[(df.A < 0.5) & (df.B < 0.5)]
result2 = pd.eval('df[(df.A < 0.5) & (df.B < 0.5)]')
np.allclose(result1, result2)
Explanation: The @ character here marks a variable name rather than a column name, and lets you efficiently evaluate expressions involving the two "namespaces": the namespace of columns, and the namespace of Python objects.
Notice that this @ character is only supported by the DataFrame.eval() method, not by the pandas.eval() function, because the pandas.eval() function only has access to the one (Python) namespace.
DataFrame.query() Method
The DataFrame has another method based on evaluated strings, called the query() method.
Consider the following:
End of explanation
result2 = df.query('A < 0.5 and B < 0.5')
np.allclose(result1, result2)
Explanation: As with the example used in our discussion of DataFrame.eval(), this is an expression involving columns of the DataFrame.
It cannot be expressed using the DataFrame.eval() syntax, however!
Instead, for this type of filtering operation, you can use the query() method:
End of explanation
Cmean = df['C'].mean()
result1 = df[(df.A < Cmean) & (df.B < Cmean)]
result2 = df.query('A < @Cmean and B < @Cmean')
np.allclose(result1, result2)
Explanation: In addition to being a more efficient computation, compared to the masking expression this is much easier to read and understand.
Note that the query() method also accepts the @ flag to mark local variables:
End of explanation
x = df[(df.A < 0.5) & (df.B < 0.5)]
Explanation: Performance: When to Use These Functions
When considering whether to use these functions, there are two considerations: computation time and memory use.
Memory use is the most predictable aspect. As already mentioned, every compound expression involving NumPy arrays or Pandas DataFrames will result in implicit creation of temporary arrays:
For example, this:
End of explanation
tmp1 = df.A < 0.5
tmp2 = df.B < 0.5
tmp3 = tmp1 & tmp2
x = df[tmp3]
Explanation: Is roughly equivalent to this:
End of explanation
df.values.nbytes
Explanation: If the size of the temporary DataFrames is significant compared to your available system memory (typically several gigabytes) then it's a good idea to use an eval() or query() expression.
You can check the approximate size of your array in bytes using this:
End of explanation
<END_TASK>