<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Linear algebra overview projection example
Linear algebra is the study of vectors and linear transformations. This notebook introduces concepts form linear algebra in a birds-eye overview. The goal is not to get into the details, but to give the reader a taste of the different types of thinking
Step1:  Prerequisites
Linear algebra builds upon high school math concepts like
Step2:  Vector addition
Step3:  Vector length $\|\vec{u}\|$
Step4:  Unit-length vectors $\hat{u}$
Step5:  Dot product
Definition
The dot product of two vectors is proporional to the lengths for which the two vectors extend in the same direction.
If $\vec{u}=(u_1,u_2)$ and $\vec{v}=(v_1,v_2)$, then
Step6:  Intuition
Step8:  Projections
A projection of the vector $\vec{v}$ in the direction $\vec{d}$ is denoted $\Pi_{\vec{d}}(\vec{v})$. The formula for computing the projections uses the dot product operation
Step9:  Projections play an important role in physics. For example, when solving a two dimensional projectile problem we often decompose vector quantities like forces $\vec{F}$, velocities $\vec{v}$, and momenta $\vec{p}$ into their $x$- and $y$-components
Step11:  Take 1
Step13:  Vector functions
Observe that the function P is a vector function—a function that takes vectors as inputs and produces vectors as outputs. In mathematical notation we write this as
$$
  P 
Step15:  Take 3
Step16:  Equivalence relationship between linear transformstions $T$ and matrices $M_T$
Step17:  Matrix operations
Addition (denoted $A+B$)
Subtraction, the inverse of addition (denoted $A-B$)
Scaling by a constant $\alpha$  (denoted $\alpha A$)
Matrix-vector product (denoted $A\vec{x}$) 
Matrix product (denoted $AB$) 
Matrix inverse (denoted $A^{-1}$)
Trace (denoted $\textrm{Tr}(A)$)
Determinant (denoted $\textrm{det}(A)$ or $|A|$)
Matrix-vector product
$$
  A \vec{x}
  \quad
  \Leftrightarrow
  \quad
    \begin{bmatrix}
    a_{11} & a_{12} \
    a_{21} & a_{22} \
    a_{31} & a_{32} 
    \end{bmatrix}
    !!
    \begin{bmatrix}
    x_1 \
    x_2 
    \end{bmatrix}
 \equiv 
    \begin{bmatrix}
    x_1a_{11} + x_2a_{12} \
    x_1a_{21} + x_2a_{22} \
    x_1a_{31} + x_2a_{32} 
    \end{bmatrix}
$$
The matrix-vector product is defined this way so it can represent linear transformations.
Step18:  Matrix-matrix product
$$
  AB
  \quad \Leftrightarrow  \quad
\begin{bmatrix}
 a_{11}         &  a_{12}    \[1.5mm]
 a_{21}         &  a_{22}    \[1.5mm]
 a_{31}         &  a_{32}  
\end{bmatrix}
!!
\begin{bmatrix}
 b_{11}         &  b_{12}    \[1.5mm]
 b_{21}         &  b_{22}    \
\end{bmatrix}
=
\begin{bmatrix}
a_{11}b_{11} + a_{12}b_{21}     &  a_{11}b_{12} + a_{12}b_{22}   \[1.5mm]
a_{21}b_{11} + a_{22}b_{21}     &  a_{21}b_{12} + a_{22}b_{22}   \[1.5mm]
a_{31}b_{11} + a_{32}b_{21}     &  a_{31}b_{12} + a_{32}b_{22} 
\end{bmatrix}
$$
Step19:  The matrix-matrix product implements composition of linear transformations
Step20:  Matrix inverse
For an invertible matrix $A$, the matrix inverse $A^{-1}$ acts to undo the effects of $A$
Step21:  Matrix equations
Suppose we're given the equation
$$
 A\vec{x} = \vec{b}
$$
and we want to solve for $\vec{x}$.
One way to find $\vec{x}$ is to multiply both sides of the equation by $A^{-1}$
<ASSISTANT_TASK:>
Python Code:
# setup SymPy
from sympy import *
x, y, z, t = symbols('x y z t')
init_printing()
# a vector is a special type of matrix (an n-vector is either a nx1 or a 1xn matrix)
Vector = Matrix   # define alias Vector so I don't have to explain this during video
# setup plotting
%matplotlib inline
import matplotlib.pyplot as mpl
from plot_helpers import plot_vec, plot_vecs, plot_line, plot_plane, autoscale_arrows
Explanation: Linear algebra overview projection example
Linear algebra is the study of vectors and linear transformations. This notebook introduces concepts form linear algebra in a birds-eye overview. The goal is not to get into the details, but to give the reader a taste of the different types of thinking: computational, geometrical, and theoretical, that are used in linear algebra.
Plan
Prerequisites
Vectors
Definition 
Geomterical interpretation
Dot product
Projections
Projection operation
Example: projection onto the line with equation $x-y=0$
Vector functions
Linear property: $f(a\mathbf{x} + b\mathbf{y}) = af(\mathbf{x}) + bf(\mathbf{y})$
Projection transformation P
Matrix representation of linear transformations
Linear transformation <--> Matrix-vector product equivalence
Show matrix representation M_P of Projection transformation P
Matrices
Definition 
Matrix operations
Matrix-vector product 
Matrix-matrix product
Trace
Determinant
Matrix inverse
Matrix equations
Reduced row echelon form
End of explanation
# define two vectors
u = Vector([1,1])
v = Vector([1,-1])
u
v
plot_vecs(u, v)
autoscale_arrows()
Explanation: Prerequisites
Linear algebra builds upon high school math concepts like:
  - Geometry (lines, curves, areas, triangles)
  - Numbers (integers, rationals, reals, complex numbers)
  - Functions ($f(x)$ takes an input $x$ and produces an output $y$)
Vectors
End of explanation
# graphical
plot_vecs(u,v)
plot_vec(v, at=u, color='b')
plot_vec(u+v, color='r')
autoscale_arrows()
# algebraic
u+v
Explanation: Vector addition
End of explanation
u.norm()
Explanation: Vector length $\|\vec{u}\|$
End of explanation
uhat = u/u.norm()
plot_vecs(u, uhat)
uhat
Explanation: Unit-length vectors $\hat{u}$
End of explanation
u = Vector([2,2])
v = Vector([3,0])
plot_vecs(u,v)
autoscale_arrows()
u.dot(v)
Explanation: Dot product
Definition
The dot product of two vectors is proporional to the lengths for which the two vectors extend in the same direction.
If $\vec{u}=(u_1,u_2)$ and $\vec{v}=(v_1,v_2)$, then:
$$
  \vec{u} \cdot \vec{v} = u_1v_1 + u_2v_2 = \|\vec{u}\| \|\vec{v}\| \cos \theta_{uv},
$$
where $\theta_{uv}$ is the angle between the vectors.
End of explanation
# split the vector u into two parts:
u_parallel_to_v = Vector([2,0])
u_perp_to_v = Vector([0,2])
plot_vecs(u, v, u_parallel_to_v, u_perp_to_v)
autoscale_arrows()
u == u_parallel_to_v + u_perp_to_v
# the dot product uses only the part of u that is parallel to v
u.dot(v) == u_parallel_to_v.dot(v) == u_parallel_to_v.norm()*v.norm()
# two vetors that are perpendicular, have zero dot product together
u_perp_to_v.dot(v)
Explanation: Intuition
End of explanation
def proj(v, d):
    Computes the projection of vector `v` onto direction `d`.
    return v.dot( d/d.norm() )*( d/d.norm() )
v = Vector([2,2])
d = Vector([3,0])
proj_v_on_d = proj(v,d)
plot_vecs(d, v, proj_v_on_d)
autoscale_arrows()
Explanation: Projections
A projection of the vector $\vec{v}$ in the direction $\vec{d}$ is denoted $\Pi_{\vec{d}}(\vec{v})$. The formula for computing the projections uses the dot product operation:
$$
   \Pi_{\vec{d}}(\vec{v})
      \ \equiv \ 
      (\vec{v} \cdot \hat{d}) \hat{d}
      \ =  \ 
      \left(\vec{v} \cdot \frac{\vec{d}}{\|\vec{d}\|} \right) \frac{\vec{d}}{\|\vec{d}\|}.
$$
General projection operation
End of explanation
# The line with equation y = x can also be written as a paramteric equation 
# [x,y] = [0,0] + s*[1,1] where d = [1,1] is called the direction vector the line
d = Vector([1,1])
plot_line(d,[0,0])
Explanation: Projections play an important role in physics. For example, when solving a two dimensional projectile problem we often decompose vector quantities like forces $\vec{F}$, velocities $\vec{v}$, and momenta $\vec{p}$ into their $x$- and $y$-components: $(F_x,F_y)$, $(v_x,v_y)$, and $(p_x,p_y)$. This decomposition of vectors can transform a complicated two-dimensional problem into two simpler one-dimensaional problems, which can be solved independetly.
Example: projection onto the line with equation $y=x$
End of explanation
# want a function that computes the projection onto the line with equation y = x for any vec
def P(vec):
    Compute the projection of vector `vec` onto line y=x.
    return proj(vec, d)
v = Vector([5,0])
plot_line(d,[0,0])
plot_vecs(v, P(v))
P(v)
Explanation: Take 1: using projection operation
End of explanation
ihat = Vector([1,0])
jhat = Vector([0,1])
Pihat = P(ihat)
Pjhat = P(jhat)
Pihat, Pjhat
def P2(vec):
    Compute the projection of vector `vec` onto line y=x.
    return vec[0]*Pihat + vec[1]*Pjhat
v = Vector([5,0])
plot_line(d,[0,0])
plot_vecs(v, P2(v))
Explanation: Vector functions
Observe that the function P is a vector function—a function that takes vectors as inputs and produces vectors as outputs. In mathematical notation we write this as
$$
  P : \mathbb{R}^2 \to \mathbb{R}^2.
$$
Linear property:
A linear transformation $T$ is a vector function that obeys the linear property:
$$
 T(a\vec{x} + b\vec{y}) = aT(\vec{x}) + bT(\vec{y}).
$$
Take 2: projection transformation P
The projection $P$ is a linear transformation, so it obeeys:
$$
 P\left( \begin{bmatrix}a \ b \end{bmatrix} \right)
 = P(a\hat{\imath} + b\hat{\jmath}) 
 = aP(\hat{\imath}) + bP(\hat{\jmath}).
$$
End of explanation
M_P = Matrix([[1,1],
              [1,1]])/2
M_P
def P3(vec):
    Compute the projection of vector `vec` onto the line y=x.
    return M_P*vec
v = Vector([4,0])
plot_line(d, [0,0])
plot_vecs(v, P3(v))
M_P.shape
Explanation: Take 3: linear transformation as matrix-vector product
Matrix definition
$$
  \alpha \vec{u} + \beta \vec{v}
  =
  \alpha 
  \begin{bmatrix}u_1 \ u_2 \end{bmatrix}
  +
  \beta
  \begin{bmatrix}v_1 \ v_2 \end{bmatrix}
  =
  \begin{bmatrix}u_1 & v_1 \ u_2 & v_2 \end{bmatrix}
  !
  \begin{bmatrix} \alpha \ \beta \end{bmatrix}.
$$
End of explanation
A = Matrix([[1,2],
            [3,4],
            [5,6]])
A
A.shape
Explanation: Equivalence relationship between linear transformstions $T$ and matrices $M_T$:
$$
  T : \mathbb{R}^n \to \mathbb{R}^m
  \qquad
  \Leftrightarrow
  \qquad
  M_T \in \mathbb{R}^{m \times n}
$$
Matrices
A matrix is a two-dimensional array of numbers.
Example
End of explanation
a_11, a_12, a_21, a_22, a_31, a_32 = symbols('a_11 a_12 a_21 a_22 a_31 a_32')
x_1, x_2 = symbols('x_1 x_2')
A = Matrix([
        [a_11, a_12],
        [a_21, a_22], 
        [a_31, a_32]])
x = Vector([x_1,x_2])
A*x
Explanation: Matrix operations
Addition (denoted $A+B$)
Subtraction, the inverse of addition (denoted $A-B$)
Scaling by a constant $\alpha$  (denoted $\alpha A$)
Matrix-vector product (denoted $A\vec{x}$) 
Matrix product (denoted $AB$) 
Matrix inverse (denoted $A^{-1}$)
Trace (denoted $\textrm{Tr}(A)$)
Determinant (denoted $\textrm{det}(A)$ or $|A|$)
Matrix-vector product
$$
  A \vec{x}
  \quad
  \Leftrightarrow
  \quad
    \begin{bmatrix}
    a_{11} & a_{12} \
    a_{21} & a_{22} \
    a_{31} & a_{32} 
    \end{bmatrix}
    !!
    \begin{bmatrix}
    x_1 \
    x_2 
    \end{bmatrix}
 \equiv 
    \begin{bmatrix}
    x_1a_{11} + x_2a_{12} \
    x_1a_{21} + x_2a_{22} \
    x_1a_{31} + x_2a_{32} 
    \end{bmatrix}
$$
The matrix-vector product is defined this way so it can represent linear transformations.
End of explanation
b_11, b_12, b_21, b_22 = symbols('b_11 b_12 b_21 b_22')
B = Matrix([[b_11, b_12],
            [b_21, b_22]])
A*B
#  (AB)_ij = dot product of ith row of A with jth col of B
(A*B)[2,1] == A[2,:].dot( B[:,1])
Explanation: Matrix-matrix product
$$
  AB
  \quad \Leftrightarrow  \quad
\begin{bmatrix}
 a_{11}         &  a_{12}    \[1.5mm]
 a_{21}         &  a_{22}    \[1.5mm]
 a_{31}         &  a_{32}  
\end{bmatrix}
!!
\begin{bmatrix}
 b_{11}         &  b_{12}    \[1.5mm]
 b_{21}         &  b_{22}    \
\end{bmatrix}
=
\begin{bmatrix}
a_{11}b_{11} + a_{12}b_{21}     &  a_{11}b_{12} + a_{12}b_{22}   \[1.5mm]
a_{21}b_{11} + a_{22}b_{21}     &  a_{21}b_{12} + a_{22}b_{22}   \[1.5mm]
a_{31}b_{11} + a_{32}b_{21}     &  a_{31}b_{12} + a_{32}b_{22} 
\end{bmatrix}
$$
End of explanation
A*(B*x)
expand( A*(B*x) ) == expand( (A*B)*x )
# analogy with ordinary functions...
x = symbols('x')
def f(x):
    return 2*x
def g(x):
    return  3*x
f(g(x))
def h(x):
    return 6*x
h(x)
Explanation: The matrix-matrix product implements composition of linear transformations:
End of explanation
A = Matrix([[1,2],
            [3,9]])
A.inv()
A.inv()*A
Explanation: Matrix inverse
For an invertible matrix $A$, the matrix inverse $A^{-1}$ acts to undo the effects of $A$:
$$
 A^{-1} A \vec{v} = \vec{v}.
$$
End of explanation
A = Matrix([[1,2],
            [3,9]])
b = Vector([5,21]) 
x = A.inv()*b
x
# verify A*x == b
A*x 
Explanation: Matrix equations
Suppose we're given the equation
$$
 A\vec{x} = \vec{b}
$$
and we want to solve for $\vec{x}$.
One way to find $\vec{x}$ is to multiply both sides of the equation by $A^{-1}$:
$$
 A^{-1}A\vec{x} = A^{-1}\vec{b}
$$
since $A^{-1}$ cancels $A$ we obtain:
$$
 \vec{x} = A^{-1}\vec{b}.
$$
Example
$$
    \begin{bmatrix}
    1 & 2 \
    3 & 9
    \end{bmatrix}
    !!
    \begin{bmatrix}
    x_1 \
    x_2 
    \end{bmatrix}
    =
    \begin{bmatrix}
    5 \
    21 
    \end{bmatrix}  
\qquad
\Rightarrow
\qquad
    \begin{bmatrix}
    x_1 \
    x_2 
    \end{bmatrix}
    =
    \begin{bmatrix}
    1 & 2 \
    3 & 9
    \end{bmatrix}^{-1}
    !!
    \begin{bmatrix}
    5 \
    21 
    \end{bmatrix}  
$$
End of explanation
<END_TASK>