<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Indexing
Setting up the data
Let's create the structures that will be used later in this notebook
Step1:  We can index elements in an array using the square bracket and indices
Step2:  If we omit an index of a multidimensional array it returns the whole row (or, in general, a N-1 dimensional array)
Step3:  The same thing can be achieved with using 
Step4:  We can assign new values to elements in an array using indexing
Step5:  Index slicing
Index slicing is the technical name for the syntax M[lower
Step6:  Array slices are mutable
Step7:  We can omit any of the three parameters in M[lower
Step8:  Negative indices counts from the end of the array (positive index from the begining)
Step9:  Index slicing works exactly the same way for multidimensional arrays
Step10:  Indexing and Array Memory Management
Numpy arrays support two different way of storing data into memory, namely
F-Contiguous 
i.e. column-wise storage, Fortran-like
C-Contiguous
i.e. row-wise storage, C-like
The storage strategy is controlled by the parameter order of np.array
<img src="images/storage_simple.png" />
Step11:  Note
Step12:  <img src="images/storage_index.png" />
Fancy Indexing
Fancy indexing is the name for when an array or list is used in-place of an index
Step13:  We can also index masks
Step14:  Alternatively
Step15:  This feature is very useful to conditionally select elements from an array, using for example comparison operators
Step16:  Alternatively, we can use the condition (mask) array directly within brackets to index the array
Step17:  Exercises on Indexing
Index slicing is the technical name for the syntax M[lower
Step18:  np.repeat
Step19:  np.tile
Step20:  np.concatenate
Step21:  np.hstack and np.vstack
Step22:  Iterating over array elements
Generally, we want to avoid iterating over the elements of arrays whenever we can (at all costs). The reason is that in a interpreted language like Python (or MATLAB), iterations are really slow compared to vectorized operations. 
However, sometimes iterations are unavoidable. For such cases, the Python for loop is the most convenient way to iterate over an array
Step23:  When we need to iterate over each element of an array and modify its elements, it is convenient to use the enumerate function to obtain both the element and its index in the for loop
Step25:  Vectorizing functions
As mentioned several times by now, to get good performance we should try to avoid looping over elements in our vectors and matrices, and instead use vectorized algorithms. The first step in converting a scalar algorithm to a vectorized algorithm is to make sure that the functions we write work with vector inputs.
Step26:  OK, that didn't work because we didn't write the Theta function so that it can handle with vector input... 
To get a vectorized version of Theta we can use the Numpy function np.vectorize. In many cases it can automatically vectorize a function
Step27:  np.frompyfunc
Universal functions (Ufuncs) work on arrays, element-by-element, or on scalars. 
Ufuncs accept a set of scalars as input, and produce a set of scalars as output.
Step28:  Excercise
Avoiding Vectorize
Implement the function to accept vector input from the beginning 
This requires "more effort" but might give better performance
Vectorisation and Broadcasting
Vectorizing code is the key to writing efficient numerical calculation with Python/Numpy. 
That means that as much as possible of a program should be formulated in terms of matrix and vector operations, like matrix-matrix multiplication.
However, sometimes it may happen that we need to perform operations between ndarray objects of different size and shape so we might be tempted of performing this operation explicitly (i.e. explicit for loop).
Although we will get correct results (assumed we iterated arrays in the right way), this will produce very inefficient code.
For this reason, NumPy has a very important concept called Broadcasting.
Broadcasting is Numpy's terminology for performing mathematical operations between arrays with different shapes (assuming scalars being array with 0-dim).
Working example
Assume we have some data in a matrix D
Step29:  And we also have another vector adj of values that contains some adjusting factors that we might want to apply to each sample (row) of data in D
Step30:  Naive Solution
Step31:  This is a reasonable approach when coding in a low-level programming language
Step32:  Nice, it's shorter too, and slightly faster! To appreciate even more performance gain, of our np.tile solution, we could try increasing the size of D to a bigger structure
Step33:  The loop-in-Python method takes ~1.5 seconds, the stretching method takes ~48 milliseconds, a ~75x speedup.
And now, finally, comes the interesting part. 
You see, the operation we just did - stretching one array so that its shape matches that of another and then applying some element-wise operation between them - is actually pretty common. 
This often happens when we want to take a lower-dimensional array and use it to perform a computation along some axis of a higher-dimensional array. 
In fact, when taken to the extreme this is exactly what happens when we perform an operation between an array and a scalar - the scalar is stretched across the whole array so that the element-wise operation gets the same scalar value for each element it computes.
Numpy generalizes this concept into broadcasting - a set of rules that permit element-wise computations between arrays of different shapes, as long as some constraints apply. 
Incidentally, this lets Numpy achieve two separate goals - usefulness as well as more consistent and general semantics. 
Binary operators like * are element-wise, but what happens when we apply them between arrays of different shapes? Should it work or should it be rejected? If it works, how should it work? 
Broadcasting defines the semantics of these operations.
Step34:  How Broadcasting works
Element-wise operations on arrays are only valid when the arrays' shapes are either equal or compatible. 
The equal shapes case is trivial - this is the stretched array from the example above. What does compatible mean, though?
To determine if two shapes are compatible, Numpy compares their dimensions, starting with the trailing ones and working its way backwards. 
For example, for the shape (4, 3, 2) the trailing dimension is 2, and working from 2 "backwards" produces
Step35:  Element-wise array-array operations
When we add, subtract, multiply and divide arrays with each other, the default behaviour is element-wise operations
Step36:  If we multiply arrays with compatible shapes, we get an element-wise multiplication of each row
Step37:  Matrix algebra
What about matrix mutiplication? 
There are two ways. 
We can either use the np.dot function, which applies a matrix-matrix, matrix-vector, or inner vector multiplication to its two arguments
Step38:  A new dedicated Infix operator for Matrix Multiplication
Python 3.5 (released on the 24th Aug.) introduces a new binary operator to be used for matrix multiplication, called @ . (Mnemonic
<ASSISTANT_TASK:>
Python Code:
import numpy as np
np.random.seed(42)  # Setting the random seed
# a vector: the argument to the array function is a Python list
v = np.random.rand(10)
v
# a matrix: the argument to the array function is a nested Python list
M = np.random.rand(10, 2)
M
Explanation: Indexing
Setting up the data
Let's create the structures that will be used later in this notebook
End of explanation
# v is a vector, and has only one dimension, taking one index
v[0]
# M is a matrix, or a 2 dimensional array, taking two indices 
M[1,1]
Explanation: We can index elements in an array using the square bracket and indices:
End of explanation
M[1] 
Explanation: If we omit an index of a multidimensional array it returns the whole row (or, in general, a N-1 dimensional array)
End of explanation
M[1,:] # row 1
M[:,1] # column 1
Explanation: The same thing can be achieved with using : instead of an index:
End of explanation
M[0,0] = 1
M
# also works for rows and columns
M[1,:] = 0
M[:,1] = -1
M
Explanation: We can assign new values to elements in an array using indexing:
End of explanation
a = np.array([1,2,3,4,5])
a
a[1:3]
Explanation: Index slicing
Index slicing is the technical name for the syntax M[lower:upper:step] to extract part of an array:
End of explanation
a[1:3] = [-2,-3]
a
Explanation: Array slices are mutable: if they are assigned a new value the original array from which the slice was extracted is modified:
End of explanation
a[::] # lower, upper, step all take the default values
a[::2] # step is 2, lower and upper defaults to the beginning and end of the array
a[:3] # first three elements
a[3:] # elements from index 3
Explanation: We can omit any of the three parameters in M[lower:upper:step]:
End of explanation
a = np.array([1,2,3,4,5])
a[-1] # the last element in the array
a[-3:] # the last three elements
Explanation: Negative indices counts from the end of the array (positive index from the begining):
End of explanation
A = np.array([[n+m*10 for n in range(5)] 
              for m in range(5)])
A
# a block from the original array
A[1:4, 1:4]
# strides
A[::2, ::2]
Explanation: Index slicing works exactly the same way for multidimensional arrays:
End of explanation
import numpy as np
FC = np.array([[1, 2, 3],    [4, 5, 6], 
               [7, 8, 9],    [10, 11, 12]], order='F')
CC = np.array([[1, 2, 3],    [4, 5, 6], 
               [7, 8, 9],    [10, 11, 12]], order='C')
Explanation: Indexing and Array Memory Management
Numpy arrays support two different way of storing data into memory, namely
F-Contiguous 
i.e. column-wise storage, Fortran-like
C-Contiguous
i.e. row-wise storage, C-like
The storage strategy is controlled by the parameter order of np.array
<img src="images/storage_simple.png" />
End of explanation
FC[0, 1]
CC[0, 1]
FC.shape
CC.shape
Explanation: Note: no changes in meaning for indexing operations
End of explanation
row_indices = [1, 2, 3]
A[row_indices]
col_indices = [1, 2, -1] # remember, index -1 means the last element
A[row_indices, col_indices]
Explanation: <img src="images/storage_index.png" />
Fancy Indexing
Fancy indexing is the name for when an array or list is used in-place of an index:
End of explanation
b = np.array([n for n in range(5)])
b
row_mask = np.array([True, False, True, False, False])
b[row_mask]
Explanation: We can also index masks: 
If the index mask is an Numpy array of with data type bool, then an element is selected (True) or not (False) depending on the value of the index mask at the position each element:
End of explanation
# same thing
row_mask = np.array([1,0,1,0,0], dtype=bool)
b[row_mask]
Explanation: Alternatively:
End of explanation
x = np.arange(0, 10, 0.5)
x
mask = (5 < x)
mask
x[mask]
Explanation: This feature is very useful to conditionally select elements from an array, using for example comparison operators:
End of explanation
x[(5 < x)]
Explanation: Alternatively, we can use the condition (mask) array directly within brackets to index the array
End of explanation
a = np.array([[1, 2], [3, 4]])
Explanation: Exercises on Indexing
Index slicing is the technical name for the syntax M[lower:upper:step] to extract part of an array
Ex 2.1
Generate a three-dimensional array of any size containing random numbers taken from an uniform distribution (guess the numpy function in np.random). Then print out separately the first entry along the three axis (i.e. x, y, z)  
hint: Slicing with numpy arrays works quite like Python lists
Ex 2.2
Create a vector and print out elements in reverse order
Hint: Use slicing for this exercise
Ex 2.3
Generate a $7 \times 7$ matrix and replace all the elements in odd rows and even columns with 1.
Hint: Use slicing to solve this exercise!
Note: Take a look at the original matrix, then.
Use fancy indexing to get all the elements of the previous matrix that are equals to 1
Ex 2.4
Generate a 10 x 10 matrix of numbers A. Then, generate a numpy array of integers in range 1-9. Pick 5 random values (with no repetition) from this array and use these values to extract rows from the original matrix A.
Ex 2.5
Repeat the previous exercise but this time extract columns from A
Ex 2.6
Generate an array of numbers from 0 to 20 with step 0.5. 
Extract all the values greater than a randomly generated number in the same range.
hint: Try to write the condition as an expression and save it to a variable. Then, use this variable in square brackets to index.... this is when the magic happens!
Stacking and repeating arrays
Using function repeat, tile, vstack, hstack, and concatenate we can create larger vectors and matrices from smaller ones:
np.tile and np.repeat
End of explanation
# repeat each element 3 times
np.repeat(a, 3)
Explanation: np.repeat
End of explanation
# tile the matrix 3 times 
np.tile(a, 3)
Explanation: np.tile
End of explanation
b = np.array([[5, 6]])
np.concatenate((a, b), axis=0)
np.concatenate((a, b.T), axis=1)
Explanation: np.concatenate
End of explanation
np.vstack((a,b))
np.hstack((a,b.T))
Explanation: np.hstack and np.vstack
End of explanation
v = np.array([1,2,3,4])
for element in v:
    print(element)
M = np.array([[1,2], [3,4]])
for row in M:
    print("row", row)
    
    for element in row:
        print(element)
Explanation: Iterating over array elements
Generally, we want to avoid iterating over the elements of arrays whenever we can (at all costs). The reason is that in a interpreted language like Python (or MATLAB), iterations are really slow compared to vectorized operations. 
However, sometimes iterations are unavoidable. For such cases, the Python for loop is the most convenient way to iterate over an array:
End of explanation
for row_idx, row in enumerate(M):
    print("row_idx", row_idx, "row", row)
    
    for col_idx, element in enumerate(row):
        print("col_idx", col_idx, "element", element)
       
        # update the matrix M: square each element
        M[row_idx, col_idx] = element ** 2
# each element in M is now squared
M
Explanation: When we need to iterate over each element of an array and modify its elements, it is convenient to use the enumerate function to obtain both the element and its index in the for loop:
End of explanation
def Theta(x):
    
    Scalar implemenation of the Heaviside step function.
    
    if x >= 0:
        return 1
    else:
        return 0
Theta(array([-3,-2,-1,0,1,2,3]))
Explanation: Vectorizing functions
As mentioned several times by now, to get good performance we should try to avoid looping over elements in our vectors and matrices, and instead use vectorized algorithms. The first step in converting a scalar algorithm to a vectorized algorithm is to make sure that the functions we write work with vector inputs.
End of explanation
Theta_vec = np.vectorize(Theta)
Theta_vec(np.array([-3,-2,-1,0,1,2,3]))
Explanation: OK, that didn't work because we didn't write the Theta function so that it can handle with vector input... 
To get a vectorized version of Theta we can use the Numpy function np.vectorize. In many cases it can automatically vectorize a function:
np.vectorize
End of explanation
Theta_ufunc = np.frompyfunc(Theta, 1, 1)
print("Result: ", Theta_ufunc(np.arange(4)))
Explanation: np.frompyfunc
Universal functions (Ufuncs) work on arrays, element-by-element, or on scalars. 
Ufuncs accept a set of scalars as input, and produce a set of scalars as output.
End of explanation
D = np.array([ [0.3, 2.5, 3.5],
               [2.9, 27.5, 0],
               [0.4, 1.3, 23.9],
               [14.4, 6, 2.3]])
Explanation: Excercise
Avoiding Vectorize
Implement the function to accept vector input from the beginning 
This requires "more effort" but might give better performance
Vectorisation and Broadcasting
Vectorizing code is the key to writing efficient numerical calculation with Python/Numpy. 
That means that as much as possible of a program should be formulated in terms of matrix and vector operations, like matrix-matrix multiplication.
However, sometimes it may happen that we need to perform operations between ndarray objects of different size and shape so we might be tempted of performing this operation explicitly (i.e. explicit for loop).
Although we will get correct results (assumed we iterated arrays in the right way), this will produce very inefficient code.
For this reason, NumPy has a very important concept called Broadcasting.
Broadcasting is Numpy's terminology for performing mathematical operations between arrays with different shapes (assuming scalars being array with 0-dim).
Working example
Assume we have some data in a matrix D:
End of explanation
adj = np.array([9, 4, 4])
Explanation: And we also have another vector adj of values that contains some adjusting factors that we might want to apply to each sample (row) of data in D
End of explanation
%%timeit
# Create a new array filled with zeros, of the same shape as macros.
result = np.zeros_like(D)
# Now multiply each row of macros by cal_per_macro. In Numpy, `*` is
# element-wise multiplication between two arrays.
for i in range(D.shape[0]):
    result[i, :] = D[i, :] * adj
    
result
Explanation: Naive Solution
End of explanation
%%timeit 
adj_stretch = np.tile(adj, (D.shape[0], 1))
D * adj_stretch
adj_stretch
Explanation: This is a reasonable approach when coding in a low-level programming language: allocate the output, loop over input performing some operation, write result into output. 
In Numpy, however, this is fairly bad for performance because the looping is done in (slow) Python code instead of internally by Numpy in (fast) C code
(we've proven this already when we compared ndarray vs list)
Using np.tile
Idea: Leverage on np.tile function to replicate adj over the number
of rows D has (i.e. D.shape[0]).
End of explanation
D_large = np.random.rand(10**6, 10)
adj_large = np.random.rand(10)
D_large.shape, adj_large.shape
%%timeit
# Create a new array filled with zeros, of the same shape as macros.
result_large = np.zeros_like(D_large)
# Now multiply each row of macros by cal_per_macro. In Numpy, `*` is
# element-wise multiplication between two arrays.
for i in range(D_large.shape[0]):
    result_large[i, :] = D_large[i, :] * adj_large
%%timeit 
adj_large_stretch = np.tile(adj_large, (D_large.shape[0], 1))
D_large * adj_large_stretch
Explanation: Nice, it's shorter too, and slightly faster! To appreciate even more performance gain, of our np.tile solution, we could try increasing the size of D to a bigger structure:
End of explanation
%%timeit
## Back to our example
D * adj  # Broadcasting
Explanation: The loop-in-Python method takes ~1.5 seconds, the stretching method takes ~48 milliseconds, a ~75x speedup.
And now, finally, comes the interesting part. 
You see, the operation we just did - stretching one array so that its shape matches that of another and then applying some element-wise operation between them - is actually pretty common. 
This often happens when we want to take a lower-dimensional array and use it to perform a computation along some axis of a higher-dimensional array. 
In fact, when taken to the extreme this is exactly what happens when we perform an operation between an array and a scalar - the scalar is stretched across the whole array so that the element-wise operation gets the same scalar value for each element it computes.
Numpy generalizes this concept into broadcasting - a set of rules that permit element-wise computations between arrays of different shapes, as long as some constraints apply. 
Incidentally, this lets Numpy achieve two separate goals - usefulness as well as more consistent and general semantics. 
Binary operators like * are element-wise, but what happens when we apply them between arrays of different shapes? Should it work or should it be rejected? If it works, how should it work? 
Broadcasting defines the semantics of these operations.
End of explanation
v1 = np.arange(0, 5)
v1 * 2
v1 + 2
A = np.array([[n+m*10 for n in range(5)] for m in range(5)])
print('A * 2: ', '\n', A * 2)
print('A + 2: ', '\n', A + 2)
Explanation: How Broadcasting works
Element-wise operations on arrays are only valid when the arrays' shapes are either equal or compatible. 
The equal shapes case is trivial - this is the stretched array from the example above. What does compatible mean, though?
To determine if two shapes are compatible, Numpy compares their dimensions, starting with the trailing ones and working its way backwards. 
For example, for the shape (4, 3, 2) the trailing dimension is 2, and working from 2 "backwards" produces: 2, then 3, then 4.
If two dimensions are equal, or if one of them equals 1, the comparison continues. Otherwise, you'll see a ValueError raised (saying something like "operands could not be broadcast together with shapes ...").
When one of the shapes runs out of dimensions (because it has less dimensions than the other shape), Numpy will use 1 in the comparison process until the other shape's dimensions run out as well.
Once Numpy determines that two shapes are compatible, the shape of the result is simply the maximum of the two shapes' sizes in each dimension.
More here: Broadcasting Arrays in NumPy
Scalar-array operations
We can use the usual arithmetic operators to multiply, add, subtract, and divide arrays with scalar numbers.
End of explanation
A * A # element-wise multiplication
v1 * v1
Explanation: Element-wise array-array operations
When we add, subtract, multiply and divide arrays with each other, the default behaviour is element-wise operations:
End of explanation
A.shape, v1.shape
A * v1  #Broadcasting
Explanation: If we multiply arrays with compatible shapes, we get an element-wise multiplication of each row:
End of explanation
np.dot(A, A)
np.dot(A, v1)
np.dot(v1, v1)
Explanation: Matrix algebra
What about matrix mutiplication? 
There are two ways. 
We can either use the np.dot function, which applies a matrix-matrix, matrix-vector, or inner vector multiplication to its two arguments:
End of explanation
A @ v1
Explanation: A new dedicated Infix operator for Matrix Multiplication
Python 3.5 (released on the 24th Aug.) introduces a new binary operator to be used for matrix multiplication, called @ . (Mnemonic: @ is * for mATrices.)
Some useful hints:
PEP465 Description: https://www.python.org/dev/peps/pep-0465/#abstract
Motivations
Rationale for Specification Details
End of explanation
<END_TASK>