<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <center><h1>Applications of Machine Learning with Artificial Neural Networks and Supervised Regression</h1></center>
<br>
<center><i>Lucas Barbosa</i></center><hr>
<h3>The Architecture</h3>
<p>The structure of the ANN is going to be comprised of three layers
Step1:  <p>The importance in separating testing data from training data is to always make sure the model is according to the real world by comparing to testing values. The test data doesn’t go through the same pipelines as the training data does. The input data values are passed through the input neurons as matrices for <strong>computational speed-ups</strong>. </p>
<br>
<center>
$ X = \begin{bmatrix} X_{1,1} & X_{1,2}\ X_{2,1} & X_{2,2} \ X_{3,1} & X_{3,2} \end{bmatrix} $
</center>
<p>Before processing anymore data there is one more thing to account for. The input is in different units to the output. The network won’t be smart enough to map a generalisation between different units of data, one being hours and the other a score out of 100. We can take advantage of the fact that all the data is positive and divided the individual values by their respective maximum to get a number between 0 and 1.</p>
Step2:  <p>The next phase now for the input is for it to multiplied using the <strong>dot product</strong> to the first set of weights on the first layer of synapses. With a total of 6 weights where three are connected to each input neuron, a 2x3 matrix is formed
<ASSISTANT_TASK:>
Python Code:
import numpy as np
from scipy import optimize
# training data
x_train = np.array(([3,5],[5,1],[10,2],[6,1.5]), dtype=float)
y_train = np.array(([75],[82],[93],[70]), dtype=float)
# testing data
x_test = np.array(([4, 5.5],[4.5, 1],[9,2.5],[6,2]), dtype=float)
y_test = np.array(([70],[89],[85],[75]), dtype=float)
Explanation: <center><h1>Applications of Machine Learning with Artificial Neural Networks and Supervised Regression</h1></center>
<br>
<center><i>Lucas Barbosa</i></center><hr>
<h3>The Architecture</h3>
<p>The structure of the ANN is going to be comprised of three layers:</p>
<br>
<center><img src="img/simple_net.png" alt="Neural Net Structure"/></center>
<p>The input and output layers will have 2 neurons and 1 neuron respectively, this is dependant on the dimensionally of the data. For the sake of simplicity the hidden layer will consists of only three hidden units (hidden neurons). When defining the structure of the ANN it all comes down to the quantity of:</p>
<table>
    <thead>
        <tr>
            <th><center>Hyper-parameters</center></th>
            <th><center>Parameters</center></th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>Are fine-tuned while ANN is being trained to minimise the error of the predictions.</td>
            <td>Stay static the entire time unless changed or coded manually.</td>
        </tr>
    </tbody>
</table>
<p>Examples of hyper-parameters are the amount of neurons in each layer and even the amount of layers themselves. These should be determined before approaching implementation in code. Parameters however such as weights and regularisation values are changed throughout the life span of the ANN.</p>
<h3>Forward Propagation</h3>
<p>Forward propagation is the process of advancing our input data (hours studied and slept) through the actual network. It’ll be useful to both Mathematically and Programatically visualise the ANN process until we converge to our final output result.</p>
<p>The only libraries being used are numpy for matrix math and scipy for Mathematical Optimisation. Before anything goes anywhere the data required for the network to be trained comes in twofold. Training data and testing data.</p>
End of explanation
def scale_data(hours, test_score):
    MAX_SCORE = 100
    hours = hours / np.amax(hours, axis=0)
    test_score /= MAX_SCORE
    return hours, test_score
# normalize data
x_train, y_train = scale_data(x_train, y_train)
x_test, y_test   = scale_data(x_test, y_test)
x_train
y_train
Explanation: <p>The importance in separating testing data from training data is to always make sure the model is according to the real world by comparing to testing values. The test data doesn’t go through the same pipelines as the training data does. The input data values are passed through the input neurons as matrices for <strong>computational speed-ups</strong>. </p>
<br>
<center>
$ X = \begin{bmatrix} X_{1,1} & X_{1,2}\ X_{2,1} & X_{2,2} \ X_{3,1} & X_{3,2} \end{bmatrix} $
</center>
<p>Before processing anymore data there is one more thing to account for. The input is in different units to the output. The network won’t be smart enough to map a generalisation between different units of data, one being hours and the other a score out of 100. We can take advantage of the fact that all the data is positive and divided the individual values by their respective maximum to get a number between 0 and 1.</p>
End of explanation
class Neural_Network(object):
    
    def __init__(self):
        # define hyperparameters
        self.input_layer_size = 2
        self.hidden_layer_size = 3
        self.output_layer_size = 1
        
        #define parameters
        self.W1 = np.random.randn(self.input_layer_size, self.hidden_layer_size)
        self.W2 = np.random.randn(self.hidden_layer_size, self.output_layer_size)
        
    # forward propagation
    def forward(self, X):
        self.z2 = np.dot(X, self.W1)
        self.a2 = self.sigmoid(self.z2)
        self.z3 = np.dot(self.a2, self.W2)
        prediction = self.sigmoid(self.z3)
        return prediction
    
    # activation functions
    def sigmoid(self, z):
        return 1 / (1 + np.exp(-z))
NN = Neural_Network()
NN.forward(x_train)
y_train
Explanation: <p>The next phase now for the input is for it to multiplied using the <strong>dot product</strong> to the first set of weights on the first layer of synapses. With a total of 6 weights where three are connected to each input neuron, a 2x3 matrix is formed:</p>
<br>
<center>$ W^{(1)} = \begin{bmatrix} W^{(1)}{1,1} & W^{(1)}{1,2} & W^{(1)}{1,3} \ W^{(1)}{2,1} & W^{(1)}{2,2} & W^{(1)}{2,3} \end{bmatrix} $</center><br>
The activity of the second layer is produced though the matrix multiplication of the input and the first set of weights. This activity results in a 3x3 matrix z2
<center>$ z^{(2)} = W^{(1)} X $</center>
<br>
<center>$ z^{(2)} = \begin{bmatrix}
X_{1,1} W^{(1)}{1,1} + X{1,2} W^{(1)}{2,1} & X{1,1} W^{(1)}{1,2} + X{1,2} W^{(1)}{2,2} & X{1,1} W^{(1)}{1,3} + X{1,2} W^{(1)}{1,3} \ 
X{2,1} W^{(1)}{1,1} + X{2,2} W^{(1)}{2,1} & X{2,1} W^{(1)}{1,2} + X{2,2} W^{(1)}{2,2} & X{2,1} W^{(1)}{1,3} + X{2,2} W^{(1)}{2,3} \ 
X{3,1} W^{(1)}{1,1} + X{2,3} W^{(1)}{2,1} & X{3,1} W^{(1)}{1,2} + X{3,2} W^{(1)}{2,2} & X{3,1} W^{(1)}{1,3} + X{2,3} W^{(1)}_{2,3}
\end{bmatrix} $</center>
<p>The activity of the second layer z2 now needs to be activated through an <strong>activation function</strong>. The non-linear activation function being used will be the sigmoid function.</p>
<br>
<center>$ \varphi(x) = \frac{1}{1 + e^{-x}} $</center><br>
<p>This function needs to be applied to <strong>every</strong> entry in the activity matrix z2 to produce the activation of the second layer a2.</p>
<br>
<center>$ \varphi(z^{(2)}) = \begin{bmatrix}
\varphi(X_{1,1} W^{(1)}{1,1} + X{1,2} W^{(1)}{2,1}) & \varphi(X{1,1} W^{(1)}{1,2} + X{1,2} W^{(1)}{2,2}) & \varphi(X{1,1} W^{(1)}{1,3} + X{1,2} W^{(1)}{1,3}) \
\varphi(X{2,1} W^{(1)}{1,1} + X{2,2} W^{(1)}{2,1}) & \varphi(X{2,1} W^{(1)}{1,2} + X{2,2} W^{(1)}{2,2}) & \varphi(X{2,1} W^{(1)}{1,3} + X{2,2} W^{(1)}{2,3}) \
\varphi(X{3,1} W^{(1)}{1,1} + X{2,3} W^{(1)}{2,1}) & \varphi(X{3,1} W^{(1)}{1,2} + X{3,2} W^{(1)}{2,2}) & \varphi(X{3,1} W^{(1)}{1,3} + X{2,3} W^{(1)}_{2,3})
\end{bmatrix} $</center><br>
<center>$ a^{(2)} = \varphi(z^{(2)}) $</center><br>
<p>Once the activation has been computed it needs to multiplied by the <strong>second</strong> set of weights on the next synapses. This time there are 3 weights from 3 hidden neurons going into one output neuron, formulating a 3x1 matrix of new weights:</p>
<br>
<center>$ W^{(2)} = \begin{bmatrix}
W^{(2)}{1,1} \ 
W^{(2)}{2,1} \
W^{(2)}_{3,1}
\end{bmatrix} $</center>
<p>The matrix multiplication of the second layer activation and the second set of weights results in a 3x1 matrix yielding the activity for the third and final layer z3.</p>
<br>
<center>$ z^{(3)} = a^{(2)}W^{(2)} $</center> <br>
<center>$ z^{(3)} = \begin{bmatrix}
W^{(2)}{1,1} a^{(2)}{1,1} + W^{(2)}{1,1} a^{(2)}{1,2} + W^{(2)}{1,1} a^{(2)}{1,3} \ 
W^{(2)}{2,1} a^{(2)}{2,1} + W^{(2)}{2,1} a^{(2)}{2,2} + W^{(2)}{2,1} a^{(2)}{2,3} \ 
W^{(2)}{3,1} a^{(2)}{3,1} + W^{(2)}{2,1} a^{(2)}{3,2} + W^{(2)}{3,1} a^{(2)}{3,3} 
\end{bmatrix} $</center>
<p>The final computation is the activation function of the third layer activity z3. This result will also yield our final output prediciton from the output layer.</p>
<br>
<center>$ \varphi(z^{(3)}) = \begin{bmatrix}
\varphi(W^{(2)}{1,1} a^{(2)}{1,1} + W^{(2)}{1,1} a^{(2)}{1,2} + W^{(2)}{1,1} a^{(2)}{1,3}) \
\varphi(W^{(2)}{2,1} a^{(2)}{2,1} + W^{(2)}{2,1} a^{(2)}{2,2} + W^{(2)}{2,1} a^{(2)}{2,3}) \
\varphi(W^{(2)}{3,1} a^{(2)}{3,1} + W^{(2)}{2,1} a^{(2)}{3,2} + W^{(2)}{3,1} a^{(2)}{3,3})
\end{bmatrix}  $</center> <br>
<center>$ \hat{y} = \varphi(z^{(3)}) $</center>
<p>The forward propagation algorithm has been shown Mathematically, now its time to put all this into code. Before a class is built, weights need to be <strong>randomly</strong> initialised.</p>
End of explanation
<END_TASK>