<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Manual Neural Network
In this notebook we will manually build out a neural network that mimics the TensorFlow API. This will greatly help your understanding when working with the real TensorFlow!
Quick Note on Super() and OOP
Step1:  The child class will use its own initialization method, if not specified otherwise.
Step2:  If we want to use initialization from the parent class, we can do that using
Step6:  Operation
Step7:  Example Operations
Addition
Step8:  Multiplication
Step9:  Matrix Multiplication
Step11:  Placeholders
Step13:  Variables
Step15:  Graph
Step16:  A Basic Graph
$$ z = Ax + b $$
With A=10 and b=1
$$ z = 10x + 1 $$
Just need a placeholder for x and then once x is filled in we can solve it!
Step17:  Session
Step20:  Traversing Operation Nodes
More details about tree post order traversal
Step21:  The result should look like
Step22:  Looks like we did it!
Step23:  Activation Function
Step24:  Sigmoid as an Operation
Step25:  Classification Example
Step26:  Defining the Perceptron
$$ y = mx + b $$
$$ y = -x + 5 $$
$$ f1 = mf2 + b , m = 1$$
$$ f1 = -f2 + 5 $$
$$ f1 + f2 - 5 = 0 $$
Convert to a Matrix Representation of Features
$$ w^Tx + b = 0 $$
$$   \Big(1, 1\Big)f - 5 = 0 $$
Then if the result is > 0 its label 1, if it is less than 0, it is label=0
Example Point
Let's say we have the point f1=2 , f2=2 otherwise stated as (8,10). Then we have
Step27:  Or if we have (4,-10)
Step28:  Using an Example Session Graph
<ASSISTANT_TASK:>
Python Code:
class SimpleClass():
    def __init__(self, str_input):
        print("SIMPLE" + str_input)
class ExtendedClass(SimpleClass):
    def __init__(self):
        print('EXTENDED')
Explanation: Manual Neural Network
In this notebook we will manually build out a neural network that mimics the TensorFlow API. This will greatly help your understanding when working with the real TensorFlow!
Quick Note on Super() and OOP
End of explanation
s = ExtendedClass()
Explanation: The child class will use its own initialization method, if not specified otherwise.
End of explanation
class ExtendedClass(SimpleClass):
    def __init__(self):
        super().__init__(" My String")
        print('EXTENDED')
s = ExtendedClass()
Explanation: If we want to use initialization from the parent class, we can do that using: 
python
super().__init__()
End of explanation
class Operation():
    
    An Operation is a node in a "Graph". TensorFlow will also use this concept of a Graph.
    
    This Operation class will be inherited by other classes that actually compute the specific
    operation, such as adding or matrix multiplication.
    
    def __init__(self, input_nodes = []):
        
        Intialize an Operation
        
        # The list of input nodes
        self.input_nodes = input_nodes 
        
        # Initialize list of nodes consuming this node's output
        self.output_nodes = [] 
        
        # For every node in the input, we append this operation (self) to the list of
        # the consumers of the input nodes
        for node in input_nodes:
            node.output_nodes.append(self)
        
        # There will be a global default graph (TensorFlow works this way)
        # We will then append this particular operation
        # Append this operation to the list of operations in the currently active default graph
        _default_graph.operations.append(self)
  
    def compute(self):
         
        This is a placeholder function. It will be overwritten by the actual specific operation
        that inherits from this class.
        
        pass
Explanation: Operation
End of explanation
class add(Operation):
    def __init__(self, x, y):
        super().__init__([x, y])
    def compute(self, x_var, y_var):
        self.inputs = [x_var, y_var]
        return x_var + y_var
Explanation: Example Operations
Addition
End of explanation
class multiply(Operation):
    def __init__(self, a, b):
        super().__init__([a, b])
    
    def compute(self, a_var, b_var):
        self.inputs = [a_var, b_var]
        return a_var * b_var
Explanation: Multiplication
End of explanation
class matmul(Operation):
    def __init__(self, a, b):
        super().__init__([a, b])
    
    def compute(self, a_mat, b_mat):
        self.inputs = [a_mat, b_mat]
        return a_mat.dot(b_mat)
Explanation: Matrix Multiplication
End of explanation
class Placeholder():
    
    A placeholder is a node that needs to be provided a value for computing the output in the Graph.
    In case of supervised learning, X (input) and Y (output) will require placeholders.
    
    def __init__(self):
        self.output_nodes = []
        _default_graph.placeholders.append(self)
Explanation: Placeholders
End of explanation
class Variable():
    
    This variable is a changeable parameter of the Graph.
    For a simple neural networks, it will be weights and biases.
    
    
    def __init__(self, initial_value = None):
        self.value = initial_value
        self.output_nodes = []
        
        _default_graph.variables.append(self)
Explanation: Variables
End of explanation
class Graph():  
    def __init__(self):
        self.operations = []
        self.placeholders = []
        self.variables = []
        
    def set_as_default(self):
        
        Sets this Graph instance as the Global Default Graph
        
        global _default_graph
        _default_graph = self
Explanation: Graph
End of explanation
g = Graph()
g.set_as_default()
print("Operations:")
print(g.operations)
print("Placeholders:")
print(g.placeholders)
print("Variables:")
print(g.variables)
A = Variable(10)
print("Operations:")
print(g.operations)
print("Placeholders:")
print(g.placeholders)
print("Variables:")
print(g.variables)
b = Variable(1)
print("Operations:")
print(g.operations)
print("Placeholders:")
print(g.placeholders)
print("Variables:")
print(g.variables)
# Will be filled out later
x = Placeholder()
print("Operations:")
print(g.operations)
print("Placeholders:")
print(g.placeholders)
print("Variables:")
print(g.variables)
y = multiply(A,x)
print("Operations:")
print(g.operations)
print("Placeholders:")
print(g.placeholders)
print("Variables:")
print(g.variables)
z = add(y, b)
print("Operations:")
print(g.operations)
print("Placeholders:")
print(g.placeholders)
print("Variables:")
print(g.variables)
Explanation: A Basic Graph
$$ z = Ax + b $$
With A=10 and b=1
$$ z = 10x + 1 $$
Just need a placeholder for x and then once x is filled in we can solve it!
End of explanation
import numpy as np
Explanation: Session
End of explanation
def traverse_postorder(operation):
     
    PostOrder Traversal of Nodes. 
    Basically makes sure computations are done in the correct order (Ax first , then Ax + b). 
    
    
    nodes_postorder = []
    def recurse(node):
        if isinstance(node, Operation):
            for input_node in node.input_nodes:
                recurse(input_node)
        nodes_postorder.append(node)
    recurse(operation)
    return nodes_postorder
class Session:
    def run(self, operation, feed_dict = {}):
         
          operation: The operation to compute
          feed_dict: Dictionary mapping placeholders to input values (the data)  
        
        
        # Puts nodes in correct order
        nodes_postorder = traverse_postorder(operation)
        print("Post Order:")
        print(nodes_postorder)
        
        for node in nodes_postorder:
            if type(node) == Placeholder:
                node.output = feed_dict[node]
                
            elif type(node) == Variable:
                node.output = node.value
                
            else: # Operation
                node.inputs = [input_node.output for input_node in node.input_nodes]
                node.output = node.compute(*node.inputs)
                
            # Convert lists to numpy arrays
            if type(node.output) == list:
                node.output = np.array(node.output)
        
        # Return the requested node value
        return operation.output
sess = Session()
result = sess.run(operation = z, 
                  feed_dict = {x : 10})
Explanation: Traversing Operation Nodes
More details about tree post order traversal: https://en.wikipedia.org/wiki/Tree_traversal#Post-order_(LRN)
End of explanation
result
10 * 10 + 1
# Running just y = Ax
# The post order should be only up to 
result = sess.run(operation = y, 
                  feed_dict = {x : 10})
result
Explanation: The result should look like:
Variable (A), Placeholder (x), Multiple operation (Ax), Variable (b), Add (Ax + b)
End of explanation
g = Graph()
g.set_as_default()
A = Variable([[10, 20], [30, 40]])
b = Variable([1, 1])
x = Placeholder()
y = matmul(A,x)
z = add(y,b)
sess = Session()
result = sess.run(operation = z,
                  feed_dict = {x : 10})
result
Explanation: Looks like we did it!
End of explanation
import matplotlib.pyplot as plt
%matplotlib inline
# Defining sigmoid function
def sigmoid(z):
    return 1 / (1 + np.exp(-z))
sample_z = np.linspace(-10, 10, 100)
sample_a = sigmoid(sample_z)
plt.figure(figsize = (8, 8))
plt.title("Sigmoid")
plt.plot(sample_z, sample_a)
Explanation: Activation Function
End of explanation
class Sigmoid(Operation):
    def __init__(self, z):
        # a is the input node
        super().__init__([z])
    def compute(self, z_val):
        return 1 / (1 + np.exp(-z_val))
Explanation: Sigmoid as an Operation
End of explanation
from sklearn.datasets import make_blobs
# Creating 50 samples divided into 2 blobs with 2 features
data = make_blobs(n_samples = 50,
                  n_features = 2,
                  centers = 2,
                  random_state = 75)
data
features = data[0]
plt.scatter(features[:,0],features[:,1])
labels = data[1]
plt.scatter(x = features[:,0], 
            y = features[:,1], 
            c = labels,
            cmap = 'coolwarm')
# DRAW A LINE THAT SEPERATES CLASSES
x = np.linspace(0, 11 ,10)
y = -x + 5
plt.scatter(features[:,0],
            features[:,1],
            c = labels,
            cmap = 'coolwarm')
plt.plot(x,y)
Explanation: Classification Example
End of explanation
z = np.array([1, 1]).dot(np.array([[8], [10]])) - 5
print(z)
a = 1 / (1 + np.exp(-z))
print(a)
Explanation: Defining the Perceptron
$$ y = mx + b $$
$$ y = -x + 5 $$
$$ f1 = mf2 + b , m = 1$$
$$ f1 = -f2 + 5 $$
$$ f1 + f2 - 5 = 0 $$
Convert to a Matrix Representation of Features
$$ w^Tx + b = 0 $$
$$   \Big(1, 1\Big)f - 5 = 0 $$
Then if the result is > 0 its label 1, if it is less than 0, it is label=0
Example Point
Let's say we have the point f1=2 , f2=2 otherwise stated as (8,10). Then we have:
$$ 
\begin{pmatrix} 
1 , 1
\end{pmatrix} 
\begin{pmatrix} 
8 \
10
\end{pmatrix} + 5 =  $$
End of explanation
z = np.array([1,1]).dot(np.array([[2],[-10]])) - 5
print(z)
a = 1 / (1 + np.exp(-z))
print(a)
Explanation: Or if we have (4,-10)
End of explanation
g = Graph()
g.set_as_default()
x = Placeholder()
w = Variable([1,1])
b = Variable(-5)
z = add(matmul(w,x),b)
a = Sigmoid(z)
sess = Session()
sess.run(operation = a,
         feed_dict = {x : [8, 10]})
sess.run(operation = a,
         feed_dict = {x : [2, -10]})
Explanation: Using an Example Session Graph
End of explanation
<END_TASK>