<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <a href="https
Step1:  Transforming an input to a known output
Step2:  relation between input and output is linear
Step3:  Defining the model to train
untrained single unit (neuron) also outputs a line from same input, although another one 
The Artificial Neuron
Step5:  Defining a layer with a random number of neurons and inputs
Step6:  Output of a single untrained neuron
Step7:  Loss - Mean Squared Error
Loss function is the prerequisite to training. We need an objective to optimize for. We calculate the difference between what we get as output and what we would like to get.
Mean Squared Error
$MSE = {\frac {1}{n}}\sum {i=1}^{n}(Y{i}-{\hat {Y_{i}}})^{2}$
https
Step8:  Minimize Loss by changing parameters of neuron
Move in parameter space in the direction of a descent
<img src='https
Step9:  Training
Step10:  Line drawn by neuron after training
result after training is not perfect, but almost looks like the same line
https
Step11:  Prebuilt Optimizers do this job (but a bit more efficient and sohpisticated)
Step12:  More data points, more noisy
Step13:  Lines model draws over time
Initial Step
Step14:  After 500 Steps
Step15:  Final Step
Step16:  Understandinging the effect of activation functions
Typically, the output of a neuron is transformed using an activation function which compresses the output to a value between 0 and 1 (sigmoid), or between -1 and 1 (tanh) or sets all negative values to zero (relu).
<img src='https
Step17:  Logictic Regression
So far we were inferring a continous value for another, now we want to classify. Imagine we have a line that separates two categories in two dimensions.
Step19:  We compress output between 0 and 1 using sigmoid to match y
everything below 0.5 counts as 0, everthing above as 1
Step20:  We have 2d input now
Step21:  Reconsidering the loss function
cross entropy is an alternative to squared error 
cross entropy can be used as an error measure when a network's outputs can be thought of as representing independent hypotheses
activations can be understood as representing the probability that each hypothesis might be true
the loss indicates the distance between what the network believes this distribution should be, and what the teacher says it should be 
http
Step22:  The same solution using high level Keas API
<ASSISTANT_TASK:>
Python Code:
!pip install -q tf-nightly-gpu-2.0-preview
import tensorflow as tf
print(tf.__version__)
# a small sanity check, does tf seem to work ok?
hello = tf.constant('Hello TF!')
print("This works: {}".format(hello))
# this should return True even on Colab
tf.test.is_gpu_available()
tf.test.is_built_with_cuda()
!nvidia-smi
tf.executing_eagerly()
Explanation: <a href="https://colab.research.google.com/github/DJCordhose/ai/blob/master/notebooks/tf2/tf-low-level.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>
Introduction to Neural Networks with Low Level TensorFlow 2
Based on 
* This thread is a crash course on everything you need to know to use TensorFlow 2.0 + Keras for deep learning research: https://twitter.com/fchollet/status/1105139360226140160
* Colab Notebook tf.keras for Researchers: https://colab.research.google.com/drive/17u-pRZJnKN0gO5XZmq8n5A2bKGrfKEUg#scrollTo=UHOOlixcQ9Gl
* Effective TensorFlow 2: https://www.tensorflow.org/alpha/guide/effective_tf2
End of explanation
input = [[-1], [0], [1], [2], [3], [4]]
output = [[2], [1], [0], [-1], [-2], [-3]]
import matplotlib.pyplot as plt
plt.xlabel('input')
plt.ylabel('output')
plt.plot(input, output, 'ro')
Explanation: Transforming an input to a known output
End of explanation
plt.plot(input, output)
plt.plot(input, output, 'ro')
Explanation: relation between input and output is linear
End of explanation
w = tf.constant([[1.5], [-2], [1]], dtype='float32')
x = tf.constant([[10, 6, 8]], dtype='float32')
b = tf.constant([6], dtype='float32')
y = tf.matmul(x, w) + b
print(y)
Explanation: Defining the model to train
untrained single unit (neuron) also outputs a line from same input, although another one 
The Artificial Neuron: Foundation of Deep Neural Networks (simplified, more later)
a neuron takes a number of numerical inputs
multiplies each with a weight, sums up all weighted input and  
adds bias (constant) to that sum
from this it creates a single numerical output
for one input (one dimension) this would be a description of a line
for more dimensions this describes a hyper plane that can serve as a decision boundary
this is typically expressed as a matrix multplication plus an addition
<img src='https://djcordhose.github.io/ai/img/insurance/neuron211.jpg'>
This can be expressed using a matrix multiplication
End of explanation
from tensorflow.keras.layers import Layer
class LinearLayer(Layer):
  y = w.x + b
  def __init__(self, units=1, input_dim=1):
      super(LinearLayer, self).__init__()
      w_init = tf.random_normal_initializer(stddev=2)
      self.w = tf.Variable(
          initial_value = w_init(shape=(input_dim, units), dtype='float32'),
          trainable=True)
      b_init = tf.zeros_initializer()
      self.b = tf.Variable(
          initial_value = b_init(shape=(units,), dtype='float32'),
          trainable=True)
  def call(self, inputs):
      return tf.matmul(inputs, self.w) + self.b
    
linear_layer = LinearLayer()
Explanation: Defining a layer with a random number of neurons and inputs
End of explanation
x = tf.constant(input, dtype=tf.float32)
y_true = tf.constant(output, dtype=tf.float32)
y_true
y_pred = linear_layer(x)
y_pred
plt.plot(x, y_pred)
plt.plot(input, output, 'ro')
Explanation: Output of a single untrained neuron
End of explanation
loss_fn = tf.losses.mean_squared_error
# loss_fn = tf.losses.mean_absolute_error
loss = loss_fn(y_true=tf.squeeze(y_true), y_pred=tf.squeeze(y_pred))
print(loss)
tf.keras.losses.mean_squared_error == tf.losses.mean_squared_error
Explanation: Loss - Mean Squared Error
Loss function is the prerequisite to training. We need an objective to optimize for. We calculate the difference between what we get as output and what we would like to get.
Mean Squared Error
$MSE = {\frac {1}{n}}\sum {i=1}^{n}(Y{i}-{\hat {Y_{i}}})^{2}$
https://en.wikipedia.org/wiki/Mean_squared_error
End of explanation
# a simple example
# f(x) = x^2
# f'(x) = 2x
# x = 4
# f(4) = 16
# f'(4) = 8 (that's what we expect)
def tape_sample():
  x = tf.constant(4.0)
  # open a GradientTape
  with tf.GradientTape() as tape:
    tape.watch(x)
    y = x * x
  dy_dx = tape.gradient(y, x)
  print(dy_dx)
  
# just a function in order not to interfere with x on the global scope  
tape_sample()
Explanation: Minimize Loss by changing parameters of neuron
Move in parameter space in the direction of a descent
<img src='https://djcordhose.github.io/ai/img/gradients.jpg'>
https://twitter.com/colindcarroll/status/1090266016259534848
Job of the optimizer
<img src='https://djcordhose.github.io/ai/img/manning/optimizer.png' height=500>
For this we need partial derivations
TensorFlow offers automatic differentiation: https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/GradientTape
tape will record operations for automatic differentiation
either by making it record explicily (watch) or 
by declaring a varible to be trainable (which we did in the layer above)
End of explanation
linear_layer = LinearLayer()
linear_layer.w, linear_layer.b
linear_layer.trainable_weights
EPOCHS = 200
learning_rate = 1e-2
losses = []
weights = []
biases = []
weights_gradient = []
biases_gradient = []
for step in range(EPOCHS):
  with tf.GradientTape() as tape:
    # forward pass
    y_pred = linear_layer(x)
    # loss value for this batch
    loss = loss_fn(y_true=tf.squeeze(y_true), y_pred=tf.squeeze(y_pred))
    
  # just for logging
  losses.append(loss.numpy())
  weights.append(linear_layer.w.numpy()[0][0])
  biases.append(linear_layer.b.numpy()[0])
  # get gradients of weights wrt the loss
  gradients = tape.gradient(loss, linear_layer.trainable_weights)
  weights_gradient.append(gradients[0].numpy()[0][0])
  biases_gradient.append(gradients[1].numpy()[0])
  
  # backward pass, changing trainable weights
  linear_layer.w.assign_sub(learning_rate * gradients[0])
  linear_layer.b.assign_sub(learning_rate * gradients[1])
print(loss)
plt.xlabel('epochs')
plt.ylabel('loss')
# plt.yscale('log')
plt.plot(losses)
plt.figure(figsize=(20, 10))
plt.plot(weights)
plt.plot(biases)
plt.plot(weights_gradient)
plt.plot(biases_gradient)
plt.legend(['slope', 'offset', 'gradient slope', 'gradient offset'])
Explanation: Training
End of explanation
y_pred = linear_layer(x)
y_pred
plt.plot(x, y_pred)
plt.plot(input, output, 'ro')
# single neuron and single input: one weight and one bias
# slope m ~ -1
# y-axis offset y0 ~ 1
# https://en.wikipedia.org/wiki/Linear_equation#Slope%E2%80%93intercept_form
linear_layer.trainable_weights
Explanation: Line drawn by neuron after training
result after training is not perfect, but almost looks like the same line
https://en.wikipedia.org/wiki/Linear_equation#Slope%E2%80%93intercept_form
End of explanation
optimizer = tf.keras.optimizers.SGD(learning_rate=1e-2)
EPOCHS = 500
losses = []
linear_layer = LinearLayer()
for step in range(EPOCHS):
  with tf.GradientTape() as tape:
    # Forward pass.
    y_pred = linear_layer(x)
    # Loss value for this batch.
    loss = loss_fn(y_true=tf.squeeze(y_true), y_pred=tf.squeeze(y_pred))
    
  losses.append(loss)
     
  # Get gradients of weights wrt the loss.
  gradients = tape.gradient(loss, linear_layer.trainable_weights)
  
  # Update the weights of our linear layer.
  optimizer.apply_gradients(zip(gradients, linear_layer.trainable_weights))
# plt.yscale('log')
plt.ylabel("loss")
plt.xlabel("epochs")
plt.plot(losses)
y_pred = linear_layer(x)
plt.plot(x, y_pred)
plt.plot(input, output, 'ro')
linear_layer.trainable_weights
Explanation: Prebuilt Optimizers do this job (but a bit more efficient and sohpisticated)
End of explanation
import numpy as np
a = -1
b = 1
n = 50
x = tf.constant(np.random.uniform(0, 1, n), dtype='float32')
y = tf.constant(a*x+b + 0.1 * np.random.normal(0, 1, n), dtype='float32')
plt.scatter(x, y)
x = tf.reshape(x, (n, 1))
y_true = tf.reshape(y, (n, 1))
linear_layer = LinearLayer()
a = linear_layer.w.numpy()[0][0]
b = linear_layer.b.numpy()[0]
def plot_line(a, b, x, y_true):
  fig, ax = plt.subplots()
  y_pred = a * x + b
  
  line = ax.plot(x, y_pred)
  ax.plot(x, y_true, 'ro')
  return fig, line
plot_line(a, b, x, y_true)
# the problem is a little bit harder, train for a little longer
EPOCHS = 2000
losses = []
lines = []
linear_layer = LinearLayer()
for step in range(EPOCHS):
  # Open a GradientTape.
  with tf.GradientTape() as tape:
    # Forward pass.
    y_pred = linear_layer(x)
    # Loss value for this batch.
    loss = loss_fn(y_true=tf.squeeze(y_true), y_pred=tf.squeeze(y_pred))
    
  losses.append(loss)
  
  a = linear_layer.w.numpy()[0][0]
  b = linear_layer.b.numpy()[0]
  lines.append((a, b))
     
  # Get gradients of weights wrt the loss.
  gradients = tape.gradient(loss, linear_layer.trainable_weights)
  
  # Update the weights of our linear layer.
  optimizer.apply_gradients(zip(gradients, linear_layer.trainable_weights))
print(loss)
# plt.yscale('log')
plt.ylabel("loss")
plt.xlabel("epochs")
plt.plot(losses)
Explanation: More data points, more noisy
End of explanation
a, b = lines[0]
plot_line(a, b, x, y_true)
Explanation: Lines model draws over time
Initial Step
End of explanation
a, b = lines[500]
plot_line(a, b, x, y_true)
Explanation: After 500 Steps
End of explanation
a, b = lines[1999]
plot_line(a, b, x, y_true)
Explanation: Final Step
End of explanation
import numpy as np
x = tf.reshape(tf.constant(np.arange(-1, 4, 0.1), dtype='float32'), (50, 1))
y_pred = linear_layer(x)
plt.figure(figsize=(20, 10))
plt.plot(x, y_pred)
y_pred_relu = tf.nn.relu(y_pred)
plt.plot(x, y_pred_relu)
y_pred_sigmoid = tf.nn.sigmoid(y_pred)
plt.plot(x, y_pred_sigmoid)
y_pred_tanh = tf.nn.tanh(y_pred)
plt.plot(x, y_pred_tanh)
plt.plot(input, output, 'ro')
plt.legend(['no activation', 'relu', 'sigmoid', 'tanh'])
Explanation: Understandinging the effect of activation functions
Typically, the output of a neuron is transformed using an activation function which compresses the output to a value between 0 and 1 (sigmoid), or between -1 and 1 (tanh) or sets all negative values to zero (relu).
<img src='https://raw.githubusercontent.com/DJCordhose/deep-learning-crash-course-notebooks/master/img/neuron.jpg'>
Typical Activation Functions
<img src='https://djcordhose.github.io/ai/img/activation-functions.jpg'>
End of explanation
from matplotlib.colors import ListedColormap
a = -1
b = 1
n = 100
# all points
X = np.random.uniform(0, 1, (n, 2))
# our line
line_x = np.random.uniform(0, 1, n)
line_y = a*line_x+b
plt.plot(line_x, line_y, 'r')
# below and above line
y = X[:, 1] > a*X[:, 0]+b
y = y.astype(int)
plt.xlabel("x1")
plt.ylabel("x2")
plt.scatter(X[:,0], X[:,1], c=y, cmap=ListedColormap(['#AA6666', '#6666AA']), marker='o', edgecolors='k')
y
Explanation: Logictic Regression
So far we were inferring a continous value for another, now we want to classify. Imagine we have a line that separates two categories in two dimensions.
End of explanation
class SigmoidLayer(LinearLayer):
  y = sigmoid(w.x + b)
  def __init__(self, **kwargs):
      super(SigmoidLayer, self).__init__(**kwargs)
  def call(self, inputs):
      return tf.sigmoid(super().call(inputs))
Explanation: We compress output between 0 and 1 using sigmoid to match y
everything below 0.5 counts as 0, everthing above as 1
End of explanation
x = tf.constant(X, dtype='float32')
y_true = tf.constant(y, dtype='float32')
x.shape
model = SigmoidLayer(input_dim=2)
Explanation: We have 2d input now
End of explanation
loss_fn = tf.losses.binary_crossentropy
# standard optimizer using advanced properties
optimizer = tf.keras.optimizers.Adam(learning_rate=1e-1)
# https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/metrics/Accuracy
m = tf.keras.metrics.Accuracy()
EPOCHS = 1000
losses = []
accuracies = []
for step in range(EPOCHS):
  # Open a GradientTape.
  with tf.GradientTape() as tape:
    # Forward pass.
    y_pred = model(x)
    # Loss value for this batch.
    loss = loss_fn(y_true=tf.squeeze(y_true), y_pred=tf.squeeze(y_pred))
  y_pred_binary = (tf.squeeze(y_pred) > 0.5).numpy().astype(float)
  m.update_state(tf.squeeze(y_true), y_pred_binary)
  accuracy = m.result().numpy()
  losses.append(loss)
  accuracies.append(accuracy)
     
  # Get gradients of weights wrt the loss.
  gradients = tape.gradient(loss, model.trainable_weights)
  
  # Update the weights of our linear layer.
  optimizer.apply_gradients(zip(gradients, model.trainable_weights))
print(loss)
print(accuracy)
plt.yscale('log')
plt.ylabel("loss")
plt.xlabel("epochs")
plt.plot(losses)
plt.ylabel("accuracy")
plt.xlabel("epochs")
plt.plot(accuracies)
y_pred = model(x)
y_pred_binary = (tf.squeeze(y_pred) > 0.5).numpy().astype(float)
y_pred_binary
y_true - y_pred_binary
# below and above line
plt.xlabel("x1")
plt.ylabel("x2")
plt.scatter(X[:,0], X[:,1], c=y_pred_binary, cmap=ListedColormap(['#AA6666', '#6666AA']), marker='o', edgecolors='k')
Explanation: Reconsidering the loss function
cross entropy is an alternative to squared error 
cross entropy can be used as an error measure when a network's outputs can be thought of as representing independent hypotheses
activations can be understood as representing the probability that each hypothesis might be true
the loss indicates the distance between what the network believes this distribution should be, and what the teacher says it should be 
http://www.cse.unsw.edu.au/~billw/cs9444/crossentropy.html
End of explanation
from tensorflow.keras.layers import Dense
 
model = tf.keras.Sequential()
model.add(Dense(units=1, activation='sigmoid', input_dim=2))
model.summary()
%%time 
model.compile(loss=loss_fn, # binary cross entropy, unchanged from low level example
             optimizer=optimizer, # adam, unchanged from low level example
             metrics=['accuracy'])
# does a similar thing internally as our loop from above
history = model.fit(x, y_true, epochs=EPOCHS, verbose=0)
loss, accuracy = model.evaluate(x, y_true)
loss, accuracy
plt.yscale('log')
plt.ylabel("accuracy")
plt.xlabel("epochs")
plt.plot(history.history['accuracy'])
plt.yscale('log')
plt.ylabel("loss")
plt.xlabel("epochs")
plt.plot(history.history['loss'])
y_pred = model.predict(x)
y_pred_binary = (tf.squeeze(y_pred) > 0.5).numpy().astype(float)
# below and above line
plt.xlabel("x1")
plt.ylabel("x2")
plt.scatter(X[:,0], X[:,1], c=y_pred_binary, cmap=ListedColormap(['#AA6666', '#6666AA']), marker='o', edgecolors='k')
Explanation: The same solution using high level Keas API
End of explanation
<END_TASK>