<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <div class="alert alert-block alert-info" style="margin-top
Step1:  <a id="ref1"></a>
<h2 align=center>Neural Network Module and Training Function </h2>
define the neural network module or class, with two hidden Layers 
<img src = "https
Step2:  define the class with the Tanh activation function
Step3:  define the class for the Relu activation function
Step4:  define a function to  train the model, in this case the function returns a Python dictionary to store the training loss and accuracy on the validation data
Step5:  <a id="ref2"></a>
<h2 align=center>Prepare Data </h2>
Load the training dataset by setting the parameters <code>train </code> to <code>True</code> and convert it to a tensor  by placing a transform object int the argument <code>transform</code>
Step6:  Load the testing dataset by setting the parameters train  <code>False</code> and convert it to a tensor  by placing a transform object int the argument <code>transform</code>
Step7:  create the criterion function
Step8:  create the training-data loader and the validation-data loader object
Step9:  <a id="ref3"></a>
<h2 align=center>Define Neural Network, Criterion function, Optimizer and Train the  Model  </h2>
create the criterion function
Step10:  create  the model with 100 hidden layers
Step11:  print the model parameters 
The epoch number in the video is 35. You can try 10 for now. If you try 35, it may take a long time.
Step12:  <a id="ref4"></a>
<h2 align=center>Test Sigmoid ,Tanh and Relu  </h2>
train the network using the Sigmoid activations function
Step13:  train the network using the Tanh activations function
Step14:  train the network using the Relu activations function
Step15:  <a id="ref5"></a>
<h2 align=center>Analyse Results </h2>
compare the training loss for each activation
Step16:  compare the validation loss for each model
<ASSISTANT_TASK:>
Python Code:
!conda install -y torchvision
import torch 
import torch.nn as nn
import torchvision.transforms as transforms
import torchvision.datasets as dsets
import torch.nn.functional as F
import matplotlib.pylab as plt
import numpy as np
torch.manual_seed(2)
Explanation: <div class="alert alert-block alert-info" style="margin-top: 20px">
 <a href="http://cocl.us/pytorch_link_top"><img src = "http://cocl.us/Pytorch_top" width = 950, align = "center"></a>
<img src = "https://ibm.box.com/shared/static/ugcqz6ohbvff804xp84y4kqnvvk3bq1g.png" width = 200, align = "center">
<h1 align=center><font size = 5>Hidden  Layer Deep Network:  Sigmoid, Tanh and Relu Activations Functions  MNIST Dataset  </font></h1> 
# Table of Contents
In this lab, you will test Sigmoid, Tanh and Relu activations functions on the MNIST dataset with with two hidden Layers. 
<div class="alert alert-block alert-info" style="margin-top: 20px">
<li><a href="#ref1">Neural Network Module and Training Function   </a></li>
<li><a href="#ref2"> Prepare Data </a></li>
<li><a href="#ref3">Define Several Neural Network, Criterion function, Optimizer</a></li>
<li><a href="#ref4">Test Sigmoid ,Tanh and Relu </a></li>
<li><a href="#ref4">Analyse Results</a></li>
<br>
<p></p>
Estimated Time Needed: <strong>25 min</strong>
</div>
<hr>
You'll need the following libraries:
End of explanation
class Net(nn.Module):
    def __init__(self,D_in,H1,H2,D_out):
        super(Net,self).__init__()
        self.linear1=nn.Linear(D_in,H1)
        self.linear2=nn.Linear(H1,H2)
        self.linear3=nn.Linear(H2,D_out)
       
    def forward(self,x):
        x=torch.sigmoid(self.linear1(x)) 
        x=torch.sigmoid(self.linear2(x))
        x=self.linear3(x)
        return x
Explanation: <a id="ref1"></a>
<h2 align=center>Neural Network Module and Training Function </h2>
define the neural network module or class, with two hidden Layers 
<img src = "https://ibm.box.com/shared/static/5wtclahun0f70qlwkn2kxzh3amnbq4zg.png" width = 200, align = "center">
End of explanation
class NetTanh(nn.Module):
    def __init__(self,D_in,H1,H2,D_out):
        super(NetTanh,self).__init__()
        self.linear1=nn.Linear(D_in,H1)
        self.linear2=nn.Linear(H1,H2)
        self.linear3=nn.Linear(H2,D_out)
        
    def forward(self,x):
        x=torch.tanh(self.linear1(x))
        x=torch.tanh(self.linear2(x))
        x=self.linear3(x)
        return x
Explanation: define the class with the Tanh activation function
End of explanation
class NetRelu(nn.Module):
    def __init__(self,D_in,H1,H2,D_out):
        super(NetRelu,self).__init__()
        self.linear1=nn.Linear(D_in,H1)
        self.linear2=nn.Linear(H1,H2)
        self.linear3=nn.Linear(H2,D_out)
      
    def forward(self,x):
        x=F.relu(self.linear1(x))  
        x=F.relu(self.linear2(x))
        x=self.linear3(x)
        return x
Explanation: define the class for the Relu activation function
End of explanation
def train(model,criterion, train_loader,validation_loader, optimizer, epochs=100):
    i=0
    useful_stuff={'training_loss':[],'validation_accuracy':[]}  
    
    #n_epochs
    for epoch in range(epochs):
        for i,(x, y) in enumerate(train_loader):
            #clear gradient 
            optimizer.zero_grad()
            #make a prediction logits 
            z=model(x.view(-1,28*28))
            # calculate loss 
            loss=criterion(z,y)
    
            # calculate gradients of parameters 
            loss.backward()
            # update parameters 
            optimizer.step()
            useful_stuff['training_loss'].append(loss.data.item())
        correct=0
        for x, y in validation_loader:
            #perform a prediction on the validation  data  
            yhat=model(x.view(-1,28*28))
            
            _,lable=torch.max(yhat,1)
            correct+=(lable==y).sum().item()
 
    
        accuracy=100*(correct/len(validation_dataset))
   
        useful_stuff['validation_accuracy'].append(accuracy)
    
    return useful_stuff
Explanation: define a function to  train the model, in this case the function returns a Python dictionary to store the training loss and accuracy on the validation data
End of explanation
train_dataset=dsets.MNIST(root='./data', train=True, download=True, transform=transforms.ToTensor())
Explanation: <a id="ref2"></a>
<h2 align=center>Prepare Data </h2>
Load the training dataset by setting the parameters <code>train </code> to <code>True</code> and convert it to a tensor  by placing a transform object int the argument <code>transform</code>
End of explanation
validation_dataset=dsets.MNIST(root='./data', train=False, download=True, transform=transforms.ToTensor())
Explanation: Load the testing dataset by setting the parameters train  <code>False</code> and convert it to a tensor  by placing a transform object int the argument <code>transform</code>
End of explanation
criterion=nn.CrossEntropyLoss()
Explanation: create the criterion function
End of explanation
train_loader=torch.utils.data.DataLoader(dataset=train_dataset,batch_size=2000,shuffle=True)
validation_loader=torch.utils.data.DataLoader(dataset=validation_dataset,batch_size=5000,shuffle=False)
Explanation: create the training-data loader and the validation-data loader object
End of explanation
criterion=nn.CrossEntropyLoss()
Explanation: <a id="ref3"></a>
<h2 align=center>Define Neural Network, Criterion function, Optimizer and Train the  Model  </h2>
create the criterion function
End of explanation
input_dim=28*28
hidden_dim1=50
hidden_dim2=50
output_dim=10
Explanation: create  the model with 100 hidden layers
End of explanation
cust_epochs = 10
Explanation: print the model parameters 
The epoch number in the video is 35. You can try 10 for now. If you try 35, it may take a long time.
End of explanation
model=Net(input_dim,hidden_dim1,hidden_dim2,output_dim)
learning_rate=0.01
optimizer=torch.optim.SGD(model.parameters(),lr=learning_rate)
training_results=train(model,criterion, train_loader,validation_loader, optimizer, epochs=cust_epochs)
Explanation: <a id="ref4"></a>
<h2 align=center>Test Sigmoid ,Tanh and Relu  </h2>
train the network using the Sigmoid activations function
End of explanation
model_Tanh=NetTanh(input_dim,hidden_dim1,hidden_dim2,output_dim)
optimizer=torch.optim.SGD(model_Tanh.parameters(),lr=learning_rate)
training_results_tanch=train(model_Tanh,criterion, train_loader,validation_loader, optimizer, epochs=cust_epochs)
Explanation: train the network using the Tanh activations function
End of explanation
modelRelu=NetRelu(input_dim,hidden_dim1,hidden_dim2,output_dim)
optimizer=torch.optim.SGD(modelRelu.parameters(),lr=learning_rate)
training_results_relu=train(modelRelu,criterion, train_loader,validation_loader, optimizer, epochs=cust_epochs)
Explanation: train the network using the Relu activations function
End of explanation
plt.plot(training_results_tanch['training_loss'],label='tanh')
plt.plot(training_results['training_loss'],label='sim')
plt.plot(training_results_relu['training_loss'],label='relu')
plt.ylabel('loss')
plt.title('training loss iterations')
plt.legend()
Explanation: <a id="ref5"></a>
<h2 align=center>Analyse Results </h2>
compare the training loss for each activation
End of explanation
plt.plot(training_results_tanch['validation_accuracy'],label='tanh')
plt.plot(training_results['validation_accuracy'],label='sigmoid')
plt.plot(training_results_relu['validation_accuracy'],label='relu') 
plt.ylabel('validation accuracy')
plt.xlabel('epochs ')   
plt.legend()
Explanation: compare the validation loss for each model
End of explanation
<END_TASK>