<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Neural Network
Step1:  Activation Functions
Q1. Apply relu, elu, and softplus to x.
Step2:  Q2. Apply sigmoid and tanh to x.
Step3:  Q3. Apply softmax to x.
Step4:  Q4. Apply dropout with keep_prob=.5 to x.
Step5:  Fully Connected
Q5. Apply a fully connected layer to x with 2 outputs and then an sigmoid function.
Step6:  Convolution
Q6. Apply 2 kernels of width-height (2, 2), stride 1, and same padding to x.
Step7:  Q7. Apply 3 kernels of width-height (2, 2), stride 1, dilation_rate 2 and valid padding to x.
Step8:  Q8. Apply 4 kernels of width-height (3, 3), stride 2, and same padding to x.
Step9:  Q9. Apply 4 times of kernels of width-height (3, 3), stride 2, and same padding to x, depth-wise.
Step10:  Q10. Apply 5 kernels of height 3, stride 2, and valid padding to x.
Step11:  Q11. Apply conv2d transpose with 5 kernels of width-height (3, 3), stride 2, and same padding to x.
Step12:  Q12. Apply conv2d transpose with 5 kernels of width-height (3, 3), stride 2, and valid padding to x.
Step13:  Q13. Apply max pooling and average pooling of window size 2, stride 1, and valid padding to x.
<ASSISTANT_TASK:>
Python Code:
from __future__ import print_function
import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
%matplotlib inline
from datetime import date
date.today()
author = "kyubyong. https://github.com/Kyubyong/tensorflow-exercises"
tf.__version__
np.__version__
Explanation: Neural Network
End of explanation
_x = np.linspace(-10., 10., 1000)
x = tf.convert_to_tensor(_x)
relu = ...
elu = ...
softplus = ...
with tf.Session() as sess:
    _relu, _elu, _softplus = sess.run([relu, elu, softplus])
    plt.plot(_x, _relu, label='relu')
    plt.plot(_x, _elu, label='elu')
    plt.plot(_x, _softplus, label='softplus')
    plt.legend(bbox_to_anchor=(0.5, 1.0))
    plt.show()
Explanation: Activation Functions
Q1. Apply relu, elu, and softplus to x.
End of explanation
_x = np.linspace(-10., 10., 1000)
x = tf.convert_to_tensor(_x)
sigmoid = ...
tanh = ...
with tf.Session() as sess:
    _sigmoid, _tanh = sess.run([sigmoid, tanh])
    plt.plot(_x, _sigmoid, label='sigmoid')
    plt.plot(_x, _tanh, label='tanh')
    plt.legend(bbox_to_anchor=(0.5, 1.0))
    plt.grid()
    plt.show()
Explanation: Q2. Apply sigmoid and tanh to x.
End of explanation
_x = np.array([[1, 2, 4, 8], [2, 4, 6, 8]], dtype=np.float32)
x = tf.convert_to_tensor(_x)
out = ...
with tf.Session() as sess:
    _out = sess.run(out)
    print(_out)    
    assert np.allclose(np.sum(_out, axis=-1), 1)
Explanation: Q3. Apply softmax to x.
End of explanation
_x = np.array([[1, 2, 4, 8], [2, 4, 6, 8]], dtype=np.float32)
print("_x =\n" , _x)
x = tf.convert_to_tensor(_x)
out = ...
with tf.Session() as sess:
    _out = sess.run(out)
    print("_out =\n", _out) 
Explanation: Q4. Apply dropout with keep_prob=.5 to x.
End of explanation
x = tf.random_normal([8, 10])
Explanation: Fully Connected
Q5. Apply a fully connected layer to x with 2 outputs and then an sigmoid function.
End of explanation
tf.reset_default_graph()
x = tf.random_uniform(shape=(2, 3, 3, 3), dtype=tf.float32)
filter = tf.get_variable("filter", shape=..., dtype=tf.float32, 
                    initializer=tf.random_uniform_initializer())
out = ...
init = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(init)
    _out = sess.run(out)
    print(_out.shape)
Explanation: Convolution
Q6. Apply 2 kernels of width-height (2, 2), stride 1, and same padding to x.
End of explanation
tf.reset_default_graph()
x = tf.random_uniform(shape=(4, 10, 10, 3), dtype=tf.float32)
filter = tf.get_variable("filter", shape=..., dtype=tf.float32, 
                    initializer=tf.random_uniform_initializer())
out = ...
init = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(init)
    _out = sess.run(out)
    print(_out.shape)
Explanation: Q7. Apply 3 kernels of width-height (2, 2), stride 1, dilation_rate 2 and valid padding to x.
End of explanation
tf.reset_default_graph()
x = tf.random_uniform(shape=(4, 10, 10, 5), dtype=tf.float32)
filter = tf.get_variable("filter", shape=..., dtype=tf.float32, 
                    initializer=tf.random_uniform_initializer())
out = ...
init = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(init)
    _out = sess.run(out)
    print(_out.shape)
Explanation: Q8. Apply 4 kernels of width-height (3, 3), stride 2, and same padding to x.
End of explanation
tf.reset_default_graph()
x = tf.random_uniform(shape=(4, 10, 10, 5), dtype=tf.float32)
filter = tf.get_variable("filter", shape=..., dtype=tf.float32, 
                    initializer=tf.random_uniform_initializer())
out = ...
init = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(init)
    _out = sess.run(out)
    print(_out.shape)
Explanation: Q9. Apply 4 times of kernels of width-height (3, 3), stride 2, and same padding to x, depth-wise.
End of explanation
tf.reset_default_graph()
x = tf.random_uniform(shape=(4, 10, 5), dtype=tf.float32)
filter = tf.get_variable("filter", shape=..., dtype=tf.float32, 
                    initializer=tf.random_uniform_initializer())
out = ...
init = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(init)
    _out = sess.run(out)
    print(_out.shape)
Explanation: Q10. Apply 5 kernels of height 3, stride 2, and valid padding to x.
End of explanation
tf.reset_default_graph()
x = tf.random_uniform(shape=(4, 5, 5, 4), dtype=tf.float32)
filter = tf.get_variable("filter", shape=..., dtype=tf.float32, 
                    initializer=tf.random_uniform_initializer())
shp = x.get_shape().as_list()
output_shape = ...
out = ...
init = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(init)
    _out = sess.run(out)
    print(_out.shape)
Explanation: Q11. Apply conv2d transpose with 5 kernels of width-height (3, 3), stride 2, and same padding to x.
End of explanation
tf.reset_default_graph()
x = tf.random_uniform(shape=(4, 5, 5, 4), dtype=tf.float32)
filter = tf.get_variable("filter", shape=(3, 3, 5, 4), dtype=tf.float32, 
                    initializer=tf.random_uniform_initializer())
shp = x.get_shape().as_list()
output_shape = ...
out = ...
init = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(init)
    _out = sess.run(out)
    print(_out.shape)
Explanation: Q12. Apply conv2d transpose with 5 kernels of width-height (3, 3), stride 2, and valid padding to x.
End of explanation
_x = np.zeros((1, 3, 3, 3), dtype=np.float32)
_x[0, :, :, 0] = np.arange(1, 10, dtype=np.float32).reshape(3, 3)
_x[0, :, :, 1] = np.arange(10, 19, dtype=np.float32).reshape(3, 3)
_x[0, :, :, 2] = np.arange(19, 28, dtype=np.float32).reshape(3, 3)
print("1st channel of x =\n", _x[:, :, :, 0])
print("\n2nd channel of x =\n", _x[:, :, :, 1])
print("\n3rd channel of x =\n", _x[:, :, :, 2])
x = tf.constant(_x)
maxpool = ...
avgpool = ...
with tf.Session() as sess:
    _maxpool, _avgpool = sess.run([maxpool, avgpool])
    print("\n1st channel of max pooling =\n", _maxpool[:, :, :, 0])
    print("\n2nd channel of max pooling =\n", _maxpool[:, :, :, 1])
    print("\n3rd channel of max pooling =\n", _maxpool[:, :, :, 2])
    print("\n1st channel of avg pooling =\n", _avgpool[:, :, :, 0])
    print("\n2nd channel of avg pooling =\n", _avgpool[:, :, :, 1])
    print("\n3rd channel of avg pooling =\n", _avgpool[:, :, :, 2])
Explanation: Q13. Apply max pooling and average pooling of window size 2, stride 1, and valid padding to x.
End of explanation
<END_TASK>