<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Copyright 2019 Google LLC
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https
Step1:  Model with convolution and batch norm layer definition
Step2:  Initialize all model weights with random numbers
Step3:  Dims of conv layer weights
Step4:  Dims of batch norm layer weights
Step5:  Fuse conv and batch norm weights
Step6:  Model with folded/fused convolution and batch norm layers
Step7:  Initialize model_fused with fused weights
Step8:  Validate that model and model_fused produce the same outputs
<ASSISTANT_TASK:>
Python Code:
import tensorflow as tf
import numpy as np
np.random.seed(123)
Explanation: Copyright 2019 Google LLC
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Example of folding keras conv layer with batch norm
Imports
End of explanation
epsilon=0.001
inputs = tf.keras.Input(shape=(50, 32, 5), batch_size=4)
net = inputs
net = tf.keras.layers.Conv2D(filters=2, kernel_size=(3,3))(net)
net = tf.keras.layers.BatchNormalization(epsilon=epsilon)(net)
net = tf.keras.layers.ReLU()(net)
net = tf.keras.layers.Flatten()(net)
model = tf.keras.Model(inputs, net)
model.summary()
model.layers
Explanation: Model with convolution and batch norm layer definition
End of explanation
# we will set all weights to randmom numbers so that even bias will be non zero 
# it will help to validated numerical correctness of conv and batch norm fusion
all_weights = model.get_weights()
for i in range(len(all_weights)):
  all_weights[i] = np.random.random(all_weights[i].shape)
model.set_weights(all_weights)
Explanation: Initialize all model weights with random numbers
End of explanation
ind_conv_layer = 1
assert(isinstance(model.layers[ind_conv_layer], tf.keras.layers.Conv2D))
conv_weights = model.layers[ind_conv_layer].get_weights()
print("conv weights shape " + str(conv_weights[0].shape))
print("conv bias shape " + str(conv_weights[1].shape))
Explanation: Dims of conv layer weights
End of explanation
ind_batch_norm_layer = 2
assert(isinstance(model.layers[ind_batch_norm_layer], tf.keras.layers.BatchNormalization))
bn_weights = model.layers[ind_batch_norm_layer].get_weights()
gamma = bn_weights[0]
print("gamma shape " + str(gamma.shape))
betta = bn_weights[1]
print("betta shape " + str(gamma.shape))
mean = bn_weights[2]
print("mean shape " + str(gamma.shape))
variance = bn_weights[3]
print("variance shape " + str(gamma.shape))
Explanation: Dims of batch norm layer weights
End of explanation
new_conv_weights = np.multiply(conv_weights[0], gamma) / np.sqrt(variance + epsilon)
new_bias = betta + np.multiply((conv_weights[1] - mean), gamma) / np.sqrt(variance + epsilon)
Explanation: Fuse conv and batch norm weights
End of explanation
inputs = tf.keras.Input(shape=(50, 32, 5), batch_size=4)
net = inputs
net = tf.keras.layers.Conv2D(filters=2, kernel_size=(3,3))(net)
net = tf.keras.layers.Flatten()(net)
model_fused = tf.keras.Model(inputs, net)
model_fused.summary()
Explanation: Model with folded/fused convolution and batch norm layers
End of explanation
all_weights_fused = model_fused.get_weights()
all_weights_fused[0] = new_conv_weights
all_weights_fused[1] = new_bias
model_fused.set_weights(all_weights_fused)
Explanation: Initialize model_fused with fused weights
End of explanation
input_data = np.random.random(inputs.shape)
outputs = model.predict(input_data)
outputs_fused = model_fused.predict(input_data)
np.allclose(outputs, outputs_fused)
Explanation: Validate that model and model_fused produce the same outputs
End of explanation
<END_TASK>