<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Tutorial 4. Immediate mode
In this tutorial we will talk about a cute feature about Caffe2
Step1:  Now, as we have known before, in the normal mode, when you create an operator, we are declaring it only, and nothing gets actually executed. Let's re-confirm that.
Step2:  Entering and exiting immediate mode.
Entering immediate mode is easy
Step3:  Now that you have enabled immediate mode, any operators you run will simultaneously be executed in a separate immediate workspace. Note - the main workspace that you are working on is not affected. We designed the immediate workspace to be separate from the main workspace, so that nothing in the main workspace gets polluted.
Step4:  Manually feeding blobs
But wait, you say - what if I want to create an operator that uses an input that is "declared" but not present yet? Since the immediate workspace does not have the input, we will encounter an exception
Step5:  This is because immediate mode, being completely imperative, requires any input to be used to already exist in the immediate workspace. To make the immediate mode aware of such external inputs, we can manually feed blobs to the immediate workspace.
Step6:  When is immediate mode useful?
You might want to use immediate mode when you are not very sure about the shape of the intermediate results, such as in a CNN where there are multiple convolution and pooling layers. Let's say that you are creating an MNIST convnet model but don't want to calculate the number of dimensions for the final FC layer. Here is what you might want to do.
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
from caffe2.python import cnn, core, visualize, workspace, model_helper, brew
import numpy as np
import os
core.GlobalInit(['caffe2', '--caffe2_log_level=-1'])
Explanation: Tutorial 4. Immediate mode
In this tutorial we will talk about a cute feature about Caffe2: immediate mode.
From the previous tutorials you have seen that Caffe2 declares a network, and during this declaration phase, nothing gets actually executed - it's like writing the source of a program, and "compilation/execution" only happens later.
This sometimes gets a bit tricky if we are in a researchy mind, and want to inspect typical intermediate outputs as we go. This is when the immediate mode come to help. At a high level, what the immediate mode does is to run the corresponding operators as you write them. The results live under a special workspace that can then be accessed via FetchImmediate() and FeedImmediate() runs.
Let's show some examples.
End of explanation
workspace.ResetWorkspace()
# declaration
op = core.CreateOperator("GaussianFill", [], "X", shape=[3, 5])
print('Before execution, workspace contains X: {}'
      .format(workspace.HasBlob("X")))
# execution
workspace.RunOperatorOnce(op)
print('After execution, workspace contains X: {}'
      .format(workspace.HasBlob("X")))
Explanation: Now, as we have known before, in the normal mode, when you create an operator, we are declaring it only, and nothing gets actually executed. Let's re-confirm that.
End of explanation
workspace.StartImmediate()
Explanation: Entering and exiting immediate mode.
Entering immediate mode is easy: you basically invoke workspace.StartImmediate(). Since immediate mode has quite a lot of side effects, it would be good to read through the warning message to make sure you understand the implications.
(If you don't want to see the messages, pass i_know=True to StartImmediate to suppress that.)
End of explanation
# declaration, and since we are in immediate mode, run it in the immediate workspace.
op = core.CreateOperator("GaussianFill", [], "X", shape=[3, 5])
print('Before execution, does workspace contain X? {}'
      .format(workspace.HasBlob("X")))
print('But we can access it using the Immediate related functions.'
      'Here is a list of immediate blobs:')
print(workspace.ImmediateBlobs())
print('The content is like this:')
print(workspace.FetchImmediate('X'))
# After the immediate execution, you can invoke StopImmediate() to clean up.
workspace.StopImmediate()
Explanation: Now that you have enabled immediate mode, any operators you run will simultaneously be executed in a separate immediate workspace. Note - the main workspace that you are working on is not affected. We designed the immediate workspace to be separate from the main workspace, so that nothing in the main workspace gets polluted.
End of explanation
workspace.StartImmediate(i_know=True)
op = core.CreateOperator("Relu", "X", "Y")
Explanation: Manually feeding blobs
But wait, you say - what if I want to create an operator that uses an input that is "declared" but not present yet? Since the immediate workspace does not have the input, we will encounter an exception:
End of explanation
X = np.random.randn(2, 3).astype(np.float32)
workspace.FeedImmediate("X", X)
# Now, we can safely run CreateOperator since immediate mode knows what X looks like
op = core.CreateOperator("Relu", "X", "Y")
print("Example input is:\n{}".format(workspace.FetchImmediate("X")))
print("Example output is:\n{}".format(workspace.FetchImmediate("Y")))
workspace.StopImmediate()
Explanation: This is because immediate mode, being completely imperative, requires any input to be used to already exist in the immediate workspace. To make the immediate mode aware of such external inputs, we can manually feed blobs to the immediate workspace.
End of explanation
model = model_helper.ModelHelper(name="mnist")
# Start the immediate mode.
workspace.StartImmediate(i_know=True)
data_folder = os.path.join(os.path.expanduser('~'), 'caffe2_notebooks', 'tutorial_data')
data_uint8, label = model.TensorProtosDBInput(
    [], ["data_uint8", "label"], batch_size=64,
    db=os.path.join(data_folder, 'mnist/mnist-train-nchw-leveldb'),
    db_type='leveldb')
data = model.net.Cast(data_uint8, "data", to=core.DataType.FLOAT)
data = model.net.Scale(data, data, scale=float(1./256))
data = model.net.StopGradient(data, data)
conv1 = brew.conv(model, data, 'conv1', 1, 20, 5)
pool1 = brew.max_pool(model, conv1, 'pool1', kernel=2, stride=2)
conv2 = brew.conv(model, pool1, 'conv2', 20, 50, 5)
pool2 = brew.max_pool(model, conv2, 'pool2', kernel=2, stride=2)
# What is the shape of pool2 again...?
feature_dimensions = workspace.FetchImmediate("pool2").shape[1:]
print("Feature dimensions before FC layer: {}".format(feature_dimensions))
fc3 = brew.fc(model, pool2, 'fc3', int(np.prod(feature_dimensions)), 500)
fc3 = brew.relu(model, fc3, fc3)
pred = brew.fc(model, fc3, 'pred', 500, 10)
softmax = brew.softmax(model, pred, 'softmax')
# Let's see if the dimensions are all correct:
for blob in ["data", "conv1", "pool1", "conv2", "pool2", "fc3", "pred"]:
    print("Blob {} has shape: {}".format(
          blob, workspace.FetchImmediate(blob).shape))
# Let's also visualize a sample input.
print("Sample input:")
visualize.NCHW.ShowMultiple(workspace.FetchImmediate("data"))
workspace.StopImmediate()
Explanation: When is immediate mode useful?
You might want to use immediate mode when you are not very sure about the shape of the intermediate results, such as in a CNN where there are multiple convolution and pooling layers. Let's say that you are creating an MNIST convnet model but don't want to calculate the number of dimensions for the final FC layer. Here is what you might want to do.
End of explanation
<END_TASK>