<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Tracking iterative executions
One of the main primitives in our framework is Pipeline, which runs the same list of procedures over and over. It is essential to have a tool to notify the user on its progress; for situations where plain tqdm bar is just not enough we've developed Notifier and Monitor classes that provide fine-grained updates on the state of iterative flow. This notebooks shows how to
Step1:  Notifier
In the simplest case we use Notifier as a drop-in replacement for tqdm progress bar, and the behaviour remains the same. Positional argument bar tells which progress bar implementation to use
Step2:  As some of the loops are running hundreds of iterations per second, we should take special care of speed of updating
Step3:  Monitors
We will return to Notifier and its other arguments in just a second; for now, let's look at other feature
Step4:  Under the hood Monitor creates a separate process, that checks the state of a resource every frequency seconds and can fetch collected data on demand.
There are a number of resources that can be tracked
Step5:  This feature is immensely helpful during both research and deploy stages, so we included it in the Notifier itself
Step6:  Pipeline
As was already mentioned, one of the prime purposes of our notification system is to be used in tandem with Pipeline. We create a rather simple one with following variables
Step7:  Vanilla pipeline
The easiest way to track progress is to supply notifier argument to the run method. It works with
Step8:  Track pipeline variables
We can use notifier to track values of pipeline variables at each iteration
Step9:  Obviously, we can use the same resource monitors, as before, by passing additional items to monitors. There is also file argument, that allows us to log the progress to an external storage
Step10:  Live plots
One of the distinct features of Notifier is its ability of plotting data on the fly
Step11:  It can work with images also. As the rendering of plots might take some time, we want to do so once every 10 iterations and achieve so by using frequency parameter
Step13:  Advanced usage of Notifier
Different people prefer different types of notifications
<ASSISTANT_TASK:>
Python Code:
%load_ext autoreload
%autoreload 2
import sys
import warnings
warnings.filterwarnings("ignore")
import torch
import numpy as np
from tqdm import tqdm_notebook, tqdm
sys.path.append('../..')
from batchflow import Notifier, Pipeline, Dataset, I, W, V, L, B
from batchflow.monitor import *
# Set GPU
%env CUDA_VISIBLE_DEVICES=0
DEVICE = torch.device('cuda:0')
torch.ones((1, 1, 1), device=DEVICE)
BAR = 't' # can be changed to 'n' to use Jupyter Notebook progress bar
Explanation: Tracking iterative executions
One of the main primitives in our framework is Pipeline, which runs the same list of procedures over and over. It is essential to have a tool to notify the user on its progress; for situations where plain tqdm bar is just not enough we've developed Notifier and Monitor classes that provide fine-grained updates on the state of iterative flow. This notebooks shows how to:
track utilization for a wide variety of resources (cpu, gpu, memory, etc)
create an iteration-wise notifications for values of pipeline variables and other entities
plot graphs (for example, loss values) on the fly
End of explanation
for item in Notifier(BAR)(range(5)):
    print(item)
Explanation: Notifier
In the simplest case we use Notifier as a drop-in replacement for tqdm progress bar, and the behaviour remains the same. Positional argument bar tells which progress bar implementation to use:
t stands for plain text tqdm
n allows to make use of modern tqdm_notebook graphics
a automatically changes from text-based to GUI, depending on the environment it is being used
End of explanation
%time for item in Notifier('t')(range(100000)): pass
%time for item in tqdm(range(100000)): pass
%time for item in Notifier('n')(range(100000)): pass
%time for item in tqdm_notebook(range(100000)): pass
Explanation: As some of the loops are running hundreds of iterations per second, we should take special care of speed of updating: as we can see, the overhead is negligeable for both text and graphical versions of the Notifier:
End of explanation
with monitor_cpu(frequency=0.1) as cpu_monitor:
    for _ in Notifier(BAR)(range(10)):
        _ = np.random.random((1000, 10000))
        
cpu_monitor.visualize()
Explanation: Monitors
We will return to Notifier and its other arguments in just a second; for now, let's look at other feature: Monitors. As we intent to use our pipelines to work with enormous neural networks on gigantic datasets, we must finely control all of our resources: CPU, RAM, GPU load, etc. Monitors provide us tools to visualize resource utilization over a period of time: mere addition of a special context manager allows us to look into CPU usage in details:
End of explanation
with monitor_resource(['uss', 'gpu', 'gpu_memory'], frequency=0.1) as (uss_monitor, gpu_monitor, gpum_monitor):
    for _ in Notifier(BAR)(range(42)):
        cpu_data = np.random.random((1000, 10000))
        gpu_data = torch.ones((256, 512, 2096), device=DEVICE)
        gpu_op = torch.mvlgamma(torch.erfinv(gpu_data), 1)         # intense operation
        torch.cuda.empty_cache()
        
uss_monitor.visualize()
gpu_monitor.visualize()
gpum_monitor.visualize()
Explanation: Under the hood Monitor creates a separate process, that checks the state of a resource every frequency seconds and can fetch collected data on demand.
There are a number of resources that can be tracked:
- CPU utilization (cpu)
- RAM used by the current process (uss)
- total RAM used on the machine (memory)
- GPU utilization (gpu)
- GPU memory used (gpu_memory)
For convenience we also provide context manager to track multiple entities at once:
End of explanation
notifier = Notifier(BAR, monitors=['memory', 'cpu'])
for _ in notifier(range(100)):
    _ = np.random.random((1000, 100))
notifier.visualize()
Explanation: This feature is immensely helpful during both research and deploy stages, so we included it in the Notifier itself: it has never been so easy to always keep track of all the resources!
Note that monitors connected to an instance of Notifier provide information on the resource usage not only every frequency seconds, but also at the end of each iteration.
End of explanation
pipeline = (
    Pipeline()
    .init_variable('loss_history', [])
    .init_variable('image')
    .update(V('loss_history', mode='a'), 100 * 2 ** (-I()))
    .update(V('image'), L(np.random.random)((30, 30)))
) << Dataset(10)
Explanation: Pipeline
As was already mentioned, one of the prime purposes of our notification system is to be used in tandem with Pipeline. We create a rather simple one with following variables:
list loss_history that is updated at each iteration with a new value
2D array image that is randomly generated at each iteration
End of explanation
pipeline.reset('all')
_ = pipeline.run(1, n_iters=10, notifier=BAR)
Explanation: Vanilla pipeline
The easiest way to track progress is to supply notifier argument to the run method. It works with:
an instance of Notifier
string (t, n, a) would be used as positional argument for Notifier creation in the bowels of Pipeline
dictionary with parameters of Notifier initialization
End of explanation
pipeline.reset('all')
_ = pipeline.run(1, n_iters=10, notifier=Notifier(BAR, monitors='loss_history'))
pipeline.notifier.visualize()
Explanation: Track pipeline variables
We can use notifier to track values of pipeline variables at each iteration: conveniently, we use the same monitor argument:
End of explanation
pipeline.reset('all')
_ = pipeline.run(1, n_iters=50, notifier=Notifier(BAR, monitors=['cpu', 'loss_history'], file='notifications.txt'))
pipeline.notifier.visualize()
!head notifications.txt -n 13
Explanation: Obviously, we can use the same resource monitors, as before, by passing additional items to monitors. There is also file argument, that allows us to log the progress to an external storage:
End of explanation
pipeline.reset('all')
_ = pipeline.run(1, n_iters=10, notifier=Notifier('n', graphs=['memory', 'loss_history']))
Explanation: Live plots
One of the distinct features of Notifier is its ability of plotting data on the fly: to do so, we must supply graphs argument. Its semantics are exactly the same as those of monitors parameter.
End of explanation
pipeline.reset('all')
_ = pipeline.run(1, n_iters=100, notifier=Notifier('n', graphs=['memory', 'loss_history', 'image'], frequency=10))
Explanation: It can work with images also. As the rendering of plots might take some time, we want to do so once every 10 iterations and achieve so by using frequency parameter:
End of explanation
def custom_plotter(ax=None, container=None, **kwargs):
     Zero-out center area of the image, change plot parameters. 
    container['data'][10:20, 10:20] = 0
    
    ax.imshow(container['data'])
    ax.set_title(container['name'], fontsize=18)
    
    ax.set_xlabel('axis one', fontsize=18)
    ax.set_ylabel('axis two', fontsize=18)
pipeline.reset('all')
_ = pipeline.run(1, n_iters=100,
                 notifier=Notifier('n',
                                   graphs=[{'source': 'memory',
                                            'name': 'my custom monitor'},
                                           {'source': 'image',
                                            'name': 'amazing plot',
                                            'plot_function': custom_plotter}],
                                   frequency=10)
)
Explanation: Advanced usage of Notifier
Different people prefer different types of notifications: for some modest print is enough, some build complex systems to bring notifications to their mobile devices. Our Notifier provides you fine control on how the data is shown so you can tune it for yourself.
Previously, we passed strings as monitor/graphs parameters to identify, which exactly pipeline variable or resource should be tracked. Under the hood, each of them is parsed to a dictionary with parameters:
source determines the data generator
name used for plot titles and near bar text descriptions
plot_function can be called for custom plotting mechanism
We can leverage those parameters make all the plots to our liking:
End of explanation
<END_TASK>