<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 VAEs on sparse data
The following notebook provides an example of how to load a dataset, setup parameters for it, create the model
and train it for a few epochs.
In the notebook, we will use with the RCV1 dataset (assuming it has been setup previously). For details on how to set it up, run python rcv2.py in the optvaedatasets folder
Step1:  Model Parameters
The model parameters have been saved here, we'll load them and look at them
These are what the model will be built based on
Step2:  For the moment, we will leave everything as is. Some worthwhile parameters to note
Step3:  Load dataset
Lets load the RCV1(v2) dataset and visualize how the dataset &lt;dict&gt; is structured
We'll need to append some parameters from the dataset into the default parameters dict that we will use to create the model
Also, compute the idf vectors for the entire dataset (the term frequencies will be multiplied dynamically) inside the model
Step4:  Setup
Create directory for configuration files. The configuration file for a single experiment is in the pickle file.
We will use this directory to save checkpoint files as well
Step5:  Training the model
We can now train the model we created
This is the overall setup for the file train.py
<ASSISTANT_TASK:>
Python Code:
import sys,os,glob
from collections import OrderedDict
import numpy as np
from utils.misc import readPickle, createIfAbsent
sys.path.append('../')
from optvaedatasets.load import loadDataset as loadDataset_OVAE
from sklearn.feature_extraction.text import TfidfTransformer
Explanation: VAEs on sparse data
The following notebook provides an example of how to load a dataset, setup parameters for it, create the model
and train it for a few epochs.
In the notebook, we will use with the RCV1 dataset (assuming it has been setup previously). For details on how to set it up, run python rcv2.py in the optvaedatasets folder
End of explanation
default_params = readPickle('../optvaeutils/default_settings.pkl')[0]
for k in default_params:
    print '(',k,default_params[k],')',
print
Explanation: Model Parameters
The model parameters have been saved here, we'll load them and look at them
These are what the model will be built based on
End of explanation
default_params['opt_type'] = 'finopt' #set to finopt to optimize var. params, none otherwise
default_params['n_steps'] = 5
#temporary directory where checkpoints are saved
default_params['savedir'] = './tmp'
Explanation: For the moment, we will leave everything as is. Some worthwhile parameters to note: 
* n_steps: Number of steps of optimizing $\psi(x)$, the local variational parameters as output by the inference network. We'll set this to 10 below for the moment.
* dim_stochastic: Number of latent dimensions.
End of explanation
dset = loadDataset_OVAE('rcv2')
#Visualize structure of dataset dict
for k in dset:
    print k, type(dset[k]),
    if hasattr(dset[k],'shape'):
        print dset[k].shape
    elif type(dset[k]) is not list:
        print dset[k]
    else:
        print
#Add parameters to default_params
for k in ['dim_observations','data_type']:
    default_params[k] = dset[k]
default_params['max_word_count'] =dset['train'].max()
#Create IDF
additional_attrs        = {}
tfidf                   = TfidfTransformer(norm=None) 
tfidf.fit(dset['train'])
additional_attrs['idf'] = tfidf.idf_ 
from optvaemodels.vae import VAE as Model
import optvaemodels.vae_learn as Learn
import optvaemodels.vae_evaluate as Evaluate
Explanation: Load dataset
Lets load the RCV1(v2) dataset and visualize how the dataset &lt;dict&gt; is structured
We'll need to append some parameters from the dataset into the default parameters dict that we will use to create the model
Also, compute the idf vectors for the entire dataset (the term frequencies will be multiplied dynamically) inside the model
End of explanation
default_params['savedir']+='-rcv2-'+default_params['opt_type']
createIfAbsent(default_params['savedir'])
pfile= default_params['savedir']+'/'+default_params['unique_id']+'-config.pkl'
print 'Training model from scratch. Parameters in: ',pfile
model = Model(default_params, paramFile = pfile, additional_attrs = additional_attrs)
Explanation: Setup
Create directory for configuration files. The configuration file for a single experiment is in the pickle file.
We will use this directory to save checkpoint files as well
End of explanation
savef      = os.path.join(default_params['savedir'],default_params['unique_id']) #Prefix for saving in checkpoint directory
savedata   = Learn.learn( model, 
                                dataset     = dset['train'],
                                epoch_start = 0 , 
                                epoch_end   = 3,  #epochs -- set w/ default_params['epochs'] 
                                batch_size  = default_params['batch_size'], #batch size 
                                savefreq    = default_params['savefreq'], #frequency of saving
                                savefile    = savef,
                                dataset_eval= dset['valid']
                                )
for k in savedata:
    print k, type(savedata[k]), savedata[k].shape
Explanation: Training the model
We can now train the model we created
This is the overall setup for the file train.py
End of explanation
<END_TASK>