<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Paramz Tutorial
A simple introduction into Paramz based gradient based optimization of parameterized models.
Paramz is a python based parameterized modelling framework, that handles parameterization, printing, randomizing and many other parameter based operations to be done to a parameterized model.
In this example we will make use of the rosenbrock function of scipy. We will write a paramz model calling the scipy rosen function as an objective function and its gradients and use it to show the features of Paramz.
Step1:  The starting position of the rosen function is set to be 
$$ x_0 = [-1,1] $$
Step2:  For paramz to understand your model there is three steps involved
Step3:  The class created above only holds the information about the parameters, we still have to implement the objective function to optimize over. For now the class can be instantiated but is not functional yet.
Step4:  Step Two
Step5:  Step Three
Step6:  Model Usage
Having implemented a paramz model with its necessary functions, the whole set of functionality of paramz is available for us. We will instantiate a rosen model class to play around with.
Step7:  This rosen model is a fully working parameterized model for gradient based optimization of the rosen function of scipy.
Printing and Naming
All Parameterized and Param objects are named and can be accessed by name. This ensures a cleaner model creation and printing, when big models are created. In our simple example we only have a position and the model name itself
Step8:  Or use the notebook representation
Step9:  Note the model just printing the shape (in the value column) of the parameters, as parameters can be any sized arrays or matrices (with arbitrary numbers of dimensions). 
We can print the actual values of the parameters directly, either by programmatically assigned variable
Step10:  Or by name
Step11:  We can redefine the name freely, as long as it does not exist already
Step12:  Now r.position will not be accessible anymore!
Step13:  Setting Parameters and Automated Updates
Param objects represent the parameters for the model. We told the model in the initialization that the position parameter (re-)named pos is a parameter of the model. Thus the model will listen to changes of the parameter values and update on any changes. We will set one element of the parameter and see what happens to the model
Step14:  Note that we never actually told the model to update. It listened to changes to any of its parameters and updated accordingly. This update chain is based on the hierarchy of the model structure. Specific values of parameters can be accessed through indexing, just like indexing numpy arrays. In fact Param is a derivative of ndarray and inherits all its traits. Thus, Param can be used in any calculation involved with numpy. Importantly, when using a Param parameter inside a computation, it will be returning a normal numpy array. This prevents unwanted side effects and pointer errors.
Step15:  Optimization
The optimization routine for the model can be accessed by the optimize() function. A call to optimize will setup the optimizer, do the iteration through getting and setting the parameters in an optimal 'in memory' fashion. By supplying messages=1 as an optional parameter we can print the progress of the optimization itself.
Step16:  To show the values of the positions itself, we directly print the Param object
Step17:  We could also randomize the model by using the convenience function randomize(), on the part we want to randomize. It can be any part of the model, also the whole model can be randomized
Step18:  Gradient Checking
Importantly when implementing gradient based optimization is to make sure, that the gradients implemented match the numerical gradients of the objective function. This can be achieved using the checkgrad() function in paramz. It does a triangle numerical gradient estimate around the current position of the parameter. The verbosity of the gradient checker can be adjusted using the verbose option. If verbose is False, only one bool will be returned, specifying whether the gradients check the numerical gradients or not. The option of verbose=True returns a full list of every parameter, checking each parameter individually. This can be called on each subpart of the model again. 
Here we can either directly call it on the parameter
Step19:  Or on the whole model (verbose or not)
Step20:  Or on individual parameters, note that numpy indexing is used
Step21:  Constraining Parameter Spaces
In many optimization scenarios it is necessary to constrain parameters to only take on certain ranges of values, may it be bounded in a region (between two numbers), fixed or constrained to only be positive or negative numbers. This can be achieved in paramz by applying a transformation to a parameter. For convenience the most common constraints are placed in specific functions, found by r.constrain_&lt;tab&gt;
Step22:  The printing will contain the constraints, either directly on the object, or it lists the constraints contained within a parameter. If a parameter has multiple constraints spread across the Param object all constraints contained in the whole Param object are indicated with {&lt;partial constraint&gt;}
Step23:  To show the individual constraints, we look at the Param object of interest directly
Step24:  The constraints (and other indexed properties) are held by each parameter as a dictionary with the name. For example the constraints are held in a constraints dictionary, where the keys are the constraints, and the values are the indices this constraint refers to. You can either ask for the constraints of the whole model
Step25:  Or the constraints of individual Parameterized objects
Step26:  The constraints of subparts of the model are only views into the actual constaints held by the root of the model hierarchy.
Models Inside Models
The hierarchy of a Paramz model is a tree, where the nodes of the tree are Parameterized objects and the leaves are Param objects. The Model class is Parameterized itself and, thus can serve as a child itself. This opens the possibility for combining models together in a bigger model. As a simple example, we will just add two rosen models together into a single model
Step27:  The keen eyed will have noticed, that we did not set any gradients in the above definition. That is because the underlying rosen models handle their gradients directly!
Step28:  All options listed above are availible for this model now. No additional steps need to be taken!
Step29:  To show the different ways of how constraints are displayed, we constrain different parts of the model and fix parts of it too
Step30:  First, we can see, that because two models with the same name were added to dr, the framework renamed the second model to have a unique name. This only happens when two childs of one parameter share the same name. If the two childs not under the same parameter share names, it is just fine, as you can see in the name of x in both models
Step31:  Or print only one model
Step32:  We can showcase that constraints are mapped to each parameter directly. We can either access the constraints of the whole model directly
Step33:  Or for parameters directly
Step34:  Note, that the constraints are remapped to directly index the parameters locally. This directly leeds up to the in memory handling of parameters. The root node of the hierarchy holds one parameter array param_array comprising all parameters. The same goes for the gradient gradient
Step35:  Each child parameter (and subsequent parameters) have their own view into the memory of the root node
Step36:  When changing the param_array of a parameter it directly edits the memory of the root node. This is a big part of the optimization of paramz, as getting and setting parameters works directly in memory and does not need any python routines (such as loops or traversal) functionality. 
The constraints as described above, directly index the param_array of their Parameterized or Param object. That is why the remapping exists. 
This param_array has its counterpart for the optimizer, which holds the remapped parameters by the constraints. The constraints are transformation mappings, which transform model parameters param_array into optimizer parameters optimizer_array. This optimizer array is presented to the optimizer and the constraints framework handles the mapping directly.
Step37:  Note, that the optimizer array does only contain three values. This is because the first element of the the first rosen model is fixed and is not presented to the optimizer. The transformed gradients can be computed by the root node directly
<ASSISTANT_TASK:>
Python Code:
import paramz, numpy as np
from scipy.optimize import rosen_der, rosen
Explanation: Paramz Tutorial
A simple introduction into Paramz based gradient based optimization of parameterized models.
Paramz is a python based parameterized modelling framework, that handles parameterization, printing, randomizing and many other parameter based operations to be done to a parameterized model.
In this example we will make use of the rosenbrock function of scipy. We will write a paramz model calling the scipy rosen function as an objective function and its gradients and use it to show the features of Paramz.
End of explanation
x = np.array([-1,1])
Explanation: The starting position of the rosen function is set to be 
$$ x_0 = [-1,1] $$
End of explanation
class Rosen(paramz.Model): # Inherit from paramz.Model to ensure all model functionality.
    def __init__(self, x, name='rosen'): # Initialize the Rosen model with a numpy array `x` and name `name`.
        super(Rosen, self).__init__(name=name) # Call to super to make sure the structure is set up.
        self.x = paramz.Param('position', x) # setup a Param object for the position parameter.
        self.link_parameter(self.x) # Tell the model that the parameter `x` exists.
Explanation: For paramz to understand your model there is three steps involved:
Step One: Initialization of the Model
Initialize your model using the __init__() function. The init function contains a call to the super class to make sure paramz can setup the model structure. Then we setup the parameters contained for this model and lastly we tell the model that we have those parameters by linking them to self.
End of explanation
r = Rosen(x)
try:
    print(r)
except NotImplementedError as e:
    print(e)
Explanation: The class created above only holds the information about the parameters, we still have to implement the objective function to optimize over. For now the class can be instantiated but is not functional yet.
End of explanation
class Rosen(paramz.Model):
    def __init__(self, x, name='rosen'):
        super(Rosen, self).__init__(name=name)
        self.x = paramz.Param('position', x)
        self.link_parameter(self.x)
    def objective_function(self): # The function to overwrite for the framework to know about the objective to optimize
        return rosen(self.x) # Call the rosenbrock function of scipy as objective function.
Explanation: Step Two: Adding the Objective Function
The optimization of a gradient based mathematical model is based on an objective function to optimize over. The paramz framework expects the objective_function to be overwridden, returning the current objective of the model. It can make use of all parameters inside the model and you can rely on the parameters to be updated when the objective function is called. This function does not take any parameters.
End of explanation
class Rosen(paramz.Model):
    def __init__(self, x, name='rosen'):
        super(Rosen, self).__init__(name=name)
        self.x = paramz.Param('position', x)
        self.link_parameter(self.x)
    def objective_function(self):
        return self._obj
    def parameters_changed(self): # Overwrite the parameters_changed function for model updates
        self._obj = rosen(self.x) # Lazy evaluation of the rosen function only when there is an update
        self.x.gradient[:] = rosen_der(self.x) # Compuataion and storing of the gradients for the position parameter
Explanation: Step Three: Adding Update Routine for Parameter Changes
This model is now functional, except optimization. The gradients are not initialized and an optimization will stagnate, as there are no gradients to consider. The optimization of parameters requires the gradients of the parameters to be updated. For this, we provide an inversion of control based approach, in which to update parameters and set gradients of parameters. The gradients for parameters are saved in the gradient of the parameter itself. The model handles the distribution and collection of correct gradients to the optimizer itself. 
To implement the parameters_changed(self) function we overwrite the function on the class. This function has the expensive bits of computation in it, as it is only being called if an update is absolutely necessary. We also compute the objective for the current parameter set and store it as a variable, so that a call to objective_function() can be done in a lazy function and to prevent computational overhead:
End of explanation
r = Rosen(x)
Explanation: Model Usage
Having implemented a paramz model with its necessary functions, the whole set of functionality of paramz is available for us. We will instantiate a rosen model class to play around with.
End of explanation
print(r)
Explanation: This rosen model is a fully working parameterized model for gradient based optimization of the rosen function of scipy.
Printing and Naming
All Parameterized and Param objects are named and can be accessed by name. This ensures a cleaner model creation and printing, when big models are created. In our simple example we only have a position and the model name itself: rosen.
End of explanation
r
Explanation: Or use the notebook representation:
End of explanation
r.x 
Explanation: Note the model just printing the shape (in the value column) of the parameters, as parameters can be any sized arrays or matrices (with arbitrary numbers of dimensions). 
We can print the actual values of the parameters directly, either by programmatically assigned variable
End of explanation
r.position
Explanation: Or by name:
End of explanation
r.x.name = 'pos'
r
Explanation: We can redefine the name freely, as long as it does not exist already:
End of explanation
try:
    r.position
except AttributeError as v:
    print("Attribute Error: " + str(v))
Explanation: Now r.position will not be accessible anymore!
End of explanation
print("Objective before change: {}".format(r._obj))
r.x[0] = 1
print("Objective after change: {}".format(r._obj))
Explanation: Setting Parameters and Automated Updates
Param objects represent the parameters for the model. We told the model in the initialization that the position parameter (re-)named pos is a parameter of the model. Thus the model will listen to changes of the parameter values and update on any changes. We will set one element of the parameter and see what happens to the model:
End of explanation
2 * r.x
Explanation: Note that we never actually told the model to update. It listened to changes to any of its parameters and updated accordingly. This update chain is based on the hierarchy of the model structure. Specific values of parameters can be accessed through indexing, just like indexing numpy arrays. In fact Param is a derivative of ndarray and inherits all its traits. Thus, Param can be used in any calculation involved with numpy. Importantly, when using a Param parameter inside a computation, it will be returning a normal numpy array. This prevents unwanted side effects and pointer errors.
End of explanation
r.x[:] = [100,5] # Set to a difficult starting position to show the messages of the optimization.
r.optimize(messages=1) # Call the optimization and show the progress.
Explanation: Optimization
The optimization routine for the model can be accessed by the optimize() function. A call to optimize will setup the optimizer, do the iteration through getting and setting the parameters in an optimal 'in memory' fashion. By supplying messages=1 as an optional parameter we can print the progress of the optimization itself.
End of explanation
r.x
Explanation: To show the values of the positions itself, we directly print the Param object:
End of explanation
np.random.seed(100)
r.randomize()
r.x
r.x.randomize()
r.x
Explanation: We could also randomize the model by using the convenience function randomize(), on the part we want to randomize. It can be any part of the model, also the whole model can be randomized:
End of explanation
r.x.checkgrad(verbose=1)
Explanation: Gradient Checking
Importantly when implementing gradient based optimization is to make sure, that the gradients implemented match the numerical gradients of the objective function. This can be achieved using the checkgrad() function in paramz. It does a triangle numerical gradient estimate around the current position of the parameter. The verbosity of the gradient checker can be adjusted using the verbose option. If verbose is False, only one bool will be returned, specifying whether the gradients check the numerical gradients or not. The option of verbose=True returns a full list of every parameter, checking each parameter individually. This can be called on each subpart of the model again. 
Here we can either directly call it on the parameter:
End of explanation
r.checkgrad()
r.checkgrad(verbose=1)
Explanation: Or on the whole model (verbose or not):
End of explanation
r.x[[0]].checkgrad(verbose=1)
Explanation: Or on individual parameters, note that numpy indexing is used:
End of explanation
r.x[[0]].constrain_bounded(-10,-1)
r.x[[1]].constrain_positive()
Explanation: Constraining Parameter Spaces
In many optimization scenarios it is necessary to constrain parameters to only take on certain ranges of values, may it be bounded in a region (between two numbers), fixed or constrained to only be positive or negative numbers. This can be achieved in paramz by applying a transformation to a parameter. For convenience the most common constraints are placed in specific functions, found by r.constrain_&lt;tab&gt;:
Each parameter can be constrain individually, by subindexing the Param object or Parameterized objects as a whole. Note that indexing functions like numpy indexing, so we need to make sure to keep the array structure when indexing singular elements. Next we bound $x_0$ to be constrained between $-10$ and $-1$ and $x_1$ to be constrained to only positive values:
End of explanation
r
Explanation: The printing will contain the constraints, either directly on the object, or it lists the constraints contained within a parameter. If a parameter has multiple constraints spread across the Param object all constraints contained in the whole Param object are indicated with {&lt;partial constraint&gt;}:
End of explanation
r.x
Explanation: To show the individual constraints, we look at the Param object of interest directly:
End of explanation
list(r.constraints.items())
Explanation: The constraints (and other indexed properties) are held by each parameter as a dictionary with the name. For example the constraints are held in a constraints dictionary, where the keys are the constraints, and the values are the indices this constraint refers to. You can either ask for the constraints of the whole model:
End of explanation
list(r.x.constraints.items())
Explanation: Or the constraints of individual Parameterized objects:
End of explanation
class DoubleRosen(paramz.Model):
    def __init__(self, x1, x2, name='silly_double'):
        super(DoubleRosen, self).__init__(name=name) # Call super to initiate the structure of the model
        self.r1 = Rosen(x1) # Instantiate the underlying Rosen classes
        self.r2 = Rosen(x2)
        # Tell this model, which parameters it has. Models are just the same as parameters:
        self.link_parameters(self.r1, self.r2) 
    def objective_function(self):
        return self._obj # Lazy evaluation of the objective
    def parameters_changed(self):
        self._obj = self.r1._obj + self.r2._obj # Just add both objectives together to optimize both models.
Explanation: The constraints of subparts of the model are only views into the actual constaints held by the root of the model hierarchy.
Models Inside Models
The hierarchy of a Paramz model is a tree, where the nodes of the tree are Parameterized objects and the leaves are Param objects. The Model class is Parameterized itself and, thus can serve as a child itself. This opens the possibility for combining models together in a bigger model. As a simple example, we will just add two rosen models together into a single model:
End of explanation
dr = DoubleRosen(np.random.normal(size=2), np.random.normal(size=2))
Explanation: The keen eyed will have noticed, that we did not set any gradients in the above definition. That is because the underlying rosen models handle their gradients directly!
End of explanation
dr.checkgrad(verbose=1)
Explanation: All options listed above are availible for this model now. No additional steps need to be taken!
End of explanation
dr.r1.constrain_negative()
dr.r1.x[[0]].fix()
dr.r2.x[[1]].constrain_bounded(-30, 5)
dr.r2.x[[0]].constrain_positive()
dr
Explanation: To show the different ways of how constraints are displayed, we constrain different parts of the model and fix parts of it too:
End of explanation
dr.r2.checkgrad(verbose=1)
Explanation: First, we can see, that because two models with the same name were added to dr, the framework renamed the second model to have a unique name. This only happens when two childs of one parameter share the same name. If the two childs not under the same parameter share names, it is just fine, as you can see in the name of x in both models: position.
Second, the constraints are displayed in curly brackets {} if they do not span all underlying parameters. If a constraint, however, spans all parameters, it is shown without curly brackets, such as -ve for the first rosen model.
We can now just like before perform all actions paramz support on this model, as well as on sub models. For example we can check the gradients of only one part of the model:
End of explanation
dr.r1
Explanation: Or print only one model:
End of explanation
print(dr.constraints)
Explanation: We can showcase that constraints are mapped to each parameter directly. We can either access the constraints of the whole model directly:
End of explanation
print(dr.r2.constraints)
Explanation: Or for parameters directly:
End of explanation
dr.param_array
Explanation: Note, that the constraints are remapped to directly index the parameters locally. This directly leeds up to the in memory handling of parameters. The root node of the hierarchy holds one parameter array param_array comprising all parameters. The same goes for the gradient gradient:
End of explanation
dr.r2.param_array
Explanation: Each child parameter (and subsequent parameters) have their own view into the memory of the root node:
End of explanation
print(dr.param_array)
print(dr.optimizer_array)
Explanation: When changing the param_array of a parameter it directly edits the memory of the root node. This is a big part of the optimization of paramz, as getting and setting parameters works directly in memory and does not need any python routines (such as loops or traversal) functionality. 
The constraints as described above, directly index the param_array of their Parameterized or Param object. That is why the remapping exists. 
This param_array has its counterpart for the optimizer, which holds the remapped parameters by the constraints. The constraints are transformation mappings, which transform model parameters param_array into optimizer parameters optimizer_array. This optimizer array is presented to the optimizer and the constraints framework handles the mapping directly.
End of explanation
dr._transform_gradients(dr.gradient)
Explanation: Note, that the optimizer array does only contain three values. This is because the first element of the the first rosen model is fixed and is not presented to the optimizer. The transformed gradients can be computed by the root node directly:
End of explanation
<END_TASK>