<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Session 2
Step1:  import
The Python import statement makes other packages available to your current session.
There are a few forms of import, two of which are illustrated here
Step2:  Aside
Step3:  The rest of this tutorial uses v to refer to current Veneer client
Querying models
veneer-py provides two groups of functions
Step4:  You can also perform some basic topological queries on the network, such as finding all network outlets, or finding upstream/downstream links for a given node
Step5:  If you have GeoPandas installed, you can convert network to a GeoDataFrame, which is useful for visualisation, reporting and advanced filtering.
Step6:  With a GeoDataFrame, you can filter against any column
Step7:  Querying Functions and Variables
You can query all functions using v.functions(), and all variables with v.variables().
Step8:  Both v.functions() and v.variables() return a searchable list
Step9:  Note
Step10:  v.variables() only returns summary information about the variable. If you want to get at the actual time series (or the piecewise linear function or similar), use v.variable_time_series(name) (or v.variable_piecewise(name))
Step11:  Note
Step12:  Using .as_dataframe() can help to interpret the information
Step13:  This highlights that the Items column has nested information
Step14:  So, v.data_sources() returns information about all data sources. Specifically, it returns the list of data sources, and, for each one, information about the different time series. However, v.data_sources does not return the entire time series. (This is mainly for speed - on big models, this can be HUGE)
To get to the time series, either use v.data_source(name) to retrieve a single data source (and all the time series within it) OR v.data_source_item(source,name) to retrieve a particular time series.
Note
Step15:  You can see that the time series are included, but they are still nested within the returned data. Essentially, Veneer has returned the data for each input set, with each being a separate Pandas DataFrame.
You can get to an individual DataFrame back with find_one_by_Name
Step16:  You can jump to a particular data source item (typically a column from an original file) with v.data_source_item(source,name)
Step17:  Input Sets
Veneer provides access to the scenario input sets - primarily the commands for changing individual model parameters - through `v.input_sets()
Step18:  Within a given input set, the individual commands are available in the Configuration property
Step19:  Input sets are a lot more interesting when you are changing them and applying them from Python...
Running Models
You can run the current scenario, with its current settings (start/end date, input set, run configuration, etc) using v.run_model()
Step20:  After a successful run, veneer-py returns the URL of the new run - this can be used to retrieve an index of all the available results and, from there, the actual time series results
Step21:  This index lists all of the time series (under ['Results']) and can be used to retrieve individual time series, or collections of time series
Step22:  The easiest way to retrieve time series results is with v.retrieve_multiple_time_series, which takes a set of run results, some search criteria by which it identifies wanted time series. There are also options for retrieve aggregated data and the ability to rename time series as they are retrieved.
The help text for retrieve_multiple_time_series describes the options
Step23:  The following examples retrieve data from the latest run, using the index index we saved, earlier
Step24:  Other run related actions
There are numerous functions and options that are closely related to running simulations
Step25:  Note
Step26:  Manipulating model setup and the ‘Allow Scripts’ option
There are various ways to use veneer-py to change the configuration of the currently loaded model in Source
Methods to update input sets with new parameters (v.update_input_set)
Applying a particular input set (v.apply_input_set)
Use functionality under v.model. to directly change model parameters and structure
It is also possible to modify files on disk where Source is configured to 'Reload-On-Run', such as changing time series files or input set command files. This option isn't specific to Veneer/veneer-py and isn't covered in these tutorials.
We can update the input set commands for the default input set.
(Making a single change like this can be a bit fiddly!)
Step27:  At this point, you can check in the Source user interface (Edit|Scenario Input Sets) to see that the input set has been updated. Alternatively, you can query the input set again
Step28:  The Default Input Set would be applied when we next run, but if we want those parameters set on the model immediately, we can do so, with v.apply_input_set('Default Input Set')
Step29:  v.model and the Allow Scripts option
The v.model namespace contains a lot of functionality for modifying the structure of the Source model. This part of veneer-py works by generating IronPython scripts, which run directly within Source and manipulate the .NET objects that Source is built upon.
This capability (to accept IronPython scripts) is disabled, by default, in Veneer. To enable it, click the 'Allow Scripts' checkbox in the Web Server Monitoring window. If you don't enable the 'Allow Scripts' option, then the following instructions will give an error.
As an example of what can be done, the following command queries the routing model used for each link
Step30:  This shows that, for the example model, there are three Straight Through Routing links and two Storage Routing links.
get_models() has returned a list - and this is common within the v.model namespace. It's also common for functions to accept a list when changing the Source model (eg by changing assigned models, or changing parameters).
The list isn't directly useful though, because it doesn't tell you which link is which. That can be checked by asking for the results coded by name
Step31:  PEST and Parallel processing
PEST is a powerful, model independent Parameter ESTimation tool. veneer-py has some initial functionality for configuring and running PEST jobs over Source models using Veneer.
This functionality is currently the best demonstration of parallel processing using Veneer and veneer-py. This is explored in Tutorial 8.
GUI and Command Line
Veneer is distributed with a command line program, FlowMatters.Source.VeneerCmd.exe, which is an alternative to the eWater supplied RiverSystem.CommandLine.exe. The Veneer command line uses the same protocols as the plugin version of Veneer, making them interchangeable for most applications.
Note
Step32:  Each copy of the Veneer command line is a copy of the Source engine with a Veneer server.
Each running process will run until terminated, either by the user or by the end of the Python session.
Local and Remote connections
Veneer communicates via network ports, but by default, these connections are restricted to local machine - ie you cannot connect to Veneer on machine A from a Python notebook running on machine B.
If you want to allow remote connections, you need to enable the checkbox in the Web Server Monitoring window. However, in order to do so, you will first need to register the Veneer URL with Windows. Instructions are at https
<ASSISTANT_TASK:>
Python Code:
import veneer
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
Explanation: Session 2: Quick tour of Veneer
Main features of Veneer (and veneer-py)
Starting a new notebook
Querying models
Running models and Retrieving results
Manipulating model setup and the ‘Allow Scripts’ option
PEST and Parallel processing
GUI and Command Line
Local and Remote connections
Getting help
Start Source and Veneer
If you haven't already started Source, do so now, following the instructions in 0Setup_and_Hello_World.ipynb to load and start Veneer using the Tools|Web Server Monitoring option.
Which Model?
Note: This session uses ExampleProject/RiverModel1.rsproj. You are welcome to work with your own model instead, however you will need to change the notebook text at certain points to reflect the names of nodes, links and functions in your model file.
Starting a new Notebook
You can edit your local copy of this notebook. Alternatively, start a new notebook and enter the relevant code there. From an existing notebook, use the File|New Notebook|Python 3 menu to start a new notebook using Python 3 syntax.
You can also create a new notebook from the main Jupyter dashboard, using the New button.
It's probably worth using a Markdown cell at the top of the notebook to describe the purpose of the notebook (if only to remind yourself later...)
Then, in the first code block, you should import relevant Python packages and generally initialise your Python environment with anything needed for the notebook:
End of explanation
%matplotlib inline
plt.plot(range(10))
%matplotlib notebook
plt.plot(range(10))
%matplotlib inline
Explanation: import
The Python import statement makes other packages available to your current session.
There are a few forms of import, two of which are illustrated here:
import veneer makes the main veneer-py package available. You can access parts of the package using veneer.&lt;member&gt;
import pandas as pd imports the pandas package, but aliases it as pd for convenience. For common packages (such as numpy adn pandas) there is often a well-used abbreviation, such as np and pd.
from os import mkdir, makedirs imports one or more specific members of the os package and makes them available without any prefix
from os import * imports all the members of the os package and makes them available without any prefix (eg makedirs('test')).
%matplotlib
%matplotlib enables plotting by configuring a rendering engine for the matplotlib package. Anything starting with % is a special ('magic') command to Jupyter/IPython - it isn't standard Python.
The %matplotlib magic command accepts an optional parameter indicating which rendering backend to use for the current session. I've used inline, which renders charts directly within the notebook as HTML &lt;img&gt; tags. I find this best for creating self-sufficient notebooks that can be circulated and read without running. The downside is that the charts aren't interactive.
Other options worth noting:
%matplotlib - (ie without specifying a rendering backend). This gives you the default for your current system. On my Windows machine, this uses the Qt4 windowing toolkit and charts appear in their own window, with basic zoom, pan, export and layout controls.
%matplotlib notebook - Renders within the notebook with some basic interactive chart controls.
End of explanation
v = veneer.Veneer()
# Equiavelent to
# v = veneer.Veneer(host='localhost',port=9876)
Explanation: Aside: the range function is a generator. In this case, its generating a consecutive integers, starting from 0 and going up to, but not including, 10.
Try:
print(range(10))
and
print(list(range(10))
Initialising a Veneer client
veneer-py provides functions for accessing a running copy of Source/Veneer. Most of these functions are provided through an instance of the Veneer client class, within the veneer package. You can have as many clients as you need - for example, to point to different copies of Source that are running concurrently.
By default, the client will attempt to connect to a copy of Source/Veneer running on the local machine and hosted on port 9876
End of explanation
the_network = v.network()
all_nodes = the_network['features'].find_by_feature_type('node')
len(all_nodes)
all_nodes._all_values('name')
Explanation: The rest of this tutorial uses v to refer to current Veneer client
Querying models
veneer-py provides two groups of functions:
Functions for accessing the various URLs defined by Veneer (eg v.network() accesses the /network URL), and
Functions for accessing the catch-all, IronPython endpoint (eg v.model.catchments.runoff.get_param_values('SMSC') uses IronPython to query the SMSC parameter in the rainfall runoff models)
The functions in group 1 should work on all Source models, whereas the functions in group 2 tend to be more model specific.
This section deals with group 1 - functions related to specific URLs.
Network queries
You can retrieve the current node link network (including catchment polygons if present) using v.network(). The result includes a list of features, which you can search and filter for particular nodes/links/catchments.
End of explanation
outlets = the_network.outlet_nodes()
outlets
the_network.upstream_links(outlets[0])
Explanation: You can also perform some basic topological queries on the network, such as finding all network outlets, or finding upstream/downstream links for a given node
End of explanation
network_df = the_network.as_dataframe()
network_df
Explanation: If you have GeoPandas installed, you can convert network to a GeoDataFrame, which is useful for visualisation, reporting and advanced filtering.
End of explanation
network_df[network_df.feature_type=='node']
network_df[network_df.icon=='/resources/StorageNodeModel']
Explanation: With a GeoDataFrame, you can filter against any column
End of explanation
model_functions = v.functions()
model_functions
model_variables = v.variables()
model_variables
Explanation: Querying Functions and Variables
You can query all functions using v.functions(), and all variables with v.variables().
End of explanation
model_variables.find_by_Name('$Runoff')
Explanation: Both v.functions() and v.variables() return a searchable list:
End of explanation
model_variables.as_dataframe()
Explanation: Note: When you get a searchable list, you can also convert it to a Pandas Data Frame if that is more convenient
End of explanation
runoff_ts = v.variable_time_series('$Runoff')
runoff_ts.plot()
mfr_piecewise = v.variable_piecewise('MFR_Piecewise')
mfr_piecewise
Explanation: v.variables() only returns summary information about the variable. If you want to get at the actual time series (or the piecewise linear function or similar), use v.variable_time_series(name) (or v.variable_piecewise(name)):
End of explanation
data_sources = v.data_sources()
data_sources
Explanation: Note: At this stage, only Time Series and Piecewise Linear variables are supported through this mechanism
Data Sources
You can get a list of all Data Sources in the model with v.data_sources()
End of explanation
data_sources.as_dataframe()
Explanation: Using .as_dataframe() can help to interpret the information:
End of explanation
data_sources[0]['Items']
Explanation: This highlights that the Items column has nested information:
End of explanation
forcing = v.data_source('ForcingData_csv')
forcing
Explanation: So, v.data_sources() returns information about all data sources. Specifically, it returns the list of data sources, and, for each one, information about the different time series. However, v.data_sources does not return the entire time series. (This is mainly for speed - on big models, this can be HUGE)
To get to the time series, either use v.data_source(name) to retrieve a single data source (and all the time series within it) OR v.data_source_item(source,name) to retrieve a particular time series.
Note: With v.data_source_item you can retrieve the time series for an item that is associated with a particular input set.
End of explanation
forcing_as_df = forcing['Items'].find_one_by_Name('Default Input Set')['Details']
forcing_as_df[0:30] # Look at the first 30 time steps
Explanation: You can see that the time series are included, but they are still nested within the returned data. Essentially, Veneer has returned the data for each input set, with each being a separate Pandas DataFrame.
You can get to an individual DataFrame back with find_one_by_Name
End of explanation
rainfall = v.data_source_item('ForcingData_csv','Rainfall')
rainfall[0:30]
Explanation: You can jump to a particular data source item (typically a column from an original file) with v.data_source_item(source,name)
End of explanation
input_sets = v.input_sets()
input_sets
Explanation: Input Sets
Veneer provides access to the scenario input sets - primarily the commands for changing individual model parameters - through `v.input_sets()
End of explanation
input_sets.find_one_by_Name('Default Input Set')['Configuration']
Explanation: Within a given input set, the individual commands are available in the Configuration property
End of explanation
response,run_url = v.run_model()
response,run_url
Explanation: Input sets are a lot more interesting when you are changing them and applying them from Python...
Running Models
You can run the current scenario, with its current settings (start/end date, input set, run configuration, etc) using v.run_model()
End of explanation
run_url
index = v.retrieve_run(run_url)
index
Explanation: After a successful run, veneer-py returns the URL of the new run - this can be used to retrieve an index of all the available results and, from there, the actual time series results
End of explanation
index['Results'].as_dataframe()
Explanation: This index lists all of the time series (under ['Results']) and can be used to retrieve individual time series, or collections of time series
End of explanation
help(v.retrieve_multiple_time_series)
Explanation: The easiest way to retrieve time series results is with v.retrieve_multiple_time_series, which takes a set of run results, some search criteria by which it identifies wanted time series. There are also options for retrieve aggregated data and the ability to rename time series as they are retrieved.
The help text for retrieve_multiple_time_series describes the options
End of explanation
# Retrieve Downstream Flow Volume, anywhere it is recorded.
# Name the columns for the network location
ds_flow = v.retrieve_multiple_time_series(run_data=index,criteria={'RecordingVariable':'Downstream Flow Volume'},
                                          name_fn=veneer.name_for_location)
ds_flow[0:10]
# Retrieve everything available from the Water User node
# Name the columns for the variable
water_user = v.retrieve_multiple_time_series(run_data=index,criteria={'NetworkElement':'Water User'},
                                          name_fn=veneer.name_for_variable)
water_user[0:10]
Explanation: The following examples retrieve data from the latest run, using the index index we saved, earlier:
End of explanation
v.drop_all_runs()
v.configure_recording(enable=[{'RecordingVariable':'Storage Surface Area'}])
v.run_model()
surface_area = v.retrieve_multiple_time_series(criteria={'RecordingVariable':'Storage Surface Area'})
surface_area.plot()
Explanation: Other run related actions
There are numerous functions and options that are closely related to running simulations:
You can call v.run_model() with alternate start and end dates, along with other options related to the simulation, such as specifying an input set
You can call v.apply_input_set() to run the commands in a particular input set. This doesn't trigger a run, so its useful in situations where you want to run multiple input sets before a single simulation
v.configure_recording() can switch time series recorders on and off
You can drop a run from memory (v.drop_run) or drop all runs (v.drop_all_runs())
These actions will all be covered in subsequent tutorials. For now, a simple example of enabling recording of Storage Surface Area and rerunning:
End of explanation
v.drop_all_runs()
Explanation: Note: We again drop all runs before moving on...
End of explanation
default_input_set = v.input_sets()[0]
default_input_set
command_of_interest = default_input_set['Configuration'][0]
command_of_interest
default_input_set['Configuration'][0] = command_of_interest.replace('25 m','50 m')
default_input_set
v.update_input_set('Default Input Set',default_input_set)
Explanation: Manipulating model setup and the ‘Allow Scripts’ option
There are various ways to use veneer-py to change the configuration of the currently loaded model in Source
Methods to update input sets with new parameters (v.update_input_set)
Applying a particular input set (v.apply_input_set)
Use functionality under v.model. to directly change model parameters and structure
It is also possible to modify files on disk where Source is configured to 'Reload-On-Run', such as changing time series files or input set command files. This option isn't specific to Veneer/veneer-py and isn't covered in these tutorials.
We can update the input set commands for the default input set.
(Making a single change like this can be a bit fiddly!)
End of explanation
v.input_sets()
Explanation: At this point, you can check in the Source user interface (Edit|Scenario Input Sets) to see that the input set has been updated. Alternatively, you can query the input set again
End of explanation
v.apply_input_set('Default Input Set')
Explanation: The Default Input Set would be applied when we next run, but if we want those parameters set on the model immediately, we can do so, with v.apply_input_set('Default Input Set')
End of explanation
v.model.link.routing.get_models()
Explanation: v.model and the Allow Scripts option
The v.model namespace contains a lot of functionality for modifying the structure of the Source model. This part of veneer-py works by generating IronPython scripts, which run directly within Source and manipulate the .NET objects that Source is built upon.
This capability (to accept IronPython scripts) is disabled, by default, in Veneer. To enable it, click the 'Allow Scripts' checkbox in the Web Server Monitoring window. If you don't enable the 'Allow Scripts' option, then the following instructions will give an error.
As an example of what can be done, the following command queries the routing model used for each link
End of explanation
v.model.link.routing.get_models(by_name=True)
Explanation: This shows that, for the example model, there are three Straight Through Routing links and two Storage Routing links.
get_models() has returned a list - and this is common within the v.model namespace. It's also common for functions to accept a list when changing the Source model (eg by changing assigned models, or changing parameters).
The list isn't directly useful though, because it doesn't tell you which link is which. That can be checked by asking for the results coded by name
End of explanation
from veneer.manage import start
help(start)
Explanation: PEST and Parallel processing
PEST is a powerful, model independent Parameter ESTimation tool. veneer-py has some initial functionality for configuring and running PEST jobs over Source models using Veneer.
This functionality is currently the best demonstration of parallel processing using Veneer and veneer-py. This is explored in Tutorial 8.
GUI and Command Line
Veneer is distributed with a command line program, FlowMatters.Source.VeneerCmd.exe, which is an alternative to the eWater supplied RiverSystem.CommandLine.exe. The Veneer command line uses the same protocols as the plugin version of Veneer, making them interchangeable for most applications.
Note: To use the Veneer command line, it needs access to all of the DLLs distributed with Source. Currently, this means that the Veneer command line, and the various DLLs related to Veneer, need to be in the same directory as these eWater supplied files. You can copy all of the eWater files into the Veneer directory, or copy all the Veneer files into the Source installation directory (typically under C:\Program Files\eWater\Source X.Y.ZZ.WWWW). Importantly, if copying the Veneer files to the Source directory, the Veneer files must be in the same directory as the main Source files (eg RiverSystem.Forms.exe), not in the plugins directory
Starting the Veneer Command Line from Python
veneer-py includes a function, start (under veneer.manage) for starting one or more copies of the Veneer command line using a given project file.
End of explanation
help(v.network)
Explanation: Each copy of the Veneer command line is a copy of the Source engine with a Veneer server.
Each running process will run until terminated, either by the user or by the end of the Python session.
Local and Remote connections
Veneer communicates via network ports, but by default, these connections are restricted to local machine - ie you cannot connect to Veneer on machine A from a Python notebook running on machine B.
If you want to allow remote connections, you need to enable the checkbox in the Web Server Monitoring window. However, in order to do so, you will first need to register the Veneer URL with Windows. Instructions are at https://github.com/flowmatters/veneer.
Getting help
Many of the functions in veneer-py have built in documentation, which can be accessed in Jupyter/IPython using the ? command, such as:
v.network?
Alternatively, you can use the help() function, as in:
End of explanation
<END_TASK>