<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Time Series Database
Summary
This package implements a
time series database with the following functionality
Step1:  Load the database webserver.
Step2:  Import the web interface and initialize it.
Step3:  The instructions below assume that these three steps have been carried out.
Database Operations
Let's create some dummy data to aid in our demonstration. You will need to import the timeseries package to work with the TimeSeries format.
Note
Step4:  Insert Time Series
Inserts a new time series into the database. If any triggers are associated with time series insertion, then these are run and the results of their operations are also stored in the database.
Function signature
Step5:  Upsert Metadata
Inserts or updates metadata associated with a time series. Non-specified fields will be assigned a default value.
Function signature
Step6:  Delete Time Series
Deletes a time series and all associated metadata from the database.
Function signature
Step7:  Select
Queries the database for time series and/or associated metadata.
Function signature
Step8:  Augmented Select
Queries the database for time series and/or associated metadata, then executes a pre-specified function on the data that is returned.
Note
Step9:  Add Trigger
Adds a trigger that will cause a pre-defined function to be run upon execution of a particular database operation. For example, additional metadata fields may be calculated upon insertion of new time series data.
Function signature
Step10:  Remove Trigger
Removes a trigger associated with a database operation and a pre-defined function.
Function signature
Step11:  Add Vantage Point
Marks a time series as a vantage point. Vantage points are necessary to carry out vantage point similarity searches.
Function signature
Step12:  Delete Vantage Point
Unmarks a time series as a vantage point.
Function signature
Step13:  Vantage Point Similarity Search
Runs a vantage point similarity search, to find the closest (most similar) time series in the database.
Function signature
Step14:  iSAX Tree Similarity Search
Runs an iSAX tree-based similarity search, which returns a faster but only returns an approximate result.
Function signature
Step15:  iSAX Tree Representation
Returns a visual representation of the current contents of the iSAX tree.
Function signature
Step16:  Termination
Let's delete all the data before closing, so that we can start again from scratch in future demonstrations.
Step17:  Remember to terminate any outstanding processes!
<ASSISTANT_TASK:>
Python Code:
# you must specify the length of the time series when loading the database
ts_length = 100
# when running from the terminal
# python go_server_persistent.py --ts_length 100 --db_name 'demo'
# here we load the server as a subprocess for demonstration purposes
server = subprocess.Popen(['python', '../go_server_persistent.py',
                           '--ts_length', str(ts_length), '--data_dir', '../db_files', '--db_name', 'demo'])
time.sleep(5)  # make sure it loads completely
Explanation: Time Series Database
Summary
This package implements a
time series database with the following functionality:
* Insert time series data. May be followed by running a pre-defined function (trigger), if previously specified.
* Upsert (insert/update) time series metadata.
* Delete time series data and all associated metadata.
* Perform select (query) of time series data and/or metadata.
* Perform augmented select (query, followed by a pre-defined function) of time series data and/or metadata.
* Add a trigger that will cause a pre-defined function to be run upon execution of a particular database operation (e.g. calculate metadata fields after adding a new time series).
* Remove a trigger associated with a database operation and a pre-defined function.
* Add a vantage point (necessary to run vantage point similarity searches).
* Remove a vantage point and all associated data.
* Run a vantage point similarity search, to find the closest (most similar) time series in the database.
* Run an iSAX tree-based similarity search, to find the closest (most similar) time series in the database. This is a faster search technique, but it only returns an approximate answer and may not always find a match.
* Visualize the iSAX tree.
Initialization
The time series database can be accessed through a web interface, which directly executes database operations via the webserver (REST API).
Before running any database operations, you must:
Load the database server. You may pass the following arguments when loading the server.
--ts_length: Specifies the length of the time series, which must be consistent for all time series loaded into the database.
--data_dir: Specifies the directory where the database files are stored (optional).
--db_name: Specifies the database name (optional, but strongly recommended!).
End of explanation
# when running from the terminal
# python go_webserver.py
# here we load the server as a subprocess for demonstration purposes
webserver = subprocess.Popen(['python', '../go_webserver.py'])
time.sleep(5)  # make sure it loads completely
Explanation: Load the database webserver.
End of explanation
from webserver import *
web_interface = WebInterface()
Explanation: Import the web interface and initialize it.
End of explanation
from timeseries import *
def tsmaker(m, s, j):
    '''
    Helper function: randomly generates a time series for testing.
    Parameters
    ----------
    m : float
        Mean value for generating time series data
    s : float
        Standard deviation value for generating time series data
    j : float
        Quantifies the "jitter" to add to the time series data
    Returns
    -------
    A time series and associated meta data.
    '''
    # generate metadata
    meta = {}
    meta['order'] = int(np.random.choice(
        [-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5]))
    meta['blarg'] = int(np.random.choice([1, 2]))
    # generate time series data
    t = np.arange(0.0, 1.0, 0.01)
    v = norm.pdf(t, m, s) + j * np.random.randn(ts_length)
    # return time series and metadata
    return meta, TimeSeries(t, v)
# generate sample time series
num_ts = 50
mus = np.random.uniform(low=0.0, high=1.0, size=num_ts)
sigs = np.random.uniform(low=0.05, high=0.4, size=num_ts)
jits = np.random.uniform(low=0.05, high=0.2, size=num_ts)
# initialize dictionaries for time series and their metadata
primary_keys = []
tsdict = {}
metadict = {}
# fill dictionaries with randomly generated entries for database
for i, m, s, j in zip(range(num_ts), mus, sigs, jits):
    meta, tsrs = tsmaker(m, s, j)  # generate data
    pk = "ts-{}".format(i)  # generate primary key
    primary_keys.append(pk) # keep track of all primary keys
    tsdict[pk] = tsrs  # store time series data
    metadict[pk] = meta  # store metadata
Explanation: The instructions below assume that these three steps have been carried out.
Database Operations
Let's create some dummy data to aid in our demonstration. You will need to import the timeseries package to work with the TimeSeries format.
Note: the database is persistent, so can store data between sessions, but we will start with an empty database here for demonstration purposes.
End of explanation
# insert all the time series
for k in primary_keys:
    web_interface.insert_ts(pk=k, ts=tsdict[k])
# check what is in the database
web_interface.select(fields=None, additional={'sort_by': '+pk', 'limit': 10})
# successfully inserting data will yield a success code
web_interface.insert_ts(pk='sample1', ts=tsdict[primary_keys[0]])
# errors will yield an error code (e.g. attempting to insert the same primary key twice) 
web_interface.insert_ts(pk='sample1', ts=tsdict[primary_keys[0]])
# let's remove the test time series
web_interface.delete_ts('sample1')
Explanation: Insert Time Series
Inserts a new time series into the database. If any triggers are associated with time series insertion, then these are run and the results of their operations are also stored in the database.
Function signature:
insert_ts(pk, ts)
Parameters
----------
pk : any hashable type
Primary key for the new database entry
ts : TimeSeries
Time series to be inserted into the database  
Returns
-------
Result of the database operation (or error message).  
Examples:
End of explanation
# upsert the metadata
for k in primary_keys:
    web_interface.upsert_meta(k, metadict[k])
# let's check the first five entries in the database - they should include metadata
web_interface.select(fields=[], additional={'sort_by': '+pk', 'limit': 5})
Explanation: Upsert Metadata
Inserts or updates metadata associated with a time series. Non-specified fields will be assigned a default value.
Function signature:
upsert_meta(pk, md)
Parameters
----------
pk : any hashable type
Primary key for the  database entry
md : dictionary
Metadata to be upserted into the database
Returns
-------
Result of the database operation (or error message).  
Examples:
End of explanation
# example primary key to delete
primary_keys[0]
# delete an existing time series
web_interface.delete_ts(primary_keys[0])
# check what is in the database - should not include the deleted key
# note: select operations return dictionaries, so you can use the keys(), values(), and items() methods
web_interface.select(additional={'sort_by': '+pk'}).keys()
# double-check!
primary_keys[0] in web_interface.select(additional={'sort_by': '+pk'}).keys()
# add the time series and metadata back in
web_interface.insert_ts(primary_keys[0], tsdict[primary_keys[0]])
web_interface.upsert_meta(primary_keys[0], metadict[primary_keys[0]])
# check what is in the database - should include the newly added key
web_interface.select(additional={'sort_by': '+pk'}).keys()
Explanation: Delete Time Series
Deletes a time series and all associated metadata from the database.
Function signature:
delete_ts(pk)
Parameters
----------
pk : any hashable type
Primary key for the database entry to be deleted  
Returns
-------
Result of the database operation (or error message).  
Examples:
End of explanation
# select all database entries; no metadata fields
web_interface.select(additional={'sort_by': '+pk', 'limit': 10})
# select all database entries; all metadata fields
web_interface.select(fields=[], additional={'sort_by': '+pk', 'limit': 10})
# select a specific time series; all metadata fields
web_interface.select(md={'pk': 'ts-0'}, fields=[])
Explanation: Select
Queries the database for time series and/or associated metadata.
Function signature:
select(md={}, fields=None, additional=None)
Parameters
----------
md : dictionary (default={})
Criteria to apply to metadata
fields : list (default=None)
List of fields to return
additional : dictionary (default=None)
Additional criteria (e.g. 'sort_by' and 'limit')
Returns
-------
Query results (or error message).
Additional search criteria:
 sort_by: Sorts the query results in either ascending or descending order. Use + to denote ascending order and - to denote descending order.
e.g. {'sort_by': '+pk'} will sort by primary key in ascending order; {'sort_by': '-order'} will sort by the order metadata field in descending order.
 limit: Caps the number of fields that are returned when used in conjunction with sort_by.
e.g. {'sort_by': '+pk', 'limit': 5} for the top 5 primary keys
Examples:
End of explanation
# return a specific time series and the result of the 'stats' function (mean and standard deviation)
web_interface.augmented_select(
    proc='stats', target=['mean', 'std'], arg=None, md={'pk': 'ts-0'}, additional=None)
Explanation: Augmented Select
Queries the database for time series and/or associated metadata, then executes a pre-specified function on the data that is returned.
Note: the result of the function is not stored in the database.
Function signature:
augmented_select(proc, target, arg=None, md={}, additional=None)
Parameters
----------
proc : string
Name of the function to run when the trigger is met
target : string
Field names used to identify the results of the function.
arg : string (default=None)
Possible additional arguments (e.g. time series for similarity search)
md : dictionary (default={})
Criteria to apply to metadata
additional : dictionary (default=None)
Additional criteria ('sort_by' and 'order')  
Returns
-------
Query results (or error message).
Additional search criteria:
 sort_by: Sorts the query results in either ascending or descending order. Use + to denote ascending order and - to denote descending order.
e.g. {'sort_by': '+pk'} will sort by primary key in ascending order; {'sort_by': '-order'} will sort by the order metadata field in descending order.
 limit: Caps the number of fields that are returned when used in conjunction with sort_by.
e.g. {'sort_by': '+pk', 'limit': 5} for the top 5 primary keys
Available trigger functions:
 corr: Calculates the distance between two time series, using the normalize kernelized cross-correlation metric. Required argument: a TimeSeries object.
 stats: Calculates the mean and standard deviation of time series values. No arguments required.
Examples:
End of explanation
# add trigger
web_interface.add_trigger('stats', 'insert_ts', ['mean', 'std'], None)
# add a new time series with the trigger (note: not adding metadata)
web_interface.insert_ts('test', tsdict[primary_keys[0]])
# inspect the results of the trigger - should include mean and std fields
web_interface.select(md={'pk': 'test'}, fields=[])
# delete back out
web_interface.delete_ts('test')
Explanation: Add Trigger
Adds a trigger that will cause a pre-defined function to be run upon execution of a particular database operation. For example, additional metadata fields may be calculated upon insertion of new time series data.
Function signature:
add_trigger(proc, onwhat, target, arg=None)
Parameters
----------
proc : string
Name of the function to run when the trigger is hit
onwhat : string
Operation that triggers the function (e.g. 'insert_ts')
target : string
Array of field names to which to apply the results of the function
arg : string (default=None)
Possible additional arguments for the function  
Returns
-------
Result of the database operation (or error message).  
Available trigger functions:
 corr: Calculates the distance between two time series, using the normalize kernelized cross-correlation metric. Required argument: a TimeSeries object.
 stats: Calculates the mean and standard deviation of time series values. No arguments required.
Examples:
End of explanation
# remove trigger
web_interface.remove_trigger('stats', 'insert_ts')
# add a new time series without the trigger (note: not adding metadata)
web_interface.insert_ts('sample2', tsdict[primary_keys[0]])
# inspect the results of the trigger - should not include mean and std fields
web_interface.select(md={'pk': 'sample2'}, fields=[])
# delete back out
web_interface.delete_ts('sample2')
Explanation: Remove Trigger
Removes a trigger associated with a database operation and a pre-defined function.
Function signature:
remove_trigger(proc, onwhat, target=None)
Parameters
----------
proc : string
Name of the function that is run when the trigger is hit
onwhat : string
Operation that triggers the function (e.g. 'insert_ts')
target : string
Array of field names to which the results are applied. If not provided, all triggers associated with the database operation and function will be removed.  
Returns
-------
Result of the database operation (or error message).  
Examples:
End of explanation
# randomly choose time series as vantage points
num_vps = 5
random_vps = np.random.choice(range(num_ts), size=num_vps, replace=False)
vpkeys = ['ts-{}'.format(i) for i in random_vps]
# add the time series as vantage points
for i in range(num_vps):
    web_interface.insert_vp(vpkeys[i])
Explanation: Add Vantage Point
Marks a time series as a vantage point. Vantage points are necessary to carry out vantage point similarity searches.
Function signature:
insert_vp(pk)
Parameters
----------
pk : any hashable type
Primary key for the time series to be marked as a vantage point  
Returns
-------
Result of the database operation (or error message).  
Examples:
End of explanation
# delete one of the vantage points
web_interface.delete_vp(vpkeys[0])
# add it back in
web_interface.insert_vp(vpkeys[0])
Explanation: Delete Vantage Point
Unmarks a time series as a vantage point.
Function signature:
delete_vp(pk)
Parameters
----------
pk : any hashable type
Primary key for the time series to be unmarked as a vantage point  
Returns
-------
Result of the database operation (or error message).  
Examples:
End of explanation
# run similarity search on a time series already in the database
# should return the same time series
primary_keys[0], web_interface.vp_similarity_search(tsdict[primary_keys[0]], 1)
# create dummy time series for demonstration purposes
_, query = tsmaker(np.random.uniform(low=0.0, high=1.0),
                   np.random.uniform(low=0.05, high=0.4),
                   np.random.uniform(low=0.05, high=0.2))
results = web_interface.vp_similarity_search(query, 1)
results
# visualize the results
plt.plot(query, label='Query TS')
plt.plot(tsdict[list(results.keys())[0]], label='Closest TS')
plt.legend(loc='best')
plt.xticks([])
plt.show()
Explanation: Vantage Point Similarity Search
Runs a vantage point similarity search, to find the closest (most similar) time series in the database.
Function signature:
similarity_search(self, query, top=1)
Parameters
----------
query : TimeSeries
The time series being compared to those in the database
top : int
The number of closest time series to return (default=1)
Returns
-------
Primary key and distance to the closest time series (or error message if database operation fails).
Examples:
End of explanation
# run similarity search on a time series already in the database
# should return the same time series
primary_keys[0], web_interface.isax_similarity_search(tsdict[primary_keys[0]])
# create dummy time series for demonstration purposes
_, query = tsmaker(np.random.uniform(low=0.0, high=1.0),
                   np.random.uniform(low=0.05, high=0.4),
                   np.random.uniform(low=0.05, high=0.2))
# note: because this is an approximate search, it will not be able
# to find a match for all query time series
results = web_interface.isax_similarity_search(query)
results
# visualize the results
plt.plot(query, label='Query TS')
plt.plot(tsdict[list(results.keys())[0]], label='Closest TS')
plt.legend(loc='best')
plt.xticks([])
plt.show()
Explanation: iSAX Tree Similarity Search
Runs an iSAX tree-based similarity search, which returns a faster but only returns an approximate result.
Function signature:
isax_similarity_search(query)
Parameters
----------
query : TimeSeries
The time series being compared to those in the database  
Returns
-------
Primary key of the closest time series (or error message if database operation fails).
Examples:
End of explanation
# note: print() is required to visualize the tree correctly with carriage returns
print(web_interface.isax_tree())
Explanation: iSAX Tree Representation
Returns a visual representation of the current contents of the iSAX tree.
Function signature:
isax_tree()
Parameters
----------
None  
Returns
-------
Result of the database operation (or error message).
Examples:
End of explanation
# insert all the time series
for k in primary_keys:
    web_interface.delete_ts(pk=k)
# check that no data is left
web_interface.select()
Explanation: Termination
Let's delete all the data before closing, so that we can start again from scratch in future demonstrations.
End of explanation
# terminate processes before exiting
os.kill(server.pid, signal.SIGINT)
time.sleep(5)  # give it time to terminate
web_interface = None
webserver.terminate()
Explanation: Remember to terminate any outstanding processes!
End of explanation
<END_TASK>