<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 This is a notebook to explore opSim outputs in different ways, mostly useful to supernova analysis. We will look at the opsim output called Enigma_1189
Step1:  Read in OpSim output for modern versions
Step2:  Read in the OpSim DataBase into a pandas dataFrame
Step3:  The opsim database is a large file (approx 4.0 GB), but still possible to read into memory on new computers. You usually only need the Summary Table, which is about 900 MB. If you are only interested in the Deep Drilling Fields, you can use the read_sql_query to only select information pertaining to Deep Drilling Observations. This has a memory footprint of about 40 MB.
Obviously, you can reduce this further by narrowing down the columns to those of interest only. For the entire Summary Table, this step takes a few minutes on my computer. 
If you are going to do the read from disk step very often, you can further reduce the time used by storing the output on disk as a hdf5 file and reading that into memory
We will look at three different Summaries of OpSim Runs. A summary of the 
1. Deep Drilling fields
Step4:  Some properties of the OpSim Outputs
Step5:  Construct our Summary
Step6:  First Season
We can visualize the cadence during the first season using the cadence plot for a particular field
Step7:  Example to obtain the observations of in a 100 day period in a field
First find the fieldID witha  center closest to your coordinates. The fields are of radial size about 1.75 degrees.  I would suggest just going to a fieldID, as you probably don't care about the coordinates. Then the following query would get this done. Alternatively, you could also achieve some of these goals using sql queries on the opsim database.
Step8:  Plots
Step9:  This is a DDF. 
- Many observations per night
- Often about 20-25 per filter per night
Step10:  WFD field
<ASSISTANT_TASK:>
Python Code:
import numpy as np
%matplotlib inline 
import matplotlib.pyplot as plt
# Required packages sqlachemy, pandas (both are part of anaconda distribution, or can be installed with a python installer)
# One step requires the LSST stack, can be skipped for a particular OPSIM database in question
import OpSimSummary.summarize_opsim as so
from sqlalchemy import create_engine
import pandas as pd
print so.__file__
# This step requires LSST SIMS package MAF. The main goal of this step is to set DD and WFD to integer keys that 
# label an observation as Deep Drilling or for Wide Fast Deep.
# If you want to skip this step, you can use the next cell by uncommenting it, and commenting out this cell, if all you
# care about is the database used in this example. But there is no guarantee that the numbers in the cell below will work
# on other versions of opsim database outputs
#from lsst.sims.maf import db
#from lsst.sims.maf.utils import opsimUtils
# DD = 366
# WFD = 364
Explanation: This is a notebook to explore opSim outputs in different ways, mostly useful to supernova analysis. We will look at the opsim output called Enigma_1189
End of explanation
# Change dbname to point at your own location of the opsim output
dbname = '/Users/rbiswas/data/LSST/OpSimData/enigma_1189_sqlite.db'
#opsdb = db.OpsimDatabase(dbname)
#propID, propTags = opsdb.fetchPropInfo()
#DD = propTags['DD'][0]
#WFD = propTags['WFD'][0]
Explanation: Read in OpSim output for modern versions: (sqlite formats)
Description of OpSim outputs are available on the page https://confluence.lsstcorp.org/display/SIM/OpSim+Datasets+for+Cadence+Workshop+LSST2015http://tusken.astro.washington.edu:8080
Here we will use the opsim output http://ops2.tuc.noao.edu/runs/enigma_1189/data/enigma_1189_sqlite.db.gz
I have downloaded this database, unzipped and use the variable dbname to point to its location
End of explanation
engine = create_engine('sqlite:///' + dbname)
Explanation: Read in the OpSim DataBase into a pandas dataFrame
End of explanation
# Load to a dataframe
# Summary = pd.read_hdf('storage.h5', 'table')
Summary = pd.read_sql_table('Summary', engine, index_col='obsHistID')
# EnigmaDeep  = pd.read_sql_query('SELECT * FROM SUMMARY WHERE PROPID is 366', engine)
# EnigmaD  = pd.read_sql_query('SELECT * FROM SUMMARY WHERE PROPID is 366', engine)
EnigmaCombined = Summary.query('propID == [364, 366]')# & (fieldID == list(EnigmaDeep.fieldID.unique().values)')
EnigmaCombined.propID.unique()
Explanation: The opsim database is a large file (approx 4.0 GB), but still possible to read into memory on new computers. You usually only need the Summary Table, which is about 900 MB. If you are only interested in the Deep Drilling Fields, you can use the read_sql_query to only select information pertaining to Deep Drilling Observations. This has a memory footprint of about 40 MB.
Obviously, you can reduce this further by narrowing down the columns to those of interest only. For the entire Summary Table, this step takes a few minutes on my computer. 
If you are going to do the read from disk step very often, you can further reduce the time used by storing the output on disk as a hdf5 file and reading that into memory
We will look at three different Summaries of OpSim Runs. A summary of the 
1. Deep Drilling fields: These are the observations corresponding to propID of the variable DD above, and are restricted to a handful of fields
2. WFD (Main) Survey: These are the observations corresponding to the propID of the variables WFD
3. Combined Survey: These are observations combining DEEP and WFD in the DDF. Note that this leads to duplicate observations which must be subsequently dropped.
End of explanation
EnigmaCombined.fieldID.unique().size
Explanation: Some properties of the OpSim Outputs
End of explanation
Full = so.SummaryOpsim(EnigmaCombined)
fig = plt.figure(figsize=(10, 5))
ax = fig.add_subplot(111, projection='mollweide');
fig = Full.showFields(ax=fig.axes[0], marker='o', s=1)
Explanation: Construct our Summary
End of explanation
fieldList = Full.fieldIds
len(fieldList)
Explanation: First Season
We can visualize the cadence during the first season using the cadence plot for a particular field: The following plot shows how many visits we have in different filters on a particular night:
End of explanation
selected = Full.df.query('fieldID == 290 and expMJD > 49490 and expMJD < 49590')
selected.head()
# write to disk in ascii file
selected.to_csv('selected_obs.csv', index='obsHistID')
# write to disk in ascii file with selected columns
selected[['expMJD', 'night', 'filter', 'fiveSigmaDepth', 'filtSkyBrightness', 'finSeeing']].to_csv('selected_cols.csv', index='obsHistID')
Explanation: Example to obtain the observations of in a 100 day period in a field
First find the fieldID witha  center closest to your coordinates. The fields are of radial size about 1.75 degrees.  I would suggest just going to a fieldID, as you probably don't care about the coordinates. Then the following query would get this done. Alternatively, you could also achieve some of these goals using sql queries on the opsim database.
End of explanation
fig_firstSeason, firstSeasonCadence = Full.cadence_plot(fieldList[0], observedOnly=False, sql_query='night < 366')
Explanation: Plots
End of explanation
fig_firstSeason_1, firstSeasonCadence_1 = Full.cadence_plot(fieldList[0], observedOnly=True, sql_query='night < 366')
Explanation: This is a DDF. 
- Many observations per night
- Often about 20-25 per filter per night
End of explanation
fig_firstSeason_main, firstSeasonCadence_main = Full.cadence_plot(fieldList[1], observedOnly=False, sql_query='night < 366')
fig_long, figCadence_long  = Full.cadence_plot(fieldList[0], observedOnly=False, sql_query='night < 3655', nightMax=3655)
fig_2, figCadence_2  = Full.cadence_plot(fieldList[0], observedOnly=False, 
                                         sql_query='night < 720', nightMax=720, nightMin=365)
fig_SN, SN_matrix = Full.cadence_plot(fieldList[0], observedOnly=False, mjd_center=49540., mjd_range=[-30., 50.])
Explanation: WFD field
End of explanation
<END_TASK>