<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Strong Lensing in Twinkles
Overview
Strong Lensing in Twinkles simulations involves taking existing CATSIM instance catalogs and replacing specific AGN within those catalogs with a Strongly Lensed AGN system. This means adding images of the original AGN and a lens galaxy into the CATSIM instance catalogs. This is done using a piece of code in the Twinkles repository that is known as the sprinkler. This notebook will first discuss the sprinkler and then move onto verifying its outputs in instance catalogs.
Sprinkler Code Review
The sprinkler code can be found here. It involves a step inserted into the final pass method of the Instance Catalog class that allows manipulation of the array of catalog results to be manipulated before printing out. This step starts with the loading of an OM10 based catalog of strongly lensed AGN systems.
The OM10-Twinkles Catalog
The catalog we are using started with the set of the 872 rung 4 lenses from the Time Delay Challenge 1 (see discussion in this issue on why these were chosen). However, when looking through this catalog we learned that sizes were missing for the foreground lens galaxies and we have chosen to use CATSIM galaxies to add this information to the catalog as well as an SED for each of these foreground galaxies. This process was completed in this issue. In the process 11 galaxies were unable to be matched adequately to CATSIM galaxies so we end up with a final catalog in Twinkles of 861 galaxies.
Selecting Sprinkled Systems
The next step after loading the lensing catalog is to go through the instance catalog AGNs one at a time and see if they match a lens system from our OM10-Twinkles catalog. The matching is done by taking the redshift and magnitude of the CATSIM AGN and selecting any available lens systems with a source AGN that has a redshift within 0.1 dex and an LSST magnorm value (magnitude in a simulated filter with one bin at 500nm) within .25 mags.
Once the possible matches to a given instance catalog AGN are found there is a chance that it will be replaced that is set by the density parameter in the sprinkler class. In the current Twinkles run it is set at 1.0 so that 198 lens systems are sprinkled into the Twinkles instance catalogs.
If the AGN is chosen to be sprinkled then we proceed with the sprinkler and if there are multiple matches from the OM10-Twinkles catalog we randomly choose one to use as we proceed. First, we remove any disc and bulge information for that galaxy from the instance catalog leaving only the AGN data and copy this AGN entry creating 2 or 4 entries in the catalog depending upon the number of images the OM10 system we are sprinkling in has in total. Next we use the OM10 data for each image to shift the location of the AGN in each image to the correct location and adjust the magnitude for each entry depending on the OM10 magnification parameter for each AGN image. At this point we also adjust the time delay for each image with OM10 info and give all the images the same source redshift from the OM10 entry. Once the new entries are ready they are appended to the instance catalog with a new instance catalog id tag that gives us the ID of the Bulge galaxy in the instance catalog to which it is associated and the ID number of the OM10-Twinkles catalog system that is used.
The next step is to take the original entry in the instance catalog and add in the information for the foreground lens galaxy. Therefore, in this step the AGN and Disc data for the original galaxy are cleared and the redshift and magnitude for the lens galaxy from the OM10 system is added into the instance catalog. Here we also add in the radius of the lens galaxy and the sed file we want to use (the information added to the original OM10 systems in the process described in the last section). We also add in the OM10 position angles and ellipticities at this point.
Once we have gone through this process for all AGN in the instance catalog then we return to the normal instance catalog processing to write the information out to file.
Verifying Instance Catalog Output
For the sections below I am using the output from an instance catalog created using Twinkles and removing the SNe entries at the bottom. I then split it up into the bulge+disk galaxy entries and the AGNs before only selecting the objects that were sprinkled. We will also load in the OM10-Twinkles catalog of strongly lensed AGN systems for validation purposes. This catalog is where all the lens systems that end up in Twinkles catalogs are stored.
Loading datasets
Step1:  This file loaded contains extra information to a normal instance catalog output. During the sprinkling process as new id numbers are given to the sprinkled AGN all of their ids are larger than 1e11 while those unsprinkled are not. These larger Ids contain information on the OM10-Twinkles system used in the sprinkler, the image number in that system and the associated lens galaxy. We have previously added columns containing this information to the instance catalog information for the AGN.
Step2:  Validating Image Positions
Our first task is to verify that the image locations in the instance catalog are in the correct places when compared to the entries from the OM10 catalog. To do this we will find the relative ra and dec positions for each of the images in a lensed system relative to the lens galaxy from OM10 and compare the instance catalog values to the appropriate OM10 system.
Step3:  Overall, differences between the instance catalogs and the OM10 inputs seem to be within 0.01 arcseconds. It appears that the positions in the instance catalog are accurate reproductions of the OM10 data they are based upon.
Check distributions of Lens galaxies
In the next section we will plot the properties of the lens galaxies in the sprinkled data.
Step4:  Check distributions of Lensed AGN
Now we plot the properties of the lensed AGN sprinkled into the instance catalogs.
Step5:  Varaibility Validation
Finally, we take a look to make sure that the magnitudes for the AGN are not static. We do that by looking at the AGN in two more r band visits from the Twinkles simulation.
Step6:  As the previous cell shows there are no AGN with the same magnitude from the first visit to the next showing that variability is being calculated for each system. Below we plot a few light curves on the set of three visits.
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from astropy.io import fits
plt.style.use('ggplot')
%matplotlib inline
om10_cat = fits.open('../../data/twinkles_lenses_v2.fits')[1].data
sprinkled_lens_gals = pd.read_csv('../../data/sprinkled_lens_galaxies_230.txt')
sprinkled_agn = pd.read_csv('../../data/sprinkled_agn_230.txt')
Explanation: Strong Lensing in Twinkles
Overview
Strong Lensing in Twinkles simulations involves taking existing CATSIM instance catalogs and replacing specific AGN within those catalogs with a Strongly Lensed AGN system. This means adding images of the original AGN and a lens galaxy into the CATSIM instance catalogs. This is done using a piece of code in the Twinkles repository that is known as the sprinkler. This notebook will first discuss the sprinkler and then move onto verifying its outputs in instance catalogs.
Sprinkler Code Review
The sprinkler code can be found here. It involves a step inserted into the final pass method of the Instance Catalog class that allows manipulation of the array of catalog results to be manipulated before printing out. This step starts with the loading of an OM10 based catalog of strongly lensed AGN systems.
The OM10-Twinkles Catalog
The catalog we are using started with the set of the 872 rung 4 lenses from the Time Delay Challenge 1 (see discussion in this issue on why these were chosen). However, when looking through this catalog we learned that sizes were missing for the foreground lens galaxies and we have chosen to use CATSIM galaxies to add this information to the catalog as well as an SED for each of these foreground galaxies. This process was completed in this issue. In the process 11 galaxies were unable to be matched adequately to CATSIM galaxies so we end up with a final catalog in Twinkles of 861 galaxies.
Selecting Sprinkled Systems
The next step after loading the lensing catalog is to go through the instance catalog AGNs one at a time and see if they match a lens system from our OM10-Twinkles catalog. The matching is done by taking the redshift and magnitude of the CATSIM AGN and selecting any available lens systems with a source AGN that has a redshift within 0.1 dex and an LSST magnorm value (magnitude in a simulated filter with one bin at 500nm) within .25 mags.
Once the possible matches to a given instance catalog AGN are found there is a chance that it will be replaced that is set by the density parameter in the sprinkler class. In the current Twinkles run it is set at 1.0 so that 198 lens systems are sprinkled into the Twinkles instance catalogs.
If the AGN is chosen to be sprinkled then we proceed with the sprinkler and if there are multiple matches from the OM10-Twinkles catalog we randomly choose one to use as we proceed. First, we remove any disc and bulge information for that galaxy from the instance catalog leaving only the AGN data and copy this AGN entry creating 2 or 4 entries in the catalog depending upon the number of images the OM10 system we are sprinkling in has in total. Next we use the OM10 data for each image to shift the location of the AGN in each image to the correct location and adjust the magnitude for each entry depending on the OM10 magnification parameter for each AGN image. At this point we also adjust the time delay for each image with OM10 info and give all the images the same source redshift from the OM10 entry. Once the new entries are ready they are appended to the instance catalog with a new instance catalog id tag that gives us the ID of the Bulge galaxy in the instance catalog to which it is associated and the ID number of the OM10-Twinkles catalog system that is used.
The next step is to take the original entry in the instance catalog and add in the information for the foreground lens galaxy. Therefore, in this step the AGN and Disc data for the original galaxy are cleared and the redshift and magnitude for the lens galaxy from the OM10 system is added into the instance catalog. Here we also add in the radius of the lens galaxy and the sed file we want to use (the information added to the original OM10 systems in the process described in the last section). We also add in the OM10 position angles and ellipticities at this point.
Once we have gone through this process for all AGN in the instance catalog then we return to the normal instance catalog processing to write the information out to file.
Verifying Instance Catalog Output
For the sections below I am using the output from an instance catalog created using Twinkles and removing the SNe entries at the bottom. I then split it up into the bulge+disk galaxy entries and the AGNs before only selecting the objects that were sprinkled. We will also load in the OM10-Twinkles catalog of strongly lensed AGN systems for validation purposes. This catalog is where all the lens systems that end up in Twinkles catalogs are stored.
Loading datasets
End of explanation
sprinkled_agn[:20]
Explanation: This file loaded contains extra information to a normal instance catalog output. During the sprinkling process as new id numbers are given to the sprinkled AGN all of their ids are larger than 1e11 while those unsprinkled are not. These larger Ids contain information on the OM10-Twinkles system used in the sprinkler, the image number in that system and the associated lens galaxy. We have previously added columns containing this information to the instance catalog information for the AGN.
End of explanation
x_sep_inst = []
y_sep_inst = []
x_sep_om10 = []
y_sep_om10 = []
for agn_row in range(len(sprinkled_agn)):
    lens_gal_current = sprinkled_agn['lens_gal_id'][agn_row]
    twinkles_current = sprinkled_agn['twinkles_id'][agn_row]
    im_num_current = sprinkled_agn['image_num'][agn_row]
    om10_current = om10_cat[om10_cat['twinklesId'] == twinkles_current]
    lens_ra = sprinkled_lens_gals.query('id == %i' % lens_gal_current)['ra'].values[0]
    lens_dec = sprinkled_lens_gals.query('id == %i' % lens_gal_current)['dec'].values[0]
    x_sep_inst.append(((sprinkled_agn['ra'][agn_row] - lens_ra)*3600.)*np.cos(np.radians(lens_dec)))
    y_sep_inst.append((sprinkled_agn['dec'][agn_row] - lens_dec)*3600)
    x_sep_om10.append(om10_current['XIMG'][0][im_num_current])
    y_sep_om10.append(om10_current['YIMG'][0][im_num_current])
fig = plt.figure(figsize=(12, 6))
fig.add_subplot(1,2,1)
p1 = plt.hist(100*(np.array(x_sep_inst) - np.array(x_sep_om10))/np.array(x_sep_om10))
plt.xlabel('Percent Difference')
plt.title('XIMG (ra)')
fig.add_subplot(1,2,2)
p2 = plt.hist(100*(np.array(y_sep_inst) - np.array(y_sep_om10))/np.array(y_sep_om10))
plt.xlabel('Percent Difference')
plt.title('YIMG (dec)')
plt.suptitle('Difference between OM10 separations and Instance Catalog separations between images and lens galaxy')
fig = plt.figure(figsize=(12, 6))
fig.add_subplot(1,2,1)
p1 = plt.hist((np.array(x_sep_inst) - np.array(x_sep_om10)))
plt.xlabel('Difference (arcsec)')
plt.title('XIMG (ra)')
fig.add_subplot(1,2,2)
p2 = plt.hist((np.array(y_sep_inst) - np.array(y_sep_om10)))
plt.xlabel('Difference (arcsec)')
plt.title('YIMG (dec)')
plt.suptitle('Difference between OM10 separations and Instance Catalog separations between images and lens galaxy')
plt.tight_layout()
plt.subplots_adjust(top=.9)
Explanation: Validating Image Positions
Our first task is to verify that the image locations in the instance catalog are in the correct places when compared to the entries from the OM10 catalog. To do this we will find the relative ra and dec positions for each of the images in a lensed system relative to the lens galaxy from OM10 and compare the instance catalog values to the appropriate OM10 system.
End of explanation
ellip = np.sqrt(1-((sprinkled_lens_gals['minor_axis']**2)/(sprinkled_lens_gals['major_axis']**2)))
sprinkled_lens_gals['r_eff'] = sprinkled_lens_gals['major_axis']*np.sqrt(1-ellip)
fig = plt.figure(figsize=(18,6))
fig.add_subplot(1,3,1)
p1 = plt.hist(sprinkled_lens_gals['mag_norm'])
plt.xlabel('MagNorm (mags @ 500 nm)')
fig.add_subplot(1,3,2)
p2 = plt.hist(sprinkled_lens_gals['redshift'])
plt.xlabel('Redshift')
fig.add_subplot(1,3,3)
p3 = plt.hist(sprinkled_lens_gals['r_eff'])
plt.xlabel('R_Eff (arcsec)')
Explanation: Overall, differences between the instance catalogs and the OM10 inputs seem to be within 0.01 arcseconds. It appears that the positions in the instance catalog are accurate reproductions of the OM10 data they are based upon.
Check distributions of Lens galaxies
In the next section we will plot the properties of the lens galaxies in the sprinkled data.
End of explanation
#Use only the brightest AGN image in a system for magnitudes and only record the redshift once for each system
agn_magnorm = {}
agn_redshifts = []
for agn_row in range(len(sprinkled_agn)):
    lens_gal_current = sprinkled_agn['lens_gal_id'][agn_row]
    im_num_current = sprinkled_agn['image_num'][agn_row]
    om10_current = om10_cat[om10_cat['twinklesId'] == twinkles_current]
    try:
        current_val = agn_magnorm[str(lens_gal_current)]
        if sprinkled_agn['mag_norm'][agn_row] < current_val:
            agn_magnorm[str(lens_gal_current)] = sprinkled_agn['mag_norm'][agn_row]
    except KeyError:
        agn_magnorm[str(lens_gal_current)] = sprinkled_agn['mag_norm'][agn_row]
        agn_redshifts.append(sprinkled_agn['redshift'][agn_row])
fig = plt.figure(figsize=(12,6))
fig.add_subplot(1,2,1)
plt.hist(agn_magnorm.values())
plt.title('Magnitude for brightest AGN image in a system')
plt.xlabel('MagNorm (mags @ 500 nm)')
fig.add_subplot(1,2,2)
plt.hist(agn_redshifts)
plt.title('Lensed AGN Redshifts')
plt.xlabel('Source Redshifts')
plt.suptitle('Lensed AGN Properties')
plt.tight_layout()
plt.subplots_adjust(top=.9)
Explanation: Check distributions of Lensed AGN
Now we plot the properties of the lensed AGN sprinkled into the instance catalogs.
End of explanation
sprinkled_agn_2 = pd.read_csv('../../data/sprinkled_agn_185614.txt')
sprinkled_agn_3 = pd.read_csv('../../data/sprinkled_agn_204392.txt')
sprinkled_agn['mag_norm_185614'] = sprinkled_agn_2['mag_norm']
sprinkled_agn['mag_norm_204392'] = sprinkled_agn_3['mag_norm']
#Check if the mag_norm value stays the same from the first visit to the second for any AGN image
for agn_row in range(len(sprinkled_agn)):
    if sprinkled_agn['mag_norm'][agn_row] == sprinkled_agn['mag_norm_185614'][agn_row]:
        print agn_row
Explanation: Varaibility Validation
Finally, we take a look to make sure that the magnitudes for the AGN are not static. We do that by looking at the AGN in two more r band visits from the Twinkles simulation.
End of explanation
fig = plt.figure(figsize=(12,12))
for agn_row in range(10):
    plt.plot([59580.1, 59825.3, 59857.2], [sprinkled_agn['mag_norm'][agn_row], 
                            sprinkled_agn['mag_norm_185614'][agn_row], 
                            sprinkled_agn['mag_norm_204392'][agn_row]], marker='+')
plt.xlabel('Visit MJD')
plt.ylabel('MagNorm (mags @ 500 nm)')
plt.title('AGN Image MagNorms for the first three r-band visits for 5 doubly-lensed AGN systems')
plt.tight_layout()
Explanation: As the previous cell shows there are no AGN with the same magnitude from the first visit to the next showing that variability is being calculated for each system. Below we plot a few light curves on the set of three visits.
End of explanation
<END_TASK>