<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 SS82
Talking to Bruno about his project on stacking Swift observations and my project on Stripe82 SED we started to think about a collaboration to create a set of deep observations with Swift data. Deep Swift data being a great addition to the High-Energy end of Stripe82 data collection (see, for instance, LaMassa,2016).
Bruno then seached for all Swift-XRT observations inside the Stripe-82
Step1:  The base catalog
Right below, in Table 1, we can see a sample of the catalog (the first/last five lines).
In Table 2, we see a brief description of the catalog.
Step2:  Target_Name is the name of the (central) object at each observation, from that we see we have 681 unique sources out of the 3035 observations. GroupSize is the number of overlapping observations, the average number is ~54. Let's see how sparse are the observations in time and how do they distribute for each source.
Step3:  Number of observations
To have a glue about the number of observations done over each object we can look the counts shown by Table 3 and the histogram below (Figure 2).
Step4:  Filtering the data
First, a closer look to an example
To have a better idea of what we should find regarding the observation time of these sources, I'll take a particular one -- V1647ORI -- and see what we have for this source.
Step5:  If we consider each group of observations of our interest -- let me call them "chunk" -- observations that distance each other no more than "X" days (for example, X=20 days) we see from this example that it happens to exist more than one "chunk" of observations per object. Here, for instance, rows 347,344,343,346 and 338,339,336,335,341 form the cluster of observations of our interest, "chunk-1" and "chunk-2", respectively.
To select the candidates we need to run a window function over the 'start_time' sorted list, where the function has two elements (i.e, observations) to ask their distance in time. If the pair of observations is less than, say 20 days, they are selected for future processing.
Applying the filter to all objects
Now defining a 20 days window as the selection criterium to all objects in our catalog we end up with 2254 observations, done over 320 objects.
Table 5 add such information through column "obs_chunk", where "Not-Available" value means the observations that have not succeed in the filtering applied.
Note
Step6:  Filtered catalog
And here is the final catalog, where rows (i.e, observations) out of our interest are removed.
Out of our interest are the entries with an xrt_exposure shorter than 200 seconds
and individual observations (i.e, "obs_chunk == NaN").
This catalog is written to 'Swift_Master_Stripe82_groups_filtered.csv'.
<ASSISTANT_TASK:>
Python Code:
from IPython.display import HTML
HTML('''<script>
code_show=true; 
function code_toggle() {
 if (code_show){
 $('div.input').hide();
 } else {
 $('div.input').show();
 }
 code_show = !code_show
} 
$( document ).ready(code_toggle);
</script>
<form action="javascript:code_toggle()"><input type="submit" value="Click here to toggle on/off the raw code."></form>''')
from IPython.display import HTML
HTML('''
<figure>
  <img src="Stripe82_gal_projection.png" alt="Swift observations over Stripe82">
  <figcaption>Figure 1: Swift observations over Stripe82</figcaption>
</figure>
''')
Explanation: SS82
Talking to Bruno about his project on stacking Swift observations and my project on Stripe82 SED we started to think about a collaboration to create a set of deep observations with Swift data. Deep Swift data being a great addition to the High-Energy end of Stripe82 data collection (see, for instance, LaMassa,2016).
Bruno then seached for all Swift-XRT observations inside the Stripe-82:
* RA: 310 : 60
* Dec: -1.25 : 1.25
Over the Stripe, Bruno has found ~3000 observations. See Figure 1 and tables Table 1 and Table 2.
Here, I'll do the filtering of the observations to keep only those useful for Paolo's stacking.
The selection looks for observations done within a time-range of a few days; for instance, 20 days is the window size I'll use here.
If all you want is to have a look at the final/filtered catalog, go straight to final section.
Otherwise, if the code used in this filtering does matter to you, you can show them out clicking the button below.
End of explanation
import pandas
cat = pandas.read_csv('Swift_Master_Stripe82_groups.ascii',
                     delim_whitespace=True)
print "Table 1: Sample of the catalog"
pandas.concat([cat.head(5),cat.tail(5)])
print "Table 2: Summary of the catalog columns"
cat.describe(include='all')
Explanation: The base catalog
Right below, in Table 1, we can see a sample of the catalog (the first/last five lines).
In Table 2, we see a brief description of the catalog.
End of explanation
cat['start_time'] = pandas.to_datetime(cat['start_time'])
cat_grouped_by_target = cat[['Target_Name','start_time']].groupby(['Target_Name'])
cat_descr = cat_grouped_by_target.describe().unstack()
cat_time = cat_descr.sort_values([('start_time','count')],ascending=False)
del cat_descr
Explanation: Target_Name is the name of the (central) object at each observation, from that we see we have 681 unique sources out of the 3035 observations. GroupSize is the number of overlapping observations, the average number is ~54. Let's see how sparse are the observations in time and how do they distribute for each source.
End of explanation
title = "Figure 2: Number of sources(Y axis) observed number of times(X axis)"
%matplotlib inline
from matplotlib import pyplot as plt
width = 16
height = 4
plt.figure(figsize=(width, height))
yticks = [2,10,50,100,200,300]
xticks = range(51)
ax = cat_time[('start_time','count')].plot.hist(bins=xticks,xlim=(0,50),title=title,grid=True,xticks=xticks,yticks=yticks,align='left')
ax.set_xlabel('Number of observations (per source)')
print "Table 3: Number counts and dates (first/last) of the observations (per object)"
cat_time
Explanation: Number of observations
To have a glue about the number of observations done over each object we can look the counts shown by Table 3 and the histogram below (Figure 2).
End of explanation
print "Table 4: Observation carried out for source 'V1647ORI' sorted in time"
g = cat_grouped_by_target.get_group('V1647ORI')
g_sorted = g.sort_values('start_time')
g_sorted
Explanation: Filtering the data
First, a closer look to an example
To have a better idea of what we should find regarding the observation time of these sources, I'll take a particular one -- V1647ORI -- and see what we have for this source.
End of explanation
def find_clustered_observations(sorted_target_observations,time_range=10):
    # Let's select a 'time_range' days window to select valid observations
    window_size = time_range
    g_sorted = sorted_target_observations
    
    # an ordered dictionary works as a 'set' structure
    from collections import OrderedDict
    selected_allObs = OrderedDict()
    # define en identificator for each cluster of observations, to ease future filtering
    group_obs = 1
    _last_time = None
    _last_id = None
    for _row in g_sorted.iterrows():
        ind,row = _row
        if _last_time is None:
            _last_time = row.start_time
            _last_id = ind
            continue
        _delta = row.start_time - _last_time
        if _delta.days <= window_size:
            selected_allObs[_last_id] = group_obs
            selected_allObs[ind] = group_obs
        else:
            if len(selected_allObs):
                group_obs = selected_allObs.values()[-1] + 1
        _last_time = row.start_time
        _last_id = ind
    return selected_allObs
from collections import OrderedDict
obs_indx = OrderedDict()
for name,group in cat_grouped_by_target:
    g_sorted = group.sort_values('start_time')
    filtered_indxs = find_clustered_observations(g_sorted,time_range=20)
    obs_indx.update(filtered_indxs)
import pandas
obsChunks_forFilteringCat = pandas.DataFrame(obs_indx.values(),columns=['obs_chunk'],index=obs_indx.keys())
# obsChunks_forFilteringCat.sort_index()
print "Table 5: original catalog with column 'obs_chunk' to flag which rows succeed the filtering (non-NA values)."
cat_with_obsChunksFlag = cat.join(obsChunks_forFilteringCat)
# cols = list(cat_with_obsChunksFlag.columns)
# cols.insert(2,cols.pop(-1))
# cat_with_obsChunksFlag = cat_with_obsChunksFlag.ix[:,cols]
cat_with_obsChunksFlag
Explanation: If we consider each group of observations of our interest -- let me call them "chunk" -- observations that distance each other no more than "X" days (for example, X=20 days) we see from this example that it happens to exist more than one "chunk" of observations per object. Here, for instance, rows 347,344,343,346 and 338,339,336,335,341 form the cluster of observations of our interest, "chunk-1" and "chunk-2", respectively.
To select the candidates we need to run a window function over the 'start_time' sorted list, where the function has two elements (i.e, observations) to ask their distance in time. If the pair of observations is less than, say 20 days, they are selected for future processing.
Applying the filter to all objects
Now defining a 20 days window as the selection criterium to all objects in our catalog we end up with 2254 observations, done over 320 objects.
Table 5 add such information through column "obs_chunk", where "Not-Available" value means the observations that have not succeed in the filtering applied.
Note: obs_chunk values mean the groupings -- "chunks" -- formed within each object's set of observations. They are unique among each object's observations, but not accross the entire catalog.
End of explanation
cat_filtered = cat_with_obsChunksFlag.dropna(subset=['obs_chunk'])
cat_filtered
cat_filtered = cat_filtered.loc[cat_filtered.xrt_exposure >= 200]
cat_filtered
cat_filtered.describe(include='all')
cat_filtered.to_csv('Swift_Master_Stripe82_groups_filtered.csv')
cat_filtered_minimum = cat_filtered[['obsid','RA_(J2000)','Dec_(J2000)']]
cat_filtered_minimum.index.name = 'index'
cat_filtered_minimum.to_csv('Swift_Master_Stripe82_filtered_positions.csv')
%qtconsole
Explanation: Filtered catalog
And here is the final catalog, where rows (i.e, observations) out of our interest are removed.
Out of our interest are the entries with an xrt_exposure shorter than 200 seconds
and individual observations (i.e, "obs_chunk == NaN").
This catalog is written to 'Swift_Master_Stripe82_groups_filtered.csv'.
End of explanation
<END_TASK>