<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Target selection bits and bitmasks
Author
Step1:  The mask contains the name of the target bit (e.g. ELG) the bit value to which that name corresponds (e.g. 1, meaning 2-to-the-power-1), a description of the target (e.g. "ELG") and a dictionary of values that contain information for fiber assignment, such as the observing conditions allowed for the target, the initial priority with which the target class should be observed, and the initial number of observations for the target class. Note that these bits of information can be accessed individually in a number of ways
Step2:  There are corresponding masks for the BGS and MWS, which can be accessed in the same way, e.g.
Step3:  The Commissioning and Survey Validation bitmasks
In addition to the DESI Main Survey, desitarget produces targets for Commissioning ("CMX") and Survey Validation ("SV"). The CMX and SV bitmasks can be obtained and examined as follows (other manipulation of these masks is similar to the previous sub-section, above)
Step4:  Using the bitmasks to understand a file of targets
Important aside!!!
Target classes have evolved throughout the history of the desitarget code, and the bits that correspond to those targets have thus occasionally changed. It is therefore critical that you use the same version of desitarget when working with bits in a target file as was used to create that target file!
For example, say you are working with commissioning targets that were created with version 0.X.X of desitarget. The correct version of Git to use to study this file can be obtained via
Step5:  Note that if you took the file from my the examples directory, then you're using an example file that only contains a subset of columns.
Step6:  Let's consider the value of DESI_TARGET for the forty-second target
Step7:  What does this number mean? Well, let's see which target classes are defined by this integer
Step8:  Now let's see what target classes are include for the first 10 targets
Step9:  So far, we've looked at the target class for each target. Now, let's just extract target classes that correspond to a certain bit. For example, which of the first 10 targets have the 'BGS_ANY' bit set?
Step10:  Which of all of the targets are both ELG and quasar targets?
Step11:  Alternatively, more compactly"
Step12:  You should note that the forty-second target studied above pops up in these lists!
Note that desi_mask contains a couple of special bits that simply denote whether a target is a BGS or MWS target. These are called BGS_ANY and MWS_ANY. For example
Step13:  Bits representing targets for the Bright Galaxy Survey and Milky Way Survey can be manipulated in the same way as previous examples in this section. The relevant columns and masks are BGS_TARGET and bgs_mask, and MWS_TARGET and mws_mask respectively. For example
Step14:  Working with target files for CMX or SV
As noted in the previous section, Commissioning and SV have different bitmasks. Conveniently, commissioning and SV also have different _TARGET column names, allowing a user to easily distinguish which "flavor" of file they are using
Step15:  Let's see what would happen if our targets file was actually an SV file
Step16:  An advanced example
As a challenge, let's try to find all quasar targets that are close to an LRG target using our example file of targets.
First, let's retrieve all LRG and QSO targets from our file.
Step17:  We'll need the astropy spatial matching functions
Step18:  Convert the lrgs and quasars to SkyCoord objects
Step19:  Perform the match. Let's choose a radius of 1 arcminute
Step20:  Finally, write out the matching lrgs and quasars, and the distance between them
<ASSISTANT_TASK:>
Python Code:
from desitarget.targets import desi_mask, bgs_mask, mws_mask
print(desi_mask)
Explanation: Target selection bits and bitmasks
Author: Adam D. Myers, University of Wyoming
This Notebook describes how to work with target selection bitmasks for DESI.
Setting up your environment
First, ensure that your environment matches a standard DESI environment. For example:
module unload desimodules
source /project/projectdirs/desi/software/desi_environment.sh 18.7
desitarget relies on desiutil and desimodel, so you may also need to set up a wider DESI environment, as detailed at:
https://desi.lbl.gov/trac/wiki/Pipeline/GettingStarted/Laptop/JuneMeeting
It may also be useful to set up some additional environment variables that are used in some versions of the desitarget code (you could also place these in your .bash_profile.ext file):
export DESIMODEL=$HOME/git/desimodel
export DUST_DIR=/project/projectdirs/desi/software/edison/dust/v0_1/maps
export GAIA_DIR=/project/projectdirs/desi/target/gaia_dr2
Here, I've set DESIMODEL to a reasonable location. For a more detailed description of checking out the desimodel data files from svn see:
https://desi.lbl.gov/trac/wiki/Pipeline/GettingStarted/Laptop/JuneMeeting#Datafilesfordesimodel
Understanding the desitarget bitmasks
Main Survey bitmasks
The critical values that select_targets produces are the DESI_TARGET, BGS_TARGET and MWS_TARGET bit masks, which contain the target bits for the DESI main (or "dark time") survey and the Bright Galaxy Survey and Milky Way Survey respectively. Let's examine the masks that correspond to these surveys.
End of explanation
desi_mask["QSO"], desi_mask.QSO # ADM different ways of accessing the bit values.
desi_mask.names() # ADM the names of each target type.
desi_mask.names(7) # ADM the names of target classes that correspond to an integer value of 5.
# ADM note that 7 is 2**0 + 2**1 + 2**2.
desi_mask.bitnum("SKY") # ADM the integer value that corresponds to the "SKY" bit.
names = desi_mask.names()
bitnums = [desi_mask.bitnum(name) for name in names]
bitvals = [desi_mask[name] for name in names]
list(zip(names,bitnums,bitvals)) # ADM the bit and integer value for each defined name.
desi_mask["LRG"].priorities # ADM a dictionary of initial priorities for the LRG target class.
desi_mask["LRG"].obsconditions, desi_mask["LRG"].numobs, desi_mask["LRG"].priorities["MORE_ZGOOD"]
Explanation: The mask contains the name of the target bit (e.g. ELG) the bit value to which that name corresponds (e.g. 1, meaning 2-to-the-power-1), a description of the target (e.g. "ELG") and a dictionary of values that contain information for fiber assignment, such as the observing conditions allowed for the target, the initial priority with which the target class should be observed, and the initial number of observations for the target class. Note that these bits of information can be accessed individually in a number of ways:
End of explanation
bgs_mask.names()
mws_mask.names()
Explanation: There are corresponding masks for the BGS and MWS, which can be accessed in the same way, e.g.:
End of explanation
from desitarget.cmx.cmx_targetmask import cmx_mask
print(cmx_mask)
from desitarget.sv1.sv1_targetmask import desi_mask, bgs_mask, mws_mask
desi_mask.names()
Explanation: The Commissioning and Survey Validation bitmasks
In addition to the DESI Main Survey, desitarget produces targets for Commissioning ("CMX") and Survey Validation ("SV"). The CMX and SV bitmasks can be obtained and examined as follows (other manipulation of these masks is similar to the previous sub-section, above):
End of explanation
import os
from glob import glob
from astropy.io.fits import getdata
import numpy as np
# ADM replace this with any directory you know of that holds targets.
targdir = "/project/projectdirs/desi/target/catalogs/examples"
# ADM replace this with the name of any target file.
targfile = 'targets.fits'
targfile = os.path.join(targdir, targfile)
targs = getdata(targfile)
Explanation: Using the bitmasks to understand a file of targets
Important aside!!!
Target classes have evolved throughout the history of the desitarget code, and the bits that correspond to those targets have thus occasionally changed. It is therefore critical that you use the same version of desitarget when working with bits in a target file as was used to create that target file!
For example, say you are working with commissioning targets that were created with version 0.X.X of desitarget. The correct version of Git to use to study this file can be obtained via:
git checkout 0.X.X
and the corresponding .yaml file online on GitHub would be:
https://github.com/desihub/desitarget/blob/0.X.X/py/desitarget/cmx/data/cmx_targetmask.yaml
For example, for version 0.31.1 of desitarget issue:
git checkout 0.31.1
or look at:
https://github.com/desihub/desitarget/blob/0.31.1/py/desitarget/cmx/data/cmx_targetmask.yaml
Equivalently, for version 0.31.1 of desitarget for SV or the Main Survey:
https://github.com/desihub/desitarget/blob/0.31.1/py/desitarget/sv1/data/sv1_targetmask.yaml
https://github.com/desihub/desitarget/blob/0.31.1/py/desitarget/data/targetmask.yaml
Commissioning targeting bits are expected to be final as of version 0.32.0 of desitarget. SV and Main Survey bits may not yet be final.
Working with target files
The target files produced by select_targets contain many quantities from the Legacy Surveys data model sweeps files at, e.g.:
http://www.legacysurvey.org/dr7/files/#sweep-7-0-sweep-brickmin-brickmax-fits
The main columns added by select_targets are DESI_TARGET, BGS_TARGET and MWS_TARGET, which contain the output bitmasks from target selection. Let's take a closer look at how these columns can be used in conjunction with the bitmasks.
First, enter the Python prompt. Now, let's read in a file of targets. I'll assume you're working at NERSC, but set targdir, below, to wherever you have a targets- file.
End of explanation
print(targs.dtype)
Explanation: Note that if you took the file from my the examples directory, then you're using an example file that only contains a subset of columns.
End of explanation
targ = targs[41]
print(targ["DESI_TARGET"])
Explanation: Let's consider the value of DESI_TARGET for the forty-second target:
End of explanation
from desitarget.targets import desi_mask
desi_mask.names(targ["DESI_TARGET"])
Explanation: What does this number mean? Well, let's see which target classes are defined by this integer:
End of explanation
bitnames = np.array(desi_mask.names())  # ADM note the array conversion to help manipulation.
bitvals = [desi_mask[name] for name in bitnames]
for targ in targs[:10]:
    w = np.where( (targ["DESI_TARGET"] & bitvals) != 0)[0]
    print(targ["DESI_TARGET"], bitnames[w])
Explanation: Now let's see what target classes are include for the first 10 targets:
End of explanation
np.where((targs[:10]["DESI_TARGET"] & desi_mask["BGS_ANY"]) != 0)[0]
Explanation: So far, we've looked at the target class for each target. Now, let's just extract target classes that correspond to a certain bit. For example, which of the first 10 targets have the 'BGS_ANY' bit set?
End of explanation
isELG = (targs["DESI_TARGET"] & desi_mask["ELG"]) != 0
isQSO = (targs["DESI_TARGET"] & desi_mask["QSO"]) != 0
np.where(isELG & isQSO)[0]
Explanation: Which of all of the targets are both ELG and quasar targets?
End of explanation
bitvalboth = desi_mask["ELG"] + desi_mask["QSO"]
np.where(targs["DESI_TARGET"] & bitvalboth == bitvalboth)[0]
Explanation: Alternatively, more compactly"
End of explanation
print((targs[:10]["DESI_TARGET"] & desi_mask["BGS_ANY"]) != 0)
print(targs[:10]["BGS_TARGET"] != 0)
Explanation: You should note that the forty-second target studied above pops up in these lists!
Note that desi_mask contains a couple of special bits that simply denote whether a target is a BGS or MWS target. These are called BGS_ANY and MWS_ANY. For example:
End of explanation
from desitarget.targets import bgs_mask, mws_mask
bitnames = np.array(bgs_mask.names())  # ADM note the array conversion to help manipulation.
bitvals = [bgs_mask[name] for name in bitnames]
for targ in targs[:10]:
    w = np.where( (targ["BGS_TARGET"] & bitvals) != 0)[0]
    print(targ["BGS_TARGET"], bitnames[w])
Explanation: Bits representing targets for the Bright Galaxy Survey and Milky Way Survey can be manipulated in the same way as previous examples in this section. The relevant columns and masks are BGS_TARGET and bgs_mask, and MWS_TARGET and mws_mask respectively. For example:
End of explanation
import os, fitsio
# ADM replace this with any directory you know of that holds targets.
targdir = "/project/projectdirs/desi/target/catalogs/examples"
# ADM replace this with the name of any target file.
targfile = 'targets.fits'
targfile = os.path.join(targdir, targfile)
targs = fitsio.read(targfile)
# ADM load the convenient utility and use it.
from desitarget.targets import main_cmx_or_sv
[desi_target, bgs_target, mws_target], [desi_mask, bgs_mask, mws_mask], surv = main_cmx_or_sv(targs)
print(desi_target, mws_target)
print(surv)
print(bgs_mask)
Explanation: Working with target files for CMX or SV
As noted in the previous section, Commissioning and SV have different bitmasks. Conveniently, commissioning and SV also have different _TARGET column names, allowing a user to easily distinguish which "flavor" of file they are using:
Main Survey files have the columns DESI_TARGET, BGS_TARGET and MWS_TARGET.
Commissioning files have the column CMX_TARGET.
SV files have the columns SV1_DESI_TARGET, SV1_BGS_TARGET and SV1_MWS_TARGET.
A convenient utility is desitarget.targets.main_cmx_or_sv which will use the differing column names to load the appropriate mask or masks. For example, using our Main Survey example file:
End of explanation
import numpy.lib.recfunctions as rfn
for col in [desi_target, bgs_target, mws_target]:
    sv1_targs = rfn.rename_fields(targs, {col: 'SV1_'+col})
[desi_target, bgs_target, mws_target], [desi_mask, bgs_mask, mws_mask], surv = main_cmx_or_sv(sv1_targs)
print(bgs_target, mws_target)
print(surv)
print(desi_mask)
Explanation: Let's see what would happen if our targets file was actually an SV file:
End of explanation
from desitarget.targets import desi_mask
isLRG = (targs["DESI_TARGET"] & desi_mask["LRG"]) != 0
isQSO = (targs["DESI_TARGET"] & desi_mask["QSO"]) != 0
lrgs, qsos = targs[isLRG], targs[isQSO]
# ADM a sanity check.
for qso in qsos[:10]:
    print(desi_mask.names(qso["DESI_TARGET"]))
Explanation: An advanced example
As a challenge, let's try to find all quasar targets that are close to an LRG target using our example file of targets.
First, let's retrieve all LRG and QSO targets from our file.
End of explanation
from astropy.coordinates import SkyCoord
from astropy import units as u
Explanation: We'll need the astropy spatial matching functions:
End of explanation
clrgs = SkyCoord(lrgs["RA"], lrgs["DEC"], unit='degree')
cqsos = SkyCoord(qsos["RA"], qsos["DEC"], unit='degree')
Explanation: Convert the lrgs and quasars to SkyCoord objects:
End of explanation
matchrad = 20*u.arcsec
idlrgs, idqsos, sep, _ = cqsos.search_around_sky(clrgs, matchrad)
Explanation: Perform the match. Let's choose a radius of 1 arcminute:
End of explanation
lrgmatch, qsomatch = lrgs[idlrgs], qsos[idqsos]
for i in range(len(lrgmatch)):
    print("LRG coordinates: {:.4f} deg, {:.4f} deg".format(lrgmatch[i]["RA"], lrgmatch[i]["DEC"]))
    print("QSO coordinates: {:.4f} deg, {:.4f} deg".format(qsomatch[i]["RA"], qsomatch[i]["DEC"]))
    print("Angular separation: {:.4f} arcsec".format(sep.value[i]*3600))
Explanation: Finally, write out the matching lrgs and quasars, and the distance between them:
End of explanation
<END_TASK>