<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Turn magnitudes into colors
Step1:  Filter out bad data
Step2:  Create classification labels
Step3:  Load the IDs of the narrowband population
Step4:  Setup locations of images
Step5:  Copy over pre-downloaded images
Step6:  Get the images from the quarry
For technical details, see
Step7:  Make the request via curl
1)
First you need to setup you authentication information. Add it to a file like galaxy_images_training/curl_netrc which should look like
Step8:  Remove incomplete dirs, then transfer to main google drive directory
Step9:  Now link these new images to the project data directory
Step10:  Check that the directory contents are correct
<ASSISTANT_TASK:>
Python Code:
combined["g_minus_r"] = combined.gcmodel_mag - combined.rcmodel_mag
combined["r_minus_i"] = combined.rcmodel_mag - combined.icmodel_mag
combined["i_minus_z"] = combined.icmodel_mag - combined.zcmodel_mag
combined["z_minus_y"] = combined.zcmodel_mag - combined.ycmodel_mag
Explanation: Turn magnitudes into colors
End of explanation
mask =    np.isfinite(combined["g_minus_r"]) & np.isfinite(combined["r_minus_i"]) \
        & np.isfinite(combined["i_minus_z"]) & np.isfinite(combined["z_minus_y"]) \
        & np.isfinite(combined["icmodel_mag"]) \
        & (~combined.gcmodel_flux_flags) & (~combined.rcmodel_flux_flags) \
        & (~combined.icmodel_flux_flags) & (~combined.zcmodel_flux_flags) \
        & (~combined.ycmodel_flux_flags)
combined = combined[mask]
Explanation: Filter out bad data
End of explanation
low_z    = (combined.photo_z  < .15)
low_mass = (combined.log_mass > 8) & (combined.log_mass < 9)
combined["low_z_low_mass"] = (low_z & low_mass)
combined.low_z_low_mass.mean()
Explanation: Create classification labels
End of explanation
target_ids = pd.read_csv("../narrowband_deconfusion/target_galaxies-HSC_ids.csv")
target_ids.head()
contaminant_ids = pd.read_csv("../narrowband_deconfusion/contaminant_galaxies-HSC_ids.csv")
contaminant_ids.head()
Explanation: Load the IDs of the narrowband population
End of explanation
targets_path = pathlib.Path.home() / "dwarfz" \
                                   / "galaxies_narrowband" \
                                   / "target"
targets_path.mkdir(parents=True, exist_ok=True)
contaminants_path = pathlib.Path.home() / "dwarfz" \
                                        / "galaxies_narrowband" \
                                        / "contaminant"
contaminants_path.mkdir(parents=True, exist_ok=True)
Explanation: Setup locations of images
End of explanation
google_dir = pathlib.Path.home() / "Google Drive" \
                                 / "HSC_COSMOS_galaxies" \
                                 / "galaxies"
predownloaded_ids = {int(image_dir.name)
                     for image_dir in google_dir.iterdir()
                     if image_dir.is_dir()}
i = 0
for (_, _, HSC_id) in target_ids.itertuples():
    if HSC_id in predownloaded_ids:
        image_dir = google_dir / str(HSC_id)
        new_dir = targets_path / image_dir.name
        if not new_dir.is_dir():
            new_dir.symlink_to(image_dir, target_is_directory=True)
        i += 1
        
print("symlinked {} galaxies".format(i))
j = 0
for (_, _, HSC_id) in contaminant_ids.itertuples():
    if HSC_id in predownloaded_ids:
        image_dir = google_dir / str(HSC_id)
        new_dir = contaminants_path / image_dir.name
        if not new_dir.is_dir():
            new_dir.symlink_to(image_dir, target_is_directory=True)
        j += 1
        
print("symlinked {} galaxies".format(j))
# num galaxies remaining to download
target_ids.shape[0] + contaminant_ids.shape[0] - i - j
target_ids_to_download = set(target_ids.HSC_id) - predownloaded_ids
contaminant_ids_to_download = set(contaminant_ids.HSC_id) - predownloaded_ids
ids_to_download = target_ids_to_download |  contaminant_ids_to_download
Explanation: Copy over pre-downloaded images
End of explanation
galaxy_coords = combined[["catalog_2_ids", "ra", "dec"]]
galaxy_coords = galaxy_coords.rename(columns={"catalog_2_ids":"HSC_index"})
galaxy_coords = galaxy_coords.set_index("HSC_index")
galaxy_coords = galaxy_coords.loc[ids_to_download]
galaxy_coords.head()
width = "20asec"
filters = ["HSC-G", "HSC-R", "HSC-I", "HSC-Z", "HSC-Y"]
rerun = "pdr1_deep"
quarry_input_dir = pathlib.Path("galaxy_images_training") \
    / "quarry_input_files"
quarry_input_dir.mkdir(exist_ok=True)
quarry_name_format = "tmp_quarry_{:>04d}.txt"
batch_i = 0
files_in_batch = 0
max_files_per_batch = 1000
tmp_filename = quarry_input_dir / quarry_name_format.format(batch_i)
f = open(tmp_filename, mode="w")
print("#?     ra          dec       filter   sw      sh      rerun", file=f)
print_formatter = " {galaxy.ra:.6f}deg  {galaxy.dec:.6f}deg  {filter}  {width}  {width}  {rerun} # {galaxy.Index}"
for galaxy in galaxy_coords.itertuples():
    for filter in filters:            
        print(print_formatter.format(galaxy=galaxy,
                                     width=width,
                                     filter=filter,
                                     rerun=rerun),
              file=f)
        files_in_batch += 1
        if files_in_batch == max_files_per_batch:
            f.close()
            files_in_batch = 0
            batch_i += 1
            tmp_filename = quarry_input_dir / quarry_name_format.format(batch_i)
            f = open(tmp_filename, mode="w")
            print("#?     ra          dec       filter   sw      sh      rerun", file=f)
f.close()
!head -n 10 $tmp_filename
!wc -l $tmp_filename
!ls  galaxy_images_training/quarry_input_files/ | wc -l
!ls -lh galaxy_images_training/quarry_input_files/ | head -n 10
Explanation: Get the images from the quarry
For technical details, see: https://hsc-release.mtk.nao.ac.jp/das_quarry/manual.html
I'll be downloading these directly into the google drive folder. Then, when everything is complete, I'll just symlink them into the appropriate project folders, split by target and contaminant.
Create a coordinates list
End of explanation
filenames = sorted(quarry_input_dir.iterdir())
min_batch_number_to_pull = 1
max_batch_number_to_pull = 100
new_data_dir = targets_path.parent / "staging"
new_data_dir.mkdir(exist_ok=True)
for i, filename in enumerate(filenames):
    if i < min_batch_number_to_pull: 
        continue
    if i >= max_batch_number_to_pull:
        break
    print("Currently processing file: {}".format(os.path.basename(filename)), end="\r", flush=True)
    os.system(("curl -k --netrc-file galaxy_images_training/curl_netrc "
               "https://hsc-release.mtk.nao.ac.jp/das_quarry/cgi-bin/quarryImage "
               "--form list=@{filename} "
               "| tar -xvf -").format(filename=filename))
    arch_dirs = list(pathlib.Path.cwd().glob("arch-*"))
    assert(len(arch_dirs)==1)
    arch_dir = arch_dirs[0]
    with open(filename, "r") as f:
        _ = f.readline() # skip header
        line_number = 1 # 1 indexed, and then also with header
        for line in f:
            line_number += 1
            HSC_id = int(line.split("#")[-1].strip())
            HSC_dir = new_data_dir / str(HSC_id)
            HSC_dir.mkdir(exist_ok=True)
            
            image_filenames = list(arch_dir.glob( 
                str(line_number) + "-cutout-HSC-?-????-pdr1_deep.fits"
            ))
            
            if len(image_filenames) == 0:
                continue
            elif len(image_filenames) >1:
                raise RuntimeError("Too many files for line {} id {}".format(
                    line_number, HSC_id,
                ))
            image_filename = image_filenames[0]
            
            # rename with HSC id and move to within `new_data_dir`
            image_filename.rename(
                HSC_dir / image_filename.name.replace(
                    "{}-cutout".format(line_number),
                    "{}-cutout".format(HSC_id),
                )
            )
            
    arch_dir.rmdir()
Explanation: Make the request via curl
1)
First you need to setup you authentication information. Add it to a file like galaxy_images_training/curl_netrc which should look like:
machine hsc-release.mtk.nao.ac.jp login &lt;your username&gt; password &lt;your password&gt;
This allows you to script the curl calls, without being prompted for your password each time
2a) 
The curl call (in (2b)) will spit out files into a somewhat unpredicatably named directory, like arch-170928-231223. You should rename this to match the batch suffix. You really should do this right away, so you don't get confused. In general I add the rename onto the same line as the curl call:
curl ... | tar xvf - &amp;&amp; mv arch-* quarry_files_a
This only works if it finds one arch- directory, but you really shouldn't have multiple arch directories at any given time; that's a recipe for getting your galaxies mixed up.
2b) 
Here's the actual curl invocation:
curl --netrc-file galaxy_images_training/curl_netrc https://hsc-release.mtk.nao.ac.jp/das_quarry/cgi-bin/quarryImage --form list=@&lt;coord list filename&gt; | tar xvf -
End of explanation
staging_dir = google_dir.parent / "staging"
num_removed = 0
for staged_dir in staging_dir.iterdir():
    if not staged_dir.is_dir(): continue
    
    num_images = len({*staged_dir.glob("*.fits")})
    if num_images>5:
        raise ValueError("{} has {} fits files".format(staged_dir, num_images))
    elif num_images < 5:
        print("too few images in {} (n={}); removing".format(
            staged_dir,
            num_images,
        ))
        num_removed += 1
        send2trash.send2trash(str(staged_dir))
    else:
        staged_dir.rename(staged_dir.parent.parent / "galaxies" / staged_dir.name)
num_removed
Explanation: Remove incomplete dirs, then transfer to main google drive directory
End of explanation
pre_linked_ids  = {int(path.name) for path in contaminants_path.iterdir() if path.is_dir()} 
pre_linked_ids |= {int(path.name) for path in targets_path.iterdir() if path.is_dir()} 
len(pre_linked_ids)
narrowband_ids = set(target_ids.HSC_id) | set(contaminant_ids.HSC_id)
len(narrowband_ids)
all_downloaded_ids = {int(path.name) for path in google_dir.iterdir()
                      if path.is_dir()}
len(all_downloaded_ids)
num_to_link = 0
already_linked = 0
missing = 0
for HSC_id in narrowband_ids:
    if HSC_id in pre_linked_ids:
        already_linked += 1
    if HSC_id not in all_downloaded_ids:
        missing += 1
        
    if HSC_id in target_ids.HSC_id.values:
        class_path = targets_path
    elif HSC_id in contaminant_ids.HSC_id.values:
        class_path = contaminants_path
    else:
        raise ValueError("HSC id {} in neither targets nor contaminants".format(HSC_id))
        
    image_dir = google_dir / str(HSC_id)
    new_dir = class_path / image_dir.name
    if not new_dir.is_dir():
#         new_dir.symlink_to(image_dir, target_is_directory=True)
        pass
        
    num_to_link += 1
    
    
print("just linked: ", num_to_link)
print("previously linked: ", already_linked)
print("missing: ", missing)
Explanation: Now link these new images to the project data directory
End of explanation
for path in targets_path.iterdir():
    if not path.is_dir():
        continue
    
    HSC_id = int(path.name)
    if HSC_id not in target_ids.HSC_id.values:
        raise ValueError("HSC id {} should not be in target path".format(HSC_id))
for path in contaminants_path.iterdir():
    if not path.is_dir():
        continue
    
    HSC_id = int(path.name)
    if HSC_id not in contaminant_ids.HSC_id.values:
        raise ValueError("HSC id {} should not be in contaminant path".format(HSC_id))
Explanation: Check that the directory contents are correct
End of explanation
<END_TASK>