<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
Step1:  Getting data into Python
Outline
Step2:  Using numpy.loadtxt
Step3:  Using astropy.io.ascii
Step4:  Using pandas
Step5:  Specialized text formats
Step6:  See http
Step7:  astropy.io.fits
Step8:  fitsio
(pip install --no-deps fitsio)
Faster (mainly for tables)
Does a better job with ASCII table extensions
Step9:  Salvaging data from IDL
scipy.io.readsav
Step10:  Clean up downloaded files
<ASSISTANT_TASK:>
Python Code:
import os
import numpy as np
import requests
# get some CSV data from the SDSS SQL server
URL = "http://skyserver.sdss.org/dr12/en/tools/search/x_sql.aspx"
cmd = 
SELECT TOP 1000
    p.u, p.g, p.r, p.i, p.z, s.class, s.z, s.zerr
FROM
    PhotoObj AS p
JOIN
    SpecObj AS s ON s.bestobjid = p.objid
WHERE
    p.u BETWEEN 0 AND 19.6 AND
    p.g BETWEEN 0 AND 20 AND
    s.class = 'GALAXY'
if not os.path.exists('galaxy_colors.csv'):
    cmd = ' '.join(map(lambda x: x.strip(), cmd.split('\n')))
    response = requests.get(URL, params={'cmd': cmd, 'format':'csv'})
    with open('galaxy_colors.csv', 'w') as f:
        f.write(response.text)
!ls -lh galaxy_colors.csv
!more galaxy_colors.csv
Explanation: Getting data into Python
Outline:
ASCII Files: numpy.loadtxt, astropy.io.ascii, read_csv (pandas)
Reading/Writing FITS files: astropy.io.fits, fitsio
IDL .sav files: scipy.readsav
Pandas
CSV data
End of explanation
dtype=[('u', 'f8'),
       ('g', 'f8'),
       ('r', 'f8'),
       ('i', 'f8'),
       ('z', 'f8'),
       ('class', 'S10'),
       ('redshift', 'f8'),
       ('redshift_err', 'f8')]
data = np.loadtxt('galaxy_colors.csv', skiprows=2, delimiter=',', dtype=dtype)
data[:10]
Explanation: Using numpy.loadtxt
End of explanation
from astropy.io import ascii
data = ascii.read('galaxy_colors.csv', format='csv', comment='#')
type(data)
data[:10]
Explanation: Using astropy.io.ascii
End of explanation
import pandas
data = pandas.read_csv('galaxy_colors.csv', comment='#')
type(data)
data.head()
data.describe()
# Pandas reads from *lots* of different data sources
pandas.read_
Explanation: Using pandas
End of explanation
# get some data from CDS
prefix = "http://cdsarc.u-strasbg.fr/vizier/ftp/cats/J/ApJ/686/749/"
for fname in ["ReadMe", "table10.dat"]:
    if not os.path.exists(fname):
        response = requests.get(prefix + fname)
        with open(fname, 'w') as f:
            f.write(response.text)
!cat table10.dat
!cat ReadMe
# must specify the "readme" here.
data = ascii.read("table10.dat", format='cds', readme="ReadMe")
data
Explanation: Specialized text formats
End of explanation
# get an SDSS image (can search for images from http://dr12.sdss3.org/fields/)
if not os.path.exists("frame-g-006728-4-0121.fits.bz2"):
    !wget http://dr12.sdss3.org/sas/dr12/boss/photoObj/frames/301/6728/4/frame-g-006728-4-0121.fits.bz2
if not os.path.exists("frame-g-006728-4-0121.fits"):
    !bunzip2 frame-g-006728-4-0121.fits.bz2
Explanation: See http://astropy.readthedocs.org/en/stable/io/ascii/index.html for all the supported formats.
Reading FITS files
Two options: astropy.io.fits (formerly pyfits) and fitsio.
End of explanation
from astropy.io import fits
hdulist = fits.open("frame-g-006728-4-0121.fits")
hdulist
hdulist.info()
hdulist[0].data
hdulist[0].header
Explanation: astropy.io.fits
End of explanation
import fitsio
f = fitsio.FITS("frame-g-006728-4-0121.fits")
# summary of file HDUs
f
# summary of first HDU
f[0]
# Summary of 3rd HDU
f[2]
# Actually read the data.
data = f[0].read()
data
Explanation: fitsio
(pip install --no-deps fitsio)
Faster (mainly for tables)
Does a better job with ASCII table extensions
End of explanation
from scipy.io import readsav
# Note: won't work unless you have this sav file!
data = readsav("150623434_det8_8100keV.sav")
data
len(data.events)
Explanation: Salvaging data from IDL
scipy.io.readsav: Formerly a separate idlsave module by Tom Robitaille.
End of explanation
!rm galaxy_colors.csv
!rm ReadMe
!rm table10.dat
!rm frame-g-006728-4-0121.fits.bz2
!rm frame-g-006728-4-0121.fits
Explanation: Clean up downloaded files
End of explanation
<END_TASK>