<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Loading large datasets
Learning Objectives
  - Understand difference between loading data entirely in-memory and loading in batches from disk
  - Practice loading a .csv file from disk in batches using the tf.data module
Introduction
In the previous notebook, we read the the whole taxifare .csv files into memory, specifically a Pandas dataframe, before invoking tf.data.from_tensor_slices from the tf.data API. We could get away with this because it was a small sample of the dataset, but on the full taxifare dataset this wouldn't be feasible.
In this notebook we demonstrate how to read .csv files directly from disk, one batch at a time, using tf.data.TextLineDataset
Run the following cell and restart the kernel if needed
Step1:  Input function reading from CSV
We define read_dataset() which given a csv file path returns a tf.data.Dataset in which each row represents a (features,label) in the Estimator API required format 
- features
Step2:  Run the following test to make sure your implementation is correct
Step3:  We'll use the function parse_row we implemented above to 
implement a read_dataset function that
- takes as input the path to a csv file
- returns a tf.data.Dataset object containing the features, labels
We can assume that the .csv file has a header, and that your read_dataset will skip it.
Step4:  Tests
Let's create a test dataset to test our function.
Step5:  You should be able to iterate over what's returned by read_dataset. We'll print the dropofflat and fare_amount for each entry in ./test.csv
Step6:  Run the following test cell to make sure your function works properly
Step7:  Next we can implement a train_input_fn function that
- takes a input a path to a csv file along with a batch_size
- returns a dataset object that shuffle the rows and returns them in batches of batch_size
We'll reuse the read_dataset function you implemented above.
Step8:  Next, we implement a eval_input_fn simlar to train_input_fn you implemented above.
The only difference is that this function does not need to shuffle the rows.
Step9:  Create feature columns
The features of our models are the following
Step10:  In the cell below, create a variable feature_cols containing a
list of the appropriate tf.feature_column to be passed to a tf.estimator
Step11:  Choose Estimator
Next, we create an instance of a tf.estimator.DNNRegressor such that
- it has two layers of 10 units each
- it uses the features defined in the previous exercise
- it saves the trained model into the directory ./taxi_trained
- it has a random seed set to 1 for replicability and debugging
Note that we can set the random seed by passing a tf.estimator.RunConfig object to the config parameter of the tf.estimator.
Step12:  Train
With the model defined, we can now train the model on our data. In the cell below, we train the model you defined above using the train_input_function on ./tazi-train.csv for 500 steps. How many epochs of our data does this represent?
Step13:  Evaluate
Finally, we'll evaluate the performance of our model on the validation set. We evaluate the model using its .evaluate method and
the eval_input_fn function you implemented above on the ./taxi-valid.csv dataset. Note, we make sure to extract the average_loss for the dictionary returned by model.evaluate. It is the RMSE.
<ASSISTANT_TASK:>
Python Code:
import tensorflow as tf
import shutil
print(tf.__version__)
tf.enable_eager_execution()
Explanation: Loading large datasets
Learning Objectives
  - Understand difference between loading data entirely in-memory and loading in batches from disk
  - Practice loading a .csv file from disk in batches using the tf.data module
Introduction
In the previous notebook, we read the the whole taxifare .csv files into memory, specifically a Pandas dataframe, before invoking tf.data.from_tensor_slices from the tf.data API. We could get away with this because it was a small sample of the dataset, but on the full taxifare dataset this wouldn't be feasible.
In this notebook we demonstrate how to read .csv files directly from disk, one batch at a time, using tf.data.TextLineDataset
Run the following cell and restart the kernel if needed:
End of explanation
CSV_COLUMN_NAMES = ["fare_amount","dayofweek","hourofday","pickuplon","pickuplat","dropofflon","dropofflat"]
CSV_DEFAULTS = [[0.0],[1],[0],[-74.0], [40.0], [-74.0], [40.7]]
def parse_row(row):
    fields = tf.decode_csv(records = row, record_defaults = CSV_DEFAULTS)
    features = dict(zip(CSV_COLUMN_NAMES, fields))
    label = features.pop("fare_amount") # remove label from features and store
    return features, label
Explanation: Input function reading from CSV
We define read_dataset() which given a csv file path returns a tf.data.Dataset in which each row represents a (features,label) in the Estimator API required format 
- features: A python dictionary. Each key is a feature column name and its value is the tensor containing the data for that feature
- label: A Tensor containing the labels
We then invoke read_dataset() function from within the train_input_fn() and eval_input_fn(). The remaining code is as before.
End of explanation
a_row = "0.0,1,0,-74.0,40.0,-74.0,40.7"
features, labels = parse_row(a_row)
assert labels.numpy() == 0.0
assert features["pickuplon"].numpy() == -74.0
print("You rock!")
Explanation: Run the following test to make sure your implementation is correct
End of explanation
def read_dataset(csv_path):  
    dataset = tf.data.TextLineDataset(filenames = csv_path).skip(count = 1) # skip header
    dataset = dataset.map(map_func = parse_row) 
    return dataset
Explanation: We'll use the function parse_row we implemented above to 
implement a read_dataset function that
- takes as input the path to a csv file
- returns a tf.data.Dataset object containing the features, labels
We can assume that the .csv file has a header, and that your read_dataset will skip it.
End of explanation
%%writefile test.csv
fare_amount,dayofweek,hourofday,pickuplon,pickuplat,dropofflon,dropofflat
28,1,0,-73.0,41.0,-74.0,20.7
12.3,1,0,-72.0,44.0,-75.0,40.6
10,1,0,-71.0,41.0,-71.0,42.9
Explanation: Tests
Let's create a test dataset to test our function.
End of explanation
for feature, label in read_dataset("./test.csv"):
    print("dropofflat:", feature["dropofflat"].numpy())
    print("fare_amount:", label.numpy())
Explanation: You should be able to iterate over what's returned by read_dataset. We'll print the dropofflat and fare_amount for each entry in ./test.csv
End of explanation
dataset= read_dataset("./test.csv")
dataset_iterator = dataset.make_one_shot_iterator()
features, labels = dataset_iterator.get_next()
assert features["dayofweek"].numpy() == 1
assert labels.numpy() == 28
print("You rock!")
Explanation: Run the following test cell to make sure your function works properly:
End of explanation
def train_input_fn(csv_path, batch_size = 128):
    dataset = read_dataset(csv_path)
    dataset = dataset.shuffle(buffer_size = 1000).repeat(count = None).batch(batch_size = batch_size)
    return dataset
Explanation: Next we can implement a train_input_fn function that
- takes a input a path to a csv file along with a batch_size
- returns a dataset object that shuffle the rows and returns them in batches of batch_size
We'll reuse the read_dataset function you implemented above.
End of explanation
def eval_input_fn(csv_path, batch_size = 128):
    dataset = read_dataset(csv_path)
    dataset = dataset.batch(batch_size = batch_size)
    return dataset
Explanation: Next, we implement a eval_input_fn simlar to train_input_fn you implemented above.
The only difference is that this function does not need to shuffle the rows.
End of explanation
FEATURE_NAMES = CSV_COLUMN_NAMES[1:] # all but first column
print(FEATURE_NAMES)
Explanation: Create feature columns
The features of our models are the following:
End of explanation
feature_cols = [tf.feature_column.numeric_column(key = k) for k in FEATURE_NAMES]
print(feature_cols)
Explanation: In the cell below, create a variable feature_cols containing a
list of the appropriate tf.feature_column to be passed to a tf.estimator:
End of explanation
OUTDIR = "taxi_trained"
model = tf.estimator.DNNRegressor(
    hidden_units = [10,10], # specify neural architecture
    feature_columns = feature_cols, 
    model_dir = OUTDIR,
    config = tf.estimator.RunConfig(tf_random_seed = 1)
)
Explanation: Choose Estimator
Next, we create an instance of a tf.estimator.DNNRegressor such that
- it has two layers of 10 units each
- it uses the features defined in the previous exercise
- it saves the trained model into the directory ./taxi_trained
- it has a random seed set to 1 for replicability and debugging
Note that we can set the random seed by passing a tf.estimator.RunConfig object to the config parameter of the tf.estimator.
End of explanation
%%time
tf.logging.set_verbosity(tf.logging.INFO) # so loss is printed during training
shutil.rmtree(path = OUTDIR, ignore_errors = True) # start fresh each time
model.train(
    input_fn = lambda: train_input_fn(csv_path = "./taxi-train.csv"),
    steps = 500
)
Explanation: Train
With the model defined, we can now train the model on our data. In the cell below, we train the model you defined above using the train_input_function on ./tazi-train.csv for 500 steps. How many epochs of our data does this represent?
End of explanation
metrics = model.evaluate(input_fn = lambda: eval_input_fn(csv_path = "./taxi-valid.csv"))
print("RMSE on dataset = {}".format(metrics["average_loss"]**.5))
Explanation: Evaluate
Finally, we'll evaluate the performance of our model on the validation set. We evaluate the model using its .evaluate method and
the eval_input_fn function you implemented above on the ./taxi-valid.csv dataset. Note, we make sure to extract the average_loss for the dictionary returned by model.evaluate. It is the RMSE.
End of explanation
<END_TASK>