<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <h1> 2c. Refactoring to add batching and feature-creation </h1>
In this notebook, we continue reading the same small dataset, but refactor our ML pipeline in two small, but significant, ways
Step1:  <h2> 1. Refactor the input </h2>
Read data created in Lab1a, but this time make it more general and performant.  Instead of using Pandas, we will use TensorFlow's Dataset API.
Step2:  <h2> 2. Refactor the way features are created. </h2>
For now, pass these through (same as previous lab).  However, refactoring this way will enable us to break the one-to-one relationship between inputs and features.
Step3:  <h2> Create and train the model </h2>
Note that we train for num_steps * batch_size examples.
Step4:  <h3> Evaluate model </h3>
As before, evaluate on the validation data.  We'll do the third refactoring (to move the evaluation into the training loop) in the next lab.
<ASSISTANT_TASK:>
Python Code:
import tensorflow.compat.v1 as tf
import numpy as np
import shutil
print(tf.__version__)
Explanation: <h1> 2c. Refactoring to add batching and feature-creation </h1>
In this notebook, we continue reading the same small dataset, but refactor our ML pipeline in two small, but significant, ways:
<ol>
<li> Refactor the input to read data in batches.
<li> Refactor the feature creation so that it is not one-to-one with inputs.
</ol>
The Pandas function in the previous notebook also batched, only after it had read the whole data into memory -- on a large dataset, this won't be an option.
End of explanation
CSV_COLUMNS = ['fare_amount', 'pickuplon','pickuplat','dropofflon','dropofflat','passengers', 'key']
LABEL_COLUMN = 'fare_amount'
DEFAULTS = [[0.0], [-74.0], [40.0], [-74.0], [40.7], [1.0], ['nokey']]
def read_dataset(filename, mode, batch_size = 512):
  def _input_fn():
    def decode_csv(value_column):
      columns = tf.decode_csv(value_column, record_defaults = DEFAULTS)
      features = dict(zip(CSV_COLUMNS, columns))
      label = features.pop(LABEL_COLUMN)
      return features, label
    # Create list of files that match pattern
    file_list = tf.gfile.Glob(filename)
    # Create dataset from file list
    dataset = tf.data.TextLineDataset(file_list).map(decode_csv)
    if mode == tf.estimator.ModeKeys.TRAIN:
        num_epochs = None # indefinitely
        dataset = dataset.shuffle(buffer_size = 10 * batch_size)
    else:
        num_epochs = 1 # end-of-input after this
    dataset = dataset.repeat(num_epochs).batch(batch_size)
    return dataset.make_one_shot_iterator().get_next()
  return _input_fn
    
def get_train():
  return read_dataset('./taxi-train.csv', mode = tf.estimator.ModeKeys.TRAIN)
def get_valid():
  return read_dataset('./taxi-valid.csv', mode = tf.estimator.ModeKeys.EVAL)
def get_test():
  return read_dataset('./taxi-test.csv', mode = tf.estimator.ModeKeys.EVAL)
Explanation: <h2> 1. Refactor the input </h2>
Read data created in Lab1a, but this time make it more general and performant.  Instead of using Pandas, we will use TensorFlow's Dataset API.
End of explanation
INPUT_COLUMNS = [
    tf.feature_column.numeric_column('pickuplon'),
    tf.feature_column.numeric_column('pickuplat'),
    tf.feature_column.numeric_column('dropofflat'),
    tf.feature_column.numeric_column('dropofflon'),
    tf.feature_column.numeric_column('passengers'),
]
def add_more_features(feats):
  # Nothing to add (yet!)
  return feats
feature_cols = add_more_features(INPUT_COLUMNS)
Explanation: <h2> 2. Refactor the way features are created. </h2>
For now, pass these through (same as previous lab).  However, refactoring this way will enable us to break the one-to-one relationship between inputs and features.
End of explanation
tf.logging.set_verbosity(tf.logging.INFO)
OUTDIR = 'taxi_trained'
shutil.rmtree(OUTDIR, ignore_errors = True) # start fresh each time
model = tf.estimator.LinearRegressor(
      feature_columns = feature_cols, model_dir = OUTDIR)
model.train(input_fn = get_train(), steps = 100);
Explanation: <h2> Create and train the model </h2>
Note that we train for num_steps * batch_size examples.
End of explanation
def print_rmse(model, name, input_fn):
  metrics = model.evaluate(input_fn = input_fn, steps = 1)
  print('RMSE on {} dataset = {}'.format(name, np.sqrt(metrics['average_loss'])))
print_rmse(model, 'validation', get_valid())
Explanation: <h3> Evaluate model </h3>
As before, evaluate on the validation data.  We'll do the third refactoring (to move the evaluation into the training loop) in the next lab.
End of explanation
<END_TASK>