<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Copyright 2021 The TensorFlow Authors.
Step2:  Composing Decision Forest and Neural Network models
<table class="tfo-notebook-buttons" align="left">
  <td>
    <a target="_blank" href="https
Step3:  Your composed model has three stages
Step4:  Wurlitzer is needed to display the detailed training logs in Colabs (when using verbose=2 in the model constructor).
Step5:  Import libraries
Step6:  Dataset
You will use a simple synthetic dataset in this tutorial to make it easier to
interpret the final model.
Step7:  Generate some examples
Step8:  You can also plot them to get an idea of the synthetic pattern
Step9:  Note that this pattern is smooth and not axis aligned. This will advantage the neural network models. This is because it is easier for a neural network than for a decision tree to have round and non aligned decision boundaries.
On the other hand, we will train the model on a small datasets with 2500 examples. This will advantage the decision forest models. This is because decision forests are much more efficient, using all the available information from the examples (decision forests are "sample efficient").
Our ensemble of neural networks and decision forests will use the best of both worlds.
Let's create a train and test tf.data.Dataset
Step10:  Model structure
Define the model structure as follows
Step11:  Before you train the model, you can plot it to check if it is similar to the
initial diagram.
Step12:  Model training
First train the preprocessing and two neural network layers using the
backpropagation algorithm.
Step13:  Let's evaluate the preprocessing and the part with the two neural networks only
Step14:  Let's train the two Decision Forest components (one after another).
Step15:  And let's evaluate the Decision Forests individually.
Step16:  Let's evaluate the entire model composition
Step17:  To finish, let's finetune the neural network layer a bit more. Note that we do
not finetune the pre-trained embedding as the DF models depends on it (unless we
would also retrain them after).
In summary, you have
<ASSISTANT_TASK:>
Python Code:
#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
Explanation: Copyright 2021 The TensorFlow Authors.
End of explanation
#@title
!pip install graphviz -U --quiet
from graphviz import Source
Source(
digraph G {
  raw_data [label="Input features"];
  preprocess_data [label="Learnable NN pre-processing", shape=rect];
  raw_data -> preprocess_data
  subgraph cluster_0 {
    color=grey;
    a1[label="NN layer", shape=rect];
    b1[label="NN layer", shape=rect];
    a1 -> b1;
	label = "Model #1";
  }
   subgraph cluster_1 {
    color=grey;
    a2[label="NN layer", shape=rect];
    b2[label="NN layer", shape=rect];
    a2 -> b2;
	label = "Model #2";
  }
  subgraph cluster_2 {
    color=grey;
    a3[label="Decision Forest", shape=rect];
	label = "Model #3";
  }
  subgraph cluster_3 {
    color=grey;
    a4[label="Decision Forest", shape=rect];
	label = "Model #4";
  }
  preprocess_data -> a1;
  preprocess_data -> a2;
  preprocess_data -> a3;
  preprocess_data -> a4;
  b1  -> aggr;
  b2  -> aggr;
  a3 -> aggr;
  a4 -> aggr;
  aggr [label="Aggregation (mean)", shape=rect]
  aggr -> predictions
}
)
Explanation: Composing Decision Forest and Neural Network models
<table class="tfo-notebook-buttons" align="left">
  <td>
    <a target="_blank" href="https://www.tensorflow.org/decision_forests/tutorials/model_composition_colab"><img src="https://www.tensorflow.org/images/tf_logo_32px.png" />View on TensorFlow.org</a>
  </td>
  <td>
    <a target="_blank" href="https://colab.research.google.com/github/tensorflow/decision-forests/blob/main/documentation/tutorials/model_composition_colab.ipynb"><img src="https://www.tensorflow.org/images/colab_logo_32px.png" />Run in Google Colab</a>
  </td>
  <td>
    <a target="_blank" href="https://github.com/tensorflow/decision-forests/blob/main/documentation/tutorials/model_composition_colab.ipynb"><img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" />View on GitHub</a>
  </td>
  <td>
    <a href="https://storage.googleapis.com/tensorflow_docs/decision-forests/documentation/tutorials/model_composition_colab.ipynb"><img src="https://www.tensorflow.org/images/download_logo_32px.png" />Download notebook</a>
  </td>
  <td>
    <a href="https://keras.io/guides/functional_api"><img src="https://www.tensorflow.org/images/tf_logo_32px.png" />Keras Functional API</a>
  </td>
</table>
Introduction
Welcome to the model composition tutorial for TensorFlow Decision Forests
(TF-DF). This notebook shows you how to compose multiple decision forest and
neural network models together using a common preprocessing layer and the
Keras functional API.
You might want to compose models together to improve predictive performance
(ensembling), to get the best of different modeling technologies (heterogeneous
model ensembling), to train different part of the model on different datasets
(e.g. pre-training), or to create a stacked model (e.g. a model
operates on the predictions of another model).
This tutorial covers an advanced use case of model composition using the
Functional API. You can find examples for simpler scenarios of model composition
in the "feature preprocessing" section of this tutorial
and in the "using a pretrained text embedding" section of this
tutorial.
Here is the structure of the model you'll build:
End of explanation
!pip install tensorflow_decision_forests -U --quiet
Explanation: Your composed model has three stages:
The first stage is a preprocessing layer composed of a neural network and
    common to all the models in the next stage. In practice, such a
    preprocessing layer could either be a pre-trained embedding to fine-tune, or
    a randomly initialized neural network.
The second stage is an ensemble of two decision forest and two neural
    network models.
The last stage averages the predictions of the models in the second stage.
    It does not contain any learnable weights.
The neural networks are trained using the
backpropagation algorithm
and gradient descent. This algorithm has two important properties: (1) The layer
of neural network can be trained if its receives a loss gradient (more
precicely, the gradient of the loss according to the layer's output), and (2)
the algorithm "transmits" the loss gradient from the layer's output to the
layer's input (this is the "chain rule"). For these two reasons, Backpropagation
can train together multiple layers of neural networks stacked on top of each
other.
In this example, the decision forests are trained with the
Random Forest
(RF) algorithm. Unlike Backpropagation, the training of RF does not "transmit"
the loss gradient to from its output to its input. For this reasons, the
classical RF algorithm cannot be used to train or fine-tune a neural network
underneath. In other words, the "decision forest" stages cannot be used to train
the "Learnable NN pre-processing block".
Train the preprocessing and neural networks stage.
Train the decision forest stages.
Install TensorFlow Decision Forests
Install TF-DF by running the following cell.
End of explanation
!pip install wurlitzer -U --quiet
Explanation: Wurlitzer is needed to display the detailed training logs in Colabs (when using verbose=2 in the model constructor).
End of explanation
import tensorflow_decision_forests as tfdf
import os
import numpy as np
import pandas as pd
import tensorflow as tf
import math
import matplotlib.pyplot as plt
Explanation: Import libraries
End of explanation
def make_dataset(num_examples, num_features, seed=1234):
  np.random.seed(seed)
  features = np.random.uniform(-1, 1, size=(num_examples, num_features))
  noise = np.random.uniform(size=(num_examples))
  left_side = np.sqrt(
      np.sum(np.multiply(np.square(features[:, 0:2]), [1, 2]), axis=1))
  right_side = features[:, 2] * 0.7 + np.sin(
      features[:, 3] * 10) * 0.5 + noise * 0.0 + 0.5
  labels = left_side <= right_side
  return features, labels.astype(int)
Explanation: Dataset
You will use a simple synthetic dataset in this tutorial to make it easier to
interpret the final model.
End of explanation
make_dataset(num_examples=5, num_features=4)
Explanation: Generate some examples:
End of explanation
plot_features, plot_label = make_dataset(num_examples=50000, num_features=4)
plt.rcParams["figure.figsize"] = [8, 8]
common_args = dict(c=plot_label, s=1.0, alpha=0.5)
plt.subplot(2, 2, 1)
plt.scatter(plot_features[:, 0], plot_features[:, 1], **common_args)
plt.subplot(2, 2, 2)
plt.scatter(plot_features[:, 1], plot_features[:, 2], **common_args)
plt.subplot(2, 2, 3)
plt.scatter(plot_features[:, 0], plot_features[:, 2], **common_args)
plt.subplot(2, 2, 4)
plt.scatter(plot_features[:, 0], plot_features[:, 3], **common_args)
Explanation: You can also plot them to get an idea of the synthetic pattern:
End of explanation
def make_tf_dataset(batch_size=64, **args):
  features, labels = make_dataset(**args)
  return tf.data.Dataset.from_tensor_slices(
      (features, labels)).batch(batch_size)
num_features = 10
train_dataset = make_tf_dataset(
    num_examples=2500, num_features=num_features, batch_size=100, seed=1234)
test_dataset = make_tf_dataset(
    num_examples=10000, num_features=num_features, batch_size=100, seed=5678)
Explanation: Note that this pattern is smooth and not axis aligned. This will advantage the neural network models. This is because it is easier for a neural network than for a decision tree to have round and non aligned decision boundaries.
On the other hand, we will train the model on a small datasets with 2500 examples. This will advantage the decision forest models. This is because decision forests are much more efficient, using all the available information from the examples (decision forests are "sample efficient").
Our ensemble of neural networks and decision forests will use the best of both worlds.
Let's create a train and test tf.data.Dataset:
End of explanation
# Input features.
raw_features = tf.keras.layers.Input(shape=(num_features,))
# Stage 1
# =======
# Common learnable pre-processing
preprocessor = tf.keras.layers.Dense(10, activation=tf.nn.relu6)
preprocess_features = preprocessor(raw_features)
# Stage 2
# =======
# Model #1: NN
m1_z1 = tf.keras.layers.Dense(5, activation=tf.nn.relu6)(preprocess_features)
m1_pred = tf.keras.layers.Dense(1, activation=tf.nn.sigmoid)(m1_z1)
# Model #2: NN
m2_z1 = tf.keras.layers.Dense(5, activation=tf.nn.relu6)(preprocess_features)
m2_pred = tf.keras.layers.Dense(1, activation=tf.nn.sigmoid)(m2_z1)
# Model #3: DF
model_3 = tfdf.keras.RandomForestModel(num_trees=1000, random_seed=1234)
m3_pred = model_3(preprocess_features)
# Model #4: DF
model_4 = tfdf.keras.RandomForestModel(
    num_trees=1000,
    #split_axis="SPARSE_OBLIQUE", # Uncomment this line to increase the quality of this model
    random_seed=4567)
m4_pred = model_4(preprocess_features)
# Since TF-DF uses deterministic learning algorithms, you should set the model's
# training seed to different values otherwise both
# `tfdf.keras.RandomForestModel` will be exactly the same.
# Stage 3
# =======
mean_nn_only = tf.reduce_mean(tf.stack([m1_pred, m2_pred], axis=0), axis=0)
mean_nn_and_df = tf.reduce_mean(
    tf.stack([m1_pred, m2_pred, m3_pred, m4_pred], axis=0), axis=0)
# Keras Models
# ============
ensemble_nn_only = tf.keras.models.Model(raw_features, mean_nn_only)
ensemble_nn_and_df = tf.keras.models.Model(raw_features, mean_nn_and_df)
Explanation: Model structure
Define the model structure as follows:
End of explanation
from keras.utils.vis_utils import plot_model
plot_model(ensemble_nn_and_df, to_file="/tmp/model.png", show_shapes=True)
Explanation: Before you train the model, you can plot it to check if it is similar to the
initial diagram.
End of explanation
%%time
ensemble_nn_only.compile(
        optimizer=tf.keras.optimizers.Adam(),
        loss=tf.keras.losses.BinaryCrossentropy(),
        metrics=["accuracy"])
ensemble_nn_only.fit(train_dataset, epochs=20, validation_data=test_dataset)
Explanation: Model training
First train the preprocessing and two neural network layers using the
backpropagation algorithm.
End of explanation
evaluation_nn_only = ensemble_nn_only.evaluate(test_dataset, return_dict=True)
print("Accuracy (NN #1 and #2 only): ", evaluation_nn_only["accuracy"])
print("Loss (NN #1 and #2 only): ", evaluation_nn_only["loss"])
Explanation: Let's evaluate the preprocessing and the part with the two neural networks only:
End of explanation
%%time
train_dataset_with_preprocessing = train_dataset.map(lambda x,y: (preprocessor(x), y))
test_dataset_with_preprocessing = test_dataset.map(lambda x,y: (preprocessor(x), y))
model_3.fit(train_dataset_with_preprocessing)
model_4.fit(train_dataset_with_preprocessing)
Explanation: Let's train the two Decision Forest components (one after another).
End of explanation
model_3.compile(["accuracy"])
model_4.compile(["accuracy"])
evaluation_df3_only = model_3.evaluate(
    test_dataset_with_preprocessing, return_dict=True)
evaluation_df4_only = model_4.evaluate(
    test_dataset_with_preprocessing, return_dict=True)
print("Accuracy (DF #3 only): ", evaluation_df3_only["accuracy"])
print("Accuracy (DF #4 only): ", evaluation_df4_only["accuracy"])
Explanation: And let's evaluate the Decision Forests individually.
End of explanation
ensemble_nn_and_df.compile(
    loss=tf.keras.losses.BinaryCrossentropy(), metrics=["accuracy"])
evaluation_nn_and_df = ensemble_nn_and_df.evaluate(
    test_dataset, return_dict=True)
print("Accuracy (2xNN and 2xDF): ", evaluation_nn_and_df["accuracy"])
print("Loss (2xNN and 2xDF): ", evaluation_nn_and_df["loss"])
Explanation: Let's evaluate the entire model composition:
End of explanation
#@title
print(f"Accuracy (NN #1 and #2 only):\t{evaluation_nn_only['accuracy']:.6f}")
print(f"Accuracy (DF #3 only):\t\t{evaluation_df3_only['accuracy']:.6f}")
print(f"Accuracy (DF #4 only):\t\t{evaluation_df4_only['accuracy']:.6f}")
print("----------------------------------------")
print(f"Accuracy (2xNN and 2xDF):\t{evaluation_nn_and_df['accuracy']:.6f}")
def delta_percent(src_eval, key):
  src_acc = src_eval["accuracy"]
  final_acc = evaluation_nn_and_df["accuracy"]
  increase = final_acc - src_acc
  print(f"\t\t\t\t  {increase:+.6f} over {key}")
delta_percent(evaluation_nn_only, "NN #1 and #2 only")
delta_percent(evaluation_df3_only, "DF #3 only")
delta_percent(evaluation_df4_only, "DF #4 only")
Explanation: To finish, let's finetune the neural network layer a bit more. Note that we do
not finetune the pre-trained embedding as the DF models depends on it (unless we
would also retrain them after).
In summary, you have:
End of explanation
<END_TASK>