<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Word embeddings Tutorial in TensorFlow 2.0
Step1:  Embedding Layer
Create and Embedding layer to map integer encoded tokens to dense vectors
Step2:  Let's test the embedding layer by passing a vector with random integer values
Step3:  What are the limits in terms of allowed integer values? Let's try with a negative value (-1) and then a positive value above the maximum number of input dimensions (1001)
Step4:  Ok, we expected that! Let's move forward...
We can pass 2D inputs, as we would if working in a sequence or text problem. That is, (batch_size, sequence_length). The Embedding layer returns then a 3D floating point tensor of shape (batch_size, sequence_length, embedding_dimensionality).
In this case, if we pass a set of 3 sequences of length 4, we will get a tensor with dimensions (3, 4, 5)
Step5:  Learning embeddings from scratch
We will train a sentiment classifier on IMDB movie reviews
Step6:  Let's inspect the words encoder, that can be found in info.features["text"].encoder
Step7:  Let's inspect the training dataset, train_data. But first, apply a random shuffling.
Step8:  Let's try to batch the elements in the dataset with padding. Consult the tf.data.Dataset.padded_batch docstring here.
Step9:  Create a simple model
Model description
Next the Embedding layer takes the integer-encoded vocabulary and looks up the embedding vector for each word-index. These vectors are learned as the model trains. The vectors add a dimension to the output array. The resulting dimensions are
Step10:  NOTE
Step11:  Let's plot the training and validation curves
Step12:  Inspect the learnt embeddings
Step13:  Let's write the embeddings to disk in TSV (tab-separated values) so that we can visualize them with the Embedding Projector.
<ASSISTANT_TASK:>
Python Code:
import tensorflow as tf
import tensorflow_datasets as tfds
tfds.disable_progress_bar()
import matplotlib.pyplot as plt
Explanation: Word embeddings Tutorial in TensorFlow 2.0
End of explanation
embedding_layer = tf.keras.layers.Embedding(input_dim=1000, output_dim=5)
Explanation: Embedding Layer
Create and Embedding layer to map integer encoded tokens to dense vectors
End of explanation
embedding_layer(tf.constant([1, 2, 3, 999]))
Explanation: Let's test the embedding layer by passing a vector with random integer values
End of explanation
try:
  embedding_layer(tf.constant(-1))
except Exception:
  print("We expected an error to happen")
try:
  embedding_layer(tf.constant(1001))
except Exception:
  print("We expected an error to happen")
Explanation: What are the limits in terms of allowed integer values? Let's try with a negative value (-1) and then a positive value above the maximum number of input dimensions (1001)
End of explanation
real_sequence = tf.constant(
    [
     [4, 10, 3, 2],
     [2, 4, 1, 5],
     [102, 40, 81, 22],
     ]
)
real_embeddings = embedding_layer(real_sequence)
real_embeddings_np = real_embeddings.numpy()
print(real_embeddings_np.shape)
print(real_embeddings_np)
Explanation: Ok, we expected that! Let's move forward...
We can pass 2D inputs, as we would if working in a sequence or text problem. That is, (batch_size, sequence_length). The Embedding layer returns then a 3D floating point tensor of shape (batch_size, sequence_length, embedding_dimensionality).
In this case, if we pass a set of 3 sequences of length 4, we will get a tensor with dimensions (3, 4, 5)
End of explanation
(train_data, test_data), info = tfds.load(
    "imdb_reviews/subwords8k", 
    split=(tfds.Split.TRAIN, tfds.Split.TEST), 
    with_info=True, 
    as_supervised=True
)
Explanation: Learning embeddings from scratch
We will train a sentiment classifier on IMDB movie reviews
End of explanation
info
encoder = info.features["text"].encoder
encoder.subwords[:30]
encoder.vocab_size
Explanation: Let's inspect the words encoder, that can be found in info.features["text"].encoder
End of explanation
train_data = train_data.shuffle(1000)
test_data = test_data.shuffle(1000)
train_data
for encoded_text, label in train_data.take(5):
  print(f"Encoded text tensor shape: {encoded_text.shape}")
  print(f"Label: {label.numpy()}")
Explanation: Let's inspect the training dataset, train_data. But first, apply a random shuffling.
End of explanation
train_batches = train_data.padded_batch(16)
test_batches = test_data.padded_batch(16)
for train_batch, train_label in train_batches.take(4):
  print(train_batch.shape)
  print(train_batch)
  print("-"*80)
Explanation: Let's try to batch the elements in the dataset with padding. Consult the tf.data.Dataset.padded_batch docstring here.
End of explanation
embedding_layer_dim = 16
embedding_dense_dim = 16
model = tf.keras.Sequential([
  tf.keras.layers.Embedding(input_dim=encoder.vocab_size, output_dim=embedding_layer_dim, mask_zero=True),
  tf.keras.layers.GlobalAveragePooling1D(),
  tf.keras.layers.Dense(embedding_dense_dim, activation='relu'),
  tf.keras.layers.Dense(1),
])
model.summary()
simplified_model = tf.keras.Sequential([
  tf.keras.layers.Embedding(input_dim=encoder.vocab_size, output_dim=embedding_layer_dim, mask_zero=True),
  tf.keras.layers.GlobalAveragePooling1D(),
  tf.keras.layers.Dense(1),
])
simplified_model.summary()
Explanation: Create a simple model
Model description
Next the Embedding layer takes the integer-encoded vocabulary and looks up the embedding vector for each word-index. These vectors are learned as the model trains. The vectors add a dimension to the output array. The resulting dimensions are: (batch, sequence, embedding).
Next, a GlobalAveragePooling1D layer returns a fixed-length output vector for each example by averaging over the sequence dimension. This allows the model to handle input of variable length, in the simplest way possible.
This fixed-length output vector is piped through a fully-connected (Dense) layer with 16 hidden units.
The last layer is densely connected with a single output node. Using the sigmoid activation function, this value is a float between 0 and 1, representing a probability (or confidence level) that the review is positive.
End of explanation
model.compile(
    optimizer="adam",
    loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
    metrics=["accuracy"],
)
simplified_model.compile(
    optimizer="adam",
    loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
    metrics=["accuracy"],
)
# history = model.fit(train_batches, epochs=10, validation_data=test_batches, validation_steps=20)
history = simplified_model.fit(train_batches, epochs=10, validation_data=test_batches, validation_steps=20)
Explanation: NOTE: This model uses masking thanks to the mask_zero=True argument passed to the Embedding layer. If it was not set, the zero-padding would used as part of the input, so the padding length would affect the output. See the masking and padding guide for more detailed information.
End of explanation
history_dict = history.history
acc = history_dict['accuracy']
val_acc = history_dict['val_accuracy']
loss = history_dict['loss']
val_loss = history_dict['val_loss']
epochs = range(1, len(acc) + 1)
plt.figure(figsize=(12,9))
plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
plt.show()
plt.figure(figsize=(12,9))
plt.plot(epochs, acc, 'bo', label='Training acc')
plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.legend(loc='lower right')
plt.ylim((0.5,1))
plt.show()
Explanation: Let's plot the training and validation curves
End of explanation
print(model.layers)
embedding_layer = model.layers[0]
weights = embedding_layer.get_weights()[0]
print(weights.shape)
Explanation: Inspect the learnt embeddings
End of explanation
import io
encoder = info.features["text"].encoder
out_v = io.open("vecs.tsv", "w", encoding="utf-8")
out_m = io.open("meta.tsv", "w", encoding="utf-8")
# Iterate over all words in the vocabulary and fetch their embedding vector
# Skip 0 as it represents padding
for idx in range(1, len(encoder.subwords)):
  word = encoder.subwords[idx]
  vec = weights[idx]
  out_m.write(word + "\n")
  out_v.write("\t".join([str(x) for x in vec]) + "\n")
out_v.close()
out_m.close()
Explanation: Let's write the embeddings to disk in TSV (tab-separated values) so that we can visualize them with the Embedding Projector.
End of explanation
<END_TASK>