<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Vision ML on Audio, Video, Text, etc.
This notebook shows you how to use the spectrogram
of an audio file as a grayscale image input to an
ML model.
Step1:  Vision ML on video
Video consists of frames, each of which is an image.
Step3:  Text
We can break down a paragraph into sentences.
And we can do sentence-embedding to get a numeric representation of each sentence.
A paragraph now becomes an image!
<ASSISTANT_TASK:>
Python Code:
!gsutil cp gs://ml-design-patterns/audio_train/00353774.wav cello.wav
!gsutil cp gs://ml-design-patterns/audio_train/001ca53d.wav sax.wav
import matplotlib.pyplot as plt
from scipy import signal
from scipy.io import wavfile
import numpy as np
fig, ax = plt.subplots(2, 2, figsize=(15, 10))
for idx, instr in enumerate(['sax', 'cello']):
  sample_rate, samples = wavfile.read(instr + '.wav')
  ax[idx][0].plot(samples)
  _, _, spectro = signal.spectrogram(samples, sample_rate)
  img = np.log(spectro)
  ax[idx][1].imshow(img, cmap='gray', aspect='auto');
  ax[idx][1].set_title(instr)
  print(img.shape)
Explanation: Vision ML on Audio, Video, Text, etc.
This notebook shows you how to use the spectrogram
of an audio file as a grayscale image input to an
ML model.
End of explanation
!curl -O https://www.engr.colostate.edu/me/facil/dynamics/files/flame.avi
## Frame-by-frame
import cv2
import numpy as np
import matplotlib.pyplot as plt
cap = cv2.VideoCapture('flame.avi')
num_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
print(num_frames)
fig, ax = plt.subplots(1, 4, figsize=(20, 10))
for i in range(num_frames):
  ret, frame = cap.read()
  if ret:
    img = np.asarray(frame)
    if i%30 == 0:
      ax[i//30].imshow(img)
## Rolling average of 30 frames at a time
def rolling_average(cap, N):
  img = None;
  n = 0
  for i in range(N):
    ret, frame = cap.read()
    if ret:
      frame = np.asarray(frame)
      if n > 0:
        img = frame + img
      else:
        img = frame
      n += 1
  if n > 0:
    return img / n
  return img
cap = cv2.VideoCapture('flame.avi')
fig, ax = plt.subplots(1, 4, figsize=(20, 10))
for i in range(4):
  img = rolling_average(cap, 25)
  ax[i].imshow(img)
# read into a 4D shape
import tensorflow as tf
def read_video(filename):
  cap = cv2.VideoCapture(filename)
  num_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
  frames = []
  for i in range(num_frames):
    ret, frame = cap.read()
    if ret:
      frames.append(np.asarray(frame))
  return tf.convert_to_tensor(frames)
img4d = read_video('flame.avi')
print(img4d.shape)
Explanation: Vision ML on video
Video consists of frames, each of which is an image.
End of explanation
import tensorflow_hub as hub
paragraph = 
Siddhartha gave his clothes to a poor Brahmin on the road and
only retained his loincloth andearth-colored unstitched cloak.
He only ate once a day and never cooked food. He fasted fourteen
days. He fasted twenty-eight days. The flesh disappeared from
his legs and cheeks. Strange dreams were reflected in his enlarged
eyes. The nails grew long on his thin fingers and a dry, bristly
beard appeared on his chin. His glance became icy when he
encountered women; his lips curled with contempt when he passed
through a town of well-dressed people. He saw businessmen trading,
princes going to the hunt, mourners weeping over their dead,
prostitutes offering themselves, doctors attending the sick,
priests deciding the day for sowing, lovers making love, mothers
soothing their children -and all were not worth a passing glance,
everything lied, stank of lies; they were all illusions of sense,
happiness and beauty. All were doomed to decay. The world tasted
bitter. Life was pain.
print(paragraph.split('.'))
embed = hub.load("https://tfhub.dev/google/universal-sentence-encoder/4")
embeddings = embed(paragraph.split('.'))
import matplotlib.pyplot as plt
plt.figure(figsize=(5,10))
plt.imshow(embeddings.numpy(), aspect=25.0, cmap='gray');
Explanation: Text
We can break down a paragraph into sentences.
And we can do sentence-embedding to get a numeric representation of each sentence.
A paragraph now becomes an image!
End of explanation
<END_TASK>