<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Prepare data for Google Cloud AutoML Natural Language from scikit-learn
Overview
This notebook demonstrates how to prepare text data available in scikit-learn (or other libraries), so that it can be used in Google Cloud AutoML Natural Language.
The script reads the data into a pandas dataframe, and then makes some minor transformations to ensure that it is compatible with the AutoML Natural Language input specification. Finally, the CSV is saved into a CSV file, which can be downloaded from the notebook server.
Dataset
This notebook downloads the 20 newsgroups dataset using scikit-learn. This dataset contains about 18000 posts from 20 newsgroups, and is useful for text classification. More details on the dataset can be found here.
Objectives
There are 3 goals for this notebook
Step1:  Fetch data
Step2:  Clean data
Step3:  Export to CSV
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import pandas as pd
import csv
from sklearn.datasets import fetch_20newsgroups
Explanation: Prepare data for Google Cloud AutoML Natural Language from scikit-learn
Overview
This notebook demonstrates how to prepare text data available in scikit-learn (or other libraries), so that it can be used in Google Cloud AutoML Natural Language.
The script reads the data into a pandas dataframe, and then makes some minor transformations to ensure that it is compatible with the AutoML Natural Language input specification. Finally, the CSV is saved into a CSV file, which can be downloaded from the notebook server.
Dataset
This notebook downloads the 20 newsgroups dataset using scikit-learn. This dataset contains about 18000 posts from 20 newsgroups, and is useful for text classification. More details on the dataset can be found here.
Objectives
There are 3 goals for this notebook:
1. Introduce scikit-learn datasets
2. Explore pandas dataframe text manipulation
3. Import data into AutoML Natural language for text classification
What's next?
After downloading the CSV at the end of this notebook, import the data into Google Cloud AutoML Natural Language to explore classifying text.
Imports
End of explanation
newsgroups = fetch_20newsgroups(subset='all')
df = pd.DataFrame(newsgroups.data, columns=['text'])
df['categories'] = [newsgroups.target_names[index] for index in newsgroups.target]
df.head()
Explanation: Fetch data
End of explanation
# Convert multiple whitespace characters into a space
df['text'] = df['text'].str.replace('\s+',' ')
# Change newsgroup titles to use underscores rather than periods
df['categories'] = df['categories'].str.replace('.','_')
# Trim leading and tailing whitespace
df['text'] = df['text'].str.strip()
# Truncate all fields to the maximum field length of 128kB
df['text'] = df['text'].str.slice(0,131072)
# Remove any rows with empty fields
df = df.replace('', np.NaN).dropna()
# Drop duplicates
df = df.drop_duplicates(subset='text')
# Limit rows to maximum of 100,000
df = df.sample(min(100000, len(df)))
df.head()
Explanation: Clean data
End of explanation
csv_str = df.to_csv(index=False, header=False)
with open("20-newsgroups-dataset.csv", "w") as text_file:
    print(csv_str, file=text_file)
Explanation: Export to CSV
End of explanation
<END_TASK>