<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 About
Analysis of recipes with word2vec.
Partially based on https
Step1:  Prepare text processing
Step4:  Define functions for cleaning the text data.
Step5:  Initialize and train the model
this will take some time...
Step6:  Save the model for later use or continued training
Step7:  Some model evaluation examples
<ASSISTANT_TASK:>
Python Code:
import re                          # Regular Expressions
import pandas as pd                # DataFrames & Manipulation
import nltk.data                   # Sentence tokenizer
from nltk.corpus import stopwords  # Import the stop word list
from bs4 import BeautifulSoup      # HTML processing
from gensim.models.word2vec import Word2Vec
train_input = "../data/recipes.tsv.bz2"
# keep empty strings (http://pandas-docs.github.io/pandas-docs-travis/io.html#na-values)
train = pd.read_csv(train_input, delimiter="\t", quoting=3, encoding="utf-8", keep_default_na=False)
Explanation: About
Analysis of recipes with word2vec.
Partially based on https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-2-word-vectors
Prerequesites
Python libraries:
nltk - natural language processing toolkit which includes functions for cleaning text data.
gensim - word2vec implementation - cython is required for ensuring speedy computations.
End of explanation
# load sentence tokenizer model and initialize for german language
nltk.download("punkt")
tokenizer = nltk.data.load('tokenizers/punkt/german.pickle')
Explanation: Prepare text processing
End of explanation
def clean_text( input, remove_stopwords = False ):
    
    Remove HTML, non-letter characters, and convert to lower case.
    Return list of words.
    
    
    # remove HTML markup with BeautifulSoup (and keep spaces after removal)
    plainText = " ".join(BeautifulSoup(input, 'html.parser').strings)
    
    # retain only letters (include umlauts)
    onlyLetters = re.sub(u"[^a-zA-ZäöüÄÖÜß]", " ", plainText)
    
    # get lower case words
    words = onlyLetters.lower().split()
    
    # use set of stop words
    if remove_stopwords:
        stops = set(stopwords.words("english"))
        words = [w for w in words if not w in stops]
    
    return words
def clean_sentences( text, remove_stopwords = False ):
    
    Split text by sentences and clean each sentence.
    
    sentences = tokenizer.tokenize(text)
    return filter(None, [clean_text(line, remove_stopwords) for line in sentences])
sentences = []
size = train['instructions'].size
for i in xrange ( 0, size ):
    
    if (i+1) % 1000 == 0:
        print "Processing %d of %d recipies." % ( i+1, size )
    sentences += clean_sentences(train['instructions'][i])
print "Total: %d sentences.\n" % len(sentences)
print "Example Sentences:\n", "\n".join([",".join(sentence) for sentence in sentences[0:3]])
Explanation: Define functions for cleaning the text data.
End of explanation
# Set values for various parameters
num_features = 300    # Word vector dimensionality                      
min_word_count = 40   # Minimum word count                        
num_workers = 4       # Number of threads to run in parallel
context = 10          # Context window size                                                                                    
downsampling = 1e-3   # Downsample setting for frequent words
# Import the built-in logging module and configure it so that Word2Vec creates nice output messages
import logging
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
print "Training model..."
model = Word2Vec(sentences, workers=num_workers, \
            size=num_features, min_count = min_word_count, \
            window = context, sample = downsampling)
print "%d words in vocabular." % len(model.wv.vocab)
Explanation: Initialize and train the model
this will take some time...
End of explanation
# make the model much more memory-efficient.
model.init_sims(replace=True)
model_name = "recipes-words_%dfeatures_%dminwords_%dcontext" % (num_features, min_word_count, context)
model.save(model_name)
Explanation: Save the model for later use or continued training
End of explanation
model.doesnt_match("milch brot eier".split())
model.most_similar("braten")
model.most_similar("pasta")
model.most_similar("brownies")
model.most_similar("lasagne")
Explanation: Some model evaluation examples:
End of explanation
<END_TASK>