<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Context Based Sentence Classification
This program classifies sentences based on the context and predicts whether a sentence might be related to a "patient" or "doctor" spoken sentences.
<strong> Run the cell below to import all the required modules
Step2:  Data collection
The data is crawled from www.askthedoctor.com.
The website contains data of questions asked by the patients, and the corresponding answers given by the doctor.
The data is categorized into different categories based on the diseases. Here, each of the category is looped and corresponding data is stored as "patient" data or "doctor" data
Run the code below to collect data from the above website. 
Two files "patient.txt" and "doctor.txt" are saved
<strong>Note that it might take several minutes depending on the internet connection. 
<strong>Skip the below cell if the data is already saved.
Step3:  Word2Vec
Word2vec is a method of word embeddings where the words in a sentence are mapped to their corresponding vectors representation.
Here the whole dataset is considered, both patient and doctor spoken sentences to learn word embeddings.
A python library "gensim" is used to train a word2vec model
<strong> Run the code below to generate a word2vec model. 
<string> Skip the cell below if model is already created.
Step4:  Testing Word2Vec Model
Given a word, the model should be able to give similar words after being trained depending on the context words that appeared in sentences of the dataset.
Step5:  Saving the features
Each sentence vector is formed by averaging the vectors corresponding to each word in a sentence. 
The whole set of features is divided into train and test features.
<strong> This might take some time
<strong> Skip the cell if features are already saved
Step6:  Training Classifier Model
A simple SVM classifier is trained by converting a sentence into vectors.
<strong> Run the following code a train a simple SVM classifier and save the model. This might take some time
<Strong> Skip the cell if model is already generated.
Step7:  Evaluating Classifier Model
Step8:  Trying out some random sentences
<ASSISTANT_TASK:>
Python Code:
# Importing all the required modules and the helper functions 
import numpy as np 
import urllib.request
from bs4 import BeautifulSoup
from nltk import sent_tokenize
from nltk import word_tokenize
import re
from gensim.models import Word2Vec
import pickle 
# the following two modules are helper functions to generate features from the sentences
def get_feature(text,feature_dimension,wordset,model, label = None):
    features = None
    
    for sample in text:
        paragraph = sample.lower()
        sentences = sent_tokenize(paragraph)
        for sentence in sentences:
            feature_vector = np.zeros(feature_dimension)
            words = word_tokenize(sentence)
            count = 0
            for word in words:
                if word in wordset and word.isalnum():
                    count = count + 1
                    feature_vector = feature_vector + model[word]
            if count != 0:
                feature_vector = feature_vector / float(count)
                if label is not None:
                    feature_vector = np.append(feature_vector, label)
                feature_vector = feature_vector[np.newaxis]
                if features is None:
                    features = feature_vector
                else:
                    features = np.concatenate((features, feature_vector))
    return features
def generate_features(feature_dimension,wordset,model):
    with open("patient.txt") as patfile:
        patient = patfile.readlines()
    patfile.close()
    with open("doctor.txt") as docfile:
        doctor = docfile.readlines()
    docfile.close()
    patient_features = get_feature(patient,feature_dimension,wordset,model,label=0)
    doctor_features = get_feature(doctor,feature_dimension,wordset,model,label=1)
    features = np.concatenate((patient_features,doctor_features))
    return features
def predict(clf, text,feature_dimension, wordset, model):
    paragraph = text.lower()
    sentences = sent_tokenize(paragraph)   
    
    features = get_feature([text],feature_dimension,wordset,model)
    
    pred = clf.predict(features)
                       
    for i,item in enumerate(pred):
        if item == 0:
            ret = "patient"
        else:
            ret = "doctor"
        print("{} : {}".format(sentences[i],ret))
    
    print()
        
    return pred
Explanation: Context Based Sentence Classification
This program classifies sentences based on the context and predicts whether a sentence might be related to a "patient" or "doctor" spoken sentences.
<strong> Run the cell below to import all the required modules
End of explanation
base_url = "https://www.askthedoctor.com/browse-medical-questions"
base_f = urllib.request.urlopen(base_url)
base_soup = BeautifulSoup(base_f,"lxml")
# categories of diseases 
categories = [(base_anchor["href"],base_anchor["title"]) for base_div in base_soup.findAll("div",{"class":"disease_column"}) for base_anchor in base_div.findAll("a",{"itemtype":"https://schema.org/category"})]
print("Collecting data ... ")
with open("patient.txt","w") as patientfile, open("doctor.txt", "w") as doctorfile:
    for category in categories:
        topic = category[1]
        print(topic)
        try:
            url = category[0]
            f = urllib.request.urlopen(url)
            soup = BeautifulSoup(f,"lxml")
            divs = soup.findAll('div',{"class":"question_az"})
            for i,div in enumerate(divs):
                inner_url = div.find('a')['href']
                inner_f = urllib.request.urlopen(inner_url)
                inner_soup = BeautifulSoup(inner_f,"lxml")
                question = inner_soup.find('span',{"class":"quesans"})
                question = question.text.replace(","," ")
                question = re.sub('[.]+', '.',question)
                for token in sent_tokenize(question):
                    if len(word_tokenize(token)) > 3:
                        patientfile.write("{}\n".format(token))
                answer = inner_soup.find('span', {"class": "answer quesans"})
                answer = answer.text.replace( \n(adsbygoogle = window.adsbygoogle || []).push({});,"").replace("\n"," ").replace("   "," ").replace(","," ")
                answer = re.sub('[.]+', '.',answer)
                for token in sent_tokenize(answer):
                    if len(word_tokenize(token)) > 3:
                        doctorfile.write("{}\n".format(token))
        except:
            print("Error ................ {}".format(topic))
            
patientfile.close()
doctorfile.close()
print("Data saved !")
Explanation: Data collection
The data is crawled from www.askthedoctor.com.
The website contains data of questions asked by the patients, and the corresponding answers given by the doctor.
The data is categorized into different categories based on the diseases. Here, each of the category is looped and corresponding data is stored as "patient" data or "doctor" data
Run the code below to collect data from the above website. 
Two files "patient.txt" and "doctor.txt" are saved
<strong>Note that it might take several minutes depending on the internet connection. 
<strong>Skip the below cell if the data is already saved.
End of explanation
data_matrix = [] 
with open("patient.txt","r") as patfile, open("doctor.txt","r") as docfile:
    data_matrix = patfile.readlines()
    data_matrix.extend(docfile.readlines())
    
patfile.close()
docfile.close()
# converting the whole data into lower case
data_matrix = [sample.lower() for sample in data_matrix]
print("The Dataset consists of {} sentences.".format(len(data_matrix)))
# Formatting the data to provide as input to gensim package's word2vec model
words_matrix = []
for sample in data_matrix:
    sentences = sent_tokenize(sample)
    for sentence in sentences:
        words = word_tokenize(sentence)
        words_new = [word for word in words if word.isalnum()]
        words_matrix.append(words_new)
        
        
import logging
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s',\
    level=logging.INFO)
# Parameters required for training word2vec model
num_features = 300    # Word vector dimensionality
min_word_count = 5   # Minimum word count
num_workers = 4       # Number of threads to run in parallel
context = 10          # Context window size
downsampling = 1e-3   # Downsample setting for frequent words
# Initialize and train the model (this will take some time)
from gensim.models import word2vec
print("Training model...")
model = word2vec.Word2Vec(words_matrix, workers=num_workers, \
            size=num_features, min_count = min_word_count, \
            window = context, sample = downsampling)
# If you don't plan to train the model any further, calling
# init_sims will make the model much more memory-efficient.
model.init_sims(replace=True)
# It can be helpful to create a meaningful model name and
# save the model for later use. You can load it later using Word2Vec.load()
model_name = "word2vec_model"
model.save(model_name)
print("Word2Vec model saved !")
Explanation: Word2Vec
Word2vec is a method of word embeddings where the words in a sentence are mapped to their corresponding vectors representation.
Here the whole dataset is considered, both patient and doctor spoken sentences to learn word embeddings.
A python library "gensim" is used to train a word2vec model
<strong> Run the code below to generate a word2vec model. 
<string> Skip the cell below if model is already created.
End of explanation
model = Word2Vec.load("word2vec_model")
word_vectors = model.wv.syn0
print("The model has {} words in the vocabulary and the dimension of the vectors is {}".format(word_vectors.shape[0],word_vectors.shape[1]))
print("I\n{}\n".format(model.most_similar("i")))
print("swelling\n{}\n".format(model.most_similar("swelling")))
print("headache\n{}\n".format(model.most_similar("headache")))
print("fever\n{}\n".format(model.most_similar("fever")))
Explanation: Testing Word2Vec Model
Given a word, the model should be able to give similar words after being trained depending on the context words that appeared in sentences of the dataset.
End of explanation
model = Word2Vec.load("word2vec_model")
word_vectors = model.wv.syn0
feature_dimension = word_vectors.shape[1]
# all words in the vocabulary
wordset = set(model.wv.index2word)
print("Generating features ...")
features = generate_features(feature_dimension,wordset,model)
# dividing the dataset into train and test datasets.
indices = np.random.permutation(features.shape[0])
test_idx,training_idx = indices[:2000], indices[2000:]
test_features, train_features = features[test_idx,:], features[training_idx,:]
train_labels =  train_features[:,-1]
train_features = train_features[:,:-1]
test_labels =  test_features[:,-1]
test_features = test_features[:,:-1]
# Saving features 
np.save("train_features.npy",train_features)
np.save("train_labels.npy", train_labels)
np.save("test_features.npy",test_features)
np.save("test_labels.npy",test_labels)
print("Features saved")
Explanation: Saving the features
Each sentence vector is formed by averaging the vectors corresponding to each word in a sentence. 
The whole set of features is divided into train and test features.
<strong> This might take some time
<strong> Skip the cell if features are already saved
End of explanation
# Loading train features
train_features = np.load("train_features.npy")
train_labels = np.load("train_labels.npy")
from sklearn.svm import SVC
clf = SVC(kernel="linear",C=100)
print("Training classifier ....")
clf.fit(train_features,train_labels)
import pickle 
with open("clf_model.pkl","wb") as clffile:
    pickle.dump(clf,clffile)
    
clffile.close()
print("Classifier Model saved !")
Explanation: Training Classifier Model
A simple SVM classifier is trained by converting a sentence into vectors.
<strong> Run the following code a train a simple SVM classifier and save the model. This might take some time
<Strong> Skip the cell if model is already generated.
End of explanation
with open("clf_model.pkl","rb") as clffile:
    clf = pickle.load(clffile)
    
clffile.close()
   
test_features = np.load("test_features.npy")
test_labels = np.load("test_labels.npy")
pred = clf.predict(test_features)
from sklearn.metrics import accuracy_score
acc = accuracy_score(test_labels,pred)*100
print("The accuracy for {} samples for the model : {}%".format(test_features.shape[0],acc))
Explanation: Evaluating Classifier Model
End of explanation
with open("clf_model.pkl","rb") as clffile:
    clf = pickle.load(clffile)
clffile.close()
model = Word2Vec.load("word2vec_model")
word_vectors = model.wv.syn0
feature_dimension = word_vectors.shape[1]
# all words in the vocabulary
wordset = set(model.wv.index2word)
text = "i still cough few times a day. what should i do?"
res = predict(clf, text,feature_dimension, wordset, model)
text = "i have severe pain in my abdomen. do i have to go to the doctor? wash your hands everytime and follow hygenic practices"
res = predict(clf, text,feature_dimension, wordset, model)
text = "i have a sore throat. it has been there for the past week."
res = predict(clf, text,feature_dimension, wordset, model)
text = "do you have sore throat? Does your throat feel itchy? Do you have flu?"
res = predict(clf, text,feature_dimension, wordset, model)
text = "you should apply neomycin ointment on your chin"
res = predict(clf, text,feature_dimension, wordset, model)
text = "do you think I have infection which is causing my blood pressure to rise? Yes, your blood pressure is increasing because of infection"
res = predict(clf, text,feature_dimension, wordset, model)
text = "Are you comfortable? If you are not comfortable, please let me know. No I am not comfortable and in too much in pain right now."
res = predict(clf, text,feature_dimension, wordset, model)
Explanation: Trying out some random sentences
End of explanation
<END_TASK>