How to use a fixed validation set (not K-fold cross validation) in Scikit-learn for a decision tree classifier/random forest classifier?
<p>I am new to machine learning and data science. Sorry, if it is a very stupid question.   </p>

<p>I see there is an inbuilt function for cross-validation but not for a fixed validation set. I have a dataset with 50,000 samples labeled with years from 1990 to 2010. I need to train different classifiers on 1990-2008 samples, then validate on 2009 samples, and test on 2010 samples. </p>

<p>EDIT:
After @Quan Tran's answer, I tried this. This is how it should be?</p>

<pre><code># Fit a decision tree
estimator1 = DecisionTreeClassifier( max_depth = 9, max_leaf_nodes=9)
estimator1.fit(X_train, y_train)
print estimator1


# validate using validation set
acc = np.zeros((20,20))  # store accuracy 
for i in range(20):
     for j in range(20):
         estimator1 = DecisionTreeClassifier(max_depth = i+1, max_leaf_nodes=j+2)
         estimator1.fit(X_valid, y_valid)
         y_pred = estimator1.predict(X_valid)
         acc[i,j] = accuracy_score(y_valid, y_pred)

best_mod = np.where(acc == acc.max())
print best_mod
print acc[best_mod]



 # Predict target values
estimator1 = DecisionTreeClassifier(max_depth = int(best_mod[0]) + 1, max_leaf_nodes= int(best_mod[1]) + 2)
estimator1.fit(X_valid, y_valid)
y_pred = estimator1.predict(X_test)
confusion = metrics.confusion_matrix(y_test, y_pred)

TP = confusion[1, 1]
TN = confusion[0, 0]
FP = confusion[0, 1]
FN = confusion[1, 0]


# Classification Accuracy
print "======= ACCURACY ========"
print((TP + TN) / float(TP + TN + FP + FN))
print accuracy_score(y_valid, y_pred)
# store the predicted probabilities for class 
y_pred_prob = estimator1.predict_proba(X_test)[:, 1]


# plot a ROC curve for y_test and y_pred_prob
fpr, tpr, thresholds = metrics.roc_curve(y_test, y_pred_prob)
plt.plot(fpr, tpr)
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.0])
plt.title('ROC curve for DecisionTreeClassifier')
plt.xlabel('False Positive Rate (1 - Specificity)')
plt.ylabel('True Positive Rate (Sensitivity)')
plt.grid(True)

print("======= AUC ========")
print(metrics.roc_auc_score(y_test, y_pred_prob))
</code></pre>

<p>I get this answer, which is <strong>not the best accuracy</strong>.</p>

<pre><code>DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=9,
        max_features=None, max_leaf_nodes=9, min_samples_leaf=1,
        min_samples_split=2, min_weight_fraction_leaf=0.0,
        presort=False, random_state=None, splitter='best')
(array([5]), array([19]))
[ 0.8489011]
======= ACCURACY ========
0.574175824176
0.538461538462
======= AUC ========
0.547632099893
</code></pre>