Random Forests and ROC Curves in Julia
<p>I'm using the ScikitLearn flavour of the <a href="https://github.com/bensadeghi/DecisionTree.jl" rel="nofollow noreferrer">DecisionTree.jl</a> package to create a random forest model for a binary classification problem of one of the <a href="https://github.com/johnmyleswhite/RDatasets.jl" rel="nofollow noreferrer">RDatasets</a> data sets (see bottom of the DecisionTree.jl home page for what I mean by ScikitLearn flavour). I'm also using the <a href="https://github.com/JuliaStats/MLBase.jl" rel="nofollow noreferrer">MLBase</a> package for model evaluation.</p>

<p>I have built a random forest model of my data and would like to create a ROC Curve for this model. Reading the documentation available, I do understand what a ROC curve is in theory. I just can't figure out how to create one for a specific model.</p>

<p>From the <a href="https://en.wikipedia.org/wiki/Receiver_operating_characteristic" rel="nofollow noreferrer">Wikipedia page</a> the last part of the first sentence that I have marked in bold italics below is the one that is causing my confusion: "In statistics, a receiver operating characteristic (ROC), or ROC curve, is a graphical plot that illustrates the performance of a binary classifier system <strong><em>as its discrimination threshold is varied</em></strong>." There is more on the threshold value throughout the article but this still confuses me for binary classification problems. What is the threshold value and how do I vary it? </p>

<p>Also, in the <a href="http://mlbasejl.readthedocs.io/en/latest/perfeval.html#receiver-operating-characteristics-roc" rel="nofollow noreferrer">MLBase documentation on ROC Curves</a> it says "Compute an ROC instance or an ROC curve (a vector of ROC instances), based on given scores and a threshold thres." But doesn't mention this threshold anywhere else really.</p>

<p>Example code for my project is given below. Basically, I want to create a ROC curve for the random forest but I'm not sure how to or if it's even appropriate.</p>

<pre><code>using DecisionTree
using RDatasets
using MLBase

quakes_data = dataset("datasets", "quakes");

# Add in a binary column as feature column for classification
quakes_data[:MagGT5] = convert(Array{Int32,1}, quakes_data[:Mag] .&gt; 5.0)

# Getting features and labels where label = 1 is mag &gt; 1 and label = 2 is mag &lt;= 5
features = convert(Array, quakes_data[:, [1:3;5]]);
labels = convert(Array, quakes_data[:, 6]);
labels[labels.==0] = 2

# Create a random forest model with the tuning parameters I want
r_f_model = RandomForestClassifier(nsubfeatures = 3, ntrees = 50, partialsampling=0.7, maxdepth = 4)

# Train the model in-place on the dataset (there isn't a fit function without the in-place functionality)
DecisionTree.fit!(r_f_model, features, labels)

# Apply the trained model to the test features data set (here I haven't partitioned into training and test)
r_f_prediction = convert(Array{Int64,1}, DecisionTree.predict(r_f_model, features))

# Applying the model to the training set and looking at model stats
TrainingROC = roc(labels, r_f_prediction) #getting the stats around the model applied to the train set
#     p::T    # positive in ground-truth
#     n::T    # negative in ground-truth
#     tp::T   # correct positive prediction
#     tn::T   # correct negative prediction
#     fp::T   # (incorrect) positive prediction when ground-truth is negative
#     fn::T   # (incorrect) negative prediction when ground-truth is positive
</code></pre>

<p>I also read <a href="https://stackoverflow.com/questions/19972736/plotting-a-roc-curve-from-a-random-forest-classification">this</a> question and didn't find it helpful really.</p>