How to statistically identify outliers using Kmeans clustering
<p>I have following data:</p>

<pre><code>head(df.num1)
##   num_critic_for_reviews duration director_facebook_likes
## 1                    723      178                       0
## 2                    302      169                     563
## 3                    602      148                       0
## 4                    813      164                   22000
## 5                    388      100                     131
## 6                    462      132                     475
##   actor_3_facebook_likes actor_1_facebook_likes     gross num_voted_users
## 1                    855                   1000 760505847          886204
## 2                   1000                  40000 309404152          471220
## 3                    161                  11000 200074175          275868
## 4                  23000                  27000 448130642         1144337
## 5                    365                    131  46975183               8
## 6                    530                    640  73058679          212204
##   cast_total_facebook_likes facenumber_in_poster num_user_for_reviews
## 1                      4834                    0                 3054
## 2                     48350                    0                 1238
## 3                     11700                    1                  994
## 4                    106759                    0                 2701
## 5                       143                    0                  450
## 6                      1873                    1                  738
##    budget title_year actor_2_facebook_likes imdb_score aspect_ratio
## 1 2.4e+08       2009                    936        7.9          1.8
## 2 3.0e+08       2007                   5000        7.1          2.4
## 3 2.4e+08       2015                    393        6.8          2.4
## 4 2.5e+08       2012                  23000        8.5          2.4
## 5 1.0e+07       2015                     12        7.1          2.4
## 6 2.6e+08       2012                    632        6.6          2.4
##   movie_facebook_likes
## 1                33000
## 2                    0
## 3                85000
## 4               164000
## 5                    0
 ## 6                24000
</code></pre>

<p>Then I run kmeans as below:</p>

<pre><code>set.seed(111)
km_out &lt;- kmeans(df.num1,centers=3) #perform kmeans cluster with k=3
</code></pre>

<p>we now calculate the distance between the objects and cluster centers to determine the outliers and identify say 5 largest distances which are outliers (arbitrary identification). </p>

<pre><code>centers &lt;- km_out$centers[km_out$cluster, ] # "centers" is a data frame of 3    centers but the length of dataset so we can calculate distance difference easily.
distances &lt;- sqrt(rowSums((df.num1 - centers)^2))
(outliers &lt;- order(distances, decreasing=T)[1:5])# these rows are 5 top outliers
## [1] 3860 3006 2324 2335 3424
</code></pre>

<p>lets get the dataframe with distance appended:</p>

<pre><code>df.num1$distance&lt;-distances
df.num1$cluster&lt;-km_out$cluster
</code></pre>

<p>print the details about the outliers (largest five distance values)</p>

<pre><code>(df.num1[outliers,])

##      num_critic_for_reviews duration director_facebook_likes
## 3860                    202      112                       0
## 3006                     73      134                      45
## 2324                    174      134                    6000
## 2335                    105      103                      78
## 3424                    150      124                      78
##      actor_3_facebook_likes actor_1_facebook_likes   gross num_voted_users
## 3860                     38                    717  211667           53508
## 3006                      0                      9  195888            5603
## 2324                    745                    893 2298191          221552
## 2335                    101                    488  410388           13727
## 3424                      4                      6  439162          106160
##      cast_total_facebook_likes facenumber_in_poster num_user_for_reviews
## 3860                       907                    0                  131
## 3006                        11                    0                   45
## 2324                      2710                    0                  570
## 2335                       991                    1                   79
## 3424                        28                    0                  430
##       budget title_year actor_2_facebook_likes imdb_score aspect_ratio
## 3860 4.2e+09       2005                    126        7.7          2.4
## 3006 2.5e+09       2005                      2        7.1          2.4
## 2324 2.4e+09       1997                    851        8.4          1.8
## 2335 2.1e+09       2004                    336        6.9          1.8
## 3424 1.1e+09       1988                      5        8.1          1.8
##      movie_facebook_likes distance cluster
## 3860                 4000  4.1e+09       2
## 3006                  607  2.4e+09       2
## 2324                11000  2.3e+09       2
## 2335                  973  2.0e+09       2
## 3424                    0  9.8e+08       2
</code></pre>

<p>But these are just data points selected on basis of largest distance from the cluster centers.....</p>

<p>What i wud like is something based on statistical measure such as extreme value based on z score (say > 2sd defined as outlier) instead of just taking few largest distance value obs(rows)........ </p>

<p>final output ideas such as:</p>

<p><a href="https://i.stack.imgur.com/ZNEON.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ZNEON.png" alt="enter image description here"></a></p>

<p>Or better still something like this:</p>

<p><a href="https://i.stack.imgur.com/HDrNm.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/HDrNm.png" alt="enter image description here"></a></p>

<p><a href="https://i.stack.imgur.com/pi0eB.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/pi0eB.png" alt="enter image description here"></a></p>

<p>Would be obliged if some help/pointers to get the kind of result as shown above ......</p>

<p>Regards</p>