{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4301","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4301/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4301/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4301/events","html_url":"https://github.com/scikit-learn/scikit-learn/pull/4301","id":59257697,"number":4301,"title":"Choose number of clusters","user":{"login":"afouchet","id":7079913,"avatar_url":"https://avatars.githubusercontent.com/u/7079913?v=3","gravatar_id":"","url":"https://api.github.com/users/afouchet","html_url":"https://github.com/afouchet","followers_url":"https://api.github.com/users/afouchet/followers","following_url":"https://api.github.com/users/afouchet/following{/other_user}","gists_url":"https://api.github.com/users/afouchet/gists{/gist_id}","starred_url":"https://api.github.com/users/afouchet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/afouchet/subscriptions","organizations_url":"https://api.github.com/users/afouchet/orgs","repos_url":"https://api.github.com/users/afouchet/repos","events_url":"https://api.github.com/users/afouchet/events{/privacy}","received_events_url":"https://api.github.com/users/afouchet/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-02-27T16:20:09Z","updated_at":"2015-03-02T22:02:44Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/4301","html_url":"https://github.com/scikit-learn/scikit-learn/pull/4301","diff_url":"https://github.com/scikit-learn/scikit-learn/pull/4301.diff","patch_url":"https://github.com/scikit-learn/scikit-learn/pull/4301.patch"},"body":"This module implements 7 algorithms to find \"optimal\" number of clusters (stabiliity, gap statistic, distortion jump, silhouette, Calinsky and Harabasz index, and 2 \"elbow\" methods)\r\n\r\nIf a metric is needed (for example, to compute distortion, the mean distance of a point to its cluster's center), all distance of scipy.spatial.distance can be used.\r\n"},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/76836003","html_url":"https://github.com/scikit-learn/scikit-learn/pull/4301#issuecomment-76836003","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4301","id":76836003,"user":{"login":"ragv","id":9487348,"avatar_url":"https://avatars.githubusercontent.com/u/9487348?v=3","gravatar_id":"","url":"https://api.github.com/users/ragv","html_url":"https://github.com/ragv","followers_url":"https://api.github.com/users/ragv/followers","following_url":"https://api.github.com/users/ragv/following{/other_user}","gists_url":"https://api.github.com/users/ragv/gists{/gist_id}","starred_url":"https://api.github.com/users/ragv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ragv/subscriptions","organizations_url":"https://api.github.com/users/ragv/orgs","repos_url":"https://api.github.com/users/ragv/repos","events_url":"https://api.github.com/users/ragv/events{/privacy}","received_events_url":"https://api.github.com/users/ragv/received_events","type":"User","site_admin":false},"created_at":"2015-03-02T22:02:44Z","updated_at":"2015-03-02T22:02:44Z","body":"If I understand things correctly - \r\n\r\nI agree your point on not needing to split data to choose the optimal `n_clusters` since you might want to apply a particular metric (silhouette/distortion/any other) on the whole of your data...\r\nAlso `GridSearchCV` is to minimize a particular score for various combinations of parameters... And since your use case is a single parameter you could simply do a linear search...\r\n\r\nSo I suggest - split the core of your algorithm into 2 metric functions --> 1. which returns the raw value of distortion / index etc... 2. Which is a monotonic function of 1 which is always positively correlated with the optimality of k ( `n_clusters` ) such that minimizing this function over a range of k gives you the optimal value of `n_clusters`...\r\n\r\nFew naiive suggestions (could sound stupid, in which case kindly ignore it):\r\n\r\n> we expect this mean distance to decrease a lot if k < k_optimal, and to decrease very little for k > k_optimal. So I compute the clustering for all k, see the distortion (can be seen as mean distance of a point to its centroïd) an look for an inflection point\r\n\r\n1. a metric `clustering_distortion` / `distortion` that returns the distortion value given the data and labels. \r\nUsage : `distortion(data, model(n_clusters=k).fit(data).labels_)` --> distortion for the predicted data using model initialized with `n_clusters=k`.\r\n\r\n2. a metric `distortion_laplacian(data, n_clusters_range, Estimator, *args, **kwargs)` which computes the 2nd derivative of the `distortion` vs `n_clusters` curve at the point of given `n_clusters`...\r\n\r\n( or better `distortion_laplacians(data, range_n_clusters, Estimator, *args, **kwargs)`  which might be a lot more efficient since you will compute the labels for different `n_clusters` only once... and use `numpy.diff` directly on the results... )\r\n\r\nUsage for finding the optimal k\r\n\r\nSimply do a linear search - `range_k[argmin(distortion_laplacians(data, range_k, KMeans))]`\r\n\r\n> but, for the gap-statistic algorithm, you do not look for k maximizing the gap, but for the minimum k such as Gap(k) > Gap(k+1) \r\n\r\nYou could make 2 metric functions\r\n\r\n* `gap(k)` which returns the plain gap(k).\r\n* `gap_cluster_score(k)` which returns `k*(gap(k) - gap(k+1))` or even better `k*int(gap(k) > gap(k+1))`\r\n\r\n(That is crudely put, you ought to be taking in only data, labels for `gap()` like the above described distortion... similarly to avoid computing gap(k) multiple times we could have `gap_cluster_scores(Estimator, range_k, *args, **kwargs)` to return `k*(gap(k) - gap(k+1))` for a range of k values...)\r\n\r\nSimilarly you must be able to find a way to make the stability measure into a monotonically increasing metric value that is positively correlated with the k optimality...\r\n\r\nApologies if any of this sounds stupid w.r.t your situation/scikit learn's api :)"}}