{"action":"closed","issue":{"url":"https://api.github.com/repos/ContinuumIO/dask/issues/265","labels_url":"https://api.github.com/repos/ContinuumIO/dask/issues/265/labels{/name}","comments_url":"https://api.github.com/repos/ContinuumIO/dask/issues/265/comments","events_url":"https://api.github.com/repos/ContinuumIO/dask/issues/265/events","html_url":"https://github.com/ContinuumIO/dask/issues/265","id":84633382,"number":265,"title":"Block-wise implementation of randomized PCA","user":{"login":"alimanfoo","id":703554,"avatar_url":"https://avatars.githubusercontent.com/u/703554?v=3","gravatar_id":"","url":"https://api.github.com/users/alimanfoo","html_url":"https://github.com/alimanfoo","followers_url":"https://api.github.com/users/alimanfoo/followers","following_url":"https://api.github.com/users/alimanfoo/following{/other_user}","gists_url":"https://api.github.com/users/alimanfoo/gists{/gist_id}","starred_url":"https://api.github.com/users/alimanfoo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alimanfoo/subscriptions","organizations_url":"https://api.github.com/users/alimanfoo/orgs","repos_url":"https://api.github.com/users/alimanfoo/repos","events_url":"https://api.github.com/users/alimanfoo/events{/privacy}","received_events_url":"https://api.github.com/users/alimanfoo/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-06-03T15:51:31Z","updated_at":"2015-06-12T14:44:45Z","closed_at":"2015-06-12T14:44:45Z","body":"I work with large arrays of genotype data of shape (n_variants, n_samples, 2), where n_variants is ~20000000 and n_samples is currently ~1000. These data are derived from next-generation sequencing of Anopheles mosquitoes, which we are studying because of their role in malaria transmission in Africa. A typical analysis we perform is to first reduce this to a 2D array of shape (n_variants, n_samples) then perform a principal components analysis of variants (features) against samples (observations), to look for patterns in the genetic relatedness between mosquito samples. \r\n\r\nI have tried both conventional SVD via numpy.linalg and the randomized PCA implementation in scikit-learn. The randomized implementation is roughly an order of magnitude faster and uses about half the memory. Given the large number of features we have to deal with, both reductions in memory footprint and increases in speed are very valuable, enabling exploratory analysis of large datasets within an interactive session.\r\n\r\nToday we tried the dask block-wise SVD implementation which seems very promising because of the much-reduced memory requirements. If dask had a blocked implementation of randomized SVD that would be even better, making principal components analysis of these data very quick and scalable."}}