{"action":"created","issue":{"url":"https://api.github.com/repos/markvanderloo/stringdist/issues/45","labels_url":"https://api.github.com/repos/markvanderloo/stringdist/issues/45/labels{/name}","comments_url":"https://api.github.com/repos/markvanderloo/stringdist/issues/45/comments","events_url":"https://api.github.com/repos/markvanderloo/stringdist/issues/45/events","html_url":"https://github.com/markvanderloo/stringdist/issues/45","id":123685684,"number":45,"title":"stringdistmatrix gives unexpected output for large input vectors","user":{"login":"touwwouter","id":6943471,"avatar_url":"https://avatars.githubusercontent.com/u/6943471?v=3","gravatar_id":"","url":"https://api.github.com/users/touwwouter","html_url":"https://github.com/touwwouter","followers_url":"https://api.github.com/users/touwwouter/followers","following_url":"https://api.github.com/users/touwwouter/following{/other_user}","gists_url":"https://api.github.com/users/touwwouter/gists{/gist_id}","starred_url":"https://api.github.com/users/touwwouter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/touwwouter/subscriptions","organizations_url":"https://api.github.com/users/touwwouter/orgs","repos_url":"https://api.github.com/users/touwwouter/repos","events_url":"https://api.github.com/users/touwwouter/events{/privacy}","received_events_url":"https://api.github.com/users/touwwouter/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-12-23T16:20:32Z","updated_at":"2015-12-23T16:53:53Z","closed_at":null,"body":"When `stringdistmatrix` is called using a large single input vector, the result is a distance matrix of class `dist` with fewer elements than expected.\r\n\r\n```\r\nmany_words <- sapply(1:100000, function(x) paste(sample(letters, 10, replace=T),\r\n                                                 collapse=\"\"))\r\n# Needs a lot of memory\r\nd <- stringdist::stringdistmatrix(many_words, method = \"jw\")\r\nsize <- attr(d, \"Size\")\r\nstopifnot(inherits(d, \"dist\"))\r\nstopifnot(size == length(many_words))\r\nstopifnot(length(d) == size*(size - 1)/2)\r\n# Error: length(d) == size * (size - 1)/2 is not TRUE\r\n# > length(d)\r\n# [1] 704982704\r\n# > size*(size - 1)/2\r\n# [1] 4999950000\r\n```\r\n\r\nFor smaller character vectors I did not encounter any problems:\r\n```\r\nfew_words <- sapply(1:1000, function(x) paste(sample(letters, 10, replace=T),\r\n                                              collapse=\"\"))\r\nd <- stringdist::stringdistmatrix(few_words, method = \"jw\")\r\nstopifnot(inherits(d, \"dist\"))\r\nsize <- attr(d, \"Size\")\r\nstopifnot(size == length(few_words))\r\nstopifnot(length(d) == size*(size - 1)/2)\r\n# No errors raised\r\n```\r\n\r\nI can provide more info when necessary."},"comment":{"url":"https://api.github.com/repos/markvanderloo/stringdist/issues/comments/166942997","html_url":"https://github.com/markvanderloo/stringdist/issues/45#issuecomment-166942997","issue_url":"https://api.github.com/repos/markvanderloo/stringdist/issues/45","id":166942997,"user":{"login":"touwwouter","id":6943471,"avatar_url":"https://avatars.githubusercontent.com/u/6943471?v=3","gravatar_id":"","url":"https://api.github.com/users/touwwouter","html_url":"https://github.com/touwwouter","followers_url":"https://api.github.com/users/touwwouter/followers","following_url":"https://api.github.com/users/touwwouter/following{/other_user}","gists_url":"https://api.github.com/users/touwwouter/gists{/gist_id}","starred_url":"https://api.github.com/users/touwwouter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/touwwouter/subscriptions","organizations_url":"https://api.github.com/users/touwwouter/orgs","repos_url":"https://api.github.com/users/touwwouter/repos","events_url":"https://api.github.com/users/touwwouter/events{/privacy}","received_events_url":"https://api.github.com/users/touwwouter/received_events","type":"User","site_admin":false},"created_at":"2015-12-23T16:53:53Z","updated_at":"2015-12-23T16:53:53Z","body":"Thanks. I would expect a warning from the current version of stringdistmatrix, though, since it took me a while before I figured out the source of my downstream problems.\r\n\r\nOn my workstation with 40 threads, the run time is quite acceptable:\r\n```\r\n   user  system elapsed \r\n689.892   9.048  23.474\r\n```\r\n\r\nOn a single thread the elapsed time is slightly over 8 minutes:\r\n```\r\n   user  system elapsed \r\n489.896   7.756 497.547 \r\n```"}}