{"action":"closed","issue":{"url":"https://api.github.com/repos/Rdatatable/data.table/issues/1433","labels_url":"https://api.github.com/repos/Rdatatable/data.table/issues/1433/labels{/name}","comments_url":"https://api.github.com/repos/Rdatatable/data.table/issues/1433/comments","events_url":"https://api.github.com/repos/Rdatatable/data.table/issues/1433/events","html_url":"https://github.com/Rdatatable/data.table/issues/1433","id":116987423,"number":1433,"title":"dim.data.table and alloc.col are rather slow due to [[.data.frame calls","user":{"login":"cooldome","id":8496051,"avatar_url":"https://avatars.githubusercontent.com/u/8496051?v=3","gravatar_id":"","url":"https://api.github.com/users/cooldome","html_url":"https://github.com/cooldome","followers_url":"https://api.github.com/users/cooldome/followers","following_url":"https://api.github.com/users/cooldome/following{/other_user}","gists_url":"https://api.github.com/users/cooldome/gists{/gist_id}","starred_url":"https://api.github.com/users/cooldome/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cooldome/subscriptions","organizations_url":"https://api.github.com/users/cooldome/orgs","repos_url":"https://api.github.com/users/cooldome/repos","events_url":"https://api.github.com/users/cooldome/events{/privacy}","received_events_url":"https://api.github.com/users/cooldome/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Rdatatable/data.table/labels/Enhancement","name":"Enhancement","color":"bfe5bf"}],"state":"closed","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/Rdatatable/data.table/milestones/3","html_url":"https://github.com/Rdatatable/data.table/milestones/v1.9.8","labels_url":"https://api.github.com/repos/Rdatatable/data.table/milestones/3/labels","id":812342,"number":3,"title":"v1.9.8","description":"","creator":{"login":"arunsrinivasan","id":3102045,"avatar_url":"https://avatars.githubusercontent.com/u/3102045?v=3","gravatar_id":"","url":"https://api.github.com/users/arunsrinivasan","html_url":"https://github.com/arunsrinivasan","followers_url":"https://api.github.com/users/arunsrinivasan/followers","following_url":"https://api.github.com/users/arunsrinivasan/following{/other_user}","gists_url":"https://api.github.com/users/arunsrinivasan/gists{/gist_id}","starred_url":"https://api.github.com/users/arunsrinivasan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arunsrinivasan/subscriptions","organizations_url":"https://api.github.com/users/arunsrinivasan/orgs","repos_url":"https://api.github.com/users/arunsrinivasan/repos","events_url":"https://api.github.com/users/arunsrinivasan/events{/privacy}","received_events_url":"https://api.github.com/users/arunsrinivasan/received_events","type":"User","site_admin":false},"open_issues":74,"closed_issues":36,"state":"open","created_at":"2014-10-03T08:24:10Z","updated_at":"2015-11-19T20:21:29Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2015-11-15T11:15:46Z","updated_at":"2015-11-19T20:21:29Z","closed_at":"2015-11-19T20:21:29Z","body":"I was recently profiling a data.table heavy R script that involved lots of joins and linear scans on tables with number of rows from 35000 to 65000. I was surprised to see dim.data.table as top most contributor to the spent time. The reason was quickly identified, the problem in following implementation of `dim.data.table` is hidden in `x[[1L]]`:\r\n\r\n```R\r\ndim.data.table <- function(x) {\r\n    if (length(x)) c(length(x[[1L]]), length(x))\r\n    else c(0L,0L)\r\n    # TO DO: consider placing \"dim\" as an attibute updated on inserts. Saves this 'if'.\r\n}\r\n```\r\n`x[[1L]]` is behind the scene is calling `[[.data.frame`, which rather awful in terms of performance:\r\n\r\n```R\r\n> `[[.data.frame`\r\nfunction (x, ..., exact = TRUE) \r\n{\r\n    na <- nargs() - (!missing(exact))\r\n    if (!all(names(sys.call()) %in% c(\"\", \"exact\"))) \r\n        warning(\"named arguments other than 'exact' are discouraged\")\r\n    if (na < 3L) \r\n        (function(x, i, exact) if (is.matrix(i)) \r\n            as.matrix(x)[[i]]\r\n        else .subset2(x, i, exact = exact))(x, ..., exact = exact)\r\n    else {\r\n        col <- .subset2(x, ..2, exact = exact)\r\n        i <- if (is.character(..1)) \r\n            pmatch(..1, row.names(x), duplicates.ok = TRUE)\r\n        else ..1\r\n        col[[i, exact = exact]]\r\n    }\r\n}\r\n```\r\n\r\nThe second top most contributor was `alloc.col` that suffered from the same problem:\r\n```R\r\nalloc.col <- function(DT, n=getOption(\"datatable.alloccol\"), verbose=getOption(\"datatable.verbose\"))\r\n{\r\n...   \r\nfor (i in seq_along(ans)) {\r\n        # clear the same excluded by copyMostAttrib(). Primarily for data.table and as.data.table, but added here centrally (see #4890).\r\n        setattr(ans[[i]],\"names\",NULL)\r\n        setattr(ans[[i]],\"dim\",NULL)\r\n        setattr(ans[[i]],\"dimnames\",NULL)\r\n    }\r\n...\r\n}\r\n```\r\n\r\nThree `[[.data.frame` calls per column.  \r\n\r\n\r\n\r\n "}}