{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4898","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4898/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4898/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4898/events","html_url":"https://github.com/scikit-learn/scikit-learn/issues/4898","id":90997709,"number":4898,"title":"Profiling RandomForestClassifier for bottlenecks - _set_oob_score","user":{"login":"dtsmith2001","id":3189861,"avatar_url":"https://avatars.githubusercontent.com/u/3189861?v=3","gravatar_id":"","url":"https://api.github.com/users/dtsmith2001","html_url":"https://github.com/dtsmith2001","followers_url":"https://api.github.com/users/dtsmith2001/followers","following_url":"https://api.github.com/users/dtsmith2001/following{/other_user}","gists_url":"https://api.github.com/users/dtsmith2001/gists{/gist_id}","starred_url":"https://api.github.com/users/dtsmith2001/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dtsmith2001/subscriptions","organizations_url":"https://api.github.com/users/dtsmith2001/orgs","repos_url":"https://api.github.com/users/dtsmith2001/repos","events_url":"https://api.github.com/users/dtsmith2001/events{/privacy}","received_events_url":"https://api.github.com/users/dtsmith2001/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-06-25T15:34:50Z","updated_at":"2015-06-25T16:32:52Z","closed_at":null,"body":"I observed a long delay between fitting trees after constructing a **RandomForestClassifier** with n_jobs=20. I fired up iPython and used\r\n\r\n```python\r\n%runit -l forest.py forest.fit(X, y)\r\n```\r\n\r\n(where the forest object is the constructed classifier - see below). I found that the function **_set_oob_score** was taking up most of the time. This function is in the class **ForestClassifier**.\r\n\r\nI am using scikit-learn version 0.16.1 with Anaconda 3 on Windows Server 2008 R2 Enterprise x64.\r\n\r\nHere is the code I’m using to create the forest object I use with %runit:\r\n\r\n```python\r\nimport sklearn as sk\r\nfrom sklearn.ensemble import RandomForestClassifier\r\nfrom sklearn.datasets import make_classification\r\n\r\nprint(\"scikit-learn version {0}\".format(sk.__version__))\r\n\r\nX, y = make_classification(n_samples=5000,\r\n                     n_features=30, n_informative=10, n_redundant=0, n_repeated=0,\r\n                     n_classes=2, random_state=0, weights=[0.9, 0.1],  # unbalanced\r\n                     shuffle=False)\r\n\r\nprint(\"Number of cases {0} number of features {1}\".format(X.shape[0], X.shape[1]))\r\nforest = RandomForestClassifier(n_estimators=50,\r\n                     random_state=100, oob_score=True,\r\n                     bootstrap=True, verbose=2)\r\n```\r\n\r\nAt this point we are ready to use %runit as I illustrated above.\r\n\r\nMy own use case corresponds to using n_samples=400000 and n_features=3500 (n_informative is not relevant in this example).\r\n\r\nThe issue here is that **_set_oob_score** uses vastly more time than actually creating the trees, particularly with my use case of 400,000 samples and 3,500 features.\r\n\r\nHere is a completely reproducible example of the code I'm running. I can see in Task Manager that the processors are in use generating the trees, but after the number of processors in use goes back to one, there is a long delay before fit finishes. Note that this code uses 20 processors, over 17 gb memory, and takes a lot of time on my own computer. YMMV.\r\n\r\n```python\r\nimport sklearn as sk\r\nfrom sklearn.ensemble import RandomForestClassifier\r\nfrom sklearn.datasets import make_classification\r\n\r\nprint(\"scikit-learn version {0}\".format(sk.__version__))\r\n\r\nX, y = make_classification(n_samples=400000, n_features=3500,\r\n\t\t\t\t\t\t   n_informative=10, n_redundant=0, n_repeated=0,\r\n\t\t\t\t\t\t   n_classes=2, random_state=0, weights=[0.9, 0.1],  # unbalanced\r\n\t\t\t\t\t\t   shuffle=False)\r\n\r\nprint(\"Number of cases {0} number of features {1}\".format(X.shape[0], X.shape[1]))\r\nforest = RandomForestClassifier(n_estimators=50, n_jobs=20,\r\n\t\t\t\t\t\t\t\trandom_state=100, oob_score=True, bootstrap=True, verbose=2)\r\nforest.fit(X, y)\r\nprint(\"Finished\")\r\n```\r\n\r\nPlease rewrite **_set_oob_score** to improve performance.\r\n"},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/115313200","html_url":"https://github.com/scikit-learn/scikit-learn/issues/4898#issuecomment-115313200","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4898","id":115313200,"user":{"login":"dtsmith2001","id":3189861,"avatar_url":"https://avatars.githubusercontent.com/u/3189861?v=3","gravatar_id":"","url":"https://api.github.com/users/dtsmith2001","html_url":"https://github.com/dtsmith2001","followers_url":"https://api.github.com/users/dtsmith2001/followers","following_url":"https://api.github.com/users/dtsmith2001/following{/other_user}","gists_url":"https://api.github.com/users/dtsmith2001/gists{/gist_id}","starred_url":"https://api.github.com/users/dtsmith2001/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dtsmith2001/subscriptions","organizations_url":"https://api.github.com/users/dtsmith2001/orgs","repos_url":"https://api.github.com/users/dtsmith2001/repos","events_url":"https://api.github.com/users/dtsmith2001/events{/privacy}","received_events_url":"https://api.github.com/users/dtsmith2001/received_events","type":"User","site_admin":false},"created_at":"2015-06-25T16:32:52Z","updated_at":"2015-06-25T16:32:52Z","body":"I followed the instructions in the section Profiling Python code\r\n\r\nhttp://scikit-learn.org/stable/developers/performance.html\r\n\r\nand produced the following, using the smaller example in my original post.\r\n\r\n```\r\nLine #      Hits         Time  Per Hit   % Time  Line Contents\r\n==============================================================\r\n   168                                               def fit(self, X, y, sample_weight=None):\r\n   169                                                   \"\"\"Build a forest of trees from the training set (X, y).\r\n   170                                           \r\n   171                                                   Parameters\r\n   172                                                   ----------\r\n   173                                                   X : array-like or sparse matrix of shape = [n_samples, n_features]\r\n   174                                                       The training input samples. Internally, it will be converted to\r\n   175                                                       ``dtype=np.float32`` and if a sparse matrix is provided\r\n   176                                                       to a sparse ``csc_matrix``.\r\n   177                                           \r\n   178                                                   y : array-like, shape = [n_samples] or [n_samples, n_outputs]\r\n   179                                                       The target values (class labels in classification, real numbers in\r\n   180                                                       regression).\r\n   181                                           \r\n   182                                                   sample_weight : array-like, shape = [n_samples] or None\r\n   183                                                       Sample weights. If None, then samples are equally weighted. Splits\r\n   184                                                       that would create child nodes with net zero or negative weight are\r\n   185                                                       ignored while searching for a split in each node. In the case of\r\n   186                                                       classification, splits are also ignored if they would result in any\r\n   187                                                       single class carrying a negative weight in either child node.\r\n   188                                           \r\n   189                                                   Returns\r\n   190                                                   -------\r\n   191                                                   self : object\r\n   192                                                       Returns self.\r\n   193                                                   \"\"\"\r\n   194                                                   # Validate or convert input data\r\n   195         1         1411   1411.0      0.0          X = check_array(X, dtype=DTYPE, accept_sparse=\"csc\")\r\n   196         1           15     15.0      0.0          if issparse(X):\r\n   197                                                       # Pre-sort indices to avoid that each individual tree of the\r\n   198                                                       # ensemble sorts the indices.\r\n   199                                                       X.sort_indices()\r\n   200                                           \r\n   201                                                   # Remap output\r\n   202         1           20     20.0      0.0          n_samples, self.n_features_ = X.shape\r\n   203                                           \r\n   204         1           40     40.0      0.0          y = np.atleast_1d(y)\r\n   205         1            7      7.0      0.0          if y.ndim == 2 and y.shape[1] == 1:\r\n   206                                                       warn(\"A column-vector y was passed when a 1d array was\"\r\n   207                                                            \" expected. Please change the shape of y to \"\r\n   208                                                            \"(n_samples,), for example using ravel().\",\r\n   209                                                            DataConversionWarning, stacklevel=2)\r\n   210                                           \r\n   211         1            7      7.0      0.0          if y.ndim == 1:\r\n   212                                                       # reshape is necessary to preserve the data contiguity against vs\r\n   213                                                       # [:, np.newaxis] that does not.\r\n   214         1           37     37.0      0.0              y = np.reshape(y, (-1, 1))\r\n   215                                           \r\n   216         1           11     11.0      0.0          self.n_outputs_ = y.shape[1]\r\n   217                                           \r\n   218         1         1147   1147.0      0.0          y, expanded_class_weight = self._validate_y_class_weight(y)\r\n   219                                           \r\n   220         1           15     15.0      0.0          if getattr(y, \"dtype\", None) != DOUBLE or not y.flags.contiguous:\r\n   221         1           56     56.0      0.0              y = np.ascontiguousarray(y, dtype=DOUBLE)\r\n   222                                           \r\n   223         1            7      7.0      0.0          if expanded_class_weight is not None:\r\n   224                                                       if sample_weight is not None:\r\n   225                                                           sample_weight = sample_weight * expanded_class_weight\r\n   226                                                       else:\r\n   227                                                           sample_weight = expanded_class_weight\r\n   228                                           \r\n   229                                                   # Check parameters\r\n   230         1           28     28.0      0.0          self._validate_estimator()\r\n   231                                           \r\n   232         1            8      8.0      0.0          if not self.bootstrap and self.oob_score:\r\n   233                                                       raise ValueError(\"Out of bag estimation only available\"\r\n   234                                                                        \" if bootstrap=True\")\r\n   235                                           \r\n   236         1          117    117.0      0.0          random_state = check_random_state(self.random_state)\r\n   237                                           \r\n   238         1           10     10.0      0.0          if not self.warm_start:\r\n   239                                                       # Free allocated memory, if any\r\n   240         1          264    264.0      0.0              self.estimators_ = []\r\n   241                                           \r\n   242         1           13     13.0      0.0          n_more_estimators = self.n_estimators - len(self.estimators_)\r\n   243                                           \r\n   244         1            7      7.0      0.0          if n_more_estimators < 0:\r\n   245                                                       raise ValueError('n_estimators=%d must be larger or equal to '\r\n   246                                                                        'len(estimators_)=%d when warm_start==True'\r\n   247                                                                        % (self.n_estimators, len(self.estimators_)))\r\n   248                                           \r\n   249         1            7      7.0      0.0          elif n_more_estimators == 0:\r\n   250                                                       warn(\"Warm-start fitting without increasing n_estimators does not \"\r\n   251                                                            \"fit new trees.\")\r\n   252                                                   else:\r\n   253         1            8      8.0      0.0              if self.warm_start and len(self.estimators_) > 0:\r\n   254                                                           # We draw from the random state to get the random state we\r\n   255                                                           # would have got if we hadn't used a warm_start.\r\n   256                                                           random_state.randint(MAX_INT, size=len(self.estimators_))\r\n   257                                           \r\n   258         1            7      7.0      0.0              trees = []\r\n   259        51          384      7.5      0.0              for i in range(n_more_estimators):\r\n   260        50       346522   6930.4      8.7                  tree = self._make_estimator(append=False)\r\n   261        50       105158   2103.2      2.6                  tree.set_params(random_state=random_state.randint(MAX_INT))\r\n   262        50          511     10.2      0.0                  trees.append(tree)\r\n   263                                           \r\n   264                                                       # Parallel loop: we use the threading backend as the Cython code\r\n   265                                                       # for fitting the trees is internally releasing the Python GIL\r\n   266                                                       # making threading always more efficient than multiprocessing in\r\n   267                                                       # that case.\r\n   268         1           12     12.0      0.0              trees = Parallel(n_jobs=self.n_jobs, verbose=self.verbose,\r\n   269         1           67     67.0      0.0                               backend=\"threading\")(\r\n   270         1            9      9.0      0.0                  delayed(_parallel_build_trees)(\r\n   271                                                               t, self, X, y, sample_weight, i, len(trees),\r\n   272                                                               verbose=self.verbose, class_weight=self.class_weight)\r\n   273         1      3431107 3431107.0     86.3                  for i, t in enumerate(trees))\r\n   274                                           \r\n   275                                                       # Collect newly grown trees\r\n   276         1           20     20.0      0.0              self.estimators_.extend(trees)\r\n   277                                           \r\n   278         1           11     11.0      0.0          if self.oob_score:\r\n   279         1        87788  87788.0      2.2              self._set_oob_score(X, y)\r\n   280                                           \r\n   281                                                   # Decapsulate classes_ attributes\r\n   282         1           14     14.0      0.0          if hasattr(self, \"classes_\") and self.n_outputs_ == 1:\r\n   283         1           10     10.0      0.0              self.n_classes_ = self.n_classes_[0]\r\n   284         1            9      9.0      0.0              self.classes_ = self.classes_[0]\r\n   285                                           \r\n   286         1            7      7.0      0.0          return self\r\n```\r\n\r\nThe Parallel loop takes the most time, followed by\r\n\r\n```python\r\nself._make_estimator\r\ntree.set_params(...)\r\nself._set_oob_score(...)\r\n```\r\n\r\nin order of timing. Apologies for the poor formatting above, but I believe I should give the exact output of lprun rather than try to edit."}}