{"action":"created","issue":{"url":"https://api.github.com/repos/pydata/pandas/issues/10161","labels_url":"https://api.github.com/repos/pydata/pandas/issues/10161/labels{/name}","comments_url":"https://api.github.com/repos/pydata/pandas/issues/10161/comments","events_url":"https://api.github.com/repos/pydata/pandas/issues/10161/events","html_url":"https://github.com/pydata/pandas/issues/10161","id":77316447,"number":10161,"title":"duplicated() performance and bug on long rows regression from 0.15.2->0.16.0","user":{"login":"eyaler","id":4436747,"avatar_url":"https://avatars.githubusercontent.com/u/4436747?v=3","gravatar_id":"","url":"https://api.github.com/users/eyaler","html_url":"https://github.com/eyaler","followers_url":"https://api.github.com/users/eyaler/followers","following_url":"https://api.github.com/users/eyaler/following{/other_user}","gists_url":"https://api.github.com/users/eyaler/gists{/gist_id}","starred_url":"https://api.github.com/users/eyaler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eyaler/subscriptions","organizations_url":"https://api.github.com/users/eyaler/orgs","repos_url":"https://api.github.com/users/eyaler/repos","events_url":"https://api.github.com/users/eyaler/events{/privacy}","received_events_url":"https://api.github.com/users/eyaler/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/pydata/pandas/labels/Difficulty%20Intermediate","name":"Difficulty Intermediate","color":"fbca04"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Effort%20Low","name":"Effort Low","color":"006b75"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Performance","name":"Performance","color":"a10c02"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Prio-medium","name":"Prio-medium","color":"FF8C00"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Regression","name":"Regression","color":"e11d21"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Reshaping","name":"Reshaping","color":"02d7e1"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/pydata/pandas/milestones/32","html_url":"https://github.com/pydata/pandas/milestones/Next%20Major%20Release","labels_url":"https://api.github.com/repos/pydata/pandas/milestones/32/labels","id":933188,"number":32,"title":"Next Major Release","description":"after 0.16.1 of course","creator":{"login":"jreback","id":953992,"avatar_url":"https://avatars.githubusercontent.com/u/953992?v=3","gravatar_id":"","url":"https://api.github.com/users/jreback","html_url":"https://github.com/jreback","followers_url":"https://api.github.com/users/jreback/followers","following_url":"https://api.github.com/users/jreback/following{/other_user}","gists_url":"https://api.github.com/users/jreback/gists{/gist_id}","starred_url":"https://api.github.com/users/jreback/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jreback/subscriptions","organizations_url":"https://api.github.com/users/jreback/orgs","repos_url":"https://api.github.com/users/jreback/repos","events_url":"https://api.github.com/users/jreback/events{/privacy}","received_events_url":"https://api.github.com/users/jreback/received_events","type":"User","site_admin":false},"open_issues":911,"closed_issues":73,"state":"open","created_at":"2015-01-13T10:53:19Z","updated_at":"2015-09-22T11:28:02Z","due_on":"2016-01-31T05:00:00Z","closed_at":null},"comments":13,"created_at":"2015-05-17T14:03:31Z","updated_at":"2015-09-23T20:20:35Z","closed_at":null,"body":"the following works quickly in 0.15.2 and has a performance issue on the last operation df.T.duplicated() in 0.16.0 and 0.16.1\r\nalso on a private data set that works on 0.15.2 i get an error on 0.16.0 and 0.16.1 on the same operation.\r\n\r\ncode:\r\n\r\n```python\r\nimport pandas,numpy\r\n\r\ndf = pandas.DataFrame({'A': [1 for x in range(1000)],\r\n                   'B': [1 for x in range(1000)]})\r\n\r\nprint (numpy.count_nonzero(df.duplicated()))\r\nprint (numpy.count_nonzero(df.T.duplicated()))\r\n\r\ndf = pandas.DataFrame({'A': [1 for x in range(1000000)],\r\n                   'B': [1 for x in range(1000000)]})\r\n\r\nprint (numpy.count_nonzero(df.duplicated()))\r\nprint (numpy.count_nonzero(df.T.duplicated()))\r\n\r\n\r\nthis is the error i get on the private data set (code not reproduce yet with synthetic data):\r\n  File \"C:\\Anaconda3\\lib\\site-packages\\pandas\\util\\decorators.py\", line 88, in wrapper\r\n    return func(*args, **kwargs)\r\n  File \"C:\\Anaconda3\\lib\\site-packages\\pandas\\core\\frame.py\", line 2867, in duplicated\r\n    labels, shape = map(list, zip( * map(f, vals)))\r\n  File \"C:\\Anaconda3\\lib\\site-packages\\pandas\\core\\frame.py\", line 2856, in f\r\n    labels, shape = factorize(vals, size_hint=min(len(self), _SIZE_HINT_LIMIT))\r\n  File \"C:\\Anaconda3\\lib\\site-packages\\pandas\\core\\algorithms.py\", line 135, in factorize\r\n    labels = table.get_labels(vals, uniques, 0, na_sentinel)\r\n  File \"pandas\\hashtable.pyx\", line 813, in pandas.hashtable.PyObjectHashTable.get_labels (pandas\\hashtable.c:14025)\r\nValueError: Buffer has wrong number of dimensions (expected 1, got 2)\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/pydata/pandas/issues/comments/142717207","html_url":"https://github.com/pydata/pandas/issues/10161#issuecomment-142717207","issue_url":"https://api.github.com/repos/pydata/pandas/issues/10161","id":142717207,"user":{"login":"behzadnouri","id":1288998,"avatar_url":"https://avatars.githubusercontent.com/u/1288998?v=3","gravatar_id":"","url":"https://api.github.com/users/behzadnouri","html_url":"https://github.com/behzadnouri","followers_url":"https://api.github.com/users/behzadnouri/followers","following_url":"https://api.github.com/users/behzadnouri/following{/other_user}","gists_url":"https://api.github.com/users/behzadnouri/gists{/gist_id}","starred_url":"https://api.github.com/users/behzadnouri/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/behzadnouri/subscriptions","organizations_url":"https://api.github.com/users/behzadnouri/orgs","repos_url":"https://api.github.com/users/behzadnouri/repos","events_url":"https://api.github.com/users/behzadnouri/events{/privacy}","received_events_url":"https://api.github.com/users/behzadnouri/received_events","type":"User","site_admin":false},"created_at":"2015-09-23T20:20:35Z","updated_at":"2015-09-23T20:20:35Z","body":"i will patch this later today"}}