{"action":"opened","issue":{"url":"https://api.github.com/repos/pydata/pandas/issues/11668","labels_url":"https://api.github.com/repos/pydata/pandas/issues/11668/labels{/name}","comments_url":"https://api.github.com/repos/pydata/pandas/issues/11668/comments","events_url":"https://api.github.com/repos/pydata/pandas/issues/11668/events","html_url":"https://github.com/pydata/pandas/issues/11668","id":118095000,"number":11668,"title":"DataFrame .duplicated() / .drop_duplicates() flagging unique rows as duplicated in 0.17.0","user":{"login":"bijanhoule","id":7012524,"avatar_url":"https://avatars.githubusercontent.com/u/7012524?v=3","gravatar_id":"","url":"https://api.github.com/users/bijanhoule","html_url":"https://github.com/bijanhoule","followers_url":"https://api.github.com/users/bijanhoule/followers","following_url":"https://api.github.com/users/bijanhoule/following{/other_user}","gists_url":"https://api.github.com/users/bijanhoule/gists{/gist_id}","starred_url":"https://api.github.com/users/bijanhoule/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bijanhoule/subscriptions","organizations_url":"https://api.github.com/users/bijanhoule/orgs","repos_url":"https://api.github.com/users/bijanhoule/repos","events_url":"https://api.github.com/users/bijanhoule/events{/privacy}","received_events_url":"https://api.github.com/users/bijanhoule/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-11-20T18:11:39Z","updated_at":"2015-11-20T18:11:39Z","closed_at":null,"body":"Dataframe.duplicated() and .drop_duplicates() are flagging rows as duplicates when they are in fact distinct.  \r\n\r\nThis was the smallest dataset I could make to recreate the issue, but I've seen this issue on DataFrames of any size:\r\n\r\n```python\r\n>>> import pandas as pd\r\n\r\n>>> json_data = '{\"Col1\":{\"0\":\"S2#OaGwWII\",\"1\":\")A9$rw3W_I\",\"2\":\"2Ra+*_RWII\",\"3\":\"2RA`4kRWII\",\"4\":\"2R=K*_RWII\"},' \\\r\n            '\"Col2\":{\"0\":141105144406,\"1\":141107294517,\"2\":141106133624,\"3\":141108219194,\"4\":141106133614}}'\r\n>>> df = pd.read_json(json_data)\r\n>>> print(df)\r\n\r\n         Col1          Col2\r\n0  S2#OaGwWII  141105144406\r\n1  )A9$rw3W_I  141107294517\r\n2  2Ra+*_RWII  141106133624\r\n3  2RA`4kRWII  141108219194\r\n4  2R=K*_RWII  141106133614\r\n\r\n>>> df.duplicated(keep=False)\r\n\r\n0    False\r\n1    False\r\n2     True\r\n3    False\r\n4     True\r\ndtype: bool\r\n\r\n```\r\n\r\nIt also seems to depend on row order / column order; this behavior can be changed by shuffling / sampling rows or columns, e.g.:\r\n\r\n```python\r\n>>> df[['Col2', 'Col1']].duplicated(keep=False)\r\n0    False\r\n1    False\r\n2    False\r\n3    False\r\n4    False\r\ndtype: bool\r\n```\r\n\r\nI only see this behavior on 0.17.0, while 0.16.2 is fine.  More details about each environment are below:\r\n\r\n#### pandas 0.17.0 / python 3.4.3 (failing)\r\n\r\n```\r\n>>> pd.util.print_versions.show_versions()\r\n\r\nINSTALLED VERSIONS\r\n------------------\r\ncommit: None\r\npython: 3.4.3.final.0\r\npython-bits: 64\r\nOS: Linux\r\nOS-release: 2.6.18-400.1.1.el5\r\nmachine: x86_64\r\nprocessor: x86_64\r\nbyteorder: little\r\nLC_ALL: None\r\nLANG: en_US\r\n\r\npandas: 0.17.0\r\nnose: 1.3.4\r\npip: 7.1.2\r\nsetuptools: 18.4\r\nCython: 0.23.4\r\nnumpy: 1.10.1\r\nscipy: 0.16.0\r\nstatsmodels: 0.6.1\r\nIPython: 3.2.1\r\nsphinx: 1.3.1\r\npatsy: 0.4.0\r\ndateutil: 2.4.2\r\npytz: 2015.7\r\nblosc: None\r\nbottleneck: None\r\ntables: 3.2.2\r\nnumexpr: 2.4.4\r\nmatplotlib: 1.4.3\r\nopenpyxl: 1.8.5\r\nxlrd: 0.9.4\r\nxlwt: None\r\nxlsxwriter: 0.7.3\r\nlxml: 3.4.4\r\nbs4: 4.4.0\r\nhtml5lib: None\r\nhttplib2: None\r\napiclient: None\r\nsqlalchemy: 1.0.9\r\npymysql: 0.6.6.None\r\npsycopg2: 2.6 (dt dec pq3 ext)\r\n```\r\n\r\n\r\n#### pandas 0.17.0 / python 3.5 (failing)\r\n\r\n```\r\n>>> pd.util.print_versions.show_versions()\r\n\r\nINSTALLED VERSIONS\r\n------------------\r\ncommit: None\r\npython: 3.5.0.final.0\r\npython-bits: 64\r\nOS: Linux\r\nOS-release: 2.6.18-348.18.1.el5\r\nmachine: x86_64\r\nprocessor: x86_64\r\nbyteorder: little\r\nLC_ALL: None\r\nLANG: C\r\n\r\npandas: 0.17.0\r\nnose: 1.3.7\r\npip: 7.1.2\r\nsetuptools: 18.4\r\nCython: 0.23.4\r\nnumpy: 1.10.1\r\nscipy: 0.16.0\r\nstatsmodels: 0.6.1\r\nIPython: 4.0.0\r\nsphinx: 1.3.1\r\npatsy: 0.4.0\r\ndateutil: 2.4.2\r\npytz: 2015.7\r\nblosc: None\r\nbottleneck: None\r\ntables: 3.2.2\r\nnumexpr: 2.4.4\r\nmatplotlib: 1.5.0\r\nopenpyxl: 1.8.5\r\nxlrd: 0.9.4\r\nxlwt: None\r\nxlsxwriter: 0.7.7\r\nlxml: 3.4.4\r\nbs4: 4.4.1\r\nhtml5lib: None\r\nhttplib2: None\r\napiclient: None\r\nsqlalchemy: 1.0.9\r\npymysql: 0.6.7.None\r\npsycopg2: None\r\n```\r\n\r\n#### pandas 0.16.2 python 3.5 (passing)\r\n\r\n```\r\n>>> pd.util.print_versions.show_versions()\r\n\r\nINSTALLED VERSIONS\r\n------------------\r\ncommit: None\r\npython: 3.5.0.final.0\r\npython-bits: 64\r\nOS: Linux\r\nOS-release: 2.6.18-348.18.1.el5\r\nmachine: x86_64\r\nprocessor: x86_64\r\nbyteorder: little\r\nLC_ALL: None\r\nLANG: C\r\n\r\npandas: 0.16.2\r\nnose: None\r\nCython: None\r\nnumpy: 1.10.1\r\nscipy: None\r\nstatsmodels: None\r\nIPython: None\r\nsphinx: None\r\npatsy: None\r\ndateutil: 2.4.2\r\npytz: 2015.7\r\nbottleneck: None\r\ntables: None\r\nnumexpr: None\r\nmatplotlib: None\r\nopenpyxl: None\r\nxlrd: None\r\nxlwt: None\r\nxlsxwriter: None\r\nlxml: None\r\nbs4: None\r\nhtml5lib: None\r\nhttplib2: None\r\napiclient: None\r\nsqlalchemy: None\r\npymysql: None\r\npsycopg2: None\r\n```"}}