{"action":"created","issue":{"url":"https://api.github.com/repos/pydata/pandas/issues/9149","labels_url":"https://api.github.com/repos/pydata/pandas/issues/9149/labels{/name}","comments_url":"https://api.github.com/repos/pydata/pandas/issues/9149/comments","events_url":"https://api.github.com/repos/pydata/pandas/issues/9149/events","html_url":"https://github.com/pydata/pandas/issues/9149","id":52834154,"number":9149,"title":"aggregating with a dictionary that specifies a column that has all nan's fails to use numpy","user":{"login":"flyingchipmunk","id":807921,"avatar_url":"https://avatars.githubusercontent.com/u/807921?v=3","gravatar_id":"","url":"https://api.github.com/users/flyingchipmunk","html_url":"https://github.com/flyingchipmunk","followers_url":"https://api.github.com/users/flyingchipmunk/followers","following_url":"https://api.github.com/users/flyingchipmunk/following{/other_user}","gists_url":"https://api.github.com/users/flyingchipmunk/gists{/gist_id}","starred_url":"https://api.github.com/users/flyingchipmunk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/flyingchipmunk/subscriptions","organizations_url":"https://api.github.com/users/flyingchipmunk/orgs","repos_url":"https://api.github.com/users/flyingchipmunk/repos","events_url":"https://api.github.com/users/flyingchipmunk/events{/privacy}","received_events_url":"https://api.github.com/users/flyingchipmunk/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2014-12-24T19:41:44Z","updated_at":"2015-01-02T17:13:29Z","closed_at":null,"body":"As the subject says, if I try to call .agg() with a dictionary with a column that has all np.nan's it falls back to python agg functions instead of numpy.\r\n\r\nTo reproduce: (my dataset is 60 cols, 100000 rows)\r\n\r\nI imported a csv and one column was all null (np.nan). The column dtype was set to object. (that's one issue, why the large upcast container to store np.nan?)\r\n```\r\nsq = pd.read_table(sqFile, sep='\\t', skiprows = 1, nrows=None, header=0)\r\nsq_g=sq.groupby(all_key_cols, as_index=False, sort=False)\r\n```\r\n`sq_g.agg(sum)`\r\nWithout specifying a dictionary and using sum over the entire dataframe it correctly uses the cython optimized numpy.sum:\r\n`10 loops, best of 3: 48.3 ms per loop`\r\n\r\n`sq_g.agg(collections.OrderedDict({'ColumnRef53':'sum'}))`\r\nSpecifying a dictionary and a column that is of dtype object that is entirely rows of np.nan falls back to python (bad):\r\n`1 loops, best of 3: 7.26 s per loop`\r\n\r\nFor reference (ColumnRef20 has floats, ColumnRef53 has entirely np.nan's):\r\n```\r\nsq.dtypes\r\n# Row               float64\r\nRowdesc.iphost       object\r\n...\r\nColumnRef20         float64\r\nColumnRef53          object\r\n...\r\ndtype: object\r\n```\r\nMy workaround is to downcast these np.nan filled columns back to float64, then the dictionary aggregation correctly uses the numpy optimized functions and not python:\r\n\r\n```\r\n# workaround for numpy groupby issue:\r\n#  downcast columns with all NaN from object to float64 so agg() doesn't fallback to python.\r\n\r\n# first find all columns with all np.nan rows\r\ndata_cols = [x for x in sq_concat.columns.tolist() if x.startswith('Column')]\r\nall_nan = pd.isnull(sq_concat[data_cols]).all()\r\nall_nan_cols = all_nan[all_nan == True].index.values.tolist()\r\n\r\n# only need to downcast if type is object\r\nobj_downcast = sq_concat[all_nan_cols].dtypes == object\r\nobj_downcast_cols = obj_downcast[obj_downcast == True].index.values.tolist()\r\n\r\n# downcast object to np.float64\r\nfor nan_col in obj_downcast_cols:\r\n    sq_concat[nan_col] = sq_concat[nan_col].apply(np.float64)\r\n```\r\n\r\nThen the dictionary .agg() works as expected:\r\n`sq_g.agg(collections.OrderedDict({'ColumnRef53':'sum'}))`\r\n`100 loops, best of 3: 6.2 ms per loop`\r\n"},"comment":{"url":"https://api.github.com/repos/pydata/pandas/issues/comments/68543164","html_url":"https://github.com/pydata/pandas/issues/9149#issuecomment-68543164","issue_url":"https://api.github.com/repos/pydata/pandas/issues/9149","id":68543164,"user":{"login":"jreback","id":953992,"avatar_url":"https://avatars.githubusercontent.com/u/953992?v=3","gravatar_id":"","url":"https://api.github.com/users/jreback","html_url":"https://github.com/jreback","followers_url":"https://api.github.com/users/jreback/followers","following_url":"https://api.github.com/users/jreback/following{/other_user}","gists_url":"https://api.github.com/users/jreback/gists{/gist_id}","starred_url":"https://api.github.com/users/jreback/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jreback/subscriptions","organizations_url":"https://api.github.com/users/jreback/orgs","repos_url":"https://api.github.com/users/jreback/repos","events_url":"https://api.github.com/users/jreback/events{/privacy}","received_events_url":"https://api.github.com/users/jreback/received_events","type":"User","site_admin":false},"created_at":"2015-01-02T17:13:29Z","updated_at":"2015-01-02T17:13:29Z","body":"can you show a complete reproducicle example that I can copy/paste?"}}