{"action":"created","issue":{"url":"https://api.github.com/repos/pydata/pandas/issues/10511","labels_url":"https://api.github.com/repos/pydata/pandas/issues/10511/labels{/name}","comments_url":"https://api.github.com/repos/pydata/pandas/issues/10511/comments","events_url":"https://api.github.com/repos/pydata/pandas/issues/10511/events","html_url":"https://github.com/pydata/pandas/issues/10511","id":93036580,"number":10511,"title":"New DataFrame feature: listify() and unlistify()","user":{"login":"dov","id":338235,"avatar_url":"https://avatars.githubusercontent.com/u/338235?v=3","gravatar_id":"","url":"https://api.github.com/users/dov","html_url":"https://github.com/dov","followers_url":"https://api.github.com/users/dov/followers","following_url":"https://api.github.com/users/dov/following{/other_user}","gists_url":"https://api.github.com/users/dov/gists{/gist_id}","starred_url":"https://api.github.com/users/dov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dov/subscriptions","organizations_url":"https://api.github.com/users/dov/orgs","repos_url":"https://api.github.com/users/dov/repos","events_url":"https://api.github.com/users/dov/events{/privacy}","received_events_url":"https://api.github.com/users/dov/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-07-04T18:33:13Z","updated_at":"2015-07-05T06:54:52Z","closed_at":null,"body":"While building up a DataFrame in several steps, I found it difficult to add a new \"perpendicular\" column, i.e. a column that adds another dimension to already existing columns. To solve this problem I got the idea that this may be done in two steps:\r\n\r\n1. Add a new column whose values at each cell is a python list of the values the new column takes on.\r\n2. Unlistify the column thereby creating a new row for each element in the above lists.\r\n\r\nI.e. I propose two new DataFrame methods, listify() and unlistify().\r\n\r\n`listify(df, column)`: Takes as input a dataframe and the name of a column. It will do a groupby of the df for all columns except column and generate a single row where the values in the column cell is a list of the column `column` values.\r\n\r\n`unlistify(df, column)`: Takes as input a dataframe and the name of a column. It will iterate over the values of the contents of `column` for each row and generate a new row for each value.\r\n\r\nThe functions may be expanded to support multiple columns. `listify()` may e.g. support a post processing function, that will be applied on the list.\r\n\r\nThe following python code illustrates these two functions. But obviously the functionality may be implemented more efficienctly on the C-level.\r\n\r\n```python\r\n#!/usr/bin/python\r\n\r\nimport pandas as pd\r\n\r\ndef listify(df, column):\r\n  matches = [i for i,n in enumerate(df.columns)\r\n             if n==column]\r\n\r\n  if len(matches)==0:\r\n    raise Exception('Failed to find column named ' + column +'!')\r\n  if len(matches)>1:\r\n    raise Exception('More than one column named ' + column +'!')\r\n\r\n  old_index = df.index\r\n  col_idx = matches[0] + len(old_index.shape) # Since we will reset the index\r\n\r\n  column_names = list(df.index.names) + list(df.columns)\r\n  gb_cols = [c for c in column_names\r\n             if c!= column]\r\n\r\n  # Helper function to generate the squashed dataframe\r\n  def fnc(d):\r\n    row = list(d.values[0])\r\n    return pd.DataFrame([row[:col_idx]\r\n                         + [[v[col_idx] for v in list(d.values)]]\r\n                         + row[col_idx+1:]])\r\n\r\n  return (df\r\n          .reset_index()\r\n          .groupby(gb_cols)\r\n          .apply(fnc)\r\n          .rename(columns = lambda i : column_names[i])\r\n          .set_index(old_index.names)\r\n          )\r\n\r\ndef unlistify(df, column):\r\n  matches = [i for i,n in enumerate(df.columns)\r\n             if n==column]\r\n\r\n  if len(matches)==0:\r\n    raise Exception('Failed to find column named ' + column +'!')\r\n  if len(matches)>1:\r\n    raise Exception('More than one column named ' + column +'!')\r\n\r\n  col_idx = matches[0]\r\n\r\n  # Helper function to expand and repeat the column col_idx\r\n  def fnc(d):\r\n    row = list(d.values[0])\r\n    bef = row[:col_idx]\r\n    aft = row[col_idx+1:]\r\n    col = row[col_idx]\r\n    z = [bef + [c] + aft for c in col]\r\n    return pd.DataFrame(z)\r\n\r\n  col_idx += len(df.index.shape) # Since we will push reset the index\r\n  index_names = list(df.index.names)\r\n  column_names = list(index_names) + list(df.columns)\r\n  return (df\r\n          .reset_index()\r\n          .groupby(level=0,as_index=0)\r\n          .apply(fnc)\r\n          .rename(columns = lambda i :column_names[i])\r\n          .set_index(index_names)\r\n          )\r\n\r\n\r\n# Examples of how to listify and unlistify a column.\r\n\r\ndf = pd.DataFrame([[11,range(5),10],\r\n                   [22,range(3),20]],\r\n                   columns = ['A','B','C']).set_index('C')\r\nprint 'org'\r\nprint df\r\nprint '--'\r\ndf = unlistify(df,'B')\r\nprint 'unlistify(df,B)'\r\nprint df\r\nprint '--'\r\ndf = listify(df,'B')\r\nprint 'listify(df,B)'\r\nprint df\r\n```\r\nThe corresponding output:\r\n\r\n    org\r\n         A                B\r\n    C                      \r\n    10  11  [0, 1, 2, 3, 4]\r\n    20  22        [0, 1, 2]\r\n    --\r\n    unlistify(df,B)\r\n         A  B\r\n    C        \r\n    10  11  0\r\n    10  11  1\r\n    10  11  2\r\n    10  11  3\r\n    10  11  4\r\n    20  22  0\r\n    20  22  1\r\n    20  22  2\r\n    --\r\n    listify(df,B)\r\n         A                B\r\n    C                      \r\n    10  11  [0, 1, 2, 3, 4]\r\n    20  22        [0, 1, 2]\r\n"},"comment":{"url":"https://api.github.com/repos/pydata/pandas/issues/comments/118588498","html_url":"https://github.com/pydata/pandas/issues/10511#issuecomment-118588498","issue_url":"https://api.github.com/repos/pydata/pandas/issues/10511","id":118588498,"user":{"login":"shoyer","id":1217238,"avatar_url":"https://avatars.githubusercontent.com/u/1217238?v=3","gravatar_id":"","url":"https://api.github.com/users/shoyer","html_url":"https://github.com/shoyer","followers_url":"https://api.github.com/users/shoyer/followers","following_url":"https://api.github.com/users/shoyer/following{/other_user}","gists_url":"https://api.github.com/users/shoyer/gists{/gist_id}","starred_url":"https://api.github.com/users/shoyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shoyer/subscriptions","organizations_url":"https://api.github.com/users/shoyer/orgs","repos_url":"https://api.github.com/users/shoyer/repos","events_url":"https://api.github.com/users/shoyer/events{/privacy}","received_events_url":"https://api.github.com/users/shoyer/received_events","type":"User","site_admin":false},"created_at":"2015-07-05T06:54:51Z","updated_at":"2015-07-05T06:54:51Z","body":"Maybe you could show an example of how you make the dataframe? Presumably you're not reading it in from CSV. I guess my thought is that you might be able to easily create the \"unlisted\" dataframe in the first place.\r\n\r\nPipe is indeed a side point here. Mostly I mentioned it to point out that we are trying to put other libraries on equal footings to what we put in pandas proper."}}