{"action":"created","issue":{"url":"https://api.github.com/repos/ioam/holoviews/issues/284","labels_url":"https://api.github.com/repos/ioam/holoviews/issues/284/labels{/name}","comments_url":"https://api.github.com/repos/ioam/holoviews/issues/284/comments","events_url":"https://api.github.com/repos/ioam/holoviews/issues/284/events","html_url":"https://github.com/ioam/holoviews/pull/284","id":112022372,"number":284,"title":"Data API","user":{"login":"philippjfr","id":1550771,"avatar_url":"https://avatars.githubusercontent.com/u/1550771?v=3","gravatar_id":"","url":"https://api.github.com/users/philippjfr","html_url":"https://github.com/philippjfr","followers_url":"https://api.github.com/users/philippjfr/followers","following_url":"https://api.github.com/users/philippjfr/following{/other_user}","gists_url":"https://api.github.com/users/philippjfr/gists{/gist_id}","starred_url":"https://api.github.com/users/philippjfr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/philippjfr/subscriptions","organizations_url":"https://api.github.com/users/philippjfr/orgs","repos_url":"https://api.github.com/users/philippjfr/repos","events_url":"https://api.github.com/users/philippjfr/events{/privacy}","received_events_url":"https://api.github.com/users/philippjfr/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"philippjfr","id":1550771,"avatar_url":"https://avatars.githubusercontent.com/u/1550771?v=3","gravatar_id":"","url":"https://api.github.com/users/philippjfr","html_url":"https://github.com/philippjfr","followers_url":"https://api.github.com/users/philippjfr/followers","following_url":"https://api.github.com/users/philippjfr/following{/other_user}","gists_url":"https://api.github.com/users/philippjfr/gists{/gist_id}","starred_url":"https://api.github.com/users/philippjfr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/philippjfr/subscriptions","organizations_url":"https://api.github.com/users/philippjfr/orgs","repos_url":"https://api.github.com/users/philippjfr/repos","events_url":"https://api.github.com/users/philippjfr/events{/privacy}","received_events_url":"https://api.github.com/users/philippjfr/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/ioam/holoviews/milestones/7","html_url":"https://github.com/ioam/holoviews/milestones/v1.4.0","labels_url":"https://api.github.com/repos/ioam/holoviews/milestones/7/labels","id":1300858,"number":7,"title":"v1.4.0","description":"The last release is now over two months old and we've added a wide range of features including merging of the bokeh backend. There's no fixed date set for this release yet but we'll start assigning issues that we think can be fixed in the near future.\r\n\r\nAdditionally both matplotlib v1.5 and bokeh 0.10 will be released in the near future, this release should  ensure compatibility with both.","creator":{"login":"philippjfr","id":1550771,"avatar_url":"https://avatars.githubusercontent.com/u/1550771?v=3","gravatar_id":"","url":"https://api.github.com/users/philippjfr","html_url":"https://github.com/philippjfr","followers_url":"https://api.github.com/users/philippjfr/followers","following_url":"https://api.github.com/users/philippjfr/following{/other_user}","gists_url":"https://api.github.com/users/philippjfr/gists{/gist_id}","starred_url":"https://api.github.com/users/philippjfr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/philippjfr/subscriptions","organizations_url":"https://api.github.com/users/philippjfr/orgs","repos_url":"https://api.github.com/users/philippjfr/repos","events_url":"https://api.github.com/users/philippjfr/events{/privacy}","received_events_url":"https://api.github.com/users/philippjfr/received_events","type":"User","site_admin":false},"open_issues":20,"closed_issues":28,"state":"open","created_at":"2015-09-11T11:12:20Z","updated_at":"2015-10-29T14:59:51Z","due_on":null,"closed_at":null},"comments":6,"created_at":"2015-10-18T14:13:58Z","updated_at":"2015-10-29T17:57:25Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/ioam/holoviews/pulls/284","html_url":"https://github.com/ioam/holoviews/pull/284","diff_url":"https://github.com/ioam/holoviews/pull/284.diff","patch_url":"https://github.com/ioam/holoviews/pull/284.patch"},"body":"While not yet ready to be merged this PR will implement the proposals laid out in #269. I'll reproduce the main points of discussion here and outline a to-do list.\r\n\r\nThe core of the proposal is to add a core ``Columns`` Element class, which subsumes the functionality of the ``Chart``, ``DFrame`` and ``NdElement`` classes. This will allow a wide range of Element classes to interchangeably use one of three data formats:\r\n\r\n* An array based format of NxD numpy array, where N is the number of samples and D the number of dimensions/columns.\r\n\r\n* A dataframe based format with N samples and D columns, which should work with pandas, dask and blaze dataframes.\r\n\r\n* A pure-Python NdElement based data format storing the data as keys and values of an OrderedDict.\r\n\r\nThis would mean the Curve, Points, VectorField, Bars, Table, ErrorBars, Spread, Scatter, Scatter3D, Trisurface, Distribution, TimeSeries and Regression Elements would immediately support all three data formats.\r\n\r\nThe data format specific implementations of the API are implemented on utility classes, this will include:\r\n\r\n* dimension_values: Returns an array, list or pandas series of one column\r\n* range: Returns the minimum and maximum values along a dimension excluding NaNs.\r\n* select: Same as ``__getitem__`` but works for one or multiple dimensions.\r\n* groupby: Groups by the values along one or multiple dimensions returning an NdMapping type indexed by the grouped dimensions with Elements of the same type as values. (not implemented for Charts)\r\n* dframe: Converts data to a dataframe\r\n* collapse_data: Applies a function across a list of Element.data attributes\r\n* reduce: Applies a reduce function across one or multiple axes\r\n* sample: Returns a Table with of the samples specified as lists\r\n* reindex: Reorder specified dimensions dropping unspecified dimensions. (not implemented for Charts)\r\n* drop_dimension: Drops specified dimension(s)/column(s)\r\n* add_dimension: Adds named column at specified position with supplied value(s)\r\n* closest: Returns the closest samples given a list of coordinates.\r\n* shape: NxD shape of data\r\n\r\nand conversion methods, which can take an optional flag to return a clone of the Element with converted data:\r\n\r\n* array: Returns data as an array.\r\n* dframe: Returns data as a DataFrame.\r\n* mapping: Returns data as NdElement.\r\n\r\nBackwards incompatible changes:\r\n\r\n* Various operations no longer return ``ItemTables``, the variable return type is confusing and makes the implementation very messy. All operations (select, reduce, sample) return one of two types of data either a modified Columns object or a single scalar.\r\n\r\nOther features:\r\n\r\n* Currently Elements only support slicing of key dimensions, this is a common annoyance mentioned by several people. I propose we extend the slicing behavior of our types to the following with the following proposal. The ``__getitem__`` method will now check whether the first or n-th index (where n is the number of key dimensions) matches an existing dimension by name, allowing the user to select particular columns. Passing a dimension as a first index will return the values along that dimension, this matches the behavior of DataFrames and will allow us to provide an easy interface for more complex Boolean slicing, e.g.: ``columns[np.logical_or(columns['x'] > 5, columns['y'] > 10)]``. Passing a dimension as the n-th index preserves the current behavior when slicing tables, allowing you to select a particular value dimension. If the non-key indexes don't match any existing dimension they are used to slice along the vdims.\r\n\r\n* Collapse/reduce work via the same mechanism, they are basically a groupby operation over one or more key dimensions of an Element or list of Elements, then applying an aggregation function.\r\n\r\nOne of the major sticking points is how the constructor should behave for different data formats, currently this is the implemented behavior showing the mapping between an input type and the storage format that's used (some come with fallbacks if a certain condition isn't met):\r\n\r\n* ndarray -> ndarray\r\n* DataFrame -> DataFrame\r\n* NdElement -> NdElement\r\n* dict -> NdElement\r\n* list of scalar values -> ndarray\r\n* list of tuples for each row -> ndarray (if numeric) -> DataFrame (if available) -> NdElement\r\n* tuple of arrays/lists -> ndarray (if numeric) -> DataFrame (if available) -> NdElement \r\n* list of key value tuples (i.e. tuple of tuples) -> NdElement\r\n\r\nThe current to-do list is as follows:\r\n\r\n* [x] Get nosetests passing\r\n* [x] Test whether all affected Element types can be constructed with all data types.\r\n* [x] Test whether all affected Element types display using both plotting backends.\r\n* [x] Unify TableConversion interface with DFrame conversions \r\n* [ ] Achieve feature-parity between backends:\r\n   * [x] Constructors\r\n   * [x] getitem/select\r\n   * [x] dimension_values\r\n   * [x] range\r\n   * [x] shape\r\n   * [x] sample\r\n   * [x] reindex\r\n   * [x] add_dimension\r\n   * [x] closest\r\n   * [x] dframe\r\n   * [x] collapse\r\n   * [x] reduce\r\n   * [x] array\r\n   * [x] aggregate\r\n   * [ ] mapping \r\n   * [ ] drop_dimension\r\n   * [ ] Add index dimension to NdElement storage format to allow duplicate entries (numpy and pandas support this by behavior already).\r\n* [ ] Implement data-type agnostic comparisons for Columns.\r\n* [ ] Get notebook tests passing\r\n* [ ] Decide on good naming for the new module and classes\r\n* [ ] Refactor utility classes using classmethods rather than using instances (easier for now)\r\n* [ ] Write comprehensive set of unit tests providing 100% coverage of Columns module\r\n* [ ] Document main Columns class and utility classes.\r\n\r\nThat covers it for now. I won't have any time to work on this in a concentrated way so I'll be slowly ticking off the to-do list as I find time. In the mean-time we can think about the naming."},"comment":{"url":"https://api.github.com/repos/ioam/holoviews/issues/comments/152266185","html_url":"https://github.com/ioam/holoviews/pull/284#issuecomment-152266185","issue_url":"https://api.github.com/repos/ioam/holoviews/issues/284","id":152266185,"user":{"login":"jlstevens","id":890576,"avatar_url":"https://avatars.githubusercontent.com/u/890576?v=3","gravatar_id":"","url":"https://api.github.com/users/jlstevens","html_url":"https://github.com/jlstevens","followers_url":"https://api.github.com/users/jlstevens/followers","following_url":"https://api.github.com/users/jlstevens/following{/other_user}","gists_url":"https://api.github.com/users/jlstevens/gists{/gist_id}","starred_url":"https://api.github.com/users/jlstevens/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jlstevens/subscriptions","organizations_url":"https://api.github.com/users/jlstevens/orgs","repos_url":"https://api.github.com/users/jlstevens/repos","events_url":"https://api.github.com/users/jlstevens/events{/privacy}","received_events_url":"https://api.github.com/users/jlstevens/received_events","type":"User","site_admin":false},"created_at":"2015-10-29T17:57:25Z","updated_at":"2015-10-29T17:57:25Z","body":"I've just had a discussion with Philipp and here are the notes I've made:\r\n\r\n* I'm happy if ``Table`` no longer supports ``ItemTable`` as a return type.\r\n* We should preserve the input data type but if it is one of our fancy formats (e.g a dictionary or some other pure Python literal) we should have a flag to determine whether ``.data`` should become an ``NdElement`` or a pandas ``DataFrame`` (if available) by default. I say we don't use pandas unless the input is pandas.\r\n* We need to update the documentation about key and value dimensions because it is no longer about indexing. I think we said that key dimensions are sample positions or categorical values whereas the value dimensions are the values recorded at those points. The point is that the new proposal *will* support indexing and slicing of value dimensions.\r\n* ``NdElement`` needs a first ``'Index'`` dimension handled by the interface class (that needs to be written to support it!) in order to support multiple entries with duplicate keys. This index will be managed by the interface class.\r\n* ``Columns``  will be unsorted by default but will have a ``.sort()`` method for lexical sort.\r\n* For compatibility with pandas and numpy, we are considering another indexing mode: indexing with a single string would return the values of the associated dimension. In effect this is an alias to ``dimension_values`` that is familiar to pandas/numpy users. By default this would probably be a numpy array but I would consider a flag to change the return type (e.g list, xray-array, dataframe, another ``Columns`` instance). NOTE: In future we should be able to return another ``Column`` instance (the sensible thing!) because numpy will behave in a more sensible way (http://docs.scipy.org/doc/numpy-dev/release.html):\r\n\r\n> * In the future Boolean array-likes (such as lists of python bools) will always be treated as Boolean \r\n> indexes and Boolean scalars (including python True) will be a legal boolean index\r\n"}}