{"action":"created","issue":{"url":"https://api.github.com/repos/xray/xray/issues/643","labels_url":"https://api.github.com/repos/xray/xray/issues/643/labels{/name}","comments_url":"https://api.github.com/repos/xray/xray/issues/643/comments","events_url":"https://api.github.com/repos/xray/xray/issues/643/events","html_url":"https://github.com/xray/xray/issues/643","id":114732169,"number":643,"title":"\"naive\" iteration is very slow","user":{"login":"anntzer","id":1322974,"avatar_url":"https://avatars.githubusercontent.com/u/1322974?v=3","gravatar_id":"","url":"https://api.github.com/users/anntzer","html_url":"https://github.com/anntzer","followers_url":"https://api.github.com/users/anntzer/followers","following_url":"https://api.github.com/users/anntzer/following{/other_user}","gists_url":"https://api.github.com/users/anntzer/gists{/gist_id}","starred_url":"https://api.github.com/users/anntzer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anntzer/subscriptions","organizations_url":"https://api.github.com/users/anntzer/orgs","repos_url":"https://api.github.com/users/anntzer/repos","events_url":"https://api.github.com/users/anntzer/events{/privacy}","received_events_url":"https://api.github.com/users/anntzer/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-11-03T02:53:04Z","updated_at":"2015-11-03T06:39:39Z","closed_at":null,"body":"```\r\n$ ipython\r\nPython 3.5.0 (default, Sep 20 2015, 11:28:25) \r\nType \"copyright\", \"credits\" or \"license\" for more information.\r\n\r\nIPython 4.0.0 -- An enhanced Interactive Python.\r\n?         -> Introduction and overview of IPython's features.\r\n%quickref -> Quick reference.\r\nhelp      -> Python's own help system.\r\nobject?   -> Details about 'object', use 'object??' for extra details.\r\nUsing matplotlib backend: Qt4Agg\r\n\r\nIn [1]: from xray import DataArray\r\n\r\n# Iteration over a Python list\r\nIn [2]: %%timeit t = list(range(10000))\r\nfor _ in t: pass\r\n   ...: \r\n10000 loops, best of 3: 87.3 µs per loop\r\n\r\n# Iteration over a ndarray\r\nIn [3]: %%timeit t = np.arange(10000)\r\nfor _ in t: pass\r\n   ...: \r\n1000 loops, best of 3: 472 µs per loop\r\n\r\n# Iteration over a DataArray\r\nIn [4]: %%timeit t = DataArray(np.arange(10000))\r\nfor _ in t: pass\r\n   ...: \r\n1 loops, best of 3: 818 ms per loop\r\n```\r\n\r\nI'm not sure how much can be done about this as iterating over a DataArray needs to create a bunch of temporary objects (and I understand the emphasis is as usual on vectorized operations, etc.) but a >1500 fold difference certainly doesn't look good."},"comment":{"url":"https://api.github.com/repos/xray/xray/issues/comments/153258925","html_url":"https://github.com/xray/xray/issues/643#issuecomment-153258925","issue_url":"https://api.github.com/repos/xray/xray/issues/643","id":153258925,"user":{"login":"shoyer","id":1217238,"avatar_url":"https://avatars.githubusercontent.com/u/1217238?v=3","gravatar_id":"","url":"https://api.github.com/users/shoyer","html_url":"https://github.com/shoyer","followers_url":"https://api.github.com/users/shoyer/followers","following_url":"https://api.github.com/users/shoyer/following{/other_user}","gists_url":"https://api.github.com/users/shoyer/gists{/gist_id}","starred_url":"https://api.github.com/users/shoyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shoyer/subscriptions","organizations_url":"https://api.github.com/users/shoyer/orgs","repos_url":"https://api.github.com/users/shoyer/repos","events_url":"https://api.github.com/users/shoyer/events{/privacy}","received_events_url":"https://api.github.com/users/shoyer/received_events","type":"User","site_admin":false},"created_at":"2015-11-03T06:39:39Z","updated_at":"2015-11-03T06:39:39Z","body":"A useful point of reference here is to compare xray's performance to pandas:\r\n```\r\nIn [20]: %%timeit t = pd.DataFrame({'x': np.arange(10000)})\r\nfor _ in t.iterrows(): pass\r\n   ....:\r\n1 loops, best of 3: 558 ms per loop\r\n\r\nIn [21]: %%timeit t = DataArray(np.arange(10000))\r\nfor _ in t: pass\r\n   ....:\r\n1 loops, best of 3: 1.49 s per loop\r\n```\r\n\r\nSo we're about 2.5-3x slower than pandas. We might be able to catch up there with some careful optimization, but I doubt we would be able to do much better.\r\n\r\nBasically, the issue is that DataArray (like pandas's Series) is not an extension type (written in C/Cython), which means that it is much slower to construct -- iterating over it entails a lot of Python logic and a lot of memory allocation (e.g., a new dictionary for every instance).\r\n\r\nWe could imagine converting xray's core data objects to Cython extension types, but I doubt the tradeoff in easy of use would be worth it. I prefer to encourage users to use xray's easy interface to NumPy when entering into something performance critical like a loop."}}