{"action":"created","issue":{"url":"https://api.github.com/repos/numpy/numpy/issues/5844","labels_url":"https://api.github.com/repos/numpy/numpy/issues/5844/labels{/name}","comments_url":"https://api.github.com/repos/numpy/numpy/issues/5844/comments","events_url":"https://api.github.com/repos/numpy/numpy/issues/5844/events","html_url":"https://github.com/numpy/numpy/issues/5844","id":73550255,"number":5844,"title":"Decide on what the resolution rules for __op__/__rop__/__numpy_ufunc__ actually are","user":{"login":"njsmith","id":609896,"avatar_url":"https://avatars.githubusercontent.com/u/609896?v=3","gravatar_id":"","url":"https://api.github.com/users/njsmith","html_url":"https://github.com/njsmith","followers_url":"https://api.github.com/users/njsmith/followers","following_url":"https://api.github.com/users/njsmith/following{/other_user}","gists_url":"https://api.github.com/users/njsmith/gists{/gist_id}","starred_url":"https://api.github.com/users/njsmith/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/njsmith/subscriptions","organizations_url":"https://api.github.com/users/njsmith/orgs","repos_url":"https://api.github.com/users/njsmith/repos","events_url":"https://api.github.com/users/njsmith/events{/privacy}","received_events_url":"https://api.github.com/users/njsmith/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/numpy/numpy/milestones/30","html_url":"https://github.com/numpy/numpy/milestones/1.10%20blockers","labels_url":"https://api.github.com/repos/numpy/numpy/milestones/30/labels","id":518598,"number":30,"title":"1.10 blockers","description":"","creator":{"login":"seberg","id":61977,"avatar_url":"https://avatars.githubusercontent.com/u/61977?v=3","gravatar_id":"","url":"https://api.github.com/users/seberg","html_url":"https://github.com/seberg","followers_url":"https://api.github.com/users/seberg/followers","following_url":"https://api.github.com/users/seberg/following{/other_user}","gists_url":"https://api.github.com/users/seberg/gists{/gist_id}","starred_url":"https://api.github.com/users/seberg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/seberg/subscriptions","organizations_url":"https://api.github.com/users/seberg/orgs","repos_url":"https://api.github.com/users/seberg/repos","events_url":"https://api.github.com/users/seberg/events{/privacy}","received_events_url":"https://api.github.com/users/seberg/received_events","type":"User","site_admin":false},"open_issues":25,"closed_issues":6,"state":"open","created_at":"2013-12-20T16:52:18Z","updated_at":"2015-05-09T21:09:58Z","due_on":null,"closed_at":null},"comments":151,"created_at":"2015-05-06T08:09:28Z","updated_at":"2015-05-27T13:47:59Z","closed_at":null,"body":"There is a complex set of questions around how to handle method resolution in the presence of `__numpy_ufunc__`. Currently in master is an extremely complicated set of rules that isn't documented and that I don't actually understand (see #5748 for the latest set of changes to this), so it's kinda hard to know whether they are correct, but I suspect not. And this is a blocker for 1.10, b/c whatever we release in 1.10 will be set in stone forever.\r\n\r\nI strongly feel that we cannot include `__numpy_ufunc__` in a release without at least having a document somewhere describing what the actual dispatch rules are. I hope that doesn't mean we have to defer `__numpy_ufunc__` for another release, but if it does then it does.\r\n\r\nAFAICT this is how `a op b` dispatch works for ndarrays, BEFORE `__numpy_ufunc__` (i.e., this is how 1.9 works):\r\n\r\n* First Python uses the subclass rule to decide whether to invoke `a.__op__(b)` or `b.__rop__(a)`. So in the case where one of these objects is a proper subclass of the other, that object always gets to do absolutely anything, so that's fine. The interesting cases are the ones where neither is a proper subclass of the other (either because it's like, `matrix + masked array`, or because it's like `ndarray + scipy.sparse`). So without loss of generality, let's focus on the case where Python calls `a.__op__(b)`, and `a` is either an instance of `ndarray` or else an instance of a subclass of `ndarray` which has not overridden `__op__`, i.e. we're getting `ndarray.__op__(a, b)`.\r\n* `ndarray.__op__` has the following logic (see `PyArray_GenericBinaryFunction` in `number.c`):\r\n  * If `b` is not an `ndarray` at all (even a subclass), and `b` has a higher `__array_priority__` than `a`, then we return `NotImplemented` and let control pass to `b.__rop__(a)`.\r\n  * Otherwise, we call `np.op(a, b)` and let the ufunc machinery take over.\r\n* `np.op(a, b)` does the following (see `PyUFunc_GenericFunction`, `PyUFunc_GeneralizedFunction`,  in `ufunc_object.c`, and also `ufunc_generic_call` which converts `-2` return values from the previous into `NotImplemented` so you have to audit their whole call stack):\r\n  * If `b` is not an `ndarray`, and calling `np.array(b)` returns an object array (presumably because coercion failed... though I guess this could also be hit if `b.__array__()` return an object array or something), AND `b` has a higher `__array_priority__` than `a`, and `b` has an `__rop__` method, then return `NotImplemented`.\r\n  * If any of our arrays contain structured dtypes or strings, and there are no special struct ufunc loops registered, but not if any of our arrays contain objects, then return `NotImplemented`. (This is buried in `get_ufunc_arguments`, search for `return -2`.)\r\n  * Otherwise we return the actual ufunc result.\r\n\r\nNow, my suggestion is that the way we would EVENTUALLY like this to look is:\r\n\r\n* First, Python uses the subclass rule to decide whether to invoke `a.__op__(b)` or `b.__rop__(a)`. As above, let's assume that it invokes `ndarray.__op__(a, b)`.\r\n* `ndarray.__op__(a, b)` calls `np.op(a, b)` (which in turn invokes all the standard ufunc stuff, including `__numpy_ufunc__` resolution).\r\n* There is no step 3.\r\n\r\nI submit that it is obvious that IF we can make this work, then it is obviously the ideal outcome, because it is the simplest possible solution. But is it too simple? To determine this we have to answer two questions: (1) Will it adequately address all the relevant use cases? (2) Can we get there from here?\r\n\r\nSo let's compare the current rules to my dream rules.\r\n\r\nFirst, we observe that everything that currently happens inside the ufunc machinery looks like it's totally wrong. The first check can only be triggered if `b` is a non-`ndarray` that has a higher `__array_priority__` (among other things), but if we look above, we see that those conditions are sufficient to trigger the check in `ndarray.__op__`, so checking again at the ufunc level is redundant at best. And the second check is just incoherent nonsense AFAICT. The only reason to return `NotImplemented` is b/c you want to pass control to another `__(r)op__` method, and there's no reason arrays containing structured dtypes in particular should somehow magically have different `__(r)op__` methods available than other arrays. So we can just get rid of all the ufunc stuff immediately, great.\r\n\r\nThat leaves the `__array_priority__` stuff. We have two problems here: we can't just drop this immediately b/c of backcompat issues, and we need to have some way to continue to support all the use cases that this currently supports. The first problem is just a matter of having a deprecation period. For the second, observe that a class which defines a `__numpy_ufunc__` method gets complete control over what any ufunc call does, so it has *almost* as much power as a class that currently sets `__array_priority__`. The only additional power that `__array_priority__` currently gives you is that it lets you distinguish between e.g. a call to `ndarray.__add(a, b)` versus a call to `np.add(a, b)`. So the only code that really loses out from my proposed change is code which wants `a + b` and `add(a, b)` to do different things.\r\n\r\nAFAIK in the entire history of numpy there is only one situation where this power has been used on purpose: the definition of matrix classes where `a * b` is matmul, but `np.multiply(a, b)` is elmul. And we've all agreed that such classes should be deprecated and eventually phased out (cite: PEP 465).\r\n\r\nSo, I conclude that EVENTUALLY my dream rules should work great. The only problem is that we need some temporary compromises to get us from here to there. Therefore, I propose we use the following dispatch rules in numpy 1.10, with the goal of moving to my \"dream rules\" in some future version:\r\n\r\n* First, Python uses the subclass rule to decide whether to invoke `a.__op__(b)` or `b.__rop__(a)`. As above, let's assume that it invokes `ndarray.__op__(a, b)`.\r\n* `ndarray.__op__(a, b)` does the following:\r\n  * If `b` **does not define `__numpy_ufunc__` and** is not an `ndarray` at all (even a subclass), and `b` has a higher `__array_priority__` than `a`, then we **issue a deprecation warning and** return `NotImplemented` and let control pass to `b.__rop__(a)`. (bolded parts are changes compared to the current behaviour)\r\n  * If `__op__` is `__mul__` and `b->tp_class->tp_name.startswith(\"scipy.sparse.\")`, then return `NotImplemented`. (This rule is necessary in addition to the above, because `scipy.sparse` has already made a release containing `__numpy_ufunc__` methods, so the exception above doesn't apply.)\r\n  * Otherwise, we call `np.op(a, b)` and let the ufunc machinery take over.\r\n\r\nI believe that this is adequate to covers all practical use cases for the current dispatch machinery, and gives us a clean path to better dispatch machinery in the future.\r\n\r\nThe main alternative proposal is Pauli's, which involves a very complicated check (I won't try to summarize here, see [this comment and following code](https://github.com/numpy/numpy/blob/eecb2e3c07f29c0ac991d364a846a2f8293a432a/numpy/core/src/multiarray/number.c#L103)). The goal of that approach is to continue supporting classes where `a + b` and `add(a, b)` do different things. I don't think that keeping substantial additional complexity around indefinitely is worth it in order to support functionality that no-one has ever found a use for except in one very specific case (overriding `__mul__`), and where we generally agree that that one specific case should be phased out as possible.\r\n\r\nI would very much appreciate feedback from scipy.sparse and astropy in particular on whether the above covers all their concerns.\r\n\r\n(Partial) History: #4815, #5748 \r\nCC: @pv, @cowlicks, @mhvk"},"comment":{"url":"https://api.github.com/repos/numpy/numpy/issues/comments/105917060","html_url":"https://github.com/numpy/numpy/issues/5844#issuecomment-105917060","issue_url":"https://api.github.com/repos/numpy/numpy/issues/5844","id":105917060,"user":{"login":"shoyer","id":1217238,"avatar_url":"https://avatars.githubusercontent.com/u/1217238?v=3","gravatar_id":"","url":"https://api.github.com/users/shoyer","html_url":"https://github.com/shoyer","followers_url":"https://api.github.com/users/shoyer/followers","following_url":"https://api.github.com/users/shoyer/following{/other_user}","gists_url":"https://api.github.com/users/shoyer/gists{/gist_id}","starred_url":"https://api.github.com/users/shoyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shoyer/subscriptions","organizations_url":"https://api.github.com/users/shoyer/orgs","repos_url":"https://api.github.com/users/shoyer/repos","events_url":"https://api.github.com/users/shoyer/events{/privacy}","received_events_url":"https://api.github.com/users/shoyer/received_events","type":"User","site_admin":false},"created_at":"2015-05-27T13:47:59Z","updated_at":"2015-05-27T13:47:59Z","body":"> Now if a user writes a reasonable expression like np.add(dask_array1, dask_array2, out=xray_dataset), then this works perfectly... iff dask.array.__numpy_ufunc__ knows to pass out= arguments through its checks and return NotImplemented, and xray.Dataset.__numpy_ufunc__ knows how to process out= arguments too\r\n\r\nIs this actually true? AFAICT, `__numpy_ufunc__` only dispatches based on input arguments. There's no generic way to signal that the output argument should handle the operation."}}