{"action":"created","issue":{"url":"https://api.github.com/repos/scipy/scipy/issues/5436","labels_url":"https://api.github.com/repos/scipy/scipy/issues/5436/labels{/name}","comments_url":"https://api.github.com/repos/scipy/scipy/issues/5436/comments","events_url":"https://api.github.com/repos/scipy/scipy/issues/5436/events","html_url":"https://github.com/scipy/scipy/pull/5436","id":114013930,"number":5436,"title":"DOC: correct recommendation for minimize (trivial change)","user":{"login":"GaelVaroquaux","id":208217,"avatar_url":"https://avatars.githubusercontent.com/u/208217?v=3","gravatar_id":"","url":"https://api.github.com/users/GaelVaroquaux","html_url":"https://github.com/GaelVaroquaux","followers_url":"https://api.github.com/users/GaelVaroquaux/followers","following_url":"https://api.github.com/users/GaelVaroquaux/following{/other_user}","gists_url":"https://api.github.com/users/GaelVaroquaux/gists{/gist_id}","starred_url":"https://api.github.com/users/GaelVaroquaux/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GaelVaroquaux/subscriptions","organizations_url":"https://api.github.com/users/GaelVaroquaux/orgs","repos_url":"https://api.github.com/users/GaelVaroquaux/repos","events_url":"https://api.github.com/users/GaelVaroquaux/events{/privacy}","received_events_url":"https://api.github.com/users/GaelVaroquaux/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-10-29T09:45:43Z","updated_at":"2015-10-29T15:49:11Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/scipy/scipy/pulls/5436","html_url":"https://github.com/scipy/scipy/pull/5436","diff_url":"https://github.com/scipy/scipy/pull/5436.diff","patch_url":"https://github.com/scipy/scipy/pull/5436.patch"},"body":"The docstring of minimize is a bit too negative on the Nelder-Mead.\r\nUnlike what this docstring claims, the Nelder-Mead algorithm is very robust, in the sense that it doesn't break easily. It is less fast than a gradient-best method. However, those can easily be broken by a vanishing gradient. This is actually the reason that the recommended routine in\r\nMatlab optimization does not use a gradient-based solver (it actually uses the Simplex, aka Nelder-Mead). For people who do not understand at all what they do, and want a black box optimizer, it might be dangerous to push too much toward gradient-based solver.\r\n\r\nI think that the docstring is scipy downplays too much Nelder-Mead, and I am proposing a small rewording.\r\n\r\nSee the second exercise on http://www.scipy-lectures.org/advanced/mathematical_optimization/index.html#synthetic-exercices for a simple example that breaks gradient-based solvers."},"comment":{"url":"https://api.github.com/repos/scipy/scipy/issues/comments/152221858","html_url":"https://github.com/scipy/scipy/pull/5436#issuecomment-152221858","issue_url":"https://api.github.com/repos/scipy/scipy/issues/5436","id":152221858,"user":{"login":"josef-pkt","id":440735,"avatar_url":"https://avatars.githubusercontent.com/u/440735?v=3","gravatar_id":"","url":"https://api.github.com/users/josef-pkt","html_url":"https://github.com/josef-pkt","followers_url":"https://api.github.com/users/josef-pkt/followers","following_url":"https://api.github.com/users/josef-pkt/following{/other_user}","gists_url":"https://api.github.com/users/josef-pkt/gists{/gist_id}","starred_url":"https://api.github.com/users/josef-pkt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/josef-pkt/subscriptions","organizations_url":"https://api.github.com/users/josef-pkt/orgs","repos_url":"https://api.github.com/users/josef-pkt/repos","events_url":"https://api.github.com/users/josef-pkt/events{/privacy}","received_events_url":"https://api.github.com/users/josef-pkt/received_events","type":"User","site_admin":false},"created_at":"2015-10-29T15:49:11Z","updated_at":"2015-10-29T15:49:11Z","body":"(got distracted)\r\n\r\nI'm not sure about the `trust in gradients`. (I don't have an alternative proposal for the sentence.)\r\n\r\nI also think nelder-mead is still very useful. In our (statsmodels) cases it works well when the starting values are bad or the problem is badly scaled, even when we have pretty good gradients.\r\nBut in my experience in all smooth problems the gradient based methods are better in the neighborhood of the optimum. \r\n\r\nI didn't run the flat optimum example, but in my experience nelder-mead can get stuck in flat parts of the objective function, when bfgs or a pure newton method (with pretty good gradients) is still successful. (nelder-mead doesn't get gradients close enough to zero at the optimum.)\r\n\r\n(My strategy in difficult problems is often to use fmin first, and then try to switch to optimizers that use gradient and maybe hessian information to speed up or \"polish\".)\r\n"}}