{"action":"created","issue":{"url":"https://api.github.com/repos/sympy/sympy/issues/9221","labels_url":"https://api.github.com/repos/sympy/sympy/issues/9221/labels{/name}","comments_url":"https://api.github.com/repos/sympy/sympy/issues/9221/comments","events_url":"https://api.github.com/repos/sympy/sympy/issues/9221/events","html_url":"https://github.com/sympy/sympy/pull/9221","id":65189585,"number":9221,"title":"Humanise @slow","user":{"login":"pzrq","id":1217010,"avatar_url":"https://avatars.githubusercontent.com/u/1217010?v=3","gravatar_id":"","url":"https://api.github.com/users/pzrq","html_url":"https://github.com/pzrq","followers_url":"https://api.github.com/users/pzrq/followers","following_url":"https://api.github.com/users/pzrq/following{/other_user}","gists_url":"https://api.github.com/users/pzrq/gists{/gist_id}","starred_url":"https://api.github.com/users/pzrq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pzrq/subscriptions","organizations_url":"https://api.github.com/users/pzrq/orgs","repos_url":"https://api.github.com/users/pzrq/repos","events_url":"https://api.github.com/users/pzrq/events{/privacy}","received_events_url":"https://api.github.com/users/pzrq/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/sympy/sympy/labels/Testing","name":"Testing","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-03-30T11:22:39Z","updated_at":"2015-04-06T22:14:40Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/sympy/sympy/pulls/9221","html_url":"https://github.com/sympy/sympy/pull/9221","diff_url":"https://github.com/sympy/sympy/pull/9221.diff","patch_url":"https://github.com/sympy/sympy/pull/9221.patch"},"body":"This builds on the discussion in https://github.com/sympy/sympy/pull/9197\r\n\r\nBasically the idea is we should be able to make a better trade off when running the majority of the test suite on a local machine for refactoring or development (and have Travis let us know about the really slow final 1%, or alternately use `sympy.test(slow=True)`).\r\n\r\nThe current trade off is the `@slow` tests on master, after excluding the `ON_TRAVIS` extra-slow tests, take around one hour to run, and the main test suite also takes around one hour to run, i.e. **around 2 hours total** if you really want to know what the Travis output is for at least one Python interpreter before sending it to Travis. \r\n\r\nI think a better trade off would be to make around 1% of those, which happens to be around 49 additional tests, also `@slow`, which reduces the test suite run time to **around 30 minutes**. This branch also finds several tests are probably not worth leaving as `@slow` in the context of their modules where other tests often take >10x longer to run, and removes those `@slow`.\r\n\r\nTesting was performed under OSX Python 2.7, Late 2012 Mac Mini, 2.3GHz Intel Core i7, 16GB RAM.\r\n\r\nThis branch also includes additions to the test framework which print the slowest 3 tests if they took more than 10 seconds each to run, and the symmetric print that if a `@slow` test runs in under 100ms it is considered fast, though others are of course free to determine that something slower than 100ms is fast enough given the context of what is being attempted and that having test coverage by default is more important. 10s and 0.1s (100ms) are based on:\r\nhttp://www.nngroup.com/articles/response-times-3-important-limits/\r\n\r\nObviously there are exceptions to these, they are hopefully in the commit messages though I would welcome feedback from anyone else in case I've inadvertently made some things `@slow` that really should not be or if there are any other concerns.\r\n\r\nMy plan is to use this branch whether or not it is ultimately merged in, as I have several other SymPy bugs and regressions I'd like to try to fix, so I expect it to work out as a better trade off for me even if not for anyone else. That said hopefully it is useful :)"},"comment":{"url":"https://api.github.com/repos/sympy/sympy/issues/comments/90261716","html_url":"https://github.com/sympy/sympy/pull/9221#issuecomment-90261716","issue_url":"https://api.github.com/repos/sympy/sympy/issues/9221","id":90261716,"user":{"login":"toolforger","id":915906,"avatar_url":"https://avatars.githubusercontent.com/u/915906?v=3","gravatar_id":"","url":"https://api.github.com/users/toolforger","html_url":"https://github.com/toolforger","followers_url":"https://api.github.com/users/toolforger/followers","following_url":"https://api.github.com/users/toolforger/following{/other_user}","gists_url":"https://api.github.com/users/toolforger/gists{/gist_id}","starred_url":"https://api.github.com/users/toolforger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/toolforger/subscriptions","organizations_url":"https://api.github.com/users/toolforger/orgs","repos_url":"https://api.github.com/users/toolforger/repos","events_url":"https://api.github.com/users/toolforger/events{/privacy}","received_events_url":"https://api.github.com/users/toolforger/received_events","type":"User","site_admin":false},"created_at":"2015-04-06T22:14:40Z","updated_at":"2015-04-06T22:14:40Z","body":"Sorry for coming back to you so late.\n\n> @toolforger Just made the slow and fast thresholds configurable and\n> added a comment to hopefully encourage others to come up with a\n> better yardstick if one exists.\n\nI like what I see.\n\nI'm not too happy with measuring wall-clock time, it's too dependent on \nmachine performance (and also on CPU load and on whether somebody puts \nthe machine in suspended state). However, as a first experimental \napproach this is okay.\n\nI do have some thoughts how to eliminate most relevant sources of error; \nit involves a separate thread that inspects the call stack at regular \nintervals, and measuring time as a percentage of how presences in the \nstack. It would be very different from what you're doing here, but maybe \nmore useful across machines.\nWhatever the future brings in terms of better performance diagnostics, \nplease split this PR in two: One that does the changes around `@slow`, \nand one that adds the thresholds and timing diagnostics. This makes it \neasier to revert the timing diagnostics when/if we want to supersede \nthis work.\n\n> To me it's that the test runner doesn't print any single character\n> while it is running a test function, so there is no obvious\n> indication that anything is happening, or if the test has actually\n> hung (i.e. will the [calculation under test even\n> halt](http://en.wikipedia.org/wiki/Halting_problem)?).\n\nThe tests all finish, except one of them used to eat so many gigabytes \nof memory that it would always overflow memory (I hear that that's fixed \nin the mean time).\n\n> If that's actually because the test is large and can be broken into\n> several smaller tests that give the faster, more granular feedback\n> and remove the `@slow` I'd still count that as an overall win.\n\nThat's marginal, there are factors for and against that.\nUsually, however, tests are already at the smallest granularity that the \noriginal author considered useful, so attacking slowness from this angle \nisn't going to give great rewards in relation to the effort involve.\n"}}