{"action":"created","issue":{"url":"https://api.github.com/repos/opal/opal/issues/1255","labels_url":"https://api.github.com/repos/opal/opal/issues/1255/labels{/name}","comments_url":"https://api.github.com/repos/opal/opal/issues/1255/comments","events_url":"https://api.github.com/repos/opal/opal/issues/1255/events","html_url":"https://github.com/opal/opal/issues/1255","id":123693857,"number":1255,"title":"Add Auto Benchmarking","user":{"login":"wied03","id":297123,"avatar_url":"https://avatars.githubusercontent.com/u/297123?v=3","gravatar_id":"","url":"https://api.github.com/users/wied03","html_url":"https://github.com/wied03","followers_url":"https://api.github.com/users/wied03/followers","following_url":"https://api.github.com/users/wied03/following{/other_user}","gists_url":"https://api.github.com/users/wied03/gists{/gist_id}","starred_url":"https://api.github.com/users/wied03/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wied03/subscriptions","organizations_url":"https://api.github.com/users/wied03/orgs","repos_url":"https://api.github.com/users/wied03/repos","events_url":"https://api.github.com/users/wied03/events{/privacy}","received_events_url":"https://api.github.com/users/wied03/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-12-23T17:14:48Z","updated_at":"2015-12-23T17:39:27Z","closed_at":null,"body":"In order to keep track of which methods are optimize-able by JS runtimes, do something like this:\r\n\r\nhttps://gist.github.com/wied03/76cb79e6076e48eee9a5\r\n\r\nKey points from Gitter:\r\n* Use text based output (not HTML reports)\r\n* Fail the build (use Travis here) if new unoptimized methods appear\r\n* Have a white list of class#method items that are \"allowed\" to be not optimized, and fail the build if a class#method that's not on that list is not optimized. This would catch regressions, and would force one to add a class#method to the list explicitly if there's no other way to implement the method. This would also let us track progress on optimization, as the list gets shorter over time.\r\n* Might need each method check to run in its own process. The reason I mention it is that functions can get deoptimized for being too polymorphic. For example, if you call a function with more than some VM-specific number of different types of arguments (pretty sure in Safari it's 4, might be 3 in Chrome), it can decide to deoptimize the function because the cost (both performance and memory) of optimizing it for each different type of argument potentially outweighs the performance you gain from it."},"comment":{"url":"https://api.github.com/repos/opal/opal/issues/comments/166953116","html_url":"https://github.com/opal/opal/issues/1255#issuecomment-166953116","issue_url":"https://api.github.com/repos/opal/opal/issues/1255","id":166953116,"user":{"login":"iliabylich","id":1265492,"avatar_url":"https://avatars.githubusercontent.com/u/1265492?v=3","gravatar_id":"","url":"https://api.github.com/users/iliabylich","html_url":"https://github.com/iliabylich","followers_url":"https://api.github.com/users/iliabylich/followers","following_url":"https://api.github.com/users/iliabylich/following{/other_user}","gists_url":"https://api.github.com/users/iliabylich/gists{/gist_id}","starred_url":"https://api.github.com/users/iliabylich/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/iliabylich/subscriptions","organizations_url":"https://api.github.com/users/iliabylich/orgs","repos_url":"https://api.github.com/users/iliabylich/repos","events_url":"https://api.github.com/users/iliabylich/events{/privacy}","received_events_url":"https://api.github.com/users/iliabylich/received_events","type":"User","site_admin":false},"created_at":"2015-12-23T17:39:27Z","updated_at":"2015-12-23T17:39:27Z","body":"@wied03 Travis clones the main `opal` repo and switches to the post-merge branch (which becomes available in `FETCH_HEAD`). I guess the code should look like:\r\n``` sh\r\ngit checkout master\r\n# report 1\r\nBENCH_NAME=master rake bench\r\nget checkout FETCH_HEAD\r\n# report 2\r\nBENCH_NAME=pr rake bench\r\nBENCHES=master,pr rake bench:compare\r\n```\r\n\r\nAs I know, Travis runs everything in Docker containers. So it becomes possible for one docker image to affect performance of another container on the same physical server, right?\r\n\r\nOne more thing about reporting. Text-based report is the only way to get a response from Travis, but its output may become huge. Any thoughts about it?\r\n\r\nHow should we proceed with some cases when `klass#method` was fast, but incomplete, and after implementing it completely we get some performance regression (which is required due to new complexity)?"}}