{"action":"created","issue":{"url":"https://api.github.com/repos/trotzig/likadan/issues/4","labels_url":"https://api.github.com/repos/trotzig/likadan/issues/4/labels{/name}","comments_url":"https://api.github.com/repos/trotzig/likadan/issues/4/comments","events_url":"https://api.github.com/repos/trotzig/likadan/issues/4/events","html_url":"https://github.com/trotzig/likadan/issues/4","id":77028259,"number":4,"title":"Come up with strategy for running Likadan in a CI environment","user":{"login":"trotzig","id":578277,"avatar_url":"https://avatars.githubusercontent.com/u/578277?v=3","gravatar_id":"","url":"https://api.github.com/users/trotzig","html_url":"https://github.com/trotzig","followers_url":"https://api.github.com/users/trotzig/followers","following_url":"https://api.github.com/users/trotzig/following{/other_user}","gists_url":"https://api.github.com/users/trotzig/gists{/gist_id}","starred_url":"https://api.github.com/users/trotzig/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/trotzig/subscriptions","organizations_url":"https://api.github.com/users/trotzig/orgs","repos_url":"https://api.github.com/users/trotzig/repos","events_url":"https://api.github.com/users/trotzig/events{/privacy}","received_events_url":"https://api.github.com/users/trotzig/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-05-16T12:25:52Z","updated_at":"2015-07-17T21:02:01Z","closed_at":null,"body":"One of the main reasons for creating Likadan was that I wanted to run perceptual diffs in a CI environment. So far, I've mainly focused on the tool itself, and not the CI process. But I think it's time Likadan is enhanced even further so that it could be used alongside unit/feature/integration tests, linters, and other tools that aim at automatically making sure that the code is of high quality and prevent bugs.\r\n\r\nThe straight-forward way would be to run Likadan on each commit, and fail if a diff is found. It's not that simple however. I've noticed that after running Likadan manually once every day on the Brigade code base for the last month. I did this because I wanted to get more experience with the type of diffs that are normally caught. The Likadan suite for Brigade consists of 64 examples. Everything from smaller components (buttons, tooltips) to larger components (dialogs, cards) is tested. Here are a few things I've learned from running it:\r\n\r\n- Few diffs are actual regressions in style\r\n- Many diffs are intentional, and caused by iterating on the design of a component\r\n- A number of diffs are somewhat unintentional, but harmless (changing a global color), refactoring some css, etc.\r\n\r\nTo make Likadan fully automated, we need to separate the real regression diffs from other diffs. I'm not sure how to do that. There's always a bit of human activity needed to figure out which diffs are bad and which are good. Lately I've been thinking about how we can aid manual diffing with automated diffing. What if Likadan posted the diffs as a comment to the commit? Then it would be easy for a code reviewer (someone else or the author) to make the call. If you like Brigade use Gerrit for code review, we could post a comment with a link to the diffs (I'm not sure you can inline images in Gerrit). If you use the Github PR model, we could post a comment with inline diff images to the pull request. \r\n\r\nThe basic CI script for likadan would\r\n\r\n1. Check out the code before the commit\r\n2. Run `likadan` to generate baseline snapshots\r\n3. Check out the new commit\r\n4. Run `likadan` again\r\n5. Report back result\r\n\r\nAnother option I've considered is tagging a commit that you know will result in a diff with something that Likadan can use to auto-approve diffs. Let's pretend the commit message looks like this:\r\n\r\n```\r\nMake buttons wider\r\n\r\nWe want buttons to be easier to target. The previous width \r\nwasn't enough for people with small thumbs.\r\n\r\n[likadan changes \"button\"]\r\n[likadan changes \"button with icon\"]\r\n[likadan changes \"button group\"]\r\n```\r\n\r\nThen when the Likadan suite is run, the tagged diffs will be ignored. I'm not a fan of this solution right now though. I don't want to pollute the git history with those tags, plus I think it's better to make the call at review time. \r\n\r\nWhat do you think? Can you come up with an even better alternative, or perhaps refine what I'm proposing? "},"comment":{"url":"https://api.github.com/repos/trotzig/likadan/issues/comments/122414837","html_url":"https://github.com/trotzig/likadan/issues/4#issuecomment-122414837","issue_url":"https://api.github.com/repos/trotzig/likadan/issues/4","id":122414837,"user":{"login":"lencioni","id":195534,"avatar_url":"https://avatars.githubusercontent.com/u/195534?v=3","gravatar_id":"","url":"https://api.github.com/users/lencioni","html_url":"https://github.com/lencioni","followers_url":"https://api.github.com/users/lencioni/followers","following_url":"https://api.github.com/users/lencioni/following{/other_user}","gists_url":"https://api.github.com/users/lencioni/gists{/gist_id}","starred_url":"https://api.github.com/users/lencioni/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lencioni/subscriptions","organizations_url":"https://api.github.com/users/lencioni/orgs","repos_url":"https://api.github.com/users/lencioni/repos","events_url":"https://api.github.com/users/lencioni/events{/privacy}","received_events_url":"https://api.github.com/users/lencioni/received_events","type":"User","site_admin":false},"created_at":"2015-07-17T21:02:01Z","updated_at":"2015-07-17T21:02:01Z","body":"We've recently had some styling regressions at Brigade. Running likadan manually later really helped, but I think that CI integration would have prevented it entirely. @trotzig what do you think would be involved in landing this at this point? @sds might have some ideas for how we can hook directly into Gerrit."}}