{"push_id":584405747,"size":12,"distinct_size":6,"ref":"refs/heads/multiple-rooturls","head":"ab6156857fc061bd6845c029d7a62e68be5e7e46","before":"bf7c50980059daa5832fc66a9ecf9446b7eb109a","commits":[{"sha":"010afff356f1783fa98562a5c76c96bd6f2de595","author":{"email":"1fa2ef4755a9226cb9a0a4840bd89b158ac71391@mattbostock.com","name":"Matt Bostock"},"message":"Avoid url.Parse where possible\n\nWe were using `url.Parse()` to create a pointer to a new `url.URL`\nstruct but throwing away any errors generated.\n\nInstead, create the URL struct explicitly. This is cleaner IMO as it\navoids having to parse each URL, and means we have no error returned to\nworry about.\n\nSome URLs are changed from `http://` to `https://` for consistency; this\nshould also help DRY up the tests later.","distinct":false,"url":"https://api.github.com/repos/alphagov/govuk_crawler_worker/commits/010afff356f1783fa98562a5c76c96bd6f2de595"},{"sha":"c257dde9818812ce1cc3fe0e763135d463e9d0f0","author":{"email":"1fa2ef4755a9226cb9a0a4840bd89b158ac71391@mattbostock.com","name":"Matt Bostock"},"message":"Convert testURL from string into *url.URL{}","distinct":false,"url":"https://api.github.com/repos/alphagov/govuk_crawler_worker/commits/c257dde9818812ce1cc3fe0e763135d463e9d0f0"},{"sha":"e579055f41826e60fec46e3c8a2187739ec71596","author":{"email":"1fa2ef4755a9226cb9a0a4840bd89b158ac71391@mattbostock.com","name":"Matt Bostock"},"message":"Enclose unicode characters in double quotes\n\nFor consistency with the other tests; Go supports unicode by default so\nthere's no need to treat these characters any differently.","distinct":false,"url":"https://api.github.com/repos/alphagov/govuk_crawler_worker/commits/e579055f41826e60fec46e3c8a2187739ec71596"},{"sha":"73e6e55dbb2d5cba882dab14cf9caaa700427473","author":{"email":"1fa2ef4755a9226cb9a0a4840bd89b158ac71391@mattbostock.com","name":"Matt Bostock"},"message":"Use var block in test","distinct":false,"url":"https://api.github.com/repos/alphagov/govuk_crawler_worker/commits/73e6e55dbb2d5cba882dab14cf9caaa700427473"},{"sha":"5e6a4f35228776b712b634e4c2d94de11c588a22","author":{"email":"1fa2ef4755a9226cb9a0a4840bd89b158ac71391@mattbostock.com","name":"Matt Bostock"},"message":"Reduce duplication in var block\n\nBy bundling data types together.","distinct":false,"url":"https://api.github.com/repos/alphagov/govuk_crawler_worker/commits/5e6a4f35228776b712b634e4c2d94de11c588a22"},{"sha":"e1bc55124eb8efac34cd28243d2f5aed5e423c0c","author":{"email":"d44b6537b4a214e92da5b6fd78d5b10cc5ef0bda@gmail.com","name":"Kushal Pisavadia"},"message":"Merge pull request #89 from alphagov/tidy-tests\n\n[#58334884] Tidy tests","distinct":false,"url":"https://api.github.com/repos/alphagov/govuk_crawler_worker/commits/e1bc55124eb8efac34cd28243d2f5aed5e423c0c"},{"sha":"7a529ef409bf9ba4b4d6dd277894e2ee280ecac3","author":{"email":"173e4baf7722781318c4f9ba6b90abd8874b8258@digital.cabinet-office.gov.uk","name":"Alex Muller"},"message":"Add HostOnly method for the Crawler\n\nThis HostOnly method takes either a host or a host:port and returns\njust the host. It's similar to `net.SplitHostPort` but it swallows\nthe error that that method raises if you don't pass a port.","distinct":true,"url":"https://api.github.com/repos/alphagov/govuk_crawler_worker/commits/7a529ef409bf9ba4b4d6dd277894e2ee280ecac3"},{"sha":"be5ae0668f88a24ff47d579f3e5997b44601f981","author":{"email":"173e4baf7722781318c4f9ba6b90abd8874b8258@digital.cabinet-office.gov.uk","name":"Alex Muller"},"message":"Add IsAllowedHost method to Crawler\n\nThe IsAllowedHost method takes a single host and a slice of hosts,\nand returns true as long as the host is contained in the slice.","distinct":true,"url":"https://api.github.com/repos/alphagov/govuk_crawler_worker/commits/be5ae0668f88a24ff47d579f3e5997b44601f981"},{"sha":"eba78f6005dd08cc69f50f39aec966c04a364ca4","author":{"email":"173e4baf7722781318c4f9ba6b90abd8874b8258@digital.cabinet-office.gov.uk","name":"Alex Muller"},"message":"Use hostname in RelativeFilePath method\n\nIn order to support multiple hostnames for the crawler, we need to\nstore HTML from each hostname in a separate directory. The directory\nname is exactly the hostname of the crawled site.","distinct":true,"url":"https://api.github.com/repos/alphagov/govuk_crawler_worker/commits/eba78f6005dd08cc69f50f39aec966c04a364ca4"},{"sha":"4922ec973aad7cc02985e4e3ca5e6097efed8b72","author":{"email":"173e4baf7722781318c4f9ba6b90abd8874b8258@digital.cabinet-office.gov.uk","name":"Alex Muller"},"message":"Tidy tests\n\n- Use baseURL instead of rootURL.String()\n- Change `url` to `u` to not be confused with the URL package","distinct":true,"url":"https://api.github.com/repos/alphagov/govuk_crawler_worker/commits/4922ec973aad7cc02985e4e3ca5e6097efed8b72"},{"sha":"ac217b50e26cd18730c3b135828d89ca0dea90af","author":{"email":"173e4baf7722781318c4f9ba6b90abd8874b8258@digital.cabinet-office.gov.uk","name":"Alex Muller"},"message":"Rename contains method to containsInt\n\nTo make it more clear that this method only searches slices for\nintegers.","distinct":true,"url":"https://api.github.com/repos/alphagov/govuk_crawler_worker/commits/ac217b50e26cd18730c3b135828d89ca0dea90af"},{"sha":"ab6156857fc061bd6845c029d7a62e68be5e7e46","author":{"email":"173e4baf7722781318c4f9ba6b90abd8874b8258@digital.cabinet-office.gov.uk","name":"Alex Muller"},"message":"Support multiple rootURLs\n\nWe have 2 hostnames, one that we serve the site on and one which\nis used for assets. We'd like to crawl URLs from both hostnames\nand store them in separate places on disk.","distinct":true,"url":"https://api.github.com/repos/alphagov/govuk_crawler_worker/commits/ab6156857fc061bd6845c029d7a62e68be5e7e46"}]}