{"action":"created","comment":{"url":"https://api.github.com/repos/benbalter/site-inspector-ruby/pulls/comments/28212242","id":28212242,"diff_hunk":"@@ -49,8 +51,13 @@ def domain\n     www? ? PublicSuffix.parse(\"www.#{@uri.host}\") : @domain\n   end\n \n-  def request(ssl=false, www=false, followlocation=true)\n-    Typhoeus.get(uri(ssl, www), followlocation: followlocation, timeout: 10)\n+  def request(ssl=false, www=false, followlocation=true, ssl_verifypeer=true, ssl_verifyhost=true)","path":"lib/site-inspector.rb","position":34,"original_position":34,"commit_id":"18638991c14591df4fbebf7de41f819ccc07a301","original_commit_id":"18638991c14591df4fbebf7de41f819ccc07a301","user":{"login":"konklone","id":4592,"avatar_url":"https://avatars.githubusercontent.com/u/4592?v=3","gravatar_id":"","url":"https://api.github.com/users/konklone","html_url":"https://github.com/konklone","followers_url":"https://api.github.com/users/konklone/followers","following_url":"https://api.github.com/users/konklone/following{/other_user}","gists_url":"https://api.github.com/users/konklone/gists{/gist_id}","starred_url":"https://api.github.com/users/konklone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/konklone/subscriptions","organizations_url":"https://api.github.com/users/konklone/orgs","repos_url":"https://api.github.com/users/konklone/repos","events_url":"https://api.github.com/users/konklone/events{/privacy}","received_events_url":"https://api.github.com/users/konklone/received_events","type":"User","site_admin":false},"body":"Adds two additional options to `request` to disable trust validation and hostname verification, respectively. They have no effect if passed in to an `http://` request.","created_at":"2015-04-13T02:57:34Z","updated_at":"2015-04-13T02:57:34Z","html_url":"https://github.com/benbalter/site-inspector-ruby/pull/24#discussion_r28212242","pull_request_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/pulls/24","_links":{"self":{"href":"https://api.github.com/repos/benbalter/site-inspector-ruby/pulls/comments/28212242"},"html":{"href":"https://github.com/benbalter/site-inspector-ruby/pull/24#discussion_r28212242"},"pull_request":{"href":"https://api.github.com/repos/benbalter/site-inspector-ruby/pulls/24"}}},"pull_request":{"url":"https://api.github.com/repos/benbalter/site-inspector-ruby/pulls/24","id":33132675,"html_url":"https://github.com/benbalter/site-inspector-ruby/pull/24","diff_url":"https://github.com/benbalter/site-inspector-ruby/pull/24.diff","patch_url":"https://github.com/benbalter/site-inspector-ruby/pull/24.patch","issue_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/issues/24","number":24,"state":"open","locked":false,"title":"A much more detailed, granular HTTP mode","user":{"login":"konklone","id":4592,"avatar_url":"https://avatars.githubusercontent.com/u/4592?v=3","gravatar_id":"","url":"https://api.github.com/users/konklone","html_url":"https://github.com/konklone","followers_url":"https://api.github.com/users/konklone/followers","following_url":"https://api.github.com/users/konklone/following{/other_user}","gists_url":"https://api.github.com/users/konklone/gists{/gist_id}","starred_url":"https://api.github.com/users/konklone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/konklone/subscriptions","organizations_url":"https://api.github.com/users/konklone/orgs","repos_url":"https://api.github.com/users/konklone/repos","events_url":"https://api.github.com/users/konklone/events{/privacy}","received_events_url":"https://api.github.com/users/konklone/received_events","type":"User","site_admin":false},"body":"OK, the work in this branch does not have sufficient tests. I'm opening it here so I can get a sense from @benbalter whether or not this is going in a direction that's going to be accepted into site-inspector.\r\n\r\nThe goal of this work is to provide a more detailed and accurate glimpse into the behavior of domains at all four standard combinations of root and protocol:\r\n\r\n* `https://examplecom`\r\n* `http://example.com`\r\n* `https://www.example.com`\r\n* `http://www.example.com`\r\n\r\nIt is downstream from and **replaces** the `--http` flag work I did in #22, so that the `--http` flag now triggers a very different response object. It's not backwards-compatible, and more flexible about hostname verification and trust validation for TLS.\r\n\r\nThis comes out of a few needs I have:\r\n\r\n* Domains are not always rational about how redirects between the main 4 combos work, and so site-inspector's idea of a \"canonical\" domain can miss important details or make incorrect claims about the domain.\r\n* When measuring HSTS, what matters are the headers and TLS validation on the root domain, not the canonical domain.\r\n* It's relevant, and [in some cases crucial](https://github.com/benbalter/site-inspector-ruby/issues/21), to detect when the site might be live but have an invalid certificate chain. It's not as relevant, but still helpful, to capture when a site has an HTTPS certificate for an invalid hostname (e.g. an `akamai.net` hostname).\r\n\r\nIf this work is too much of a shift for this library, I'll understand, and can probably go repurpose this into a standalone thing. But it seems pretty relevant to the library's goal, and after spending some hours on it, I didn't know how to elegantly merge the original mode and this mode.\r\n\r\nHere's an example:\r\n\r\n```\r\n$ ./bin/site-inspector nasa.gov --http\r\n```\r\n\r\n```json\r\n{\r\n  \"canonical\":\"http://www.nasa.gov\",\r\n  \"canonical_protocol\":\"https\",\r\n  \"canonical_endpoint\":\"www\",\r\n  \"endpoints\":{\r\n    \"https\":{\r\n      \"www\":{\r\n        \"https_valid\":true,\r\n        \"status\":200,\r\n        \"hsts\":false,\r\n        \"hsts_header\":null,\r\n        \"redirect\":false,\r\n        \"redirect_to_external\":false,\r\n        \"redirect_to_https\":false,\r\n        \"live\":true\r\n      },\r\n      \"root\":{\r\n        \"status\":0\r\n      }\r\n    },\r\n    \"http\":{\r\n      \"www\":{\r\n        \"status\":200,\r\n        \"hsts\":false,\r\n        \"hsts_header\":null,\r\n        \"redirect\":false,\r\n        \"redirect_to_external\":false,\r\n        \"redirect_to_https\":false,\r\n        \"live\":true\r\n      },\r\n      \"root\":{\r\n        \"status\":0\r\n      }\r\n    }\r\n  },\r\n  \"live\":true,\r\n  \"broken_root\":true,\r\n  \"broken_www\":false,\r\n  \"enforce_https\":false,\r\n  \"redirect\":false,\r\n  \"hsts\":false,\r\n  \"hsts_entire_domain\":null,\r\n  \"hsts_entire_domain_preload\":null\r\n}\r\n```\r\n\r\nAnd another:\r\n\r\n```\r\n$ ./bin/site-inspector whitehouse.gov --http\r\n```\r\n\r\n```json\r\n{\r\n  \"canonical\":\"https://www.whitehouse.gov\",\r\n  \"canonical_protocol\":\"https\",\r\n  \"canonical_endpoint\":\"www\",\r\n  \"endpoints\":{\r\n    \"https\":{\r\n      \"www\":{\r\n        \"https_valid\":true,\r\n        \"status\":200,\r\n        \"hsts\":true,\r\n        \"hsts_header\":\"max-age=3600;include_subdomains\",\r\n        \"redirect\":false,\r\n        \"redirect_to_external\":false,\r\n        \"redirect_to_https\":false,\r\n        \"live\":true\r\n      },\r\n      \"root\":{\r\n        \"https_valid\":true,\r\n        \"status\":301,\r\n        \"hsts\":false,\r\n        \"hsts_header\":null,\r\n        \"redirect\":true,\r\n        \"redirect_to\":\"https://www.whitehouse.gov/\",\r\n        \"redirect_to_external\":true,\r\n        \"redirect_to_https\":true,\r\n        \"live\":true\r\n      }\r\n    },\r\n    \"http\":{\r\n      \"www\":{\r\n        \"status\":302,\r\n        \"hsts\":false,\r\n        \"hsts_header\":null,\r\n        \"redirect\":true,\r\n        \"redirect_to\":\"https://www.whitehouse.gov/\",\r\n        \"redirect_to_external\":true,\r\n        \"redirect_to_https\":true,\r\n        \"live\":true\r\n      },\r\n      \"root\":{\r\n        \"status\":301,\r\n        \"hsts\":false,\r\n        \"hsts_header\":null,\r\n        \"redirect\":true,\r\n        \"redirect_to\":\"https://www.whitehouse.gov/\",\r\n        \"redirect_to_external\":true,\r\n        \"redirect_to_https\":true,\r\n        \"live\":true\r\n      }\r\n    }\r\n  },\r\n  \"live\":true,\r\n  \"broken_root\":false,\r\n  \"broken_www\":false,\r\n  \"enforce_https\":true,\r\n  \"redirect\":false,\r\n  \"hsts\":true,\r\n  \"hsts_entire_domain\":false,\r\n  \"hsts_entire_domain_preload\":false\r\n}\r\n```\r\n\r\nI'll annotate the rest inline -- looking for feedback here, and suggestions about how this could be better integrated into the library.","created_at":"2015-04-13T02:50:05Z","updated_at":"2015-04-13T02:57:34Z","closed_at":null,"merged_at":null,"merge_commit_sha":"d1997a65ad2f35cb5de858991882b0662e64e75c","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/pulls/24/commits","review_comments_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/pulls/24/comments","review_comment_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/pulls/comments{/number}","comments_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/issues/24/comments","statuses_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/statuses/18638991c14591df4fbebf7de41f819ccc07a301","head":{"label":"benbalter:light","ref":"light","sha":"18638991c14591df4fbebf7de41f819ccc07a301","user":{"login":"benbalter","id":282759,"avatar_url":"https://avatars.githubusercontent.com/u/282759?v=3","gravatar_id":"","url":"https://api.github.com/users/benbalter","html_url":"https://github.com/benbalter","followers_url":"https://api.github.com/users/benbalter/followers","following_url":"https://api.github.com/users/benbalter/following{/other_user}","gists_url":"https://api.github.com/users/benbalter/gists{/gist_id}","starred_url":"https://api.github.com/users/benbalter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benbalter/subscriptions","organizations_url":"https://api.github.com/users/benbalter/orgs","repos_url":"https://api.github.com/users/benbalter/repos","events_url":"https://api.github.com/users/benbalter/events{/privacy}","received_events_url":"https://api.github.com/users/benbalter/received_events","type":"User","site_admin":true},"repo":{"id":12851435,"name":"site-inspector-ruby","full_name":"benbalter/site-inspector-ruby","owner":{"login":"benbalter","id":282759,"avatar_url":"https://avatars.githubusercontent.com/u/282759?v=3","gravatar_id":"","url":"https://api.github.com/users/benbalter","html_url":"https://github.com/benbalter","followers_url":"https://api.github.com/users/benbalter/followers","following_url":"https://api.github.com/users/benbalter/following{/other_user}","gists_url":"https://api.github.com/users/benbalter/gists{/gist_id}","starred_url":"https://api.github.com/users/benbalter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benbalter/subscriptions","organizations_url":"https://api.github.com/users/benbalter/orgs","repos_url":"https://api.github.com/users/benbalter/repos","events_url":"https://api.github.com/users/benbalter/events{/privacy}","received_events_url":"https://api.github.com/users/benbalter/received_events","type":"User","site_admin":true},"private":false,"html_url":"https://github.com/benbalter/site-inspector-ruby","description":"Ruby Gem to sniff information about a domain's technology and capabilities.","fork":false,"url":"https://api.github.com/repos/benbalter/site-inspector-ruby","forks_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/forks","keys_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/keys{/key_id}","collaborators_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/teams","hooks_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/hooks","issue_events_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/issues/events{/number}","events_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/events","assignees_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/assignees{/user}","branches_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/branches{/branch}","tags_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/tags","blobs_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/git/refs{/sha}","trees_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/git/trees{/sha}","statuses_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/statuses/{sha}","languages_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/languages","stargazers_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/stargazers","contributors_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/contributors","subscribers_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/subscribers","subscription_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/subscription","commits_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/commits{/sha}","git_commits_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/git/commits{/sha}","comments_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/comments{/number}","issue_comment_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/issues/comments{/number}","contents_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/contents/{+path}","compare_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/compare/{base}...{head}","merges_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/merges","archive_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/downloads","issues_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/issues{/number}","pulls_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/pulls{/number}","milestones_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/milestones{/number}","notifications_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/labels{/name}","releases_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/releases{/id}","created_at":"2013-09-15T19:47:25Z","updated_at":"2015-04-12T17:26:56Z","pushed_at":"2015-04-13T02:50:05Z","git_url":"git://github.com/benbalter/site-inspector-ruby.git","ssh_url":"git@github.com:benbalter/site-inspector-ruby.git","clone_url":"https://github.com/benbalter/site-inspector-ruby.git","svn_url":"https://github.com/benbalter/site-inspector-ruby","homepage":"https://gov-inspector.herokuapp.com","size":1929,"stargazers_count":11,"watchers_count":11,"language":"Ruby","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":7,"mirror_url":null,"open_issues_count":5,"forks":7,"open_issues":5,"watchers":11,"default_branch":"master"}},"base":{"label":"benbalter:master","ref":"master","sha":"cc6c4c73dffa71de9e2645a0aaf30b83b0b1258d","user":{"login":"benbalter","id":282759,"avatar_url":"https://avatars.githubusercontent.com/u/282759?v=3","gravatar_id":"","url":"https://api.github.com/users/benbalter","html_url":"https://github.com/benbalter","followers_url":"https://api.github.com/users/benbalter/followers","following_url":"https://api.github.com/users/benbalter/following{/other_user}","gists_url":"https://api.github.com/users/benbalter/gists{/gist_id}","starred_url":"https://api.github.com/users/benbalter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benbalter/subscriptions","organizations_url":"https://api.github.com/users/benbalter/orgs","repos_url":"https://api.github.com/users/benbalter/repos","events_url":"https://api.github.com/users/benbalter/events{/privacy}","received_events_url":"https://api.github.com/users/benbalter/received_events","type":"User","site_admin":true},"repo":{"id":12851435,"name":"site-inspector-ruby","full_name":"benbalter/site-inspector-ruby","owner":{"login":"benbalter","id":282759,"avatar_url":"https://avatars.githubusercontent.com/u/282759?v=3","gravatar_id":"","url":"https://api.github.com/users/benbalter","html_url":"https://github.com/benbalter","followers_url":"https://api.github.com/users/benbalter/followers","following_url":"https://api.github.com/users/benbalter/following{/other_user}","gists_url":"https://api.github.com/users/benbalter/gists{/gist_id}","starred_url":"https://api.github.com/users/benbalter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benbalter/subscriptions","organizations_url":"https://api.github.com/users/benbalter/orgs","repos_url":"https://api.github.com/users/benbalter/repos","events_url":"https://api.github.com/users/benbalter/events{/privacy}","received_events_url":"https://api.github.com/users/benbalter/received_events","type":"User","site_admin":true},"private":false,"html_url":"https://github.com/benbalter/site-inspector-ruby","description":"Ruby Gem to sniff information about a domain's technology and capabilities.","fork":false,"url":"https://api.github.com/repos/benbalter/site-inspector-ruby","forks_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/forks","keys_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/keys{/key_id}","collaborators_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/teams","hooks_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/hooks","issue_events_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/issues/events{/number}","events_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/events","assignees_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/assignees{/user}","branches_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/branches{/branch}","tags_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/tags","blobs_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/git/refs{/sha}","trees_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/git/trees{/sha}","statuses_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/statuses/{sha}","languages_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/languages","stargazers_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/stargazers","contributors_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/contributors","subscribers_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/subscribers","subscription_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/subscription","commits_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/commits{/sha}","git_commits_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/git/commits{/sha}","comments_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/comments{/number}","issue_comment_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/issues/comments{/number}","contents_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/contents/{+path}","compare_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/compare/{base}...{head}","merges_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/merges","archive_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/downloads","issues_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/issues{/number}","pulls_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/pulls{/number}","milestones_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/milestones{/number}","notifications_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/labels{/name}","releases_url":"https://api.github.com/repos/benbalter/site-inspector-ruby/releases{/id}","created_at":"2013-09-15T19:47:25Z","updated_at":"2015-04-12T17:26:56Z","pushed_at":"2015-04-13T02:50:05Z","git_url":"git://github.com/benbalter/site-inspector-ruby.git","ssh_url":"git@github.com:benbalter/site-inspector-ruby.git","clone_url":"https://github.com/benbalter/site-inspector-ruby.git","svn_url":"https://github.com/benbalter/site-inspector-ruby","homepage":"https://gov-inspector.herokuapp.com","size":1929,"stargazers_count":11,"watchers_count":11,"language":"Ruby","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":7,"mirror_url":null,"open_issues_count":5,"forks":7,"open_issues":5,"watchers":11,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/benbalter/site-inspector-ruby/pulls/24"},"html":{"href":"https://github.com/benbalter/site-inspector-ruby/pull/24"},"issue":{"href":"https://api.github.com/repos/benbalter/site-inspector-ruby/issues/24"},"comments":{"href":"https://api.github.com/repos/benbalter/site-inspector-ruby/issues/24/comments"},"review_comments":{"href":"https://api.github.com/repos/benbalter/site-inspector-ruby/pulls/24/comments"},"review_comment":{"href":"https://api.github.com/repos/benbalter/site-inspector-ruby/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/benbalter/site-inspector-ruby/pulls/24/commits"},"statuses":{"href":"https://api.github.com/repos/benbalter/site-inspector-ruby/statuses/18638991c14591df4fbebf7de41f819ccc07a301"}}}}