{"action":"created","issue":{"url":"https://api.github.com/repos/rapid7/metasploit-framework/issues/5333","labels_url":"https://api.github.com/repos/rapid7/metasploit-framework/issues/5333/labels{/name}","comments_url":"https://api.github.com/repos/rapid7/metasploit-framework/issues/5333/comments","events_url":"https://api.github.com/repos/rapid7/metasploit-framework/issues/5333/events","html_url":"https://github.com/rapid7/metasploit-framework/pull/5333","id":75309275,"number":5333,"title":"HTML Title Grabber","user":{"login":"stufus","id":12296344,"avatar_url":"https://avatars.githubusercontent.com/u/12296344?v=3","gravatar_id":"","url":"https://api.github.com/users/stufus","html_url":"https://github.com/stufus","followers_url":"https://api.github.com/users/stufus/followers","following_url":"https://api.github.com/users/stufus/following{/other_user}","gists_url":"https://api.github.com/users/stufus/gists{/gist_id}","starred_url":"https://api.github.com/users/stufus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stufus/subscriptions","organizations_url":"https://api.github.com/users/stufus/orgs","repos_url":"https://api.github.com/users/stufus/repos","events_url":"https://api.github.com/users/stufus/events{/privacy}","received_events_url":"https://api.github.com/users/stufus/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rapid7/metasploit-framework/labels/module","name":"module","color":"eb6420"}],"state":"closed","locked":false,"assignee":{"login":"wchen-r7","id":1170914,"avatar_url":"https://avatars.githubusercontent.com/u/1170914?v=3","gravatar_id":"","url":"https://api.github.com/users/wchen-r7","html_url":"https://github.com/wchen-r7","followers_url":"https://api.github.com/users/wchen-r7/followers","following_url":"https://api.github.com/users/wchen-r7/following{/other_user}","gists_url":"https://api.github.com/users/wchen-r7/gists{/gist_id}","starred_url":"https://api.github.com/users/wchen-r7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wchen-r7/subscriptions","organizations_url":"https://api.github.com/users/wchen-r7/orgs","repos_url":"https://api.github.com/users/wchen-r7/repos","events_url":"https://api.github.com/users/wchen-r7/events{/privacy}","received_events_url":"https://api.github.com/users/wchen-r7/received_events","type":"User","site_admin":false},"milestone":null,"comments":7,"created_at":"2015-05-11T18:10:54Z","updated_at":"2015-05-12T16:31:47Z","closed_at":"2015-05-12T16:19:31Z","pull_request":{"url":"https://api.github.com/repos/rapid7/metasploit-framework/pulls/5333","html_url":"https://github.com/rapid7/metasploit-framework/pull/5333","diff_url":"https://github.com/rapid7/metasploit-framework/pull/5333.diff","patch_url":"https://github.com/rapid7/metasploit-framework/pull/5333.patch"},"body":"# Overview\r\n\r\nOn most penetration tests (particularly internal tests on large networks), it is useful to be able to connect to a large number of webservers, obtain basic page information (server header and title) so that any interesting or anomalous servers can be easily identified.\r\n\r\nThis can be achieved using some third party tools, and to an extent can be achieved using scraper. However, this PR is for an auxiliary module which will connect to 'RHOSTS:RPORT', perform a non-greedy match for the first `<title>` tag (which will return the first title that appears on the page), the return HTTP code, the Location header (if present) and the Server header (if present).  It will optionally present this on the console if `SHOW_TITLES` is set and write to notes (http.title) if `STORE_NOTES` is set. It can be made as quiet as possible to allow background execution without filling up the console with output. The benefit of using notes can be easily seen using commands such as\r\n\r\n`notes -t http.title -S ':port=>443, :server=>\"Apache.*?\"' -R` \r\n\r\nand then launching another module specifically aimed at mod_ssl etc.\r\n\r\nNote that this is first and foremost a title grabber, not a fingerprinting module. Therefore, no output will be generated if a title cannot be obtained, even if the web server is still alive. It should be used to identify interesting hosts based on HTML title, not to identify webservers which are responding to HTTP.\r\n\r\n# Output Format\r\n\r\n## Console\r\n\r\nIf `SHOW_TITLES` is set, the code will generate console output in the format shown in the example below:\r\n\r\n`[*] [46.228.47.114:80] [C:200] [R:] [S:ATS] TITLE`\r\n\r\nFormat | Example | Description\r\n------------ | ------------- | -------------\r\nC | [C:200] | The HTTP status code from the server\r\nR | [R:http://www.mwrinfosecurity.com/] | The contents of the Location header, if present. This is useful to see at a glance if a GET request redirects you to another URL.\r\nS | [S:ATS] | The contents of the Server header, if present.\r\n(Rest of string) | TITLE | The unescaped contents of the first HTML TITLE tag\r\n\r\n## Notes\r\n\r\nIf `STORE_NOTES` is set, the code will inject a note into the notes database to cover each of the HTML titles that are obtained, as shown in the example below:\r\n\r\n`[*] Time: 2015-05-11 17:25:21 UTC Note: host=193.0.255.237 type=http.title data={:code=>200, :port=>80, :server=>\"gws\", :title=>\"Google\", :redirect=>nil}`\r\n\r\nThe format is relatively self-explanatory. \r\n\r\n# Verification\r\n\r\n- [x] Able to grab HTML titles on tcp/80 when provided with domain names\r\n\r\n`msf  auxiliary(title) > show options`\r\n```\r\nModule options (auxiliary/scanner/http/title):\r\n\r\n   Name         Current Setting                                         Required  Description\r\n   ----         ---------------                                         --------  -----------\r\n   Proxies                                                              no        A proxy chain of format type:host:port[,type:host:port][...]\r\n   RHOSTS       www.mwrinfosecurity.com www.yahoo.com www.google.co.uk  yes       The target address range or CIDR identifier\r\n   RPORT        80                                                      yes       The target port\r\n   SHOW_ERRORS  true                                                    yes       Show error messages relating to grabbing titles on the console\r\n   SHOW_TITLES  true                                                    yes       Show the titles on the console as they are grabbed\r\n   STORE_NOTES  true                                                    yes       Store the captured information in notes. Use \"notes -t http.title\" to view\r\n   THREADS      3                                                       yes       The number of concurrent threads\r\n```\r\n`msf  auxiliary(title) > exploit`\r\n```\r\n[*] [5.79.13.16:80] [C:301] [R:http://www.mwrinfosecurity.com/] [S:Apache] 301 Moved Permanently\r\n[*] [46.228.47.114:80] [C:404] [R:] [S:ATS] Yahoo\r\n[*] [46.228.47.115:80] [C:404] [R:] [S:ATS] Yahoo\r\n[*] Scanned  3 of 20 hosts (15% complete)\r\n[*] Scanned  4 of 20 hosts (20% complete)\r\n[*] [193.0.255.241:80] [C:200] [R:] [S:gws] Google\r\n[*] [193.0.255.222:80] [C:200] [R:] [S:gws] Google\r\n[*] [193.0.255.249:80] [C:200] [R:] [S:gws] Google\r\n[*] Scanned  6 of 20 hosts (30% complete)\r\n[*] [193.0.255.236:80] [C:200] [R:] [S:gws] Google\r\n[*] [193.0.255.221:80] [C:200] [R:] [S:gws] Google\r\n[*] [193.0.255.237:80] [C:200] [R:] [S:gws] Google\r\n[*] Scanned  8 of 20 hosts (40% complete)\r\n[*] Scanned 10 of 20 hosts (50% complete)\r\n[*] [193.0.255.234:80] [C:200] [R:] [S:gws] Google\r\n[*] [193.0.255.230:80] [C:200] [R:] [S:gws] Google\r\n[*] [193.0.255.245:80] [C:200] [R:] [S:gws] Google\r\n[*] [193.0.255.251:80] [C:200] [R:] [S:gws] Google\r\n[*] Scanned 14 of 20 hosts (70% complete)\r\n[*] [193.0.255.226:80] [C:200] [R:] [S:gws] Google\r\n[*] [193.0.255.211:80] [C:200] [R:] [S:gws] Google\r\n[*] [193.0.255.215:80] [C:200] [R:] [S:gws] Google\r\n[*] Scanned 15 of 20 hosts (75% complete)\r\n[*] Scanned 16 of 20 hosts (80% complete)\r\n[*] [193.0.255.219:80] [C:200] [R:] [S:gws] Google\r\n[*] Scanned 19 of 20 hosts (95% complete)\r\n[*] [193.0.255.207:80] [C:200] [R:] [S:gws] Google\r\n[*] Scanned 20 of 20 hosts (100% complete)\r\n[*] Auxiliary module execution completed\r\n```\r\n- [x] Notes from the above correctly stored in the database\r\n\r\n`msf  auxiliary(title) > notes -t http.title`\r\n\r\n```\r\n[*] Time: 2015-05-11 17:25:21 UTC Note: host=5.79.13.16 type=http.title data={:code=>301, :port=>80, :server=>\"Apache\", :title=>\"301 Moved Permanently\", :redirect=>\"http://www.mwrinfosecurity.com/\"}\r\n[*] Time: 2015-05-11 17:25:21 UTC Note: host=193.0.255.241 type=http.title data={:code=>200, :port=>80, :server=>\"gws\", :title=>\"Google\", :redirect=>nil}\r\n[*] Time: 2015-05-11 17:25:21 UTC Note: host=193.0.255.249 type=http.title data={:code=>200, :port=>80, :server=>\"gws\", :title=>\"Google\", :redirect=>nil}\r\n[*] Time: 2015-05-11 17:25:21 UTC Note: host=193.0.255.236 type=http.title data={:code=>200, :port=>80, :server=>\"gws\", :title=>\"Google\", :redirect=>nil}\r\n[*] Time: 2015-05-11 17:25:21 UTC Note: host=193.0.255.237 type=http.title data={:code=>200, :port=>80, :server=>\"gws\", :title=>\"Google\", :redirect=>nil}\r\n[*] Time: 2015-05-11 17:25:21 UTC Note: host=193.0.255.234 type=http.title data={:code=>200, :port=>80, :server=>\"gws\", :title=>\"Google\", :redirect=>nil}\r\n[*] Time: 2015-05-11 17:25:22 UTC Note: host=193.0.255.230 type=http.title data={:code=>200, :port=>80, :server=>\"gws\", :title=>\"Google\", :redirect=>nil}\r\n[*] Time: 2015-05-11 17:25:22 UTC Note: host=193.0.255.245 type=http.title data={:code=>200, :port=>80, :server=>\"gws\", :title=>\"Google\", :redirect=>nil}\r\n[*] Time: 2015-05-11 17:25:22 UTC Note: host=193.0.255.251 type=http.title data={:code=>200, :port=>80, :server=>\"gws\", :title=>\"Google\", :redirect=>nil}\r\n[*] Time: 2015-05-11 17:25:22 UTC Note: host=193.0.255.226 type=http.title data={:code=>200, :port=>80, :server=>\"gws\", :title=>\"Google\", :redirect=>nil}\r\n[*] Time: 2015-05-11 17:25:22 UTC Note: host=193.0.255.215 type=http.title data={:code=>200, :port=>80, :server=>\"gws\", :title=>\"Google\", :redirect=>nil}\r\n[*] Time: 2015-05-11 17:25:22 UTC Note: host=193.0.255.219 type=http.title data={:code=>200, :port=>80, :server=>\"gws\", :title=>\"Google\", :redirect=>nil}\r\n[*] Time: 2015-05-11 17:25:22 UTC Note: host=193.0.255.207 type=http.title data={:code=>200, :port=>80, :server=>\"gws\", :title=>\"Google\", :redirect=>nil}\r\n[*] Time: 2015-05-11 17:25:21 UTC Note: host=46.228.47.114 type=http.title data={:code=>404, :port=>80, :server=>\"ATS\", :title=>\"Yahoo\", :redirect=>nil}\r\n[*] Time: 2015-05-11 17:25:21 UTC Note: host=193.0.255.222 type=http.title data={:code=>200, :port=>80, :server=>\"gws\", :title=>\"Google\", :redirect=>nil}\r\n[*] Time: 2015-05-11 17:25:21 UTC Note: host=193.0.255.221 type=http.title data={:code=>200, :port=>80, :server=>\"gws\", :title=>\"Google\", :redirect=>nil}\r\n[*] Time: 2015-05-11 17:25:22 UTC Note: host=193.0.255.211 type=http.title data={:code=>200, :port=>80, :server=>\"gws\", :title=>\"Google\", :redirect=>nil}\r\n[*] Time: 2015-05-11 17:25:21 UTC Note: host=46.228.47.115 type=http.title data={:code=>404, :port=>80, :server=>\"ATS\", :title=>\"Yahoo\", :redirect=>nil}\r\n```\r\n\r\n- [x] Able to grab HTML titles on tcp/443 when provided with IP addresses\r\n\r\n`msf  auxiliary(title) > show options`\r\n\r\n```\r\nModule options (auxiliary/scanner/http/title):\r\n\r\n   Name         Current Setting           Required  Description\r\n   ----         ---------------           --------  -----------\r\n   Proxies                                no        A proxy chain of format type:host:port[,type:host:port][...]\r\n   RHOSTS       193.0.255.236 5.79.13.16  yes       The target address range or CIDR identifier\r\n   RPORT        443                       yes       The target port\r\n   SHOW_ERRORS  true                      yes       Show error messages relating to grabbing titles on the console\r\n   SHOW_TITLES  true                      yes       Show the titles on the console as they are grabbed\r\n   STORE_NOTES  true                      yes       Store the captured information in notes. Use \"notes -t http.title\" to view\r\n   THREADS      3                         yes       The number of concurrent threads\r\n```\r\n`msf  auxiliary(title) > exploit`\r\n```\r\n[*] [5.79.13.16:443] [C:200] [R:] [S:Apache] MWR InfoSecurity\r\n[*] [193.0.255.236:443] [C:200] [R:] [S:gws] Google\r\n[*] Scanned 2 of 2 hosts (100% complete)\r\n[*] Auxiliary module execution completed\r\n```\r\n\r\n- [x] Notes stored correctly\r\n\r\n`msf  auxiliary(title) > notes -t http.title -S ':port=>443'`\r\n```\r\n[*] Time: 2015-05-11 17:25:21 UTC Note: host=5.79.13.16 type=http.title data={:code=>200, :port=>443, :server=>\"Apache\", :title=>\"MWR InfoSecurity\", :redirect=>nil}\r\n[*] Time: 2015-05-11 17:25:21 UTC Note: host=193.0.255.236 type=http.title data={:code=>200, :port=>443, :server=>\"gws\", :title=>\"Google\", :redirect=>nil}\r\n```\r\n\r\n# Support\r\n\r\nThanks to @Meatballs1 and @hmoore-r7 for your support :smile: "},"comment":{"url":"https://api.github.com/repos/rapid7/metasploit-framework/issues/comments/101340912","html_url":"https://github.com/rapid7/metasploit-framework/pull/5333#issuecomment-101340912","issue_url":"https://api.github.com/repos/rapid7/metasploit-framework/issues/5333","id":101340912,"user":{"login":"void-in","id":1746697,"avatar_url":"https://avatars.githubusercontent.com/u/1746697?v=3","gravatar_id":"","url":"https://api.github.com/users/void-in","html_url":"https://github.com/void-in","followers_url":"https://api.github.com/users/void-in/followers","following_url":"https://api.github.com/users/void-in/following{/other_user}","gists_url":"https://api.github.com/users/void-in/gists{/gist_id}","starred_url":"https://api.github.com/users/void-in/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/void-in/subscriptions","organizations_url":"https://api.github.com/users/void-in/orgs","repos_url":"https://api.github.com/users/void-in/repos","events_url":"https://api.github.com/users/void-in/events{/privacy}","received_events_url":"https://api.github.com/users/void-in/received_events","type":"User","site_admin":false},"created_at":"2015-05-12T16:31:47Z","updated_at":"2015-05-12T16:31:47Z","body":"@wchen-r7 No issues. Not that a big deal. Will generate a PR for this. "}}