{"action":"created","comment":{"url":"https://api.github.com/repos/Yelp/osxcollector/pulls/comments/22401350","id":22401350,"diff_hunk":"@@ -2,7 +2,7 @@\n \n # -*- coding: utf-8 -*-\n #\n-# LookupDomainsFilter uses VirusTotal to lookup the values in 'md5' and add 'osxcollector_vthash' key.\n+# LookupHashesFilter uses VirusTotal to lookup the values in 'sha2' and add 'osxcollector_vthash' key.","path":"osxcollector/output_filters/virustotal/lookup_hashes.py","position":5,"original_position":5,"commit_id":"4eca164c397c739b0bc8125f17f25177b573c7a8","original_commit_id":"8ac57bc3ef04ab7bb9798b236f825e91068f6869","user":{"login":"ivanlei","id":1002798,"avatar_url":"https://avatars.githubusercontent.com/u/1002798?v=3","gravatar_id":"","url":"https://api.github.com/users/ivanlei","html_url":"https://github.com/ivanlei","followers_url":"https://api.github.com/users/ivanlei/followers","following_url":"https://api.github.com/users/ivanlei/following{/other_user}","gists_url":"https://api.github.com/users/ivanlei/gists{/gist_id}","starred_url":"https://api.github.com/users/ivanlei/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ivanlei/subscriptions","organizations_url":"https://api.github.com/users/ivanlei/orgs","repos_url":"https://api.github.com/users/ivanlei/repos","events_url":"https://api.github.com/users/ivanlei/events{/privacy}","received_events_url":"https://api.github.com/users/ivanlei/received_events","type":"User","site_admin":false},"body":"ShadowServer just doesn't offer SHA2.\r\nThe goal is to use the \"best\" hash a service will offer.","created_at":"2015-01-01T19:40:22Z","updated_at":"2015-01-01T19:40:22Z","html_url":"https://github.com/Yelp/osxcollector/pull/42#discussion_r22401350","pull_request_url":"https://api.github.com/repos/Yelp/osxcollector/pulls/42","_links":{"self":{"href":"https://api.github.com/repos/Yelp/osxcollector/pulls/comments/22401350"},"html":{"href":"https://github.com/Yelp/osxcollector/pull/42#discussion_r22401350"},"pull_request":{"href":"https://api.github.com/repos/Yelp/osxcollector/pulls/42"}}},"pull_request":{"url":"https://api.github.com/repos/Yelp/osxcollector/pulls/42","id":26708053,"html_url":"https://github.com/Yelp/osxcollector/pull/42","diff_url":"https://github.com/Yelp/osxcollector/pull/42.diff","patch_url":"https://github.com/Yelp/osxcollector/pull/42.patch","issue_url":"https://api.github.com/repos/Yelp/osxcollector/issues/42","number":42,"state":"closed","locked":false,"title":"Grequests rework","user":{"login":"ivanlei","id":1002798,"avatar_url":"https://avatars.githubusercontent.com/u/1002798?v=3","gravatar_id":"","url":"https://api.github.com/users/ivanlei","html_url":"https://github.com/ivanlei","followers_url":"https://api.github.com/users/ivanlei/followers","following_url":"https://api.github.com/users/ivanlei/following{/other_user}","gists_url":"https://api.github.com/users/ivanlei/gists{/gist_id}","starred_url":"https://api.github.com/users/ivanlei/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ivanlei/subscriptions","organizations_url":"https://api.github.com/users/ivanlei/orgs","repos_url":"https://api.github.com/users/ivanlei/repos","events_url":"https://api.github.com/users/ivanlei/events{/privacy}","received_events_url":"https://api.github.com/users/ivanlei/received_events","type":"User","site_admin":false},"body":"* **Create `osxcollector.output_files.util`**\r\nMove a bunch of utility functions that have been living alongside a semi-random OutputFilter into their own module.\r\n\r\n* **Create `osxcollector.output_files.util.http`**\r\nA submodule that deals with issuing multiple requests with grequests. This puts all the grequests code into one file and is easy to use in new filters that want to call an API.\r\n\r\n* **Add ShadowServer hash lookup filter**\r\n`osxcollector.output_filters.shadowserver.LookupHashesFilter` uses an API to identify hashes of known good files - parts of the OS and stuff. Use it to effectively mark OS files as non-suspicious. This was a great way to test the new http handling code.\r\nVirusTotal hash lookup got 25x faster. This was enough to let `AnalyzeFilter` do lookups on all hashes all the time.\r\n\r\n* **More attribution in the blacklist and related filters**\r\nRather than adding `{'osxcollector_blacklist': ['hashes']}` add `{'osxcollector_blacklist': {'hashes': ['abcdefg1234']}}`\r\nRather than adding `{'osxcollector_related': ['files']}` add `{'osxcollector_related': {'files': ['evil.dmg']}}`\r\n\r\n* **Remove the concept of \"suspicious\" from `ThreatFeedFilter`**\r\nIt was confusing and mostly just led to a lot of false positives.\r\n\r\n* **More tests**\r\nIncluding a good test driver for testing OutputFilters.\r\n\r\n* **Lots of comment improvements**\r\nYeah, just improve em.\r\n\r\n* **Handle strange encodings in domains**\r\nThis resolves #34. Basically nothing magic just tried a bunch of different things with some good test cases in place. \r\n\r\n* **Do better at finding related files**\r\nThis resolves #40 by not using a regex. There was never a need to use a regex anyhow.\r\nAlso, RelatedFilesFilter takes a `when` function like other filters that lets it do a better job of finding more related files. For example, if the `when` function is \r\n```\r\ndef when(blob):\r\n    return 'osxcollector_vthash' in blob\r\n```\r\nThen the RelatedFilesFilter will take the filename of any known malware and use it to find stuff related to that file - like the download that caused it maybe.","created_at":"2014-12-30T22:56:29Z","updated_at":"2015-01-01T19:40:22Z","closed_at":"2014-12-31T02:57:46Z","merged_at":"2014-12-31T02:57:46Z","merge_commit_sha":"9d663320d9c610a092a86ef2924a79dec246217e","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/Yelp/osxcollector/pulls/42/commits","review_comments_url":"https://api.github.com/repos/Yelp/osxcollector/pulls/42/comments","review_comment_url":"https://api.github.com/repos/Yelp/osxcollector/pulls/comments/{number}","comments_url":"https://api.github.com/repos/Yelp/osxcollector/issues/42/comments","statuses_url":"https://api.github.com/repos/Yelp/osxcollector/statuses/4eca164c397c739b0bc8125f17f25177b573c7a8","head":{"label":"Yelp:grequests_rework","ref":"grequests_rework","sha":"4eca164c397c739b0bc8125f17f25177b573c7a8","user":{"login":"Yelp","id":49071,"avatar_url":"https://avatars.githubusercontent.com/u/49071?v=3","gravatar_id":"","url":"https://api.github.com/users/Yelp","html_url":"https://github.com/Yelp","followers_url":"https://api.github.com/users/Yelp/followers","following_url":"https://api.github.com/users/Yelp/following{/other_user}","gists_url":"https://api.github.com/users/Yelp/gists{/gist_id}","starred_url":"https://api.github.com/users/Yelp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yelp/subscriptions","organizations_url":"https://api.github.com/users/Yelp/orgs","repos_url":"https://api.github.com/users/Yelp/repos","events_url":"https://api.github.com/users/Yelp/events{/privacy}","received_events_url":"https://api.github.com/users/Yelp/received_events","type":"Organization","site_admin":false},"repo":{"id":22615096,"name":"osxcollector","full_name":"Yelp/osxcollector","owner":{"login":"Yelp","id":49071,"avatar_url":"https://avatars.githubusercontent.com/u/49071?v=3","gravatar_id":"","url":"https://api.github.com/users/Yelp","html_url":"https://github.com/Yelp","followers_url":"https://api.github.com/users/Yelp/followers","following_url":"https://api.github.com/users/Yelp/following{/other_user}","gists_url":"https://api.github.com/users/Yelp/gists{/gist_id}","starred_url":"https://api.github.com/users/Yelp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yelp/subscriptions","organizations_url":"https://api.github.com/users/Yelp/orgs","repos_url":"https://api.github.com/users/Yelp/repos","events_url":"https://api.github.com/users/Yelp/events{/privacy}","received_events_url":"https://api.github.com/users/Yelp/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Yelp/osxcollector","description":"A forensic evidence collection & analysis toolkit for OSX.","fork":false,"url":"https://api.github.com/repos/Yelp/osxcollector","forks_url":"https://api.github.com/repos/Yelp/osxcollector/forks","keys_url":"https://api.github.com/repos/Yelp/osxcollector/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Yelp/osxcollector/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Yelp/osxcollector/teams","hooks_url":"https://api.github.com/repos/Yelp/osxcollector/hooks","issue_events_url":"https://api.github.com/repos/Yelp/osxcollector/issues/events{/number}","events_url":"https://api.github.com/repos/Yelp/osxcollector/events","assignees_url":"https://api.github.com/repos/Yelp/osxcollector/assignees{/user}","branches_url":"https://api.github.com/repos/Yelp/osxcollector/branches{/branch}","tags_url":"https://api.github.com/repos/Yelp/osxcollector/tags","blobs_url":"https://api.github.com/repos/Yelp/osxcollector/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Yelp/osxcollector/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Yelp/osxcollector/git/refs{/sha}","trees_url":"https://api.github.com/repos/Yelp/osxcollector/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Yelp/osxcollector/statuses/{sha}","languages_url":"https://api.github.com/repos/Yelp/osxcollector/languages","stargazers_url":"https://api.github.com/repos/Yelp/osxcollector/stargazers","contributors_url":"https://api.github.com/repos/Yelp/osxcollector/contributors","subscribers_url":"https://api.github.com/repos/Yelp/osxcollector/subscribers","subscription_url":"https://api.github.com/repos/Yelp/osxcollector/subscription","commits_url":"https://api.github.com/repos/Yelp/osxcollector/commits{/sha}","git_commits_url":"https://api.github.com/repos/Yelp/osxcollector/git/commits{/sha}","comments_url":"https://api.github.com/repos/Yelp/osxcollector/comments{/number}","issue_comment_url":"https://api.github.com/repos/Yelp/osxcollector/issues/comments/{number}","contents_url":"https://api.github.com/repos/Yelp/osxcollector/contents/{+path}","compare_url":"https://api.github.com/repos/Yelp/osxcollector/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Yelp/osxcollector/merges","archive_url":"https://api.github.com/repos/Yelp/osxcollector/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Yelp/osxcollector/downloads","issues_url":"https://api.github.com/repos/Yelp/osxcollector/issues{/number}","pulls_url":"https://api.github.com/repos/Yelp/osxcollector/pulls{/number}","milestones_url":"https://api.github.com/repos/Yelp/osxcollector/milestones{/number}","notifications_url":"https://api.github.com/repos/Yelp/osxcollector/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Yelp/osxcollector/labels{/name}","releases_url":"https://api.github.com/repos/Yelp/osxcollector/releases{/id}","created_at":"2014-08-04T18:25:29Z","updated_at":"2014-12-31T08:05:06Z","pushed_at":"2014-12-31T19:26:38Z","git_url":"git://github.com/Yelp/osxcollector.git","ssh_url":"git@github.com:Yelp/osxcollector.git","clone_url":"https://github.com/Yelp/osxcollector.git","svn_url":"https://github.com/Yelp/osxcollector","homepage":"http://yelp.github.io/osxcollector","size":1111,"stargazers_count":303,"watchers_count":303,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":8,"mirror_url":null,"open_issues_count":4,"forks":8,"open_issues":4,"watchers":303,"default_branch":"master"}},"base":{"label":"Yelp:master","ref":"master","sha":"27d37bdd351161cb1c3ef662864cddf61ab10f7f","user":{"login":"Yelp","id":49071,"avatar_url":"https://avatars.githubusercontent.com/u/49071?v=3","gravatar_id":"","url":"https://api.github.com/users/Yelp","html_url":"https://github.com/Yelp","followers_url":"https://api.github.com/users/Yelp/followers","following_url":"https://api.github.com/users/Yelp/following{/other_user}","gists_url":"https://api.github.com/users/Yelp/gists{/gist_id}","starred_url":"https://api.github.com/users/Yelp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yelp/subscriptions","organizations_url":"https://api.github.com/users/Yelp/orgs","repos_url":"https://api.github.com/users/Yelp/repos","events_url":"https://api.github.com/users/Yelp/events{/privacy}","received_events_url":"https://api.github.com/users/Yelp/received_events","type":"Organization","site_admin":false},"repo":{"id":22615096,"name":"osxcollector","full_name":"Yelp/osxcollector","owner":{"login":"Yelp","id":49071,"avatar_url":"https://avatars.githubusercontent.com/u/49071?v=3","gravatar_id":"","url":"https://api.github.com/users/Yelp","html_url":"https://github.com/Yelp","followers_url":"https://api.github.com/users/Yelp/followers","following_url":"https://api.github.com/users/Yelp/following{/other_user}","gists_url":"https://api.github.com/users/Yelp/gists{/gist_id}","starred_url":"https://api.github.com/users/Yelp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yelp/subscriptions","organizations_url":"https://api.github.com/users/Yelp/orgs","repos_url":"https://api.github.com/users/Yelp/repos","events_url":"https://api.github.com/users/Yelp/events{/privacy}","received_events_url":"https://api.github.com/users/Yelp/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Yelp/osxcollector","description":"A forensic evidence collection & analysis toolkit for OSX.","fork":false,"url":"https://api.github.com/repos/Yelp/osxcollector","forks_url":"https://api.github.com/repos/Yelp/osxcollector/forks","keys_url":"https://api.github.com/repos/Yelp/osxcollector/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Yelp/osxcollector/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Yelp/osxcollector/teams","hooks_url":"https://api.github.com/repos/Yelp/osxcollector/hooks","issue_events_url":"https://api.github.com/repos/Yelp/osxcollector/issues/events{/number}","events_url":"https://api.github.com/repos/Yelp/osxcollector/events","assignees_url":"https://api.github.com/repos/Yelp/osxcollector/assignees{/user}","branches_url":"https://api.github.com/repos/Yelp/osxcollector/branches{/branch}","tags_url":"https://api.github.com/repos/Yelp/osxcollector/tags","blobs_url":"https://api.github.com/repos/Yelp/osxcollector/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Yelp/osxcollector/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Yelp/osxcollector/git/refs{/sha}","trees_url":"https://api.github.com/repos/Yelp/osxcollector/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Yelp/osxcollector/statuses/{sha}","languages_url":"https://api.github.com/repos/Yelp/osxcollector/languages","stargazers_url":"https://api.github.com/repos/Yelp/osxcollector/stargazers","contributors_url":"https://api.github.com/repos/Yelp/osxcollector/contributors","subscribers_url":"https://api.github.com/repos/Yelp/osxcollector/subscribers","subscription_url":"https://api.github.com/repos/Yelp/osxcollector/subscription","commits_url":"https://api.github.com/repos/Yelp/osxcollector/commits{/sha}","git_commits_url":"https://api.github.com/repos/Yelp/osxcollector/git/commits{/sha}","comments_url":"https://api.github.com/repos/Yelp/osxcollector/comments{/number}","issue_comment_url":"https://api.github.com/repos/Yelp/osxcollector/issues/comments/{number}","contents_url":"https://api.github.com/repos/Yelp/osxcollector/contents/{+path}","compare_url":"https://api.github.com/repos/Yelp/osxcollector/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Yelp/osxcollector/merges","archive_url":"https://api.github.com/repos/Yelp/osxcollector/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Yelp/osxcollector/downloads","issues_url":"https://api.github.com/repos/Yelp/osxcollector/issues{/number}","pulls_url":"https://api.github.com/repos/Yelp/osxcollector/pulls{/number}","milestones_url":"https://api.github.com/repos/Yelp/osxcollector/milestones{/number}","notifications_url":"https://api.github.com/repos/Yelp/osxcollector/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Yelp/osxcollector/labels{/name}","releases_url":"https://api.github.com/repos/Yelp/osxcollector/releases{/id}","created_at":"2014-08-04T18:25:29Z","updated_at":"2014-12-31T08:05:06Z","pushed_at":"2014-12-31T19:26:38Z","git_url":"git://github.com/Yelp/osxcollector.git","ssh_url":"git@github.com:Yelp/osxcollector.git","clone_url":"https://github.com/Yelp/osxcollector.git","svn_url":"https://github.com/Yelp/osxcollector","homepage":"http://yelp.github.io/osxcollector","size":1111,"stargazers_count":303,"watchers_count":303,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":8,"mirror_url":null,"open_issues_count":4,"forks":8,"open_issues":4,"watchers":303,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/Yelp/osxcollector/pulls/42"},"html":{"href":"https://github.com/Yelp/osxcollector/pull/42"},"issue":{"href":"https://api.github.com/repos/Yelp/osxcollector/issues/42"},"comments":{"href":"https://api.github.com/repos/Yelp/osxcollector/issues/42/comments"},"review_comments":{"href":"https://api.github.com/repos/Yelp/osxcollector/pulls/42/comments"},"review_comment":{"href":"https://api.github.com/repos/Yelp/osxcollector/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/Yelp/osxcollector/pulls/42/commits"},"statuses":{"href":"https://api.github.com/repos/Yelp/osxcollector/statuses/4eca164c397c739b0bc8125f17f25177b573c7a8"}}}}