{"action":"created","issue":{"url":"https://api.github.com/repos/18F/foia-hub/issues/530","labels_url":"https://api.github.com/repos/18F/foia-hub/issues/530/labels{/name}","comments_url":"https://api.github.com/repos/18F/foia-hub/issues/530/comments","events_url":"https://api.github.com/repos/18F/foia-hub/issues/530/events","html_url":"https://github.com/18F/foia-hub/issues/530","id":58535993,"number":530,"title":"Scan government sites for PDFs","user":{"login":"konklone","id":4592,"avatar_url":"https://avatars.githubusercontent.com/u/4592?v=3","gravatar_id":"","url":"https://api.github.com/users/konklone","html_url":"https://github.com/konklone","followers_url":"https://api.github.com/users/konklone/followers","following_url":"https://api.github.com/users/konklone/following{/other_user}","gists_url":"https://api.github.com/users/konklone/gists{/gist_id}","starred_url":"https://api.github.com/users/konklone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/konklone/subscriptions","organizations_url":"https://api.github.com/users/konklone/orgs","repos_url":"https://api.github.com/users/konklone/repos","events_url":"https://api.github.com/users/konklone/events{/privacy}","received_events_url":"https://api.github.com/users/konklone/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-02-23T02:31:59Z","updated_at":"2015-02-23T02:38:59Z","closed_at":null,"body":"US Open Data's [Let Me Get That Data For You](http://lmgtdfy.usopendata.org) is a very neat system, and is an [open source Django app](https://github.com/opendata/lmgtdfy/) that uses Celery background tasks to queue up a search over a website domain.\r\n\r\nThis uses the Bing API, and the [Bing API costs money](http://datamarket.azure.com/dataset/bing/search) (there is a free tier). It could be used to, at least, do a one-time scan of some sites to gather leads and find documents.\r\n\r\nWe could also explore adapting an actual crawler to hunt down PDF links. `wget` is built for that sort of thing, and I've been using [`mixed-content-scan`](https://github.com/bramus/mixed-content-scan), which recursively crawls a domain for mixed content issues. for some separate work.\r\n\r\ncc @knowtheory @waldoj"},"comment":{"url":"https://api.github.com/repos/18F/foia-hub/issues/comments/75482073","html_url":"https://github.com/18F/foia-hub/issues/530#issuecomment-75482073","issue_url":"https://api.github.com/repos/18F/foia-hub/issues/530","id":75482073,"user":{"login":"waldoj","id":656758,"avatar_url":"https://avatars.githubusercontent.com/u/656758?v=3","gravatar_id":"","url":"https://api.github.com/users/waldoj","html_url":"https://github.com/waldoj","followers_url":"https://api.github.com/users/waldoj/followers","following_url":"https://api.github.com/users/waldoj/following{/other_user}","gists_url":"https://api.github.com/users/waldoj/gists{/gist_id}","starred_url":"https://api.github.com/users/waldoj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/waldoj/subscriptions","organizations_url":"https://api.github.com/users/waldoj/orgs","repos_url":"https://api.github.com/users/waldoj/repos","events_url":"https://api.github.com/users/waldoj/events{/privacy}","received_events_url":"https://api.github.com/users/waldoj/received_events","type":"User","site_admin":false},"created_at":"2015-02-23T02:38:59Z","updated_at":"2015-02-23T02:38:59Z","body":"You might check Common Crawl. We had to reject it for LMGTDFY you because it doesn't include CSV or XLS files (I stopped researching at that point), but perhaps it includes PDFs."}}