{"action":"created","issue":{"url":"https://api.github.com/repos/HackJersey/crime/issues/3","labels_url":"https://api.github.com/repos/HackJersey/crime/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/HackJersey/crime/issues/3/comments","events_url":"https://api.github.com/repos/HackJersey/crime/issues/3/events","html_url":"https://github.com/HackJersey/crime/issues/3","id":65324888,"number":3,"title":"Make a functioning scraper","user":{"login":"tommeagher","id":713778,"avatar_url":"https://avatars.githubusercontent.com/u/713778?v=3","gravatar_id":"","url":"https://api.github.com/users/tommeagher","html_url":"https://github.com/tommeagher","followers_url":"https://api.github.com/users/tommeagher/followers","following_url":"https://api.github.com/users/tommeagher/following{/other_user}","gists_url":"https://api.github.com/users/tommeagher/gists{/gist_id}","starred_url":"https://api.github.com/users/tommeagher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tommeagher/subscriptions","organizations_url":"https://api.github.com/users/tommeagher/orgs","repos_url":"https://api.github.com/users/tommeagher/repos","events_url":"https://api.github.com/users/tommeagher/events{/privacy}","received_events_url":"https://api.github.com/users/tommeagher/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-03-30T23:10:14Z","updated_at":"2015-04-08T18:37:11Z","closed_at":null,"body":"First, let's get a program together to scrape and parse the monthly UCR PDFs. "},"comment":{"url":"https://api.github.com/repos/HackJersey/crime/issues/comments/90998736","html_url":"https://github.com/HackJersey/crime/issues/3#issuecomment-90998736","issue_url":"https://api.github.com/repos/HackJersey/crime/issues/3","id":90998736,"user":{"login":"tommeagher","id":713778,"avatar_url":"https://avatars.githubusercontent.com/u/713778?v=3","gravatar_id":"","url":"https://api.github.com/users/tommeagher","html_url":"https://github.com/tommeagher","followers_url":"https://api.github.com/users/tommeagher/followers","following_url":"https://api.github.com/users/tommeagher/following{/other_user}","gists_url":"https://api.github.com/users/tommeagher/gists{/gist_id}","starred_url":"https://api.github.com/users/tommeagher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tommeagher/subscriptions","organizations_url":"https://api.github.com/users/tommeagher/orgs","repos_url":"https://api.github.com/users/tommeagher/repos","events_url":"https://api.github.com/users/tommeagher/events{/privacy}","received_events_url":"https://api.github.com/users/tommeagher/received_events","type":"User","site_admin":false},"created_at":"2015-04-08T18:37:11Z","updated_at":"2015-04-08T18:37:11Z","body":"OK. I have got this mostly working using a couple of command line tools, namely [docsplit](https://documentcloud.github.io/docsplit/) and [tabula-extractor](https://github.com/tabulapdf/tabula-extractor). Here's the pseudo-code version of what I've been testing.\r\n\r\n- Burst the document into its individual pages with docsplit and then run tabula-extractor on every doc in the folder and output a new csv. \r\n```docsplit pages 20150327_crimetrend.pdf```\r\n\r\nFor naming our csvs and for ultimately importing into a db, we need the date of publication and the time period the data covers. \r\n\r\n- We can grab the date of publication from the upper right corner of the doc, or the first line in the tabula output:\r\n```tabula -p 1 20150327_crimetrend.pdf | head -n 1 | cut -d , -f6-```\r\n\r\n- We can grab the time period covered by this data from line 5:\r\n```tabula -p 1 20150327_crimetrend.pdf | sed -n '5p' | cut -d , -f6-```\r\n\r\n- For each pdf in the directory, run it through tabula and just extract the unique ID:\r\n```tabula -p 1 20150327_crimetrend.pdf|grep -E '^N.+  Total'|cut -d , -f1|cut -d T -f 1```\r\nIf agency id = \"New Jersey\", then convert that to \"statewide\"\r\n\r\n- Name the file with the date YYYY-MM-DD-AGENCY-ID. For each page, pull the table and output it to a CSV file.\r\n```tabula -p 1 -a 97.55,11.19,573.48,778.62 -o YYYY-MM-DD-AGENCY-ID-.csv 20150327_crimetrend.pdf```\r\nWe may also want to add the time period covered to the filename.\r\n\r\n- Add the column heds:\r\n``` crime, month_last_yr, month_this_yr, pct_chg, ytd_last, ytd_this_yr, ytd_pct_chg, cleared, pct_cleared, juvie_cleared```\r\n\r\n\r\nIn the resulting csv, remove any row with no values after the first column and any row that starts with \"total\".\r\n\r\n\r\nNow I want to write this as a coherent Ruby script rather than a bunch of hacky command line tools piped to one another. "}}