{"action":"created","issue":{"url":"https://api.github.com/repos/caciviclab/disclosure-backend/issues/12","labels_url":"https://api.github.com/repos/caciviclab/disclosure-backend/issues/12/labels{/name}","comments_url":"https://api.github.com/repos/caciviclab/disclosure-backend/issues/12/comments","events_url":"https://api.github.com/repos/caciviclab/disclosure-backend/issues/12/events","html_url":"https://github.com/caciviclab/disclosure-backend/issues/12","id":97361879,"number":12,"title":"Corrections and auditability tracking","user":{"login":"adborden","id":509703,"avatar_url":"https://avatars.githubusercontent.com/u/509703?v=3","gravatar_id":"","url":"https://api.github.com/users/adborden","html_url":"https://github.com/adborden","followers_url":"https://api.github.com/users/adborden/followers","following_url":"https://api.github.com/users/adborden/following{/other_user}","gists_url":"https://api.github.com/users/adborden/gists{/gist_id}","starred_url":"https://api.github.com/users/adborden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adborden/subscriptions","organizations_url":"https://api.github.com/users/adborden/orgs","repos_url":"https://api.github.com/users/adborden/repos","events_url":"https://api.github.com/users/adborden/events{/privacy}","received_events_url":"https://api.github.com/users/adborden/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-07-27T00:28:52Z","updated_at":"2015-12-24T00:40:53Z","closed_at":null,"body":"A system to provide an auditability trail. From @rkiddy:\r\n> We _should_ correct some things about the data so that analyses can be\r\n    done with the data without jumping through too many hoops. But we want\r\n    to still be able to claim that this is \"data supplied from the\r\n    (state/county/city/etc)\" so we have to be careful about this.\r\n>\r\n>  We can use tools like dedupe and others to help us connect the\r\n    different names of things to the actual things, but this will have\r\n    to be checked by someone. And there will be some corrections that\r\n    will only be doable by a human being, so we need to be able to track\r\n    the fixes that are made.\r\n\r\nThe system could be implemented via DB records which reference the raw records an any \"manual\" corrections that we've made.\r\n\r\nCare should be taken that the audit records refer only to the raw data records, so that if we need to blow away the cleaned records, we can download the latest set of raw data then apply the corrections to get the cleaned data.\r\n\r\nThe corrections should be available by API so that if someone else wants to re-create our cleaned netfile data, they can do that by grabbing the raw data and applying our corrections."},"comment":{"url":"https://api.github.com/repos/caciviclab/disclosure-backend/issues/comments/167014239","html_url":"https://github.com/caciviclab/disclosure-backend/issues/12#issuecomment-167014239","issue_url":"https://api.github.com/repos/caciviclab/disclosure-backend/issues/12","id":167014239,"user":{"login":"mikeubell","id":6759416,"avatar_url":"https://avatars.githubusercontent.com/u/6759416?v=3","gravatar_id":"","url":"https://api.github.com/users/mikeubell","html_url":"https://github.com/mikeubell","followers_url":"https://api.github.com/users/mikeubell/followers","following_url":"https://api.github.com/users/mikeubell/following{/other_user}","gists_url":"https://api.github.com/users/mikeubell/gists{/gist_id}","starred_url":"https://api.github.com/users/mikeubell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikeubell/subscriptions","organizations_url":"https://api.github.com/users/mikeubell/orgs","repos_url":"https://api.github.com/users/mikeubell/repos","events_url":"https://api.github.com/users/mikeubell/events{/privacy}","received_events_url":"https://api.github.com/users/mikeubell/received_events","type":"User","site_admin":false},"created_at":"2015-12-24T00:40:53Z","updated_at":"2015-12-24T00:40:53Z","body":"The transaction ID is generated by the data entry software and aught to be unique with in a single filing, but even that may not be true.\n\n> On Dec 23, 2015, at 3:20 PM, Andrell Bower <notifications@github.com> wrote:\n> \n> I do not know of a field that would do that. There is a transaction ID field, but it is not a true transaction ID. I can't remember the specifics of how it's generated, I can ask our City Clerk about that, but a quick look at the Netfile data that I can just download to Excel from the clerk's website shows the same transaction ID for multiple contributions that don't have much in common. Really, the only forms that are reported on a transaction-basis are the 497 and 496. Form 460 is just kind of a data dump of a bunch of contributions and expenditures that happened during a specific time period.\n> The audit trail will obviously be good, but it seems that the entire effort might create more work than it saves?\n> \n> —\n> Reply to this email directly or view it on GitHub.\n> \n"}}