{"action":"created","issue":{"url":"https://api.github.com/repos/18F/openFEC/issues/1141","labels_url":"https://api.github.com/repos/18F/openFEC/issues/1141/labels{/name}","comments_url":"https://api.github.com/repos/18F/openFEC/issues/1141/comments","events_url":"https://api.github.com/repos/18F/openFEC/issues/1141/events","html_url":"https://github.com/18F/openFEC/issues/1141","id":101804129,"number":1141,"title":"Handle receipts duplicates","user":{"login":"jmcarp","id":1633460,"avatar_url":"https://avatars.githubusercontent.com/u/1633460?v=3","gravatar_id":"","url":"https://api.github.com/users/jmcarp","html_url":"https://github.com/jmcarp","followers_url":"https://api.github.com/users/jmcarp/followers","following_url":"https://api.github.com/users/jmcarp/following{/other_user}","gists_url":"https://api.github.com/users/jmcarp/gists{/gist_id}","starred_url":"https://api.github.com/users/jmcarp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmcarp/subscriptions","organizations_url":"https://api.github.com/users/jmcarp/orgs","repos_url":"https://api.github.com/users/jmcarp/repos","events_url":"https://api.github.com/users/jmcarp/events{/privacy}","received_events_url":"https://api.github.com/users/jmcarp/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-08-19T04:21:54Z","updated_at":"2015-08-20T16:02:42Z","closed_at":null,"body":"Based on conversation in #fec-partners with @jwchumley and @paulclark2:\r\n\r\n> Duplicates in Receipts:  In cases where a committee is reporting contributions earmarked through another committee (Actblue is a common earmarking committee) you are including both the transaction from the individual and the memo from the earmarking committee and thus doubling the amount.   https://fec-dev-web.18f.gov/receipts?committee_id=C00578401&min_date=01%2F01%2F2015&max_date=12%2F31%2F2016 Look at the top two transactions on this image:  http://docquery.fec.gov/cgi-bin/fecimg/?201507159000167638   as an example.  We avoid this in our House and Senate map by using the transaction codes 15, 15E, and 15J for individual contributions. This loses those that are under $200 because we don't code them.  Instead of using the codes you could try leaving out transactions where the memo code is not null and the word \"EARMARK\" shows up in the memo_text. There may be some common abbreviations  for earmark like ermk or earmk.\r\n\r\nCurrently, we're trying to exclude duplicate transactions in aggregates by excluding records with `memo_cd = 'X'`, but @jwchumley pointed out that this isn't reliable. It sounds like there are multiple options for dealing with this, so let's pick one here, and then we can implement:\r\n* Restrict receipts to `receipt_tp in ('15', '15E', '15J')`. It sounds like this only includes individual contributions, which seems like the wrong behavior for the browse receipts page, since users should be able to see contributions from both individuals and organizations.\r\n* Restrict receipts to `memo_cd is null and memo_text like '%earmark%'`. This would take some work from FEC to determine exactly which memo text indicates an earmark--we'd want to validate our rules to make sure we're not including records that should be excluded and vice versa. Also, `memo_cd` is null for the large majority of receipts, so I may be misunderstanding @jwchumley's intention here.\r\n\r\n@jwchumley and @paulclark2: how would you like to proceed here? I'm happy to help evaluating options, but will defer (as usual) to your expertise."},"comment":{"url":"https://api.github.com/repos/18F/openFEC/issues/comments/133060791","html_url":"https://github.com/18F/openFEC/issues/1141#issuecomment-133060791","issue_url":"https://api.github.com/repos/18F/openFEC/issues/1141","id":133060791,"user":{"login":"jwchumley","id":12615919,"avatar_url":"https://avatars.githubusercontent.com/u/12615919?v=3","gravatar_id":"","url":"https://api.github.com/users/jwchumley","html_url":"https://github.com/jwchumley","followers_url":"https://api.github.com/users/jwchumley/followers","following_url":"https://api.github.com/users/jwchumley/following{/other_user}","gists_url":"https://api.github.com/users/jwchumley/gists{/gist_id}","starred_url":"https://api.github.com/users/jwchumley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jwchumley/subscriptions","organizations_url":"https://api.github.com/users/jwchumley/orgs","repos_url":"https://api.github.com/users/jwchumley/repos","events_url":"https://api.github.com/users/jwchumley/events{/privacy}","received_events_url":"https://api.github.com/users/jwchumley/received_events","type":"User","site_admin":false},"created_at":"2015-08-20T16:02:42Z","updated_at":"2015-08-20T16:02:42Z","body":"Disbursements are messier than receipts because most of them don't get coded and there is a wide variation in the way people report. I think if we try to write rules to handle excluding aggregate rows we will find hundreds of cases. Discussed with Paul. We think we need to include memos for display but don't include their values in when summing. \r\n"}}