{"action":"created","issue":{"url":"https://api.github.com/repos/Netflix/ice/issues/143","labels_url":"https://api.github.com/repos/Netflix/ice/issues/143/labels{/name}","comments_url":"https://api.github.com/repos/Netflix/ice/issues/143/comments","events_url":"https://api.github.com/repos/Netflix/ice/issues/143/events","html_url":"https://github.com/Netflix/ice/issues/143","id":56416654,"number":143,"title":"Ice totals off","user":{"login":"ckelner","id":133862,"avatar_url":"https://avatars.githubusercontent.com/u/133862?v=3","gravatar_id":"","url":"https://api.github.com/users/ckelner","html_url":"https://github.com/ckelner","followers_url":"https://api.github.com/users/ckelner/followers","following_url":"https://api.github.com/users/ckelner/following{/other_user}","gists_url":"https://api.github.com/users/ckelner/gists{/gist_id}","starred_url":"https://api.github.com/users/ckelner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ckelner/subscriptions","organizations_url":"https://api.github.com/users/ckelner/orgs","repos_url":"https://api.github.com/users/ckelner/repos","events_url":"https://api.github.com/users/ckelner/events{/privacy}","received_events_url":"https://api.github.com/users/ckelner/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":33,"created_at":"2015-02-03T18:25:58Z","updated_at":"2015-02-26T15:59:49Z","closed_at":null,"body":"Hello, our Ice total is off by a large amount when totaled over the month, for individual days it is obviously much smaller (1/30th or thereabouts).  I can't share the exact amount, but suffice to say it is large enough that it is not easily missed.\r\n\r\n![screen shot 2015-02-03 at 1 07 34 pm](https://cloud.githubusercontent.com/assets/133862/6026251/4ded6b0c-aba7-11e4-9081-16e5e1a80d3c.png)\r\n\r\nI wrote a dumb py script (see https://gist.github.com/ckelner/962b9e52db11fc73cf68#file-aws_csv_totaler-py) to total our CSV by LineItem (our CSV is 17million lines and about 7Gb, so it is hard to handle any other way).  The value it spits out is equal to the value AWS shows in the billing console.  The totals at the bottom of the CSV also line up w/ the AWS billing console and the py script.\r\n\r\nMy colleague, @AlexHawk31 has done some deep dive comparisons on the numbers (she's really good at that), and it looks like charges for unused reserved instances aren't showing up for us.  Every account that has unused RIs is off by some value.\r\n\r\nTo provide some more context, that may be helpful, we have 70+ AWS accounts under a single master account.  We also have several hundred RI purchases (not in this month, just overall).  We also added two new accounts under the master account to consolidated billing this month.\r\n\r\nAdditionally we also run more than one processor for redundancy, uptime, etc, as seen in https://github.com/Netflix/ice/issues/141.  Not sure if that matters or not (attempting to figure that out in issue 141).\r\n\r\nI've seen a lot of fixes which involve \"dumping local ice files and ice s3 files\" -- I'd like to avoid doing that and rather find the root of the problem.  I'm happy to make code changes, but I wanted to come here first and start a conversation about it."},"comment":{"url":"https://api.github.com/repos/Netflix/ice/issues/comments/76203795","html_url":"https://github.com/Netflix/ice/issues/143#issuecomment-76203795","issue_url":"https://api.github.com/repos/Netflix/ice/issues/143","id":76203795,"user":{"login":"ckelner","id":133862,"avatar_url":"https://avatars.githubusercontent.com/u/133862?v=3","gravatar_id":"","url":"https://api.github.com/users/ckelner","html_url":"https://github.com/ckelner","followers_url":"https://api.github.com/users/ckelner/followers","following_url":"https://api.github.com/users/ckelner/following{/other_user}","gists_url":"https://api.github.com/users/ckelner/gists{/gist_id}","starred_url":"https://api.github.com/users/ckelner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ckelner/subscriptions","organizations_url":"https://api.github.com/users/ckelner/orgs","repos_url":"https://api.github.com/users/ckelner/repos","events_url":"https://api.github.com/users/ckelner/events{/privacy}","received_events_url":"https://api.github.com/users/ckelner/received_events","type":"User","site_admin":false},"created_at":"2015-02-26T15:59:49Z","updated_at":"2015-02-26T15:59:49Z","body":"Just a bit more info, I tried an undocumented config option suggested by another Ice user `ice.resourceGroupCost=files` to see if it made any difference with the stock codebase.  It results in the same differences we saw without that configuration option unfortunately.  "}}