{"action":"created","issue":{"url":"https://api.github.com/repos/snowplow/snowplow/issues/2158","labels_url":"https://api.github.com/repos/snowplow/snowplow/issues/2158/labels{/name}","comments_url":"https://api.github.com/repos/snowplow/snowplow/issues/2158/comments","events_url":"https://api.github.com/repos/snowplow/snowplow/issues/2158/events","html_url":"https://github.com/snowplow/snowplow/issues/2158","id":114836683,"number":2158,"title":"Optimize weather data cache","user":{"login":"fblundun","id":5892914,"avatar_url":"https://avatars.githubusercontent.com/u/5892914?v=3","gravatar_id":"","url":"https://api.github.com/users/fblundun","html_url":"https://github.com/fblundun","followers_url":"https://api.github.com/users/fblundun/followers","following_url":"https://api.github.com/users/fblundun/following{/other_user}","gists_url":"https://api.github.com/users/fblundun/gists{/gist_id}","starred_url":"https://api.github.com/users/fblundun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fblundun/subscriptions","organizations_url":"https://api.github.com/users/fblundun/orgs","repos_url":"https://api.github.com/users/fblundun/repos","events_url":"https://api.github.com/users/fblundun/events{/privacy}","received_events_url":"https://api.github.com/users/fblundun/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/snowplow/snowplow/labels/3.%20ETL","name":"3. ETL","color":"c3d69b"}],"state":"open","locked":false,"assignee":{"login":"chuwy","id":681171,"avatar_url":"https://avatars.githubusercontent.com/u/681171?v=3","gravatar_id":"","url":"https://api.github.com/users/chuwy","html_url":"https://github.com/chuwy","followers_url":"https://api.github.com/users/chuwy/followers","following_url":"https://api.github.com/users/chuwy/following{/other_user}","gists_url":"https://api.github.com/users/chuwy/gists{/gist_id}","starred_url":"https://api.github.com/users/chuwy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chuwy/subscriptions","organizations_url":"https://api.github.com/users/chuwy/orgs","repos_url":"https://api.github.com/users/chuwy/repos","events_url":"https://api.github.com/users/chuwy/events{/privacy}","received_events_url":"https://api.github.com/users/chuwy/received_events","type":"User","site_admin":false},"milestone":null,"comments":9,"created_at":"2015-11-03T15:06:01Z","updated_at":"2015-11-04T09:29:30Z","closed_at":null,"body":"Simplified example using a single spatial dimension called \"displacement\" rather than 2 spatial dimensions + time.\r\n\r\nAssume that the user has specified a tolerance of 1km, meaning that our reported weather must be the actual weather somewhere within 1km of the event.\r\n\r\nAs I understand it, the current strategy is:\r\n\r\nFor each event, round its displacement to the nearest 2km. If the rounded displacement is in the cache, use the cached value; otherwise, look up the weather information, add it to the cache with the rounded displacement as key, and return it.\r\n\r\nThe disadvantage of this strategy is that the returned results are unnecessarily innaccurate.\r\n\r\nAn alternative strategy:\r\n\r\nInstead of the cache being a Map[Displacement, Weather] we make it a Map[RoundedDownDisplacement, (ActualDisplacement, Weather)]. Each key represents an interval of 1km. For example, the key 7 represents the interval of all displacements between 7km and 8km.\r\n\r\nFor each event, let d be the actual displacement of the event in km. Check the cache for floor(d-1), floor(d), and ceiling(d). If any one of those keys is in the cache and if the corresponding actual displacement value of the key is within 1km of the non-rounded displacement of the event, use that key's weather. Otherwise, add `floor(d): (d, getWeather(d))` to the cache and return getWeather(d).\r\n\r\nFor example, suppose an event has displacement 3.7km, and the cache looks like this:\r\n\r\n```\r\n{\r\n  2: (2.3, sunny),\r\n  4: (4.4, rainy)\r\n}\r\n```\r\nFirst we see that floor(3.7-1)=2 is in the cache, but the difference between 2.3 and 3.7 is more than 1 (our specified tolerance) so we cannot use this cached value. Next we see that floor(3.7-1)=3 is not in the cache at all. Finally we see that ceiling(3.7)=4 is in the cache and 4.4 - 3.7 < 1 so we use \"rainy\" as the weather value for the event.\r\n\r\nIf 4 had not been in the cache, we not be able to use any cached value and would have to look up 3.7 in the API and add it to the cache:\r\n\r\n```\r\n{\r\n 2: (2.3, sunny),\r\n 3: (3.7, slightly rainy)\r\n}\r\n```\r\n\r\nNote that we look up the value from the API for 3.7km exactly, rather than 4km (as we would under the original strategy).\r\n\r\nIt should be possible to extend this idea from a single dimension to 3 dimensions (latitude, longitude, and time). Instead of intervals of 1km, the keys in the cache would represent cubes of 1km x 1km x 1hr, and we would check up to 27 keys rather than up to 3.\r\n\r\nAny thoughts? Is there an easier way to achieve this? Does this make any sense at all?"},"comment":{"url":"https://api.github.com/repos/snowplow/snowplow/issues/comments/153654874","html_url":"https://github.com/snowplow/snowplow/issues/2158#issuecomment-153654874","issue_url":"https://api.github.com/repos/snowplow/snowplow/issues/2158","id":153654874,"user":{"login":"alexanderdean","id":200511,"avatar_url":"https://avatars.githubusercontent.com/u/200511?v=3","gravatar_id":"","url":"https://api.github.com/users/alexanderdean","html_url":"https://github.com/alexanderdean","followers_url":"https://api.github.com/users/alexanderdean/followers","following_url":"https://api.github.com/users/alexanderdean/following{/other_user}","gists_url":"https://api.github.com/users/alexanderdean/gists{/gist_id}","starred_url":"https://api.github.com/users/alexanderdean/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexanderdean/subscriptions","organizations_url":"https://api.github.com/users/alexanderdean/orgs","repos_url":"https://api.github.com/users/alexanderdean/repos","events_url":"https://api.github.com/users/alexanderdean/events{/privacy}","received_events_url":"https://api.github.com/users/alexanderdean/received_events","type":"User","site_admin":false},"created_at":"2015-11-04T09:29:30Z","updated_at":"2015-11-04T09:29:30Z","body":"I think that's survivable, because the event fingerprint doesn't include derived contexts... (Therefore deduplication won't be affected)."}}