{"action":"created","issue":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/4755","labels_url":"https://api.github.com/repos/influxdb/influxdb/issues/4755/labels{/name}","comments_url":"https://api.github.com/repos/influxdb/influxdb/issues/4755/comments","events_url":"https://api.github.com/repos/influxdb/influxdb/issues/4755/events","html_url":"https://github.com/influxdb/influxdb/pull/4755","id":116411935,"number":4755,"title":"tsm1 WAL cache","user":{"login":"otoolep","id":536312,"avatar_url":"https://avatars.githubusercontent.com/u/536312?v=3","gravatar_id":"","url":"https://api.github.com/users/otoolep","html_url":"https://github.com/otoolep","followers_url":"https://api.github.com/users/otoolep/followers","following_url":"https://api.github.com/users/otoolep/following{/other_user}","gists_url":"https://api.github.com/users/otoolep/gists{/gist_id}","starred_url":"https://api.github.com/users/otoolep/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/otoolep/subscriptions","organizations_url":"https://api.github.com/users/otoolep/orgs","repos_url":"https://api.github.com/users/otoolep/repos","events_url":"https://api.github.com/users/otoolep/events{/privacy}","received_events_url":"https://api.github.com/users/otoolep/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/influxdb/influxdb/labels/2%20-%20Working","name":"2 - Working","color":"CCCCCC"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-11-11T20:09:27Z","updated_at":"2015-11-12T06:55:01Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/influxdb/influxdb/pulls/4755","html_url":"https://github.com/influxdb/influxdb/pull/4755","diff_url":"https://github.com/influxdb/influxdb/pull/4755.diff","patch_url":"https://github.com/influxdb/influxdb/pull/4755.patch"},"body":"This work shows the tsm1 cache taking shape. The idea behind this cache is that it is the queryable version of the data that exists in the WAL. In fact it may be a super-set of data in the WAL, and even data in the WAL has been compacted into tsm files data in the cache may be queried instead of hitting tsm files. Systems with large amounts of RAM may use this to increase query performance.\r\n\r\nOf course all caches must have an eviction policy, and this is where checkpoints come in. When the WAL compacts data into tsm files, it informs the cache via the checkpoint mechanism. This allows the cache to evict data received up to and including the checkpoint, if necessary.\r\n\r\nIf data cannot be added to the cache, because its maximum memory footprint has been reached, it will reject additions.\r\n\r\nPerformance is critical in this code, and this code will be carefully tested and benchmarked. Some distinct types have been introduced to help with construction, but these types must not impact performance."},"comment":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/comments/156018110","html_url":"https://github.com/influxdb/influxdb/pull/4755#issuecomment-156018110","issue_url":"https://api.github.com/repos/influxdb/influxdb/issues/4755","id":156018110,"user":{"login":"otoolep","id":536312,"avatar_url":"https://avatars.githubusercontent.com/u/536312?v=3","gravatar_id":"","url":"https://api.github.com/users/otoolep","html_url":"https://github.com/otoolep","followers_url":"https://api.github.com/users/otoolep/followers","following_url":"https://api.github.com/users/otoolep/following{/other_user}","gists_url":"https://api.github.com/users/otoolep/gists{/gist_id}","starred_url":"https://api.github.com/users/otoolep/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/otoolep/subscriptions","organizations_url":"https://api.github.com/users/otoolep/orgs","repos_url":"https://api.github.com/users/otoolep/repos","events_url":"https://api.github.com/users/otoolep/events{/privacy}","received_events_url":"https://api.github.com/users/otoolep/received_events","type":"User","site_admin":false},"created_at":"2015-11-12T06:55:00Z","updated_at":"2015-11-12T06:55:00Z","body":"Some more work here @pauldix @jwilder \r\n\r\nThis change sees the addition of an LRU for the entries-by-keys, allowing the least-recently-used keys to be evicted if they are before the checkpoint.\r\n\r\nI was trying to avoid subdividing entries (a set of values) by time, but I don't think I can avoid it. A non-stop write into a single key will increase the footprint of the cache, but would never allow the single key to be evicted since its checkpoint would continually be updated.\r\n\r\nIt does appear we'll have to partition entries (a set of values) by time as well as key."}}