{"action":"opened","issue":{"url":"https://api.github.com/repos/btrask/stronglink/issues/22","labels_url":"https://api.github.com/repos/btrask/stronglink/issues/22/labels{/name}","comments_url":"https://api.github.com/repos/btrask/stronglink/issues/22/comments","events_url":"https://api.github.com/repos/btrask/stronglink/issues/22/events","html_url":"https://github.com/btrask/stronglink/issues/22","id":66640121,"number":22,"title":"Intersection query caching","user":{"login":"btrask","id":301663,"avatar_url":"https://avatars.githubusercontent.com/u/301663?v=3","gravatar_id":"","url":"https://api.github.com/users/btrask","html_url":"https://github.com/btrask","followers_url":"https://api.github.com/users/btrask/followers","following_url":"https://api.github.com/users/btrask/following{/other_user}","gists_url":"https://api.github.com/users/btrask/gists{/gist_id}","starred_url":"https://api.github.com/users/btrask/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/btrask/subscriptions","organizations_url":"https://api.github.com/users/btrask/orgs","repos_url":"https://api.github.com/users/btrask/repos","events_url":"https://api.github.com/users/btrask/events{/privacy}","received_events_url":"https://api.github.com/users/btrask/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-04-06T16:07:55Z","updated_at":"2015-04-06T16:07:55Z","closed_at":null,"body":"Due to the strict requirements of our queries, intersections are necessarily slow in cases where an extremely common clause is joined with an extremely rare one. I don't see any way to speed this up aside from caching.\r\n\r\nWe should be able to do caching intelligently, basically incrementally caching of only the items needed to satisfy a given request. Caches never have to be invalidated and can just be slowly built forever. So that's nice.\r\n\r\nWhere to store these caches is an open question. Storing them in the main database seems like a bad idea for several reasons. Maybe a second database or separate files. Durability is not necessary.\r\n\r\nNote that LevelDB eats file descriptors for lunch (we currently cap it to 1/3 of the process's limit). But that's not a problem specific to caching."}}