{"action":"created","issue":{"url":"https://api.github.com/repos/google/cayley/issues/277","labels_url":"https://api.github.com/repos/google/cayley/issues/277/labels{/name}","comments_url":"https://api.github.com/repos/google/cayley/issues/277/comments","events_url":"https://api.github.com/repos/google/cayley/issues/277/events","html_url":"https://github.com/google/cayley/pull/277","id":90491561,"number":277,"title":"Add migration tool and use protos in Bolt store","user":{"login":"barakmich","id":98356,"avatar_url":"https://avatars.githubusercontent.com/u/98356?v=3","gravatar_id":"","url":"https://api.github.com/users/barakmich","html_url":"https://github.com/barakmich","followers_url":"https://api.github.com/users/barakmich/followers","following_url":"https://api.github.com/users/barakmich/following{/other_user}","gists_url":"https://api.github.com/users/barakmich/gists{/gist_id}","starred_url":"https://api.github.com/users/barakmich/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/barakmich/subscriptions","organizations_url":"https://api.github.com/users/barakmich/orgs","repos_url":"https://api.github.com/users/barakmich/repos","events_url":"https://api.github.com/users/barakmich/events{/privacy}","received_events_url":"https://api.github.com/users/barakmich/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-06-23T21:13:49Z","updated_at":"2015-06-24T14:37:53Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/google/cayley/pulls/277","html_url":"https://github.com/google/cayley/pull/277","diff_url":"https://github.com/google/cayley/pull/277.diff","patch_url":"https://github.com/google/cayley/pull/277.patch"},"body":"Fixes #227 \r\n\r\nSo JSON serialization has a pretty sizeable overhead. By converting the bolt data store to using Protobufs instead of JSON, there's a pretty sizeable speedup for queries on Bolt:\r\n\r\n```\r\n$ benchx boltnoproto boltwproto\r\nbenchmark                                   old ns/op     new ns/op     delta\tmult\r\nBenchmarkNamePredicate                      536192        519102        -3.19%\t-1.03x\r\nBenchmarkLargeSetsNoIntersection            25406759      16517821      -34.99%\t-1.54x\r\nBenchmarkVeryLargeSetsSmallIntersection     55738157      38773174      -30.44%\t-1.44x\r\nBenchmarkHelplessContainsChecker            72689377      50568672      -30.43%\t-1.44x\r\nBenchmarkHelplessNotContainsFilms           125709712     82282494      -34.55%\t-1.53x\r\nBenchmarkHelplessNotContainsActors          101334859     67082261      -33.80%\t-1.51x\r\nBenchmarkNetAndSpeed                        5530942       4847584       -12.36%\t-1.14x\r\nBenchmarkKeanuAndNet                        4399839       4081278       -7.24%\t-1.08x\r\nBenchmarkKeanuAndSpeed                      5144942       4554785       -11.47%\t-1.13x\r\nBenchmarkKeanuOther                         16995806      12897262      -24.12%\t-1.32x\r\nBenchmarkKeanuBullockOther                  26998991      19858152      -26.45%\t-1.36x\r\n```\r\n\r\nThis means that data in the data file needs to migrate. So, I added the plumbing and rules by which migrations can happen. This is a general framework moving forward that will allow us to change the data format without forcing everybody to reload their data (the movie db takes me about a minute to load, but about 20 seconds to convert)"},"comment":{"url":"https://api.github.com/repos/google/cayley/issues/comments/114891351","html_url":"https://github.com/google/cayley/pull/277#issuecomment-114891351","issue_url":"https://api.github.com/repos/google/cayley/issues/277","id":114891351,"user":{"login":"Quentin-M","id":1332289,"avatar_url":"https://avatars.githubusercontent.com/u/1332289?v=3","gravatar_id":"","url":"https://api.github.com/users/Quentin-M","html_url":"https://github.com/Quentin-M","followers_url":"https://api.github.com/users/Quentin-M/followers","following_url":"https://api.github.com/users/Quentin-M/following{/other_user}","gists_url":"https://api.github.com/users/Quentin-M/gists{/gist_id}","starred_url":"https://api.github.com/users/Quentin-M/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Quentin-M/subscriptions","organizations_url":"https://api.github.com/users/Quentin-M/orgs","repos_url":"https://api.github.com/users/Quentin-M/repos","events_url":"https://api.github.com/users/Quentin-M/events{/privacy}","received_events_url":"https://api.github.com/users/Quentin-M/received_events","type":"User","site_admin":false},"created_at":"2015-06-24T14:37:52Z","updated_at":"2015-06-24T14:37:52Z","body":"**Benefits**: \r\n- Reduce the mean time of my requests from 324ms to 111ms (-65.74%)\r\n\r\n**Drawbacks**: \r\n- The migration tool's memory consumption grown up to 25GB while converting a 13.96GB database (13,094,491 quads). I believe it has been fixed by 41f5573 thought.\r\n- The new database size is 26.84GB (+92%)"}}