{"action":"created","issue":{"url":"https://api.github.com/repos/boltdb/bolt/issues/337","labels_url":"https://api.github.com/repos/boltdb/bolt/issues/337/labels{/name}","comments_url":"https://api.github.com/repos/boltdb/bolt/issues/337/comments","events_url":"https://api.github.com/repos/boltdb/bolt/issues/337/events","html_url":"https://github.com/boltdb/bolt/issues/337","id":64861745,"number":337,"title":"Batching inserts does not improve performance and increases disk space usage","user":{"login":"alexsacr","id":3968929,"avatar_url":"https://avatars.githubusercontent.com/u/3968929?v=3","gravatar_id":"","url":"https://api.github.com/users/alexsacr","html_url":"https://github.com/alexsacr","followers_url":"https://api.github.com/users/alexsacr/followers","following_url":"https://api.github.com/users/alexsacr/following{/other_user}","gists_url":"https://api.github.com/users/alexsacr/gists{/gist_id}","starred_url":"https://api.github.com/users/alexsacr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexsacr/subscriptions","organizations_url":"https://api.github.com/users/alexsacr/orgs","repos_url":"https://api.github.com/users/alexsacr/repos","events_url":"https://api.github.com/users/alexsacr/events{/privacy}","received_events_url":"https://api.github.com/users/alexsacr/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-03-27T22:18:01Z","updated_at":"2015-03-28T04:44:56Z","closed_at":null,"body":"A benchmark performed [here](https://sacrilege.io/benchmarking-bolt/) showed inserting 500k records in batches of 1000 did not increase write performance compared to performing individual transactions.  In addition, batching inserts doubled the file size of the resulting DB on disk.\r\n\r\nThe dataset consisted of a constant string value (`\"877.11\"`), and a sequentially increasing string key beginning at an arbitrary unix timestamp (`\"1427323829\"`).\r\n\r\nSequential insertions were performed by calling `db.Update()` with a single key-value combination:\r\n\r\n```\r\ndb.Update(func(tx *bolt.Tx) error {\r\n    bkt := tx.Bucket([]byte(bucketName))\r\n    err := bkt.Put([]byte(key), []byte(value))\r\n    return err\r\n})\r\n```\r\n\r\nBatch insertions were made via slices (ignoring the array allocation issue here):\r\n\r\n```\r\nfunc batchInsert(db *bolt.DB, keys []string, value string) {\r\n\r\n\tif len(keys) < batchSize {\r\n\t\tinsertSlice(db, keys, value)\r\n\t\treturn\r\n\t}\r\n\r\n\tnumKeys := len(keys)\r\n\tbatches := len(keys) / batchSize\r\n\toffset := numKeys - (numKeys % batchSize)\r\n\r\n\tfor i := 0; i < batches; i++ {\r\n\t\tinsertSlice(db, keys[i:(i+1)*1000], value)\r\n\t}\r\n\r\n\tif offset != 0 {\r\n\t\tinsertSlice(db, keys[offset:], value)\r\n\t}\r\n\r\n}\r\n\r\nfunc insertSlice(db *bolt.DB, keys []string, value string) {\r\n\r\n\tdb.Update(func(tx *bolt.Tx) error {\r\n\t\tbkt := tx.Bucket([]byte(bucketName))\r\n\t\tfor _, k := range keys {\r\n\t\t\terr := bkt.Put([]byte(k), []byte(value))\r\n\t\t\tif err != nil {\r\n\t\t\t\tpanic(err)\r\n\t\t\t}\r\n\t\t}\r\n\t\treturn nil\r\n\t})\r\n\r\n}\r\n```\r\n\r\nIf you don't spot anything glaringly wrong with my use of Bolt, the code used to run the benchmark can be found [here](https://github.com/alexsacr/bolt-benchmark).  \r\n\r\nTo grab the code and run only the insertion benchmarks (one with individual transactions, one with batches of 1k), something like this ought to work:\r\n\r\n```\r\ngit clone https://github.com/alexsacr/bolt-benchmark.git\r\ncd bolt-benchmark/\r\ngo test -bench=500k\r\n```"},"comment":{"url":"https://api.github.com/repos/boltdb/bolt/issues/comments/87159915","html_url":"https://github.com/boltdb/bolt/issues/337#issuecomment-87159915","issue_url":"https://api.github.com/repos/boltdb/bolt/issues/337","id":87159915,"user":{"login":"dtfinch","id":1477446,"avatar_url":"https://avatars.githubusercontent.com/u/1477446?v=3","gravatar_id":"","url":"https://api.github.com/users/dtfinch","html_url":"https://github.com/dtfinch","followers_url":"https://api.github.com/users/dtfinch/followers","following_url":"https://api.github.com/users/dtfinch/following{/other_user}","gists_url":"https://api.github.com/users/dtfinch/gists{/gist_id}","starred_url":"https://api.github.com/users/dtfinch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dtfinch/subscriptions","organizations_url":"https://api.github.com/users/dtfinch/orgs","repos_url":"https://api.github.com/users/dtfinch/repos","events_url":"https://api.github.com/users/dtfinch/events{/privacy}","received_events_url":"https://api.github.com/users/dtfinch/received_events","type":"User","site_admin":false},"created_at":"2015-03-28T04:44:56Z","updated_at":"2015-03-28T04:44:56Z","body":"I haven't run it yet, but I think the first call to insertSlice is missing a multiplication, causing it to insert very large batches.\r\n\r\ninsertSlice(db, keys[i\\***batchSize**:(i+1)*batchSize], value)\r\n\r\n"}}