{"push_id":693266250,"size":6,"distinct_size":6,"ref":"refs/heads/multi-threaded_index","head":"77cb5ae293d246ef7243f9737384f868756688f0","before":"517117dd93abcfcecbf634f51237b7b440d38ace","commits":[{"sha":"4d4803a49ebf9519c1d2695bbf98d07685e321b7","author":{"email":"6518d16a85396e511b329a97b21dda0c9acebabc@digital.cabinet-office.gov.uk","name":"Alex Tomlins"},"message":"Update migration test to go via BulkLoader\n\nAlso add a test for reindexing with a number of documents over the batch\nsize.","distinct":true,"url":"https://api.github.com/repos/alphagov/rummager/commits/4d4803a49ebf9519c1d2695bbf98d07685e321b7"},{"sha":"2ac9c328b6fe16bd07ba8eaeaf5115857e24ea89","author":{"email":"6518d16a85396e511b329a97b21dda0c9acebabc@digital.cabinet-office.gov.uk","name":"Alex Tomlins"},"message":"Make BulkLoader use a Logging.logger\n\nTo be consistent with the rest of the app.","distinct":true,"url":"https://api.github.com/repos/alphagov/rummager/commits/2ac9c328b6fe16bd07ba8eaeaf5115857e24ea89"},{"sha":"b4caaeaafa6bd9c4fcf3b40a66b85eefac57be24","author":{"email":"6518d16a85396e511b329a97b21dda0c9acebabc@digital.cabinet-office.gov.uk","name":"Alex Tomlins"},"message":"Parallelise bulk loading of a new index.\n\nWe can get much more performance while bulk loading a new index (either\nmigrating an indes, or loading an index from an external dump), by\nprocessing batches of records in parallel.  This is because it allows\nelastisearch to use all the resources available to it.  See [1] for more\ndetails.\n\nI've picked a default concurrency level of 12 to match our\ninfrastructure.  Elasticsearch defaults to using an indexing thread pool\nsize that matches the number of CPU cores available.  Our\napi-elasticsearch machines have 4 cores, and we run a 3-node cluster,\nhence a value of 12 makes sense. This may need to be tuned after\nbenchmarking etc.\n\nThis change halved the indexing time on my dev VM.\n\n[1] https://www.elastic.co/blog/performance-considerations-elasticsearch-indexing","distinct":true,"url":"https://api.github.com/repos/alphagov/rummager/commits/b4caaeaafa6bd9c4fcf3b40a66b85eefac57be24"},{"sha":"4d5a591029e65c60f6d721ca82899e7ec92befa1","author":{"email":"6518d16a85396e511b329a97b21dda0c9acebabc@digital.cabinet-office.gov.uk","name":"Alex Tomlins"},"message":"Switch migrate_from_unaliased_index to use BulkLoader\n\nThis removes the last usage of Index#populate_from","distinct":true,"url":"https://api.github.com/repos/alphagov/rummager/commits/4d5a591029e65c60f6d721ca82899e7ec92befa1"},{"sha":"7452a22a111dc74170b15c817a2b230581bdd1a0","author":{"email":"6518d16a85396e511b329a97b21dda0c9acebabc@digital.cabinet-office.gov.uk","name":"Alex Tomlins"},"message":"Remove Index#populate_from method and related settings\n\nThis is no longer used.  All bulk loading of indexes is now handled by\nthe BulkLoader class.","distinct":true,"url":"https://api.github.com/repos/alphagov/rummager/commits/7452a22a111dc74170b15c817a2b230581bdd1a0"},{"sha":"77cb5ae293d246ef7243f9737384f868756688f0","author":{"email":"6518d16a85396e511b329a97b21dda0c9acebabc@digital.cabinet-office.gov.uk","name":"Alex Tomlins"},"message":"Fix migrate_index task with no existing index.\n\nSince the refactoring, this was erroring if there was no existing index\nto copy documents from.","distinct":true,"url":"https://api.github.com/repos/alphagov/rummager/commits/77cb5ae293d246ef7243f9737384f868756688f0"}]}