{"push_id":692420373,"size":2,"distinct_size":2,"ref":"refs/heads/multi-threaded_index","head":"4b2cf6d75f279602b19530a9945e6093939b80aa","before":"8d5facd94af27b280c8cff009a03e06539b5d5a2","commits":[{"sha":"ff94f4627c21b186f0e4002b4a2799a97ff97a2a","author":{"email":"6518d16a85396e511b329a97b21dda0c9acebabc@digital.cabinet-office.gov.uk","name":"Alex Tomlins"},"message":"Make BulkLoader use a Logging.logger\n\nTo be consistent with the rest of the app.","distinct":true,"url":"https://api.github.com/repos/alphagov/rummager/commits/ff94f4627c21b186f0e4002b4a2799a97ff97a2a"},{"sha":"4b2cf6d75f279602b19530a9945e6093939b80aa","author":{"email":"6518d16a85396e511b329a97b21dda0c9acebabc@digital.cabinet-office.gov.uk","name":"Alex Tomlins"},"message":"Parallelise bulk loading of a new index.\n\nWe can get much more performance while bulk loading a new index (either\nmigrating an indes, or loading an index from an external dump), by\nprocessing batches of records in parallel.  This is because it allows\nelastisearch to use all the resources available to it.  See [1] for more\ndetails.\n\nI've picked a default concurrency level of 12 to match our\ninfrastructure.  Elasticsearch defaults to using an indexing thread pool\nsize that matches the number of CPU cores available.  Our\napi-elasticsearch machines have 4 cores, and we run a 3-node cluster,\nhence a value of 12 makes sense. This may need to be tuned after\nbenchmarking etc.\n\nThis change halved the indexing time on my dev VM.\n\n[1] https://www.elastic.co/blog/performance-considerations-elasticsearch-indexing","distinct":true,"url":"https://api.github.com/repos/alphagov/rummager/commits/4b2cf6d75f279602b19530a9945e6093939b80aa"}]}