{"push_id":564727109,"size":2,"distinct_size":0,"ref":"refs/heads/deployed-to-preview","head":"e6f35e16685c8e14787d68ca1a52c591c6c1f109","before":"d333e4d0cc955d5313d0e41c6496fce1db8aac58","commits":[{"sha":"9640e6315c31ca4850a21238691567e0957870dc","author":{"email":"2b6bbb4bffb950b0967cd3a1166c66cfcf859973@digital.cabinet-office.gov.uk","name":"Richard Boulton"},"message":"Group facet example searches\n\nWhen calculating examples for facet searches (eg, with queries such as\nfacet_organisations=100,example_scope:query,examples:10), we use\nelasticsearch's `msearch` API to send a query for each facet value to\ngather the examples. This is fine when there's a reasonably small number\nof facet values to fetch examples for, but can cause problems if too\nmany facet values are found.\n\nThere are two problems which may occur:\n\n - Elasticsearch has an internal queue limit on the number of searches\n   to be performed: this defaults to 1000.  If we go close to this\n   limit, we risk getting error responses saying that the queue is full.\n   I believe the 1000 limit is shared across all nodes, so each query\n   submitted by a user counts several times against the 1000 limit. I\n   heave experimentally verified that it is possible to hit the limit\n   without actually sending as many as 1000 queries simultaneously.\n\n - Elasticsearch may take too long to perform the individual queries\n   involved in the `msearch` call and time out.\n\nFor our current database, the most pathological field is the\norganisations field, which can trigger nearly 900 searches for examples\nin the worst case.  This takes 9 seconds to run on my dev VM, and times\nout on the live site (when called by accessing the publicly exposed\nsearch API directly).  There are no \"user-facing\" pages on the live site\nwhich trigger such pathological cases, though, because facet examples\nare only requested for highly filtered searches.\n\nTo avoid these problems, instead of sending all the searches at once, we\nsend them in batches of 50.  This means that we wait for 50 queries to\nhave been fully completed before sending the next 50, which avoids both\nthe queue size problem and the timeout problem.\n\nIn future, this workaround will be insufficient to return results fast\nenough.  An alternative approach would be to use a custom elasticsearch\n\"aggregation\" to gather information on the facet examples in a single\npass through the index.  This ought to be many times faster, but is\ndependent on us upgrading the version of elasticsearch in use in\nproduction (we're currently on a version which doesn't support\naggregations).","distinct":false,"url":"https://api.github.com/repos/alphagov/rummager/commits/9640e6315c31ca4850a21238691567e0957870dc"},{"sha":"e6f35e16685c8e14787d68ca1a52c591c6c1f109","author":{"email":"60c6d277a8bd81de7fdde19201bf9c58a3df08f4@tomlins.org.uk","name":"Alex Tomlins"},"message":"Merge pull request #347 from alphagov/chunk-facet-example-searches\n\nGroup facet example searches","distinct":false,"url":"https://api.github.com/repos/alphagov/rummager/commits/e6f35e16685c8e14787d68ca1a52c591c6c1f109"}]}