Insert aggregation results into an index
<p>The goal is to <strong>build an Elasticsearch index with only the most recent documents in groups of related documents</strong> to track the current state of some monitoring counters and states.</p>

<p>I have crafted a simple <strong>Elasticsearch aggregation query</strong>:</p>

<pre><code>{
  "size": 0,
  "aggs": {
    "group_by_monitor": {
      "terms": {
        "field": "monitor_name"
      },
      "aggs": {
        "get_latest": {
          "top_hits": {
            "size": 1,
            "sort": [
              {
                "timestamp": {
                  "order": "desc"
                }
              }
            ]
          }
        }
      }
    }
  }
}
</code></pre>

<p>It groups related documents into buckets and select the most recent document for each bucket.</p>

<p>Here are the different ideas I had to get the job done:</p>

<ol>
<li>directly use the aggregation query to push the results into the index, but it does not seem possible : <a href="https://stackoverflow.com/questions/24813756/is-it-possible-to-put-the-results-of-an-elasticsearch-aggregation-back-into-the">Is it possible to put the results of an ElasticSearch aggregation back into the index?</a></li>
<li>use the <strong>Logstash</strong> <a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html" rel="noreferrer">Elasticsearch input plugin</a> to execute the aggregation query and the <a href="https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html" rel="noreferrer">Elasticsearch output plugin</a> to push into the index, but seems like the input plugin only looks at the <code>hits</code> field and is unable to handle aggregation results: <a href="https://discuss.elastic.co/t/aggregation-query-possible-input-es-plugin/28159" rel="noreferrer">Aggregation Query possible input ES plugin</a> !</li>
<li>use the <strong>Logstash</strong> <a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http_poller.html" rel="noreferrer">http_poller plugin</a> to get a JSON document, but it does not seem to allow specifying a body for the HTTP request !</li>
<li>use the <strong>Logstash</strong> <a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-exec.html" rel="noreferrer">exec plugin</a> to execute cURL commands to get the JSON but this seems quite cumbersome and my last resort.</li>
<li>use the <a href="http://nest.azurewebsites.net/" rel="noreferrer">NEST API</a> to build a basic application that will do polling, extract results, clean them and inject the resulting documents into the target index, but I'd like to avoid adding a new tool to maintain.</li>
</ol>

<p>Is there a reasonably complex way of accomplishing this?</p>