ElasticSearch filter on aggregations without affecting aggregation counts
<p>We're using ElasticSearch to find offers based on 5 fields, such like some 'free text', offer state and client name. We also need to aggregate on the two fields client name and offer state. So when someone enters some free text and we found say 10 docs with state closed and 8 with state open, the 'state filter' should contain closed(10) and open(8).</p>

<p>Now the problem is, when I select the state 'closed' to be included in the filter, the aggregation result for open changes to 0. I want this to remain 8. So how can I prevent the filter on the aggregations to influence the aggregation itself?</p>

<p>Here is the first query, searching for 'java':</p>

<pre><code>{
    "query": {
        "bool": {
            "filter": [
            ],
            "must": {
                "simple_query_string": {
                    "query" : "java"
                }
            }
        }
    },
    "aggs": {
        "OFFER_STATE_F": {
            "terms": {
                "size": 0,
                "field": "offer_state_f",
                "min_doc_count": 0
            }
        }
    },
    "from": 0,
    "size": 1,
    "fields": ["offer_id_ft", "offer_state_f"]
}
</code></pre>

<p>The result is this:</p>

<pre><code>{
  "hits": {
    "total": 960,
    "max_score": 0.89408284000000005,
    "hits": [
      {
        "_type": "offer",
        "_index": "select",
        "_id": "40542",
        "fields": {
          "offer_id_ft": [
            "40542"
          ],
          "offer_state_f": [
            "REJECTED"
          ]
        },
        "_score": 0.89408284000000005
      }
    ]
  },
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "timed_out": false,
  "aggregations": {
    "OFFER_STATE_F": {
      "buckets": [
        {
          "key": "REJECTED",
          "doc_count": 778
        },
        {
          "key": "ACCEPTED",
          "doc_count": 130
        },
        {
          "key": "CANCELED",
          "doc_count": 22
        },
        {
          "key": "WITHDRAWN",
          "doc_count": 13
        },
        {
          "key": "LONGLIST",
          "doc_count": 12
        },
        {
          "key": "SHORTLIST",
          "doc_count": 5
        },
        {
          "key": "INTAKE",
          "doc_count": 0
        }
      ],
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0
    }
  },
  "took": 2
}
</code></pre>

<p>As you see, the sum of the client_state_f buckets is equal to the total hits (960). Now, I include one of the states in the query, say 'ACCEPTED'. So my query becomes:</p>

<pre><code>{
    "query": {
        "bool": {
            "filter": [
                {
                    "bool": {
                        "should": [
                            {
                                "term": {
                                    "offer_state_f": "ACCEPTED"
                                }
                            }
                        ]
                    }
                }            
            ],
            "must": {
                "simple_query_string": {
                    "query" : "java"
                }
            }
        }
    },
    "aggs": {
        "OFFER_STATE_F": {
            "terms": {
                "size": 0,
                "field": "offer_state_f",
                "min_doc_count": 0
            }
        }
    },
    "from": 0,
    "size": 1,
    "fields": ["offer_id_ft", "offer_state_f"]
}
</code></pre>

<p>What I want is 130 results, but the client_state_f buckets stilling summing up to 960. But what I got is this:</p>

<pre><code>{
  "hits": {
    "total": 130,
    "max_score": 0.89408284000000005,
    "hits": [
      {
        "_type": "offer",
        "_index": "select",
        "_id": "16884",
        "fields": {
          "offer_id_ft": [
            "16884"
          ],
          "offer_state_f": [
            "ACCEPTED"
          ]
        },
        "_score": 0.89408284000000005
      }
    ]
  },
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "timed_out": false,
  "aggregations": {
    "OFFER_STATE_F": {
      "buckets": [
        {
          "key": "ACCEPTED",
          "doc_count": 130
        },
        {
          "key": "CANCELED",
          "doc_count": 0
        },
        {
          "key": "INTAKE",
          "doc_count": 0
        },
        {
          "key": "LONGLIST",
          "doc_count": 0
        },
        {
          "key": "REJECTED",
          "doc_count": 0
        },
        {
          "key": "SHORTLIST",
          "doc_count": 0
        },
        {
          "key": "WITHDRAWN",
          "doc_count": 0
        }
      ],
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0
    }
  },
  "took": 10
}
</code></pre>

<p>As you can see, only the ACCEPTED bucket is filled, all the others are 0.</p>