Script fails in upsert, with logstash output elasticsearch plugin
<p><strong>Environment</strong></p>

<p>DB: Sybase
Logstash: 2.2.0 with JDBC Plugin, Elasticsearch Output plugin</p>

<p><strong>SQL Query:</strong></p>

<pre><code>select res.id as 'res.id', res.name as 'res.name', tag.name as 'tag.name'
from Res res, ResTags rt, Tags tag
where res.id *= rt.resrow and rt.tagid *= tag.id
</code></pre>

<p><strong>SQL Result:</strong></p>

<pre><code>res.id | res.name | tag.name
0 | result0 | null
0 | result0 | tagA
1 | result1 | tagA
1 | result1 | tagB
2 | result2 | tagA
2 | result2 | tagC
</code></pre>

<p><strong>Index Mapping:</strong></p>

<pre><code>{ 
 "mappings": { 
  "res": { 
   "properties": { 
    "id": { "type": "long"},
    "name": { "type": "string" },
    "tags": {
      "type": "nested",
      "properties": { "tagname": { "type": "string" }}
    }
   }
 }
}
</code></pre>

<p><strong>Conf File:</strong></p>

<pre><code>input {
 jdbc {
  jdbc_driver_library =&gt; "jtds-1.3.1.jar"
  jdbc_driver_class =&gt; "Java::net.sourceforge.jtds.jdbc.Driver"
  jdbc_connection_string =&gt; "jdbc:jtds:sybase://hostname.com:1234/schema"
  jdbc_user =&gt; "george"
  jdbc_password =&gt; "monkey"
  jdbc_fetch_size =&gt; 100
  statement_filepath =&gt; "/home/george/sql"
 }
}
output {
 elasticsearch {
  action =&gt; "update"
  index =&gt; "myres"
  document_type =&gt; "res"
  document_id =&gt; "%{res.id}"
  script_lang =&gt; "groovy"
  hosts =&gt; [ "my.other.host.com:5921" ]
  upsert =&gt; ' {
    "id" : %{res.id},
    "name" : "%{res.name}",
    "tags" :[{ "tagname": "%{tag.name}" }]
  }'
  script =&gt; '
    if (ctx._source.res.tags.containsValue(null)) {
      // if null has been added replace it with actual value
      cts._source.res.tags = [{"tagname": "%{tag.name}" }];
    else {
      // if you find the tag, then do nothing
      if (ctx._source.res.tags.containsValue("%{tag.name}")) {}
      else {
        // if the value you try to add is not null
        if (%{tag.name} != null)
           // add it as a new object into the tag array
           ctx._source.res.tags += {"tagname": "%{tag.name}"};
        }
     }
   '
 }
}
</code></pre>

<p>The GOAL is to add the multiple rows returned from the database into ES, concatenating the tags as new objects (this is simplified example, so add_tag and filters do not do the job, as I have json structure deeper than 2 levels (nested of nested, etc))</p>

<p>The desired outcome after the bulk upload into ES would be:</p>

<pre><code>{
 "hits": {
   "total": 3,
   "max_score": 1,
   "hits": [ {
     "_index": "myres",
     "_type": "res",
     "_id": 0,
     "_score": 1,
     "_source": {
       "res": {
         "id":0,
         "name": "result0",
         "tags": [{"tagname": "tagA"}],
         "@version": "2",
         "@timestamp": "2016-xx-yy..."
       }
      },{
     "_index": "myres",
     "_type": "res",
     "_id": 1,
     "_score": 1,
     "_source": {
       "res": {
         "id":1,
         "name": "result1",
         "tags": [{"tagname": "tagA"},{"tagname": "tagB"}],
         "@version": "2",
         "@timestamp": "2016-xx-yy..."
       }
      }{
     "_index": "myres",
     "_type": "res",
     "_id": 2,
     "_score": 1,
     "_source": {
       "res": {
         "id":2,
         "name": "result2",
         "tags": [{"tagname": "tagA"},{"tagname": "tagC"],
         "@version": "2",
         "@timestamp": "2016-xx-yy..."
       }
      }
     }
...
</code></pre>

<p><strong>ISSUE:</strong> if in the conf, output section the script is not commented out, the below error pops out. If the script is not included, then only the initial tags (as expected) are imported, and the second ones are not.</p>

<p>It looks like script is not working within elasticsearch output.</p>

<p><strong>ERROR message:</strong></p>

<pre><code>    [400] {"error":"ActionRequestValidationException[Validation Failed: 
1: script or doc is missing;
2: script or doc is missing;
3: script or doc is missing;],"status":400]} {:class=&gt; ... bla bla ...}
</code></pre>

<p><strong>NOTES</strong></p>

<ul>
<li>To avoid wasting peoples' time, doc_as_upsert => true also does not work as expected. It just keeps on updating / overwriting and just keeps the latest row of the db.</li>
<li>Also, the river plugin for jdbc to ES does not support nested of nested structure so that does not work eithe</li>
</ul>