logstash - splitting an event into two, based on an included array
<p>I'm working with logstash for the first time, and I'm trying to take JSON reports from amavisd-new in for searching and analysis. Amavisd-new is able to write the json logging to redis, and I have everything importing perfectly, and have started learning my way through all this.</p>

<p>But I have one issue - the format of the JSON report from amavis looks like the following - note that "recipients" has an array, with one entry for each recipient.</p>

<p>I'd like to split the entire event into two - one for each recipient, leaving all the other fields the same, but replacing the "action", "ccat_main", "queued_as", etc fields from each recipient array member into the main even.</p>

<p>The idea would be that one incoming event with two recipients would result in two separate log events in logstash - one for each person.</p>

<p>I've looked at split for events, but I'm not seeing how to do this - I can't seem to find any appropriate examples anywhere.</p>

<p>So, for real-word examples, given this:</p>

<pre><code> {
    "@timestamp" =&gt; "2014-05-06T09:29:47.048Z",
    "time_unix" =&gt; 1399368587.048,
    "time_iso_week_date" =&gt; "2014-W19-2",
    "partition" =&gt; "19",
    "type" =&gt; "amavis",
    "host" =&gt; "mailer.example.net",
    "queued_as" =&gt; ["3gNFyR4Mfjzc3", "3gNFyR4n6Lzc4"],
    "recipients" =&gt; [
      { "action" =&gt; "PASS",
        "ccat_main" =&gt; "Clean",
        "queued_as" =&gt; "3gNFyR4Mfjzc3",
        "rcpt_is_local" =&gt; false,
        "rcpt_to" =&gt; "recip2@example.org",
        "smtp_code" =&gt; "250",
        "smtp_response" =&gt; "250 2.0.0 from MTA(smtp:[::1]:10013): 250 2.0.0 Ok: queued as 3gNFyR4Mfjzc3",
        "spam_score" =&gt; -2.0
      },
      { "action" =&gt; "PASS",
        "ccat_main" =&gt; "Clean",
        "mail_id_related" =&gt; "men7HTERZaOF",
        "penpals_age" =&gt; 1114599,
        "queued_as" =&gt; "3gNFyR4n6Lzc4",
        "rcpt_is_local" =&gt; true,
        "rcpt_to" =&gt; "recip1@example.net",
        "smtp_code" =&gt; "250",
        "smtp_response" =&gt; "250 2.0.0 from MTA(smtp:[::1]:10013): 250 2.0.0 Ok: queued as 3gNFyR4n6Lzc4",
        "spam_score" =&gt; -5.272
      }
    ],
    "smtp_code"  =&gt; ["250"],
  }
</code></pre>

<p>I'd like to end up with two different events, like these:</p>

<pre><code>  {
    "@timestamp" =&gt; "2014-05-06T09:29:47.048Z",
    "time_unix" =&gt; 1399368587.048,
    "time_iso_week_date" =&gt; "2014-W19-2",
    "partition" =&gt; "19",
    "type" =&gt; "amavis",
    "host" =&gt; "mailer.example.net",
    "queued_as" =&gt; ["3gNFyR4Mfjzc3", "3gNFyR4n6Lzc4"],
    "action" =&gt; "PASS",
    "ccat_main" =&gt; "Clean",
    "queued_as" =&gt; "3gNFyR4Mfjzc3",
    "rcpt_is_local" =&gt; false,
    "rcpt_to" =&gt; "recip2@example.org",
    "smtp_code" =&gt; "250",
    "smtp_response" =&gt; "250 2.0.0 from MTA(smtp:[::1]:10013): 250 2.0.0 Ok: queued as 3gNFyR4Mfjzc3",
    "spam_score" =&gt; -2.0
    "smtp_code"  =&gt; ["250"],
  }
</code></pre>

<p>and</p>

<pre><code>  {
    "@timestamp" =&gt; "2014-05-06T09:29:47.048Z",
    "time_unix" =&gt; 1399368587.048,
    "time_iso_week_date" =&gt; "2014-W19-2",
    "partition" =&gt; "19",
    "type" =&gt; "amavis",
    "host" =&gt; "mailer.example.net",
    "queued_as" =&gt; ["3gNFyR4Mfjzc3", "3gNFyR4n6Lzc4"],
    "recipients" =&gt; [
    "action" =&gt; "PASS",
    "ccat_main" =&gt; "Clean",
    "mail_id_related" =&gt; "men7HTERZaOF",
    "penpals_age" =&gt; 1114599,
    "queued_as" =&gt; "3gNFyR4n6Lzc4",
    "rcpt_is_local" =&gt; true,
    "rcpt_to" =&gt; "recip1@example.net",
    "smtp_code" =&gt; "250",
    "smtp_response" =&gt; "250 2.0.0 from MTA(smtp:[::1]:10013): 250 2.0.0 Ok: queued as 3gNFyR4n6Lzc4",
    "spam_score" =&gt; -5.272
    "smtp_code"  =&gt; ["250"],
  }
</code></pre>

<p><strong>EDIT:</strong></p>

<p>Okay, I simply used the split filter - I should have seen that.  But there's one thing that's confusing me.</p>

<p>When there's a single recipient, it passes the block right through - the result in kibana looks like:</p>

<pre><code>recipients      {
  "action": "PASS",
  "bypass_banned_checks": true,
  "bypass_spam_checks": true,
  "ccat_main": "Clean",
  "queued_as": "3qv7Km4Ybpz14Kyh",
  "rcpt_is_local": true,
  "rcpt_to": "user@domain.com",
  "rid": "552213780",
  "smtp_code": "250",
  "smtp_response": "250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 3qv7Km4Ybpz14Kyh"
}
</code></pre>

<p>But when there are 2 or more recipients, the new events each look like this, with their appropriate info:</p>

<pre><code>recipients.action       PASS
recipients.ccat_main        CleanTag
recipients.queued_as        3qv7Ly4Pqvz4wyS
recipients.rcpt_is_local        true
recipients.rcpt_to      user@domain.com
recipients.rid      552278239
recipients.smtp_code        250
recipients.smtp_response        250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 3qv7Ly4Pqvz4wyS
recipients.whitelisted      true
</code></pre>

<p>Why the difference between the two?  I think I'd prefer to keep the recipients field as a hash of values, so what's the best way to get make the split events consistent with the single event?</p>