The Event Transformation Agent has several modes that modify the contents of incoming Events.

Use the Event Transformation Agent to adjust formatting of incoming Events for further downstream processing. For example: extract all URLs from an email body; emit individual Events for all IP addresses from a SIEM alert; find all email addresses in a Pastebin post; ignore duplicate events.

Features

The Event Transformation Agent has several modes of operation:

extract: Use regular expressions to extract text from fields in incoming Events.

explode: Specify a field in an incoming Event that contains an array and Tines will emit individual Events for each element of the array.

deduplicate: Specify a field in an incoming Events and only emit the Event if that field contains a unique value.

message_only: Add a string to the incoming event.

delay: Wait a specific number of seconds before emitting an event.

implode: Reassemle a previously exploded array into a single event.

Extract Mode

  • Specify one or more Ruby compatible regular expressions to match and extract text from incoming Events.
  • Any matched text will be included in an array in a new Event. If there are no matches, no Event will be emitted.
  • Specify the name of the key for each array of matched text.

Explode Mode

  • Specify a field containing an array in an incoming Event.
  • A new Event will be emitted for each element of the array.
  • Specify the name of the key to hold the individual array elements in the emitted Event.
  • The newly emitted Event will also contain an index key describing the exploded element’s position in the original array.
  • If the specified field is not a valid array, no Event will be generated.

Deduplicate Mode

  • Define how many historical events should be used to determine if value in field is unique.
  • Specify a period of time to check for duplicates.
  • Specify the field in an incoming Event to use as a measure of uniqueness.
  • If an Event is deemed unique, it will be emitted. If it is deemed a duplicate, it will be ignored and a corresponding log entry will be created.

Message_only mode

  • Add a string to the incoming event in a message key.
  • Contain specified string in a custom field name.
  • Use liquid templating to modify the string.

Delay mode

  • Specify a value in seconds (between 0 and 3600 (1 hour)) to wait before emitting a received event.
  • Use natural language to specify a point in the future to run the event, e.g.: “Tomorrow at 9am”

Implode mode

  • An implode mode agent will collect individual events which have been emitted as part of an explode and store them in memory. When all (or a specified number) of the exploded events have been collected they will be emitted in a single event.
  • Specify a GUID emitted in an event as part of an explode operation
  • Specify the number of events that the implode should collect.

Configuration Options

Common

  • mode: Specify ‘extract’, ‘explode’, ‘deduplicate’, ‘delay’, ‘deduplicate’ or ‘implode’
  • message: Include an optional message that will be added to each emitted Event
  • expected_update_period_in_days: (Optional) Set this key to the maximum amount of time expected to pass between Events being created by this Agent. If this period passes without any Events being emitted, the Agent will be flagged as “Not Working”.

Extract Mode

  • conditions: When using ‘extract’ mode, define an array of regular expression configuration blocks to match text in incoming Events:
  • path: Specify the wrapped JSON path for the field containing text to extract.
  • regexp: Specify the regular expression to be used to extract text.
  • to: Specify the name of the field to contain the array of matches.

Explode Mode

  • path: When using ‘explode’ mode, define the wrapped JSON path for the field containing an array of elements to be emitted as individual events.
  • to: Specify the name of the field to contain the array of matches.

Deduplicate Mode

  • path: a JSON wrapped path the value of which should be used to determine the uniqueness of the event.
  • lookback: (Optional: must use either lookback or period) Number of past events (maximum of 1000) to examine for uniqueness. Use a value of 0 to emit Event regardless of uniqueness.
  • period: (Optional: must use either lookback or period) Number of seconds to use for deduplication. When an event is received, subsequent events will not be emitted until this period has elapsed.
  • emit_duplicate: (Optional) Set to true if duplicate events should be emitted. 'unique_event': false will be included in emitted event.

Message_only mode

  • message: a string which will be included in the emitted event.

Delay mode

  • seconds: (Optional: must use either seconds or until) an integer greater than 0.
  • until: (Optional: must use either seconds or until) a date/time in the future at which point the event will be emitted.

Implode mode

  • guid_path: The unique GUID used to identify exploded events
  • size_path: The number of events implode should collect before emitting an event.

Emitted Events

Events emitted by the Event Transformation Agent look similar to the below:

Extract Mode

{
  "email_addresses": [
    "alice@example.com"
  ],
  "ips": [
    "10.1.1.12"
  ],
  "urls": [
    "http://example.com",
    "https://tines.io"
  ]
}

Explode Mode

{
  "index": 0,
  "users": 
  {
    "name": "alice",
    "age": 85
  }
}

Deduplicate Mode

{
  "unique_event": true
}

Message_only mode

{
  "message": "This is an automatically generated message from Tines.",
  "another_message": "This is a second message in a custom field."
}

Delay mode

{
  "seconds": 25
}

Implode mode

{
 "implode":[
    {
       "receive_webhook":{
          "numbers":[
             1,
             2,
             3
          ]
       },
       "explode_array":{
          "guid":"8bc687b5-764f-492e-9210-21ac502ffa98",
          "index":0,
          "size":3,
          "number":1
       }
    },
    {
       "receive_webhook":{
          "numbers":[
             1,
             2,
             3
          ]
       },
       "explode_array":{
          "guid":"8bc687b5-764f-492e-9210-21ac502ffa98",
          "index":1,
          "size":3,
          "number":2
       }
    },
    {
       "receive_webhook":{
          "numbers":[
             1,
             2,
             3
          ]
       },
       "explode_array":{
          "guid":"8bc687b5-764f-492e-9210-21ac502ffa98",
          "index":2,
          "size":3,
          "number":3
       }
    }
 ]
}

Example Configuration Options

Extract Mode

Given the incoming Event below, extract all email addresses and store them in a field called ‘email_addresses’ in a new Event.

{
  "text": "You received to email to alice@example.com sent from bob@example.com",
}
{
  "mode": "extract",
  "matchers": [
    {
      "path": "{{.text}}",
      "regexp": "\\b[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,4}\\b",
      "to": "email_addresses"
    }
  ],
  "expected_update_period_in_days": "1"
}

Given the incoming Event below, extract all email addresses from ‘text’ store them in a field called ‘email_addresses’; extract all URLs from ‘referrers’ and store them in a field called ‘urls’; and extract all IP addresses from ‘servers’ and store them in a field called ‘ips’.

{
  "text": "You received to email to alice@example.com sent from bob@example.com",
  "metadata":{
    "servers":"10.1.1.1, 10.2.3.4, 10.15.6.8",
    "referrers": "https://tines.io and https://www.example.com"
  }
}
{
  "mode": "extract",
  "matchers": [
    {
      "path": "{{.text}}",
      "regexp": "\\b[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,4}\\b",
      "to": "email_addresses"
    },
    {
      "path": "{{.metadata.referrers}}",
      "regexp": "https?:\\/\\/[\\S]+",
      "to": "urls"
    },          
    {
      "path": "{{.metadata.servers}}",
      "regexp": "\\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\b",
      "to": "ip_addresses"
    }          
  ],
  "message": "Email addresses taken from text, Urls from referrers and IPs from servers.",
  "expected_update_period_in_days": "1"
}

Explode Mode

Given the incoming Event below, emit all elements from the ‘numbers’ array as separate Events in a field called “number”.

{
  "numbers": [8, 13, 21]
}
{
  "mode": "explode",
  "path": "{{.numbers}}",
  "to": "number",
  "expected_update_period_in_days": "1"
}

Given the incoming Event below, emit all elements from the ‘people’ array as separate Events in a field called “person”.

{
  "numbers": [8, 13, 21],
  "people": [
  {
    "name":"Alice",
    "job":"Engineer",
    "language":"English"
  },
  {
    "name":"Bob",
    "job":"Student",
    "language":"English"
  }
  ]
}
{
  "mode": "explode",
  "path": "{{.people}}",
  "to": "person",
  "expected_update_period_in_days": "1"
}

Deduplicate Mode

Check the last 100 received Events and only emit an Event if the value in the {{.person.name}} field is unique.

{
  "mode": "deduplicate",
  "lookback": "100",
  "path": "{{.person.name}}",
  "expected_update_period_in_days": "2"
}

Message_only mode

Include a simple message in the emitted event.

{
  "mode": "message_only",
  "message": "This is an automatically generated message from Tines",
  "another_message": "This is another message from Tines. The time is: {{ 'now' | date: '%Y-%m-%d %H:%M' }}.",
  "expected_update_period_in_days": "2"
}

Delay mode

Wait 10 minutes before emitting an event.

{
  "mode": "delay",
  "seconds": 600,
  "expected_update_period_in_days": "2"
}

Implode mode

Reassemble an array of exploded URLs from a previous agent in the story.

{
  "mode": "implode",
  "guid_path": "{{.explode_urls.guid}}",
  "size_path": "{{.explode_urls.size}}",
  "expected_update_period_in_days": "2"
}