The HTTP Request Agent sends HTTP requests using POST or GET Methods to a specified url.

Use a HTTP Request Agent to interact with REST APIs and web applications. For example: find all tweets mentioning specific keywords; update a JIRA ticket based on incoming Events; create a Pager Duty incident; check VirusTotal for an MD5 hash.

Features

  • Send requests on receipt of incoming Events, or run on a schedule.
  • Send requests using ‘GET’, ‘POST’, ‘PUT’, ‘PATCH’ and ‘DELETE’ methods.
  • Enable/disable SSL verification.
  • Optionally specify HTTP headers, including user agent.
  • Include information from incoming Events in HTTP requests
  • Request response will be emitted as a new Event

Configuration Options

  • url: Specify where the request should be sent. Include the URI scheme (‘http’ or ‘https’).
  • method: (Optional) Specify the HTTP method to use, i.e.: ‘get’, ‘post’, ‘put’, ‘patch’, or ‘delete’. Defaults to ‘post’.
  • payload: (Optional) Specify key-value parameters to include in the body of the request. Use wrapped JSONPaths to include data from incoming Events.
  • content_type: (Optional) Specify the content type to use with the request. Shorthands are provided for the following, common content types:
    • ‘application/json; charset=utf-8’: json
    • ‘text/xml; charset=utf-8’: xml
    • ‘application/x-www-form-urlencoded’: form
    • ‘multipart/form-data’: data
  • headers: (Optional) Specify a hash of headers to send with the request.
  • basic_auth: (Optional) Specify HTTP basic auth parameters: “username:password”, or [“username”, “password”].
  • disable_ssl_verification: (Optional) Set to ‘true’ to disable ssl verification.
  • user_agent: (Optional) Specify a custom User-Agent name (default: “Tines - Advanced Security Automation. https://tines.io”).
  • fail_on_status: (Optional) When true Tines will consider the agent run to have failed if the returned status code is anything other than 2xx. For example, if the target server returns a response with status code 404, the job will enter a retry cycle.
  • retry_on_status: (Optional) Specify the array of status codes that should cause a retry. If the HTTP response received by the agent has one of these codes, then it will be retried. If this option is included then fail_on_status will not affect retry behaviour.
    • Each array element can be either a single status code (e.g. 400), or a range of status codes (e.g. 400-499). Ranges are inclusive of starting and ending values.
    • The retry schedule consists of 25 retries with exponential back-off plus random “jitter”, starting at 5 seconds after the initial failure and gradually increasing to 10 minutes after the most recent failure, ie. [5, 10, 20, 40, 80, 160, 320, 600, 600, ...]. The jitter added is of random duration and up to (10 * (retry_count + 1)) seconds. Total back-off time over the 25 retries is approx. 3h 20mins, [5 * (2**retry_count), 10 * 60].min + (rand(10) * (retry_count + 1)).
  • log_error_on_status: [0, 400-499, 500-599] by default. Specify the array of status codes that should cause an error to be logged. If the HTTP response received by the agent has one of these codes, then an error will be logged. If this option is included then fail_on_status will not affect logging behaviour.
    • Each array element can be either a single status code (e.g. 400), or a range of status codes (e.g. 400-499). Ranges are inclusive of starting and ending values.
  • mutual_tls: (Optional) Credentials to use mutual TLS for the request. Must be an object with the following keys:

    • root_certificate: The root certificate for the certificate authority (CA) responsible for signatures
    • client_certificate: The certificate issued by the CA for this client
    • client_private_key: The private key for the client certificate

    For convenience, this can also be an interpolated Mutual TLS credential containing the required information.

Emitted Events

Events emitted by the HTTP Request Agent will include the ‘body’, ‘headers’ and response ‘code’ from the returned response. For example:

{
  "body": "ok",
  "headers": {
    "Date": "Mon, 1 Jan 2018 10:10:00 UTC",
    "Content-Type": "text/html; charset=utf-8",
    "Transfer-Encoding": "chunked",
    "Connection": "keep-alive",
    "Set-Cookie": "__cfduid=df0297dac2e4057e71e36fb67009723e91519037460; expires=Tue, 01-Jan-19 10:10:00 UTC; path=/; domain=.example.com; HttpOnly",
    "Via": "1.1 vegur",
    "Strict-Transport-Security": "max-age=15552000",
    "X-Content-Type-Options": "nosniff"
  },
  "status": 200
}

Example Configuration Options

The below samples use the postman-echo.com utility.

Send a simple GET request:

{
  "url": "https://postman-echo.com/get?foo1=bar1&foo2=bar2'",
  "method": "get"
}

Send a POST request with data from an incoming Event:

{
  "url": "https://postman-echo.com/post",
  "content_type": "json",
  "method": "post",
  "payload": {
    "user": "alice",
    "title": "{{ .person.title }}",
    "age": "85"
  },
  "headers": {}
}

Retry a request on 429 & 5xx errors, and log errors for other 4xx errors:

{
  "url": "https://postman-echo.com/post",
  "content_type": "json",
  "method": "post",
  "payload": {
    "user": "alice"
  },
  "retry_on_status": ["429", "500-599"],
  "log_error_on_status": ["400-428", "430-499"]
}

Send a request to a service that requires Basic authentication (password is accessed using the credential widget), include a custom header:

{
  "url": "https://postman-echo.com/basic-auth",
  "method": "get",
  "headers": {
    "X-Tines-Request": "123456"
  },
  "basic_auth": "postman:{% credential Postman %}"
}

Submit a file emitted as an attachment from an IMAP agent to Virustotal

{
  "url": "https://www.virustotal.com/vtapi/v2/file/scan",
  "content_type": "data",
  "method": "post",
  "payload": {
    "file": {
      "contents": "{{.get_email_with_attachment.attachments[0].base64encodedcontents | base64_decode}}",
      "filename": "{{.get_email_with_attachment.attachments[0].filename}}"
    },
    "apikey": "{% credential Virustotal %}"
  },
  "headers": {}
}