Performance of Lua-Output in high-alert throughput

Hi, I’m planning on using Suricata for HTTP traffic analysis and high-throughput logging.

In my use-case, I need to identify HTTP transactions (request+response) based on some conditions and log them for further processing by a Vector Daemon for log ingestion and forwarding.

During some local testing I’ve encountered some challenges:

  1. I couldn’t find a way to use the EVE-HTTP configuration to log the request/response payloads.

  2. I couldn’t find a way to use the EVE-ALERT configuration to log the request/response headers.

  3. I do have dump-all-headers within EVE-HTTP, and I do have http-body/http-body-printable within EVE-ALERT, but it seems that they are mutually exclusive. I couldn’t find a way to have them both in a single log.

Due to the above, I’m resorting to using the Lua Output module, where I can get access to all the required fields, however, reading the following documentation I’m concerned about the possible performance implications and bottlenecks:

  # 'luajit.states' is used to control how many states are preallocated.
  # State use: per detect script: 1 per detect thread. Per output script: 1 per
  # script.
  • Do I understand correctly that the Lua output scripts will only run in single thread per script?
  • Is there a way to configure a Lua-Output script to run concurrently?
  • Are there any other ideas on optimizing the system for a high-alert-throughput workload?

Thanks!

What version of Suricata are you using and how are you forwarding the traffic?
Can you also share your suricata.yaml and some example output?
The http EVE output should be quite verbose if you enable all the options present in the yaml.

@Andreas_Herz apologies for the late reply. Please see below the configuration and output samples.

What version of Suricata are you using and how are you forwarding the traffic?

I’m running Suricata version 7.0.0-beta1

I’m forwarding the traffic by connecting to the network interface, i.e $ suricata -i eth0

Can you also share your suricata.yaml and some example output?

Sure, here’s what my suricata.yaml looks like (taking out any irrelevant parts) :

outputs:
  - eve-log:
      enabled: yes
      filetype: regular
      filename: eve.json
      types:
        - alert:
            payload-printable: no # irrelevant, see P.S. below
            http-body-printable: yes
            dump-all-headers: both # has no effect here (taken from "types:http")
        - http:
            http-body-printable: yes # has no effect here (taken from "types:alert")
            payload-printable: yes # has no effect here (taken from "types:alert")
            enabled: yes
            extended: yes
            dump-all-headers: both

In order to trigger the `eve-log:types:alert" output I’ve created the following rule:

alert http any any -> any any ( \
    msg: "Successful HTTP response"; \
    http.stat_code; \
    content: "2"; \
    startswith; \
    sid: 1; \
)

I’m running the following command to trigger a single eve-output:

curl -H 'content-type: application/json' -X POST -d'{\"one\": 2,\n\"foobar\":1\n}' -H 'accept: application/json' http://foass.1001010.com/

Inside eve.json I can see two separate log lines, one for each output rule.
First, alert is logged:

{
  "timestamp": "2023-01-22T17:26:59.049806+0000",
  "flow_id": 1581203739460706,
  "in_iface": "eth0",
  "event_type": "alert",
  "src_ip": "64.111.122.131",
  "src_port": 80,
  "dest_ip": "192.168.65.3",
  "dest_port": 62978,
  "proto": "TCP",
  "pkt_src": "wire/pcap",
  "tx_id": 0,
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 1,
    "rev": 0,
    "signature": "Successful HTTP response",
    "category": "",
    "severity": 3
  },
  "http": {
    "hostname": "foass.1001010.com",
    "url": "/",
    "http_user_agent": "curl/7.76.1",
    "http_content_type": "application/json",
    "http_method": "POST",
    "protocol": "HTTP/1.1",
    "status": 200,
    "length": 129,
    "http_request_body_printable": "{\"one\": 2,\\n\"bar\":2\\n}",
    "http_response_body_printable": "{\n  \"message\": \"What the fuck sort of call was that?  Next time, check your fucking typing\", \n  \"subtitle\": \"The Server\"\n}\n"
  },
  "files": [
    {
      "filename": "/",
      "gaps": false,
      "state": "UNKNOWN",
      "stored": false,
      "size": 123,
      "tx_id": 0
    }
  ],
  "app_proto": "http",
  "direction": "to_client",
  "flow": {
    "pkts_toserver": 4,
    "pkts_toclient": 4,
    "bytes_toserver": 405,
    "bytes_toclient": 608,
    "start": "2023-01-22T17:26:53.630296+0000",
    "src_ip": "192.168.65.3",
    "dest_ip": "64.111.122.131",
    "src_port": 62978,
    "dest_port": 80
  }
}

Second http is logged:

{
  "timestamp": "2023-01-22T17:26:59.063414+0000",
  "flow_id": 1581203739460706,
  "in_iface": "eth0",
  "event_type": "http",
  "src_ip": "192.168.65.3",
  "src_port": 62978,
  "dest_ip": "64.111.122.131",
  "dest_port": 80,
  "proto": "TCP",
  "pkt_src": "wire/pcap",
  "tx_id": 0,
  "http": {
    "hostname": "foass.1001010.com",
    "url": "/",
    "http_user_agent": "curl/7.76.1",
    "http_content_type": "application/json",
    "accept": "application/json",
    "content_type": "application/json",
    "http_method": "POST",
    "protocol": "HTTP/1.1",
    "status": 200,
    "length": 129,
    "request_headers": [
      {
        "name": "Host",
        "value": "foass.1001010.com"
      },
      {
        "name": "User-Agent",
        "value": "curl/7.76.1"
      },
      {
        "name": "content-type",
        "value": "application/json"
      },
      {
        "name": "accept",
        "value": "application/json"
      },
      {
        "name": "Content-Length",
        "value": "22"
      }
    ],
    "response_headers": [
      {
        "name": "Date",
        "value": "Sun, 22 Jan 2023 17:26:53 GMT"
      },
      {
        "name": "Server",
        "value": "Apache"
      },
      {
        "name": "Upgrade",
        "value": "h2"
      },
      {
        "name": "Connection",
        "value": "Upgrade"
      },
      {
        "name": "Cache-Control",
        "value": "max-age=172800"
      },
      {
        "name": "Expires",
        "value": "Tue, 24 Jan 2023 17:26:53 GMT"
      },
      {
        "name": "Vary",
        "value": "User-Agent"
      },
      {
        "name": "Transfer-Encoding",
        "value": "chunked"
      },
      {
        "name": "Content-Type",
        "value": "application/json"
      }
    ]
  }
}

As you can see, alert log has a printable body but no headers while the http log has the headers in a convenient format, but no body.

P.S

I did, however find that the option eve-log:{ types:alert: { payload-printable: yes } } enables logging the response payload, however, it includes only the response. In my use-case I need both.

{
  "timestamp": "2023-01-22T17:41:48.575909+0000",
  "flow_id": 2188154957904930,
  "in_iface": "eth0",
  "event_type": "alert",
  "src_ip": "64.111.122.131",
  "src_port": 80,
  "dest_ip": "192.168.65.3",
  "dest_port": 61400,
  "proto": "TCP",
  "pkt_src": "wire/pcap",
  "tx_id": 0,
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 1,
    "rev": 0,
    "signature": "Successful HTTP response",
    "category": "",
    "severity": 3
  },
  "http": {
    "hostname": "foass.1001010.com",
    "url": "/",
    "http_user_agent": "curl/7.76.1",
    "http_content_type": "application/json",
    "http_method": "POST",
    "protocol": "HTTP/1.1",
    "status": 200,
    "length": 129,
    "http_request_body_printable": "{\"one\": 2,\\n\"bar\":2\\n}",
    "http_response_body_printable": "{\n  \"message\": \"What the fuck sort of call was that?  Next time, check your fucking typing\", \n  \"subtitle\": \"The Server\"\n}\n"
  },
  "files": [
    {
      "filename": "/",
      "gaps": false,
      "state": "UNKNOWN",
      "stored": false,
      "size": 123,
      "tx_id": 0
    }
  ],
  "app_proto": "http",
  "direction": "to_client",
  "flow": {
    "pkts_toserver": 4,
    "pkts_toclient": 4,
    "bytes_toserver": 405,
    "bytes_toclient": 608,
    "start": "2023-01-22T17:41:43.181789+0000",
    "src_ip": "192.168.65.3",
    "dest_ip": "64.111.122.131",
    "src_port": 61400,
    "dest_port": 80
  },
  "payload_printable": "HTTP/1.1 200 OK\r\nDate: Sun, 22 Jan 2023 17:41:43 GMT\r\nServer: Apache\r\nUpgrade: h2\r\nConnection: Upgrade\r\nCache-Control: max-age=172800\r\nExpires: Tue, 24 Jan 2023 17:41:43 GMT\r\nVary: User-Agent\r\nTransfer-Encoding: chunked\r\nContent-Type: application/json\r\n\r\n7b\r\n{\n  \"message\": \"What the **** sort of call was that?  Next time, check your ****** typing\", \n  \"subtitle\": \"The Server\"\n}\n\r\n",
  "stream": 1
}

I’m don’t understand what is the conclusion here.

Can Suricata include full payload and headers of HTTP requests and responses in a single log without involving Lua Output scripting?

Can anybody confirm or deny that Lua Output scripts can only run on a single thread per script?

If anybody comes across this issue, I found that Zeek implements the exact functionality I was looking for pretty much out of the box.

https://docs.zeek.org/en/master/scripts/base/protocols/http/main.zeek.html#namespace-HTTP