# Performance of Lua-Output in high-alert throughput

**URL:** <https://forum.suricata.io/t/performance-of-lua-output-in-high-alert-throughput/3134>\
**Category:** Developers\
**Tags:** rules, suricata, lua\
**Created:** [January 16, 2023, 5:52pm UTC](https://forum.suricata.io/t/performance-of-lua-output-in-high-alert-throughput/3134 "2023-01-16T17:52:57Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![sapien](https://avatars.discourse-cdn.com/v4/letter/s/65b543/32.png) [@sapien](https://forum.suricata.io/u/sapien)\
**Post date:** [January 16, 2023, 5:52pm UTC](https://forum.suricata.io/t/performance-of-lua-output-in-high-alert-throughput/3134/1 "2023-01-16T17:52:57Z")

</div>

Hi, I’m planning on using Suricata for HTTP traffic analysis and high-throughput logging.

In my use-case, I need to identify HTTP transactions (request+response) based on some conditions and log them for further processing by a Vector Daemon for log ingestion and forwarding.

During some local testing I’ve encountered some challenges:

1. I couldn’t find a way to use the _EVE-HTTP_ configuration to log the request/response **payloads**.

2. I couldn’t find a way to use the _EVE-ALERT_ configuration to log the request/response **headers**.

3. I **do** have `dump-all-headers` within _EVE-HTTP_, and I **do** have `http-body/http-body-printable` within EVE-ALERT, but it seems that they are mutually exclusive. I couldn’t find a way to have them both in a single log.

Due to the above, I’m resorting to using the _Lua Output_ module, where I can get access to all the required fields, however, reading the following documentation I’m concerned about the possible performance implications and bottlenecks:

```auto
  # 'luajit.states' is used to control how many states are preallocated.
  # State use: per detect script: 1 per detect thread. Per output script: 1 per
  # script.

```

- Do I understand correctly that the Lua output scripts will only run in single thread per script?
- Is there a way to configure a Lua-Output script to run concurrently?
- Are there any other ideas on optimizing the system for a high-alert-throughput workload?

Thanks!

---

<div class="post-metadata">

**Author:** ![Andreas\_Herz](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.suricata.io/andreas_herz/32/52_2.png) [@Andreas\_Herz](https://forum.suricata.io/u/Andreas_Herz)\
**Post date:** [January 17, 2023, 8:21pm UTC](https://forum.suricata.io/t/performance-of-lua-output-in-high-alert-throughput/3134/2 "2023-01-17T20:21:52Z")

</div>

What version of Suricata are you using and how are you forwarding the traffic?  
Can you also share your suricata.yaml and some example output?  
The http EVE output should be quite verbose if you enable all the options present in the yaml.

---

<div class="post-metadata">

**Author:** ![sapien](https://avatars.discourse-cdn.com/v4/letter/s/65b543/32.png) [@sapien](https://forum.suricata.io/u/sapien)\
**Post date:** [January 22, 2023, 5:50pm UTC](https://forum.suricata.io/t/performance-of-lua-output-in-high-alert-throughput/3134/3 "2023-01-22T17:50:00Z")

</div>

@Andreas_Herz apologies for the late reply. Please see below the configuration and output samples.

> What version of Suricata are you using and how are you forwarding the traffic?

I’m running Suricata version `7.0.0-beta1`

I’m forwarding the traffic by connecting to the network interface, i.e `$ suricata -i eth0`

> Can you also share your suricata.yaml and some example output?

Sure, here’s what my `suricata.yaml` looks like (taking out any irrelevant parts) :

```auto
outputs:
  - eve-log:
      enabled: yes
      filetype: regular
      filename: eve.json
      types:
        - alert:
            payload-printable: no # irrelevant, see P.S. below
            http-body-printable: yes
            dump-all-headers: both # has no effect here (taken from "types:http")
        - http:
            http-body-printable: yes # has no effect here (taken from "types:alert")
            payload-printable: yes # has no effect here (taken from "types:alert")
            enabled: yes
            extended: yes
            dump-all-headers: both

```

In order to trigger the `eve-log:types:alert" output I’ve created the following rule:

```auto
alert http any any -> any any ( \
    msg: "Successful HTTP response"; \
    http.stat_code; \
    content: "2"; \
    startswith; \
    sid: 1; \
)

```

I’m running the following command to trigger a single `eve-output`:

`curl -H 'content-type: application/json' -X POST -d'{\"one\": 2,\n\"foobar\":1\n}' -H 'accept: application/json' http://foass.1001010.com/`

Inside `eve.json` I can see two separate log lines, one for each output rule.  
First, `alert` is logged:

```auto
{
  "timestamp": "2023-01-22T17:26:59.049806+0000",
  "flow_id": 1581203739460706,
  "in_iface": "eth0",
  "event_type": "alert",
  "src_ip": "64.111.122.131",
  "src_port": 80,
  "dest_ip": "192.168.65.3",
  "dest_port": 62978,
  "proto": "TCP",
  "pkt_src": "wire/pcap",
  "tx_id": 0,
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 1,
    "rev": 0,
    "signature": "Successful HTTP response",
    "category": "",
    "severity": 3
  },
  "http": {
    "hostname": "foass.1001010.com",
    "url": "/",
    "http_user_agent": "curl/7.76.1",
    "http_content_type": "application/json",
    "http_method": "POST",
    "protocol": "HTTP/1.1",
    "status": 200,
    "length": 129,
    "http_request_body_printable": "{\"one\": 2,\\n\"bar\":2\\n}",
    "http_response_body_printable": "{\n \"message\": \"What the fuck sort of call was that? Next time, check your fucking typing\", \n \"subtitle\": \"The Server\"\n}\n"
  },
  "files": [
    {
      "filename": "/",
      "gaps": false,
      "state": "UNKNOWN",
      "stored": false,
      "size": 123,
      "tx_id": 0
    }
  ],
  "app_proto": "http",
  "direction": "to_client",
  "flow": {
    "pkts_toserver": 4,
    "pkts_toclient": 4,
    "bytes_toserver": 405,
    "bytes_toclient": 608,
    "start": "2023-01-22T17:26:53.630296+0000",
    "src_ip": "192.168.65.3",
    "dest_ip": "64.111.122.131",
    "src_port": 62978,
    "dest_port": 80
  }
}

```

Second `http` is logged:

```auto
{
  "timestamp": "2023-01-22T17:26:59.063414+0000",
  "flow_id": 1581203739460706,
  "in_iface": "eth0",
  "event_type": "http",
  "src_ip": "192.168.65.3",
  "src_port": 62978,
  "dest_ip": "64.111.122.131",
  "dest_port": 80,
  "proto": "TCP",
  "pkt_src": "wire/pcap",
  "tx_id": 0,
  "http": {
    "hostname": "foass.1001010.com",
    "url": "/",
    "http_user_agent": "curl/7.76.1",
    "http_content_type": "application/json",
    "accept": "application/json",
    "content_type": "application/json",
    "http_method": "POST",
    "protocol": "HTTP/1.1",
    "status": 200,
    "length": 129,
    "request_headers": [
      {
        "name": "Host",
        "value": "foass.1001010.com"
      },
      {
        "name": "User-Agent",
        "value": "curl/7.76.1"
      },
      {
        "name": "content-type",
        "value": "application/json"
      },
      {
        "name": "accept",
        "value": "application/json"
      },
      {
        "name": "Content-Length",
        "value": "22"
      }
    ],
    "response_headers": [
      {
        "name": "Date",
        "value": "Sun, 22 Jan 2023 17:26:53 GMT"
      },
      {
        "name": "Server",
        "value": "Apache"
      },
      {
        "name": "Upgrade",
        "value": "h2"
      },
      {
        "name": "Connection",
        "value": "Upgrade"
      },
      {
        "name": "Cache-Control",
        "value": "max-age=172800"
      },
      {
        "name": "Expires",
        "value": "Tue, 24 Jan 2023 17:26:53 GMT"
      },
      {
        "name": "Vary",
        "value": "User-Agent"
      },
      {
        "name": "Transfer-Encoding",
        "value": "chunked"
      },
      {
        "name": "Content-Type",
        "value": "application/json"
      }
    ]
  }
}

```

As you can see, `alert` log has a printable `body` but no `headers` while the `http` log has the headers in a convenient format, but no `body`.

**P.S**

I did, however find that the option `eve-log:{ types:alert: { payload-printable: yes } }` enables logging the **response** payload, however, it includes **only** the **response**. In my use-case I need both.

```auto
{
  "timestamp": "2023-01-22T17:41:48.575909+0000",
  "flow_id": 2188154957904930,
  "in_iface": "eth0",
  "event_type": "alert",
  "src_ip": "64.111.122.131",
  "src_port": 80,
  "dest_ip": "192.168.65.3",
  "dest_port": 61400,
  "proto": "TCP",
  "pkt_src": "wire/pcap",
  "tx_id": 0,
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 1,
    "rev": 0,
    "signature": "Successful HTTP response",
    "category": "",
    "severity": 3
  },
  "http": {
    "hostname": "foass.1001010.com",
    "url": "/",
    "http_user_agent": "curl/7.76.1",
    "http_content_type": "application/json",
    "http_method": "POST",
    "protocol": "HTTP/1.1",
    "status": 200,
    "length": 129,
    "http_request_body_printable": "{\"one\": 2,\\n\"bar\":2\\n}",
    "http_response_body_printable": "{\n \"message\": \"What the fuck sort of call was that? Next time, check your fucking typing\", \n \"subtitle\": \"The Server\"\n}\n"
  },
  "files": [
    {
      "filename": "/",
      "gaps": false,
      "state": "UNKNOWN",
      "stored": false,
      "size": 123,
      "tx_id": 0
    }
  ],
  "app_proto": "http",
  "direction": "to_client",
  "flow": {
    "pkts_toserver": 4,
    "pkts_toclient": 4,
    "bytes_toserver": 405,
    "bytes_toclient": 608,
    "start": "2023-01-22T17:41:43.181789+0000",
    "src_ip": "192.168.65.3",
    "dest_ip": "64.111.122.131",
    "src_port": 61400,
    "dest_port": 80
  },
  "payload_printable": "HTTP/1.1 200 OK\r\nDate: Sun, 22 Jan 2023 17:41:43 GMT\r\nServer: Apache\r\nUpgrade: h2\r\nConnection: Upgrade\r\nCache-Control: max-age=172800\r\nExpires: Tue, 24 Jan 2023 17:41:43 GMT\r\nVary: User-Agent\r\nTransfer-Encoding: chunked\r\nContent-Type: application/json\r\n\r\n7b\r\n{\n \"message\": \"What the ****sort of call was that? Next time, check your****** typing\", \n \"subtitle\": \"The Server\"\n}\n\r\n",
  "stream": 1
}

```

---

<div class="post-metadata">

**Author:** ![sapien](https://avatars.discourse-cdn.com/v4/letter/s/65b543/32.png) [@sapien](https://forum.suricata.io/u/sapien)\
**Post date:** [February 1, 2023, 7:03pm UTC](https://forum.suricata.io/t/performance-of-lua-output-in-high-alert-throughput/3134/4 "2023-02-01T19:03:25Z")

</div>

I’m don’t understand what is the conclusion here.

Can Suricata include full payload and headers of HTTP requests and responses in a single log without involving Lua Output scripting?

Can anybody confirm or deny that Lua Output scripts can only run on a single thread per script?

---

<div class="post-metadata">

**Author:** ![sapien](https://avatars.discourse-cdn.com/v4/letter/s/65b543/32.png) [@sapien](https://forum.suricata.io/u/sapien)\
**Post date:** [February 2, 2023, 5:48pm UTC](https://forum.suricata.io/t/performance-of-lua-output-in-high-alert-throughput/3134/5 "2023-02-02T17:48:59Z")

</div>

If anybody comes across this issue, I found that Zeek implements the exact functionality I was looking for pretty much out of the box.

[https://docs.zeek.org/en/master/scripts/base/protocols/http/main.zeek.html#namespace-HTTP](https://docs.zeek.org/en/master/scripts/base/protocols/http/main.zeek.html#namespace-HTTP)
