Ipfw: Write to ipfw divert socket failed: Message too long

1

Hi all

We’re using suricata 7.0.6 on OpenBSD 7.6

Suricata runs in worker runmode on a KVM virtual machine using virtio as network driver

The firewall is setup in ipfw divert mode so that suricata acts as IPS

Everything worked fine till the moment we upgraded from suricata 6 and OpenBSD 7.5 to the new setup.

After that as soon as the pf filtering is started suricata crashes with the following error:

> ipfw: Write to ipfw divert socket failed: Message too long

The relevant log is as follows:

82557 - Suricata-Main] 2024-10-21 12:39:43 Notice: suricata: This is Suricata version 7.0.6 RELEASE running in SYSTEM mode
[82557 - Suricata-Main] 2024-10-21 12:39:43 Info: cpu: CPUs/cores online: 4
[82557 - Suricata-Main] 2024-10-21 12:39:43 Info: exception-policy: master exception-policy set to: auto
[82557 - Suricata-Main] 2024-10-21 12:39:43 Info: suricata: Use pid file /var/run/suricata/suricata.pid from config file.
[4146 - Suricata-Main] 2024-10-21 12:39:43 Info: conf: Running in live mode, activating unix socket
[4146 - Suricata-Main] 2024-10-21 12:39:43 Info: logopenfile: fast output device (regular) initialized: fast.log
[4146 - Suricata-Main] 2024-10-21 12:39:43 Info: logopenfile: stats output device (regular) initialized: stats.log
[4146 - Suricata-Main] 2024-10-21 12:39:43 Info: alert-syslog: Syslog output initialized
[4146 - Suricata-Main] 2024-10-21 12:39:43 Info: suricata: Packets will start being processed before signatures are active.
[4146 - Suricata-Main] 2024-10-21 12:39:44 Info: unix-manager: unix socket '/var/run/suricata/suricata-command.socket'
[4146 - RX-700] 2024-10-21 12:39:44 Info: ipfw: Thread 'RX-700' will run on port 700 (item 0)
[4146 - Suricata-Main] 2024-10-21 12:39:44 Notice: threads: Threads created -> RX: 1 W: 8 TX: 1 FM: 1 FR: 1   Engine started.
[4146 - Suricata-Main] 2024-10-21 12:39:44 Notice: detect: rule reload starting
[4146 - Suricata-Main] 2024-10-21 12:39:47 Info: detect: 2 rule files processed. 45389 rules successfully loaded, 0 rules failed, 0
[4146 - Suricata-Main] 2024-10-21 12:39:47 Info: threshold-config: Threshold config parsed: 0 rule(s) found
[4146 - Suricata-Main] 2024-10-21 12:39:47 Info: detect: 45392 signatures processed. 1233 are IP-only rules, 3711 are inspecting packet payload, 40245 inspect application layer, 108 are decoder event only
[4146 - TX#00] 2024-10-21 12:39:48 Warning: ipfw: Write to ipfw divert socket failed: Message too long
[4146 - Suricata-Main] 2024-10-21 12:40:09 Notice: detect: rule reload complete
[4146 - Suricata-Main] 2024-10-21 12:40:09 Notice: suricata: Signature(s) loaded, Detect thread(s) activated.
[4146 - Suricata-Main] 2024-10-21 12:40:09 Error: threads: thread TX#00 failed

We did some checks and changes in config and googled around with no luck.

May someone suggest what to check next?

TIA for your help and best regards

2

Could you add the suricata.yaml and also the full run command for Suricata itself?

3

Hi and thank you for your answer

Command to run susricata

/usr/local/bin/suricata -D -d 700

The suricata.yaml is attached

I’m sure I’m missing something obvious but can’t see it
Thanks
suricata.yaml (84.8 KB)

4

One thought resulting from the Message too long error, have you disabled hardware offloads?

5

I did but since nothing changed I reverted it back

6

This is the loop log

[43591 - Suricata-Main] 2024-10-22 09:05:58 Error: threads: thread TX#00 failed
[54443 - Suricata-Main] 2024-10-22 09:06:01 Notice: suricata: This is Suricata version 7.0.6 RELEASE running in SYSTEM mode
[54443 - Suricata-Main] 2024-10-22 09:06:01 Info: cpu: CPUs/cores online: 4
[54443 - Suricata-Main] 2024-10-22 09:06:01 Info: exception-policy: master exception-policy set to: auto
[54443 - Suricata-Main] 2024-10-22 09:06:01 Info: suricata: Use pid file /var/run/suricata/suricata.pid from config file.
[58870 - Suricata-Main] 2024-10-22 09:06:01 Info: conf: Running in live mode, activating unix socket
[58870 - Suricata-Main] 2024-10-22 09:06:01 Info: logopenfile: fast output device (regular) initialized: fast.log
[58870 - Suricata-Main] 2024-10-22 09:06:01 Info: logopenfile: stats output device (regular) initialized: stats.log
[58870 - Suricata-Main] 2024-10-22 09:06:01 Info: alert-syslog: Syslog output initialized
[58870 - Suricata-Main] 2024-10-22 09:06:01 Info: suricata: Packets will start being processed before signatures are active.
[58870 - Suricata-Main] 2024-10-22 09:06:02 Info: unix-manager: unix socket '/var/run/suricata/suricata-command.socket'
[58870 - RX-700] 2024-10-22 09:06:02 Info: ipfw: Thread 'RX-700' will run on port 700 (item 0)
[58870 - Suricata-Main] 2024-10-22 09:06:02 Notice: threads: Threads created -> RX: 1 W: 8 TX: 1 FM: 1 FR: 1   Engine started.
[58870 - Suricata-Main] 2024-10-22 09:06:02 Notice: detect: rule reload starting
[58870 - TX#00] 2024-10-22 09:06:04 Warning: ipfw: Write to ipfw divert socket failed: Message too long
[58870 - Suricata-Main] 2024-10-22 09:06:05 Info: detect: 2 rule files processed. 45401 rules successfully loaded, 0 rules failed, 0
[58870 - Suricata-Main] 2024-10-22 09:06:05 Info: threshold-config: Threshold config parsed: 0 rule(s) found
[58870 - Suricata-Main] 2024-10-22 09:06:05 Info: detect: 45404 signatures processed. 1233 are IP-only rules, 3711 are inspecting packet payload, 40257 inspect application layer, 108 are decoder event only
[58870 - Suricata-Main] 2024-10-22 09:06:24 Notice: detect: rule reload complete
[58870 - Suricata-Main] 2024-10-22 09:06:24 Notice: suricata: Signature(s) loaded, Detect thread(s) activated.
[58870 - Suricata-Main] 2024-10-22 09:06:24 Error: threads: thread TX#00 failed

It keeps crashig :frowning:

7

Do you see any other log messages related to that timestamp with the error message?

Might be something OpenBSD specific

8

No other logs are present

I actually have a different installation (totally different) on a physical setup and it works

May it be something related to virtio dirver on OpenBSD?

Just a shot in the dark :frowning:

Very unfrtunatelly I can’t change it for performance reasons so at the moment suricata is excluded from pf filtering on the virtual setup

9

I think our divert support has only been developed and tested on FreeBSD, and I cannot remember any reports of it working on OpenBSD. So it may be impossible w/o code changes.

10

It worked fine till OpenBSD 7.4 and still works on OpenBSD 7.5 in physical setup.
Maybe OpenBSD changed something that breaks divert socket …