High CPU usage with 100mbps, 100 parallel connections

Help
1
  • Suricata version: 6.0.20
  • Operating system and/or Linux distribution: Operating system and/or Linux distribution: Oracle Linux 9 (x86_64) UEK Release 7
  • How you installed Suricata - el repository

Hello,

I am running suricata on Oracle Linux 9 (x86_64) UEK Release 7 with 4 (v)CPUs.

It has three NICs. I have set suricata to monitor two of it’s interfaces using following command line:

/sbin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet=eth0 --af-packet=nic3 --user suricata src host or dst host or src host or dst host

The bpf filter is based on suggestion from https://forum.suricata.io/t/ignoring-traffic-when-suricata-machine-is-router

I run iperf server on said VM with command line “iperf -s”
From remote machine I run command: iperf -c -l 64 -b 1m -t 1000000000 -P 100

Upon running CPU usage of suricata jumps to approximately 50 - 60 %.

Htop gives following results:

image
image1644×224 21.4 KB

perf top -p $(pidof suricata) gives following results:

image
image974×922 230 KB

stats.log is following:
stats.log (15.5 KB)

The only rules loaded are the ones that come with installation of suricata i.e. I have not configured any sources.

The above cpu is almost identical even if I remove suricata.rules file.

What can I do so that CPU usage of suricata stays below 10%?

Thanks in advance!

2

Please upgrade to Suricata 7.0.7 first since Suricata 6 is EOL.

In addition to that please post your suricata.yaml and suricata.log.

Ideally add the lscpu output so see which CPU is used and ethtool -i on the both NICs being used.

3

Hi @Andreas_Herz ,

Thanks! I will get that information shortly. Could you please till then point to the documentation\link that denote that suricata 6 is EOL. I need that information to create user story :slight_smile:

Thanks in advance!

4
1 Like
5

I got your files via DM, but ideally attach those here so others could help or benefit as well.

This looks like a VM, what virtualization solution is used there?

You did not specify specifics on the af-packet section in the suricata.yaml. I would recommend to read through or docs on how to configure this section (Suricata User Guide — Suricata 7.0.7 documentation)

Runmode workers for example could result in higher performance and less CPU usage.

Besides that iperf is not a realistic nor ideal tool to test Suricata.

6

Hello @Andreas_Herz ,

This looks like a VM, what virtualization solution is used there?

We are using openstack\kvm.

You did not specify specifics on the af-packet section in the suricata.yaml

Apart from specifying af-packet on command line, I have not made any other changes. Is there anything specific you suggest I should look at?

Runmode workers for example could result in higher performance and less CPU usage.

Can it be substantially big enough to bring CPU usage from 50-70% to 10%?

Besides that iperf is not a realistic nor ideal tool to test Suricata.

We are using iperf to test load on the server, not for suricata. We noticed that suricata is taking big chunk of cpu while doing that test. We need to keep on using iperf for performance testing.

Thanks!

7

As I said, read our documentation. There are several parts, especially the af-packet section, see 12.1. Suricata.yaml — Suricata 7.0.7 documentation and 11. Performance — Suricata 7.0.7 documentation

This depends on the traffic you forward and the amount of signatures. If there is some solid traffic I would expect the CPU to be pretty occupied which is expected. Based on your stats.log you are already running into packet drops.

The iperf traffic pattern is not very realistic for a IDS/NSM like Suricata. But it is expected for Suricata to use quite some CPU resource for all the detection it is doing.