I am currently using Suricata version 6.0.14, which I installed from source using make and make install on a Debian 11 system. Could anyone please help me adjust the configuration to avoid packet drops? Any suggestions or tips on optimizing the setup would be greatly appreciated.
Average network traffic ~1Gbps.
This is the config file that I’m using
suricata (1).yaml (72.7 KB)
Before continuing, you should strongly consider to upgrade to Suricata 7.0.7 (or any of the recent 7.0.x releases) as 6.0.14 is no longer supported.
Hardware specs and NIC details as well as the packet capture method will be needed to make any progress.
However, upgrading Suricata should be done before additional analysis is performed.
Thank you for your response. However, at the moment, my system cannot be upgraded to the latest version due to some external reasons. Could you please assist me with version 6.0.14 for now?
Post your hardware details, especially CPU, Memory, NIC and ideally the stats.log
Here is all the information I can provide, including details of the CPU, interface, memory, and stats.log.
Note: I am currently running Suricata with 16 CPUs, but packet drops are still occurring.
Does the swap memory always being at 100% affect Suricata’s packet drop? Please clarify this for me.
System Info.zip (207.4 KB)
100% swap is always bad but depends on the rest of the settings on the system.
What NIC is that? run lspci | grep Ethernet
And if you look into the stats.log, most of the time the kernel_packets increase while the kernel_drops are stable, there are just some timeframes where the drops increase as well. Maybe some sort of traffic burst or elephant flow.
You can run htop and observe if you can correlate it to those burst if it happens, so if the drops start to increase in a short timeframe if there is one cpu occupied all the time.
Here’s the output of lspci | grep Ethernet:
lspci | grep Ethernet
19:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
19:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
19:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
19:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
af:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
af:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
[quote="Andreas Herz, post:6, topic:4884, username:Andreas_Herz"]
Maybe some sort of traffic burst or elephant flow.
[/quote]
How can I handle with case traffic burst or elephant flow?I would first try to observe the pattern and see if you can relate it to such bursts, feel free to post the output of htop as well.
If you spot some elephant flows you could try to prefilter those or throttle the burst, depending on how the forwarding looks like.
I mean, how can I configure suricata.yaml to handle these case (increase Suricata’s RAM/CPU,…)? Sorry if I misunderstood something.
You need to find the root cause first as I said run htop and observe the stats and see if you can narrow it down to burst/elephant flows.
Ideally you can filter some traffic from your mirror port, packetbroker or what device you use to forward the traffic.
In some cases you can use bpf filter in the interface section for Suricata to filter some stuff early.
1 Like
Tracking down performance issues often involves several steps. By using htop and viewing the Suricata threads, it’ll be easy to see how the worker threads (named W#nn) are utilized. An elephant flow (large flow, sometimes high speed) or a burst can show up in a few different ways but often it’ll have a majority of cpu time. I suggest using htop, choosing setup (F2), and choose the option to “show custom thread names”.
The memory and CPU resources seem sufficient to handle 1Gbps with out issue if the traffic is “well balanced”. Elephant flows and bursts are inevitable and we sometimes find that elephant flows contain “trusted/known” traffic that is a candidate for not processing by adding the information on the system(s) creating the elephant flows to the BPF such that they’re not processed by Suricata
1 Like