How fix “*** buffer overflow detected ***: suricata terminated aborted (core dumped)” before run suricatasc
Please include some more details, like suricata version, how you installed, start up options, etc.
Suricata v8.0.0…compiled and runned with dpdk support
I think that problem is in mempool and mtu values…but how calculate the optimized mempool values without drop packets…My server has 96G ram, 2 Numa with 16 threads each, 4 ports in IPS mode with 7 threads each, 4 cpu to management and 28 to workers in high prio(cpu affinity)
Please include more details: dpdk yaml section, start up options, etc.
dpdk:
eal-params:
proc-type: primary
interface
- interface: 0000:01:00.0
threads: 7
promisc: true
multicast: true
checksum-checks: false
checksum-checks-offload: false
mtu: 9000
mempool-size: 262143
mempool-cache-size: 511
rx-descriptors: 8192
tx-descriptors: 8192
copy-mode: ips
copy-iface: 0000:01:00.1 - interface: 0000:02:00.0
threads: 7
promisc: true
multicast: true
checksum-checks: false
checksum-checks-offload: false
mtu: 9000
mempool-size: 262143
mempool-cache-size: 511
rx-descriptors: 8192
tx-descriptors: 8192
copy-mode: ips
copy-iface: 0000:02:00.1 - interface: 0000:01:00.1
threads: 7
promisc: true
multicast: true
checksum-checks: false
checksum-checks-offload: false
mtu: 9000
mempool-size: 262143
mempool-cache-size: 511
rx-descriptors: 8192
tx-descriptors: 8192
copy-mode: ips
copy-iface: 0000:01:00.0 - interface: 0000:02:00.1
threads: 7
promisc: true
multicast: true
checksum-checks: false
checksum-checks-offload: false
mtu: 9000
mempool-size: 262143
mempool-cache-size: 511
rx-descriptors: 8192
tx-descriptors: 8192
copy-mode: ips
copy-iface: 0000:02:00.0 - interface: default
threads: auto
promisc: true
multicast: true
checksum-checks: true
checksum-checks-offload: true
mtu: 1500
rss-hash-functions: auto
mempool-size: 65535
mempool-cache-size: 257
rx-descriptors: 1024
tx-descriptors: 1024
copy-mode: none
copy-iface: none
runmode: workers
threading:
set-cpu-affinity: yes
cpu-affinity:
- management-cpu-set:
cpu: [ 0,8,16,24 ] # include only these CPUs in affinity settings
- receive-cpu-set:
cpu: [ 0,8,16,24 ] # include only these CPUs in affinity settings
- worker-cpu-set:
cpu: [ “1-7”,“9-15”,“17-23”,“25-31” ]
mode: “exclusive”
prio:
high: [ “1-7”,“9-15”,“17-23”,“25-31” ]
default: “high”
Port 01:00.0 and 01:00.1 on Numa 0 and Port 02:00.0 and 02:00.1 on Numa 1
I’m running suricata with support dpdk and libtcmalloc Google library:
LD_PRELOAD=“/usr/lib64/libtcmalloc_minimal.so.4” suricata -c /etc/suricata/suricata.yaml --dpdk
With this configuration I have less number of dropped packets but when I run suricatasc it kills the suricata process and return this output
*** buffer overflow detected ***: suricata terminated aborted (core dumped)
and I need to run suricatasc every 10 sec to see the interface statistics
Currently, I have no advice for you, only questions.
Can you try to allocate 1 GB hugepages instead of 2MB?
What happens when you omit the tcmalloc from your setup? Does it happen in IDS mode?
Can you tell from where the *** buffer overflow detected *** message comes from? Perhaps, could you inspect the core dump and provide more info?
Can you find a minimal reproducible example? E.g. can you try to reduce the number of threads or can you try to substitute your physical interfaces with net_null interfaces? Your dpdk section would then look like:
dpdk:
eal-params:
proc-type: primary
vdev: ["net_null0", "net_null1", "net_null2", "net_null3"]
interfaces:
- interface: net_null1
threads: 3
...
Thank you.
In the worst-case scenario, you can iteratively obtain stats from eve.json
fixed with dpdk-hugepages.py --pagesize 1G --setup 8G
thanks
Another question.
I would like to take advantage of the number of threads (increase to 14 instead 7) by making bonding interfaces in my dpdk configuration. In that case I think the correct configuration would be:
vdev: [‘net_bonding0,mode=5,slave=0000:04:00.0,slave=0000:82:00.0’,‘net_bonding1,mode=5,slave=0000:04:00.1,slave=0000:82:00.1’] right?
but. How would the cpu-affinity configuration to avoid errors such as " NIC is on NUMA -1, 7 threads on different NUMA node"
I don’t think Suricata can approach this problem easily.
You can try to disable Hyperthreading so that you only have “physical” cores. Sometimes it even increases performance with DPDK.
Alternatively, you can launch two Suricata processes, one on each NUMA node - in the DPDK EAL settings you would differentiate between the processes with the file-prefix: option.
Btw considering your NICs are on different NUMA nodes - wouldn’t it be possible/better to rewire your server so that 82:00.0 and 82:00.0 would be peers on one NUMA node and 04:00.0 with 04:00.1 on the other. The whole idea about being NUMA local is to have that complete pipeline to have on one NUMA node - from packet RX, through processing, to TX - and that is violated in your case as you have 1 port on one NUMA node and the other (copy) port on the other NUMA node.
If I have 04:00.0 and 04:00.1 in a bonding I would have the same problem.
In this case it would be:
[‘net_bonding0,mode=5,slave=0000:04:00.0,slave=0000:04:00.1’,‘net_bonding1,mode=5,slave=0000:82:00.0,slave=0000:82:00.1’] right ?
but the threads are loaded consecutively… I mean:
for net_bonding0 →
- 0000:04:00.0 [1-7]
- 0000:04:00.1 [9-15](these threads correspond to another Numa)
My question is this:
Can I assign threads non-consecutively?
for example:
for net_bonding0 →
- 0000:04:00.0 [1-7]
- 0000:04:00.1 [17-23]
total 14 threads
It would only apply to the TAP/IPS scenario where (meaning, not creating a bonding interface):
04:00.0 is assigned with CPUs 0-7
82:00.0 is assigned with CPUs 8-15
04:00.1 is assigned with CPUs 16-23
82:00.1 is assigned with CPUs 24-31
With regards to the better CPU assignment to the individual interfaces, I’ve created a feature ticket for myself and will look into it.