Incorrect check of the number of required cores for suricata dpdk/ips mode

Alexander · July 20, 2023, 6:18pm

Hello guys,

I am trying to start the suricata in ips mode with 2 threads per interface and can’t do it.

My computer has 8 cores.
suricata : latest master
dpdk: 23.03.0

configuration in suricata.yaml:

dpdk:
eal-params:
proc-type: primary

interfaces:
- interface: 0000:01:00.2 # PCIe address of the NIC port
threads: 2
promisc: true # promiscuous mode - capture all packets
multicast: true # enables also detection on multicast packets
checksum-checks: true # if Suricata should validate checksums
checksum-checks-offload: true # if possible offload checksum validation to the NIC (saves Suricata resources)
mtu: 1500 # Set MTU of the device in bytes
mempool-size: 65535 # The number of elements in the mbuf pool
mempool-cache-size: 257
rx-descriptors: 1024
tx-descriptors: 1024
copy-mode: ips
copy-iface: 0000:01:00.3 # or PCIe address of the second interface

- interface: 0000:01:00.3 # PCIe address of the NIC port
  threads: 2
  promisc: true # promiscuous mode - capture all packets
  multicast: true # enables also detection on multicast packets
  checksum-checks: true # if Suricata should validate checksums
  checksum-checks-offload: true # if possible offload checksum validation to the NIC (saves Suricata resources)
  mtu: 1500 # Set MTU of the device in bytes
  mempool-size: 65535 # The number of elements in the mbuf pool
  mempool-cache-size: 257
  rx-descriptors: 1024
  tx-descriptors: 1024
  copy-mode: ips
  copy-iface: 0000:01:00.2 # or PCIe address of the second interface

- interface: default
  threads: auto
  promisc: true
  multicast: true
  checksum-checks: true
  checksum-checks-offload: true
  mtu: 1500
  rss-hash-functions: auto
  mempool-size: 65535
  mempool-cache-size: 257
  rx-descriptors: 1024
  tx-descriptors: 1024
  copy-mode: none
  copy-iface: none

The Problem: Suricata can’t start:

sudo LD_LIBRARY_PATH=/usr/local/lib/x86_64-linux-gnu/ /usr/local/bin/suricata --dpdk
…
Error: dpdk: Interfaces requested more cores than configured in the threading section [ConfigSetThreads:runmode-dpdk.c:439]
Error: dpdk: 0000:01:00.2: fail to load config of interface [DeviceValidateOutIfaceConfig:runmode-dpdk.c:1240]
Error: dpdk: 0000:01:00.3: failed to configure [ParseDpdkConfigAndConfigureDevice:runmode-dpdk.c:1520]

I added tracing to the function static int ConfigSetThreads(DPDKIfaceConfig *iconf, const char *entry_str) (from runmode-dpdk.c )

static int ConfigSetThreads(DPDKIfaceConfig *iconf, const char *entry_str)
{
//ADymov
SCLogError("ConfigSetThreads for interface "%s" ", iconf->iface);
//ADymov

…
…
…

//ADymov
total_cpus += iconf->threads;
if (total_cpus > sched_cpus) {
SCLogError(“Interfaces requested more cores than configured in the threading section: %d > %d”, total_cpus, sched_cpus);
SCReturnInt(-ERANGE);
}
else
{
SCLogError(“Interfaces requested less cores than configured in the threading section: %d <= %d”, total_cpus, sched_cpus);
}
//ADymov

SCReturnInt(0);

}

Builded the suricata and repeated test:

sudo LD_LIBRARY_PATH=/usr/local/lib/x86_64-linux-gnu/ /usr/local/bin/suricata --dpdk
…

Error: dpdk: ConfigSetThreads for interface “0000:01:00.2” [ConfigSetThreads:runmode-dpdk.c:358]
Warning: dpdk: “all” specified in worker CPU cores affinity, excluding management threads [ConfigSetThreads:runmode-dpdk.c:380]
Error: dpdk: Interfaces requested less cores than configured in the threading section: 2 <= 7 [ConfigSetThreads:runmode-dpdk.c:444]
Error: dpdk: ConfigSetThreads for interface “0000:01:00.3” [ConfigSetThreads:runmode-dpdk.c:358]
Error: dpdk: Interfaces requested less cores than configured in the threading section: 4 <= 7 [ConfigSetThreads:runmode-dpdk.c:444]
Notice: dpdk: 0000:01:00.2: unable to determine NIC’s NUMA node, degraded performance can be expected [ReceiveDPDKThreadInit:source-dpdk.c:555]
Error: dpdk: ConfigSetThreads for interface “0000:01:00.3” [ConfigSetThreads:runmode-dpdk.c:358]
Error: dpdk: Interfaces requested less cores than configured in the threading section: 6 <= 7 [ConfigSetThreads:runmode-dpdk.c:444]
Error: dpdk: ConfigSetThreads for interface “0000:01:00.2” [ConfigSetThreads:runmode-dpdk.c:358]
Error: dpdk: Interfaces requested more cores than configured in the threading section: 8 > 7 [ConfigSetThreads:runmode-dpdk.c:439]
Error: dpdk: 0000:01:00.2: fail to load config of interface [DeviceValidateOutIfaceConfig:runmode-dpdk.c:1240]
Error: dpdk: 0000:01:00.3: failed to configure [ParseDpdkConfigAndConfigureDevice:runmode-dpdk.c:1520]

I have 8 cores. 1 core for management. So 7 core i can use. I expected suricata takes 4 core for ips mode, but she requires 8 cores.
The ConfigSetThreads() function is called twice for each interface and requires twice as many cores as needed.
If i disable this check, i can set threads: parameter to 2 or 3 and the suricata works faster and all cores are used.

set threads: parameter to 1 → trex shows speed: Total-Tx :6 Gbps, Total-Rx: 1.35 Gbps
set threads: parameter to 2 → trex shows speed: Total-Tx :6 Gbps, Total-Rx: 2.70 Gbps
set threads: parameter to 3 → trex shows speed: Total-Tx :6 Gbps, Total-Rx: 3.0 Gbps

But suricata prohibits assigning a parameter threads > 1.
Is it mistake?
Is there something that I am doing wrong ?

Thank you for any hint/help

lukashino · July 25, 2023, 8:43am

Hi Alex!

thanks for pointing this out, this is indeed a bug. Until I propose a fix I can suggest you to set the interfaces’ threads number to auto. An example of what I mean is:

- interface: 0000:01:00.3 # PCIe address of the NIC port
  threads: auto
  ...

and also configure threading section like this:

cpu-affinity:
    - management-cpu-set:
        cpu: [ 0 ] 
    - worker-cpu-set:
        cpu: [ 1-4 ]

The threads for individual interfaces will then be equally derived automatically from the worker-cpu-set.

The PR is submitted (along with the socket ID problem you found):

Lukas

Alexander · July 25, 2023, 1:33pm

Hi, Lukas!

Thank you for you answer.

Until I propose a fix I can suggest you to set the interfaces’ threads number to auto.
The problem is that in ips mode you can not specify the value of auto.

…

DPDK capture support

RX queues (and TX queues in IPS mode) are assigned to cores in 1:1 ratio

interfaces:
- interface: 0000:01:00.2 # PCIe address of the NIC port
# Threading: possible values are either “auto” or number of threads
# - auto takes all cores
# in IPS mode it is required to specify the number of cores and the numbers on both interfaces must match
threads: auto
…

If I set threads: auto, I get an error:

…
TELEMETRY: No legacy callbacks, legacy socket not created
Error: dpdk: ConfigSetThreads for interface “0000:01:00.2” [ConfigSetThreads:runmode-dpdk.c:358]
Warning: dpdk: “all” specified in worker CPU cores affinity, excluding management threads [ConfigSetThreads:runmode-dpdk.c:380]
Error: dpdk: ConfigSetThreads for interface “0000:01:00.3” [ConfigSetThreads:runmode-dpdk.c:358]
Error: dpdk: 0000:01:00.2: configured 4 RX queues but copy interface 0000:01:00.3 has 3 TX queues - number of queues must be equal [DeviceValidateOutIfaceConfig:runmode-dpdk.c:1247]
Error: dpdk: 0000:01:00.2: failed to configure [ParseDpdkConfigAndConfigureDevice:runmode-dpdk.c:1520]
test@vpp-desk:~$

The PR is submitted (along with the socket ID problem you found):
Thank you. We will wait until it appears in the master branch.

Lukas, thank you very much!

lukashino · July 25, 2023, 1:58pm

Hi Alex,
thanks for trying it out, in that case, 0000:01:00.2: configured 4 RX queues but copy interface 0000:01:00.3 has 3 TX queues is important as Suricata automatically took 4 threads for the first interface and 3 for the second one.
From the error log, I assume your CPU affinity seems something like this:

cpu-affinity:
    - management-cpu-set:
        cpu: [ 0 ] 
    - worker-cpu-set:
        cpu: [ 1-7 ]

To be inline with your original intention - 2 threads per interface I would suggest to have your worker CPU set as:

cpu-affinity:
    - management-cpu-set:
        cpu: [ 0 ] 
    - worker-cpu-set:
        cpu: [ 1-4 ]

(For the IPS mode, the number of cores set in the CPU affinity must be divisible by the number of interfaces). After the PR is merged you will be able to specify the exact number of threads per interface.

Alexander · July 25, 2023, 4:30pm

Hi, Lukas!

Thank you for you answer.

From the error log, I assume your CPU affinity seems something like this: …

Exactly. I have the following settings:

cpu-affinity:
- management-cpu-set:
cpu: [ 0 ]
- worker-cpu-set:
cpu: [ “all” ]

And when I start the suricata, I get an error.

With settings dpdk/copy-mode: ips and dpdk/threads: auto

1)After changing to:

cpu-affinity:
- management-cpu-set:
cpu: [ 0 ]
- worker-cpu-set:
cpu: [ 1-4 ]

suricata works fine and uses 2 cores per interface (4 cores).

2)After changing to:

cpu-affinity:
- management-cpu-set:
cpu: [ 0 ]
- worker-cpu-set:
cpu: [ 1-6 ]

suricata works fine and uses 3 cores per interface (6 cores).

After the PR is merged you will be able to specify the exact number of threads per interface.
Thank you. We will wait until it appears in the master branch.

Lukas, thank you very much!

ajee · July 27, 2023, 1:53am

Hi @lukashino I met the same issue “Interfaces requested more cores than configured in the threading section”, after git fetch the latest code, it works well.

Thanks

Topic		Replies	Views
Suricata with dpdk can not set up Help suricata	13	844	April 24, 2023
Hello, I'm a novice. May I ask you a question（dpdk,suricata）Thank you Help	15	1090	November 17, 2022
High packet drop rate in dpdk runmode Help suricata	13	1822	October 9, 2022
DPDK Performance decreases with more threads Help	9	537	February 26, 2024
Suricata with DPDK capture.dpdk.imissed drops Help	8	403	August 16, 2023

Incorrect check of the number of required cores for suricata dpdk/ips mode

DPDK capture support

RX queues (and TX queues in IPS mode) are assigned to cores in 1:1 ratio

Related Topics