1. dpdk-testpmd failed 2. configured RX queues 16 is higher than device maximum (8)

Help
1

Hello

I was trying to run Suricata with DPDK enabled. But I observed below issues. I’m not sure if I missed any step.

Issue-1. I ran the command: dpdk-devbind.py -b uio_pci_generic 0000:1b:00.0 and the command executed successfully.
But when I ran the command: /usr/local/bin/dpdk-testpmd -a 0000:13:00.0 -l 1,2 – -i,it throws the bellow error.

testpmd: No probed ethernet devices

but Suricata started successfully with the command: “suricata --dpdk -vvv” and threads set to 8(threads:8) in DPDK configuration in suricata.yaml

Issue:2.With threads set to auto or 16 in DPDK configuration in suricata.yaml, the suricata failed to start with the below message:

[10926] Error: dpdk: 0000:13:00.0: configured RX queues 16 is higher than device maximum (8)
[10926] Error: dpdk: 0000:13:00.0: failed to configure

The VM has been configured with 16 vCPUs. How to configure Suricata to use all available vCPUs.

Can you please help resolve the above mentioned issues?

The following are the steps which I executed.

[root@vIDS-R7 ~]# uname -a
Linux vIDS-R7 5.4.17-2136.320.7.1.el7uek.x86_64 #2 SMP Sat Jun 10 15:43:32 PDT 2023 x86_64 x86_64 x86_64 GNU/Linux
[root@vIDS-R7 ~]#

[root@vIDS-R7 ~]# pkg-config --modversion libdpdk
22.11.2
[root@vIDS-R7 ~]#

[root@vIDS-R7 ~]# lspci -D | grep ‘Network|Ethernet’
0000:0b:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
0000:13:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
0000:1b:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
[root@vIDS-R7 ~]# dpdk-devbind.py --status-dev net

[root@vIDS-R7 ~]# lscpu | grep -i numa
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
[root@vIDS-R7 ~]#

[root@vIDS-R7 ~]# numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
node 0 size: 63784 MB
node 0 free: 48191 MB
node distances:
node 0
0: 10
[root@vIDS-R7 ~]#

[root@vIDS-R7 ~]# cat /proc/meminfo | grep -i hugepage
AnonHugePages: 382976 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 1024
HugePages_Free: 721
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
[root@vIDS-R7 ~]#

Network devices using DPDK-compatible driver

0000:13:00.0 ‘VMXNET3 Ethernet Controller 07b0’ drv=uio_pci_generic unused=vmxnet3,vfio-pci
0000:1b:00.0 ‘VMXNET3 Ethernet Controller 07b0’ drv=uio_pci_generic unused=vmxnet3,vfio-pci

Network devices using kernel driver

0000:0b:00.0 ‘VMXNET3 Ethernet Controller 07b0’ if=ens192 drv=vmxnet3 unused=vfio-pci,uio_pci_generic Active
[root@vIDS-R7 ~]#

[root@vIDS-R7 ~]# /usr/local/bin/dpdk-testpmd -a 0000:13:00.0 -l 1,2 – -i
EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode ‘PA’
EAL: VFIO support initialized
testpmd: No probed ethernet devices
Interactive-mode selected
testpmd: create a new mbuf pool <mb_pool_0>: n=155456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Done
testpmd>

[root@vIDS-R7 ~]# suricata --build-info | grep -i dpdk
DPDK support: yes
DPDK Bond PMD: yes
[root@vIDS-R7 ~]#

[root@vIDS-R7 ~]# suricata --dpdk -vvv
[7398] Notice: suricata: This is Suricata version 7.0.0-rc2 RELEASE running in SYSTEM mode
[7398] Info: cpu: CPUs/cores online: 16
[7398] Config: device: Adding interface 0000:13:00.0 from config file
[7398] Info: suricata: Setting engine mode to IDS mode by default

[7562] Warning: dpdk: 0000:13:00.0: NIC is on NUMA -1, 8 threads on different NUMA node(s)
[7398] Config: flow-manager: using 1 flow manager threads
[7563] Perf: threads: Setting prio 0 for thread “FM#01”, thread id 7563
[7398] Config: flow-manager: using 1 flow recycler threads
[7564] Perf: threads: Setting prio 0 for thread “FR#01”, thread id 7564
[7565] Perf: threads: Setting prio 0 for thread “CW”, thread id 7565
[7566] Perf: threads: Setting prio 0 for thread “CS”, thread id 7566
[7398] Info: unix-manager: unix socket ‘/var/run/suricata/suricata-command.socket’
[7567] Perf: threads: Setting prio 0 for thread “US”, thread id 7567
[7398] Warning: suricata: setrlimit has no effet when running as root.
[7398] Notice: threads: Threads created → W: 8 FM: 1 FR: 1 Engine started.

Thanks
Maloy

2

Hi Maloy, welcome to the Suricata community.

Issue 1
Regarding issue 1 - seems suspicious, the thing that is off for me are different PCIe addresses in between command dpdk-devbind and dpdk-testpmd.
Maybe make sure both devices were bound to uio_pci_generic or (preferably) vfio-pci driver. I’ve seen you tried to probe also vfio-pci driver - in case it didn’t work for you, maybe also try enabling noiommu mode for the test:
echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

Issue 2
WRT to the second issue - each device has a limited number of queues which it can create - it seems that your limit is 8. However, reading through the documentation of vmxnet3 driver it seems that newer version of VMXNET3 driver can support up to 32 queues!

Vmxnet3 version 6, hw ver 17 This version enhanced vmxnet3 to support queues up to 32 and also removed power-of-two limitations on the queues.
http://doc.dpdk.org/guides/nics/vmxnet3.html

I am not that versatile in VMXNET3 setting but please try to verify that you are on the newest driver/VMware version and try again.

Please report back with your findings :slight_smile:

PS: don’t worry about NIC is on NUMA -1, 8 threads on different NUMA node, it will be fixed in the upcoming Suricata version.