With HP OFED that was a good catch, sometimes server vendors seem to redistribute their own OFED. So if you have HP server then it’s fine, if you have Dell they also seem to have their own OFED.
Mellanox cards have treated me well - I only had to install the OFED and it was up and running without having to change drivers as with the Intel. But this is really a peculiar problem.
I have thought of two things - RHEL 8 should be able to install DPDK through the package manager - to make sure DPDK is compiled correctly (yum, dnf).
But prior to that remove/Uninstall every DPDK file you find on the system. Maybe there are some leftover files from the previous DPDK installations that are missing with your system? DPDK files have rte in the name, libs can be librte*, header files as rte_*
only one primary app can run in DPDK within one “context” - so to run dpdk-testpmd in parallel to Suricata assign file-prefix to the EAL parameters of the testpmd app
dpdk-testpmd -l 0,2 -a 0:0f:00.0 -a 0:0f:00.1 --file-prefix test1 -- -i --forward-mode=rxonly
When testing with mellanox, I found that I didn’t need to use devbind at all. The driver in the kernel can run in both kernel and dpdk mode. So at least in Ubuntu, it works out of the box w/o tricks.
Yep good catch regarding left overs, but did remove those yesterday evening while starting fresh with a dpdk 23.07 installation. I’ll have a closer look at this, thanks!
Btw is there a lead for you in this?
dpdk-testpmd -l 0,2 -a 0:0f:00.0 -a 0:0f:00.1 --file-prefix test1 – -i --forward-mode=rxonly
EAL: Detected CPU lcores: 128
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/test1/mp_socket
EAL: Selected IOVA mode ‘VA’
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:0f:00.0 (socket 0)
mlx5_net: No available register for sampler.
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:0f:00.1 (socket 0)
mlx5_net: No available register for sampler.
Interactive-mode selected
Set rxonly packet forwarding mode
testpmd: Flow tunnel offload support might be limited or unavailable on port 0
testpmd: Flow tunnel offload support might be limited or unavailable on port 1
testpmd: create a new mbuf pool <mb_pool_0>: n=155456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 0)
Port 0: B8:83:03:95:CC:24
Configuring Port 1 (socket 0)
Port 1: B8:83:03:95:CC:25
Checking link statuses…
Done
Lukas, RH8.8 has this version in it’s repos:
Name : dpdk
Version : 21.11
Release : 3.el8
Is this worth an attempt, or will it hinder Suricata v7 with performance/options?
Oke this also requires OFED to be removed due to depencies from the RH dpdk package, so back to basics instead of cutting edge.
With RH 8.8 stock rpms and kernel drivers. Note: mlx5_ib seems related to the verbs because when I unload this driver because it is not an Infiniband card, the Verbs. error pops up.
]# dpdk-testpmd -l 0,2 -a 0:0f:00.0 -a 0:0f:00.1 --file-prefix test1 – -i --forward-mode=rxonly
EAL: Detected CPU lcores: 128
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/test1/mp_socket
EAL: Selected IOVA mode ‘VA’
EAL: No available 1048576 kB hugepages reported
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:0f:00.0 (socket 0)
mlx5_net: No available register for sampler.
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:0f:00.1 (socket 0)
mlx5_net: No available register for sampler.
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
Set rxonly packet forwarding mode
testpmd: create a new mbuf pool <mb_pool_0>: n=155456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 0)
Port 0: B8:83:03:95:CC:24
Configuring Port 1 (socket 0)
Port 1: B8:83:03:95:CC:25
Checking link statuses…
Done
]# gdb --args suricata -c /etc/suricata/suricata.yaml --user root --dpdk
run
…
[New Thread 0x7fffa335d700 (LWP 43004)]
mlx5_common: Failed to create TIS using DevX
mlx5_net: Failed to TIS 0 for bonding device mlx5_0.
mlx5_net: TIS allocation failure
mlx5_net: probe of PCI device 0000:0f:00.0 aborted after encountering an error: Cannot allocate memory
mlx5_common: Failed to load driver mlx5_eth
EAL: Requested device 0000:0f:00.0 cannot be used
mlx5_common: Failed to create TIS using DevX
mlx5_net: Failed to TIS 0 for bonding device mlx5_1.
mlx5_net: TIS allocation failure
mlx5_net: probe of PCI device 0000:0f:00.1 aborted after encountering an error: Cannot allocate memory
mlx5_common: Failed to load driver mlx5_eth
EAL: Requested device 0000:0f:00.1 cannot be used
[New Thread 0x7fffa2b5c700 (LWP 43029)]
TELEMETRY: No legacy callbacks, legacy socket not created
[New Thread 0x7fffa235b700 (LWP 43031)]
Notice: log-pcap: Ring buffer initialized with 589 files. [PcapLogInitRingBuffer:log-pcap.c:1044]
[New Thread 0x7fffa1665700 (LWP 43032)]
Error: dpdk: Interface “0000:0f:00.0”: No such device [ConfigSetIface:runmode-dpdk.c:350]
I’ve noticed mlx5_net: probe of PCI device 0000:0f:00.0 aborted after encountering an error: Cannot allocate memory
In one of your previous post I’ve also seen that you use 24 threads for each interface - can you try to lower it down to 1 thread on each interface? If it would start up then I would allocate more hugepages.
Ok, only 1 interface (Mellanox), no other interfaces and no dpdk Intel nics bonded. Suricata still fails but this pops up:
suricata[45866]: EAL: No available 1048576 kB hugepages reported
]# dpdk-hugepages.py -s
Node Pages Size Total
0 16384 2Mb 32Gb
1 16384 2Mb 32Gb
Hugepages mounted on /dev/hugepages
That is only a notice/warning that DPDK could not find 1GB hugepages - as long as you have at least 2MB pages then it is fine.
If you would have 1GB pages allocated then it would “complain” about No available 2048 kB hugepages
With that size of allocated hugepages that is definitely not the issue
Well just to be sure, testing with 1GB hugepages:
grubby --update-kernel=ALL --args=“transparent_hugepage=never hugepagesz=1G hugepages=1 default_hugepagesz=1G”
Haha indeed: EAL: No available 2048 kB hugepages reported
Well thanks for all the effort, much appriciated! We gonna order a supported 10G Intel nic and remove this Mellanox card. Learned a lot, thanks Lukas, I need to read more before I do
Ok.
let me ask/suggest one final thing - I can see your testpmd is probably working so I feel like we are really close. Have you tried to autogen && configure && make Suricata after installing DPDK through yum?
This is to update DPDK compile flags in the Suricata build process - they might be slightly altered in between DPDK installations.
Yep I did indeed, everytime I alter a library(s) such as Hyperscan or DPDK, I clean up suricata (make distclean) and create a new fresh installation of suricata just to be sure everything matches. Even when it
uses a shared library.
Btw: if you like to do some more research on this Mellanox fenomena, I have a second server we need to build up from scratch so we could test on this box?
