Suricata, dpdk and Mellanox: mlx5_common: Verbs device not found: 0000:0f:00.0

Help
1

Hello,

DPDK and Suricata 7 are running fine with 2 Intel 810-C cards (both with 2x100Gb ports). Adding a Mellanox card prevents Suricata from starting up, just stops after some time.

Did some Suricata gdb to get more info.

Cheers,
Andre

Red Hat Enterprise Linux release 8.8 (Ootpa)
Kernel 4.18.0-477.15.1.el8_8.x86_64
Suricata “7.0.1-dev (2786ccb08 2023-08-04)”
DPDK EAL: RTE Version: ‘DPDK 23.11.0-rc0’

]# dpdk-devbind.py -s
Network devices using DPDK-compatible driver

0000:0f:00.0 ‘MT27710 Family [ConnectX-4 Lx] 1015’ drv=vfio-pci unused=mlx5_core
0000:0f:00.1 ‘MT27710 Family [ConnectX-4 Lx] 1015’ drv=vfio-pci unused=mlx5_core
0000:10:00.0 ‘Ethernet Controller E810-C for QSFP 1592’ drv=vfio-pci unused=ice
0000:10:00.1 ‘Ethernet Controller E810-C for QSFP 1592’ drv=vfio-pci unused=ice
0000:84:00.0 ‘Ethernet Controller E810-C for QSFP 1592’ drv=vfio-pci unused=ice
0000:84:00.1 ‘Ethernet Controller E810-C for QSFP 1592’ drv=vfio-pci unused=ice

]# dmesg
[ 64.819193] ice 0000:84:00.0: Removed PTP clock
[ 67.030977] ice 0000:10:00.0: Removed PTP clock
[ 68.551582] mlx5_core 0000:0f:00.0: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), active vports(0)
[ 68.560530] mlx5_core 0000:0f:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), active vports(0)
[ 73.037492] mlx5_core 0000:0f:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), active vports(0)
[ 73.670393] mlx5_core 0000:0f:00.0: E-Switch: cleanup
[ 75.335550] mlx5_core 0000:0f:00.1: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), active vports(0)
[ 75.347470] mlx5_core 0000:0f:00.1: E-Switch: Disable: mode(LEGACY), nvfs(0), active vports(0)
[ 79.791446] mlx5_core 0000:0f:00.1: E-Switch: Disable: mode(LEGACY), nvfs(0), active vports(0)
[ 80.498328] mlx5_core 0000:0f:00.1: E-Switch: cleanup

]# gdb --args suricata -c /etc/suricata/suricata.yaml --user root --dpdk
(gdb) run
…
[New Thread 0x7fff91669700 (LWP 6370)]
mlx5_common: Verbs device not found: 0000:0f:00.0
mlx5_common: Failed to initialize device context.
EAL: Requested device 0000:0f:00.0 cannot be used
mlx5_common: Verbs device not found: 0000:0f:00.1
mlx5_common: Failed to initialize device context.
EAL: Requested device 0000:0f:00.1 cannot be used
[New Thread 0x7fff90e68700 (LWP 6371)]
TELEMETRY: No legacy callbacks, legacy socket not created
Error: dpdk: Interface “0000:0f:00.0”: No such device [ConfigSetIface:runmode-dpdk.c:350]
[Thread 0x7fff91669700 (LWP 6370) exited]

2

Hi André,

my first guess would be that something must be wrong with the NIC driver. Did you install the mlx5 driver as suggested?
https://doc.dpdk.org/guides/platform/mlx5.html

Can you try running dpdk-testpmd application to see if that fails too?

dpdk-testpmd -l 0,2 -a 0000:0f:00.0 -a 0000:0f:00.1 -- -i --forward-mode=rxonly

It will run in an interactive mode so then you just can use commands like:

start
stop
quit

Also for reference - I see that you use older version of Suricata - can you try to use at least 7.0.0 or compile the master branch again?

3

Hi Lukas,
No did not follow a RTFM, was happy dpdk bind worked out of the box and guessed that was enough, sorry! I’ll will follow your checks and TFM :wink: Thanks for the swift reply!
Cheers,
André

4

Using RH8 provided libibverbs, rdma-core and ib_uverbs but still a fail. Digging further.

$ suricata -V
This is Suricata version 7.0.1-dev (2786ccb08 2023-08-04)
Is more recent then 7.0.0?

dpdk-testpmd fails to.

EAL: Detected CPU lcores: 128
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode ‘VA’
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:0f:00.0 (socket 0)
mlx5_common: Verbs device not found: 0000:0f:00.0
mlx5_common: Failed to initialize device context.
EAL: Requested device 0000:0f:00.0 cannot be used
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:0f:00.1 (socket 0)
mlx5_common: Verbs device not found: 0000:0f:00.1
mlx5_common: Failed to initialize device context.
EAL: Requested device 0000:0f:00.1 cannot be used
EAL: Bus (pci) probe failed.
testpmd: No probed ethernet devices

5

Considering I see:

you’ve probably built DPDK by yourself right?

In that case try rebuilding DPDK.

When you configure the DPDK folder you should see something like below. When you first configured the DPDK sources with meson, the mlx5 driver (or some part of it) was likely missing and that’s why it now crashes. So you need to reconfigure the DPDK sources with meson again (i.e. uninstall DPDK and remove the build folder and configure DPDK sources again with meson). To check the previous meson log - you can find it in the DPDK folder in the build/meson-logs/meson-log.txt file. You should see mlx5 in the net: category (and definitely not see anything mlx5 in the skipped part.

:

Drivers Enabled
===============

common:
        cpt, dpaax, iavf, octeontx, octeontx2, cnxk, mlx5, qat, 
        sfc_efx, 
bus:
        auxiliary, dpaa, fslmc, ifpga, pci, vdev, vmbus, 
mempool:
        bucket, cnxk, dpaa, dpaa2, octeontx, octeontx2, ring, stack, 
        
dma:
        cnxk, dpaa, hisilicon, idxd, ioat, skeleton, 
net:
        af_packet, ark, atlantic, avp, axgbe, bnx2x, bnxt, bond, 
        cnxk, cxgbe, dpaa, dpaa2, e1000, ena, enetc, enetfec, 
        enic, failsafe, fm10k, hinic, hns3, i40e, iavf, ice, 
        igc, ionic, ipn3ke, ixgbe, kni, liquidio, memif, mlx5, 
        netvsc, nfb, nfp, ngbe, null, octeontx, octeontx2, octeontx_ep, 
        pfe, qede, ring, sfc, softnic, tap, thunderx, txgbe, 
        vdev_netvsc, vhost, virtio, vmxnet3, 
raw:
        cnxk_bphy, dpaa2_cmdif, dpaa2_qdma, ifpga, ntb, skeleton, 
crypto:
        bcmfs, caam_jr, ccp, cnxk, dpaa_sec, dpaa2_sec, mlx5, nitrox, 
        null, octeontx, octeontx2, openssl, scheduler, virtio, 
compress:
        mlx5, octeontx, zlib, 
regex:
        mlx5, octeontx2, 
vdpa:
        ifc, mlx5, sfc, 
event:
        cnxk, dlb2, dpaa, dpaa2, dsw, octeontx2, opdl, skeleton, 
        sw, octeontx, 
baseband:
        acc100, fpga_5gnr_fec, fpga_lte_fec, la12xx, null, turbo_sw, 
gpu:
        

Message: 
=================
Content Skipped
=================

libs:
        
drivers:
        common/mvep:    missing dependency, "libmusdk"
        net/af_xdp:     missing dependency, "libxdp" and "libbpf"
        net/mlx4:       missing dependency, "mlx4"
        net/mvneta:     missing dependency, "libmusdk"
        net/mvpp2:      missing dependency, "libmusdk"
        net/pcap:       missing dependency, "libpcap"
        raw/ioat:       replaced by dmadev drivers
        crypto/armv8:   missing dependency, "libAArch64crypto"
        crypto/ipsec_mb:        missing dependency, "libIPSec_MB"
        crypto/mvsam:   missing dependency, "libmusdk"
        compress/isal:  missing dependency, "libisal"
        gpu/cuda:       missing dependency, "cuda.h"

Uninstall commands (from within the DPDK source folder):

sudo ninja -C build uninstall
rm -rf build
sudo meson ....
ninja..
ninja install
6

Indeed build it from the git source and it already has mlx4 and 5 support build with it, checked the meson log file but also a dkdp-devbind should not work if it was missing I guess?

just performed a
$ sudo mstconfig -d 0000:0f:00.0 reset
and rebooting, found this via almighty Google regarding the verbs error : Re: mlx5_common: No Verbs device matches PCI device 0000:01:00.1
Cannot find a hint I’m having a Mellanox LAG/bond but who know what NetworkManager is f… up in the background (blacklisted the mac addresses from NetwerkManager so it should keep it’s hands of the card)

7

Ah heh. Indeed, if the whole mlx5 would be missing from the DPDK installation then dpdk-devbind should not show it but tbh I am not certain about this. I was thinking maybe some part of the driver was only missing.

Indeed, I also found that mailing list thread and that’s when I thought something is off with the installation process.

Did the reset help or did you mention it as your current state? Also, is only bond not working but individual interfaces work?
Maybe you could substitute that with the DPDK bond?
https://doc.dpdk.org/guides/prog_guide/link_bonding_poll_mode_drv_lib.html

Pass
--vdev 'net_bonding0,bond_opt0=..,bond opt1=..' as an EAL parameter

Your suricata.yaml interface will then work with the net_bonding0 as the name of the interface.

https://docs.suricata.io/en/latest/capture-hardware/dpdk.html

8

I’m not using a bond at all, just a hit regaring my verbs search did me have a look at that url :wink:
How essential is 5.5.1.4. Enable Switchdev Mode to dpdk to function proper with Mellanox?

“Then switchdev mode is enabled:
echo switchdev > /sys/class/net//compat/devlink/mode”

But this Mellanox card does not show this option:
ls -al /sys/class/net/ens3f0/ → no compat directory here

9

Btw, this is a result of not using dpdk bind but just start suricata with only the kernel driver for Mellanox:
mlx5_common: DevX create TIS failed errno=121 status=0x3 syndrome=0x6a6678
mlx5_net: Failed to create TIS 0/0 for [bonding] device mlx5_0.
mlx5_net: TIS allocation failure
mlx5_net: probe of PCI device 0000:0f:00.0 aborted after encountering an error: Cannot allocate memory
mlx5_common: Failed to load driver mlx5_eth
EAL: Requested device 0000:0f:00.0 cannot be used
mlx5_common: DevX create TIS failed errno=121 status=0x3 syndrome=0x6a6678
mlx5_net: Failed to create TIS 0/0 for [bonding] device mlx5_1.
mlx5_net: TIS allocation failure
mlx5_net: probe of PCI device 0000:0f:00.1 aborted after encountering an error: Cannot allocate memory
mlx5_common: Failed to load driver mlx5_eth
EAL: Requested device 0000:0f:00.1 cannot be used
[New Thread 0x7fff8fa10700 (LWP 71208)]
TELEMETRY: No legacy callbacks, legacy socket not created

11

Thanks, regarding the new dpdk bond feature:

EAL: failed to parse device “ids_bond0”
EAL: Unable to parse device ‘ids_bond0,mode=0,slave=0000:0f:00.0,slave=0000:0f:00.1’
Error: dpdk: DPDK EAL initialization error: Operation not permitted [InitEal:runmode-dpdk.c:301]
[Inferior 1 (process 106888) exited with code 01]

]# dpdk-devbind.py -s

  • Network devices using DPDK-compatible driver
    ============================================
    0000:10:00.0 ‘Ethernet Controller E810-C for QSFP 1592’ drv=vfio-pci unused=ice
    0000:10:00.1 ‘Ethernet Controller E810-C for QSFP 1592’ drv=vfio-pci unused=ice
    0000:84:00.0 ‘Ethernet Controller E810-C for QSFP 1592’ drv=vfio-pci unused=ice
    0000:84:00.1 ‘Ethernet Controller E810-C for QSFP 1592’ drv=vfio-pci unused=ice

  • Network devices using kernel driver
    ===================================
    0000:0f:00.0 ‘MT27710 Family [ConnectX-4 Lx] 1015’ if=ens3f0 drv=mlx5_core unused=vfio-pci
    0000:0f:00.1 ‘MT27710 Family [ConnectX-4 Lx] 1015’ if=ens3f1 drv=mlx5_core unused=vfio-pci

  • suricata
    dpdk:
    eal-params:
    proc-type: primary
    vdev: ‘ids_bond0,mode=0,slave=0000:0f:00.0,slave=0000:0f:00.1’
    interfaces:

    • interface: 0000:10:00.0
      threads: 24 # auto
      promisc: true # promiscuous mode - capture all packets
      multicast: true # enables also detection on multicast packets
      checksum-checks: true # if Suricata should validate checksums
      checksum-checks-offload: true # if possible offload checksum validation to the NIC (saves Suricata resources)
      mtu: 1518 # Set MTU of the device in bytes
      mempool-cache-size: 511
      rx-descriptors: 1024 # 1024
      tx-descriptors: 1024 # 1024
      copy-mode: none
      copy-iface: none

    • interface: default
      threads: 24 # auto
      promisc: true
      multicast: true
      checksum-checks: true
      checksum-checks-offload: true
      mtu: 1518
      rss-hash-functions: auto
      mempool-size: 65535
      mempool-cache-size: 511
      rx-descriptors: 1024 # 1024
      tx-descriptors: 1024 # 1024
      copy-mode: none
      copy-iface: none

    • interface: 0000:84:00.0

    • interface: 0000:10:00.1

    • interface: ids_bond0
      threads: 6

related dmesg:
kernel: NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #280!!!

12

Hey,
the virtual device name should be called net_bonding followed by something (as e.g. net_bonding0)

This is so that DPDK knows what virtual device to initialize. (and the DPDK driver for the bond is called net_bonding)

But I suggest only trying this feature when your MLX5 cards are working separately.

if I recall correctly I used this tutorial to set them up once…
https://docs.nvidia.com/networking/display/MLNXOFEDv531050/Installing+Mellanox+OFED#:~:text=Extract%20the%20MLNX_OFED%20package%20on,Products%20>%20Software>%20InfiniBand%20Drivers.&text=Download%20and%20install%20Mellanox%20Technologies%20GPG-KEY.&text=Verify%20that%20the%20key%20was%20successfully%20imported.

13

I’m doing something major wrong I guess:

mlx5_common: DevX create TIS failed errno=121 status=0x3 syndrome=0x6a6678
mlx5_net: Failed to create TIS 0/0 for [bonding] device mlx5_0.
mlx5_net: TIS allocation failure
mlx5_net: probe of PCI device 0000:0f:00.0 aborted after encountering an error: Cannot allocate memory
mlx5_common: Failed to load driver mlx5_eth
EAL: Requested device 0000:0f:00.0 cannot be used
mlx5_common: DevX create TIS failed errno=121 status=0x3 syndrome=0x6a6678
mlx5_net: Failed to create TIS 0/0 for [bonding] device mlx5_1.
mlx5_net: TIS allocation failure
mlx5_net: probe of PCI device 0000:0f:00.1 aborted after encountering an error: Cannot allocate memory
mlx5_common: Failed to load driver mlx5_eth
EAL: Requested device 0000:0f:00.1 cannot be used
[New Thread 0x7fff90e68700 (LWP 110141)]
TELEMETRY: No legacy callbacks, legacy socket not created
[New Thread 0x7fff90667700 (LWP 110142)]
…
[New Thread 0x7ffe37b0a700 (LWP 110216)]
ice_vsi_config_outer_vlan_stripping(): Single VLAN mode (SVM) does not support qinq
bond_ethdev_parse_slave_port_kvarg(126) - Invalid slave port value (0000:0f:00.0) specified
bond_ethdev_configure(4073) - Failed to parse slave ports for bonded device net_bonding0
Port4 dev_configure = -1
Error: dpdk: net_bonding0: failed to configure the device (port 4, err Operation not permitted) [DeviceConfigure:runmode-dpdk.c:1419]

Thread 3 “rte_mp_handle” received signal SIG32, Real-time event 32.
[Switching to Thread 0x7fffd0ee8700 (LWP 110010)]
0x00007ffff679c987 in recvmsg () from /lib64/libpthread.so.0

14

Not sure what you tried/didn’t try and how you set up the NIC. But if you in any way configured the NIC to be in the bond mode (outside of standard Linux configuration) I would revert it to work as independent Ethernet ports only. It also seems you can’t start DPDK applications with individual ports so before going for the bond, please try to run it with individual ports first.

Have you found this?

15

Haha no nothing with a bond, this is just were we got each other wrong because of the Google dorking we do :wink:

I first started with checking if the Mellanox card was supported in DPDK, it was but 23.11 release notes say no:
grep -i MCX4121A doc/guides/rel_notes/release_23_11.rst
but : doc/guides/rel_notes/release_23_03.rst: * NVIDIA\ |reg| ConnectX\ |reg|-4 Lx 25G MCX4121A-ACAT (2x25G)

So maybe that’s the problem?
Otherwise I continued and used dpdk-devbind.py to bind the Mellanox devides, but it seems that is not the way, it used some parts of the kernel driver dpdk when using Mellanox cards?

So there was never a bond config or such in place regarding this Mellanox card and it is a Ethernet only.

16

Ah okay.

Well I think it doesn’t say it’s not supported on 23.11, it only says it is not tested.
You can of course try previous DPDK versions to see if it wouldn’t help but I wouldn’t expect that to be the issue.

You say that you bind the device through dpdk-devbind. But MLX5 driver should be bifurcated so that you don’t need to bind anything to the NIC. You should not bind it vfio-pci or something like.

I see it in the first post, that you bonded it vfio-pci - oh well try to bind it back to mlx5_core

1 Like
17

Yep I’ve correct that already to use mlx5_core indeed. Now have some food and see later on if I have magic moment. And thanks for your precious time and great support, much appreciated!

Regarding dpdk 23.07:
EAL: libmlx4.so.1: cannot open shared object file: No such file or directory
EAL: FATAL: Cannot init plugins
EAL: Cannot init plugins
Error: dpdk: DPDK EAL initialization error: Operation not permitted [InitEal:runmode-dpdk.c:301]
[Inferior 1 (process 154999) exited with code 01]

Seems like it wants to intilialize the mlx_4 driver, let’s see if we can skip that.

18

Ok update for this moment:

dpdk 23.07: meson build -Ddisable_drivers=mlx4,mana

]# dpdk-proc-info -v
EAL: RTE Version: ‘DPDK 23.07.0’

]# suricata -V
This is Suricata version 7.0.1-dev (2786ccb08 2023-08-04)

]# gdb --args suricata -c /etc/suricata/suricata.yaml --user root --dpdk
[New Thread 0x7fff91a84700 (LWP 333121)]
mlx5_common: DevX create TIS failed errno=121 status=0x3 syndrome=0x6a6678
mlx5_net: Failed to create TIS 0/0 for [bonding] device mlx5_0.
mlx5_net: TIS allocation failure
mlx5_net: probe of PCI device 0000:0f:00.0 aborted after encountering an error: Cannot allocate memory
mlx5_common: Failed to load driver mlx5_eth
EAL: Requested device 0000:0f:00.0 cannot be used
mlx5_common: DevX create TIS failed errno=121 status=0x3 syndrome=0x6a6678
mlx5_net: Failed to create TIS 0/0 for [bonding] device mlx5_1.
mlx5_net: TIS allocation failure
mlx5_net: probe of PCI device 0000:0f:00.1 aborted after encountering an error: Cannot allocate memory
mlx5_common: Failed to load driver mlx5_eth
EAL: Requested device 0000:0f:00.1 cannot be used
[New Thread 0x7fff91283700 (LWP 333126)]
TELEMETRY: No legacy callbacks, legacy socket not created
[New Thread 0x7fff90a82700 (LWP 333127)]
Notice: log-pcap: Ring buffer initialized with 586 files. [PcapLogInitRingBuffer:log-pcap.c:1044]
[New Thread 0x7fff8bb0a700 (LWP 333128)]

]# dpdk-testpmd -l 0,2 -a 0000:0f:00.0 -a 0000:0f:00.1 – -i --forward-mode=rxonly
EAL: Detected CPU lcores: 128
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode ‘VA’
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:0f:00.0 (socket 0)
mlx5_net: No available register for sampler.
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:0f:00.1 (socket 0)
mlx5_net: No available register for sampler.
Interactive-mode selected
Set rxonly packet forwarding mode
testpmd: Flow tunnel offload support might be limited or unavailable on port 0
testpmd: Flow tunnel offload support might be limited or unavailable on port 1
testpmd: create a new mbuf pool <mb_pool_0>: n=155456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 0)
Port 0: B8:83:03:95:CC:24
Configuring Port 1 (socket 0)
Port 1: B8:83:03:95:CC:25
Checking link statuses…
Done

19

Interesting, after installing OFED HP Software Delivery Repository for Redhat 8.8 and reboot:

…[New Thread 0x7fff91a84700 (LWP 6745)]
mlx5_common: Verbs device not found: 0000:0f:00.0
mlx5_common: Failed to initialize device context.
EAL: Requested device 0000:0f:00.0 cannot be used
mlx5_common: Verbs device not found: 0000:0f:00.1
mlx5_common: Failed to initialize device context.
EAL: Requested device 0000:0f:00.1 cannot be used
[New Thread 0x7fff91283700 (LWP 6747)]
TELEMETRY: No legacy callbacks, legacy socket not created
Error: dpdk: Interface “0000:0f:00.0”: No such device [ConfigSetIface:runmode-dpdk.c:350]

]# dpdk-testpmd -l 0,2 -a 0000:0f:00.0 -a 0000:0f:00.1 – -i --forward-mode=rxonly
EAL: Detected CPU lcores: 128
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode ‘VA’
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:0f:00.0 (socket 0)
mlx5_common: Verbs device not found: 0000:0f:00.0
mlx5_common: Failed to initialize device context.
EAL: Requested device 0000:0f:00.0 cannot be used
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:0f:00.1 (socket 0)
mlx5_common: Verbs device not found: 0000:0f:00.1
mlx5_common: Failed to initialize device context.
EAL: Requested device 0000:0f:00.1 cannot be used
EAL: Bus (pci) probe failed.
testpmd: No probed ethernet devices
Interactive-mode selected
Set rxonly packet forwarding mode
testpmd: create a new mbuf pool <mb_pool_0>: n=155456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Done

20

Gues I also need to do much more reading. I had the idea DPDK was a replacement for kernel drivers, but seen the effort to get a Mellanox card DPDK ready with OFED got me puzzled.

21

Also interesting: running Suricata with de Intel nics via DPDK bind, going well. So now I want to do a dpdk test with de Mellanox card (no dpdk bind, not configured with Suricata) and that is a no go:

]# dpdk-testpmd -l 0,2 -a 0:0f:00.0 -a 0:0f:00.1 – -i --forward-mode=rxonly
EAL: Detected CPU lcores: 128
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Cannot create lock on ‘/var/run/dpdk/rte/config’. Is another primary process running?
EAL: FATAL: Cannot init config
EAL: Cannot init config
EAL: Error - exiting with code: 1
Cause: Cannot init EAL: Success