High Packet Drop Rate with DPDK in Suricata 7.0.2

Help
,
1

result：

[root@localhost ~]# dpdk-telemetry.py
--> /ethdev/xstats,0
{
  "/ethdev/xstats": {
    "rx_good_packets": 185439852,
    "tx_good_packets": 0,
    "rx_good_bytes": 135409616741,
    "tx_good_bytes": 0,
    "rx_missed_errors": 504199616,
    "rx_errors": 0,
    "tx_errors": 0,
    "rx_mbuf_allocation_errors": 0
  }
}
  • Suricata Version: 7.0.2,installed from source
  • Operating Mode: IDS
  • DPDK Version: 22.11,installed from source
  • CPU: 32 cores
  • Network Interface: Intel X710 with i40e dirver
  • OS: Kylin Linux Advanced Server V10 (SP4) x86_64
  • HugePages
[root@localhost ~]# grep Huge /proc/meminfo
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
HugePages_Total:    2048
HugePages_Free:     1743
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         4194304 kB

dpdk in suricata.yaml

dpdk:
  eal-params:
    proc-type: primary

  interfaces:
    - interface: 0000:41:00.0 # PCIe address of the NIC port
      threads: 8
      promisc: true # promiscuous mode - capture all packets
      multicast: false # enables also detection on multicast packets
      checksum-checks: false # if Suricata should validate checksums
      checksum-checks-offload: false # if possible offload checksum validation to the NIC (saves Suricata resources)
      mtu: 1500 # Set MTU of the device in bytes
      mempool-size: 262144 # The number of elements in the mbuf pool
      mempool-cache-size: 512
      rx-descriptors: 4096
      tx-descriptors: 4096
      copy-mode: none
      copy-iface: none # or PCIe address of the second interface
      rss-hash-functions: auto

threading:
  set-cpu-affinity: yes
  cpu-affinity:
    - management-cpu-set:
        cpu: [ 8,9 ]  # include only these CPUs in affinity settings
    - receive-cpu-set:
        cpu: [ 10 ]  # include only these CPUs in affinity settings
    - worker-cpu-set:
        cpu: [ 24-31 ]
        mode: "exclusive"
        # Use explicitly 3 threads and don't compute number by using
        # detect-thread-ratio variable:
        # threads: 3
        prio:
          low: [ 0 ]
          medium: [ "1-2" ]
          high: [ 3 ]
          default: "medium

[root@localhost ~]# cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
1024
[root@localhost ~]# cat /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
1024
[root@localhost ~]# dpdk-hugepages.py --show
Node Pages Size Total
0    1024  2Mb    2Gb
1    1024  2Mb    2Gb

Hugepages mounted on /dev/hugepages

[root@localhost ~]# lscpu
架构：                              x86_64
CPU 运行模式：                      32-bit, 64-bit
字节序：                            Little Endian
Address sizes:                      48 bits physical, 48 bits virtual
CPU:                                32
在线 CPU 列表：                     0-31
每个核的线程数：                    2
每个座的核数：                      8
座：                                2
NUMA 节点：                         2
厂商 ID：                           HygonGenuine
BIOS Vendor ID:                     Chengdu Hygon
CPU 系列：                          24
型号：                              2
型号名称：                          Hygon C86 3350  8-core Processor
BIOS Model name:                    Hygon C86 3350  8-core Processor
步进：                              2
CPU MHz：                           3000.127
BogoMIPS：                          6000.25
虚拟化：                            AMD-V
L1d 缓存：                          512 KiB
L1i 缓存：                          1 MiB
L2 缓存：                           8 MiB
L3 缓存：                           32 MiB
NUMA 节点0 CPU：                    0-7,16-23
NUMA 节点1 CPU：                    8-15,24-31
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Mitigation; untrained return thunk; SMT vulnerable
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, STIBP disabled, RSB filling, PBRSB-eIBRS Not affec
                                    ted
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected
标记：                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse
                                     sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc
                                     cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcn
                                    t aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse
                                    3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx c
                                    pb hw_pstate sme ssbd csv ibpb vmmcall csv2 fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clf
                                    lushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_loc
                                    k nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vm
                                    save_vmload vgif overflow_recov succor smca

suricata.yaml (83.4 KB)

Does anyone know the reason for the high packet loss rate? I modified rx-descriptors to 32768, but it didn’t work.

2

I changed the NIC driver from igb_uio to vifo-pci, and the packet loss rate was still high. :sob:

3

Try perf to check whether the workload is balanced distributed among your worker cpus. You are using the auto rss function, so I suspect that maybe all packets are being processed by one core.

4

Generally, I didn’t have much luck with x710. Can you try other NIC - mlx5 (Nvidia) NIC? Intel NICs seem to only support low number of descriptors so even if you set 32k descriptors, it is auto-adjusted according to NIC’s capabilities, in case of x710 it is only 4096, if I recall correctly.

You can always try to increase number of cores for the traffic to be more distributed.

5

1.Is it possible to modify the following code and then reinstall dpdk?
dpdk/drivers/net/i40e/i40e_rxtx.h → #define I40E_MAX_RING_DESC 32768
2.management-cpu-set、receive-cpu-set、worker-cpu-set Must these CPUs be within the same NUMA node?

6

Multiple cores are working.

7

I’ve pushed a patch set recently increasing the maximums where (according to the NICs datasheets it was possible). You may on the DPDK PMD level increase it but the actual driver/hw architecture will still limit you.

Receive-cpu-set is not used at all in workers runmode (only in autofp). It’s better to have them all in one NUMA because the one core from other NUMA node can be kicking out data from the cache where most of the cores reside. But having more cores is better in general - so 2 fully used NUMA nodes is better than sticking to just 1 NUMA.