# High Packet Drop Rate with DPDK compared to AF\_PACKET in Suricata 7.0.7 **URL:** \ **Category:** Help\ **Tags:** suricata, dpdk\ **Created:** [October 14, 2024, 11:46am UTC](https://forum.suricata.io/t/high-packet-drop-rate-with-dpdk-compared-to-af-packet-in-suricata-7-0-7/4896 "2024-10-14T11:46:33Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![vendul0g](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.suricata.io/vendul0g/32/3808_2.png) [@vendul0g](https://forum.suricata.io/u/vendul0g)\ **Post date:** [October 14, 2024, 11:46am UTC](https://forum.suricata.io/t/high-packet-drop-rate-with-dpdk-compared-to-af-packet-in-suricata-7-0-7/4896/1 "2024-10-14T11:46:33Z")
I hope this message finds you well. I’m reaching out to seek assistance regarding a significant performance issue I’ve been experiencing with Suricata version **7.0.7 RELEASE** running in **IDS mode** on my system. Specifically, I’m encountering a **high packet drop rate (~45%)** when operating in **DPDK run mode** , whereas the performance with **AF\_PACKET** is notably better. * * * ### **System Overview** - **Suricata Version:** 7.0.7 installed from source - **Operating Mode:** IDS - **Hardware Specifications:** - **CPU:** 20 cores (cores 0-19) ```auto $ lscpu -e CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ MINMHZ MHZ 0 0 0 0 0:0:0:0 si 4900,0000 800,0000 800.000 1 0 0 0 0:0:0:0 si 4900,0000 800,0000 800.000 2 0 0 1 4:4:1:0 si 4900,0000 800,0000 800.000 3 0 0 1 4:4:1:0 si 4900,0000 800,0000 800.000 4 0 0 2 8:8:2:0 si 4900,0000 800,0000 800.000 5 0 0 2 8:8:2:0 si 4900,0000 800,0000 800.000 6 0 0 3 12:12:3:0 si 4900,0000 800,0000 800.000 7 0 0 3 12:12:3:0 si 4900,0000 800,0000 800.000 8 0 0 4 16:16:4:0 si 5000,0000 800,0000 800.000 9 0 0 4 16:16:4:0 si 5000,0000 800,0000 800.574 10 0 0 5 20:20:5:0 si 5000,0000 800,0000 800.000 11 0 0 5 20:20:5:0 si 5000,0000 800,0000 800.000 12 0 0 6 24:24:6:0 si 4900,0000 800,0000 848.286 13 0 0 6 24:24:6:0 si 4900,0000 800,0000 800.000 14 0 0 7 28:28:7:0 si 4900,0000 800,0000 800.000 15 0 0 7 28:28:7:0 si 4900,0000 800,0000 800.000 16 0 0 8 36:36:9:0 si 3800,0000 800,0000 800.000 17 0 0 9 37:37:9:0 si 3800,0000 800,0000 800.000 18 0 0 10 38:38:9:0 si 3800,0000 800,0000 800.000 19 0 0 11 39:39:9:0 si 3800,0000 800,0000 800.001 ``` - **Network Interface:** **Intel X540-T2** with `vfio-pci` driver. - **PCI Address:** 0000:05:00.0 (Cause of the `vfio-pci`, it has no IP address) - **OS** : ubuntu 22.04 - **Memory:** 64 GB - HugePages: 4096 GB ```auto $ grep Huge /proc/meminfo AnonHugePages: 0 kB ShmemHugePages: 8192 kB FileHugePages: 0 kB HugePages_Total: 4096 HugePages_Free: 4095 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 8388608 kB ``` - **Files:** - **suricata.yaml:** Attached - **suricata.log:** Attached * * * ### **Suricata Configuration Highlights** Below are the key configurations from my `suricata.yaml` that pertain to this issue: ```yaml dpdk: eal-params: proc-type: primary allow: ["0000:05:00.0"] interfaces: - interface: 0000:05:00.0 threads: 8 promisc: true multicast: false checksum-checks: false checksum-checks-offload: false mtu: 1500 mempool-size: 262144 mempool-cache-size: 512 rx-descriptors: 4096 tx-descriptors: 4096 copy-mode: none copy-iface: none rss-hash-functions: auto threading: set-cpu-affinity: yes cpu-affinity: - management-cpu-set: cpu: [16,17] - receive-cpu-set: cpu: [18] - verdict-cpu-set: cpu: [19] - worker-cpu-set: cpu: [0,2,4,6,8,10,12,14] mode: exclusive prio: default: high detect-thread-ratio: 1.0 stack-size: 8mb ``` **Additional Notable Configurations:** - **Mempool Configuration:** - `mempool-size`: 262144 - `mempool-cache-size`: 512 - **RX and TX Descriptors:** - Both set to 4,096 per queue - **Threading:** - 8 worker threads assigned to cores `0,2,4,6,8,10,12,14` - Management, receive, and verdict threads assigned to cores 16, 17, 18, and 19 respectively - **App-Layer Protocols:** - Multiple protocols enabled (HTTP, TLS, SSH, etc.) with specific detection ports - **Runmode:** Workers * * * ### **Observed Performance Issues** When running Suricata in **DPDK mode** , the following metrics were observed from the logs: - **Total Packets Received:** ~17,559,078 - **Packets Dropped (`rx_missed_errors`):** ~7,943,058 - **Packet Drop Percentage:** **Approximately 45.24%** **Comparison with AF\_PACKET Mode:** - In **AF\_PACKET mode** , the packet drop rate is significantly lower, and overall performance is more stable and efficient. - **Total packets:** 17,796,504 - **Drops:** 3,787,138 - **Percentage:** 21.28% * * * ### **Requests for Assistance** Given the complexity of the issue and the critical nature of maintaining low packet drop rates for effective intrusion detection, I kindly request the community’s assistance with the following: 1. **Configuration Review:** 2. **Additional Optimization Tips:** * * * ### **Attached Files** For your reference and detailed analysis, I have attached the following files: 1. **suricata.yaml:** Comprehensive configuration file outlining all current settings. 2. **suricata.log:** Log output capturing the initialization, configuration, and performance metrics during a run in DPDK mode. * * * Best regards, Álvaro ---
**Author:** ![vendul0g](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.suricata.io/vendul0g/32/3808_2.png) [@vendul0g](https://forum.suricata.io/u/vendul0g)\ **Post date:** [October 14, 2024, 11:48am UTC](https://forum.suricata.io/t/high-packet-drop-rate-with-dpdk-compared-to-af-packet-in-suricata-7-0-7/4896/2 "2024-10-14T11:48:36Z")
[suricata.log](https://forum.suricata.io/uploads/short-url/kvTG2kQnWiACw4p60n5cpekcAb9.log) (76.7 KB) [suricata.yaml](https://forum.suricata.io/uploads/short-url/5yKwVwxYEwF545cAMImxFXJlYAL.yaml) (88.2 KB) ---
**Author:** ![lukashino](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.suricata.io/lukashino/32/1150_2.png) [@lukashino](https://forum.suricata.io/u/lukashino)\ **Post date:** [October 15, 2024, 10:14am UTC](https://forum.suricata.io/t/high-packet-drop-rate-with-dpdk-compared-to-af-packet-in-suricata-7-0-7/4896/3 "2024-10-15T10:14:22Z")
Some comments: 1. You actually have 8 GB of hugepage memory allocated - you have allocated 4096 hugepages of size 2048 kB == 8 GB 2. In your CPU setting it would be good to know what is your CPU and if you have Hyperthreading enabled. If you have HT enabled then it would be good to determine what are the Hyperthreaded cores. You can determine the CPU pairs by looking into /proc/cpuinfo and by pairing `core id`. So if you only use 8 cores use the independent cores only. Use of Hyperthreaded cores may boost the performance a little but 2 individual cores will always be better than 2 hyperthreaded ones. 3. This CPU alignment applies to management CPUs as well (although to a lesser degree since the operation there is not so demanding) 4. Receive/verdict CPU sets are not used in workers runmode 5. I would set mempool size to 262143 and mp cache to 511 (there is some DPDK internal math that suggest these numbers but this very likely won’t cause the issue) 6. With my experience with DPDK, it is better to set RX/TX descriptors to 32768. However, Intel cards (at least X710) support setting only 4096 descriptors. It seems like they are not a good fit for Suricata. I am not sure how it is with X540 but I assume it will be the same - try setting 32768 RX/TX descriptors, run Suricata in a very verbose mode (`-vvvv`) and observe if it says something about lowering down the descriptors. ---
**Author:** ![vendul0g](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.suricata.io/vendul0g/32/3808_2.png) [@vendul0g](https://forum.suricata.io/u/vendul0g)\ **Post date:** [October 16, 2024, 9:59am UTC](https://forum.suricata.io/t/high-packet-drop-rate-with-dpdk-compared-to-af-packet-in-suricata-7-0-7/4896/4 "2024-10-16T09:59:02Z")
Hi Lukas. Thanks for answering 1. I set 8 GB of HugePages due to i saw in the `suricata.log` this sentence ```auto [181614 - Suricata-Main] 2024-10-14 13:40:37 Perf: hugepages: Hugepages on NUMA node 0 can be set to 348 (only using 302/4095 2048kB hugepages) ``` So i reduce the size. However, i setup the Hugepages size to 20 GB, but the performance of the executions does not improve. Due to your answer, i have set the Hugepages size to 20 GB again, and the log got in the `suricata.log` is the next: ```auto Perf: hugepages: Hugepages on NUMA node 0 can be set to 347 (only using 301/20479 2048kB hugepages) [SystemHugepageEvaluateHugepages:util-hugepages.c:406] ``` 2, 3. Yes. I have Hyperthreading enabled in the CPU. The caracteristics are: - Model: `12th Gen Intel(R) Core(TM) i7-12700K` - Phyical Cores: 12 - Total threads: 20 - Hyperthreading: enabled - Hyperthreaded cores: (`Core ID: Processor`) - **0:** Processors 0 & 1 - **4:** Processors 2 & 3 - **8:** Processors 4 & 5 - **12:** Processors 6 & 7 - **16:** Processors 8 & 9 - **20:** Processors 10 & 11 - **24:** Processors 12 & 13 - **28:** Processors 14 & 15 - Independent cores (Non-Hyperthreaded): (`Core ID: Processor`) - **36:** Processor 16 - **37:** Processor 17 - **38:** Processor 18 - **39:** Processor 19 **Summary:** - **Total Physical Cores:** 12 - **8 Performance (P) Cores with Hyperthreading:** Core IDs 0, 4, 8, 12, 16, 20, 24, 28 - **4 Efficient (E) Cores without Hyperthreading:** Core IDs 36, 37, 38, 39 To assign the cores in the `cpu-affininty`, i though that enabling the physical processor instead of hyperthreading would be better. But now i don’t know what to put in the `cpu-affinity`. Can you help me? Trying to make a first approach, i though it would be better assign the P-cores to the workers cpu set, and the E-cores to the management, leaving this config: ```auto threading: cpu-affinity: - management-cpu-set: cpu: [16-19] - worker-cpu-set: cpu: [0,2,4,6,8,10,12,14] mode: exclusive prio: default: high ``` I would be very grateful if you could tell me what is the best option. 1. Okay. I have delete the both config from the `cpu-affinity` section 2. Values set as you say, but it does not seem to be the root cause of the low performance 3. When setting the `RX`/`TX` descriptors to 32768, it happens what you said. In the `suricata.log` i can see this: ```auto [7722 - Suricata-Main] 2024-10-16 11:39:17 Warning: dpdk: 0000:05:00.0: device queue descriptors adjusted (RX: from 32768 to 4096, TX: from 32768 to 4096) ``` So the NIC is also downgrading the `RX`/`TX` descriptors to 4096. Maybe a solution to reach a better performance with DPDK mode would be to change to other NIC which allows the adjustment of `RX`/`TX` descriptors? Hope you can help me. Thank you very much! ### Attached files - `suricata.yaml` - `suricata.log` of the new execution - `cpuinfo.log` → `cat /proc/cpuinfo` [suricata.yaml](https://forum.suricata.io/uploads/short-url/4U9jqfsvbSdTpVTMsR9dIUugIeE.yaml) (88.2 KB) [suricata.log](https://forum.suricata.io/uploads/short-url/5X5NC3dlSFMotnUjDYTU0LYrOFJ.log) (75.6 KB) [cpuinfo.log](https://forum.suricata.io/uploads/short-url/4WB5D3jiZd39Tfnu3WWEr8bdMFX.log) (33.5 KB) ---
**Author:** ![lukashino](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.suricata.io/lukashino/32/1150_2.png) [@lukashino](https://forum.suricata.io/u/lukashino)\ **Post date:** [October 23, 2024, 8:04am UTC](https://forum.suricata.io/t/high-packet-drop-rate-with-dpdk-compared-to-af-packet-in-suricata-7-0-7/4896/5 "2024-10-23T08:04:07Z")
Hi @vendul0g, thanks for the detailed report. **Hugepages** The message about hugepages is meant to say that you are overallocated hugepages and some can be freed. So in your case you don’t even need to allocate 8 GB but only 1 GB and the system can have more memory available for other stuff. Increasing the hugepage allocation will not help because Suricata simply won’t use them. **CPU affinity** Yes, your thinking is correct, assigning `0,2,4,6,8,10,12,14` seems like the best idea. For the management cores - maybe I would reduce the CPU list to 16-17, leaving 18-19 to the other system tasks. **NIC compatiblity** I have a good experience with Mellanox/NVIDIA NICs, here is a list of what I work with: - MCX516A-CCAT - MCX623106AE-CDAT - MCX75310AAS-NEAT With these cards, you can set a higher number of RX/TX descriptors which should solve the issue. ---
**Author:** ![vendul0g](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.suricata.io/vendul0g/32/3808_2.png) [@vendul0g](https://forum.suricata.io/u/vendul0g)\ **Post date:** [October 24, 2024, 9:18am UTC](https://forum.suricata.io/t/high-packet-drop-rate-with-dpdk-compared-to-af-packet-in-suricata-7-0-7/4896/6 "2024-10-24T09:18:21Z")
But i don’t understand how can be possible that `af-packet` mode could perform better than DPDK in the same hardware. Have you ever seen something like that? This is my AF-PACKET config setup: ```auto af-packet: - interface: enp5s0f0 threads: 20 cluster-id: 99 cluster-type: cluster_qm defrag: yes use-mmap: yes mmap-locked: yes tpacket-v3: yes use-emergency-flush: yes disable-promisc: no checksum-checks: no ``` In `AF-PACKET` mode, i use `ixgbe` driver. For DPDK setup, i change the driver to `vfio-pci`. I don’t know if this could be the problem of that different. ---
**Author:** ![lukashino](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.suricata.io/lukashino/32/1150_2.png) [@lukashino](https://forum.suricata.io/u/lukashino)\ **Post date:** [October 24, 2024, 10:28am UTC](https://forum.suricata.io/t/high-packet-drop-rate-with-dpdk-compared-to-af-packet-in-suricata-7-0-7/4896/7 "2024-10-24T10:28:26Z")
To my knowledge, the kernel (af-packet) is, independently from Suricata, receiving packets into its structures, from the HW queues to the software queues and those are bigger than the HW queues. DPDK Suricata receives packets directly from those HW queues and those are read from only in batches and where Suricata needs to process the batch and only then get another batch. These queues might get full on some workload bursts and because kernel receives the data continuously independently from the Suricata processing, the workload spikes are handled better. I stumbled upon this problem here, I forgot to reply to André though. > [@DPDK Performance decreases with more threads](https://forum.suricata.io/t/dpdk-performance-decreases-with-more-threads/4283): > > Hello all, My setup: DPDK 23.11, compiled on the host Suricata 7.0.2 compilled with DPDK and hyperscan Debian 12.2 Hardware: Intel XL710 40GbE on Numa-node 1 CPU 2x Intel 6258R Got suricata configured with workers, pinned on numa-node 1. Mgmt workers also on numa-node1 DPDK config: dpdk: eal-params: proc-type: primary allow: ["0000:d8:00.0"] interfaces: - interface: 0000:d8:00.0 threads: 8 promisc: true multicast: true checksum-checks: true … @vendul0g if you compile DPDK yourself there is one suggestion in the thread - increase the count of RX/TX descriptors by editing the respective files in DPDK (and then compile and install it) as André and myself suggested. DPDK compilation is pretty easy, once cloned you need only something like: ```bash # uninstall the previously installed DPDK meson --prefix=/usr/ build # to be installed global wise ninja -C build sudo ninja -C build install ``` ---
**Author:** ![vendul0g](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.suricata.io/vendul0g/32/3808_2.png) [@vendul0g](https://forum.suricata.io/u/vendul0g)\ **Post date:** [October 28, 2024, 1:26pm UTC](https://forum.suricata.io/t/high-packet-drop-rate-with-dpdk-compared-to-af-packet-in-suricata-7-0-7/4896/8 "2024-10-28T13:26:48Z")
Hi Lukas. It works! As you suggested, I rebuilt DPDK with the modified `IXGBE_MAX_RING_DESC` values, and it’s working. I’ll explain the process here in case anyone else needs guidance. In my case, I have an X540-AT2 NIC, which limits the `RX` and `TX` descriptors to 4096. To identify this issue, I ran Suricata in verbose mode: `suricata --dpdk -vvvv`, and noticed the following entries in the logs: `[7722 - Suricata-Main] 2024-10-16 11:39:17 Warning: dpdk: 0000:05:00.0: device queue descriptors adjusted (RX: from 32768 to 4096, TX: from 32768 to 4096)`. Because of this restriction, DPDK’s performance was very slow, to the point where even `AF_PACKET` mode outperformed DPDK. To resolve this, you first need to identify which driver your network card (NIC) uses—in my case, it’s `ixgbe`. Next, navigate to the directory where you have DPDK downloaded (or download it if you haven’t already). Then modify one file, depending on your NIC’s driver: - `dpdk/drivers/net/ixgbe/ixgbe_rxtx.h` → `#define IXGBE_MAX_RING_DESC 32768` Finally, reinstall DPDK: ```bash # uninstall the previously installed DPDK meson --prefix=/usr/ build # for a system-wide installation ninja -C build sudo ninja -C build install ``` Thank you so much, Lukas. You’ve been an enormous help! ---
**Author:** ![lukashino](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.suricata.io/lukashino/32/1150_2.png) [@lukashino](https://forum.suricata.io/u/lukashino)\ **Post date:** [October 28, 2024, 4:45pm UTC](https://forum.suricata.io/t/high-packet-drop-rate-with-dpdk-compared-to-af-packet-in-suricata-7-0-7/4896/9 "2024-10-28T16:45:51Z")
Hi vendul0g, great to hear that, I’m glad we have pushed this together. This could be actually a good reason to send a patch to DPDK to increase the RX/TX descriptors as they capped them unnecessarily low - I will try to follow-up on this.