Suricata 7 , dpdk , Redhat 8 initial setup advice and tips appreciated

atbohmer · April 6, 2023, 11:08am

Hi,
Finnaly managd to get it working all together, but please need to reflect if I made the right choices and configuration assumptions. For now running with root priviliges but hope to run suricata with dpdk under a non root account.

HPE ProLiant DL380 Gen10 Plus
2 cpu nodes, 128 cores
128Gb memory
2 x Intel 810 100Gb nic’s for spanport data

DPDK source from git version 23.07.0-rc0
Suricata source from git version 7.0.0-rc2-dev (caf9940fd 2023-04-04)

info on the Intel 810 card(s) before dpdk setup:

$ ethtool -i eth7
driver: ice
version: 4.18.0-425.13.1.el8_7.x86_64
firmware-version: 3.20 0x80012763 1.3189.0
expansion-rom-version:
bus-info: 0000:84:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

$ ethtool -l eth7
Channel parameters for eth7:
Pre-set maximums:
RX:             128
TX:             128
Other:          1
Combined:       128
Current hardware settings:
RX:             0
TX:             0
Other:          1
Combined:       128

Boot options: grub option intel_iommu=on added

$ dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:10:00.0 'Ethernet Controller E810-C for QSFP 1592' drv=vfio-pci unused=ice
0000:10:00.1 'Ethernet Controller E810-C for QSFP 1592' drv=vfio-pci unused=ice
0000:84:00.0 'Ethernet Controller E810-C for QSFP 1592' drv=vfio-pci unused=ice
0000:84:00.1 'Ethernet Controller E810-C for QSFP 1592' drv=vfio-pci unused=ice

Huge pages:

$ dpdk-hugepages.py -s
Node Pages Size Total
0    16384 2Mb    32Gb
1    16384 2Mb    32Gb

Hugepages mounted on /dev/hugepages

Suricata config regarding dpdk:

dpdk:
  eal-params:
    proc-type: primary

  interfaces:
      threads: auto


      mempool-cache-size: 257
      copy-mode: none

    - interface: default
      threads: auto
      promisc: true
      multicast: true
      checksum-checks: true
      checksum-checks-offload: true
      mtu: 1518
      rss-hash-functions: auto
      mempool-size: 65535
      mempool-cache-size: 257
      copy-mode: none
      copy-iface: none

    - interface: 0000:10:00.1
      threads: auto
      promisc: true
      multicast: true
      checksum-checks: true
      checksum-checks-offload: true
      mtu: 1518
      rss-hash-functions: auto
      mempool-size: 65535
      mempool-cache-size: 257
      copy-mode: none
      copy-iface: none

    - interface: 0000:84:00.0
      threads: auto
      promisc: true
      multicast: true
      checksum-checks: true
      checksum-checks-offload: true
      mtu: 1518
      rss-hash-functions: auto
      mempool-size: 65535
      mempool-cache-size: 257
      copy-mode: none
      copy-iface: none

    - interface: 0000:84:00.1
      threads: auto
      promisc: true
      multicast: true
      checksum-checks: true
      checksum-checks-offload: true
      mtu: 1518
      rss-hash-functions: auto
      mempool-size: 65535
      mempool-cache-size: 257
      copy-mode: none
      copy-iface: none

suricata systemd service file:

# Sample Suricata systemd unit file.
[Unit]
Description=Suricata Intrusion Detection Service
After=syslog.target network-online.target

[Service]
# Environment file to pick up $OPTIONS. On Fedora/EL this would be
# /etc/sysconfig/suricata, or on Debian/Ubuntu, /etc/default/suricata.
EnvironmentFile=-/etc/sysconfig/suricata
#EnvironmentFile=-/etc/default/suricata
ExecStartPre=/bin/rm -f /var/run/suricata.pid
ExecStart=/usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid $OPTIONS
ExecReload=/bin/kill -USR2 $MAINPID
LimitNOFILE=1024000
LimitMEMLOCK=infinity
LimitMEMLOCKSoft=infinity

[Install]
WantedBy=multi-user.target

Starting suricata takes way to long , maybe has todo with number of threads?

 tail -f -s0 /var/log/suricata/suricata.log 
[5956 - Suricata-Main] 2023-04-06 11:46:40 Info: runmodes: 0000:10:00.0: creating 128 threads
[5956 - Suricata-Main] 2023-04-06 11:46:43 Info: dpdk: 0000:10:00.1: creating packet mbuf pool mempool_0000:10:00.1 of size 65535, cache size 257, mbuf size 2176
[5956 - Suricata-Main] 2023-04-06 11:46:43 Info: runmodes: 0000:10:00.1: creating 128 threads
[15216 - Suricata-Main] 2023-04-06 12:32:32 Notice: suricata: This is Suricata version 7.0.0-rc2-dev (caf9940fd 2023-04-04) running in SYSTEM mode
[15216 - Suricata-Main] 2023-04-06 12:32:32 Info: cpu: CPUs/cores online: 128
[15216 - Suricata-Main] 2023-04-06 12:32:32 Info: suricata: Preparing unexpected signal handling
[15216 - Suricata-Main] 2023-04-06 12:32:32 Info: privs: dropped the caps for main thread
[15216 - Suricata-Main] 2023-04-06 12:32:32 Info: conf: Running in live mode, activating unix socket
[15216 - Suricata-Main] 2023-04-06 12:32:32 Info: logopenfile: eve-log output device (regular) initialized: /data/sensor_data/suricata/eve.json
[15216 - Suricata-Main] 2023-04-06 12:32:32 Info: logopenfile: stats output device (regular) initialized: stats.log
[15216 - Suricata-Main] 2023-04-06 12:32:45 Info: detect: 1 rule files processed. 70441 rules successfully loaded, 0 rules failed
[15216 - Suricata-Main] 2023-04-06 12:32:45 Info: threshold-config: Threshold config parsed: 0 rule(s) found
[15216 - Suricata-Main] 2023-04-06 12:32:45 Info: detect: 70444 signatures processed. 2352 are IP-only rules, 14806 are inspecting packet payload, 53255 inspect application layer, 0 are decoder event only
[15216 - Suricata-Main] 2023-04-06 12:33:18 Info: dpdk: 0000:10:00.0: creating packet mbuf pool mempool_0000:10:00.0 of size 65535, cache size 257, mbuf size 2176
[15216 - Suricata-Main] 2023-04-06 12:33:18 Info: runmodes: 0000:10:00.0: creating 128 threads
[15216 - Suricata-Main] 2023-04-06 12:33:34 Info: dpdk: 0000:10:00.1: creating packet mbuf pool mempool_0000:10:00.1 of size 65535, cache size 257, mbuf size 2176
[15216 - Suricata-Main] 2023-04-06 12:33:38 Info: runmodes: 0000:10:00.1: creating 128 threads

[15216 - Suricata-Main] 2023-04-06 12:40:48 Info: dpdk: 0000:84:00.0: creating packet mbuf pool mempool_0000:84:00.0 of size 65535, cache size 257, mbuf size 2176
[15216 - Suricata-Main] 2023-04-06 12:41:10 Info: runmodes: 0000:84:00.0: creating 128 threads
[20857 - W#128-84:00.0] 2023-04-06 12:56:42 Warning: dpdk: 0000:84:00.0: NIC is on NUMA 1, 128 threads on different NUMA node(s)
[15216 - Suricata-Main] 2023-04-06 12:56:47 Info: dpdk: 0000:84:00.1: creating packet mbuf pool mempool_0000:84:00.1 of size 65535, cache size 257, mbuf size 2176
[15216 - Suricata-Main] 2023-04-06 12:57:20 Info: runmodes: 0000:84:00.1: creating 128 threads

vjulien · April 7, 2023, 5:15am

By listing 4 interfaces each with threads: auto with 128 cores, I think it is going to create 512 threads (4x128). That is probably excessive even for this machine. What happens if you limit it to 16 or 32 per interface?

atbohmer · April 7, 2023, 7:22am

Indeed, it took a hour to start, but it started Is there some rule of thumbd regarding theads, interfaces and cores of just try and find out?

32 Instead of auto for threads fired up suricate within minutes:

tail -f -s0 /var/log/suricata/suricata.log

[275632 - Suricata-Main] 2023-04-07 09:17:31 Info: runmodes: 0000:10:00.0: creating 32 threads
[275632 - Suricata-Main] 2023-04-07 09:17:32 Info: dpdk: 0000:10:00.1: creating packet mbuf pool mempool_0000:10:00.1 of size 65535, cache size 257, mbuf size 2176
[275632 - Suricata-Main] 2023-04-07 09:17:32 Info: runmodes: 0000:10:00.1: creating 128 threads
[276456 - Suricata-Main] 2023-04-07 09:19:18 Notice: suricata: This is Suricata version 7.0.0-rc2-dev (caf9940fd 2023-04-04) running in SYSTEM mode
[276456 - Suricata-Main] 2023-04-07 09:19:18 Info: cpu: CPUs/cores online: 128
[276456 - Suricata-Main] 2023-04-07 09:19:18 Info: suricata: Preparing unexpected signal handling
[276456 - Suricata-Main] 2023-04-07 09:19:18 Info: privs: dropped the caps for main thread
[276456 - Suricata-Main] 2023-04-07 09:19:18 Info: conf: Running in live mode, activating unix socket
[276456 - Suricata-Main] 2023-04-07 09:19:18 Info: logopenfile: eve-log output device (regular) initialized: /data/sensor_data/suricata/eve.json
[276456 - Suricata-Main] 2023-04-07 09:19:18 Info: logopenfile: stats output device (regular) initialized: stats.log
[276456 - Suricata-Main] 2023-04-07 09:19:31 Info: detect: 1 rule files processed. 70150 rules successfully loaded, 0 rules failed
[276456 - Suricata-Main] 2023-04-07 09:19:31 Info: threshold-config: Threshold config parsed: 0 rule(s) found
[276456 - Suricata-Main] 2023-04-07 09:19:31 Info: detect: 70153 signatures processed. 2344 are IP-only rules, 14514 are inspecting packet payload, 53264 inspect application layer, 0 are decoder event only
[276456 - Suricata-Main] 2023-04-07 09:20:05 Info: dpdk: 0000:10:00.0: creating packet mbuf pool mempool_0000:10:00.0 of size 65535, cache size 257, mbuf size 2176
[276456 - Suricata-Main] 2023-04-07 09:20:05 Info: runmodes: 0000:10:00.0: creating 32 threads
[276456 - Suricata-Main] 2023-04-07 09:20:07 Info: dpdk: 0000:10:00.1: creating packet mbuf pool mempool_0000:10:00.1 of size 65535, cache size 257, mbuf size 2176
[276456 - Suricata-Main] 2023-04-07 09:20:07 Info: runmodes: 0000:10:00.1: creating 32 threads
[276456 - Suricata-Main] 2023-04-07 09:20:08 Info: dpdk: 0000:84:00.0: creating packet mbuf pool mempool_0000:84:00.0 of size 65535, cache size 257, mbuf size 2176
[276456 - Suricata-Main] 2023-04-07 09:20:08 Info: runmodes: 0000:84:00.0: creating 32 threads
[276869 - W#32-84:00.0] 2023-04-07 09:20:09 Warning: dpdk: 0000:84:00.0: NIC is on NUMA 1, 32 threads on different NUMA node(s)
[276456 - Suricata-Main] 2023-04-07 09:20:09 Info: dpdk: 0000:84:00.1: creating packet mbuf pool mempool_0000:84:00.1 of size 65535, cache size 257, mbuf size 2176
[276456 - Suricata-Main] 2023-04-07 09:20:09 Info: runmodes: 0000:84:00.1: creating 32 threads
[276905 - W#32-84:00.1] 2023-04-07 09:20:17 Warning: dpdk: 0000:84:00.1: NIC is on NUMA 1, 32 threads on different NUMA node(s)
[276456 - Suricata-Main] 2023-04-07 09:20:21 Info: unix-manager: unix socket ‘/var/run/suricata/suricata-command.socket’
[276456 - Suricata-Main] 2023-04-07 09:20:25 Notice: threads: Threads created → W: 128 FM: 1 FR: 1 Engine started.

atbohmer · April 7, 2023, 8:19am

Regarding CPU affinity and NUMA, how do I pin those dpdk interfaces? I know how to find out which interface relates to which node, but how to make configure this right in suricata?

lukashino · April 20, 2023, 7:08pm

I think you could order your interface in within the interfaces list so that you first start with the first NUMA node and then define the NICs located on the other NUMA node.

something like

dpdk:
  interfaces:
    - interface: 7b:00.0 (node 0)
      threads: 4
    - interface: 7b:00.1 (node 0)
      threads: 4
    - interface: 8c:00.0 (node 1)
      threads: 4
    - interface: 8c:00.0 (node 1)
      threads: 4

threading:
  set-cpu-affinity: yes
  workers: [0-7 (NUMA 0), 64-71 (NUMA 1)]

Hopefully my “illustration” is correct and helpful.

atbohmer · April 21, 2023, 9:08am

Thanks, but my NUMA config is more complicated then 1-10 for 0 and 11-20 for 1, it is mixed sigh:
NUMA node0 CPU(s): 0-31,64-95
NUMA node1 CPU(s): 32-63,96-127

So this works for me the way suricata handles affinity with threads: 24 per interface:
cpu: [ “6-29”,“38-61”,“64-87”,“96-119” ]
And this interface order:
- interface: 0000:10:00.0
- interface: 0000:84:00.0
- interface: 0000:10:00.1
- interface: 0000:84:00.1

lukashino · April 21, 2023, 10:10am

Perfect, great to hear that.