Suricata IDS on 100Gbps link via AF_PACKET and an Intel E810-CQDA2 with ICE driver

I recently decided to move from dedicated FPGA capture cards and PF_RING to less expensive Intel E810 and AF_PACKET and after much experimentation I have figured out the optimal settings for the NIC and system to achieve 0 packet loss in Suricata and infinitesimal packet loss on the NIC, a quick rundown is below. Though we do not in fact achieve 100Gbps on the link, I have so far logged line rates to 64Gbps++ and things are all good.

Be aware that on some distros, this one being Suse LEAP 15, when the Intel card is installed, Suse will want to install the ice and i40e drivers and irdma. The performance I have achieved was using the latest ice driver downloaded from Intel and compiled on the system, the drivers installed by the package manager provide less performance and you do not need irdma whatsoever, it just gets in the way. I am unaware of a method to prevent the package manager from updating to the distros version of these kernel modules, as there are other binaries included in the kernel-firmware-intel package that should be installed and updated, so the entire package can not be excluded, thus each time you update, you must manually remove irdma, ice and i40e installed by the package manager, and recompile the ice driver.

############################################################################

System Information
Manufacturer: Dell Inc.
Product Name: PowerEdge R450

2x Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz

2x 64 GB DDR 3200 MT/s

E810-CQDA2 inserted into a 16GT/s, x16 PCI slot in Numa Node1

NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63

###########################################################################
Enable hugepages using systemd and related scripts below which reserve and assign the memory to NUMA node1

 cat /usr/lib/systemd/system/hugetlb-gigantic-pages.service
[Unit]
Description=HugeTLB Gigantic Pages Reservation
DefaultDependencies=no
Before=dev-hugepages.mount
ConditionPathExists=/sys/devices/system/node
ConditionKernelCommandLine=hugepagesz=1G

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/lib/systemd/hugetlb-reserve-pages.sh

[Install]
WantedBy=sysinit.target

Script executed by hugpages startup systemd service

#!/bin/sh

nodes_path=/sys/devices/system/node/
if [ ! -d $nodes_path ]; then
        echo "ERROR: $nodes_path does not exist"
        exit 1
fi

reserve_pages()
{
        echo $1 > $nodes_path/$2/hugepages/hugepages-1048576kB/nr_hugepages
}

reserve_pages 62 node1

###########################################################################

Update the grub config and add the following kernel parameters, the CPUs listed correspond to the CPUs that are pinned for use by Suricata workers - I have probably over-killed with the number of CPU assigned so you might want to play with this a bit, I will do so as time permits, all I wanted right now was guaranteed performance with next to nil packet drops on the NIC and Suricata.

GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G

############################################################################

I chose to use the Suricata systemd startup script to also configure the card using ExecStartPre , you can also choose to make these settings permanent using the Network Manager nmcli

I set the card MTU to 9200 because our network supports jumbo frames

cat /etc/systemd/system/suricata.service
# Sample Suricata systemd unit file.
[Unit]
Description=Suricata Intrusion Detection Service
After=syslog.target network-online.target systemd-tmpfiles-setup.service
Documentation=man:suricata(1)

[Service]
Type=simple
# Environment file to pick up $OPTIONS. On Fedora/EL this would be
# /etc/sysconfig/suricata, or on Debian/Ubuntu, /etc/default/suricata.
EnvironmentFile=-/etc/sysconfig/suricata
ExecStartPre=/opt/suricata/usr/bin/intel-810-setup.sh
ExecStartPre=/bin/rm -f /var/run/suricata.pid
ExecStart=/opt/suricata/bin/suricata --af-packet --user suri --group suri --pidfile /var/run/suricata.pid
ExecReload=/bin/kill -USR2 $MAINPID
RemainAfterExit=true

[Install]
WantedBy=multi-user.target

#########################################
The script used by the systemd service config above

cat /opt/suricata/usr/bin/intel-810-setup.sh 
#!/bin/bash

/sbin/rmmod ice && /sbin/modprobe ice
/usr/bin/sleep 1
/usr/bin/ifconfig p3p1 down
/usr/bin/sleep 1
/usr/sbin/ethtool -L p3p1 combined 32
/usr/sbin/ethtool -K p3p1 rxhash on
/usr/sbin/ethtool -K p3p1 ntuple on
/usr/bin/ifconfig p3p1 up
/usr/bin/sleep 1
/root/intel-ice-driver/PROCGB/Linux/ice-1.14.13/scripts/set_irq_affinity local p3p1
/usr/sbin/ethtool -X p3p1 hkey 6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A equal 32
/usr/sbin/ethtool -A p3p1 rx off tx off
/usr/sbin/ethtool -C p3p1 adaptive-rx off adaptive-tx off rx-usecs 125
/usr/sbin/ethtool -G p3p1 rx 512
for proto in tcp4 udp4 tcp6 udp6; do /usr/sbin/ethtool -N p3p1 rx-flow-hash $proto sdfn; done
/sbin/ip link set dev p3p1 mtu 9200
/sbin/ip link set p3p1 promisc off arp off
for i in rx tx tso gso gro lro sg txvlan rxvlan; do /usr/sbin/ethtool -K p3p1 $i off; done;

############################################################################

Disable IRQ Balance

systemctl stop irqbalance.service
systemctl disable irqbalance.service

############################################################################
From the Suricata configuration

cpu-affinity:
    - management-cpu-set:
        cpu: [ 2,4,6,8 ]  # include only these cpus in affinity settings
        mode: balanced
        prio:
          default: "low"
    - receive-cpu-set:
        cpu: [ 1,3,5,7 ]
    - worker-cpu-set:
        cpu: [ 9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63 ]  # include only these cpus in affinity settings
        mode: exclusive
        prio:
          high: [ 9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63 ]
          default: "high"

############################################################################

A script output that shows me the current stats for the NIC and Suricata

./check-suricata-drop-statuses.sh
capture.kernel_drops | Total | 0
tcp.ssn_memcap_drop | Total | 0
tcp.segment_memcap_drop | Total | 0
capture.kernel_drops | Total | 0
tcp.ssn_memcap_drop | Total | 0
tcp.segment_memcap_drop | Total | 0
capture.kernel_drops | Total | 0
tcp.ssn_memcap_drop | Total | 0
tcp.segment_memcap_drop | Total | 0
capture.kernel_drops | Total | 0
tcp.ssn_memcap_drop | Total | 0
tcp.segment_memcap_drop | Total | 0
capture.kernel_drops | Total | 0
tcp.ssn_memcap_drop | Total | 0
tcp.segment_memcap_drop | Total | 0
capture.kernel_drops | Total | 0
tcp.ssn_memcap_drop | Total | 0
tcp.segment_memcap_drop | Total | 0
capture.kernel_drops | Total | 0
tcp.ssn_memcap_drop | Total | 0
tcp.segment_memcap_drop | Total | 0
capture.kernel_drops | Total | 0
tcp.ssn_memcap_drop | Total | 0
tcp.segment_memcap_drop | Total | 0
capture.kernel_drops | Total | 0
tcp.ssn_memcap_drop | Total | 0
tcp.segment_memcap_drop | Total | 0
capture.kernel_drops | Total | 0
tcp.ssn_memcap_drop | Total | 0
tcp.segment_memcap_drop | Total | 0
10: p3p1: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 9200 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 6c:fe:54:40:3a:e0 brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped missed mcast
23822230095352 23775288957 0 146601 0 121415
TX: bytes packets errors dropped carrier collsns
1075893 5910 0 0 0 0
altname enp152s0
NIC drop percentage: 0.000617 <-- Yes indeed that is percentage, basically nothing!

The script that outputs the above stats

#!/bin/bash

/usr/bin/fgrep drop /opt/suricata/var/log/suricata/stats.log | /usr/bin/tail -n 30

/sbin/ip -s link show dev p3p1
/sbin/ip -s link show dev p3p1 | perl -e 'while(<>) { if($_ =~ /\d+\s+(\d+)\s+\d+\s+(\d+)\s+\d+\s+\d+/) { $v1 = $1; $v2 = $2; $t = (($v2 / $v1) * 100); $s = sprintf("%.6f", $t); print "NIC drop percentage: $s\n"; exit; }}'

############################################################################

I also recommend perusing this great informational link which has good tips for system Bios settings, be sure to set your system to maximum performance

Thanks for posting !
A few of questions out of curiosity if ok :
Which Suricata version are you running?
What type of traffic is this (like corporate/university/ISP etc)?
Are you running af_packet regular or the XDP variant?
What kind of a ruleset are you running ?

Hi Peter,

Suricata version is currently 7.0.6
It is monitoring open research university traffic thus the jumbo frames
Regular AF_PACKET
Enabled sources:

  • etnetera/aggressive
  • et/pro
    Total number of enabled rules loaded: 63029

Greg

Awesome - thanks for sharing the info!