Suricata IDS on 100Gbps link via AF_PACKET and an Intel E810-CQDA2 with ICE driver

Hello back again with more real world stats and a golden config. This has been a real adventure as I have found tuning Suricata and the underlying system for high throughput is not a simple task, and I have been round and round with tuning as time permits. Now I have reached the point where Suricata has been running for days with no issue, and so I believe that I have finally determined the perfect configurations.

The current stats below reflect throughput where the bursts during the most busy period were up to 64Gbps and averaging 30Gbps for periods of time, then slowing down to 15Gbps. The most important aspect of this is that Suricata is now catching everything it should. The alerts I have ginned up that let me know when PCRE rules are failing no longer fire, and this the result I have been chasing.

Observing throughput visually on another system running a Silicom FPGA, I saw bursting periods of 10 or 15 seconds where throughput was up to 64Gbps, longer periods of sustained 30Gbps, and falling eventually to a sustained 15Gbps. Making calculations from the Suricata stats.log I determined the only time the NIC and capture kernels were dropping any significant traffic was during the 5 minute period where we burst to 64Gbps, after which the drop averages calculated continually fall off, meaning basically that overall there are no 5 minute windows where the capture kernel or NIC drops any significant traffic except at for very high line rates as the trend below illustrates. The NIC is now tuned so well it is really not in effect anymore but I wanted to include the stats anway.

Capture kernel drop percentage 4.6100
NIC drop percentage: 0.353842

Capture kernel drop percentage 2.9100
NIC drop percentage: 0.246900

Capture kernel drop percentage 2.6800
NIC drop percentage: 0.219003

Capture kernel drop percentage 1.3300
NIC drop percentage: 0.112731

Capture kernel drop percentage 0.7900
NIC drop percentage: 0.072851

Capture kernel drop percentage 0.1300
NIC drop percentage: 0.003810

If you have the same high throughput needs as I and are thinking of setting up a system based on this tips page, be aware that on some distros, this one being Suse LEAP 15, when the Intel card is installed, Suse will want to install the ice and i40e drivers and irdma. Yet the performance I have achieved is had by removing the OS installed drivers and compiling the latest ice driver downloaded from Intel on the IDS system inself. The drivers installed by the package manager provide less performance and you do not need irdma whatsoever, it just gets in the way.

I am unaware of a method to prevent the package manager from updating and installing the kernel modules contained in the distro version, as there are other binaries included in the kernel-firmware-intel package that should be installed and updated, you can not selectively choose what will be installed from that package, so the entire package can not be excluded, thus each time you update, you must manually remove irdma, ice and i40e installed by the package manager, and recompile and install the custom ice driver.

############################################################################

System Information
Manufacturer: Dell Inc.
Product Name: PowerEdge R450

2x Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz

2x 64 GB DDR 3200 MT/s

E810-CQDA2 inserted into a 16GT/s, x16 PCI slot in Numa Node1

NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63

###########################################################################
Enable hugepages using systemd and related scripts below which reserve and assign the memory to NUMA node1

 cat /usr/lib/systemd/system/hugetlb-gigantic-pages.service
[Unit]
Description=HugeTLB Gigantic Pages Reservation
DefaultDependencies=no
Before=dev-hugepages.mount
ConditionPathExists=/sys/devices/system/node
ConditionKernelCommandLine=hugepagesz=1G

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/lib/systemd/hugetlb-reserve-pages.sh

[Install]
WantedBy=sysinit.target

Script executed by hugepages startup systemd service

#!/bin/sh

nodes_path=/sys/devices/system/node/
if [ ! -d $nodes_path ]; then
        echo "ERROR: $nodes_path does not exist"
        exit 1
fi

reserve_pages()
{
        echo $1 > $nodes_path/$2/hugepages/hugepages-1048576kB/nr_hugepages
}

reserve_pages 62 node1

###########################################################################

Update the grub config and add the following kernel parameters, the CPUs listed under isolcpus correspond to the CPUs that are pinned for use by Suricata workers - this isolates the CPUs from the scheduler so that they should be reserved only for Suricata's use.  I've read that doing this "could" have a bit of a negative effect as the scheduler than can not help if there is an issue, but it appears so far this is a good update.

GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G isolcpus=1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63

############################################################################

I chose to use the Suricata systemd startup script to also configure the card using ExecStartPre , you can also choose to make these settings permanent using nmcli but as you will see below you really want to keep NetworkManager out of the picture, so better not use nmcli.

I set the card MTU to 9200 because our network supports jumbo frames and though it is not a significant part of the traffic it does exist.

cat /etc/systemd/system/suricata.service
# Sample Suricata systemd unit file.
[Unit]
Description=Suricata Intrusion Detection Service
After=syslog.target network-online.target systemd-tmpfiles-setup.service
Documentation=man:suricata(1)

[Service]
Type=simple
# Environment file to pick up $OPTIONS. On Fedora/EL this would be
# /etc/sysconfig/suricata, or on Debian/Ubuntu, /etc/default/suricata.
LimitMEMLOCK=infinity
EnvironmentFile=-/etc/sysconfig/suricata
ExecStartPre=/opt/suricata/usr/bin/intel-810-setup.sh
ExecStartPre=/bin/rm -f /var/run/suricata.pid
ExecStart=/opt/suricata/bin/suricata --af-packet --user suri --group suri --pidfile /var/run/suricata.pid
ExecReload=/bin/kill -USR2 $MAINPID
RemainAfterExit=true

[Install]
WantedBy=multi-user.target

#########################################
This is the script used by the systemd service config above, note that I assigned IRQ Affinity to the CPUs in NUMA0, and the reason is all CPUs in NUMA1 are exclusively reserved for Suricata, and I want to keep it that way; please refer back to the grub config setting. :slight_smile:

cat /opt/suricata/usr/bin/intel-810-setup.sh 
#!/bin/bash

/sbin/rmmod ice && /sbin/modprobe ice
/usr/bin/sleep 1
/usr/bin/ifconfig p3p1 down
/usr/bin/sleep 1
/usr/sbin/ethtool -L p3p1 combined 28 # Increase combined channels to match workers
/usr/sbin/ethtool -K p3p1 rxhash on
/usr/sbin/ethtool -K p3p1 ntuple on
/usr/bin/ifconfig p3p1 up
/usr/bin/sleep 1
/root/installers2/PROCGB/Linux/ice-1.15.4/scripts/set_irq_affinity local p3p1
/usr/sbin/ethtool -X p3p1 hkey 6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A equal 28  # Match RSS queues to combined channels
/usr/sbin/ethtool -A p3p1 rx off tx off
/usr/sbin/ethtool -C p3p1 adaptive-rx off adaptive-tx off rx-usecs 125
/usr/sbin/ethtool -G p3p1 rx 8192
for proto in tcp4 udp4 tcp6 udp6; do /usr/sbin/ethtool -N p3p1 rx-flow-hash $proto sdfn; done
/sbin/ip link set dev p3p1 mtu 9200
/sbin/ip link set p3p1 promisc on arp off
for i in rx tx tso gso gro lro sg txvlan rxvlan; do /usr/sbin/ethtool -K p3p1 $i off; done;

# IRQ affinity setup
# Use NUMA0 CPUs, avoid CPU 0 and SMT overlap
# Assigned CPUs: 2, 4, 6, ..., 56

IRQ_CPUS_BITMASKS=(
  0000000000000004  # CPU 2
  0000000000000010  # CPU 4
  0000000000000040  # CPU 6
  0000000000000100  # CPU 8
  0000000000000400  # CPU 10
  0000000000001000  # CPU 12
  0000000000004000  # CPU 14
  0000000000010000  # CPU 16
  0000000000040000  # CPU 18
  0000000000100000  # CPU 20
  0000000000400000  # CPU 22
  0000000001000000  # CPU 24
  0000000004000000  # CPU 26
  0000000010000000  # CPU 28
  0000000040000000  # CPU 30
  0000000100000000  # CPU 32
  0000000400000000  # CPU 34
  0000001000000000  # CPU 36
  0000004000000000  # CPU 38
  0000010000000000  # CPU 40
  0000040000000000  # CPU 42
  0000100000000000  # CPU 44
  0000400000000000  # CPU 46
  0001000000000000  # CPU 48
  0004000000000000  # CPU 50
  0010000000000000  # CPU 52
  0040000000000000  # CPU 54
  0100000000000000  # CPU 56
)

i=0
for IRQ in $(grep p3p1 /proc/interrupts | awk '{print $1}' | sed 's/://'); do
    echo ${IRQ_CPUS_BITMASKS[$((i % ${#IRQ_CPUS_BITMASKS[@]}))]} > /proc/irq/$IRQ/smp_affinity
    i=$((i + 1))
done

# Disable RPS — Suricata uses AF_PACKET with 1:1 thread-to-queue mapping
for RX_QUEUE in /sys/class/net/p3p1/queues/rx-*; do
    echo 0 > $RX_QUEUE/rps_cpus
done

############################################################################

Disable IRQ Balance

systemctl stop irqbalance.service
systemctl disable irqbalance.service

############################################################################

Create the following file with the entries shown and reboot or execute sysctl --system to reset without rebooting

vi /etc/sysctl.d/99-suricata-tuning.conf
net.core.rmem_max = 268435456
net.core.netdev_max_backlog = 4096

############################################################################

Even though network manager was not managing the Intel E810 I did the following as it can’t hurt just to be sure that it will never (due to some unforseen change in network manager after an update) inadvertently begin managing the capture device.

vi /etc/NetworkManager/conf.d/unmanaged-captures.conf
[keyfile]
unmanaged-devices=interface-name:p3p1

############################################################################

The Suricata configuration (commented lines removed for brevity

%YAML 1.1
---

vars:
  address-groups:
    HOME_NET: "[]"

    EXTERNAL_NET: "!$HOME_NET"

    HTTP_SERVERS: "$HOME_NET"
    SMTP_SERVERS: "[]"
    SQL_SERVERS: "$HOME_NET"
    DNS_SERVERS: "[]"
    TELNET_SERVERS: "$HOME_NET"
    AIM_SERVERS: "[]"
    DNP3_SERVER: "$HOME_NET"
    DNP3_CLIENT: "$HOME_NET"
    MODBUS_CLIENT: "$HOME_NET"
    MODBUS_SERVER: "$HOME_NET"
    ENIP_CLIENT: "$HOME_NET"
    ENIP_SERVER: "$HOME_NET"

  port-groups:
    HTTP_PORTS: "[36,80,81,82,83,84,85,86,87,88,89,90,311,383,555,591,593,631,801,808,818,901,972,1158,1220,1414,1533,1741,1830,1942,2231,2301,2381,2809,2980,3029,3037,3057,3128,3443,3702,4000,4343,4848,5000,5117,5250,5600,6080,6173,6988,7000,7001,7071,7144,7145,7510,7770,7777,7778,7779,8000,8008,8014,8028,8080,8081,8082,8085,8088,8090,8118,8123,8180,8181,8222,8243,8280,8300,8333,8344,8500,8509,8800,8888,8899,8983,9000,9060,9080,9090,9091,9111,9290,9443,9999,10000,11371,12601,13014,15489,29991,33300,34412,34443,34444,41080,44449,50000,50002,51423,53331,55252,55555,56712]"
    SHELLCODE_PORTS: "!80"
    ORACLE_PORTS: "[1521,1522,1523]"
    SSH_PORTS: 22
    DNP3_PORTS: 20000
    MODBUS_PORTS: 502
    FILE_DATA_PORTS: "[$HTTP_PORTS,110,143]"
    FTP_PORTS: "[21,2100,3535]"
    GENEVE_PORTS: 6081
    VXLAN_PORTS: 4789
    TEREDO_PORTS: 3544

default-rule-path: /opt/suricata/var/lib/suricata/rules
rule-files:
  - suricata.rules

classification-file: /opt/suricata/var/lib/suricata/rules/classification.config
reference-config-file: /opt/suricata/var/lib/suricata/rules/reference.config
threshold-file: /opt/suricata/etc/suricata/threshold.config

default-log-dir: /opt/suricata/var/log/suricata/

stats:
  enabled: yes
  interval: 320

outputs:
  - fast:
      enabled: no
      filename: fast.log
      append: yes

  - eve-log:
      enabled: yes
      filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
      filename: eve.json
      types:
        - alert:
            payload: yes               # enable dumping payload in Base64
            payload-buffer-size: 20kb   # max size of payload buffer to output in eve-log
            packet: no                 # enable dumping of packet (without stream segments)
            metadata: yes              # add L7/applayer fields, flowbit and other vars to the alert

            tagged-packets: yes

            xff:
              enabled: no
              mode: extra-data
              deployment: reverse
              header: X-Forwarded-For
        - tls:
            extended: no     # enable this for extended logging information
            session-resumption: no

        - stats:
            totals: yes        # stats for all threads merged together
            threads: no       # per thread stats
            deltas: no        # include delta values

  - unified2-alert:
      enabled: no
      filename: snort.u2

      limit: 1024mb

      payload: yes

      xff:
        enabled: no
        mode: extra-data
        deployment: reverse
        header: X-Forwarded-For

  - http-log:
      enabled: no
      filename: http.log
      append: yes

  - tls-log:
      enabled: no  # Log TLS connections.
      filename: tls.log # File to store TLS logs.
      append: yes

  - tls-store:
      enabled: no

  - dns-log:
      enabled: no
      filename: dns.log
      append: yes

  - pcap-log:
      enabled: no
      filename: log.pcap

      limit: 1000mb

      max-files: 2000

      mode: normal # normal, multi or sguil.

      use-stream-depth: no #If set to "yes" packets seen after reaching stream inspection depth are ignored. "no" logs all packets
      honor-pass-rules: no # If set to "yes", flows in which a pass rule matched will stopped being logged.

  - alert-debug:
      enabled: no
      filename: alert-debug.log
      append: yes

  - alert-prelude:
      enabled: no
      profile: suricata
      log-packet-content: no
      log-packet-header: yes

  - stats:
      enabled: yes
      filename: stats.log
      append: no        # append to file (yes) or overwrite it (no)
      totals: yes        # stats for all threads merged together
      threads: no      # per thread stats
      null-values: yes  # print counters that have value 0

  - syslog:
      enabled: no
      facility: local5

  - drop:
      enabled: no
      filename: drop.log
      append: yes
      filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'

  - file-store:
      enabled: no       # set to yes to enable
      log-dir: files    # directory to store the files
      force-magic: no   # force logging magic on all stored files
      force-filestore: no # force storing of all files

  - file-log:
      enabled: no
      filename: files-json.log
      append: yes

      force-magic: no   # force logging magic on all logged files

  - tcp-data:
      enabled: no
      type: file
      filename: tcp-data.log

  - http-body-data:
      enabled: no
      type: file
      filename: http-data.log

  - lua:
      enabled: no
      scripts:

logging:
  default-log-level: info

  default-output-filter:

  outputs:
  - console:
      enabled: yes
  - file:
      enabled: yes
      filename: /opt/suricata/var/log/suricata/suricata.log
  - syslog:
      enabled: no
      facility: local5
      format: "[%i] <%d> -- "

af-packet:
  - interface: p3p1
    threads: 28
    cluster-id: 99
    cluster-type: cluster_qm
    defrag: no
    use-mmap: yes
    mmap-locked: yes
    tpacket-v3: yes
    ring-size: 32768
    block-size: 262144
    block-cnt: 128
    block-timeout: 50
    use-emergency-flush: yes
    buffer-size: 262144
    checksum-checks: auto

pcap-file:
  checksum-checks: auto

app-layer:
  protocols:
    rfb:
      enabled: yes
      detection-ports:
        dp: 5900, 5901, 5902, 5903, 5904, 5905, 5906, 5907, 5908, 5909
    mqtt:
      enabled: no
    krb5:
      enabled: yes
    snmp:
      enabled: yes
    ikev2:
      enabled: yes
    tls:
      enabled: no
      detection-ports:
        dp: 443
      encryption-handling: bypass
    dcerpc:
      enabled: yes
    ftp:
      enabled: yes
    ssh:
      enabled: yes
    http2:
      enabled: no
    smtp:
      enabled: detection-only
      raw-extraction: no
      mime:
        decode-mime: yes

        decode-base64: yes
        decode-quoted-printable: yes

        header-value-depth: 2000

        extract-urls: yes
        body-md5: no
      inspected-tracker:
        content-limit: 100000
        content-inspect-min-size: 32768
        content-inspect-window: 6144
    imap:
      enabled: detection-only
    msn:
      enabled: detection-only
    smb:
      enabled: yes
      detection-ports:
        dp: 139, 445
    nfs:
      enabled: no
    tftp:
      enabled: yes
    dns:
      global-memcap: 128mb
      state-memcap: 4096kb

      tcp:
        enabled: yes
        detection-ports:
          dp: 53
      udp:
        enabled: yes
        detection-ports:
          dp: 53
    http:
      enabled: yes
      memcap: 4096mb

      libhtp:
         default-config:
           personality: IDS

           request-body-limit: 100kb
           response-body-limit: 100kb

           request-body-minimal-inspect-size: 32kb
           request-body-inspect-window: 4kb
           response-body-minimal-inspect-size: 40kb
           response-body-inspect-window: 16kb

           response-body-decompress-layer-limit: 2

           http-body-inline: auto

           double-decode-path: no
           double-decode-query: no

         server-config:

    modbus:

      enabled: no
      detection-ports:
        dp: 502

      stream-depth: 0

    dnp3:
      enabled: no
      detection-ports:
        dp: 20000

    enip:
      enabled: no
      detection-ports:
        dp: 44818
        sp: 44818

    ntp:
      enabled: no
    dhcp:
      enabled: yes
    sip:
      enabled: no
    rdp:
      enabled: no

asn1-max-frames: 256

coredump:
  max-dump: unlimited

host-mode: sniffer-only

max-pending-packets: 30000

runmode: workers

autofp-scheduler: active-packets

default-packet-size: 9022

unix-command:
  enabled: auto

legacy:
  uricontent: enabled

  action-order:
    - pass
    - alert
    - drop
    - reject

engine-analysis:
  rules-fast-pattern: yes
  rules: yes

pcre:
  match-limit: 6000
  match-limit-recursion: 3500

host-os-policy:
  windows: []
  bsd: []
  bsd-right: []
  old-linux: []
  linux: [0.0.0.0/0]
  old-solaris: []
  solaris: []
  hpux10: []
  hpux11: []
  irix: []
  macos: []
  vista: []
  windows2k3: []

defrag:
  memcap: 12288mb
  hash-size: 655360
  trackers: 65535 # number of defragmented flows to follow
  max-frags: 65535 # number of fragments to keep (higher than trackers)
  prealloc: yes
  timeout: 60

flow:
  memcap: 16384mb
  hash-size: 1048576
  prealloc: 1000000
  emergency-recovery: 70
  managers: 4 # default to one flow manager
  recyclers: 4 # default to one flow recycler thread

vlan:
  use-for-tracking: true

flow-timeouts:

  default:
    new: 15
    established: 120
    closed: 12
    bypassed: 100
    emergency-new: 10
    emergency-established: 100
    emergency-closed: 0
    emergency-bypassed: 50
  tcp:
    new: 60
    established: 300
    closed: 12
    bypassed: 100
    emergency-new: 5
    emergency-established: 100
    emergency-closed: 10
    emergency-bypassed: 50
  udp:
    new: 15
    established: 120
    bypassed: 100
    emergency-new: 10
    emergency-established: 100
    emergency-bypassed: 50
  icmp:
    new: 15
    established: 120
    bypassed: 100
    emergency-new: 10
    emergency-established: 100
    emergency-bypassed: 50

stream:
  memcap: 2048mb
  checksum-validation: no       # reject wrong csums
  inline: yes                   # auto will use inline mode in IPS mode, yes or no set it statically
  bypass: no
  drop-invalid: no
  reassembly:
    memcap: 4096mb
    depth: 2mb                  # reassemble 1mb into a stream
    toserver-chunk-size: 8192
    toclient-chunk-size: 8192
    randomize-chunk-size: no
  tcp-reject-non-reply: yes

host:
  hash-size: 61440
  prealloc: 10000
  memcap: 1024mb

decoder:
  teredo:
    enabled: false

detect:
  profile: custom
  custom-values:
    toclient-groups: 200
    toserver-groups: 200
  sgh-mpm-context: auto
  inspection-recursion-limit: 3000
  delayed-detect: yes

  prefilter:
    default: auto

  grouping:
    tcp-whitelist: 53, 80, 139, 443, 445, 1433, 3306, 3389, 6666, 6667, 8080
    udp-whitelist: 53, 123, 135, 5060

  profiling:
    grouping:
      dump-to-disk: false
      include-rules: false      # very verbose
      include-mpm-stats: false
mpm-algo: hs

spm-algo: hs

threading:
  set-cpu-affinity: yes
  cpu-affinity:
    - management-cpu-set:
        cpu: [ 2,4,6,8 ]  # include only these cpus in affinity settings
        mode: "exclusive"
        prio:
          default: "low"
    - receive-cpu-set:
        cpu: [ 1,3,5,7 ]
    - worker-cpu-set:
        cpu: [ 9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63 ]  # include only these cpus in affinity settings
        mode: exclusive
        prio:
          high: [ 9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63 ]
          default: "high"
  detect-thread-ratio: 1

luajit:
  states: 128

profiling:

  rules:

    enabled: no
    filename: rule_perf.log
    append: yes

    sort: avgticks

    limit: 10

    json: no

  keywords:
    enabled: yes
    filename: keyword_perf.log
    append: yes

  rulegroups:
    enabled: no
    filename: rule_group_perf.log
    append: yes

  packets:

    enabled: no
    filename: packet_stats.log
    append: yes

    csv:

      enabled: no
      filename: packet_stats.csv
  locks:
    enabled: no
    filename: lock_stats.log
    append: yes

  pcap-log:
    enabled: no
    filename: pcaplog_stats.log
    append: yes

nfq:

nflog:
  - group: 2
    buffer-size: 18432
  - group: default
    qthreshold: 1
    qtimeout: 100
    max-size: 20000

capture:

netmap:

    checksum-checks: auto

ipfw:

napatech:
    hba: -1

    use-all-streams: yes

    streams: ["0-3"]

mpipe:

  load-balance: dynamic

  iqueue-packets: 2028

  inputs:

  stack:
    size128: 0
    size256: 9
    size512: 0
    size1024: 0
    size1664: 7
    size4096: 0
    size10386: 0
    size16384: 0

cuda:
  mpm:
    data-buffer-size-min-limit: 0
    data-buffer-size-max-limit: 1500
    cudabuffer-buffer-size: 500mb
    gpu-transfer-size: 50mb
    batching-timeout: 2000
    device-id: 0
    cuda-streams: 2

############################################################################

A script output that shows me the current stats for the NIC and Suricata

./check-suricata-drop-statuses.sh
capture.kernel_drops | Total | 0
tcp.ssn_memcap_drop | Total | 0
tcp.segment_memcap_drop | Total | 0
capture.kernel_drops | Total | 0
tcp.ssn_memcap_drop | Total | 0
tcp.segment_memcap_drop | Total | 0
capture.kernel_drops | Total | 0
tcp.ssn_memcap_drop | Total | 0
tcp.segment_memcap_drop | Total | 0
capture.kernel_drops | Total | 0
tcp.ssn_memcap_drop | Total | 0
tcp.segment_memcap_drop | Total | 0
capture.kernel_drops | Total | 0
tcp.ssn_memcap_drop | Total | 0
tcp.segment_memcap_drop | Total | 0
capture.kernel_drops | Total | 0
tcp.ssn_memcap_drop | Total | 0
tcp.segment_memcap_drop | Total | 0
capture.kernel_drops | Total | 0
tcp.ssn_memcap_drop | Total | 0
tcp.segment_memcap_drop | Total | 0
capture.kernel_drops | Total | 0
tcp.ssn_memcap_drop | Total | 0
tcp.segment_memcap_drop | Total | 0
capture.kernel_drops | Total | 0
tcp.ssn_memcap_drop | Total | 0
tcp.segment_memcap_drop | Total | 0
capture.kernel_drops | Total | 0
tcp.ssn_memcap_drop | Total | 0
tcp.segment_memcap_drop | Total | 0
10: p3p1: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 9200 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 6c:fe:54:40:3a:e0 brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped missed mcast
23822230095352 23775288957 0 146601 0 121415
TX: bytes packets errors dropped carrier collsns
1075893 5910 0 0 0 0
altname enp152s0
NIC drop percentage: 0.000617 <-- Yes indeed that is percentage, basically nothing!

The script that outputs the above stats

#!/bin/bash

/usr/bin/fgrep drop /opt/suricata/var/log/suricata/stats.log | /usr/bin/tail -n 30

/sbin/ip -s link show dev p3p1
/sbin/ip -s link show dev p3p1 | perl -e 'while(<>) { if($_ =~ /\d+\s+(\d+)\s+\d+\s+(\d+)\s+\d+\s+\d+/) { $v1 = $1; $v2 = $2; $t = (($v2 / $v1) * 100); $s = sprintf("%.6f", $t); print "NIC drop percentage: $s\n"; exit; }}'

############################################################################

I also recommend perusing this great informational link which has good tips for system Bios settings, be sure to set your system to maximum performance

Thanks for posting !
A few of questions out of curiosity if ok :
Which Suricata version are you running?
What type of traffic is this (like corporate/university/ISP etc)?
Are you running af_packet regular or the XDP variant?
What kind of a ruleset are you running ?

Hi Peter,

Suricata version is currently 7.0.6
It is monitoring open research university traffic thus the jumbo frames
Regular AF_PACKET
Enabled sources:

  • etnetera/aggressive
  • et/pro
    Total number of enabled rules loaded: 63029

Greg

Awesome - thanks for sharing the info!

Can you share your full suricata.yaml (exclude sensitive data)?
Especially since you did set a rather high MTU it might be interesting to see how you adjusted some values related to that.

The suricata yaml and other tuning aspects are continually updated in the main post as I learn more.

1 Like

Greetings,

I have been doing a lot of work as time permits fine tuning the setup even further and I learn more an more each time. Tuning Suricata is a bit of a mysterious adventure, tweaks you make that seem like they would help sometimes have the opposit effect, I think a lot of this is due to how network traffic ebbs and flows, so I tweak I might make at one point may be during a slower period, and once traffic heats up again I find it might have been an oops; so it is a bit tedious for someone like me that does not understand Suricata like the devs might, but it is all in good fun and I am learning a lot which is always enjoyable. If you are interested in what I am doing I suggest revisiting this page every once in a while.

Thank you for the updates. Please keep it going and share any experience - it is very useful to the community i think.

I believe (knock on wood) I have finally landed on a golden configuration and hopefully have made the final updates to this tips and tricks page, Suricata has been running for 24 hours now and monitoring up to 64Gbps bursts, periods of 30Gbps++ sustained and an average 15Gbps with no issues, and very minimal NIC or capture kernel drop stats seen only at the highest throughput, and most importantly, my checks to alert me when HTTP PCRE matching starts failing have been silent.

Great article! I have a question: why is the IRQ affinity set to CPUs on NUMA node 0, instead of to the same NUMA (node 1) as the NIC?

Hello,

The CPUs on NUMA1 are reserved exclusively for Suricata and if you
noticed are removed from influence by the scheduler as well, using the
CPUs on NUMA0 to handle interrupts helps ensure their exclusivity is
maintained, or at least that is the goal. :slight_smile:

Greg