Suricata misses detecting alerts when receiving high-traffic packets

Hello,
I encountered some issues: Suricata misses detecting alerts with 1 Gbps traffic, while it can detect all alerts at lower traffic (100 Mbps).
Maybe I made a misconfiguration in suricata.yaml. How to improve this?

Testing Environment:

  • Test Machine: CPU 64 cores, Memory 100GB
  • Suricata Version: v7.0.5
  • OS Version: Ubuntu 22.04

Here’s what I observed:

  • Suricata’s CPU usage looks normal, no signs of overload.
  • In the stats.log, I found that there are no packet drops, and the decoder appears to be functioning normally, whether the traffic is 1 Gbps or 100 Mbps.
  • My test data and suricata rules are mainly focused on TCP.
stats.log for 100mbps
capture.kernel_packets                        | Total                     | 424
capture.afpacket.polls                        | Total                     | 306871
capture.afpacket.poll_timeout                 | Total                     | 306837
capture.afpacket.poll_data                    | Total                     | 34
decoder.pkts                                  | Total                     | 423
decoder.bytes                                 | Total                     | 145560
decoder.ipv4                                  | Total                     | 373
decoder.ipv6                                  | Total                     | 11
decoder.ethernet                              | Total                     | 423
decoder.arp                                   | Total                     | 11
decoder.unknown_ethertype                     | Total                     | 28
decoder.tcp                                   | Total                     | 347
tcp.syn                                       | Total                     | 13
tcp.synack                                    | Total                     | 11
decoder.udp                                   | Total                     | 37
decoder.avg_pkt_size                          | Total                     | 344
decoder.max_pkt_size                          | Total                     | 1514
flow.total                                    | Total                     | 31
flow.active                                   | Total                     | 1
flow.tcp                                      | Total                     | 18
flow.udp                                      | Total                     | 13
flow.wrk.spare_sync_avg                       | Total                     | 100
flow.wrk.spare_sync                           | Total                     | 21
flow.wrk.flows_evicted_needs_work             | Total                     | 8
flow.wrk.flows_evicted_pkt_inject             | Total                     | 13
flow.wrk.flows_injected                       | Total                     | 8
tcp.sessions                                  | Total                     | 11
tcp.ssn_from_pool                             | Total                     | 11
tcp.ack_unseen_data                           | Total                     | 6
tcp.segment_from_cache                        | Total                     | 32
tcp.segment_from_pool                         | Total                     | 34
tcp.overlap                                   | Total                     | 4
detect.alert                                  | Total                     | 3
app_layer.flow.http                           | Total                     | 4
app_layer.tx.http                             | Total                     | 4
app_layer.flow.tls                            | Total                     | 7
app_layer.flow.dns_udp                        | Total                     | 2
app_layer.tx.dns_udp                          | Total                     | 4
app_layer.flow.failed_udp                     | Total                     | 11
flow.end.state.new                            | Total                     | 17
flow.end.state.established                    | Total                     | 7
flow.end.state.closed                         | Total                     | 6
flow.end.tcp_state.established                | Total                     | 1
flow.end.tcp_state.close_wait                 | Total                     | 4
flow.end.tcp_state.closed                     | Total                     | 6
flow.mgr.full_hash_pass                       | Total                     | 49
flow.mgr.rows_per_sec                         | Total                     | 26214
flow.spare                                    | Total                     | 20022
flow.mgr.rows_maxlen                          | Total                     | 1
flow.mgr.flows_checked                        | Total                     | 65
flow.mgr.flows_notimeout                      | Total                     | 35
flow.mgr.flows_timeout                        | Total                     | 30
flow.mgr.flows_evicted                        | Total                     | 30
flow.mgr.flows_evicted_needs_work             | Total                     | 8
flow.recycler.recycled                        | Total                     | 22
flow.recycler.queue_max                       | Total                     | 4
tcp.memuse                                    | Total                     | 38797312
tcp.reassembly_memuse                         | Total                     | 7340032
flow.memuse                                   | Total                     | 23318816
stats.log for 1Gbps
capture.kernel_packets                        | Total                     | 422
capture.afpacket.polls                        | Total                     | 307732
capture.afpacket.poll_timeout                 | Total                     | 307702
capture.afpacket.poll_data                    | Total                     | 30
decoder.pkts                                  | Total                     | 422
decoder.bytes                                 | Total                     | 145456
decoder.ipv4                                  | Total                     | 373
decoder.ipv6                                  | Total                     | 10
decoder.ethernet                              | Total                     | 422
decoder.arp                                   | Total                     | 11
decoder.unknown_ethertype                     | Total                     | 28
decoder.tcp                                   | Total                     | 347
tcp.syn                                       | Total                     | 13
tcp.synack                                    | Total                     | 11
decoder.udp                                   | Total                     | 36
decoder.avg_pkt_size                          | Total                     | 344
decoder.max_pkt_size                          | Total                     | 1514
flow.total                                    | Total                     | 33
flow.active                                   | Total                     | 1
flow.tcp                                      | Total                     | 21
flow.udp                                      | Total                     | 12
flow.tcp_reuse                                | Total                     | 3
flow.wrk.spare_sync_avg                       | Total                     | 100
flow.wrk.spare_sync                           | Total                     | 21
flow.wrk.flows_evicted_needs_work             | Total                     | 11
flow.wrk.flows_evicted_pkt_inject             | Total                     | 18
flow.wrk.flows_evicted                        | Total                     | 3
flow.wrk.flows_injected                       | Total                     | 11
tcp.sessions                                  | Total                     | 11
tcp.ssn_from_pool                             | Total                     | 11
tcp.ack_unseen_data                           | Total                     | 19
tcp.segment_from_cache                        | Total                     | 15
tcp.segment_from_pool                         | Total                     | 25
tcp.overlap                                   | Total                     | 2
detect.alert                                  | Total                     | 2
app_layer.flow.http                           | Total                     | 2
app_layer.tx.http                             | Total                     | 2
app_layer.flow.tls                            | Total                     | 3
app_layer.flow.failed_tcp                     | Total                     | 2
app_layer.flow.dns_udp                        | Total                     | 1
app_layer.tx.dns_udp                          | Total                     | 2
app_layer.flow.failed_udp                     | Total                     | 11
flow.end.state.new                            | Total                     | 22
flow.end.state.established                    | Total                     | 7
flow.end.state.closed                         | Total                     | 3
flow.end.tcp_state.syn_sent                   | Total                     | 3
flow.end.tcp_state.established                | Total                     | 1
flow.end.tcp_state.time_wait                  | Total                     | 1
flow.end.tcp_state.close_wait                 | Total                     | 4
flow.end.tcp_state.closed                     | Total                     | 2
flow.mgr.full_hash_pass                       | Total                     | 49
flow.mgr.rows_per_sec                         | Total                     | 26214
flow.spare                                    | Total                     | 20018
flow.mgr.rows_maxlen                          | Total                     | 1
flow.mgr.flows_checked                        | Total                     | 60
flow.mgr.flows_notimeout                      | Total                     | 31
flow.mgr.flows_timeout                        | Total                     | 29
flow.mgr.flows_evicted                        | Total                     | 29
flow.mgr.flows_evicted_needs_work             | Total                     | 11
flow.recycler.recycled                        | Total                     | 18
flow.recycler.queue_max                       | Total                     | 2
tcp.memuse                                    | Total                     | 38797312
tcp.reassembly_memuse                         | Total                     | 7340032
flow.memuse                                   | Total                     | 23318816
af_packet.yaml
%YAML 1.1
---
af-packet:
  - interface: default
    threads: 8
    cluster-id: 10
    cluster-type: cluster_flow
    defrag: yes
    use-mmap: yes
    tpacket-v3: yes
    ring-size: 65535
  - interface: ens7f0
    threads: auto
    cluster-id: 13
    cluster-type: cluster_flow
    defrag: yes
    use-mmap: yes
    tpacket-v3: yes
    ring-size: 65535

Please upgrade first to the latest stable release 7.0.10.

Is the traffic replayed or live traffic?

Yes, this is replayed traffic from same test-pcap.
Additionally, if I directly use the test-pcap for Suricata to read, it can also detect all the alerts (3 alerts).

This is my suricata.yaml (removed some comments due to limitations of upload)

suricata.yaml
%YAML 1.1
---

# Suricata configuration file. In addition to the comments describing all
# options in this file, full documentation can be found at:
# https://docs.suricata.io/en/latest/configuration/suricata-yaml.html

# This configuration file generated by Suricata @PACKAGE_VERSION@.
suricata-version: "@MAJOR_MINOR@"


vars:
  address-groups:
    include: /opt/config/suricata_net.yaml

    HTTP_SERVERS: "$HOME_NET"
    SMTP_SERVERS: "$HOME_NET"
    SQL_SERVERS: "$HOME_NET"
    DNS_SERVERS: "$HOME_NET"
    TELNET_SERVERS: "$HOME_NET"
    AIM_SERVERS: "$EXTERNAL_NET"
    DC_SERVERS: "$HOME_NET"
    DNP3_SERVER: "$HOME_NET"
    DNP3_CLIENT: "$HOME_NET"
    MODBUS_CLIENT: "$HOME_NET"
    MODBUS_SERVER: "$HOME_NET"
    ENIP_CLIENT: "$HOME_NET"
    ENIP_SERVER: "$HOME_NET"

  port-groups:
    HTTP_PORTS: "80"
    SHELLCODE_PORTS: "!80"
    ORACLE_PORTS: 1521
    SSH_PORTS: 22
    DNP3_PORTS: 20000
    MODBUS_PORTS: 502
    FILE_DATA_PORTS: "[$HTTP_PORTS,110,143]"
    FTP_PORTS: 21
    GENEVE_PORTS: 6081
    VXLAN_PORTS: 4789
    TEREDO_PORTS: 3544


default-log-dir: /var/log/suricata/

# Global stats configuration
stats:
  enabled: yes
  interval: 8
  decoder-events: true
  stream-events: false

outputs:
  - fast:
      enabled: yes
      filename: fast.log
      append: yes

  - eve-log:
      enabled: yes
      filetype: regular 
      filename: eve.json
      level: Debug
      pcap-file: false
      community-id: true
      community-id-seed: 0

      xff:
        enabled: no
        mode: extra-data
        deployment: reverse
        header: X-Forwarded-For

      types:
        - alert:
            tagged-packets: yes
        - frame:
            enabled: no
        - anomaly:
            enabled: yes
            types:
               decode: yes
               stream: yes
               applayer: yes
        - http:
            extended: yes   
        - dns:
            enabled: yes
            requests: yes
            responses: yes
        - tls:
            extended: yes
        - files:
            force-magic: no
        - drop:
            alerts: yes     
            flows: all       
        - smtp:
        #- dnp3
        - ftp
        - rdp
        - nfs
        - smb
        - tftp
        - ike
        - dcerpc
        - krb5
        - bittorrent-dht
        - snmp
        - rfb
        - sip
        - quic
        - dhcp:
            enabled: yes
            extended: no
        - ssh
        - mqtt:
        - http2
        - pgsql:
            enabled: no
        - stats:
            totals: yes       # stats for all threads merged together
            threads: no       # per thread stats
            deltas: no        # include delta values
        - flow
        - stream:
            event-set: true
      
  - http-log:
      enabled: no
      filename: http.log
      append: yes
  - tls-log:
      enabled: no  # Log TLS connections.
      filename: tls.log # File to store TLS logs.
      append: yes
  - tls-store:
      enabled: no
  - pcap-log:
      enabled: no
      filename: log.pcap
      limit: 1000mb
      max-files: 2000
      compression: none
      mode: normal 
      use-stream-depth: no 
      honor-pass-rules: no 
  - alert-debug:
      enabled: no
      filename: alert-debug.log
      append: yes
  - stats:
      enabled: yes
      filename: stats.log
      append: yes       
      totals: yes       
  - syslog:
      enabled: no
      facility: local5
  - file-store:
      version: 2
      enabled: no
      xff:
        enabled: no
        mode: extra-data
        deployment: reverse
        header: X-Forwarded-For
  - tcp-data:
      enabled: no
      type: file
      filename: tcp-data.log
  - http-body-data:
      enabled: no
      type: file
      filename: http-data.log
  - lua:
      enabled: no
      scripts:

logging:
  default-log-level: debug 
  default-output-filter:
  outputs:
  - console:
      enabled: yes
  - file:
      enabled: yes
      level: debug
      filename: suricata.log

  - syslog:
      enabled: no
      facility: local5
      format: "[%i] <%d> -- "


# af-packet:
include: /opt/config/suricata_afpacket.yaml

af-xdp:
  - interface: default
dpdk:
  eal-params:
    proc-type: primary

  interfaces:
    - interface: 0000:3b:00.0 # PCIe address of the NIC port
      threads: auto
      
      promisc: true # promiscuous mode - capture all packets
      multicast: true # enables also detection on multicast packets
      checksum-checks: true # if Suricata should validate checksums
      checksum-checks-offload: true # if possible offload checksum validation to the NIC (saves Suricata resources)
      mtu: 1500 # Set MTU of the device in bytes
      mempool-size: 65535 # The number of elements in the mbuf pool
      mempool-cache-size: 257
      rx-descriptors: 1024
      tx-descriptors: 1024
      copy-mode: none
      copy-iface: none # or PCIe address of the second interface

    - interface: default
      threads: auto
      promisc: true
      multicast: true
      checksum-checks: true
      checksum-checks-offload: true
      mtu: 1500
      rss-hash-functions: auto
      mempool-size: 65535
      mempool-cache-size: 257
      rx-descriptors: 1024
      tx-descriptors: 1024
      copy-mode: none
      copy-iface: none

pcap:
  - interface: eth0
  - interface: default

pcap-file:
  checksum-checks: auto

app-layer:
  # error-policy: ignore
  protocols:
    telnet:
      enabled: yes
    rfb:
      enabled: yes
      detection-ports:
        dp: 5900, 5901, 5902, 5903, 5904, 5905, 5906, 5907, 5908, 5909
    mqtt:
      enabled: yes
    krb5:
      enabled: yes
    bittorrent-dht:
      enabled: yes
    snmp:
      enabled: yes
    ike:
      enabled: yes
    tls:
      enabled: yes
      detection-ports:
        dp: 443
      ja3-fingerprints: yes

    pgsql:
      enabled: no
      stream-depth: 0
    dcerpc:
      enabled: yes
    ftp:
      enabled: yes
    rdp:
      enabled: yes
    ssh:
      enabled: yes
    http2:
      enabled: yes
    smtp:
      enabled: yes
      raw-extraction: no
      mime:
        decode-mime: yes
        decode-base64: yes
        decode-quoted-printable: yes
        header-value-depth: 2000
        extract-urls: yes
        body-md5: no
      inspected-tracker:
        content-limit: 100000
        content-inspect-min-size: 32768
        content-inspect-window: 4096
    imap:
      enabled: detection-only
    smb:
      enabled: yes
      detection-ports:
        dp: 139, 445

    nfs:
      enabled: yes
    tftp:
      enabled: yes
    dns:
      tcp:
        enabled: yes
        detection-ports:
          dp: 53
      udp:
        enabled: yes
        detection-ports:
          dp: 53
    http:
      enabled: yes

      libhtp:
         default-config:
           personality: IDS
           request-body-limit: 4mb
             #request-body-limit: 100kb
           response-body-limit: 4mb
             #response-body-limit: 100kb
           request-body-minimal-inspect-size: 32kb
           request-body-inspect-window: 4kb
           response-body-minimal-inspect-size: 40kb
           response-body-inspect-window: 16kb
           response-body-decompress-layer-limit: 2
           http-body-inline: auto
           swf-decompression:
             enabled: yes
             type: both
             compress-depth: 100kb
             decompress-depth: 100kb
           double-decode-path: no
           double-decode-query: no
         server-config:
    modbus:
      enabled: yes
      detection-ports:
        dp: 502
      stream-depth: 0

    dnp3:
      enabled: yes
      detection-ports:
        dp: 20000

    enip:
      enabled: yes
      detection-ports:
        dp: 44818
        sp: 44818

    ntp:
      enabled: yes

    quic:
      enabled: yes

    dhcp:
      enabled: yes

    sip:
      enabled: yes

asn1-max-frames: 256


datasets:
  defaults:
  rules:

security:
  limit-noproc: true
  landlock:
    enabled: no
  lua:

coredump:
  max-dump: unlimited

host-mode: sniffer-only

max-pending-packets: 65534


runmode: workers

unix-command:
  enabled: on
  filename: /var/run/suricata/suricata-command.socket

legacy:
  uricontent: enabled

exception-policy: auto

engine-analysis:
  rules-fast-pattern: yes
  rules: yes

pcre:
  match-limit: 3500
  match-limit-recursion: 1500

host-os-policy:
  windows: [0.0.0.0/0]
  bsd: []
  bsd-right: []
  old-linux: []
  linux: []
  old-solaris: []
  solaris: []
  hpux10: []
  hpux11: []
  irix: []
  macos: []
  vista: []
  windows2k3: []

defrag:
  memcap: 2gb
  hash-size: 65536
  trackers: 65535 # number of defragmented flows to follow
  max-frags: 65535 # number of fragments to keep (higher than trackers)
  prealloc: yes
  timeout: 60

flow:
  memcap: 4gb
  hash-size: 262144
  prealloc: 20000
  emergency-recovery: 60

vlan:
  use-for-tracking: true

livedev:
  use-for-tracking: false

flow-timeouts:

  default:
    new: 30
    established: 300
    closed: 0
    bypassed: 100
    emergency-new: 10
    emergency-established: 100
    emergency-closed: 0
    emergency-bypassed: 50
  tcp:
    new: 60
    established: 600
    closed: 60
    bypassed: 100
    emergency-new: 5
    emergency-established: 100
    emergency-closed: 10
    emergency-bypassed: 50
  udp:
    new: 30
    established: 300
    bypassed: 100
    emergency-new: 10
    emergency-established: 100
    emergency-bypassed: 50
  icmp:
    new: 30
    established: 300
    bypassed: 100
    emergency-new: 10
    emergency-established: 100
    emergency-bypassed: 50

stream:
  memcap: 4gb
  checksum-validation: no      # reject incorrect csums
  inline: auto                  # auto will use inline mode in IPS mode, yes or no set it statically
  reassembly:
    memcap: 2gb
    depth: 5mb                  # reassemble 1mb into a stream
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560
    randomize-chunk-size: yes

host:
  hash-size: 4096
  prealloc: 1000
  memcap: 512mb

decoder:

  teredo:
    enabled: true
    ports: $TEREDO_PORTS # syntax: '[3544, 1234]' or '3533' or 'any'.

  vxlan:
    enabled: true
    ports: $VXLAN_PORTS # syntax: '[8472, 4789]' or '4789'.

  geneve:
    enabled: true
    ports: $GENEVE_PORTS # syntax: '[6081, 1234]' or '6081'.

detect:
  profile: high
  custom-values:
    toclient-groups: 3
    toserver-groups: 25
  sgh-mpm-context: auto
  inspection-recursion-limit: 6000
  delayed-detect: no

  prefilter:
    default: mpm

  grouping:

  profiling:
    grouping:
      dump-to-disk: false
      include-rules: false      # very verbose
      include-mpm-stats: false

mpm-algo: auto
spm-algo: auto
threading:
  set-cpu-affinity: no
  cpu-affinity:
    - management-cpu-set:
        cpu: [ 0 ]  # include only these CPUs in affinity settings
    - receive-cpu-set:
        cpu: [ 0 ]  # include only these CPUs in affinity settings
    - worker-cpu-set:
        cpu: [ "all" ]
        mode: "exclusive"
        prio:
          low: [ 0 ]
          medium: [ 0 ]
          high: [ 3 ]
          default: "medium"

  detect-thread-ratio: 1.0
  stack-size: 8mb

luajit:
  states: 128

profiling:
  rules:
    enabled: yes
    filename: rule_perf.log
    append: yes
    limit: 10
    json: yes

  keywords:
    enabled: yes
    filename: keyword_perf.log
    append: yes

  prefilter:
    enabled: yes
    filename: prefilter_perf.log
    append: yes

  rulegroups:
    enabled: yes
    filename: rule_group_perf.log
    append: yes

  packets:

    enabled: yes
    filename: packet_stats.log
    append: yes

    csv:
      enabled: no
      filename: packet_stats.csv

  locks:
    enabled: no
    filename: lock_stats.log
    append: yes

  pcap-log:
    enabled: no
    filename: pcaplog_stats.log
    append: yes


nfq:
nflog:
  - group: 2
    buffer-size: 18432
  - group: default
    qthreshold: 1
    qtimeout: 100
    max-size: 20000

capture:
netmap:
 - interface: default

pfring:
  - interface: default
    checksum-checks: no
    cluster-id: 10
    cluster-type: cluster_flow
    bypass: yes

  - interface: default

ipfw:

napatech:
  
    streams: ["0-3"]
    enable-stream-stats: no
    auto-config: yes
    hardware-bypass: yes
    inline: no
    ports: [0-1,2-3]
    hashmode: hash5tuplesorted



default-rule-path: /var/suricata/idsrules/
rule-files:
 - final1.rules


classification-file: /etc/suricata/classification.config
reference-config-file: /etc/suricata/reference.config

How do you replay exactly?
What are the exact diffs between the two runs?

@Andreas_Herz
I use tcpreplay-edit to send the test-pcap with different traffic.
tcpreplay-edit --mtu-trunc --mbps=100(1000) -i ens4f0 test.pcap
And wait for 10 minutes (some detections only appear after a timeout) to observe, and for each round of testing, I restart Suricata to keep it clean.

I think it is related to app_layer.flow.failed_tcp, but I’m not sure how it happens.
UPDATE: Reviewing the eve-log, it was found that in high traffic scenarios, some sessions may be missing packets, which could cause the TCP 3whs to fail.