Kernel_drops larger than kernel_packets?

According to the suricata stats log, the kernel_drops count is larger than the kernel_packets count, which is very confusing:

# tail -n 1 /opt/isd/data/suricata/stats.json | python3 -c 'import sys,json; print(json.load(sys.stdin)["stats"]["capture"])'
{'kernel_packets': 121843217, 'kernel_drops': 207464408, 'bypassed': 0}

Admittedly the machine is likely underpowered for the amount of traffic it receives (we’re in the process of acquiring a more powerful device) but these numbers don’t make sense. Does anyone have ideas of what could be causing this?

We are using pf_ring and listening to a teamed interface. We normally see about 2.5 - 4gbps traffic through this interface.

OS: Rocky Linux

Suricata build info:

# suricata --build-info
This is Suricata version 6.0.5 RELEASE
Features: PCAP_SET_BUFF PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON PROFILING TLS TLS_C11 MAGIC RUST
SIMD support: SSE_4_2 SSE_4_1 SSE_3
Atomic intrinsics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 8.5.0 20210514 (Red Hat 8.5.0-4), C version 201112
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: _Thread_local
compiled with LibHTP v0.5.40, linked against LibHTP v0.5.40

Suricata Configuration:
  AF_PACKET support:                       yes
  eBPF support:                            no
  XDP support:                             no
  PF_RING support:                         yes
  NFQueue support:                         no
  NFLOG support:                           no
  IPFW support:                            no
  Netmap support:                          no
  DAG enabled:                             no
  Napatech enabled:                        no
  WinDivert enabled:                       no

  Unix socket enabled:                     yes
  Detection enabled:                       yes

  Libmagic support:                        yes
  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  hiredis support:                         no
  hiredis async with libevent:             no
  Prelude support:                         no
  PCRE jit:                                yes
  LUA support:                             no
  libluajit:                               no
  GeoIP2 support:                          no
  Non-bundled htp:                         no
  Hyperscan support:                       no
  Libnet support:                          no
  liblz4 support:                          yes
  HTTP2 decompression:                     no

  Rust support:                            yes
  Rust strict mode:                        no
  Rust compiler path:                      /usr/bin/rustc
  Rust compiler version:                   rustc 1.54.0 (Red Hat 1.54.0-3.module+el8.5.0+13074+d655d86c)
  Cargo path:                              /usr/bin/cargo
  Cargo version:                           cargo 1.54.0
  Cargo vendor:                            yes

  Python support:                          yes
  Python path:                             /usr/bin/python3
  Python distutils                         yes
  Python yaml                              yes
  Install suricatactl:                     yes
  Install suricatasc:                      yes
  Install suricata-update:                 not bundled

  Profiling enabled:                       yes
  Profiling locks enabled:                 no

  Plugin support (experimental):           yes

Development settings:
  Coccinelle / spatch:                     no
  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no

Generic build parameters:
  Installation prefix:                     /usr
  Configuration directory:                 /etc/suricata/
  Log directory:                           /var/log/suricata/

  --prefix                                 /usr
  --sysconfdir                             /etc
  --localstatedir                          /var
  --datarootdir                            /usr/share

  Host:                                    x86_64-pc-linux-gnu
  Compiler:                                gcc (exec name) / gcc (real)
  GCC Protect enabled:                     no
  GCC march native enabled:                yes
  GCC Profile enabled:                     no
  Position Independent Executable enabled: no
  CFLAGS                                   -D_GNU_SOURCE -std=c11 -march=native -I${srcdir}/../rust/gen -I${srcdir}/../rust/dist
  PCAP_CFLAGS
  SECCFLAGS

lscpu:

# lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              12
On-line CPU(s) list: 0-11
Thread(s) per core:  1
Core(s) per socket:  6
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
BIOS Vendor ID:      Intel
CPU family:          6
Model:               85
Model name:          Intel(R) Xeon(R) Bronze 3204 CPU @ 1.90GHz
BIOS Model name:     Intel(R) Xeon(R) Bronze 3204 CPU @ 1.90GHz
Stepping:            7
CPU MHz:             1900.000
BogoMIPS:            3800.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            8448K
NUMA node0 CPU(s):   0,2,4,6,8,10
NUMA node1 CPU(s):   1,3,5,7,9,11
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm arat pln pts pku ospke avx512_vnni md_clear flush_l1d arch_capabilities

And I’ve attached the suricata.yaml file (I’ve ommitted the address and port groups - rest assured in the actual configuration those are present)
suricata.yaml (10.4 KB)

Let me know if I can provide any other information that could help. Thanks so much!

Hi. The packet counts make sense because total packets = kernel_packets + kernel_drops.
kernel_packets are given to Suricata and kernel_drops are dropped.
Do you have more drops in your stats in addition to kernel_drops? A complete stats log would be nice.
I agree that the machine is underspecced CPU wise, but the drop count still seems very high.
Are you bonding a mirror TAP with TX and RX on one interface each using PR_RING?
If you look at htop during load, how is the kernel space vs user space CPU usage?

Here is a complete stats.json entry. I’ll try to get you the other requested information and will follow up.

{
  "timestamp": "2022-05-25T15:28:25.351199+0000",
  "event_type": "stats",
  "stats": {
    "uptime": 52242,
    "capture": {
      "kernel_packets": 5254241490,
      "kernel_drops": 8210580877,
      "bypassed": 0
    },
    "decoder": {
      "pkts": 5254323710,
      "bytes": 4636824074443,
      "invalid": 38,
      "ipv4": 5257398002,
      "ipv6": 14,
      "ethernet": 5254323758,
      "chdlc": 0,
      "raw": 0,
      "null": 0,
      "sll": 0,
      "tcp": 4648903012,
      "udp": 204663247,
      "sctp": 0,
      "icmpv4": 2713222,
      "icmpv6": 14,
      "ppp": 0,
      "pppoe": 0,
      "geneve": 0,
      "gre": 48,
      "vlan": 5254323696,
      "vlan_qinq": 1460084293,
      "vxlan": 16,
      "vntag": 0,
      "ieee8021ah": 0,
      "teredo": 0,
      "ipv4_in_ipv6": 0,
      "ipv6_in_ipv6": 0,
      "mpls": 0,
      "avg_pkt_size": 882,
      "max_pkt_size": 1518,
      "max_mac_addrs_src": 0,
      "max_mac_addrs_dst": 0,
      "erspan": 0,
      "event": {
        "ipv4": {
          "pkt_too_small": 0,
          "hlen_too_small": 0,
          "iplen_smaller_than_hlen": 0,
          "trunc_pkt": 0,
          "opt_invalid": 0,
          "opt_invalid_len": 0,
          "opt_malformed": 0,
          "opt_pad_required": 0,
          "opt_eol_required": 0,
          "opt_duplicate": 0,
          "opt_unknown": 0,
          "wrong_ip_version": 0,
          "icmpv6": 0,
          "frag_pkt_too_large": 0,
          "frag_overlap": 13159,
          "frag_ignored": 0
        },
        "icmpv4": {
          "pkt_too_small": 0,
          "unknown_type": 0,
          "unknown_code": 575726,
          "ipv4_trunc_pkt": 0,
          "ipv4_unknown_ver": 0
        },
        "icmpv6": {
          "unknown_type": 0,
          "unknown_code": 0,
          "pkt_too_small": 0,
          "ipv6_unknown_version": 0,
          "ipv6_trunc_pkt": 0,
          "mld_message_with_invalid_hl": 0,
          "unassigned_type": 0,
          "experimentation_type": 0
        },
        "ipv6": {
          "pkt_too_small": 0,
          "trunc_pkt": 0,
          "trunc_exthdr": 0,
          "exthdr_dupl_fh": 0,
          "exthdr_useless_fh": 0,
          "exthdr_dupl_rh": 0,
          "exthdr_dupl_hh": 0,
          "exthdr_dupl_dh": 0,
          "exthdr_dupl_ah": 0,
          "exthdr_dupl_eh": 0,
          "exthdr_invalid_optlen": 0,
          "wrong_ip_version": 0,
          "exthdr_ah_res_not_null": 0,
          "hopopts_unknown_opt": 0,
          "hopopts_only_padding": 0,
          "dstopts_unknown_opt": 0,
          "dstopts_only_padding": 0,
          "rh_type_0": 0,
          "zero_len_padn": 0,
          "fh_non_zero_reserved_field": 0,
          "data_after_none_header": 0,
          "unknown_next_header": 0,
          "icmpv4": 0,
          "frag_pkt_too_large": 0,
          "frag_overlap": 0,
          "frag_invalid_length": 0,
          "frag_ignored": 0,
          "ipv4_in_ipv6_too_small": 0,
          "ipv4_in_ipv6_wrong_version": 0,
          "ipv6_in_ipv6_too_small": 0,
          "ipv6_in_ipv6_wrong_version": 0
        },
        "tcp": {
          "pkt_too_small": 0,
          "hlen_too_small": 0,
          "invalid_optlen": 0,
          "opt_invalid_len": 1033,
          "opt_duplicate": 0
        },
        "udp": {
          "pkt_too_small": 5,
          "hlen_too_small": 0,
          "hlen_invalid": 3
        },
        "sll": {
          "pkt_too_small": 0
        },
        "ethernet": {
          "pkt_too_small": 0
        },
        "ppp": {
          "pkt_too_small": 0,
          "vju_pkt_too_small": 0,
          "ip4_pkt_too_small": 0,
          "ip6_pkt_too_small": 0,
          "wrong_type": 0,
          "unsup_proto": 0
        },
        "pppoe": {
          "pkt_too_small": 0,
          "wrong_code": 0,
          "malformed_tags": 0
        },
        "gre": {
          "pkt_too_small": 0,
          "wrong_version": 0,
          "version0_recur": 0,
          "version0_flags": 0,
          "version0_hdr_too_big": 0,
          "version0_malformed_sre_hdr": 0,
          "version1_chksum": 0,
          "version1_route": 0,
          "version1_ssr": 0,
          "version1_recur": 0,
          "version1_flags": 0,
          "version1_no_key": 0,
          "version1_wrong_protocol": 0,
          "version1_malformed_sre_hdr": 0,
          "version1_hdr_too_big": 0
        },
        "vlan": {
          "header_too_small": 0,
          "unknown_type": 0,
          "too_many_layers": 0
        },
        "ieee8021ah": {
          "header_too_small": 0
        },
        "vntag": {
          "header_too_small": 0,
          "unknown_type": 0
        },
        "ipraw": {
          "invalid_ip_version": 0
        },
        "ltnull": {
          "pkt_too_small": 0,
          "unsupported_type": 0
        },
        "sctp": {
          "pkt_too_small": 0
        },
        "mpls": {
          "header_too_small": 0,
          "pkt_too_small": 0,
          "bad_label_router_alert": 0,
          "bad_label_implicit_null": 0,
          "bad_label_reserved": 0,
          "unknown_payload_type": 0
        },
        "vxlan": {
          "unknown_payload_type": 16
        },
        "geneve": {
          "unknown_payload_type": 0
        },
        "erspan": {
          "header_too_small": 0,
          "unsupported_version": 0,
          "too_many_vlan_layers": 0
        },
        "dce": {
          "pkt_too_small": 0
        },
        "chdlc": {
          "pkt_too_small": 0
        }
      },
      "too_many_layers": 0
    },
    "flow": {
      "memcap": 0,
      "tcp": 30968960,
      "udp": 18716494,
      "icmpv4": 184379,
      "icmpv6": 14,
      "tcp_reuse": 91877,
      "get_used": 0,
      "get_used_eval": 0,
      "get_used_eval_reject": 0,
      "get_used_eval_busy": 0,
      "get_used_failed": 0,
      "wrk": {
        "spare_sync_avg": 100,
        "spare_sync": 240130,
        "spare_sync_incomplete": 0,
        "spare_sync_empty": 0,
        "flows_evicted_needs_work": 5219595,
        "flows_evicted_pkt_inject": 10029435,
        "flows_evicted": 21651841,
        "flows_injected": 4217709
      },
      "mgr": {
        "full_hash_pass": 218,
        "closed_pruned": 0,
        "new_pruned": 0,
        "est_pruned": 0,
        "bypassed_pruned": 0,
        "rows_maxlen": 20,
        "flows_checked": 50616928,
        "flows_notimeout": 40154955,
        "flows_timeout": 10461973,
        "flows_timeout_inuse": 0,
        "flows_evicted": 27791655,
        "flows_evicted_needs_work": 4217709
      },
      "spare": 18826,
      "emerg_mode_entered": 0,
      "emerg_mode_over": 0,
      "memuse": 146636224
    },
    "defrag": {
      "ipv4": {
        "fragments": 8029648,
        "reassembled": 3074837,
        "timeouts": 0
      },
      "ipv6": {
        "fragments": 0,
        "reassembled": 0,
        "timeouts": 0
      },
      "max_frag_hits": 0
    },
    "flow_bypassed": {
      "local_pkts": 0,
      "local_bytes": 0,
      "local_capture_pkts": 0,
      "local_capture_bytes": 0,
      "closed": 0,
      "pkts": 0,
      "bytes": 0
    },
    "tcp": {
      "sessions": 20201198,
      "ssn_memcap_drop": 0,
      "pseudo": 458,
      "pseudo_failed": 0,
      "invalid_checksum": 0,
      "no_flow": 0,
      "syn": 22895620,
      "synack": 22094798,
      "rst": 16756798,
      "midstream_pickups": 0,
      "pkt_on_wrong_thread": 0,
      "segment_memcap_drop": 287286385,
      "stream_depth_reached": 147786,
      "reassembly_gap": 124898565,
      "overlap": 115097,
      "overlap_diff_data": 0,
      "insert_data_normal_fail": 282748724,
      "insert_data_overlap_fail": 310,
      "insert_list_fail": 0,
      "memuse": 52735496,
      "reassembly_memuse": 268434356
    },
    "detect": {
      "engines": [
        {
          "id": 0,
          "last_reload": "2022-05-25T00:58:23.695140+0000",
          "rules_loaded": 32365,
          "rules_failed": 1
        }
      ],
      "alert": 1210023,
      "mpm_list": 3,
      "nonmpm_list": 201,
      "fnonmpm_list": 129,
      "match_list": 132
    },
    "app_layer": {
      "flow": {
        "http": 10651,
        "ftp": 4,
        "smtp": 454,
        "tls": 39598,
        "ssh": 804,
        "imap": 0,
        "smb": 2,
        "dcerpc_tcp": 9,
        "dns_tcp": 3026,
        "modbus": 0,
        "enip": 0,
        "dnp3": 0,
        "nfs_tcp": 0,
        "ntp": 143764,
        "ftp-data": 1,
        "tftp": 95,
        "ikev2": 1380,
        "krb5_tcp": 0,
        "dhcp": 3419,
        "snmp": 122753,
        "sip": 5547,
        "rfb": 0,
        "mqtt": 0,
        "rdp": 0,
        "http2": 1,
        "failed_tcp": 22601,
        "dcerpc_udp": 5,
        "dns_udp": 14827082,
        "nfs_udp": 0,
        "krb5_udp": 0,
        "failed_udp": 3612449
      },
      "tx": {
        "http": 53996,
        "ftp": 40,
        "smtp": 1293,
        "tls": 0,
        "ssh": 0,
        "imap": 0,
        "smb": 16,
        "dcerpc_tcp": 185,
        "dns_tcp": 8510,
        "modbus": 0,
        "enip": 0,
        "dnp3": 0,
        "nfs_tcp": 0,
        "ntp": 197375,
        "ftp-data": 0,
        "tftp": 93,
        "ikev2": 5401,
        "krb5_tcp": 0,
        "dhcp": 206597,
        "snmp": 1813350,
        "sip": 9307,
        "rfb": 0,
        "mqtt": 0,
        "rdp": 0,
        "http2": 8,
        "dcerpc_udp": 3,
        "dns_udp": 31772369,
        "nfs_udp": 0,
        "krb5_udp": 0
      },
      "expectations": 0
    },
    "http": {
      "memuse": 213550,
      "memcap": 0
    },
    "ftp": {
      "memuse": 192,
      "memcap": 0
    },
    "file_store": {
      "open_files": 0
    }
  }
}

I have never seen this with capture methods like af_packet. I don’t know about pf_ring team interfaces, but can you try a non-team interface?
It could be a bug or config issue related to pf_ring.

We’ve migrated to a new machine, and are seeing better drop rates, although I am pretty certain it should be even lower given the fact we’ve allocated more (14) CPUs and these are stronger (seeing ~10% kernel drops, I feel like it should be below 5%).

And just as an FYI for anyone who might stumble upon this thread - I did not see any improvement by bypassing the network team. I’m not convinced that using a network team was the bottleneck.

We use pf_ring but I believe we simply use the default configurations.

I encountered the same problem. Do you make sense of why it happens now?