Suricata 7 with multiple NICs and cpu affinity -- af-xdp: Failed to create socket

scloder · July 27, 2023, 4:56pm

Hello,

I am running into an error in a system with 96 cores and two intel XL710 40Gbps NICs (using Ubuntu 22.04.2 native drivers) when trying to use af-xdp capture mode. I have build info below, as well as cpu affinity and af-xdp conf. I have used set_irq_affinity.sh to set affinity per NUMA node and setup the RSS hash key and other ethtool params per the suricata readthedocs. Using two managers and two recycler cores as well.

Here is where the error occurs:

[2800 - W#45-enp216s0f0] 2023-07-27 11:10:51 Perf: threads: Setting prio -2 for thread "W#45-enp216s0f0" to cpu/core 48, thread id 2800
[2803 - W#46-enp216s0f0] 2023-07-27 11:10:52 Perf: threads: Setting prio -2 for thread "W#46-enp216s0f0" to cpu/core 49, thread id 2803
[2656 - Suricata-Main] 2023-07-27 11:10:52 Info: ioctl: enp59s0f0: RX RSS queues: 46
[2656 - Suricata-Main] 2023-07-27 11:10:52 Info: runmodes: enp59s0f0: creating 46 threads
[2806 - W#01-enp59s0f0] 2023-07-27 11:10:52 Perf: threads: Setting prio -2 for thread "W#01-enp59s0f0" to cpu/core 50, thread id 2806
[2806 - W#01-enp59s0f0] 2023-07-27 11:10:52 Error: af-xdp: Failed to create socket: Invalid argument
[2807 - W#02-enp59s0f0] 2023-07-27 11:10:52 Perf: threads: Setting prio -2 for thread "W#02-enp59s0f0" to cpu/core 51, thread id 2807

build-info:

This is Suricata version 7.0.0 RELEASE
Features: PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HTTP2_DECOMPRESSION HAVE_LUA HAVE_LUAJIT HAVE_LIBJANSSON TLS TLS_C11 MAGIC RUST 
SIMD support: SSE_4_2 SSE_4_1 SSE_3 
Atomic intrinsics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version Ubuntu Clang 14.0.0, C version 201112
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: _Thread_local
compiled with LibHTP v0.5.45, linked against LibHTP v0.5.45

Suricata Configuration:
  AF_PACKET support:                       yes
  AF_XDP support:                          yes
  DPDK support:                            no
  eBPF support:                            yes
  XDP support:                             yes
  PF_RING support:                         no
  NFQueue support:                         no
  NFLOG support:                           no
  IPFW support:                            no
  Netmap support:                          no 
  DAG enabled:                             no
  Napatech enabled:                        no
  WinDivert enabled:                       no

  Unix socket enabled:                     yes
  Detection enabled:                       yes

  Libmagic support:                        yes
  libjansson support:                      yes
  hiredis support:                         no
  hiredis async with libevent:             no
  PCRE jit:                                yes
  LUA support:                             yes, through luajit
  libluajit:                               yes
  GeoIP2 support:                          yes
  Non-bundled htp:                         no
  Hyperscan support:                       yes
  Libnet support:                          yes
  liblz4 support:                          no
  Landlock support:                        yes

  Rust support:                            yes
  Rust strict mode:                        yes
  Rust compiler path:                      /usr/bin/rustc
  Rust compiler version:                   rustc 1.66.1 (90743e729 2023-01-10) (built from a source tarball)
  Cargo path:                              /usr/bin/cargo
  Cargo version:                           cargo 1.66.1

  Python support:                          yes
  Python path:                             /usr/bin/python3
  Install suricatactl:                     yes
  Install suricatasc:                      yes
  Install suricata-update:                 yes

  Profiling enabled:                       no
  Profiling locks enabled:                 no
  Profiling rules enabled:                 no

  Plugin support (experimental):           yes
  DPDK Bond PMD:                           no

Development settings:
  Coccinelle / spatch:                     no
  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no
  Fuzz targets enabled:                    no

Generic build parameters:
  Installation prefix:                     /usr
  Configuration directory:                 /etc/suricata/
  Log directory:                           /var/log/suricata/

  --prefix                                 /usr
  --sysconfdir                             /etc
  --localstatedir                          /var
  --datarootdir                            /usr/share

  Host:                                    x86_64-pc-linux-gnu
  Compiler:                                clang (exec name) / g++ (real)
  GCC Protect enabled:                     no
  GCC march native enabled:                yes
  GCC Profile enabled:                     no
  Position Independent Executable enabled: no
  CFLAGS                                   -g -O2 -fPIC -std=c11 -march=native -I${srcdir}/../rust/gen -I${srcdir}/../rust/dist
  PCAP_CFLAGS                               -I/usr/include
  SECCFLAGS

cpu and NIC config:

  cpu-affinity:
    - management-cpu-set:
        cpu: [0,1,2,3 ]  # include only these CPUs in affinity settings
    - receive-cpu-set:
        cpu: [ 0 ]  # include only these CPUs in affinity settings
    - worker-cpu-set:
        cpu: [ "4-95" ]
        mode: "exclusive"
        # Use explicitly 3 threads and don't compute number by using
        # detect-thread-ratio variable:
        # threads: 3
        prio:
          low: [ 0 ]
          medium: [ 0,1,2,3 ]
          high: [ "4-95" ]
          default: "high"

af-xdp:
  - interface: enp216s0f0
    threads: 46
    force-xdp-mode: drv
  - interface: enp59s0f0
    threads: 46
    force-xdp-mode: drv

Andreas_Herz · July 31, 2023, 7:58pm

Can you post the whole suricata.yaml (feel free to remove sensitive infos) and als the full suricata.log alongside the start command?

scloder · August 1, 2023, 2:00pm

It seems there is a character limit, so I can upload the yaml, hopefully that works for you.
suricata_af_xdp.yaml (83.5 KB)

Here is the start line and the suricata.log:
start line:

/usr/bin/suricata -c /home/user/suricata_resources/suricata_af_xdp.yaml --af-xdp -D -vvvv

suricata.log:

Notice: suricata: This is Suricata version 7.0.0 RELEASE running in SYSTEM mode [LogVersion:suricata.c:1154]
Info: cpu: CPUs/cores online: 96 [UtilCpuPrintSummary:util-cpu.c:182]
Config: device: Adding interface enp216s0f0 from config file [LiveBuildDeviceListCustom:util-device.c:295]
Config: device: Adding interface enp59s0f0 from config file [LiveBuildDeviceListCustom:util-device.c:295]
Config: luajit: luajit states preallocated: 128 [LuajitSetupStatesPool:util-luajit.c:99]
Info: suricata: Setting engine mode to IDS mode by default [PostConfLoadedSetup:suricata.c:2698]
Info: exception-policy: master exception-policy set to: auto [ExceptionPolicyMasterParse:util-exception-policy.c:200]
Config: exception-policy: app-layer.error-policy: ignore (defined via 'exception-policy' master switch) [ExceptionPolicyGetDefault:util-exception-policy.c:220]
Config: app-layer-htp: 'default' server has 'request-body-minimal-inspect-size' set to 31964 and 'request-body-inspect-window' set to 4119 after randomization. [HTPConfigSetDefaultsPhase2:app-layer-htp.c:2567]
Config: app-layer-htp: 'default' server has 'response-body-minimal-inspect-size' set to 39242 and 'response-body-inspect-window' set to 17014 after randomization. [HTPConfigSetDefaultsPhase2:app-layer-htp.c:2580]
Config: smb: read: max record size: 16777216, max queued chunks 64, max queued size 67108864 [suricata::smb::smb::rs_smb_register_parser:smb.rs:2428]
Config: smb: write: max record size: 16777216, max queued chunks 64, max queued size 67108864 [suricata::smb::smb::rs_smb_register_parser:smb.rs:2430]
Config: app-layer-enip: Protocol detection and parser disabled for enip protocol. [RegisterENIPUDPParsers:app-layer-enip.c:539]
Config: app-layer-dnp3: Protocol detection and parser disabled for DNP3. [RegisterDNP3Parsers:app-layer-dnp3.c:1565]
Info: ioctl: enp216s0f0: MTU 1500 [GetIfaceMTU:util-ioctl.c:110]
Info: ioctl: enp59s0f0: MTU 1500 [GetIfaceMTU:util-ioctl.c:110]
[106633 - Suricata-Main] 2023-08-01 08:50:52 Info: ioctl: enp59s0f0: MTU 1500
[106634 - Suricata-Main] 2023-08-01 08:50:52 Config: host: allocated 262144 bytes of memory for the host hash... 4096 buckets of size 64
[106634 - Suricata-Main] 2023-08-01 08:50:52 Config: host: preallocated 1000 hosts of size 136
[106634 - Suricata-Main] 2023-08-01 08:50:52 Config: host: host memory usage: 398144 bytes, maximum: 33554432
[106634 - Suricata-Main] 2023-08-01 08:50:52 Config: coredump-config: Core dump size set to unlimited.
[106634 - Suricata-Main] 2023-08-01 08:50:52 Config: exception-policy: defrag.memcap-policy: ignore (defined via 'exception-policy' master switch)
[106634 - Suricata-Main] 2023-08-01 08:50:52 Config: defrag-hash: allocated 3670016 bytes of memory for the defrag hash... 65536 buckets of size 56
[106634 - Suricata-Main] 2023-08-01 08:50:52 Config: defrag-hash: preallocated 65535 defrag trackers of size 160
[106634 - Suricata-Main] 2023-08-01 08:50:52 Config: defrag-hash: defrag memory usage: 14155616 bytes, maximum: 4294967296
[106634 - Suricata-Main] 2023-08-01 08:50:52 Config: exception-policy: flow.memcap-policy: ignore (defined via 'exception-policy' master switch)
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: flow: flow size 296, memcap allows for 79805135 flows. Per hash row in perfect conditions 9
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: stream-tcp: stream "prealloc-sessions": 200000 (per thread)
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: stream-tcp: stream "memcap": 12884901888
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: stream-tcp: stream "midstream" session pickups: disabled
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: stream-tcp: stream "async-oneside": disabled
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: stream-tcp: stream "checksum-validation": disabled
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: exception-policy: stream.memcap-policy: ignore (defined via 'exception-policy' master switch)
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: exception-policy: stream.reassembly.memcap-policy: ignore (defined via 'exception-policy' master switch)
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: exception-policy: stream.midstream-policy: ignore (defined via 'exception-policy' master switch)
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: stream-tcp: stream."inline": disabled
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: stream-tcp: stream "bypass": enabled
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: stream-tcp: stream "max-syn-queued": 10
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: stream-tcp: stream "max-synack-queued": 5
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: stream-tcp: stream.reassembly "memcap": 25769803776
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: stream-tcp: stream.reassembly "depth": 1048576
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: stream-tcp: stream.reassembly "toserver-chunk-size": 2650
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: stream-tcp: stream.reassembly "toclient-chunk-size": 2523
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: stream-tcp: stream.reassembly.raw: enabled
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: stream-tcp: stream.liberal-timestamps: disabled
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: stream-tcp-reassemble: stream.reassembly "segment-prealloc": 204800
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: stream-tcp-reassemble: stream.reassembly "max-regions": 8
[106634 - Suricata-Main] 2023-08-01 08:50:56 Info: conf: Running in live mode, activating unix socket
[106634 - Suricata-Main] 2023-08-01 08:50:56 Info: logopenfile: eve-log output device (regular) initialized: eve-%Y-%m-%d-%H:%M.json
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: output-json: Enabling eve community_id logging.
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'alert'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'frame'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'anomaly'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'http'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'dns'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'tls'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'files'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'smtp'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'ftp'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'rdp'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'nfs'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'smb'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'tftp'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'ike'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'dcerpc'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'krb5'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'bittorrent-dht'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'rfb'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'sip'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'dhcp'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'ssh'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'mqtt'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'http2'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'pgsql'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: runmodes: enabling 'eve-log' module 'stats'
[106634 - Suricata-Main] 2023-08-01 08:50:56 Info: logopenfile: stats output device (regular) initialized: stats.log
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: landlock: Landlock is not enabled in configuration
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: suricata: Delayed detect disabled
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: detect: pattern matchers: MPM: hs, SPM: hs
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: detect: grouping: tcp-whitelist (default) 53, 80, 139, 443, 445, 1433, 3306, 3389, 6666, 6667, 8080
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: detect: grouping: udp-whitelist (default) 53, 135, 5060
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: detect: prefilter engines: MPM
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: reputation: IP reputation disabled
[106634 - Suricata-Main] 2023-08-01 08:50:56 Config: detect: Loading rule file: /var/lib/suricata/rules/suricata.rules
[106634 - Suricata-Main] 2023-08-01 08:51:12 Info: detect: 1 rule files processed. 60792 rules successfully loaded, 0 rules failed
[106634 - Suricata-Main] 2023-08-01 08:51:12 Info: threshold-config: Threshold config parsed: 0 rule(s) found
[106634 - Suricata-Main] 2023-08-01 08:51:12 Info: detect: 60795 signatures processed. 1296 are IP-only rules, 10500 are inspecting packet payload, 48859 inspect application layer, 102 are decoder event only
[106634 - Suricata-Main] 2023-08-01 08:51:12 Config: detect: building signature grouping structure, stage 1: preprocessing rules... complete
[106634 - Suricata-Main] 2023-08-01 08:51:13 Warning: detect-flowbits: flowbit 'smb.trans2.mid66' is checked but not set. Checked in 42330 and 0 other sigs
[106634 - Suricata-Main] 2023-08-01 08:51:13 Perf: detect: TCP toserver: 41 port groups, 37 unique SGH's, 4 copies
[106634 - Suricata-Main] 2023-08-01 08:51:13 Perf: detect: TCP toclient: 21 port groups, 19 unique SGH's, 2 copies
[106634 - Suricata-Main] 2023-08-01 08:51:13 Perf: detect: UDP toserver: 41 port groups, 36 unique SGH's, 5 copies
[106634 - Suricata-Main] 2023-08-01 08:51:13 Perf: detect: UDP toclient: 21 port groups, 16 unique SGH's, 5 copies
[106634 - Suricata-Main] 2023-08-01 08:51:13 Perf: detect: OTHER toserver: 254 proto groups, 5 unique SGH's, 249 copies
[106634 - Suricata-Main] 2023-08-01 08:51:13 Perf: detect: OTHER toclient: 254 proto groups, 0 unique SGH's, 254 copies
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: Unique rule groups: 113
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: Builtin MPM "toserver TCP packet": 34
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: Builtin MPM "toclient TCP packet": 15
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: Builtin MPM "toserver TCP stream": 33
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: Builtin MPM "toclient TCP stream": 16
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: Builtin MPM "toserver UDP packet": 36
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: Builtin MPM "toclient UDP packet": 16
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: Builtin MPM "other IP packet": 5
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_uri (http)": 24
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_uri (http2)": 24
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_raw_uri (http)": 4
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_raw_uri (http2)": 4
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_request_line (http)": 12
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_request_line (http2)": 12
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_client_body (http)": 16
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_client_body (http2)": 16
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_response_line (http)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_response_line (http2)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_header (http)": 24
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_header (http)": 24
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_header (http2)": 24
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_header (http2)": 24
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_header_names (http)": 15
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_header_names (http)": 15
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_header_names (http2)": 15
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_header_names (http2)": 15
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_accept (http)": 8
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_accept (http2)": 8
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_accept_enc (http)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_accept_enc (http2)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_accept_lang (http)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_accept_lang (http2)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_referer (http)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_referer (http2)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_connection (http)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_connection (http2)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_connection (http)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_connection (http2)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_content_len (http)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_content_len (http2)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_content_len (http)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_content_len (http2)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_content_type (http)": 6
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_content_type (http2)": 6
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_content_type (http)": 6
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_content_type (http2)": 6
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http.server (http)": 4
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http.server (http2)": 4
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http.location (http)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http.location (http2)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_protocol (http)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_protocol (http)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_protocol (http2)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_protocol (http2)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_start (http)": 6
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_start (http)": 6
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_raw_header (http)": 4
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_raw_header (http)": 4
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_raw_header (http2)": 4
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_raw_header (http2)": 4
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_method (http)": 6
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_method (http2)": 6
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_cookie (http)": 6
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_cookie (http)": 6
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_cookie (http2)": 6
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_cookie (http2)": 6
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_user_agent (http)": 14
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_user_agent (http2)": 14
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_host (http)": 4
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_host (http)": 4
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_host (http2)": 4
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_host (http2)": 4
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_raw_host (http)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver http_raw_host (http2)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_stat_msg (http)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_stat_msg (http2)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_stat_code (http)": 4
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient http_stat_code (http2)": 4
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver dns_query (dns)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver dns_query (dns)": 1
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver tls.sni (tls)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver tls.sni (tls)": 1
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver tls.cert_issuer (tls)": 4
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient tls.cert_issuer (tls)": 4
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver tls.cert_subject (tls)": 5
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient tls.cert_subject (tls)": 5
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient tls.cert_serial (tls)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver tls.cert_serial (tls)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient tls.cert_fingerprint (tls)": 1
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver tls.cert_fingerprint (tls)": 1
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient tls.certs (tls)": 3
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver tls.certs (tls)": 3
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver ja3.hash (tls)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver ja3.hash (quic)": 2
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient ja3s.hash (tls)": 1
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient ja3s.hash (quic)": 1
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver ssh.proto (ssh)": 1
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient ssh.proto (ssh)": 1
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient file_data (nfs)": 29
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver file_data (nfs)": 29
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient file_data (smb)": 29
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver file_data (smb)": 29
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient file_data (ftp)": 29
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver file_data (ftp)": 29
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient file_data (ftp-data)": 29
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver file_data (ftp-data)": 29
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient file_data (http)": 29
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver file_data (http)": 29
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toclient file_data (http2)": 29
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver file_data (http2)": 29
[106634 - Suricata-Main] 2023-08-01 08:55:31 Perf: detect: AppLayer MPM "toserver file_data (smtp)": 29
[106634 - Suricata-Main] 2023-08-01 08:56:14 Config: affinity: Found affinity definition for "management-cpu-set"
[106634 - Suricata-Main] 2023-08-01 08:56:14 Config: affinity: Found affinity definition for "receive-cpu-set"
[106634 - Suricata-Main] 2023-08-01 08:56:14 Config: affinity: Found affinity definition for "worker-cpu-set"
[106634 - Suricata-Main] 2023-08-01 08:56:14 Config: affinity: Using default prio 'high' for set 'worker-cpu-set'
[106634 - Suricata-Main] 2023-08-01 08:56:14 Info: ioctl: enp216s0f0: RX RSS queues: 46
[106634 - Suricata-Main] 2023-08-01 08:56:14 Info: runmodes: enp216s0f0: creating 46 threads
[106720 - W#01-enp216s0f0] 2023-08-01 08:56:14 Perf: threads: Setting prio -2 for thread "W#01-enp216s0f0" to cpu/core 4, thread id 106720
[106727 - W#02-enp216s0f0] 2023-08-01 08:56:15 Perf: threads: Setting prio -2 for thread "W#02-enp216s0f0" to cpu/core 5, thread id 106727
[106730 - W#03-enp216s0f0] 2023-08-01 08:56:15 Perf: threads: Setting prio -2 for thread "W#03-enp216s0f0" to cpu/core 6, thread id 106730
[106733 - W#04-enp216s0f0] 2023-08-01 08:56:15 Perf: threads: Setting prio -2 for thread "W#04-enp216s0f0" to cpu/core 7, thread id 106733
[106736 - W#05-enp216s0f0] 2023-08-01 08:56:16 Perf: threads: Setting prio -2 for thread "W#05-enp216s0f0" to cpu/core 8, thread id 106736
[106739 - W#06-enp216s0f0] 2023-08-01 08:56:16 Perf: threads: Setting prio -2 for thread "W#06-enp216s0f0" to cpu/core 9, thread id 106739
[106742 - W#07-enp216s0f0] 2023-08-01 08:56:16 Perf: threads: Setting prio -2 for thread "W#07-enp216s0f0" to cpu/core 10, thread id 106742
[106745 - W#08-enp216s0f0] 2023-08-01 08:56:17 Perf: threads: Setting prio -2 for thread "W#08-enp216s0f0" to cpu/core 11, thread id 106745
[106748 - W#09-enp216s0f0] 2023-08-01 08:56:17 Perf: threads: Setting prio -2 for thread "W#09-enp216s0f0" to cpu/core 12, thread id 106748
[106751 - W#10-enp216s0f0] 2023-08-01 08:56:17 Perf: threads: Setting prio -2 for thread "W#10-enp216s0f0" to cpu/core 13, thread id 106751
[106754 - W#11-enp216s0f0] 2023-08-01 08:56:17 Perf: threads: Setting prio -2 for thread "W#11-enp216s0f0" to cpu/core 14, thread id 106754
[106757 - W#12-enp216s0f0] 2023-08-01 08:56:18 Perf: threads: Setting prio -2 for thread "W#12-enp216s0f0" to cpu/core 15, thread id 106757
[106760 - W#13-enp216s0f0] 2023-08-01 08:56:18 Perf: threads: Setting prio -2 for thread "W#13-enp216s0f0" to cpu/core 16, thread id 106760
[106763 - W#14-enp216s0f0] 2023-08-01 08:56:18 Perf: threads: Setting prio -2 for thread "W#14-enp216s0f0" to cpu/core 17, thread id 106763
[106766 - W#15-enp216s0f0] 2023-08-01 08:56:18 Perf: threads: Setting prio -2 for thread "W#15-enp216s0f0" to cpu/core 18, thread id 106766
[106769 - W#16-enp216s0f0] 2023-08-01 08:56:19 Perf: threads: Setting prio -2 for thread "W#16-enp216s0f0" to cpu/core 19, thread id 106769
[106772 - W#17-enp216s0f0] 2023-08-01 08:56:19 Perf: threads: Setting prio -2 for thread "W#17-enp216s0f0" to cpu/core 20, thread id 106772
[106775 - W#18-enp216s0f0] 2023-08-01 08:56:19 Perf: threads: Setting prio -2 for thread "W#18-enp216s0f0" to cpu/core 21, thread id 106775
[106778 - W#19-enp216s0f0] 2023-08-01 08:56:20 Perf: threads: Setting prio -2 for thread "W#19-enp216s0f0" to cpu/core 22, thread id 106778
[106781 - W#20-enp216s0f0] 2023-08-01 08:56:20 Perf: threads: Setting prio -2 for thread "W#20-enp216s0f0" to cpu/core 23, thread id 106781
[106784 - W#21-enp216s0f0] 2023-08-01 08:56:20 Perf: threads: Setting prio -2 for thread "W#21-enp216s0f0" to cpu/core 24, thread id 106784
[106787 - W#22-enp216s0f0] 2023-08-01 08:56:20 Perf: threads: Setting prio -2 for thread "W#22-enp216s0f0" to cpu/core 25, thread id 106787
[106790 - W#23-enp216s0f0] 2023-08-01 08:56:21 Perf: threads: Setting prio -2 for thread "W#23-enp216s0f0" to cpu/core 26, thread id 106790
[106793 - W#24-enp216s0f0] 2023-08-01 08:56:21 Perf: threads: Setting prio -2 for thread "W#24-enp216s0f0" to cpu/core 27, thread id 106793
[106796 - W#25-enp216s0f0] 2023-08-01 08:56:21 Perf: threads: Setting prio -2 for thread "W#25-enp216s0f0" to cpu/core 28, thread id 106796
[106799 - W#26-enp216s0f0] 2023-08-01 08:56:21 Perf: threads: Setting prio -2 for thread "W#26-enp216s0f0" to cpu/core 29, thread id 106799
[106802 - W#27-enp216s0f0] 2023-08-01 08:56:22 Perf: threads: Setting prio -2 for thread "W#27-enp216s0f0" to cpu/core 30, thread id 106802
[106805 - W#28-enp216s0f0] 2023-08-01 08:56:22 Perf: threads: Setting prio -2 for thread "W#28-enp216s0f0" to cpu/core 31, thread id 106805
[106808 - W#29-enp216s0f0] 2023-08-01 08:56:22 Perf: threads: Setting prio -2 for thread "W#29-enp216s0f0" to cpu/core 32, thread id 106808
[106811 - W#30-enp216s0f0] 2023-08-01 08:56:22 Perf: threads: Setting prio -2 for thread "W#30-enp216s0f0" to cpu/core 33, thread id 106811
[106814 - W#31-enp216s0f0] 2023-08-01 08:56:23 Perf: threads: Setting prio -2 for thread "W#31-enp216s0f0" to cpu/core 34, thread id 106814
[106817 - W#32-enp216s0f0] 2023-08-01 08:56:23 Perf: threads: Setting prio -2 for thread "W#32-enp216s0f0" to cpu/core 35, thread id 106817
[106820 - W#33-enp216s0f0] 2023-08-01 08:56:23 Perf: threads: Setting prio -2 for thread "W#33-enp216s0f0" to cpu/core 36, thread id 106820
[106823 - W#34-enp216s0f0] 2023-08-01 08:56:24 Perf: threads: Setting prio -2 for thread "W#34-enp216s0f0" to cpu/core 37, thread id 106823
[106826 - W#35-enp216s0f0] 2023-08-01 08:56:24 Perf: threads: Setting prio -2 for thread "W#35-enp216s0f0" to cpu/core 38, thread id 106826
[106829 - W#36-enp216s0f0] 2023-08-01 08:56:24 Perf: threads: Setting prio -2 for thread "W#36-enp216s0f0" to cpu/core 39, thread id 106829
[106832 - W#37-enp216s0f0] 2023-08-01 08:56:24 Perf: threads: Setting prio -2 for thread "W#37-enp216s0f0" to cpu/core 40, thread id 106832
[106835 - W#38-enp216s0f0] 2023-08-01 08:56:25 Perf: threads: Setting prio -2 for thread "W#38-enp216s0f0" to cpu/core 41, thread id 106835
[106838 - W#39-enp216s0f0] 2023-08-01 08:56:25 Perf: threads: Setting prio -2 for thread "W#39-enp216s0f0" to cpu/core 42, thread id 106838
[106841 - W#40-enp216s0f0] 2023-08-01 08:56:25 Perf: threads: Setting prio -2 for thread "W#40-enp216s0f0" to cpu/core 43, thread id 106841
[106844 - W#41-enp216s0f0] 2023-08-01 08:56:25 Perf: threads: Setting prio -2 for thread "W#41-enp216s0f0" to cpu/core 44, thread id 106844
[106847 - W#42-enp216s0f0] 2023-08-01 08:56:26 Perf: threads: Setting prio -2 for thread "W#42-enp216s0f0" to cpu/core 45, thread id 106847
[106850 - W#43-enp216s0f0] 2023-08-01 08:56:26 Perf: threads: Setting prio -2 for thread "W#43-enp216s0f0" to cpu/core 46, thread id 106850
[106853 - W#44-enp216s0f0] 2023-08-01 08:56:26 Perf: threads: Setting prio -2 for thread "W#44-enp216s0f0" to cpu/core 47, thread id 106853
[106856 - W#45-enp216s0f0] 2023-08-01 08:56:27 Perf: threads: Setting prio -2 for thread "W#45-enp216s0f0" to cpu/core 48, thread id 106856
[106859 - W#46-enp216s0f0] 2023-08-01 08:56:27 Perf: threads: Setting prio -2 for thread "W#46-enp216s0f0" to cpu/core 49, thread id 106859
[106634 - Suricata-Main] 2023-08-01 08:56:27 Info: ioctl: enp59s0f0: RX RSS queues: 46
[106634 - Suricata-Main] 2023-08-01 08:56:27 Info: runmodes: enp59s0f0: creating 46 threads
[106862 - W#01-enp59s0f0] 2023-08-01 08:56:27 Perf: threads: Setting prio -2 for thread "W#01-enp59s0f0" to cpu/core 50, thread id 106862
[106862 - W#01-enp59s0f0] 2023-08-01 08:56:27 Error: af-xdp: Failed to create socket: Invalid argument
[106863 - W#02-enp59s0f0] 2023-08-01 08:56:27 Perf: threads: Setting prio -2 for thread "W#02-enp59s0f0" to cpu/core 51, thread id 106863

antsknows · August 6, 2023, 1:52am

Hi @scloder ,
Have u solved this problem?
I’ve met the same thing, ubuntu16.04, 32 workers.

"Error: af-xdp: Failed to create socket: Invalid argument [OpenXSKSocket:source-af-xdp.c:444] "

Strangely, this issue is intermittent. After restarting the machine, it will work normally for a certain period of time. However, once Suricata has been restarted multiple times, it will occur again

scloder · August 15, 2023, 4:53pm

I haven’t made any progress. I am currently going back to using af-packet for now to see if I can get it working better. Hopefully others like yourself can add their experience here or elsewhere so we can get a better way to target the problem.

sbhardwaj · August 16, 2023, 6:31am

Hi!
Do you also see anything on your system logs?

Topic		Replies	Views
Seems parameters '--unix-socket' and '--af-apcket' are incompatible?	3	253	September 19, 2023
Af_packet + RSS, some workers doing nothing / not loaded Help	13	1350	June 12, 2021
Suricata high capture.kernel_drops count Help	23	7261	September 10, 2020
Verifying AF_XDP installation Help	1	429	May 7, 2023
How to enable more capture threads in autofp mode? Help suricata	2	415	May 19, 2023

Suricata 7 with multiple NICs and cpu affinity -- af-xdp: Failed to create socket

Related topics