Suricata-7 crashed while processing a packet, even before Initialisation

1

I am running Suricata-7 in NFQ mode

It crashed during Init

Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00000000005b859e in StatsIncr (tv=tv@entry=0x6988280, id=) at counters.c:181

[Current thread is 1 (Thread 0x7ff542ffd700 (LWP 108))]
(gdb) bt
#0 0x00000000005b859e in StatsIncr (tv=tv@entry=0x6988280, id=) at counters.c:181
#1 0x00000000005bebc3 in CaptureStatsUpdate (tv=0x6988280, p=0x7ff53a847e60) at decode.c:912
#2 0x000000000056c1a5 in TmqhOutputPacketpool (t=0x6988280, p=0x7ff53a847e60) at tmqh-packetpool.c:383
#3 0x00000000006511a9 in TmThreadsSlotProcessPkt (tv=0x6988280, s=, p=0x7ff53a847e60) at /__w/suricata-inline/suricata-inline/oisf/src/tm-threads.h:196
#4 NFQCallBack (qh=0x7ff53a848b10, nfmsg=, nfa=0x7ff542ffa120, data=0x1b46a60) at source-nfq.c:565
#5 0x00007ff55c4e12a9 in ?? () from /lib/x86_64-linux-gnu/libnetfilter_queue.so.1
#6 0x00007ff55c4d8bea in ?? () from /lib/x86_64-linux-gnu/libnfnetlink.so.0
#7 0x00007ff55c4d93a0 in nfnl_process () from /lib/x86_64-linux-gnu/libnfnetlink.so.0
#8 0x00007ff55c4d973e in nfnl_catch () from /lib/x86_64-linux-gnu/libnfnetlink.so.0
#9 0x00007ff55c4e19a5 in nfq_set_queue_maxlen () from /lib/x86_64-linux-gnu/libnetfilter_queue.so.1
#10 0x0000000000650c7d in NFQInitThread (t=0x1b46a60, queue_maxlen=64384) at source-nfq.c:635
#11 ReceiveNFQThreadInit (tv=, initdata=0x1b46a60, data=0x7ff542ffc420) at source-nfq.c:718
#12 0x000000000056e4ca in TmThreadsSlotPktAcqLoop (td=0x6988280) at tm-threads.c:264
#13 0x00007ff55c2a9609 in start_thread (arg=) at pthread_create.c:477
#14 0x00007ff55b7b4353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

(gdb) p tv->perf_private_ctx
$6 = {
head = 0x0,
size = 0,
initialized = 0
}

Usual Init workflow seems to be:
TmThreadsSlotPktAcqLoop() is invoked through SlotThreadInit()
Init flag is set when done: TmThreadsSetFlag(tv, THV_INIT_DONE)
ReceiveNFQLoop() is invoked to receive packets through NFQCallBack()

In the crash case, I see the following:
TmThreadsSlotPktAcqLoop() is invoked through SlotThreadInit()
ReceiveNFQThreadInit() is still in progress
NFQCallBack() gets called with uninitialised thread variables

Does anyone have any pointers to figure out what’s happening?

Thanks in advance!

2

Please provide the suricata.yaml and stats.log as well as the suricata.log and the run command.

What exact version is used?

Also add suricata --build-info

3

Unable attach suricata.yaml due to - “New uses are not allowed to add attachments”

suricata --build-info

This is Suricata version 7.0.6 RELEASE
Features: NFQ PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HTTP2_DECOMPRESSION HAVE_JA3 HAVE_JA4 HAVE_LIBJANSSON TLS TLS_C11 MAGIC RUST POPCNT64
SIMD support: SSE_2
Atomic intrinsics: 1 2 4 8 byte(s)
64-bits, Little-endian architecture
GCC version 14.1.0, C version 201112
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: _Thread_local
compiled with LibHTP v0.5.48, linked against LibHTP v0.5.48

Command

suricata -c suricata.yaml -k none -q0:7 --pidfile /var/suricata/suricata.pid

However, I found the root cause, and a potential fix

During Init, the following sequence led to the crash ReceiveNFQThreadInit() → NFQCallBack() → TmThreadsSlotProcessPkt() → TmqhOutputPacketpool() → CaptureStatsUpdate()
static inline TmEcode TmThreadsSlotProcessPkt(ThreadVars *tv, TmSlot *s, Packet p)
{
if (s == NULL) {
tv->tmqh_out(tv, p); ← Call to TmqhOutputPacketpool(), as (TmSlot) was NULL
return TM_ECODE_OK;
}
…
}

But, TmqhOutputPacketpool() made an unconditional call to CaptureStatsUpdate(), while (ThreadVars *t) was not yet initialised (ReceiveNFQThreadInit was still in progress)

(gdb) p tv->perf_private_ctx
$6 = {
head = 0x0,
size = 0,
initialized = 0
}

$3 = (ThreadVars *) 0x6988280
(gdb) p *tv
$4 = {
t = 140691367778048,
tm_func = 0x56e3d0 ,
name = “W-NFQ#3\000\000\000\000\000\000\000\000”,
…
flags_sc_atomic__ = 5,

flags = 5 → 101 → Bit(0) and Bit(2) are set
THV_USE and THV_PAUSE are set
THV_INIT_DONE is not set

CaptureStatsUpdate() tried to use tv->perf_private_ctx, which led to the crash

Potential fix:

Avoid calling CaptureStatsUpdate() when the thread in not Initialised
if (t != NULL && TmThreadsCheckFlag(t, THV_INIT_DONE)) { ← Check for thread Init flag
CaptureStatsUpdate(t, p);
}