SIGABRT w/ Suricata 7.0.4/7.0.5 in af-packet mode

Developers
1

Hi,

since 7.0.4, I recognized multiple coredumps due to a SIGABRT. This was the case on different hosts (all on debian 12.5) and on different interfaces with different numbers of threads and cpu cores.

Core was generated by `/usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (threadid=<optimized out>, 
    signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
44	./nptl/pthread_kill.c: Datei oder Verzeichnis nicht gefunden.
[Current thread is 1 (Thread 0x7faa1f7ff6c0 (LWP 64476))]
(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, 
    signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
#1  0x00007fab2acf2e8f in __pthread_kill_internal (signo=6, 
    threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  0x00007fab2aca3fb2 in __GI_raise (sig=sig@entry=6)
    at ../sysdeps/posix/raise.c:26
#3  0x00007fab2ac8e472 in __GI_abort () at ./stdlib/abort.c:79
#4  0x00007fab2ace7430 in __libc_message (action=action@entry=do_abort, 
    fmt=fmt@entry=0x7fab2ae01459 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#5  0x00007fab2acfc7aa in malloc_printerr (
    str=str@entry=0x7fab2ae04590 "malloc(): invalid next size (unsorted)")
    at ./malloc/malloc.c:5660
#6  0x00007fab2acff8e4 in _int_malloc (av=av@entry=0x7fab18000030, 
    bytes=bytes@entry=36865) at ./malloc/malloc.c:4001
#7  0x00007fab2ad00362 in _int_realloc (av=av@entry=0x7fab18000030, 
    oldp=oldp@entry=0x7f8cf4ffd710, oldsize=oldsize@entry=34832, 
    nb=nb@entry=36880) at ./malloc/malloc.c:4874
#8  0x00007fab2ad0120f in __GI___libc_realloc (
    oldmem=oldmem@entry=0x7f8cf4ffd720, bytes=bytes@entry=36864)
    at ./malloc/malloc.c:3489
#9  0x000056029421562e in SCReallocFunc (ptr=ptr@entry=0x7f8cf4ffd720, 
    size=size@entry=36864) at ./src/util-mem.c:46
#10 0x00005602942f1c45 in StreamTcpReassembleRealloc (optr=0x7f8cf4ffd720, 
    orig_size=34816, size=36864) at ./src/stream-tcp-reassemble.c:236
#11 0x000056029432149b in GrowRegionToSize (sb=0x7faa06beb3c8, size=35046, 
    region=0x7faa06beb3c8, cfg=0x560294af2198 <stream_config+56>)
    at ./src/util-streaming-buffer.c:722
#12 GrowToSize (size=35046, cfg=0x560294af2198 <stream_config+56>, 
    sb=0x7faa06beb3c8) at ./src/util-streaming-buffer.c:746
#13 StreamingBufferInsertAt (sb=sb@entry=0x7faa06beb3c8, 
    cfg=0x560294af2198 <stream_config+56>, seg=seg@entry=0x7faa12710ba4, 
    data=0x7f9cf7604000 "", data_len=<optimized out>, offset=<optimized out>)
    at ./src/util-streaming-buffer.c:1526
#14 0x00005602942f0d99 in InsertSegmentDataCustom (data_len=1298, 
    data=0x7f9cf7604000 "", seg=0x7faa12710b80, stream=0x7faa06beb390)
    at ./src/stream-tcp-list.c:99
#15 StreamTcpReassembleInsertSegment (tv=tv@entry=0x56036d7d8090, 
    ra_ctx=ra_ctx@entry=0x7faa11b7f450, stream=stream@entry=0x7faa06beb390, 
    seg=0x7faa12710b80, p=p@entry=0x7faa11b46580, pkt_seq=<optimized out>, 
    pkt_data=<optimized out>, pkt_datalen=<optimized out>)
    at ./src/stream-tcp-list.c:654
#16 0x00005602942f38a1 in StreamTcpReassembleHandleSegmentHandleData (
    tv=tv@entry=0x56036d7d8090, ra_ctx=ra_ctx@entry=0x7faa11b7f450, 
    ssn=ssn@entry=0x7faa06beb380, stream=stream@entry=0x7faa06beb390, 
    p=p@entry=0x7faa11b46580)
    at /usr/include/x86_64-linux-gnu/bits/byteswap.h:52
#17 0x00005602942f3aba in StreamTcpReassembleHandleSegment (
    tv=tv@entry=0x56036d7d8090, ra_ctx=0x7faa11b7f450, 
    ssn=ssn@entry=0x7faa06beb380, stream=0x7faa06beb390, 
    p=p@entry=0x7faa11b46580) at ./src/stream-tcp-reassemble.c:2016
#18 0x00005602942ea17e in HandleEstablishedPacketToClient (
    stt=<optimized out>, p=<optimized out>, ssn=<optimized out>, 
    tv=<optimized out>) at ./src/stream-tcp.c:2777
#19 StreamTcpPacketStateEstablished (tv=0x56036d7d8090, p=0x7faa11b46580, 
    stt=stt@entry=0x7faa11b7f1a0, ssn=0x7faa06beb380)
    at ./src/stream-tcp.c:3223
#20 0x00005602942ec598 in StreamTcpStateDispatch (tv=tv@entry=0x56036d7d8090, 
    p=p@entry=0x7faa11b46580, stt=stt@entry=0x7faa11b7f1a0, 
    ssn=ssn@entry=0x7faa06beb380, state=<optimized out>)
    at ./src/stream-tcp.c:5236
#21 0x00005602942edeb6 in StreamTcpPacket (tv=tv@entry=0x56036d7d8090, 
    p=p@entry=0x7faa11b46580, stt=stt@entry=0x7faa11b7f1a0, 
    pq=pq@entry=0x7faa11b652a0) at ./src/stream-tcp.c:5433
#22 0x00005602942ef0f9 in StreamTcp (tv=tv@entry=0x56036d7d8090, 
    p=p@entry=0x7faa11b46580, data=0x7faa11b7f1a0, pq=pq@entry=0x7faa11b652a0)
    at ./src/stream-tcp.c:5745
#23 0x00005602942ac575 in FlowWorkerStreamTCPUpdate (
    tv=tv@entry=0x56036d7d8090, fw=fw@entry=0x7faa11b65270, 
    p=p@entry=0x7faa11b46580, 
    detect_thread=detect_thread@entry=0x7faa06def450, 
    timeout=timeout@entry=false) at ./src/flow-worker.c:391
#24 0x00005602942acae7 in FlowWorker (tv=0x56036d7d8090, p=0x7faa11b46580, 
    data=0x7faa11b65270) at ./src/flow-worker.c:619
#25 0x000056029420295f in TmThreadsSlotVarRun (tv=tv@entry=0x56036d7d8090, 
    p=p@entry=0x7faa11b46580, slot=<optimized out>) at ./src/tm-threads.c:135
#26 0x00005602942d9f24 in TmThreadsSlotProcessPkt (tv=0x56036d7d8090, 
    s=<optimized out>, p=0x7faa11b46580) at ./src/tm-threads.h:200
#27 0x00005602942da262 in AFPParsePacketV3 (pbd=0x7f9cf7600000, 
    ppd=0x7f9cf7603f68, ptv=0x7faa11b46fc0) at ./src/source-af-packet.c:1013
#28 AFPWalkBlock (pbd=0x7f9cf7600000, ptv=0x7faa11b46fc0)
    at ./src/source-af-packet.c:1032
#29 AFPReadFromRingV3 (ptv=ptv@entry=0x7faa11b46fc0)
    at ./src/source-af-packet.c:1079
#30 0x00005602942dade4 in ReceiveAFPLoop (tv=0x56036d7d8090, 
    data=0x7faa11b46fc0, slot=<optimized out>) at ./src/source-af-packet.c:1431
#31 0x00005602942040b4 in TmThreadsSlotPktAcqLoop (td=0x56036d7d8090)
    at ./src/tm-threads.c:318
#32 0x00007fab2acf1134 in start_thread (arg=<optimized out>)
    at ./nptl/pthread_create.c:442
#33 0x00007fab2ad717dc in clone3 ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

I usually compile suricata using the debian src package w/ some modifications, but these are completely unrelated to the code mentioned above. The configure options are

CONFIGURE_ARGS = --enable-af-packet --enable-xdp --enable-dpdk \
	--enable-gccprotect --disable-gccmarch-native \
	--with-libnss-includes=/usr/include/nss --with-libnss-libraries=/usr/lib/$(DEB_HOST_MULTIARCH) \
	--with-libnspr-includes=/usr/include/nspr --with-libnspr-libraries=/usr/lib/$(DEB_HOST_MULTIARCH) \
	--with-libevent-includes=/usr/include --with-libevent-libraries=/usr/lib/$(DEB_HOST_MULTIARCH) \
	--disable-coccinelle \
	--enable-geoip --enable-hiredis \
	--enable-non-bundled-htp \
	--disable-suricata-update \
	$(ENABLE_LUAJIT) \
	$(ENABLE_HYPERSCAN) \
	$(ENABLE_UNITTESTS) \
	$(ENABLE_EBPF)

Did someone else experience this, and what can I do to further investigate this?

Thanks, maja

2

Are you able to rebuild with address sanitizer enabled? Looks like a memory corruption based on the bt. ASAN usually catches things a bit earlier.

3

Thanks, I recompiled it w/ “–enable-asan”.

Unfortunately, it takes a few days until a core dump occurs.
I will report back if I have one.

4

Ok, --enable-asan did not have any effect. I had to set the CFLAGS to -fsanitize=address,undefined -Wformat -Werror=format-security -Werror=array-bounds -g -O0.

With that, the address sanitizer now complaints about a heap buffer overflow. But indeed in one of our added code for base64 converting and dumping the first few bytes of http request and response headers.

Thus, even if this is not solved in the narrow sense, I will mark the topic as solved.

But thanks anyway for pointing me at the ASAN feature of gcc.