Empty fast.log when 2GB pcap

Help
,
1

Hello, I am currently testing suricata with huge pcap files. Suricata with testfile2 creating empty fast.log. Eve.json does not contain event_type: alert, but 99% “event_type”:“flow” and several “event_type”:“stats”.
With testfile1 all is fine: fast.log and eve.json contain alerts
default ETOpen rules, suricata 6.03
cmd: suricata -k none -c /suricata.yaml -r testfile1.pcap or testfile2.pcap

Testfile1: bigFlows.pcap from Sample Captures
Testfile2: ca 1.8GB pcap from http://mawi.wide.ad.jp/mawi/samplepoint-F/2021/202110311400.pcap.gz

Tuning options from offical surica docu did not help. What else I can check?

surica.log and stats.log from Testfile2
thankssuricata.log (5.0 KB)
stats.log (57.8 KB)

suricata --build-info
This is Suricata version 6.0.3 RELEASE
Features: NFQ PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LUAJIT HAVE_LIBJANSSON TLS TLS_C11 MAGIC RUST
SIMD support: none
Atomic intrinsics: 1 2 4 8 byte(s)
64-bits, Little-endian architecture
GCC version 7.5.0, C version 201112
compiled with _FORTIFY_SOURCE=2
L1 cache line size (CLS)=64
thread local storage method: _Thread_local
compiled with LibHTP v0.5.38, linked against LibHTP v0.5.38

Suricata Configuration:
AF_PACKET support: yes
eBPF support: no
XDP support: no
PF_RING support: no
NFQueue support: yes
NFLOG support: no
IPFW support: no
Netmap support: no
DAG enabled: no
Napatech enabled: no
WinDivert enabled: no

Unix socket enabled: yes
Detection enabled: yes

Libmagic support: yes
libnss support: yes
libnspr support: yes
libjansson support: yes
hiredis support: yes
hiredis async with libevent: yes
Prelude support: no
PCRE jit: yes
LUA support: yes, through luajit
libluajit: yes
GeoIP2 support: yes
Non-bundled htp: yes
Hyperscan support: yes
Libnet support: yes
liblz4 support: yes
HTTP2 decompression: no

Rust support: yes
Rust strict mode: no
Rust compiler path: /usr/bin/rustc
Rust compiler version: rustc 1.47.0
Cargo path: /usr/bin/cargo
Cargo version: cargo 1.46.0
Cargo vendor: yes

Python support: yes
Python path: /usr/bin/python3
Python distutils yes
Python yaml yes
Install suricatactl: yes
Install suricatasc: yes
Install suricata-update: yes

Profiling enabled: no
Profiling locks enabled: no

Plugin support (experimental): yes

Development settings:
Coccinelle / spatch: no
Unit tests enabled: no
Debug output enabled: no
Debug validation enabled: no

Generic build parameters:
Installation prefix: /usr
Configuration directory: /etc/suricata/
Log directory: /var/log/suricata/

–prefix /usr
–sysconfdir /etc
–localstatedir /var
–datarootdir /usr/share

Host: x86_64-pc-linux-gnu
Compiler: gcc (exec name) / g++ (real)
GCC Protect enabled: yes
GCC march native enabled: no
GCC Profile enabled: no
Position Independent Executable enabled: yes
CFLAGS -g -O2 -fdebug-prefix-map=/build/suricata-uRINne/suricata-6.0.3=. -fstack-protector-strong -Wformat -Werror=format-security -std=c11 -I${srcdir}/…/rust/gen -I${srcdir}/…/rust/dist
PCAP_CFLAGS -I/usr/include
SECCFLAGS -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security

2

Testfile2 has a really small snaplen, so almost all the data packets are truncated. Suricata doesn’t support this, it requires a full capture. In your stats log see the decoder.invalid and the average packet size as an indicator. Haven’t checked the other pcaps yet.

3

okay, what IDS dataset do you recommend to evaluate suricata?

4

how can I verify if pcap file is full capture before running the it with suricata?

5

where I can find more about it in offical documentation?

6

Hí,

Here you have a lot of pcaps files. They contain malware traffic to correctly test Suricata, alerts, etc.

https://www.malware-traffic-analysis.net/2021/index.html

7

I know them, but I need labeled dataset so I can calculate detection ratio of suricata