we’re running v8.0.1 (Debian Trixie) and have statistics logging per thread enabled: stats.threads == true. Our system is a dual CPU with 128 cores each, out of which we are using 120 per CPU for Suricata, so 120 Suricate workers per CPU (and thus per NIC) and 240 workers in total.
We do get statistics for individual threads but only up to and including worker #99 per interface.
[1] Info: runmodes: pcap0: creating 120 threads
[27] Perf: threads: Setting prio -2 for thread “W#01-pcap0” to cpu/core 4, thread id 27
[28] Perf: threads: Setting prio -2 for thread “W#02-pcap0” to cpu/core 5, thread id 28
…
[145] Perf: threads: Setting prio -2 for thread “W#119-pcap0” to cpu/core 190, thread id 145
[146] Perf: threads: Setting prio -2 for thread “W#120-pcap0” to cpu/core 191, thread id 146
…
[1] Notice: threads: Threads created → W: 240 FM: 2 FR: 2 Engine started.
If we look at the logging, we only get output from workers 1 to 99 per interface:
$ tail -1 eve.1.json | jq .stats.threads | grep “W#.*pcap0” | wc -l
99
Hi John, I did a small artificial test and was able to get per thread stats in both stats.log and eve stats log for 240 worker threads. Are you suppressing null-values? Maybe these threads are getting no traffic?
according to ethtool (almost) all queues are receiving traffic. Those not receiving traffic still report afpacket counters like this (we did: not send any traffic to interface pcap1 during this test; ethtool counters all are 0 for this interface): “W#98-pcap1”: { “capture”: { “afpacket”: { “polls”: 4078, “poll_timeout”: 4077 } } },
I’ll test with null-values enabled and see what happens.
it took some time before we could test again.
First of all: null value logging enabled or not does not make any difference.
There is a difference in results between the stats.log and the statistics logging in eve.1.json:
in the stats.log, counters for all workers, thus also for worker W#100 and above, are reported. Those are not present in the eve-log.
but: starting with worker 100, the name of the worker is capped, see this snippet:
capture.kernel_packets | W#98-pcap0 | 10434906
capture.kernel_packets | W#99-pcap0 | 5962934
capture.kernel_packets | W#100-pcap | 7453505
capture.kernel_packets | W#101-pcap | 5963147
capture.kernel_packets | W#102-pcap | 17888172
in the eve logs, the same (capped) worker names (as in stats.log) are reported but starting with worker #100, all counters (except for afpacket.polls and afpacket.poll_timeout) are 0.
I also checked with Suricate version 7 but due to the CPU core numbering I do not hit the 100 number of workers per interface. This means I cannot compare the results with v7.