Performance: usec feedback

Help
1

Seems if you are sitting on kernel 4.19+ it gets easier to gather/display much needed stats as there are better/improved/newer tools available.
One of those is pretty easy to install:

apt-get install bpfcc-tools python3-bpfcc

and to use:

cpudist-bpfcc -p `pidof suricata` 600 1

This would basically show all usec done by Suricata for the 10min period
It can also be done per thread which can come very handy in troubleshooting:

cpudist-bpfcc -L -p `pidof suricata` 600 1

This (screenshot attached) is what I noticed on a heavily tested machine inside a peak of 10min. Wondering out of curiosity what everyone else is seeing . Runing 6.0.0-dev (26123e05f 2020-03-20) with AFPv3, 40 threads, 4.19 kernel.

Screenshot from 2020-03-26 13-11-17
Screenshot from 2020-03-26 13-11-17722×384 13.6 KB

2

There are some examples and explanation of how the tool can be used here - https://github.com/iovisor/bcc/blob/master/tools/cpudist_example.txt
and ways to detect lock contention and on-CPU/off-CPU time.

3

That looks awesome. Going to try throwing it at a couple of things, including suricata.

4

Nice, thanks! Looking forward to the feedback - it would be helpful to “calibrate” any conclusions i think.
Interested on PID views as well. I could follow up with parallel findings/printouts and we can maybe kind of compare results.