Cpu affinity and 1 receive/management core 100%

Help
1

Hello,

In order to have cu affinity, created this config:

- management-cpu-set:
    cpu: [ "1-5","32-37" ]  # include only these CPUs in affinity settings
- receive-cpu-set:
    cpu: [ "0-5","32-37" ]  # include only these CPUs in affinity settings

Now is see core 32 almost all of the time at 100% cpu, the other cores from the receive/management range dance between 0 and 10% . Sometimes core 32 decreases and load seems to spread to the other cores in the range, but that is just for a few seconds and 32 hits 100% again.
Looking at an overload problem?

  • Red Hat Enterprise Linux release 8.8 (Ootpa)
    4.18.0-477.15.1.el8_8.x86_64 #1 SMP Fri Jun 2 08:27:19 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux
  • This is Suricata version 7.0.1-dev (becb8cefc 2023-08-11)
  • DPDK 23.11.0-rc0

Cheers,
Andre

2

what is your worker-cpu-set?

btw receive-cpu-set is only used in Autofp mode, so not relevant for workers runmode - it is possible to comment it out.

Considering you probably have many workers enabled, have you considered increasing number of managers and recyclers - by default only 1 is enabled and that might explain why 1 core is hitting 100% CPU usage.

TBH having 100% CPU on management cores seems weird to me anyway - no matter how big your flow table is and how many workers you have.
But I would try to increase the manager/recycler count and see if that would help

3

Thanks for the tips, but to no avail or did I configure it not oke?

flow:
#memcap: 128mb
#memcap-policy: ignore
#hash-size: 65536
#prealloc: 10000
emergency-recovery: 30
managers: 5 # default to one flow manager
recyclers: 5 # default to one flow recycler thread
memcap: 4gb
hash-size: 1048576
prealloc: 1048576

threading:
cpu-affinity:
- management-cpu-set:
cpu: [ “1-5”,“32-37” ] # include only these CPUs in affinity settings
# receive-cpu-set is only used in Autofp mode
#- receive-cpu-set:
# cpu: [ “0-5”,“32-37” ] # include only these CPUs in affinity settings
- worker-cpu-set:
cpu: [ “6-31”,“38-63”,“64-95”,“96-113” ]
mode: “exclusive”
prio:
low: [ 0 ]
medium: [ “1” ]
cpu: [ “6-31”,“38-63”,“64-95”,“96-113” ]
default: “high”

4

$ perf top -C 32

34.37% [kernel] [k] do_select
32.10% [kernel] [k] __fget_light
18.89% [kernel] [k] fput_many
3.42% [kernel] [k] __x86_return_thunk
1.42% [kernel] [k] __fdget
1.41% [kernel] [k] fput
0.89% [kernel] [k] memset_erms
0.75% [kernel] [k] copy_user_enhanced_fast_string
0.54% [kernel] [k] read_tsc

5

Ok, the config seems okay.

I’ve tested the config and also noticed high usage of the functions that you mentioned. Then I tried to replace 2MB with 1GB hugepages. And both this problem and the UNIX socket problem went away.
I think we can take that as a solution though I’ll try to understand a bit more why 2MB hugepages are causing the issues…

You can try it out with e.g.
sudo dpdk-hugepages.py --pagesize 1G --setup 8G

But note that hugepages require continuous memory so it is less likely to obtain 1GB hugepages than 2MB when memory gets fragmented as the OS runs.

1 Like
6

]# hugeadm --page-sizes-all
2097152
1073741824

]# dpdk-hugepages.py --pagesize 1G --setup 8G

]# hugeadm --page-sizes
1073741824

Now indeed the cpu cores allocated for suricata management are running smooth, nice catch!

7

What kind of CPU is that if ok to ask ?

8

Hey Peter, a real one or a HT version you mean? Or AMD, Intel or such?

9

I guess Peter meant the CPU model name that you have installed on your machine. So he was looking for info from e.g. lscpu like Intel® Xeon® Processor E5-4617

10

Yes, as Lukas said , if it is like Intel or AMD etc .

11

Guess I understand Lukas accent better then Peters :wink: I’ve an accent myself in English too so just kidding :sunglasses:

Btw ‘t if ok to ask ?’ I ask you guys much more then I can give so always feel free to ask Peter!

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 128
On-line CPU(s) list: 0-127
Thread(s) per core: 2
Core(s) per socket: 32
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
BIOS Vendor ID: Intel(R) Corporation
CPU family: 6
Model: 106
Model name: Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz
BIOS Model name: Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz
Stepping: 6
CPU MHz: 2600.000
CPU max MHz: 3400.0000
CPU min MHz: 800.0000
BogoMIPS: 5200.00
Virtualization: VT-x
L1d cache: 48K
L1i cache: 32K
L2 cache: 1280K
L3 cache: 49152K
NUMA node0 CPU(s): 0-31,64-95
NUMA node1 CPU(s): 32-63,96-127

1 Like