Hi, this could go to any forum from pfsense, freebsd, suricata or netmap so a first pickle right off the bat.
With pfsense 2.5.0/FreeBSD 12.2, I observe interesting behavior of Suricata not detecting and setting correct count of threads in workers mode with netmap. Not sure to understand what’s the culprit.
When set to auto it reports:
14/2/2021 – 11:31:36 - – This is Suricata version 5.0.5 RELEASE running in SYSTEM mode
14/2/2021 – 11:31:36 - – CPUs/cores online: 16
14/2/2021 – 11:31:36 - – HTTP memcap: 671088640
14/2/2021 – 11:31:39 - – all 2 packet processing threads, 4 management threads initialized, engine started.
No other setting is affecting that outcome. But system seems to be aware of more RSS queues:
dmesg | grep ^ix0 | more
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0x6000-0x601f mem 0xfd100000-0xfd17ffff,0xfd1fc000-0xfd1fffff irq 18 at device 0.0 on pci4
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 16 RX queues 16 TX queues
ix0: Using MSI-X interrupts with 17 vectors
ix0: allocated for 16 queues
ix0: allocated for 16 rx queues
ix0: Ethernet address: ec:f4:bb:c3:9b:4a
ix0: PCI Express Bus: Speed 5.0GT/s Unknown
ix0: netmap queues/slots: TX 16/2048, RX 16/2048
sysctl dev.ix | grep queue
dev.ix.0.iflib.txq15.r_enqueues: 9920
dev.ix.0.iflib.txq14.r_enqueues: 11135
dev.ix.0.iflib.txq13.r_enqueues: 10007
dev.ix.0.iflib.txq12.r_enqueues: 11404
dev.ix.0.iflib.txq11.r_enqueues: 9926
dev.ix.0.iflib.txq10.r_enqueues: 10089
dev.ix.0.iflib.txq09.r_enqueues: 10018
dev.ix.0.iflib.txq08.r_enqueues: 10184
dev.ix.0.iflib.txq07.r_enqueues: 10433
dev.ix.0.iflib.txq06.r_enqueues: 10228
dev.ix.0.iflib.txq05.r_enqueues: 10099
dev.ix.0.iflib.txq04.r_enqueues: 10291
dev.ix.0.iflib.txq03.r_enqueues: 10328
dev.ix.0.iflib.txq02.r_enqueues: 10251
dev.ix.0.iflib.txq01.r_enqueues: 10415
dev.ix.0.iflib.txq00.r_enqueues: 0
When I manually set threads to 8 or 16 for testing it is working fine.
Netmap
netmap:
- interface: default
threads: 8
copy-mode: ips
13/3/2021 – 14:58:38 - – Going to use 8 thread(s)
13/3/2021 – 14:58:38 - – opened netmap:ix0-0/R from ix0: 0x80e84b000
13/3/2021 – 14:58:38 - – opened netmap:ix0^ from ix0^: 0x80e84b300
13/3/2021 – 14:58:38 - – opened netmap:ix0-1/R from ix0: 0xa0fffd000
13/3/2021 – 14:58:38 - – opened netmap:ix0^ from ix0^: 0xa0fffd300
13/3/2021 – 14:58:39 - – opened netmap:ix0-2/R from ix0: 0xc0f5fc000
13/3/2021 – 14:58:39 - – opened netmap:ix0^ from ix0^: 0xc0f5fc300
13/3/2021 – 14:58:39 - – opened netmap:ix0-3/R from ix0: 0xe0f306000
13/3/2021 – 14:58:39 - – opened netmap:ix0^ from ix0^: 0xe0f306300
13/3/2021 – 14:58:39 - – opened netmap:ix0-4/R from ix0: 0x100e1f4000
13/3/2021 – 14:58:40 - – opened netmap:ix0^ from ix0^: 0x100e1f4300
13/3/2021 – 14:58:40 - – opened netmap:ix0-5/R from ix0: 0x120d9fc000
13/3/2021 – 14:58:40 - – opened netmap:ix0^ from ix0^: 0x120d9fc300
13/3/2021 – 14:58:40 - – opened netmap:ix0-6/R from ix0: 0x140cbd1000
13/3/2021 – 14:58:40 - – opened netmap:ix0^ from ix0^: 0x140cbd1300
13/3/2021 – 14:58:40 - – opened netmap:ix0-7/R from ix0: 0x160b7fc000
13/3/2021 – 14:58:41 - – opened netmap:ix0^ from ix0^: 0x160b7fc300
13/3/2021 – 14:58:41 - – Going to use 1 thread(s)
13/3/2021 – 14:58:41 - – opened netmap:ix0^ from ix0^: 0x180b7ab000
13/3/2021 – 14:58:41 - – opened netmap:ix0-0/T from ix0: 0x180b7ab300
13/3/2021 – 14:58:41 - – all 9 packet processing threads, 4 management threads initialized, engine started.
Of course difference in performance between auto & manual is devastating with 8 threads dealing with 10Gb/s (about 9-9.5Gb with iperf3 and -P 4) and having some spare capacity with CPU/IRQs.
System is based on E5-2667v2 running as VM with 16 cores.
13/3/2021 – 14:58:35 - – This is Suricata version 5.0.5 RELEASE running in SYSTEM mode
13/3/2021 – 14:58:35 - – CPUs/cores online: 16
13/3/2021 – 14:58:35 - – HTTP memcap: 671088640
13/3/2021 – 14:58:35 - – using flow hash instead of active packets
13/3/2021 – 14:58:35 - – Netmap: Setting IPS mode
13/3/2021 – 14:58:35 - – fast output device (regular) initialized: alerts.log
13/3/2021 – 14:58:35 - – stats output device (regular) initialized: stats.log
13/3/2021 – 14:58:35 - – Syslog output initialized
13/3/2021 – 14:58:36 - – 3 rule files processed. 2188 rules successfully loaded, 0 rules failed
13/3/2021 – 14:58:36 - – Threshold config parsed: 4 rule(s) found
13/3/2021 – 14:58:36 - – 2188 signatures processed. 58 are IP-only rules, 459 are inspecting packet payload, 1391 inspect application layer, 103 are decoder event only
Where should I look to see why auto thread setting does not properly detect threads based on RSS queues (16)? It wouldn’t be an issue but pfsense Suricata package is pre-scripted so every update overwrites the files I needed to change i.e. /usr/local/pkg/suricata/suricata_generate_yaml.php
Thanks upfront for any guidance.
M