Intel E810-C only uses one RSS queue

Help
1

Dear community,

we are using Suricata 6.0.19 (soon 6.0.20) on Debian 11 (bullseye) from custom-built Debian packages based on official Debian’s packaging but updated with current sources so we can have current 6.x versions.

Recently we are looking at a setup that involves a 48-core machine with Intel E810-C NICs, among others. Here are some details:

$ sudo lspci | grep 41
41:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-C for QSFP (rev 02)
41:00.1 Ethernet controller: Intel Corporation Ethernet Controller E810-C for QSFP (rev 02)

$ sudo ethtool --driver S2
driver: ice
version: 5.10.0-28-amd64
firmware-version: 4.30 0x8001bcf8 1.3429.0
expansion-rom-version: 
bus-info: 0000:41:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

In order to handle the incoming traffic (>10Gb) we would like to use RSS and cluster_qm to balance incoming flows across 22 threads, so we followed the docs and configured the interface like this:

ethtool -L S2 combined 22
ethtool -X S2 hkey 6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A equal $QUEUES
ethtool -K S2 rxhash on
ethtool -K S2 ntuple on
for i in rx tx tso gso gro lro tx-nocache-copy sg txvlan rxvlan; do ethtool -K S2 $i off; done
for proto in tcp4 udp4 tcp6 udp6; do ethtool -N S2 rx-flow-hash $proto sdfn; done
ethtool -C S2 adaptive-rx off rx-usecs 62
ethtool -G S2 rx 512
ethtool -X S2 hfunc toeplitz
/usr/local/bin/set_irq_affinity 1-22 S2

which are accepted and configure the flow hashing correctly (as far as I can see):

$ sudo ethtool -l S2
Channel parameters for S2:
Pre-set maximums:
RX:		48
TX:		48
Other:		1
Combined:	48
Current hardware settings:
RX:		0
TX:		0
Other:		1
Combined:	22

$ sudo ethtool --show-rxfh  S2
RX flow hash indirection table for S2 with 22 RX ring(s):
    0:      0     1     2     3     4     5     6     7
    8:      8     9    10    11    12    13    14    15
   16:     16    17    18    19    20    21     0     1
   24:      2     3     4     5     6     7     8     9
[...]
  936:     12    13    14    15    16    17    18    19
  944:     20    21     0     1     2     3     4     5
  952:      6     7     8     9    10    11    12    13
  960:     14    15    16    17    18    19    20    21
  968:      0     1     2     3     4     5     6     7
  976:      8     9    10    11    12    13    14    15
  984:     16    17    18    19    20    21     0     1
  992:      2     3     4     5     6     7     8     9
 1000:     10    11    12    13    14    15    16    17
 1008:     18    19    20    21     0     1     2     3
 1016:      4     5     6     7     8     9    10    11
 1024:     12    13    14    15    16    17    18    19
 1032:     20    21     0     1     2     3     4     5
 1040:      6     7     8     9    10    11    12    13
 1048:     14    15    16    17    18    19    20    21
 1056:      0     1     2     3     4     5     6     7
 1064:      8     9    10    11    12    13    14    15
 1072:     16    17    18    19    20    21     0     1
 1080:      2     3     4     5     6     7     8     9
 1088:     10    11    12    13    14    15    16    17
 1096:     18    19    20    21     0     1     2     3
 1104:      4     5     6     7     8     9    10    11
 1112:     12    13    14    15    16    17    18    19
 1120:     20    21     0     1     2     3     4     5
 1128:      6     7     8     9    10    11    12    13
 1136:     14    15    16    17    18    19    20    21
 1144:      0     1     2     3     4     5     6     7
 1152:      8     9    10    11    12    13    14    15
 1160:     16    17    18    19    20    21     0     1
 1168:      2     3     4     5     6     7     8     9
 1176:     10    11    12    13    14    15    16    17
 1184:     18    19    20    21     0     1     2     3
 1192:      4     5     6     7     8     9    10    11
 1200:     12    13    14    15    16    17    18    19
 1208:     20    21     0     1     2     3     4     5
 1216:      6     7     8     9    10    11    12    13
 1224:     14    15    16    17    18    19    20    21
 1232:      0     1     2     3     4     5     6     7
 1240:      8     9    10    11    12    13    14    15
 1248:     16    17    18    19    20    21     0     1
 1256:      2     3     4     5     6     7     8     9
 1264:     10    11    12    13    14    15    16    17
 1272:     18    19    20    21     0     1     2     3
 1280:      4     5     6     7     8     9    10    11
 1288:     12    13    14    15    16    17    18    19
 1296:     20    21     0     1     2     3     4     5
 1304:      6     7     8     9    10    11    12    13
 1312:     14    15    16    17    18    19    20    21
 1320:      0     1     2     3     4     5     6     7
 1328:      8     9    10    11    12    13    14    15
 1336:     16    17    18    19    20    21     0     1
 1344:      2     3     4     5     6     7     8     9
 1352:     10    11    12    13    14    15    16    17
 1360:     18    19    20    21     0     1     2     3
 1368:      4     5     6     7     8     9    10    11
 1376:     12    13    14    15    16    17    18    19
 1384:     20    21     0     1     2     3     4     5
 1392:      6     7     8     9    10    11    12    13
 1400:     14    15    16    17    18    19    20    21
 1408:      0     1     2     3     4     5     6     7
 1416:      8     9    10    11    12    13    14    15
 1424:     16    17    18    19    20    21     0     1
 1432:      2     3     4     5     6     7     8     9
 1440:     10    11    12    13    14    15    16    17
 1448:     18    19    20    21     0     1     2     3
 1456:      4     5     6     7     8     9    10    11
 1464:     12    13    14    15    16    17    18    19
 1472:     20    21     0     1     2     3     4     5
 1480:      6     7     8     9    10    11    12    13
 1488:     14    15    16    17    18    19    20    21
 1496:      0     1     2     3     4     5     6     7
 1504:      8     9    10    11    12    13    14    15
 1512:     16    17    18    19    20    21     0     1
 1520:      2     3     4     5     6     7     8     9
 1528:     10    11    12    13    14    15    16    17
 1536:     18    19    20    21     0     1     2     3
 1544:      4     5     6     7     8     9    10    11
 1552:     12    13    14    15    16    17    18    19
 1560:     20    21     0     1     2     3     4     5
 1568:      6     7     8     9    10    11    12    13
 1576:     14    15    16    17    18    19    20    21
 1584:      0     1     2     3     4     5     6     7
 1592:      8     9    10    11    12    13    14    15
 1600:     16    17    18    19    20    21     0     1
 1608:      2     3     4     5     6     7     8     9
 1616:     10    11    12    13    14    15    16    17
 1624:     18    19    20    21     0     1     2     3
 1632:      4     5     6     7     8     9    10    11
 1640:     12    13    14    15    16    17    18    19
 1648:     20    21     0     1     2     3     4     5
 1656:      6     7     8     9    10    11    12    13
 1664:     14    15    16    17    18    19    20    21
 1672:      0     1     2     3     4     5     6     7
 1680:      8     9    10    11    12    13    14    15
 1688:     16    17    18    19    20    21     0     1
 1696:      2     3     4     5     6     7     8     9
 1704:     10    11    12    13    14    15    16    17
 1712:     18    19    20    21     0     1     2     3
 1720:      4     5     6     7     8     9    10    11
 1728:     12    13    14    15    16    17    18    19
 1736:     20    21     0     1     2     3     4     5
 1744:      6     7     8     9    10    11    12    13
 1752:     14    15    16    17    18    19    20    21
 1760:      0     1     2     3     4     5     6     7
 1768:      8     9    10    11    12    13    14    15
 1776:     16    17    18    19    20    21     0     1
 1784:      2     3     4     5     6     7     8     9
 1792:     10    11    12    13    14    15    16    17
 1800:     18    19    20    21     0     1     2     3
 1808:      4     5     6     7     8     9    10    11
 1816:     12    13    14    15    16    17    18    19
 1824:     20    21     0     1     2     3     4     5
 1832:      6     7     8     9    10    11    12    13
 1840:     14    15    16    17    18    19    20    21
 1848:      0     1     2     3     4     5     6     7
 1856:      8     9    10    11    12    13    14    15
 1864:     16    17    18    19    20    21     0     1
 1872:      2     3     4     5     6     7     8     9
 1880:     10    11    12    13    14    15    16    17
 1888:     18    19    20    21     0     1     2     3
 1896:      4     5     6     7     8     9    10    11
 1904:     12    13    14    15    16    17    18    19
 1912:     20    21     0     1     2     3     4     5
 1920:      6     7     8     9    10    11    12    13
 1928:     14    15    16    17    18    19    20    21
 1936:      0     1     2     3     4     5     6     7
 1944:      8     9    10    11    12    13    14    15
 1952:     16    17    18    19    20    21     0     1
 1960:      2     3     4     5     6     7     8     9
 1968:     10    11    12    13    14    15    16    17
 1976:     18    19    20    21     0     1     2     3
 1984:      4     5     6     7     8     9    10    11
 1992:     12    13    14    15    16    17    18    19
 2000:     20    21     0     1     2     3     4     5
 2008:      6     7     8     9    10    11    12    13
 2016:     14    15    16    17    18    19    20    21
 2024:      0     1     2     3     4     5     6     7
 2032:      8     9    10    11    12    13    14    15
 2040:     16    17    18    19    20    21     0     1
RSS hash key:
6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a
RSS hash function:
    toeplitz: on
    xor: off
    crc32: off

Suricata is configured to use the queues to receive and feed its workers:

af-packet:
  - interface: S2
    cluster-id: 95
    cluster-type: cluster_qm
    threads: 22
    defrag: no
    use-mmap: yes
    mmap-locked: yes
    tpacket-v3: yes
    rollover: no
    use-emergency-flush: yes
    ring-size: 200000
    block-size: 4194304

irqbalance is disabled.
However, we only see one queue being filled with packets:

$ cat /proc/interrupts | grep S2
 358:          2   30997486          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080769-edge      ice-S2-TxRx-0
 359:          0          2          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080770-edge      ice-S2-TxRx-1
 360:          0          0          2          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080771-edge      ice-S2-TxRx-2
 361:          0          0          0          2          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080772-edge      ice-S2-TxRx-3
 362:          0          0          0          0          2          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080773-edge      ice-S2-TxRx-4
 363:          0          0          0          0          0          2          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080774-edge      ice-S2-TxRx-5
 364:          0          0          0          0          0          0          2          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080775-edge      ice-S2-TxRx-6
 365:          0          0          0          0          0          0          0          2          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080776-edge      ice-S2-TxRx-7
 366:          0          0          0          0          0          0          0          0          2          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080777-edge      ice-S2-TxRx-8
 367:          0          0          0          0          0          0          0          0          0          2          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080778-edge      ice-S2-TxRx-9
 368:          0          0          0          0          0          0          0          0          0          0          2          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080779-edge      ice-S2-TxRx-10
 369:          0          0          0          0          0          0          0          0          0          0          0          2          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080780-edge      ice-S2-TxRx-11
 370:          0          0          0          0          0          0          0          0          0          0          0          0          2          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080781-edge      ice-S2-TxRx-12
 371:          0          0          0          0          0          0          0          0          0          0          0          0          0          2          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080782-edge      ice-S2-TxRx-13
 372:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          2          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080783-edge      ice-S2-TxRx-14
 373:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          2          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080784-edge      ice-S2-TxRx-15
 374:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          2          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080785-edge      ice-S2-TxRx-16
 375:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          2          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080786-edge      ice-S2-TxRx-17
 376:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          2          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080787-edge      ice-S2-TxRx-18
 377:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          2          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080788-edge      ice-S2-TxRx-19
 378:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          2          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080789-edge      ice-S2-TxRx-20
 379:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          2          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI 34080790-edge      ice-S2-TxRx-21


$ sudo ethtool -S S2
NIC statistics:
     rx_unicast: 9620668884
     tx_unicast: 0
     rx_multicast: 0
     tx_multicast: 0
     rx_broadcast: 0
     tx_broadcast: 0
     rx_bytes: 4740939939827
     tx_bytes: 0
     rx_dropped: 2423536194
     rx_unknown_protocol: 0
     rx_alloc_fail: 0
     rx_pg_alloc_fail: 0
     rx_gro_dropped: 0
     tx_errors: 0
     tx_linearize: 0
     tx_busy: 0
     tx_restart: 0
     tx_queue_0_packets: 0
     tx_queue_0_bytes: 0
     tx_queue_1_packets: 0
     tx_queue_1_bytes: 0
     tx_queue_2_packets: 0
     tx_queue_2_bytes: 0
     tx_queue_3_packets: 0
     tx_queue_3_bytes: 0
     tx_queue_4_packets: 0
     tx_queue_4_bytes: 0
     tx_queue_5_packets: 0
     tx_queue_5_bytes: 0
     tx_queue_6_packets: 0
     tx_queue_6_bytes: 0
     tx_queue_7_packets: 0
     tx_queue_7_bytes: 0
     tx_queue_8_packets: 0
     tx_queue_8_bytes: 0
     tx_queue_9_packets: 0
     tx_queue_9_bytes: 0
     tx_queue_10_packets: 0
     tx_queue_10_bytes: 0
     tx_queue_11_packets: 0
     tx_queue_11_bytes: 0
     tx_queue_12_packets: 0
     tx_queue_12_bytes: 0
     tx_queue_13_packets: 0
     tx_queue_13_bytes: 0
     tx_queue_14_packets: 0
     tx_queue_14_bytes: 0
     tx_queue_15_packets: 0
     tx_queue_15_bytes: 0
     tx_queue_16_packets: 0
     tx_queue_16_bytes: 0
     tx_queue_17_packets: 0
     tx_queue_17_bytes: 0
     tx_queue_18_packets: 0
     tx_queue_18_bytes: 0
     tx_queue_19_packets: 0
     tx_queue_19_bytes: 0
     tx_queue_20_packets: 0
     tx_queue_20_bytes: 0
     tx_queue_21_packets: 0
     tx_queue_21_bytes: 0
     rx_queue_0_packets: 7197132596
     rx_queue_0_bytes: 3520334819668
     rx_queue_1_packets: 0
     rx_queue_1_bytes: 0
     rx_queue_2_packets: 0
     rx_queue_2_bytes: 0
     rx_queue_3_packets: 0
     rx_queue_3_bytes: 0
     rx_queue_4_packets: 0
     rx_queue_4_bytes: 0
     rx_queue_5_packets: 0
     rx_queue_5_bytes: 0
     rx_queue_6_packets: 0
     rx_queue_6_bytes: 0
     rx_queue_7_packets: 0
     rx_queue_7_bytes: 0
     rx_queue_8_packets: 0
     rx_queue_8_bytes: 0
     rx_queue_9_packets: 0
     rx_queue_9_bytes: 0
     rx_queue_10_packets: 0
     rx_queue_10_bytes: 0
     rx_queue_11_packets: 0
     rx_queue_11_bytes: 0
     rx_queue_12_packets: 0
     rx_queue_12_bytes: 0
     rx_queue_13_packets: 0
     rx_queue_13_bytes: 0
     rx_queue_14_packets: 0
     rx_queue_14_bytes: 0
     rx_queue_15_packets: 0
     rx_queue_15_bytes: 0
     rx_queue_16_packets: 0
     rx_queue_16_bytes: 0
     rx_queue_17_packets: 0
     rx_queue_17_bytes: 0
     rx_queue_18_packets: 0
     rx_queue_18_bytes: 0
     rx_queue_19_packets: 0
     rx_queue_19_bytes: 0
     rx_queue_20_packets: 0
     rx_queue_20_bytes: 0
     rx_queue_21_packets: 0
     rx_queue_21_bytes: 0
     rx_bytes.nic: 9046705927373
     tx_bytes.nic: 0
     rx_unicast.nic: 18393137976
     tx_unicast.nic: 0
     rx_multicast.nic: 0
     tx_multicast.nic: 0
     rx_broadcast.nic: 0
     tx_broadcast.nic: 0
     tx_errors.nic: 0
     tx_timeout.nic: 0
     rx_size_64.nic: 0
     tx_size_64.nic: 0
     rx_size_127.nic: 10744343323
     tx_size_127.nic: 0
     rx_size_255.nic: 1322854237
     tx_size_255.nic: 0
     rx_size_511.nic: 579743433
     tx_size_511.nic: 0
     rx_size_1023.nic: 952646762
     tx_size_1023.nic: 0
     rx_size_1522.nic: 2552097960
     tx_size_1522.nic: 0
     rx_size_big.nic: 2241452267
     tx_size_big.nic: 0
     link_xon_rx.nic: 0
     link_xon_tx.nic: 0
     link_xoff_rx.nic: 0
     link_xoff_tx.nic: 0
     tx_dropped_link_down.nic: 0
     rx_undersize.nic: 0
     rx_fragments.nic: 0
     rx_oversize.nic: 0
     rx_jabber.nic: 0
     rx_csum_bad.nic: 0
     rx_length_errors.nic: 0
     rx_dropped.nic: 158
     rx_crc_errors.nic: 17246
     illegal_bytes.nic: 24
     mac_local_faults.nic: 0
     mac_remote_faults.nic: 0
     fdir_sb_match.nic: 0
     fdir_sb_status.nic: 1
     tx_priority_0_xon.nic: 0
     tx_priority_0_xoff.nic: 0
     tx_priority_1_xon.nic: 0
     tx_priority_1_xoff.nic: 0
     tx_priority_2_xon.nic: 0
     tx_priority_2_xoff.nic: 0
     tx_priority_3_xon.nic: 0
     tx_priority_3_xoff.nic: 0
     tx_priority_4_xon.nic: 0
     tx_priority_4_xoff.nic: 0
     tx_priority_5_xon.nic: 0
     tx_priority_5_xoff.nic: 0
     tx_priority_6_xon.nic: 0
     tx_priority_6_xoff.nic: 0
     tx_priority_7_xon.nic: 0
     tx_priority_7_xoff.nic: 0
     rx_priority_0_xon.nic: 0
     rx_priority_0_xoff.nic: 0
     rx_priority_1_xon.nic: 0
     rx_priority_1_xoff.nic: 0
     rx_priority_2_xon.nic: 0
     rx_priority_2_xoff.nic: 0
     rx_priority_3_xon.nic: 0
     rx_priority_3_xoff.nic: 0
     rx_priority_4_xon.nic: 0
     rx_priority_4_xoff.nic: 0
     rx_priority_5_xon.nic: 0
     rx_priority_5_xoff.nic: 0
     rx_priority_6_xon.nic: 0
     rx_priority_6_xoff.nic: 0
     rx_priority_7_xon.nic: 0
     rx_priority_7_xoff.nic: 0

Any ideas?

2

Can you post the output if you just run this? Just to ensure the IRQ Affinity was properly set

Does it also happen with cluster_flow?

3 
$ sudo /usr/local/bin/set_irq_affinity 1-22 S2
IFACE CORE MASK -> FILE
=======================
S2 1 2 -> /proc/irq/300/smp_affinity

Yes with cluster_flow as well

4

Is this the full output? It shouldn’t be just one line.

It should look more like this:

/usr/bin/set_irq_affinity 1-22 eth6
IFACE CORE MASK -> FILE
=======================
eth6 1 2 -> /proc/irq/1397/smp_affinity
eth6 2 4 -> /proc/irq/1398/smp_affinity
eth6 3 8 -> /proc/irq/1399/smp_affinity
eth6 4 10 -> /proc/irq/1400/smp_affinity
eth6 5 20 -> /proc/irq/1401/smp_affinity
eth6 6 40 -> /proc/irq/1402/smp_affinity
eth6 7 80 -> /proc/irq/1403/smp_affinity
eth6 8 100 -> /proc/irq/1404/smp_affinity
eth6 9 200 -> /proc/irq/1405/smp_affinity
eth6 10 400 -> /proc/irq/1406/smp_affinity
eth6 11 800 -> /proc/irq/1407/smp_affinity
eth6 12 1000 -> /proc/irq/1408/smp_affinity
eth6 13 2000 -> /proc/irq/1409/smp_affinity
eth6 14 4000 -> /proc/irq/1410/smp_affinity
eth6 15 8000 -> /proc/irq/1411/smp_affinity
eth6 16 10000 -> /proc/irq/1412/smp_affinity
eth6 17 20000 -> /proc/irq/1413/smp_affinity
eth6 18 40000 -> /proc/irq/1414/smp_affinity
eth6 19 80000 -> /proc/irq/1415/smp_affinity
eth6 20 100000 -> /proc/irq/1416/smp_affinity
eth6 21 200000 -> /proc/irq/1417/smp_affinity
eth6 22 400000 -> /proc/irq/1418/smp_affinity
5

ah we had an

post-up ethtool -L S2 combined 1

in there, because of using cluster_flow … without the output is now

/usr/local/bin/set_irq_affinity 1-22 S2
IFACE CORE MASK -> FILE
=======================
S2 1 2 -> /proc/irq/300/smp_affinity
S2 2 4 -> /proc/irq/301/smp_affinity
S2 3 8 -> /proc/irq/302/smp_affinity
S2 4 10 -> /proc/irq/303/smp_affinity
S2 5 20 -> /proc/irq/304/smp_affinity
S2 6 40 -> /proc/irq/305/smp_affinity
S2 7 80 -> /proc/irq/306/smp_affinity
S2 8 100 -> /proc/irq/307/smp_affinity
S2 9 200 -> /proc/irq/308/smp_affinity
S2 10 400 -> /proc/irq/309/smp_affinity
S2 11 800 -> /proc/irq/310/smp_affinity
S2 12 1000 -> /proc/irq/311/smp_affinity
S2 13 2000 -> /proc/irq/312/smp_affinity
S2 14 4000 -> /proc/irq/313/smp_affinity
S2 15 8000 -> /proc/irq/314/smp_affinity
S2 16 10000 -> /proc/irq/315/smp_affinity
S2 17 20000 -> /proc/irq/316/smp_affinity
S2 18 40000 -> /proc/irq/317/smp_affinity
S2 19 80000 -> /proc/irq/318/smp_affinity
S2 20 100000 -> /proc/irq/319/smp_affinity
S2 21 200000 -> /proc/irq/320/smp_affinity
S2 22 400000 -> /proc/irq/321/smp_affinity

still only one queue filled

6

I will try to reproduce it on the E810 setup I have. Can you add also the lscpu output with the NUMA node infos and also the affinity section in the suricata.yaml?

7 
lscpu
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Byte Order:                         Little Endian
Address sizes:                      52 bits physical, 57 bits virtual
CPU(s):                             48
On-line CPU(s) list:                0-47
Thread(s) per core:                 2
Core(s) per socket:                 24
Socket(s):                          1
NUMA node(s):                       1
Vendor ID:                          AuthenticAMD
CPU family:                         25
Model:                              17
Model name:                         AMD EPYC 9224 24-Core Processor
Stepping:                           1
Frequency boost:                    enabled
CPU MHz:                            3670.537
CPU max MHz:                        3706.0540
CPU min MHz:                        1500.0000
BogoMIPS:                           4992.39
Virtualization:                     AMD-V
L1d cache:                          768 KiB
L1i cache:                          768 KiB
L2 cache:                           24 MiB
L3 cache:                           64 MiB
NUMA node0 CPU(s):                  0-47
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Mitigation; safe RET
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr
                                    _opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma 
                                    cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a m
                                    isalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3
                                     invpcid_single hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx5
                                    12dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cq
                                    m_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock
                                     nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif avx512vbmi umip
                                     pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca fs
                                    rm flush_l1d
8 
%YAML 1.1
---

threading:
  set-cpu-affinity: yes
  cpu-affinity:
    - management-cpu-set:
        cpu: [ "1-7" ]
    - receive-cpu-set:
       cpu: [ 0 ]
    - worker-cpu-set:
        cpu: [ "8-47" ]
        mode: "exclusive"
        prio:
          high: [ "8-47" ]
          default: "high"
9

The worker cpu range should match the one that you pass to the affinity script so “1-22” and the amount of threads defined in the af-packet section. So try to adjust the “8-47” to “1-22” and ideally post the suricata.log afterwards as well to make sure it’s properly applied.

10

changed it still same behavior with only one queue beeing filled

%YAML 1.1
---

threading:
  set-cpu-affinity: yes
  cpu-affinity:
    - management-cpu-set:
        cpu: [ "1-7" ]
    - receive-cpu-set:
       cpu: [ 0 ]
    - worker-cpu-set:
        cpu: [ "1-22" ]
        mode: "exclusive"
        prio:
          high: [ "1-22" ]
          default: "high"

28/6/2024 -- 13:53:20 - <Notice> - This is Suricata version 6.0.19 RELEASE running in SYSTEM mode
28/6/2024 -- 13:53:20 - <Info> - CPUs/cores online: 48
28/6/2024 -- 13:53:20 - <Info> - Setting engine mode to IDS mode by default
28/6/2024 -- 13:53:20 - <Info> - HTTP memcap: 4294967296
28/6/2024 -- 13:53:20 - <Info> - fast output device (regular) initialized: fast.log
28/6/2024 -- 13:53:20 - <Info> - Setting logging socket of non-blocking in live mode.
28/6/2024 -- 13:53:20 - <Info> - eve-log output device (unix_stream) initialized: /tmp/stats.sock
28/6/2024 -- 13:53:20 - <Info> - Setting logging socket of non-blocking in live mode.
28/6/2024 -- 13:53:20 - <Info> - eve-log output device (unix_stream) initialized: /tmp/suri.sock
28/6/2024 -- 13:53:20 - <Info> - DNP3 log sub-module initialized.
28/6/2024 -- 13:53:20 - <Info> - DNP3 log sub-module initialized.
28/6/2024 -- 13:53:20 - <Info> - stats output device (regular) initialized: stats.log
28/6/2024 -- 13:53:28 - <Info> - 2 rule files processed. 52405 rules successfully loaded, 0 rules failed
28/6/2024 -- 13:53:28 - <Info> - Threshold config parsed: 0 rule(s) found
28/6/2024 -- 13:53:30 - <Info> - 52435 signatures processed. 2 are IP-only rules, 8641 are inspecting packet payload, 43745 inspect application layer, 0 are decoder event only
28/6/2024 -- 13:53:50 - <Info> - Going to use 22 thread(s)
28/6/2024 -- 13:53:52 - <Info> - Using unix socket file '/var/run/suricata-command.socket'
28/6/2024 -- 13:53:52 - <Notice> - all 22 packet processing threads, 6 management threads initialized, engine started.
28/6/2024 -- 13:53:53 - <Info> - All AFP capture threads are running.
11

In this context, how are these comments in suricata.yaml to be understood?
This really means, worker-cpu should NOT set in NIC affinity config?

1555   #
1556   cpu-affinity:
1557     - management-cpu-set:
1558         cpu: [ 0 ]  # include only these CPUs in affinity settings
1559     - receive-cpu-set:
1560         cpu: [ 0 ]  # include only these CPUs in affinity settings
1561     - worker-cpu-set:
1562         cpu: [ "all" ]
1563         mode: "exclusive"
12

FTR: This was eventually addressed by removing mislabeled IEEE 802.1ad (QinQ) headers in which a wrong ethertype was specified in the outer layer. Such packets seemed to break the NIC’s hashing capabilities and resulted in one queue being used.