Suricata 7.0.1 dpdk mode with memif vdev, does it support receive packets from multiple RSS queues?

1

Hi, Lukas and others

Several days ago, I have tried successfully suricata dpdk ips mode with libmemif. Now I have a new question: when config dpdk mode with 3 threads, then use vpp packet generator to simulate packets to memif interface.
From suricata log, only one rx thread received packets and other threads show 0 as below:
Perf: dpdk: net_memif1: total RX stats: packets 0 bytes: 0 missed: 0 errors: 0 nombufs: 0 [ReceiveDPDKThreadExitStats:source-dpdk.c:639]
Perf: dpdk: net_memif1: total TX stats: packets 413836372 bytes: 26485527808 errors: 0 [ReceiveDPDKThreadExitStats:source-dpdk.c:644]
Perf: dpdk: (W#01-net_memif1) received packets 0 [ReceiveDPDKThreadExitStats:source-dpdk.c:649]
Perf: dpdk: (W#02-net_memif1) received packets 0 [ReceiveDPDKThreadExitStats:source-dpdk.c:649]
Perf: dpdk: (W#03-net_memif1) received packets 0 [ReceiveDPDKThreadExitStats:source-dpdk.c:649]
Perf: dpdk: net_memif0: total RX stats: packets 0 bytes: 0 missed: 0 errors: 0 nombufs: 0 [ReceiveDPDKThreadExitStats:source-dpdk.c:639]
Perf: dpdk: net_memif0: total TX stats: packets 0 bytes: 0 errors: 0 [ReceiveDPDKThreadExitStats:source-dpdk.c:644]
Perf: dpdk: (W#01-net_memif0) received packets 413839776 [ReceiveDPDKThreadExitStats:source-dpdk.c:649]
Perf: dpdk: (W#02-net_memif0) received packets 0 [ReceiveDPDKThreadExitStats:source-dpdk.c:649]
Perf: dpdk: (W#03-net_memif0) received packets 0 [ReceiveDPDKThreadExitStats:source-dpdk.c:649]
Info: counters: Alerts: 0 [StatsLogSummary:counters.c:878]

And in the log, also shows:
Config: dpdk: net_memif0: RSS not supported [DeviceInitPortConf:runmode-dpdk.c:1143]

In my test evn, one rx thread can process 2.39Mpps indicated in vpp terminal. Can suricata dpdk mode with memif vdev support multiple RSS queues? If so, we can improve total PPS through more threads setting.

Thanks

Andy

2

I haven’t worked with either VPP or memif PMD much…
But from the docs, it seems like memif in DPDK does not support RSS natively.
In the DPDK docs, testpmd is only shown with one queue - https://doc.dpdk.org/guides/nics/memif.html

In VPP docs they refer to “TenGigabitEthernet” which sounds like a virtual driver name. Could you possibly explore that?
https://s3-docs.fd.io/vpp/22.02/developer/corearchitecture/multi_thread.html

It is always a good idea to test if dpdk-testpmd works with the desired feature (e.g. configure RSS on memif PMD). testpmd command then would be something like:

#./dpdk-testpmd -l 0,1,2,3 --vdev=net_memif,socket=/run/vpp/memif.sock -- -i --rxq=3 --txq=3

The last resort could be implementing autofp mode to support DPDK capture mode.

@Alexander sorry for tagging but did you have luck to setup multithreaded Suricata?

3

Lukas, thanks. I will try the methods that you suggested.

4

After run: dpdk-testpmd -l 1-2 --vdev=net_memif,role=slave,id=0,socket-abstract=no,soc ket=/run/vpp/memif.sock – -i --rxq=3 --txq=3
testpmd> show config rxtx
io packet forwarding packets/burst=32
nb forwarding cores=1 - nb forwarding ports=1
port 0: RX queue number: 3 Tx queue number: 3
Rx offloads=0x0 Tx offloads=0x0
RX queue: 0
RX desc=0 - RX free threshold=0
RX threshold registers: pthresh=0 hthresh=0 wthresh=0
RX Offloads=0x0
TX queue: 0
TX desc=0 - TX free threshold=0
TX threshold registers: pthresh=0 hthresh=0 wthresh=0
TX offloads=0x0 - TX RS bit threshold=0
From the info: RX queue number is 3 but only have RX queue 0, TX is the same.
Top command result:
1710 root 20 0 64.1g 125120 71224 S 100.0 1.5 0:25.77 dpdk-testpmd

Seems testpmd with memif can’t use multiple queues.

5

From Example setup — The Vector Packet Processor v22.02-0-g7911f29c5 documentation
DBGvpp# create memif id 0 slave rx-queues 2 tx-queues 2
DBGvpp# set int state memif0/0 up
DBGvpp# set int ip address memif0/0 192.168.1.1/24

Seems that memif support multiple rx,tx queues. So I configured vpp and testpmd as below:
memif with 2 rx/tx queues:
vpp config:
create interface memif id 0 master rx-queues 2 tx-queues 2
create interface memif id 1 master rx-queues 2 tx-queues 2
set int state memif0/0 up
set int state memif0/1 up
create packet-generator interface pg0
set int state pg0 up
create packet-generator interface pg1
set int state pg1 up

set int l2 xconn pg0 memif0/0
set int l2 xconn memif0/0 pg0
set int l2 xconn pg1 memif0/1
set int l2 xconn memif0/1 pg1

*packet-generator new { *

  • name memif *
  • limit -1 *
  • node ethernet-input *
  • size 64-64 *
  • interface pg0 *
  • worker 2 *
  • data { *
  • IP4: 42:01:0a:00:00:0e → 02:fe:4b:6e:4d:c4 *
  • UDP: 172.16.2.4 → 172.16.0.24 *
  • UDP: 1234 → 1234 *
  •  length 30 checksum 0 incrementing 1       \*
  • } *
    }

dpdk-testpmd -l 5-7 --vdev=net_memif,role=slave,id=0,socket-abstract=no,socket=/run/vpp/memif.sock – -i --rxq=2 --txq=2

vpp# sh memif
sockets
id listener filename
0 yes (2) /run/vpp/memif.sock

interface memif0/0
remote-name “DPDK 22.11.2”
remote-interface “net_memif”
socket-id 0 id 0 mode ethernet
flags admin-up connected
listener-fd 19 conn-fd 18
num-s2m-rings 2 num-m2s-rings 2 buffer-size 0 num-regions 1
region 0 size 8454656 fd 20
master-to-slave ring 0:
region 0 offset 33024 ring-size 1024 int-fd 23
head 0 tail 0 flags 0x0001 interrupts 0
master-to-slave ring 1:
region 0 offset 49536 ring-size 1024 int-fd 24
head 0 tail 0 flags 0x0001 interrupts 0
slave-to-master ring 0:
region 0 offset 0 ring-size 1024 int-fd 21
head 0 tail 0 flags 0x0001 interrupts 0
slave-to-master ring 1:
region 0 offset 16512 ring-size 1024 int-fd 22
head 0 tail 0 flags 0x0001 interrupts 0
interface memif0/1
socket-id 0 id 1 mode ethernet
flags admin-up
listener-fd 19 conn-fd 0
num-s2m-rings 0 num-m2s-rings 0 buffer-size 0 num-regions 0

vpp# monitor int memif0/0
rx: 14.11Mpps 7.22Gbps tx: 24.61Mpps 12.59Gbps
rx: 14.12Mpps 7.23Gbps tx: 24.58Mpps 12.59Gbps

I am still not sure that queue 0 and queue 1 are both transmitting packets.

6

Hi Andy,

you could check stats of the testpmd interfaces. Stats should be printed as you exit the testpmd app or interactively with show port stats all

7

root@debian:~# dpdk-testpmd -l 3-7 -n 4 --vdev=net_memif,role=slave,id=0,socket-abstract=no,socket=/run/vpp/memif.sock – --nb-cores=4 --rxq=2 --txq=2 -i
EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode ‘VA’
memif_set_role(): Role argument “slave” is deprecated, use “client”
Interactive-mode selected
Warning: NUMA should be configured manually by using --port-numa-config and --ring-numa-config parameters along with --numa.
testpmd: create a new mbuf pool <mb_pool_0>: n=179456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc

Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.

Configuring Port 0 (socket 0)
Port 0: 36:8B:0F:52:34:E5
Checking link statuses…
Done
testpmd> set corelist 4,5,6,7
testpmd> start
io packet forwarding - ports=1 - cores=2 - streams=2 - NUMA support enabled, MP allocation mode: native
Logical Core 4 (socket 0) forwards packets on 1 streams:
*** RX P=0/Q=0 (socket 0) → TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00***
Logical Core 5 (socket 0) forwards packets on 1 streams:
*** RX P=0/Q=1 (socket 0) → TX P=0/Q=1 (socket 0) peer=02:00:00:00:00:00***

  • io packet forwarding packets/burst=32*
  • nb forwarding cores=4 - nb forwarding ports=1*
  • port 0: RX queue number: 2 Tx queue number: 2*
  • Rx offloads=0x0 Tx offloads=0x0*
  • RX queue: 0*
  •  RX desc=0 - RX free threshold=0*

  •  RX threshold registers: pthresh=0 hthresh=0  wthresh=0*

  •  RX Offloads=0x0*
  • TX queue: 0*
  •  TX desc=0 - TX free threshold=0*

  •  TX threshold registers: pthresh=0 hthresh=0  wthresh=0*

  •  TX offloads=0x0 - TX RS bit threshold=0*

testpmd> show port stats all

  • ######################## NIC statistics for port 0 ########################*

  • RX-packets: 1797477024 RX-missed: 0 RX-bytes: 115038531584*

  • RX-errors: 0*

  • RX-nombuf: 0*

  • TX-packets: 1797242784 TX-errors: 0 TX-bytes: 115023538176*

  • Throughput (since last show)*

  • Rx-pps: 0 Rx-bps: 0*

  • Tx-pps: 0 Tx-bps: 0*

  • ############################################################################*
    testpmd> stop
    Telling cores to stop…
    Waiting for lcores to finish…

  • ------- Forward Stats for RX Port= 0/Queue= 1 → TX Port= 0/Queue= 1 -------*

  • RX-packets: 1919570400 TX-packets: 1919390848 TX-dropped: 179552*

  • ---------------------- Forward statistics for port 0 ----------------------*

  • RX-packets: 1919570400 RX-dropped: 0 RX-total: 1919570400*

  • TX-packets: 1919390848 TX-dropped: 179552 TX-total: 1919570400*

  • ----------------------------------------------------------------------------*

  • +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++*

  • RX-packets: 1919570400 RX-dropped: 0 RX-total: 1919570400*

  • TX-packets: 1919390848 TX-dropped: 179552 TX-total: 1919570400*

  • ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*

Done.
testpmd>

From vpp side:
vpp# sh memif
sockets

  • id listener filename*
  • 0 yes (2) /run/vpp/memif.sock*

interface memif0/0

  • remote-name “DPDK 22.11.2”*
  • remote-interface “net_memif”*
  • socket-id 0 id 0 mode ethernet*
  • flags admin-up connected*
  • listener-fd 19 conn-fd 20*
  • num-s2m-rings 2 num-m2s-rings 2 buffer-size 0 num-regions 1*
  • region 0 size 8454656 fd 21*
  • master-to-slave ring 0:*
  •  region 0 offset 33024 ring-size 1024 int-fd 24*

  •  head 1024 tail 0 flags 0x0001 interrupts 0*
  • master-to-slave ring 1:*
  •  region 0 offset 49536 ring-size 1024 int-fd 25*

  •  head 27104 tail 27104 flags 0x0001 interrupts 2*
  • slave-to-master ring 0:*
  •  region 0 offset 0 ring-size 1024 int-fd 22*

  •  head 0 tail 0 flags 0x0001 interrupts 0*
  • slave-to-master ring 1:*
  •  region 0 offset 16512 ring-size 1024 int-fd 23*

  •  head 38016 tail 38016 flags 0x0001 interrupts 0*

interface memif0/1

  • socket-id 0 id 1 mode ethernet*
  • flags admin-up*
  • listener-fd 19 conn-fd 0*
  • num-s2m-rings 1 num-m2s-rings 1 buffer-size 0 num-regions 0*
  • local-disc-reason “disconnected”*

So from vpp side, two rings were used, I guess here ring means “queue”

From testpmd side, two queues are connected in"start" log. But from “stop” log, only Queue= 1 transmitted actual packets.

8

I used testpmd as memif master and suricata as memif client, 3 queues can work and show packets on every queue.

testpmd side:
dpdk-testpmd -l 4-7 --vdev=net_memif,role=server,id=0,socket-abstract=no,socket=/run/vpp/memif.sock – -i --rxq=3 --txq=3
testpmd>
set corelist 5,6,7

set fwd flowgen # flow packet generator
port stop all
port start all
start
show port stats all
stop
testpmd> stop
Telling cores to stop…
Waiting for lcores to finish…

------- Forward Stats for RX Port= 0/Queue= 0 → TX Port= 0/Queue= 0 -------

  • RX-packets: 0 TX-packets: 84606400 TX-dropped: 1552001376*

  • ------- Forward Stats for RX Port= 0/Queue= 1 → TX Port= 0/Queue= 1 -------*

  • RX-packets: 0 TX-packets: 84607936 TX-dropped: 1546333120*

  • ------- Forward Stats for RX Port= 0/Queue= 2 → TX Port= 0/Queue= 2 -------*

  • RX-packets: 0 TX-packets: 84609216 TX-dropped: 1546005664*

---------------------- Forward statistics for port 0 ----------------------
RX-packets: 0 RX-dropped: 0 RX-total: 0
TX-packets: 253823552 TX-dropped: 4644340160 TX-total: 4898163712

+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
RX-packets: 0 RX-dropped: 0 RX-total: 0
TX-packets: 253823552 TX-dropped: 4644340160 TX-total: 4898163712
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

suricata side:
dpdk:
eal-params:
proc-type: primary
file-prefix: suricata
vdev: [“net_memif0,role=slave,id=0,socket-abstract=no,socket=/run/vpp/memif.sock”]

interfaces:
- interface: net_memif0
checksum-checks: true
checksum-checks-offload: true
copy-iface: none
copy-mode: ids
mempool-cache-size: 512
mempool-size: 65535
mtu: 1500
multicast: true
promisc: true
rss-hash-functions: auto
rx-descriptors: 1024
socket-id: 0
tx-descriptors: 1024
threads: 3

Info: suricata: time elapsed 115.568s [SCPrintElapsedTime:suricata.c:1168]
Perf: flow-manager: 1024 flows processed [FlowRecycler:flow-manager.c:1123]
Perf: dpdk: Port 0 (net_memif0) - rx_good_packets: 332528032 [PrintDPDKPortXstats:source-dpdk.c:612]
Perf: dpdk: Port 0 (net_memif0) - rx_good_bytes: 19951681920 [PrintDPDKPortXstats:source-dpdk.c:612]
Perf: dpdk: Port 0 (net_memif0) - rx_q0_packets: 110840320 [PrintDPDKPortXstats:source-dpdk.c:612]
Perf: dpdk: Port 0 (net_memif0) - rx_q0_bytes: 6650419200 [PrintDPDKPortXstats:source-dpdk.c:612]
Perf: dpdk: Port 0 (net_memif0) - rx_q1_packets: 110843104 [PrintDPDKPortXstats:source-dpdk.c:612]
Perf: dpdk: Port 0 (net_memif0) - rx_q1_bytes: 6650586240 [PrintDPDKPortXstats:source-dpdk.c:612]
Perf: dpdk: Port 0 (net_memif0) - rx_q2_packets: 110844608 [PrintDPDKPortXstats:source-dpdk.c:612]
Perf: dpdk: Port 0 (net_memif0) - rx_q2_bytes: 6650676480 [PrintDPDKPortXstats:source-dpdk.c:612]
Perf: dpdk: net_memif0: total RX stats: packets 332528032 bytes: 19951681920 missed: 0 errors: 0 nombufs: 0 [ReceiveDPDKThreadExitStats:source-dpdk.c:639]
Perf: dpdk: (W#01-net_memif0) received packets 110840320 [ReceiveDPDKThreadExitStats:source-dpdk.c:649]
Perf: dpdk: (W#02-net_memif0) received packets 110843104 [ReceiveDPDKThreadExitStats:source-dpdk.c:649]
Perf: dpdk: (W#03-net_memif0) received packets 110844608 [ReceiveDPDKThreadExitStats:source-dpdk.c:649]
Info: counters: Alerts: 0 [StatsLogSummary:counters.c:878]
Perf: ippair: ippair memory usage: 414144 bytes, maximum: 16777216 [IPPairPrintStats:ippair.c:296]
Perf: host: host memory usage: 398144 bytes, maximum: 33554432 [HostPrintStats:host.c:299]
Perf: dpdk: net_memif0: closing device [DPDKCloseDevice:util-dpdk.c:51]
Notice: device: net_memif0: packets: 332528032, drops: 0 (0.00%), invalid chksum: 0 [LiveDeviceListClean:util-device.c:325]

9

Hey Andy,

as far as I understand - multi-queue setup works with memif interface (both in other DPDK apps and in Suricata), though, you need to ensure to distribute/generate packets to those individual queues because the memif interface itself will not distribute it. Am I correct?

10

Lukas,

Yes, you are right.