DPDK interface bandwith

Help
1

Hello,
How to determine bandwith usage on an networkinterface without IP configuration (span data) when the interface is bind to dpdk? Before dpdk I installed Suricata with pfring and could see bandwith usage with pfcount.
Not realy a specifig suricata question, but I need to interpret some suricata stats in comparison to ao the actual bandwith.
Thanks!
Andre

2

Hi,

to be fair, I am not sure. After googling a bit, I’ve found FloWatcher-DPDK but that is likely outdated.
Paper: https://nonsns.github.io/paper/rossi19tnsm.pdf
Github: GitHub - ztz1989/FloWatcher-DPDK: In the repository, we present FloWatcher-DPDK, a lightweight software traffic monitor based on Intel(R) DPDK

But I would encourage you to join e.g. DPDK users mailing list and ask there. Maybe some built-in application has it already available.

I do not need it often but when I did need something like this, I think I just rebound the interface to the kernel driver and used some kernel application from which I could assume the real bandwidth.

3

Lukas shouldn’t Suricata itself be able to produce this data (based on dpdk stats)?

4

I mean, data-wise, yes.
It can count the number of packets/bytes so yes but I am aware of stats that would allow user to easily interpret those data in terms of bandwidth (I assume the unit would be e.g. Gbps).

Feeding the data to some kind of consumer service (ELK, scripts, Zabbix etc.) could work as illustrated in 9.6. Statistics — Suricata 6.0.0 documentation but I thought the question was more about the command-line tool.

5

Correct, no stats from Suricata, but plain from the command line regarding DPDK. Seen there are more applications which could use DPDK, my mindset is I want to know wat DPDK sees as actual bandwith without any interpretation from an app depending in DPDK.

6

I’ve thought of a very rough estimate by using dpdk-testpmd application.

You could run dpdk-testpmd (bundled with DPDK), in rxonly forward mode and in interactive mode and look at the stats from the interface.

However timing-wise, you probably would need to measure it yourself. So listing stats of the port in the beginning and then listing the stats again after e.g. 10 seconds. Then do the math to get average bandwidth from number of seen bytes in that inverval / 10 seconds

testpmd commands that could help:
running the testpmd app:
sudo dpdk-testpmd -a 0000:3b:00.1 -l0,2 -- -i --forward-mode=rxonly

testpmd interactive commands:

start - start receiving pkts
show port (x)stats 0
stop
quit
7

Thanks Lukas, also for the presentation yesterday regarding suricata and dpdk! At the end you mentioned some stats to look for and how to tackle them, like capture.dpdk.ierrors . Do you maybe have a more comprehensive presentation or document regarding this subject?

8

Unfortunately, not at this moment. But maybe this will help you out.

The structure for (simple) stats only contains imissed, ierrors and no_mbufs as shown during the webinar.
The structure is here: https://doc.dpdk.org/api/structrte__eth__stats.html

Essentially:

  • imissed - when physical RX queues of the NIC are full - this means that Suricata does not poll packets of the queues fast enough.
  • no_mbufs - when the NIC does not have available mbufs (DPDK term for let’s say packet objects/descriptors) in the mempool. So physical NIC queues still have space for packets but the NIC cannot get an object which would hold the newly received packet.
  • ierrors - aggregate error counter of all errors that can happen within the NIC

Now what are counter-measures, when you see some of these counters increasing:

  • imissed - maybe you can try to increase mempool to overcome sudden packet bursts or you are hitting your performance limit
  • no_mbufs - definitely increase mempool size
  • ierrors - if it is the same as imissed/no_mbufs counter, then it doesn’t tell you much, but if it is different then some other error might be happening - then head to xstats (explained below) and possibly inspect further. Examples of extra errors might be CRC errors, length errors - undersized or oversized (MTU) packets. These other error counters might uncover some errors with the traffic that you are trying to inspect.

Bear in mind that DPDK is very close to the hardware and stats might be represented differently on different NICs - you can see this not only from xstats (explained below) but also from stats structure itself. E.g. after this patch from 2016 imissed counters should not be counted in ierrors counters, yet, as you’ve seen during the webinar, they still are there for mlx5 driver. So don’t panic when you don’t see those there :slight_smile:
Another stats-related thing that varies across implementations of the NIC drivers → per-queue stats (which you might’ve noticed in rte_stats structure). For this reason, Suricata does not process these stats by default and the statistics can be obtained from the extended stats (xstats) if available.

Now let’s talk a bit about xstats - extended NIC statistics. As shown in the webinar (1:09:16), these are enabled with verbose logging in Suricata and are printed out at the end of the Suricata log. They may give you a more in-depth view of what your NIC sees. For instance, some drivers can do a histogram of received packet sizes, types of packets (unicast/broadcast) or it may inform you of errors that happened. However, as the previous paragraph states, they can be different from driver to driver. They often rely on hardware counters so some drivers might contain more xstats than others. I think the names of individual errors are often self-explanatory, or a little bit of Googling may help. But generally speaking, if you are experiencing some errors when receiving packets and those errors are not related to imissed/no_mbufs counters, then this might be an interesting place to look into.
While most error counter names are a bit NIC/driver dependent here are the basic abbreviations often used within xstats:

  • phy - stands for physical, often described how many packets/bytes your NIC really saw (but when the NIC is heavily overloaded, these counters may still be off)
  • _qX_ - stands for queue number X - e.g. rx_q6_bytes - number of bytes received by the receive queue number 6

Some examples how different the xstats of individual NIC drivers can be:

1 Like