Hello guys.
I am using Suricata [5.0.4_2] and for some reason I cannot upgrade to newer versions. When I start Suricata I start to have some problems as below and after a while I cannot reach the interface. Is upgrading a solution to get around this issue or is there a way to get around it somehow? thanks
a kernel: 384.201232 [1791] netmap_ring_reinit called for igb1 TX0 a kernel: 384.208850 [1816] netmap_ring_reinit total 1 errors a kernel: 384.214334 [1820] netmap_ring_reinit igb1 TX0 reinit, cur 168 → 151 tail 1024 → 1024 a kernel: 384.223221 [1791] netmap_ring_reinit called for igb1 TX0 a kernel: 384.229115 [1816] netmap_ring_reinit total 1 errors a kernel: 384.234651 [1820] netmap_ring_reinit igb1 TX0 reinit, cur 154 → 151 tail 1024 → 1024 a kernel: 384.243425 [1791] netmap_ring_reinit called for igb1 TX0 a kernel: 384.249914 [1816] netmap_ring_reinit total 1 errors a kernel: 384255420 [1820] netmap_ring_reinit igb1 TX0 reinit, cur 154 → 151 tail 1024 → 1024 a kernel: 384.264156 [1791] netmap_ring_reinit called for igb1 TX0 a kernel: 384270142 [1816] netmap_ring_reinit total 1 errors a kernel: 384.275660 [1820] netmap_ring_reinit igb1 TX0 reinit, cur 153 → 151 tail 1024 → 1024 a kernel: 384.285959 [1791] netmap_ring_reinit called for igb1 TX0 a kernel: 384.291939 [1816] netmap_ring_reinit total 1 errors a kernel: 384.297467 [1820] netmap_ring_reinit igb1 TX0 reinit, cur 152 → 151 tail 1024 → 1024 a kernel: 384.306215 [1791] netmap_ring_reinit called for igb1 TX0 a kernel: 384.312225 [1816] netmap_ring_reinit total 1 errors a kernel: 384.317732 [1820] netmap_ring_reinit igb1 TX0 reinit, cur 154 → 151 tail 1024 → 1024
Some additional information about your setup would be helpful for formulating a troubleshooting plan.
I’ll start by saying that as of August 2022, Suricata 5.x is officially end-of-life. That means no more updates or fixes in that code branch. The current release is 6.0.9, and 7.0 is about to go to Release Candidate status.
Here is a list of additional information that would be helpful to know about your system:
What kind of hardware are you running Suricata on? Is it a dedicated appliance or a generic PC?
What is the underlying operating system? Is it FreeBSD, Linux, or something else? My first guess, based on the interface name of igb1, is FreeBSD.
Why can’t you upgrade? That would be the simplest first test for a fix. There have been some changes to the netmap code in newer Suricata versions. Also, if your OS is FreeBSD and is a dated version like your Suricata version, then there have have been a number of driver and netmap device bug fixes applied to the newer FreeBSD versions.
Thank you for your valuable reply. And sorry for the incomplete explanation. Yes, I am using Freebsd 12.3 stable and I can say that it is a dedicated device. I’ve been doing tests with some packages that belong to me, but there are errors that are not found on every device. so this one too. I am trying to understand if I can figure it out about problem before upgrading it. upgrading is little expensive in time
That particular error normally comes from different Suricata threads accessing the same netmap buffer(s) in an uncoordinated fashion (meaning without proper locking mechanisms in place). One thread modifies a buffer, then another thread comes along and makes a different change to the same buffer. The netmap device code in FreeBSD notices this and logs the error you see. This particularly can happen when you use the “autofp” threading mode with netmap operation. Try altering your setup to use “workers” mode threading if you are currently using “autofp”. I believe “autofp” is the default, and many users never change it in the suricata.yaml configuration file.
But the real fix for this error is in the new Suricata 6.0.9 release which explicitly addresses the thread locking problem with “autofp” mode. You should really consider upgrading your entire setup. Go to a more recent supported version of FreeBSD, then install the Suricata 6.0.9 package from FreeBSD ports.
Hello Mr Bill, first of all thank you for your reply. I didn’t want to answer directly that day, I wanted to test it first. and even if it throws errors like this in my testing process;
a kernel: 471.868741 [4046] netmap_transmit igb1 full hwcur 221 hwtail 868 qlen 376 a kernel: 471.876627 [4046] netmap_transmit igb1 full hwcur 221 hwtail 868 qlen 376 a kernel: 480.278206 [4046] netmap_transmit igb1 full hwcur 867 hwtail 869 qlen 1021
didn’t stop it from working. It’s just a little slowing down. It doesn’t seem to be a problem for now. Thank you again. I never thought it could be from there, and I couldn’t find any direction on that in my research.
Thank you for the further testing. Just so I am clear, are you now testing on Suricata 6.0.9? And if so, was it compiled with both the NETMAP and NETMAP_V14 options enabled, or just with NETMAP enabled? The NETMAP configure option uses a single host stack ring while the NETMAP_V14 option adds multiple host stack rings using the same number as the physical NIC exposes.
This error indicates the netmap TX buffer is not getting emptied fast enough. It is a good hint for continued troubleshooting.