Suricata detection falls off after indeterminate period of time

Greetings,

A few weeks ago I happened to notice that Suricata will fail to detect some of the rules loaded after a period of time, rules such as “ET WEB_SERVER Possible CVE-2014-6271 Attempt in Header” or “ET WEB_SERVER Possible CVE-2014-6271 Attempt in HTTP Cookie” or any of the log4j ET Pro rule detection will fail and the only way to get Suricata to begin detecting them again is to restart it.

More concisely, Suricata works perfectly from startup until time X which is always random, where it will begin to fail to detect the aforementioned example rules as well as others I am sure.

I set up a remote server to send Shellshock requests to a host on our network and a job on the Suricata server that runs every 5 minutes to determine when those requests have not been detected and it happens randomly yet quite often, examples times where Suricata was restarted due to this problem

10/30/22 18:15:11
10/30/22 11:15:12
10/30/22 08:50:11
10/29/22 16:15:12
10/29/22 03:45:11
10/28/22 19:20:12
10/28/22 18:25:11
10/28/22 16:40:12
10/28/22 14:20:11
10/27/22 22:30:12
10/27/22 22:15:11
10/27/22 15:50:11
10/26/22 20:45:12
10/26/22 19:35:12
10/25/22 22:35:11
10/25/22 20:05:12
10/25/22 14:00:12

Curious if any of the Suricata devs have an idea of what might be the problem? My set up is a Fiberblaze 40G FPGA and PF_RING

System load and available memory have been ruled out already, those are definitely not the issue.

Thank you in advance for any input.

Greg

Hi Greg,

What version of Suricata?
How is it being supervised (systemd, …)?
Do you have logs and/or core dumps?
More context would help determine what the next step would be.

Hi Jeff,

6.0.8 and yes supervised by systemd

What logs are you looking for? The only entries in the messages log are
related to my process which kills Suricata and restarts it upon
determining the problem exists. I should note that Suricata continues
to alert on simple rules like ET Known Hostile IP which don’t require
parsing into the packets.

Sorry I don’t have any debug as I can’t run Suricata in debug mode, it
overwhelms the system when running in debug seeing as it’s continually
monitoring 5Gbps++

I compile Suricata as

LIBS=“-lrt” ./configure --prefix=/opt/suricata --enable-pfring=yes --with-libpfring-includes=/usr/include --with-libpfring-libraries=/usr/lib --with-libhs-includes=/usr/local/include/hs --with-libhs-libraries=/usr/local/lib64 --enable-af-packet=no

The box is RHEL 7.9 kernel 3.10.0-1160.71.1.el7.x86_64

Thank you for your assistance.

Greg

Hi Jeff,

We have full network packet captures so I am working on reviewing the
network data to see if there’s any problem with the TAP data funneling
to IDS.

I will update with further information when I have it, likely tomorrow
(Wednesday)

Thanks again,

Greg

Greg,

Thanks for the additional information.

When you give the next update, I suggest including network stats, including packet drop/loss values.

Hi Jeff,

No packet loss is reported by Suricata, everything seems copacetic as
far as Suricata is concerned, yet the issue happens.

That said, last night I noticed that the inlet temp on the Suricata
system was elevated and the capture card in the box quite hot, so I
increased the fan speeds in the box to 100% which cooled everything down
a bit and the issue went away, so now I am looking at the issue perhaps
stemming from the capture card running too warn at times and am looking
to move the system to a row or racks with better cooling.

I don’t know for sure this is the problem, but it certainly can not be
ruled out so for now I think Suricata can consider the issue resolved
unless after the move it happens again.

Thanks so much for this great software!

Greg