Rules with pcre keyword not alerting

I am standing up a new Suricata server to replace our current one but the new server does not seem to be generating alerts for any rules with the pcre keyword. I have both old and new servers receiving the same packet feed and using the same ET Pro ruleset. Both are generating the same alerts except the new server is not alerting for rules that contain the pcre keyword.

The primary difference between them is the old server is running 6.0.3 from source and the new is running 6.0.4 from rpm. There are also various configuration differences between them due to the change in underlying hardware.

Is there a configuration setting that may be causing this? Is there a difference in PCRE implementation between 6.0.3 and 6.0.4? Is it possible that the rpm version was compiled without PCRE support?

PCRE is a required component of Suricata. You can view the runtime libraries used by Suricata with ldd $(which suricata)

The same version of PCRE is used in 6.0.4 and earlier versions of Suricata. Suricata 7 uses a newer version of the PCRE library.

There are a few PCRE configuration settings (most users never change these) in suricata.yaml:

 #recursion and match limits for PCRE where supported
 pcre:
   match-limit: 3500
   match-limit-recursion: 1500

If both servers receive exactly the same traffic and are using the same rulesets, I don’t see a reason why 6.0.3 would generate alerts and 6.0.4 wouldn’t.

The only configuration change from 6.0.3 to 6.0.4 are the settings for stream.memcap and stream.reassembly.memcap: instead of 32MB and 64MB (respectively), the new values are 64MB and 256MB (respectively).

Yes this is very strange. I probably wouldn’t even have noticed if we weren’t keeping an eye on log4shell scanning activity against our Internet-facing servers. A common scenario we see is on the same session the old sensor will fire three alerts and the new sensor will fire two alerts. The alert missing from the new sensor is from a rule with the pcre keyword.

Both sensors have the default pcre values in suricata.yaml. Stream.reassembly.memcap is well above the default and identical for both sensors. I ran ldd on both sensors. The new one actually has quite a few more runtime libraries than the old sensor, but they both have libpcre.so.1.

Does Suricata rely on anything in the underlying OS for pcre to work properly? I’m grasping at straws at this point.

The PCRE implementation is largely independent of the underlying OS.

Is it possible that the servers are receiving nearly identical but not identical traffic?

What hardware are you hosting the old and new servers on? Which NIC and packet capture method are you using?

There are some VLAN tag differences between the two, but otherwise they are receiving the same feed from our packet broker.

Old sensor:
Mellanox fiber NIC
Intel Xeon processor
cluster_qm
cpu-affinity

New sensor:
Intel copper NIC
AMD EPYC processor
cluster_flow
no cpu-affinity

I was trying to keep the new sensor as close to default as possible except where I believed there would be a tangible benefit to performance. If anything jumps out as a probably culprit, please let me know. In the meantime, I will be methodically changing settings until I get the new sensor to trigger on pcre alerts.

Since you mention vlans. Suricata uses vlan tags as a part of the flow hash by default, could that be a problem? You could try disabling vlan for tracking in the yaml file

Good thought but that didn’t seem to do the trick.

The strange thing is the sensor seems to be seeing all the traffic just fine. It will make the app layer eve.json entries and alert on other rules. But anything with the pcre keyword won’t alert. The old sensor does alert so I know the rules are good.

Could you provide (via DM if needed) a few rules and a smallish pcap that demonstrate the issue?

Here is the solution for anyone that runs into this in the future. I found that this issue was only occurring when Suricata was run under systemd and not when I ran it directly on the command line. I ran Suricata under systemd with debug logging (-vvvv) and found the following two lines:

- RWX pages denied by OS
- PCRE won’t use JIT as OS doesn’t allow RWX pages

I looked in /etc/systemd/system/multi-user.target.wants/suricata.service and found it had the line:
MemoryDenyWriteExecute=true

After changing this setting to false Suricata is running PCRE as expected.

This is odd. We’d not expect the outcome of the pcre checks to be affected by this, other than in how efficiently they run.

Could it be that Suricata was timing out on sessions before pcre had a chance to finish? I’m willing to test some things out if there’s anything you want me to try.