Heartbeat Alert - rate_filter on really common traffic

Rules
1

I’m looking to create a heartbeat alert from our 100’s of sensors. We are exploring using a rate_filter on common traffic (microsoft.com or google.com) where the alert is suppressed for 12 hours, so we can confirm the sensor and all of the underlying alert delivery is working correctly ~2 times a day.

Will that overwhelm the sensor (performance or otherwise)? Is there a better way to implement a heartbeat using a different mechanism?

Can rate_limit be woven into the alert, so no other file needs to be modified?

2

Hi Don,

If I understand your question correctly, you will be

  • injecting a heartbeat into the traffic monitored and inspected by 100s of sensors
  • Each sensor has a rule to generate an alert (with the goal of suppressing the alert to twice daily, every 12 hours)

What frequency will you be injecting the heartbeat packet?

Many Suricata deployments produces tens of thousands of alerts per second

If you’re adding one alert at a given frequency (likely not less than 1 heartbeat packet per second) and the sensors are not underpowered, an extra alert per second or so won’t negatively impact Suricata performance on your sensors.

That said, if your sensors are underpowered and having performance issues (manifested mostly through the drop rate but there are other factors indicating the sensor is having issues with the ingress traffic rate) now, I suggest that those be addressed first.

Lastly, please use the latest supported Suricata (currently Suricata 6.0.12) as it will contain all of the latest security, evasion and bug fixes.

3

Hi Jeff - Thanks for the quick response. In our configuration, we are not in a position to inject/initiate traffic that would allow for a scheduled alert. Instead, I was thinking of using web traffic to common domains (one rule monitoring traffic to *.microsoft.com *.google.com, *.dell.com, *.cisco.com *.hp.com) and configure suppression to only alert 1 time per hour (or some predictable frequency).

4

Thanks for the clarification. I think that might not be optimal unless the frequency of one of the well-known domains was quite low and/or the rule contained enough logic to generate just a few alerts per second.

Have you considered a monitoring agent that uses suricatasc to check each sensor’s uptime? This would require an agent local to the sensor as suricatasc connects to a UNIX domain socket created by Suricata and on which Suricata will respond to commands (uptime is one of them). Any logic that can create JSON and connect to a UNIX domain socket will work (even Javascript, if necessary :slight_smile: )

Perhaps other community members could share how they monitor Suricata instances.

5

Objective: Confirm (1) Suricata monitor port sees traffic; (2) confirm alert is firing as expected; and (3) confirm the alert information is being brought into our custom log aggregation system correctly.

I have a SIEM agent on the sensor, used to monitor logs and various hardware/os information…but not Suricata. Suricata is handled separately.

6

Suricata emits stats every 8 seconds (configurable time period but default is 8s). These are indicated with an event_type of stats in the eve.json log. Among the values it contains are an alert count and packets/bytes received count. That would help with (1) and perhaps (2).

7

Yep - I’m monitoring stats.log and i have some alerts generated from that info. Because it is a custom log aggregation process, I am mostly concerned with monitoring the overall collection and ingestion of the Alert log info.

BTW - I really appreciate your time and suggestions. I know this is a unique test case, but one I’m determined to find an answer to.

8

Perhaps you could use Prometheus to scrape stats?

We’ve considered adding a heartbeat mechanism to Suricata but have not done so yet so this may or may not help with your use case.

9

Hi Jeff,

You mentioned “injecting a heartbeat into the traffic monitored” – What would be the best way to do this, if you only had the sensor available to you?

Thanks for the links to Prometheus. I’m not sure that is an option at this time.

10

Follow-up: Would something like TCPREPLAY or SCAPY allow us to dump a PCAP onto the monitor port of Suricata?

11

Hi Don,

Prometheus is a great option for monitoring Suricata and the sensor on which it’s running. I mentioned that because it would likely be helpful in your environment even if your more immediate need is something else.

Can you give more context as to how each sensor is receiving packets and the command line used to invoke Suricata?

12

I registered here just to reply because I was looking for this solution.

I made a custom rule for this today, alert icmp 127.0.0.0/8 any <> 127.0.0.0/8 any (msg:"heartbeat alert; analyst ignore";) (please forgive formatting or errors here).

And a scapy program to send the packet. Set it on a cronjob for every four hours.

from scapy.all import *

sendp(Ether()/IP(src="127.1.2.3", dst=”127.4.5.6”)/ICMP(payload="Ignore Me"),int="{your_monitor_interface_here}" )

I may have made a couple syntax errors here, but the ideas all worked in testing and production. Added benefit of this not being a packet that would be seen in a monitor interface, ever, and it will work with other IDS tools attached to that interface. Good Hunting!

1 Like
13

Thanks for that. Scapy is one we evaluated.

I should have come back to update:
We have a custom signature for a unique DNS query and the associated rule to alert when seen. Then we use TCPReplay to replay a pcap of that DNS request. Works reliably.

14

Cool! Yeah, my issue with the tcpreplay was that our SIEM uses the time of the alert, and suricata was seeing the traffic as the original timestamp. Didn’t quote work as intended. Gonna edit my command with an update too. Finally got to look at what I wrote today…

15 
from scapy.all import *

sendp(Ether()/IP(src="127.1.2.3", dst=”127.4.5.6”)/ICMP()/"Ignore Me",int="{your_monitor_interface_here}" )