Heartbeat Alert - rate_filter on really common traffic

Rules
1

I’m looking to create a heartbeat alert from our 100’s of sensors. We are exploring using a rate_filter on common traffic (microsoft.com or google.com) where the alert is suppressed for 12 hours, so we can confirm the sensor and all of the underlying alert delivery is working correctly ~2 times a day.

Will that overwhelm the sensor (performance or otherwise)? Is there a better way to implement a heartbeat using a different mechanism?

Can rate_limit be woven into the alert, so no other file needs to be modified?

2

Hi Don,

If I understand your question correctly, you will be

  • injecting a heartbeat into the traffic monitored and inspected by 100s of sensors
  • Each sensor has a rule to generate an alert (with the goal of suppressing the alert to twice daily, every 12 hours)

What frequency will you be injecting the heartbeat packet?

Many Suricata deployments produces tens of thousands of alerts per second

If you’re adding one alert at a given frequency (likely not less than 1 heartbeat packet per second) and the sensors are not underpowered, an extra alert per second or so won’t negatively impact Suricata performance on your sensors.

That said, if your sensors are underpowered and having performance issues (manifested mostly through the drop rate but there are other factors indicating the sensor is having issues with the ingress traffic rate) now, I suggest that those be addressed first.

Lastly, please use the latest supported Suricata (currently Suricata 6.0.12) as it will contain all of the latest security, evasion and bug fixes.

3

Hi Jeff - Thanks for the quick response. In our configuration, we are not in a position to inject/initiate traffic that would allow for a scheduled alert. Instead, I was thinking of using web traffic to common domains (one rule monitoring traffic to *.microsoft.com *.google.com, *.dell.com, *.cisco.com *.hp.com) and configure suppression to only alert 1 time per hour (or some predictable frequency).

4

Thanks for the clarification. I think that might not be optimal unless the frequency of one of the well-known domains was quite low and/or the rule contained enough logic to generate just a few alerts per second.

Have you considered a monitoring agent that uses suricatasc to check each sensor’s uptime? This would require an agent local to the sensor as suricatasc connects to a UNIX domain socket created by Suricata and on which Suricata will respond to commands (uptime is one of them). Any logic that can create JSON and connect to a UNIX domain socket will work (even Javascript, if necessary :slight_smile: )

Perhaps other community members could share how they monitor Suricata instances.

5

Objective: Confirm (1) Suricata monitor port sees traffic; (2) confirm alert is firing as expected; and (3) confirm the alert information is being brought into our custom log aggregation system correctly.

I have a SIEM agent on the sensor, used to monitor logs and various hardware/os information…but not Suricata. Suricata is handled separately.

6

Suricata emits stats every 8 seconds (configurable time period but default is 8s). These are indicated with an event_type of stats in the eve.json log. Among the values it contains are an alert count and packets/bytes received count. That would help with (1) and perhaps (2).

7

Yep - I’m monitoring stats.log and i have some alerts generated from that info. Because it is a custom log aggregation process, I am mostly concerned with monitoring the overall collection and ingestion of the Alert log info.

BTW - I really appreciate your time and suggestions. I know this is a unique test case, but one I’m determined to find an answer to.

8

Perhaps you could use Prometheus to scrape stats?

We’ve considered adding a heartbeat mechanism to Suricata but have not done so yet so this may or may not help with your use case.

9

Hi Jeff,

You mentioned “injecting a heartbeat into the traffic monitored” – What would be the best way to do this, if you only had the sensor available to you?

Thanks for the links to Prometheus. I’m not sure that is an option at this time.

10

Follow-up: Would something like TCPREPLAY or SCAPY allow us to dump a PCAP onto the monitor port of Suricata?

11

Hi Don,

Prometheus is a great option for monitoring Suricata and the sensor on which it’s running. I mentioned that because it would likely be helpful in your environment even if your more immediate need is something else.

Can you give more context as to how each sensor is receiving packets and the command line used to invoke Suricata?