Building a Threat Intelligence Pipeline for Suricata: Architecture, Challenges, and Practical Considerations

Community Announcements
,
1

Introduction

One of the important challenges in developing network security systems is integrating external Threat Intelligence sources with intrusion detection and prevention mechanisms.

Suricata provides a powerful network traffic inspection engine and a flexible rule system. However, using external Threat Intelligence feeds in practice requires addressing several additional challenges: retrieving and normalizing Indicators of Compromise (IOCs), converting them into detection rules, managing updates, and preventing false positives.

As part of the ongoing development of Suri Oculus, a network security monitoring, management, and analysis platform built around Suricata, we are working on improving our Threat Intelligence processing capabilities.

In this post, I would like to share some architectural decisions, technical challenges, and open questions related to integrating external IOC sources with Suricata.

1. General Architecture

Suri Oculus uses a modular architecture that includes a C++ backend, Redis, Python services, and a web interface.

The Threat Intelligence subsystem is intended to retrieve information from external sources, process it, and make it available for use by Suricata.

The general pipeline looks like this:

Threat Intelligence Sources → Download → Parsing → IOC Normalization → Validation → Rule Generation → Suricata

In practice, this process involves several stages:

  1. Downloading data from external Threat Intelligence sources.

  2. Parsing the received data and extracting supported IOC types.

  3. Normalizing indicators and removing duplicates.

  4. Validating indicators and determining their applicability.

  5. Converting IOCs into Suricata rules or other supported matching mechanisms.

  6. Validating the generated configuration before deployment.

  7. Updating active rules and managing their lifecycle.

One of the sources we are considering is ThreatFox by abuse.ch, which provides information about indicators associated with malicious activity.

However, the architecture should not depend on the format or availability of a single Threat Intelligence provider.

2. Challenges in Converting IOCs into Suricata Rules

At first glance, converting Indicators of Compromise into IDS/IPS rules seems relatively straightforward.

For example, if a source provides an IP address or domain name, a rule can be generated to detect communication with the corresponding resource.

In practice, however, the situation is considerably more complex.

IP Addresses and IP Indicators

Suppose an external source provides the following indicator:

203.0.113.10:443

This address is used for illustration purposes only.

When generating a rule automatically, several factors must be considered:

  • Transport protocol: TCP or UDP.

  • Direction of the network connection.

  • Purpose of the indicator: C2 communication, malware distribution, or other malicious activity.

  • Age and reliability of the IOC.

  • Suricata operating mode: IDS or IPS.

It is particularly important not to assume that every IP indicator represents TCP traffic.

Some Threat Intelligence sources provide protocol information, but this information may be missing or ambiguous.

Automatically converting every IP indicator into a TCP rule can therefore result in missed UDP traffic.

We are considering a dedicated normalization stage where the protocol, IP address, port, and other indicator attributes are stored separately.

If the transport protocol is unknown, the system should not assign one arbitrarily without an explicitly defined policy.

URLs and HTTPS

Another example involves URL-based indicators:

https://example.com/malware/payload

For unencrypted HTTP traffic, Suricata can inspect the hostname and URI using the corresponding HTTP inspection buffers.

However, for HTTPS traffic without TLS decryption, the URL path is not visible to normal HTTP inspection.

Consequently, automatically converting an HTTPS URL into a rule using http.uri does not guarantee detection of requests to that resource.

Possible alternatives include examining available TLS metadata, DNS queries, and other network characteristics.

However, this introduces an important distinction: a domain name and a complete URL are not equivalent indicators.

Blocking an entire domain based on a single malicious URL may be overly aggressive, particularly when the domain belongs to shared infrastructure or a legitimate hosting service.

Therefore, URL-based IOC processing must account for which parts of an indicator are actually observable in network traffic.

3. Detection or Prevention?

One of the most important design questions is whether all Threat Intelligence indicators should automatically produce blocking rules.

Suricata supports actions such as alert, drop, and other mechanisms for handling rule matches.

However, automatically applying drop to every IOC introduces significant risks.

External feeds may contain outdated indicators, IP addresses that have changed ownership, public services, and infrastructure shared by legitimate and malicious applications.

We are therefore considering separating indicators according to their confidence level and intended use.

A possible model includes:

  • Detection — generating alert rules for monitoring and recording matches.

  • Prevention — using drop only for indicators that meet stricter confidence and validation requirements.

  • Review — retaining ambiguous indicators for further investigation without automatically activating them.

This approach separates Threat Intelligence collection from decisions about traffic blocking.

It is also important to remember that drop provides actual traffic blocking only when Suricata is operating in an appropriate IPS configuration.

4. Managing the IOC Lifecycle

Threat Intelligence is dynamic.

Indicators appear, change, become outdated, and disappear from provider feeds.

Therefore, periodically downloading an IOC list is not sufficient.

A reliable system must manage the lifecycle of each indicator.

We are considering storing the following attributes:

  • Unique IOC identifier.

  • Indicator type and normalized value.

  • Source of the information.

  • First-seen and last-seen timestamps.

  • Last update timestamp.

  • Available threat classification and confidence information.

  • Indicator status: active, expired, or disabled.

  • Association with the corresponding Suricata rule.

Special attention must be paid to indicators appearing in multiple sources.

Deduplication should preserve IOC provenance rather than simply discarding repeated values.

Furthermore, an indicator disappearing from a provider’s latest feed does not necessarily mean it has immediately become irrelevant.

A separate expiration and rule removal policy is therefore required.

5. Performance and Scalability

Large Threat Intelligence feeds introduce another challenge: the impact of rule volume on Suricata performance.

Converting every IOC into a separate signature rule is not always the most efficient approach.

Depending on the indicator type and deployment scenario, alternative mechanisms may be more appropriate:

  • IP reputation and related matching mechanisms.

  • Suricata Datasets for storing and matching collections of values.

  • Rule grouping and optimization.

  • Pre-filtering indicators before deployment.

  • Limiting the number of active indicators according to defined criteria.

This is particularly important for Suri Oculus because one of the project’s objectives is to support Linux systems with limited computing resources.

We want to avoid situations where enabling a large Threat Intelligence feed results in disproportionate memory consumption or significantly reduced packet inspection performance.

6. Safe Rule Updates

Automatic Threat Intelligence updates should not disrupt Suricata operation.

For this reason, downloading new data and activating it should be treated as separate operations.

The preferred workflow is:

Download → Parse → Normalize → Generate → Validate → Activate

Before activating a new configuration, the generated rules and Suricata configuration must be validated.

If validation fails, the previous working configuration should remain available.

Atomic replacement of rule files, consistent SID management, and the ability to roll back to a previous version are also important.

We believe the reliability of this update mechanism is just as important as the number of supported Threat Intelligence sources.

7. Questions for the Suricata Community

While designing this subsystem, several questions have emerged. I would be interested in hearing from Suricata developers and experienced users about their approaches.

1. Rule Generation vs. Datasets

Which approach do you prefer for large, frequently updated collections of IP addresses and domains?

Are there practical performance considerations or limitations that should influence the choice between individual signature rules and Datasets?

2. Handling Ambiguous IOCs

How do you handle IP indicators when the source does not provide reliable transport protocol information?

Do you prefer generating more general rules, or excluding such indicators from automatic rule generation?

3. Automatic Blocking

Do you use external Threat Intelligence feeds to generate drop rules automatically?

If so, what validation, filtering, and exception mechanisms do you use to minimize false positives and unintended blocking?

4. Expired Indicators

What IOC lifecycle policy has proven most practical in your environment?

Should an indicator be removed immediately when it disappears from a source, or should it remain active for an additional period?

5. Large Threat Intelligence Feeds

Which approaches have worked best for integrating large IOC feeds while maintaining reasonable memory consumption, detection performance, and update reliability?

Conclusion

Integrating Threat Intelligence with Suricata involves much more than converting lists of IP addresses and domains into detection rules.

In practice, it requires careful consideration of network protocols, encrypted traffic visibility, source data quality, performance, and the risks associated with automatic blocking.

With Suri Oculus, we aim to develop a mechanism that allows external Threat Intelligence sources to be integrated in a controlled, reliable, and reproducible way.

Our goal is not to replace existing Suricata rule management tools, but to complement them with Threat Intelligence integration and analysis capabilities within a unified platform.

I would appreciate technical feedback, recommendations, and examples from anyone who has implemented similar solutions.

Suri Oculus project: https://suri-oculus.com