Best Tools for Building a Real-Time Network Attack Detection Pipeline with Machine Learning

Help
1

Hi everyone,
I’m currently working on building a real-time network intrusion detection pipeline using machine learning, and I’d appreciate some guidance on the best tools and practices.

So far, I’ve installed Suricata in AF_Packet mode and enabled the pcap-log option to save captured packets. This allows me to collect traffic data in .pcap format.

For the next steps, I’m considering using either CICFlowMeter or Zeek to process the .pcap files and extract network flow features for use in a machine learning model. However, I’m not entirely sure which tool would be more suitable or if there’s a better approach.

If anyone has experience with this kind of setup especially regarding real-time processing, feature extraction, or integration with ML models I’d love to hear your suggestions!

Thanks in advance!

2

Can’t help you with the ML, but why not just use Suricata flow records directly? No reason to use Suricata to create pcap’s to pass to Zeek to generate flow logs when Suricata already outputs rich flow logs.

2 Likes
3

Thank you for your attention! I’m working on my final-year project, where I aim to compare the pros and cons of using an IDS (like Suricata) versus a machine learning-based approach for detecting network attacks.
I’ve already developed a simple ML model to enhance Suricata’s detection capabilities. However, for the comparison phase, I need to evaluate both methods independently. This allows me to ensure a fair comparison between traditional IDS detection and my ML based approach.

4

One way to compare signature-based based approach vs. ML-based approach would be to use Suricata for both signature-based detection of traffic (see 8.1. Rules Format — Suricata 7.0.10 documentation) and for ML-based detection over various log types (flow, HTTP, TLS, etc.) that Suricata can generate (17.1. EVE — Suricata 7.0.10 documentation).

1 Like
5

Not sure if you know this already but here is a 4 piece blog post by Markus Kont with hands on OSS Jupyter Playbooks examples of ML and Suricata data for detection:

1 Like