Performance highly dependent on flow:established

Rules
1

Hey guys,

I am a CS student and new to the IDS domain.

I am currently trying to generate Suricata rules from NIDS datasets using ML algorithms. I know that suricata rules are often formulated by experts. Therefore, the generation might not necessarily be promising. However, it is only a proof of concept.

I mainly use the rule options flowbits, flowint, stream_size, etc.

When I apply my generated rule set to a PCAP file, the precision of the classification is
significantly higher if I add the flow:established option to each rule.

This is a simplified example:

alert ip any any <> any any (sid:102; flow:to_server; ttl:31; flowbits:set,sttl_is_31; flowbits:noalert;)
alert ip any any -> any any (sid:103; msg:"insert fancy alert msg"; flow:established; flowbits:isnotset,sttl_is_31;)

I have two questions:

  • I read in the Suricata documentation that flow:established , for example, only considers
    TCP flows with completed 3WHS. However, that doesn’t explain to me why it reduces
    the number of false positives. Does anyone have a guess as to why my performance is changing?
  • Docs also say that flow:established is also for UDP, although the state is handled different. With the signature 103, I am only able to detect TCP flows. All UDP flows with ttl 31 are not detected.

Sorry if these questions sound stupid, but I’m new and trying to understand the workings of Suricata under the hood.

Thanks and Kind Regards
Eike

2

Do you have a change to share the pcaps with us?
This would help debugging.

3

I am using the PCAP files from UNSW-NB15, but I think my questions are rather abstract and debugging isn’t required: CloudStor

In the meantime, I am a little further along with my findings.

Could it be that flowbits are not completely evaluated until the flow reaches the “established” state? If so, my results are comprehensible.

Unfortunately, I cannot detect all UDP flows that do not have destination packets. This is because the “established” state is not reached without packets from either side of the communication. Is there any chance to use UDP flows and flowbits together in a useful way?

4

One reason that could explain why false positives will go down when established flag is required (without inspecting the pcaps) is the presence of scanning activities in those pcaps

a successful attack (and detection of it) dose indeed require the connection flow to be established (3WHS)

so if the pcaps have noisy scanning activities that wont constitute a successful attack (but still exhibit your detection criteria), then this might explain why the establish flag will increase your detection performance.

but then this is all without looking at the pcaps.