Setting flowbits for AND logic on same packet

Let me explain more clearly.
Suppose I have these rules :

alert http any any -> any any (msg:"RULE A"; flowbits: set, test; **some content matching on packet X**)

alert http any any -> any any (msg:"RULE B"; flowbits: isset, test; **More content matching on packet X**)

Given a scenario were this two rules should be triggered on the same packet (for example an HTTP request) I can prove that sometimes the second rule will not work.
Now a simple solution can be just merging the two rules in one but for a scalability reason I would prefer to leave them separated.

Can someone please tell me if this suricata behaviour is wanted on purpose?
Am I doing something wrong?

@vjulien

Are you testing with a pcap or live traffic?
Is suricata logging any packet loss in stats.log when you are testing these rules?

I’m testing on a pcap, I do not see any packet loss :

------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Date: 10/28/2020 -- 14:08:29 (uptime: 0d, 00h 00m 00s)
------------------------------------------------------------------------------------
Counter                                       | TM Name                   | Value
------------------------------------------------------------------------------------
decoder.pkts                                  | Total                     | 15
decoder.bytes                                 | Total                     | 13316
decoder.ipv4                                  | Total                     | 15
decoder.ethernet                              | Total                     | 15
decoder.tcp                                   | Total                     | 15
decoder.avg_pkt_size                          | Total                     | 887
decoder.max_pkt_size                          | Total                     | 7306
flow.tcp                                      | Total                     | 1
tcp.sessions                                  | Total                     | 1
tcp.syn                                       | Total                     | 1
tcp.synack                                    | Total                     | 1
detect.alert                                  | Total                     | 1
app_layer.flow.http                           | Total                     | 1
app_layer.tx.http                             | Total                     | 1
flow.spare                                    | Total                     | 10000
flow_mgr.rows_checked                         | Total                     | 65536
flow_mgr.rows_skipped                         | Total                     | 65536
tcp.memuse                                    | Total                     | 573440
tcp.reassembly_memuse                         | Total                     | 98304
flow.memuse                                   | Total                     | 7474632


A pcap you can share? Sounds strange if the flowbit checking rule (“RULE B”) on hits on some runs.

I can’t :frowning: but without the rules I believe it would be useless.
Btw the “RULE B” is hitting only when I remove the isset flowbit check.

Sounds strange to me too!
Anyway It isn’t the same RULE B hitting, but the pattern is the same :

  • Rule A setting a flowbit over a packet
  • Rule B checking that flowbit and hitting on the same packet

I agree that’s not respecting the logic behind the flowbit but seems really strange that it worked in the other rules

Ah. Sounds like there should be some difference between the flowbit rules that are working as intended and then ones that are not.

I have never tried setting a flowbit and checking it on the same packet.

Thanks for the help :slight_smile: !
Btw I really have no idea…

We have lots of rules in the ET/ETPRO set where this sort of logic occurs.

Actually I’ve found where is the problem.
It’s the pcre, these signatures are working :

alert http any any -> any any ( msg:"RULE A"; flow: established,to_server; http.method; content:"POST"; http.content_type; content:"x-www-form-urlencoded"; flowbits: set, test; sid:5;)
alert http any any -> any any ( msg:"RULE B"; flowbits: isset, test; flow: established,to_server; http.method; content:"POST"; sid:6; )

And these not (only the first one is alerting) :

alert http any any -> any any ( msg:"RULE A"; flow: established,to_server; http.method; content:"POST"; http.content_type; content:"x-www-form-urlencoded"; http.request_body; pcre:"/=/"; flowbits: set, test; sid:5;)
alert http any any -> any any ( msg:"RULE B"; flowbits: isset, test; flow: established,to_server; http.method; content:"POST"; sid:6; )
1 Like

I can share this pcap apache_struts_exploit.pcapng (13.3 KB)

I just tested the pcap and rules in my environment and I get the same results.
Looks like a bug.

1 Like

I remember that a signature that was matched once in HTTP-Buffer was not alerted again.However, “RULE B” did not match (due to flowbits), so it is questionable that there is no log.

I tested it with the attached signature. The difference is that I removed the flowbits. The transaction identified in the attached pcap consists of two packets. Since there are no flowbits, “RULE B” is alerted first, followed by “RULE A”.

alert http any any -> any any ( msg:"RULE A"; flow: established,to_server; http.method; content:"POST"; http.content_type; content:"x-www-form-urlencoded"; http.request_body; pcre:"/=/";  sid:5;)
alert http any any -> any any ( msg:"RULE B"; flow: established,to_server; http.method; content:"POST"; sid:6; )

[root@idps-pcap ~]# suricata -c /etc/suricata/suricata.yaml --simulate-ips -k none -r /root/pcap/apache_struts_CVE-2017-9791_exploit_runtime.pcapng

Signature without flowbits
[root@idps-pcap ~]# cat fast.log
10/27/2020-20:15:54.455637 [**] [1:6:0] RULE B [**] [Classification: (null)] [Priority: 3] {TCP} 18.255.255.16:46316 → 172.19.1.4:8080
10/27/2020-20:15:54.458066 [**] [1:5:0] RULE A [**] [Classification: (null)] [Priority: 3] {TCP} 18.255.255.16:46316 → 172.19.1.4:8080

Signature without flowbits
[root@idps-pcap ~]# cat rule_perf.log
…
{
“timestamp”: “2020-11-02T22:03:09.297646+0900”,
“sort”: “number of matches”,
“rules”: [{
“signature_id”: 5,
“gid”: 1,
“rev”: 0,
“checks”: 2,
“matches”: 1,
“ticks_total”: 232396,
“ticks_max”: 157732,
“ticks_avg”: 116198,
“ticks_avg_match”: 157732,
“ticks_avg_nomatch”: 74664,
“percent”: 85
}, {
“signature_id”: 6,
“gid”: 1,
“rev”: 0,
“checks”: 1,
“matches”: 1,
“ticks_total”: 37948,
“ticks_max”: 37948,
“ticks_avg”: 37948,
“ticks_avg_match”: 37948,
“ticks_avg_nomatch”: 0,
“percent”: 14
}
…

Signature with flowbits
[root@idps-pcap ~]# cat fast.log
10/27/2020-20:15:54.458066 [**] [1:5:0] RULE A [**] [Classification: (null)] [Priority: 3] {TCP} 18.255.255.16:46316 → 172.19.1.4:8080

Signature with flowbits
{
“timestamp”: “2020-11-02T22:24:02.665481+0900”,
“sort”: “number of matches”,
“rules”: [{
“signature_id”: 5,
“gid”: 1,
“rev”: 0,
“checks”: 2,
“matches”: 1,
“ticks_total”: 110382,
“ticks_max”: 63468,
“ticks_avg”: 55191,
“ticks_avg_match”: 46914,
“ticks_avg_nomatch”: 63468,
“percent”: 95
}, {
“signature_id”: 6,
“gid”: 1,
“rev”: 0,
“checks”: 1,
“matches”: 0,
“ticks_total”: 5420,
“ticks_max”: 5420,
“ticks_avg”: 5420,
“ticks_avg_match”: 0,
“ticks_avg_nomatch”: 5420,
“percent”: 4
}

And I tested raw. TCP raw streams are simply detected as they are identified. When tcp-stream signature is tested against the raw stream, it is alerted twice each according to the set chunk-size (2560, randomize-chunk=no).

alert tcp-stream any any -> any any ( msg:"RULE A"; flow: established,to_server; content:"POST"; content:"x-www-form-urlencoded";  pcre:"/=/"; flowbits: set, test; sid:5;)
alert tcp-stream any any -> any any ( msg:"RULE B"; flowbits: isset, test; flow: established,to_server;  content:"POST"; sid:6; )

[root@idps-pcap ~]# cat fast.log
10/27/2020-20:15:54.455637 [**] [1:5:0] RULE A [**] [Classification: (null)] [Priority: 3] {TCP} 18.255.255.16:46316 → 172.19.1.4:8080
10/27/2020-20:15:54.455637 [**] [1:6:0] RULE B [**] [Classification: (null)] [Priority: 3] {TCP} 18.255.255.16:46316 → 172.19.1.4:8080
10/27/2020-20:15:54.458066 [**] [1:5:0] RULE A [**] [Classification: (null)] [Priority: 3] {TCP} 18.255.255.16:46316 → 172.19.1.4:8080
10/27/2020-20:15:54.458066 [**] [1:6:0] RULE B [**] [Classification: (null)] [Priority: 3] {TCP} 18.255.255.16:46316 → 172.19.1.4:8080

[root@idps-pcap ~]# cat rule_perf.log
…
{
“timestamp”: “2020-11-02T22:17:10.948494+0900”,
“sort”: “number of matches”,
“rules”: [{
“signature_id”: 5,
“gid”: 1,
“rev”: 0,
“checks”: 2,
“matches”: 2,
“ticks_total”: 150100,
“ticks_max”: 87294,
“ticks_avg”: 75050,
“ticks_avg_match”: 75050,
“ticks_avg_nomatch”: 0,
“percent”: 70
}, {
“signature_id”: 6,
“gid”: 1,
“rev”: 0,
“checks”: 2,
“matches”: 2,
“ticks_total”: 62208,
“ticks_max”: 31778,
“ticks_avg”: 31104,
“ticks_avg_match”: 31104,
“ticks_avg_nomatch”: 0,
“percent”: 29
}]
}
…

1 Like

So the problem relies in the protocol parser! (?)
Can we consider this as a bug then?

I think this warrants generating a redmine ticket -> https://redmine.openinfosecfoundation.org/projects/suricata