Setting flowbits for AND logic on same packet

dcasa · October 28, 2020, 12:04pm

Let me explain more clearly.
Suppose I have these rules :

alert http any any -> any any (msg:"RULE A"; flowbits: set, test; **some content matching on packet X**)

alert http any any -> any any (msg:"RULE B"; flowbits: isset, test; **More content matching on packet X**)

Given a scenario were this two rules should be triggered on the same packet (for example an HTTP request) I can prove that sometimes the second rule will not work.
Now a simple solution can be just merging the two rules in one but for a scalability reason I would prefer to leave them separated.

Can someone please tell me if this suricata behaviour is wanted on purpose?
Am I doing something wrong?

@vjulien

syoc · October 28, 2020, 1:40pm

Are you testing with a pcap or live traffic?
Is suricata logging any packet loss in stats.log when you are testing these rules?

dcasa · October 28, 2020, 2:14pm

I’m testing on a pcap, I do not see any packet loss :

------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Date: 10/28/2020 -- 14:08:29 (uptime: 0d, 00h 00m 00s)
------------------------------------------------------------------------------------
Counter                                       | TM Name                   | Value
------------------------------------------------------------------------------------
decoder.pkts                                  | Total                     | 15
decoder.bytes                                 | Total                     | 13316
decoder.ipv4                                  | Total                     | 15
decoder.ethernet                              | Total                     | 15
decoder.tcp                                   | Total                     | 15
decoder.avg_pkt_size                          | Total                     | 887
decoder.max_pkt_size                          | Total                     | 7306
flow.tcp                                      | Total                     | 1
tcp.sessions                                  | Total                     | 1
tcp.syn                                       | Total                     | 1
tcp.synack                                    | Total                     | 1
detect.alert                                  | Total                     | 1
app_layer.flow.http                           | Total                     | 1
app_layer.tx.http                             | Total                     | 1
flow.spare                                    | Total                     | 10000
flow_mgr.rows_checked                         | Total                     | 65536
flow_mgr.rows_skipped                         | Total                     | 65536
tcp.memuse                                    | Total                     | 573440
tcp.reassembly_memuse                         | Total                     | 98304
flow.memuse                                   | Total                     | 7474632

syoc · October 28, 2020, 2:21pm

A pcap you can share? Sounds strange if the flowbit checking rule (“RULE B”) on hits on some runs.

dcasa · October 28, 2020, 2:45pm

I can’t but without the rules I believe it would be useless.
Btw the “RULE B” is hitting only when I remove the isset flowbit check.

Sounds strange to me too!
Anyway It isn’t the same RULE B hitting, but the pattern is the same :

Rule A setting a flowbit over a packet
Rule B checking that flowbit and hitting on the same packet

I agree that’s not respecting the logic behind the flowbit but seems really strange that it worked in the other rules

syoc · October 28, 2020, 3:25pm

Ah. Sounds like there should be some difference between the flowbit rules that are working as intended and then ones that are not.

I have never tried setting a flowbit and checking it on the same packet.

dcasa · October 28, 2020, 3:30pm

Thanks for the help !
Btw I really have no idea…

jae · October 29, 2020, 2:29am

We have lots of rules in the ET/ETPRO set where this sort of logic occurs.

dcasa · October 29, 2020, 8:24am

Actually I’ve found where is the problem.
It’s the pcre, these signatures are working :

alert http any any -> any any ( msg:"RULE A"; flow: established,to_server; http.method; content:"POST"; http.content_type; content:"x-www-form-urlencoded"; flowbits: set, test; sid:5;)
alert http any any -> any any ( msg:"RULE B"; flowbits: isset, test; flow: established,to_server; http.method; content:"POST"; sid:6; )

And these not (only the first one is alerting) :

alert http any any -> any any ( msg:"RULE A"; flow: established,to_server; http.method; content:"POST"; http.content_type; content:"x-www-form-urlencoded"; http.request_body; pcre:"/=/"; flowbits: set, test; sid:5;)
alert http any any -> any any ( msg:"RULE B"; flowbits: isset, test; flow: established,to_server; http.method; content:"POST"; sid:6; )

dcasa · October 29, 2020, 9:43am

I can share this pcap apache_struts_exploit.pcapng (13.3 KB)

syoc · November 2, 2020, 8:37am

I just tested the pcap and rules in my environment and I get the same results.
Looks like a bug.

Jungho · November 2, 2020, 1:47pm

I remember that a signature that was matched once in HTTP-Buffer was not alerted again.However, “RULE B” did not match (due to flowbits), so it is questionable that there is no log.

I tested it with the attached signature. The difference is that I removed the flowbits. The transaction identified in the attached pcap consists of two packets. Since there are no flowbits, “RULE B” is alerted first, followed by “RULE A”.

alert http any any -> any any ( msg:"RULE A"; flow: established,to_server; http.method; content:"POST"; http.content_type; content:"x-www-form-urlencoded"; http.request_body; pcre:"/=/";  sid:5;)
alert http any any -> any any ( msg:"RULE B"; flow: established,to_server; http.method; content:"POST"; sid:6; )

[root@idps-pcap ~]# suricata -c /etc/suricata/suricata.yaml --simulate-ips -k none -r /root/pcap/apache_struts_CVE-2017-9791_exploit_runtime.pcapng

Signature without flowbits
[root@idps-pcap ~]# cat fast.log
10/27/2020-20:15:54.455637 [**] [1:6:0] RULE B [**] [Classification: (null)] [Priority: 3] {TCP} 18.255.255.16:46316 → 172.19.1.4:8080
10/27/2020-20:15:54.458066 [**] [1:5:0] RULE A [**] [Classification: (null)] [Priority: 3] {TCP} 18.255.255.16:46316 → 172.19.1.4:8080

Signature without flowbits
[root@idps-pcap ~]# cat rule_perf.log
…
{
“timestamp”: “2020-11-02T22:03:09.297646+0900”,
“sort”: “number of matches”,
“rules”: [{
“signature_id”: 5,
“gid”: 1,
“rev”: 0,
“checks”: 2,
“matches”: 1,
“ticks_total”: 232396,
“ticks_max”: 157732,
“ticks_avg”: 116198,
“ticks_avg_match”: 157732,
“ticks_avg_nomatch”: 74664,
“percent”: 85
}, {
“signature_id”: 6,
“gid”: 1,
“rev”: 0,
“checks”: 1,
“matches”: 1,
“ticks_total”: 37948,
“ticks_max”: 37948,
“ticks_avg”: 37948,
“ticks_avg_match”: 37948,
“ticks_avg_nomatch”: 0,
“percent”: 14
}
…

Signature with flowbits
[root@idps-pcap ~]# cat fast.log
10/27/2020-20:15:54.458066 [**] [1:5:0] RULE A [**] [Classification: (null)] [Priority: 3] {TCP} 18.255.255.16:46316 → 172.19.1.4:8080

Signature with flowbits
{
“timestamp”: “2020-11-02T22:24:02.665481+0900”,
“sort”: “number of matches”,
“rules”: [{
“signature_id”: 5,
“gid”: 1,
“rev”: 0,
“checks”: 2,
“matches”: 1,
“ticks_total”: 110382,
“ticks_max”: 63468,
“ticks_avg”: 55191,
“ticks_avg_match”: 46914,
“ticks_avg_nomatch”: 63468,
“percent”: 95
}, {
“signature_id”: 6,
“gid”: 1,
“rev”: 0,
“checks”: 1,
“matches”: 0,
“ticks_total”: 5420,
“ticks_max”: 5420,
“ticks_avg”: 5420,
“ticks_avg_match”: 0,
“ticks_avg_nomatch”: 5420,
“percent”: 4
}

And I tested raw. TCP raw streams are simply detected as they are identified. When tcp-stream signature is tested against the raw stream, it is alerted twice each according to the set chunk-size (2560, randomize-chunk=no).

alert tcp-stream any any -> any any ( msg:"RULE A"; flow: established,to_server; content:"POST"; content:"x-www-form-urlencoded";  pcre:"/=/"; flowbits: set, test; sid:5;)
alert tcp-stream any any -> any any ( msg:"RULE B"; flowbits: isset, test; flow: established,to_server;  content:"POST"; sid:6; )

[root@idps-pcap ~]# cat fast.log
10/27/2020-20:15:54.455637 [**] [1:5:0] RULE A [**] [Classification: (null)] [Priority: 3] {TCP} 18.255.255.16:46316 → 172.19.1.4:8080
10/27/2020-20:15:54.455637 [**] [1:6:0] RULE B [**] [Classification: (null)] [Priority: 3] {TCP} 18.255.255.16:46316 → 172.19.1.4:8080
10/27/2020-20:15:54.458066 [**] [1:5:0] RULE A [**] [Classification: (null)] [Priority: 3] {TCP} 18.255.255.16:46316 → 172.19.1.4:8080
10/27/2020-20:15:54.458066 [**] [1:6:0] RULE B [**] [Classification: (null)] [Priority: 3] {TCP} 18.255.255.16:46316 → 172.19.1.4:8080

[root@idps-pcap ~]# cat rule_perf.log
…
{
“timestamp”: “2020-11-02T22:17:10.948494+0900”,
“sort”: “number of matches”,
“rules”: [{
“signature_id”: 5,
“gid”: 1,
“rev”: 0,
“checks”: 2,
“matches”: 2,
“ticks_total”: 150100,
“ticks_max”: 87294,
“ticks_avg”: 75050,
“ticks_avg_match”: 75050,
“ticks_avg_nomatch”: 0,
“percent”: 70
}, {
“signature_id”: 6,
“gid”: 1,
“rev”: 0,
“checks”: 2,
“matches”: 2,
“ticks_total”: 62208,
“ticks_max”: 31778,
“ticks_avg”: 31104,
“ticks_avg_match”: 31104,
“ticks_avg_nomatch”: 0,
“percent”: 29
}]
}
…

dcasa · November 2, 2020, 3:17pm

So the problem relies in the protocol parser! (?)
Can we consider this as a bug then?

jae · November 24, 2020, 4:24pm

I think this warrants generating a redmine ticket -> https://redmine.openinfosecfoundation.org/projects/suricata

Topic		Replies	Views
Different behaviour between byte_test and content when using flowbits Help	0	580	May 11, 2021
Some match bypass? Rules rules	27	1952	September 17, 2021
Performance highly dependent on flow:established Rules	3	1375	July 23, 2021
Write Suricata Rules trigger condition to both http request and http response Rules	1	493	July 31, 2023
Suricata rules about network scan Rules rules	2	866	January 18, 2023

Setting flowbits for AND logic on same packet

Related Topics