V6.0.9: Custom Rule Failing to Load for Base64 Dataset - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - datasets are only supported for sticky buffers

[ADudeWhoSurfs (Ads)]

Hi Community!

I need some help with an issue I am experiencing with custom dataset and rules.

NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"

● suricata.service - LSB: Next Generation IDS/IPS
     Loaded: loaded (/etc/init.d/suricata; generated)
     Active: active (running) since Tue 2023-01-31 20:47:48 UTC; 13min ago
       Docs: man:systemd-sysv-generator(8)
    Process: 635721 ExecStart=/etc/init.d/suricata start (code=exited, status=0/SUCCESS)
      Tasks: 66 (limit: 462579)
     Memory: 1.1G
     CGroup: /system.slice/suricata.service
             └─635739 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -vvv

Jan 31 20:47:48 <hostname> systemd[1]: Starting LSB: Next Generation IDS/IPS...
Jan 31 20:47:48 <hostname> suricata[635721]: Starting suricata in IDS (af-packet) mode... done.
Jan 31 20:47:48 <hostname> systemd[1]: Started LSB: Next Generation IDS/IPS.

This is Suricata version 6.0.9 RELEASE

# This configuration file generated by:
#     Suricata 6.0.8

My dataset(s) are base64-encoded line-separated strings representing hot domains and IP’s and defined in my config file:

datasets:
  dns-ioc-alert-dataset:
    type: string
    state: /etc/suricata/rules/domains_base64_threatfox_base64.lst
  rf-ip-ioc-alert-dataset:
    type: string
    state: /etc/suricata/rules/ipv4s_base64_rf_base64.lst
  rf-ip-ioc-alert-dataset:
    type: string
    state: /etc/suricata/rules/domains_base64_rf_base64.lst

default-rule-path: /var/lib/suricata/rules

rule-files:
  - suricata.rules
  - /etc/suricata/rules/local.rules
  # - /etc/suricata/rules/*.rules
  - /etc/suricata/rules/ads-ioc-dataset.rules
  - /etc/suricata/rules/ads-crypto-currency-rules.rules

My rules file is as follows:

########### XXXXX #############

#alert dns any any -> any any (msg: "dns list test"; dns.query; to_sha256; dataset:isset,dns-ioc-alert-dataset.sha256; sid:1000003; rev:1;)
alert dns any any -> any any (msg: "dns list test"; dns.query; dataset:set,dns-ioc-alert-dataset; sid:1000004; rev:1;)
alert http any any -> any any (msg: "http user-agent test"; http.user_agent; dataset:set,dns-ioc-alert-dataset; sid:1000005; rev:1;)
#alert ip any any <> any any (msg: "IOC DATASET MATCH Traffic identified ingress or egress to custom IoC dataset"; dataset:set,domains_base64_rf_base64.lst; priority:4; sid:1000006; rev:1; gid:2;)
#alert ip any any -> any any (msg: "IOC DATASET MATCH Traffic identified ingress or egress to custom IoC dataset"; dataset:set,domains_base64_rf_base64.lst; priority:4; sid:1000006; rev:1; gid:2;)
alert ip any any -> any any (msg: "IOC DATASET MATCH Traffic identified ingress or egress to custom IoC dataset"; dataset:set,ipv4s_base64_rf_base64.lst; sid:1000007; rev:1;)

• The strange thing that I cannot comprehend, is that rule 1 and 2 are successful… difference is that they match different protocols. However, I tried changing my rules to match tcp, http, dns etc.
• I think this is related to the fact that the “working rules” with my dataset scan on application-layer protocols with Protocol Keywords
• Simply put, I.E dns.query & http.user_agent are in working rules 1000004,1000005 compared to 1000006 and so forth
• I left some rules commented-out, to illustrate some of the testing I did through process of elimination style approach

Log Output (var/log/suricata/suricata.log)

31/1/2023 -- 20:47:48 - <Info> - stats output device (regular) initialized: stats.log
31/1/2023 -- 20:47:48 - <Config> - dataset: dns-ioc-alert-dataset loading from '/etc/suricata/rules/domains_base64_threatfox_base64.lst'
31/1/2023 -- 20:47:48 - <Config> - dataset: dns-ioc-alert-dataset loaded 5390 records
31/1/2023 -- 20:47:48 - <Config> - dataset: rf-ip-ioc-alert-dataset loading from '/etc/suricata/rules/domains_base64_rf_base64.lst'
31/1/2023 -- 20:47:48 - <Config> - dataset: rf-ip-ioc-alert-dataset loaded 100159 records
31/1/2023 -- 20:47:48 - <Config> - Delayed detect disabled

31/1/2023 -- 20:47:48 - <Config> - IP reputation disabled
31/1/2023 -- 20:47:48 - <Config> - Loading rule file: /var/lib/suricata/rules/suricata.rules
31/1/2023 -- 20:47:53 - <Config> - Loading rule file: /etc/suricata/rules/local.rules
31/1/2023 -- 20:47:53 - <Config> - Loading rule file: /etc/suricata/rules/ads-ioc-dataset.rules
31/1/2023 -- 20:47:53 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - datasets are only supported for sticky buffers
31/1/2023 -- 20:47:53 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert ip any any -> any any (msg: "IOC DATASET MATCH Traffic identified ingress or egress to custom IoC dataset"; dataset:set,ipv4s_base64_rf_base64.lst; sid:1000007; rev:1;)" from file /etc/suricata/rules/ads-ioc-dataset.rules at line 8
31/1/2023 -- 20:47:53 - <Warning> - [ERRCODE: SC_ERR_NO_RULES(42)] - No rule files match the pattern /etc/suricata/rules/ads-crypto-currency-rules.rules
31/1/2023 -- 20:47:53 - <Config> - No rules loaded from /etc/suricata/rules/ads-crypto-currency-rules.rules.
31/1/2023 -- 20:47:53 - <Info> - 4 rule files processed. 38819 rules successfully loaded, 1 rules failed

...
31/1/2023 -- 20:48:05 - <Info> - All AFP capture threads are running.

Related Issues:

• I could only see the below which do not seem to match, no other references:

Some errors are reported when starting Suricata

Some errors are reported when starting Suricata： [199226] 29/11/2021 – 13:47:56 - (detect-pcre.c:890) (DetectPcreSetup) – [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - Expression seen with a sticky buffer still set; either (1) reset sticky buffer with pkt_data or (2) use a sticky buffer providing “raw http uri”. [199226] 29/11/2021 – 13:47:56 - (detect-engine-loader.c:185) (DetectLoadSigFile) – [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature “alert http any any <> any any (msg:…

Error Code issues

Hello, i am using suricata-6.0.3 version , when I start Suricata I will always get below error. And the suricata.log is as attached.suricata.log (90.2 KB) [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - pcre with /R (relative) needs preceding match in the same buffer [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - can’t use a relative keyword like within/distance with a absolute relative keyword like depth/offset for the same content. [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - rule 2836763 mixes keyword…

Unless it’s also me, I can’t find any clarification on what the “Sticky Buffer” actually entails or some context on it to reverse-engineer my errors from the suricata.log

TYIA!

[ADudeWhoSurfs (Ads)]

I see where I may be at fault here

• Is this issue occurring because I am specifying ip protocol (“any”) but expecting Suricata to match the src/ip against either from my dataset: (since I have two types, 1=domains and 1=ipv4s)
  • Domains (assuming Suricata can read src/ip address of the packet and then dns-lookup that IP and base64-decode, match it against a domain-related dataset entry)
  • IP addresses (assuming Suricata is able to read src/ip address of the packet, cross-correlate a match against a base64-decoded IP-related dataset entry)

Retrospectively, do I need to use HTTP and DNS keywords to actually match packet content against my datasets? I.E http.uri / dns.query etc.

• If so, does anyone have any examples using SSL/TLS keywords that could cover multiple protocols? Not for DNS as the DNS lookup is made to a NS which is not the remote host, but at least HTTPS and other protocols that utilize SSL encrypted comms where a TLS handshake is completed for each protocol as they use SSL.

• Related, but not inline with the above… within my rule, do I also need to use any Transformations to manipulate the data? Or when my rules are processed, does Suricata base64-decode my dataset and then use the plain-text strings to match against rule configuration?

[ADudeWhoSurfs (Ads)]

Hello!

I think this is now resolved with the following rule configs:

# `to_sha256` transform not required as Suricata will base64-decode the strings from datasets upon initialization
# alert dns any any -> any any (msg: "dns list test"; dns.query; to_sha256; dataset:isset,dns-ioc-alert-dataset.sha256; sid:1000003; rev:1;)
################################ DNS RULES | SID 1000004-1000015 ################################
# alert dns any any -> any any (msg: "IOC DATASET MATCH DNS traffic identified ingress or egress to custom IoC dataset list test"; dns.query; dataset:set,dns-ioc-alert-dataset; dataset:set,rf-ip-ioc-alert-dataset; dataset:set,rf-domain-ioc-alert-dataset; sid:1000004; rev:1; gid:2;)
alert dns any any -> any any (msg: "IOC DATASET MATCH DNS traffic identified ingress or egress to custom IoC dataset list test"; dns.query; dataset:set,dns-ioc-alert-dataset; sid:1000004; rev:1; gid:2;)
alert dns any any -> any any (msg: "IOC DATASET MATCH DNS traffic identified ingress or egress to custom IoC dataset list test"; dns.query; dataset:set,rf-ip-ioc-alert-dataset; sid:1000005; rev:1; gid:2;)
alert dns any any -> any any (msg: "IOC DATASET MATCH DNS traffic identified ingress or egress to custom IoC dataset list test"; dns.query; dataset:set,rf-domain-ioc-alert-dataset; sid:1000006; rev:1; gid:2;)
#### Testing neverssl.com to identify which rules match based on dataset. Ensure to import www.neverssl.com, base64-encoded into the dataset for a match. ####
alert dns any any -> any any (msg: "NEVERSSLDOTCOM TEST traffic"; dns.query; content:"neverssl"; nocase; sid:1000007; rev:1; gid:3;)
################################ EO DNS RULES | SID 1000004-1000015 ################################
################################ HTTP RULES | SID 1000015-1000025 ################################
alert http $HOME_NET any -> any any (msg: "http user-agent test"; http.user_agent; content:"curl"; nocase; sid:1000015; rev:1; gid:3;)
################################ EO HTTP RULES | SID 1000015-1000025 ################################
################ IP RULES | SID -1000035 ################
#### Match any bidirectional traffic for IP (any) traffic to custom IoC base64-encoded IP strings datase ####
# Below rule is incompatible with Suricata rules as Datasets are not supported by the sticky buffer
# alert ip any any <> any any (msg: "IP IOC DATASET MATCH Traffic identified ingress or egress to custom IoC dataset"; dataset:set,rf-ip-ioc-alert-dataset; priority:4; sid:1000025; rev:1; gid:2;)
################ EO IP RULES | SID 1000025-1000035 ################
################################ SSL/TLS RULES | SID 1000035-1000045 ################################
#### Match any TLS (<v1.3) plain-text SSL handshake negotations where the TLS SNI contains a match from IoC datasets, thus attempting to capture all encrypted protocols for traffic to and from imported IoC's dataset ####
alert tls $HOME_NET any <> any any (msg: "IOC DATASET MATCH SSL_TLS traffic identified ingress or egress to custom IoC dataset list test"; tls.sni; dataset:set,dns-ioc-alert-dataset; sid:1000025; rev:1; gid:2;)
alert tls $HOME_NET any <> any any (msg: "IOC DATASET MATCH SSL_TLS traffic identified ingress or egress to custom IoC dataset list test"; tls.sni; dataset:set,rf-ip-ioc-alert-dataset; sid:1000026; rev:1; gid:2;)
alert tls $HOME_NET any <> any any (msg: "IOC DATASET MATCH SSL_TLS traffic identified ingress or egress to custom IoC dataset list test"; tls.sni; dataset:set,rf-domain-ioc-alert-dataset; sid:1000027; rev:1; gid:2;)
#### Match for TLS traffic where the TLS certificate has expired ####
alert tls $HOME_NET any <> any any (msg: "SSL_TLS EXPIRED CERT traffic identified ingress or egress to expired SSL_TLS certs"; tls_cert_expired; sid:1000028; rev:1; gid:3;)
################################ EO SSL/TLS RULES | SID 1000035-1000045 ################################

Log output:

1/2/2023 -- 20:27:50 - <Info> - stats output device (regular) initialized: stats.log
1/2/2023 -- 20:27:50 - <Config> - dataset: dns-ioc-alert-dataset loading from '/etc/suricata/rules/domains_base64_threatfox_base64.lst'
1/2/2023 -- 20:27:50 - <Config> - dataset: dns-ioc-alert-dataset loaded 5730 records
1/2/2023 -- 20:27:50 - <Config> - dataset: rf-ip-ioc-alert-dataset loading from '/etc/suricata/rules/ipv4s_base64_rf_base64.lst'
1/2/2023 -- 20:27:50 - <Config> - dataset: rf-ip-ioc-alert-dataset loaded 4905 records
1/2/2023 -- 20:27:50 - <Config> - dataset: rf-domain-ioc-alert-dataset loading from '/etc/suricata/rules/domains_base64_rf_base64.lst'
1/2/2023 -- 20:27:50 - <Config> - dataset: rf-domain-ioc-alert-dataset loaded 100251 records

1/2/2023 -- 20:27:50 - <Config> - IP reputation disabled
1/2/2023 -- 20:27:50 - <Config> - Loading rule file: /var/lib/suricata/rules/suricata.rules
1/2/2023 -- 20:27:56 - <Config> - Loading rule file: /etc/suricata/rules/local.rules
1/2/2023 -- 20:27:56 - <Config> - Loading rule file: /etc/suricata/rules/ads-ioc-dataset.rules
1/2/2023 -- 20:27:56 - <Warning> - [ERRCODE: SC_ERR_NO_RULES(42)] - No rule files match the pattern /etc/suricata/rules/ads-crypto-currency-rules.rules
1/2/2023 -- 20:27:56 - <Config> - No rules loaded from /etc/suricata/rules/ads-crypto-currency-rules.rules.
1/2/2023 -- 20:27:56 - <Info> - 4 rule files processed. 38834 rules successfully loaded, 0 rules failed

Last questions:

1. Apart from testing traffic and verifying fast.log, is there anyway for me to see my custom rules within my configuration loaded? To validate it is as expected. I tried to use suricatasc but does not seem to be possible there, or I was trying cat /var/lib/suricata/rules/suricata.rules | grep 10000XX but didn’t seem to work?
2. Within my rule, do I also need to use any Transformations to manipulate the data? Or when my rules are processed, does Suricata base64-decode my dataset and then use the plain-text strings to match against rule configuration?
3. Can you concatenate/aggregate multiple datasets in one rule?

[Andreas_Herz]

@ADudeWhoSurfs

Apart from testing traffic and verifying fast.log, is there anyway for me to see my custom rules within my configuration loaded? To validate it is as expected. I tried to use suricatasc but does not seem to be possible there, or I was trying cat /var/lib/suricata/rules/suricata.rules | grep 10000XX but didn’t seem to work?

You can see this logline:

1/2/2023 -- 20:27:56 - <Info> - 4 rule files processed. 38834 rules successfully loaded, 0 rules failed

Which already tells you at least the amount of rules loaded and that none failed.
If the grep didn’t work is the file actually the correct one and do you see it in the file?

2. This depends on the way you define the set, see 6.34. Datasets — Suricata 6.0.10 documentation so if you use the type string it will convert the input to that properly but also expects the loading of the data to be the same.

3. Might be worth a try, but could reduce the readability