Dataset rules not triggering

Rules
1

Hi.

I have read the dataset documentation and I’m having a hard time getting my test rules to fire.
I’m sure that it’s something silly I have overlooked, so hopefully someone can point it out for me.
Dataset config in suricata.yaml

root@753edddc3735:/etc/suricata# suricata --dump-config | grep datasets
datasets = (null)
datasets.defaults = (null)
datasets.defaults.memcap = 100mb
datasets.defaults.hashsize = 2048
datasets.test-list = (null)
datasets.test-list.type = string
datasets.test-list.load = /etc/suricata/rules/testlist.lst

Dataset file content

root@753edddc3735:/etc/suricata# cat rules/testlist.lst | base64 -d
google.com
8.8.8.8

Rule file content

root@753edddc3735:/etc/suricata# cat test.rule 
alert ip any any -> any any (msg:"test dataset"; ipv4.hdr; dataset:isset, test-list, type string; sid:1;)
#alert ip any any -> 8.8.8.8 any (msg:"test regular"; sid:2;)
#alert ip any any -> any any (msg:"test ipv4.hdr"; ipv4.hdr; content:"|08 08 08 08|"; sid:3;)
alert http any any -> any any (msg:"test dataset 2"; http.host; dataset:isset, test-list, type string, load /etc/suricata/rules/testlist.lst; sid:4;)
alert http any any -> any any (msg:"test dataset 3"; http.host; dataset:isset, test-list, type string; sid:5;)
alert http any any -> any any (msg:"test dataset 4"; http.host; dataset:isset, test-list; sid:6;)
alert http any any -> any any (msg:"test regular http"; http.host; content:"google.com"; sid:7;)

Relevant logs from starting Suricata

root@753edddc3735:/etc/suricata# suricata -c suricata.yaml -S test.rule -i eth0 -vvv
7/2/2021 -- 13:16:27 - <Notice> - This is Suricata version 6.0.1 RELEASE running in SYSTEM mode
[.....]
7/2/2021 -- 13:16:27 - <Config> - dataset: test-list loading from '/etc/suricata/rules/testlist.lst'
7/2/2021 -- 13:16:27 - <Config> - dataset: test-list loaded 2 records
[.....]
7/2/2021 -- 13:16:27 - <Info> - 5 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 4 inspect application layer, 0 are decoder event only
7/2/2021 -- 13:16:27 - <Config> - building signature grouping structure, stage 1: preprocessing rules... complete

Generating traffic

root@753edddc3735:/# wget google.com

Suricata does proper http parsing

root@753edddc3735:/# cat /var/log/suricata/eve.json | jq '. | select(.event_type == "http") | .http'
{
  "hostname": "google.com",
  "url": "/",
  "http_user_agent": "Wget/1.20.3 (linux-gnu)",
  "http_content_type": "text/html",
  "http_method": "GET",
  "protocol": "HTTP/1.1",
  "status": 301,
  "redirect": "http://www.google.com/",
  "length": 219
}
{
  "hostname": "www.google.com",
  "url": "/",
  "http_user_agent": "Wget/1.20.3 (linux-gnu)",
  "http_method": "GET",
  "protocol": "HTTP/1.1",
  "length": 0
}

Only the regular http rule using the content keyword matches

cat /var/log/suricata/eve.json | jq '. | select(.event_type == "alert") | .alert.signature'
"test regular http"
"test regular http"
2

One thing that get me and i have ended up chasing my tail sort of speak the first time - please double check that each line contains only the datasets base 64 string and nothing else - no spaces at the end of the line,tabs,carriage returns etc …

3

I have a 0a so I think I’m good, but thanks for the tip.

root@753edddc3735:/etc/suricata# cat rules/testlist.lst | base64 -d | xxd 
00000000: 676f 6f67 6c65 2e63 6f6d 0a38 2e38 2e38  google.com.8.8.8
00000010: 2e38 0a                                  .8

root@753edddc3735:/etc/suricata# cat rules/testlist.lst | xxd
00000000: 5a32 3976 5a32 786c 4c6d 4e76 6251 6f3d  Z29vZ2xlLmNvbQo=
00000010: 0a4f 4334 344c 6a67 754f 416f 3d0a       .OC44LjguOAo=.
4

You might want to check the CR at the end

Z29vZ2xlLmNvbQo=

which is
google.com<CR>
vs

Z29vZ2xlLmNvbQ==

which is just
google.com