Suricata and IP blacklist

Help
41

I used ipv4.hdr just for trying something. Honestly I am not even sure if that is the most recommended buffer. I am trying to get the IP address from the package and pass it to the dataset to verify if it is on it or not. In the official doc I found ipv4.hdr clearly classified as a “sticky buffer”, and having nothing else better at hand, I invited it to the party.

First I was thinking to use something like this:

alert ip any any -> any any (msg:"TEST Bad IP"; ipv4.hdr; offset:12, offset:16; dataset:isset,test-badip; sid:10; rev:1;)

Where offsets 12 and 16 are source and destination IP from the IPv4 header.

ipv4 header
ipv4 header1105×354 8.82 KB

But got an error stating that for using offset, I must place a content: first. If I use content:, I am checking just a single, specific and manually entered IP on each rule, like 8.8.8.8, but I need to check thousands of IPs dinamically. That’s the reason for using either iprep or dataset, because basically, both engines offers to match on large amounts of data against data on each packet.

Again, I am not even sure if that is the most recommended buffer to use. If you know a better one, please share it. I am ready to try anything.

Or maybe we could (if it does not exist yet and/or is too dificult) add a new sticky buffer, just to return the IP address and to be used along with dataset.

But before we can use or test the rule, the main issue is the dataset configuration, not being detected or accepted. I think I need to focus my efforts in resolve that first.

As for the blacklist for the iprep, if you want to take a look, I can post it in Pastebin and share the link, but is nothing else but a list of IPs having around 50500 entries on it. All from public and free blacklists.

42

I think we could troubleshoot step by step.
Lets start with IPrep as originally mentioned.

Can you share (if possible) a smaller subset of the list that is not generating the desired result in your setup ?(could be smaller list 100-200 or 1k entries)

43

Hi @pevma ,

I though we reached the point where we were considering there’s a bug for iprep, at least for my Suricata installation. But I am open to all ideas, so if you are available to help me, I’m ready :grinning:.

If you see my post here, I already did some tests and confirmed iprep engine is working as expected for all items below line 1170 within the blacklist. Any item beyond that point it won’t be blocked, regardless the category to which belongs to. I have tested the same “ip,catID,rep_value” before and after that point to get these results.

So, how many entries do you want on the list?
Meanwhile I’m sharing this portion of the list.

And I could try both iprep and datset. The last one with a little help, because I have not been able to configure it for even test it.

Thank you

44

Hi, anything new?

I am happy to share with you that I have found a way to setup a dataset.

I was missing a small detail and is the fact that the list must be encoded with base64 . I was passing just a plain text list, but I just was able to see the error regarding the format after trying many different combinations. Then I encoded my list to base64 before passing the list to Suricata.

base64 /etc/suricata/rules/dataset/test-badip.lst >> /etc/suricata/rules/dataset/test-badip64.lst

Anyway, setting up the dataset from the rule instead of the config file, was accepted with this format:

alert ip any any -> any any (msg:"TEST Bad IP"; ipv4.hdr; dataset:isset,test-badip64,type string,load /etc/suricata/rules/test-badip64.lst; classtype:bad-ip; sid:10; rev:1;)

With the above config there are not alerts and processing the new file ends without issues.

Source updated
Source is valid
Source activated in "Default SELKS ruleset"

This represents a great step forward, since although configuring a dataset from the configuration file is failing (or at least not being detected by Scirius), at least I was able to confirm an alternative path that works.

What I need now is help to adjust the rule to make it operational. Current rule is enabled, applied transformation “drop”, but it is not blocking any traffic. I suspect the sticky buffer I am using is not the right one or I am missing something else to extract the IP address so the dataset can check match.

45

I am investigating currently, will feedback.

1 Like
46

I can confirm there seems to be a problem with iprep, even when the memcaps are adjusted.
Updating the issue here - Bug #4280: Suricata is not fully reading or loading the iprep files - Suricata - Open Information Security Foundation

47

Hi guys,

I am currently researching and testing both solutions: iprep and dataset, to avoid wasting time and to be able to either solve the problem, or at least to help you in your investigation and to provide as much details and feedback as possible. If I get stuck with one way, I jump to the other, until I can’t move forward. So if you guys are working in both directions, please feel free to feedback about both lines of investigations and I will try both.

Thank you for all your hard work and help.

48

Comment:

Since the dataset rules are not working, I was wondering if Suricata is really reading/processing the dataset or it’s failing like is currently happening with iprep .

A couple of days ago, while I was trying to create the dataset from the config file, Suricata was restarting without issues (fast, no errors). But now that I tried to reference a file ( /etc/suricata/rules/tss-badip64.lst ) encoded in base64 from the config file, Suricata got stuck while loading in the following lines:

user1@server1:~$ sudo tail -f /var/log/suricata/suricata.log
[19990] 5/2/2021 -- 11:58:35 - (source-af-packet.c:2822) <Perf> (ReceiveAFPThreadExitStats) -- (W#01-enxa..2d70) Kernel: Packets 18141, dropped 0
[19991] 5/2/2021 -- 11:58:35 - (source-af-packet.c:2822) <Perf> (ReceiveAFPThreadExitStats) -- (W#01-enxa..2e2e) Kernel: Packets 55516, dropped 0
[19989] 5/2/2021 -- 11:58:35 - (counters.c:854) <Info> (StatsLogSummary) -- Alerts: 0
[19989] 5/2/2021 -- 11:58:35 - (ippair.c:294) <Perf> (IPPairPrintStats) -- ippair memory usage: 414144 bytes, maximum: 16777216
[19989] 5/2/2021 -- 11:58:35 - (host.c:299) <Perf> (HostPrintStats) -- host memory usage: 10442152 bytes, maximum: 33554432
[19989] 5/2/2021 -- 11:58:35 - (detect-engine-build.c:1722) <Info> (SigAddressCleanupStage1) -- cleaning up signature grouping structure... complete
[19989] 5/2/2021 -- 11:58:35 - (util-device.c:359) <Notice> (LiveDeviceListClean) -- Stats for 'enxa0cec8d92d70':  pkts: 18141, drop: 0 (0.00%), invalid chksum: 0
[19989] 5/2/2021 -- 11:58:35 - (util-device.c:359) <Notice> (LiveDeviceListClean) -- Stats for 'enxa0cec8d92e2e':  pkts: 55516, drop: 0 (0.00%), invalid chksum: 0
[19989] 5/2/2021 -- 11:58:35 - (util-mpm-hs.c:1078) <Perf> (MpmHSGlobalCleanup) -- Cleaning up Hyperscan global scratch
[19989] 5/2/2021 -- 11:58:35 - (util-mpm-hs.c:1086) <Perf> (MpmHSGlobalCleanup) -- Clearing Hyperscan database cache

Does it means that it is really trying to load the encoded file (around 1MB)?

Another question:
This time I keep testing the config that seems to work. No dataset setup in the config file, but within the rules. Scirus shows no errors and the dataset seems to be accepted, but after applied Suricata Ruleset Actions from Scirius, which push all new settings to Suricata, I noticed the following lines in the logs (which I turned more verbose to get more details):

[27694] 5/2/2021 -- 13:47:35 - (detect-engine-loader.c:251) <Config> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/scirius.rules
[27694] 5/2/2021 -- 13:47:35 - (datasets.c:298) <Config> (DatasetLoadString) -- dataset: tss-badip64 loading from '/etc/suricata/rules/test-badip64.lst'
[27694] 5/2/2021 -- 13:47:35 - (datasets.c:365) <Config> (DatasetLoadString) -- dataset: test-badip64 loaded 0 records
[27694] 5/2/2021 -- 13:47:35 - (detect-engine-loader.c:355) <Info> (SigLoadSignatures) -- 1 rule files processed. 2 rules successfully loaded, 0 rules failed

Especially this line
[27694] 5/2/2021 -- 13:47:35 - (datasets.c:365) <Config> (DatasetLoadString) -- dataset: test-badip64 loaded 0 records

Does it means Suricata is not loading the dataset file, or it can’t read it, and that is why the rule isn’t dropping any traffic?

Could you please provide a valid sticky buffer that I can use to check match against an IP dataset ? The closest I’ve been is using a combination that can include ipv4.hdr , content and offset , but it doesn’t seem to be filtering the IP address, which is the field I need to compare against the dataset.

After reading the sticky buffers documentation for days and tried many different combinations without any luck, I’m starting to think that I won’t be able to use a dataset rule to drop packets to/from an IP given a blacklists, with any of the sticky buffers currently available. If I’m wrong, please show me the right combination. Or perhaps we might need a new sticky buffer like ip.addr

49

The ipaddress in the packet header is a raw 32 bit number, not a nice dotted quad notation. Did you account for that in your data?

50

I’m sorry, but I don’t understand your comment. Could you explain this in a different way?

51

He means that the IP Address in a packet header is not like 141.81.0.10, it would in fact be a raw 32 bit number.
141.81.0.10 would for example be 0x8d51000a. Did you take this into account?

52

Hi,

Honestly, I confess that have no Idea what you’re talking about. Are you talking about IPREP engine or DATASET engine? If this is about IPREP, engine works fine up to 1170 items in a list, or distributed in several lists. If this is about DATASET, originally each item in the list was with the format 141.81.0.10. The only transformation I did was converting the whole list to base64, due to an error stating the list has to be in that format. Also due to this comment in your forum.

I do not recall any other explicit modification I have done, but I can check whatever you need. Just please tell me where to look.

Thanks

53

This is referring to the ipv4.hdr keyword. Read his comment and the comment I made here Suricata and IP blacklist - #39 by syoc one more time.

The data “over the wire” is not the ASCII, readable, characters 8.8.8.8.
It’s in HEX, see CyberChef.
And no, you cannot match on content:"08080808";, that would be matching on printable characters.
You need to match on the binary representation when using the ipv4.hdr. That would mean content:"|08 08 08 08|"; as I mentioned in my post.

1 Like
54

I got it now :sweat_smile:. Thank you all for this explanation :+1:. I’ll try modifying the rule later.

55

Hi guys,

I attempted to modify the rule to use the raw 32 bit number in the ipv4.hdr. The problem is that I am not trying to check match for a single number specified within the rule, but against one or more lists with several IPs.

Can you please show me a valid transformation for the rule to dynamically convert IP address within ipv4.hdr into a nice dotted quad notation?

drop ip any any -> any any (msg:"TEST Bad IP Dataset"; ipv4.hdr; dataset:isset,test-badip64,type string,load /etc/suricata/rules/test-badip64.lst; classtype:bad-ip; sid:10; rev:1;)

Or perhaps should I convert each item within the dataset to be a raw 32 bit number, then convert the list to a base64 file?

Thanks

56

I am on very thin ice here, but it seems like datasets might not do substring matches[1].
Meaning the content in the dataset file needs to be equal to the entire content of the sticky buffer you are matching on.
The ipv4.hdr contains more than just the IP addresses, including random data such as client port numbers making it quite unsuitable for dataset usage.
IPRep is the correct way to handle this.
If you need this right now and don’t want to dig further into why it’s not working yourself you can just generate one IP rule for each IP using a script. Suricata will handle it just fine.

[1] suricata/datasets.c at b66d013294e258a05f9f0c8506e059ba7415f5fd · OISF/suricata · GitHub

1 Like
57

@syoc you are correct, I overlooked this originally. The dataset will do a lookup of the entire ipv4.hdr data, so not just the bytes at some offset. So this approach won’t work at all.

2 Likes
58

Hi,

I appreciate your efforts and time dedicated to this issue. My goal remains the same: use Suricata to block several IPs given one or several blacklists. Either IPREP or DATASET will fit my needs.

In fact IPREP works just fine, but it’s a shame that it’s currently limited to only 1170 items among all the lists. I hope the bug can be fixed soon.

On the other hand, it seems DATASET could do the job as well, but so far I have been unable to find the right combination to create a valid rule. After many different tests, I tend to think that none of the current sticky buffers will fit the type of rule that I need. Perhaps if you guys could add a new sticky buffer that contains only the source/destination IP, then it would be easy to create the rule I need.

I’ll check what @syoc suggest to see if I can use it.

These are just ideas to consider. In the end, I will use the one that works (either IPREP or DATASET) :wink:

Thank you

1 Like
59

Hi @ManuelFFF , I’ve been able to look into this and it seems there is a bug related to loading a iprep list file that contains both ipv4 and ipv6 addresses. I’ll post a patch soon, but as a temporary workaround you could try splitting the ipv4 and ipv6 addresses into separate files. The bug is in the per file loading routine, so splitting them into the separate files should be enough.

grep -v : scirius-iprep.list > ipv4-only.list
grep : scirius-iprep.list > ipv6-only.list

Then load them:

reputation-files:
  - ipv4-only.list
  - ipv6-only.list

Can you try this to see if it works for you?

1 Like
60

If you’re able to test a patch, you could try this