Suricata and IP blacklist

The IP rep functionality uses the host table, which by default has a fairly low memcap. Have you tried increasing that to see if it changes anything?

Hi and thank you for answering.

What a coincidence! I was just checking the following article

Speed

I’ve been playing with data sets of up to a million entries. Loading it takes hardly any time and I’m confident larger numbers will work just fine. The host table just needs bigger memcaps and hash sizes.

and I was wondering if what you suggest could help.

I was about to ask you guys what values would you recommend to start testing?
Current values:

# Host table:
#
# Host table is used by the tagging and per host thresholding subsystems.
#
host:
  hash-size: 4096
  prealloc: 1000
  memcap: 32mb

Thank you in advance!

I’d say double it, test it, repeat, until it works :slight_smile:

I’ll start working on it! Question: In each iteration is it advisable to double the values of the three variables or only the first two?

Thanks

The memcap is the only critical one, but I think its good to double all of them to avoid possible performance issues.

OK, thanks. I’ll share feedback later

I misunderstood that when all are in one file it behaves properly , otherwise not - sorry about this!

Hi,

Just to be in the same page, when all the IPs are in a single list AND there are just a few lines (I don’t know what the limit is), the blocking mechanism works as expected.

But, by mixing all the lists into a single list, without any limits, the number of lines amounts to more than 50,000 and only a few IPs are blocked.

Hi @vjulien and @pevma,

Some updates:

Take for instance IP 95.141.17.10 . If I add that IP to either file /etc/suricata/rules/scirius-iprep.list or /etc/suricata/rules/iprep/scirius-iprep.list , as line # 21605 ( 95.141.17.10,31,100 ), then restart Suricata and test access from TEST PC behind Suricata, the IP it is NOT being blocked.

Other combination is mixing some blacklists into file /etc/suricata/rules/iprep/test-iprep.list. Here IP 95.141.17.10 will be in line # 3136 ( 95.141.17.10,31,100 ). Then restart Suricata and test access from TEST PC behind Suricata, the IP it is NOT being blocked.

However, if I add same IP to file /etc/suricata/rules/iprep/test-iprep.list , as line # 12 ( 95.141.17.10,31,100 ), then restart Suricata and test access from TEST PC behind Suricata, the IP is successfully blocked.

I’m still modifying variables under Host Table to see if with higher values would help IPREP mechanism to work properly.

Current values are:

# Host table:
#
# Host table is used by the tagging and per host thresholding subsystems.
#
host:
  hash-size: 4194304
  prealloc: 1024000
  memcap: 16384mb

Do you advise to keep increasing (doubling) all values or should I stop and try something else? Variable memcap just reached 16GB, but the issue has not been resolved yet, nor I can notice any other impact in Suricata.

Thank you

Think increasing those as mentioned by @vjulien should do the trick then.

The amount of space used by memcap is taken from HDD or from RAM?

RAM while Suricata is running.

Thank you @syoc . That’s was I was suspecting, since now I see the swap and also CPU increasing consumption for suricata process.

Current values for Host table are the following:

host:
  hash-size: 536870912
  prealloc: 131072000
  memcap: 2097152mb

After last restart Suricata is not coming up yet (got stuck). I am not an expert, but for me, the above values are huge already. I am not sure if @vjulien agree with me. Current config it is starting to impact the machine performance.

I think I have reached the limit. Perhaps it is time to start trying something different.

Any recommendations @vjulien ? If you need more details, please let me know.

Thank you

Another update

Currently using only one categories file and two reputation files

# IP Reputation
reputation-categories-file: /etc/suricata/rules/scirius-categories.txt
default-reputation-path: /etc/suricata/rules/iprep/
reputation-files:
 - scirius-iprep.list
 - test-iprep.list

I mixed all blacklists into test-iprep.list , having around 50500 IPs in the list. List includes the IPs I used as small sample in previous post. I am including their line numbers

10067	176.221.42.32,31,100
6326	113.212.69.128,31,100
4177	108.62.59.27,31,100
3370	95.141.17.244,31,100
33835	216.151.137.155,36,100
32561	173.234.225.161,36,100
30574	108.62.56.222,36,100
30106	95.141.17.10,36,100

None of these IPs are being blocked by Suricata. Same result if I leave all blacklists separated in individual iprep lists.

However if I move these IPs next to the top in the same list, then restart Suricata, the same IPs are being blocked as expected. Again, I am not an expert, but it looks to me that Suricata is not reading/loading/processing the entire list.

After few tests I have narrowed the working range up to 1170 lines of IPs within the list. If IP falls beyond that point, it won’t be blocked. That’s not a large number of IPs to check.

Unfortunately solution suggested by Victor Julien did not work.

In this article, somebody was testing iprep with data sets up to million entries with positive results! Having a small list with only 1170 entries vs a million entries working fine, makes me think that perhaps Suricata is not reading values properly from the Host table or Host table mechanism is not working properly. Is there a way to check the host table or a command to force loading data (other that USR2 or restart Suricata)?

Any other ideas or anything else I could check or try?

Thanks

Hi,

While you guys are still researching what could be wrong with IPREP in my Suricata, I would like to try a parallel approaching so I can keep moving forward in my project.

I have been reading about datasets and apparently I could achieve the same goal (block many IPs given one or several blacklists) with a mechanism similar to IPREP. The problem is that after reading the official documentation I still have a lot of questions and I would appreciate if somebody could give me a hand and some clues.

I have configured a dataset in /etc/suricata/selks6-addin.yaml. I think dataset keyword is easier than datarep. Please check and comment what I have done so far:

datasets:
  defaults:
    memcap: 100mb
    hashsize: 2048
  bad-ip :
     type: string
     state: bad-ip.lst

Also I am not sure about the fields for the rules, but I want to build something similat to IPREP. Please check and comment what I have done so far:

From original doc …

alert http any any -> any any (msg: “http user-agent test”; http.user_agent; dataset:set,ua-seen; sid:234; rev:1;)

Customizing …

drop ip any any -> any any (msg:"TEST Bad IPs"; dataset:isset, bad-ip; sid:10; rev:1;)

I’m not sure what other values I can use for the field where dns.query is, so I am just guessing ip . Not sure if this make sense.

If somebody could please help me with these settings.

Thank you in advance

I have been reading a lot about dataset keyword in Suricata, but I’m still unable to configure it properly.

I have configured a dataset in /etc/suricata/selks6-addin.yaml .

datasets:
  defaults:
    memcap: 100mb
    hashsize: 2048
  test-badip :
     type: string
     state: test-badip.lst

Restarted Suricata

And also I have created two signatures:

dataset
alert ip any any -> any any (msg:"TEST Bad IP"; dataset:isset,test-badip; sid:10; rev:1;)

GeoIP
alert ip any any -> any any (msg:"TEST Bad GeoIP"; geoip:any,RU,CN,KR,KP,UA; sid:11; rev:1;)

Both signatures are rejected by Suricata when attempt to load them.

  • Where can I get a list of available “sticky buffers” that I can use in the dataset signature (specially to get the IP address for the signature)?
  • I have tried different combinations for geoip and I have compared my rule against some examples I have found on the Net. I can’t find what it’s wrong with my geoip signature.

Can somebody please help me whit this?

Thanks

Try running Suricata with --list-keywords. Then check the documentation for the keyword you want to use to see if it is sticky buffer. Stick buffer keywords are used before the corresponding content match.

I do not think that dataset feature can be used with IP matching, like you are trying to do.
IPrep is really the way to go, honestly not sure why it’s not working with a not very large list.

As for the GeoIP issue, can you post the message suricata prints when rejecting the geoip signatures?

Hi @syoc ,

Thank you for answering. At this point I have read a lot of documentation (including the official doc), but there are parts or details that are not clear to me. I wish I could do it on my own, but so far I keep failing. That is why I am asking for help here.

Having IPREP working would be great! I do think it is the right engine to be used. I have tried everything I can, with many different combinations and also tried recommendations from Suricata and SELKS teams, but it looks like there is a bug and now I am stuck. So I decided to try Peter’s advice from here:

Apparently datasets can be used for the same purpose. The mechanism is similar to IPREP. So I am trying hard to have it operational, but again I’m facing errors due to my lack of knowledge.

Here are the last errors I got. Please remember that I am running Suricata as part of SELKS suite, so I must enter rules via Scirius or I might lose them after a Scirius restart.

* **SC_ERR_DATASET** : dataset test-badip not defined
* **SC_ERR_INVALID_SIGNATURE** : failed to set up dataset 'test-badip'.
* **SC_ERR_INVALID_SIGNATURE** : error parsing signature "alert ip any any -> any any (msg:"TEST Bad IP"; ipv4.hdr; dataset:isset,test-badip; sid:10; rev:1;) "
* **SC_ERR_INVALID_SIGNATURE** : error parsing signature "alert ip any any -> any any (msg:"TEST Bad GeoIP"; geoip:RU,CN,KR,KP,UA; sid:11; rev:1;)"

I also tried to add a dataset as described here, from another Suricata forum post, but failed as well.

* **SC_ERR_DATASET** : dataset dns-seen not defined
* **SC_ERR_INVALID_SIGNATURE** : failed to set up dataset 'dns-seen'.
* **SC_ERR_INVALID_SIGNATURE** : error parsing signature "alert dns any any -> any any (msg: "dns list test"; dns.query; dataset:isset,dns-seen; sid:123; rev:1;)"

I really appreciate if you or somebody else could help me.

Thank you

Ah, never knew about the ipv4.hdr buffer. Very interesting.

The issue, however is that you are matching against the raw ipv4 header.
This means that the source and destination IP is not represented as a plaintext IP.
If you want a rule to match traffic towards 8.8.8.8 for instance you need a rule with ipv4.hdr; content:"|08 08 08 08|"; as you are matching the binary content directly.

Can you upload you iprep list that’s not working somewhere?