Suricata and IP blacklist

Well, Pevma got ahead of me and already published the post with SELKS’s answer :sweat_smile:. Thank you

Hi there,

I had to pause my researching due to hardware issues. Issues were resolved and now I can move forward.

Summarizing what I have so far:

  • Running Suricata v6 and v7 as part of SELKS 6 suite
  • Setup Suricata in IPS mode
  • I am implementing the L2 config approaching (af-packet). Machine 150 is hosting Suricata, so I want Suricata to ignore that traffic for detection.

selks6-interfaces-config.yaml

af-packet:
  - interface: enxa0cec8d92d70
    threads: 1
    defrag: yes
    cluster-type: cluster_flow
    cluster-id: 98
    copy-mode: ips
    copy-iface: enxa0cec8d92e2e
    tpacket-v3: no
    ring-size: 2048
    buffer-size: 64535
    use-mmap: yes
    bpf-filter: not host 192.168.1.150
 
 - interface: enxa0cec8d92e2e
    threads: 1
    cluster-id: 97
    defrag: yes
    cluster-type: cluster_flow
    copy-mode: ips
    copy-iface: enxa0cec8d92d70
    tpacket-v3: no
    ring-size: 2048
    buffer-size: 64535
    use-mmap: yes
    bpf-filter: not host 192.168.1.150

pcap:
  - interface: enxa0cec8d92d70
    cluster-id: 98
    bpf-filter: "not host 192.168.1.150"
  - interface: enxa0cec8d92e2e
    cluster-id: 97
    bpf-filter: "not host 192.168.1.150"

pfring:
  - interface: enxa0cec8d92d70
    cluster-id: 98
    bpf-filter: not host 192.168.1.150
  - interface: enxa0cec8d92e2e
    cluster-id: 97
    bpf-filter: not host 192.168.1.150

netmap:
  - interface: enxa0cec8d92d70
    cluster-id: 98
    bpf-filter: not host 192.168.1.150
  - interface: enxa0cec8d92e2e
    cluster-id: 97
    bpf-filter: not host 192.168.1.150
  • Added custom Suricata rule via Scirius (webGUI for SELKS)
  • Added to Suricata config

# IP Reputation
reputation-categories-file: /etc/suricata/rules/scirius-categories.txt
default-reputation-path: /etc/suricata/rules
reputation-files:
- scirius-iprep.list
- test-iprep.list

default-rule-path: /etc/suricata/rules
rule-files:
- scirius.rules

scirius-categories.txt
1,2400000,ET DROP Spamhaus DROP Listed Traffic Inbound
2,2402000,ET DROP Dshield Block Listed Source
3,2403300,ET CINS Active Threat Intelligence Poor Reputation IP
4,2404000,ET CNC Shadowserver Reported CnC Server IP
5,2404029,ET CNC Shadowserver Reported CnC Server
6,2404300,ET CNC Feodo Tracker Reported CnC Server
7,2500000,ET COMPROMISED Known Compromised or Hostile Host Traffic
8,2520000,ET TOR Known Tor Exit Node Traffic
9,2522000,ET TOR Known Tor Relay/Router (Not Exit) Node Traffic
10,2525000,ET 3CORESec Poor Reputation IP
30,TESTBadIP,TEST Known Bad IP Reputation

test-iprep.list
149.20.4.15,30,100

scirius.rules
# Rules file for Default SELKS ruleset generated by Scirius at 2021-01-26 15:49:40.671556+00:00
drop ip any any -> any any (msg:"TEST IP Bad Reputation Blacklist"; iprep:any,TESTBadIP,>,99; sid:1; rev:1;)

  • The internal af-packet bridge in Suricata seems to be working fine, as I can reach a test machine behind Suricata and also that machine can reach Internet

So far so good.
Now I am testing the only rule enabled in Suricata, which should block IP 149.20.4.15 (www.debian.org). Well, traffic is not being blocked. What could I be doing wrong or missing?

Thank you

Hi,

I have Suricata running in IPS mode (L2 configuration) already. I was missing a couple of settings in to convert rules from alert to drop. After that I have been running tests. All rules are IP-Only using the iprep control. During these tests all other Suricata rules are disabled.

First I just added an IP reputation list with a couple of IPs. All were successfully blocked. after that I decided to try next level and automate the sources. Downloaded a few lists from Internet and parsed all list to match the categories within Suricata. The last results showed me that not all the traffic from the new reputation list it’s being blocked. Does anybody have any idea about what could be failing or what should I check? If you need more details, let me know.

Please help!

Thanks

Hi,

I continued trying different things. This morning I have been trying this new config, combining several feeds into one iprep file:

  • Removed all iprep files
  • Left only two iprep files in
  • Added all feeds into test-iprep.list
  • Restarted Suricata
  • Tried access from test machine behind Suricata

Results:

  • Custom iprep file has now 1117 lines
  • First 5 sites (IPs added manually) remain being blocked
  • At least 8 IPs (this is just a small sample) from different feeds where NOT blocked while were in separate iprep files, and same IPs are not being blocked now that are in the same iprep file.
  • I did an additional test leaving in the file test-iprep.list only the 5 IPs added manually plus the 8 IPs in conflict. This time the 8 IPs in conflict were successfully blocked. I kept their categories within the iprep file:
149.20.4.15,30,100
128.31.0.62,30,100
176.221.42.32,31,100
113.212.69.128,31,100
108.62.59.27,31,100
95.141.17.244,31,100
216.151.137.155,36,100
173.234.225.161,36,100
108.62.56.222,36,100
95.141.17.10,36,100

This leads me to believe that Suricata is not fully reading the iprep files, or that there is a limit to the number of lines it can read / load.

Is there anything we can do to fix this issue? Is there a variable or config to modify/remove this limit?

Please help!

Thank you

I think it may not be reading in all lists as you suggest , might worth to open an issue on the suricata redmine tracker.

Oh, that’s a pity. I was hopping that it was something I missed or configured incorrectly.
Could you confirm/reproduce the same issue on your end?

Is this the site to report a bug?
https://redmine.openinfosecfoundation.org/projects/suricata/issues?set_filter=1&tracker_id=1

Report submitted
https://redmine.openinfosecfoundation.org/issues/4280

If somebody could, please tell me if datasets will serve the same purpose, if this could avoid the problem I have right now, and if somebody could give me more details about its implementation.

Thank you

The IP rep functionality uses the host table, which by default has a fairly low memcap. Have you tried increasing that to see if it changes anything?

Hi and thank you for answering.

What a coincidence! I was just checking the following article

Speed

I’ve been playing with data sets of up to a million entries. Loading it takes hardly any time and I’m confident larger numbers will work just fine. The host table just needs bigger memcaps and hash sizes.

and I was wondering if what you suggest could help.

I was about to ask you guys what values would you recommend to start testing?
Current values:

# Host table:
#
# Host table is used by the tagging and per host thresholding subsystems.
#
host:
  hash-size: 4096
  prealloc: 1000
  memcap: 32mb

Thank you in advance!

I’d say double it, test it, repeat, until it works :slight_smile:

I’ll start working on it! Question: In each iteration is it advisable to double the values of the three variables or only the first two?

Thanks

The memcap is the only critical one, but I think its good to double all of them to avoid possible performance issues.

OK, thanks. I’ll share feedback later

I misunderstood that when all are in one file it behaves properly , otherwise not - sorry about this!

Hi,

Just to be in the same page, when all the IPs are in a single list AND there are just a few lines (I don’t know what the limit is), the blocking mechanism works as expected.

But, by mixing all the lists into a single list, without any limits, the number of lines amounts to more than 50,000 and only a few IPs are blocked.

Hi @vjulien and @pevma,

Some updates:

Take for instance IP 95.141.17.10 . If I add that IP to either file /etc/suricata/rules/scirius-iprep.list or /etc/suricata/rules/iprep/scirius-iprep.list , as line # 21605 ( 95.141.17.10,31,100 ), then restart Suricata and test access from TEST PC behind Suricata, the IP it is NOT being blocked.

Other combination is mixing some blacklists into file /etc/suricata/rules/iprep/test-iprep.list. Here IP 95.141.17.10 will be in line # 3136 ( 95.141.17.10,31,100 ). Then restart Suricata and test access from TEST PC behind Suricata, the IP it is NOT being blocked.

However, if I add same IP to file /etc/suricata/rules/iprep/test-iprep.list , as line # 12 ( 95.141.17.10,31,100 ), then restart Suricata and test access from TEST PC behind Suricata, the IP is successfully blocked.

I’m still modifying variables under Host Table to see if with higher values would help IPREP mechanism to work properly.

Current values are:

# Host table:
#
# Host table is used by the tagging and per host thresholding subsystems.
#
host:
  hash-size: 4194304
  prealloc: 1024000
  memcap: 16384mb

Do you advise to keep increasing (doubling) all values or should I stop and try something else? Variable memcap just reached 16GB, but the issue has not been resolved yet, nor I can notice any other impact in Suricata.

Thank you

Think increasing those as mentioned by @vjulien should do the trick then.

The amount of space used by memcap is taken from HDD or from RAM?

RAM while Suricata is running.

Thank you @syoc . That’s was I was suspecting, since now I see the swap and also CPU increasing consumption for suricata process.

Current values for Host table are the following:

host:
  hash-size: 536870912
  prealloc: 131072000
  memcap: 2097152mb

After last restart Suricata is not coming up yet (got stuck). I am not an expert, but for me, the above values are huge already. I am not sure if @vjulien agree with me. Current config it is starting to impact the machine performance.

I think I have reached the limit. Perhaps it is time to start trying something different.

Any recommendations @vjulien ? If you need more details, please let me know.

Thank you