Suricata 8.0.7 with NFQUEUE causes outbound TCP connections to hang, even with no OUTPUT queue

Help
,
1

Hello,

I’m running Suricata in IPS mode on a public Debian server, using nftables and NFQUEUE. When Suricata is running, new outbound TCP connections from the server can hang, even though the NFQUEUE rule is in the INPUT chain only.

Stopping Suricata immediately restores normal outbound connectivity.

System

  • Debian 13 (Trixie)
  • Suricata 8.0.7
  • nftables
  • NFQUEUE queue 0
  • Suricata started with -q 0
  • IPS configuration:
nfq:
  mode: accept
  fail-open: yes

suricata -T succeeds and Suricata processes packets normally.

Problem

The clearest example is a PHP application on the server which makes an HTTPS request to OpenWeatherMap.

With Suricata running:

CONNECT=0.000000s START=0.000000s TOTAL=10.002769s

The curl command was using --connect-timeout 10, so the connection never completed.

With Suricata stopped, the same connection establishes immediately:

* Trying 141.95.47.140:443...
* Connected to api.openweathermap.org (141.95.47.140) port 443

DNS resolution works correctly, and routing is correct:

141.95.47.139 via 147.135.136.254 dev eno1 src 147.135.136.17 uid 0

The destination IP can change because OpenWeatherMap has multiple addresses.

A tcpdump running on any during the failed connection did not show the expected TCP SYN.

Important nftables detail

The permanent NFQUEUE rule is in the INPUT chain:

chain input {
    type filter hook input priority filter; policy drop;

    ip saddr @admin_ip accept
    iif "lo" accept

    counter packets 942605 bytes 113912625 queue flags bypass to 0

    ct state established,related accept
    ip protocol icmp accept
    ip6 nexthdr ipv6-icmp accept
    meta nfproto ipv6 drop

    ...
}

The complete relevant structure is:

table inet filter {
    set admin_ip {
        type ipv4_addr
        elements = { 87.88.33.47 }
    }

    chain input_early {
        type filter hook input priority -20; policy accept;
        ip saddr @admin_ip accept
    }

    chain input {
        type filter hook input priority filter; policy drop;

        ip saddr @admin_ip accept
        iif "lo" accept

        counter packets 942605 bytes 113912625 queue flags bypass to 0

        ct state established,related accept
        ip protocol icmp accept
        ip6 nexthdr ipv6-icmp accept
        meta nfproto ipv6 drop

        tcp dport 6969 accept
        tcp dport { 80, 443 } accept
        tcp dport { 25, 110, 143, 465, 587, 993, 995 } accept
        tcp dport 10000 ip saddr @admin_ip accept
        tcp dport 20000 accept
        tcp dport 8443 accept
        tcp dport 8081 accept

        ct state new tcp flags syn tcp dport 35565-35569 limit rate 15/second burst 30 packets accept
        ct state new tcp flags syn tcp dport 35565-35569 drop
        tcp dport 35565-35569 accept

        tcp flags syn ct state new limit rate over 60/second burst 100 packets drop
        tcp flags syn ct state new limit rate over 30/second burst 50 packets drop

        tcp dport 10050 ip saddr @admin_ip accept

        counter packets 1834164 bytes 119245484 drop
    }

    chain forward {
        type filter hook forward priority filter; policy drop;
        ct state established,related accept
        iifname "pterodactyl0" accept
        oifname "pterodactyl0" accept
    }

    chain output {
        type filter hook output priority filter; policy accept;

        # Temporary test only:
        ct state new tcp dport 443 queue flags bypass to 0
    }
}

The OUTPUT NFQUEUE rule above was added only as a diagnostic test. It did not solve the problem. The outbound HTTPS connection still timed out with Suricata running.

The normal OUTPUT policy is:

policy accept

and:

iptables -S OUTPUT
-P OUTPUT ACCEPT

Another important observation

My home IP (87.88.33.47) is in admin_ip.

Therefore incoming connections from my home PC are accepted here:

ip saddr @admin_ip accept

before reaching the NFQUEUE rule.

Nevertheless, I initially observed significant delays in some HTTP/API connections from my home PC.

For example, the main website was fast:

https://www.extra-ordinaire.com/

with curl taking about 0.08 s.

The API endpoint:

https://lesmdc.net/meg-e/src/

could take about 134 seconds.

The reason for that particular API delay was subsequently identified as a PHP cURL call to OpenWeatherMap without a timeout. The OpenWeatherMap connection itself fails only while Suricata is running, so this application-level delay is a consequence rather than the root cause.

NFQUEUE status

The NFQUEUE itself does not appear saturated.

/proc/net/netfilter/nfnetlink_queue showed:

0 709319     0 2 65531     0     0       30  1

The queue length is zero.

Suricata counters

The latest dump-counters while Suricata was running showed:

rules_loaded: 53115
ips.accepted: 3449
ips.blocked: 1023
ips.rejected: 0
drop_reason.nfq_error: 0
detect.alert: 20
detect.alert_queue_overflow: 0
RX-NFQ#0.decoder.pkts: 4473
TX#00.ips.accepted: 3449
TX#00.ips.blocked: 1023

There were no NFQUEUE errors reported.

What I have tested

  • DNS resolution: OK
  • Routing: OK
  • ICMP: normal, approximately 7–9 ms from my home connection
  • Normal inbound HTTPS: works
  • Outbound HTTPS with Suricata stopped: works immediately
  • Outbound HTTPS with Suricata running: TCP connection hangs
  • NFQUEUE queue is not full
  • OUTPUT nftables policy is ACCEPT
  • iptables OUTPUT is ACCEPT
  • Adding a temporary OUTPUT -> NFQUEUE 0 rule did not solve the problem
  • tcpdump does not see the expected outbound SYN while Suricata is running
  • Stopping Suricata immediately restores outbound connectivity

Question

I would like to understand why simply running Suricata with NFQUEUE on INPUT can prevent new outbound TCP connections from being established, even though:

  1. OUTPUT is ACCEPT;
  2. there is no permanent OUTPUT NFQUEUE rule;
  3. the INPUT NFQUEUE queue is not saturated;
  4. fail-open: yes is configured;
  5. the outbound SYN does not appear in tcpdump when Suricata is running.

Is this a known interaction between Suricata 8.0.7, NFQUEUE, conntrack and nftables?

Is there something specific I should check regarding NFQUEUE/conntrack state, hook ordering, or Suricata’s NFQ handling?

I have currently stopped Suricata to restore normal outbound connectivity.

Thanks for any help.

2

I have just upgraded the server from Suricata 8.0.7 to Suricata 8.0.8 on Debian 13 (Trixie), and unfortunately I can confirm that the NFQUEUE issue I previously reported is still reproducible.

After the upgrade, Suricata was started again with my existing NFQUEUE configuration.

Immediately afterwards, outbound HTTPS connections from the server started failing again.

For example:

curl -I https://deb.debian.org/debian/
curl: (35) Recv failure: Connection reset by peer

APT showed similar failures against multiple unrelated HTTPS repositories:

Could not wait for server fd - select (11: Resource temporarily unavailable)

This affected Debian, Rspamd, Elastic, Virtualmin, EveBox and packagecloud repositories, so it was clearly not specific to a remote server.

I then stopped Suricata:

systemctl stop suricata

Without changing anything else, the same test immediately worked:

HTTP/2 200
server: Apache
...

So the behaviour is the same as with 8.0.7:

  • Debian 13

  • Suricata 8.0.8

  • nftables

  • Suricata using NFQUEUE

  • NFQUEUE rule located in the INPUT path

  • outbound TCP/HTTPS connections from the server are nevertheless affected

  • stopping Suricata immediately restores normal outbound connectivity

Therefore, at least with my configuration, the issue does not appear to be fixed in 8.0.8.

There was also a separate packaging/dependency issue during the upgrade.

Installing Suricata 8.0.8 from trixie-backports installed libvectorscan5 and removed both libhyperscan5 and Rspamd:

The following additional packages will be installed:
  libvectorscan5

The following packages will be REMOVED:
  libhyperscan5 rspamd

As a consequence, Rspamd was no longer installed after the Suricata upgrade.

To restore mail filtering, I had to remove Suricata again and reinstall Rspamd, which in turn reinstalled libhyperscan5 and removed libvectorscan5.

So there appear to be two separate issues here:

  1. the NFQUEUE connectivity problem is still reproducible with Suricata 8.0.8;

  2. the Debian 13 package dependency change from Hyperscan to Vectorscan conflicts with the currently installed Rspamd package and causes Rspamd to be removed during the upgrade.

I can provide the current nftables ruleset, Suricata NFQUEUE configuration and additional package metadata/debug output if useful.