Suricatasc error: Reload already in progress

Hello,

Just modified some rules and tried to reload them, but this fails via suricatasc.

ruleset-stats
Success:
[
{
“id”: 0,
“rules_failed”: 0,
“rules_loaded”: 64310
}
]
reload-rules
Error:
“Reload already in progress”
quit

How to debug/fix?
Cheers,
Andre

  • System
    Red Hat Enterprise Linux release 8.7 (Ootpa) 4.18.0-425.13.1.el8_7.x86_64
    SElinux enforcing but permissive setting does not help suricatsc
    Suricata version 7.0.0-rc2-dev (6487c689f 2023-05-07)

  • suricata --build-info
    This is Suricata version 7.0.0-rc2-dev (6487c689f 2023-05-07)
    Features: PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HTTP2_DECOMPRESSION HAVE_LIBJANSSON TLS TLS_C11 MAGIC RUST
    SIMD support: SSE_4_2 SSE_4_1 SSE_3
    Atomic intrinsics: 1 2 4 8 16 byte(s)
    64-bits, Little-endian architecture
    GCC version 8.5.0 20210514 (Red Hat 8.5.0-16), C version 201112
    compiled with -fstack-protector
    compiled with _FORTIFY_SOURCE=2
    L1 cache line size (CLS)=64
    thread local storage method: _Thread_local
    compiled with LibHTP v0.5.43, linked against LibHTP v0.5.42

Suricata Configuration:
AF_PACKET support: yes
AF_XDP support: no
DPDK support: yes
eBPF support: no
XDP support: no
PF_RING support: no
NFQueue support: no
NFLOG support: no
IPFW support: no
Netmap support: no
DAG enabled: no
Napatech enabled: no
WinDivert enabled: no

Unix socket enabled: yes
Detection enabled: yes

Libmagic support: yes
libjansson support: yes
hiredis support: no
hiredis async with libevent: no
PCRE jit: yes
LUA support: no
libluajit: no
GeoIP2 support: yes
Non-bundled htp: no
Hyperscan support: yes
Libnet support: no
liblz4 support: yes
Landlock support: no

Rust support: yes
Rust strict mode: no
Rust compiler path: /usr/bin/rustc
Rust compiler version: rustc 1.62.1 (Red Hat 1.62.1-1.module+el8.7.0+16002+ac58477b)
Cargo path: /usr/bin/cargo
Cargo version: cargo 1.62.1

Python support: yes
Python path: /usr/bin/python3
Install suricatactl: yes
Install suricatasc: yes
Install suricata-update: no, not bundled

Profiling enabled: no
Profiling locks enabled: no

Plugin support (experimental): yes

Development settings:
Coccinelle / spatch: no
Unit tests enabled: no
Debug output enabled: no
Debug validation enabled: no
Fuzz targets enabled: no

Generic build parameters:
Installation prefix: /usr
Configuration directory: /etc/suricata/
Log directory: /var/log/suricata/

–prefix /usr
–sysconfdir /etc
–localstatedir /var
–datarootdir /usr/share

Host: x86_64-pc-linux-gnu
Compiler: gcc (exec name) / g++ (real)
GCC Protect enabled: yes
GCC march native enabled: yes
GCC Profile enabled: no
Position Independent Executable enabled: no
CFLAGS -ggdb -fPIC -std=c11 -march=native -I/usr/local/include -include rte_config.h -march=native -I${srcdir}/…/rust/gen -I${srcdir}/…/rust/dist
PCAP_CFLAGS -I/usr/local/include
SECCFLAGS -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security

Based on log output, can you check if perhaps a previous reload is in fact not complete? I suppose this could happen if some of the dpdk interface have no traffic and our dpdk is missing some logic :slight_smile:

Yep, every night suricata-update runs and will reload in case of changes. And indeed, 1 out of 4 dpdk interfaces receives traffic, testing in progress. So better only use dpdk interfaces with traffic? If so, I also need to check cpu affinity after modifying the dpdk config in suricata.

Can you check the logs to see if it reports reloads completing? Should look something like

May 10 13:17:20 c2758 suricata[180592]: Notice: detect: rule reload starting [DetectEngineReload:detect-engine.c:4586]
May 10 13:19:00 c2758 suricata[180592]: Info: detect: 2 rule files processed. 66197 rules successfully loaded, 0 rules failed [SigLoadSignatures:detect-engine-loader.c:363]
May 10 13:19:01 c2758 suricata[180592]: Info: threshold-config: Threshold config parsed: 10 rule(s) found [SCThresholdConfParseFile:util-threshold-config.c:1112]
May 10 13:19:01 c2758 suricata[180592]: Info: detect: 66200 signatures processed. 1510 are IP-only rules, 10585 are inspecting packet payload, 53884 inspect application layer, 108 are decoder event only [SigAddressPrepareStage1:detect-engine-build.c:1456]
May 10 13:19:16 c2758 suricata[180592]: Notice: detect: rule reload complete [DetectEngineReload:detect-engine.c:4667]

10/5/2023 – 02:01:19 - – Enabled 184 rules for flowbit dependencies.
10/5/2023 – 02:01:19 - – Backing up current rules.
10/5/2023 – 02:01:19 - – Recording existing file /var/lib/suricata/rules/suricata.rules with hash ‘342836fe982f95196400fa6a73e933e5’.
10/5/2023 – 02:01:23 - – Writing rules to /var/lib/suricata/rules/suricata.rules: total: 82469; enabled: 64082; added: 11; removed 3; modified: 1529
10/5/2023 – 02:01:24 - – Loading /etc/suricata/classification.config
10/5/2023 – 02:01:24 - – Loading /usr/share/suricata/classification.config
10/5/2023 – 02:01:24 - – Loading rules/classification.config
10/5/2023 – 02:01:24 - – Writing /var/lib/suricata/rules/classification.config
10/5/2023 – 02:01:24 - – Skipping test, disabled by configuration.
10/5/2023 – 02:01:24 - – Running suricatasc -c ruleset-reload-nonblocking.
10/5/2023 – 02:01:24 - – Reload command exited with error: 1

I will test some scenarios to see when error:1 is triggered

This looks like suricata-update output, I’m looking for output from suricata itself.

Me too, but there is no suricata logging round 10/5/2023 – 02:01:24 ?!? Maybe start suricata service with -vvv option

Ok, just started suricata service with extra options -vvv , expect suricata-update rond 2am to succeed but a manual reload tomorrow during working hours to fail with an error 1. I’ll be back.

Ok think I’ve found a possible lead in logrotate. suricata-update runs at 2am, all good but around 03:12am suricata still runs but eve.json files are not updating anymore.
And indeed suricatsc just now:
reload-rules
Error:
“Reload already in progress”

This logrote.conf worked for suricata v6, but shreds suricata 7 ?
/var/log/suricata/.log /data/sensor_data/suricata/eve..json /data/sensor_data/suricata/stats.log
{
daily
missingok
rotate 2
compress
minsize 500k
sharedscripts
postrotate
/bin/kill -HUP cat /var/run/suricata.pid 2> /dev/null 2> /dev/null || true
endscript
}

2013 strikes again? :wink:
Bug #911: HUP signal

Still hoping to see the suricata output. We need it to determine if suricata itself believes it completed.

There is no logging at the moment supreme. Maybe better to start suricata in a screen console session? Or better ideas?

Maybe a clue overhere when restarting suricata via systemd?

May 11 13:16:23 systemd[1]: Stopping Suricata Intrusion Detection Service…
– Subject: Unit suricata.service has begun shutting down
– Defined-By: systemd
– Support: https://access.redhat.com/support
**-- **
– Unit suricata.service has begun shutting down.
May 11 13:17:53 systemd[1]: suricata.service: State ‘stop-sigterm’ timed out. Killing.
May 11 13:17:53 systemd[1]: suricata.service: Killing process 1755717 (Suricata-Main) with signal SIGKILL.
May 11 13:17:57 systemd[1]: suricata.service: Main process exited, code=killed, status=9/KILL
May 11 13:17:57 systemd[1]: suricata.service: Failed with result ‘timeout’.
– Subject: Unit failed
– Defined-By: systemd
– Support: Red Hat Customer Experience & Engagement - Red Hat Customer Portal
**-- **
– The unit suricata.service has entered the ‘failed’ state with result ‘timeout’.
May 11 13:17:57 systemd[1]: Stopped Suricata Intrusion Detection Service.
– Subject: Unit suricata.service has finished shutting down
– Defined-By: systemd
– Support: Red Hat Customer Experience & Engagement - Red Hat Customer Portal
**-- **
– Unit suricata.service has finished shutting down.
May 11 13:46:57 systemd[1]: Starting Suricata Intrusion Detection Service…
– Subject: Unit suricata.service has begun start-up
– Defined-By: systemd
– Support: Red Hat Customer Experience & Engagement - Red Hat Customer Portal
**-- **
– Unit suricata.service has begun starting up.
May 11 13:46:57 systemd[1]: Started Suricata Intrusion Detection Service.
– Subject: Unit suricata.service has finished start-up
– Defined-By: systemd
– Support: Red Hat Customer Experience & Engagement - Red Hat Customer Portal
**-- **
– Unit suricata.service has finished starting up.

You could journalctl -xf -u suricata.

Yep i know, thanks, but in my opinion it does not show a clue but but more retrospective about that point I’m not a developper :sunglasses: I’ve attached a tar archive with logging (tar cjf)
surlogs.tgz (36.3 KB)

This confirms that the reload doesn’t complete from suricata’s pov.

1 Like

Thanks for confirming I’m officially blind :wink: Great this is picked up so swift, great release v7!

This is now fixed in the master branch.