Suricatasc error: Reload already in progress

atbohmer · May 9, 2023, 1:55pm

Hello,

Just modified some rules and tried to reload them, but this fails via suricatasc.

ruleset-stats
Success:
[
{
“id”: 0,
“rules_failed”: 0,
“rules_loaded”: 64310
}
]
reload-rules
Error:
“Reload already in progress”
quit

How to debug/fix?
Cheers,
Andre

System
Red Hat Enterprise Linux release 8.7 (Ootpa) 4.18.0-425.13.1.el8_7.x86_64
SElinux enforcing but permissive setting does not help suricatsc
Suricata version 7.0.0-rc2-dev (6487c689f 2023-05-07)
suricata --build-info
This is Suricata version 7.0.0-rc2-dev (6487c689f 2023-05-07)
Features: PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HTTP2_DECOMPRESSION HAVE_LIBJANSSON TLS TLS_C11 MAGIC RUST
SIMD support: SSE_4_2 SSE_4_1 SSE_3
Atomic intrinsics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 8.5.0 20210514 (Red Hat 8.5.0-16), C version 201112
compiled with -fstack-protector
compiled with _FORTIFY_SOURCE=2
L1 cache line size (CLS)=64
thread local storage method: _Thread_local
compiled with LibHTP v0.5.43, linked against LibHTP v0.5.42

Suricata Configuration:
AF_PACKET support: yes
AF_XDP support: no
DPDK support: yes
eBPF support: no
XDP support: no
PF_RING support: no
NFQueue support: no
NFLOG support: no
IPFW support: no
Netmap support: no
DAG enabled: no
Napatech enabled: no
WinDivert enabled: no

Unix socket enabled: yes
Detection enabled: yes

Libmagic support: yes
libjansson support: yes
hiredis support: no
hiredis async with libevent: no
PCRE jit: yes
LUA support: no
libluajit: no
GeoIP2 support: yes
Non-bundled htp: no
Hyperscan support: yes
Libnet support: no
liblz4 support: yes
Landlock support: no

Rust support: yes
Rust strict mode: no
Rust compiler path: /usr/bin/rustc
Rust compiler version: rustc 1.62.1 (Red Hat 1.62.1-1.module+el8.7.0+16002+ac58477b)
Cargo path: /usr/bin/cargo
Cargo version: cargo 1.62.1

Python support: yes
Python path: /usr/bin/python3
Install suricatactl: yes
Install suricatasc: yes
Install suricata-update: no, not bundled

Profiling enabled: no
Profiling locks enabled: no

Plugin support (experimental): yes

Development settings:
Coccinelle / spatch: no
Unit tests enabled: no
Debug output enabled: no
Debug validation enabled: no
Fuzz targets enabled: no

Generic build parameters:
Installation prefix: /usr
Configuration directory: /etc/suricata/
Log directory: /var/log/suricata/

–prefix /usr
–sysconfdir /etc
–localstatedir /var
–datarootdir /usr/share

Host: x86_64-pc-linux-gnu
Compiler: gcc (exec name) / g++ (real)
GCC Protect enabled: yes
GCC march native enabled: yes
GCC Profile enabled: no
Position Independent Executable enabled: no
CFLAGS -ggdb -fPIC -std=c11 -march=native -I/usr/local/include -include rte_config.h -march=native -I${srcdir}/…/rust/gen -I${srcdir}/…/rust/dist
PCAP_CFLAGS -I/usr/local/include
SECCFLAGS -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security

vjulien · May 9, 2023, 5:43pm

Based on log output, can you check if perhaps a previous reload is in fact not complete? I suppose this could happen if some of the dpdk interface have no traffic and our dpdk is missing some logic

atbohmer · May 10, 2023, 11:18am

Yep, every night suricata-update runs and will reload in case of changes. And indeed, 1 out of 4 dpdk interfaces receives traffic, testing in progress. So better only use dpdk interfaces with traffic? If so, I also need to check cpu affinity after modifying the dpdk config in suricata.

vjulien · May 10, 2023, 11:34am

Can you check the logs to see if it reports reloads completing? Should look something like

May 10 13:17:20 c2758 suricata[180592]: Notice: detect: rule reload starting [DetectEngineReload:detect-engine.c:4586]
May 10 13:19:00 c2758 suricata[180592]: Info: detect: 2 rule files processed. 66197 rules successfully loaded, 0 rules failed [SigLoadSignatures:detect-engine-loader.c:363]
May 10 13:19:01 c2758 suricata[180592]: Info: threshold-config: Threshold config parsed: 10 rule(s) found [SCThresholdConfParseFile:util-threshold-config.c:1112]
May 10 13:19:01 c2758 suricata[180592]: Info: detect: 66200 signatures processed. 1510 are IP-only rules, 10585 are inspecting packet payload, 53884 inspect application layer, 108 are decoder event only [SigAddressPrepareStage1:detect-engine-build.c:1456]
May 10 13:19:16 c2758 suricata[180592]: Notice: detect: rule reload complete [DetectEngineReload:detect-engine.c:4667]

atbohmer · May 10, 2023, 11:51am

10/5/2023 – 02:01:19 - – Enabled 184 rules for flowbit dependencies.
10/5/2023 – 02:01:19 - – Backing up current rules.
10/5/2023 – 02:01:19 - – Recording existing file /var/lib/suricata/rules/suricata.rules with hash ‘342836fe982f95196400fa6a73e933e5’.
10/5/2023 – 02:01:23 - – Writing rules to /var/lib/suricata/rules/suricata.rules: total: 82469; enabled: 64082; added: 11; removed 3; modified: 1529
10/5/2023 – 02:01:24 - – Loading /etc/suricata/classification.config
10/5/2023 – 02:01:24 - – Loading /usr/share/suricata/classification.config
10/5/2023 – 02:01:24 - – Loading rules/classification.config
10/5/2023 – 02:01:24 - – Writing /var/lib/suricata/rules/classification.config
10/5/2023 – 02:01:24 - – Skipping test, disabled by configuration.
10/5/2023 – 02:01:24 - – Running suricatasc -c ruleset-reload-nonblocking.
10/5/2023 – 02:01:24 - – Reload command exited with error: 1

atbohmer · May 10, 2023, 11:52am

I will test some scenarios to see when error:1 is triggered

vjulien · May 10, 2023, 2:46pm

This looks like suricata-update output, I’m looking for output from suricata itself.

atbohmer · May 10, 2023, 3:49pm

Me too, but there is no suricata logging round 10/5/2023 – 02:01:24 ?!? Maybe start suricata service with -vvv option

atbohmer · May 10, 2023, 3:56pm

Ok, just started suricata service with extra options -vvv , expect suricata-update rond 2am to succeed but a manual reload tomorrow during working hours to fail with an error 1. I’ll be back.

atbohmer · May 11, 2023, 8:22am

Ok think I’ve found a possible lead in logrotate. suricata-update runs at 2am, all good but around 03:12am suricata still runs but eve.json files are not updating anymore.
And indeed suricatsc just now:
reload-rules
Error:
“Reload already in progress”

This logrote.conf worked for suricata v6, but shreds suricata 7 ?
/var/log/suricata/.log /data/sensor_data/suricata/eve..json /data/sensor_data/suricata/stats.log
{
daily
missingok
rotate 2
compress
minsize 500k
sharedscripts
postrotate
/bin/kill -HUP cat /var/run/suricata.pid 2> /dev/null 2> /dev/null || true
endscript
}

atbohmer · May 11, 2023, 9:55am

2013 strikes again?
Bug #911: HUP signal

vjulien · May 11, 2023, 11:57am

Still hoping to see the suricata output. We need it to determine if suricata itself believes it completed.

atbohmer · May 11, 2023, 12:04pm

There is no logging at the moment supreme. Maybe better to start suricata in a screen console session? Or better ideas?

atbohmer · May 11, 2023, 1:10pm

Maybe a clue overhere when restarting suricata via systemd?

May 11 13:16:23 systemd[1]: Stopping Suricata Intrusion Detection Service…
– Subject: Unit suricata.service has begun shutting down
– Defined-By: systemd
– Support: https://access.redhat.com/support
**-- **
– Unit suricata.service has begun shutting down.
May 11 13:17:53 systemd[1]: suricata.service: State ‘stop-sigterm’ timed out. Killing.
May 11 13:17:53 systemd[1]: suricata.service: Killing process 1755717 (Suricata-Main) with signal SIGKILL.
May 11 13:17:57 systemd[1]: suricata.service: Main process exited, code=killed, status=9/KILL
May 11 13:17:57 systemd[1]: suricata.service: Failed with result ‘timeout’.
– Subject: Unit failed
– Defined-By: systemd
– Support: Red Hat Customer Experience & Engagement - Red Hat Customer Portal
**-- **
– The unit suricata.service has entered the ‘failed’ state with result ‘timeout’.
May 11 13:17:57 systemd[1]: Stopped Suricata Intrusion Detection Service.
– Subject: Unit suricata.service has finished shutting down
– Defined-By: systemd
– Support: Red Hat Customer Experience & Engagement - Red Hat Customer Portal
**-- **
– Unit suricata.service has finished shutting down.
May 11 13:46:57 systemd[1]: Starting Suricata Intrusion Detection Service…
– Subject: Unit suricata.service has begun start-up
– Defined-By: systemd
– Support: Red Hat Customer Experience & Engagement - Red Hat Customer Portal
**-- **
– Unit suricata.service has begun starting up.
May 11 13:46:57 systemd[1]: Started Suricata Intrusion Detection Service.
– Subject: Unit suricata.service has finished start-up
– Defined-By: systemd
– Support: Red Hat Customer Experience & Engagement - Red Hat Customer Portal
**-- **
– Unit suricata.service has finished starting up.

ish · May 11, 2023, 6:25pm

You could journalctl -xf -u suricata.

atbohmer · May 12, 2023, 10:14am

Yep i know, thanks, but in my opinion it does not show a clue but but more retrospective about that point I’m not a developper I’ve attached a tar archive with logging (tar cjf)
surlogs.tgz (36.3 KB)

vjulien · May 12, 2023, 2:10pm

This confirms that the reload doesn’t complete from suricata’s pov.

vjulien · May 12, 2023, 3:04pm

atbohmer · May 12, 2023, 5:59pm

Thanks for confirming I’m officially blind Great this is picked up so swift, great release v7!

vjulien · May 13, 2023, 5:43am

This is now fixed in the master branch.

Topic		Replies	Views
Suricata is not running after 260k loaded rules Help suricata	6	410	January 18, 2023
Suricatasc reload-tenant to suricata segatiation fault suricata Help suricata	3	145	February 28, 2024
Running suricata-update Help	6	1453	December 29, 2021
[errcode: sc_err_conf_yaml_error(242)] Help	4	1952	June 7, 2023
<Error> - [ERRCODE: SC_ERR_NO_RULES_LOADED(43)] - Loading signatures failed Rules	33	7763	October 12, 2020

Suricatasc error: Reload already in progress

Related topics