Reload-tenants (suricatasc) fails for 'workers' runmode

1

I am using Suricata 7.0.2. Multi-tenant is enabled with 3 tenants. ‘reload-tenants’ via suricatasc fails with below errors:

For runmode "autofp’, reload-tenants works fine. The issue is seen with ‘workers’ mode.

reload-tenants
An exception occured: L<function get_linenumber at 0x7ff302742050>: Unable to get message from server

suricata.log

[7982] 14/1/2024 -- 16:33:17 - (detect-engine.c:DetectEngineCtxInitReal:2502) <Config>  -- pattern matchers: MPM: hs, SPM: hs
[7983] 14/1/2024 -- 16:33:17 - (detect-engine.c:DetectEngineCtxLoadConf:2940) <Config>  -- grouping: udp-whitelist (default) 53, 135, 5060
[7981] 14/1/2024 -- 16:33:17 - (detect-engine.c:DetectEngineCtxLoadConf:2970) <Config>  -- prefilter engines: MPM
[7983] 14/1/2024 -- 16:33:17 - (reputation.c:SRepInit:612) <Config>  -- IP reputation disabled
[7983] 14/1/2024 -- 16:33:25 - (util-threshold-config.c:SCThresholdConfParseFile:1045) <Info>  -- Threshold config parsed: 0 rule(s) found
[7983] 14/1/2024 -- 16:33:25 - (detect-engine-build.c:SigAddressPrepareStage1:1503) <Info>  -- 5171 signatures processed. 0 are IP-only rules, 126 are inspecting packet payload, 5045 inspect application layer, 0 are decoder event only
[7983] 14/1/2024 -- 16:33:25 - (detect-engine-build.c:SigAddressPrepareStage1:1506) <Config>  -- building signature grouping structure, stage 1: preprocessing rules... complete
[7981] 14/1/2024 -- 16:33:30 - (detect-engine-loader.c:SigLoadSignatures:351) <Info>  -- 1 rule files processed. 16751 rules successfully loaded, 0 rules failed
[7981] 14/1/2024 -- 16:33:30 - (util-threshold-config.c:SCThresholdConfParseFile:1045) <Info>  -- Threshold config parsed: 0 rule(s) found
[7982] 14/1/2024 -- 16:33:30 - (detect-engine-loader.c:SigLoadSignatures:351) <Info>  -- 1 rule files processed. 16751 rules successfully loaded, 0 rules failed
[7982] 14/1/2024 -- 16:33:30 - (util-threshold-config.c:SCThresholdConfParseFile:1045) <Info>  -- Threshold config parsed: 0 rule(s) found
[7981] 14/1/2024 -- 16:33:30 - (detect-engine-build.c:SigAddressPrepareStage1:1503) <Info>  -- 16751 signatures processed. 0 are IP-only rules, 1409 are inspecting packet payload, 15329 inspect application layer, 0 are decoder event only
[7981] 14/1/2024 -- 16:33:30 - (detect-engine-build.c:SigAddressPrepareStage1:1506) <Config>  -- building signature grouping structure, stage 1: preprocessing rules... complete
[7982] 14/1/2024 -- 16:33:30 - (detect-engine-build.c:SigAddressPrepareStage1:1503) <Info>  -- 16751 signatures processed. 0 are IP-only rules, 1409 are inspecting packet payload, 15329 inspect application layer, 0 are decoder event only
[7982] 14/1/2024 -- 16:33:30 - (detect-engine-build.c:SigAddressPrepareStage1:1506) <Config>  -- building signature grouping structure, stage 1: preprocessing rules... complete
2

Same question as in the other thread you opened, post the details about run, logs and config.

3

suricata.yaml (we call it idps_engine.yaml)

%YAML 1.1
---

vars:
  address-groups:
    HOME_NET: "[100.1.1.0/28,100.1.1.2/32,100.1.2.0/28,100.1.2.16/28,100.1.3.0/28,100.1.3.16/28,100.1.4.0/28,100.1.4.16/28,100.1.11.0/28,100.1.11.16/28,100.1.12.0/28,100.1.12.16/28,100.1.13.0/28,100.1.13.16/28,100.1.14.0/28,100.1.14.16/28,100.1.254.0/28,192.168.1.0/24,104.1.1.0/28,104.1.2.0/28,104.1.2.16/28,104.1.3.0/28,104.1.3.16/28,104.1.4.0/28,104.1.4.16/28,104.1.11.0/28,104.1.11.16/28,104.1.12.0/28,104.1.12.16/28,104.1.13.0/28,104.1.13.16/28,104.1.14.0/28,104.1.14.16/28,104.1.254.0/28]"
    EXTERNAL_NET: "!$HOME_NET"
    HTTP_SERVERS: "$HOME_NET"
    SMTP_SERVERS: "$HOME_NET"
    SQL_SERVERS: "$HOME_NET"
    DNS_SERVERS: "$HOME_NET"
    TELNET_SERVERS: "$HOME_NET"
    AIM_SERVERS: "$HOME_NET"
    DC_SERVERS: "$HOME_NET"
    DNP3_SERVERS: "$HOME_NET"
    MODBUS_SERVERS: "$HOME_NET"
    ENIP_SERVERS: "$HOME_NET"
    DNP3_CLIENT: "$HOME_NET"
    MODBUS_CLIENT: "$HOME_NET"
    ENIP_CLIENT: "$HOME_NET"
  port-groups:
    HTTP_PORTS: "[80,443]"
    SSH_PORTS: "22"
    SHELLCODE_PORTS: "!80"
    ORACLE_PORTS: "1521"
    DNP3_PORTS: "20000"
    MODBUS_PORTS: "502"
    FILE_DATA_PORTS: "[$HTTP_PORTS,110,143]"
    GENEVE_PORTS: "6081"
    VXLAN_PORTS: "4789"
    TEREDO_PORTS: "3544"
    FTP_PORTS: "21"

exception-policy: pass-flow

stream:
  midstream: true
  drop-invalid: no
  midstream-policy: auto

default-log-dir: /var/aryaka/nexus/idps_ni/

logging:
  default-log-level: info
  default-log-format: "[%i] %t - (%f:%n:%l) <%d>  -- "
  outputs:
  - console:
      enabled: false
  - file:
      enabled: true
      level: config
      filename: idps_engine.log
      format: "[%i] %t - (%f:%n:%l) <%d>  -- "

stats:
    enabled: true
    interval: 3600

outputs:
  - stats-json:
      enabled: true
      filename: stats_301.json
  - stats:
      enabled: false
      filename: stats_301.log
      totals: true
      threads: true
      decoder-events: true
      tcp: true
      http: true
      tls: true
      smtp: true
      ssh: true
      flow: false
      dns: true

  - eve-log:
      enabled: true
      log-level: notice
      filetype: regular
      filename: idps_events_301.json
      types:
      - alert:
          enabled: true
          tagged-packets: true
          payload: false
          payload-printable: false
          http-body: false
          metadata:
            app-layer: true
            flow: false
      - anomaly:
            enabled: false
            types:
              decode: false
              stream: false
              applayer: false
              packethdr: false
      - http:
          enabled: true
          extended: false
          body: false
          header: false
      - dns:
          enabled: false
          query: true
          answer: true
          transaction: true
      - tls:
          enabled: false
          certs: true
          subject: true
          issuer: true
      - files:
          enabled: false
          force-magic: true
      - smtp:
          enabled: false
          commands: true
          helo: true
          rcpt: true
      - flow:
          enabled: false
          extended: true
      - ssh:
          enabled: false
          handshake: true
      - netflow:
          enabled: false

  - eve-log:
      enabled: false
      filetype: regular
      filename: idps_stats_301.json
      types:
        - stats

  - syslog:
      enabled: false
      facility: local0
      level: notice

  - http-log:
      enabled: false

  - tls-log:
      enabled: false

  - tls-store:
      enabled: false

app-layer:
  protocols:
    krb5:
      enabled: true
    mqtt:
      enabled: false
    rfb:
      enabled: true
      detection-ports:
        dp: 5900, 5901, 5902, 5903, 5904, 5905, 5906, 5907, 5908, 5909
    snmp:
      enabled: true
    ikev2:
      enabled: true
    tls:
      enabled: true
      encryption-handling: bypass
      detection-ports:
        dp: 443
    dcerpc:
      enabled: true
    ftp:
      enabled: true
    rdp:
      enabled: true
    ssh:
      enabled: true
    smtp:
      enabled: true
    imap:
      enabled: detection-only
    smb:
      enabled: true
      detection-ports:
        dp: 139, 445
    nfs:
      enabled: true
    tftp:
      enabled: true
    dns:
      enabled: true
      tcp:
        enabled: true
        detection-ports:
          dp: 53
      udp:
        enabled: true
        detection-ports:
          dp: 53
    http:
      enabled: true
    http2:
      enabled: true
    modbus:
      enabled: false
      detection-ports:
        dp: 502
    dnp3:
      enabled: false
      detection-ports:
    enabled: 20000
    enip:
      enabled: false
      detection-ports:
        dp: 44818
        sp: 44818
    ntp:
      enabled: true
    dhcp:
      enabled: true
    sip:
      enabled: true
flow:
    enabled: false
    detection: none

coredump:
  max-dump: unlimited

runmode: workers

auto-scheduler: hash

asn1-max-frames: 256

host-mode: auto

max-pending-packets: 4096

default-packet-size: 2048

unix-command:
  enabled: yes
  filename: /var/aryaka/nexus/idps_ni/custom.socket

multi-detect:
  enabled: yes
  selector: vlan
  loaders: 3
  tenants:
  - id: 1
    yaml: /var/aryaka/nexus/idps_ni/config/idps_engine_lan.yaml
  - id: 2
    yaml: /var/aryaka/nexus/idps_ni/config/idps_engine_wan.yaml
  - id: 3
    yaml: /var/aryaka/nexus/idps_ni/config/idps_engine_adv.yaml
  mappings:
    - vlan-id: 1000
      tenant-id: 1
    - vlan-id: 1001
      tenant-id: 2
    - vlan-id: 1002
      tenant-id: 3

detect:
  profile: high

classification-file: /var/aryaka/nexus/idps_ni/rules/classification.config
reference-config-file: /var/aryaka/nexus/idps_ni/rules/reference.config
threshold-file: /var/aryaka/nexus/idps_ni/rules/threshold.config

==============================
suritcata.log (we call it idps_engine.log)

[17802] 1/3/2024 – 15:04:04. ---- timestamp at which reload-tenants was issued in suricatasc

$ cat idps_engine.log
[17801] 1/3/2024 -- 15:02:29 - (suricata.c:LogVersion:1178) <Notice>  -- This is Suricata version 7.0.2 RELEASE running in SYSTEM mode
[17801] 1/3/2024 -- 15:02:29 - (util-cpu.c:UtilCpuPrintSummary:182) <Info>  -- CPUs/cores online: 8
[17801] 1/3/2024 -- 15:02:29 - (util-exception-policy.c:ExceptionPolicyMasterParse:200) <Info>  -- master exception-policy set to: pass-flow
[17801] 1/3/2024 -- 15:02:29 - (util-exception-policy.c:ExceptionPolicyGetDefault:220) <Config>  -- app-layer.error-policy: pass-flow (defined via 'exception-policy' master switch)
[17801] 1/3/2024 -- 15:02:29 - (app-layer-htp.c:HTPConfigSetDefaultsPhase2:2567) <Config>  -- 'default' server has 'request-body-minimal-inspect-size' set to 34111 and 'request-body-inspect-window' set to 4249 after randomization.
[17801] 1/3/2024 -- 15:02:29 - (app-layer-htp.c:HTPConfigSetDefaultsPhase2:2580) <Config>  -- 'default' server has 'response-body-minimal-inspect-size' set to 32569 and 'response-body-inspect-window' set to 3942 after randomization.
[17801] 1/3/2024 -- 15:02:29 - (app-layer-enip.c:RegisterENIPUDPParsers:539) <Config>  -- Protocol detection and parser disabled for enip protocol.
[17801] 1/3/2024 -- 15:02:29 - (app-layer-dnp3.c:RegisterDNP3Parsers:1565) <Config>  -- Protocol detection and parser disabled for DNP3.
[17801] 1/3/2024 -- 15:02:29 - (host.c:HostInitConfig:259) <Config>  -- allocated 262144 bytes of memory for the host hash... 4096 buckets of size 64
[17801] 1/3/2024 -- 15:02:29 - (host.c:HostInitConfig:283) <Config>  -- preallocated 1000 hosts of size 136
[17801] 1/3/2024 -- 15:02:29 - (host.c:HostInitConfig:285) <Config>  -- host memory usage: 398144 bytes, maximum: 16777216
[17801] 1/3/2024 -- 15:02:29 - (util-coredump-config.c:CoredumpLoadConfig:148) <Config>  -- Core dump size is unlimited.
[17801] 1/3/2024 -- 15:02:29 - (util-exception-policy.c:PickPacketAction:126) <Warning>  -- flow actions not supported for defrag.memcap-policy, defaulting to "pass-packet"
[17801] 1/3/2024 -- 15:02:29 - (util-exception-policy.c:ExceptionPolicyGetDefault:220) <Config>  -- defrag.memcap-policy: pass-packet (defined via 'exception-policy' master switch)
[17801] 1/3/2024 -- 15:02:29 - (defrag-hash.c:DefragInitConfig:254) <Config>  -- allocated 229376 bytes of memory for the defrag hash... 4096 buckets of size 56
[17801] 1/3/2024 -- 15:02:29 - (defrag-hash.c:DefragInitConfig:288) <Config>  -- defrag memory usage: 229376 bytes, maximum: 16777216
[17801] 1/3/2024 -- 15:02:29 - (util-exception-policy.c:PickPacketAction:126) <Warning>  -- flow actions not supported for flow.memcap-policy, defaulting to "pass-packet"
[17801] 1/3/2024 -- 15:02:29 - (util-exception-policy.c:ExceptionPolicyGetDefault:220) <Config>  -- flow.memcap-policy: pass-packet (defined via 'exception-policy' master switch)
[17801] 1/3/2024 -- 15:02:29 - (flow.c:FlowInitConfig:675) <Config>  -- flow size 296, memcap allows for 0 flows. Per hash row in perfect conditions 0
[17801] 1/3/2024 -- 15:02:29 - (stream-tcp.c:StreamTcpInitConfig:410) <Config>  -- stream "prealloc-sessions": 2048 (per thread)
[17801] 1/3/2024 -- 15:02:29 - (stream-tcp.c:StreamTcpInitConfig:429) <Config>  -- stream "memcap": 67108864
[17801] 1/3/2024 -- 15:02:29 - (stream-tcp.c:StreamTcpInitConfig:437) <Config>  -- stream "midstream" session pickups: enabled
[17801] 1/3/2024 -- 15:02:29 - (stream-tcp.c:StreamTcpInitConfig:445) <Config>  -- stream "async-oneside": disabled
[17801] 1/3/2024 -- 15:02:29 - (stream-tcp.c:StreamTcpInitConfig:462) <Config>  -- stream "checksum-validation": enabled
[17801] 1/3/2024 -- 15:02:29 - (util-exception-policy.c:ExceptionPolicyGetDefault:220) <Config>  -- stream.memcap-policy: pass-flow (defined via 'exception-policy' master switch)
[17801] 1/3/2024 -- 15:02:29 - (util-exception-policy.c:ExceptionPolicyGetDefault:220) <Config>  -- stream.reassembly.memcap-policy: pass-flow (defined via 'exception-policy' master switch)
[17801] 1/3/2024 -- 15:02:29 - (stream-tcp.c:StreamTcpInitConfig:494) <Config>  -- stream."inline": enabled
[17801] 1/3/2024 -- 15:02:29 - (stream-tcp.c:StreamTcpInitConfig:507) <Config>  -- stream "bypass": disabled
[17801] 1/3/2024 -- 15:02:29 - (stream-tcp.c:StreamTcpInitConfig:529) <Config>  -- stream "max-syn-queued": 10
[17801] 1/3/2024 -- 15:02:29 - (stream-tcp.c:StreamTcpInitConfig:542) <Config>  -- stream "max-synack-queued": 5
[17801] 1/3/2024 -- 15:02:29 - (stream-tcp.c:StreamTcpInitConfig:564) <Config>  -- stream.reassembly "memcap": 268435456
[17801] 1/3/2024 -- 15:02:29 - (stream-tcp.c:StreamTcpInitConfig:586) <Config>  -- stream.reassembly "depth": 1048576
[17801] 1/3/2024 -- 15:02:29 - (stream-tcp.c:StreamTcpInitConfig:659) <Config>  -- stream.reassembly "toserver-chunk-size": 2483
[17801] 1/3/2024 -- 15:02:29 - (stream-tcp.c:StreamTcpInitConfig:661) <Config>  -- stream.reassembly "toclient-chunk-size": 2440
[17801] 1/3/2024 -- 15:02:29 - (stream-tcp.c:StreamTcpInitConfig:673) <Config>  -- stream.reassembly.raw: enabled
[17801] 1/3/2024 -- 15:02:29 - (stream-tcp.c:StreamTcpInitConfig:682) <Config>  -- stream.liberal-timestamps: disabled
[17801] 1/3/2024 -- 15:02:29 - (stream-tcp-reassemble.c:StreamTcpReassemblyConfig:491) <Config>  -- stream.reassembly "segment-prealloc": 2048
[17801] 1/3/2024 -- 15:02:29 - (stream-tcp-reassemble.c:StreamTcpReassemblyConfig:514) <Config>  -- stream.reassembly "max-regions": 8
[17801] 1/3/2024 -- 15:02:29 - (util-logopenfile.c:SCConfLogOpenGeneric:659) <Info>  -- stats-json output device (regular) initialized: stats_301.json
[17801] 1/3/2024 -- 15:02:29 - (util-logopenfile.c:SCConfLogOpenGeneric:659) <Info>  -- eve-log output device (regular) initialized: idps_events_301.json
[17801] 1/3/2024 -- 15:02:29 - (runmodes.c:RunModeInitializeEveOutput:726) <Config>  -- enabling 'eve-log' module 'alert'
[17801] 1/3/2024 -- 15:02:29 - (runmodes.c:RunModeInitializeEveOutput:726) <Config>  -- enabling 'eve-log' module 'anomaly'
[17801] 1/3/2024 -- 15:02:29 - (runmodes.c:RunModeInitializeEveOutput:726) <Config>  -- enabling 'eve-log' module 'http'
[17801] 1/3/2024 -- 15:02:29 - (runmodes.c:RunModeInitializeEveOutput:767) <Warning>  -- No output module named eve-log.http
[17801] 1/3/2024 -- 15:02:29 - (runmodes.c:RunModeInitializeEveOutput:726) <Config>  -- enabling 'eve-log' module 'dns'
[17801] 1/3/2024 -- 15:02:29 - (runmodes.c:RunModeInitializeEveOutput:726) <Config>  -- enabling 'eve-log' module 'tls'
[17801] 1/3/2024 -- 15:02:29 - (runmodes.c:RunModeInitializeEveOutput:726) <Config>  -- enabling 'eve-log' module 'files'
[17801] 1/3/2024 -- 15:02:29 - (runmodes.c:RunModeInitializeEveOutput:726) <Config>  -- enabling 'eve-log' module 'smtp'
[17801] 1/3/2024 -- 15:02:29 - (runmodes.c:RunModeInitializeEveOutput:726) <Config>  -- enabling 'eve-log' module 'flow'
[17801] 1/3/2024 -- 15:02:29 - (runmodes.c:RunModeInitializeEveOutput:726) <Config>  -- enabling 'eve-log' module 'ssh'
[17801] 1/3/2024 -- 15:02:29 - (runmodes.c:RunModeInitializeEveOutput:726) <Config>  -- enabling 'eve-log' module 'netflow'
[17801] 1/3/2024 -- 15:02:29 - (suricata.c:SetupDelayedDetect:2502) <Config>  -- Delayed detect disabled
[17801] 1/3/2024 -- 15:02:29 - (detect-engine-loader.c:DetectLoadersInit:473) <Info>  -- using 3 detect loader threads
[17801] 1/3/2024 -- 15:02:29 - (detect-engine.c:DetectEngineMultiTenantSetup:4247) <Config>  -- multi-tenant selector type vlan
[17801] 1/3/2024 -- 15:02:29 - (detect-engine.c:DetectEngineMultiTenantSetup:4279) <Config>  -- multi-detect is enabled (multi tenancy). Selector: vlan
[17801] 1/3/2024 -- 15:02:29 - (detect-engine.c:DetectEngineMultiTenantSetupLoadVlanMappings:4205) <Config>  -- vlan 1000 connected to tenant-id 1
[17801] 1/3/2024 -- 15:02:29 - (detect-engine.c:DetectEngineMultiTenantSetupLoadVlanMappings:4205) <Config>  -- vlan 1001 connected to tenant-id 2
[17801] 1/3/2024 -- 15:02:29 - (detect-engine.c:DetectEngineMultiTenantSetupLoadVlanMappings:4205) <Config>  -- vlan 1002 connected to tenant-id 3
[17802] 1/3/2024 -- 15:02:29 - (detect-engine.c:DetectEngineCtxInitReal:2502) <Config>  -- pattern matchers: MPM: hs, SPM: hs
[17802] 1/3/2024 -- 15:02:29 - (detect-engine.c:DetectEngineCtxLoadConf:2914) <Config>  -- grouping: tcp-whitelist (default) 53, 80, 139, 443, 445, 1433, 3306, 3389, 6666, 6667, 8080
[17803] 1/3/2024 -- 15:02:29 - (detect-engine.c:DetectEngineCtxInitReal:2502) <Config>  -- pattern matchers: MPM: hs, SPM: hs
[17802] 1/3/2024 -- 15:02:29 - (detect-engine.c:DetectEngineCtxLoadConf:2940) <Config>  -- grouping: udp-whitelist (default) 53, 135, 5060
[17803] 1/3/2024 -- 15:02:29 - (detect-engine.c:DetectEngineCtxLoadConf:2914) <Config>  -- grouping: tcp-whitelist (default) 53, 80, 139, 443, 445, 1433, 3306, 3389, 6666, 6667, 8080
[17802] 1/3/2024 -- 15:02:29 - (detect-engine.c:DetectEngineCtxLoadConf:2970) <Config>  -- prefilter engines: MPM
[17803] 1/3/2024 -- 15:02:29 - (detect-engine.c:DetectEngineCtxLoadConf:2940) <Config>  -- grouping: udp-whitelist (default) 53, 135, 5060
[17803] 1/3/2024 -- 15:02:29 - (detect-engine.c:DetectEngineCtxLoadConf:2970) <Config>  -- prefilter engines: MPM
[17804] 1/3/2024 -- 15:02:29 - (detect-engine.c:DetectEngineCtxInitReal:2502) <Config>  -- pattern matchers: MPM: hs, SPM: hs
[17804] 1/3/2024 -- 15:02:29 - (detect-engine.c:DetectEngineCtxLoadConf:2914) <Config>  -- grouping: tcp-whitelist (default) 53, 80, 139, 443, 445, 1433, 3306, 3389, 6666, 6667, 8080
[17804] 1/3/2024 -- 15:02:29 - (detect-engine.c:DetectEngineCtxLoadConf:2940) <Config>  -- grouping: udp-whitelist (default) 53, 135, 5060
[17804] 1/3/2024 -- 15:02:29 - (detect-engine.c:DetectEngineCtxLoadConf:2970) <Config>  -- prefilter engines: MPM
[17802] 1/3/2024 -- 15:02:29 - (reputation.c:SRepInit:612) <Config>  -- IP reputation disabled
[17804] 1/3/2024 -- 15:02:29 - (reputation.c:SRepInit:612) <Config>  -- IP reputation disabled
[17803] 1/3/2024 -- 15:02:29 - (reputation.c:SRepInit:612) <Config>  -- IP reputation disabled
[17803] 1/3/2024 -- 15:02:29 - (detect-engine-loader.c:ProcessSigFiles:248) <Config>  -- Loading rule file: /var/aryaka/nexus/idps_ni/rules/wan/wan_rules.rules
[17802] 1/3/2024 -- 15:02:29 - (detect-engine-loader.c:ProcessSigFiles:248) <Config>  -- Loading rule file: /var/aryaka/nexus/idps_ni/rules/lan/lan_rules.rules
[17804] 1/3/2024 -- 15:02:29 - (detect-engine-loader.c:ProcessSigFiles:248) <Config>  -- Loading rule file: /var/aryaka/nexus/idps_ni/rules/adv/adv_rules.rules
[17804] 1/3/2024 -- 15:02:36 - (detect-engine-loader.c:SigLoadSignatures:351) <Info>  -- 1 rule files processed. 5171 rules successfully loaded, 0 rules failed
[17804] 1/3/2024 -- 15:02:36 - (util-threshold-config.c:SCThresholdConfParseFile:1045) <Info>  -- Threshold config parsed: 0 rule(s) found
[17804] 1/3/2024 -- 15:02:36 - (detect-engine-build.c:SigAddressPrepareStage1:1503) <Info>  -- 5171 signatures processed. 0 are IP-only rules, 126 are inspecting packet payload, 5045 inspect application layer, 0 are decoder event only
[17804] 1/3/2024 -- 15:02:36 - (detect-engine-build.c:SigAddressPrepareStage1:1506) <Config>  -- building signature grouping structure, stage 1: preprocessing rules... complete
[17802] 1/3/2024 -- 15:02:43 - (detect-engine-loader.c:SigLoadSignatures:351) <Info>  -- 1 rule files processed. 16751 rules successfully loaded, 0 rules failed
[17802] 1/3/2024 -- 15:02:44 - (util-threshold-config.c:SCThresholdConfParseFile:1045) <Info>  -- Threshold config parsed: 0 rule(s) found
[17803] 1/3/2024 -- 15:02:44 - (detect-engine-loader.c:SigLoadSignatures:351) <Info>  -- 1 rule files processed. 16751 rules successfully loaded, 0 rules failed
[17803] 1/3/2024 -- 15:02:44 - (util-threshold-config.c:SCThresholdConfParseFile:1045) <Info>  -- Threshold config parsed: 0 rule(s) found
[17802] 1/3/2024 -- 15:02:44 - (detect-engine-build.c:SigAddressPrepareStage1:1503) <Info>  -- 16751 signatures processed. 0 are IP-only rules, 1409 are inspecting packet payload, 15329 inspect application layer, 0 are decoder event only
[17802] 1/3/2024 -- 15:02:44 - (detect-engine-build.c:SigAddressPrepareStage1:1506) <Config>  -- building signature grouping structure, stage 1: preprocessing rules... complete
[17803] 1/3/2024 -- 15:02:44 - (detect-engine-build.c:SigAddressPrepareStage1:1503) <Info>  -- 16751 signatures processed. 0 are IP-only rules, 1409 are inspecting packet payload, 15329 inspect application layer, 0 are decoder event only
[17803] 1/3/2024 -- 15:02:44 - (detect-engine-build.c:SigAddressPrepareStage1:1506) <Config>  -- building signature grouping structure, stage 1: preprocessing rules... complete
[17801] 1/3/2024 -- 15:02:56 - (flow-manager.c:FlowManagerThreadSpawn:948) <Config>  -- using 1 flow manager threads
[17801] 1/3/2024 -- 15:02:56 - (flow-manager.c:FlowRecyclerThreadSpawn:1154) <Config>  -- using 1 flow recycler threads
[17801] 1/3/2024 -- 15:02:56 - (unix-manager.c:UnixNew:136) <Info>  -- unix socket '/var/aryaka/nexus/idps_ni/custom.socket'
[17801] 1/3/2024 -- 15:02:56 - (tm-threads.c:TmThreadWaitOnThreadRunning:1893) <Notice>  -- Threads created -> W: 8 FM: 1 FR: 1   Engine started.
[17802] 1/3/2024 -- 15:04:04 - (detect-engine.c:DetectEngineCtxInitReal:2502) <Config>  -- pattern matchers: MPM: hs, SPM: hs
[17802] 1/3/2024 -- 15:04:04 - (detect-engine.c:DetectEngineCtxLoadConf:2914) <Config>  -- grouping: tcp-whitelist (default) 53, 80, 139, 443, 445, 1433, 3306, 3389, 6666, 6667, 8080
[17802] 1/3/2024 -- 15:04:04 - (detect-engine.c:DetectEngineCtxLoadConf:2940) <Config>  -- grouping: udp-whitelist (default) 53, 135, 5060
[17802] 1/3/2024 -- 15:04:04 - (detect-engine.c:DetectEngineCtxLoadConf:2970) <Config>  -- prefilter engines: MPM
[17803] 1/3/2024 -- 15:04:04 - (detect-engine.c:DetectEngineCtxInitReal:2502) <Config>  -- pattern matchers: MPM: hs, SPM: hs
[17803] 1/3/2024 -- 15:04:04 - (detect-engine.c:DetectEngineCtxLoadConf:2914) <Config>  -- grouping: tcp-whitelist (default) 53, 80, 139, 443, 445, 1433, 3306, 3389, 6666, 6667, 8080
[17804] 1/3/2024 -- 15:04:04 - (detect-engine.c:DetectEngineCtxInitReal:2502) <Config>  -- pattern matchers: MPM: hs, SPM: hs
[17803] 1/3/2024 -- 15:04:04 - (detect-engine.c:DetectEngineCtxLoadConf:2940) <Config>  -- grouping: udp-whitelist (default) 53, 135, 5060
[17804] 1/3/2024 -- 15:04:04 - (detect-engine.c:DetectEngineCtxLoadConf:2914) <Config>  -- grouping: tcp-whitelist (default) 53, 80, 139, 443, 445, 1433, 3306, 3389, 6666, 6667, 8080
[17803] 1/3/2024 -- 15:04:04 - (detect-engine.c:DetectEngineCtxLoadConf:2970) <Config>  -- prefilter engines: MPM
[17804] 1/3/2024 -- 15:04:04 - (detect-engine.c:DetectEngineCtxLoadConf:2940) <Config>  -- grouping: udp-whitelist (default) 53, 135, 5060
[17804] 1/3/2024 -- 15:04:04 - (detect-engine.c:DetectEngineCtxLoadConf:2970) <Config>  -- prefilter engines: MPM
[17802] 1/3/2024 -- 15:04:04 - (reputation.c:SRepInit:612) <Config>  -- IP reputation disabled
[17803] 1/3/2024 -- 15:04:04 - (reputation.c:SRepInit:612) <Config>  -- IP reputation disabled
[17804] 1/3/2024 -- 15:04:04 - (reputation.c:SRepInit:612) <Config>  -- IP reputation disabled
[17802] 1/3/2024 -- 15:04:04 - (detect-engine-loader.c:ProcessSigFiles:248) <Config>  -- Loading rule file: /var/aryaka/nexus/idps_ni/rules/lan/lan_rules.rules
[17804] 1/3/2024 -- 15:04:04 - (detect-engine-loader.c:ProcessSigFiles:248) <Config>  -- Loading rule file: /var/aryaka/nexus/idps_ni/rules/adv/adv_rules.rules
[17803] 1/3/2024 -- 15:04:04 - (detect-engine-loader.c:ProcessSigFiles:248) <Config>  -- Loading rule file: /var/aryaka/nexus/idps_ni/rules/wan/wan_rules.rules
[17804] 1/3/2024 -- 15:04:11 - (detect-engine-loader.c:SigLoadSignatures:351) <Info>  -- 1 rule files processed. 5171 rules successfully loaded, 0 rules failed
[17804] 1/3/2024 -- 15:04:11 - (util-threshold-config.c:SCThresholdConfParseFile:1045) <Info>  -- Threshold config parsed: 0 rule(s) found
[17804] 1/3/2024 -- 15:04:11 - (detect-engine-build.c:SigAddressPrepareStage1:1503) <Info>  -- 5171 signatures processed. 0 are IP-only rules, 126 are inspecting packet payload, 5045 inspect application layer, 0 are decoder event only
[17804] 1/3/2024 -- 15:04:11 - (detect-engine-build.c:SigAddressPrepareStage1:1506) <Config>  -- building signature grouping structure, stage 1: preprocessing rules... complete
[17803] 1/3/2024 -- 15:04:16 - (detect-engine-loader.c:SigLoadSignatures:351) <Info>  -- 1 rule files processed. 16751 rules successfully loaded, 0 rules failed
[17803] 1/3/2024 -- 15:04:16 - (util-threshold-config.c:SCThresholdConfParseFile:1045) <Info>  -- Threshold config parsed: 0 rule(s) found
[17803] 1/3/2024 -- 15:04:16 - (detect-engine-build.c:SigAddressPrepareStage1:1503) <Info>  -- 16751 signatures processed. 0 are IP-only rules, 1409 are inspecting packet payload, 15329 inspect application layer, 0 are decoder event only
[17803] 1/3/2024 -- 15:04:16 - (detect-engine-build.c:SigAddressPrepareStage1:1506) <Config>  -- building signature grouping structure, stage 1: preprocessing rules... complete
[17802] 1/3/2024 -- 15:04:16 - (detect-engine-loader.c:SigLoadSignatures:351) <Info>  -- 1 rule files processed. 16751 rules successfully loaded, 0 rules failed
[17802] 1/3/2024 -- 15:04:16 - (util-threshold-config.c:SCThresholdConfParseFile:1045) <Info>  -- Threshold config parsed: 0 rule(s) found
[17802] 1/3/2024 -- 15:04:16 - (detect-engine-build.c:SigAddressPrepareStage1:1503) <Info>  -- 16751 signatures processed. 0 are IP-only rules, 1409 are inspecting packet payload, 15329 inspect application layer, 0 are decoder event only
[17802] 1/3/2024 -- 15:04:16 - (detect-engine-build.c:SigAddressPrepareStage1:1506) <Config>  -- building signature grouping structure, stage 1: preprocessing rules... complete

==============================
------------------ sruicatasc hangs and never completes------------

[root@active:anap:il3-301 ~]$ suricatasc /var/aryaka/nexus/idps_ni/custom.socket
Command list: shutdown, command-list, help, version, uptime, running-mode, capture-mode, conf-get, dump-counters, reload-rules, ruleset-reload-rules, ruleset-reload-nonblocking, ruleset-reload-time, ruleset-stats, ruleset-failed-rules, register-tenant-handler, unregister-tenant-handler, register-tenant, reload-tenant, reload-tenants, unregister-tenant, add-hostbit, remove-hostbit, list-hostbit, reopen-log-files, memcap-set, memcap-show, memcap-list, dataset-add, dataset-remove, get-flow-stats-by-id, dataset-dump, dataset-clear, dataset-lookup, log-level, iface-stat, iface-list, iface-bypassed-stat, ebpf-bypassed-stat, quit
>>> reload-tenants
4

Can you also post the following configuration file for the tenants?

  tenants:
  - id: 1
    yaml: /var/aryaka/nexus/idps_ni/config/idps_engine_lan.yaml
  - id: 2
    yaml: /var/aryaka/nexus/idps_ni/config/idps_engine_wan.yaml
  - id: 3
    yaml: /var/aryaka/nexus/idps_ni/config/idps_engine_adv.yaml
5

==================
idps_engine_lan.yaml

%YAML 1.1
---

vars:
  address-groups:
    HOME_NET: "[100.1.1.0/28,100.1.1.2/32,100.1.2.0/28,100.1.2.16/28,100.1.3.0/28,100.1.3.16/28,100.1.4.0/28,100.1.4.16/28,100.1.11.0/28,100.1.11.16/28,100.1.12.0/28,100.1.12.16/28,100.1.13.0/28,100.1.13.16/28,100.1.14.0/28,100.1.14.16/28,100.1.254.0/28,192.168.1.0/24,104.1.1.0/28,104.1.2.0/28,104.1.2.16/28,104.1.3.0/28,104.1.3.16/28,104.1.4.0/28,104.1.4.16/28,104.1.11.0/28,104.1.11.16/28,104.1.12.0/28,104.1.12.16/28,104.1.13.0/28,104.1.13.16/28,104.1.14.0/28,104.1.14.16/28,104.1.254.0/28]"
    EXTERNAL_NET: "!$HOME_NET"
    HTTP_SERVERS: "$HOME_NET"
    SMTP_SERVERS: "$HOME_NET"
    SQL_SERVERS: "$HOME_NET"
    DNS_SERVERS: "$HOME_NET"
    TELNET_SERVERS: "$HOME_NET"
    AIM_SERVERS: "$HOME_NET"
    DC_SERVERS: "$HOME_NET"
    DNP3_SERVERS: "$HOME_NET"
    MODBUS_SERVERS: "$HOME_NET"
    ENIP_SERVERS: "$HOME_NET"
    DNP3_CLIENT: "$HOME_NET"
    MODBUS_CLIENT: "$HOME_NET"
    ENIP_CLIENT: "$HOME_NET"
  port-groups:
    HTTP_PORTS: "[80,443]"
    SSH_PORTS: "22"
    SHELLCODE_PORTS: "!80"
    ORACLE_PORTS: "1521"
    DNP3_PORTS: "20000"
    MODBUS_PORTS: "502"
    FILE_DATA_PORTS: "[$HTTP_PORTS,110,143]"
    GENEVE_PORTS: "6081"
    VXLAN_PORTS: "4789"
    TEREDO_PORTS: "3544"
    FTP_PORTS: "21"

default-rule-path: /var/aryaka/nexus/idps_ni/rules/lan/
rule-files:
  - lan_rules.rules

==================
idps_engine_wan.yaml

%YAML 1.1
---

vars:
  address-groups:
    HOME_NET: "[100.1.1.0/28,100.1.1.2/32,100.1.2.0/28,100.1.2.16/28,100.1.3.0/28,100.1.3.16/28,100.1.4.0/28,100.1.4.16/28,100.1.11.0/28,100.1.11.16/28,100.1.12.0/28,100.1.12.16/28,100.1.13.0/28,100.1.13.16/28,100.1.14.0/28,100.1.14.16/28,100.1.254.0/28,192.168.1.0/24,104.1.1.0/28,104.1.2.0/28,104.1.2.16/28,104.1.3.0/28,104.1.3.16/28,104.1.4.0/28,104.1.4.16/28,104.1.11.0/28,104.1.11.16/28,104.1.12.0/28,104.1.12.16/28,104.1.13.0/28,104.1.13.16/28,104.1.14.0/28,104.1.14.16/28,104.1.254.0/28]"
    EXTERNAL_NET: "!$HOME_NET"
    HTTP_SERVERS: "$HOME_NET"
    SMTP_SERVERS: "$HOME_NET"
    SQL_SERVERS: "$HOME_NET"
    DNS_SERVERS: "$HOME_NET"
    TELNET_SERVERS: "$HOME_NET"
    AIM_SERVERS: "$HOME_NET"
    DC_SERVERS: "$HOME_NET"
    DNP3_SERVERS: "$HOME_NET"
    MODBUS_SERVERS: "$HOME_NET"
    ENIP_SERVERS: "$HOME_NET"
    DNP3_CLIENT: "$HOME_NET"
    MODBUS_CLIENT: "$HOME_NET"
    ENIP_CLIENT: "$HOME_NET"
  port-groups:
    HTTP_PORTS: "[80,443]"
    SSH_PORTS: "22"
    SHELLCODE_PORTS: "!80"
    ORACLE_PORTS: "1521"
    DNP3_PORTS: "20000"
    MODBUS_PORTS: "502"
    FILE_DATA_PORTS: "[$HTTP_PORTS,110,143]"
    GENEVE_PORTS: "6081"
    VXLAN_PORTS: "4789"
    TEREDO_PORTS: "3544"
    FTP_PORTS: "21"

default-rule-path: /var/aryaka/nexus/idps_ni/rules/wan/
rule-files:
  - wan_rules.rules

==================
idps_engine_adv.yaml

%YAML 1.1
---

vars:
  address-groups:
    HOME_NET: "[100.1.1.0/28,100.1.1.2/32,100.1.2.0/28,100.1.2.16/28,100.1.3.0/28,100.1.3.16/28,100.1.4.0/28,100.1.4.16/28,100.1.11.0/28,100.1.11.16/28,100.1.12.0/28,100.1.12.16/28,100.1.13.0/28,100.1.13.16/28,100.1.14.0/28,100.1.14.16/28,100.1.254.0/28,192.168.1.0/24,104.1.1.0/28,104.1.2.0/28,104.1.2.16/28,104.1.3.0/28,104.1.3.16/28,104.1.4.0/28,104.1.4.16/28,104.1.11.0/28,104.1.11.16/28,104.1.12.0/28,104.1.12.16/28,104.1.13.0/28,104.1.13.16/28,104.1.14.0/28,104.1.14.16/28,104.1.254.0/28]"
    EXTERNAL_NET: "!$HOME_NET"
    HTTP_SERVERS: "$HOME_NET"
    SMTP_SERVERS: "$HOME_NET"
    SQL_SERVERS: "$HOME_NET"
    DNS_SERVERS: "$HOME_NET"
    TELNET_SERVERS: "$HOME_NET"
    AIM_SERVERS: "$HOME_NET"
    DC_SERVERS: "$HOME_NET"
    DNP3_SERVERS: "$HOME_NET"
    MODBUS_SERVERS: "$HOME_NET"
    ENIP_SERVERS: "$HOME_NET"
    DNP3_CLIENT: "$HOME_NET"
    MODBUS_CLIENT: "$HOME_NET"
    ENIP_CLIENT: "$HOME_NET"
  port-groups:
    HTTP_PORTS: "[80,443]"
    SSH_PORTS: "22"
    SHELLCODE_PORTS: "!80"
    ORACLE_PORTS: "1521"
    DNP3_PORTS: "20000"
    MODBUS_PORTS: "502"
    FILE_DATA_PORTS: "[$HTTP_PORTS,110,143]"
    GENEVE_PORTS: "6081"
    VXLAN_PORTS: "4789"
    TEREDO_PORTS: "3544"
    FTP_PORTS: "21"

default-rule-path: /var/aryaka/nexus/idps_ni/rules/adv/
rule-files:
  - adv_rules.rules
6

I’m using Suri 7.0.3 and cannot replicate your issue. reload-tenants completes – but it does take longer using your configuration file than the default suricata configuration file with the same tenant configuration.

What’re the hardware details for the deployment machine?

7

Thanks for your response. You are right that reload-tenants really takes long time. It only happens for ‘workers’ mode. For ‘auto-fp’ mode the operation is completed quite quick. The issue is that reload-tenants throws an error like below and we never know if the operation was indeed successful or not.

reload-tenants
An exception occured: L<function get_linenumber at 0x7ff302742050>: Unable to get message from server

For our use-case, we use reload-tenants to reflect any *.yaml changes in suricata and the config update is dynamic (as the variables like HOME_NET changes).

We have several hatrdware platforms, I see the issue on all platforms (including fairly latest intel x86 processors).
$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 94
Model name: Intel(R) Xeon(R) CPU E3-1275 v5 @ 3.60GHz
Stepping: 3
CPU MHz: 3600.000
BogoMIPS: 7200.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 8192K
NUMA node0 CPU(s): 0-7
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 dtherm arat pln pts hwp hwp_notify hwp_act_window hwp_epp

8

What command line are you using to start Suricata?

9

invoked with ‘-c’, ‘-l’ and -‘a’ options ( ‘-a’ is custom option we added)
‘-c’ to set suri->conf_filename
‘-l’ to set suri->set_logdir
-‘a’ to set suri->run_mode and set EngineModeSetIPS

10

Does the suricata binary crash or create a core-dump when this error occurs?

reload-tenants
An exception occured: L<function get_linenumber at 0x7ff302742050>: Unable to get message from server

11

No crash/core-dump and suricata continues to function. It’s just that suricatasc takes too long and throws an exception. We use suricatasc ‘reload-tenants’ mechanism to update suricata w/ config changes, hence ‘reload-tenants’ is critical.

12

Thanks for the additional information.

From the information you’ve provided, it looks like the exception (noted earlier) occurs within suricatasc because of the extra time taken to complete the reload-tenants command.

suricatasc is a python application that uses logic in suricatasc.py. There, there’s a 5 minute timeout before giving up. Perhaps you could try adjusting 600 to 900 and see if that helps. Suggestion: make the timeout large enough, but not too large to let issues go unnoticed.

Here’s the current code from suricatasc.py

    def send_command(self, command, arguments=None):
        if command not in self.cmd_list and command != 'command-list':
            raise SuricataCommandException("L{}: Command not found: {}".format(get_linenumber(), command))

        cmdmsg = {}
        cmdmsg['command'] = command
        if arguments:
            cmdmsg['arguments'] = arguments
        if self.verbose:
            print("SND: " + json.dumps(cmdmsg))
        cmdmsg_str = json.dumps(cmdmsg) + "\n"
        if sys.version < '3':
            self.socket.send(cmdmsg_str)
        else:
            self.socket.send(bytes(cmdmsg_str, 'iso-8859-1'))

        ready = select.select([self.socket], [], [], 600)
        if ready[0]:
            cmdret = self.json_recv()
        else:
            cmdret = None
        if not cmdret:
            raise SuricataReturnException("L{}: Unable to get message from server".format(get_linenumber))

        if self.verbose:
            print("RCV: "+ json.dumps(cmdret))

        return cmdret
13

Thank you for your response. Changing timeout to higher value (as high as 1500) did not help, same error. It appears the issue is with suricata not responding (as if SigGroupBuild is waiting for ever after SigAddressPrepareStage1 is complete). What’s puzzling is ‘autofp’ works fine, the issue is only seen with ‘workers’ mode

14

Have done more debugging. Wanted to provide details, hope it’s not a problem in sharing details in this forum.

For ‘workers’ mode, DetectEngineReloadThreads() goes into an infinite loop.

inside DetectEngineReloadThreads(), after call to InjectPackets(), there is a retry block that checks for (threads_done < no_of_detect_tvs).

For ‘autofp’ mode, (threads_done == no_of_detect_tvs) and retry block is executed only once. For ‘workers’ mode, (threads_done < no_of_detect_tvs) is true and retry happens forever.

inside InjectPackets(),
for ‘autofp’ mode, (detect_tvs[i]->inq != NULL), so PKT_SRC_DETECT_RELOAD_FLUSH packet is enqueued.
for ‘workers’ mode, (detect_tvs[i]->inq == NULL), so no packet is injected.

after InjectPackets() call
for ‘autofp’ mode, (new_det_ctx[i]->so_far_used_by_detect) == 1) is TRUE and threads_done is incremented. This leads to satisfying (threads_done == no_of_detect_tvs) condition
for ‘workers’ mode, (new_det_ctx[i]->so_far_used_by_detect) == 1) is FALSE and TmThreadsCaptureBreakLoop() is invoked, but could’t figure out how its execution is supposed to result in setting new_det_ctx[i]->so_far_used_by_detect to 1.

15

Thanks for your perseverance to get more context for this issue.

Please file a bug at our redmine site: https://redmine.openinfosecfoundation.org/ with the details.