Suricata6.0.0-beta1 on OpenWrt Illegal Instruction error

root@OpenWrt:/# gdb suricata
GNU gdb (GDB) 8.3.1
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "mips64-openwrt-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from suricata...
(No debugging symbols found in suricata)
(gdb) bt
No stack.
(gdb) run -c /etc/suricata/suricata.yaml -i eth0 -s rules/*.rules
Starting program: /usr/bin/suricata -c /etc/suricata/suricata.yaml -i eth0 -s rules/*.rules
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
Error opening file /var/log/suricata//suricata.log
[1967] 5/9/2020 -- 10:54:06 - (suricata.c:1065) <Notice> (LogVersion) -- This is Suricata version 6.0.0-dev running in SYSTEM mode

Program received signal SIGILL, Illegal instruction.
0x000000aaab85f3ac in ?? ()
(gdb) bt
#0  0x000000aaab85f3ac in ?? ()
warning: GDB can't find the start of the function at 0xaaab85f3ac.

    GDB is unable to find the start of the function at 0xaaab85f3ac
and thus can't determine the size of that function's stack frame.
This means that GDB may be unable to access that stack frame, or
the frames below it.
    This problem is most likely caused by an invalid program counter or
stack pointer.
    However, if you think GDB should simply search farther back
from 0xaaab85f3ac for code which looks like the beginning of a
function, you can increase the range of the search using the `set
heuristic-fence-post' command.
(gdb) 

I’m rebuilding the firmware with the -ggdb CFLAG

building with the CFLAG -ggdb did not produce any different results. Suggestions?

So I tried with 5.0.3, and still received the same Illegal Instruction… Both 5.0.3 and 6.0.0-beta1 seem to crash in LIBHTP? I clone the LibHTP master branch into the suricata directory before building.

root@OpenWrt:/etc# suricata -c /etc/suricata/suricata.yaml -i eth0
Error opening file /var/log/suricata//suricata.log
7/9/2020 -- 04:48:53 - <Notice> - This is Suricata version 5.0.3 RELEASE running in SYSTEM mode
Illegal instruction
root@OpenWrt:/etc# suricata -vvvv -c /etc/suricata/suricata.yaml -i eth0
Error opening file /var/log/suricata//suricata.log
7/9/2020 -- 04:49:01 - <Debug> - sc_log_global_log_level: 10
7/9/2020 -- 04:49:01 - <Debug> - sc_lc->log_format: %t - <%d> - 
7/9/2020 -- 04:49:01 - <Debug> - SCLogSetOPFilter: filter: (null)
7/9/2020 -- 04:49:01 - <Notice> - This is Suricata version 5.0.3 RELEASE running in SYSTEM mode
7/9/2020 -- 04:49:01 - <Debug> - CPUs Summary: 
7/9/2020 -- 04:49:01 - <Debug> - CPUs configured: 2
7/9/2020 -- 04:49:01 - <Info> - CPUs/cores online: 2
7/9/2020 -- 04:49:01 - <Debug> - Entering ... >>
7/9/2020 -- 04:49:01 - <Debug> - Returning: 0 ... <<
7/9/2020 -- 04:49:01 - <Debug> - failed to lookup configuration parameter 'capture.disable-offloading'
7/9/2020 -- 04:49:01 - <Debug> - failed to lookup configuration parameter 'capture.checksum-validation'
7/9/2020 -- 04:49:01 - <Debug> - Entering ... >>
7/9/2020 -- 04:49:01 - <Debug> - Entering ... >>
7/9/2020 -- 04:49:01 - <Debug> - Returning ... <<
7/9/2020 -- 04:49:01 - <Debug> - Returning ... <<
7/9/2020 -- 04:49:01 - <Debug> - Returning ... <<
7/9/2020 -- 04:49:01 - <Debug> - Returning ... <<
7/9/2020 -- 04:49:01 - <Debug> - Returning ... <<
7/9/2020 -- 04:49:01 - <Debug> - Returning ... <<
7/9/2020 -- 04:49:01 - <Debug> - Returning ... <<
7/9/2020 -- 04:49:01 - <Debug> - Returning ... <<
7/9/2020 -- 04:49:01 - <Debug> - Returning: 0 ... <<
...
7/9/2020 -- 04:49:05 - <Debug> - Entering ... >>
7/9/2020 -- 04:49:05 - <Debug> - Returning ... <<
7/9/2020 -- 04:49:05 - <Debug> - Entering ... >>
7/9/2020 -- 04:49:05 - <Debug> - LIBHTP default config: 0x154945940
7/9/2020 -- 04:49:05 - <Debug> - LIBHTP default: personality = IDS
7/9/2020 -- 04:49:05 - <Debug> - LIBHTP default: personality = IDS
7/9/2020 -- 04:49:05 - <Debug> - LIBHTP default: personality=IDS (2)
7/9/2020 -- 04:49:05 - <Debug> - LIBHTP personality set to IDS
Illegal instruction

Additional info:

I also get the Illegal instruction when doing a --dump-features.

I don’t really know what to try next… I think my approach would be to try things like:

  • disable pcre jit
  • disable compiler optimization (-O0)
  • if possible, disable any other optional feature/dependency

It would be helpful if gdb would work, but I have no idea how to do such things.

Maybe its worth looking if there are other suricata cross compiling efforts and how they may be different. E.g. I think Debian cross compiles in their build system.

If it helps, this is the Makefile I’m using inside of OpenWrt. You can see the arguments I’m building with, the deps, etc…

#
# Copyright (C) 2006-2015 OpenWrt.org
#
# This is free software, licensed under the GNU General Public License v2.
# See /LICENSE for more information.
#
include $(TOPDIR)/rules.mk

PKG_NAME:=suricata
PKG_VERSION:=6.0.0-beta1

PKG_SOURCE_PROTO:=git
PKG_SOURCE_DATE:=2020-09-04
PKG_SOURCE_VERSION:=fbdc7765254983360cf2a988e78754c68e52994c
PKG_SOURCE_URL:=https://github.com/OISF/suricata.git
PKG_HASH:=skip

PKG_FIXUP:=autoreconf
PKG_FIXUP:=patch-libtool
#PKG_FIXUP:=gettext-version
PKG_INSTALL:=1

PKG_BUILD_DEPENDS:=rustup/host

include $(INCLUDE_DIR)/package.mk
include $(INCLUDE_DIR)/nls.mk
include ../../lang/rustup/rustc-triple.mk

define Package/suricata6
    SUBMENU:=Firewall
    SECTION:=net
    CATEGORY:=Network
    DEPENDS:=+libpcre +libpcap +libnet-1.2.x +libyaml +zlib +libmagic \
       +jansson +libnfnetlink +lua +liblz4 +libnss +libopenssl \
	  +python3 +python3-yaml +libyaml +libcap-ng +luajit +libmaxminddb \
	  $(ICONV_DEPENDS)
    TITLE:=OISF Suricata IDS
    URL:=https://www.openinfosecfoundation.org/
endef

TARGET_CFLAGS += -ggdb

CONFIGURE_VARS += \
	CARGO_HOME=$(CARGO_HOME) \
	RUSTUP_HOME=$(RUSTUP_HOME) \
	ac_cv_path_CARGO="$(CARGO_HOME)/bin/cargo" \
	ac_cv_path_RUSTC="$(CARGO_HOME)/bin/rustc"

CONFIGURE_ARGS = \
  	--prefix="/usr/" \
  	--sysconfdir="/etc" \
  	--enable-nfqueue \
  	--localstatedir="/var" \
  	--enable-nfqueue \
  	--enable-debug \
	--enable-lua \
	--enable-geoip \
	--disable-gccmarch-native \
	--with-suricata-update \
	--host=$(RUSTC_TARGET_ARCH)

define Build/Prepare
	$(call Build/Prepare/Default)
	cd $(PKG_BUILD_DIR) && git clone https://github.com/OISF/libhtp && \
	cargo install --force cbindgen
	cd $(PKG_BUILD_DIR) && ./autogen.sh
endef

define Build/Compile
	$(call Build/Compile/Default)
endef

define Package/suricata6/install
	$(INSTALL_DIR) $(1)/usr/bin
	$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/suricata $(1)/usr/bin/suricata
	$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/suricatactl $(1)/usr/bin/suricatactl
	$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/suricatasc $(1)/usr/bin/suricatasc

	$(INSTALL_DIR) $(1)/usr/lib
	$(CP) $(PKG_INSTALL_DIR)/usr/lib/* $(1)/usr/lib/
	$(INSTALL_DIR) $(1)/usr/lib/pkgconfig
	$(INSTALL_DIR) $(1)/usr/lib/pthon3.8

	$(INSTALL_DIR) $(1)/usr/include
	$(CP) $(PKG_INSTALL_DIR)/usr/include/* $(1)/usr/include/
	$(INSTALL_DIR) $(1)/usr/include/htp

	$(INSTALL_DIR) $(1)/usr/share
	$(INSTALL_DIR) $(1)/usr/share/suricata
	$(INSTALL_DIR) $(1)/usr/share/doc

	$(INSTALL_DIR) $(1)/etc/suricata
	$(CP) $(PKG_BUILD_DIR)/suricata.yaml \
	$(PKG_BUILD_DIR)/etc/classification.config \
	$(PKG_BUILD_DIR)/threshold.config \
	$(PKG_BUILD_DIR)/etc/reference.config \
	$(1)/etc/suricata/

	$(INSTALL_DIR) $(1)/etc/suricata/rules
	$(CP) $(PKG_BUILD_DIR)/rules/*.rules $(1)/etc/suricata/rules/
	$(INSTALL_DIR) $(1)/etc/init.d
#	$(INSTALL_BIN) ./files/suricata.init $(1)/etc/init.d/suricata
endef

$(eval $(call BuildPackage,suricata6))

Have you been able to make any progress on this?

No. I am willing to do whatever you need to me to help, but keep in mind my knowledge base is limited. Allow and indulge some stupid questions, and I’ll do what I can.

For example. One of the complicating issues is going to be the OpenWrt environment. I can make the connections between the build system and OpenWrt. I have it building with -g3 and -ggdb3, and --enable-debug. Consistently, when I run gdb on suricata (gdb --args suricata -c /etc/suricata/suricata.yaml -i eth0) it tells me:

Reading symbols from suricata...
(No debugging symbols found in suricata)

Starting program: /usr/bin/suricata -v -c /etc/suricata/suricata.yaml -i eth0
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.

Now, I’m building with the right flags, but it can’t find symbols (no suricata.sym is generated. Is it being created by the build system and I’m just failing to copy it over? I get the same error when I try suricata --dump-features, and it always seems to error at libHTP

Below is what I’m moving over to the device (and where). All that is important to know is that $(PKG_INSTALL_DIR) is the buildroot for what was installed via make install.

define Package/suricata6/install
	$(INSTALL_DIR) $(1)/usr/bin
	$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/suricata $(1)/usr/bin/suricata
	$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/suricatactl $(1)/usr/bin/suricatactl
	$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/suricatasc $(1)/usr/bin/suricatasc

	$(INSTALL_DIR) $(1)/usr/lib
	$(CP) $(PKG_INSTALL_DIR)/usr/lib/* $(1)/usr/lib/

	$(INSTALL_DIR) $(1)/usr/include
	$(CP) -r $(PKG_INSTALL_DIR)/usr/include/* $(1)/usr/include/

	$(INSTALL_DIR) $(1)/usr/share
	$(CP) -r $(PKG_INSTALL_DIR)/usr/share/* $(1)/usr/share/

	$(INSTALL_DIR) $(1)/etc/suricata
	$(CP) $(PKG_BUILD_DIR)/suricata.yaml \
	$(PKG_BUILD_DIR)/etc/classification.config \
	$(PKG_BUILD_DIR)/threshold.config \
	$(PKG_BUILD_DIR)/etc/reference.config \
	$(1)/etc/suricata/

You can see my attached build log for the gritty details.

suricata6.log.tgz (36.8 KB)

Finally getting somewhere… But I have no idea what it means :slight_smile: I’m using gdbserver and remote-gdb so i can read the symbols.

Is this a Floating Point issue?

grommish@norwits:~/openwrt$ ./scripts/remote-gdb 192.168.1.1:9000 build_dir/target-mips64_octeon3_64_musl/suricata-6.0.0-beta1/ipkg-install/usr/bin/suricata
Using target mips64_octeon3_64 (musl, )
GNU gdb (GDB) 8.3.1
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=mips64-openwrt-linux-musl".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from build_dir/target-mips64_octeon3_64_musl/suricata-6.0.0-beta1/ipkg-install/usr/bin/suricata...
warning: Unsupported auto-load script at offset 0 in section .debug_gdb_scripts
of file /home/grommish/openwrt/build_dir/target-mips64_octeon3_64_musl/suricata-6.0.0-beta1/ipkg-install/usr/bin/suricata.
Use `info auto-load python-scripts [REGEXP]' to list them.
0x000000fff7f58c00 in _dlstart () from /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/lib/ld-musl-mips64-sf.so.1
(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) n
Program not restarted.
(gdb) bt
#0  0x000000fff7f58c00 in _dlstart ()
   from /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/lib/ld-musl-mips64-sf.so.1
#1  0x0000000000000000 in ?? ()
Backtrace stopped: frame did not save the PC
(gdb) bt
#0  0x000000fff7f58c00 in _dlstart ()
   from /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/lib/ld-musl-mips64-sf.so.1
#1  0x0000000000000000 in ?? ()
Backtrace stopped: frame did not save the PC
(gdb) set remote exec-file /usr/bin/suricata
(gdb) set args -vvvv -c /etc/suricata/suricata.yaml -i eth0
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /home/grommish/openwrt/build_dir/target-mips64_octeon3_64_musl/suricata-6.0.0-beta1/ipkg-install/usr/bin/suricata -vvvv -c /etc/suricata/suricata.yaml -i eth0

Program received signal SIGILL, Illegal instruction.
0x0000000120d7d87c in decfloat (f=0xffffffd650, c=49, bits=53, emin=-1074, sign=1, pok=1) at src/internal/floatscan.c:67
67	src/internal/floatscan.c: No such file or directory.
(gdb) 

What does bt show if you issue it after it crashes with the SIGILL. src/internal/floatscan.c is not in Suricata or libhtp, so it will be interesting how we get there. Might be in a supporting lib.

(gdb) bt
#0  0x0000000120d7d87c in decfloat (f=0xffffffd650, c=49, bits=53, emin=-1074, sign=1, pok=1) at src/internal/floatscan.c:67
#1  0x0000000120b67f94 in strtox (s=0xffffffd808 "100", p=0xffffffd788, prec=prec@entry=1) at src/stdlib/strtod.c:11
#2  0x0000000120b6805c in strtod (s=<optimized out>, p=<optimized out>) at src/stdlib/strtod.c:24
#3  0x0000000120400e18 in ParseSizeString (size=0x12100dd60 "100kb", res=0xffffffd968) at util-misc.c:113
#4  0x0000000120401324 in ParseSizeStringU32 (size=<optimized out>, res=0x120fd6160 <cfglist+48>) at util-misc.c:190
#5  0x0000000120256568 in HTPConfigParseParameters (cfg_prec=0x120fd6130 <cfglist>, s=0x12100dbe0, tree=0x120ffed80) at app-layer-htp.c:2528
#6  0x000000012025c378 in HTPConfigParseParameters (tree=<optimized out>, s=<optimized out>, cfg_prec=<optimized out>) at app-layer-htp.c:2465
#7  HTPConfigure () at app-layer-htp.c:2817
#8  0x000000012025cab4 in RegisterHTPParsers () at app-layer-htp.c:3101
#9  0x0000000120268764 in AppLayerParserRegisterProtocolParsers () at app-layer-parser.c:1561
#10 0x000000012022bf28 in AppLayerSetup () at app-layer.c:812
#11 0x00000001203c06a0 in PostConfLoadedSetup (suri=0x120feba30 <suricata>) at suricata.c:2494
#12 0x00000001203c1930 in SuricataMain (argc=<optimized out>, argv=<optimized out>) at suricata.c:2777
#13 0x000000012022658c in main (argc=<optimized out>, argv=<optimized out>) at main.c:22
(gdb) 

Here is the only floatscan.c in my buildroot

./build_dir/toolchain-mips64_octeon3_64_gcc-10.2.0_musl/musl-1.1.24/src/internal/floatscan.c

Right, so since the SIGILL happens inside a musl function, I suspect the issue is in the way musl is compiled for the hardware. Is it also cross compiled?

Yes… host is x86_64 Ubuntu 20.04, build target is a mips64 MUSL target.

  	--prefix="/usr/" \
  	--sysconfdir="/etc" \
  	--enable-nfqueue \
  	--localstatedir="/var" \
  	--enable-nfqueue \
  	--enable-debug \
	--enable-lua \
	--enable-geoip \
	--disable-gccmarch-native \
	--with-suricata-update \
	--host=$(RUSTC_TARGET_ARCH)

these are my arg calls for ./configure (after autogen.sh). RUSTC_TARGET_ARCH, in my case, is mips64-unknown-linux-muslabi64

I had asked regarding soft-float/hard-float, as the mips64 Octeon3 chip doesn’t have a hardware FPU… It’s software emulated…

 │ If your target CPU does not have a Floating Point Unit (FPU) or a                                                                                    │  
  │ kernel FPU emulator, but you still wish to support floating point                                                                                    │  
  │ functions, then everything will need to be compiled with soft floating                                                                               │  
  │ point support (-msoft-float). 

This is enabled for me.

Here is the file in question it is complaining about:

grommish@norwits:~/openwrt$ ./scripts/remote-gdb 192.168.1.1:9000 build_dir/target-mips64_octeon3_64_musl/suricata-6.0.0-beta1/src/suricata
Using target mips64_octeon3_64 (musl, )
GNU gdb (GDB) 8.3.1
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=mips64-openwrt-linux-musl".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
"/home/grommish/openwrt/build_dir/target-mips64_octeon3_64_musl/suricata-6.0.0-beta1/src/suricata": not in executable format: file format not recognized
Reading symbols from /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/bin/suricata...
warning: Unsupported auto-load script at offset 0 in section .debug_gdb_scripts
of file /home/grommish/openwrt/staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/bin/suricata.
Use `info auto-load python-scripts [REGEXP]' to list them.
0x000000fff7f58c00 in _dlstart () from /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/lib/ld-musl-mips64-sf.so.1
(gdb) set remote exec-file /usr/bin/suricata
(gdb) set args -c /etc/suricata/suricata.yaml -i eth0
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /home/grommish/openwrt/staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/bin/suricata -c /etc/suricata/suricata.yaml -i eth0

Program received signal SIGILL, Illegal instruction.
0x0000000120d7d87c in decfloat (f=0xffffffd670, c=49, bits=53, emin=-1074, sign=1, pok=1) at src/internal/floatscan.c:67
67	src/internal/floatscan.c: No such file or directory.
(gdb) x/i $pc
=> 0x120d7d87c <decfloat+60>:	sdc1	$f24,8376(sp)
(gdb) 

Someone suggested running the x/i $pc in gdb to see if it shows anyting interesting. That was the result.

grommish@norwits:~/openwrt$ ./scripts/remote-gdb 192.168.1.1:9000 build_dir/target-mips64_octeon3_64_musl/suricata-6.0.0-beta1/src/suricata
Using target mips64_octeon3_64 (musl, )
GNU gdb (GDB) 8.3.1
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=mips64-openwrt-linux-musl".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
"/home/grommish/openwrt/build_dir/target-mips64_octeon3_64_musl/suricata-6.0.0-beta1/src/suricata": not in executable format: file format not recognized
Reading symbols from /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/bin/suricata...
warning: Unsupported auto-load script at offset 0 in section .debug_gdb_scripts
of file /home/grommish/openwrt/staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/bin/suricata.
Use `info auto-load python-scripts [REGEXP]' to list them.
0x000000fff7f58c00 in _dlstart () from /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/lib/ld-musl-mips64-sf.so.1
(gdb) set remote exec-file /usr/bin/suricata
(gdb) set args -v -c /etc/suricata/suricata.yaml -i eth0
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /home/grommish/openwrt/staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/bin/suricata -v -c /etc/suricata/suricata.yaml -i eth0

Program received signal SIGILL, Illegal instruction.
0x0000000120ea19cc in decfloat (f=0xffffffd7a0, c=49, bits=53, emin=-1074, sign=1, pok=1) at src/internal/floatscan.c:67
67	src/internal/floatscan.c: No such file or directory.
(gdb) where
#0  0x0000000120ea19cc in decfloat (f=0xffffffd7a0, c=49, bits=53, emin=-1074, sign=1, pok=1) at src/internal/floatscan.c:67
#1  0x0000000120c8c0e4 in strtox (s=0xffffffd978 "100", p=0xffffffd8f0, prec=prec@entry=1) at src/stdlib/strtod.c:11
#2  0x0000000120c8c1ac in strtod (s=<optimized out>, p=<optimized out>) at src/stdlib/strtod.c:24
#3  0x0000000120510890 in ParseSizeString (size=0x12113ed60 "100kb", res=0xffffffdac8) at util-misc.c:113
#4  0x0000000120510d80 in ParseSizeStringU32 (size=0x12113ed60 "100kb", res=0x121107b10 <cfglist+48>) at util-misc.c:190
#5  0x0000000120275060 in HTPConfigParseParameters (cfg_prec=0x121107ae0 <cfglist>, s=0x12113ebe0, tree=0x12112fd80) at app-layer-htp.c:2528
#6  0x0000000120276b34 in HTPConfigure () at app-layer-htp.c:2817
#7  0x0000000120277de4 in RegisterHTPParsers () at app-layer-htp.c:3101
#8  0x0000000120288ef4 in AppLayerParserRegisterProtocolParsers () at app-layer-parser.c:1561
#9  0x000000012022f1cc in AppLayerSetup () at app-layer.c:812
#10 0x00000001204ab720 in PostConfLoadedSetup (suri=0x12111d570 <suricata>) at suricata.c:2494
#11 0x00000001204ac8d0 in SuricataMain (argc=6, argv=0xffffffdd38) at suricata.c:2777
#12 0x00000001202268bc in main (argc=6, argv=0xffffffdd38) at main.c:22
(gdb) list 67
file: "src/internal/floatscan.c", line number: 67, symbol: "???"
62	in src/internal/floatscan.c
file: "src/internal/floatscan.c", line number: 67, symbol: "???"
62		return neg ? -y : y;
63	}
64	
65	
66	static long double decfloat(FILE *f, int c, int bits, int emin, int sign, int pok)
67	{
68		uint32_t x[KMAX];
69		static const uint32_t th[] = { LD_B1B_MAX };
70		int i, j, k, a, z;
71		long long lrp=0, dc=0;
(gdb) x/i 0xffffffd7a0
   0xffffffd7a0:	dsra32	zero,zero,0x3
(gdb) x/1fw 0xffffffd7a0
0xffffffd7a0:	3.57331108e-43
(gdb) x/1dw 0xffffffd7a0
0xffffffd7a0:	255
(gdb) 

In layout asm:

0x120ea19cc <decfloat+60>       sdc1   $f24,8376(sp) 
``

Suggestions?

Note that libc must be built with the compiler flag -msoft-float as well.

It’s not clear if that’s the case in your setup.

Yes. I am seeing -msoft-float in the gcc args. Doing an objdump on floatscan.lo pulled apart the decfloat function, and I failed to see any FPU calls (since I don’t have a hardware FPU), but when I run it on the device, I see the sdc1 call, which is where it was failing.

I am currently building out to ensure ccache isn’t causing issues and removing it from the build. I’ll test again (to make sure ccache wasn’t just placing already compiled objects back into the build-directory), and will let you know.

I am able to do whatever is suggested and put the output, just keep in mind I probably won’t understand what you’re asking to see :slight_smile:

Example: info sharedlibrary returns the following:

(gdb) info sharedlibrary
From                To                  Syms Read   Shared Object Library
0x000000fff7f4a700  0x000000fff7fd4a7c  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/lib/ld-musl-mips64-sf.so.1
0x000000fff7ede810  0x000000fff7f1b540  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/liblua.so.5.1.5
0x000000fff7ea6360  0x000000fff7ec3290  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/libhtp.so.2
0x000000fff7e2f7c0  0x000000fff7e8d850  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/liblz4.so.1
0x000000fff7e14fd0  0x000000fff7e1afb0  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/libmaxminddb.so.0
0x000000fff7dc87c0  0x000000fff7df9d00  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/libmagic.so.1
0x000000fff7da9f40  0x000000fff7dae500  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/libcap-ng.so.0
0x000000fff7d79a70  0x000000fff7d92a60  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/libnet.so.9
0x000000fff7d52150  0x000000fff7d61f00  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/libjansson.so.4
0x000000fff7d0a630  0x000000fff7d3ace0  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/libyaml-0.so.2
0x000000fff7c91410  0x000000fff7cdd680  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/libpcre.so.1
0x000000fff7c54810  0x000000fff7c70180  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/libz.so.1
0x000000fff7be89d0  0x000000fff7c281c0  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/libpcap.so.1
0x000000fff7a4c330  0x000000fff7ba3920  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/libnss3.so
0x000000fff79dd530  0x000000fff7a01340  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/libnssutil3.so
0x000000fff79910e0  0x000000fff79be770  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/libsmime3.so
0x000000fff78f8cf0  0x000000fff79699a0  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/libssl3.so
0x000000fff7873880  0x000000fff78d1d50  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/libsoftokn3.so
0x000000fff7858c60  0x000000fff785ab90  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/libplds4.so
0x000000fff7843080  0x000000fff7846a20  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/libplc4.so
0x000000fff77e0c10  0x000000fff7827f00  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/libnspr4.so
0x000000fff778e960  0x000000fff77c13d0  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/lib/libgcc_s.so.1
0x000000fff7743cd0  0x000000fff7772230  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/liblzma.so.5
0x000000fff7715620  0x000000fff772e470  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/libbz2.so.1.0
0x000000fff754e450  0x000000fff76e7100  Yes         /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/usr/lib/libsqlite3.so.0

Edit: It was suggested that musl might be the issue, so I’m going to attempt to rebuild out with glibc instead and report back

Does this information (info shared lib) correlate with the output of ldd <suricata-binary>?

This is from the suricata bin on-device.

root@OpenWrt:/# ldd /usr/bin/suricata
        /lib/ld-musl-mips64-sf.so.1 (0xfff357b000)
        liblua.so.5.1.5 => /usr/lib/liblua.so.5.1.5 (0xfff3520000)
        libhtp.so.2 => /usr/lib/libhtp.so.2 (0xfff34e7000)
        liblz4.so.1 => /usr/lib/liblz4.so.1 (0xfff3473000)
        libmaxminddb.so.0 => /usr/lib/libmaxminddb.so.0 (0xfff345a000)
        libmagic.so.1 => /usr/lib/libmagic.so.1 (0xfff3406000)
        libcap-ng.so.0 => /usr/lib/libcap-ng.so.0 (0xfff33ef000)
        libnet.so.9 => /usr/lib/libnet.so.9 (0xfff33bb000)
        libjansson.so.4 => /usr/lib/libjansson.so.4 (0xfff3395000)
        libyaml-0.so.2 => /usr/lib/libyaml-0.so.2 (0xfff334e000)
        libpcre.so.1 => /usr/lib/libpcre.so.1 (0xfff32cb000)
        libz.so.1 => /usr/lib/libz.so.1 (0xfff3296000)
        libpcap.so.1 => /usr/lib/libpcap.so.1 (0xfff3220000)
        libnss3.so => /usr/lib/libnss3.so (0xfff3066000)
        libnssutil3.so => /usr/lib/libnssutil3.so (0xfff3017000)
        libsmime3.so => /usr/lib/libsmime3.so (0xfff2fce000)
        libssl3.so => /usr/lib/libssl3.so (0xfff2f30000)
        libsoftokn3.so => /usr/lib/libsoftokn3.so (0xfff2eaf000)
        libplds4.so => /usr/lib/libplds4.so (0xfff2e9b000)
        libplc4.so => /usr/lib/libplc4.so (0xfff2e85000)
        libnspr4.so => /usr/lib/libnspr4.so (0xfff2e18000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xfff2dc7000)
        libc.so => /lib/ld-musl-mips64-sf.so.1 (0xfff357b000)
        liblzma.so.5 => /usr/lib/liblzma.so.5 (0xfff2d82000)
        libbz2.so.1.0 => /usr/lib/libbz2.so.1.0 (0xfff2d55000)
        libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0xfff2b7b000)
root@OpenWrt:/#