Suricata6.0.0-beta1 on OpenWrt Illegal Instruction error

I am cross compiling Suricata 6.0.0-beta1 for a mips64 Octeon3 SoC running OpenWrt.

I have incorporated Rust/Cargo into OpenWrt, along with everything else to get Suricata to build (although I’m still having issues with suricata-update, but that’s another issue) (Thank you so much for fixing the cross_compile issues I was having)

When I attempt to run suricata, I get an Illegal Instruction error. Attached are the logs from a -vvvv run, and a full --dump-config

Any help would be appreciated!

suricata6.log (71.0 KB) suricata6-dumpconfig.log (17.9 KB)

Can you add the output of suricata --build-info?

Yes, sir.

suricata6-buildinfo.log (4.2 KB)

Nothing stands out to me. Are you able to run in gdb to find at what line it aborts?

(gdb) run -vvvv -c /etc/suricata/suricata.yaml -s rules/*.rules -i eth0                                                                                                                
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /usr/bin/suricata -vvvv -c /etc/suricata/suricata.yaml -s rules/*.rules -i eth0
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.

....

Program received signal SIGILL, Illegal instruction.
0x000000aaab85f3ac in ?? ()
(gdb) 

Is what it gives me. I do use --enable-debug, but I’ve not used gdb, but I can follow instructions.

Can you share the output of bt when inside this gdb session?

You may need to add -ggdb to your CFLAGS before compiling to get something useful out of the bt, but lets try without first.

root@OpenWrt:/# gdb suricata
GNU gdb (GDB) 8.3.1
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "mips64-openwrt-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from suricata...
(No debugging symbols found in suricata)
(gdb) bt
No stack.
(gdb) run -c /etc/suricata/suricata.yaml -i eth0 -s rules/*.rules
Starting program: /usr/bin/suricata -c /etc/suricata/suricata.yaml -i eth0 -s rules/*.rules
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
Error opening file /var/log/suricata//suricata.log
[1967] 5/9/2020 -- 10:54:06 - (suricata.c:1065) <Notice> (LogVersion) -- This is Suricata version 6.0.0-dev running in SYSTEM mode

Program received signal SIGILL, Illegal instruction.
0x000000aaab85f3ac in ?? ()
(gdb) bt
#0  0x000000aaab85f3ac in ?? ()
warning: GDB can't find the start of the function at 0xaaab85f3ac.

    GDB is unable to find the start of the function at 0xaaab85f3ac
and thus can't determine the size of that function's stack frame.
This means that GDB may be unable to access that stack frame, or
the frames below it.
    This problem is most likely caused by an invalid program counter or
stack pointer.
    However, if you think GDB should simply search farther back
from 0xaaab85f3ac for code which looks like the beginning of a
function, you can increase the range of the search using the `set
heuristic-fence-post' command.
(gdb) 

I’m rebuilding the firmware with the -ggdb CFLAG

building with the CFLAG -ggdb did not produce any different results. Suggestions?

So I tried with 5.0.3, and still received the same Illegal Instruction… Both 5.0.3 and 6.0.0-beta1 seem to crash in LIBHTP? I clone the LibHTP master branch into the suricata directory before building.

root@OpenWrt:/etc# suricata -c /etc/suricata/suricata.yaml -i eth0
Error opening file /var/log/suricata//suricata.log
7/9/2020 -- 04:48:53 - <Notice> - This is Suricata version 5.0.3 RELEASE running in SYSTEM mode
Illegal instruction
root@OpenWrt:/etc# suricata -vvvv -c /etc/suricata/suricata.yaml -i eth0
Error opening file /var/log/suricata//suricata.log
7/9/2020 -- 04:49:01 - <Debug> - sc_log_global_log_level: 10
7/9/2020 -- 04:49:01 - <Debug> - sc_lc->log_format: %t - <%d> - 
7/9/2020 -- 04:49:01 - <Debug> - SCLogSetOPFilter: filter: (null)
7/9/2020 -- 04:49:01 - <Notice> - This is Suricata version 5.0.3 RELEASE running in SYSTEM mode
7/9/2020 -- 04:49:01 - <Debug> - CPUs Summary: 
7/9/2020 -- 04:49:01 - <Debug> - CPUs configured: 2
7/9/2020 -- 04:49:01 - <Info> - CPUs/cores online: 2
7/9/2020 -- 04:49:01 - <Debug> - Entering ... >>
7/9/2020 -- 04:49:01 - <Debug> - Returning: 0 ... <<
7/9/2020 -- 04:49:01 - <Debug> - failed to lookup configuration parameter 'capture.disable-offloading'
7/9/2020 -- 04:49:01 - <Debug> - failed to lookup configuration parameter 'capture.checksum-validation'
7/9/2020 -- 04:49:01 - <Debug> - Entering ... >>
7/9/2020 -- 04:49:01 - <Debug> - Entering ... >>
7/9/2020 -- 04:49:01 - <Debug> - Returning ... <<
7/9/2020 -- 04:49:01 - <Debug> - Returning ... <<
7/9/2020 -- 04:49:01 - <Debug> - Returning ... <<
7/9/2020 -- 04:49:01 - <Debug> - Returning ... <<
7/9/2020 -- 04:49:01 - <Debug> - Returning ... <<
7/9/2020 -- 04:49:01 - <Debug> - Returning ... <<
7/9/2020 -- 04:49:01 - <Debug> - Returning ... <<
7/9/2020 -- 04:49:01 - <Debug> - Returning ... <<
7/9/2020 -- 04:49:01 - <Debug> - Returning: 0 ... <<
...
7/9/2020 -- 04:49:05 - <Debug> - Entering ... >>
7/9/2020 -- 04:49:05 - <Debug> - Returning ... <<
7/9/2020 -- 04:49:05 - <Debug> - Entering ... >>
7/9/2020 -- 04:49:05 - <Debug> - LIBHTP default config: 0x154945940
7/9/2020 -- 04:49:05 - <Debug> - LIBHTP default: personality = IDS
7/9/2020 -- 04:49:05 - <Debug> - LIBHTP default: personality = IDS
7/9/2020 -- 04:49:05 - <Debug> - LIBHTP default: personality=IDS (2)
7/9/2020 -- 04:49:05 - <Debug> - LIBHTP personality set to IDS
Illegal instruction

Additional info:

I also get the Illegal instruction when doing a --dump-features.

I don’t really know what to try next… I think my approach would be to try things like:

  • disable pcre jit
  • disable compiler optimization (-O0)
  • if possible, disable any other optional feature/dependency

It would be helpful if gdb would work, but I have no idea how to do such things.

Maybe its worth looking if there are other suricata cross compiling efforts and how they may be different. E.g. I think Debian cross compiles in their build system.

If it helps, this is the Makefile I’m using inside of OpenWrt. You can see the arguments I’m building with, the deps, etc…

#
# Copyright (C) 2006-2015 OpenWrt.org
#
# This is free software, licensed under the GNU General Public License v2.
# See /LICENSE for more information.
#
include $(TOPDIR)/rules.mk

PKG_NAME:=suricata
PKG_VERSION:=6.0.0-beta1

PKG_SOURCE_PROTO:=git
PKG_SOURCE_DATE:=2020-09-04
PKG_SOURCE_VERSION:=fbdc7765254983360cf2a988e78754c68e52994c
PKG_SOURCE_URL:=https://github.com/OISF/suricata.git
PKG_HASH:=skip

PKG_FIXUP:=autoreconf
PKG_FIXUP:=patch-libtool
#PKG_FIXUP:=gettext-version
PKG_INSTALL:=1

PKG_BUILD_DEPENDS:=rustup/host

include $(INCLUDE_DIR)/package.mk
include $(INCLUDE_DIR)/nls.mk
include ../../lang/rustup/rustc-triple.mk

define Package/suricata6
    SUBMENU:=Firewall
    SECTION:=net
    CATEGORY:=Network
    DEPENDS:=+libpcre +libpcap +libnet-1.2.x +libyaml +zlib +libmagic \
       +jansson +libnfnetlink +lua +liblz4 +libnss +libopenssl \
	  +python3 +python3-yaml +libyaml +libcap-ng +luajit +libmaxminddb \
	  $(ICONV_DEPENDS)
    TITLE:=OISF Suricata IDS
    URL:=https://www.openinfosecfoundation.org/
endef

TARGET_CFLAGS += -ggdb

CONFIGURE_VARS += \
	CARGO_HOME=$(CARGO_HOME) \
	RUSTUP_HOME=$(RUSTUP_HOME) \
	ac_cv_path_CARGO="$(CARGO_HOME)/bin/cargo" \
	ac_cv_path_RUSTC="$(CARGO_HOME)/bin/rustc"

CONFIGURE_ARGS = \
  	--prefix="/usr/" \
  	--sysconfdir="/etc" \
  	--enable-nfqueue \
  	--localstatedir="/var" \
  	--enable-nfqueue \
  	--enable-debug \
	--enable-lua \
	--enable-geoip \
	--disable-gccmarch-native \
	--with-suricata-update \
	--host=$(RUSTC_TARGET_ARCH)

define Build/Prepare
	$(call Build/Prepare/Default)
	cd $(PKG_BUILD_DIR) && git clone https://github.com/OISF/libhtp && \
	cargo install --force cbindgen
	cd $(PKG_BUILD_DIR) && ./autogen.sh
endef

define Build/Compile
	$(call Build/Compile/Default)
endef

define Package/suricata6/install
	$(INSTALL_DIR) $(1)/usr/bin
	$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/suricata $(1)/usr/bin/suricata
	$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/suricatactl $(1)/usr/bin/suricatactl
	$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/suricatasc $(1)/usr/bin/suricatasc

	$(INSTALL_DIR) $(1)/usr/lib
	$(CP) $(PKG_INSTALL_DIR)/usr/lib/* $(1)/usr/lib/
	$(INSTALL_DIR) $(1)/usr/lib/pkgconfig
	$(INSTALL_DIR) $(1)/usr/lib/pthon3.8

	$(INSTALL_DIR) $(1)/usr/include
	$(CP) $(PKG_INSTALL_DIR)/usr/include/* $(1)/usr/include/
	$(INSTALL_DIR) $(1)/usr/include/htp

	$(INSTALL_DIR) $(1)/usr/share
	$(INSTALL_DIR) $(1)/usr/share/suricata
	$(INSTALL_DIR) $(1)/usr/share/doc

	$(INSTALL_DIR) $(1)/etc/suricata
	$(CP) $(PKG_BUILD_DIR)/suricata.yaml \
	$(PKG_BUILD_DIR)/etc/classification.config \
	$(PKG_BUILD_DIR)/threshold.config \
	$(PKG_BUILD_DIR)/etc/reference.config \
	$(1)/etc/suricata/

	$(INSTALL_DIR) $(1)/etc/suricata/rules
	$(CP) $(PKG_BUILD_DIR)/rules/*.rules $(1)/etc/suricata/rules/
	$(INSTALL_DIR) $(1)/etc/init.d
#	$(INSTALL_BIN) ./files/suricata.init $(1)/etc/init.d/suricata
endef

$(eval $(call BuildPackage,suricata6))

Have you been able to make any progress on this?

No. I am willing to do whatever you need to me to help, but keep in mind my knowledge base is limited. Allow and indulge some stupid questions, and I’ll do what I can.

For example. One of the complicating issues is going to be the OpenWrt environment. I can make the connections between the build system and OpenWrt. I have it building with -g3 and -ggdb3, and --enable-debug. Consistently, when I run gdb on suricata (gdb --args suricata -c /etc/suricata/suricata.yaml -i eth0) it tells me:

Reading symbols from suricata...
(No debugging symbols found in suricata)

Starting program: /usr/bin/suricata -v -c /etc/suricata/suricata.yaml -i eth0
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.

Now, I’m building with the right flags, but it can’t find symbols (no suricata.sym is generated. Is it being created by the build system and I’m just failing to copy it over? I get the same error when I try suricata --dump-features, and it always seems to error at libHTP

Below is what I’m moving over to the device (and where). All that is important to know is that $(PKG_INSTALL_DIR) is the buildroot for what was installed via make install.

define Package/suricata6/install
	$(INSTALL_DIR) $(1)/usr/bin
	$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/suricata $(1)/usr/bin/suricata
	$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/suricatactl $(1)/usr/bin/suricatactl
	$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/suricatasc $(1)/usr/bin/suricatasc

	$(INSTALL_DIR) $(1)/usr/lib
	$(CP) $(PKG_INSTALL_DIR)/usr/lib/* $(1)/usr/lib/

	$(INSTALL_DIR) $(1)/usr/include
	$(CP) -r $(PKG_INSTALL_DIR)/usr/include/* $(1)/usr/include/

	$(INSTALL_DIR) $(1)/usr/share
	$(CP) -r $(PKG_INSTALL_DIR)/usr/share/* $(1)/usr/share/

	$(INSTALL_DIR) $(1)/etc/suricata
	$(CP) $(PKG_BUILD_DIR)/suricata.yaml \
	$(PKG_BUILD_DIR)/etc/classification.config \
	$(PKG_BUILD_DIR)/threshold.config \
	$(PKG_BUILD_DIR)/etc/reference.config \
	$(1)/etc/suricata/

You can see my attached build log for the gritty details.

suricata6.log.tgz (36.8 KB)