I am cross compiling Suricata 6.0.0-beta1 for a mips64 Octeon3 SoC running OpenWrt.
I have incorporated Rust/Cargo into OpenWrt, along with everything else to get Suricata to build (although I’m still having issues with suricata-update, but that’s another issue) (Thank you so much for fixing the cross_compile issues I was having)
When I attempt to run suricata, I get an Illegal Instruction error. Attached are the logs from a -vvvv run, and a full --dump-config
(gdb) run -vvvv -c /etc/suricata/suricata.yaml -s rules/*.rules -i eth0
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /usr/bin/suricata -vvvv -c /etc/suricata/suricata.yaml -s rules/*.rules -i eth0
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
....
Program received signal SIGILL, Illegal instruction.
0x000000aaab85f3ac in ?? ()
(gdb)
Is what it gives me. I do use --enable-debug, but I’ve not used gdb, but I can follow instructions.
root@OpenWrt:/# gdb suricata
GNU gdb (GDB) 8.3.1
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "mips64-openwrt-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from suricata...
(No debugging symbols found in suricata)
(gdb) bt
No stack.
(gdb) run -c /etc/suricata/suricata.yaml -i eth0 -s rules/*.rules
Starting program: /usr/bin/suricata -c /etc/suricata/suricata.yaml -i eth0 -s rules/*.rules
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
Error opening file /var/log/suricata//suricata.log
[1967] 5/9/2020 -- 10:54:06 - (suricata.c:1065) <Notice> (LogVersion) -- This is Suricata version 6.0.0-dev running in SYSTEM mode
Program received signal SIGILL, Illegal instruction.
0x000000aaab85f3ac in ?? ()
(gdb) bt
#0 0x000000aaab85f3ac in ?? ()
warning: GDB can't find the start of the function at 0xaaab85f3ac.
GDB is unable to find the start of the function at 0xaaab85f3ac
and thus can't determine the size of that function's stack frame.
This means that GDB may be unable to access that stack frame, or
the frames below it.
This problem is most likely caused by an invalid program counter or
stack pointer.
However, if you think GDB should simply search farther back
from 0xaaab85f3ac for code which looks like the beginning of a
function, you can increase the range of the search using the `set
heuristic-fence-post' command.
(gdb)
So I tried with 5.0.3, and still received the same Illegal Instruction… Both 5.0.3 and 6.0.0-beta1 seem to crash in LIBHTP? I clone the LibHTP master branch into the suricata directory before building.
I don’t really know what to try next… I think my approach would be to try things like:
disable pcre jit
disable compiler optimization (-O0)
if possible, disable any other optional feature/dependency
It would be helpful if gdb would work, but I have no idea how to do such things.
Maybe its worth looking if there are other suricata cross compiling efforts and how they may be different. E.g. I think Debian cross compiles in their build system.
No. I am willing to do whatever you need to me to help, but keep in mind my knowledge base is limited. Allow and indulge some stupid questions, and I’ll do what I can.
For example. One of the complicating issues is going to be the OpenWrt environment. I can make the connections between the build system and OpenWrt. I have it building with -g3 and -ggdb3, and --enable-debug. Consistently, when I run gdb on suricata (gdb --args suricata -c /etc/suricata/suricata.yaml -i eth0) it tells me:
Reading symbols from suricata...
(No debugging symbols found in suricata)
Starting program: /usr/bin/suricata -v -c /etc/suricata/suricata.yaml -i eth0
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
Now, I’m building with the right flags, but it can’t find symbols (no suricata.sym is generated. Is it being created by the build system and I’m just failing to copy it over? I get the same error when I try suricata --dump-features, and it always seems to error at libHTP…
Below is what I’m moving over to the device (and where). All that is important to know is that $(PKG_INSTALL_DIR) is the buildroot for what was installed via make install.
Finally getting somewhere… But I have no idea what it means I’m using gdbserver and remote-gdb so i can read the symbols.
Is this a Floating Point issue?
grommish@norwits:~/openwrt$ ./scripts/remote-gdb 192.168.1.1:9000 build_dir/target-mips64_octeon3_64_musl/suricata-6.0.0-beta1/ipkg-install/usr/bin/suricata
Using target mips64_octeon3_64 (musl, )
GNU gdb (GDB) 8.3.1
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=mips64-openwrt-linux-musl".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from build_dir/target-mips64_octeon3_64_musl/suricata-6.0.0-beta1/ipkg-install/usr/bin/suricata...
warning: Unsupported auto-load script at offset 0 in section .debug_gdb_scripts
of file /home/grommish/openwrt/build_dir/target-mips64_octeon3_64_musl/suricata-6.0.0-beta1/ipkg-install/usr/bin/suricata.
Use `info auto-load python-scripts [REGEXP]' to list them.
0x000000fff7f58c00 in _dlstart () from /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/lib/ld-musl-mips64-sf.so.1
(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) n
Program not restarted.
(gdb) bt
#0 0x000000fff7f58c00 in _dlstart ()
from /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/lib/ld-musl-mips64-sf.so.1
#1 0x0000000000000000 in ?? ()
Backtrace stopped: frame did not save the PC
(gdb) bt
#0 0x000000fff7f58c00 in _dlstart ()
from /home/grommish/openwrt/scripts/../staging_dir/target-mips64_octeon3_64_musl/root-octeon/lib/ld-musl-mips64-sf.so.1
#1 0x0000000000000000 in ?? ()
Backtrace stopped: frame did not save the PC
(gdb) set remote exec-file /usr/bin/suricata
(gdb) set args -vvvv -c /etc/suricata/suricata.yaml -i eth0
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /home/grommish/openwrt/build_dir/target-mips64_octeon3_64_musl/suricata-6.0.0-beta1/ipkg-install/usr/bin/suricata -vvvv -c /etc/suricata/suricata.yaml -i eth0
Program received signal SIGILL, Illegal instruction.
0x0000000120d7d87c in decfloat (f=0xffffffd650, c=49, bits=53, emin=-1074, sign=1, pok=1) at src/internal/floatscan.c:67
67 src/internal/floatscan.c: No such file or directory.
(gdb)
What does bt show if you issue it after it crashes with the SIGILL. src/internal/floatscan.c is not in Suricata or libhtp, so it will be interesting how we get there. Might be in a supporting lib.
(gdb) bt
#0 0x0000000120d7d87c in decfloat (f=0xffffffd650, c=49, bits=53, emin=-1074, sign=1, pok=1) at src/internal/floatscan.c:67
#1 0x0000000120b67f94 in strtox (s=0xffffffd808 "100", p=0xffffffd788, prec=prec@entry=1) at src/stdlib/strtod.c:11
#2 0x0000000120b6805c in strtod (s=<optimized out>, p=<optimized out>) at src/stdlib/strtod.c:24
#3 0x0000000120400e18 in ParseSizeString (size=0x12100dd60 "100kb", res=0xffffffd968) at util-misc.c:113
#4 0x0000000120401324 in ParseSizeStringU32 (size=<optimized out>, res=0x120fd6160 <cfglist+48>) at util-misc.c:190
#5 0x0000000120256568 in HTPConfigParseParameters (cfg_prec=0x120fd6130 <cfglist>, s=0x12100dbe0, tree=0x120ffed80) at app-layer-htp.c:2528
#6 0x000000012025c378 in HTPConfigParseParameters (tree=<optimized out>, s=<optimized out>, cfg_prec=<optimized out>) at app-layer-htp.c:2465
#7 HTPConfigure () at app-layer-htp.c:2817
#8 0x000000012025cab4 in RegisterHTPParsers () at app-layer-htp.c:3101
#9 0x0000000120268764 in AppLayerParserRegisterProtocolParsers () at app-layer-parser.c:1561
#10 0x000000012022bf28 in AppLayerSetup () at app-layer.c:812
#11 0x00000001203c06a0 in PostConfLoadedSetup (suri=0x120feba30 <suricata>) at suricata.c:2494
#12 0x00000001203c1930 in SuricataMain (argc=<optimized out>, argv=<optimized out>) at suricata.c:2777
#13 0x000000012022658c in main (argc=<optimized out>, argv=<optimized out>) at main.c:22
(gdb)
Right, so since the SIGILL happens inside a musl function, I suspect the issue is in the way musl is compiled for the hardware. Is it also cross compiled?
I had asked regarding soft-float/hard-float, as the mips64 Octeon3 chip doesn’t have a hardware FPU… It’s software emulated…
│ If your target CPU does not have a Floating Point Unit (FPU) or a │
│ kernel FPU emulator, but you still wish to support floating point │
│ functions, then everything will need to be compiled with soft floating │
│ point support (-msoft-float).
This is enabled for me.
Here is the file in question it is complaining about:
I’m using gdbserver and remote-gdb so i can read the symbols.