Suricata 7.0.0-rc1 crash under RHEL9

Hi all,

Using Suricata 7.0.0-rc1 under RHEL 9.1 host (fully patched) using af-packet crashes:

[17337.852669] traps: Suricata-Main[45474] trap divide error ip:55562126a77d sp:7ffd016951f0 error:0 in suricata[555621232000+603000]
[23889.603869] gmain invoked oom-killer: gfp_mask=0x1140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
[23889.603887] CPU: 0 PID: 775 Comm: gmain Not tainted 5.14.0-162.12.1.el9_1.x86_64 #1
[23889.603890] Hardware name: Red Hat KVM/RHEL, BIOS 1.16.0-4.el9 04/01/2014
[23889.603895] Call Trace:
[23889.603903] dump_stack_lvl+0x34/0x48
[23889.603911] dump_header+0x4a/0x201
[23889.603917] oom_kill_process.cold+0xb/0x10
[23889.603921] out_of_memory+0xed/0x2d0
[23889.603927] __alloc_pages_slowpath.constprop.0+0x7cc/0x8a0
[23889.603932] __alloc_pages+0x1fe/0x230
[23889.603935] folio_alloc+0x17/0x50
[23889.603940] __filemap_get_folio+0x1b6/0x330
[23889.603944] ? do_sync_mmap_readahead+0x14b/0x270
[23889.603947] filemap_fault+0x454/0x7a0
[23889.603950] ? next_uptodate_page+0x160/0x1f0
[23889.603952] ? filemap_map_pages+0x307/0x4a0
[23889.603956] __xfs_filemap_fault+0x66/0x280 [xfs]
[23889.604243] __do_fault+0x36/0x110
[23889.604247] do_read_fault+0xea/0x190
[23889.604251] do_fault+0x8c/0x2c0
[23889.604254] __handle_mm_fault+0x3cb/0x750
[23889.604259] handle_mm_fault+0xc5/0x2a0
[23889.604262] do_user_addr_fault+0x1bb/0x690
[23889.604284] exc_page_fault+0x62/0x150
[23889.604289] asm_exc_page_fault+0x22/0x30
[23889.604295] RIP: 0033:0x7f13f2633720
[23889.604313] Code: Unable to access opcode bytes at RIP 0x7f13f26336f6.
[23889.604314] RSP: 002b:00007f13f0febf88 EFLAGS: 00010202
[23889.604317] RAX: 0000561fc1394860 RBX: 0000561fc1394860 RCX: 0000000000000001
[23889.604318] RDX: 00007f13f2563ae0 RSI: 0000000000005d50 RDI: 0000561fc1395130
[23889.604319] RBP: 0000561fc1395130 R08: 0000561fc1394860 R09: 00007f13f0febec0
[23889.604320] R10: 00007ffca0bd1080 R11: 00007ffca0bd1090 R12: 0000000000000001
[23889.604322] R13: 0000000000000016 R14: 00007f13f0febff0 R15: 0000561fc1395130
[23889.604351] Mem-Info:
[23889.604355] active_anon:432640 inactive_anon:895743 isolated_anon:0
active_file:15 inactive_file:1010 isolated_file:0
unevictable:0 dirty:0 writeback:0
slab_reclaimable:8419 slab_unreclaimable:11849
mapped:117342 shmem:423 pagetables:5956 bounce:0
kernel_misc_reclaimable:0
free:25689 free_pcp:682 free_cma:0
[23889.604360] Node 0 active_anon:1730560kB inactive_anon:3582972kB active_file:60kB inactive_file:4040kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:469368kB dirty:0kB writeback:0kB shmem:1692kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 1677312kB writeback_tmp:0kB kernel_stack:5184kB pagetables:23824kB all_unreclaimable? yes
[23889.604365] Node 0 DMA free:14336kB min:172kB low:212kB high:252kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[23889.604372] lowmem_reserve: 0 1936 5866 5866 5866
[23889.604376] Node 0 DMA32 free:37728kB min:22244kB low:27804kB high:33364kB reserved_highatomic:0KB active_anon:762788kB inactive_anon:1062640kB active_file:0kB inactive_file:1260kB unevictable:0kB writepending:0kB present:2080628kB managed:2015092kB mlocked:0kB bounce:0kB free_pcp:1012kB local_pcp:468kB free_cma:0kB
[23889.604381] lowmem_reserve: 0 0 3930 3930 3930
[23889.604384] Node 0 Normal free:50692kB min:45164kB low:56452kB high:67740kB reserved_highatomic:12288KB active_anon:967772kB inactive_anon:2520332kB active_file:4kB inactive_file:2940kB unevictable:0kB writepending:0kB present:4194304kB managed:4032728kB mlocked:0kB bounce:0kB free_pcp:1716kB local_pcp:776kB free_cma:0kB
[23889.604388] lowmem_reserve: 0 0 0 0 0
[23889.604391] Node 0 DMA: 04kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 12048kB (M) 34096kB (M) = 14336kB
[23889.604402] Node 0 DMA32: 85
4kB (UME) 13088kB (UME) 31716kB (UME) 8932kB (UME) 4564kB (UME) 29128kB (UME) 13256kB (UME) 8512kB (UME) 51024kB (UM) 02048kB 04096kB = 37860kB
[23889.604415] Node 0 Normal: 5984kB (UMEH) 7238kB (UMEH) 80116kB (UMEH) 932kB (ME) 464kB (M) 6128kB (ME) 5256kB (M) 13512kB (M) 201024kB (M) 02048kB 0*4096kB = 50720kB
[23889.604430] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[23889.604432] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[23889.604433] 62213 total pagecache pages
[23889.604434] 60735 pages in swap cache
[23889.604435] Swap cache stats: add 2628344, delete 2567592, find 685736/853012
[23889.604436] Free swap = 0kB
[23889.604437] Total swap = 4194300kB
[23889.604439] 1572731 pages RAM
[23889.604440] 0 pages HighMem/MovableOnly
[23889.604441] 56936 pages reserved
[23889.604441] 0 pages cma reserved
[23889.604442] 0 pages hwpoisoned
[23889.604443] Tasks state (memory values in pages):
[23889.604443] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[23889.604451] [ 631] 0 631 6527 460 81920 150 -250 systemd-journal
[23889.604454] [ 644] 0 644 8360 203 81920 538 -1000 systemd-udevd
[23889.604458] [ 710] 0 710 22953 96 65536 105 -1000 auditd
[23889.604460] [ 712] 0 712 1950 48 53248 50 0 sedispatch
[23889.604463] [ 733] 81 733 2709 58 57344 111 -900 dbus-broker-lau
[23889.604465] [ 734] 81 734 1282 97 49152 38 -900 dbus-broker
[23889.604467] [ 738] 992 738 644212 245 204800 825 0 polkitd
[23889.604470] [ 739] 0 739 41040 253 86016 665 0 rsyslogd
[23889.604472] [ 742] 0 742 5036 240 77824 357 0 systemd-logind
[23889.604474] [ 771] 0 771 64147 714 139264 520 0 NetworkManager
[23889.604477] [ 773] 990 773 21245 152 69632 156 0 chronyd
[23889.604479] [ 783] 0 783 4023 2 65536 406 -1000 sshd
[23889.604481] [ 784] 0 784 1828 21 49152 24 0 rhsmcertd
[23889.604483] [ 786] 0 786 64087 213 131072 3088 0 tuned
[23889.604485] [ 795] 0 795 2137 48 53248 171 0 crond
[23889.604487] [ 855] 0 855 761 18 45056 3 0 agetty
[23889.604498] [ 861] 0 861 1403 19 49152 3 0 agetty
[23889.604502] [ 1015] 0 1015 9529 26 69632 91 0 master
[23889.604522] [ 1026] 89 1026 11303 42 77824 206 0 qmgr
[23889.604525] [ 44402] 0 44402 4785 54 77824 517 0 sshd
[23889.604527] [ 44406] 0 44406 5436 403 86016 353 100 systemd
[23889.604529] [ 44410] 0 44410 43090 308 102400 986 100 (sd-pam)
[23889.604532] [ 44417] 0 44417 4785 95 77824 473 0 sshd
[23889.604534] [ 44418] 0 44418 1911 164 53248 63 0 bash
[23889.604536] [ 44455] 0 44455 4785 56 77824 514 0 sshd
[23889.604538] [ 44458] 0 44458 4842 101 77824 531 0 sshd
[23889.604540] [ 44459] 0 44459 1938 123 53248 118 0 bash
[23889.604542] [ 44573] 0 44573 4785 464 81920 106 0 sshd
[23889.604544] [ 44576] 0 44576 4785 429 81920 138 0 sshd
[23889.604546] [ 44577] 0 44577 1936 185 53248 53 0 bash
[23889.604548] [ 44680] 0 44680 3720 11 61440 329 0 crond
[23889.604550] [ 44681] 0 44681 1780 2 53248 74 0 run-parts
[23889.604552] [ 44693] 0 44693 1780 1 53248 66 0 update_intel_fe
[23889.604554] [ 44694] 0 44694 1636 0 53248 38 0 sed
[23889.604556] [ 44697] 0 44697 498979 239248 4018176 252445 0 python
[23889.604558] [ 44701] 1000 44701 83418 8828 610304 16332 0 splunkd
[23889.604561] [ 44727] 1000 44727 29718 12 151552 3479 0 splunkd
[23889.604563] [ 44870] 0 44870 1813 1 57344 85 0 run-zeek
[23889.604565] [ 44876] 0 44876 116104 10306 544768 23858 0 zeek
[23889.604568] [ 44919] 0 44919 1813 2 49152 85 0 run-zeek
[23889.604572] [ 44925] 0 44925 890754 374125 6270976 361332 0 zeek
[23889.604575] [ 45002] 0 45002 1813 2 53248 85 0 run-zeek
[23889.604578] [ 45008] 0 45008 70769 6572 462848 26196 0 zeek
[23889.604582] [ 45060] 0 45060 1813 2 53248 85 0 run-zeek
[23889.604586] [ 45066] 0 45066 241531 47442 1757184 123509 0 zeek
[23889.604590] [ 45304] 89 45304 11649 126 90112 282 0 tlsmgr
[23889.604594] [ 45642] 0 45642 313197 127731 1953792 97694 0 Suricata-Main
[23889.604599] [ 45643] 0 45643 1404 23 61440 0 0 tail
[23889.604603] [ 47752] 0 47752 3720 0 61440 348 0 crond
[23889.604608] [ 47753] 0 47753 1780 2 57344 74 0 run-parts
[23889.604612] [ 47765] 0 47765 1780 0 57344 67 0 update_intel_fe
[23889.604616] [ 47766] 0 47766 1636 0 49152 38 0 sed
[23889.604620] [ 47769] 0 47769 488441 349318 3944448 133009 0 python
[23889.604626] [ 50165] 0 50165 3720 177 61440 163 0 crond
[23889.604630] [ 50166] 0 50166 1780 75 49152 0 0 run-parts
[23889.604633] [ 50178] 0 50178 1780 62 61440 0 0 update_intel_fe
[23889.604635] [ 50179] 0 50179 1636 28 45056 0 0 sed
[23889.604637] [ 50182] 0 50182 215926 212474 1769472 0 0 python
[23889.604639] [ 50183] 89 50183 11292 240 77824 0 0 pickup
[23889.604641] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=NetworkManager.service,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-7.scope,task=zeek,pid=44925,uid=0
[23889.604684] Out of memory: Killed process 44925 (zeek) total-vm:3563016kB, anon-rss:1496500kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:6124kB oom_score_adj:0
[26856.516170] traps: Suricata-Main[45642] trap divide error ip:55fe8af2f77d sp:7ffd53719b10 error:0 in suricata[55fe8aef7000+603000]

Thanks for posing the OOM details.

Does your deployment have a resource limiting mechanism in place? Sometimes resource limits are imposed using cgroups and the OOM may have been triggered from the resource limit being reached or exceeded.

Suricata memory usage varies according to

  • Worker thread count
  • Traffic mix
    and other factors so it would help to have your deployment’s
  • Suricata configuration (suricata.yaml)
  • Packet acquisition method (e.g., netmap, afpacket, dpdk, etc)
  • Existing memory limits (if used)
  • Statistics log (stats.log)