JQ cheat sheet for parsing Suricata EVE outputs

Guides
1

JQ quick commands for some common usage situations for Suricata EVE logs

As shared by @cthomas in July’s 2023 webinar: Using JQ to parse Suricata logs.

Pick out single event type

jq -c 'select(.alert)' eve.json

Pick out event type with single condition

jq 'select(.event_type=="tls" and .tls.subject==.tls.issuerdn)' eve.json

jq 'select(.event_type=="tls" and .tls.version=="TLSv1")' eve.json

Pick out event type with multiple conditions

jq 'select(.event_type=="flow" and .flow.state=="established" and .flow.age >1800 and .app_proto=="smb")' eve.json

jq 'select(.event_type=="flow" and .flow.state=="established" and .flow.age>1800 and .app_proto=="smb" and .timestamp>"2019-04-29T14:38")' eve.json

String comparisons

jq -c 'select(.event_type=="alert")' eve.json | jq -c 'select(.alert.signature | contains("HTTP") )'

jq -c 'select((.event_type=="alert") and (.alert.signature | contains("HTTP") ) )' eve.json

alternatives:

   startswith("str")

   endswith("str")

Conditions, but only a single field

jq -c 'select((.event_type=="alert") and (.alert.signature | contains("HTTP") ) ) | .alert.signature' eve.json

Counting events

jq 'select(.event_type=="smb" and .smb.status == "STATUS_ACCESS_DENIED")' eve.json | jq .flow_id | sort | uniq -c

grep '"event_type":' eve.json  | jq .event_type | sort -rn  | uniq -c | sort -rn

Conditions and multiple fields

jq -c 'select((.event_type=="alert") and (.alert.signature | contains("HTTP") ) ) | .alert.signature, .flow.bytes_toclient' eve.json

Joining multiple fields

jq -r 'select((.event_type=="alert") and (.alert.signature | contains("HTTP") ) ) | [.alert.signature, .flow.bytes_toclient] | join(", ")' eve.json

Importing (into csv)

jq -r 'select((.event_type=="alert") and (.alert.signature | contains("HTTP") ) ) | [.alert.signature, .flow.bytes_toclient] | join(", ")' eve.json | sort -d > /tmp/test.csv

Mixing data types

(pre-command for combining filestore)

find filestore/ | grep "json$" | xargs cat >> full.json

jq -r '.src_ip +" "+ .dest_ip +" "+ (.src_port | tostring) +" "+
(.dest_port | tostring)' full.json |
while read line ; do read sip dip sp dp <<<$(echo $line) ; echo "srcip = $sip --- dstip = $dip --- srcport = $sp --- dstport = $dp" ; done

Avoiding ‘null’ empty fields

(has null in IPS ‘drop’ eve type)

jq '.alert.signature' drops.json

(removes any null lines)

jq '.alert.signature | select ( . != null)' drops.json

(gets fields that exist on your own priority)

jq 'if .alert.signature then .alert.signature elif .metadata.flowbits > 0 then .metadata.flowbits[] else .proto end' drops.json

DNS rrname sort

jq 'select(.event_type=="dns")' eve.json | jq .dns.rrname | sort -rn | uniq -c | sort -rn

PCAP splitting script

#!/bin/bash
njobs="\j"      # special bash job info

jq -c -r '.src_ip +" "+ .dest_ip +" "+ (.src_port|tostring) +" "+ (.dest_port|tostring)' filestore_combined.json  |

while read line ; do
   while (( ${njobs@P} >= 29)) ; do    wait -n ;    done     # wait for this process job count to  be below 29
   read sip dip sp dp <<<$(echo $line)
   echo "srcip = $sip --- $dstip = $dip --- srcport = $sp --- dstport = $dp"

tcpdump -r /path/to/200someGB.pcap "host $sip and host $dip and port $sp and port $dp" -w /tmp/split-pcaps/${sip}_${sp}-${dip}_${dp}.pcap &

done

Extras Base vs Split EVE json

Default EVE

outputs:
  # a line based alerts log similar to Snort's fast.log
  - fast:
      enabled: yes
      filename: fast.log
      append: yes
      #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'

  # Extensible Event Format (nicknamed EVE) event log in JSON format
  - eve-log:
      enabled: yes
      filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
      filename: eve.json
    ...
    ...
    ...
      types:
        - alert:
            # payload: yes             # enable dumping payload in Base64
            # payload-buffer-size: 4kb # max size of payload buffer to output in eve-log
            # payload-printable: yes   # enable dumping payload in printable (lossy) format
            # packet: yes              # enable dumping of packet (without stream segments)
            # metadata: no             # enable inclusion of app layer metadata with alert. Default yes
            # http-body: yes           # Requires metadata; enable dumping of HTTP body in Base64
            # http-body-printable: yes # Requires metadata; enable dumping of HTTP body in printable format

            # Enable the logging of tagged packets for rules using the
            # "tag" keyword.
            tagged-packets: yes

Split EVE

outputs:
  # a line based alerts log similar to Snort's fast.log
  - fast:
      enabled: yes
      filename: /dev/null
      append: yes
      #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'

  # Stats.log contains data from various counters of the Suricata engine.
  - eve-log:
      enabled: yes
      append: no        # append to file (yes) or overwrite it (no)
      filename: stats.json
      types:
      - stats:
          totals: yes       # stats for all threads merged together
          threads: no       # per thread stats
          deltas: no        # include delta values
    ...
    ...
    ...
  # Extensible Event Format (nicknamed EVE) event log in JSON format
  - eve-log:
      enabled: yes
      filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
      filename: eve.json
    ...
    ...
    ...
        #- stats:
        #    totals: yes       # stats for all threads merged together
        #    threads: no       # per thread stats
        #    deltas: no        # include delta values
1 Like