Suricata the output of eve.log does not include data on http connections

asmirnov2 · November 18, 2021, 4:21pm

I downloaded dashboards for ELK Suricata, but there is no data on any HTTP dashboard. Checked the presence of fields in the output (For example http.server.keyword, http.status) and there are none. Please tell me if I have configured logging incorrectly or suricata does not check http?

asmirnov2 · November 18, 2021, 4:24pm

asmirnov2 · November 18, 2021, 4:41pm

At the same time, I see in the logs that suricata captures packets with port 80

jufajardini · November 18, 2021, 4:46pm

Hi,

Could you maybe share your suricata.yaml config file and/or confirm that you have http enabled in it?

Suricata is capable of logging a lot of info on http (please check
15.1.1. Eve JSON Output — Suricata 7.0.0-dev documentation )

Is that output you’ve shared from stats.log?

asmirnov2 · November 19, 2021, 10:30am

# Extensible Event Format (nicknamed EVE) event log in JSON format
  - eve-log:
      enabled: yes
      filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
      filename: eve.json
      # Enable for multi-threaded eve.json output; output files are amended with
      # with an identifier, e.g., eve.9.json
      #threaded: false
      #prefix: "@cee: " # prefix to prepend to each log entry
      # the following are valid when type: syslog above
      identity: "Suricata-IDSSU-01"
      facility: local5
      level: Info ## possible levels: Emergency, Alert, Critical,
                   ## Error, Warning, Notice, Info, Debug

      # include the name of the input pcap file in pcap file processing mode
      pcap-file: false

      # Community Flow ID
      # Adds a 'community_id' field to EVE records. These are meant to give
      # records a predictable flow ID that can be used to match records to
      # output of other tools such as Zeek (Bro).
      #
      # Takes a 'seed' that needs to be same across sensors and tools
      # to make the id less predictable.

      # enable/disable the community id feature.
      community-id: true
      # Seed value for the ID output. Valid values are 0-65535.
      community-id-seed: 0

      # HTTP X-Forwarded-For support by adding an extra field or overwriting
      # the source or destination IP address (depending on flow direction)
      # with the one reported in the X-Forwarded-For HTTP header. This is
      # helpful when reviewing alerts for traffic that is being reverse
      # or forward proxied.
      xff:
        enabled: no
        # Two operation modes are available: "extra-data" and "overwrite".
        mode: extra-data
        # Two proxy deployments are supported: "reverse" and "forward". In
        # a "reverse" deployment the IP address used is the last one, in a
        # "forward" deployment the first IP address is used.
        deployment: reverse
        # Header name where the actual IP address will be reported. If more
        # than one IP address is present, the last IP address will be the
        # one taken into consideration.
        header: X-Forwarded-For

      types:
        - alert:
            # payload: yes             # enable dumping payload in Base64
            # payload-buffer-size: 4kb # max size of payload buffer to output in eve-log
            # payload-printable: yes   # enable dumping payload in printable (lossy) format
            # packet: yes              # enable dumping of packet (without stream segments)
            metadata: yes             # enable inclusion of app layer metadata with alert. Default yes
            http-body: yes           # Requires metadata; enable dumping of HTTP body in Base64
            http-body-printable: yes # Requires metadata; enable dumping of HTTP body in printable format

            # Enable the logging of tagged packets for rules using the
            # "tag" keyword.
            tagged-packets: yes
        - anomaly:
            # Anomaly log records describe unexpected conditions such
            # as truncated packets, packets with invalid IP/UDP/TCP
            # length values, and other events that render the packet
            # invalid for further processing or describe unexpected
            # behavior on an established stream. Networks which
            # experience high occurrences of anomalies may experience
            # packet processing degradation.
            #
            # Anomalies are reported for the following:
            # 1. Decode: Values and conditions that are detected while
            # decoding individual packets. This includes invalid or
            # unexpected values for low-level protocol lengths as well
            # as stream related events (TCP 3-way handshake issues,
            # unexpected sequence number, etc).
            # 2. Stream: This includes stream related events (TCP
            # 3-way handshake issues, unexpected sequence number,
            # etc).
            # 3. Application layer: These denote application layer
            # specific conditions that are unexpected, invalid or are
            # unexpected given the application monitoring state.
            #
            # By default, anomaly logging is enabled. When anomaly
            # logging is enabled, applayer anomaly reporting is
            # also enabled.
            enabled: yes
            #
            # Choose one or more types of anomaly logging and whether to enable
            # logging of the packet header for packet anomalies.
            types:
              # decode: no
              stream: yes
              applayer: yes
            #packethdr: no
        - http:
            extended: yes     # enable this for extended logging information
            # custom allows additional HTTP fields to be included in eve-log.
            # the example below adds three additional fields when uncommented
            custom: [Accept-Encoding, Accept-Language, Authorization]
            # set this value to one and only one from {both, request, response}
            # to dump all HTTP headers for every HTTP request and/or response
            dump-all-headers: both
        - dns:
            # This configuration uses the new DNS logging format,
            # the old configuration is still available:
            # https://suricata.readthedocs.io/en/latest/output/eve/eve-json-output.html#dns-v1-format

            # As of Suricata 5.0, version 2 of the eve dns output
            # format is the default.
            # version: 2

            # Enable/disable this logger. Default: enabled.
            enabled: yes

            # Control logging of requests and responses:
            # - requests: enable logging of DNS queries
            # - responses: enable logging of DNS answers
            # By default both requests and responses are logged.
            #requests: no
            #responses: no

            # Format of answer logging:
            # - detailed: array item per answer
            # - grouped: answers aggregated by type
            # Default: all
            # formats: [detailed, grouped]

            # DNS record types to log, based on the query type.
            # Default: all.
            # types: [a, aaaa, cname, mx, ns, ptr, txt]
        - tls:
            extended: yes     # enable this for extended logging information
            # output TLS transaction where the session is resumed using a
            # session id
            # session-resumption: no
            # custom controls which TLS fields that are included in eve-log
            #custom: [subject, issuer, session_resumed, serial, fingerprint, sni, version, not_before, not_after, certificate, chain, ja3, ja3s]
            custom: [subject, issuer, session_resumed, serial, fingerprint, sni, version, not_before, not_after, chain, ja3s]
        - files:
            force-magic: no   # force logging magic on all logged files
            # force logging of checksums, available hash functions are md5,
            # sha1 and sha256
            # force-hash: [md5]
        #- drop:
        #    alerts: yes      # log alerts that caused drops
        #    flows: all       # start or all: 'start' logs only a single drop
        #                     # per flow direction. All logs each dropped pkt.
        - smtp:
            extended: yes # enable this for extended logging information
            # this includes: bcc, message-id, subject, x_mailer, user-agent
            # custom fields logging from the list:
            #  reply-to, bcc, message-id, subject, x-mailer, user-agent, received,
            #  x-originating-ip, in-reply-to, references, importance, priority,
            #  sensitivity, organization, content-md5, date
            custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc]
            # output md5 of fields: body, subject
            # for the body you need to set app-layer.protocols.smtp.mime.body-md5
            # to yes
            # md5: [body, subject]

        #- dnp3
        - ftp
        - rdp
        - nfs
        - smb
        - tftp
        - ikev2
        - dcerpc
        - krb5
        - snmp
        - rfb
        - sip
        - dhcp:
            enabled: yes
            # When extended mode is on, all DHCP messages are logged
            # with full detail. When extended mode is off (the
            # default), just enough information to map a MAC address
            # to an IP address is logged.
            extended: yes
        - ssh
        - mqtt:
            # passwords: yes           # enable output of passwords
        # HTTP2 logging. HTTP2 support is currently experimental and
        # disabled by default. To enable, uncomment the following line
        # and be sure to enable http2 in the app-layer section.
        # - http2
        - stats:
            totals: yes       # stats for all threads merged together
            threads: no       # per thread stats
            deltas: no        # include delta values
        # bi-directional flows
        - flow
        # uni-directional flows
        #- netflow

        # Metadata event type. Triggered whenever a pktvar is saved
        # and will include the pktvars, flowvars, flowbits and
        # flowints.
        #- metadata

  # a line based log of HTTP requests (no alerts)
  - http-log:
      enabled: yes
      filename: http.log
      append: yes
      extended: yes     # enable this for extended logging information
      custom: yes       # enable the custom logging format (defined by customformat)
      customformat: "%{%D-%H:%M:%S}t.%z %{X-Forwarded-For}i %H %m %h %u %s %B %a:%p -> %A:%P"
      #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'

  # a line based log of TLS handshake parameters (no alerts)
  - tls-log:
      enabled: yes  # Log TLS connections.
      filename: tls.log # File to store TLS logs.
      append: yes
      extended: yes     # Log extended information like fingerprint
      #custom: yes       # enabled the custom logging format (defined by customformat)
      #customformat: "%{%D-%H:%M:%S}t.%z %a:%p -> %A:%P %v %n %d %D"
      #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'
      # output TLS transaction where the session is resumed using a
      # session id
      #session-resumption: no

  # output module to store certificates chain to disk
  - tls-store:
      #enabled: yes
      #certs-log-dir: certs # directory to store the certificates files

asmirnov2 · November 19, 2021, 10:33am

Yes, this is status.log. Moreover, http-data.log and http.log are empty

jufajardini · November 19, 2021, 1:20pm

What Suricata version are you running?

Can you confirm that http is enabled in the app-layer.protocols section in your suricata.yaml file?

asmirnov2 · November 19, 2021, 2:23pm

Suricata version 6.0.3

asmirnov2 · November 19, 2021, 2:23pm

http:
  enabled: yes
  # memcap:                   Maximum memory capacity for HTTP
  #                           Default is unlimited, values can be 64mb, e.g.

  # default-config:           Used when no server-config matches
  #   personality:            List of personalities used by default
  #   request-body-limit:     Limit reassembly of request body for inspection
  #                           by http_client_body & pcre /P option.
  #   response-body-limit:    Limit reassembly of response body for inspection
  #                           by file_data, http_server_body & pcre /Q option.
  #
  #   For advanced options, see the user guide


  # server-config:            List of server configurations to use if address matches
  #   address:                List of IP addresses or networks for this block
  #   personality:            List of personalities used by this block
  #
  #                           Then, all the fields from default-config can be overloaded
  #
  # Currently Available Personalities:
  #   Minimal, Generic, IDS (default), IIS_4_0, IIS_5_0, IIS_5_1, IIS_6_0,
  #   IIS_7_0, IIS_7_5, Apache_2
  libhtp:
     default-config:
       personality: IDS

asmirnov2 · November 19, 2021, 2:27pm

libhtp:
default-config:
personality: IDS

       # Can be specified in kb, mb, gb.  Just a number indicates
       # it's in bytes.
       request-body-limit: 100kb
       response-body-limit: 100kb

       # inspection limits
       request-body-minimal-inspect-size: 32kb
       request-body-inspect-window: 4kb
       response-body-minimal-inspect-size: 40kb
       response-body-inspect-window: 16kb

       # response body decompression (0 disables)
       response-body-decompress-layer-limit: 2

       # auto will use http-body-inline mode in IPS mode, yes or no set it statically
       http-body-inline: auto

       # Decompress SWF files.
       # Two types: 'deflate', 'lzma', 'both' will decompress deflate and lzma
       # compress-depth:
       # Specifies the maximum amount of data to decompress,
       # set 0 for unlimited.
       # decompress-depth:
       # Specifies the maximum amount of decompressed data to obtain,
       # set 0 for unlimited.
       swf-decompression:
         enabled: yes
         type: both
         compress-depth: 100kb
         decompress-depth: 100kb

       # Use a random value for inspection sizes around the specified value.
       # This lowers the risk of some evasion techniques but could lead
       # to detection change between runs. It is set to 'yes' by default.
       #randomize-inspection-sizes: yes
       # If "randomize-inspection-sizes" is active, the value of various
       # inspection size will be chosen from the [1 - range%, 1 + range%]
       # range
       # Default value of "randomize-inspection-range" is 10.
       #randomize-inspection-range: 10

       # decoding
       double-decode-path: no
       double-decode-query: no

asmirnov2 · November 19, 2021, 2:29pm

       # Can enable LZMA decompression
       #lzma-enabled: false
       # Memory limit usage for LZMA decompression dictionary
       # Data is decompressed until dictionary reaches this size
       #lzma-memlimit: 1mb
       # Maximum decompressed size with a compression ratio
       # above 2048 (only LZMA can reach this ratio, deflate cannot)
       #compression-bomb-limit: 1mb
       # Maximum time spent decompressing a single transaction in usec
       #decompression-time-limit: 100000

     server-config:

       #- apache:
       #    address: [192.168.1.0/24, 127.0.0.0/8, "::1"]
       #    personality: Apache_2
       #    # Can be specified in kb, mb, gb.  Just a number indicates
       #    # it's in bytes.
       #    request-body-limit: 4096
       #    response-body-limit: 4096
       #    double-decode-path: no
       #    double-decode-query: no

       #- iis7:
       #    address:
       #      - 192.168.0.0/24
       #      - 192.168.10.0/24
       #    personality: IIS_7_0
       #    # Can be specified in kb, mb, gb.  Just a number indicates
       #    # it's in bytes.
       #    request-body-limit: 4096
       #    response-body-limit: 4096
       #    double-decode-path: no
       #    double-decode-query: no

Andreas_Herz · November 27, 2021, 11:34pm

I see that it matches erspan,gre, vlan so how does your captured traffic look like exactly? Maybe it’s an issue within the traffic or the capsulation. You can also run tcpdump on the interface to create a pcap and look into that if it’s correct http traffic and rerun it against suricata with the -r feature and see if http events are seen.

Topic		Replies	Views
Suricata, HTTP request/response data is not outputting to eve-log, is there any way? Help community , suricata	3	108	August 13, 2024
Suricata no longer write into Eve files Help	2	2096	May 7, 2021
After installation, Suricata writes only statistics to eve Help	5	538	December 2, 2021
No log of http protocol appears in eve.json and fast.log Help	3	820	September 28, 2020
Suricata cannot parse tls/http traffice Help	10	357	September 19, 2024

Suricata the output of eve.log does not include data on http connections

Related topics