What happend when suricata receive a new tcp flow?

Developers
1

Hi. all.
Due to some tcp packet from layer-4 loaded-balance device, and L4 LB device save the real source ip address in tcp options field, so that the backend server can see the real source.
So I add codes into DecodeTCPOptions function in src/decode-tcp.c. like that:

typedef struct Packet_
{
    Address src;
    Address dst;
    Address toa_src;
    union {
        Port sp;
        struct {
            uint8_t type;
            uint8_t code;
        } icmp_s;
    };
    union {
        Port dp;
        struct {
            uint8_t type;
            uint8_t code;
        } icmp_d;
    };
    Port toa_sp;
/*.........*/
} Packet;
static void DecodeTCPOptions(Packet *p, const uint8_t *pkt, uint16_t pktlen)
{
    uint8_t tcp_opt_cnt = 0;
    TCPOpt tcp_opts[TCP_OPTMAX];

    const TCPHdr *tcph = PacketGetTCP(p);
    uint16_t plen = pktlen;
    while (plen)
    {
        const uint8_t type = *pkt;
        if (type == TCP_OPT_EOL) {
            break;
        } else if (type == TCP_OPT_NOP) {
            pkt++;
            plen--;

        } else {
            if (plen < 2) {
                break;
            }

            const uint8_t olen = *(pkt+1);

            if (unlikely(olen > plen || olen < 2)) {
                ENGINE_SET_INVALID_EVENT(p, TCP_OPT_INVALID_LEN);
                return;
            }

            tcp_opts[tcp_opt_cnt].type = type;
            tcp_opts[tcp_opt_cnt].len  = olen;
            tcp_opts[tcp_opt_cnt].data = (olen > 2) ? (pkt+2) : NULL;

            switch (type) {
                case TCP_OPT_TOA:
                    if (olen != TCP_OPT_TOA_LEN) {
                        ENGINE_SET_EVENT(p,TCP_OPT_INVALID_LEN);
                    } else {
                        uint16_t source_port = (pkt[2] << 8) | pkt[3];
                        p->toa_sp = source_port;
                        p->toa_src.family = AF_INET;
                        memcpy(&p->toa_src.addr_data32[0], pkt+4, sizeof(uint32_t));
                        SCLogNotice("DecodeTCPOptions => pkt[4]: %d.%d.%d.%d", pkt[4],pkt[5],pkt[6],pkt[7]);
                    }
                    break;

And I print toa_srcip in DecodeTCPPacket function, like that:

    if (likely(tcp_opt_len > 0)) {
        DecodeTCPOptions(p, pkt + TCP_HEADER_LEN, tcp_opt_len);
    }

    p->sp = TCP_GET_RAW_SRC_PORT(tcph);
    p->dp = TCP_GET_RAW_DST_PORT(tcph);

    if (PacketIsIPv4(p)) {
        char srcip[46] = {0}, dstip[46] = {0}, toa_srcip[46] = {0};
        PrintInet(AF_INET, (const void *)GET_IPV4_SRC_ADDR_PTR(p), srcip, sizeof(srcip));
        PrintInet(AF_INET, (const void *)GET_IPV4_DST_ADDR_PTR(p), dstip, sizeof(dstip));
        PrintInet(AF_INET, (const void *)GET_IPV4_TOA_SRC_ADDR_PTR(p), toa_srcip, sizeof(toa_srcip));
        SCLogNotice("DecodeTCPPacket => src_ip: %s, src_port: %d, dst_ip: %s, dst_port: %d, toa_src: %s, toa_sp: %d", srcip, p->sp, dstip, p->dp, toa_srcip, p->toa_sp);
    }

I also add print in output-json-alert.c, like that:

static int AlertJson(ThreadVars *tv, JsonAlertLogThread *aft, const Packet *p)
{
    AlertJsonOutputCtx *json_output_ctx = aft->json_output_ctx;

    if (p->alerts.cnt == 0 && !(p->flags & PKT_HAS_TAG))
        return TM_ECODE_OK;

    for (int i = 0; i < p->alerts.cnt; i++) {
        const PacketAlert *pa = &p->alerts.alerts[i];
        if (unlikely(pa->s == NULL)) {
            continue;
        }

        JsonAddrInfo addr = json_addr_info_zero;
        if (PacketIsIPv4(p)) {
            char srcip[46] = {0}, dstip[46] = {0}, toa_srcip[46] = {0};
            uint16_t src_port, dst_port = 0;
            src_port = GET_TCP_SRC_PORT(p);
            dst_port = GET_TCP_DST_PORT(p);
            PrintInet(AF_INET, (const void *)GET_IPV4_SRC_ADDR_PTR(p), srcip, sizeof(srcip));
            PrintInet(AF_INET, (const void *)GET_IPV4_DST_ADDR_PTR(p), dstip, sizeof(dstip));
            PrintInet(AF_INET, (const void *)GET_IPV4_TOA_SRC_ADDR_PTR(p), toa_srcip, sizeof(toa_srcip));
            SCLogNotice("AlertJson => p.src_ip: %s, p.src_port: %d, p.dst_ip: %s, p.dst_port: %d, p.toa_srcip: %s", srcip, src_port, dstip, dst_port, toa_srcip);
        }

I rebuild the code and deploy it on server. run suricata, and tail the suricata.log. As you can see, suricata parse toa_srcip successed in DecodeTCPPacket, but in AlertJson, the toa_srcip is 0.0.0.0

root@suricata01:/home/suricata/logs# grep 10.115.217.171 suricata.log | grep 37745
[38 - W#13-eth1] 2024-09-06 17:33:20 Notice: decode-tcp: DecodeTCPPacket => src_ip: 10.115.217.171, src_port: 37745, dst_ip: 10.38.161.152, dst_port: 80, toa_src: 120.92.33.71, toa_sp: 31069
[38 - W#13-eth1] 2024-09-06 17:33:20 Notice: decode-tcp: DecodeTCPPacket => src_ip: 10.38.161.152, src_port: 80, dst_ip: 10.115.217.171, dst_port: 37745, toa_src: 120.92.33.71, toa_sp: 31069
[38 - W#13-eth1] 2024-09-06 17:33:20 Notice: decode-tcp: DecodeTCPPacket => src_ip: 10.115.217.171, src_port: 37745, dst_ip: 10.38.161.152, dst_port: 80, toa_src: 120.92.33.71, toa_sp: 31069
[38 - W#13-eth1] 2024-09-06 17:33:20 Notice: decode-tcp: DecodeTCPPacket => src_ip: 10.115.217.171, src_port: 37745, dst_ip: 10.38.161.152, dst_port: 80, toa_src: 120.92.33.71, toa_sp: 31069
[38 - W#13-eth1] 2024-09-06 17:33:20 Notice: decode-tcp: DecodeTCPPacket => src_ip: 10.38.161.152, src_port: 80, dst_ip: 10.115.217.171, dst_port: 37745, toa_src: 120.92.33.71, toa_sp: 31069
[38 - W#13-eth1] 2024-09-06 17:33:20 Notice: decode-tcp: DecodeTCPPacket => src_ip: 10.38.161.152, src_port: 80, dst_ip: 10.115.217.171, dst_port: 37745, toa_src: 120.92.33.71, toa_sp: 31069
[38 - W#13-eth1] 2024-09-06 17:33:20 Notice: decode-tcp: DecodeTCPPacket => src_ip: 10.38.161.152, src_port: 80, dst_ip: 10.115.217.171, dst_port: 37745, toa_src: 120.92.33.71, toa_sp: 31069
[38 - W#13-eth1] 2024-09-06 17:33:20 Notice: decode-tcp: DecodeTCPPacket => src_ip: 10.38.161.152, src_port: 80, dst_ip: 10.115.217.171, dst_port: 37745, toa_src: 120.92.33.71, toa_sp: 31069
[38 - W#13-eth1] 2024-09-06 17:43:15 Notice: output-json-alert: AlertJson => p.src_ip: 10.115.217.171, p.src_port: 37745, p.dst_ip: 10.38.161.152, p.dst_port: 80, p.toa_srcip: 0.0.0.0
[38 - W#13-eth1] 2024-09-06 17:43:15 Notice: output-json: JsonAddrInfoInit => src_ip: 10.115.217.171, dst_ip: 10.38.161.152, toa_src_ip: 0.0.0.0, src_port: 37745, dst_port: 80
[38 - W#13-eth1] 2024-09-06 17:43:15 Notice: output-json-flow: CreateEveHeaderFromFlow => src_ip: 10.115.217.171, src_port: 37745, dst_ip: 10.38.161.152, dst_port: 80, toa_src_ip: 0.0.0.0
root@suricata01:/home/suricata/logs#

So my question is that, What is the sequence in which Suricata processes TCP streams? Why is the toa_src value in the Packet structure seen correctly when printed in DecodeTCPPacket, but it changes to 0.0.0.0 in AlertJson? It takes about ten minutes from the DecodeTCPPacket function to the AlertJson function. Is this time too long?

2

Some alerts are generated w/o (real) packets. For example when a flow times out, a fake packet is set up based on the flow tuple and this is used to trigger the detect/output pipeline. So in this case the TOA option will not be available.

I guess there are roughly 2 options:

  1. if every packet holds this info somehow, you could replace the IP address in the packets so that the flow tuple simply has the TAO address
  2. otherwise, store the TAO address in the flow (look at the FlowStorage API and retrieve it from there during output
1 Like
3

Thanks for your reply. I’ll try the second option.

4

Hi, Can you tell me where this part in the code ?

5

Try to have a look at e.g. static void FlowWorkerFlowTimeout(ThreadVars *tv, Packet *p, FlowWorkerThreadData *fw, void *detect_thread) function

1 Like