How to stop the log entry from being split into multiple fields

[nilushancosta]

I have two Kubernetes clusters where I have Fluent Bit and OpenSearch running. I am using Fluent Bit to collect container logs and publish them to OpenSearch. I have a container which logs the following entry

time = 2024-07-12T11:24:45.502Z level = INFO module = test/success_failure message = "Executed" elapsedTime = "diff value" context = "context value"

There was a difference in how this log entry was returned from OpenSearch. So I added a new [OUTPUT] to fluent bit and had a look at the log from stdout in fluent bit. I noticed a difference as follows.

Cluster 1

[0] kube.var.log.containers.successfailure-ioo-2191798088-86877c5cbc-m2c7q_test_main-1526763574-08ec06be85b774ee1efaedd94daccbf791d1.log: [[1720776684.156919996, {}], {"time"=>"2024-07-12T09:31:24.156919996Z", "stream"=>"stderr", "_p"=>"F", "time"=>"2024-07-12T09:31:24.156Z", "level"=>"INFO", "module"=>"test/success_failure", "message"=>"Executed", "elapsedTime"=>"diff value", "context"=>"context value", "kubernetes"=>{<TRUNCATED>}}]

Cluster 2

[0] kube.var.log.containers.success-failure-deployment-7dc4844cb6-l2kzk_test_success-failure-deployment-83639e969cc614a2b31f5c692631.log: [[1720776582.874229284, {}], {"log"=>"time = 2024-07-12T09:29:42.868Z level = INFO module = test/success_failure message = "Executed" elapsedTime = "diff value" context = "context value"", "stream"=>"stderr", "time"=>"2024-07-12T09:29:42.874229284Z", "kubernetes"=>{<TRUNCATED}}]

As you can see, in Cluster 2, the log entry gets added to a log field. So in OpenSearch, the entry is in a field named log. But the same does not happen in Cluster 1. What configuration would cause this? I want to make sure that any log collected by Fluent Bit gets published to a field named log like in Cluster 2.I am using Helm to deploy Fluent Bit. The configmap that gets created has the following configs

data:
  custom_parsers.conf: |
    [PARSER]
        Name docker_no_time
        Format json
        Time_Keep Off
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off
        Flush 1
        Log_Level info
        Parsers_File /fluent-bit/etc/parsers.conf
        Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On

    [INPUT]
        Name tail
        Inotify_Watcher false
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Refresh_Interval 5
        Tag kube.*
        Mem_Buf_Limit 256MB
        Skip_Long_Lines On

    [FILTER]
        Name kubernetes
        Buffer_Size 32MB
        K8S-Logging.Parser On
        K8S-Logging.Exclude On
        Keep_Log Off
        Match kube.*
        Merge_Log On
        tls.verify Off
        Use_Kubelet true

    [FILTER]
        Name lua
        Match kube.*
        script /fluent-bit/scripts/droplogs.lua
        call drop

    [OUTPUT]
        Name opensearch
        Host opensearch
        HTTP_Passwd ${OPENSEARCH_PASSWORD}
        HTTP_User test
        Logstash_Format On
        Logstash_DateFormat %Y-%m-%d
        Logstash_Prefix container-logs
        Match *
        Port 9200
        Replace_Dots On
        Suppress_Type_Name On
        tls On
        tls.verify Off
        Trace_Error On

droplogs.lua is a custom filter which I use to drop some logs. However it does not modify the log entry like I saw in the output

[emmacz]

pls. share droplogs.lua

[patrick-stephens]

Merge_log attempts to parse the log message so I suspect this is what is doing it.

See #10293 (reply in thread) as well.

[nilushancosta]

Thanks @patrick-stephens . Merge_log was what caused the difference