Disc buffering very slow

[tontondematt]

Hello,

I am running fluentbit as my goto log forwarding solution in my eks clusters. I installed it using this https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html and have been using it for quite some time.
Recently we decide to migrate our nodes to much bigger ones, I have nodes running up to 500 pods.

My fluenbit is running in a hybrid mode with the primary buffering in memory and filesystem to deal with back pressure.
I am having performance issues while running in a hybrid mode. My node has to handle an injection speed of about 500 req/s.

Visible on this grafana dashaboard when running in memory buffering only everything is working as expected

And here when file buffering is turned on every thing just slow down by about a staggering 40 times for the same amount of injected data.

I have looked into what's happening on my storage (ebs on aws) i don't see anything alarming.

I stripped down my configuration to some really basic stuff

application-log.conf: |

[INPUT]
    Name                tail
    Tag                 application.*
    Exclude_Path        /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
    Path                /var/log/containers/*.log
    multiline.parser    docker, cri
    DB                  /var/fluent-bit/state/flb_container.db
    Mem_Buf_Limit       512MB
    Skip_Long_Lines     On
    Refresh_Interval    1
    Rotate_Wait         30
    storage.type        filesystem
    Read_from_Head      ${READ_FROM_HEAD}
    Buffer_Chunk_Size   16k
    Buffer_Max_Size     16k
    
[FILTER]
    Name                kubernetes
    Match               application.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_Tag_Prefix     application.var.log.containers.
    Merge_Log           On
    Keep_Log            Off
    K8S-Logging.Parser  On
    K8S-Logging.Exclude Off
    Labels              On
    Annotations         Off
    Use_Kubelet         On
    Kubelet_Port        10250
    Buffer_Size         0

[OUTPUT]
    Name                cloudwatch
    Match               application.*
    region              ${AWS_REGION}
    log_group_name      /aws/containerinsights/${CLUSTER_NAME}/application
    log_stream_prefix   ${HOST_NAME}-
    auto_create_group   true

fluent-bit.conf: |

[SERVICE]
    Flush                     1
    Grace                     30
    Log_Level                 error
    Daemon                    off
    Parsers_File              parsers.conf
    HTTP_Server               ${HTTP_SERVER}
    HTTP_Listen               0.0.0.0
    HTTP_Port                 ${HTTP_PORT}
    storage.path              /var/fluent-bit/state/flb-storage/
    storage.sync              normal
    storage.checksum          on
    storage.backlog.mem_limit 100M

Everytime I activate filestorage on my tail input "storage.type filesystem" everything stalls

Is this a normal behavior ?
Appreciate some pointers and some help !

[patrick-stephens]

Are you using the AWS image? If so I'd switch to the OSS one as there are a lot of performance improvements since the version used by AWS.

[tontondematt]

Hey Pat I will give it a try. I have a question though, will I be able to achieve anything close to only memory buffering at my current rate using a hybrid approach ? I dropped my from 4,5 k record to approx 200 , that seems huge ...
What kind of overhead can I expect ?