[Q&A] How to pass in record value to S3 path or object key

[lisazhao30]

What is a problem?

How can I pass in a value from a record dynamically to the S3 output path or object key?

When I try accessing via ${record["id"], it returns that exact string rather than the actual id of that record. The records coming in can be different ids, but I have set the buffer parameters to include the id.

My log looks like:
logs {"id": "123", "log": "some log message", "time": "..."}

Describe the configuration of Fluentd

Code:

 <match logs>
      @type copy
      <store>
        @type s3
        aws_key_id #
        aws_sec_key #
        s3_bucket #
        s3_region #
        path logs/${record["id"]}
        s3_object_key_format %{path}/%{time_slice}worker="#{Socket.gethostname}"_%{index}.%{file_extension}
        <buffer time, id>
          @type file
          path /tmp/fluentd-buffers/logs
          timekey 10m
          timekey_wait 1m
          chunk_limit_size 256m
        </buffer>
      </store>
      <store>
        @type stdout
      </store>
    </match>

Describe the logs of Fluentd

No error messages encountered.

Environment

- Fluentd version: 1.18.0
- TD Agent version:
- Fluent Package version:
- Docker image (tag):
- Operating system:
- Kernel version: