Configure logging for standalone Elastic Agents is not working

[whyyouwannaknow]

Hello,

I have installed Elastic Agent version 8.10.4 using "apt" packet manager and I would like to set those settings to configure logging information.

agent.logging.level: info
agent.logging.to_files: true
agent.logging.files:
    # Configure the path where the logs are written. The default is the logs directory
    # under the home path (the binary location).
    path: /var/log/elastic-agent

    # The name of the files where the logs are written to.
    name: elastic-agent

    # Configure log file size limit. If limit is reached, log file will be
    # automatically rotated
    rotateeverybytes: 524288000 # = 500MB

    # Number of rotated log files to keep. Oldest files will be deleted first.
    keepfiles: 7

    # The permissions mask to apply when rotating log files. The default value is 0600.
    # Must be a valid Unix-style file permissions mask expressed in octal notation.
    permissions: 0605

    # Enable log file rotation on time intervals in addition to size-based rotation.
    # Intervals must be at least 1s. Values of 1m, 1h, 24h, 7*24h, 30*24h, and 365*24h
    # are boundary-aligned with minutes, hours, days, weeks, months, and years as
    # reported by the local system clock. All other intervals are calculated from the
    # Unix epoch. Defaults to disabled.
    interval: 24h

    # Rotate existing logs on startup rather than appending to the existing
    # file. Defaults to true.
    rotateonstartup: false

I especially would like to have the settings rotateonstartup: false and the permissions: 0605 to work.

At the moment, the given above settings are not working at all. Everytime I restart the agent using systemd it generates a new log file, the permissions are not applied and all the other parameters are not taken into account.

I do not really understand why, but it looks like it doesn't really work to set those settings into the /etc/elastic-agent/elastic-agent.yml file. Below is the full content of this file if needed :

# ================================ General =====================================
# Beats is configured under Fleet, you can define most settings
# from the Kibana UI. You can update this file to configure the settings that
# are not supported by Fleet.
fleet:
  enabled: true

# agent.download:
#   # source of the artifacts, requires elastic like structure and naming of the binaries
#   # e.g /windows-x86.zip
#   sourceURI: "https://artifacts.elastic.co/downloads/beats/"
#   # path to the directory containing downloaded packages
#   target_directory: "${path.data}/downloads"
#   # timeout for downloading package
#   timeout: 120s
#   # file path to a public key used for verifying downloaded artifacts
#   # if not file is present Elastic Agent will try to load public key from elastic.co website.
#   pgpfile: "${path.data}/elastic.pgp"
#   # install_path describes the location of installed packages/programs. It is also used
#   # for reading program specifications.
#   install_path: "${path.data}/install"

# agent.process:
#   # minimal port number for spawned processes
#   min_port: 10000
#   # maximum port number for spawned processes
#   max_port: 30000
#   # timeout for creating new processes. when process is not successfully created by this timeout
#   # start operation is considered a failure
#   spawn_timeout: 30s

# agent.retry:
#   # enabled determines whether retry is possible. Default is false.
#   enabled: true
#   # retries_count specifies number of retries. Default is 3.
#   # Retry count of 1 means it will be retried one time after one failure.
#   retries_count: 3
#   # delay specifies delay in ms between retries. Default is 30s
#   delay: 30s
#   # max_delay specifies maximum delay in ms between retries. Default is 300s
#   max_delay: 5m
#   # Exponential determines whether delay is treated as exponential.
#   # With 30s delay and 3 retries: 30, 60, 120s
#   # Default is false
#   exponential: false
# BEGIN ANSIBLE MANAGED BLOCK
agent.logging.level: info
agent.logging.to_files: true
agent.logging.files:
    # Configure the path where the logs are written. The default is the logs directory
    # under the home path (the binary location).
    path: /var/log/elastic-agent

    # The name of the files where the logs are written to.
    name: elastic-agent

    # Configure log file size limit. If limit is reached, log file will be
    # automatically rotated
    rotateeverybytes: 524288000 # = 500MB

    # Number of rotated log files to keep. Oldest files will be deleted first.
    keepfiles: 7

    # The permissions mask to apply when rotating log files. The default value is 0600.
    # Must be a valid Unix-style file permissions mask expressed in octal notation.
    permissions: 0605

    # Enable log file rotation on time intervals in addition to size-based rotation.
    # Intervals must be at least 1s. Values of 1m, 1h, 24h, 7*24h, 30*24h, and 365*24h
    # are boundary-aligned with minutes, hours, days, weeks, months, and years as
    # reported by the local system clock. All other intervals are calculated from the
    # Unix epoch. Defaults to disabled.
    interval: 24h

    # Rotate existing logs on startup rather than appending to the existing
    # file. Defaults to true.
    rotateonstartup: false
# END ANSIBLE MANAGED BLOCK

I will also add that those settings were configured after the enrollment of the Elastic Agent as a Fleet Server. Maybe those need to be applied before enrolling the Elastic Agent to a policy for it to work?

Thank you for the help!

EDIT : It looks like that restarting the whole Linux system (the VM), did solve the problem. Something's off with this!

Regards,

[jameswiggins]

I am experiencing the same issue. Adding agent.logging settings to the elastic-agent.yml file as described in Configure logging for standalone Elastic Agents has absolutely no impact on the logging behavior of the standalone elastic agent.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

Thanks for reporting this, confirmed that changing the value of agent.logging.level in the /Library/Elastic/Agent/elastic-agent.yml for an installed agent on my Mac has no effect.

Clearly there is a missing test case for this, we have a test in place for changing it via Fleet but not for a standalone agent like this and the handling must have diverged.

elastic-agent/testing/integration/log_level_test.go

Line 36 in 3f22ccb

func TestSetLogLevelFleetManaged(t *testing.T) {

[yavor-martian]

This one has still not been fixed in latest versions?

[michel-laterman]

I believe currently the agent fails to pass log levels to components, but instead applies a filter when receiving them

[michel-laterman]

I think what happens with our core components; is that the env var (sent through the command line -E flag) overwrites config that is passed to the component.

[michel-laterman]

I am unable to recreate this issue on a Linux VM when using tar installers, or the .deb installer. I am also unable to recreate on Windows with the .zip installer; but it is present on my mac - this looks to be OS specific.

[yavor-martian]

@michel-laterman can you also try running it with the Docker image in EKS - it should be reproducible there.
Especially with the setting agent.logging.to_files: true

[michel-laterman]

I don't have too much experiance with k8s deployments, and I was unable to recreate to bug.
I only tried changing the log.level on a 9.3.0-SNAPSHOT image deployed in k8s.

I referred to https://www.elastic.co/docs/reference/fleet/running-on-kubernetes-managed-by-fleet

I created a daemonset with kubectl create -f elastic-agent-standalone-kubernetes.yaml.
Then edited the ConfigMap in file to include agent.logging.level: debug and applied it with kubectl create -f elastic-agent-standalone-kubernetes.yaml.
Finally I forced the daemonset to restart with kubectl rollout restart ds elastic-agent-standalone -n kube-system.

I was able to verify debug logs appeared (from filebeat).

Do you have a different workflow that I can test this with @yavor-martian?

[yavor-martian]

@michel-laterman the log.level works for me as well but the agent.logging.to_files: true is not, at least the last time in 8.17 when I had events not being ingested in Elastic and wanted them to be written locally to a file.
This behavior was specific only to the standalone agent - https://www.elastic.co/docs/reference/fleet/running-on-kubernetes-standalone

[michel-laterman]

After retrying on Macos today, I have been unable to recreate the problem 🤦; the new log level is used after a process is restarted.

So to recap: I have been unable to recreate the issue by setting agent.logging.level in standalone installs across macos, windows, linux (using tar and apt), and on our docker container on k8s using v9.3.0-SNAPSHOTS.

I'll try to set agent.logging.to_files: true as suggested in standalone macos and k8s.

cc @cmacknz

[michel-laterman]

I can confirm to_files is being replaced on the k8s deployments.

Execing into a container with this setting set to true, we can see it's present in the config file that is mounted:

[root@desktop-control-plane elastic-agent]# cat /etc/elastic-agent/agent.yml
outputs:
  default:
    type: elasticsearch
    hosts:
      - >-
        ${ES_HOST}
    api_key: ${API_KEY}
agent:
  logging:
    level: info
    to_files: true
  monitoring:

But the the agent does not end up running this:

[root@desktop-control-plane elastic-agent]# elastic-agent inspect
agent:
  logging:
    level: info
    to_file: false
  monitoring:

(note that the config to_files is correct, but inspect displays to_file)