API-Server occurs memory alarm in K8S Env.

[bh7274]

Description

In the case of the Airflow API server running on Kubernetes, there are situations where memory usage exceeds 75%, 80%, or 85% — triggering alerts — regardless of the configured memory limit.
Would it be possible to provide a configuration option to set the maximum memory usage threshold (as a percentage)?

Use case/motivation

In on-premises environments, resource alerts often indicate serious issues.
Providing such a configuration would help prevent these problems.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

This looks like a deployment issue - not airflow itself, I think you need to look at your k8s configuration and configure it.

Guessing of course. But this is most likely thing deployment manager (you) should be able to configure I guess. converting into discussion, maybe you can provide more information or maybe someone might help with better guidance

[bh7274]

When running the airflow api-server with args, there is no way to set a separate memory limit.
When a large number of tasks occur at once, memory req is 4GB, limit is 6GB.
Even if memory req is set to 6GB, limit is set to 9GB, memory alarms are still detected through Prometheus, Grafana, and OCP Alter Manager. (Alarms are triggered when memory usage exceeds 80%.)
(Test environment: wrcp 24.02, OCP 3.x, 4.x, HW: Dell R740, public: 10GB, Internal: 40GB, SSD, Rook Ceph)
For example, in the case of a Java application, I can limit the maximum usage of heap memory to a percentage to prevent an alarm from occurring, but currently, there is no separate option for the airflow api-server, so it cannot be set.

[potiuk]

AFAIK There is no such limit for Python app, but you might likely configure something in your K8s deployment.

[bh7274]

Yes, gunicorn provides an option to restart the process after a certain number of requests, but I'm not sure if this is the right way to do it. Or, you could calculate the allocated memory and manually call garbage collection. What do you think?

[potiuk]

This issue was about configuring alarms. BTW We are not using gunicorn any more - just starlette and uvicorn I believe.

Forcing garbage collection in Python is not a good idea. But if you think there is a memory leak somewhere and api-server takes more memory for you than expected you can try to track your memory allocation with memray - like one of the users did for scheduler in #56641 and report it.

If you want to add "--limit-max-requests" - that might also be a good idea. PRs are most welcome. Opening an issue describing this new feature is also a good idea, but you will have to wait for someone to volunteer to implement it so contributing PR on your own if you need it is the fastest way (and you can test it locally by patching it in your version first to see if it works - which adds value in such contribution).

[bh7274]

Thank you for answer, I'll wait a little bit and if not commit I'll make PR
(I also saw the same scheduler issue. happy to fix it !!)
PS. Additionally, it was observed that the memory allocated by the api-server was returned to the OS late after use (RSS/WSS Memory).