Skip to content

[BUG] New ReqServer memory leak in 3006.10 #67931

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
OrangeDog opened this issue Apr 1, 2025 · 0 comments
Open

[BUG] New ReqServer memory leak in 3006.10 #67931

OrangeDog opened this issue Apr 1, 2025 · 0 comments
Assignees
@dwoz
Labels
Bug broken, incorrect, or confusing behavior memory-leak
Milestone
Sulfur v3006.11

Comments

@OrangeDog
Copy link
Contributor

Description
I recently upgraded from 3006.9 to 3006.10, and a new memory leak appears to have been introduced.
I now have to restart salt-master basically every day to avoid OOM issues, and will probably have to revert to .9

Setup
Master is on KVM. I have only twelve minions, on a mix of machine, but all the same version as the master.
They run state.apply every fifteen minutes, and a couple run some other scheduled script that doesn't interact with the master.
Both gitfs and git_pillar are in use, as well as x509 (v1) states and mine requests.

Steps to Reproduce the behavior
Salt process shortly after a restart:

    PID USER       PRI  NI  VIRT   RES   SHR S  CPU% MEM%▽  TIME+  Command
    977 root        20   0  693M 87812  8960 S   0.0  1.5  1h23:53 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager
4046378 root        20   0  394M 71880 18692 S   0.0  1.2  0:00.91 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master FileServerUpdate
4046359 root        20   0  216M 65960 23808 S   0.0  1.1  0:01.21 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master MainProcess
4046376 root        20   0  223M 62496 13700 S   0.0  1.1  0:00.50 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master Maintenance
4046392 root        20   0  306M 62256 11140 S   0.0  1.1  0:01.02 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-4
4046382 root        20   0  306M 60976 11012 S   0.0  1.0  0:00.99 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-1
4046384 root        20   0  304M 57008  7172 S   0.0  1.0  0:01.01 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-2
4046380 root        20   0  303M 56240  6404 S   0.0  1.0  0:00.98 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-0
4046385 root        20   0  303M 56240  6404 S   0.0  1.0  0:01.04 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-3
4046379 root        20   0  584M 48944  6020 S   0.0  0.8  0:00.01 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorkerQueue
4046372 root        20   0  296M 48624  6660 S   0.0  0.8  0:00.05 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master PubServerChannel._publish_daemon
4046377 root        20   0  216M 47520  5252 S   0.0  0.8  0:00.03 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer ReqServer_ProcessManager
4046373 root        20   0  217M 47360  5508 S   0.0  0.8  0:00.01 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master EventPublisher
    947 root        20   0  168M 24728   256 S   0.0  0.4  0:00.54 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion

Salt processes thirty minutes later:

    PID USER       PRI  NI  VIRT   RES   SHR S  CPU% MEM%▽  TIME+  Command
4046392 root        20   0  616M  126M 31236 S   0.0  2.2  0:44.65 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-4
4046380 root        20   0  617M  126M 31236 S   0.0  2.2  0:45.11 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-0
4046382 root        20   0  615M  124M 31108 S   0.0  2.2  0:50.88 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-1
4046385 root        20   0  615M  123M 31108 S   0.0  2.1  0:38.10 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-3
4046384 root        20   0  614M  122M 31108 S   0.0  2.1  0:37.49 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-2
    977 root        20   0  694M 88836  9984 S   0.0  1.5  1h23:57 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager
4046378 root        20   0  398M 75720 18180 S   0.0  1.3  0:07.85 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master FileServerUpdate
4046376 root        20   0  231M 70304 14340 S   0.0  1.2  0:11.13 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master Maintenance
4046359 root        20   0  216M 65704 23552 S   0.0  1.1  0:01.24 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master MainProcess
4046372 root        20   0  297M 54256 10884 S   0.0  0.9  0:00.30 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master PubServerChannel._publish_daemon
4046373 root        20   0  254M 52932  5508 S   0.0  0.9  0:00.68 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master EventPublisher
4046379 root        20   0  584M 50096  6020 S   0.0  0.9  0:08.88 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorkerQueue
4046377 root        20   0  216M 47520  5252 S   0.0  0.8  0:00.07 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer ReqServer_ProcessManager
    947 root        20   0  168M 24728   256 S   0.0  0.4  0:00.54 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion

24 hours later:

    PID USER       PRI  NI  VIRT   RES   SHR S  CPU% MEM%▽  TIME+  Command
4046392 root        20   0  690M  174M  4868 S   0.0  3.0 30:42.23 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-4
4046384 root        20   0  755M  173M  4996 S   0.0  3.0 29:52.55 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-2
4046380 root        20   0  753M  173M  4996 S   0.0  3.0 30:54.94 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-0
4046382 root        20   0  688M  172M  4996 S   0.0  3.0 30:44.35 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-1
4046385 root        20   0  689M  172M  4868 S   0.0  3.0 30:51.85 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-3
    977 root        20   0  692M 83744  3840 S   0.0  1.4  1h27:26 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager
4046378 root        20   0  402M 63048  1284 S   0.0  1.1  0:29.32 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master FileServerUpdate
 503265 root        20   0  230M 58304  3456 S   0.0  1.0  0:28.76 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master Maintenance
4046373 root        20   0  254M 56928  2564 S   0.0  1.0  0:26.97 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master EventPublisher
4046372 root        20   0  299M 46704  2948 S   0.0  0.8  0:09.60 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master PubServerChannel._publish_daemon
4046379 root        20   0  584M 45744  1156 S   0.0  0.8  6:18.52 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorkerQueue
4046377 root        20   0  216M 43552  1284 S   0.0  0.7  0:03.25 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer ReqServer_ProcessManager
4046359 root        20   0  216M 43432  1280 S   0.0  0.7  0:04.29 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master MainProcess
    947 root        20   0  168M 24728   256 S   0.0  0.4  0:00.54 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion

Expected behavior
Memory usage should not keep growing.

Versions Report

salt --versions-report 
Salt Version:
          Salt: 3006.10

Python Version:
        Python: 3.10.16 (main, Mar  6 2025, 02:23:15) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.17.1
      cherrypy: unknown
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: 1.9.0
  looseversion: 1.0.2
      M2Crypto: 0.38.0
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: 1.17.0
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.18.1
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 24.04.2 noble
        locale: utf-8
       machine: x86_64
       release: 6.8.0-56-generic
        system: Linux
       version: Ubuntu 24.04.2 noble

Additional information
Version report from before the upgrade is here: #67207
@OrangeDog OrangeDog added Bug broken, incorrect, or confusing behavior needs-triage labels Apr 1, 2025
@twangboy twangboy added this to the Sulfur v3006.11 milestone Apr 1, 2025
@dwoz dwoz self-assigned this Apr 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dwoz dwoz

Labels
Bug broken, incorrect, or confusing behavior memory-leak
Projects
None yet
Milestone
Sulfur v3006.11
Development

No branches or pull requests
3 participants
@OrangeDog @dwoz @twangboy