[BUG] AttributeError: 'AsyncAuth' object has no attribute '_creds'

[afischer-opentext-com]

Description

Not frequently but from time to time we see the following error in our salt minion logs

2025-04-07 05:16:07,893 [salt.crypt       :981 ][DEBUG   ][4916] Decrypting the current master AES key
2025-04-07 05:16:07,893 [salt.crypt       :215 ][DEBUG   ][4916] salt.crypt.get_rsa_key: Loading private key
2025-04-07 05:16:07,893 [salt.crypt       :907 ][DEBUG   ][4916] Loaded minion key: C:\salt\conf\pki\minion\minion.pem
2025-04-07 05:16:07,909 [salt.crypt       :223 ][DEBUG   ][4916] salt.crypt.get_rsa_pub_key: Loading public key
2025-04-07 05:16:07,909 [salt.crypt       :262 ][DEBUG   ][4916] salt.crypt.verify_signature: Loading public key
2025-04-07 05:16:07,909 [salt.crypt       :223 ][DEBUG   ][4916] salt.crypt.get_rsa_pub_key: Loading public key
2025-04-07 05:16:07,909 [salt.crypt       :264 ][DEBUG   ][4916] salt.crypt.verify_signature: Verifying signature
2025-04-07 05:16:07,909 [salt.channel.client:358 ][DEBUG   ][4916] Closing AsyncReqChannel instance
2025-04-07 05:16:07,909 [tornado.application:640 ][ERROR   ][4916] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x000002A6A1C3E7A0>, <salt.ext.tornado.concurrent.Future object at 0x000002A6A1A6BD00>)
Traceback (most recent call last):
  File "C:\salt\Lib\site-packages\salt\ext\tornado\ioloop.py", line 606, in _run_callback
    ret = callback()
  File "C:\salt\Lib\site-packages\salt\ext\tornado\stack_context.py", line 278, in null_wrapper
    return fn(*args, **kwargs)
  File "C:\salt\Lib\site-packages\salt\ext\tornado\ioloop.py", line 628, in _discard_future_result
    future.result()
  File "C:\salt\Lib\site-packages\salt\ext\tornado\concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "C:\salt\Lib\site-packages\salt\ext\tornado\gen.py", line 1070, in run
    yielded = self.gen.send(value)
  File "C:\salt\Lib\site-packages\salt\crypt.py", line 712, in _authenticate
    elif self._creds["aes"] != creds["aes"]:
AttributeError: 'AsyncAuth' object has no attribute '_creds'

When this error appears the impact is that communication with the salt-master is interrupted and we loose any feedback on potentially running jobs as they continue to operate but results are sent to nowhere and the minion may require a manual restart. I tried to investigate if this issue is fixed but the code logic on the master branch looks identical to me, so opening the ticket...

Setup

These are Windows 3006.8 salt-minions connected (or not) with a 3006.9 salt-master.

Minion configuration

auth_tries: 10
autoload_dynamic_modules: false
id: 'minion1'
ipc_mode: tcp
log_level_logfile: debug
master:
- master1
- master2
- master3
master_alive_interval: 30
master_tries: -1
master_type: failover
multiprocessing: false
pillarenv: staging
ping_interval: 60
playground: aws
rejected_retry: true
retry_dns_count: 60
s3.path_style: true
saltenv: staging
use_superseded:
- module.run
winrepo_cache_expire_max: 3600
winrepo_cache_expire_min: 60

Please be as specific as possible and give set-up details.

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior

It is not reliably reproducible.

Expected behavior

No exception shall appear, connection to the salt master shall be stable.

Versions Report

C:\salt>.\salt-call.exe --versions-report
Salt Version:
          Salt: 3006.8

Python Version:
        Python: 3.10.14 (heads/main:c1ec015, Apr  3 2024, 21:36:37) [MSC v.1938 64 bit (AMD64)]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.7
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 25.0.2
        relenv: 0.16.0
         smmap: 4.0.0
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist:
        locale: utf-8
       machine: AMD64
       release: 2022Server
        system: Windows
       version: 2022Server 10.0.20348 SP0 Multiprocessor Free