[BUG] Windows 3006.10 minions time out with an error in minion.py and no longer communicate with the master 3006.10 after upgrade from 3005-1.2 with multiprocessing: True in minion config

[geisingerDev]

Description
3006.10 minions time out with an error in minion.py and no longer communicate with the master 3006.10.

Windows Server 2019
Windows Server 2016 Standard
Windows Server 2022

Please be as specific as possible and give set-up details.

• on-prem machine - vmware VM
• cloud machine, AWS

Steps to Reproduce the behavior
upgrade to salt from 3005.1-2 to 3006.10

2025-04-13 13:02:06,101 [salt.utils.process:1004][ERROR   ][2880] An un-handled exception from the multiprocessing process 'ProcessPayload(jid=20250413170104743058)' was caught:
Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\utils\process.py", line 999, in wrapped_run_func
    return run_func()
  File "C:\Program Files\Salt Project\Salt\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\minion.py", line 1927, in _target
    run_func(minion_instance, opts, data)
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\minion.py", line 1921, in run_func
    return Minion._thread_return(minion_instance, opts, data)
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\minion.py", line 2157, in _thread_return
    minion_instance._return_pub(ret)
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\minion.py", line 2385, in _return_pub
    ret_val = self._send_req_sync(load, timeout=timeout)
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\minion.py", line 1650, in _send_req_sync
    raise TimeoutError("Request timed out")
TimeoutError: Request timed out

Expected behavior
No timeouts should occur with default minion config of multiprocessing: True. please refer behavior in 3005.1-2, there never used to be this problem.

Screenshots

Minion Versions Report
Windows server 2019

PS C:\Users\vchoudhury_srv> salt-call --versions
Salt Version:
          Salt: 3006.10

Python Version:
        Python: 3.10.16 (heads/main:c504d17, Mar  6 2025, 02:25:38) [MSC v.1943 64 bit (AMD64)]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.7
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 25.0.2
        relenv: 0.18.1
         smmap: 4.0.0
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist:
        locale: utf-8
       machine: AMD64
       release: 2016Server
        system: Windows
       version: 2016Server 10.0.14393 SP0 Multiprocessor Free

Windows Server 2016

PS C:\Windows\system32> salt-call --versions
Salt Version:
          Salt: 3006.10

Python Version:
        Python: 3.10.16 (heads/main:c504d17, Mar  6 2025, 02:25:38) [MSC v.1943 64 bit (AMD64)]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.7
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 25.0.2
        relenv: 0.18.1
         smmap: 4.0.0
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist:
        locale: utf-8
       machine: AMD64
       release: 2019Server
        system: Windows
       version: 2019Server 10.0.17763 SP0 Multiprocessor Free

master versions

[root@rlx8gdcpsamp1v ~]# salt --versions
Salt Version:
          Salt: 3006.10

Python Version:
        Python: 3.10.16 (main, Mar  6 2025, 02:23:15) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.18.1
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

Salt Extensions:
        SSEAPE: 8.17.0.6

System Versions:
          dist: rhel 8.10 Ootpa
        locale: utf-8
       machine: x86_64
       release: 4.18.0-553.42.1.el8_10.x86_64
        system: Linux
       version: Red Hat Enterprise Linux 8.10 Ootpa

Additional context
Add any other context about the problem here.

[dwoz]

@geisingerDev are you able to provide trace level logs for salt.minion?

in the minion config:

log_granular_levels:
  'salt.minion': 'trace'

[geisingerDev]

all minions currently have multiprocessing: False.
minion config is managed by the master. I will have to manually update multiprocessing: True (and wait for the issue to reappear on some test servers).
ETA 1 week