Python 3.12 tests performance degradation

[hkad98]

Hello community,

I want to ask if someone encountered performance degradation for Python 3.12. Recently, I added support for It, and its performance is worse than that of 311, 310, 39, and 38.

Python version	Test execution time
3.12	4m 59s
3.11	2m 9s
3.10	2m 8s

Link to tests: https://github.com/gooddata/gooddata-python-sdk/actions/runs/9778991463/job/26997125183?pr=710

I use the latest version of pytest. I will keep investigating what is going on, and I will let you know if I find something.

[RonnyPfannschmidt]

this is a most curious an schocking data find

would i be possible/feasible to do profiler reports for those testruns to see whats causing that spike

[squiddy]

I am seeing the same, happy to provide profiler reports later. The increase is exactly on the commit that switched to 3.12.

• Python 3.12.3
• py.test 8.3.4
• pytest-xdist 3.6.1

[giampaolo]

I also see a difference:

~/svn/psutil {master}$ time python3.12 -m pytest --no-header psutil/tests/test_misc.py::TestMisc::test_version
============================= test session starts =============================
collected 1 item                                                                                                                      

psutil/tests/test_misc.py::TestMisc::test_version PASSED

============================== 1 passed in 0.05s ==============================

real    0m0,422s
user    0m0,382s
sys     0m0,040s
~/svn/psutil {master}$ time python3.8 -m pytest --no-header psutil/tests/test_misc.py::TestMisc::test_version
============================= test session starts =============================
collected 1 item                                                                                                                      

psutil/tests/test_misc.py::TestMisc::test_version PASSED

============================== 1 passed in 0.03s ==============================

real    0m0,356s
user    0m0,309s
sys     0m0,048s

Kind of related: I recently wrote a blogpost about pytest startup time, which is basically what is being tested above:
https://gmpy.dev/blog/2025/speedup-pytest-startup

[hhartzer]

I'm seeing this as well. Dramatic increase in test time over Python 3.11.

The "collection" part of pytest takes about 2 seconds with 3.11 for my codebase and about 32 seconds with 3.12. This is for 264 test items. Identical package sets.

The actual run time of the tests is also much higher. pytest reports about 81 seconds on 3.11 and 182 seconds on 3.12.

One aspect of my application is slower as well with 3.12, but we're talking about a 17% increase and not 2x-15x.

[RonnyPfannschmidt]

Please compare times with --assert=plain

[hhartzer]

With --assert=plain:

3.11: About 2 seconds collecting and 81 seconds testing.

3.12: About 5 seconds collecting and 155 seconds testing

So notably faster on 3.12 with --assert=plain, but still much slower especially in test execution. It's interesting how much of an impact it made on test collection.

[RonnyPfannschmidt]

There's various 3.12 versions with issue

Black refuses to run on them

[hhartzer]

Black runs fine for me on 3.11 or 3.12.

[hhartzer]

Is there anything else that I should try to test?

[hkad98]

Long time no see, I was able to fix the issue in our repository. The problem was in coverage. We run tests using tox.

Our setup in tox.ini:

[testenv]
runner = uv-venv-lock-runner
package = wheel
wheel_build_env = .pkg
dependency_groups =
    test
commands =
    pytest -v --cov=src/gooddata_sdk --cov-report=xml tests {posargs}

I tried to remove coverage, and it started to work. Then I chat about it with Cursor, and it proposed to add COVERAGE_CORE=sysmon, which helped.

New setup in tox.ini looks like this:

[testenv]
runner = uv-venv-lock-runner
package = wheel
wheel_build_env = .pkg
dependency_groups =
    test
setenv =
    COVERAGE_CORE=sysmon
commands =
    pytest -v --cov=src/gooddata_sdk --cov-report=xml tests {posargs}

Coverage documentation says:

In Python 3.12 and above, you can try an experimental core based on the new sys.monitoring module by defining a COVERAGE_CORE=sysmon environment variable. This should be faster, though plugins and dynamic contexts are not yet supported with it.