Skip to content

Conversation

ogrisel
Copy link
Contributor

@ogrisel ogrisel commented Aug 27, 2025

Closes #13678.

This would help projects avoid wasting their precious CI resources when running extensive thread-safety tests (which will become more popular in the new free-threading era).

I also changed the logic to mark the test as XFAIL conditionally on the presence of the "CI" environment variable instead of always skipping (related to #7022). This makes it possible to check that the test pass as expected when running locally.

If you think this is too risky, I can remove the condition and always mark the offending params with the xfail mark instead of skipping.

EDIT: I will do the later right away because I have already observed such a random failure on the CI: https://github.com/pytest-dev/pytest/actions/runs/17269674978/job/49010727146

EDIT 2: that does not work because xfail_strict = true is configured in pyproject.toml. Let me skip instead...

Copy link
Member

@RonnyPfannschmidt RonnyPfannschmidt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At first glance this looks nice

There may be some nitpicks but they relate to preexisting patterns

@ogrisel
Copy link
Contributor Author

ogrisel commented Aug 27, 2025

I think I understand why the CI var was not properly detected: we probably need to add it to the passenv directive of tox. Let me try that quickly.

@ogrisel
Copy link
Contributor Author

ogrisel commented Aug 27, 2025

Ok so many other unrelated test started to fail because they are not written in a way that expect to be ever executed in the presence of the CI environment variable... Let me revert 000876e and 5066d98...

"""
import os, time
def test_long_sleep_and_raise():
time.sleep(1 if "CI" in os.environ else 0.1)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall I remove the if "CI" in os.environ condition that can never hold as long as pytest's CI is configured to not propagate this env var via its tox.ini config?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's actually propagate it snd check why tox doesn't automatically

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI is not in the list of variables passed by default: https://tox.wiki/en/latest/config.html#pass_env

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I tried to add CI to passenv (see 000876e above in this PR) this broke many other unrelated tests:

https://github.com/pytest-dev/pytest/actions/runs/17271882336/job/49018562088

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In that case, those tests are incorrectly sensitive to said env var. I recommend fixing them all in a separate PR (that would also bypass this env var in tox) by ensuring adequate isolation in them (monkeypatch.delenv('CI')). Once that PR is merged, you'd be able to return to this one.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if the is_ci_env could be computed early, before the test runs instead of having this check within. It could be checked before the tests implement shielding through os.environ manipulation. Perhaps, a session fixture in conftest.py would be useful in general. Thoughts?

Copy link
Contributor Author

@ogrisel ogrisel Aug 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this particular case, once the CI variable is properly propagated (after #13684 is merged and this PR synced with main), just increasing the sleep time as originally intended might be enough to make the test stable again and we would not have skip those tests anymore.

Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bot:chronographer:provided (automation) changelog entry is part of PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make it possible to terminate pytest on deadlock after via the faulthandler timeout
3 participants