Skip to content

Parity tests fail under the documented direct pytest invocation when the ambient python3 lacks PyYAML (11 failures) #5359

Open

Description

@mehdikarimi55

Summary

Following AGENTS.md's prescribed invocation — uv sync --extra test then .venv/bin/python -m pytest — 11 parity tests fail on a fresh macOS checkout whose ambient python3 does not have PyYAML installed (e.g. Homebrew Python):

FAILED tests/test_check_prerequisites_python_parity.py::test_all_variants_resolve_requested_template[composed]
FAILED tests/test_check_prerequisites_python_parity.py::test_all_variants_validate_requested_template_in_text_mode[composed]
FAILED tests/test_create_new_feature_python_parity.py::test_all_variants_materialize_composed_spec_template
FAILED tests/test_resolve_template_python_parity.py (7 tests, all [composed]-style)
FAILED tests/test_setup_plan_python_parity.py::test_all_variants_materialize_composed_plan_template
FAILED tests/test_setup_tasks_python_parity.py::test_all_variants_return_composed_tasks_template

Each failure is the bash variant reporting Error: PyYAML is required to resolve preset template composition.

Root cause

run() in tests/parity_helpers.py defaults to clean_env(), which is a copy of os.environ — the script subprocesses resolve python3 -> python -> py -3 through the ambient PATH.

  • In CI, tests run via uv run pytest, and uv run prepends the project venv's bin/ to PATH, so the scripts find the venv interpreter (which has PyYAML) and everything passes.
  • Under the direct .venv/bin/python -m pytest invocation documented in AGENTS.md, PATH is untouched, so the scripts find the ambient python3 (no PyYAML) and fail.

tests/parity_helpers.py already contains make_python3_path_shim() whose docstring warns about exactly this hazard ("silently regains/loses that venv's site-packages (e.g. PyYAML)"), and tests/extensions/test_extension_agent_context.py contains a similar _ensure_test_python_on_path() helper — the composed-template tests just don't get the same treatment.

Suggested fix

Prepend the running interpreter's own bin directory to PATH inside parity_helpers.clean_env() — the same thing uv run does — so both invocations give the script subprocesses the same Python the suite runs on. Tests that install their own shim (the no_yaml_python3_shim cases) prepend afterwards and keep precedence, so the negative tests are unaffected. One tests/test_check_prerequisites_python_parity.py defines a file-local _clean_env() duplicate and needs the same treatment (delegate to the shared helper).

Reproduction

git clone https://github.com/github/spec-kit && cd spec-kit
uv sync --extra test          # or: python -m venv .venv && pip install -e . pytest
# ensure the ambient `python3` has no PyYAML:
python3 -c "import yaml" || echo "ambient python3 has no PyYAML"
.venv/bin/python -m pytest tests/test_resolve_template_python_parity.py -q   # 7 failed
PATH="$PWD/.venv/bin:$PATH" .venv/bin/python -m pytest tests/test_resolve_template_python_parity.py -q   # all pass

Observed on macOS 15 (Apple Silicon), Python 3.12, spec-kit main at 08252af.

AI disclosure

Issue filed after diagnosis performed with AI assistance: OpenCode running GLM 5.3 Flash (Z.ai), agent mode under human supervision by @mehdikarimi55; environment reproduction, root-cause analysis, and verification of the PATH-based fix.

Activity

  1. added
    triage-must-haveVerdict: high-value, important work for Spec Kit — do first
    on Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    triage-must-haveVerdict: high-value, important work for Spec Kit — do first

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions