Doc tests migration

.claude/skills/doc-tests/SKILL.md

@@ -25,7 +25,7 @@ Use this skill when adding tests to documentation guides in the `agentgateway/we
    > **Critical**: Most Kubernetes topic pages (e.g. `content/docs/kubernetes/latest/resiliency/timeouts/request.md`) are thin wrappers that only contain `{{< reuse "agw-docs/pages/..." >}}`. **Always place doc-test blocks in the reuse file** (`assets/agw-docs/pages/...`), never in the content wrapper. This way both `latest` and `main` versions automatically inherit the tests — you only need to add them once.
 3. **Extractor** resolves `{{< reuse "..." >}}` from `assets/`, so the script is built from the expanded content. Reference the **content file** in `test:` sources; the extractor will follow reuse.
 4. **Block order**: Selected blocks are emitted in document order (by file and `start_line`). Hidden blocks (e.g. "start server in background") must appear *before* any visible block that depends on them (e.g. curl). The extractor sorts selected blocks by `(file_path, start_line)` so hidden blocks are not deferred to the end.
-5. **Byte-identical blocks are silently dropped**: `build_script()` in `scripts/doc_test_extract.py` keeps a `seen` set of block contents and skips any block whose content (after stripping leading and trailing newlines) exactly matches an earlier selected block. Only the **first** copy reaches the generated script — there is no warning. See "Repeated commands across sections" under step 3 for what this breaks and how to avoid it.
+5. **Byte-identical blocks are silently dropped**: `build_script()` in `docs-tests/scripts/doc_test_extract.py` keeps a `seen` set of block contents and skips any block whose content (after stripping leading and trailing newlines) exactly matches an earlier selected block. Only the **first** copy reaches the generated script — there is no warning. See "Repeated commands across sections" under step 3 for what this breaks and how to avoid it.
 
 ---
 
@@ -335,8 +335,7 @@ Before generating, review any `yaml`/`yml` fenced blocks tagged with `paths=` to
 
 ### 9. Generate and verify
 
-- From the repo root directory: `python3 scripts/doc_test_run.py --repo-root . --generate-only`
-- From repo root: `python3 scripts/doc_test_run.py --generate-only`
+- From the repo root directory: `python3 ../docs-tests/scripts/doc_test_run.py --repo-root . --generate-only`
 - Inspect `out/tests/generated/*.sh`: order of steps, no unresolved shortcodes, env vars and backgrounding correct.
 - Run a script manually, e.g. `bash out/tests/generated/<script-name>.sh` (standalone tests do not use a kind cluster; use `--generate-only` and run the script in an env that has the binary/Docker/etc.).
 
@@ -442,5 +441,5 @@ When in doubt, flag the failure to the user rather than silently adjusting the t
 ## Reference
 
 - Full framework: [scripts/TEST_FRAMEWORK.md](../../../scripts/TEST_FRAMEWORK.md)
-- Extractor: `scripts/doc_test_extract.py` (block selection, reuse resolution, block order sort)
-- Runner: `scripts/doc_test_run.py` (discovers `test:` pages, generates scripts, optional kind run)
+- Extractor: `docs-tests/scripts/doc_test_extract.py` (block selection, reuse resolution, block order sort)
+- Runner: `docs-tests/scripts/doc_test_run.py` (discovers `test:` pages, generates scripts, optional kind run)

.cursor/skills/doc-tests/SKILL.md

@@ -92,8 +92,8 @@ Write `file:` values **version-relative** so a page can be copied between `main`
 
 ### 8. Generate and verify
 
-- From **website** directory: `python3 scripts/doc_test_run.py --repo-root . --generate-only`
-- From repo root: `python3 website/scripts/doc_test_run.py --repo-root website --generate-only`
+- From **website** directory: `python3 ../docs-tests/scripts/doc_test_run.py --repo-root . --generate-only`
+- From repo root: `python3 docs-tests/scripts/doc_test_run.py --repo-root website --generate-only`
 - Inspect `out/tests/generated/*.sh`: order of steps, no unresolved shortcodes, env vars and backgrounding correct.
 - Run a script manually, e.g. `bash out/tests/generated/<script-name>.sh` (standalone tests do not use a kind cluster; use `--generate-only` and run the script in an env that has the binary/Docker/etc.).
 
@@ -125,5 +125,5 @@ Write `file:` values **version-relative** so a page can be copied between `main`
 ## Reference
 
 - Full framework: [website/scripts/TEST_FRAMEWORK.md](website/scripts/TEST_FRAMEWORK.md)
-- Extractor: `website/scripts/doc_test_extract.py` (block selection, reuse resolution, block order sort)
-- Runner: `website/scripts/doc_test_run.py` (discovers `test:` pages, generates scripts, optional kind run)
+- Extractor: `docs-tests/scripts/doc_test_extract.py` (block selection, reuse resolution, block order sort)
+- Runner: `docs-tests/scripts/doc_test_run.py` (discovers `test:` pages, generates scripts, optional kind run)

.github/workflows/doc-tests.yaml

@@ -26,11 +26,21 @@ jobs:
     outputs:
       matrix: ${{ steps.list.outputs.matrix }}
       has_tests: ${{ steps.list.outputs.has_tests }}
+      matrix_credentialed: ${{ steps.list.outputs.matrix_credentialed }}
+      has_credentialed_tests: ${{ steps.list.outputs.has_credentialed_tests }}
       dev_version: ${{ steps.dev-version.outputs.dev_version }}
     steps:
       - name: Checkout website repo
         uses: actions/checkout@v6
 
+      # Public repo, no token needed. The doc-test scripts live here now, not in this
+      # repo's own scripts/ dir -- see docs-tests/scripts/doc_test_run.py etc. below.
+      - name: Checkout docs-tests
+        uses: actions/checkout@v6
+        with:
+          repository: solo-io/docs-tests
+          path: docs-tests
+
       - name: Set up Python
         uses: actions/setup-python@v6
         with:
@@ -79,10 +89,15 @@ jobs:
       - name: List test cases
         id: list
         run: |
+          # 'functional' and 'live' run here and block the PR ('functional' = static
+          # validation against a real cluster, 'live' = unauthenticated check against a
+          # real external endpoint). 'schema' (config-vs-schema, no cluster) has its own
+          # job below. 'credentialed' is discovered separately and only run on the daily
+          # schedule, non-blocking.
           if [ -n "$CHANGED_FILES" ]; then
-            TESTS=$(python3 scripts/doc_test_run.py --repo-root . --list-tests --file $CHANGED_FILES)
+            TESTS=$(python3 docs-tests/scripts/doc_test_run.py --repo-root . --list-tests --types functional,live --file $CHANGED_FILES)
           else
-            TESTS=$(python3 scripts/doc_test_run.py --repo-root . --list-tests)
+            TESTS=$(python3 docs-tests/scripts/doc_test_run.py --repo-root . --list-tests --types functional,live)
           fi
           COUNT=$(echo "$TESTS" | jq 'length')
           if [ "$COUNT" -gt 0 ]; then
@@ -92,7 +107,21 @@ jobs:
           fi
           MATRIX=$(echo "$TESTS" | jq -c '(length | [., 40] | min) as $shards | . as $tests | {include: [range($shards) | . as $i | ([$tests | to_entries[] | select(.key % $shards == $i) | .value] as $shard | {shard_index: $i, test_count: ($shard | length), tests: ($shard | tojson), shard_name: ($shard | [.[].test] | join(", ") | if length > 80 then .[:77] + "..." else . end)})]}')
           echo "matrix=${MATRIX}" >> $GITHUB_OUTPUT
-          echo "Discovered $COUNT test case(s)"
+          echo "Discovered $COUNT functional/live test case(s)"
+
+          # 'credentialed' is not filtered by changed files -- it only ever runs on the
+          # schedule trigger (see run-test-credentialed's `if:`), never on a PR, so
+          # there's no "changed" set to narrow it against.
+          TESTS_CREDENTIALED=$(python3 docs-tests/scripts/doc_test_run.py --repo-root . --list-tests --types credentialed)
+          COUNT_CREDENTIALED=$(echo "$TESTS_CREDENTIALED" | jq 'length')
+          if [ "$COUNT_CREDENTIALED" -gt 0 ]; then
+            echo "has_credentialed_tests=true" >> $GITHUB_OUTPUT
+          else
+            echo "has_credentialed_tests=false" >> $GITHUB_OUTPUT
+          fi
+          MATRIX_CREDENTIALED=$(echo "$TESTS_CREDENTIALED" | jq -c '(length | [., 40] | min) as $shards | . as $tests | {include: [range($shards) | . as $i | ([$tests | to_entries[] | select(.key % $shards == $i) | .value] as $shard | {shard_index: $i, test_count: ($shard | length), tests: ($shard | tojson), shard_name: ($shard | [.[].test] | join(", ") | if length > 80 then .[:77] + "..." else . end)})]}')
+          echo "matrix_credentialed=${MATRIX_CREDENTIALED}" >> $GITHUB_OUTPUT
+          echo "Discovered $COUNT_CREDENTIALED credentialed test case(s)"
         env:
           CHANGED_FILES: ${{ steps.changed-files.outputs.files }}
 
@@ -173,6 +202,54 @@ jobs:
           echo "dev_version=$RESOLVED" >> "$GITHUB_OUTPUT"
           echo "Resolved dev build version: $RESOLVED"
 
+  schema-check:
+    name: Schema validation (no cluster)
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout website repo
+        uses: actions/checkout@v6
+
+      - name: Checkout docs-tests
+        uses: actions/checkout@v6
+        with:
+          repository: solo-io/docs-tests
+          path: docs-tests
+
+      - name: Checkout agentgateway (for CRD schemas)
+        uses: actions/checkout@v6
+        with:
+          repository: agentgateway/agentgateway
+          path: agentgateway-product
+
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: '3.x'
+
+      - name: Install Python dependencies
+        run: pip install pyyaml jsonschema
+
+      - name: Run schema checks
+        run: |
+          mkdir -p out/tests/generated
+          python3 docs-tests/scripts/doc_test_schema_check.py \
+            --repo-root . \
+            --crd-dir agentgateway-product/controller/install/helm/agentgateway-crds/templates \
+            --docs-tests-root docs-tests \
+            --report-file out/tests/generated/test-results.yaml \
+            --verbose
+
+      # Named test-result-schema so it's picked up by the same test-result-* pattern the
+      # report job already downloads and merges -- no changes needed there. Writes a
+      # test-results.yaml in the exact shape doc_test_run.py writes (see write_report()).
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: test-result-schema
+          path: out/tests/generated/test-results.yaml
+          retention-days: 1
+
   run-test:
     name: "${{ matrix.shard_name }}"
     needs: discover
@@ -187,6 +264,12 @@ jobs:
       - name: Checkout website repo
         uses: actions/checkout@v6
 
+      - name: Checkout docs-tests
+        uses: actions/checkout@v6
+        with:
+          repository: solo-io/docs-tests
+          path: docs-tests
+
       - name: Set up Python
         uses: actions/setup-python@v6
         with:
@@ -235,15 +318,16 @@ jobs:
           # Used by the install-agentgateway-binary snippet's `gh run download` of the
           # nightly release-binary-linux artifact from agentgateway/agentgateway.
           GH_TOKEN: ${{ github.token }}
+          DOCS_TESTS_ROOT: ${{ github.workspace }}/docs-tests
         run: |
           FAILED=0
           TEST_INDEX=0
           while IFS=$'\t' read -r file test; do
-            python3 scripts/doc_test_run.py --repo-root . --file "$file" --test "$test" \
+            python3 docs-tests/scripts/doc_test_run.py --repo-root . --file "$file" --test "$test" \
               --report-file "out/tests/generated/shard/${TEST_INDEX}/test-results.yaml" || FAILED=1
             TEST_INDEX=$((TEST_INDEX + 1))
           done < <(echo "$SHARD_TESTS" | jq -r '.[] | "\(.file)\t\(.test)"')
-          python3 scripts/merge_test_results.py out/tests/generated/shard/ out/tests/generated/test-results.yaml
+          python3 docs-tests/scripts/merge_test_results.py out/tests/generated/shard/ out/tests/generated/test-results.yaml
           exit $FAILED
 
       - name: Upload test results
@@ -263,15 +347,119 @@ jobs:
           retention-days: 1
           if-no-files-found: ignore
 
+  # 'credentialed' = checks against a real vendor (e.g. a real OpenAI key, a real Entra
+  # tenant). These never block a doc PR: they only run on the daily schedule, and
+  # continue-on-error means a credentialed failure never fails this workflow. Results
+  # are uploaded under a name that does NOT match report's `test-result-*` download
+  # pattern, so they never enter the blocking pass/fail gate.
+  #
+  # No credentialed scenario exists in this repo yet (matrix_credentialed is currently
+  # always empty -- every OSS traffic-management test that reaches 'live' or beyond
+  # needs a vendor this repo doesn't have; see the plan's Phase B/F notes on the Entra
+  # tenant gap), so this job is a structural no-op today: `has_credentialed_tests` stays
+  # 'false' and it skips. It starts doing real work the moment a page declares
+  # `type: credentialed` in its front matter. A dedicated summary/Slack step for this
+  # type is deliberately not built yet -- there's nothing to exercise it against; add
+  # one alongside the first real credentialed test.
+  run-test-credentialed:
+    name: "[credentialed, non-blocking] ${{ matrix.shard_name }}"
+    needs: discover
+    if: github.event_name == 'schedule' && needs.discover.outputs.has_credentialed_tests == 'true'
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    continue-on-error: true
+
+    strategy:
+      matrix: ${{ fromJson(needs.discover.outputs.matrix_credentialed) }}
+      fail-fast: false
+    steps:
+      - name: Checkout website repo
+        uses: actions/checkout@v6
+
+      - name: Checkout docs-tests
+        uses: actions/checkout@v6
+        with:
+          repository: solo-io/docs-tests
+          path: docs-tests
+
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: '3.x'
+
+      - name: Set up Go
+        uses: actions/setup-go@v6
+        with:
+          go-version: 'stable'
+          cache: false
+
+      - name: Set up Node.js
+        uses: actions/setup-node@v6
+        with:
+          node-version: 'lts/*'
+
+      - name: Install Python dependencies
+        run: pip install pyyaml
+
+      - name: Install cloud-provider-kind
+        run: go install sigs.k8s.io/cloud-provider-kind@latest
+
+      - name: Install yamltest
+        run: npm install -g yamltest@latest
+
+      - name: Run doc tests
+        env:
+          SHARD_TESTS: ${{ matrix.tests }}
+          DEBUG_MODE: true
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+          PYTHONUNBUFFERED: '1'
+          GH_TOKEN: ${{ github.token }}
+          DOCS_TESTS_ROOT: ${{ github.workspace }}/docs-tests
+        run: |
+          FAILED=0
+          TEST_INDEX=0
+          while IFS=$'\t' read -r file test; do
+            python3 docs-tests/scripts/doc_test_run.py --repo-root . --file "$file" --test "$test" \
+              --report-file "out/tests/generated/shard/${TEST_INDEX}/test-results.yaml" || FAILED=1
+            TEST_INDEX=$((TEST_INDEX + 1))
+          done < <(echo "$SHARD_TESTS" | jq -r '.[] | "\(.file)\t\(.test)"')
+          python3 docs-tests/scripts/merge_test_results.py out/tests/generated/shard/ out/tests/generated/test-results.yaml
+          exit $FAILED
+
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: credentialed-test-result-${{ strategy.job-index }}
+          path: out/tests/generated/test-results.yaml
+          retention-days: 1
+
+      - name: Upload test context (on failure)
+        if: failure()
+        uses: actions/upload-artifact@v7
+        with:
+          name: credentialed-test-context-${{ strategy.job-index }}
+          path: out/tests/generated/context/
+          retention-days: 1
+          if-no-files-found: ignore
+
   report:
     name: Aggregate results and report
-    needs: [discover, run-test]
+    needs: [discover, run-test, schema-check]
     if: always() && needs.discover.outputs.has_tests == 'true'
     runs-on: ubuntu-latest
     steps:
       - name: Checkout website repo
         uses: actions/checkout@v6
 
+      # Public repo, no token needed. The doc-test scripts live here now, not in this
+      # repo's own scripts/ dir -- see docs-tests/scripts/merge_test_results.py etc. below.
+      - name: Checkout docs-tests
+        uses: actions/checkout@v6
+        with:
+          repository: solo-io/docs-tests
+          path: docs-tests
+
       - name: Set up Python
         uses: actions/setup-python@v6
         with:
@@ -287,19 +475,19 @@ jobs:
           path: collected-results/
 
       - name: Merge test results
-        run: python3 scripts/merge_test_results.py collected-results/ out/tests/generated/test-results.yaml
+        run: python3 docs-tests/scripts/merge_test_results.py collected-results/ out/tests/generated/test-results.yaml
 
       - name: Generate job summary
         id: summary
         run: |
           RESULTS_FILE=out/tests/generated/test-results.yaml
 
           # GitHub Step Summary (Markdown)
-          python3 scripts/report_summary.py "$RESULTS_FILE" >> "$GITHUB_STEP_SUMMARY"
+          python3 docs-tests/scripts/report_summary.py "$RESULTS_FILE" >> "$GITHUB_STEP_SUMMARY"
 
           # Slack Block Kit payload
           RUN_URL="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
-          SLACK_PAYLOAD=$(python3 scripts/report_summary.py --slack --run-url "$RUN_URL" "$RESULTS_FILE")
+          SLACK_PAYLOAD=$(python3 docs-tests/scripts/report_summary.py --slack --run-url "$RUN_URL" "$RESULTS_FILE")
           echo "slack_blocks<<EOF" >> "$GITHUB_OUTPUT"
           echo "$SLACK_PAYLOAD" | jq '.main' >> "$GITHUB_OUTPUT"
           echo "EOF" >> "$GITHUB_OUTPUT"
@@ -320,7 +508,7 @@ jobs:
       - name: List untested docs
         if: always()
         run: |
-          python3 scripts/list_untested_docs.py \
+          python3 docs-tests/scripts/list_untested_docs.py \
             --docs-dir content/docs \
             --output out/tests/generated/untested-docs.txt
 

.github/workflows/playwright-screenshots-kube.yml

@@ -26,9 +26,17 @@ jobs:
       - name: Checkout website repo
         uses: actions/checkout@v6
 
-      # Cluster bring-up reuses the doc-tests machinery (scripts/doc_test_run.py): Python +
-      # pyyaml, Go for cloud-provider-kind, and yamltest. kind/kubectl/helm are preinstalled on
-      # the runner (same assumption as .github/workflows/doc-tests.yaml).
+      # Public repo, no token needed. Cluster bring-up reuses the doc-tests machinery
+      # (docs-tests/scripts/doc_test_run.py), which lives here now, not in this repo's own
+      # scripts/ dir. Python + pyyaml, Go for cloud-provider-kind, and yamltest.
+      # kind/kubectl/helm are preinstalled on the runner (same assumption as
+      # .github/workflows/doc-tests.yaml).
+      - name: Checkout docs-tests
+        uses: actions/checkout@v6
+        with:
+          repository: solo-io/docs-tests
+          path: docs-tests
+
       - name: Set up Python
         uses: actions/setup-python@v6
         with:
@@ -91,7 +99,7 @@ jobs:
             page="content/docs/kubernetes/$v/observability/ui.md"
             echo "::group::kube capture — $v"
             group_open=1
-            python3 scripts/doc_test_run.py --repo-root . \
+            python3 docs-tests/scripts/doc_test_run.py --repo-root . \
               --file "$page" --test capture \
               --keep-cluster --keep-cluster-file "out/tests/kept-$v.txt"