Skip to content

docs: add an advisory PR-time doc-integrity check to docs-ui-tests - #2991

Draft
marcel-rbro wants to merge 4 commits into
masterfrom
docs/docs-tests-pr-integrity-check
Draft

marcel-rbro wants to merge 4 commits into
masterfrom
docs/docs-tests-pr-integrity-check

Conversation

@marcel-rbro

@marcel-rbro marcel-rbro commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Part of #2671. Addresses the cause behind #2988.

The harness only ran weekly, so #2919 moved a doc on 09-01 and nothing surfaced it until 09-07 — into an unassigned issue that then sat 10 days.

Splits the workflow into two jobs:

Job Runs on Does
doc-integrity (new) PRs touching sources/platform/** or docs-tests/** doc-side checks only — each assertion's source_file/source_quote still resolves
ui-drift (unchanged) weekly + manual full suite against staging, files a drift issue

Informative only: continue-on-error: true, so a failure never blocks the merge. Findings render as a table in the job summary. Promoting it to a gate later means dropping that flag and marking the check required.

No playwright install and no secrets — the doc-side project never opens a browser, so it also works on fork PRs. Verified against the pre-#2988 state with no browser present: all 16 stale references reported in 4.3s.

Note: this PR only touches .github/workflows/, so the job won't self-trigger here. It first runs on the next docs PR after merge.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FSj1mTUbMKgmu2JoGJLSKD

Splits the workflow into two jobs so doc-side drift is caught in review
instead of up to a week later.

doc-integrity runs on pull requests touching sources/platform/** or
docs-tests/**. It runs only the `integrity` project — static checks that
each assertion's source_file and source_quote still resolve. No browser
and no credentials, so it needs no Playwright install and works on fork
PRs, where secrets are unavailable.

The check is informative only: `continue-on-error` keeps a failure from
blocking the merge, and findings are written to the job summary as a
table of assertion IDs and their doc sources, so a reviewer sees them
without opening logs. Promoting it to a gate later means removing
`continue-on-error` and marking it required in branch protection.

ui-drift is unchanged except for an `if` guard keeping it off pull
requests. It still runs weekly and on demand, against staging, and still
files a drift issue on failure.

The motivation is #2951: #2919 moved account/settings.md on 09-01, the
baseline went stale, and nothing surfaced it until the scheduled run on
09-07 — into an unassigned issue that sat another 10 days. Run against
that state locally, this job reports all 16 stale back-references in
4.3 seconds with no browser and no credentials.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FSj1mTUbMKgmu2JoGJLSKD
@marcel-rbro marcel-rbro added documentation Improvements or additions to documentation. t-docs Issues owned by technical writing team. labels Sep 17, 2026
@marcel-rbro marcel-rbro self-assigned this Sep 17, 2026
@apify-service-account apify-service-account linked an issue Sep 17, 2026 that may be closed by this pull request
6 tasks
@apify-service-account apify-service-account added the tested Temporary label used only programatically for some analytics. label Sep 17, 2026
@apify-service-account

apify-service-account commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

✅ Preview for this PR (commit b3ddd08b) is ready at https://pr-2991.preview.docs.apify.com (see action run).

@TC-MO

TC-MO commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

I spy with my little 👀 a red PR check?

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@marcel-rbro
marcel-rbro requested a review from TC-MO October 14, 2026 13:28

@TC-MO TC-MO left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, running the doc-side check on PRs fixes the gap that let #2951 go unnoticed for a week. I ran pnpm test:integrity locally with no browsers installed, and it works as described.

I'd like two changes before merge. Both matter because master is already stale: #2980 broke own-oauth-client-option, and #3016 tracks it. With the current wording, the summary on every docs PR would blame that PR for this failure. CI retries also list each finding twice. The other comments are optional suggestions.

Comment thread .github/workflows/docs-ui-tests.yaml Resolved
Comment thread .github/workflows/docs-ui-tests.yaml Resolved
Comment thread .github/workflows/docs-ui-tests.yaml Resolved
Comment thread .github/workflows/docs-ui-tests.yaml Resolved
Comment thread .github/workflows/docs-ui-tests.yaml Resolved
Comment thread .github/workflows/docs-ui-tests.yaml Resolved
Comment thread .github/workflows/docs-ui-tests.yaml Resolved
marcel-rbro and others added 2 commits October 21, 2026 12:02
- Compare against the base commit and split findings into new vs.
  already unresolved on the base branch, with neutral wording
- Run with --retries=0 so CI's retry doesn't list findings twice
- Emit a ::warning annotation per new finding
- Summarize only on failure; drop the success branch
- Run on changes to this workflow; cancel superseded PR runs
- Shorten the header and job comments

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@marcel-rbro
marcel-rbro requested a review from TC-MO November 2, 2026 14:04
@marcel-rbro
marcel-rbro marked this pull request as draft November 3, 2026 15:26
@marcel-rbro

Copy link
Copy Markdown
Contributor Author

We can wait with this for now. It's blocked by the captcha issue.

@marcel-rbro
marcel-rbro removed the request for review from TC-MO November 10, 2026 09:30

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation. t-docs Issues owned by technical writing team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Docs-as-tests: automated Console UI drift detection

3 participants