Skip to content

test(vibe-tests): compare two ways to add Astryx to Tailwind apps - #5821

Merged
nynexman4464 merged 1 commit into
nynexman4464/vibe-tests/setup-evaluator-correctnessfrom
nynexman4464/vibe-tests/strategy-pilot
Sep 3, 2026
Merged

test(vibe-tests): compare two ways to add Astryx to Tailwind apps#5821
nynexman4464 merged 1 commit into
nynexman4464/vibe-tests/setup-evaluator-correctnessfrom
nynexman4464/vibe-tests/strategy-pilot

Conversation

@nynexman4464

@nynexman4464 nynexman4464 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Top of a three-part stack. The measurement and evaluator corrections this
pilot depended on now sit below it:
#5855 (hermetic measurement) → #5856 (evaluator correctness) → this.

Problem

The setup evaluation has one existing-app recipe. An application that already
has its own reset, token vocabulary, and visual language can adopt a design
system in more than one way, and those ways disagree with each other:

  • keep the host authoritative over typography and colour, and make the design
    system disappear into it — which means an app-owned theme rather than a stock
    theme stylesheet;
  • keep the host's global ownership untouched and contain the design system to
    one new subtree — which means deliberately not importing the reset.

A single recipe cannot represent both, and combining them hands an implementer
contradictory instructions.

They are not equal options, and the documents no longer read as though they
were. Host-aligned adoption is the normal objective for an established app: the
host stays authoritative and the whole recipe runs on supported public API, at a
maintenance cost — an app-owned theme has to track the host's own tokens as they
change. Guest containment is the exceptional mechanism, for an application whose
global ownership cannot move and where scoping the design system to one nested
subtree is a hard requirement of the work. It has no fully supported path today:
it depends on suppressing a root attribute sync no public API exposes, and it
gives up theming for content rendered outside the contained subtree. It is
therefore as much a probe of those two product gaps as it is a recipe.

Changes

Two opt-in strategies. Each is a self-contained document under
internal/vibe-tests/setup-test/guidance/, explored additively beside the
existing controls. Each states its own cost, and each states its own standing:
the host-aligned document is the normal established-app objective on supported
API, the guest-contained document says plainly that it is the exceptional case
and that its central mechanism is a workaround for a missing capability. A
strategy document is deliberately not combined with the generic existing-app
recipe, because the two strategies disagree with parts of it.

What stays neutral is the measurement, not the product position: the pilot
characterizes both strategies per failure category and ranks neither.

Additive stages. New stages only. Existing stage ids, cell identities, and
expansion counts are unchanged, and tests assert those counts and assert that no
new cell can collide with an existing one by key or id. Iterations use new
condition ids rather than editing an existing one, so an earlier result is never
overwritten by a later run, and each revision records the one it supersedes.

A stage cell allowlist. A stage is otherwise the full cross product of its
dimensions, which cannot describe a stage that reruns a named handful of earlier
cells. An explicit allowlist keeps the expected set equal to the intended set,
so coverage and acceptance stay honest, and it is validated rather than trusted.

A recorded guidance digest. setupEnvironmentHash is the durable link
between a run and the instructions it actually received. It is what makes stale
evidence detectable: cells were failing a rule that was written into the
guidance after those runs happened, and the recorded digests prove it rather
than argue it. An executor cannot be said to have ignored an instruction it was
never handed.

Per-strategy aggregation. The report is per strategy and per failure
category, with no combined score and no ranking — collapsing two strategies that
make different tradeoffs into one number hides which category actually failed. A
run whose executor did not complete is reported as not completed and excluded
from the clean and comparable counts, whatever its measurement says.

One more intended-change allowance. s5 mandates exactly one visible
astryx-dialog-trigger, and this fixture declares a guest subtree for guest
design-system content, so that is where the trigger belongs — and where two
independent reps put it. Both grew guest-boundary from 296px to 409px, moving
its height and the two geometry fields height moves, in both schemes. That is
the insertion's own geometry, not host damage, so the contract now allows those
three fields on that probe: height, geometry.height, geometry.bottom. It
is narrower than the shell's allowance and takes no text exemption at all,
because the trigger is task-owned and the boundary's protected text already
excludes it. The boundary's width, its position, its every computed style, and
every other probe stay exact. Stage ids, cell identities, and expansion counts
are untouched.

No behavior change

  • No default changes. A strategy is reachable only from the new stages that
    name it, and every pre-existing stage keeps its exact expansion count and its
    exact cell ids — which is what the tests assert, rather than the presence of a
    metadata flag. Nothing selects a strategy automatically.
  • No core or package changes. Everything here is under
    internal/vibe-tests/. No published package, component, theme, or public API
    is touched.
  • Greenfield adoption is unaffected. These strategies describe adopting into
    an application that already has its own reset and tokens. The existing setup
    path is unchanged.
  • GitHub runs no agents. This repository prepares fixtures, tasks, and
    measurement; it never launches an executor. Nothing added here runs in CI
    beyond ordinary unit tests, and the browser-backed cases are opt-in behind an
    environment flag.
  • Acceptance is unchanged. Every valid run must still be clean.

Test plan

Repository gates from CONTRIBUTING.md, all green at stack top:

pnpm lint:strict   # 0 errors
pnpm test
pnpm build

The setup evaluation suite, including the evaluator, matrix, integrity,
aggregation, and measurement units:

pnpm -F @astryxdesign/vibe-tests setup:test   # 252 passed | 39 skipped

The browser-backed canonical cases, skipped unless explicitly enabled:

ASTRYX_CANONICAL_SETUP_BROWSER=1 \
  pnpm -F @astryxdesign/vibe-tests setup:test   # 291 passed, 0 skipped

These build real fixture applications, drive them in a browser, and assert
against measured computed styles rather than restating the rules in a copy that
can drift. The new guest-boundary case is proved both ways:
with the allowance the measured run scores clean, and with the allowance removed
the same run reports exactly six regressions — height, geometry.height, and
geometry.bottom, in light and dark, on that probe and nothing else. Mutation
proofs over the accepted run show a colour, a font, the boundary's width, the
boundary moving, a neighbouring probe's height, and host copy the boundary
either gains or loses all still fail. Both canonical strategies are exercised, including both portal
directions. The package-manager cases run a real install so an install-time
failure cannot pass unnoticed.

Guidance is asserted where it is delivered — the AGENTS.md a prepared cell
actually carries — rather than in the file it is authored in. That distinction
is the point: two cells were once scored against a rule present in the file and
absent from every sandbox an executor opened.

Each lower rung was built and tested independently against its own bytes
(pnpm lint:strict, pnpm build, setup:test) before this one.

No changeset: this touches internal/ only and publishes nothing.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Sep 1, 2026
@vercel

vercel Bot commented Sep 1, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
astryx Ready Ready Preview Sep 3, 2026 9:10am UTC

Request Review

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

PR Analysis Report

📚 Storybook Preview

View Storybook for this PR
GitHub Pages may take up to a minute to hydrate after deploy.

🧪 Sandbox Preview

View Sandbox for this PR
GitHub Pages may take up to a minute to hydrate after deploy.

No new or modified components detected.

Bundle Size Summary

No component packages changed.

Accessibility Audit

Status: No accessibility violations detected.


Generated by PR Enrichment workflow | Storybook | Sandbox | View full report

@nynexman4464
nynexman4464 force-pushed the nynexman4464/vibe-tests/strategy-pilot branch from 3cc00fb to f94034f Compare September 1, 2026 22:09
@nynexman4464
nynexman4464 changed the base branch from main to nynexman4464/vibe-tests/setup-evaluator-correctness September 1, 2026 22:11
@nynexman4464 nynexman4464 changed the title test(vibe-tests): compare established Tailwind adoption strategies test(vibe-tests): compare two ways to add Astryx to Tailwind apps Sep 1, 2026
github-actions Bot added a commit that referenced this pull request Sep 1, 2026
@nynexman4464
nynexman4464 force-pushed the nynexman4464/vibe-tests/strategy-pilot branch from f94034f to 8bea950 Compare September 1, 2026 23:33
github-actions Bot added a commit that referenced this pull request Sep 2, 2026
@nynexman4464
nynexman4464 marked this pull request as ready for review September 2, 2026 16:19
The setup evaluation has one existing-app recipe. An application that
already has its own reset, token vocabulary, and visual language can
adopt a design system in more than one way, and those ways disagree:
keep the host authoritative over typography and colour and make the
design system disappear into it, which means an app-owned theme rather
than a stock theme stylesheet; or keep the host's global ownership
untouched and contain the design system to one new subtree, which means
deliberately not importing the reset. A single recipe cannot represent
both, and combining them hands an implementer contradictory
instructions.

Two opt-in strategy conditions, each a self-contained document, explored
additively beside the existing controls. They are not product equals and
the documents no longer read as though they were. Host-aligned adoption
is the normal objective for an established app and runs entirely on
supported public API; its cost is maintenance, since an app-owned theme
has to track the host's tokens as they change. Guest containment is the
exceptional mechanism, for an app whose global ownership cannot move and
where scoping the design system to one nested subtree is a hard
requirement; it has no fully supported path today, because it depends on
suppressing a root attribute sync no public API exposes and it gives up
theming for content rendered outside the contained subtree. It is
therefore as much a probe of those two product gaps as a recipe.

Stages are additive. Existing stage ids, cell identities, and expansion
counts are unchanged, and tests assert those counts and assert that no
new cell can collide with an existing one by key or id. Iterations take
new condition ids rather than editing an existing one, so an earlier
result is never overwritten by a later run, and each revision records
the one it supersedes. What stays neutral is the measurement:
aggregation reports the two strategies separately, per failure category,
with no combined score and no ranking.

The s5 contract gains one more intended-change allowance. s5 mandates
exactly one visible astryx-dialog-trigger, and this fixture declares a
guest subtree for guest design-system content, so that is where the
trigger belongs and where two independent reps put it. Both grew
guest-boundary from 296px to 409px, moving its height and the two
geometry fields height moves, in both schemes. That is the insertion's
own geometry, not host damage, so the contract now allows those three
fields on that probe: height, geometry.height, and geometry.bottom. It
is narrower than the shell's allowance and takes no text exemption at
all, because the trigger is task-owned and the boundary's protected text
already excludes it. The boundary's width, its position, its every
computed style, and every other probe stay exact. A canonical browser
case inserts the trigger into the guest subtree and measures the real
app, and mutation proofs over that measured run show a colour, a font,
the boundary's width, the boundary moving, a neighbouring probe's
height, and host copy the boundary either gains or loses all still fail.

No default changes, no core or package changes, and greenfield adoption
is unaffected. This repository prepares fixtures, tasks, and
measurement; it never launches an executor.
@nynexman4464
nynexman4464 force-pushed the nynexman4464/vibe-tests/strategy-pilot branch from 8bea950 to 22ed1ad Compare September 3, 2026 09:07
github-actions Bot added a commit that referenced this pull request Sep 3, 2026
@nynexman4464
nynexman4464 merged commit 08db6e9 into main Sep 3, 2026
25 checks passed
@github-actions
github-actions Bot deleted the nynexman4464/vibe-tests/strategy-pilot branch September 4, 2026 06:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant