test(vibe-tests): compare two ways to add Astryx to Tailwind apps - #5821
Merged
nynexman4464 merged 1 commit intoSep 3, 2026
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
PR Analysis Report📚 Storybook PreviewView Storybook for this PR 🧪 Sandbox PreviewView Sandbox for this PR No new or modified components detected. Bundle Size SummaryNo component packages changed. Accessibility AuditStatus: No accessibility violations detected. Generated by PR Enrichment workflow | Storybook | Sandbox | View full report |
nynexman4464
force-pushed
the
nynexman4464/vibe-tests/strategy-pilot
branch
from
September 1, 2026 22:09
3cc00fb to
f94034f
Compare
nynexman4464
changed the base branch from
main
to
nynexman4464/vibe-tests/setup-evaluator-correctness
September 1, 2026 22:11
nynexman4464
force-pushed
the
nynexman4464/vibe-tests/strategy-pilot
branch
from
September 1, 2026 23:33
f94034f to
8bea950
Compare
nynexman4464
marked this pull request as ready for review
September 2, 2026 16:19
nynexman4464
requested review from
cixzhang and
imdreamrunner
as code owners
September 2, 2026 16:19
The setup evaluation has one existing-app recipe. An application that already has its own reset, token vocabulary, and visual language can adopt a design system in more than one way, and those ways disagree: keep the host authoritative over typography and colour and make the design system disappear into it, which means an app-owned theme rather than a stock theme stylesheet; or keep the host's global ownership untouched and contain the design system to one new subtree, which means deliberately not importing the reset. A single recipe cannot represent both, and combining them hands an implementer contradictory instructions. Two opt-in strategy conditions, each a self-contained document, explored additively beside the existing controls. They are not product equals and the documents no longer read as though they were. Host-aligned adoption is the normal objective for an established app and runs entirely on supported public API; its cost is maintenance, since an app-owned theme has to track the host's tokens as they change. Guest containment is the exceptional mechanism, for an app whose global ownership cannot move and where scoping the design system to one nested subtree is a hard requirement; it has no fully supported path today, because it depends on suppressing a root attribute sync no public API exposes and it gives up theming for content rendered outside the contained subtree. It is therefore as much a probe of those two product gaps as a recipe. Stages are additive. Existing stage ids, cell identities, and expansion counts are unchanged, and tests assert those counts and assert that no new cell can collide with an existing one by key or id. Iterations take new condition ids rather than editing an existing one, so an earlier result is never overwritten by a later run, and each revision records the one it supersedes. What stays neutral is the measurement: aggregation reports the two strategies separately, per failure category, with no combined score and no ranking. The s5 contract gains one more intended-change allowance. s5 mandates exactly one visible astryx-dialog-trigger, and this fixture declares a guest subtree for guest design-system content, so that is where the trigger belongs and where two independent reps put it. Both grew guest-boundary from 296px to 409px, moving its height and the two geometry fields height moves, in both schemes. That is the insertion's own geometry, not host damage, so the contract now allows those three fields on that probe: height, geometry.height, and geometry.bottom. It is narrower than the shell's allowance and takes no text exemption at all, because the trigger is task-owned and the boundary's protected text already excludes it. The boundary's width, its position, its every computed style, and every other probe stay exact. A canonical browser case inserts the trigger into the guest subtree and measures the real app, and mutation proofs over that measured run show a colour, a font, the boundary's width, the boundary moving, a neighbouring probe's height, and host copy the boundary either gains or loses all still fail. No default changes, no core or package changes, and greenfield adoption is unaffected. This repository prepares fixtures, tasks, and measurement; it never launches an executor.
nynexman4464
force-pushed
the
nynexman4464/vibe-tests/strategy-pilot
branch
from
September 3, 2026 09:07
8bea950 to
22ed1ad
Compare
github-actions
Bot
deleted the
nynexman4464/vibe-tests/strategy-pilot
branch
September 4, 2026 06:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The setup evaluation has one existing-app recipe. An application that already
has its own reset, token vocabulary, and visual language can adopt a design
system in more than one way, and those ways disagree with each other:
system disappear into it — which means an app-owned theme rather than a stock
theme stylesheet;
one new subtree — which means deliberately not importing the reset.
A single recipe cannot represent both, and combining them hands an implementer
contradictory instructions.
They are not equal options, and the documents no longer read as though they
were. Host-aligned adoption is the normal objective for an established app: the
host stays authoritative and the whole recipe runs on supported public API, at a
maintenance cost — an app-owned theme has to track the host's own tokens as they
change. Guest containment is the exceptional mechanism, for an application whose
global ownership cannot move and where scoping the design system to one nested
subtree is a hard requirement of the work. It has no fully supported path today:
it depends on suppressing a root attribute sync no public API exposes, and it
gives up theming for content rendered outside the contained subtree. It is
therefore as much a probe of those two product gaps as it is a recipe.
Changes
Two opt-in strategies. Each is a self-contained document under
internal/vibe-tests/setup-test/guidance/, explored additively beside theexisting controls. Each states its own cost, and each states its own standing:
the host-aligned document is the normal established-app objective on supported
API, the guest-contained document says plainly that it is the exceptional case
and that its central mechanism is a workaround for a missing capability. A
strategy document is deliberately not combined with the generic existing-app
recipe, because the two strategies disagree with parts of it.
What stays neutral is the measurement, not the product position: the pilot
characterizes both strategies per failure category and ranks neither.
Additive stages. New stages only. Existing stage ids, cell identities, and
expansion counts are unchanged, and tests assert those counts and assert that no
new cell can collide with an existing one by key or id. Iterations use new
condition ids rather than editing an existing one, so an earlier result is never
overwritten by a later run, and each revision records the one it supersedes.
A stage cell allowlist. A stage is otherwise the full cross product of its
dimensions, which cannot describe a stage that reruns a named handful of earlier
cells. An explicit allowlist keeps the expected set equal to the intended set,
so coverage and acceptance stay honest, and it is validated rather than trusted.
A recorded guidance digest.
setupEnvironmentHashis the durable linkbetween a run and the instructions it actually received. It is what makes stale
evidence detectable: cells were failing a rule that was written into the
guidance after those runs happened, and the recorded digests prove it rather
than argue it. An executor cannot be said to have ignored an instruction it was
never handed.
Per-strategy aggregation. The report is per strategy and per failure
category, with no combined score and no ranking — collapsing two strategies that
make different tradeoffs into one number hides which category actually failed. A
run whose executor did not complete is reported as not completed and excluded
from the clean and comparable counts, whatever its measurement says.
One more intended-change allowance.
s5mandates exactly one visibleastryx-dialog-trigger, and this fixture declares a guest subtree for guestdesign-system content, so that is where the trigger belongs — and where two
independent reps put it. Both grew
guest-boundaryfrom 296px to 409px, movingits height and the two geometry fields height moves, in both schemes. That is
the insertion's own geometry, not host damage, so the contract now allows those
three fields on that probe:
height,geometry.height,geometry.bottom. Itis narrower than the shell's allowance and takes no text exemption at all,
because the trigger is task-owned and the boundary's protected text already
excludes it. The boundary's width, its position, its every computed style, and
every other probe stay exact. Stage ids, cell identities, and expansion counts
are untouched.
No behavior change
name it, and every pre-existing stage keeps its exact expansion count and its
exact cell ids — which is what the tests assert, rather than the presence of a
metadata flag. Nothing selects a strategy automatically.
internal/vibe-tests/. No published package, component, theme, or public APIis touched.
an application that already has its own reset and tokens. The existing setup
path is unchanged.
measurement; it never launches an executor. Nothing added here runs in CI
beyond ordinary unit tests, and the browser-backed cases are opt-in behind an
environment flag.
clean.Test plan
Repository gates from
CONTRIBUTING.md, all green at stack top:The setup evaluation suite, including the evaluator, matrix, integrity,
aggregation, and measurement units:
pnpm -F @astryxdesign/vibe-tests setup:test # 252 passed | 39 skippedThe browser-backed canonical cases, skipped unless explicitly enabled:
ASTRYX_CANONICAL_SETUP_BROWSER=1 \ pnpm -F @astryxdesign/vibe-tests setup:test # 291 passed, 0 skippedThese build real fixture applications, drive them in a browser, and assert
against measured computed styles rather than restating the rules in a copy that
can drift. The new
guest-boundarycase is proved both ways:with the allowance the measured run scores clean, and with the allowance removed
the same run reports exactly six regressions —
height,geometry.height, andgeometry.bottom, in light and dark, on that probe and nothing else. Mutationproofs over the accepted run show a colour, a font, the boundary's width, the
boundary moving, a neighbouring probe's height, and host copy the boundary
either gains or loses all still fail. Both canonical strategies are exercised, including both portal
directions. The package-manager cases run a real install so an install-time
failure cannot pass unnoticed.
Guidance is asserted where it is delivered — the
AGENTS.mda prepared cellactually carries — rather than in the file it is authored in. That distinction
is the point: two cells were once scored against a rule present in the file and
absent from every sandbox an executor opened.
Each lower rung was built and tested independently against its own bytes
(
pnpm lint:strict,pnpm build,setup:test) before this one.No changeset: this touches
internal/only and publishes nothing.