Skip to content

Implement the Subject Sources foundation - #1265

Draft
JeroenDeDauw wants to merge 8 commits into
masterfrom
subject-sources-foundation
Draft

Implement the Subject Sources foundation#1265
JeroenDeDauw wants to merge 8 commits into
masterfrom
subject-sources-foundation

Conversation

@JeroenDeDauw

@JeroenDeDauw JeroenDeDauw commented Aug 7, 2026

Copy link
Copy Markdown
Member

Fixes #993

The ADR 23 foundation. With
only the local Source registered, every path resolves what it did before: nothing user-visible changes and nothing
stored changes.

What lands

T1 — SubjectId is a (source, localId) pair. A bare s + 14 nanoid characters still means a local Subject and
is still stored as that bare string; anything else is sourceKey:localId. Every place an id arrives as text — a REST
path, a revision slot, a relation target, a Lua call — goes through SubjectIdParser, which is handed the local Source
key and resolves nothing. Local slot JSON, Neo4j id and API output are unchanged, pinned by a byte-identical slot
test.

T2 — the Source contract, its registry, and LocalSource. One interface, one registry keyed by Source key, the
local revision slot refactored into the default registered Source under the MediaWiki Wiki ID. Extensions contribute
Sources through NeoWikiRegistrar::addSource(), alongside the Property Type and Page Property Provider registries.
ADR 23's "Source interface contract" open question is resolved in the ADR.

T3 — a Schema reference carries its Source. SchemaReference pairs a Source with the unchanged SchemaName, in
the two places a reference is persisted: a Subject's schema and a relation property's targetSchema. Resolution
routes through the Source the reference names, which need not be the Subject's.

T4 — a relation target naming an unregistered Source is an error-severity violation, raised on the write and
validate paths and never on read of persisted data. It blocks like every other error (ADR 26): under
$wgNeoWikiEnforceValidation, and only when the edit introduces it, so a Subject already carrying such a target stays
editable. Where one is stored anyway, it degrades on read — the projections drop the edge and the triple, and
resolution yields no Subject.

Design calls

  • Serialized form sourceKey:localId, split at the first colon. Source key [A-Za-z][A-Za-z0-9_-]{0,63}; a
    foreign localId is RFC 3986 pchar, minus percent-encoding and minus & and ', capped at 256 characters. Every
    PHP grammar is \z-anchored, so a trailing newline cannot pass.
  • The local key is the Wiki ID, and is exempt from that grammar. A wiki whose id is not a well-formed Source key
    still resolves its own bare ids; only the explicit <wikiId>:<localId> spelling is unusable there.
  • An explicitly-local reference canonicalizes to bare, for Subject ids and Schema references alike, so one thing
    has one identity — which the id-keyed maps rely on.
  • A foreign Schema reference persists as {source, name}. A bare string is always a local name, so a Schema title
    containing a colon (ISO:9001) stays valid.
  • No client-side source validation (ADR 25). A server violation renders through the field-error path the editor
    already has.
  • Sources register lazily, as factories built on first use: registration runs under an early hook where
    constructing clients would be waste, and a Source nobody references is never built.
  • A Source vouches for what it serves. Everything a Source returns is readable by every reader of this wiki, so
    NeoWiki performs no per-user authorization on sourced Subjects; a Source over restricted data serves only its
    unrestricted part. Read paths decide on isLocal(), never on whether a hosting page was found — local Subjects
    keep the page-based checks. Per-user granularity for sourced data is out of scope until demanded (ADR 27).
  • A local slot holds local Subjects only. The slot now accepts qualified ids and object Schema references, so
    deserialization skips a non-local subject key with a warning and the index, Neo4j and RDF projections agree. Over
    REST and Lua, schema and targetSchema are references — a bare string when local, the reserved {source, name}
    object otherwise; consumers tolerate the object form, source-aware display is
    Frontend source-awareness: display sourced Subjects and Schema references, enter cross-source relation targets #1373.
  • isEditable() and foreign-source isValidLocalId() are contract surface with no consumer until sourced-subject
    rendering lands. They are frozen now because a Source author must answer them from the start.
  • Foreign relation edges and unresolvable Schema references degrade — skipped with a warning — rather than
    project. The projection skip covers every foreign target, resolvable or not; T4 refuses only the unresolvable ones.
  • RelationId shared SubjectId's $-anchor newline flaw, fixed here. Pre-existing, one line.

Considered, omitted

  • A write-capability stub on Source. Write-back brings its own method when it is built.
  • A TypeScript SubjectIdParser. Only canonical ids reach the frontend, so there is nothing to canonicalize there.
  • Registry-aware colon splitting of Schema-name strings. Rejected: registering a Source later would retroactively
    reinterpret a Schema name that already contains a colon.
  • Neo4j stub nodes for foreign relation targets. The stub would carry this wiki's wiki_id, and one without it scopes
    no query correctly.

Follow-ups filed

Reviewing

The three commits stage the foundation — identity, the Source contract, Schema references and the target guard —
with the review findings from #1330 folded in and co-authored. Five
further commits act on three independent blind-spot reviews (authorization model, tolerant read-side boundary,
local-only slots, registry honesty, UI guards and docs). PHP and TypeScript assert the id grammars against one shared fixture, tests/vectors/subject-ids.json.

AI-authored — Claude Code, Opus 5 (max); detailed in-session spec from @JeroenDeDauw; diff not yet human-reviewed; phpcs, phpstan, PHPUnit and the TypeScript gates green locally at each revision, and all CI checks green on the current head.

Production notes

Design and orchestration by Fable 5 (max); implementation and the post-review fix batch by Opus 5 (max) subagents.
Four independent AI review passes — code, security, tests, and a diff scan against a superseded branch — with every
blocking finding fixed before this description was written.

@JeroenDeDauw

Copy link
Copy Markdown
Member Author

I did not fully review this yet

@JeroenDeDauw
JeroenDeDauw force-pushed the subject-sources-foundation branch from 2392e27 to 1e809f2 Compare August 25, 2026 20:55
@alistair3149 alistair3149 self-assigned this Aug 31, 2026
@alistair3149
alistair3149 force-pushed the subject-sources-foundation branch from 1e809f2 to cf92872 Compare August 31, 2026 17:19
@JeroenDeDauw

Copy link
Copy Markdown
Member Author

Note: before merging I'd like to ALSO finish my own review. Prior/additional review by others is useful and welcome

@JeroenDeDauw
JeroenDeDauw force-pushed the subject-sources-foundation branch 2 times, most recently from 4402842 to 087bcdd Compare September 1, 2026 23:01
@JeroenDeDauw

Copy link
Copy Markdown
Member Author

Thanks @alistair3149. I've resumed my review and identified some structural issues (not related to your changes). Will continue on this tomorrow.

@JeroenDeDauw
JeroenDeDauw marked this pull request as draft September 1, 2026 23:42
JeroenDeDauw added a commit that referenced this pull request Sep 7, 2026
For #630

The Relations work map had drifted from master. Both authoring paths for
intermediate structures have shipped (mapping-side node synthesis, the
tree editor with in-place target creation), the relation representation
in RDF is recorded as decided in NativeRdfProjection.md, the cross-Source
guard moved from the closed
#1043 to
#1265, Subjects move
between pages on their own, the relations endpoint is specified and
measured in #1324, and
two links and an anchor were broken. The doc also listed shipped in-flow
creation as forward work.

Rewrite it as a work map for someone starting the implementation: what
exists, the six proposed decisions and which of them ratification gates,
the questions that are still open, and the work ordered by what it waits
on. A fifth shorter, with the PR-by-PR recital, the decided RDF question,
and the design-call reassurances gone.

Considered, omitted: a GitHub Discussion link, since no thread exists and
the epic is the venue; roadmap dates, which live in the ECHOLOT roadmap.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
JeroenDeDauw and others added 7 commits September 8, 2026 01:12
A Subject's id becomes the Source that produced it paired with that Source's
own id for it (ADR 23). A bare `s` + 14 nanoid characters keeps meaning a local
Subject, so nothing stored changes; everything else serializes as
`sourceKey:localId`, split at the first colon.

Canonicalizing an id that names the local wiki explicitly needs to know which
Source key is local, which the value object deliberately does not, so that is
SubjectIdParser's job. Every place an id arrives as text - a REST path, a
revision slot, a relation target, a Lua call - goes through the parser rather
than constructing a SubjectId directly, so an id naming this wiki explicitly
resolves to the same identity as its bare form. A caller-supplied id on
creation must be a local one, since no path creates a Subject in another
Source.

Both suites run the same parse vectors, so PHP and TypeScript are asserted
equal rather than assumed so, and the slot serialization of a local-only page
is pinned byte for byte.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: alistair3149 <alistair31494322@gmail.com>
A Subject now comes from a Source, per the contract ADR 23 freezes: the object
that knows how to fetch it, what its ids look like, which Schemas come with it,
and whether it may be edited. The local revision slot becomes one such Source,
registered under the MediaWiki Wiki ID, which is what a bare Subject id
resolves to.

Behaviour-preserving: with only the local Source registered, resolving through
the registry reaches exactly what the repository did. An id naming a Source this
wiki does not have resolves to no Subject and logs a warning, rather than
breaking the page that names it.

Sources are registered up front but built on first use, because building the
local one reaches the graph projection, which a wiki may not have configured.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…targets

A Schema is now referenced as (Source, name) rather than by name alone (ADR 23),
in the two places a reference is stored: a Subject's `schema` field and a
relation property's `targetSchema`. Schema identity stays the name (ADR 17), so
SchemaName is untouched; the reference wraps it.

A Schema of this wiki keeps being written bare, which is also its page title, so
stored Schemas and Subjects are unchanged. Resolution routes through the Source
the reference names, which need not be the Source of the Subject using it; a
reference this wiki cannot resolve reports schema-not-found, the degraded state
the editor already renders, rather than failing.

A local Schema name may itself contain a colon (`ISO:9001`), so a stored string
is never split: it is always one local name. A Schema from elsewhere is stored
as a `{source, name}` object instead, which no local name can be mistaken for.
The qualified `source:name` spelling survives only as a one-way rendering for
people.

Resolution reaches the projection boundary too: the Neo4j and RDF projectors now
resolve a Subject's Schema through its own Source. A relation to a Subject of
another Source gets no Neo4j edge, since the stub node would carry this wiki's
wiki_id, and its RDF triple is named under that Source's own base URI.

Relation targets get ADR 23's v1 guard: a Relation whose target names a Source
this wiki has not registered is refused on the write and validate paths, since
nothing can resolve it. Cross-Source relations open up once resolution for them
exists.

Refused whatever $wgNeoWikiEnforceValidation is set to, because enforcement is
about how strictly a wiki holds data to its Schemas and this is not a Schema
question. Only a violation the edit introduces blocks it, so a Subject that
already carries such a target stays editable, and no read of persisted data is
ever rejected.

For the same reason the check is not Schema-scoped, unlike every other one here:
it runs over every relation value on the proposed Subject, including a Statement
the Schema does not declare and a Subject whose Schema cannot be loaded at all.
Such a target would be unreadable from the moment it is written, which no later
Schema change fixes.

The server is the only validator (ADR 25); the editor renders the violation
through the field-error path it already has.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: alistair3149 <alistair31494322@gmail.com>
A Source vouches for what it returns: everything it hands back is readable by
every reader of this wiki. NeoWiki performs no per-user authorization on a
sourced Subject, because there is no page here to authorize against. A Source
over data with restrictions of its own must serve only the unrestricted part,
which it decides without knowing who is asking.

The read paths now say so rather than arriving at it. GetSubjectQuery
discriminates on whether the id is local: a local Subject keeps the #1046
behaviour exactly, page gate and revision-read allowance included, while a
sourced one is served without consulting the subject-to-page index at all —
that index holds local ids only (ADR 32), so asking it was always going to
answer nothing.

The page-subjects relation expansion stops dropping sourced targets. It gated
every target on a resolvable, readable page, which no sourced target can have,
so a relation to another Source rendered as an absent target next to the same
relation being served by the single-Subject endpoint.

The accepted limitation is that sourced data has no per-user granularity. ADR 27
would have to lift it before a Source may serve data readable to some users and
not others.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A Schema of another Source is stored as `{source, name}` (ADR 23), and the
server emits references faithfully. Three consumers still assumed the bare
string every reference used to be.

`nw.getSchema` fataled on it. The documented idiom is
`nw.getSchema( subject.schema )`, and a sourced Subject's `schema` field is a
table, which the `?string` parameter turned into a TypeError rather than a Lua
error or a nil. It now takes either form and resolves through the Source the
reference names, so the idiom holds for every Subject; a table that is not a
reference returns nil, like a name nothing offers.

The create dry-run validated against a Schema of this wiki whichever Source the
reference named, so previewing a Subject that follows a sourced Schema answered
schema-not-found. It now resolves the reference the way its sibling
ValidateSubjectUpdateQuery does. The REST endpoint's `schema` body parameter is
still typed as a string, so only the local form reaches it over HTTP.

UpdateStatementAction was the one with teeth: it derived a Statement's property
type from a same-named Schema of this wiki while validating the result against
the foreign one, so setting a value on a sourced Subject was coerced to whatever
type the local Schema gave that property.

The frontend still carries a bare Schema name, so the deserializer collapses a
reference to its name rather than dropping it. Interim, until sourced Subjects
render; the wire type now says what the server actually sends.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A Subject of another Source is fetched from that Source; it is never stored in a
page's content (ADR 23). The slot's consumers disagreed about what to do with an
entry keyed otherwise, which imported or hand-edited content can carry: the
subject-to-page index drops it (ADR 32 keys the index by bare local ids), while
both projectors wrote it — Neo4j giving a foreign id this wiki's `wiki_id`,
against what graph-model.md says that property means.

Deserialization now applies the index's rule: a `subjects` key that is not a
local id is skipped, and a `mainSubject` that is not one is read as absent. Both
are warnings rather than throws, because the content is persisted and a page has
to render regardless — which is also why the qualified-but-local spelling
`thisWikiId:s...` is skipped rather than canonicalized: the index cannot record
it, so admitting it here would put a Subject in the graph that nothing can find
a page for.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three claims about the registry had drifted from the code.

Asking whether an id resolves built the Source it names. canResolve() went
through getSource(), which calls a Closure registration and caches what it
returns, so a validation pass over relation targets connected to every Source
those targets name — to answer a question the registration alone settles. It now
answers from key presence.

The lazy-registration rationale named a dependency that no longer exists: since
ADR 32 the subject-to-page index is the neowiki_subject_page table, not part of
the graph projection, so the local Source builds fine without a graph backend.
The Closure overload still earns its place — registration runs under an early
hook on every request, whether or not anything reads a Subject — so all four
texts say that instead.

localId validation was described as source-delegated in ADR 23 and on
SubjectId::isValid(). Source::isValidLocalId() is the seam for it and no caller
consults it: a foreign localId is checked against the shared serialization
grammar alone. Both texts now say so, and the method stays.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two guards. The relation-property editor showed an empty Schema picker for a
target Schema from another Source, inviting a selection that would replace the
reference with a bare local name; it now shows the reference read-only, so only
what the editor can express is editable there. The subjects manager derived a
copy-IRI under this wiki's base for any id, which for a Subject of another
Source asserts ownership of an entity elsewhere — the refusal the export side
already makes in SubjectIriResolver. Neither Subject can reach those surfaces
today; both are guards for when one can.

The docs catch up with the id and reference forms: the Lua and parser-function
pages describe the qualified id, and the Lua page says a Subject's `schema` may
be a reference table that getSchema takes. The Neo4j uniqueness constraint says
why a bare Subject id is unique on its own rather than assuming it. The id
vectors point at docs/api/, where subject-format.md lives. The set-main-subject
and ordering endpoints stop advertising a qualified id: they address Subjects on
one page, which are local by construction.

Plus nits: the create endpoint's 400 gives the reason rather than blaming the
format; a detached docblock, a duplicate assignment, and a dead initialisation;
the ADR 23 amendment note moves to its Status line; and the batch read path
notes an unregistered Source at debug rather than warning on every page view,
since the write path already refuses such a target as a violation.

Tests cover two deliberate edges that had none: a wiki whose id is not a
well-formed Source key still registers and resolves its own bare ids, and cannot
be named explicitly. The Source contract gains the keying invariant its callers
depend on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement the Subject Sources foundation (ADR 23)

2 participants