Skip to content

Fix Pydantic annotation metadata handling across 2.10-2.13#3863

Merged
rudolfix merged 4 commits intodevelfrom
fix/pydantic-annotation
May 8, 2026
Merged

Fix Pydantic annotation metadata handling across 2.10-2.13#3863
rudolfix merged 4 commits intodevelfrom
fix/pydantic-annotation

Conversation

@Travior
Copy link
Copy Markdown
Contributor

@Travior Travior commented Apr 14, 2026

Fixes #3862 #3861

Summary

  • Fix discriminated RootModel handling across Pydantic 2.10 → 2.13
  • Preserve root-level Annotated[...] metadata when dlt rebuilds Pydantic models for schema contracts
  • Add regressions for discriminated-union schema inference and root-metadata semantics

Problem 1 — Pydantic 2.13 moved where the discriminator lives

For tagged unions wrapped in a RootModel, Pydantic 2.13 moved the discriminator off the Annotated metadata of the root field and onto root_field.discriminator directly:

root_field.annotation root_field.discriminator
≤ 2.12 Annotated[Union[...], FieldInfo(discriminator="kind")] None
≥ 2.13 Union[...] (bare) "kind"

dlt assumed the discriminator was always recoverable from root_field.annotation, so on 2.13 discriminated root models were treated like a plain RootModel with a single root column, producing e.g. NOT NULL constraint failed: click_events.root.

This is partially fixed already, but only for the case where 2.13 strips the annotation down to a bare Union. It misses the case where root_field.annotation is still an Annotated[...] on 2.13 because additional metadata (e.g. an AfterValidator) is present — the discriminator lives on root_field.discriminator but the annotation is not bare, so the rescue path was skipped.

Problem 2 — dlt was corrupting Annotated[...] metadata

While rebuilding models for schema contracts, dlt did:

return Annotated[_process_annotation(a_t), tuple(a_m)]

This collapses N metadata entries into one tuple entry, changing Pydantic's semantics. For example:

Annotated[Union[ClickEvent, PurchaseEvent],
          Field(discriminator="kind"),
          AfterValidator(reject_forbidden_clicks)]

was effectively rebuilt as:

Annotated[Union[...], (FieldInfo(...), AfterValidator(...))]

For regular BaseModel this went unnoticed because create_model(__base__=model, ...) re-inherits the original, correctly-spread metadata. For RootModel there is no __base__ rescue (it's built with type(name, (PydanticRootModel[ann],), ...)), so validators and other root-level metadata were silently dropped.

Fix

  • Support both discriminator representations:
    • older Pydantic: inside Annotated[..., FieldInfo(...)]
    • newer Pydantic: on root_field.discriminator, including when root_field.annotation is still Annotated due to other metadata
  • Preserve root-model discriminator semantics when rebuilding contract-adjusted RootModel[...]
  • Rebuild Annotated[...] metadata by preserving entries individually instead of packing them into a tuple

@cloudflare-workers-and-pages
Copy link
Copy Markdown

cloudflare-workers-and-pages Bot commented Apr 14, 2026

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
docs d4bcea4 Commit Preview URL

Branch Preview URL
May 05 2026, 12:20 PM

@Travior Travior force-pushed the fix/pydantic-annotation branch from 4200fb4 to 4cb4e32 Compare April 14, 2026 10:03
@Travior Travior self-assigned this Apr 14, 2026
if is_annotated(t_):
a_t, *a_m = get_args(t_)
return Annotated[_process_annotation(a_t), tuple(a_m)] # type: ignore[return-value]
return Annotated[(_process_annotation(a_t), *a_m)] # type: ignore[return-value]
Copy link
Copy Markdown
Contributor Author

@Travior Travior Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me it's unclear why we previously wrapped other annotation metadata into tuples, losing their semantics:
Annotated[str, FieldInfo(...), AfterValidator(...)]
would be turned into:
Annotated[str, (FieldInfo(...), AfterValidator(...)].

Pydantic then ignores this tuple and we effectively lose the validator (or other annotation metadata)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is certainly a bug. I probably overlooked that in claude-generated code.

@Travior Travior changed the title Fix Pydantic root model metadata handling across 2.10-2.13 Fix Pydantic annotation metadata handling across 2.10-2.13 Apr 14, 2026
@Travior Travior marked this pull request as ready for review April 14, 2026 21:18
@Travior Travior requested a review from rudolfix April 20, 2026 13:25
Copy link
Copy Markdown
Collaborator

@rudolfix rudolfix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! it is very hard to verify Pydantic quirks - I added two more tests to have (hopefully) complete coverage - test_apply_contract_preserves_nested_annotated_metadata_entries checks tuple bug directly

if is_annotated(t_):
a_t, *a_m = get_args(t_)
return Annotated[_process_annotation(a_t), tuple(a_m)] # type: ignore[return-value]
return Annotated[(_process_annotation(a_t), *a_m)] # type: ignore[return-value]
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is certainly a bug. I probably overlooked that in claude-generated code.

@rudolfix rudolfix merged commit 9ee723f into devel May 8, 2026
170 of 179 checks passed
@rudolfix rudolfix deleted the fix/pydantic-annotation branch May 8, 2026 08:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants