Repository navigation

Add experimental Hugging Face .tok tokenizer backend - #218

Draft

JacobSzwejbka wants to merge 4 commits into

meta-pytorch:mainfrom

JacobSzwejbka:hf-rust-tokenizer-tok-followup

Draft

Add experimental Hugging Face .tok tokenizer backend#218
JacobSzwejbka wants to merge 4 commits into
meta-pytorch:mainfrom
JacobSzwejbka:hf-rust-tokenizer-tok-followup

Commits on Sep 24, 2026
- Add optional Hugging Face .tok tokenizer backend
  JacobSzwejbka
  committed
  c250cc9
- Fix Rust tokenizer adapter semantics
  JacobSzwejbka
  committed
  1669237
- Support Hugging Face v1 JSON and experimental tok
  JacobSzwejbka
  committed
  5c68507
- Refresh tokenizer size measurements
  JacobSzwejbka
  committed
  48970bb