Skip to content

Add opt-in Hugging Face Rust JSON tokenizer integration - #22548

Draft
JacobSzwejbka wants to merge 5 commits into
pytorch:mainfrom
JacobSzwejbka:hf-rust-tokenizer-tok
Draft

Add opt-in Hugging Face Rust JSON tokenizer integration#22548
JacobSzwejbka wants to merge 5 commits into
pytorch:mainfrom
JacobSzwejbka:hf-rust-tokenizer-tok

Make Hugging Face integration JSON-only

1e71054
Select commit
Loading
Failed to load commit list.
Meta CLA / Meta CLA Check succeeded Sep 24, 2026 in 0s

Contributor License Agreement is valid!

View more details on Meta CLA