Skip to content

Add opt-in Hugging Face Rust JSON tokenizer integration - #22548

Draft
JacobSzwejbka wants to merge 5 commits into
pytorch:mainfrom
JacobSzwejbka:hf-rust-tokenizer-tok
Draft

Add opt-in Hugging Face Rust JSON tokenizer integration#22548
JacobSzwejbka wants to merge 5 commits into
pytorch:mainfrom
JacobSzwejbka:hf-rust-tokenizer-tok

Make Hugging Face integration JSON-only

1e71054
Select commit
Loading
Failed to load commit list.
Meta CodeSync / Meta Import Checks succeeded Sep 24, 2026 in 0s

All checks passed, this Pull Request can be imported!

This check validates that the Pull Request can be imported to Meta internal infrastructure

View more details on Meta CodeSync