Skip token type ids in Hugging Face backend

[Marchematics]

Motivation

Some PreTrainedTokenizerFast tokenizers return token_type_ids. Hugging Face generation rejects this key for models that do not accept it, causing evaluation to fail.

Modification

Disable token_type_ids in Hugging Face generation and perplexity tokenization paths, with regression tests for both paths.

Fixes #1533

Checklist

• Bug fix covered by unit tests
• Lint checks passed
• No documentation changes required

[Copilot]

🟡 Changes recommended

The change only covers one generation path, but other generation/tokenization paths in the same module still forward all tokenizer outputs and may continue to pass token_type_ids into model calls.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR updates the HuggingFace v4.33+ chat-template generation path to explicitly skip token_type_ids during tokenization, and adds a unit test assertion to enforce that behavior.

Changes:

• Pass return_token_type_ids=False to tokenizer.batch_encode_plus(...) in HuggingFacewithChatTemplate.generate.
• Extend test_generate_basic to assert return_token_type_ids is disabled in the tokenizer call.

File summaries

File	Description
tests/models/test_huggingface_above_v4_33.py	Adds an assertion that batch_encode_plus is called with return_token_type_ids=False.
opencompass/models/huggingface_above_v4_33.py	Disables token_type_ids in the chat-template generate tokenization kwargs.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Marchematics]

Updated the remaining generation and PPL paths with tests.

[Marchematics]

Updated the base generation and PPL tokenization paths too.