[BugFix] Normalize reward loss over valid pairs

[fallintoplace]

Summary

• Track how many chosen/rejected pairs actually contribute to GPT2RewardModel.compute_reward_loss.
• Normalize by that valid pair count instead of the original batch size when identical pairs are skipped.
• Keep all-identical batches returning a scalar zero, while preserving an autograd connection to the reward tensors.

Root cause

The identical-sequence fix correctly skipped pairs with no preference signal, but the final loss still divided by the original batch size. Mixed batches with skipped samples therefore silently scaled the loss and gradients down.

Tests

• /Users/hoangvu/.pyenv/versions/3.10.13/bin/python -m pytest test/test_rlhf.py::test_compute_reward_loss_identical_sequences test/test_rlhf.py::test_compute_reward_loss_normalizes_by_non_identical_sequences
• /Users/hoangvu/.pyenv/versions/3.10.13/bin/python -m pytest test/test_rlhf.py (2 passed, 75 skipped; optional LLM dependencies unavailable locally)

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3886

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• [ROCm] MI350 CI jobs will have longer queue times due to CI migration

⏳ No Failures, 23 Pending

As of commit 798243e with merge base 92a3f7f ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.