[RFC] Reinforcement learning from human feedback

[pppppM]

Discuss the Implementation of RLHF

[xiaohangguo]

Paper Reading List

• Aligning Language Models with Offline Reinforcement Learning from Human
  Feedback
• Training language models to follow instructions with human feedback
• Improving alignment of dialogue agents via targeted human judgements
• Fine-Tuning Language Models from Human Preferences
• DeepSpeed-Chat: Easy, Fast and Affordable RLHF
  Training of ChatGPT-like Models at All Scales

---

选了几篇比较经典RLHF（Reinforcement learning from human feedback）论文

[maxchiron]

大佬慢点，我先入个门😭
https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html

[helonggood]

我认领这篇：Aligning Language Models with Offline Reinforcement Learning from Human Feedback，研究研究，到时候分享一个ppt

[helonggood]

【腾讯文档】RLHF_OFFLINE_PPT
https://docs.qq.com/slide/DZmJLbkp4aUZTWmJT
分享PPT，大家有兴趣可以阅览讨论。
文章研究摘要：
1、文章提出一个来自人类反馈 (RLHF) 框架的离线强化学习，以使用生成的样本对齐 LM，而无需与 RL 环境交互。
2、主要研究了如下三种离线RLHF方法来对齐人类偏好：maximum likelihood estimation (MLE) with filtering, reward-weighted regression (RWR), and Decision Transformer (DT) 。
3、通过采用类似监督学习的损失函数，结果表明这三种方法相较于PPO，优势体现在：机器学习系统 (MLSys)更简单 、更少的（大约 12.3%）计算资源、更稳定的模型训练。
4、同时，DT 对齐优于其他离线 RLHF 方法，也优于 PPO，结论就是DT方法最好。

[Sun-9923]

我认领这篇：DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales

[xiaohangguo]

我认领openAI这篇吧：《Training language models to follow instructions with human feedback》，到时候一起ppt圆桌分享哇。

[maxchiron]

我认领这篇：Fine-Tuning Language Models from Human Preferences

[amulil]

我认领这篇：Improving alignment of dialogue agents via targeted human judgements

[KooSung]

考虑支持视觉的RLHF-V吗？
code: https://github.com/RLHF-V/RLHF-V, https://github.com/thunlp/Muffin