RLHF

Reinforcement learning from human feedback (RLHF) is a training stage in which people compare or rate a model's responses. Those judgments are used to train a separate "reward model," which then guides further training so the main model produces responses people prefer.

RLHF is a major reason modern chatbots follow instructions, keep a consistent tone, and decline some requests. Variants use feedback written by AI systems or by rules rather than by people.

Because the method rewards responses that raters like, it can also encourage models to sound confident or agreeable rather than accurate, a behavior sometimes called sycophancy.