Red teaming

Red teaming borrows a military and security term for having a team act as an adversary. For AI, red teamers try to get a model to produce harmful content, leak information, break its rules, or take unsafe actions, so the developer can fix problems before or after release.

Red teams can be internal staff, hired outside experts, or government testers. Results are often summarized in system cards, usually without full detail, so others cannot check them.

Red teaming finds some problems. It cannot prove a model is free of them, because testers can only try a limited number of approaches.