Your AI models are failing in production—Here’s how to fix model selection

Enterprises must know if the fashions that energy their purposes and brokers work in real-life situations. This sort of analysis can generally be complicated as a result of it’s onerous to foretell particular situations. A revamped model of the RewardBench benchmark appears to offer organizations a greater thought of a mannequin’s real-life efficiency. 

The Allen Institute of AI (Ai2) launched RewardBench 2, an up to date model of its reward mannequin benchmark, RewardBench, which they declare supplies a extra holistic view of mannequin efficiency and assesses how fashions align with an enterprise’s targets and requirements. 

Ai2 constructed RewardBench with classification duties that measure correlations by inference-time compute and downstream coaching. RewardBench primarily offers with reward fashions (RM), which may act as judges and consider LLM outputs. RMs assign a rating or a “reward” that guides reinforcement studying with human suggestions (RHLF).

Nathan Lambert, a senior analysis scientist at Ai2, instructed VentureBeat that the primary RewardBench labored as supposed when it was launched. Nonetheless, the mannequin atmosphere quickly advanced, and so ought to its benchmarks. 

“As reward fashions turned extra superior and use instances extra nuanced, we shortly acknowledged with the neighborhood that the primary model didn’t absolutely seize the complexity of real-world human preferences,” he stated. 

Lambert added that with RewardBench 2, “we got down to enhance each the breadth and depth of analysis—incorporating extra numerous, difficult prompts and refining the methodology to mirror higher how people really choose AI outputs in follow.” He stated the second model makes use of unseen human prompts, has a tougher scoring setup and new domains. 

Utilizing evaluations for fashions that consider

Whereas reward fashions take a look at how nicely fashions work, it’s additionally necessary that RMs align with firm values; in any other case, the fine-tuning and reinforcement studying course of can reinforce unhealthy habits, corresponding to hallucinations, scale back generalization, and rating dangerous responses too excessive.

RewardBench 2 covers six totally different domains: factuality, exact instruction following, math, security, focus and ties.

“Enterprises ought to use RewardBench 2 in two alternative ways relying on their utility. In the event that they’re performing RLHF themselves, they need to undertake the perfect practices and datasets from main fashions in their very own pipelines as a result of reward fashions want on-policy coaching recipes (i.e. reward fashions that mirror the mannequin they’re attempting to coach with RL). For inference time scaling or knowledge filtering, RewardBench 2 has proven that they’ll choose the perfect mannequin for his or her area and see correlated efficiency,” Lambert stated. 

Lambert famous that benchmarks like RewardBench provide customers a strategy to consider the fashions they’re selecting based mostly on the “dimensions that matter most to them, reasonably than counting on a slim one-size-fits-all rating.” He stated the thought of efficiency, which many analysis strategies declare to evaluate, may be very subjective as a result of a superb response from a mannequin extremely depends upon the context and targets of the consumer. On the identical time, human preferences get very nuanced. 

Ai 2 launched the primary model of RewardBench in March 2024. On the time, the corporate stated it was the primary benchmark and leaderboard for reward fashions. Since then, a number of strategies for benchmarking and bettering RM have emerged. Researchers at Meta’s FAIR got here out with reWordBench. DeepSeek launched a brand new approach known as Self-Principled Critique Tuning for smarter and scalable RM. 

How fashions carried out

Since RewardBench 2 is an up to date model of RewardBench, Ai2 examined each present and newly educated fashions to see in the event that they proceed to rank excessive. These included quite a lot of fashions, corresponding to variations of Gemini, Claude, GPT-4.1, and Llama-3.1, together with datasets and fashions like Qwen, Skywork, and its personal Tulu. 

The corporate discovered that bigger reward fashions carry out greatest on the benchmark as a result of their base fashions are stronger. General, the strongest-performing fashions are variants of Llama-3.1 Instruct. When it comes to focus and security, Skywork knowledge “is especially useful,” and Tulu did nicely on factuality. 

Ai2 stated that whereas they consider RewardBench 2 “is a step ahead in broad, multi-domain accuracy-based analysis” for reward fashions, they cautioned that mannequin analysis must be primarily used as a information to choose fashions that work greatest with an enterprise’s wants.