DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systems

One-line takeaway

DecisionPivot diagnoses decision models used for routing, guardrails and verification in LLM systems: 48 tasks, 40,033 items, 7 roles, with contrast groups testing correctness under controlled variations. Across 40 decision and 14 general models, two 27B models 2.6 points apart on the reference track score 70.1% vs 44.2% on contrast groups.

DecisionPivot 是给 LLM 系统里的“判定模型”（路由、护栏、输出校验）做诊断的基准：48 个任务、40,033 题、7 种判定角色，核心轨用对照组检验模型在细节变化下是否判对。评测 40 个判定模型和 14 个通用模型发现：两个参考轨只差 2.6 分的 27B 模型，对照组准确率分别是 70.1% 和 44.2%。