TOPIC · BENCHMARK · 62 PAPERS

Benchmarks & data

LLMs Reasoning Alignment Agents Multimodal Visual generation Vision & 3D Robotics & embodied AI Efficient training & inference Safety & interpretability AI for Science Code & SWE Benchmarks & data Reinforcement learning

Last 30 days 60

#02 ▲220 ★1.6k ❝22 2026-09-21

RRSI: Regularized Recursive Self-Improvement of Agent Harnesses

Adds regularisation to recursive self-improvement of agent harnesses so gains survive out-of-distribution tests.

Google · Peng Xia, Rujun Han, Zifeng Wang +11

Agents Benchmarks & data arXiv Code HF

#11 ▲459 ★218 ❝0 2026-10-15

Has GPT-6-Astra Achieved Embodied Intelligence?

Evaluates GPT-6-Astra on 12 robotics benchmarks and discusses what a general model's embodied competence means for the field.

SenseNova · SenseNova, Joint Research Team, Zhongang Cai +59

Robotics & embodied AI Benchmarks & data arXiv Code HF

#23 ▲272 ★107 ❝0 2026-10-15

FDE-Gym: The Last Mile of AI Delivery

FDE-Gym offers 95 real delivery tasks testing whether agents can clarify vague requirements and ship working solutions like forward-deployed engineers.

DeepLethe · Dongxu Yang, Hongda Yan, Renxin Wang +23

Agents Benchmarks & data arXiv Code HF

#25 ▲490 ★30 ❝0 2026-10-14

BizAgentBench: Unveiling Hidden Challenges in Deploying Long-Horizon Agents for Real-World Business

BizBench, built on real payment procedures, checks whether long-horizon agents violate policies and constraints mid-execution.

ant-international · Enhan Zhao, Yongqi Tong, Xuejun Yu +6

Agents Benchmarks & data arXiv Code HF

#26 ▲727 ★5 ❝0 2026-10-15

PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews

ReviewFact labels factual errors in peer reviews; PremiseCheck combines an LLM with cheap decision models to verify them efficiently.

Affiliation unverified · Dingjie Song, Zhiling Yan, Ziqian Zhang +2

Benchmarks & data Efficient training & inference arXiv Code HF

#34 ▲163 ★36 ❝1 2026-09-22

The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks

Taste-Bench measures an agent's 'taste' for long-horizon decisions and explores ways to improve it.

Microsoft · Wenbo Pan, Zhichao Liu, Shujie Liu +6

Agents Benchmarks & data arXiv Code HF

#38 ▲21 ★2.8k ❝22 2026-09-29

TabFM: A Zero-Shot Foundation Model for Tabular Data

TabFM is a 400M-parameter tabular foundation model trained on synthetic tables that predicts zero-shot in a single forward pass.

Google · Weihao Kong, Erez Louidor Ilan, Shuxin Nie +6

LLMs Benchmarks & data arXiv Code HF

#44 ▲197 ★123 ❝0 2026-10-07

UltraText Bench: A Comprehensive Bilingual Benchmark for Evaluating Visual Text Rendering in Image Generation

UltraText Bench uses 432 bilingual prompts to evaluate dense visual text rendering in image generators.

Westlake University · Deyuan Liu, Yihao Hu, Jingxuan Zhang +13

Visual generation Benchmarks & data arXiv Code HF

#53 ▲233 ★42 ❝0 2026-10-09

AgentSWE: Can Coding Agents Build the Agent You Actually Want?

AgentSWE benchmarks whether coding agents can create, edit and optimise agents that truly meet the stated requirement.

Affiliation unverified · Jiahao Wang, Hongjin Qian, Yuyang Hu +2

Agents Code & SWE Benchmarks & data arXiv Code HF

#55 ▲73 ★572 ❝0 2026-09-21

GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay

GameHorizon provides multi-horizon gameplay data and evaluation to measure planning and control across model families.

Tencent · Yiran Wang, Xingyilang Yin, Junfu Pu +12

Benchmarks & data Agents arXiv Code HF

#63 ▲155 ★19 ❝0 2026-10-13

Into the Unknown: Benchmarking AI agents on open scientific research problems

Into the Unknown asks agents to write follow-up papers on unpublished directions from US national-lab scientists to test open research ability.

Carnegie Mellon University · Seungone Kim, Jiayi Geng, Young-Jun Lee +25

Agents AI for Science Benchmarks & data arXiv Code HF

#64 ▲291 ★5 ❝0 2026-10-16

OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners

OR-License benchmarks whether LLM agents can turn original client materials from 31 paid operations-research engagements into expert-grade solutions.

Data Science & Artificial Intelligence Lab at HKUST · Peisong Wang, Feixue Liu, Yize Meng +16

Benchmarks & data Agents arXiv Code HF

#65 ▲34 ★637 ❝9 2026-09-29

Systematically Exploring the Capabilities of GPT-6 Astra as Embodied Policies

Systematically tests GPT-6 Astra as an embodied policy across six domains, alone and combined with learned policies.

Galbot · Galbot Team, Xuchuan Chen, Xiaoqian Cheng +31

Robotics & embodied AI Benchmarks & data arXiv Code HF

#80 ▲153 ★13 ❝1 2026-10-06

Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?

Learn2Play Bench uses text games with novel or counterintuitive rules to test whether agents learn from experience.

National University of Singapore · Yibo Li, Jinhang Qiu, Zhi Zheng +4

Agents Benchmarks & data arXiv Code HF

#89 ▲149 ★38 ❝0 2026-10-14

MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environments

MAI-Gym is an open all-in-one GUI agent platform unifying evaluation, data collection and online RL at up to 20,000 parallel environments.

Tongyi-MAI · Panrong Tong, Quyu Kong, Pengxiang Li +12

Agents Reinforcement learning Benchmarks & data arXiv Code HF

#91 ▲110 ★113 ❝0 2026-10-14

Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents

Doc2Harness directly benchmarks whether LLMs can engineer a long-horizon coding-agent harness from a development document.

Institute of Computing Technology · Yuyao Ge, Shenghua Liu, Peizhong Gao +8

Agents Benchmarks & data Code & SWE arXiv Code HF

#93 ▲170 ★18 2026-10-19

EgoXperience: Learning World-Grounded Egocentric Representations from Human Experience

EgoXperience jointly reconstructs observer motion, scene geometry and hand-object interaction from egocentric video, with an auto-annotation engine and benchmark.

Ropedia · Xinhao Ji, Hui En Pang, Hao Li +12

Vision & 3D Robotics & embodied AI Benchmarks & data arXiv Code HF

#97 ▲74 ★49 ❝3 2026-10-01

PROWBench: Do Video Models Render What the Program Specifies?

PROWBench tests whether video models faithfully render program-specified world events, aimed at next-gen game engines.

Alaya Lab · Zheng-Hui Huang, Guixu Lin, Yu-Ju Tsai +6

Visual generation Benchmarks & data arXiv Code HF

#99 ▲206 ★6 ❝0 2026-10-13

RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignment

RoTA-Bench evaluates VLMs' 3D spatial reasoning by having them rotate objects to match target silhouettes interactively.

MAC-AutoML · Qingchuan Ma, Yuexiao Ma, Xiang Meng +7

Multimodal Benchmarks & data Reasoning arXiv Code HF

#101 ▲76 ★56 ❝0 2026-10-10

Last Secure Code Benchmark: Is the Code You Vibe Secure?

Builds 450 tasks from 150 real vulnerabilities to check whether code written by vibe-coding agents is secure.

Qwen · Tianyi Wu, Chao Ge, Peng Chen +12

Code & SWE Safety & interpretability Benchmarks & data arXiv Code HF

#105 ▲44 ❝40 2026-09-22

JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

Uses a decision-only judge as a cheap first pass and escalates to stronger judges only when unsure.

Carnegie Mellon University · Yubo Li, Yidi Miao, Ramayya Krishnan +1

Benchmarks & data Efficient training & inference arXiv HF

#107 ▲156 ★13 ❝0 2026-10-19

Holi-World: Building Holistic World Intelligence from Human-Ego Experience

Holi-World is an ecosystem for learning world intelligence from human egocentric experience, from capture to evaluation.

Ropedia · Hao Li, Xiyuan Zheng, Yuanyuan Gao +7

Vision & 3D Benchmarks & data arXiv Code HF

#113 ▲92 ★83 ❝0 2026-10-14

More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning

Compares ten agentic-RL self-distillation methods against GRPO under identical conditions to check real gains and costs.

Zhejiang University · Linhao Zhong, Bozhen Fang, Xuehao Wang +7

Agents Reinforcement learning Benchmarks & data arXiv Code HF

#116 ▲287 ❝0 2026-10-01

RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversations

RealCompanion releases ten real AI-companion relationships (27,218 messages over up to 120 days) to test long-term understanding of a person.

Quis Lab · Arman Behnam, Sunglyoung Kim, Jiayi Yu +2

Benchmarks & data Agents arXiv HF

#130 ▲165 ❝0 2026-10-09

Scaling Agentic Data for Long-Horizon Terminal Intelligence

Scales synthesis of high-quality long-horizon terminal trajectories to train agents for professional tasks.

alibaba · Shun Zou, Ziyu Ma, Yi Zou +6

Agents Benchmarks & data arXiv HF

#132 ▲132 ★9 ❝0 2026-10-07

LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets

LiveMACE uses live financial markets to evaluate persistent LLM agents on both outcomes and process diagnostics.

Fudan University · Jun Zhao, Leiming Fu, Yanbo Wen +9

Agents Benchmarks & data arXiv Code HF

#134 ▲34 ★178 ❝4 2026-10-02

4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes

4DCodeBench asks agents to reconstruct dynamic scenes from video as executable graphics programs.

4DCodeBench · Ruihong Shen, Žiga Kovačič, Peter Kulits +6

Benchmarks & data Code & SWE Vision & 3D arXiv Code HF

#137 ▲111 ★20 ❝0 2026-10-08

SoftwareGym: Scaling Computer-Use Agent Training with Diverse Synthesized Software Environments

SoftwareGym automatically synthesises diverse, executable, verifiable GUI software environments to scale computer-use agent training.

Shanghai Jiao Tong University · Tian Jin, Wenhao Wang, Jing Kang +4

Agents Benchmarks & data arXiv Code HF

#145 ▲43 ★48 ❝1 2026-10-14

Orbia: Do Generated Videos Form a Consistent 3D World?

Orbia is a benchmark for 3D consistency of interactive world models, testing whether scenes persist on revisits.

Tencent · Dongyue Lu, Guian Fang, Hur Lim +20

Visual generation Benchmarks & data arXiv Code HF

#146 ▲134 ★7 ❝0 2026-09-26

VoxMem: Benchmarking Multimodal Memory in Large Audio Language Models

VoxMem benchmarks whether audio LLMs remember who said what, how, and what was audible—information beyond the transcript.

The University of Melbourne · Yang Xiao, Vidhyasaharan Sethu, Eun-Jung Holden +1

Multimodal Benchmarks & data arXiv Code HF

#148 ▲60 ★135 ❝0 2026-10-13

OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding

OmniCompass evaluates video grounding and understanding on separate vision-language and audio-visual tracks.

VCLab · Xuyang Liu, Shihao Wang, Xinyu Wei +5

Multimodal Benchmarks & data arXiv Code HF

#161 ▲72 ★13 ❝0 2026-09-27

TraceDance: An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Traces

TraceDance automatically builds targeted benchmarks for specific undesirable agent behaviours from real deployment traces.

ByteDance · Dehai Min, Daoan Zhang, Yiming Zeng +13

Agents Benchmarks & data Safety & interpretability arXiv Code HF

#167 ▲75 ★46 ❝0 2026-09-29

EngiWorld: What Can Frontier Agents Deliver in Professional Engineering Environments?

EngiWorld has 1,301 expert tasks across six engineering domains (CAD, CAE, EDA…) and 26 professional software platforms.

Zhiman AI · Hongcheng Gao, Hailong Qu, Yu Lei +19

Agents Benchmarks & data arXiv Code HF

#176 ▲65 ★5 ❝3 2026-09-30

More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Models

Finds JEV-like direct-decision models under-use ordinal scales, over-predicting middle labels.

Jilin University · Tianxiang Gao, Jinzhe Li, Zhiyuan Li +2

Benchmarks & data arXiv Code HF

#187 ▲90 ❝1 2026-09-25

Duplex-MPE: Benchmarking Multi-Party Interaction in Full-Duplex Dialogue

Duplex-MPE uses 2,000 multi-party scenarios to test when a full-duplex voice assistant should answer, stay silent or stop.

Peking University · Chengqian Ma, Wenhao Feng, Weixuan Jin +7

Multimodal Benchmarks & data arXiv HF

#200 ▲106 ★4 ❝0 2026-09-30

WorldAuditBench: Interactive 3D World Auditing with Multimodal Agents

WorldAuditBench has multimodal agents navigate 3D worlds to find anomalies like floating objects or walk-through walls.

University of California, Santa Barbara · Ziyan Jiang, Jingbo Yang, Jiabao Ji +5

Multimodal Agents Benchmarks & data arXiv Code HF

#202 ▲63 ★14 ❝1 2026-09-30

Science or Slop?: Benchmarking and Mitigating Scientific Slop in AI-Generated Papers

SciSlopBench collects 390 AI-generated papers to detect plausible-looking but broken 'scientific slop' and explores mitigations.

Seoul National University · Yerim Oh, Young-Jun Lee, Jaewoo Ahn +2

Benchmarks & data AI for Science arXiv Code HF

#204 ▲74 ★17 ❝0 2026-10-09

EmbodiedCLBench: Evaluating Continual Learning for Self-Evolving Embodied Agents

EmbodiedCLBench is the first embodied continual-learning benchmark, testing whether agents recombine learned skills on new tasks.

PKU-VaLuE-Lab · Jie Huang, Yanan Chen, Ruixun Liu +6

Robotics & embodied AI Agents Benchmarks & data arXiv Code HF

#208 ▲103 ❝0 2026-09-29

Predictive Credit: Measuring What Scientific Explanations Add to Experimental Forecasts

Proposes 'predictive credit' to measure whether research agents' explanations improve experimental forecasts; preregistered tests were inconclusive.

Carnegie Mellon University · Jingjie Ning, Xueqi Li, Yibo Kong +1

AI for Science Benchmarks & data arXiv HF

#215 ▲72 ★2 ❝2 2026-10-06

CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?

CheckerBench tests whether long-horizon agents can build working static-analysis checkers, with 300 tasks from 297 CVEs.

Humanlaya Data Lab · Hang He, Li Wang, Hao Chen +9

Code & SWE Agents Benchmarks & data arXiv Code HF

#224 ▲25 ★185 ❝0 2026-10-14

BlenderWorld: Benchmarking Visual Agents for 3D Creation from Real-World References

Qwen's BlenderWorld benchmarks visual agents creating and editing 3D content in Blender from real-world references.

Qwen · Xinyu Fang, Yuchi Wang, Weihao Yu +12

Benchmarks & data Agents Multimodal arXiv Code HF

#228 ▲75 ★11 ❝0 2026-09-30

EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videos

EgoTools targets tool-centric reasoning in real egocentric videos, covering hand-tool-object relations and procedures.

Ropedia · Shulin Tian, Junsu Kim, Shuai Liu +17

Multimodal Reasoning Benchmarks & data arXiv Code HF

#234 ▲40 ★41 ❝1 2026-10-07

RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments

RobotWorld uses 84 simulated tasks (manipulation, locomotion, driving, aerial) to test multimodal agents at robot use.

Affiliation unverified · Zhiqin Yang, Chenxin Li, Xiaomeng Hu +30

Robotics & embodied AI Agents Benchmarks & data arXiv Code HF

#237 ▲31 ★26 ❝1 2026-09-24

ExplorationBench: Measuring AI Systems' Exploration in Verifiable Alien Worlds

Tencent Hunyuan's ExplorationBench measures scientific exploration in verifiable alien worlds, ruling out recall from pre-training.

Tencent Hunyuan · Ming Zhang, Zhenghao Xiang, Peizhong Gao +17

AI for Science Benchmarks & data Agents arXiv Code HF

#238 ▲85 ★5 ❝0 2026-10-06

DecepEval: A Benchmark for Evaluating Deception in LLM Agents

DecepEval uses 1,532 instances and a 'deception diamond' framework to test when LLM agents are likely to deceive.

Xi'an Jiaotong University · Yiming Xu, Hongyue Yu, Beihua Yang +8

Safety & interpretability Benchmarks & data Agents arXiv Code HF

#247 ▲123 ❝0 2026-09-30

GraphForge: Training Working Agents with Graph-Anchored Workspace Synthesis

GraphForge synthesises verifiable workspace tasks grounded in real files via evidence graphs to train working agents.

University of Science and Technology of China · Qisheng Su, Hanchen Wang, Guanru Zhu +9

Agents Benchmarks & data arXiv HF

#249 ▲56 ★21 ❝0 2026-10-01

SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation

SimuVerity's 101 tasks check whether agent-generated Simulink models meet engineering requirements, not just whether they run.

Xi'an Jiaotong University · Ruiqi Zhang, Jiahao Wang, Mingxuan Li +9

Agents Benchmarks & data Code & SWE arXiv Code HF

#254 ▲53 ★4 ❝0 2026-10-15

From Research Goals to Scientific Closure: Evaluating Agents Through Evidence-Supported Physics Research

PhysClosureBench uses 50 expert-verified physics tasks to test whether agents can run investigations to well-supported conclusions.

Shanghai AI Laboratory · Yijia Mao, Song Dai, Tao Chen +14

AI for Science Agents Benchmarks & data arXiv Code HF

#264 ▲60 ★26 ❝0 2026-10-01

EditHero: A Benchmark for Long-Horizon Part-Level 3D Editing and Vibe Modeling

EditHero is a benchmark for long-horizon, part-level 3D editing, checking that each revision changes only what was asked.

Affiliation unverified · Ruihan Yu, Yu-Ju Tsai, Muyao Niu +9

Vision & 3D Benchmarks & data arXiv Code HF

#276 ▲69 ❝1 2026-09-30

PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Loop

PhysVista evaluates VLMs' physical intelligence through a perception-reasoning-assessment loop.

university of science and technology of china · Xinge Peng, Yiting Lu, Tianwu Zhi +4

Multimodal Benchmarks & data Reasoning arXiv HF

#283 ▲35 ★29 ❝0 2026-10-16

RAP: A Dataset for Paper-to-Poster Generation via Declarative Markup from Human-Crafted Exemplars

RAP pairs 25K+ papers with posters derendered into HTML/CSS slots, enabling render-and-compare evaluation of paper-to-poster generation.

meituan · Haonan Han, Yukang Lin, Minliang Lin +7

Benchmarks & data Code & SWE arXiv Code HF

#288 ▲33 ★12 ❝0 2026-10-16

Execution Before Memory: Diagnosing Memory Failures in Video World Models

MIST separates instruction execution from memory in video world model tests, finding the bottleneck is usually execution rather than forgetting.

The University of Hong Kong · Weiyang Jin, Jiwen Yu, Sihan Xu +2

Benchmarks & data Visual generation arXiv Code HF

#290 ▲37 ★7 2026-10-19

AnyGoal: On World Model Generalization Beyond Expert Rollouts

AnyGoal benchmarks whether world models generalize to recombinations of familiar state and action factors; current models consistently fail.

galilai-group · Heejeong Nam, Divyat Mahajan, Quentin Le Lidec +2

Benchmarks & data Agents arXiv Code HF

#291 ▲24 ★11 ❝0 2026-10-20

MMEnterprise-Bench: Evaluating Enterprise Agents over Evolving Multimodal Records

MMEnterprise-Bench has 1,613 questions over evolving multimodal workplace records, testing evidence reasoning, conflict resolution and unanswerability.

Affiliation unverified · Seohyun Lee, Jongha Kim, Dohwan Ko +4

Benchmarks & data Agents arXiv Code HF

#296 ▲28 ★3 ❝0 2026-10-19

AdaExploreBench: Benchmarking LLM Agents for Proactive Exploration and Test-time Adaptation in Evolving Environments

AdaExploreBench uses hidden, silently drifting rules to test whether LLM agents proactively explore and adapt at test time.

HKUST · Rui Wang, Zishen Lai, Jiayu Liu +15

Benchmarks & data Agents arXiv Code HF

#298 ▲26 ★4 ❝0 2026-10-09

APEX-Harm: Evaluating Agent Safety in Shared Professional Workspaces

APEX-Harm tests finance, consulting and legal agents against prompt and script injection in large shared workspaces.

King Abdullah University of Science and Technology · Ziyi Yang, Adel Bibi, Guohao Li +2

Safety & interpretability Agents Benchmarks & data arXiv Code HF

#300 ▲21 ★9 ❝0 2026-09-24

IndicBankBench: Evaluating Safety and Reliability of Language Model Assistants in Indian Retail Banking

IndicBankBench evaluates Indian retail-banking assistants on 799 cases across safety, tool use and answer quality.

National Payments Corporation of India · Suvradip Paul, Chandra Bhushan, Harsh Sharma +4

Safety & interpretability Benchmarks & data arXiv Code HF

#301 ▲34 ❝0 2026-10-14

Interlock: From Local Correctness to Task Completion in Long-Horizon Agents

Interlock represents tasks as typed completion graphs to synthesize long-horizon agent tasks and rubrics that check end-to-end completion.

LongCat · Linji Hao, Yanji He, Zihan Wang +6

Agents Benchmarks & data arXiv HF

#303 ▲33 ❝0 2026-10-16

MTEdit: Packed Trajectories Beat Rollouts for Multi-Turn Image Editing

MTEdit builds 500K multi-turn image-editing trajectories by generating several edit states in one packed canvas, plus a benchmark.

SenseNova · Houyuan Chen, Fanzhou Wang, Xiangyu Fan +8

Visual generation Benchmarks & data arXiv HF

#306 ▲23 ❝0 2026-09-24

Where Does Retrieval-Based Open-Ended Evaluation Fail? Automatic Taxonomy Induction from Long-Form Medical Answer Factuality Verification

Builds taxonomies of where retrieval-based factuality checking of long medical answers fails.

Center for Language and Speech Processing @ JHU · Heyuan Huang, Jirui Dai, Alexandra DeLucia +5

AI for Science Benchmarks & data arXiv HF

Earlier (archive) 2

▲82 ★216 ❝0 2026-09-18

OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

Defines native audio-visual dialogue for omni models and provides synthetic data, a benchmark and training recipes.

Qwen · Haolin He, Yunfei Chu, Qi Chen +15

Multimodal Benchmarks & data arXiv Code HF

▲81 ★108 ❝1 2026-09-18

RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents

RecreationWorld asks computer-use agents to observe running software and re-implement it, with verifiable environments on five platforms.

Qwen · Shuai Bai, Jiayong Deng, Sicheng Fan +30

Agents Code & SWE Benchmarks & data arXiv Code HF

Search this topic in the archive →