// AI news · 2026-10-28 · Hugging Face Blog
LLMs' reliability in judging research novelty assessed
A study evaluates LLMs' reliability in judging research novelty using human-authored ideas from ICLR 2026 reviews.
Published 2026-10-28 00:12 (Beijing time)Source Hugging Face BlogAuto-summarised, fact-checked against source
01Summary
Results show significant variation across models due to prompt changes, with no consistent improvement from retrieval or reasoning. Custom novelty systems underperform baseline prompts. AI-generated ideas are frequently misjudged as novel, and stylistic differences do not explain the errors.
Summarised from the original article by an open-source model, then checked automatically: every number and English name must appear in the source. The original article is authoritative.
02Key points
- Novelty judgments vary widely across LLMs with minor prompt changes.
- Retrieval and reasoning don't consistently improve evaluation accuracy.
- Custom novelty systems underperform simple prompt-based baselines.
- AI-generated ideas are often misjudged as novel despite weak originality.
03Source
Can we trust LLMs to judge whether a research idea is novel?
Hugging Face Blog · published 2026-10-28 00:12 Beijing time
05Related news
- Experts frequently follow AI agent biases2026-10-27 · Anthropic Research
- Anthropic Expands GSA Partnership with 40% Claude Discount2026-10-27 · Anthropic News
- Anthropic Discloses Enterprise Ecosystem Strategy in IPO Filings2026-10-27 · IT之家
- Anthropic IPO Highlights AI's Existential Risks2026-10-27 · WIRED
- Anthropic Sued Over Use of 274 Elvis Songs2026-10-26 · Guardian Tech
- Anthropic expands AI for Science to institutions2026-10-28 · Anthropic News
From AIMap AI news: collected from 1,100+ official and media sources. Summaries only restate the original article; follow the link for the full story. All AI news · Today