Engineering & computing · OECD FoS 1.2 · China MoE 0835 软件工程
Software engineering
软件工程
L3Systemic transformation ↑ Rising ↓ from L4 Score 3.10 · Last reviewed 2026-10-21 · Next review 2026-12-20
BIGGEST BREAKTHROUGH 2021-07-07 AI · majorField · major
Codex and HumanEval: evaluating LLMs trained on code ↗
Origin of code LLMs and Copilot
Why this level
Lowered to L3: adoption is the highest of any field (Stack Overflow 2026: over 70% of respondents use AI daily), but Google’s ~75% and Anthropic’s 90%+ code shares are vendor claims and cannot alone justify L4. METR’s RCT found experienced OSS developers 19% slower in early 2025; 2026 follow-ups estimate speed-ups but with design limits, so independent productivity evidence is mixed.
↓ L4 → L3
S = 0.30·method + 0.35·discovery + 0.25·adoption + 0.10·education (education scores 0 when there is no evidence; if empty, the other weights are renormalised)
Sources
- survey.stackoverflow.co/2026/ai
- metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study
- metr.org/blog/2026-02-24-uplift-update
Composite 2.5–3.49 (or ≥3.5 with any of method/discovery/adoption below 3): AI is standard infrastructure with multiple independent breakthroughs. All levels →
Research directions
Coding agentsAutomated repair & testingCode reviewSpec-to-deploy automation
Papers
- Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
Developers believed +20% but measured −19%
2025-07-12 · arXiv
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Real-repo issue benchmark; main yardstick for coding agents
2023-10-10 · arXiv
- The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
Early productivity evidence on a simple task
2023-02-13 · arXiv
- Competition-level code generation with AlphaCode
First median-human level in programming contests
2022-12-09 · Science
- Evaluating Large Language Models Trained on Code
Origin of code LLMs and Copilot
2021-07-07 · arXiv
- SWE-DIVE: Multi-Turn Coding Agent Evaluation with Dynamic User Simulation
Evaluates coding agents in multi-turn sessions with simulated users giving under-specified, evolving requests.
2026-10-21 · arXiv · Lorenz Brehme, Benjamin Steenhoek, Ruth Breu
- PAUSE: Search-Free Steering of Coding Agents Before Their First Source-Code Edit
Steers coding agents with a single planning intervention before their first source edit, cheaply improving patch success.
2026-10-21 · arXiv · Viral Patel, Boyin Tan, Jason Xue, Youcheng Sun
- TraceLite: Efficient Trace Reduction for SWE Agents via Optical Compression and Redundancy Removal
Training-free compression of SWE-agent traces by rendering history as images and removing redundancy.
2026-10-20 · arXiv · Jingyu Xiao, Qianxi Liu, Yintong Huo, Yuan Jiang
- InterWebCoder: Learning Interactive Webpage Generation from Executable Tests via Reinforcement Learning
Synthesises interactive webpages with executable tests and trains MLLMs with SFT plus GRPO for functional correctness.
2026-10-20 · arXiv · Jingyu Xiao, Dasen Dai, Yuxuan Wan, Jen-tse Huang
- From Code Generator to Engineering Companion: A Plant-aware Agent for PLC Programming
A plant-aware PLC programming copilot with graduated tool access that keeps engineers in control.
2026-10-20 · arXiv · Joschka Kersting, Gesa Benndorf
- Co-Evolving Policy and Reward for Code Generation: A Mechanistic Study of Single-Adapter Drift
Tests whether SPARK-style policy–reward co-evolution transfers to RL with verifiable rewards for code generation.
2026-10-19 · arXiv · Amarsaikhan Batjargal, Noriko Tomuro
- When Coding Agents Outpace Review: Evaluating PRD-Driven Development in Mature Enterprise Repositories
PACE benchmark (43 enterprise tasks, 750 audited code points) measures whether coding agents fulfil full PRDs.
2026-10-19 · arXiv · Tianqi Liu, Yandong Guan, Songze Li, Wenkai Wang
- SPLASH: Contrastive Audit Pressure for Detecting and Localizing Security Hallucinations in Generated Code
SPLASH contrasts security- vs style-review log-probabilities to detect and localise insecure LLM-generated code.
2026-10-19 · arXiv · Mirza Masfiqur Rahman, MD Ajwad Akil, Imtiaz Karim, Ashish Kundu
- From arXiv, bioRxiv, medRxiv and OpenAlex (last ~30 days); summaries written by AIMap from the abstract.
- Matrix-Game 4.0: Harnessing Video Models for Real-Time Playable Game Worlds 2026-10-20
- S-Agent 2.0: Spatial Reasoning with a General Coding Agent 2026-10-20
- SolarRenderer: Few-Step Generative Reference Rendering 2026-10-20
- InfiniDream: Open-Ended Photorealistic 3D Worlds Through Procedural Generation 2026-10-19
- HexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents 2026-10-16
- Self-Evolving Coding Agents via Harness-Policy Optimization 2026-10-16
- All related papers in the archive →
Milestones 10
Big glowing nodes are leaps: AI leap cool · Field leap warm · double ring = both
-
Industry
Stack Overflow 2026 survey (30k+): over 70% use AI daily, 31% of them 4+ h/day; 48% trust AI only when easily verifiableIndependent large-sample adoption data
Stack Overflow -
Industry
Anthropic: 90%+ of its code written by ClaudeVendor claim: 90%+ of code written by Claude
TechSpotVendor claim -
Industry
Google: ~75% of new code is AI-generatedVendor claim: ~75% of new code AI-generated
The VergeVendor claim -
Paper
METR update: late-2025 follow-up estimates 4–18% speed-ups, CIs span zero; design to changeShows how hard independent productivity measurement remains
METR -
Paper
METR RCT: early-2025 AI made experienced OSS developers 19% slowerDevelopers believed +20% but measured −19%
arXivPreprint -
Benchmark
SWE-bench: resolving real GitHub issues (ICLR 2024)Real-repo issue benchmark; main yardstick for coding agents
arXivPreprint -
Paper
Copilot RCT: developers finish a task 55.8% fasterEarly productivity evidence on a simple task
arXivPreprint -
Competition
AlphaCode: competition-level code generation (Science)First median-human level in programming contests
Science -
Benchmark ⓘ note
Codex and HumanEval: evaluating LLMs trained on codeOrigin of code LLMs and Copilot
arXivPreprint
No milestones match this filter.
More papers & repos 7
| Type | Title | Date |
|---|---|---|
| Paper | Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity Preprint | 2025-07-12 |
| Paper | SWE-bench: Can Language Models Resolve Real-World GitHub Issues? Preprint | 2023-10-10 |
| Paper | The Impact of AI on Developer Productivity: Evidence from GitHub Copilot Preprint | 2023-02-13 |
| Paper | Competition-level code generation with AlphaCode | 2022-12-09 |
| Paper | Evaluating Large Language Models Trained on Code Preprint | 2021-07-07 |
| Repo | SWE-bench/SWE-bench To be verified | — |
| Repo | openai/codex To be verified | — |
Latest news 12
- Cortex: A Fixed-Point Theory of Governed Coding Agents - Technical ReportarXiv cs.AIenPaperPreprint
- Stress-Query and Temporal Reliability of Vision-Language Models as copilots in Autonomous UAV NavigationarXiv cs.ROenPaperPreprint
- How Effectively Do Security Skills Guide Coding Agents in Vulnerability Detection in Linux Kernel?arXiv cs.CRenPaperPreprint
- The results of the 2026 Developer Survey are here!Stack Overflow Blogen