Coding model and tool benchmarks, with hands-on comparisons using the same tasks.
The author ran Qwen2.5 7B/32B/72B on a single AMD MI300X with vLLM ROCm, priced at $2.99/GPU-hr. The BF16 baseline was 7B at $0.227/M, 32B at $0.77/M, and 72B at $1.67/M output tokens.
Why it matters: The author benchmarked FP8 quantization on the MI300X and found that per-token billing can hide the model's output degrading into gibberish, then gave a reusable way to verify it.
GitHub 发布代码评审离线基准 ReviewBench，基于 1.039 亿个 GitHub PR 的分布特征，构建了覆盖 19 种语言、219 个公开 PR 的评测集，并公开数据集、评分规则与 LLM 评审模型配置。
Awaiting translation
Why it matters: GitHub 公开了 AI 代码评审基准的数据集、评分规则与评测入口，读者可据此对比不同评审智能体。
The author tested 10 open-source prompt injection detectors against 629 AgentDojo injection attacks—each buried in real tool output—plus 97 benign samples.
Why it matters: The author tested 10 open-source detectors against 629 real injection attacks, with full comparison data at both default thresholds and after calibration.
On September 3, 2026, OpenAI released GPT-6 Astra and Astra Pro, initially limited to enterprises in the Daybreak cybersecurity program, with paid ChatGPT, the API, and AWS opening up over the following days.
Why it matters: We break down the benchmark comparison between GPT-6 Astra and Fable 5.1, pointing out that the tested versions and harnesses differ across teams, so readers can judge which scores are actually comparable.
Cline 让八个模型在自家 harness 里做 IMO 2026 六道题，证明由 GPT-5.5 和 Claude Opus 5 双盲按 0–7 分制评分、Gemini 3.1 Pro 仲裁，金牌线为 29 分。
Awaiting translation
Why it matters: Cline 用同一套 harness 盲评八个模型做 IMO 2026，给出分数与单次成本对照，可看开源权重模型的实际性价比。
Cline has open-sourced its evaluation methodology for open-weight models, along with a hill-climbing score and trace worth over one thousand dollars, available for download and analysis. The post lays out five heuristics from the Hill Climber's Checklist: set a North Star metric, quantify noise, break down failure modes by task/model/vendor, don't assume more thinking is always better, and keep a private evaluation set.
Why it matters: Cline shares its evaluation methodology and a trace worth over a thousand dollars, offering five transferable hill-climbing heuristics that teams building their own harness can reference.
Martin Fowler 用 Sonnet 4.6 生成、Opus 4.8 盲评的方式，对小型、中型和较大型三类业务逻辑任务分别跑 TDD 与非 TDD 方案，结论是两者质量没有明显差异，非 TDD 方案在设计和测试质量上还多次略高，变异分数也没有实质差别。
Awaiting translation
Why it matters: 作者用同一批任务对比 TDD 与非 TDD 智能体实现，给出 token 成本与设计质量差异，并反思哪些 TDD 收益在智能体循环里失效。
The Terminal-Bench team releases Terminal-Bench 3.0, whose first version spans 7 domains and 74 tasks, with the strongest model passing about 34%. Building on Terminal-Bench 2.1, this release broadens task diversity and adds CI/CD, semantic versioning, and result migration to keep improving the benchmark.
Why it matters: Terminal-Bench 3.0 rebuilds the benchmark with 74 tasks and CI/CD-based versioning, so readers can see how the new benchmark separates models.
The Terminal-Bench team has shipped a batch of new Harbor features that let datasets be released by version and let leaderboards migrate to new versions by reusing, re-evaluating, or rerunning trials. Tasks use semantic versioning: patch-level changes reuse old results as-is, validator changes only require re-evaluating saved artifacts, and only major changes that significantly alter the agent environment require a rerun. Dataset versions follow the highest version number among the tasks, and leaderboards use diffs to rerun only the tasks with major changes.
Why it matters: The Terminal-Bench team maintains the benchmark like software, laying out concrete mechanisms for task semantic versioning and leaderboard upgrades that you can carry over to your own evaluation pipeline.
Cline 用一条提示词启动 17 小时连续运行的编码智能体，以 GPT-5.6-Sol 为 leader 模型，把 Kimi K3 在 Terminal-Bench 2.1 上的成绩从基线 69/89（77.5%，$79）提升到确认运行 79/89（88.8%，$49.8），超过 Moonshot 自报的 88.3%。
Awaiting translation
Why it matters: Cline 用一条提示词让智能体自主完成 17 小时爬山，把 Terminal-Bench 2.1 从 77.5% 提到 88.8%，可看具体修了哪些 harness 问题。
Terminal-Bench 发布 Challenges，一种长周期、高 token 消耗的单任务基准，要求智能体在无时间与资源限制下自主完成整个项目，首批开放三个挑战。
Awaiting translation
Why it matters: Terminal-Bench 官方推出长周期单任务基准，并公开三个挑战的实测失败模式，可供评估智能体长时自主能力时参考。
In AGENTS.md, Augment Code front-loads roughly 2.5k characters of Karpathy-style coding rules, then runs 40 OpenClaw PRs through Auggie, Claude Code, and Codex for comparison.
Why it matters: A head-to-head test of three coding agents on the same set of PRs shows that prompt constraints mainly cut costs rather than improve quality, and it also surfaces differences between the harnesses.
Augment Code pulled dozens of AGENTS.md files from its own monorepo and used its internal benchmark suite AuggieBench to compare how the same tasks performed with and without the file. The best files delivered a quality boost equivalent to upgrading from Haiku to Opus, while the worst made the output worse than having no AGENTS.md at all.
Why it matters: Augment Code used internal benchmarks to quantify how much the different ways of writing AGENTS.md actually differ, so readers can adjust their own repo's documentation structure accordingly.
Terminal-Bench has released version 2.0 and the Harbor package. The former is a more rigorously validated, harder benchmark for evaluating agents; the latter is for evaluating and optimizing agents. Harbor rewrites Terminal-Bench's test harness, supports deploying containers in the cloud, provides rollout interfaces for RL and SFT, and works with any agent you can put in a container.
Why it matters: Terminal-Bench 2.0 and Harbor are released together, so readers can see how the agent evaluation benchmark is validated and how to scale it in the cloud.
Martin Fowler 试用 Kiro、spec-kit 和 Tessl 三款自称实现 spec-driven development（SDD）的工具，把 SDD 归纳为 spec-first、spec-anchored、spec-as-source 三个层次，并指出目前所有方案都停留在 spec-first。
Awaiting translation
Why it matters: 作者亲手试用 Kiro、spec-kit 和 Tessl 三款 SDD 工具，给出 spec-first、spec-anchored、spec-as-source 三层划分，并指出小任务被过度规格化的问题。
Terminal-Bench 发布首个版本，用于量化 AI 智能体在终端中执行复杂任务的能力，首发数据集 Terminal-Bench-Core-v0 包含 80 个手工编写并人工验证的任务，每个任务配有独立 Docker 环境、人工验证的解法与测试用例。
Awaiting translation
Why it matters: Terminal-Bench 给出 80 个带 Docker 环境和测试用例的终端任务，可用来横向比较不同智能体在命令行中的实际表现。