Skip to content

Testing and verification

Getting agents to write code that passes tests and verifying their claims that a task is done.

32 curated itemsRelated topicsCode reviewHooks and automationWorkflows

Latest curated items

Items 21–32 · 32 total
7/22Wed
  1. Augment Code · Blog62

    What is loop engineering, and how are leading software engineering teams using it?

    Augment Code proposes loop engineering: designing agent loops that run from trigger to execution to validation to outcome, with agents handling the intermediate steps and humans stepping in only at checkpoints that require judgment. The article compares loop engineering with prompt engineering and context engineering as distinct layers, lays out five stages—trigger, execution, validation, outcome, and improvement—and describes four team-level loops already running in production: code review, ticket-to-PR, vulnerability remediation, and incident response.

    Why it matters: Augment Code breaks loop engineering into five stages—trigger, execution, validation, outcome, and improvement—and lays out four team-level loop patterns already running in production.

7/20Mon
  1. Addy Osmani · Blog74

    Addy Osmani on software factories: the visible factory and the hidden factory, where validation is the bottleneck

    Addy Osmani proposes that a software factory has three layers—loop, harness, and factory. The factory isn’t a smarter agent; it’s multiple loops with harnesses feeding into a single review gate, with humans controlling the outer loop.

    Why it matters: The author breaks the software factory into three layers—loop, harness, and factory—and points out that validation, not generation, is the real bottleneck.

7/11Sat
  1. Habr · Kova13v80

    Using an evidence contract to constrain Claude Code's test conclusions: a QA Skill package

    A QA engineer distilled six months of experience doing web testing on Claude Code into an open-source Skill package called paranoid-qa. At its core is an evidence contract: Pass/Fail can only be based on actual artifacts like screenshots, request bodies, and logs; anything unverified gets marked Not tested; anything blocked by the environment gets marked Blocked; and forms must verify the real submitted payload.

    Why it matters: The author codified six months of QA experience into a testing Skill package for Claude Code, along with an evidence contract and failure checklist that can be reused directly.

7/8Wed
  1. AI Hero · Skills Updates65

    AI Hero skills repo ships v1.1: adds /wayfinder, renames /to-spec and /to-tickets

    AI Hero's skills repo ships v1.1, renaming /to-prd to /to-spec, merging /to-plan and /to-issues into /to-tickets, and adding new Skills like /wayfinder, /research, and /prototype.

    Why it matters: The author walks through the full Skill flow from grilling to deployment and gives the migration commands for the renames, the merge, and the new /wayfinder—useful for anyone building an AI development workflow.

6/15Mon
  1. Jesse Vincent78

    Superpowers 6 发布:构建提速最高 50%、token 花费降低最高 60%

    Superpowers 6 发布,作者称在 Anthropic 评测基准上构建耗时降低 50%、token 花费降低 60%,主要来自合并规范符合性与代码质量两个评审 agent、预先生成评审用的 diff 包让评审者少跑 git,以及调整编排器对任务所需 agent 类型的指引。

    Awaiting translation

    Why it matters: 作者用自建评测套件量化了 Superpowers 6 在构建耗时和 token 花费上的改进,并公开了实验记录与失败结论。

3/13Fri
  1. Ryan Lopopolo74

    智能体时代的生产函数变了:实现不再稀缺,验证才是

    作者认为,软件组织一直把人的实现时间当作稀缺投入,智能体打破了这个假设,因此策略必须可执行、验证必须随实现规模扩展。他以在大型 TypeScript 代码库开启 ESLint 的 no-await-in-loop 规则为例,发现 600 处违规,过去这需要昂贵的迁移,现在一个 PR 就能完成修复并补齐测试覆盖。

    Awaiting translation

    Why it matters: 作者以开启 ESLint 规则、迁移 600 处违规的亲身实践,说明智能体时代实现成本下降后,约束与验证为何成为新的稀缺环节。

3/9Mon
  1. Paper Compute · Engineering Blog78

    我让 Agent 玩 1000 回合宝可梦,它始终没走出卧室

    作者构建的宝可梦红版自主 Agent 跑了 1000 回合仍停在卧室,原因是它把背景瓦片图地址 0xC4F2 当成文本框状态标志,一直按 A 键却不知道卡住。

    Awaiting translation

    Why it matters: 作者用 1000 回合卡在卧室的失败复盘,说明记录并回放 Agent 会话状态比改提示词更能定位静默故障。

2/23Mon
  1. OpenAI Developer Blog · Codex72

    用 Codex 跑 25 小时长时程任务:一份可复用的项目记忆文件栈

    OpenAI 用 GPT-5.3-Codex 在 Extra High 推理档下从空仓库连续运行约 25 小时、消耗约 13M token、生成约 3 万行代码,做出一个可测试的设计工具。

    Awaiting translation

    Why it matters: 作者用 25 小时、13M token 的实测展示长时程智能体如何靠持久化项目记忆和逐里程碑验证保持不跑偏。

1/22Thu
11/19Wed
  1. OpenAI · Codex Cookbook67

    How to modernize a legacy codebase in phases with Codex CLI

    In the Codex Cookbook, OpenAI lays out a complete workflow for modernizing a legacy codebase with Codex CLI, using a COBOL portfolio system as the example and moving through five phases built around an ExecPlan design document.

    Why it matters: Using a COBOL portfolio system as the example, it offers reusable documents and a validation workflow for modernizing legacy code in phases with Codex CLI.

8/29Fri
  1. OpenAI · Codex Cookbook74

    在 GitLab CI/CD 中用 Codex CLI 自动做代码质量检查与安全修复

    OpenAI Codex Cookbook 给出把 Codex CLI 接入 GitLab CI/CD 的完整做法,用于生成 CodeClimate JSON 代码质量报告、把 SAST 结果整理成 security_priority.md,并让 Codex 输出可 git apply 的补丁。

    Awaiting translation

    Why it matters: 官方 Cookbook 给出把 Codex CLI 接入 GitLab CI 的完整配置,含提示词约束、JSON 标记提取与 diff 校验,可直接照搬。

6/4Wed
  1. Martin Fowler · Exploring Generative AI66

    自主编码智能体实测:一次 OpenAI Codex 运行记录

    Martin Fowler 给 OpenAI Codex 布置了一个前端标签格式化的化妆类小任务,并完整公开了 Codex 的日志和生成的 PR。日志显示 Codex 主要靠 grep 反复文本搜索定位代码,中途因把 AGENTS.md 误写成 AGENT.md 来回折腾,还因删掉 .yarnrc 导致测试无法运行,最终 PR 里有两个回归测试失败。

    Awaiting translation

    Why it matters: 作者完整记录 Codex 自主完成一次前端小任务的日志,并对比 6 次运行结果,展示后台编码智能体在环境配置和代码复用上的真实短板。