Addy Osmani on testing strategies for AI agent outputs
Understand the story
On October 5, 2026, Addy Osmani shared his thoughts on X about validating the quality of AI agent outputs, suggesting that agents should be equipped with ways to verify their outputs—testing being one part of that. He recommends four kinds of tests: first, end-to-end tests that simulate real user flows, serving as ground truth; second, property-based tests that declare what should never happen and generate thousands of cases to try to break those constraints; third, when swapping out a system, compare the old and new versions using random inputs; and fourth, require tests to run fast and deterministically, otherwise the agent may learn to retry instead of fixing the problem. So far this post stays at the level of methodological advice, and the report mentions no specific tools, real-world cases, or follow-up updates.
Generated by AI from reports · Updated 2 days ago
Report timeline
Follow the reports to explore different perspectives.
- Addy Osmani · X给 AI 智能体配上测试来校验产出质量
Addy Osmani 建议为 AI 智能体配备校验产出的手段，测试是其中一部分。他推荐四类测试：模拟真实用户流程的端到端测试作为基准真相；用基于属性的测试声明绝不应发生的情况并生成数千用例尝试突破约束；替换系统时用随机输入对比新旧版本；要求测试快速且确定性运行，否则智能体可能学会重试而非修复问题。
Interest in this story
Current interest 3·Peak within comparable coverage 10（Oct 5, 15:00）·Change over 24 hours within comparable coverage -50%
The trend compares the same participants under continuous, complete observation, so its coverage may be narrower than the current score. Hover or tap to view hourly interest; use the left and right arrow keys to navigate.