https://caozuohua.github.io/tags/%E6%95%85%E9%9A%9C%E6%B3%A8%E5%85%A5/Recent content in 故障注入 on CAO ZUOHUAHugozh-cnFri, 18 Sep 2026 00:00:00 +0000https://caozuohua.github.io/posts/2026-09-18-agent-framework-evaluation-3-failure-evidence/Fri, 18 Sep 2026 00:00:00 +0000https://caozuohua.github.io/posts/2026-09-18-agent-framework-evaluation-3-failure-evidence/<p>这是 <a href="https://caozuohua.github.io/posts/2026-09-18-agent-framework-evaluation-series/">Agent 框架评测实验室系列</a>第三篇。</p> <p>一个 Agent 在 demo 中完成任务，最能说明的是：这次模型、网络和工具刚好都没有出大问题。企业系统真正区分框架能力的地方，往往发生在错误之后。</p> <h2 id="七类最低故障矩阵">七类最低故障矩阵</h2> <p>生产变更 Alpha 固定测试七类场景：</p> <table> <thead> <tr> <th>场景</th> <th>要证明什么</th> </tr> </thead> <tbody> <tr> <td>审批通过</td> <td>精确意图获批后才可继续</td> </tr> <tr> <td>审批拒绝</td> <td>拒绝不能通过新 ID 或重试绕过</td> </tr> <tr> <td>取消</td> <td>正在执行的 attempt 中断，预算结算</td> </tr> <tr> <td>工具失败</td> <td>错误可见，重试有界，不伪装成功</td> </tr> <tr> <td>worker 崩溃</td> <td>lease 和 reservation 可回收</td> </tr> <tr> <td>进程重启</td> <td>从已提交 checkpoint 恢复，不重跑完成阶段</td> </tr> <tr> <td>重复副作用</td> <td>同一外部操作最多生效一次</td> </tr> </tbody> </table> <p>这些测试不能只检查“最后还是成功了”。还要检查事件顺序、预算余额、checkpoint 哈希、工具调用次数和幂等收据。</p> <h2 id="审批不是一个布尔值">审批不是一个布尔值</h2> <p>如果审批只保存 <code>approved=true</code>，就无法确认批准的是哪次操作。安全审批至少绑定：</p> <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#66d9ef">class</span> <span style="color:#a6e22e">ApprovalDecision</span>(BaseModel): </span></span><span style="display:flex;"><span> approval_id: UUID </span></span><span style="display:flex;"><span> job_id: UUID </span></span><span style="display:flex;"><span> attempt_id: UUID </span></span><span style="display:flex;"><span> operation_id: UUID </span></span><span style="display:flex;"><span> action: str </span></span><span style="display:flex;"><span> arguments_sha256: str </span></span><span style="display:flex;"><span> decision: Literal[<span style="color:#e6db74">&#34;approved&#34;</span>, <span style="color:#e6db74">&#34;rejected&#34;</span>] </span></span><span style="display:flex;"><span> approver_id: str </span></span><span style="display:flex;"><span> revision: int </span></span></code></pre></div><p>恢复时重新计算参数摘要。job、operation、action、参数或 revision 任一不一致，都不能复用原批准。测试不仅要覆盖正确批准，还要专门提交“看起来很像”的错误批准。</p>