指南
切片
整体比率可能保持稳定，而测试套件的某一部分却已崩溃。切片是测试套件中一个经过声明、单独度量的部分，从而让这种崩溃可见。切片附带两条纪律，因为同时查看测试套件的许多部分，正是评估把噪声当作发现的常见方式。
声明切片
切片在 oloproof.yaml 中命名。最常见的一类读取每个用例 metadata 中的某个键：
slices: [metadata.lang, metadata.topic]
min_slice_support: 30{"id": "c000", "input": {"lang": "en", "topic": "billing", "hard": false, "n": 0}, "expected": {"label": "yes"}, "metadata": {"lang": "en", "topic": "billing"}}该文件接受的所有切片种类：
| 切片 | 用例分组依据 | 前提 |
|---|---|---|
| metadata.<key> | 用例 metadata 中该键的值 | 用例上存在该键 |
| relevant_position | 第一个相关段落被检索到的位置：top_k、beyond_top_k 或 not_retrieved | RAG 系统以及 expected.relevant |
| context_truncated | token 预算是否从上下文中丢弃了段落 | 带有 token_budget 的 RAG 系统 |
| first_tool | 智能体首先调用的工具 | 一条 agent_trajectory/v1 记录 |
| repeated_action | 智能体是否重复了完全相同的调用 | 一条 agent_trajectory/v1 记录 |
| route | 掌握过控制权的智能体，写作 triage>billing | 步骤中注明其智能体的 agent_trajectory/v1 记录 |
| trajectory_length:4,8 | 步数，按你声明的边界分桶 | 一条 agent_trajectory/v1 记录 |
| confidence:0.5 | 模型给该行的分数，按你声明的边界分段 | 一个 predictive: 块 |
文件无法识别的名称会在任何内容运行之前被拒绝：
Configuration error: unknown slice 'lang'; declare metadata.<key>, relevant_position, context_truncated, first_tool, repeated_action, route, trajectory_length:<bounds> or confidence:<bounds>切片只做报告，从不参与门控
运行会在单独的标题下打印其切片。一个修好了英语却弄坏了法语的候选：
│ metadata.lang=de │ ok │ 100.0% │ [95.4%, 100.0%] │ 80 / 80 observed · 0 missing · 0 excluded · exploratory · support 80 │
│ metadata.lang=en │ ok │ 100.0% │ [95.4%, 100.0%] │ 80 / 80 observed · 0 missing · 0 excluded · exploratory · support 80 │
│ metadata.lang=fr │ ok │ 32.5% │ [22.4%, 43.9%] │ 26 / 80 observed · 0 missing · 0 excluded · exploratory · support 80 │
│ metadata.topic=account │ ok │ 88.3% │ [81.2%, 93.5%] │ 106 / 120 observed · 0 missing · 0 excluded · exploratory · support 120 │
│ metadata.topic=billing │ ok │ 66.7% │ [57.4%, 75.1%] │ 80 / 120 observed · 0 missing · 0 excluded · exploratory · support 120 │该表的标题是 Slices (exploratory; never gated)，而且说到做到。那次运行的整体比率为 77.5%，其门控允许了发布，而法语只有这个值的三分之一左右。针对运行的发布规则指定的是整体指标，作用范围限于单个运行某个切片的规则会被拒绝：
Configuration error: release rule 'fr-floor' targets slice/metadata.lang=fr; a run's slice results are exploratory, and only a comparison rule gates a slice, inside a family with a correction这是第一条纪律。一个被切成十种切片的测试套件有十次机会偶然显示出差异，读取切片的门控会因噪声而阻止发布。切片的用途是找到该往哪里看。
哪些切片显得突出
运行还会将其切片与运行自身的比率进行排序，并标记出存在差异的切片，所用的是 Benjamini-Yekutieli 程序，无论切片之间如何重叠，错误标记的比例都能得到控制。终端的切片表在 BY mark 列中显示每个切片的结果——marked、not marked，或者对于太稀疏而无法检验的切片留空——并附上用于排序的 p 值。这些结果也会存储在运行中，并导出到其证据包的 run.json 中：
jq -c '.run.slice_metrics[] | {scope, estimate, fdr_marked, fdr_p_value}' .oloproof/bundles/RUN_ID/run.json{"scope":"slice/metadata.lang=de","estimate":1.0,"fdr_marked":true,"fdr_p_value":0.0001}
{"scope":"slice/metadata.lang=en","estimate":1.0,"fdr_marked":true,"fdr_p_value":0.0001}
{"scope":"slice/metadata.lang=fr","estimate":0.325,"fdr_marked":true,"fdr_p_value":0.0001}
{"scope":"slice/metadata.topic=account","estimate":0.8833333333333333,"fdr_marked":true,"fdr_p_value":0.0037003540039062493}
{"scope":"slice/metadata.topic=billing","estimate":0.6666666666666666,"fdr_marked":true,"fdr_p_value":0.008582189941406249}未被标记的切片并不表示已证明它与运行一致；它只是没有被标出。标记用于引导注意力，不会进入任何门控。
支持度
min_slice_support 是切片获得区间所需的最少用例数。低于该值时，会显示估计值，但不给出区间。以下来自一个将其设为 4 的项目：
│ metadata.topic=account │ answer_correct │ 100.0% │ │ 1 / 1 observed · 0 missing · 0 excluded · exploratory · support 1 │
│ │ │ │ │ < 4, no interval │默认值为 30。调低该值的小型测试套件会在稀疏切片上得到区间，而这些区间宽到足以自己说明问题。
在比较中对切片进行门控
如果某个切片确实不能退化，可以将比较规则的作用范围限定在该切片上。作用范围是在切片名称前加上 slice/ 前缀，并且规则要在读取任何证据之前声明它所需的支持度：
version: 1
rules:
- {id: overall, metric: ok, kind: non_inferiority, margin: 0.05}
- {id: fr-not-worse, metric: ok, kind: non_inferiority, margin: 0.05, scope: slice/metadata.lang=fr, min_support: 30}
- {id: en-not-worse, metric: ok, kind: non_inferiority, margin: 0.05, scope: slice/metadata.lang=en, min_support: 30}
- {id: de-not-worse, metric: ok, kind: non_inferiority, margin: 0.05, scope: slice/metadata.lang=de, min_support: 30}
families:
- {id: languages, correction: holm, rules: [fr-not-worse, en-not-worse, de-not-worse]}不写前缀时，作用范围会被拒绝：
Configuration error: rule 'fr-not-worse' has unknown scope 'metadata.lang=fr'; a comparison rule decides on the global scope or on slice/<name>=<value>加上前缀后，上面的候选与其基线比较的结果：
oloproof compare CANDIDATE_RUN_ID BASELINE_RUN_ID --policy compare.yamlComparison sha256:cd2693b505e645919d397e8d7a87eaf0a841ab0c11c0b95484912363381b72e0 of run_01M3C46CYA163JSMSA5QZ6KAZA against run_01M3C46B3PSNP1ZTC8X05AG0GE · 240 paired cases
ok: -5.0 points [-15.2, +4.9] · 240 paired · 0 missing · 0 excluded
ok [slice/metadata.lang=de]: +8.8 points [-0.5, +21.5] · 80 paired · 0 missing · 0 excluded · exploratory · support 80
ok [slice/metadata.lang=en]: +18.8 points [+7.0, +34.4] · 80 paired · 0 missing · 0 excluded · exploratory · support 80
ok [slice/metadata.lang=fr]: -42.5 points [-60.3, -26.8] · 80 paired · 0 missing · 0 excluded · exploratory · support 80
ok [slice/metadata.topic=account]: +7.5 points [+0.9, +17.1] · 120 paired · 0 missing · 0 excluded · exploratory · support 120
ok [slice/metadata.topic=billing]: -17.5 points [-35.0, -0.1] · 120 paired · 0 missing · 0 excluded · exploratory · support 120
Decisions
overall ok non-inferiority, margin 5.0 points INSUFFICIENT_EVIDENCE interval_overlaps_margin
no sample size would make this PASS: the difference itself (-5.0 points) is outside the margin, so more cases would move it toward FAIL
fr-not-worse ok non-inferiority, margin 5.0 points FAIL upper_bound_below_margin
en-not-worse ok non-inferiority, margin 5.0 points PASS lower_bound_above_margin
de-not-worse ok non-inferiority, margin 5.0 points PASS lower_bound_above_margin
Family languages: Holm over 3 rules; adjusted levels 0.01667
Gate: BLOCK (exit 1)这是第二条纪律。作用范围限于切片的规则只有在 families: 条目内才参与门控，该条目列出了其错误失败需要被共同控制的规则。holm 是校正方法：族中失败的规则会在更严格的水平上重新检验，该水平根据族中规则的数量确定，这样三次偶然失败的机会不会累加成三倍的风险。在置信水平为 0.95、共三条规则时，最强的失败在 0.05 / 3 下重新检验，其次在 0.05 / 2 下，最后一个在 0.05 下。只有失败会被重新检验：通过并不是拒绝，不需要校正。这里有一条规则失败，它在 0.01667 下被重新检验，法语仍然失败。整体规则无法作出判定，其下方的建议说明，更多用例只会让它更接近 FAIL。
不属于任何族的切片规则会被拒绝：
Configuration error: slice rule 'fr-not-worse' belongs to no family; slice rules multiply the chances of a false FAIL, so they gate only inside a family with a correction