chore: import zh skill advanced-evaluation

README.wehub.md

@@ -0,0 +1,9 @@
+# WeHub 来源说明
+
+- Skill 名称：`advanced-evaluation`
+- 中文类目：设计并校准 LLM-as-judge 评测 rubric
+- 上游仓库：`muratcankoylan__Agent-Skills-for-Context-Engineering`
+- 上游路径：`skills/advanced-evaluation/SKILL.md`
+- 上游链接：https://github.com/muratcankoylan/Agent-Skills-for-Context-Engineering/blob/HEAD/skills/advanced-evaluation/SKILL.md
+- 本仓库为 WeHub 中文 Skill 汉化包，基于 skill 市场筛选 Top200 清单整理
+- 原作者、版权和许可证信息以上游仓库为准

SKILL.md

@@ -0,0 +1,409 @@
+---
+name: advanced-evaluation
+description: 本技能应用于高级 LLM 评估：LLM 作为裁判系统、直接评分、成对比较、量规校准、评估者偏差缓解、置信度评分以及自动化质量评估。
+---
+
+# 高级评估
+
+本技能涵盖了使用 LLM 作为裁判来评估 LLM 输出的生产级技术。它将来自学术论文、行业实践及实际实施经验的研究成果综合为构建可靠评估系统的可操作模式。
+
+**核心见解**：LLM 作为裁判并非单一技术，而是一系列方法的统称，每种方法适用于不同的评估场景。选择正确的方法并缓解已知偏差是本技能培养的核心能力。
+
+## 何时激活
+
+在以下情况激活本技能：
+
+- 构建针对 LLM 输出的 LLM 作为裁判系统
+- 比较多个模型的回复以选出最佳结果
+- 在评估团队间建立一致的质量标准
+- 调试显示不一致结果的评估系统
+- 设计针对提示词或模型更改的 A/B 测试
+- 专门为 LLM 或人/LLM 混合裁判创建量规
+- 分析自动评估与人工评估之间的相关性
+
+对于其他技能负责的相邻工作，请勿激活本技能：
+- 通用确定性检查、回归套件、生产质量门禁或结果指标：`evaluation`
+- 自主循环治理、锁定量规、回滚或 PR 审批边界：`harness-engineering`
+- 评估工具的工具 API 契约：`tool-design`
+
+## 核心概念
+
+### 评估分类法
+
+根据是否存在真实基准，在两种主要方法之间选择：
+
+**直接评分**——在存在客观标准时使用（事实准确性、指令遵循、毒性）。单个 LLM 按既定量表对一条回复进行评分。对于定义明确的标准，可实现中等到高可靠性。需注意分数校准漂移和不一致的量表解读问题。
+
+**成对比较**——用于主观偏好（语气、风格、说服力）。LLM 比较两条回复并选出较好者。对于主观任务，成对方法通常比开放式直接评分与人类偏好具有更好的相关性（claim-advanced-evaluation-position-swap）。需注意位置偏差和长度偏差。
+
+### 偏差全景
+
+在每个评估系统中缓解以下系统性偏差：
+
+**位置偏差**：处于第一位置的回复会获得更优对待。通过交换位置评估两次，然后应用多数投票或一致性检查来缓解。
+
+**长度偏差**：较长的回复无论质量如何都能获得更高分数。通过显式提示忽略长度并应用长度归一化评分来缓解。
+
+**自我增强偏差**：模型对自己输出的评分更高。通过使用不同的模型进行生成和评估来缓解。
+
+**冗长偏差**：即使不必要，过度详细的回复也能获得更高分数。通过使用惩罚无关细节的特定标准量规来缓解。
+
+**权威偏差**：无论准确性如何，自信的语气能获得更高分数。通过要求引用证据并增加事实核查层来缓解。
+
+### 指标选择框架
+
+将指标与评估任务结构匹配：
+
+| 任务类型 | 主要指标 | 次要指标 |
+|-----------|-----------------|-------------------|
+| 二分类（通过/不通过） | 召回率、精确率、F1 | Cohen's kappa |
+| 序数量表（1-5 评分） | Spearman's rho、Kendall's tau | Cohen's kappa（加权） |
+| 成对偏好 | 一致率、位置一致性 | 置信度校准 |
+| 多标签 | 宏平均 F1、微平均 F1 | 每个标签的精确率/召回率 |
+
+优先关注系统性不一致模式，而非绝对一致率，因为在特定标准上与人类持续不一致的裁判，比存在随机噪声的裁判问题更严重。
+
+## 评估方法
+
+### 直接评分实现
+
+通过三个组件构建直接评分：清晰的标准、校准的量表以及结构化输出格式。
+
+**标准定义模式**：
+```
+Criterion: [名称]
+Description: [该标准衡量什么]
+Weight: [相对重要性，0-1]
+```
+
+**量表校准**——根据量规详细程度选择量表粒度：
+- 1-3：带中性选项的二分类，认知负荷最低
+- 1-5：标准 Likert 量表，粒度和可靠性的最佳平衡
+- 1-10：仅在有详细逐级量规时使用，因为校准更困难
+
+**直接评分提示词结构**：
+```
+You are an expert evaluator assessing response quality.
+
+## Task
+Evaluate the following response against each criterion.
+
+## Original Prompt
+{prompt}
+
+## Response to Evaluate
+{response}
+
+## Criteria
+{for each criterion: name, description, weight}
+
+## Instructions
+For each criterion:
+1. Find specific evidence in the response
+2. Score according to the rubric (1-{max} scale)
+3. Justify your score with evidence
+4. Suggest one specific improvement
+
+## Output Format
+Respond with structured JSON containing scores, justifications, and summary.
+```
+
+在评分提示词中要求在分数之前提供证据，以便裁判在给出数字之前必须将其判断锚定在可观察到的输出特征上。
+
+### 成对比较实现
+
+在每个成对评估中应用位置偏差缓解：
+
+1. 首先运行确定性前置检查：两个候选回复必须满足相同的 schema、来源证据要求和范围约束。
+2. 第一次裁判：回复 A 在第一位置，回复 B 在第二位置。
+3. 第二次裁判：回复 B 在第一位置，回复 A 在第二位置。
+4. 一致性检查：如果两次判定的胜者不一致，返回平局并降低置信度。
+5. 最终裁决：一致的胜者，带平均置信度和明确的破平理由。
+
+**成对比较提示词结构**：
+```
+You are an expert evaluator comparing two AI responses.
+
+## Critical Instructions
+- Do NOT prefer responses because they are longer
+- Do NOT prefer responses based on position (first vs second)
+- Focus ONLY on quality according to the specified criteria
+- Ties are acceptable when responses are genuinely equivalent
+
+## Original Prompt
+{prompt}
+
+## Response A
+{response_a}
+
+## Response B
+{response_b}
+
+## Comparison Criteria
+{criteria list}
+
+## Instructions
+1. Analyze each response independently first
+2. Compare them on each criterion
+3. Determine overall winner with confidence level
+
+## Output Format
+JSON with per-criterion comparison, overall winner, confidence (0-1), and reasoning.
+```
+
+**置信度校准**——将置信度映射到位置一致性：
+- 两次判定一致：置信度 = 各次置信度的平均值
+- 两次判定不一致：置信度 = 0.5，裁决 = 平局
+
+### 量规生成
+
+生成量规可减少与开放式评分相比的评估方差。除非在目标评估集上测量过，否则将确切的方差减少视为特定于工作负载。
+
+**包含以下量规组件**：
+1. **等级描述**：每个分数等级的清晰边界
+2. **特征**：定义每个等级的可观察特征
+3. **示例**：每个等级的代表性文本（可选但有价值）
+4. **边缘情况**：针对模糊情况的指引
+5. **评分指南**：一致应用的一般原则
+
+**根据用例设置严格度校准**：
+- **宽松**：通过门槛较低，适合鼓励迭代
+- **均衡**：典型的生产环境期望
+- **严格**：对安全关键或高风险的评估采用高标准
+
+使量规适应领域——使用领域特定术语。代码可读性量规提及变量、函数和注释。医学准确性量规引用临床术语和证据标准。
+
+## 实践指南
+
+### 评估流水线设计
+
+使用以下层次构建生产级评估系统：标准加载器（量规 + 权重）-> 主要评分器（直接或成对）-> 偏差缓解（位置交换等）-> 置信度评分（校准）-> 输出（分数 + 理由 + 置信度）。参见[评估流水线图](./references/evaluation-pipeline.md)获取完整可视化布局。
+
+### 决策框架：直接 vs. 成对
+
+应用以下决策树：
+
+```
+是否存在客观真实基准？
++-- 是 -> 直接评分
+|   示例：事实准确性、指令遵循、格式合规
+|
++-- 否 -> 是偏好或质量判断吗？
+    +-- 是 -> 成对比较
+    |   示例：语气、风格、说服力、创造力
+    |
+    +-- 否 -> 考虑基于参考的评估
+        示例：摘要（与原文比较）、翻译（与参考译文比较）
+```
+
+### 扩展评估
+
+对于高流量评估，应用以下策略之一：
+
+1. **LLM 面板（PoLL）**：使用多个模型作为裁判并汇总投票，以减少单个模型偏差。成本更高，但对于高风险决策更可靠。
+
+2. **分层评估**：使用快速廉价的模型进行筛选，使用昂贵模型处理边缘情况。需要对筛选阈值进行校准。
+
+3. **人在回路中**：自动化处理明确案例，将低置信度决策路由到人工审核。设计反馈循环以逐步改进自动评估。
+
+## 示例
+
+### 示例 1：准确性直接评分
+
+**输入**：
+```
+Prompt: "What causes seasons on Earth?"
+Response: "Seasons are caused by Earth's tilted axis. As Earth orbits the Sun,
+different hemispheres receive more direct sunlight at different times of year."
+Criterion: Factual Accuracy (weight: 1.0)
+Scale: 1-5
+```
+
+**输出**：
+```json
+{
+  "criterion": "Factual Accuracy",
+  "score": 5,
+  "evidence": [
+    "Correctly identifies axial tilt as primary cause",
+    "Correctly explains differential sunlight by hemisphere",
+    "No factual errors present"
+  ],
+  "justification": "Response accurately explains the cause of seasons with correct
+scientific reasoning. Both the axial tilt and its effect on sunlight distribution
+are correctly described.",
+  "improvement": "Could add the specific tilt angle (23.5 degrees) for completeness."
+}
+```
+
+### 示例 2：带位置交换的成对比较
+
+**输入**：
+```
+Prompt: "Explain machine learning to a beginner"
+Response A: [Technical explanation with jargon]
+Response B: [Simple analogy-based explanation]
+Criteria: ["clarity", "accessibility"]
+```
+
+**第一次判定（A 在前）**：
+```json
+{ "winner": "B", "confidence": 0.8 }
+```
+
+**第二次判定（B 在前）**：
+```json
+{ "winner": "A", "confidence": 0.6 }
+```
+（注意：胜者是 A，因为 B 处于第一位置）
+
+**映射后的第二次判定**：
+```json
+{ "winner": "B", "confidence": 0.6 }
+```
+
+**最终结果**：
+```json
+{
+  "winner": "B",
+  "confidence": 0.7,
+  "positionConsistency": {
+    "consistent": true,
+    "firstPassWinner": "B",
+    "secondPassWinner": "B"
+  }
+}
+```
+
+### 示例 3：量规生成
+
+**输入**：
+```
+criterionName: "Code Readability"
+criterionDescription: "How easy the code is to understand and maintain"
+domain: "software engineering"
+scale: "1-5"
+strictness: "balanced"
+```
+
+**输出**（节选）：
+```json
+{
+  "levels": [
+    {
+      "score": 1,
+      "label": "Poor",
+      "description": "Code is difficult to understand without significant effort",
+      "characteristics": [
+        "No meaningful variable or function names",
+        "No comments or documentation",
+        "Deeply nested or convoluted logic"
+      ]
+    },
+    {
+      "score": 3,
+      "label": "Adequate",
+      "description": "Code is understandable with some effort",
+      "characteristics": [
+        "Most variables have meaningful names",
+        "Basic comments present for complex sections",
+        "Logic is followable but could be cleaner"
+      ]
+    },
+    {
+      "score": 5,
+      "label": "Excellent",
+      "description": "Code is immediately clear and maintainable",
+      "characteristics": [
+        "All names are descriptive and consistent",
+        "Comprehensive documentation",
+        "Clean, modular structure"
+      ]
+    }
+  ],
+  "edgeCases": [
+    {
+      "situation": "Code is well-structured but uses domain-specific abbreviations",
+      "guidance": "Score based on readability for domain experts, not general audience"
+    }
+  ]
+}
+```
+
+## 指南
+
+1. **始终在分数之前要求提供证据**——证据优先的提示词使判断更易于审计，并减少无依据的数字评分
+
+2. **在成对比较中始终交换位置**——单次判定会被位置偏差污染
+
+3. **使量表粒度与量规详细程度匹配**——没有详细等级描述时不要使用 1-10 分制
+
+4. **区分客观标准和主观标准**——客观标准使用直接评分，主观标准使用成对比较
+
+5. **包含置信度分数**——根据位置一致性和证据强度进行校准
+
+6. **显式定义边缘情况**——模糊情况导致的评估方差最大
+
+7. **使用领域特定的量规**——通用量规会产生通用（不太有用）的评估结果
+
+8. **针对人工判断进行验证**——自动评估只有在与人工评估相关时才有价值
+
+9. **监控系统性偏差**——按标准、回复类型、模型追踪不一致模式
+
+10. **为迭代而设计**——评估系统通过反馈循环不断改进
+
+## 注意事项
+
+1. **评分没有理由**：分数缺乏依据，难以调试。始终要求在分数之前提供基于证据的理由。
+
+2. **单次成对比较**：位置未交换时，位置偏差会污染结果。始终交换位置评估两次并检查一致性。
+
+3. **标准过载**：同时衡量多个方面的标准会产生不可靠的分数。强制一个标准 = 一个可衡量的方面。
+
+4. **缺少边缘情况指引**：没有显式指令时，评估者处理模糊情况的方式不一致。在量规中包含边缘情况及其明确解决规则。
+
+5. **忽略置信度校准**：高置信度的错误判断比低置信度的更糟糕。将置信度校准到位置一致性和证据强度。
+
+6. **量规漂移**：随着质量标准演变或模型能力提升，量规会变得不准确。安排定期量规审查，并对照新的人工标注示例重新锚定分数等级。
+
+7. **评估提示词敏感性**：评估提示词中的微小措辞变化可能导致分数大幅波动。对评估提示词进行版本控制，并在部署提示词更改前运行回归测试。
+
+8. **未控制的长度偏差**：即使简洁性更受偏好，较长的回复也会系统性地获得更高分数。在评估提示词中添加显式的长度中立指令，并通过长度控制的测试对进行验证。
+
+## 集成
+
+本技能负责裁判设计和偏差缓解。相邻技能负责更广泛的质量门禁和基础设施：
+
+- `evaluation`：通用确定性检查、回归套件、质量门禁和生产监控
+- `context-fundamentals`：裁判提示词的上下文结构
+- `tool-design`：评估工具的 schemas 和错误处理
+- `context-optimization`：高流量评估的令牌和延迟效率
+- `harness-engineering`：自主循环的锁定评估器表面和治理
+
+## 参考资料
+
+内部参考：
+- [LLM 作为裁判实现模式](./references/implementation-patterns.md) - 何时阅读：从头构建评估流水线或将 LLM 裁判集成到 CI/CD 中时
+- [偏差缓解技术](./references/bias-mitigation.md) - 何时阅读：评估结果显示不一致或可疑的评分模式时
+- [指标选择指南](./references/metrics-guide.md) - 何时阅读：选择统计指标以验证评估可靠性时
+- [评估流水线图](./references/evaluation-pipeline.md) - 何时阅读：设计多阶段评估系统的架构时
+
+外部研究：
+- [Eugene Yan: Evaluating the Effectiveness of LLM-Evaluators](https://eugeneyan.com/writing/llm-evaluators/) - 何时阅读：调研 LLM 评估最新技术现状时
+- [Judging LLM-as-a-Judge (Zheng et al., 2023)](https://arxiv.org/abs/2306.05685) - 何时阅读：理解位置偏差和 MT-Bench 方法论时
+- [G-Eval: NLG Evaluation using GPT-4 (Liu et al., 2023)](https://arxiv.org/abs/2303.16634) - 何时阅读：实现思维链评估评分时
+- [Large Language Models are not Fair Evaluators (Wang et al., 2023)](https://arxiv.org/abs/2305.17926) - 何时阅读：诊断评估输出中的系统性偏差时
+
+本集合中的相关技能：
+- evaluation——基础评估概念
+- context-fundamentals——评估提示词的上下文结构
+- tool-design——构建评估工具
+
+---
+
+## 技能元数据
+
+**创建日期**：2025-12-24
+**最后更新**：2026-05-15
+**作者**：Agent Skills for Context Engineering 贡献者
+**版本**：2.1.0

references/bias-mitigation.md

@@ -0,0 +1,286 @@
+# LLM 评估中的偏差缓解技术
+
+本文档详细介绍了用于减轻 LLM 作为评判者（LLM-as-a-Judge）系统中已知偏差的具体技术。
+
+## 位置偏差
+
+### 问题描述
+
+在成对比较中，LLM 会系统性地偏好处于特定位置的回答。研究表明：
+- GPT 存在轻微的首位偏差（在平局情况下，约 55% 的偏好倾向于第一个位置）
+- Claude 表现出类似的模式
+- 较小的模型通常表现出更强的偏差
+
+### 缓解措施：位置交换协议
+
+```python
+async def position_swap_comparison(response_a, response_b, prompt, criteria):
+    # 第 1 轮：原始顺序
+    result_ab = await compare(response_a, response_b, prompt, criteria)
+    
+    # 第 2 轮：交换顺序
+    result_ba = await compare(response_b, response_a, prompt, criteria)
+    
+    # 映射第二轮结果（第二轮中 A 在第二位 → 对应第一轮中 B 在第一位）
+    result_ba_mapped = {
+        'winner': {'A': 'B', 'B': 'A', 'TIE': 'TIE'}[result_ba['winner']],
+        'confidence': result_ba['confidence']
+    }
+    
+    # 一致性检查
+    if result_ab['winner'] == result_ba_mapped['winner']:
+        return {
+            'winner': result_ab['winner'],
+            'confidence': (result_ab['confidence'] + result_ba_mapped['confidence']) / 2,
+            'position_consistent': True
+        }
+    else:
+        # 不一致表明位置偏差起了作用
+        return {
+            'winner': 'TIE',
+            'confidence': 0.5,
+            'position_consistent': False,
+            'bias_detected': True
+        }
+```
+
+### 替代方案：多次打乱
+
+为了更高的可靠性，可以使用多种位置顺序：
+
+```python
+async def multi_shuffle_comparison(response_a, response_b, prompt, criteria, n_shuffles=3):
+    results = []
+    for i in range(n_shuffles):
+        if i % 2 == 0:
+            r = await compare(response_a, response_b, prompt, criteria)
+        else:
+            r = await compare(response_b, response_a, prompt, criteria)
+            r['winner'] = {'A': 'B', 'B': 'A', 'TIE': 'TIE'}[r['winner']]
+        results.append(r)
+    
+    # 多数投票
+    winners = [r['winner'] for r in results]
+    final_winner = max(set(winners), key=winners.count)
+    agreement = winners.count(final_winner) / len(winners)
+    
+    return {
+        'winner': final_winner,
+        'confidence': agreement,
+        'n_shuffles': n_shuffles
+    }
+```
+
+## 长度偏差
+
+### 问题描述
+
+LLM 倾向于给较长的回答打更高的分，而不管其质量如何。这表现为：
+- 冗长的回答获得虚高的分数
+- 简洁但完整的回答被扣分
+- 填充和重复反而得到奖励
+
+### 缓解措施：显式提示
+
+在提示词中包含反长度偏差的指令：
+
+```
+关键评估指南：
+- 不要因为回答更长就给予偏好
+- 简洁、完整的回答与详细的回答同样有价值
+- 对不必要的冗长或重复内容进行扣分
+- 关注信息密度，而非字数
+```
+
+### 缓解措施：长度归一化评分
+
+```python
+def length_normalized_score(score, response_length, target_length=500):
+    """根据回答长度调整分数。"""
+    length_ratio = response_length / target_length
+    
+    if length_ratio > 2.0:
+        # 对过长的回答进行扣分
+        penalty = (length_ratio - 2.0) * 0.1
+        return max(score - penalty, 1)
+    elif length_ratio < 0.3:
+        # 对过短的回答进行扣分
+        penalty = (0.3 - length_ratio) * 0.5
+        return max(score - penalty, 1)
+    else:
+        return score
+```
+
+### 缓解措施：独立的长度评判标准
+
+将长度作为一个独立的、显式的评判标准，使其不被隐式奖励：
+
+```python
+criteria = [
+    {"name": "Accuracy", "description": "事实正确性", "weight": 0.4},
+    {"name": "Completeness", "description": "涵盖关键要点", "weight": 0.3},
+    {"name": "Conciseness", "description": "无多余内容", "weight": 0.3}  # 显式指定
+]
+```
+
+## 自我增强偏差
+
+### 问题描述
+
+模型对自己（或类似模型）生成的输出给出的评分，高于对不同模型生成的输出。
+
+### 缓解措施：跨模型评估
+
+使用与生成模型不同的模型家族进行评估：
+
+```python
+def get_evaluator_model(generator_model):
+    """选择评估者以避免自我增强偏差。"""
+    if 'gpt' in generator_model.lower():
+        return 'claude-4-5-sonnet'
+    elif 'claude' in generator_model.lower():
+        return 'gpt-5.2'
+    else:
+        return 'gpt-5.2'  # 默认值
+```
+
+### 缓解措施：盲评
+
+在评估之前从回答中移除模型归属信息：
+
+```python
+def anonymize_response(response, model_name):
+    """移除可识别模型的模式。"""
+    patterns = [
+        f"As {model_name}",
+        "I am an AI",
+        "I don't have personal opinions",
+        # 特定于模型的模式
+    ]
+    anonymized = response
+    for pattern in patterns:
+        anonymized = anonymized.replace(pattern, "[REDACTED]")
+    return anonymized
+```
+
+## 冗长偏差
+
+### 问题描述
+
+详细的解释即使包含了无关或不正确的内容，也会获得更高的分数。
+
+### 缓解措施：相关性加权评分
+
+```python
+async def relevance_weighted_evaluation(response, prompt, criteria):
+    # 首先，评估每个片段的相关性
+    relevance_scores = await assess_relevance(response, prompt)
+    
+    # 按相关性加权评估
+    segments = split_into_segments(response)
+    weighted_scores = []
+    for segment, relevance in zip(segments, relevance_scores):
+        if relevance > 0.5:  # 只计算相关的片段
+            score = await evaluate_segment(segment, prompt, criteria)
+            weighted_scores.append(score * relevance)
+    
+    return sum(weighted_scores) / len(weighted_scores)
+```
+
+### 缓解措施：包含冗长扣分项的评分细则
+
+在评分细则中加入显式的冗长扣分项：
+
+```python
+rubric_levels = [
+    {
+        "score": 5,
+        "description": "完整且简洁。包含所有必要信息，无多余内容。",
+        "characteristics": ["每个句子都有价值", "无重复", "范围适当"]
+    },
+    {
+        "score": 3,
+        "description": "完整但冗长。包含不必要的细节或重复。",
+        "characteristics": ["主要要点已涵盖", "存在一些偏离主题的内容", "可以更简洁"]
+    },
+    # ... 以此类推
+]
+```
+
+## 权威偏差
+
+### 问题描述
+
+自信、权威的语气会获得更高的评分，而不论其准确性如何。
+
+### 缓解措施：证据要求
+
+要求对主张提供明确的证据：
+
+```
+对于回答中的每个主张：
+1. 判断其是否为事实性主张
+2. 注意是否提供了证据或来源
+3. 基于可验证性评分，而非自信程度
+
+重要提示：没有证据的自信主张，其得分不应高于有证据的谨慎主张。
+```
+
+### 缓解措施：事实核查层
+
+在评分之前增加一个事实核查步骤：
+
+```python
+async def fact_checked_evaluation(response, prompt, criteria):
+    # 提取主张
+    claims = await extract_claims(response)
+    
+    # 对每个主张进行事实核查
+    fact_check_results = await asyncio.gather(*[
+        verify_claim(claim) for claim in claims
+    ])
+    
+    # 根据事实核查结果调整分数
+    accuracy_factor = sum(r['verified'] for r in fact_check_results) / len(fact_check_results)
+    
+    base_score = await evaluate(response, prompt, criteria)
+    return base_score * (0.7 + 0.3 * accuracy_factor)  # 至少保留原始分的 70%
+```
+
+## 聚合偏差检测
+
+在生产环境中监控系统性偏差：
+
+```python
+class BiasMonitor:
+    def __init__(self):
+        self.evaluations = []
+    
+    def record(self, evaluation):
+        self.evaluations.append(evaluation)
+    
+    def detect_position_bias(self):
+        """检测第一位置的胜率是否高于预期。"""
+        first_wins = sum(1 for e in self.evaluations if e['first_position_winner'])
+        expected = len(self.evaluations) * 0.5
+        z_score = (first_wins - expected) / (expected * 0.5) ** 0.5
+        return {'bias_detected': abs(z_score) > 2, 'z_score': z_score}
+    
+    def detect_length_bias(self):
+        """检测较长的回答是否得分更高。"""
+        from scipy.stats import spearmanr
+        lengths = [e['response_length'] for e in self.evaluations]
+        scores = [e['score'] for e in self.evaluations]
+        corr, p_value = spearmanr(lengths, scores)
+        return {'bias_detected': corr > 0.3 and p_value < 0.05, 'correlation': corr}
+```
+
+## 汇总表
+
+| 偏差类型 | 主要缓解措施 | 辅助缓解措施 | 检测方法 |
+|----------|-------------|-------------|---------|
+| 位置偏差 | 位置交换 | 多次打乱 | 一致性检查 |
+| 长度偏差 | 显式提示 | 长度归一化 | 长度-分数相关性 |
+| 自我增强偏差 | 跨模型评估 | 匿名化 | 模型对比研究 |
+| 冗长偏差 | 相关性加权 | 评分细则扣分 | 相关性评分 |
+| 权威偏差 | 证据要求 | 事实核查层 | 自信度-准确度相关性 |

references/evaluation-pipeline.md

@@ -0,0 +1,43 @@
+# 评估管线流程图
+
+生产环境评估管线的可视化布局。
+
+```
+┌─────────────────────────────────────────────────┐
+│                  评估管线                        │
+├─────────────────────────────────────────────────┤
+│                                                   │
+│  输入：回复 + 提示 + 上下文                       │
+│           │                                       │
+│           ▼                                       │
+│  ┌─────────────────────┐                         │
+│  │   标准加载器         │ ◄── 评分细则、权重      │
+│  └──────────┬──────────┘                         │
+│             │                                     │
+│             ▼                                     │
+│  ┌─────────────────────┐                         │
+│  │   主评分器           │ ◄── 直接评分或成对比较  │
+│  └──────────┬──────────┘                         │
+│             │                                     │
+│             ▼                                     │
+│  ┌─────────────────────┐                         │
+│  │   偏差缓解           │ ◄── 位置交换等          │
+│  └──────────┬──────────┘                         │
+│             │                                     │
+│             ▼                                     │
+│  ┌─────────────────────┐                         │
+│  │   置信度评分         │ ◄── 校准               │
+│  └──────────┬──────────┘                         │
+│             │                                     │
+│             ▼                                     │
+│  输出：分数 + 理由说明 + 置信度                   │
+│                                                   │
+└─────────────────────────────────────────────────┘
+```
+
+## 管线阶段
+
+1. **标准加载器**：从配置中加载评分细则和标准权重
+2. **主评分器**：执行直接评分或成对比较
+3. **偏差缓解**：运行位置交换、长度归一化及其他去偏处理
+4. **置信度评分**：基于位置一致性和证据强度校准置信度

references/implementation-patterns.md

@@ -0,0 +1,314 @@
+# LLM-as-Judge 实现模式
+
+本文档提供了构建生产级 LLM 评估系统的详细实现模式。
+
+## 模式 1：结构化评估流水线
+
+最可靠的评估系统遵循结构化的流水线来分离关注点：
+
+```
+Input Validation → Criteria Loading → Scoring → Bias Mitigation → Output Formatting
+```
+
+### 输入验证层
+
+在评估开始之前，验证以下内容：
+
+1. **响应存在性**：待评估的响应不能为空
+2. **提示词存在性**：用于提供上下文的原始提示词不能为空
+3. **标准有效性**：至少有一条标准，且包含名称和描述
+4. **权重归一化**：权重之和为 1.0（或对其进行归一化）
+
+```python
+def validate_input(response, prompt, criteria):
+    if not response or not response.strip():
+        raise ValueError("Response cannot be empty")
+    if not prompt or not prompt.strip():
+        raise ValueError("Prompt cannot be empty")
+    if not criteria or len(criteria) == 0:
+        raise ValueError("At least one criterion required")
+    
+    # Normalize weights
+    total_weight = sum(c.get('weight', 1) for c in criteria)
+    for c in criteria:
+        c['weight'] = c.get('weight', 1) / total_weight
+```
+
+### 标准加载层
+
+标准应从配置中加载，而非硬编码：
+
+```python
+class CriteriaLoader:
+    def __init__(self, rubric_path=None):
+        self.rubrics = self._load_rubrics(rubric_path)
+    
+    def get_criteria(self, task_type):
+        return self.rubrics.get(task_type, self.default_criteria)
+    
+    def get_rubric(self, criterion_name):
+        return self.rubrics.get(criterion_name, {}).get('levels', [])
+```
+
+### 评分层
+
+评分层负责实际的 LLM 调用：
+
+```python
+async def score_response(response, prompt, criteria, rubric, model):
+    system_prompt = build_system_prompt(criteria, rubric)
+    user_prompt = build_user_prompt(response, prompt, criteria)
+    
+    result = await generate_text(
+        model=model,
+        system=system_prompt,
+        prompt=user_prompt,
+        temperature=0.3  # Lower temperature for consistency
+    )
+    
+    return parse_scores(result.text)
+```
+
+### 偏差缓解层
+
+对于成对比较，始终包含位置交换：
+
+```python
+async def compare_with_bias_mitigation(response_a, response_b, prompt, criteria, model):
+    # First pass: A first
+    pass1 = await compare_pair(response_a, response_b, prompt, criteria, model)
+    
+    # Second pass: B first
+    pass2 = await compare_pair(response_b, response_a, prompt, criteria, model)
+    
+    # Map pass2 winner back
+    pass2_mapped = map_winner(pass2.winner)  # A→B, B→A, TIE→TIE
+    
+    # Check consistency
+    if pass1.winner == pass2_mapped:
+        return {
+            'winner': pass1.winner,
+            'confidence': (pass1.confidence + pass2.confidence) / 2,
+            'consistent': True
+        }
+    else:
+        return {
+            'winner': 'TIE',
+            'confidence': 0.5,
+            'consistent': False
+        }
+```
+
+## 模式 2：分层评估
+
+对于复杂的评估，使用分层方法：
+
+```
+Quick Screen (cheap model) → Detailed Evaluation (expensive model) → Human Review (edge cases)
+```
+
+### 快速筛选实现
+
+```python
+async def quick_screen(response, prompt, threshold=0.7):
+    """Fast, cheap screening for obvious passes/fails."""
+    result = await generate_text(
+        model='gpt-5.2',  # Cheaper model
+        prompt=f"Rate 0-1 if this response adequately addresses the prompt:\n\nPrompt: {prompt}\n\nResponse: {response}",
+        temperature=0
+    )
+    score = float(result.text.strip())
+    return score, score > threshold
+```
+
+### 详细评估
+
+```python
+async def detailed_evaluation(response, prompt, criteria):
+    """Full evaluation for borderline or important cases."""
+    result = await generate_text(
+        model='gpt-5.2',  # More capable model
+        system=DETAILED_EVALUATION_PROMPT,
+        prompt=build_detailed_prompt(response, prompt, criteria),
+        temperature=0.3
+    )
+    return parse_detailed_scores(result.text)
+```
+
+## 模式 3：LLM 评委专家组（PoLL）
+
+对于高风险评估，使用多个模型：
+
+```python
+async def poll_evaluation(response, prompt, criteria, models):
+    """Aggregate judgments from multiple LLM judges."""
+    results = await asyncio.gather(*[
+        score_with_model(response, prompt, criteria, model)
+        for model in models
+    ])
+    
+    # Aggregate scores
+    aggregated = aggregate_scores(results)
+    
+    # Calculate agreement
+    agreement = calculate_agreement(results)
+    
+    return {
+        'scores': aggregated,
+        'agreement': agreement,
+        'individual_results': results
+    }
+
+def aggregate_scores(results):
+    """Aggregate scores using median (robust to outliers)."""
+    scores = {}
+    for criterion in results[0]['scores'].keys():
+        criterion_scores = [r['scores'][criterion] for r in results]
+        scores[criterion] = {
+            'score': statistics.median(criterion_scores),
+            'std': statistics.stdev(criterion_scores) if len(criterion_scores) > 1 else 0
+        }
+    return scores
+```
+
+## 模式 4：置信度校准
+
+置信度得分应与实际可靠性相匹配：
+
+```python
+def calibrate_confidence(raw_confidence, position_consistent, evidence_count):
+    """Calibrate confidence based on multiple signals."""
+    
+    # Base confidence from model output
+    calibrated = raw_confidence
+    
+    # Position consistency is a strong signal
+    if not position_consistent:
+        calibrated *= 0.6  # Significant reduction
+    
+    # More evidence = higher confidence
+    evidence_factor = min(evidence_count / 3, 1.0)  # Cap at 3 pieces
+    calibrated *= (0.7 + 0.3 * evidence_factor)
+    
+    return min(calibrated, 0.99)  # Never 100% confident
+```
+
+## 模式 5：输出格式化
+
+始终使用一致的 Schema 返回结构化输出：
+
+```python
+@dataclass
+class ScoreResult:
+    criterion: str
+    score: float
+    max_score: float
+    justification: str
+    evidence: List[str]
+    improvement: str
+
+@dataclass
+class EvaluationResult:
+    success: bool
+    scores: List[ScoreResult]
+    overall_score: float
+    weighted_score: float
+    summary: Dict[str, Any]
+    metadata: Dict[str, Any]
+
+def format_output(scores, metadata) -> EvaluationResult:
+    """Format evaluation results consistently."""
+    return EvaluationResult(
+        success=True,
+        scores=scores,
+        overall_score=sum(s.score for s in scores) / len(scores),
+        weighted_score=calculate_weighted_score(scores),
+        summary=generate_summary(scores),
+        metadata=metadata
+    )
+```
+
+## 错误处理模式
+
+### 优雅降级
+
+```python
+async def evaluate_with_fallback(response, prompt, criteria):
+    try:
+        return await full_evaluation(response, prompt, criteria)
+    except RateLimitError:
+        # Fall back to simpler evaluation
+        return await simple_evaluation(response, prompt, criteria)
+    except ParseError as e:
+        # Return partial results with error flag
+        return {
+            'success': False,
+            'partial_results': e.partial_data,
+            'error': str(e)
+        }
+```
+
+### 重试逻辑
+
+```python
+async def evaluate_with_retry(response, prompt, criteria, max_retries=3):
+    for attempt in range(max_retries):
+        try:
+            result = await evaluate(response, prompt, criteria)
+            if is_valid_result(result):
+                return result
+        except TransientError:
+            await asyncio.sleep(2 ** attempt)  # Exponential backoff
+    
+    raise EvaluationError("Max retries exceeded")
+```
+
+## 测试模式
+
+### 解析单元测试
+
+```python
+def test_score_parsing():
+    raw_output = '{"scores": [{"criterion": "Accuracy", "score": 4}]}'
+    result = parse_scores(raw_output)
+    assert result.scores[0].criterion == "Accuracy"
+    assert result.scores[0].score == 4
+
+def test_malformed_output():
+    raw_output = 'Invalid JSON'
+    with pytest.raises(ParseError):
+        parse_scores(raw_output)
+```
+
+### 集成测试（使用真实 API）
+
+```python
+@pytest.mark.integration
+async def test_full_evaluation_pipeline():
+    result = await evaluate(
+        response="Water boils at 100°C at sea level.",
+        prompt="At what temperature does water boil?",
+        criteria=[{"name": "Accuracy", "description": "Factual correctness", "weight": 1}]
+    )
+    
+    assert result.success
+    assert len(result.scores) == 1
+    assert result.scores[0].score >= 4  # Should score high for accurate response
+```
+
+### 偏差检测测试
+
+```python
+async def test_position_bias_mitigation():
+    # Same response in both positions should tie
+    result = await compare(
+        response_a="Same response",
+        response_b="Same response",
+        prompt="Test prompt",
+        criteria=["quality"],
+        swap_positions=True
+    )
+    
+    assert result.winner == "TIE"
+    assert result.consistent == True
+```

references/metrics-guide.md

@@ -0,0 +1,336 @@
+---
+name: metric-selection-guide-llm-evaluation
+description: LLM 评估指标选择指南
+metadata:
+  type: reference
+---
+
+# LLM 评估指标选择指南
+
+本文档为不同评估场景下的指标选择提供参考。
+
+## 指标类别
+
+### 分类指标
+
+适用于二元或多分类评估任务（通过/不通过，正确/不正确）。
+
+#### 精确率
+
+```
+Precision = True Positives / (True Positives + False Positives)
+```
+
+**解释**：在被评判为「好」的所有回答中，实际好的比例是多少？
+
+**适用场景**：误报（假阳性）代价较高时（例如批准不安全的内容）
+
+```python
+def precision(predictions, ground_truth):
+    true_positives = sum(1 for p, g in zip(predictions, ground_truth) if p == 1 and g == 1)
+    predicted_positives = sum(predictions)
+    return true_positives / predicted_positives if predicted_positives > 0 else 0
+```
+
+#### 召回率
+
+```
+Recall = True Positives / (True Positives + False Negatives)
+```
+
+**解释**：在所有实际好的回答中，评判者识别出了多少？
+
+**适用场景**：漏报（假阴性）代价较高时（例如过滤中遗漏了优质内容）
+
+```python
+def recall(predictions, ground_truth):
+    true_positives = sum(1 for p, g in zip(predictions, ground_truth) if p == 1 and g == 1)
+    actual_positives = sum(ground_truth)
+    return true_positives / actual_positives if actual_positives > 0 else 0
+```
+
+#### F1 分数
+
+```
+F1 = 2 * (Precision * Recall) / (Precision + Recall)
+```
+
+**解释**：精确率与召回率的调和平均数
+
+**适用场景**：需要一个能平衡两者的单一数值时
+
+```python
+def f1_score(predictions, ground_truth):
+    p = precision(predictions, ground_truth)
+    r = recall(predictions, ground_truth)
+    return 2 * p * r / (p + r) if (p + r) > 0 else 0
+```
+
+### 一致性指标
+
+适用于比较自动化评估与人工判断。
+
+#### Cohen's Kappa（κ 系数）
+
+```
+κ = (Observed Agreement - Expected Agreement) / (1 - Expected Agreement)
+```
+
+**解释**：排除了随机一致性的实际一致性
+- κ > 0.8：几乎完全一致
+- κ 0.6–0.8：高度一致
+- κ 0.4–0.6：中等一致
+- κ < 0.4：一致性较差
+
+**适用场景**：二元或分类判断
+
+```python
+def cohens_kappa(judge1, judge2):
+    from sklearn.metrics import cohen_kappa_score
+    return cohen_kappa_score(judge1, judge2)
+```
+
+#### 加权 Kappa
+
+适用于不一致严重程度有差别的有序量表：
+
+```python
+def weighted_kappa(judge1, judge2):
+    from sklearn.metrics import cohen_kappa_score
+    return cohen_kappa_score(judge1, judge2, weights='quadratic')
+```
+
+**解释**：对较大分歧的惩罚力度高于小分歧
+
+### 相关性指标
+
+适用于有序/连续评分。
+
+#### Spearman 秩相关系数（ρ）
+
+**解释**：衡量排名之间的相关性，而非绝对值
+- ρ > 0.9：非常强的相关性
+- ρ 0.7–0.9：强相关性
+- ρ 0.5–0.7：中等相关性
+- ρ < 0.5：弱相关性
+
+**适用场景**：顺序比具体数值更重要时
+
+```python
+def spearmans_rho(scores1, scores2):
+    from scipy.stats import spearmanr
+    rho, p_value = spearmanr(scores1, scores2)
+    return {'rho': rho, 'p_value': p_value}
+```
+
+#### Kendall 秩相关系数（τ）
+
+**解释**：与 Spearman 类似，但基于配对一致性
+
+**适用场景**：数据中存在大量并列值时
+
+```python
+def kendalls_tau(scores1, scores2):
+    from scipy.stats import kendalltau
+    tau, p_value = kendalltau(scores1, scores2)
+    return {'tau': tau, 'p_value': p_value}
+```
+
+#### Pearson 相关系数（r）
+
+**解释**：评分之间的线性相关性
+
+**适用场景**：评分的实际数值有意义，而不仅仅是排序时
+
+```python
+def pearsons_r(scores1, scores2):
+    from scipy.stats import pearsonr
+    r, p_value = pearsonr(scores1, scores2)
+    return {'r': r, 'p_value': p_value}
+```
+
+### 成对比较指标
+
+#### 一致率
+
+```
+Agreement = (Matching Decisions) / (Total Comparisons)
+```
+
+**解释**：简单的百分比一致性
+
+```python
+def pairwise_agreement(decisions1, decisions2):
+    matches = sum(1 for d1, d2 in zip(decisions1, decisions2) if d1 == d2)
+    return matches / len(decisions1)
+```
+
+#### 位置一致性
+
+```
+Consistency = (Consistent across position swaps) / (Total comparisons)
+```
+
+**解释**：交换位置后，判断结果发生变化的频率有多高？
+
+```python
+def position_consistency(results):
+    consistent = sum(1 for r in results if r['position_consistent'])
+    return consistent / len(results)
+```
+
+## 选择决策树
+
+```
+评估任务的类型是什么？
+│
+├── 二元分类（通过/不通过）
+│   └── 使用：Precision、Recall、F1、Cohen's κ
+│
+├── 有序量表（1–5 评分）
+│   ├── 与人工判断比较？
+│   │   └── 使用：Spearman's ρ、Weighted κ
+│   └── 比较两个自动化评判？
+│       └── 使用：Kendall's τ、Spearman's ρ
+│
+├── 成对偏好
+│   └── 使用：一致率、位置一致性
+│
+└── 多标签分类
+    └── 使用：Macro-F1、Micro-F1、逐标签指标
+```
+
+## 按使用场景选择指标
+
+### 使用场景 1：验证自动化评估
+
+**目标**：确保自动化评估与人工判断相关
+
+**推荐指标**：
+1. 主要指标：Spearman's ρ（有序量表）或 Cohen's κ（分类）
+2. 次要指标：逐标准一致性
+3. 诊断指标：混淆矩阵（用于系统误差分析）
+
+```python
+def validate_automated_eval(automated_scores, human_scores, criteria):
+    results = {}
+    
+    # Overall correlation
+    results['overall_spearman'] = spearmans_rho(automated_scores, human_scores)
+    
+    # Per-criterion agreement
+    for criterion in criteria:
+        auto_crit = [s[criterion] for s in automated_scores]
+        human_crit = [s[criterion] for s in human_scores]
+        results[f'{criterion}_spearman'] = spearmans_rho(auto_crit, human_crit)
+    
+    return results
+```
+
+### 使用场景 2：比较两个模型
+
+**目标**：判断哪个模型产出更优
+
+**推荐指标**：
+1. 主要指标：胜率（来自成对比较）
+2. 次要指标：位置一致性（偏差检查）
+3. 诊断指标：逐标准细分
+
+```python
+def compare_models(model_a_outputs, model_b_outputs, prompts):
+    results = []
+    for a, b, p in zip(model_a_outputs, model_b_outputs, prompts):
+        comparison = await compare_with_position_swap(a, b, p)
+        results.append(comparison)
+    
+    return {
+        'a_wins': sum(1 for r in results if r['winner'] == 'A'),
+        'b_wins': sum(1 for r in results if r['winner'] == 'B'),
+        'ties': sum(1 for r in results if r['winner'] == 'TIE'),
+        'position_consistency': position_consistency(results)
+    }
+```
+
+### 使用场景 3：质量监控
+
+**目标**：追踪评估质量随时间的变化
+
+**推荐指标**：
+1. 主要指标：与人工抽检的滚动一致性
+2. 次要指标：评分分布稳定性
+3. 诊断指标：偏差指标（位置、长度）
+
+```python
+class QualityMonitor:
+    def __init__(self, window_size=100):
+        self.window = deque(maxlen=window_size)
+    
+    def add_evaluation(self, automated, human_spot_check=None):
+        self.window.append({
+            'automated': automated,
+            'human': human_spot_check,
+            'length': len(automated['response'])
+        })
+    
+    def get_metrics(self):
+        # Filter to evaluations with human spot-checks
+        with_human = [e for e in self.window if e['human'] is not None]
+        
+        if len(with_human) < 10:
+            return {'insufficient_data': True}
+        
+        auto_scores = [e['automated']['score'] for e in with_human]
+        human_scores = [e['human']['score'] for e in with_human]
+        
+        return {
+            'correlation': spearmans_rho(auto_scores, human_scores),
+            'mean_difference': np.mean([a - h for a, h in zip(auto_scores, human_scores)]),
+            'length_correlation': spearmans_rho(
+                [e['length'] for e in self.window],
+                [e['automated']['score'] for e in self.window]
+            )
+        }
+```
+
+## 解读指标结果
+
+### 良好评估系统的指标参考
+
+| 指标 | 良好 | 可接受 | 值得关注 |
+|------|------|--------|----------|
+| Spearman's ρ | > 0.8 | 0.6–0.8 | < 0.6 |
+| Cohen's κ | > 0.7 | 0.5–0.7 | < 0.5 |
+| 位置一致性 | > 0.9 | 0.8–0.9 | < 0.8 |
+| 长度相关性 | < 0.2 | 0.2–0.4 | > 0.4 |
+
+### 预警信号
+
+1. **一致性高但相关性低**：可能存在校准问题
+2. **位置一致性低**：位置偏差正在影响结果
+3. **长度相关性高**：长度偏差拉高了评分
+4. **逐标准方差大**：某些标准的定义可能不够清晰
+
+## 报告模板
+
+```markdown
+## 评估系统指标报告
+
+### 与人工的一致性
+- Spearman's ρ：0.82（p < 0.001）
+- Cohen's κ：0.74
+- 样本量：500 次评估
+
+### 偏差指标
+- 位置一致性：91%
+- 长度与评分相关性：0.12
+
+### 各标准表现
+| 标准 | Spearman's ρ | κ |
+|-----------|--------------|---|
+| 准确性 | 0.88 | 0.79 |
+| 清晰度 | 0.76 | 0.68 |
+| 完整性 | 0.81 | 0.72 |
+
+### 建议
+- 所有指标均在可接受范围内
+- 关注「清晰度」标准——一致性较低可能表明需要优化评分细则

scripts/evaluation_example.py

@@ -0,0 +1,392 @@
+"""Advanced Evaluation Example
+
+Use when: building LLM-as-judge evaluation pipelines, comparing model outputs
+with position-bias mitigation, or generating domain-specific scoring rubrics.
+
+This module demonstrates the three core evaluation patterns from the
+advanced-evaluation skill: direct scoring, pairwise comparison with position
+swapping, and rubric generation. All functions use pseudocode-style examples
+that work across Python environments without specific dependencies.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+__all__ = [
+    "direct_scoring_example",
+    "pairwise_comparison_example",
+    "rubric_generation_example",
+]
+
+
+# =============================================================================
+# DIRECT SCORING EXAMPLE
+# =============================================================================
+
+
+def direct_scoring_example() -> dict[str, Any]:
+    """Rate a single response against defined criteria using direct scoring.
+
+    Use when: evaluating objective criteria like factual accuracy, instruction
+    following, or toxicity where a clear ground truth or rubric exists.
+
+    Returns:
+        Dictionary containing per-criterion scores, evidence, justifications,
+        and a weighted summary.
+    """
+
+    # Input
+    prompt: str = "Explain quantum entanglement to a high school student"
+    response: str = (
+        "Quantum entanglement is like having two magical coins that are connected. "
+        "When you flip one and it lands on heads, the other instantly shows tails, "
+        'no matter how far apart they are. Scientists call this "spooky action at a distance."'
+    )
+
+    criteria: list[dict[str, Any]] = [
+        {"name": "Accuracy", "description": "Scientific correctness", "weight": 0.4},
+        {"name": "Clarity", "description": "Understandable for audience", "weight": 0.3},
+        {"name": "Engagement", "description": "Interesting and memorable", "weight": 0.3},
+    ]
+
+    # System prompt for the evaluator
+    system_prompt: str = (
+        "You are an expert evaluator. Assess the response against each criterion.\n\n"
+        "For each criterion:\n"
+        "1. Find specific evidence in the response\n"
+        "2. Score according to the rubric (1-5 scale)\n"
+        "3. Justify your score with evidence\n"
+        "4. Suggest one specific improvement\n\n"
+        "Be objective and consistent. Base scores on explicit evidence."
+    )
+
+    # User prompt structure
+    user_prompt: str = f"""## Original Prompt
+{prompt}
+
+## Response to Evaluate
+{response}
+
+## Criteria
+1. **Accuracy** (weight: 0.4): Scientific correctness
+2. **Clarity** (weight: 0.3): Understandable for audience
+3. **Engagement** (weight: 0.3): Interesting and memorable
+
+## Output Format
+Respond with valid JSON:
+{{
+  "scores": [
+    {{
+      "criterion": "Accuracy",
+      "score": 4,
+      "evidence": ["quote or observation"],
+      "justification": "why this score",
+      "improvement": "specific suggestion"
+    }}
+  ],
+  "summary": {{
+    "assessment": "overall quality summary",
+    "strengths": ["strength 1"],
+    "weaknesses": ["weakness 1"]
+  }}
+}}"""
+
+    # Expected output structure
+    expected_output: dict[str, Any] = {
+        "scores": [
+            {
+                "criterion": "Accuracy",
+                "score": 4,
+                "evidence": ["Correctly uses analogy", "Mentions spooky action at a distance"],
+                "justification": "Core concept is correct, analogy is appropriate",
+                "improvement": "Could mention it's a quantum mechanical phenomenon",
+            },
+            {
+                "criterion": "Clarity",
+                "score": 5,
+                "evidence": ["Simple coin analogy", "No jargon"],
+                "justification": "Appropriate for high school level",
+                "improvement": "None needed",
+            },
+            {
+                "criterion": "Engagement",
+                "score": 4,
+                "evidence": ["Magical coins", "Spooky action quote"],
+                "justification": "Memorable imagery and Einstein quote",
+                "improvement": "Could add a real-world application",
+            },
+        ],
+        "summary": {
+            "assessment": "Good explanation suitable for the target audience",
+            "strengths": ["Clear analogy", "Age-appropriate language"],
+            "weaknesses": ["Could be more comprehensive"],
+        },
+    }
+
+    # Calculate weighted score
+    total_weight: float = sum(c["weight"] for c in criteria)
+    weighted_score: float = sum(
+        s["score"] * next(c["weight"] for c in criteria if c["name"] == s["criterion"])
+        for s in expected_output["scores"]
+    ) / total_weight
+
+    print(f"Weighted Score: {weighted_score:.2f}/5")
+    return expected_output
+
+
+# =============================================================================
+# PAIRWISE COMPARISON WITH POSITION BIAS MITIGATION
+# =============================================================================
+
+
+def pairwise_comparison_example() -> dict[str, Any]:
+    """Compare two responses with position-swapped bias mitigation.
+
+    Use when: evaluating subjective preferences like tone, style, or
+    persuasiveness where pairwise comparison achieves higher human-judge
+    agreement than direct scoring.
+
+    Returns:
+        Dictionary containing the winner, confidence score, and whether
+        position consistency was achieved across both passes.
+    """
+
+    prompt: str = "Explain machine learning to a beginner"
+
+    response_a: str = (
+        "Machine learning is a subset of artificial intelligence that enables "
+        "systems to learn and improve from experience without being explicitly "
+        "programmed. It uses statistical techniques to give computers the ability "
+        "to identify patterns in data."
+    )
+
+    response_b: str = (
+        "Imagine teaching a dog a new trick. You show the dog what to do, give "
+        "treats when it's right, and eventually it learns. Machine learning works "
+        "similarly - we show computers lots of examples, tell them when they're "
+        "right, and they learn to recognize patterns on their own."
+    )
+
+    criteria: list[str] = ["clarity", "accessibility", "accuracy"]
+
+    # System prompt emphasizing bias awareness
+    system_prompt: str = (
+        "You are an expert evaluator comparing two AI responses.\n\n"
+        "CRITICAL INSTRUCTIONS:\n"
+        "- Do NOT prefer responses because they are longer\n"
+        "- Do NOT prefer responses based on position (first vs second)\n"
+        "- Focus ONLY on quality according to the specified criteria\n"
+        "- Ties are acceptable when responses are genuinely equivalent"
+    )
+
+    # Build evaluation prompt for a given ordering
+    def evaluate_pass(
+        first_response: str,
+        second_response: str,
+        first_label: str,
+        second_label: str,
+    ) -> str:
+        """Build evaluation prompt for one pass of position-swapped comparison.
+
+        Use when: constructing the prompt for a single evaluation pass before
+        swapping response positions for bias mitigation.
+        """
+        return f"""## Original Prompt
+{prompt}
+
+## Response {first_label}
+{first_response}
+
+## Response {second_label}
+{second_response}
+
+## Comparison Criteria
+{', '.join(criteria)}
+
+## Output Format
+{{
+  "comparison": [
+    {{"criterion": "clarity", "winner": "A|B|TIE", "reasoning": "..."}}
+  ],
+  "result": {{
+    "winner": "A|B|TIE",
+    "confidence": 0.0-1.0,
+    "reasoning": "overall reasoning"
+  }}
+}}"""
+
+    # Position bias mitigation protocol
+    print("Pass 1: A in first position")
+    pass1_result: dict[str, Any] = {"winner": "B", "confidence": 0.8}
+
+    print("Pass 2: B in first position (swapped)")
+    pass2_result: dict[str, Any] = {"winner": "A", "confidence": 0.75}  # A because B was first
+
+    # Map pass2 result back (swap labels)
+    def map_winner(winner: str) -> str:
+        """Map winner label after position swap."""
+        return {"A": "B", "B": "A", "TIE": "TIE"}[winner]
+
+    pass2_mapped: str = map_winner(pass2_result["winner"])
+    print(f"Pass 2 mapped winner: {pass2_mapped}")
+
+    # Check consistency
+    consistent: bool = pass1_result["winner"] == pass2_mapped
+
+    final_result: dict[str, Any]
+    if consistent:
+        final_result = {
+            "winner": pass1_result["winner"],
+            "confidence": (pass1_result["confidence"] + pass2_result["confidence"]) / 2,
+            "position_consistent": True,
+        }
+    else:
+        final_result = {
+            "winner": "TIE",
+            "confidence": 0.5,
+            "position_consistent": False,
+            "bias_detected": True,
+        }
+
+    print(f"\nFinal Result: {final_result}")
+    return final_result
+
+
+# =============================================================================
+# RUBRIC GENERATION
+# =============================================================================
+
+
+def rubric_generation_example() -> dict[str, Any]:
+    """Generate a domain-specific scoring rubric for consistent evaluation.
+
+    Use when: establishing evaluation standards for a new criterion, reducing
+    scoring variance (rubrics cut variance by 40-60%), or onboarding new
+    evaluators to an existing evaluation pipeline.
+
+    Returns:
+        Dictionary containing score levels, characteristics, examples,
+        scoring guidelines, and edge case handling.
+    """
+
+    criterion_name: str = "Code Readability"
+    criterion_description: str = "How easy the code is to understand and maintain"
+    domain: str = "software engineering"
+    scale: str = "1-5"
+    strictness: str = "balanced"
+
+    system_prompt: str = (
+        f"You are an expert in creating evaluation rubrics.\n"
+        f"Create clear, actionable rubrics with distinct boundaries between levels.\n\n"
+        f"Strictness: {strictness}\n"
+        f"- lenient: Lower bar for passing scores\n"
+        f"- balanced: Fair, typical expectations\n"
+        f"- strict: High standards, critical evaluation"
+    )
+
+    user_prompt: str = f"""Create a scoring rubric for:
+
+**Criterion**: {criterion_name}
+**Description**: {criterion_description}
+**Scale**: {scale}
+**Domain**: {domain}
+
+Generate:
+1. Clear descriptions for each score level
+2. Specific characteristics that define each level
+3. Brief example text for each level
+4. General scoring guidelines
+5. Edge cases with guidance"""
+
+    # Expected rubric structure
+    rubric: dict[str, Any] = {
+        "criterion": criterion_name,
+        "scale": {"min": 1, "max": 5},
+        "levels": [
+            {
+                "score": 1,
+                "label": "Poor",
+                "description": "Code is difficult to understand without significant effort",
+                "characteristics": [
+                    "No meaningful variable or function names",
+                    "No comments or documentation",
+                    "Deeply nested or convoluted logic",
+                ],
+                "example": "def f(x): return x[0]*x[1]+x[2]",
+            },
+            {
+                "score": 3,
+                "label": "Adequate",
+                "description": "Code is understandable with some effort",
+                "characteristics": [
+                    "Most variables have meaningful names",
+                    "Basic comments for complex sections",
+                    "Logic is followable but could be cleaner",
+                ],
+                "example": (
+                    "def calc_total(items): # calculate sum\n"
+                    "    total = 0\n"
+                    "    for i in items: total += i\n"
+                    "    return total"
+                ),
+            },
+            {
+                "score": 5,
+                "label": "Excellent",
+                "description": "Code is immediately clear and maintainable",
+                "characteristics": [
+                    "All names are descriptive and consistent",
+                    "Comprehensive documentation",
+                    "Clean, modular structure",
+                ],
+                "example": (
+                    "def calculate_total_price(items: List[Item]) -> Decimal:\n"
+                    "    '''Calculate the total price of all items.'''\n"
+                    "    return sum(item.price for item in items)"
+                ),
+            },
+        ],
+        "scoring_guidelines": [
+            "Focus on readability, not cleverness",
+            "Consider the intended audience (team skill level)",
+            "Consistency matters more than style preference",
+        ],
+        "edge_cases": [
+            {
+                "situation": "Code uses domain-specific abbreviations",
+                "guidance": "Score based on readability for domain experts, not general audience",
+            },
+            {
+                "situation": "Code is auto-generated",
+                "guidance": "Apply same standards but note in evaluation",
+            },
+        ],
+    }
+
+    print("Generated Rubric:")
+    for level in rubric["levels"]:
+        print(f"  {level['score']}: {level['label']} - {level['description']}")
+
+    return rubric
+
+
+# =============================================================================
+# MAIN
+# =============================================================================
+
+if __name__ == "__main__":
+    print("=" * 60)
+    print("DIRECT SCORING EXAMPLE")
+    print("=" * 60)
+    direct_scoring_example()
+
+    print("\n" + "=" * 60)
+    print("PAIRWISE COMPARISON EXAMPLE")
+    print("=" * 60)
+    pairwise_comparison_example()
+
+    print("\n" + "=" * 60)
+    print("RUBRIC GENERATION EXAMPLE")
+    print("=" * 60)
+    rubric_generation_example()