chore: import zh skill context-compression
This commit is contained in:
wehub-skill-sync
@@ -0,0 +1,9 @@
|
||||
# WeHub 来源说明
|
||||
|
||||
- Skill 名称：`context-compression`
|
||||
- 中文类目：诊断与缓解上下文健康度问题
|
||||
- 上游仓库：`muratcankoylan__agent-skills-for-context-engineering`
|
||||
- 上游路径：`skills/context-compression/SKILL.md`
|
||||
- 上游链接：https://github.com/muratcankoylan/agent-skills-for-context-engineering/blob/HEAD/skills/context-compression/SKILL.md
|
||||
- 本仓库为 WeHub 中文 Skill 汉化包，基于 skill 市场筛选 Top200 清单整理
|
||||
- 原作者、版权和许可证信息以上游仓库为准
|
||||
@@ -0,0 +1,278 @@
|
||||
---name: context-compression
|
||||
description: 此技能适用于长时间运行的 Agent 会话需要上下文压缩、结构化摘要、精简、按任务优化 Token，以及保存决策、文件、风险和后续行动的可持久化交接摘要的场景。
|
||||
---
|
||||
|
||||
# 上下文压缩策略
|
||||
|
||||
当 Agent 会话生成数百万 Token 的对话历史时，压缩就变得必不可少。天真的做法是激进压缩以最小化每次请求的 Token 数。而正确的优化目标是按任务优化 Token 数：完成一个任务所消耗的 Token 总数，包括因压缩丢失关键信息而重新获取的成本。
|
||||
|
||||
## 何时启用
|
||||
|
||||
在以下情况启用此技能：
|
||||
- Agent 会话超出上下文窗口限制
|
||||
- 代码库超出上下文窗口（5M+ Token 的系统）
|
||||
- 设计对话摘要策略
|
||||
- 调试 Agent「忘记」自己修改过哪些文件的情况
|
||||
- 构建压缩质量评估框架
|
||||
- 创建可持久化的交接摘要，以保存决策、文件、风险和后续行动
|
||||
|
||||
对于其他技能负责的相邻工作，不要启用此技能：
|
||||
- 通用的 Token 效率策略，如掩码、前缀缓存或分区：请使用 `context-optimization`。
|
||||
- 在选择缓解方案之前诊断长上下文为何失败：请使用 `context-degradation`。
|
||||
- 将原始输出、日志或计划写入文件而不做摘要：请使用 `filesystem-context`。
|
||||
- 设计跨会话的长期语义记忆：请使用 `memory-systems`。
|
||||
|
||||
## 核心概念
|
||||
|
||||
上下文压缩在 Token 节省与信息损失之间进行权衡。根据会话特性，从以下三种生产就绪的方法中选择：
|
||||
|
||||
1. **锚定迭代摘要**：适用于需要跟踪文件的长时间运行会话。维护结构化、持久的摘要，包含明确的章节——会话意图、文件修改、决策和后续步骤。当触发压缩时，仅对新增截断的部分进行摘要，并与现有摘要合并，而不是从头重新生成。这样可以防止逐次全面重新生成摘要时产生的漂移——每次重新生成都有可能丢失模型认为优先级低但任务实际需要的细节。结构化方式能强制保留信息，因为专门的章节充当了摘要器必须填写的检查清单，从而捕捉到无声的信息丢失。
|
||||
|
||||
2. **不透明压缩**：仅适用于重新获取成本低且需要最大程度节省 Token 的短会话。它生成针对重建保真度优化的压缩表示，可实现 99% 以上的压缩比，但完全牺牲了可解释性。这一权衡很重要：如果不进行基于探测的评估，就无法验证保留了哪些内容，因此在调试或工件跟踪至关重要时，切勿使用此方法。
|
||||
|
||||
3. **重新生成式全文摘要**：适用于摘要可读性至关重要且会话具有清晰阶段边界的情况。它在每次触发压缩时生成详细的结构化摘要。其弱点是在多次循环中累积细节丢失——每次完整重新生成都是一次新的扫描，可能会降低先前摘要中保留细节的优先级。
|
||||
|
||||
## 详细主题
|
||||
|
||||
### 为按任务优化 Token 数，而非按请求优化 Token 数
|
||||
|
||||
衡量从任务开始到完成的 Token 总消耗量，而不是每次请求的 Token 数。当压缩丢弃了文件路径、错误消息或决策依据时，Agent 必须重新探索、重新读取文件并重新推导结论——这会浪费远比压缩节省的更多的 Token。一个每请求多节省 0.5% Token 但导致 20% 更多重新获取成本的策略，总体成本反而更高。将重新获取频率作为主要质量信号进行跟踪：如果 Agent 反复要求重新读取已经处理过的文件，说明压缩过于激进。
|
||||
|
||||
### 首先解决工件追踪问题
|
||||
|
||||
工件追踪完整性通常是压缩评估中最薄弱的维度（见 claim-context-compression-factory-benchmark）。应主动解决此问题，因为通用摘要无法可靠地维护它。
|
||||
|
||||
在每个压缩周期中显式保留以下类别：
|
||||
- 创建了哪些文件（完整路径）
|
||||
- 修改了哪些文件以及修改了什么（包括函数名，而不仅仅是文件名）
|
||||
- 读取了但未修改的文件
|
||||
- 特定标识符：函数名、变量名、错误消息、错误码
|
||||
|
||||
在 Agent 脚手架中实现独立的工件索引或显式的文件状态跟踪，而不是依赖摘要器来捕获这些细节。即使是带有专门文件章节的结构化摘要，在长时间会话中也难以保证完整性。
|
||||
|
||||
### 使用必填章节构建摘要
|
||||
|
||||
构建带有显式章节的结构化摘要，以防止无声的信息丢失。每个章节充当摘要器必须填写的检查清单，使遗漏变得可见而非无声。
|
||||
|
||||
```markdown
|
||||
## 会话意图
|
||||
[用户试图完成什么]
|
||||
|
||||
## 已修改文件
|
||||
- auth.controller.ts：修复了 JWT Token 生成
|
||||
- config/redis.ts：更新了连接池
|
||||
- tests/auth.test.ts：为新配置添加了 mock 设置
|
||||
|
||||
## 已做决策
|
||||
- 使用 Redis 连接池替代每请求连接
|
||||
- 对临时故障采用指数退避重试逻辑
|
||||
|
||||
## 当前状态
|
||||
- 14 个测试通过，2 个失败
|
||||
- 剩余工作：为 Session Service 测试设置 mock
|
||||
|
||||
## 后续步骤
|
||||
1. 修复剩余测试失败
|
||||
2. 运行完整测试套件
|
||||
3. 更新文档
|
||||
```
|
||||
|
||||
根据 Agent 的领域调整章节。调试 Agent 需要「根因」和「错误消息」；迁移 Agent 需要「源模式」和「目标模式」。结构本身比具体的章节更重要——任何显式的模式都比自由格式的摘要表现更好。
|
||||
|
||||
### 策略性地选择压缩触发时机
|
||||
|
||||
何时触发压缩与如何压缩同等重要。根据会话的可预测性选择触发策略：
|
||||
|
||||
| 策略 | 触发点 | 权衡 |
|
||||
|----------|---------------|-----------|
|
||||
| 固定阈值 | 上下文使用率达到 70-80% | 简单但可能过早压缩 |
|
||||
| 滑动窗口 | 保留最近 N 轮对话 + 摘要 | 上下文大小可预测 |
|
||||
| 基于重要性 | 优先压缩低相关性的章节 | 复杂但能保留信号 |
|
||||
| 任务边界 | 在逻辑任务完成时压缩 | 摘要清晰但时机不可预测 |
|
||||
|
||||
对于编码 Agent，默认使用带结构化摘要的滑动窗口——它提供了可预测性与质量的最佳平衡。当会话具有清晰的阶段转换时（例如，研究然后实现然后测试），使用任务边界触发器。
|
||||
|
||||
### 使用探测而非指标评估压缩
|
||||
|
||||
传统的指标如 ROUGE 或嵌入相似度无法捕捉功能性的压缩质量。一份摘要可能在词汇重叠上得分很高，却遗漏了 Agent 继续工作所需的那个文件路径。
|
||||
|
||||
使用基于探测的评估：压缩后，提出测试关键信息是否幸存的问题。如果 Agent 回答正确，说明压缩保留了正确的信息。如果回答错误，它就会猜测或产生幻觉。
|
||||
|
||||
| 探测类型 | 测试内容 | 示例问题 |
|
||||
|------------|---------------|------------------|
|
||||
| 召回 | 事实保留 | "原始错误消息是什么？" |
|
||||
| 工件 | 文件跟踪 | "我们修改了哪些文件？" |
|
||||
| 延续 | 任务规划 | "我们下一步应该做什么？" |
|
||||
| 决策 | 推理链 | "关于 Redis 问题我们做了什么决定？" |
|
||||
|
||||
### 从六个维度评估压缩
|
||||
|
||||
从以下维度评估编码 Agent 的压缩质量。准确性和工件追踪完整性通常比词汇相似度更能区分不同方法（见 claim-context-compression-factory-benchmark），因此压缩需要超出通用摘要之外的特殊处理。
|
||||
|
||||
1. **准确性**：技术细节是否正确——文件路径、函数名、错误码？
|
||||
2. **上下文感知**：回复是否反映了当前的对话状态？
|
||||
3. **工件追踪**：Agent 是否知道哪些文件被读取或修改了？
|
||||
4. **完整性**：回复是否涵盖了问题的所有部分？
|
||||
5. **连续性**：能否在不重新获取信息的情况下继续工作？
|
||||
6. **指令遵循**：回复是否尊重已声明的约束条件？
|
||||
|
||||
## 实用指南
|
||||
|
||||
### 对大型代码库应用三阶段压缩工作流
|
||||
|
||||
对于超出上下文窗口的代码库或 Agent 系统，通过三个顺序阶段进行压缩。每个阶段都缩小上下文，使下一阶段能在预算内运行。
|
||||
|
||||
1. **研究阶段**：探索架构图、文档和关键接口。将探索结果压缩为对组件、依赖和边界的结构化分析。输出：一份替代原始探索的研究文档。
|
||||
|
||||
2. **规划阶段**：将研究文档转化为包含函数签名、类型定义和数据流的实现规格说明。一个 5M Token 的代码库在此阶段可压缩至约 2000 词的规格说明。
|
||||
|
||||
3. **实现阶段**：根据规格说明执行。上下文始终聚焦于规格说明加当前工作文件，而非原始代码库探索。此阶段很少需要进一步压缩，因为规格说明已经足够精简。
|
||||
|
||||
### 使用示例工件作为压缩种子
|
||||
|
||||
当提供了手动迁移示例或参考 PR 时，将其作为模板来理解目标模式，而不是从头开始探索代码库。该示例揭示了静态分析无法发现的约束条件：哪些不变量必须保持，哪些服务在变更时会中断，以及一个干净实现是什么样的。
|
||||
|
||||
这在 Agent 无法区分本质复杂性（业务需求）与偶然复杂性（遗留变通方案）时最为重要。示例工件隐含地编码了这一区分，从而节省了原本会花在试错探索上的 Token。
|
||||
|
||||
### 逐步实施锚定迭代摘要
|
||||
|
||||
1. 根据 Agent 的领域（调试、迁移、功能开发）定义显式的摘要章节
|
||||
2. 第一次触发压缩时，将截断的历史摘要到这些章节中
|
||||
3. 后续压缩时，仅摘要新增截断的内容——不要重新摘要已有的摘要
|
||||
4. 将新信息合并到现有章节中，而不是重新生成它们，按文件路径和决策标识进行去重
|
||||
5. 标记每条信息来自哪个压缩周期——当摘要发生漂移时，这有助于调试
|
||||
|
||||
### 根据会话特征选择正确的方法
|
||||
|
||||
**在以下情况使用锚定迭代摘要：**
|
||||
- 会话长时间运行（100 条以上消息）
|
||||
- 文件跟踪很重要（编码、调试）
|
||||
- 需要验证已保留的信息
|
||||
|
||||
**在以下情况使用不透明压缩：**
|
||||
- 需要最大程度节省 Token
|
||||
- 会话相对较短
|
||||
- 重新获取成本低（例如，无需访问文件系统）
|
||||
|
||||
**在以下情况使用重新生成式摘要：**
|
||||
- 摘要的可解释性对人工审核至关重要
|
||||
- 会话具有清晰的阶段边界
|
||||
- 每次触发压缩时接受完整的上下文审查
|
||||
|
||||
### 按方法校准压缩比
|
||||
|
||||
| 方法 | 压缩比 | 质量评分 | 权衡 |
|
||||
|--------|-------------------|---------------|-----------|
|
||||
| 锚定迭代 | 98.6% | 3.70 | 质量最佳，压缩率略低 |
|
||||
| 重新生成式 | 98.7% | 3.44 | 质量好，压缩率适中 |
|
||||
| 不透明压缩 | 99.3% | 3.35 | 压缩率最高，质量有损失 |
|
||||
|
||||
将这些作为特定来源的基准数据，而非通用常量（见 claim-context-compression-factory-benchmark）。对于任何存在重新获取成本的任务，这一权衡通常倾向于结构化方法。
|
||||
|
||||
## 示例
|
||||
|
||||
**示例 1：调试会话压缩**
|
||||
|
||||
原始上下文（89,000 Token，178 条消息）：
|
||||
- `/api/auth/login` 端点返回 401 错误
|
||||
- 跟踪了 auth controller、middleware、session store
|
||||
- 发现过期的 Redis 连接
|
||||
- 修复了连接池，添加了重试逻辑
|
||||
- 14 个测试通过，2 个失败
|
||||
|
||||
压缩后的结构化摘要：
|
||||
|
||||
```markdown
|
||||
## 会话意图
|
||||
调试 /api/auth/login 上使用有效凭证仍返回 401 Unauthorized 错误的问题。
|
||||
|
||||
## 根因
|
||||
Session Store 中存在过期的 Redis 连接。JWT 生成正确，但会话无法持久化。
|
||||
|
||||
## 已修改文件
|
||||
- auth.controller.ts：未修改（仅读取）
|
||||
- middleware/cors.ts：未修改（已检查）
|
||||
- config/redis.ts：修复了连接池配置
|
||||
- services/session.service.ts：为临时故障添加了重试逻辑
|
||||
- tests/auth.test.ts：更新了 mock 设置
|
||||
|
||||
## 测试状态
|
||||
14 个通过，2 个失败（mock 设置问题）
|
||||
|
||||
## 后续步骤
|
||||
1. 修复剩余测试失败（mock session service）
|
||||
2. 运行完整测试套件
|
||||
3. 部署到预发布环境
|
||||
```
|
||||
|
||||
**示例 2：探测回复质量**
|
||||
|
||||
压缩后，询问「原始错误是什么？」：
|
||||
|
||||
好回复（结构化摘要）：
|
||||
> "原始错误是 /api/auth/login 端点返回了 401 Unauthorized 响应。用户使用有效凭证时收到了此错误。根因是 Session Store 中存在过期的 Redis 连接。"
|
||||
|
||||
差回复（激进压缩）：
|
||||
> "我们当时在调试一个认证问题。登录失败了。我们修复了一些配置问题。"
|
||||
|
||||
结构化回复保留了端点、错误码和根因。激进回复丢失了所有技术细节。
|
||||
|
||||
## 指南
|
||||
|
||||
1. 按任务优化 Token 数，而非按请求优化 Token 数
|
||||
2. 使用带有显式文件跟踪章节的结构化摘要
|
||||
3. 在上下文使用率达到 70-80% 时触发压缩
|
||||
4. 实施增量合并而非完全重新生成
|
||||
5. 使用基于探测的评估测试压缩质量
|
||||
6. 如果文件跟踪至关重要，则单独跟踪工件
|
||||
7. 接受略低的压缩比以换取更好的质量保留
|
||||
8. 监控重新获取频率作为压缩质量信号
|
||||
|
||||
## 注意事项
|
||||
|
||||
1. **切勿压缩工具定义或模式**：压缩函数调用模式、API 规格说明或工具定义会完全破坏 Agent 的功能。Agent 无法调用参数名或类型已被摘要掉的工具。将工具定义视为不可变的锚点，使其绕过压缩。
|
||||
|
||||
2. **压缩摘要会产生事实幻觉**：当 LLM 对对话历史进行摘要时，可能会引入听起来合理但原始内容中从未出现的细节。在丢弃原始内容之前，务必对照源材料验证压缩输出——尤其是摘要器可能会「四舍五入」或编造的文件路径、错误码和数值。
|
||||
|
||||
3. **压缩会破坏工件引用**：文件路径、提交 SHA、变量名和代码片段在压缩过程中会被改写或丢弃。当摘要说「更新了配置文件」而 Agent 需要 `config/redis.ts` 时，就会导致重新探索。在专门的章节中逐字保留标识符，而不是将其嵌入散文。
|
||||
|
||||
4. **早期轮次包含不可替代的约束**：会话的前几轮通常包含任务设置、用户约束和架构决策，这些是无法重新推导出来的。保护早期轮次免受压缩，或将其约束提取到持久的序言中，使其贯穿所有压缩周期。
|
||||
|
||||
5. **激进的比例会在多个周期中叠加**：95% 的压缩比单独看似乎安全，但重复应用会使损失累积。经过三轮 95% 的压缩后，仅剩下 0.0125% 的原始 Token。校准压缩比时应假设会有多个压缩周期，而非仅一次。
|
||||
|
||||
6. **代码和散文需要不同的压缩策略**：散文压缩效果好，因为自然语言存在冗余。但代码并非如此——从函数签名或导入路径中移除一个 Token 就可能使其完全无用。应用领域特定的压缩策略：激进地压缩散文章节，同时逐字保留代码块和结构化数据。
|
||||
|
||||
7. **基于探测的评估会带来虚假信心**：探测可能通过，尽管关键信息已丢失，因为探测只测试它们所问的内容。一组检查文件名但不检查函数签名的探测集，会遗漏签名丢失的问题。设计覆盖所有六个评估维度的探测，并在不同评估轮次中轮换探测集，以避免盲点。
|
||||
|
||||
## 集成
|
||||
|
||||
此技能与集合中的其他几个技能相关联：
|
||||
|
||||
- context-degradation——压缩是应对性能下降的一种缓解策略
|
||||
- context-optimization——压缩是众多优化技术之一
|
||||
- evaluation——基于探测的评估适用于压缩测试
|
||||
- memory-systems——压缩与草稿板和摘要记忆模式相关
|
||||
|
||||
## 参考
|
||||
|
||||
内部参考：
|
||||
- [评估框架参考](./references/evaluation-framework.md)——何时阅读：构建或校准基于探测的评估管道时，或需要用于压缩质量评估的评分标准和 LLM 评判配置时
|
||||
|
||||
本集合中的相关技能：
|
||||
- context-degradation——何时阅读：在应用压缩作为缓解措施之前，诊断 Agent 在长时间会话中性能为何下降时
|
||||
- context-optimization——何时阅读：仅靠压缩不够，需要更广泛的优化策略（剪枝、缓存、路由）时
|
||||
- evaluation——何时阅读：设计超出压缩特定探测的评估框架时，包括通用的 LLM-as-judge 方法
|
||||
|
||||
外部资源：
|
||||
- Factory Research：评估 AI Agent 的上下文压缩（2025 年 12 月）——何时阅读：需要压缩方法比较的基准数据或包含 36,000 条消息的评估数据集时
|
||||
- LLM-as-judge 评估方法论研究（Zheng 等人，2023 年）——何时阅读：实施或验证 LLM 评判评分，以了解偏差模式和校准方法时
|
||||
- Netflix Engineering：「无限软件危机」——大规模三阶段工作流与上下文压缩（AI Summit 2025）——何时阅读：为大型代码库实施三阶段压缩工作流，或了解生产级上下文管理时
|
||||
|
||||
---
|
||||
|
||||
## 技能元数据
|
||||
|
||||
**创建日期**：2025-12-22
|
||||
**最后更新**：2026-05-15
|
||||
**作者**：上下文工程贡献者的 Agent 技能
|
||||
**版本**：1.3.0
|
||||
@@ -0,0 +1,211 @@
|
||||
# 上下文压缩评估框架
|
||||
|
||||
本文档提供了用于衡量上下文压缩质量的完整评估框架，包括探针类型、评分标准和大语言模型裁判配置。
|
||||
|
||||
## 探针类型
|
||||
|
||||
### 召回探针
|
||||
|
||||
测试对对话历史中具体细节的事实性保留。
|
||||
|
||||
**结构：**
|
||||
```
|
||||
问题：[询问截断历史中的具体事实]
|
||||
预期：[应被保留的精确细节]
|
||||
评分：技术细节的匹配准确度
|
||||
```
|
||||
|
||||
**示例：**
|
||||
- "启动本次调试会话的原始错误信息是什么？"
|
||||
- "我们决定使用哪个版本的依赖？"
|
||||
- "执行失败的具体命令是什么？"
|
||||
|
||||
### 制品探针
|
||||
|
||||
测试文件跟踪与修改感知能力。
|
||||
|
||||
**结构：**
|
||||
```
|
||||
问题：[询问创建、修改或检查过的文件]
|
||||
预期：[包含变更描述的完整列表]
|
||||
评分：文件列表的完整性与变更描述的准确性
|
||||
```
|
||||
|
||||
**示例：**
|
||||
- "我们修改了哪些文件？请描述每个文件中的变化。"
|
||||
- "本次会话中我们创建了哪些新文件？"
|
||||
- "我们检查过但未修改的配置文件有哪些？"
|
||||
|
||||
### 延续探针
|
||||
|
||||
测试无需重新获取上下文即可继续工作的能力。
|
||||
|
||||
**结构：**
|
||||
```
|
||||
问题：[询问下一步或当前状态]
|
||||
预期：[基于会话历史的可操作下一步]
|
||||
评分：无需请求重新读取文件即可继续的能力
|
||||
```
|
||||
|
||||
**示例：**
|
||||
- "我们接下来应该做什么？"
|
||||
- "还有哪些测试仍然失败？原因是什么？"
|
||||
- "上一步中我们有什么未完成的工作？"
|
||||
|
||||
### 决策探针
|
||||
|
||||
测试对推理链与决策理由的保留能力。
|
||||
|
||||
**结构：**
|
||||
```
|
||||
问题：[询问某个决策的缘由]
|
||||
预期：[导致该决策的推理过程]
|
||||
评分：决策上下文及备选方案考虑的保留程度
|
||||
```
|
||||
|
||||
**示例：**
|
||||
- "我们讨论过 Redis 问题的几种方案。最终的决定是什么？为什么？"
|
||||
- "为什么我们选择了连接池方案而非每次请求单独连接？"
|
||||
- "在身份验证修复中，我们考虑过哪些备选方案？"
|
||||
|
||||
## 评分标准
|
||||
|
||||
### 准确度维度
|
||||
|
||||
| 标准 | 问题 | 0 分 | 3 分 | 5 分 |
|
||||
|-----------|----------|---------|---------|---------|
|
||||
| accuracy_factual | 事实、文件路径和技术细节是否正确？ | 完全错误或编造 | 基本准确，存在轻微错误 | 完全准确 |
|
||||
| accuracy_technical | 代码引用和技术概念是否正确？ | 重大技术错误 | 基本正确，存在轻微问题 | 技术精确 |
|
||||
|
||||
### 上下文感知维度
|
||||
|
||||
| 标准 | 问题 | 0 分 | 3 分 | 5 分 |
|
||||
|-----------|----------|---------|---------|---------|
|
||||
| context_conversation_state | 回答是否反映了当前对话状态？ | 未意识到先前上下文 | 大致知晓，存在盲区 | 完全了解对话历史 |
|
||||
| context_artifact_state | 回答是否反映了访问过的文件/制品？ | 未意识到制品 | 部分了解制品 | 完全了解制品状态 |
|
||||
|
||||
### 制品轨迹维度
|
||||
|
||||
| 标准 | 问题 | 0 分 | 3 分 | 5 分 |
|
||||
|-----------|----------|---------|---------|---------|
|
||||
| artifact_files_created | 代理是否知道创建了哪些文件？ | 一无所知 | 知道大部分文件 | 完全知晓 |
|
||||
| artifact_files_modified | 代理是否知道哪些文件被修改以及修改了什么？ | 一无所知 | 较好了解大部分修改 | 完全了解所有修改 |
|
||||
| artifact_key_details | 代理是否能记住函数名、变量名、错误信息？ | 无法回忆 | 能回忆大部分关键细节 | 完美回忆 |
|
||||
|
||||
### 完整性维度
|
||||
|
||||
| 标准 | 问题 | 0 分 | 3 分 | 5 分 |
|
||||
|-----------|----------|---------|---------|---------|
|
||||
| completeness_coverage | 回答是否涵盖问题的所有部分？ | 忽略大部分内容 | 涵盖大部分内容 | 全面覆盖所有内容 |
|
||||
| completeness_depth | 是否提供了足够的细节？ | 肤浅或缺少细节 | 细节充分 | 细节详尽 |
|
||||
|
||||
### 连续性维度
|
||||
|
||||
| 标准 | 问题 | 0 分 | 3 分 | 5 分 |
|
||||
|-----------|----------|---------|---------|---------|
|
||||
| continuity_work_state | 代理能否在不重新获取先前访问过的信息的情况下继续工作？ | 无法继续，需重新获取所有上下文 | 可继续，只需最小限度重新获取 | 可无缝继续 |
|
||||
| continuity_todo_state | 代理是否保持对未完成任务的感知？ | 丢失所有待办事项记录 | 较好的感知，存在少量盲区 | 完美的任务感知 |
|
||||
| continuity_reasoning | 代理是否保留先前决策背后的理由？ | 无法记住推理过程 | 大致能记住推理过程 | 极好地保留 |
|
||||
|
||||
### 指令遵循维度
|
||||
|
||||
| 标准 | 问题 | 0 分 | 3 分 | 5 分 |
|
||||
|-----------|----------|---------|---------|---------|
|
||||
| instruction_format | 回答是否遵循了要求的格式？ | 忽略格式 | 基本遵循格式 | 完美遵循格式 |
|
||||
| instruction_constraints | 回答是否遵守了给定的约束条件？ | 忽略约束 | 基本遵守约束 | 完全遵守所有约束 |
|
||||
|
||||
## 大语言模型裁判配置
|
||||
|
||||
### 系统提示词
|
||||
|
||||
```
|
||||
你是一名专业评估员，负责评估 AI 助手在软件开发对话中的回答。
|
||||
|
||||
你的任务是根据特定的评分标准对回答进行评分。针对每个标准：
|
||||
1. 仔细阅读标准问题
|
||||
2. 检查回答中的相关证据
|
||||
3. 根据评分指南给出 0-5 分的分数
|
||||
4. 提供简要的评分理由
|
||||
|
||||
请保持客观一致。关注回答中实际呈现的内容，而非可能包含的内容。
|
||||
```
|
||||
|
||||
### 裁判输入格式
|
||||
|
||||
```json
|
||||
{
|
||||
"probe_question": "原始的报错信息是什么？",
|
||||
"model_response": "[待评估的回答]",
|
||||
"compacted_context": "[所提供的压缩上下文]",
|
||||
"ground_truth": "[可选：已知的正确答案]",
|
||||
"rubric_criteria": ["accuracy_factual", "accuracy_technical", "context_conversation_state"]
|
||||
}
|
||||
```
|
||||
|
||||
### 裁判输出格式
|
||||
|
||||
```json
|
||||
{
|
||||
"criterionResults": [
|
||||
{
|
||||
"criterionId": "accuracy_factual",
|
||||
"score": 5,
|
||||
"reasoning": "回答正确识别了 401 错误、具体端点和根本原因。"
|
||||
}
|
||||
],
|
||||
"aggregateScore": 4.8,
|
||||
"dimensionScores": {
|
||||
"accuracy": 4.9,
|
||||
"context_awareness": 4.5,
|
||||
"artifact_trail": 3.2,
|
||||
"completeness": 5.0,
|
||||
"continuity": 4.8,
|
||||
"instruction_following": 5.0
|
||||
}
|
||||
}
|
||||
|
||||
## 基准测试结果参考
|
||||
|
||||
各压缩方法的表现（基于 36,000+ 条消息）：
|
||||
|
||||
| 方法 | 总体 | 准确度 | 上下文 | 制品 | 完整性 | 连续性 | 指令 |
|
||||
|--------|---------|----------|---------|----------|----------|------------|-------------|
|
||||
| 锚定迭代式 | 3.70 | 4.04 | 4.01 | 2.45 | 4.44 | 3.80 | 4.99 |
|
||||
| 再生式 | 3.44 | 3.74 | 3.56 | 2.33 | 4.37 | 3.67 | 4.95 |
|
||||
| 不透明式 | 3.35 | 3.43 | 3.64 | 2.19 | 4.37 | 3.77 | 4.92 |
|
||||
|
||||
**主要发现：**
|
||||
|
||||
1. **准确度差距**：最佳与最差方法之间相差 0.61 分
|
||||
2. **上下文感知差距**：相差 0.45 分，锚定迭代方法占优
|
||||
3. **制品轨迹**：普遍薄弱（2.19-2.45），需要专门处理
|
||||
4. **完整性与指令遵循**：差异化程度极低
|
||||
|
||||
## 统计考量
|
||||
|
||||
- 0.26-0.35 分的差异在不同任务类型和会话长度上具有一致性
|
||||
- 该模式在短会话和长会话中均成立
|
||||
- 该模式在调试、功能实现和代码审查任务中均成立
|
||||
- 样本量：跨数百个压缩点的 36,611 条消息
|
||||
|
||||
## 实现说明
|
||||
|
||||
### 探针生成
|
||||
|
||||
在每个压缩点根据截断历史生成探针：
|
||||
1. 提取事实性陈述，用于召回探针
|
||||
2. 提取文件操作记录，用于制品探针
|
||||
3. 提取未完成任务，用于延续探针
|
||||
4. 提取决策节点，用于决策探针
|
||||
|
||||
### 评分流程
|
||||
|
||||
1. 将探针问题 + 模型回答 + 压缩上下文输入裁判
|
||||
2. 对照评分标准中的每个标准进行评估
|
||||
3. 输出带有分数和推理过程的结构化 JSON
|
||||
4. 计算维度得分（加权平均）
|
||||
5. 计算总体得分（各维度得分的算术平均）
|
||||
|
||||
### 盲测原则
|
||||
|
||||
裁判不应知晓被评估的回答是由哪种压缩方法产生的。这可以防止对已知方法产生偏见。
|
||||
@@ -0,0 +1,862 @@
|
||||
"""
|
||||
Context Compression Evaluation
|
||||
|
||||
Public API for evaluating context compression quality using probe-based
|
||||
assessment. This module provides three composable components:
|
||||
|
||||
- **ProbeGenerator**: Extracts factual claims, file operations, and decisions
|
||||
from conversation history, then generates typed probes for evaluation.
|
||||
Use when: building a compression evaluation pipeline and needing to
|
||||
automatically derive test questions from raw conversation history.
|
||||
|
||||
- **CompressionEvaluator**: Scores probe responses against a multi-dimensional
|
||||
rubric (accuracy, context awareness, artifact trail, completeness,
|
||||
continuity, instruction following). Use when: comparing compression methods
|
||||
or validating that a compression strategy preserves critical information.
|
||||
|
||||
- **StructuredSummarizer**: Implements anchored iterative summarization with
|
||||
explicit sections for session intent, file tracking, decisions, and next
|
||||
steps. Use when: compressing long-running coding sessions where file
|
||||
tracking and decision rationale must survive compression.
|
||||
|
||||
Top-level convenience function:
|
||||
- **evaluate_compression_quality**: End-to-end pipeline that generates probes,
|
||||
collects model responses, evaluates them, and returns a scored summary with
|
||||
recommendations. Use when: running a one-shot compression quality check
|
||||
without wiring up individual components.
|
||||
|
||||
PRODUCTION NOTES:
|
||||
- The LLM judge calls are stubbed for demonstration. Production systems
|
||||
should implement actual API calls to a frontier model.
|
||||
- Token estimation uses simplified heuristics. Production systems should
|
||||
use model-specific tokenizers.
|
||||
- Ground truth extraction uses pattern matching. Production systems may
|
||||
benefit from more sophisticated fact extraction.
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import List, Dict, Optional, Callable
|
||||
from enum import Enum
|
||||
import json
|
||||
import re
|
||||
|
||||
__all__ = [
|
||||
"ProbeType",
|
||||
"Probe",
|
||||
"CriterionResult",
|
||||
"EvaluationResult",
|
||||
"RUBRIC_CRITERIA",
|
||||
"ProbeGenerator",
|
||||
"CompressionEvaluator",
|
||||
"StructuredSummarizer",
|
||||
"evaluate_compression_quality",
|
||||
]
|
||||
|
||||
|
||||
class ProbeType(Enum):
|
||||
"""Types of evaluation probes for compression quality assessment."""
|
||||
RECALL = "recall"
|
||||
ARTIFACT = "artifact"
|
||||
CONTINUATION = "continuation"
|
||||
DECISION = "decision"
|
||||
|
||||
|
||||
@dataclass
|
||||
class Probe:
|
||||
"""A probe question for evaluating compression quality.
|
||||
|
||||
Use when: constructing evaluation inputs for CompressionEvaluator.
|
||||
Each probe targets a specific information category that compression
|
||||
may have lost.
|
||||
"""
|
||||
probe_type: ProbeType
|
||||
question: str
|
||||
ground_truth: Optional[str] = None
|
||||
context_reference: Optional[str] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class CriterionResult:
|
||||
"""Result for a single evaluation criterion."""
|
||||
criterion_id: str
|
||||
score: float
|
||||
reasoning: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class EvaluationResult:
|
||||
"""Complete evaluation result for a probe response.
|
||||
|
||||
Contains per-criterion scores, per-dimension aggregates, and an
|
||||
overall aggregate score.
|
||||
"""
|
||||
probe: Probe
|
||||
response: str
|
||||
criterion_results: List[CriterionResult]
|
||||
aggregate_score: float
|
||||
dimension_scores: Dict[str, float] = field(default_factory=dict)
|
||||
|
||||
|
||||
# Evaluation Rubrics
|
||||
|
||||
RUBRIC_CRITERIA: Dict[str, List[Dict]] = {
|
||||
"accuracy": [
|
||||
{
|
||||
"id": "accuracy_factual",
|
||||
"question": "Are facts, file paths, and technical details correct?",
|
||||
"weight": 0.6
|
||||
},
|
||||
{
|
||||
"id": "accuracy_technical",
|
||||
"question": "Are code references and technical concepts correct?",
|
||||
"weight": 0.4
|
||||
}
|
||||
],
|
||||
"context_awareness": [
|
||||
{
|
||||
"id": "context_conversation_state",
|
||||
"question": "Does the response reflect current conversation state?",
|
||||
"weight": 0.5
|
||||
},
|
||||
{
|
||||
"id": "context_artifact_state",
|
||||
"question": "Does the response reflect which files/artifacts were accessed?",
|
||||
"weight": 0.5
|
||||
}
|
||||
],
|
||||
"artifact_trail": [
|
||||
{
|
||||
"id": "artifact_files_created",
|
||||
"question": "Does the agent know which files were created?",
|
||||
"weight": 0.3
|
||||
},
|
||||
{
|
||||
"id": "artifact_files_modified",
|
||||
"question": "Does the agent know which files were modified?",
|
||||
"weight": 0.4
|
||||
},
|
||||
{
|
||||
"id": "artifact_key_details",
|
||||
"question": "Does the agent remember function names, variable names, error messages?",
|
||||
"weight": 0.3
|
||||
}
|
||||
],
|
||||
"completeness": [
|
||||
{
|
||||
"id": "completeness_coverage",
|
||||
"question": "Does the response address all parts of the question?",
|
||||
"weight": 0.6
|
||||
},
|
||||
{
|
||||
"id": "completeness_depth",
|
||||
"question": "Is sufficient detail provided?",
|
||||
"weight": 0.4
|
||||
}
|
||||
],
|
||||
"continuity": [
|
||||
{
|
||||
"id": "continuity_work_state",
|
||||
"question": "Can the agent continue without re-fetching information?",
|
||||
"weight": 0.4
|
||||
},
|
||||
{
|
||||
"id": "continuity_todo_state",
|
||||
"question": "Does the agent maintain awareness of pending tasks?",
|
||||
"weight": 0.3
|
||||
},
|
||||
{
|
||||
"id": "continuity_reasoning",
|
||||
"question": "Does the agent retain rationale behind previous decisions?",
|
||||
"weight": 0.3
|
||||
}
|
||||
],
|
||||
"instruction_following": [
|
||||
{
|
||||
"id": "instruction_format",
|
||||
"question": "Does the response follow the requested format?",
|
||||
"weight": 0.5
|
||||
},
|
||||
{
|
||||
"id": "instruction_constraints",
|
||||
"question": "Does the response respect stated constraints?",
|
||||
"weight": 0.5
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
class ProbeGenerator:
|
||||
"""Generate typed probes from conversation history.
|
||||
|
||||
Use when: automatically deriving evaluation questions from raw
|
||||
conversation history at compression points. Extracts facts, file
|
||||
operations, and decisions via pattern matching, then produces
|
||||
one probe per category.
|
||||
|
||||
For production systems, replace the regex-based extraction with
|
||||
an LLM-based extractor for higher recall.
|
||||
"""
|
||||
|
||||
def __init__(self, conversation_history: str) -> None:
|
||||
self.history = conversation_history
|
||||
self.extracted_facts = self._extract_facts()
|
||||
self.extracted_files = self._extract_files()
|
||||
self.extracted_decisions = self._extract_decisions()
|
||||
|
||||
def generate_probes(self) -> List[Probe]:
|
||||
"""Generate all probe types for evaluation.
|
||||
|
||||
Use when: preparing evaluation inputs at a compression point.
|
||||
Returns one probe per category (recall, artifact, continuation,
|
||||
decision) based on extractable content from the history.
|
||||
"""
|
||||
probes: List[Probe] = []
|
||||
|
||||
# Recall probes
|
||||
if self.extracted_facts:
|
||||
probes.append(Probe(
|
||||
probe_type=ProbeType.RECALL,
|
||||
question="What was the original error or issue that started this session?",
|
||||
ground_truth=self.extracted_facts.get("original_error"),
|
||||
context_reference="session_start"
|
||||
))
|
||||
|
||||
# Artifact probes
|
||||
if self.extracted_files:
|
||||
probes.append(Probe(
|
||||
probe_type=ProbeType.ARTIFACT,
|
||||
question="Which files have we modified? Describe what changed in each.",
|
||||
ground_truth=json.dumps(self.extracted_files),
|
||||
context_reference="file_operations"
|
||||
))
|
||||
|
||||
# Continuation probes
|
||||
probes.append(Probe(
|
||||
probe_type=ProbeType.CONTINUATION,
|
||||
question="What should we do next?",
|
||||
ground_truth=self.extracted_facts.get("next_steps"),
|
||||
context_reference="task_state"
|
||||
))
|
||||
|
||||
# Decision probes
|
||||
if self.extracted_decisions:
|
||||
probes.append(Probe(
|
||||
probe_type=ProbeType.DECISION,
|
||||
question="What key decisions did we make and why?",
|
||||
ground_truth=json.dumps(self.extracted_decisions),
|
||||
context_reference="decision_points"
|
||||
))
|
||||
|
||||
return probes
|
||||
|
||||
def _extract_facts(self) -> Dict[str, str]:
|
||||
"""Extract factual claims from history."""
|
||||
facts: Dict[str, str] = {}
|
||||
|
||||
# Extract error patterns
|
||||
error_patterns = [
|
||||
r"error[:\s]+(.+?)(?:\n|$)",
|
||||
r"(\d{3})\s+(Unauthorized|Not Found|Internal Server Error)",
|
||||
r"exception[:\s]+(.+?)(?:\n|$)"
|
||||
]
|
||||
|
||||
for pattern in error_patterns:
|
||||
match = re.search(pattern, self.history, re.IGNORECASE)
|
||||
if match:
|
||||
facts["original_error"] = match.group(0).strip()
|
||||
break
|
||||
|
||||
# Extract next steps
|
||||
next_step_patterns = [
|
||||
r"next[:\s]+(.+?)(?:\n|$)",
|
||||
r"TODO[:\s]+(.+?)(?:\n|$)",
|
||||
r"remaining[:\s]+(.+?)(?:\n|$)"
|
||||
]
|
||||
|
||||
for pattern in next_step_patterns:
|
||||
match = re.search(pattern, self.history, re.IGNORECASE)
|
||||
if match:
|
||||
facts["next_steps"] = match.group(0).strip()
|
||||
break
|
||||
|
||||
return facts
|
||||
|
||||
def _extract_files(self) -> List[Dict[str, str]]:
|
||||
"""Extract file operations from history."""
|
||||
files: List[Dict[str, str]] = []
|
||||
|
||||
# Common file patterns
|
||||
file_patterns = [
|
||||
r"(?:modified|changed|updated|edited)\s+([^\s]+\.[a-z]+)",
|
||||
r"(?:created|added)\s+([^\s]+\.[a-z]+)",
|
||||
r"(?:read|examined|opened)\s+([^\s]+\.[a-z]+)"
|
||||
]
|
||||
|
||||
for pattern in file_patterns:
|
||||
matches = re.findall(pattern, self.history, re.IGNORECASE)
|
||||
for match in matches:
|
||||
if match not in [f["path"] for f in files]:
|
||||
files.append({
|
||||
"path": match,
|
||||
"operation": "modified" if "modif" in pattern else "created" if "creat" in pattern else "read"
|
||||
})
|
||||
|
||||
return files
|
||||
|
||||
def _extract_decisions(self) -> List[Dict[str, str]]:
|
||||
"""Extract decision points from history."""
|
||||
decisions: List[Dict[str, str]] = []
|
||||
|
||||
decision_patterns = [
|
||||
r"decided to\s+(.+?)(?:\n|$)",
|
||||
r"chose\s+(.+?)(?:\n|$)",
|
||||
r"going with\s+(.+?)(?:\n|$)",
|
||||
r"will use\s+(.+?)(?:\n|$)"
|
||||
]
|
||||
|
||||
for pattern in decision_patterns:
|
||||
matches = re.findall(pattern, self.history, re.IGNORECASE)
|
||||
for match in matches:
|
||||
decisions.append({
|
||||
"decision": match.strip(),
|
||||
"context": pattern.split("\\s+")[0]
|
||||
})
|
||||
|
||||
return decisions[:5] # Limit to 5 decisions
|
||||
|
||||
|
||||
class CompressionEvaluator:
|
||||
"""Evaluate compression quality using probes and LLM judge.
|
||||
|
||||
Use when: comparing compression methods or validating that a specific
|
||||
compression pass preserved critical information. Scores responses
|
||||
across six dimensions (accuracy, context awareness, artifact trail,
|
||||
completeness, continuity, instruction following) and produces an
|
||||
aggregate quality score.
|
||||
|
||||
The evaluate() method is the primary entry point. Call it once per
|
||||
probe, then call get_summary() to retrieve aggregated results.
|
||||
"""
|
||||
|
||||
def __init__(self, model: str = "gpt-5.2") -> None:
|
||||
self.model = model
|
||||
self.results: List[EvaluationResult] = []
|
||||
|
||||
def evaluate(self,
|
||||
probe: Probe,
|
||||
response: str,
|
||||
compressed_context: str) -> EvaluationResult:
|
||||
"""Evaluate a single probe response against the rubric.
|
||||
|
||||
Use when: scoring how well a model's response (given compressed
|
||||
context) answers a probe question. Returns per-criterion scores,
|
||||
per-dimension aggregates, and an overall score.
|
||||
|
||||
Args:
|
||||
probe: The probe question with expected ground truth.
|
||||
response: The model's response to evaluate.
|
||||
compressed_context: The compressed context that was provided
|
||||
to the model when generating the response.
|
||||
|
||||
Returns:
|
||||
EvaluationResult with scores and reasoning across all
|
||||
applicable dimensions.
|
||||
"""
|
||||
# Get relevant criteria based on probe type
|
||||
criteria = self._get_criteria_for_probe(probe.probe_type)
|
||||
|
||||
# Evaluate each criterion
|
||||
criterion_results: List[CriterionResult] = []
|
||||
for criterion in criteria:
|
||||
result = self._evaluate_criterion(
|
||||
criterion,
|
||||
probe,
|
||||
response,
|
||||
compressed_context
|
||||
)
|
||||
criterion_results.append(result)
|
||||
|
||||
# Calculate dimension scores
|
||||
dimension_scores = self._calculate_dimension_scores(criterion_results)
|
||||
|
||||
# Calculate aggregate score
|
||||
aggregate_score = sum(dimension_scores.values()) / len(dimension_scores) if dimension_scores else 0.0
|
||||
|
||||
result = EvaluationResult(
|
||||
probe=probe,
|
||||
response=response,
|
||||
criterion_results=criterion_results,
|
||||
aggregate_score=aggregate_score,
|
||||
dimension_scores=dimension_scores
|
||||
)
|
||||
|
||||
self.results.append(result)
|
||||
return result
|
||||
|
||||
def get_summary(self) -> Dict:
|
||||
"""Get summary of all evaluation results.
|
||||
|
||||
Use when: all probes have been evaluated and an aggregate
|
||||
report is needed to compare methods or make a go/no-go
|
||||
decision on a compression strategy.
|
||||
|
||||
Returns:
|
||||
Dictionary with total evaluations, average score,
|
||||
per-dimension averages, and weakest/strongest dimensions.
|
||||
"""
|
||||
if not self.results:
|
||||
return {"error": "No evaluations performed"}
|
||||
|
||||
avg_score = sum(r.aggregate_score for r in self.results) / len(self.results)
|
||||
|
||||
# Average dimension scores
|
||||
dimension_totals: Dict[str, float] = {}
|
||||
dimension_counts: Dict[str, int] = {}
|
||||
|
||||
for result in self.results:
|
||||
for dim, score in result.dimension_scores.items():
|
||||
dimension_totals[dim] = dimension_totals.get(dim, 0) + score
|
||||
dimension_counts[dim] = dimension_counts.get(dim, 0) + 1
|
||||
|
||||
avg_dimensions = {
|
||||
dim: dimension_totals[dim] / dimension_counts[dim]
|
||||
for dim in dimension_totals
|
||||
}
|
||||
|
||||
return {
|
||||
"total_evaluations": len(self.results),
|
||||
"average_score": avg_score,
|
||||
"dimension_averages": avg_dimensions,
|
||||
"weakest_dimension": min(avg_dimensions, key=avg_dimensions.get) if avg_dimensions else None,
|
||||
"strongest_dimension": max(avg_dimensions, key=avg_dimensions.get) if avg_dimensions else None,
|
||||
}
|
||||
|
||||
def _get_criteria_for_probe(self, probe_type: ProbeType) -> List[Dict]:
|
||||
"""Get relevant criteria for probe type."""
|
||||
criteria: List[Dict] = []
|
||||
|
||||
# All probes get accuracy and completeness
|
||||
criteria.extend(RUBRIC_CRITERIA["accuracy"])
|
||||
criteria.extend(RUBRIC_CRITERIA["completeness"])
|
||||
|
||||
# Add type-specific criteria
|
||||
if probe_type == ProbeType.ARTIFACT:
|
||||
criteria.extend(RUBRIC_CRITERIA["artifact_trail"])
|
||||
elif probe_type == ProbeType.CONTINUATION:
|
||||
criteria.extend(RUBRIC_CRITERIA["continuity"])
|
||||
elif probe_type == ProbeType.RECALL:
|
||||
criteria.extend(RUBRIC_CRITERIA["context_awareness"])
|
||||
elif probe_type == ProbeType.DECISION:
|
||||
criteria.extend(RUBRIC_CRITERIA["context_awareness"])
|
||||
criteria.extend(RUBRIC_CRITERIA["continuity"])
|
||||
|
||||
criteria.extend(RUBRIC_CRITERIA["instruction_following"])
|
||||
|
||||
return criteria
|
||||
|
||||
def _evaluate_criterion(self,
|
||||
criterion: Dict,
|
||||
probe: Probe,
|
||||
response: str,
|
||||
context: str) -> CriterionResult:
|
||||
"""
|
||||
Evaluate a single criterion using LLM judge.
|
||||
|
||||
PRODUCTION NOTE: This is a stub implementation.
|
||||
Production systems should call the actual LLM API:
|
||||
|
||||
```python
|
||||
result = openai.chat.completions.create(
|
||||
model="gpt-5.2",
|
||||
messages=[
|
||||
{"role": "system", "content": JUDGE_SYSTEM_PROMPT},
|
||||
{"role": "user", "content": self._format_judge_input(criterion, probe, response, context)}
|
||||
]
|
||||
)
|
||||
return self._parse_judge_output(result)
|
||||
```
|
||||
"""
|
||||
# Stub implementation - in production, call LLM judge
|
||||
score = self._heuristic_score(criterion, response, probe.ground_truth)
|
||||
reasoning = f"Evaluated {criterion['id']} based on response content."
|
||||
|
||||
return CriterionResult(
|
||||
criterion_id=criterion["id"],
|
||||
score=score,
|
||||
reasoning=reasoning
|
||||
)
|
||||
|
||||
def _heuristic_score(self,
|
||||
criterion: Dict,
|
||||
response: str,
|
||||
ground_truth: Optional[str]) -> float:
|
||||
"""
|
||||
Heuristic scoring for demonstration.
|
||||
|
||||
Production systems should use LLM judge instead.
|
||||
"""
|
||||
score = 3.0 # Base score
|
||||
|
||||
# Adjust based on response length and content
|
||||
if len(response) < 50:
|
||||
score -= 1.0 # Too short
|
||||
elif len(response) > 500:
|
||||
score += 0.5 # Detailed
|
||||
|
||||
# Check for technical content
|
||||
if any(ext in response for ext in [".ts", ".py", ".js", ".md"]):
|
||||
score += 0.5 # Contains file references
|
||||
|
||||
overlap_ratio = self._ground_truth_overlap_ratio(response, ground_truth)
|
||||
if overlap_ratio >= 0.75:
|
||||
score += 1.0
|
||||
elif overlap_ratio >= 0.4:
|
||||
score += 0.5
|
||||
elif ground_truth:
|
||||
score -= 0.5
|
||||
|
||||
return min(5.0, max(0.0, score))
|
||||
|
||||
def _ground_truth_overlap_ratio(self,
|
||||
response: str,
|
||||
ground_truth: Optional[str]) -> float:
|
||||
if not ground_truth:
|
||||
return 0.0
|
||||
|
||||
terms = self._extract_ground_truth_terms(ground_truth)
|
||||
if not terms:
|
||||
return 1.0 if ground_truth.lower() in response.lower() else 0.0
|
||||
|
||||
response_lower = response.lower()
|
||||
matches = sum(1 for term in terms if term in response_lower)
|
||||
return matches / len(terms)
|
||||
|
||||
def _extract_ground_truth_terms(self, ground_truth: str) -> List[str]:
|
||||
try:
|
||||
parsed = json.loads(ground_truth)
|
||||
except json.JSONDecodeError:
|
||||
return [ground_truth.lower()] if ground_truth.strip() else []
|
||||
|
||||
terms: List[str] = []
|
||||
|
||||
def collect(value) -> None:
|
||||
if isinstance(value, str):
|
||||
normalized = value.strip().lower()
|
||||
if normalized:
|
||||
terms.append(normalized)
|
||||
elif isinstance(value, dict):
|
||||
for nested in value.values():
|
||||
collect(nested)
|
||||
elif isinstance(value, list):
|
||||
for nested in value:
|
||||
collect(nested)
|
||||
|
||||
collect(parsed)
|
||||
return list(dict.fromkeys(terms))
|
||||
|
||||
def _calculate_dimension_scores(self,
|
||||
criterion_results: List[CriterionResult]) -> Dict[str, float]:
|
||||
"""Calculate dimension scores from criterion results."""
|
||||
dimension_scores: Dict[str, float] = {}
|
||||
|
||||
for dimension, criteria in RUBRIC_CRITERIA.items():
|
||||
criterion_ids = [c["id"] for c in criteria]
|
||||
relevant_results = [
|
||||
r for r in criterion_results
|
||||
if r.criterion_id in criterion_ids
|
||||
]
|
||||
|
||||
if relevant_results:
|
||||
# Weighted average
|
||||
total_weight = sum(
|
||||
c["weight"] for c in criteria
|
||||
if c["id"] in [r.criterion_id for r in relevant_results]
|
||||
)
|
||||
weighted_sum = sum(
|
||||
r.score * next(c["weight"] for c in criteria if c["id"] == r.criterion_id)
|
||||
for r in relevant_results
|
||||
)
|
||||
dimension_scores[dimension] = weighted_sum / total_weight if total_weight > 0 else 0.0
|
||||
|
||||
return dimension_scores
|
||||
|
||||
|
||||
class StructuredSummarizer:
|
||||
"""Generate structured summaries with explicit sections.
|
||||
|
||||
Use when: implementing anchored iterative summarization for
|
||||
long-running coding sessions. Maintains a persistent summary
|
||||
with dedicated sections for session intent, file modifications,
|
||||
decisions, current state, and next steps.
|
||||
|
||||
Call update_from_span() each time a new content span is truncated.
|
||||
The summarizer merges new information into existing sections rather
|
||||
than regenerating, preventing cumulative detail loss.
|
||||
"""
|
||||
|
||||
TEMPLATE = """## Session Intent
|
||||
{intent}
|
||||
|
||||
## Files Modified
|
||||
{files_modified}
|
||||
|
||||
## Files Read (Not Modified)
|
||||
{files_read}
|
||||
|
||||
## Decisions Made
|
||||
{decisions}
|
||||
|
||||
## Current State
|
||||
{current_state}
|
||||
|
||||
## Next Steps
|
||||
{next_steps}
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.sections: Dict = {
|
||||
"intent": "",
|
||||
"files_modified": [],
|
||||
"files_read": [],
|
||||
"decisions": [],
|
||||
"current_state": "",
|
||||
"next_steps": []
|
||||
}
|
||||
|
||||
def update_from_span(self, new_content: str) -> str:
|
||||
"""Update summary from newly truncated content span.
|
||||
|
||||
Use when: a compression trigger fires and a portion of
|
||||
conversation history is about to be discarded. Pass the
|
||||
content that will be truncated; the summarizer extracts
|
||||
structured information and merges it with prior state.
|
||||
|
||||
Args:
|
||||
new_content: The conversation span being truncated.
|
||||
|
||||
Returns:
|
||||
Formatted summary string with all sections populated.
|
||||
"""
|
||||
# Extract information from new content
|
||||
new_info = self._extract_from_content(new_content)
|
||||
|
||||
# Merge with existing sections
|
||||
self._merge_sections(new_info)
|
||||
|
||||
# Generate formatted summary
|
||||
return self._format_summary()
|
||||
|
||||
def _extract_from_content(self, content: str) -> Dict:
|
||||
"""Extract structured information from content."""
|
||||
extracted: Dict = {
|
||||
"intent": "",
|
||||
"files_modified": [],
|
||||
"files_read": [],
|
||||
"decisions": [],
|
||||
"current_state": "",
|
||||
"next_steps": []
|
||||
}
|
||||
|
||||
# Extract file modifications
|
||||
mod_pattern = r"(?:modified|changed|updated|fixed)\s+([^\s]+\.[a-z]+)[:\s]*(.+?)(?:\n|$)"
|
||||
for match in re.finditer(mod_pattern, content, re.IGNORECASE):
|
||||
extracted["files_modified"].append({
|
||||
"path": match.group(1),
|
||||
"change": match.group(2).strip()[:100]
|
||||
})
|
||||
|
||||
# Extract file reads
|
||||
read_pattern = r"(?:read|examined|opened|checked)\s+([^\s]+\.[a-z]+)"
|
||||
for match in re.finditer(read_pattern, content, re.IGNORECASE):
|
||||
file_path = match.group(1)
|
||||
if file_path not in [f["path"] for f in extracted["files_modified"]]:
|
||||
extracted["files_read"].append(file_path)
|
||||
|
||||
# Extract decisions
|
||||
decision_pattern = r"(?:decided|chose|going with|will use)\s+(.+?)(?:\n|$)"
|
||||
for match in re.finditer(decision_pattern, content, re.IGNORECASE):
|
||||
extracted["decisions"].append(match.group(1).strip()[:150])
|
||||
|
||||
return extracted
|
||||
|
||||
def _merge_sections(self, new_info: Dict) -> None:
|
||||
"""Merge new information with existing sections."""
|
||||
# Update intent if empty
|
||||
if new_info["intent"] and not self.sections["intent"]:
|
||||
self.sections["intent"] = new_info["intent"]
|
||||
|
||||
# Merge file lists (deduplicate by path)
|
||||
existing_mod_paths = [f["path"] for f in self.sections["files_modified"]]
|
||||
for file_info in new_info["files_modified"]:
|
||||
if file_info["path"] not in existing_mod_paths:
|
||||
self.sections["files_modified"].append(file_info)
|
||||
|
||||
# Merge read files
|
||||
for file_path in new_info["files_read"]:
|
||||
if file_path not in self.sections["files_read"]:
|
||||
self.sections["files_read"].append(file_path)
|
||||
|
||||
# Append decisions
|
||||
self.sections["decisions"].extend(new_info["decisions"])
|
||||
|
||||
# Update current state (latest wins)
|
||||
if new_info["current_state"]:
|
||||
self.sections["current_state"] = new_info["current_state"]
|
||||
|
||||
# Merge next steps
|
||||
self.sections["next_steps"].extend(new_info["next_steps"])
|
||||
|
||||
def _format_summary(self) -> str:
|
||||
"""Format sections into summary string."""
|
||||
files_modified_str = "\n".join(
|
||||
f"- {f['path']}: {f['change']}"
|
||||
for f in self.sections["files_modified"]
|
||||
) or "None"
|
||||
|
||||
files_read_str = "\n".join(
|
||||
f"- {f}" for f in self.sections["files_read"]
|
||||
) or "None"
|
||||
|
||||
decisions_str = "\n".join(
|
||||
f"- {d}" for d in self.sections["decisions"][-5:] # Keep last 5
|
||||
) or "None"
|
||||
|
||||
next_steps_str = "\n".join(
|
||||
f"{i+1}. {s}" for i, s in enumerate(self.sections["next_steps"][-5:])
|
||||
) or "None"
|
||||
|
||||
return self.TEMPLATE.format(
|
||||
intent=self.sections["intent"] or "Not specified",
|
||||
files_modified=files_modified_str,
|
||||
files_read=files_read_str,
|
||||
decisions=decisions_str,
|
||||
current_state=self.sections["current_state"] or "In progress",
|
||||
next_steps=next_steps_str
|
||||
)
|
||||
|
||||
|
||||
def evaluate_compression_quality(
|
||||
original_history: str,
|
||||
compressed_context: str,
|
||||
model_response_fn: Callable[[str, str], str],
|
||||
) -> Dict:
|
||||
"""Evaluate compression quality for a conversation end-to-end.
|
||||
|
||||
Use when: running a one-shot quality check on a compression pass.
|
||||
Generates probes from original history, collects model responses
|
||||
using the compressed context, evaluates each response, and returns
|
||||
a scored summary with actionable recommendations.
|
||||
|
||||
Args:
|
||||
original_history: The full conversation before compression.
|
||||
compressed_context: The compressed version to evaluate.
|
||||
model_response_fn: Callable that takes (compressed_context, question)
|
||||
and returns the model's response string.
|
||||
|
||||
Returns:
|
||||
Dictionary with total evaluations, average score, per-dimension
|
||||
averages, weakest/strongest dimensions, and recommendations list.
|
||||
"""
|
||||
# Generate probes
|
||||
generator = ProbeGenerator(original_history)
|
||||
probes = generator.generate_probes()
|
||||
|
||||
# Evaluate each probe
|
||||
evaluator = CompressionEvaluator()
|
||||
|
||||
for probe in probes:
|
||||
# Get model response using compressed context
|
||||
response = model_response_fn(compressed_context, probe.question)
|
||||
|
||||
# Evaluate response
|
||||
evaluator.evaluate(probe, response, compressed_context)
|
||||
|
||||
# Get summary
|
||||
summary = evaluator.get_summary()
|
||||
|
||||
# Add recommendations
|
||||
summary["recommendations"] = []
|
||||
|
||||
if summary.get("weakest_dimension") == "artifact_trail":
|
||||
summary["recommendations"].append(
|
||||
"Consider implementing separate artifact tracking outside compression"
|
||||
)
|
||||
|
||||
if summary.get("average_score", 0) < 3.5:
|
||||
summary["recommendations"].append(
|
||||
"Compression quality is below threshold - consider less aggressive compression"
|
||||
)
|
||||
|
||||
return summary
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Demo: generate probes and evaluate a sample compression
|
||||
|
||||
sample_history = """
|
||||
User reported error: 401 Unauthorized on /api/auth/login endpoint.
|
||||
Examined auth.controller.ts - JWT generation looks correct.
|
||||
Examined middleware/cors.ts - no issues found.
|
||||
Modified config/redis.ts: Fixed connection pooling configuration.
|
||||
Modified services/session.service.ts: Added retry logic for transient failures.
|
||||
Decided to use Redis connection pool instead of per-request connections.
|
||||
Modified tests/auth.test.ts: Updated mock setup for new config.
|
||||
14 tests passing, 2 failing (mock setup issues).
|
||||
Next: Fix remaining test failures in session service mocks.
|
||||
"""
|
||||
|
||||
sample_compressed = """
|
||||
## Session Intent
|
||||
Debug 401 Unauthorized on /api/auth/login.
|
||||
|
||||
## Root Cause
|
||||
Stale Redis connection in session store.
|
||||
|
||||
## Files Modified
|
||||
- config/redis.ts: Fixed connection pooling
|
||||
- services/session.service.ts: Added retry logic
|
||||
- tests/auth.test.ts: Updated mock setup
|
||||
|
||||
## Test Status
|
||||
14 passing, 2 failing
|
||||
|
||||
## Next Steps
|
||||
1. Fix remaining test failures
|
||||
"""
|
||||
|
||||
# Stub model response function
|
||||
def mock_model_response(context: str, question: str) -> str:
|
||||
if "error" in question.lower():
|
||||
return "The original error was a 401 Unauthorized on /api/auth/login."
|
||||
if "files" in question.lower():
|
||||
return "Modified config/redis.ts, services/session.service.ts, tests/auth.test.ts."
|
||||
if "next" in question.lower():
|
||||
return "Fix remaining test failures in session service mocks."
|
||||
if "decision" in question.lower():
|
||||
return "Decided to use Redis connection pool instead of per-request connections."
|
||||
return "No specific information available."
|
||||
|
||||
# Run evaluation
|
||||
result = evaluate_compression_quality(
|
||||
original_history=sample_history,
|
||||
compressed_context=sample_compressed,
|
||||
model_response_fn=mock_model_response,
|
||||
)
|
||||
|
||||
print("=== Compression Quality Evaluation ===")
|
||||
print(f"Total evaluations: {result['total_evaluations']}")
|
||||
print(f"Average score: {result['average_score']:.2f}")
|
||||
print()
|
||||
print("Dimension averages:")
|
||||
for dim, score in result.get("dimension_averages", {}).items():
|
||||
print(f" {dim}: {score:.2f}")
|
||||
print()
|
||||
print(f"Weakest dimension: {result.get('weakest_dimension')}")
|
||||
print(f"Strongest dimension: {result.get('strongest_dimension')}")
|
||||
print()
|
||||
if result.get("recommendations"):
|
||||
print("Recommendations:")
|
||||
for rec in result["recommendations"]:
|
||||
print(f" - {rec}")
|
||||
else:
|
||||
print("No recommendations - compression quality looks acceptable.")
|
||||
@@ -0,0 +1,56 @@
|
||||
import importlib.util
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
MODULE_PATH = (
|
||||
Path(__file__).resolve().parents[1] / "scripts" / "compression_evaluator.py"
|
||||
)
|
||||
MODULE_SPEC = importlib.util.spec_from_file_location(
|
||||
"compression_evaluator", MODULE_PATH
|
||||
)
|
||||
if MODULE_SPEC is None or MODULE_SPEC.loader is None:
|
||||
raise RuntimeError(f"Unable to load compression_evaluator.py from {MODULE_PATH}")
|
||||
COMPRESSION_EVALUATOR = importlib.util.module_from_spec(MODULE_SPEC)
|
||||
MODULE_SPEC.loader.exec_module(COMPRESSION_EVALUATOR)
|
||||
|
||||
|
||||
class CompressionEvaluatorTests(unittest.TestCase):
|
||||
def test_json_ground_truth_terms_score_when_response_mentions_artifacts(
|
||||
self,
|
||||
) -> None:
|
||||
evaluator = COMPRESSION_EVALUATOR.CompressionEvaluator()
|
||||
|
||||
rich_score = evaluator._heuristic_score(
|
||||
{"id": "artifact_files_modified"},
|
||||
"We modified src/app.py and updated README.md during the session.",
|
||||
'[{"path": "src/app.py", "operation": "modified"}, {"path": "README.md", "operation": "updated"}]',
|
||||
)
|
||||
poor_score = evaluator._heuristic_score(
|
||||
{"id": "artifact_files_modified"},
|
||||
"We changed some files but I do not remember which ones.",
|
||||
'[{"path": "src/app.py", "operation": "modified"}, {"path": "README.md", "operation": "updated"}]',
|
||||
)
|
||||
|
||||
self.assertGreater(rich_score, poor_score)
|
||||
self.assertGreaterEqual(rich_score, 4.0)
|
||||
|
||||
def test_plain_text_ground_truth_still_uses_substring_match(self) -> None:
|
||||
evaluator = COMPRESSION_EVALUATOR.CompressionEvaluator()
|
||||
|
||||
exact_score = evaluator._heuristic_score(
|
||||
{"id": "continuity_work_state"},
|
||||
"Next: fix the websocket timeout before rerunning tests.",
|
||||
"fix the websocket timeout",
|
||||
)
|
||||
missing_score = evaluator._heuristic_score(
|
||||
{"id": "continuity_work_state"},
|
||||
"Next: inspect logs again.",
|
||||
"fix the websocket timeout",
|
||||
)
|
||||
|
||||
self.assertGreater(exact_score, missing_score)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user