chore: import upstream snapshot with attribution
Continuous Integration / Pre-commit Linter (push) Has been cancelled
Continuous Integration / Mypy Check (Python 3.10) (push) Has been cancelled
Continuous Integration / Mypy Check (Python 3.11) (push) Has been cancelled
Continuous Integration / Mypy Check (Python 3.12) (push) Has been cancelled
Continuous Integration / Mypy Check (Python 3.13) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.10) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.11) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.12) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.13) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.14) (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.10) (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.11) (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.12) (push) Has been cancelled
Copybara PR Handler / close-imported-pr (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.13) (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.14) (push) Has been cancelled
Continuous Integration / Pre-commit Linter (push) Has been cancelled
Continuous Integration / Mypy Check (Python 3.10) (push) Has been cancelled
Continuous Integration / Mypy Check (Python 3.11) (push) Has been cancelled
Continuous Integration / Mypy Check (Python 3.12) (push) Has been cancelled
Continuous Integration / Mypy Check (Python 3.13) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.10) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.11) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.12) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.13) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.14) (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.10) (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.11) (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.12) (push) Has been cancelled
Copybara PR Handler / close-imported-pr (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.13) (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.14) (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,78 @@
|
||||
# Log Probabilities Demo Agent
|
||||
|
||||
## Overview
|
||||
|
||||
This sample demonstrates how to access and display log probabilities from language model responses using the `avg_logprobs` and `logprobs_result` fields in `LlmResponse`. It shows how to configure an ADK agent to request log probabilities and how to use an `after_model_callback` to analyze and append confidence metrics to the response.
|
||||
|
||||
## Sample Inputs
|
||||
|
||||
- `What is the capital of France?`
|
||||
|
||||
*A factual, straightforward question. The agent will answer confidently (e.g., "Paris"), resulting in a high average log probability and confidence score near 100%.*
|
||||
|
||||
- `What are the philosophical implications of artificial consciousness?`
|
||||
|
||||
*A complex, open-ended question. The agent will provide a nuanced answer with varied vocabulary, resulting in a lower average log probability and medium/low confidence score.*
|
||||
|
||||
## Graph
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
User[User Input] --> RootAgent[root_agent: logprobs_demo_agent]
|
||||
RootAgent --> Callback[after_model_callback: append_logprobs_to_response]
|
||||
Callback -- Appended Logprobs Analysis --> Response[User Response]
|
||||
```
|
||||
|
||||
## How To
|
||||
|
||||
### 1. Enabling Log Probabilities
|
||||
|
||||
To enable log probability collection, configure `generate_content_config` on the `Agent` using `types.GenerateContentConfig`:
|
||||
|
||||
```python
|
||||
from google.genai import types
|
||||
|
||||
root_agent = Agent(
|
||||
name="logprobs_demo_agent",
|
||||
generate_content_config=types.GenerateContentConfig(
|
||||
response_logprobs=True, # Enable log probability collection
|
||||
logprobs=5, # Collect top 5 alternatives for analysis
|
||||
temperature=0.7,
|
||||
),
|
||||
after_model_callback=append_logprobs_to_response,
|
||||
)
|
||||
```
|
||||
|
||||
### 2. Extracting Log Probabilities in a Callback
|
||||
|
||||
The `after_model_callback` receives the `LlmResponse` object, which contains the `avg_logprobs` and `logprobs_result` fields. You can use this data for confidence analysis, quality filtering, or appending information to the response content:
|
||||
|
||||
```python
|
||||
async def append_logprobs_to_response(
|
||||
callback_context: CallbackContext, llm_response: LlmResponse
|
||||
) -> LlmResponse:
|
||||
if llm_response.avg_logprobs is not None:
|
||||
print(f"📊 Average log probability: {llm_response.avg_logprobs:.4f}")
|
||||
|
||||
# Analyze confidence
|
||||
confidence_level = (
|
||||
"High" if llm_response.avg_logprobs >= -0.5
|
||||
else "Medium" if llm_response.avg_logprobs >= -1.0
|
||||
else "Low"
|
||||
)
|
||||
|
||||
# Access detailed candidates
|
||||
if llm_response.logprobs_result and llm_response.logprobs_result.top_candidates:
|
||||
num_candidates = len(llm_response.logprobs_result.top_candidates)
|
||||
|
||||
return llm_response
|
||||
```
|
||||
|
||||
### 3. Understanding Log Probabilities
|
||||
|
||||
- **Range**: -∞ to 0 (0 = 100% confident, -1 ≈ 37% confident, -2 ≈ 14% confident)
|
||||
- **Confidence Levels**:
|
||||
- **High**: `>= -0.5` (typically factual, straightforward responses)
|
||||
- **Medium**: `-1.0` to `-0.5` (reasonably confident responses)
|
||||
- **Low**: `< -1.0` (uncertain or complex responses)
|
||||
- **Use Cases**: Quality control, uncertainty detection, and response filtering.
|
||||
@@ -0,0 +1,15 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from . import agent
|
||||
@@ -0,0 +1,116 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Sample agent demonstrating log probability usage.
|
||||
|
||||
This agent shows how to access log probabilities from language model responses.
|
||||
The after_model_callback appends confidence information to demonstrate how
|
||||
logprobs can be extracted and used.
|
||||
"""
|
||||
|
||||
from google.adk.agents.callback_context import CallbackContext
|
||||
from google.adk.agents.llm_agent import Agent
|
||||
from google.adk.models.llm_response import LlmResponse
|
||||
from google.genai import types
|
||||
|
||||
|
||||
async def append_logprobs_to_response(
|
||||
callback_context: CallbackContext, llm_response: LlmResponse
|
||||
) -> LlmResponse:
|
||||
"""After-model callback that appends log probability information to response.
|
||||
|
||||
This callback demonstrates how to access avg_logprobs and logprobs_result
|
||||
from the LlmResponse and append the information to the response content.
|
||||
|
||||
Args:
|
||||
callback_context: The current callback context
|
||||
llm_response: The LlmResponse containing logprobs data
|
||||
|
||||
Returns:
|
||||
Modified LlmResponse with logprobs information appended
|
||||
"""
|
||||
# Build log probability analysis
|
||||
if llm_response.avg_logprobs is None:
|
||||
print("⚠️ No log probability data available")
|
||||
logprobs_info = """
|
||||
|
||||
---
|
||||
|
||||
### 📊 Log Probability Analysis
|
||||
|
||||
⚠️ *No log probability data available*"""
|
||||
else:
|
||||
print(f"📊 Average log probability: {llm_response.avg_logprobs:.4f}")
|
||||
|
||||
# Build confidence analysis
|
||||
confidence_level = (
|
||||
"High"
|
||||
if llm_response.avg_logprobs >= -0.5
|
||||
else "Medium"
|
||||
if llm_response.avg_logprobs >= -1.0
|
||||
else "Low"
|
||||
)
|
||||
|
||||
logprobs_info = f"""
|
||||
|
||||
---
|
||||
|
||||
### 📊 Log Probability Analysis
|
||||
|
||||
* **Average Log Probability**: {llm_response.avg_logprobs:.4f}
|
||||
* **Confidence Level**: {confidence_level}
|
||||
* **Confidence Score**: {100 * (2 ** llm_response.avg_logprobs):.1f}%"""
|
||||
|
||||
# Optionally include detailed logprobs_result information
|
||||
if (
|
||||
llm_response.logprobs_result
|
||||
and llm_response.logprobs_result.top_candidates
|
||||
):
|
||||
logprobs_info += (
|
||||
"\n* **Top Alternatives Analyzed**:"
|
||||
f" {len(llm_response.logprobs_result.top_candidates)}"
|
||||
)
|
||||
|
||||
# Append logprobs analysis to the response
|
||||
if llm_response.content and llm_response.content.parts:
|
||||
if not any(
|
||||
"### 📊 Log Probability Analysis" in p.text
|
||||
for p in llm_response.content.parts
|
||||
if p.text
|
||||
):
|
||||
llm_response.content.parts.append(types.Part(text=logprobs_info))
|
||||
|
||||
return llm_response
|
||||
|
||||
|
||||
# Create a simple agent that demonstrates logprobs usage
|
||||
root_agent = Agent(
|
||||
name="logprobs_demo_agent",
|
||||
description=(
|
||||
"A simple agent that demonstrates log probability extraction and"
|
||||
" display."
|
||||
),
|
||||
instruction="""
|
||||
You are a helpful AI assistant. Answer user questions normally and naturally.
|
||||
|
||||
After you respond, you'll see log probability analysis appended to your response.
|
||||
You don't need to include the log probability analysis in your response yourself.
|
||||
""",
|
||||
generate_content_config=types.GenerateContentConfig(
|
||||
response_logprobs=True, # Enable log probability collection
|
||||
logprobs=5, # Collect top 5 alternatives for analysis
|
||||
temperature=0.7, # Moderate temperature for varied responses
|
||||
),
|
||||
after_model_callback=append_logprobs_to_response,
|
||||
)
|
||||
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"events": [
|
||||
{
|
||||
"author": "user",
|
||||
"content": {
|
||||
"parts": [
|
||||
{
|
||||
"text": "Hello"
|
||||
}
|
||||
],
|
||||
"role": "user"
|
||||
},
|
||||
"id": "e-1",
|
||||
"invocationId": "i-1",
|
||||
"nodeInfo": {
|
||||
"path": ""
|
||||
}
|
||||
},
|
||||
{
|
||||
"author": "logprobs_demo_agent",
|
||||
"content": {
|
||||
"parts": [
|
||||
{
|
||||
"text": "Hello there! How can I help you today?"
|
||||
},
|
||||
{
|
||||
"text": "\n\n---\n\n### \ud83d\udcca Log Probability Analysis\n\n* **Average Log Probability**: -0.1564\n* **Confidence Level**: High\n* **Confidence Score**: 89.7%\n* **Top Alternatives Analyzed**: 10"
|
||||
}
|
||||
],
|
||||
"role": "model"
|
||||
},
|
||||
"finishReason": "STOP",
|
||||
"id": "e-2",
|
||||
"invocationId": "i-1",
|
||||
"nodeInfo": {
|
||||
"path": "logprobs_demo_agent@1"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"events": [
|
||||
{
|
||||
"author": "user",
|
||||
"content": {
|
||||
"parts": [
|
||||
{
|
||||
"text": "Tell me a joke"
|
||||
}
|
||||
],
|
||||
"role": "user"
|
||||
},
|
||||
"id": "e-1",
|
||||
"invocationId": "i-1",
|
||||
"nodeInfo": {
|
||||
"path": ""
|
||||
}
|
||||
},
|
||||
{
|
||||
"author": "logprobs_demo_agent",
|
||||
"content": {
|
||||
"parts": [
|
||||
{
|
||||
"text": "Why don't scientists trust atoms?\n\nBecause they make up everything!"
|
||||
},
|
||||
{
|
||||
"text": "\n\n---\n\n### \ud83d\udcca Log Probability Analysis\n\n* **Average Log Probability**: -0.4411\n* **Confidence Level**: High\n* **Confidence Score**: 73.7%\n* **Top Alternatives Analyzed**: 15"
|
||||
}
|
||||
],
|
||||
"role": "model"
|
||||
},
|
||||
"finishReason": "STOP",
|
||||
"id": "e-2",
|
||||
"invocationId": "i-1",
|
||||
"nodeInfo": {
|
||||
"path": "logprobs_demo_agent@1"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"events": [
|
||||
{
|
||||
"author": "user",
|
||||
"content": {
|
||||
"parts": [
|
||||
{
|
||||
"text": "What is AI?"
|
||||
}
|
||||
],
|
||||
"role": "user"
|
||||
},
|
||||
"id": "e-1",
|
||||
"invocationId": "i-1",
|
||||
"nodeInfo": {
|
||||
"path": ""
|
||||
}
|
||||
},
|
||||
{
|
||||
"author": "logprobs_demo_agent",
|
||||
"content": {
|
||||
"parts": [
|
||||
{
|
||||
"text": "Artificial Intelligence (AI) refers to the simulation of human intelligence processes by machines, especially computer systems. These processes include learning (the acquisition of information and rules for using the information), reasoning (using rules to reach approximate or definite conclusions), and self-correction.\n\nThe ultimate goal of AI is to enable machines to perform tasks that typically require human intelligence, such as understanding natural language, recognizing patterns, solving problems, making decisions, and even learning from experience."
|
||||
},
|
||||
{
|
||||
"text": "\n\n---\n\n### \ud83d\udcca Log Probability Analysis\n\n* **Average Log Probability**: -0.1382\n* **Confidence Level**: High\n* **Confidence Score**: 90.9%\n* **Top Alternatives Analyzed**: 92"
|
||||
}
|
||||
],
|
||||
"role": "model"
|
||||
},
|
||||
"finishReason": "STOP",
|
||||
"id": "e-2",
|
||||
"invocationId": "i-1",
|
||||
"nodeInfo": {
|
||||
"path": "logprobs_demo_agent@1"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user