chore: import upstream snapshot with attribution
Continuous Integration / Pre-commit Linter (push) Has been cancelled
Continuous Integration / Mypy Check (Python 3.10) (push) Has been cancelled
Continuous Integration / Mypy Check (Python 3.11) (push) Has been cancelled
Continuous Integration / Mypy Check (Python 3.12) (push) Has been cancelled
Continuous Integration / Mypy Check (Python 3.13) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.10) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.11) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.12) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.13) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.14) (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.10) (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.11) (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.12) (push) Has been cancelled
Copybara PR Handler / close-imported-pr (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.13) (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.14) (push) Has been cancelled
Continuous Integration / Pre-commit Linter (push) Has been cancelled
Continuous Integration / Mypy Check (Python 3.10) (push) Has been cancelled
Continuous Integration / Mypy Check (Python 3.11) (push) Has been cancelled
Continuous Integration / Mypy Check (Python 3.12) (push) Has been cancelled
Continuous Integration / Mypy Check (Python 3.13) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.10) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.11) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.12) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.13) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.14) (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.10) (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.11) (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.12) (push) Has been cancelled
Copybara PR Handler / close-imported-pr (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.13) (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.14) (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,131 @@
|
||||
# Cache Analysis Research Assistant
|
||||
|
||||
## Overview
|
||||
|
||||
This sample demonstrates ADK context caching features using a comprehensive research assistant agent designed to test both Gemini 2.0 Flash and 2.5 Flash context caching capabilities. The sample showcases the difference between explicit ADK caching and Google's built-in implicit caching.
|
||||
|
||||
### Key Features
|
||||
|
||||
- **App-Level Cache Configuration**: Context cache settings applied at the App level following ADK best practices.
|
||||
- **Large Context Instructions**: Over 4,200 tokens in system instructions to trigger context caching thresholds.
|
||||
- **Comprehensive Tool Suite**: 7 specialized research and analysis tools.
|
||||
- **Multi-Model Support**: Compatible with any Gemini model, automatically adapting the experiment type.
|
||||
- **Performance Metrics**: Detailed token usage tracking, including `cached_content_token_count`.
|
||||
|
||||
## Sample Inputs
|
||||
|
||||
- `Hello, what can you do for me?`
|
||||
|
||||
*General question that does not trigger function calls, serving as a baseline query.*
|
||||
|
||||
- `What is artificial intelligence and how does it work in modern applications?`
|
||||
|
||||
*General question exploring domain knowledge without specific tool requests.*
|
||||
|
||||
- `Use benchmark_performance with system_name='E-commerce Platform', metrics=['latency', 'throughput'], duration='standard', load_profile='realistic'.`
|
||||
|
||||
*Specific request triggering the benchmark_performance tool with explicit parameters.*
|
||||
|
||||
- `Call analyze_user_behavior_patterns with user_segment='premium_customers', time_period='last_30_days', metrics=['engagement', 'conversion'].`
|
||||
|
||||
*Specific request triggering data analysis tools with required parameters.*
|
||||
|
||||
## Graph
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
Agent[Agent: cache_analysis_assistant] --> Tool1[Tool: analyze_data_patterns]
|
||||
Agent --> Tool2[Tool: research_literature]
|
||||
Agent --> Tool3[Tool: generate_test_scenarios]
|
||||
Agent --> Tool4[Tool: benchmark_performance]
|
||||
Agent --> Tool5[Tool: optimize_system_performance]
|
||||
Agent --> Tool6[Tool: analyze_security_vulnerabilities]
|
||||
Agent --> Tool7[Tool: design_scalability_architecture]
|
||||
```
|
||||
|
||||
## How To
|
||||
|
||||
### 1. Cache Configuration
|
||||
|
||||
Context caching is configured at the App level using `ContextCacheConfig`. This ensures that the agent's extensive system instructions and tool definitions are cached for repeated invocations.
|
||||
|
||||
```python
|
||||
from google.adk.agents.context_cache_config import ContextCacheConfig
|
||||
from google.adk.apps.app import App
|
||||
|
||||
cache_analysis_app = App(
|
||||
name="cache_analysis",
|
||||
root_agent=cache_analysis_agent,
|
||||
context_cache_config=ContextCacheConfig(
|
||||
min_tokens=4096,
|
||||
ttl_seconds=600, # 10 minutes for research sessions
|
||||
cache_intervals=3, # Maximum invocations before cache refresh
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
### 2. Run Cache Experiments
|
||||
|
||||
The `run_cache_experiments.py` script automates the execution of prompts and compares caching performance between models:
|
||||
|
||||
```bash
|
||||
# Test any Gemini model - script automatically determines experiment type
|
||||
python run_cache_experiments.py <model_name> --output results.json
|
||||
|
||||
# Examples:
|
||||
python run_cache_experiments.py gemini-2.5-flash --output gemini_2_5_results.json
|
||||
|
||||
# Run multiple iterations for averaged results
|
||||
python run_cache_experiments.py gemini-2.5-flash --repeat 3 --output averaged_results.json
|
||||
```
|
||||
|
||||
### 3. Direct Agent Usage
|
||||
|
||||
You can also run or debug the agent directly using the ADK CLI:
|
||||
|
||||
```bash
|
||||
# Run the agent directly
|
||||
adk run contributing/samples/cache_analysis/agent.py
|
||||
|
||||
# Web interface for debugging
|
||||
adk web contributing/samples/cache_analysis
|
||||
```
|
||||
|
||||
### 4. Experiment Types
|
||||
|
||||
The script automatically adapts the experiment based on the specified model name:
|
||||
|
||||
#### Models with "2.5" (e.g., `gemini-2.5-flash`)
|
||||
|
||||
- **Explicit Caching**: ADK explicit caching + Google's implicit caching.
|
||||
- **Implicit Only**: Google's built-in implicit caching alone.
|
||||
- **Measures**: The added benefit and performance differences of explicit caching over built-in implicit caching.
|
||||
|
||||
#### Other Models (e.g., `gemini-2.0-flash`)
|
||||
|
||||
- **Explicit Caching**: ADK explicit caching enabled.
|
||||
- **Uncached**: Caching completely disabled.
|
||||
- **Measures**: Baseline performance and cost benefits of context caching.
|
||||
|
||||
### 5. Expected Results
|
||||
|
||||
- **Performance Improvements**: Simple text agents typically see a 30-70% latency reduction with caching. Tool-heavy agents may experience slight cache setup overhead but still provide significant cost benefits.
|
||||
- **Cost Savings**: Up to 75% reduction in input token costs for cached content (paying only 25% of normal input cost).
|
||||
- **Token Metrics**: Successful cache hits are indicated by non-zero `cached_content_token_count` values.
|
||||
|
||||
### 6. Troubleshooting
|
||||
|
||||
#### Zero Cached Tokens
|
||||
|
||||
If `cached_content_token_count` is always `0`:
|
||||
|
||||
- Verify model names match exactly (e.g., `gemini-2.5-flash`).
|
||||
- Check that the `min_tokens` threshold (4,096 tokens) is met by the prompt and system instructions.
|
||||
- Ensure proper App-based configuration is used rather than passing standalone agents without App wrappers.
|
||||
|
||||
#### Session Errors
|
||||
|
||||
If you encounter "Session not found" errors:
|
||||
|
||||
- Verify `runner.app_name` is used for session creation.
|
||||
- Ensure correct initialization of `InMemoryRunner` with the `App` object.
|
||||
@@ -0,0 +1,17 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from . import agent
|
||||
|
||||
__all__ = ['agent']
|
||||
@@ -0,0 +1,853 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Cache Analysis Research Assistant Agent.
|
||||
|
||||
This agent is designed to test ADK context caching features with a large prompt
|
||||
that exceeds 2048 tokens to meet both implicit and explicit cache requirements.
|
||||
"""
|
||||
|
||||
import random
|
||||
import time
|
||||
from typing import Any
|
||||
from typing import Dict
|
||||
from typing import List
|
||||
from typing import Optional
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from google.adk import Agent
|
||||
from google.adk.agents.context_cache_config import ContextCacheConfig
|
||||
from google.adk.apps.app import App
|
||||
|
||||
# Load environment variables from .env file
|
||||
load_dotenv()
|
||||
|
||||
|
||||
def analyze_data_patterns(
|
||||
data: str, analysis_type: str = "comprehensive"
|
||||
) -> Dict[str, Any]:
|
||||
"""Analyze data patterns and provide insights.
|
||||
|
||||
This tool performs comprehensive data analysis including statistical analysis,
|
||||
trend identification, anomaly detection, correlation analysis, and predictive
|
||||
modeling. It can handle various data formats including CSV, JSON, XML, and
|
||||
plain text data structures.
|
||||
|
||||
Args:
|
||||
data: The input data to analyze. Can be structured (JSON, CSV) or
|
||||
unstructured text data. For structured data, include column headers
|
||||
and ensure proper formatting. For time series data, include
|
||||
timestamps in ISO format.
|
||||
analysis_type: Type of analysis to perform. Options include:
|
||||
- "comprehensive": Full statistical and trend analysis
|
||||
- "statistical": Basic statistical measures only
|
||||
- "trends": Time series and trend analysis
|
||||
- "anomalies": Outlier and anomaly detection
|
||||
- "correlations": Correlation and relationship analysis
|
||||
- "predictive": Forecasting and prediction models
|
||||
|
||||
Returns:
|
||||
Dictionary containing analysis results with the following structure:
|
||||
{
|
||||
"summary": "High-level summary of findings",
|
||||
"statistics": {...}, # Statistical measures
|
||||
"trends": {...}, # Trend analysis results
|
||||
"anomalies": [...], # List of detected anomalies
|
||||
"correlations": {...}, # Correlation matrix and relationships
|
||||
"predictions": {...}, # Forecasting results if applicable
|
||||
"recommendations": [...] # Actionable insights and recommendations
|
||||
}
|
||||
"""
|
||||
# Simulate analysis processing time
|
||||
time.sleep(0.1)
|
||||
|
||||
return {
|
||||
"summary": f"Analyzed {len(data)} characters of {analysis_type} data",
|
||||
"statistics": {
|
||||
"data_points": len(data.split()),
|
||||
"analysis_type": analysis_type,
|
||||
"processing_time": "0.1 seconds",
|
||||
},
|
||||
"recommendations": [
|
||||
"Continue monitoring data trends",
|
||||
"Consider additional data sources for correlation analysis",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def research_literature(
|
||||
topic: str,
|
||||
sources: Optional[List[str]] = None,
|
||||
depth: str = "comprehensive",
|
||||
time_range: str = "recent",
|
||||
) -> Dict[str, Any]:
|
||||
"""Research academic and professional literature on specified topics.
|
||||
|
||||
This tool performs comprehensive literature research across multiple academic
|
||||
databases, professional journals, conference proceedings, and industry reports.
|
||||
It can analyze research trends, identify key authors and institutions, extract
|
||||
methodological approaches, and synthesize findings across multiple sources.
|
||||
|
||||
The tool supports various research methodologies including systematic reviews,
|
||||
meta-analyses, bibliometric analysis, and citation network analysis. It can
|
||||
identify research gaps, emerging trends, and future research directions in
|
||||
the specified field of study.
|
||||
|
||||
Args:
|
||||
topic: The research topic or query. Can be specific (e.g., "context caching
|
||||
in large language models") or broad (e.g., "machine learning optimization").
|
||||
Use specific keywords and phrases for better results. Boolean operators
|
||||
(AND, OR, NOT) are supported for complex queries.
|
||||
sources: List of preferred sources to search. Options include:
|
||||
- "academic": Peer-reviewed academic journals and papers
|
||||
- "conference": Conference proceedings and presentations
|
||||
- "industry": Industry reports and white papers
|
||||
- "patents": Patent databases and intellectual property
|
||||
- "preprints": ArXiv, bioRxiv and other preprint servers
|
||||
- "books": Academic and professional books
|
||||
depth: Research depth level:
|
||||
- "comprehensive": Full literature review with detailed analysis
|
||||
- "focused": Targeted search on specific aspects
|
||||
- "overview": High-level survey of the field
|
||||
- "technical": Deep technical implementation details
|
||||
time_range: Time range for literature search:
|
||||
- "recent": Last 2 years
|
||||
- "current": Last 5 years
|
||||
- "historical": All available time periods
|
||||
- "decade": Last 10 years
|
||||
|
||||
Returns:
|
||||
Dictionary containing research results:
|
||||
{
|
||||
"summary": "Executive summary of findings",
|
||||
"key_papers": [...], # Most relevant papers found
|
||||
"authors": [...], # Key researchers in the field
|
||||
"institutions": [...], # Leading research institutions
|
||||
"trends": {...}, # Research trends and evolution
|
||||
"methodologies": [...], # Common research approaches
|
||||
"gaps": [...], # Identified research gaps
|
||||
"citations": {...}, # Citation network analysis
|
||||
"recommendations": [...] # Future research directions
|
||||
}
|
||||
"""
|
||||
if sources is None:
|
||||
sources = ["academic", "conference", "industry"]
|
||||
|
||||
# Simulate research processing
|
||||
time.sleep(0.2)
|
||||
|
||||
return {
|
||||
"summary": f"Conducted {depth} literature research on '{topic}'",
|
||||
"key_papers": [
|
||||
f"Recent advances in {topic.lower()}: A systematic review",
|
||||
f"Methodological approaches to {topic.lower()} optimization",
|
||||
f"Future directions in {topic.lower()} research",
|
||||
],
|
||||
"trends": {
|
||||
"emerging_topics": [f"{topic} optimization", f"{topic} scalability"],
|
||||
"methodology_trends": [
|
||||
"experimental validation",
|
||||
"theoretical analysis",
|
||||
],
|
||||
},
|
||||
"recommendations": [
|
||||
f"Focus on practical applications of {topic}",
|
||||
"Consider interdisciplinary approaches",
|
||||
"Investigate scalability challenges",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def generate_test_scenarios(
|
||||
system_type: str,
|
||||
complexity: str = "medium",
|
||||
coverage: Optional[List[str]] = None,
|
||||
constraints: Optional[Dict[str, Any]] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Generate comprehensive test scenarios for system validation.
|
||||
|
||||
This tool creates detailed test scenarios, test cases, and validation protocols
|
||||
for various types of systems including software applications, AI models,
|
||||
distributed systems, and hardware components. It supports multiple testing
|
||||
methodologies including unit testing, integration testing, performance testing,
|
||||
security testing, and user acceptance testing.
|
||||
|
||||
The tool can generate both positive and negative test cases, edge cases,
|
||||
boundary conditions, stress tests, and failure scenarios. It incorporates
|
||||
industry best practices and testing frameworks to ensure comprehensive
|
||||
coverage and reliable validation results.
|
||||
|
||||
Args:
|
||||
system_type: Type of system to test. Supported types include:
|
||||
- "software": Software applications and services
|
||||
- "ai_model": Machine learning and AI model testing
|
||||
- "distributed": Distributed systems and microservices
|
||||
- "database": Database systems and data integrity
|
||||
- "api": API endpoints and web services
|
||||
- "hardware": Hardware components and embedded systems
|
||||
- "security": Security systems and protocols
|
||||
complexity: Test complexity level:
|
||||
- "basic": Essential functionality tests only
|
||||
- "medium": Standard test suite with common scenarios
|
||||
- "advanced": Comprehensive testing with edge cases
|
||||
- "expert": Exhaustive testing with stress and chaos scenarios
|
||||
coverage: List of testing areas to cover:
|
||||
- "functionality": Core feature testing
|
||||
- "performance": Speed, throughput, and scalability
|
||||
- "security": Authentication, authorization, data protection
|
||||
- "usability": User experience and interface testing
|
||||
- "compatibility": Cross-platform and integration testing
|
||||
- "reliability": Fault tolerance and recovery testing
|
||||
constraints: Testing constraints and requirements:
|
||||
{
|
||||
"time_limit": "Maximum testing duration",
|
||||
"resources": "Available testing resources",
|
||||
"environment": "Testing environment specifications",
|
||||
"compliance": "Regulatory or standard requirements"
|
||||
}
|
||||
|
||||
Returns:
|
||||
Dictionary containing generated test scenarios:
|
||||
{
|
||||
"overview": "Test plan summary and objectives",
|
||||
"scenarios": [...], # Detailed test scenarios
|
||||
"test_cases": [...], # Individual test cases
|
||||
"edge_cases": [...], # Boundary and edge conditions
|
||||
"performance_tests": [...], # Performance validation tests
|
||||
"security_tests": [...], # Security and vulnerability tests
|
||||
"automation": {...}, # Test automation recommendations
|
||||
"metrics": {...}, # Success criteria and metrics
|
||||
"schedule": {...} # Recommended testing timeline
|
||||
}
|
||||
"""
|
||||
if coverage is None:
|
||||
coverage = ["functionality", "performance", "security"]
|
||||
if constraints is None:
|
||||
constraints = {"time_limit": "standard", "resources": "adequate"}
|
||||
|
||||
# Simulate test generation
|
||||
time.sleep(0.15)
|
||||
|
||||
num_scenarios = {"basic": 5, "medium": 10, "advanced": 20, "expert": 35}.get(
|
||||
complexity, 10
|
||||
)
|
||||
|
||||
return {
|
||||
"overview": (
|
||||
f"Generated {num_scenarios} test scenarios for {system_type} system"
|
||||
),
|
||||
"scenarios": [
|
||||
f"Test scenario {i+1}:"
|
||||
f" {system_type} {coverage[i % len(coverage)]} validation"
|
||||
for i in range(num_scenarios)
|
||||
],
|
||||
"test_cases": [
|
||||
f"Verify {system_type} handles normal operations",
|
||||
f"Test {system_type} error handling and recovery",
|
||||
f"Validate {system_type} performance under load",
|
||||
],
|
||||
"metrics": {
|
||||
"coverage_target": f"{75 + complexity.index(complexity) * 5}%",
|
||||
"success_criteria": "All critical tests pass",
|
||||
"performance_benchmark": f"{system_type} specific benchmarks",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def optimize_system_performance(
|
||||
system_type: str,
|
||||
current_metrics: Dict[str, Any],
|
||||
target_improvements: Dict[str, Any],
|
||||
constraints: Optional[Dict[str, Any]] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Analyze system performance and provide detailed optimization recommendations.
|
||||
|
||||
This tool performs comprehensive system performance analysis including bottleneck
|
||||
identification, resource utilization assessment, scalability planning, and provides
|
||||
specific optimization strategies tailored to the system type and constraints.
|
||||
|
||||
Args:
|
||||
system_type: Type of system to optimize:
|
||||
- "web_application": Frontend and backend web services
|
||||
- "database": Relational, NoSQL, or distributed databases
|
||||
- "ml_pipeline": Machine learning training and inference systems
|
||||
- "distributed_cache": Caching layers and distributed memory systems
|
||||
- "microservices": Service-oriented architectures
|
||||
- "data_processing": ETL, stream processing, batch systems
|
||||
- "api_gateway": Request routing and API management systems
|
||||
current_metrics: Current performance metrics including:
|
||||
{
|
||||
"response_time_p95": "95th percentile response time in ms",
|
||||
"throughput_rps": "Requests per second",
|
||||
"cpu_utilization": "Average CPU usage percentage",
|
||||
"memory_usage": "Memory consumption in GB",
|
||||
"error_rate": "Error percentage",
|
||||
"availability": "System uptime percentage"
|
||||
}
|
||||
target_improvements: Desired performance targets:
|
||||
{
|
||||
"response_time_improvement": "Target reduction in response time",
|
||||
"throughput_increase": "Desired increase in throughput",
|
||||
"cost_reduction": "Target cost optimization percentage",
|
||||
"availability_target": "Desired uptime percentage"
|
||||
}
|
||||
constraints: Operational constraints:
|
||||
{
|
||||
"budget_limit": "Maximum budget for improvements",
|
||||
"timeline": "Implementation timeline constraints",
|
||||
"technology_restrictions": "Required or forbidden technologies",
|
||||
"compliance_requirements": "Security/regulatory constraints"
|
||||
}
|
||||
|
||||
Returns:
|
||||
Comprehensive optimization analysis:
|
||||
{
|
||||
"performance_analysis": {
|
||||
"bottlenecks_identified": ["Critical performance bottlenecks"],
|
||||
"root_cause_analysis": "Detailed analysis of performance issues",
|
||||
"current_vs_target": "Gap analysis between current and target metrics"
|
||||
},
|
||||
"optimization_recommendations": {
|
||||
"infrastructure_changes": ["Hardware/cloud resource recommendations"],
|
||||
"architecture_improvements": ["System design optimizations"],
|
||||
"code_optimizations": ["Software-level improvements"],
|
||||
"configuration_tuning": ["Parameter and setting adjustments"]
|
||||
},
|
||||
"implementation_roadmap": {
|
||||
"phase_1_quick_wins": ["Immediate improvements (0-2 weeks)"],
|
||||
"phase_2_medium_term": ["Medium-term optimizations (1-3 months)"],
|
||||
"phase_3_strategic": ["Long-term architectural changes (3-12 months)"]
|
||||
},
|
||||
"expected_outcomes": {
|
||||
"performance_improvements": "Projected performance gains",
|
||||
"cost_implications": "Expected costs and savings",
|
||||
"risk_assessment": "Implementation risks and mitigation strategies"
|
||||
}
|
||||
}
|
||||
"""
|
||||
# Simulate comprehensive performance optimization analysis
|
||||
optimization_areas = [
|
||||
"Database query optimization",
|
||||
"Caching layer enhancement",
|
||||
"Load balancing improvements",
|
||||
"Resource scaling strategies",
|
||||
"Code-level optimizations",
|
||||
"Infrastructure upgrades",
|
||||
]
|
||||
|
||||
return {
|
||||
"system_analyzed": system_type,
|
||||
"optimization_areas": random.sample(
|
||||
optimization_areas, k=min(4, len(optimization_areas))
|
||||
),
|
||||
"performance_score": random.randint(65, 95),
|
||||
"implementation_complexity": random.choice(["Low", "Medium", "High"]),
|
||||
"estimated_improvement": f"{random.randint(15, 45)}%",
|
||||
"recommendations": [
|
||||
"Implement distributed caching for frequently accessed data",
|
||||
"Optimize database queries and add strategic indexes",
|
||||
"Configure auto-scaling based on traffic patterns",
|
||||
"Implement asynchronous processing for heavy operations",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def analyze_security_vulnerabilities(
|
||||
system_components: List[str],
|
||||
security_scope: str = "comprehensive",
|
||||
compliance_frameworks: Optional[List[str]] = None,
|
||||
threat_model: str = "enterprise",
|
||||
) -> Dict[str, Any]:
|
||||
"""Perform comprehensive security vulnerability analysis and risk assessment.
|
||||
|
||||
This tool conducts detailed security analysis including vulnerability identification,
|
||||
threat modeling, compliance gap analysis, and provides prioritized remediation
|
||||
strategies based on risk levels and business impact.
|
||||
|
||||
Args:
|
||||
system_components: List of system components to analyze:
|
||||
- "web_frontend": User interfaces, SPAs, mobile apps
|
||||
- "api_endpoints": REST/GraphQL APIs, microservices
|
||||
- "database_layer": Data storage and access systems
|
||||
- "authentication": User auth, SSO, identity management
|
||||
- "data_processing": ETL, analytics, ML pipelines
|
||||
- "infrastructure": Servers, containers, cloud services
|
||||
- "network_layer": Load balancers, firewalls, CDNs
|
||||
security_scope: Analysis depth:
|
||||
- "basic": Standard vulnerability scanning
|
||||
- "comprehensive": Full security assessment
|
||||
- "compliance_focused": Regulatory compliance analysis
|
||||
- "threat_modeling": Advanced threat analysis
|
||||
compliance_frameworks: Required compliance standards:
|
||||
["SOC2", "GDPR", "HIPAA", "PCI-DSS", "ISO27001"]
|
||||
threat_model: Threat landscape consideration:
|
||||
- "startup": Basic threat model for early-stage companies
|
||||
- "enterprise": Corporate threat landscape
|
||||
- "high_security": Government/financial sector threats
|
||||
- "public_facing": Internet-exposed systems
|
||||
|
||||
Returns:
|
||||
Security analysis results:
|
||||
{
|
||||
"vulnerability_assessment": {
|
||||
"critical_vulnerabilities": ["High-priority security issues"],
|
||||
"moderate_risks": ["Medium-priority concerns"],
|
||||
"informational": ["Low-priority observations"],
|
||||
"risk_score": "Overall security risk rating (1-10)"
|
||||
},
|
||||
"threat_analysis": {
|
||||
"attack_vectors": ["Potential attack methods"],
|
||||
"threat_actors": ["Relevant threat actor profiles"],
|
||||
"attack_likelihood": "Probability assessment",
|
||||
"potential_impact": "Business impact analysis"
|
||||
},
|
||||
"compliance_status": {
|
||||
"framework_compliance": "Compliance percentage per framework",
|
||||
"gaps_identified": ["Non-compliant areas"],
|
||||
"certification_readiness": "Readiness for compliance audits"
|
||||
},
|
||||
"remediation_plan": {
|
||||
"immediate_actions": ["Critical fixes (0-2 weeks)"],
|
||||
"short_term_improvements": ["Important fixes (1-2 months)"],
|
||||
"strategic_initiatives": ["Long-term security enhancements"],
|
||||
"resource_requirements": "Personnel and budget needs"
|
||||
}
|
||||
}
|
||||
"""
|
||||
# Simulate security vulnerability analysis
|
||||
vulnerability_types = [
|
||||
"SQL Injection",
|
||||
"Cross-Site Scripting (XSS)",
|
||||
"Authentication Bypass",
|
||||
"Insecure Direct Object References",
|
||||
"Security Misconfiguration",
|
||||
"Sensitive Data Exposure",
|
||||
"Insufficient Logging",
|
||||
"CSRF",
|
||||
]
|
||||
|
||||
return {
|
||||
"components_analyzed": len(system_components),
|
||||
"critical_vulnerabilities": random.randint(0, 3),
|
||||
"moderate_risks": random.randint(2, 8),
|
||||
"overall_security_score": random.randint(6, 9),
|
||||
"compliance_percentage": random.randint(75, 95),
|
||||
"top_recommendations": [
|
||||
"Implement input validation and parameterized queries",
|
||||
"Enable comprehensive security logging and monitoring",
|
||||
"Review and update authentication and authorization controls",
|
||||
"Conduct regular security training for development team",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def design_scalability_architecture(
|
||||
current_architecture: str,
|
||||
expected_growth: Dict[str, Any],
|
||||
scalability_requirements: Dict[str, Any],
|
||||
technology_preferences: Optional[List[str]] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Design comprehensive scalability architecture for anticipated growth.
|
||||
|
||||
This tool analyzes current system architecture and designs scalable solutions
|
||||
to handle projected growth in users, data, traffic, and complexity while
|
||||
maintaining performance, reliability, and cost-effectiveness.
|
||||
|
||||
Args:
|
||||
current_architecture: Current system architecture type:
|
||||
- "monolith": Single-tier monolithic application
|
||||
- "service_oriented": SOA with multiple services
|
||||
- "microservices": Containerized microservice architecture
|
||||
- "serverless": Function-as-a-Service architecture
|
||||
- "hybrid": Mixed architecture patterns
|
||||
expected_growth: Projected growth metrics:
|
||||
{
|
||||
"user_growth_multiplier": "Expected increase in users",
|
||||
"data_volume_growth": "Projected data storage needs",
|
||||
"traffic_increase": "Expected traffic growth percentage",
|
||||
"geographic_expansion": "New regions/markets",
|
||||
"feature_complexity": "Additional functionality scope"
|
||||
}
|
||||
scalability_requirements: Scalability constraints and targets:
|
||||
{
|
||||
"performance_sla": "Response time requirements",
|
||||
"availability_target": "Uptime requirements",
|
||||
"consistency_model": "Data consistency needs",
|
||||
"budget_constraints": "Cost limitations",
|
||||
"deployment_model": "On-premise/cloud preferences"
|
||||
}
|
||||
technology_preferences: Preferred or required technologies:
|
||||
["kubernetes", "aws", "microservices", "nosql", etc.]
|
||||
|
||||
Returns:
|
||||
Scalability architecture design:
|
||||
{
|
||||
"architecture_recommendation": {
|
||||
"target_architecture": "Recommended architecture pattern",
|
||||
"migration_strategy": "Path from current to target architecture",
|
||||
"technology_stack": "Recommended technologies and frameworks"
|
||||
},
|
||||
"scalability_patterns": {
|
||||
"horizontal_scaling": "Auto-scaling and load distribution strategies",
|
||||
"data_partitioning": "Database sharding and data distribution",
|
||||
"caching_strategy": "Multi-level caching implementation",
|
||||
"async_processing": "Background job and queue systems"
|
||||
},
|
||||
"infrastructure_design": {
|
||||
"compute_resources": "Server/container resource planning",
|
||||
"data_storage": "Database and storage architecture",
|
||||
"network_topology": "CDN, load balancing, and routing",
|
||||
"monitoring_observability": "Logging, metrics, and alerting"
|
||||
},
|
||||
"implementation_phases": {
|
||||
"foundation_setup": "Core infrastructure preparation",
|
||||
"service_decomposition": "Breaking down monolithic components",
|
||||
"data_migration": "Database and storage transitions",
|
||||
"traffic_migration": "Gradual user traffic transition"
|
||||
}
|
||||
}
|
||||
"""
|
||||
# Simulate scalability architecture design
|
||||
architecture_patterns = [
|
||||
"Event-driven microservices",
|
||||
"CQRS with Event Sourcing",
|
||||
"Federated GraphQL architecture",
|
||||
"Serverless-first design",
|
||||
"Hybrid cloud architecture",
|
||||
"Edge-computing integration",
|
||||
]
|
||||
|
||||
return {
|
||||
"recommended_pattern": random.choice(architecture_patterns),
|
||||
"scalability_factor": f"{random.randint(5, 50)}x current capacity",
|
||||
"implementation_timeline": f"{random.randint(6, 18)} months",
|
||||
"estimated_cost_increase": f"{random.randint(20, 80)}%",
|
||||
"key_technologies": random.sample(
|
||||
[
|
||||
"Kubernetes",
|
||||
"Docker",
|
||||
"Redis",
|
||||
"PostgreSQL",
|
||||
"MongoDB",
|
||||
"Apache Kafka",
|
||||
"Elasticsearch",
|
||||
"AWS Lambda",
|
||||
"CloudFront",
|
||||
],
|
||||
k=4,
|
||||
),
|
||||
"success_metrics": [
|
||||
"Response time under load",
|
||||
"Auto-scaling effectiveness",
|
||||
"Cost per transaction",
|
||||
"System availability",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def benchmark_performance(
|
||||
system_name: str,
|
||||
metrics: Optional[List[str]] = None,
|
||||
duration: str = "standard",
|
||||
load_profile: str = "realistic",
|
||||
) -> Dict[str, Any]:
|
||||
"""Perform comprehensive performance benchmarking and analysis.
|
||||
|
||||
This tool conducts detailed performance benchmarking across multiple dimensions
|
||||
including response time, throughput, resource utilization, scalability limits,
|
||||
and system stability under various load conditions. It supports both synthetic
|
||||
and realistic workload testing with configurable parameters and monitoring.
|
||||
|
||||
The benchmarking process includes baseline establishment, performance profiling,
|
||||
bottleneck identification, capacity planning, and optimization recommendations.
|
||||
It can simulate various user patterns, network conditions, and system configurations
|
||||
to provide comprehensive performance insights.
|
||||
|
||||
Args:
|
||||
system_name: Name or identifier of the system to benchmark. Should be
|
||||
specific enough to identify the exact system configuration
|
||||
being tested.
|
||||
metrics: List of performance metrics to measure:
|
||||
- "latency": Response time and request processing delays
|
||||
- "throughput": Requests per second and data processing rates
|
||||
- "cpu": CPU utilization and processing efficiency
|
||||
- "memory": Memory usage and allocation patterns
|
||||
- "disk": Disk I/O performance and storage operations
|
||||
- "network": Network bandwidth and communication overhead
|
||||
- "scalability": System behavior under increasing load
|
||||
- "stability": Long-term performance and reliability
|
||||
duration: Benchmarking duration:
|
||||
- "quick": 5-10 minutes for rapid assessment
|
||||
- "standard": 30-60 minutes for comprehensive testing
|
||||
- "extended": 2-4 hours for stability and endurance testing
|
||||
- "continuous": Ongoing monitoring and measurement
|
||||
load_profile: Type of load pattern to simulate:
|
||||
- "constant": Steady, consistent load throughout test
|
||||
- "realistic": Variable load mimicking real usage patterns
|
||||
- "peak": High-intensity load testing for capacity limits
|
||||
- "stress": Beyond-capacity testing for failure analysis
|
||||
- "spike": Sudden load increases to test elasticity
|
||||
|
||||
Returns:
|
||||
Dictionary containing comprehensive benchmark results:
|
||||
{
|
||||
"summary": "Performance benchmark executive summary",
|
||||
"baseline": {...}, # Baseline performance measurements
|
||||
"results": {...}, # Detailed performance metrics
|
||||
"bottlenecks": [...], # Identified performance bottlenecks
|
||||
"scalability": {...}, # Scalability analysis results
|
||||
"recommendations": [...], # Performance optimization suggestions
|
||||
"capacity": {...}, # Capacity planning insights
|
||||
"monitoring": {...} # Ongoing monitoring recommendations
|
||||
}
|
||||
"""
|
||||
if metrics is None:
|
||||
metrics = ["latency", "throughput", "cpu", "memory"]
|
||||
|
||||
# Simulate benchmarking
|
||||
time.sleep(0.3)
|
||||
|
||||
return {
|
||||
"summary": f"Completed {duration} performance benchmark of {system_name}",
|
||||
"baseline": {
|
||||
"avg_latency": f"{random.uniform(50, 200):.2f}ms",
|
||||
"throughput": f"{random.randint(100, 1000)} requests/sec",
|
||||
"cpu_usage": f"{random.uniform(20, 80):.1f}%",
|
||||
},
|
||||
"results": {
|
||||
metric: f"Measured {metric} performance within expected ranges"
|
||||
for metric in metrics
|
||||
},
|
||||
"recommendations": [
|
||||
f"Optimize {system_name} for better {metrics[0]} performance",
|
||||
f"Consider scaling {system_name} for higher throughput",
|
||||
"Monitor performance trends over time",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
# Create the cache analysis research assistant agent
|
||||
cache_analysis_agent = Agent(
|
||||
name="cache_analysis_assistant",
|
||||
description="""
|
||||
Advanced Research and Analysis Assistant specializing in comprehensive system analysis,
|
||||
performance benchmarking, literature research, and test scenario generation for
|
||||
technical systems and AI applications.
|
||||
""",
|
||||
instruction="""
|
||||
|
||||
You are an expert Research and Analysis Assistant with deep expertise across multiple
|
||||
technical domains, specializing in comprehensive system analysis, performance optimization,
|
||||
security assessment, and architectural design. Your role encompasses both strategic planning
|
||||
and tactical implementation guidance for complex technical systems.
|
||||
|
||||
**Core Competencies and Expertise Areas:**
|
||||
|
||||
**Data Analysis & Pattern Recognition:**
|
||||
- Advanced statistical analysis including multivariate analysis, time series forecasting,
|
||||
regression modeling, and machine learning applications for pattern discovery
|
||||
- Trend identification across large datasets using statistical process control, anomaly
|
||||
detection algorithms, and predictive modeling techniques
|
||||
- Root cause analysis methodologies for complex system behaviors and performance issues
|
||||
- Data quality assessment and validation frameworks for ensuring analytical integrity
|
||||
- Visualization design principles for effective communication of analytical findings
|
||||
- Business intelligence and reporting strategies for different stakeholder audiences
|
||||
|
||||
**Academic & Professional Research:**
|
||||
- Systematic literature reviews following PRISMA guidelines and meta-analysis techniques
|
||||
- Citation network analysis and research impact assessment using bibliometric methods
|
||||
- Research gap identification through comprehensive domain mapping and trend analysis
|
||||
- Synthesis methodologies for integrating findings from diverse research sources
|
||||
- Research methodology design including experimental design, survey methods, and case studies
|
||||
- Peer review processes and academic publication strategies for research dissemination
|
||||
- Industry research integration including white papers, technical reports, and conference proceedings
|
||||
- Patent landscape analysis and intellectual property research for innovation assessment
|
||||
|
||||
**Test Design & Validation:**
|
||||
- Comprehensive test strategy development following industry frameworks (ISTQB, TMMI, TPI)
|
||||
- Test automation architecture design including framework selection and implementation strategies
|
||||
- Quality assurance methodologies encompassing functional, non-functional, and security testing
|
||||
- Risk-based testing approaches for optimizing test coverage within resource constraints
|
||||
- Continuous integration and deployment testing strategies for DevOps environments
|
||||
- Performance testing including load, stress, volume, and endurance testing protocols
|
||||
- Usability testing methodologies and user experience validation frameworks
|
||||
- Compliance testing for regulatory requirements across different industries
|
||||
|
||||
**Performance Engineering & Optimization:**
|
||||
- System performance analysis using APM tools, profiling techniques, and monitoring strategies
|
||||
- Capacity planning methodologies for both current needs and future growth projections
|
||||
- Scalability assessment including horizontal and vertical scaling strategies
|
||||
- Resource optimization techniques for compute, memory, storage, and network resources
|
||||
- Database performance tuning including query optimization, indexing strategies, and partitioning
|
||||
- Caching strategies implementation across multiple layers (application, database, CDN)
|
||||
- Load balancing and traffic distribution optimization for high-availability systems
|
||||
- Performance budgeting and SLA definition for service-level agreements
|
||||
|
||||
**Security & Compliance Analysis:**
|
||||
- Comprehensive security risk assessment including threat modeling and vulnerability analysis
|
||||
- Security architecture review and design for both defensive and offensive security perspectives
|
||||
- Compliance framework analysis for standards including SOC2, GDPR, HIPAA, PCI-DSS, ISO27001
|
||||
- Incident response planning and security monitoring strategy development
|
||||
- Security testing methodologies including penetration testing and security code review
|
||||
- Privacy impact assessment and data protection strategy development
|
||||
- Security training program design for technical and non-technical audiences
|
||||
- Cybersecurity governance and policy development for organizational security posture
|
||||
|
||||
**System Architecture & Design:**
|
||||
- Distributed systems design including microservices, service mesh, and event-driven architectures
|
||||
- Cloud architecture design for AWS, Azure, GCP with multi-cloud and hybrid strategies
|
||||
- Scalability patterns implementation including CQRS, Event Sourcing, and saga patterns
|
||||
- Database design and data modeling for both relational and NoSQL systems
|
||||
- API design following REST, GraphQL, and event-driven communication patterns
|
||||
- Infrastructure as Code (IaC) implementation using Terraform, CloudFormation, and Ansible
|
||||
- Container orchestration with Kubernetes including service mesh and observability
|
||||
- DevOps pipeline design encompassing CI/CD, monitoring, logging, and alerting strategies
|
||||
|
||||
**Research Methodology Framework:**
|
||||
|
||||
**Systematic Approach:**
|
||||
- Begin every analysis with clear problem definition, success criteria, and scope boundaries
|
||||
- Establish baseline measurements and define key performance indicators before analysis
|
||||
- Use structured analytical frameworks appropriate to the domain and problem type
|
||||
- Apply scientific methods including hypothesis formation, controlled experimentation, and validation
|
||||
- Implement peer review processes and cross-validation techniques when possible
|
||||
- Document methodology transparently to enable reproducibility and peer verification
|
||||
|
||||
**Information Synthesis:**
|
||||
- Integrate quantitative data with qualitative insights for comprehensive understanding
|
||||
- Cross-reference multiple authoritative sources to validate findings and reduce bias
|
||||
- Identify conflicting information and analyze reasons for discrepancies
|
||||
- Synthesize complex technical concepts into actionable business recommendations
|
||||
- Maintain awareness of information currency and source reliability
|
||||
- Apply critical thinking to distinguish correlation from causation in analytical findings
|
||||
|
||||
**Quality Assurance Standards:**
|
||||
- Implement multi-stage review processes for all analytical outputs
|
||||
- Use statistical significance testing and confidence intervals where appropriate
|
||||
- Clearly distinguish between established facts, supported inferences, and speculative conclusions
|
||||
- Provide uncertainty estimates and risk assessments for all recommendations
|
||||
- Include limitations analysis and recommendations for additional research or data collection
|
||||
- Ensure all analysis follows industry best practices and professional standards
|
||||
|
||||
**Communication and Reporting Excellence:**
|
||||
|
||||
**Audience Adaptation:**
|
||||
- Tailor communication style to technical level and role of the intended audience
|
||||
- Provide executive summaries for strategic decision-makers alongside detailed technical analysis
|
||||
- Use progressive disclosure to present information at appropriate levels of detail
|
||||
- Include visual elements and structured formats to enhance comprehension
|
||||
- Anticipate questions and provide preemptive clarification on complex topics
|
||||
|
||||
**Documentation Standards:**
|
||||
- Follow structured reporting templates appropriate to the analysis type
|
||||
- Include methodology sections that enable reproduction of analytical work
|
||||
- Provide clear action items with priority levels and implementation timelines
|
||||
- Include risk assessments and mitigation strategies for all recommendations
|
||||
- Maintain version control and change tracking for iterative analytical processes
|
||||
|
||||
**Tool Utilization Guidelines:**
|
||||
|
||||
When users request analysis or research, strategically leverage the available tools:
|
||||
|
||||
**For Data Analysis Requests:**
|
||||
- Use analyze_data_patterns for statistical analysis, trend identification, and pattern discovery
|
||||
- Apply appropriate statistical methods based on data type, sample size, and research questions
|
||||
- Provide confidence intervals and statistical significance testing where applicable
|
||||
- Include data visualization recommendations and interpretation guidance
|
||||
|
||||
**For Literature Research:**
|
||||
- Use research_literature for comprehensive academic and professional literature reviews
|
||||
- Focus on peer-reviewed sources while including relevant industry reports and white papers
|
||||
- Provide synthesis of findings with identification of research gaps and conflicting viewpoints
|
||||
- Include citation analysis and research impact assessment when relevant
|
||||
|
||||
**For Testing Strategy:**
|
||||
- Use generate_test_scenarios for comprehensive test planning and validation protocol design
|
||||
- Balance test coverage with practical constraints including time, budget, and resource limitations
|
||||
- Include both functional and non-functional testing considerations
|
||||
- Provide automation recommendations and implementation guidance
|
||||
|
||||
**For Performance Analysis:**
|
||||
- Use benchmark_performance for detailed performance assessment and optimization analysis
|
||||
- Include both current performance evaluation and future scalability considerations
|
||||
- Provide specific, measurable recommendations with expected impact quantification
|
||||
- Consider cost implications and return on investment for optimization recommendations
|
||||
|
||||
**For System Optimization:**
|
||||
- Use optimize_system_performance for comprehensive system improvement strategies
|
||||
- Include both technical optimizations and operational process improvements
|
||||
- Provide phased implementation approaches with quick wins and long-term strategic initiatives
|
||||
- Consider interdependencies between system components and potential unintended consequences
|
||||
|
||||
**For Security Assessment:**
|
||||
- Use analyze_security_vulnerabilities for comprehensive security risk evaluation
|
||||
- Include both technical vulnerabilities and procedural/operational security gaps
|
||||
- Provide risk-prioritized remediation plans with business impact consideration
|
||||
- Include compliance requirements and regulatory considerations
|
||||
|
||||
**For Architecture Design:**
|
||||
- Use design_scalability_architecture for strategic technical architecture planning
|
||||
- Consider both current requirements and future growth projections
|
||||
- Include technology stack recommendations with rationale and trade-off analysis
|
||||
- Provide migration strategies and implementation roadmaps for architecture transitions
|
||||
|
||||
**Professional Standards and Ethics:**
|
||||
|
||||
**Analytical Integrity:**
|
||||
- Maintain objectivity and avoid confirmation bias in all analytical work
|
||||
- Acknowledge limitations in data, methodology, or analytical scope
|
||||
- Provide balanced perspectives that consider alternative explanations and interpretations
|
||||
- Use peer review and validation processes to ensure analytical quality
|
||||
- Stay current with best practices and methodological advances in relevant domains
|
||||
|
||||
**Stakeholder Communication:**
|
||||
- Provide clear, actionable recommendations that align with organizational capabilities
|
||||
- Include risk assessments and uncertainty estimates for all strategic recommendations
|
||||
- Consider implementation feasibility including technical, financial, and organizational constraints
|
||||
- Offer both immediate tactical improvements and long-term strategic initiatives
|
||||
- Maintain transparency about analytical processes and potential sources of error
|
||||
|
||||
Your ultimate goal is to provide insights that are technically rigorous, strategically sound,
|
||||
and practically implementable. Every analysis should contribute to improved decision-making
|
||||
and measurable business outcomes while maintaining the highest standards of professional
|
||||
excellence and analytical integrity.
|
||||
""",
|
||||
tools=[
|
||||
analyze_data_patterns,
|
||||
research_literature,
|
||||
generate_test_scenarios,
|
||||
benchmark_performance,
|
||||
optimize_system_performance,
|
||||
analyze_security_vulnerabilities,
|
||||
design_scalability_architecture,
|
||||
],
|
||||
)
|
||||
|
||||
# Create the app with context caching configuration
|
||||
# Note: Context cache config is set at the App level
|
||||
cache_analysis_app = App(
|
||||
name="cache_analysis",
|
||||
root_agent=cache_analysis_agent,
|
||||
context_cache_config=ContextCacheConfig(
|
||||
min_tokens=4096,
|
||||
ttl_seconds=600, # 10 mins for research sessions
|
||||
cache_intervals=3, # Maximum invocations before cache refresh
|
||||
),
|
||||
)
|
||||
|
||||
# Export as app since it's an App, not an Agent
|
||||
app = cache_analysis_app
|
||||
|
||||
# Backward compatibility export - ADK still expects root_agent in some contexts
|
||||
root_agent = cache_analysis_agent
|
||||
@@ -0,0 +1,717 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""
|
||||
Cache Performance Experiments for ADK Context Caching
|
||||
|
||||
This script runs two experiments to compare caching performance:
|
||||
A. Gemini 2.0 Flash: Cache enabled vs disabled (explicit caching test)
|
||||
B. Gemini 2.5 Flash: Implicit vs explicit caching comparison
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import copy
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
import time
|
||||
from typing import Any
|
||||
from typing import Dict
|
||||
from typing import List
|
||||
|
||||
try:
|
||||
# Try relative imports first (when run as module)
|
||||
from .agent import app
|
||||
from .utils import get_test_prompts
|
||||
from .utils import run_experiment_batch
|
||||
except ImportError:
|
||||
# Fallback to direct imports (when run as script)
|
||||
from agent import app
|
||||
from utils import get_test_prompts
|
||||
from utils import run_experiment_batch
|
||||
|
||||
from google.adk.cli.utils import logs
|
||||
from google.adk.runners import InMemoryRunner
|
||||
from google.adk.utils.cache_performance_analyzer import CachePerformanceAnalyzer
|
||||
|
||||
APP_NAME = "cache_analysis_experiments"
|
||||
USER_ID = "cache_researcher"
|
||||
|
||||
|
||||
def create_agent_variant(base_app, model_name: str, cache_enabled: bool):
|
||||
"""Create an app variant with specified model and cache settings."""
|
||||
import datetime
|
||||
|
||||
from google.adk.agents.context_cache_config import ContextCacheConfig
|
||||
from google.adk.apps.app import App
|
||||
|
||||
# Extract the root agent and modify its model
|
||||
agent_copy = copy.deepcopy(base_app.root_agent)
|
||||
agent_copy.model = model_name
|
||||
|
||||
# Prepend dynamic timestamp to instruction to avoid implicit cache reuse across runs
|
||||
current_timestamp = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
dynamic_prefix = f"Current session started at: {current_timestamp}\n\n"
|
||||
agent_copy.instruction = dynamic_prefix + agent_copy.instruction
|
||||
|
||||
# Update agent name to reflect configuration
|
||||
cache_status = "cached" if cache_enabled else "no_cache"
|
||||
agent_copy.name = (
|
||||
f"cache_analysis_{model_name.replace('.', '_').replace('-', '_')}_{cache_status}"
|
||||
)
|
||||
|
||||
if cache_enabled:
|
||||
# Use standardized cache config
|
||||
cache_config = ContextCacheConfig(
|
||||
min_tokens=4096,
|
||||
ttl_seconds=600, # 10 mins for research sessions
|
||||
cache_intervals=3, # Maximum invocations before cache refresh
|
||||
)
|
||||
else:
|
||||
# Disable caching by setting config to None
|
||||
cache_config = None
|
||||
|
||||
# Create new App with updated configuration
|
||||
app_copy = App(
|
||||
name=f"{base_app.name}_{cache_status}",
|
||||
root_agent=agent_copy,
|
||||
context_cache_config=cache_config,
|
||||
)
|
||||
|
||||
return app_copy
|
||||
|
||||
|
||||
async def run_cache_comparison_experiment(
|
||||
model_name: str,
|
||||
description: str,
|
||||
cached_label: str,
|
||||
uncached_label: str,
|
||||
experiment_title: str,
|
||||
reverse_order: bool = False,
|
||||
request_delay: float = 2.0,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Run a cache performance comparison experiment for a specific model.
|
||||
|
||||
Args:
|
||||
model_name: Model to test (e.g., "gemini-2.5-flash")
|
||||
description: Description of what the experiment tests
|
||||
cached_label: Label for the cached experiment variant
|
||||
uncached_label: Label for the uncached experiment variant
|
||||
experiment_title: Title to display for the experiment
|
||||
|
||||
Returns:
|
||||
Dictionary containing experiment results and performance comparison
|
||||
"""
|
||||
print("=" * 80)
|
||||
print(f"EXPERIMENT {model_name}: {experiment_title}")
|
||||
print("=" * 80)
|
||||
print(f"Testing: {description}")
|
||||
print(f"Model: {model_name}")
|
||||
print()
|
||||
|
||||
# Create app variants
|
||||
app_cached = create_agent_variant(app, model_name, cache_enabled=True)
|
||||
app_uncached = create_agent_variant(app, model_name, cache_enabled=False)
|
||||
|
||||
# Get test prompts
|
||||
prompts = get_test_prompts()
|
||||
|
||||
# Create runners
|
||||
runner_cached = InMemoryRunner(app=app_cached, app_name=None)
|
||||
runner_uncached = InMemoryRunner(app=app_uncached, app_name=None)
|
||||
|
||||
# Create sessions for each experiment to avoid cross-contamination
|
||||
session_cached = await runner_cached.session_service.create_session(
|
||||
app_name=runner_cached.app_name, user_id=USER_ID
|
||||
)
|
||||
session_uncached = await runner_uncached.session_service.create_session(
|
||||
app_name=runner_uncached.app_name, user_id=USER_ID
|
||||
)
|
||||
|
||||
if not reverse_order: # Default: uncached first
|
||||
print("▶️ Running experiments in DEFAULT ORDER (uncached first)")
|
||||
print()
|
||||
|
||||
# Test uncached version first
|
||||
results_uncached = await run_experiment_batch(
|
||||
app_uncached.root_agent.name,
|
||||
runner_uncached,
|
||||
USER_ID,
|
||||
session_uncached.id,
|
||||
prompts,
|
||||
f"Experiment {model_name} - {uncached_label}",
|
||||
request_delay=request_delay,
|
||||
)
|
||||
|
||||
# Brief pause between experiments
|
||||
await asyncio.sleep(5)
|
||||
|
||||
# Test cached version second
|
||||
results_cached = await run_experiment_batch(
|
||||
app_cached.root_agent.name,
|
||||
runner_cached,
|
||||
USER_ID,
|
||||
session_cached.id,
|
||||
prompts,
|
||||
f"Experiment {model_name} - {cached_label}",
|
||||
request_delay=request_delay,
|
||||
)
|
||||
else:
|
||||
print("🔄 Running experiments in ALTERNATE ORDER (cached first)")
|
||||
print()
|
||||
|
||||
# Test cached version first
|
||||
results_cached = await run_experiment_batch(
|
||||
app_cached.root_agent.name,
|
||||
runner_cached,
|
||||
USER_ID,
|
||||
session_cached.id,
|
||||
prompts,
|
||||
f"Experiment {model_name} - {cached_label}",
|
||||
request_delay=request_delay,
|
||||
)
|
||||
|
||||
# Brief pause between experiments
|
||||
await asyncio.sleep(5)
|
||||
|
||||
# Test uncached version second
|
||||
results_uncached = await run_experiment_batch(
|
||||
app_uncached.root_agent.name,
|
||||
runner_uncached,
|
||||
USER_ID,
|
||||
session_uncached.id,
|
||||
prompts,
|
||||
f"Experiment {model_name} - {uncached_label}",
|
||||
request_delay=request_delay,
|
||||
)
|
||||
|
||||
# Analyze cache performance using CachePerformanceAnalyzer
|
||||
performance_analysis = await analyze_cache_performance_from_sessions(
|
||||
runner_cached,
|
||||
session_cached,
|
||||
runner_uncached,
|
||||
session_uncached,
|
||||
model_name,
|
||||
)
|
||||
|
||||
# Extract metrics from analyzer for backward compatibility
|
||||
cached_analysis = performance_analysis.get("cached_analysis", {})
|
||||
uncached_analysis = performance_analysis.get("uncached_analysis", {})
|
||||
|
||||
cached_total_prompt_tokens = cached_analysis.get("total_prompt_tokens", 0)
|
||||
cached_total_cached_tokens = cached_analysis.get("total_cached_tokens", 0)
|
||||
cached_cache_hit_ratio = cached_analysis.get("cache_hit_ratio_percent", 0.0)
|
||||
cached_cache_utilization_ratio = cached_analysis.get(
|
||||
"cache_utilization_ratio_percent", 0.0
|
||||
)
|
||||
cached_avg_cached_tokens_per_request = cached_analysis.get(
|
||||
"avg_cached_tokens_per_request", 0.0
|
||||
)
|
||||
cached_requests_with_hits = cached_analysis.get("requests_with_cache_hits", 0)
|
||||
total_cached_requests = cached_analysis.get("total_requests", 0)
|
||||
|
||||
uncached_total_prompt_tokens = uncached_analysis.get("total_prompt_tokens", 0)
|
||||
uncached_total_cached_tokens = uncached_analysis.get("total_cached_tokens", 0)
|
||||
uncached_cache_hit_ratio = uncached_analysis.get(
|
||||
"cache_hit_ratio_percent", 0.0
|
||||
)
|
||||
uncached_cache_utilization_ratio = uncached_analysis.get(
|
||||
"cache_utilization_ratio_percent", 0.0
|
||||
)
|
||||
uncached_avg_cached_tokens_per_request = uncached_analysis.get(
|
||||
"avg_cached_tokens_per_request", 0.0
|
||||
)
|
||||
uncached_requests_with_hits = uncached_analysis.get(
|
||||
"requests_with_cache_hits", 0
|
||||
)
|
||||
total_uncached_requests = uncached_analysis.get("total_requests", 0)
|
||||
|
||||
summary = {
|
||||
"experiment": model_name,
|
||||
"description": description,
|
||||
"model": model_name,
|
||||
"cached_results": results_cached,
|
||||
"uncached_results": results_uncached,
|
||||
"cache_analysis": {
|
||||
"cached_experiment": {
|
||||
"cache_hit_ratio_percent": cached_cache_hit_ratio,
|
||||
"cache_utilization_ratio_percent": cached_cache_utilization_ratio,
|
||||
"total_prompt_tokens": cached_total_prompt_tokens,
|
||||
"total_cached_tokens": cached_total_cached_tokens,
|
||||
"avg_cached_tokens_per_request": (
|
||||
cached_avg_cached_tokens_per_request
|
||||
),
|
||||
"requests_with_cache_hits": cached_requests_with_hits,
|
||||
"total_requests": total_cached_requests,
|
||||
},
|
||||
"uncached_experiment": {
|
||||
"cache_hit_ratio_percent": uncached_cache_hit_ratio,
|
||||
"cache_utilization_ratio_percent": (
|
||||
uncached_cache_utilization_ratio
|
||||
),
|
||||
"total_prompt_tokens": uncached_total_prompt_tokens,
|
||||
"total_cached_tokens": uncached_total_cached_tokens,
|
||||
"avg_cached_tokens_per_request": (
|
||||
uncached_avg_cached_tokens_per_request
|
||||
),
|
||||
"requests_with_cache_hits": uncached_requests_with_hits,
|
||||
"total_requests": total_uncached_requests,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
print(f"📊 EXPERIMENT {model_name} CACHE ANALYSIS:")
|
||||
print(f" 🔥 {cached_label}:")
|
||||
print(
|
||||
f" Cache Hit Ratio: {cached_cache_hit_ratio:.1f}%"
|
||||
f" ({cached_total_cached_tokens:,} /"
|
||||
f" {cached_total_prompt_tokens:,} tokens)"
|
||||
)
|
||||
print(
|
||||
f" Cache Utilization: {cached_cache_utilization_ratio:.1f}%"
|
||||
f" ({cached_requests_with_hits}/{total_cached_requests} requests)"
|
||||
)
|
||||
print(
|
||||
" Avg Cached Tokens/Request:"
|
||||
f" {cached_avg_cached_tokens_per_request:.0f}"
|
||||
)
|
||||
print(f" ❄️ {uncached_label}:")
|
||||
print(
|
||||
f" Cache Hit Ratio: {uncached_cache_hit_ratio:.1f}%"
|
||||
f" ({uncached_total_cached_tokens:,} /"
|
||||
f" {uncached_total_prompt_tokens:,} tokens)"
|
||||
)
|
||||
print(
|
||||
f" Cache Utilization: {uncached_cache_utilization_ratio:.1f}%"
|
||||
f" ({uncached_requests_with_hits}/{total_uncached_requests} requests)"
|
||||
)
|
||||
print(
|
||||
" Avg Cached Tokens/Request:"
|
||||
f" {uncached_avg_cached_tokens_per_request:.0f}"
|
||||
)
|
||||
print()
|
||||
|
||||
# Add performance analysis to summary
|
||||
summary["performance_analysis"] = performance_analysis
|
||||
|
||||
return summary
|
||||
|
||||
|
||||
async def analyze_cache_performance_from_sessions(
|
||||
runner_cached,
|
||||
session_cached,
|
||||
runner_uncached,
|
||||
session_uncached,
|
||||
model_name: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""Analyze cache performance using CachePerformanceAnalyzer."""
|
||||
print("📊 ANALYZING CACHE PERFORMANCE WITH CachePerformanceAnalyzer...")
|
||||
|
||||
analyzer_cached = CachePerformanceAnalyzer(runner_cached.session_service)
|
||||
analyzer_uncached = CachePerformanceAnalyzer(runner_uncached.session_service)
|
||||
|
||||
# Analyze cached experiment
|
||||
try:
|
||||
cached_analysis = await analyzer_cached.analyze_agent_cache_performance(
|
||||
session_cached.id,
|
||||
USER_ID,
|
||||
runner_cached.app_name,
|
||||
f"cache_analysis_{model_name.replace('.', '_').replace('-', '_')}_cached",
|
||||
)
|
||||
print(f" 🔥 Cached Experiment Analysis:")
|
||||
print(f" Status: {cached_analysis['status']}")
|
||||
if cached_analysis["status"] == "active":
|
||||
print(
|
||||
" Cache Hit Ratio:"
|
||||
f" {cached_analysis['cache_hit_ratio_percent']:.1f}%"
|
||||
f" ({cached_analysis['total_cached_tokens']:,} /"
|
||||
f" {cached_analysis['total_prompt_tokens']:,} tokens)"
|
||||
)
|
||||
print(
|
||||
" Cache Utilization:"
|
||||
f" {cached_analysis['cache_utilization_ratio_percent']:.1f}%"
|
||||
f" ({cached_analysis['requests_with_cache_hits']}/{cached_analysis['total_requests']}"
|
||||
" requests)"
|
||||
)
|
||||
print(
|
||||
" Avg Cached Tokens/Request:"
|
||||
f" {cached_analysis['avg_cached_tokens_per_request']:.0f}"
|
||||
)
|
||||
print(
|
||||
f" Requests with cache: {cached_analysis['requests_with_cache']}"
|
||||
)
|
||||
print(
|
||||
" Avg invocations used:"
|
||||
f" {cached_analysis['avg_invocations_used']:.1f}"
|
||||
)
|
||||
print(f" Cache refreshes: {cached_analysis['cache_refreshes']}")
|
||||
print(f" Total invocations: {cached_analysis['total_invocations']}")
|
||||
except Exception as e:
|
||||
print(f" ❌ Error analyzing cached experiment: {e}")
|
||||
cached_analysis = {"status": "error", "error": str(e)}
|
||||
|
||||
# Analyze uncached experiment
|
||||
try:
|
||||
uncached_analysis = await analyzer_uncached.analyze_agent_cache_performance(
|
||||
session_uncached.id,
|
||||
USER_ID,
|
||||
runner_uncached.app_name,
|
||||
f"cache_analysis_{model_name.replace('.', '_').replace('-', '_')}_no_cache",
|
||||
)
|
||||
print(f" ❄️ Uncached Experiment Analysis:")
|
||||
print(f" Status: {uncached_analysis['status']}")
|
||||
if uncached_analysis["status"] == "active":
|
||||
print(
|
||||
" Cache Hit Ratio:"
|
||||
f" {uncached_analysis['cache_hit_ratio_percent']:.1f}%"
|
||||
f" ({uncached_analysis['total_cached_tokens']:,} /"
|
||||
f" {uncached_analysis['total_prompt_tokens']:,} tokens)"
|
||||
)
|
||||
print(
|
||||
" Cache Utilization:"
|
||||
f" {uncached_analysis['cache_utilization_ratio_percent']:.1f}%"
|
||||
f" ({uncached_analysis['requests_with_cache_hits']}/{uncached_analysis['total_requests']}"
|
||||
" requests)"
|
||||
)
|
||||
print(
|
||||
" Avg Cached Tokens/Request:"
|
||||
f" {uncached_analysis['avg_cached_tokens_per_request']:.0f}"
|
||||
)
|
||||
print(
|
||||
" Requests with cache:"
|
||||
f" {uncached_analysis['requests_with_cache']}"
|
||||
)
|
||||
print(
|
||||
" Avg invocations used:"
|
||||
f" {uncached_analysis['avg_invocations_used']:.1f}"
|
||||
)
|
||||
print(f" Cache refreshes: {uncached_analysis['cache_refreshes']}")
|
||||
print(f" Total invocations: {uncached_analysis['total_invocations']}")
|
||||
except Exception as e:
|
||||
print(f" ❌ Error analyzing uncached experiment: {e}")
|
||||
uncached_analysis = {"status": "error", "error": str(e)}
|
||||
|
||||
print()
|
||||
|
||||
return {
|
||||
"cached_analysis": cached_analysis,
|
||||
"uncached_analysis": uncached_analysis,
|
||||
}
|
||||
|
||||
|
||||
def get_experiment_labels(model_name: str) -> Dict[str, str]:
|
||||
"""Get experiment labels and titles for a given model."""
|
||||
# Determine experiment type based on model name
|
||||
if "2.5" in model_name:
|
||||
# Gemini 2.5 models have implicit caching
|
||||
return {
|
||||
"description": "Google implicit caching vs ADK explicit caching",
|
||||
"cached_label": "Explicit Caching",
|
||||
"uncached_label": "Implicit Caching",
|
||||
"experiment_title": "Implicit vs Explicit Caching",
|
||||
}
|
||||
else:
|
||||
# Other models (2.0, etc.) test explicit caching vs no caching
|
||||
return {
|
||||
"description": "ADK explicit caching enabled vs disabled",
|
||||
"cached_label": "Cached",
|
||||
"uncached_label": "Uncached",
|
||||
"experiment_title": "Cache Performance Comparison",
|
||||
}
|
||||
|
||||
|
||||
def calculate_averaged_results(
|
||||
all_results: List[Dict[str, Any]], model_name: str
|
||||
) -> Dict[str, Any]:
|
||||
"""Calculate averaged results from multiple experiment runs."""
|
||||
if not all_results:
|
||||
raise ValueError("No results to average")
|
||||
|
||||
# Calculate average cache metrics
|
||||
cache_hit_ratios = [
|
||||
r["cache_analysis"]["cache_hit_ratio_percent"] for r in all_results
|
||||
]
|
||||
cache_utilization_ratios = [
|
||||
r["cache_analysis"]["cache_utilization_ratio_percent"]
|
||||
for r in all_results
|
||||
]
|
||||
total_prompt_tokens = [
|
||||
r["cache_analysis"]["total_prompt_tokens"] for r in all_results
|
||||
]
|
||||
total_cached_tokens = [
|
||||
r["cache_analysis"]["total_cached_tokens"] for r in all_results
|
||||
]
|
||||
avg_cached_tokens_per_request = [
|
||||
r["cache_analysis"]["avg_cached_tokens_per_request"] for r in all_results
|
||||
]
|
||||
requests_with_cache_hits = [
|
||||
r["cache_analysis"]["requests_with_cache_hits"] for r in all_results
|
||||
]
|
||||
|
||||
def safe_average(values):
|
||||
"""Calculate average, handling empty lists."""
|
||||
return sum(values) / len(values) if values else 0.0
|
||||
|
||||
# Create averaged result
|
||||
averaged_result = {
|
||||
"experiment": model_name,
|
||||
"description": all_results[0]["description"],
|
||||
"model": model_name,
|
||||
"individual_runs": (
|
||||
all_results
|
||||
), # Keep all individual results for reference
|
||||
"averaged_cache_analysis": {
|
||||
"cache_hit_ratio_percent": safe_average(cache_hit_ratios),
|
||||
"cache_utilization_ratio_percent": safe_average(
|
||||
cache_utilization_ratios
|
||||
),
|
||||
"total_prompt_tokens": safe_average(total_prompt_tokens),
|
||||
"total_cached_tokens": safe_average(total_cached_tokens),
|
||||
"avg_cached_tokens_per_request": safe_average(
|
||||
avg_cached_tokens_per_request
|
||||
),
|
||||
"requests_with_cache_hits": safe_average(requests_with_cache_hits),
|
||||
},
|
||||
"statistics": {
|
||||
"runs_completed": len(all_results),
|
||||
"cache_hit_ratio_std": _calculate_std(cache_hit_ratios),
|
||||
"cache_utilization_std": _calculate_std(cache_utilization_ratios),
|
||||
"cached_tokens_per_request_std": _calculate_std(
|
||||
avg_cached_tokens_per_request
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
# Print averaged results
|
||||
print("\n📊 AVERAGED CACHE ANALYSIS RESULTS:")
|
||||
print("=" * 80)
|
||||
avg_cache = averaged_result["averaged_cache_analysis"]
|
||||
stats = averaged_result["statistics"]
|
||||
|
||||
print(f" Runs completed: {stats['runs_completed']}")
|
||||
print(
|
||||
f" Average Cache Hit Ratio: {avg_cache['cache_hit_ratio_percent']:.1f}%"
|
||||
f" (±{stats['cache_hit_ratio_std']:.1f}%)"
|
||||
)
|
||||
print(
|
||||
" Average Cache Utilization:"
|
||||
f" {avg_cache['cache_utilization_ratio_percent']:.1f}%"
|
||||
f" (±{stats['cache_utilization_std']:.1f}%)"
|
||||
)
|
||||
print(
|
||||
" Average Cached Tokens/Request:"
|
||||
f" {avg_cache['avg_cached_tokens_per_request']:.0f}"
|
||||
f" (±{stats['cached_tokens_per_request_std']:.0f})"
|
||||
)
|
||||
print()
|
||||
|
||||
return averaged_result
|
||||
|
||||
|
||||
def _calculate_std(values):
|
||||
"""Calculate standard deviation."""
|
||||
if len(values) <= 1:
|
||||
return 0.0
|
||||
mean = sum(values) / len(values)
|
||||
variance = sum((x - mean) ** 2 for x in values) / len(values)
|
||||
return variance**0.5
|
||||
|
||||
|
||||
def save_results(results: Dict[str, Any], filename: str):
|
||||
"""Save experiment results to JSON file."""
|
||||
with open(filename, "w") as f:
|
||||
json.dump(results, f, indent=2)
|
||||
print(f"💾 Results saved to: {filename}")
|
||||
|
||||
|
||||
async def main():
|
||||
"""Run cache performance experiment for a specific model."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="ADK Cache Performance Experiment"
|
||||
)
|
||||
parser.add_argument(
|
||||
"model",
|
||||
help="Model to test (e.g., gemini-2.5-flash)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
help="Output filename for results (default: cache_{model}_results.json)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--repeat",
|
||||
type=int,
|
||||
default=1,
|
||||
help=(
|
||||
"Number of times to repeat each experiment for averaged results"
|
||||
" (default: 1)"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--cached-first",
|
||||
action="store_true",
|
||||
help="Run cached experiment first (default: uncached first)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--request-delay",
|
||||
type=float,
|
||||
default=2.0,
|
||||
help=(
|
||||
"Delay in seconds between API requests to avoid overloading (default:"
|
||||
" 2.0)"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--log-level",
|
||||
choices=["DEBUG", "INFO", "WARNING", "ERROR"],
|
||||
default="INFO",
|
||||
help="Set logging level (default: INFO)",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Setup logger with specified level
|
||||
log_level = getattr(logging, args.log_level.upper())
|
||||
logs.setup_adk_logger(log_level)
|
||||
|
||||
# Set default output filename based on model
|
||||
if not args.output:
|
||||
args.output = (
|
||||
f"cache_{args.model.replace('.', '_').replace('-', '_')}_results.json"
|
||||
)
|
||||
|
||||
print("🧪 ADK CONTEXT CACHE PERFORMANCE EXPERIMENT")
|
||||
print("=" * 80)
|
||||
print(f"Start time: {time.strftime('%Y-%m-%d %H:%M:%S')}")
|
||||
print(f"Model: {args.model}")
|
||||
print(f"Repetitions: {args.repeat}")
|
||||
print()
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
# Get experiment labels for the model
|
||||
labels = get_experiment_labels(args.model)
|
||||
|
||||
# Run the experiment multiple times if repeat > 1
|
||||
if args.repeat == 1:
|
||||
# Single run
|
||||
result = await run_cache_comparison_experiment(
|
||||
model_name=args.model,
|
||||
reverse_order=args.cached_first,
|
||||
request_delay=args.request_delay,
|
||||
**labels,
|
||||
)
|
||||
else:
|
||||
# Multiple runs with averaging
|
||||
print(f"🔄 Running experiment {args.repeat} times for averaged results")
|
||||
print("=" * 80)
|
||||
|
||||
all_results = []
|
||||
for run_num in range(args.repeat):
|
||||
print(f"\n🏃 RUN {run_num + 1}/{args.repeat}")
|
||||
print("-" * 40)
|
||||
|
||||
run_result = await run_cache_comparison_experiment(
|
||||
model_name=args.model,
|
||||
reverse_order=args.cached_first,
|
||||
request_delay=args.request_delay,
|
||||
**labels,
|
||||
)
|
||||
all_results.append(run_result)
|
||||
|
||||
# Brief pause between runs
|
||||
if run_num < args.repeat - 1:
|
||||
print("⏸️ Pausing 10 seconds between runs...")
|
||||
await asyncio.sleep(10)
|
||||
|
||||
# Calculate averaged results
|
||||
result = calculate_averaged_results(all_results, args.model)
|
||||
|
||||
# Add completion metadata
|
||||
result["end_time"] = time.strftime("%Y-%m-%d %H:%M:%S")
|
||||
result["total_duration"] = time.time() - start_time
|
||||
result["repetitions"] = args.repeat
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n⚠️ Experiment interrupted by user")
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
print(f"\n❌ Experiment failed: {e}")
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
# Save results
|
||||
save_results(result, args.output)
|
||||
|
||||
# Print final summary
|
||||
print("=" * 80)
|
||||
print("🎉 EXPERIMENT COMPLETED SUCCESSFULLY!")
|
||||
print("=" * 80)
|
||||
|
||||
# Handle both single and averaged results
|
||||
if args.repeat == 1:
|
||||
cached_exp = result["cache_analysis"]["cached_experiment"]
|
||||
uncached_exp = result["cache_analysis"]["uncached_experiment"]
|
||||
labels = get_experiment_labels(args.model)
|
||||
print(f"{args.model}:")
|
||||
print(f" 🔥 {labels['cached_label']}:")
|
||||
print(f" Cache Hit Ratio: {cached_exp['cache_hit_ratio_percent']:.1f}%")
|
||||
print(
|
||||
" Cache Utilization:"
|
||||
f" {cached_exp['cache_utilization_ratio_percent']:.1f}%"
|
||||
)
|
||||
print(
|
||||
" Cached Tokens/Request:"
|
||||
f" {cached_exp['avg_cached_tokens_per_request']:.0f}"
|
||||
)
|
||||
print(f" ❄️ {labels['uncached_label']}:")
|
||||
print(
|
||||
f" Cache Hit Ratio: {uncached_exp['cache_hit_ratio_percent']:.1f}%"
|
||||
)
|
||||
print(
|
||||
" Cache Utilization:"
|
||||
f" {uncached_exp['cache_utilization_ratio_percent']:.1f}%"
|
||||
)
|
||||
print(
|
||||
" Cached Tokens/Request:"
|
||||
f" {uncached_exp['avg_cached_tokens_per_request']:.0f}"
|
||||
)
|
||||
else:
|
||||
# For averaged results, show summary comparison
|
||||
cached_exp = result["averaged_cache_analysis"]["cached_experiment"]
|
||||
uncached_exp = result["averaged_cache_analysis"]["uncached_experiment"]
|
||||
labels = get_experiment_labels(args.model)
|
||||
print(f"{args.model} (averaged over {args.repeat} runs):")
|
||||
print(f" 🔥 {labels['cached_label']} vs ❄️ {labels['uncached_label']}:")
|
||||
print(
|
||||
f" Cache Hit Ratio: {cached_exp['cache_hit_ratio_percent']:.1f}% vs"
|
||||
f" {uncached_exp['cache_hit_ratio_percent']:.1f}%"
|
||||
)
|
||||
print(
|
||||
" Cache Utilization:"
|
||||
f" {cached_exp['cache_utilization_ratio_percent']:.1f}% vs"
|
||||
f" {uncached_exp['cache_utilization_ratio_percent']:.1f}%"
|
||||
)
|
||||
|
||||
print(f"\nTotal execution time: {result['total_duration']:.2f} seconds")
|
||||
print(f"Results saved to: {args.output}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,272 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Utility functions for cache analysis experiments."""
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from typing import Any
|
||||
from typing import Dict
|
||||
from typing import List
|
||||
|
||||
from google.adk.runners import InMemoryRunner
|
||||
|
||||
|
||||
async def call_agent_async(
|
||||
runner: InMemoryRunner, user_id: str, session_id: str, prompt: str
|
||||
) -> Dict[str, Any]:
|
||||
"""Call agent asynchronously and return response with token usage."""
|
||||
from google.genai import types
|
||||
|
||||
response_parts = []
|
||||
token_usage = {
|
||||
"prompt_token_count": 0,
|
||||
"candidates_token_count": 0,
|
||||
"cached_content_token_count": 0,
|
||||
"total_token_count": 0,
|
||||
}
|
||||
|
||||
async for event in runner.run_async(
|
||||
user_id=user_id,
|
||||
session_id=session_id,
|
||||
new_message=types.Content(parts=[types.Part(text=prompt)], role="user"),
|
||||
):
|
||||
if event.content and event.content.parts:
|
||||
for part in event.content.parts:
|
||||
if hasattr(part, "text") and part.text:
|
||||
response_parts.append(part.text)
|
||||
|
||||
# Collect token usage information
|
||||
if event.usage_metadata:
|
||||
if (
|
||||
hasattr(event.usage_metadata, "prompt_token_count")
|
||||
and event.usage_metadata.prompt_token_count
|
||||
):
|
||||
token_usage[
|
||||
"prompt_token_count"
|
||||
] += event.usage_metadata.prompt_token_count
|
||||
if (
|
||||
hasattr(event.usage_metadata, "candidates_token_count")
|
||||
and event.usage_metadata.candidates_token_count
|
||||
):
|
||||
token_usage[
|
||||
"candidates_token_count"
|
||||
] += event.usage_metadata.candidates_token_count
|
||||
if (
|
||||
hasattr(event.usage_metadata, "cached_content_token_count")
|
||||
and event.usage_metadata.cached_content_token_count
|
||||
):
|
||||
token_usage[
|
||||
"cached_content_token_count"
|
||||
] += event.usage_metadata.cached_content_token_count
|
||||
if (
|
||||
hasattr(event.usage_metadata, "total_token_count")
|
||||
and event.usage_metadata.total_token_count
|
||||
):
|
||||
token_usage[
|
||||
"total_token_count"
|
||||
] += event.usage_metadata.total_token_count
|
||||
|
||||
response_text = "".join(response_parts)
|
||||
|
||||
return {"response_text": response_text, "token_usage": token_usage}
|
||||
|
||||
|
||||
def get_test_prompts() -> List[str]:
|
||||
"""Get a standardized set of test prompts for cache analysis experiments.
|
||||
|
||||
Designed for consistent behavior:
|
||||
- Prompts 1-5: Will NOT trigger function calls (general questions)
|
||||
- Prompts 6-10: Will trigger function calls (specific tool requests)
|
||||
"""
|
||||
return [
|
||||
# === PROMPTS THAT WILL NOT TRIGGER FUNCTION CALLS ===
|
||||
# (General questions that don't match specific tool descriptions)
|
||||
"Hello, what can you do for me?",
|
||||
(
|
||||
"What is artificial intelligence and how does it work in modern"
|
||||
" applications?"
|
||||
),
|
||||
"Explain the difference between machine learning and deep learning.",
|
||||
"What are the main challenges in implementing AI systems at scale?",
|
||||
"How do recommendation systems work in modern e-commerce platforms?",
|
||||
# === PROMPTS THAT WILL TRIGGER FUNCTION CALLS ===
|
||||
# (Specific requests with all required parameters clearly specified)
|
||||
(
|
||||
"Use benchmark_performance with system_name='E-commerce Platform',"
|
||||
" metrics=['latency', 'throughput'], duration='standard',"
|
||||
" load_profile='realistic'."
|
||||
),
|
||||
(
|
||||
"Call analyze_user_behavior_patterns with"
|
||||
" user_segment='premium_customers', time_period='last_30_days',"
|
||||
" metrics=['engagement', 'conversion']."
|
||||
),
|
||||
(
|
||||
"Run market_research_analysis for industry='fintech',"
|
||||
" focus_areas=['user_experience', 'security'],"
|
||||
" report_depth='comprehensive'."
|
||||
),
|
||||
(
|
||||
"Execute competitive_analysis with competitors=['Netflix',"
|
||||
" 'Disney+'], analysis_type='feature_comparison',"
|
||||
" output_format='detailed'."
|
||||
),
|
||||
(
|
||||
"Perform content_performance_evaluation on content_type='video',"
|
||||
" platform='social_media', success_metrics=['views', 'engagement']."
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
async def run_experiment_batch(
|
||||
agent_name: str,
|
||||
runner: InMemoryRunner,
|
||||
user_id: str,
|
||||
session_id: str,
|
||||
prompts: List[str],
|
||||
experiment_name: str,
|
||||
request_delay: float = 2.0,
|
||||
) -> Dict[str, Any]:
|
||||
"""Run a batch of prompts and collect cache metrics."""
|
||||
results = []
|
||||
|
||||
print(f"🧪 Running {experiment_name}")
|
||||
print(f"Agent: {agent_name}")
|
||||
print(f"Session: {session_id}")
|
||||
print(f"Prompts: {len(prompts)}")
|
||||
print(f"Request delay: {request_delay}s between calls")
|
||||
print("-" * 60)
|
||||
|
||||
for i, prompt in enumerate(prompts, 1):
|
||||
print(f"[{i}/{len(prompts)}] Running test prompt...")
|
||||
print(f"Prompt: {prompt[:100]}...")
|
||||
|
||||
try:
|
||||
agent_response = await call_agent_async(
|
||||
runner, user_id, session_id, prompt
|
||||
)
|
||||
|
||||
result = {
|
||||
"prompt_number": i,
|
||||
"prompt": prompt,
|
||||
"response_length": len(agent_response["response_text"]),
|
||||
"success": True,
|
||||
"error": None,
|
||||
"token_usage": agent_response["token_usage"],
|
||||
}
|
||||
|
||||
# Extract token usage for individual prompt statistics
|
||||
prompt_tokens = agent_response["token_usage"].get("prompt_token_count", 0)
|
||||
cached_tokens = agent_response["token_usage"].get(
|
||||
"cached_content_token_count", 0
|
||||
)
|
||||
|
||||
print(
|
||||
"✅ Completed (Response:"
|
||||
f" {len(agent_response['response_text'])} chars)"
|
||||
)
|
||||
print(
|
||||
f" 📊 Tokens - Prompt: {prompt_tokens:,}, Cached: {cached_tokens:,}"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
result = {
|
||||
"prompt_number": i,
|
||||
"prompt": prompt,
|
||||
"response_length": 0,
|
||||
"success": False,
|
||||
"error": str(e),
|
||||
"token_usage": {
|
||||
"prompt_token_count": 0,
|
||||
"candidates_token_count": 0,
|
||||
"cached_content_token_count": 0,
|
||||
"total_token_count": 0,
|
||||
},
|
||||
}
|
||||
|
||||
print(f"❌ Failed: {e}")
|
||||
|
||||
results.append(result)
|
||||
|
||||
# Configurable pause between requests to avoid API overload
|
||||
if i < len(prompts): # Don't sleep after the last request
|
||||
print(f" ⏸️ Waiting {request_delay}s before next request...")
|
||||
await asyncio.sleep(request_delay)
|
||||
|
||||
successful_requests = sum(1 for r in results if r["success"])
|
||||
|
||||
# Calculate cache statistics for this batch
|
||||
total_prompt_tokens = sum(
|
||||
r.get("token_usage", {}).get("prompt_token_count", 0) for r in results
|
||||
)
|
||||
total_cached_tokens = sum(
|
||||
r.get("token_usage", {}).get("cached_content_token_count", 0)
|
||||
for r in results
|
||||
)
|
||||
|
||||
# Calculate cache hit ratio
|
||||
if total_prompt_tokens > 0:
|
||||
cache_hit_ratio = (total_cached_tokens / total_prompt_tokens) * 100
|
||||
else:
|
||||
cache_hit_ratio = 0.0
|
||||
|
||||
# Calculate cache utilization
|
||||
requests_with_cache_hits = sum(
|
||||
1
|
||||
for r in results
|
||||
if r.get("token_usage", {}).get("cached_content_token_count", 0) > 0
|
||||
)
|
||||
cache_utilization_ratio = (
|
||||
(requests_with_cache_hits / len(prompts)) * 100 if prompts else 0.0
|
||||
)
|
||||
|
||||
# Average cached tokens per request
|
||||
avg_cached_tokens_per_request = (
|
||||
total_cached_tokens / len(prompts) if prompts else 0.0
|
||||
)
|
||||
|
||||
summary = {
|
||||
"experiment_name": experiment_name,
|
||||
"agent_name": agent_name,
|
||||
"total_requests": len(prompts),
|
||||
"successful_requests": successful_requests,
|
||||
"results": results,
|
||||
"cache_statistics": {
|
||||
"cache_hit_ratio_percent": cache_hit_ratio,
|
||||
"cache_utilization_ratio_percent": cache_utilization_ratio,
|
||||
"total_prompt_tokens": total_prompt_tokens,
|
||||
"total_cached_tokens": total_cached_tokens,
|
||||
"avg_cached_tokens_per_request": avg_cached_tokens_per_request,
|
||||
"requests_with_cache_hits": requests_with_cache_hits,
|
||||
},
|
||||
}
|
||||
|
||||
print("-" * 60)
|
||||
print(f"✅ {experiment_name} completed:")
|
||||
print(f" Total requests: {len(prompts)}")
|
||||
print(f" Successful: {successful_requests}/{len(prompts)}")
|
||||
print(" 📊 BATCH CACHE STATISTICS:")
|
||||
print(
|
||||
f" Cache Hit Ratio: {cache_hit_ratio:.1f}%"
|
||||
f" ({total_cached_tokens:,} / {total_prompt_tokens:,} tokens)"
|
||||
)
|
||||
print(
|
||||
f" Cache Utilization: {cache_utilization_ratio:.1f}%"
|
||||
f" ({requests_with_cache_hits}/{len(prompts)} requests)"
|
||||
)
|
||||
print(f" Avg Cached Tokens/Request: {avg_cached_tokens_per_request:.0f}")
|
||||
print()
|
||||
|
||||
return summary
|
||||
Reference in New Issue
Block a user