mirrors/vxcontrol--pentagi
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:28:13 +08:00
commit 0878425be3
1160 changed files with 491311 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/tests/anthropic-report.md
+537
View File
@@ -0,0 +1,537 @@
# LLM Agent Testing Report
Generated: Thu, 29 Jan 2026 17:36:55 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
| simple | claude-haiku-4-5 | false | 23/23 (100.00%) | 1.239s |
| simple_json | claude-haiku-4-5 | false | 5/5 (100.00%) | 1.181s |
| primary_agent | claude-sonnet-4-5 | true | 23/23 (100.00%) | 3.542s |
| assistant | claude-sonnet-4-5 | true | 23/23 (100.00%) | 3.484s |
| generator | claude-opus-4-5 | true | 22/23 (95.65%) | 3.806s |
| refiner | claude-sonnet-4-5 | true | 23/23 (100.00%) | 3.512s |
| adviser | claude-sonnet-4-5 | true | 23/23 (100.00%) | 3.846s |
| reflector | claude-haiku-4-5 | true | 23/23 (100.00%) | 1.750s |
| searcher | claude-haiku-4-5 | true | 23/23 (100.00%) | 2.005s |
| enricher | claude-haiku-4-5 | true | 23/23 (100.00%) | 1.274s |
| coder | claude-sonnet-4-5 | true | 23/23 (100.00%) | 3.591s |
| installer | claude-sonnet-4-5 | true | 23/23 (100.00%) | 3.467s |
| pentester | claude-sonnet-4-5 | true | 23/23 (100.00%) | 3.390s |
**Total**: 280/281 (99.64%) successful tests
**Overall average latency**: 2.878s
## Detailed Results
### simple (claude-haiku-4-5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.141s | |
| Text Transform Uppercase | ✅ Pass | 0.758s | |
| Count from 1 to 5 | ✅ Pass | 0.671s | |
| Math Calculation | ✅ Pass | 0.670s | |
| Basic Echo Function | ✅ Pass | 0.896s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.771s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.898s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.729s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.771s | |
| Search Query Function | ✅ Pass | 1.083s | |
| Ask Advice Function | ✅ Pass | 1.788s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.829s | |
| Basic Context Memory Test | ✅ Pass | 0.800s | |
| Function Argument Memory Test | ✅ Pass | 0.818s | |
| Function Response Memory Test | ✅ Pass | 0.688s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.943s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.809s | |
| Penetration Testing Methodology | ✅ Pass | 2.576s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.713s | |
| SQL Injection Attack Type | ✅ Pass | 0.678s | |
| Penetration Testing Framework | ✅ Pass | 2.781s | |
| Web Application Security Scanner | ✅ Pass | 2.740s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.944s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.239s
---
### simple_json (claude-haiku-4-5)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Person Information JSON | ✅ Pass | 1.094s | |
| Project Information JSON | ✅ Pass | 0.980s | |
| Vulnerability Report Memory Test | ✅ Pass | 1.344s | |
| User Profile JSON | ✅ Pass | 1.302s | |
| Streaming Person Information JSON Streaming | ✅ Pass | 1.181s | |
**Summary**: 5/5 (100.00%) successful tests
**Average latency**: 1.181s
---
### primary_agent (claude-sonnet-4-5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.570s | |
| Text Transform Uppercase | ✅ Pass | 1.693s | |
| Count from 1 to 5 | ✅ Pass | 2.039s | |
| Math Calculation | ✅ Pass | 1.850s | |
| Basic Echo Function | ✅ Pass | 2.055s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.790s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.265s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.289s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.807s | |
| Search Query Function | ✅ Pass | 2.814s | |
| Ask Advice Function | ✅ Pass | 2.926s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.151s | |
| Basic Context Memory Test | ✅ Pass | 2.079s | |
| Function Argument Memory Test | ✅ Pass | 2.591s | |
| Function Response Memory Test | ✅ Pass | 2.866s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 5.508s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.358s | |
| Penetration Testing Methodology | ✅ Pass | 9.430s | |
| Vulnerability Assessment Tools | ✅ Pass | 9.419s | |
| SQL Injection Attack Type | ✅ Pass | 3.307s | |
| Penetration Testing Framework | ✅ Pass | 7.157s | |
| Web Application Security Scanner | ✅ Pass | 5.690s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.796s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.542s
---
### assistant (claude-sonnet-4-5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.461s | |
| Text Transform Uppercase | ✅ Pass | 2.354s | |
| Count from 1 to 5 | ✅ Pass | 2.622s | |
| Math Calculation | ✅ Pass | 1.745s | |
| Basic Echo Function | ✅ Pass | 2.333s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.756s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.387s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.253s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.586s | |
| Search Query Function | ✅ Pass | 2.288s | |
| Ask Advice Function | ✅ Pass | 2.792s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.011s | |
| Basic Context Memory Test | ✅ Pass | 1.945s | |
| Function Argument Memory Test | ✅ Pass | 2.272s | |
| Function Response Memory Test | ✅ Pass | 2.866s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.404s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.110s | |
| Penetration Testing Methodology | ✅ Pass | 9.989s | |
| Vulnerability Assessment Tools | ✅ Pass | 9.019s | |
| SQL Injection Attack Type | ✅ Pass | 3.210s | |
| Penetration Testing Framework | ✅ Pass | 6.954s | |
| Web Application Security Scanner | ✅ Pass | 6.797s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.967s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.484s
---
### generator (claude-opus-4-5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.321s | |
| Text Transform Uppercase | ✅ Pass | 2.552s | |
| Count from 1 to 5 | ✅ Pass | 2.619s | |
| Math Calculation | ✅ Pass | 1.929s | |
| Basic Echo Function | ✅ Pass | 2.562s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.360s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.331s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.475s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.969s | |
| Search Query Function | ✅ Pass | 2.670s | |
| Ask Advice Function | ✅ Pass | 3.021s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.796s | |
| Basic Context Memory Test | ✅ Pass | 3.351s | |
| Function Argument Memory Test | ✅ Pass | 2.976s | |
| Function Response Memory Test | ✅ Pass | 2.796s | |
| Penetration Testing Memory with Tool Call | ❌ Fail | 3.838s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.065s | |
| Penetration Testing Methodology | ✅ Pass | 10.315s | |
| Vulnerability Assessment Tools | ✅ Pass | 8.043s | |
| SQL Injection Attack Type | ✅ Pass | 3.519s | |
| Penetration Testing Framework | ✅ Pass | 7.758s | |
| Web Application Security Scanner | ✅ Pass | 7.778s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.486s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 3.806s
---
### refiner (claude-sonnet-4-5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.404s | |
| Text Transform Uppercase | ✅ Pass | 2.447s | |
| Count from 1 to 5 | ✅ Pass | 2.261s | |
| Math Calculation | ✅ Pass | 1.882s | |
| Basic Echo Function | ✅ Pass | 2.163s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.718s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.771s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.208s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.094s | |
| Search Query Function | ✅ Pass | 2.448s | |
| Ask Advice Function | ✅ Pass | 2.774s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.988s | |
| Basic Context Memory Test | ✅ Pass | 1.760s | |
| Function Argument Memory Test | ✅ Pass | 2.900s | |
| Function Response Memory Test | ✅ Pass | 2.596s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.326s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.033s | |
| Penetration Testing Methodology | ✅ Pass | 7.834s | |
| Vulnerability Assessment Tools | ✅ Pass | 8.705s | |
| SQL Injection Attack Type | ✅ Pass | 2.849s | |
| Penetration Testing Framework | ✅ Pass | 6.204s | |
| Web Application Security Scanner | ✅ Pass | 8.261s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.140s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.512s
---
### adviser (claude-sonnet-4-5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.245s | |
| Text Transform Uppercase | ✅ Pass | 2.271s | |
| Count from 1 to 5 | ✅ Pass | 2.626s | |
| Math Calculation | ✅ Pass | 2.379s | |
| Basic Echo Function | ✅ Pass | 2.195s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.984s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.513s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.233s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.467s | |
| Search Query Function | ✅ Pass | 1.824s | |
| Ask Advice Function | ✅ Pass | 2.907s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.212s | |
| Basic Context Memory Test | ✅ Pass | 2.029s | |
| Function Argument Memory Test | ✅ Pass | 3.062s | |
| Function Response Memory Test | ✅ Pass | 2.245s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.578s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.265s | |
| Penetration Testing Methodology | ✅ Pass | 10.992s | |
| Vulnerability Assessment Tools | ✅ Pass | 10.983s | |
| SQL Injection Attack Type | ✅ Pass | 3.834s | |
| Penetration Testing Framework | ✅ Pass | 11.421s | |
| Web Application Security Scanner | ✅ Pass | 5.404s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.776s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.846s
---
### reflector (claude-haiku-4-5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.054s | |
| Text Transform Uppercase | ✅ Pass | 1.121s | |
| Count from 1 to 5 | ✅ Pass | 1.197s | |
| Math Calculation | ✅ Pass | 0.758s | |
| Basic Echo Function | ✅ Pass | 0.950s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.839s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.877s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.067s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.055s | |
| Search Query Function | ✅ Pass | 1.245s | |
| Ask Advice Function | ✅ Pass | 1.219s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.137s | |
| Basic Context Memory Test | ✅ Pass | 1.086s | |
| Function Argument Memory Test | ✅ Pass | 1.374s | |
| Function Response Memory Test | ✅ Pass | 1.842s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.876s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.243s | |
| Penetration Testing Methodology | ✅ Pass | 5.903s | |
| Vulnerability Assessment Tools | ✅ Pass | 4.234s | |
| SQL Injection Attack Type | ✅ Pass | 1.474s | |
| Penetration Testing Framework | ✅ Pass | 4.361s | |
| Web Application Security Scanner | ✅ Pass | 2.855s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.474s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.750s
---
### searcher (claude-haiku-4-5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.103s | |
| Text Transform Uppercase | ✅ Pass | 0.949s | |
| Count from 1 to 5 | ✅ Pass | 1.459s | |
| Math Calculation | ✅ Pass | 0.803s | |
| Basic Echo Function | ✅ Pass | 1.227s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.397s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.554s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.528s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.932s | |
| Search Query Function | ✅ Pass | 1.231s | |
| Ask Advice Function | ✅ Pass | 1.183s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.042s | |
| Basic Context Memory Test | ✅ Pass | 1.258s | |
| Function Argument Memory Test | ✅ Pass | 1.074s | |
| Function Response Memory Test | ✅ Pass | 1.228s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.921s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.275s | |
| Penetration Testing Methodology | ✅ Pass | 7.863s | |
| Vulnerability Assessment Tools | ✅ Pass | 4.928s | |
| SQL Injection Attack Type | ✅ Pass | 3.186s | |
| Penetration Testing Framework | ✅ Pass | 3.992s | |
| Web Application Security Scanner | ✅ Pass | 2.703s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.267s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.005s
---
### enricher (claude-haiku-4-5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.833s | |
| Text Transform Uppercase | ✅ Pass | 0.743s | |
| Count from 1 to 5 | ✅ Pass | 0.913s | |
| Math Calculation | ✅ Pass | 0.633s | |
| Basic Echo Function | ✅ Pass | 1.190s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.224s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.673s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.493s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.793s | |
| Search Query Function | ✅ Pass | 0.886s | |
| Ask Advice Function | ✅ Pass | 0.980s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.917s | |
| Basic Context Memory Test | ✅ Pass | 0.795s | |
| Function Argument Memory Test | ✅ Pass | 0.700s | |
| Function Response Memory Test | ✅ Pass | 0.803s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.235s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.656s | |
| Penetration Testing Methodology | ✅ Pass | 2.759s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.843s | |
| SQL Injection Attack Type | ✅ Pass | 0.720s | |
| Penetration Testing Framework | ✅ Pass | 4.301s | |
| Web Application Security Scanner | ✅ Pass | 1.662s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.550s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.274s
---
### coder (claude-sonnet-4-5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.400s | |
| Text Transform Uppercase | ✅ Pass | 2.366s | |
| Count from 1 to 5 | ✅ Pass | 2.549s | |
| Math Calculation | ✅ Pass | 1.765s | |
| Basic Echo Function | ✅ Pass | 2.240s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.441s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.591s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.306s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.554s | |
| Search Query Function | ✅ Pass | 2.253s | |
| Ask Advice Function | ✅ Pass | 2.974s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.105s | |
| Basic Context Memory Test | ✅ Pass | 2.040s | |
| Function Argument Memory Test | ✅ Pass | 2.789s | |
| Function Response Memory Test | ✅ Pass | 2.121s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.801s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.181s | |
| Penetration Testing Methodology | ✅ Pass | 10.242s | |
| SQL Injection Attack Type | ✅ Pass | 3.511s | |
| Vulnerability Assessment Tools | ✅ Pass | 10.006s | |
| Penetration Testing Framework | ✅ Pass | 6.971s | |
| Web Application Security Scanner | ✅ Pass | 7.101s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.264s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.591s
---
### installer (claude-sonnet-4-5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.929s | |
| Text Transform Uppercase | ✅ Pass | 2.695s | |
| Count from 1 to 5 | ✅ Pass | 2.422s | |
| Math Calculation | ✅ Pass | 1.983s | |
| Basic Echo Function | ✅ Pass | 2.220s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.774s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.213s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.149s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.649s | |
| Search Query Function | ✅ Pass | 2.268s | |
| Ask Advice Function | ✅ Pass | 2.830s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.583s | |
| Basic Context Memory Test | ✅ Pass | 2.037s | |
| Function Argument Memory Test | ✅ Pass | 2.461s | |
| Function Response Memory Test | ✅ Pass | 2.316s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.875s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.429s | |
| Penetration Testing Methodology | ✅ Pass | 11.365s | |
| SQL Injection Attack Type | ✅ Pass | 2.569s | |
| Vulnerability Assessment Tools | ✅ Pass | 9.120s | |
| Penetration Testing Framework | ✅ Pass | 6.013s | |
| Web Application Security Scanner | ✅ Pass | 7.011s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.827s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.467s
---
### pentester (claude-sonnet-4-5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.950s | |
| Text Transform Uppercase | ✅ Pass | 2.156s | |
| Count from 1 to 5 | ✅ Pass | 2.490s | |
| Math Calculation | ✅ Pass | 1.791s | |
| Basic Echo Function | ✅ Pass | 2.504s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.898s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.821s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.111s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.741s | |
| Search Query Function | ✅ Pass | 2.284s | |
| Ask Advice Function | ✅ Pass | 2.957s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.078s | |
| Basic Context Memory Test | ✅ Pass | 2.196s | |
| Function Argument Memory Test | ✅ Pass | 2.287s | |
| Function Response Memory Test | ✅ Pass | 2.888s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.658s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.876s | |
| Penetration Testing Methodology | ✅ Pass | 8.949s | |
| Vulnerability Assessment Tools | ✅ Pass | 9.236s | |
| SQL Injection Attack Type | ✅ Pass | 3.461s | |
| Penetration Testing Framework | ✅ Pass | 6.547s | |
| Web Application Security Scanner | ✅ Pass | 5.307s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.761s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.390s
---
examples/tests/bedrock-report.md
+537
View File
@@ -0,0 +1,537 @@
# LLM Agent Testing Report
Generated: Wed, 04 Mar 2026 14:58:03 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
| simple | openai.gpt-oss-120b-1:0 | true | 23/23 (100.00%) | 0.706s |
| simple_json | openai.gpt-oss-120b-1:0 | true | 5/5 (100.00%) | 0.766s |
| primary_agent | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | true | 23/23 (100.00%) | 4.416s |
| assistant | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | true | 23/23 (100.00%) | 4.147s |
| generator | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | true | 23/23 (100.00%) | 4.768s |
| refiner | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | true | 23/23 (100.00%) | 4.212s |
| adviser | us.anthropic.claude-opus-4-6-v1 | true | 23/23 (100.00%) | 6.599s |
| reflector | us.anthropic.claude-haiku-4-5-20251001-v1:0 | true | 23/23 (100.00%) | 2.272s |
| searcher | us.anthropic.claude-haiku-4-5-20251001-v1:0 | true | 23/23 (100.00%) | 2.303s |
| enricher | us.anthropic.claude-haiku-4-5-20251001-v1:0 | true | 23/23 (100.00%) | 2.467s |
| coder | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | true | 23/23 (100.00%) | 4.197s |
| installer | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | true | 23/23 (100.00%) | 4.483s |
| pentester | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | true | 23/23 (100.00%) | 4.427s |
**Total**: 281/281 (100.00%) successful tests
**Overall average latency**: 3.697s
## Detailed Results
### simple (openai.gpt-oss-120b-1:0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.618s | |
| Text Transform Uppercase | ✅ Pass | 0.564s | |
| Count from 1 to 5 | ✅ Pass | 0.772s | |
| Math Calculation | ✅ Pass | 0.501s | |
| Basic Echo Function | ✅ Pass | 0.553s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.639s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.467s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.600s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.639s | |
| Search Query Function | ✅ Pass | 0.968s | |
| Ask Advice Function | ✅ Pass | 0.628s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.657s | |
| Basic Context Memory Test | ✅ Pass | 0.669s | |
| Function Argument Memory Test | ✅ Pass | 0.845s | |
| Function Response Memory Test | ✅ Pass | 0.488s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 0.714s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.738s | |
| Penetration Testing Methodology | ✅ Pass | 0.619s | |
| Vulnerability Assessment Tools | ✅ Pass | 0.723s | |
| SQL Injection Attack Type | ✅ Pass | 0.853s | |
| Penetration Testing Framework | ✅ Pass | 0.553s | |
| Web Application Security Scanner | ✅ Pass | 0.661s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.756s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.706s
---
### simple_json (openai.gpt-oss-120b-1:0)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Vulnerability Report Memory Test | ✅ Pass | 0.963s | |
| Person Information JSON | ✅ Pass | 0.892s | |
| Project Information JSON | ✅ Pass | 0.625s | |
| User Profile JSON | ✅ Pass | 0.740s | |
| Streaming Person Information JSON Streaming | ✅ Pass | 0.608s | |
**Summary**: 5/5 (100.00%) successful tests
**Average latency**: 0.766s
---
### primary_agent (us.anthropic.claude-sonnet-4-5-20250929-v1:0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 3.138s | |
| Text Transform Uppercase | ✅ Pass | 2.612s | |
| Count from 1 to 5 | ✅ Pass | 4.291s | |
| Math Calculation | ✅ Pass | 2.252s | |
| Basic Echo Function | ✅ Pass | 2.710s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.231s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.166s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.644s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 4.604s | |
| Search Query Function | ✅ Pass | 2.456s | |
| Ask Advice Function | ✅ Pass | 3.235s | |
| Streaming Search Query Function Streaming | ✅ Pass | 5.160s | |
| Basic Context Memory Test | ✅ Pass | 3.276s | |
| Function Argument Memory Test | ✅ Pass | 5.419s | |
| Function Response Memory Test | ✅ Pass | 4.129s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 7.036s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.372s | |
| Penetration Testing Methodology | ✅ Pass | 8.965s | |
| Vulnerability Assessment Tools | ✅ Pass | 8.967s | |
| SQL Injection Attack Type | ✅ Pass | 3.332s | |
| Penetration Testing Framework | ✅ Pass | 6.086s | |
| Web Application Security Scanner | ✅ Pass | 8.666s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.799s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 4.416s
---
### assistant (us.anthropic.claude-sonnet-4-5-20250929-v1:0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.647s | |
| Text Transform Uppercase | ✅ Pass | 4.615s | |
| Count from 1 to 5 | ✅ Pass | 2.519s | |
| Math Calculation | ✅ Pass | 2.116s | |
| Basic Echo Function | ✅ Pass | 2.474s | |
| Streaming Simple Math Streaming | ✅ Pass | 3.953s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.768s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.609s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.033s | |
| Search Query Function | ✅ Pass | 2.985s | |
| Ask Advice Function | ✅ Pass | 3.034s | |
| Streaming Search Query Function Streaming | ✅ Pass | 4.928s | |
| Basic Context Memory Test | ✅ Pass | 2.231s | |
| Function Argument Memory Test | ✅ Pass | 2.451s | |
| Function Response Memory Test | ✅ Pass | 3.166s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.586s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.071s | |
| Penetration Testing Methodology | ✅ Pass | 10.633s | |
| Vulnerability Assessment Tools | ✅ Pass | 7.906s | |
| SQL Injection Attack Type | ✅ Pass | 5.364s | |
| Penetration Testing Framework | ✅ Pass | 9.337s | |
| Web Application Security Scanner | ✅ Pass | 4.870s | |
| Penetration Testing Tool Selection | ✅ Pass | 5.071s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 4.147s
---
### generator (us.anthropic.claude-sonnet-4-5-20250929-v1:0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.165s | |
| Text Transform Uppercase | ✅ Pass | 2.657s | |
| Count from 1 to 5 | ✅ Pass | 5.377s | |
| Math Calculation | ✅ Pass | 4.765s | |
| Basic Echo Function | ✅ Pass | 4.964s | |
| Streaming Simple Math Streaming | ✅ Pass | 3.777s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.953s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.834s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.848s | |
| Search Query Function | ✅ Pass | 4.715s | |
| Ask Advice Function | ✅ Pass | 2.895s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.916s | |
| Basic Context Memory Test | ✅ Pass | 2.932s | |
| Function Argument Memory Test | ✅ Pass | 3.663s | |
| Function Response Memory Test | ✅ Pass | 5.374s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.607s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 6.857s | |
| Penetration Testing Methodology | ✅ Pass | 8.748s | |
| Vulnerability Assessment Tools | ✅ Pass | 13.187s | |
| SQL Injection Attack Type | ✅ Pass | 3.252s | |
| Penetration Testing Framework | ✅ Pass | 8.061s | |
| Web Application Security Scanner | ✅ Pass | 5.568s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.540s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 4.768s
---
### refiner (us.anthropic.claude-sonnet-4-5-20250929-v1:0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.429s | |
| Text Transform Uppercase | ✅ Pass | 3.501s | |
| Count from 1 to 5 | ✅ Pass | 2.639s | |
| Math Calculation | ✅ Pass | 2.235s | |
| Basic Echo Function | ✅ Pass | 2.677s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.270s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.043s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.846s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 4.618s | |
| Search Query Function | ✅ Pass | 4.727s | |
| Ask Advice Function | ✅ Pass | 3.741s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.863s | |
| Basic Context Memory Test | ✅ Pass | 4.924s | |
| Function Argument Memory Test | ✅ Pass | 3.036s | |
| Function Response Memory Test | ✅ Pass | 3.341s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.100s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 5.602s | |
| Penetration Testing Methodology | ✅ Pass | 8.381s | |
| Vulnerability Assessment Tools | ✅ Pass | 8.499s | |
| SQL Injection Attack Type | ✅ Pass | 2.908s | |
| Penetration Testing Framework | ✅ Pass | 8.594s | |
| Web Application Security Scanner | ✅ Pass | 7.187s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.700s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 4.212s
---
### adviser (us.anthropic.claude-opus-4-6-v1)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 3.568s | |
| Text Transform Uppercase | ✅ Pass | 3.961s | |
| Count from 1 to 5 | ✅ Pass | 4.415s | |
| Math Calculation | ✅ Pass | 2.137s | |
| Basic Echo Function | ✅ Pass | 2.199s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.102s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.540s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.644s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 7.616s | |
| Search Query Function | ✅ Pass | 3.461s | |
| Ask Advice Function | ✅ Pass | 4.363s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.361s | |
| Basic Context Memory Test | ✅ Pass | 2.789s | |
| Function Argument Memory Test | ✅ Pass | 8.947s | |
| Function Response Memory Test | ✅ Pass | 1.805s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.173s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.831s | |
| Penetration Testing Methodology | ✅ Pass | 11.607s | |
| Vulnerability Assessment Tools | ✅ Pass | 17.733s | |
| SQL Injection Attack Type | ✅ Pass | 2.430s | |
| Penetration Testing Framework | ✅ Pass | 27.779s | |
| Web Application Security Scanner | ✅ Pass | 12.400s | |
| Penetration Testing Tool Selection | ✅ Pass | 13.896s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 6.599s
---
### reflector (us.anthropic.claude-haiku-4-5-20251001-v1:0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.471s | |
| Text Transform Uppercase | ✅ Pass | 1.595s | |
| Count from 1 to 5 | ✅ Pass | 1.970s | |
| Math Calculation | ✅ Pass | 1.297s | |
| Basic Echo Function | ✅ Pass | 1.696s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.504s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.380s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.779s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.729s | |
| Search Query Function | ✅ Pass | 1.743s | |
| Ask Advice Function | ✅ Pass | 1.773s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.839s | |
| Basic Context Memory Test | ✅ Pass | 1.724s | |
| Function Argument Memory Test | ✅ Pass | 1.739s | |
| Function Response Memory Test | ✅ Pass | 1.821s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.116s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.909s | |
| Penetration Testing Methodology | ✅ Pass | 5.090s | |
| Vulnerability Assessment Tools | ✅ Pass | 4.686s | |
| SQL Injection Attack Type | ✅ Pass | 2.546s | |
| Penetration Testing Framework | ✅ Pass | 4.166s | |
| Web Application Security Scanner | ✅ Pass | 3.870s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.802s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.272s
---
### searcher (us.anthropic.claude-haiku-4-5-20251001-v1:0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.638s | |
| Text Transform Uppercase | ✅ Pass | 1.109s | |
| Count from 1 to 5 | ✅ Pass | 1.542s | |
| Math Calculation | ✅ Pass | 1.733s | |
| Basic Echo Function | ✅ Pass | 1.894s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.911s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.893s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.598s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.605s | |
| Search Query Function | ✅ Pass | 1.742s | |
| Ask Advice Function | ✅ Pass | 1.724s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.456s | |
| Basic Context Memory Test | ✅ Pass | 1.793s | |
| Function Argument Memory Test | ✅ Pass | 1.953s | |
| Function Response Memory Test | ✅ Pass | 1.806s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.409s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.881s | |
| Penetration Testing Methodology | ✅ Pass | 5.688s | |
| Vulnerability Assessment Tools | ✅ Pass | 4.266s | |
| SQL Injection Attack Type | ✅ Pass | 2.000s | |
| Penetration Testing Framework | ✅ Pass | 4.033s | |
| Web Application Security Scanner | ✅ Pass | 4.243s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.051s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.303s
---
### enricher (us.anthropic.claude-haiku-4-5-20251001-v1:0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.233s | |
| Text Transform Uppercase | ✅ Pass | 1.515s | |
| Count from 1 to 5 | ✅ Pass | 1.582s | |
| Math Calculation | ✅ Pass | 1.561s | |
| Basic Echo Function | ✅ Pass | 1.587s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.622s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.743s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.453s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.323s | |
| Search Query Function | ✅ Pass | 1.791s | |
| Ask Advice Function | ✅ Pass | 2.094s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.205s | |
| Basic Context Memory Test | ✅ Pass | 1.731s | |
| Function Argument Memory Test | ✅ Pass | 1.818s | |
| Function Response Memory Test | ✅ Pass | 2.317s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.740s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.877s | |
| Penetration Testing Methodology | ✅ Pass | 4.790s | |
| Vulnerability Assessment Tools | ✅ Pass | 4.254s | |
| SQL Injection Attack Type | ✅ Pass | 2.044s | |
| Penetration Testing Framework | ✅ Pass | 6.264s | |
| Web Application Security Scanner | ✅ Pass | 4.793s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.384s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.467s
---
### coder (us.anthropic.claude-sonnet-4-5-20250929-v1:0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.027s | |
| Text Transform Uppercase | ✅ Pass | 5.865s | |
| Count from 1 to 5 | ✅ Pass | 4.366s | |
| Math Calculation | ✅ Pass | 2.321s | |
| Basic Echo Function | ✅ Pass | 4.897s | |
| Streaming Simple Math Streaming | ✅ Pass | 3.941s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.624s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.860s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.398s | |
| Search Query Function | ✅ Pass | 4.321s | |
| Ask Advice Function | ✅ Pass | 3.065s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.623s | |
| Basic Context Memory Test | ✅ Pass | 2.514s | |
| Function Argument Memory Test | ✅ Pass | 3.229s | |
| Function Response Memory Test | ✅ Pass | 2.771s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.283s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.003s | |
| Penetration Testing Methodology | ✅ Pass | 11.748s | |
| Vulnerability Assessment Tools | ✅ Pass | 8.343s | |
| SQL Injection Attack Type | ✅ Pass | 3.425s | |
| Penetration Testing Framework | ✅ Pass | 5.599s | |
| Web Application Security Scanner | ✅ Pass | 5.566s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.723s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 4.197s
---
### installer (us.anthropic.claude-sonnet-4-5-20250929-v1:0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.495s | |
| Text Transform Uppercase | ✅ Pass | 2.385s | |
| Count from 1 to 5 | ✅ Pass | 2.851s | |
| Math Calculation | ✅ Pass | 4.008s | |
| Basic Echo Function | ✅ Pass | 3.840s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.631s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.243s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.520s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.584s | |
| Search Query Function | ✅ Pass | 5.006s | |
| Ask Advice Function | ✅ Pass | 3.081s | |
| Streaming Search Query Function Streaming | ✅ Pass | 5.535s | |
| Basic Context Memory Test | ✅ Pass | 5.053s | |
| Function Argument Memory Test | ✅ Pass | 2.839s | |
| Function Response Memory Test | ✅ Pass | 5.648s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.765s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.324s | |
| Penetration Testing Methodology | ✅ Pass | 6.697s | |
| Vulnerability Assessment Tools | ✅ Pass | 8.405s | |
| SQL Injection Attack Type | ✅ Pass | 7.206s | |
| Penetration Testing Framework | ✅ Pass | 9.822s | |
| Web Application Security Scanner | ✅ Pass | 6.015s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.145s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 4.483s
---
### pentester (us.anthropic.claude-sonnet-4-5-20250929-v1:0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.531s | |
| Text Transform Uppercase | ✅ Pass | 4.248s | |
| Count from 1 to 5 | ✅ Pass | 2.429s | |
| Math Calculation | ✅ Pass | 2.792s | |
| Basic Echo Function | ✅ Pass | 3.709s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.008s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.826s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.912s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.994s | |
| Search Query Function | ✅ Pass | 2.333s | |
| Ask Advice Function | ✅ Pass | 6.841s | |
| Streaming Search Query Function Streaming | ✅ Pass | 4.218s | |
| Basic Context Memory Test | ✅ Pass | 4.731s | |
| Function Argument Memory Test | ✅ Pass | 3.151s | |
| Function Response Memory Test | ✅ Pass | 3.061s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 6.495s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.016s | |
| Penetration Testing Methodology | ✅ Pass | 11.347s | |
| Vulnerability Assessment Tools | ✅ Pass | 7.938s | |
| SQL Injection Attack Type | ✅ Pass | 3.653s | |
| Penetration Testing Framework | ✅ Pass | 9.077s | |
| Web Application Security Scanner | ✅ Pass | 6.831s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.679s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 4.427s
---
examples/tests/custom-openai-report.md
+537
View File
@@ -0,0 +1,537 @@
# LLM Agent Testing Report
Generated: Sat, 19 Jul 2025 17:43:14 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
| simple | gpt-4.1-mini | false | 23/23 (100.00%) | 0.818s |
| simple_json | gpt-4.1-mini | false | 5/5 (100.00%) | 0.899s |
| primary_agent | o3-mini | true | 23/23 (100.00%) | 1.864s |
| assistant | o3-mini | true | 23/23 (100.00%) | 2.421s |
| generator | o3-mini | true | 23/23 (100.00%) | 2.449s |
| refiner | gpt-4.1 | false | 23/23 (100.00%) | 0.651s |
| adviser | o3-mini | true | 23/23 (100.00%) | 2.291s |
| reflector | o3-mini | true | 23/23 (100.00%) | 2.277s |
| searcher | gpt-4.1-mini | false | 23/23 (100.00%) | 0.586s |
| enricher | gpt-4.1-mini | false | 23/23 (100.00%) | 0.684s |
| coder | gpt-4.1 | false | 23/23 (100.00%) | 0.678s |
| installer | gpt-4.1 | false | 23/23 (100.00%) | 0.705s |
| pentester | o3-mini | true | 23/23 (100.00%) | 1.678s |
**Total**: 281/281 (100.00%) successful tests
**Overall average latency**: 1.416s
## Detailed Results
### simple (gpt-4.1-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.446s | |
| Text Transform Uppercase | ✅ Pass | 0.487s | |
| Count from 1 to 5 | ✅ Pass | 0.480s | |
| Math Calculation | ✅ Pass | 0.359s | |
| Basic Echo Function | ✅ Pass | 0.734s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.536s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.689s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.691s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.800s | |
| Search Query Function | ✅ Pass | 0.743s | |
| Ask Advice Function | ✅ Pass | 0.793s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.674s | |
| Basic Context Memory Test | ✅ Pass | 0.553s | |
| Function Argument Memory Test | ✅ Pass | 2.938s | |
| Function Response Memory Test | ✅ Pass | 0.431s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.026s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.470s | |
| Penetration Testing Methodology | ✅ Pass | 0.554s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.948s | |
| SQL Injection Attack Type | ✅ Pass | 0.467s | |
| Penetration Testing Framework | ✅ Pass | 0.653s | |
| Web Application Security Scanner | ✅ Pass | 0.734s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.603s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.818s
---
### simple_json (gpt-4.1-mini)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Person Information JSON | ✅ Pass | 0.876s | |
| Project Information JSON | ✅ Pass | 0.824s | |
| User Profile JSON | ✅ Pass | 0.625s | |
| Vulnerability Report Memory Test | ✅ Pass | 1.412s | |
| Streaming Person Information JSON Streaming | ✅ Pass | 0.752s | |
**Summary**: 5/5 (100.00%) successful tests
**Average latency**: 0.899s
---
### primary_agent (o3-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.093s | |
| Text Transform Uppercase | ✅ Pass | 1.701s | |
| Count from 1 to 5 | ✅ Pass | 1.818s | |
| Math Calculation | ✅ Pass | 1.486s | |
| Basic Echo Function | ✅ Pass | 1.455s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.616s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.687s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.196s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.582s | |
| Search Query Function | ✅ Pass | 2.201s | |
| Ask Advice Function | ✅ Pass | 1.284s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.159s | |
| Basic Context Memory Test | ✅ Pass | 1.657s | |
| Function Argument Memory Test | ✅ Pass | 1.547s | |
| Function Response Memory Test | ✅ Pass | 1.592s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.030s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.658s | |
| Penetration Testing Methodology | ✅ Pass | 1.440s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.278s | |
| SQL Injection Attack Type | ✅ Pass | 3.660s | |
| Penetration Testing Framework | ✅ Pass | 1.768s | |
| Web Application Security Scanner | ✅ Pass | 2.324s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.628s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.864s
---
### assistant (o3-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.966s | |
| Text Transform Uppercase | ✅ Pass | 3.316s | |
| Count from 1 to 5 | ✅ Pass | 2.169s | |
| Math Calculation | ✅ Pass | 2.319s | |
| Basic Echo Function | ✅ Pass | 1.490s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.615s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.004s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.455s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.434s | |
| Search Query Function | ✅ Pass | 1.913s | |
| Ask Advice Function | ✅ Pass | 1.892s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.965s | |
| Basic Context Memory Test | ✅ Pass | 2.646s | |
| Function Argument Memory Test | ✅ Pass | 2.116s | |
| Function Response Memory Test | ✅ Pass | 1.654s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.538s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.939s | |
| Penetration Testing Methodology | ✅ Pass | 1.959s | |
| Vulnerability Assessment Tools | ✅ Pass | 5.623s | |
| SQL Injection Attack Type | ✅ Pass | 3.432s | |
| Penetration Testing Framework | ✅ Pass | 3.295s | |
| Web Application Security Scanner | ✅ Pass | 2.242s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.697s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.421s
---
### generator (o3-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.072s | |
| Text Transform Uppercase | ✅ Pass | 2.268s | |
| Count from 1 to 5 | ✅ Pass | 2.519s | |
| Math Calculation | ✅ Pass | 1.813s | |
| Basic Echo Function | ✅ Pass | 1.947s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.684s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.177s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.508s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.968s | |
| Search Query Function | ✅ Pass | 2.275s | |
| Ask Advice Function | ✅ Pass | 1.337s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.214s | |
| Basic Context Memory Test | ✅ Pass | 3.678s | |
| Function Argument Memory Test | ✅ Pass | 1.936s | |
| Function Response Memory Test | ✅ Pass | 2.254s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.923s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.286s | |
| Penetration Testing Methodology | ✅ Pass | 1.886s | |
| Vulnerability Assessment Tools | ✅ Pass | 4.566s | |
| Penetration Testing Framework | ✅ Pass | 2.827s | |
| SQL Injection Attack Type | ✅ Pass | 7.667s | |
| Web Application Security Scanner | ✅ Pass | 1.864s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.652s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.449s
---
### refiner (gpt-4.1)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.471s | |
| Text Transform Uppercase | ✅ Pass | 0.567s | |
| Count from 1 to 5 | ✅ Pass | 0.473s | |
| Math Calculation | ✅ Pass | 0.820s | |
| Basic Echo Function | ✅ Pass | 0.724s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.409s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.991s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.687s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.576s | |
| Search Query Function | ✅ Pass | 0.741s | |
| Ask Advice Function | ✅ Pass | 0.747s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.666s | |
| Basic Context Memory Test | ✅ Pass | 0.587s | |
| Function Argument Memory Test | ✅ Pass | 0.427s | |
| Function Response Memory Test | ✅ Pass | 0.417s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 0.790s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.556s | |
| Penetration Testing Methodology | ✅ Pass | 0.625s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.048s | |
| SQL Injection Attack Type | ✅ Pass | 0.626s | |
| Penetration Testing Framework | ✅ Pass | 0.681s | |
| Web Application Security Scanner | ✅ Pass | 0.582s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.738s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.651s
---
### adviser (o3-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.596s | |
| Text Transform Uppercase | ✅ Pass | 1.729s | |
| Count from 1 to 5 | ✅ Pass | 2.232s | |
| Math Calculation | ✅ Pass | 1.427s | |
| Basic Echo Function | ✅ Pass | 1.771s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.078s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.871s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.118s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.231s | |
| Search Query Function | ✅ Pass | 1.984s | |
| Ask Advice Function | ✅ Pass | 1.953s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.493s | |
| Basic Context Memory Test | ✅ Pass | 3.430s | |
| Function Argument Memory Test | ✅ Pass | 1.782s | |
| Function Response Memory Test | ✅ Pass | 2.374s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.427s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.778s | |
| Penetration Testing Methodology | ✅ Pass | 1.660s | |
| Vulnerability Assessment Tools | ✅ Pass | 5.158s | |
| SQL Injection Attack Type | ✅ Pass | 3.258s | |
| Penetration Testing Framework | ✅ Pass | 3.163s | |
| Web Application Security Scanner | ✅ Pass | 2.294s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.880s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.291s
---
### reflector (o3-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.007s | |
| Text Transform Uppercase | ✅ Pass | 1.557s | |
| Count from 1 to 5 | ✅ Pass | 2.252s | |
| Math Calculation | ✅ Pass | 1.688s | |
| Basic Echo Function | ✅ Pass | 2.140s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.109s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.549s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.758s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.720s | |
| Search Query Function | ✅ Pass | 1.641s | |
| Ask Advice Function | ✅ Pass | 1.753s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.430s | |
| Basic Context Memory Test | ✅ Pass | 2.423s | |
| Function Argument Memory Test | ✅ Pass | 1.887s | |
| Function Response Memory Test | ✅ Pass | 1.891s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.169s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.610s | |
| Penetration Testing Methodology | ✅ Pass | 2.438s | |
| Vulnerability Assessment Tools | ✅ Pass | 5.552s | |
| SQL Injection Attack Type | ✅ Pass | 4.227s | |
| Penetration Testing Framework | ✅ Pass | 3.024s | |
| Web Application Security Scanner | ✅ Pass | 2.037s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.493s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.277s
---
### searcher (gpt-4.1-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.412s | |
| Text Transform Uppercase | ✅ Pass | 0.457s | |
| Count from 1 to 5 | ✅ Pass | 0.472s | |
| Math Calculation | ✅ Pass | 0.449s | |
| Basic Echo Function | ✅ Pass | 0.602s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.441s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.409s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.551s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.711s | |
| Search Query Function | ✅ Pass | 0.607s | |
| Ask Advice Function | ✅ Pass | 0.703s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.550s | |
| Basic Context Memory Test | ✅ Pass | 0.535s | |
| Function Argument Memory Test | ✅ Pass | 0.463s | |
| Function Response Memory Test | ✅ Pass | 0.404s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.016s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.481s | |
| Penetration Testing Methodology | ✅ Pass | 0.656s | |
| Vulnerability Assessment Tools | ✅ Pass | 0.931s | |
| SQL Injection Attack Type | ✅ Pass | 0.456s | |
| Penetration Testing Framework | ✅ Pass | 0.910s | |
| Web Application Security Scanner | ✅ Pass | 0.494s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.757s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.586s
---
### enricher (gpt-4.1-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.392s | |
| Text Transform Uppercase | ✅ Pass | 0.516s | |
| Count from 1 to 5 | ✅ Pass | 0.443s | |
| Math Calculation | ✅ Pass | 0.354s | |
| Basic Echo Function | ✅ Pass | 0.559s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.420s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.392s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.585s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.577s | |
| Search Query Function | ✅ Pass | 0.726s | |
| Ask Advice Function | ✅ Pass | 0.754s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.288s | |
| Basic Context Memory Test | ✅ Pass | 0.636s | |
| Function Argument Memory Test | ✅ Pass | 0.455s | |
| Function Response Memory Test | ✅ Pass | 0.361s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.522s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.639s | |
| Penetration Testing Methodology | ✅ Pass | 0.682s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.128s | |
| SQL Injection Attack Type | ✅ Pass | 0.499s | |
| Penetration Testing Framework | ✅ Pass | 0.570s | |
| Web Application Security Scanner | ✅ Pass | 0.497s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.717s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.684s
---
### coder (gpt-4.1)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.402s | |
| Text Transform Uppercase | ✅ Pass | 0.621s | |
| Count from 1 to 5 | ✅ Pass | 0.478s | |
| Math Calculation | ✅ Pass | 0.342s | |
| Basic Echo Function | ✅ Pass | 0.708s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.430s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.407s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.062s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.617s | |
| Search Query Function | ✅ Pass | 0.568s | |
| Ask Advice Function | ✅ Pass | 0.948s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.574s | |
| Basic Context Memory Test | ✅ Pass | 0.600s | |
| Function Argument Memory Test | ✅ Pass | 0.618s | |
| Function Response Memory Test | ✅ Pass | 0.630s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.698s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.448s | |
| Penetration Testing Methodology | ✅ Pass | 0.626s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.099s | |
| SQL Injection Attack Type | ✅ Pass | 0.887s | |
| Penetration Testing Framework | ✅ Pass | 0.547s | |
| Web Application Security Scanner | ✅ Pass | 0.624s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.648s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.678s
---
### installer (gpt-4.1)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.600s | |
| Text Transform Uppercase | ✅ Pass | 0.452s | |
| Count from 1 to 5 | ✅ Pass | 0.598s | |
| Math Calculation | ✅ Pass | 0.400s | |
| Basic Echo Function | ✅ Pass | 0.881s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.367s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.479s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.076s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.656s | |
| Search Query Function | ✅ Pass | 0.829s | |
| Ask Advice Function | ✅ Pass | 0.657s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.655s | |
| Basic Context Memory Test | ✅ Pass | 0.584s | |
| Function Argument Memory Test | ✅ Pass | 0.518s | |
| Function Response Memory Test | ✅ Pass | 0.551s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 0.854s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.457s | |
| Penetration Testing Methodology | ✅ Pass | 0.673s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.284s | |
| SQL Injection Attack Type | ✅ Pass | 0.774s | |
| Penetration Testing Framework | ✅ Pass | 0.559s | |
| Web Application Security Scanner | ✅ Pass | 1.209s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.094s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.705s
---
### pentester (o3-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.491s | |
| Text Transform Uppercase | ✅ Pass | 1.742s | |
| Count from 1 to 5 | ✅ Pass | 1.592s | |
| Math Calculation | ✅ Pass | 1.670s | |
| Basic Echo Function | ✅ Pass | 1.463s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.149s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.209s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.322s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.224s | |
| Search Query Function | ✅ Pass | 1.431s | |
| Ask Advice Function | ✅ Pass | 1.612s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.112s | |
| Basic Context Memory Test | ✅ Pass | 1.616s | |
| Function Argument Memory Test | ✅ Pass | 1.315s | |
| Function Response Memory Test | ✅ Pass | 1.260s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.090s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.003s | |
| Penetration Testing Methodology | ✅ Pass | 2.127s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.243s | |
| SQL Injection Attack Type | ✅ Pass | 2.215s | |
| Penetration Testing Framework | ✅ Pass | 1.702s | |
| Web Application Security Scanner | ✅ Pass | 1.352s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.648s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.678s
---
examples/tests/deepinfra-report.md
+537
View File
@@ -0,0 +1,537 @@
# LLM Agent Testing Report
Generated: Tue, 30 Sep 2025 19:10:56 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
| simple | Qwen/Qwen3-Next-80B-A3B-Instruct | false | 23/23 (100.00%) | 1.284s |
| simple_json | Qwen/Qwen3-Next-80B-A3B-Instruct | false | 5/5 (100.00%) | 1.261s |
| primary_agent | moonshotai/Kimi-K2-Instruct-0905 | false | 22/23 (95.65%) | 1.406s |
| assistant | moonshotai/Kimi-K2-Instruct-0905 | true | 21/23 (91.30%) | 1.397s |
| generator | google/gemini-2.5-pro | true | 22/23 (95.65%) | 7.349s |
| refiner | deepseek-ai/DeepSeek-R1-0528-Turbo | true | 22/23 (95.65%) | 4.424s |
| adviser | google/gemini-2.5-pro | true | 23/23 (100.00%) | 6.986s |
| reflector | Qwen/Qwen3-Next-80B-A3B-Instruct | true | 23/23 (100.00%) | 1.277s |
| searcher | Qwen/Qwen3-32B | true | 23/23 (100.00%) | 6.780s |
| enricher | Qwen/Qwen3-32B | true | 23/23 (100.00%) | 6.705s |
| coder | anthropic/claude-4-sonnet | true | 23/23 (100.00%) | 2.953s |
| installer | google/gemini-2.5-flash | true | 23/23 (100.00%) | 2.703s |
| pentester | moonshotai/Kimi-K2-Instruct-0905 | true | 22/23 (95.65%) | 1.303s |
**Total**: 275/281 (97.86%) successful tests
**Overall average latency**: 3.670s
## Detailed Results
### simple (Qwen/Qwen3-Next-80B-A3B-Instruct)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.578s | |
| Text Transform Uppercase | ✅ Pass | 2.580s | |
| Count from 1 to 5 | ✅ Pass | 0.964s | |
| Math Calculation | ✅ Pass | 0.900s | |
| Basic Echo Function | ✅ Pass | 1.061s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.022s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.949s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.736s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.425s | |
| Search Query Function | ✅ Pass | 1.348s | |
| Ask Advice Function | ✅ Pass | 1.016s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.939s | |
| Basic Context Memory Test | ✅ Pass | 1.248s | |
| Function Argument Memory Test | ✅ Pass | 1.044s | |
| Function Response Memory Test | ✅ Pass | 0.887s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.120s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.931s | |
| Penetration Testing Methodology | ✅ Pass | 1.612s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.659s | |
| SQL Injection Attack Type | ✅ Pass | 0.930s | |
| Penetration Testing Framework | ✅ Pass | 1.154s | |
| Web Application Security Scanner | ✅ Pass | 1.297s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.111s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.284s
---
### simple_json (Qwen/Qwen3-Next-80B-A3B-Instruct)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Vulnerability Report Memory Test | ✅ Pass | 1.469s | |
| Person Information JSON | ✅ Pass | 1.179s | |
| User Profile JSON | ✅ Pass | 1.124s | |
| Project Information JSON | ✅ Pass | 1.246s | |
| Streaming Person Information JSON Streaming | ✅ Pass | 1.283s | |
**Summary**: 5/5 (100.00%) successful tests
**Average latency**: 1.261s
---
### primary_agent (moonshotai/Kimi-K2-Instruct-0905)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.603s | |
| Text Transform Uppercase | ✅ Pass | 2.651s | |
| Count from 1 to 5 | ✅ Pass | 0.755s | |
| Math Calculation | ✅ Pass | 0.793s | |
| Basic Echo Function | ✅ Pass | 1.418s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.783s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.653s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.644s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.157s | |
| Search Query Function | ✅ Pass | 1.301s | |
| Ask Advice Function | ✅ Pass | 1.400s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.985s | |
| Basic Context Memory Test | ✅ Pass | 1.052s | |
| Function Argument Memory Test | ✅ Pass | 1.118s | |
| Function Response Memory Test | ✅ Pass | 0.731s | |
| Penetration Testing Memory with Tool Call | ❌ Fail | 2.041s | no tool calls found, expected at least 1 |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.632s | |
| Penetration Testing Methodology | ✅ Pass | 1.588s | |
| Vulnerability Assessment Tools | ✅ Pass | 0.885s | |
| SQL Injection Attack Type | ✅ Pass | 0.796s | |
| Penetration Testing Framework | ✅ Pass | 4.317s | |
| Web Application Security Scanner | ✅ Pass | 0.679s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.336s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 1.406s
---
### assistant (moonshotai/Kimi-K2-Instruct-0905)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.648s | |
| Text Transform Uppercase | ✅ Pass | 2.672s | |
| Count from 1 to 5 | ✅ Pass | 0.759s | |
| Math Calculation | ✅ Pass | 0.772s | |
| Basic Echo Function | ❌ Fail | 1.552s | no tool calls found, expected at least 1 |
| Streaming Simple Math Streaming | ✅ Pass | 0.702s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.661s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.752s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.377s | |
| Search Query Function | ✅ Pass | 2.002s | |
| Ask Advice Function | ✅ Pass | 1.351s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.864s | |
| Basic Context Memory Test | ✅ Pass | 1.075s | |
| Function Argument Memory Test | ✅ Pass | 0.667s | |
| Function Response Memory Test | ✅ Pass | 0.603s | |
| Penetration Testing Memory with Tool Call | ❌ Fail | 2.146s | no tool calls found, expected at least 1 |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.626s | |
| Penetration Testing Methodology | ✅ Pass | 1.616s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.152s | |
| SQL Injection Attack Type | ✅ Pass | 0.886s | |
| Penetration Testing Framework | ✅ Pass | 3.288s | |
| Web Application Security Scanner | ✅ Pass | 0.714s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.229s | |
**Summary**: 21/23 (91.30%) successful tests
**Average latency**: 1.397s
---
### generator (google/gemini-2.5-pro)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 5.339s | |
| Text Transform Uppercase | ✅ Pass | 8.288s | |
| Math Calculation | ✅ Pass | 3.081s | |
| Count from 1 to 5 | ✅ Pass | 5.671s | |
| Streaming Simple Math Streaming | ✅ Pass | 4.258s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.547s | |
| Basic Echo Function | ✅ Pass | 8.848s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.689s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Search Query Function | ✅ Pass | 4.440s | |
| JSON Response Function | ✅ Pass | 8.128s | |
| Ask Advice Function | ✅ Pass | 4.194s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.103s | |
| Function Argument Memory Test | ✅ Pass | 4.669s | |
| Basic Context Memory Test | ✅ Pass | 7.525s | |
| Function Response Memory Test | ✅ Pass | 7.255s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 6.236s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 5.003s | |
| Penetration Testing Methodology | ✅ Pass | 10.267s | |
| SQL Injection Attack Type | ✅ Pass | 6.586s | |
| Vulnerability Assessment Tools | ❌ Fail | 21.490s | expected text 'network' not found |
| Penetration Testing Tool Selection | ✅ Pass | 4.211s | |
| Penetration Testing Framework | ✅ Pass | 18.186s | |
| Web Application Security Scanner | ✅ Pass | 15.008s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 7.349s
---
### refiner (deepseek-ai/DeepSeek-R1-0528-Turbo)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Text Transform Uppercase | ✅ Pass | 3.027s | |
| Count from 1 to 5 | ✅ Pass | 1.407s | |
| Simple Math | ✅ Pass | 5.472s | |
| Math Calculation | ✅ Pass | 2.814s | |
| Basic Echo Function | ✅ Pass | 2.816s | |
| Streaming Simple Math Streaming | ✅ Pass | 3.297s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.343s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.435s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.203s | |
| Search Query Function | ✅ Pass | 3.924s | |
| Ask Advice Function | ✅ Pass | 2.326s | |
| Streaming Search Query Function Streaming | ❌ Fail | 3.341s | no tool calls found, expected at least 1 |
| Basic Context Memory Test | ✅ Pass | 4.284s | |
| Function Argument Memory Test | ✅ Pass | 2.489s | |
| Function Response Memory Test | ✅ Pass | 1.821s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.045s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 9.472s | |
| Vulnerability Assessment Tools | ✅ Pass | 4.717s | |
| Penetration Testing Methodology | ✅ Pass | 8.188s | |
| SQL Injection Attack Type | ✅ Pass | 6.380s | |
| Penetration Testing Framework | ✅ Pass | 10.979s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.450s | |
| Web Application Security Scanner | ✅ Pass | 10.505s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 4.424s
---
### adviser (google/gemini-2.5-pro)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 5.136s | |
| Text Transform Uppercase | ✅ Pass | 5.091s | |
| Count from 1 to 5 | ✅ Pass | 5.334s | |
| Math Calculation | ✅ Pass | 3.554s | |
| Streaming Simple Math Streaming | ✅ Pass | 4.277s | |
| Basic Echo Function | ✅ Pass | 5.468s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.383s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 4.941s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Search Query Function | ✅ Pass | 4.618s | |
| JSON Response Function | ✅ Pass | 8.534s | |
| Ask Advice Function | ✅ Pass | 4.124s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.168s | |
| Basic Context Memory Test | ✅ Pass | 5.123s | |
| Function Argument Memory Test | ✅ Pass | 3.921s | |
| Function Response Memory Test | ✅ Pass | 6.008s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.767s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 8.001s | |
| Penetration Testing Methodology | ✅ Pass | 11.466s | |
| SQL Injection Attack Type | ✅ Pass | 8.174s | |
| Vulnerability Assessment Tools | ✅ Pass | 15.468s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.422s | |
| Web Application Security Scanner | ✅ Pass | 15.610s | |
| Penetration Testing Framework | ✅ Pass | 20.072s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 6.986s
---
### reflector (Qwen/Qwen3-Next-80B-A3B-Instruct)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.579s | |
| Text Transform Uppercase | ✅ Pass | 1.261s | |
| Count from 1 to 5 | ✅ Pass | 0.888s | |
| Math Calculation | ✅ Pass | 0.900s | |
| Basic Echo Function | ✅ Pass | 0.943s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.933s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.168s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.156s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.828s | |
| Search Query Function | ✅ Pass | 1.027s | |
| Ask Advice Function | ✅ Pass | 0.974s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.438s | |
| Basic Context Memory Test | ✅ Pass | 0.978s | |
| Function Argument Memory Test | ✅ Pass | 0.914s | |
| Function Response Memory Test | ✅ Pass | 0.950s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.307s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.194s | |
| Penetration Testing Methodology | ✅ Pass | 1.691s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.906s | |
| SQL Injection Attack Type | ✅ Pass | 1.504s | |
| Penetration Testing Framework | ✅ Pass | 1.183s | |
| Web Application Security Scanner | ✅ Pass | 1.622s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.013s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.277s
---
### searcher (Qwen/Qwen3-32B)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 5.632s | |
| Text Transform Uppercase | ✅ Pass | 5.010s | |
| Count from 1 to 5 | ✅ Pass | 4.092s | |
| Basic Echo Function | ✅ Pass | 3.654s | |
| Math Calculation | ✅ Pass | 6.467s | |
| Streaming Simple Math Streaming | ✅ Pass | 5.738s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.966s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.840s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 5.198s | |
| Search Query Function | ✅ Pass | 3.664s | |
| Streaming Search Query Function Streaming | ✅ Pass | 4.344s | |
| Ask Advice Function | ✅ Pass | 6.628s | |
| Function Argument Memory Test | ✅ Pass | 4.112s | |
| Basic Context Memory Test | ✅ Pass | 7.697s | |
| Function Response Memory Test | ✅ Pass | 4.331s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.918s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 7.309s | |
| Penetration Testing Methodology | ✅ Pass | 12.251s | |
| SQL Injection Attack Type | ✅ Pass | 9.998s | |
| Vulnerability Assessment Tools | ✅ Pass | 17.638s | |
| Penetration Testing Framework | ✅ Pass | 12.965s | |
| Web Application Security Scanner | ✅ Pass | 10.203s | |
| Penetration Testing Tool Selection | ✅ Pass | 5.265s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 6.780s
---
### enricher (Qwen/Qwen3-32B)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Text Transform Uppercase | ✅ Pass | 5.822s | |
| Simple Math | ✅ Pass | 8.812s | |
| Count from 1 to 5 | ✅ Pass | 4.942s | |
| Math Calculation | ✅ Pass | 5.295s | |
| Basic Echo Function | ✅ Pass | 3.634s | |
| Streaming Simple Math Streaming | ✅ Pass | 3.727s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.957s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 7.226s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Search Query Function | ✅ Pass | 3.308s | |
| JSON Response Function | ✅ Pass | 7.863s | |
| Ask Advice Function | ✅ Pass | 4.381s | |
| Streaming Search Query Function Streaming | ✅ Pass | 4.571s | |
| Basic Context Memory Test | ✅ Pass | 7.343s | |
| Function Response Memory Test | ✅ Pass | 4.464s | |
| Function Argument Memory Test | ✅ Pass | 6.395s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 7.066s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 5.124s | |
| Penetration Testing Methodology | ✅ Pass | 11.162s | |
| Vulnerability Assessment Tools | ✅ Pass | 11.898s | |
| SQL Injection Attack Type | ✅ Pass | 8.524s | |
| Penetration Testing Framework | ✅ Pass | 11.747s | |
| Web Application Security Scanner | ✅ Pass | 10.317s | |
| Penetration Testing Tool Selection | ✅ Pass | 6.616s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 6.705s
---
### coder (anthropic/claude-4-sonnet)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 4.038s | |
| Text Transform Uppercase | ✅ Pass | 1.810s | |
| Count from 1 to 5 | ✅ Pass | 1.872s | |
| Math Calculation | ✅ Pass | 2.400s | |
| Basic Echo Function | ✅ Pass | 2.111s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.348s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.517s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.253s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.561s | |
| Search Query Function | ✅ Pass | 2.016s | |
| Ask Advice Function | ✅ Pass | 2.363s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.742s | |
| Basic Context Memory Test | ✅ Pass | 2.941s | |
| Function Argument Memory Test | ✅ Pass | 1.764s | |
| Function Response Memory Test | ✅ Pass | 1.999s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.912s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.101s | |
| Penetration Testing Methodology | ✅ Pass | 5.195s | |
| Vulnerability Assessment Tools | ✅ Pass | 5.804s | |
| SQL Injection Attack Type | ✅ Pass | 2.188s | |
| Penetration Testing Framework | ✅ Pass | 5.985s | |
| Web Application Security Scanner | ✅ Pass | 4.381s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.602s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.953s
---
### installer (google/gemini-2.5-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.469s | |
| Text Transform Uppercase | ✅ Pass | 2.376s | |
| Count from 1 to 5 | ✅ Pass | 1.039s | |
| Math Calculation | ✅ Pass | 1.579s | |
| Basic Echo Function | ✅ Pass | 1.417s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.214s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.954s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.251s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.778s | |
| Search Query Function | ✅ Pass | 2.765s | |
| Ask Advice Function | ✅ Pass | 2.787s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.896s | |
| Basic Context Memory Test | ✅ Pass | 3.006s | |
| Function Argument Memory Test | ✅ Pass | 1.224s | |
| Function Response Memory Test | ✅ Pass | 1.743s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.311s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.248s | |
| Penetration Testing Methodology | ✅ Pass | 2.712s | |
| SQL Injection Attack Type | ✅ Pass | 1.453s | |
| Vulnerability Assessment Tools | ✅ Pass | 14.593s | |
| Penetration Testing Framework | ✅ Pass | 4.916s | |
| Web Application Security Scanner | ✅ Pass | 3.753s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.673s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.703s
---
### pentester (moonshotai/Kimi-K2-Instruct-0905)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.661s | |
| Text Transform Uppercase | ✅ Pass | 0.641s | |
| Count from 1 to 5 | ✅ Pass | 1.036s | |
| Math Calculation | ✅ Pass | 0.594s | |
| Basic Echo Function | ✅ Pass | 1.246s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.594s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.479s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.881s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.316s | |
| Search Query Function | ✅ Pass | 1.443s | |
| Ask Advice Function | ✅ Pass | 1.316s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.617s | |
| Basic Context Memory Test | ✅ Pass | 0.840s | |
| Function Argument Memory Test | ✅ Pass | 0.659s | |
| Function Response Memory Test | ✅ Pass | 0.611s | |
| Penetration Testing Memory with Tool Call | ❌ Fail | 2.006s | no tool calls found, expected at least 1 |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.633s | |
| Penetration Testing Methodology | ✅ Pass | 1.000s | |
| Vulnerability Assessment Tools | ✅ Pass | 0.791s | |
| SQL Injection Attack Type | ✅ Pass | 5.163s | |
| Penetration Testing Framework | ✅ Pass | 0.761s | |
| Web Application Security Scanner | ✅ Pass | 0.598s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.073s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 1.303s
---
examples/tests/deepseek-report.md
+537
View File
@@ -0,0 +1,537 @@
# LLM Agent Testing Report
Generated: Wed, 27 May 2026 22:59:39 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
| simple | deepseek-v4-flash | false | 23/23 (100.00%) | 1.039s |
| simple_json | deepseek-v4-flash | false | 5/5 (100.00%) | 0.989s |
| primary_agent | deepseek-v4-pro | true | 23/23 (100.00%) | 2.896s |
| assistant | deepseek-v4-pro | true | 22/23 (95.65%) | 2.929s |
| generator | deepseek-v4-pro | true | 23/23 (100.00%) | 3.046s |
| refiner | deepseek-v4-pro | true | 23/23 (100.00%) | 2.932s |
| adviser | deepseek-v4-pro | true | 23/23 (100.00%) | 2.667s |
| reflector | deepseek-v4-flash | true | 23/23 (100.00%) | 1.288s |
| searcher | deepseek-v4-flash | true | 23/23 (100.00%) | 0.991s |
| enricher | deepseek-v4-flash | true | 23/23 (100.00%) | 0.981s |
| coder | deepseek-v4-pro | true | 23/23 (100.00%) | 2.817s |
| installer | deepseek-v4-flash | true | 23/23 (100.00%) | 1.246s |
| pentester | deepseek-v4-pro | true | 23/23 (100.00%) | 3.086s |
**Total**: 280/281 (99.64%) successful tests
**Overall average latency**: 2.139s
## Detailed Results
### simple (deepseek-v4-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.973s | |
| Text Transform Uppercase | ✅ Pass | 0.709s | |
| Count from 1 to 5 | ✅ Pass | 0.903s | |
| Math Calculation | ✅ Pass | 0.698s | |
| Basic Echo Function | ✅ Pass | 1.075s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.775s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.871s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.081s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.254s | |
| Search Query Function | ✅ Pass | 0.963s | |
| Ask Advice Function | ✅ Pass | 1.142s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.976s | |
| Basic Context Memory Test | ✅ Pass | 0.768s | |
| Function Argument Memory Test | ✅ Pass | 0.865s | |
| Function Response Memory Test | ✅ Pass | 0.782s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.414s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.847s | |
| Penetration Testing Methodology | ✅ Pass | 1.116s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.258s | |
| SQL Injection Attack Type | ✅ Pass | 0.876s | |
| Penetration Testing Framework | ✅ Pass | 1.179s | |
| Web Application Security Scanner | ✅ Pass | 1.190s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.180s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.039s
---
### simple_json (deepseek-v4-flash)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Person Information JSON | ✅ Pass | 0.872s | |
| Project Information JSON | ✅ Pass | 0.944s | |
| Vulnerability Report Memory Test | ✅ Pass | 1.059s | |
| User Profile JSON | ✅ Pass | 0.976s | |
| Streaming Person Information JSON Streaming | ✅ Pass | 1.093s | |
**Summary**: 5/5 (100.00%) successful tests
**Average latency**: 0.989s
---
### primary_agent (deepseek-v4-pro)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.178s | |
| Text Transform Uppercase | ✅ Pass | 1.700s | |
| Count from 1 to 5 | ✅ Pass | 4.825s | |
| Math Calculation | ✅ Pass | 1.798s | |
| Basic Echo Function | ✅ Pass | 2.080s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.931s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.996s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.133s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.657s | |
| Search Query Function | ✅ Pass | 2.248s | |
| Ask Advice Function | ✅ Pass | 2.200s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.060s | |
| Basic Context Memory Test | ✅ Pass | 3.547s | |
| Function Argument Memory Test | ✅ Pass | 2.111s | |
| Function Response Memory Test | ✅ Pass | 2.269s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.453s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.267s | |
| Penetration Testing Methodology | ✅ Pass | 4.575s | |
| Vulnerability Assessment Tools | ✅ Pass | 7.988s | |
| SQL Injection Attack Type | ✅ Pass | 2.190s | |
| Penetration Testing Framework | ✅ Pass | 5.697s | |
| Web Application Security Scanner | ✅ Pass | 2.219s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.468s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.896s
---
### assistant (deepseek-v4-pro)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.123s | |
| Text Transform Uppercase | ✅ Pass | 1.972s | |
| Count from 1 to 5 | ✅ Pass | 6.415s | |
| Math Calculation | ✅ Pass | 1.814s | |
| Basic Echo Function | ✅ Pass | 2.143s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.970s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.243s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.283s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.509s | |
| Search Query Function | ✅ Pass | 2.421s | |
| Ask Advice Function | ✅ Pass | 2.316s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.957s | |
| Basic Context Memory Test | ✅ Pass | 2.203s | |
| Function Argument Memory Test | ✅ Pass | 2.293s | |
| Function Response Memory Test | ✅ Pass | 2.042s | |
| Penetration Testing Memory with Tool Call | ❌ Fail | 3.318s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.571s | |
| Penetration Testing Methodology | ✅ Pass | 3.431s | |
| SQL Injection Attack Type | ✅ Pass | 2.289s | |
| Vulnerability Assessment Tools | ✅ Pass | 10.808s | |
| Penetration Testing Framework | ✅ Pass | 3.233s | |
| Web Application Security Scanner | ✅ Pass | 3.258s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.752s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 2.929s
---
### generator (deepseek-v4-pro)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.071s | |
| Text Transform Uppercase | ✅ Pass | 2.231s | |
| Count from 1 to 5 | ✅ Pass | 6.367s | |
| Math Calculation | ✅ Pass | 1.514s | |
| Basic Echo Function | ✅ Pass | 1.995s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.534s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.143s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.014s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.431s | |
| Search Query Function | ✅ Pass | 2.231s | |
| Ask Advice Function | ✅ Pass | 2.042s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.115s | |
| Basic Context Memory Test | ✅ Pass | 2.382s | |
| Function Argument Memory Test | ✅ Pass | 2.525s | |
| Function Response Memory Test | ✅ Pass | 2.067s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.271s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.383s | |
| Penetration Testing Methodology | ✅ Pass | 4.247s | |
| SQL Injection Attack Type | ✅ Pass | 2.293s | |
| Vulnerability Assessment Tools | ✅ Pass | 12.485s | |
| Penetration Testing Framework | ✅ Pass | 4.695s | |
| Web Application Security Scanner | ✅ Pass | 2.208s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.794s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.046s
---
### refiner (deepseek-v4-pro)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.115s | |
| Text Transform Uppercase | ✅ Pass | 1.996s | |
| Count from 1 to 5 | ✅ Pass | 4.321s | |
| Math Calculation | ✅ Pass | 1.722s | |
| Basic Echo Function | ✅ Pass | 2.028s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.742s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.044s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.170s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.374s | |
| Search Query Function | ✅ Pass | 2.454s | |
| Ask Advice Function | ✅ Pass | 2.166s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.974s | |
| Basic Context Memory Test | ✅ Pass | 3.341s | |
| Function Argument Memory Test | ✅ Pass | 2.086s | |
| Function Response Memory Test | ✅ Pass | 2.400s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.446s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.015s | |
| Penetration Testing Methodology | ✅ Pass | 3.273s | |
| SQL Injection Attack Type | ✅ Pass | 3.515s | |
| Vulnerability Assessment Tools | ✅ Pass | 11.480s | |
| Penetration Testing Framework | ✅ Pass | 4.129s | |
| Web Application Security Scanner | ✅ Pass | 2.298s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.323s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.932s
---
### adviser (deepseek-v4-pro)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.074s | |
| Text Transform Uppercase | ✅ Pass | 1.958s | |
| Count from 1 to 5 | ✅ Pass | 4.874s | |
| Math Calculation | ✅ Pass | 1.681s | |
| Basic Echo Function | ✅ Pass | 2.290s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.667s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.040s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.949s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.484s | |
| Search Query Function | ✅ Pass | 2.248s | |
| Ask Advice Function | ✅ Pass | 1.988s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.986s | |
| Basic Context Memory Test | ✅ Pass | 2.873s | |
| Function Argument Memory Test | ✅ Pass | 2.211s | |
| Function Response Memory Test | ✅ Pass | 1.874s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.502s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.064s | |
| Penetration Testing Methodology | ✅ Pass | 3.537s | |
| Vulnerability Assessment Tools | ✅ Pass | 5.184s | |
| SQL Injection Attack Type | ✅ Pass | 4.424s | |
| Penetration Testing Framework | ✅ Pass | 3.939s | |
| Web Application Security Scanner | ✅ Pass | 2.206s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.267s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.667s
---
### reflector (deepseek-v4-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.798s | |
| Text Transform Uppercase | ✅ Pass | 0.888s | |
| Count from 1 to 5 | ✅ Pass | 7.083s | |
| Math Calculation | ✅ Pass | 0.887s | |
| Basic Echo Function | ✅ Pass | 1.023s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.759s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.762s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.272s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.026s | |
| Search Query Function | ✅ Pass | 0.935s | |
| Ask Advice Function | ✅ Pass | 1.016s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.064s | |
| Basic Context Memory Test | ✅ Pass | 0.733s | |
| Function Argument Memory Test | ✅ Pass | 0.869s | |
| Function Response Memory Test | ✅ Pass | 0.791s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.358s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.853s | |
| Penetration Testing Methodology | ✅ Pass | 1.117s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.876s | |
| SQL Injection Attack Type | ✅ Pass | 0.844s | |
| Penetration Testing Framework | ✅ Pass | 1.286s | |
| Web Application Security Scanner | ✅ Pass | 1.159s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.207s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.288s
---
### searcher (deepseek-v4-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.813s | |
| Text Transform Uppercase | ✅ Pass | 0.770s | |
| Count from 1 to 5 | ✅ Pass | 0.893s | |
| Math Calculation | ✅ Pass | 0.742s | |
| Basic Echo Function | ✅ Pass | 0.971s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.695s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.929s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.123s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.010s | |
| Search Query Function | ✅ Pass | 0.922s | |
| Ask Advice Function | ✅ Pass | 1.081s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.909s | |
| Basic Context Memory Test | ✅ Pass | 0.755s | |
| Function Argument Memory Test | ✅ Pass | 0.789s | |
| Function Response Memory Test | ✅ Pass | 0.742s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.220s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.870s | |
| Penetration Testing Methodology | ✅ Pass | 0.981s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.004s | |
| SQL Injection Attack Type | ✅ Pass | 0.899s | |
| Penetration Testing Framework | ✅ Pass | 1.223s | |
| Web Application Security Scanner | ✅ Pass | 1.287s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.151s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.991s
---
### enricher (deepseek-v4-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.759s | |
| Text Transform Uppercase | ✅ Pass | 0.897s | |
| Count from 1 to 5 | ✅ Pass | 0.872s | |
| Math Calculation | ✅ Pass | 0.710s | |
| Basic Echo Function | ✅ Pass | 0.948s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.870s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.746s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.899s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.144s | |
| Search Query Function | ✅ Pass | 0.972s | |
| Ask Advice Function | ✅ Pass | 0.989s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.015s | |
| Basic Context Memory Test | ✅ Pass | 0.858s | |
| Function Argument Memory Test | ✅ Pass | 0.726s | |
| Function Response Memory Test | ✅ Pass | 0.754s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.414s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.851s | |
| Penetration Testing Methodology | ✅ Pass | 0.991s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.597s | |
| SQL Injection Attack Type | ✅ Pass | 0.857s | |
| Penetration Testing Framework | ✅ Pass | 1.269s | |
| Web Application Security Scanner | ✅ Pass | 1.036s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.377s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.981s
---
### coder (deepseek-v4-pro)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.791s | |
| Text Transform Uppercase | ✅ Pass | 6.028s | |
| Count from 1 to 5 | ✅ Pass | 1.873s | |
| Math Calculation | ✅ Pass | 1.707s | |
| Basic Echo Function | ✅ Pass | 2.051s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.951s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.989s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.446s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.494s | |
| Search Query Function | ✅ Pass | 2.010s | |
| Ask Advice Function | ✅ Pass | 2.242s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.133s | |
| Basic Context Memory Test | ✅ Pass | 3.207s | |
| Function Argument Memory Test | ✅ Pass | 2.075s | |
| Function Response Memory Test | ✅ Pass | 1.921s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.561s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.345s | |
| Penetration Testing Methodology | ✅ Pass | 3.598s | |
| Vulnerability Assessment Tools | ✅ Pass | 8.780s | |
| SQL Injection Attack Type | ✅ Pass | 2.581s | |
| Penetration Testing Framework | ✅ Pass | 2.960s | |
| Web Application Security Scanner | ✅ Pass | 2.493s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.548s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.817s
---
### installer (deepseek-v4-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.185s | |
| Text Transform Uppercase | ✅ Pass | 1.103s | |
| Count from 1 to 5 | ✅ Pass | 1.222s | |
| Math Calculation | ✅ Pass | 0.962s | |
| Basic Echo Function | ✅ Pass | 0.990s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.908s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.216s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.106s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.264s | |
| Search Query Function | ✅ Pass | 1.140s | |
| Ask Advice Function | ✅ Pass | 1.158s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.171s | |
| Basic Context Memory Test | ✅ Pass | 1.188s | |
| Function Argument Memory Test | ✅ Pass | 0.905s | |
| Function Response Memory Test | ✅ Pass | 1.120s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.758s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.140s | |
| Penetration Testing Methodology | ✅ Pass | 1.382s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.060s | |
| SQL Injection Attack Type | ✅ Pass | 1.298s | |
| Penetration Testing Framework | ✅ Pass | 1.423s | |
| Web Application Security Scanner | ✅ Pass | 1.586s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.352s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.246s
---
### pentester (deepseek-v4-pro)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.845s | |
| Text Transform Uppercase | ✅ Pass | 6.297s | |
| Count from 1 to 5 | ✅ Pass | 1.982s | |
| Math Calculation | ✅ Pass | 1.653s | |
| Basic Echo Function | ✅ Pass | 1.838s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.884s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.004s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.209s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.702s | |
| Search Query Function | ✅ Pass | 1.898s | |
| Ask Advice Function | ✅ Pass | 2.049s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.145s | |
| Basic Context Memory Test | ✅ Pass | 3.227s | |
| Function Argument Memory Test | ✅ Pass | 1.892s | |
| Function Response Memory Test | ✅ Pass | 2.334s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.631s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.276s | |
| Penetration Testing Methodology | ✅ Pass | 3.126s | |
| SQL Injection Attack Type | ✅ Pass | 2.551s | |
| Vulnerability Assessment Tools | ✅ Pass | 14.848s | |
| Penetration Testing Framework | ✅ Pass | 3.944s | |
| Web Application Security Scanner | ✅ Pass | 1.612s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.012s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.086s
---
examples/tests/gemini-report.md
+537
View File
@@ -0,0 +1,537 @@
# LLM Agent Testing Report
Generated: Wed, 27 May 2026 22:33:44 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
| simple | gemini-3.1-flash-lite | true | 23/23 (100.00%) | 0.768s |
| simple_json | gemini-3.1-flash-lite | true | 5/5 (100.00%) | 0.715s |
| primary_agent | gemini-3.1-pro-preview | true | 22/23 (95.65%) | 4.190s |
| assistant | gemini-3.1-pro-preview | true | 22/23 (95.65%) | 4.239s |
| generator | gemini-3.1-pro-preview | true | 21/23 (91.30%) | 4.434s |
| refiner | gemini-3.1-pro-preview | true | 23/23 (100.00%) | 4.221s |
| adviser | gemini-3.1-pro-preview | true | 21/23 (91.30%) | 4.304s |
| reflector | gemini-3.5-flash | true | 23/23 (100.00%) | 1.692s |
| searcher | gemini-3.5-flash | true | 23/23 (100.00%) | 1.817s |
| enricher | gemini-3.5-flash | true | 23/23 (100.00%) | 1.724s |
| coder | gemini-3.1-pro-preview | true | 23/23 (100.00%) | 4.103s |
| installer | gemini-3.5-flash | true | 22/23 (95.65%) | 2.891s |
| pentester | gemini-3.1-pro-preview | true | 23/23 (100.00%) | 4.197s |
**Total**: 274/281 (97.51%) successful tests
**Overall average latency**: 3.171s
## Detailed Results
### simple (gemini-3.1-flash-lite)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.367s | |
| Text Transform Uppercase | ✅ Pass | 0.517s | |
| Count from 1 to 5 | ✅ Pass | 0.704s | |
| Math Calculation | ✅ Pass | 0.530s | |
| Basic Echo Function | ✅ Pass | 0.524s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.506s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.626s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.663s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.578s | |
| Search Query Function | ✅ Pass | 0.544s | |
| Ask Advice Function | ✅ Pass | 0.576s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.529s | |
| Basic Context Memory Test | ✅ Pass | 0.551s | |
| Function Argument Memory Test | ✅ Pass | 0.490s | |
| Function Response Memory Test | ✅ Pass | 0.688s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.597s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.546s | |
| Penetration Testing Methodology | ✅ Pass | 1.218s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.050s | |
| SQL Injection Attack Type | ✅ Pass | 0.661s | |
| Penetration Testing Framework | ✅ Pass | 0.762s | |
| Web Application Security Scanner | ✅ Pass | 0.750s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.673s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.768s
---
### simple_json (gemini-3.1-flash-lite)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Person Information JSON | ✅ Pass | 0.664s | |
| Vulnerability Report Memory Test | ✅ Pass | 0.961s | |
| User Profile JSON | ✅ Pass | 0.594s | |
| Project Information JSON | ✅ Pass | 0.766s | |
| Streaming Person Information JSON Streaming | ✅ Pass | 0.590s | |
**Summary**: 5/5 (100.00%) successful tests
**Average latency**: 0.715s
---
### primary_agent (gemini-3.1-pro-preview)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 3.812s | |
| Text Transform Uppercase | ✅ Pass | 3.313s | |
| Count from 1 to 5 | ✅ Pass | 4.086s | |
| Math Calculation | ✅ Pass | 2.632s | |
| Basic Echo Function | ❌ Fail | 4.651s | no tool calls found, expected at least 1 |
| Streaming Simple Math Streaming | ✅ Pass | 3.243s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.803s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.800s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 5.060s | |
| Search Query Function | ✅ Pass | 4.701s | |
| Ask Advice Function | ✅ Pass | 6.206s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.818s | |
| Basic Context Memory Test | ✅ Pass | 3.449s | |
| Function Argument Memory Test | ✅ Pass | 4.006s | |
| Function Response Memory Test | ✅ Pass | 4.055s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.735s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.365s | |
| Penetration Testing Methodology | ✅ Pass | 5.392s | |
| Vulnerability Assessment Tools | ✅ Pass | 5.336s | |
| SQL Injection Attack Type | ✅ Pass | 3.569s | |
| Penetration Testing Framework | ✅ Pass | 4.948s | |
| Web Application Security Scanner | ✅ Pass | 5.728s | |
| Penetration Testing Tool Selection | ✅ Pass | 5.643s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 4.190s
---
### assistant (gemini-3.1-pro-preview)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.818s | |
| Text Transform Uppercase | ✅ Pass | 3.634s | |
| Count from 1 to 5 | ✅ Pass | 4.193s | |
| Math Calculation | ✅ Pass | 3.305s | |
| Basic Echo Function | ❌ Fail | 5.872s | no tool calls found, expected at least 1 |
| Streaming Simple Math Streaming | ✅ Pass | 3.279s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.548s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.377s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.867s | |
| Search Query Function | ✅ Pass | 3.219s | |
| Ask Advice Function | ✅ Pass | 5.921s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.671s | |
| Basic Context Memory Test | ✅ Pass | 3.188s | |
| Function Argument Memory Test | ✅ Pass | 4.451s | |
| Function Response Memory Test | ✅ Pass | 3.309s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 6.425s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.503s | |
| Penetration Testing Methodology | ✅ Pass | 5.211s | |
| Vulnerability Assessment Tools | ✅ Pass | 5.501s | |
| SQL Injection Attack Type | ✅ Pass | 4.066s | |
| Penetration Testing Framework | ✅ Pass | 5.491s | |
| Web Application Security Scanner | ✅ Pass | 5.037s | |
| Penetration Testing Tool Selection | ✅ Pass | 6.604s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 4.239s
---
### generator (gemini-3.1-pro-preview)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 3.066s | |
| Text Transform Uppercase | ✅ Pass | 4.138s | |
| Count from 1 to 5 | ✅ Pass | 3.613s | |
| Math Calculation | ✅ Pass | 3.089s | |
| Basic Echo Function | ❌ Fail | 4.109s | no tool calls found, expected at least 1 |
| Streaming Simple Math Streaming | ✅ Pass | 3.430s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.411s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.347s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 4.487s | |
| Search Query Function | ✅ Pass | 3.946s | |
| Ask Advice Function | ✅ Pass | 4.281s | |
| Streaming Search Query Function Streaming | ❌ Fail | 3.204s | no tool calls found, expected at least 1 |
| Basic Context Memory Test | ✅ Pass | 4.219s | |
| Function Argument Memory Test | ✅ Pass | 4.124s | |
| Function Response Memory Test | ✅ Pass | 3.339s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 7.760s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.648s | |
| Penetration Testing Methodology | ✅ Pass | 7.310s | |
| Vulnerability Assessment Tools | ✅ Pass | 9.153s | |
| SQL Injection Attack Type | ✅ Pass | 3.768s | |
| Penetration Testing Framework | ✅ Pass | 3.997s | |
| Web Application Security Scanner | ✅ Pass | 4.903s | |
| Penetration Testing Tool Selection | ✅ Pass | 6.634s | |
**Summary**: 21/23 (91.30%) successful tests
**Average latency**: 4.434s
---
### refiner (gemini-3.1-pro-preview)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 3.559s | |
| Text Transform Uppercase | ✅ Pass | 4.160s | |
| Count from 1 to 5 | ✅ Pass | 3.893s | |
| Math Calculation | ✅ Pass | 2.809s | |
| Basic Echo Function | ✅ Pass | 4.024s | |
| Streaming Simple Math Streaming | ✅ Pass | 3.314s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.412s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.594s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 4.043s | |
| Search Query Function | ✅ Pass | 3.295s | |
| Ask Advice Function | ✅ Pass | 4.504s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.448s | |
| Basic Context Memory Test | ✅ Pass | 3.640s | |
| Function Argument Memory Test | ✅ Pass | 3.824s | |
| Function Response Memory Test | ✅ Pass | 4.427s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 7.450s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.576s | |
| Penetration Testing Methodology | ✅ Pass | 5.822s | |
| Vulnerability Assessment Tools | ✅ Pass | 6.900s | |
| SQL Injection Attack Type | ✅ Pass | 3.912s | |
| Penetration Testing Framework | ✅ Pass | 4.930s | |
| Web Application Security Scanner | ✅ Pass | 5.459s | |
| Penetration Testing Tool Selection | ✅ Pass | 6.079s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 4.221s
---
### adviser (gemini-3.1-pro-preview)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.811s | |
| Text Transform Uppercase | ✅ Pass | 3.618s | |
| Count from 1 to 5 | ✅ Pass | 5.524s | |
| Math Calculation | ✅ Pass | 3.096s | |
| Basic Echo Function | ✅ Pass | 5.575s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.725s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.862s | |
| Streaming Basic Echo Function Streaming | ❌ Fail | 3.410s | no tool calls found, expected at least 1 |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.542s | |
| Search Query Function | ❌ Fail | 4.056s | no tool calls found, expected at least 1 |
| Ask Advice Function | ✅ Pass | 3.966s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.561s | |
| Basic Context Memory Test | ✅ Pass | 3.647s | |
| Function Argument Memory Test | ✅ Pass | 4.782s | |
| Function Response Memory Test | ✅ Pass | 4.070s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 6.209s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.835s | |
| Penetration Testing Methodology | ✅ Pass | 6.464s | |
| Vulnerability Assessment Tools | ✅ Pass | 6.427s | |
| SQL Injection Attack Type | ✅ Pass | 3.244s | |
| Penetration Testing Framework | ✅ Pass | 5.763s | |
| Web Application Security Scanner | ✅ Pass | 5.714s | |
| Penetration Testing Tool Selection | ✅ Pass | 5.082s | |
**Summary**: 21/23 (91.30%) successful tests
**Average latency**: 4.304s
---
### reflector (gemini-3.5-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.016s | |
| Text Transform Uppercase | ✅ Pass | 1.398s | |
| Count from 1 to 5 | ✅ Pass | 1.549s | |
| Math Calculation | ✅ Pass | 1.023s | |
| Basic Echo Function | ✅ Pass | 1.100s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.953s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.477s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.172s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.330s | |
| Search Query Function | ✅ Pass | 0.972s | |
| Ask Advice Function | ✅ Pass | 1.386s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.149s | |
| Basic Context Memory Test | ✅ Pass | 1.303s | |
| Function Argument Memory Test | ✅ Pass | 1.155s | |
| Function Response Memory Test | ✅ Pass | 1.319s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.053s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.048s | |
| Penetration Testing Methodology | ✅ Pass | 2.124s | |
| Vulnerability Assessment Tools | ✅ Pass | 3.923s | |
| SQL Injection Attack Type | ✅ Pass | 1.678s | |
| Penetration Testing Framework | ✅ Pass | 2.756s | |
| Web Application Security Scanner | ✅ Pass | 1.608s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.421s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.692s
---
### searcher (gemini-3.5-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.992s | |
| Text Transform Uppercase | ✅ Pass | 1.363s | |
| Count from 1 to 5 | ✅ Pass | 1.590s | |
| Math Calculation | ✅ Pass | 0.971s | |
| Basic Echo Function | ✅ Pass | 1.336s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.916s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.542s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.124s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.079s | |
| Search Query Function | ✅ Pass | 1.121s | |
| Ask Advice Function | ✅ Pass | 1.387s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.260s | |
| Basic Context Memory Test | ✅ Pass | 1.399s | |
| Function Argument Memory Test | ✅ Pass | 1.557s | |
| Function Response Memory Test | ✅ Pass | 1.474s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.472s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.838s | |
| Penetration Testing Methodology | ✅ Pass | 2.772s | |
| Vulnerability Assessment Tools | ✅ Pass | 4.313s | |
| SQL Injection Attack Type | ✅ Pass | 1.202s | |
| Penetration Testing Framework | ✅ Pass | 2.830s | |
| Web Application Security Scanner | ✅ Pass | 2.120s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.116s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.817s
---
### enricher (gemini-3.5-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.981s | |
| Text Transform Uppercase | ✅ Pass | 1.215s | |
| Count from 1 to 5 | ✅ Pass | 1.425s | |
| Math Calculation | ✅ Pass | 0.932s | |
| Basic Echo Function | ✅ Pass | 1.084s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.958s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.604s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.237s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.128s | |
| Search Query Function | ✅ Pass | 0.993s | |
| Ask Advice Function | ✅ Pass | 1.090s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.009s | |
| Basic Context Memory Test | ✅ Pass | 1.461s | |
| Function Argument Memory Test | ✅ Pass | 0.955s | |
| Function Response Memory Test | ✅ Pass | 1.856s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.277s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.771s | |
| Penetration Testing Methodology | ✅ Pass | 2.691s | |
| Vulnerability Assessment Tools | ✅ Pass | 3.802s | |
| SQL Injection Attack Type | ✅ Pass | 1.611s | |
| Penetration Testing Framework | ✅ Pass | 2.182s | |
| Web Application Security Scanner | ✅ Pass | 2.253s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.135s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.724s
---
### coder (gemini-3.1-pro-preview)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 3.599s | |
| Text Transform Uppercase | ✅ Pass | 3.774s | |
| Count from 1 to 5 | ✅ Pass | 3.470s | |
| Math Calculation | ✅ Pass | 3.170s | |
| Basic Echo Function | ✅ Pass | 4.147s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.895s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.859s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.385s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 5.047s | |
| Search Query Function | ✅ Pass | 3.097s | |
| Ask Advice Function | ✅ Pass | 3.666s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.431s | |
| Basic Context Memory Test | ✅ Pass | 3.474s | |
| Function Argument Memory Test | ✅ Pass | 3.706s | |
| Function Response Memory Test | ✅ Pass | 3.504s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 6.181s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.421s | |
| Penetration Testing Methodology | ✅ Pass | 5.411s | |
| Vulnerability Assessment Tools | ✅ Pass | 8.503s | |
| SQL Injection Attack Type | ✅ Pass | 3.996s | |
| Penetration Testing Framework | ✅ Pass | 4.476s | |
| Web Application Security Scanner | ✅ Pass | 4.807s | |
| Penetration Testing Tool Selection | ✅ Pass | 6.343s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 4.103s
---
### installer (gemini-3.5-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.355s | |
| Text Transform Uppercase | ✅ Pass | 3.024s | |
| Count from 1 to 5 | ✅ Pass | 2.930s | |
| Math Calculation | ✅ Pass | 1.667s | |
| Basic Echo Function | ✅ Pass | 2.377s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.014s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.497s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.267s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.773s | |
| Search Query Function | ✅ Pass | 2.193s | |
| Ask Advice Function | ✅ Pass | 1.839s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.064s | |
| Basic Context Memory Test | ✅ Pass | 2.661s | |
| Function Argument Memory Test | ✅ Pass | 2.603s | |
| Function Response Memory Test | ✅ Pass | 2.563s | |
| Penetration Testing Memory with Tool Call | ❌ Fail | 5.230s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
| Cybersecurity Workflow Memory Test | ✅ Pass | 5.456s | |
| Penetration Testing Methodology | ✅ Pass | 4.745s | |
| Vulnerability Assessment Tools | ✅ Pass | 5.977s | |
| SQL Injection Attack Type | ✅ Pass | 2.499s | |
| Penetration Testing Framework | ✅ Pass | 4.620s | |
| Web Application Security Scanner | ✅ Pass | 4.105s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.022s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 2.891s
---
### pentester (gemini-3.1-pro-preview)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.995s | |
| Text Transform Uppercase | ✅ Pass | 3.980s | |
| Count from 1 to 5 | ✅ Pass | 3.836s | |
| Math Calculation | ✅ Pass | 3.256s | |
| Basic Echo Function | ✅ Pass | 4.078s | |
| Streaming Simple Math Streaming | ✅ Pass | 3.645s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.757s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.314s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 4.023s | |
| Search Query Function | ✅ Pass | 3.725s | |
| Ask Advice Function | ✅ Pass | 4.173s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.589s | |
| Basic Context Memory Test | ✅ Pass | 3.997s | |
| Function Argument Memory Test | ✅ Pass | 4.753s | |
| Function Response Memory Test | ✅ Pass | 4.861s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 5.704s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.880s | |
| Penetration Testing Methodology | ✅ Pass | 6.182s | |
| Vulnerability Assessment Tools | ✅ Pass | 5.410s | |
| SQL Injection Attack Type | ✅ Pass | 3.714s | |
| Penetration Testing Framework | ✅ Pass | 4.977s | |
| Web Application Security Scanner | ✅ Pass | 5.953s | |
| Penetration Testing Tool Selection | ✅ Pass | 5.708s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 4.197s
---
examples/tests/glm-report.md
+537
View File
@@ -0,0 +1,537 @@
# LLM Agent Testing Report
Generated: Thu, 28 May 2026 12:25:41 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
| simple | glm-4.5-air | false | 23/23 (100.00%) | 3.213s |
| simple_json | glm-4.5-air | false | 5/5 (100.00%) | 3.923s |
| primary_agent | glm-5-turbo | true | 23/23 (100.00%) | 4.488s |
| assistant | glm-5-turbo | true | 22/23 (95.65%) | 3.383s |
| generator | glm-5.1 | true | 23/23 (100.00%) | 7.081s |
| refiner | glm-5.1 | true | 22/23 (95.65%) | 6.864s |
| adviser | glm-5.1 | true | 23/23 (100.00%) | 6.553s |
| reflector | glm-4.5-air | true | 23/23 (100.00%) | 2.760s |
| searcher | glm-4.5-air | true | 23/23 (100.00%) | 2.889s |
| enricher | glm-4.5-air | true | 23/23 (100.00%) | 3.001s |
| coder | glm-5.1 | true | 22/23 (95.65%) | 7.172s |
| installer | glm-4.5-air | true | 23/23 (100.00%) | 6.146s |
| pentester | glm-5.1 | true | 23/23 (100.00%) | 6.772s |
**Total**: 278/281 (98.93%) successful tests
**Overall average latency**: 5.007s
## Detailed Results
### simple (glm-4.5-air)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 4.356s | |
| Text Transform Uppercase | ✅ Pass | 1.140s | |
| Count from 1 to 5 | ✅ Pass | 1.534s | |
| Math Calculation | ✅ Pass | 1.975s | |
| Basic Echo Function | ✅ Pass | 1.529s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.911s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.859s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.170s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.415s | |
| Search Query Function | ✅ Pass | 3.528s | |
| Ask Advice Function | ✅ Pass | 1.866s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.467s | |
| Basic Context Memory Test | ✅ Pass | 1.478s | |
| Function Argument Memory Test | ✅ Pass | 1.072s | |
| Function Response Memory Test | ✅ Pass | 2.755s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.953s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.161s | |
| Penetration Testing Methodology | ✅ Pass | 7.480s | |
| Vulnerability Assessment Tools | ✅ Pass | 18.361s | |
| SQL Injection Attack Type | ✅ Pass | 1.326s | |
| Penetration Testing Framework | ✅ Pass | 7.132s | |
| Web Application Security Scanner | ✅ Pass | 3.189s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.219s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.213s
---
### simple_json (glm-4.5-air)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Person Information JSON | ✅ Pass | 4.516s | |
| Vulnerability Report Memory Test | ✅ Pass | 5.454s | |
| Project Information JSON | ✅ Pass | 1.830s | |
| User Profile JSON | ✅ Pass | 3.002s | |
| Streaming Person Information JSON Streaming | ✅ Pass | 4.808s | |
**Summary**: 5/5 (100.00%) successful tests
**Average latency**: 3.923s
---
### primary_agent (glm-5-turbo)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 5.935s | |
| Text Transform Uppercase | ✅ Pass | 3.695s | |
| Count from 1 to 5 | ✅ Pass | 3.836s | |
| Math Calculation | ✅ Pass | 3.798s | |
| Basic Echo Function | ✅ Pass | 2.191s | |
| Streaming Simple Math Streaming | ✅ Pass | 3.538s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.337s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.491s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.857s | |
| Search Query Function | ✅ Pass | 1.869s | |
| Ask Advice Function | ✅ Pass | 1.275s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.561s | |
| Basic Context Memory Test | ✅ Pass | 5.012s | |
| Function Argument Memory Test | ✅ Pass | 3.120s | |
| Function Response Memory Test | ✅ Pass | 1.293s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.664s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.496s | |
| Penetration Testing Methodology | ✅ Pass | 22.967s | |
| Vulnerability Assessment Tools | ✅ Pass | 9.455s | |
| SQL Injection Attack Type | ✅ Pass | 5.215s | |
| Penetration Testing Framework | ✅ Pass | 5.249s | |
| Web Application Security Scanner | ✅ Pass | 5.172s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.180s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 4.488s
---
### assistant (glm-5-turbo)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.980s | |
| Text Transform Uppercase | ✅ Pass | 2.896s | |
| Count from 1 to 5 | ✅ Pass | 2.572s | |
| Math Calculation | ✅ Pass | 3.321s | |
| Basic Echo Function | ✅ Pass | 1.431s | |
| Streaming Simple Math Streaming | ✅ Pass | 3.935s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.138s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.242s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.378s | |
| Search Query Function | ✅ Pass | 1.954s | |
| Ask Advice Function | ✅ Pass | 1.638s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.390s | |
| Basic Context Memory Test | ✅ Pass | 4.105s | |
| Function Argument Memory Test | ✅ Pass | 1.651s | |
| Function Response Memory Test | ✅ Pass | 2.104s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 6.155s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 5.861s | |
| Penetration Testing Methodology | ✅ Pass | 5.751s | |
| Vulnerability Assessment Tools | ❌ Fail | 6.224s | expected text 'network' not found |
| SQL Injection Attack Type | ✅ Pass | 4.402s | |
| Penetration Testing Framework | ✅ Pass | 6.187s | |
| Web Application Security Scanner | ✅ Pass | 5.577s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.904s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 3.383s
---
### generator (glm-5.1)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 5.598s | |
| Text Transform Uppercase | ✅ Pass | 4.686s | |
| Count from 1 to 5 | ✅ Pass | 4.534s | |
| Math Calculation | ✅ Pass | 4.466s | |
| Basic Echo Function | ✅ Pass | 5.033s | |
| Streaming Simple Math Streaming | ✅ Pass | 3.667s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.827s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.858s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 5.855s | |
| Search Query Function | ✅ Pass | 5.443s | |
| Ask Advice Function | ✅ Pass | 3.380s | |
| Streaming Search Query Function Streaming | ✅ Pass | 4.654s | |
| Basic Context Memory Test | ✅ Pass | 7.745s | |
| Function Argument Memory Test | ✅ Pass | 9.781s | |
| Function Response Memory Test | ✅ Pass | 6.321s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 6.979s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 5.237s | |
| Penetration Testing Methodology | ✅ Pass | 13.391s | |
| Vulnerability Assessment Tools | ✅ Pass | 19.520s | |
| SQL Injection Attack Type | ✅ Pass | 14.260s | |
| Penetration Testing Framework | ✅ Pass | 7.151s | |
| Web Application Security Scanner | ✅ Pass | 12.338s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.118s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 7.081s
---
### refiner (glm-5.1)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 7.346s | |
| Text Transform Uppercase | ✅ Pass | 7.388s | |
| Count from 1 to 5 | ✅ Pass | 7.160s | |
| Math Calculation | ✅ Pass | 4.476s | |
| Basic Echo Function | ✅ Pass | 7.189s | |
| Streaming Simple Math Streaming | ✅ Pass | 9.466s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.994s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.243s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 6.498s | |
| Search Query Function | ✅ Pass | 5.493s | |
| Ask Advice Function | ✅ Pass | 4.021s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.686s | |
| Basic Context Memory Test | ✅ Pass | 5.153s | |
| Function Argument Memory Test | ✅ Pass | 5.468s | |
| Function Response Memory Test | ✅ Pass | 3.245s | |
| Penetration Testing Memory with Tool Call | ❌ Fail | 7.345s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.404s | |
| Penetration Testing Methodology | ✅ Pass | 12.575s | |
| Vulnerability Assessment Tools | ✅ Pass | 16.644s | |
| SQL Injection Attack Type | ✅ Pass | 5.347s | |
| Penetration Testing Framework | ✅ Pass | 11.479s | |
| Web Application Security Scanner | ✅ Pass | 12.258s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.992s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 6.864s
---
### adviser (glm-5.1)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 7.436s | |
| Text Transform Uppercase | ✅ Pass | 4.739s | |
| Count from 1 to 5 | ✅ Pass | 7.323s | |
| Math Calculation | ✅ Pass | 3.884s | |
| Basic Echo Function | ✅ Pass | 4.145s | |
| Streaming Simple Math Streaming | ✅ Pass | 4.102s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.763s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 5.518s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.657s | |
| Search Query Function | ✅ Pass | 3.205s | |
| Ask Advice Function | ✅ Pass | 3.559s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.959s | |
| Basic Context Memory Test | ✅ Pass | 6.817s | |
| Function Argument Memory Test | ✅ Pass | 6.147s | |
| Function Response Memory Test | ✅ Pass | 3.902s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 5.443s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 5.905s | |
| Penetration Testing Methodology | ✅ Pass | 14.182s | |
| Vulnerability Assessment Tools | ✅ Pass | 19.240s | |
| SQL Injection Attack Type | ✅ Pass | 5.849s | |
| Penetration Testing Framework | ✅ Pass | 11.161s | |
| Web Application Security Scanner | ✅ Pass | 11.757s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.019s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 6.553s
---
### reflector (glm-4.5-air)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.349s | |
| Text Transform Uppercase | ✅ Pass | 0.952s | |
| Count from 1 to 5 | ✅ Pass | 3.336s | |
| Math Calculation | ✅ Pass | 0.828s | |
| Basic Echo Function | ✅ Pass | 1.318s | |
| Streaming Simple Math Streaming | ✅ Pass | 8.738s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.624s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.163s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.864s | |
| Search Query Function | ✅ Pass | 1.759s | |
| Ask Advice Function | ✅ Pass | 1.985s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.501s | |
| Basic Context Memory Test | ✅ Pass | 1.776s | |
| Function Argument Memory Test | ✅ Pass | 1.869s | |
| Function Response Memory Test | ✅ Pass | 1.430s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.434s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.338s | |
| Penetration Testing Methodology | ✅ Pass | 6.325s | |
| Vulnerability Assessment Tools | ✅ Pass | 5.048s | |
| SQL Injection Attack Type | ✅ Pass | 2.589s | |
| Penetration Testing Framework | ✅ Pass | 7.158s | |
| Web Application Security Scanner | ✅ Pass | 5.490s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.597s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.760s
---
### searcher (glm-4.5-air)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.144s | |
| Text Transform Uppercase | ✅ Pass | 1.230s | |
| Count from 1 to 5 | ✅ Pass | 2.526s | |
| Math Calculation | ✅ Pass | 0.975s | |
| Basic Echo Function | ✅ Pass | 1.962s | |
| Streaming Simple Math Streaming | ✅ Pass | 6.438s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.872s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.207s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.837s | |
| Search Query Function | ✅ Pass | 2.366s | |
| Ask Advice Function | ✅ Pass | 1.564s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.558s | |
| Basic Context Memory Test | ✅ Pass | 1.415s | |
| Function Argument Memory Test | ✅ Pass | 1.913s | |
| Function Response Memory Test | ✅ Pass | 1.785s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.208s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.507s | |
| Penetration Testing Methodology | ✅ Pass | 5.352s | |
| Vulnerability Assessment Tools | ✅ Pass | 14.510s | |
| SQL Injection Attack Type | ✅ Pass | 2.434s | |
| Penetration Testing Framework | ✅ Pass | 5.262s | |
| Web Application Security Scanner | ✅ Pass | 2.488s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.891s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.889s
---
### enricher (glm-4.5-air)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.166s | |
| Text Transform Uppercase | ✅ Pass | 1.054s | |
| Count from 1 to 5 | ✅ Pass | 1.600s | |
| Math Calculation | ✅ Pass | 1.054s | |
| Basic Echo Function | ✅ Pass | 1.737s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.306s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.486s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.403s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.452s | |
| Search Query Function | ✅ Pass | 1.895s | |
| Ask Advice Function | ✅ Pass | 1.631s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.771s | |
| Basic Context Memory Test | ✅ Pass | 1.640s | |
| Function Argument Memory Test | ✅ Pass | 2.034s | |
| Function Response Memory Test | ✅ Pass | 1.623s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.927s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.506s | |
| Penetration Testing Methodology | ✅ Pass | 8.173s | |
| Vulnerability Assessment Tools | ✅ Pass | 12.181s | |
| SQL Injection Attack Type | ✅ Pass | 1.488s | |
| Penetration Testing Framework | ✅ Pass | 14.886s | |
| Web Application Security Scanner | ✅ Pass | 2.242s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.759s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.001s
---
### coder (glm-5.1)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 4.213s | |
| Text Transform Uppercase | ✅ Pass | 4.692s | |
| Count from 1 to 5 | ✅ Pass | 5.795s | |
| Math Calculation | ✅ Pass | 9.208s | |
| Basic Echo Function | ✅ Pass | 5.473s | |
| Streaming Simple Math Streaming | ✅ Pass | 3.389s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 8.238s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.129s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.931s | |
| Search Query Function | ✅ Pass | 3.264s | |
| Ask Advice Function | ✅ Pass | 7.299s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.841s | |
| Basic Context Memory Test | ✅ Pass | 5.589s | |
| Function Argument Memory Test | ✅ Pass | 6.503s | |
| Function Response Memory Test | ✅ Pass | 3.301s | |
| Penetration Testing Memory with Tool Call | ❌ Fail | 6.407s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
| Cybersecurity Workflow Memory Test | ✅ Pass | 6.279s | |
| Penetration Testing Methodology | ✅ Pass | 17.570s | |
| Vulnerability Assessment Tools | ✅ Pass | 19.116s | |
| SQL Injection Attack Type | ✅ Pass | 5.596s | |
| Penetration Testing Framework | ✅ Pass | 16.889s | |
| Web Application Security Scanner | ✅ Pass | 11.733s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.497s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 7.172s
---
### installer (glm-4.5-air)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 6.253s | |
| Text Transform Uppercase | ✅ Pass | 2.297s | |
| Count from 1 to 5 | ✅ Pass | 3.713s | |
| Math Calculation | ✅ Pass | 2.951s | |
| Basic Echo Function | ✅ Pass | 2.057s | |
| Streaming Simple Math Streaming | ✅ Pass | 3.884s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.290s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.606s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 4.104s | |
| Search Query Function | ✅ Pass | 3.696s | |
| Ask Advice Function | ✅ Pass | 3.538s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.188s | |
| Basic Context Memory Test | ✅ Pass | 3.317s | |
| Function Argument Memory Test | ✅ Pass | 1.915s | |
| Function Response Memory Test | ✅ Pass | 3.620s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 5.784s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.012s | |
| Penetration Testing Methodology | ✅ Pass | 14.559s | |
| Vulnerability Assessment Tools | ✅ Pass | 16.449s | |
| SQL Injection Attack Type | ✅ Pass | 8.793s | |
| Penetration Testing Framework | ✅ Pass | 22.365s | |
| Web Application Security Scanner | ✅ Pass | 15.217s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.740s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 6.146s
---
### pentester (glm-5.1)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 5.261s | |
| Text Transform Uppercase | ✅ Pass | 6.631s | |
| Count from 1 to 5 | ✅ Pass | 4.222s | |
| Math Calculation | ✅ Pass | 3.124s | |
| Basic Echo Function | ✅ Pass | 3.221s | |
| Streaming Simple Math Streaming | ✅ Pass | 6.244s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.420s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.163s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 4.696s | |
| Search Query Function | ✅ Pass | 2.983s | |
| Ask Advice Function | ✅ Pass | 5.813s | |
| Streaming Search Query Function Streaming | ✅ Pass | 9.633s | |
| Basic Context Memory Test | ✅ Pass | 4.770s | |
| Function Argument Memory Test | ✅ Pass | 7.549s | |
| Function Response Memory Test | ✅ Pass | 3.946s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.723s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.120s | |
| Penetration Testing Methodology | ✅ Pass | 17.474s | |
| Vulnerability Assessment Tools | ✅ Pass | 21.649s | |
| SQL Injection Attack Type | ✅ Pass | 6.494s | |
| Penetration Testing Framework | ✅ Pass | 12.428s | |
| Web Application Security Scanner | ✅ Pass | 8.468s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.718s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 6.772s
---
examples/tests/kimi-report.md
+537
View File
@@ -0,0 +1,537 @@
# LLM Agent Testing Report
Generated: Wed, 27 May 2026 23:34:01 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
| simple | kimi-k2.5 | true | 23/23 (100.00%) | 1.452s |
| simple_json | kimi-k2.5 | false | 5/5 (100.00%) | 1.241s |
| primary_agent | kimi-k2.5 | true | 23/23 (100.00%) | 3.602s |
| assistant | kimi-k2.5 | true | 23/23 (100.00%) | 3.660s |
| generator | kimi-k2.6 | true | 23/23 (100.00%) | 3.471s |
| refiner | kimi-k2.6 | true | 23/23 (100.00%) | 3.700s |
| adviser | kimi-k2.6 | true | 23/23 (100.00%) | 3.886s |
| reflector | kimi-k2.5 | true | 23/23 (100.00%) | 1.526s |
| searcher | kimi-k2.5 | true | 23/23 (100.00%) | 1.383s |
| enricher | kimi-k2.5 | true | 23/23 (100.00%) | 1.271s |
| coder | kimi-k2.6 | true | 23/23 (100.00%) | 3.321s |
| installer | kimi-k2.5 | true | 23/23 (100.00%) | 3.381s |
| pentester | kimi-k2.6 | true | 23/23 (100.00%) | 3.432s |
**Total**: 281/281 (100.00%) successful tests
**Overall average latency**: 2.812s
## Detailed Results
### simple (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.967s | |
| Text Transform Uppercase | ✅ Pass | 0.754s | |
| Count from 1 to 5 | ✅ Pass | 0.898s | |
| Math Calculation | ✅ Pass | 0.592s | |
| Basic Echo Function | ✅ Pass | 1.073s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.716s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.857s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.202s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.030s | |
| Search Query Function | ✅ Pass | 0.991s | |
| Ask Advice Function | ✅ Pass | 1.341s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.123s | |
| Basic Context Memory Test | ✅ Pass | 0.844s | |
| Function Argument Memory Test | ✅ Pass | 0.811s | |
| Function Response Memory Test | ✅ Pass | 0.827s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.086s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.772s | |
| Penetration Testing Methodology | ✅ Pass | 2.927s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.957s | |
| SQL Injection Attack Type | ✅ Pass | 0.878s | |
| Penetration Testing Framework | ✅ Pass | 3.484s | |
| Web Application Security Scanner | ✅ Pass | 3.183s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.066s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.452s
---
### simple_json (kimi-k2.5)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Project Information JSON | ✅ Pass | 1.047s | |
| Person Information JSON | ✅ Pass | 1.194s | |
| Vulnerability Report Memory Test | ✅ Pass | 2.029s | |
| User Profile JSON | ✅ Pass | 0.951s | |
| Streaming Person Information JSON Streaming | ✅ Pass | 0.984s | |
**Summary**: 5/5 (100.00%) successful tests
**Average latency**: 1.241s
---
### primary_agent (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 3.805s | |
| Text Transform Uppercase | ✅ Pass | 2.413s | |
| Count from 1 to 5 | ✅ Pass | 2.075s | |
| Math Calculation | ✅ Pass | 1.364s | |
| Basic Echo Function | ✅ Pass | 1.724s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.629s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.851s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.895s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.622s | |
| Search Query Function | ✅ Pass | 1.869s | |
| Ask Advice Function | ✅ Pass | 2.412s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.302s | |
| Basic Context Memory Test | ✅ Pass | 2.121s | |
| Function Argument Memory Test | ✅ Pass | 2.266s | |
| Function Response Memory Test | ✅ Pass | 1.705s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.334s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.341s | |
| Penetration Testing Methodology | ✅ Pass | 11.752s | |
| Vulnerability Assessment Tools | ✅ Pass | 7.759s | |
| SQL Injection Attack Type | ✅ Pass | 2.492s | |
| Penetration Testing Framework | ✅ Pass | 12.965s | |
| Web Application Security Scanner | ✅ Pass | 8.206s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.925s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.602s
---
### assistant (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 3.792s | |
| Text Transform Uppercase | ✅ Pass | 1.857s | |
| Count from 1 to 5 | ✅ Pass | 3.005s | |
| Math Calculation | ✅ Pass | 1.790s | |
| Basic Echo Function | ✅ Pass | 1.660s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.064s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.648s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.884s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.934s | |
| Search Query Function | ✅ Pass | 1.837s | |
| Ask Advice Function | ✅ Pass | 2.343s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.786s | |
| Basic Context Memory Test | ✅ Pass | 2.777s | |
| Function Argument Memory Test | ✅ Pass | 2.055s | |
| Function Response Memory Test | ✅ Pass | 1.940s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.760s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.796s | |
| Penetration Testing Methodology | ✅ Pass | 10.671s | |
| Vulnerability Assessment Tools | ✅ Pass | 10.371s | |
| SQL Injection Attack Type | ✅ Pass | 2.529s | |
| Penetration Testing Framework | ✅ Pass | 11.063s | |
| Web Application Security Scanner | ✅ Pass | 6.830s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.769s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.660s
---
### generator (kimi-k2.6)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 3.818s | |
| Text Transform Uppercase | ✅ Pass | 4.989s | |
| Count from 1 to 5 | ✅ Pass | 2.068s | |
| Math Calculation | ✅ Pass | 1.499s | |
| Basic Echo Function | ✅ Pass | 1.577s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.753s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.802s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 6.221s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.911s | |
| Search Query Function | ✅ Pass | 1.693s | |
| Ask Advice Function | ✅ Pass | 2.063s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.638s | |
| Basic Context Memory Test | ✅ Pass | 2.781s | |
| Function Argument Memory Test | ✅ Pass | 1.929s | |
| Function Response Memory Test | ✅ Pass | 1.722s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.330s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 6.770s | |
| Penetration Testing Methodology | ✅ Pass | 5.593s | |
| Vulnerability Assessment Tools | ✅ Pass | 9.689s | |
| SQL Injection Attack Type | ✅ Pass | 3.238s | |
| Penetration Testing Framework | ✅ Pass | 6.540s | |
| Web Application Security Scanner | ✅ Pass | 4.246s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.942s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.471s
---
### refiner (kimi-k2.6)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 3.930s | |
| Text Transform Uppercase | ✅ Pass | 1.828s | |
| Count from 1 to 5 | ✅ Pass | 2.079s | |
| Math Calculation | ✅ Pass | 2.422s | |
| Basic Echo Function | ✅ Pass | 1.638s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.739s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.457s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.056s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.574s | |
| Search Query Function | ✅ Pass | 3.115s | |
| Ask Advice Function | ✅ Pass | 1.961s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.807s | |
| Basic Context Memory Test | ✅ Pass | 1.837s | |
| Function Argument Memory Test | ✅ Pass | 2.727s | |
| Function Response Memory Test | ✅ Pass | 2.079s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.531s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.579s | |
| Penetration Testing Methodology | ✅ Pass | 12.756s | |
| Vulnerability Assessment Tools | ✅ Pass | 10.127s | |
| SQL Injection Attack Type | ✅ Pass | 4.953s | |
| Penetration Testing Framework | ✅ Pass | 8.007s | |
| Web Application Security Scanner | ✅ Pass | 4.581s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.309s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.700s
---
### adviser (kimi-k2.6)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 4.341s | |
| Text Transform Uppercase | ✅ Pass | 1.662s | |
| Count from 1 to 5 | ✅ Pass | 2.051s | |
| Basic Echo Function | ✅ Pass | 1.700s | |
| Math Calculation | ✅ Pass | 4.090s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.907s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.154s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 6.487s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.825s | |
| Search Query Function | ✅ Pass | 1.781s | |
| Ask Advice Function | ✅ Pass | 2.140s | |
| Streaming Search Query Function Streaming | ✅ Pass | 4.836s | |
| Basic Context Memory Test | ✅ Pass | 2.497s | |
| Function Argument Memory Test | ✅ Pass | 1.844s | |
| Function Response Memory Test | ✅ Pass | 2.066s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 5.170s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 6.002s | |
| Penetration Testing Methodology | ✅ Pass | 7.523s | |
| Vulnerability Assessment Tools | ✅ Pass | 7.854s | |
| SQL Injection Attack Type | ✅ Pass | 2.703s | |
| Penetration Testing Framework | ✅ Pass | 10.072s | |
| Web Application Security Scanner | ✅ Pass | 5.844s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.817s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.886s
---
### reflector (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.557s | |
| Text Transform Uppercase | ✅ Pass | 0.691s | |
| Count from 1 to 5 | ✅ Pass | 0.821s | |
| Math Calculation | ✅ Pass | 0.788s | |
| Basic Echo Function | ✅ Pass | 1.170s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.658s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.917s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.215s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.349s | |
| Search Query Function | ✅ Pass | 1.016s | |
| Ask Advice Function | ✅ Pass | 1.553s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.116s | |
| Basic Context Memory Test | ✅ Pass | 1.271s | |
| Function Argument Memory Test | ✅ Pass | 0.825s | |
| Function Response Memory Test | ✅ Pass | 0.693s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.984s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.656s | |
| Penetration Testing Methodology | ✅ Pass | 3.017s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.796s | |
| SQL Injection Attack Type | ✅ Pass | 0.961s | |
| Penetration Testing Framework | ✅ Pass | 4.904s | |
| Web Application Security Scanner | ✅ Pass | 2.988s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.130s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.526s
---
### searcher (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.629s | |
| Text Transform Uppercase | ✅ Pass | 0.720s | |
| Count from 1 to 5 | ✅ Pass | 0.826s | |
| Math Calculation | ✅ Pass | 1.210s | |
| Basic Echo Function | ✅ Pass | 1.057s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.705s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.748s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.239s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.191s | |
| Search Query Function | ✅ Pass | 1.057s | |
| Ask Advice Function | ✅ Pass | 1.340s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.069s | |
| Basic Context Memory Test | ✅ Pass | 1.075s | |
| Function Argument Memory Test | ✅ Pass | 0.814s | |
| Function Response Memory Test | ✅ Pass | 0.680s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.213s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.775s | |
| Penetration Testing Methodology | ✅ Pass | 3.708s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.575s | |
| SQL Injection Attack Type | ✅ Pass | 0.897s | |
| Penetration Testing Framework | ✅ Pass | 2.486s | |
| Web Application Security Scanner | ✅ Pass | 1.508s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.267s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.383s
---
### enricher (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.721s | |
| Text Transform Uppercase | ✅ Pass | 0.799s | |
| Count from 1 to 5 | ✅ Pass | 0.791s | |
| Math Calculation | ✅ Pass | 0.712s | |
| Basic Echo Function | ✅ Pass | 1.013s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.621s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.699s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.240s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.281s | |
| Search Query Function | ✅ Pass | 1.034s | |
| Ask Advice Function | ✅ Pass | 1.261s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.968s | |
| Basic Context Memory Test | ✅ Pass | 0.962s | |
| Function Argument Memory Test | ✅ Pass | 0.781s | |
| Function Response Memory Test | ✅ Pass | 0.746s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.290s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.715s | |
| Penetration Testing Methodology | ✅ Pass | 3.325s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.531s | |
| SQL Injection Attack Type | ✅ Pass | 1.343s | |
| Penetration Testing Framework | ✅ Pass | 2.634s | |
| Web Application Security Scanner | ✅ Pass | 1.549s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.217s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.271s
---
### coder (kimi-k2.6)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.077s | |
| Text Transform Uppercase | ✅ Pass | 1.523s | |
| Count from 1 to 5 | ✅ Pass | 1.973s | |
| Math Calculation | ✅ Pass | 1.502s | |
| Basic Echo Function | ✅ Pass | 1.630s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.642s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.130s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.123s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.829s | |
| Search Query Function | ✅ Pass | 1.558s | |
| Ask Advice Function | ✅ Pass | 1.854s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.841s | |
| Basic Context Memory Test | ✅ Pass | 2.435s | |
| Function Argument Memory Test | ✅ Pass | 1.975s | |
| Function Response Memory Test | ✅ Pass | 2.004s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.342s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.180s | |
| Penetration Testing Methodology | ✅ Pass | 8.157s | |
| SQL Injection Attack Type | ✅ Pass | 3.469s | |
| Vulnerability Assessment Tools | ✅ Pass | 18.587s | |
| Penetration Testing Framework | ✅ Pass | 3.799s | |
| Web Application Security Scanner | ✅ Pass | 3.946s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.792s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.321s
---
### installer (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.393s | |
| Text Transform Uppercase | ✅ Pass | 2.517s | |
| Count from 1 to 5 | ✅ Pass | 1.722s | |
| Math Calculation | ✅ Pass | 1.790s | |
| Basic Echo Function | ✅ Pass | 1.364s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.430s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.340s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.897s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.757s | |
| Search Query Function | ✅ Pass | 1.746s | |
| Ask Advice Function | ✅ Pass | 2.257s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.568s | |
| Basic Context Memory Test | ✅ Pass | 2.824s | |
| Function Argument Memory Test | ✅ Pass | 2.347s | |
| Function Response Memory Test | ✅ Pass | 1.724s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.100s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.024s | |
| Penetration Testing Methodology | ✅ Pass | 11.036s | |
| Vulnerability Assessment Tools | ✅ Pass | 5.959s | |
| SQL Injection Attack Type | ✅ Pass | 5.710s | |
| Penetration Testing Framework | ✅ Pass | 10.269s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.241s | |
| Web Application Security Scanner | ✅ Pass | 6.747s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.381s
---
### pentester (kimi-k2.6)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 4.828s | |
| Text Transform Uppercase | ✅ Pass | 1.994s | |
| Count from 1 to 5 | ✅ Pass | 2.194s | |
| Math Calculation | ✅ Pass | 1.846s | |
| Basic Echo Function | ✅ Pass | 1.777s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.363s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.947s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.943s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.911s | |
| Search Query Function | ✅ Pass | 2.339s | |
| Ask Advice Function | ✅ Pass | 2.164s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.754s | |
| Basic Context Memory Test | ✅ Pass | 2.243s | |
| Function Argument Memory Test | ✅ Pass | 1.688s | |
| Function Response Memory Test | ✅ Pass | 2.078s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.638s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.884s | |
| Penetration Testing Methodology | ✅ Pass | 7.438s | |
| SQL Injection Attack Type | ✅ Pass | 2.465s | |
| Vulnerability Assessment Tools | ✅ Pass | 16.920s | |
| Penetration Testing Framework | ✅ Pass | 6.907s | |
| Web Application Security Scanner | ✅ Pass | 4.049s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.557s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.432s
---
examples/tests/moonshot-report.md
+537
View File
@@ -0,0 +1,537 @@
# LLM Agent Testing Report
Generated: Thu, 29 Jan 2026 17:23:17 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
| simple | kimi-k2-0905-preview | true | 23/23 (100.00%) | 2.527s |
| simple_json | kimi-k2-0905-preview | false | 5/5 (100.00%) | 3.434s |
| primary_agent | kimi-k2.5 | true | 23/23 (100.00%) | 4.712s |
| assistant | kimi-k2.5 | true | 23/23 (100.00%) | 4.800s |
| generator | kimi-k2.5 | true | 23/23 (100.00%) | 4.819s |
| refiner | kimi-k2.5 | true | 23/23 (100.00%) | 5.105s |
| adviser | kimi-k2.5 | true | 23/23 (100.00%) | 4.209s |
| reflector | kimi-k2-0905-preview | true | 23/23 (100.00%) | 2.616s |
| searcher | kimi-k2-0905-preview | true | 23/23 (100.00%) | 2.564s |
| enricher | kimi-k2-0905-preview | true | 23/23 (100.00%) | 2.497s |
| coder | kimi-k2.5 | true | 23/23 (100.00%) | 5.042s |
| installer | kimi-k2-turbo-preview | true | 23/23 (100.00%) | 1.057s |
| pentester | kimi-k2-turbo-preview | true | 23/23 (100.00%) | 1.050s |
**Total**: 281/281 (100.00%) successful tests
**Overall average latency**: 3.417s
## Detailed Results
### simple (kimi-k2-0905-preview)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.969s | |
| Text Transform Uppercase | ✅ Pass | 1.409s | |
| Count from 1 to 5 | ✅ Pass | 2.185s | |
| Math Calculation | ✅ Pass | 1.264s | |
| Basic Echo Function | ✅ Pass | 3.142s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.245s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.640s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.451s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 4.030s | |
| Search Query Function | ✅ Pass | 3.010s | |
| Ask Advice Function | ✅ Pass | 4.312s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.796s | |
| Basic Context Memory Test | ✅ Pass | 2.255s | |
| Function Argument Memory Test | ✅ Pass | 1.492s | |
| Function Response Memory Test | ✅ Pass | 1.159s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 8.136s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.327s | |
| Penetration Testing Methodology | ✅ Pass | 1.652s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.114s | |
| SQL Injection Attack Type | ✅ Pass | 2.276s | |
| Penetration Testing Framework | ✅ Pass | 1.798s | |
| Web Application Security Scanner | ✅ Pass | 1.280s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.168s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.527s
---
### simple_json (kimi-k2-0905-preview)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Person Information JSON | ✅ Pass | 2.695s | |
| Project Information JSON | ✅ Pass | 2.609s | |
| Streaming Person Information JSON Streaming | ✅ Pass | 2.682s | |
| User Profile JSON | ✅ Pass | 3.029s | |
| Vulnerability Report Memory Test | ✅ Pass | 6.151s | |
**Summary**: 5/5 (100.00%) successful tests
**Average latency**: 3.434s
---
### primary_agent (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 3.156s | |
| Text Transform Uppercase | ✅ Pass | 2.189s | |
| Count from 1 to 5 | ✅ Pass | 4.424s | |
| Math Calculation | ✅ Pass | 1.450s | |
| Basic Echo Function | ✅ Pass | 1.899s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.228s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.028s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.610s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.485s | |
| Search Query Function | ✅ Pass | 1.712s | |
| Ask Advice Function | ✅ Pass | 2.803s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.520s | |
| Basic Context Memory Test | ✅ Pass | 2.999s | |
| Function Argument Memory Test | ✅ Pass | 2.119s | |
| Function Response Memory Test | ✅ Pass | 2.689s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.852s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.575s | |
| Penetration Testing Methodology | ✅ Pass | 15.432s | |
| Vulnerability Assessment Tools | ✅ Pass | 15.437s | |
| SQL Injection Attack Type | ✅ Pass | 6.549s | |
| Penetration Testing Framework | ✅ Pass | 9.912s | |
| Web Application Security Scanner | ✅ Pass | 13.061s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.227s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 4.712s
---
### assistant (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.533s | |
| Text Transform Uppercase | ✅ Pass | 2.086s | |
| Count from 1 to 5 | ✅ Pass | 4.770s | |
| Math Calculation | ✅ Pass | 2.369s | |
| Basic Echo Function | ✅ Pass | 2.024s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.364s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.988s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.295s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.751s | |
| Search Query Function | ✅ Pass | 1.407s | |
| Ask Advice Function | ✅ Pass | 2.156s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.947s | |
| Basic Context Memory Test | ✅ Pass | 3.082s | |
| Function Argument Memory Test | ✅ Pass | 2.787s | |
| Function Response Memory Test | ✅ Pass | 2.670s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.721s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.700s | |
| Penetration Testing Methodology | ✅ Pass | 15.716s | |
| Vulnerability Assessment Tools | ✅ Pass | 17.163s | |
| SQL Injection Attack Type | ✅ Pass | 4.081s | |
| Penetration Testing Framework | ✅ Pass | 10.991s | |
| Web Application Security Scanner | ✅ Pass | 14.581s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.207s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 4.800s
---
### generator (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 3.573s | |
| Text Transform Uppercase | ✅ Pass | 3.412s | |
| Count from 1 to 5 | ✅ Pass | 4.818s | |
| Math Calculation | ✅ Pass | 1.616s | |
| Basic Echo Function | ✅ Pass | 1.914s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.830s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.596s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.960s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.316s | |
| Search Query Function | ✅ Pass | 1.623s | |
| Ask Advice Function | ✅ Pass | 1.644s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.135s | |
| Basic Context Memory Test | ✅ Pass | 3.843s | |
| Function Argument Memory Test | ✅ Pass | 1.947s | |
| Function Response Memory Test | ✅ Pass | 4.476s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.291s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.153s | |
| Penetration Testing Methodology | ✅ Pass | 11.999s | |
| SQL Injection Attack Type | ✅ Pass | 4.011s | |
| Vulnerability Assessment Tools | ✅ Pass | 17.057s | |
| Penetration Testing Framework | ✅ Pass | 12.720s | |
| Web Application Security Scanner | ✅ Pass | 11.890s | |
| Penetration Testing Tool Selection | ✅ Pass | 5.008s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 4.819s
---
### refiner (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.575s | |
| Text Transform Uppercase | ✅ Pass | 1.467s | |
| Count from 1 to 5 | ✅ Pass | 2.710s | |
| Math Calculation | ✅ Pass | 2.063s | |
| Basic Echo Function | ✅ Pass | 2.177s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.719s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.366s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.311s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.040s | |
| Search Query Function | ✅ Pass | 1.610s | |
| Ask Advice Function | ✅ Pass | 2.109s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.672s | |
| Basic Context Memory Test | ✅ Pass | 3.430s | |
| Function Argument Memory Test | ✅ Pass | 2.247s | |
| Function Response Memory Test | ✅ Pass | 3.155s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 6.300s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.164s | |
| Penetration Testing Methodology | ✅ Pass | 11.814s | |
| SQL Injection Attack Type | ✅ Pass | 4.782s | |
| Vulnerability Assessment Tools | ✅ Pass | 19.726s | |
| Penetration Testing Framework | ✅ Pass | 17.103s | |
| Web Application Security Scanner | ✅ Pass | 14.709s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.152s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 5.105s
---
### adviser (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 3.087s | |
| Text Transform Uppercase | ✅ Pass | 2.282s | |
| Count from 1 to 5 | ✅ Pass | 1.617s | |
| Math Calculation | ✅ Pass | 2.105s | |
| Basic Echo Function | ✅ Pass | 2.211s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.083s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.229s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.895s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.561s | |
| Search Query Function | ✅ Pass | 1.690s | |
| Ask Advice Function | ✅ Pass | 2.217s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.371s | |
| Basic Context Memory Test | ✅ Pass | 2.929s | |
| Function Argument Memory Test | ✅ Pass | 1.864s | |
| Function Response Memory Test | ✅ Pass | 3.357s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.003s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.202s | |
| Penetration Testing Methodology | ✅ Pass | 13.754s | |
| SQL Injection Attack Type | ✅ Pass | 2.961s | |
| Vulnerability Assessment Tools | ✅ Pass | 16.462s | |
| Penetration Testing Framework | ✅ Pass | 6.220s | |
| Web Application Security Scanner | ✅ Pass | 12.801s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.884s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 4.209s
---
### reflector (kimi-k2-0905-preview)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.968s | |
| Text Transform Uppercase | ✅ Pass | 1.353s | |
| Count from 1 to 5 | ✅ Pass | 2.131s | |
| Math Calculation | ✅ Pass | 1.275s | |
| Basic Echo Function | ✅ Pass | 2.887s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.255s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.752s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.544s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 4.087s | |
| Search Query Function | ✅ Pass | 3.010s | |
| Ask Advice Function | ✅ Pass | 4.280s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.324s | |
| Basic Context Memory Test | ✅ Pass | 2.626s | |
| Function Argument Memory Test | ✅ Pass | 1.436s | |
| Function Response Memory Test | ✅ Pass | 1.267s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 7.401s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.327s | |
| Penetration Testing Methodology | ✅ Pass | 2.447s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.007s | |
| SQL Injection Attack Type | ✅ Pass | 2.523s | |
| Penetration Testing Framework | ✅ Pass | 1.611s | |
| Web Application Security Scanner | ✅ Pass | 2.019s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.628s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.616s
---
### searcher (kimi-k2-0905-preview)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.976s | |
| Text Transform Uppercase | ✅ Pass | 1.436s | |
| Count from 1 to 5 | ✅ Pass | 2.454s | |
| Math Calculation | ✅ Pass | 1.724s | |
| Basic Echo Function | ✅ Pass | 2.867s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.358s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.621s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.471s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.997s | |
| Search Query Function | ✅ Pass | 2.949s | |
| Ask Advice Function | ✅ Pass | 4.267s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.800s | |
| Basic Context Memory Test | ✅ Pass | 2.723s | |
| Function Argument Memory Test | ✅ Pass | 1.476s | |
| Function Response Memory Test | ✅ Pass | 1.480s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 7.167s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.430s | |
| Penetration Testing Methodology | ✅ Pass | 2.091s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.125s | |
| SQL Injection Attack Type | ✅ Pass | 2.275s | |
| Penetration Testing Framework | ✅ Pass | 1.584s | |
| Web Application Security Scanner | ✅ Pass | 1.475s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.217s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.564s
---
### enricher (kimi-k2-0905-preview)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.380s | |
| Text Transform Uppercase | ✅ Pass | 1.480s | |
| Count from 1 to 5 | ✅ Pass | 2.090s | |
| Math Calculation | ✅ Pass | 1.219s | |
| Basic Echo Function | ✅ Pass | 2.964s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.157s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.274s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.458s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.441s | |
| Search Query Function | ✅ Pass | 2.873s | |
| Ask Advice Function | ✅ Pass | 4.240s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.152s | |
| Basic Context Memory Test | ✅ Pass | 2.943s | |
| Function Argument Memory Test | ✅ Pass | 1.419s | |
| Function Response Memory Test | ✅ Pass | 1.301s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.490s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 7.617s | |
| Penetration Testing Methodology | ✅ Pass | 2.430s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.518s | |
| SQL Injection Attack Type | ✅ Pass | 1.380s | |
| Penetration Testing Framework | ✅ Pass | 1.676s | |
| Web Application Security Scanner | ✅ Pass | 1.834s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.091s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.497s
---
### coder (kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.846s | |
| Text Transform Uppercase | ✅ Pass | 1.852s | |
| Count from 1 to 5 | ✅ Pass | 5.834s | |
| Math Calculation | ✅ Pass | 2.142s | |
| Basic Echo Function | ✅ Pass | 2.158s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.971s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.713s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.089s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.249s | |
| Search Query Function | ✅ Pass | 2.192s | |
| Ask Advice Function | ✅ Pass | 3.064s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.303s | |
| Basic Context Memory Test | ✅ Pass | 3.178s | |
| Function Argument Memory Test | ✅ Pass | 1.985s | |
| Function Response Memory Test | ✅ Pass | 3.064s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.572s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.607s | |
| Penetration Testing Methodology | ✅ Pass | 17.249s | |
| SQL Injection Attack Type | ✅ Pass | 2.996s | |
| Vulnerability Assessment Tools | ✅ Pass | 17.738s | |
| Penetration Testing Framework | ✅ Pass | 14.365s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.291s | |
| Web Application Security Scanner | ✅ Pass | 14.502s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 5.042s
---
### installer (kimi-k2-turbo-preview)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.799s | |
| Text Transform Uppercase | ✅ Pass | 0.831s | |
| Count from 1 to 5 | ✅ Pass | 0.949s | |
| Math Calculation | ✅ Pass | 0.844s | |
| Basic Echo Function | ✅ Pass | 1.011s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.789s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.455s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.465s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.008s | |
| Search Query Function | ✅ Pass | 1.117s | |
| Ask Advice Function | ✅ Pass | 1.262s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.063s | |
| Basic Context Memory Test | ✅ Pass | 0.915s | |
| Function Argument Memory Test | ✅ Pass | 0.787s | |
| Function Response Memory Test | ✅ Pass | 0.798s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.997s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.827s | |
| Penetration Testing Methodology | ✅ Pass | 0.989s | |
| Vulnerability Assessment Tools | ✅ Pass | 0.864s | |
| SQL Injection Attack Type | ✅ Pass | 0.902s | |
| Penetration Testing Framework | ✅ Pass | 1.020s | |
| Web Application Security Scanner | ✅ Pass | 1.263s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.345s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.057s
---
### pentester (kimi-k2-turbo-preview)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.856s | |
| Text Transform Uppercase | ✅ Pass | 0.816s | |
| Count from 1 to 5 | ✅ Pass | 0.893s | |
| Math Calculation | ✅ Pass | 0.829s | |
| Basic Echo Function | ✅ Pass | 0.923s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.820s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.375s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.587s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.088s | |
| Search Query Function | ✅ Pass | 0.976s | |
| Ask Advice Function | ✅ Pass | 1.218s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.231s | |
| Basic Context Memory Test | ✅ Pass | 0.975s | |
| Function Argument Memory Test | ✅ Pass | 0.927s | |
| Function Response Memory Test | ✅ Pass | 0.830s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.959s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.795s | |
| Penetration Testing Methodology | ✅ Pass | 1.035s | |
| Vulnerability Assessment Tools | ✅ Pass | 0.987s | |
| SQL Injection Attack Type | ✅ Pass | 0.972s | |
| Penetration Testing Framework | ✅ Pass | 1.222s | |
| Web Application Security Scanner | ✅ Pass | 0.834s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.989s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.050s
---
examples/tests/novita-report.md
+537
View File
@@ -0,0 +1,537 @@
# LLM Agent Testing Report
Generated: Mon, 02 Mar 2026 15:08:50 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
| simple | deepseek/deepseek-v3.2 | false | 22/23 (95.65%) | 2.458s |
| simple_json | deepseek/deepseek-v3.2 | false | 5/5 (100.00%) | 2.148s |
| primary_agent | moonshotai/kimi-k2.5 | true | 22/23 (95.65%) | 2.658s |
| assistant | moonshotai/kimi-k2.5 | true | 22/23 (95.65%) | 3.286s |
| generator | moonshotai/kimi-k2.5 | true | 22/23 (95.65%) | 2.686s |
| refiner | moonshotai/kimi-k2.5 | true | 22/23 (95.65%) | 3.071s |
| adviser | zai-org/glm-5 | true | 23/23 (100.00%) | 9.204s |
| reflector | qwen/qwen3.5-35b-a3b | true | 23/23 (100.00%) | 3.375s |
| searcher | qwen/qwen3.5-35b-a3b | true | 22/23 (95.65%) | 3.648s |
| enricher | qwen/qwen3.5-35b-a3b | true | 23/23 (100.00%) | 3.332s |
| coder | moonshotai/kimi-k2.5 | true | 23/23 (100.00%) | 3.067s |
| installer | moonshotai/kimi-k2-instruct | true | 20/23 (86.96%) | 1.480s |
| pentester | moonshotai/kimi-k2.5 | true | 23/23 (100.00%) | 2.818s |
**Total**: 272/281 (96.80%) successful tests
**Overall average latency**: 3.401s
## Detailed Results
### simple (deepseek/deepseek-v3.2)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.288s | |
| Text Transform Uppercase | ✅ Pass | 1.461s | |
| Count from 1 to 5 | ✅ Pass | 1.353s | |
| Math Calculation | ✅ Pass | 1.379s | |
| Basic Echo Function | ✅ Pass | 2.869s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.182s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.654s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.997s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 4.244s | |
| Search Query Function | ✅ Pass | 2.317s | |
| Ask Advice Function | ✅ Pass | 3.651s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.320s | |
| Basic Context Memory Test | ✅ Pass | 2.409s | |
| Function Argument Memory Test | ✅ Pass | 1.077s | |
| Function Response Memory Test | ✅ Pass | 1.354s | |
| Penetration Testing Memory with Tool Call | ❌ Fail | 4.833s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.674s | |
| Penetration Testing Methodology | ✅ Pass | 2.582s | |
| Vulnerability Assessment Tools | ✅ Pass | 5.695s | |
| SQL Injection Attack Type | ✅ Pass | 1.572s | |
| Penetration Testing Framework | ✅ Pass | 2.639s | |
| Web Application Security Scanner | ✅ Pass | 1.808s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.174s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 2.458s
---
### simple_json (deepseek/deepseek-v3.2)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Vulnerability Report Memory Test | ✅ Pass | 2.759s | |
| Person Information JSON | ✅ Pass | 1.929s | |
| Project Information JSON | ✅ Pass | 1.792s | |
| User Profile JSON | ✅ Pass | 2.154s | |
| Streaming Person Information JSON Streaming | ✅ Pass | 2.102s | |
**Summary**: 5/5 (100.00%) successful tests
**Average latency**: 2.148s
---
### primary_agent (moonshotai/kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 4.139s | |
| Text Transform Uppercase | ✅ Pass | 0.784s | |
| Count from 1 to 5 | ✅ Pass | 0.869s | |
| Math Calculation | ✅ Pass | 1.179s | |
| Basic Echo Function | ✅ Pass | 1.499s | |
| Simple Math Streaming | ❌ Fail | 0.242s | API returned unexpected status code: 429: |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.832s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.742s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.169s | |
| Search Query Function | ✅ Pass | 1.403s | |
| Ask Advice Function | ✅ Pass | 1.781s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.215s | |
| Basic Context Memory Test | ✅ Pass | 2.855s | |
| Function Argument Memory Test | ✅ Pass | 1.847s | |
| Function Response Memory Test | ✅ Pass | 2.417s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.549s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.638s | |
| Penetration Testing Methodology | ✅ Pass | 3.428s | |
| Vulnerability Assessment Tools | ✅ Pass | 3.630s | |
| SQL Injection Attack Type | ✅ Pass | 1.428s | |
| Penetration Testing Framework | ✅ Pass | 13.651s | |
| Web Application Security Scanner | ✅ Pass | 2.220s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.612s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 2.658s
---
### assistant (moonshotai/kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.280s | |
| Text Transform Uppercase | ✅ Pass | 0.778s | |
| Count from 1 to 5 | ✅ Pass | 1.746s | |
| Math Calculation | ✅ Pass | 1.138s | |
| Basic Echo Function | ✅ Pass | 1.738s | |
| Simple Math Streaming | ❌ Fail | 0.282s | API returned unexpected status code: 429: |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.009s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.181s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.819s | |
| Search Query Function | ✅ Pass | 2.981s | |
| Ask Advice Function | ✅ Pass | 1.336s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.939s | |
| Basic Context Memory Test | ✅ Pass | 3.554s | |
| Function Argument Memory Test | ✅ Pass | 6.030s | |
| Function Response Memory Test | ✅ Pass | 2.626s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.105s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.154s | |
| Penetration Testing Methodology | ✅ Pass | 4.826s | |
| Vulnerability Assessment Tools | ✅ Pass | 11.738s | |
| SQL Injection Attack Type | ✅ Pass | 3.703s | |
| Penetration Testing Framework | ✅ Pass | 13.123s | |
| Web Application Security Scanner | ✅ Pass | 1.985s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.496s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 3.286s
---
### generator (moonshotai/kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.148s | |
| Text Transform Uppercase | ✅ Pass | 0.817s | |
| Count from 1 to 5 | ✅ Pass | 0.931s | |
| Math Calculation | ✅ Pass | 1.174s | |
| Basic Echo Function | ✅ Pass | 1.500s | |
| Simple Math Streaming | ❌ Fail | 0.243s | API returned unexpected status code: 429: |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.691s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.370s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.270s | |
| Search Query Function | ✅ Pass | 1.396s | |
| Ask Advice Function | ✅ Pass | 1.799s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.598s | |
| Basic Context Memory Test | ✅ Pass | 1.747s | |
| Function Argument Memory Test | ✅ Pass | 1.747s | |
| Function Response Memory Test | ✅ Pass | 0.828s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.171s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.225s | |
| Penetration Testing Methodology | ✅ Pass | 3.684s | |
| Vulnerability Assessment Tools | ✅ Pass | 10.069s | |
| SQL Injection Attack Type | ✅ Pass | 1.440s | |
| Penetration Testing Framework | ✅ Pass | 17.198s | |
| Web Application Security Scanner | ✅ Pass | 1.930s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.790s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 2.686s
---
### refiner (moonshotai/kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.052s | |
| Text Transform Uppercase | ✅ Pass | 1.245s | |
| Count from 1 to 5 | ✅ Pass | 2.057s | |
| Math Calculation | ✅ Pass | 1.189s | |
| Basic Echo Function | ✅ Pass | 1.490s | |
| Simple Math Streaming | ❌ Fail | 0.267s | API returned unexpected status code: 429: |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.289s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.278s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.328s | |
| Search Query Function | ✅ Pass | 1.366s | |
| Ask Advice Function | ✅ Pass | 1.970s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.560s | |
| Basic Context Memory Test | ✅ Pass | 2.548s | |
| Function Argument Memory Test | ✅ Pass | 5.412s | |
| Function Response Memory Test | ✅ Pass | 1.155s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 5.038s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.165s | |
| Penetration Testing Methodology | ✅ Pass | 3.756s | |
| Vulnerability Assessment Tools | ✅ Pass | 13.463s | |
| SQL Injection Attack Type | ✅ Pass | 5.187s | |
| Penetration Testing Framework | ✅ Pass | 6.667s | |
| Web Application Security Scanner | ✅ Pass | 2.588s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.554s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 3.071s
---
### adviser (zai-org/glm-5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 3.329s | |
| Text Transform Uppercase | ✅ Pass | 1.953s | |
| Count from 1 to 5 | ✅ Pass | 6.180s | |
| Math Calculation | ✅ Pass | 6.157s | |
| Basic Echo Function | ✅ Pass | 4.979s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.658s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.993s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.868s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 6.030s | |
| Search Query Function | ✅ Pass | 4.583s | |
| Ask Advice Function | ✅ Pass | 3.598s | |
| Streaming Search Query Function Streaming | ✅ Pass | 5.290s | |
| Basic Context Memory Test | ✅ Pass | 2.464s | |
| Function Argument Memory Test | ✅ Pass | 4.195s | |
| Function Response Memory Test | ✅ Pass | 2.751s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 8.420s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.993s | |
| Penetration Testing Methodology | ✅ Pass | 17.127s | |
| Vulnerability Assessment Tools | ✅ Pass | 56.258s | |
| SQL Injection Attack Type | ✅ Pass | 7.521s | |
| Penetration Testing Framework | ✅ Pass | 27.639s | |
| Web Application Security Scanner | ✅ Pass | 22.824s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.873s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 9.204s
---
### reflector (qwen/qwen3.5-35b-a3b)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.177s | |
| Text Transform Uppercase | ✅ Pass | 1.761s | |
| Count from 1 to 5 | ✅ Pass | 2.230s | |
| Math Calculation | ✅ Pass | 2.020s | |
| Basic Echo Function | ✅ Pass | 1.069s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.739s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.650s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.308s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.554s | |
| Search Query Function | ✅ Pass | 1.355s | |
| Ask Advice Function | ✅ Pass | 1.450s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.319s | |
| Basic Context Memory Test | ✅ Pass | 3.054s | |
| Function Argument Memory Test | ✅ Pass | 1.053s | |
| Function Response Memory Test | ✅ Pass | 1.318s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.568s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.063s | |
| Penetration Testing Methodology | ✅ Pass | 5.066s | |
| Vulnerability Assessment Tools | ✅ Pass | 21.070s | |
| SQL Injection Attack Type | ✅ Pass | 5.581s | |
| Penetration Testing Framework | ✅ Pass | 12.000s | |
| Web Application Security Scanner | ✅ Pass | 3.616s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.594s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.375s
---
### searcher (qwen/qwen3.5-35b-a3b)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.605s | |
| Text Transform Uppercase | ✅ Pass | 2.089s | |
| Count from 1 to 5 | ✅ Pass | 8.093s | |
| Math Calculation | ✅ Pass | 1.417s | |
| Basic Echo Function | ✅ Pass | 1.186s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.070s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.164s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.441s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.266s | |
| Search Query Function | ✅ Pass | 1.045s | |
| Ask Advice Function | ✅ Pass | 1.284s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.722s | |
| Basic Context Memory Test | ✅ Pass | 3.651s | |
| Function Argument Memory Test | ✅ Pass | 6.143s | |
| Function Response Memory Test | ✅ Pass | 5.972s | |
| Penetration Testing Memory with Tool Call | ❌ Fail | 2.381s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.153s | |
| Penetration Testing Methodology | ✅ Pass | 6.202s | |
| Vulnerability Assessment Tools | ✅ Pass | 18.668s | |
| SQL Injection Attack Type | ✅ Pass | 2.414s | |
| Penetration Testing Framework | ✅ Pass | 4.669s | |
| Web Application Security Scanner | ✅ Pass | 3.988s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.274s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 3.648s
---
### enricher (qwen/qwen3.5-35b-a3b)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.084s | |
| Text Transform Uppercase | ✅ Pass | 2.561s | |
| Count from 1 to 5 | ✅ Pass | 1.884s | |
| Math Calculation | ✅ Pass | 1.308s | |
| Basic Echo Function | ✅ Pass | 1.227s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.092s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 15.459s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.655s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.266s | |
| Search Query Function | ✅ Pass | 1.511s | |
| Ask Advice Function | ✅ Pass | 1.737s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.579s | |
| Basic Context Memory Test | ✅ Pass | 2.538s | |
| Function Argument Memory Test | ✅ Pass | 1.031s | |
| Function Response Memory Test | ✅ Pass | 0.992s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.167s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.353s | |
| Penetration Testing Methodology | ✅ Pass | 7.191s | |
| Vulnerability Assessment Tools | ✅ Pass | 13.350s | |
| SQL Injection Attack Type | ✅ Pass | 3.672s | |
| Penetration Testing Framework | ✅ Pass | 3.777s | |
| Web Application Security Scanner | ✅ Pass | 4.323s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.877s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.332s
---
### coder (moonshotai/kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.418s | |
| Text Transform Uppercase | ✅ Pass | 0.783s | |
| Count from 1 to 5 | ✅ Pass | 1.644s | |
| Math Calculation | ✅ Pass | 1.177s | |
| Basic Echo Function | ✅ Pass | 1.741s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.137s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.736s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.831s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.516s | |
| Search Query Function | ✅ Pass | 1.649s | |
| Ask Advice Function | ✅ Pass | 1.316s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.804s | |
| Basic Context Memory Test | ✅ Pass | 2.790s | |
| Function Argument Memory Test | ✅ Pass | 1.301s | |
| Function Response Memory Test | ✅ Pass | 3.491s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.182s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.580s | |
| Penetration Testing Methodology | ✅ Pass | 4.092s | |
| Vulnerability Assessment Tools | ✅ Pass | 5.292s | |
| SQL Injection Attack Type | ✅ Pass | 1.417s | |
| Penetration Testing Framework | ✅ Pass | 14.947s | |
| Web Application Security Scanner | ✅ Pass | 4.822s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.867s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.067s
---
### installer (moonshotai/kimi-k2-instruct)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.799s | |
| Text Transform Uppercase | ✅ Pass | 1.102s | |
| Count from 1 to 5 | ✅ Pass | 1.179s | |
| Math Calculation | ✅ Pass | 1.124s | |
| Basic Echo Function | ✅ Pass | 1.514s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.115s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.312s | |
| Streaming Basic Echo Function Streaming | ❌ Fail | 1.588s | expected function 'echo' not found in tool calls: invalid JSON in tool call echo: unexpected end of JSON input |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.410s | |
| Search Query Function | ✅ Pass | 1.787s | |
| Ask Advice Function | ✅ Pass | 2.906s | |
| Streaming Search Query Function Streaming | ❌ Fail | 1.552s | expected function 'search' not found in tool calls: invalid JSON in tool call search: unexpected end of JSON input |
| Basic Context Memory Test | ✅ Pass | 2.793s | |
| Function Argument Memory Test | ✅ Pass | 1.024s | |
| Function Response Memory Test | ✅ Pass | 0.989s | |
| Penetration Testing Memory with Tool Call | ❌ Fail | 2.065s | no tool calls found, expected at least 1 |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.145s | |
| Penetration Testing Methodology | ✅ Pass | 1.163s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.252s | |
| SQL Injection Attack Type | ✅ Pass | 1.508s | |
| Penetration Testing Framework | ✅ Pass | 1.003s | |
| Web Application Security Scanner | ✅ Pass | 1.105s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.601s | |
**Summary**: 20/23 (86.96%) successful tests
**Average latency**: 1.480s
---
### pentester (moonshotai/kimi-k2.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.792s | |
| Text Transform Uppercase | ✅ Pass | 0.903s | |
| Count from 1 to 5 | ✅ Pass | 2.272s | |
| Math Calculation | ✅ Pass | 3.338s | |
| Basic Echo Function | ✅ Pass | 1.732s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.144s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.128s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.879s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.055s | |
| Search Query Function | ✅ Pass | 2.093s | |
| Ask Advice Function | ✅ Pass | 1.955s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.601s | |
| Basic Context Memory Test | ✅ Pass | 2.653s | |
| Function Argument Memory Test | ✅ Pass | 1.192s | |
| Function Response Memory Test | ✅ Pass | 0.701s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.381s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.075s | |
| Penetration Testing Methodology | ✅ Pass | 3.893s | |
| Vulnerability Assessment Tools | ✅ Pass | 3.376s | |
| SQL Injection Attack Type | ✅ Pass | 1.423s | |
| Penetration Testing Framework | ✅ Pass | 16.347s | |
| Web Application Security Scanner | ✅ Pass | 2.108s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.766s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.818s
---
examples/tests/ollama-cloud-report.md
+537
View File
@@ -0,0 +1,537 @@
# LLM Agent Testing Report
Generated: Mon, 23 Mar 2026 16:29:35 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
| simple | nemotron-3-super:cloud | false | 22/23 (95.65%) | 22.627s |
| simple_json | nemotron-3-super:cloud | false | 5/5 (100.00%) | 27.380s |
| primary_agent | qwen3-coder-next:cloud | false | 23/23 (100.00%) | 4.101s |
| assistant | nemotron-3-super:cloud | false | 23/23 (100.00%) | 22.773s |
| generator | qwen3-coder-next:cloud | false | 23/23 (100.00%) | 2.348s |
| refiner | glm-5:cloud | false | 23/23 (100.00%) | 7.964s |
| adviser | minimax-m2.7:cloud | false | 23/23 (100.00%) | 3.990s |
| reflector | glm-5:cloud | false | 23/23 (100.00%) | 9.201s |
| searcher | qwen3.5:397b-cloud | false | 20/23 (86.96%) | 46.174s |
| enricher | minimax-m2.7:cloud | false | 23/23 (100.00%) | 6.153s |
| coder | qwen3-coder-next:cloud | false | 23/23 (100.00%) | 3.223s |
| installer | devstral-2:123b-cloud | false | 23/23 (100.00%) | 8.049s |
| pentester | qwen3-coder-next:cloud | false | 23/23 (100.00%) | 2.227s |
**Total**: 277/281 (98.58%) successful tests
**Overall average latency**: 11.851s
## Detailed Results
### simple (nemotron-3-super:cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.444s | |
| Text Transform Uppercase | ✅ Pass | 10.749s | |
| Count from 1 to 5 | ✅ Pass | 19.076s | |
| Math Calculation | ✅ Pass | 6.549s | |
| Basic Echo Function | ✅ Pass | 16.980s | |
| Streaming Simple Math Streaming | ✅ Pass | 18.192s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 8.274s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 10.723s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 20.851s | |
| Search Query Function | ✅ Pass | 16.304s | |
| Ask Advice Function | ✅ Pass | 33.178s | |
| Streaming Search Query Function Streaming | ✅ Pass | 14.129s | |
| Basic Context Memory Test | ✅ Pass | 25.373s | |
| Function Argument Memory Test | ✅ Pass | 34.212s | |
| Function Response Memory Test | ✅ Pass | 22.402s | |
| Penetration Testing Memory with Tool Call | ❌ Fail | 39.672s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
| Cybersecurity Workflow Memory Test | ✅ Pass | 14.158s | |
| Penetration Testing Methodology | ✅ Pass | 57.038s | |
| Vulnerability Assessment Tools | ✅ Pass | 40.968s | |
| SQL Injection Attack Type | ✅ Pass | 9.713s | |
| Penetration Testing Framework | ✅ Pass | 65.247s | |
| Web Application Security Scanner | ✅ Pass | 15.760s | |
| Penetration Testing Tool Selection | ✅ Pass | 18.418s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 22.627s
---
### simple_json (nemotron-3-super:cloud)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Vulnerability Report Memory Test | ✅ Pass | 30.553s | |
| Person Information JSON | ✅ Pass | 23.124s | |
| Project Information JSON | ✅ Pass | 25.124s | |
| User Profile JSON | ✅ Pass | 31.109s | |
| Streaming Person Information JSON Streaming | ✅ Pass | 26.988s | |
**Summary**: 5/5 (100.00%) successful tests
**Average latency**: 27.380s
---
### primary_agent (qwen3-coder-next:cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.931s | |
| Text Transform Uppercase | ✅ Pass | 0.763s | |
| Count from 1 to 5 | ✅ Pass | 3.641s | |
| Math Calculation | ✅ Pass | 0.932s | |
| Basic Echo Function | ✅ Pass | 0.869s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.700s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.713s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.812s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.959s | |
| Search Query Function | ✅ Pass | 0.933s | |
| Ask Advice Function | ✅ Pass | 33.999s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.998s | |
| Basic Context Memory Test | ✅ Pass | 8.687s | |
| Function Argument Memory Test | ✅ Pass | 6.186s | |
| Function Response Memory Test | ✅ Pass | 2.615s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.527s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.732s | |
| Penetration Testing Methodology | ✅ Pass | 6.404s | |
| Vulnerability Assessment Tools | ✅ Pass | 3.203s | |
| SQL Injection Attack Type | ✅ Pass | 2.766s | |
| Penetration Testing Framework | ✅ Pass | 8.059s | |
| Web Application Security Scanner | ✅ Pass | 1.546s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.339s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 4.101s
---
### assistant (nemotron-3-super:cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 13.460s | |
| Text Transform Uppercase | ✅ Pass | 8.769s | |
| Count from 1 to 5 | ✅ Pass | 6.944s | |
| Math Calculation | ✅ Pass | 31.133s | |
| Basic Echo Function | ✅ Pass | 10.710s | |
| Streaming Simple Math Streaming | ✅ Pass | 16.396s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 7.617s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 13.783s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 20.992s | |
| Search Query Function | ✅ Pass | 13.170s | |
| Ask Advice Function | ✅ Pass | 17.042s | |
| Streaming Search Query Function Streaming | ✅ Pass | 18.533s | |
| Basic Context Memory Test | ✅ Pass | 23.059s | |
| Function Argument Memory Test | ✅ Pass | 25.397s | |
| Function Response Memory Test | ✅ Pass | 24.149s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 39.970s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 8.683s | |
| Penetration Testing Methodology | ✅ Pass | 63.505s | |
| Vulnerability Assessment Tools | ✅ Pass | 34.659s | |
| SQL Injection Attack Type | ✅ Pass | 26.478s | |
| Penetration Testing Framework | ✅ Pass | 58.539s | |
| Web Application Security Scanner | ✅ Pass | 33.683s | |
| Penetration Testing Tool Selection | ✅ Pass | 7.095s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 22.773s
---
### generator (qwen3-coder-next:cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.633s | |
| Text Transform Uppercase | ✅ Pass | 0.664s | |
| Count from 1 to 5 | ✅ Pass | 1.159s | |
| Math Calculation | ✅ Pass | 7.640s | |
| Basic Echo Function | ✅ Pass | 1.056s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.638s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.870s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.083s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.946s | |
| Search Query Function | ✅ Pass | 0.880s | |
| Ask Advice Function | ✅ Pass | 1.013s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.946s | |
| Basic Context Memory Test | ✅ Pass | 1.315s | |
| Function Argument Memory Test | ✅ Pass | 0.722s | |
| Function Response Memory Test | ✅ Pass | 3.881s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 6.597s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.642s | |
| Penetration Testing Methodology | ✅ Pass | 9.073s | |
| Vulnerability Assessment Tools | ✅ Pass | 3.598s | |
| SQL Injection Attack Type | ✅ Pass | 5.544s | |
| Penetration Testing Framework | ✅ Pass | 1.304s | |
| Web Application Security Scanner | ✅ Pass | 1.261s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.537s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.348s
---
### refiner (glm-5:cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 3.032s | |
| Text Transform Uppercase | ✅ Pass | 5.144s | |
| Count from 1 to 5 | ✅ Pass | 1.324s | |
| Math Calculation | ✅ Pass | 22.879s | |
| Basic Echo Function | ✅ Pass | 2.930s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.628s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.727s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.291s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.312s | |
| Search Query Function | ✅ Pass | 1.351s | |
| Ask Advice Function | ✅ Pass | 1.539s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.405s | |
| Basic Context Memory Test | ✅ Pass | 5.605s | |
| Function Argument Memory Test | ✅ Pass | 2.879s | |
| Function Response Memory Test | ✅ Pass | 1.468s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 15.864s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.101s | |
| Penetration Testing Methodology | ✅ Pass | 16.016s | |
| Vulnerability Assessment Tools | ✅ Pass | 24.194s | |
| SQL Injection Attack Type | ✅ Pass | 7.143s | |
| Penetration Testing Framework | ✅ Pass | 22.774s | |
| Web Application Security Scanner | ✅ Pass | 34.765s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.788s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 7.964s
---
### adviser (minimax-m2.7:cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.791s | |
| Text Transform Uppercase | ✅ Pass | 1.907s | |
| Count from 1 to 5 | ✅ Pass | 5.236s | |
| Math Calculation | ✅ Pass | 4.146s | |
| Basic Echo Function | ✅ Pass | 1.855s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.809s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.696s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 7.184s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.898s | |
| Search Query Function | ✅ Pass | 4.785s | |
| Ask Advice Function | ✅ Pass | 1.973s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.405s | |
| Basic Context Memory Test | ✅ Pass | 2.504s | |
| Function Argument Memory Test | ✅ Pass | 2.930s | |
| Function Response Memory Test | ✅ Pass | 2.279s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.874s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.765s | |
| Penetration Testing Methodology | ✅ Pass | 3.184s | |
| Vulnerability Assessment Tools | ✅ Pass | 14.047s | |
| SQL Injection Attack Type | ✅ Pass | 5.317s | |
| Penetration Testing Framework | ✅ Pass | 5.817s | |
| Web Application Security Scanner | ✅ Pass | 4.591s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.773s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.990s
---
### reflector (glm-5:cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 18.770s | |
| Text Transform Uppercase | ✅ Pass | 3.024s | |
| Count from 1 to 5 | ✅ Pass | 1.798s | |
| Math Calculation | ✅ Pass | 2.176s | |
| Basic Echo Function | ✅ Pass | 4.612s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.214s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 17.066s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.913s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.675s | |
| Search Query Function | ✅ Pass | 1.432s | |
| Ask Advice Function | ✅ Pass | 1.765s | |
| Streaming Search Query Function Streaming | ✅ Pass | 11.441s | |
| Basic Context Memory Test | ✅ Pass | 5.525s | |
| Function Argument Memory Test | ✅ Pass | 10.351s | |
| Function Response Memory Test | ✅ Pass | 3.459s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 7.899s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.117s | |
| Penetration Testing Methodology | ✅ Pass | 10.196s | |
| Vulnerability Assessment Tools | ✅ Pass | 63.102s | |
| SQL Injection Attack Type | ✅ Pass | 4.516s | |
| Penetration Testing Framework | ✅ Pass | 17.855s | |
| Web Application Security Scanner | ✅ Pass | 9.839s | |
| Penetration Testing Tool Selection | ✅ Pass | 6.867s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 9.201s
---
### searcher (qwen3.5:397b-cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.317s | |
| Text Transform Uppercase | ✅ Pass | 1.459s | |
| Count from 1 to 5 | ✅ Pass | 3.248s | |
| Math Calculation | ✅ Pass | 1.474s | |
| Basic Echo Function | ✅ Pass | 2.060s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.405s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.521s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.976s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.263s | |
| Search Query Function | ✅ Pass | 0.955s | |
| Ask Advice Function | ✅ Pass | 1.196s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.967s | |
| Basic Context Memory Test | ✅ Pass | 2.587s | |
| Function Argument Memory Test | ✅ Pass | 1.596s | |
| Function Response Memory Test | ✅ Pass | 1.408s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.083s | |
| Penetration Testing Methodology | ✅ Pass | 5.672s | |
| Penetration Testing Memory with Tool Call | ❌ Fail | 600.006s | Post "https://ollama\.com/api/chat?ts=1774282315": net/http: request canceled \(Client\.Timeout exceeded while awaiting headers\) |
| SQL Injection Attack Type | ✅ Pass | 4.083s | |
| Vulnerability Assessment Tools | ❌ Fail | 391.303s | 500 Internal Server Error: Internal Server Error \(ref: 9d3370b7\-e2d2\-4086\-9f8b\-948f29fcd567\) |
| Penetration Testing Framework | ❌ Fail | 30.619s | expected text 'exploitation' not found |
| Web Application Security Scanner | ✅ Pass | 2.647s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.136s | |
**Summary**: 20/23 (86.96%) successful tests
**Average latency**: 46.174s
---
### enricher (minimax-m2.7:cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 3.381s | |
| Text Transform Uppercase | ✅ Pass | 2.794s | |
| Count from 1 to 5 | ✅ Pass | 4.367s | |
| Math Calculation | ✅ Pass | 2.078s | |
| Basic Echo Function | ✅ Pass | 1.977s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.884s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.745s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.099s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.883s | |
| Search Query Function | ✅ Pass | 3.617s | |
| Ask Advice Function | ✅ Pass | 1.873s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.564s | |
| Basic Context Memory Test | ✅ Pass | 1.803s | |
| Function Argument Memory Test | ✅ Pass | 2.661s | |
| Function Response Memory Test | ✅ Pass | 2.227s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 5.442s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.429s | |
| Penetration Testing Methodology | ✅ Pass | 5.753s | |
| Vulnerability Assessment Tools | ✅ Pass | 70.743s | |
| SQL Injection Attack Type | ✅ Pass | 4.696s | |
| Penetration Testing Framework | ✅ Pass | 6.033s | |
| Web Application Security Scanner | ✅ Pass | 3.398s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.064s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 6.153s
---
### coder (qwen3-coder-next:cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.654s | |
| Text Transform Uppercase | ✅ Pass | 12.005s | |
| Count from 1 to 5 | ✅ Pass | 0.667s | |
| Math Calculation | ✅ Pass | 0.633s | |
| Basic Echo Function | ✅ Pass | 0.819s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.612s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.699s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.923s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 4.229s | |
| Search Query Function | ✅ Pass | 0.873s | |
| Ask Advice Function | ✅ Pass | 1.075s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.897s | |
| Basic Context Memory Test | ✅ Pass | 1.072s | |
| Function Argument Memory Test | ✅ Pass | 1.312s | |
| Function Response Memory Test | ✅ Pass | 0.642s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.623s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.710s | |
| Penetration Testing Methodology | ✅ Pass | 15.597s | |
| Vulnerability Assessment Tools | ✅ Pass | 12.591s | |
| SQL Injection Attack Type | ✅ Pass | 1.284s | |
| Penetration Testing Framework | ✅ Pass | 2.334s | |
| Web Application Security Scanner | ✅ Pass | 10.475s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.387s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.223s
---
### installer (devstral-2:123b-cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.698s | |
| Text Transform Uppercase | ✅ Pass | 0.738s | |
| Count from 1 to 5 | ✅ Pass | 0.793s | |
| Math Calculation | ✅ Pass | 0.709s | |
| Basic Echo Function | ✅ Pass | 0.842s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.028s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.740s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 158.484s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.955s | |
| Search Query Function | ✅ Pass | 0.851s | |
| Ask Advice Function | ✅ Pass | 1.002s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.813s | |
| Basic Context Memory Test | ✅ Pass | 0.851s | |
| Function Argument Memory Test | ✅ Pass | 0.757s | |
| Function Response Memory Test | ✅ Pass | 0.761s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.261s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.730s | |
| Penetration Testing Methodology | ✅ Pass | 2.598s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.323s | |
| SQL Injection Attack Type | ✅ Pass | 1.026s | |
| Penetration Testing Framework | ✅ Pass | 4.639s | |
| Web Application Security Scanner | ✅ Pass | 1.504s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.016s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 8.049s
---
### pentester (qwen3-coder-next:cloud)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.654s | |
| Text Transform Uppercase | ✅ Pass | 1.256s | |
| Count from 1 to 5 | ✅ Pass | 0.716s | |
| Math Calculation | ✅ Pass | 0.993s | |
| Basic Echo Function | ✅ Pass | 0.798s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.657s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.658s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.956s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.019s | |
| Search Query Function | ✅ Pass | 0.841s | |
| Ask Advice Function | ✅ Pass | 8.525s | |
| Streaming Search Query Function Streaming | ✅ Pass | 7.950s | |
| Basic Context Memory Test | ✅ Pass | 10.081s | |
| Function Argument Memory Test | ✅ Pass | 0.815s | |
| Function Response Memory Test | ✅ Pass | 0.675s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.659s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.804s | |
| Penetration Testing Methodology | ✅ Pass | 1.960s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.314s | |
| SQL Injection Attack Type | ✅ Pass | 1.424s | |
| Penetration Testing Framework | ✅ Pass | 1.673s | |
| Web Application Security Scanner | ✅ Pass | 1.148s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.646s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.227s
---
examples/tests/ollama-llama318b-instruct-report.md
+537
View File
@@ -0,0 +1,537 @@
# LLM Agent Testing Report
Generated: Sat, 17 Jan 2026 16:40:42 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
| simple | llama3.1:8b-instruct-q8_0 | false | 22/23 (95.65%) | 1.339s |
| simple_json | llama3.1:8b-instruct-q8_0 | false | 5/5 (100.00%) | 0.834s |
| primary_agent | llama3.1:8b-instruct-q8_0 | false | 21/23 (91.30%) | 1.335s |
| assistant | llama3.1:8b-instruct-q8_0 | false | 22/23 (95.65%) | 1.328s |
| generator | llama3.1:8b-instruct-q8_0 | false | 22/23 (95.65%) | 1.289s |
| refiner | llama3.1:8b-instruct-q8_0 | false | 21/23 (91.30%) | 1.246s |
| adviser | llama3.1:8b-instruct-q8_0 | false | 22/23 (95.65%) | 1.253s |
| reflector | llama3.1:8b-instruct-q8_0 | false | 22/23 (95.65%) | 1.305s |
| searcher | llama3.1:8b-instruct-q8_0 | false | 22/23 (95.65%) | 1.321s |
| enricher | llama3.1:8b-instruct-q8_0 | false | 22/23 (95.65%) | 1.320s |
| coder | llama3.1:8b-instruct-q8_0 | false | 21/23 (91.30%) | 1.321s |
| installer | llama3.1:8b-instruct-q8_0 | false | 21/23 (91.30%) | 1.277s |
| pentester | llama3.1:8b-instruct-q8_0 | false | 22/23 (95.65%) | 1.312s |
**Total**: 265/281 (94.31%) successful tests
**Overall average latency**: 1.295s
## Detailed Results
### simple (llama3.1:8b-instruct-q8_0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.373s | |
| Text Transform Uppercase | ✅ Pass | 0.250s | |
| Count from 1 to 5 | ✅ Pass | 0.297s | |
| Math Calculation | ✅ Pass | 0.265s | |
| Basic Echo Function | ✅ Pass | 0.388s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.455s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.293s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.403s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ❌ Fail | 0.643s | expected function 'respond\_with\_json' not found in tool calls: expected function respond\_with\_json not found in tool calls |
| Search Query Function | ✅ Pass | 0.806s | |
| Ask Advice Function | ✅ Pass | 0.686s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.853s | |
| Basic Context Memory Test | ✅ Pass | 1.018s | |
| Function Argument Memory Test | ✅ Pass | 1.243s | |
| Function Response Memory Test | ✅ Pass | 0.258s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 0.709s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.947s | |
| Penetration Testing Methodology | ✅ Pass | 1.805s | |
| Vulnerability Assessment Tools | ✅ Pass | 5.452s | |
| SQL Injection Attack Type | ✅ Pass | 5.091s | |
| Penetration Testing Framework | ✅ Pass | 0.966s | |
| Web Application Security Scanner | ✅ Pass | 4.471s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.104s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 1.339s
---
### simple_json (llama3.1:8b-instruct-q8_0)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Vulnerability Report Memory Test | ✅ Pass | 0.721s | |
| Person Information JSON | ✅ Pass | 0.867s | |
| Project Information JSON | ✅ Pass | 0.989s | |
| User Profile JSON | ✅ Pass | 0.978s | |
| Streaming Person Information JSON Streaming | ✅ Pass | 0.611s | |
**Summary**: 5/5 (100.00%) successful tests
**Average latency**: 0.834s
---
### primary_agent (llama3.1:8b-instruct-q8_0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.374s | |
| Text Transform Uppercase | ✅ Pass | 0.263s | |
| Count from 1 to 5 | ✅ Pass | 0.287s | |
| Math Calculation | ✅ Pass | 0.228s | |
| Basic Echo Function | ✅ Pass | 0.582s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.329s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.314s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.523s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ❌ Fail | 0.701s | expected function 'respond\_with\_json' not found in tool calls: expected function respond\_with\_json not found in tool calls |
| Search Query Function | ✅ Pass | 0.724s | |
| Ask Advice Function | ✅ Pass | 0.772s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.757s | |
| Basic Context Memory Test | ✅ Pass | 1.178s | |
| Function Argument Memory Test | ✅ Pass | 0.871s | |
| Function Response Memory Test | ✅ Pass | 0.239s | |
| Penetration Testing Memory with Tool Call | ❌ Fail | 0.945s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.705s | |
| Penetration Testing Methodology | ✅ Pass | 2.971s | |
| Vulnerability Assessment Tools | ✅ Pass | 5.949s | |
| SQL Injection Attack Type | ✅ Pass | 3.361s | |
| Penetration Testing Framework | ✅ Pass | 1.945s | |
| Web Application Security Scanner | ✅ Pass | 4.296s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.377s | |
**Summary**: 21/23 (91.30%) successful tests
**Average latency**: 1.335s
---
### assistant (llama3.1:8b-instruct-q8_0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.320s | |
| Text Transform Uppercase | ✅ Pass | 0.278s | |
| Count from 1 to 5 | ✅ Pass | 0.351s | |
| Math Calculation | ✅ Pass | 0.234s | |
| Basic Echo Function | ✅ Pass | 0.648s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.213s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.363s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.606s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ❌ Fail | 0.776s | expected function 'respond\_with\_json' not found in tool calls: expected function respond\_with\_json not found in tool calls |
| Search Query Function | ✅ Pass | 0.663s | |
| Ask Advice Function | ✅ Pass | 0.854s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.659s | |
| Basic Context Memory Test | ✅ Pass | 1.354s | |
| Function Argument Memory Test | ✅ Pass | 0.669s | |
| Function Response Memory Test | ✅ Pass | 0.233s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.226s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.462s | |
| Penetration Testing Methodology | ✅ Pass | 3.679s | |
| Vulnerability Assessment Tools | ✅ Pass | 6.123s | |
| SQL Injection Attack Type | ✅ Pass | 2.013s | |
| Penetration Testing Framework | ✅ Pass | 2.886s | |
| Web Application Security Scanner | ✅ Pass | 4.149s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.773s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 1.328s
---
### generator (llama3.1:8b-instruct-q8_0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.371s | |
| Text Transform Uppercase | ✅ Pass | 0.287s | |
| Count from 1 to 5 | ✅ Pass | 0.373s | |
| Math Calculation | ✅ Pass | 0.245s | |
| Basic Echo Function | ✅ Pass | 0.475s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.245s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.284s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.684s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ❌ Fail | 0.857s | expected function 'respond\_with\_json' not found in tool calls: expected function respond\_with\_json not found in tool calls |
| Search Query Function | ✅ Pass | 0.585s | |
| Ask Advice Function | ✅ Pass | 0.947s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.567s | |
| Basic Context Memory Test | ✅ Pass | 1.460s | |
| Function Argument Memory Test | ✅ Pass | 0.274s | |
| Function Response Memory Test | ✅ Pass | 0.245s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.535s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.251s | |
| Penetration Testing Methodology | ✅ Pass | 4.738s | |
| Vulnerability Assessment Tools | ✅ Pass | 6.499s | |
| SQL Injection Attack Type | ✅ Pass | 0.240s | |
| Penetration Testing Framework | ✅ Pass | 3.607s | |
| Web Application Security Scanner | ✅ Pass | 3.937s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.936s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 1.289s
---
### refiner (llama3.1:8b-instruct-q8_0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.240s | |
| Text Transform Uppercase | ✅ Pass | 0.243s | |
| Count from 1 to 5 | ✅ Pass | 0.325s | |
| Math Calculation | ✅ Pass | 0.241s | |
| Basic Echo Function | ✅ Pass | 0.596s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.274s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.271s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.574s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ❌ Fail | 0.847s | expected function 'respond\_with\_json' not found in tool calls: expected function respond\_with\_json not found in tool calls |
| Search Query Function | ✅ Pass | 0.592s | |
| Ask Advice Function | ✅ Pass | 0.924s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.550s | |
| Basic Context Memory Test | ✅ Pass | 1.093s | |
| Function Argument Memory Test | ✅ Pass | 0.286s | |
| Function Response Memory Test | ✅ Pass | 0.233s | |
| Penetration Testing Memory with Tool Call | ❌ Fail | 1.316s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.257s | |
| Penetration Testing Methodology | ✅ Pass | 4.687s | |
| Vulnerability Assessment Tools | ✅ Pass | 6.461s | |
| SQL Injection Attack Type | ✅ Pass | 0.250s | |
| Penetration Testing Framework | ✅ Pass | 4.012s | |
| Web Application Security Scanner | ✅ Pass | 3.463s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.901s | |
**Summary**: 21/23 (91.30%) successful tests
**Average latency**: 1.246s
---
### adviser (llama3.1:8b-instruct-q8_0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.275s | |
| Text Transform Uppercase | ✅ Pass | 0.260s | |
| Count from 1 to 5 | ✅ Pass | 0.308s | |
| Math Calculation | ✅ Pass | 0.244s | |
| Basic Echo Function | ✅ Pass | 0.550s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.258s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.274s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.570s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ❌ Fail | 1.157s | expected function 'respond\_with\_json' not found in tool calls: expected function respond\_with\_json not found in tool calls |
| Search Query Function | ✅ Pass | 0.573s | |
| Ask Advice Function | ✅ Pass | 0.925s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.546s | |
| Basic Context Memory Test | ✅ Pass | 1.180s | |
| Function Argument Memory Test | ✅ Pass | 0.291s | |
| Function Response Memory Test | ✅ Pass | 0.245s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.473s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.254s | |
| Penetration Testing Methodology | ✅ Pass | 4.640s | |
| Vulnerability Assessment Tools | ✅ Pass | 6.311s | |
| SQL Injection Attack Type | ✅ Pass | 0.255s | |
| Penetration Testing Framework | ✅ Pass | 4.100s | |
| Web Application Security Scanner | ✅ Pass | 3.210s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.900s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 1.253s
---
### reflector (llama3.1:8b-instruct-q8_0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.245s | |
| Text Transform Uppercase | ✅ Pass | 0.264s | |
| Count from 1 to 5 | ✅ Pass | 0.297s | |
| Math Calculation | ✅ Pass | 0.238s | |
| Basic Echo Function | ✅ Pass | 0.564s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.243s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.272s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.575s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ❌ Fail | 1.144s | expected function 'respond\_with\_json' not found in tool calls: expected function respond\_with\_json not found in tool calls |
| Search Query Function | ✅ Pass | 0.575s | |
| Ask Advice Function | ✅ Pass | 0.927s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.549s | |
| Basic Context Memory Test | ✅ Pass | 1.260s | |
| Function Argument Memory Test | ✅ Pass | 0.278s | |
| Function Response Memory Test | ✅ Pass | 0.242s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.538s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.255s | |
| Penetration Testing Methodology | ✅ Pass | 5.074s | |
| Vulnerability Assessment Tools | ✅ Pass | 6.746s | |
| SQL Injection Attack Type | ✅ Pass | 0.248s | |
| Penetration Testing Framework | ✅ Pass | 4.304s | |
| Web Application Security Scanner | ✅ Pass | 3.257s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.902s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 1.305s
---
### searcher (llama3.1:8b-instruct-q8_0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.257s | |
| Text Transform Uppercase | ✅ Pass | 0.266s | |
| Count from 1 to 5 | ✅ Pass | 0.316s | |
| Math Calculation | ✅ Pass | 0.260s | |
| Basic Echo Function | ✅ Pass | 0.573s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.235s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.279s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.574s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ❌ Fail | 1.142s | expected function 'respond\_with\_json' not found in tool calls: expected function respond\_with\_json not found in tool calls |
| Search Query Function | ✅ Pass | 0.575s | |
| Ask Advice Function | ✅ Pass | 0.924s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.547s | |
| Basic Context Memory Test | ✅ Pass | 1.481s | |
| Function Argument Memory Test | ✅ Pass | 0.288s | |
| Function Response Memory Test | ✅ Pass | 0.270s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.471s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.256s | |
| Penetration Testing Methodology | ✅ Pass | 5.094s | |
| Vulnerability Assessment Tools | ✅ Pass | 6.750s | |
| SQL Injection Attack Type | ✅ Pass | 0.266s | |
| Penetration Testing Framework | ✅ Pass | 4.493s | |
| Web Application Security Scanner | ✅ Pass | 3.142s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.900s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 1.321s
---
### enricher (llama3.1:8b-instruct-q8_0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.239s | |
| Text Transform Uppercase | ✅ Pass | 0.247s | |
| Count from 1 to 5 | ✅ Pass | 0.315s | |
| Math Calculation | ✅ Pass | 0.258s | |
| Basic Echo Function | ✅ Pass | 0.575s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.243s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.298s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.575s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ❌ Fail | 1.164s | expected function 'respond\_with\_json' not found in tool calls: expected function respond\_with\_json not found in tool calls |
| Search Query Function | ✅ Pass | 0.556s | |
| Ask Advice Function | ✅ Pass | 0.923s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.542s | |
| Basic Context Memory Test | ✅ Pass | 1.482s | |
| Function Argument Memory Test | ✅ Pass | 0.273s | |
| Function Response Memory Test | ✅ Pass | 0.267s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.561s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.257s | |
| Penetration Testing Methodology | ✅ Pass | 4.954s | |
| Vulnerability Assessment Tools | ✅ Pass | 6.721s | |
| SQL Injection Attack Type | ✅ Pass | 0.268s | |
| Penetration Testing Framework | ✅ Pass | 4.513s | |
| Web Application Security Scanner | ✅ Pass | 3.210s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.900s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 1.320s
---
### coder (llama3.1:8b-instruct-q8_0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.243s | |
| Text Transform Uppercase | ✅ Pass | 0.260s | |
| Count from 1 to 5 | ✅ Pass | 0.311s | |
| Math Calculation | ✅ Pass | 0.234s | |
| Basic Echo Function | ✅ Pass | 0.576s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.268s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.277s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.577s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ❌ Fail | 0.878s | expected function 'respond\_with\_json' not found in tool calls: expected function respond\_with\_json not found in tool calls |
| Search Query Function | ✅ Pass | 0.577s | |
| Ask Advice Function | ✅ Pass | 0.923s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.547s | |
| Basic Context Memory Test | ✅ Pass | 1.531s | |
| Function Argument Memory Test | ✅ Pass | 0.276s | |
| Function Response Memory Test | ✅ Pass | 0.237s | |
| Penetration Testing Memory with Tool Call | ❌ Fail | 1.404s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.257s | |
| Penetration Testing Methodology | ✅ Pass | 4.907s | |
| Vulnerability Assessment Tools | ✅ Pass | 6.909s | |
| SQL Injection Attack Type | ✅ Pass | 0.261s | |
| Penetration Testing Framework | ✅ Pass | 4.538s | |
| Web Application Security Scanner | ✅ Pass | 3.478s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.900s | |
**Summary**: 21/23 (91.30%) successful tests
**Average latency**: 1.321s
---
### installer (llama3.1:8b-instruct-q8_0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.247s | |
| Text Transform Uppercase | ✅ Pass | 0.256s | |
| Count from 1 to 5 | ✅ Pass | 0.302s | |
| Math Calculation | ✅ Pass | 0.225s | |
| Basic Echo Function | ✅ Pass | 0.573s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.267s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.276s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.574s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ❌ Fail | 0.877s | expected function 'respond\_with\_json' not found in tool calls: expected function respond\_with\_json not found in tool calls |
| Search Query Function | ✅ Pass | 0.580s | |
| Ask Advice Function | ✅ Pass | 0.925s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.544s | |
| Basic Context Memory Test | ✅ Pass | 1.422s | |
| Function Argument Memory Test | ✅ Pass | 0.281s | |
| Function Response Memory Test | ✅ Pass | 0.250s | |
| Penetration Testing Memory with Tool Call | ❌ Fail | 1.250s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.258s | |
| Penetration Testing Methodology | ✅ Pass | 4.888s | |
| Vulnerability Assessment Tools | ✅ Pass | 6.426s | |
| SQL Injection Attack Type | ✅ Pass | 0.258s | |
| Penetration Testing Framework | ✅ Pass | 4.368s | |
| Web Application Security Scanner | ✅ Pass | 3.404s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.899s | |
**Summary**: 21/23 (91.30%) successful tests
**Average latency**: 1.277s
---
### pentester (llama3.1:8b-instruct-q8_0)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.251s | |
| Text Transform Uppercase | ✅ Pass | 0.258s | |
| Count from 1 to 5 | ✅ Pass | 0.312s | |
| Math Calculation | ✅ Pass | 0.434s | |
| Basic Echo Function | ✅ Pass | 0.575s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.242s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.270s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.575s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ❌ Fail | 0.876s | expected function 'respond\_with\_json' not found in tool calls: expected function respond\_with\_json not found in tool calls |
| Search Query Function | ✅ Pass | 0.574s | |
| Ask Advice Function | ✅ Pass | 0.926s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.545s | |
| Basic Context Memory Test | ✅ Pass | 1.406s | |
| Function Argument Memory Test | ✅ Pass | 0.293s | |
| Function Response Memory Test | ✅ Pass | 0.250s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.249s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.258s | |
| Penetration Testing Methodology | ✅ Pass | 5.022s | |
| Vulnerability Assessment Tools | ✅ Pass | 6.589s | |
| SQL Injection Attack Type | ✅ Pass | 0.241s | |
| Penetration Testing Framework | ✅ Pass | 4.368s | |
| Web Application Security Scanner | ✅ Pass | 3.743s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.900s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 1.312s
---
examples/tests/ollama-llama318b-report.md
+537
View File
@@ -0,0 +1,537 @@
# LLM Agent Testing Report
Generated: Sat, 19 Jul 2025 19:43:32 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
| simple | llama3.1:8b | false | 23/23 (100.00%) | 0.641s |
| simple_json | llama3.1:8b | false | 5/5 (100.00%) | 0.514s |
| primary_agent | llama3.1:8b | false | 23/23 (100.00%) | 0.545s |
| assistant | llama3.1:8b | false | 23/23 (100.00%) | 0.543s |
| generator | llama3.1:8b | false | 23/23 (100.00%) | 0.512s |
| refiner | llama3.1:8b | false | 23/23 (100.00%) | 0.528s |
| adviser | llama3.1:8b | false | 23/23 (100.00%) | 0.538s |
| reflector | llama3.1:8b | false | 23/23 (100.00%) | 0.545s |
| searcher | llama3.1:8b | false | 23/23 (100.00%) | 0.533s |
| enricher | llama3.1:8b | false | 23/23 (100.00%) | 0.546s |
| coder | llama3.1:8b | false | 23/23 (100.00%) | 0.565s |
| installer | llama3.1:8b | false | 23/23 (100.00%) | 0.546s |
| pentester | llama3.1:8b | false | 23/23 (100.00%) | 0.543s |
**Total**: 281/281 (100.00%) successful tests
**Overall average latency**: 0.548s
## Detailed Results
### simple (llama3.1:8b)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.290s | |
| Text Transform Uppercase | ✅ Pass | 0.323s | |
| Count from 1 to 5 | ✅ Pass | 0.366s | |
| Math Calculation | ✅ Pass | 0.314s | |
| Basic Echo Function | ✅ Pass | 0.431s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.312s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.472s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.542s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.403s | |
| Search Query Function | ✅ Pass | 0.411s | |
| Ask Advice Function | ✅ Pass | 0.502s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.575s | |
| Basic Context Memory Test | ✅ Pass | 0.457s | |
| Function Argument Memory Test | ✅ Pass | 0.356s | |
| Function Response Memory Test | ✅ Pass | 0.405s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.218s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.315s | |
| Penetration Testing Methodology | ✅ Pass | 1.245s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.782s | |
| SQL Injection Attack Type | ✅ Pass | 0.319s | |
| Penetration Testing Framework | ✅ Pass | 1.296s | |
| Web Application Security Scanner | ✅ Pass | 0.962s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.437s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.641s
---
### simple_json (llama3.1:8b)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Vulnerability Report Memory Test | ✅ Pass | 0.812s | |
| Person Information JSON | ✅ Pass | 0.427s | |
| Project Information JSON | ✅ Pass | 0.410s | |
| User Profile JSON | ✅ Pass | 0.445s | |
| Streaming Person Information JSON Streaming | ✅ Pass | 0.472s | |
**Summary**: 5/5 (100.00%) successful tests
**Average latency**: 0.514s
---
### primary_agent (llama3.1:8b)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.306s | |
| Text Transform Uppercase | ✅ Pass | 0.313s | |
| Count from 1 to 5 | ✅ Pass | 0.348s | |
| Math Calculation | ✅ Pass | 0.306s | |
| Basic Echo Function | ✅ Pass | 0.408s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.306s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.419s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.517s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.388s | |
| Search Query Function | ✅ Pass | 0.401s | |
| Ask Advice Function | ✅ Pass | 0.470s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.573s | |
| Basic Context Memory Test | ✅ Pass | 0.633s | |
| Function Argument Memory Test | ✅ Pass | 0.334s | |
| Function Response Memory Test | ✅ Pass | 0.303s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 0.530s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.273s | |
| Penetration Testing Methodology | ✅ Pass | 1.205s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.701s | |
| SQL Injection Attack Type | ✅ Pass | 0.444s | |
| Penetration Testing Framework | ✅ Pass | 1.015s | |
| Web Application Security Scanner | ✅ Pass | 0.924s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.400s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.545s
---
### assistant (llama3.1:8b)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.309s | |
| Text Transform Uppercase | ✅ Pass | 0.321s | |
| Count from 1 to 5 | ✅ Pass | 0.349s | |
| Math Calculation | ✅ Pass | 0.303s | |
| Basic Echo Function | ✅ Pass | 0.403s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.302s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.423s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.518s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.430s | |
| Search Query Function | ✅ Pass | 0.401s | |
| Ask Advice Function | ✅ Pass | 0.467s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.515s | |
| Basic Context Memory Test | ✅ Pass | 0.638s | |
| Function Argument Memory Test | ✅ Pass | 0.347s | |
| Function Response Memory Test | ✅ Pass | 0.304s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 0.534s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.274s | |
| Penetration Testing Methodology | ✅ Pass | 1.197s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.663s | |
| SQL Injection Attack Type | ✅ Pass | 0.341s | |
| Penetration Testing Framework | ✅ Pass | 1.142s | |
| Web Application Security Scanner | ✅ Pass | 0.889s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.398s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.543s
---
### generator (llama3.1:8b)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.303s | |
| Text Transform Uppercase | ✅ Pass | 0.327s | |
| Count from 1 to 5 | ✅ Pass | 0.346s | |
| Math Calculation | ✅ Pass | 0.302s | |
| Basic Echo Function | ✅ Pass | 0.404s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.304s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.418s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.519s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.439s | |
| Search Query Function | ✅ Pass | 0.399s | |
| Ask Advice Function | ✅ Pass | 0.470s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.511s | |
| Basic Context Memory Test | ✅ Pass | 0.473s | |
| Function Argument Memory Test | ✅ Pass | 0.294s | |
| Function Response Memory Test | ✅ Pass | 0.305s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 0.530s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.279s | |
| Penetration Testing Methodology | ✅ Pass | 0.812s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.864s | |
| SQL Injection Attack Type | ✅ Pass | 0.305s | |
| Penetration Testing Framework | ✅ Pass | 0.795s | |
| Web Application Security Scanner | ✅ Pass | 0.970s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.398s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.512s
---
### refiner (llama3.1:8b)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.301s | |
| Text Transform Uppercase | ✅ Pass | 0.313s | |
| Count from 1 to 5 | ✅ Pass | 0.350s | |
| Math Calculation | ✅ Pass | 0.305s | |
| Basic Echo Function | ✅ Pass | 0.405s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.304s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.420s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.520s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.428s | |
| Search Query Function | ✅ Pass | 0.400s | |
| Ask Advice Function | ✅ Pass | 0.468s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.509s | |
| Basic Context Memory Test | ✅ Pass | 0.450s | |
| Function Argument Memory Test | ✅ Pass | 0.339s | |
| Function Response Memory Test | ✅ Pass | 0.300s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 0.529s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.274s | |
| Penetration Testing Methodology | ✅ Pass | 1.232s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.385s | |
| SQL Injection Attack Type | ✅ Pass | 0.397s | |
| Penetration Testing Framework | ✅ Pass | 1.209s | |
| Web Application Security Scanner | ✅ Pass | 0.906s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.397s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.528s
---
### adviser (llama3.1:8b)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.307s | |
| Text Transform Uppercase | ✅ Pass | 0.315s | |
| Count from 1 to 5 | ✅ Pass | 0.349s | |
| Math Calculation | ✅ Pass | 0.304s | |
| Basic Echo Function | ✅ Pass | 0.406s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.301s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.421s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.517s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.432s | |
| Search Query Function | ✅ Pass | 0.399s | |
| Ask Advice Function | ✅ Pass | 0.470s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.508s | |
| Basic Context Memory Test | ✅ Pass | 0.477s | |
| Function Argument Memory Test | ✅ Pass | 0.339s | |
| Function Response Memory Test | ✅ Pass | 0.303s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 0.532s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.275s | |
| Penetration Testing Methodology | ✅ Pass | 1.082s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.479s | |
| SQL Injection Attack Type | ✅ Pass | 0.315s | |
| Penetration Testing Framework | ✅ Pass | 1.092s | |
| Web Application Security Scanner | ✅ Pass | 1.331s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.400s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.538s
---
### reflector (llama3.1:8b)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.303s | |
| Text Transform Uppercase | ✅ Pass | 0.316s | |
| Count from 1 to 5 | ✅ Pass | 0.356s | |
| Math Calculation | ✅ Pass | 0.301s | |
| Basic Echo Function | ✅ Pass | 0.401s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.307s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.418s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.518s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.424s | |
| Search Query Function | ✅ Pass | 0.401s | |
| Ask Advice Function | ✅ Pass | 0.467s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.511s | |
| Basic Context Memory Test | ✅ Pass | 0.485s | |
| Function Argument Memory Test | ✅ Pass | 0.366s | |
| Function Response Memory Test | ✅ Pass | 0.307s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 0.542s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.277s | |
| Penetration Testing Methodology | ✅ Pass | 1.486s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.552s | |
| SQL Injection Attack Type | ✅ Pass | 0.313s | |
| Penetration Testing Framework | ✅ Pass | 1.079s | |
| Web Application Security Scanner | ✅ Pass | 0.999s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.399s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.545s
---
### searcher (llama3.1:8b)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.307s | |
| Text Transform Uppercase | ✅ Pass | 0.315s | |
| Count from 1 to 5 | ✅ Pass | 0.343s | |
| Math Calculation | ✅ Pass | 0.304s | |
| Basic Echo Function | ✅ Pass | 0.407s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.300s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.422s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.517s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.430s | |
| Search Query Function | ✅ Pass | 0.400s | |
| Ask Advice Function | ✅ Pass | 0.468s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.516s | |
| Basic Context Memory Test | ✅ Pass | 0.472s | |
| Function Argument Memory Test | ✅ Pass | 0.352s | |
| Function Response Memory Test | ✅ Pass | 0.302s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 0.528s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.276s | |
| Penetration Testing Methodology | ✅ Pass | 1.057s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.729s | |
| SQL Injection Attack Type | ✅ Pass | 0.444s | |
| Penetration Testing Framework | ✅ Pass | 1.007s | |
| Web Application Security Scanner | ✅ Pass | 0.888s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.468s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.533s
---
### enricher (llama3.1:8b)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.302s | |
| Text Transform Uppercase | ✅ Pass | 0.317s | |
| Count from 1 to 5 | ✅ Pass | 0.352s | |
| Math Calculation | ✅ Pass | 0.264s | |
| Basic Echo Function | ✅ Pass | 0.397s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.303s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.424s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.516s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.425s | |
| Search Query Function | ✅ Pass | 0.400s | |
| Ask Advice Function | ✅ Pass | 0.466s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.387s | |
| Basic Context Memory Test | ✅ Pass | 0.484s | |
| Function Argument Memory Test | ✅ Pass | 0.337s | |
| Function Response Memory Test | ✅ Pass | 0.301s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 0.534s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.274s | |
| Penetration Testing Methodology | ✅ Pass | 1.201s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.817s | |
| SQL Injection Attack Type | ✅ Pass | 0.526s | |
| Penetration Testing Framework | ✅ Pass | 1.105s | |
| Web Application Security Scanner | ✅ Pass | 0.971s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.453s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.546s
---
### coder (llama3.1:8b)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.312s | |
| Text Transform Uppercase | ✅ Pass | 0.316s | |
| Count from 1 to 5 | ✅ Pass | 0.349s | |
| Math Calculation | ✅ Pass | 0.301s | |
| Basic Echo Function | ✅ Pass | 0.401s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.305s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.425s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.518s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.429s | |
| Search Query Function | ✅ Pass | 0.399s | |
| Ask Advice Function | ✅ Pass | 0.469s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.556s | |
| Basic Context Memory Test | ✅ Pass | 0.638s | |
| Function Argument Memory Test | ✅ Pass | 0.380s | |
| Function Response Memory Test | ✅ Pass | 0.310s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 0.530s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.275s | |
| Penetration Testing Methodology | ✅ Pass | 1.201s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.092s | |
| SQL Injection Attack Type | ✅ Pass | 0.315s | |
| Penetration Testing Framework | ✅ Pass | 1.159s | |
| Web Application Security Scanner | ✅ Pass | 0.896s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.403s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.565s
---
### installer (llama3.1:8b)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.305s | |
| Text Transform Uppercase | ✅ Pass | 0.315s | |
| Count from 1 to 5 | ✅ Pass | 0.354s | |
| Math Calculation | ✅ Pass | 0.303s | |
| Basic Echo Function | ✅ Pass | 0.405s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.306s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.417s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.518s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.431s | |
| Search Query Function | ✅ Pass | 0.398s | |
| Ask Advice Function | ✅ Pass | 0.467s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.508s | |
| Basic Context Memory Test | ✅ Pass | 0.639s | |
| Function Argument Memory Test | ✅ Pass | 0.337s | |
| Function Response Memory Test | ✅ Pass | 0.304s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 0.530s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.277s | |
| Penetration Testing Methodology | ✅ Pass | 1.198s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.696s | |
| SQL Injection Attack Type | ✅ Pass | 0.469s | |
| Penetration Testing Framework | ✅ Pass | 1.076s | |
| Web Application Security Scanner | ✅ Pass | 0.890s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.399s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.546s
---
### pentester (llama3.1:8b)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.303s | |
| Text Transform Uppercase | ✅ Pass | 0.316s | |
| Count from 1 to 5 | ✅ Pass | 0.356s | |
| Math Calculation | ✅ Pass | 0.302s | |
| Basic Echo Function | ✅ Pass | 0.404s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.301s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.420s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.520s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.431s | |
| Search Query Function | ✅ Pass | 0.399s | |
| Ask Advice Function | ✅ Pass | 0.467s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.510s | |
| Basic Context Memory Test | ✅ Pass | 0.505s | |
| Function Argument Memory Test | ✅ Pass | 0.334s | |
| Function Response Memory Test | ✅ Pass | 0.306s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 0.534s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.274s | |
| Penetration Testing Methodology | ✅ Pass | 1.208s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.880s | |
| SQL Injection Attack Type | ✅ Pass | 0.308s | |
| Penetration Testing Framework | ✅ Pass | 0.987s | |
| Web Application Security Scanner | ✅ Pass | 1.013s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.398s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.543s
---
examples/tests/ollama-qwen332b-fp16-tc-report.md
+537
View File
@@ -0,0 +1,537 @@
# LLM Agent Testing Report
Generated: Sat, 19 Jul 2025 21:18:34 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
| simple | qwen3:32b-fp16-tc | true | 23/23 (100.00%) | 7.029s |
| simple_json | qwen3:32b-fp16-tc | true | 5/5 (100.00%) | 6.073s |
| primary_agent | qwen3:32b-fp16-tc | true | 22/23 (95.65%) | 6.596s |
| assistant | qwen3:32b-fp16-tc | true | 23/23 (100.00%) | 7.374s |
| generator | qwen3:32b-fp16-tc | true | 23/23 (100.00%) | 6.395s |
| refiner | qwen3:32b-fp16-tc | true | 23/23 (100.00%) | 7.367s |
| adviser | qwen3:32b-fp16-tc | true | 23/23 (100.00%) | 7.065s |
| reflector | qwen3:32b-fp16-tc | true | 23/23 (100.00%) | 6.974s |
| searcher | qwen3:32b-fp16-tc | true | 23/23 (100.00%) | 6.736s |
| enricher | qwen3:32b-fp16-tc | true | 22/23 (95.65%) | 6.578s |
| coder | qwen3:32b-fp16-tc | true | 23/23 (100.00%) | 7.086s |
| installer | qwen3:32b-fp16-tc | true | 23/23 (100.00%) | 6.952s |
| pentester | qwen3:32b-fp16-tc | true | 23/23 (100.00%) | 7.140s |
**Total**: 279/281 (99.29%) successful tests
**Overall average latency**: 6.925s
## Detailed Results
### simple (qwen3:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 8.338s | |
| Text Transform Uppercase | ✅ Pass | 7.296s | |
| Count from 1 to 5 | ✅ Pass | 4.953s | |
| Math Calculation | ✅ Pass | 4.497s | |
| Basic Echo Function | ✅ Pass | 3.309s | |
| Streaming Simple Math Streaming | ✅ Pass | 9.747s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 6.707s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.416s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 5.116s | |
| Search Query Function | ✅ Pass | 3.968s | |
| Ask Advice Function | ✅ Pass | 4.378s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.260s | |
| Basic Context Memory Test | ✅ Pass | 5.115s | |
| Function Argument Memory Test | ✅ Pass | 6.694s | |
| Function Response Memory Test | ✅ Pass | 4.319s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 14.800s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.892s | |
| Penetration Testing Methodology | ✅ Pass | 9.962s | |
| Vulnerability Assessment Tools | ✅ Pass | 12.119s | |
| SQL Injection Attack Type | ✅ Pass | 7.969s | |
| Penetration Testing Framework | ✅ Pass | 11.575s | |
| Web Application Security Scanner | ✅ Pass | 14.461s | |
| Penetration Testing Tool Selection | ✅ Pass | 5.754s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 7.029s
---
### simple_json (qwen3:32b-fp16-tc)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Vulnerability Report Memory Test | ✅ Pass | 5.983s | |
| Person Information JSON | ✅ Pass | 4.989s | |
| Project Information JSON | ✅ Pass | 6.805s | |
| User Profile JSON | ✅ Pass | 6.519s | |
| Streaming Person Information JSON Streaming | ✅ Pass | 6.068s | |
**Summary**: 5/5 (100.00%) successful tests
**Average latency**: 6.073s
---
### primary_agent (qwen3:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 7.310s | |
| Text Transform Uppercase | ✅ Pass | 5.831s | |
| Count from 1 to 5 | ✅ Pass | 5.415s | |
| Math Calculation | ✅ Pass | 6.598s | |
| Basic Echo Function | ✅ Pass | 3.385s | |
| Streaming Simple Math Streaming | ✅ Pass | 4.320s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.366s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.270s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 5.612s | |
| Search Query Function | ✅ Pass | 4.001s | |
| Ask Advice Function | ✅ Pass | 4.534s | |
| Streaming Search Query Function Streaming | ✅ Pass | 4.845s | |
| Basic Context Memory Test | ✅ Pass | 5.125s | |
| Function Argument Memory Test | ✅ Pass | 4.316s | |
| Function Response Memory Test | ✅ Pass | 3.577s | |
| Penetration Testing Memory with Tool Call | ❌ Fail | 11.379s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.505s | |
| Penetration Testing Methodology | ✅ Pass | 11.729s | |
| Vulnerability Assessment Tools | ✅ Pass | 13.465s | |
| SQL Injection Attack Type | ✅ Pass | 7.851s | |
| Penetration Testing Framework | ✅ Pass | 11.415s | |
| Web Application Security Scanner | ✅ Pass | 12.780s | |
| Penetration Testing Tool Selection | ✅ Pass | 5.079s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 6.596s
---
### assistant (qwen3:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 7.196s | |
| Text Transform Uppercase | ✅ Pass | 5.213s | |
| Count from 1 to 5 | ✅ Pass | 3.672s | |
| Math Calculation | ✅ Pass | 5.501s | |
| Basic Echo Function | ✅ Pass | 3.435s | |
| Streaming Simple Math Streaming | ✅ Pass | 5.058s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.833s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 5.393s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 5.098s | |
| Search Query Function | ✅ Pass | 4.025s | |
| Ask Advice Function | ✅ Pass | 5.241s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.946s | |
| Basic Context Memory Test | ✅ Pass | 4.055s | |
| Function Argument Memory Test | ✅ Pass | 7.927s | |
| Function Response Memory Test | ✅ Pass | 21.505s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 10.776s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.533s | |
| Penetration Testing Methodology | ✅ Pass | 11.291s | |
| Vulnerability Assessment Tools | ✅ Pass | 12.372s | |
| SQL Injection Attack Type | ✅ Pass | 10.011s | |
| Penetration Testing Framework | ✅ Pass | 16.996s | |
| Web Application Security Scanner | ✅ Pass | 6.533s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.978s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 7.374s
---
### generator (qwen3:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 8.272s | |
| Text Transform Uppercase | ✅ Pass | 6.269s | |
| Count from 1 to 5 | ✅ Pass | 5.975s | |
| Math Calculation | ✅ Pass | 5.078s | |
| Basic Echo Function | ✅ Pass | 3.326s | |
| Streaming Simple Math Streaming | ✅ Pass | 6.757s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.235s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.513s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 5.065s | |
| Search Query Function | ✅ Pass | 2.729s | |
| Ask Advice Function | ✅ Pass | 4.952s | |
| Streaming Search Query Function Streaming | ✅ Pass | 4.660s | |
| Basic Context Memory Test | ✅ Pass | 4.273s | |
| Function Argument Memory Test | ✅ Pass | 4.981s | |
| Function Response Memory Test | ✅ Pass | 6.514s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 8.339s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.602s | |
| Penetration Testing Methodology | ✅ Pass | 9.196s | |
| Vulnerability Assessment Tools | ✅ Pass | 15.506s | |
| SQL Injection Attack Type | ✅ Pass | 6.542s | |
| Penetration Testing Framework | ✅ Pass | 11.258s | |
| Web Application Security Scanner | ✅ Pass | 10.277s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.751s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 6.395s
---
### refiner (qwen3:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 9.163s | |
| Text Transform Uppercase | ✅ Pass | 6.860s | |
| Count from 1 to 5 | ✅ Pass | 5.760s | |
| Math Calculation | ✅ Pass | 6.596s | |
| Basic Echo Function | ✅ Pass | 3.326s | |
| Streaming Simple Math Streaming | ✅ Pass | 6.044s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.567s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.097s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.782s | |
| Search Query Function | ✅ Pass | 2.999s | |
| Ask Advice Function | ✅ Pass | 4.545s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.240s | |
| Basic Context Memory Test | ✅ Pass | 3.805s | |
| Function Argument Memory Test | ✅ Pass | 13.018s | |
| Function Response Memory Test | ✅ Pass | 13.484s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 18.941s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.375s | |
| Penetration Testing Methodology | ✅ Pass | 9.131s | |
| Vulnerability Assessment Tools | ✅ Pass | 16.578s | |
| SQL Injection Attack Type | ✅ Pass | 6.729s | |
| Penetration Testing Framework | ✅ Pass | 9.926s | |
| Web Application Security Scanner | ✅ Pass | 7.386s | |
| Penetration Testing Tool Selection | ✅ Pass | 5.078s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 7.367s
---
### adviser (qwen3:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 7.319s | |
| Text Transform Uppercase | ✅ Pass | 4.659s | |
| Count from 1 to 5 | ✅ Pass | 7.788s | |
| Math Calculation | ✅ Pass | 5.550s | |
| Basic Echo Function | ✅ Pass | 3.435s | |
| Streaming Simple Math Streaming | ✅ Pass | 6.069s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 6.911s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.180s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 5.828s | |
| Search Query Function | ✅ Pass | 2.529s | |
| Ask Advice Function | ✅ Pass | 4.512s | |
| Streaming Search Query Function Streaming | ✅ Pass | 4.623s | |
| Basic Context Memory Test | ✅ Pass | 3.887s | |
| Function Argument Memory Test | ✅ Pass | 6.185s | |
| Function Response Memory Test | ✅ Pass | 7.947s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 11.167s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 6.547s | |
| Penetration Testing Methodology | ✅ Pass | 10.852s | |
| Vulnerability Assessment Tools | ✅ Pass | 17.335s | |
| SQL Injection Attack Type | ✅ Pass | 7.411s | |
| Penetration Testing Framework | ✅ Pass | 12.124s | |
| Web Application Security Scanner | ✅ Pass | 11.661s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.973s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 7.065s
---
### reflector (qwen3:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 11.883s | |
| Text Transform Uppercase | ✅ Pass | 4.865s | |
| Count from 1 to 5 | ✅ Pass | 8.229s | |
| Math Calculation | ✅ Pass | 5.889s | |
| Basic Echo Function | ✅ Pass | 4.971s | |
| Streaming Simple Math Streaming | ✅ Pass | 4.694s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 6.196s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.101s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 4.744s | |
| Search Query Function | ✅ Pass | 4.014s | |
| Ask Advice Function | ✅ Pass | 4.346s | |
| Streaming Search Query Function Streaming | ✅ Pass | 4.364s | |
| Basic Context Memory Test | ✅ Pass | 4.967s | |
| Function Argument Memory Test | ✅ Pass | 7.148s | |
| Function Response Memory Test | ✅ Pass | 8.042s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 10.223s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.460s | |
| Penetration Testing Methodology | ✅ Pass | 9.655s | |
| Vulnerability Assessment Tools | ✅ Pass | 13.905s | |
| SQL Injection Attack Type | ✅ Pass | 5.332s | |
| Penetration Testing Framework | ✅ Pass | 13.050s | |
| Web Application Security Scanner | ✅ Pass | 11.131s | |
| Penetration Testing Tool Selection | ✅ Pass | 5.172s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 6.974s
---
### searcher (qwen3:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 7.555s | |
| Text Transform Uppercase | ✅ Pass | 4.951s | |
| Count from 1 to 5 | ✅ Pass | 4.161s | |
| Math Calculation | ✅ Pass | 3.418s | |
| Basic Echo Function | ✅ Pass | 2.865s | |
| Streaming Simple Math Streaming | ✅ Pass | 10.309s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.476s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.271s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 5.118s | |
| Search Query Function | ✅ Pass | 5.118s | |
| Ask Advice Function | ✅ Pass | 5.088s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.300s | |
| Basic Context Memory Test | ✅ Pass | 4.086s | |
| Function Argument Memory Test | ✅ Pass | 3.538s | |
| Function Response Memory Test | ✅ Pass | 4.366s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 10.349s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.474s | |
| Penetration Testing Methodology | ✅ Pass | 8.283s | |
| Vulnerability Assessment Tools | ✅ Pass | 24.191s | |
| SQL Injection Attack Type | ✅ Pass | 7.553s | |
| Penetration Testing Framework | ✅ Pass | 12.626s | |
| Web Application Security Scanner | ✅ Pass | 12.450s | |
| Penetration Testing Tool Selection | ✅ Pass | 5.361s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 6.736s
---
### enricher (qwen3:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 6.379s | |
| Text Transform Uppercase | ✅ Pass | 4.917s | |
| Count from 1 to 5 | ✅ Pass | 4.756s | |
| Math Calculation | ✅ Pass | 4.864s | |
| Basic Echo Function | ✅ Pass | 3.522s | |
| Streaming Simple Math Streaming | ✅ Pass | 7.023s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.537s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.523s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.829s | |
| Search Query Function | ✅ Pass | 4.096s | |
| Ask Advice Function | ✅ Pass | 5.520s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.941s | |
| Basic Context Memory Test | ✅ Pass | 5.351s | |
| Function Argument Memory Test | ✅ Pass | 4.132s | |
| Function Response Memory Test | ✅ Pass | 4.927s | |
| Penetration Testing Memory with Tool Call | ❌ Fail | 12.571s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.384s | |
| Penetration Testing Methodology | ✅ Pass | 16.065s | |
| Vulnerability Assessment Tools | ✅ Pass | 10.939s | |
| SQL Injection Attack Type | ✅ Pass | 7.665s | |
| Penetration Testing Framework | ✅ Pass | 15.214s | |
| Web Application Security Scanner | ✅ Pass | 10.771s | |
| Penetration Testing Tool Selection | ✅ Pass | 5.347s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 6.578s
---
### coder (qwen3:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 6.090s | |
| Text Transform Uppercase | ✅ Pass | 5.618s | |
| Count from 1 to 5 | ✅ Pass | 5.186s | |
| Math Calculation | ✅ Pass | 7.975s | |
| Basic Echo Function | ✅ Pass | 3.275s | |
| Streaming Simple Math Streaming | ✅ Pass | 11.679s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.395s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.268s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 5.099s | |
| Search Query Function | ✅ Pass | 4.003s | |
| Ask Advice Function | ✅ Pass | 4.074s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.796s | |
| Basic Context Memory Test | ✅ Pass | 4.440s | |
| Function Argument Memory Test | ✅ Pass | 15.400s | |
| Function Response Memory Test | ✅ Pass | 9.491s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 7.311s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.199s | |
| Penetration Testing Methodology | ✅ Pass | 9.748s | |
| Vulnerability Assessment Tools | ✅ Pass | 13.082s | |
| SQL Injection Attack Type | ✅ Pass | 6.824s | |
| Penetration Testing Framework | ✅ Pass | 10.664s | |
| Web Application Security Scanner | ✅ Pass | 12.258s | |
| Penetration Testing Tool Selection | ✅ Pass | 5.093s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 7.086s
---
### installer (qwen3:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 7.516s | |
| Text Transform Uppercase | ✅ Pass | 7.313s | |
| Count from 1 to 5 | ✅ Pass | 6.568s | |
| Math Calculation | ✅ Pass | 7.159s | |
| Basic Echo Function | ✅ Pass | 3.013s | |
| Streaming Simple Math Streaming | ✅ Pass | 10.104s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.982s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.514s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.809s | |
| Search Query Function | ✅ Pass | 4.973s | |
| Ask Advice Function | ✅ Pass | 5.545s | |
| Streaming Search Query Function Streaming | ✅ Pass | 4.130s | |
| Basic Context Memory Test | ✅ Pass | 4.978s | |
| Function Argument Memory Test | ✅ Pass | 5.363s | |
| Function Response Memory Test | ✅ Pass | 7.220s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 11.346s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.498s | |
| Penetration Testing Methodology | ✅ Pass | 8.142s | |
| Vulnerability Assessment Tools | ✅ Pass | 14.207s | |
| SQL Injection Attack Type | ✅ Pass | 9.205s | |
| Penetration Testing Framework | ✅ Pass | 11.698s | |
| Web Application Security Scanner | ✅ Pass | 11.854s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.745s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 6.952s
---
### pentester (qwen3:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 7.421s | |
| Text Transform Uppercase | ✅ Pass | 5.115s | |
| Count from 1 to 5 | ✅ Pass | 7.193s | |
| Math Calculation | ✅ Pass | 3.295s | |
| Basic Echo Function | ✅ Pass | 2.843s | |
| Streaming Simple Math Streaming | ✅ Pass | 8.829s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.051s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.529s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 5.107s | |
| Search Query Function | ✅ Pass | 4.109s | |
| Ask Advice Function | ✅ Pass | 6.434s | |
| Streaming Search Query Function Streaming | ✅ Pass | 4.367s | |
| Basic Context Memory Test | ✅ Pass | 3.979s | |
| Function Argument Memory Test | ✅ Pass | 5.832s | |
| Function Response Memory Test | ✅ Pass | 22.963s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 8.087s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.515s | |
| Penetration Testing Methodology | ✅ Pass | 10.706s | |
| Vulnerability Assessment Tools | ✅ Pass | 12.930s | |
| SQL Injection Attack Type | ✅ Pass | 8.345s | |
| Penetration Testing Framework | ✅ Pass | 11.880s | |
| Web Application Security Scanner | ✅ Pass | 8.811s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.864s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 7.140s
---
examples/tests/ollama-qwq-32b-fp16-tc-report.md
+537
View File
@@ -0,0 +1,537 @@
# LLM Agent Testing Report
Generated: Sat, 19 Jul 2025 20:33:51 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
| simple | qwq:32b-fp16-tc | true | 23/23 (100.00%) | 6.716s |
| simple_json | qwq:32b-fp16-tc | true | 5/5 (100.00%) | 6.216s |
| primary_agent | qwq:32b-fp16-tc | true | 23/23 (100.00%) | 9.193s |
| assistant | qwq:32b-fp16-tc | true | 23/23 (100.00%) | 8.104s |
| generator | qwq:32b-fp16-tc | true | 23/23 (100.00%) | 9.544s |
| refiner | qwq:32b-fp16-tc | true | 23/23 (100.00%) | 9.373s |
| adviser | qwq:32b-fp16-tc | true | 23/23 (100.00%) | 8.474s |
| reflector | qwq:32b-fp16-tc | true | 23/23 (100.00%) | 8.746s |
| searcher | qwq:32b-fp16-tc | true | 23/23 (100.00%) | 8.270s |
| enricher | qwq:32b-fp16-tc | true | 23/23 (100.00%) | 10.131s |
| coder | qwq:32b-fp16-tc | true | 23/23 (100.00%) | 8.886s |
| installer | qwq:32b-fp16-tc | true | 22/23 (95.65%) | 8.990s |
| pentester | qwq:32b-fp16-tc | true | 23/23 (100.00%) | 10.520s |
**Total**: 280/281 (99.64%) successful tests
**Overall average latency**: 8.864s
## Detailed Results
### simple (qwq:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 6.681s | |
| Text Transform Uppercase | ✅ Pass | 4.573s | |
| Count from 1 to 5 | ✅ Pass | 10.128s | |
| Math Calculation | ✅ Pass | 5.587s | |
| Basic Echo Function | ✅ Pass | 2.728s | |
| Streaming Simple Math Streaming | ✅ Pass | 6.202s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.603s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.625s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 4.333s | |
| Search Query Function | ✅ Pass | 3.209s | |
| Ask Advice Function | ✅ Pass | 3.321s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.065s | |
| Basic Context Memory Test | ✅ Pass | 3.660s | |
| Function Argument Memory Test | ✅ Pass | 5.600s | |
| Function Response Memory Test | ✅ Pass | 3.156s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 8.576s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.602s | |
| Penetration Testing Methodology | ✅ Pass | 15.436s | |
| Vulnerability Assessment Tools | ✅ Pass | 21.553s | |
| SQL Injection Attack Type | ✅ Pass | 7.660s | |
| Penetration Testing Framework | ✅ Pass | 15.103s | |
| Web Application Security Scanner | ✅ Pass | 10.527s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.523s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 6.716s
---
### simple_json (qwq:32b-fp16-tc)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Vulnerability Report Memory Test | ✅ Pass | 11.014s | |
| Person Information JSON | ✅ Pass | 6.958s | |
| Project Information JSON | ✅ Pass | 4.410s | |
| User Profile JSON | ✅ Pass | 3.958s | |
| Streaming Person Information JSON Streaming | ✅ Pass | 4.737s | |
**Summary**: 5/5 (100.00%) successful tests
**Average latency**: 6.216s
---
### primary_agent (qwq:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 6.427s | |
| Text Transform Uppercase | ✅ Pass | 5.094s | |
| Count from 1 to 5 | ✅ Pass | 6.045s | |
| Math Calculation | ✅ Pass | 8.572s | |
| Basic Echo Function | ✅ Pass | 5.126s | |
| Streaming Simple Math Streaming | ✅ Pass | 7.438s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.221s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 4.700s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 4.894s | |
| Search Query Function | ✅ Pass | 4.260s | |
| Ask Advice Function | ✅ Pass | 4.531s | |
| Streaming Search Query Function Streaming | ✅ Pass | 4.014s | |
| Basic Context Memory Test | ✅ Pass | 4.387s | |
| Function Argument Memory Test | ✅ Pass | 5.627s | |
| Function Response Memory Test | ✅ Pass | 6.668s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 6.791s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.742s | |
| Penetration Testing Methodology | ✅ Pass | 23.601s | |
| Vulnerability Assessment Tools | ✅ Pass | 21.807s | |
| SQL Injection Attack Type | ✅ Pass | 27.442s | |
| Penetration Testing Framework | ✅ Pass | 23.325s | |
| Web Application Security Scanner | ✅ Pass | 15.780s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.938s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 9.193s
---
### assistant (qwq:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 10.946s | |
| Text Transform Uppercase | ✅ Pass | 6.941s | |
| Count from 1 to 5 | ✅ Pass | 4.256s | |
| Math Calculation | ✅ Pass | 11.927s | |
| Basic Echo Function | ✅ Pass | 4.216s | |
| Streaming Simple Math Streaming | ✅ Pass | 10.500s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.883s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 4.938s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.806s | |
| Search Query Function | ✅ Pass | 5.634s | |
| Ask Advice Function | ✅ Pass | 4.006s | |
| Streaming Search Query Function Streaming | ✅ Pass | 5.245s | |
| Basic Context Memory Test | ✅ Pass | 3.060s | |
| Function Argument Memory Test | ✅ Pass | 4.733s | |
| Function Response Memory Test | ✅ Pass | 8.668s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 12.198s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.656s | |
| Penetration Testing Methodology | ✅ Pass | 12.831s | |
| Vulnerability Assessment Tools | ✅ Pass | 18.861s | |
| SQL Injection Attack Type | ✅ Pass | 8.588s | |
| Penetration Testing Framework | ✅ Pass | 17.076s | |
| Web Application Security Scanner | ✅ Pass | 14.477s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.937s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 8.104s
---
### generator (qwq:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 7.366s | |
| Text Transform Uppercase | ✅ Pass | 4.111s | |
| Count from 1 to 5 | ✅ Pass | 4.739s | |
| Math Calculation | ✅ Pass | 12.855s | |
| Basic Echo Function | ✅ Pass | 4.534s | |
| Streaming Simple Math Streaming | ✅ Pass | 10.861s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.929s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 4.671s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 5.297s | |
| Search Query Function | ✅ Pass | 7.993s | |
| Ask Advice Function | ✅ Pass | 3.878s | |
| Streaming Search Query Function Streaming | ✅ Pass | 5.270s | |
| Basic Context Memory Test | ✅ Pass | 3.761s | |
| Function Argument Memory Test | ✅ Pass | 4.728s | |
| Function Response Memory Test | ✅ Pass | 4.591s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 29.808s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.493s | |
| Penetration Testing Methodology | ✅ Pass | 18.866s | |
| Vulnerability Assessment Tools | ✅ Pass | 19.203s | |
| SQL Injection Attack Type | ✅ Pass | 20.241s | |
| Penetration Testing Framework | ✅ Pass | 19.454s | |
| Web Application Security Scanner | ✅ Pass | 13.553s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.303s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 9.544s
---
### refiner (qwq:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 6.994s | |
| Text Transform Uppercase | ✅ Pass | 6.657s | |
| Count from 1 to 5 | ✅ Pass | 4.197s | |
| Math Calculation | ✅ Pass | 12.493s | |
| Basic Echo Function | ✅ Pass | 4.838s | |
| Streaming Simple Math Streaming | ✅ Pass | 9.617s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.921s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 4.528s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 4.596s | |
| Search Query Function | ✅ Pass | 8.016s | |
| Ask Advice Function | ✅ Pass | 4.720s | |
| Streaming Search Query Function Streaming | ✅ Pass | 4.481s | |
| Basic Context Memory Test | ✅ Pass | 3.840s | |
| Function Argument Memory Test | ✅ Pass | 8.249s | |
| Function Response Memory Test | ✅ Pass | 24.309s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 8.445s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.230s | |
| Penetration Testing Methodology | ✅ Pass | 16.988s | |
| Vulnerability Assessment Tools | ✅ Pass | 15.847s | |
| SQL Injection Attack Type | ✅ Pass | 22.903s | |
| Penetration Testing Framework | ✅ Pass | 18.108s | |
| Web Application Security Scanner | ✅ Pass | 12.641s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.945s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 9.373s
---
### adviser (qwq:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 8.448s | |
| Text Transform Uppercase | ✅ Pass | 5.223s | |
| Count from 1 to 5 | ✅ Pass | 4.137s | |
| Math Calculation | ✅ Pass | 29.630s | |
| Basic Echo Function | ✅ Pass | 3.791s | |
| Streaming Simple Math Streaming | ✅ Pass | 9.284s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 6.324s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 5.104s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 4.474s | |
| Search Query Function | ✅ Pass | 5.012s | |
| Ask Advice Function | ✅ Pass | 3.713s | |
| Streaming Search Query Function Streaming | ✅ Pass | 5.286s | |
| Basic Context Memory Test | ✅ Pass | 4.592s | |
| Function Argument Memory Test | ✅ Pass | 9.007s | |
| Function Response Memory Test | ✅ Pass | 4.417s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 7.419s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.139s | |
| Penetration Testing Methodology | ✅ Pass | 13.577s | |
| Vulnerability Assessment Tools | ✅ Pass | 21.854s | |
| SQL Injection Attack Type | ✅ Pass | 9.491s | |
| Penetration Testing Framework | ✅ Pass | 14.146s | |
| Web Application Security Scanner | ✅ Pass | 11.518s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.300s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 8.474s
---
### reflector (qwq:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 5.056s | |
| Text Transform Uppercase | ✅ Pass | 4.968s | |
| Count from 1 to 5 | ✅ Pass | 4.893s | |
| Math Calculation | ✅ Pass | 9.789s | |
| Basic Echo Function | ✅ Pass | 4.689s | |
| Streaming Simple Math Streaming | ✅ Pass | 17.710s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 6.866s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 6.350s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.813s | |
| Search Query Function | ✅ Pass | 6.374s | |
| Ask Advice Function | ✅ Pass | 3.841s | |
| Streaming Search Query Function Streaming | ✅ Pass | 4.710s | |
| Basic Context Memory Test | ✅ Pass | 4.339s | |
| Function Argument Memory Test | ✅ Pass | 6.259s | |
| Function Response Memory Test | ✅ Pass | 13.187s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 8.633s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.170s | |
| Penetration Testing Methodology | ✅ Pass | 17.012s | |
| Vulnerability Assessment Tools | ✅ Pass | 20.805s | |
| SQL Injection Attack Type | ✅ Pass | 9.169s | |
| Penetration Testing Framework | ✅ Pass | 17.306s | |
| Web Application Security Scanner | ✅ Pass | 16.287s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.913s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 8.746s
---
### searcher (qwq:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 5.501s | |
| Text Transform Uppercase | ✅ Pass | 5.733s | |
| Count from 1 to 5 | ✅ Pass | 4.384s | |
| Math Calculation | ✅ Pass | 19.789s | |
| Basic Echo Function | ✅ Pass | 3.466s | |
| Streaming Simple Math Streaming | ✅ Pass | 11.112s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.044s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 4.030s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 5.001s | |
| Search Query Function | ✅ Pass | 7.560s | |
| Ask Advice Function | ✅ Pass | 4.992s | |
| Streaming Search Query Function Streaming | ✅ Pass | 4.649s | |
| Basic Context Memory Test | ✅ Pass | 4.280s | |
| Function Argument Memory Test | ✅ Pass | 11.166s | |
| Function Response Memory Test | ✅ Pass | 4.679s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 7.225s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.642s | |
| Penetration Testing Methodology | ✅ Pass | 18.262s | |
| Vulnerability Assessment Tools | ✅ Pass | 13.810s | |
| SQL Injection Attack Type | ✅ Pass | 10.062s | |
| Penetration Testing Framework | ✅ Pass | 17.466s | |
| Web Application Security Scanner | ✅ Pass | 13.754s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.590s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 8.270s
---
### enricher (qwq:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 5.136s | |
| Text Transform Uppercase | ✅ Pass | 6.673s | |
| Count from 1 to 5 | ✅ Pass | 4.038s | |
| Math Calculation | ✅ Pass | 18.707s | |
| Basic Echo Function | ✅ Pass | 4.421s | |
| Streaming Simple Math Streaming | ✅ Pass | 9.519s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.789s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 5.283s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 4.865s | |
| Search Query Function | ✅ Pass | 10.054s | |
| Ask Advice Function | ✅ Pass | 3.730s | |
| Streaming Search Query Function Streaming | ✅ Pass | 4.154s | |
| Basic Context Memory Test | ✅ Pass | 4.669s | |
| Function Argument Memory Test | ✅ Pass | 3.649s | |
| Function Response Memory Test | ✅ Pass | 16.702s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 6.791s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.484s | |
| Penetration Testing Methodology | ✅ Pass | 18.141s | |
| Vulnerability Assessment Tools | ✅ Pass | 22.787s | |
| SQL Injection Attack Type | ✅ Pass | 39.473s | |
| Penetration Testing Framework | ✅ Pass | 18.883s | |
| Web Application Security Scanner | ✅ Pass | 12.108s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.941s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 10.131s
---
### coder (qwq:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 6.236s | |
| Text Transform Uppercase | ✅ Pass | 5.392s | |
| Count from 1 to 5 | ✅ Pass | 5.107s | |
| Math Calculation | ✅ Pass | 8.484s | |
| Basic Echo Function | ✅ Pass | 4.541s | |
| Streaming Simple Math Streaming | ✅ Pass | 9.311s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.351s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 5.162s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 4.865s | |
| Search Query Function | ✅ Pass | 15.405s | |
| Ask Advice Function | ✅ Pass | 4.197s | |
| Streaming Search Query Function Streaming | ✅ Pass | 4.541s | |
| Basic Context Memory Test | ✅ Pass | 3.293s | |
| Function Argument Memory Test | ✅ Pass | 5.456s | |
| Function Response Memory Test | ✅ Pass | 11.370s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 15.621s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.115s | |
| Penetration Testing Methodology | ✅ Pass | 22.034s | |
| Vulnerability Assessment Tools | ✅ Pass | 19.513s | |
| SQL Injection Attack Type | ✅ Pass | 18.884s | |
| Penetration Testing Framework | ✅ Pass | 12.967s | |
| Web Application Security Scanner | ✅ Pass | 9.560s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.956s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 8.886s
---
### installer (qwq:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 7.613s | |
| Text Transform Uppercase | ✅ Pass | 5.875s | |
| Count from 1 to 5 | ✅ Pass | 3.987s | |
| Math Calculation | ✅ Pass | 23.690s | |
| Basic Echo Function | ✅ Pass | 3.616s | |
| Streaming Simple Math Streaming | ✅ Pass | 9.350s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.202s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 4.302s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.813s | |
| Search Query Function | ❌ Fail | 6.340s | expected function 'search' not found in tool calls: expected function search not found in tool calls |
| Ask Advice Function | ✅ Pass | 3.913s | |
| Streaming Search Query Function Streaming | ✅ Pass | 5.425s | |
| Basic Context Memory Test | ✅ Pass | 3.478s | |
| Function Argument Memory Test | ✅ Pass | 6.654s | |
| Function Response Memory Test | ✅ Pass | 5.056s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 8.050s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.954s | |
| Penetration Testing Methodology | ✅ Pass | 15.131s | |
| Vulnerability Assessment Tools | ✅ Pass | 20.484s | |
| SQL Injection Attack Type | ✅ Pass | 27.444s | |
| Penetration Testing Framework | ✅ Pass | 12.985s | |
| Web Application Security Scanner | ✅ Pass | 15.344s | |
| Penetration Testing Tool Selection | ✅ Pass | 5.053s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 8.990s
---
### pentester (qwq:32b-fp16-tc)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 6.272s | |
| Text Transform Uppercase | ✅ Pass | 5.369s | |
| Count from 1 to 5 | ✅ Pass | 3.969s | |
| Math Calculation | ✅ Pass | 20.641s | |
| Basic Echo Function | ✅ Pass | 3.630s | |
| Streaming Simple Math Streaming | ✅ Pass | 8.335s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.560s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 4.832s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 4.319s | |
| Search Query Function | ✅ Pass | 7.127s | |
| Ask Advice Function | ✅ Pass | 4.739s | |
| Streaming Search Query Function Streaming | ✅ Pass | 6.342s | |
| Basic Context Memory Test | ✅ Pass | 4.692s | |
| Function Argument Memory Test | ✅ Pass | 12.869s | |
| Function Response Memory Test | ✅ Pass | 26.694s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 9.736s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.734s | |
| Penetration Testing Methodology | ✅ Pass | 18.070s | |
| Vulnerability Assessment Tools | ✅ Pass | 25.093s | |
| SQL Injection Attack Type | ✅ Pass | 34.538s | |
| Penetration Testing Framework | ✅ Pass | 9.951s | |
| Web Application Security Scanner | ✅ Pass | 11.550s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.882s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 10.520s
---
examples/tests/openai-report.md
+537
View File
@@ -0,0 +1,537 @@
# LLM Agent Testing Report
Generated: Thu, 29 Jan 2026 17:38:42 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
| simple | gpt-4.1-mini | false | 23/23 (100.00%) | 0.995s |
| simple_json | gpt-4.1-mini | false | 5/5 (100.00%) | 1.027s |
| primary_agent | o4-mini | true | 23/23 (100.00%) | 2.302s |
| assistant | o4-mini | true | 23/23 (100.00%) | 2.415s |
| generator | o3 | true | 23/23 (100.00%) | 2.079s |
| refiner | o3 | true | 23/23 (100.00%) | 3.682s |
| adviser | gpt-5.2 | true | 23/23 (100.00%) | 1.193s |
| reflector | o4-mini | true | 23/23 (100.00%) | 2.591s |
| searcher | gpt-4.1-mini | false | 23/23 (100.00%) | 0.855s |
| enricher | gpt-4.1-mini | false | 23/23 (100.00%) | 0.874s |
| coder | o3 | true | 23/23 (100.00%) | 1.798s |
| installer | o4-mini | true | 23/23 (100.00%) | 1.432s |
| pentester | o4-mini | true | 23/23 (100.00%) | 1.506s |
**Total**: 281/281 (100.00%) successful tests
**Overall average latency**: 1.796s
## Detailed Results
### simple (gpt-4.1-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.368s | |
| Text Transform Uppercase | ✅ Pass | 0.724s | |
| Math Calculation | ✅ Pass | 0.571s | |
| Count from 1 to 5 | ✅ Pass | 3.392s | |
| Basic Echo Function | ✅ Pass | 0.888s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.704s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.664s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.938s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.968s | |
| Search Query Function | ✅ Pass | 0.878s | |
| Ask Advice Function | ✅ Pass | 1.225s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.808s | |
| Basic Context Memory Test | ✅ Pass | 0.777s | |
| Function Argument Memory Test | ✅ Pass | 0.666s | |
| Function Response Memory Test | ✅ Pass | 0.620s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.191s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.649s | |
| Penetration Testing Methodology | ✅ Pass | 0.931s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.566s | |
| SQL Injection Attack Type | ✅ Pass | 0.782s | |
| Penetration Testing Framework | ✅ Pass | 0.904s | |
| Web Application Security Scanner | ✅ Pass | 0.751s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.919s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.995s
---
### simple_json (gpt-4.1-mini)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Person Information JSON | ✅ Pass | 0.926s | |
| Project Information JSON | ✅ Pass | 0.859s | |
| Vulnerability Report Memory Test | ✅ Pass | 1.562s | |
| User Profile JSON | ✅ Pass | 0.883s | |
| Streaming Person Information JSON Streaming | ✅ Pass | 0.901s | |
**Summary**: 5/5 (100.00%) successful tests
**Average latency**: 1.027s
---
### primary_agent (o4-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.376s | |
| Text Transform Uppercase | ✅ Pass | 1.929s | |
| Count from 1 to 5 | ✅ Pass | 1.718s | |
| Math Calculation | ✅ Pass | 1.156s | |
| Basic Echo Function | ✅ Pass | 2.535s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.765s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.355s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.773s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Search Query Function | ✅ Pass | 1.626s | |
| JSON Response Function | ✅ Pass | 4.824s | |
| Ask Advice Function | ✅ Pass | 2.918s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.077s | |
| Basic Context Memory Test | ✅ Pass | 2.291s | |
| Function Argument Memory Test | ✅ Pass | 1.976s | |
| Function Response Memory Test | ✅ Pass | 1.534s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.414s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 2.189s | |
| Penetration Testing Methodology | ✅ Pass | 2.109s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.882s | |
| SQL Injection Attack Type | ✅ Pass | 3.378s | |
| Penetration Testing Framework | ✅ Pass | 1.863s | |
| Web Application Security Scanner | ✅ Pass | 2.422s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.821s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.302s
---
### assistant (o4-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.010s | |
| Text Transform Uppercase | ✅ Pass | 1.451s | |
| Count from 1 to 5 | ✅ Pass | 1.825s | |
| Math Calculation | ✅ Pass | 1.186s | |
| Basic Echo Function | ✅ Pass | 3.803s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.108s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.409s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.680s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 5.114s | |
| Search Query Function | ✅ Pass | 2.948s | |
| Ask Advice Function | ✅ Pass | 1.913s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.845s | |
| Basic Context Memory Test | ✅ Pass | 1.961s | |
| Function Argument Memory Test | ✅ Pass | 1.367s | |
| Function Response Memory Test | ✅ Pass | 1.961s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.599s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.941s | |
| Penetration Testing Methodology | ✅ Pass | 2.459s | |
| Vulnerability Assessment Tools | ✅ Pass | 4.370s | |
| SQL Injection Attack Type | ✅ Pass | 3.904s | |
| Penetration Testing Framework | ✅ Pass | 2.310s | |
| Web Application Security Scanner | ✅ Pass | 2.158s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.206s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.415s
---
### generator (o3)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.007s | |
| Text Transform Uppercase | ✅ Pass | 2.139s | |
| Count from 1 to 5 | ✅ Pass | 1.782s | |
| Math Calculation | ✅ Pass | 2.060s | |
| Basic Echo Function | ✅ Pass | 2.894s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.271s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.244s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.827s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.864s | |
| Search Query Function | ✅ Pass | 1.262s | |
| Ask Advice Function | ✅ Pass | 1.421s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.493s | |
| Basic Context Memory Test | ✅ Pass | 3.737s | |
| Function Argument Memory Test | ✅ Pass | 1.326s | |
| Function Response Memory Test | ✅ Pass | 1.881s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.361s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.761s | |
| Penetration Testing Methodology | ✅ Pass | 2.348s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.881s | |
| SQL Injection Attack Type | ✅ Pass | 2.790s | |
| Penetration Testing Framework | ✅ Pass | 2.106s | |
| Web Application Security Scanner | ✅ Pass | 1.683s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.678s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.079s
---
### refiner (o3)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 3.448s | |
| Text Transform Uppercase | ✅ Pass | 2.546s | |
| Count from 1 to 5 | ✅ Pass | 5.522s | |
| Math Calculation | ✅ Pass | 3.212s | |
| Basic Echo Function | ✅ Pass | 1.892s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.309s | |
| Streaming Simple Math Streaming | ✅ Pass | 4.889s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.371s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.058s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.136s | |
| Search Query Function | ✅ Pass | 10.011s | |
| Basic Context Memory Test | ✅ Pass | 4.091s | |
| Function Argument Memory Test | ✅ Pass | 1.994s | |
| Ask Advice Function | ✅ Pass | 14.955s | |
| Function Response Memory Test | ✅ Pass | 3.540s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.963s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.079s | |
| Penetration Testing Methodology | ✅ Pass | 2.170s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.531s | |
| SQL Injection Attack Type | ✅ Pass | 1.760s | |
| Penetration Testing Framework | ✅ Pass | 1.550s | |
| Web Application Security Scanner | ✅ Pass | 2.946s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.708s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.682s
---
### adviser (gpt-5.2)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.572s | |
| Text Transform Uppercase | ✅ Pass | 0.817s | |
| Count from 1 to 5 | ✅ Pass | 0.921s | |
| Math Calculation | ✅ Pass | 0.793s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.662s | |
| Basic Echo Function | ✅ Pass | 4.649s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.657s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.768s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.151s | |
| Search Query Function | ✅ Pass | 0.899s | |
| Ask Advice Function | ✅ Pass | 0.991s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.171s | |
| Basic Context Memory Test | ✅ Pass | 0.786s | |
| Function Argument Memory Test | ✅ Pass | 0.761s | |
| Function Response Memory Test | ✅ Pass | 0.904s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.756s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.567s | |
| Penetration Testing Methodology | ✅ Pass | 0.936s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.874s | |
| SQL Injection Attack Type | ✅ Pass | 0.945s | |
| Penetration Testing Framework | ✅ Pass | 0.835s | |
| Web Application Security Scanner | ✅ Pass | 0.841s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.185s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.193s
---
### reflector (o4-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.239s | |
| Text Transform Uppercase | ✅ Pass | 1.427s | |
| Count from 1 to 5 | ✅ Pass | 1.759s | |
| Math Calculation | ✅ Pass | 1.513s | |
| Basic Echo Function | ✅ Pass | 1.532s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.474s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.386s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 8.476s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.630s | |
| Search Query Function | ✅ Pass | 1.996s | |
| Ask Advice Function | ✅ Pass | 2.403s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.610s | |
| Basic Context Memory Test | ✅ Pass | 2.136s | |
| Function Argument Memory Test | ✅ Pass | 2.317s | |
| Function Response Memory Test | ✅ Pass | 1.938s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.983s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.886s | |
| Penetration Testing Methodology | ✅ Pass | 2.069s | |
| Vulnerability Assessment Tools | ✅ Pass | 3.294s | |
| SQL Injection Attack Type | ✅ Pass | 1.435s | |
| Web Application Security Scanner | ✅ Pass | 1.750s | |
| Penetration Testing Framework | ✅ Pass | 5.447s | |
| Penetration Testing Tool Selection | ✅ Pass | 7.874s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 2.591s
---
### searcher (gpt-4.1-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.365s | |
| Text Transform Uppercase | ✅ Pass | 0.696s | |
| Count from 1 to 5 | ✅ Pass | 0.633s | |
| Math Calculation | ✅ Pass | 0.560s | |
| Basic Echo Function | ✅ Pass | 0.908s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.632s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.704s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.772s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.944s | |
| Search Query Function | ✅ Pass | 0.715s | |
| Ask Advice Function | ✅ Pass | 0.996s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.768s | |
| Basic Context Memory Test | ✅ Pass | 0.698s | |
| Function Argument Memory Test | ✅ Pass | 0.701s | |
| Function Response Memory Test | ✅ Pass | 0.602s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.197s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.595s | |
| Penetration Testing Methodology | ✅ Pass | 1.064s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.512s | |
| SQL Injection Attack Type | ✅ Pass | 0.747s | |
| Penetration Testing Framework | ✅ Pass | 1.084s | |
| Web Application Security Scanner | ✅ Pass | 0.797s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.973s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.855s
---
### enricher (gpt-4.1-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.673s | |
| Text Transform Uppercase | ✅ Pass | 0.599s | |
| Count from 1 to 5 | ✅ Pass | 0.696s | |
| Math Calculation | ✅ Pass | 0.686s | |
| Basic Echo Function | ✅ Pass | 0.969s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.575s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.699s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.797s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.863s | |
| Search Query Function | ✅ Pass | 2.113s | |
| Ask Advice Function | ✅ Pass | 0.978s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.747s | |
| Basic Context Memory Test | ✅ Pass | 0.677s | |
| Function Argument Memory Test | ✅ Pass | 0.922s | |
| Function Response Memory Test | ✅ Pass | 0.611s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.221s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.708s | |
| Penetration Testing Methodology | ✅ Pass | 0.905s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.199s | |
| SQL Injection Attack Type | ✅ Pass | 0.759s | |
| Penetration Testing Framework | ✅ Pass | 0.752s | |
| Web Application Security Scanner | ✅ Pass | 0.834s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.106s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.874s
---
### coder (o3)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.846s | |
| Text Transform Uppercase | ✅ Pass | 1.455s | |
| Count from 1 to 5 | ✅ Pass | 1.774s | |
| Math Calculation | ✅ Pass | 1.376s | |
| Basic Echo Function | ✅ Pass | 1.224s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.248s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.365s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.026s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.378s | |
| Search Query Function | ✅ Pass | 2.455s | |
| Ask Advice Function | ✅ Pass | 1.263s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.306s | |
| Basic Context Memory Test | ✅ Pass | 2.486s | |
| Function Argument Memory Test | ✅ Pass | 1.768s | |
| Function Response Memory Test | ✅ Pass | 2.899s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.468s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.163s | |
| Penetration Testing Methodology | ✅ Pass | 1.939s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.276s | |
| SQL Injection Attack Type | ✅ Pass | 3.775s | |
| Penetration Testing Framework | ✅ Pass | 1.902s | |
| Web Application Security Scanner | ✅ Pass | 1.195s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.757s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.798s
---
### installer (o4-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.259s | |
| Text Transform Uppercase | ✅ Pass | 1.103s | |
| Count from 1 to 5 | ✅ Pass | 1.479s | |
| Math Calculation | ✅ Pass | 1.098s | |
| Basic Echo Function | ✅ Pass | 1.458s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.221s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.150s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.093s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.167s | |
| Search Query Function | ✅ Pass | 1.190s | |
| Ask Advice Function | ✅ Pass | 1.344s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.060s | |
| Basic Context Memory Test | ✅ Pass | 1.780s | |
| Function Argument Memory Test | ✅ Pass | 1.371s | |
| Function Response Memory Test | ✅ Pass | 1.473s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.580s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.675s | |
| Penetration Testing Methodology | ✅ Pass | 1.638s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.012s | |
| SQL Injection Attack Type | ✅ Pass | 1.645s | |
| Penetration Testing Framework | ✅ Pass | 1.624s | |
| Web Application Security Scanner | ✅ Pass | 1.865s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.639s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.432s
---
### pentester (o4-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.229s | |
| Text Transform Uppercase | ✅ Pass | 1.321s | |
| Count from 1 to 5 | ✅ Pass | 1.642s | |
| Math Calculation | ✅ Pass | 1.335s | |
| Basic Echo Function | ✅ Pass | 1.047s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.165s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.275s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.970s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.298s | |
| Search Query Function | ✅ Pass | 1.158s | |
| Ask Advice Function | ✅ Pass | 1.220s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.040s | |
| Basic Context Memory Test | ✅ Pass | 1.583s | |
| Function Argument Memory Test | ✅ Pass | 1.313s | |
| Function Response Memory Test | ✅ Pass | 1.448s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.786s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.637s | |
| Penetration Testing Methodology | ✅ Pass | 1.640s | |
| Vulnerability Assessment Tools | ✅ Pass | 4.050s | |
| SQL Injection Attack Type | ✅ Pass | 1.678s | |
| Penetration Testing Framework | ✅ Pass | 1.841s | |
| Web Application Security Scanner | ✅ Pass | 1.649s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.307s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.506s
---
examples/tests/openrouter-report.md
+537
View File
@@ -0,0 +1,537 @@
# LLM Agent Testing Report
Generated: Tue, 30 Sep 2025 18:46:00 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
| simple | openai/gpt-4.1-mini | false | 23/23 (100.00%) | 1.594s |
| simple_json | openai/gpt-4.1-mini | false | 5/5 (100.00%) | 1.682s |
| primary_agent | openai/gpt-5 | true | 23/23 (100.00%) | 7.285s |
| assistant | openai/gpt-5 | true | 23/23 (100.00%) | 8.135s |
| generator | anthropic/claude-sonnet-4.5 | true | 23/23 (100.00%) | 4.525s |
| refiner | google/gemini-2.5-pro | true | 21/23 (91.30%) | 5.576s |
| adviser | google/gemini-2.5-pro | true | 22/23 (95.65%) | 5.532s |
| reflector | openai/gpt-4.1-mini | false | 23/23 (100.00%) | 1.556s |
| searcher | x-ai/grok-3-mini | true | 22/23 (95.65%) | 4.511s |
| enricher | openai/gpt-4.1-mini | true | 23/23 (100.00%) | 1.597s |
| coder | anthropic/claude-sonnet-4.5 | true | 23/23 (100.00%) | 4.445s |
| installer | google/gemini-2.5-flash | true | 23/23 (100.00%) | 3.276s |
| pentester | moonshotai/kimi-k2-0905 | true | 22/23 (95.65%) | 2.301s |
**Total**: 276/281 (98.22%) successful tests
**Overall average latency**: 4.150s
## Detailed Results
### simple (openai/gpt-4.1-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Text Transform Uppercase | ✅ Pass | 2.727s | |
| Simple Math | ✅ Pass | 2.809s | |
| Count from 1 to 5 | ✅ Pass | 3.158s | |
| Math Calculation | ✅ Pass | 1.255s | |
| Basic Echo Function | ✅ Pass | 1.112s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.109s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.179s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.270s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.334s | |
| Search Query Function | ✅ Pass | 1.375s | |
| Ask Advice Function | ✅ Pass | 1.433s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.436s | |
| Basic Context Memory Test | ✅ Pass | 1.293s | |
| Function Argument Memory Test | ✅ Pass | 1.326s | |
| Function Response Memory Test | ✅ Pass | 1.378s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.802s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.454s | |
| Penetration Testing Methodology | ✅ Pass | 1.216s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.509s | |
| SQL Injection Attack Type | ✅ Pass | 2.427s | |
| Penetration Testing Framework | ✅ Pass | 1.526s | |
| Web Application Security Scanner | ✅ Pass | 1.093s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.419s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.594s
---
### simple_json (openai/gpt-4.1-mini)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Project Information JSON | ✅ Pass | 1.574s | |
| User Profile JSON | ✅ Pass | 1.531s | |
| Person Information JSON | ✅ Pass | 1.706s | |
| Vulnerability Report Memory Test | ✅ Pass | 2.108s | |
| Streaming Person Information JSON Streaming | ✅ Pass | 1.488s | |
**Summary**: 5/5 (100.00%) successful tests
**Average latency**: 1.682s
---
### primary_agent (openai/gpt-5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Text Transform Uppercase | ✅ Pass | 5.678s | |
| Simple Math | ✅ Pass | 6.979s | |
| Math Calculation | ✅ Pass | 4.546s | |
| Count from 1 to 5 | ✅ Pass | 8.078s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.289s | |
| Basic Echo Function | ✅ Pass | 7.959s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.885s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 12.785s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 5.854s | |
| Ask Advice Function | ✅ Pass | 2.945s | |
| Basic Context Memory Test | ✅ Pass | 6.477s | |
| Streaming Search Query Function Streaming | ✅ Pass | 10.439s | |
| Function Argument Memory Test | ✅ Pass | 5.706s | |
| Search Query Function | ✅ Pass | 17.551s | |
| Function Response Memory Test | ✅ Pass | 6.284s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 5.072s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 12.418s | |
| Penetration Testing Methodology | ✅ Pass | 8.316s | |
| SQL Injection Attack Type | ✅ Pass | 5.413s | |
| Vulnerability Assessment Tools | ✅ Pass | 11.698s | |
| Penetration Testing Framework | ✅ Pass | 5.109s | |
| Web Application Security Scanner | ✅ Pass | 4.251s | |
| Penetration Testing Tool Selection | ✅ Pass | 5.821s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 7.285s
---
### assistant (openai/gpt-5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Text Transform Uppercase | ✅ Pass | 4.176s | |
| Simple Math | ✅ Pass | 4.241s | |
| Count from 1 to 5 | ✅ Pass | 4.418s | |
| Math Calculation | ✅ Pass | 2.466s | |
| Streaming Simple Math Streaming | ✅ Pass | 4.288s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.402s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 4.997s | |
| Basic Echo Function | ✅ Pass | 14.115s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Ask Advice Function | ✅ Pass | 3.039s | |
| Search Query Function | ✅ Pass | 9.098s | |
| Function Argument Memory Test | ✅ Pass | 3.562s | |
| Basic Context Memory Test | ✅ Pass | 8.180s | |
| Function Response Memory Test | ✅ Pass | 4.814s | |
| Streaming Search Query Function Streaming | ✅ Pass | 15.423s | |
| JSON Response Function | ✅ Pass | 24.602s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 7.121s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 6.563s | |
| SQL Injection Attack Type | ✅ Pass | 7.029s | |
| Penetration Testing Methodology | ✅ Pass | 16.605s | |
| Vulnerability Assessment Tools | ✅ Pass | 17.711s | |
| Web Application Security Scanner | ✅ Pass | 3.749s | |
| Penetration Testing Framework | ✅ Pass | 7.171s | |
| Penetration Testing Tool Selection | ✅ Pass | 9.317s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 8.135s
---
### generator (anthropic/claude-sonnet-4.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 4.796s | |
| Text Transform Uppercase | ✅ Pass | 4.900s | |
| Count from 1 to 5 | ✅ Pass | 3.211s | |
| Math Calculation | ✅ Pass | 2.543s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.894s | |
| Basic Echo Function | ✅ Pass | 3.969s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.810s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.255s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.649s | |
| Search Query Function | ✅ Pass | 3.659s | |
| Ask Advice Function | ✅ Pass | 3.011s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.007s | |
| Basic Context Memory Test | ✅ Pass | 2.584s | |
| Function Argument Memory Test | ✅ Pass | 3.795s | |
| Function Response Memory Test | ✅ Pass | 3.613s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.593s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.289s | |
| Penetration Testing Methodology | ✅ Pass | 11.070s | |
| Vulnerability Assessment Tools | ✅ Pass | 9.953s | |
| SQL Injection Attack Type | ✅ Pass | 4.623s | |
| Web Application Security Scanner | ✅ Pass | 6.242s | |
| Penetration Testing Framework | ✅ Pass | 9.207s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.393s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 4.525s
---
### refiner (google/gemini-2.5-pro)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 4.765s | |
| Text Transform Uppercase | ✅ Pass | 8.021s | |
| Count from 1 to 5 | ✅ Pass | 5.828s | |
| Math Calculation | ✅ Pass | 3.337s | |
| Basic Echo Function | ✅ Pass | 3.749s | |
| Streaming Simple Math Streaming | ✅ Pass | 4.356s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 2.657s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.911s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.989s | |
| Search Query Function | ✅ Pass | 4.489s | |
| Ask Advice Function | ✅ Pass | 3.256s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.329s | |
| Function Argument Memory Test | ❌ Fail | 1.329s | expected text 'Go programming language' not found |
| Basic Context Memory Test | ✅ Pass | 4.987s | |
| Function Response Memory Test | ❌ Fail | 1.624s | expected text '22' not found |
| Penetration Testing Memory with Tool Call | ✅ Pass | 6.209s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.357s | |
| Penetration Testing Methodology | ✅ Pass | 9.174s | |
| Vulnerability Assessment Tools | ✅ Pass | 11.362s | |
| SQL Injection Attack Type | ✅ Pass | 7.982s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.862s | |
| Web Application Security Scanner | ✅ Pass | 11.705s | |
| Penetration Testing Framework | ✅ Pass | 15.968s | |
**Summary**: 21/23 (91.30%) successful tests
**Average latency**: 5.576s
---
### adviser (google/gemini-2.5-pro)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 4.749s | |
| Count from 1 to 5 | ✅ Pass | 4.983s | |
| Text Transform Uppercase | ✅ Pass | 7.715s | |
| Math Calculation | ✅ Pass | 3.160s | |
| Basic Echo Function | ✅ Pass | 3.491s | |
| Streaming Simple Math Streaming | ✅ Pass | 3.304s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.330s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.641s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.319s | |
| Search Query Function | ✅ Pass | 3.352s | |
| Ask Advice Function | ✅ Pass | 2.876s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.286s | |
| Basic Context Memory Test | ✅ Pass | 5.184s | |
| Function Argument Memory Test | ✅ Pass | 3.338s | |
| Function Response Memory Test | ❌ Fail | 1.962s | expected text '22' not found |
| Penetration Testing Memory with Tool Call | ✅ Pass | 5.316s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.781s | |
| Penetration Testing Methodology | ✅ Pass | 10.426s | |
| Vulnerability Assessment Tools | ✅ Pass | 12.932s | |
| SQL Injection Attack Type | ✅ Pass | 6.701s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.242s | |
| Penetration Testing Framework | ✅ Pass | 13.500s | |
| Web Application Security Scanner | ✅ Pass | 12.631s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 5.532s
---
### reflector (openai/gpt-4.1-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.664s | |
| Text Transform Uppercase | ✅ Pass | 3.352s | |
| Count from 1 to 5 | ✅ Pass | 1.470s | |
| Math Calculation | ✅ Pass | 1.184s | |
| Basic Echo Function | ✅ Pass | 1.459s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.206s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.110s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.144s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.294s | |
| Search Query Function | ✅ Pass | 1.555s | |
| Ask Advice Function | ✅ Pass | 1.328s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.209s | |
| Basic Context Memory Test | ✅ Pass | 1.465s | |
| Function Argument Memory Test | ✅ Pass | 1.186s | |
| Function Response Memory Test | ✅ Pass | 1.476s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.031s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.505s | |
| Penetration Testing Methodology | ✅ Pass | 1.356s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.687s | |
| SQL Injection Attack Type | ✅ Pass | 1.316s | |
| Penetration Testing Framework | ✅ Pass | 1.093s | |
| Web Application Security Scanner | ✅ Pass | 1.298s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.387s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.556s
---
### searcher (x-ai/grok-3-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 4.840s | |
| Text Transform Uppercase | ✅ Pass | 5.601s | |
| Count from 1 to 5 | ✅ Pass | 4.014s | |
| Math Calculation | ✅ Pass | 3.175s | |
| Basic Echo Function | ✅ Pass | 4.012s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.994s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.596s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.705s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 5.163s | |
| Search Query Function | ✅ Pass | 3.663s | |
| Ask Advice Function | ✅ Pass | 4.934s | |
| Streaming Search Query Function Streaming | ✅ Pass | 5.816s | |
| Basic Context Memory Test | ✅ Pass | 3.479s | |
| Function Argument Memory Test | ✅ Pass | 3.226s | |
| Function Response Memory Test | ✅ Pass | 3.040s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 5.192s | |
| Penetration Testing Methodology | ✅ Pass | 4.935s | |
| Cybersecurity Workflow Memory Test | ❌ Fail | 8.973s | expected text 'example\.com' not found |
| Vulnerability Assessment Tools | ✅ Pass | 6.358s | |
| SQL Injection Attack Type | ✅ Pass | 3.042s | |
| Penetration Testing Framework | ✅ Pass | 5.377s | |
| Web Application Security Scanner | ✅ Pass | 4.338s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.267s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 4.511s
---
### enricher (openai/gpt-4.1-mini)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.796s | |
| Text Transform Uppercase | ✅ Pass | 3.117s | |
| Count from 1 to 5 | ✅ Pass | 1.902s | |
| Math Calculation | ✅ Pass | 0.887s | |
| Basic Echo Function | ✅ Pass | 1.260s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.943s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.273s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.393s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.519s | |
| Search Query Function | ✅ Pass | 1.304s | |
| Ask Advice Function | ✅ Pass | 1.661s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.592s | |
| Basic Context Memory Test | ✅ Pass | 1.266s | |
| Function Argument Memory Test | ✅ Pass | 1.239s | |
| Function Response Memory Test | ✅ Pass | 1.617s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.076s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.278s | |
| Penetration Testing Methodology | ✅ Pass | 1.934s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.300s | |
| SQL Injection Attack Type | ✅ Pass | 1.211s | |
| Penetration Testing Framework | ✅ Pass | 1.614s | |
| Web Application Security Scanner | ✅ Pass | 1.195s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.334s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.597s
---
### coder (anthropic/claude-sonnet-4.5)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 5.233s | |
| Text Transform Uppercase | ✅ Pass | 5.161s | |
| Count from 1 to 5 | ✅ Pass | 3.227s | |
| Math Calculation | ✅ Pass | 2.882s | |
| Basic Echo Function | ✅ Pass | 3.143s | |
| Streaming Simple Math Streaming | ✅ Pass | 2.506s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 3.003s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.763s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 3.419s | |
| Search Query Function | ✅ Pass | 3.017s | |
| Ask Advice Function | ✅ Pass | 2.999s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.992s | |
| Basic Context Memory Test | ✅ Pass | 3.126s | |
| Function Argument Memory Test | ✅ Pass | 3.670s | |
| Function Response Memory Test | ✅ Pass | 3.248s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.631s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.598s | |
| Penetration Testing Methodology | ✅ Pass | 11.220s | |
| Vulnerability Assessment Tools | ✅ Pass | 9.139s | |
| SQL Injection Attack Type | ✅ Pass | 4.317s | |
| Penetration Testing Framework | ✅ Pass | 7.797s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.140s | |
| Web Application Security Scanner | ✅ Pass | 8.004s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 4.445s
---
### installer (google/gemini-2.5-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 3.212s | |
| Text Transform Uppercase | ✅ Pass | 2.828s | |
| Count from 1 to 5 | ✅ Pass | 0.800s | |
| Math Calculation | ✅ Pass | 1.529s | |
| Basic Echo Function | ✅ Pass | 1.841s | |
| Streaming Simple Math Streaming | ✅ Pass | 3.011s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.480s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.747s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.678s | |
| Search Query Function | ✅ Pass | 1.535s | |
| Ask Advice Function | ✅ Pass | 2.439s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.017s | |
| Basic Context Memory Test | ✅ Pass | 2.790s | |
| Function Response Memory Test | ✅ Pass | 0.868s | |
| Function Argument Memory Test | ✅ Pass | 3.503s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.933s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.969s | |
| Penetration Testing Methodology | ✅ Pass | 6.046s | |
| Vulnerability Assessment Tools | ✅ Pass | 8.005s | |
| SQL Injection Attack Type | ✅ Pass | 2.731s | |
| Web Application Security Scanner | ✅ Pass | 4.804s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.752s | |
| Penetration Testing Framework | ✅ Pass | 13.820s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 3.276s
---
### pentester (moonshotai/kimi-k2-0905)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 2.485s | |
| Text Transform Uppercase | ✅ Pass | 1.845s | |
| Count from 1 to 5 | ✅ Pass | 1.481s | |
| Math Calculation | ✅ Pass | 1.625s | |
| Basic Echo Function | ✅ Pass | 1.611s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.693s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.843s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.580s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.278s | |
| Search Query Function | ✅ Pass | 1.168s | |
| Search Query Function Streaming | ❌ Fail | 0.747s | streaming tool call func returned an error: tool call name is required |
| Ask Advice Function | ✅ Pass | 4.458s | |
| Basic Context Memory Test | ✅ Pass | 1.566s | |
| Function Argument Memory Test | ✅ Pass | 1.148s | |
| Function Response Memory Test | ✅ Pass | 0.811s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.292s | |
| Penetration Testing Methodology | ✅ Pass | 2.511s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 11.318s | |
| Vulnerability Assessment Tools | ✅ Pass | 0.730s | |
| SQL Injection Attack Type | ✅ Pass | 3.724s | |
| Penetration Testing Framework | ✅ Pass | 0.782s | |
| Web Application Security Scanner | ✅ Pass | 3.533s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.674s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 2.301s
---
examples/tests/qwen-report.md
+537
View File
@@ -0,0 +1,537 @@
# LLM Agent Testing Report
Generated: Thu, 28 May 2026 16:06:51 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
| simple | qwen3.5-flash | false | 23/23 (100.00%) | 1.194s |
| simple_json | qwen3.5-flash | false | 4/5 (80.00%) | 0.945s |
| primary_agent | qwen3.6-plus | true | 23/23 (100.00%) | 6.079s |
| assistant | qwen3.6-plus | true | 23/23 (100.00%) | 5.512s |
| generator | qwen3.7-max | true | 23/23 (100.00%) | 5.172s |
| refiner | qwen3.7-max | true | 23/23 (100.00%) | 5.455s |
| adviser | qwen3.7-max | true | 23/23 (100.00%) | 4.750s |
| reflector | qwen3.5-flash | true | 23/23 (100.00%) | 1.222s |
| searcher | qwen3.5-flash | true | 23/23 (100.00%) | 1.083s |
| enricher | qwen3.5-flash | true | 23/23 (100.00%) | 0.972s |
| coder | qwen3-coder-plus | true | 23/23 (100.00%) | 1.702s |
| installer | qwen3-coder-flash | true | 23/23 (100.00%) | 1.124s |
| pentester | qwen3.6-plus | true | 23/23 (100.00%) | 9.207s |
**Total**: 280/281 (99.64%) successful tests
**Overall average latency**: 3.575s
## Detailed Results
### simple (qwen3.5-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.515s | |
| Text Transform Uppercase | ✅ Pass | 1.075s | |
| Count from 1 to 5 | ✅ Pass | 1.066s | |
| Math Calculation | ✅ Pass | 1.122s | |
| Basic Echo Function | ✅ Pass | 1.252s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.603s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.569s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.800s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.795s | |
| Search Query Function | ✅ Pass | 1.148s | |
| Ask Advice Function | ✅ Pass | 1.347s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.834s | |
| Basic Context Memory Test | ✅ Pass | 0.655s | |
| Function Argument Memory Test | ✅ Pass | 1.019s | |
| Function Response Memory Test | ✅ Pass | 1.023s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.142s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.996s | |
| Penetration Testing Methodology | ✅ Pass | 2.497s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.849s | |
| SQL Injection Attack Type | ✅ Pass | 0.552s | |
| Penetration Testing Framework | ✅ Pass | 1.971s | |
| Web Application Security Scanner | ✅ Pass | 1.461s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.152s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.194s
---
### simple_json (qwen3.5-flash)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Project Information JSON | ✅ Pass | 0.751s | |
| User Profile JSON | ✅ Pass | 0.767s | |
| Person Information JSON | ✅ Pass | 1.137s | |
| Vulnerability Report Memory Test | ❌ Fail | 1.318s | got map\[string\]interface \{\}\{"$schema":"http://json\-schema\.org/draft\-07/schema\#", "properties":map\[string\]interface \{\}\{"open\_ports":\... |
| Streaming Person Information JSON Streaming | ✅ Pass | 0.747s | |
**Summary**: 4/5 (80.00%) successful tests
**Average latency**: 0.945s
---
### primary_agent (qwen3.6-plus)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 5.447s | |
| Text Transform Uppercase | ✅ Pass | 5.747s | |
| Count from 1 to 5 | ✅ Pass | 6.848s | |
| Math Calculation | ✅ Pass | 3.912s | |
| Basic Echo Function | ✅ Pass | 2.791s | |
| Streaming Simple Math Streaming | ✅ Pass | 4.355s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.205s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.995s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 4.066s | |
| Search Query Function | ✅ Pass | 2.232s | |
| Ask Advice Function | ✅ Pass | 2.860s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.094s | |
| Basic Context Memory Test | ✅ Pass | 4.240s | |
| Function Argument Memory Test | ✅ Pass | 2.635s | |
| Function Response Memory Test | ✅ Pass | 9.775s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 7.318s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.524s | |
| Penetration Testing Methodology | ✅ Pass | 13.685s | |
| Vulnerability Assessment Tools | ✅ Pass | 12.372s | |
| SQL Injection Attack Type | ✅ Pass | 11.677s | |
| Penetration Testing Framework | ✅ Pass | 12.500s | |
| Web Application Security Scanner | ✅ Pass | 9.014s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.509s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 6.079s
---
### assistant (qwen3.6-plus)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 4.896s | |
| Text Transform Uppercase | ✅ Pass | 4.750s | |
| Count from 1 to 5 | ✅ Pass | 5.467s | |
| Math Calculation | ✅ Pass | 3.192s | |
| Basic Echo Function | ✅ Pass | 2.627s | |
| Streaming Simple Math Streaming | ✅ Pass | 3.657s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.401s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 5.018s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.385s | |
| Search Query Function | ✅ Pass | 2.864s | |
| Ask Advice Function | ✅ Pass | 3.599s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.866s | |
| Basic Context Memory Test | ✅ Pass | 4.909s | |
| Function Argument Memory Test | ✅ Pass | 3.381s | |
| Function Response Memory Test | ✅ Pass | 4.146s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 5.778s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 5.842s | |
| Penetration Testing Methodology | ✅ Pass | 9.568s | |
| SQL Injection Attack Type | ✅ Pass | 6.860s | |
| Vulnerability Assessment Tools | ✅ Pass | 18.097s | |
| Penetration Testing Framework | ✅ Pass | 10.065s | |
| Web Application Security Scanner | ✅ Pass | 9.724s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.665s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 5.512s
---
### generator (qwen3.7-max)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 5.569s | |
| Text Transform Uppercase | ✅ Pass | 4.279s | |
| Math Calculation | ✅ Pass | 2.508s | |
| Basic Echo Function | ✅ Pass | 2.450s | |
| Count from 1 to 5 | ✅ Pass | 13.152s | |
| Streaming Simple Math Streaming | ✅ Pass | 3.691s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.566s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.392s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.320s | |
| Search Query Function | ✅ Pass | 3.027s | |
| Ask Advice Function | ✅ Pass | 4.213s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.036s | |
| Basic Context Memory Test | ✅ Pass | 7.051s | |
| Function Argument Memory Test | ✅ Pass | 3.966s | |
| Function Response Memory Test | ✅ Pass | 7.543s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.481s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.203s | |
| Penetration Testing Methodology | ✅ Pass | 6.768s | |
| SQL Injection Attack Type | ✅ Pass | 4.673s | |
| Vulnerability Assessment Tools | ✅ Pass | 18.743s | |
| Penetration Testing Framework | ✅ Pass | 5.278s | |
| Web Application Security Scanner | ✅ Pass | 3.018s | |
| Penetration Testing Tool Selection | ✅ Pass | 4.024s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 5.172s
---
### refiner (qwen3.7-max)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 4.724s | |
| Text Transform Uppercase | ✅ Pass | 4.732s | |
| Count from 1 to 5 | ✅ Pass | 8.230s | |
| Math Calculation | ✅ Pass | 2.684s | |
| Basic Echo Function | ✅ Pass | 2.505s | |
| Streaming Simple Math Streaming | ✅ Pass | 5.229s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 6.093s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.982s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.442s | |
| Search Query Function | ✅ Pass | 2.913s | |
| Ask Advice Function | ✅ Pass | 3.594s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.382s | |
| Basic Context Memory Test | ✅ Pass | 4.152s | |
| Function Argument Memory Test | ✅ Pass | 3.718s | |
| Function Response Memory Test | ✅ Pass | 5.162s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 4.590s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.049s | |
| Penetration Testing Methodology | ✅ Pass | 5.504s | |
| SQL Injection Attack Type | ✅ Pass | 5.041s | |
| Vulnerability Assessment Tools | ✅ Pass | 20.343s | |
| Penetration Testing Framework | ✅ Pass | 10.747s | |
| Web Application Security Scanner | ✅ Pass | 10.091s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.551s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 5.455s
---
### adviser (qwen3.7-max)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 3.761s | |
| Text Transform Uppercase | ✅ Pass | 6.663s | |
| Count from 1 to 5 | ✅ Pass | 8.046s | |
| Math Calculation | ✅ Pass | 2.987s | |
| Basic Echo Function | ✅ Pass | 2.306s | |
| Streaming Simple Math Streaming | ✅ Pass | 3.074s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 2.033s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 7.517s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.833s | |
| Search Query Function | ✅ Pass | 2.622s | |
| Ask Advice Function | ✅ Pass | 3.735s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.859s | |
| Basic Context Memory Test | ✅ Pass | 4.274s | |
| Function Argument Memory Test | ✅ Pass | 5.488s | |
| Function Response Memory Test | ✅ Pass | 7.424s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 2.711s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 4.780s | |
| Penetration Testing Methodology | ✅ Pass | 6.058s | |
| Vulnerability Assessment Tools | ✅ Pass | 13.545s | |
| SQL Injection Attack Type | ✅ Pass | 3.451s | |
| Penetration Testing Framework | ✅ Pass | 5.348s | |
| Web Application Security Scanner | ✅ Pass | 6.648s | |
| Penetration Testing Tool Selection | ✅ Pass | 3.077s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 4.750s
---
### reflector (qwen3.5-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.664s | |
| Text Transform Uppercase | ✅ Pass | 0.999s | |
| Count from 1 to 5 | ✅ Pass | 0.706s | |
| Math Calculation | ✅ Pass | 0.808s | |
| Basic Echo Function | ✅ Pass | 1.256s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.597s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.636s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.226s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.959s | |
| Search Query Function | ✅ Pass | 1.138s | |
| Ask Advice Function | ✅ Pass | 1.422s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.120s | |
| Basic Context Memory Test | ✅ Pass | 0.691s | |
| Function Argument Memory Test | ✅ Pass | 0.528s | |
| Function Response Memory Test | ✅ Pass | 0.553s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.283s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.122s | |
| Penetration Testing Methodology | ✅ Pass | 2.639s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.019s | |
| SQL Injection Attack Type | ✅ Pass | 1.036s | |
| Penetration Testing Framework | ✅ Pass | 1.655s | |
| Web Application Security Scanner | ✅ Pass | 3.147s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.902s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.222s
---
### searcher (qwen3.5-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.541s | |
| Text Transform Uppercase | ✅ Pass | 0.541s | |
| Count from 1 to 5 | ✅ Pass | 0.660s | |
| Math Calculation | ✅ Pass | 0.556s | |
| Basic Echo Function | ✅ Pass | 0.701s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.537s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.012s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.886s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.744s | |
| Search Query Function | ✅ Pass | 1.350s | |
| Ask Advice Function | ✅ Pass | 1.986s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.726s | |
| Basic Context Memory Test | ✅ Pass | 1.089s | |
| Function Argument Memory Test | ✅ Pass | 0.558s | |
| Function Response Memory Test | ✅ Pass | 0.620s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.284s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.607s | |
| Penetration Testing Methodology | ✅ Pass | 2.167s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.673s | |
| SQL Injection Attack Type | ✅ Pass | 1.089s | |
| Penetration Testing Framework | ✅ Pass | 1.859s | |
| Web Application Security Scanner | ✅ Pass | 1.590s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.127s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.083s
---
### enricher (qwen3.5-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.582s | |
| Text Transform Uppercase | ✅ Pass | 1.107s | |
| Count from 1 to 5 | ✅ Pass | 0.693s | |
| Math Calculation | ✅ Pass | 0.516s | |
| Basic Echo Function | ✅ Pass | 0.700s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.523s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.544s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.772s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.830s | |
| Search Query Function | ✅ Pass | 1.114s | |
| Ask Advice Function | ✅ Pass | 0.989s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.729s | |
| Basic Context Memory Test | ✅ Pass | 0.571s | |
| Function Argument Memory Test | ✅ Pass | 0.820s | |
| Function Response Memory Test | ✅ Pass | 0.683s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.762s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.029s | |
| Penetration Testing Methodology | ✅ Pass | 2.050s | |
| Vulnerability Assessment Tools | ✅ Pass | 1.990s | |
| SQL Injection Attack Type | ✅ Pass | 0.651s | |
| Penetration Testing Framework | ✅ Pass | 1.320s | |
| Web Application Security Scanner | ✅ Pass | 1.317s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.060s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 0.972s
---
### coder (qwen3-coder-plus)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 1.889s | |
| Text Transform Uppercase | ✅ Pass | 1.323s | |
| Count from 1 to 5 | ✅ Pass | 0.900s | |
| Math Calculation | ✅ Pass | 1.026s | |
| Basic Echo Function | ✅ Pass | 1.016s | |
| Streaming Simple Math Streaming | ✅ Pass | 1.920s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 1.298s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.987s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.585s | |
| Search Query Function | ✅ Pass | 1.567s | |
| Ask Advice Function | ✅ Pass | 1.272s | |
| Streaming Search Query Function Streaming | ✅ Pass | 0.972s | |
| Basic Context Memory Test | ✅ Pass | 1.434s | |
| Function Argument Memory Test | ✅ Pass | 0.764s | |
| Function Response Memory Test | ✅ Pass | 1.068s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.771s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.781s | |
| Penetration Testing Methodology | ✅ Pass | 3.191s | |
| Vulnerability Assessment Tools | ✅ Pass | 3.556s | |
| SQL Injection Attack Type | ✅ Pass | 0.965s | |
| Penetration Testing Framework | ✅ Pass | 5.216s | |
| Web Application Security Scanner | ✅ Pass | 3.193s | |
| Penetration Testing Tool Selection | ✅ Pass | 1.441s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.702s
---
### installer (qwen3-coder-flash)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 0.558s | |
| Text Transform Uppercase | ✅ Pass | 0.527s | |
| Count from 1 to 5 | ✅ Pass | 0.651s | |
| Math Calculation | ✅ Pass | 0.965s | |
| Basic Echo Function | ✅ Pass | 1.248s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.919s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.591s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.761s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 0.869s | |
| Search Query Function | ✅ Pass | 0.856s | |
| Ask Advice Function | ✅ Pass | 1.283s | |
| Streaming Search Query Function Streaming | ✅ Pass | 1.259s | |
| Basic Context Memory Test | ✅ Pass | 0.981s | |
| Function Argument Memory Test | ✅ Pass | 0.649s | |
| Function Response Memory Test | ✅ Pass | 1.007s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 1.940s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.070s | |
| Penetration Testing Methodology | ✅ Pass | 2.822s | |
| Vulnerability Assessment Tools | ✅ Pass | 2.937s | |
| SQL Injection Attack Type | ✅ Pass | 0.620s | |
| Penetration Testing Framework | ✅ Pass | 1.331s | |
| Web Application Security Scanner | ✅ Pass | 1.018s | |
| Penetration Testing Tool Selection | ✅ Pass | 0.972s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 1.124s
---
### pentester (qwen3.6-plus)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 3.761s | |
| Text Transform Uppercase | ✅ Pass | 4.980s | |
| Count from 1 to 5 | ✅ Pass | 6.066s | |
| Math Calculation | ✅ Pass | 5.125s | |
| Basic Echo Function | ✅ Pass | 4.145s | |
| Streaming Simple Math Streaming | ✅ Pass | 4.829s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.837s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 3.264s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.894s | |
| Search Query Function | ✅ Pass | 3.922s | |
| Ask Advice Function | ✅ Pass | 3.311s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.728s | |
| Basic Context Memory Test | ✅ Pass | 4.075s | |
| Function Argument Memory Test | ✅ Pass | 5.545s | |
| Function Response Memory Test | ✅ Pass | 10.448s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 5.993s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 3.695s | |
| Penetration Testing Methodology | ✅ Pass | 9.881s | |
| Vulnerability Assessment Tools | ✅ Pass | 10.935s | |
| Penetration Testing Framework | ✅ Pass | 12.705s | |
| Web Application Security Scanner | ✅ Pass | 8.109s | |
| Penetration Testing Tool Selection | ✅ Pass | 2.886s | |
| SQL Injection Attack Type | ✅ Pass | 86.622s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 9.207s
---
examples/tests/vllm-qwen332b-fp16-report.md
+537
View File
@@ -0,0 +1,537 @@
# LLM Agent Testing Report
Generated: Sun, 15 Mar 2026 15:53:05 UTC
## Overall Results
| Agent | Model | Reasoning | Success Rate | Average Latency |
|-------|-------|-----------|--------------|-----------------|
| simple | Qwen/Qwen3.5-27B-FP8 | true | 23/23 (100.00%) | 14.417s |
| simple_json | Qwen/Qwen3.5-27B-FP8 | false | 5/5 (100.00%) | 48.110s |
| primary_agent | Qwen/Qwen3.5-27B-FP8 | true | 23/23 (100.00%) | 40.143s |
| assistant | Qwen/Qwen3.5-27B-FP8 | true | 22/23 (95.65%) | 52.153s |
| generator | Qwen/Qwen3.5-27B-FP8 | true | 23/23 (100.00%) | 50.132s |
| refiner | Qwen/Qwen3.5-27B-FP8 | true | 23/23 (100.00%) | 48.599s |
| adviser | Qwen/Qwen3.5-27B-FP8 | true | 22/23 (95.65%) | 51.045s |
| reflector | Qwen/Qwen3.5-27B-FP8 | true | 23/23 (100.00%) | 20.053s |
| searcher | Qwen/Qwen3.5-27B-FP8 | true | 23/23 (100.00%) | 15.935s |
| enricher | Qwen/Qwen3.5-27B-FP8 | true | 23/23 (100.00%) | 17.074s |
| coder | Qwen/Qwen3.5-27B-FP8 | true | 23/23 (100.00%) | 54.885s |
| installer | Qwen/Qwen3.5-27B-FP8 | true | 23/23 (100.00%) | 55.538s |
| pentester | Qwen/Qwen3.5-27B-FP8 | true | 23/23 (100.00%) | 54.749s |
**Total**: 279/281 (99.29%) successful tests
**Overall average latency**: 39.712s
## Detailed Results
### simple (Qwen/Qwen3.5-27B-FP8)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 7.676s | |
| Text Transform Uppercase | ✅ Pass | 0.659s | |
| Count from 1 to 5 | ✅ Pass | 0.505s | |
| Math Calculation | ✅ Pass | 16.490s | |
| Basic Echo Function | ✅ Pass | 19.115s | |
| Streaming Simple Math Streaming | ✅ Pass | 4.274s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 10.085s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 0.804s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 53.536s | |
| Search Query Function | ✅ Pass | 0.823s | |
| Ask Advice Function | ✅ Pass | 1.980s | |
| Streaming Search Query Function Streaming | ✅ Pass | 36.073s | |
| Basic Context Memory Test | ✅ Pass | 1.551s | |
| Function Argument Memory Test | ✅ Pass | 0.306s | |
| Function Response Memory Test | ✅ Pass | 5.207s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 11.206s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 0.317s | |
| Penetration Testing Methodology | ✅ Pass | 27.195s | |
| Vulnerability Assessment Tools | ✅ Pass | 30.694s | |
| SQL Injection Attack Type | ✅ Pass | 2.421s | |
| Penetration Testing Framework | ✅ Pass | 54.771s | |
| Web Application Security Scanner | ✅ Pass | 40.336s | |
| Penetration Testing Tool Selection | ✅ Pass | 5.550s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 14.417s
---
### simple_json (Qwen/Qwen3.5-27B-FP8)
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Vulnerability Report Memory Test | ✅ Pass | 65.427s | |
| Project Information JSON | ✅ Pass | 43.413s | |
| Person Information JSON | ✅ Pass | 53.143s | |
| User Profile JSON | ✅ Pass | 42.996s | |
| Streaming Person Information JSON Streaming | ✅ Pass | 35.570s | |
**Summary**: 5/5 (100.00%) successful tests
**Average latency**: 48.110s
---
### primary_agent (Qwen/Qwen3.5-27B-FP8)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 46.317s | |
| Text Transform Uppercase | ✅ Pass | 3.599s | |
| Count from 1 to 5 | ✅ Pass | 77.707s | |
| Math Calculation | ✅ Pass | 49.296s | |
| Basic Echo Function | ✅ Pass | 46.650s | |
| Streaming Simple Math Streaming | ✅ Pass | 5.358s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 1.253s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 17.503s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 54.723s | |
| Search Query Function | ✅ Pass | 1.968s | |
| Ask Advice Function | ✅ Pass | 2.593s | |
| Streaming Search Query Function Streaming | ✅ Pass | 38.959s | |
| Basic Context Memory Test | ✅ Pass | 37.345s | |
| Function Argument Memory Test | ✅ Pass | 90.263s | |
| Function Response Memory Test | ✅ Pass | 3.072s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 15.508s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 1.638s | |
| Penetration Testing Methodology | ✅ Pass | 20.098s | |
| Vulnerability Assessment Tools | ✅ Pass | 178.331s | |
| SQL Injection Attack Type | ✅ Pass | 42.430s | |
| Penetration Testing Framework | ✅ Pass | 50.972s | |
| Web Application Security Scanner | ✅ Pass | 75.701s | |
| Penetration Testing Tool Selection | ✅ Pass | 61.984s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 40.143s
---
### assistant (Qwen/Qwen3.5-27B-FP8)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 46.006s | |
| Text Transform Uppercase | ✅ Pass | 4.004s | |
| Count from 1 to 5 | ✅ Pass | 66.849s | |
| Math Calculation | ✅ Pass | 46.121s | |
| Basic Echo Function | ✅ Pass | 56.990s | |
| Streaming Simple Math Streaming | ✅ Pass | 4.660s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 14.354s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 10.898s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 42.389s | |
| Search Query Function | ✅ Pass | 1.672s | |
| Ask Advice Function | ✅ Pass | 2.230s | |
| Streaming Search Query Function Streaming | ✅ Pass | 39.160s | |
| Basic Context Memory Test | ✅ Pass | 30.534s | |
| Function Argument Memory Test | ✅ Pass | 81.043s | |
| Function Response Memory Test | ✅ Pass | 3.833s | |
| Penetration Testing Memory with Tool Call | ❌ Fail | 16.904s | expected function 'generate\_report' not found in tool calls: expected function generate\_report not found in tool calls |
| Cybersecurity Workflow Memory Test | ✅ Pass | 63.736s | |
| Penetration Testing Methodology | ✅ Pass | 16.286s | |
| SQL Injection Attack Type | ✅ Pass | 47.840s | |
| Vulnerability Assessment Tools | ✅ Pass | 437.897s | |
| Penetration Testing Framework | ✅ Pass | 37.735s | |
| Web Application Security Scanner | ✅ Pass | 67.262s | |
| Penetration Testing Tool Selection | ✅ Pass | 61.109s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 52.153s
---
### generator (Qwen/Qwen3.5-27B-FP8)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 45.295s | |
| Text Transform Uppercase | ✅ Pass | 4.303s | |
| Count from 1 to 5 | ✅ Pass | 59.045s | |
| Math Calculation | ✅ Pass | 66.939s | |
| Basic Echo Function | ✅ Pass | 34.650s | |
| Streaming Simple Math Streaming | ✅ Pass | 3.368s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 11.610s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 67.278s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 40.016s | |
| Search Query Function | ✅ Pass | 2.165s | |
| Ask Advice Function | ✅ Pass | 2.305s | |
| Streaming Search Query Function Streaming | ✅ Pass | 14.124s | |
| Basic Context Memory Test | ✅ Pass | 72.382s | |
| Function Argument Memory Test | ✅ Pass | 70.020s | |
| Function Response Memory Test | ✅ Pass | 2.337s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 16.301s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 54.758s | |
| Penetration Testing Methodology | ✅ Pass | 20.628s | |
| Vulnerability Assessment Tools | ✅ Pass | 342.586s | |
| SQL Injection Attack Type | ✅ Pass | 78.224s | |
| Penetration Testing Framework | ✅ Pass | 25.054s | |
| Web Application Security Scanner | ✅ Pass | 58.230s | |
| Penetration Testing Tool Selection | ✅ Pass | 61.399s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 50.132s
---
### refiner (Qwen/Qwen3.5-27B-FP8)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 45.873s | |
| Text Transform Uppercase | ✅ Pass | 4.709s | |
| Count from 1 to 5 | ✅ Pass | 62.154s | |
| Math Calculation | ✅ Pass | 37.647s | |
| Basic Echo Function | ✅ Pass | 37.346s | |
| Streaming Simple Math Streaming | ✅ Pass | 3.470s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.181s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 68.362s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 37.968s | |
| Search Query Function | ✅ Pass | 2.222s | |
| Ask Advice Function | ✅ Pass | 59.187s | |
| Streaming Search Query Function Streaming | ✅ Pass | 13.251s | |
| Basic Context Memory Test | ✅ Pass | 3.312s | |
| Function Argument Memory Test | ✅ Pass | 67.582s | |
| Function Response Memory Test | ✅ Pass | 2.802s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 16.111s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 72.918s | |
| Penetration Testing Methodology | ✅ Pass | 14.653s | |
| Vulnerability Assessment Tools | ✅ Pass | 345.334s | |
| SQL Injection Attack Type | ✅ Pass | 82.511s | |
| Penetration Testing Framework | ✅ Pass | 11.861s | |
| Web Application Security Scanner | ✅ Pass | 48.987s | |
| Penetration Testing Tool Selection | ✅ Pass | 75.316s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 48.599s
---
### adviser (Qwen/Qwen3.5-27B-FP8)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 46.219s | |
| Text Transform Uppercase | ✅ Pass | 5.548s | |
| Count from 1 to 5 | ✅ Pass | 70.699s | |
| Math Calculation | ✅ Pass | 60.313s | |
| Basic Echo Function | ✅ Pass | 39.678s | |
| Streaming Simple Math Streaming | ✅ Pass | 3.608s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.352s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 68.389s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 33.105s | |
| Search Query Function | ✅ Pass | 1.318s | |
| Ask Advice Function | ✅ Pass | 58.002s | |
| Streaming Search Query Function Streaming | ✅ Pass | 10.630s | |
| Basic Context Memory Test | ✅ Pass | 16.399s | |
| Function Argument Memory Test | ✅ Pass | 62.118s | |
| Function Response Memory Test | ✅ Pass | 2.379s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 16.699s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 64.896s | |
| Penetration Testing Methodology | ✅ Pass | 15.182s | |
| Vulnerability Assessment Tools | ❌ Fail | 314.943s | expected text 'network' not found |
| SQL Injection Attack Type | ✅ Pass | 51.973s | |
| Penetration Testing Framework | ✅ Pass | 10.408s | |
| Web Application Security Scanner | ✅ Pass | 140.844s | |
| Penetration Testing Tool Selection | ✅ Pass | 76.315s | |
**Summary**: 22/23 (95.65%) successful tests
**Average latency**: 51.045s
---
### reflector (Qwen/Qwen3.5-27B-FP8)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 6.835s | |
| Text Transform Uppercase | ✅ Pass | 1.439s | |
| Count from 1 to 5 | ✅ Pass | 10.836s | |
| Math Calculation | ✅ Pass | 14.311s | |
| Basic Echo Function | ✅ Pass | 19.567s | |
| Streaming Simple Math Streaming | ✅ Pass | 0.513s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.329s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 51.378s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 28.011s | |
| Search Query Function | ✅ Pass | 0.793s | |
| Ask Advice Function | ✅ Pass | 29.709s | |
| Streaming Search Query Function Streaming | ✅ Pass | 10.034s | |
| Basic Context Memory Test | ✅ Pass | 1.082s | |
| Function Argument Memory Test | ✅ Pass | 41.331s | |
| Function Response Memory Test | ✅ Pass | 0.904s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 14.044s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 25.000s | |
| Penetration Testing Methodology | ✅ Pass | 5.131s | |
| Vulnerability Assessment Tools | ✅ Pass | 75.303s | |
| SQL Injection Attack Type | ✅ Pass | 5.444s | |
| Penetration Testing Framework | ✅ Pass | 55.241s | |
| Web Application Security Scanner | ✅ Pass | 14.433s | |
| Penetration Testing Tool Selection | ✅ Pass | 49.539s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 20.053s
---
### searcher (Qwen/Qwen3.5-27B-FP8)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 6.835s | |
| Text Transform Uppercase | ✅ Pass | 0.289s | |
| Count from 1 to 5 | ✅ Pass | 9.973s | |
| Math Calculation | ✅ Pass | 13.611s | |
| Basic Echo Function | ✅ Pass | 29.473s | |
| Streaming Simple Math Streaming | ✅ Pass | 10.019s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.336s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 50.127s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.682s | |
| Search Query Function | ✅ Pass | 0.708s | |
| Ask Advice Function | ✅ Pass | 29.709s | |
| Streaming Search Query Function Streaming | ✅ Pass | 9.131s | |
| Basic Context Memory Test | ✅ Pass | 0.501s | |
| Function Argument Memory Test | ✅ Pass | 24.274s | |
| Function Response Memory Test | ✅ Pass | 0.357s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 12.952s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 29.898s | |
| Penetration Testing Methodology | ✅ Pass | 4.896s | |
| Vulnerability Assessment Tools | ✅ Pass | 18.455s | |
| SQL Injection Attack Type | ✅ Pass | 6.357s | |
| Penetration Testing Framework | ✅ Pass | 34.478s | |
| Web Application Security Scanner | ✅ Pass | 12.739s | |
| Penetration Testing Tool Selection | ✅ Pass | 59.684s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 15.935s
---
### enricher (Qwen/Qwen3.5-27B-FP8)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 38.613s | |
| Text Transform Uppercase | ✅ Pass | 0.288s | |
| Count from 1 to 5 | ✅ Pass | 9.635s | |
| Math Calculation | ✅ Pass | 5.748s | |
| Basic Echo Function | ✅ Pass | 15.700s | |
| Streaming Simple Math Streaming | ✅ Pass | 11.636s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 0.397s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 49.174s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 1.357s | |
| Search Query Function | ✅ Pass | 1.935s | |
| Ask Advice Function | ✅ Pass | 29.692s | |
| Streaming Search Query Function Streaming | ✅ Pass | 5.592s | |
| Basic Context Memory Test | ✅ Pass | 0.396s | |
| Function Argument Memory Test | ✅ Pass | 19.758s | |
| Function Response Memory Test | ✅ Pass | 0.352s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.148s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 34.780s | |
| Penetration Testing Methodology | ✅ Pass | 4.457s | |
| Vulnerability Assessment Tools | ✅ Pass | 18.442s | |
| SQL Injection Attack Type | ✅ Pass | 0.516s | |
| Penetration Testing Framework | ✅ Pass | 63.726s | |
| Web Application Security Scanner | ✅ Pass | 35.554s | |
| Penetration Testing Tool Selection | ✅ Pass | 41.806s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 17.074s
---
### coder (Qwen/Qwen3.5-27B-FP8)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 39.481s | |
| Text Transform Uppercase | ✅ Pass | 3.370s | |
| Count from 1 to 5 | ✅ Pass | 78.615s | |
| Math Calculation | ✅ Pass | 32.595s | |
| Basic Echo Function | ✅ Pass | 16.466s | |
| Streaming Simple Math Streaming | ✅ Pass | 14.916s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 4.345s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 74.466s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.116s | |
| Search Query Function | ✅ Pass | 2.488s | |
| Ask Advice Function | ✅ Pass | 61.793s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.247s | |
| Basic Context Memory Test | ✅ Pass | 44.846s | |
| Function Argument Memory Test | ✅ Pass | 11.874s | |
| Function Response Memory Test | ✅ Pass | 1.275s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.892s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 82.178s | |
| Penetration Testing Methodology | ✅ Pass | 154.106s | |
| SQL Injection Attack Type | ✅ Pass | 19.059s | |
| Vulnerability Assessment Tools | ✅ Pass | 298.388s | |
| Penetration Testing Framework | ✅ Pass | 114.067s | |
| Web Application Security Scanner | ✅ Pass | 150.870s | |
| Penetration Testing Tool Selection | ✅ Pass | 47.893s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 54.885s
---
### installer (Qwen/Qwen3.5-27B-FP8)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 41.183s | |
| Text Transform Uppercase | ✅ Pass | 4.185s | |
| Count from 1 to 5 | ✅ Pass | 42.353s | |
| Math Calculation | ✅ Pass | 36.572s | |
| Basic Echo Function | ✅ Pass | 13.272s | |
| Streaming Simple Math Streaming | ✅ Pass | 12.815s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 7.686s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 66.695s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.003s | |
| Search Query Function | ✅ Pass | 2.361s | |
| Ask Advice Function | ✅ Pass | 60.607s | |
| Streaming Search Query Function Streaming | ✅ Pass | 3.050s | |
| Basic Context Memory Test | ✅ Pass | 29.974s | |
| Function Argument Memory Test | ✅ Pass | 12.821s | |
| Function Response Memory Test | ✅ Pass | 1.053s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.913s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 72.414s | |
| Penetration Testing Methodology | ✅ Pass | 95.008s | |
| SQL Injection Attack Type | ✅ Pass | 56.874s | |
| Vulnerability Assessment Tools | ✅ Pass | 410.320s | |
| Penetration Testing Framework | ✅ Pass | 113.426s | |
| Web Application Security Scanner | ✅ Pass | 152.774s | |
| Penetration Testing Tool Selection | ✅ Pass | 36.011s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 55.538s
---
### pentester (Qwen/Qwen3.5-27B-FP8)
#### Basic Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| Simple Math | ✅ Pass | 3.622s | |
| Text Transform Uppercase | ✅ Pass | 3.788s | |
| Count from 1 to 5 | ✅ Pass | 61.459s | |
| Math Calculation | ✅ Pass | 57.892s | |
| Basic Echo Function | ✅ Pass | 10.738s | |
| Streaming Simple Math Streaming | ✅ Pass | 13.100s | |
| Streaming Count from 1 to 3 Streaming | ✅ Pass | 5.418s | |
| Streaming Basic Echo Function Streaming | ✅ Pass | 54.160s | |
#### Advanced Tests
| Test | Result | Latency | Error |
|------|--------|---------|-------|
| JSON Response Function | ✅ Pass | 2.458s | |
| Search Query Function | ✅ Pass | 2.340s | |
| Ask Advice Function | ✅ Pass | 60.580s | |
| Streaming Search Query Function Streaming | ✅ Pass | 2.590s | |
| Basic Context Memory Test | ✅ Pass | 62.238s | |
| Function Argument Memory Test | ✅ Pass | 7.654s | |
| Function Response Memory Test | ✅ Pass | 1.304s | |
| Penetration Testing Memory with Tool Call | ✅ Pass | 3.581s | |
| Cybersecurity Workflow Memory Test | ✅ Pass | 71.250s | |
| Penetration Testing Methodology | ✅ Pass | 152.774s | |
| SQL Injection Attack Type | ✅ Pass | 82.404s | |
| Vulnerability Assessment Tools | ✅ Pass | 294.588s | |
| Penetration Testing Framework | ✅ Pass | 107.775s | |
| Web Application Security Scanner | ✅ Pass | 166.016s | |
| Penetration Testing Tool Selection | ✅ Pass | 31.482s | |
**Summary**: 23/23 (100.00%) successful tests
**Average latency**: 54.749s
---