chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,315 @@
|
||||
# Crawl4AI Stress Testing and Benchmarking
|
||||
|
||||
This directory contains tools for stress testing Crawl4AI's `arun_many` method and dispatcher system with high volumes of URLs to evaluate performance, concurrency handling, and potentially detect memory issues. It also includes a benchmarking system to track performance over time.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Run a default stress test (small config) and generate a report
|
||||
# (Assumes run_all.sh is updated to call run_benchmark.py)
|
||||
./run_all.sh
|
||||
```
|
||||
*Note: `run_all.sh` might need to be updated if it directly called the old script.*
|
||||
|
||||
## Overview
|
||||
|
||||
The stress testing system works by:
|
||||
|
||||
1. Generating a local test site with heavy HTML pages (regenerated by default for each test).
|
||||
2. Starting a local HTTP server to serve these pages.
|
||||
3. Running Crawl4AI's `arun_many` method against this local site using the `MemoryAdaptiveDispatcher` with configurable concurrency (`max_sessions`).
|
||||
4. Monitoring performance metrics via the `CrawlerMonitor` and optionally logging memory usage.
|
||||
5. Optionally generating detailed benchmark reports with visualizations using `benchmark_report.py`.
|
||||
|
||||
## Available Tools
|
||||
|
||||
- `test_stress_sdk.py` - Main stress testing script utilizing `arun_many` and dispatchers.
|
||||
- `benchmark_report.py` - Report generator for comparing test results (assumes compatibility with `test_stress_sdk.py` outputs).
|
||||
- `run_benchmark.py` - Python script with predefined test configurations that orchestrates tests using `test_stress_sdk.py`.
|
||||
- `run_all.sh` - Simple wrapper script (may need updating).
|
||||
|
||||
## Usage Guide
|
||||
|
||||
### Using Predefined Configurations (Recommended)
|
||||
|
||||
The `run_benchmark.py` script offers the easiest way to run standardized tests:
|
||||
|
||||
```bash
|
||||
# Quick test (50 URLs, 4 max sessions)
|
||||
python run_benchmark.py quick
|
||||
|
||||
# Medium test (500 URLs, 16 max sessions)
|
||||
python run_benchmark.py medium
|
||||
|
||||
# Large test (1000 URLs, 32 max sessions)
|
||||
python run_benchmark.py large
|
||||
|
||||
# Extreme test (2000 URLs, 64 max sessions)
|
||||
python run_benchmark.py extreme
|
||||
|
||||
# Custom configuration
|
||||
python run_benchmark.py custom --urls 300 --max-sessions 24 --chunk-size 50
|
||||
|
||||
# Run 'small' test in streaming mode
|
||||
python run_benchmark.py small --stream
|
||||
|
||||
# Override max_sessions for the 'medium' config
|
||||
python run_benchmark.py medium --max-sessions 20
|
||||
|
||||
# Skip benchmark report generation after the test
|
||||
python run_benchmark.py small --no-report
|
||||
|
||||
# Clean up reports and site files before running
|
||||
python run_benchmark.py medium --clean
|
||||
```
|
||||
|
||||
#### `run_benchmark.py` Parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
| -------------------- | --------------- | --------------------------------------------------------------------------- |
|
||||
| `config` | *required* | Test configuration: `quick`, `small`, `medium`, `large`, `extreme`, `custom`|
|
||||
| `--urls` | config-specific | Number of URLs (required for `custom`) |
|
||||
| `--max-sessions` | config-specific | Max concurrent sessions managed by dispatcher (required for `custom`) |
|
||||
| `--chunk-size` | config-specific | URLs per batch for non-stream logging (required for `custom`) |
|
||||
| `--stream` | False | Enable streaming results (disables batch logging) |
|
||||
| `--monitor-mode` | DETAILED | `DETAILED` or `AGGREGATED` display for the live monitor |
|
||||
| `--use-rate-limiter` | False | Enable basic rate limiter in the dispatcher |
|
||||
| `--port` | 8000 | HTTP server port |
|
||||
| `--no-report` | False | Skip generating comparison report via `benchmark_report.py` |
|
||||
| `--clean` | False | Clean up reports and site files before running |
|
||||
| `--keep-server-alive`| False | Keep local HTTP server running after test |
|
||||
| `--use-existing-site`| False | Use existing site on specified port (no local server start/site gen) |
|
||||
| `--skip-generation` | False | Use existing site files but start local server |
|
||||
| `--keep-site` | False | Keep generated site files after test |
|
||||
|
||||
#### Predefined Configurations
|
||||
|
||||
| Configuration | URLs | Max Sessions | Chunk Size | Description |
|
||||
| ------------- | ------ | ------------ | ---------- | -------------------------------- |
|
||||
| `quick` | 50 | 4 | 10 | Quick test for basic validation |
|
||||
| `small` | 100 | 8 | 20 | Small test for routine checks |
|
||||
| `medium` | 500 | 16 | 50 | Medium test for thorough checks |
|
||||
| `large` | 1000 | 32 | 100 | Large test for stress testing |
|
||||
| `extreme` | 2000 | 64 | 200 | Extreme test for limit testing |
|
||||
|
||||
### Direct Usage of `test_stress_sdk.py`
|
||||
|
||||
For fine-grained control or debugging, you can run the stress test script directly:
|
||||
|
||||
```bash
|
||||
# Test with 200 URLs and 32 max concurrent sessions
|
||||
python test_stress_sdk.py --urls 200 --max-sessions 32 --chunk-size 40
|
||||
|
||||
# Clean up previous test data first
|
||||
python test_stress_sdk.py --clean-reports --clean-site --urls 100 --max-sessions 16 --chunk-size 20
|
||||
|
||||
# Change the HTTP server port and use aggregated monitor
|
||||
python test_stress_sdk.py --port 8088 --urls 100 --max-sessions 16 --monitor-mode AGGREGATED
|
||||
|
||||
# Enable streaming mode and use rate limiting
|
||||
python test_stress_sdk.py --urls 50 --max-sessions 8 --stream --use-rate-limiter
|
||||
|
||||
# Change report output location
|
||||
python test_stress_sdk.py --report-path custom_reports --urls 100 --max-sessions 16
|
||||
```
|
||||
|
||||
#### `test_stress_sdk.py` Parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
| -------------------- | ---------- | -------------------------------------------------------------------- |
|
||||
| `--urls` | 100 | Number of URLs to test |
|
||||
| `--max-sessions` | 16 | Maximum concurrent crawling sessions managed by the dispatcher |
|
||||
| `--chunk-size` | 10 | Number of URLs per batch (relevant for non-stream logging) |
|
||||
| `--stream` | False | Enable streaming results (disables batch logging) |
|
||||
| `--monitor-mode` | DETAILED | `DETAILED` or `AGGREGATED` display for the live `CrawlerMonitor` |
|
||||
| `--use-rate-limiter` | False | Enable a basic `RateLimiter` within the dispatcher |
|
||||
| `--site-path` | "test_site"| Path to store/use the generated test site |
|
||||
| `--port` | 8000 | Port for the local HTTP server |
|
||||
| `--report-path` | "reports" | Path to save test result summary (JSON) and memory samples (CSV) |
|
||||
| `--skip-generation` | False | Use existing test site files but still start local server |
|
||||
| `--use-existing-site`| False | Use existing site on specified port (no local server/site gen) |
|
||||
| `--keep-server-alive`| False | Keep local HTTP server running after test completion |
|
||||
| `--keep-site` | False | Keep the generated test site files after test completion |
|
||||
| `--clean-reports` | False | Clean up report directory before running |
|
||||
| `--clean-site` | False | Clean up site directory before/after running (see script logic) |
|
||||
|
||||
### Generating Reports Only
|
||||
|
||||
If you only want to generate a benchmark report from existing test results (assuming `benchmark_report.py` is compatible):
|
||||
|
||||
```bash
|
||||
# Generate a report from existing test results in ./reports/
|
||||
python benchmark_report.py
|
||||
|
||||
# Limit to the most recent 5 test results
|
||||
python benchmark_report.py --limit 5
|
||||
|
||||
# Specify a custom source directory for test results
|
||||
python benchmark_report.py --reports-dir alternate_results
|
||||
```
|
||||
|
||||
#### `benchmark_report.py` Parameters (Assumed)
|
||||
|
||||
| Parameter | Default | Description |
|
||||
| --------------- | -------------------- | ----------------------------------------------------------- |
|
||||
| `--reports-dir` | "reports" | Directory containing `test_stress_sdk.py` result files |
|
||||
| `--output-dir` | "benchmark_reports" | Directory to save generated HTML reports and charts |
|
||||
| `--limit` | None (all results) | Limit comparison to N most recent test results |
|
||||
| `--output-file` | Auto-generated | Custom output filename for the HTML report |
|
||||
|
||||
## Understanding the Test Output
|
||||
|
||||
### Real-time Progress Display (`CrawlerMonitor`)
|
||||
|
||||
When running `test_stress_sdk.py`, the `CrawlerMonitor` provides a live view of the crawling process managed by the dispatcher.
|
||||
|
||||
- **DETAILED Mode (Default):** Shows individual task status (Queued, Active, Completed, Failed), timings, memory usage per task (if `psutil` is available), overall queue statistics, and memory pressure status (if `psutil` available).
|
||||
- **AGGREGATED Mode:** Shows summary counts (Queued, Active, Completed, Failed), overall progress percentage, estimated time remaining, average URLs/sec, and memory pressure status.
|
||||
|
||||
### Batch Log Output (Non-Streaming Mode Only)
|
||||
|
||||
If running `test_stress_sdk.py` **without** the `--stream` flag, you will *also* see per-batch summary lines printed to the console *after* the monitor display, once each chunk of URLs finishes processing:
|
||||
|
||||
```
|
||||
Batch | Progress | Start Mem | End Mem | URLs/sec | Success/Fail | Time (s) | Status
|
||||
───────────────────────────────────────────────────────────────────────────────────────────
|
||||
1 | 10.0% | 50.1 MB | 55.3 MB | 23.8 | 10/0 | 0.42 | Success
|
||||
2 | 20.0% | 55.3 MB | 60.1 MB | 24.1 | 10/0 | 0.41 | Success
|
||||
...
|
||||
```
|
||||
|
||||
This display provides chunk-specific metrics:
|
||||
- **Batch**: The batch number being reported.
|
||||
- **Progress**: Overall percentage of total URLs processed *after* this batch.
|
||||
- **Start Mem / End Mem**: Memory usage before and after processing this batch (if tracked).
|
||||
- **URLs/sec**: Processing speed *for this specific batch*.
|
||||
- **Success/Fail**: Number of successful and failed URLs *in this batch*.
|
||||
- **Time (s)**: Wall-clock time taken to process *this batch*.
|
||||
- **Status**: Color-coded status for the batch outcome.
|
||||
|
||||
### Summary Output
|
||||
|
||||
After test completion, a final summary is displayed:
|
||||
|
||||
```
|
||||
================================================================================
|
||||
Test Completed
|
||||
================================================================================
|
||||
Test ID: 20250418_103015
|
||||
Configuration: 100 URLs, 16 max sessions, Chunk: 10, Stream: False, Monitor: DETAILED
|
||||
Results: 100 successful, 0 failed (100 processed, 100.0% success)
|
||||
Performance: 5.85 seconds total, 17.09 URLs/second avg
|
||||
Memory Usage: Start: 50.1 MB, End: 75.3 MB, Max: 78.1 MB, Growth: 25.2 MB
|
||||
Results summary saved to reports/test_summary_20250418_103015.json
|
||||
```
|
||||
|
||||
### HTML Report Structure (Generated by `benchmark_report.py`)
|
||||
|
||||
(This section remains the same, assuming `benchmark_report.py` generates these)
|
||||
The benchmark report contains several sections:
|
||||
1. **Summary**: Overview of the latest test results and trends
|
||||
2. **Performance Comparison**: Charts showing throughput across tests
|
||||
3. **Memory Usage**: Detailed memory usage graphs for each test
|
||||
4. **Detailed Results**: Tabular data of all test metrics
|
||||
5. **Conclusion**: Automated analysis of performance and memory patterns
|
||||
|
||||
### Memory Metrics
|
||||
|
||||
(This section remains conceptually the same)
|
||||
Memory growth is the key metric for detecting leaks...
|
||||
|
||||
### Performance Metrics
|
||||
|
||||
(This section remains conceptually the same, though "URLs per Worker" is less relevant - focus on overall URLs/sec)
|
||||
Key performance indicators include:
|
||||
- **URLs per Second**: Higher is better (throughput)
|
||||
- **Success Rate**: Should be 100% in normal conditions
|
||||
- **Total Processing Time**: Lower is better
|
||||
- **Dispatcher Efficiency**: Observe queue lengths and wait times in the monitor (Detailed mode)
|
||||
|
||||
### Raw Data Files
|
||||
|
||||
Raw data is saved in the `--report-path` directory (default `./reports/`):
|
||||
|
||||
- **JSON files** (`test_summary_*.json`): Contains the final summary for each test run.
|
||||
- **CSV files** (`memory_samples_*.csv`): Contains time-series memory samples taken during the test run.
|
||||
|
||||
Example of reading raw data:
|
||||
```python
|
||||
import json
|
||||
import pandas as pd
|
||||
|
||||
# Load test summary
|
||||
test_id = "20250418_103015" # Example ID
|
||||
with open(f'reports/test_summary_{test_id}.json', 'r') as f:
|
||||
results = json.load(f)
|
||||
|
||||
# Load memory samples
|
||||
memory_df = pd.read_csv(f'reports/memory_samples_{test_id}.csv')
|
||||
|
||||
# Analyze memory_df (e.g., calculate growth, plot)
|
||||
if not memory_df['memory_info_mb'].isnull().all():
|
||||
growth = memory_df['memory_info_mb'].iloc[-1] - memory_df['memory_info_mb'].iloc[0]
|
||||
print(f"Total Memory Growth: {growth:.1f} MB")
|
||||
else:
|
||||
print("No valid memory samples found.")
|
||||
|
||||
print(f"Avg URLs/sec: {results['urls_processed'] / results['total_time_seconds']:.2f}")
|
||||
```
|
||||
|
||||
## Visualization Dependencies
|
||||
|
||||
(This section remains the same)
|
||||
For full visualization capabilities in the HTML reports generated by `benchmark_report.py`, install additional dependencies...
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
benchmarking/ # Or your top-level directory name
|
||||
├── benchmark_reports/ # Generated HTML reports (by benchmark_report.py)
|
||||
├── reports/ # Raw test result data (from test_stress_sdk.py)
|
||||
├── test_site/ # Generated test content (temporary)
|
||||
├── benchmark_report.py# Report generator
|
||||
├── run_benchmark.py # Test runner with predefined configs
|
||||
├── test_stress_sdk.py # Main stress test implementation using arun_many
|
||||
└── run_all.sh # Simple wrapper script (may need updates)
|
||||
#└── requirements.txt # Optional: Visualization dependencies for benchmark_report.py
|
||||
```
|
||||
|
||||
## Cleanup
|
||||
|
||||
To clean up after testing:
|
||||
|
||||
```bash
|
||||
# Remove the test site content (if not using --keep-site)
|
||||
rm -rf test_site
|
||||
|
||||
# Remove all raw reports and generated benchmark reports
|
||||
rm -rf reports benchmark_reports
|
||||
|
||||
# Or use the --clean flag with run_benchmark.py
|
||||
python run_benchmark.py medium --clean
|
||||
```
|
||||
|
||||
## Use in CI/CD
|
||||
|
||||
(This section remains conceptually the same, just update script names)
|
||||
These tests can be integrated into CI/CD pipelines:
|
||||
```bash
|
||||
# Example CI script
|
||||
python run_benchmark.py medium --no-report # Run test without interactive report gen
|
||||
# Check exit code
|
||||
if [ $? -ne 0 ]; then echo "Stress test failed!"; exit 1; fi
|
||||
# Optionally, run report generator and check its output/metrics
|
||||
# python benchmark_report.py
|
||||
# check_report_metrics.py reports/test_summary_*.json || exit 1
|
||||
exit 0
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **HTTP Server Port Conflict**: Use `--port` with `run_benchmark.py` or `test_stress_sdk.py`.
|
||||
- **Memory Tracking Issues**: The `SimpleMemoryTracker` uses platform commands (`ps`, `/proc`, `tasklist`). Ensure these are available and the script has permission. If it consistently fails, memory reporting will be limited.
|
||||
- **Visualization Missing**: Related to `benchmark_report.py` and its dependencies.
|
||||
- **Site Generation Issues**: Check permissions for creating `./test_site/`. Use `--skip-generation` if you want to manage the site manually.
|
||||
- **Testing Against External Site**: Ensure the external site is running and use `--use-existing-site --port <correct_port>`.
|
||||
Executable
+887
@@ -0,0 +1,887 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Benchmark reporting tool for Crawl4AI stress tests.
|
||||
Generates visual reports and comparisons between test runs.
|
||||
"""
|
||||
|
||||
import os
|
||||
import json
|
||||
import glob
|
||||
import argparse
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from rich.console import Console
|
||||
from rich.table import Table
|
||||
from rich.panel import Panel
|
||||
|
||||
# Initialize rich console
|
||||
console = Console()
|
||||
|
||||
# Try to import optional visualization dependencies
|
||||
VISUALIZATION_AVAILABLE = True
|
||||
try:
|
||||
import pandas as pd
|
||||
import matplotlib.pyplot as plt
|
||||
import matplotlib as mpl
|
||||
import numpy as np
|
||||
import seaborn as sns
|
||||
except ImportError:
|
||||
VISUALIZATION_AVAILABLE = False
|
||||
console.print("[yellow]Warning: Visualization dependencies not found. Install with:[/yellow]")
|
||||
console.print("[yellow]pip install pandas matplotlib seaborn[/yellow]")
|
||||
console.print("[yellow]Only text-based reports will be generated.[/yellow]")
|
||||
|
||||
# Configure plotting if available
|
||||
if VISUALIZATION_AVAILABLE:
|
||||
# Set plot style for dark theme
|
||||
plt.style.use('dark_background')
|
||||
sns.set_theme(style="darkgrid")
|
||||
|
||||
# Custom color palette based on Nord theme
|
||||
nord_palette = ["#88c0d0", "#81a1c1", "#a3be8c", "#ebcb8b", "#bf616a", "#b48ead", "#5e81ac"]
|
||||
sns.set_palette(nord_palette)
|
||||
|
||||
class BenchmarkReporter:
|
||||
"""Generates visual reports and comparisons for Crawl4AI stress tests."""
|
||||
|
||||
def __init__(self, reports_dir="reports", output_dir="benchmark_reports"):
|
||||
"""Initialize the benchmark reporter.
|
||||
|
||||
Args:
|
||||
reports_dir: Directory containing test result files
|
||||
output_dir: Directory to save generated reports
|
||||
"""
|
||||
self.reports_dir = Path(reports_dir)
|
||||
self.output_dir = Path(output_dir)
|
||||
self.output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Configure matplotlib if available
|
||||
if VISUALIZATION_AVAILABLE:
|
||||
# Ensure the matplotlib backend works in headless environments
|
||||
mpl.use('Agg')
|
||||
|
||||
# Set up styling for plots with dark theme
|
||||
mpl.rcParams['figure.figsize'] = (12, 8)
|
||||
mpl.rcParams['font.size'] = 12
|
||||
mpl.rcParams['axes.labelsize'] = 14
|
||||
mpl.rcParams['axes.titlesize'] = 16
|
||||
mpl.rcParams['xtick.labelsize'] = 12
|
||||
mpl.rcParams['ytick.labelsize'] = 12
|
||||
mpl.rcParams['legend.fontsize'] = 12
|
||||
mpl.rcParams['figure.facecolor'] = '#1e1e1e'
|
||||
mpl.rcParams['axes.facecolor'] = '#2e3440'
|
||||
mpl.rcParams['savefig.facecolor'] = '#1e1e1e'
|
||||
mpl.rcParams['text.color'] = '#e0e0e0'
|
||||
mpl.rcParams['axes.labelcolor'] = '#e0e0e0'
|
||||
mpl.rcParams['xtick.color'] = '#e0e0e0'
|
||||
mpl.rcParams['ytick.color'] = '#e0e0e0'
|
||||
mpl.rcParams['grid.color'] = '#444444'
|
||||
mpl.rcParams['figure.edgecolor'] = '#444444'
|
||||
|
||||
def load_test_results(self, limit=None):
|
||||
"""Load all test results from the reports directory.
|
||||
|
||||
Args:
|
||||
limit: Optional limit on number of most recent tests to load
|
||||
|
||||
Returns:
|
||||
Dictionary mapping test IDs to result data
|
||||
"""
|
||||
result_files = glob.glob(str(self.reports_dir / "test_results_*.json"))
|
||||
|
||||
# Sort files by modification time (newest first)
|
||||
result_files.sort(key=os.path.getmtime, reverse=True)
|
||||
|
||||
if limit:
|
||||
result_files = result_files[:limit]
|
||||
|
||||
results = {}
|
||||
for file_path in result_files:
|
||||
try:
|
||||
with open(file_path, 'r') as f:
|
||||
data = json.load(f)
|
||||
test_id = data.get('test_id')
|
||||
if test_id:
|
||||
results[test_id] = data
|
||||
|
||||
# Try to load the corresponding memory samples
|
||||
csv_path = self.reports_dir / f"memory_samples_{test_id}.csv"
|
||||
if csv_path.exists():
|
||||
try:
|
||||
memory_df = pd.read_csv(csv_path)
|
||||
results[test_id]['memory_samples'] = memory_df
|
||||
except Exception as e:
|
||||
console.print(f"[yellow]Warning: Could not load memory samples for {test_id}: {e}[/yellow]")
|
||||
except Exception as e:
|
||||
console.print(f"[red]Error loading {file_path}: {e}[/red]")
|
||||
|
||||
console.print(f"Loaded {len(results)} test results")
|
||||
return results
|
||||
|
||||
def generate_summary_table(self, results):
|
||||
"""Generate a summary table of test results.
|
||||
|
||||
Args:
|
||||
results: Dictionary mapping test IDs to result data
|
||||
|
||||
Returns:
|
||||
Rich Table object
|
||||
"""
|
||||
table = Table(title="Crawl4AI Stress Test Summary", show_header=True)
|
||||
|
||||
# Define columns
|
||||
table.add_column("Test ID", style="cyan")
|
||||
table.add_column("Date", style="bright_green")
|
||||
table.add_column("URLs", justify="right")
|
||||
table.add_column("Workers", justify="right")
|
||||
table.add_column("Success %", justify="right")
|
||||
table.add_column("Time (s)", justify="right")
|
||||
table.add_column("Mem Growth", justify="right")
|
||||
table.add_column("URLs/sec", justify="right")
|
||||
|
||||
# Add rows
|
||||
for test_id, data in sorted(results.items(), key=lambda x: x[0], reverse=True):
|
||||
# Parse timestamp from test_id
|
||||
try:
|
||||
date_str = datetime.strptime(test_id, "%Y%m%d_%H%M%S").strftime("%Y-%m-%d %H:%M")
|
||||
except:
|
||||
date_str = "Unknown"
|
||||
|
||||
# Calculate success percentage
|
||||
total_urls = data.get('url_count', 0)
|
||||
successful = data.get('successful_urls', 0)
|
||||
success_pct = (successful / total_urls * 100) if total_urls > 0 else 0
|
||||
|
||||
# Calculate memory growth if available
|
||||
mem_growth = "N/A"
|
||||
if 'memory_samples' in data:
|
||||
samples = data['memory_samples']
|
||||
if len(samples) >= 2:
|
||||
# Try to extract numeric values from memory_info strings
|
||||
try:
|
||||
first_mem = float(samples.iloc[0]['memory_info'].split()[0])
|
||||
last_mem = float(samples.iloc[-1]['memory_info'].split()[0])
|
||||
mem_growth = f"{last_mem - first_mem:.1f} MB"
|
||||
except:
|
||||
pass
|
||||
|
||||
# Calculate URLs per second
|
||||
time_taken = data.get('total_time_seconds', 0)
|
||||
urls_per_sec = total_urls / time_taken if time_taken > 0 else 0
|
||||
|
||||
table.add_row(
|
||||
test_id,
|
||||
date_str,
|
||||
str(total_urls),
|
||||
str(data.get('workers', 'N/A')),
|
||||
f"{success_pct:.1f}%",
|
||||
f"{data.get('total_time_seconds', 0):.2f}",
|
||||
mem_growth,
|
||||
f"{urls_per_sec:.1f}"
|
||||
)
|
||||
|
||||
return table
|
||||
|
||||
def generate_performance_chart(self, results, output_file=None):
|
||||
"""Generate a performance comparison chart.
|
||||
|
||||
Args:
|
||||
results: Dictionary mapping test IDs to result data
|
||||
output_file: File path to save the chart
|
||||
|
||||
Returns:
|
||||
Path to the saved chart file or None if visualization is not available
|
||||
"""
|
||||
if not VISUALIZATION_AVAILABLE:
|
||||
console.print("[yellow]Skipping performance chart - visualization dependencies not available[/yellow]")
|
||||
return None
|
||||
|
||||
# Extract relevant data
|
||||
data = []
|
||||
for test_id, result in results.items():
|
||||
urls = result.get('url_count', 0)
|
||||
workers = result.get('workers', 0)
|
||||
time_taken = result.get('total_time_seconds', 0)
|
||||
urls_per_sec = urls / time_taken if time_taken > 0 else 0
|
||||
|
||||
# Parse timestamp from test_id for sorting
|
||||
try:
|
||||
timestamp = datetime.strptime(test_id, "%Y%m%d_%H%M%S")
|
||||
data.append({
|
||||
'test_id': test_id,
|
||||
'timestamp': timestamp,
|
||||
'urls': urls,
|
||||
'workers': workers,
|
||||
'time_seconds': time_taken,
|
||||
'urls_per_sec': urls_per_sec
|
||||
})
|
||||
except:
|
||||
console.print(f"[yellow]Warning: Could not parse timestamp from {test_id}[/yellow]")
|
||||
|
||||
if not data:
|
||||
console.print("[yellow]No valid data for performance chart[/yellow]")
|
||||
return None
|
||||
|
||||
# Convert to DataFrame and sort by timestamp
|
||||
df = pd.DataFrame(data)
|
||||
df = df.sort_values('timestamp')
|
||||
|
||||
# Create the plot
|
||||
fig, ax1 = plt.subplots(figsize=(12, 6))
|
||||
|
||||
# Plot URLs per second as bars with properly set x-axis
|
||||
x_pos = range(len(df['test_id']))
|
||||
bars = ax1.bar(x_pos, df['urls_per_sec'], color='#88c0d0', alpha=0.8)
|
||||
ax1.set_ylabel('URLs per Second', color='#88c0d0')
|
||||
ax1.tick_params(axis='y', labelcolor='#88c0d0')
|
||||
|
||||
# Properly set x-axis labels
|
||||
ax1.set_xticks(x_pos)
|
||||
ax1.set_xticklabels(df['test_id'].tolist(), rotation=45, ha='right')
|
||||
|
||||
# Add worker count as text on each bar
|
||||
for i, bar in enumerate(bars):
|
||||
height = bar.get_height()
|
||||
workers = df.iloc[i]['workers']
|
||||
ax1.text(i, height + 0.1,
|
||||
f'W: {workers}', ha='center', va='bottom', fontsize=9, color='#e0e0e0')
|
||||
|
||||
# Add a second y-axis for total URLs
|
||||
ax2 = ax1.twinx()
|
||||
ax2.plot(x_pos, df['urls'], '-', color='#bf616a', alpha=0.8, markersize=6, marker='o')
|
||||
ax2.set_ylabel('Total URLs', color='#bf616a')
|
||||
ax2.tick_params(axis='y', labelcolor='#bf616a')
|
||||
|
||||
# Set title and layout
|
||||
plt.title('Crawl4AI Performance Benchmarks')
|
||||
plt.tight_layout()
|
||||
|
||||
# Save the figure
|
||||
if output_file is None:
|
||||
output_file = self.output_dir / "performance_comparison.png"
|
||||
plt.savefig(output_file, dpi=100, bbox_inches='tight')
|
||||
plt.close()
|
||||
|
||||
return output_file
|
||||
|
||||
def generate_memory_charts(self, results, output_prefix=None):
|
||||
"""Generate memory usage charts for each test.
|
||||
|
||||
Args:
|
||||
results: Dictionary mapping test IDs to result data
|
||||
output_prefix: Prefix for output file names
|
||||
|
||||
Returns:
|
||||
List of paths to the saved chart files
|
||||
"""
|
||||
if not VISUALIZATION_AVAILABLE:
|
||||
console.print("[yellow]Skipping memory charts - visualization dependencies not available[/yellow]")
|
||||
return []
|
||||
|
||||
output_files = []
|
||||
|
||||
for test_id, result in results.items():
|
||||
if 'memory_samples' not in result:
|
||||
continue
|
||||
|
||||
memory_df = result['memory_samples']
|
||||
|
||||
# Check if we have enough data points
|
||||
if len(memory_df) < 2:
|
||||
continue
|
||||
|
||||
# Try to extract numeric values from memory_info strings
|
||||
try:
|
||||
memory_values = []
|
||||
for mem_str in memory_df['memory_info']:
|
||||
# Extract the number from strings like "142.8 MB"
|
||||
value = float(mem_str.split()[0])
|
||||
memory_values.append(value)
|
||||
|
||||
memory_df['memory_mb'] = memory_values
|
||||
except Exception as e:
|
||||
console.print(f"[yellow]Could not parse memory values for {test_id}: {e}[/yellow]")
|
||||
continue
|
||||
|
||||
# Create the plot
|
||||
plt.figure(figsize=(10, 6))
|
||||
|
||||
# Plot memory usage over time
|
||||
plt.plot(memory_df['elapsed_seconds'], memory_df['memory_mb'],
|
||||
color='#88c0d0', marker='o', linewidth=2, markersize=4)
|
||||
|
||||
# Add annotations for chunk processing
|
||||
chunk_size = result.get('chunk_size', 0)
|
||||
url_count = result.get('url_count', 0)
|
||||
if chunk_size > 0 and url_count > 0:
|
||||
# Estimate chunk processing times
|
||||
num_chunks = (url_count + chunk_size - 1) // chunk_size # Ceiling division
|
||||
total_time = result.get('total_time_seconds', memory_df['elapsed_seconds'].max())
|
||||
chunk_times = np.linspace(0, total_time, num_chunks + 1)[1:]
|
||||
|
||||
for i, time_point in enumerate(chunk_times):
|
||||
if time_point <= memory_df['elapsed_seconds'].max():
|
||||
plt.axvline(x=time_point, color='#4c566a', linestyle='--', alpha=0.6)
|
||||
plt.text(time_point, memory_df['memory_mb'].min(), f'Chunk {i+1}',
|
||||
rotation=90, verticalalignment='bottom', fontsize=8, color='#e0e0e0')
|
||||
|
||||
# Set labels and title
|
||||
plt.xlabel('Elapsed Time (seconds)', color='#e0e0e0')
|
||||
plt.ylabel('Memory Usage (MB)', color='#e0e0e0')
|
||||
plt.title(f'Memory Usage During Test {test_id}\n({url_count} URLs, {result.get("workers", "?")} Workers)',
|
||||
color='#e0e0e0')
|
||||
|
||||
# Add grid and set y-axis to start from zero
|
||||
plt.grid(True, alpha=0.3, color='#4c566a')
|
||||
|
||||
# Add test metadata as text
|
||||
info_text = (
|
||||
f"URLs: {url_count}\n"
|
||||
f"Workers: {result.get('workers', 'N/A')}\n"
|
||||
f"Chunk Size: {result.get('chunk_size', 'N/A')}\n"
|
||||
f"Total Time: {result.get('total_time_seconds', 0):.2f}s\n"
|
||||
)
|
||||
|
||||
# Calculate memory growth
|
||||
if len(memory_df) >= 2:
|
||||
first_mem = memory_df.iloc[0]['memory_mb']
|
||||
last_mem = memory_df.iloc[-1]['memory_mb']
|
||||
growth = last_mem - first_mem
|
||||
growth_rate = growth / result.get('total_time_seconds', 1)
|
||||
|
||||
info_text += f"Memory Growth: {growth:.1f} MB\n"
|
||||
info_text += f"Growth Rate: {growth_rate:.2f} MB/s"
|
||||
|
||||
plt.figtext(0.02, 0.02, info_text, fontsize=9, color='#e0e0e0',
|
||||
bbox=dict(facecolor='#3b4252', alpha=0.8, edgecolor='#4c566a'))
|
||||
|
||||
# Save the figure
|
||||
if output_prefix is None:
|
||||
output_file = self.output_dir / f"memory_chart_{test_id}.png"
|
||||
else:
|
||||
output_file = Path(f"{output_prefix}_memory_{test_id}.png")
|
||||
|
||||
plt.tight_layout()
|
||||
plt.savefig(output_file, dpi=100, bbox_inches='tight')
|
||||
plt.close()
|
||||
|
||||
output_files.append(output_file)
|
||||
|
||||
return output_files
|
||||
|
||||
def generate_comparison_report(self, results, title=None, output_file=None):
|
||||
"""Generate a comprehensive comparison report of multiple test runs.
|
||||
|
||||
Args:
|
||||
results: Dictionary mapping test IDs to result data
|
||||
title: Optional title for the report
|
||||
output_file: File path to save the report
|
||||
|
||||
Returns:
|
||||
Path to the saved report file
|
||||
"""
|
||||
if not results:
|
||||
console.print("[yellow]No results to generate comparison report[/yellow]")
|
||||
return None
|
||||
|
||||
if output_file is None:
|
||||
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
output_file = self.output_dir / f"comparison_report_{timestamp}.html"
|
||||
|
||||
# Create data for the report
|
||||
rows = []
|
||||
for test_id, data in results.items():
|
||||
# Calculate metrics
|
||||
urls = data.get('url_count', 0)
|
||||
workers = data.get('workers', 0)
|
||||
successful = data.get('successful_urls', 0)
|
||||
failed = data.get('failed_urls', 0)
|
||||
time_seconds = data.get('total_time_seconds', 0)
|
||||
|
||||
# Calculate additional metrics
|
||||
success_rate = (successful / urls) * 100 if urls > 0 else 0
|
||||
urls_per_second = urls / time_seconds if time_seconds > 0 else 0
|
||||
urls_per_worker = urls / workers if workers > 0 else 0
|
||||
|
||||
# Calculate memory growth if available
|
||||
mem_start = None
|
||||
mem_end = None
|
||||
mem_growth = None
|
||||
if 'memory_samples' in data:
|
||||
samples = data['memory_samples']
|
||||
if len(samples) >= 2:
|
||||
try:
|
||||
first_mem = float(samples.iloc[0]['memory_info'].split()[0])
|
||||
last_mem = float(samples.iloc[-1]['memory_info'].split()[0])
|
||||
mem_start = first_mem
|
||||
mem_end = last_mem
|
||||
mem_growth = last_mem - first_mem
|
||||
except:
|
||||
pass
|
||||
|
||||
# Parse timestamp from test_id
|
||||
try:
|
||||
timestamp = datetime.strptime(test_id, "%Y%m%d_%H%M%S")
|
||||
except:
|
||||
timestamp = None
|
||||
|
||||
rows.append({
|
||||
'test_id': test_id,
|
||||
'timestamp': timestamp,
|
||||
'date': timestamp.strftime("%Y-%m-%d %H:%M:%S") if timestamp else "Unknown",
|
||||
'urls': urls,
|
||||
'workers': workers,
|
||||
'chunk_size': data.get('chunk_size', 0),
|
||||
'successful': successful,
|
||||
'failed': failed,
|
||||
'success_rate': success_rate,
|
||||
'time_seconds': time_seconds,
|
||||
'urls_per_second': urls_per_second,
|
||||
'urls_per_worker': urls_per_worker,
|
||||
'memory_start': mem_start,
|
||||
'memory_end': mem_end,
|
||||
'memory_growth': mem_growth
|
||||
})
|
||||
|
||||
# Sort data by timestamp if possible
|
||||
if VISUALIZATION_AVAILABLE:
|
||||
# Convert to DataFrame and sort by timestamp
|
||||
df = pd.DataFrame(rows)
|
||||
if 'timestamp' in df.columns and not df['timestamp'].isna().all():
|
||||
df = df.sort_values('timestamp', ascending=False)
|
||||
else:
|
||||
# Simple sorting without pandas
|
||||
rows.sort(key=lambda x: x.get('timestamp', datetime.now()), reverse=True)
|
||||
df = None
|
||||
|
||||
# Generate HTML report
|
||||
html = []
|
||||
html.append('<!DOCTYPE html>')
|
||||
html.append('<html lang="en">')
|
||||
html.append('<head>')
|
||||
html.append('<meta charset="UTF-8">')
|
||||
html.append('<meta name="viewport" content="width=device-width, initial-scale=1.0">')
|
||||
html.append(f'<title>{title or "Crawl4AI Benchmark Comparison"}</title>')
|
||||
html.append('<style>')
|
||||
html.append('''
|
||||
body {
|
||||
font-family: Arial, sans-serif;
|
||||
line-height: 1.6;
|
||||
margin: 0;
|
||||
padding: 20px;
|
||||
max-width: 1200px;
|
||||
margin: 0 auto;
|
||||
color: #e0e0e0;
|
||||
background-color: #1e1e1e;
|
||||
}
|
||||
h1, h2, h3 {
|
||||
color: #81a1c1;
|
||||
}
|
||||
table {
|
||||
border-collapse: collapse;
|
||||
width: 100%;
|
||||
margin-bottom: 20px;
|
||||
}
|
||||
th, td {
|
||||
text-align: left;
|
||||
padding: 12px;
|
||||
border-bottom: 1px solid #444;
|
||||
}
|
||||
th {
|
||||
background-color: #2e3440;
|
||||
font-weight: bold;
|
||||
}
|
||||
tr:hover {
|
||||
background-color: #2e3440;
|
||||
}
|
||||
a {
|
||||
color: #88c0d0;
|
||||
text-decoration: none;
|
||||
}
|
||||
a:hover {
|
||||
text-decoration: underline;
|
||||
}
|
||||
.chart-container {
|
||||
margin: 30px 0;
|
||||
text-align: center;
|
||||
background-color: #2e3440;
|
||||
padding: 20px;
|
||||
border-radius: 8px;
|
||||
}
|
||||
.chart-container img {
|
||||
max-width: 100%;
|
||||
height: auto;
|
||||
border: 1px solid #444;
|
||||
box-shadow: 0 0 10px rgba(0,0,0,0.3);
|
||||
}
|
||||
.card {
|
||||
border: 1px solid #444;
|
||||
border-radius: 8px;
|
||||
padding: 15px;
|
||||
margin-bottom: 20px;
|
||||
background-color: #2e3440;
|
||||
box-shadow: 0 0 10px rgba(0,0,0,0.2);
|
||||
}
|
||||
.highlight {
|
||||
background-color: #3b4252;
|
||||
font-weight: bold;
|
||||
}
|
||||
.status-good {
|
||||
color: #a3be8c;
|
||||
}
|
||||
.status-warning {
|
||||
color: #ebcb8b;
|
||||
}
|
||||
.status-bad {
|
||||
color: #bf616a;
|
||||
}
|
||||
''')
|
||||
html.append('</style>')
|
||||
html.append('</head>')
|
||||
html.append('<body>')
|
||||
|
||||
# Header
|
||||
html.append(f'<h1>{title or "Crawl4AI Benchmark Comparison"}</h1>')
|
||||
html.append(f'<p>Report generated on {datetime.now().strftime("%Y-%m-%d %H:%M:%S")}</p>')
|
||||
|
||||
# Summary section
|
||||
html.append('<div class="card">')
|
||||
html.append('<h2>Summary</h2>')
|
||||
html.append('<p>This report compares the performance of Crawl4AI across multiple test runs.</p>')
|
||||
|
||||
# Summary metrics
|
||||
data_available = (VISUALIZATION_AVAILABLE and df is not None and not df.empty) or (not VISUALIZATION_AVAILABLE and len(rows) > 0)
|
||||
if data_available:
|
||||
# Get the latest test data
|
||||
if VISUALIZATION_AVAILABLE and df is not None and not df.empty:
|
||||
latest_test = df.iloc[0]
|
||||
latest_id = latest_test['test_id']
|
||||
else:
|
||||
latest_test = rows[0] # First row (already sorted by timestamp)
|
||||
latest_id = latest_test['test_id']
|
||||
|
||||
html.append('<h3>Latest Test Results</h3>')
|
||||
html.append('<ul>')
|
||||
html.append(f'<li><strong>Test ID:</strong> {latest_id}</li>')
|
||||
html.append(f'<li><strong>Date:</strong> {latest_test["date"]}</li>')
|
||||
html.append(f'<li><strong>URLs:</strong> {latest_test["urls"]}</li>')
|
||||
html.append(f'<li><strong>Workers:</strong> {latest_test["workers"]}</li>')
|
||||
html.append(f'<li><strong>Success Rate:</strong> {latest_test["success_rate"]:.1f}%</li>')
|
||||
html.append(f'<li><strong>Time:</strong> {latest_test["time_seconds"]:.2f} seconds</li>')
|
||||
html.append(f'<li><strong>Performance:</strong> {latest_test["urls_per_second"]:.1f} URLs/second</li>')
|
||||
|
||||
# Check memory growth (handle both pandas and dict mode)
|
||||
memory_growth_available = False
|
||||
if VISUALIZATION_AVAILABLE and df is not None:
|
||||
if pd.notna(latest_test["memory_growth"]):
|
||||
html.append(f'<li><strong>Memory Growth:</strong> {latest_test["memory_growth"]:.1f} MB</li>')
|
||||
memory_growth_available = True
|
||||
else:
|
||||
if latest_test["memory_growth"] is not None:
|
||||
html.append(f'<li><strong>Memory Growth:</strong> {latest_test["memory_growth"]:.1f} MB</li>')
|
||||
memory_growth_available = True
|
||||
|
||||
html.append('</ul>')
|
||||
|
||||
# If we have more than one test, show trend
|
||||
if (VISUALIZATION_AVAILABLE and df is not None and len(df) > 1) or (not VISUALIZATION_AVAILABLE and len(rows) > 1):
|
||||
if VISUALIZATION_AVAILABLE and df is not None:
|
||||
prev_test = df.iloc[1]
|
||||
else:
|
||||
prev_test = rows[1]
|
||||
|
||||
# Calculate performance change
|
||||
perf_change = ((latest_test["urls_per_second"] / prev_test["urls_per_second"]) - 1) * 100 if prev_test["urls_per_second"] > 0 else 0
|
||||
|
||||
status_class = ""
|
||||
if perf_change > 5:
|
||||
status_class = "status-good"
|
||||
elif perf_change < -5:
|
||||
status_class = "status-bad"
|
||||
|
||||
html.append('<h3>Performance Trend</h3>')
|
||||
html.append('<ul>')
|
||||
html.append(f'<li><strong>Performance Change:</strong> <span class="{status_class}">{perf_change:+.1f}%</span> compared to previous test</li>')
|
||||
|
||||
# Memory trend if available
|
||||
memory_trend_available = False
|
||||
if VISUALIZATION_AVAILABLE and df is not None:
|
||||
if pd.notna(latest_test["memory_growth"]) and pd.notna(prev_test["memory_growth"]):
|
||||
mem_change = latest_test["memory_growth"] - prev_test["memory_growth"]
|
||||
memory_trend_available = True
|
||||
else:
|
||||
if latest_test["memory_growth"] is not None and prev_test["memory_growth"] is not None:
|
||||
mem_change = latest_test["memory_growth"] - prev_test["memory_growth"]
|
||||
memory_trend_available = True
|
||||
|
||||
if memory_trend_available:
|
||||
mem_status = ""
|
||||
if mem_change < -1: # Improved (less growth)
|
||||
mem_status = "status-good"
|
||||
elif mem_change > 1: # Worse (more growth)
|
||||
mem_status = "status-bad"
|
||||
|
||||
html.append(f'<li><strong>Memory Trend:</strong> <span class="{mem_status}">{mem_change:+.1f} MB</span> change in memory growth</li>')
|
||||
|
||||
html.append('</ul>')
|
||||
|
||||
html.append('</div>')
|
||||
|
||||
# Generate performance chart if visualization is available
|
||||
if VISUALIZATION_AVAILABLE:
|
||||
perf_chart = self.generate_performance_chart(results)
|
||||
if perf_chart:
|
||||
html.append('<div class="chart-container">')
|
||||
html.append('<h2>Performance Comparison</h2>')
|
||||
html.append(f'<img src="{os.path.relpath(perf_chart, os.path.dirname(output_file))}" alt="Performance Comparison Chart">')
|
||||
html.append('</div>')
|
||||
else:
|
||||
html.append('<div class="chart-container">')
|
||||
html.append('<h2>Performance Comparison</h2>')
|
||||
html.append('<p>Charts not available - install visualization dependencies (pandas, matplotlib, seaborn) to enable.</p>')
|
||||
html.append('</div>')
|
||||
|
||||
# Generate memory charts if visualization is available
|
||||
if VISUALIZATION_AVAILABLE:
|
||||
memory_charts = self.generate_memory_charts(results)
|
||||
if memory_charts:
|
||||
html.append('<div class="chart-container">')
|
||||
html.append('<h2>Memory Usage</h2>')
|
||||
|
||||
for chart in memory_charts:
|
||||
test_id = chart.stem.split('_')[-1]
|
||||
html.append(f'<h3>Test {test_id}</h3>')
|
||||
html.append(f'<img src="{os.path.relpath(chart, os.path.dirname(output_file))}" alt="Memory Chart for {test_id}">')
|
||||
|
||||
html.append('</div>')
|
||||
else:
|
||||
html.append('<div class="chart-container">')
|
||||
html.append('<h2>Memory Usage</h2>')
|
||||
html.append('<p>Charts not available - install visualization dependencies (pandas, matplotlib, seaborn) to enable.</p>')
|
||||
html.append('</div>')
|
||||
|
||||
# Detailed results table
|
||||
html.append('<h2>Detailed Results</h2>')
|
||||
|
||||
# Add the results as an HTML table
|
||||
html.append('<table>')
|
||||
|
||||
# Table headers
|
||||
html.append('<tr>')
|
||||
for col in ['Test ID', 'Date', 'URLs', 'Workers', 'Success %', 'Time (s)', 'URLs/sec', 'Mem Growth (MB)']:
|
||||
html.append(f'<th>{col}</th>')
|
||||
html.append('</tr>')
|
||||
|
||||
# Table rows - handle both pandas DataFrame and list of dicts
|
||||
if VISUALIZATION_AVAILABLE and df is not None:
|
||||
# Using pandas DataFrame
|
||||
for _, row in df.iterrows():
|
||||
html.append('<tr>')
|
||||
html.append(f'<td>{row["test_id"]}</td>')
|
||||
html.append(f'<td>{row["date"]}</td>')
|
||||
html.append(f'<td>{row["urls"]}</td>')
|
||||
html.append(f'<td>{row["workers"]}</td>')
|
||||
html.append(f'<td>{row["success_rate"]:.1f}%</td>')
|
||||
html.append(f'<td>{row["time_seconds"]:.2f}</td>')
|
||||
html.append(f'<td>{row["urls_per_second"]:.1f}</td>')
|
||||
|
||||
# Memory growth cell
|
||||
if pd.notna(row["memory_growth"]):
|
||||
html.append(f'<td>{row["memory_growth"]:.1f}</td>')
|
||||
else:
|
||||
html.append('<td>N/A</td>')
|
||||
|
||||
html.append('</tr>')
|
||||
else:
|
||||
# Using list of dicts (when pandas is not available)
|
||||
for row in rows:
|
||||
html.append('<tr>')
|
||||
html.append(f'<td>{row["test_id"]}</td>')
|
||||
html.append(f'<td>{row["date"]}</td>')
|
||||
html.append(f'<td>{row["urls"]}</td>')
|
||||
html.append(f'<td>{row["workers"]}</td>')
|
||||
html.append(f'<td>{row["success_rate"]:.1f}%</td>')
|
||||
html.append(f'<td>{row["time_seconds"]:.2f}</td>')
|
||||
html.append(f'<td>{row["urls_per_second"]:.1f}</td>')
|
||||
|
||||
# Memory growth cell
|
||||
if row["memory_growth"] is not None:
|
||||
html.append(f'<td>{row["memory_growth"]:.1f}</td>')
|
||||
else:
|
||||
html.append('<td>N/A</td>')
|
||||
|
||||
html.append('</tr>')
|
||||
|
||||
html.append('</table>')
|
||||
|
||||
# Conclusion section
|
||||
html.append('<div class="card">')
|
||||
html.append('<h2>Conclusion</h2>')
|
||||
|
||||
if VISUALIZATION_AVAILABLE and df is not None and not df.empty:
|
||||
# Using pandas for statistics (when available)
|
||||
# Calculate some overall statistics
|
||||
avg_urls_per_sec = df['urls_per_second'].mean()
|
||||
max_urls_per_sec = df['urls_per_second'].max()
|
||||
|
||||
# Determine if we have a trend
|
||||
if len(df) > 1:
|
||||
trend_data = df.sort_values('timestamp')
|
||||
first_perf = trend_data.iloc[0]['urls_per_second']
|
||||
last_perf = trend_data.iloc[-1]['urls_per_second']
|
||||
|
||||
perf_change = ((last_perf / first_perf) - 1) * 100 if first_perf > 0 else 0
|
||||
|
||||
if perf_change > 10:
|
||||
trend_desc = "significantly improved"
|
||||
trend_class = "status-good"
|
||||
elif perf_change > 5:
|
||||
trend_desc = "improved"
|
||||
trend_class = "status-good"
|
||||
elif perf_change < -10:
|
||||
trend_desc = "significantly decreased"
|
||||
trend_class = "status-bad"
|
||||
elif perf_change < -5:
|
||||
trend_desc = "decreased"
|
||||
trend_class = "status-bad"
|
||||
else:
|
||||
trend_desc = "remained stable"
|
||||
trend_class = ""
|
||||
|
||||
html.append(f'<p>Overall performance has <span class="{trend_class}">{trend_desc}</span> over the test period.</p>')
|
||||
|
||||
html.append(f'<p>Average throughput: <strong>{avg_urls_per_sec:.1f}</strong> URLs/second</p>')
|
||||
html.append(f'<p>Maximum throughput: <strong>{max_urls_per_sec:.1f}</strong> URLs/second</p>')
|
||||
|
||||
# Memory leak assessment
|
||||
if 'memory_growth' in df.columns and not df['memory_growth'].isna().all():
|
||||
avg_growth = df['memory_growth'].mean()
|
||||
max_growth = df['memory_growth'].max()
|
||||
|
||||
if avg_growth < 5:
|
||||
leak_assessment = "No significant memory leaks detected"
|
||||
leak_class = "status-good"
|
||||
elif avg_growth < 10:
|
||||
leak_assessment = "Minor memory growth observed"
|
||||
leak_class = "status-warning"
|
||||
else:
|
||||
leak_assessment = "Potential memory leak detected"
|
||||
leak_class = "status-bad"
|
||||
|
||||
html.append(f'<p><span class="{leak_class}">{leak_assessment}</span>. Average memory growth: <strong>{avg_growth:.1f} MB</strong> per test.</p>')
|
||||
else:
|
||||
# Manual calculations without pandas
|
||||
if rows:
|
||||
# Calculate average and max throughput
|
||||
total_urls_per_sec = sum(row['urls_per_second'] for row in rows)
|
||||
avg_urls_per_sec = total_urls_per_sec / len(rows)
|
||||
max_urls_per_sec = max(row['urls_per_second'] for row in rows)
|
||||
|
||||
html.append(f'<p>Average throughput: <strong>{avg_urls_per_sec:.1f}</strong> URLs/second</p>')
|
||||
html.append(f'<p>Maximum throughput: <strong>{max_urls_per_sec:.1f}</strong> URLs/second</p>')
|
||||
|
||||
# Memory assessment (simplified without pandas)
|
||||
growth_values = [row['memory_growth'] for row in rows if row['memory_growth'] is not None]
|
||||
if growth_values:
|
||||
avg_growth = sum(growth_values) / len(growth_values)
|
||||
|
||||
if avg_growth < 5:
|
||||
leak_assessment = "No significant memory leaks detected"
|
||||
leak_class = "status-good"
|
||||
elif avg_growth < 10:
|
||||
leak_assessment = "Minor memory growth observed"
|
||||
leak_class = "status-warning"
|
||||
else:
|
||||
leak_assessment = "Potential memory leak detected"
|
||||
leak_class = "status-bad"
|
||||
|
||||
html.append(f'<p><span class="{leak_class}">{leak_assessment}</span>. Average memory growth: <strong>{avg_growth:.1f} MB</strong> per test.</p>')
|
||||
else:
|
||||
html.append('<p>No test data available for analysis.</p>')
|
||||
|
||||
html.append('</div>')
|
||||
|
||||
# Footer
|
||||
html.append('<div style="margin-top: 30px; text-align: center; color: #777; font-size: 0.9em;">')
|
||||
html.append('<p>Generated by Crawl4AI Benchmark Reporter</p>')
|
||||
html.append('</div>')
|
||||
|
||||
html.append('</body>')
|
||||
html.append('</html>')
|
||||
|
||||
# Write the HTML file
|
||||
with open(output_file, 'w') as f:
|
||||
f.write('\n'.join(html))
|
||||
|
||||
# Print a clickable link for terminals that support it (iTerm, VS Code, etc.)
|
||||
file_url = f"file://{os.path.abspath(output_file)}"
|
||||
console.print(f"[green]Comparison report saved to: {output_file}[/green]")
|
||||
console.print(f"[blue underline]Click to open report: {file_url}[/blue underline]")
|
||||
return output_file
|
||||
|
||||
def run(self, limit=None, output_file=None):
|
||||
"""Generate a full benchmark report.
|
||||
|
||||
Args:
|
||||
limit: Optional limit on number of most recent tests to include
|
||||
output_file: Optional output file path
|
||||
|
||||
Returns:
|
||||
Path to the generated report file
|
||||
"""
|
||||
# Load test results
|
||||
results = self.load_test_results(limit=limit)
|
||||
|
||||
if not results:
|
||||
console.print("[yellow]No test results found. Run some tests first.[/yellow]")
|
||||
return None
|
||||
|
||||
# Generate and display summary table
|
||||
summary_table = self.generate_summary_table(results)
|
||||
console.print(summary_table)
|
||||
|
||||
# Generate comparison report
|
||||
title = f"Crawl4AI Benchmark Report ({len(results)} test runs)"
|
||||
report_file = self.generate_comparison_report(results, title=title, output_file=output_file)
|
||||
|
||||
if report_file:
|
||||
console.print(f"[bold green]Report generated successfully: {report_file}[/bold green]")
|
||||
return report_file
|
||||
else:
|
||||
console.print("[bold red]Failed to generate report[/bold red]")
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
"""Main entry point for the benchmark reporter."""
|
||||
parser = argparse.ArgumentParser(description="Generate benchmark reports for Crawl4AI stress tests")
|
||||
|
||||
parser.add_argument("--reports-dir", type=str, default="reports",
|
||||
help="Directory containing test result files")
|
||||
parser.add_argument("--output-dir", type=str, default="benchmark_reports",
|
||||
help="Directory to save generated reports")
|
||||
parser.add_argument("--limit", type=int, default=None,
|
||||
help="Limit to most recent N test results")
|
||||
parser.add_argument("--output-file", type=str, default=None,
|
||||
help="Custom output file path for the report")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Create the benchmark reporter
|
||||
reporter = BenchmarkReporter(reports_dir=args.reports_dir, output_dir=args.output_dir)
|
||||
|
||||
# Generate the report
|
||||
report_file = reporter.run(limit=args.limit, output_file=args.output_file)
|
||||
|
||||
if report_file:
|
||||
print(f"Report generated at: {report_file}")
|
||||
return 0
|
||||
else:
|
||||
print("Failed to generate report")
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,34 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Hammer /crawl with many concurrent requests to prove GLOBAL_SEM works.
|
||||
"""
|
||||
|
||||
import asyncio, httpx, json, uuid, argparse
|
||||
|
||||
API = "http://localhost:8020/crawl"
|
||||
URLS_PER_CALL = 1 # keep it minimal so each arun() == 1 page
|
||||
CONCURRENT_CALLS = 20 # way above your cap
|
||||
|
||||
payload_template = {
|
||||
"browser_config": {"type": "BrowserConfig", "params": {"headless": True}},
|
||||
"crawler_config": {
|
||||
"type": "CrawlerRunConfig",
|
||||
"params": {"cache_mode": "BYPASS", "verbose": False},
|
||||
}
|
||||
}
|
||||
|
||||
async def one_call(client):
|
||||
payload = payload_template.copy()
|
||||
payload["urls"] = [f"https://httpbin.org/anything/{uuid.uuid4()}"]
|
||||
r = await client.post(API, json=payload)
|
||||
r.raise_for_status()
|
||||
return r.json()["server_peak_memory_mb"]
|
||||
|
||||
async def main():
|
||||
async with httpx.AsyncClient(timeout=60) as client:
|
||||
tasks = [asyncio.create_task(one_call(client)) for _ in range(CONCURRENT_CALLS)]
|
||||
mem_usages = await asyncio.gather(*tasks)
|
||||
print("Calls finished OK, server peaks reported:", mem_usages)
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,4 @@
|
||||
pandas>=1.5.0
|
||||
matplotlib>=3.5.0
|
||||
seaborn>=0.12.0
|
||||
rich>=12.0.0
|
||||
Executable
+259
@@ -0,0 +1,259 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Run a complete Crawl4AI benchmark test using test_stress_sdk.py and generate a report.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import glob
|
||||
import argparse
|
||||
import subprocess
|
||||
import time
|
||||
from datetime import datetime
|
||||
|
||||
from rich.console import Console
|
||||
from rich.text import Text
|
||||
|
||||
console = Console()
|
||||
|
||||
# Updated TEST_CONFIGS to use max_sessions
|
||||
TEST_CONFIGS = {
|
||||
"quick": {"urls": 50, "max_sessions": 4, "chunk_size": 10, "description": "Quick test (50 URLs, 4 sessions)"},
|
||||
"small": {"urls": 100, "max_sessions": 8, "chunk_size": 20, "description": "Small test (100 URLs, 8 sessions)"},
|
||||
"medium": {"urls": 500, "max_sessions": 16, "chunk_size": 50, "description": "Medium test (500 URLs, 16 sessions)"},
|
||||
"large": {"urls": 1000, "max_sessions": 32, "chunk_size": 100,"description": "Large test (1000 URLs, 32 sessions)"},
|
||||
"extreme": {"urls": 2000, "max_sessions": 64, "chunk_size": 200,"description": "Extreme test (2000 URLs, 64 sessions)"},
|
||||
}
|
||||
|
||||
# Arguments to forward directly if present in custom_args
|
||||
FORWARD_ARGS = {
|
||||
"urls": "--urls",
|
||||
"max_sessions": "--max-sessions",
|
||||
"chunk_size": "--chunk-size",
|
||||
"port": "--port",
|
||||
"monitor_mode": "--monitor-mode",
|
||||
}
|
||||
# Boolean flags to forward if True
|
||||
FORWARD_FLAGS = {
|
||||
"stream": "--stream",
|
||||
"use_rate_limiter": "--use-rate-limiter",
|
||||
"keep_server_alive": "--keep-server-alive",
|
||||
"use_existing_site": "--use-existing-site",
|
||||
"skip_generation": "--skip-generation",
|
||||
"keep_site": "--keep-site",
|
||||
"clean_reports": "--clean-reports", # Note: clean behavior is handled here, but pass flag if needed
|
||||
"clean_site": "--clean-site", # Note: clean behavior is handled here, but pass flag if needed
|
||||
}
|
||||
|
||||
def run_benchmark(config_name, custom_args=None, compare=True, clean=False):
|
||||
"""Runs the stress test and optionally the report generator."""
|
||||
if config_name not in TEST_CONFIGS and config_name != "custom":
|
||||
console.print(f"[bold red]Unknown configuration: {config_name}[/bold red]")
|
||||
return False
|
||||
|
||||
# Print header
|
||||
title = "Crawl4AI SDK Benchmark Test"
|
||||
if config_name != "custom":
|
||||
title += f" - {TEST_CONFIGS[config_name]['description']}"
|
||||
else:
|
||||
# Safely get custom args for title
|
||||
urls = custom_args.get('urls', '?') if custom_args else '?'
|
||||
sessions = custom_args.get('max_sessions', '?') if custom_args else '?'
|
||||
title += f" - Custom ({urls} URLs, {sessions} sessions)"
|
||||
|
||||
console.print(f"\n[bold blue]{title}[/bold blue]")
|
||||
console.print("=" * (len(title) + 4)) # Adjust underline length
|
||||
|
||||
console.print("\n[bold white]Preparing test...[/bold white]")
|
||||
|
||||
# --- Command Construction ---
|
||||
# Use the new script name
|
||||
cmd = ["python", "test_stress_sdk.py"]
|
||||
|
||||
# Apply config or custom args
|
||||
args_to_use = {}
|
||||
if config_name != "custom":
|
||||
args_to_use = TEST_CONFIGS[config_name].copy()
|
||||
# If custom args are provided (e.g., boolean flags), overlay them
|
||||
if custom_args:
|
||||
args_to_use.update(custom_args)
|
||||
elif custom_args: # Custom config
|
||||
args_to_use = custom_args.copy()
|
||||
|
||||
# Add arguments with values
|
||||
for key, arg_name in FORWARD_ARGS.items():
|
||||
if key in args_to_use:
|
||||
cmd.extend([arg_name, str(args_to_use[key])])
|
||||
|
||||
# Add boolean flags
|
||||
for key, flag_name in FORWARD_FLAGS.items():
|
||||
if args_to_use.get(key, False): # Check if key exists and is True
|
||||
# Special handling for clean flags - apply locally, don't forward?
|
||||
# Decide if test_stress_sdk.py also needs --clean flags or if run_benchmark handles it.
|
||||
# For now, let's assume run_benchmark handles cleaning based on its own --clean flag.
|
||||
# We'll forward other flags.
|
||||
if key not in ["clean_reports", "clean_site"]:
|
||||
cmd.append(flag_name)
|
||||
|
||||
# Handle the top-level --clean flag for run_benchmark
|
||||
if clean:
|
||||
# Pass clean flags to the stress test script as well, if needed
|
||||
# This assumes test_stress_sdk.py also uses --clean-reports and --clean-site
|
||||
cmd.append("--clean-reports")
|
||||
cmd.append("--clean-site")
|
||||
console.print("[yellow]Applying --clean: Cleaning reports and site before test.[/yellow]")
|
||||
# Actual cleaning logic might reside here or be delegated entirely
|
||||
|
||||
console.print(f"\n[bold white]Running stress test:[/bold white] {' '.join(cmd)}")
|
||||
start = time.time()
|
||||
|
||||
# Execute the stress test script
|
||||
# Use Popen to stream output
|
||||
try:
|
||||
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, encoding='utf-8', errors='replace')
|
||||
while True:
|
||||
line = proc.stdout.readline()
|
||||
if not line:
|
||||
break
|
||||
console.print(line.rstrip()) # Print line by line
|
||||
proc.wait() # Wait for the process to complete
|
||||
except FileNotFoundError:
|
||||
console.print(f"[bold red]Error: Script 'test_stress_sdk.py' not found. Make sure it's in the correct directory.[/bold red]")
|
||||
return False
|
||||
except Exception as e:
|
||||
console.print(f"[bold red]Error running stress test subprocess: {e}[/bold red]")
|
||||
return False
|
||||
|
||||
|
||||
if proc.returncode != 0:
|
||||
console.print(f"[bold red]Stress test failed with exit code {proc.returncode}[/bold red]")
|
||||
return False
|
||||
|
||||
duration = time.time() - start
|
||||
console.print(f"[bold green]Stress test completed in {duration:.1f} seconds[/bold green]")
|
||||
|
||||
# --- Report Generation (Optional) ---
|
||||
if compare:
|
||||
# Assuming benchmark_report.py exists and works with the generated reports
|
||||
report_script = "benchmark_report.py" # Keep configurable if needed
|
||||
report_cmd = ["python", report_script]
|
||||
console.print(f"\n[bold white]Generating benchmark report: {' '.join(report_cmd)}[/bold white]")
|
||||
|
||||
# Run the report command and capture output
|
||||
try:
|
||||
report_proc = subprocess.run(report_cmd, capture_output=True, text=True, check=False, encoding='utf-8', errors='replace') # Use check=False to handle potential errors
|
||||
|
||||
# Print the captured output from benchmark_report.py
|
||||
if report_proc.stdout:
|
||||
console.print("\n" + report_proc.stdout)
|
||||
if report_proc.stderr:
|
||||
console.print("[yellow]Report generator stderr:[/yellow]\n" + report_proc.stderr)
|
||||
|
||||
if report_proc.returncode != 0:
|
||||
console.print(f"[bold yellow]Benchmark report generation script '{report_script}' failed with exit code {report_proc.returncode}[/bold yellow]")
|
||||
# Don't return False here, test itself succeeded
|
||||
else:
|
||||
console.print(f"[bold green]Benchmark report script '{report_script}' completed.[/bold green]")
|
||||
|
||||
# Find and print clickable links to the reports
|
||||
# Assuming reports are saved in 'benchmark_reports' by benchmark_report.py
|
||||
report_dir = "benchmark_reports"
|
||||
if os.path.isdir(report_dir):
|
||||
report_files = glob.glob(os.path.join(report_dir, "comparison_report_*.html"))
|
||||
if report_files:
|
||||
try:
|
||||
latest_report = max(report_files, key=os.path.getctime)
|
||||
report_path = os.path.abspath(latest_report)
|
||||
report_url = pathlib.Path(report_path).as_uri() # Better way to create file URI
|
||||
console.print(f"[bold cyan]Click to open report: [link={report_url}]{report_url}[/link][/bold cyan]")
|
||||
except Exception as e:
|
||||
console.print(f"[yellow]Could not determine latest report: {e}[/yellow]")
|
||||
|
||||
chart_files = glob.glob(os.path.join(report_dir, "memory_chart_*.png"))
|
||||
if chart_files:
|
||||
try:
|
||||
latest_chart = max(chart_files, key=os.path.getctime)
|
||||
chart_path = os.path.abspath(latest_chart)
|
||||
chart_url = pathlib.Path(chart_path).as_uri()
|
||||
console.print(f"[cyan]Memory chart: [link={chart_url}]{chart_url}[/link][/cyan]")
|
||||
except Exception as e:
|
||||
console.print(f"[yellow]Could not determine latest chart: {e}[/yellow]")
|
||||
else:
|
||||
console.print(f"[yellow]Benchmark report directory '{report_dir}' not found. Cannot link reports.[/yellow]")
|
||||
|
||||
except FileNotFoundError:
|
||||
console.print(f"[bold red]Error: Report script '{report_script}' not found.[/bold red]")
|
||||
except Exception as e:
|
||||
console.print(f"[bold red]Error running report generation subprocess: {e}[/bold red]")
|
||||
|
||||
|
||||
# Prompt to exit
|
||||
console.print("\n[bold green]Benchmark run finished. Press Enter to exit.[/bold green]")
|
||||
try:
|
||||
input() # Wait for user input
|
||||
except EOFError:
|
||||
pass # Handle case where input is piped or unavailable
|
||||
|
||||
return True
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Run a Crawl4AI SDK benchmark test and generate a report")
|
||||
|
||||
# --- Arguments ---
|
||||
parser.add_argument("config", choices=list(TEST_CONFIGS) + ["custom"],
|
||||
help="Test configuration: quick, small, medium, large, extreme, or custom")
|
||||
|
||||
# Arguments for 'custom' config or to override presets
|
||||
parser.add_argument("--urls", type=int, help="Number of URLs")
|
||||
parser.add_argument("--max-sessions", type=int, help="Max concurrent sessions (replaces --workers)")
|
||||
parser.add_argument("--chunk-size", type=int, help="URLs per batch (for non-stream logging)")
|
||||
parser.add_argument("--port", type=int, help="HTTP server port")
|
||||
parser.add_argument("--monitor-mode", type=str, choices=["DETAILED", "AGGREGATED"], help="Monitor display mode")
|
||||
|
||||
# Boolean flags / options
|
||||
parser.add_argument("--stream", action="store_true", help="Enable streaming results (disables batch logging)")
|
||||
parser.add_argument("--use-rate-limiter", action="store_true", help="Enable basic rate limiter")
|
||||
parser.add_argument("--no-report", action="store_true", help="Skip generating comparison report")
|
||||
parser.add_argument("--clean", action="store_true", help="Clean up reports and site before running")
|
||||
parser.add_argument("--keep-server-alive", action="store_true", help="Keep HTTP server running after test")
|
||||
parser.add_argument("--use-existing-site", action="store_true", help="Use existing site on specified port")
|
||||
parser.add_argument("--skip-generation", action="store_true", help="Use existing site files without regenerating")
|
||||
parser.add_argument("--keep-site", action="store_true", help="Keep generated site files after test")
|
||||
# Removed url_level_logging as it's implicitly handled by stream/batch mode now
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
custom_args = {}
|
||||
|
||||
# Populate custom_args from explicit command-line args
|
||||
if args.urls is not None: custom_args["urls"] = args.urls
|
||||
if args.max_sessions is not None: custom_args["max_sessions"] = args.max_sessions
|
||||
if args.chunk_size is not None: custom_args["chunk_size"] = args.chunk_size
|
||||
if args.port is not None: custom_args["port"] = args.port
|
||||
if args.monitor_mode is not None: custom_args["monitor_mode"] = args.monitor_mode
|
||||
if args.stream: custom_args["stream"] = True
|
||||
if args.use_rate_limiter: custom_args["use_rate_limiter"] = True
|
||||
if args.keep_server_alive: custom_args["keep_server_alive"] = True
|
||||
if args.use_existing_site: custom_args["use_existing_site"] = True
|
||||
if args.skip_generation: custom_args["skip_generation"] = True
|
||||
if args.keep_site: custom_args["keep_site"] = True
|
||||
# Clean flags are handled by the 'clean' argument passed to run_benchmark
|
||||
|
||||
# Validate custom config requirements
|
||||
if args.config == "custom":
|
||||
required_custom = ["urls", "max_sessions", "chunk_size"]
|
||||
missing = [f"--{arg}" for arg in required_custom if arg not in custom_args]
|
||||
if missing:
|
||||
console.print(f"[bold red]Error: 'custom' config requires: {', '.join(missing)}[/bold red]")
|
||||
return 1
|
||||
|
||||
success = run_benchmark(
|
||||
config_name=args.config,
|
||||
custom_args=custom_args, # Pass all collected custom args
|
||||
compare=not args.no_report,
|
||||
clean=args.clean
|
||||
)
|
||||
return 0 if success else 1
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,168 @@
|
||||
"""
|
||||
Test script for the CrawlerMonitor component.
|
||||
This script simulates a crawler with multiple tasks to demonstrate the real-time monitoring capabilities.
|
||||
"""
|
||||
|
||||
import time
|
||||
import uuid
|
||||
import random
|
||||
import threading
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Add the parent directory to the path to import crawl4ai
|
||||
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "../..")))
|
||||
|
||||
from crawl4ai.components.crawler_monitor import CrawlerMonitor
|
||||
from crawl4ai.models import CrawlStatus
|
||||
|
||||
def simulate_crawler_task(monitor, task_id, url, simulate_failure=False):
|
||||
"""Simulate a crawler task with different states."""
|
||||
# Task starts in the QUEUED state
|
||||
wait_time = random.uniform(0.5, 3.0)
|
||||
time.sleep(wait_time)
|
||||
|
||||
# Update to IN_PROGRESS state
|
||||
monitor.update_task(
|
||||
task_id=task_id,
|
||||
status=CrawlStatus.IN_PROGRESS,
|
||||
start_time=time.time(),
|
||||
wait_time=wait_time
|
||||
)
|
||||
|
||||
# Simulate task running
|
||||
process_time = random.uniform(1.0, 5.0)
|
||||
for i in range(int(process_time * 2)):
|
||||
# Simulate memory usage changes
|
||||
memory_usage = random.uniform(5.0, 25.0)
|
||||
monitor.update_task(
|
||||
task_id=task_id,
|
||||
memory_usage=memory_usage,
|
||||
peak_memory=max(memory_usage, monitor.get_task_stats(task_id).get("peak_memory", 0))
|
||||
)
|
||||
time.sleep(0.5)
|
||||
|
||||
# Update to COMPLETED or FAILED state
|
||||
if simulate_failure and random.random() < 0.8: # 80% chance of failure if simulate_failure is True
|
||||
monitor.update_task(
|
||||
task_id=task_id,
|
||||
status=CrawlStatus.FAILED,
|
||||
end_time=time.time(),
|
||||
error_message="Simulated failure: Connection timeout",
|
||||
memory_usage=0.0
|
||||
)
|
||||
else:
|
||||
monitor.update_task(
|
||||
task_id=task_id,
|
||||
status=CrawlStatus.COMPLETED,
|
||||
end_time=time.time(),
|
||||
memory_usage=0.0
|
||||
)
|
||||
|
||||
def update_queue_stats(monitor, num_queued_tasks):
|
||||
"""Update queue statistics periodically."""
|
||||
while monitor.is_running:
|
||||
queued_tasks = [
|
||||
task for task_id, task in monitor.get_all_task_stats().items()
|
||||
if task["status"] == CrawlStatus.QUEUED.name
|
||||
]
|
||||
|
||||
total_queued = len(queued_tasks)
|
||||
|
||||
if total_queued > 0:
|
||||
current_time = time.time()
|
||||
wait_times = [
|
||||
current_time - task.get("enqueue_time", current_time)
|
||||
for task in queued_tasks
|
||||
]
|
||||
highest_wait_time = max(wait_times) if wait_times else 0.0
|
||||
avg_wait_time = sum(wait_times) / len(wait_times) if wait_times else 0.0
|
||||
else:
|
||||
highest_wait_time = 0.0
|
||||
avg_wait_time = 0.0
|
||||
|
||||
monitor.update_queue_statistics(
|
||||
total_queued=total_queued,
|
||||
highest_wait_time=highest_wait_time,
|
||||
avg_wait_time=avg_wait_time
|
||||
)
|
||||
|
||||
# Simulate memory pressure based on number of active tasks
|
||||
active_tasks = len([
|
||||
task for task_id, task in monitor.get_all_task_stats().items()
|
||||
if task["status"] == CrawlStatus.IN_PROGRESS.name
|
||||
])
|
||||
|
||||
if active_tasks > 8:
|
||||
monitor.update_memory_status("CRITICAL")
|
||||
elif active_tasks > 4:
|
||||
monitor.update_memory_status("PRESSURE")
|
||||
else:
|
||||
monitor.update_memory_status("NORMAL")
|
||||
|
||||
time.sleep(1.0)
|
||||
|
||||
def test_crawler_monitor():
|
||||
"""Test the CrawlerMonitor with simulated crawler tasks."""
|
||||
# Total number of URLs to crawl
|
||||
total_urls = 50
|
||||
|
||||
# Initialize the monitor
|
||||
monitor = CrawlerMonitor(urls_total=total_urls, refresh_rate=0.5)
|
||||
|
||||
# Start the monitor
|
||||
monitor.start()
|
||||
|
||||
# Start thread to update queue statistics
|
||||
queue_stats_thread = threading.Thread(target=update_queue_stats, args=(monitor, total_urls))
|
||||
queue_stats_thread.daemon = True
|
||||
queue_stats_thread.start()
|
||||
|
||||
try:
|
||||
# Create task threads
|
||||
threads = []
|
||||
for i in range(total_urls):
|
||||
task_id = str(uuid.uuid4())
|
||||
url = f"https://example.com/page{i}"
|
||||
|
||||
# Add task to monitor
|
||||
monitor.add_task(task_id, url)
|
||||
|
||||
# Determine if this task should simulate failure
|
||||
simulate_failure = (i % 10 == 0) # Every 10th task
|
||||
|
||||
# Create and start thread for this task
|
||||
thread = threading.Thread(
|
||||
target=simulate_crawler_task,
|
||||
args=(monitor, task_id, url, simulate_failure)
|
||||
)
|
||||
thread.daemon = True
|
||||
threads.append(thread)
|
||||
|
||||
# Start threads with delay to simulate tasks being added over time
|
||||
batch_size = 5
|
||||
for i in range(0, len(threads), batch_size):
|
||||
batch = threads[i:i+batch_size]
|
||||
for thread in batch:
|
||||
thread.start()
|
||||
time.sleep(0.5) # Small delay between starting threads
|
||||
|
||||
# Wait a bit before starting the next batch
|
||||
time.sleep(2.0)
|
||||
|
||||
# Wait for all threads to complete
|
||||
for thread in threads:
|
||||
thread.join()
|
||||
|
||||
# Keep monitor running a bit longer to see the final state
|
||||
time.sleep(5.0)
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\nTest interrupted by user")
|
||||
finally:
|
||||
# Stop the monitor
|
||||
monitor.stop()
|
||||
print("\nCrawler monitor test completed")
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_crawler_monitor()
|
||||
@@ -0,0 +1,410 @@
|
||||
import asyncio
|
||||
import time
|
||||
import psutil
|
||||
import logging
|
||||
import random
|
||||
from typing import List, Dict
|
||||
import uuid
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Import your crawler components
|
||||
from crawl4ai.models import DisplayMode, CrawlStatus, CrawlResult
|
||||
from crawl4ai.async_configs import CrawlerRunConfig, BrowserConfig, CacheMode
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
from crawl4ai import MemoryAdaptiveDispatcher, CrawlerMonitor
|
||||
|
||||
# Global configuration
|
||||
STREAM = False # Toggle between streaming and non-streaming modes
|
||||
|
||||
# Configure logging to file only (to avoid breaking the rich display)
|
||||
os.makedirs("logs", exist_ok=True)
|
||||
file_handler = logging.FileHandler("logs/memory_stress_test.log")
|
||||
file_handler.setFormatter(logging.Formatter('%(asctime)s [%(levelname)s] %(message)s'))
|
||||
|
||||
# Root logger - only to file, not console
|
||||
root_logger = logging.getLogger()
|
||||
root_logger.setLevel(logging.INFO)
|
||||
root_logger.addHandler(file_handler)
|
||||
|
||||
# Our test logger also writes to file only
|
||||
logger = logging.getLogger("memory_stress_test")
|
||||
logger.setLevel(logging.INFO)
|
||||
logger.addHandler(file_handler)
|
||||
logger.propagate = False # Don't propagate to root logger
|
||||
|
||||
# Create a memory restrictor to simulate limited memory environment
|
||||
class MemorySimulator:
|
||||
def __init__(self, target_percent: float = 85.0, aggressive: bool = False):
|
||||
"""Simulates memory pressure by allocating memory"""
|
||||
self.target_percent = target_percent
|
||||
self.memory_blocks: List[bytearray] = []
|
||||
self.aggressive = aggressive
|
||||
|
||||
def apply_pressure(self, additional_percent: float = 0.0):
|
||||
"""Fill memory until we reach target percentage"""
|
||||
current_percent = psutil.virtual_memory().percent
|
||||
target = self.target_percent + additional_percent
|
||||
|
||||
if current_percent >= target:
|
||||
return # Already at target
|
||||
|
||||
logger.info(f"Current memory: {current_percent}%, target: {target}%")
|
||||
|
||||
# Calculate how much memory we need to allocate
|
||||
total_memory = psutil.virtual_memory().total
|
||||
target_usage = (target / 100.0) * total_memory
|
||||
current_usage = (current_percent / 100.0) * total_memory
|
||||
bytes_to_allocate = int(target_usage - current_usage)
|
||||
|
||||
if bytes_to_allocate <= 0:
|
||||
return
|
||||
|
||||
# Allocate in smaller chunks to avoid overallocation
|
||||
if self.aggressive:
|
||||
# Use larger chunks for faster allocation in aggressive mode
|
||||
chunk_size = min(bytes_to_allocate, 200 * 1024 * 1024) # 200MB chunks
|
||||
else:
|
||||
chunk_size = min(bytes_to_allocate, 50 * 1024 * 1024) # 50MB chunks
|
||||
|
||||
try:
|
||||
logger.info(f"Allocating {chunk_size / (1024 * 1024):.1f}MB to reach target memory usage")
|
||||
self.memory_blocks.append(bytearray(chunk_size))
|
||||
time.sleep(0.5) # Give system time to register the allocation
|
||||
except MemoryError:
|
||||
logger.warning("Unable to allocate more memory")
|
||||
|
||||
def release_pressure(self, percent: float = None):
|
||||
"""
|
||||
Release allocated memory
|
||||
If percent is specified, release that percentage of blocks
|
||||
"""
|
||||
if not self.memory_blocks:
|
||||
return
|
||||
|
||||
if percent is None:
|
||||
# Release all
|
||||
logger.info(f"Releasing all {len(self.memory_blocks)} memory blocks")
|
||||
self.memory_blocks.clear()
|
||||
else:
|
||||
# Release specified percentage
|
||||
blocks_to_release = int(len(self.memory_blocks) * (percent / 100.0))
|
||||
if blocks_to_release > 0:
|
||||
logger.info(f"Releasing {blocks_to_release} of {len(self.memory_blocks)} memory blocks ({percent}%)")
|
||||
self.memory_blocks = self.memory_blocks[blocks_to_release:]
|
||||
|
||||
def spike_pressure(self, duration: float = 5.0):
|
||||
"""
|
||||
Create a temporary spike in memory pressure then release
|
||||
Useful for forcing requeues
|
||||
"""
|
||||
logger.info(f"Creating memory pressure spike for {duration} seconds")
|
||||
# Save current blocks count
|
||||
initial_blocks = len(self.memory_blocks)
|
||||
|
||||
# Create spike with extra 5%
|
||||
self.apply_pressure(additional_percent=5.0)
|
||||
|
||||
# Schedule release after duration
|
||||
asyncio.create_task(self._delayed_release(duration, initial_blocks))
|
||||
|
||||
async def _delayed_release(self, delay: float, target_blocks: int):
|
||||
"""Helper for spike_pressure - releases extra blocks after delay"""
|
||||
await asyncio.sleep(delay)
|
||||
|
||||
# Remove blocks added since spike started
|
||||
if len(self.memory_blocks) > target_blocks:
|
||||
logger.info(f"Releasing memory spike ({len(self.memory_blocks) - target_blocks} blocks)")
|
||||
self.memory_blocks = self.memory_blocks[:target_blocks]
|
||||
|
||||
# Test statistics collector
|
||||
class TestResults:
|
||||
def __init__(self):
|
||||
self.start_time = time.time()
|
||||
self.completed_urls: List[str] = []
|
||||
self.failed_urls: List[str] = []
|
||||
self.requeued_count = 0
|
||||
self.memory_warnings = 0
|
||||
self.max_memory_usage = 0.0
|
||||
self.max_queue_size = 0
|
||||
self.max_wait_time = 0.0
|
||||
self.url_to_attempt: Dict[str, int] = {} # Track retries per URL
|
||||
|
||||
def log_summary(self):
|
||||
duration = time.time() - self.start_time
|
||||
logger.info("===== TEST SUMMARY =====")
|
||||
logger.info(f"Stream mode: {'ON' if STREAM else 'OFF'}")
|
||||
logger.info(f"Total duration: {duration:.1f} seconds")
|
||||
logger.info(f"Completed URLs: {len(self.completed_urls)}")
|
||||
logger.info(f"Failed URLs: {len(self.failed_urls)}")
|
||||
logger.info(f"Requeue events: {self.requeued_count}")
|
||||
logger.info(f"Memory warnings: {self.memory_warnings}")
|
||||
logger.info(f"Max memory usage: {self.max_memory_usage:.1f}%")
|
||||
logger.info(f"Max queue size: {self.max_queue_size}")
|
||||
logger.info(f"Max wait time: {self.max_wait_time:.1f} seconds")
|
||||
|
||||
# Log URLs with multiple attempts
|
||||
retried_urls = {url: count for url, count in self.url_to_attempt.items() if count > 1}
|
||||
if retried_urls:
|
||||
logger.info(f"URLs with retries: {len(retried_urls)}")
|
||||
# Log the top 5 most retried
|
||||
top_retries = sorted(retried_urls.items(), key=lambda x: x[1], reverse=True)[:5]
|
||||
for url, count in top_retries:
|
||||
logger.info(f" URL {url[-30:]} had {count} attempts")
|
||||
|
||||
# Write summary to a separate human-readable file
|
||||
with open("logs/test_summary.txt", "w") as f:
|
||||
f.write(f"Stream mode: {'ON' if STREAM else 'OFF'}\n")
|
||||
f.write(f"Total duration: {duration:.1f} seconds\n")
|
||||
f.write(f"Completed URLs: {len(self.completed_urls)}\n")
|
||||
f.write(f"Failed URLs: {len(self.failed_urls)}\n")
|
||||
f.write(f"Requeue events: {self.requeued_count}\n")
|
||||
f.write(f"Memory warnings: {self.memory_warnings}\n")
|
||||
f.write(f"Max memory usage: {self.max_memory_usage:.1f}%\n")
|
||||
f.write(f"Max queue size: {self.max_queue_size}\n")
|
||||
f.write(f"Max wait time: {self.max_wait_time:.1f} seconds\n")
|
||||
|
||||
# Custom monitor with stats tracking
|
||||
# Custom monitor that extends CrawlerMonitor with test-specific tracking
|
||||
class StressTestMonitor(CrawlerMonitor):
|
||||
def __init__(self, test_results: TestResults, **kwargs):
|
||||
# Initialize the parent CrawlerMonitor
|
||||
super().__init__(**kwargs)
|
||||
self.test_results = test_results
|
||||
|
||||
def update_memory_status(self, status: str):
|
||||
if status != self.memory_status:
|
||||
logger.info(f"Memory status changed: {self.memory_status} -> {status}")
|
||||
if "CRITICAL" in status or "PRESSURE" in status:
|
||||
self.test_results.memory_warnings += 1
|
||||
|
||||
# Track peak memory usage in test results
|
||||
current_memory = psutil.virtual_memory().percent
|
||||
self.test_results.max_memory_usage = max(self.test_results.max_memory_usage, current_memory)
|
||||
|
||||
# Call parent method to update the dashboard
|
||||
super().update_memory_status(status)
|
||||
|
||||
def update_queue_statistics(self, total_queued: int, highest_wait_time: float, avg_wait_time: float):
|
||||
# Track queue metrics in test results
|
||||
self.test_results.max_queue_size = max(self.test_results.max_queue_size, total_queued)
|
||||
self.test_results.max_wait_time = max(self.test_results.max_wait_time, highest_wait_time)
|
||||
|
||||
# Call parent method to update the dashboard
|
||||
super().update_queue_statistics(total_queued, highest_wait_time, avg_wait_time)
|
||||
|
||||
def update_task(self, task_id: str, **kwargs):
|
||||
# Track URL status changes for test results
|
||||
if task_id in self.stats:
|
||||
old_status = self.stats[task_id].status
|
||||
|
||||
# If this is a requeue event (requeued due to memory pressure)
|
||||
if 'error_message' in kwargs and 'requeued' in kwargs['error_message']:
|
||||
if not hasattr(self.stats[task_id], 'counted_requeue') or not self.stats[task_id].counted_requeue:
|
||||
self.test_results.requeued_count += 1
|
||||
self.stats[task_id].counted_requeue = True
|
||||
|
||||
# Track completion status for test results
|
||||
if 'status' in kwargs:
|
||||
new_status = kwargs['status']
|
||||
if old_status != new_status:
|
||||
if new_status == CrawlStatus.COMPLETED:
|
||||
if task_id not in self.test_results.completed_urls:
|
||||
self.test_results.completed_urls.append(task_id)
|
||||
elif new_status == CrawlStatus.FAILED:
|
||||
if task_id not in self.test_results.failed_urls:
|
||||
self.test_results.failed_urls.append(task_id)
|
||||
|
||||
# Call parent method to update the dashboard
|
||||
super().update_task(task_id, **kwargs)
|
||||
self.live.update(self._create_table())
|
||||
|
||||
# Generate test URLs - use example.com with unique paths to avoid browser caching
|
||||
def generate_test_urls(count: int) -> List[str]:
|
||||
urls = []
|
||||
for i in range(count):
|
||||
# Add random path and query parameters to create unique URLs
|
||||
path = f"/path/{uuid.uuid4()}"
|
||||
query = f"?test={i}&random={random.randint(1, 100000)}"
|
||||
urls.append(f"https://example.com{path}{query}")
|
||||
return urls
|
||||
|
||||
# Process result callback
|
||||
async def process_result(result, test_results: TestResults):
|
||||
# Track attempt counts
|
||||
if result.url not in test_results.url_to_attempt:
|
||||
test_results.url_to_attempt[result.url] = 1
|
||||
else:
|
||||
test_results.url_to_attempt[result.url] += 1
|
||||
|
||||
if "requeued" in result.error_message:
|
||||
test_results.requeued_count += 1
|
||||
logger.debug(f"Requeued due to memory pressure: {result.url}")
|
||||
elif result.success:
|
||||
test_results.completed_urls.append(result.url)
|
||||
logger.debug(f"Successfully processed: {result.url}")
|
||||
else:
|
||||
test_results.failed_urls.append(result.url)
|
||||
logger.warning(f"Failed to process: {result.url} - {result.error_message}")
|
||||
|
||||
# Process multiple results (used in non-streaming mode)
|
||||
async def process_results(results, test_results: TestResults):
|
||||
for result in results:
|
||||
await process_result(result, test_results)
|
||||
|
||||
# Main test function for extreme memory pressure simulation
|
||||
async def run_memory_stress_test(
|
||||
url_count: int = 100,
|
||||
target_memory_percent: float = 92.0, # Push to dangerous levels
|
||||
chunk_size: int = 20, # Larger chunks for more chaos
|
||||
aggressive: bool = False,
|
||||
spikes: bool = True
|
||||
):
|
||||
test_results = TestResults()
|
||||
memory_simulator = MemorySimulator(target_percent=target_memory_percent, aggressive=aggressive)
|
||||
|
||||
logger.info(f"Starting stress test with {url_count} URLs in {'STREAM' if STREAM else 'NON-STREAM'} mode")
|
||||
logger.info(f"Target memory usage: {target_memory_percent}%")
|
||||
|
||||
# First, elevate memory usage to create pressure
|
||||
logger.info("Creating initial memory pressure...")
|
||||
memory_simulator.apply_pressure()
|
||||
|
||||
# Create test URLs in chunks to simulate real-world crawling where URLs are discovered
|
||||
all_urls = generate_test_urls(url_count)
|
||||
url_chunks = [all_urls[i:i+chunk_size] for i in range(0, len(all_urls), chunk_size)]
|
||||
|
||||
# Set up the crawler components - low memory thresholds to create more requeues
|
||||
browser_config = BrowserConfig(headless=True, verbose=False)
|
||||
run_config = CrawlerRunConfig(
|
||||
cache_mode=CacheMode.BYPASS,
|
||||
verbose=False,
|
||||
stream=STREAM # Use the global STREAM variable to set mode
|
||||
)
|
||||
|
||||
# Create monitor with reference to test results
|
||||
monitor = StressTestMonitor(
|
||||
test_results=test_results,
|
||||
display_mode=DisplayMode.DETAILED,
|
||||
max_visible_rows=20,
|
||||
total_urls=url_count # Pass total URLs count
|
||||
)
|
||||
|
||||
# Create dispatcher with EXTREME settings - pure survival mode
|
||||
# These settings are designed to create a memory battleground
|
||||
dispatcher = MemoryAdaptiveDispatcher(
|
||||
memory_threshold_percent=63.0, # Start throttling at just 60% memory
|
||||
critical_threshold_percent=70.0, # Start requeuing at 70% - incredibly aggressive
|
||||
recovery_threshold_percent=55.0, # Only resume normal ops when plenty of memory available
|
||||
check_interval=0.1, # Check extremely frequently (100ms)
|
||||
max_session_permit=20 if aggressive else 10, # Double the concurrent sessions - pure chaos
|
||||
fairness_timeout=10.0, # Extremely low timeout - rapid priority changes
|
||||
monitor=monitor
|
||||
)
|
||||
|
||||
# Set up spike schedule if enabled
|
||||
if spikes:
|
||||
spike_intervals = []
|
||||
# Create 3-5 random spike times
|
||||
num_spikes = random.randint(3, 5)
|
||||
for _ in range(num_spikes):
|
||||
# Schedule spikes at random chunks
|
||||
chunk_index = random.randint(1, len(url_chunks) - 1)
|
||||
spike_intervals.append(chunk_index)
|
||||
logger.info(f"Scheduled memory spikes at chunks: {spike_intervals}")
|
||||
|
||||
try:
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
# Process URLs in chunks to simulate discovering URLs over time
|
||||
for chunk_index, url_chunk in enumerate(url_chunks):
|
||||
logger.info(f"Processing chunk {chunk_index+1}/{len(url_chunks)} ({len(url_chunk)} URLs)")
|
||||
|
||||
# Regular pressure increases
|
||||
if chunk_index % 2 == 0:
|
||||
logger.info("Increasing memory pressure...")
|
||||
memory_simulator.apply_pressure()
|
||||
|
||||
# Memory spike if scheduled for this chunk
|
||||
if spikes and chunk_index in spike_intervals:
|
||||
logger.info(f"⚠️ CREATING MASSIVE MEMORY SPIKE at chunk {chunk_index+1} ⚠️")
|
||||
# Create a nightmare scenario - multiple overlapping spikes
|
||||
memory_simulator.spike_pressure(duration=10.0) # 10-second spike
|
||||
|
||||
# 50% chance of double-spike (pure evil)
|
||||
if random.random() < 0.5:
|
||||
await asyncio.sleep(2.0) # Wait 2 seconds
|
||||
logger.info("💀 DOUBLE SPIKE - EXTREME MEMORY PRESSURE 💀")
|
||||
memory_simulator.spike_pressure(duration=8.0) # 8-second overlapping spike
|
||||
|
||||
if STREAM:
|
||||
# Stream mode - process results as they come in
|
||||
async for result in dispatcher.run_urls_stream(
|
||||
urls=url_chunk,
|
||||
crawler=crawler,
|
||||
config=run_config
|
||||
):
|
||||
await process_result(result, test_results)
|
||||
else:
|
||||
# Non-stream mode - get all results at once
|
||||
results = await dispatcher.run_urls(
|
||||
urls=url_chunk,
|
||||
crawler=crawler,
|
||||
config=run_config
|
||||
)
|
||||
await process_results(results, test_results)
|
||||
|
||||
# Simulate discovering more URLs while others are still processing
|
||||
await asyncio.sleep(1)
|
||||
|
||||
# RARELY release pressure - make the system fight for resources
|
||||
if chunk_index % 5 == 4: # Less frequent releases
|
||||
release_percent = random.choice([10, 15, 20]) # Smaller, inconsistent releases
|
||||
logger.info(f"Releasing {release_percent}% of memory blocks - brief respite")
|
||||
memory_simulator.release_pressure(percent=release_percent)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Test error: {str(e)}")
|
||||
raise
|
||||
finally:
|
||||
# Release memory pressure
|
||||
memory_simulator.release_pressure()
|
||||
# Log final results
|
||||
test_results.log_summary()
|
||||
|
||||
# Check for success criteria
|
||||
if len(test_results.completed_urls) + len(test_results.failed_urls) < url_count:
|
||||
logger.error(f"TEST FAILED: Not all URLs were processed. {url_count - len(test_results.completed_urls) - len(test_results.failed_urls)} URLs missing.")
|
||||
return False
|
||||
|
||||
logger.info("TEST PASSED: All URLs were processed without crashing.")
|
||||
return True
|
||||
|
||||
# Command-line entry point
|
||||
if __name__ == "__main__":
|
||||
# Parse command line arguments
|
||||
url_count = int(sys.argv[1]) if len(sys.argv) > 1 else 100
|
||||
target_memory = float(sys.argv[2]) if len(sys.argv) > 2 else 85.0
|
||||
|
||||
# Check if stream mode is specified
|
||||
if len(sys.argv) > 3:
|
||||
STREAM = sys.argv[3].lower() in ('true', 'yes', '1', 'stream')
|
||||
|
||||
# Check if aggressive mode is specified
|
||||
aggressive = False
|
||||
if len(sys.argv) > 4:
|
||||
aggressive = sys.argv[4].lower() in ('true', 'yes', '1', 'aggressive')
|
||||
|
||||
print(f"Starting test with {url_count} URLs, {target_memory}% memory target")
|
||||
print(f"Stream mode: {STREAM}, Aggressive: {aggressive}")
|
||||
print("Logs will be written to the logs directory")
|
||||
print("Live display starting now...")
|
||||
|
||||
# Run the test
|
||||
result = asyncio.run(run_memory_stress_test(
|
||||
url_count=url_count,
|
||||
target_memory_percent=target_memory,
|
||||
aggressive=aggressive
|
||||
))
|
||||
|
||||
# Exit with status code
|
||||
sys.exit(0 if result else 1)
|
||||
@@ -0,0 +1,36 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Quick sanity‑check for /config/dump endpoint.
|
||||
|
||||
Usage:
|
||||
python test_config_dump.py [http://localhost:8020]
|
||||
|
||||
If the server isn’t running, start it first:
|
||||
uvicorn deploy.docker.server:app --port 8020
|
||||
"""
|
||||
|
||||
import sys, json, textwrap, requests
|
||||
|
||||
# BASE = sys.argv[1] if len(sys.argv) > 1 else "http://localhost:8020"
|
||||
BASE = sys.argv[1] if len(sys.argv) > 1 else "http://localhost:11235"
|
||||
URL = f"{BASE.rstrip('/')}/config/dump"
|
||||
|
||||
CASES = [
|
||||
# --- CrawlRunConfig variants ---
|
||||
"CrawlerRunConfig()",
|
||||
"CrawlerRunConfig(stream=True, cache_mode=CacheMode.BYPASS)",
|
||||
"CrawlerRunConfig(js_only=True, wait_until='networkidle')",
|
||||
|
||||
# --- BrowserConfig variants ---
|
||||
"BrowserConfig()",
|
||||
"BrowserConfig(headless=False, extra_args=['--disable-gpu'])",
|
||||
"BrowserConfig(browser_mode='builtin', proxy_config={'server': 'http://1.2.3.4:8080'})",
|
||||
]
|
||||
|
||||
for code in CASES:
|
||||
print("\n=== POST:", code)
|
||||
resp = requests.post(URL, json={"code": code}, timeout=15)
|
||||
if resp.ok:
|
||||
print(json.dumps(resp.json(), indent=2)[:400] + "...")
|
||||
else:
|
||||
print("ERROR", resp.status_code, resp.text[:200])
|
||||
@@ -0,0 +1,520 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Stress test for Crawl4AI's Docker API server (/crawl and /crawl/stream endpoints).
|
||||
|
||||
This version targets a running Crawl4AI API server, sending concurrent requests
|
||||
to test its ability to handle multiple crawl jobs simultaneously.
|
||||
It uses httpx for async HTTP requests and logs results per batch of requests,
|
||||
including server-side memory usage reported by the API.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
import uuid
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
import os
|
||||
import shutil
|
||||
from typing import List, Dict, Optional, Union, AsyncGenerator, Tuple
|
||||
import httpx
|
||||
import pathlib # Import pathlib explicitly
|
||||
from rich.console import Console
|
||||
from rich.panel import Panel
|
||||
from rich.syntax import Syntax
|
||||
|
||||
# --- Constants ---
|
||||
DEFAULT_API_URL = "http://localhost:11235" # Default port
|
||||
DEFAULT_API_URL = "http://localhost:8020" # Default port
|
||||
DEFAULT_URL_COUNT = 100
|
||||
DEFAULT_MAX_CONCURRENT_REQUESTS = 1
|
||||
DEFAULT_CHUNK_SIZE = 10
|
||||
DEFAULT_REPORT_PATH = "reports_api"
|
||||
DEFAULT_STREAM_MODE = True
|
||||
REQUEST_TIMEOUT = 180.0
|
||||
|
||||
# Initialize Rich console
|
||||
console = Console()
|
||||
|
||||
# --- API Health Check (Unchanged) ---
|
||||
async def check_server_health(client: httpx.AsyncClient, health_endpoint: str = "/health"):
|
||||
"""Check if the API server is healthy."""
|
||||
console.print(f"[bold cyan]Checking API server health at {client.base_url}{health_endpoint}...[/]", end="")
|
||||
try:
|
||||
response = await client.get(health_endpoint, timeout=10.0)
|
||||
response.raise_for_status()
|
||||
health_data = response.json()
|
||||
version = health_data.get('version', 'N/A')
|
||||
console.print(f"[bold green] Server OK! Version: {version}[/]")
|
||||
return True
|
||||
except (httpx.RequestError, httpx.HTTPStatusError) as e:
|
||||
console.print(f"\n[bold red]Server health check FAILED:[/]")
|
||||
console.print(f"Error: {e}")
|
||||
console.print(f"Is the server running and accessible at {client.base_url}?")
|
||||
return False
|
||||
except Exception as e:
|
||||
console.print(f"\n[bold red]An unexpected error occurred during health check:[/]")
|
||||
console.print(e)
|
||||
return False
|
||||
|
||||
# --- API Stress Test Class ---
|
||||
class ApiStressTest:
|
||||
"""Orchestrates the stress test by sending concurrent requests to the API."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
api_url: str,
|
||||
url_count: int,
|
||||
max_concurrent_requests: int,
|
||||
chunk_size: int,
|
||||
report_path: str,
|
||||
stream_mode: bool,
|
||||
):
|
||||
self.api_base_url = api_url.rstrip('/')
|
||||
self.url_count = url_count
|
||||
self.max_concurrent_requests = max_concurrent_requests
|
||||
self.chunk_size = chunk_size
|
||||
self.report_path = pathlib.Path(report_path)
|
||||
self.report_path.mkdir(parents=True, exist_ok=True)
|
||||
self.stream_mode = stream_mode
|
||||
|
||||
# Ignore repo path and set it to current file path
|
||||
self.repo_path = pathlib.Path(__file__).parent.resolve()
|
||||
|
||||
|
||||
self.test_id = time.strftime("%Y%m%d_%H%M%S")
|
||||
self.results_summary = {
|
||||
"test_id": self.test_id, "api_url": api_url, "url_count": url_count,
|
||||
"max_concurrent_requests": max_concurrent_requests, "chunk_size": chunk_size,
|
||||
"stream_mode": stream_mode, "start_time": "", "end_time": "",
|
||||
"total_time_seconds": 0, "successful_requests": 0, "failed_requests": 0,
|
||||
"successful_urls": 0, "failed_urls": 0, "total_urls_processed": 0,
|
||||
"total_api_calls": 0,
|
||||
"server_memory_metrics": { # To store aggregated server memory info
|
||||
"batch_mode_avg_delta_mb": None,
|
||||
"batch_mode_max_delta_mb": None,
|
||||
"stream_mode_avg_max_snapshot_mb": None,
|
||||
"stream_mode_max_max_snapshot_mb": None,
|
||||
"samples": [] # Store individual request memory results
|
||||
}
|
||||
}
|
||||
self.http_client = httpx.AsyncClient(base_url=self.api_base_url, timeout=REQUEST_TIMEOUT, limits=httpx.Limits(max_connections=max_concurrent_requests + 5, max_keepalive_connections=max_concurrent_requests))
|
||||
|
||||
async def close_client(self):
|
||||
"""Close the httpx client."""
|
||||
await self.http_client.aclose()
|
||||
|
||||
async def run(self) -> Dict:
|
||||
"""Run the API stress test."""
|
||||
# No client memory tracker needed
|
||||
urls_to_process = [f"https://httpbin.org/anything/{uuid.uuid4()}" for _ in range(self.url_count)]
|
||||
url_chunks = [urls_to_process[i:i+self.chunk_size] for i in range(0, len(urls_to_process), self.chunk_size)]
|
||||
|
||||
self.results_summary["start_time"] = time.strftime("%Y-%m-%d %H:%M:%S")
|
||||
start_time = time.time()
|
||||
|
||||
console.print(f"\n[bold cyan]Crawl4AI API Stress Test - {self.url_count} URLs, {self.max_concurrent_requests} concurrent requests[/bold cyan]")
|
||||
console.print(f"[bold cyan]Target API:[/bold cyan] {self.api_base_url}, [bold cyan]Mode:[/bold cyan] {'Streaming' if self.stream_mode else 'Batch'}, [bold cyan]URLs per Request:[/bold cyan] {self.chunk_size}")
|
||||
# Removed client memory log
|
||||
|
||||
semaphore = asyncio.Semaphore(self.max_concurrent_requests)
|
||||
|
||||
# Updated Batch logging header
|
||||
console.print("\n[bold]API Request Batch Progress:[/bold]")
|
||||
# Adjusted spacing and added Peak
|
||||
console.print("[bold] Batch | Progress | SrvMem Peak / Δ|Max (MB) | Reqs/sec | S/F URLs | Time (s) | Status [/bold]")
|
||||
# Adjust separator length if needed, looks okay for now
|
||||
console.print("─" * 95)
|
||||
|
||||
# No client memory monitor task needed
|
||||
|
||||
tasks = []
|
||||
total_api_calls = len(url_chunks)
|
||||
self.results_summary["total_api_calls"] = total_api_calls
|
||||
|
||||
try:
|
||||
for i, chunk in enumerate(url_chunks):
|
||||
task = asyncio.create_task(self._make_api_request(
|
||||
chunk=chunk,
|
||||
batch_idx=i + 1,
|
||||
total_batches=total_api_calls,
|
||||
semaphore=semaphore
|
||||
# No memory tracker passed
|
||||
))
|
||||
tasks.append(task)
|
||||
|
||||
api_results = await asyncio.gather(*tasks)
|
||||
|
||||
# Process aggregated results including server memory
|
||||
total_successful_requests = sum(1 for r in api_results if r['request_success'])
|
||||
total_failed_requests = total_api_calls - total_successful_requests
|
||||
total_successful_urls = sum(r['success_urls'] for r in api_results)
|
||||
total_failed_urls = sum(r['failed_urls'] for r in api_results)
|
||||
total_urls_processed = total_successful_urls + total_failed_urls
|
||||
|
||||
# Aggregate server memory metrics
|
||||
valid_samples = [r for r in api_results if r.get('server_delta_or_max_mb') is not None] # Filter results with valid mem data
|
||||
self.results_summary["server_memory_metrics"]["samples"] = valid_samples # Store raw samples with both peak and delta/max
|
||||
|
||||
if valid_samples:
|
||||
delta_or_max_values = [r['server_delta_or_max_mb'] for r in valid_samples]
|
||||
if self.stream_mode:
|
||||
# Stream mode: delta_or_max holds max snapshot
|
||||
self.results_summary["server_memory_metrics"]["stream_mode_avg_max_snapshot_mb"] = sum(delta_or_max_values) / len(delta_or_max_values)
|
||||
self.results_summary["server_memory_metrics"]["stream_mode_max_max_snapshot_mb"] = max(delta_or_max_values)
|
||||
else: # Batch mode
|
||||
# delta_or_max holds delta
|
||||
self.results_summary["server_memory_metrics"]["batch_mode_avg_delta_mb"] = sum(delta_or_max_values) / len(delta_or_max_values)
|
||||
self.results_summary["server_memory_metrics"]["batch_mode_max_delta_mb"] = max(delta_or_max_values)
|
||||
|
||||
# Aggregate peak values for batch mode
|
||||
peak_values = [r['server_peak_memory_mb'] for r in valid_samples if r.get('server_peak_memory_mb') is not None]
|
||||
if peak_values:
|
||||
self.results_summary["server_memory_metrics"]["batch_mode_avg_peak_mb"] = sum(peak_values) / len(peak_values)
|
||||
self.results_summary["server_memory_metrics"]["batch_mode_max_peak_mb"] = max(peak_values)
|
||||
|
||||
|
||||
self.results_summary.update({
|
||||
"successful_requests": total_successful_requests,
|
||||
"failed_requests": total_failed_requests,
|
||||
"successful_urls": total_successful_urls,
|
||||
"failed_urls": total_failed_urls,
|
||||
"total_urls_processed": total_urls_processed,
|
||||
})
|
||||
|
||||
except Exception as e:
|
||||
console.print(f"[bold red]An error occurred during task execution: {e}[/bold red]")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
# No finally block needed for monitor task
|
||||
|
||||
end_time = time.time()
|
||||
self.results_summary.update({
|
||||
"end_time": time.strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"total_time_seconds": end_time - start_time,
|
||||
# No client memory report
|
||||
})
|
||||
self._save_results()
|
||||
return self.results_summary
|
||||
|
||||
async def _make_api_request(
|
||||
self,
|
||||
chunk: List[str],
|
||||
batch_idx: int,
|
||||
total_batches: int,
|
||||
semaphore: asyncio.Semaphore
|
||||
# No memory tracker
|
||||
) -> Dict:
|
||||
"""Makes a single API request for a chunk of URLs, handling concurrency and logging server memory."""
|
||||
request_success = False
|
||||
success_urls = 0
|
||||
failed_urls = 0
|
||||
status = "Pending"
|
||||
status_color = "grey"
|
||||
server_memory_metric = None # Store delta (batch) or max snapshot (stream)
|
||||
api_call_start_time = time.time()
|
||||
|
||||
async with semaphore:
|
||||
try:
|
||||
# No client memory sampling
|
||||
|
||||
endpoint = "/crawl/stream" if self.stream_mode else "/crawl"
|
||||
payload = {
|
||||
"urls": chunk,
|
||||
"browser_config": {"type": "BrowserConfig", "params": {"headless": True}},
|
||||
"crawler_config": {
|
||||
"type": "CrawlerRunConfig",
|
||||
"params": {"cache_mode": "BYPASS", "stream": self.stream_mode}
|
||||
}
|
||||
}
|
||||
|
||||
if self.stream_mode:
|
||||
max_server_mem_snapshot = 0.0 # Track max memory seen in this stream
|
||||
async with self.http_client.stream("POST", endpoint, json=payload) as response:
|
||||
initial_status_code = response.status_code
|
||||
response.raise_for_status()
|
||||
|
||||
completed_marker_received = False
|
||||
async for line in response.aiter_lines():
|
||||
if line:
|
||||
try:
|
||||
data = json.loads(line)
|
||||
if data.get("status") == "completed":
|
||||
completed_marker_received = True
|
||||
break
|
||||
elif data.get("url"):
|
||||
if data.get("success"): success_urls += 1
|
||||
else: failed_urls += 1
|
||||
# Extract server memory snapshot per result
|
||||
mem_snapshot = data.get('server_memory_mb')
|
||||
if mem_snapshot is not None:
|
||||
max_server_mem_snapshot = max(max_server_mem_snapshot, float(mem_snapshot))
|
||||
except json.JSONDecodeError:
|
||||
console.print(f"[Batch {batch_idx}] [red]Stream decode error for line:[/red] {line}")
|
||||
failed_urls = len(chunk)
|
||||
break
|
||||
request_success = completed_marker_received
|
||||
if not request_success:
|
||||
failed_urls = len(chunk) - success_urls
|
||||
server_memory_metric = max_server_mem_snapshot # Use max snapshot for stream logging
|
||||
|
||||
else: # Batch mode
|
||||
response = await self.http_client.post(endpoint, json=payload)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
# Extract server memory delta from the response
|
||||
server_memory_metric = data.get('server_memory_delta_mb')
|
||||
server_peak_mem_mb = data.get('server_peak_memory_mb')
|
||||
|
||||
if data.get("success") and "results" in data:
|
||||
request_success = True
|
||||
results_list = data.get("results", [])
|
||||
for result_item in results_list:
|
||||
if result_item.get("success"): success_urls += 1
|
||||
else: failed_urls += 1
|
||||
if len(results_list) != len(chunk):
|
||||
console.print(f"[Batch {batch_idx}] [yellow]Warning: Result count ({len(results_list)}) doesn't match URL count ({len(chunk)})[/yellow]")
|
||||
failed_urls = len(chunk) - success_urls
|
||||
else:
|
||||
request_success = False
|
||||
failed_urls = len(chunk)
|
||||
# Try to get memory from error detail if available
|
||||
detail = data.get('detail')
|
||||
if isinstance(detail, str):
|
||||
try: detail_json = json.loads(detail)
|
||||
except: detail_json = {}
|
||||
elif isinstance(detail, dict):
|
||||
detail_json = detail
|
||||
else: detail_json = {}
|
||||
server_peak_mem_mb = detail_json.get('server_peak_memory_mb', None)
|
||||
server_memory_metric = detail_json.get('server_memory_delta_mb', None)
|
||||
console.print(f"[Batch {batch_idx}] [red]API request failed:[/red] {detail_json.get('error', 'No details')}")
|
||||
|
||||
|
||||
except httpx.HTTPStatusError as e:
|
||||
request_success = False
|
||||
failed_urls = len(chunk)
|
||||
console.print(f"[Batch {batch_idx}] [bold red]HTTP Error {e.response.status_code}:[/] {e.request.url}")
|
||||
try:
|
||||
error_detail = e.response.json()
|
||||
# Attempt to extract memory info even from error responses
|
||||
detail_content = error_detail.get('detail', {})
|
||||
if isinstance(detail_content, str): # Handle if detail is stringified JSON
|
||||
try: detail_content = json.loads(detail_content)
|
||||
except: detail_content = {}
|
||||
server_memory_metric = detail_content.get('server_memory_delta_mb', None)
|
||||
server_peak_mem_mb = detail_content.get('server_peak_memory_mb', None)
|
||||
console.print(f"Response: {error_detail}")
|
||||
except Exception:
|
||||
console.print(f"Response Text: {e.response.text[:200]}...")
|
||||
except httpx.RequestError as e:
|
||||
request_success = False
|
||||
failed_urls = len(chunk)
|
||||
console.print(f"[Batch {batch_idx}] [bold red]Request Error:[/bold] {e.request.url} - {e}")
|
||||
except Exception as e:
|
||||
request_success = False
|
||||
failed_urls = len(chunk)
|
||||
console.print(f"[Batch {batch_idx}] [bold red]Unexpected Error:[/bold] {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
|
||||
finally:
|
||||
api_call_time = time.time() - api_call_start_time
|
||||
total_processed_urls = success_urls + failed_urls
|
||||
|
||||
if request_success and failed_urls == 0: status_color, status = "green", "Success"
|
||||
elif request_success and success_urls > 0: status_color, status = "yellow", "Partial"
|
||||
else: status_color, status = "red", "Failed"
|
||||
|
||||
current_total_urls = batch_idx * self.chunk_size
|
||||
progress_pct = min(100.0, (current_total_urls / self.url_count) * 100)
|
||||
reqs_per_sec = 1.0 / api_call_time if api_call_time > 0 else float('inf')
|
||||
|
||||
# --- New Memory Formatting ---
|
||||
mem_display = " N/A " # Default
|
||||
peak_mem_value = None
|
||||
delta_or_max_value = None
|
||||
|
||||
if self.stream_mode:
|
||||
# server_memory_metric holds max snapshot for stream
|
||||
if server_memory_metric is not None:
|
||||
mem_display = f"{server_memory_metric:.1f} (Max)"
|
||||
delta_or_max_value = server_memory_metric # Store for aggregation
|
||||
else: # Batch mode - expect peak and delta
|
||||
# We need to get peak and delta from the API response
|
||||
peak_mem_value = locals().get('server_peak_mem_mb', None) # Get from response data if available
|
||||
delta_value = server_memory_metric # server_memory_metric holds delta for batch
|
||||
|
||||
if peak_mem_value is not None and delta_value is not None:
|
||||
mem_display = f"{peak_mem_value:.1f} / {delta_value:+.1f}"
|
||||
delta_or_max_value = delta_value # Store delta for aggregation
|
||||
elif peak_mem_value is not None:
|
||||
mem_display = f"{peak_mem_value:.1f} / N/A"
|
||||
elif delta_value is not None:
|
||||
mem_display = f"N/A / {delta_value:+.1f}"
|
||||
delta_or_max_value = delta_value # Store delta for aggregation
|
||||
|
||||
# --- Updated Print Statement with Adjusted Padding ---
|
||||
console.print(
|
||||
f" {batch_idx:<5} | {progress_pct:6.1f}% | {mem_display:>24} | {reqs_per_sec:8.1f} | " # Increased width for memory column
|
||||
f"{success_urls:^7}/{failed_urls:<6} | {api_call_time:8.2f} | [{status_color}]{status:<7}[/{status_color}] " # Added trailing space
|
||||
)
|
||||
|
||||
# --- Updated Return Dictionary ---
|
||||
return_data = {
|
||||
"batch_idx": batch_idx,
|
||||
"request_success": request_success,
|
||||
"success_urls": success_urls,
|
||||
"failed_urls": failed_urls,
|
||||
"time": api_call_time,
|
||||
# Return both peak (if available) and delta/max
|
||||
"server_peak_memory_mb": peak_mem_value, # Will be None for stream mode
|
||||
"server_delta_or_max_mb": delta_or_max_value # Delta for batch, Max for stream
|
||||
}
|
||||
# Add back the specific batch mode delta if needed elsewhere, but delta_or_max covers it
|
||||
# if not self.stream_mode:
|
||||
# return_data["server_memory_delta_mb"] = delta_value
|
||||
return return_data
|
||||
|
||||
# No _periodic_memory_sample needed
|
||||
|
||||
def _save_results(self) -> None:
|
||||
"""Saves the results summary to a JSON file."""
|
||||
results_path = self.report_path / f"api_test_summary_{self.test_id}.json"
|
||||
try:
|
||||
# No client memory path to convert
|
||||
with open(results_path, 'w', encoding='utf-8') as f:
|
||||
json.dump(self.results_summary, f, indent=2, default=str)
|
||||
except Exception as e:
|
||||
console.print(f"[bold red]Failed to save results summary: {e}[/bold red]")
|
||||
|
||||
|
||||
# --- run_full_test Function ---
|
||||
async def run_full_test(args):
|
||||
"""Runs the full API stress test process."""
|
||||
client = httpx.AsyncClient(base_url=args.api_url, timeout=REQUEST_TIMEOUT)
|
||||
|
||||
if not await check_server_health(client):
|
||||
console.print("[bold red]Aborting test due to server health check failure.[/]")
|
||||
await client.aclose()
|
||||
return
|
||||
await client.aclose()
|
||||
|
||||
test = ApiStressTest(
|
||||
api_url=args.api_url,
|
||||
url_count=args.urls,
|
||||
max_concurrent_requests=args.max_concurrent_requests,
|
||||
chunk_size=args.chunk_size,
|
||||
report_path=args.report_path,
|
||||
stream_mode=args.stream,
|
||||
)
|
||||
results = {}
|
||||
try:
|
||||
results = await test.run()
|
||||
finally:
|
||||
await test.close_client()
|
||||
|
||||
if not results:
|
||||
console.print("[bold red]Test did not produce results.[/bold red]")
|
||||
return
|
||||
|
||||
console.print("\n" + "=" * 80)
|
||||
console.print("[bold green]API Stress Test Completed[/bold green]")
|
||||
console.print("=" * 80)
|
||||
|
||||
success_rate_reqs = results["successful_requests"] / results["total_api_calls"] * 100 if results["total_api_calls"] > 0 else 0
|
||||
success_rate_urls = results["successful_urls"] / results["url_count"] * 100 if results["url_count"] > 0 else 0
|
||||
urls_per_second = results["total_urls_processed"] / results["total_time_seconds"] if results["total_time_seconds"] > 0 else 0
|
||||
reqs_per_second = results["total_api_calls"] / results["total_time_seconds"] if results["total_time_seconds"] > 0 else 0
|
||||
|
||||
|
||||
console.print(f"[bold cyan]Test ID:[/bold cyan] {results['test_id']}")
|
||||
console.print(f"[bold cyan]Target API:[/bold cyan] {results['api_url']}")
|
||||
console.print(f"[bold cyan]Configuration:[/bold cyan] {results['url_count']} URLs, {results['max_concurrent_requests']} concurrent client requests, URLs/Req: {results['chunk_size']}, Stream: {results['stream_mode']}")
|
||||
console.print(f"[bold cyan]API Requests:[/bold cyan] {results['successful_requests']} successful, {results['failed_requests']} failed ({results['total_api_calls']} total, {success_rate_reqs:.1f}% success)")
|
||||
console.print(f"[bold cyan]URL Processing:[/bold cyan] {results['successful_urls']} successful, {results['failed_urls']} failed ({results['total_urls_processed']} processed, {success_rate_urls:.1f}% success)")
|
||||
console.print(f"[bold cyan]Performance:[/bold cyan] {results['total_time_seconds']:.2f}s total | Avg Reqs/sec: {reqs_per_second:.2f} | Avg URLs/sec: {urls_per_second:.2f}")
|
||||
|
||||
# Report Server Memory
|
||||
mem_metrics = results.get("server_memory_metrics", {})
|
||||
mem_samples = mem_metrics.get("samples", [])
|
||||
if mem_samples:
|
||||
num_samples = len(mem_samples)
|
||||
if results['stream_mode']:
|
||||
avg_mem = mem_metrics.get("stream_mode_avg_max_snapshot_mb")
|
||||
max_mem = mem_metrics.get("stream_mode_max_max_snapshot_mb")
|
||||
avg_str = f"{avg_mem:.1f}" if avg_mem is not None else "N/A"
|
||||
max_str = f"{max_mem:.1f}" if max_mem is not None else "N/A"
|
||||
console.print(f"[bold cyan]Server Memory (Stream):[/bold cyan] Avg Max Snapshot: {avg_str} MB | Max Max Snapshot: {max_str} MB (across {num_samples} requests)")
|
||||
else: # Batch mode
|
||||
avg_delta = mem_metrics.get("batch_mode_avg_delta_mb")
|
||||
max_delta = mem_metrics.get("batch_mode_max_delta_mb")
|
||||
avg_peak = mem_metrics.get("batch_mode_avg_peak_mb")
|
||||
max_peak = mem_metrics.get("batch_mode_max_peak_mb")
|
||||
|
||||
avg_delta_str = f"{avg_delta:.1f}" if avg_delta is not None else "N/A"
|
||||
max_delta_str = f"{max_delta:.1f}" if max_delta is not None else "N/A"
|
||||
avg_peak_str = f"{avg_peak:.1f}" if avg_peak is not None else "N/A"
|
||||
max_peak_str = f"{max_peak:.1f}" if max_peak is not None else "N/A"
|
||||
|
||||
console.print(f"[bold cyan]Server Memory (Batch):[/bold cyan] Avg Peak: {avg_peak_str} MB | Max Peak: {max_peak_str} MB | Avg Delta: {avg_delta_str} MB | Max Delta: {max_delta_str} MB (across {num_samples} requests)")
|
||||
else:
|
||||
console.print("[bold cyan]Server Memory:[/bold cyan] No memory data reported by server.")
|
||||
|
||||
|
||||
# No client memory report
|
||||
summary_path = pathlib.Path(args.report_path) / f"api_test_summary_{results['test_id']}.json"
|
||||
console.print(f"[bold green]Results summary saved to {summary_path}[/bold green]")
|
||||
|
||||
if results["failed_requests"] > 0:
|
||||
console.print(f"\n[bold yellow]Warning: {results['failed_requests']} API requests failed ({100-success_rate_reqs:.1f}% failure rate)[/bold yellow]")
|
||||
if results["failed_urls"] > 0:
|
||||
console.print(f"[bold yellow]Warning: {results['failed_urls']} URLs failed to process ({100-success_rate_urls:.1f}% URL failure rate)[/bold yellow]")
|
||||
if results["total_urls_processed"] < results["url_count"]:
|
||||
console.print(f"\n[bold red]Error: Only {results['total_urls_processed']} out of {results['url_count']} target URLs were processed![/bold red]")
|
||||
|
||||
|
||||
# --- main Function (Argument parsing mostly unchanged) ---
|
||||
def main():
|
||||
"""Main entry point for the script."""
|
||||
parser = argparse.ArgumentParser(description="Crawl4AI API Server Stress Test")
|
||||
|
||||
parser.add_argument("--api-url", type=str, default=DEFAULT_API_URL, help=f"Base URL of the Crawl4AI API server (default: {DEFAULT_API_URL})")
|
||||
parser.add_argument("--urls", type=int, default=DEFAULT_URL_COUNT, help=f"Total number of unique URLs to process via API calls (default: {DEFAULT_URL_COUNT})")
|
||||
parser.add_argument("--max-concurrent-requests", type=int, default=DEFAULT_MAX_CONCURRENT_REQUESTS, help=f"Maximum concurrent API requests from this client (default: {DEFAULT_MAX_CONCURRENT_REQUESTS})")
|
||||
parser.add_argument("--chunk-size", type=int, default=DEFAULT_CHUNK_SIZE, help=f"Number of URLs per API request payload (default: {DEFAULT_CHUNK_SIZE})")
|
||||
parser.add_argument("--stream", action="store_true", default=DEFAULT_STREAM_MODE, help=f"Use the /crawl/stream endpoint instead of /crawl (default: {DEFAULT_STREAM_MODE})")
|
||||
parser.add_argument("--report-path", type=str, default=DEFAULT_REPORT_PATH, help=f"Path to save reports and logs (default: {DEFAULT_REPORT_PATH})")
|
||||
parser.add_argument("--clean-reports", action="store_true", help="Clean up report directory before running")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
console.print("[bold underline]Crawl4AI API Stress Test Configuration[/bold underline]")
|
||||
console.print(f"API URL: {args.api_url}")
|
||||
console.print(f"Total URLs: {args.urls}, Concurrent Client Requests: {args.max_concurrent_requests}, URLs per Request: {args.chunk_size}")
|
||||
console.print(f"Mode: {'Streaming' if args.stream else 'Batch'}")
|
||||
console.print(f"Report Path: {args.report_path}")
|
||||
console.print("-" * 40)
|
||||
if args.clean_reports: console.print("[cyan]Option: Clean reports before test[/cyan]")
|
||||
console.print("-" * 40)
|
||||
|
||||
if args.clean_reports:
|
||||
report_dir = pathlib.Path(args.report_path)
|
||||
if report_dir.exists():
|
||||
console.print(f"[yellow]Cleaning up reports directory: {args.report_path}[/yellow]")
|
||||
shutil.rmtree(args.report_path)
|
||||
report_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
try:
|
||||
asyncio.run(run_full_test(args))
|
||||
except KeyboardInterrupt:
|
||||
console.print("\n[bold yellow]Test interrupted by user.[/bold yellow]")
|
||||
except Exception as e:
|
||||
console.print(f"\n[bold red]An unexpected error occurred:[/bold red] {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
|
||||
if __name__ == "__main__":
|
||||
# No need to modify sys.path for SimpleMemoryTracker as it's removed
|
||||
main()
|
||||
@@ -0,0 +1,203 @@
|
||||
"""Lite Crawl4AI API stress‑tester.
|
||||
|
||||
✔ batch or stream mode (single unified path)
|
||||
✔ global stats + JSON summary
|
||||
✔ rich table progress
|
||||
✔ Typer CLI with presets (quick / soak)
|
||||
|
||||
Usage examples:
|
||||
python api_stress_test.py # uses quick preset
|
||||
python api_stress_test.py soak # 5 K URLs stress run
|
||||
python api_stress_test.py --urls 200 --concurrent 10 --chunk 20
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio, json, time, uuid, pathlib, statistics
|
||||
from typing import List, Dict, Optional
|
||||
|
||||
import httpx, typer
|
||||
from rich.console import Console
|
||||
from rich.table import Table
|
||||
|
||||
# ───────────────────────── defaults / presets ──────────────────────────
|
||||
PRESETS = {
|
||||
"quick": dict(urls=1, concurrent=1, chunk=1, stream=False),
|
||||
"debug": dict(urls=10, concurrent=2, chunk=5, stream=False),
|
||||
"soak": dict(urls=5000, concurrent=20, chunk=50, stream=True),
|
||||
}
|
||||
|
||||
API_HEALTH_ENDPOINT = "/health"
|
||||
REQUEST_TIMEOUT = 180.0
|
||||
|
||||
console = Console()
|
||||
app = typer.Typer(add_completion=False, rich_markup_mode="rich")
|
||||
|
||||
# ───────────────────────── helpers ─────────────────────────────────────
|
||||
async def _check_health(client: httpx.AsyncClient) -> None:
|
||||
resp = await client.get(API_HEALTH_ENDPOINT, timeout=10)
|
||||
resp.raise_for_status()
|
||||
console.print(f"[green]Server healthy — version {resp.json().get('version','?')}[/]")
|
||||
|
||||
async def _iter_results(resp: httpx.Response, stream: bool):
|
||||
"""Yield result dicts from batch JSON or ND‑JSON stream."""
|
||||
if stream:
|
||||
async for line in resp.aiter_lines():
|
||||
if not line:
|
||||
continue
|
||||
rec = json.loads(line)
|
||||
if rec.get("status") == "completed":
|
||||
break
|
||||
yield rec
|
||||
else:
|
||||
data = resp.json()
|
||||
for rec in data.get("results", []):
|
||||
yield rec, data # rec + whole payload for memory delta/peak
|
||||
|
||||
async def _consume_stream(resp: httpx.Response) -> Dict:
|
||||
stats = {"success_urls": 0, "failed_urls": 0, "mem_metric": 0.0}
|
||||
async for line in resp.aiter_lines():
|
||||
if not line:
|
||||
continue
|
||||
rec = json.loads(line)
|
||||
if rec.get("status") == "completed":
|
||||
break
|
||||
if rec.get("success"):
|
||||
stats["success_urls"] += 1
|
||||
else:
|
||||
stats["failed_urls"] += 1
|
||||
mem = rec.get("server_memory_mb")
|
||||
if mem is not None:
|
||||
stats["mem_metric"] = max(stats["mem_metric"], float(mem))
|
||||
return stats
|
||||
|
||||
def _consume_batch(body: Dict) -> Dict:
|
||||
stats = {"success_urls": 0, "failed_urls": 0}
|
||||
for rec in body.get("results", []):
|
||||
if rec.get("success"):
|
||||
stats["success_urls"] += 1
|
||||
else:
|
||||
stats["failed_urls"] += 1
|
||||
stats["mem_metric"] = body.get("server_memory_delta_mb")
|
||||
stats["peak"] = body.get("server_peak_memory_mb")
|
||||
return stats
|
||||
|
||||
async def _fetch_chunk(
|
||||
client: httpx.AsyncClient,
|
||||
urls: List[str],
|
||||
stream: bool,
|
||||
semaphore: asyncio.Semaphore,
|
||||
) -> Dict:
|
||||
endpoint = "/crawl/stream" if stream else "/crawl"
|
||||
payload = {
|
||||
"urls": urls,
|
||||
"browser_config": {"type": "BrowserConfig", "params": {"headless": True}},
|
||||
"crawler_config": {"type": "CrawlerRunConfig",
|
||||
"params": {"cache_mode": "BYPASS", "stream": stream}},
|
||||
}
|
||||
|
||||
async with semaphore:
|
||||
start = time.perf_counter()
|
||||
|
||||
if stream:
|
||||
# ---- streaming request ----
|
||||
async with client.stream("POST", endpoint, json=payload) as resp:
|
||||
resp.raise_for_status()
|
||||
stats = await _consume_stream(resp)
|
||||
else:
|
||||
# ---- batch request ----
|
||||
resp = await client.post(endpoint, json=payload)
|
||||
resp.raise_for_status()
|
||||
stats = _consume_batch(resp.json())
|
||||
|
||||
stats["elapsed"] = time.perf_counter() - start
|
||||
return stats
|
||||
|
||||
|
||||
# ───────────────────────── core runner ─────────────────────────────────
|
||||
async def _run(api: str, urls: int, concurrent: int, chunk: int, stream: bool, report: pathlib.Path):
|
||||
client = httpx.AsyncClient(base_url=api, timeout=REQUEST_TIMEOUT, limits=httpx.Limits(max_connections=concurrent+5))
|
||||
await _check_health(client)
|
||||
|
||||
url_list = [f"https://httpbin.org/anything/{uuid.uuid4()}" for _ in range(urls)]
|
||||
chunks = [url_list[i:i+chunk] for i in range(0, len(url_list), chunk)]
|
||||
sem = asyncio.Semaphore(concurrent)
|
||||
|
||||
table = Table(show_header=True, header_style="bold magenta")
|
||||
table.add_column("Batch", style="dim", width=6)
|
||||
table.add_column("Success/Fail", width=12)
|
||||
table.add_column("Mem", width=14)
|
||||
table.add_column("Time (s)")
|
||||
|
||||
agg_success = agg_fail = 0
|
||||
deltas, peaks = [], []
|
||||
|
||||
start = time.perf_counter()
|
||||
tasks = [asyncio.create_task(_fetch_chunk(client, c, stream, sem)) for c in chunks]
|
||||
for idx, coro in enumerate(asyncio.as_completed(tasks), 1):
|
||||
res = await coro
|
||||
agg_success += res["success_urls"]
|
||||
agg_fail += res["failed_urls"]
|
||||
if res["mem_metric"] is not None:
|
||||
deltas.append(res["mem_metric"])
|
||||
if res["peak"] is not None:
|
||||
peaks.append(res["peak"])
|
||||
|
||||
mem_txt = f"{res['mem_metric']:.1f}" if res["mem_metric"] is not None else "‑"
|
||||
if res["peak"] is not None:
|
||||
mem_txt = f"{res['peak']:.1f}/{mem_txt}"
|
||||
|
||||
table.add_row(str(idx), f"{res['success_urls']}/{res['failed_urls']}", mem_txt, f"{res['elapsed']:.2f}")
|
||||
|
||||
console.print(table)
|
||||
total_time = time.perf_counter() - start
|
||||
|
||||
summary = {
|
||||
"urls": urls,
|
||||
"concurrent": concurrent,
|
||||
"chunk": chunk,
|
||||
"stream": stream,
|
||||
"success_urls": agg_success,
|
||||
"failed_urls": agg_fail,
|
||||
"elapsed_sec": round(total_time, 2),
|
||||
"avg_mem": round(statistics.mean(deltas), 2) if deltas else None,
|
||||
"max_mem": max(deltas) if deltas else None,
|
||||
"avg_peak": round(statistics.mean(peaks), 2) if peaks else None,
|
||||
"max_peak": max(peaks) if peaks else None,
|
||||
}
|
||||
console.print("\n[bold green]Done:[/]" , summary)
|
||||
|
||||
report.mkdir(parents=True, exist_ok=True)
|
||||
path = report / f"api_test_{int(time.time())}.json"
|
||||
path.write_text(json.dumps(summary, indent=2))
|
||||
console.print(f"[green]Summary → {path}")
|
||||
|
||||
await client.aclose()
|
||||
|
||||
# ───────────────────────── Typer CLI ──────────────────────────────────
|
||||
@app.command()
|
||||
def main(
|
||||
preset: str = typer.Argument("quick", help="quick / debug / soak or custom"),
|
||||
api_url: str = typer.Option("http://localhost:8020", show_default=True),
|
||||
urls: int = typer.Option(None, help="Total URLs to crawl"),
|
||||
concurrent: int = typer.Option(None, help="Concurrent API requests"),
|
||||
chunk: int = typer.Option(None, help="URLs per request"),
|
||||
stream: bool = typer.Option(None, help="Use /crawl/stream"),
|
||||
report: pathlib.Path = typer.Option("reports_api", help="Where to save JSON summary"),
|
||||
):
|
||||
"""Run a stress test against a running Crawl4AI API server."""
|
||||
if preset not in PRESETS and any(v is None for v in (urls, concurrent, chunk, stream)):
|
||||
console.print(f"[red]Unknown preset '{preset}' and custom params missing[/]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
cfg = PRESETS.get(preset, {})
|
||||
urls = urls or cfg.get("urls")
|
||||
concurrent = concurrent or cfg.get("concurrent")
|
||||
chunk = chunk or cfg.get("chunk")
|
||||
stream = stream if stream is not None else cfg.get("stream", False)
|
||||
|
||||
console.print(f"[cyan]API:[/] {api_url} | URLs: {urls} | Concurrency: {concurrent} | Chunk: {chunk} | Stream: {stream}")
|
||||
asyncio.run(_run(api_url, urls, concurrent, chunk, stream, report))
|
||||
|
||||
if __name__ == "__main__":
|
||||
app()
|
||||
@@ -0,0 +1,129 @@
|
||||
"""
|
||||
Crawl4AI Docker API stress tester.
|
||||
|
||||
Examples
|
||||
--------
|
||||
python test_stress_docker_api.py --urls 1000 --concurrency 32
|
||||
python test_stress_docker_api.py --urls 1000 --concurrency 32 --stream
|
||||
python test_stress_docker_api.py --base-url http://10.0.0.42:11235 --http2
|
||||
"""
|
||||
|
||||
import argparse, asyncio, json, secrets, statistics, time
|
||||
from typing import List, Tuple
|
||||
import httpx
|
||||
from rich.console import Console
|
||||
from rich.progress import Progress, BarColumn, TimeElapsedColumn, TimeRemainingColumn
|
||||
from rich.table import Table
|
||||
|
||||
console = Console()
|
||||
|
||||
|
||||
# ───────────────────────── helpers ─────────────────────────
|
||||
def make_fake_urls(n: int) -> List[str]:
|
||||
base = "https://httpbin.org/anything/"
|
||||
return [f"{base}{secrets.token_hex(8)}" for _ in range(n)]
|
||||
|
||||
|
||||
async def fire(
|
||||
client: httpx.AsyncClient, endpoint: str, payload: dict, sem: asyncio.Semaphore
|
||||
) -> Tuple[bool, float]:
|
||||
async with sem:
|
||||
print(f"POST {endpoint} with {len(payload['urls'])} URLs")
|
||||
t0 = time.perf_counter()
|
||||
try:
|
||||
if endpoint.endswith("/stream"):
|
||||
async with client.stream("POST", endpoint, json=payload) as r:
|
||||
r.raise_for_status()
|
||||
async for _ in r.aiter_lines():
|
||||
pass
|
||||
else:
|
||||
r = await client.post(endpoint, json=payload)
|
||||
r.raise_for_status()
|
||||
return True, time.perf_counter() - t0
|
||||
except Exception:
|
||||
return False, time.perf_counter() - t0
|
||||
|
||||
|
||||
def pct(lat: List[float], p: float) -> str:
|
||||
"""Return percentile string even for tiny samples."""
|
||||
if not lat:
|
||||
return "-"
|
||||
if len(lat) == 1:
|
||||
return f"{lat[0]:.2f}s"
|
||||
lat_sorted = sorted(lat)
|
||||
k = (p / 100) * (len(lat_sorted) - 1)
|
||||
lo = int(k)
|
||||
hi = min(lo + 1, len(lat_sorted) - 1)
|
||||
frac = k - lo
|
||||
val = lat_sorted[lo] * (1 - frac) + lat_sorted[hi] * frac
|
||||
return f"{val:.2f}s"
|
||||
|
||||
|
||||
# ───────────────────────── main ─────────────────────────
|
||||
def parse_args() -> argparse.Namespace:
|
||||
p = argparse.ArgumentParser(description="Stress test Crawl4AI Docker API")
|
||||
p.add_argument("--urls", type=int, default=100, help="number of URLs")
|
||||
p.add_argument("--concurrency", type=int, default=1, help="max POSTs in flight")
|
||||
p.add_argument("--chunk-size", type=int, default=50, help="URLs per request")
|
||||
p.add_argument("--base-url", default="http://localhost:11235", help="API root")
|
||||
# p.add_argument("--base-url", default="http://localhost:8020", help="API root")
|
||||
p.add_argument("--stream", action="store_true", help="use /crawl/stream")
|
||||
p.add_argument("--http2", action="store_true", help="enable HTTP/2")
|
||||
p.add_argument("--headless", action="store_true", default=True)
|
||||
return p.parse_args()
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
args = parse_args()
|
||||
|
||||
urls = make_fake_urls(args.urls)
|
||||
batches = [urls[i : i + args.chunk_size] for i in range(0, len(urls), args.chunk_size)]
|
||||
endpoint = "/crawl/stream" if args.stream else "/crawl"
|
||||
sem = asyncio.Semaphore(args.concurrency)
|
||||
|
||||
async with httpx.AsyncClient(base_url=args.base_url, http2=args.http2, timeout=None) as client:
|
||||
with Progress(
|
||||
"[progress.description]{task.description}",
|
||||
BarColumn(),
|
||||
"[progress.percentage]{task.percentage:>3.0f}%",
|
||||
TimeElapsedColumn(),
|
||||
TimeRemainingColumn(),
|
||||
) as progress:
|
||||
task_id = progress.add_task("[cyan]bombarding…", total=len(batches))
|
||||
tasks = []
|
||||
for chunk in batches:
|
||||
payload = {
|
||||
"urls": chunk,
|
||||
"browser_config": {"type": "BrowserConfig", "params": {"headless": args.headless}},
|
||||
"crawler_config": {"type": "CrawlerRunConfig", "params": {"cache_mode": "BYPASS", "stream": args.stream}},
|
||||
}
|
||||
tasks.append(asyncio.create_task(fire(client, endpoint, payload, sem)))
|
||||
progress.advance(task_id)
|
||||
|
||||
results = await asyncio.gather(*tasks)
|
||||
|
||||
ok_latencies = [dt for ok, dt in results if ok]
|
||||
err_count = sum(1 for ok, _ in results if not ok)
|
||||
|
||||
table = Table(title="Docker API Stress‑Test Summary")
|
||||
table.add_column("total", justify="right")
|
||||
table.add_column("errors", justify="right")
|
||||
table.add_column("p50", justify="right")
|
||||
table.add_column("p95", justify="right")
|
||||
table.add_column("max", justify="right")
|
||||
|
||||
table.add_row(
|
||||
str(len(results)),
|
||||
str(err_count),
|
||||
pct(ok_latencies, 50),
|
||||
pct(ok_latencies, 95),
|
||||
f"{max(ok_latencies):.2f}s" if ok_latencies else "-",
|
||||
)
|
||||
console.print(table)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
asyncio.run(main())
|
||||
except KeyboardInterrupt:
|
||||
console.print("\n[yellow]aborted by user[/]")
|
||||
@@ -0,0 +1,500 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Stress test for Crawl4AI's arun_many and dispatcher system.
|
||||
This version uses a local HTTP server and focuses on testing
|
||||
the SDK's ability to handle multiple URLs concurrently, with per-batch logging.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import time
|
||||
import pathlib
|
||||
import random
|
||||
import secrets
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
import subprocess
|
||||
import signal
|
||||
from typing import List, Dict, Optional, Union, AsyncGenerator
|
||||
import shutil
|
||||
from rich.console import Console
|
||||
|
||||
# Crawl4AI components
|
||||
from crawl4ai import (
|
||||
AsyncWebCrawler,
|
||||
CrawlerRunConfig,
|
||||
BrowserConfig,
|
||||
MemoryAdaptiveDispatcher,
|
||||
CrawlerMonitor,
|
||||
DisplayMode,
|
||||
CrawlResult,
|
||||
RateLimiter,
|
||||
CacheMode,
|
||||
)
|
||||
|
||||
# Constants
|
||||
DEFAULT_SITE_PATH = "test_site"
|
||||
DEFAULT_PORT = 8000
|
||||
DEFAULT_MAX_SESSIONS = 16
|
||||
DEFAULT_URL_COUNT = 1
|
||||
DEFAULT_CHUNK_SIZE = 1 # Define chunk size for batch logging
|
||||
DEFAULT_REPORT_PATH = "reports"
|
||||
DEFAULT_STREAM_MODE = False
|
||||
DEFAULT_MONITOR_MODE = "DETAILED"
|
||||
|
||||
# Initialize Rich console
|
||||
console = Console()
|
||||
|
||||
# --- SiteGenerator Class (Unchanged) ---
|
||||
class SiteGenerator:
|
||||
"""Generates a local test site with heavy pages for stress testing."""
|
||||
|
||||
def __init__(self, site_path: str = DEFAULT_SITE_PATH, page_count: int = DEFAULT_URL_COUNT):
|
||||
self.site_path = pathlib.Path(site_path)
|
||||
self.page_count = page_count
|
||||
self.images_dir = self.site_path / "images"
|
||||
self.lorem_words = " ".join("lorem ipsum dolor sit amet " * 100).split()
|
||||
|
||||
self.html_template = """<!doctype html>
|
||||
<html>
|
||||
<head>
|
||||
<title>Test Page {page_num}</title>
|
||||
<meta charset="utf-8">
|
||||
</head>
|
||||
<body>
|
||||
<h1>Test Page {page_num}</h1>
|
||||
{paragraphs}
|
||||
{images}
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
def generate_site(self) -> None:
|
||||
self.site_path.mkdir(parents=True, exist_ok=True)
|
||||
self.images_dir.mkdir(exist_ok=True)
|
||||
console.print(f"Generating {self.page_count} test pages...")
|
||||
for i in range(self.page_count):
|
||||
paragraphs = "\n".join(f"<p>{' '.join(random.choices(self.lorem_words, k=200))}</p>" for _ in range(5))
|
||||
images = "\n".join(f'<img src="https://picsum.photos/seed/{secrets.token_hex(8)}/300/200" loading="lazy" alt="Random image {j}"/>' for j in range(3))
|
||||
page_path = self.site_path / f"page_{i}.html"
|
||||
page_path.write_text(self.html_template.format(page_num=i, paragraphs=paragraphs, images=images), encoding="utf-8")
|
||||
if (i + 1) % (self.page_count // 10 or 1) == 0 or i == self.page_count - 1:
|
||||
console.print(f"Generated {i+1}/{self.page_count} pages")
|
||||
self._create_index_page()
|
||||
console.print(f"[bold green]Successfully generated {self.page_count} test pages in [cyan]{self.site_path}[/cyan][/bold green]")
|
||||
|
||||
def _create_index_page(self) -> None:
|
||||
index_content = """<!doctype html><html><head><title>Test Site Index</title><meta charset="utf-8"></head><body><h1>Test Site Index</h1><p>This is an automatically generated site for testing Crawl4AI.</p><div class="page-links">\n"""
|
||||
for i in range(self.page_count):
|
||||
index_content += f' <a href="page_{i}.html">Test Page {i}</a><br>\n'
|
||||
index_content += """ </div></body></html>"""
|
||||
(self.site_path / "index.html").write_text(index_content, encoding="utf-8")
|
||||
|
||||
# --- LocalHttpServer Class (Unchanged) ---
|
||||
class LocalHttpServer:
|
||||
"""Manages a local HTTP server for serving test pages."""
|
||||
def __init__(self, site_path: str = DEFAULT_SITE_PATH, port: int = DEFAULT_PORT):
|
||||
self.site_path = pathlib.Path(site_path)
|
||||
self.port = port
|
||||
self.process = None
|
||||
|
||||
def start(self) -> None:
|
||||
if not self.site_path.exists(): raise FileNotFoundError(f"Site directory {self.site_path} does not exist")
|
||||
console.print(f"Attempting to start HTTP server in [cyan]{self.site_path}[/cyan] on port {self.port}...")
|
||||
try:
|
||||
cmd = ["python", "-m", "http.server", str(self.port)]
|
||||
creationflags = 0; preexec_fn = None
|
||||
if sys.platform == 'win32': creationflags = subprocess.CREATE_NEW_PROCESS_GROUP
|
||||
self.process = subprocess.Popen(cmd, cwd=str(self.site_path), stdout=subprocess.PIPE, stderr=subprocess.PIPE, creationflags=creationflags)
|
||||
time.sleep(1.5)
|
||||
if self.is_running(): console.print(f"[bold green]HTTP server started successfully (PID: {self.process.pid})[/bold green]")
|
||||
else:
|
||||
console.print("[bold red]Failed to start HTTP server. Checking logs...[/bold red]")
|
||||
stdout, stderr = self.process.communicate(); print(stdout.decode(errors='ignore')); print(stderr.decode(errors='ignore'))
|
||||
self.stop(); raise RuntimeError("HTTP server failed to start.")
|
||||
except Exception as e: console.print(f"[bold red]Error starting HTTP server: {str(e)}[/bold red]"); self.stop(); raise
|
||||
|
||||
def stop(self) -> None:
|
||||
if self.process and self.is_running():
|
||||
console.print(f"Stopping HTTP server (PID: {self.process.pid})...")
|
||||
try:
|
||||
if sys.platform == 'win32': self.process.send_signal(signal.CTRL_BREAK_EVENT); time.sleep(0.5)
|
||||
self.process.terminate()
|
||||
try: stdout, stderr = self.process.communicate(timeout=5); console.print("[bold yellow]HTTP server stopped[/bold yellow]")
|
||||
except subprocess.TimeoutExpired: console.print("[bold red]Server did not terminate gracefully, killing...[/bold red]"); self.process.kill(); stdout, stderr = self.process.communicate(); console.print("[bold yellow]HTTP server killed[/bold yellow]")
|
||||
except Exception as e: console.print(f"[bold red]Error stopping HTTP server: {str(e)}[/bold red]"); self.process.kill()
|
||||
finally: self.process = None
|
||||
elif self.process: console.print("[dim]HTTP server process already stopped.[/dim]"); self.process = None
|
||||
|
||||
def is_running(self) -> bool:
|
||||
if not self.process: return False
|
||||
return self.process.poll() is None
|
||||
|
||||
# --- SimpleMemoryTracker Class (Unchanged) ---
|
||||
class SimpleMemoryTracker:
|
||||
"""Basic memory tracker that doesn't rely on psutil."""
|
||||
def __init__(self, report_path: str = DEFAULT_REPORT_PATH, test_id: Optional[str] = None):
|
||||
self.report_path = pathlib.Path(report_path); self.report_path.mkdir(parents=True, exist_ok=True)
|
||||
self.test_id = test_id or time.strftime("%Y%m%d_%H%M%S")
|
||||
self.start_time = time.time(); self.memory_samples = []; self.pid = os.getpid()
|
||||
self.csv_path = self.report_path / f"memory_samples_{self.test_id}.csv"
|
||||
with open(self.csv_path, 'w', encoding='utf-8') as f: f.write("timestamp,elapsed_seconds,memory_info_mb\n")
|
||||
|
||||
def sample(self) -> Dict:
|
||||
try:
|
||||
memory_mb = self._get_memory_info_mb()
|
||||
memory_str = f"{memory_mb:.1f} MB" if memory_mb is not None else "Unknown"
|
||||
timestamp = time.time(); elapsed = timestamp - self.start_time
|
||||
sample = {"timestamp": timestamp, "elapsed_seconds": elapsed, "memory_mb": memory_mb, "memory_str": memory_str}
|
||||
self.memory_samples.append(sample)
|
||||
with open(self.csv_path, 'a', encoding='utf-8') as f: f.write(f"{timestamp},{elapsed:.2f},{memory_mb if memory_mb is not None else ''}\n")
|
||||
return sample
|
||||
except Exception as e: return {"memory_mb": None, "memory_str": "Error"}
|
||||
|
||||
def _get_memory_info_mb(self) -> Optional[float]:
|
||||
pid_str = str(self.pid)
|
||||
try:
|
||||
if sys.platform == 'darwin': result = subprocess.run(["ps", "-o", "rss=", "-p", pid_str], capture_output=True, text=True, check=True, encoding='utf-8'); return int(result.stdout.strip()) / 1024.0
|
||||
elif sys.platform == 'linux':
|
||||
with open(f"/proc/{pid_str}/status", encoding='utf-8') as f:
|
||||
for line in f:
|
||||
if line.startswith("VmRSS:"): return int(line.split()[1]) / 1024.0
|
||||
return None
|
||||
elif sys.platform == 'win32': result = subprocess.run(["tasklist", "/fi", f"PID eq {pid_str}", "/fo", "csv", "/nh"], capture_output=True, text=True, check=True, encoding='cp850', errors='ignore'); parts = result.stdout.strip().split('","'); return int(parts[4].strip().replace('"', '').replace(' K', '').replace(',', '')) / 1024.0 if len(parts) >= 5 else None
|
||||
else: return None
|
||||
except: return None # Catch all exceptions for robustness
|
||||
|
||||
def get_report(self) -> Dict:
|
||||
if not self.memory_samples: return {"error": "No memory samples collected"}
|
||||
total_time = time.time() - self.start_time; valid_samples = [s['memory_mb'] for s in self.memory_samples if s['memory_mb'] is not None]
|
||||
start_mem = valid_samples[0] if valid_samples else None; end_mem = valid_samples[-1] if valid_samples else None
|
||||
max_mem = max(valid_samples) if valid_samples else None; avg_mem = sum(valid_samples) / len(valid_samples) if valid_samples else None
|
||||
growth = (end_mem - start_mem) if start_mem is not None and end_mem is not None else None
|
||||
return {"test_id": self.test_id, "total_time_seconds": total_time, "sample_count": len(self.memory_samples), "valid_sample_count": len(valid_samples), "csv_path": str(self.csv_path), "platform": sys.platform, "start_memory_mb": start_mem, "end_memory_mb": end_mem, "max_memory_mb": max_mem, "average_memory_mb": avg_mem, "memory_growth_mb": growth}
|
||||
|
||||
|
||||
# --- CrawlerStressTest Class (Refactored for Per-Batch Logging) ---
|
||||
class CrawlerStressTest:
|
||||
"""Orchestrates the stress test using arun_many per chunk and a dispatcher."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
url_count: int = DEFAULT_URL_COUNT,
|
||||
port: int = DEFAULT_PORT,
|
||||
max_sessions: int = DEFAULT_MAX_SESSIONS,
|
||||
chunk_size: int = DEFAULT_CHUNK_SIZE, # Added chunk_size
|
||||
report_path: str = DEFAULT_REPORT_PATH,
|
||||
stream_mode: bool = DEFAULT_STREAM_MODE,
|
||||
monitor_mode: str = DEFAULT_MONITOR_MODE,
|
||||
use_rate_limiter: bool = False
|
||||
):
|
||||
self.url_count = url_count
|
||||
self.server_port = port
|
||||
self.max_sessions = max_sessions
|
||||
self.chunk_size = chunk_size # Store chunk size
|
||||
self.report_path = pathlib.Path(report_path)
|
||||
self.report_path.mkdir(parents=True, exist_ok=True)
|
||||
self.stream_mode = stream_mode
|
||||
self.monitor_mode = DisplayMode[monitor_mode.upper()]
|
||||
self.use_rate_limiter = use_rate_limiter
|
||||
|
||||
self.test_id = time.strftime("%Y%m%d_%H%M%S")
|
||||
self.results_summary = {
|
||||
"test_id": self.test_id, "url_count": url_count, "max_sessions": max_sessions,
|
||||
"chunk_size": chunk_size, "stream_mode": stream_mode, "monitor_mode": monitor_mode,
|
||||
"rate_limiter_used": use_rate_limiter, "start_time": "", "end_time": "",
|
||||
"total_time_seconds": 0, "successful_urls": 0, "failed_urls": 0,
|
||||
"urls_processed": 0, "chunks_processed": 0
|
||||
}
|
||||
|
||||
async def run(self) -> Dict:
|
||||
"""Run the stress test and return results."""
|
||||
memory_tracker = SimpleMemoryTracker(report_path=self.report_path, test_id=self.test_id)
|
||||
urls = [f"http://localhost:{self.server_port}/page_{i}.html" for i in range(self.url_count)]
|
||||
# Split URLs into chunks based on self.chunk_size
|
||||
url_chunks = [urls[i:i+self.chunk_size] for i in range(0, len(urls), self.chunk_size)]
|
||||
|
||||
self.results_summary["start_time"] = time.strftime("%Y-%m-%d %H:%M:%S")
|
||||
start_time = time.time()
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
wait_for_images=False, verbose=False,
|
||||
stream=self.stream_mode, # Still pass stream mode, affects arun_many return type
|
||||
cache_mode=CacheMode.BYPASS
|
||||
)
|
||||
|
||||
total_successful_urls = 0
|
||||
total_failed_urls = 0
|
||||
total_urls_processed = 0
|
||||
start_memory_sample = memory_tracker.sample()
|
||||
start_memory_str = start_memory_sample.get("memory_str", "Unknown")
|
||||
|
||||
# monitor = CrawlerMonitor(display_mode=self.monitor_mode, total_urls=self.url_count)
|
||||
monitor = None
|
||||
rate_limiter = RateLimiter(base_delay=(0.1, 0.3)) if self.use_rate_limiter else None
|
||||
dispatcher = MemoryAdaptiveDispatcher(max_session_permit=self.max_sessions, monitor=monitor, rate_limiter=rate_limiter)
|
||||
|
||||
console.print(f"\n[bold cyan]Crawl4AI Stress Test - {self.url_count} URLs, {self.max_sessions} max sessions[/bold cyan]")
|
||||
console.print(f"[bold cyan]Mode:[/bold cyan] {'Streaming' if self.stream_mode else 'Batch'}, [bold cyan]Monitor:[/bold cyan] {self.monitor_mode.name}, [bold cyan]Chunk Size:[/bold cyan] {self.chunk_size}")
|
||||
console.print(f"[bold cyan]Initial Memory:[/bold cyan] {start_memory_str}")
|
||||
|
||||
# Print batch log header only if not streaming
|
||||
if not self.stream_mode:
|
||||
console.print("\n[bold]Batch Progress:[/bold] (Monitor below shows overall progress)")
|
||||
console.print("[bold] Batch | Progress | Start Mem | End Mem | URLs/sec | Success/Fail | Time (s) | Status [/bold]")
|
||||
console.print("─" * 90)
|
||||
|
||||
monitor_task = asyncio.create_task(self._periodic_memory_sample(memory_tracker, 2.0))
|
||||
|
||||
try:
|
||||
async with AsyncWebCrawler(
|
||||
config=BrowserConfig( verbose = False)
|
||||
) as crawler:
|
||||
# Process URLs chunk by chunk
|
||||
for chunk_idx, url_chunk in enumerate(url_chunks):
|
||||
batch_start_time = time.time()
|
||||
chunk_success = 0
|
||||
chunk_failed = 0
|
||||
|
||||
# Sample memory before the chunk
|
||||
start_mem_sample = memory_tracker.sample()
|
||||
start_mem_str = start_mem_sample.get("memory_str", "Unknown")
|
||||
|
||||
# --- Call arun_many for the current chunk ---
|
||||
try:
|
||||
# Note: dispatcher/monitor persist across calls
|
||||
results_gen_or_list: Union[AsyncGenerator[CrawlResult, None], List[CrawlResult]] = \
|
||||
await crawler.arun_many(
|
||||
urls=url_chunk,
|
||||
config=config,
|
||||
dispatcher=dispatcher # Reuse the same dispatcher
|
||||
)
|
||||
|
||||
if self.stream_mode:
|
||||
# Process stream results if needed, but batch logging is less relevant
|
||||
async for result in results_gen_or_list:
|
||||
total_urls_processed += 1
|
||||
if result.success: chunk_success += 1
|
||||
else: chunk_failed += 1
|
||||
# In stream mode, batch summary isn't as meaningful here
|
||||
# We could potentially track completion per chunk async, but it's complex
|
||||
|
||||
else: # Batch mode
|
||||
# Process the list of results for this chunk
|
||||
for result in results_gen_or_list:
|
||||
total_urls_processed += 1
|
||||
if result.success: chunk_success += 1
|
||||
else: chunk_failed += 1
|
||||
|
||||
except Exception as e:
|
||||
console.print(f"[bold red]Error processing chunk {chunk_idx+1}: {e}[/bold red]")
|
||||
chunk_failed = len(url_chunk) # Assume all failed in the chunk on error
|
||||
total_urls_processed += len(url_chunk) # Count them as processed (failed)
|
||||
|
||||
# --- Log batch results (only if not streaming) ---
|
||||
if not self.stream_mode:
|
||||
batch_time = time.time() - batch_start_time
|
||||
urls_per_sec = len(url_chunk) / batch_time if batch_time > 0 else 0
|
||||
end_mem_sample = memory_tracker.sample()
|
||||
end_mem_str = end_mem_sample.get("memory_str", "Unknown")
|
||||
|
||||
progress_pct = (total_urls_processed / self.url_count) * 100
|
||||
|
||||
if chunk_failed == 0: status_color, status = "green", "Success"
|
||||
elif chunk_success == 0: status_color, status = "red", "Failed"
|
||||
else: status_color, status = "yellow", "Partial"
|
||||
|
||||
console.print(
|
||||
f" {chunk_idx+1:<5} | {progress_pct:6.1f}% | {start_mem_str:>9} | {end_mem_str:>9} | {urls_per_sec:8.1f} | "
|
||||
f"{chunk_success:^7}/{chunk_failed:<6} | {batch_time:8.2f} | [{status_color}]{status:<7}[/{status_color}]"
|
||||
)
|
||||
|
||||
# Accumulate totals
|
||||
total_successful_urls += chunk_success
|
||||
total_failed_urls += chunk_failed
|
||||
self.results_summary["chunks_processed"] += 1
|
||||
|
||||
# Optional small delay between starting chunks if needed
|
||||
# await asyncio.sleep(0.1)
|
||||
|
||||
except Exception as e:
|
||||
console.print(f"[bold red]An error occurred during the main crawl loop: {e}[/bold red]")
|
||||
finally:
|
||||
if 'monitor_task' in locals() and not monitor_task.done():
|
||||
monitor_task.cancel()
|
||||
try: await monitor_task
|
||||
except asyncio.CancelledError: pass
|
||||
|
||||
end_time = time.time()
|
||||
self.results_summary.update({
|
||||
"end_time": time.strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"total_time_seconds": end_time - start_time,
|
||||
"successful_urls": total_successful_urls,
|
||||
"failed_urls": total_failed_urls,
|
||||
"urls_processed": total_urls_processed,
|
||||
"memory": memory_tracker.get_report()
|
||||
})
|
||||
self._save_results()
|
||||
return self.results_summary
|
||||
|
||||
async def _periodic_memory_sample(self, tracker: SimpleMemoryTracker, interval: float):
|
||||
"""Background task to sample memory periodically."""
|
||||
while True:
|
||||
tracker.sample()
|
||||
try:
|
||||
await asyncio.sleep(interval)
|
||||
except asyncio.CancelledError:
|
||||
break # Exit loop on cancellation
|
||||
|
||||
def _save_results(self) -> None:
|
||||
results_path = self.report_path / f"test_summary_{self.test_id}.json"
|
||||
try:
|
||||
with open(results_path, 'w', encoding='utf-8') as f: json.dump(self.results_summary, f, indent=2, default=str)
|
||||
# console.print(f"\n[bold green]Results summary saved to {results_path}[/bold green]") # Moved summary print to run_full_test
|
||||
except Exception as e: console.print(f"[bold red]Failed to save results summary: {e}[/bold red]")
|
||||
|
||||
|
||||
# --- run_full_test Function (Adjusted) ---
|
||||
async def run_full_test(args):
|
||||
"""Run the complete test process from site generation to crawling."""
|
||||
server = None
|
||||
site_generated = False
|
||||
|
||||
# --- Site Generation --- (Same as before)
|
||||
if not args.use_existing_site and not args.skip_generation:
|
||||
if os.path.exists(args.site_path): console.print(f"[yellow]Removing existing site directory: {args.site_path}[/yellow]"); shutil.rmtree(args.site_path)
|
||||
site_generator = SiteGenerator(site_path=args.site_path, page_count=args.urls); site_generator.generate_site(); site_generated = True
|
||||
elif args.use_existing_site: console.print(f"[cyan]Using existing site assumed to be running on port {args.port}[/cyan]")
|
||||
elif args.skip_generation:
|
||||
console.print(f"[cyan]Skipping site generation, using existing directory: {args.site_path}[/cyan]")
|
||||
if not os.path.exists(args.site_path) or not os.path.isdir(args.site_path): console.print(f"[bold red]Error: Site path '{args.site_path}' does not exist or is not a directory.[/bold red]"); return
|
||||
|
||||
# --- Start Local Server --- (Same as before)
|
||||
server_started = False
|
||||
if not args.use_existing_site:
|
||||
server = LocalHttpServer(site_path=args.site_path, port=args.port)
|
||||
try: server.start(); server_started = True
|
||||
except Exception as e:
|
||||
console.print(f"[bold red]Failed to start local server. Aborting test.[/bold red]")
|
||||
if site_generated and not args.keep_site: console.print(f"[yellow]Cleaning up generated site: {args.site_path}[/yellow]"); shutil.rmtree(args.site_path)
|
||||
return
|
||||
|
||||
try:
|
||||
# --- Run the Stress Test ---
|
||||
test = CrawlerStressTest(
|
||||
url_count=args.urls,
|
||||
port=args.port,
|
||||
max_sessions=args.max_sessions,
|
||||
chunk_size=args.chunk_size, # Pass chunk_size
|
||||
report_path=args.report_path,
|
||||
stream_mode=args.stream,
|
||||
monitor_mode=args.monitor_mode,
|
||||
use_rate_limiter=args.use_rate_limiter
|
||||
)
|
||||
results = await test.run() # Run the test which now handles chunks internally
|
||||
|
||||
# --- Print Summary ---
|
||||
console.print("\n" + "=" * 80)
|
||||
console.print("[bold green]Test Completed[/bold green]")
|
||||
console.print("=" * 80)
|
||||
|
||||
# (Summary printing logic remains largely the same)
|
||||
success_rate = results["successful_urls"] / results["url_count"] * 100 if results["url_count"] > 0 else 0
|
||||
urls_per_second = results["urls_processed"] / results["total_time_seconds"] if results["total_time_seconds"] > 0 else 0
|
||||
|
||||
console.print(f"[bold cyan]Test ID:[/bold cyan] {results['test_id']}")
|
||||
console.print(f"[bold cyan]Configuration:[/bold cyan] {results['url_count']} URLs, {results['max_sessions']} sessions, Chunk: {results['chunk_size']}, Stream: {results['stream_mode']}, Monitor: {results['monitor_mode']}")
|
||||
console.print(f"[bold cyan]Results:[/bold cyan] {results['successful_urls']} successful, {results['failed_urls']} failed ({results['urls_processed']} processed, {success_rate:.1f}% success)")
|
||||
console.print(f"[bold cyan]Performance:[/bold cyan] {results['total_time_seconds']:.2f} seconds total, {urls_per_second:.2f} URLs/second avg")
|
||||
|
||||
mem_report = results.get("memory", {})
|
||||
mem_info_str = "Memory tracking data unavailable."
|
||||
if mem_report and not mem_report.get("error"):
|
||||
start_mb = mem_report.get('start_memory_mb'); end_mb = mem_report.get('end_memory_mb'); max_mb = mem_report.get('max_memory_mb'); growth_mb = mem_report.get('memory_growth_mb')
|
||||
mem_parts = []
|
||||
if start_mb is not None: mem_parts.append(f"Start: {start_mb:.1f} MB")
|
||||
if end_mb is not None: mem_parts.append(f"End: {end_mb:.1f} MB")
|
||||
if max_mb is not None: mem_parts.append(f"Max: {max_mb:.1f} MB")
|
||||
if growth_mb is not None: mem_parts.append(f"Growth: {growth_mb:.1f} MB")
|
||||
if mem_parts: mem_info_str = ", ".join(mem_parts)
|
||||
csv_path = mem_report.get('csv_path')
|
||||
if csv_path: console.print(f"[dim]Memory samples saved to: {csv_path}[/dim]")
|
||||
|
||||
console.print(f"[bold cyan]Memory Usage:[/bold cyan] {mem_info_str}")
|
||||
console.print(f"[bold green]Results summary saved to {results['memory']['csv_path'].replace('memory_samples', 'test_summary').replace('.csv', '.json')}[/bold green]") # Infer summary path
|
||||
|
||||
|
||||
if results["failed_urls"] > 0: console.print(f"\n[bold yellow]Warning: {results['failed_urls']} URLs failed to process ({100-success_rate:.1f}% failure rate)[/bold yellow]")
|
||||
if results["urls_processed"] < results["url_count"]: console.print(f"\n[bold red]Error: Only {results['urls_processed']} out of {results['url_count']} URLs were processed![/bold red]")
|
||||
|
||||
|
||||
finally:
|
||||
# --- Stop Server / Cleanup --- (Same as before)
|
||||
if server_started and server and not args.keep_server_alive: server.stop()
|
||||
elif server_started and server and args.keep_server_alive:
|
||||
console.print(f"[bold cyan]Server is kept running on port {args.port}. Press Ctrl+C to stop it.[/bold cyan]")
|
||||
try: await asyncio.Future() # Keep running indefinitely
|
||||
except KeyboardInterrupt: console.print("\n[bold yellow]Stopping server due to user interrupt...[/bold yellow]"); server.stop()
|
||||
|
||||
if site_generated and not args.keep_site: console.print(f"[yellow]Cleaning up generated site: {args.site_path}[/yellow]"); shutil.rmtree(args.site_path)
|
||||
elif args.clean_site and os.path.exists(args.site_path): console.print(f"[yellow]Cleaning up site directory as requested: {args.site_path}[/yellow]"); shutil.rmtree(args.site_path)
|
||||
|
||||
|
||||
# --- main Function (Added chunk_size argument) ---
|
||||
def main():
|
||||
"""Main entry point for the script."""
|
||||
parser = argparse.ArgumentParser(description="Crawl4AI SDK High Volume Stress Test using arun_many")
|
||||
|
||||
# Test parameters
|
||||
parser.add_argument("--urls", type=int, default=DEFAULT_URL_COUNT, help=f"Number of URLs to test (default: {DEFAULT_URL_COUNT})")
|
||||
parser.add_argument("--max-sessions", type=int, default=DEFAULT_MAX_SESSIONS, help=f"Maximum concurrent crawling sessions (default: {DEFAULT_MAX_SESSIONS})")
|
||||
parser.add_argument("--chunk-size", type=int, default=DEFAULT_CHUNK_SIZE, help=f"Number of URLs per batch for logging (default: {DEFAULT_CHUNK_SIZE})") # Added
|
||||
parser.add_argument("--stream", action="store_true", default=DEFAULT_STREAM_MODE, help=f"Enable streaming mode (disables batch logging) (default: {DEFAULT_STREAM_MODE})")
|
||||
parser.add_argument("--monitor-mode", type=str, default=DEFAULT_MONITOR_MODE, choices=["DETAILED", "AGGREGATED"], help=f"Display mode for the live monitor (default: {DEFAULT_MONITOR_MODE})")
|
||||
parser.add_argument("--use-rate-limiter", action="store_true", default=False, help="Enable a basic rate limiter (default: False)")
|
||||
|
||||
# Environment parameters
|
||||
parser.add_argument("--site-path", type=str, default=DEFAULT_SITE_PATH, help=f"Path to generate/use the test site (default: {DEFAULT_SITE_PATH})")
|
||||
parser.add_argument("--port", type=int, default=DEFAULT_PORT, help=f"Port for the local HTTP server (default: {DEFAULT_PORT})")
|
||||
parser.add_argument("--report-path", type=str, default=DEFAULT_REPORT_PATH, help=f"Path to save reports and logs (default: {DEFAULT_REPORT_PATH})")
|
||||
|
||||
# Site/Server management
|
||||
parser.add_argument("--skip-generation", action="store_true", help="Use existing test site folder without regenerating")
|
||||
parser.add_argument("--use-existing-site", action="store_true", help="Do not generate site or start local server; assume site exists on --port")
|
||||
parser.add_argument("--keep-server-alive", action="store_true", help="Keep the local HTTP server running after test")
|
||||
parser.add_argument("--keep-site", action="store_true", help="Keep the generated test site files after test")
|
||||
parser.add_argument("--clean-reports", action="store_true", help="Clean up report directory before running")
|
||||
parser.add_argument("--clean-site", action="store_true", help="Clean up site directory before running (if generating) or after")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Display config
|
||||
console.print("[bold underline]Crawl4AI SDK Stress Test Configuration[/bold underline]")
|
||||
console.print(f"URLs: {args.urls}, Max Sessions: {args.max_sessions}, Chunk Size: {args.chunk_size}") # Added chunk size
|
||||
console.print(f"Mode: {'Streaming' if args.stream else 'Batch'}, Monitor: {args.monitor_mode}, Rate Limit: {args.use_rate_limiter}")
|
||||
console.print(f"Site Path: {args.site_path}, Port: {args.port}, Report Path: {args.report_path}")
|
||||
console.print("-" * 40)
|
||||
# (Rest of config display and cleanup logic is the same)
|
||||
if args.use_existing_site: console.print("[cyan]Mode: Using existing external site/server[/cyan]")
|
||||
elif args.skip_generation: console.print("[cyan]Mode: Using existing site files, starting local server[/cyan]")
|
||||
else: console.print("[cyan]Mode: Generating site files, starting local server[/cyan]")
|
||||
if args.keep_server_alive: console.print("[cyan]Option: Keep server alive after test[/cyan]")
|
||||
if args.keep_site: console.print("[cyan]Option: Keep site files after test[/cyan]")
|
||||
if args.clean_reports: console.print("[cyan]Option: Clean reports before test[/cyan]")
|
||||
if args.clean_site: console.print("[cyan]Option: Clean site directory[/cyan]")
|
||||
console.print("-" * 40)
|
||||
|
||||
if args.clean_reports:
|
||||
if os.path.exists(args.report_path): console.print(f"[yellow]Cleaning up reports directory: {args.report_path}[/yellow]"); shutil.rmtree(args.report_path)
|
||||
os.makedirs(args.report_path, exist_ok=True)
|
||||
if args.clean_site and not args.use_existing_site:
|
||||
if os.path.exists(args.site_path): console.print(f"[yellow]Cleaning up site directory as requested: {args.site_path}[/yellow]"); shutil.rmtree(args.site_path)
|
||||
|
||||
# Run
|
||||
try: asyncio.run(run_full_test(args))
|
||||
except KeyboardInterrupt: console.print("\n[bold yellow]Test interrupted by user.[/bold yellow]")
|
||||
except Exception as e: console.print(f"\n[bold red]An unexpected error occurred:[/bold red] {e}"); import traceback; traceback.print_exc()
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user