wehub-resource-sync
431 lines
9.8 KiB
Markdown
431 lines
9.8 KiB
Markdown
# ScrapeGraphAI Testing Infrastructure
|
|
|
|
Comprehensive testing infrastructure for ScrapeGraphAI with support for unit tests, integration tests, and performance benchmarks.
|
|
|
|
## Table of Contents
|
|
|
|
- [Overview](#overview)
|
|
- [Test Organization](#test-organization)
|
|
- [Running Tests](#running-tests)
|
|
- [Test Fixtures](#test-fixtures)
|
|
- [Performance Benchmarking](#performance-benchmarking)
|
|
- [Mock Server](#mock-server)
|
|
- [CI/CD Integration](#cicd-integration)
|
|
|
|
## Overview
|
|
|
|
The testing infrastructure includes:
|
|
|
|
- **Unit Tests**: Fast, isolated tests with mocked dependencies
|
|
- **Integration Tests**: Tests with real LLM providers and websites
|
|
- **Performance Benchmarks**: Track performance metrics and detect regressions
|
|
- **Mock HTTP Server**: Consistent testing without external dependencies
|
|
- **Multi-Provider Support**: Test compatibility across different LLM providers
|
|
|
|
## Test Organization
|
|
|
|
```
|
|
tests/
|
|
├── conftest.py # Shared fixtures and pytest configuration
|
|
├── pytest.ini # Pytest settings (in project root)
|
|
├── fixtures/
|
|
│ ├── mock_server/ # Mock HTTP server for testing
|
|
│ │ ├── __init__.py
|
|
│ │ └── server.py
|
|
│ ├── benchmarking.py # Performance benchmarking utilities
|
|
│ ├── helpers.py # Test utilities and helpers
|
|
│ ├── data/ # Test data files
|
|
│ └── html/ # HTML fixtures
|
|
├── integration/ # Integration tests
|
|
│ ├── test_smart_scraper_integration.py
|
|
│ ├── test_multi_graph_integration.py
|
|
│ └── test_file_formats_integration.py
|
|
├── graphs/ # Graph-specific tests
|
|
├── nodes/ # Node-specific tests
|
|
└── utils/ # Utility tests
|
|
```
|
|
|
|
## Running Tests
|
|
|
|
### All Tests
|
|
|
|
```bash
|
|
pytest
|
|
```
|
|
|
|
### Unit Tests Only
|
|
|
|
```bash
|
|
pytest -m "unit or not integration"
|
|
```
|
|
|
|
### Integration Tests
|
|
|
|
```bash
|
|
pytest --integration
|
|
```
|
|
|
|
### With Coverage
|
|
|
|
```bash
|
|
pytest --cov=scrapegraphai --cov-report=html
|
|
```
|
|
|
|
### Performance Benchmarks
|
|
|
|
```bash
|
|
pytest --benchmark -m benchmark
|
|
```
|
|
|
|
### Slow Tests
|
|
|
|
```bash
|
|
pytest --slow
|
|
```
|
|
|
|
### Specific Test File
|
|
|
|
```bash
|
|
pytest tests/integration/test_smart_scraper_integration.py
|
|
```
|
|
|
|
### Verbose Output
|
|
|
|
```bash
|
|
pytest -v
|
|
```
|
|
|
|
## Test Fixtures
|
|
|
|
### LLM Provider Fixtures
|
|
|
|
Pre-configured fixtures for all supported LLM providers:
|
|
|
|
```python
|
|
def test_with_openai(openai_config):
|
|
"""Use OpenAI configuration."""
|
|
scraper = SmartScraperGraph(
|
|
prompt="...",
|
|
source="...",
|
|
config=openai_config
|
|
)
|
|
```
|
|
|
|
Available fixtures:
|
|
- `openai_config` - OpenAI GPT-3.5
|
|
- `openai_gpt4_config` - OpenAI GPT-4
|
|
- `ollama_config` - Ollama (local)
|
|
- `anthropic_config` - Anthropic Claude
|
|
- `groq_config` - Groq
|
|
- `azure_config` - Azure OpenAI
|
|
- `gemini_config` - Google Gemini
|
|
|
|
### Mock LLM Fixtures
|
|
|
|
For unit testing without API calls:
|
|
|
|
```python
|
|
def test_with_mock_llm(mock_llm_model, mock_embedder_model):
|
|
"""Use mocked LLM for fast unit tests."""
|
|
# Test logic here
|
|
```
|
|
|
|
### File Fixtures
|
|
|
|
Temporary files for testing:
|
|
|
|
```python
|
|
def test_json_scraping(temp_json_file):
|
|
"""Use temporary JSON file."""
|
|
scraper = JSONScraperGraph(
|
|
prompt="...",
|
|
source=temp_json_file,
|
|
config=config
|
|
)
|
|
```
|
|
|
|
Available fixtures:
|
|
- `temp_json_file`
|
|
- `temp_html_file`
|
|
- `temp_xml_file`
|
|
- `temp_csv_file`
|
|
|
|
### Mock HTTP Server
|
|
|
|
Local HTTP server for consistent testing:
|
|
|
|
```python
|
|
def test_with_mock_server(mock_server):
|
|
"""Use mock HTTP server."""
|
|
url = mock_server.get_url("/products")
|
|
|
|
scraper = SmartScraperGraph(
|
|
prompt="Extract products",
|
|
source=url,
|
|
config=config
|
|
)
|
|
```
|
|
|
|
Available endpoints:
|
|
- `/` - Home page
|
|
- `/products` - Products listing
|
|
- `/projects` - Projects listing
|
|
- `/api/data.json` - JSON endpoint
|
|
- `/api/data.xml` - XML endpoint
|
|
- `/api/data.csv` - CSV endpoint
|
|
- `/slow` - Slow response (2s delay)
|
|
- `/error/404` - 404 error
|
|
- `/error/500` - 500 error
|
|
- `/rate-limited` - Rate limiting simulation
|
|
- `/pagination?page=N` - Paginated content
|
|
|
|
## Performance Benchmarking
|
|
|
|
### Using the Benchmark Tracker
|
|
|
|
```python
|
|
def test_performance(benchmark_tracker):
|
|
"""Track performance metrics."""
|
|
import time
|
|
|
|
start = time.perf_counter()
|
|
# ... run scraping ...
|
|
end = time.perf_counter()
|
|
|
|
from tests.fixtures.benchmarking import BenchmarkResult
|
|
|
|
result = BenchmarkResult(
|
|
test_name="my_test",
|
|
execution_time=end - start,
|
|
token_usage=1000,
|
|
api_calls=2,
|
|
success=True
|
|
)
|
|
|
|
benchmark_tracker.record(result)
|
|
```
|
|
|
|
### Generating Reports
|
|
|
|
After running benchmarks:
|
|
|
|
```python
|
|
# In your test or conftest.py
|
|
tracker.save_results()
|
|
report = tracker.generate_report()
|
|
print(report)
|
|
```
|
|
|
|
### Comparing Against Baseline
|
|
|
|
```bash
|
|
# Save baseline
|
|
pytest --benchmark -m benchmark
|
|
cp benchmark_results/benchmark_results.json baseline.json
|
|
|
|
# Run tests and compare
|
|
pytest --benchmark -m benchmark
|
|
|
|
# Compare programmatically
|
|
from tests.fixtures.benchmarking import pytest_benchmark_compare
|
|
comparison = pytest_benchmark_compare(
|
|
Path("baseline.json"),
|
|
Path("benchmark_results/benchmark_results.json")
|
|
)
|
|
```
|
|
|
|
## Test Markers
|
|
|
|
### Available Markers
|
|
|
|
- `@pytest.mark.unit` - Unit tests (fast, no external deps)
|
|
- `@pytest.mark.integration` - Integration tests (require network)
|
|
- `@pytest.mark.slow` - Slow-running tests
|
|
- `@pytest.mark.benchmark` - Performance benchmarks
|
|
- `@pytest.mark.requires_api_key` - Tests requiring API keys
|
|
- `@pytest.mark.llm_provider(name)` - Tests for specific LLM provider
|
|
|
|
### Usage Example
|
|
|
|
```python
|
|
@pytest.mark.integration
|
|
@pytest.mark.requires_api_key
|
|
@pytest.mark.slow
|
|
def test_comprehensive_scraping(openai_config):
|
|
"""This test requires API keys and network access."""
|
|
# Test implementation
|
|
```
|
|
|
|
## Environment Variables
|
|
|
|
Set these environment variables for integration tests:
|
|
|
|
```bash
|
|
# LLM API Keys
|
|
export OPENAI_APIKEY="sk-..."
|
|
export ANTHROPIC_APIKEY="sk-ant-..."
|
|
export GROQ_APIKEY="gsk_..."
|
|
export GEMINI_APIKEY="..."
|
|
|
|
# Azure OpenAI
|
|
export AZURE_OPENAI_KEY="..."
|
|
export AZURE_OPENAI_ENDPOINT="https://..."
|
|
|
|
# Test Configuration
|
|
export TEST_WEBSITE_URL="https://scrapegrah-ai-website-for-tests.onrender.com"
|
|
export OLLAMA_BASE_URL="http://localhost:11434"
|
|
```
|
|
|
|
## CI/CD Integration
|
|
|
|
### GitHub Actions
|
|
|
|
The test suite runs automatically on:
|
|
- Push to main, pre/beta, dev branches
|
|
- Pull requests
|
|
- Daily scheduled runs
|
|
- Manual workflow dispatch
|
|
|
|
### Test Jobs
|
|
|
|
1. **Unit Tests**: Run on multiple OS and Python versions
|
|
2. **Integration Tests**: Test with real LLM providers
|
|
3. **Performance Benchmarks**: Track performance metrics
|
|
4. **Code Quality**: Linting, formatting, type checking
|
|
|
|
### Viewing Results
|
|
|
|
- Test results are uploaded as artifacts
|
|
- Coverage reports are sent to Codecov
|
|
- Performance benchmarks are saved for comparison
|
|
|
|
## Writing New Tests
|
|
|
|
### Unit Test Template
|
|
|
|
```python
|
|
import pytest
|
|
from unittest.mock import Mock, patch
|
|
|
|
class TestMyFeature:
|
|
@pytest.fixture
|
|
def setup(self):
|
|
"""Setup fixture for tests."""
|
|
return {"data": "value"}
|
|
|
|
def test_my_function(self, setup, mock_llm_model):
|
|
"""Test description."""
|
|
# Arrange
|
|
# Act
|
|
# Assert
|
|
```
|
|
|
|
### Integration Test Template
|
|
|
|
```python
|
|
import pytest
|
|
from scrapegraphai.graphs import SmartScraperGraph
|
|
|
|
@pytest.mark.integration
|
|
@pytest.mark.requires_api_key
|
|
class TestMyIntegration:
|
|
def test_real_scraping(self, openai_config, mock_server):
|
|
"""Test with real LLM provider."""
|
|
url = mock_server.get_url("/test-page")
|
|
|
|
scraper = SmartScraperGraph(
|
|
prompt="Extract data",
|
|
source=url,
|
|
config=openai_config
|
|
)
|
|
|
|
result = scraper.run()
|
|
|
|
assert result is not None
|
|
assert isinstance(result, dict)
|
|
```
|
|
|
|
### Benchmark Test Template
|
|
|
|
```python
|
|
import pytest
|
|
import time
|
|
from tests.fixtures.benchmarking import BenchmarkResult
|
|
|
|
@pytest.mark.benchmark
|
|
class TestMyBenchmark:
|
|
def test_performance(self, benchmark_tracker, openai_config):
|
|
"""Benchmark test description."""
|
|
start = time.perf_counter()
|
|
|
|
# Run operation to benchmark
|
|
|
|
end = time.perf_counter()
|
|
|
|
result = BenchmarkResult(
|
|
test_name="my_benchmark",
|
|
execution_time=end - start,
|
|
success=True
|
|
)
|
|
|
|
benchmark_tracker.record(result)
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Tests Timeout
|
|
|
|
Increase timeout in pytest.ini or per-test:
|
|
|
|
```python
|
|
@pytest.mark.timeout(120) # 2 minutes
|
|
def test_long_running():
|
|
pass
|
|
```
|
|
|
|
### API Rate Limits
|
|
|
|
Use mock server or implement rate limiting in tests:
|
|
|
|
```python
|
|
from tests.fixtures.helpers import RateLimitHelper
|
|
|
|
rate_limiter = RateLimitHelper(max_requests=5, time_window=60)
|
|
```
|
|
|
|
### Flaky Tests
|
|
|
|
Mark tests as flaky and allow retries:
|
|
|
|
```python
|
|
@pytest.mark.flaky(reruns=3, reruns_delay=2)
|
|
def test_sometimes_fails():
|
|
pass
|
|
```
|
|
|
|
## Best Practices
|
|
|
|
1. **Use appropriate markers** - Mark tests correctly for proper filtering
|
|
2. **Mock external dependencies** - Use mock server and fixtures
|
|
3. **Test isolation** - Each test should be independent
|
|
4. **Clear assertions** - Use helper functions for better error messages
|
|
5. **Performance tracking** - Use benchmarking for critical paths
|
|
6. **Documentation** - Document test purpose and requirements
|
|
7. **Cleanup** - Use fixtures and context managers for proper cleanup
|
|
|
|
## Contributing
|
|
|
|
When adding tests:
|
|
|
|
1. Follow existing test structure and naming conventions
|
|
2. Add appropriate markers
|
|
3. Document test requirements (API keys, network, etc.)
|
|
4. Update this README if adding new test infrastructure
|
|
5. Ensure tests pass in CI before submitting PR
|
|
|
|
## Additional Resources
|
|
|
|
- [pytest Documentation](https://docs.pytest.org/)
|
|
- [pytest-cov Documentation](https://pytest-cov.readthedocs.io/)
|
|
- [ScrapeGraphAI Documentation](https://docs.scrapegraphai.com/introduction)
|