scrapegraphai--scrapegraph-ai/tests/README_TESTING.md

ScrapeGraphAI Testing Infrastructure

Comprehensive testing infrastructure for ScrapeGraphAI with support for unit tests, integration tests, and performance benchmarks.

Table of Contents

• Overview
• Test Organization
• Running Tests
• Test Fixtures
• Performance Benchmarking
• Mock Server
• CI/CD Integration

Overview

The testing infrastructure includes:

• Unit Tests: Fast, isolated tests with mocked dependencies
• Integration Tests: Tests with real LLM providers and websites
• Performance Benchmarks: Track performance metrics and detect regressions
• Mock HTTP Server: Consistent testing without external dependencies
• Multi-Provider Support: Test compatibility across different LLM providers

Test Organization

tests/
├── conftest.py                 # Shared fixtures and pytest configuration
├── pytest.ini                  # Pytest settings (in project root)
├── fixtures/
│   ├── mock_server/           # Mock HTTP server for testing
│   │   ├── __init__.py
│   │   └── server.py
│   ├── benchmarking.py        # Performance benchmarking utilities
│   ├── helpers.py             # Test utilities and helpers
│   ├── data/                  # Test data files
│   └── html/                  # HTML fixtures
├── integration/               # Integration tests
│   ├── test_smart_scraper_integration.py
│   ├── test_multi_graph_integration.py
│   └── test_file_formats_integration.py
├── graphs/                    # Graph-specific tests
├── nodes/                     # Node-specific tests
└── utils/                     # Utility tests

Running Tests

All Tests

pytest

Unit Tests Only

pytest -m "unit or not integration"

Integration Tests

pytest --integration

With Coverage

pytest --cov=scrapegraphai --cov-report=html

Performance Benchmarks

pytest --benchmark -m benchmark

Slow Tests

pytest --slow

Specific Test File

pytest tests/integration/test_smart_scraper_integration.py

Verbose Output

pytest -v

Test Fixtures

LLM Provider Fixtures

Pre-configured fixtures for all supported LLM providers:

def test_with_openai(openai_config):
    """Use OpenAI configuration."""
    scraper = SmartScraperGraph(
        prompt="...",
        source="...",
        config=openai_config
    )

Available fixtures:

• openai_config - OpenAI GPT-3.5
• openai_gpt4_config - OpenAI GPT-4
• ollama_config - Ollama (local)
• anthropic_config - Anthropic Claude
• groq_config - Groq
• azure_config - Azure OpenAI
• gemini_config - Google Gemini

Mock LLM Fixtures

For unit testing without API calls:

def test_with_mock_llm(mock_llm_model, mock_embedder_model):
    """Use mocked LLM for fast unit tests."""
    # Test logic here

File Fixtures

Temporary files for testing:

def test_json_scraping(temp_json_file):
    """Use temporary JSON file."""
    scraper = JSONScraperGraph(
        prompt="...",
        source=temp_json_file,
        config=config
    )

Available fixtures:

• temp_json_file
• temp_html_file
• temp_xml_file
• temp_csv_file

Mock HTTP Server

Local HTTP server for consistent testing:

def test_with_mock_server(mock_server):
    """Use mock HTTP server."""
    url = mock_server.get_url("/products")

    scraper = SmartScraperGraph(
        prompt="Extract products",
        source=url,
        config=config
    )

Available endpoints:

• / - Home page
• /products - Products listing
• /projects - Projects listing
• /api/data.json - JSON endpoint
• /api/data.xml - XML endpoint
• /api/data.csv - CSV endpoint
• /slow - Slow response (2s delay)
• /error/404 - 404 error
• /error/500 - 500 error
• /rate-limited - Rate limiting simulation
• /pagination?page=N - Paginated content

Performance Benchmarking

Using the Benchmark Tracker

def test_performance(benchmark_tracker):
    """Track performance metrics."""
    import time

    start = time.perf_counter()
    # ... run scraping ...
    end = time.perf_counter()

    from tests.fixtures.benchmarking import BenchmarkResult

    result = BenchmarkResult(
        test_name="my_test",
        execution_time=end - start,
        token_usage=1000,
        api_calls=2,
        success=True
    )

    benchmark_tracker.record(result)

Generating Reports

After running benchmarks:

# In your test or conftest.py
tracker.save_results()
report = tracker.generate_report()
print(report)

Comparing Against Baseline

# Save baseline
pytest --benchmark -m benchmark
cp benchmark_results/benchmark_results.json baseline.json

# Run tests and compare
pytest --benchmark -m benchmark

# Compare programmatically
from tests.fixtures.benchmarking import pytest_benchmark_compare
comparison = pytest_benchmark_compare(
    Path("baseline.json"),
    Path("benchmark_results/benchmark_results.json")
)

Test Markers

Available Markers

• @pytest.mark.unit - Unit tests (fast, no external deps)
• @pytest.mark.integration - Integration tests (require network)
• @pytest.mark.slow - Slow-running tests
• @pytest.mark.benchmark - Performance benchmarks
• @pytest.mark.requires_api_key - Tests requiring API keys
• @pytest.mark.llm_provider(name) - Tests for specific LLM provider

Usage Example

@pytest.mark.integration
@pytest.mark.requires_api_key
@pytest.mark.slow
def test_comprehensive_scraping(openai_config):
    """This test requires API keys and network access."""
    # Test implementation

Environment Variables

Set these environment variables for integration tests:

# LLM API Keys
export OPENAI_APIKEY="sk-..."
export ANTHROPIC_APIKEY="sk-ant-..."
export GROQ_APIKEY="gsk_..."
export GEMINI_APIKEY="..."

# Azure OpenAI
export AZURE_OPENAI_KEY="..."
export AZURE_OPENAI_ENDPOINT="https://..."

# Test Configuration
export TEST_WEBSITE_URL="https://scrapegrah-ai-website-for-tests.onrender.com"
export OLLAMA_BASE_URL="http://localhost:11434"

CI/CD Integration

GitHub Actions

The test suite runs automatically on:

• Push to main, pre/beta, dev branches
• Pull requests
• Daily scheduled runs
• Manual workflow dispatch

Test Jobs

1. Unit Tests: Run on multiple OS and Python versions
2. Integration Tests: Test with real LLM providers
3. Performance Benchmarks: Track performance metrics
4. Code Quality: Linting, formatting, type checking

Viewing Results

• Test results are uploaded as artifacts
• Coverage reports are sent to Codecov
• Performance benchmarks are saved for comparison

Writing New Tests

Unit Test Template

import pytest
from unittest.mock import Mock, patch

class TestMyFeature:
    @pytest.fixture
    def setup(self):
        """Setup fixture for tests."""
        return {"data": "value"}

    def test_my_function(self, setup, mock_llm_model):
        """Test description."""
        # Arrange
        # Act
        # Assert

Integration Test Template

import pytest
from scrapegraphai.graphs import SmartScraperGraph

@pytest.mark.integration
@pytest.mark.requires_api_key
class TestMyIntegration:
    def test_real_scraping(self, openai_config, mock_server):
        """Test with real LLM provider."""
        url = mock_server.get_url("/test-page")

        scraper = SmartScraperGraph(
            prompt="Extract data",
            source=url,
            config=openai_config
        )

        result = scraper.run()

        assert result is not None
        assert isinstance(result, dict)

Benchmark Test Template

import pytest
import time
from tests.fixtures.benchmarking import BenchmarkResult

@pytest.mark.benchmark
class TestMyBenchmark:
    def test_performance(self, benchmark_tracker, openai_config):
        """Benchmark test description."""
        start = time.perf_counter()

        # Run operation to benchmark

        end = time.perf_counter()

        result = BenchmarkResult(
            test_name="my_benchmark",
            execution_time=end - start,
            success=True
        )

        benchmark_tracker.record(result)

Troubleshooting

Tests Timeout

Increase timeout in pytest.ini or per-test:

@pytest.mark.timeout(120)  # 2 minutes
def test_long_running():
    pass

API Rate Limits

Use mock server or implement rate limiting in tests:

from tests.fixtures.helpers import RateLimitHelper

rate_limiter = RateLimitHelper(max_requests=5, time_window=60)

Flaky Tests

Mark tests as flaky and allow retries:

@pytest.mark.flaky(reruns=3, reruns_delay=2)
def test_sometimes_fails():
    pass

Best Practices

1. Use appropriate markers - Mark tests correctly for proper filtering
2. Mock external dependencies - Use mock server and fixtures
3. Test isolation - Each test should be independent
4. Clear assertions - Use helper functions for better error messages
5. Performance tracking - Use benchmarking for critical paths
6. Documentation - Document test purpose and requirements
7. Cleanup - Use fixtures and context managers for proper cleanup

Contributing

When adding tests:

1. Follow existing test structure and naming conventions
2. Add appropriate markers
3. Document test requirements (API keys, network, etc.)
4. Update this README if adding new test infrastructure
5. Ensure tests pass in CI before submitting PR

Additional Resources

• pytest Documentation
• pytest-cov Documentation
• ScrapeGraphAI Documentation