promptfoo--promptfoo/docs/scheduler-architecture.md

Adaptive Rate Limit Scheduler - Architecture

Overview

The adaptive rate limit scheduler automatically handles provider rate limits during evaluations. It's zero-configuration - users don't need to change anything. The scheduler transparently wraps all provider calls with intelligent rate limit detection, retry logic, and adaptive concurrency management.

Design Goals

The scheduler addresses common challenges when running evaluations against rate-limited APIs:

• No manual tuning: Users shouldn't need to guess the right -j (concurrency) value
• Automatic recovery: Rate limit errors (429) should be retried, not fail permanently
• Prevent cascading failures: High concurrency shouldn't cause mass failures
• Zero configuration: Works out of the box with sensible defaults

Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                              Evaluator                                       │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                     RateLimitRegistry                                │    │
│  │  (Central coordinator - one per evaluation)                          │    │
│  │                                                                      │    │
│  │  ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐   │    │
│  │  │ProviderRateLimit │  │ProviderRateLimit │  │ProviderRateLimit │   │    │
│  │  │     State        │  │     State        │  │     State        │   │    │
│  │  │  (openai/key1)   │  │  (openai/key2)   │  │  (anthropic)     │   │    │
│  │  │                  │  │                  │  │                  │   │    │
│  │  │ ┌─────────────┐  │  │ ┌─────────────┐  │  │ ┌─────────────┐  │   │    │
│  │  │ │  SlotQueue  │  │  │ │  SlotQueue  │  │  │ │  SlotQueue  │  │   │    │
│  │  │ │ (FIFO)      │  │  │ │ (FIFO)      │  │  │ │ (FIFO)      │  │   │    │
│  │  │ └─────────────┘  │  │ └─────────────┘  │  │ └─────────────┘  │   │    │
│  │  │ ┌─────────────┐  │  │ ┌─────────────┐  │  │ ┌─────────────┐  │   │    │
│  │  │ │  Adaptive   │  │  │ │  Adaptive   │  │  │ │  Adaptive   │  │   │    │
│  │  │ │ Concurrency │  │  │ │ Concurrency │  │  │ │ Concurrency │  │   │    │
│  │  │ └─────────────┘  │  │ └─────────────┘  │  │ └─────────────┘  │   │    │
│  │  └──────────────────┘  └──────────────────┘  └──────────────────┘   │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────────────┘

Component Responsibilities

RateLimitRegistry

File: src/scheduler/rateLimitRegistry.ts

Central coordinator that:

• Creates/retrieves per-provider state based on rate limit keys
• Routes provider calls to the appropriate state
• Aggregates metrics across all providers
• Emits events for monitoring

// Usage (automatic in evaluator)
const result = await registry.execute(provider, () => provider.callApi(...), {
  getHeaders: (result) => result.metadata?.headers,
  isRateLimited: (result, error) => error?.message?.includes('429'),
  getRetryAfter: (result, error) => parseRetryAfter(headers['retry-after']),
});

ProviderRateLimitState

File: src/scheduler/providerRateLimitState.ts

Per-provider state manager that:

• Manages the slot queue for concurrency control
• Tracks rate limit headers from responses
• Implements retry logic with exponential backoff
• Adapts concurrency based on success/failure patterns
• Collects latency metrics

SlotQueue

File: src/scheduler/slotQueue.ts

FIFO queue with concurrency limiting:

• Acquires/releases "slots" for concurrent requests
• Blocks when at max concurrency or quota exhausted
• Tracks remaining requests/tokens from headers
• Schedules queue processing after rate limit windows

Key insight: Race-condition-free slot allocation. All requests queue, then slots are allocated in FIFO order.

AdaptiveConcurrency

File: src/scheduler/adaptiveConcurrency.ts

Dynamic concurrency adjustment:

• On rate limit: Reduce concurrency by 50% (multiplicative decrease)
• On sustained success: Increase by 1 (additive increase)
• Proactive throttling: Reduce when approaching limits (via headers)

This implements AIMD (Additive Increase, Multiplicative Decrease) - the same algorithm TCP uses for congestion control.

HeaderParser

File: src/scheduler/headerParser.ts

Parses rate limit headers from multiple providers:

• OpenAI: x-ratelimit-remaining-requests, x-ratelimit-limit-requests
• Anthropic: anthropic-ratelimit-requests-remaining
• Generic: retry-after, retry-after-ms, ratelimit-reset

RetryPolicy

File: src/scheduler/retryPolicy.ts

Determines retry behavior:

• Exponential backoff with jitter
• Respects server retry-after headers
• Configurable max retries (default: 3)
• Retries on: rate limits, timeouts, 502/503/504

Data Flow

1. Evaluator calls registry.execute(provider, callFn)
       │
       ▼
2. Registry gets/creates ProviderRateLimitState for this provider
       │
       ▼
3. State.executeWithRetry() is called
       │
       ▼
4. SlotQueue.acquire() - wait for available slot
       │
       ▼
5. Execute callFn() - actual provider API call
       │
       ▼
6. Parse response headers → update rate limit state
       │
       ▼
7. Check if rate limited:
   ├─ Yes → retry with backoff, reduce concurrency
   └─ No  → record success, maybe increase concurrency
       │
       ▼
8. SlotQueue.release() - free slot for next request
       │
       ▼
9. Return result (or throw after max retries)

Rate Limit Key Generation

Each provider gets a unique "rate limit key" based on:

• Provider ID (e.g., "openai:chat:gpt-4o")
• API key hash (different keys = different rate limits)
• Organization ID (if applicable)

This ensures:

• Same provider + same key = shared rate limit state
• Same provider + different keys = separate rate limits
• Different providers = completely isolated

Key Design Decisions

1. Zero Configuration

Users shouldn't need to tune rate limit settings. The scheduler learns from response headers and adapts automatically.

2. Fail-Safe Defaults

• Default max concurrency: 4 (conservative)
• Default retry delay: 60 seconds (when no header)
• Max retries: 3 (prevents infinite loops)

3. Proactive Throttling

Don't wait for 429 errors. When headers show <10% remaining quota, proactively reduce concurrency.

4. Per-Provider Isolation

Different providers have different rate limits. Don't let OpenAI rate limits affect Anthropic calls.

5. Transparent Integration

The scheduler wraps provider.callApi() without changing the interface. Existing code works unchanged.

Metrics

The scheduler tracks:

• totalRequests - All requests attempted
• completedRequests - Successful completions
• failedRequests - Permanent failures (after retries)
• rateLimitHits - Times 429 was encountered
• retriedRequests - Requests that required retry
• avgLatencyMs, p50LatencyMs, p99LatencyMs - Latency distribution

Events

For monitoring/debugging, the scheduler emits:

• slot:acquired / slot:released - Concurrency tracking
• ratelimit:hit - Rate limit encountered
• ratelimit:learned - First time seeing provider's limits
• ratelimit:warning - Approaching rate limit
• concurrency:increased / concurrency:decreased - Adaptive changes
• request:retrying - Retry in progress

Testing

256 tests covering:

• Unit tests for each component
• Edge cases (negative values, zero values, overflow)
• Race condition prevention
• Integration with evaluator

Performance Characteristics

• Overhead: Minimal - just slot acquisition and header parsing
• Memory: O(providers) - one state object per unique rate limit key
• Latency buffer: Circular buffer, last 100 requests, O(1) insertion