lmcache--lmcache/examples/chunk_statistics

Chunk Statistics Example

This example demonstrates how to use LMCache's chunk statistics feature to track and analyze KV cache chunk reuse patterns.

Overview

Chunk statistics provides insights into cache efficiency by tracking:

• Total chunks processed
• Unique chunks encountered
• Duplicate chunks (cache hits)
• Reuse rate (duplicate/total ratio)

Prerequisites

• LMCache installed with vLLM integration
• A model for testing (e.g., /data1/deepseek/DeepSeek-V2-Lite-Chat)

Examples

Example 1: Memory Bloom Filter Strategy

The memory bloom filter strategy uses a probabilistic data structure for efficient duplicate detection with minimal memory overhead.

Configuration

See memory_bloom_filter.yaml for the configuration file.

Running the Example

# Start vLLM with chunk statistics enabled
LMCACHE_CONFIG_FILE=memory_bloom_filter.yaml \
PYTHONHASHSEED=0 \
python3 -m vllm.entrypoints.cli.main serve <model_path> \
--load-format dummy \
-tp 2 \
--trust-remote-code \
--served-model-name vllm_cpu_offload \
--gpu-memory-utilization 0.5 \
--max-num-seqs 64 \
--no-enable-prefix-caching \
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}'

Query Statistics

# Get current statistics (default port: 6999 for scheduler)
curl http://localhost:6999/chunk_statistics/status

# Pretty print JSON output
curl http://localhost:6999/chunk_statistics/status | jq .

# Start statistics collection (if not auto-started)
curl -X POST http://localhost:6999/chunk_statistics/start

# Stop statistics collection
curl -X POST http://localhost:6999/chunk_statistics/stop

# Reset statistics
curl -X POST http://localhost:6999/chunk_statistics/reset

Expected Output

{
  "enabled": true,
  "total_requests": 3,
  "timing": {
    "lookup_time_seconds": 0.044486284255981445,
    "record_statistics_time_seconds": 6.246566772460938e-05,
    "check_exit_conditions_time_seconds": 5.7220458984375e-06,
    "total_time_seconds": 0.04455447196960449,
    "overhead_time_seconds": 6.818771362304688e-05,
    "overhead_percentage": 0.1530434782608696
  },
  "total_chunks": 12,
  "unique_chunks": 9,
  "duplicate_chunks": 3,
  "reuse_rate": 0.25,
  "async_queue": {
    "enabled": true,
    "capacity": 100000,
    "current_size": 0,
    "max_size_reached": 0,
    "full_blocks": 0,
    "utilization": 0.0
  },
  "bloom_filter": {
    "size_mb": 11.426279067993164,
    "hash_count": 6,
    "item_count": 9,
    "bits_set": 54,
    "fill_rate": 5.633768549952377e-07,
    "expected_elements": 10000000,
    "false_positive_rate": 0.01
  },
  "timestamp": 1763026696.7670634,
  "auto_exit_enabled": false,
  "auto_exit_timeout_hours": 0.0,
  "auto_exit_target_unique_chunks": null
}

Example 2: File Hash Strategy

The file hash strategy writes chunk hashes to disk for exact tracking and offline analysis.

Configuration

See file_hash.yaml for the configuration file.

Running the Example

# Start vLLM with file hash strategy
LMCACHE_CONFIG_FILE=file_hash.yaml \
PYTHONHASHSEED=0 \
python3 -m vllm.entrypoints.cli.main serve <model_path> \
--load-format dummy \
-tp 2 \
--trust-remote-code \
--served-model-name vllm_cpu_offload \
--gpu-memory-utilization 0.5 \
--max-num-seqs 64 \
--no-enable-prefix-caching \
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}'

Analyze Collected Data

Use the provided Python script to analyze the collected chunk hashes:

# Analyze chunk hashes from default directory
python analyze_chunk_hashes.py --input-dir /tmp/lmcache_chunk_statistics

# Export results to JSON file
python analyze_chunk_hashes.py --input-dir /tmp/lmcache_chunk_statistics --output analysis_results.json

Example 3: Auto-Stop Configuration

This example demonstrates automatic stopping based on time or chunk count.

Configuration

See auto_stop.yaml for the configuration file.

Running the Example

# Statistics will automatically stop after configured time or chunk count
LMCACHE_CONFIG_FILE=auto_stop.yaml \
PYTHONHASHSEED=0 \
python3 -m vllm.entrypoints.cli.main serve <model_path> \
--load-format dummy \
-tp 2 \
--trust-remote-code \
--served-model-name vllm_cpu_offload \
--gpu-memory-utilization 0.5 \
--max-num-seqs 64 \
--no-enable-prefix-caching \
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}'

Configuration Options

Memory Bloom Filter Strategy

Option	Default	Description
chunk_statistics_mem_bf_expected_chunks	20000000	Expected number of chunks for capacity planning
chunk_statistics_mem_bf_false_positive_rate	0.01	Target false positive rate (1%)

File Hash Strategy

Option	Default	Description
chunk_statistics_file_output_dir	/tmp/lmcache_chunk_statistics	Directory for storing chunk hash files
chunk_statistics_file_rotation_size	104857600	File size threshold for rotation (100MB)
chunk_statistics_file_max_count	100	Maximum number of files to keep

Understanding the Metrics

Reuse Rate

The reuse rate indicates cache efficiency:

• 0.0: No cache reuse (all chunks are unique)
• 0.5: 50% of chunks are duplicates
• 0.9: 90% of chunks are duplicates (high cache efficiency)

Bloom Filter Metrics

• size_mb: Memory used by the bloom filter
• fill_rate: Percentage of bits set in the bloom filter (0.0 to 1.0)
• false_positive_rate: Configured target false positive rate

Async Queue Metrics

• capacity: Maximum queue size
• current_size: Current number of items in queue
• max_size_reached: Peak queue size observed
• full_blocks: Number of times the queue was full
• utilization: Current queue utilization (0.0 to 1.0)

Best Practices

1. Choose the Right Strategy:

   • Use memory_bloom_filter for real-time monitoring with minimal overhead
   • Use file_hash for exact tracking and offline analysis

2. Tune Bloom Filter Parameters:

   • Set expected_chunks based on your workload size
   • Lower false_positive_rate increases memory usage but improves accuracy

3. Monitor Memory Usage:

   • Track bloom_filter_size_mb to ensure it fits in available memory
   • Adjust expected_chunks if memory usage is too high

4. File Rotation:

   • Configure appropriate file_rotation_size to balance file size and count
   • Set file_max_count to prevent unlimited disk usage

Troubleshooting

Statistics Not Updating

Problem: Statistics remain at zero or don't update.

Solution:

• Verify enable_chunk_statistics is set to true
• Check that statistics collection is started
• Ensure requests are being processed

High Memory Usage

Problem: Bloom filter consuming too much memory.

Solution:

• Reduce chunk_statistics_mem_bf_expected_chunks
• Increase chunk_statistics_mem_bf_false_positive_rate
• Consider switching to file_hash strategy

File System Full

Problem: Disk space exhausted with file hash strategy.

Solution:

• Reduce chunk_statistics_file_max_count
• Decrease chunk_statistics_file_rotation_size
• Implement external log rotation

Additional Resources

• Chunk Statistics Documentation
• Internal API Server Documentation