chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,666 @@
|
||||
# LLM Query Cache Cleanup Tool - User Guide
|
||||
|
||||
## Overview
|
||||
|
||||
This tool cleans up LightRAG's LLM query cache from KV storage implementations. It specifically targets query caches generated during RAG query operations (modes: `mix`, `hybrid`, `local`, `global`), including both query and keywords caches.
|
||||
|
||||
## Supported Storage Types
|
||||
|
||||
1. **JsonKVStorage** - File-based JSON storage
|
||||
2. **RedisKVStorage** - Redis database storage
|
||||
3. **PGKVStorage** - PostgreSQL database storage
|
||||
4. **MongoKVStorage** - MongoDB database storage
|
||||
5. **OpenSearchKVStorage** - OpenSearch index storage
|
||||
|
||||
## Cache Types
|
||||
|
||||
The tool cleans up the following query cache types:
|
||||
|
||||
### Query Cache Modes (4 types)
|
||||
- `mix:*` - Mixed mode query caches
|
||||
- `hybrid:*` - Hybrid mode query caches
|
||||
- `local:*` - Local mode query caches
|
||||
- `global:*` - Global mode query caches
|
||||
|
||||
### Cache Content Types (2 types)
|
||||
- `*:query:*` - Query result caches
|
||||
- `*:keywords:*` - Keywords extraction caches
|
||||
|
||||
### Cache Key Format
|
||||
```
|
||||
<mode>:<cache_type>:<hash>
|
||||
```
|
||||
|
||||
Examples:
|
||||
- `mix:query:5ce04d25e957c290216cee5bfe6344fa`
|
||||
- `mix:keywords:fee77b98244a0b047ce95e21060de60e`
|
||||
- `global:query:abc123def456...`
|
||||
- `local:keywords:789xyz...`
|
||||
|
||||
**Important Note**: This tool does NOT clean extraction caches (`default:extract:*` and `default:summary:*`). Use the migration tool or manual deletion for those caches.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- The tool reads storage configuration from environment variables
|
||||
- Ensure the target storage is properly configured and accessible
|
||||
- Backup important data before running cleanup operations
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run from the LightRAG project root directory:
|
||||
|
||||
```bash
|
||||
python -m lightrag.tools.clean_llm_query_cache
|
||||
# or
|
||||
python lightrag/tools/clean_llm_query_cache.py
|
||||
```
|
||||
|
||||
### Interactive Workflow
|
||||
|
||||
The tool guides you through the following steps:
|
||||
|
||||
#### 1. Select Storage Type
|
||||
```
|
||||
============================================================
|
||||
LLM Query Cache Cleanup Tool - LightRAG
|
||||
============================================================
|
||||
|
||||
=== Storage Setup ===
|
||||
|
||||
Supported KV Storage Types:
|
||||
[1] JsonKVStorage
|
||||
[2] RedisKVStorage
|
||||
[3] PGKVStorage
|
||||
[4] MongoKVStorage
|
||||
[5] OpenSearchKVStorage
|
||||
|
||||
Select storage type (1-5) (Press Enter to exit): 1
|
||||
```
|
||||
|
||||
**Note**: You can press Enter or type `0` at any prompt to exit gracefully.
|
||||
|
||||
#### 2. Storage Validation
|
||||
The tool will:
|
||||
- Check required environment variables
|
||||
- Auto-detect workspace configuration
|
||||
- Initialize and connect to storage
|
||||
- Verify connection status
|
||||
|
||||
```
|
||||
Checking configuration...
|
||||
✓ All required environment variables are set
|
||||
|
||||
Initializing storage...
|
||||
- Storage Type: JsonKVStorage
|
||||
- Workspace: space1
|
||||
- Connection Status: ✓ Success
|
||||
```
|
||||
|
||||
#### 3. View Cache Statistics
|
||||
|
||||
The tool displays a detailed breakdown of query caches by mode and type:
|
||||
|
||||
```
|
||||
Counting query cache records...
|
||||
|
||||
📊 Query Cache Statistics (Before Cleanup):
|
||||
┌────────────┬────────────┬────────────┬────────────┐
|
||||
│ Mode │ Query │ Keywords │ Total │
|
||||
├────────────┼────────────┼────────────┼────────────┤
|
||||
│ mix │ 1,234 │ 567 │ 1,801 │
|
||||
│ hybrid │ 890 │ 423 │ 1,313 │
|
||||
│ local │ 2,345 │ 1,123 │ 3,468 │
|
||||
│ global │ 678 │ 345 │ 1,023 │
|
||||
├────────────┼────────────┼────────────┼────────────┤
|
||||
│ Total │ 5,147 │ 2,458 │ 7,605 │
|
||||
└────────────┴────────────┴────────────┴────────────┘
|
||||
```
|
||||
|
||||
#### 4. Select Cleanup Scope
|
||||
|
||||
Choose what type of caches to delete:
|
||||
|
||||
```
|
||||
=== Cleanup Options ===
|
||||
[1] Delete all query caches (both query and keywords)
|
||||
[2] Delete query caches only (keep keywords)
|
||||
[3] Delete keywords caches only (keep query)
|
||||
[0] Cancel
|
||||
|
||||
Select cleanup option (0-3): 1
|
||||
```
|
||||
|
||||
**Cleanup Types:**
|
||||
- **Option 1 (all)**: Deletes both query and keywords caches across all modes
|
||||
- **Option 2 (query)**: Deletes only query caches, preserves keywords caches
|
||||
- **Option 3 (keywords)**: Deletes only keywords caches, preserves query caches
|
||||
|
||||
#### 5. Confirm Deletion
|
||||
|
||||
Review the cleanup plan and confirm:
|
||||
|
||||
```
|
||||
============================================================
|
||||
Cleanup Confirmation
|
||||
============================================================
|
||||
Storage: JsonKVStorage (workspace: space1)
|
||||
Cleanup Type: all
|
||||
Records to Delete: 7,605 / 7,605
|
||||
|
||||
⚠️ WARNING: This will delete ALL query caches across all modes!
|
||||
|
||||
Continue with deletion? (y/n): y
|
||||
```
|
||||
|
||||
#### 6. Execute Cleanup
|
||||
|
||||
The tool performs batch deletion with real-time progress:
|
||||
|
||||
**JsonKVStorage Example:**
|
||||
```
|
||||
=== Starting Cleanup ===
|
||||
💡 Processing 1,000 records at a time from JsonKVStorage
|
||||
|
||||
Batch 1/8: ████░░░░░░░░░░░░░░░░ 1,000/7,605 (13.1%) ✓
|
||||
Batch 2/8: ████████░░░░░░░░░░░░ 2,000/7,605 (26.3%) ✓
|
||||
...
|
||||
Batch 8/8: ████████████████████ 7,605/7,605 (100.0%) ✓
|
||||
|
||||
Persisting changes to storage...
|
||||
✓ Changes persisted successfully
|
||||
```
|
||||
|
||||
**RedisKVStorage Example:**
|
||||
```
|
||||
=== Starting Cleanup ===
|
||||
💡 Processing Redis keys in batches of 1,000
|
||||
|
||||
Batch 1: Deleted 1,000 keys (Total: 1,000) ✓
|
||||
Batch 2: Deleted 1,000 keys (Total: 2,000) ✓
|
||||
...
|
||||
```
|
||||
|
||||
**PostgreSQL Example:**
|
||||
```
|
||||
=== Starting Cleanup ===
|
||||
💡 Executing PostgreSQL DELETE query
|
||||
|
||||
✓ Deleted 7,605 records in 0.45s
|
||||
```
|
||||
|
||||
**MongoDB Example:**
|
||||
```
|
||||
=== Starting Cleanup ===
|
||||
💡 Executing MongoDB deleteMany operations
|
||||
|
||||
Pattern 1/8: Deleted 1,234 records ✓
|
||||
Pattern 2/8: Deleted 567 records ✓
|
||||
...
|
||||
Total deleted: 7,605 records
|
||||
```
|
||||
|
||||
**OpenSearchKVStorage Example:**
|
||||
```
|
||||
=== Starting Cleanup ===
|
||||
💡 Processing 1,000 records at a time from OpenSearchKVStorage
|
||||
|
||||
Batch 1/8: ████░░░░░░░░░░░░░░░░ 1,000/7,605 (13.1%) ✓
|
||||
Batch 2/8: ████████░░░░░░░░░░░░ 2,000/7,605 (26.3%) ✓
|
||||
...
|
||||
```
|
||||
|
||||
#### 7. Review Cleanup Report
|
||||
|
||||
The tool provides a comprehensive final report:
|
||||
|
||||
**Successful Cleanup:**
|
||||
```
|
||||
============================================================
|
||||
Cleanup Complete - Final Report
|
||||
============================================================
|
||||
|
||||
📊 Statistics:
|
||||
Total records to delete: 7,605
|
||||
Total batches: 8
|
||||
Successful batches: 8
|
||||
Failed batches: 0
|
||||
Successfully deleted: 7,605
|
||||
Failed to delete: 0
|
||||
Success rate: 100.00%
|
||||
|
||||
📈 Before/After Comparison:
|
||||
Total caches before: 7,605
|
||||
Total caches after: 0
|
||||
Net reduction: 7,605
|
||||
|
||||
============================================================
|
||||
✓ SUCCESS: All records cleaned up successfully!
|
||||
============================================================
|
||||
|
||||
📊 Query Cache Statistics (After Cleanup):
|
||||
┌────────────┬────────────┬────────────┬────────────┐
|
||||
│ Mode │ Query │ Keywords │ Total │
|
||||
├────────────┼────────────┼────────────┼────────────┤
|
||||
│ mix │ 0 │ 0 │ 0 │
|
||||
│ hybrid │ 0 │ 0 │ 0 │
|
||||
│ local │ 0 │ 0 │ 0 │
|
||||
│ global │ 0 │ 0 │ 0 │
|
||||
├────────────┼────────────┼────────────┼────────────┤
|
||||
│ Total │ 0 │ 0 │ 0 │
|
||||
└────────────┴────────────┴────────────┴────────────┘
|
||||
```
|
||||
|
||||
**Cleanup with Errors:**
|
||||
```
|
||||
============================================================
|
||||
Cleanup Complete - Final Report
|
||||
============================================================
|
||||
|
||||
📊 Statistics:
|
||||
Total records to delete: 7,605
|
||||
Total batches: 8
|
||||
Successful batches: 7
|
||||
Failed batches: 1
|
||||
Successfully deleted: 6,605
|
||||
Failed to delete: 1,000
|
||||
Success rate: 86.85%
|
||||
|
||||
📈 Before/After Comparison:
|
||||
Total caches before: 7,605
|
||||
Total caches after: 1,000
|
||||
Net reduction: 6,605
|
||||
|
||||
⚠️ Errors encountered: 1
|
||||
|
||||
Error Details:
|
||||
------------------------------------------------------------
|
||||
|
||||
Error Summary:
|
||||
- ConnectionError: 1 occurrence(s)
|
||||
|
||||
First 5 errors:
|
||||
|
||||
1. Batch 3
|
||||
Type: ConnectionError
|
||||
Message: Connection timeout after 30s
|
||||
Records lost: 1,000
|
||||
|
||||
============================================================
|
||||
⚠️ WARNING: Cleanup completed with errors!
|
||||
Please review the error details above.
|
||||
============================================================
|
||||
```
|
||||
|
||||
## Technical Details
|
||||
|
||||
### Workspace Handling
|
||||
|
||||
The tool retrieves workspace in the following priority order:
|
||||
|
||||
1. **Storage-specific workspace environment variables**
|
||||
- PGKVStorage: `POSTGRES_WORKSPACE`
|
||||
- MongoKVStorage: `MONGODB_WORKSPACE`
|
||||
- RedisKVStorage: `REDIS_WORKSPACE`
|
||||
- OpenSearchKVStorage: `OPENSEARCH_WORKSPACE`
|
||||
|
||||
2. **Generic workspace environment variable**
|
||||
- `WORKSPACE`
|
||||
|
||||
3. **Default value**
|
||||
- Empty string (uses storage's default workspace)
|
||||
|
||||
### Batch Deletion
|
||||
|
||||
- Default batch size: 1000 records/batch
|
||||
- Prevents memory overflow and connection timeouts
|
||||
- Each batch is processed independently
|
||||
- Failed batches are logged but don't stop cleanup
|
||||
|
||||
### Storage-Specific Deletion Strategies
|
||||
|
||||
#### JsonKVStorage
|
||||
- Collects all matching keys first (snapshot approach)
|
||||
- Deletes in batches with lock protection
|
||||
- Fast in-memory operations
|
||||
|
||||
#### RedisKVStorage
|
||||
- Uses SCAN with pattern matching
|
||||
- Pipeline DELETE for batch operations
|
||||
- Cursor-based iteration for large datasets
|
||||
|
||||
#### PostgreSQL
|
||||
- Single DELETE query with OR conditions
|
||||
- Efficient server-side bulk deletion
|
||||
- Uses LIKE patterns for mode/type matching
|
||||
|
||||
#### MongoDB
|
||||
- Multiple deleteMany operations (one per pattern)
|
||||
- Regex-based document matching
|
||||
- Returns exact deletion counts
|
||||
|
||||
### Pattern Matching Implementation
|
||||
|
||||
**JsonKVStorage:**
|
||||
```python
|
||||
# Direct key prefix matching
|
||||
if key.startswith("mix:query:") or key.startswith("mix:keywords:")
|
||||
```
|
||||
|
||||
**RedisKVStorage:**
|
||||
```python
|
||||
# SCAN with namespace-prefixed patterns
|
||||
pattern = f"{namespace}:mix:query:*"
|
||||
cursor, keys = await redis.scan(cursor, match=pattern)
|
||||
```
|
||||
|
||||
**PostgreSQL:**
|
||||
```python
|
||||
# SQL LIKE conditions
|
||||
WHERE id LIKE 'mix:query:%' OR id LIKE 'mix:keywords:%'
|
||||
```
|
||||
|
||||
**MongoDB:**
|
||||
```python
|
||||
# Regex queries on _id field
|
||||
{"_id": {"$regex": "^mix:query:"}}
|
||||
```
|
||||
|
||||
**OpenSearchKVStorage:**
|
||||
```python
|
||||
# Scan raw hits, then match cache key prefixes in Python
|
||||
if hit["_id"].startswith("mix:query:"):
|
||||
```
|
||||
|
||||
## Error Handling & Resilience
|
||||
|
||||
The tool implements comprehensive error tracking:
|
||||
|
||||
### Batch-Level Error Tracking
|
||||
- Each batch is independently error-checked
|
||||
- Failed batches are logged with full details
|
||||
- Successful batches commit even if later batches fail
|
||||
- Real-time progress shows ✓ (success) or ✗ (failed)
|
||||
|
||||
### Error Reporting
|
||||
After cleanup completes, a detailed report includes:
|
||||
- **Statistics**: Total records, success/failure counts, success rate
|
||||
- **Before/After Comparison**: Net reduction in cache count
|
||||
- **Error Summary**: Grouped by error type with occurrence counts
|
||||
- **Error Details**: Batch number, error type, message, and records lost
|
||||
- **Recommendations**: Clear indication of success or need for review
|
||||
|
||||
### Verification
|
||||
- Post-cleanup count verification
|
||||
- Before/after statistics comparison
|
||||
- Identifies partial cleanup scenarios
|
||||
|
||||
## Important Notes
|
||||
|
||||
1. **Irreversible Operation**
|
||||
- Deleted caches cannot be recovered
|
||||
- Always backup important data before cleanup
|
||||
- Test on non-production data first
|
||||
|
||||
2. **Performance Impact**
|
||||
- Query performance may degrade temporarily after cleanup
|
||||
- Caches will rebuild on subsequent queries
|
||||
- Consider cleanup during off-peak hours
|
||||
|
||||
3. **Selective Cleanup**
|
||||
- Choose cleanup scope carefully
|
||||
- Keywords caches may be valuable for future queries
|
||||
- Query caches rebuild faster than keywords caches
|
||||
|
||||
4. **Workspace Isolation**
|
||||
- Cleanup only affects the selected workspace
|
||||
- Other workspaces remain untouched
|
||||
- Verify workspace before confirming cleanup
|
||||
|
||||
5. **Interrupt and Resume**
|
||||
- Cleanup can be interrupted at any time (Ctrl+C)
|
||||
- Already deleted records cannot be recovered
|
||||
- No automatic resume - must run tool again
|
||||
|
||||
## Storage Configuration
|
||||
|
||||
The tool supports multiple configuration methods with the following priority:
|
||||
|
||||
1. **Environment variables** (highest priority)
|
||||
2. **Default values** (lowest priority)
|
||||
|
||||
### Environment Variable Configuration
|
||||
|
||||
Configure storage settings in your `.env` file:
|
||||
|
||||
#### Workspace Configuration (Optional)
|
||||
|
||||
```bash
|
||||
# Generic workspace (shared by all storages)
|
||||
WORKSPACE=space1
|
||||
|
||||
# Or configure independent workspace for specific storage
|
||||
POSTGRES_WORKSPACE=pg_space
|
||||
MONGODB_WORKSPACE=mongo_space
|
||||
REDIS_WORKSPACE=redis_space
|
||||
```
|
||||
|
||||
**Workspace Priority**: Storage-specific > Generic WORKSPACE > Empty string
|
||||
|
||||
#### JsonKVStorage
|
||||
|
||||
```bash
|
||||
WORKING_DIR=./rag_storage
|
||||
```
|
||||
|
||||
#### RedisKVStorage
|
||||
|
||||
```bash
|
||||
REDIS_URI=redis://localhost:6379
|
||||
```
|
||||
|
||||
#### PGKVStorage
|
||||
|
||||
```bash
|
||||
POSTGRES_HOST=localhost
|
||||
POSTGRES_PORT=5432
|
||||
POSTGRES_USER=your_username
|
||||
POSTGRES_PASSWORD=your_password
|
||||
POSTGRES_DATABASE=your_database
|
||||
```
|
||||
|
||||
#### MongoKVStorage
|
||||
|
||||
```bash
|
||||
MONGO_URI=mongodb://root:root@localhost:27017/
|
||||
MONGO_DATABASE=LightRAG
|
||||
```
|
||||
|
||||
#### OpenSearchKVStorage
|
||||
|
||||
```bash
|
||||
OPENSEARCH_HOSTS=localhost:9200
|
||||
OPENSEARCH_WORKSPACE=search_space
|
||||
```
|
||||
|
||||
If environment variables are not provided, the tool falls back to built-in defaults where available.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Missing Environment Variables
|
||||
```
|
||||
⚠️ Warning: Missing environment variables: POSTGRES_USER, POSTGRES_PASSWORD
|
||||
```
|
||||
**Solution**: Add missing variables to your `.env` file
|
||||
|
||||
### Connection Failed
|
||||
```
|
||||
✗ Initialization failed: Connection refused
|
||||
```
|
||||
**Solutions**:
|
||||
- Check if database service is running
|
||||
- Verify connection parameters (host, port, credentials)
|
||||
- Check firewall settings
|
||||
- Ensure network connectivity for remote databases
|
||||
|
||||
### No Caches Found
|
||||
```
|
||||
⚠️ No query caches found in storage
|
||||
```
|
||||
**Possible Reasons**:
|
||||
- No queries have been run yet
|
||||
- Caches were already cleaned
|
||||
- Wrong workspace selected
|
||||
- Different storage type was used for queries
|
||||
|
||||
### Partial Cleanup
|
||||
```
|
||||
⚠️ WARNING: Cleanup completed with errors!
|
||||
```
|
||||
**Solutions**:
|
||||
- Check error details in the report
|
||||
- Verify storage connection stability
|
||||
- Re-run tool to clean remaining caches
|
||||
- Check storage capacity and permissions
|
||||
|
||||
## Use Cases
|
||||
|
||||
### Use Case 1: Clean All Query Caches
|
||||
|
||||
**Scenario**: Free up storage space by removing all query caches
|
||||
|
||||
```bash
|
||||
# Run tool
|
||||
python -m lightrag.tools.clean_llm_query_cache
|
||||
|
||||
# Select: Storage type -> Option 1 (all) -> Confirm (y)
|
||||
```
|
||||
|
||||
**Result**: All query and keywords caches deleted, maximum storage freed
|
||||
|
||||
### Use Case 2: Refresh Query Caches Only
|
||||
|
||||
**Scenario**: Force query cache rebuild while keeping keywords
|
||||
|
||||
```bash
|
||||
# Run tool
|
||||
python -m lightrag.tools.clean_llm_query_cache
|
||||
|
||||
# Select: Storage type -> Option 2 (query only) -> Confirm (y)
|
||||
```
|
||||
|
||||
**Result**: Query caches deleted, keywords preserved for faster rebuild
|
||||
|
||||
### Use Case 3: Clean Stale Keywords
|
||||
|
||||
**Scenario**: Remove outdated keywords while keeping recent query results
|
||||
|
||||
```bash
|
||||
# Run tool
|
||||
python -m lightrag.tools.clean_llm_query_cache
|
||||
|
||||
# Select: Storage type -> Option 3 (keywords only) -> Confirm (y)
|
||||
```
|
||||
|
||||
**Result**: Keywords deleted, query caches preserved
|
||||
|
||||
### Use Case 4: Workspace-Specific Cleanup
|
||||
|
||||
**Scenario**: Clean caches for a specific workspace
|
||||
|
||||
```bash
|
||||
# Configure workspace
|
||||
export WORKSPACE=development
|
||||
|
||||
# Run tool
|
||||
python -m lightrag.tools.clean_llm_query_cache
|
||||
|
||||
# Select: Storage type -> Cleanup option -> Confirm (y)
|
||||
```
|
||||
|
||||
**Result**: Only development workspace caches cleaned
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Backup Before Cleanup**
|
||||
- Always backup your storage before major cleanup
|
||||
- Test cleanup on non-production data first
|
||||
- Document cleanup decisions
|
||||
|
||||
2. **Monitor Performance**
|
||||
- Watch storage metrics during cleanup
|
||||
- Monitor query performance after cleanup
|
||||
- Allow time for cache rebuild
|
||||
|
||||
3. **Scheduled Cleanup**
|
||||
- Clean caches periodically (weekly/monthly)
|
||||
- Automate cleanup for development environments
|
||||
- Keep production cleanup manual for safety
|
||||
|
||||
4. **Selective Deletion**
|
||||
- Consider cleanup scope based on needs
|
||||
- Keywords caches are harder to rebuild
|
||||
- Query caches rebuild automatically
|
||||
|
||||
5. **Storage Capacity**
|
||||
- Monitor storage usage trends
|
||||
- Clean caches before reaching capacity limits
|
||||
- Archive old data if needed
|
||||
|
||||
## Comparison with Migration Tool
|
||||
|
||||
| Feature | Cleanup Tool | Migration Tool |
|
||||
|---------|-------------|----------------|
|
||||
| **Purpose** | Delete query caches | Migrate extraction caches |
|
||||
| **Cache Types** | mix/hybrid/local/global | default:extract/summary |
|
||||
| **Modes** | query, keywords | extract, summary |
|
||||
| **Operation** | Deletion | Copy between storages |
|
||||
| **Reversible** | No | Yes (source unchanged) |
|
||||
| **Use Case** | Free storage, refresh caches | Change storage backend |
|
||||
|
||||
## Limitations
|
||||
|
||||
1. **Single Storage Operation**
|
||||
- Can only clean one storage type at a time
|
||||
- To clean multiple storages, run tool multiple times
|
||||
|
||||
2. **No Dry Run Mode**
|
||||
- Deletion is immediate after confirmation
|
||||
- No preview-only mode available
|
||||
- Test on non-production first
|
||||
|
||||
3. **No Selective Mode Cleanup**
|
||||
- Cannot clean only specific modes (e.g., only `mix`)
|
||||
- Cleanup applies to all modes for selected cache type
|
||||
- All-or-nothing per cache type
|
||||
|
||||
4. **No Scheduled Cleanup**
|
||||
- Manual execution required
|
||||
- No built-in scheduling
|
||||
- Use cron/scheduler if automation needed
|
||||
|
||||
5. **Verification Limitations**
|
||||
- Post-cleanup verification may fail in error scenarios
|
||||
- Manual verification recommended for critical operations
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
Potential improvements for future versions:
|
||||
|
||||
- Selective mode cleanup (e.g., clean only `mix` mode)
|
||||
- Age-based cleanup (delete caches older than X days)
|
||||
- Size-based cleanup (delete largest caches first)
|
||||
- Dry run mode for safe preview
|
||||
- Automated scheduling support
|
||||
- Cache statistics export
|
||||
- Incremental cleanup with pause/resume
|
||||
|
||||
## Support
|
||||
|
||||
For issues, questions, or feature requests:
|
||||
- Check the error details in the cleanup report
|
||||
- Review storage configuration
|
||||
- Verify workspace settings
|
||||
- Test with a small dataset first
|
||||
- Report bugs through project issue tracker
|
||||
@@ -0,0 +1,468 @@
|
||||
# LLM Cache Migration Tool - User Guide
|
||||
|
||||
## Overview
|
||||
|
||||
This tool migrates LightRAG's LLM response cache between different KV storage implementations. It specifically migrates caches generated during file extraction (mode `default`), including entity extraction and summary caches.
|
||||
|
||||
## Supported Storage Types
|
||||
|
||||
1. **JsonKVStorage** - File-based JSON storage
|
||||
2. **RedisKVStorage** - Redis database storage
|
||||
3. **PGKVStorage** - PostgreSQL database storage
|
||||
4. **MongoKVStorage** - MongoDB database storage
|
||||
5. **OpenSearchKVStorage** - OpenSearch index storage
|
||||
|
||||
## Cache Types
|
||||
|
||||
The tool migrates the following cache types:
|
||||
- `default:extract:*` - Entity and relationship extraction caches
|
||||
- `default:summary:*` - Entity and relationship summary caches
|
||||
|
||||
**Note**: Query caches (modes like `mix`,`local`, `global`, etc.) are NOT migrated.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
The LLM Cache Migration Tool reads the storage configuration of the LightRAG Server and provides an LLM migration option to select source and destination storage. Ensure that both the source and destination storage have been correctly configured and are accessible via the LightRAG Server before cache migration.
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run from the LightRAG project root directory:
|
||||
|
||||
```bash
|
||||
python -m lightrag.tools.migrate_llm_cache
|
||||
# or
|
||||
python lightrag/tools/migrate_llm_cache.py
|
||||
```
|
||||
|
||||
### Interactive Workflow
|
||||
|
||||
The tool guides you through the following steps:
|
||||
|
||||
#### 1. Select Source Storage Type
|
||||
```
|
||||
Supported KV Storage Types:
|
||||
[1] JsonKVStorage
|
||||
[2] RedisKVStorage
|
||||
[3] PGKVStorage
|
||||
[4] MongoKVStorage
|
||||
[5] OpenSearchKVStorage
|
||||
|
||||
Select Source storage type (1-5) (Press Enter to exit): 1
|
||||
```
|
||||
|
||||
**Note**: You can press Enter or type `0` at any storage selection prompt to exit gracefully.
|
||||
|
||||
#### 2. Source Storage Validation
|
||||
The tool will:
|
||||
- Check required environment variables
|
||||
- Auto-detect workspace configuration
|
||||
- Initialize and connect to storage
|
||||
- Count cache records available for migration
|
||||
|
||||
```
|
||||
Checking environment variables...
|
||||
✓ All required environment variables are set
|
||||
|
||||
Initializing Source storage...
|
||||
- Storage Type: JsonKVStorage
|
||||
- Workspace: space1
|
||||
- Connection Status: ✓ Success
|
||||
|
||||
Counting cache records...
|
||||
- Total: 8,734 records
|
||||
```
|
||||
|
||||
**Progress Display by Storage Type:**
|
||||
- **JsonKVStorage**: Fast in-memory counting, displays final count without incremental progress
|
||||
```
|
||||
Counting cache records...
|
||||
- Total: 8,734 records
|
||||
```
|
||||
- **RedisKVStorage**: Real-time scanning progress with incremental counts
|
||||
```
|
||||
Scanning Redis keys... found 8,734 records
|
||||
```
|
||||
- **PostgreSQL**: Quick COUNT(*) query, shows timing only if operation takes >1 second
|
||||
```
|
||||
Counting PostgreSQL records... (took 2.3s)
|
||||
```
|
||||
- **MongoDB**: Fast count_documents(), shows timing only if operation takes >1 second
|
||||
```
|
||||
Counting MongoDB documents... (took 1.8s)
|
||||
```
|
||||
- **OpenSearchKVStorage**: PIT-based scan with timing shown when noticeable
|
||||
```
|
||||
Scanning OpenSearch documents... (took 1.5s)
|
||||
```
|
||||
|
||||
#### 3. Select Target Storage Type
|
||||
|
||||
The tool automatically excludes the source storage type from the target selection and renumbers the remaining options sequentially:
|
||||
|
||||
```
|
||||
Available Storage Types for Target (source: JsonKVStorage excluded):
|
||||
[1] RedisKVStorage
|
||||
[2] PGKVStorage
|
||||
[3] MongoKVStorage
|
||||
[4] OpenSearchKVStorage
|
||||
|
||||
Select Target storage type (1-4) (Press Enter or 0 to exit): 1
|
||||
```
|
||||
|
||||
**Important Notes:**
|
||||
- You **cannot** select the same storage type for both source and target
|
||||
- Options are automatically renumbered (e.g., [1], [2], [3] instead of [2], [3], [4])
|
||||
- You can press Enter or type `0` to exit at this point as well
|
||||
|
||||
The tool then validates the target storage following the same process as the source (checking environment variables, initializing connection, counting records).
|
||||
|
||||
#### 4. Confirm Migration
|
||||
|
||||
```
|
||||
==================================================
|
||||
Migration Confirmation
|
||||
Source: JsonKVStorage (workspace: space1) - 8,734 records
|
||||
Target: MongoKVStorage (workspace: space1) - 0 records
|
||||
Batch Size: 1,000 records/batch
|
||||
Memory Mode: Streaming (memory-optimized)
|
||||
|
||||
⚠️ Warning: Target storage already has 0 records
|
||||
Migration will overwrite records with the same keys
|
||||
|
||||
Continue? (y/n): y
|
||||
```
|
||||
|
||||
#### 5. Execute Migration
|
||||
|
||||
The tool uses **streaming migration** by default for memory efficiency. Observe migration progress:
|
||||
|
||||
```
|
||||
=== Starting Streaming Migration ===
|
||||
💡 Memory-optimized mode: Processing 1,000 records at a time
|
||||
|
||||
Batch 1/9: ████████░░░░░░░░░░░░ 1000/8734 (11.4%) - default:extract ✓
|
||||
Batch 2/9: ████████████░░░░░░░░ 2000/8734 (22.9%) - default:extract ✓
|
||||
...
|
||||
Batch 9/9: ████████████████████ 8734/8734 (100.0%) - default:summary ✓
|
||||
|
||||
Persisting data to disk...
|
||||
✓ Data persisted successfully
|
||||
```
|
||||
|
||||
**Key Features:**
|
||||
- **Streaming mode**: Processes data in batches without loading entire dataset into memory
|
||||
- **Real-time progress**: Shows progress bar with precise percentage and cache type
|
||||
- **Success indicators**: ✓ for successful batches, ✗ for failed batches
|
||||
- **Constant memory usage**: Handles millions of records efficiently
|
||||
|
||||
#### 6. Review Migration Report
|
||||
|
||||
The tool provides a comprehensive final report showing statistics and any errors encountered:
|
||||
|
||||
**Successful Migration:**
|
||||
```
|
||||
Migration Complete - Final Report
|
||||
|
||||
📊 Statistics:
|
||||
Total source records: 8,734
|
||||
Total batches: 9
|
||||
Successful batches: 9
|
||||
Failed batches: 0
|
||||
Successfully migrated: 8,734
|
||||
Failed to migrate: 0
|
||||
Success rate: 100.00%
|
||||
|
||||
✓ SUCCESS: All records migrated successfully!
|
||||
```
|
||||
|
||||
**Migration with Errors:**
|
||||
```
|
||||
Migration Complete - Final Report
|
||||
|
||||
📊 Statistics:
|
||||
Total source records: 8,734
|
||||
Total batches: 9
|
||||
Successful batches: 8
|
||||
Failed batches: 1
|
||||
Successfully migrated: 7,734
|
||||
Failed to migrate: 1,000
|
||||
Success rate: 88.55%
|
||||
|
||||
⚠️ Errors encountered: 1
|
||||
|
||||
Error Details:
|
||||
------------------------------------------------------------
|
||||
|
||||
Error Summary:
|
||||
- ConnectionError: 1 occurrence(s)
|
||||
|
||||
First 5 errors:
|
||||
|
||||
1. Batch 2
|
||||
Type: ConnectionError
|
||||
Message: Connection timeout after 30s
|
||||
Records lost: 1,000
|
||||
|
||||
⚠️ WARNING: Migration completed with errors!
|
||||
Please review the error details above.
|
||||
```
|
||||
|
||||
## Technical Details
|
||||
|
||||
### Workspace Handling
|
||||
|
||||
The tool retrieves workspace in the following priority order:
|
||||
|
||||
1. **Storage-specific workspace environment variables**
|
||||
- PGKVStorage: `POSTGRES_WORKSPACE`
|
||||
- MongoKVStorage: `MONGODB_WORKSPACE`
|
||||
- RedisKVStorage: `REDIS_WORKSPACE`
|
||||
- OpenSearchKVStorage: `OPENSEARCH_WORKSPACE`
|
||||
|
||||
2. **Generic workspace environment variable**
|
||||
- `WORKSPACE`
|
||||
|
||||
3. **Default value**
|
||||
- Empty string (uses storage's default workspace)
|
||||
|
||||
### Batch Migration
|
||||
|
||||
- Default batch size: 1000 records/batch
|
||||
- Avoids memory overflow from loading too much data at once
|
||||
- Each batch is committed independently, supporting resume capability
|
||||
|
||||
### Memory-Efficient Pagination
|
||||
|
||||
For large datasets, the tool implements storage-specific pagination strategies:
|
||||
|
||||
- **JsonKVStorage**: Direct in-memory access (data already loaded in shared storage)
|
||||
- **RedisKVStorage**: Cursor-based SCAN with pipeline batching (1000 keys/batch)
|
||||
- **PGKVStorage**: SQL LIMIT/OFFSET pagination (1000 records/batch)
|
||||
- **MongoKVStorage**: Cursor streaming with batch_size (1000 documents/batch)
|
||||
- **OpenSearchKVStorage**: PIT + `search_after` scan of the KV index (1000 documents/batch)
|
||||
|
||||
This ensures the tool can handle millions of cache records without memory issues.
|
||||
|
||||
### Prefix Filtering Implementation
|
||||
|
||||
The tool uses optimized filtering methods for different storage types:
|
||||
|
||||
- **JsonKVStorage**: Direct dictionary iteration with lock protection
|
||||
- **RedisKVStorage**: SCAN command with namespace-prefixed patterns + pipeline for bulk GET
|
||||
- **PGKVStorage**: SQL LIKE queries with proper field mapping (id, return_value, etc.)
|
||||
- **MongoKVStorage**: MongoDB regex queries on `_id` field with cursor streaming
|
||||
- **OpenSearchKVStorage**: Full-index scan with `_id` prefix filtering and `_source` passthrough
|
||||
|
||||
## Error Handling & Resilience
|
||||
|
||||
The tool implements comprehensive error tracking to ensure transparent and resilient migrations:
|
||||
|
||||
### Batch-Level Error Tracking
|
||||
- Each batch is independently error-checked
|
||||
- Failed batches are logged but don't stop the migration
|
||||
- Successful batches are committed even if later batches fail
|
||||
- Real-time progress shows ✓ (success) or ✗ (failed) for each batch
|
||||
|
||||
### Error Reporting
|
||||
After migration completes, a detailed report includes:
|
||||
- **Statistics**: Total records, success/failure counts, success rate
|
||||
- **Error Summary**: Grouped by error type with occurrence counts
|
||||
- **Error Details**: Batch number, error type, message, and records lost
|
||||
- **Recommendations**: Clear indication of success or need for review
|
||||
|
||||
### No Double Data Loading
|
||||
- Unlike traditional verification approaches, the tool does NOT reload all target data
|
||||
- Errors are detected during migration, not after
|
||||
- This eliminates memory overhead and handles pre-existing target data correctly
|
||||
|
||||
## Important Notes
|
||||
|
||||
1. **Data Overwrite Warning**
|
||||
- Migration will overwrite records with the same keys in the target storage
|
||||
- Tool displays a warning if target storage already has data
|
||||
- Data migration can be performed repeatedly
|
||||
- Pre-existing data in target storage is handled correctly
|
||||
3. **Interrupt and Resume**
|
||||
- Migration can be interrupted at any time (Ctrl+C)
|
||||
- Already migrated data will remain in target storage
|
||||
- Re-running will overwrite existing records
|
||||
- Failed batches can be manually retried
|
||||
4. **Performance Considerations**
|
||||
- Large data migration may take considerable time
|
||||
- Recommend migrating during off-peak hours
|
||||
- Ensure stable network connection (for remote databases)
|
||||
- Memory usage stays constant regardless of dataset size
|
||||
|
||||
## Storage Configuration
|
||||
|
||||
The tool supports multiple configuration methods with the following priority:
|
||||
|
||||
1. **Environment variables** (highest priority)
|
||||
2. **Default values** (lowest priority)
|
||||
|
||||
#### Option A: Environment Variable Configuration
|
||||
|
||||
Configure storage settings in your `.env` file:
|
||||
|
||||
#### Workspace Configuration (Optional)
|
||||
|
||||
```bash
|
||||
# Generic workspace (shared by all storages)
|
||||
WORKSPACE=space1
|
||||
|
||||
# Or configure independent workspace for specific storage
|
||||
POSTGRES_WORKSPACE=pg_space
|
||||
MONGODB_WORKSPACE=mongo_space
|
||||
REDIS_WORKSPACE=redis_space
|
||||
OPENSEARCH_WORKSPACE=os_space
|
||||
```
|
||||
|
||||
**Workspace Priority**: Storage-specific > Generic WORKSPACE > Empty string
|
||||
|
||||
#### JsonKVStorage
|
||||
|
||||
```bash
|
||||
WORKING_DIR=./rag_storage
|
||||
```
|
||||
|
||||
#### RedisKVStorage
|
||||
|
||||
```bash
|
||||
REDIS_URI=redis://localhost:6379
|
||||
```
|
||||
|
||||
#### PGKVStorage
|
||||
|
||||
```bash
|
||||
POSTGRES_HOST=localhost
|
||||
POSTGRES_PORT=5432
|
||||
POSTGRES_USER=your_username
|
||||
POSTGRES_PASSWORD=your_password
|
||||
POSTGRES_DATABASE=your_database
|
||||
```
|
||||
|
||||
#### MongoKVStorage
|
||||
|
||||
```bash
|
||||
MONGO_URI=mongodb://root:root@localhost:27017/
|
||||
MONGO_DATABASE=LightRAG
|
||||
```
|
||||
|
||||
#### OpenSearchKVStorage
|
||||
|
||||
```bash
|
||||
OPENSEARCH_HOSTS=localhost:9200
|
||||
OPENSEARCH_WORKSPACE=os_space
|
||||
```
|
||||
|
||||
If environment variables are not provided, the tool falls back to built-in defaults where available. JsonKVStorage uses `WORKING_DIR` or defaults to `./rag_storage`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Missing Environment Variables
|
||||
```
|
||||
✗ Missing required environment variables: POSTGRES_USER, POSTGRES_PASSWORD
|
||||
```
|
||||
**Solution**: Add missing variables to your `.env` file
|
||||
|
||||
### Connection Failed
|
||||
```
|
||||
✗ Initialization failed: Connection refused
|
||||
```
|
||||
**Solutions**:
|
||||
- Check if database service is running
|
||||
- Verify connection parameters (host, port, credentials)
|
||||
- Check firewall settings
|
||||
|
||||
**Solutions**:
|
||||
- Check migration process for error logs
|
||||
- Re-run migration tool
|
||||
- Check target storage capacity and permissions
|
||||
|
||||
## Example Scenarios
|
||||
|
||||
### Scenario 1: JSON to MongoDB Migration
|
||||
|
||||
Use case: Migrating from single-machine development to production
|
||||
|
||||
```bash
|
||||
# 1. Configure environment variables
|
||||
WORKSPACE=production
|
||||
MONGO_URI=mongodb://user:pass@prod-server:27017/
|
||||
MONGO_DATABASE=LightRAG
|
||||
|
||||
# 2. Run tool
|
||||
python -m lightrag.tools.migrate_llm_cache
|
||||
|
||||
# 3. Select: 1 (JsonKVStorage) -> 1 (MongoKVStorage - renumbered from 4)
|
||||
```
|
||||
|
||||
**Note**: After selecting JsonKVStorage as source, MongoKVStorage will be shown as option [1] in the target selection since options are renumbered after excluding the source.
|
||||
|
||||
### Scenario 2: Redis to PostgreSQL
|
||||
|
||||
Use case: Migrating from cache storage to relational database
|
||||
|
||||
```bash
|
||||
# 1. Ensure both databases are accessible
|
||||
REDIS_URI=redis://old-redis:6379
|
||||
POSTGRES_HOST=new-postgres-server
|
||||
# ... Other PostgreSQL configs
|
||||
|
||||
# 2. Run tool
|
||||
python -m lightrag.tools.migrate_llm_cache
|
||||
|
||||
# 3. Select: 2 (RedisKVStorage) -> 2 (PGKVStorage - renumbered from 3)
|
||||
```
|
||||
|
||||
**Note**: After selecting RedisKVStorage as source, PGKVStorage will be shown as option [2] in the target selection.
|
||||
|
||||
### Scenario 3: Different Workspaces Migration
|
||||
|
||||
Use case: Migrating data between different workspace environments
|
||||
|
||||
```bash
|
||||
# Configure separate workspaces for source and target
|
||||
POSTGRES_WORKSPACE=dev_workspace # For development environment
|
||||
MONGODB_WORKSPACE=prod_workspace # For production environment
|
||||
|
||||
# Run tool
|
||||
python -m lightrag.tools.migrate_llm_cache
|
||||
|
||||
# Select: 3 (PGKVStorage with dev_workspace) -> 3 (MongoKVStorage with prod_workspace)
|
||||
```
|
||||
|
||||
**Note**: This allows you to migrate between different logical data partitions while changing storage backends.
|
||||
|
||||
## Tool Limitations
|
||||
|
||||
1. **Same Storage Type Not Allowed**
|
||||
- You cannot migrate between the same storage type (e.g., PostgreSQL to PostgreSQL)
|
||||
- This is enforced by the tool automatically excluding the source storage type from target selection
|
||||
- For same-storage migrations (e.g., database switches), use database-native tools instead
|
||||
2. **Only Default Mode Caches**
|
||||
- Only migrates `default:extract:*` and `default:summary:*`
|
||||
- Query caches are not included
|
||||
4. **Network Dependency**
|
||||
- Tool requires stable network connection for remote databases
|
||||
- Large datasets may fail if connection is interrupted
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Backup Before Migration**
|
||||
- Always backup your data before migration
|
||||
- Test migration on non-production data first
|
||||
|
||||
2. **Verify Results**
|
||||
- Check the verification output after migration
|
||||
- Manually verify a few cache entries if needed
|
||||
|
||||
3. **Monitor Performance**
|
||||
- Watch database resource usage during migration
|
||||
- Consider migrating in smaller batches if needed
|
||||
|
||||
4. **Clean Old Data**
|
||||
- After successful migration, consider cleaning old cache data
|
||||
- Keep backups for a reasonable period before deletion
|
||||
@@ -0,0 +1,121 @@
|
||||
# Offline Vector Storage (VDB) Rebuild Tool
|
||||
|
||||
`lightrag-rebuild-vdb` restores consistency between LightRAG's authoritative
|
||||
data sources and its vector storages by dropping each vector storage and
|
||||
rebuilding it from scratch:
|
||||
|
||||
| Vector storage | Authoritative source |
|
||||
|---|---|
|
||||
| `entities_vdb` | graph nodes |
|
||||
| `relationships_vdb` | graph edges |
|
||||
| `chunks_vdb` | `text_chunks` KV store |
|
||||
|
||||
## When do I need this?
|
||||
|
||||
LightRAG performs multi-step writes (graph + vector storage). If a vector
|
||||
storage write fails at runtime — embedder outage, network timeout, context
|
||||
overflow on a high-degree entity — the graph and the vector storage drift
|
||||
apart. The most common symptom is the edge-count drift reported in issue
|
||||
[#2917](https://github.com/HKUDS/LightRAG/issues/2917): graph edges with no
|
||||
vector counterpart, so `local`/`hybrid` queries miss relations that exist in
|
||||
the graph.
|
||||
|
||||
Since v(next), `amerge_entities` raises `VectorStorageConsistencyError` when
|
||||
this happens, with a pointer to this tool. No data is lost in this situation:
|
||||
the graph and the `text_chunks` KV store hold everything needed to rebuild
|
||||
the vectors.
|
||||
|
||||
A full drop + rebuild also clears *reverse* orphans (records present in the
|
||||
vector storage but absent from the graph), which incremental repair cannot
|
||||
reliably do.
|
||||
|
||||
You can also use this tool after changing the embedding model or embedding
|
||||
dimension. Existing vector records were generated in the old embedding space
|
||||
(and some vector backends bind collections/tables to the configured dimension),
|
||||
so run the tool with the updated `.env` and choose **Rebuild ALL vector
|
||||
storages** to regenerate `entities_vdb`, `relationships_vdb`, and `chunks_vdb`
|
||||
from the authoritative graph/KV sources. The consistency check is not enough
|
||||
for this case because it only detects missing graph → VDB records, not vectors
|
||||
that exist but were embedded with a previous model or dimension.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Stop the LightRAG Server first!
|
||||
lightrag-rebuild-vdb
|
||||
# or
|
||||
python -m lightrag.tools.rebuild_vdb
|
||||
```
|
||||
|
||||
The tool reads the same `.env` / environment configuration as the server
|
||||
(`LIGHTRAG_GRAPH_STORAGE`, `LIGHTRAG_VECTOR_STORAGE`, `LIGHTRAG_KV_STORAGE`,
|
||||
`WORKSPACE`, `WORKING_DIR`, `EMBEDDING_*`, backend connection settings) and
|
||||
builds its embedding function through the exact factory the server uses —
|
||||
run it with the same `.env` so rebuilt vectors live in the same embedding
|
||||
space the server queries against.
|
||||
|
||||
Menu options:
|
||||
|
||||
1. **Consistency check (diagnose only)** — probes every graph entity/relation
|
||||
for a vector counterpart and reports what is missing. Run this first to
|
||||
decide whether a rebuild is worth the embedding cost. The check covers
|
||||
the graph → VDB direction only; reverse orphans can only be cleared by a
|
||||
full rebuild. Legacy reverse-order relation ids (from old custom-KG
|
||||
imports) are recognized and not misreported as missing. The check issues
|
||||
read queries only and does not run a rebuild (no drop + re-embed). It is
|
||||
**not** strictly side-effect-free, though: the tool initializes every
|
||||
storage on startup — exactly as the server does — and for some backends
|
||||
that includes schema/DDL setup and one-time legacy migrations (e.g. Qdrant
|
||||
upserts into the new collection, PostgreSQL batch-inserts into the new
|
||||
table, Milvus may create a temp collection and drop/rename the original).
|
||||
Treat running the tool — even just for a check — like starting the server:
|
||||
stop other writers first.
|
||||
2. **Rebuild entities + relationships VDB** — sufficient for the #2917
|
||||
merge-failure scenario.
|
||||
3. **Rebuild chunks VDB**.
|
||||
4. **Rebuild ALL vector storages** — use this after changing the embedding
|
||||
model or embedding dimension.
|
||||
|
||||
## Important notes
|
||||
|
||||
- **Stop the server first.** The tool drops and rewrites vector storages;
|
||||
concurrent writers (any backend, not just file-based ones) can corrupt
|
||||
data or lose updates.
|
||||
- **Embedding model/dimension changes.** Run the tool with the new embedding
|
||||
configuration and rebuild all vector storages. A consistency check can still
|
||||
pass when every vector record exists but was created with the old embedding
|
||||
model or dimension.
|
||||
- **Embedding cost.** A rebuild re-embeds every affected record. On large
|
||||
datasets this means real API cost and time. Use the check mode first, and
|
||||
rebuild only the storages that need it.
|
||||
- **Idempotent / crash-safe.** Sources (graph, `text_chunks`) are never
|
||||
modified. If the tool crashes between drop and rewrite, just re-run it.
|
||||
- **`__created_at__` reset.** Backends that store creation timestamps in
|
||||
vector records (nano, faiss) will show fresh timestamps after a rebuild.
|
||||
No query logic depends on them.
|
||||
- **Custom-KG placeholder entities.** `UNKNOWN` placeholder nodes created by
|
||||
`ainsert_custom_kg` are rebuilt faithfully from the graph; they may gain a
|
||||
vector record they previously lacked (improving their retrievability).
|
||||
- **Chunk enumeration is backend-specific.** `BaseKVStorage` has no key
|
||||
enumeration API, so the tool scans each KV backend directly (JsonKV,
|
||||
Redis, PostgreSQL, MongoDB, OpenSearch). When a new KV backend is added,
|
||||
`enumerate_kv_keys()` in `rebuild_vdb.py` must be extended.
|
||||
|
||||
## Library usage
|
||||
|
||||
The core rebuild/check functions are plain async functions that accept your
|
||||
own initialized storage instances:
|
||||
|
||||
```python
|
||||
from lightrag.tools.rebuild_vdb import (
|
||||
check_vdb_consistency,
|
||||
rebuild_chunks_vdb,
|
||||
rebuild_entities_vdb,
|
||||
rebuild_relationships_vdb,
|
||||
)
|
||||
|
||||
report = await check_vdb_consistency(graph, entities_vdb, relationships_vdb)
|
||||
if not report["consistent"]:
|
||||
await rebuild_entities_vdb(graph, entities_vdb, global_config)
|
||||
await rebuild_relationships_vdb(graph, relationships_vdb, global_config)
|
||||
```
|
||||
@@ -0,0 +1,179 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Diagnostic tool to check LightRAG initialization status.
|
||||
|
||||
This tool helps developers verify that their LightRAG instance is properly
|
||||
initialized and ready to use. It should be called AFTER initialize_storages()
|
||||
to validate that all components are correctly set up.
|
||||
|
||||
Usage:
|
||||
# Basic usage in your code:
|
||||
rag = LightRAG(...)
|
||||
await rag.initialize_storages()
|
||||
await check_lightrag_setup(rag, verbose=True)
|
||||
|
||||
# Run demo from command line:
|
||||
python -m lightrag.tools.check_initialization --demo
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Add parent directory to path for imports
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
|
||||
|
||||
from lightrag import LightRAG
|
||||
from lightrag.base import StoragesStatus
|
||||
|
||||
|
||||
async def check_lightrag_setup(rag_instance: LightRAG, verbose: bool = False) -> bool:
|
||||
"""
|
||||
Check if a LightRAG instance is properly initialized.
|
||||
|
||||
Args:
|
||||
rag_instance: The LightRAG instance to check
|
||||
verbose: If True, print detailed diagnostic information
|
||||
|
||||
Returns:
|
||||
True if properly initialized, False otherwise
|
||||
"""
|
||||
issues = []
|
||||
warnings = []
|
||||
|
||||
print("🔍 Checking LightRAG initialization status...\n")
|
||||
|
||||
# Check storage initialization status
|
||||
if not hasattr(rag_instance, "_storages_status"):
|
||||
issues.append("LightRAG instance missing _storages_status attribute")
|
||||
elif rag_instance._storages_status != StoragesStatus.INITIALIZED:
|
||||
issues.append(
|
||||
f"Storages not initialized (status: {rag_instance._storages_status.name})"
|
||||
)
|
||||
else:
|
||||
print("✅ Storage status: INITIALIZED")
|
||||
|
||||
# Check individual storage components
|
||||
storage_components = [
|
||||
("full_docs", "Document storage"),
|
||||
("text_chunks", "Text chunks storage"),
|
||||
("entities_vdb", "Entity vector database"),
|
||||
("relationships_vdb", "Relationship vector database"),
|
||||
("chunks_vdb", "Chunks vector database"),
|
||||
("doc_status", "Document status tracker"),
|
||||
("llm_response_cache", "LLM response cache"),
|
||||
("full_entities", "Entity storage"),
|
||||
("full_relations", "Relation storage"),
|
||||
("chunk_entity_relation_graph", "Graph storage"),
|
||||
]
|
||||
|
||||
if verbose:
|
||||
print("\n📦 Storage Components:")
|
||||
|
||||
for component, description in storage_components:
|
||||
if not hasattr(rag_instance, component):
|
||||
issues.append(f"Missing storage component: {component} ({description})")
|
||||
else:
|
||||
storage = getattr(rag_instance, component)
|
||||
if storage is None:
|
||||
warnings.append(f"Storage {component} is None (might be optional)")
|
||||
elif hasattr(storage, "_storage_lock"):
|
||||
if storage._storage_lock is None:
|
||||
issues.append(f"Storage {component} not initialized (lock is None)")
|
||||
elif verbose:
|
||||
print(f" ✅ {description}: Ready")
|
||||
elif verbose:
|
||||
print(f" ✅ {description}: Ready")
|
||||
|
||||
# Check pipeline status
|
||||
try:
|
||||
from lightrag.kg.shared_storage import get_namespace_data
|
||||
|
||||
get_namespace_data("pipeline_status", workspace=rag_instance.workspace)
|
||||
print("✅ Pipeline status: INITIALIZED")
|
||||
except KeyError:
|
||||
issues.append(
|
||||
"Pipeline status not initialized - call rag.initialize_storages() first"
|
||||
)
|
||||
except Exception as e:
|
||||
issues.append(f"Error checking pipeline status: {str(e)}")
|
||||
|
||||
# Print results
|
||||
print("\n" + "=" * 50)
|
||||
|
||||
if issues:
|
||||
print("❌ Issues found:\n")
|
||||
for issue in issues:
|
||||
print(f" • {issue}")
|
||||
|
||||
print("\n📝 To fix, run this initialization sequence:\n")
|
||||
print(" await rag.initialize_storages()")
|
||||
print(
|
||||
"\n📚 Documentation: https://github.com/HKUDS/LightRAG#important-initialization-requirements"
|
||||
)
|
||||
|
||||
if warnings and verbose:
|
||||
print("\n⚠️ Warnings (might be normal):")
|
||||
for warning in warnings:
|
||||
print(f" • {warning}")
|
||||
|
||||
return False
|
||||
else:
|
||||
print("✅ LightRAG is properly initialized and ready to use!")
|
||||
|
||||
if warnings and verbose:
|
||||
print("\n⚠️ Warnings (might be normal):")
|
||||
for warning in warnings:
|
||||
print(f" • {warning}")
|
||||
|
||||
return True
|
||||
|
||||
|
||||
async def demo():
|
||||
"""Demonstrate the diagnostic tool with a test instance."""
|
||||
from lightrag.llm.openai import openai_embed, gpt_4o_mini_complete
|
||||
|
||||
print("=" * 50)
|
||||
print("LightRAG Initialization Diagnostic Tool")
|
||||
print("=" * 50)
|
||||
|
||||
# Create test instance
|
||||
rag = LightRAG(
|
||||
working_dir="./test_diagnostic",
|
||||
embedding_func=openai_embed,
|
||||
llm_model_func=gpt_4o_mini_complete,
|
||||
)
|
||||
|
||||
print("\n🔄 Initializing storages...\n")
|
||||
await rag.initialize_storages() # Auto-initializes pipeline_status
|
||||
|
||||
print("\n🔍 Checking initialization status:\n")
|
||||
await check_lightrag_setup(rag, verbose=True)
|
||||
|
||||
# Cleanup
|
||||
import shutil
|
||||
|
||||
shutil.rmtree("./test_diagnostic", ignore_errors=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(description="Check LightRAG initialization status")
|
||||
parser.add_argument(
|
||||
"--demo", action="store_true", help="Run a demonstration with a test instance"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--verbose",
|
||||
"-v",
|
||||
action="store_true",
|
||||
help="Show detailed diagnostic information",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.demo:
|
||||
asyncio.run(demo())
|
||||
else:
|
||||
print("Run with --demo to see the diagnostic tool in action")
|
||||
print("Or import this module and use check_lightrag_setup() with your instance")
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,200 @@
|
||||
"""
|
||||
Download all necessary cache files for offline deployment.
|
||||
|
||||
This module provides a CLI command to download tiktoken model cache files
|
||||
for offline environments where internet access is not available.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
# Known tiktoken encoding names (not model names)
|
||||
# These need to be loaded with tiktoken.get_encoding() instead of tiktoken.encoding_for_model()
|
||||
TIKTOKEN_ENCODING_NAMES = {"cl100k_base", "p50k_base", "r50k_base", "o200k_base"}
|
||||
|
||||
|
||||
def download_tiktoken_cache(cache_dir: str = None, models: list = None):
|
||||
"""Download tiktoken models to local cache
|
||||
|
||||
Args:
|
||||
cache_dir: Directory to store the cache files. If None, uses tiktoken's default location.
|
||||
models: List of model names or encoding names to download. If None, downloads common ones.
|
||||
|
||||
Returns:
|
||||
Tuple of (success_count, failed_models, actual_cache_dir)
|
||||
"""
|
||||
# If user specified a cache directory, set it BEFORE importing tiktoken
|
||||
# tiktoken reads TIKTOKEN_CACHE_DIR at import time
|
||||
user_specified_cache = cache_dir is not None
|
||||
|
||||
if user_specified_cache:
|
||||
cache_dir = os.path.abspath(cache_dir)
|
||||
os.environ["TIKTOKEN_CACHE_DIR"] = cache_dir
|
||||
cache_path = Path(cache_dir)
|
||||
cache_path.mkdir(parents=True, exist_ok=True)
|
||||
print(f"Using specified cache directory: {cache_dir}")
|
||||
else:
|
||||
# Check if TIKTOKEN_CACHE_DIR is already set in environment
|
||||
env_cache_dir = os.environ.get("TIKTOKEN_CACHE_DIR")
|
||||
if env_cache_dir:
|
||||
cache_dir = env_cache_dir
|
||||
print(f"Using TIKTOKEN_CACHE_DIR from environment: {cache_dir}")
|
||||
else:
|
||||
# Use tiktoken's default location (tempdir/data-gym-cache)
|
||||
import tempfile
|
||||
|
||||
cache_dir = os.path.join(tempfile.gettempdir(), "data-gym-cache")
|
||||
print(f"Using tiktoken default cache directory: {cache_dir}")
|
||||
|
||||
# Now import tiktoken (it will use the cache directory we determined)
|
||||
try:
|
||||
import tiktoken
|
||||
except ImportError:
|
||||
print("Error: tiktoken is not installed.")
|
||||
print("Install with: pip install tiktoken")
|
||||
sys.exit(1)
|
||||
|
||||
# Common models used by LightRAG and OpenAI
|
||||
if models is None:
|
||||
models = [
|
||||
"gpt-4o-mini", # Default model for LightRAG
|
||||
"gpt-4o", # GPT-4 Omni
|
||||
"gpt-4", # GPT-4
|
||||
"gpt-3.5-turbo", # GPT-3.5 Turbo
|
||||
"text-embedding-ada-002", # Legacy embedding model
|
||||
"text-embedding-3-small", # Small embedding model
|
||||
"text-embedding-3-large", # Large embedding model
|
||||
"cl100k_base", # Default encoding for LightRAG
|
||||
]
|
||||
|
||||
print(f"\nDownloading {len(models)} tiktoken models...")
|
||||
print("=" * 70)
|
||||
|
||||
success_count = 0
|
||||
failed_models = []
|
||||
|
||||
for i, model in enumerate(models, 1):
|
||||
try:
|
||||
print(f"[{i}/{len(models)}] Downloading {model}...", end=" ", flush=True)
|
||||
# Use get_encoding for encoding names, encoding_for_model for model names
|
||||
if model in TIKTOKEN_ENCODING_NAMES:
|
||||
encoding = tiktoken.get_encoding(model)
|
||||
else:
|
||||
encoding = tiktoken.encoding_for_model(model)
|
||||
# Trigger download by encoding a test string
|
||||
encoding.encode("test")
|
||||
print("✓ Done")
|
||||
success_count += 1
|
||||
except KeyError as e:
|
||||
print(f"✗ Failed: Unknown model or encoding '{model}'")
|
||||
failed_models.append((model, str(e)))
|
||||
except Exception as e:
|
||||
print(f"✗ Failed: {e}")
|
||||
failed_models.append((model, str(e)))
|
||||
|
||||
print("=" * 70)
|
||||
print(f"\n✓ Successfully cached {success_count}/{len(models)} models")
|
||||
|
||||
if failed_models:
|
||||
print(f"\n✗ Failed to download {len(failed_models)} models:")
|
||||
for model, error in failed_models:
|
||||
print(f" - {model}: {error}")
|
||||
|
||||
print(f"\nCache location: {cache_dir}")
|
||||
print("\nFor offline deployment:")
|
||||
print(" 1. Copy directory to offline server:")
|
||||
print(f" tar -czf tiktoken_cache.tar.gz {cache_dir}")
|
||||
print(" scp tiktoken_cache.tar.gz user@offline-server:/path/to/")
|
||||
print("")
|
||||
print(" 2. On offline server, extract and set environment variable:")
|
||||
print(" tar -xzf tiktoken_cache.tar.gz")
|
||||
print(" export TIKTOKEN_CACHE_DIR=/path/to/tiktoken_cache")
|
||||
print("")
|
||||
print(" 3. Or copy to default location:")
|
||||
print(f" cp -r {cache_dir} ~/.tiktoken_cache/")
|
||||
|
||||
return success_count, failed_models
|
||||
|
||||
|
||||
def main():
|
||||
"""Main entry point for the CLI command"""
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="lightrag-download-cache",
|
||||
description="Download cache files for LightRAG offline deployment",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
# Download to default location (~/.tiktoken_cache)
|
||||
lightrag-download-cache
|
||||
|
||||
# Download to specific directory
|
||||
lightrag-download-cache --cache-dir ./offline_cache/tiktoken
|
||||
|
||||
# Download specific models only
|
||||
lightrag-download-cache --models gpt-4o-mini gpt-4
|
||||
|
||||
For more information, visit: https://github.com/HKUDS/LightRAG
|
||||
""",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--cache-dir",
|
||||
help="Cache directory path (default: ~/.tiktoken_cache)",
|
||||
default=None,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--models",
|
||||
nargs="+",
|
||||
help="Specific models to download (default: common models)",
|
||||
default=None,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--version", action="version", version="%(prog)s (LightRAG cache downloader)"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
print("=" * 70)
|
||||
print("LightRAG Offline Cache Downloader")
|
||||
print("=" * 70)
|
||||
|
||||
try:
|
||||
success_count, failed_models = download_tiktoken_cache(
|
||||
args.cache_dir, args.models
|
||||
)
|
||||
|
||||
print("\n" + "=" * 70)
|
||||
print("Download Complete")
|
||||
print("=" * 70)
|
||||
|
||||
# Exit with error code if all downloads failed
|
||||
if success_count == 0:
|
||||
print("\n✗ All downloads failed. Please check your internet connection.")
|
||||
sys.exit(1)
|
||||
# Exit with warning code if some downloads failed
|
||||
elif failed_models:
|
||||
print(
|
||||
f"\n⚠ Some downloads failed ({len(failed_models)}/{success_count + len(failed_models)})"
|
||||
)
|
||||
sys.exit(2)
|
||||
else:
|
||||
print("\n✓ All cache files downloaded successfully!")
|
||||
sys.exit(0)
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n\n✗ Download interrupted by user")
|
||||
sys.exit(130)
|
||||
except Exception as e:
|
||||
print(f"\n\n✗ Error: {e}")
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,39 @@
|
||||
import argparse
|
||||
import getpass
|
||||
|
||||
from lightrag.api.passwords import hash_password
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Generate a bcrypt password value for AUTH_ACCOUNTS."
|
||||
)
|
||||
parser.add_argument(
|
||||
"password",
|
||||
nargs="?",
|
||||
help="Password to hash. If omitted, a secure prompt is used.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--username",
|
||||
help="Optional username. When provided, output is ready to paste into AUTH_ACCOUNTS.",
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = build_parser()
|
||||
args = parser.parse_args(argv)
|
||||
password = args.password or getpass.getpass("Password: ")
|
||||
if not password:
|
||||
parser.error("password cannot be empty")
|
||||
|
||||
hashed_password = hash_password(password)
|
||||
if args.username:
|
||||
print(f"{args.username}:{hashed_password}")
|
||||
else:
|
||||
print(hashed_password)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,95 @@
|
||||
# 3D GraphML Viewer
|
||||
|
||||
一个基于 Dear ImGui 和 ModernGL 的交互式 3D 图可视化工具。
|
||||
|
||||
## 功能特点
|
||||
|
||||
- **3D 交互式可视化**: 使用 ModernGL 实现高性能的 3D 图形渲染
|
||||
- **多种布局算法**: 支持多种图布局方式
|
||||
- Spring 布局
|
||||
- Circular 布局
|
||||
- Shell 布局
|
||||
- Random 布局
|
||||
- **社区检测**: 支持图社区结构的自动检测和可视化
|
||||
- **交互控制**:
|
||||
- WASD + QE 键控制相机移动
|
||||
- 鼠标右键拖拽控制视角
|
||||
- 节点选择和高亮
|
||||
- 可调节节点大小和边宽度
|
||||
- 可控制标签显示
|
||||
- 可在节点的Connections间快速跳转
|
||||
- **社区检测**: 支持图社区结构的自动检测和可视化
|
||||
- **交互控制**:
|
||||
- WASD + QE 键控制相机移动
|
||||
- 鼠标右键拖拽控制视角
|
||||
- 节点选择和高亮
|
||||
- 可调节节点大小和边宽度
|
||||
- 可控制标签显示
|
||||
|
||||
## 技术栈
|
||||
|
||||
- **imgui_bundle**: 用户界面
|
||||
- **ModernGL**: OpenGL 图形渲染
|
||||
- **NetworkX**: 图数据结构和算法
|
||||
- **NumPy**: 数值计算
|
||||
- **community**: 社区检测
|
||||
|
||||
## 使用方法
|
||||
|
||||
1. **启动程序**:
|
||||
```bash
|
||||
pip install lightrag-hku[tools]
|
||||
lightrag-viewer
|
||||
```
|
||||
|
||||
2. **加载字体**:
|
||||
- 将中文字体文件 `font.ttf` 放置在 `assets` 目录下
|
||||
- 或者修改 `CUSTOM_FONT` 常量来使用其他字体文件
|
||||
|
||||
3. **加载图文件**:
|
||||
- 点击界面上的 "Load GraphML" 按钮
|
||||
- 选择 GraphML 格式的图文件
|
||||
|
||||
4. **交互控制**:
|
||||
- **相机移动**:
|
||||
- W: 前进
|
||||
- S: 后退
|
||||
- A: 左移
|
||||
- D: 右移
|
||||
- Q: 上升
|
||||
- E: 下降
|
||||
- **视角控制**:
|
||||
- 按住鼠标右键拖动来旋转视角
|
||||
- **节点交互**:
|
||||
- 鼠标悬停可高亮节点
|
||||
- 点击可选中节点
|
||||
|
||||
5. **可视化设置**:
|
||||
- 可通过 UI 控制面板调整:
|
||||
- 布局类型
|
||||
- 节点大小
|
||||
- 边的宽度
|
||||
- 标签显示
|
||||
- 标签大小
|
||||
- 背景颜色
|
||||
|
||||
## 自定义设置
|
||||
|
||||
- **节点缩放**: 通过 `node_scale` 参数调整节点大小
|
||||
- **边宽度**: 通过 `edge_width` 参数调整边的宽度
|
||||
- **标签显示**: 可通过 `show_labels` 开关标签显示
|
||||
- **标签大小**: 使用 `label_size` 调整标签大小
|
||||
- **标签颜色**: 通过 `label_color` 设置标签颜色
|
||||
- **视距控制**: 使用 `label_culling_distance` 控制标签显示的最大距离
|
||||
|
||||
## 性能优化
|
||||
|
||||
- 使用 ModernGL 进行高效的图形渲染
|
||||
- 视距裁剪优化标签显示
|
||||
- 社区检测算法优化大规模图的可视化效果
|
||||
|
||||
## 系统要求
|
||||
|
||||
- Python 3.10+
|
||||
- OpenGL 3.3+ 兼容的显卡
|
||||
- 支持的操作系统：Windows/Linux/MacOS
|
||||
@@ -0,0 +1,136 @@
|
||||
# LightRAG 3D Graph Viewer
|
||||
|
||||
An interactive 3D graph visualization tool included in the LightRAG package for visualizing and analyzing RAG (Retrieval-Augmented Generation) graphs and other graph structures.
|
||||
|
||||
|
||||
|
||||
## Installation
|
||||
|
||||
### Quick Install
|
||||
```bash
|
||||
pip install lightrag-hku[tools] # Install with visualization tool only
|
||||
# or
|
||||
pip install lightrag-hku[api,tools] # Install with both API and visualization tools
|
||||
```
|
||||
|
||||
## Launch the Viewer
|
||||
```bash
|
||||
lightrag-viewer
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- **3D Interactive Visualization**: High-performance 3D graphics rendering using ModernGL
|
||||
- **Multiple Layout Algorithms**: Support for various graph layouts
|
||||
- Spring layout
|
||||
- Circular layout
|
||||
- Shell layout
|
||||
- Random layout
|
||||
- **Community Detection**: Automatic detection and visualization of graph community structures
|
||||
- **Interactive Controls**:
|
||||
- WASD + QE keys for camera movement
|
||||
- Right mouse drag for view angle control
|
||||
- Node selection and highlighting
|
||||
- Adjustable node size and edge width
|
||||
- Configurable label display
|
||||
- Quick navigation between node connections
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **imgui_bundle**: User interface
|
||||
- **ModernGL**: OpenGL graphics rendering
|
||||
- **NetworkX**: Graph data structures and algorithms
|
||||
- **NumPy**: Numerical computations
|
||||
- **community**: Community detection
|
||||
|
||||
## Interactive Controls
|
||||
|
||||
### Camera Movement
|
||||
- W: Move forward
|
||||
- S: Move backward
|
||||
- A: Move left
|
||||
- D: Move right
|
||||
- Q: Move up
|
||||
- E: Move down
|
||||
|
||||
### View Control
|
||||
- Hold right mouse button and drag to rotate view
|
||||
|
||||
### Node Interaction
|
||||
- Hover mouse to highlight nodes
|
||||
- Click to select nodes
|
||||
|
||||
## Visualization Settings
|
||||
|
||||
Adjustable via UI control panel:
|
||||
- Layout type
|
||||
- Node size
|
||||
- Edge width
|
||||
- Label visibility
|
||||
- Label size
|
||||
- Background color
|
||||
|
||||
## Customization Options
|
||||
|
||||
- **Node Scaling**: Adjust node size via `node_scale` parameter
|
||||
- **Edge Width**: Modify edge width using `edge_width` parameter
|
||||
- **Label Display**: Toggle label visibility with `show_labels`
|
||||
- **Label Size**: Adjust label size using `label_size`
|
||||
- **Label Color**: Set label color through `label_color`
|
||||
- **View Distance**: Control maximum label display distance with `label_culling_distance`
|
||||
|
||||
## System Requirements
|
||||
|
||||
- Python 3.9+
|
||||
- Graphics card with OpenGL 3.3+ support
|
||||
- Supported Operating Systems: Windows/Linux/MacOS
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Command Not Found**
|
||||
```bash
|
||||
# Make sure you installed with the 'tools' option
|
||||
pip install lightrag-hku[tools]
|
||||
|
||||
# Verify installation
|
||||
pip list | grep lightrag-hku
|
||||
```
|
||||
|
||||
2. **ModernGL Initialization Failed**
|
||||
```bash
|
||||
# Check OpenGL version
|
||||
glxinfo | grep "OpenGL version"
|
||||
|
||||
# Update graphics drivers if needed
|
||||
```
|
||||
|
||||
3. **Font Loading Issues**
|
||||
- The required fonts are included in the package
|
||||
- If issues persist, check your graphics drivers
|
||||
|
||||
## Usage with LightRAG
|
||||
|
||||
The viewer is particularly useful for:
|
||||
- Visualizing RAG knowledge graphs
|
||||
- Analyzing document relationships
|
||||
- Exploring semantic connections
|
||||
- Debugging retrieval patterns
|
||||
|
||||
## Performance Optimizations
|
||||
|
||||
- Efficient graphics rendering using ModernGL
|
||||
- View distance culling for label display optimization
|
||||
- Community detection algorithms for optimized visualization of large-scale graphs
|
||||
|
||||
## Support
|
||||
|
||||
- GitHub Issues: [LightRAG Repository](https://github.com/HKUDS/LightRAG)
|
||||
- Documentation: [LightRAG Docs](https://URL-to-docs)
|
||||
|
||||
## License
|
||||
|
||||
This tool is part of LightRAG and is distributed under the MIT License. See `LICENSE` for more information.
|
||||
|
||||
Note: This visualization tool is an optional component of the LightRAG package. Install with the [tools] option to access the viewer functionality.
|
||||
Binary file not shown.
@@ -0,0 +1,92 @@
|
||||
Copyright (c) 2023 Vercel, in collaboration with basement.studio
|
||||
|
||||
This Font Software is licensed under the SIL Open Font License, Version 1.1.
|
||||
This license is copied below, and is also available with a FAQ at:
|
||||
http://scripts.sil.org/OFL
|
||||
|
||||
-----------------------------------------------------------
|
||||
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
|
||||
-----------------------------------------------------------
|
||||
|
||||
PREAMBLE
|
||||
The goals of the Open Font License (OFL) are to stimulate worldwide
|
||||
development of collaborative font projects, to support the font creation
|
||||
efforts of academic and linguistic communities, and to provide a free and
|
||||
open framework in which fonts may be shared and improved in partnership
|
||||
with others.
|
||||
|
||||
The OFL allows the licensed fonts to be used, studied, modified and
|
||||
redistributed freely as long as they are not sold by themselves. The
|
||||
fonts, including any derivative works, can be bundled, embedded,
|
||||
redistributed and/or sold with any software provided that any reserved
|
||||
names are not used by derivative works. The fonts and derivatives,
|
||||
however, cannot be released under any other type of license. The
|
||||
requirement for fonts to remain under this license does not apply
|
||||
to any document created using the fonts or their derivatives.
|
||||
|
||||
DEFINITIONS
|
||||
"Font Software" refers to the set of files released by the Copyright
|
||||
Holder(s) under this license and clearly marked as such. This may
|
||||
include source files, build scripts and documentation.
|
||||
|
||||
"Reserved Font Name" refers to any names specified as such after the
|
||||
copyright statement(s).
|
||||
|
||||
"Original Version" refers to the collection of Font Software components as
|
||||
distributed by the Copyright Holder(s).
|
||||
|
||||
"Modified Version" refers to any derivative made by adding to, deleting,
|
||||
or substituting -- in part or in whole -- any of the components of the
|
||||
Original Version, by changing formats or by porting the Font Software to a
|
||||
new environment.
|
||||
|
||||
"Author" refers to any designer, engineer, programmer, technical
|
||||
writer or other person who contributed to the Font Software.
|
||||
|
||||
PERMISSION AND CONDITIONS
|
||||
Permission is hereby granted, free of charge, to any person obtaining
|
||||
a copy of the Font Software, to use, study, copy, merge, embed, modify,
|
||||
redistribute, and sell modified and unmodified copies of the Font
|
||||
Software, subject to the following conditions:
|
||||
|
||||
1) Neither the Font Software nor any of its individual components,
|
||||
in Original or Modified Versions, may be sold by itself.
|
||||
|
||||
2) Original or Modified Versions of the Font Software may be bundled,
|
||||
redistributed and/or sold with any software, provided that each copy
|
||||
contains the above copyright notice and this license. These can be
|
||||
included either as stand-alone text files, human-readable headers or
|
||||
in the appropriate machine-readable metadata fields within text or
|
||||
binary files as long as those fields can be easily viewed by the user.
|
||||
|
||||
3) No Modified Version of the Font Software may use the Reserved Font
|
||||
Name(s) unless explicit written permission is granted by the corresponding
|
||||
Copyright Holder. This restriction only applies to the primary font name as
|
||||
presented to the users.
|
||||
|
||||
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
|
||||
Software shall not be used to promote, endorse or advertise any
|
||||
Modified Version, except to acknowledge the contribution(s) of the
|
||||
Copyright Holder(s) and the Author(s) or with their explicit written
|
||||
permission.
|
||||
|
||||
5) The Font Software, modified or unmodified, in part or in whole,
|
||||
must be distributed entirely under this license, and must not be
|
||||
distributed under any other license. The requirement for fonts to
|
||||
remain under this license does not apply to any document created
|
||||
using the Font Software.
|
||||
|
||||
TERMINATION
|
||||
This license becomes null and void if any of the above conditions are
|
||||
not met.
|
||||
|
||||
DISCLAIMER
|
||||
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
||||
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
|
||||
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
|
||||
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
|
||||
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
|
||||
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
|
||||
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
||||
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
|
||||
OTHER DEALINGS IN THE FONT SOFTWARE.
|
||||
@@ -0,0 +1,93 @@
|
||||
Copyright (c) 2022--2024, atelierAnchor <https://atelier-anchor.com>,
|
||||
with Reserved Font Name <Smiley> and <得意黑>.
|
||||
|
||||
This Font Software is licensed under the SIL Open Font License, Version 1.1.
|
||||
This license is copied below, and is also available with a FAQ at:
|
||||
http://scripts.sil.org/OFL
|
||||
|
||||
-----------------------------------------------------------
|
||||
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
|
||||
-----------------------------------------------------------
|
||||
|
||||
PREAMBLE
|
||||
The goals of the Open Font License (OFL) are to stimulate worldwide
|
||||
development of collaborative font projects, to support the font creation
|
||||
efforts of academic and linguistic communities, and to provide a free and
|
||||
open framework in which fonts may be shared and improved in partnership
|
||||
with others.
|
||||
|
||||
The OFL allows the licensed fonts to be used, studied, modified and
|
||||
redistributed freely as long as they are not sold by themselves. The
|
||||
fonts, including any derivative works, can be bundled, embedded,
|
||||
redistributed and/or sold with any software provided that any reserved
|
||||
names are not used by derivative works. The fonts and derivatives,
|
||||
however, cannot be released under any other type of license. The
|
||||
requirement for fonts to remain under this license does not apply
|
||||
to any document created using the fonts or their derivatives.
|
||||
|
||||
DEFINITIONS
|
||||
"Font Software" refers to the set of files released by the Copyright
|
||||
Holder(s) under this license and clearly marked as such. This may
|
||||
include source files, build scripts and documentation.
|
||||
|
||||
"Reserved Font Name" refers to any names specified as such after the
|
||||
copyright statement(s).
|
||||
|
||||
"Original Version" refers to the collection of Font Software components as
|
||||
distributed by the Copyright Holder(s).
|
||||
|
||||
"Modified Version" refers to any derivative made by adding to, deleting,
|
||||
or substituting -- in part or in whole -- any of the components of the
|
||||
Original Version, by changing formats or by porting the Font Software to a
|
||||
new environment.
|
||||
|
||||
"Author" refers to any designer, engineer, programmer, technical
|
||||
writer or other person who contributed to the Font Software.
|
||||
|
||||
PERMISSION & CONDITIONS
|
||||
Permission is hereby granted, free of charge, to any person obtaining
|
||||
a copy of the Font Software, to use, study, copy, merge, embed, modify,
|
||||
redistribute, and sell modified and unmodified copies of the Font
|
||||
Software, subject to the following conditions:
|
||||
|
||||
1) Neither the Font Software nor any of its individual components,
|
||||
in Original or Modified Versions, may be sold by itself.
|
||||
|
||||
2) Original or Modified Versions of the Font Software may be bundled,
|
||||
redistributed and/or sold with any software, provided that each copy
|
||||
contains the above copyright notice and this license. These can be
|
||||
included either as stand-alone text files, human-readable headers or
|
||||
in the appropriate machine-readable metadata fields within text or
|
||||
binary files as long as those fields can be easily viewed by the user.
|
||||
|
||||
3) No Modified Version of the Font Software may use the Reserved Font
|
||||
Name(s) unless explicit written permission is granted by the corresponding
|
||||
Copyright Holder. This restriction only applies to the primary font name as
|
||||
presented to the users.
|
||||
|
||||
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
|
||||
Software shall not be used to promote, endorse or advertise any
|
||||
Modified Version, except to acknowledge the contribution(s) of the
|
||||
Copyright Holder(s) and the Author(s) or with their explicit written
|
||||
permission.
|
||||
|
||||
5) The Font Software, modified or unmodified, in part or in whole,
|
||||
must be distributed entirely under this license, and must not be
|
||||
distributed under any other license. The requirement for fonts to
|
||||
remain under this license does not apply to any document created
|
||||
using the Font Software.
|
||||
|
||||
TERMINATION
|
||||
This license becomes null and void if any of the above conditions are
|
||||
not met.
|
||||
|
||||
DISCLAIMER
|
||||
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
||||
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
|
||||
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
|
||||
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
|
||||
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
|
||||
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
|
||||
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
||||
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
|
||||
OTHER DEALINGS IN THE FONT SOFTWARE.
|
||||
Binary file not shown.
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,8 @@
|
||||
imgui_bundle
|
||||
moderngl
|
||||
networkx
|
||||
numpy
|
||||
pyglm
|
||||
python-louvain
|
||||
scipy
|
||||
tk
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,720 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Qdrant Legacy Data Preparation Tool for LightRAG
|
||||
|
||||
This tool copies data from new collections to legacy collections for testing
|
||||
the data migration logic in setup_collection function.
|
||||
|
||||
New Collections (with workspace_id):
|
||||
- lightrag_vdb_chunks
|
||||
- lightrag_vdb_entities
|
||||
- lightrag_vdb_relationships
|
||||
|
||||
Legacy Collections (without workspace_id, dynamically named as {workspace}_{suffix}):
|
||||
- {workspace}_chunks (e.g., space1_chunks)
|
||||
- {workspace}_entities (e.g., space1_entities)
|
||||
- {workspace}_relationships (e.g., space1_relationships)
|
||||
|
||||
The tool:
|
||||
1. Filters source data by workspace_id
|
||||
2. Verifies workspace data exists before creating legacy collections
|
||||
3. Removes workspace_id field to simulate legacy data format
|
||||
4. Copies only the specified workspace's data to legacy collections
|
||||
|
||||
Usage:
|
||||
python -m lightrag.tools.prepare_qdrant_legacy_data
|
||||
# or
|
||||
python lightrag/tools/prepare_qdrant_legacy_data.py
|
||||
|
||||
# Specify custom workspace
|
||||
python -m lightrag.tools.prepare_qdrant_legacy_data --workspace space1
|
||||
|
||||
# Process specific collection types only
|
||||
python -m lightrag.tools.prepare_qdrant_legacy_data --types chunks,entities
|
||||
|
||||
# Dry run (preview only, no actual changes)
|
||||
python -m lightrag.tools.prepare_qdrant_legacy_data --dry-run
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import configparser
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
import pipmaster as pm
|
||||
from dotenv import load_dotenv
|
||||
from qdrant_client import QdrantClient, models # type: ignore
|
||||
|
||||
# Add project root to path for imports
|
||||
sys.path.insert(
|
||||
0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
)
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv(dotenv_path=".env", override=False)
|
||||
|
||||
# Ensure qdrant-client is installed
|
||||
if not pm.is_installed("qdrant-client"):
|
||||
pm.install("qdrant-client")
|
||||
|
||||
# Collection namespace mapping: new collection pattern -> legacy suffix
|
||||
# Legacy collection will be named as: {workspace}_{suffix}
|
||||
COLLECTION_NAMESPACES = {
|
||||
"chunks": {
|
||||
"new": "lightrag_vdb_chunks",
|
||||
"suffix": "chunks",
|
||||
},
|
||||
"entities": {
|
||||
"new": "lightrag_vdb_entities",
|
||||
"suffix": "entities",
|
||||
},
|
||||
"relationships": {
|
||||
"new": "lightrag_vdb_relationships",
|
||||
"suffix": "relationships",
|
||||
},
|
||||
}
|
||||
|
||||
# Default batch size for copy operations
|
||||
DEFAULT_BATCH_SIZE = 500
|
||||
|
||||
# Field to remove from legacy data
|
||||
WORKSPACE_ID_FIELD = "workspace_id"
|
||||
|
||||
# ANSI color codes for terminal output
|
||||
BOLD_CYAN = "\033[1;36m"
|
||||
BOLD_GREEN = "\033[1;32m"
|
||||
BOLD_YELLOW = "\033[1;33m"
|
||||
BOLD_RED = "\033[1;31m"
|
||||
RESET = "\033[0m"
|
||||
|
||||
|
||||
@dataclass
|
||||
class CopyStats:
|
||||
"""Copy operation statistics"""
|
||||
|
||||
collection_type: str
|
||||
source_collection: str
|
||||
target_collection: str
|
||||
total_records: int = 0
|
||||
copied_records: int = 0
|
||||
failed_records: int = 0
|
||||
errors: List[Dict[str, Any]] = field(default_factory=list)
|
||||
elapsed_time: float = 0.0
|
||||
|
||||
def add_error(self, batch_idx: int, error: Exception, batch_size: int):
|
||||
"""Record batch error"""
|
||||
self.errors.append(
|
||||
{
|
||||
"batch": batch_idx,
|
||||
"error_type": type(error).__name__,
|
||||
"error_msg": str(error),
|
||||
"records_lost": batch_size,
|
||||
"timestamp": time.time(),
|
||||
}
|
||||
)
|
||||
self.failed_records += batch_size
|
||||
|
||||
|
||||
class QdrantLegacyDataPreparationTool:
|
||||
"""Tool for preparing legacy data in Qdrant for migration testing"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
workspace: str = "space1",
|
||||
batch_size: int = DEFAULT_BATCH_SIZE,
|
||||
dry_run: bool = False,
|
||||
clear_target: bool = False,
|
||||
):
|
||||
"""
|
||||
Initialize the tool.
|
||||
|
||||
Args:
|
||||
workspace: Workspace to use for filtering new collection data
|
||||
batch_size: Number of records to process per batch
|
||||
dry_run: If True, only preview operations without making changes
|
||||
clear_target: If True, delete target collection before copying data
|
||||
"""
|
||||
self.workspace = workspace
|
||||
self.batch_size = batch_size
|
||||
self.dry_run = dry_run
|
||||
self.clear_target = clear_target
|
||||
self._client: Optional[QdrantClient] = None
|
||||
|
||||
def _get_client(self) -> QdrantClient:
|
||||
"""Get or create QdrantClient instance"""
|
||||
if self._client is None:
|
||||
config = configparser.ConfigParser()
|
||||
config.read("config.ini", "utf-8")
|
||||
|
||||
self._client = QdrantClient(
|
||||
url=os.environ.get(
|
||||
"QDRANT_URL", config.get("qdrant", "uri", fallback=None)
|
||||
),
|
||||
api_key=os.environ.get(
|
||||
"QDRANT_API_KEY",
|
||||
config.get("qdrant", "apikey", fallback=None),
|
||||
),
|
||||
)
|
||||
return self._client
|
||||
|
||||
def print_header(self):
|
||||
"""Print tool header"""
|
||||
print("\n" + "=" * 60)
|
||||
print("Qdrant Legacy Data Preparation Tool - LightRAG")
|
||||
print("=" * 60)
|
||||
if self.dry_run:
|
||||
print(f"{BOLD_YELLOW}⚠️ DRY RUN MODE - No changes will be made{RESET}")
|
||||
if self.clear_target:
|
||||
print(
|
||||
f"{BOLD_RED}⚠️ CLEAR TARGET MODE - Target collections will be deleted first{RESET}"
|
||||
)
|
||||
print(f"Workspace: {BOLD_CYAN}{self.workspace}{RESET}")
|
||||
print(f"Batch Size: {self.batch_size}")
|
||||
print("=" * 60)
|
||||
|
||||
def check_connection(self) -> bool:
|
||||
"""Check Qdrant connection"""
|
||||
try:
|
||||
client = self._get_client()
|
||||
# Try to list collections to verify connection
|
||||
client.get_collections()
|
||||
print(f"{BOLD_GREEN}✓{RESET} Qdrant connection successful")
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"{BOLD_RED}✗{RESET} Qdrant connection failed: {e}")
|
||||
return False
|
||||
|
||||
def get_collection_info(self, collection_name: str) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Get collection information.
|
||||
|
||||
Args:
|
||||
collection_name: Name of the collection
|
||||
|
||||
Returns:
|
||||
Dictionary with collection info (vector_size, count) or None if not exists
|
||||
"""
|
||||
client = self._get_client()
|
||||
|
||||
if not client.collection_exists(collection_name):
|
||||
return None
|
||||
|
||||
info = client.get_collection(collection_name)
|
||||
count = client.count(collection_name=collection_name, exact=True).count
|
||||
|
||||
# Handle both object and dict formats for vectors config
|
||||
vectors_config = info.config.params.vectors
|
||||
if isinstance(vectors_config, dict):
|
||||
# Named vectors format or dict format
|
||||
if vectors_config:
|
||||
first_key = next(iter(vectors_config.keys()), None)
|
||||
if first_key and hasattr(vectors_config[first_key], "size"):
|
||||
vector_size = vectors_config[first_key].size
|
||||
distance = vectors_config[first_key].distance
|
||||
else:
|
||||
# Try to get from dict values
|
||||
first_val = next(iter(vectors_config.values()), {})
|
||||
vector_size = (
|
||||
first_val.get("size")
|
||||
if isinstance(first_val, dict)
|
||||
else getattr(first_val, "size", None)
|
||||
)
|
||||
distance = (
|
||||
first_val.get("distance")
|
||||
if isinstance(first_val, dict)
|
||||
else getattr(first_val, "distance", None)
|
||||
)
|
||||
else:
|
||||
vector_size = None
|
||||
distance = None
|
||||
else:
|
||||
# Standard single vector format
|
||||
vector_size = vectors_config.size
|
||||
distance = vectors_config.distance
|
||||
|
||||
return {
|
||||
"name": collection_name,
|
||||
"vector_size": vector_size,
|
||||
"count": count,
|
||||
"distance": distance,
|
||||
}
|
||||
|
||||
def delete_collection(self, collection_name: str) -> bool:
|
||||
"""
|
||||
Delete a collection if it exists.
|
||||
|
||||
Args:
|
||||
collection_name: Name of the collection to delete
|
||||
|
||||
Returns:
|
||||
True if deleted or doesn't exist
|
||||
"""
|
||||
client = self._get_client()
|
||||
|
||||
if not client.collection_exists(collection_name):
|
||||
return True
|
||||
|
||||
if self.dry_run:
|
||||
target_info = self.get_collection_info(collection_name)
|
||||
count = target_info["count"] if target_info else 0
|
||||
print(
|
||||
f" {BOLD_YELLOW}[DRY RUN]{RESET} Would delete collection '{collection_name}' ({count:,} records)"
|
||||
)
|
||||
return True
|
||||
|
||||
try:
|
||||
target_info = self.get_collection_info(collection_name)
|
||||
count = target_info["count"] if target_info else 0
|
||||
client.delete_collection(collection_name=collection_name)
|
||||
print(
|
||||
f" {BOLD_RED}✗{RESET} Deleted collection '{collection_name}' ({count:,} records)"
|
||||
)
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f" {BOLD_RED}✗{RESET} Failed to delete collection: {e}")
|
||||
return False
|
||||
|
||||
def create_legacy_collection(
|
||||
self, collection_name: str, vector_size: int, distance: models.Distance
|
||||
) -> bool:
|
||||
"""
|
||||
Create legacy collection if it doesn't exist.
|
||||
|
||||
Args:
|
||||
collection_name: Name of the collection to create
|
||||
vector_size: Dimension of vectors
|
||||
distance: Distance metric
|
||||
|
||||
Returns:
|
||||
True if created or already exists
|
||||
"""
|
||||
client = self._get_client()
|
||||
|
||||
if client.collection_exists(collection_name):
|
||||
print(f" Collection '{collection_name}' already exists")
|
||||
return True
|
||||
|
||||
if self.dry_run:
|
||||
print(
|
||||
f" {BOLD_YELLOW}[DRY RUN]{RESET} Would create collection '{collection_name}' with {vector_size}d vectors"
|
||||
)
|
||||
return True
|
||||
|
||||
try:
|
||||
client.create_collection(
|
||||
collection_name=collection_name,
|
||||
vectors_config=models.VectorParams(
|
||||
size=vector_size,
|
||||
distance=distance,
|
||||
),
|
||||
hnsw_config=models.HnswConfigDiff(
|
||||
payload_m=16,
|
||||
m=0,
|
||||
),
|
||||
)
|
||||
print(
|
||||
f" {BOLD_GREEN}✓{RESET} Created collection '{collection_name}' with {vector_size}d vectors"
|
||||
)
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f" {BOLD_RED}✗{RESET} Failed to create collection: {e}")
|
||||
return False
|
||||
|
||||
def _get_workspace_filter(self) -> models.Filter:
|
||||
"""Create workspace filter for Qdrant queries"""
|
||||
return models.Filter(
|
||||
must=[
|
||||
models.FieldCondition(
|
||||
key=WORKSPACE_ID_FIELD,
|
||||
match=models.MatchValue(value=self.workspace),
|
||||
)
|
||||
]
|
||||
)
|
||||
|
||||
def get_workspace_count(self, collection_name: str) -> int:
|
||||
"""
|
||||
Get count of records for the current workspace in a collection.
|
||||
|
||||
Args:
|
||||
collection_name: Name of the collection
|
||||
|
||||
Returns:
|
||||
Count of records for the workspace
|
||||
"""
|
||||
client = self._get_client()
|
||||
return client.count(
|
||||
collection_name=collection_name,
|
||||
count_filter=self._get_workspace_filter(),
|
||||
exact=True,
|
||||
).count
|
||||
|
||||
def copy_collection_data(
|
||||
self,
|
||||
source_collection: str,
|
||||
target_collection: str,
|
||||
collection_type: str,
|
||||
workspace_count: int,
|
||||
) -> CopyStats:
|
||||
"""
|
||||
Copy data from source to target collection.
|
||||
|
||||
This filters by workspace_id and removes it from payload to simulate legacy data format.
|
||||
|
||||
Args:
|
||||
source_collection: Source collection name
|
||||
target_collection: Target collection name
|
||||
collection_type: Type of collection (chunks, entities, relationships)
|
||||
workspace_count: Pre-computed count of workspace records
|
||||
|
||||
Returns:
|
||||
CopyStats with operation results
|
||||
"""
|
||||
client = self._get_client()
|
||||
stats = CopyStats(
|
||||
collection_type=collection_type,
|
||||
source_collection=source_collection,
|
||||
target_collection=target_collection,
|
||||
)
|
||||
|
||||
start_time = time.time()
|
||||
stats.total_records = workspace_count
|
||||
|
||||
if workspace_count == 0:
|
||||
print(f" No records for workspace '{self.workspace}', skipping")
|
||||
stats.elapsed_time = time.time() - start_time
|
||||
return stats
|
||||
|
||||
print(f" Workspace records: {workspace_count:,}")
|
||||
|
||||
if self.dry_run:
|
||||
print(
|
||||
f" {BOLD_YELLOW}[DRY RUN]{RESET} Would copy {workspace_count:,} records to '{target_collection}'"
|
||||
)
|
||||
stats.copied_records = workspace_count
|
||||
stats.elapsed_time = time.time() - start_time
|
||||
return stats
|
||||
|
||||
# Batch copy using scroll with workspace filter
|
||||
workspace_filter = self._get_workspace_filter()
|
||||
offset = None
|
||||
batch_idx = 0
|
||||
|
||||
while True:
|
||||
# Scroll source collection with workspace filter
|
||||
result = client.scroll(
|
||||
collection_name=source_collection,
|
||||
scroll_filter=workspace_filter,
|
||||
limit=self.batch_size,
|
||||
offset=offset,
|
||||
with_vectors=True,
|
||||
with_payload=True,
|
||||
)
|
||||
points, next_offset = result
|
||||
|
||||
if not points:
|
||||
break
|
||||
|
||||
batch_idx += 1
|
||||
|
||||
# Transform points: remove workspace_id from payload
|
||||
new_points = []
|
||||
for point in points:
|
||||
new_payload = dict(point.payload or {})
|
||||
# Remove workspace_id to simulate legacy format
|
||||
new_payload.pop(WORKSPACE_ID_FIELD, None)
|
||||
|
||||
# Use original id from payload if available, otherwise use point.id
|
||||
original_id = new_payload.get("id")
|
||||
if original_id:
|
||||
# Generate a simple deterministic id for legacy format
|
||||
# Use original id directly (legacy format didn't have workspace prefix)
|
||||
import hashlib
|
||||
import uuid
|
||||
|
||||
hashed = hashlib.sha256(original_id.encode("utf-8")).digest()
|
||||
point_id = uuid.UUID(bytes=hashed[:16], version=4).hex
|
||||
else:
|
||||
point_id = str(point.id)
|
||||
|
||||
new_points.append(
|
||||
models.PointStruct(
|
||||
id=point_id,
|
||||
vector=point.vector,
|
||||
payload=new_payload,
|
||||
)
|
||||
)
|
||||
|
||||
try:
|
||||
# Upsert to target collection
|
||||
client.upsert(
|
||||
collection_name=target_collection, points=new_points, wait=True
|
||||
)
|
||||
stats.copied_records += len(new_points)
|
||||
|
||||
# Progress bar
|
||||
progress = (stats.copied_records / workspace_count) * 100
|
||||
bar_length = 30
|
||||
filled = int(bar_length * stats.copied_records // workspace_count)
|
||||
bar = "█" * filled + "░" * (bar_length - filled)
|
||||
|
||||
print(
|
||||
f"\r Copying: {bar} {stats.copied_records:,}/{workspace_count:,} ({progress:.1f}%) ",
|
||||
end="",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
stats.add_error(batch_idx, e, len(new_points))
|
||||
print(
|
||||
f"\n {BOLD_RED}✗{RESET} Batch {batch_idx} failed: {type(e).__name__}: {e}"
|
||||
)
|
||||
|
||||
if next_offset is None:
|
||||
break
|
||||
offset = next_offset
|
||||
|
||||
print() # New line after progress bar
|
||||
stats.elapsed_time = time.time() - start_time
|
||||
|
||||
return stats
|
||||
|
||||
def process_collection_type(self, collection_type: str) -> Optional[CopyStats]:
|
||||
"""
|
||||
Process a single collection type.
|
||||
|
||||
Args:
|
||||
collection_type: Type of collection (chunks, entities, relationships)
|
||||
|
||||
Returns:
|
||||
CopyStats or None if error
|
||||
"""
|
||||
namespace_config = COLLECTION_NAMESPACES.get(collection_type)
|
||||
if not namespace_config:
|
||||
print(f"{BOLD_RED}✗{RESET} Unknown collection type: {collection_type}")
|
||||
return None
|
||||
|
||||
source = namespace_config["new"]
|
||||
# Generate legacy collection name dynamically: {workspace}_{suffix}
|
||||
target = f"{self.workspace}_{namespace_config['suffix']}"
|
||||
|
||||
print(f"\n{'=' * 50}")
|
||||
print(f"Processing: {BOLD_CYAN}{collection_type}{RESET}")
|
||||
print(f"{'=' * 50}")
|
||||
print(f" Source: {source}")
|
||||
print(f" Target: {target}")
|
||||
|
||||
# Check source collection
|
||||
source_info = self.get_collection_info(source)
|
||||
if source_info is None:
|
||||
print(
|
||||
f" {BOLD_YELLOW}⚠{RESET} Source collection '{source}' does not exist, skipping"
|
||||
)
|
||||
return None
|
||||
|
||||
print(f" Source vector dimension: {source_info['vector_size']}d")
|
||||
print(f" Source distance metric: {source_info['distance']}")
|
||||
print(f" Source total records: {source_info['count']:,}")
|
||||
|
||||
# Check workspace data exists BEFORE creating legacy collection
|
||||
workspace_count = self.get_workspace_count(source)
|
||||
print(f" Workspace '{self.workspace}' records: {workspace_count:,}")
|
||||
|
||||
if workspace_count == 0:
|
||||
print(
|
||||
f" {BOLD_YELLOW}⚠{RESET} No data found for workspace '{self.workspace}' in '{source}', skipping"
|
||||
)
|
||||
return None
|
||||
|
||||
# Clear target collection if requested
|
||||
if self.clear_target:
|
||||
if not self.delete_collection(target):
|
||||
return None
|
||||
|
||||
# Create target collection only after confirming workspace data exists
|
||||
if not self.create_legacy_collection(
|
||||
target, source_info["vector_size"], source_info["distance"]
|
||||
):
|
||||
return None
|
||||
|
||||
# Copy data with workspace filter
|
||||
stats = self.copy_collection_data(
|
||||
source, target, collection_type, workspace_count
|
||||
)
|
||||
|
||||
# Print result
|
||||
if stats.failed_records == 0:
|
||||
print(
|
||||
f" {BOLD_GREEN}✓{RESET} Copied {stats.copied_records:,} records in {stats.elapsed_time:.2f}s"
|
||||
)
|
||||
else:
|
||||
print(
|
||||
f" {BOLD_YELLOW}⚠{RESET} Copied {stats.copied_records:,} records, "
|
||||
f"{BOLD_RED}{stats.failed_records:,} failed{RESET} in {stats.elapsed_time:.2f}s"
|
||||
)
|
||||
|
||||
return stats
|
||||
|
||||
def print_summary(self, all_stats: List[CopyStats]):
|
||||
"""Print summary of all operations"""
|
||||
print("\n" + "=" * 60)
|
||||
print("Summary")
|
||||
print("=" * 60)
|
||||
|
||||
total_copied = sum(s.copied_records for s in all_stats)
|
||||
total_failed = sum(s.failed_records for s in all_stats)
|
||||
total_time = sum(s.elapsed_time for s in all_stats)
|
||||
|
||||
for stats in all_stats:
|
||||
status = (
|
||||
f"{BOLD_GREEN}✓{RESET}"
|
||||
if stats.failed_records == 0
|
||||
else f"{BOLD_YELLOW}⚠{RESET}"
|
||||
)
|
||||
print(
|
||||
f" {status} {stats.collection_type}: {stats.copied_records:,}/{stats.total_records:,} "
|
||||
f"({stats.source_collection} → {stats.target_collection})"
|
||||
)
|
||||
|
||||
print("-" * 60)
|
||||
print(f" Total records copied: {BOLD_CYAN}{total_copied:,}{RESET}")
|
||||
if total_failed > 0:
|
||||
print(f" Total records failed: {BOLD_RED}{total_failed:,}{RESET}")
|
||||
print(f" Total time: {total_time:.2f}s")
|
||||
|
||||
if self.dry_run:
|
||||
print(f"\n{BOLD_YELLOW}⚠️ DRY RUN - No actual changes were made{RESET}")
|
||||
|
||||
# Print error details if any
|
||||
all_errors = []
|
||||
for stats in all_stats:
|
||||
all_errors.extend(stats.errors)
|
||||
|
||||
if all_errors:
|
||||
print(f"\n{BOLD_RED}Errors ({len(all_errors)}){RESET}")
|
||||
for i, error in enumerate(all_errors[:5], 1):
|
||||
print(
|
||||
f" {i}. Batch {error['batch']}: {error['error_type']}: {error['error_msg']}"
|
||||
)
|
||||
if len(all_errors) > 5:
|
||||
print(f" ... and {len(all_errors) - 5} more errors")
|
||||
|
||||
print("=" * 60)
|
||||
|
||||
async def run(self, collection_types: Optional[List[str]] = None):
|
||||
"""
|
||||
Run the data preparation tool.
|
||||
|
||||
Args:
|
||||
collection_types: List of collection types to process (default: all)
|
||||
"""
|
||||
self.print_header()
|
||||
|
||||
# Check connection
|
||||
if not self.check_connection():
|
||||
return
|
||||
|
||||
# Determine which collection types to process
|
||||
if collection_types:
|
||||
types_to_process = [t.strip() for t in collection_types]
|
||||
invalid_types = [
|
||||
t for t in types_to_process if t not in COLLECTION_NAMESPACES
|
||||
]
|
||||
if invalid_types:
|
||||
print(
|
||||
f"{BOLD_RED}✗{RESET} Invalid collection types: {', '.join(invalid_types)}"
|
||||
)
|
||||
print(f" Valid types: {', '.join(COLLECTION_NAMESPACES.keys())}")
|
||||
return
|
||||
else:
|
||||
types_to_process = list(COLLECTION_NAMESPACES.keys())
|
||||
|
||||
print(f"\nCollection types to process: {', '.join(types_to_process)}")
|
||||
|
||||
# Process each collection type
|
||||
all_stats = []
|
||||
for ctype in types_to_process:
|
||||
stats = self.process_collection_type(ctype)
|
||||
if stats:
|
||||
all_stats.append(stats)
|
||||
|
||||
# Print summary
|
||||
if all_stats:
|
||||
self.print_summary(all_stats)
|
||||
else:
|
||||
print(f"\n{BOLD_YELLOW}⚠{RESET} No collections were processed")
|
||||
|
||||
|
||||
def parse_args():
|
||||
"""Parse command line arguments"""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Prepare legacy data in Qdrant for migration testing",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
python -m lightrag.tools.prepare_qdrant_legacy_data
|
||||
python -m lightrag.tools.prepare_qdrant_legacy_data --workspace space1
|
||||
python -m lightrag.tools.prepare_qdrant_legacy_data --types chunks,entities
|
||||
python -m lightrag.tools.prepare_qdrant_legacy_data --dry-run
|
||||
""",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--workspace",
|
||||
type=str,
|
||||
default="space1",
|
||||
help="Workspace name (default: space1)",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--types",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Comma-separated list of collection types (chunks, entities, relationships)",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--batch-size",
|
||||
type=int,
|
||||
default=DEFAULT_BATCH_SIZE,
|
||||
help=f"Batch size for copy operations (default: {DEFAULT_BATCH_SIZE})",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--dry-run",
|
||||
action="store_true",
|
||||
help="Preview operations without making changes",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--clear-target",
|
||||
action="store_true",
|
||||
help="Delete target collections before copying (for clean test environment)",
|
||||
)
|
||||
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
async def main():
|
||||
"""Main entry point"""
|
||||
args = parse_args()
|
||||
|
||||
collection_types = None
|
||||
if args.types:
|
||||
collection_types = [t.strip() for t in args.types.split(",")]
|
||||
|
||||
tool = QdrantLegacyDataPreparationTool(
|
||||
workspace=args.workspace,
|
||||
batch_size=args.batch_size,
|
||||
dry_run=args.dry_run,
|
||||
clear_target=args.clear_target,
|
||||
)
|
||||
|
||||
await tool.run(collection_types=collection_types)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user