learningcircuit--local-deep-research/docs/troubleshooting.md

Troubleshooting Guide

This guide covers common issues and their solutions.

Table of Contents

• LLM Connection Issues
• Search Engine Issues
• Rate Limiting
• Database Issues
• WebSocket/Real-time Updates
• Docker Issues
• API Issues
• Performance Issues
• Resource Exhaustion

---

LLM Connection Issues

Ollama Not Connecting

Symptoms:

• "Failed to connect to Ollama"
• "Connection refused" errors
• Empty responses from LLM

Solutions:

1. Verify Ollama is running:

   curl http://localhost:11434/api/tags
   

2. Check the URL configuration:

   • Default: http://localhost:11434
   • For Docker: Use http://host.docker.internal:11434 or your host IP
   • Settings location: llm.ollama.url

3. Verify the model is pulled:

   ollama list
   ollama pull llama3.2
   

4. Check Docker networking:

   # If running LDR in Docker, Ollama on host:
   docker run --add-host=host.docker.internal:host-gateway ...
   

OpenAI API Errors

Symptoms:

• "Invalid API key"
• "Rate limit exceeded"
• "Model not found"

Solutions:

1. Verify API key format:

   • Should start with sk-
   • Check for leading/trailing whitespace

2. Check API key permissions:

   • Ensure key has access to the model you're using
   • Verify organization ID if using org-scoped keys

3. Rate limits:

   • Wait and retry for rate limit errors
   • Consider using a higher tier API key
   • Reduce questions_per_iteration setting

4. Model availability:

   • Verify model name is correct (e.g., gpt-4, not gpt4)
   • Check if model is available in your region

OpenRouter Issues

Symptoms:

• Authentication failures
• Model not available

Solutions:

1. API key format:

   # Settings
   llm.openrouter.api_key = <your-key-here>
   

2. Model naming:

   • Use full model paths: anthropic/claude-3-opus
   • Check available models at openrouter.ai/docs

---

Search Engine Issues

DuckDuckGo Returning No Results

Symptoms:

• Empty search results
• "No results found" consistently

Solutions:

1. Rate limiting: DuckDuckGo aggressively rate limits. Solutions:

   • Switch to SearXNG or another engine
   • Increase wait time in rate limiting settings
   • Use search.rate_limiting.profile = conservative

2. Check network: Verify you can access DuckDuckGo directly

3. Try alternative engines:

   # In settings
   search.tool = "searxng"  # or "brave", "tavily", etc.
   

SearXNG Setup Issues

Symptoms:

• "Connection refused"
• "404 Not Found"

Solutions:

1. Verify SearXNG is running:

   curl http://localhost:8080/search?q=test&format=json
   

2. Check URL configuration:

   • Settings UI: Settings → Search → SearXNG → Instance URL
   • Setting key: search.engine.web.searxng.default_params.instance_url
   • Env var: LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_INSTANCE_URL

3. Ensure JSON format is enabled in SearXNG settings

4. Docker networking: From inside the LDR container, localhost is the container itself, not your host. Use http://searxng:8080 if SearXNG is a sibling service in Docker Compose, or http://host.docker.internal:8080 if SearXNG runs on the host (Mac/Windows/WSL2 — see the Windows/WSL2 FAQ entry for the full recipe).

API Key Issues for Search Engines

Symptoms:

• "API key required"
• "Unauthorized" errors

Solutions:

1. Verify key is set:

   • Check in Settings > Search > [Engine Name]
   • Or via environment variable

2. Engine-specific settings:

   Engine	Setting Key
   Brave	search.engine.brave.api_key
   Tavily	search.engine.tavily.api_key
   Serper	search.engine.serper.api_key
   SerpAPI	search.engine.serpapi.api_key

---

Rate Limiting

"Rate limit exceeded" Errors

Symptoms:

• Searches failing with rate limit errors
• Long waits between searches
• Inconsistent search performance

Solutions:

1. View current rate limit status:

   python -m local_deep_research.web_search_engines.rate_limiting status
   

2. Reset rate limits for an engine:

   python -m local_deep_research.web_search_engines.rate_limiting reset --engine duckduckgo
   

3. Adjust rate limiting profile:

   # Options: conservative, balanced, aggressive
   search.rate_limiting.profile = conservative
   

4. Use the langgraph-agent strategy (the default) to distribute load — it selects engines dynamically per query and can route around rate-limited ones.

Rate Limiting CLI Commands

# View status
python -m local_deep_research.web_search_engines.rate_limiting status
python -m local_deep_research.web_search_engines.rate_limiting status --engine arxiv

# Reset learned rates
python -m local_deep_research.web_search_engines.rate_limiting reset --engine duckduckgo

# Clean old data
python -m local_deep_research.web_search_engines.rate_limiting cleanup --days 30

# Export data
python -m local_deep_research.web_search_engines.rate_limiting export --format csv

---

Database Issues

"Database is locked" Errors

Symptoms:

• SQLite lock errors
• Operations timing out
• Concurrent access failures

This is likely a bug. If you encounter persistent "database is locked" errors, please:

1. Collect logs:

   • Check the application logs for error details
   • Note what action triggered the error

2. Report the issue:

   • Open an issue at GitHub Issues
   • Include the logs and steps to reproduce

Temporary workarounds:

1. Check for zombie processes:

   ps aux | grep python
   # Kill any stuck LDR processes
   

2. Restart the application to release any held locks

Encryption/SQLCipher Issues

Symptoms:

• "file is not a database"
• "database disk image is malformed"
• Cannot open user database

Solutions:

1. Verify SQLCipher is installed:

   pip show sqlcipher3-binary
   

2. Check password/key:

   • User databases are encrypted with derived keys
   • Password changes require re-encryption

3. For corrupted databases:

   • Check ~/.local/share/local-deep-research/users/ for backups
   • Consider creating a new user account

4. Integrity check:

   • Use the /auth/integrity-check endpoint
   • Or run manual SQLite integrity checks

Migration Issues

Symptoms:

• Schema version mismatch
• Missing tables or columns

Solutions:

1. Check version:

   from local_deep_research import __version__
   print(__version__)
   

2. Run migrations (if applicable):

   • Migrations are typically automatic on startup
   • Check logs for migration errors

---

WebSocket/Real-time Updates

Progress Updates Not Showing

Symptoms:

• Research starts but no progress shown
• UI appears stuck
• Results appear suddenly at end

Solutions:

1. Check browser console for WebSocket errors

2. Verify SocketIO connection:

   • Open browser DevTools > Network > WS
   • Look for /socket.io connections

3. Authentication / expired session:

   • WebSocket connections require an authenticated session.
   • If your session has expired, or the server was restarted while your tab was open, the handshake is rejected and the UI may silently fall back to HTTP polling with no progress shown.
   • Log out and back in to restore the connection.

4. Firewall/proxy issues:

   • WebSocket needs persistent connections
   • Some proxies don't support WebSocket
   • Try direct connection (no proxy)

5. Fallback to polling:

   • The client automatically falls back to HTTP polling
   • Check if polling requests are working

Connection Drops

Symptoms:

• Frequent disconnections
• "transport close" errors

Solutions:

1. Check network stability

2. Adjust timeout settings:

   • Default ping timeout: 20 seconds
   • Default ping interval: 5 seconds

3. For reverse proxy setups:

   # Nginx example (see docs/deployment/reverse-proxy.md for the full config)
   location /socket.io {
       proxy_pass http://127.0.0.1:5000;   # not "localhost" — may resolve to ::1 first
       proxy_http_version 1.1;
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       proxy_set_header Host $host;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       # Required when terminating TLS at the proxy: without X-Forwarded-Proto
       # the same-origin WebSocket check (the default) sees http:// and rejects
       # the browser's https Origin. Also needed for secure cookies / HSTS.
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_read_timeout 86400;
   }
   

   See Deploying behind a reverse proxy for the complete nginx/Caddy configuration (uploads, redirects, SSE timeouts).

---

Docker Issues

Port Conflicts

macOS AirPlay (Port 5000)

Symptom: "Address already in use" error or container starts but http://localhost:5000 is unreachable

Diagnose:

lsof -i :5000
sudo lsof -i :5000  # May need sudo for system services
# If "ControlCe" or "AirPlayXPC" appears → AirPlay is the cause

Solutions:

1. Disable AirPlay Receiver (macOS 12 Monterey and later):

   • System Settings → General → AirDrop & Handoff → Toggle OFF "AirPlay Receiver"

2. Use a different port (recommended if you need AirPlay):

   # docker-compose.yml
   ports:
     - "8080:5000"  # Access at http://localhost:8080
   

   Or with Docker CLI:

   docker run -p 8080:5000 ...
   

Note: Other services that may use port 5000 include Flask development servers, Synology DSM, and some VPN software. The diagnostic commands above will help identify the culprit.

---

Container Won't Start

Symptoms:

• Container exits immediately
• "exec format error"
• Port already in use

Solutions:

1. Check logs:

   docker logs local-deep-research
   

2. Port conflicts:

   # Check what's using port 5000
   lsof -i :5000
   
   # Use different port
   docker run -p 8080:5000 ...
   

3. Architecture mismatch:

   • Ensure image matches your CPU architecture (amd64/arm64)

GPU Not Working

Symptoms:

• Ollama running on CPU instead of GPU
• "CUDA not available"

Solutions:

1. Use GPU-specific compose file:

   docker compose -f docker-compose.yml -f docker-compose.gpu.override.yml up
   

2. Verify NVIDIA runtime:

   docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi
   

3. Install nvidia-container-toolkit:

   # Ubuntu/Debian
   sudo apt-get install -y nvidia-container-toolkit
   sudo systemctl restart docker
   

Volume/Permission Issues

Symptoms:

• "Permission denied" errors
• Data not persisting

Solutions:

1. Check volume ownership:

   ls -la ~/.local/share/local-deep-research/
   

2. Fix permissions:

   sudo chown -R $(id -u):$(id -g) ~/.local/share/local-deep-research/
   

---

API Issues

CSRF Token Errors

Symptoms:

• "CSRF token missing"
• "CSRF validation failed"

Solutions:

1. Fetch token before requests:

   # Get CSRF token from server
   resp = session.get("http://localhost:5000/auth/csrf-token")
   csrf = resp.json()["csrf_token"]
   
   # Include in requests
   session.post(
       "http://localhost:5000/api/v1/quick_summary",
       json={"query": "..."},
       headers={"X-CSRFToken": csrf}
   )
   

2. Use the LDRClient which handles CSRF automatically:

   from local_deep_research.api.client import LDRClient
   
   with LDRClient() as client:
       client.login(username, password)
       result = client.quick_research("query")
   

Authentication Failures

Symptoms:

• "Login required"
• Session expires unexpectedly

Solutions:

1. Verify credentials:

   • Username is case-sensitive
   • Check for password special characters

2. Session issues:

   • Clear cookies and re-login
   • Check session timeout settings

3. For API access:

   • Consider using API keys instead of sessions
   • Check api.enabled setting

---

Performance Issues

Slow Research

Symptoms:

• Research taking too long
• High memory usage
• Timeouts

Solutions:

1. Reduce iterations:

   search.iterations = 2  # Instead of default 4
   

2. Reduce questions per iteration:

   search.questions_per_iteration = 3  # Instead of 5
   

3. Use a lighter strategy:

   search.search_strategy = source-based  # Instead of the agentic default
   

4. Limit search results:

   search.max_results = 5  # Instead of 10
   

5. Use snippet-only mode:

   search.snippets_only = true  # Skip full content retrieval
   

Memory Issues

Symptoms:

• Out of memory errors
• System becomes unresponsive

Solutions:

1. Limit concurrent research:

   • Reduce queue size
   • Wait for research to complete before starting new ones

2. Use smaller models:

   • llama3.2:3b instead of larger variants
   • Quantized models (Q4, Q5)

3. Increase swap space (Linux):

   sudo fallocate -l 8G /swapfile
   sudo chmod 600 /swapfile
   sudo mkswap /swapfile
   sudo swapon /swapfile
   

---

Resource Exhaustion

File Descriptor Exhaustion

Symptoms:

• sqlite3.OperationalError: unable to open database file
• OSError: [Errno 24] Too many open files
• Cascading failures across unrelated operations (logging, HTTP requests, WebSocket connections fail simultaneously)

Why it happens:

Each SQLCipher WAL-mode connection uses 2 file descriptors (main db + WAL), plus 1 shared SHM fd per database. With per-user encrypted databases, the QueuePool alone uses users × (pool_size × 2 + 1) FDs at steady state (41 per user with defaults), up to users × ((20 + 40) × 2 + 1) = users × 121 under load. The default Linux soft ulimit of 1024 is tight for multi-user deployments.

Diagnosis:

# Inside Docker (PID 1 is the app due to exec in entrypoint)
ls /proc/1/fd | wc -l
cat /proc/1/limits | grep "open files"

# Bare-metal Linux
ls /proc/$(pgrep -fo ldr-web)/fd | wc -l

# Detailed view — show database-related FDs
lsof -p <PID> | grep -E '\.db|\.wal|\.shm'

Solutions:

1. The app includes automatic credential cleanup (~5 minutes) and periodic pool disposal (every 30 minutes) — this normally handles cleanup transparently
2. Docker: The daemon default FD limit (typically 1M+) is appropriate. Do not set a lower nofile ulimit — this was intentionally removed from docker-compose.yml
3. Bare-metal Linux: The default soft limit of 1024 may be too low. Increase it:

   ulimit -n 65536
   

4. Restart the application to release all file descriptors

For the technical details of the cleanup architecture, see Architecture - Thread & Resource Lifecycle.

---

Debug Logging

Security note: Log files are unencrypted and may contain sensitive information such as research queries. Ensure appropriate file permissions.

Enable File Logging

By default, LDR logs to the console. To enable persistent file logging:

export LDR_ENABLE_FILE_LOGGING=true

Log File Locations

Platform	Path
Linux	~/.local/share/local-deep-research/logs/
macOS	~/Library/Application Support/local-deep-research/logs/
Windows	%USERPROFILE%\AppData\Local\local-deep-research\logs\
Custom	Set LDR_DATA_DIR environment variable

Log files:

• ldr_web.log - Main application log
• Logs rotate at 10MB with 7-day retention (compressed)

Docker Logging

# Live log stream
docker compose logs -f local-deep-research

# Last 100 lines
docker compose logs --tail 100 local-deep-research

# Follow logs with timestamps
docker compose logs -f -t local-deep-research

Verbose File Logging

To capture DEBUG-level output to log files:

export LDR_ENABLE_FILE_LOGGING=true

Log files will include DEBUG-level messages. See log file locations above.

Security note: Log files are unencrypted and may contain sensitive information such as research queries. Ensure appropriate file permissions.

---

Getting Help

If you're still experiencing issues:

1. Check logs:

   • Console output
   • Log files (see Debug Logging above)

2. Search existing issues:

   • GitHub Issues

3. Create a new issue with:

   • LDR version
   • Operating system
   • Docker/native installation
   • Steps to reproduce
   • Relevant logs

---

See Also

• Architecture Overview - System architecture
• FAQ - Frequently asked questions
• Search Engines Guide - Detailed engine documentation
• Architecture - Thread & Resource Lifecycle - Resource cleanup layers and FD budget