learningcircuit--local-deep-research/docs/troubleshooting.md

# Troubleshooting Guide

This guide covers common issues and their solutions.

## Table of Contents

- [LLM Connection Issues](#llm-connection-issues)
- [Search Engine Issues](#search-engine-issues)
- [Rate Limiting](#rate-limiting)
- [Database Issues](#database-issues)
- [WebSocket/Real-time Updates](#websocketreal-time-updates)
- [Docker Issues](#docker-issues)
- [API Issues](#api-issues)
- [Performance Issues](#performance-issues)
- [Resource Exhaustion](#resource-exhaustion)

---

## LLM Connection Issues

### Ollama Not Connecting

**Symptoms:**
- "Failed to connect to Ollama"
- "Connection refused" errors
- Empty responses from LLM

**Solutions:**

1. **Verify Ollama is running:**
   ```bash
   curl http://localhost:11434/api/tags
   ```

2. **Check the URL configuration:**
   - Default: `http://localhost:11434`
   - For Docker: Use `http://host.docker.internal:11434` or your host IP
   - Settings location: `llm.ollama.url`

3. **Verify the model is pulled:**
   ```bash
   ollama list
   ollama pull llama3.2
   ```

4. **Check Docker networking:**
   ```bash
   # If running LDR in Docker, Ollama on host:
   docker run --add-host=host.docker.internal:host-gateway ...
   ```

### OpenAI API Errors

**Symptoms:**
- "Invalid API key"
- "Rate limit exceeded"
- "Model not found"

**Solutions:**

1. **Verify API key format:**
   - Should start with `sk-`
   - Check for leading/trailing whitespace

2. **Check API key permissions:**
   - Ensure key has access to the model you're using
   - Verify organization ID if using org-scoped keys

3. **Rate limits:**
   - Wait and retry for rate limit errors
   - Consider using a higher tier API key
   - Reduce `questions_per_iteration` setting

4. **Model availability:**
   - Verify model name is correct (e.g., `gpt-4`, not `gpt4`)
   - Check if model is available in your region

### OpenRouter Issues

**Symptoms:**
- Authentication failures
- Model not available

**Solutions:**

1. **API key format:**
   ```
   # Settings
   llm.openrouter.api_key = <your-key-here>
   ```

2. **Model naming:**
   - Use full model paths: `anthropic/claude-3-opus`
   - Check available models at openrouter.ai/docs

---

## Search Engine Issues

### DuckDuckGo Returning No Results

**Symptoms:**
- Empty search results
- "No results found" consistently

**Solutions:**

1. **Rate limiting:** DuckDuckGo aggressively rate limits. Solutions:
   - Switch to SearXNG or another engine
   - Increase wait time in rate limiting settings
   - Use `search.rate_limiting.profile = conservative`

2. **Check network:** Verify you can access DuckDuckGo directly

3. **Try alternative engines:**
   ```python
   # In settings
   search.tool = "searxng"  # or "brave", "tavily", etc.
   ```

### SearXNG Setup Issues

**Symptoms:**
- "Connection refused"
- "404 Not Found"

**Solutions:**

1. **Verify SearXNG is running:**
   ```bash
   curl http://localhost:8080/search?q=test&format=json
   ```

2. **Check URL configuration:**
   - Settings UI: **Settings → Search → SearXNG → Instance URL**
   - Setting key: `search.engine.web.searxng.default_params.instance_url`
   - Env var: `LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_INSTANCE_URL`

3. **Ensure JSON format is enabled** in SearXNG settings

4. **Docker networking:** From inside the LDR container, `localhost` is the container itself, not your host. Use `http://searxng:8080` if SearXNG is a sibling service in Docker Compose, or `http://host.docker.internal:8080` if SearXNG runs on the host (Mac/Windows/WSL2 — see the [Windows/WSL2 FAQ entry](faq.md#port-5000-not-accessible-on-windows) for the full recipe).

### API Key Issues for Search Engines

**Symptoms:**
- "API key required"
- "Unauthorized" errors

**Solutions:**

1. **Verify key is set:**
   - Check in Settings > Search > [Engine Name]
   - Or via environment variable

2. **Engine-specific settings:**
   | Engine | Setting Key |
   |--------|-------------|
   | Brave | `search.engine.brave.api_key` |
   | Tavily | `search.engine.tavily.api_key` |
   | Serper | `search.engine.serper.api_key` |
   | SerpAPI | `search.engine.serpapi.api_key` |

---

## Rate Limiting

### "Rate limit exceeded" Errors

**Symptoms:**
- Searches failing with rate limit errors
- Long waits between searches
- Inconsistent search performance

**Solutions:**

1. **View current rate limit status:**
   ```bash
   python -m local_deep_research.web_search_engines.rate_limiting status
   ```

2. **Reset rate limits for an engine:**
   ```bash
   python -m local_deep_research.web_search_engines.rate_limiting reset --engine duckduckgo
   ```

3. **Adjust rate limiting profile:**
   ```
   # Options: conservative, balanced, aggressive
   search.rate_limiting.profile = conservative
   ```

4. **Use the langgraph-agent strategy** (the default) to distribute load — it
   selects engines dynamically per query and can route around rate-limited ones.

### Rate Limiting CLI Commands

```bash
# View status
python -m local_deep_research.web_search_engines.rate_limiting status
python -m local_deep_research.web_search_engines.rate_limiting status --engine arxiv

# Reset learned rates
python -m local_deep_research.web_search_engines.rate_limiting reset --engine duckduckgo

# Clean old data
python -m local_deep_research.web_search_engines.rate_limiting cleanup --days 30

# Export data
python -m local_deep_research.web_search_engines.rate_limiting export --format csv
```

---

## Database Issues

### "Database is locked" Errors

**Symptoms:**
- SQLite lock errors
- Operations timing out
- Concurrent access failures

**This is likely a bug.** If you encounter persistent "database is locked" errors, please:

1. **Collect logs:**
   - Check the application logs for error details
   - Note what action triggered the error

2. **Report the issue:**
   - Open an issue at [GitHub Issues](https://github.com/LearningCircuit/local-deep-research/issues)
   - Include the logs and steps to reproduce

**Temporary workarounds:**

1. **Check for zombie processes:**
   ```bash
   ps aux | grep python
   # Kill any stuck LDR processes
   ```

2. **Restart the application** to release any held locks

### Encryption/SQLCipher Issues

**Symptoms:**
- "file is not a database"
- "database disk image is malformed"
- Cannot open user database

**Solutions:**

1. **Verify SQLCipher is installed:**
   ```bash
   pip show sqlcipher3-binary
   ```

2. **Check password/key:**
   - User databases are encrypted with derived keys
   - Password changes require re-encryption

3. **For corrupted databases:**
   - Check `~/.local/share/local-deep-research/users/` for backups
   - Consider creating a new user account

4. **Integrity check:**
   - Use the `/auth/integrity-check` endpoint
   - Or run manual SQLite integrity checks

### Migration Issues

**Symptoms:**
- Schema version mismatch
- Missing tables or columns

**Solutions:**

1. **Check version:**
   ```python
   from local_deep_research import __version__
   print(__version__)
   ```

2. **Run migrations** (if applicable):
   - Migrations are typically automatic on startup
   - Check logs for migration errors

---

## WebSocket/Real-time Updates

### Progress Updates Not Showing

**Symptoms:**
- Research starts but no progress shown
- UI appears stuck
- Results appear suddenly at end

**Solutions:**

1. **Check browser console** for WebSocket errors

2. **Verify SocketIO connection:**
   - Open browser DevTools > Network > WS
   - Look for `/socket.io` connections

3. **Authentication / expired session:**
   - WebSocket connections require an authenticated session.
   - If your session has expired, or the server was restarted while your tab was open, the handshake is rejected and the UI may silently fall back to HTTP polling with no progress shown.
   - Log out and back in to restore the connection.

4. **Firewall/proxy issues:**
   - WebSocket needs persistent connections
   - Some proxies don't support WebSocket
   - Try direct connection (no proxy)

5. **Fallback to polling:**
   - The client automatically falls back to HTTP polling
   - Check if polling requests are working

### Connection Drops

**Symptoms:**
- Frequent disconnections
- "transport close" errors

**Solutions:**

1. **Check network stability**

2. **Adjust timeout settings:**
   - Default ping timeout: 20 seconds
   - Default ping interval: 5 seconds

3. **For reverse proxy setups:**
   ```nginx
   # Nginx example (see docs/deployment/reverse-proxy.md for the full config)
   location /socket.io {
       proxy_pass http://127.0.0.1:5000;   # not "localhost" — may resolve to ::1 first
       proxy_http_version 1.1;
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       proxy_set_header Host $host;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       # Required when terminating TLS at the proxy: without X-Forwarded-Proto
       # the same-origin WebSocket check (the default) sees http:// and rejects
       # the browser's https Origin. Also needed for secure cookies / HSTS.
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_read_timeout 86400;
   }
   ```

   See [Deploying behind a reverse proxy](deployment/reverse-proxy.md) for the
   complete nginx/Caddy configuration (uploads, redirects, SSE timeouts).

---

## Docker Issues

### Port Conflicts

#### macOS AirPlay (Port 5000)

**Symptom:** "Address already in use" error or container starts but http://localhost:5000 is unreachable

**Diagnose:**
```bash
lsof -i :5000
sudo lsof -i :5000  # May need sudo for system services
# If "ControlCe" or "AirPlayXPC" appears → AirPlay is the cause
```

**Solutions:**

1. **Disable AirPlay Receiver** (macOS 12 Monterey and later):
   - System Settings → General → AirDrop & Handoff → Toggle OFF "AirPlay Receiver"

2. **Use a different port** (recommended if you need AirPlay):
   ```yaml
   # docker-compose.yml
   ports:
     - "8080:5000"  # Access at http://localhost:8080
   ```

   Or with Docker CLI:
   ```bash
   docker run -p 8080:5000 ...
   ```

**Note:** Other services that may use port 5000 include Flask development servers, Synology DSM, and some VPN software. The diagnostic commands above will help identify the culprit.

---

### Container Won't Start

**Symptoms:**
- Container exits immediately
- "exec format error"
- Port already in use

**Solutions:**

1. **Check logs:**
   ```bash
   docker logs local-deep-research
   ```

2. **Port conflicts:**
   ```bash
   # Check what's using port 5000
   lsof -i :5000

   # Use different port
   docker run -p 8080:5000 ...
   ```

3. **Architecture mismatch:**
   - Ensure image matches your CPU architecture (amd64/arm64)

### GPU Not Working

**Symptoms:**
- Ollama running on CPU instead of GPU
- "CUDA not available"

**Solutions:**

1. **Use GPU-specific compose file:**
   ```bash
   docker compose -f docker-compose.yml -f docker-compose.gpu.override.yml up
   ```

2. **Verify NVIDIA runtime:**
   ```bash
   docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi
   ```

3. **Install nvidia-container-toolkit:**
   ```bash
   # Ubuntu/Debian
   sudo apt-get install -y nvidia-container-toolkit
   sudo systemctl restart docker
   ```

### Volume/Permission Issues

**Symptoms:**
- "Permission denied" errors
- Data not persisting

**Solutions:**

1. **Check volume ownership:**
   ```bash
   ls -la ~/.local/share/local-deep-research/
   ```

2. **Fix permissions:**
   ```bash
   sudo chown -R $(id -u):$(id -g) ~/.local/share/local-deep-research/
   ```

---

## API Issues

### CSRF Token Errors

**Symptoms:**
- "CSRF token missing"
- "CSRF validation failed"

**Solutions:**

1. **Fetch token before requests:**
   ```python
   # Get CSRF token from server
   resp = session.get("http://localhost:5000/auth/csrf-token")
   csrf = resp.json()["csrf_token"]

   # Include in requests
   session.post(
       "http://localhost:5000/api/v1/quick_summary",
       json={"query": "..."},
       headers={"X-CSRFToken": csrf}
   )
   ```

2. **Use the LDRClient** which handles CSRF automatically:
   ```python
   from local_deep_research.api.client import LDRClient

   with LDRClient() as client:
       client.login(username, password)
       result = client.quick_research("query")
   ```

### Authentication Failures

**Symptoms:**
- "Login required"
- Session expires unexpectedly

**Solutions:**

1. **Verify credentials:**
   - Username is case-sensitive
   - Check for password special characters

2. **Session issues:**
   - Clear cookies and re-login
   - Check session timeout settings

3. **For API access:**
   - Consider using API keys instead of sessions
   - Check `api.enabled` setting

---

## Performance Issues

### Slow Research

**Symptoms:**
- Research taking too long
- High memory usage
- Timeouts

**Solutions:**

1. **Reduce iterations:**
   ```
   search.iterations = 2  # Instead of default 4
   ```

2. **Reduce questions per iteration:**
   ```
   search.questions_per_iteration = 3  # Instead of 5
   ```

3. **Use a lighter strategy:**
   ```
   search.search_strategy = source-based  # Instead of the agentic default
   ```

4. **Limit search results:**
   ```
   search.max_results = 5  # Instead of 10
   ```

5. **Use snippet-only mode:**
   ```
   search.snippets_only = true  # Skip full content retrieval
   ```

### Memory Issues

**Symptoms:**
- Out of memory errors
- System becomes unresponsive

**Solutions:**

1. **Limit concurrent research:**
   - Reduce queue size
   - Wait for research to complete before starting new ones

2. **Use smaller models:**
   - `llama3.2:3b` instead of larger variants
   - Quantized models (Q4, Q5)

3. **Increase swap space** (Linux):
   ```bash
   sudo fallocate -l 8G /swapfile
   sudo chmod 600 /swapfile
   sudo mkswap /swapfile
   sudo swapon /swapfile
   ```

---

## Resource Exhaustion

### File Descriptor Exhaustion

**Symptoms:**
- `sqlite3.OperationalError: unable to open database file`
- `OSError: [Errno 24] Too many open files`
- Cascading failures across unrelated operations (logging, HTTP requests, WebSocket connections fail simultaneously)

**Why it happens:**

Each SQLCipher WAL-mode connection uses 2 file descriptors (main db + WAL), plus 1 shared SHM fd per database. With per-user encrypted databases, the QueuePool alone uses `users × (pool_size × 2 + 1)` FDs at steady state (41 per user with defaults), up to `users × ((20 + 40) × 2 + 1) = users × 121` under load. The default Linux soft ulimit of 1024 is tight for multi-user deployments.

**Diagnosis:**

```bash
# Inside Docker (PID 1 is the app due to exec in entrypoint)
ls /proc/1/fd | wc -l
cat /proc/1/limits | grep "open files"

# Bare-metal Linux
ls /proc/$(pgrep -fo ldr-web)/fd | wc -l

# Detailed view — show database-related FDs
lsof -p <PID> | grep -E '\.db|\.wal|\.shm'
```

**Solutions:**

1. The app includes automatic credential cleanup (~5 minutes) and periodic pool disposal (every 30 minutes) — this normally handles cleanup transparently
2. **Docker:** The daemon default FD limit (typically 1M+) is appropriate. Do not set a lower `nofile` ulimit — this was intentionally removed from `docker-compose.yml`
3. **Bare-metal Linux:** The default soft limit of 1024 may be too low. Increase it:
   ```bash
   ulimit -n 65536
   ```
4. Restart the application to release all file descriptors

For the technical details of the cleanup architecture, see [Architecture - Thread & Resource Lifecycle](./architecture.md#thread--resource-lifecycle).

---

## Debug Logging

> **Security note:** Log files are unencrypted and may contain sensitive information such as research queries. Ensure appropriate file permissions.

### Enable File Logging

By default, LDR logs to the console. To enable persistent file logging:

```bash
export LDR_ENABLE_FILE_LOGGING=true
```

### Log File Locations

| Platform | Path |
|----------|------|
| Linux | `~/.local/share/local-deep-research/logs/` |
| macOS | `~/Library/Application Support/local-deep-research/logs/` |
| Windows | `%USERPROFILE%\AppData\Local\local-deep-research\logs\` |
| Custom | Set `LDR_DATA_DIR` environment variable |

**Log files:**
- `ldr_web.log` - Main application log
- Logs rotate at 10MB with 7-day retention (compressed)

### Docker Logging

```bash
# Live log stream
docker compose logs -f local-deep-research

# Last 100 lines
docker compose logs --tail 100 local-deep-research

# Follow logs with timestamps
docker compose logs -f -t local-deep-research
```

### Verbose File Logging

To capture DEBUG-level output to log files:

```bash
export LDR_ENABLE_FILE_LOGGING=true
```

Log files will include DEBUG-level messages. See log file locations above.

**Security note:** Log files are unencrypted and may contain sensitive information such as research queries. Ensure appropriate file permissions.

---

## Getting Help

If you're still experiencing issues:

1. **Check logs:**
   - Console output
   - Log files (see [Debug Logging](#debug-logging) above)

2. **Search existing issues:**
   - [GitHub Issues](https://github.com/LearningCircuit/local-deep-research/issues)

3. **Create a new issue** with:
   - LDR version
   - Operating system
   - Docker/native installation
   - Steps to reproduce
   - Relevant logs

---

## See Also

- [Architecture Overview](./architecture/OVERVIEW.md) - System architecture
- [FAQ](./faq.md) - Frequently asked questions
- [Search Engines Guide](./search-engines.md) - Detailed engine documentation
- [Architecture - Thread & Resource Lifecycle](./architecture.md#thread--resource-lifecycle) - Resource cleanup layers and FD budget