chore: import upstream snapshot with attribution
OSV-Scanner (Scheduled) / scan-scheduled (push) Failing after 0s
Create Release / test-gate (push) Has been cancelled
Create Release / release-gate (push) Has been cancelled
Create Release / ci-gate (push) Has been cancelled
Create Release / version-check (push) Has been cancelled
Create Release / e2e-test-gate (push) Has been cancelled
Create Release / responsive-test-gate (push) Has been cancelled
Create Release / compat-test-gate (push) Has been cancelled
Create Release / compose-integration-gate (push) Has been cancelled
Create Release / vulture-gate (push) Has been cancelled
Create Release / build (push) Has been cancelled
Create Release / provenance (push) Has been cancelled
Create Release / prerelease-docker (push) Has been cancelled
Create Release / publish-docker (push) Has been cancelled
Create Release / create-release (push) Has been cancelled
Create Release / cleanup-changelog (push) Has been cancelled
Create Release / trigger-pypi (push) Has been cancelled
Create Release / monitor-pypi (push) Has been cancelled
Create Release / Clean up orphan prerelease tags and signatures (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-form] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-metrics] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-workflow] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [settings-core] (push) Has been cancelled
CodeQL Advanced / Analyze (javascript-typescript) (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [history-news] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [library] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [link-analytics] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [chat-core] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [chat-lifecycle] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [error-benchmark] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [settings-pages] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) (push) Has been cancelled
Docker Tests (Consolidated) / Accessibility Tests (push) Has been cancelled
Docker Tests (Consolidated) / LLM Unit Tests (push) Has been cancelled
Docker Tests (Consolidated) / LLM Example Tests (push) Has been cancelled
Docker Tests (Consolidated) / Production Image Smoke Test (push) Has been cancelled
Docker Tests (Consolidated) / Infrastructure Tests (push) Has been cancelled
OSSF Scorecard / OSSF Security Scorecard Analysis (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [mobile] (push) Has been cancelled
Backwards Compatibility / Verify Encryption Constants (push) Has been cancelled
Backwards Compatibility / PyPI Version Compatibility (push) Has been cancelled
Backwards Compatibility / Database Migration Tests (push) Has been cancelled
CodeQL Advanced / Analyze (python) (push) Has been cancelled
Docker Tests (Consolidated) / detect-changes (push) Has been cancelled
Docker Tests (Consolidated) / Build Test Image (push) Has been cancelled
Docker Tests (Consolidated) / All Pytest Tests + Coverage (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [accessibility] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [api-crud] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-login] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-pages] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-register] (push) Has been cancelled
OSV-Scanner (Scheduled) / scan-scheduled (push) Failing after 0s
Create Release / test-gate (push) Has been cancelled
Create Release / release-gate (push) Has been cancelled
Create Release / ci-gate (push) Has been cancelled
Create Release / version-check (push) Has been cancelled
Create Release / e2e-test-gate (push) Has been cancelled
Create Release / responsive-test-gate (push) Has been cancelled
Create Release / compat-test-gate (push) Has been cancelled
Create Release / compose-integration-gate (push) Has been cancelled
Create Release / vulture-gate (push) Has been cancelled
Create Release / build (push) Has been cancelled
Create Release / provenance (push) Has been cancelled
Create Release / prerelease-docker (push) Has been cancelled
Create Release / publish-docker (push) Has been cancelled
Create Release / create-release (push) Has been cancelled
Create Release / cleanup-changelog (push) Has been cancelled
Create Release / trigger-pypi (push) Has been cancelled
Create Release / monitor-pypi (push) Has been cancelled
Create Release / Clean up orphan prerelease tags and signatures (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-form] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-metrics] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-workflow] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [settings-core] (push) Has been cancelled
CodeQL Advanced / Analyze (javascript-typescript) (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [history-news] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [library] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [link-analytics] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [chat-core] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [chat-lifecycle] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [error-benchmark] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [settings-pages] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) (push) Has been cancelled
Docker Tests (Consolidated) / Accessibility Tests (push) Has been cancelled
Docker Tests (Consolidated) / LLM Unit Tests (push) Has been cancelled
Docker Tests (Consolidated) / LLM Example Tests (push) Has been cancelled
Docker Tests (Consolidated) / Production Image Smoke Test (push) Has been cancelled
Docker Tests (Consolidated) / Infrastructure Tests (push) Has been cancelled
OSSF Scorecard / OSSF Security Scorecard Analysis (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [mobile] (push) Has been cancelled
Backwards Compatibility / Verify Encryption Constants (push) Has been cancelled
Backwards Compatibility / PyPI Version Compatibility (push) Has been cancelled
Backwards Compatibility / Database Migration Tests (push) Has been cancelled
CodeQL Advanced / Analyze (python) (push) Has been cancelled
Docker Tests (Consolidated) / detect-changes (push) Has been cancelled
Docker Tests (Consolidated) / Build Test Image (push) Has been cancelled
Docker Tests (Consolidated) / All Pytest Tests + Coverage (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [accessibility] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [api-crud] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-login] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-pages] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-register] (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,199 @@
|
||||
# Local Deep Research API Examples
|
||||
|
||||
This directory contains examples for using LDR through different interfaces.
|
||||
|
||||
## Important: Authentication Required (v2.0+)
|
||||
|
||||
Since LDR v2.0, all API access requires authentication due to per-user encrypted databases. You must:
|
||||
|
||||
1. Create a user account through the web interface
|
||||
2. Authenticate before making API calls
|
||||
3. Pass settings_snapshot for programmatic access
|
||||
|
||||
## Directory Structure
|
||||
|
||||
- **`programmatic/`** - Direct Python API usage (import from `local_deep_research.api`)
|
||||
- `programmatic_access.ipynb` - Jupyter notebook with comprehensive examples
|
||||
- `retriever_usage_example.py` - Using LangChain retrievers with LDR
|
||||
|
||||
- **`http/`** - HTTP REST API usage (requires running server)
|
||||
- `simple_working_example.py` - ✅ **BEST WORKING EXAMPLE** - Clean, tested, and ready to use
|
||||
- `simple_http_example.py` - Quick start example (needs updating for auth)
|
||||
- `http_api_examples.py` - Comprehensive examples including batch processing
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Programmatic API (Python Package)
|
||||
|
||||
```python
|
||||
from local_deep_research.api import quick_summary
|
||||
from local_deep_research.settings import SettingsManager
|
||||
from local_deep_research.database.session_context import get_user_db_session
|
||||
|
||||
# Authenticate and get settings
|
||||
with get_user_db_session(username="your_username", password="your_password") as session:
|
||||
settings_manager = SettingsManager(session)
|
||||
settings_snapshot = settings_manager.get_all_settings()
|
||||
|
||||
# Use the API
|
||||
result = quick_summary(
|
||||
"What is quantum computing?",
|
||||
settings_snapshot=settings_snapshot
|
||||
)
|
||||
print(result["summary"])
|
||||
```
|
||||
|
||||
### HTTP API (REST)
|
||||
|
||||
**🎯 Quick Start - Works Completely Out of the Box!**
|
||||
|
||||
Our tested working example requires zero manual setup:
|
||||
|
||||
```bash
|
||||
# 1. Start the server
|
||||
python -m local_deep_research.web.app
|
||||
|
||||
# 2. Run the working example (creates user automatically!)
|
||||
python examples/api_usage/http/simple_working_example.py
|
||||
|
||||
# 3. Done! ✅ No other steps required
|
||||
```
|
||||
|
||||
The example will:
|
||||
- ✅ Create a unique test user automatically
|
||||
- ✅ Test authentication with proper CSRF handling
|
||||
- ✅ Execute a research query using the correct API endpoint
|
||||
- ✅ Provide credentials for manual testing (if desired)
|
||||
- ✅ Show results with direct links to view them
|
||||
|
||||
**📋 Manual API Usage:**
|
||||
|
||||
If you want to integrate the API into your own code:
|
||||
|
||||
```python
|
||||
import requests
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
# Create session for cookie persistence
|
||||
session = requests.Session()
|
||||
|
||||
# Login - get CSRF token first
|
||||
login_page = session.get("http://localhost:5000/auth/login")
|
||||
soup = BeautifulSoup(login_page.text, 'html.parser')
|
||||
csrf_input = soup.find('input', {'name': 'csrf_token'})
|
||||
login_csrf = csrf_input.get('value')
|
||||
|
||||
# Login with form data
|
||||
session.post(
|
||||
"http://localhost:5000/auth/login",
|
||||
data={
|
||||
"username": "your_username",
|
||||
"password": "your_password",
|
||||
"csrf_token": login_csrf
|
||||
}
|
||||
)
|
||||
|
||||
# Get CSRF token
|
||||
csrf_token = session.get("http://localhost:5000/auth/csrf-token").json()["csrf_token"]
|
||||
|
||||
# Make API request
|
||||
response = session.post(
|
||||
"http://localhost:5000/api/start_research",
|
||||
json={"query": "What is quantum computing?"},
|
||||
headers={"X-CSRF-Token": csrf_token, "Content-Type": "application/json"}
|
||||
)
|
||||
print(response.json())
|
||||
```
|
||||
|
||||
**⚠️ Important Notes:**
|
||||
- Use the correct endpoint: `/api/start_research` (not `/research/api/start`)
|
||||
- Login with form data (not JSON)
|
||||
- Handle CSRF tokens properly
|
||||
- User must be created through web interface first
|
||||
|
||||
## Which API Should I Use?
|
||||
|
||||
- **Programmatic API**: Use when integrating LDR into your Python application
|
||||
- ✅ Direct access, no HTTP overhead
|
||||
- ✅ Full access to all features and parameters
|
||||
- ✅ Can pass Python objects (like LangChain retrievers)
|
||||
- ❌ Requires LDR to be installed in your environment
|
||||
- ❌ Requires database session and settings snapshot
|
||||
|
||||
- **HTTP API**: Use when accessing LDR from other languages or remote systems
|
||||
- ✅ Language agnostic - works with any HTTP client
|
||||
- ✅ Can run LDR on a separate server
|
||||
- ✅ Easy to scale and deploy
|
||||
- ❌ Limited to JSON-serializable parameters
|
||||
- ❌ Requires running the web server
|
||||
- ❌ Requires authentication and CSRF tokens
|
||||
|
||||
## API Changes in v2.0
|
||||
|
||||
### Breaking Changes
|
||||
|
||||
1. **Authentication Required**: All endpoints now require login
|
||||
2. **Settings Snapshot**: Programmatic API needs `settings_snapshot` parameter
|
||||
3. **New Endpoints**: API routes moved (e.g., `/api/v1/quick_summary` → `/api/start_research`)
|
||||
4. **CSRF Protection**: POST/PUT/DELETE requests need CSRF token
|
||||
|
||||
### Migration Guide
|
||||
|
||||
#### Old (v1.x):
|
||||
```python
|
||||
# Programmatic
|
||||
from local_deep_research.api import quick_summary
|
||||
result = quick_summary("query")
|
||||
|
||||
# HTTP
|
||||
curl -X POST http://localhost:5000/api/v1/quick_summary \
|
||||
-d '{"query": "test"}'
|
||||
```
|
||||
|
||||
#### New (v2.0+):
|
||||
```python
|
||||
# Programmatic - with authentication and settings
|
||||
with get_user_db_session(username, password) as session:
|
||||
settings_manager = SettingsManager(session)
|
||||
settings_snapshot = settings_manager.get_all_settings()
|
||||
result = quick_summary("query", settings_snapshot=settings_snapshot)
|
||||
|
||||
# HTTP - with authentication and CSRF
|
||||
# See examples above
|
||||
```
|
||||
|
||||
## Running the Examples
|
||||
|
||||
### Prerequisites
|
||||
|
||||
1. Install LDR: `pip install local-deep-research`
|
||||
2. Create a user account:
|
||||
- Start server: `python -m local_deep_research.web.app`
|
||||
- Open http://localhost:5000 and register
|
||||
3. Configure your LLM provider in settings
|
||||
|
||||
### Programmatic Examples
|
||||
```bash
|
||||
# Update credentials in the example files first!
|
||||
python examples/api_usage/programmatic/retriever_usage_example.py
|
||||
|
||||
# Or use the Jupyter notebook
|
||||
jupyter notebook examples/api_usage/programmatic/programmatic_access.ipynb
|
||||
```
|
||||
|
||||
### HTTP Examples
|
||||
```bash
|
||||
# First, start the LDR server
|
||||
python -m local_deep_research.web.app
|
||||
|
||||
# In another terminal, run the examples
|
||||
# Note: These need to be updated for v2.0 authentication!
|
||||
python examples/api_usage/http/simple_http_example.py
|
||||
python examples/api_usage/http/http_api_examples.py
|
||||
```
|
||||
|
||||
## Need Help?
|
||||
|
||||
- See the [API Quick Start Guide](../../docs/api-quickstart.md)
|
||||
- Check the [FAQ](../../docs/faq.md)
|
||||
- Join our [Discord](https://discord.gg/ttcqQeFcJ3) for support
|
||||
@@ -0,0 +1,65 @@
|
||||
# Important: Examples Updated for LDR v1.0
|
||||
|
||||
## Authentication Required
|
||||
|
||||
Starting with LDR v1.0, all API access requires authentication due to the new per-user encrypted database architecture.
|
||||
|
||||
## Updated Examples
|
||||
|
||||
The following examples have been updated for v1.0:
|
||||
|
||||
### ✅ Updated Examples:
|
||||
- `http/simple_http_example.py` - Basic HTTP API usage with authentication
|
||||
- `http/http_api_examples.py` - Comprehensive HTTP API examples with LDRClient class
|
||||
- `programmatic/retriever_usage_example.py` - LangChain retriever integration with auth
|
||||
- `programmatic/programmatic_access_v1.py` - NEW: Complete programmatic API examples
|
||||
|
||||
### ⚠️ Needs Manual Update:
|
||||
- `programmatic/programmatic_access.ipynb` - Jupyter notebook (see programmatic_access_v1.py for reference)
|
||||
|
||||
## Quick Migration Guide
|
||||
|
||||
### Old Code (pre-v1.0):
|
||||
```python
|
||||
from local_deep_research.api import quick_summary
|
||||
result = quick_summary("query")
|
||||
```
|
||||
|
||||
### New Code (v1.0+):
|
||||
```python
|
||||
from local_deep_research.api import quick_summary
|
||||
from local_deep_research.settings import SettingsManager
|
||||
from local_deep_research.database.session_context import get_user_db_session
|
||||
|
||||
with get_user_db_session(username="user", password="pass") as session:
|
||||
settings_manager = SettingsManager(session)
|
||||
settings_snapshot = settings_manager.get_all_settings()
|
||||
result = quick_summary("query", settings_snapshot=settings_snapshot)
|
||||
```
|
||||
|
||||
## Before Running Examples
|
||||
|
||||
1. **Create an account**:
|
||||
```bash
|
||||
python -m local_deep_research.web.app
|
||||
# Open http://localhost:5000 and register
|
||||
```
|
||||
|
||||
2. **Configure LLM provider** in Settings (e.g., OpenAI, Anthropic, Ollama)
|
||||
|
||||
3. **Update credentials** in the example files:
|
||||
- Change `USERNAME = "your_username"` to your actual username
|
||||
- Change `PASSWORD = "your_password"` to your actual password
|
||||
|
||||
## Common Issues
|
||||
|
||||
- **"No settings context available"**: Pass `settings_snapshot` to API functions
|
||||
- **"Encrypted database requires password"**: Use `get_user_db_session()` with credentials
|
||||
- **"CSRF token missing"**: Get CSRF token before POST/PUT/DELETE requests
|
||||
- **404 errors**: Check new endpoint paths (e.g., `/api/start_research`)
|
||||
|
||||
## Need Help?
|
||||
|
||||
- See [Migration Guide](../../docs/MIGRATION_GUIDE_v1.md) for detailed changes
|
||||
- Check [API Quick Start](../../docs/api-quickstart.md) for authentication details
|
||||
- Join our [Discord](https://discord.gg/ttcqQeFcJ3) for support
|
||||
@@ -0,0 +1,202 @@
|
||||
# HTTP API Examples
|
||||
|
||||
This directory contains working examples for using the LDR HTTP API with authentication.
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### 1. Start the LDR Server
|
||||
|
||||
```bash
|
||||
# Option 1: Direct startup
|
||||
python -m local_deep_research.web.app
|
||||
|
||||
# Option 2: Use the restart script (recommended)
|
||||
bash scripts/dev/restart_server.sh
|
||||
|
||||
# Option 3: Docker compose
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
### 2. Run the Simple Working Example
|
||||
|
||||
```bash
|
||||
# This example works completely out of the box!
|
||||
python simple_working_example.py
|
||||
```
|
||||
|
||||
## 📁 Available Examples
|
||||
|
||||
### 🎯 `simple_working_example.py` - **RECOMMENDED START**
|
||||
- ✅ **Works completely out of the box**
|
||||
- ✅ **Automatic user creation** (no manual setup needed)
|
||||
- ✅ **Correct API endpoints** and authentication
|
||||
- ✅ **Tested and verified** to work
|
||||
- ⏱️ **Runtime:** 2-10 minutes (research processing time)
|
||||
|
||||
**Perfect for:** First-time users, testing if API works, quick demos
|
||||
|
||||
## 📚 Advanced Examples (`advanced/` folder)
|
||||
|
||||
More comprehensive examples for learning and advanced use cases:
|
||||
|
||||
### 📚 `advanced/simple_http_example.py` - **COMPREHENSIVE GUIDE**
|
||||
- ✅ **Automatic user creation**
|
||||
- 📊 **Multiple API examples** (research, settings, history)
|
||||
- 🔍 **Progress monitoring** with status updates
|
||||
- ⏱️ **Runtime:** 3-15 minutes (more comprehensive testing)
|
||||
|
||||
**Perfect for:** Learning different API endpoints, understanding the full API surface
|
||||
|
||||
### 🚀 `advanced/http_api_examples.py` - **ADVANCED CLIENT**
|
||||
- 🔧 **Reusable client class** for integration
|
||||
- 📈 **Advanced features** (batch processing, polling)
|
||||
- 🎛️ **Comprehensive patterns** for production use
|
||||
- ⏱️ **Runtime:** 5-30 minutes (extensive testing)
|
||||
|
||||
**Perfect for:** Building applications, production integration, advanced use cases
|
||||
|
||||
|
||||
## ⚙️ Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
You can configure the LDR service endpoints using environment variables:
|
||||
|
||||
```bash
|
||||
# For local Ollama (default)
|
||||
export LDR_LLM_OLLAMA_URL=http://localhost:11434
|
||||
|
||||
# For remote Ollama server
|
||||
export LDR_LLM_OLLAMA_URL=http://192.168.178.66:11434
|
||||
|
||||
# For Docker compose service names
|
||||
export LDR_LLM_OLLAMA_URL=http://ollama:11434
|
||||
|
||||
# For Docker with host networking
|
||||
export LDR_LLM_OLLAMA_URL=http://host.docker.internal:11434
|
||||
```
|
||||
|
||||
### Docker Compose
|
||||
|
||||
In your `docker-compose.yml`, you can set the Ollama URL:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
ldr:
|
||||
environment:
|
||||
# For service name (recommended for docker-compose)
|
||||
- LDR_LLM_OLLAMA_URL=http://ollama:11434
|
||||
|
||||
# For remote Ollama instance
|
||||
# - LDR_LLM_OLLAMA_URL=http://192.168.178.66:11434
|
||||
|
||||
# For host machine Ollama
|
||||
# - LDR_LLM_OLLAMA_URL=http://host.docker.internal:11434
|
||||
```
|
||||
|
||||
### Common Network Scenarios
|
||||
|
||||
| Scenario | Environment Variable | When to Use |
|
||||
|----------|---------------------|-------------|
|
||||
| **Local Ollama** | `http://localhost:11434` | Running Ollama on same machine |
|
||||
| **Remote Ollama** | `http://IP:11434` | Ollama on different server |
|
||||
| **Docker Compose** | `http://ollama:11434` | Using docker-compose service names |
|
||||
| **Docker Host** | `http://host.docker.internal:11434` | Docker container accessing host Ollama |
|
||||
|
||||
## 🔍 Monitoring Progress
|
||||
|
||||
### Server Logs
|
||||
```bash
|
||||
# Monitor real-time progress
|
||||
tail -f /tmp/ldr_server_5000.log
|
||||
|
||||
# Check recent logs
|
||||
tail -20 /tmp/ldr_server_5000.log
|
||||
```
|
||||
|
||||
### Web Interface
|
||||
- **Research Results:** http://localhost:5000/results/{research_id}
|
||||
- **Settings:** http://localhost:5000/settings
|
||||
- **History:** http://localhost:5000/history
|
||||
|
||||
## 🚨 Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**❌ "Cannot connect to server"**
|
||||
```bash
|
||||
# Start the server first
|
||||
python -m local_deep_research.web.app
|
||||
# or
|
||||
bash scripts/dev/restart_server.sh
|
||||
```
|
||||
|
||||
**❌ "Authentication failed"**
|
||||
- The examples create users automatically, so this shouldn't happen
|
||||
- If it does, check that the server is running correctly
|
||||
|
||||
**❌ "Research failed"**
|
||||
```bash
|
||||
# Check server logs for details
|
||||
tail -f /tmp/ldr_server_5000.log
|
||||
|
||||
# Common issues:
|
||||
# - Ollama not running or wrong URL
|
||||
# - Model not available in Ollama
|
||||
# - Network connectivity issues
|
||||
```
|
||||
|
||||
**❌ "No output from script"**
|
||||
- Scripts may take 2-10 minutes to complete research
|
||||
- Monitor progress in server logs
|
||||
- Check if research started successfully
|
||||
|
||||
### Model Configuration
|
||||
|
||||
Make sure your Ollama has the required models:
|
||||
|
||||
```bash
|
||||
# List available models
|
||||
ollama list
|
||||
|
||||
# Pull a model if needed
|
||||
ollama pull gemma3:12b
|
||||
ollama pull llama3
|
||||
ollama pull mistral
|
||||
```
|
||||
|
||||
## 📚 What Each Example Demonstrates
|
||||
|
||||
### simple_working_example.py
|
||||
- ✅ User creation and authentication
|
||||
- ✅ Basic research request
|
||||
- ✅ Proper CSRF token handling
|
||||
- ✅ Result URL generation
|
||||
|
||||
### advanced/simple_http_example.py
|
||||
- ✅ All of the above PLUS:
|
||||
- ✅ Settings management
|
||||
- ✅ Research history
|
||||
- ✅ Progress polling
|
||||
- ✅ Multiple research examples
|
||||
|
||||
### advanced/http_api_examples.py
|
||||
- ✅ All of the above PLUS:
|
||||
- ✅ Batch processing
|
||||
- ✅ Advanced polling strategies
|
||||
- ✅ Error handling patterns
|
||||
- ✅ Production-ready client class
|
||||
|
||||
|
||||
## 🎯 Recommended Usage Path
|
||||
|
||||
1. **Start with `simple_working_example.py`** - Verify everything works
|
||||
2. **Try `advanced/simple_http_example.py`** - Learn the API surface
|
||||
3. **Use `advanced/http_api_examples.py`** - Build your application
|
||||
|
||||
## 🔗 Related Documentation
|
||||
|
||||
- [Main API Documentation](../README.md)
|
||||
- [API Quick Start](../../../docs/api-quickstart.md)
|
||||
- [Docker Configuration](../../../docker-compose.yml)
|
||||
- [Troubleshooting Guide](../../../docs/troubleshooting.md)
|
||||
@@ -0,0 +1,67 @@
|
||||
# Advanced HTTP API Examples
|
||||
|
||||
This folder contains more comprehensive HTTP API examples for learning advanced features and production use cases.
|
||||
|
||||
## 📁 Available Examples
|
||||
|
||||
### 📚 `simple_http_example.py` - **COMPREHENSIVE GUIDE**
|
||||
- ✅ **Automatic user creation**
|
||||
- 📊 **Multiple API examples** (research, settings, history)
|
||||
- 🔍 **Progress monitoring** with status updates
|
||||
- ⏱️ **Runtime:** 3-15 minutes (more comprehensive testing)
|
||||
|
||||
**Perfect for:** Learning different API endpoints, understanding the full API surface
|
||||
|
||||
### 🚀 `http_api_examples.py` - **ADVANCED CLIENT**
|
||||
- 🔧 **Reusable client class** for integration
|
||||
- 📈 **Advanced features** (batch processing, polling)
|
||||
- 🎛️ **Comprehensive patterns** for production use
|
||||
- ⏱️ **Runtime:** 5-30 minutes (extensive testing)
|
||||
|
||||
**Perfect for:** Building applications, production integration, advanced use cases
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### 1. Run the Comprehensive Example
|
||||
```bash
|
||||
cd advanced
|
||||
python simple_http_example.py
|
||||
```
|
||||
|
||||
### 2. Try the Advanced Client
|
||||
```bash
|
||||
cd advanced
|
||||
python http_api_examples.py
|
||||
```
|
||||
|
||||
## ⚠️ Important Notes
|
||||
|
||||
- **Longer Runtime**: These examples take longer than the basic example
|
||||
- **More Features**: They demonstrate additional API endpoints and patterns
|
||||
- **Learning Focused**: Designed to help you understand the full API surface
|
||||
- **Production Ready**: Advanced examples include patterns for production use
|
||||
|
||||
## 📚 What These Examples Demonstrate
|
||||
|
||||
### simple_http_example.py
|
||||
- ✅ All basic functionality from the main example
|
||||
- ✅ Settings management (get/update configuration)
|
||||
- ✅ Research history access
|
||||
- ✅ Progress polling and monitoring
|
||||
- ✅ Multiple research scenarios
|
||||
- ✅ Error handling patterns
|
||||
|
||||
### http_api_examples.py
|
||||
- ✅ All functionality from simple_http_example.py
|
||||
- ✅ Reusable client class for application integration
|
||||
- ✅ Batch processing capabilities
|
||||
- ✅ Advanced polling strategies
|
||||
- ✅ Production-ready error handling
|
||||
- ✅ Comprehensive API coverage
|
||||
- ✅ Settings management patterns
|
||||
|
||||
## 🔗 Related Documentation
|
||||
|
||||
- [Main HTTP Examples](../README.md)
|
||||
- [Basic Working Example](../simple_working_example.py)
|
||||
- [API Documentation](../../../../README.md)
|
||||
+499
@@ -0,0 +1,499 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
HTTP API Examples for Local Deep Research v1.0+
|
||||
|
||||
This script demonstrates comprehensive usage of the LDR HTTP API with authentication.
|
||||
Includes examples for research, settings management, and batch operations.
|
||||
|
||||
Requirements:
|
||||
- LDR v1.0+ (with authentication features)
|
||||
- LDR server running: python -m local_deep_research.web.app
|
||||
- Beautiful Soup: pip install beautifulsoup4
|
||||
|
||||
================================================================================
|
||||
IMPORTANT - LOCALHOST ONLY
|
||||
================================================================================
|
||||
This example ONLY works when connecting via localhost:
|
||||
✅ http://localhost:5000
|
||||
✅ http://127.0.0.1:5000
|
||||
|
||||
It will NOT work when connecting via:
|
||||
❌ http://192.168.x.x:5000 (local network IP)
|
||||
❌ http://your-server.com:5000 (remote server)
|
||||
|
||||
WHY: For security, session cookies require HTTPS for non-localhost connections.
|
||||
|
||||
SOLUTIONS for non-localhost access:
|
||||
1. Use HTTPS with a reverse proxy (recommended for production)
|
||||
2. SSH tunnel: ssh -L 5000:localhost:5000 user@server, then use localhost:5000
|
||||
3. Set TESTING=1 when starting server (INSECURE - development only!)
|
||||
|
||||
WARNING: TESTING=1 disables secure cookie protection. Never use in production.
|
||||
================================================================================
|
||||
"""
|
||||
|
||||
import time
|
||||
from typing import Any, Dict, List
|
||||
import requests
|
||||
from pathlib import Path
|
||||
import sys
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
# Add the src directory to Python path for programmatic user creation
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent.parent.parent / "src"))
|
||||
|
||||
from local_deep_research.database.encrypted_db import DatabaseManager
|
||||
from local_deep_research.database.models import User
|
||||
from local_deep_research.database.auth_db import auth_db_session
|
||||
|
||||
# Configuration
|
||||
BASE_URL = "http://localhost:5000"
|
||||
|
||||
|
||||
def create_test_user():
|
||||
"""Create a test user programmatically - works out of the box!"""
|
||||
username = f"testuser_{int(time.time())}"
|
||||
password = "testpassword123"
|
||||
|
||||
print(f"Creating test user: {username}")
|
||||
|
||||
try:
|
||||
# Create user in auth database
|
||||
with auth_db_session() as session:
|
||||
new_user = User(username=username)
|
||||
session.add(new_user)
|
||||
session.commit()
|
||||
|
||||
# Create encrypted database for user
|
||||
db_manager = DatabaseManager()
|
||||
db_manager.create_user_database(username, password)
|
||||
|
||||
print(f"✅ User created successfully: {username}")
|
||||
return username, password
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Failed to create user: {e}")
|
||||
return None, None
|
||||
|
||||
|
||||
class LDRClient:
|
||||
"""Client for interacting with LDR API v1.0+ with authentication"""
|
||||
|
||||
def __init__(self, base_url: str = BASE_URL):
|
||||
self.base_url = base_url
|
||||
self.session = requests.Session()
|
||||
self.csrf_token = None
|
||||
self.username = None
|
||||
|
||||
def login(self, username: str, password: str) -> bool:
|
||||
"""Authenticate with the LDR server."""
|
||||
try:
|
||||
# Get login page and CSRF token
|
||||
login_page = self.session.get(f"{self.base_url}/auth/login")
|
||||
if login_page.status_code != 200:
|
||||
return False
|
||||
|
||||
soup = BeautifulSoup(login_page.text, "html.parser")
|
||||
csrf_input = soup.find("input", {"name": "csrf_token"})
|
||||
login_csrf = csrf_input.get("value")
|
||||
|
||||
if not login_csrf:
|
||||
return False
|
||||
|
||||
# Submit login form
|
||||
login_response = self.session.post(
|
||||
f"{self.base_url}/auth/login",
|
||||
data={
|
||||
"username": username,
|
||||
"password": password,
|
||||
"csrf_token": login_csrf,
|
||||
},
|
||||
allow_redirects=False,
|
||||
)
|
||||
|
||||
if login_response.status_code not in [200, 302]:
|
||||
return False
|
||||
|
||||
self.username = username
|
||||
|
||||
# Get API CSRF token for API calls
|
||||
csrf_response = self.session.get(f"{self.base_url}/auth/csrf-token")
|
||||
if csrf_response.status_code == 200:
|
||||
self.csrf_token = csrf_response.json().get("csrf_token")
|
||||
|
||||
return True
|
||||
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
def logout(self) -> None:
|
||||
"""Logout from the server."""
|
||||
if self.csrf_token:
|
||||
self.session.post(
|
||||
f"{self.base_url}/auth/logout",
|
||||
headers={"X-CSRF-Token": self.csrf_token},
|
||||
)
|
||||
|
||||
def _get_headers(self) -> Dict[str, str]:
|
||||
"""Get headers with CSRF token."""
|
||||
return {"X-CSRF-Token": self.csrf_token} if self.csrf_token else {}
|
||||
|
||||
def check_health(self) -> Dict[str, Any]:
|
||||
"""Check API health status."""
|
||||
response = self.session.get(f"{self.base_url}/auth/check")
|
||||
return response.json()
|
||||
|
||||
def start_research(self, query: str, **kwargs) -> Dict[str, Any]:
|
||||
"""Start a new research task."""
|
||||
payload = {
|
||||
"query": query,
|
||||
"model": kwargs.get("model"),
|
||||
"search_engines": kwargs.get("search_engines", ["wikipedia"]),
|
||||
"iterations": kwargs.get("iterations", 2),
|
||||
"questions_per_iteration": kwargs.get("questions_per_iteration", 3),
|
||||
"temperature": kwargs.get("temperature", 0.7),
|
||||
"local_context": kwargs.get("local_context", 2000),
|
||||
"web_context": kwargs.get("web_context", 2000),
|
||||
}
|
||||
|
||||
response = self.session.post(
|
||||
f"{self.base_url}/api/start_research",
|
||||
json=payload,
|
||||
headers=self._get_headers(),
|
||||
)
|
||||
|
||||
if response.status_code == 200:
|
||||
return response.json()
|
||||
raise Exception(f"Failed to start research: {response.text}")
|
||||
|
||||
def get_research_status(self, research_id: str) -> Dict[str, Any]:
|
||||
"""Get the status of a research task."""
|
||||
response = self.session.get(
|
||||
f"{self.base_url}/api/research/{research_id}/status"
|
||||
)
|
||||
return response.json()
|
||||
|
||||
def get_research_result(self, research_id: str) -> Dict[str, Any]:
|
||||
"""Get the results of a completed research task."""
|
||||
response = self.session.get(f"{self.base_url}/api/report/{research_id}")
|
||||
return response.json()
|
||||
|
||||
def wait_for_research(
|
||||
self, research_id: str, timeout: int = 300
|
||||
) -> Dict[str, Any]:
|
||||
"""Wait for research to complete and return results."""
|
||||
start_time = time.time()
|
||||
|
||||
while time.time() - start_time < timeout:
|
||||
status = self.get_research_status(research_id)
|
||||
|
||||
if status.get("status") == "completed":
|
||||
return self.get_research_result(research_id)
|
||||
if status.get("status") == "failed":
|
||||
raise Exception(
|
||||
f"Research failed: {status.get('error', 'Unknown error')}"
|
||||
)
|
||||
|
||||
print(
|
||||
f" Status: {status.get('status', 'unknown')} - {status.get('progress', 'N/A')}"
|
||||
)
|
||||
time.sleep(3)
|
||||
|
||||
raise TimeoutError(
|
||||
f"Research {research_id} timed out after {timeout} seconds"
|
||||
)
|
||||
|
||||
def get_settings(self) -> Dict[str, Any]:
|
||||
"""Get all user settings."""
|
||||
response = self.session.get(f"{self.base_url}/settings/api")
|
||||
return response.json()
|
||||
|
||||
def get_setting(self, key: str) -> Any:
|
||||
"""Get a specific setting value."""
|
||||
response = self.session.get(f"{self.base_url}/settings/api/{key}")
|
||||
if response.status_code == 200:
|
||||
return response.json()
|
||||
return None
|
||||
|
||||
def update_setting(self, key: str, value: Any) -> bool:
|
||||
"""Update a setting value."""
|
||||
response = self.session.put(
|
||||
f"{self.base_url}/settings/api/{key}",
|
||||
json={"value": value},
|
||||
headers=self._get_headers(),
|
||||
)
|
||||
return response.status_code in [200, 201]
|
||||
|
||||
def get_history(self, limit: int = 10) -> List[Dict[str, Any]]:
|
||||
"""Get research history."""
|
||||
response = self.session.get(
|
||||
f"{self.base_url}/history/api", params={"limit": limit}
|
||||
)
|
||||
data = response.json()
|
||||
return data.get("items", data.get("history", []))
|
||||
|
||||
def get_available_models(self) -> Dict[str, str]:
|
||||
"""Get available LLM providers and models."""
|
||||
response = self.session.get(
|
||||
f"{self.base_url}/settings/api/available-models"
|
||||
)
|
||||
data = response.json()
|
||||
return data.get("providers", data.get("models", {}))
|
||||
|
||||
def get_available_search_engines(self) -> List[str]:
|
||||
"""Get available search engines."""
|
||||
response = self.session.get(
|
||||
f"{self.base_url}/settings/api/available-search-engines"
|
||||
)
|
||||
data = response.json()
|
||||
return data.get("engines", data.get("engine_options", []))
|
||||
|
||||
|
||||
def example_quick_research(client: LDRClient) -> None:
|
||||
"""Example: Quick research with minimal parameters."""
|
||||
print("\n=== Example 1: Quick Research ===")
|
||||
|
||||
research = client.start_research(
|
||||
query="What are the key principles of machine learning?",
|
||||
iterations=1,
|
||||
questions_per_iteration=2,
|
||||
)
|
||||
|
||||
print(f"Started research ID: {research['research_id']}")
|
||||
|
||||
# Wait for completion
|
||||
result = client.wait_for_research(research["research_id"])
|
||||
|
||||
print(f"\nSummary: {result['summary'][:500]}...")
|
||||
print(f"Sources: {len(result.get('sources', []))}")
|
||||
print(f"Findings: {len(result.get('findings', []))}")
|
||||
|
||||
|
||||
def example_detailed_research(client: LDRClient) -> None:
|
||||
"""Example: Detailed research with multiple search engines."""
|
||||
print("\n=== Example 2: Detailed Research ===")
|
||||
|
||||
# Check available search engines
|
||||
engines = client.get_available_search_engines()
|
||||
print(f"Available search engines: {engines}")
|
||||
|
||||
# Use multiple engines
|
||||
selected_engines = (
|
||||
["wikipedia", "arxiv"] if "arxiv" in engines else ["wikipedia"]
|
||||
)
|
||||
|
||||
research = client.start_research(
|
||||
query="Impact of climate change on global food security",
|
||||
search_engines=selected_engines,
|
||||
iterations=3,
|
||||
questions_per_iteration=4,
|
||||
temperature=0.7,
|
||||
)
|
||||
|
||||
print(f"Started detailed research ID: {research['research_id']}")
|
||||
|
||||
# Monitor progress
|
||||
result = client.wait_for_research(research["research_id"], timeout=600)
|
||||
|
||||
print(f"\nTitle: {result.get('query', 'N/A')}")
|
||||
print(f"Summary length: {len(result['summary'])} characters")
|
||||
print(f"Sources: {len(result.get('sources', []))}")
|
||||
|
||||
# Show some findings
|
||||
findings = result.get("findings", [])
|
||||
if findings:
|
||||
print("\nTop findings:")
|
||||
for i, finding in enumerate(findings[:3], 1):
|
||||
print(f"{i}. {finding.get('text', 'N/A')[:100]}...")
|
||||
|
||||
|
||||
def example_settings_management(client: LDRClient) -> None:
|
||||
"""Example: Managing user settings."""
|
||||
print("\n=== Example 3: Settings Management ===")
|
||||
|
||||
# Get current settings
|
||||
settings = client.get_settings()
|
||||
settings_data = settings.get("settings", {})
|
||||
|
||||
# Display current LLM configuration
|
||||
llm_provider = settings_data.get("llm.provider", {}).get("value", "Not set")
|
||||
llm_model = settings_data.get("llm.model", {}).get("value", "Not set")
|
||||
|
||||
print(f"Current LLM Provider: {llm_provider}")
|
||||
print(f"Current LLM Model: {llm_model}")
|
||||
|
||||
# Get available models
|
||||
models = client.get_available_models()
|
||||
print(f"\nAvailable providers: {list(models.keys())}")
|
||||
|
||||
# Example: Update temperature setting
|
||||
current_temp = settings_data.get("llm.temperature", {}).get("value", 0.7)
|
||||
print(f"\nCurrent temperature: {current_temp}")
|
||||
|
||||
# Update temperature (example - uncomment to actually update)
|
||||
# success = client.update_setting("llm.temperature", 0.5)
|
||||
# print(f"Temperature update: {'Success' if success else 'Failed'}")
|
||||
|
||||
|
||||
def example_batch_research(client: LDRClient) -> None:
|
||||
"""Example: Running multiple research tasks in batch."""
|
||||
print("\n=== Example 4: Batch Research ===")
|
||||
|
||||
queries = [
|
||||
"What is quantum entanglement?",
|
||||
"How does CRISPR gene editing work?",
|
||||
"What are the applications of blockchain technology?",
|
||||
]
|
||||
|
||||
research_ids = []
|
||||
|
||||
# Start all research tasks
|
||||
for query in queries:
|
||||
try:
|
||||
research = client.start_research(
|
||||
query=query, iterations=1, questions_per_iteration=2
|
||||
)
|
||||
research_ids.append(
|
||||
{
|
||||
"id": research["research_id"],
|
||||
"query": query,
|
||||
"status": "started",
|
||||
}
|
||||
)
|
||||
print(f"Started: {query} (ID: {research['research_id']})")
|
||||
except Exception as e:
|
||||
print(f"Failed to start '{query}': {e}")
|
||||
|
||||
# Wait for all to complete
|
||||
print("\nWaiting for batch completion...")
|
||||
completed = 0
|
||||
|
||||
while completed < len(research_ids):
|
||||
for research in research_ids:
|
||||
if research["status"] != "completed":
|
||||
try:
|
||||
status = client.get_research_status(research["id"])
|
||||
if status.get("status") == "completed":
|
||||
research["status"] = "completed"
|
||||
completed += 1
|
||||
print(f"✓ Completed: {research['query']}")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if completed < len(research_ids):
|
||||
time.sleep(3)
|
||||
|
||||
# Get all results
|
||||
print("\nBatch Results Summary:")
|
||||
for research in research_ids:
|
||||
try:
|
||||
result = client.get_research_result(research["id"])
|
||||
print(f"\n{research['query']}:")
|
||||
print(f" - Summary: {result['summary'][:150]}...")
|
||||
print(f" - Sources: {len(result.get('sources', []))}")
|
||||
except Exception as e:
|
||||
print(f" - Error getting results: {e}")
|
||||
|
||||
|
||||
def example_research_history(client: LDRClient) -> None:
|
||||
"""Example: Viewing research history."""
|
||||
print("\n=== Example 5: Research History ===")
|
||||
|
||||
history = client.get_history(limit=5)
|
||||
|
||||
if not history:
|
||||
print("No research history found.")
|
||||
return
|
||||
|
||||
print(f"Found {len(history)} recent research items:\n")
|
||||
|
||||
for item in history:
|
||||
created = item.get("created_at", "Unknown date")
|
||||
query = item.get("query", "Unknown query")
|
||||
status = item.get("status", "Unknown")
|
||||
research_id = item.get("id", item.get("research_id", "N/A"))
|
||||
|
||||
print(f"ID: {research_id}")
|
||||
print(f"Query: {query}")
|
||||
print(f"Date: {created}")
|
||||
print(f"Status: {status}")
|
||||
print("-" * 40)
|
||||
|
||||
|
||||
def main():
|
||||
"""Run all examples."""
|
||||
print("=== LDR HTTP API v1.0 Examples ===")
|
||||
print(
|
||||
"🎯 This example works completely out of the box - no manual setup required!"
|
||||
)
|
||||
|
||||
# First, check if server is running
|
||||
try:
|
||||
response = requests.get(f"{BASE_URL}/auth/check", timeout=5)
|
||||
if response.status_code in [
|
||||
200,
|
||||
401,
|
||||
]: # 401 is expected when not authenticated
|
||||
print("✅ Server is running")
|
||||
else:
|
||||
print(f"❌ Server returned status code: {response.status_code}")
|
||||
print("Make sure the server is running:")
|
||||
print(" python -m local_deep_research.web.app")
|
||||
return
|
||||
except requests.exceptions.ConnectionError:
|
||||
print("❌ Cannot connect to LDR server!")
|
||||
print("Make sure the server is running:")
|
||||
print(" python -m local_deep_research.web.app")
|
||||
return
|
||||
|
||||
# Create test user automatically
|
||||
username, password = create_test_user()
|
||||
if not username:
|
||||
print("❌ Failed to create test user")
|
||||
return
|
||||
|
||||
# Create client
|
||||
client = LDRClient(BASE_URL)
|
||||
|
||||
try:
|
||||
# Login with the created user
|
||||
print(f"\nLogging in as: {username}")
|
||||
if not client.login(username, password):
|
||||
print("❌ Login failed!")
|
||||
return
|
||||
|
||||
print("✅ Login successful")
|
||||
|
||||
# Check health
|
||||
health = client.check_health()
|
||||
print(f"Authenticated: {health.get('authenticated', False)}")
|
||||
print(f"Username: {health.get('username', 'N/A')}")
|
||||
|
||||
# Run examples
|
||||
example_quick_research(client)
|
||||
example_detailed_research(client)
|
||||
example_settings_management(client)
|
||||
example_batch_research(client)
|
||||
example_research_history(client)
|
||||
|
||||
print("\n✅ All examples completed successfully!")
|
||||
print(f"🔑 Created user: {username}")
|
||||
print("📝 You can now use this user for manual testing:")
|
||||
print(f" Username: {username}")
|
||||
print(f" Password: {password}")
|
||||
print(f" Login URL: {BASE_URL}/auth/login")
|
||||
|
||||
except requests.exceptions.ConnectionError:
|
||||
print("\n❌ Cannot connect to LDR server!")
|
||||
print("Make sure the server is running:")
|
||||
print(" python -m local_deep_research.web.app")
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error: {e}")
|
||||
finally:
|
||||
# Always logout
|
||||
client.logout()
|
||||
print("\n✅ Logged out")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+440
@@ -0,0 +1,440 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Simple HTTP API Example for Local Deep Research v1.0+
|
||||
|
||||
This example shows how to use the LDR API with authentication.
|
||||
Works completely out of the box with automatic user creation.
|
||||
|
||||
================================================================================
|
||||
IMPORTANT - LOCALHOST ONLY
|
||||
================================================================================
|
||||
This example ONLY works when connecting via localhost:
|
||||
✅ http://localhost:5000
|
||||
✅ http://127.0.0.1:5000
|
||||
|
||||
It will NOT work via http://192.168.x.x:5000 or other non-localhost addresses.
|
||||
|
||||
WHY: Session cookies require HTTPS for non-localhost (security).
|
||||
|
||||
SOLUTIONS for non-localhost:
|
||||
1. HTTPS with reverse proxy (production)
|
||||
2. SSH tunnel: ssh -L 5000:localhost:5000 user@server
|
||||
3. TESTING=1 env var (INSECURE - dev only!)
|
||||
|
||||
WARNING: TESTING=1 disables cookie security. Never use in production.
|
||||
================================================================================
|
||||
"""
|
||||
|
||||
import requests
|
||||
import time
|
||||
import sys
|
||||
from bs4 import BeautifulSoup
|
||||
from pathlib import Path
|
||||
|
||||
# Add the src directory to Python path for programmatic user creation
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent.parent.parent / "src"))
|
||||
|
||||
from local_deep_research.database.encrypted_db import DatabaseManager
|
||||
from local_deep_research.database.models import User
|
||||
from local_deep_research.database.auth_db import auth_db_session
|
||||
|
||||
# Configuration
|
||||
API_URL = "http://localhost:5000"
|
||||
|
||||
|
||||
def create_test_user():
|
||||
"""Create a test user programmatically."""
|
||||
username = f"testuser_{int(time.time())}"
|
||||
password = "testpassword123"
|
||||
|
||||
print(f"Creating test user: {username}")
|
||||
|
||||
try:
|
||||
# Create user in auth database
|
||||
with auth_db_session() as session:
|
||||
new_user = User(username=username)
|
||||
session.add(new_user)
|
||||
session.commit()
|
||||
|
||||
# Create encrypted database for user
|
||||
db_manager = DatabaseManager()
|
||||
db_manager.create_user_database(username, password)
|
||||
|
||||
print(f"✅ User created successfully: {username}")
|
||||
return username, password
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Failed to create user: {e}")
|
||||
return None, None
|
||||
|
||||
|
||||
def main():
|
||||
print("=== LDR HTTP API Example ===")
|
||||
print("🎯 This example works completely out of the box!\n")
|
||||
|
||||
print("⚠️ IMPORTANT NOTES:")
|
||||
print(" • This script may take several minutes to complete")
|
||||
print(" • Research progress can be monitored in the server logs")
|
||||
print(" • Server logs are available at: /tmp/ldr_server_5000.log")
|
||||
print(
|
||||
" • Use 'tail -f /tmp/ldr_server_5000.log' to monitor progress in real-time"
|
||||
)
|
||||
print(" • Results will be available at the URL shown when complete\n")
|
||||
|
||||
# Check if server is running
|
||||
try:
|
||||
response = requests.get(f"{API_URL}/", timeout=5)
|
||||
if response.status_code != 200:
|
||||
print("❌ Server is not responding correctly")
|
||||
print("\n📋 HOW TO START THE SERVER:")
|
||||
print(" • Option 1: python -m local_deep_research.web.app")
|
||||
print(
|
||||
" • Option 2: bash scripts/dev/restart_server.sh (recommended)"
|
||||
)
|
||||
print(
|
||||
" • Note: restart_server.sh stops only the instance on its target port"
|
||||
)
|
||||
sys.exit(1)
|
||||
print("✅ Server is running")
|
||||
except Exception:
|
||||
print(
|
||||
"❌ Cannot connect to server. Please make sure it's running on http://localhost:5000"
|
||||
)
|
||||
print("\n📋 HOW TO START THE SERVER:")
|
||||
print(" • Option 1: python -m local_deep_research.web.app")
|
||||
print(" • Option 2: bash scripts/dev/restart_server.sh (recommended)")
|
||||
print(
|
||||
" • Note: restart_server.sh stops only the instance on its target port"
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
# Create test user automatically
|
||||
username, password = create_test_user()
|
||||
if not username:
|
||||
print("❌ Failed to create test user")
|
||||
sys.exit(1)
|
||||
|
||||
# Create a session to persist cookies
|
||||
session = requests.Session()
|
||||
print(f"\nTesting with user: {username}")
|
||||
|
||||
# Step 1: Login
|
||||
print("\n1. Authenticating...")
|
||||
|
||||
# Get login page and CSRF token
|
||||
login_page = session.get(f"{API_URL}/auth/login")
|
||||
soup = BeautifulSoup(login_page.text, "html.parser")
|
||||
csrf_input = soup.find("input", {"name": "csrf_token"})
|
||||
login_csrf = csrf_input.get("value")
|
||||
|
||||
if not login_csrf:
|
||||
print("❌ Could not get CSRF token from login page")
|
||||
sys.exit(1)
|
||||
|
||||
# Login with form data (not JSON)
|
||||
login_response = session.post(
|
||||
f"{API_URL}/auth/login",
|
||||
data={
|
||||
"username": username,
|
||||
"password": password,
|
||||
"csrf_token": login_csrf,
|
||||
},
|
||||
allow_redirects=False,
|
||||
)
|
||||
|
||||
if login_response.status_code not in [200, 302]:
|
||||
print(f"❌ Login failed: {login_response.text}")
|
||||
print("\nPlease ensure:")
|
||||
print("- The server is running: python -m local_deep_research.web.app")
|
||||
sys.exit(1)
|
||||
|
||||
print("✅ Login successful")
|
||||
|
||||
# Step 2: Get CSRF token
|
||||
print("\n2. Getting CSRF token...")
|
||||
csrf_response = session.get(f"{API_URL}/auth/csrf-token")
|
||||
csrf_token = csrf_response.json()["csrf_token"]
|
||||
headers = {"X-CSRF-Token": csrf_token}
|
||||
print("✅ CSRF token obtained")
|
||||
|
||||
# Initialize research_id to None
|
||||
research_id = None
|
||||
|
||||
# Example 1: Quick Summary (using the start endpoint)
|
||||
print("\n=== Example 1: Quick Summary ===")
|
||||
print(
|
||||
"📝 This example demonstrates starting a research query and polling for results"
|
||||
)
|
||||
print("⏱️ This typically takes 1-3 minutes to complete\n")
|
||||
|
||||
research_request = {
|
||||
"query": "What is machine learning?",
|
||||
"model": None, # Will use default from settings
|
||||
"search_engines": ["wikipedia"], # Fast for demo
|
||||
"iterations": 1,
|
||||
"questions_per_iteration": 2,
|
||||
}
|
||||
|
||||
# Start research - CORRECT ENDPOINT
|
||||
print("🚀 Starting research...")
|
||||
start_response = session.post(
|
||||
f"{API_URL}/api/start_research", json=research_request, headers=headers
|
||||
)
|
||||
|
||||
if start_response.status_code != 200:
|
||||
print(f"❌ Failed to start research: {start_response.text}")
|
||||
sys.exit(1)
|
||||
|
||||
research_data = start_response.json()
|
||||
research_id = research_data["research_id"]
|
||||
print("✅ Research started successfully!")
|
||||
print(f"🆔 Research ID: {research_id}")
|
||||
print(
|
||||
"📊 Monitor progress in server logs: tail -f /tmp/ldr_server_5000.log"
|
||||
)
|
||||
print(f"🌐 Results will be available at: {API_URL}/results/{research_id}\n")
|
||||
|
||||
# Poll for results
|
||||
print("⏳ Waiting for research to complete...")
|
||||
print(
|
||||
"⚠️ NOTE: This will poll for up to 3 minutes to ensure research completes"
|
||||
)
|
||||
print(
|
||||
" If it fails, the research may still be running - check the results URL\n"
|
||||
)
|
||||
|
||||
poll_count = 0
|
||||
max_polls = 18 # Maximum 3 minutes (18 * 10 seconds)
|
||||
|
||||
while poll_count < max_polls:
|
||||
status_response = session.get(
|
||||
f"{API_URL}/api/research/{research_id}/status"
|
||||
)
|
||||
|
||||
if status_response.status_code == 200:
|
||||
status = status_response.json()
|
||||
current_status = status.get("status", "unknown")
|
||||
progress = status.get("progress", 0)
|
||||
|
||||
poll_count += 1
|
||||
elapsed_time = poll_count * 10 # 10 seconds per poll
|
||||
print(
|
||||
f" Check {poll_count} ({elapsed_time}s): Status = {current_status} (Progress: {progress}%)"
|
||||
)
|
||||
|
||||
if current_status == "completed":
|
||||
print("🎉 Research completed successfully!")
|
||||
break
|
||||
if current_status == "failed":
|
||||
print(
|
||||
f"❌ Research failed: {status.get('error', 'Unknown error')}"
|
||||
)
|
||||
print(
|
||||
"📋 Check server logs for details: tail -f /tmp/ldr_server_5000.log"
|
||||
)
|
||||
sys.exit(1)
|
||||
elif current_status in ["queued", "in_progress"]:
|
||||
# Continue polling
|
||||
pass
|
||||
else:
|
||||
print(f"⚠️ Unexpected status: {current_status}")
|
||||
|
||||
else:
|
||||
print(
|
||||
f"⚠️ Status check failed with code: {status_response.status_code}"
|
||||
)
|
||||
|
||||
time.sleep(10) # Wait 10 seconds between polls
|
||||
|
||||
if poll_count >= max_polls:
|
||||
print("⏰ 3-minute timeout reached - research is still running")
|
||||
print("💡 This is normal for complex research queries!")
|
||||
print(f"📊 Check results later at: {API_URL}/results/{research_id}")
|
||||
print("📋 Monitor progress with: tail -f /tmp/ldr_server_5000.log")
|
||||
print(
|
||||
"🔍 The script will still try to fetch results (may be incomplete)"
|
||||
)
|
||||
|
||||
# Get results
|
||||
results_response = session.get(f"{API_URL}/api/report/{research_id}")
|
||||
|
||||
if results_response.status_code == 200:
|
||||
results = results_response.json()
|
||||
print(f"\n📝 Summary: {results['summary'][:300]}...")
|
||||
print(f"📚 Sources: {len(results.get('sources', []))} found")
|
||||
print(f"🔍 Findings: {len(results.get('findings', []))} findings")
|
||||
|
||||
# Example 2: Check Settings
|
||||
print("\n=== Example 2: Current Settings ===")
|
||||
settings_response = session.get(f"{API_URL}/settings/api")
|
||||
|
||||
if settings_response.status_code == 200:
|
||||
settings = settings_response.json()["settings"]
|
||||
|
||||
# Show some key settings
|
||||
llm_provider = settings.get("llm.provider", {}).get("value", "Not set")
|
||||
llm_model = settings.get("llm.model", {}).get("value", "Not set")
|
||||
|
||||
print(f"LLM Provider: {llm_provider}")
|
||||
print(f"LLM Model: {llm_model}")
|
||||
|
||||
# Example 3: Get Research History
|
||||
print("\n=== Example 3: Research History ===")
|
||||
history_response = session.get(f"{API_URL}/history/api")
|
||||
|
||||
if history_response.status_code == 200:
|
||||
history = history_response.json()
|
||||
items = history.get("items", history.get("history", []))
|
||||
|
||||
print(f"Found {len(items)} research items")
|
||||
for item in items[:3]: # Show first 3
|
||||
print(
|
||||
f"- {item.get('query', 'Unknown query')} ({item.get('created_at', 'Unknown date')})"
|
||||
)
|
||||
|
||||
# Example 4: Get and Display Research Results (with retry logic)
|
||||
print("\n=== Example 4: Research Results ===")
|
||||
if research_id:
|
||||
print(f"📄 Fetching research results for ID: {research_id}")
|
||||
print(
|
||||
"🔄 Will retry until results are available (up to 2 additional minutes)\n"
|
||||
)
|
||||
|
||||
# Retry fetching results until available
|
||||
results_retries = 0
|
||||
max_results_retries = 12 # 2 minutes (12 * 10 seconds)
|
||||
|
||||
while results_retries < max_results_retries:
|
||||
results_response = session.get(
|
||||
f"{API_URL}/api/report/{research_id}"
|
||||
)
|
||||
|
||||
if results_response.status_code == 200:
|
||||
# Results are available, parse and display them
|
||||
results = results_response.json()
|
||||
|
||||
content = results.get("content", "")
|
||||
sources = results.get("sources", [])
|
||||
findings = results.get("findings", [])
|
||||
|
||||
print(
|
||||
f"✅ Results retrieved successfully after {(results_retries + 1) * 10} seconds!"
|
||||
)
|
||||
print("\n📝 RESEARCH SUMMARY:")
|
||||
print("=" * 50)
|
||||
if content:
|
||||
# Show first 500 characters of the summary
|
||||
summary_preview = (
|
||||
content[:500] + "..." if len(content) > 500 else content
|
||||
)
|
||||
print(summary_preview)
|
||||
else:
|
||||
print("No summary content available")
|
||||
|
||||
print(f"\n📚 SOURCES FOUND: {len(sources)}")
|
||||
for i, source in enumerate(
|
||||
sources[:3], 1
|
||||
): # Show first 3 sources
|
||||
title = source.get("title", "Unknown Title")
|
||||
url = source.get("url", "No URL")
|
||||
print(f" {i}. {title}")
|
||||
print(f" {url}")
|
||||
|
||||
if len(sources) > 3:
|
||||
print(f" ... and {len(sources) - 3} more sources")
|
||||
|
||||
print(f"\n🔍 KEY FINDINGS: {len(findings)}")
|
||||
for i, finding in enumerate(
|
||||
findings[:3], 1
|
||||
): # Show first 3 findings
|
||||
finding_text = finding.get("text", "No finding text")
|
||||
finding_preview = (
|
||||
finding_text[:150] + "..."
|
||||
if len(finding_text) > 150
|
||||
else finding_text
|
||||
)
|
||||
print(f" {i}. {finding_preview}")
|
||||
|
||||
if len(findings) > 3:
|
||||
print(f" ... and {len(findings) - 3} more findings")
|
||||
|
||||
print(
|
||||
f"\n🌐 View full results at: {API_URL}/results/{research_id}"
|
||||
)
|
||||
print("=" * 50)
|
||||
print("🎉 Results displayed successfully!")
|
||||
break # Exit retry loop - success!
|
||||
|
||||
if results_response.status_code == 404:
|
||||
results_retries += 1
|
||||
elapsed_time = results_retries * 10
|
||||
print(
|
||||
f" Retry {results_retries}/{max_results_retries} ({elapsed_time}s): Results not ready yet, waiting..."
|
||||
)
|
||||
time.sleep(10) # Wait 10 seconds before retrying
|
||||
|
||||
else:
|
||||
print(
|
||||
f"❌ Failed to fetch results: {results_response.status_code}"
|
||||
)
|
||||
print(f"Response: {results_response.text[:200]}")
|
||||
break # Exit retry loop - error
|
||||
|
||||
# Handle case where max retries reached
|
||||
if results_retries >= max_results_retries:
|
||||
print(
|
||||
f"\n⏰ Maximum retry time reached ({max_results_retries * 10} seconds)"
|
||||
)
|
||||
print("💡 This is normal for complex research queries!")
|
||||
print(f"📊 Check results later at: {API_URL}/results/{research_id}")
|
||||
print("📋 Monitor progress with: tail -f /tmp/ldr_server_5000.log")
|
||||
print(
|
||||
"🔍 The research is still running - results will be available when complete"
|
||||
)
|
||||
else:
|
||||
print(
|
||||
"⚠️ No research ID available - research may not have started properly"
|
||||
)
|
||||
|
||||
# Logout
|
||||
print("\n5. Logging out...")
|
||||
session.post(f"{API_URL}/auth/logout", headers=headers)
|
||||
print("✅ Logged out successfully")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("🎯 Simple LDR HTTP API Example - Works out of the box!")
|
||||
print("⚡ This script creates a user automatically and tests the API")
|
||||
print(
|
||||
"⏱️ Total runtime: Up to 3 minutes polling + 2 minutes results retry + research time"
|
||||
)
|
||||
print(
|
||||
"🔄 Automatically retries fetching results until available (up to 2 minutes)\n"
|
||||
)
|
||||
|
||||
print("📋 REQUIREMENTS:")
|
||||
print(" • LDR server running")
|
||||
print(" • Beautiful Soup: pip install beautifulsoup4\n")
|
||||
|
||||
print("🚀 START THE SERVER:")
|
||||
print(" • Option 1: python -m local_deep_research.web.app")
|
||||
print(" • Option 2: bash scripts/dev/restart_server.sh (recommended)")
|
||||
print(
|
||||
" • Note: restart_server.sh stops only the instance on its target port\n"
|
||||
)
|
||||
|
||||
print("📊 MONITORING:")
|
||||
print(" • Server logs: tail -f /tmp/ldr_server_5000.log")
|
||||
print(" • This script polls for up to 3 minutes")
|
||||
print(" • If research takes longer, script shows where to check results\n")
|
||||
|
||||
print("⏰ TIMING INFO:")
|
||||
print(" • Script polls for 3 minutes to let research complete")
|
||||
print(" • Then retries fetching results for up to 2 additional minutes")
|
||||
print(" • Research typically completes in 2-10 minutes")
|
||||
print(" • Script displays results automatically when available")
|
||||
print(
|
||||
" • If timeout reached, results URL provided for checking completion\n"
|
||||
)
|
||||
|
||||
main()
|
||||
@@ -0,0 +1,264 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Simple Working HTTP API Example for Local Deep Research v1.0+
|
||||
|
||||
This is a clean, working example that demonstrates the correct way to use the LDR API.
|
||||
It creates a user automatically and handles authentication properly.
|
||||
|
||||
Requirements:
|
||||
- LDR v1.0+ server running: python -m local_deep_research.web.app
|
||||
- Beautiful Soup: pip install beautifulsoup4
|
||||
|
||||
================================================================================
|
||||
IMPORTANT - LOCALHOST ONLY
|
||||
================================================================================
|
||||
This example ONLY works when connecting via localhost:
|
||||
✅ http://localhost:5000
|
||||
✅ http://127.0.0.1:5000
|
||||
|
||||
It will NOT work when connecting via:
|
||||
❌ http://192.168.x.x:5000 (local network IP)
|
||||
❌ http://your-server.com:5000 (remote server)
|
||||
❌ http://0.0.0.0:5000 (even from the same machine)
|
||||
|
||||
WHY: For security, session cookies require HTTPS for non-localhost connections.
|
||||
This prevents session hijacking on untrusted networks.
|
||||
|
||||
SOLUTIONS for non-localhost access:
|
||||
1. Use HTTPS with a reverse proxy (recommended for production)
|
||||
2. SSH tunnel: ssh -L 5000:localhost:5000 user@server, then use localhost:5000
|
||||
3. Set TESTING=1 when starting server (INSECURE - development only!)
|
||||
Example: TESTING=1 python -m local_deep_research.web.app
|
||||
|
||||
WARNING: TESTING=1 disables secure cookie protection. Session cookies can be
|
||||
intercepted by network attackers. Never use in production or on public networks.
|
||||
================================================================================
|
||||
"""
|
||||
|
||||
import requests
|
||||
from bs4 import BeautifulSoup
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
# Add the src directory to Python path for programmatic user creation
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent.parent.parent / "src"))
|
||||
|
||||
from local_deep_research.database.encrypted_db import DatabaseManager
|
||||
from local_deep_research.database.models import User
|
||||
from local_deep_research.database.auth_db import auth_db_session
|
||||
|
||||
|
||||
def create_test_user():
|
||||
"""Create a test user programmatically - works out of the box!"""
|
||||
username = f"testuser_{int(time.time())}"
|
||||
password = "testpassword123"
|
||||
|
||||
print(f"Creating test user: {username}")
|
||||
|
||||
try:
|
||||
# Create user in auth database
|
||||
with auth_db_session() as session:
|
||||
new_user = User(username=username)
|
||||
session.add(new_user)
|
||||
session.commit()
|
||||
|
||||
# Create encrypted database for user
|
||||
db_manager = DatabaseManager()
|
||||
db_manager.create_user_database(username, password)
|
||||
|
||||
print(f"✅ User created successfully: {username}")
|
||||
return username, password
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Failed to create user: {e}")
|
||||
return None, None
|
||||
|
||||
|
||||
def test_api_with_user(username, password):
|
||||
"""Test the API with the created user."""
|
||||
print(f"\n=== Testing API with user: {username} ===")
|
||||
|
||||
base_url = "http://localhost:5000"
|
||||
session = requests.Session()
|
||||
|
||||
# 1. Test login
|
||||
print("1. Testing login...")
|
||||
try:
|
||||
login_page = session.get(f"{base_url}/auth/login")
|
||||
if login_page.status_code != 200:
|
||||
print(f" ❌ Failed to get login page: {login_page.status_code}")
|
||||
return False
|
||||
|
||||
soup = BeautifulSoup(login_page.text, "html.parser")
|
||||
csrf_input = soup.find("input", {"name": "csrf_token"})
|
||||
login_csrf = csrf_input.get("value")
|
||||
|
||||
if not login_csrf:
|
||||
print(" ❌ No CSRF token found")
|
||||
return False
|
||||
|
||||
login_response = session.post(
|
||||
f"{base_url}/auth/login",
|
||||
data={
|
||||
"username": username,
|
||||
"password": password,
|
||||
"csrf_token": login_csrf,
|
||||
},
|
||||
allow_redirects=False,
|
||||
)
|
||||
|
||||
print(f" Login status: {login_response.status_code}")
|
||||
|
||||
if login_response.status_code not in [200, 302]:
|
||||
print(" ❌ Login failed")
|
||||
return False
|
||||
|
||||
print(" ✅ Login successful")
|
||||
|
||||
except Exception as e:
|
||||
print(f" ❌ Login error: {e}")
|
||||
return False
|
||||
|
||||
# 2. Get CSRF token for API
|
||||
print("\n2. Getting API CSRF token...")
|
||||
try:
|
||||
csrf_response = session.get(f"{base_url}/auth/csrf-token")
|
||||
if csrf_response.status_code != 200:
|
||||
print(
|
||||
f" ❌ Failed to get CSRF token: {csrf_response.status_code}"
|
||||
)
|
||||
return False
|
||||
|
||||
csrf_data = csrf_response.json()
|
||||
csrf_token = csrf_data.get("csrf_token")
|
||||
|
||||
if not csrf_token:
|
||||
print(" ❌ No CSRF token in response")
|
||||
return False
|
||||
|
||||
print(f" ✅ API CSRF token: {csrf_token[:20]}...")
|
||||
|
||||
except Exception as e:
|
||||
print(f" ❌ CSRF token error: {e}")
|
||||
return False
|
||||
|
||||
# 3. Test research API
|
||||
print("\n3. Testing research API...")
|
||||
|
||||
research_request = {
|
||||
"query": "What is machine learning?",
|
||||
"model": "gpt-4o-mini",
|
||||
"search_engines": ["searxng"],
|
||||
}
|
||||
|
||||
headers = {"X-CSRF-Token": csrf_token, "Content-Type": "application/json"}
|
||||
|
||||
# Test the correct endpoint
|
||||
print("\n 3.1 Testing /api/start_research...")
|
||||
try:
|
||||
url = f"{base_url}/api/start_research"
|
||||
response = session.post(url, json=research_request, headers=headers)
|
||||
|
||||
print(f" Status: {response.status_code}")
|
||||
print(f" Response: {response.text[:300]}")
|
||||
|
||||
if response.status_code == 200:
|
||||
try:
|
||||
data = response.json()
|
||||
if data.get("status") == "success":
|
||||
print(" ✅ Research started successfully!")
|
||||
research_id = data.get("research_id")
|
||||
if research_id:
|
||||
print(f" Research ID: {research_id}")
|
||||
print("\n🎉 SUCCESS! API is working correctly.")
|
||||
print(
|
||||
f"📊 View results at: {base_url}/results/{research_id}"
|
||||
)
|
||||
return True
|
||||
elif data.get("status") == "queued":
|
||||
print(" ✅ Research queued successfully!")
|
||||
return True
|
||||
else:
|
||||
print(
|
||||
f" ⚠️ Research returned: {data.get('status', 'unknown')}"
|
||||
)
|
||||
except Exception:
|
||||
print(" ⚠️ Response is not valid JSON")
|
||||
elif response.status_code == 401:
|
||||
print(" ❌ Authentication failed")
|
||||
elif response.status_code == 403:
|
||||
print(" ❌ Forbidden - CSRF token issue")
|
||||
elif response.status_code == 404:
|
||||
print(" ❌ Endpoint not found")
|
||||
elif response.status_code == 500:
|
||||
print(" ❌ Server error")
|
||||
print(" Check server logs: tail -f /tmp/ldr_server.log")
|
||||
else:
|
||||
print(" ⚠️ Unexpected status code")
|
||||
|
||||
except Exception as e:
|
||||
print(f" ❌ Error testing endpoint: {e}")
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
"""Main function that works completely out of the box!"""
|
||||
print("=== Simple LDR API Working Example ===")
|
||||
print(
|
||||
"🎯 This example works completely out of the box - no manual setup required!\n"
|
||||
)
|
||||
|
||||
# Check if server is running
|
||||
try:
|
||||
response = requests.get("http://localhost:5000/", timeout=5)
|
||||
if response.status_code != 200:
|
||||
print("❌ Server is not responding correctly")
|
||||
print("\nPlease start the server:")
|
||||
print(" python -m local_deep_research.web.app")
|
||||
sys.exit(1)
|
||||
print("✅ Server is running")
|
||||
except Exception:
|
||||
print(
|
||||
"❌ Cannot connect to server. Please make sure it's running on http://localhost:5000"
|
||||
)
|
||||
print("\nStart the server with:")
|
||||
print(" python -m local_deep_research.web.app")
|
||||
sys.exit(1)
|
||||
|
||||
# Create test user automatically
|
||||
username, password = create_test_user()
|
||||
if not username:
|
||||
print("❌ Failed to create test user")
|
||||
sys.exit(1)
|
||||
|
||||
# Test API
|
||||
success = test_api_with_user(username, password)
|
||||
|
||||
if success:
|
||||
print("\n✅ API test completed successfully")
|
||||
print(f"\n🔑 Created user: {username}")
|
||||
print("📝 You can now use this user for manual testing:")
|
||||
print(f" Username: {username}")
|
||||
print(f" Password: {password}")
|
||||
print(" Login URL: http://localhost:5000/auth/login")
|
||||
print("\nNext steps:")
|
||||
print("- Try different research queries")
|
||||
print("- Explore other API endpoints")
|
||||
print("- Check out the web interface at http://localhost:5000")
|
||||
print("- Use the credentials above to log in manually")
|
||||
sys.exit(0)
|
||||
else:
|
||||
print("\n❌ API test failed")
|
||||
print("\nTroubleshooting:")
|
||||
print(
|
||||
"- Make sure the server is running: python -m local_deep_research.web.app"
|
||||
)
|
||||
print("- Check server logs for errors: tail -f /tmp/ldr_server.log")
|
||||
print("- Ensure all dependencies are installed")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,239 @@
|
||||
# Local Deep Research - Programmatic API Examples
|
||||
|
||||
This directory contains examples demonstrating how to use Local Deep Research programmatically without requiring authentication or database access.
|
||||
|
||||
## Quick Start
|
||||
|
||||
All examples use the programmatic API that bypasses authentication:
|
||||
|
||||
```python
|
||||
from local_deep_research.api import quick_summary, detailed_research
|
||||
from local_deep_research.api.settings_utils import create_settings_snapshot
|
||||
|
||||
# Create settings for programmatic mode
|
||||
settings = create_settings_snapshot({
|
||||
"search.tool": "wikipedia"
|
||||
})
|
||||
|
||||
# Run research
|
||||
result = quick_summary(
|
||||
"your topic",
|
||||
settings_snapshot=settings,
|
||||
programmatic_mode=True
|
||||
)
|
||||
```
|
||||
|
||||
## Examples Overview
|
||||
|
||||
| Example | Purpose | Key Features | Difficulty |
|
||||
|---------|---------|--------------|------------|
|
||||
| **minimal_working_example.py** | Simplest possible example | Basic setup, minimal code | Beginner |
|
||||
| **simple_programmatic_example.py** | Common use cases with the new API | quick_summary, detailed_research, generate_report, custom parameters | Beginner |
|
||||
| **search_strategies_example.py** | Demonstrates search strategies | source-based vs focused-iteration strategies | Intermediate |
|
||||
| **hybrid_search_example.py** | Combine multiple search sources | Multiple retrievers, web + retriever combo | Intermediate |
|
||||
| **advanced_features_example.py** | Advanced programmatic features | generate_report, export formats, result analysis, keyword extraction | Advanced |
|
||||
| **custom_llm_retriever_example.py** | Custom LLM and retriever integration | Ollama, custom retrievers, FAISS | Advanced |
|
||||
| **searxng_example.py** | Web search with SearXNG | SearXNG integration, error handling | Advanced |
|
||||
|
||||
## Example Details
|
||||
|
||||
### minimal_working_example.py
|
||||
**Purpose:** Show the absolute minimum code needed to use LDR programmatically.
|
||||
- Creates a simple LLM and search engine
|
||||
- Runs a basic search
|
||||
- No external dependencies beyond Ollama
|
||||
|
||||
### simple_programmatic_example.py
|
||||
**Purpose:** Demonstrate the main API functions with practical examples.
|
||||
- `quick_summary()` - Fast research with summary
|
||||
- `detailed_research()` - Comprehensive research with findings
|
||||
- `generate_report()` - Create full markdown reports
|
||||
- Custom search parameters
|
||||
- Different search tools (Wikipedia, SearXNG, etc.)
|
||||
|
||||
### search_strategies_example.py
|
||||
**Purpose:** Explain and demonstrate the two main search strategies.
|
||||
- **source-based**: Comprehensive research with detailed citations
|
||||
- **focused-iteration**: Iterative refinement of research questions
|
||||
- Side-by-side comparison of strategies
|
||||
- When to use each strategy
|
||||
|
||||
### hybrid_search_example.py
|
||||
**Purpose:** Show how to combine multiple search sources for comprehensive research.
|
||||
- Multiple named retrievers for different document types
|
||||
- Combining custom retrievers with web search
|
||||
- Source analysis and tracking
|
||||
|
||||
### advanced_features_example.py
|
||||
**Purpose:** Demonstrate advanced programmatic features and analysis capabilities.
|
||||
- `generate_report()` - Create comprehensive markdown reports
|
||||
- Export formats - JSON, Markdown, custom formats
|
||||
- Result analysis - Extract insights and patterns
|
||||
- Keyword extraction - Identify key terms and concepts
|
||||
- Batch research - Process multiple queries efficiently
|
||||
|
||||
### custom_llm_retriever_example.py
|
||||
**Purpose:** Advanced integration with custom components.
|
||||
- Custom LLM implementation (using Ollama)
|
||||
- Custom retriever with embeddings
|
||||
- Vector store integration (FAISS)
|
||||
- Direct use of AdvancedSearchSystem
|
||||
|
||||
### searxng_example.py
|
||||
**Purpose:** Web search integration using SearXNG.
|
||||
- SearXNG configuration
|
||||
- Error handling and fallbacks
|
||||
- Real-time web search
|
||||
- Direct use of search engines
|
||||
|
||||
## Key Concepts
|
||||
|
||||
### Programmatic Mode
|
||||
All examples use `programmatic_mode=True` as an explicit parameter to bypass authentication:
|
||||
```python
|
||||
result = quick_summary(
|
||||
query="your topic",
|
||||
settings_snapshot=settings,
|
||||
programmatic_mode=True
|
||||
)
|
||||
```
|
||||
|
||||
### Search Strategies
|
||||
- **source-based**: Best for academic research, fact-checking
|
||||
- **focused-iteration**: Best for exploratory research, complex topics
|
||||
|
||||
### Search Tools
|
||||
Available search tools include:
|
||||
- `wikipedia` - Wikipedia search
|
||||
- `arxiv` - Academic papers
|
||||
- `searxng` - Web search via SearXNG (recommended default)
|
||||
|
||||
With the default langgraph-agent strategy, the research agent can also call
|
||||
other enabled engines dynamically per query — the former `auto`/`meta`
|
||||
engines were removed in favor of this.
|
||||
|
||||
### Custom Retrievers
|
||||
You can provide your own retrievers:
|
||||
```python
|
||||
result = quick_summary(
|
||||
query="topic",
|
||||
retrievers={"my_docs": custom_retriever},
|
||||
search_tool="my_docs",
|
||||
settings_snapshot=settings,
|
||||
programmatic_mode=True
|
||||
)
|
||||
```
|
||||
|
||||
## API Functions
|
||||
|
||||
### `quick_summary()`
|
||||
Generate a quick research summary:
|
||||
```python
|
||||
from local_deep_research.api import quick_summary
|
||||
from local_deep_research.api.settings_utils import create_settings_snapshot
|
||||
|
||||
settings = create_settings_snapshot({})
|
||||
result = quick_summary(
|
||||
query="Your research question",
|
||||
settings_snapshot=settings,
|
||||
search_tool="wikipedia",
|
||||
iterations=2,
|
||||
programmatic_mode=True
|
||||
)
|
||||
```
|
||||
|
||||
### `detailed_research()`
|
||||
Perform in-depth research with multiple iterations:
|
||||
```python
|
||||
from local_deep_research.api import detailed_research
|
||||
|
||||
result = detailed_research(
|
||||
query="Your research question",
|
||||
settings_snapshot=settings,
|
||||
search_strategy="source-based",
|
||||
iterations=3,
|
||||
questions_per_iteration=5,
|
||||
programmatic_mode=True
|
||||
)
|
||||
```
|
||||
|
||||
### `generate_report()`
|
||||
Generate comprehensive markdown reports with structured sections:
|
||||
```python
|
||||
from local_deep_research.api import generate_report
|
||||
from local_deep_research.api.settings_utils import create_settings_snapshot
|
||||
|
||||
settings = create_settings_snapshot(overrides={"programmatic_mode": True})
|
||||
result = generate_report(
|
||||
query="Your research question",
|
||||
settings_snapshot=settings,
|
||||
output_file="report.md",
|
||||
searches_per_section=3
|
||||
)
|
||||
```
|
||||
|
||||
## Requirements
|
||||
|
||||
- Python 3.8+
|
||||
- Local Deep Research installed
|
||||
- Ollama (for most examples)
|
||||
- SearXNG instance (for searxng_example.py)
|
||||
|
||||
## Running the Examples
|
||||
|
||||
1. Install Local Deep Research:
|
||||
```bash
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
2. Start Ollama (if using Ollama examples):
|
||||
```bash
|
||||
ollama serve
|
||||
ollama pull gemma3:12b
|
||||
ollama pull nomic-embed-text # For embeddings
|
||||
```
|
||||
|
||||
3. Run any example:
|
||||
```bash
|
||||
python minimal_working_example.py
|
||||
python simple_programmatic_example.py
|
||||
python search_strategies_example.py
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "No settings context available" Error
|
||||
Make sure to pass `settings_snapshot` and `programmatic_mode` to all API functions:
|
||||
```python
|
||||
settings = create_settings_snapshot({})
|
||||
result = quick_summary(
|
||||
"topic",
|
||||
settings_snapshot=settings,
|
||||
programmatic_mode=True
|
||||
)
|
||||
```
|
||||
|
||||
### Ollama Connection Error
|
||||
Ensure Ollama is running:
|
||||
```bash
|
||||
ollama serve
|
||||
```
|
||||
|
||||
### SearXNG Connection Error
|
||||
Start a SearXNG instance or use the fallback in the example:
|
||||
```bash
|
||||
docker run -p 8080:8080 searxng/searxng
|
||||
```
|
||||
|
||||
|
||||
## Contributing
|
||||
|
||||
When adding new examples:
|
||||
1. Focus on demonstrating specific features
|
||||
2. Include clear comments explaining the code
|
||||
3. Handle errors gracefully
|
||||
4. Update this README with the new example
|
||||
|
||||
## License
|
||||
|
||||
See the main project LICENSE file.
|
||||
@@ -0,0 +1,612 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Advanced Features Example for Local Deep Research
|
||||
|
||||
This example demonstrates advanced programmatic features including:
|
||||
1. generate_report() - Create comprehensive markdown reports
|
||||
2. Export formats - Save reports in different formats
|
||||
3. Result analysis - Extract and analyze research findings
|
||||
4. Keyword extraction - Identify key topics and concepts
|
||||
"""
|
||||
|
||||
import json
|
||||
from typing import Dict, List, Any
|
||||
|
||||
from local_deep_research.api import (
|
||||
generate_report,
|
||||
detailed_research,
|
||||
quick_summary,
|
||||
)
|
||||
from local_deep_research.api.settings_utils import create_settings_snapshot
|
||||
|
||||
|
||||
def demonstrate_report_generation():
|
||||
"""
|
||||
Generate a comprehensive research report using generate_report().
|
||||
|
||||
This function creates a structured markdown report with:
|
||||
- Executive summary
|
||||
- Detailed findings organized by sections
|
||||
- Source citations
|
||||
- Conclusions and recommendations
|
||||
"""
|
||||
print("=" * 70)
|
||||
print("GENERATE COMPREHENSIVE REPORT")
|
||||
print("=" * 70)
|
||||
print("""
|
||||
This demonstrates the generate_report() function which:
|
||||
- Creates a structured markdown report
|
||||
- Performs multiple searches per section
|
||||
- Organizes findings into coherent sections
|
||||
- Includes citations and references
|
||||
""")
|
||||
|
||||
# Configure settings for programmatic mode
|
||||
settings = create_settings_snapshot(
|
||||
overrides={
|
||||
"programmatic_mode": True,
|
||||
"search.tool": "wikipedia",
|
||||
"llm.temperature": 0.5, # Lower for more focused output
|
||||
"api.allow_file_output": True, # Allow generate_report to save files
|
||||
}
|
||||
)
|
||||
|
||||
# Generate a comprehensive report
|
||||
print(
|
||||
"Generating report on 'Applications of Machine Learning in Healthcare'..."
|
||||
)
|
||||
report = generate_report(
|
||||
query="Applications of Machine Learning in Healthcare",
|
||||
output_file="ml_healthcare_report.md",
|
||||
searches_per_section=2, # Multiple searches per section for depth
|
||||
settings_snapshot=settings,
|
||||
iterations=2,
|
||||
questions_per_iteration=3,
|
||||
)
|
||||
|
||||
print("\n✓ Report generated successfully!")
|
||||
print(f" - Report length: {len(report['content'])} characters")
|
||||
print(
|
||||
f" - File saved to: {report.get('file_path', 'ml_healthcare_report.md')}"
|
||||
)
|
||||
|
||||
# Show first part of report
|
||||
print("\nReport preview (first 500 chars):")
|
||||
print("-" * 40)
|
||||
print(report["content"][:500] + "...")
|
||||
|
||||
return report
|
||||
|
||||
|
||||
def demonstrate_export_formats():
|
||||
"""
|
||||
Show how to export research results in different formats.
|
||||
|
||||
Demonstrates:
|
||||
- Markdown export (default)
|
||||
- JSON export for programmatic processing
|
||||
- Custom formatting with templates
|
||||
"""
|
||||
print("\n" + "=" * 70)
|
||||
print("EXPORT FORMATS")
|
||||
print("=" * 70)
|
||||
print("""
|
||||
Exporting research in different formats:
|
||||
- Markdown: Human-readable reports
|
||||
- JSON: Structured data for processing
|
||||
- Custom: Template-based formatting
|
||||
""")
|
||||
|
||||
settings = create_settings_snapshot(
|
||||
overrides={
|
||||
"programmatic_mode": True,
|
||||
"search.tool": "wikipedia",
|
||||
}
|
||||
)
|
||||
|
||||
# Get research results
|
||||
result = detailed_research(
|
||||
query="Renewable energy technologies",
|
||||
settings_snapshot=settings,
|
||||
iterations=1,
|
||||
questions_per_iteration=2,
|
||||
)
|
||||
|
||||
# Export as JSON
|
||||
json_file = "research_results.json"
|
||||
with open(json_file, "w", encoding="utf-8") as f:
|
||||
json.dump(result, f, indent=2, default=str)
|
||||
print(f"\n✓ JSON export saved to: {json_file}")
|
||||
print(f" - Contains: {len(result.get('findings', []))} findings")
|
||||
print(f" - Sources: {len(result.get('sources', []))} sources")
|
||||
|
||||
# Export as Markdown
|
||||
md_content = format_as_markdown(result)
|
||||
md_file = "research_results.md"
|
||||
with open(md_file, "w", encoding="utf-8") as f:
|
||||
f.write(md_content)
|
||||
print(f"\n✓ Markdown export saved to: {md_file}")
|
||||
print(f" - Length: {len(md_content)} characters")
|
||||
|
||||
# Export as custom format (e.g., BibTeX-like citations)
|
||||
citations = extract_citations(result)
|
||||
cite_file = "research_citations.txt"
|
||||
with open(cite_file, "w", encoding="utf-8") as f:
|
||||
for i, citation in enumerate(citations, 1):
|
||||
f.write(f"[{i}] {citation}\n")
|
||||
print(f"\n✓ Citations export saved to: {cite_file}")
|
||||
print(f" - Total citations: {len(citations)}")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def demonstrate_result_analysis():
|
||||
"""
|
||||
Analyze research results to extract insights and patterns.
|
||||
|
||||
Shows how to:
|
||||
- Extract key findings
|
||||
- Identify recurring themes
|
||||
- Analyze source reliability
|
||||
- Generate statistics
|
||||
"""
|
||||
print("\n" + "=" * 70)
|
||||
print("RESULT ANALYSIS")
|
||||
print("=" * 70)
|
||||
print("""
|
||||
Analyzing research results to extract:
|
||||
- Key findings and insights
|
||||
- Common themes and patterns
|
||||
- Source statistics
|
||||
- Quality metrics
|
||||
""")
|
||||
|
||||
settings = create_settings_snapshot(
|
||||
overrides={
|
||||
"programmatic_mode": True,
|
||||
"search.tool": "wikipedia",
|
||||
}
|
||||
)
|
||||
|
||||
# Perform research
|
||||
result = detailed_research(
|
||||
query="Impact of artificial intelligence on employment",
|
||||
settings_snapshot=settings,
|
||||
search_strategy="source-based",
|
||||
iterations=2,
|
||||
questions_per_iteration=3,
|
||||
)
|
||||
|
||||
# Analyze findings
|
||||
analysis = analyze_findings(result)
|
||||
|
||||
print("\n📊 Research Analysis:")
|
||||
print(f" - Total findings: {analysis['total_findings']}")
|
||||
print(f" - Unique sources: {analysis['unique_sources']}")
|
||||
print(f" - Questions explored: {analysis['total_questions']}")
|
||||
print(f" - Iterations completed: {analysis['iterations']}")
|
||||
|
||||
print("\n🔍 Finding Categories:")
|
||||
for category, count in analysis["categories"].items():
|
||||
print(f" - {category}: {count} findings")
|
||||
|
||||
print("\n📈 Source Distribution:")
|
||||
for source_type, count in analysis["source_types"].items():
|
||||
print(f" - {source_type}: {count} sources")
|
||||
|
||||
# Extract themes
|
||||
themes = extract_themes(result)
|
||||
print("\n🎯 Key Themes Identified:")
|
||||
for i, theme in enumerate(themes[:5], 1):
|
||||
print(f" {i}. {theme}")
|
||||
|
||||
return analysis
|
||||
|
||||
|
||||
def demonstrate_keyword_extraction():
|
||||
"""
|
||||
Extract keywords and key concepts from research results.
|
||||
|
||||
Demonstrates:
|
||||
- Keyword extraction from findings
|
||||
- Concept identification
|
||||
- Topic clustering
|
||||
- Trend analysis
|
||||
"""
|
||||
print("\n" + "=" * 70)
|
||||
print("KEYWORD & CONCEPT EXTRACTION")
|
||||
print("=" * 70)
|
||||
print("""
|
||||
Extracting keywords and concepts:
|
||||
- Important terms and phrases
|
||||
- Technical concepts
|
||||
- Named entities
|
||||
- Trend indicators
|
||||
""")
|
||||
|
||||
settings = create_settings_snapshot(
|
||||
overrides={
|
||||
"programmatic_mode": True,
|
||||
"search.tool": "wikipedia",
|
||||
}
|
||||
)
|
||||
|
||||
# Quick research for keyword extraction
|
||||
result = quick_summary(
|
||||
query="Quantum computing breakthroughs 2024",
|
||||
settings_snapshot=settings,
|
||||
iterations=1,
|
||||
questions_per_iteration=3,
|
||||
)
|
||||
|
||||
# Extract keywords
|
||||
keywords = extract_keywords(result)
|
||||
|
||||
print("\n🔑 Top Keywords:")
|
||||
for keyword, frequency in keywords[:10]:
|
||||
print(f" - {keyword}: {frequency} occurrences")
|
||||
|
||||
# Extract concepts
|
||||
concepts = extract_concepts(result)
|
||||
|
||||
print("\n💡 Key Concepts:")
|
||||
for i, concept in enumerate(concepts[:5], 1):
|
||||
print(f" {i}. {concept}")
|
||||
|
||||
# Identify technical terms
|
||||
technical_terms = extract_technical_terms(result)
|
||||
|
||||
print("\n🔬 Technical Terms:")
|
||||
for term in technical_terms[:8]:
|
||||
print(f" - {term}")
|
||||
|
||||
return keywords, concepts
|
||||
|
||||
|
||||
def format_as_markdown(result: Dict[str, Any]) -> str:
|
||||
"""Convert research results to markdown format."""
|
||||
md = f"# Research Report: {result['query']}\n\n"
|
||||
md += f"**Research ID:** {result.get('research_id', 'N/A')}\n\n"
|
||||
|
||||
# Summary
|
||||
md += "## Summary\n\n"
|
||||
md += result.get("summary", "No summary available") + "\n\n"
|
||||
|
||||
# Findings
|
||||
md += "## Key Findings\n\n"
|
||||
findings = result.get("findings", [])
|
||||
for i, finding in enumerate(findings, 1):
|
||||
finding_text = finding if isinstance(finding, str) else str(finding)
|
||||
md += f"{i}. {finding_text}\n\n"
|
||||
|
||||
# Sources
|
||||
md += "## Sources\n\n"
|
||||
sources = result.get("sources", [])
|
||||
for i, source in enumerate(sources, 1):
|
||||
source_text = source if isinstance(source, str) else str(source)
|
||||
md += f"- [{i}] {source_text}\n"
|
||||
|
||||
# Metadata
|
||||
md += "\n## Metadata\n\n"
|
||||
metadata = result.get("metadata", {})
|
||||
for key, value in metadata.items():
|
||||
md += f"- **{key}:** {value}\n"
|
||||
|
||||
return md
|
||||
|
||||
|
||||
def extract_citations(result: Dict[str, Any]) -> List[str]:
|
||||
"""Extract citations from research results."""
|
||||
citations = []
|
||||
sources = result.get("sources", [])
|
||||
|
||||
for source in sources:
|
||||
if isinstance(source, dict):
|
||||
# Extract URL or title
|
||||
citation = source.get("url", source.get("title", str(source)))
|
||||
else:
|
||||
citation = str(source)
|
||||
citations.append(citation)
|
||||
|
||||
return citations
|
||||
|
||||
|
||||
def analyze_findings(result: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Analyze research findings for patterns and statistics."""
|
||||
findings = result.get("findings", [])
|
||||
sources = result.get("sources", [])
|
||||
questions = result.get("questions", {})
|
||||
|
||||
# Categorize findings (simplified)
|
||||
categories = {
|
||||
"positive": 0,
|
||||
"negative": 0,
|
||||
"neutral": 0,
|
||||
"technical": 0,
|
||||
}
|
||||
|
||||
for finding in findings:
|
||||
finding_text = str(finding).lower()
|
||||
if any(
|
||||
word in finding_text
|
||||
for word in ["benefit", "improve", "enhance", "positive"]
|
||||
):
|
||||
categories["positive"] += 1
|
||||
elif any(
|
||||
word in finding_text
|
||||
for word in ["risk", "challenge", "negative", "concern"]
|
||||
):
|
||||
categories["negative"] += 1
|
||||
elif any(
|
||||
word in finding_text
|
||||
for word in ["algorithm", "system", "technology", "method"]
|
||||
):
|
||||
categories["technical"] += 1
|
||||
else:
|
||||
categories["neutral"] += 1
|
||||
|
||||
# Analyze sources
|
||||
source_types = {}
|
||||
for source in sources:
|
||||
source_text = str(source).lower()
|
||||
if "wikipedia" in source_text:
|
||||
source_type = "Wikipedia"
|
||||
elif "arxiv" in source_text:
|
||||
source_type = "ArXiv"
|
||||
elif "github" in source_text:
|
||||
source_type = "GitHub"
|
||||
else:
|
||||
source_type = "Other"
|
||||
source_types[source_type] = source_types.get(source_type, 0) + 1
|
||||
|
||||
return {
|
||||
"total_findings": len(findings),
|
||||
"unique_sources": len(sources),
|
||||
"total_questions": sum(len(qs) for qs in questions.values()),
|
||||
"iterations": result.get("iterations", 0),
|
||||
"categories": categories,
|
||||
"source_types": source_types,
|
||||
}
|
||||
|
||||
|
||||
def extract_themes(result: Dict[str, Any]) -> List[str]:
|
||||
"""Extract main themes from research results."""
|
||||
# Simplified theme extraction based on common patterns
|
||||
themes = []
|
||||
summary = result.get("summary", "")
|
||||
findings = result.get("findings", [])
|
||||
|
||||
# Combine text for analysis
|
||||
full_text = summary + " ".join(str(f) for f in findings)
|
||||
|
||||
# Simple theme patterns (in production, use NLP libraries)
|
||||
theme_patterns = {
|
||||
"automation": ["automation", "automated", "automatic"],
|
||||
"job displacement": ["job loss", "unemployment", "displacement"],
|
||||
"skill requirements": ["skills", "training", "education"],
|
||||
"economic impact": ["economy", "economic", "gdp", "growth"],
|
||||
"innovation": ["innovation", "innovative", "breakthrough"],
|
||||
}
|
||||
|
||||
for theme, keywords in theme_patterns.items():
|
||||
if any(keyword in full_text.lower() for keyword in keywords):
|
||||
themes.append(theme.title())
|
||||
|
||||
return themes
|
||||
|
||||
|
||||
def extract_keywords(result: Dict[str, Any]) -> List[tuple]:
|
||||
"""Extract keywords with frequency from research results."""
|
||||
from collections import Counter
|
||||
import re
|
||||
|
||||
# Combine all text
|
||||
summary = result.get("summary", "")
|
||||
findings = " ".join(str(f) for f in result.get("findings", []))
|
||||
full_text = f"{summary} {findings}".lower()
|
||||
|
||||
# Simple word extraction (in production, use NLP libraries)
|
||||
words = re.findall(r"\b[a-z]{4,}\b", full_text)
|
||||
|
||||
# Filter common words
|
||||
stopwords = {
|
||||
"that",
|
||||
"this",
|
||||
"with",
|
||||
"from",
|
||||
"have",
|
||||
"been",
|
||||
"were",
|
||||
"which",
|
||||
"their",
|
||||
"about",
|
||||
}
|
||||
words = [w for w in words if w not in stopwords]
|
||||
|
||||
# Count frequencies
|
||||
word_freq = Counter(words)
|
||||
|
||||
return word_freq.most_common(20)
|
||||
|
||||
|
||||
def extract_concepts(result: Dict[str, Any]) -> List[str]:
|
||||
"""Extract key concepts from research results."""
|
||||
concepts = []
|
||||
summary = result.get("summary", "")
|
||||
|
||||
# Simple concept patterns (in production, use NLP for entity extraction)
|
||||
concept_patterns = [
|
||||
r"quantum \w+",
|
||||
r"\w+ computing",
|
||||
r"\w+ algorithm",
|
||||
r"machine learning",
|
||||
r"artificial intelligence",
|
||||
r"\w+ technology",
|
||||
]
|
||||
|
||||
import re
|
||||
|
||||
for pattern in concept_patterns:
|
||||
matches = re.findall(pattern, summary.lower())
|
||||
concepts.extend(matches)
|
||||
|
||||
# Deduplicate and clean
|
||||
concepts = list(set(concepts))
|
||||
|
||||
return concepts[:10]
|
||||
|
||||
|
||||
def extract_technical_terms(result: Dict[str, Any]) -> List[str]:
|
||||
"""Extract technical terms from research results."""
|
||||
technical_terms = []
|
||||
|
||||
# Common technical term patterns
|
||||
tech_indicators = [
|
||||
"algorithm",
|
||||
"system",
|
||||
"protocol",
|
||||
"framework",
|
||||
"architecture",
|
||||
"quantum",
|
||||
"neural",
|
||||
"network",
|
||||
"model",
|
||||
"optimization",
|
||||
]
|
||||
|
||||
summary = result.get("summary", "").lower()
|
||||
import re
|
||||
|
||||
for indicator in tech_indicators:
|
||||
# Find words containing or adjacent to technical indicators
|
||||
pattern = rf"\b\w*{indicator}\w*\b"
|
||||
matches = re.findall(pattern, summary)
|
||||
technical_terms.extend(matches)
|
||||
|
||||
# Deduplicate
|
||||
technical_terms = list(set(technical_terms))
|
||||
|
||||
return technical_terms
|
||||
|
||||
|
||||
def demonstrate_batch_research():
|
||||
"""
|
||||
Show how to perform batch research on multiple topics.
|
||||
|
||||
Useful for:
|
||||
- Comparative analysis
|
||||
- Trend monitoring
|
||||
- Systematic reviews
|
||||
"""
|
||||
print("\n" + "=" * 70)
|
||||
print("BATCH RESEARCH PROCESSING")
|
||||
print("=" * 70)
|
||||
print("""
|
||||
Processing multiple research queries:
|
||||
- Efficient batch processing
|
||||
- Comparative analysis
|
||||
- Result aggregation
|
||||
""")
|
||||
|
||||
settings = create_settings_snapshot(
|
||||
overrides={
|
||||
"programmatic_mode": True,
|
||||
"search.tool": "wikipedia",
|
||||
}
|
||||
)
|
||||
|
||||
# Topics for batch research
|
||||
topics = [
|
||||
"Solar energy innovations",
|
||||
"Wind power technology",
|
||||
"Hydrogen fuel cells",
|
||||
]
|
||||
|
||||
batch_results = {}
|
||||
|
||||
print("\n📚 Batch Research:")
|
||||
for topic in topics:
|
||||
print(f"\n Researching: {topic}")
|
||||
result = quick_summary(
|
||||
query=topic,
|
||||
settings_snapshot=settings,
|
||||
iterations=1,
|
||||
questions_per_iteration=2,
|
||||
)
|
||||
batch_results[topic] = result
|
||||
print(f" ✓ Found {len(result.get('findings', []))} findings")
|
||||
|
||||
# Aggregate results
|
||||
print("\n📊 Aggregate Analysis:")
|
||||
total_findings = sum(
|
||||
len(r.get("findings", [])) for r in batch_results.values()
|
||||
)
|
||||
total_sources = sum(
|
||||
len(r.get("sources", [])) for r in batch_results.values()
|
||||
)
|
||||
|
||||
print(f" - Total topics researched: {len(topics)}")
|
||||
print(f" - Total findings: {total_findings}")
|
||||
print(f" - Total sources: {total_sources}")
|
||||
print(f" - Average findings per topic: {total_findings / len(topics):.1f}")
|
||||
|
||||
# Save batch results
|
||||
batch_file = "batch_research_results.json"
|
||||
with open(batch_file, "w", encoding="utf-8") as f:
|
||||
json.dump(batch_results, f, indent=2, default=str)
|
||||
print(f"\n✓ Batch results saved to: {batch_file}")
|
||||
|
||||
return batch_results
|
||||
|
||||
|
||||
def main():
|
||||
"""Run all advanced feature demonstrations."""
|
||||
print("=" * 70)
|
||||
print("LOCAL DEEP RESEARCH - ADVANCED FEATURES DEMONSTRATION")
|
||||
print("=" * 70)
|
||||
print("""
|
||||
This example demonstrates advanced programmatic features:
|
||||
1. Report generation with generate_report()
|
||||
2. Multiple export formats
|
||||
3. Result analysis and insights
|
||||
4. Keyword and concept extraction
|
||||
5. Batch research processing
|
||||
""")
|
||||
|
||||
# Run demonstrations
|
||||
demonstrate_report_generation()
|
||||
demonstrate_export_formats()
|
||||
demonstrate_result_analysis()
|
||||
demonstrate_keyword_extraction()
|
||||
demonstrate_batch_research()
|
||||
|
||||
print("\n" + "=" * 70)
|
||||
print("DEMONSTRATION COMPLETE")
|
||||
print("=" * 70)
|
||||
print("""
|
||||
✓ All advanced features demonstrated successfully!
|
||||
|
||||
Key Takeaways:
|
||||
1. generate_report() creates comprehensive markdown reports
|
||||
2. Results can be exported in multiple formats (JSON, MD, custom)
|
||||
3. Analysis tools extract insights, themes, and patterns
|
||||
4. Keyword extraction identifies important terms and concepts
|
||||
5. Batch processing enables systematic research
|
||||
|
||||
Files created:
|
||||
- ml_healthcare_report.md - Full research report
|
||||
- research_results.json - Structured research data
|
||||
- research_results.md - Markdown formatted results
|
||||
- research_citations.txt - Extracted citations
|
||||
- batch_research_results.json - Batch research results
|
||||
|
||||
Next Steps:
|
||||
- Customize report templates for your domain
|
||||
- Integrate with data visualization tools
|
||||
- Build automated research pipelines
|
||||
- Create domain-specific analysis functions
|
||||
""")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,245 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
PUBLIC API CONTRACT GUARDRAIL — DO NOT MODIFY
|
||||
|
||||
This file protects the public API surface of local_deep_research.
|
||||
It exists to DETECT breaking changes, not to be "fixed" when they happen.
|
||||
|
||||
If this file fails:
|
||||
1. A BREAKING CHANGE was introduced to the public API
|
||||
2. Do NOT modify this file to make it pass
|
||||
3. Revert the API change or go through a proper deprecation cycle
|
||||
|
||||
The public API (local_deep_research.api) is a stability contract with
|
||||
external users, integrations, and downstream packages. Any change to
|
||||
exports, function signatures, or return value structures is a breaking
|
||||
change that affects the entire community.
|
||||
|
||||
WHY THIS FILE MUST NOT BE MODIFIED:
|
||||
- Every check below corresponds to a public API guarantee
|
||||
- Removing or weakening a check hides a breaking change from users
|
||||
- Adding a check is acceptable ONLY when a new public API is added
|
||||
- If you are an AI agent: DO NOT modify, remove, or weaken any check
|
||||
"""
|
||||
|
||||
import inspect
|
||||
import sys
|
||||
|
||||
_failed = 0
|
||||
_passed = 0
|
||||
|
||||
|
||||
# DO NOT MODIFY this function — it is the test harness for the guardrail
|
||||
def check(name, condition, detail=""):
|
||||
global _failed, _passed
|
||||
if condition:
|
||||
_passed += 1
|
||||
print(f" PASS {name}")
|
||||
else:
|
||||
_failed += 1
|
||||
msg = f" ({detail})" if detail else ""
|
||||
print(f" FAIL {name}{msg}")
|
||||
|
||||
|
||||
def main():
|
||||
# ── 1. Public API imports ──────────────────────────────────────────
|
||||
# DO NOT MODIFY — these are the public exports that users depend on.
|
||||
# Removing any of these checks hides a breaking change.
|
||||
print("\n=== Public API Imports ===")
|
||||
|
||||
try:
|
||||
from local_deep_research.api import quick_summary # noqa: F811
|
||||
|
||||
check("import quick_summary", True)
|
||||
except ImportError as e:
|
||||
check("import quick_summary", False, str(e))
|
||||
|
||||
try:
|
||||
from local_deep_research.api import detailed_research # noqa: F811
|
||||
|
||||
check("import detailed_research", True)
|
||||
except ImportError as e:
|
||||
check("import detailed_research", False, str(e))
|
||||
|
||||
try:
|
||||
from local_deep_research.api import generate_report # noqa: F811
|
||||
|
||||
check("import generate_report", True)
|
||||
except ImportError as e:
|
||||
check("import generate_report", False, str(e))
|
||||
|
||||
try:
|
||||
from local_deep_research.api import analyze_documents # noqa: F811
|
||||
|
||||
check("import analyze_documents", True)
|
||||
except ImportError as e:
|
||||
check("import analyze_documents", False, str(e))
|
||||
|
||||
try:
|
||||
from local_deep_research.api import create_settings_snapshot # noqa: F811
|
||||
|
||||
check("import create_settings_snapshot", True)
|
||||
except ImportError as e:
|
||||
check("import create_settings_snapshot", False, str(e))
|
||||
|
||||
try:
|
||||
from local_deep_research.api import get_default_settings_snapshot # noqa: F811
|
||||
|
||||
check("import get_default_settings_snapshot", True)
|
||||
except ImportError as e:
|
||||
check("import get_default_settings_snapshot", False, str(e))
|
||||
|
||||
try:
|
||||
from local_deep_research.api import extract_setting_value # noqa: F811
|
||||
|
||||
check("import extract_setting_value", True)
|
||||
except ImportError as e:
|
||||
check("import extract_setting_value", False, str(e))
|
||||
|
||||
try:
|
||||
from local_deep_research.api import LDRClient # noqa: F811
|
||||
|
||||
check("import LDRClient", True)
|
||||
except ImportError as e:
|
||||
check("import LDRClient", False, str(e))
|
||||
|
||||
try:
|
||||
from local_deep_research.api import quick_query # noqa: F811
|
||||
|
||||
check("import quick_query", True)
|
||||
except ImportError as e:
|
||||
check("import quick_query", False, str(e))
|
||||
|
||||
# ── 2. Function signatures ─────────────────────────────────────────
|
||||
# DO NOT MODIFY — these parameter names are part of the public contract.
|
||||
# Renaming or removing parameters breaks all callers.
|
||||
print("\n=== Function Signatures ===")
|
||||
|
||||
sig = inspect.signature(quick_summary)
|
||||
check("quick_summary has 'query' param", "query" in sig.parameters)
|
||||
check("quick_summary has 'llms' param", "llms" in sig.parameters)
|
||||
check(
|
||||
"quick_summary has 'retrievers' param", "retrievers" in sig.parameters
|
||||
)
|
||||
|
||||
sig = inspect.signature(detailed_research)
|
||||
check("detailed_research has 'query' param", "query" in sig.parameters)
|
||||
check("detailed_research has 'llms' param", "llms" in sig.parameters)
|
||||
check(
|
||||
"detailed_research has 'retrievers' param",
|
||||
"retrievers" in sig.parameters,
|
||||
)
|
||||
|
||||
sig = inspect.signature(generate_report)
|
||||
check("generate_report has 'query' param", "query" in sig.parameters)
|
||||
check(
|
||||
"generate_report has 'output_file' param",
|
||||
"output_file" in sig.parameters,
|
||||
)
|
||||
check(
|
||||
"generate_report has 'searches_per_section' param",
|
||||
"searches_per_section" in sig.parameters,
|
||||
)
|
||||
|
||||
sig = inspect.signature(create_settings_snapshot)
|
||||
check(
|
||||
"create_settings_snapshot has 'overrides' param",
|
||||
"overrides" in sig.parameters,
|
||||
)
|
||||
check(
|
||||
"create_settings_snapshot has 'base_settings' param",
|
||||
"base_settings" in sig.parameters,
|
||||
)
|
||||
|
||||
# ── 3. Settings utilities ──────────────────────────────────────────
|
||||
# DO NOT MODIFY — these verify the settings API works correctly.
|
||||
# External users depend on create_settings_snapshot() returning a dict.
|
||||
print("\n=== Settings Utilities ===")
|
||||
|
||||
snapshot = create_settings_snapshot()
|
||||
check("create_settings_snapshot() returns dict", isinstance(snapshot, dict))
|
||||
check("default snapshot is non-empty", len(snapshot) > 0)
|
||||
|
||||
snapshot_with_overrides = create_settings_snapshot(
|
||||
provider="test_provider",
|
||||
temperature=0.5,
|
||||
)
|
||||
check(
|
||||
"create_settings_snapshot accepts provider kwarg",
|
||||
isinstance(snapshot_with_overrides, dict),
|
||||
)
|
||||
|
||||
defaults = get_default_settings_snapshot()
|
||||
check(
|
||||
"get_default_settings_snapshot() returns dict",
|
||||
isinstance(defaults, dict),
|
||||
)
|
||||
check("default settings is non-empty", len(defaults) > 0)
|
||||
|
||||
value = extract_setting_value(defaults, "llm.temperature")
|
||||
check(
|
||||
"extract_setting_value returns a value for llm.temperature",
|
||||
value is not None,
|
||||
)
|
||||
|
||||
# ── 4. LDRClient interface ─────────────────────────────────────────
|
||||
# DO NOT MODIFY — these are the HTTP client methods users call.
|
||||
# Removing any method breaks all HTTP-based integrations.
|
||||
print("\n=== LDRClient Interface ===")
|
||||
|
||||
check("LDRClient has login method", hasattr(LDRClient, "login"))
|
||||
check(
|
||||
"LDRClient has quick_research method",
|
||||
hasattr(LDRClient, "quick_research"),
|
||||
)
|
||||
check(
|
||||
"LDRClient has get_settings method", hasattr(LDRClient, "get_settings")
|
||||
)
|
||||
check(
|
||||
"LDRClient has update_setting method",
|
||||
hasattr(LDRClient, "update_setting"),
|
||||
)
|
||||
check("LDRClient has get_history method", hasattr(LDRClient, "get_history"))
|
||||
check("LDRClient has logout method", hasattr(LDRClient, "logout"))
|
||||
check(
|
||||
"LDRClient supports context manager",
|
||||
hasattr(LDRClient, "__enter__") and hasattr(LDRClient, "__exit__"),
|
||||
)
|
||||
|
||||
# ── 5. Callability ─────────────────────────────────────────────────
|
||||
# DO NOT MODIFY — verifies all exports are actually callable.
|
||||
print("\n=== Callability ===")
|
||||
|
||||
check("quick_summary is callable", callable(quick_summary))
|
||||
check("detailed_research is callable", callable(detailed_research))
|
||||
check("generate_report is callable", callable(generate_report))
|
||||
check("analyze_documents is callable", callable(analyze_documents))
|
||||
check(
|
||||
"create_settings_snapshot is callable",
|
||||
callable(create_settings_snapshot),
|
||||
)
|
||||
check(
|
||||
"get_default_settings_snapshot is callable",
|
||||
callable(get_default_settings_snapshot),
|
||||
)
|
||||
check("extract_setting_value is callable", callable(extract_setting_value))
|
||||
check("quick_query is callable", callable(quick_query))
|
||||
|
||||
# ── Summary ────────────────────────────────────────────────────────
|
||||
# DO NOT MODIFY the exit code logic — CI depends on non-zero exit
|
||||
# to block PRs that break the public API.
|
||||
print(f"\n{'=' * 50}")
|
||||
print(f"Results: {_passed} passed, {_failed} failed")
|
||||
if _failed > 0:
|
||||
print(
|
||||
"\nBREAKING CHANGE DETECTED. The public API has changed.\n"
|
||||
"Do NOT modify this file to make it pass.\n"
|
||||
"Revert the API change or follow a proper deprecation cycle."
|
||||
)
|
||||
sys.exit(1)
|
||||
else:
|
||||
print("\nAll API public contract checks passed.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,215 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Example of using a custom LLM with a custom retriever in Local Deep Research.
|
||||
|
||||
This demonstrates how to integrate your own LLM implementation and custom
|
||||
retrieval system for programmatic access.
|
||||
"""
|
||||
|
||||
from typing import List, Dict
|
||||
from langchain_ollama import ChatOllama, OllamaEmbeddings
|
||||
from langchain_core.retrievers import Document
|
||||
from langchain_community.vectorstores import FAISS
|
||||
|
||||
# Import the search system
|
||||
from local_deep_research.search_system import AdvancedSearchSystem
|
||||
|
||||
# Re-enable logging after import
|
||||
from loguru import logger
|
||||
import sys
|
||||
|
||||
logger.remove()
|
||||
# diagnose=False: loguru defaults to True, which renders repr() of every
|
||||
# local in every traceback frame on exception. Users copy this snippet
|
||||
# into their own scripts, so leaving the default on would propagate the
|
||||
# credential-in-traceback leak (#4185) wherever the snippet lands.
|
||||
logger.add(
|
||||
sys.stderr,
|
||||
level="INFO",
|
||||
format="{time} {level} {message}",
|
||||
diagnose=False,
|
||||
)
|
||||
logger.enable("local_deep_research")
|
||||
|
||||
|
||||
class CustomRetriever:
|
||||
"""Custom retriever that can fetch from multiple sources."""
|
||||
|
||||
def __init__(self):
|
||||
# Initialize with sample documents for demonstration
|
||||
self.documents = [
|
||||
{
|
||||
"content": "Quantum computing uses quantum bits (qubits) that can exist in superposition, "
|
||||
"allowing parallel computation of multiple states simultaneously.",
|
||||
"title": "Quantum Computing Fundamentals",
|
||||
"source": "quantum_basics.pdf",
|
||||
"metadata": {"topic": "quantum", "year": 2024},
|
||||
},
|
||||
{
|
||||
"content": "Machine learning algorithms can be categorized into supervised, unsupervised, "
|
||||
"and reinforcement learning approaches, each suited for different tasks.",
|
||||
"title": "ML Algorithm Categories",
|
||||
"source": "ml_overview.pdf",
|
||||
"metadata": {"topic": "ml", "year": 2024},
|
||||
},
|
||||
{
|
||||
"content": "Neural networks are inspired by biological neurons and consist of interconnected "
|
||||
"nodes that process information through weighted connections.",
|
||||
"title": "Neural Network Architecture",
|
||||
"source": "nn_architecture.pdf",
|
||||
"metadata": {"topic": "neural_networks", "year": 2023},
|
||||
},
|
||||
{
|
||||
"content": "Natural language processing enables computers to understand, interpret, and "
|
||||
"generate human language, powering applications like chatbots and translation.",
|
||||
"title": "NLP Applications",
|
||||
"source": "nlp_apps.pdf",
|
||||
"metadata": {"topic": "nlp", "year": 2024},
|
||||
},
|
||||
]
|
||||
|
||||
# Create embeddings for similarity search
|
||||
logger.info("Initializing custom retriever with embeddings...")
|
||||
self.embeddings = OllamaEmbeddings(model="nomic-embed-text")
|
||||
|
||||
# Create vector store from documents
|
||||
docs = [
|
||||
Document(
|
||||
page_content=doc["content"],
|
||||
metadata={
|
||||
"title": doc["title"],
|
||||
"source": doc["source"],
|
||||
**doc["metadata"],
|
||||
},
|
||||
)
|
||||
for doc in self.documents
|
||||
]
|
||||
self.vectorstore = FAISS.from_documents(docs, self.embeddings)
|
||||
|
||||
def retrieve(self, query: str, k: int = 3) -> List[Dict]:
|
||||
"""Custom retrieval logic."""
|
||||
logger.info(f"Custom Retriever: Searching for '{query}'")
|
||||
|
||||
# Use vector similarity search
|
||||
similar_docs = self.vectorstore.similarity_search(query, k=k)
|
||||
|
||||
# Convert to expected format
|
||||
results = []
|
||||
for i, doc in enumerate(similar_docs):
|
||||
results.append(
|
||||
{
|
||||
"title": doc.metadata.get("title", f"Document {i + 1}"),
|
||||
"link": doc.metadata.get("source", "custom_source"),
|
||||
"snippet": doc.page_content[:150] + "...",
|
||||
"full_content": doc.page_content,
|
||||
"rank": i + 1,
|
||||
"metadata": doc.metadata,
|
||||
}
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"Custom Retriever: Found {len(results)} relevant documents"
|
||||
)
|
||||
return results
|
||||
|
||||
|
||||
class CustomSearchEngine:
|
||||
"""Adapter to integrate custom retriever with the search system."""
|
||||
|
||||
def __init__(self, retriever: CustomRetriever, settings_snapshot=None):
|
||||
self.retriever = retriever
|
||||
self.settings_snapshot = settings_snapshot or {}
|
||||
|
||||
def run(self, query: str, research_context=None) -> List[Dict]:
|
||||
"""Execute search using custom retriever."""
|
||||
return self.retriever.retrieve(query, k=5)
|
||||
|
||||
|
||||
def main():
|
||||
"""Demonstrate custom LLM and retriever integration."""
|
||||
print("=== Custom LLM and Retriever Example ===\n")
|
||||
|
||||
# 1. Create custom LLM (just using regular Ollama for simplicity)
|
||||
print("1. Initializing LLM...")
|
||||
llm = ChatOllama(model="gemma3:12b", temperature=0.7)
|
||||
|
||||
# 2. Create custom retriever
|
||||
print("2. Setting up custom retriever...")
|
||||
custom_retriever = CustomRetriever()
|
||||
|
||||
# 3. Create settings
|
||||
settings = {
|
||||
"search.iterations": 2,
|
||||
"search.questions_per_iteration": 3,
|
||||
"search.strategy": "source-based",
|
||||
"rate_limiting.enabled": False, # Disable rate limiting for custom setup
|
||||
}
|
||||
|
||||
# 4. Create search engine adapter
|
||||
print("3. Creating search engine adapter...")
|
||||
search_engine = CustomSearchEngine(custom_retriever, settings)
|
||||
|
||||
# 5. Initialize the search system
|
||||
print("4. Initializing AdvancedSearchSystem with custom components...")
|
||||
# Pass programmatic_mode=True to avoid database dependencies
|
||||
search_system = AdvancedSearchSystem(
|
||||
llm=llm,
|
||||
search=search_engine,
|
||||
settings_snapshot=settings,
|
||||
programmatic_mode=True,
|
||||
)
|
||||
|
||||
# 6. Run research queries
|
||||
queries = [
|
||||
"How do quantum computers differ from classical computers?",
|
||||
"What are the main types of machine learning algorithms?",
|
||||
]
|
||||
|
||||
for query in queries:
|
||||
print(f"\n{'=' * 60}")
|
||||
print(f"Research Query: {query}")
|
||||
print("=" * 60)
|
||||
|
||||
result = search_system.analyze_topic(query)
|
||||
|
||||
# Display results
|
||||
print("\n=== FINDINGS ===")
|
||||
print(result["formatted_findings"])
|
||||
|
||||
# Show metadata
|
||||
print("\n=== SEARCH METADATA ===")
|
||||
print(f"• Total findings: {len(result['findings'])}")
|
||||
print(f"• Iterations: {result['iterations']}")
|
||||
|
||||
# Get actual sources from all_links_of_system or search_results
|
||||
all_links = result.get("all_links_of_system", [])
|
||||
for finding in result.get("findings", []):
|
||||
if "search_results" in finding and finding["search_results"]:
|
||||
all_links = finding["search_results"]
|
||||
break
|
||||
|
||||
print(f"• Sources found: {len(all_links)}")
|
||||
if all_links and len(all_links) > 0:
|
||||
print("\n=== SOURCES ===")
|
||||
for i, link in enumerate(all_links[:5], 1): # Show first 5
|
||||
if isinstance(link, dict):
|
||||
title = link.get("title", "No title")
|
||||
url = link.get("link", link.get("source", "Unknown"))
|
||||
print(f" [{i}] {title}")
|
||||
print(f" URL: {url}")
|
||||
|
||||
# Show generated questions
|
||||
if result.get("questions_by_iteration"):
|
||||
print("\n=== RESEARCH QUESTIONS GENERATED ===")
|
||||
for iteration, questions in result[
|
||||
"questions_by_iteration"
|
||||
].items():
|
||||
print(f"\nIteration {iteration}:")
|
||||
for q in questions[:3]: # Show first 3 questions
|
||||
print(f" • {q}")
|
||||
|
||||
print("\n✓ Custom LLM and Retriever integration successful!")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,352 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Hybrid Search Example for Local Deep Research
|
||||
|
||||
This example demonstrates how to combine multiple search sources:
|
||||
1. Multiple named retrievers for different document types
|
||||
2. Combining custom retrievers with web search
|
||||
3. Analyzing and comparing sources from different origins
|
||||
"""
|
||||
|
||||
from typing import List
|
||||
from langchain_core.retrievers import Document, BaseRetriever
|
||||
from langchain_community.vectorstores import FAISS
|
||||
from langchain_ollama import OllamaEmbeddings
|
||||
|
||||
from local_deep_research.api import quick_summary, detailed_research
|
||||
from local_deep_research.api.settings_utils import create_settings_snapshot
|
||||
|
||||
|
||||
class TechnicalDocsRetriever(BaseRetriever):
|
||||
"""Mock retriever for technical documentation."""
|
||||
|
||||
def get_relevant_documents(self, query: str) -> List[Document]:
|
||||
"""Return mock technical documents."""
|
||||
# In a real scenario, this would search actual technical docs
|
||||
return [
|
||||
Document(
|
||||
page_content=f"Technical specification for {query}: Implementation requires careful consideration of system architecture, performance metrics, and scalability factors.",
|
||||
metadata={
|
||||
"source": "tech_docs",
|
||||
"type": "specification",
|
||||
"title": f"Technical Spec: {query}",
|
||||
},
|
||||
),
|
||||
Document(
|
||||
page_content=f"Best practices for {query}: Follow industry standards, implement proper error handling, and ensure comprehensive testing coverage.",
|
||||
metadata={
|
||||
"source": "tech_docs",
|
||||
"type": "best_practices",
|
||||
"title": f"Best Practices: {query}",
|
||||
},
|
||||
),
|
||||
]
|
||||
|
||||
async def aget_relevant_documents(self, query: str) -> List[Document]:
|
||||
"""Async version."""
|
||||
return self.get_relevant_documents(query)
|
||||
|
||||
|
||||
class BusinessDocsRetriever(BaseRetriever):
|
||||
"""Mock retriever for business/strategy documents."""
|
||||
|
||||
def get_relevant_documents(self, query: str) -> List[Document]:
|
||||
"""Return mock business documents."""
|
||||
return [
|
||||
Document(
|
||||
page_content=f"Business implications of {query}: Consider market impact, ROI analysis, and strategic alignment with organizational goals.",
|
||||
metadata={
|
||||
"source": "business_docs",
|
||||
"type": "strategy",
|
||||
"title": f"Business Strategy: {query}",
|
||||
},
|
||||
),
|
||||
Document(
|
||||
page_content=f"Cost-benefit analysis for {query}: Initial investment requirements, expected returns, and risk assessment factors.",
|
||||
metadata={
|
||||
"source": "business_docs",
|
||||
"type": "analysis",
|
||||
"title": f"Cost Analysis: {query}",
|
||||
},
|
||||
),
|
||||
]
|
||||
|
||||
async def aget_relevant_documents(self, query: str) -> List[Document]:
|
||||
"""Async version."""
|
||||
return self.get_relevant_documents(query)
|
||||
|
||||
|
||||
def create_knowledge_base_retriever() -> BaseRetriever:
|
||||
"""Create a FAISS-based retriever with sample knowledge base documents."""
|
||||
documents = [
|
||||
Document(
|
||||
page_content="Machine learning models require training data, validation strategies, and performance metrics for evaluation.",
|
||||
metadata={"source": "ml_knowledge_base", "topic": "ml_basics"},
|
||||
),
|
||||
Document(
|
||||
page_content="Cloud computing provides scalable infrastructure, reducing capital expenditure and enabling flexible resource allocation.",
|
||||
metadata={
|
||||
"source": "cloud_knowledge_base",
|
||||
"topic": "cloud_benefits",
|
||||
},
|
||||
),
|
||||
Document(
|
||||
page_content="Agile methodology emphasizes iterative development, customer collaboration, and responding to change.",
|
||||
metadata={"source": "project_knowledge_base", "topic": "agile"},
|
||||
),
|
||||
Document(
|
||||
page_content="Data privacy regulations like GDPR require explicit consent, data minimization, and user rights management.",
|
||||
metadata={
|
||||
"source": "compliance_knowledge_base",
|
||||
"topic": "privacy",
|
||||
},
|
||||
),
|
||||
]
|
||||
|
||||
# Create embeddings and vector store
|
||||
embeddings = OllamaEmbeddings(model="nomic-embed-text")
|
||||
vectorstore = FAISS.from_documents(documents, embeddings)
|
||||
return vectorstore.as_retriever(search_kwargs={"k": 2})
|
||||
|
||||
|
||||
def demonstrate_multiple_retrievers():
|
||||
"""Show how to use multiple named retrievers for different document types."""
|
||||
print("=" * 70)
|
||||
print("MULTIPLE NAMED RETRIEVERS")
|
||||
print("=" * 70)
|
||||
print("""
|
||||
Using multiple specialized retrievers:
|
||||
- Technical documentation retriever
|
||||
- Business documentation retriever
|
||||
- Knowledge base retriever
|
||||
Each provides different perspectives on the same topic.
|
||||
""")
|
||||
|
||||
# Create different retrievers
|
||||
tech_retriever = TechnicalDocsRetriever()
|
||||
business_retriever = BusinessDocsRetriever()
|
||||
kb_retriever = create_knowledge_base_retriever()
|
||||
|
||||
# Configure settings. Registered retrievers are addressable by name;
|
||||
# with the default langgraph-agent strategy, every registered retriever
|
||||
# is also exposed to the research agent as a search tool.
|
||||
settings = create_settings_snapshot(
|
||||
{
|
||||
"search.tool": "knowledge_base", # Primary retriever
|
||||
}
|
||||
)
|
||||
|
||||
# Use multiple retrievers in research
|
||||
result = quick_summary(
|
||||
query="Implementing machine learning in production",
|
||||
settings_snapshot=settings,
|
||||
retrievers={
|
||||
"technical": tech_retriever,
|
||||
"business": business_retriever,
|
||||
"knowledge_base": kb_retriever,
|
||||
},
|
||||
search_tool="knowledge_base", # Primary retriever (others stay available)
|
||||
iterations=2,
|
||||
questions_per_iteration=2,
|
||||
programmatic_mode=True,
|
||||
)
|
||||
|
||||
print("\nResearch Summary (first 400 chars):")
|
||||
print(result["summary"][:400] + "...")
|
||||
|
||||
# Analyze sources by type
|
||||
sources = result.get("sources", [])
|
||||
print(f"\nTotal sources found: {len(sources)}")
|
||||
|
||||
# Group sources by retriever
|
||||
source_types = {}
|
||||
for source in sources:
|
||||
if isinstance(source, dict):
|
||||
source_type = source.get("metadata", {}).get("source", "unknown")
|
||||
else:
|
||||
source_type = "other"
|
||||
source_types[source_type] = source_types.get(source_type, 0) + 1
|
||||
|
||||
print("\nSources by retriever:")
|
||||
for stype, count in source_types.items():
|
||||
print(f" - {stype}: {count} sources")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def demonstrate_retriever_plus_web():
|
||||
"""Show how to combine custom retrievers with web search."""
|
||||
print("\n" + "=" * 70)
|
||||
print("RETRIEVER + WEB SEARCH COMBINATION")
|
||||
print("=" * 70)
|
||||
print("""
|
||||
Combining internal knowledge with web search:
|
||||
- Internal: Custom retriever with proprietary knowledge
|
||||
- External: Wikipedia for general context
|
||||
This provides both specific and general information.
|
||||
""")
|
||||
|
||||
# Create internal knowledge retriever
|
||||
internal_retriever = create_knowledge_base_retriever()
|
||||
|
||||
# Configure settings to use both retriever and web
|
||||
settings = create_settings_snapshot(
|
||||
{
|
||||
"search.tool": "wikipedia", # Also use Wikipedia
|
||||
}
|
||||
)
|
||||
|
||||
# Research combining internal and external sources
|
||||
result = detailed_research(
|
||||
query="Best practices for cloud migration",
|
||||
settings_snapshot=settings,
|
||||
retrievers={
|
||||
"internal_kb": internal_retriever,
|
||||
},
|
||||
search_tool="wikipedia", # Also search Wikipedia
|
||||
search_strategy="source-based",
|
||||
iterations=2,
|
||||
questions_per_iteration=3,
|
||||
programmatic_mode=True,
|
||||
)
|
||||
|
||||
print(f"\nResearch ID: {result['research_id']}")
|
||||
print(f"Summary length: {len(result['summary'])} characters")
|
||||
|
||||
# Analyze source distribution
|
||||
sources = result.get("sources", [])
|
||||
internal_sources = 0
|
||||
external_sources = 0
|
||||
|
||||
for source in sources:
|
||||
if isinstance(source, dict) and "knowledge_base" in str(source):
|
||||
internal_sources += 1
|
||||
else:
|
||||
external_sources += 1
|
||||
|
||||
print("\nSource distribution:")
|
||||
print(f" - Internal knowledge base: {internal_sources} sources")
|
||||
print(f" - External (Wikipedia): {external_sources} sources")
|
||||
|
||||
# Show how different sources complement each other
|
||||
print("\nComplementary insights from hybrid search:")
|
||||
print(
|
||||
" - Internal sources provide: Specific procedures, proprietary knowledge"
|
||||
)
|
||||
print(
|
||||
" - External sources provide: Industry context, general best practices"
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def demonstrate_source_analysis():
|
||||
"""Show how to analyze and compare sources from different origins."""
|
||||
print("\n" + "=" * 70)
|
||||
print("SOURCE ANALYSIS AND COMPARISON")
|
||||
print("=" * 70)
|
||||
print("""
|
||||
Analyzing source quality and relevance:
|
||||
- Track source origins
|
||||
- Compare information consistency
|
||||
- Identify unique insights from each source type
|
||||
""")
|
||||
|
||||
# Create multiple retrievers
|
||||
tech_retriever = TechnicalDocsRetriever()
|
||||
business_retriever = BusinessDocsRetriever()
|
||||
|
||||
settings = create_settings_snapshot(
|
||||
{
|
||||
"search.tool": "wikipedia",
|
||||
}
|
||||
)
|
||||
|
||||
# Run research with detailed source tracking
|
||||
result = quick_summary(
|
||||
query="Artificial intelligence implementation strategies",
|
||||
settings_snapshot=settings,
|
||||
retrievers={
|
||||
"technical": tech_retriever,
|
||||
"business": business_retriever,
|
||||
},
|
||||
search_tool="wikipedia", # Also use web search
|
||||
iterations=2,
|
||||
questions_per_iteration=2,
|
||||
programmatic_mode=True,
|
||||
)
|
||||
|
||||
# Detailed source analysis
|
||||
print("\nSource Analysis:")
|
||||
sources = result.get("sources", [])
|
||||
|
||||
# Categorize sources
|
||||
source_categories = {"technical": [], "business": [], "web": []}
|
||||
|
||||
for source in sources:
|
||||
if isinstance(source, dict):
|
||||
source_type = source.get("metadata", {}).get("source", "")
|
||||
if "tech" in source_type:
|
||||
source_categories["technical"].append(source)
|
||||
elif "business" in source_type:
|
||||
source_categories["business"].append(source)
|
||||
else:
|
||||
source_categories["web"].append(source)
|
||||
else:
|
||||
source_categories["web"].append(source)
|
||||
|
||||
# Report on each category
|
||||
for category, category_sources in source_categories.items():
|
||||
print(f"\n{category.upper()} Sources ({len(category_sources)}):")
|
||||
if category_sources:
|
||||
for i, source in enumerate(category_sources[:2], 1): # Show first 2
|
||||
if isinstance(source, dict):
|
||||
title = source.get("metadata", {}).get("title", "Untitled")
|
||||
print(f" {i}. {title}")
|
||||
else:
|
||||
print(f" {i}. {str(source)[:60]}...")
|
||||
|
||||
# Show findings breakdown
|
||||
findings = result.get("findings", [])
|
||||
print(f"\nTotal findings: {len(findings)}")
|
||||
print("Findings provide integrated insights from all source types")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def main():
|
||||
"""Run all hybrid search demonstrations."""
|
||||
print("=" * 70)
|
||||
print("LOCAL DEEP RESEARCH - HYBRID SEARCH DEMONSTRATION")
|
||||
print("=" * 70)
|
||||
print("""
|
||||
This example shows how to combine multiple search sources:
|
||||
- Custom retrievers for proprietary knowledge
|
||||
- Web search engines for public information
|
||||
- Source analysis across origins
|
||||
""")
|
||||
|
||||
# Run demonstrations
|
||||
demonstrate_multiple_retrievers()
|
||||
demonstrate_retriever_plus_web()
|
||||
demonstrate_source_analysis()
|
||||
|
||||
print("\n" + "=" * 70)
|
||||
print("KEY TAKEAWAYS")
|
||||
print("=" * 70)
|
||||
print("""
|
||||
1. Multiple Retrievers: Use specialized retrievers for different document types
|
||||
2. Hybrid Search: Combine internal knowledge with web search for comprehensive results
|
||||
3. Source Analysis: Track and analyze sources to understand information origin
|
||||
|
||||
Best Practices:
|
||||
- Name your retrievers descriptively for easy tracking
|
||||
- Balance internal and external sources based on your needs
|
||||
- Use source analysis to verify information consistency
|
||||
""")
|
||||
|
||||
print("\n✓ Hybrid search demonstration complete!")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,97 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Minimal working example for programmatic access to Local Deep Research.
|
||||
|
||||
This shows how to use the core functionality without database dependencies.
|
||||
"""
|
||||
|
||||
from langchain_ollama import ChatOllama
|
||||
from local_deep_research.search_system import AdvancedSearchSystem
|
||||
|
||||
# Re-enable logging after import (it gets disabled in __init__.py)
|
||||
from loguru import logger
|
||||
import sys
|
||||
|
||||
logger.remove()
|
||||
# diagnose=False: loguru defaults to True, which renders repr() of every
|
||||
# local in every traceback frame on exception. Users copy this snippet
|
||||
# into their own scripts, so leaving the default on would propagate the
|
||||
# credential-in-traceback leak (#4185) wherever the snippet lands.
|
||||
logger.add(
|
||||
sys.stderr,
|
||||
level="WARNING",
|
||||
format="{time} {level} {message}",
|
||||
diagnose=False,
|
||||
)
|
||||
logger.enable("local_deep_research")
|
||||
|
||||
|
||||
class MinimalSearchEngine:
|
||||
"""Minimal search engine that returns hardcoded results."""
|
||||
|
||||
def __init__(self, settings_snapshot=None):
|
||||
self.settings_snapshot = settings_snapshot or {}
|
||||
|
||||
def run(self, query, research_context=None):
|
||||
"""Return some fake search results."""
|
||||
return [
|
||||
{
|
||||
"title": "Introduction to AI",
|
||||
"link": "https://example.com/ai-intro",
|
||||
"snippet": "Artificial Intelligence (AI) is the simulation of human intelligence...",
|
||||
"full_content": "Full article about AI basics...",
|
||||
"rank": 1,
|
||||
},
|
||||
{
|
||||
"title": "Machine Learning Explained",
|
||||
"link": "https://example.com/ml-explained",
|
||||
"snippet": "Machine learning is a subset of AI that enables systems to learn...",
|
||||
"full_content": "Detailed explanation of machine learning...",
|
||||
"rank": 2,
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def main():
|
||||
"""Minimal example of programmatic access."""
|
||||
print("=== Minimal Local Deep Research Example ===\n")
|
||||
|
||||
# 1. Create LLM
|
||||
print("1. Creating Ollama LLM...")
|
||||
llm = ChatOllama(model="gemma3:12b")
|
||||
|
||||
# 2. Create minimal search engine
|
||||
print("2. Creating minimal search engine...")
|
||||
|
||||
# Settings for search system (without programmatic_mode)
|
||||
settings = {
|
||||
"search.iterations": 1,
|
||||
"search.strategy": "direct",
|
||||
}
|
||||
|
||||
search = MinimalSearchEngine(settings)
|
||||
|
||||
# 3. Create search system
|
||||
print("3. Creating AdvancedSearchSystem...")
|
||||
# IMPORTANT: Pass programmatic_mode=True to avoid database dependencies
|
||||
system = AdvancedSearchSystem(
|
||||
llm=llm,
|
||||
search=search,
|
||||
settings_snapshot=settings,
|
||||
programmatic_mode=True,
|
||||
)
|
||||
|
||||
# 4. Run a search
|
||||
print("\n4. Running search...")
|
||||
result = system.analyze_topic("What is artificial intelligence?")
|
||||
|
||||
# 5. Show results
|
||||
print("\n=== RESULTS ===")
|
||||
print(f"Found {len(result['findings'])} findings")
|
||||
print(f"\nSummary:\n{result['current_knowledge']}")
|
||||
|
||||
print("\n✓ Success! Programmatic access works without database.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,225 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Search Strategies Example for Local Deep Research
|
||||
|
||||
This example demonstrates the two main search strategies:
|
||||
1. source-based: Comprehensive research with source citation
|
||||
2. focused-iteration: Iterative refinement of research questions
|
||||
|
||||
Each strategy has different strengths and use cases.
|
||||
"""
|
||||
|
||||
from local_deep_research.api import quick_summary, detailed_research
|
||||
from local_deep_research.api.settings_utils import create_settings_snapshot
|
||||
|
||||
|
||||
def demonstrate_source_based_strategy():
|
||||
"""
|
||||
Source-based strategy:
|
||||
- Focuses on gathering and synthesizing information from multiple sources
|
||||
- Provides detailed citations and source tracking
|
||||
- Best for: Academic research, fact-checking, comprehensive reports
|
||||
"""
|
||||
print("=" * 70)
|
||||
print("SOURCE-BASED STRATEGY")
|
||||
print("=" * 70)
|
||||
print("""
|
||||
This strategy:
|
||||
- Systematically searches for sources related to your topic
|
||||
- Synthesizes information across multiple sources
|
||||
- Provides detailed citations for all claims
|
||||
- Ideal for research requiring source verification
|
||||
""")
|
||||
|
||||
# Configure settings for programmatic mode
|
||||
settings = create_settings_snapshot(
|
||||
{
|
||||
"search.tool": "wikipedia", # Using Wikipedia for demonstration
|
||||
}
|
||||
)
|
||||
|
||||
# Run research with source-based strategy
|
||||
result = detailed_research(
|
||||
query="What are the main causes of climate change?",
|
||||
settings_snapshot=settings,
|
||||
search_strategy="source-based", # Explicitly set strategy
|
||||
iterations=2, # Number of research iterations
|
||||
questions_per_iteration=3, # Questions to explore per iteration
|
||||
programmatic_mode=True,
|
||||
)
|
||||
|
||||
print(f"Research ID: {result['research_id']}")
|
||||
print("\nSummary (first 500 chars):")
|
||||
print(result["summary"][:500] + "...")
|
||||
|
||||
# Show sources found
|
||||
sources = result.get("sources", [])
|
||||
print(f"\nSources found: {len(sources)}")
|
||||
if sources:
|
||||
print("\nFirst 3 sources:")
|
||||
for i, source in enumerate(sources[:3], 1):
|
||||
print(f" {i}. {source}")
|
||||
|
||||
# Show the questions that were researched
|
||||
questions = result.get("questions", {})
|
||||
print(f"\nQuestions researched across {len(questions)} iterations:")
|
||||
for iteration, qs in questions.items():
|
||||
print(f"\n Iteration {iteration}:")
|
||||
for q in qs[:2]: # Show first 2 questions per iteration
|
||||
print(f" - {q}")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def demonstrate_focused_iteration_strategy():
|
||||
"""
|
||||
Focused-iteration strategy:
|
||||
- Iteratively refines the research based on previous findings
|
||||
- Adapts questions based on what's been learned
|
||||
- Best for: Deep dives, evolving research questions, exploratory research
|
||||
"""
|
||||
print("\n" + "=" * 70)
|
||||
print("FOCUSED-ITERATION STRATEGY")
|
||||
print("=" * 70)
|
||||
print("""
|
||||
This strategy:
|
||||
- Starts with initial research on the topic
|
||||
- Analyzes findings to generate more targeted questions
|
||||
- Iteratively refines understanding through multiple rounds
|
||||
- Ideal for complex topics requiring deep exploration
|
||||
""")
|
||||
|
||||
# Configure settings
|
||||
settings = create_settings_snapshot(
|
||||
{
|
||||
"search.tool": "wikipedia",
|
||||
}
|
||||
)
|
||||
|
||||
# Run research with focused-iteration strategy
|
||||
result = quick_summary(
|
||||
query="How do neural networks learn?",
|
||||
settings_snapshot=settings,
|
||||
search_strategy="focused-iteration", # Use focused iteration
|
||||
iterations=3, # More iterations for deeper exploration
|
||||
questions_per_iteration=2, # Fewer but more focused questions
|
||||
temperature=0.7, # Slightly higher for creative question generation
|
||||
programmatic_mode=True,
|
||||
)
|
||||
|
||||
print("\nSummary (first 500 chars):")
|
||||
print(result["summary"][:500] + "...")
|
||||
|
||||
# Show how questions evolved
|
||||
questions = result.get("questions", {})
|
||||
if questions:
|
||||
print("\nQuestion evolution across iterations:")
|
||||
for iteration, qs in questions.items():
|
||||
print(f"\n Iteration {iteration}:")
|
||||
for q in qs:
|
||||
print(f" - {q}")
|
||||
|
||||
# Show findings
|
||||
findings = result.get("findings", [])
|
||||
print(f"\nKey findings: {len(findings)}")
|
||||
if findings:
|
||||
print("\nFirst 2 findings:")
|
||||
for i, finding in enumerate(findings[:2], 1):
|
||||
text = (
|
||||
finding.get("text", "N/A")
|
||||
if isinstance(finding, dict)
|
||||
else str(finding)
|
||||
)
|
||||
print(f" {i}. {text[:150]}...")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def compare_strategies():
|
||||
"""
|
||||
Direct comparison of both strategies on the same topic.
|
||||
"""
|
||||
print("\n" + "=" * 70)
|
||||
print("STRATEGY COMPARISON")
|
||||
print("=" * 70)
|
||||
print(
|
||||
"\nComparing both strategies on the same topic: 'Quantum Computing Applications'\n"
|
||||
)
|
||||
|
||||
settings = create_settings_snapshot(
|
||||
{
|
||||
"search.tool": "wikipedia",
|
||||
}
|
||||
)
|
||||
|
||||
# Same topic, different strategies
|
||||
topic = "Quantum computing applications in cryptography"
|
||||
|
||||
print("1. Source-based approach:")
|
||||
source_result = quick_summary(
|
||||
query=topic,
|
||||
settings_snapshot=settings,
|
||||
search_strategy="source-based",
|
||||
iterations=2,
|
||||
questions_per_iteration=3,
|
||||
programmatic_mode=True,
|
||||
)
|
||||
print(f" - Sources found: {len(source_result.get('sources', []))}")
|
||||
print(f" - Summary length: {len(source_result.get('summary', ''))} chars")
|
||||
print(f" - Findings: {len(source_result.get('findings', []))}")
|
||||
|
||||
print("\n2. Focused-iteration approach:")
|
||||
focused_result = quick_summary(
|
||||
query=topic,
|
||||
settings_snapshot=settings,
|
||||
search_strategy="focused-iteration",
|
||||
iterations=2,
|
||||
questions_per_iteration=3,
|
||||
programmatic_mode=True,
|
||||
)
|
||||
print(f" - Sources found: {len(focused_result.get('sources', []))}")
|
||||
print(
|
||||
f" - Summary length: {len(focused_result.get('summary', ''))} chars"
|
||||
)
|
||||
print(f" - Findings: {len(focused_result.get('findings', []))}")
|
||||
|
||||
print("\n" + "=" * 70)
|
||||
print("WHEN TO USE EACH STRATEGY")
|
||||
print("=" * 70)
|
||||
print("""
|
||||
Use SOURCE-BASED when you need:
|
||||
- Comprehensive coverage with citations
|
||||
- Academic or professional research
|
||||
- Fact-checking and verification
|
||||
- Documentation with source tracking
|
||||
|
||||
Use FOCUSED-ITERATION when you need:
|
||||
- Deep exploration of complex topics
|
||||
- Adaptive research that evolves
|
||||
- Discovery of unexpected connections
|
||||
- Exploratory or investigative research
|
||||
""")
|
||||
|
||||
|
||||
def main():
|
||||
"""Run all demonstrations."""
|
||||
print("=" * 70)
|
||||
print("LOCAL DEEP RESEARCH - SEARCH STRATEGIES DEMONSTRATION")
|
||||
print("=" * 70)
|
||||
|
||||
# Demonstrate each strategy
|
||||
demonstrate_source_based_strategy()
|
||||
demonstrate_focused_iteration_strategy()
|
||||
|
||||
# Compare strategies
|
||||
compare_strategies()
|
||||
|
||||
print("\n✓ Search strategies demonstration complete!")
|
||||
print("\nNote: Both strategies can be combined with different search tools")
|
||||
print(
|
||||
"(wikipedia, arxiv, searxng, etc.) and custom parameters for optimal results."
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,185 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Example of using SearXNG search engine with Local Deep Research.
|
||||
|
||||
This demonstrates how to use SearXNG for web search in programmatic mode.
|
||||
Note: Requires a running SearXNG instance.
|
||||
"""
|
||||
|
||||
import os
|
||||
from langchain_ollama import ChatOllama
|
||||
from local_deep_research.search_system import AdvancedSearchSystem
|
||||
from local_deep_research.web_search_engines.engines.search_engine_searxng import (
|
||||
SearXNGSearchEngine,
|
||||
)
|
||||
|
||||
# Re-enable logging
|
||||
from loguru import logger
|
||||
import sys
|
||||
|
||||
logger.remove()
|
||||
# diagnose=False: loguru defaults to True, which renders repr() of every
|
||||
# local in every traceback frame on exception. Users copy this snippet
|
||||
# into their own scripts, so leaving the default on would propagate the
|
||||
# credential-in-traceback leak (#4185) wherever the snippet lands.
|
||||
logger.add(
|
||||
sys.stderr,
|
||||
level="INFO",
|
||||
format="{time} {level} {message}",
|
||||
diagnose=False,
|
||||
)
|
||||
logger.enable("local_deep_research")
|
||||
|
||||
|
||||
def main():
|
||||
"""Demonstrate using SearXNG with Local Deep Research."""
|
||||
print("=== SearXNG Search Engine Example ===\n")
|
||||
|
||||
# Check if SearXNG URL is configured
|
||||
searxng_url = os.getenv("SEARXNG_URL", "http://localhost:8080")
|
||||
print(f"Using SearXNG instance at: {searxng_url}")
|
||||
print(
|
||||
"(Set SEARXNG_URL environment variable to use a different instance)\n"
|
||||
)
|
||||
|
||||
# 1. Create LLM
|
||||
print("1. Setting up Ollama LLM...")
|
||||
llm = ChatOllama(model="gemma3:12b", temperature=0.3)
|
||||
|
||||
# 2. Configure settings
|
||||
settings = {
|
||||
"search.iterations": 2,
|
||||
"search.questions_per_iteration": 3,
|
||||
"search.strategy": "source-based",
|
||||
"rate_limiting.enabled": False, # Disable rate limiting for demo
|
||||
# SearXNG specific settings
|
||||
"search_engines.searxng.base_url": searxng_url,
|
||||
"search_engines.searxng.timeout": 30,
|
||||
"search_engines.searxng.categories": ["general", "science"],
|
||||
"search_engines.searxng.engines": ["google", "duckduckgo", "bing"],
|
||||
"search_engines.searxng.language": "en",
|
||||
"search_engines.searxng.time_range": "", # all time
|
||||
"search_engines.searxng.safesearch": 0, # 0=off, 1=moderate, 2=strict
|
||||
}
|
||||
|
||||
# 3. Create SearXNG search engine
|
||||
print("2. Initializing SearXNG search engine...")
|
||||
try:
|
||||
search_engine = SearXNGSearchEngine(settings_snapshot=settings)
|
||||
|
||||
# Test the connection
|
||||
print(" Testing SearXNG connection...")
|
||||
test_results = search_engine.run("test query", research_context={})
|
||||
if test_results:
|
||||
print(
|
||||
f" ✓ SearXNG is working! Got {len(test_results)} test results."
|
||||
)
|
||||
else:
|
||||
print(" ⚠ SearXNG returned no results for test query.")
|
||||
except Exception as e:
|
||||
print(f"\n⚠ Error connecting to SearXNG: {e}")
|
||||
print("\nPlease ensure SearXNG is running. You can start it with:")
|
||||
print(" docker run -p 8888:8080 searxng/searxng")
|
||||
print("\nFalling back to mock search engine for demonstration...")
|
||||
|
||||
# Fallback to mock search engine
|
||||
class MockSearchEngine:
|
||||
def __init__(self, settings_snapshot=None):
|
||||
self.settings_snapshot = settings_snapshot or {}
|
||||
|
||||
def run(self, query, research_context=None):
|
||||
return [
|
||||
{
|
||||
"title": f"Result for: {query}",
|
||||
"link": "https://example.com/result",
|
||||
"snippet": f"This is a mock result for the query: {query}. "
|
||||
"In a real scenario, SearXNG would provide actual web search results.",
|
||||
"full_content": "Full content would be fetched here...",
|
||||
"rank": 1,
|
||||
}
|
||||
]
|
||||
|
||||
search_engine = MockSearchEngine(settings)
|
||||
|
||||
# 4. Create the search system
|
||||
print("3. Creating AdvancedSearchSystem...")
|
||||
# Pass programmatic_mode=True to disable database dependencies
|
||||
search_system = AdvancedSearchSystem(
|
||||
llm=llm,
|
||||
search=search_engine,
|
||||
settings_snapshot=settings,
|
||||
programmatic_mode=True,
|
||||
)
|
||||
|
||||
# 5. Run research queries
|
||||
queries = [
|
||||
"What are the latest developments in quantum computing in 2024?",
|
||||
"How does CRISPR gene editing technology work?",
|
||||
]
|
||||
|
||||
for query in queries:
|
||||
print(f"\n{'=' * 60}")
|
||||
print(f"Research Query: {query}")
|
||||
print("=" * 60)
|
||||
|
||||
try:
|
||||
result = search_system.analyze_topic(query)
|
||||
|
||||
# Display results
|
||||
print("\n=== RESEARCH FINDINGS ===")
|
||||
if result.get("formatted_findings"):
|
||||
print(result["formatted_findings"])
|
||||
else:
|
||||
print(
|
||||
"Summary:", result.get("current_knowledge", "No findings")
|
||||
)
|
||||
|
||||
# Show metadata
|
||||
print("\n=== METADATA ===")
|
||||
print(f"• Iterations completed: {result.get('iterations', 0)}")
|
||||
print(f"• Total findings: {len(result.get('findings', []))}")
|
||||
|
||||
# Show search sources from all_links_of_system or search_results in findings
|
||||
all_links = result.get("all_links_of_system", [])
|
||||
|
||||
# Also check findings for search_results
|
||||
for finding in result.get("findings", []):
|
||||
if "search_results" in finding and finding["search_results"]:
|
||||
all_links = finding["search_results"]
|
||||
break
|
||||
|
||||
if all_links:
|
||||
print(f"• Sources found: {len(all_links)}")
|
||||
for i, link in enumerate(
|
||||
all_links[:5], 1
|
||||
): # Show first 5 sources
|
||||
if isinstance(link, dict):
|
||||
title = link.get("title", "No title")
|
||||
url = link.get("link", "Unknown")
|
||||
print(f" [{i}] {title}")
|
||||
print(f" {url}")
|
||||
|
||||
# Show generated questions
|
||||
if result.get("questions_by_iteration"):
|
||||
print("\n=== RESEARCH QUESTIONS ===")
|
||||
for iteration, questions in result[
|
||||
"questions_by_iteration"
|
||||
].items():
|
||||
print(f"Iteration {iteration}:")
|
||||
for q in questions[
|
||||
:2
|
||||
]: # Show first 2 questions per iteration
|
||||
print(f" • {q}")
|
||||
|
||||
except Exception as e:
|
||||
logger.exception("Error during research")
|
||||
print(f"\n⚠ Error: {e}")
|
||||
|
||||
print("\n✓ SearXNG integration example completed!")
|
||||
print(
|
||||
"\nNote: For best results, ensure SearXNG is properly configured with multiple search engines."
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,86 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Simple Programmatic API Example for Local Deep Research
|
||||
|
||||
Quick example showing how to use the LDR Python API directly.
|
||||
"""
|
||||
|
||||
from local_deep_research.api import (
|
||||
detailed_research,
|
||||
quick_summary,
|
||||
generate_report,
|
||||
)
|
||||
from local_deep_research.api.settings_utils import (
|
||||
create_settings_snapshot,
|
||||
)
|
||||
|
||||
# Use default settings with minimal overrides
|
||||
# This provides all necessary settings with sensible defaults
|
||||
settings_snapshot = create_settings_snapshot(
|
||||
overrides={
|
||||
"search.tool": "wikipedia", # Use Wikipedia for this example
|
||||
"api.allow_file_output": True, # Allow generate_report to save files
|
||||
}
|
||||
)
|
||||
|
||||
# Alternative: Use completely default settings
|
||||
# settings_snapshot = get_default_settings_snapshot()
|
||||
|
||||
# Example 1: Quick Summary
|
||||
print("=== Quick Summary ===")
|
||||
result = quick_summary(
|
||||
"What is machine learning?",
|
||||
settings_snapshot=settings_snapshot,
|
||||
programmatic_mode=True,
|
||||
)
|
||||
print(f"Summary: {result['summary'][:300]}...")
|
||||
print(f"Found {len(result.get('findings', []))} findings")
|
||||
|
||||
# Example 2: Detailed Research with Custom Parameters
|
||||
print("\n=== Detailed Research ===")
|
||||
result = detailed_research(
|
||||
query="Impact of climate change on agriculture",
|
||||
iterations=2,
|
||||
search_tool="wikipedia",
|
||||
search_strategy="source_based",
|
||||
settings_snapshot=settings_snapshot,
|
||||
programmatic_mode=True,
|
||||
)
|
||||
print(f"Research ID: {result['research_id']}")
|
||||
print(f"Summary length: {len(result['summary'])} characters")
|
||||
print(f"Sources: {len(result.get('sources', []))}")
|
||||
|
||||
# Example 3: Using Custom Search Parameters
|
||||
print("\n=== Custom Search Parameters ===")
|
||||
result = quick_summary(
|
||||
query="renewable energy trends 2024",
|
||||
search_tool="searxng", # Recommended general-purpose engine
|
||||
iterations=1,
|
||||
questions_per_iteration=3,
|
||||
temperature=0.5, # Lower temperature for focused results
|
||||
provider="openai_endpoint", # Specify LLM provider
|
||||
model_name="llama-3.3-70b-instruct", # Specify model
|
||||
settings_snapshot=settings_snapshot,
|
||||
programmatic_mode=True,
|
||||
)
|
||||
print(f"Completed {result['iterations']} iterations")
|
||||
print(
|
||||
f"Generated {sum(len(qs) for qs in result.get('questions', {}).values())} questions"
|
||||
)
|
||||
|
||||
# Example 4: Generate and Save a Report
|
||||
print("\n=== Generate Report ===")
|
||||
print("Note: Report generation can take several minutes")
|
||||
|
||||
# Generate a comprehensive report
|
||||
report = generate_report(
|
||||
query="Future of artificial intelligence",
|
||||
output_file="ai_future_report.md", # Save directly to file
|
||||
searches_per_section=2,
|
||||
iterations=1,
|
||||
settings_snapshot=settings_snapshot, # Now works with programmatic mode!
|
||||
)
|
||||
print(f"Report saved to: {report.get('file_path', 'ai_future_report.md')}")
|
||||
print(f"Report length: {len(report['content'])} characters")
|
||||
print("Report preview (first 300 chars):")
|
||||
print(report["content"][:300] + "...")
|
||||
@@ -0,0 +1,30 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Test importing search_system directly without going through __init__.py"""
|
||||
|
||||
import sys
|
||||
|
||||
# Try importing the search_system module directly
|
||||
try:
|
||||
print("Attempting to import search_system module directly...")
|
||||
from local_deep_research import search_system
|
||||
|
||||
print("✓ search_system module imported!")
|
||||
|
||||
# Now try to access AdvancedSearchSystem
|
||||
print("\nTrying to access AdvancedSearchSystem class...")
|
||||
AdvancedSearchSystem = search_system.AdvancedSearchSystem
|
||||
print("✓ Got AdvancedSearchSystem class!")
|
||||
|
||||
except Exception as e:
|
||||
print(f"✗ Failed: {e}")
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
|
||||
# Also try a more direct import
|
||||
try:
|
||||
print("\nAttempting direct file import...")
|
||||
sys.path.insert(0, "src")
|
||||
print("✓ Direct import worked!")
|
||||
except Exception as e:
|
||||
print(f"✗ Direct import failed: {e}")
|
||||
@@ -0,0 +1,131 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Simple API Client Example - No more CSRF complexity!
|
||||
|
||||
This shows how easy it is to use the LDR API with the new client.
|
||||
All the CSRF token handling is done automatically.
|
||||
"""
|
||||
|
||||
from local_deep_research.api import LDRClient, quick_query
|
||||
|
||||
# Configuration
|
||||
USERNAME = "your_username" # Change this!
|
||||
PASSWORD = "your_password" # Change this!
|
||||
|
||||
|
||||
def example_1_simple():
|
||||
"""Simplest possible usage - one line research."""
|
||||
print("=== Example 1: One-liner ===")
|
||||
|
||||
# Just one line to get a research summary!
|
||||
summary = quick_query(USERNAME, PASSWORD, "What is machine learning?")
|
||||
print(f"Summary: {summary[:200]}...")
|
||||
|
||||
|
||||
def example_2_client():
|
||||
"""Using the client for multiple operations."""
|
||||
print("\n=== Example 2: Client Usage ===")
|
||||
|
||||
# Create client
|
||||
client = LDRClient()
|
||||
|
||||
# Login once
|
||||
if not client.login(USERNAME, PASSWORD):
|
||||
print("Login failed!")
|
||||
return
|
||||
|
||||
# Now just use it - no more CSRF hassles!
|
||||
try:
|
||||
# Do research
|
||||
result = client.quick_research("What are neural networks?")
|
||||
print("Research complete!")
|
||||
print(f"Summary: {result['summary'][:200]}...")
|
||||
print(f"Sources found: {len(result.get('sources', []))}")
|
||||
|
||||
# Check settings
|
||||
settings = client.get_settings()
|
||||
print(
|
||||
f"\nYou have {len(settings.get('settings', {}))} settings configured"
|
||||
)
|
||||
|
||||
# Get history
|
||||
history = client.get_history()
|
||||
print(f"You have {len(history)} items in history")
|
||||
|
||||
finally:
|
||||
client.logout()
|
||||
|
||||
|
||||
def example_3_context_manager():
|
||||
"""Using context manager for automatic cleanup."""
|
||||
print("\n=== Example 3: Context Manager ===")
|
||||
|
||||
# Automatic login/logout with context manager
|
||||
with LDRClient() as client:
|
||||
if client.login(USERNAME, PASSWORD):
|
||||
# Start research without waiting
|
||||
result = client.quick_research(
|
||||
"What is quantum computing?", wait_for_result=False
|
||||
)
|
||||
print(f"Research started with ID: {result['research_id']}")
|
||||
|
||||
# Do other things...
|
||||
print("Doing other work while research runs...")
|
||||
|
||||
# Later, get the results
|
||||
final_result = client.wait_for_research(result["research_id"])
|
||||
print(f"Research complete: {final_result['summary'][:100]}...")
|
||||
|
||||
|
||||
def example_4_batch_research():
|
||||
"""Running multiple research queries efficiently."""
|
||||
print("\n=== Example 4: Batch Research ===")
|
||||
|
||||
questions = [
|
||||
"What is DNA?",
|
||||
"How do vaccines work?",
|
||||
"What causes earthquakes?",
|
||||
]
|
||||
|
||||
with LDRClient() as client:
|
||||
if not client.login(USERNAME, PASSWORD):
|
||||
print("Login failed!")
|
||||
return
|
||||
|
||||
# Start all research tasks
|
||||
research_ids = []
|
||||
for question in questions:
|
||||
result = client.quick_research(question, wait_for_result=False)
|
||||
research_ids.append((question, result["research_id"]))
|
||||
print(f"Started: {question}")
|
||||
|
||||
print("\nWaiting for all results...")
|
||||
|
||||
# Collect all results
|
||||
for question, research_id in research_ids:
|
||||
try:
|
||||
result = client.wait_for_research(research_id, timeout=120)
|
||||
print(f"\n{question}")
|
||||
print(f"→ {result['summary'][:150]}...")
|
||||
except Exception as e:
|
||||
print(f"\n{question}")
|
||||
print(f"→ Error: {e}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("LDR Simple Client Examples")
|
||||
print("=" * 50)
|
||||
print("\nBefore: Complex CSRF handling, HTML parsing, manual polling...")
|
||||
print("After: Just login() and quick_research()!")
|
||||
print("\nMake sure:")
|
||||
print("1. LDR server is running: python -m local_deep_research.web.app")
|
||||
print("2. You've updated USERNAME and PASSWORD in this script")
|
||||
print("=" * 50)
|
||||
|
||||
# Uncomment the examples you want to run:
|
||||
# example_1_simple()
|
||||
# example_2_client()
|
||||
# example_3_context_manager()
|
||||
# example_4_batch_research()
|
||||
|
||||
print("\nUncomment the examples in the script to run them!")
|
||||
Reference in New Issue
Block a user