wehub/lfnovo--open-notebook
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:10:23 +08:00
commit fbab2c6005
567 changed files with 114434 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/0-START-HERE/index.md
+65
View File
@@ -0,0 +1,65 @@
# Open Notebook - Start Here
**Open Notebook** is a privacy-focused AI research assistant. Upload documents, chat with AI, generate notes, and create podcasts—all with complete control over your data.
## Choose Your Path
### 🚀 I want to use OpenAI (Fastest)
**5 minutes to running.** GPT, simple setup, powerful results.
→ [OpenAI Quick Start](quick-start-openai.md)
---
### ☁️ I want to use other cloud AI (Anthropic, Google, OpenRouter, etc.)
**5 minutes to running.** Choose from 17+ AI providers.
→ [Cloud Providers Quick Start](quick-start-cloud.md)
---
### 🏠 I want to run locally (Ollama or LMStudio, completely private)
**5 minutes to running.** Keep everything private, on your machine. No costs.
→ [Local Quick Start](quick-start-local.md)
**Already have Ollama installed?** → [External Ollama Guide](quick-start-external-ollama.md)
---
## What Can You Do?
- 📄 **Upload Content**: PDFs, web links, audio, video, text
- 🤖 **Chat with AI**: Ask questions about your documents with citations
- 📝 **Generate Notes**: AI creates summaries and insights
- 🎙️ **Create Podcasts**: Turn research into professional audio content
- 🔍 **Search**: Full-text and semantic search across all content
- ⚙️ **Transform**: Extract insights, analyze themes, create summaries
## Why Open Notebook?
| Feature | Open Notebook | Notebook LM |
|---------|---|---|
| **Privacy** | Self-hosted, your control | Cloud, Google's servers |
| **AI Choice** | 17+ providers | Google's models only |
| **Podcast Speakers** | 1-4 customizable | 2 only |
| **Cost** | Completely free | Free (but your data) |
| **Offline** | Yes | No |
## Prerequisites
- **Docker**: All paths use Docker (free)
- **AI Provider**: Either a cloud API key OR use free local models (Ollama)
---
## Next Steps
1. Pick your path above ⬆️
2. Follow the 5-minute quick start
3. Create your first notebook
4. Start uploading documents!
---
**Need Help?** Join our [Discord community](https://discord.gg/37XJPXfz2w) or see [Full Documentation](../index.md).
docs/0-START-HERE/quick-start-cloud.md
+214
View File
@@ -0,0 +1,214 @@
# Quick Start - Cloud AI Providers (5 minutes)
Get Open Notebook running with **Anthropic, Google, Groq, or other cloud providers**. Same simplicity as OpenAI, with more choices.
## Prerequisites
1. **Docker Desktop** installed
- [Download here](https://www.docker.com/products/docker-desktop/)
- Already have it? Skip to step 2
2. **API Key** from your chosen provider:
- **OpenRouter** (100+ models, one key): https://openrouter.ai/keys
- **Anthropic (Claude)**: https://console.anthropic.com/
- **Google (Gemini)**: https://aistudio.google.com/
- **Groq** (fast, free tier): https://console.groq.com/
- **Mistral**: https://console.mistral.ai/
- **DeepSeek**: https://platform.deepseek.com/
- **xAI (Grok)**: https://console.x.ai/
## Step 1: Create Configuration (1 min)
Create a new folder `open-notebook` and add this file:
**docker-compose.yml**:
```yaml
services:
surrealdb:
image: surrealdb/surrealdb:v2
command: start --user root --pass password rocksdb:/mydata/mydatabase.db
ports:
# Localhost only — the database uses default credentials, so never
# publish this port on 0.0.0.0
- "127.0.0.1:8000:8000"
volumes:
- ./surreal_data:/mydata
# Removed the healthcheck because the v2 image is too minimal to run wget/curl
restart: always
open_notebook:
image: lfnovo/open_notebook:v1-latest
pull_policy: always
ports:
- "8502:8502" # Web UI
- "5055:5055" # API
environment:
- OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
- SURREAL_URL=ws://surrealdb:8000/rpc
- SURREAL_USER=root
- SURREAL_PASSWORD=password
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=open_notebook
volumes:
- ./notebook_data:/app/data
depends_on:
- surrealdb
restart: always
```
**Edit the file:**
- Replace `change-me-to-a-secret-string` with your own secret (any string works)
---
## Step 2: Start Services (1 min)
Open terminal in your `open-notebook` folder:
```bash
docker compose up -d
```
Wait 15-20 seconds for services to start.
---
## Step 3: Access Open Notebook (instant)
Open your browser:
```
http://localhost:8502
```
You should see the Open Notebook interface!
---
## Step 4: Configure Your AI Provider (1 min)
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select your provider (e.g., Anthropic, Google, Groq, OpenRouter)
4. Give it a name, paste your API key
5. Click **Save**
6. Click **Test Connection** — should show success
7. Click **Discover Models** → **Register Models**
Your provider's models are now available!
> **Multiple providers**: You can add credentials for as many providers as you want. Just repeat this step for each provider.
---
## Step 5: Configure Your Model (1 min)
1. Go to **Settings** (gear icon)
2. Navigate to **Models**
3. Select your provider's model:
| Provider | Recommended Model | Notes |
|----------|-------------------|-------|
| **OpenRouter** | `anthropic/claude-3.5-sonnet` | Access 100+ models |
| **Anthropic** | `claude-3-5-sonnet-latest` | Best reasoning |
| **Google** | `gemini-3.5-flash` | Large context, fast |
| **Groq** | `llama-3.3-70b-versatile` | Ultra-fast |
| **Mistral** | `mistral-large-latest` | Strong European option |
4. Click **Save**
---
## Step 6: Create Your First Notebook (1 min)
1. Click **New Notebook**
2. Name: "My Research"
3. Click **Create**
---
## Step 7: Add Content & Chat (2 min)
1. Click **Add Source**
2. Choose **Web Link**
3. Paste any article URL
4. Wait for processing
5. Go to **Chat** and ask questions!
---
## Verification Checklist
- [ ] Docker is running
- [ ] You can access `http://localhost:8502`
- [ ] Provider credential is configured and tested
- [ ] Models are registered
- [ ] You created a notebook
- [ ] Chat works
**All checked?** You're ready to research!
---
## Provider Comparison
| Provider | Speed | Quality | Context | Cost |
|----------|-------|---------|---------|------|
| **OpenRouter** | Varies | Varies | Varies | Varies (100+ models) |
| **Anthropic** | Medium | Excellent | 200K | $$$ |
| **Google** | Fast | Very Good | 1M+ | $$ |
| **Groq** | Ultra-fast | Good | 128K | $ (free tier) |
| **Mistral** | Fast | Good | 128K | $$ |
| **DeepSeek** | Medium | Very Good | 64K | $ |
---
## Troubleshooting
### "Model not found" Error
1. Go to **Settings** → **API Keys**
2. Click **Test Connection** on your credential
3. If valid, click **Discover Models** → **Register Models**
4. Check you have credits/access for the model
### "Cannot connect to server"
```bash
docker ps # Check all services running
docker compose logs # View logs
docker compose restart # Restart everything
```
### Provider-Specific Issues
**Anthropic**: Ensure key starts with `sk-ant-`
**Google**: Use AI Studio key, not Cloud Console
**Groq**: Free tier has rate limits; upgrade if needed
---
## Cost Estimates
Approximate costs per 1K tokens:
| Provider | Input | Output |
|----------|-------|--------|
| Anthropic (Sonnet) | $0.003 | $0.015 |
| Google (Flash) | $0.0001 | $0.0004 |
| Groq (Llama 70B) | Free tier available | - |
| Mistral (Large) | $0.002 | $0.006 |
Check provider websites for current pricing.
---
## Next Steps
1. **Add Your Content**: PDFs, web links, documents
2. **Explore Features**: Podcasts, transformations, search
3. **Full Documentation**: [See all features](../3-USER-GUIDE/index.md)
---
**Need help?** Join our [Discord community](https://discord.gg/37XJPXfz2w)!
docs/0-START-HERE/quick-start-external-ollama.md
+214
View File
@@ -0,0 +1,214 @@
# Quick Start - External Ollama
Run Open Notebook with a **separately installed Ollama** (not via Docker). This avoids Docker running the Ollama service while you use your own local Ollama installation.
## Prerequisites
1. **Docker Desktop** installed (for SurrealDB and Open Notebook)
- [Download here](https://www.docker.com/products/docker-desktop/)
2. **Ollama** installed separately
- [Download here](https://ollama.ai/)
- Verify: run `ollama --version`
3. **Models downloaded** in Ollama:
```bash
ollama pull mistral
ollama pull nomic-embed-text
```
---
## Step 1: Start Ollama (1 min)
Start the Ollama server:
```bash
# Default: runs on http://localhost:11434
ollama serve
```
Keep this terminal open. Ollama will run in the background.
**Optional: Start Ollama on a custom port or network interface:**
```bash
OLLAMA_HOST=0.0.0.0:11434 ollama serve
```
---
## Step 2: Create Configuration (1 min)
Create a new folder `open-notebook-external-ollama` and add these files:
**docker-compose.yml**:
```yaml
services:
surrealdb:
image: surrealdb/surrealdb:v2
command: start --user root --pass password rocksdb:/mydata/mydatabase.db
user: root
ports:
# Localhost only — the database uses default credentials, so never
# publish this port on 0.0.0.0
- "127.0.0.1:8000:8000"
volumes:
- ./surreal_data:/mydata
open_notebook:
image: lfnovo/open_notebook:v1-latest
pull_policy: always
ports:
- "8502:8502" # Web UI (React frontend)
- "5055:5055" # API (required!)
environment:
# Encryption key for credential storage (required)
- OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
# Database (required)
- SURREAL_URL=ws://surrealdb:8000/rpc
- SURREAL_USER=root
- SURREAL_PASSWORD=password
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=open_notebook
volumes:
- ./notebook_data:/app/data
depends_on:
- surrealdb
restart: always
```
**Note:** No Ollama service in Docker — we use the host's Ollama.
---
## Step 3: Connect Open Notebook to Host Ollama (1 min)
When Open Notebook runs inside Docker, it cannot reach `localhost:11434` on your host directly. Use the special hostname:
| Host OS | Ollama URL in Open Notebook |
|---------|----------------------------|
| Linux | `http://host.containers.internal:11434` |
| macOS | `http://host.docker.internal:11434` |
| Windows | `http://host.docker.internal:11434` |
---
## Step 4: Start Open Notebook (1 min)
Open terminal in your `open-notebook-external-ollama` folder:
```bash
docker compose up -d
```
Wait 10-15 seconds for services to start.
---
## Step 5: Configure Ollama Provider (1 min)
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **Ollama**
4. Give it a name (e.g., "Local Ollama")
5. Enter the base URL:
- **Windows/macOS:** `http://host.docker.internal:11434`
- **Linux:** `http://host.containers.internal:11434`
6. Click **Save**
7. Click **Test Connection** — should show success
8. Click **Discover Models** → **Register Models**
---
## Step 6: Configure Models (1 min)
1. Go to **Settings** → **Models**
2. Set:
- **Language Model**: `ollama/mistral` (or whichever model you downloaded)
- **Embedding Model**: `ollama/nomic-embed-text`
3. Click **Save**
---
## Step 7: Access Open Notebook (instant)
Open your browser:
```
http://localhost:8502
```
---
## Verification Checklist
- [ ] Ollama is running (`ollama serve` in terminal)
- [ ] Docker is running
- [ ] You can access `http://localhost:8502`
- [ ] Ollama credential is configured with host URL and tested
- [ ] Models are registered
- [ ] Chat works
---
## Troubleshooting
### "Connection failed" when testing Ollama credential
1. Verify Ollama is running:
```bash
curl http://localhost:11434/api/version
```
2. Check firewall allows local connections on port 11434
3. For Windows/macOS, ensure `host.docker.internal` is reachable from inside the container:
```bash
docker exec <open_notebook_container> curl http://host.docker.internal:11434/api/version
```
### Ollama not starting
```bash
# Check Ollama logs
ollama list
# Pull a model again
ollama pull mistral
```
### "Address already in use" for SurrealDB
```bash
docker compose down
docker compose up -d
```
---
## Why External Ollama?
| Approach | Ollama in Docker | Ollama External |
|----------|-----------------|-----------------|
| **Resource isolation** | Separated | Shares with host |
| **GPU access** | Requires Docker GPU setup | Native GPU access |
| **Model management** | Via `docker exec` | Via terminal directly |
| **Memory usage** | Isolated from host | Shared with host apps |
**External Ollama** is recommended if you:
- Already have Ollama installed and configured
- Want GPU access without Docker GPU passthrough complexity
- Prefer managing models via command line directly
---
## Going Further
- **Add more models**: Run `ollama pull <model>`, then re-discover from Open Notebook
- **Check Ollama status**: `ollama list` shows downloaded models
- **Customize Ollama**: Edit `~/.ollama/config.yaml` for advanced settings
---
**Need Help?** Join our [Discord community](https://discord.gg/37XJPXfz2w)
docs/0-START-HERE/quick-start-local.md
+308
View File
@@ -0,0 +1,308 @@
# Quick Start - Local & Private (5 minutes)
Get Open Notebook running with **100% local AI** using Ollama. No cloud API keys needed, completely private.
**Already have Ollama installed?** See [External Ollama Guide](quick-start-external-ollama.md) instead.
## Prerequisites
1. **Docker Desktop** installed
- [Download here](https://www.docker.com/products/docker-desktop/)
- Already have it? Skip to step 2
2. **Local LLM** - Choose one:
- **Ollama** (recommended): [Download here](https://ollama.ai/)
- **LM Studio** (GUI alternative): [Download here](https://lmstudio.ai)
## Step 1: Choose Your Setup (1 min)
### Local Machine (Same Computer)
Everything runs on your machine. Recommended for testing/learning.
### Remote Server (Raspberry Pi, NAS, Cloud VM)
Run on a different computer, access from another. Needs network configuration.
---
## Step 2: Create Configuration (1 min)
Create a new folder `open-notebook-local` and add this file:
**docker-compose.yml**:
```yaml
services:
surrealdb:
image: surrealdb/surrealdb:v2
command: start --user root --pass password rocksdb:/mydata/mydatabase.db
user: root
ports:
# Localhost only — the database uses default credentials, so never
# publish this port on 0.0.0.0
- "127.0.0.1:8000:8000"
volumes:
- ./surreal_data:/mydata
open_notebook:
image: lfnovo/open_notebook:v1-latest
pull_policy: always
ports:
- "8502:8502" # Web UI (React frontend)
- "5055:5055" # API (required!)
environment:
# Encryption key for credential storage (required)
- OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
# Database (required)
- SURREAL_URL=ws://surrealdb:8000/rpc
- SURREAL_USER=root
- SURREAL_PASSWORD=password
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=open_notebook
# Ollama (required when running Ollama via Docker, as in this compose file)
- OLLAMA_BASE_URL=http://ollama:11434
volumes:
- ./notebook_data:/app/data
depends_on:
- surrealdb
restart: always
ollama:
image: ollama/ollama:latest
ports:
- "11434:11434"
volumes:
- ./ollama_models:/root/.ollama
restart: always
# Optional: set GPU support if available
#deploy:
# resources:
# reservations:
# devices:
# - driver: nvidia
# count: 1
# capabilities: [gpu]
```
**Edit the file:**
- Replace `change-me-to-a-secret-string` with your own secret (any string works)
---
## Step 3: Start Services (1 min)
Open terminal in your `open-notebook-local` folder:
```bash
docker compose up -d
```
Wait 10-15 seconds for all services to start.
---
## Step 4: Download a Model (2-3 min)
Ollama needs at least one language model. Pick one:
```bash
# Fastest & smallest (recommended for testing)
docker exec open-notebook-local-ollama-1 ollama pull mistral
# OR: Better quality but slower
docker exec open-notebook-local-ollama-1 ollama pull neural-chat
# OR: Even better quality, more VRAM needed
docker exec open-notebook-local-ollama-1 ollama pull llama2
```
This downloads the model (will take 1-5 minutes depending on your internet).
---
## Step 5: Access Open Notebook (instant)
Open your browser:
```
http://localhost:8502
```
You should see the Open Notebook interface.
---
## Step 6: Configure Ollama Provider (1 min)
1. Go to **Manage** → **Models**
2. Click **Add Credential**
3. Select provider: **Ollama**
4. Give it a name (e.g., "Local Ollama")
5. Enter the base URL: `http://ollama:11434`
6. Click **Save**
7. Click **Test Connection** — should show success
8. Click **Discover Models** → **Register Models**
---
## Step 7: Configure Local Model (1 min)
1. Go to **Manage** → **Models**
2. Set:
- **Language Model**: `ollama/mistral` (or whichever model you downloaded)
- **Embedding Model**: `ollama/nomic-embed-text` (auto-downloads if missing)
3. Click **Save**
---
## Step 8: Create Your First Notebook (1 min)
1. Click **New Notebook**
2. Name: "My Private Research"
3. Click **Create**
---
## Step 9: Add Local Content (1 min)
1. Click **Add Source**
2. Choose **Text**
3. Paste some text or a local document
4. Click **Add**
---
## Step 10: Chat With Your Content (1 min)
1. Go to **Chat**
2. Type: "What did you learn from this?"
3. Click **Send**
4. Watch as the local Ollama model responds!
---
## Verification Checklist
- [ ] Docker is running
- [ ] You can access `http://localhost:8502`
- [ ] Ollama credential is configured and tested
- [ ] Models are registered
- [ ] You created a notebook
- [ ] Chat works with local model
**All checked?** You have a completely **private, offline** research assistant!
---
## Advantages of Local Setup
- **No API costs** - Free forever
- **No internet required** - True offline capability
- **Privacy first** - Your data never leaves your machine
- **No subscriptions** - No monthly bills
**Trade-off:** Slower than cloud models (depends on your CPU/GPU)
---
## Troubleshooting
### "ollama: command not found"
Docker image name might be different:
```bash
docker ps # Find the Ollama container name
docker exec <container_name> ollama pull mistral
```
### Model Download Stuck
Check internet connection and restart:
```bash
docker compose restart ollama
```
Then retry the model pull command.
### "Address already in use" Error
```bash
docker compose down
docker compose up -d
```
### Low Performance
Check if GPU is available:
```bash
# Show available GPUs
docker exec open-notebook-local-ollama-1 ollama ps
# Enable GPU in docker-compose.yml
```
Then restart: `docker compose restart ollama`
### Adding More Models
```bash
# List available models
docker exec open-notebook-local-ollama-1 ollama list
# Pull additional model
docker exec open-notebook-local-ollama-1 ollama pull neural-chat
```
---
## Next Steps
**Now that it's running:**
1. **Add Your Own Content**: PDFs, documents, articles (see 3-USER-GUIDE)
2. **Explore Features**: Podcasts, transformations, search
3. **Full Documentation**: [See all features](../3-USER-GUIDE/index.md)
4. **Scale Up**: Deploy to a server with better hardware for faster responses
5. **Benchmark Models**: Try different models to find the speed/quality tradeoff you prefer
## Alternative: Using LM Studio Instead of Ollama
**Prefer a GUI?** LM Studio is easier for non-technical users:
1. Download LM Studio: https://lmstudio.ai
2. Open the app, download a model from the library
3. Go to "Local Server" tab, start server (port 1234)
4. In Open Notebook, go to **Settings** → **API Keys**
5. Click **Add Credential** → Select **OpenAI-Compatible**
6. Enter base URL: `http://host.docker.internal:1234/v1`
7. Enter API key: `lm-studio` (placeholder)
8. Click **Save**, then **Test Connection**
9. Configure in Settings → Models → Select your LM Studio model
**Note**: LM Studio runs outside Docker, use `host.docker.internal` to connect.
---
## Going Further
- **Switch models**: Change in Settings → Models anytime
- **Add more models**:
- Ollama: Run `ollama pull <model>`, then re-discover models from the credential
- LM Studio: Download from the app library
- **Deploy to server**: Same docker-compose.yml works anywhere
- **Use cloud hybrid**: Keep some local models, add cloud provider credentials for complex tasks
---
## Common Model Choices
| Model | Speed | Quality | VRAM | Best For |
|-------|-------|---------|------|----------|
| **mistral** | Fast | Good | 4GB | Testing, general use |
| **neural-chat** | Medium | Better | 6GB | Balanced, recommended |
| **llama2** | Slow | Best | 8GB+ | Complex reasoning |
| **phi** | Very Fast | Fair | 2GB | Minimal hardware |
---
**Need Help?** Join our [Discord community](https://discord.gg/37XJPXfz2w) - many users run local setups!
docs/0-START-HERE/quick-start-openai.md
+197
View File
@@ -0,0 +1,197 @@
# Quick Start - OpenAI (5 minutes)
Get Open Notebook running with OpenAI's GPT models. Fast, powerful, and simple.
## Prerequisites
1. **Docker Desktop** installed
- [Download here](https://www.docker.com/products/docker-desktop/)
- Already have it? Skip to step 2
2. **OpenAI API Key** (required)
- Go to https://platform.openai.com/api-keys
- Create account → Create new secret key
- Add at least $5 in credits to your account
- Copy the key (starts with `sk-`)
## Step 1: Create Configuration (1 min)
Create a new folder `open-notebook` and add this file:
**docker-compose.yml**:
```yaml
services:
surrealdb:
image: surrealdb/surrealdb:v2
command: start --user root --pass password rocksdb:/mydata/mydatabase.db
ports:
# Localhost only — the database uses default credentials, so never
# publish this port on 0.0.0.0
- "127.0.0.1:8000:8000"
volumes:
- ./surreal_data:/mydata
open_notebook:
image: lfnovo/open_notebook:v1-latest
pull_policy: always
ports:
- "8502:8502" # Web UI
- "5055:5055" # API
environment:
# Encryption key for credential storage (required)
- OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
# Database (required)
- SURREAL_URL=ws://surrealdb:8000/rpc
- SURREAL_USER=root
- SURREAL_PASSWORD=password
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=open_notebook
volumes:
- ./notebook_data:/app/data
depends_on:
- surrealdb
restart: always
```
**Edit the file:**
- Replace `change-me-to-a-secret-string` with your own secret (any string works)
---
## Step 2: Start Services (1 min)
Open terminal in your `open-notebook` folder:
```bash
docker compose up -d
```
Wait 15-20 seconds for services to start.
---
## Step 3: Access Open Notebook (instant)
Open your browser:
```
http://localhost:8502
```
You should see the Open Notebook interface!
---
## Step 4: Configure Your OpenAI Provider (1 min)
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **OpenAI**
4. Give it a name (e.g., "My OpenAI Key")
5. Paste your OpenAI API key
6. Click **Save**
7. Click **Test Connection** — should show success
8. Click **Discover Models** → **Register Models**
Your OpenAI models are now available!
---
## Step 5: Create Your First Notebook (1 min)
1. Click **New Notebook**
2. Name: "My Research"
3. Click **Create**
---
## Step 6: Add a Source (1 min)
1. Click **Add Source**
2. Choose **Web Link**
3. Paste: `https://en.wikipedia.org/wiki/Artificial_intelligence`
4. Click **Add**
5. Wait for processing (30-60 seconds)
---
## Step 7: Chat With Your Content (1 min)
1. Go to **Chat**
2. Type: "What is artificial intelligence?"
3. Click **Send**
4. Watch as GPT responds with information from your source!
---
## Verification Checklist
- [ ] Docker is running
- [ ] You can access `http://localhost:8502`
- [ ] OpenAI credential is configured and tested
- [ ] You created a notebook
- [ ] You added a source
- [ ] Chat works
**All checked?** You have a fully working AI research assistant!
---
## Using Different Models
In your notebook, go to **Settings** → **Models** to choose:
- `gpt-4o` - Best quality (recommended)
- `gpt-4o-mini` - Fast and cheap (good for testing)
---
## Troubleshooting
### "Port 8502 already in use"
Change the port in docker-compose.yml:
```yaml
ports:
- "8503:8502" # Use 8503 instead
```
Then access at `http://localhost:8503`
### "API key not working"
1. Go to **Settings** → **API Keys**
2. Click **Test Connection** on your OpenAI credential
3. If it fails, verify your key at https://platform.openai.com
4. Delete the credential and create a new one with the correct key
### "Cannot connect to server"
```bash
docker ps # Check all services running
docker compose logs # View logs
docker compose restart # Restart everything
```
---
## Next Steps
1. **Add Your Own Content**: PDFs, web links, documents
2. **Explore Features**: Podcasts, transformations, search
3. **Full Documentation**: [See all features](../3-USER-GUIDE/index.md)
---
## Cost Estimate
OpenAI pricing (approximate):
- **Conversation**: $0.01-0.10 per 1K tokens
- **Embeddings**: $0.02 per 1M tokens
- **Typical usage**: $1-5/month for light use, $20-50/month for heavy use
Check https://openai.com/pricing for current rates.
---
**Need help?** Join our [Discord community](https://discord.gg/37XJPXfz2w)!
docs/1-INSTALLATION/docker-compose.md
+374
View File
@@ -0,0 +1,374 @@
# Docker Compose Installation (Recommended)
Multi-container setup with separate services. **Best for most users.**
> **Alternative Registry:** All images are available on both Docker Hub (`lfnovo/open_notebook`) and GitHub Container Registry (`ghcr.io/lfnovo/open-notebook`). Use GHCR if Docker Hub is blocked or you prefer GitHub-native workflows.
## Prerequisites
- **Docker Desktop** installed ([Download](https://www.docker.com/products/docker-desktop/))
- **5-10 minutes** of your time
- **API key** for at least one AI provider (OpenAI recommended for beginners)
## Step 1: Get docker-compose.yml (1 min)
**Option A: Download from repository**
```bash
curl -o docker-compose.yml https://raw.githubusercontent.com/lfnovo/open-notebook/main/docker-compose.yml
```
**Option B: Use the official file from the repo**
The official `docker-compose.yml` is in the root of our repository: [View on GitHub](https://github.com/lfnovo/open-notebook/blob/main/docker-compose.yml)
Copy that file to your project folder.
**Option C: Create manually**
Create a file called `docker-compose.yml` with this content:
```yaml
services:
surrealdb:
image: surrealdb/surrealdb:v2
# Credentials default to root:root for a zero-config local setup. Before
# exposing this instance to a network, set SURREAL_USER / SURREAL_PASSWORD
# in a .env file (see .env.example) — they are applied here and to the
# open_notebook service below, so the two always stay in sync.
# List (exec) form so each interpolated value stays a single argument —
# a password containing spaces would otherwise be split into several.
command: ["start", "--log", "info", "--user", "${SURREAL_USER:-root}", "--pass", "${SURREAL_PASSWORD:-root}", "rocksdb:/mydata/mydatabase.db"]
user: root # Required for bind mounts on Linux
ports:
# Bound to localhost only: the open_notebook service reaches this over
# the internal compose network regardless, so the host port is purely
# for local debugging (e.g. Surrealist, `surreal sql`). Exposing this
# on 0.0.0.0 would let anyone who can reach the host connect with the
# default root:root credentials.
- "127.0.0.1:8000:8000"
volumes:
- ./surreal_data:/mydata
environment:
- SURREAL_EXPERIMENTAL_GRAPHQL=true
restart: always
pull_policy: always
open_notebook:
image: lfnovo/open_notebook:v1-latest
ports:
- "8502:8502" # Web UI
- "5055:5055" # REST API
environment:
# REQUIRED: Change this to your own secret string
# This encrypts your API keys in the database
- OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
# Database connection. SURREAL_USER / SURREAL_PASSWORD default to root:root
# for local use; override them in a .env file before exposing the instance
# (the same values configure the surrealdb service above).
- SURREAL_URL=ws://surrealdb:8000/rpc
- SURREAL_USER=${SURREAL_USER:-root}
- SURREAL_PASSWORD=${SURREAL_PASSWORD:-root}
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=open_notebook
volumes:
- ./notebook_data:/app/data
depends_on:
- surrealdb
restart: always
pull_policy: always
```
**Edit the file:**
- Replace `change-me-to-a-secret-string` with your own secret (any string works, e.g., `my-super-secret-key-123`)
- (Optional) To use database credentials other than the default `root:root`, create a `.env` file next to `docker-compose.yml` with `SURREAL_USER=...` and `SURREAL_PASSWORD=...` — both services pick them up automatically ([.env.example](https://github.com/lfnovo/open-notebook/blob/main/.env.example) shows the full format)
---
## Step 2: Start Services (2 min)
Open terminal in the `open-notebook` folder:
```bash
docker compose up -d
```
Wait 15-20 seconds for all services to start:
```
✅ surrealdb running on :8000
✅ open_notebook running on :8502 (UI) and :5055 (API)
```
Check status:
```bash
docker compose ps
```
---
## Step 3: Verify Installation (1 min)
**API Health:**
```bash
curl http://localhost:5055/health
# Should return: {"status": "healthy"}
```
**Frontend Access:**
Open browser to:
```
http://localhost:8502
```
You should see the Open Notebook interface!
---
## Step 4: Configure AI Provider (2 min)
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select your provider (e.g., OpenAI, Anthropic, Google)
4. Give it a name, paste your API key
5. Click **Save**
6. Click **Test Connection** — should show success
7. Click **Discover Models** → **Register Models**
Your models are now available!
> **Need an API key?** Get one from your chosen provider:
> - **OpenAI**: https://platform.openai.com/api-keys
> - **Anthropic**: https://console.anthropic.com/
> - **Google**: https://aistudio.google.com/
> - **Groq**: https://console.groq.com/
---
## Step 5: First Notebook (2 min)
1. Click **New Notebook**
2. Name: "My Research"
3. Description: "Getting started"
4. Click **Create**
Done! You now have a fully working Open Notebook instance.
---
## Configuration
### Adding Ollama (Free Local Models)
Instead of manually editing, use our ready-made example:
```bash
# Download the Ollama example
curl -o docker-compose.yml https://raw.githubusercontent.com/lfnovo/open-notebook/main/examples/docker-compose-ollama.yml
# Or copy from repo
cp examples/docker-compose-ollama.yml docker-compose.yml
```
See [examples/docker-compose-ollama.yml](../../examples/docker-compose-ollama.yml) for the complete setup.
**Manual setup:** Add this to your existing `docker-compose.yml`:
```yaml
ollama:
image: ollama/ollama:latest
ports:
- "11434:11434"
volumes:
- ollama_models:/root/.ollama
restart: always
volumes:
ollama_models:
```
Then restart and pull a model:
```bash
docker compose restart
docker exec open-notebook-local-ollama-1 ollama pull mistral
```
Configure Ollama in the Settings UI:
1. Go to **Settings** → **API Keys**
2. Click **Add Credential** → Select **Ollama**
3. Enter base URL: `http://ollama:11434`
4. Click **Save**, then **Test Connection**
5. Click **Discover Models** → **Register Models**
---
## Environment Variables Reference
| Variable | Purpose | Example |
|----------|---------|---------|
| `OPEN_NOTEBOOK_ENCRYPTION_KEY` | Encryption key for credentials | `my-secret-key` |
| `SURREAL_URL` | Database connection | `ws://surrealdb:8000/rpc` |
| `SURREAL_USER` | Database user | `root` |
| `SURREAL_PASSWORD` | Database password | `root` |
| `SURREAL_NAMESPACE` | Database namespace | `open_notebook` |
| `SURREAL_DATABASE` | Database name | `open_notebook` |
| `API_URL` | API external URL | `http://localhost:5055` |
| `OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE` | Override embedding batch size for stricter/local providers (recommended: `8` for CPU-only local setups) | `50` |
See [Environment Reference](../5-CONFIGURATION/environment-reference.md) for complete list.
---
## Common Tasks
### Stop Services
```bash
docker compose down
```
### View Logs
```bash
# All services
docker compose logs -f
# Specific service
docker compose logs -f api
```
### Restart Services
```bash
docker compose restart
```
### Update to Latest Version
```bash
docker compose down
docker compose pull
docker compose up -d
```
### Remove All Data
```bash
docker compose down -v
```
---
## Troubleshooting
### "Cannot connect to API" Error
1. Check if Docker is running:
```bash
docker ps
```
2. Check if services are running:
```bash
docker compose ps
```
3. Check API logs:
```bash
docker compose logs api
```
4. Wait longer - services can take 20-30 seconds to start on first run
---
### Port Already in Use
If you get "Port 8502 already in use", change the port:
```yaml
ports:
- "8503:8502" # Use 8503 instead
- "5055:5055" # Keep API port same
```
Then access at `http://localhost:8503`
---
### Credential Issues
1. Go to **Settings** → **API Keys**
2. Click **Test Connection** on the credential
3. If it fails, verify key at provider's website
4. Check you have credits in your account
5. Delete and re-create the credential if needed
---
### Database Connection Issues
Check SurrealDB is running:
```bash
docker compose logs surrealdb
```
Reset database:
```bash
docker compose down -v
docker compose up -d
```
### Database Permission Denied (Linux)
If you see `Permission denied` or `Failed to create RocksDB directory` in SurrealDB logs:
```bash
docker compose logs surrealdb | grep -i permission
```
This happens because SurrealDB runs as a non-root user but Docker creates bind mount directories as root. Add `user: root` to the surrealdb service:
```yaml
surrealdb:
image: surrealdb/surrealdb:v2
user: root # Fix for Linux bind mount permissions
# ... rest of config
```
Then restart:
```bash
docker compose down -v
docker compose up -d
```
---
## Alternative Setups
Looking for different configurations? Check out our [examples/](../../examples/) folder:
- **[Ollama Setup](../../examples/docker-compose-ollama.yml)** - Run local AI models (free, private)
- **[Single Container](../../examples/docker-compose-single.yml)** - All-in-one container (deprecated, will be removed in v2)
- **[Development](../../examples/docker-compose-dev.yml)** - For contributors and developers
Each example includes detailed comments and usage instructions.
---
## Next Steps
1. **Add Content**: Sources, notebooks, documents
2. **Configure Models**: Settings → Models (choose your preferences)
3. **Explore Features**: Chat, search, transformations
4. **Read Guide**: [User Guide](../3-USER-GUIDE/index.md)
---
## Production Deployment
For production use, see:
- [Security Hardening](../5-CONFIGURATION/security.md)
- [Reverse Proxy](../5-CONFIGURATION/reverse-proxy.md)
---
## Getting Help
- **Discord**: [Community support](https://discord.gg/37XJPXfz2w)
- **Issues**: [GitHub Issues](https://github.com/lfnovo/open-notebook/issues)
- **Docs**: [Full documentation](../index.md)
docs/1-INSTALLATION/from-source.md
+195
View File
@@ -0,0 +1,195 @@
# From Source Installation
Clone the repository and run locally. **For developers and contributors.**
## Prerequisites
- **Python 3.11+** - [Download](https://www.python.org/)
- **Node.js 18+** - [Download](https://nodejs.org/)
- **Git** - [Download](https://git-scm.com/)
- **Docker** (for SurrealDB) - [Download](https://docker.com/)
- **uv** (Python package manager) - `curl -LsSf https://astral.sh/uv/install.sh | sh`
- API key from OpenAI or similar (or use Ollama for free)
## Quick Setup (10 minutes)
### 1. Clone Repository
```bash
git clone https://github.com/lfnovo/open-notebook.git
cd open-notebook
# If you forked it:
git clone https://github.com/YOUR_USERNAME/open-notebook.git
cd open-notebook
git remote add upstream https://github.com/lfnovo/open-notebook.git
```
### 2. Install Python Dependencies
```bash
uv sync
uv pip install python-magic
```
#### 2.1 Alternative: Conda Setup (Optional)
If you prefer using **Conda** to manage your environments, follow these steps instead of the standard `uv sync`:
```bash
# Create and activate the environment
conda create -n open-notebook python=3.11 -y
conda activate open-notebook
# Install uv inside conda to maintain compatibility with the Makefile
conda install -c conda-forge uv nodejs -y
# Sync dependencies
uv sync
```
> **Note**: Installing `uv` inside your Conda environment ensures that commands like `make start-all` and `make api` continue to work seamlessly.
### 3. Start SurrealDB
```bash
# Terminal 1
make database
# or: docker compose up surrealdb
```
### 4. Set Environment Variables
```bash
cp .env.example .env
# Edit .env and set:
# OPEN_NOTEBOOK_ENCRYPTION_KEY=my-secret-key
```
After starting the app, configure AI providers via the **Manage → Models** UI in the browser.
### 5. Start API
```bash
# Terminal 2
make api
# or: uv run --env-file .env uvicorn api.main:app --host 0.0.0.0 --port 5055
```
### 6. Start Worker
Source and note processing (content extraction, embedding, insights) is dispatched
as background jobs that a **separate worker** process consumes. Without it, every
source stays stuck at `Source processing status: CommandStatus.NEW` forever.
```bash
# Terminal 3
make worker
# or: uv run --env-file .env surreal-commands-worker --import-modules commands
```
> `make start-all` starts Database + API + Worker + Frontend together; the steps
> above run them individually so you can see each process's logs.
### 7. Start Frontend
```bash
# Terminal 4
cd frontend && npm install && npm run dev
```
### 8. Access
- **Frontend**: http://localhost:3000
- **API Docs**: http://localhost:5055/docs
- **Database**: http://localhost:8000
### 9. Configure AI Provider
1. Open http://localhost:3000
2. Go to **Manage** → **Models**
3. Click **Add Credential** → Select your provider → Paste API key
4. Click **Save**, then **Test Connection**
5. Click **Discover Models** → **Register Models**
---
## Development Workflow
### Code Quality
```bash
# Format and lint Python
make ruff
# or: ruff check . --fix
# Type checking
make lint
# or: uv run python -m mypy .
```
### Run Tests
```bash
uv run pytest tests/
```
### Common Commands
```bash
# Start everything
make start-all
# View API docs
open http://localhost:5055/docs
# Check database migrations
# (Auto-run on API startup)
# Clean up
make clean
```
---
## Troubleshooting
### Python version too old
```bash
python --version # Check version
uv sync --python 3.11 # Use specific version
```
### npm: command not found
Install Node.js from https://nodejs.org/
### Database connection errors
```bash
docker ps # Check SurrealDB running
docker logs surrealdb # View logs
```
### Port 5055 already in use
```bash
# Use different port
uv run uvicorn api.main:app --port 5056
```
---
## Next Steps
1. Read [Development Guide](../7-DEVELOPMENT/quick-start.md)
2. See [Architecture Overview](../7-DEVELOPMENT/architecture.md)
3. Check [Contributing Guide](../7-DEVELOPMENT/contributing.md)
---
## Getting Help
- **Discord**: [Community](https://discord.gg/37XJPXfz2w)
- **Issues**: [GitHub Issues](https://github.com/lfnovo/open-notebook/issues)
docs/1-INSTALLATION/index.md
+159
View File
@@ -0,0 +1,159 @@
# Installation Guide
Choose your installation route based on your setup and use case.
## Quick Decision: Which Route?
### 🚀 I want the easiest setup (Recommended for most)
**→ [Docker Compose](docker-compose.md)** - Multi-container setup, production-ready
- ✅ All features working
- ✅ Clear separation of services
- ✅ Easy to scale
- ✅ Works on Mac, Windows, Linux
- ⏱️ 5 minutes to running
---
### 🏠 I want everything in one container (Deprecated)
**→ [Single Container](single-container.md)** - Deprecated, will be removed in v2
- ⚠️ **Deprecated** — please use Docker Compose instead
- Still supported until v2 release
---
### 👨‍💻 I want to develop/contribute (Developers only)
**→ [From Source](from-source.md)** - Clone repo, set up locally
- ✅ Full control over code
- ✅ Easy to debug
- ✅ Can modify and test
- ⚠️ Requires Python 3.11+, Node.js
- ⏱️ 10 minutes to running
---
### 🪟 I'm on Windows and can't use Docker/WSL
**→ [Windows Native](windows-native.md)** - Run natively, no Docker or WSL
- ✅ Works on Windows ARM64
- ✅ For systems without Hyper-V/Docker Desktop
- ⚠️ Requires Python 3.12+, Node.js, SurrealDB, uv
- ⏱️ 15 minutes to running
---
## System Requirements
### Minimum
- **RAM**: 4GB
- **Storage**: 2GB for app + space for documents
- **CPU**: Any modern processor
- **Network**: Internet (optional for offline setup)
### Recommended
- **RAM**: 8GB+
- **Storage**: 10GB+ for documents and models
- **CPU**: Multi-core processor
- **GPU**: Optional (speeds up local AI models)
---
## AI Provider Options
### Cloud-Based (Pay-as-you-go)
- **OpenAI** - GPT-4, GPT-4o, fast and capable
- **Anthropic (Claude)** - Claude 3.5 Sonnet, excellent reasoning
- **Google Gemini** - Multimodal, cost-effective
- **Groq** - Ultra-fast inference
- **Others**: Mistral, DeepSeek, xAI, OpenRouter
**Cost**: Usually $0.01-$0.10 per 1K tokens
**Speed**: Fast (sub-second)
**Privacy**: Your data sent to cloud
### Local (Free, Private)
- **Ollama** - Run open-source models locally
- **LM Studio** - Desktop app for local models
- **Hugging Face models** - Download and run
**Cost**: $0 (just electricity)
**Speed**: Depends on your hardware (slow to medium)
**Privacy**: 100% offline
---
## Choose a Route
**Already know which way to go?** Pick your installation path:
- [Docker Compose](docker-compose.md) - **Most users**
- [Single Container](single-container.md) - **Deprecated**
- [From Source](from-source.md) - **Developers**
> **Privacy-first?** Any installation method works with Ollama for 100% local AI. See [Local Quick Start](../0-START-HERE/quick-start-local.md).
---
## Pre-Installation Checklist
Before installing, you'll need:
- [ ] **Docker** (for Docker routes) or **Node.js 18+** (for source)
- [ ] **AI Provider API key** (OpenAI, Anthropic, etc.) OR willingness to use free local models
- [ ] **At least 4GB RAM** available
- [ ] **Stable internet** (or offline setup with Ollama)
---
## Detailed Installation Instructions
### For Docker Users
1. Install [Docker Desktop](https://docker.com/products/docker-desktop)
2. Follow [Docker Compose](docker-compose.md) installation
3. Follow the step-by-step guide
4. Access at `http://localhost:8502`
### For Source Installation (Developers)
1. Have Python 3.11+, Node.js 18+, Git installed
2. Follow [From Source](from-source.md)
3. Run `make start-all`
4. Access at `http://localhost:8502` (frontend) or `http://localhost:5055` (API)
---
## After Installation
Once you're up and running:
1. **Configure Models** - Choose your AI provider in Settings
2. **Create First Notebook** - Start organizing research
3. **Add Sources** - PDFs, web links, documents
4. **Explore Features** - Chat, search, transformations
5. **Read Full Guide** - [User Guide](../3-USER-GUIDE/index.md)
---
## Troubleshooting During Installation
**Having issues?** Check the troubleshooting section in your chosen installation guide, or see [Quick Fixes](../6-TROUBLESHOOTING/quick-fixes.md).
---
## Need Help?
- **Discord**: [Join community](https://discord.gg/37XJPXfz2w)
- **GitHub Issues**: [Report problems](https://github.com/lfnovo/open-notebook/issues)
- **Docs**: See [Full Documentation](../index.md)
---
## Production Deployment
Installing for production use? See additional resources:
- [Security Hardening](../5-CONFIGURATION/security.md)
- [Reverse Proxy Setup](../5-CONFIGURATION/reverse-proxy.md)
- [Performance Tuning](../5-CONFIGURATION/advanced.md)
---
**Ready to install?** Pick a route above! ⬆️
docs/1-INSTALLATION/single-container.md
+146
View File
@@ -0,0 +1,146 @@
# Single Container Installation (Deprecated)
> **Deprecation Notice:** The single-container image (`v1-latest-single`) is **deprecated** and will be removed in v2. Please migrate to [Docker Compose](docker-compose.md), which is the recommended installation method for all users. The single-container image will continue to receive updates until v2 is released, but no new features or documentation will target it.
All-in-one container setup. **Simpler than Docker Compose, but less flexible.**
**Best for:** PikaPods, Railway, shared hosting, minimal setups
> **Alternative Registry:** Images available on both Docker Hub (`lfnovo/open_notebook:v1-latest-single`) and GitHub Container Registry (`ghcr.io/lfnovo/open-notebook:v1-latest-single`).
## Prerequisites
- Docker installed (for local testing)
- API key from OpenAI, Anthropic, or another provider
- 5 minutes
## Quick Setup
### For Local Testing (Docker)
```yaml
# docker-compose.yml
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest-single
pull_policy: always
ports:
- "8502:8502" # Web UI (React frontend)
- "5055:5055" # API
environment:
- OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
- SURREAL_URL=ws://localhost:8000/rpc
- SURREAL_USER=root
- SURREAL_PASSWORD=root
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=open_notebook
volumes:
- ./data:/app/data
restart: always
```
Run:
```bash
docker compose up -d
```
Access: `http://localhost:8502`
Then configure your AI provider:
1. Go to **Settings** → **API Keys**
2. Click **Add Credential** → Select your provider → Paste API key
3. Click **Save**, then **Test Connection**
4. Click **Discover Models** → **Register Models**
### For Cloud Platforms
**PikaPods:**
1. Click "New App"
2. Search "Open Notebook"
3. Set environment variables (at minimum: `OPEN_NOTEBOOK_ENCRYPTION_KEY`)
4. Click "Deploy"
5. Open the app → Go to **Settings → API Keys** to configure your AI provider
**Railway:**
1. Create new project
2. Add `lfnovo/open_notebook:v1-latest-single`
3. Set environment variables (at minimum: `OPEN_NOTEBOOK_ENCRYPTION_KEY`)
4. Deploy
5. Open the app → Go to **Settings → API Keys** to configure your AI provider
**Render:**
1. Create new Web Service
2. Use Docker image: `lfnovo/open_notebook:v1-latest-single`
3. Set environment variables in dashboard (at minimum: `OPEN_NOTEBOOK_ENCRYPTION_KEY`)
4. Configure persistent disk for `/app/data` and `/mydata`
**DigitalOcean App Platform:**
1. Create new app from Docker Hub
2. Use image: `lfnovo/open_notebook:v1-latest-single`
3. Set port to 8502
4. Add environment variables (at minimum: `OPEN_NOTEBOOK_ENCRYPTION_KEY`)
5. Configure persistent storage
**Heroku:**
```bash
# Using heroku.yml
heroku container:push web
heroku container:release web
heroku config:set OPEN_NOTEBOOK_ENCRYPTION_KEY=your-secret-key
```
**Coolify:**
1. Add new service → Docker Image
2. Image: `lfnovo/open_notebook:v1-latest-single`
3. Port: 8502
4. Add environment variables (at minimum: `OPEN_NOTEBOOK_ENCRYPTION_KEY`)
5. Enable persistent volumes
6. Coolify handles HTTPS automatically
**EasyPanel:**
Open Notebook ships an EasyPanel template at [`examples/easypanel/`](https://github.com/lfnovo/open-notebook/tree/main/examples/easypanel). Unlike the single-image options above, the template provisions **two services** — the Open Notebook app and a dedicated SurrealDB instance — and generates the database password, encryption key, and (optionally) the app password for you.
- **One-click (recommended):** once the template is published to the official [EasyPanel template gallery](https://github.com/easypanel-io/templates), create a new service from "Open Notebook", set an app password (or leave it blank to auto-generate one), and deploy.
- **Manual:** copy `examples/easypanel/` into `templates/open-notebook` in a checkout of [`easypanel-io/templates`](https://github.com/easypanel-io/templates), run the templates playground (`npm run dev`), and create the template from the generated JSON in your EasyPanel instance.
After deployment, open the EasyPanel domain and configure your AI provider in **Settings → API Keys**. See [`examples/easypanel/README.md`](https://github.com/lfnovo/open-notebook/blob/main/examples/easypanel/README.md) for details.
---
## Environment Variables
| Variable | Purpose | Example |
|----------|---------|---------|
| `OPEN_NOTEBOOK_ENCRYPTION_KEY` | Encryption key for credentials (required) | `my-secret-key` |
| `SURREAL_URL` | Database | `ws://localhost:8000/rpc` |
| `SURREAL_USER` | DB user | `root` |
| `SURREAL_PASSWORD` | DB password | `root` |
| `SURREAL_NAMESPACE` | DB namespace | `open_notebook` |
| `SURREAL_DATABASE` | DB name | `open_notebook` |
| `API_URL` | External URL (for remote access) | `https://myapp.example.com` |
AI provider API keys are configured via the **Settings → API Keys** UI after deployment.
---
## Limitations vs Docker Compose
| Feature | Single Container | Docker Compose |
|---------|------------------|-----------------|
| Setup time | 2 minutes | 5 minutes |
| Complexity | Minimal | Moderate |
| Services | All bundled | Separated |
| Scalability | Limited | Excellent |
| Memory usage | ~800MB | ~1.2GB |
---
## Next Steps
Same as Docker Compose setup - just access via `http://localhost:8502` (local) or your platform's URL (cloud).
1. Go to **Settings → API Keys** to add your AI provider credential
2. **Test Connection** and **Discover Models**
See [Docker Compose](docker-compose.md) for full post-install guide.
docs/1-INSTALLATION/windows-native.md
+285
View File
@@ -0,0 +1,285 @@
# Open Notebook Windows Installation Guide (Native, No Docker)
This guide documents how to install and run [Open Notebook](https://github.com/lfnovo/open-notebook) on Windows **natively without Docker or WSL**.
## Who Is This For?
- **Windows ARM64 users** - Docker Desktop and WSL2 have limitations on ARM64
- **Users without Hyper-V** - Some Windows editions don't support Docker
- **Users who prefer native installs** - Simpler architecture, easier debugging
## What This Guide Covers
- Native Windows installation steps
- Critical configuration fixes for Windows
- Troubleshooting common issues
- Upgrade and maintenance scripts
## Prerequisites
| Software | Installation | Required |
| ------------ | -------------------------------- | -------- |
| Git | `winget install Git.Git` | Yes |
| Python 3.12+ | Via uv (installed automatically) | Yes |
| Node.js 18+ | `winget install OpenJS.NodeJS` | Yes |
| uv | `pip install uv` | Yes |
| SurrealDB | `scoop install surrealdb` | Yes |
## Quick Start
1. **Clone and setup:**
```bash
cd %USERPROFILE%\Projects # or your preferred location
git clone https://github.com/lfnovo/open-notebook.git
cd open-notebook
uv sync
cd frontend && npm install && cd ..
```
2. **Configure `.env`:**
- Copy `.env.example` to `.env`
- Add your API keys
- **CRITICAL:** Change `SURREAL_URL` from `localhost` to `127.0.0.1`:
```env
SURREAL_URL="ws://127.0.0.1:8000/rpc"
```
3. **Start the four services**, each in its own terminal, from the `open-notebook` folder.
> Open Notebook does not ship a launcher script — start the services manually as below (or wrap them in your own `.bat`, see [Optional: one-click launcher](#optional-one-click-launcher)).
```batch
REM Optional: point Open Notebook at a separate data folder (see Issue 4 below).
REM Set this in each terminal before running, or skip to use ./data.
set DATA_FOLDER=%USERPROFILE%\Projects\open-notebook-data
REM Terminal 1 — SurrealDB
surreal start --user root --pass root --bind 127.0.0.1:8000 rocksdb:%DATA_FOLDER%\surrealdb
REM Terminal 2 — API
uv run --env-file .env run_api.py
REM Terminal 3 — Worker (module form avoids the Windows "canonicalize" error, see Issue 3)
set PYTHONPATH=%CD%
uv run --env-file .env python -m surreal_commands.cli.worker --import-modules commands
REM Terminal 4 — Frontend
cd frontend && npm run dev
```
4. **Open the app:** http://127.0.0.1:3000
## Directory Structure (Recommended)
```
YourProjectsFolder\
├── open-notebook\ # Source code (git clone)
│ ├── .venv\ # Python virtual environment (created by uv)
│ ├── frontend\ # Next.js frontend
│ ├── commands\ # Worker command modules
│ └── .env # Your configuration
├── open-notebook-data\ # Data storage (SEPARATE from code!)
│ ├── surrealdb\ # Database files
│ ├── uploads\ # Uploaded documents
│ └── sqlite-db\ # LangGraph checkpoints
└── start-open-notebook.bat # Optional launcher you create yourself (see below)
```
**Why separate data folder?** Prevents accidental data loss when updating/reinstalling code.
## Optional: one-click launcher
Open Notebook does not ship a launcher, but you can save the following as
`start-open-notebook.bat` (anywhere you like) to start all four services with a
double-click. Adjust `ROOT` and `DATA_ROOT` to match your setup.
```batch
@echo off
REM --- adjust these two paths ---
set ROOT=%USERPROFILE%\Projects\open-notebook
set DATA_ROOT=%USERPROFILE%\Projects\open-notebook-data
set DATA_FOLDER=%DATA_ROOT%
set PYTHONPATH=%ROOT%
cd /d %ROOT%
start "SurrealDB" surreal start --user root --pass root --bind 127.0.0.1:8000 rocksdb:%DATA_ROOT%\surrealdb
start "API" cmd /k "uv run --env-file .env run_api.py"
start "Worker" cmd /k "uv run --env-file .env python -m surreal_commands.cli.worker --import-modules commands"
start "Frontend" cmd /k "cd /d %ROOT%\frontend && npm run dev"
```
Then open http://127.0.0.1:3000.
## Critical Windows Fixes
### Issue 1: Wrong Python Version
**Symptom:**
```
ModuleNotFoundError: No module named 'langgraph.checkpoint.sqlite'
```
Traceback shows system Python (e.g., `C:\Python314\`) instead of venv.
**Cause:** Windows may have multiple Python versions. The venv's `activate.bat` doesn't always override correctly.
**Solution:** Use `uv run` instead of direct python calls:
```batch
REM Wrong:
.venv\Scripts\python.exe run_api.py
REM Correct:
uv run python run_api.py
```
### Issue 2: Database Health Check Timeout
**Symptom:**
```
WARNING: Database health check timed out after 2 seconds
```
Frontend shows "Database is offline" even though SurrealDB is running.
**Cause:** `.env` uses `localhost` but SurrealDB binds to `127.0.0.1`.
**Solution:** In `.env`, change:
```env
# Wrong:
SURREAL_URL="ws://localhost:8000/rpc"
# Correct:
SURREAL_URL="ws://127.0.0.1:8000/rpc"
```
### Issue 3: Worker "Failed to canonicalize script path"
**Symptom:**
```
Failed to canonicalize script path
```
**Cause:** The `surreal-commands-worker.exe` can't find the Python `commands` module.
**Solution:** Use Python module invocation with PYTHONPATH:
```batch
set PYTHONPATH=%ROOT%
uv run --env-file .env python -m surreal_commands.cli.worker --import-modules commands
```
### Issue 4: DATA_FOLDER Path Parsing Error
**Symptom:**
```
warning: Failed to parse environment file .env at position X
```
**Cause:** `uv` can't parse Windows paths with backslashes in `.env`.
**Solution:** Keep `DATA_FOLDER` **commented out** in `.env`. Set it via batch file:
```batch
set DATA_FOLDER=C:\path\to\open-notebook-data
```
## Configuration Files
### Modifying `open_notebook/config.py`
The default `config.py` uses a hardcoded data path. Modify it to read from environment:
```python
import os
# ROOT DATA FOLDER - can be overridden via DATA_FOLDER environment variable
DATA_FOLDER = os.environ.get("DATA_FOLDER", "./data")
# Rest of file uses DATA_FOLDER...
```
### Required `.env` Settings
```env
# Database - MUST use 127.0.0.1!
SURREAL_URL="ws://127.0.0.1:8000/rpc"
SURREAL_USER="root"
SURREAL_PASSWORD="root"
SURREAL_NAMESPACE="open_notebook"
SURREAL_DATABASE="open_notebook"
# API Keys (uncomment and fill in)
OPENAI_API_KEY=your-key-here
ANTHROPIC_API_KEY=your-key-here
GOOGLE_API_KEY=your-key-here
```
## Available AI Models
Once running, add models in Settings. Common model names:
| Provider | Models |
| --------- | ------------------------------------------------------------ |
| OpenAI | `gpt-4o`, `gpt-4o-mini`, `gpt-4-turbo`, `text-embedding-3-small` |
| Anthropic | `claude-sonnet-4-20250514`, `claude-3-5-sonnet-20241022`, `claude-3-5-haiku-20241022` |
| Google | `gemini-3.5-flash`, `gemini-2.5-flash`, `gemini-2.5-pro` |
| DeepSeek | `deepseek-chat`, `deepseek-reasoner` |
## Upgrading
When a new version is released:
```batch
cd open-notebook
git pull
uv sync
cd frontend && npm install && cd ..
```
Then restart all services. Your `.env` and data are preserved.
## Services & Ports
| Service | Port | URL |
| --------- | ---- | -------------------------- |
| SurrealDB | 8000 | ws://127.0.0.1:8000 |
| API | 5055 | http://127.0.0.1:5055/docs |
| Frontend | 3000 | http://127.0.0.1:3000 |
## Troubleshooting
### Services won't start
- Check if ports are in use: `netstat -ano | findstr :8000`
- Kill existing processes: `taskkill /F /PID <pid>`
### Frontend can't connect to API
- Verify API is running: http://127.0.0.1:5055/docs
- Check `.env` has `API_URL=http://localhost:5055`
### Worker not processing commands
- Check Worker window for errors
- Verify PYTHONPATH is set in startup script
## Contributing
Found another Windows-specific issue? Please share your solution!
---
*Tested on Windows 11 ARM64 with Open Notebook v1.6.0*
*Created: January 2026*
docs/2-CORE-CONCEPTS/ai-context-rag.md
+450
View File
@@ -0,0 +1,450 @@
# AI Context & RAG - How Open Notebook Uses Your Research
Open Notebook uses different approaches to make AI models aware of your research depending on the feature. This section explains **RAG** (used in Ask) and **full-content context** (used in Chat).
---
## The Problem: Making AI Aware of Your Data
### Traditional Approaches (and their problems)
**Option 1: Fine-Tuning**
- Train the model on your data
- Pro: Model becomes specialized
- Con: Expensive, slow, permanent (can't unlearn)
**Option 2: Send Everything to Cloud**
- Upload all your data to ChatGPT/Claude API
- Pro: Works well, fast
- Con: Privacy nightmare, data leaves your control, expensive
**Option 3: Ignore Your Data**
- Just use the base model without your research
- Pro: Private, free
- Con: AI doesn't know anything about your specific topic
### Open Notebook's Dual Approach
**For Chat**: Sends the entire selected content to the LLM
- Simple and transparent: You select sources, they're sent in full
- Maximum context: AI sees everything you choose
- You control which sources are included
**For Ask (RAG)**: Retrieval-Augmented Generation
- RAG = Retrieval-Augmented Generation
- The insight: *Search your content, find relevant pieces, send only those*
- Automatic: AI decides what's relevant based on your question
---
## How RAG Works: Three Stages
### Stage 1: Content Preparation
When you upload a source, Open Notebook prepares it for retrieval:
```
1. EXTRACT TEXT
PDF → text
URL → webpage text
Audio → transcribed text
Video → subtitles + transcription
2. CHUNK INTO PIECES
Long documents → break into ~500-word chunks
Why? AI context has limits; smaller pieces are more precise
3. CREATE EMBEDDINGS
Each chunk → semantic vector (numbers representing meaning)
Why? Allows finding chunks by similarity, not just keywords
4. STORE IN DATABASE
Chunks + embeddings + metadata → searchable storage
```
**Example:**
```
Source: "AI Safety Research 2026" (50-page PDF)
↓
Extracted: 50 pages of text
↓
Chunked: 150 chunks (~500 words each)
↓
Embedded: Each chunk gets a vector (1536 numbers for OpenAI)
↓
Stored: Ready for search
```
---
### Stage 2: Query Time (What You Search For)
When you ask a question, the system finds relevant content:
```
1. YOU ASK A QUESTION
"What does the paper say about alignment?"
2. SYSTEM CONVERTS QUESTION TO EMBEDDING
Your question → vector (same way chunks are vectorized)
3. SIMILARITY SEARCH
Find chunks most similar to your question
(using vector math, not keyword matching)
4. RETURN TOP RESULTS
Usually top 5-10 most similar chunks
5. YOU GET BACK
✓ The relevant chunks
✓ Where they came from (sources + page numbers)
✓ Relevance scores
```
**Example:**
```
Q: "What does the paper say about alignment?"
↓
Q vector: [0.23, -0.51, 0.88, ..., 0.12]
↓
Search: Compare to all chunk vectors
↓
Results:
- Chunk 47 (alignment section): similarity 0.94
- Chunk 63 (safety approaches): similarity 0.88
- Chunk 12 (related work): similarity 0.71
```
---
### Stage 3: Augmentation (How AI Uses It)
Now you have the relevant pieces. The AI uses them:
```
SYSTEM BUILDS A PROMPT:
"You are an AI research assistant.
The user has the following research materials:
[CHUNK 47 CONTENT]
[CHUNK 63 CONTENT]
User question: 'What does the paper say about alignment?'
Answer based on the above materials."
AI RESPONDS:
"Based on the research materials, the paper approaches
alignment through [pulls from chunks] and emphasizes
[pulls from chunks]..."
SYSTEM ADDS CITATIONS:
"- See research materials page 15 for approach details
- See research materials page 23 for emphasis on X"
```
---
## Two Search Modes: Exact vs. Semantic
Open Notebook provides two different search strategies for different goals.
### 1. Text Search (Keyword Matching)
**How it works:**
- Uses BM25 ranking (the same algorithm Google uses)
- Finds chunks containing your keywords
- Ranks by relevance (how often keywords appear, position, etc.)
**When to use:**
- "I remember the exact phrase 'X' and want to find it"
- "I'm looking for a specific name or number"
- "I need the exact quote"
**Example:**
```
Search: "transformer architecture"
Results:
1. Chunk with "transformer architecture" 3 times
2. Chunk with "transformer" and "architecture" separately
3. Chunk with "transformer-based models"
```
### 2. Vector Search (Semantic Similarity)
**How it works:**
- Converts your question to a vector (number embedding)
- Finds chunks with similar vectors
- No keywords needed—finds conceptually similar content
**When to use:**
- "Find content about X (without saying exact words)"
- "I'm exploring a concept"
- "Find similar ideas even if worded differently"
**Example:**
```
Search: "what's the mechanism for model understanding?"
Results (no "understanding" in any chunk):
1. Chunk about interpretability and mechanistic analysis
2. Chunk about feature analysis
3. Chunk about attention mechanisms
Why? The vectors are semantically similar to your concept.
```
---
## Context Management: Your Control Panel
Here's where Open Notebook is different: **You decide what the AI sees.**
### The Three Levels
| Level | What's Shared | Example Cost | Privacy | Use Case |
|-------|---------------|--------------|---------|----------|
| **Full Content** | Complete source text | 10,000 tokens | Low | Detailed analysis, close reading |
| **Summary Only** | AI-generated summary | 2,000 tokens | High | Background material, references |
| **Not in Context** | Nothing | 0 tokens | Max | Confidential, irrelevant, or archived |
### How It Works
**Full Content:**
```
You: "What's the methodology in paper A?"
System:
- Searches paper A
- Retrieves full paper content (or large chunks)
- Sends to AI: "Here's paper A. Answer about methodology."
- AI analyzes complete content
- Result: Detailed, precise answer
```
**Summary Only:**
```
You: "I want to chat using paper A and B"
System:
- For Paper A: Sends AI-generated summary (not full text)
- For Paper B: Sends full content (detailed analysis)
- AI sees 2 sources but in different detail levels
- Result: Uses summaries for context, details for focused content
```
**Not in Context:**
```
You: "I have 10 sources but only want 5 in context"
System:
- Paper A-E: In context (sent to AI)
- Paper F-J: Not in context (AI can't see them, doesn't search them)
- AI never knows these 5 sources exist
- Result: Tight, focused context
```
### Why This Matters
**Privacy**: You control what leaves your system
```
Scenario: Confidential company docs + public research
Control: Public research in context → Confidential docs excluded
Result: AI never sees confidential content
```
**Cost**: You control token usage
```
Scenario: 100 sources for background + 5 for detailed analysis
Control: Full content for 5 detailed, summaries for 95 background
Result: 80% lower token cost than sending everything
```
**Quality**: You control what the AI focuses on
```
Scenario: 20 sources, question requires deep analysis
Control: Full content for relevant source, exclude others
Result: AI doesn't get distracted; gives better answer
```
---
## The Difference: Chat vs. Ask
**IMPORTANT**: These use completely different approaches!
### Chat: Full-Content Context (NO RAG)
**How it works:**
```
YOU:
1. Select which sources to include in context
2. Set context level (full/summary/excluded)
3. Ask question
SYSTEM:
- Takes ALL selected sources (respecting context levels)
- Sends the ENTIRE content to the LLM at once
- NO search, NO retrieval, NO chunking
- AI sees everything you selected
AI:
- Responds based on the full content you provided
- Can reference any part of selected sources
- Conversational: context stays for follow-ups
```
**Use this when**:
- You know which sources are relevant
- You want conversational back-and-forth
- You want AI to see the complete context
- You're doing close reading or analysis
**Advantages:**
- Simple and transparent
- AI sees everything (no missed content)
- Conversational flow
**Limitations:**
- Limited by LLM context window
- You must manually select relevant sources
- Sends more tokens (higher cost with many sources)
---
### Ask: RAG - Automatic Retrieval
**How it works:**
```
YOU:
Ask one complex question
SYSTEM:
1. Analyzes your question
2. Searches across ALL your sources automatically
3. Finds relevant chunks using vector similarity
4. Retrieves only the most relevant pieces
5. Sends ONLY those chunks to the LLM
6. Synthesizes into comprehensive answer
AI:
- Sees ONLY the retrieved chunks (not full sources)
- Answers based on what was found to be relevant
- One-shot answer (not conversational)
```
**Use this when**:
- You have many sources and don't know which are relevant
- You want the AI to search automatically
- You need a comprehensive answer to a complex question
- You want to minimize tokens sent to LLM
**Advantages:**
- Automatic search (you don't pick sources)
- Works across many sources at once
- Cost-effective (sends only relevant chunks)
**Limitations:**
- Not conversational (single question/answer)
- AI only sees retrieved chunks (might miss context)
- Search quality depends on how well question matches content
---
## What This Means: Privacy by Design
Open Notebook's RAG approach gives you something you don't get with ChatGPT or Claude directly:
**You control the boundary between:**
- What stays private (on your system)
- What goes to AI (explicitly chosen)
- What the AI can see (context levels)
### The Audit Trail
Because everything is retrieved explicitly, you can ask:
- "Which sources did the AI use for this answer?" → See citations
- "What exactly did the AI see?" → See chunks in context level
- "Is the AI's claim actually in my sources?" → Verify citation
This prevents hallucinations or misrepresentation better than most systems.
---
## How Embeddings Work (Simplified)
The magic of semantic search comes from embeddings. Here's the intuition:
### The Idea
Instead of storing text, store it as a list of numbers (vectors) that represent "meaning."
```
Chunk: "The transformer uses attention mechanisms"
Vector: [0.23, -0.51, 0.88, 0.12, ..., 0.34]
(1536 numbers for OpenAI)
Another chunk: "Attention allows models to focus on relevant parts"
Vector: [0.24, -0.48, 0.87, 0.15, ..., 0.35]
(similar numbers = similar meaning!)
```
### Why This Works
Words that are semantically similar produce similar vectors. So:
- "alignment" and "interpretability" have similar vectors
- "transformer" and "attention" have related vectors
- "cat" and "dog" are more similar than "cat" and "radiator"
### How Search Works
```
Your question: "How do models understand their decisions?"
Question vector: [0.25, -0.50, 0.86, 0.14, ..., 0.33]
Compare to all stored vectors. Find the most similar:
- Chunk about interpretability: similarity 0.94
- Chunk about explainability: similarity 0.91
- Chunk about feature attribution: similarity 0.88
Return the top matches.
```
This is why semantic search finds conceptually similar content even when words are different.
---
## Key Design Decisions
### 1. Search, Don't Train
**Why?** Fine-tuning is slow and permanent. Search is flexible and reversible.
### 2. Explicit Retrieval, Not Implicit Knowledge
**Why?** You can verify what the AI saw. You have audit trails. You control what leaves your system.
### 3. Multiple Search Types
**Why?** Different questions need different search (keyword vs. semantic). Giving you both is more powerful.
### 4. Context as a Permission System
**Why?** Not everything you save needs to reach AI. You control granularly.
---
## Summary
Open Notebook gives you **two ways** to work with AI:
### Chat (Full-Content)
- Sends entire selected sources to LLM
- Manual control: you pick sources
- Conversational: back-and-forth dialog
- Transparent: you know exactly what AI sees
- Best for: focused analysis, close reading
### Ask (RAG)
- Searches and retrieves relevant chunks automatically
- Automatic: AI finds what's relevant
- One-shot: single comprehensive answer
- Efficient: sends only relevant pieces
- Best for: broad questions across many sources
**Both approaches:**
1. Keep your data private (doesn't leave your system by default)
2. Give you control (you choose which features to use)
3. Create audit trails (citations show what was used)
4. Support multiple AI providers
**Coming Soon**: The community is working on adding RAG capabilities to Chat as well, giving you the best of both worlds.
docs/2-CORE-CONCEPTS/chat-vs-transformations.md
+353
View File
@@ -0,0 +1,353 @@
# Chat vs. Ask vs. Transformations - Which Tool for Which Job?
Open Notebook offers different ways to work with your research. Understanding when to use each is key to using the system effectively.
---
## The Three Interaction Modes
### 1. CHAT - Conversational Exploration with Manual Context
**What it is:** Have a conversation with AI about selected sources.
**The flow:**
```
1. You select which sources to include ("in context")
2. You ask a question
3. AI responds using ONLY those sources
4. You ask follow-up questions (context stays same)
5. You change sources or context level, then continue
```
**Context management:** You explicitly choose which sources the AI can see.
**Conversational:** Multiple questions with shared history.
**Example:**
```
You: [Select sources: "paper1.pdf", "research_notes.txt"]
[Set context: Full content for paper1, Summary for notes]
You: "What's the main argument in these sources?"
AI: "Paper 1 argues X [citation]. Your notes emphasize Y [citation]."
You: "How do they differ?"
AI: "Paper 1 focuses on X [citation], while your notes highlight Y [citation]..."
You: [Now select different sources]
You: "Compare to this other perspective"
AI: "This new source takes a different approach..."
```
**Best for:**
- Exploring a focused topic with specific sources
- Having a dialogue (multiple back-and-forth questions)
- When you know which sources matter
- When you want tight control over what goes to AI
---
### 2. ASK - Automated Comprehensive Search
**What it is:** Ask one complex question, system automatically finds relevant content.
**The flow:**
```
1. You ask a comprehensive question
2. System analyzes the question
3. System automatically searches your sources
4. System retrieves relevant chunks
5. System synthesizes answer from all results
6. You get one detailed answer (not conversational)
```
**Context management:** Automatic. System figures out what's relevant.
**Non-conversational:** One question → one answer. No follow-ups.
**Example:**
```
You: "How do these papers compare their approaches to alignment?
What does each one recommend?"
System:
- Breaks down the question into search strategies
- Searches all sources for alignment approaches
- Searches all sources for recommendations
- Retrieves top 10 relevant chunks
- Synthesizes: "Paper A recommends X [citation].
Paper B recommends Y [citation].
They differ in Z."
You: [Get back one comprehensive answer]
[If you want to follow up, use Chat instead]
```
**Best for:**
- Comprehensive, one-time questions
- Comparing multiple sources at once
- When you want the system to decide what's relevant
- Complex questions that need multiple search angles
- When you don't need a back-and-forth conversation
---
### 3. TRANSFORMATIONS - Template-Based Processing
**What it is:** Apply a reusable template to a source and get structured output.
**The flow:**
```
1. You define a transformation (or choose a preset)
"Extract: main argument, methodology, limitations"
2. You apply it to ONE source at a time
(You can repeat for other sources)
3. For the source:
- Source content + transformation prompt → AI
- Result stored as new insight/note
4. You get back
- Structured output (main argument, methodology, limitations)
- Saved as a note in your notebook
```
**Context management:** Works on one source at a time.
**Reusable:** Apply the same template to different sources (one by one).
**Note**: Currently processes one source at a time. Batch processing (multiple sources at once) is planned for a future release.
**Example:**
```
You: Define transformation
"For each academic paper, extract:
- Main research question
- Methodology used
- Key findings
- Limitations and gaps
- Recommended next research"
You: Apply to paper 1
System:
- Runs the transformation on paper 1
- Result stored as new note
You: Apply same transformation to paper 2, 3, etc.
After 10 papers:
- You have 10 structured notes with consistent format
- Perfect for writing a literature review or comparison
```
**Best for:**
- Extracting the same information from each source (run repeatedly)
- Creating structured summaries with consistent format
- Building a knowledge base of categorized insights
- When you want reusable templates you can apply to each source
---
## Decision Tree: Which Tool to Use?
```
What are you trying to do?
│
├─→ "I want to have a conversation about this topic"
│ └─→ Is the conversation exploratory or fixed?
│ ├─→ Exploratory (I'll ask follow-ups)
│ │ └─→ USE: CHAT
│ │
│ └─→ Fixed (One question → done)
│ └─→ Go to next question
│
├─→ "I need to compare these sources or get a comprehensive answer"
│ └─→ USE: ASK
│
├─→ "I want to extract the same info from each source (one at a time)"
│ └─→ USE: TRANSFORMATIONS (apply to each source)
│
└─→ "I just want to read and search"
└─→ USE: Search (text or vector)
OR read your notes
```
---
## Side-by-Side Comparison
| Aspect | CHAT | ASK | TRANSFORMATIONS |
|--------|------|-----|-----------------|
| **What's it for?** | Conversational exploration | Comprehensive Q&A | Template-based extraction |
| **# of questions** | Multiple (conversational) | One | One template per source |
| **Context control** | Manual (you choose) | Automatic (system searches) | One source at a time |
| **Conversational?** | Yes (follow-ups work) | No (one question only) | No (single operation) |
| **Output** | Natural conversation | Natural answer | Structured note |
| **Time** | Quick (back-and-forth) | Longer (comprehensive) | Per source |
| **Best when** | Exploring & uncertain | Need full picture | Want consistent format |
| **Model speed** | Any | Fast preferred | Any |
---
## Workflow Examples
### Example 1: Academic Research
```
Goal: Write literature review from 15 papers
Step 1: TRANSFORMATIONS
- Define: "Extract abstract, methodology, findings, relevance"
- Apply to paper 1 → get structured note
- Apply to paper 2 → get structured note
- ... repeat for all 15 papers
- Result: 15 structured notes with consistent format
Step 2: Read the notes
- Now you have consistent summaries
Step 3: CHAT or ASK
- Chat: "Help me organize these by theme"
- Ask: "What are the common methodologies across these papers?"
Step 4: Write your review
- Use the transformations as foundation
- Use chat/ask insights for structure
```
### Example 2: Product Research
```
Goal: Understand customer feedback from interviews
Step 1: Add sources (interview transcripts)
Step 2: ASK
- "What are the top 10 pain points mentioned?"
- Get comprehensive answer with citations
Step 3: CHAT
- "Can you help me group these by severity?"
- Continue conversation to prioritize
Step 4: TRANSFORMATIONS (optional)
- Define: "Extract: pain point, frequency, who mentioned it"
- Apply to each interview (one by one)
- Get structured data for analysis
```
### Example 3: Policy Analysis
```
Goal: Compare policy documents
Step 1: Add all policy documents as sources
Step 2: ASK
- "How do these policies differ on climate measures?"
- System searches all docs, gives comprehensive comparison
Step 3: CHAT (if needed)
- "Which policy is most aligned with X goals?"
- Have discussion about trade-offs
Step 4: Export notes
- Save AI responses as notes for reports
```
---
## Context Management: The Control Panel
All three modes let you control what the AI sees.
### In CHAT and TRANSFORMATIONS
```
You choose:
- Which sources to include
- Context level for each:
✓ Full Content (send complete text)
✓ Summary Only (send AI summary, not full text)
✓ Not in Context (exclude entirely)
Example:
Paper A: Full Content (analyzing closely)
Paper B: Summary Only (background)
Paper C: Not in Context (confidential)
```
### In ASK
```
Context is automatic:
- System searches ALL your sources
- Retrieves most relevant chunks
- Sends those to AI
But you can:
- Search in specific notebook
- Filter by source type
- Use the results to decide context for follow-up Chat
```
---
## Model Selection
Each mode works with different models:
### CHAT
- **Any model** works fine
- Fast models (GPT-4o mini, Claude Haiku): Quick responses, good for conversation
- Powerful models (GPT-4o, Claude Sonnet): Better reasoning, better for complex topics
### ASK
- **Fast models preferred** (because it processes multiple searches)
- Can use powerful models if you want deep synthesis
- Example: GPT-4 for strategy planning, GPT-4o-mini for quick facts
### TRANSFORMATIONS
- **Any model** works
- Fast models (cost-effective for batch processing)
- Powerful models (better quality extractions)
---
## Advanced: Chaining Modes Together
You can combine these modes:
```
TRANSFORMATIONS → CHAT
1. Use transformations to extract structured data
2. Use chat to discuss the results
ASK → TRANSFORMATIONS
1. Use Ask to understand what matters
2. Use Transformations to extract it from remaining sources
CHAT → Save as Note → TRANSFORMATIONS
1. Have conversation (Chat)
2. Save good responses as notes
3. Use those notes as context for transformations
```
---
## Summary: When to Use Each
| Situation | Use | Why |
|-----------|-----|-----|
| "I want to explore a topic with follow-up questions" | **CHAT** | Conversational, you control context |
| "I need a comprehensive answer to one complex question" | **ASK** | Automatic search, synthesized answer |
| "I want consistent summaries from each source" | **TRANSFORMATIONS** | Template reuse, apply to each source |
| "I'm comparing two specific sources" | **CHAT** | Select just those 2, have discussion |
| "I need to categorize each source by X criteria" | **TRANSFORMATIONS** | Extract category from each source |
| "I want to understand the big picture across all sources" | **ASK** | Automatic comprehensive search |
| "I want to build a knowledge base" | **TRANSFORMATIONS** | Create structured note from each source |
| "I want to iterate on understanding" | **CHAT** | Multiple questions, refine thinking |
The key insight: **Different questions need different tools.** Open Notebook gives you all three because research rarely fits one mode.
docs/2-CORE-CONCEPTS/index.md
+70
View File
@@ -0,0 +1,70 @@
# Core Concepts - Understand the Mental Model
Before diving into how to use Open Notebook, it's important to understand **how it thinks**. These core concepts explain the "why" behind the design.
## The Five Mental Models
### 1. [Notebooks, Sources, and Notes](notebooks-sources-notes.md)
How Open Notebook organizes your research. Understand the three-tier container structure and how information flows from raw materials to finished insights.
**Key idea**: A notebook is a scoped research container. Sources are inputs (PDFs, URLs, etc.). Notes are outputs (your insights, AI-generated summaries, captured responses).
---
### 2. [AI Context & RAG](ai-context-rag.md)
How Open Notebook makes AI aware of your research - two different approaches.
**Key idea**: **Chat** sends entire selected sources to the LLM (full context, conversational). **Ask** uses RAG (retrieval-augmented generation) to automatically search and retrieve only relevant chunks. Different tools for different needs.
---
### 3. [Chat vs. Transformations](chat-vs-transformations.md)
Why Open Notebook has different interaction modes and when to use each one.
**Key idea**: Chat is conversational exploration (you control context). Transformations are insight extractions. They reduced content to smaller bits of concentrated/dense information, which is much more suitable for an AI to use.
---
### 4. [Context Management](chat-vs-transformations.md#context-management-the-control-panel)
Your control panel for privacy and cost. Decide what data actually reaches AI.
**Key idea**: You choose three levels—not in context (private), summary only (condensed), or full content (complete access). This gives you fine-grained control.
---
### 5. [Podcasts Explained](podcasts-explained.md)
Why Open Notebook can turn research into audio and why this matters.
**Key idea**: Podcasts transform your research into a different consumption format. Instead of reading, someone can listen and absorb your insights passively.
---
## Read This Section If:
- **You're new to Open Notebook** — Start here to understand how the system works conceptually before learning the features
- **You're confused about Chat vs Ask** — Section 2 explains the difference (full-content vs RAG)
- **You're wondering when to use Chat vs Transformations** — Section 3 clarifies the differences
- **You want to understand privacy controls** — Section 4 shows you what you can control
- **You're curious about podcasts** — Section 5 explains the architecture and why it's different from competitors
---
## The Big Picture
Open Notebook is built on a simple insight: **Your research deserves to stay yours**.
That means:
- **Privacy by default** — Your data doesn't leave your infrastructure unless you explicitly choose
- **AI as a tool, not a gatekeeper** — You decide which sources the AI sees, not the AI deciding for you
- **Flexible consumption** — Read, listen, search, chat, or transform your research however makes sense
These core concepts explain how that works.
---
## Next Steps
1. **Just want to use it?** → Go to [User Guide](../3-USER-GUIDE/index.md)
2. **Want to understand it first?** → Read the 5 sections above (15 min)
3. **Setting up for the first time?** → Go to [Installation](../1-INSTALLATION/index.md)
docs/2-CORE-CONCEPTS/notebooks-sources-notes.md
+284
View File
@@ -0,0 +1,284 @@
# Notebooks, Sources, and Notes - The Container Model
Open Notebook organizes research in three connected layers. Understanding this hierarchy is key to using the system effectively.
## The Three-Layer Structure
```
┌─────────────────────────────────────┐
│ NOTEBOOK (The Container) │
│ "My AI Safety Research 2026" │
├─────────────────────────────────────┤
│ │
│ SOURCES (The Raw Materials) │
│ ├─ safety_paper.pdf │
│ ├─ alignment_video.mp4 │
│ └─ prompt_injection_article.html │
│ │
│ NOTES (The Processed Insights) │
│ ├─ AI Summary (auto-generated) │
│ ├─ Key Concepts (transformation) │
│ ├─ My Research Notes (manual) │
│ └─ Chat Insights (from conversation)
│ │
└─────────────────────────────────────┘
```
---
## 1. NOTEBOOKS - The Research Container
### What Is a Notebook?
A **notebook** is a *scoped container* for a research project or topic. It's your research workspace.
Think of it like a physical notebook: everything inside is about the same topic, shares the same context, and builds toward the same goals.
### What Goes In?
- **A description** — "This notebook collects research on X topic"
- **Sources** — The raw materials you add
- **Notes** — Your insights and outputs
- **Conversation history** — Your chats and questions
### Why This Matters
**Isolation**: Each notebook is completely separate. Sources in Notebook A never appear in Notebook B. This lets you:
- Keep different research topics completely isolated
- Reuse source names across notebooks without conflicts
- Control which AI context applies to which research
**Shared Context**: All sources and notes in a notebook inherit the notebook's context. If your notebook is titled "AI Safety 2026" with description "Focusing on alignment and interpretability," that context applies to all AI interactions within that notebook.
**Parallel Projects**: You can have 10 notebooks running simultaneously. Each one is its own isolated research environment.
### Example
```
Notebook: "Customer Research - Product Launch"
Description: "User interviews and feedback for Q1 2026 launch"
→ All sources added to this notebook are about customer feedback
→ All notes generated are in that context
→ When you chat, the AI knows you're analyzing product launch feedback
→ Different from your "Market Analysis - Competitors" notebook
```
---
## 2. SOURCES - The Raw Materials
### What Is a Source?
A **source** is a *single piece of input material* — the raw content you bring in. Sources never change; they're just processed and indexed.
### What Can Be a Source?
- **PDFs** — Research papers, reports, documents
- **Web links** — Articles, blog posts, web pages
- **Audio files** — Podcasts, interviews, lectures
- **Video files** — Tutorials, presentations, recordings
- **Plain text** — Notes, transcripts, passages
- **Uploaded text** — Paste content directly
### What Happens When You Add a Source?
```
1. EXTRACTION
File/URL → Extract text and metadata
(OCR for PDFs, web scraping for URLs, speech-to-text for audio)
2. CHUNKING
Long text → Break into searchable chunks
(Prevents "too much context" in single query)
3. EMBEDDING
Each chunk → Generate semantic vector
(Allows AI to find conceptually similar content)
4. STORAGE
Chunks + vectors → Store in database
(Ready for search and retrieval)
```
### Key Properties
**Immutable**: Once added, the source doesn't change. If you need a new version, add it as a new source.
**Indexed**: Sources are automatically indexed for search (both text and semantic).
**Scoped**: A source belongs to exactly one notebook.
**Referenceable**: Other sources and notes can reference this source by citation.
### Example
```
Source: "openai_charter.pdf"
Type: PDF document
What happens:
→ PDF is uploaded
→ Text is extracted (including images)
→ Text is split into 50 chunks (paragraphs, sections)
→ Each chunk gets an embedding vector
→ Now searchable by: "OpenAI's approach to safety"
```
---
## 3. NOTES - The Processed Insights
### What Is a Note?
A **note** is a *processed output* — something you created or AI created based on your sources. Notes are the "results" of your research work.
### Types of Notes
#### Manual Notes
You write them yourself. They're your original thinking, capturing:
- What you learned from sources
- Your analysis and interpretations
- Your next steps and questions
#### AI-Generated Notes
Created by applying AI processing to sources:
- **Transformations** — Structured extraction (main points, key concepts, methodology)
- **Chat Responses** — Answers you saved from conversations
- **Ask Results** — Comprehensive answers saved to your notebook
#### Captured Insights
Notes you explicitly saved from interactions:
- "Save this response as a note"
- "Save this transformation result"
- Convert any AI output into a permanent note
### What Can Notes Contain?
- **Text** — Your writing or AI-generated content
- **Citations** — References to specific sources
- **Metadata** — When created, how created (manual/AI), which sources influenced it
- **Tags** — Your categorization (optional but useful)
### Why Notes Matter
**Knowledge Accumulation**: Notes become your actual knowledge base. They're what you take away from the research.
**Searchable**: Notes are searchable along with sources. "Find everything about X" includes your notes, not just sources.
**Citable**: Notes can cite sources, creating an audit trail of where insights came from.
**Shareable**: Notes are your outputs. You can share them, publish them, or build on them in other projects.
---
## How They Connect: The Data Flow
```
YOU
│
├─→ Create Notebook ("AI Research")
│
├─→ Add Sources (papers, articles, videos)
│ └─→ System: Extract, embed, index
│
├─→ Search Sources (text or semantic)
│ └─→ System: Find relevant chunks
│
├─→ Apply Transformations (extract insights)
│ └─→ Creates Notes
│
├─→ Chat with Sources (explore with context control)
│ ├─→ Can save responses as Notes
│ └─→ Notes include citations
│
├─→ Ask Questions (automated comprehensive search)
│ ├─→ Can save results as Notes
│ └─→ Notes include citations
│
└─→ Generate Podcast (transform notebook into audio)
└─→ Uses all sources + notes for content
```
---
## Key Design Decisions
### 1. One Notebook Per Source
Each source belongs to exactly one notebook. This creates clear boundaries:
- No ambiguity about which research project a source is in
- Easy to isolate or export a complete project
- Clean permissions model (if someone gets access to notebook, they get access to all its sources)
### 2. Immutable Sources, Mutable Notes
Sources never change (once added, always the same). But notes can be edited or deleted. Why?
- Sources are evidence → evidence shouldn't be altered
- Notes are your thinking → thinking evolves as you learn
### 3. Explicit Context Control
Sources don't automatically go to AI. You decide which sources are "in context" for each interaction:
- Chat: You manually select which sources to include
- Ask: System automatically figures out which sources to search
- Transformations: You choose which sources to transform
This is different from systems that always send everything to AI.
---
## Mental Models Explained
### Notebook as Boundaries
Think of a notebook like a Git repository:
- Everything in it is about the same topic
- You can clone/fork it (copy to new project)
- It has clear entry/exit points
- You know exactly what's included
### Sources as Evidence
Think of sources like exhibits in a legal case:
- Once filed, they don't change
- They can be cited and referenced
- They're the ground truth for what you're basing claims on
- Multiple sources can be cross-referenced
### Notes as Synthesis
Think of notes like your case brief:
- You write them based on evidence
- They're your interpretation
- You can cite which evidence supports each claim
- They're what you actually share or act on
---
## Common Questions
### Can I move a source to a different notebook?
Not directly. Each source is tied to one notebook. If you want it in multiple notebooks, add it again (uploads are fast if it's already processed).
### Can a note reference sources from a different notebook?
No. Notes stay within their notebook and reference sources within that notebook. This keeps boundaries clean.
### What if I want to group sources within a notebook?
Use tags. You can tag sources ("primary research," "background," "methodology") and filter by tags.
### Can I merge two notebooks?
Not built-in, but you can manually copy sources from one notebook to another by re-uploading them.
---
## Summary
| Concept | Purpose | Lifecycle | Scope |
|---------|---------|-----------|-------|
| **Notebook** | Container + context | Create once, configure | All its sources + notes |
| **Source** | Raw material | Add → Process → Store | One notebook |
| **Note** | Processed output | Create/capture → Edit → Share | One notebook |
This three-layer model gives you:
- **Clear organization** (everything scoped to projects)
- **Privacy control** (isolated notebooks)
- **Audit trails** (notes cite sources)
- **Flexibility** (notes can be manual or AI-generated)
docs/2-CORE-CONCEPTS/podcasts-explained.md
+426
View File
@@ -0,0 +1,426 @@
# Podcasts Explained - Research as Audio Dialogue
Podcasts are Open Notebook's highest-level transformation: converting your research into audio dialogue for a different consumption pattern.
---
## Why Podcasts Matter
### The Problem
Research naturally accumulates as text: PDFs, articles, web pages, notes. This creates a friction point:
**To consume research, you must:**
- Sit down at a desk
- Focus intently
- Read actively
- Take notes
- Set aside dedicated time
**But much of life is passive time:**
- Commuting
- Exercising
- Doing dishes
- Driving
- Walking
- Idle moments
### The Solution
Convert your research into audio dialogue so you can consume it passively.
```
Before (Text-based):
Research pile → Must schedule reading time → Requires focus
After (Podcast):
Research pile → Podcast → Can listen while commuting
→ Absorb while exercising
→ Understand while walking
→ Engage without screen time
```
---
## What Makes It Special: Open Notebook vs. Competitors
### Google Notebook LM Podcasts
- **Fixed format**: 2 hosts, always conversational
- **Limited customization**: You can't choose who the "hosts" are
- **One TTS voice per speaker**: Can't customize voices
- **Only uses cloud services**: No local options
### Open Notebook Podcasts
- **Customizable format**: 1-4 speakers, you design them
- **Rich speaker profiles**: Create personas with backstories and expertise
- **Multiple TTS options**:
- OpenAI (natural, fast)
- Google TTS (high quality)
- ElevenLabs (beautiful voices, accents)
- Local TTS (privacy-first, no API calls)
- **Async generation**: Doesn't block your work
- **Full control**: Choose outline structure, tone, depth
---
## How Podcast Generation Works
### Stage 1: Content Selection
You choose what goes into the podcast:
```
Notebook content → Which sources? → Which notes?
→ Which topics to focus on?
→ Depth of coverage?
```
### Stage 2: Episode Profile
You define how you want the podcast structured:
```
Episode Profile
├─ Topic: "AI Safety Approaches"
├─ Length: 20 minutes
├─ Tone: Academic but accessible
├─ Format: Debate (2 speakers with opposing views)
├─ Audience: Researchers new to the field
└─ Focus areas: Main approaches, pros/cons, open questions
```
### Stage 3: Speaker Configuration
You create speaker personas (1-4 speakers):
```
Speaker 1: "Expert Alex"
├─ Expertise: "Deep knowledge of alignment research"
├─ Personality: "Rigorous, academic, patient with explanation"
├─ Accent: (Optional) "British English"
└─ Voice Model: Selected from model registry (e.g., OpenAI TTS)
└─ Optional per-speaker override of the episode's default voice model
Speaker 2: "Researcher Sam"
├─ Expertise: "Field observer, pragmatic perspective"
├─ Personality: "Curious, asks clarifying questions"
├─ Accent: "American English"
└─ Voice Model: Selected from model registry (e.g., ElevenLabs TTS)
```
### Stage 4: Outline Generation
System generates episode outline:
```
EPISODE: "AI Safety Approaches"
1. Introduction (2 min)
Alex: Introduces topic and speakers
Sam: What will we cover today?
2. Main Approaches (8 min)
Alex: Explains top 3 approaches
Sam: Asks about tradeoffs
3. Debate: Best approach? (6 min)
Alex: Advocates for approach A
Sam: Argues for approach B
4. Open Questions (3 min)
Both: What's unsolved?
5. Conclusion (1 min)
Recap and where to learn more
```
### Stage 5: Dialogue Generation
System generates dialogue based on outline:
```
Alex: "Today we're exploring three major approaches to AI alignment..."
Sam: "That's a great start. Can you break down what we mean by alignment?"
Alex: "Good question. Alignment means ensuring AI systems pursue the goals
we actually want them to pursue, not just what we literally asked for.
There's a classic example of a paperclip maximizer..."
Sam: "Interesting. So it's about solving the intention problem?"
Alex: "Exactly. And that's where the three approaches come in..."
```
### Stage 6: Text-to-Speech
System converts dialogue to audio using the voice models configured in the model registry. Credentials are automatically resolved from each model's configuration.
```
Alex's text → Voice model (from registry) → Alex's voice (audio file)
Sam's text → Voice model (from registry) → Sam's voice (audio file)
Audio files → Mix together → Final podcast MP3
```
---
## When Things Go Wrong: Failures & Retry
Podcast generation involves multiple steps (outline, transcript, TTS) and depends on external AI providers. Sometimes things fail.
### What Happens on Failure
When podcast generation fails (e.g., wrong model configured, API key expired, provider outage):
- The episode is marked as **Failed** with a red badge
- The **error message** from the AI provider is displayed so you can understand what went wrong
- No duplicate episodes are created — automatic retries are disabled to prevent confusion
### How to Retry a Failed Episode
1. Go to the podcast's **Episodes** tab
2. Find the failed episode — it shows a red "FAILED" badge and an error details box
3. Click the **Retry** button
4. The failed episode is deleted and a new generation job is submitted
5. The new episode appears with "pending" status
### Common Failure Causes
| Error | What to Do |
|-------|-----------|
| Invalid API key | Check Settings -> Credentials for the TTS and language model providers |
| Model not found | Verify the model exists in the model registry and has valid credentials configured |
| Rate limit exceeded | Wait a few minutes and retry |
| Provider unavailable | Check provider status page; retry later |
---
## Key Architecture Decisions
### 1. Asynchronous Processing
Podcasts are generated in the background. You upload → system processes → you download when ready.
**Why?** Podcast generation takes time (10+ minutes for a 30-minute episode). Blocking would lock up your interface.
### 2. Multi-Speaker Support
Unlike Google Notebook LM (always 2 hosts), you choose 1-4 speakers.
**Why?** Different discussions work better with different formats:
- Expert monologue (1 speaker)
- Interview (2 speakers: host + expert)
- Debate (2 speakers: opposing views)
- Panel discussion (3-4 speakers: different expertise)
### 3. Speaker Customization
You create rich speaker profiles, not just "Host A" and "Host B".
**Why?** Makes podcasts more engaging and authentic. Different speakers bring different perspectives.
### 4. Multiple TTS Providers
You're not locked into one voice provider.
**Why?**
- Cost optimization (some providers cheaper)
- Quality preferences (some voices more natural)
- Privacy options (local TTS for sensitive content)
- Accessibility (different accents, genders, styles)
### 5. Local TTS Option
Can generate podcasts entirely offline with local text-to-speech.
**Why?** For sensitive research, never send audio to external APIs.
---
## Use Cases Show Why This Matters
### Academic Publishing
```
Traditional: Academic paper → PDF
Problem: Hard to consume, linear reading required
Open Notebook:
Research materials → Podcast (expert explaining methodology)
→ Podcast (debate format: different interpretations)
→ Different consumption for different audiences
```
### Content Creation
```
Blog creator: Has research pile on a topic
Problem: Doesn't have time to write the article
Solution:
Add research → Create podcast → Transcribe → Becomes article
OR: Podcast BECOMES the content (upload to podcast platforms)
```
### Educational Content
```
Educator: Has reading materials for a course
Problem: Students don't read the papers
Solution:
Create podcast with expert explaining papers
Students listen → Better engagement → Discussions can reference podcast
```
### Market Research
```
Product manager: Has interviews with customers
Problem: Too many hours of audio to review
Solution:
Create podcast with debate format (customer perspective vs. team perspective)
Much more engaging than raw transcripts
```
### Knowledge Transfer
```
Domain expert: Leaving the organization
Problem: How to preserve expertise?
Solution:
Create expert-mode podcast explaining frameworks, decision-making, context
New team member listens, gets context faster than reading 100 documents
```
---
## The Difference: Active vs. Passive Learning
### Text-Based Research (Active)
- **Effort**: High (must focus, read, synthesize)
- **When**: Dedicated study time
- **Cost**: Time is expensive (can't multitask)
- **Best for**: Deep dives, precise information
- **Format**: Whatever you write (notes, articles, books)
### Audio Podcast (Passive)
- **Effort**: Low (just listen)
- **When**: Anywhere, anytime
- **Cost**: Low (can multitask)
- **Best for**: Overview, context, exploration
- **Format**: Dialogue (more engaging than narration)
**They complement each other:**
1. **First encounter**: Listen to podcast (passive, get context)
2. **Deep dive**: Read source materials (active, precise)
3. **Mastery**: Both together (understand big picture + details)
---
## How Podcasts Fit Into Your Workflow
```
1. Build notebook (add sources)
↓
2. Apply transformations (extract insights)
↓
3. Chat/Ask (explore content)
↓
4. Decide on podcast
├─→ Create speaker profiles
├─→ Define episode profile
├─→ Configure voice models (from model registry)
└─→ Generate podcast
↓
5. Listen while commuting/exercising
↓
6. Reference sources for deep dive
↓
7. Repeat for different formats/speakers/focus
```
---
## Advanced: Multiple Podcasts from Same Research
You can create different podcasts from the same sources:
### Example: AI Safety Research
```
Podcast 1: "Expert Monologue"
Speaker: Researcher explaining field
Format: Educational, comprehensive
Audience: Students new to field
Podcast 2: "Debate Format"
Speakers: Optimist vs. skeptic
Format: Discussion of tradeoffs
Audience: Advanced researchers
Podcast 3: "Interview Format"
Speakers: Journalist + expert
Format: Q&A about practical applications
Audience: Industry practitioners
```
Each tells the same story from different angles.
---
## Privacy & Data Considerations
### Where Your Data Goes
**Option 1: Cloud TTS (Faster, Higher Quality)**
```
Your outline → API call to TTS provider
→ Audio returned
→ Stored in your notebook
Provider sees: Your outlined script (not raw sources)
Privacy level: Medium (outline is shared, sources aren't)
```
**Option 2: Local TTS (Slower, Maximum Privacy)**
```
Your outline → Local TTS engine (runs on your machine)
→ Audio generated locally
→ Stored in your notebook
Provider sees: Nothing
Privacy level: Maximum (everything local)
```
### Recommendation
- **Sensitive research**: Use local TTS, no API calls
- **Less sensitive**: Use ElevenLabs or Google (both handle audio data professionally)
- **Mixed**: Use local TTS for speakers reading sensitive content
---
## Cost Considerations
### Cloud TTS Costs
| Provider | Cost | Quality | Speed |
|----------|------|---------|-------|
| OpenAI | ~$0.015 per minute | Good | Fast |
| Google | ~$0.004 per minute | Excellent | Fast |
| ElevenLabs | ~$0.10 per minute | Exceptional | Medium |
| Local TTS | Free | Basic | Slow |
A 30-minute podcast costs:
- OpenAI: ~$0.45
- Google: ~$0.12
- ElevenLabs: ~$3.00
- Local: Free (but slow)
---
## Summary: Why Podcasts Are Special
**Podcasts transform your research consumption:**
| Aspect | Text | Podcast |
|--------|------|---------|
| **How consumed?** | Active reading | Passive listening |
| **Where consumed?** | Desk | Anywhere |
| **Multitasking** | Hard | Easy |
| **Time commitment** | Scheduled | Flexible |
| **Format** | Whatever | Natural dialogue |
| **Engagement** | Academic | Conversational |
| **Accessibility** | Text-based | Audio-based |
**In Open Notebook specifically:**
- **Full customization** — you create speakers and format
- **Privacy options** — local TTS for sensitive content
- **Cost control** — choose TTS provider based on budget
- **Non-blocking** — generates in background
- **Multiple versions** — create different podcasts from same research
This is why podcasts matter: they change *when* and *how* you can consume your research.
docs/3-USER-GUIDE/adding-sources.md
+429
View File
@@ -0,0 +1,429 @@
# Adding Sources - Getting Content Into Your Notebook
Sources are the raw materials of your research. This guide covers how to add different types of content.
---
## Quick-Start: Add Your First Source
### Option 1: Upload a File (PDF, Word, etc.)
```
1. In your notebook, click "Add Source"
2. Select "Upload File"
3. Choose a file from your computer
4. Click "Upload"
5. Wait 30-60 seconds for processing
6. Done! Source appears in your notebook
```
### Option 2: Add a Web Link
```
1. Click "Add Source"
2. Select "Web Link"
3. Paste URL: https://example.com/article
4. Click "Add"
5. Wait for processing (usually faster than files)
6. Done!
```
### Option 3: Paste Text
```
1. Click "Add Source"
2. Select "Text"
3. Paste or type your content
4. Click "Save"
5. Done! Immediately available
```
---
## Supported File Types
### Documents
- **PDF** (.pdf) — Best support, including scanned PDFs with OCR
- **Word** (.docx, .doc) — Full support
- **PowerPoint** (.pptx) — Slides converted to text
- **Excel** (.xlsx, .xls) — Spreadsheet data
- **EPUB** (.epub) — eBook files
- **Markdown** (.md, .txt) — Plain text formats
- **HTML** (.html, .htm) — Web page files
**File size limits:** Up to ~100MB (varies by system)
**Processing time:** 10 seconds - 2 minutes (depending on length and file type)
### Audio & Video
- **Audio**: MP3, WAV, M4A, OGG, FLAC (~30 seconds - 3 minutes per hour)
- **Video**: MP4, AVI, MOV, MKV, WebM (~3-10 minutes per hour)
- **YouTube**: Direct URL support
- **Podcasts**: RSS feed URL
**Automatic transcription**: Audio/video is transcribed to text automatically. This requires enabling speech-to-text in settings.
### Web Content
- **Articles**: Blog posts, news articles, Medium
- **YouTube**: Full videos or playlists
- **PDFs online**: Direct PDF links
- **News**: News site articles
**Just paste the URL** in "Web Link" section.
### What Doesn't Work
- Paywalled content (WSJ, FT, etc.) — Can't extract
- Password-protected PDFs — Can't open
- Pure image files (.jpg, .png) — Except scanned PDFs which have OCR
- Very large files (>100MB) — Timeout
---
## What Happens When You Add a Source
The system automatically does four things:
```
1. EXTRACT TEXT
File/URL → Readable text
(PDFs get OCR if scanned)
(Videos get transcribed if enabled)
2. BREAK INTO CHUNKS
Long text → ~500-word pieces
(So search finds specific parts, not whole document)
3. CREATE EMBEDDINGS
Each chunk → Vector representation
(Enables semantic/concept search)
4. INDEX & STORE
Everything → Database
(Ready to search and retrieve)
```
**Time to use:** After the progress bar completes, the source is ready immediately. Embeddings are created in the background.
---
## Step-by-Step for Different Types
### PDFs
**Best practices:**
```
Clean PDFs:
1. Upload → Done
2. Processing time: ~30-60 seconds
Scanned/Image PDFs:
1. Upload same way
2. System auto-detects and uses OCR
3. Processing time: ~2-3 minutes
4. (Higher, due to OCR overhead)
Large PDFs (50+ pages):
1. Consider splitting into smaller files
2. Or upload as-is (system handles it)
3. Processing time scales with size
```
**Common issues:**
- "Can't extract text" → PDF is corrupted or has copy protection
- Solution: Try opening in Adobe. If it won't, the PDF is likely protected.
### Web Links / Articles
**Best practices:**
```
1. Copy full URL from browser: https://example.com/article-title
2. Paste in "Web Link"
3. Click Add
4. Wait for extraction
Processing time: Usually 5-15 seconds
```
**What works:**
- Standard web articles
- Blog posts
- News articles
- Wikipedia pages
- Medium posts
- Substack articles
**What doesn't work:**
- Twitter threads (unreliable)
- Paywalled articles (can't access)
- JavaScript-heavy sites (content not extracted)
**Pro tip:** If it doesn't work, copy the article text and paste as "Text" instead.
### Audio Files
**Best practices:**
```
1. Ensure speech-to-text is enabled in Settings
2. Upload MP3, WAV, or M4A file
3. System automatically transcribes to text
4. Processing time: ~1 minute per 5 minutes of audio
Example:
- 1-hour podcast → 12 minutes processing
- 10-minute recording → 2 minutes processing
```
**Quality matters:**
- Clear audio: Fast transcription
- Muffled/noisy audio: Slower, less accurate transcription
- Background noise: Try to minimize before uploading
**Tip:** If audio quality is poor, the AI might misinterpret content. You can manually correct transcription if needed.
### YouTube Videos
**Best practices:**
```
Two ways to add:
Method 1: Direct URL
1. Copy YouTube URL: https://www.youtube.com/watch?v=...
2. Paste in "Web Link"
3. Click Add
4. System extracts captions (if available) + transcript
Method 2: Playlist
1. Paste playlist URL
2. System adds all videos as separate sources
3. Each video processed separately
4. Takes longer (multiple videos)
```
**What's extracted:**
- Captions/subtitles (if available)
- Transcription (if captions aren't available)
- Basic metadata (title, channel, length)
**Processing:**
- 10-minute video: ~2-3 minutes
- 1-hour video: ~10-15 minutes
### Text / Paste Content
**Best practices:**
```
1. Select "Text" when adding source
2. Paste or type content
3. System processes immediately
4. No wait time needed
Good for:
- Notes you want to reference
- Quotes from books
- Transcripts you have handy
- Quick research snippets
```
---
## Managing Your Sources
### Viewing Source Details
```
Click on source → See:
- Original file name/title
- When it was added
- Size and format
- Processing status
- Number of chunks
```
### Organizing with Metadata
You can add to each source:
- **Title**: Better name than original filename
- **Tags**: Category labels ("primary research", "background", "competitor analysis")
- **Description**: A few notes about what it contains
**Why this matters:**
- Makes sources easier to find
- Helps when contextualizing for Chat
- Useful for organizing large notebooks
### Searching Within Sources
```
After sources are added, you can:
Text search: "Find exact phrase"
Vector search: "Find conceptually similar"
Both search across all sources in notebook.
Results show:
- Which source
- Which section
- Relevance score
```
---
## Context Management: How Sources Get Used
You control how AI accesses sources:
### Three Levels (for Chat)
**Full Content:**
```
AI sees: Complete source text
Cost: 100% of tokens
Use when: Analyzing in detail, need precise citations
Example: "Analyze this methodology paper closely"
```
**Summary Only:**
```
AI sees: AI-generated summary (not full text)
Cost: ~10-20% of tokens
Use when: Background material, reference context
Example: "Use this as context but focus on the main source"
```
**Not in Context:**
```
AI sees: Nothing (excluded)
Cost: 0 tokens
Use when: Confidential, not relevant, or archived
Example: "Keep this in notebook but don't use in this conversation"
```
### How to Set Context (in Chat)
```
1. Go to Chat
2. Click "Select Context Sources"
3. For each source:
- Toggle ON/OFF (include/exclude)
- Choose level (Full/Summary/Excluded)
4. Click "Save"
5. Now chat uses these settings
```
---
## Common Mistakes
| Mistake | What Happens | How to Fix |
|---------|--------------|-----------|
| Upload 200 sources at once | System gets slow, processing stalls | Add 10-20 at a time, wait for processing |
| Use full content for all sources | Token usage skyrockets, expensive | Use "Summary" or "Excluded" for background material |
| Add huge PDFs without splitting | Processing is slow, search results less precise | Consider splitting large PDFs into chapters |
| Forget source titles | Can't distinguish between similar sources | Rename sources with descriptive titles right after uploading |
| Don't tag sources | Hard to find and organize later | Add tags immediately: "primary", "background", etc. |
| Mix languages in one source | Transcription/embedding quality drops | Keep each language in separate sources |
| Use same source multiple times | Takes up space, creates confusion | Add once; reuse in multiple chats/notebooks |
---
## Processing Status & Troubleshooting
### What the Status Indicators Mean
```
🟡 Processing
→ Source is being extracted and embedded
→ Wait 30 seconds - 3 minutes depending on size
→ Don't use in Chat yet
🟢 Ready
→ Source is processed and searchable
→ Can use immediately in Chat
→ Can apply transformations
🔴 Error
→ Something went wrong
→ Common reasons:
- Unsupported file format
- File too large or corrupted
- Network timeout
⚪ Not in Context
→ Source added but excluded from Chat
→ Still searchable, not sent to AI
```
### Common Errors & Solutions
**"Unsupported file type"**
- You tried to upload a format not in the list (e.g., `.webp` image)
- Solution: Convert to supported format (PDF for documents, MP3 for audio)
**"Processing timeout"**
- Very large file (>100MB) or very long audio
- Solution: Split into smaller pieces or try uploading again
**"Transcription failed"**
- Audio quality too poor or language not detected
- Solution: Re-record with better quality, or paste text transcript manually
**"Web link won't extract"**
- Website blocks automated access or uses JavaScript for content
- Solution: Copy the article text and paste as "Text" instead
---
## Tips for Best Results
### For PDFs
- Clean, digital PDFs work best
- Remove copy protection if present (legally)
- Scanned PDFs work but take longer
### For Web Articles
- Use full URL including domain
- Avoid cookie/popup-laden sites
- If extraction fails, copy-paste text instead
### For Audio
- Clear, well-recorded audio transcribes better
- Remove background noise if possible
- YouTube videos usually have good transcriptions built-in
### For Large Documents
- Consider splitting into smaller sources
- Gives more precise search results
- Processing is faster for smaller pieces
### For Organization
- Name sources clearly (not "document_2.pdf")
- Add tags immediately after uploading
- Use descriptions for complex documents
---
## What Comes After: Using Your Sources
Once you've added sources, you can:
- **Chat** → Ask questions (see [Chat Effectively](chat-effectively.md))
- **Search** → Find specific content (see [Search Effectively](search.md))
- **Transformations** → Extract structured insights (see [Working with Notes](working-with-notes.md))
- **Ask** → Get comprehensive answers (see [Search Effectively](search.md))
- **Podcasts** → Turn into audio (see [Creating Podcasts](creating-podcasts.md))
---
## Summary Checklist
Before adding sources, confirm:
- [ ] File is in supported format
- [ ] File is under 100MB (or splitting large ones)
- [ ] Web links are full URLs (not shortened)
- [ ] Audio files have clear speech (if transcription-dependent)
- [ ] You've named source clearly
- [ ] You've added tags for organization
- [ ] You understand context levels (Full/Summary/Excluded)
Done! Sources are now ready for Chat, Search, Transformations, and more.
docs/3-USER-GUIDE/api-configuration.md
+386
View File
@@ -0,0 +1,386 @@
# API Configuration
Configure AI provider credentials through the Settings UI. No file editing required.
> **Credential System**: Open Notebook uses encrypted credentials stored in the database. Each credential connects to a provider and allows you to discover, register, and test models.
---
## Overview
Open Notebook manages AI provider access through a **credential-based system**:
1. You create a **credential** for each provider (API key + settings)
2. Credentials are **encrypted** and stored in the database
3. You **test connections** to verify credentials work
4. You **discover and register models** from each credential
5. Models are linked to credentials for direct configuration
---
## Encryption Setup
Before storing credentials, you must configure an encryption key.
### Setting the Encryption Key
Add `OPEN_NOTEBOOK_ENCRYPTION_KEY` to your docker-compose.yml:
```yaml
environment:
- OPEN_NOTEBOOK_ENCRYPTION_KEY=my-secret-passphrase
```
Any string works as a key — it will be securely derived via SHA-256 internally.
> **Warning**: If you change or lose the encryption key, **all stored credentials become unreadable**. Back up your encryption key securely and separately from your database backups.
### Docker Secrets Support
Both password and encryption key support Docker secrets:
```yaml
# docker-compose.yml
services:
open_notebook:
environment:
- OPEN_NOTEBOOK_PASSWORD_FILE=/run/secrets/app_password
- OPEN_NOTEBOOK_ENCRYPTION_KEY_FILE=/run/secrets/encryption_key
secrets:
- app_password
- encryption_key
secrets:
app_password:
file: ./secrets/password.txt
encryption_key:
file: ./secrets/encryption_key.txt
```
### Encryption Details
API keys stored in the database are encrypted using Fernet (AES-128-CBC + HMAC-SHA256).
| Configuration | Behavior |
|---------------|----------|
| Encryption key set | Keys encrypted with your key |
| No encryption key set | Storing credentials is disabled |
---
## Accessing Credential Configuration
1. Click **Settings** in the navigation bar
2. Select **API Keys** tab
3. You'll see existing credentials and an **Add Credential** button
```
Navigation: Settings → API Keys
```
---
## Supported Providers
### Cloud Providers
| Provider | Required Fields | Optional Fields |
|----------|-----------------|-----------------|
| OpenAI | API Key | — |
| Anthropic | API Key | — |
| Google Gemini | API Key | — |
| Groq | API Key | — |
| Mistral | API Key | — |
| DeepSeek | API Key | — |
| xAI | API Key | — |
| OpenRouter | API Key | — |
| Voyage AI | API Key | — |
| ElevenLabs | API Key | — |
### Local/Self-Hosted
| Provider | Required Fields | Notes |
|----------|-----------------|-------|
| Ollama | Base URL | Typically `http://localhost:11434` or `http://ollama:11434` |
### Enterprise
| Provider | Required Fields | Optional Fields |
|----------|-----------------|-----------------|
| Azure OpenAI | API Key, URL Base (Azure endpoint) | Service-specific endpoints (LLM, Embedding, STT, TTS) |
| OpenAI-Compatible | Base URL | API Key, Service-specific configs |
| Vertex AI | Project ID, Location, Credentials Path | — |
---
## Creating a Credential
### Step 1: Add Credential
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select your provider
4. Give it a descriptive name (e.g., "My OpenAI Key", "Work Anthropic")
5. Fill in the required fields (API key, base URL, etc.)
6. Click **Save**
### Step 2: Test Connection
1. On your new credential card, click **Test Connection**
2. Wait for the result:
| Result | Meaning |
|--------|---------|
| Success | Key is valid, provider accessible |
| Invalid API key | Check key format and value |
| Connection failed | Check URL, network, firewall |
### Step 3: Discover Models
1. Click **Discover Models** on the credential card
2. The system queries the provider for available models
3. Review the discovered models
### Step 4: Register Models
1. Select the models you want to use
2. Click **Register Models**
3. The models are now available throughout Open Notebook
---
## Multi-Credential Support
Each provider can have **multiple credentials**. This is useful when:
- You have different API keys for different projects
- You want to test with different endpoints
- Multiple team members need separate credentials
### Creating Multiple Credentials
1. Click **Add Credential** again
2. Select the same provider
3. Fill in different credentials
4. Each credential can discover and register its own models
### How Models Link to Credentials
When you register models from a credential, those models are linked to that specific credential. This means:
- Each model knows which API key to use
- You can have models from different credentials for the same provider
- Deleting a credential removes its linked models
---
## Testing Connections
Click **Test Connection** to verify your credential:
| Result | Meaning |
|--------|---------|
| Success | Key is valid, provider accessible |
| Invalid API key | Check key format and value |
| Connection failed | Check URL, network, firewall |
| Model not available | Key valid but model access restricted |
Test uses inexpensive models (e.g., `gpt-3.5-turbo`, `claude-3-haiku`) to minimize cost.
---
## Configuring Specific Providers
### Simple Providers (API Key Only)
For OpenAI, Anthropic, Google, Groq, Mistral, DeepSeek, xAI, OpenRouter:
1. Add credential with your API key
2. Test connection
3. Discover and register models
### Ollama (URL-Based)
1. Add credential with provider **Ollama**
2. Enter the base URL (e.g., `http://ollama:11434`)
3. Test connection
4. Discover and register models
Ollama allows localhost and private IPs since it runs locally.
### Azure OpenAI
1. Add credential with provider **Azure OpenAI**
2. Enter your API key
3. Enter your Azure endpoint in the **URL Base** field (e.g., `https://myresource.openai.azure.com`)
4. Test connection
5. Discover and register models
The URL Base field is automatically mapped to the Azure endpoint. The API version defaults to `2024-10-21` if not set via environment variable.
### OpenAI-Compatible
For custom OpenAI-compatible servers (LM Studio, vLLM, etc.):
1. Add credential with provider **OpenAI-Compatible**
2. Enter the base URL
3. Enter API key (if required)
4. Optionally configure per-service URLs
Supports separate configurations for:
- LLM (language models)
- Embedding
- STT (speech-to-text)
- TTS (text-to-speech)
### Vertex AI
Google Cloud's enterprise AI platform:
| Field | Example |
|-------|---------|
| Project ID | `my-gcp-project` |
| Location | `us-central1` |
| Credentials Path | `/path/to/service-account.json` |
---
## Migrating from Environment Variables
If you have existing API keys in environment variables (from a previous version):
1. Open **Settings → API Keys**
2. A banner appears: "Environment variables detected"
3. Click **Migrate to Database**
4. Keys are copied to the database (encrypted)
5. Original environment variables remain unchanged
### Migration Behavior
| Scenario | Action |
|----------|--------|
| Key in env only | Migrated to database |
| Key in database only | No change |
| Key in both | Database version kept (skipped) |
### After Migration
- Database credentials are used for all operations
- You can remove the API key environment variables from your docker-compose.yml
- Keep `OPEN_NOTEBOOK_ENCRYPTION_KEY` — it's still required
### Migration Banner Visibility
The migration banner only appears when:
- You have environment variables configured
- Those providers are **not** already in the database
- If all env providers are already migrated, the banner won't show
---
## Migrating from ProviderConfig (v1.1 → v1.2)
If you're upgrading from an older version that used the ProviderConfig system:
- The migration happens automatically on first startup
- Your existing configurations are converted to credentials
- Check **Settings → API Keys** to verify the migration succeeded
- If you see issues, check the API logs for migration messages
---
## Key Storage Security
### Encryption
API keys stored in the database are encrypted using Fernet (AES-128-CBC + HMAC-SHA256).
| Configuration | Behavior |
|---------------|----------|
| Encryption key set | Keys encrypted with your key |
| No encryption key set | Storing API keys in database is disabled |
### Default Credentials
| Setting | Default Value | Production Recommendation |
|---------|---------------|---------------------------|
| Password | None - auth is fully disabled until set | Set `OPEN_NOTEBOOK_PASSWORD` |
| Encryption Key | None (must be set) | Set `OPEN_NOTEBOOK_ENCRYPTION_KEY` to any secret string |
**For production deployments, always set custom credentials.**
---
## Deleting Credentials
1. Click the **Delete** button on the credential card
2. Confirm deletion
3. Credential and all its linked models are removed from the database
---
## Troubleshooting
### Credential Not Saving
| Symptom | Cause | Solution |
|---------|-------|----------|
| Save button disabled | Empty or invalid input | Enter a valid key |
| Error on save | Encryption key not set | Set `OPEN_NOTEBOOK_ENCRYPTION_KEY` in docker-compose.yml |
| Error on save | Database connection issue | Check database status |
### Test Connection Fails
| Error | Cause | Solution |
|-------|-------|----------|
| Invalid API key | Wrong key or format | Verify key from provider dashboard |
| Connection refused | Wrong URL | Check base URL format |
| Timeout | Network issue | Check firewall, proxy settings |
| 403 Forbidden | IP restriction | Whitelist your server IP |
### Migration Issues
| Problem | Solution |
|---------|----------|
| No migration banner | No env vars detected, or already migrated |
| Partial migration | Check error list, fix and retry |
| Keys not working after migration | Clear browser cache, restart services |
### Provider Shows "Not Configured"
1. Check if a credential exists for this provider (Settings → API Keys)
2. Test the credential connection
3. Verify key format matches provider requirements
4. Re-discover and register models if needed
---
## Provider-Specific Notes
### OpenAI
- Keys start with `sk-proj-` (project keys) or `sk-` (legacy)
- Requires billing enabled on account
### Anthropic
- Keys start with `sk-ant-`
- Check account has API access enabled
### Google Gemini
- Keys start with `AIzaSy`
- Free tier has rate limits
### Ollama
- No API key required
- Default URL: `http://localhost:11434` (local) or `http://ollama:11434` (Docker)
- Ensure Ollama server is running
### Azure OpenAI
- Enter your Azure endpoint in the **URL Base** field (format: `https://{resource-name}.openai.azure.com`)
- API version defaults to `2024-10-21`; override via `AZURE_OPENAI_API_VERSION` environment variable if needed
- Deployment names configured separately when registering models via the credential's Discover Models dialog
---
## Related
- **[AI Providers](../5-CONFIGURATION/ai-providers.md)** — Provider setup instructions and recommendations
- **[Security](../5-CONFIGURATION/security.md)** — Password and encryption configuration
- **[Environment Reference](../5-CONFIGURATION/environment-reference.md)** — All configuration options
docs/3-USER-GUIDE/chat-effectively.md
+554
View File
@@ -0,0 +1,554 @@
# Chat Effectively - Conversations with Your Research
Chat is your main tool for exploratory questions and back-and-forth dialogue. This guide covers how to use it effectively.
---
## Quick-Start: Your First Chat
```
1. Go to your notebook
2. Click "Chat"
3. Select which sources to include (context)
4. Type your question
5. Click "Send"
6. Read the response
7. Ask a follow-up (context stays same)
8. Repeat until satisfied
```
That's it! But doing it *well* requires understanding how context works.
---
## Context Management: The Key to Good Chat
Context controls **what the AI is allowed to see**. This is your most important control.
### The Three Levels Explained
**FULL CONTENT**
- AI sees: Complete source text
- Cost: 100 tokens per 1K tokens of source
- Best for: Detailed analysis, precise citations
- Example: "Analyze this research paper closely"
```
You set: Paper A → Full Content
AI sees: Every word of Paper A
AI can: Cite specific sentences, notice nuances
Result: Precise, detailed answers (higher cost)
```
**SUMMARY ONLY**
- AI sees: AI-generated 200-word summary (not full text)
- Cost: ~10-20% of full content cost
- Best for: Background material, reference context
- Example: "Use this for background, focus on the main paper"
```
You set: Paper B → Summary Only
AI sees: Condensed summary, key points
AI can: Reference main ideas but not details
Result: Faster, cheaper answers (loses precision)
```
**NOT IN CONTEXT**
- AI sees: Nothing
- Cost: 0 tokens
- Best for: Confidential, irrelevant, archived content
- Example: "Keep this in notebook but don't use now"
```
You set: Paper C → Not in Context
AI sees: Nothing (completely excluded)
AI can: Never reference it
Result: No cost, no privacy risk for that source
```
### Setting Context (Step by Step)
```
1. Click "Select Sources"
(Shows list of all sources in notebook)
2. For each source:
□ Checkbox: Include or exclude
Level dropdown:
├─ Full Content
├─ Summary Only
└─ Excluded
3. Check your selections
Example:
✓ Paper A (Full Content) - "Main focus"
✓ Paper B (Summary Only) - "Background"
✓ Paper C (Excluded) - "Keep private"
□ Paper D (Not included) - "Not relevant"
4. Click "Save Context"
5. Now chat uses these settings
```
### Context Strategies
**Strategy 1: Minimalist**
- Main source: Full Content
- Everything else: Excluded
- Result: Focused, cheap, precise
```
Use when:
- Analyzing one source deeply
- Budget-conscious
- Want focused answers
```
**Strategy 2: Comprehensive**
- All sources: Full Content
- Result: All context considered, expensive
```
Use when:
- Comprehensive analysis
- Unlimited budget
- Want AI to see everything
```
**Strategy 3: Tiered**
- Primary sources: Full Content
- Secondary sources: Summary Only
- Background/reference: Excluded
- Result: Balanced cost/quality
```
Use when:
- Mix of important and reference material
- Want thorough but not expensive
- Most common strategy
```
**Strategy 4: Privacy-First**
- Sensitive docs: Excluded
- Public research: Full Content
- Result: Never send confidential data
```
Use when:
- Company confidential materials
- Personal sensitive data
- Complying with data protection
```
---
## Asking Effective Questions
### Good Questions vs. Poor Questions
**Poor Question**
```
"What do you think?"
Problems:
- Too vague (about what?)
- No context (what am I analyzing?)
- Can't verify answer (citing what?)
Result: Generic, shallow answer
```
**Good Question**
```
"Based on the paper's methodology section,
what are the three main limitations the authors acknowledge?
Please cite which pages mention each one."
Strengths:
- Specific about what you want
- Clear scope (methodology section)
- Asks for citations
- Requires deep reading
Result: Precise, verifiable, useful answer
```
### Question Patterns That Work
**Factual Questions**
```
"What does the paper say about X?"
"Who are the authors?"
"What year was this published?"
Result: Simple, factual answers with citations
```
**Analysis Questions**
```
"How does this approach differ from the traditional method?"
"What are the main assumptions underlying this argument?"
"Why do you think the author chose this methodology?"
Result: Deeper thinking, comparison, critique
```
**Synthesis Questions**
```
"How do these two sources approach the problem differently?"
"What's the common theme across all three papers?"
"If we combine these approaches, what would we get?"
Result: Cross-source insights, connections
```
**Actionable Questions**
```
"What are the practical implications of this research?"
"How could we apply these findings to our situation?"
"What's the next logical research direction?"
Result: Practical, forward-looking answers
```
### The SPECIFIC Formula
Good questions have:
1. **SCOPE** - What are you analyzing?
"In this research paper..."
"Looking at these three articles..."
"Based on your experience..."
2. **SPECIFICITY** - Exactly what do you want?
"...the methodology..."
"...main findings..."
"...recommended next steps..."
3. **CONSTRAINT** - Any limits?
"...in 3 bullet points..."
"...with citations to page numbers..."
"...comparing these two approaches..."
4. **VERIFICATION** - How can you check it?
"...with specific quotes..."
"...cite your sources..."
"...link to the relevant section..."
**Example:**
```
Poor: "What about transformers?"
Good: "In this research paper on machine learning,
explain the transformer architecture in 2-3 sentences,
then cite which page describes the attention mechanism."
```
---
## Follow-Up Questions (The Real Power of Chat)
Chat's strength is dialogue. You ask, get an answer, ask more.
### Building on Responses
```
First question:
"What's the main finding?"
AI: "The study shows X [citation]"
Follow-up question:
"How does that compare to Y research?"
AI: "The key difference is Z [citation]"
Next question:
"Why do you think that difference matters?"
AI: "Because it affects A, B, C [explained]"
```
### Iterating Toward Understanding
```
Round 1: Get overview
"What's this source about?"
Round 2: Get details
"What's the most important part?"
Round 3: Compare
"How does it relate to my notes on X?"
Round 4: Apply
"What should I do with this information?"
```
### Changing Direction
```
Context stays same, but you ask new questions:
Question 1: "What's the methodology?"
Question 2: "What are the limitations?"
Question 3: "What about the ethical implications?"
Question 4: "Who else has done similar work?"
All in one conversation, reusing context.
```
### Adjusting Context Between Rounds
```
After question 3, you realize:
"I need more context from another source"
1. Click "Adjust Context"
2. Add new source or change context level
3. Your conversation history stays
4. Continue asking with new context
```
---
## Citations and Verification
Citations are how you verify that the AI's answer is accurate.
### Understanding Citations
```
AI Response with Citation:
"The paper reports a 95% accuracy rate [see page 12]"
What this means:
✓ The claim "95% accuracy rate" is from page 12
✓ You can verify by reading page 12
✓ If page 12 doesn't say that, the AI hallucinated
```
### Requesting Better Citations
```
If you get a response without citations:
Ask: "Please cite the page number for that claim"
or: "Show me where you found that information"
AI will:
- Find the citation
- Provide page numbers
- Show you the source
```
### Verification Workflow
```
1. Get answer from Chat
2. Check citation (which source? which page?)
3. Click citation link (if available)
4. See the actual text in source
5. Does it really say what AI claimed?
If YES: Great, you can use this answer
If NO: The AI hallucinated, ask for correction
```
---
## Common Chat Patterns
### Pattern 1: Deep Dive into One Source
```
1. Set context: One source (Full Content)
2. Question 1: Overview
3. Question 2: Main argument
4. Question 3: Evidence for argument
5. Question 4: Limitations
6. Question 5: Next steps
Result: Complete understanding of one source
```
### Pattern 2: Comparative Analysis
```
1. Set context: 2-3 sources (all Full Content)
2. Question 1: What does each source say about X?
3. Question 2: How do they agree?
4. Question 3: How do they disagree?
5. Question 4: Which approach is stronger?
Result: Understanding differences and trade-offs
```
### Pattern 3: Research Exploration
```
1. Set context: Many sources (mix of Full/Summary)
2. Question 1: What are the main perspectives?
3. Question 2: What's missing from these views?
4. Question 3: What questions does this raise?
5. Question 4: What should I research next?
Result: Understanding landscape and gaps
```
### Pattern 4: Problem Solving
```
1. Set context: Relevant sources (Full Content)
2. Question 1: What's the problem?
3. Question 2: What approaches exist?
4. Question 3: Pros and cons of each?
5. Question 4: Which would work best for [my situation]?
Result: Decision-making informed by research
```
---
## Optimizing for Cost
Chat uses tokens for every response. Here's how to use efficiently:
### Reduce Token Usage
**Minimize context**
```
Option A: All sources, Full Content
Cost per response: 5,000 tokens
Option B: Only relevant sources, Summary Only
Cost per response: 1,000 tokens
Savings: 80% cheaper, same conversation
```
**Shorter questions**
```
Verbose: "Could you please analyze the methodology
section of this paper and explain in detail
what the authors did?"
Concise: "Summarize the methodology in 2-3 points."
Savings: 20-30% per response
```
**Use cheaper models**
```
GPT-4o: $0.15 per 1M input tokens
GPT-4o-mini: $0.03 per 1M input tokens
Claude Sonnet: $0.90 per 1M input tokens
For chat: Mini/Haiku models are usually fine
For deep analysis: Sonnet/Opus worth the cost
```
### Budget Strategies
**Exploration budget**
- Use cheap model
- Broad context (understand landscape)
- Short questions
- Result: Low cost, good overview
**Analysis budget**
- Use powerful model
- Focused context (main source only)
- Detailed questions
- Result: Higher cost, deep insights
**Synthesis budget**
- Use powerful model for final synthesis
- Multiple sources (Full Content)
- Complex comparative questions
- Result: Expensive but valuable output
---
## Troubleshooting Chat Issues
### Poor Responses
| Problem | Cause | Solution |
|---------|-------|----------|
| Generic answers | Vague question | Be specific (see question patterns) |
| Missing context | Not enough in context | Add sources or change to Full Content |
| Incorrect info | Source not in context | Add the relevant source |
| Hallucinating | Model confused | Ask for citations, verify claims |
| Shallow analysis | Wrong model | Switch to more powerful model |
### High Costs
| Problem | Cause | Solution |
|---------|-------|----------|
| Expensive per response | Too much context | Use Summary Only or exclude sources |
| Many follow-ups | Exploratory chat | Use Ask instead for single comprehensive answer |
| Long conversations | Keeping history | Archive old chats, start fresh |
| Large sources | Full text in context | Use Summary Only for large documents |
---
## Best Practices
### Before You Chat
- [ ] Add sources you'll need
- [ ] Decide context strategy (Tiered is usually best)
- [ ] Choose model (cheaper for exploration, powerful for analysis)
- [ ] Have a question in mind
### During Chat
- [ ] Ask specific questions (use SPECIFIC formula)
- [ ] Check citations for factual claims
- [ ] Follow up on unclear points
- [ ] Adjust context if you need different sources
### After Chat
- [ ] Save good responses as notes
- [ ] Archive conversation if you're done
- [ ] Organize notes for future reference
- [ ] Use insights in other features (Ask, Transformations, Podcasts)
---
## When to Use Chat vs. Ask
**Use CHAT when:**
- You want a dialogue
- You're exploring a topic
- You'll ask multiple related questions
- You want to adjust context during conversation
- You're not sure exactly what you need
**Use ASK when:**
- You have one specific question
- You want a comprehensive answer
- You want the system to auto-search
- You want one response, not dialogue
- You want maximum tokens spent on search
---
## Summary: Chat as Conversation
Chat is fundamentally different from asking ChatGPT directly:
| Aspect | ChatGPT | Open Notebook Chat |
|--------|---------|-------------------|
| **Source control** | None (uses training) | You control which sources are visible |
| **Cost control** | Per token | Per token, but context is your choice |
| **Iteration** | Works | Works, with your sources changing dynamically |
| **Citations** | Made up often | Tied to your sources (verifiable) |
| **Privacy** | Your data to OpenAI | Your data stays local (unless you choose) |
The key insight: **Chat is retrieval-augmented generation.** AI sees only what you put in context. You control the conversation and the information flow.
That's why Chat is powerful for research. You're not just talking to an AI; you're having a conversation with your research itself.
docs/3-USER-GUIDE/citations.md
+299
View File
@@ -0,0 +1,299 @@
# Citations - Verify and Trust AI Responses
Citations connect AI responses to your source materials. This guide covers how to use and verify them.
---
## Why Citations Matter
Every AI-generated response in Open Notebook includes citations to your sources. This lets you:
- **Verify claims** - Check that AI actually read what it claims
- **Find original context** - See the full passage around a quote
- **Catch hallucinations** - Spot when AI makes things up
- **Build credibility** - Your notes have traceable sources
---
## Quick Start: Using Citations
### Reading Citations
```
AI Response:
"The study found a 95% accuracy rate [1] using the proposed method."
[1] = Click to see source
What happens when you click:
→ Opens the source document
→ Highlights the relevant section
→ You can verify the claim
```
### Requesting Better Citations
If a response lacks citations, ask:
```
"Please cite the specific page or section for that claim."
"Where in the document does it say that?"
"Can you quote the exact text?"
```
---
## How Citations Work
### Automatic Generation
When AI references your sources, citations are generated automatically:
```
1. AI analyzes your question
2. Retrieves relevant source chunks
3. Generates response with inline citations
4. Links citations to original source locations
```
### Citation Format
```
Inline format:
"The researchers concluded X [1] and Y [2]."
Reference list:
[1] Paper Title - Section 3.2
[2] Report Name - Page 15
Clickable: Each [number] links to the source
```
---
## Verifying Citations
### The Verification Workflow
```
Step 1: Read AI response
"The model achieved 95% accuracy [1]"
Step 2: Click citation [1]
→ Opens source document
→ Shows relevant passage
Step 3: Verify the claim
Does source actually say 95%?
Is context correct?
Any nuance missed?
Step 4: Trust or correct
✓ Accurate → Use the insight
✗ Wrong → Ask AI to correct
```
### What to Check
| Check | Why |
|-------|-----|
| **Exact numbers** | AI sometimes rounds or misremembers |
| **Context** | Quote might mean something different in context |
| **Attribution** | Is this the source's claim or someone they cited? |
| **Completeness** | Did AI miss important caveats? |
---
## Citations in Different Features
### Chat Citations
```
Context: Sources you selected
Citations: Reference chunks used in response
Verification: Click to see original text
Save: Citations preserved when saving as note
```
### Ask Feature Citations
```
Context: Auto-searched across all sources
Citations: Multiple sources synthesized
Verification: Each source linked separately
Quality: Often more comprehensive than Chat
```
### Transformation Citations
```
Context: Single source being transformed
Citations: Points back to original document
Verification: Compare output to source
Use: When you need structured extraction
```
---
## Saving Citations
### In Notes
When you save an AI response as a note, citations are preserved:
```
Original response:
"According to the paper [1], the method works by..."
Saved note includes:
- The text
- The citation link
- Reference to source document
```
### Exporting
Citations work in exports:
| Format | Citation Behavior |
|--------|-------------------|
| **Markdown** | Links preserved as `[text](link)` |
| **Copy/Paste** | Plain text with reference numbers |
| **PDF** | Clickable references (if supported) |
---
## Citation Quality Tips
### Get Better Citations
**Be specific in questions:**
```
Poor: "What does it say about X?"
Good: "What does page 15 say about X? Please quote directly."
```
**Request citation format:**
```
"Include page numbers for each claim."
"Cite specific sections, not just document names."
```
**Use Full Content context:**
```
Summary Only → Less precise citations
Full Content → Exact quotes possible
```
### When Citations Are Missing
| Situation | Cause | Solution |
|-----------|-------|----------|
| No citations | AI used general knowledge | Ask: "Base your answer only on my sources" |
| Vague citations | Source not in Full Content | Change context level |
| Wrong citations | AI confused sources | Ask to verify with quotes |
---
## Common Issues
### "Citation doesn't match claim"
```
Problem: AI says X, but source says Y
What happened:
- AI paraphrased incorrectly
- AI combined multiple sources confusingly
- Source was taken out of context
Solution:
1. Click citation to see original
2. Note the discrepancy
3. Ask AI: "The source says Y, not X. Please correct."
```
### "Can't find cited section"
```
Problem: Citation link doesn't show relevant text
What happened:
- Source was chunked differently than expected
- Information spread across multiple sections
- Processing missed some content
Solution:
1. Search within source for key terms
2. Ask AI for more specific location
3. Re-process source if needed
```
### "No citations at all"
```
Problem: AI response has no source references
What happened:
- Sources not in context
- Question asked for opinion/general knowledge
- Model didn't find relevant content
Solution:
1. Check context settings
2. Rephrase: "Based on my sources, what..."
3. Add more relevant sources
```
---
## Best Practices
### For Research Integrity
1. **Always verify important claims** - Don't trust AI blindly
2. **Check context** - Quotes can be misleading out of context
3. **Note limitations** - AI might miss nuance
4. **Keep source access** - Don't delete sources you cite
### For Academic Work
1. **Use Full Content** for documents you'll cite
2. **Request specific page numbers**
3. **Cross-check with original sources**
4. **Document your verification process**
### For Professional Use
1. **Verify before sharing** - Check claims clients will see
2. **Keep citation trail** - Save notes with sources linked
3. **Be transparent** - Note when insights are AI-assisted
---
## Summary
```
Citations = Your verification system
How to use:
1. Read AI response
2. Note citation markers [1], [2], etc.
3. Click to see original source
4. Verify claim matches source
5. Trust verified insights
When citations fail:
- Ask for specific quotes
- Change to Full Content
- Request page numbers
- Verify manually
Why it matters:
- AI can hallucinate
- Context can change meaning
- Trust requires verification
- Good research needs sources
```
Citations aren't just references — they're your quality control. Use them to build research you can trust.
docs/3-USER-GUIDE/creating-podcasts.md
+678
View File
@@ -0,0 +1,678 @@
# Creating Podcasts - Turn Research into Audio
Podcasts let you consume your research passively. This guide covers the complete workflow from setup to download.
---
## Quick-Start: Your First Podcast (5 Minutes)
```
1. Go to your notebook
2. Click "Generate Podcast"
3. Select sources to include
4. Choose a speaker profile (or use default)
5. Click "Generate"
6. Wait 3-10 minutes (non-blocking)
7. Download MP3 when ready
8. Done!
```
That's the minimum. Let's make it better.
---
## Step-by-Step: The Complete Workflow
### Step 1: Prepare Your Notebook
```
Before generating, make sure:
✓ You have sources added
(At least 1-2 sources)
✓ Sources have been processed
(Green "Ready" status)
✓ Notes are organized
(If you want notes included)
✓ You know your message
(What's the main story?)
Typical preparation: 5-10 minutes
```
### Step 2: Choose Content
```
Click "Generate Podcast"
You'll see:
- List of all sources in notebook
- List of all notes
Select which to include:
☑ Paper A (primary source)
☑ Paper B (supporting source)
☐ Old note (not relevant)
✓ Analysis note (important)
What to include:
- Primary sources: Always include
- Supporting sources: Usually include
- Notes: Include your analysis/insights
- Everything: Can overload podcast
Recommended: 3-5 sources per podcast
```
### Step 3: Choose Episode Profile
An episode profile defines the structure and tone.
**Option A: Use Preset Profile**
```
Open Notebook provides preset profiles:
Academic Presentation (Monologue)
├─ 1 speaker
├─ Tone: Educational
└─ Format: Expert explaining topic
Expert Interview (2-speaker)
├─ 2 speakers: Host + Expert
├─ Tone: Q&A, conversational
└─ Format: Interview with expert
Debate Format (2-speaker)
├─ 2 speakers: Pro vs. Con
├─ Tone: Discussion, disagreement
└─ Format: Debate about the topic
Panel Discussion (3-4 speaker)
├─ 3-4 speakers: Different perspectives
├─ Tone: Thoughtful discussion
└─ Format: Each brings different expertise
Solo Explanation (Monologue)
├─ 1 speaker
├─ Tone: Conversational, friendly
└─ Format: Personal explanation
```
**Pick based on your content:**
- One main idea → Academic Presentation
- You want to explain → Solo Explanation
- Two competing views → Debate Format
- Multiple perspectives → Panel Discussion
- Want to explore → Expert Interview
### Step 4: Customize Episode Profile (Optional)
If presets don't fit, customize:
```
Episode Profile
├─ Title: "AI Safety in 2026"
├─ Description: "Exploring current approaches"
├─ Length target: 20 minutes
├─ Tone: "Academic but accessible"
├─ Focus areas:
│ ├─ Main approaches to alignment
│ ├─ Pros and cons comparison
│ └─ Open questions
├─ Audience: "Researchers new to field"
└─ Format: "Debate between two perspectives"
How to set:
1. Click "Customize"
2. Edit each field
3. Click "Save Profile"
4. System uses your profile for outline generation
```
### Step 5: Create or Select Speakers
Speakers are the "voice" of your podcast.
**Option A: Use Preset Speakers**
```
Open Notebook provides preset profiles:
"Expert Alex"
- Expertise: Deep knowledge
- Personality: Rigorous, patient
- Voice Model: Selected from model registry
"Curious Sam"
- Expertise: Curious newcomer
- Personality: Asks questions
- Voice Model: Selected from model registry
"Skeptic Jordan"
- Expertise: Critical perspective
- Personality: Challenges assumptions
- Voice Model: Selected from model registry
For your first podcast: Use presets
For custom podcast: Create your own
```
**Option B: Create Custom Speakers**
```
Click "Add Speaker"
Fill in:
Name: "Dr. Research Expert"
Expertise:
"20 years in AI safety research,
deep knowledge of alignment approaches"
Personality:
"Rigorous, academic style,
explains clearly, asks good questions"
Voice Configuration:
- Voice Model: Select from model registry (e.g., OpenAI TTS, Google TTS, ElevenLabs)
- Voice: Choose from available voices for the selected model
- Per-speaker override: Each speaker can optionally use a different voice model
Credentials are automatically resolved from the model configuration.
Example:
Name: Dr. Research Expert
Expertise: AI safety alignment research
Personality: Rigorous, academic but accessible
Voice Model: ElevenLabs TTS (from registry), Voice: professional male
```
### Step 6: Generate Podcast
```
1. Review your setup:
Sources: ✓ Selected
Profile: ✓ Episode profile chosen
Speakers: ✓ Speakers configured
2. Click "Generate Podcast"
3. System begins:
- Analyzing your content
- Creating outline
- Writing dialogue
- Generating audio
- Mixing speakers
4. Status shows progress:
20% Outline generation
40% Dialogue writing
60% Audio synthesis
80% Mixing
100% Complete
Processing time:
- 5 minutes of content: 3-5 minutes
- 15 minutes of content: 5-10 minutes
- 30 minutes of content: 10-20 minutes
```
### Step 7: Review and Download
```
When complete:
Preview:
- Play audio sample
- Review transcript
- Check duration
Options:
✓ Download as MP3 - Save to computer
✓ Stream directly - Listen in browser
✓ Share link - Get shareable URL (if public)
✓ Regenerate - Try different speakers/profile
Download:
1. Click "Download as MP3"
2. Choose quality: 128kbps / 192kbps / 320kbps
3. Save file: podcast_[notebook]_[date].mp3
4. Listen!
```
---
## Understanding What Happens Behind the Scenes
### The Generation Pipeline
```
Stage 1: CONTENT ANALYSIS (1 minute)
Your sources → What's the main story?
→ Key themes?
→ Debate points?
Stage 2: OUTLINE CREATION (2-3 minutes)
Themes → Episode structure
→ Section breakdown
→ Talking points
Stage 3: DIALOGUE WRITING (2-3 minutes)
Outline → Convert to natural dialogue
→ Add speaker personalities
→ Create flow and transitions
Stage 4: AUDIO SYNTHESIS (3-5 minutes per speaker)
Script + Speaker → Text-to-speech
→ Individual audio files
→ High quality audio
Stage 5: MIXING & MASTERING (1-2 minutes)
Multiple audio → Combine speakers
→ Level audio
→ Add polish
→ Final MP3
Total: 10-20 minutes for typical podcast
```
---
## Text-to-Speech Providers
Different providers, different qualities.
### OpenAI (Recommended)
```
Voices: 5 options (Alloy, Echo, Fable, Onyx, Shimmer)
Quality: Good, natural sounding
Speed: Fast
Cost: ~$0.015 per minute
Best for: General purpose, natural speech
Example: "I have to say, the research shows..."
```
### Google TTS
```
Voices: Many options, various accents
Quality: Excellent, very natural
Speed: Fast
Cost: ~$0.004 per minute
Best for: High quality output, accents
Example: "The research demonstrates that..."
```
### ElevenLabs
```
Voices: 100+ voices, highly customizable
Quality: Exceptional, very expressive
Speed: Slower (5-10 seconds per phrase)
Cost: ~$0.10 per minute
Best for: Premium quality, emotional range
Example: [Can convey emotion and tone]
```
### Local TTS (Free)
```
Voices: Limited, basic options
Quality: Basic, robotic
Speed: Depends on hardware (slow)
Cost: Free (local processing)
Best for: Privacy, testing, offline use
Example: "The research shows..."
Privacy: Everything stays on your computer
```
### Which Provider to Choose?
```
For your first podcast: Google (quality/cost balance)
For privacy-sensitive: Local TTS (free, private)
For premium quality: ElevenLabs (best voices)
For budget: Google (cheapest quality option)
For speed: OpenAI (fast generation)
```
---
## Tips for Better Podcasts
### Choose Right Profile
```
Single source analysis → Academic Presentation
"Explaining one paper to someone new"
Comparing two approaches → Debate Format
"Pros and cons of different methods"
Multiple sources + insights → Panel Discussion
"Different experts discussing topic"
Narrative exploration → Expert Interview
"Host interviewing research expert"
Personal take → Solo Explanation
"You explaining your analysis"
```
### Create Good Speakers
```
Good Speaker:
✓ Clear expertise (know what they're talking about)
✓ Distinct personality (not generic)
✓ Good voice choice (matches personality)
✓ Realistic backstory (feels like real person)
Bad Speaker:
✗ Generic expertise ("good at research")
✗ No personality ("just reads")
✗ Mismatched voice (deep voice for young person)
✗ Contradicts personality (serious person uses casual voice)
```
### Focus Content
```
Better: Podcast on ONE specific topic
"How transformers work" (15 minutes, focused)
Worse: Podcast on everything
"All of AI 2025" (2 hours, unfocused)
Guideline:
- 5-10 minutes: One narrow topic
- 15-20 minutes: One broad topic
- 30+ minutes: Multiple related subtopics
Shorter is usually better for podcasts.
```
### Optimize Source Selection
```
Too much content:
"Here are all 20 papers"
→ Podcast becomes 2+ hours
→ Unfocused
→ Low quality
Right amount:
"Here are 3 key papers"
→ Podcast is 15-20 minutes
→ Focused
→ High quality
Rule: 3-5 sources per podcast
Remove long background papers
Keep focused on main topic
```
---
## Quality Troubleshooting
### Audio Sounds Robotic
**Problem**: TTS voice sounds unnatural
**Solutions**:
```
1. Switch provider: Try Google or ElevenLabs instead
2. Choose different voice: Some voices more natural
3. Shorter sentences: Very long sentences sound robotic
4. Adjust pacing: Ask for "natural, conversational pacing"
```
### Audio Sounds Unclear
**Problem**: Hard to understand what's being said
**Solutions**:
```
1. Re-generate with different speaker
2. Try different TTS provider
3. Use speakers with clear accents
4. Lower background noise (if any)
5. Increase speech rate (if too slow)
```
### Missing Content
**Problem**: Important information isn't in podcast
**Solutions**:
```
1. Include that source in content selection
2. Review generated outline (check before generating)
3. Regenerate with clearer profile instructions
4. Try different model (more thorough model)
```
### Speakers Don't Match
**Problem**: Speakers sound like same person
**Solutions**:
```
1. Choose different voice models from the registry for each speaker
2. Choose very different voice options
3. Increase personality differences in profile
4. Try different speaker count (2 vs 3 vs 4)
```
### Generation Failed
**Problem**: "Podcast generation failed"
**Solutions**:
```
1. Check internet connection (especially TTS)
2. Try again (might be temporary issue)
3. Use local TTS (doesn't need internet)
4. Reduce source count (less to process)
5. Contact support if persistent
```
---
## Advanced: Multiple Podcasts from Same Research
You can generate different podcasts from one notebook:
```
Podcast 1: Overview
Profile: Academic Presentation
Sources: Papers A, B, C
Speakers: One expert
Length: 15 minutes
→ Use for "What's this about?" understanding
Podcast 2: Deep Dive
Profile: Expert Interview
Sources: Paper A (Full) + B, C (Summary)
Speakers: Expert + Interviewer
Length: 30 minutes
→ Use for detailed exploration
Podcast 3: Debate
Profile: Debate Format
Sources: Papers A vs B (different approaches)
Speakers: Pro-A speaker + Pro-B speaker
Length: 20 minutes
→ Use for comparing approaches
```
Each tells the same story from different angles.
---
## Exporting and Sharing
### Download MP3
```
1. Generation complete
2. Click "Download"
3. Choose quality:
- 128 kbps: Smallest file, lower quality
- 192 kbps: Balanced (recommended)
- 320 kbps: Highest quality, largest file
4. Save to computer
5. Use in podcast app, upload to platform, etc.
```
### Export Transcript
```
1. Click "Export Transcript"
2. Get full dialogue as text
3. Useful for:
- Blog post content
- Show notes
- Searchable text version
- Accessibility
```
### Share Link
```
If podcast is public:
1. Click "Share"
2. Get shareable link
3. Others can listen/download
4. Useful for:
- Sharing with team
- Public distribution
- Embedding on website
```
### Publish to Podcast Platforms
```
If you want to distribute (future feature):
1. Download MP3
2. Upload to platform (Spotify, Apple Podcasts, etc.)
3. Add metadata (title, description, episode notes)
4. Your research becomes a published podcast!
```
---
## Best Practices
### Before Generation
- [ ] Sources are processed and ready
- [ ] You've chosen content to include
- [ ] You have a clear episode profile
- [ ] Speakers are well-defined
- [ ] Content is focused (3-5 sources max)
### During Generation
- Don't close the browser (use background processing)
- Check back in 5-15 minutes
- Review transcript when complete
- Listen to sample before downloading
### After Generation
- [ ] Download MP3 to computer
- [ ] Save in organized folder
- [ ] Add metadata (title, description, date)
- [ ] Test listening in podcast app
- [ ] Share with colleagues for feedback
---
## Use Cases
### Academic Researcher
```
Podcast: Explaining your dissertation
Speakers: You + colleague
Content: Your papers + supporting research
Use: Share with advisors, test explanations
```
### Content Creator
```
Podcast: Research-to-podcast article
Speakers: Narrator + expert
Content: Articles you've researched
Use: Transform article into podcast version
```
### Team Research
```
Podcast: Weekly research updates
Speakers: Multiple team members
Content: This week's papers
Use: Team updates, knowledge sharing
```
### Learning/Teaching
```
Podcast: Teaching material
Speakers: Teacher + inquisitive student
Content: Textbook + examples
Use: Students learn while commuting
```
---
## Cost Breakdown Example
### Generate 15-minute podcast with ElevenLabs
```
Generation (outline + dialogue):
No charge (included in service)
Text-to-speech:
2 speakers × 15 minutes = 30 minutes TTS
ElevenLabs: $0.10 per minute
Cost: 30 × $0.10 = $3.00
Processing:
Included (no additional cost)
Total: $3.00 per podcast
Cheaper options:
With Google TTS: ~$0.12
With OpenAI: ~$0.45
With Local TTS: ~$0.00
```
---
## Summary: Podcasts as Research Tool
Podcasts transform how you consume research:
```
Before: Reading papers takes time, focus
After: Listen while commuting, exercising, doing chores
Before: Can't share complex research easily
After: Share audio of your analysis
Before: Different consumption styles isolated
After: Same research, multiple formats (read/listen)
```
Podcasts aren't just for entertainment—they're a tool for making research more accessible, shareable, and consumable.
That's why they're important for Open Notebook.
docs/3-USER-GUIDE/index.md
+208
View File
@@ -0,0 +1,208 @@
# User Guide - How to Use Open Notebook
This guide covers practical, step-by-step usage of Open Notebook features. You already understand the concepts; now learn how to actually use them.
> **Prerequisite**: Review [2-CORE-CONCEPTS](../2-CORE-CONCEPTS/index.md) first to understand the mental models (notebooks, sources, notes, chat, transformations, podcasts).
---
## Start Here
### [Interface Overview](interface-overview.md)
Learn the layout before diving in. Understand the three-panel design and where everything is.
---
## Eight Core Features
### 1. [Adding Sources](adding-sources.md)
How to bring content into your notebook. Supports PDFs, web links, audio, video, text, and more.
**Quick links:**
- Upload a PDF or document
- Add a web link or article
- Transcribe audio or video
- Paste text directly
- Common mistakes + fixes
---
### 2. [Working with Notes](working-with-notes.md)
Creating, organizing, and using notes (both manual and AI-generated).
**Quick links:**
- Create a manual note
- Save AI responses as notes
- Apply transformations to generate insights
- Organize with tags and naming
- Use notes across your notebook
---
### 3. [Chat Effectively](chat-effectively.md)
Have conversations with AI about your sources. Manage context to control what AI sees.
**Quick links:**
- Start your first chat
- Select which sources go in context
- Ask effective questions
- Use follow-ups productively
- Understand citations and verify claims
---
### 4. [Creating Podcasts](creating-podcasts.md)
Convert your research into audio dialogue for passive consumption.
**Quick links:**
- Create your first podcast
- Choose or customize speakers
- Select TTS provider
- Generate and download
- Common audio quality fixes
---
### 5. [Search Effectively](search.md)
Two search modes: text-based (keyword) and vector-based (semantic). Know when to use each.
**Quick links:**
- Text search vs vector search (when to use)
- Running effective searches
- Using the Ask feature for comprehensive answers
- Saving search results as notes
- Troubleshooting poor results
---
### 6. [Transformations](transformations.md)
Batch-process sources with predefined templates. Extract the same insights from multiple documents.
**Quick links:**
- Built-in transformation templates
- Creating custom transformations
- Applying to single or multiple sources
- Managing transformation output
---
### 7. [Citations](citations.md)
Verify AI claims by tracing them back to source material. Understand the citation system.
**Quick links:**
- Reading and clicking citations
- Verifying claims against sources
- Requesting better citations
- Saving cited content as notes
---
### 8. [API Configuration](api-configuration.md)
Configure AI provider API keys directly through the Settings UI.
**Quick links:**
- Add API keys without editing files
- Test provider connections
- Migrate from environment variables
- Manage Azure and OpenAI-compatible providers
- Understand key storage and encryption
---
## Which Feature for Which Task?
```
Task: "I want to explore a topic with follow-ups"
→ Use: Chat (add sources, select context, have conversation)
Task: "I want one comprehensive answer"
→ Use: Search / Ask (system finds relevant content)
Task: "I want to extract the same info from many sources"
→ Use: Transformations (define template, apply to all)
Task: "I want summaries of all my sources"
→ Use: Transformations (with built-in summary template)
Task: "I want to share my research in audio form"
→ Use: Podcasts (create speakers, generate episode)
Task: "I want to find that quote I remember"
→ Use: Search / Text Search (keyword matching)
Task: "I'm exploring a concept without knowing exact words"
→ Use: Search / Vector Search (semantic similarity)
Task: "I need to add or change my AI provider API keys"
→ Use: Settings / API Keys (configure providers without editing files)
```
---
## Quick-Start Checklist: First 15 Minutes
**Step 1: Create a Notebook (1 min)**
- Name: Something descriptive ("Q1 Market Research", "AI Safety Papers", etc.)
- Description: 1-2 sentences about what you're researching
- This is your research container
**Step 2: Add Your First Source (3 min)**
- Pick one: PDF, web link, or text
- Follow [Adding Sources](adding-sources.md)
- Wait for processing (usually 30-60 seconds)
**Step 3: Chat About It (3 min)**
- Go to Chat
- Select your source (set context to "Full Content")
- Ask a simple question: "What are the main points?"
- See AI respond with citations
**Step 4: Save Insight as Note (2 min)**
- Good response? Click "Save as Note"
- Name it something useful ("Main points from source X")
- Now you have a captured insight
**Step 5: Explore More (6 min)**
- Add another source
- Chat about both together
- Ask a question that compares them
- Follow up with clarifying questions
**Done!** You've used the core workflow: notebook → sources → chat → notes
---
## Common Mistakes to Avoid
| Mistake | Problem | Fix |
|---------|---------|-----|
| Adding everything to one notebook | No isolation between projects | Create separate notebooks for different topics |
| Expecting AI to know your context | Questions get generic answers | Describe your research focus in chat context |
| Forgetting to cite sources | You can't verify claims | Click citations to check source chunks |
| Using Chat for one-time questions | Slower than Ask | Use Ask for comprehensive Q&A, Chat for exploration |
| Adding huge PDFs without chunking | Slow processing, poor search | Break into multiple smaller sources if possible |
| Using same context for all chats | Expensive, unfocused | Adjust context level for each chat |
| Ignoring vector search | Only finding exact keywords | Use vector search to explore conceptually |
---
## Next Steps
1. **Follow each guide** in order (sources → notes → chat → podcasts → search)
2. **Create your first notebook** with real content
3. **Practice each feature** with your own research
4. **Return to CORE-CONCEPTS** if you need to understand the "why"
---
## Getting Help
- **Feature not working?** → Check the feature's guide (look for "Troubleshooting" section)
- **Error message?** → Check [6-TROUBLESHOOTING](../6-TROUBLESHOOTING/index.md)
- **Understanding how something works?** → Check [2-CORE-CONCEPTS](../2-CORE-CONCEPTS/index.md)
- **Setting up for the first time?** → Go back to [1-INSTALLATION](../1-INSTALLATION/index.md)
- **For developers** → See [7-DEVELOPMENT](../7-DEVELOPMENT/index.md)
---
**Ready to start?** Pick the guide for what you want to do first!
docs/3-USER-GUIDE/interface-overview.md
+377
View File
@@ -0,0 +1,377 @@
# Interface Overview - Finding Your Way Around
Open Notebook uses a clean three-panel layout. This guide shows you where everything is.
---
## The Main Layout
```
┌─────────────────────────────────────────────────────────────┐
│ [Logo] Notebooks Search Podcasts Models Settings │
├──────────────┬──────────────┬───────────────────────────────┤
│ │ │ │
│ SOURCES │ NOTES │ CHAT │
│ │ │ │
│ Your docs │ Your │ Talk to AI about │
│ PDFs, URLs │ insights │ your sources │
│ Videos │ summaries │ │
│ │ │ │
│ [+Add] │ [+Write] │ [Type here...] │
│ │ │ │
└──────────────┴──────────────┴───────────────────────────────┘
```
---
## Navigation Bar
The top navigation takes you to main sections:
| Icon | Page | What It Does |
|------|------|--------------|
| **Notebooks** | Main workspace | Your research projects |
| **Search** | Ask & Search | Query across all notebooks |
| **Podcasts** | Audio generation | Manage podcast profiles |
| **Models** | AI configuration | Set up providers and models |
| **Settings** | Preferences | App configuration |
---
## Left Panel: Sources
Your research materials live here.
### What You'll See
```
┌─────────────────────────┐
│ Sources (5) │
│ [+ Add Source] │
├─────────────────────────┤
│ ┌─────────────────┐ │
│ │ 📄 Paper.pdf │ │
│ │ 🟢 Full Content │ │
│ │ [⋮ Menu] │ │
│ └─────────────────┘ │
│ │
│ ┌─────────────────┐ │
│ │ 🔗 Article URL │ │
│ │ 🟡 Summary Only │ │
│ │ [⋮ Menu] │ │
│ └─────────────────┘ │
└─────────────────────────┘
```
### Source Card Elements
- **Icon** - File type (PDF, URL, video, etc.)
- **Title** - Document name
- **Context indicator** - What AI can see:
- 🟢 Full Content
- 🟡 Summary Only
- ⛔ Not in Context
- **Menu (⋮)** - Edit, transform, delete
### Add Source Button
Click to add:
- File upload (PDF, DOCX, etc.)
- Web URL
- YouTube video
- Plain text
---
## Middle Panel: Notes
Your insights and AI-generated content.
### What You'll See
```
┌─────────────────────────┐
│ Notes (3) │
│ [+ Write Note] │
├─────────────────────────┤
│ ┌─────────────────┐ │
│ │ 📝 My Analysis │ │
│ │ Manual note │ │
│ │ Jan 3, 2026 │ │
│ └─────────────────┘ │
│ │
│ ┌─────────────────┐ │
│ │ 🤖 Summary │ │
│ │ From transform │ │
│ │ Jan 2, 2026 │ │
│ └─────────────────┘ │
└─────────────────────────┘
```
### Note Card Elements
- **Icon** - Note type (manual 📝 or AI 🤖)
- **Title** - Note name
- **Origin** - How it was created
- **Date** - When created
### Write Note Button
Click to:
- Create manual note
- Add your own insights
- Markdown supported
---
## Right Panel: Chat
Your AI conversation space.
### What You'll See
```
┌───────────────────────────────┐
│ Chat │
│ Session: Research Discussion │
│ [+ New Session] [Sessions ▼] │
├───────────────────────────────┤
│ │
│ You: What's the main │
│ finding? │
│ │
│ AI: Based on the paper [1], │
│ the main finding is... │
│ [Save as Note] │
│ │
│ You: Tell me more about │
│ the methodology. │
│ │
├───────────────────────────────┤
│ Context: 3 sources (12K tok) │
├───────────────────────────────┤
│ [Type your message...] [↑] │
└───────────────────────────────┘
```
### Chat Elements
- **Session selector** - Switch between conversations
- **Message history** - Your conversation
- **Save as Note** - Keep good responses
- **Context indicator** - What AI can see
- **Input field** - Type your questions
---
## Context Indicators
These show what AI can access:
### Token Counter
```
Context: 3 sources (12,450 tokens)
↑ ↑
Sources Approximate cost indicator
included
```
### Per-Source Indicators
| Indicator | Meaning | AI Access |
|-----------|---------|-----------|
| 🟢 Full Content | Complete text | Everything |
| 🟡 Summary Only | AI summary | Key points only |
| ⛔ Not in Context | Excluded | Nothing |
Click any source to change its context level.
---
## Podcasts Tab
Inside a notebook, switch to Podcasts:
```
┌───────────────────────────────┐
│ [Chat] [Podcasts] │
├───────────────────────────────┤
│ Episode Profile: [Select ▼] │
│ │
│ Speakers: │
│ ├─ Host: Alex (voice model) │
│ └─ Guest: Sam (voice model) │
│ │
│ Include: │
│ ☑ Paper.pdf │
│ ☑ My Analysis (note) │
│ ☐ Background article │
│ │
│ [Generate Podcast] │
└───────────────────────────────┘
```
---
## Settings Page
Access via navigation bar → Settings:
### Key Sections
| Section | What It Controls |
|---------|------------------|
| **Processing** | Document and URL extraction engines |
| **Embedding** | Auto-embed settings |
| **Files** | Auto-delete uploads after processing |
| **YouTube** | Preferred transcript languages |
---
## Models Page
Configure AI providers:
```
┌───────────────────────────────────────┐
│ Models │
├───────────────────────────────────────┤
│ Language Models │
│ ┌─────────────────────────────────┐ │
│ │ GPT-4o (OpenAI) [Edit] │ │
│ │ Claude Sonnet (Anthropic) │ │
│ │ Llama 3.3 (Ollama) [⭐] │ │
│ └─────────────────────────────────┘ │
│ [+ Add Model] │
│ │
│ Embedding Models │
│ ┌─────────────────────────────────┐ │
│ │ text-embedding-3-small [⭐] │ │
│ └─────────────────────────────────┘ │
│ │
│ Text-to-Speech │
│ ┌─────────────────────────────────┐ │
│ │ OpenAI TTS [⭐] │ │
│ │ Google TTS │ │
│ └─────────────────────────────────┘ │
└───────────────────────────────────────┘
```
- **⭐** = Default model for that category
- **[Edit]** = Modify configuration
- **[+ Add]** = Add new model
---
## Search Page
Query across all notebooks:
```
┌───────────────────────────────────────┐
│ Search │
├───────────────────────────────────────┤
│ [What are you looking for? ] [🔍] │
│ │
│ Search type: [Text ▼] [Vector ▼] │
│ Search in: [Sources] [Notes] │
├───────────────────────────────────────┤
│ Results (15) │
│ │
│ 📄 Paper.pdf - Notebook: Research │
│ "...the transformer model..." │
│ │
│ 📝 My Analysis - Notebook: Research │
│ "...key findings include..." │
└───────────────────────────────────────┘
```
---
## Common Actions
### Create a Notebook
```
Notebooks page → [+ New Notebook] → Enter name → Create
```
### Add a Source
```
Inside notebook → [+ Add Source] → Choose type → Upload/paste → Wait for processing
```
### Ask a Question
```
Inside notebook → Chat panel → Type question → Enter → Read response
```
### Save AI Response
```
Get good response → Click [Save as Note] → Edit title → Save
```
### Change Context Level
```
Click source → Context dropdown → Select level → Changes apply immediately
```
### Generate Podcast
```
Podcasts tab → Select profile → Choose sources → [Generate] → Wait → Download
```
---
## Keyboard Shortcuts
| Key | Action |
|-----|--------|
| `Enter` | Send chat message |
| `Shift + Enter` | New line in chat |
| `Escape` | Close dialogs |
| `Ctrl/Cmd + F` | Browser find |
---
## Mobile View
On smaller screens, the three-panel layout stacks vertically:
```
┌─────────────────┐
│ SOURCES │
│ (tap to expand)
├─────────────────┤
│ NOTES │
│ (tap to expand)
├─────────────────┤
│ CHAT │
│ (always visible)
└─────────────────┘
```
- Panels collapse to save space
- Tap headers to expand/collapse
- Chat remains accessible
- Full functionality preserved
---
## Tips for Efficient Navigation
1. **Use keyboard** - Enter sends messages, Escape closes dialogs
2. **Context first** - Set source context before chatting
3. **Sessions** - Create new sessions for different topics
4. **Search globally** - Use Search page to find across all notebooks
5. **Models page** - Bookmark your preferred models
---
Now you know where everything is. Start with [Adding Sources](adding-sources.md) to begin your research!
docs/3-USER-GUIDE/search.md
+475
View File
@@ -0,0 +1,475 @@
# Search Effectively - Finding What You Need
Search is your gateway into your research. This guide covers two search modes and when to use each.
---
## Quick-Start: Find Something
### Simple Search
```
1. Go to your notebook
2. Type in search box
3. See results (both sources and notes)
4. Click result to view source/note
5. Done!
That works for basic searches.
But you can do much better...
```
---
## Two Search Modes Explained
Open Notebook has two fundamentally different search approaches.
### Search Type 1: TEXT SEARCH (Keyword Matching)
**How it works:**
- You search for words: "transformer"
- System finds chunks containing "transformer"
- Ranked by relevance: frequency, position, context
**Speed:** Very fast (instant)
**When to use:**
- You remember exact words or phrases
- You're looking for specific terms
- You want precise keyword matches
- You need exact quotes
**Example:**
```
Search: "attention mechanism"
Results:
1. "The attention mechanism allows..." (perfect match)
2. "Attention and other mechanisms..." (partial match)
3. "How mechanisms work in attention..." (includes words separately)
All contain "attention" AND "mechanism"
Ranked by how close together they are
```
**What it finds:**
- Exact phrases: "transformer model"
- Individual words: transformer OR model (too broad)
- Names: "Vaswani et al."
- Numbers: "1994", "GPT-4"
- Technical terms: "LSTM", "convolution"
**What it doesn't find:**
- Similar words: searching "attention" won't find "focus"
- Synonyms: searching "large" won't find "big"
- Concepts: searching "similarity" won't find "likeness"
---
### Search Type 2: VECTOR SEARCH (Semantic/Concept Matching)
**How it works:**
- Your search converted to embedding (vector)
- All chunks converted to embeddings
- System finds most similar embeddings
- Ranked by semantic similarity
**Speed:** A bit slower (1-2 seconds)
**When to use:**
- You're exploring a concept
- You don't know exact words
- You want semantically similar content
- You're discovering, not searching
**Example:**
```
Search: "What's the mechanism for understanding in models?"
(Notice: No chunk likely says exactly that)
Results:
1. "Mechanistic interpretability allows understanding..." (semantic match)
2. "Feature attribution reveals how models work..." (conceptually similar)
3. "Attention visualization shows model decisions..." (same topic)
None contain your exact words
But all are semantically related
```
**What it finds:**
- Similar concepts: "understanding" + "interpretation" + "explainability" (all related)
- Paraphrases: "big" and "large" (same meaning)
- Related ideas: "safety" relates to "alignment" (connected concepts)
- Analogies: content about biological learning when searching "learning"
**What it doesn't find:**
- Exact keywords: if you search a rare word, vector search might miss it
- Specific numbers: "1994" vs "1993" are semantically different
- Technical jargon: "LSTM" and "RNN" are different even if related
---
## Decision: Text Search vs. Vector Search?
```
Question: "Do I remember the exact words?"
→ YES: Use TEXT SEARCH
Example: "I remember the paper said 'attention is all you need'"
→ NO: Use VECTOR SEARCH
Example: "I'm looking for content about how models process information"
→ UNSURE: Try TEXT SEARCH first (faster)
If no results, try VECTOR SEARCH
Text search: "I know what I'm looking for"
Vector search: "I'm exploring an idea"
```
---
## Step-by-Step: Using Each Search
### Text Search
```
1. Go to search box
2. Type your keywords: "transformer", "attention", "2017"
3. Press Enter
4. Results appear (usually instant)
5. Click result to see context
Results show:
- Which source contains it
- How many times it appears
- Relevance score
- Preview of surrounding text
```
### Vector Search
```
1. Go to search box
2. Type your concept: "How do models understand language?"
3. Choose "Vector Search" from dropdown
4. Press Enter
5. Results appear (1-2 seconds)
6. Click result to see context
Results show:
- Semantically related chunks
- Similarity score (higher = more related)
- Preview of surrounding text
- Different sources mixed together
```
---
## The Ask Feature (Automated Search)
Ask is different from simple search. It automatically searches, synthesizes, and answers.
### How Ask Works
```
Stage 1: QUESTION UNDERSTANDING
"Compare the approaches in my papers"
→ System: "This asks for comparison"
Stage 2: SEARCH STRATEGY
→ System: "I should search for each approach separately"
Stage 3: PARALLEL SEARCHES
→ Search 1: "Approach in paper A"
→ Search 2: "Approach in paper B"
(Multiple searches happen at once)
Stage 4: ANALYSIS & SYNTHESIS
→ Per-result analysis: "Based on paper A, the approach is..."
→ Per-result analysis: "Based on paper B, the approach is..."
→ Final synthesis: "Comparing A and B: A differs from B in..."
Result: Comprehensive answer, not just search results
```
### When to Use Ask vs. Simple Search
| Task | Use | Why |
|------|-----|-----|
| "Find the quote about X" | **TEXT SEARCH** | Need exact words |
| "What does source A say about X?" | **TEXT SEARCH** | Direct, fast answer |
| "Find content about X" | **VECTOR SEARCH** | Semantic discovery |
| "Compare A and B" | **ASK** | Comprehensive synthesis |
| "What's the big picture?" | **ASK** | Full analysis needed |
| "How do these sources relate?" | **ASK** | Cross-source synthesis |
| "I remember something about X" | **TEXT SEARCH** | Recall memory |
| "I'm exploring the topic of X" | **VECTOR SEARCH** | Discovery mode |
---
## Advanced Search Strategies
### Strategy 1: Simple Search with Follow-Up
```
1. Text search: "attention mechanism"
Results: 50 matches
2. Too many. Follow up with vector search:
"Why is attention useful?" (concept search)
Results: Most relevant papers/notes
3. Better results with less noise
```
### Strategy 2: Ask for Comprehensive, Then Search for Details
```
1. Ask: "What are the main approaches to X?"
Result: Comprehensive answer about A, B, C
2. Use that to identify specific sources
3. Text search in those specific sources:
"Why did they choose method X?"
Result: Detailed information
```
### Strategy 3: Vector Search for Discovery, Text for Verification
```
1. Vector search: "How do transformers generalize?"
Results: Related conceptual papers
2. Skim to understand landscape
3. Text search in promising sources:
"generalization", "extrapolation", "transfer"
Results: Specific passages to read carefully
```
### Strategy 4: Combine Search with Chat
```
1. Vector search: "What's new in AI 2026?"
Results: Latest papers
2. Go to Chat
3. Add those papers to context
4. Ask detailed follow-up questions
5. Get deep analysis of results
```
---
## Search Quality Issues & Fixes
### Getting No Results
| Problem | Cause | Solution |
|---------|-------|----------|
| Text search: no results | Word doesn't appear | Try vector search instead |
| Vector search: no results | Concept not in content | Try broader search term |
| Both empty | Content not in notebook | Add sources to notebook |
| | Sources not processed | Wait for processing to complete |
### Getting Too Many Results
| Problem | Cause | Solution |
|---------|-------|----------|
| 1000+ results | Search too broad | Be more specific |
| | All sources | Filter by source |
| | Keyword matches rare words | Use vector search instead |
### Getting Wrong Results
| Problem | Cause | Solution |
|---------|-------|----------|
| Results irrelevant | Search term has multiple meanings | Provide more context |
| | Using text search for concepts | Try vector search |
| Different meaning | Homonym (word means multiple things) | Add context (e.g., "attention mechanism") |
### Getting Low Quality Results
| Problem | Cause | Solution |
|---------|-------|----------|
| Results don't match intent | Vague search term | Be specific ("Who invented X?" vs "X") |
| | Concept not well-represented | Add more sources on that topic |
| | Vector embedding not trained on domain | Use text search as fallback |
---
## Tips for Better Searches
### For Text Search
1. **Be specific** — "attention mechanism" not just "attention"
2. **Use exact phrases** — Put quotes around: "attention is all you need"
3. **Include context** — "LSTM vs attention" not just "attention"
4. **Use technical terms** — These are usually more precise
5. **Try synonyms** — If first search fails, try related terms
### For Vector Search
1. **Ask a question** — "What's the best way to X?" is better than "best way"
2. **Use natural language** — Explain what you're looking for
3. **Be specific about intent** — "Compare X and Y" not "X and Y"
4. **Include context** — "In machine learning, how..." vs just "how..."
5. **Think conceptually** — What idea are you exploring?
### General Tips
1. **Start broad, then narrow** — "AI papers" → "transformers" → "attention mechanism"
2. **Try both search types** — Each finds different things
3. **Use Ask for complex questions** — Don't just search
4. **Save good results as notes** — Create knowledge base
5. **Filter by source if needed** — "Search in Paper A only"
---
## Search Examples
### Example 1: Finding a Specific Fact
**Goal:** "Find the date the transformer was introduced"
```
Step 1: Text search
"transformer 2017" (or year you remember)
If that works: Done!
If no results: Try
"attention is all you need" (famous paper title)
Check result for exact date
```
### Example 2: Exploring a Concept
**Goal:** "Find content about alignment interpretability"
```
Step 1: Vector search
"How do we make AI interpretable?"
Results: Papers on interpretability, transparency, alignment
Step 2: Review results
See which papers are most relevant
Step 3: Deep dive
Go to Chat, add top 2-3 papers
Ask detailed questions about alignment
```
### Example 3: Comprehensive Answer
**Goal:** "How do different approaches to AI safety compare?"
```
Step 1: Ask
"Compare the main approaches to AI safety in my sources"
Result: Comprehensive analysis comparing approaches
Step 2: Identify sources
From answer, see which papers were most relevant
Step 3: Deep dive
Text search in those papers:
"limitations", "critiques", "open problems"
Step 4: Save as notes
Create comparison note from Ask result
```
### Example 4: Finding Pattern
**Goal:** "Find all papers mentioning transformers"
```
Step 1: Text search
"transformer"
Results: All papers mentioning "transformer"
Step 2: Vector search
"neural network architecture for sequence processing"
Results: Papers that don't say "transformer" but discuss similar concept
Step 3: Combine
Union of text + vector results shows full landscape
Step 4: Analyze
Go to Chat with all results
Ask: "What's common across all these?"
```
---
## Search in the Workflow
How search fits with other features:
```
SOURCES
↓
SEARCH (find what matters)
├─ Text search (precise)
├─ Vector search (exploration)
└─ Ask (comprehensive)
↓
CHAT (explore with follow-ups)
↓
TRANSFORMATIONS (batch extract)
↓
NOTES (save insights)
```
### Workflow Example
```
1. Add 10 papers to notebook
2. Search: "What's the state of the art?"
(Vector search explores landscape)
3. Ask: "Compare these 3 approaches"
(Comprehensive synthesis)
4. Chat: Deep questions about winner
(Follow-up exploration)
5. Save best insights as notes
(Knowledge capture)
6. Transform remaining papers
(Batch extraction for later)
7. Create podcast from notes + sources
(Share findings)
```
---
## Summary: Know Your Search
**TEXT SEARCH** — "I know what I'm looking for"
- Fast, precise, keyword-based
- Use when you remember exact words/phrases
- Best for: Finding specific facts, quotes, technical terms
- Speed: Instant
**VECTOR SEARCH** — "I'm exploring an idea"
- Slow-ish, concept-based, semantic
- Use when you're discovering connections
- Best for: Concept exploration, related ideas, synonyms
- Speed: 1-2 seconds
**ASK** — "I want a comprehensive answer"
- Auto-searches, auto-analyzes, synthesizes
- Use for complex questions needing multiple sources
- Best for: Comparisons, big-picture questions, synthesis
- Speed: 10-30 seconds
Pick the right tool for your search goal, and you'll find what you need faster.
docs/3-USER-GUIDE/transformations.md
+402
View File
@@ -0,0 +1,402 @@
# Transformations - Batch Processing Your Sources
Transformations apply the same analysis to multiple sources at once. Instead of asking the same question repeatedly, define a template and run it across your content.
---
## When to Use Transformations
| Use Transformations When | Use Chat Instead When |
|-------------------------|----------------------|
| Same analysis on many sources | One-off questions |
| Consistent output format needed | Exploratory conversation |
| Batch processing | Follow-up questions needed |
| Creating structured notes | Context changes between questions |
**Example**: You have 10 papers and want a summary of each. Transformation does it in one operation.
---
## Quick Start: Your First Transformation
```
1. Go to your notebook
2. Click "Transformations" in navigation
3. Select a built-in template (e.g., "Summary")
4. Select sources to transform
5. Click "Apply"
6. Wait for processing
7. New notes appear automatically
```
---
## Built-in Transformations
Open Notebook includes ready-to-use templates:
### Summary
```
What it does: Creates a 200-300 word overview
Output: Key points, main arguments, conclusions
Best for: Quick reference, getting the gist
```
### Key Concepts
```
What it does: Extracts main ideas and terminology
Output: List of concepts with explanations
Best for: Learning new topics, building vocabulary
```
### Methodology
```
What it does: Extracts research approach
Output: How the study was conducted
Best for: Academic papers, research review
```
### Takeaways
```
What it does: Extracts actionable insights
Output: What you should do with this information
Best for: Business documents, practical guides
```
### Questions
```
What it does: Generates questions the source raises
Output: Open questions, gaps, follow-up research
Best for: Literature review, research planning
```
---
## Creating Custom Transformations
### Step-by-Step
```
1. Go to "Transformations" page
2. Click "Create New"
3. Enter a name: "Academic Paper Analysis"
4. Write your prompt template:
"Analyze this academic paper and extract:
1. **Research Question**: What problem does this address?
2. **Hypothesis**: What did they predict?
3. **Methodology**: How did they test it?
4. **Key Findings**: What did they discover? (numbered list)
5. **Limitations**: What caveats do the authors mention?
6. **Future Work**: What do they suggest next?
Be specific and cite page numbers where possible."
5. Click "Save"
6. Your transformation appears in the list
```
### Prompt Template Tips
**Be specific about format:**
```
Good: "List 5 key points as bullet points"
Bad: "What are the key points?"
```
**Request structure:**
```
Good: "Create sections for: Summary, Methods, Results"
Bad: "Tell me about this paper"
```
**Ask for citations:**
```
Good: "Cite page numbers for each claim"
Bad: (no citation request)
```
**Set length expectations:**
```
Good: "In 200-300 words, summarize..."
Bad: "Summarize this"
```
---
## Applying Transformations
### To a Single Source
```
1. In Sources panel, click source menu (⋮)
2. Select "Transform"
3. Choose transformation template
4. Click "Apply"
5. Note appears when done
```
### To Multiple Sources (Batch)
```
1. Go to Transformations page
2. Select your template
3. Check multiple sources
4. Click "Apply to Selected"
5. Processing runs in parallel
6. One note per source created
```
### Processing Time
| Sources | Typical Time |
|---------|--------------|
| 1 source | 30 seconds - 1 minute |
| 5 sources | 2-3 minutes |
| 10 sources | 4-5 minutes |
| 20+ sources | 8-10 minutes |
Processing runs in background. You can continue working.
---
## Transformation Examples
### Literature Review Template
```
Name: Literature Review Entry
Prompt:
"For this research paper, create a literature review entry:
**Citation**: [Author(s), Year, Title, Journal]
**Research Question**: What problem is addressed?
**Methodology**: What approach was used?
**Sample**: What population/data was studied?
**Key Findings**:
1. [Finding with page citation]
2. [Finding with page citation]
3. [Finding with page citation]
**Strengths**: What did this study do well?
**Limitations**: What are the gaps?
**Relevance**: How does this connect to my research?
Keep each section to 2-3 sentences."
```
### Meeting Notes Template
```
Name: Meeting Summary
Prompt:
"From this meeting transcript, extract:
**Attendees**: Who was present
**Date/Time**: When it occurred
**Key Decisions**: What was decided (numbered)
**Action Items**:
- [ ] Task (Owner, Due Date)
**Open Questions**: Unresolved issues
**Next Steps**: What happens next
Format as clear, scannable notes."
```
### Competitor Analysis Template
```
Name: Competitor Analysis
Prompt:
"Analyze this company/product document:
**Company**: Name and overview
**Products/Services**: What they offer
**Target Market**: Who they serve
**Pricing**: If available
**Strengths**: Competitive advantages
**Weaknesses**: Gaps or limitations
**Opportunities**: How we compare
**Threats**: What they do better
Be objective and cite specific details."
```
### Technical Documentation Template
```
Name: API Documentation Summary
Prompt:
"Extract from this technical document:
**Overview**: What does this do? (1-2 sentences)
**Authentication**: How to authenticate
**Key Endpoints**:
- Endpoint 1: [method] [path] - [purpose]
- Endpoint 2: ...
**Common Parameters**: Frequently used params
**Rate Limits**: If mentioned
**Error Codes**: Key error responses
**Example Usage**: Simple code example if possible
Keep technical but concise."
```
---
## Managing Transformations
### Edit a Transformation
```
1. Go to Transformations page
2. Find your template
3. Click "Edit"
4. Modify the prompt
5. Click "Save"
```
### Delete a Transformation
```
1. Go to Transformations page
2. Find the template
3. Click "Delete"
4. Confirm
```
### Reorder/Organize
Built-in transformations appear first, then custom ones alphabetically.
---
## Transformation Output
### Where Results Go
- Each source produces one note
- Notes appear in your notebook's Notes panel
- Notes are tagged with transformation name
- Original source is linked
### Note Naming
```
Default: "[Transformation Name] - [Source Title]"
Example: "Summary - Research Paper 2025.pdf"
```
### Editing Output
```
1. Click the generated note
2. Click "Edit"
3. Refine the content
4. Save
```
---
## Best Practices
### Template Design
1. **Start specific** - Vague prompts give vague results
2. **Use formatting** - Headings, bullets, numbered lists
3. **Request citations** - Make results verifiable
4. **Set length** - Prevent overly long or short output
5. **Test first** - Run on one source before batch
### Source Selection
1. **Similar content** - Same transformation on similar sources
2. **Reasonable size** - Very long sources may need splitting
3. **Processed status** - Ensure sources are fully processed
### Quality Control
1. **Review samples** - Check first few outputs before trusting batch
2. **Edit as needed** - Transformations are starting points
3. **Iterate prompts** - Refine based on results
---
## Common Issues
### Generic Output
**Problem**: Results are too vague
**Solution**: Make prompt more specific, add format requirements
### Missing Information
**Problem**: Key details not extracted
**Solution**: Explicitly ask for what you need in prompt
### Inconsistent Format
**Problem**: Each note looks different
**Solution**: Add clear formatting instructions to prompt
### Too Long/Short
**Problem**: Output doesn't match expectations
**Solution**: Specify word count or section lengths
### Processing Fails
**Problem**: Transformation doesn't complete
**Solution**:
- Check source is processed
- Try shorter/simpler prompt
- Process sources individually
---
## Transformations vs. Chat vs. Ask
| Feature | Transformations | Chat | Ask |
|---------|----------------|------|-----|
| **Input** | Predefined template | Your questions | Your question |
| **Scope** | One source at a time | Selected sources | Auto-searched |
| **Output** | Structured note | Conversation | Comprehensive answer |
| **Best for** | Batch processing | Exploration | One-shot answers |
| **Follow-up** | Run again | Ask more | New query |
---
## Summary
```
Transformations = Batch AI Processing
How to use:
1. Define template (or use built-in)
2. Select sources
3. Apply transformation
4. Get structured notes
When to use:
- Same analysis on many sources
- Consistent output needed
- Building structured knowledge base
- Saving time on repetitive tasks
Tips:
- Be specific in prompts
- Request formatting
- Test before batch
- Edit output as needed
```
Transformations turn repetitive analysis into one-click operations. Define once, apply many times.
docs/3-USER-GUIDE/working-with-notes.md
+581
View File
@@ -0,0 +1,581 @@
# Working with Notes - Capturing and Organizing Insights
Notes are your processed knowledge. This guide covers how to create, organize, and use them effectively.
---
## What Are Notes?
Notes are your **research output** — the insights you capture from analyzing sources. They can be:
- **Manual** — You write them yourself
- **AI-Generated** — From Chat responses, Ask results, or Transformations
- **Hybrid** — AI insight + your edits and additions
Unlike sources (which never change), notes are mutable — you edit, refine, and organize them.
---
## Quick-Start: Create Your First Note
### Method 1: Manual Note (Write Yourself)
```
1. In your notebook, go to "Notes" section
2. Click "Create New Note"
3. Give it a title: "Key insights from source X"
4. Write your content (markdown supported)
5. Click "Save"
6. Done! Note appears in your notebook
```
### Method 2: Save from Chat
```
1. Have a Chat conversation
2. Get a good response from AI
3. Click "Save as Note" button under response
4. Give the note a title
5. Add any additional context
6. Click "Save"
7. Done! Note appears in your notebook
```
### Method 3: Apply Transformation
```
1. Go to "Transformations"
2. Select a template (or create custom)
3. Click "Apply to sources"
4. Select which sources to transform
5. Wait for processing
6. New notes automatically appear
7. Done! Each source produces one note
```
---
## Creating Manual Notes
### Basic Structure
```
Title: "What you're capturing"
(Make it descriptive)
Content:
- Main points
- Your analysis
- Questions raised
- Next steps
Metadata:
- Tags: How to categorize
- Related sources: Which documents influenced this
- Date: Auto-added when created
```
### Markdown Support
You can format notes with markdown:
```markdown
# Heading
## Subheading
### Sub-subheading
**Bold text** for emphasis
*Italic text* for secondary emphasis
- Bullet lists
- Like this
1. Numbered lists
2. Like this
> Quotes and important callouts
[Links work](https://example.com)
```
### Example Note Structure
```markdown
# Key Findings from "AI Safety Paper 2025"
## Main Argument
The paper argues that X approach is better than Y because...
## Methodology
The authors use [methodology] to test this hypothesis.
## Key Results
- Result 1: [specific finding with citation]
- Result 2: [specific finding with citation]
- Result 3: [specific finding with citation]
## Gaps & Limitations
1. The paper assumes X, which might not hold in Y scenario
2. Limited to Z population/domain
3. Future work needed on A, B, C
## My Thoughts
- This connects to previous research on...
- Potential application in...
## Next Steps
- [ ] Read the referenced paper on X
- [ ] Find similar studies on Y
- [ ] Discuss implications with team
```
---
## AI-Generated Notes: Three Sources
### 1. Save from Chat
```
Workflow:
Chat → Good response → "Save as Note"
→ Edit if needed → Save
When to use:
- AI response answers your question well
- You want to keep the answer for reference
- You're building a knowledge base from conversations
Quality:
- Quality = quality of your Chat question
- Better context = better responses = better notes
- Ask specific questions for useful notes
```
### 2. Save from Ask
```
Workflow:
Ask → Comprehensive answer → "Save as Note"
→ Edit if needed → Save
When to use:
- You need a one-time comprehensive answer
- You want to save the synthesized result
- Building a knowledge base of comprehensive answers
Quality:
- System automatically found relevant sources
- Results already have citations
- Often higher quality than Chat (more thorough)
```
### 3. Transformations (Batch Processing)
```
Workflow:
Define transformation → Apply to sources → Notes auto-created
→ Review & edit → Organize
Example Transformation:
Template: "Extract: main argument, methodology, key findings"
Apply to: 5 sources
Result: 5 new notes with consistent structure
When to use:
- Same extraction from many sources
- Building structured knowledge base
- Creating consistent summaries
```
---
## Using Transformations for Batch Insights
### Built-in Transformations
Open Notebook comes with presets:
**Summary**
```
Extracts: Main points, key arguments, conclusions
Output: 200-300 word summary of source
Best for: Quick reference summaries
```
**Key Concepts**
```
Extracts: Main ideas, concepts, terminology
Output: List of concepts with explanations
Best for: Learning and terminology
```
**Methodology**
```
Extracts: Research approach, methods, data
Output: How the research was conducted
Best for: Academic sources, methodology review
```
**Takeaways**
```
Extracts: Actionable insights, recommendations
Output: What you should do with this information
Best for: Practical/business sources
```
### How to Apply Transformation
```
1. Go to "Transformations"
2. Select a template
3. Click "Apply"
4. Select which sources (one or many)
5. Wait for processing (usually 30 seconds - 2 minutes)
6. New notes appear in your notebook
7. Edit if needed
```
### Create Custom Transformation
```
1. Click "Create Custom Transformation"
2. Write your extraction template:
Example:
"For this academic paper, extract:
- Central research question
- Hypothesis tested
- Methodology used
- Key findings (numbered)
- Limitations acknowledged
- Recommendations for future work"
3. Click "Save Template"
4. Apply to one or many sources
5. System generates notes with consistent structure
```
---
## Organizing Notes
### Naming Conventions
**Option 1: Date-based**
```
2026-01-03 - Key points from X source
2026-01-04 - Comparison between A and B
Benefit: Easy to see what you did when
```
**Option 2: Topic-based**
```
AI Safety - Alignment approaches
AI Safety - Interpretability research
Benefit: Groups by subject matter
```
**Option 3: Type-based**
```
SUMMARY: Paper on X
QUESTION: What about Y?
INSIGHT: Connection between Z and W
Benefit: Easy to filter by type
```
**Option 4: Source-based**
```
From: Paper A - Main insights
From: Video B - Interesting implications
Benefit: Easy to trace back to sources
```
**Best practice:** Combine approaches
```
[Date] [Source] - [Topic] - [Type]
2026-01-03 - Paper A - AI Safety - Takeaways
```
### Using Tags
Tags are labels for categorization. Add them when creating notes:
```
Example tags:
- "primary-research" (direct source analysis)
- "background" (supporting material)
- "methodology" (about research methods)
- "insights" (your original thinking)
- "questions" (open questions raised)
- "follow-up" (needs more work)
- "published" (ready to share/use)
```
**Benefits of tags:**
- Filter notes by tag
- Find all notes of a type
- Organize workflow (e.g., find all "follow-up" notes)
### Note Linking & References
You can reference sources within notes:
```markdown
# Analysis of Paper A
As shown in Paper A (see "main argument" section),
the authors argue that...
## Related Sources
- Paper B discusses similar approach
- Video C shows practical application
- My note on "Comparative analysis" has more
```
---
## Editing and Refining Notes
### Improving AI-Generated Notes
```
AI Note:
"The paper discusses machine learning"
What you might change:
"The paper proposes a supervised learning approach
to classification problems, using neural networks
with attention mechanisms (see pp. 15-18)."
How to edit:
1. Click note
2. Click "Edit"
3. Refine the content
4. Click "Save"
```
### Adding Citations
```
When saving from Chat/Ask:
- Citations auto-added
- Shows which sources informed answer
- You can verify by clicking
When manual notes:
- Add manually: "From Paper A, page 15: ..."
- Or reference: "As discussed in [source]"
```
---
## Searching Your Notes
Notes are fully searchable:
### Text Search
```
Find exact phrase: "attention mechanism"
Results: All notes containing that phrase
Use when: Looking for specific terms or quotes
```
### Vector/Semantic Search
```
Find concept: "How do models understand?"
Results: Notes about interpretability, mechanistic understanding, etc.
Use when: Exploring conceptually (words not exact)
```
### Combined Search
```
Text search notes → Find keyword matches
Vector search notes → Find conceptual matches
Both work across sources + notes together
```
---
## Exporting and Sharing Notes
### Options
**Copy to clipboard**
```
Click "Share" → "Copy" → Paste anywhere
Good for: Sharing one note via email/chat
```
**Export as Markdown**
```
Click "Share" → "Export as MD" → Saves as .md file
Good for: Sharing with others, version control
```
**Create note collection**
```
Select multiple notes → "Export collection"
→ Creates organized markdown document
Good for: Sharing a topic overview
```
**Publish to web**
```
Click "Publish" → Get shareable link
Good for: Publishing publicly (if desired)
```
---
## Organizing Your Notebook's Notes
### By Research Phase
**Phase 1: Discovery**
- Initial summaries
- Questions raised
- Interesting findings
**Phase 2: Deep Dive**
- Detailed analysis
- Comparative insights
- Methodology reviews
**Phase 3: Synthesis**
- Connections across sources
- Original thinking
- Conclusions
### By Content Type
**Summaries**
- High-level overviews
- Generated by transformations
- Quick reference
**Questions**
- Open questions
- Things to research more
- Gaps to fill
**Insights**
- Your original analysis
- Connections made
- Conclusions reached
**Tasks**
- Follow-up research
- Sources to add
- People to contact
---
## Using Notes in Other Features
### In Chat
```
You can reference notes:
"Based on my note 'Key findings from A',
how does this compare to B?"
Notes become part of context.
Treated like sources but smaller/more focused.
```
### In Transformations
```
Notes can be transformed:
1. Select notes as input
2. Apply transformation
3. Get new derived notes
Example: Transform 5 analysis notes → Create synthesis
```
### In Podcasts
```
Notes are used to create podcast content:
1. Generate podcast for notebook
2. System includes notes in content selection
3. Notes become part of episode outline
```
---
## Best Practices
### For Manual Notes
1. **Write clearly** — Future you will appreciate it
2. **Add context** — Why this matters, not just what it says
3. **Link to sources** — You can verify later
4. **Date them** — Track your thinking over time
5. **Tag immediately** — Don't defer organization
### For AI-Generated Notes
1. **Review before saving** — Verify quality
2. **Edit for clarity** — AI might miss nuance
3. **Add your thoughts** — Make it your own
4. **Include citations** — Understand sources
5. **Organize right away** — While context is fresh
### For Organization
1. **Consistent naming** — Your future self will thank you
2. **Tag everything** — Makes filtering later much easier
3. **Link related notes** — Create knowledge network
4. **Review periodically** — Refactor as understanding evolves
5. **Archive old notes** — Keep working space clean
---
## Common Mistakes
| Mistake | Problem | Solution |
|---------|---------|----------|
| Save every Chat response | Notebook becomes cluttered with low-quality notes | Only save good responses that answer your questions |
| Don't add tags | Can't find notes later | Tag immediately when creating |
| Poor note titles | Can't remember what's in them | Use descriptive titles, include key concept |
| Never link notes together | Miss connections between ideas | Add references to related notes |
| Forget the source | Can't verify claims later | Always link back to source |
| Never edit AI notes | Keep generic AI responses | Refine for clarity and context |
| Create one giant note | Too long to be useful | Split into focused notes by subtopic |
---
## Summary: Note Lifecycle
```
1. CREATE
├─ Manual: Write from scratch
├─ From Chat: Save good response
├─ From Ask: Save synthesis
└─ From Transform: Batch process
2. EDIT & REFINE
├─ Improve clarity
├─ Add context
├─ Fix AI mistakes
└─ Add citations
3. ORGANIZE
├─ Name clearly
├─ Add tags
├─ Link related
└─ Categorize
4. USE
├─ Reference in Chat
├─ Transform for synthesis
├─ Export for sharing
└─ Build on with new questions
5. MAINTAIN
├─ Periodically review
├─ Update as understanding grows
├─ Archive when done
└─ Learn from organized knowledge
```
Your notes become your actual knowledge base. The more you invest in organizing them, the more valuable they become.
docs/4-AI-PROVIDERS/index.md
+219
View File
@@ -0,0 +1,219 @@
# AI Providers - Comparison & Selection Guide
Open Notebook supports 17+ AI providers. This guide helps you **choose the right provider** for your needs.
> 💡 **Just want to set up a provider?** Skip to the [Configuration Guide](../5-CONFIGURATION/ai-providers.md) for detailed setup instructions.
---
## Quick Decision: Which Provider?
### Cloud Providers (Easiest)
**OpenAI (Recommended)**
- Cost: ~$0.03-0.15 per 1K tokens
- Speed: Very fast
- Quality: Excellent
- Best for: Most users (best quality/price balance)
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#openai)
**Anthropic (Claude)**
- Cost: ~$0.80-3.00 per 1M tokens
- Speed: Fast
- Quality: Excellent
- Best for: Long context (200K tokens), reasoning, latest AI
- Advantage: Superior long-context handling
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#anthropic-claude)
**Google Gemini**
- Cost: ~$0.075-0.30 per 1K tokens
- Speed: Very fast
- Quality: Good to excellent
- Best for: Multimodal (images, audio, video)
- Advantage: Longest context (up to 2M tokens)
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#google-gemini)
**Groq (Ultra-Fast)**
- Cost: ~$0.05 per 1M tokens (cheapest)
- Speed: Ultra-fast (fastest available)
- Quality: Good
- Best for: Budget-conscious, transformations, speed-critical tasks
- Disadvantage: Limited model selection
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#groq)
**OpenRouter (100+ Models)**
- Cost: Pay-per-model (varies widely)
- Speed: Varies by model
- Quality: Varies by model
- Best for: Model comparison, testing, unified billing
- Advantage: One API key for 100+ models from different providers
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#openrouter)
**DashScope (Qwen)**
- Cost: ~$0.01-0.06 per 1K tokens
- Speed: Fast
- Quality: Good
- Best for: Users in Asia, Alibaba Cloud ecosystem
- Advantage: Competitive pricing, strong multilingual support
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#dashscope-qwen)
**MiniMax**
- Cost: Varies by model
- Speed: Fast
- Quality: Good
- Best for: Long context tasks (204K tokens)
- Advantage: Very long context window
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#minimax)
### Local / Self-Hosted (Free)
**Ollama (Recommended for Local)**
- Cost: Free (electricity only)
- Speed: Depends on hardware (slow on CPU, fast on GPU)
- Quality: Good (open-source models)
- Setup: 10 minutes
- Best for: Privacy-first, offline use
- Privacy: 100% local, nothing leaves your machine
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#ollama-recommended-for-local)
**LM Studio (Alternative)**
- Cost: Free (electricity only)
- Speed: Depends on hardware
- Quality: Good (same models as Ollama)
- Setup: 15 minutes (GUI interface)
- Best for: Non-technical users who prefer GUI over CLI
- Privacy: 100% local
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#lm-studio-local-alternative)
### Enterprise
**Azure OpenAI**
- Cost: Same as OpenAI (usage-based)
- Speed: Very fast
- Quality: Excellent (same models as OpenAI)
- Setup: 10 minutes (more complex)
- Best for: Enterprise, compliance (HIPAA, SOC2), VPC integration
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#azure-openai)
---
## Comparison Table
| Provider | Speed | Cost | Quality | Privacy | Setup | Context |
|----------|-------|------|---------|---------|-------|---------|
| **OpenAI** | Very Fast | $$ | Excellent | Low | 5 min | 128K |
| **Anthropic** | Fast | $$ | Excellent | Low | 5 min | 200K |
| **Google** | Very Fast | $$ | Good-Excellent | Low | 5 min | 2M |
| **Groq** | Ultra Fast | $ | Good | Low | 5 min | 32K |
| **OpenRouter** | Varies | Varies | Varies | Low | 5 min | Varies |
| **DashScope** | Fast | $ | Good | Low | 5 min | Varies |
| **MiniMax** | Fast | $$ | Good | Low | 5 min | 204K |
| **Ollama** | Slow-Medium | Free | Good | Max | 10 min | Varies |
| **LM Studio** | Slow-Medium | Free | Good | Max | 15 min | Varies |
| **Azure** | Very Fast | $$ | Excellent | High | 10 min | 128K |
---
## Choosing Your Provider
### I want the easiest setup
→ **OpenAI** — Most popular, best community support
### I have unlimited budget
→ **OpenAI** — Best quality
### I want to save money
→ **Groq** — Cheapest cloud ($0.05 per 1M tokens)
### I want privacy/offline
→ **Ollama** — Free, local, private
### I want a GUI (not CLI)
→ **LM Studio** — Desktop app
### I'm in an enterprise
→ **Azure OpenAI** — Compliance, support
### I need long context (200K+ tokens)
→ **Anthropic** — Best long-context model
### I need multimodal (images, audio, video)
→ **Google Gemini** — Best multimodal support
### I want access to many models with one API key
→ **OpenRouter** — 100+ models, unified billing
---
## Ready to Set Up Your Provider?
Now that you've chosen a provider, follow the detailed setup instructions:
→ **[AI Providers Configuration Guide](../5-CONFIGURATION/ai-providers.md)**
This guide includes:
- Step-by-step setup instructions for each provider via the Settings UI
- How to add credentials, test connections, and discover models
- Model selection and recommendations
- Provider-specific troubleshooting
- Hardware requirements (for local providers)
- Cost optimization tips
---
## Cost Estimator
### OpenAI
```
Light use (10 chats/day): $1-5/month
Medium use (50 chats/day): $10-30/month
Heavy use (all-day use): $50-100+/month
```
### Anthropic
```
Light use: $1-3/month
Medium use: $5-20/month
Heavy use: $20-50+/month
```
### Groq
```
Light use: $0-1/month
Medium use: $2-5/month
Heavy use: $5-20/month
```
### Ollama
```
Any use: Free (electricity only)
8GB GPU running 24/7: ~$10/month electricity
```
---
## Next Steps
1. **You've chosen a provider** (from this comparison guide)
2. **Follow the setup guide**: [AI Providers Configuration](../5-CONFIGURATION/ai-providers.md)
3. **Add your credential** in Settings → API Keys
4. **Test your connection** and discover models
5. **Start using Open Notebook!**
---
## Need Help?
- **Setup issues?** See [AI Providers Configuration](../5-CONFIGURATION/ai-providers.md) for detailed troubleshooting per provider
- **General problems?** Check [Troubleshooting Guide](../6-TROUBLESHOOTING/index.md)
- **Questions?** Join [Discord community](https://discord.gg/37XJPXfz2w)
docs/5-CONFIGURATION/advanced.md
+544
View File
@@ -0,0 +1,544 @@
# Advanced Configuration
Performance tuning, debugging, and advanced features.
---
## Performance Tuning
### Concurrency Control
```env
# Max concurrent database operations (default: 5)
# Increase: Faster processing, more conflicts
# Decrease: Slower, fewer conflicts
SURREAL_COMMANDS_MAX_TASKS=5
```
**Guidelines:**
- CPU: 2 cores → 2-3 tasks
- CPU: 4 cores → 5 tasks (default)
- CPU: 8+ cores → 10-20 tasks
Higher concurrency = more throughput but more database conflicts (retries handle this).
### Retry Strategy
```env
# How to wait between retries
SURREAL_COMMANDS_RETRY_WAIT_STRATEGY=exponential_jitter
# Options:
# - exponential_jitter (recommended)
# - exponential
# - fixed
# - random
```
For high-concurrency deployments, use `exponential_jitter` to prevent thundering herd.
### Timeout Tuning
```env
# Client timeout (default: 300 seconds)
API_CLIENT_TIMEOUT=300
# LLM timeout (default: 60 seconds)
ESPERANTO_LLM_TIMEOUT=60
```
**Guideline:** Set `API_CLIENT_TIMEOUT` > `ESPERANTO_LLM_TIMEOUT` + buffer
```
Example:
ESPERANTO_LLM_TIMEOUT=120
API_CLIENT_TIMEOUT=180 # 120 + 60 second buffer
```
---
## Batching
### TTS Batch Size
For podcast generation, control concurrent TTS requests:
```env
# Default: 5
TTS_BATCH_SIZE=2
```
**Providers and recommendations:**
- OpenAI: 5 (can handle many concurrent)
- Google: 4 (good concurrency)
- ElevenLabs: 2 (limited concurrent requests)
- Local TTS: 1 (single-threaded)
Lower = slower but more stable. Higher = faster but more load on provider.
---
## Logging & Debugging
### Enable Detailed Logging
```bash
# Start with debug logging
RUST_LOG=debug # For Rust components
LOGLEVEL=DEBUG # For Python components
```
### Debug Specific Components
```bash
# Only surreal operations
RUST_LOG=surrealdb=debug
# Only langchain
LOGLEVEL=langchain:debug
# Only specific module
RUST_LOG=open_notebook::database=debug
```
### LangSmith Tracing
For debugging LLM workflows:
```env
LANGCHAIN_TRACING_V2=true
LANGCHAIN_ENDPOINT="https://api.smith.langchain.com"
LANGCHAIN_API_KEY=your-key
LANGCHAIN_PROJECT="Open Notebook"
```
Then visit https://smith.langchain.com to see traces.
---
## Port Configuration
### Default Ports
```
Frontend: 8502 (Docker deployment)
Frontend: 3000 (Development from source)
API: 5055
SurrealDB: 8000
```
### Changing Frontend Port
Edit `docker-compose.yml`:
```yaml
services:
open-notebook:
ports:
- "8001:8502" # Change from 8502 to 8001
```
Access at: `http://localhost:8001`
API auto-detects to: `http://localhost:5055` ✓
### Changing API Port
```yaml
services:
open-notebook:
ports:
- "127.0.0.1:8502:8502" # Frontend
- "5056:5055" # Change API from 5055 to 5056
environment:
- API_URL=http://localhost:5056 # Update API_URL
```
Access API directly: `http://localhost:5056/docs`
**Note:** When changing API port, you must set `API_URL` explicitly since auto-detection assumes port 5055.
### Changing SurrealDB Port
```yaml
services:
surrealdb:
ports:
- "127.0.0.1:8001:8000" # Change from 8000 to 8001 (localhost only)
environment:
- SURREAL_URL=ws://surrealdb:8001/rpc # Update connection URL
```
**Important:** Internal Docker network uses container name (`surrealdb`), not `localhost`.
---
## SSL/TLS Configuration
### Custom CA Certificate
For self-signed certs on local providers:
```env
ESPERANTO_SSL_CA_BUNDLE=/path/to/ca-bundle.pem
```
### Disable Verification (Development Only)
```env
# WARNING: Only for testing/development
# Vulnerable to MITM attacks
ESPERANTO_SSL_VERIFY=false
```
---
## Multi-Provider Setup
### Use Different Providers for Different Tasks
Configure multiple AI providers via **Settings → API Keys**. Each provider gets its own credential:
1. Add a credential for your main language model provider (e.g., OpenAI, Anthropic)
2. Add a credential for embeddings (e.g., Voyage AI, or use the same provider)
3. Add a credential for TTS (e.g., ElevenLabs, or OpenAI-Compatible for local Speaches)
4. Each credential's models are registered and available independently
### Multiple Endpoints for OpenAI-Compatible
When using OpenAI-Compatible providers, you can configure per-service URLs in a single credential:
1. Go to **Settings** → **API Keys**
2. Click **Add Credential** → Select **OpenAI-Compatible**
3. Configure separate URLs for LLM, Embedding, TTS, and STT
4. Click **Save**, then **Test Connection**
---
## Security Hardening
### Change Default Credentials
```env
# Don't use defaults in production
SURREAL_USER=your_secure_username
SURREAL_PASSWORD=$(openssl rand -base64 32) # Generate secure password
```
### Add Password Protection
```env
# Protect your Open Notebook instance
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
### Use HTTPS
```env
# Always use HTTPS in production
API_URL=https://mynotebook.example.com
```
### Firewall Rules
Restrict access to your Open Notebook:
- Port 8502 (frontend): Only from your IP
- Port 5055 (API): Only from frontend
- Port 8000 (SurrealDB): Never expose to internet
---
## Web Scraping & Content Extraction
Open Notebook uses multiple services for content extraction:
### Firecrawl
For advanced web scraping:
```env
FIRECRAWL_API_KEY=your-key
```
Get key from: https://firecrawl.dev/
### Jina AI
Alternative web extraction:
```env
JINA_API_KEY=your-key
```
Get key from: https://jina.ai/
---
## Environment Variable Groups
### Credential Storage (Required)
```env
OPEN_NOTEBOOK_ENCRYPTION_KEY # Required for storing credentials
```
AI provider API keys are configured via **Settings → API Keys** (not environment variables).
### Database
```env
SURREAL_URL
SURREAL_USER
SURREAL_PASSWORD
SURREAL_NAMESPACE
SURREAL_DATABASE
```
### Performance
```env
SURREAL_COMMANDS_MAX_TASKS
SURREAL_COMMANDS_RETRY_ENABLED
SURREAL_COMMANDS_RETRY_MAX_ATTEMPTS
SURREAL_COMMANDS_RETRY_WAIT_STRATEGY
SURREAL_COMMANDS_RETRY_WAIT_MIN
SURREAL_COMMANDS_RETRY_WAIT_MAX
```
### API Settings
```env
API_URL
INTERNAL_API_URL
API_CLIENT_TIMEOUT
ESPERANTO_LLM_TIMEOUT
```
### Audio/TTS
```env
TTS_BATCH_SIZE
```
> **Note:** `ELEVENLABS_API_KEY` is deprecated. Configure ElevenLabs via **Settings → API Keys**.
### Debugging
```env
LANGCHAIN_TRACING_V2
LANGCHAIN_ENDPOINT
LANGCHAIN_API_KEY
LANGCHAIN_PROJECT
```
---
## Testing Configuration
### Quick Test
```bash
# Test API health
curl http://localhost:5055/health
# Test with sample (requires configured credential and registered models)
curl -X POST http://localhost:5055/api/chat \
-H "Content-Type: application/json" \
-d '{"message":"Hello"}'
```
### Validate Config
```bash
# Check environment variables are set
env | grep OPEN_NOTEBOOK_ENCRYPTION_KEY
# Verify database connection
python -c "import os; print(os.getenv('SURREAL_URL'))"
```
---
## Troubleshooting Performance
### High Memory Usage
```env
# Reduce concurrency
SURREAL_COMMANDS_MAX_TASKS=2
# Reduce TTS batch size
TTS_BATCH_SIZE=1
```
### High CPU Usage
```env
# Check worker count
SURREAL_COMMANDS_MAX_TASKS
# Reduce if maxed out:
SURREAL_COMMANDS_MAX_TASKS=5
```
### Slow Responses
```env
# Check timeout settings
API_CLIENT_TIMEOUT=300
# Check retry config
SURREAL_COMMANDS_RETRY_MAX_ATTEMPTS=3
```
### Database Conflicts
```env
# Reduce concurrency
SURREAL_COMMANDS_MAX_TASKS=3
# Use jitter strategy
SURREAL_COMMANDS_RETRY_WAIT_STRATEGY=exponential_jitter
```
---
## Backup & Restore
### Data Locations
| Path | Contents |
|------|----------|
| `./data` or `/app/data` | Uploads, podcasts, checkpoints |
| `./surreal_data` or `/mydata` | SurrealDB database files |
### Quick Backup
```bash
# Stop services (recommended for consistency)
docker compose down
# Create timestamped backup
tar -czf backup-$(date +%Y%m%d-%H%M%S).tar.gz \
notebook_data/ surreal_data/
# Restart services
docker compose up -d
```
### Automated Backup Script
```bash
#!/bin/bash
# backup.sh - Run daily via cron
BACKUP_DIR="/path/to/backups"
DATE=$(date +%Y%m%d-%H%M%S)
# Create backup
tar -czf "$BACKUP_DIR/open-notebook-$DATE.tar.gz" \
/path/to/notebook_data \
/path/to/surreal_data
# Keep only last 7 days
find "$BACKUP_DIR" -name "open-notebook-*.tar.gz" -mtime +7 -delete
echo "Backup complete: open-notebook-$DATE.tar.gz"
```
Add to cron:
```bash
# Daily backup at 2 AM
0 2 * * * /path/to/backup.sh >> /var/log/open-notebook-backup.log 2>&1
```
### Restore
```bash
# Stop services
docker compose down
# Remove old data (careful!)
rm -rf notebook_data/ surreal_data/
# Extract backup
tar -xzf backup-20240115-120000.tar.gz
# Restart services
docker compose up -d
```
### Migration Between Servers
```bash
# On source server
docker compose down
tar -czf open-notebook-migration.tar.gz notebook_data/ surreal_data/
# Transfer to new server
scp open-notebook-migration.tar.gz user@newserver:/path/
# On new server
tar -xzf open-notebook-migration.tar.gz
docker compose up -d
```
---
## Container Management
### Common Commands
```bash
# Start services
docker compose up -d
# Stop services
docker compose down
# View logs (all services)
docker compose logs -f
# View logs (specific service)
docker compose logs -f api
# Restart specific service
docker compose restart api
# Update to latest version
docker compose down
docker compose pull
docker compose up -d
# Check resource usage
docker stats
# Check service health
docker compose ps
```
### Clean Up
```bash
# Remove stopped containers
docker compose rm
# Remove unused images
docker image prune
# Full cleanup (careful!)
docker system prune -a
```
---
## Summary
**Most deployments need:**
- One AI provider API key
- Default database settings
- Default timeouts
**Tune performance only if:**
- You have specific bottlenecks
- High-concurrency workload
- Custom hardware (very fast or very slow)
**Advanced features:**
- Firecrawl for better web scraping
- LangSmith for debugging workflows
- Custom CA bundles for self-signed certs
docs/5-CONFIGURATION/ai-providers.md
+550
View File
@@ -0,0 +1,550 @@
# AI Providers - Configuration Guide
Complete setup instructions for each AI provider via the **Settings UI**.
> **New in v1.2**: All AI provider credentials are now managed through the Settings UI. Environment variables for API keys are deprecated.
---
## How Provider Setup Works
Open Notebook uses a **credential-based system** for managing AI providers:
1. **Get your API key** from the provider's website
2. **Open Settings** → **API Keys** → **Add Credential**
3. **Test the connection** to verify it works
4. **Discover & Register Models** to make them available
5. **Start using** the provider in your notebooks
> **Prerequisite**: You must set `OPEN_NOTEBOOK_ENCRYPTION_KEY` in your docker-compose.yml before storing credentials. See [API Configuration](../3-USER-GUIDE/api-configuration.md#encryption-setup) for details.
---
## Cloud Providers (Recommended for Most)
### OpenAI
**Cost:** ~$0.03-0.15 per 1K tokens (varies by model)
**Get Your API Key:**
1. Go to https://platform.openai.com/api-keys
2. Create account (if needed)
3. Create new API key (starts with "sk-proj-")
4. Add $5+ credits to account
**Configure in Open Notebook:**
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **OpenAI**
4. Give it a name (e.g., "My OpenAI Key")
5. Paste your API key
6. Click **Save**, then **Test Connection**
7. Click **Discover Models** to find available models
8. Click **Register Models** to make them available
**Available Models (in Open Notebook):**
- `gpt-4o` — Best quality, fast (latest version)
- `gpt-4o-mini` — Fast, cheap, good for testing
- `o1` — Advanced reasoning model (slower, more expensive)
- `o1-mini` — Faster reasoning model
**Recommended:**
- For general use: `gpt-4o` (best balance)
- For testing/cheap: `gpt-4o-mini` (90% cheaper)
- For complex reasoning: `o1` (best for hard problems)
**Cost Estimate:**
```
Light use: $1-5/month
Medium use: $10-30/month
Heavy use: $50-100+/month
```
**Troubleshooting:**
- "Invalid API key" → Check key starts with "sk-proj-" and test the connection in Settings
- "Rate limit exceeded" → Wait or upgrade account
- "Model not available" → Try gpt-4o-mini instead, or re-discover models
---
### Anthropic (Claude)
**Cost:** ~$0.80-3.00 per 1M tokens (cheaper than OpenAI for long context)
**Get Your API Key:**
1. Go to https://console.anthropic.com/
2. Create account or login
3. Go to API keys section
4. Create new API key (starts with "sk-ant-")
**Configure in Open Notebook:**
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **Anthropic**
4. Give it a name, paste your API key
5. Click **Save**, then **Test Connection**
6. Click **Discover Models** → **Register Models**
**Available Models:**
- `claude-sonnet-4-5-20250929` — Latest, best quality (recommended)
- `claude-3-5-sonnet-20241022` — Previous generation, still excellent
- `claude-3-5-haiku-20241022` — Fast, cheap
- `claude-opus-4-5-20251101` — Most powerful, expensive
**Recommended:**
- For general use: `claude-sonnet-4-5` (best overall, latest)
- For cheap: `claude-3-5-haiku` (80% cheaper)
- For complex: `claude-opus-4-5` (most capable)
**Cost Estimate:**
```
Sonnet: $3-20/month (typical use)
Haiku: $0.50-3/month
Opus: $10-50+/month
```
**Advantages:**
- Great long-context support (200K tokens)
- Excellent reasoning
- Fast processing
**Troubleshooting:**
- "Invalid API key" → Check it starts with "sk-ant-" and test in Settings
- "Overloaded" → Anthropic is busy, retry later
- "Model unavailable" → Re-discover models from the credential
---
### Google Gemini
**Cost:** ~$0.075-0.30 per 1K tokens (competitive with OpenAI)
**Get Your API Key:**
1. Go to https://aistudio.google.com/app/apikey
2. Create account or login
3. Create new API key
**Configure in Open Notebook:**
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **Google Gemini**
4. Give it a name, paste your API key
5. Click **Save**, then **Test Connection**
6. Click **Discover Models** → **Register Models**
**Available Models:**
- `gemini-2.5-pro` — Strongest, best for long context (1M tokens)
- `gemini-3.5-flash` — Fast, good for general use
- `gemini-3.1-flash-lite` — Fastest and cheapest
- `gemini-2.5-flash` — Previous-gen stable, cheaper
**Recommended:**
- For general use: `gemini-3.5-flash` (best value, latest)
- For cheap: `gemini-3.1-flash-lite` (very cheap)
- For complex/long context: `gemini-2.5-pro` (1M token context)
**Advantages:**
- Very long context (1M tokens)
- Multimodal (images, audio, video)
- Good for podcasts
**Troubleshooting:**
- "API key invalid" → Get fresh key from aistudio.google.com
- "Quota exceeded" → Free tier limited, upgrade account
- "Model not found" → Re-discover models from the credential
---
### Groq
**Cost:** ~$0.05 per 1M tokens (cheapest, but limited models)
**Get Your API Key:**
1. Go to https://console.groq.com/keys
2. Create account or login
3. Create new API key
**Configure in Open Notebook:**
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **Groq**
4. Give it a name, paste your API key
5. Click **Save**, then **Test Connection**
6. Click **Discover Models** → **Register Models**
**Available Models:**
- `llama-3.3-70b-versatile` — Best on Groq (recommended)
- `llama-3.1-70b-versatile` — Fast, capable
- `mixtral-8x7b-32768` — Good alternative
- `gemma2-9b-it` — Small, very fast
**Recommended:**
- For quality: `llama-3.3-70b-versatile` (best overall)
- For speed: `gemma2-9b-it` (ultra-fast)
- For balance: `llama-3.1-70b-versatile`
**Advantages:**
- Ultra-fast inference
- Very cheap
- Great for transformations/batch work
**Disadvantages:**
- Limited model selection
- Smaller models than OpenAI/Anthropic
**Troubleshooting:**
- "Rate limited" → Free tier has limits, upgrade
- "Model not available" → Re-discover models from the credential
---
### OpenRouter
**Cost:** Varies by model ($0.05-15 per 1M tokens)
**Get Your API Key:**
1. Go to https://openrouter.ai/keys
2. Create account or login
3. Add credits to your account
4. Create new API key
**Configure in Open Notebook:**
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **OpenRouter**
4. Give it a name, paste your API key
5. Click **Save**, then **Test Connection**
6. Click **Discover Models** → **Register Models**
**Available Models (100+ options):**
- OpenAI: `openai/gpt-4o`, `openai/o1`
- Anthropic: `anthropic/claude-sonnet-4.5`, `anthropic/claude-3.5-haiku`
- Google: `google/gemini-3.5-flash`, `google/gemini-2.5-pro`
- Meta: `meta-llama/llama-3.3-70b-instruct`, `meta-llama/llama-3.1-405b-instruct`
- Mistral: `mistralai/mistral-large-2411`
- DeepSeek: `deepseek/deepseek-chat`
- And many more...
**Recommended:**
- For quality: `anthropic/claude-sonnet-4.5` (best overall)
- For speed/cost: `google/gemini-2.5-flash` (very fast, cheap)
- For open-source: `meta-llama/llama-3.3-70b-instruct`
- For reasoning: `openai/o1`
**Advantages:**
- One API key for 100+ models
- Unified billing
- Easy model comparison
- Access to models that may have waitlists elsewhere
**Cost Estimate:**
```
Light use: $1-5/month
Medium use: $10-30/month
Heavy use: Depends on models chosen
```
**Troubleshooting:**
- "Invalid API key" → Check it starts with "sk-or-"
- "Insufficient credits" → Add credits at openrouter.ai
- "Model not available" → Check model ID spelling (use full path)
---
### DashScope (Qwen)
**Cost:** ~$0.01-0.06 per 1K tokens (varies by model)
**Get Your API Key:**
1. Go to https://dashscope.console.aliyun.com/
2. Create an Alibaba Cloud account (if needed)
3. Navigate to API Keys section
4. Create a new API key
**Configure in Open Notebook:**
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **DashScope (Qwen)**
4. Give it a name, paste your API key
5. Click **Save**, then **Test Connection**
6. Click **Discover Models** → **Register Models**
**Available Models:**
- `qwen-max` — Most capable Qwen model
- `qwen-plus` — Good balance of quality and speed
- `qwen-turbo` — Fastest, cheapest
**Recommended:**
- For quality: `qwen-max` (best overall)
- For general use: `qwen-plus` (good balance)
- For speed/cost: `qwen-turbo` (cheapest)
**Troubleshooting:**
- "Invalid API key" → Check the key in the DashScope console
- "Model not available" → Re-discover models from the credential
---
### MiniMax
**Cost:** Varies by model
**Get Your API Key:**
1. Go to https://platform.minimaxi.com/
2. Create an account (if needed)
3. Navigate to API Keys section
4. Create a new API key
**Configure in Open Notebook:**
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **MiniMax**
4. Give it a name, paste your API key
5. Click **Save**, then **Test Connection**
6. Click **Discover Models** → **Register Models**
**Available Models:**
- `MiniMax-M2.5` — Most capable, 204K context
- `MiniMax-M2.5-highspeed` — Faster variant, 204K context
**Recommended:**
- For quality: `MiniMax-M2.5` (best overall)
- For speed: `MiniMax-M2.5-highspeed` (faster responses)
**Advantages:**
- Very long context (204K tokens)
- Competitive pricing
**Troubleshooting:**
- "Invalid API key" → Check the key in the MiniMax platform
- "Model not available" → Re-discover models from the credential
---
## Self-Hosted / Local
### Ollama (Recommended for Local)
**Cost:** Free (electricity only)
**Setup Ollama:**
1. Install Ollama: https://ollama.ai
2. Run Ollama in background: `ollama serve`
3. Download a model: `ollama pull mistral`
**Configure in Open Notebook:**
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **Ollama**
4. Give it a name (e.g., "Local Ollama")
5. Enter the base URL:
- Same machine (non-Docker): `http://localhost:11434`
- Docker with Ollama on host: `http://host.docker.internal:11434`
- Docker with Ollama container: `http://ollama:11434`
6. Click **Save**, then **Test Connection**
7. Click **Discover Models** → **Register Models**
See [Ollama Setup Guide](ollama.md) for detailed network configuration.
**Context Window (`num_ctx`):**
Ollama models default to a **8,192-token** context window. This default is intentionally
conservative so models run reliably on consumer GPUs (≈8GB VRAM) without running out of memory.
If your hardware can handle more, set an optional **Context Window (num_ctx)** value on the
Ollama credential (Settings → API Keys → edit the Ollama credential). It applies to all models
that use that credential. Leave it empty to keep the default.
- Raise it (e.g. `32768`) when ingesting large documents or using long chat histories.
- If you hit "out of memory" errors, lower it or leave it at the default.
**Available Models:**
- `llama3.3:70b` — Best quality (requires 40GB+ RAM)
- `llama3.1:8b` — Recommended, balanced (8GB RAM)
- `qwen2.5:7b` — Excellent for code and reasoning
- `mistral:7b` — Good general purpose
- `phi3:3.8b` — Small, fast (4GB RAM)
- `gemma2:9b` — Google's model, balanced
- Many more: `ollama list` to see available
**Recommended:**
- For quality (with GPU): `llama3.3:70b` (best)
- For general use: `llama3.1:8b` (best balance)
- For speed/low memory: `phi3:3.8b` (very fast)
- For coding: `qwen2.5:7b` (excellent at code)
**Hardware Requirements:**
```
GPU (NVIDIA/AMD):
8GB VRAM: Runs most models fine
6GB VRAM: Works, slower
4GB VRAM: Small models only
CPU-only:
16GB+ RAM: Slow but works
8GB RAM: Very slow
4GB RAM: Not recommended
```
**Advantages:**
- Completely private (runs locally)
- Free (electricity only)
- No API key needed
- Works offline
**Disadvantages:**
- Slower than cloud (unless on GPU)
- Smaller models than cloud
- Requires local hardware
**Troubleshooting:**
- "Connection refused" → Ollama not running or wrong URL in credential
- "Model not found" → Download it: `ollama pull modelname`
- "Out of memory" → Use smaller model or add more RAM
---
### LM Studio (Local Alternative)
**Cost:** Free
**Setup LM Studio:**
1. Download LM Studio: https://lmstudio.ai
2. Open app
3. Download a model from library
4. Go to "Local Server" tab
5. Start server (default port: 1234)
**Configure in Open Notebook:**
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **OpenAI-Compatible**
4. Give it a name (e.g., "LM Studio")
5. Enter the base URL: `http://host.docker.internal:1234/v1` (Docker) or `http://localhost:1234/v1` (local)
6. API key: `lm-studio` (placeholder, LM Studio doesn't require one)
7. Click **Save**, then **Test Connection**
**Advantages:**
- GUI interface (easier than Ollama CLI)
- Good model selection
- Privacy-focused
- Works offline
**Disadvantages:**
- Desktop only (Mac/Windows/Linux)
- Slower than cloud
- Requires local GPU
---
### Custom OpenAI-Compatible
For Text Generation UI, vLLM, or other OpenAI-compatible endpoints:
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **OpenAI-Compatible**
4. Enter the base URL for your endpoint (e.g., `http://localhost:8000/v1`)
5. Enter API key if required
6. Optionally configure per-service URLs (LLM, Embedding, TTS, STT)
7. Click **Save**, then **Test Connection**
See [OpenAI-Compatible Setup](openai-compatible.md) for detailed instructions.
---
## Enterprise
### Azure OpenAI
**Cost:** Same as OpenAI (usage-based)
**Configure in Open Notebook:**
1. Create Azure OpenAI service in Azure portal
2. Deploy GPT-4/3.5-turbo model
3. Get your endpoint and key
4. Go to **Settings** → **API Keys**
5. Click **Add Credential**
6. Select provider: **Azure OpenAI**
7. Fill in: API Key, Endpoint, API Version
8. Optionally configure service-specific endpoints (LLM, Embedding)
9. Click **Save**, then **Test Connection**
**Advantages:**
- Enterprise support
- VPC integration
- Compliance (HIPAA, SOC2, etc.)
**Disadvantages:**
- More complex setup
- Higher overhead
- Requires Azure account
---
## Embeddings (For Search/Semantic Features)
By default, Open Notebook uses the LLM provider's embeddings. Embedding models are discovered and registered through the same credential system — when you discover models from a credential, embedding models are included alongside language models.
---
## Choosing Your Provider
**1. Don't want to run locally and don't want to mess around with different providers:**
Use OpenAI
- Cloud-based
- Good quality
- Reasonable cost
- Simplest setup, supports all modes (text, embedding, tts, stt, etc)
**For budget-conscious:** Groq, OpenRouter or Ollama
- Groq: Super cheap cloud
- Ollama: Free, but local
- OpenRouter: many open source models very accessible
**For privacy-first:** Ollama or LM Studio and Speaches ([TTS](local-tts.md), [STT](local-stt.md))
- Everything stays local
- Works offline
- No API keys sent anywhere
**For enterprise:** Azure OpenAI
- Compliance
- VPC integration
- Support
---
## Next Steps
1. **Choose your provider** from above
2. **Get API key** (if cloud) or install locally (if Ollama)
3. **Set `OPEN_NOTEBOOK_ENCRYPTION_KEY`** in your docker-compose.yml (required for storing credentials)
4. **Open Settings** → **API Keys** → **Add Credential**
5. **Test Connection** to verify it works
6. **Discover & Register Models** to make them available
7. **Verify it works** with a test chat
> **Multiple providers**: You can add credentials for as many providers as you want. Create separate credentials for different projects or team members.
Done!
---
## Legacy: Environment Variables (Deprecated)
> **Deprecated**: Configuring AI provider API keys via environment variables is deprecated. Use the Settings UI instead. Environment variables may still work as a fallback but are no longer the recommended approach.
If you are migrating from an older version that used environment variables, go to **Settings** → **API Keys** and click the **Migrate to Database** button to import your existing keys into the credential system.
---
## Related
- **[API Configuration](../3-USER-GUIDE/api-configuration.md)** — Detailed credential management guide
- **[Environment Reference](environment-reference.md)** - Complete list of all environment variables
- **[Advanced Configuration](advanced.md)** - Timeouts, SSL, performance tuning
- **[Ollama Setup](ollama.md)** - Detailed Ollama configuration guide
- **[OpenAI-Compatible](openai-compatible.md)** - LM Studio and other compatible providers
- **[Local TTS Setup](local-tts.md)** - Text-to-speech with Speaches
- **[Local STT Setup](local-stt.md)** - Speech-to-text with Speaches
- **[Troubleshooting](../6-TROUBLESHOOTING/quick-fixes.md)** - Common issues and fixes
docs/5-CONFIGURATION/database.md
+52
View File
@@ -0,0 +1,52 @@
# Database - SurrealDB Configuration
Open Notebook uses SurrealDB for its database needs.
---
## Default Configuration
Open Notebook should work out of the box with SurrealDB as long as the environment variables are correctly setup.
### DB running in the same docker compose as Open Notebook (recommended)
The example above is for when you are running SurrealDB as a separate docker container, which is the method described [here](../1-INSTALLATION/docker-compose.md) (and our recommended method).
```env
SURREAL_URL="ws://surrealdb:8000/rpc"
SURREAL_USER="root"
SURREAL_PASSWORD="root"
SURREAL_NAMESPACE="open_notebook"
SURREAL_DATABASE="open_notebook"
```
### DB running in the host machine and Open Notebook running in Docker
If ON is running in docker and SurrealDB is on your host machine, you need to point to it.
```env
SURREAL_URL="ws://your-machine-ip:8000/rpc" #or host.docker.internal
SURREAL_USER="root"
SURREAL_PASSWORD="root"
SURREAL_NAMESPACE="open_notebook"
SURREAL_DATABASE="open_notebook"
```
> **Note:** If SurrealDB runs in Docker with its port published on `127.0.0.1` only (the documented default), it won't be reachable at your machine's IP. Re-publish the port deliberately — see `docker-compose.override.yml.example` in the repo root — behind a firewall or SSH tunnel, with real credentials set.
### Open Notebook and Surreal are running on the same machine
If you are running both services locally or if you are using the deprecated [single container setup](../1-INSTALLATION/single-container.md)
```env
SURREAL_URL="ws://localhost:8000/rpc"
SURREAL_USER="root"
SURREAL_PASSWORD="root"
SURREAL_NAMESPACE="open_notebook"
SURREAL_DATABASE="open_notebook"
```
## Multiple databases
You can have multiple namespaces in one SurrealDB instance and you can also have multiple databases in one instance. So, if you want to setup multiple open noteobok deployments for different users, you don't need to deploy multiple databases.
docs/5-CONFIGURATION/environment-reference.md
+307
View File
@@ -0,0 +1,307 @@
# Complete Environment Reference
Comprehensive list of all environment variables available in Open Notebook.
---
## API Configuration
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `API_URL` | No | Auto-detected | URL where frontend reaches API (e.g., http://localhost:5055) |
| `INTERNAL_API_URL` | No | http://localhost:5055 | Internal API URL for Next.js server-side proxying |
| `API_CLIENT_TIMEOUT` | No | 300 | Client timeout in seconds (how long to wait for API response) |
| `OPEN_NOTEBOOK_PASSWORD` | No | None | Password to protect Open Notebook instance |
| `OPEN_NOTEBOOK_ENCRYPTION_KEY` | **Yes** | None | Secret string to encrypt credentials stored in database (any string works). **Required** for the credential system. Supports Docker secrets via `_FILE` suffix. |
| `FRONTEND_BIND_HOST` | No | `0.0.0.0` (in Docker) | Network interface for Next.js to bind to. Default `0.0.0.0` ensures accessibility from reverse proxies. (Replaces `HOSTNAME`, which container runtimes such as Podman override with the container/pod hostname, causing Next.js to bind to the wrong address) |
| `API_HOST` | No | `0.0.0.0` (in Docker) | Network interface for the API (uvicorn) to bind to. Set to `::` for IPv6 dual-stack environments (listens on IPv6 and, on Linux defaults, IPv4 too) |
| `OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB` | No | 100 | Maximum request body size (in MB) the API will accept, enforced before auth/routing. Raise this if you need to upload larger audio/video files. A fronting reverse proxy's own limit (e.g. nginx `client_max_body_size`) still applies and should be raised to match. |
> **Important**: `OPEN_NOTEBOOK_ENCRYPTION_KEY` is required for storing AI provider credentials via the Settings UI. Without it, you cannot save credentials. If you change or lose this key, all stored credentials become unreadable.
---
## Database: SurrealDB
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `SURREAL_URL` | Yes | ws://surrealdb:8000/rpc | SurrealDB WebSocket connection URL |
| `SURREAL_USER` | Yes | root | SurrealDB username |
| `SURREAL_PASSWORD` | Yes | root | SurrealDB password |
| `SURREAL_NAMESPACE` | Yes | open_notebook | SurrealDB namespace |
| `SURREAL_DATABASE` | Yes | open_notebook | SurrealDB database name |
---
## Database: Retry Configuration
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `SURREAL_COMMANDS_RETRY_ENABLED` | No | true | Enable retries on failure |
| `SURREAL_COMMANDS_RETRY_MAX_ATTEMPTS` | No | 3 | Maximum retry attempts |
| `SURREAL_COMMANDS_RETRY_WAIT_STRATEGY` | No | exponential_jitter | Retry wait strategy (exponential_jitter/exponential/fixed/random) |
| `SURREAL_COMMANDS_RETRY_WAIT_MIN` | No | 1 | Minimum wait time between retries (seconds) |
| `SURREAL_COMMANDS_RETRY_WAIT_MAX` | No | 30 | Maximum wait time between retries (seconds) |
---
## Database: Concurrency
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `SURREAL_COMMANDS_MAX_TASKS` | No | 5 | Maximum concurrent database tasks |
---
## LLM Timeouts
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `ESPERANTO_LLM_TIMEOUT` | No | 60 | LLM inference timeout in seconds |
| `ESPERANTO_SSL_VERIFY` | No | true | Verify SSL certificates (false = development only) |
| `ESPERANTO_SSL_CA_BUNDLE` | No | None | Path to custom CA certificate bundle |
---
## Embeddings
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE` | No | 50 | Number of texts sent per embedding batch. Lower this for CPU-only or stricter OpenAI-compatible embedding providers. |
| `OPEN_NOTEBOOK_MIN_CHUNK_SIZE` | No | 5 | Minimum chunk size in tokens. Chunks below this threshold are dropped before embedding to avoid degenerate single-character fragments that some providers (e.g. llama.cpp) return null embeddings for. Set to `0` to disable filtering. |
---
## API / CORS
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `CORS_ORIGINS` | No | `*` | Comma-separated list of origins allowed to call the API (e.g. `https://app.example.com,https://www.example.com`). Default `*` accepts any origin; **for production, set this explicitly to your frontend origin(s)**. Changes require an API restart. The API logs a warning on startup when unset. |
**When to change this**:
- You access the UI at a custom domain (reverse proxy, HTTPS, public deployment).
- The frontend runs on a different port than `3000`.
- You serve the frontend from a different host than the API (e.g. CDN).
Example for a production deployment behind a reverse proxy:
```bash
CORS_ORIGINS=https://notebook.example.com
```
---
## Text-to-Speech (TTS)
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `TTS_BATCH_SIZE` | No | 5 | Concurrent TTS requests (1-5, depends on provider) |
| `ESPERANTO_TTS_TIMEOUT` | No | 300 | Text-to-speech request timeout in seconds (passed through to Esperanto). Increase it for slow or self-hosted TTS providers that take longer than 5 minutes to synthesize a segment, otherwise long podcast segments can fail with a timeout. |
---
## Content Extraction
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `FIRECRAWL_API_KEY` | No | None | Firecrawl API key for advanced web scraping |
| `JINA_API_KEY` | No | None | Jina AI API key for web extraction |
**Setup:**
- Firecrawl: https://firecrawl.dev/
- Jina: https://jina.ai/
---
## Network / Proxy
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `HTTP_PROXY` | No | None | HTTP proxy URL for outbound HTTP requests |
| `HTTPS_PROXY` | No | None | HTTPS proxy URL for outbound HTTPS requests |
| `NO_PROXY` | No | None | Comma-separated list of hosts to bypass proxy |
Route all outbound HTTP requests through a proxy server. Useful for corporate/firewalled environments.
The underlying libraries (esperanto, content-core, podcast-creator) automatically detect proxy settings from these standard environment variables.
**Affects:**
- AI provider API calls (OpenAI, Anthropic, Google, Groq, etc.)
- Content extraction from URLs (web scraping, YouTube transcripts)
- Podcast generation (LLM and TTS provider calls)
**Format:** `http://[user:pass@]host:port` or `https://[user:pass@]host:port`
**Examples:**
```bash
# Basic proxy
HTTP_PROXY=http://proxy.corp.com:8080
HTTPS_PROXY=http://proxy.corp.com:8080
# Authenticated proxy
HTTP_PROXY=http://user:password@proxy.corp.com:8080
HTTPS_PROXY=http://user:password@proxy.corp.com:8080
# Bypass proxy for local hosts
NO_PROXY=localhost,127.0.0.1,.local
```
---
## Debugging & Monitoring
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `LANGCHAIN_TRACING_V2` | No | false | Enable LangSmith tracing |
| `LANGCHAIN_ENDPOINT` | No | https://api.smith.langchain.com | LangSmith endpoint |
| `LANGCHAIN_API_KEY` | No | None | LangSmith API key |
| `LANGCHAIN_PROJECT` | No | Open Notebook | LangSmith project name |
**Setup:** https://smith.langchain.com/
---
## Environment Variables by Use Case
### Minimal Setup (New Installation)
```
OPEN_NOTEBOOK_ENCRYPTION_KEY=my-secret-key
SURREAL_URL=ws://surrealdb:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=password
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=open_notebook
```
Then configure AI providers via **Settings → API Keys** in the browser.
### Production Deployment
```
OPEN_NOTEBOOK_ENCRYPTION_KEY=your-strong-secret-key
OPEN_NOTEBOOK_PASSWORD=your-secure-password
API_URL=https://mynotebook.example.com
SURREAL_USER=production_user
SURREAL_PASSWORD=secure_password
```
### Self-Hosted Behind Reverse Proxy
```
OPEN_NOTEBOOK_ENCRYPTION_KEY=your-secret-key
API_URL=https://mynotebook.example.com
```
### Corporate Environment (Behind Proxy)
```
OPEN_NOTEBOOK_ENCRYPTION_KEY=your-secret-key
HTTP_PROXY=http://proxy.corp.com:8080
HTTPS_PROXY=http://proxy.corp.com:8080
NO_PROXY=localhost,127.0.0.1
```
### High-Performance Deployment
```
OPEN_NOTEBOOK_ENCRYPTION_KEY=your-secret-key
SURREAL_COMMANDS_MAX_TASKS=10
TTS_BATCH_SIZE=5
API_CLIENT_TIMEOUT=600
```
### Debugging
```
OPEN_NOTEBOOK_ENCRYPTION_KEY=your-secret-key
LANGCHAIN_TRACING_V2=true
LANGCHAIN_API_KEY=your-key
```
---
## Validation
Check if a variable is set:
```bash
# Check single variable
echo $OPEN_NOTEBOOK_ENCRYPTION_KEY
# Check multiple
env | grep -E "OPEN_NOTEBOOK|API_URL"
# Print all config
env | grep -E "^[A-Z_]+=" | sort
```
---
## Notes
- **Case-sensitive:** `OPEN_NOTEBOOK_ENCRYPTION_KEY` ≠ `open_notebook_encryption_key`
- **No spaces:** `OPEN_NOTEBOOK_ENCRYPTION_KEY=my-key` not `OPEN_NOTEBOOK_ENCRYPTION_KEY = my-key`
- **Quote values:** Use quotes for values with spaces: `API_URL="http://my server:5055"`
- **Restart required:** Changes take effect after restarting services
- **Secrets:** Don't commit encryption keys or passwords to git
- **AI Providers:** Configure via **Settings → API Keys** in the browser (not via env vars)
- **Migration:** Use Settings UI to migrate existing env vars to the credential system. See [API Configuration](../3-USER-GUIDE/api-configuration.md#migrating-from-environment-variables)
---
## Quick Setup Checklist
- [ ] Set `OPEN_NOTEBOOK_ENCRYPTION_KEY` in docker-compose.yml
- [ ] Set database credentials (`SURREAL_*`)
- [ ] Start services
- [ ] Open browser → Go to **Settings → API Keys**
- [ ] **Add Credential** for your AI provider
- [ ] **Test Connection** to verify
- [ ] **Discover & Register Models**
- [ ] Set `API_URL` if behind reverse proxy
- [ ] Change `SURREAL_PASSWORD` in production
- [ ] Try a test chat
Done!
---
## Legacy: AI Provider Environment Variables (Deprecated)
> **Deprecated**: The following AI provider API key environment variables are deprecated. Configure providers via the Settings UI instead. These variables may still work as a fallback but are no longer recommended.
If you have these variables configured from a previous installation, click the **Migrate to Database** button in **Settings → API Keys** to import them into the credential system, then remove them from your configuration.
| Variable | Provider | Replacement |
|----------|----------|-------------|
| `OPENAI_API_KEY` | OpenAI | Settings → API Keys → Add OpenAI Credential |
| `ANTHROPIC_API_KEY` | Anthropic | Settings → API Keys → Add Anthropic Credential |
| `GOOGLE_API_KEY` | Google Gemini | Settings → API Keys → Add Google Credential |
| `GEMINI_API_BASE_URL` | Google Gemini | Configure in Google Gemini credential |
| `VERTEX_PROJECT` | Vertex AI | Settings → API Keys → Add Vertex AI Credential |
| `VERTEX_LOCATION` | Vertex AI | Configure in Vertex AI credential |
| `GOOGLE_APPLICATION_CREDENTIALS` | Vertex AI | Configure in Vertex AI credential |
| `GROQ_API_KEY` | Groq | Settings → API Keys → Add Groq Credential |
| `MISTRAL_API_KEY` | Mistral | Settings → API Keys → Add Mistral Credential |
| `DEEPSEEK_API_KEY` | DeepSeek | Settings → API Keys → Add DeepSeek Credential |
| `XAI_API_KEY` | xAI | Settings → API Keys → Add xAI Credential |
| `OLLAMA_API_BASE` | Ollama | Settings → API Keys → Add Ollama Credential |
| `OPENROUTER_API_KEY` | OpenRouter | Settings → API Keys → Add OpenRouter Credential |
| `OPENROUTER_BASE_URL` | OpenRouter | Configure in OpenRouter credential |
| `VOYAGE_API_KEY` | Voyage AI | Settings → API Keys → Add Voyage AI Credential |
| `ELEVENLABS_API_KEY` | ElevenLabs | Settings → API Keys → Add ElevenLabs Credential |
| `OPENAI_COMPATIBLE_BASE_URL` | OpenAI-Compatible | Settings → API Keys → Add OpenAI-Compatible Credential |
| `OPENAI_COMPATIBLE_API_KEY` | OpenAI-Compatible | Configure in OpenAI-Compatible credential |
| `OPENAI_COMPATIBLE_BASE_URL_LLM` | OpenAI-Compatible | Configure per-service URL in credential |
| `OPENAI_COMPATIBLE_API_KEY_LLM` | OpenAI-Compatible | Configure per-service key in credential |
| `OPENAI_COMPATIBLE_BASE_URL_EMBEDDING` | OpenAI-Compatible | Configure per-service URL in credential |
| `OPENAI_COMPATIBLE_API_KEY_EMBEDDING` | OpenAI-Compatible | Configure per-service key in credential |
| `OPENAI_COMPATIBLE_BASE_URL_STT` | OpenAI-Compatible | Configure per-service URL in credential |
| `OPENAI_COMPATIBLE_API_KEY_STT` | OpenAI-Compatible | Configure per-service key in credential |
| `OPENAI_COMPATIBLE_BASE_URL_TTS` | OpenAI-Compatible | Configure per-service URL in credential |
| `OPENAI_COMPATIBLE_API_KEY_TTS` | OpenAI-Compatible | Configure per-service key in credential |
| `DASHSCOPE_API_KEY` | DashScope (Qwen) | Settings → API Keys → Add DashScope Credential |
| `MINIMAX_API_KEY` | MiniMax | Settings → API Keys → Add MiniMax Credential |
| `AZURE_OPENAI_API_KEY` | Azure OpenAI | Settings → API Keys → Add Azure OpenAI Credential |
| `AZURE_OPENAI_ENDPOINT` | Azure OpenAI | Configure in Azure OpenAI credential |
| `AZURE_OPENAI_API_VERSION` | Azure OpenAI | Configure in Azure OpenAI credential |
| `AZURE_OPENAI_API_KEY_LLM` | Azure OpenAI | Configure per-service in credential |
| `AZURE_OPENAI_ENDPOINT_LLM` | Azure OpenAI | Configure per-service in credential |
| `AZURE_OPENAI_API_VERSION_LLM` | Azure OpenAI | Configure per-service in credential |
| `AZURE_OPENAI_API_KEY_EMBEDDING` | Azure OpenAI | Configure per-service in credential |
| `AZURE_OPENAI_ENDPOINT_EMBEDDING` | Azure OpenAI | Configure per-service in credential |
| `AZURE_OPENAI_API_VERSION_EMBEDDING` | Azure OpenAI | Configure per-service in credential |
docs/5-CONFIGURATION/index.md
+328
View File
@@ -0,0 +1,328 @@
# Configuration - Essential Settings
Configuration is how you customize Open Notebook for your specific setup. This section covers what you need to know.
---
## What Needs Configuration?
Three things:
1. **AI Provider** — Which LLM/embedding service you're using (OpenAI, Anthropic, Ollama, etc.)
2. **Database** — How to connect to SurrealDB (usually pre-configured)
3. **Server** — API URL, ports, timeouts (usually auto-detected)
---
## Quick Decision: Which Provider?
### Option 1: Cloud Provider (Fastest)
- **OpenRouter (recommended)** (access to all models with one key)
- **OpenAI** (GPT)
- **Anthropic** (Claude)
- **Google Gemini** (multi-modal, long context)
- **Groq** (ultra-fast inference)
Setup: Get API key → Add credential in Settings UI → Done
→ Go to **[AI Providers Guide](ai-providers.md)**
### Option 2: Local (Free & Private)
- **Ollama** (open-source models, on your machine)
→ Go to **[Ollama Setup](ollama.md)**
### Option 3: OpenAI-Compatible
- **LM Studio** (local)
- **Custom endpoints**
→ Go to **[OpenAI-Compatible Guide](openai-compatible.md)**
---
## Configuration File
Use the right file depending on your setup.
### `.env` (Local Development)
You will only use .env if you are running Open Notebook locally.
```
Located in: project root
Use for: Development on your machine
Format: KEY=value, one per line
```
### `docker.env` (Docker Deployment)
You will use this file to hold your environment variables if you are using docker-compose and prefer not to put the variables directly in the compose file.
```
Located in: project root (or ./docker)
Use for: Docker deployments
Format: Same as .env
Loaded by: docker-compose.yml
```
---
## Most Important Settings
All of the settings provided below are to be placed inside your environment file (.env or docker.env depending on your setup).
### Surreal Database
This is the database used by the app.
```
SURREAL_URL=ws://surrealdb:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=root # Change in production!
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=open_notebook
```
> The only thing that is critical to not miss is the hostname in the `SURREAL_URL`. Check what URL to use based on your deployment, [here](database.md).
### AI Provider (Credentials)
We need access to LLMs in order for the app to work. AI provider credentials are configured via the **Settings UI**:
1. Set `OPEN_NOTEBOOK_ENCRYPTION_KEY` in your environment (required for storing credentials)
2. Start services
3. Go to **Settings → API Keys → Add Credential**
4. Select your provider, paste your API key
5. **Test Connection** → **Discover Models** → **Register Models**
```
# Required in your .env or docker-compose.yml:
OPEN_NOTEBOOK_ENCRYPTION_KEY=my-secret-key
```
> **Ollama users**: Add an Ollama credential in Settings → API Keys with the correct base URL. See [Ollama Setup](ollama.md) for network configuration help.
> **LM Studio / OpenAI-Compatible**: Add an OpenAI-Compatible credential in Settings → API Keys. See [OpenAI-Compatible Guide](openai-compatible.md).
### API URL (If Behind Reverse Proxy)
You only need to worry about this if you are deploying on a proxy or if you are changing port information. Otherwise, skip this.
```
API_URL=https://your-domain.com
# Usually auto-detected. Only set if needed.
```
Auto-detection works for most setups.
---
## Configuration by Scenario
### Scenario 1: Docker on Localhost (Default)
```env
# In docker.env:
OPEN_NOTEBOOK_ENCRYPTION_KEY=my-secret-key
# Everything else uses defaults
# Then configure AI provider in Settings → API Keys
```
### Scenario 2: Docker on Remote Server
```env
# In docker.env:
OPEN_NOTEBOOK_ENCRYPTION_KEY=my-secret-key
API_URL=http://your-server-ip:5055
```
### Scenario 3: Behind Reverse Proxy (Nginx/Cloudflare)
```env
# In docker.env:
OPEN_NOTEBOOK_ENCRYPTION_KEY=my-secret-key
API_URL=https://your-domain.com
# The reverse proxy handles HTTPS
```
### Scenario 4: Using Ollama Locally
```env
# In .env:
OPEN_NOTEBOOK_ENCRYPTION_KEY=my-secret-key
# Then add Ollama credential in Settings → API Keys
```
### Scenario 5: Using Azure OpenAI
```env
# In docker.env:
OPEN_NOTEBOOK_ENCRYPTION_KEY=my-secret-key
# Then add Azure OpenAI credential in Settings → API Keys
```
---
## Configuration Sections
### [AI Providers](ai-providers.md)
- OpenAI configuration
- Anthropic configuration
- Google Gemini configuration
- Groq configuration
- Ollama configuration
- Azure OpenAI configuration
- OpenAI-compatible configuration
### [Database](database.md)
- SurrealDB setup
- Connection strings
- Database vs. namespace
- Running your own SurrealDB
### [Advanced](advanced.md)
- Ports and networking
- Timeouts and concurrency
- SSL/security
- Retry configuration
- Worker concurrency
- Language models & embeddings
- Speech-to-text & text-to-speech
- Debugging and logging
### [Reverse Proxy](reverse-proxy.md)
- Nginx, Caddy, Traefik configs
- Custom domain setup
- SSL/HTTPS configuration
- Coolify and other platforms
### [Security](security.md)
- Password protection
- API authentication
- Production hardening
- Firewall configuration
### [Local TTS](local-tts.md)
- Speaches setup for local text-to-speech
- GPU acceleration
- Voice options
- Docker networking
### [Local STT](local-stt.md)
- Speaches setup for local speech-to-text
- Whisper model options
- GPU acceleration
- Docker networking
### [Ollama](ollama.md)
- Setting up and pointing to an Ollama server
- Downloading models
- Using embedding
### [OpenAI-Compatible Providers](openai-compatible.md)
- LM Studio, vLLM, Text Generation WebUI
- Connection configuration
- Docker networking
- Troubleshooting
### [Complete Reference](environment-reference.md)
- All environment variables
- Grouped by category
- What each one does
- Default values
---
## How to Add Configuration
### Method 1: Settings UI (For AI Provider Credentials)
The recommended way to configure AI providers:
```
1. Open Open Notebook in your browser
2. Go to Settings → API Keys
3. Click "Add Credential"
4. Select provider, enter API key
5. Click Save, then Test Connection
6. Click Discover Models → Register Models
```
No file editing, no restarts. Credentials stored securely (encrypted) in database.
→ **[Full Guide: API Configuration](../3-USER-GUIDE/api-configuration.md)**
### Method 2: Edit `.env` File (Infrastructure Settings)
For database, network, and encryption key settings:
```bash
1. Open .env in your editor
2. Set OPEN_NOTEBOOK_ENCRYPTION_KEY and database vars
3. Save
4. Restart services
```
### Method 3: Set Docker Environment (Deployment)
```bash
# In docker-compose.yml:
services:
api:
environment:
- OPEN_NOTEBOOK_ENCRYPTION_KEY=my-secret-key
- API_URL=https://your-domain.com
```
---
## Verification
After configuration, verify it works:
```
1. Open your notebook
2. Go to Settings → Models
3. You should see your configured provider
4. Try a simple Chat question
5. If it responds, configuration is correct!
```
---
## Common Mistakes
| Mistake | Problem | Fix |
|---------|---------|-----|
| No credential configured | Models not available | Add credential in Settings → API Keys |
| Missing encryption key | Can't save credentials | Set OPEN_NOTEBOOK_ENCRYPTION_KEY |
| Wrong database URL | Can't start API | Check SURREAL_URL format |
| Expose port 5055 | "Can't connect to server" | Expose 5055 in docker-compose |
| Typo in env var | Settings ignored | Check spelling (case-sensitive!) |
| Don't restart | Old config still used | Restart services after env changes |
---
## What Comes After Configuration
Once configured:
1. **[Quick Start](../0-START-HERE/index.md)** — Run your first notebook
2. **[Installation](../1-INSTALLATION/index.md)** — Multi-route deployment guides
3. **[User Guide](../3-USER-GUIDE/index.md)** — How to use each feature
---
## Getting Help
- **Configuration error?** → Check [Troubleshooting](../6-TROUBLESHOOTING/quick-fixes.md)
- **Provider-specific issue?** → Check [AI Providers](ai-providers.md)
- **Need complete reference?** → See [Environment Reference](environment-reference.md)
---
## Summary
**Minimal configuration to run:**
1. Set `OPEN_NOTEBOOK_ENCRYPTION_KEY` in your environment
2. Start services
3. Add AI provider credential in Settings → API Keys
4. Done!
Everything else is optional optimization.
docs/5-CONFIGURATION/local-stt.md
+366
View File
@@ -0,0 +1,366 @@
# Local Speech-to-Text Setup
Run speech-to-text locally for free, private audio/video transcription using OpenAI-compatible STT servers.
---
## Why Local STT?
| Benefit | Description |
|---------|-------------|
| **Free** | No per-minute costs after setup |
| **Private** | Audio never leaves your machine |
| **Unlimited** | No rate limits or quotas |
| **Offline** | Works without internet |
---
## Quick Start with Speaches
[Speaches](https://github.com/speaches-ai/speaches) is an open-source, OpenAI-compatible server that supports both TTS and STT. It uses [faster-whisper](https://github.com/SYSTRAN/faster-whisper) for transcription.
> **💡 Ready-made Docker Compose files available:**
> - **[docker-compose-speaches.yml](../../examples/docker-compose-speaches.yml)** - Speaches + Open Notebook
> - **[docker-compose-full-local.yml](../../examples/docker-compose-full-local.yml)** - Speaches + Ollama (100% local setup)
>
> These include complete setup instructions and configuration examples. Just copy and run!
### Step 1: Create Docker Compose File
Create a folder and add `docker-compose.yml`:
```yaml
services:
speaches:
image: ghcr.io/speaches-ai/speaches:latest-cpu
container_name: speaches
ports:
- "8969:8000"
volumes:
- hf-hub-cache:/home/ubuntu/.cache/huggingface/hub
restart: unless-stopped
volumes:
hf-hub-cache:
```
### Step 2: Start and Download Model
```bash
# Start Speaches
docker compose up -d
# Wait for startup
sleep 10
# Download Whisper model (~500MB for small)
docker compose exec speaches uv tool run speaches-cli model download Systran/faster-whisper-small
```
Models can also be downloaded automatically on first use, but pre-downloading avoids delays.
### Step 3: Test
```bash
# Create a test audio file (or use your own)
# Then transcribe it:
curl "http://localhost:8969/v1/audio/transcriptions" \
-F "file=@test.mp3" \
-F "model=Systran/faster-whisper-small"
```
You should see the transcribed text in the response.
### Step 4: Configure Open Notebook
**Via Settings UI (Recommended):**
1. Go to **Settings** → **API Keys**
2. Click **Add Credential** → Select **OpenAI-Compatible**
3. Enter base URL for STT: `http://host.docker.internal:8969/v1` (Docker) or `http://localhost:8969/v1` (local)
4. Click **Save**, then **Test Connection**
**Legacy (Deprecated) — Environment variables:**
```yaml
# In your Open Notebook docker-compose.yml
environment:
- OPENAI_COMPATIBLE_BASE_URL_STT=http://host.docker.internal:8969/v1
```
```bash
# Local development
export OPENAI_COMPATIBLE_BASE_URL_STT=http://localhost:8969/v1
```
### Step 5: Add Model in Open Notebook
1. Go to **Settings** → **Models**
2. Click **Add Model** in Speech-to-Text section
3. Configure:
- **Provider**: `openai_compatible`
- **Model Name**: `Systran/faster-whisper-small`
- **Display Name**: `Local Whisper`
4. Click **Save**
5. Set as default if desired
---
## Available Models
Speaches supports various Whisper model sizes. Larger models are more accurate but slower:
| Model | Size | Speed | Accuracy | VRAM (GPU) |
|-------|------|-------|----------|------------|
| `Systran/faster-whisper-tiny` | ~75 MB | Fastest | Basic | ~1 GB |
| `Systran/faster-whisper-base` | ~150 MB | Fast | Good | ~1 GB |
| `Systran/faster-whisper-small` | ~500 MB | Medium | Better | ~2 GB |
| `Systran/faster-whisper-medium` | ~1.5 GB | Slow | Great | ~5 GB |
| `Systran/faster-whisper-large-v3` | ~3 GB | Slowest | Best | ~10 GB |
| `Systran/faster-distil-whisper-small.en` | ~400 MB | Fast | Good (English only) | ~2 GB |
### List Available Models
```bash
docker compose exec speaches uv tool run speaches-cli registry ls --task automatic-speech-recognition
```
### Recommended Models
- **For speed**: `Systran/faster-whisper-tiny` or `Systran/faster-whisper-base`
- **For balance**: `Systran/faster-whisper-small` (recommended)
- **For accuracy**: `Systran/faster-whisper-large-v3`
---
## GPU Acceleration
For faster transcription with NVIDIA GPUs:
```yaml
services:
speaches:
image: ghcr.io/speaches-ai/speaches:latest-cuda
container_name: speaches
ports:
- "8969:8000"
volumes:
- hf-hub-cache:/home/ubuntu/.cache/huggingface/hub
environment:
- WHISPER__TTL=-1 # Keep model in VRAM (recommended if you have enough memory)
restart: unless-stopped
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
volumes:
hf-hub-cache:
```
### Keep Model in Memory
By default, Speaches unloads models after some time. To keep the Whisper model loaded for instant transcription:
```yaml
environment:
- WHISPER__TTL=-1 # Never unload
```
This is recommended if you have enough RAM/VRAM, as loading the model can take a few seconds.
---
## Docker Networking
When configuring your OpenAI-Compatible credential in **Settings → API Keys**, use the appropriate STT base URL for your setup:
### Open Notebook in Docker (macOS/Windows)
**STT Base URL:** `http://host.docker.internal:8969/v1`
### Open Notebook in Docker (Linux)
**STT Base URL (Option 1 — Docker bridge IP):** `http://172.17.0.1:8969/v1`
**Option 2:** Use host networking mode (`docker run --network host ...`), then use: `http://localhost:8969/v1`
### Remote Server
Run Speaches on a different machine:
**STT Base URL:** `http://server-ip:8969/v1` (replace with your server's IP)
---
## Language Support
Whisper supports 99+ languages. Specify the language for better accuracy:
```bash
curl "http://localhost:8969/v1/audio/transcriptions" \
-F "file=@audio.mp3" \
-F "model=Systran/faster-whisper-small" \
-F "language=ru"
```
Common language codes:
- `en` - English
- `ru` - Russian
- `es` - Spanish
- `fr` - French
- `de` - German
- `zh` - Chinese
- `ja` - Japanese
---
## Troubleshooting
### Service Won't Start
```bash
# Check logs
docker compose logs speaches
# Verify port available
lsof -i :8969
# Restart
docker compose down && docker compose up -d
```
### Connection Refused
```bash
# Test Speaches is running
curl http://localhost:8969/v1/models
# From inside Open Notebook container
docker exec -it open-notebook curl http://host.docker.internal:8969/v1/models
```
### Model Download Fails
Models are downloaded automatically on first use. If download fails:
```bash
# Check available disk space
df -h
# Check Docker logs for errors
docker compose logs speaches
# Restart and try again
docker compose restart speaches
```
### Poor Transcription Quality
- Use a larger model (`faster-whisper-medium` or `large-v3`)
- Specify the correct language
- Ensure audio quality is good (clear speech, minimal background noise)
- Try different audio formats (WAV often works better than MP3)
### Slow Transcription
| Solution | How |
|----------|-----|
| Use GPU | Switch to `latest-cuda` image |
| Smaller model | Use `faster-whisper-tiny` or `base` |
| More CPU | Allocate more cores in Docker |
| SSD storage | Move Docker volumes to SSD |
---
## Performance Tips
### Recommended Specs
| Component | Minimum | Recommended |
|-----------|---------|-------------|
| CPU | 2 cores | 4+ cores |
| RAM | 2 GB | 8+ GB |
| Storage | 5 GB | 10 GB (for multiple models) |
| GPU | None | NVIDIA (optional, much faster) |
### Resource Limits
```yaml
services:
speaches:
# ... other config
mem_limit: 4g
cpus: 2
```
### Monitor Usage
```bash
docker stats speaches
```
---
## Comparison: Local vs Cloud
| Aspect | Local (Speaches) | Cloud (OpenAI Whisper) |
|--------|------------------|------------------------|
| **Cost** | Free | $0.006/min |
| **Privacy** | Complete | Data sent to provider |
| **Speed** | Depends on hardware | Usually faster |
| **Quality** | Excellent (same Whisper) | Excellent |
| **Setup** | Moderate | Simple API key |
| **Offline** | Yes | No |
| **Languages** | 99+ | 99+ |
### When to Use Local
- Privacy-sensitive content
- High-volume transcription
- Development/testing
- Offline environments
- Cost control
### When to Use Cloud
- Limited hardware
- Time-sensitive projects
- No GPU available
- Simple setup preferred
---
## Using Both TTS and STT
Speaches supports both TTS and STT in one server. In **Settings → API Keys**, add a single **OpenAI-Compatible** credential and configure both the TTS and STT base URLs to point to the same Speaches server (e.g., `http://localhost:8969/v1`).
See **[Local TTS Setup](local-tts.md)** for TTS configuration.
---
## Other Local STT Options
Any OpenAI-compatible STT server works:
| Server | Description |
|--------|-------------|
| [Speaches](https://github.com/speaches-ai/speaches) | TTS + STT in one (recommended) |
| [faster-whisper-server](https://github.com/fedirz/faster-whisper-server) | Lightweight STT only |
| [whisper.cpp](https://github.com/ggerganov/whisper.cpp) | C++ implementation with server mode |
| [LocalAI](https://github.com/mudler/LocalAI) | Multi-model local AI server |
The key requirements:
1. Server implements `/v1/audio/transcriptions` endpoint
2. Add an OpenAI-Compatible credential in **Settings → API Keys** with the STT base URL
3. Add model with provider `openai_compatible`
---
## Related
- **[Local TTS Setup](local-tts.md)** - Text-to-speech with Speaches
- **[OpenAI-Compatible Providers](openai-compatible.md)** - General compatible provider setup
- **[AI Providers](ai-providers.md)** - All provider configuration
docs/5-CONFIGURATION/local-tts.md
+344
View File
@@ -0,0 +1,344 @@
# Local Text-to-Speech Setup
Run text-to-speech locally for free, private podcast generation using OpenAI-compatible TTS servers.
---
## Why Local TTS?
| Benefit | Description |
|---------|-------------|
| **Free** | No per-character costs after setup |
| **Private** | Audio never leaves your machine |
| **Unlimited** | No rate limits or quotas |
| **Offline** | Works without internet |
---
## Quick Start with Speaches
[Speaches](https://github.com/speaches-ai/speaches) is an open-source, OpenAI-compatible TTS server.
> **💡 Ready-made Docker Compose files available:**
> - **[docker-compose-speaches.yml](../../examples/docker-compose-speaches.yml)** - Speaches + Open Notebook
> - **[docker-compose-full-local.yml](../../examples/docker-compose-full-local.yml)** - Speaches + Ollama (100% local setup)
>
> These include complete setup instructions and configuration examples. Just copy and run!
### Step 1: Create Docker Compose File
Create a folder and add `docker-compose.yml`:
```yaml
services:
speaches:
image: ghcr.io/speaches-ai/speaches:latest-cpu
container_name: speaches
ports:
- "8969:8000"
volumes:
- hf-hub-cache:/home/ubuntu/.cache/huggingface/hub
restart: unless-stopped
volumes:
hf-hub-cache:
```
### Step 2: Start and Download Model
```bash
# Start Speaches
docker compose up -d
# Wait for startup
sleep 10
# Download voice model (~500MB)
docker compose exec speaches uv tool run speaches-cli model download speaches-ai/Kokoro-82M-v1.0-ONNX
```
### Step 3: Test
```bash
curl "http://localhost:8969/v1/audio/speech" -s \
-H "Content-Type: application/json" \
--output test.mp3 \
--data '{
"input": "Hello! Local TTS is working.",
"model": "speaches-ai/Kokoro-82M-v1.0-ONNX",
"voice": "af_bella"
}'
```
Play `test.mp3` to verify.
### Step 4: Configure Open Notebook
**Via Settings UI (Recommended):**
1. Go to **Settings** → **API Keys**
2. Click **Add Credential** → Select **OpenAI-Compatible**
3. Enter base URL for TTS: `http://host.docker.internal:8969/v1` (Docker) or `http://localhost:8969/v1` (local)
4. Click **Save**, then **Test Connection**
**Legacy (Deprecated) — Environment variables:**
```yaml
# In your Open Notebook docker-compose.yml
environment:
- OPENAI_COMPATIBLE_BASE_URL_TTS=http://host.docker.internal:8969/v1
```
```bash
# Local development
export OPENAI_COMPATIBLE_BASE_URL_TTS=http://localhost:8969/v1
```
### Step 5: Add Model in Open Notebook
1. Go to **Settings** → **Models**
2. Click **Add Model** in Text-to-Speech section
3. Configure:
- **Provider**: `openai_compatible`
- **Model Name**: `speaches-ai/Kokoro-82M-v1.0-ONNX`
- **Display Name**: `Local TTS`
4. Click **Save**
5. Set as default if desired
---
## Available Voices
The Kokoro model includes multiple voices:
### Female Voices
| Voice ID | Description |
|----------|-------------|
| `af_bella` | Clear, professional |
| `af_sarah` | Warm, friendly |
| `af_nicole` | Energetic, expressive |
### Male Voices
| Voice ID | Description |
|----------|-------------|
| `am_adam` | Deep, authoritative |
| `am_michael` | Friendly, conversational |
### British Accents
| Voice ID | Description |
|----------|-------------|
| `bf_emma` | British female, professional |
| `bm_george` | British male, formal |
### Test Different Voices
```bash
for voice in af_bella af_sarah am_adam am_michael; do
curl "http://localhost:8969/v1/audio/speech" -s \
-H "Content-Type: application/json" \
--output "test_${voice}.mp3" \
--data "{
\"input\": \"Hello, this is the ${voice} voice.\",
\"model\": \"speaches-ai/Kokoro-82M-v1.0-ONNX\",
\"voice\": \"${voice}\"
}"
done
```
---
## GPU Acceleration
For faster generation with NVIDIA GPUs:
```yaml
services:
speaches:
image: ghcr.io/speaches-ai/speaches:latest-cuda
container_name: speaches
ports:
- "8969:8000"
volumes:
- hf-hub-cache:/home/ubuntu/.cache/huggingface/hub
restart: unless-stopped
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
volumes:
hf-hub-cache:
```
---
## Docker Networking
When configuring your OpenAI-Compatible credential in **Settings → API Keys**, use the appropriate TTS base URL for your setup:
### Open Notebook in Docker (macOS/Windows)
**TTS Base URL:** `http://host.docker.internal:8969/v1`
### Open Notebook in Docker (Linux)
**TTS Base URL (Option 1 — Docker bridge IP):** `http://172.17.0.1:8969/v1`
**Option 2:** Use host networking mode (`docker run --network host ...`), then use: `http://localhost:8969/v1`
### Remote Server
Run Speaches on a different machine:
**TTS Base URL:** `http://server-ip:8969/v1` (replace with your server's IP)
---
## Multi-Speaker Podcasts
Configure different voices for each speaker:
```
Speaker 1 (Host):
Model: speaches-ai/Kokoro-82M-v1.0-ONNX
Voice: af_bella
Speaker 2 (Guest):
Model: speaches-ai/Kokoro-82M-v1.0-ONNX
Voice: am_adam
Speaker 3 (Narrator):
Model: speaches-ai/Kokoro-82M-v1.0-ONNX
Voice: bf_emma
```
---
## Troubleshooting
### Service Won't Start
```bash
# Check logs
docker compose logs speaches
# Verify port available
lsof -i :8969
# Restart
docker compose down && docker compose up -d
```
### Connection Refused
```bash
# Test Speaches is running
curl http://localhost:8969/v1/models
# From inside Open Notebook container
docker exec -it open-notebook curl http://host.docker.internal:8969/v1/models
```
### Model Not Found
```bash
# List downloaded models
docker compose exec speaches uv tool run speaches-cli model list
# Download if missing
docker compose exec speaches uv tool run speaches-cli model download speaches-ai/Kokoro-82M-v1.0-ONNX
```
### Poor Audio Quality
- Try different voices
- Adjust speed: `"speed": 0.9` to `1.2`
- Check model downloaded completely
- Allocate more memory
### Slow Generation
| Solution | How |
|----------|-----|
| Use GPU | Switch to `latest-cuda` image |
| More CPU | Allocate more cores in Docker |
| Faster model | Use smaller/quantized models |
| SSD storage | Move Docker volumes to SSD |
---
## Performance Tips
### Recommended Specs
| Component | Minimum | Recommended |
|-----------|---------|-------------|
| CPU | 2 cores | 4+ cores |
| RAM | 2 GB | 4+ GB |
| Storage | 5 GB | 10 GB (for multiple models) |
| GPU | None | NVIDIA (optional) |
### Resource Limits
```yaml
services:
speaches:
# ... other config
mem_limit: 4g
cpus: 2
```
### Monitor Usage
```bash
docker stats speaches
```
---
## Comparison: Local vs Cloud
| Aspect | Local (Speaches) | Cloud (OpenAI/ElevenLabs) |
|--------|------------------|---------------------------|
| **Cost** | Free | $0.015-0.10/min |
| **Privacy** | Complete | Data sent to provider |
| **Speed** | Depends on hardware | Usually faster |
| **Quality** | Good | Excellent |
| **Setup** | Moderate | Simple API key |
| **Offline** | Yes | No |
| **Voices** | Limited | Many options |
### When to Use Local
- Privacy-sensitive content
- High-volume generation
- Development/testing
- Offline environments
- Cost control
### When to Use Cloud
- Premium quality needs
- Multiple languages
- Time-sensitive projects
- Limited hardware
---
## Other Local TTS Options
Any OpenAI-compatible TTS server works. The key is:
1. Server implements `/v1/audio/speech` endpoint
2. Add an OpenAI-Compatible credential in **Settings → API Keys** with the TTS base URL
3. Add model with provider `openai_compatible`
---
## Related
- **[Local STT Setup](local-stt.md)** - Speech-to-text with Speaches
- **[OpenAI-Compatible Providers](openai-compatible.md)** - General compatible provider setup
- **[AI Providers](ai-providers.md)** - All provider configuration
- **[Creating Podcasts](../3-USER-GUIDE/creating-podcasts.md)** - Using TTS for podcasts
docs/5-CONFIGURATION/mcp-integration.md
+199
View File
@@ -0,0 +1,199 @@
# Model Context Protocol (MCP) Integration
Open Notebook can be seamlessly integrated into your AI workflows using the **Model Context Protocol (MCP)**, enabling direct access to your notebooks, sources, and chat functionality from AI assistants like Claude Desktop and VS Code extensions.
## What is MCP?
The [Model Context Protocol](https://modelcontextprotocol.io) is an open standard that allows AI applications to securely connect to external data sources and tools. With the Open Notebook MCP server, you can:
- 📚 **Access your notebooks** directly from Claude Desktop or VS Code
- 🔍 **Search your research content** without leaving your AI assistant
- 💬 **Create and manage chat sessions** with your research as context
- 📝 **Generate notes** and insights on-the-fly
- 🤖 **Automate workflows** using the full Open Notebook API
## Quick Setup
### For Claude Desktop
1. **Install the MCP server** (automatically from PyPI):
```bash
# No manual installation needed! Claude Desktop will use uvx to run it automatically
```
2. **Configure Claude Desktop**:
**macOS/Linux**: Edit `~/Library/Application Support/Claude/claude_desktop_config.json`
```json
{
"mcpServers": {
"open-notebook": {
"command": "uvx",
"args": ["open-notebook-mcp"],
"env": {
"OPEN_NOTEBOOK_URL": "http://localhost:5055",
"OPEN_NOTEBOOK_PASSWORD": "your_password_here"
}
}
}
}
```
**Windows**: Edit `%APPDATA%\Claude\claude_desktop_config.json`
```json
{
"mcpServers": {
"open-notebook": {
"command": "uvx",
"args": ["open-notebook-mcp"],
"env": {
"OPEN_NOTEBOOK_URL": "http://localhost:5055",
"OPEN_NOTEBOOK_PASSWORD": "your_password_here"
}
}
}
}
```
3. **Restart Claude Desktop** and start using your notebooks in conversations!
### For VS Code (Cline and other MCP-compatible extensions)
Add to your VS Code settings or `.vscode/mcp.json`:
```json
{
"servers": {
"open-notebook": {
"command": "uvx",
"args": ["open-notebook-mcp"],
"env": {
"OPEN_NOTEBOOK_URL": "http://localhost:5055",
"OPEN_NOTEBOOK_PASSWORD": "your_password_here"
}
}
}
}
```
## Configuration
- **OPEN_NOTEBOOK_URL**: URL to your Open Notebook API (default: `http://localhost:5055`)
- **OPEN_NOTEBOOK_PASSWORD**: Optional - only needed if you've enabled password protection
### For Remote Servers
If your Open Notebook instance is running on a remote server, update the URL accordingly:
```json
"OPEN_NOTEBOOK_URL": "http://192.168.1.100:5055"
```
Or with a domain:
```json
"OPEN_NOTEBOOK_URL": "https://notebook.yourdomain.com/api"
```
## What You Can Do
Once connected, you can ask Claude or your AI assistant to:
- _"Search my research notebooks for information about [topic]"_
- _"Create a new note summarizing the key points from our conversation"_
- _"List all my notebooks"_
- _"Start a chat session about [specific source or topic]"_
- _"What sources do I have in my [notebook name] notebook?"_
- _"Add this PDF to my research notebook"_
- _"Show me all notes in [notebook name]"_
The MCP server provides full access to Open Notebook's capabilities, allowing you to manage your research seamlessly from within your AI assistant.
## Available Tools
The Open Notebook MCP server exposes these capabilities:
### Notebooks
- List notebooks
- Get notebook details
- Create new notebooks
- Update notebook information
- Delete notebooks
### Sources
- List sources in a notebook
- Get source details
- Add new sources (links, files, text)
- Update source metadata
- Delete sources
### Notes
- List notes in a notebook
- Get note details
- Create new notes
- Update notes
- Delete notes
### Chat
- Create chat sessions
- Send messages to chat sessions
- Get chat history
- List chat sessions
### Search
- Vector search across content
- Text search across content
- Filter by notebook
### Models
- List configured AI models
- Get model details
- Create model configurations
- Update model settings
### Settings
- Get application settings
- Update settings
## MCP Server Repository
The Open Notebook MCP server is developed and maintained by the Epochal team:
**🔗 GitHub**: [Epochal-dev/open-notebook-mcp](https://github.com/Epochal-dev/open-notebook-mcp)
Contributions, issues, and feature requests are welcome!
## Finding the Server
The Open Notebook MCP server is published to the official MCP Registry:
- **Registry**: Search for "open-notebook" at [registry.modelcontextprotocol.io](https://registry.modelcontextprotocol.io)
- **PyPI**: [pypi.org/project/open-notebook-mcp](https://pypi.org/project/open-notebook-mcp)
- **GitHub**: [Epochal-dev/open-notebook-mcp](https://github.com/Epochal-dev/open-notebook-mcp)
## Troubleshooting
### Connection Errors
1. Verify the `OPEN_NOTEBOOK_URL` is correct and accessible
2. If using password protection, ensure `OPEN_NOTEBOOK_PASSWORD` is set correctly
3. For remote servers, make sure port 5055 is accessible from your machine
4. Check firewall settings if connecting to a remote server
## Using with Other MCP Clients
The Open Notebook MCP server follows the standard MCP protocol and can be used with any MCP-compatible client. Check your client's documentation for configuration details.
## Learn More
- [Model Context Protocol Documentation](https://modelcontextprotocol.io)
docs/5-CONFIGURATION/ollama.md
+744
View File
@@ -0,0 +1,744 @@
# Ollama Setup Guide
Ollama provides free, local AI models that run on your own hardware. This guide covers everything you need to know about setting up Ollama with Open Notebook, including different deployment scenarios and network configurations.
## Why Choose Ollama?
- **🆓 Completely Free**: No API costs after initial setup
- **🔒 Full Privacy**: Your data never leaves your local network
- **📱 Offline Capable**: Works without internet connection
- **🚀 Fast**: Local inference with no network latency
- **🧠 Reasoning Models**: Support for advanced reasoning models like DeepSeek-R1
- **💾 Model Variety**: Access to hundreds of open-source models
## Quick Start
### 1. Install Ollama
**Linux/macOS:**
```bash
curl -fsSL https://ollama.ai/install.sh | sh
```
**Windows:**
Download and install from [ollama.ai](https://ollama.ai/download)
### 2. Pull Required Models
```bash
# Language models (choose one or more)
ollama pull qwen3 # Excellent general purpose, 7B parameters
ollama pull gemma3 # Google's model, good performance
ollama pull deepseek-r1 # Advanced reasoning model
ollama pull phi4 # Microsoft's efficient model
# Embedding model (required for search)
ollama pull mxbai-embed-large # Best embedding model for Ollama
```
### 3. Configure Open Notebook
**Via Settings UI (Recommended):**
1. Go to **Settings** → **API Keys**
2. Click **Add Credential** → Select **Ollama**
3. Enter the base URL (see [Network Configuration](#network-configuration-guide) below for correct URL)
4. Click **Save**, then **Test Connection**
5. Click **Discover Models** → **Register Models**
**Legacy (Deprecated) — Environment variables:**
```bash
# For local installation:
export OLLAMA_API_BASE=http://localhost:11434
# For Docker installation:
export OLLAMA_API_BASE=http://host.docker.internal:11434
```
> **Note**: The `OLLAMA_API_BASE` environment variable is deprecated. Configure Ollama via Settings → API Keys instead.
## Network Configuration Guide
When adding an Ollama credential in **Settings → API Keys**, you need to enter the correct base URL. The correct URL depends on your deployment scenario:
### Scenario 1: Local Installation (Same Machine)
When both Open Notebook and Ollama run directly on your machine:
**Base URL to enter in Settings → API Keys:** `http://localhost:11434`
Alternative: `http://127.0.0.1:11434` (use if you have DNS resolution issues with localhost)
### Scenario 2: Open Notebook in Docker, Ollama on Host
When Open Notebook runs in Docker but Ollama runs on your host machine:
**Base URL to enter in Settings → API Keys:** `http://host.docker.internal:11434`
**⚠️ CRITICAL: Ollama must accept external connections:**
```bash
# Start Ollama with external access enabled
export OLLAMA_HOST=0.0.0.0:11434
ollama serve
```
**⚠️ LINUX USERS: Extra configuration required!**
On Linux, `host.docker.internal` doesn't resolve automatically like it does on macOS/Windows. You must add `extra_hosts` to your docker-compose.yml:
```yaml
# Add to your docker-compose.yml (requires surrealdb service, see installation guide)
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest
# ... other settings ...
extra_hosts:
- "host.docker.internal:host-gateway"
```
Without this, you'll get connection errors like:
```
httpcore.ConnectError: [Errno -2] Name or service not known
```
**Why `host.docker.internal`?**
- Docker containers can't reach `localhost` on the host
- `host.docker.internal` is Docker's special hostname for the host machine
- Available on Docker Desktop for Mac/Windows; **requires `extra_hosts` on Linux**
**Why `OLLAMA_HOST=0.0.0.0:11434`?**
- By default, Ollama only binds to localhost and rejects external connections
- Docker containers are considered "external" even when running on the same machine
- Setting `OLLAMA_HOST=0.0.0.0:11434` allows connections from Docker containers
### Scenario 3: Both in Docker (Same Compose)
When both Open Notebook and Ollama run in the same Docker Compose stack:
**Base URL to enter in Settings → API Keys:** `http://ollama:11434`
**Docker Compose Example:**
```yaml
# Requires surrealdb service — see full base setup:
# https://github.com/lfnovo/open-notebook/blob/main/docker-compose.yml
services:
open-notebook:
image: lfnovo/open_notebook:v1-latest
pull_policy: always
ports:
- "8502:8502"
- "5055:5055"
environment:
- OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
volumes:
- ./notebook_data:/app/data
depends_on:
- ollama
ollama:
image: ollama/ollama:latest
ports:
- "11434:11434"
volumes:
- ollama_data:/root/.ollama
# Optional: GPU support
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
volumes:
ollama_data:
```
### Scenario 4: Remote Ollama Server
When Ollama runs on a different machine in your network:
**Base URL to enter in Settings → API Keys:** `http://192.168.1.100:11434` (replace with your Ollama server's IP)
**Security Note:** Only use this in trusted networks. Ollama doesn't have built-in authentication.
### Scenario 5: Ollama with Custom Port
If you've configured Ollama to use a different port:
```bash
# Start Ollama on custom port
OLLAMA_HOST=0.0.0.0:8080 ollama serve
```
**Base URL to enter in Settings → API Keys:** `http://localhost:8080`
## Model Recommendations
### Language Models
| Model | Size | Best For | Quality | Speed |
|-------|------|----------|---------|-------|
| **qwen3** | 7B | General purpose, coding | Excellent | Fast |
| **deepseek-r1** | 7B | Reasoning, problem-solving | Exceptional | Medium |
| **gemma3** | 7B | Balanced performance | Very Good | Fast |
| **phi4** | 14B | Efficiency on small hardware | Good | Very Fast |
| **llama3** | 8B | General purpose | Very Good | Medium |
### Embedding Models
| Model | Best For | Performance |
|-------|----------|-------------|
| **mxbai-embed-large** | General search | Excellent |
| **nomic-embed-text** | Document similarity | Good |
| **all-minilm** | Lightweight option | Fair |
### Installation Commands
```bash
# Essential models
ollama pull qwen3 # Primary language model
ollama pull mxbai-embed-large # Search embeddings
# Optional reasoning model
ollama pull deepseek-r1 # Advanced reasoning
# Alternative language models
ollama pull gemma3 # Google's model
ollama pull phi4 # Microsoft's efficient model
```
## Hardware Requirements
### Minimum Requirements
- **RAM**: 8GB (for 7B models)
- **Storage**: 10GB free space per model
- **CPU**: Modern multi-core processor
### Recommended Setup
- **RAM**: 16GB+ (for multiple models)
- **Storage**: SSD with 50GB+ free space
- **GPU**: NVIDIA GPU with 8GB+ VRAM (optional but faster)
### GPU Acceleration
**NVIDIA GPU (CUDA):**
```bash
# Install NVIDIA Container Toolkit for Docker
# Then use the Docker Compose example above with GPU support
# For local installation, Ollama auto-detects CUDA
ollama pull qwen3
```
**Apple Silicon (M1/M2/M3):**
```bash
# Ollama automatically uses Metal acceleration
# No additional setup required
ollama pull qwen3
```
**AMD GPUs:**
```bash
# ROCm support varies by model and system
# Check Ollama documentation for latest compatibility
```
## Troubleshooting
### Model Name Configuration (Critical)
**⚠️ IMPORTANT: Model names must exactly match the output of `ollama list`**
This is the most common cause of "Failed to send message" errors. Open Notebook requires the **exact model name** as it appears in Ollama.
**Step 1: Get the exact model name**
```bash
ollama list
```
Example output:
```
NAME ID SIZE MODIFIED
mxbai-embed-large:latest 468836162de7 669 MB 7 minutes ago
gemma3:12b f4031aab637d 8.1 GB 2 months ago
qwen3:32b 030ee887880f 20 GB 9 days ago
```
**Step 2: Use the exact name when adding the model in Open Notebook**
| ✅ Correct | ❌ Wrong |
|-----------|----------|
| `gemma3:12b` | `gemma3` (missing tag) |
| `qwen3:32b` | `qwen3-32b` (wrong format) |
| `mxbai-embed-large:latest` | `mxbai-embed-large` (missing tag) |
**Note:** Some models use `:latest` as the default tag. If `ollama list` shows `model:latest`, you must use `model:latest` in Open Notebook, not just `model`.
**Step 3: Configure in Open Notebook**
1. Go to **Settings → Models**
2. Click **Add Model**
3. Enter the **exact name** from `ollama list`
4. Select provider: `ollama`
5. Select type: `language` (for chat) or `embedding` (for search)
6. Save the model
7. Set it as the default for the appropriate task (chat, transformation, etc.)
### Common Issues
**1. "Ollama unavailable" in Open Notebook**
**Check Ollama is running:**
```bash
curl http://localhost:11434/api/tags
```
**Verify credential is configured:**
Check **Settings → API Keys** for an Ollama credential with the correct base URL.
**⚠️ IMPORTANT: Enable external connections (most common fix):**
```bash
# If Open Notebook runs in Docker or on a different machine,
# Ollama must bind to all interfaces, not just localhost
export OLLAMA_HOST=0.0.0.0:11434
ollama serve
```
> **Why this is needed:** By default, Ollama only accepts connections from `localhost` (127.0.0.1). When Open Notebook runs in Docker or on a different machine, it can't reach Ollama unless you configure `OLLAMA_HOST=0.0.0.0:11434` to accept external connections.
**Restart Ollama:**
```bash
# Linux/macOS
sudo systemctl restart ollama
# or
ollama serve
# Windows
# Restart from system tray or Services
```
**2. Docker networking issues**
**From inside Open Notebook container, test Ollama:**
```bash
# Get into container
docker exec -it open-notebook bash
# Test connection
curl http://host.docker.internal:11434/api/tags
```
**If this fails on Linux** with "Name or service not known", you need to add `extra_hosts` to your docker-compose.yml. See the [Docker-Specific Troubleshooting](#docker-specific-troubleshooting) section below.
**3. Models not downloading**
**Check disk space:**
```bash
df -h
```
**Manual model pull:**
```bash
ollama pull qwen3 --verbose
```
**Clear failed downloads:**
```bash
ollama rm qwen3
ollama pull qwen3
```
**4. Slow performance**
**Check model size vs available RAM:**
```bash
ollama ps # Show running models
free -h # Check available memory
```
**Use smaller models:**
```bash
ollama pull phi4 # Instead of larger models
ollama pull gemma3:2b # 2B parameter variant
```
**5. Port conflicts**
**Check what's using port 11434:**
```bash
lsof -i :11434
netstat -tulpn | grep 11434
```
**Use custom port:**
```bash
OLLAMA_HOST=0.0.0.0:8080 ollama serve
```
Then update the base URL in **Settings → API Keys** to `http://localhost:8080`
**6. "Failed to send message" in Chat**
**Symptom:** Chat shows "Failed to send message" toast notification. Logs may show:
```
Error executing chat: Model is not a LanguageModel: None
```
**Causes (in order of likelihood):**
1. **Model name mismatch**: The model name in Open Notebook doesn't exactly match `ollama list`
2. **No default model configured**: You haven't set a default chat model in Settings → Models
3. **Model was deleted**: You removed the model from Ollama but didn't update Open Notebook's defaults
4. **Model record deleted**: The model was removed from Open Notebook but is still set as default
**Solutions:**
**Check 1: Verify model names match exactly**
```bash
# Get exact model names from Ollama
ollama list
# Compare with what's configured in Open Notebook
# Go to Settings → Models and verify the names match EXACTLY
```
**Check 2: Verify default models are set**
1. Go to **Settings → Models**
2. Scroll to **Default Models** section
3. Ensure **Default Chat Model** has a value selected
4. If empty, select an available language model
**Check 3: Refresh after changes**
If you've added/removed models in Ollama:
1. Refresh the Open Notebook page
2. Go to Settings → Models
3. Re-add any missing models with exact names from `ollama list`
4. Re-select default models if needed
**Check 4: Test the model directly**
```bash
# Verify Ollama can use the model
ollama run gemma3:12b "Hello, world"
```
### Docker-Specific Troubleshooting
**1. Linux: `host.docker.internal` not resolving (Most Common)**
If you see `Name or service not known` errors on Linux, add `extra_hosts` to your docker-compose.yml:
```yaml
# Add to your docker-compose.yml (requires surrealdb service, see installation guide)
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest
extra_hosts:
- "host.docker.internal:host-gateway"
environment:
# ... rest of your config
```
Then in **Settings → API Keys**, use base URL: `http://host.docker.internal:11434`
This maps `host.docker.internal` to your host machine's IP. macOS/Windows Docker Desktop does this automatically, but Linux requires explicit configuration.
**2. Host networking on Linux (alternative):**
```bash
# Use host networking if host.docker.internal doesn't work
docker run --network host lfnovo/open_notebook:v1-latest # for quick testing only
```
Then in **Settings → API Keys**, use base URL: `http://localhost:11434`
**3. Custom bridge network:**
```yaml
version: '3.8'
networks:
ollama_network:
driver: bridge
services:
open-notebook:
networks:
- ollama_network
environment:
ollama:
networks:
- ollama_network
```
Then in **Settings → API Keys**, use base URL: `http://ollama:11434`
**4. Firewall issues:**
```bash
# Allow Ollama port through firewall
sudo ufw allow 11434
# or
sudo firewall-cmd --add-port=11434/tcp --permanent
```
## Performance Optimization
### Model Management
**List installed models:**
```bash
ollama list
```
**Remove unused models:**
```bash
ollama rm model_name
```
**Show running models:**
```bash
ollama ps
```
**Preload models for faster startup:**
```bash
# Keep model in memory
curl http://localhost:11434/api/generate -d '{
"model": "qwen3",
"prompt": "test",
"keep_alive": -1
}'
```
### System Optimization
**Linux: Increase file limits:**
```bash
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
```
**macOS: Increase memory limits:**
```bash
# Add to ~/.zshrc or ~/.bash_profile
export OLLAMA_MAX_LOADED_MODELS=2
export OLLAMA_NUM_PARALLEL=4
```
**Docker: Resource allocation:**
```yaml
services:
ollama:
deploy:
resources:
limits:
memory: 8G
cpus: '4'
```
## Advanced Configuration
### Environment Variables
```bash
# Ollama server configuration
export OLLAMA_HOST=0.0.0.0:11434 # Bind to all interfaces
export OLLAMA_KEEP_ALIVE=5m # Keep models in memory
export OLLAMA_MAX_LOADED_MODELS=3 # Max concurrent models
export OLLAMA_MAX_QUEUE=512 # Request queue size
export OLLAMA_NUM_PARALLEL=4 # Parallel request handling
export OLLAMA_FLASH_ATTENTION=1 # Enable flash attention (if supported)
# Open Notebook configuration (configure via Settings → API Keys instead)
# OLLAMA_API_BASE=http://localhost:11434 # Deprecated — use Settings UI
```
### SSL Configuration (Self-Signed Certificates)
If you're running Ollama behind a reverse proxy with self-signed SSL certificates (e.g., Caddy, nginx with custom certs), you may encounter SSL verification errors:
```
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate
```
**Solutions:**
**Option 1: Use a custom CA bundle (recommended)**
```bash
# Point to your CA certificate file
export ESPERANTO_SSL_CA_BUNDLE=/path/to/your/ca-bundle.pem
```
**Option 2: Disable SSL verification (development only)**
```bash
# WARNING: Only use in trusted development environments
export ESPERANTO_SSL_VERIFY=false
```
**Docker Compose example with SSL configuration:**
```yaml
# Add to your docker-compose.yml (requires surrealdb service, see installation guide)
services:
open-notebook:
image: lfnovo/open_notebook:v1-latest
pull_policy: always
environment:
- OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
# Option 1: Custom CA bundle (if Ollama uses self-signed SSL)
- ESPERANTO_SSL_CA_BUNDLE=/certs/ca-bundle.pem
# Option 2: Disable verification (dev only)
# - ESPERANTO_SSL_VERIFY=false
volumes:
- /path/to/your/ca-bundle.pem:/certs/ca-bundle.pem:ro
```
> **Security Note:** Disabling SSL verification exposes you to man-in-the-middle attacks. Always prefer using a custom CA bundle in production environments.
### Custom Model Imports
**Import custom models:**
```bash
# Create Modelfile
cat > Modelfile << EOF
FROM qwen3
PARAMETER temperature 0.7
PARAMETER top_p 0.9
SYSTEM "You are a helpful research assistant."
EOF
# Create custom model
ollama create my-research-model -f Modelfile
```
**Use in Open Notebook:**
1. Go to Models
2. Add new model: `my-research-model`
3. Set as default for specific tasks
### Monitoring and Logging
**Monitor Ollama logs:**
```bash
# Linux (systemd)
journalctl -u ollama -f
# Docker
docker logs -f ollama
# Manual run with verbose logging
OLLAMA_DEBUG=1 ollama serve
```
**Resource monitoring:**
```bash
# CPU and memory usage
htop
# GPU usage (NVIDIA)
nvidia-smi -l 1
# Model-specific metrics
ollama ps
```
## Integration Examples
### Python Script Integration
```python
import requests
import os
# Test Ollama connection
ollama_base = os.environ.get('OLLAMA_API_BASE', 'http://localhost:11434')
response = requests.get(f'{ollama_base}/api/tags')
print(f"Available models: {response.json()}")
# Generate text
payload = {
"model": "qwen3",
"prompt": "Explain quantum computing",
"stream": False
}
response = requests.post(f'{ollama_base}/api/generate', json=payload)
print(response.json()['response'])
```
### Health Check Script
```bash
#!/bin/bash
# ollama-health-check.sh
OLLAMA_API_BASE=${OLLAMA_API_BASE:-"http://localhost:11434"}
echo "Checking Ollama health..."
if curl -s "${OLLAMA_API_BASE}/api/tags" > /dev/null; then
echo "✅ Ollama is running"
echo "Available models:"
curl -s "${OLLAMA_API_BASE}/api/tags" | jq -r '.models[].name'
else
echo "❌ Ollama is not accessible at ${OLLAMA_API_BASE}"
exit 1
fi
```
## Migration from Other Providers
### Coming from OpenAI
**Similar performance models:**
- GPT-4 → `qwen3` or `deepseek-r1`
- GPT-3.5 → `gemma3` or `phi4`
- text-embedding-ada-002 → `mxbai-embed-large`
**Cost comparison:**
- OpenAI: $0.01-0.06 per 1K tokens
- Ollama: $0 after hardware investment
### Coming from Anthropic
**Claude replacement suggestions:**
- Claude 3.5 Sonnet → `deepseek-r1` (reasoning)
- Claude 3 Haiku → `phi4` (speed)
## Best Practices
### Security
1. **Network Security:**
- Run Ollama only on trusted networks
- Use firewall rules to limit access
- Consider VPN for remote access
2. **Model Verification:**
- Only pull models from trusted sources
- Verify model checksums when possible
3. **Resource Limits:**
- Set memory and CPU limits in production
- Monitor resource usage regularly
### Performance
1. **Model Selection:**
- Use appropriate model size for your hardware
- Smaller models for simple tasks
- Reasoning models only when needed
2. **Resource Management:**
- Preload frequently used models
- Remove unused models regularly
- Monitor system resources
3. **Network Optimization:**
- Use local networks for better latency
- Consider SSD storage for faster model loading
## Getting Help
**Community Resources:**
- [Ollama GitHub](https://github.com/jmorganca/ollama) - Official repository
- [Ollama Discord](https://discord.gg/ollama) - Community support
- [Open Notebook Discord](https://discord.gg/37XJPXfz2w) - Integration help
**Debugging Resources:**
- Check Ollama logs for error messages
- Test connection with curl commands
- Verify environment variables
- Monitor system resources
This comprehensive guide should help you successfully deploy and optimize Ollama with Open Notebook. Start with the Quick Start section and refer to specific scenarios as needed.
docs/5-CONFIGURATION/openai-compatible.md
+405
View File
@@ -0,0 +1,405 @@
# OpenAI-Compatible Providers
Use any server that implements the OpenAI API format with Open Notebook. This includes LM Studio, Text Generation WebUI, vLLM, and many others.
---
## What is OpenAI-Compatible?
Many AI tools implement the same API format as OpenAI:
```
POST /v1/chat/completions
POST /v1/embeddings
POST /v1/audio/speech
```
Open Notebook can connect to any server using this format.
---
## Common Compatible Servers
| Server | Use Case | URL |
|--------|----------|-----|
| **LM Studio** | Desktop GUI for local models | https://lmstudio.ai |
| **Text Generation WebUI** | Full-featured local inference | https://github.com/oobabooga/text-generation-webui |
| **vLLM** | High-performance serving | https://github.com/vllm-project/vllm |
| **Ollama** | Simple local models | (Use native Ollama provider instead) |
| **LocalAI** | Local AI inference | https://github.com/mudler/LocalAI |
| **llama.cpp server** | Lightweight inference | https://github.com/ggerganov/llama.cpp |
---
## Quick Setup: LM Studio
### Step 1: Install and Start LM Studio
1. Download from https://lmstudio.ai
2. Install and launch
3. Download a model (e.g., Llama 3)
4. Start the local server (default: port 1234)
### Step 2: Configure in Settings UI (Recommended)
1. Go to **Settings** → **API Keys**
2. Click **Add Credential** → Select **OpenAI-Compatible**
3. Enter base URL: `http://host.docker.internal:1234/v1` (Docker) or `http://localhost:1234/v1` (local)
4. API key: `lm-studio` (placeholder, LM Studio doesn't require one)
5. Click **Save**, then **Test Connection**
**Legacy (Deprecated) — Environment variables:**
```bash
export OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1
export OPENAI_COMPATIBLE_API_KEY=not-needed
```
### Step 3: Add Model in Open Notebook
1. Go to **Settings** → **Models**
2. Click **Add Model**
3. Configure:
- **Provider**: `openai_compatible`
- **Model Name**: Your model name from LM Studio
- **Display Name**: `LM Studio - Llama 3`
4. Click **Save**
---
## Configuration via Settings UI
The recommended way to configure OpenAI-compatible providers is through the Settings UI:
1. Go to **Settings** → **API Keys**
2. Click **Add Credential** → Select **OpenAI-Compatible**
3. Enter your base URL and API key (if needed)
4. Optionally configure per-service URLs for LLM, Embedding, TTS, and STT
5. Click **Save**, then **Test Connection**
## Legacy: Environment Variables (Deprecated)
> **Deprecated**: These environment variables are deprecated. Use the Settings UI instead.
### Language Models (Chat)
```bash
OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1
OPENAI_COMPATIBLE_API_KEY=optional-api-key
```
### Embeddings
```bash
OPENAI_COMPATIBLE_BASE_URL_EMBEDDING=http://localhost:1234/v1
OPENAI_COMPATIBLE_API_KEY_EMBEDDING=optional-api-key
```
### Text-to-Speech
```bash
OPENAI_COMPATIBLE_BASE_URL_TTS=http://localhost:8969/v1
OPENAI_COMPATIBLE_API_KEY_TTS=optional-api-key
```
### Speech-to-Text
```bash
OPENAI_COMPATIBLE_BASE_URL_STT=http://localhost:9000/v1
OPENAI_COMPATIBLE_API_KEY_STT=optional-api-key
```
---
## Docker Networking
When Open Notebook runs in Docker and your compatible server runs on the host, use the appropriate base URL when adding your credential in **Settings → API Keys**:
### macOS / Windows
**Base URL:** `http://host.docker.internal:1234/v1`
### Linux
**Base URL (Option 1 — Docker bridge IP):** `http://172.17.0.1:1234/v1`
**Option 2:** Use host networking mode: `docker run --network host ...`
Then use base URL: `http://localhost:1234/v1`
### Same Docker Network
```yaml
# docker-compose.yml
services:
open-notebook:
# ...
lm-studio:
# your LM Studio container
ports:
- "1234:1234"
```
**Base URL in Settings → API Keys:** `http://lm-studio:1234/v1`
---
## Text Generation WebUI Setup
### Start with API Enabled
```bash
python server.py --api --listen
```
### Configure Open Notebook
In **Settings → API Keys**, add an **OpenAI-Compatible** credential with base URL: `http://localhost:5000/v1`
### Docker Compose Example
```yaml
# Add to your docker-compose.yml (requires surrealdb service, see installation guide)
services:
text-gen:
image: atinoda/text-generation-webui:default
ports:
- "5000:5000"
- "7860:7860"
volumes:
- ./models:/app/models
command: --api --listen
open-notebook:
image: lfnovo/open_notebook:v1-latest
pull_policy: always
depends_on:
- text-gen
```
Then in **Settings → API Keys**, add an **OpenAI-Compatible** credential with base URL: `http://text-gen:5000/v1`
---
## vLLM Setup
### Start vLLM Server
```bash
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-8B-Instruct \
--port 8000
```
### Configure Open Notebook
In **Settings → API Keys**, add an **OpenAI-Compatible** credential with base URL: `http://localhost:8000/v1`
### Docker Compose with GPU
```yaml
# Add to your docker-compose.yml (requires surrealdb service, see installation guide)
services:
vllm:
image: vllm/vllm-openai:latest
command: --model meta-llama/Llama-3.1-8B-Instruct
ports:
# Localhost only (vLLM has no authentication by default), on host port
# 8001 because SurrealDB already publishes 8000. Open Notebook reaches
# vLLM over the compose network at http://vllm:8000/v1 regardless.
- "127.0.0.1:8001:8000"
volumes:
- ~/.cache/huggingface:/root/.cache/huggingface
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
open-notebook:
image: lfnovo/open_notebook:v1-latest
pull_policy: always
depends_on:
- vllm
```
Then in **Settings → API Keys**, add an **OpenAI-Compatible** credential with base URL: `http://vllm:8000/v1`
---
## Adding Models in Open Notebook
### Via Settings UI
1. Go to **Settings** → **Models**
2. Click **Add Model** in appropriate section
3. Select **Provider**: `openai_compatible`
4. Enter **Model Name**: exactly as the server expects
5. Enter **Display Name**: your preferred name
6. Click **Save**
### Model Name Format
The model name must match what your server expects:
| Server | Model Name Format |
|--------|-------------------|
| LM Studio | As shown in LM Studio UI |
| vLLM | HuggingFace model path |
| Text Gen WebUI | As loaded in UI |
| llama.cpp | Model file name |
---
## Testing Connection
### Test API Endpoint
```bash
# Test chat completions
curl http://localhost:1234/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "your-model-name",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
### Test from Inside Docker
```bash
docker exec -it open-notebook curl http://host.docker.internal:1234/v1/models
```
---
## Troubleshooting
### Connection Refused
```
Problem: Cannot connect to server
Solutions:
1. Verify server is running
2. Check port is correct
3. Test with curl directly
4. Check Docker networking (use host.docker.internal)
5. Verify firewall allows connection
```
### Model Not Found
```
Problem: Server returns "model not found"
Solutions:
1. Check model is loaded in server
2. Verify exact model name spelling
3. List available models: curl http://localhost:1234/v1/models
4. Update model name in Open Notebook
```
### Slow Responses
```
Problem: Requests take very long
Solutions:
1. Check server resources (RAM, GPU)
2. Use smaller/quantized model
3. Reduce context length
4. Enable GPU acceleration if available
```
### Authentication Errors
```
Problem: 401 or authentication failed
Solutions:
1. Check if server requires API key
2. Set the API key in your credential (Settings → API Keys)
3. Some servers need any non-empty key (use a placeholder like "not-needed")
```
### Timeout Errors
```
Problem: Request times out
Solutions:
1. Model may be loading (first request slow)
2. Increase timeout settings
3. Check server logs for errors
4. Reduce request size
```
---
## Multiple Compatible Endpoints
You can use different compatible servers for different purposes. When adding an **OpenAI-Compatible** credential in **Settings → API Keys**, you can configure per-service URLs:
- **LLM URL**: e.g., `http://localhost:1234/v1` (LM Studio)
- **Embedding URL**: e.g., `http://localhost:8080/v1` (different server)
- **TTS URL**: e.g., `http://localhost:8969/v1` (Speaches)
- **STT URL**: e.g., `http://localhost:9000/v1` (Speaches)
Alternatively, add each as a separate credential with its own base URL.
---
## Performance Tips
### Model Selection
| Model Size | RAM Needed | Speed |
|------------|------------|-------|
| 7B | 8GB | Fast |
| 13B | 16GB | Medium |
| 70B | 64GB+ | Slow |
### Quantization
Use quantized models (Q4, Q5) for faster inference with less RAM:
```
llama-3-8b-q4_k_m.gguf → ~4GB RAM, fast
llama-3-8b-f16.gguf → ~16GB RAM, slower
```
### GPU Acceleration
Enable GPU in your server for much faster inference:
- LM Studio: Settings → GPU layers
- vLLM: Automatic with CUDA
- llama.cpp: `--n-gpu-layers 35`
---
## Comparison: Native vs Compatible
| Aspect | Native Provider | OpenAI Compatible |
|--------|-----------------|-------------------|
| **Setup** | API key only | Server + configuration |
| **Models** | Provider's models | Any compatible model |
| **Cost** | Pay per token | Free (local) |
| **Speed** | Usually fast | Depends on hardware |
| **Features** | Full support | Basic features |
Use OpenAI-compatible when:
- Running local models
- Using custom/fine-tuned models
- Privacy requirements
- Cost control
---
## Related
- **[Local TTS Setup](local-tts.md)** - Text-to-speech with Speaches
- **[Local STT Setup](local-stt.md)** - Speech-to-text with Speaches
- **[AI Providers](ai-providers.md)** - All provider options
- **[Ollama Setup](ollama.md)** - Native Ollama integration
docs/5-CONFIGURATION/reverse-proxy.md
+903
View File
@@ -0,0 +1,903 @@
# Reverse Proxy Configuration
Deploy Open Notebook behind nginx, Caddy, Traefik, or other reverse proxies with custom domains and HTTPS.
---
## Simplified Setup (v1.1+)
Starting with v1.1, Open Notebook uses Next.js rewrites to simplify configuration. **You only need to proxy to one port** - Next.js handles internal API routing automatically.
### How It Works
```
Browser → Reverse Proxy → Port 8502 (Next.js)
↓ (internal proxy)
Port 5055 (FastAPI)
```
Next.js automatically forwards `/api/*` requests to the FastAPI backend, so your reverse proxy only needs one port!
---
## Quick Configuration Examples
### Nginx (Recommended)
```nginx
server {
listen 443 ssl http2;
server_name notebook.example.com;
ssl_certificate /etc/nginx/ssl/fullchain.pem;
ssl_certificate_key /etc/nginx/ssl/privkey.pem;
# Allow file uploads up to 100MB
client_max_body_size 100M;
# Single location block - that's it!
location / {
proxy_pass http://open-notebook:8502;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_cache_bypass $http_upgrade;
}
}
# HTTP to HTTPS redirect
server {
listen 80;
server_name notebook.example.com;
return 301 https://$server_name$request_uri;
}
```
### Caddy
```caddy
notebook.example.com {
reverse_proxy open-notebook:8502 {
transport http {
read_timeout 600s
write_timeout 600s
}
}
}
```
Caddy handles HTTPS automatically. The timeout settings ensure long-running operations (transformations, podcast generation) don't fail.
### Traefik
```yaml
# Add this to your docker-compose.yml alongside the surrealdb service
# See full base setup: https://github.com/lfnovo/open-notebook/blob/main/docker-compose.yml
services:
open-notebook:
image: lfnovo/open_notebook:v1-latest
pull_policy: always
environment:
- API_URL=https://notebook.example.com
labels:
- "traefik.enable=true"
- "traefik.http.routers.notebook.rule=Host(`notebook.example.com`)"
- "traefik.http.routers.notebook.entrypoints=websecure"
- "traefik.http.routers.notebook.tls.certresolver=myresolver"
- "traefik.http.services.notebook.loadbalancer.server.port=8502"
# Timeout for long-running operations (transformations, podcasts)
- "traefik.http.services.notebook.loadbalancer.responseforwarding.flushinterval=100ms"
networks:
- traefik-network
```
**Note**: For Traefik v2+, you may also need to configure `serversTransport` timeouts in your static configuration:
```yaml
# traefik.yml (static configuration)
serversTransport:
forwardingTimeouts:
dialTimeout: 30s
responseHeaderTimeout: 600s
idleConnTimeout: 90s
```
### Coolify
1. Create new service using [Docker Compose](../1-INSTALLATION/docker-compose.md)
2. Set port to **8502**
3. Add environment: `API_URL=https://your-domain.com`
4. Enable HTTPS in Coolify
5. Done!
---
## Environment Variables
```bash
# Required for reverse proxy setups
API_URL=https://your-domain.com
# Optional: For multi-container deployments
# INTERNAL_API_URL=http://api-service:5055
```
**Important**: Set `API_URL` to your public URL (with https://).
**Note on the bind address**: The Docker images bind Next.js to `0.0.0.0` by default, which ensures it listens on all interfaces and is accessible from reverse proxies. You typically don't need to change this; if you do, set `FRONTEND_BIND_HOST` (the `HOSTNAME` variable is unreliable because container runtimes such as Podman override it with the container/pod hostname).
---
## Understanding API_URL
The frontend uses a three-tier priority system to determine the API URL:
1. **Runtime Configuration** (Highest Priority): `API_URL` environment variable set at container runtime
2. **Build-time Configuration**: `NEXT_PUBLIC_API_URL` baked into the Docker image
3. **Auto-detection** (Fallback): Infers from the incoming HTTP request headers
### Auto-Detection Details
When `API_URL` is not set, the Next.js frontend:
- Analyzes the incoming HTTP request
- Extracts the hostname from the `host` header
- Respects the `X-Forwarded-Proto` header (for HTTPS behind reverse proxies)
- Constructs the API URL as `{protocol}://{hostname}:5055`
- Example: Request to `http://10.20.30.20:8502` → API URL becomes `http://10.20.30.20:5055`
**Why set API_URL explicitly?**
- **Reliability**: Auto-detection can fail with complex proxy setups
- **HTTPS**: Ensures frontend uses `https://` when behind SSL-terminating proxy
- **Custom domains**: Works correctly with domain names instead of IP addresses
- **Port mapping**: Avoids exposing port 5055 in the URL when using reverse proxy
**Important**: Don't include `/api` at the end - the system adds this automatically!
---
## Complete Docker Compose Example
> **Note:** This example only shows the open-notebook and nginx services. You also need a `surrealdb` service. See the [full base docker-compose.yml](https://github.com/lfnovo/open-notebook/blob/main/docker-compose.yml) for the complete setup.
```yaml
services:
open-notebook:
image: lfnovo/open_notebook:v1-latest
pull_policy: always
container_name: open-notebook
environment:
- API_URL=https://notebook.example.com
- OPEN_NOTEBOOK_ENCRYPTION_KEY=${OPEN_NOTEBOOK_ENCRYPTION_KEY}
- OPEN_NOTEBOOK_PASSWORD=${OPEN_NOTEBOOK_PASSWORD}
volumes:
- ./notebook_data:/app/data
# Only expose to localhost (nginx handles public access)
ports:
- "127.0.0.1:8502:8502"
restart: unless-stopped
nginx:
image: nginx:alpine
container_name: nginx-proxy
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
- ./ssl:/etc/nginx/ssl:ro
depends_on:
- open-notebook
restart: unless-stopped
```
---
## Full Nginx Configuration
```nginx
events {
worker_connections 1024;
}
http {
upstream notebook {
server open-notebook:8502;
}
# HTTP redirect
server {
listen 80;
server_name notebook.example.com;
return 301 https://$server_name$request_uri;
}
# HTTPS server
server {
listen 443 ssl http2;
server_name notebook.example.com;
ssl_certificate /etc/nginx/ssl/fullchain.pem;
ssl_certificate_key /etc/nginx/ssl/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
# Allow file uploads up to 100MB
client_max_body_size 100M;
# Security headers
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
# Proxy settings
location / {
proxy_pass http://notebook;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_cache_bypass $http_upgrade;
# Timeouts for long-running operations (transformations, podcasts, etc.)
# 600s matches the frontend timeout for slow LLM operations
proxy_read_timeout 600s;
proxy_connect_timeout 60s;
proxy_send_timeout 600s;
}
}
}
```
---
## Direct API Access (Optional)
If external scripts or integrations need direct API access, route `/api/*` directly:
```nginx
# Direct API access (for external integrations)
location /api/ {
proxy_pass http://open-notebook:5055/api/;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# Frontend (handles all other traffic)
location / {
proxy_pass http://open-notebook:8502;
# ... same headers as above
}
```
**Note**: This is only needed for external API integrations. Browser traffic works fine with single-port setup.
---
## Advanced Scenarios
### Remote Server Access (LAN/VPS)
Accessing Open Notebook from a different machine on your network:
**Step 1: Get your server IP**
```bash
# On the server running Open Notebook:
hostname -I
# or
ifconfig | grep "inet "
# Note the IP (e.g., 192.168.1.100)
```
**Step 2: Configure API_URL**
```bash
# In docker-compose.yml or .env:
API_URL=http://192.168.1.100:5055
```
**Step 3: Expose ports**
```yaml
# Add to your docker-compose.yml (requires surrealdb service, see installation guide)
services:
open-notebook:
image: lfnovo/open_notebook:v1-latest
pull_policy: always
environment:
- API_URL=http://192.168.1.100:5055
ports:
- "8502:8502"
- "5055:5055"
```
**Step 4: Access from client machine**
```bash
# In browser on other machine:
http://192.168.1.100:8502
```
**Troubleshooting**:
- Check firewall: `sudo ufw allow 8502 && sudo ufw allow 5055`
- Verify connectivity: `ping 192.168.1.100` from client machine
- Test port: `telnet 192.168.1.100 8502` from client machine
---
### API on Separate Subdomain
Host the API and frontend on different subdomains:
**docker-compose.yml:**
```yaml
# Add to your docker-compose.yml (requires surrealdb service, see installation guide)
services:
open-notebook:
image: lfnovo/open_notebook:v1-latest
pull_policy: always
environment:
- API_URL=https://api.notebook.example.com
- OPEN_NOTEBOOK_ENCRYPTION_KEY=${OPEN_NOTEBOOK_ENCRYPTION_KEY}
# Don't expose ports (nginx handles routing)
```
**nginx.conf:**
```nginx
# Frontend server
server {
listen 443 ssl http2;
server_name notebook.example.com;
ssl_certificate /etc/nginx/ssl/fullchain.pem;
ssl_certificate_key /etc/nginx/ssl/privkey.pem;
location / {
proxy_pass http://open-notebook:8502;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_cache_bypass $http_upgrade;
}
}
# API server (separate subdomain)
server {
listen 443 ssl http2;
server_name api.notebook.example.com;
ssl_certificate /etc/nginx/ssl/fullchain.pem;
ssl_certificate_key /etc/nginx/ssl/privkey.pem;
location / {
proxy_pass http://open-notebook:5055;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
```
**Use case**: Separate DNS records, different rate limiting, or isolated API access control.
---
### Multi-Container Deployment (Advanced)
For complex deployments with separate frontend and API containers:
**docker-compose.yml:**
```yaml
services:
frontend:
image: lfnovo/open_notebook_frontend:v1-latest
pull_policy: always
environment:
- API_URL=https://notebook.example.com
ports:
- "8502:8502"
api:
image: lfnovo/open_notebook_api:v1-latest
pull_policy: always
environment:
- OPEN_NOTEBOOK_ENCRYPTION_KEY=${OPEN_NOTEBOOK_ENCRYPTION_KEY}
ports:
- "5055:5055"
depends_on:
- surrealdb
surrealdb:
image: surrealdb/surrealdb:latest
command: start --log trace --user root --pass root file:/mydata/database.db
ports:
# Localhost only — the database uses default credentials, so never
# publish this port on 0.0.0.0
- "127.0.0.1:8000:8000"
volumes:
- ./surreal_data:/mydata
```
**nginx.conf:**
```nginx
http {
upstream frontend {
server frontend:8502;
}
upstream api {
server api:5055;
}
server {
listen 443 ssl http2;
server_name notebook.example.com;
# API routes
location /api/ {
proxy_pass http://api/api/;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# Frontend (catch-all)
location / {
proxy_pass http://frontend;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_cache_bypass $http_upgrade;
}
}
}
```
**Note**: Most users should use the [Docker Compose](../1-INSTALLATION/docker-compose.md) approach (`v1-latest`). Multi-container with separate nginx is only needed for custom scaling or isolation requirements.
---
## SSL Certificates
### Let's Encrypt with Certbot
```bash
# Install certbot
sudo apt install certbot python3-certbot-nginx
# Get certificate
sudo certbot --nginx -d notebook.example.com
# Auto-renewal (usually configured automatically)
sudo certbot renew --dry-run
```
### Let's Encrypt with Caddy
Caddy handles SSL automatically - no configuration needed!
### Self-Signed (Development Only)
```bash
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout ssl/privkey.pem \
-out ssl/fullchain.pem \
-subj "/CN=localhost"
```
---
## Troubleshooting
### "Unable to connect to server"
1. **Check API_URL is set**:
```bash
docker exec open-notebook env | grep API_URL
```
2. **Verify reverse proxy reaches container**:
```bash
curl -I http://localhost:8502
```
3. **Check browser console** (F12):
- Look for connection errors
- Check what URL it's trying to reach
### Mixed Content Errors
Frontend using HTTPS but trying to reach HTTP API:
```bash
# Ensure API_URL uses https://
API_URL=https://notebook.example.com # Not http://
```
### WebSocket Issues
Ensure your proxy supports WebSocket upgrades:
```nginx
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
```
### 502 Bad Gateway
1. Check container is running: `docker ps`
2. Check container logs: `docker logs open-notebook`
3. Verify nginx can reach container (same network)
### Timeout Errors
**Symptoms:**
- `socket hang up` or `ECONNRESET` errors
- `Timeout after 30000ms` errors
- Operations fail after exactly 30 seconds
**Cause:** Your reverse proxy has a default timeout (often 30s) that's shorter than Open Notebook's operations.
**Solutions by proxy:**
**Nginx:**
```nginx
proxy_read_timeout 600s;
proxy_send_timeout 600s;
```
**Caddy:**
```caddy
reverse_proxy open-notebook:8502 {
transport http {
read_timeout 600s
write_timeout 600s
}
}
```
**Traefik (static config):**
```yaml
serversTransport:
forwardingTimeouts:
responseHeaderTimeout: 600s
```
**Application-level timeouts:**
If you still experience timeouts after configuring your proxy, you can also adjust the application timeouts:
```bash
# In .env file:
API_CLIENT_TIMEOUT=600 # API client timeout (default: 300s)
ESPERANTO_LLM_TIMEOUT=180 # LLM inference timeout (default: 60s)
```
See [Advanced Configuration](advanced.md) for more timeout options.
---
### How to Debug Configuration Issues
**Step 1: Check browser console** (F12 → Console tab)
```
Look for messages starting with 🔧 [Config]
These show the configuration detection process
You'll see which API URL is being used
```
**Example good output:**
```
✅ [Config] Runtime API URL from server: https://your-domain.com
```
**Example bad output:**
```
❌ [Config] Failed to fetch runtime config
⚠️ [Config] Using auto-detected URL: http://localhost:5055
```
**Step 2: Test API directly**
```bash
# Should return JSON config
curl https://your-domain.com/api/config
# Expected output:
{"status":"ok","credentials_configured":true,...}
```
**Step 3: Check Docker logs**
```bash
docker logs open-notebook
# Look for:
# - Frontend startup: "▲ Next.js ready on http://0.0.0.0:8502"
# - API startup: "INFO: Uvicorn running on http://0.0.0.0:5055"
# - Connection errors or CORS issues
```
**Step 4: Verify environment variable**
```bash
docker exec open-notebook env | grep API_URL
# Should show:
# API_URL=https://your-domain.com
```
---
### Frontend Adds `:5055` to URL (Versions ≤ 1.0.10)
**Symptoms** (only in older versions):
- You set `API_URL=https://your-domain.com`
- Browser console shows: "Attempted URL: https://your-domain.com:5055/api/config"
- CORS errors with "Status code: (null)"
**Root Cause:**
In versions ≤ 1.0.10, the frontend's config endpoint was at `/api/runtime-config`, which got intercepted by reverse proxies routing all `/api/*` requests to the backend. This prevented the frontend from reading the `API_URL` environment variable.
**Solution:**
Upgrade to version 1.0.11 or later. The config endpoint has been moved to `/config` which avoids the `/api/*` routing conflict.
**Verification:**
Check browser console (F12) - should see: `✅ [Config] Runtime API URL from server: https://your-domain.com`
**If you can't upgrade**, explicitly configure the `/config` route:
```nginx
# Only needed for versions ≤ 1.0.10
location = /config {
proxy_pass http://open-notebook:8502;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
}
```
---
### File Upload Errors (413 Payload Too Large)
**Symptoms:**
```
CORS header 'Access-Control-Allow-Origin' missing. Status code: 413.
Error creating source. Please try again.
```
**Root Cause:**
When uploading files, your reverse proxy may reject the request due to body size limits *before* it reaches the application. Since the error happens at the proxy level, CORS headers are not included in the response.
**Version Requirement:**
- **Open Notebook v1.3.2+** is required for file uploads >10MB
- Uses Next.js 16+ which supports the `proxyClientMaxBodySize` configuration option
- Check your version: Settings → About (bottom of settings page)
**Solutions:**
1. **Nginx - Increase body size limit**:
```nginx
server {
# Allow larger file uploads (default is 1MB)
client_max_body_size 100M;
# Add CORS headers to error responses
error_page 413 = @cors_error_413;
location @cors_error_413 {
add_header 'Access-Control-Allow-Origin' '*' always;
add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, DELETE, OPTIONS' always;
add_header 'Access-Control-Allow-Headers' '*' always;
return 413 '{"detail": "File too large. Maximum size is 100MB."}';
}
location / {
# ... your existing proxy configuration
}
}
```
2. **Traefik - Increase buffer size**:
```yaml
# In your traefik configuration
http:
middlewares:
large-body:
buffering:
maxRequestBodyBytes: 104857600 # 100MB
```
Apply middleware to your router:
```yaml
labels:
- "traefik.http.routers.notebook.middlewares=large-body"
```
3. **Kubernetes Ingress (nginx-ingress)**:
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: open-notebook
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "100m"
# Add CORS headers for error responses
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "Access-Control-Allow-Origin: *";
```
4. **Caddy**:
```caddy
notebook.example.com {
request_body {
max_size 100MB
}
reverse_proxy open-notebook:8502 {
transport http {
read_timeout 600s
write_timeout 600s
}
}
}
```
**Note:** Open Notebook's API includes CORS headers in error responses, but this only works for errors that reach the application. Proxy-level errors (like 413 from nginx) need to be configured at the proxy level.
---
### CORS Errors
**Symptoms:**
```
Access-Control-Allow-Origin header is missing
Cross-Origin Request Blocked
Response to preflight request doesn't pass access control check
```
**Possible Causes:**
1. **Missing proxy headers**:
```nginx
# Make sure these are set:
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
```
2. **API_URL protocol mismatch**:
```bash
# Frontend is HTTPS, but API_URL is HTTP:
API_URL=http://notebook.example.com # ❌ Wrong
API_URL=https://notebook.example.com # ✅ Correct
```
3. **Reverse proxy not forwarding `/api/*` correctly**:
```nginx
# Make sure this works:
location /api/ {
proxy_pass http://open-notebook:5055/api/; # Note the trailing slash!
}
```
---
### Missing Authorization Header
**Symptoms:**
```json
{"detail": "Missing authorization header"}
```
This happens when:
- You have set `OPEN_NOTEBOOK_PASSWORD` for authentication
- You're trying to access `/api/config` directly without logging in first
**Solution:**
This is **expected behavior**! The frontend handles authentication automatically. Just:
1. Access the frontend URL (not `/api/` directly)
2. Log in through the UI
3. The frontend will handle authorization headers for all API calls
**For API integrations:** Include the password in the Authorization header:
```bash
curl -H "Authorization: Bearer your-password-here" \
https://your-domain.com/api/config
```
---
### SSL/TLS Certificate Errors
**Symptoms:**
- Browser shows "Your connection is not private"
- Certificate warnings
- Mixed content errors
**Solutions:**
1. **Use Let's Encrypt** (recommended):
```bash
sudo certbot --nginx -d notebook.example.com
```
2. **Check certificate paths** in nginx:
```nginx
ssl_certificate /etc/nginx/ssl/fullchain.pem; # Full chain
ssl_certificate_key /etc/nginx/ssl/privkey.pem; # Private key
```
3. **Verify certificate is valid**:
```bash
openssl x509 -in /etc/nginx/ssl/fullchain.pem -text -noout
```
4. **For development**, use self-signed (not for production):
```bash
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout ssl/privkey.pem -out ssl/fullchain.pem \
-subj "/CN=localhost"
```
---
## Best Practices
1. **Always use HTTPS** in production
2. **Set API_URL explicitly** when using reverse proxies to avoid auto-detection issues
3. **Bind to localhost** (`127.0.0.1:8502`) and let proxy handle public access for security
4. **Enable security headers** (HSTS, X-Frame-Options, X-Content-Type-Options, X-XSS-Protection)
5. **Set up certificate renewal** for Let's Encrypt (usually automatic with certbot)
6. **Keep ports 5055 and 8502 accessible** from your reverse proxy container (use Docker networks)
7. **Use environment files** (`.env` or `docker.env`) to manage configuration securely
8. **Test your configuration** before going live:
- Check browser console for config messages
- Test API: `curl https://your-domain.com/api/config`
- Verify authentication works
- Check long-running operations (podcast generation)
9. **Monitor logs** regularly: `docker logs open-notebook`
10. **Don't include `/api` in API_URL** - the system adds this automatically
---
## Legacy Configurations (Pre-v1.1)
If you're running Open Notebook **version 1.0.x or earlier**, you may need to use the legacy two-port configuration where you explicitly route `/api/*` to port 5055.
**Check your version:**
```bash
docker exec open-notebook cat /app/package.json | grep version
```
**If version < 1.1.0**, you may need:
- Explicit `/api/*` routing to port 5055 in reverse proxy
- Explicit `/config` endpoint routing for versions ≤ 1.0.10
- See the "Frontend Adds `:5055` to URL" troubleshooting section above
**Recommendation:** Upgrade to v1.1+ for simplified configuration and better performance.
---
## Related
- **[Security Configuration](security.md)** - Password protection and hardening
- **[Advanced Configuration](advanced.md)** - Ports, timeouts, and SSL settings
- **[Troubleshooting](../6-TROUBLESHOOTING/connection-issues.md)** - Connection problems
- **[Docker Deployment](../1-INSTALLATION/docker-compose.md)** - Complete deployment guide
docs/5-CONFIGURATION/security.md
+422
View File
@@ -0,0 +1,422 @@
# Security Configuration
Protect your Open Notebook deployment with password authentication and production hardening.
---
## API Key Encryption
Open Notebook encrypts API keys stored in the database using Fernet symmetric encryption (AES-128-CBC with HMAC-SHA256).
### Configuration Methods
| Method | Documentation |
|--------|---------------|
| **Settings UI** | [API Configuration Guide](../3-USER-GUIDE/api-configuration.md) |
| **Environment Variables** | This page (below) |
### Setup
Set the encryption key to any secret string:
```bash
# .env or docker.env
OPEN_NOTEBOOK_ENCRYPTION_KEY=my-secret-passphrase
```
Any string works — it will be securely derived via SHA-256 internally. Use a strong passphrase for production deployments.
### Default Credentials
| Setting | Default | Security Level |
|---------|---------|----------------|
| Password | None - auth is fully disabled until `OPEN_NOTEBOOK_PASSWORD` is set | Development only |
| Encryption Key | **None** (must be configured) | Required for API key storage |
**The encryption key has no default.** You must set `OPEN_NOTEBOOK_ENCRYPTION_KEY` before using the API key configuration feature. Without it, encrypting/decrypting API keys will fail.
### Docker Secrets Support
Both settings support Docker secrets via `_FILE` suffix:
```yaml
environment:
- OPEN_NOTEBOOK_PASSWORD_FILE=/run/secrets/app_password
- OPEN_NOTEBOOK_ENCRYPTION_KEY_FILE=/run/secrets/encryption_key
```
### Security Notes
| Scenario | Behavior |
|----------|----------|
| Key configured | API keys encrypted with your key |
| No key configured | Encryption/decryption will fail (key is required) |
| Key changed | Old encrypted keys become unreadable |
| Legacy data | Unencrypted keys still work (graceful fallback) |
### Key Management
- **Keep secret**: Never commit the encryption key to version control
- **Backup securely**: Store the key separately from database backups
- **No rotation yet**: Changing the key requires re-saving all API keys
- **Per-deployment**: Each instance should have its own encryption key
---
## When to Use Password Protection
### Use it for:
- Public cloud deployments (PikaPods, Railway, DigitalOcean)
- Shared network environments
- Any deployment accessible beyond localhost
### You can skip it for:
- Local development on your machine
- Private, isolated networks
- Single-user local setups
---
## Quick Setup
### Docker Deployment
```yaml
# Add to your docker-compose.yml (requires surrealdb service, see installation guide)
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest
pull_policy: always
environment:
- OPEN_NOTEBOOK_ENCRYPTION_KEY=your-secret-encryption-key
- OPEN_NOTEBOOK_PASSWORD=your_secure_password
# ... rest of config
```
Or using environment file:
```bash
# docker.env
OPEN_NOTEBOOK_ENCRYPTION_KEY=your-secret-encryption-key
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
> **Important**: The encryption key is **required** for credential storage. Without it, you cannot save AI provider credentials via the Settings UI. If you change or lose the encryption key, all stored credentials become unreadable.
### Development Setup
```bash
# .env
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
---
## Password Requirements
### Good Passwords
```bash
# Strong: 20+ characters, mixed case, numbers, symbols
OPEN_NOTEBOOK_PASSWORD=MySecure2024!Research#Tool
OPEN_NOTEBOOK_PASSWORD=Notebook$Dev$2024$Strong!
# Generated (recommended)
OPEN_NOTEBOOK_PASSWORD=$(openssl rand -base64 24)
```
### Bad Passwords
```bash
# DON'T use these
OPEN_NOTEBOOK_PASSWORD=password123
OPEN_NOTEBOOK_PASSWORD=opennotebook
OPEN_NOTEBOOK_PASSWORD=admin
```
---
## How It Works
### Frontend Protection
1. Login form appears on first visit
2. Password stored in browser session
3. Session persists until browser closes
4. Clear browser data to log out
### API Protection
All API endpoints require authentication:
```bash
# Authenticated request
curl -H "Authorization: Bearer your_password" \
http://localhost:5055/api/notebooks
# Unauthenticated (will fail)
curl http://localhost:5055/api/notebooks
# Returns: {"detail": "Missing authorization header"}
```
### Unprotected Endpoints
These work without authentication:
- `/health` - System health check
- `/docs` - API documentation
- `/openapi.json` - OpenAPI spec
---
## API Authentication Examples
### curl
```bash
# List notebooks
curl -H "Authorization: Bearer your_password" \
http://localhost:5055/api/notebooks
# Create notebook
curl -X POST \
-H "Authorization: Bearer your_password" \
-H "Content-Type: application/json" \
-d '{"name": "My Notebook", "description": "Research notes"}' \
http://localhost:5055/api/notebooks
# Upload file
curl -X POST \
-H "Authorization: Bearer your_password" \
-F "file=@document.pdf" \
http://localhost:5055/api/sources/upload
```
### Python
```python
import requests
class OpenNotebookClient:
def __init__(self, base_url: str, password: str):
self.base_url = base_url
self.headers = {"Authorization": f"Bearer {password}"}
def get_notebooks(self):
response = requests.get(
f"{self.base_url}/api/notebooks",
headers=self.headers
)
return response.json()
def create_notebook(self, name: str, description: str = None):
response = requests.post(
f"{self.base_url}/api/notebooks",
headers=self.headers,
json={"name": name, "description": description}
)
return response.json()
# Usage
client = OpenNotebookClient("http://localhost:5055", "your_password")
notebooks = client.get_notebooks()
```
### JavaScript/TypeScript
```javascript
const API_URL = 'http://localhost:5055';
const PASSWORD = 'your_password';
async function getNotebooks() {
const response = await fetch(`${API_URL}/api/notebooks`, {
headers: {
'Authorization': `Bearer ${PASSWORD}`
}
});
return response.json();
}
```
---
## Production Hardening
### Docker Security
```yaml
# Add to your docker-compose.yml (requires surrealdb service, see installation guide)
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest
pull_policy: always
ports:
- "127.0.0.1:8502:8502" # Bind to localhost only
environment:
- OPEN_NOTEBOOK_PASSWORD=your_secure_password
security_opt:
- no-new-privileges:true
deploy:
resources:
limits:
memory: 2G
cpus: "1.0"
restart: always
```
### Firewall Configuration
```bash
# UFW (Ubuntu)
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw deny 8502/tcp # Block direct access
sudo ufw deny 5055/tcp # Block direct API access
sudo ufw enable
# iptables
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT
iptables -A INPUT -p tcp --dport 8502 -j DROP
iptables -A INPUT -p tcp --dport 5055 -j DROP
```
### Reverse Proxy with SSL
See [Reverse Proxy Configuration](reverse-proxy.md) for complete nginx/Caddy/Traefik setup with HTTPS.
### CORS Origins
The API accepts cross-origin requests from any origin by default (`*`). This is convenient for development and diverse self-hosted setups, but it's not recommended for internet-facing production deployments because any website the user visits can issue authenticated cross-origin requests to your API.
When `CORS_ORIGINS` is not set, the API logs a startup warning prompting you to configure it.
**For production, set `CORS_ORIGINS` to your frontend's actual origin(s):**
```bash
# Single origin
CORS_ORIGINS=https://notebook.example.com
# Multiple origins (comma-separated)
CORS_ORIGINS=https://notebook.example.com,https://admin.example.com
```
**Guidelines:**
- Always use HTTPS origins in production.
- List only the exact origins that should be allowed to call the API.
- Include the scheme and port (if non-default): `https://example.com`, `http://192.168.1.10:3000`.
- Changes require an API restart to take effect.
**Error responses** (401, 404, 500, etc.) also respect the configured origins — they only include `Access-Control-Allow-Origin` for allowed origins, so error bodies are not leaked cross-origin when `CORS_ORIGINS` is configured.
---
## Security Limitations
Open Notebook's password protection provides **basic access control**, not enterprise-grade security:
| Feature | Status |
|---------|--------|
| Password transmission | Plain text (use HTTPS!) |
| Password storage | In memory |
| User management | Single password for all |
| Session timeout | None (until browser close) |
| Rate limiting | None |
| Audit logging | None |
### Risk Mitigation
1. **Always use HTTPS** - Encrypt traffic with TLS
2. **Strong passwords** - 20+ characters, complex
3. **Network security** - Firewall, VPN for sensitive deployments
4. **Regular updates** - Keep containers and dependencies updated
5. **Monitoring** - Check logs for suspicious activity
6. **Backups** - Regular backups of data
---
## Enterprise Considerations
For deployments requiring advanced security:
| Need | Solution |
|------|----------|
| SSO/OAuth | Implement OAuth2/SAML proxy |
| Role-based access | Custom middleware |
| Audit logging | Log aggregation service |
| Rate limiting | API gateway or nginx |
| Data encryption | Encrypt volumes at rest |
| Network segmentation | Docker networks, VPC |
---
## Troubleshooting
### Password Not Working
```bash
# Check env var is set
docker exec open-notebook env | grep OPEN_NOTEBOOK_PASSWORD
# Check logs
docker logs open-notebook | grep -i auth
# Test API directly
curl -H "Authorization: Bearer your_password" \
http://localhost:5055/health
```
### 401 Unauthorized Errors
```bash
# Check header format
curl -v -H "Authorization: Bearer your_password" \
http://localhost:5055/api/notebooks
# Verify password matches
echo "Password length: $(echo -n $OPEN_NOTEBOOK_PASSWORD | wc -c)"
```
### Cannot Access After Setting Password
1. Clear browser cache and cookies
2. Try incognito/private mode
3. Check browser console for errors
4. Verify password is correct in environment
### Security Testing
```bash
# Without password (should fail)
curl http://localhost:5055/api/notebooks
# Expected: {"detail": "Missing authorization header"}
# With correct password (should succeed)
curl -H "Authorization: Bearer your_password" \
http://localhost:5055/api/notebooks
# Health check (should work without password)
curl http://localhost:5055/health
```
---
## Reporting Security Issues
If you discover security vulnerabilities:
1. **Do NOT open public issues**
2. Contact maintainers directly
3. Provide detailed information
4. Allow time for fixes before disclosure
---
## Related
- **[Reverse Proxy](reverse-proxy.md)** - HTTPS and SSL setup
- **[Advanced Configuration](advanced.md)** - Ports, timeouts, and SSL settings
- **[Environment Reference](environment-reference.md)** - All configuration options
docs/6-TROUBLESHOOTING/ai-chat-issues.md
+444
View File
@@ -0,0 +1,444 @@
# AI & Chat Issues - Model Configuration & Quality
Problems with AI models, chat, and response quality.
> **Note:** Open Notebook now shows descriptive error messages for AI provider failures. Instead of a generic "An unexpected error occurred", you'll see specific messages like "Authentication failed. Please check your API key" or "Rate limit exceeded. Please wait a moment and try again." These messages help you diagnose and fix issues faster.
---
## "Failed to send message" Error
**Symptom:** Chat shows "Failed to send message" toast. Logs show:
```
Error executing chat: Model is not a LanguageModel: None
```
**Cause:** No valid language model configured for chat
**Solutions:**
### Solution 1: Check Default Model Configuration
```
1. Go to Settings → Models
2. Scroll to "Default Models" section
3. Verify "Default Chat Model" has a model selected
4. If empty, select an available language model
5. Click Save
```
### Solution 2: Verify Model Names (Ollama Users)
```bash
# Get exact model names
ollama list
# Example output:
# NAME SIZE MODIFIED
# gemma3:12b 8.1 GB 2 months ago
# The model name in Open Notebook must be EXACTLY "gemma3:12b"
# NOT "gemma3" or "gemma3-12b"
```
### Solution 3: Re-add Missing Models
```
1. Note the exact model names from your provider
2. Go to Settings → Models
3. Delete any misconfigured models
4. Add models with exact names
5. Set new defaults
```
### Solution 4: Check Model Still Exists
```bash
# For Ollama: verify model is installed
ollama list
# For cloud providers: verify API key is valid
# and you have access to the model
```
> **Tip:** This error often occurs when you delete a model from Ollama but forget to update the default models in Open Notebook. Always re-configure defaults after removing models.
---
## "Models not available" or "Models not showing"
**Symptom:** Settings → Models shows empty, or "No models configured"
**Cause:** No credential configured, or credential has invalid API key
**Solutions:**
### Solution 1: Add Credential via Settings UI
```
1. Go to Settings → API Keys
2. Click "Add Credential"
3. Select your provider (e.g., OpenAI, Anthropic, Google)
4. Enter your API key
5. Click Save, then Test Connection
6. Click Discover Models → Register Models
7. Go to Settings → Models to verify
```
### Solution 2: Check Key is Valid
```
1. Go to Settings → API Keys
2. Click "Test Connection" on your credential
3. If it shows "Invalid API key":
- Get a fresh key from the provider's website
- Delete the credential and create a new one
```
### Solution 3: Switch Provider
```
1. Go to Settings → API Keys
2. Add a credential for a different provider
3. Test Connection → Discover Models → Register Models
4. Go to Settings → Models to select the new provider's models
```
---
## "Invalid API key" or "Unauthorized"
**Symptom:** Error when trying to chat: "Invalid API key"
**Cause:** Credential has wrong, expired, or revoked API key
**Solutions:**
### Step 1: Test Your Credential
```
1. Go to Settings → API Keys
2. Click "Test Connection" on your credential
3. If it fails, proceed to Step 2
```
### Step 2: Get Fresh Key
```
Go to provider's dashboard:
- OpenAI: https://platform.openai.com/api-keys (starts with sk-proj-)
- Anthropic: https://console.anthropic.com/ (starts with sk-ant-)
- Google: https://aistudio.google.com/app/apikey (starts with AIzaSy)
Generate new key and copy exactly (no extra spaces)
```
### Step 3: Update Credential
```
1. Go to Settings → API Keys
2. Delete the old credential
3. Click "Add Credential" → select provider
4. Paste the new key
5. Click Save, then Test Connection
6. Re-discover and register models if needed
```
### Step 4: Verify in UI
```
1. Go to Settings → Models
2. Verify models are available
3. Try a test chat
```
---
## Chat Returns Generic/Bad Responses
**Symptom:** AI responses are shallow, generic, or wrong
**Cause:** Bad context, vague question, or wrong model
**Solutions:**
### Solution 1: Check Context
```
1. In Chat, click "Select Sources"
2. Verify sources you want are CHECKED
3. Set them to "Full Content" (not "Summary Only")
4. Click "Save"
5. Try chat again
```
### Solution 2: Ask Better Question
```
Bad: "What do you think?"
Good: "Based on the paper's methodology, what are 3 limitations?"
Bad: "Tell me about X"
Good: "Summarize X in 3 bullet points with page citations"
```
### Solution 3: Use Stronger Model
```
OpenAI:
Current: gpt-4o-mini → Switch to: gpt-4o
Anthropic:
Current: claude-3-5-haiku → Switch to: claude-3-5-sonnet
To change:
1. Settings → Models
2. Select model
3. Try chat again
```
### Solution 4: Add More Sources
```
If: "Response seems incomplete"
Try: Add more relevant sources to provide context
```
---
## Chat is Very Slow
**Symptom:** Chat responses take minutes
**Cause:** Large context, slow model, or overloaded API
**Solutions:**
### Solution 1: Use Faster Model
```bash
Fastest: Groq (any model)
Fast: OpenAI gpt-4o-mini
Medium: Anthropic claude-3-5-haiku
Slow: Anthropic claude-3-5-sonnet
Switch in: Settings → Models
```
### Solution 2: Reduce Context
```
1. Chat → Select Sources
2. Uncheck sources you don't need
3. Or switch to "Summary Only" for background sources
4. Save and try again
```
### Solution 3: Increase Timeout
```bash
# In .env:
API_CLIENT_TIMEOUT=600 # 10 minutes
# Restart:
docker compose restart
```
### Solution 4: Check System Load
```bash
# See if API is overloaded:
docker stats
# If CPU >80% or memory >90%:
# Reduce: SURREAL_COMMANDS_MAX_TASKS=2
# Restart: docker compose restart
```
---
## Chat Doesn't Remember History
**Symptom:** Each message treated as separate, no context between questions
**Cause:** Chat history not saved or new chat started
**Solution:**
```
1. Make sure you're in same Chat (not new Chat)
2. Check Chat title at top
3. If it's blank, start new Chat with a title
4. Each named Chat keeps its history
5. If you start new Chat, history is separate
```
---
## "Rate limit exceeded"
**Symptom:** Error: "Rate limit exceeded" or "Too many requests"
**Cause:** Hit provider's API rate limit
**Solutions:**
### For Cloud Providers (OpenAI, Anthropic, etc.)
**Immediate:**
- Wait 1-2 minutes
- Try again
**Short term:**
- Use cheaper/smaller model
- Reduce concurrent operations
- Space out requests
**Long term:**
- Upgrade your account
- Switch to different provider
- Use Ollama (local, no limits)
### Check Account Status
```
OpenAI: https://platform.openai.com/account/usage/overview
Anthropic: https://console.anthropic.com/account/billing/overview
Google: Google Cloud Console
```
### For Ollama (Local)
- No rate limits
- Use `ollama pull mistral` for best model
- Restart if hitting resource limits
---
## "Context length exceeded" or "Token limit"
**Symptom:** Error about too many tokens
**Cause:** Sources too large for model
**Solutions:**
### Solution 1: Use Model with Longer Context
```
Current: GPT-4o (128K tokens) → Switch to: Claude (200K tokens)
Current: Claude Haiku (200K) → Switch to: Gemini (1M tokens)
To change: Settings → Models
```
### Solution 2: Reduce Context
```
1. Select fewer sources
2. Or use "Summary Only" instead of "Full Content"
3. Or split large documents into smaller pieces
```
### Solution 3: For Ollama (Local)
```bash
# Use smaller model:
ollama pull phi # Very small
# Instead of: ollama pull neural-chat # Large
```
---
## "API call failed" or Timeout
**Symptom:** Generic API error, response times out
**Cause:** Provider API down, network issue, or slow service
**Solutions:**
### Check Provider Status
```
OpenAI: https://status.openai.com/
Anthropic: Check website
Google: Google Cloud Status
Groq: Check website
```
### Retry Operation
```
1. Wait 30 seconds
2. Try again
```
### Use Different Model/Provider
```
1. Settings → Models
2. Try different provider
3. If OpenAI down, use Anthropic
```
### Check Network
```bash
# Verify internet working:
ping google.com
# Test API endpoint directly:
curl https://api.openai.com/v1/models \
-H "Authorization: Bearer YOUR_KEY"
```
---
## Responses Include Hallucinations
**Symptom:** AI makes up facts that aren't in sources
**Cause:** Sources not in context, or model guessing
**Solutions:**
### Solution 1: Verify Context
```
1. Click citation in response
2. Check source actually says that
3. If not, sources weren't in context
4. Add source to context and try again
```
### Solution 2: Request Citations
```
Ask: "Answer this with citations to specific pages"
The AI will be more careful if asked for citations
```
### Solution 3: Use Stronger Model
```
Weaker models hallucinate more
Switch to: GPT-4o or Claude Sonnet
```
---
## High API Costs
**Symptom:** API bills are higher than expected
**Cause:** Using expensive model, large context, many requests
**Solutions:**
### Use Cheaper Model
```
Expensive: gpt-4o
Cheaper: gpt-4o-mini (10x cheaper)
Expensive: Claude Sonnet
Cheaper: Claude Haiku (5x cheaper)
Groq: Ultra cheap but fewer models
```
### Reduce Context
```
In Chat:
1. Select fewer sources
2. Use "Summary Only" for background
3. Ask more specific questions
```
### Switch to Ollama (Free)
```bash
# Install Ollama
# Run: ollama serve
# Download: ollama pull mistral
# Set: OLLAMA_API_BASE=http://localhost:11434
# Cost: Free!
```
---
## Still Having Chat Issues?
- Try [Quick Fixes](quick-fixes.md)
- Try [Chat Effectively Guide](../3-USER-GUIDE/chat-effectively.md)
- Check logs: `docker compose logs api | grep -i "error"`
- Ask for help: [Troubleshooting Index](index.md#getting-help)
docs/6-TROUBLESHOOTING/connection-issues.md
+447
View File
@@ -0,0 +1,447 @@
# Connection Issues - Network & API Problems
Frontend can't reach API or services won't communicate.
---
## "Cannot connect to server" (Most Common)
**What it looks like:**
- Browser shows error page
- "Unable to reach API"
- "Cannot connect to server"
- UI loads but can't create notebooks
**Diagnosis:**
```bash
# Check if API is running
docker ps | grep api
# Should see "api" service running
# Check if API is responding
curl http://localhost:5055/health
# Should show: {"status":"ok"}
# Check if frontend is running
docker ps | grep frontend
# Should see "frontend" or React service running
```
**Solutions:**
### Solution 1: API Not Running
```bash
# Start API
docker compose up api -d
# Wait 5 seconds
sleep 5
# Verify it's running
docker compose logs api | tail -20
```
### Solution 2: Port Not Exposed
```bash
# Check docker-compose.yml has port mapping:
# api:
# ports:
# - "5055:5055"
# If missing, add it and restart:
docker compose down
docker compose up -d
```
### Solution 3: API_URL Mismatch
```bash
# In .env, check API_URL:
cat .env | grep API_URL
# Should match your frontend URL:
# Frontend: http://localhost:8502
# API_URL: http://localhost:5055
# If wrong, fix it:
# API_URL=http://localhost:5055
# Then restart:
docker compose restart frontend
```
### Solution 4: Firewall Blocking
```bash
# Verify port 5055 is accessible
netstat -tlnp | grep 5055
# Should show port listening
# If on different machine, try:
# Instead of localhost, use your IP:
API_URL=http://192.168.1.100:5055
```
### Solution 5: Services Not Started
```bash
# Restart everything
docker compose restart
# Wait 10 seconds
sleep 10
# Check all services
docker compose ps
# All should show "Up"
```
---
## Connection Refused
**What it looks like:**
```
Connection refused
ECONNREFUSED
Error: socket hang up
```
**Diagnosis:**
- API port (5055) not open
- API crashed
- Wrong IP/hostname
**Solution:**
```bash
# Step 1: Check if API is running
docker ps | grep api
# Step 2: Check if port is listening
lsof -i :5055
# or
netstat -tlnp | grep 5055
# Step 3: Check API logs
docker compose logs api | tail -30
# Look for errors
# Step 4: Restart API
docker compose restart api
docker compose logs api | grep -i "error"
```
---
## Timeout / Slow Connection
**What it looks like:**
- Page loads slowly
- Request times out
- "Gateway timeout" error
**Causes:**
- API is overloaded
- Network is slow
- Reverse proxy issue
**Solutions:**
### Check API Performance
```bash
# See CPU/memory usage
docker stats
# Check logs for slow operations
docker compose logs api | grep "slow\|timeout"
```
### Reduce Load
```bash
# In .env:
SURREAL_COMMANDS_MAX_TASKS=2
API_CLIENT_TIMEOUT=600
# Restart
docker compose restart
```
### Check Network
```bash
# Test latency
ping localhost
# Test API directly
time curl http://localhost:5055/health
# Should be < 100ms
```
---
## 502 Bad Gateway (Reverse Proxy)
**What it looks like:**
```
502 Bad Gateway
The server is temporarily unable to service the request
```
**Cause:** Reverse proxy can't reach API
**Solutions:**
### Check Backend is Running
```bash
# From the reverse proxy server
curl http://localhost:5055/health
# Should work
```
### Check Reverse Proxy Config
```nginx
# Nginx example (correct):
location /api {
proxy_pass http://localhost:5055/api;
proxy_http_version 1.1;
}
# Common mistake (wrong):
location /api {
proxy_pass http://localhost:5055; # Missing /api
}
```
### Set API_URL for HTTPS
```bash
# In .env:
API_URL=https://yourdomain.com
# Restart
docker compose restart
```
---
## Intermittent Disconnects
**What it looks like:**
- Works sometimes, fails other times
- Sporadic "cannot connect" errors
- Works then stops working
**Cause:** Transient network issue or database conflicts
**Solutions:**
### Enable Retry Logic
```bash
# In .env:
SURREAL_COMMANDS_RETRY_ENABLED=true
SURREAL_COMMANDS_RETRY_MAX_ATTEMPTS=5
SURREAL_COMMANDS_RETRY_WAIT_STRATEGY=exponential_jitter
# Restart
docker compose restart
```
### Reduce Concurrency
```bash
# In .env:
SURREAL_COMMANDS_MAX_TASKS=2
# Restart
docker compose restart
```
### Check Network Stability
```bash
# Monitor connection
ping google.com
# Long-running test
ping -c 100 google.com | grep "packet loss"
# Should be 0% loss
```
---
## Different Machine / Remote Access
**You want to access Open Notebook from another computer**
**Solution:**
### Step 1: Get Your Machine IP
```bash
# On the server running Open Notebook:
ifconfig | grep "inet "
# or
hostname -I
# Note the IP (e.g., 192.168.1.100)
```
### Step 2: Update API_URL
```bash
# In .env:
API_URL=http://192.168.1.100:5055
# Restart
docker compose restart
```
### Step 3: Access from Other Machine
```bash
# In browser on other machine:
http://192.168.1.100:8502
# (or your server IP)
```
### Step 4: Verify Port is Exposed
```bash
# On server:
docker compose ps
# Should show port mapping:
# 0.0.0.0:8502->8502/tcp
# 0.0.0.0:5055->5055/tcp
```
### If Still Doesn't Work
```bash
# Check firewall on server
sudo ufw status
# May need to open ports:
sudo ufw allow 8502
sudo ufw allow 5055
# Check on different machine:
telnet 192.168.1.100 5055
# Should connect
```
---
## CORS Error (Browser Console)
**What it looks like:**
```
Cross-Origin Request Blocked
Access-Control-Allow-Origin
```
**In browser console (F12):**
```
CORS policy: Response to preflight request doesn't pass access control check
```
**Cause:** Frontend and API URLs don't match
**Solution:**
```bash
# Check browser console error for what URLs are being used
# The error shows:
# - Requesting from: http://localhost:8502
# - Trying to reach: http://localhost:5055
# Make sure API_URL matches:
API_URL=http://localhost:5055
# And protocol matches (http/https)
# Restart
docker compose restart frontend
```
---
## Testing Connection
**Full diagnostic:**
```bash
# 1. Services running?
docker compose ps
# All should show "Up"
# 2. Ports listening?
netstat -tlnp | grep -E "8502|5055|8000"
# 3. API responding?
curl http://localhost:5055/health
# 4. Frontend accessible?
curl http://localhost:8502 | head
# 5. Network OK?
ping google.com
# 6. No firewall?
sudo ufw status | grep -E "5055|8502|8000"
```
---
## Checklist for Remote Access
- [ ] Server IP noted (e.g., 192.168.1.100)
- [ ] Ports 8502, 5055, 8000 exposed in docker-compose
- [ ] API_URL set to server IP
- [ ] Firewall allows ports 8502, 5055, 8000
- [ ] Can reach server from client machine (ping IP)
- [ ] All services running (docker compose ps)
- [ ] Can curl API from client (curl http://IP:5055/health)
---
## SSL Certificate Errors
**What it looks like:**
```
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
Connection error when using HTTPS endpoints
Works with HTTP but fails with HTTPS
```
**Cause:** Self-signed certificates not trusted by Python's SSL verification
**Solutions:**
### Solution 1: Use Custom CA Bundle (Recommended)
```bash
# In .env:
ESPERANTO_SSL_CA_BUNDLE=/path/to/your/ca-bundle.pem
# For Docker, mount the certificate:
# In docker-compose.yml:
volumes:
- /path/to/your/ca-bundle.pem:/certs/ca-bundle.pem:ro
environment:
- ESPERANTO_SSL_CA_BUNDLE=/certs/ca-bundle.pem
```
### Solution 2: Disable SSL Verification (Development Only)
```bash
# WARNING: Only use in trusted development environments
# In .env:
ESPERANTO_SSL_VERIFY=false
```
### Solution 3: Use HTTP Instead
If services are on a trusted local network, HTTP is acceptable:
```
Change the base URL in your credential (Settings → API Keys) from https:// to http://
Example: http://localhost:1234/v1
```
> **Security Note:** Disabling SSL verification exposes you to man-in-the-middle attacks. Always prefer custom CA bundle or HTTP on trusted networks.
---
## Still Having Issues?
- Check [Quick Fixes](quick-fixes.md)
- Check [FAQ](faq.md)
- Check logs: `docker compose logs`
- Try restart: `docker compose restart`
- Check firewall: `sudo ufw status`
- Ask for help on [Discord](https://discord.gg/37XJPXfz2w)
docs/6-TROUBLESHOOTING/faq.md
+258
View File
@@ -0,0 +1,258 @@
# Frequently Asked Questions
Common questions about Open Notebook usage, configuration, and best practices.
---
## General Usage
### What is Open Notebook?
Open Notebook is an open-source, privacy-focused alternative to Google's Notebook LM. It allows you to:
- Create and manage research notebooks
- Chat with your documents using AI
- Generate podcasts from your content
- Search across all your sources with semantic search
- Transform and analyze your content
### How is it different from Google Notebook LM?
**Privacy**: Your data stays local by default. Only your chosen AI providers receive queries.
**Flexibility**: Support for 17+ AI providers (OpenAI, Anthropic, Google, local models, etc.)
**Customization**: Open source, so you can modify and extend functionality
**Control**: You control your data, models, and processing
### Can I use Open Notebook offline?
**Partially**: The application runs locally, but requires internet for:
- AI model API calls (unless using local models like Ollama)
- Web content scraping
**Fully offline**: Possible with local models (Ollama) for basic functionality.
### What file types are supported?
**Documents**: PDF, DOCX, TXT, Markdown
**Web Content**: URLs, YouTube videos
**Media**: MP3, WAV, M4A (audio), MP4, AVI, MOV (video)
**Other**: Direct text input, CSV, code files
### How much does it cost?
**Software**: Free (open source)
**AI API costs**: Pay-per-use to providers:
- OpenAI: ~$0.50-5 per 1M tokens
- Anthropic: ~$3-75 per 1M tokens
- Google: Often free tier available
- Local models: Free after initial setup
**Typical monthly costs**: $5-50 for moderate usage.
---
## AI Models and Providers
### Which AI provider should I choose?
**For beginners**: OpenAI (reliable, well-documented)
**For privacy**: Local models (Ollama) or European providers (Mistral)
**For cost optimization**: Groq, Google (free tier), or OpenRouter
**For long context**: Anthropic (200K tokens) or Google Gemini (1M tokens)
### Can I use multiple providers?
**Yes**: Configure different providers for different tasks:
- OpenAI for chat
- Google for embeddings
- ElevenLabs for text-to-speech
- Anthropic for complex reasoning
### What are the best model combinations?
**Budget-friendly**:
- Language: `gpt-4o-mini` (OpenAI) or `deepseek-chat`
- Embedding: `text-embedding-3-small` (OpenAI)
**High-quality**:
- Language: `claude-3-5-sonnet` (Anthropic) or `gpt-4o` (OpenAI)
- Embedding: `text-embedding-3-large` (OpenAI)
**Privacy-focused**:
- Language: Local Ollama models (mistral, llama3)
- Embedding: Local embedding models
### How do I optimize AI costs?
**Model selection**:
- Use smaller models for simple tasks (gpt-4o-mini, claude-3-5-haiku)
- Use larger models only for complex reasoning
- Leverage free tiers when available
**Usage optimization**:
- Use "Summary Only" context for background sources
- Ask more specific questions
- Use local models (Ollama) for frequent tasks
---
## Data Management
### Where is my data stored?
**Local storage**: By default, all data is stored locally:
- Database: SurrealDB files in `surreal_data/`
- Uploads: Files in `data/uploads/`
- Podcasts: Generated audio in `data/podcasts/`
- No external data transmission (except to chosen AI providers)
### How do I backup my data?
```bash
# Create backup
tar -czf backup-$(date +%Y%m%d).tar.gz data/ surreal_data/
# Restore backup
tar -xzf backup-20240101.tar.gz
```
### Can I sync data between devices?
**Currently**: No built-in sync functionality.
**Workarounds**:
- Use shared network storage for data directories
- Manual backup/restore between devices
### What happens if I delete a notebook?
**Soft deletion**: Notebooks are marked as archived, not permanently deleted.
**Recovery**: Archived notebooks can be restored from the database.
---
## Best Practices
### How should I organize my notebooks?
- **By topic**: Separate notebooks for different research areas
- **By project**: One notebook per project or course
- **By time period**: Monthly or quarterly notebooks
**Recommended size**: 20-100 sources per notebook for best performance.
### How do I get the best search results?
- Use descriptive queries ("data analysis methods" not just "data")
- Combine multiple related terms
- Use natural language (ask questions as you would to a human)
- Try both text search (keywords) and vector search (concepts)
### How can I improve chat responses?
- Provide context: Reference specific sources or topics
- Be specific: Ask detailed questions rather than general ones
- Request citations: "Answer with page citations"
- Use follow-up questions: Build on previous responses
### What are the security best practices?
- Never share API keys publicly
- Use `OPEN_NOTEBOOK_PASSWORD` for public deployments
- Use HTTPS for production (via reverse proxy)
- Keep Docker images updated
- Encrypt backups if they contain sensitive data
---
## Technical Questions
### Can I use Open Notebook programmatically?
**Yes**: Open Notebook provides a REST API:
- Full API documentation at `http://localhost:5055/docs`
- Support for all UI functionality
- Authentication via password header
### Can I run Open Notebook in production?
**Yes**: Designed for production use with:
- Docker deployment
- Security features (password protection)
- Monitoring and logging
- Reverse proxy support (nginx, Caddy, Traefik)
### What are the system requirements?
**Minimum**:
- 4GB RAM
- 2 CPU cores
- 10GB disk space
**Recommended**:
- 8GB+ RAM
- 4+ CPU cores
- SSD storage
- For local models: 16GB+ RAM, GPU recommended
---
## Timeout and Performance
### Why do I get timeout errors?
**Common causes**:
- Large context (too many sources)
- Slow AI provider
- Local models on CPU (slow)
- First request (model loading)
**Solutions**:
```bash
# In .env:
API_CLIENT_TIMEOUT=600 # 10 minutes for slow setups
ESPERANTO_LLM_TIMEOUT=180 # 3 minutes for model inference
```
### Recommended timeouts by setup:
| Setup | API_CLIENT_TIMEOUT |
|-------|-------------------|
| Cloud APIs (OpenAI, Anthropic) | 300 (default) |
| Local Ollama with GPU | 600 |
| Local Ollama with CPU | 1200 |
| Remote LM Studio | 900 |
---
## Getting Help
### My question isn't answered here
1. Check the troubleshooting guides in this section
2. Search existing GitHub issues
3. Ask in the Discord community
4. Create a GitHub issue with detailed information
### How do I report a bug?
Include:
- Steps to reproduce
- Expected vs actual behavior
- Error messages and logs
- System information
- Configuration details (without API keys)
Submit to: [GitHub Issues](https://github.com/lfnovo/open-notebook/issues)
### Where can I get help?
- **Discord**: https://discord.gg/37XJPXfz2w (fastest)
- **GitHub Issues**: Bug reports and feature requests
- **Documentation**: This docs site
---
## Related
- [Quick Fixes](quick-fixes.md) - Common issues with 1-minute solutions
- [AI & Chat Issues](ai-chat-issues.md) - Model and chat problems
- [Connection Issues](connection-issues.md) - Network and API problems
docs/6-TROUBLESHOOTING/index.md
+240
View File
@@ -0,0 +1,240 @@
# Troubleshooting - Problem Solving Guide
Having issues? Use this guide to diagnose and fix problems.
---
## How to Use This Guide
**Step 1: Identify your problem**
- What's the symptom? (error message, behavior, something not working?)
- When did it happen? (during install, while using, after update?)
**Step 2: Find the right guide**
- Look below for your symptom
- Go to the specific troubleshooting guide
**Step 3: Follow the steps**
- Guides are organized by symptom, not by root cause
- Each has diagnostic steps and solutions
---
## Quick Problem Map
### During Installation
- **Docker won't start** → [Quick Fixes](quick-fixes.md#9-services-wont-start-or-docker-error)
- **Port already in use** → [Quick Fixes](quick-fixes.md#3-port-x-already-in-use)
- **Permission denied** → [Quick Fixes](quick-fixes.md#9-services-wont-start-or-docker-error)
- **Can't connect to database** → [Connection Issues](connection-issues.md)
### When Starting
- **API won't start** → [Quick Fixes](quick-fixes.md#9-services-wont-start-or-docker-error)
- **Frontend won't load** → [Connection Issues](connection-issues.md)
- **"Cannot connect to server" error** → [Connection Issues](connection-issues.md)
### Settings / Configuration
- **Models not showing** → [AI & Chat Issues](ai-chat-issues.md)
- **"Invalid API key"** → [AI & Chat Issues](ai-chat-issues.md)
- **Can't find Settings** → [Quick Fixes](quick-fixes.md)
### Using Features
- **Chat not working** → [AI & Chat Issues](ai-chat-issues.md)
- **Chat responses are slow** → [AI & Chat Issues](ai-chat-issues.md)
- **Chat gives bad answers** → [AI & Chat Issues](ai-chat-issues.md)
### Adding Content
- **Can't upload PDF** → [Quick Fixes](quick-fixes.md#4-cannot-process-file-or-unsupported-format)
- **File won't process** → [Quick Fixes](quick-fixes.md#4-cannot-process-file-or-unsupported-format)
- **Web link won't extract** → [Quick Fixes](quick-fixes.md#4-cannot-process-file-or-unsupported-format)
### Search
- **Search returns no results** → [Quick Fixes](quick-fixes.md#7-search-returns-nothing)
- **Search returns wrong results** → [Quick Fixes](quick-fixes.md#7-search-returns-nothing)
### Podcasts
- **Can't generate podcast** → [Quick Fixes](quick-fixes.md#8-podcast-generation-failed)
- **Podcast shows "FAILED" badge** → Check the error message displayed on the episode, then use the **Retry** button. See [Podcasts Explained](../2-CORE-CONCEPTS/podcasts-explained.md#when-things-go-wrong-failures--retry)
- **Podcast audio is robotic** → [Quick Fixes](quick-fixes.md#8-podcast-generation-failed)
- **Podcast generation times out** → [Quick Fixes](quick-fixes.md#8-podcast-generation-failed)
---
## Troubleshooting by Error Message
### "Cannot connect to server"
→ [Connection Issues](connection-issues.md) — Frontend can't reach API
### "Invalid API key"
→ [AI & Chat Issues](ai-chat-issues.md) — Wrong or missing API key
### "Models not available"
→ [AI & Chat Issues](ai-chat-issues.md) — Model not configured
### "Connection refused"
→ [Connection Issues](connection-issues.md) — Service not running or port wrong
### "Port already in use"
→ [Quick Fixes](quick-fixes.md#3-port-x-already-in-use) — Port conflict
### "Permission denied"
→ [Quick Fixes](quick-fixes.md#9-services-wont-start-or-docker-error) — File permissions issue
### "Unsupported file type"
→ [Quick Fixes](quick-fixes.md#4-cannot-process-file-or-unsupported-format) — File format not supported
### "Processing timeout"
→ [Quick Fixes](quick-fixes.md#5-chat-is-very-slow) — File too large or slow processing
---
## Troubleshooting by Component
### Frontend (Browser/UI)
- Can't access UI → [Connection Issues](connection-issues.md)
- UI is slow → [Quick Fixes](quick-fixes.md)
- Button/feature missing → [Quick Fixes](quick-fixes.md)
### API (Backend)
- API won't start → [Quick Fixes](quick-fixes.md#9-services-wont-start-or-docker-error)
- API errors in logs → [Quick Fixes](quick-fixes.md#9-services-wont-start-or-docker-error)
- API is slow → [Quick Fixes](quick-fixes.md)
### Database
- Can't connect to database → [Connection Issues](connection-issues.md)
- Data lost after restart → [FAQ](faq.md#how-do-i-backup-my-data)
### AI / Chat
- Chat not working → [AI & Chat Issues](ai-chat-issues.md)
- Bad responses → [AI & Chat Issues](ai-chat-issues.md)
- Cost too high → [AI & Chat Issues](ai-chat-issues.md#high-api-costs)
### Sources
- Can't upload file → [Quick Fixes](quick-fixes.md#4-cannot-process-file-or-unsupported-format)
- File won't process → [Quick Fixes](quick-fixes.md#4-cannot-process-file-or-unsupported-format)
### Podcasts
- Won't generate → [Quick Fixes](quick-fixes.md#8-podcast-generation-failed)
- Bad audio quality → [Quick Fixes](quick-fixes.md#8-podcast-generation-failed)
---
## Diagnostic Checklist
**When something isn't working:**
- [ ] Check if services are running: `docker ps`
- [ ] Check logs: `docker compose logs api` (or frontend, surrealdb)
- [ ] Verify ports are exposed: `netstat -tlnp` or `lsof -i :5055`
- [ ] Test connectivity: `curl http://localhost:5055/health`
- [ ] Check environment variables: `docker inspect <container>`
- [ ] Try restarting: `docker compose restart`
- [ ] Check firewall/antivirus isn't blocking
---
## Getting Help
If you can't find the answer here:
1. **Check the relevant guide** — Read completely, try all steps
2. **Check the FAQ** — [Frequently Asked Questions](faq.md)
3. **Search our Discord** — Others may have had same issue
4. **Check logs** — Most issues show error messages in logs
5. **Report on GitHub** — Include error message, steps to reproduce
### How to Report an Issue
Include:
1. Error message (exact)
2. Steps to reproduce
3. Logs: `docker compose logs`
4. Your setup: Docker/local, provider, OS
5. What you've already tried
→ [Report on GitHub](https://github.com/lfnovo/open-notebook/issues)
---
## Guides
### [Quick Fixes](quick-fixes.md)
Top 10 most common issues with 1-minute solutions.
### [Connection Issues](connection-issues.md)
Frontend can't reach API, network problems.
### [AI & Chat Issues](ai-chat-issues.md)
Chat not working, bad responses, slow performance.
### [FAQ](faq.md)
Frequently asked questions about usage, costs, and best practices.
---
## Common Solutions
**Service won't start?**
```bash
# Check logs
docker compose logs
# Restart everything
docker compose restart
# Nuclear option: rebuild
docker compose down
docker compose up --build
```
**Port conflict?**
```bash
# Find what's using port 5055
lsof -i :5055
# Kill it or use different port
```
**Can't connect?**
```bash
# Test API directly
curl http://localhost:5055/health
# Should return: {"status":"ok"}
```
**Slow performance?**
```bash
# Check resource usage
docker stats
# Reduce concurrency in .env
SURREAL_COMMANDS_MAX_TASKS=2
```
**High costs?**
```bash
# Switch to cheaper model
# In Settings → Models → Choose gpt-4o-mini (OpenAI)
# Or use Ollama (free)
```
---
## Still Stuck?
**Before asking for help:**
1. Read the relevant guide completely
2. Try all the steps
3. Check the logs
4. Restart services
5. Search existing issues on GitHub
**Then:**
- **Discord**: https://discord.gg/37XJPXfz2w (fastest response)
- **GitHub Issues**: https://github.com/lfnovo/open-notebook/issues
docs/6-TROUBLESHOOTING/quick-fixes.md
+372
View File
@@ -0,0 +1,372 @@
# Quick Fixes - Top 11 Issues & Solutions
Common problems with 1-minute solutions.
---
## #1: "Cannot connect to server"
**Symptom:** Browser shows error "Cannot connect to server" or "Unable to reach API"
**Cause:** Frontend can't reach API
**Solution (1 minute):**
```bash
# Step 1: Check if API is running
docker ps | grep api
# Step 2: Verify port 5055 is accessible
curl http://localhost:5055/health
# Expected output: {"status":"ok"}
# If that doesn't work:
# Step 3: Restart services
docker compose restart
# Step 4: Try again
# Open http://localhost:8502 in browser
```
**If still broken:**
- Check `API_URL` in .env (should match your frontend URL)
- See [Connection Issues](connection-issues.md)
---
## #2: "Invalid API key" or "Models not showing"
**Symptom:** Settings → Models shows "No models available"
**Cause:** No credential configured, or credential has invalid API key
**Solution (1 minute):**
```
1. Go to Settings → API Keys
2. If no credential exists, click "Add Credential" and add one
3. If a credential exists, click "Test Connection"
4. If test fails, delete and re-create with correct key
5. After test passes, click "Discover Models" → "Register Models"
6. Go to Settings → Models to verify models appear
```
**If still broken:**
- Make sure key has no extra spaces
- Generate a fresh key from provider dashboard
- Check that `OPEN_NOTEBOOK_ENCRYPTION_KEY` is set in docker-compose.yml
- See [AI & Chat Issues](ai-chat-issues.md)
---
## #3: "Port X already in use"
**Symptom:** Docker error "Port 8502 is already allocated"
**Cause:** Another service using that port
**Solution (1 minute):**
```bash
# Option 1: Stop the other service
# Find what's using port 8502
lsof -i :8502
# Kill it or close the app
# Option 2: Use different port
# Edit docker-compose.yml
# Change: - "8502:8502"
# To: - "8503:8502"
# Then restart
docker compose restart
# Access at: http://localhost:8503
```
---
## #4: "Cannot process file" or "Unsupported format"
**Symptom:** Upload fails or says "File format not supported"
**Cause:** File type not supported or too large
**Solution (1 minute):**
```bash
# Check if file format is supported:
# ✓ PDF, DOCX, PPTX, XLSX (documents)
# ✓ MP3, WAV, M4A (audio)
# ✓ MP4, AVI, MOV (video)
# ✓ URLs/web links
# ✗ Pure images (.jpg without OCR)
# ✗ Files > 100MB
# Try these:
# - Convert to PDF if possible
# - Split large files
# - Try uploading again
```
---
## #5: "Chat is very slow"
**Symptom:** Chat responses take minutes or timeout
**Cause:** Slow AI provider, large context, or overloaded system
**Solution (1 minute):**
```bash
# Step 1: Check which model you're using
# Settings → Models
# Note the model name
# Step 2: Try a cheaper/faster model
# OpenAI: Switch to gpt-4o-mini (10x cheaper, slightly faster)
# Anthropic: Switch to claude-3-5-haiku (fastest)
# Groq: Use any model (ultra-fast)
# Step 3: Reduce context
# Chat: Select fewer sources
# Use "Summary Only" instead of "Full Content"
# Step 4: Check if API is overloaded
docker stats
# Look at CPU/memory usage
```
For deep dive: See [AI & Chat Issues](ai-chat-issues.md)
---
## #6: "Chat gives bad responses"
**Symptom:** AI responses are generic, wrong, or irrelevant
**Cause:** Bad context, vague question, or wrong model
**Solution (1 minute):**
```bash
# Step 1: Make sure sources are in context
# Click "Select Sources" in Chat
# Verify relevant sources are checked and set to "Full Content"
# Step 2: Ask a specific question
# Bad: "What do you think?"
# Good: "Based on the paper's methodology section, what are the 3 main limitations?"
# Step 3: Try a more powerful model
# OpenAI: Use gpt-4o (better reasoning)
# Anthropic: Use claude-3-5-sonnet (best reasoning)
# Step 4: Check citations
# Click citations to verify AI actually saw those sources
```
For detailed help: See [Chat Effectively](../3-USER-GUIDE/chat-effectively.md)
---
## #7: "Search returns nothing"
**Symptom:** Search shows 0 results even though content exists
**Cause:** Wrong search type or poor query
**Solution (1 minute):**
```bash
# Try a different search type:
# If you searched with KEYWORDS:
# Try VECTOR SEARCH instead
# (Concept-based, not keyword-based)
# If you searched for CONCEPTS:
# Try TEXT SEARCH instead
# (Look for specific words in your query)
# Try simpler search:
# Instead of: "How do transformers work in neural networks?"
# Try: "transformers" or "neural networks"
# Check sources are processed:
# Go to notebook
# All sources should show green "Ready" status
```
For detailed help: See [Search Effectively](../3-USER-GUIDE/search.md)
---
## #8: "Podcast generation failed"
**Symptom:** "Podcast generation failed" error
**Cause:** Insufficient content, API quota, or network issue
**Solution (1 minute):**
```bash
# Step 1: Make sure you have content
# Select at least 1-2 sources
# Avoid single-sentence sources
# Step 2: Try again
# Sometimes it's a temporary API issue
# Wait 30 seconds and retry
# Step 3: Check your TTS provider has quota
# OpenAI: Check account has credits
# ElevenLabs: Check monthly quota
# Google: Check API quota
# Step 4: Try different TTS provider
# In podcast generation, choose "Google" or "Local"
# instead of "ElevenLabs"
```
For detailed help: See [FAQ](faq.md)
---
## #9: "Services won't start" or Docker error
**Symptom:** Docker error when running `docker compose up`
**Cause:** Corrupt configuration, permission issue, or resource issue
**Solution (1 minute):**
```bash
# Step 1: Check logs
docker compose logs
# Step 2: Try restart
docker compose restart
# Step 3: If that fails, rebuild
docker compose down
docker compose up --build
# Step 4: Check disk space
df -h
# Need at least 5GB free
# Step 5: Check Docker has enough memory
# Docker settings → Resources → Memory: 4GB+
```
---
## #10: "Database says 'too many connections'"
**Symptom:** Error about database connections
**Cause:** Too many concurrent operations
**Solution (1 minute):**
```bash
# In .env, reduce concurrency:
SURREAL_COMMANDS_MAX_TASKS=2
# Then restart:
docker compose restart
# This makes it slower but more stable
```
---
## #11: Slow Startup or Download Timeouts (China/Slow Networks)
**Symptom:** Container crashes on startup, worker enters FATAL state, or pip/uv downloads fail
**Cause:** Slow network or restricted access to Python package repositories
**Solution:**
### Increase Download Timeout
```yaml
# In docker-compose.yml environment:
environment:
- UV_HTTP_TIMEOUT=600 # 10 minutes (default is 30s)
```
### Use Chinese Mirrors (if in China)
```yaml
environment:
- UV_HTTP_TIMEOUT=600
- UV_INDEX_URL=https://pypi.tuna.tsinghua.edu.cn/simple
- PIP_INDEX_URL=https://pypi.tuna.tsinghua.edu.cn/simple
```
**Alternative Chinese mirrors:**
- Tsinghua: `https://pypi.tuna.tsinghua.edu.cn/simple`
- Aliyun: `https://mirrors.aliyun.com/pypi/simple/`
- Huawei: `https://repo.huaweicloud.com/repository/pypi/simple`
**Note:** First startup may take several minutes while dependencies download. Subsequent starts will be faster.
---
## Quick Troubleshooting Checklist
When something breaks:
- [ ] **Restart services:** `docker compose restart`
- [ ] **Check logs:** `docker compose logs`
- [ ] **Verify connectivity:** `curl http://localhost:5055/health`
- [ ] **Check .env:** API keys set? API_URL correct?
- [ ] **Check resources:** `docker stats` (CPU/memory)
- [ ] **Clear cache:** `docker system prune` (free space)
- [ ] **Rebuild if needed:** `docker compose up --build`
---
## Nuclear Options (Last Resort)
**Completely reset (will lose all data in Docker):**
```bash
docker compose down -v
docker compose up --build
```
**Reset to defaults:**
```bash
# Backup your .env first!
cp .env .env.backup
# Reset to example
cp .env.example .env
# Edit with your API keys
# Restart
docker compose up
```
---
## Prevention Tips
1. **Keep backups** — Export your notebooks regularly
2. **Monitor logs** — Check `docker compose logs` periodically
3. **Update regularly** — Pull latest image: `docker pull lfnovo/open_notebook:latest`
4. **Document changes** — Keep notes on what you configured
5. **Test after updates** — Verify everything works
---
## Still Stuck?
- **Look up your exact error** in [Troubleshooting Index](index.md)
- **Check the FAQ** in [FAQ](faq.md)
- **Check logs:** `docker compose logs | head -50`
- **Ask for help:** [Discord](https://discord.gg/37XJPXfz2w) or [GitHub Issues](https://github.com/lfnovo/open-notebook/issues)
docs/7-DEVELOPMENT/api-reference.md
+223
View File
@@ -0,0 +1,223 @@
# API Reference
Complete REST API for Open Notebook. All endpoints are served from the API backend (default: `http://localhost:5055`).
**Base URL**: `http://localhost:5055` (development) or environment-specific production URL
**Interactive Docs**: Use FastAPI's built-in Swagger UI at `http://localhost:5055/docs` for live testing and exploration. This is the primary reference for all endpoints, request/response schemas, and real-time testing.
---
## Quick Start
### 1. Authentication
Simple password-based (development only):
```bash
curl http://localhost:5055/api/notebooks \
-H "Authorization: Bearer your_password"
```
**⚠️ Production**: Replace with OAuth/JWT. See [Security Configuration](../5-CONFIGURATION/security.md) for details.
### 2. Base API Flow
Most operations follow this pattern:
1. Create a **Notebook** (container for research)
2. Add **Sources** (PDFs, URLs, text)
3. Query via **Chat** or **Search**
4. View results and **Notes**
### 3. Testing Endpoints
Instead of memorizing endpoints, use the interactive API docs:
- Navigate to `http://localhost:5055/docs`
- Try requests directly in the browser
- See request/response schemas in real-time
- Test with your own data
---
## API Endpoints Overview
### Main Resource Types
**Notebooks** - Research projects containing sources and notes
- `GET/POST /notebooks` - List and create
- `GET/PUT/DELETE /notebooks/{id}` - Read, update, delete
**Sources** - Content items (PDFs, URLs, text)
- `GET/POST /sources` - List and add content
- `GET /sources/{id}` - Fetch source details
- `POST /sources/{id}/retry` - Retry failed processing
- `GET /sources/{id}/download` - Download original file
**Notes** - User-created or AI-generated research notes
- `GET/POST /notes` - List and create
- `GET/PUT/DELETE /notes/{id}` - Read, update, delete
**Chat** - Conversational AI interface
- `GET/POST /chat/sessions` - Manage chat sessions
- `POST /chat/execute` - Send message and get response
- `POST /chat/context` - Prepare context for chat
**Search** - Find content by text or semantic similarity
- `POST /search` - Full-text or vector search
- `POST /ask` - Ask a question (search + synthesize)
**Transformations** - Custom prompts for extracting insights
- `GET/POST /transformations` - Create custom extraction rules
- `POST /sources/{id}/insights` - Apply transformation to source
**Models** - Configure AI providers
- `GET /models` - Available models
- `GET /models/defaults` - Current defaults
- `POST /models/config` - Set defaults
**Credentials** - Manage AI provider credentials
- `GET/POST /credentials` - List and create credentials
- `GET/PUT/DELETE /credentials/{id}` - CRUD operations
- `POST /credentials/{id}/test` - Test connection
- `POST /credentials/{id}/discover` - Discover models from provider
- `POST /credentials/{id}/register-models` - Register discovered models
- `GET /credentials/status` - Provider status overview
- `GET /credentials/env-status` - Environment variable status
- `POST /credentials/migrate-from-env` - Migrate env vars to credentials
**Health & Status**
- `GET /health` - Health check
- `GET /commands/{id}` - Track async operations
---
## Authentication
### Current (Development)
All requests require password header:
```bash
curl -H "Authorization: Bearer your_password" http://localhost:5055/api/notebooks
```
Password configured via `OPEN_NOTEBOOK_PASSWORD` environment variable.
> **📖 See [Security Configuration](../5-CONFIGURATION/security.md)** for complete authentication setup, API examples, and production hardening.
### Production
**⚠️ Not secure.** Replace with:
- OAuth 2.0 (recommended)
- JWT tokens
- API keys
See [Security Configuration](../5-CONFIGURATION/security.md) for production setup.
---
## Common Patterns
### Pagination
```bash
# List sources with limit/offset
curl 'http://localhost:5055/sources?limit=20&offset=10'
```
### Filtering & Sorting
```bash
# Filter by notebook, sort by date
curl 'http://localhost:5055/sources?notebook_id=notebook:abc&sort_by=created&sort_order=asc'
```
### Async Operations
Some operations (source processing, podcast generation) return immediately with a command ID:
```bash
# Submit async operation
curl -X POST http://localhost:5055/sources -F async_processing=true
# Response: {"id": "source:src001", "command_id": "command:cmd123"}
# Poll status
curl http://localhost:5055/commands/command:cmd123
```
### Streaming Responses
The `/ask` endpoint streams responses as Server-Sent Events:
```bash
curl -N 'http://localhost:5055/ask' \
-H "Content-Type: application/json" \
-d '{"question": "What is AI?"}'
# Outputs: data: {"type":"strategy",...}
# data: {"type":"answer",...}
# data: {"type":"final_answer",...}
```
### Multipart File Upload
```bash
curl -X POST http://localhost:5055/sources \
-F "type=upload" \
-F "notebook_id=notebook:abc" \
-F "file=@document.pdf"
```
---
## Error Handling
All errors return JSON with status code:
```json
{"detail": "Notebook not found"}
```
### Common Status Codes
| Code | Meaning | Example |
|------|---------|---------|
| 200 | Success | Operation completed |
| 400 | Bad Request | Invalid input |
| 404 | Not Found | Resource doesn't exist |
| 409 | Conflict | Resource already exists |
| 500 | Server Error | Database/processing error |
---
## Tips for Developers
1. **Start with interactive docs** (`http://localhost:5055/docs`) - this is the definitive reference
2. **Enable logging** for debugging (check API logs: `docker logs`)
3. **Streaming endpoints** require special handling (Server-Sent Events, not standard JSON)
4. **Async operations** return immediately; always poll status before assuming completion
5. **Vector search** requires embedding model configured (check `/models`)
6. **Model overrides** are per-request; set in body, not config
7. **CORS enabled** in development; configure for production
---
## Learning Path
1. **Authentication**: Add `X-Password` header to all requests
2. **Create a notebook**: `POST /notebooks` with name and description
3. **Add a source**: `POST /sources` with file, URL, or text
4. **Query your content**: `POST /chat/execute` to ask questions
5. **Explore advanced features**: Search, transformations, streaming
---
## Production Considerations
- Replace password auth with OAuth/JWT (see [Security](../5-CONFIGURATION/security.md))
- Add rate limiting via reverse proxy (Nginx, CloudFlare, Kong)
- Enable CORS restrictions (currently allows all origins)
- Use HTTPS via reverse proxy (see [Reverse Proxy](../5-CONFIGURATION/reverse-proxy.md))
- Set up API versioning strategy (currently implicit)
See [Security Configuration](../5-CONFIGURATION/security.md) and [Reverse Proxy Setup](../5-CONFIGURATION/reverse-proxy.md) for complete production setup.
docs/7-DEVELOPMENT/architecture.md
+891
View File
@@ -0,0 +1,891 @@
# Open Notebook Architecture
## High-Level Overview
Open Notebook follows a three-tier architecture with clear separation of concerns:
```
┌─────────────────────────────────────────────────────────┐
│ Your Browser │
│ Access: http://your-server-ip:8502 │
└────────────────┬────────────────────────────────────────┘
│
▼
┌───────────────┐
│ Port 8502 │ ← Next.js Frontend (what you see)
│ Frontend │ Also proxies API requests internally!
└───────┬───────┘
│ proxies /api/* requests ↓
▼
┌───────────────┐
│ Port 5055 │ ← FastAPI Backend (handles requests)
│ API │
└───────┬───────┘
│
▼
┌───────────────┐
│ SurrealDB │ ← Database (internal, auto-configured)
│ (Port 8000) │
└───────────────┘
```
**Key Points:**
- **v1.1+**: Next.js automatically proxies `/api/*` requests to the backend, simplifying reverse proxy setup
- Your browser loads the frontend from port 8502
- The frontend needs to know where to find the API - when accessing remotely, set: `API_URL=http://your-server-ip:5055`
- **Behind reverse proxy?** You only need to proxy to port 8502 now! See [Reverse Proxy Configuration](../5-CONFIGURATION/reverse-proxy.md)
---
## Detailed Architecture
Open Notebook is built on a **three-tier, async-first architecture** designed for scalability, modularity, and multi-provider AI flexibility. The system separates concerns across frontend, API, and database layers, with LangGraph powering intelligent workflows and Esperanto enabling seamless integration with 17 AI providers.
**Core Philosophy**:
- Privacy-first: Users control their data and AI provider choice
- Async/await throughout: Non-blocking operations for responsive UX
- Domain-Driven Design: Clear separation between domain models, repositories, and orchestrators
- Multi-provider flexibility: Swap AI providers without changing application code
- Self-hosted capable: All components deployable in isolated environments
---
## Three-Tier Architecture
### Layer 1: Frontend (React/Next.js @ port 3000)
**Purpose**: Responsive, interactive user interface for research, notes, chat, and podcast management.
**Technology Stack**:
- **Framework**: Next.js 15 with React 19
- **Language**: TypeScript with strict type checking
- **State Management**: Zustand (lightweight store) + TanStack Query (server state)
- **Styling**: Tailwind CSS + Shadcn/ui component library
- **Build Tool**: Webpack (bundled via Next.js)
**Key Responsibilities**:
- Render notebooks, sources, notes, chat sessions, and podcasts
- Handle user interactions (create, read, update, delete operations)
- Manage complex UI state (modals, file uploads, real-time search)
- Stream responses from API (chat, podcast generation)
- Display embeddings, vector search results, and insights
**Communication Pattern**:
- All data fetched via REST API (async requests to port 5055)
- Configured base URL: `http://localhost:5055` (dev) or environment-specific (prod)
- TanStack Query handles caching, refetching, and data synchronization
- Zustand stores global state (user, notebooks, selected context)
- CORS enabled on API side for cross-origin requests
**Component Architecture**:
- `/src/app/`: Next.js App Router (pages, layouts)
- `/src/components/`: Reusable React components (buttons, forms, cards)
- `/src/hooks/`: Custom hooks (useNotebook, useChat, useSearch)
- `/src/lib/`: Utility functions, API clients, validators
- `/src/styles/`: Global CSS, Tailwind config
---
### Layer 2: API (FastAPI @ port 5055)
**Purpose**: RESTful backend exposing operations on notebooks, sources, notes, chat sessions, and AI models.
**Technology Stack**:
- **Framework**: FastAPI 0.104+ (async Python web framework)
- **Language**: Python 3.11+
- **Validation**: Pydantic v2 (request/response schemas)
- **Logging**: Loguru (structured JSON logging)
- **Testing**: Pytest (unit and integration tests)
**Architecture**:
```
FastAPI App (main.py)
├── Routers (HTTP endpoints)
│ ├── routers/notebooks.py (CRUD operations)
│ ├── routers/sources.py (content ingestion, upload)
│ ├── routers/notes.py (note management)
│ ├── routers/chat.py (conversation sessions)
│ ├── routers/search.py (full-text + vector search)
│ ├── routers/transformations.py (custom transformations)
│ ├── routers/models.py (AI model configuration)
│ └── routers/*.py (11 additional routers)
│
├── Services (business logic)
│ ├── *_service.py (orchestration, graph invocation)
│ ├── command_service.py (async job submission)
│ └── middleware (auth, logging)
│
├── Models (Pydantic schemas)
│ └── models.py (validation, serialization)
│
└── Lifespan (startup/shutdown)
└── AsyncMigrationManager (database schema migrations)
```
**Key Responsibilities**:
1. **HTTP Interface**: Accept REST requests, validate, return JSON responses
2. **Business Logic**: Orchestrate domain models, repository operations, and workflows
3. **Async Job Queue**: Submit long-running tasks (podcast generation, source processing)
4. **Database Migrations**: Run schema updates on startup
5. **Error Handling**: Catch exceptions, return appropriate HTTP status codes
6. **Logging**: Track operations for debugging and monitoring
**Startup Flow**:
1. Load `.env` environment variables
2. Initialize FastAPI app with CORS + auth middleware
3. Run AsyncMigrationManager (creates/updates database schema)
4. Register all routers (20+ endpoints)
5. Server ready on port 5055
**Request-Response Cycle**:
```
HTTP Request → Router → Service → Domain/Repository → SurrealDB
↓
LangGraph (optional)
↓
Response ← Pydantic serialization ← Service ← Result
```
---
### Layer 3: Database (SurrealDB @ port 8000)
**Purpose**: Graph database with built-in vector embeddings, semantic search, and relationship management.
**Technology Stack**:
- **Database**: SurrealDB (multi-model, ACID transactions)
- **Query Language**: SurrealQL (SQL-like syntax with graph operations)
- **Async Driver**: Async Rust client for Python
- **Migrations**: `.surrealql` files in `open_notebook/database/migrations/`, registered in `AsyncMigrationManager` (auto-run on API startup)
**Core Tables**:
| Table | Purpose | Key Fields |
|-------|---------|-----------|
| `notebook` | Research project container | id, name, description, archived, created, updated |
| `source` | Content item (PDF, URL, text) | id, title, full_text, topics, asset, created, updated |
| `source_embedding` | Vector embeddings for semantic search | id, source, embedding, chunk_text, chunk_index |
| `note` | User-created research notes | id, title, content, note_type (human/ai), created, updated |
| `chat_session` | Conversation session | id, notebook_id, title, messages (JSON), created, updated |
| `transformation` | Custom transformation rules | id, name, description, prompt, created, updated |
| `source_insight` | Transformation output | id, source_id, insight_type, content, created, updated |
| `reference` | Relationship: source → notebook | out (source), in (notebook) |
| `artifact` | Relationship: note → notebook | out (note), in (notebook) |
**Relationship Graph**:
```
Notebook
↓ (referenced_by)
Source
├→ SourceEmbedding (1:many for chunked text)
├→ SourceInsight (1:many for transformation outputs)
└→ Note (via artifact relationship)
├→ Embedding (semantic search)
└→ Topics (tags)
ChatSession
├→ Notebook
└→ Messages (stored as JSON array)
```
**Vector Search Capability**:
- Embeddings stored natively in SurrealDB
- Full-text search on `source.full_text` and `note.content`
- Cosine similarity search on embedding vectors
- Semantic search integrates with search endpoint
**Connection Management**:
- Async connection pooling (configurable size)
- Transaction support for multi-record operations
- Schema auto-validation via migrations
- Query timeout protection (prevent infinite queries)
---
## Tech Stack Rationale
### Why Python + FastAPI?
**Python**:
- Rich AI/ML ecosystem (LangChain, LangGraph, transformers, scikit-learn)
- Rapid prototyping and deployment
- Extensive async support (asyncio, async/await)
- Strong type hints (Pydantic, mypy)
**FastAPI**:
- Modern, async-first framework
- Automatic OpenAPI documentation (Swagger UI @ /docs)
- Built-in request validation (Pydantic)
- Excellent performance (benchmarked near C/Rust speeds)
- Easy middleware/dependency injection
### Why Next.js + React + TypeScript?
**Next.js**:
- Full-stack React framework with SSR/SSG
- File-based routing (intuitive project structure)
- Built-in API routes (optional backend co-location)
- Optimized image/code splitting
- Easy deployment (Vercel, Docker, self-hosted)
**React 19**:
- Component-based UI (reusable, testable)
- Excellent tooling and community
- Client-side state management (Zustand)
- Server-side state sync (TanStack Query)
**TypeScript**:
- Type safety catches errors at compile time
- Better IDE autocomplete and refactoring
- Documentation via types (self-documenting code)
- Easier onboarding for new contributors
### Why SurrealDB?
**SurrealDB**:
- Native graph database (relationships are first-class)
- Built-in vector embeddings (no separate vector DB)
- ACID transactions (data consistency)
- Multi-model (relational + document + graph)
- Full-text search + semantic search in one query
- Self-hosted (unlike managed Pinecone/Weaviate)
- Flexible SurrealQL (SQL-like syntax)
**Alternative Considered**: PostgreSQL + pgvector (more mature but separate extensions)
### Why Esperanto for AI Providers?
**Esperanto Library**:
- Unified interface to 17 providers (OpenAI, Anthropic, Google, Groq, Ollama, Mistral, DeepSeek, xAI, OpenRouter, Azure, Vertex, and more)
- Multi-provider embeddings (OpenAI, Google, Ollama, Mistral, Voyage)
- TTS/STT integration (OpenAI, Groq, ElevenLabs, Google)
- Smart provider selection (fallback logic, cost optimization)
- Per-request model override support
- Local Ollama support (completely self-hosted option)
**Alternative Considered**: LangChain's provider abstraction (more verbose, less flexible)
---
## LangGraph Workflows
LangGraph is a state machine library that orchestrates multi-step AI workflows. Open Notebook uses five core workflows:
### 1. **Source Processing Workflow** (`open_notebook/graphs/source.py`)
**Purpose**: Ingest content (PDF, URL, text) and prepare for search/insights.
**Flow**:
```
Input (file/URL/text)
↓
Extract Content (content-core library)
↓
Clean & tokenize text
↓
Generate Embeddings (Esperanto)
↓
Create SourceEmbedding records (chunked + indexed)
↓
Extract Topics (LLM summarization)
↓
Save to SurrealDB
↓
Output (Source record with embeddings)
```
**State Dict**:
```python
{
"content_state": {"file_path" | "url" | "content": str},
"source_id": str,
"full_text": str,
"embeddings": List[Dict],
"topics": List[str],
"notebook_ids": List[str],
}
```
**Invoked By**: Sources API (`POST /sources`)
---
### 2. **Chat Workflow** (`open_notebook/graphs/chat.py`)
**Purpose**: Conduct multi-turn conversations with AI model, referencing notebook context.
**Flow**:
```
User Message
↓
Build Context (selected sources/notes)
↓
Add Message to Session
↓
Create Chat Prompt (system + history + context)
↓
Call LLM (via Esperanto)
↓
Stream Response
↓
Save AI Message to ChatSession
↓
Output (complete message)
```
**State Dict**:
```python
{
"session_id": str,
"messages": List[BaseMessage],
"context": Dict[str, Any], # sources, notes, snippets
"response": str,
"model_override": Optional[str],
}
```
**Key Features**:
- Message history persisted in SurrealDB (SqliteSaver checkpoint)
- Context building via `build_context_for_chat()` utility
- Token counting to prevent overflow
- Per-message model override support
**Invoked By**: Chat API (`POST /chat/execute`)
---
### 3. **Ask Workflow** (`open_notebook/graphs/ask.py`)
**Purpose**: Answer user questions by searching sources and synthesizing responses.
**Flow**:
```
User Question
↓
Plan Search Strategy (LLM generates searches)
↓
Execute Searches (vector + text search)
↓
Score & Rank Results
↓
Provide Answers (LLM synthesizes from results)
↓
Stream Responses
↓
Output (final answer)
```
**State Dict**:
```python
{
"question": str,
"strategy": SearchStrategy,
"answers": List[str],
"final_answer": str,
"sources_used": List[Source],
}
```
**Streaming**: Uses `astream()` to emit updates in real-time (strategy → answers → final answer)
**Invoked By**: Search API (`POST /ask` with streaming)
---
### 4. **Transformation Workflow** (`open_notebook/graphs/transformation.py`)
**Purpose**: Apply custom transformations to sources (extract summaries, key points, etc).
**Flow**:
```
Source + Transformation Rule
↓
Generate Prompt (Jinja2 template)
↓
Call LLM
↓
Parse Output
↓
Create SourceInsight record
↓
Output (insight with type + content)
```
**Example Transformations**:
- Summary (5-sentence overview)
- Key Points (bulleted list)
- Quotes (notable excerpts)
- Q&A (generated questions and answers)
**Invoked By**: Sources API (`POST /sources/{id}/insights`)
---
### 5. **Prompt Workflow** (`open_notebook/graphs/prompt.py`)
**Purpose**: Generic LLM task execution (e.g., auto-generate note titles, analyze content).
**Flow**:
```
Input Text + Prompt
↓
Call LLM (simple request-response)
↓
Output (completion)
```
**Used For**: Note title generation, content analysis, etc.
---
## AI Provider Integration Pattern
### ModelManager: Centralized Factory
Located in `open_notebook/ai/models.py`, ModelManager handles:
1. **Provider Detection**: Check environment variables for available providers
2. **Model Selection**: Choose best model based on context size and task
3. **Fallback Logic**: If primary provider unavailable, try backup
4. **Cost Optimization**: Prefer cheaper models for simple tasks
5. **Token Calculation**: Estimate cost before LLM call
**Usage**:
```python
from open_notebook.ai.provision import provision_langchain_model
# Get best LLM for context size
model = await provision_langchain_model(
task="chat", # or "search", "extraction"
model_override="anthropic/claude-opus-4", # optional
context_size=8000, # estimated tokens
)
# Invoke model
response = await model.ainvoke({"input": prompt})
```
### Multi-Provider Support
**LLM Providers**:
- OpenAI (gpt-4, gpt-4-turbo, gpt-3.5-turbo)
- Anthropic (claude-opus, claude-sonnet, claude-haiku)
- Google (gemini-3.5-flash, gemini-2.5-pro)
- Groq (mixtral, llama-2)
- Ollama (local models)
- Mistral (mistral-large, mistral-medium)
- DeepSeek (deepseek-chat)
- xAI (grok)
**Embedding Providers**:
- OpenAI (text-embedding-3-large, text-embedding-3-small)
- Google (embedding-001)
- Ollama (local embeddings)
- Mistral (mistral-embed)
- Voyage (voyage-large-2)
**TTS Providers**:
- OpenAI (tts-1, tts-1-hd)
- Groq (no TTS, fallback to OpenAI)
- ElevenLabs (multilingual voices)
- Google TTS (text-to-speech)
### Per-Request Override
Every LangGraph invocation accepts a `config` parameter to override models:
```python
result = await graph.ainvoke(
input={...},
config={
"configurable": {
"model_override": "anthropic/claude-opus-4" # Use Claude instead
}
}
)
```
---
## Design Patterns
### 1. **Domain-Driven Design (DDD)**
**Domain Objects** (`open_notebook/domain/`):
- `Notebook`: Research container with relationships to sources/notes
- `Source`: Content item (PDF, URL, text) with embeddings
- `Note`: User-created or AI-generated research note
- `ChatSession`: Conversation history for a notebook
- `Transformation`: Custom rule for extracting insights
**Repository Pattern**:
- Database access layer (`open_notebook/database/repository.py`)
- `repo_query()`: Execute SurrealQL queries
- `repo_create()`: Insert records
- `repo_upsert()`: Merge records
- `repo_delete()`: Remove records
**Entity Methods**:
```python
# Domain methods (business logic)
notebook = await Notebook.get(id)
await notebook.save()
notes = await notebook.get_notes()
sources = await notebook.get_sources()
```
### 2. **Async-First Architecture**
**All I/O is async**:
- Database queries: `await repo_query(...)`
- LLM calls: `await model.ainvoke(...)`
- File I/O: `await upload_file.read()`
- Graph invocations: `await graph.ainvoke(...)`
**Benefits**:
- Non-blocking request handling (FastAPI serves multiple concurrent requests)
- Better resource utilization (I/O waiting doesn't block CPU)
- Natural fit for Python async/await syntax
**Example**:
```python
@router.post("/sources")
async def create_source(source_data: SourceCreate):
# All operations are non-blocking
source = Source(title=source_data.title)
await source.save() # async database operation
await graph.ainvoke({...}) # async LangGraph invocation
return SourceResponse(...)
```
### 3. **Service Pattern**
Services orchestrate domain objects, repositories, and workflows:
```python
# illustrative example (see api/podcast_service.py for a real service)
class NotebookService:
async def get_notebook_with_stats(notebook_id: str):
notebook = await Notebook.get(notebook_id)
sources = await notebook.get_sources()
notes = await notebook.get_notes()
return {
"notebook": notebook,
"source_count": len(sources),
"note_count": len(notes),
}
```
**Responsibilities**:
- Validate inputs (Pydantic)
- Orchestrate database operations
- Invoke workflows (LangGraph graphs)
- Handle errors and return appropriate status codes
- Log operations
### 4. **Streaming Pattern**
For long-running operations (ask workflow, podcast generation), stream results as Server-Sent Events:
```python
@router.post("/ask", response_class=StreamingResponse)
async def ask(request: AskRequest):
async def stream_response():
async for chunk in ask_graph.astream(input={...}):
yield f"data: {json.dumps(chunk)}\n\n"
return StreamingResponse(stream_response(), media_type="text/event-stream")
```
### 5. **Job Queue Pattern**
For async background tasks (source processing), use Surreal-Commands job queue:
```python
# Submit job
command_id = await CommandService.submit_command_job(
app="open_notebook",
command="process_source",
input={...}
)
# Poll status
status = await source.get_status()
```
---
## Service Communication Patterns
### Frontend → API
1. **REST requests** (HTTP GET/POST/PUT/DELETE)
2. **JSON request/response bodies**
3. **Standard HTTP status codes** (200, 400, 404, 500)
4. **Optional streaming** (Server-Sent Events for long operations)
**Example**:
```typescript
// Frontend
const response = await fetch("http://localhost:5055/sources", {
method: "POST",
body: formData, // multipart/form-data for file upload
});
const source = await response.json();
```
### API → SurrealDB
1. **SurrealQL queries** (similar to SQL)
2. **Async driver** with connection pooling
3. **Type-safe record IDs** (record_id syntax)
4. **Transaction support** for multi-step operations
**Example**:
```python
# API
result = await repo_query(
"SELECT * FROM source WHERE notebook = $notebook_id",
{"notebook_id": ensure_record_id(notebook_id)}
)
```
### API → AI Providers (via Esperanto)
1. **Esperanto unified interface**
2. **Per-request provider override**
3. **Automatic fallback on failure**
4. **Token counting and cost estimation**
**Example**:
```python
# API
model = await provision_langchain_model(task="chat")
response = await model.ainvoke({"input": prompt})
```
### API → Job Queue (Surreal-Commands)
1. **Async job submission**
2. **Fire-and-forget pattern**
3. **Status polling via `/commands/{id}` endpoint**
4. **Job completion callbacks (optional)**
**Example**:
```python
# Submit async source processing
command_id = await CommandService.submit_command_job(...)
# Client polls status
response = await fetch(f"http://localhost:5055/commands/{command_id}")
status = await response.json() # returns { status: "running|queued|completed|failed" }
```
---
## Database Schema Overview
### Core Schema Structure
**Tables** (20+):
- Notebooks (with soft-delete via `archived` flag)
- Sources (content + metadata)
- SourceEmbeddings (vector chunks)
- Notes (user-created + AI-generated)
- ChatSessions (conversation history)
- Transformations (custom rules)
- SourceInsights (transformation outputs)
- Relationships (notebook→source, notebook→note)
**Migrations**:
- Automatic on API startup
- Located in `open_notebook/database/migrations/`
- Numbered sequentially (`1.surrealql`, `2.surrealql`, …) and registered explicitly in `AsyncMigrationManager` (no auto-discovery)
- Tracked in `_sbl_migrations` table
- Rollback via `N_down.surrealql` files (manual)
### Relationship Model
**Graph Relationships**:
```
Notebook
← reference ← Source (many:many)
← artifact ← Note (many:many)
Source
→ source_embedding (one:many)
→ source_insight (one:many)
→ embedding (via source_embedding)
ChatSession
→ messages (JSON array in database)
→ notebook_id (reference to Notebook)
Transformation
→ source_insight (one:many)
```
**Query Example** (get all sources in a notebook with counts):
```sql
SELECT id, title,
count(<-reference.in) as note_count,
count(<-embedding.in) as embedded_chunks
FROM source
WHERE notebook = $notebook_id
ORDER BY updated DESC
```
---
## Key Architectural Decisions
### 1. **Async Throughout**
All I/O operations are non-blocking to maximize concurrency and responsiveness.
**Trade-off**: Slightly more complex code (async/await syntax) vs. high throughput.
### 2. **Multi-Provider from Day 1**
Built-in support for 17 AI providers prevents vendor lock-in.
**Trade-off**: Added complexity in ModelManager vs. flexibility and cost optimization.
### 3. **Graph-First Workflows**
LangGraph state machines for complex multi-step operations (ask, chat, transformations).
**Trade-off**: Steeper learning curve vs. maintainable, debuggable workflows.
### 4. **Self-Hosted Database**
SurrealDB for graph + vector search in one system (no external dependencies).
**Trade-off**: Operational responsibility vs. simplified architecture and cost savings.
### 5. **Job Queue for Long-Running Tasks**
Async job submission (source processing, podcast generation) prevents request timeouts.
**Trade-off**: Eventual consistency vs. responsive user experience.
---
## Important Quirks & Gotchas
### API Startup
- **Migrations run automatically** on every startup; check logs for errors
- **SurrealDB must be running** before starting API (connection test in lifespan)
- **Auth middleware is basic** (password-only); upgrade to OAuth/JWT for production
### Database Operations
- **Record IDs use SurrealDB syntax** (table:id format, e.g., "notebook:abc123")
- **ensure_record_id()** helper prevents malformed IDs
- **Soft deletes** via `archived` field (data not removed, just marked inactive)
- **Timestamps in ISO 8601 format** (created, updated fields)
### LangGraph Workflows
- **State persistence** via SqliteSaver in `/data/sqlite-db/`
- **No built-in timeout**; long workflows may block requests (use streaming for UX)
- **Model fallback** automatic if primary provider unavailable
- **Checkpoint IDs** must be unique per session (avoid collisions)
### AI Provider Integration
- **Esperanto library** handles all provider APIs (no direct API calls)
- **Per-request override** via RunnableConfig (temporary, not persistent)
- **Cost estimation** via token counting (not 100% accurate, use for guidance)
- **Fallback logic** tries cheaper models if primary fails
### File Uploads
- **Stored in `/data/uploads/`** directory (not database)
- **Unique filename generation** prevents overwrites (counter suffix)
- **Content-core library** extracts text from 50+ file types
- **Large files** may block API briefly (sync content extraction)
---
## Performance Considerations
### Optimization Strategies
1. **Connection Pooling**: SurrealDB async driver with configurable pool size
2. **Query Caching**: TanStack Query on frontend (client-side caching)
3. **Embedding Reuse**: Vector search uses pre-computed embeddings
4. **Chunking**: Sources split into chunks for better search relevance
5. **Async Operations**: Non-blocking I/O for high concurrency
6. **Lazy Loading**: Frontend requests only needed data (pagination)
### Bottlenecks
1. **LLM Calls**: Latency depends on provider (typically 1-30 seconds)
2. **Embedding Generation**: Time proportional to content size and provider
3. **Vector Search**: Similarity computation over all embeddings
4. **Content Extraction**: Sync operation in source processing
### Monitoring
- **API Logs**: Check loguru output for errors and slow operations
- **Database Queries**: SurrealDB metrics available via admin UI
- **Token Usage**: Estimated via `estimate_tokens()` utility
- **Job Status**: Poll `/commands/{id}` for async operations
---
## Extension Points
### Adding a New Workflow
1. Create `open_notebook/graphs/workflow_name.py`
2. Define StateDict and node functions
3. Build graph with `.add_node()` / `.add_edge()`
4. Create service in `api/workflow_service.py`
5. Register router in `api/main.py`
6. Add tests in `tests/test_workflow.py`
### Adding a New Data Model
1. Create model in `open_notebook/domain/model_name.py`
2. Inherit from BaseModel (domain object)
3. Implement `save()`, `get()`, `delete()` methods (CRUD)
4. Add repository functions if complex queries needed
5. Create database migration in `open_notebook/database/migrations/` (and register it in `AsyncMigrationManager`)
6. Add API routes and models in `api/`
### Adding a New AI Provider
1. Configure Esperanto for new provider (see .env.example)
2. ModelManager automatically detects via environment variables
3. Override via per-request config (no code changes needed)
4. Test fallback logic if provider unavailable
---
## Deployment Considerations
### Development
- All services on localhost (3000, 5055, 8000)
- Auto-reload on file changes (Next.js, FastAPI)
- Hot-reload database migrations
- Open API docs at http://localhost:5055/docs
### Production
- **Frontend**: Deploy to Vercel, Netlify, or Docker
- **API**: Docker container (see Dockerfile)
- **Database**: SurrealDB container or managed service
- **Environment**: Secure .env file with API keys
- **SSL/TLS**: Reverse proxy (Nginx, CloudFlare)
- **Rate Limiting**: Add at proxy layer
- **Auth**: Replace PasswordAuthMiddleware with OAuth/JWT
- **Monitoring**: Log aggregation (CloudWatch, DataDog, etc)
---
## Summary
Open Notebook's architecture provides a solid foundation for privacy-focused, AI-powered research. The separation of concerns (frontend/API/database), async-first design, and multi-provider flexibility enable rapid development and easy deployment. LangGraph workflows orchestrate complex AI tasks, while Esperanto abstracts provider details. The result is a scalable, maintainable system that puts users in control of their data and AI provider choice.
docs/7-DEVELOPMENT/change-playbooks.md
+209
View File
@@ -0,0 +1,209 @@
# Change Playbooks
Step-by-step guides for common types of changes in the Open Notebook codebase. Each playbook lists the files to touch **in order**, what to do at each step, and what to test.
> **For AI agents:** Read the relevant playbook BEFORE implementing. Follow the sequence — skipping steps causes incomplete changes that break other layers.
---
## How to Use This Document
1. Identify what type of change your issue requires
2. Follow the playbook step by step
3. If a change spans multiple types (e.g., new field + new endpoint), combine the relevant playbooks
4. When in doubt, read existing examples in the codebase — look at the most recent similar change via `git log`
---
## Playbook: Add a Field to an Existing Model
**Example:** "Add `language` field to Source"
| Step | File(s) | What to Do |
|------|---------|------------|
| 1 | `open_notebook/domain/<model>.py` | Add field with type hint and default value. Follow existing patterns in the class. |
| 2 | `open_notebook/database/migrations/N.surrealql` | Create migration. Use next number in sequence. `DEFINE FIELD` for new fields, `UPDATE` for backfilling existing records. Register it in `AsyncMigrationManager` (`async_migrate.py`) — migrations are not auto-discovered. |
| 3 | `api/models.py` | Add field to `*Create`, `*Update` (Optional), and `*Response` schemas. |
| 4 | `frontend/src/lib/types/api.ts` | Add field to the corresponding TypeScript interface (`*Response`, `Create*Request`, `Update*Request`). |
| 5 | Frontend component (if user-facing) | Display or edit the field in the relevant component. |
| 6 | `frontend/src/lib/locales/*/` | Add i18n strings if the field has a user-visible label. All 7 locales. |
| 7 | Tests | Add/update tests covering the new field — at minimum, API test for create/read. |
**Verify:** Restart API (migration auto-runs), check logs for migration success, test via `/docs`.
---
## Playbook: New API Endpoint
**Example:** "Add endpoint to export notebook as PDF"
| Step | File(s) | What to Do |
|------|---------|------------|
| 1 | `api/models.py` | Define request/response Pydantic schemas. Naming: `<Feature>Request`, `<Feature>Response`. |
| 2 | `api/routers/<resource>.py` | Add endpoint to existing router, OR create new router file if it's a new resource. Follow the pattern: validate → call service → return response. |
| 3 | `api/<resource>_service.py` | Business logic goes here, not in the router. Create new service file if needed. |
| 4 | `api/main.py` | If new router file: register with `app.include_router()`. |
| 5 | `frontend/src/lib/types/api.ts` | Add TypeScript types matching the Pydantic schemas. |
| 6 | `frontend/src/lib/api/<resource>.ts` | Add method to the API module. Follow existing pattern (axios call, return `response.data`). |
| 7 | `frontend/src/lib/hooks/use-<resource>.ts` | Add React Query hook. `useQuery` for GET, `useMutation` for POST/PUT/DELETE. Include cache invalidation and toast. |
| 8 | Frontend component/page | Wire up the hook in the UI. |
| 9 | Tests | API test (status codes, validation, error cases). |
**Naming conventions:**
- Routers: `@router.get("/resources/{id}")` (plural, lowercase, kebab for multi-word)
- Services: functions are `async`, named descriptively (`process_source`, `generate_podcast`)
- Hooks: `useResources()` for list, `useResource(id)` for single, `useCreateResource()` for mutation
---
## Playbook: New LangGraph Workflow
**Example:** "Add a summarization workflow"
| Step | File(s) | What to Do |
|------|---------|------------|
| 1 | `prompts/<workflow_name>/*.jinja` | Create Jinja2 prompt templates. Use `Prompter` from ai-prompter. |
| 2 | `open_notebook/graphs/<workflow_name>.py` | Define `StateDict` (TypedDict), node functions, build graph with `StateGraph`. Use `provision_langchain_model()` for model selection. Wrap LLM calls with `classify_error()`. |
| 3 | `api/<resource>_service.py` | Invoke graph: `await graph.ainvoke(state, config)`. |
| 4 | `api/routers/<resource>.py` | Expose endpoint to trigger the workflow. |
| 5 | `commands/<workflow>_commands.py` | If the workflow should run async: create command with `CommandInput`/`CommandOutput`. Register in command service. |
| 6 | Frontend integration | API module → hook → component. |
| 7 | Tests | Test graph nodes individually with mocked LLM responses. |
**Key patterns:**
- Nodes are sync functions (LangGraph requirement) but can call async code via ThreadPoolExecutor
- Use `classify_error()` to convert raw exceptions to typed `OpenNotebookError` subclasses
- Use `provision_langchain_model()` for model selection — never hardcode a provider
- State is a TypedDict, NOT a Pydantic model
---
## Playbook: Bug Fix (Single Layer)
**Example:** "order_by parameter not working on sources endpoint"
| Step | What to Do |
|------|------------|
| 1 | **Identify the layer.** Read the issue and determine: frontend, API router, service, domain model, database, or graph. |
| 2 | **Read the relevant AGENTS.md** (root, `open_notebook/`, or `frontend/`) and the matching page in `docs/7-DEVELOPMENT/`. They document the rules and gotchas. |
| 3 | **Reproduce.** Use the API docs (`/docs`), browser, or a test to confirm the bug. |
| 4 | **Fix.** Make the minimal change needed. Don't refactor surrounding code. |
| 5 | **Add a test** that reproduces the bug and verifies the fix. |
| 6 | **Run existing tests** to verify no regression: `uv run pytest tests/` |
---
## Playbook: Bug Fix (Cross-Layer)
**Example:** "Creating a source via URL doesn't show in notebook"
| Step | What to Do |
|------|------------|
| 1 | **Trace the data flow.** Start from where the user sees the problem (frontend) and trace backward: component → hook → API call → router → service → domain → database. |
| 2 | **Identify where the chain breaks.** Use API docs to test the backend independently of the frontend. Use SurrealDB queries to check if data was persisted. |
| 3 | **Fix at the right layer.** Don't patch the symptom in the frontend if the bug is in the service. |
| 4 | **Verify the full chain** after fixing. |
| 5 | **Add tests** at the layer where the bug was. |
---
## Playbook: Database Migration
**Example:** "Add index on source.notebook_id for query performance"
| Step | File(s) | What to Do |
|------|---------|------------|
| 1 | `open_notebook/database/migrations/N.surrealql` (+ `N_down.surrealql`) | Write SurrealQL. Use next number in sequence. Check existing migrations for patterns. |
| 2 | `open_notebook/database/async_migrate.py` | Register the new files in `AsyncMigrationManager.__init__` — migrations are hard-coded, not auto-discovered. |
| 3 | Domain model (if schema change) | Update field definitions to match. |
| 4 | API schemas (if new/changed fields) | Update Pydantic models. |
| 5 | **Verify:** Restart API and check logs | Migrations auto-run on startup. Look for errors in Loguru output. |
**Important:**
- Migrations are numbered and run in order
- They're tracked in the `_sbl_migrations` table — won't re-run
- One migration per PR that needs one, numbered in merge order; never consolidate after a migration lands on main (dev images apply it immediately) — see [ADR-006](decisions/ADR-006-migration-granularity.md)
- For destructive changes (DROP FIELD), consider data preservation
- Test with existing data, not just empty database
---
## Playbook: Frontend-Only Change
**Example:** "Improve notebook list loading state"
| Step | File(s) | What to Do |
|------|---------|------------|
| 1 | Identify component | Components are in `frontend/src/app/` (pages) or `frontend/src/components/` (shared). |
| 2 | Make changes | Follow existing patterns: functional components, hooks for state, Tailwind for styling. |
| 3 | i18n strings | If adding user-visible text, add to ALL locale files under `frontend/src/lib/locales/`. |
| 4 | Test in browser | Check responsive layout, dark mode (if applicable), loading states, empty states, error states. |
**Key patterns:**
- `'use client'` directive at top of components using hooks
- State: `useState` for local, Zustand for global, TanStack Query for server
- Styling: Tailwind utility classes, Shadcn/ui components from `components/ui/`
- Types: Define in `lib/types/api.ts`, import everywhere
---
## Playbook: New Background Command
**Example:** "Add command to rebuild all embeddings for a notebook"
| Step | File(s) | What to Do |
|------|---------|------------|
| 1 | `commands/<name>_commands.py` | Define `CommandInput` and `CommandOutput` Pydantic classes. Write the command function. |
| 2 | Register command | Add to the command service so it can be submitted via `CommandService.submit_command_job()`. |
| 3 | API endpoint | Add endpoint that submits the command and returns the command ID. |
| 4 | Frontend (polling) | Use `/commands/{command_id}` endpoint to poll for status. Show progress to user. |
**Pattern:**
- Commands are fire-and-forget: submit returns immediately with a command ID
- Retry config: `max_attempts`, `stop_on` exceptions (ValueError = no retry)
- Exponential backoff with jitter for transient failures
---
## Playbook: i18n / Translation Update
**Example:** "Add translations for new settings page"
| Step | File(s) | What to Do |
|------|---------|------------|
| 1 | `frontend/src/lib/locales/en-US/index.ts` | Add English strings first. Group by feature. |
| 2 | All other locale files | Add the same keys to: `pt-BR`, `zh-CN`, `zh-TW`, `ja-JP`, `ru-RU`, `bn-IN`. Use English as placeholder if translation unavailable. |
| 3 | Component | Use `const { t } = useTranslation()` and access via `t('section.key')`. |
**7 locales total.** Don't forget any.
### Adding a whole new language
| Step | File(s) | What to Do |
|------|---------|------------|
| 1 | `frontend/src/lib/locales/<code>/index.ts` | Copy the structure from `en-US/index.ts` and translate all strings. |
| 2 | `frontend/src/lib/locales/index.ts` | Register the locale: import it, add to `resources`, add to the `languages` array (`{ code, label }`). |
| 3 | `frontend/src/lib/utils/date-locale.ts` | Import the matching `date-fns/locale` and add it to `LOCALE_MAP`. |
| 4 | **Test** | Switch languages via the UI language toggle; missing keys fall back to en-US. |
---
## Quick Reference: File Locations by Layer
| Layer | Location | Schema/Types | Tests |
|-------|----------|-------------|-------|
| Domain models | `open_notebook/domain/` | Pydantic fields | `tests/` |
| Database | `open_notebook/database/repository.py` | SurrealQL | `tests/` |
| Migrations | `open_notebook/database/migrations/*.surrealql` | SurrealQL | Auto-run on startup |
| AI/LLM | `open_notebook/ai/` | Esperanto types | `tests/` |
| Graphs | `open_notebook/graphs/` | TypedDict state | `tests/` |
| Prompts | `prompts/**/*.jinja` | Jinja2 context | — |
| Commands | `commands/` | CommandInput/Output | `tests/` |
| API routers | `api/routers/` | `api/models.py` | `tests/` |
| API services | `api/*_service.py` | — | `tests/` |
| Frontend types | `frontend/src/lib/types/` | TypeScript interfaces | — |
| Frontend API | `frontend/src/lib/api/` | — | — |
| Frontend hooks | `frontend/src/lib/hooks/` | — | `frontend/src/test/` |
| Frontend components | `frontend/src/components/` | Props interfaces | `frontend/src/test/` |
| Frontend pages | `frontend/src/app/` | — | — |
| i18n | `frontend/src/lib/locales/` | — | — |
docs/7-DEVELOPMENT/code-standards.md
+375
View File
@@ -0,0 +1,375 @@
# Code Standards
This document outlines coding standards and best practices for Open Notebook contributions. All code should follow these guidelines to ensure consistency, readability, and maintainability.
## Python Standards
### Code Formatting
We follow **PEP 8** with some specific guidelines:
- Use **Ruff** for linting and formatting
- Maximum line length: **88 characters**
- Use **double quotes** for strings
- Use **trailing commas** in multi-line structures
### Type Hints
Always use type hints for function parameters and return values:
```python
from typing import List, Optional, Dict, Any
from pydantic import BaseModel
async def process_content(
content: str,
options: Optional[Dict[str, Any]] = None
) -> ProcessedContent:
"""Process content with optional configuration."""
# Implementation
```
### Async/Await Patterns
Use async/await consistently throughout the codebase:
```python
# Good
async def fetch_data(url: str) -> Dict[str, Any]:
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
return await response.json()
# Bad - mixing sync and async
def fetch_data(url: str) -> Dict[str, Any]:
loop = asyncio.get_event_loop()
return loop.run_until_complete(async_fetch(url))
```
### Error Handling
Use structured error handling with custom exceptions:
```python
from open_notebook.exceptions import DatabaseOperationError, InvalidInputError
async def create_notebook(name: str, description: str) -> Notebook:
"""Create a new notebook with validation."""
if not name.strip():
raise InvalidInputError("Notebook name cannot be empty")
try:
notebook = Notebook(name=name, description=description)
await notebook.save()
return notebook
except Exception as e:
raise DatabaseOperationError(f"Failed to create notebook: {str(e)}")
```
### Documentation (Google-style Docstrings)
Use Google-style docstrings for all functions, classes, and modules:
```python
async def vector_search(
query: str,
limit: int = 10,
minimum_score: float = 0.2
) -> List[SearchResult]:
"""Perform vector search across embedded content.
Args:
query: Search query string
limit: Maximum number of results to return
minimum_score: Minimum similarity score for results
Returns:
List of search results sorted by relevance score
Raises:
InvalidInputError: If query is empty or limit is invalid
DatabaseOperationError: If search operation fails
"""
# Implementation
```
#### Module Docstrings
```python
"""
Notebook domain model and operations.
This module contains the core Notebook class and related operations for
managing research notebooks within the Open Notebook system.
"""
```
#### Class Docstrings
```python
class Notebook(BaseModel):
"""A research notebook containing sources, notes, and chat sessions.
Notebooks are the primary organizational unit in Open Notebook, allowing
users to group related research materials and maintain separate contexts
for different projects.
Attributes:
name: The notebook's display name
description: Optional description of the notebook's purpose
archived: Whether the notebook is archived (default: False)
created: Timestamp of creation
updated: Timestamp of last update
"""
```
#### Function Docstrings
```python
async def create_notebook(
name: str,
description: str = "",
user_id: Optional[str] = None
) -> Notebook:
"""Create a new notebook with validation.
Args:
name: The notebook name (required, non-empty)
description: Optional notebook description
user_id: Optional user ID for multi-user deployments
Returns:
The created notebook instance
Raises:
InvalidInputError: If name is empty or invalid
DatabaseOperationError: If creation fails
Example:
```python
notebook = await create_notebook(
name="AI Research",
description="Research on AI applications"
)
```
"""
```
## FastAPI Standards
### Router Organization
Organize endpoints by domain:
```python
# api/routers/notebooks.py
from fastapi import APIRouter, HTTPException, Query
from typing import List, Optional
router = APIRouter()
@router.get("/notebooks", response_model=List[NotebookResponse])
async def get_notebooks(
archived: Optional[bool] = Query(None, description="Filter by archived status"),
order_by: str = Query("updated desc", description="Order by field and direction"),
):
"""Get all notebooks with optional filtering and ordering."""
# Implementation
```
### Request/Response Models
Use Pydantic models for validation:
```python
from pydantic import BaseModel, Field
from typing import Optional
class NotebookCreate(BaseModel):
name: str = Field(..., description="Name of the notebook", min_length=1)
description: str = Field(default="", description="Description of the notebook")
class NotebookResponse(BaseModel):
id: str
name: str
description: str
archived: bool
created: str
updated: str
```
### Error Handling
Use consistent error responses:
```python
from fastapi import HTTPException
from loguru import logger
try:
result = await some_operation()
return result
except InvalidInputError as e:
raise HTTPException(status_code=400, detail=str(e))
except DatabaseOperationError as e:
logger.error(f"Database error: {str(e)}")
raise HTTPException(status_code=500, detail="Internal server error")
```
### API Documentation
Use FastAPI's automatic documentation features:
```python
@router.post(
"/notebooks",
response_model=NotebookResponse,
summary="Create a new notebook",
description="Create a new notebook with the specified name and description.",
responses={
201: {"description": "Notebook created successfully"},
400: {"description": "Invalid input data"},
500: {"description": "Internal server error"}
}
)
async def create_notebook(notebook: NotebookCreate):
"""Create a new notebook."""
# Implementation
```
## Database Standards
### SurrealDB Patterns
Use the repository pattern consistently:
```python
from open_notebook.database.repository import repo_create, repo_query, repo_update
# Create records
async def create_notebook(data: Dict[str, Any]) -> Dict[str, Any]:
"""Create a new notebook record."""
return await repo_create("notebook", data)
# Query with parameters
async def find_notebooks_by_user(user_id: str) -> List[Dict[str, Any]]:
"""Find notebooks for a specific user."""
return await repo_query(
"SELECT * FROM notebook WHERE user_id = $user_id",
{"user_id": user_id}
)
# Update records
async def update_notebook(notebook_id: str, data: Dict[str, Any]) -> Dict[str, Any]:
"""Update a notebook record."""
return await repo_update("notebook", notebook_id, data)
```
### Schema Management
Use migrations for schema changes:
```surrealql
-- migrations/8.surrealql
DEFINE TABLE IF NOT EXISTS new_feature SCHEMAFULL;
DEFINE FIELD IF NOT EXISTS name ON TABLE new_feature TYPE string;
DEFINE FIELD IF NOT EXISTS description ON TABLE new_feature TYPE option<string>;
DEFINE FIELD IF NOT EXISTS created ON TABLE new_feature TYPE datetime DEFAULT time::now();
DEFINE FIELD IF NOT EXISTS updated ON TABLE new_feature TYPE datetime DEFAULT time::now();
```
## TypeScript Standards
### Basic Guidelines
Follow TypeScript best practices:
- Use strict mode enabled in `tsconfig.json`
- Use proper type annotations for all variables and functions
- Avoid using `any` type unless absolutely necessary
- Use `interface` for object shapes, `type` for unions and other advanced types
### Component Structure
- Use functional components with hooks
- Keep components focused and single-responsibility
- Extract reusable logic into custom hooks
- Use proper TypeScript types for props
### Error Handling
- Handle errors explicitly
- Provide meaningful error messages
- Log errors appropriately
- Don't suppress errors silently
## Code Quality Tools
We use these tools to maintain code quality:
- **Ruff**: Linting and code formatting
- Run with: `uv run ruff check . --fix`
- Format with: `uv run ruff format .`
- **MyPy**: Static type checking
- Run with: `uv run python -m mypy .`
- **Pytest**: Testing framework
- Run with: `uv run pytest`
## Common Patterns
### Async Database Operations
```python
async def get_notebook_with_sources(notebook_id: str) -> Notebook:
"""Retrieve notebook with all related sources."""
notebook_data = await repo_query(
"SELECT * FROM notebook WHERE id = $id",
{"id": notebook_id}
)
if not notebook_data:
raise InvalidInputError(f"Notebook {notebook_id} not found")
sources_data = await repo_query(
"SELECT * FROM source WHERE notebook_id = $notebook_id",
{"notebook_id": notebook_id}
)
return Notebook(
**notebook_data[0],
sources=[Source(**s) for s in sources_data]
)
```
### Model Validation
```python
from pydantic import BaseModel, validator
class NotebookInput(BaseModel):
name: str
description: str = ""
@validator('name')
def name_not_empty(cls, v):
if not v.strip():
raise ValueError('Name cannot be empty')
return v.strip()
```
## Code Review Checklist
Before submitting code for review, ensure:
- [ ] Code follows PEP 8 / TypeScript best practices
- [ ] Type hints are present for all functions
- [ ] Docstrings are complete and accurate
- [ ] Error handling is appropriate
- [ ] Tests are included and passing
- [ ] No debug code (console.logs, print statements) left behind
- [ ] Commit messages are clear and follow conventions
- [ ] Documentation is updated if needed
---
**See also:**
- [Testing Guide](testing.md) - How to write tests
- [Contributing Guide](contributing.md) - Overall contribution workflow
docs/7-DEVELOPMENT/content-processing.md
+56
View File
@@ -0,0 +1,56 @@
# Content Processing: Chunking, Embedding, Context & Encryption
Design notes for the utilities in `open_notebook/utils/` that turn raw content into searchable, LLM-consumable data. These are cross-cutting: sources, notes and insights all flow through them.
## Chunking (`utils/chunking.py`)
Content is split with content-type-aware LangChain splitters (`HTMLHeaderTextSplitter`, `MarkdownHeaderTextSplitter`, `RecursiveCharacterTextSplitter`). Content type detection uses the file extension first; heuristics can override a PLAIN extension when confidence ≥ 0.8. Oversized chunks from the HTML/Markdown splitters get a secondary split.
**Why the 400-token default.** `OPEN_NOTEBOOK_CHUNK_SIZE` defaults to 400 tokens — ~20% below the 512-token ceiling of BERT-family embedders (e.g. `mxbai-embed-large`). The buffer absorbs three error sources: tokenizer mismatch (we measure with `o200k_base`, the embedder tokenizes with WordPiece), splitter overshoot, and special tokens. For embedders with large windows (OpenAI `text-embedding-3` family: 8191 tokens) raise it, e.g.:
```bash
export OPEN_NOTEBOOK_CHUNK_SIZE=1500
export OPEN_NOTEBOOK_CHUNK_OVERLAP=150
```
`OPEN_NOTEBOOK_CHUNK_OVERLAP` defaults to 15% of chunk size. Both are **token-based** (not characters), minimum chunk size 100, and require an app restart to take effect.
## Embedding (`utils/embedding.py`)
- `generate_embedding(text)` — unified entry point: short text (≤ chunk size) embeds directly; long text is chunked, each chunk embedded, and the results combined via **mean pooling** (normalize each → mean → normalize result, numpy).
- `generate_embeddings(texts)` — batch path used by `embed_source_command`: batches of 50 with per-batch retry, to stay under provider payload limits.
- Empty/whitespace-only input raises `ValueError` — which background commands treat as a permanent (non-retried) failure by design.
- The embedding model comes from `model_manager` (see [credentials.md](credentials.md) for how provider config is resolved).
**Who triggers embedding** (see also the domain rules in `open_notebook/AGENTS.md`):
| Content | Trigger |
|---|---|
| Note | `Note.save()` auto-submits `embed_note` |
| Insight | `create_insight_command` submits `embed_insight` |
| Source | explicit `source.vectorize()` → `embed_source` (NOT automatic on save) |
| Everything | `rebuild_embeddings_command` fans out individual jobs |
All embedding is fire-and-forget through the surreal-commands worker — nothing embeds if the worker isn't running.
## Context building (`utils/context_builder.py`)
The single implementation behind both context consumers:
- `build_notebook_context()` backs `POST /api/chat/context` (chat panel + podcast generation): it assembles source/note contexts from the inclusion config, whose status strings are matched textually ("not in" skips, "insights" → short context, "full content" → long context). Without a config, every source and note is included with its short context. Per-item failures are logged and skipped.
- `build_source_context()` backs the source-chat graph: one source's short context plus its insights, truncated to a token budget by dropping insights (last-fetched first).
- Every call re-fetches — there is no cache layer.
- Token counting uses `o200k_base` via tiktoken and is an estimate (±5-10% vs. the actual model); `token_count()` falls back to a coarse estimate if tiktoken is unavailable.
## Encryption (`utils/encryption.py`) {#encryption}
Field-level encryption for sensitive values (API keys) stored in the database, using Fernet (AES-128-CBC + HMAC-SHA256).
- Key source: `OPEN_NOTEBOOK_ENCRYPTION_KEY_FILE` (Docker secrets) → `OPEN_NOTEBOOK_ENCRYPTION_KEY`. **No default** — credential storage is unavailable until the key is set.
- Any string works as key: it's derived to a Fernet key via SHA-256, lazily on first use.
- Decryption falls back gracefully: an `InvalidToken` (legacy unencrypted data) returns the original value, so pre-encryption databases keep working.
- Key rotation is **not implemented** — changing the key orphans previously encrypted values.
## Text utilities (`utils/text_utils.py`)
`clean_thinking_content()` strips `<think>…</think>` blocks from model output (extended-thinking models); used in every graph that consumes LLM responses. It handles malformed output (missing opening tag) and bypasses extraction for content > 100KB for performance.
docs/7-DEVELOPMENT/contributing.md
+228
View File
@@ -0,0 +1,228 @@
# Contributing to Open Notebook
Thank you for your interest in contributing to Open Notebook! We welcome contributions from developers of all skill levels. This guide will help you understand our contribution workflow and what makes a good contribution.
## 🚦 Issue-First Workflow (for anything non-trivial)
**To maintain project coherence and avoid wasted effort, non-trivial work starts with an issue:**
1. **Create an issue first** - Before writing any code, create an issue describing the bug or feature
2. **Propose your solution** - Explain how you plan to implement the fix or feature
3. **Wait for assignment** - A maintainer will review and assign the issue to you if approved
4. **Only then start coding** - This ensures your work aligns with the project's vision and architecture
**When you can skip the issue and just open a PR:**
- Typos, broken links, and small documentation clarifications
- Small, obvious bug fixes — a few lines, one clear right answer, no design decisions
- Translation fixes or completing missing i18n keys
**When an issue is definitely required first:**
- New features, of any size
- Architecture or structural changes
- Breaking changes
- Anything where the *how* has more than one reasonable answer
**Already coded something sizeable without an issue?** Don't throw it away: mark the PR as **draft**, open an issue describing the problem and your approach, and link it from the PR. Our triage is fast — expect a call within 1–2 days.
**Why this process?**
- Prevents duplicate work
- Ensures solutions align with our architecture and design principles
- Saves your time by getting feedback before coding
- Helps maintainers manage the project direction
> ⚠️ **Non-trivial pull requests without an issue may be closed**, even if the code is good. We want to respect your time by making sure work is aligned before it starts.
## Code of Conduct
By participating in this project, you are expected to uphold our [Code of Conduct](/CODE_OF_CONDUCT.md). Be respectful, constructive, and collaborative.
## How Can I Contribute?
### Reporting Bugs
1. **Search existing issues** - Check if the bug was already reported
2. **Create a bug report** - Use the [Bug Report template](https://github.com/lfnovo/open-notebook/issues/new?template=bug_report.yml)
3. **Provide details** - Include:
- Steps to reproduce
- Expected vs actual behavior
- Logs, screenshots, or error messages
- Your environment (OS, Docker version, Open Notebook version)
4. **Indicate if you want to fix it** - Check the "I would like to work on this" box if you're interested
### Suggesting Features
1. **Search existing issues** - Check if the feature was already suggested
2. **Create a feature request** - Use the [Feature Request template](https://github.com/lfnovo/open-notebook/issues/new?template=feature_request.yml)
3. **Explain the value** - Describe why this feature would be helpful
4. **Propose implementation** - If you have ideas on how to implement it, share them
5. **Indicate if you want to build it** - Check the "I would like to work on this" box if you're interested
### Contributing Code (Pull Requests)
**IMPORTANT: Follow the issue-first workflow above before starting any PR**
Once your issue is assigned:
1. **Fork the repo** and create your branch from `main`
2. **Understand our vision and principles** - Read [VISION.md](../../VISION.md) (what the product is and where it's going) and [design-principles.md](design-principles.md) (engineering practices)
3. **Follow our architecture** - Refer to the architecture documentation to understand project structure
4. **Write quality code** - Follow the standards outlined in [code-standards.md](code-standards.md)
5. **Test your changes** - See [testing.md](testing.md) for test guidelines
6. **Update documentation** - If you changed functionality, update the relevant docs
7. **Create your PR**:
- Reference the issue number (e.g., "Fixes #123")
- Describe what changed and why
- Include screenshots for UI changes
- Keep PRs focused - one issue per PR
### What Makes a Good Contribution?
✅ **We love PRs that:**
- Solve a real problem described in an issue
- Follow our architecture and coding standards
- Include tests and documentation
- Are well-scoped (focused on one thing)
- Have clear commit messages
❌ **We may close PRs that:**
- Are non-trivial and don't have an associated approved issue (small obvious fixes are exempt — see the issue-first workflow above)
- Introduce breaking changes without discussion
- Conflict with our architectural vision
- Lack tests or documentation
- Try to solve multiple unrelated problems
### AI-Assisted and Agent-Generated PRs
A large share of contributions — including our own — are written with coding agents (Claude Code, Cursor, Copilot, etc.). That's welcome. The tool doesn't change the contract; **the operator does not stop being the author**:
1. **You own the PR.** You must have read, understood, and be able to explain every line of the diff. "The agent wrote it" is never an answer in review.
2. **Issue-first still applies to anything non-trivial.** Agents make it cheap to produce large unsolicited PRs — those get closed like any other unassigned PR, regardless of code quality. (Small obvious fixes are exempt, same as for humans; a sizeable PR without an issue can be converted to draft while its issue goes through triage.)
3. **Tests must have actually run.** Paste real output. An agent *claiming* tests pass is not test evidence.
4. **Point your agent at the right context.** The repo ships `AGENTS.md` files (root, `open_notebook/`, `frontend/`) with the normative rules, and [change-playbooks.md](change-playbooks.md) with step-by-step recipes — agents that read them produce PRs that pass review faster.
5. **Keep it scoped.** Agents tend to "improve" surrounding code along the way. Unrelated refactors belong in separate issues/PRs.
Disclosure of AI assistance is appreciated but optional — responsibility for the result is what matters, and it's yours either way.
## Git Commit Messages
- Use the present tense ("Add feature" not "Added feature")
- Use the imperative mood ("Move cursor to..." not "Moves cursor to...")
- Limit the first line to 72 characters or less
- Reference issues and pull requests liberally after the first line
## Development Workflow
### Branch Strategy
We use a **feature branch workflow**:
1. **Main Branch**: `main` - production-ready code
2. **Feature Branches**: `feature/description` - new features
3. **Bug Fixes**: `fix/description` - bug fixes
4. **Documentation**: `docs/description` - documentation updates
### Making Changes
1. **Create a feature branch**:
```bash
git checkout -b feature/amazing-new-feature
```
2. **Make your changes** following our coding standards
3. **Test your changes**:
```bash
# Run tests
uv run pytest
# Run linting
uv run ruff check .
# Run formatting
uv run ruff format .
```
4. **Commit your changes**:
```bash
git add .
git commit -m "feat: add amazing new feature"
```
5. **Push and create PR**:
```bash
git push origin feature/amazing-new-feature
# Then create a Pull Request on GitHub
```
### Keeping Your Fork Updated
```bash
# Fetch upstream changes
git fetch upstream
# Switch to main and merge
git checkout main
git merge upstream/main
# Push to your fork
git push origin main
```
## Pull Request Process
When you create a pull request:
1. **Link your issue** - Reference the issue number in PR description
2. **Describe your changes** - Explain what changed and why
3. **Provide test evidence** - Screenshots, test results, or logs
4. **Check PR template** - Ensure you've completed all required sections
5. **Wait for review** - A maintainer will review your PR within a week
### PR Review Expectations
- Code review feedback is about the code, not the person
- Be open to suggestions and alternative approaches
- Address review comments with clarity and respect
- Ask questions if feedback is unclear
## Current Priority Areas
We're actively looking for contributions in these areas:
1. **Frontend Enhancement** - Help improve the Next.js/React UI with real-time updates and better UX
2. **Testing** - Expand test coverage across all components
3. **Performance** - Async processing improvements and caching
4. **Documentation** - API examples and user guides
5. **Integrations** - New content sources and AI providers
## Getting Help
### Community Support
- **Discord**: [Join our Discord server](https://discord.gg/37XJPXfz2w) for real-time help
- **GitHub Discussions**: For longer-form questions and ideas
- **GitHub Issues**: For bug reports and feature requests
### Documentation References
- [VISION.md](../../VISION.md) - Product identity and current posture
- [Design Principles](design-principles.md) - Engineering practices and anti-patterns
- [Decision Records](decisions/README.md) - Why things are the way they are
- [Code Standards](code-standards.md) - Coding guidelines by language
- [Testing Guide](testing.md) - How to write tests
- [Development Setup](development-setup.md) - Getting started locally
## Recognition
We recognize contributions through:
- **GitHub credits** on releases
- **Community recognition** in Discord
- **Contribution statistics** in project analytics
- **Maintainer consideration** for active contributors
---
Thank you for contributing to Open Notebook! Your contributions help make research more accessible and private for everyone.
For questions about this guide or contributing in general, please reach out on [Discord](https://discord.gg/37XJPXfz2w) or open a GitHub Discussion.
docs/7-DEVELOPMENT/credentials.md
+65
View File
@@ -0,0 +1,65 @@
# Credential System
How Open Notebook stores, encrypts and provisions AI provider credentials — from the settings UI down to Esperanto model instantiation. This is the single reference for the subsystem; the frontend and backend halves are documented together on purpose.
## Overview
Users can configure provider credentials through the UI instead of environment variables. Keys are stored as individual `Credential` records in SurrealDB, encrypted with Fernet, and resolved at model-provisioning time with a database-first, environment-variable-fallback strategy.
```
Settings UI ──► /credentials API ──► Credential record (encrypted, SurrealDB)
│
Model record ──credential─┘ (preferred: direct link)
│
ModelManager.get_model()
│
credential.to_esperanto_config() ──► Esperanto AIFactory
│
(no linked credential?)
└──► key_provider.provision_provider_keys() ──► env vars ──► Esperanto
```
## The `Credential` domain model (`open_notebook/domain/credential.py`)
- One record per credential (e.g. "My OpenAI Key", "Work Anthropic") — multiple credentials per provider are supported.
- Fields: `name`, `provider`, `modalities`, `api_key` (Pydantic `SecretStr`, masked in logs), plus provider-specific config (`base_url`, `endpoint`, `api_version`, mode-specific endpoints, `project`, `location`, `credentials_path`).
- `api_key` is encrypted with `encrypt_value()` before save and decrypted on read (`get()` / `get_all()` are overridden). Encryption requires `OPEN_NOTEBOOK_ENCRYPTION_KEY` (see [content-processing.md](content-processing.md#encryption) for the encryption utility itself).
- `to_esperanto_config()` builds the config dict passed to Esperanto's `AIFactory.create_*`.
- `provider_config.py` still exists only to migrate legacy `ProviderConfig` records.
## Provisioning: two paths
1. **Credential-linked model (preferred).** A `Model` record has a `credential` field pointing at a Credential. `ModelManager.get_model()` calls `credential.to_esperanto_config()` and passes the config directly — no env var mutation, multiple credentials per provider work naturally.
2. **Env-var fallback (`open_notebook/ai/key_provider.py`).** When a model has no linked credential, `provision_provider_keys(provider)` copies DB-stored keys into `os.environ` so Esperanto can read them; pre-existing env vars are left untouched when no DB config exists. The `PROVIDER_CONFIG` map in `key_provider.py` defines the env-var ↔ config-field mapping for simple providers; multi-field providers (Vertex, Azure, OpenAI-compatible) are handled by the dedicated `_provision_vertex()` / `_provision_azure()` / `_provision_openai_compatible()` functions.
## The API surface (`api/routers/credentials.py`)
CRUD plus lifecycle operations: `POST /credentials/{id}/test` (connection check), `/discover` (list available models), `/register-models` (create Model records from discovery), and two migration endpoints (`/migrate-from-env`, `/migrate-from-provider-config`). Swagger at `/docs` documents the shapes.
**Supported providers (17)** are defined once in the provider registry (`open_notebook/ai/provider_registry.py` `PROVIDERS`) — env vars, modalities, test models, discovery URLs and docs links all live there, and `connection_tester.TEST_MODELS`, `credentials_service.PROVIDER_ENV_CONFIG`/`PROVIDER_MODALITIES` and `model_discovery.OPENAI_COMPAT_PROVIDERS` are derived from it. `GET /api/providers` exposes the registry to clients — the frontend fetches it at runtime (`useProviders()` in `frontend/src/lib/hooks/use-providers.ts`) and renders providers in response order (the registry declaration order). One manual copy remains, enforced by `tests/test_credential_provider_validation.py`: the `SupportedProvider` Literal in `api/models.py` (typing can't be derived at runtime):
- Simple API key: openai, anthropic, google, groq, mistral, deepseek, xai, openrouter, voyage, elevenlabs, deepgram, dashscope, minimax
- URL-based: ollama
- Multi-field: azure, vertex, openai_compatible
**Security properties**:
- API key values are never returned by any endpoint — only metadata (`has_api_key`, counts).
- Every URL field passes `validate_url()` (SSRF protection); private IPs/localhost are allowed by design for self-hosted services (Ollama, LM Studio). Hostname resolution runs in `asyncio.to_thread` to avoid blocking the event loop.
- Connection testing of Vertex credentials collapses "file missing / not JSON / wrong shape" errors into one generic message so the tester can't be used as a filesystem oracle.
## Connection testing (`open_notebook/ai/connection_tester.py`)
`test_provider_connection()` makes a minimal API call using the cheapest model per provider (`TEST_MODELS` map). URL-based providers get a server ping instead (`/api/tags` for Ollama, `/models` for OpenAI-compatible). Error messages are normalized for the UI: 401 → "Invalid API key", rate-limit → success ("connection works"), model-not-found → success ("key valid").
## Frontend half
- `src/lib/api/credentials.ts` — typed client mirroring the endpoints above. The `Credential` interface never carries the key value, only `has_api_key`.
- `src/lib/hooks/use-credentials.ts` — TanStack Query hooks (`useCredentials`, `useCreateCredential`, `useTestCredential`, …) with toast feedback. Mutations invalidate `CREDENTIAL_QUERY_KEYS.all` + provider/model keys; test results are kept in local state, not the query cache.
## Migration paths
Both migration endpoints are idempotent summaries (`migrated` / `skipped` / `errors`):
- **From env vars**: creates Credential records for providers whose env vars are set.
- **From legacy ProviderConfig**: converts old singleton records into individual Credentials.
docs/7-DEVELOPMENT/decisions/ADR-001-surrealdb.md
+27
View File
@@ -0,0 +1,27 @@
# ADR-001: SurrealDB as the database
- **Status**: Accepted
- **Date**: 2026-07 (retroactive record — decision dates from project inception; long-form rationale maintained in [#372](https://github.com/lfnovo/open-notebook/issues/372))
- **Related**: #372, #378, #381, [VISION.md](../../../VISION.md) (Platform v-next cluster)
## Context
Open Notebook needs document storage (sources with metadata), graph relationships (notebooks ↔ sources ↔ notes), vector embeddings for semantic search, and background jobs — while staying easy to self-host for privacy-focused users. A traditional stack would be Postgres + Redis + Celery + a vector DB: four services to operate.
## Decision
Use **SurrealDB** as the single database: documents, graph relationships, vector embeddings and (via surreal-commands) job queueing in one service. Stay with it and work through the challenges; reconsider only under the exit criteria listed in #372 (unworkable transaction conflicts, performance that tuning can't fix, unpatched critical security issue, or a mature alternative with the same consolidated benefits).
## Alternatives considered
- **PostgreSQL + pgvector** — maturity and ecosystem, but loses graph queries and still needs Celery/Redis for jobs.
- **SQLite + LiteFS** — ultimate simplicity, but poor concurrency and no graph features.
- **MongoDB + Redis + Celery** — familiar tooling, but three services kills the self-hosting simplicity advantage.
- **Hybrid (Postgres + Neo4j)** — best of both worlds at an ops cost we don't want to impose on self-hosters.
## Consequences
- One container to run — the biggest infra advantage for self-hosted users.
- Younger ecosystem: we document more and contribute back; fewer established tuning practices.
- Transaction conflicts under concurrency turned out to be log verbosity, not failures (#362, #373) — handled with Tenacity retries.
- Major-version upgrades need deliberate migration work (v3: #378, part of the Platform v-next cluster).
docs/7-DEVELOPMENT/decisions/ADR-002-external-libraries.md
+32
View File
@@ -0,0 +1,32 @@
# ADR-002: Delegate platform/media support to focused external libraries
- **Status**: Accepted
- **Date**: 2026-07 (retroactive record — decision dates from project inception)
- **Related**: [PDR-002](PDR-002-provider-agnostic-core.md), [credentials.md](../credentials.md), [content-processing.md](../content-processing.md)
## Context
Open Notebook's value is the knowledge layer: organizing, understanding and generating from research content. But delivering it requires two areas of heavy, ever-changing integration code: AI provider APIs (17 providers across LLM/embedding/TTS/STT) and content extraction (50+ file types, URLs, video/audio platforms). Building that in-repo would bloat the codebase and pull maintenance attention away from the product core.
## Decision
**If a capability is about platform/media support and will require heavy integration coding, it doesn't belong in this repository.** It lives in a focused external library, keeping this project centered on the knowledge layer — and keeping those parts replaceable by other frameworks if we ever decide to swap them.
Applied today:
- **[Esperanto](https://github.com/lfnovo/esperanto)** — all model access (LLM, embeddings, TTS, STT) through one `AIFactory` interface. Application code selects models via the `Model` registry and `provision_langchain_model()`, never by instantiating provider clients directly.
- **[Content Core](https://github.com/lfnovo/content-core)** — all content extraction (files, URLs, media) through `extract_content()`.
- The same rule covers **[podcast-creator](https://github.com/lfnovo/podcast-creator)** for audio generation.
## Alternatives considered
- **Provider SDKs / extraction logic in-repo** — spreads platform-specific code across every feature and makes each new provider/format a cross-cutting change.
- **LangChain provider classes directly** — workable for models, but couples every callsite to per-provider packages and quirks; Esperanto normalizes config (and we control the library).
- **Single provider + "compatible" endpoints** — simplest, but breaks the no-lock-in promise and excludes local-first users.
## Consequences
- Adding a provider or content format is mostly an upstream change, plus registry/config sync here (e.g. the four `SupportedProvider` locations — see `open_notebook/AGENTS.md`).
- The libraries are independently versioned, testable and swappable; the boundary keeps this repo's scope honest.
- Debugging sometimes spans two repos; issues whose root cause is upstream get the `upstream` + library labels (`esperanto`, `content-core`, `podcast-creator`).
- Provider-specific *capabilities* (not just plumbing) remain constrained by [PDR-002](PDR-002-provider-agnostic-core.md).
docs/7-DEVELOPMENT/decisions/ADR-003-streamlit-to-nextjs.md
+26
View File
@@ -0,0 +1,26 @@
# ADR-003: Migrate the UI from Streamlit to Next.js
- **Status**: Accepted
- **Date**: 2026-07 (retroactive record — migration shipped with the v1→v2 platform rework)
- **Related**: [frontend.md](../frontend.md), API-first principle in [VISION.md](../../../VISION.md)
## Context
The original UI was Streamlit: fast to build, but it coupled UI and backend logic in one process, made external integrations hard, and gave us limited control over API behavior. The API-first principle — every capability accessible via REST, the UI being just one client — was structurally impossible to honor.
## Decision
Rebuild the frontend as a **Next.js/React application** consuming the same FastAPI REST API that external clients use. Business logic lives behind the API; the frontend is a pure client (TanStack Query + Zustand over axios).
## Alternatives considered
- **Stay on Streamlit** — lowest effort, but permanently blocks API-first and a polished UX.
- **Server-rendered templates (FastAPI + Jinja)** — simpler stack, but poor interactivity for chat/streaming-heavy UX.
- **Other SPA frameworks (Vue, Svelte)** — viable; React/Next.js won on ecosystem, component library availability (Radix/Shadcn) and contributor familiarity.
## Consequences
- The API is complete by construction — anything the UI does, an integration can do (this later enabled the MCP direction, #878).
- Two build systems and a larger contributor surface (TypeScript + Python).
- i18n, theming and accessibility became first-class frontend concerns (7 locales today).
- Legacy Streamlit remnants were removed over time; migrations assume API-driven access.
docs/7-DEVELOPMENT/decisions/ADR-004-background-workers.md
+27
View File
@@ -0,0 +1,27 @@
# ADR-004: Long-running work runs on background workers
- **Status**: Accepted
- **Date**: 2026-07 (retroactive record — decision dates from the async rework)
- **Related**: #381, [ADR-001](ADR-001-surrealdb.md), [podcasts.md](../podcasts.md)
## Context
Open Notebook routinely processes work that takes seconds to minutes: ingesting and embedding large volumes of content, generating insights, producing podcast episodes. Users run it on machines of very different sizes — from small home servers to beefy workstations — so the same job can be fast on one deployment and slow on another. None of that may lock product usage: the API and UI must stay responsive while heavy work happens (async-first principle).
## Decision
**Long-running operations run as background jobs on a dedicated worker process, never inline in the API request cycle.** Submission is fire-and-forget (returns a job id immediately), status is observable (`/commands/{id}`), failures are explicit (permanent vs. retriable), and the UI polls or resumes rather than blocking.
The *queue implementation* is deliberately an implementation detail behind this decision. Today it's [surreal-commands](https://github.com/lfnovo/surreal-commands) — chosen because it reuses the SurrealDB we already run, adding zero infrastructure for self-hosters ([ADR-001](ADR-001-surrealdb.md)). A move to Celery is under evaluation as part of the Platform v-next cluster (#381); if it happens, it replaces the implementation, not this decision.
## Alternatives considered
- **Run heavy work inline in API handlers** — simplest, but locks the product for minutes and ties job success to the HTTP connection.
- **FastAPI BackgroundTasks / asyncio tasks** — no persistence: jobs die with the process, no status tracking, no retry.
- **Celery + Redis from the start** — battle-tested, but two extra services for self-hosters before we knew we needed the features.
## Consequences
- A worker process is **required** for anything async to actually run (documented in the root `AGENTS.md`; forgetting it is a silent-queue failure mode).
- Features must be designed for the job model: idempotent-ish under retry, explicit permanent failures (`ValueError` → no retry), status exposed to the UI (e.g. podcasts use `max_attempts: 1` + an explicit retry endpoint).
- Deployment has one more moving part than a monolith — the price of not locking usage on slow machines.
docs/7-DEVELOPMENT/decisions/ADR-005-release-confidence-process.md
+69
View File
@@ -0,0 +1,69 @@
# ADR-005: Releases pass a risk-based confidence process, gated on the real image
- **Status**: Accepted
- **Date**: 2026-07 (established during the v1.11.0 release)
- **Related**: [RELEASE_PROCESS.md](../../../.github/RELEASE_PROCESS.md), `scripts/release-test/`, [ADR-004](ADR-004-background-workers.md)
## Context
Releases grew from a handful of fixes to 50+ commits spanning security
hardening, features, migrations and dependency changes. Verification was
ad-hoc: a green test suite on `main` plus whatever manual checks the release
owner remembered. v1.11.0 proved the gap empirically — the unit suite was
fully green while `sort_by=title` returned a 500 (a SEARCH-index interaction
only a real SurrealDB exhibits) and clearing credential fields silently
no-oped (two mirror-image bugs, frontend and API, that only an end-to-end
path reveals). Neither class of bug is catchable by mocked tests, and neither
was: both were found by the process this record establishes.
## Decision
**Every stable release passes a risk-based confidence process before cutting,
and the final gate runs against the built Docker image — the artifact users
receive — not the repository.**
The process (mechanics in [RELEASE_PROCESS.md](../../../.github/RELEASE_PROCESS.md)):
1. **Changelog audit first** — the release diff, fully represented in the
CHANGELOG, is the input for both the test plan and the communication.
2. **Risk matrix over test list** — each change is classified by what it can
break and for whom, then assigned to a bucket: **A** (automated now),
**B** (automatable with investment — build the muscle when it compounds),
**C** (release-owner judgment: real credentials, real TTS, UX, the pushed
image). Security changes are probed for the inverse risk: does the
protection break legitimate use?
3. **The image gate** — fresh-install and upgrade-with-data scenarios run
against real containers (`make release-test`), because packaging bugs
(supervisord flags, uv sync modes, migration ordering) never appear in the
suite.
4. **Fix loop with a re-test policy** — findings become focused PRs; what
re-runs after each merge is defined up front. Pre-existing bugs that are
not release regressions become backlog issues, not scope creep.
5. **Human gates stay human** — the pushed-image verification and the release
publication require the release owner explicitly; automation prepares,
people pull the trigger.
6. **Retro closes the loop** — accepted improvements are applied to the
process docs and scripts in the same session.
## Alternatives considered
- **Keep ad-hoc verification** — free, but v1.11.0 showed it misses exactly
the bug classes that hurt users most (integration and packaging).
- **Full CI-based E2E on every PR** — highest coverage, but a real
SurrealDB + worker + image build pipeline on every PR is slow and expensive;
the release boundary is where artifact-level confidence pays off.
- **Community soak (RC tags)** — previously abandoned: slow feedback and low
participation; a deliberate confidence process front-loads what soaking
found late.
## Consequences
- Cutting a release costs hours, not minutes — deliberately: the cost scales
with release size, which is the point of the risk matrix.
- Release muscle is versioned in-repo (`scripts/release-test/`, make targets)
and compounds: bucket-B investments from one release become bucket-A
automation for the next.
- The upgrade scenario requires published previous images to remain available
on the registries.
- The process assumes a release owner in the loop for buckets B/C — it is a
confidence process, not full automation.
docs/7-DEVELOPMENT/decisions/ADR-006-migration-granularity.md
+35
View File
@@ -0,0 +1,35 @@
# ADR-006: Migration granularity follows merge granularity, not release granularity
- **Status**: Accepted
- **Date**: 2026-07
- **Related**: #1085 (first case decided under this policy), #1031 (surreal-basics migration runner — policy carries over), [change-playbooks.md](../change-playbooks.md) (Database Migration playbook)
## Context
Multiple issues in the same release cycle can each need a schema migration. The intuitive worry: merging them one by one produces several small migrations (19, 20, 21, 22…) inside a single release, which "feels" messier than one consolidated migration per release.
Two facts about this project shape the decision:
1. Migrations run **automatically at API startup**, in sequence, tracked in `_sbl_migrations`. Users upgrading across any version span run all pending migrations transparently — they never see the count.
2. A `v1-dev` image is published on **every push to main**. A migration is therefore effectively *released the moment it lands on main* — dev-image users apply it immediately, before any versioned release exists.
## Decision
**One migration per PR that needs one; numbers allocated in merge order; never consolidate after a migration has touched main.**
- Each migration lives in the PR that motivates it, with its `_down` counterpart, reviewed together with the code that requires it.
- Multiple small migrations per release is the normal, healthy state — not a smell.
- Consolidation is only allowed **before merge**, as a development-time choice: when two in-flight PRs touch the *same table*, coordinate them (stack the PRs, or fold the schema change into the first one to land).
- Branch protection's strict mode makes number collisions between parallel PRs surface as a required update-branch before merge; renumbering is part of that rebase.
## Alternatives considered
- **One consolidated migration per release** — rejected: post-hoc squashing breaks every `v1-dev` user whose `_sbl_migrations` already recorded the individual migrations, and it decouples schema changes from the PRs that explain them.
- **Batch migrations in a release branch** — rejected for the same dev-image reason, plus it would hold merged features hostage to release timing.
## Consequences
- Release notes may list several migrations; that is an implementation detail, not a user-facing cost.
- Debugging stays cheap: "migration 21 broke it" is bisectable; a slice of a consolidated release migration is not.
- If parallel batches make number collisions frequent, add a cheap CI check (duplicate or gapped numbers fail the build).
- When the migration runner moves to surreal-basics (#1031), this policy transfers unchanged — it is about granularity and immutability-after-main, not about the runner.
docs/7-DEVELOPMENT/decisions/PDR-001-single-user-first.md
+26
View File
@@ -0,0 +1,26 @@
# PDR-001: Single-user first; new features must not preclude multi-user
- **Status**: Accepted
- **Date**: 2026-07
- **Related**: [#712](https://github.com/lfnovo/open-notebook/issues/712) (multi-user umbrella), [VISION.md](../../../VISION.md) (Current Posture)
## Context
Multi-user support is a recurring community request (#712 tracks it), and it would be a deep platform redesign: auth, data scoping, permissions, and what "multi-user" even means for a privacy-first self-hosted tool (household/team vs. SaaS-style). That strategic call hasn't been made. Meanwhile, features keep shipping — and each one can silently bake in single-tenancy assumptions that make the future redesign more expensive.
## Decision
Open Notebook remains a **single-user product for now** — but this is a *directional constraint*, not a verdict: **new features must not gratuitously preclude multi-user.** Concretely, when designing a feature, avoid hard-coding single-tenancy where a neutral choice costs the same: data models that could carry an owner scope, auth touchpoints that assume exactly one identity, global singletons for per-user state.
This record does **not** commit to building multi-user. It commits to keeping the door open until the vision call in #712 is made.
## Alternatives considered
- **Decide multi-user now** — premature: the product core is still stabilizing (see Current Posture in VISION.md) and the design space (team vs. SaaS) is unresolved.
- **Ignore it until decided** — cheapest today, but every single-tenant assumption shipped becomes migration debt.
## Consequences
- Card design/triage gains a concrete check: "does this preclude multi-user?"
- Slight design overhead on data-model and auth decisions.
- When the #712 vision call is made, a new PDR supersedes or extends this one.
docs/7-DEVELOPMENT/decisions/PDR-002-provider-agnostic-core.md
+27
View File
@@ -0,0 +1,27 @@
# PDR-002: Provider-agnostic core by default; provider-exclusive capabilities require a PDR
- **Status**: Accepted
- **Date**: 2026-07
- **Related**: [ADR-002](ADR-002-external-libraries.md), [VISION.md](../../../VISION.md) (Principles + Current Posture)
## Context
Our philosophy is to support as many providers as makes sense — users choose their AI, including fully local. The cost of that democratic stance is that we don't immediately adopt capabilities unique to one provider, even compelling ones. But "portable by default" must not harden into "never": some provider-exclusive capabilities (including paid-only ones) may be worth adopting deliberately. The current phase (see VISION.md posture) is basics-first; that phase is expected to evolve — the decision is to evolve, not to stand still.
## Decision
Two rules with different lifespans:
1. **Durable principle**: the core is provider-agnostic. Features must work across the provider matrix by default. Adopting a provider-exclusive capability is *allowed*, but it is a deliberate product decision that requires its own PDR (stating scope, fallback behavior for other providers, and why the exclusivity is worth it).
2. **Current posture** (temporal, lives in VISION.md): in the basics-first phase, we lean portable. Expanding into provider-exclusive premium capabilities is a phase change, recorded as a short PDR when it happens — that will be the principle working, not a reversal.
## Alternatives considered
- **Hard rule: never provider-exclusive** — protects portability but forfeits real capabilities and doesn't reflect our actual intent.
- **No rule: adopt case-by-case silently** — drifts into de-facto lock-in one convenient feature at a time.
## Consequences
- Triage/design gets a crisp test: "does this only work on provider X?" → needs a PDR, not just an implementation.
- Feature requests for provider-exclusive capabilities aren't auto-rejected — they're routed to a deliberate decision.
- Slower adoption of shiny provider features, by design, during the basics-first phase.
docs/7-DEVELOPMENT/decisions/README.md
+52
View File
@@ -0,0 +1,52 @@
# Decision Records
The project's decision log: short, dated, immutable records of structural decisions. They answer *"why is it like this?"* months later, and prevent settled discussions from being reopened without knowing they were settled.
Two kinds, same format:
- **ADR** (Architecture Decision Record) — technical choices: `ADR-NNN-slug.md`
- **PDR** (Product Decision Record) — product direction and scope: `PDR-NNN-slug.md`
The **current rules** distilled from these records live in [VISION.md](../../../VISION.md) (product identity + posture) and [design-principles.md](../design-principles.md) (engineering practices). Records are the memory; those pages are the law.
## Rules
1. **Records are immutable.** Reversing a decision means writing a *new* record and marking the old one `Superseded by ADR-NNN` in its Status line — never editing history.
2. **Write it in the same PR.** A design that resolves an open structural question ships with its record. Half a page, written while the context is loaded — not a documentation session later.
3. **Keep it to half a page.** Four sections: Context, Decision, Alternatives considered, Consequences. If it needs more, link an issue or doc for the depth.
4. **Number sequentially** within each prefix (ADR-005 comes after ADR-004, independent of PDRs).
## Template
```markdown
# ADR-NNN: <Title>
- **Status**: Accepted | Superseded by ADR-NNN
- **Date**: YYYY-MM
- **Related**: #issue, other records
## Context
What was the situation and the forces at play? (2-5 sentences)
## Decision
What we decided, stated as a rule someone can follow.
## Alternatives considered
What else was on the table and why it lost. (bullets)
## Consequences
What this makes easier, what it makes harder, what to watch. (bullets)
```
## Index
| Record | Title | Status |
|---|---|---|
| [ADR-001](ADR-001-surrealdb.md) | SurrealDB as the database | Accepted |
| [ADR-002](ADR-002-external-libraries.md) | Delegate platform/media support to focused external libraries | Accepted |
| [ADR-003](ADR-003-streamlit-to-nextjs.md) | Migrate the UI from Streamlit to Next.js | Accepted |
| [ADR-004](ADR-004-background-workers.md) | Long-running work runs on background workers | Accepted |
| [ADR-005](ADR-005-release-confidence-process.md) | Releases pass a risk-based confidence process, gated on the real image | Accepted |
| [ADR-006](ADR-006-migration-granularity.md) | Migration granularity follows merge granularity, not release granularity | Accepted |
| [PDR-001](PDR-001-single-user-first.md) | Single-user first; don't preclude multi-user | Accepted |
| [PDR-002](PDR-002-provider-agnostic-core.md) | Provider-agnostic core by default | Accepted |
docs/7-DEVELOPMENT/design-principles.md
+162
View File
@@ -0,0 +1,162 @@
# Design Principles
Engineering practices and decision-making guidance for contributors.
> **Looking for the product vision?** What Open Notebook is (and is not), the durable product
> principles, and the current posture live in **[VISION.md](../../VISION.md)** — read that first.
> The reasoning behind past structural choices lives in the
> **[decision records](decisions/README.md)**.
## 🎨 UI/UX Principles
### Focus on Content, Not Chrome
- Minimize UI clutter and distractions
- Content should occupy most of the screen space
- Controls appear when needed, not always visible
- Consistent layout across different views
### Progressive Disclosure
- Show simple options first, advanced options on demand
- Don't overwhelm new users with every possible setting
- Provide sensible defaults that work for 80% of use cases
- Make power features discoverable but not intrusive
### Responsive and Fast
- UI should feel instant for common operations
- Show loading states for operations that take time
- Cache and optimize where possible
- Degrade gracefully on slow connections
## 🔧 Technical Principles
### Clean Separation of Concerns
**Layers should not leak**:
- Frontend should not know about database structure
- API should not contain business logic (delegate to domain layer)
- Domain models should not know about HTTP requests
- Database layer should not know about AI providers
### Type Safety and Validation
**Catch errors early**:
- Use Pydantic models for all API boundaries
- Type hints throughout Python codebase
- TypeScript for frontend code
- Validate data at system boundaries
### Test What Matters
**Focus on valuable tests**:
- Test business logic and domain models
- Test API contracts and error handling
- Don't test framework code (FastAPI, React, etc.)
- Integration tests for critical workflows
### Database as Source of Truth
**SurrealDB is our single source of truth**:
- All state persisted in database
- No business logic in database layer
- Use SurrealDB features (record links, queries) appropriately
- Schema migrations for all schema changes
## 🚫 Anti-Patterns to Avoid
### Feature Creep
**What it looks like**:
- Adding features because they're "cool" or "easy"
- Building features for edge cases before common cases work well
- Trying to be everything to everyone
**Instead**: Focus on core use cases; say no to features that don't align with the
[vision](../../VISION.md); build extensibility points for edge cases.
### Premature Optimization
**What it looks like**:
- Optimizing code before knowing if it's slow
- Complex caching strategies without measuring impact
- Trading code clarity for marginal performance gains
**Instead**: Measure first, optimize second; focus on algorithmic improvements; profile before
making performance changes.
### Over-Engineering
**What it looks like**:
- Building abstraction layers "in case we need them later"
- Implementing design patterns for 3-line functions
- Creating frameworks instead of solving problems
**Instead**: Start simple, refactor when patterns emerge; optimize for readability; use
abstractions when they simplify, not complicate.
### Breaking Changes Without Migration Path
**What it looks like**:
- Changing database schema without migration scripts
- Modifying API contracts without versioning
- Removing features without deprecation warnings
**Instead**: Always provide migration scripts for schema changes; deprecate before removing;
document breaking changes clearly.
## 🤝 Decision-Making Framework
When evaluating new features or changes, ask:
### 1. Does it align with our vision?
- Does it help users own their research data?
- Does it support privacy and self-hosting?
- Does it fit our core use cases? (See [VISION.md](../../VISION.md))
### 2. Does it follow our principles?
- Is it simple to use and understand?
- Does it work via API?
- Does it support multiple providers?
- Can it be extended by users?
### 3. Is the implementation sound?
- Does it maintain separation of concerns?
- Is it properly typed and validated?
- Does it include tests?
- Is it documented?
### 4. What is the cost?
- How much complexity does it add?
- How much maintenance burden?
- Does it introduce new dependencies?
- Will it be used enough to justify the cost?
### 5. Are there alternatives?
- Can existing features solve this problem?
- Can this be built as a plugin or extension?
- Should this be a separate tool instead?
**When a decision resolves a structural question** — architecture or product — capture it as a
[decision record](decisions/README.md) in the same PR. Half a page, written while the context is
still loaded.
---
## For Contributors
When proposing a feature or change:
1. **Reference the vision and principles** — explain how your proposal aligns with
[VISION.md](../../VISION.md)
2. **Identify trade-offs** — be honest about what you're trading for what
3. **Suggest alternatives** — show you've considered other approaches
4. **Be open to feedback** — maintainers may see concerns you don't
**Remember**: A "no" to a feature isn't a judgment on you or your idea. It means we're staying
focused on our core vision. We appreciate all contributions and ideas!
---
**Questions about these principles?** Open a discussion on GitHub or join our [Discord](https://discord.gg/37XJPXfz2w).
docs/7-DEVELOPMENT/development-setup.md
+441
View File
@@ -0,0 +1,441 @@
# Local Development Setup
This guide walks you through setting up Open Notebook for local development. Follow these steps to get the full stack running on your machine.
## Prerequisites
Before you start, ensure you have the following installed:
- **Python 3.11+** - Check with: `python --version`
- **uv** (recommended) or **pip** - Install from: https://github.com/astral-sh/uv
- **SurrealDB** - Via Docker or binary (see below)
- **Docker** (optional) - For containerized database
- **Node.js 18+** (optional) - For frontend development
- **Git** - For version control
## Step 1: Clone and Initial Setup
```bash
# Clone the repository
git clone https://github.com/lfnovo/open-notebook.git
cd open-notebook
# Add upstream remote for keeping your fork updated
git remote add upstream https://github.com/lfnovo/open-notebook.git
```
## Step 2: Install Python Dependencies
```bash
# Using uv (recommended)
uv sync
# Or using pip
pip install -e .
```
## Step 3: Environment Variables
Create a `.env` file in the project root with your configuration:
```bash
# Copy from example
cp .env.example .env
```
Edit `.env` with your settings:
```bash
# Database
SURREAL_URL=ws://localhost:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=password
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=development
# Credential encryption (required for storing API keys)
OPEN_NOTEBOOK_ENCRYPTION_KEY=my-dev-secret-key
# Application
APP_PASSWORD= # Optional password protection
DEBUG=true
LOG_LEVEL=DEBUG
```
### AI Provider Configuration
After starting the API and frontend, configure your AI provider via the Settings UI:
1. Open **http://localhost:3000** → **Settings** → **API Keys**
2. Click **Add Credential** → Select your provider
3. Enter your API key (get from provider dashboard)
4. Click **Save**, then **Test Connection**
5. Click **Discover Models** → **Register Models**
Popular providers:
- **OpenAI** - https://platform.openai.com/api-keys
- **Anthropic (Claude)** - https://console.anthropic.com/
- **Google** - https://ai.google.dev/
- **Groq** - https://console.groq.com/
For local development, you can also use:
- **Ollama** - Run locally without API keys (see "Local Ollama" below)
> **Note:** API key environment variables (e.g., `OPENAI_API_KEY`) are deprecated. Use the Settings UI to manage credentials instead.
## Step 4: Start SurrealDB
### Option A: Using Docker (Recommended)
```bash
# Start SurrealDB in memory (publish the port on localhost only — the
# database uses default credentials, so never publish it on 0.0.0.0)
docker run -d --name surrealdb -p 127.0.0.1:8000:8000 \
surrealdb/surrealdb:v2 start \
--user root --pass password \
memory
# Or with persistent storage
docker run -d --name surrealdb -p 127.0.0.1:8000:8000 \
-v surrealdb_data:/data \
surrealdb/surrealdb:v2 start \
--user root --pass password \
file:/data/surreal.db
```
### Option B: Using Make
```bash
make database
```
### Option C: Using Docker Compose
```bash
docker compose up -d surrealdb
```
### Verify SurrealDB is Running
```bash
# Should show server information
curl http://localhost:8000/
```
## Step 5: Run Database Migrations
Database migrations run automatically when you start the API. The first startup will apply any pending migrations.
To verify migrations manually:
```bash
# API will run migrations on startup
uv run python -m api.main
```
Check the logs - you should see messages like:
```
Running migration 001_initial_schema
Running migration 002_add_vectors
...
Migrations completed successfully
```
## Step 6: Start the API Server
In a new terminal window:
```bash
# Terminal 2: Start API (port 5055)
uv run --env-file .env uvicorn api.main:app --host 0.0.0.0 --port 5055
# Or using the shortcut
make api
```
You should see:
```
INFO: Application startup complete
INFO: Uvicorn running on http://0.0.0.0:5055
```
### Verify API is Running
```bash
# Check health endpoint
curl http://localhost:5055/health
# View API documentation
open http://localhost:5055/docs
```
## Step 7: Start the Frontend (Optional)
If you want to work on the frontend, start Next.js in another terminal:
```bash
# Terminal 3: Start Next.js frontend (port 3000)
cd frontend
npm install # First time only
npm run dev
```
You should see:
```
> next dev
▲ Next.js 16.x
- Local: http://localhost:3000
```
### Access the Frontend
Open your browser to: http://localhost:3000
## Verification Checklist
After setup, verify everything is working:
- [ ] **SurrealDB**: `curl http://localhost:8000/` returns content
- [ ] **API**: `curl http://localhost:5055/health` returns `{"status": "ok"}`
- [ ] **API Docs**: `open http://localhost:5055/docs` works
- [ ] **Database**: API logs show migrations completing
- [ ] **Frontend** (optional): `http://localhost:3000` loads
## Development Workflows: When to Use What?
| Workflow | Use Case | Speed | Production Parity |
|----------|----------|-------|-------------------|
| **Local Services** (`make start-all`) | Day-to-day development, fastest iteration | ⚡⚡⚡ Fast | Medium |
| **Docker Compose** (`make dev`) | Testing containerized setup | ⚡⚡ Medium | High |
| **Local Docker Build** (`make docker-build-local`) | Testing Dockerfile changes | ⚡ Slow | Very High |
| **Multi-platform Build** (`make docker-push`) | Publishing releases (see [Release Process](../../.github/RELEASE_PROCESS.md)) | 🐌 Very Slow | Exact |
Local services give hot reload, direct log access and easy debugging; Docker Compose (`examples/docker-compose-dev.yml` via `make dev`, `examples/docker-compose-full-local.yml` via `make full`) is closer to production. Use `make docker-build-local` before touching anything Docker-related in a PR.
## Starting Services Together
### Quick Start All Services
```bash
make start-all # SurrealDB + API + worker + frontend
make status # see what's running
make stop-all # stop everything
```
### Individual Terminals (Recommended for Development)
**Terminal 1 - Database:**
```bash
make database
```
**Terminal 2 - API:**
```bash
make api
```
**Terminal 3 - Background worker** (required for podcasts, embeddings, source processing):
```bash
make worker-start
```
**Terminal 4 - Frontend:**
```bash
cd frontend && npm run dev
```
### Performance Tips
1. Use `make start-all` instead of Docker for daily work
2. Keep SurrealDB running between sessions (`make database`)
3. Use `make docker-build-local` only when testing Dockerfile changes
4. Skip multi-platform builds until ready to publish
5. Clean caches when things get weird: `make clean-cache`, `docker system prune -a`
## Development Tools Setup
### Pre-commit Hooks (Optional but Recommended)
Install git hooks to automatically check code quality:
```bash
uv run pre-commit install
```
Now your commits will be checked before they're made.
### Code Quality Commands
```bash
# Lint Python code (auto-fix)
make ruff
# or: ruff check . --fix
# Type check Python code
make lint
# or: uv run python -m mypy .
# Run tests
uv run pytest
# Run tests with coverage
uv run pytest --cov=open_notebook
```
## Common Development Tasks
### Running Tests
```bash
# Run all tests
uv run pytest
# Run specific test file
uv run pytest tests/test_notebooks.py
# Run with coverage report
uv run pytest --cov=open_notebook --cov-report=html
```
### Creating a Feature Branch
```bash
# Create and switch to new branch
git checkout -b feature/my-feature
# Make changes, then commit
git add .
git commit -m "feat: add my feature"
# Push to your fork
git push origin feature/my-feature
```
### Updating from Upstream
```bash
# Fetch latest changes
git fetch upstream
# Rebase your branch
git rebase upstream/main
# Push updated branch
git push origin feature/my-feature -f
```
## Troubleshooting
### "Connection refused" on SurrealDB
**Problem**: API can't connect to SurrealDB
**Solutions**:
1. Check if SurrealDB is running: `docker ps | grep surrealdb`
2. Verify URL in `.env`: Should be `ws://localhost:8000/rpc`
3. Restart SurrealDB: `docker stop surrealdb && docker rm surrealdb`
4. Then restart with: `docker run -d --name surrealdb -p 127.0.0.1:8000:8000 surrealdb/surrealdb:v2 start --user root --pass password memory`
### "Address already in use"
**Problem**: Port 5055 or 3000 is already in use
**Solutions**:
```bash
# Find process using port
lsof -i :5055 # Check port 5055
# Kill process (macOS/Linux)
kill -9 <PID>
# Or use different port
uvicorn api.main:app --port 5056
```
### Module not found errors
**Problem**: Import errors when running API
**Solutions**:
```bash
# Reinstall dependencies
uv sync
# Or with pip
pip install -e .
```
### Database migration failures
**Problem**: API fails to start with migration errors
**Solutions**:
1. Check SurrealDB is running: `curl http://localhost:8000/`
2. Check credentials in `.env` match your SurrealDB setup
3. Check logs for specific migration error: `make api 2>&1 | grep -i migration`
4. Verify database exists: Check SurrealDB console at http://localhost:8000/
### Migrations not applying
**Problem**: Database schema seems outdated
**Solutions**:
1. Restart API - migrations run on startup: `make api`
2. Check logs show "Migrations completed successfully"
3. Verify `/migrations/` folder exists and has files
4. Check SurrealDB is writable and not in read-only mode
## Optional: Local Ollama Setup
For testing with local AI models:
```bash
# Install Ollama from https://ollama.ai
# Pull a model (e.g., Mistral 7B)
ollama pull mistral
```
Then configure via the Settings UI:
1. Go to **Settings** → **API Keys** → **Add Credential** → **Ollama**
2. Enter base URL: `http://localhost:11434`
3. Click **Save**, then **Test Connection**
4. Click **Discover Models** → **Register Models**
## Optional: Docker Development Environment
Run entire stack in Docker:
```bash
# Start all services
docker compose --profile multi up
# Logs
docker compose logs -f
# Stop services
docker compose down
```
## Next Steps
After setup is complete:
1. **Read the Contributing Guide** - [contributing.md](contributing.md)
2. **Explore the Architecture** - Check the documentation
3. **Find an Issue** - Look for "good first issue" on GitHub
4. **Set Up Pre-commit** - Install git hooks for code quality
5. **Join Discord** - https://discord.gg/37XJPXfz2w
## Getting Help
If you get stuck:
- **Discord**: [Join our server](https://discord.gg/37XJPXfz2w) for real-time help
- **GitHub Issues**: Check existing issues for similar problems
- **GitHub Discussions**: Ask questions in discussions
- **Documentation**: See [code-standards.md](code-standards.md) and [testing.md](testing.md)
---
**Ready to contribute?** Go to [contributing.md](contributing.md) for the contribution workflow.
docs/7-DEVELOPMENT/frontend.md
+45
View File
@@ -0,0 +1,45 @@
# Frontend Architecture
How the Next.js app is layered and how data flows through it. Normative rules (commands, i18n, gotchas) live in [`frontend/AGENTS.md`](../../frontend/AGENTS.md); this page is the mental model.
## Layers
```
Pages (src/app/, App Router) → Feature components (src/components/) → Hooks (src/lib/hooks/)
↓
Stores (src/lib/stores/) → API modules (src/lib/api/) → Backend
```
- **Pages** — route endpoints. Router groups `(auth)` / `(dashboard)` organize routes without affecting URLs. Pages call hooks and render components.
- **Components** — feature folders (`source/`, `notebooks/`, `podcasts/`, …) own page-level state (loading, error); `components/ui/` are stateless Radix UI wrappers styled with Tailwind + CVA.
- **Hooks** (`src/lib/hooks/`) — TanStack Query wrappers. Query hooks return `{ data, isLoading, error, refetch }`; mutation hooks invalidate caches and toast. Complex hooks (`useNotebookChat`, `useAsk`) add session management, context building, SSE streaming.
- **Stores** (`src/lib/stores/`) — Zustand for auth and modal state; `persist` middleware syncs to localStorage (auth token under `auth-storage`).
- **API modules** (`src/lib/api/`) — namespaced typed clients (`sourcesApi.list()`, …) over a single axios instance with auth/FormData/401 interceptors.
Provider tree in `app/layout.tsx` (outermost → innermost): ErrorBoundary → ThemeProvider → QueryProvider → I18nProvider → ConnectionGuard → Toaster.
## Flow walkthrough: notebook chat
1. `notebooks/[id]/page.tsx` passes `notebookId` to `ChatColumn`.
2. `useNotebookChat()` queries sessions, manages message state, returns `{ messages, sendMessage(), setModelOverride() }`.
3. On send: `buildContext()` assembles selected sources/notes (token/char counts), calls `chatApi.sendMessage()`, and applies an **optimistic update** (message added locally, removed on error).
4. Response updates the TanStack Query cache; related source/note mutations elsewhere invalidate broadly so stale UI refreshes.
5. Model override before a session exists is stored as pending and applied on session creation.
## Flow walkthrough: file upload
1. `SourceDialog` collects the file; `useFileUpload` builds FormData — nested JSON fields are stringified.
2. The client interceptor deletes the Content-Type header so the browser sets the multipart boundary.
3. On success, `queryClient.invalidateQueries(['sources'])` refetches lists; `useSourceStatus` polls every 2s while the source is processing.
## Caching strategy
Query keys are hierarchical (`QUERY_KEYS.sources(notebookId)`), but invalidation is deliberately **broad** (`['sources']` catches everything) — a precision/simplicity trade-off. Frequently changing data uses `refetchOnWindowFocus: true`.
## Auth
The token is validated by an actual API call (`/notebooks`), not JWT decoding, with a 30-second cache in the auth store. The response interceptor clears auth and redirects to `/login` on 401. Logout is client-side only.
## Error handling
`getApiErrorMessage()` (`lib/utils/error-handler.ts`) tries an i18n mapping first, then falls back to the backend's descriptive message — which the backend error-classification system already makes user-friendly (see [architecture.md](architecture.md)). Mutations surface errors as toasts; an app-level ErrorBoundary catches render errors.
docs/7-DEVELOPMENT/index.md
+127
View File
@@ -0,0 +1,127 @@
# Development
Welcome to the Open Notebook development documentation! Whether you're contributing code, understanding our architecture, or maintaining the project, you'll find guidance here.
## 🎯 Pick Your Path
### 👨‍💻 I Want to Contribute Code
Start with **[Contributing Guide](contributing.md)** for the workflow, then check:
- **[Quick Start](quick-start.md)** - Clone, install, verify in 5 minutes
- **[Development Setup](development-setup.md)** - Complete local environment guide
- **[Code Standards](code-standards.md)** - How to write code that fits our style
- **[Testing](testing.md)** - How to write and run tests
**First time?** Check out our [Contributing Guide](contributing.md) for the issue-first workflow.
### 🔒 I Want to Understand Security Practices
**[Security Guidelines](security.md)** covers:
- Database query safety (preventing SurrealQL injection)
- Template rendering safety (preventing SSTI)
- File handling safety (preventing path traversal and LFI)
- Secrets management and CORS configuration
- Code review security checklist
---
### 🏗️ I Want to Understand the Architecture
**[Architecture Overview](architecture.md)** covers:
- 3-tier system design
- Tech stack and rationale
- Key components and workflows
- Design patterns we use
For deeper dives into specific subsystems:
- **[Credentials](credentials.md)** - Provider credential storage, encryption, provisioning
- **[Content Processing](content-processing.md)** - Chunking, embedding, context building, encryption
- **[Podcasts](podcasts.md)** - Profile system, model registry, job lifecycle
- **[Prompts](prompts.md)** - Prompt engineering patterns
- **[Frontend](frontend.md)** - Next.js layers and data flows
Normative rules for coding agents (and humans in a hurry) live in the `AGENTS.md` files at the
repo root, `open_notebook/`, and `frontend/`.
### 🧭 Why Is It Like This?
- **[VISION.md](../../VISION.md)** - Product identity and current posture
- **[Decision Records](decisions/README.md)** - ADRs and PDRs: the durable "why" behind structural choices
---
### 👨‍🔧 I'm a Maintainer
**[Maintainer Guide](maintainer-guide.md)** covers:
- Issue triage and management
- Pull request review process
- Communication templates
- Best practices
---
## 📚 Quick Links
| Document | For | Purpose |
|---|---|---|
| [Quick Start](quick-start.md) | New developers | Clone, install, and verify setup (5 min) |
| [Development Setup](development-setup.md) | Local development | Complete environment setup guide |
| [Contributing](contributing.md) | Code contributors | Workflow: issue → code → PR |
| [Code Standards](code-standards.md) | Writing code | Style guides for Python, FastAPI, DB |
| [Testing](testing.md) | Testing code | How to write and run tests |
| [Architecture](architecture.md) | Understanding system | System design, tech stack, workflows |
| [Credentials](credentials.md) | Understanding system | Provider credential subsystem |
| [Content Processing](content-processing.md) | Understanding system | Chunking, embedding, context building |
| [Podcasts](podcasts.md) | Understanding system | Podcast profiles and job lifecycle |
| [Prompts](prompts.md) | Understanding system | Prompt engineering patterns |
| [Frontend](frontend.md) | Understanding system | Next.js architecture and data flows |
| [Design Principles](design-principles.md) | All developers | Engineering practices and anti-patterns |
| [VISION.md](../../VISION.md) | All developers | Product identity and current posture |
| [Decision Records](decisions/README.md) | All developers | ADRs/PDRs — why things are the way they are |
| [Change Playbooks](change-playbooks.md) | Contributors & agents | Step-by-step recipes for common changes |
| [API Reference](api-reference.md) | Building integrations | Complete REST API documentation |
| [Security](security.md) | All developers | Security practices and vulnerability prevention |
| [Maintainer Guide](maintainer-guide.md) | Maintainers | Managing issues, PRs, labels |
---
## 🚀 Current Development Priorities
We're actively looking for help with:
1. **Frontend Enhancement** - Improve Next.js/React UI with real-time updates
2. **Performance** - Async processing and caching optimizations
3. **Testing** - Expand test coverage across components
4. **Documentation** - API examples and developer guides
5. **Integrations** - New content sources and AI providers
See GitHub Issues labeled `good first issue` or `help wanted`.
---
## 💬 Getting Help
- **Discord**: [Join our server](https://discord.gg/37XJPXfz2w) for real-time discussions
- **GitHub Discussions**: For architecture questions
- **GitHub Issues**: For bugs and features
Don't be shy! We're here to help new contributors succeed.
---
## 📖 Additional Resources
### External Documentation
- [FastAPI Docs](https://fastapi.tiangolo.com/)
- [SurrealDB Docs](https://surrealdb.com/docs)
- [LangChain Docs](https://python.langchain.com/)
- [Next.js Docs](https://nextjs.org/docs)
### Our Libraries
- [Esperanto](https://github.com/lfnovo/esperanto) - Multi-provider AI abstraction
- [Content Core](https://github.com/lfnovo/content-core) - Content processing
- [Podcast Creator](https://github.com/lfnovo/podcast-creator) - Podcast generation
---
Ready to get started? Head over to **[Quick Start](quick-start.md)**! 🎉
docs/7-DEVELOPMENT/maintainer-guide.md
+446
View File
@@ -0,0 +1,446 @@
# Maintainer Guide
This guide is for project maintainers to help manage contributions effectively while maintaining project quality and vision.
## Table of Contents
- [Issue Management](#issue-management)
- [Pull Request Review](#pull-request-review)
- [Merging PR Batches](#merging-pr-batches)
- [Common Scenarios](#common-scenarios)
- [Communication Templates](#communication-templates)
## Issue Management
### When a New Issue is Created
**1. Initial Triage** (within 24-48 hours)
- Issues arrive with the intake label `needs-triage` (applied by the issue templates). Triage replaces it with exactly **one state label** from the funnel below (or closes the issue).
- Add one **type** and one **area** label where they apply (see [Labels](#labels)).
- Quick assessment:
- Is it clear and well-described?
- Is it aligned with the product vision? (See [VISION.md](../../VISION.md))
- Does it duplicate an existing issue?
**2. Initial Response**
```markdown
Thanks for opening this issue! We'll review it and get back to you soon.
[If it's a bug] In the meantime, have you checked our troubleshooting guide?
[If it's a feature] You might find our [vision](https://github.com/lfnovo/open-notebook/blob/main/VISION.md) helpful for understanding what we're building toward.
```
**3. Decision Making**
Ask yourself:
- Does this align with our [vision and principles](../../VISION.md)?
- Is this something we want in the core project, or better as a plugin/extension?
- Do we have the capacity to support this feature long-term?
- Will this benefit most users, or just a specific use case?
**4. Issue Assignment**
If the contributor checked "I am a developer and would like to work on this":
**For Accepted Issues:**
```markdown
Great idea! This aligns well with our goals, particularly [specific design principle].
I see you'd like to work on this. Before you start:
1. Please share your proposed approach/solution
2. Review our [Contributing Guide](contributing.md) and [VISION.md](../../VISION.md)
3. Once we agree on the approach, I'll assign this to you
Looking forward to your thoughts!
```
**For Issues Needing Clarification:**
```markdown
Thanks for offering to work on this! Before we proceed, we need to clarify a few things:
1. [Question 1]
2. [Question 2]
Once we have these details, we can discuss the best approach.
```
**For Issues Not Aligned with Vision:**
```markdown
Thank you for the suggestion and for offering to work on this!
After reviewing against our [vision and principles](https://github.com/lfnovo/open-notebook/blob/main/VISION.md), we've decided not to pursue this in the core project because [specific reason].
However, you might be able to achieve this through [alternative approach, if applicable].
We appreciate your interest in contributing! Feel free to check out our [open issues](https://github.com/lfnovo/open-notebook/issues) for other ways to contribute.
```
### Labels
The label set is curated — **don't invent labels**. If something doesn't fit, raise it instead of adding one. Assign **one state**, **one type**, and **one area** where each applies; multiple bundling/ecosystem labels are fine.
**State funnel** — every open issue lands in exactly one state:
| Label | Meaning |
|---|---|
| `needs-triage` | Intake — applied by the issue templates, means "not triaged yet" |
| `needs-vision` | Unsure if/how this fits — strategic call for the maintainers (against [VISION.md](../../VISION.md)) |
| `needs-design` | Wanted, but the *how* isn't resolved — needs design/spec before it's ready |
| `needs-info` | Waiting on the reporter to confirm or provide more information |
| `ready` | Fully specified — the dev loop can pick it up |
| **Close** | Use GitHub's native close reasons (duplicate / not planned); link the canonical issue when duplicate/superseded |
**Type** — what kind of work it is (apply one when clear):
- `bug` · `enhancement` · `documentation`
- `installation` is an intake label applied by the issue-creation workflow (not by triage); installation reports get routed to `area: deploy`.
**Area** — which part of the system (apply always, one per issue):
| Label | What goes here |
|---|---|
| `area: chat` | Conversation/chat, RAG retrieval, agentic responses, citations |
| `area: search` | Full-text and semantic search |
| `area: sources` | Source ingestion & processing (URLs, files, extraction, chunking) |
| `area: notebooks` | Notebook features: notes, insights, transformations |
| `area: providers` | AI provider integrations + model configuration |
| `area: embeddings` | Embedding models, vectorization, semantic indexing |
| `area: podcast` | Podcast / audio generation |
| `area: database` | SurrealDB, persistence, schema, migrations |
| `area: ui` | Frontend (Next.js), UX, visual issues |
| `area: deploy` | Docker, deployment, k8s, reverse proxy, infra/setup |
| `area: offline` | Airgapped/offline operation |
| `area: i18n` | Internationalization / localization |
**Bundling / epics:**
- `umbrella` — a tracking issue grouping related work
- `tracked-in-umbrella` — covered by an umbrella; follow the umbrella for progress
- `bundled` — part of a thematic bundle
- `upstream` — root cause lives in one of our libraries, not this repo
**Ecosystem** — issues whose real home is an upstream library:
- `esperanto` (model abstraction) · `content-core` (content extraction) · `podcast-creator` (podcast generation)
**Community:**
- `good first issue` — small, well-scoped, newcomer-friendly
- `help wanted` — we'd welcome a contributor to take this
### Consolidation: one issue vs. umbrella
When several open issues circle the same topic, pick the model by **how decided the work is** — not just by shared theme:
- **Pre-vision / pre-design topic** → collapse into **one** issue (`needs-vision` or `needs-design`), capture each request's signal (👍 counts, interested contributors) in its body, and close the rest as duplicates pointing to it. A topic isn't N issues — it's one thinking space.
- **Already decomposed into real parallel tasks** → use `umbrella` + `tracked-in-umbrella`. Children stay open because each is independently pickable (e.g. the multi-user umbrella #712).
Rule of thumb: if the issues can't be worked until *we* make a call, they're one issue. If the call is made and the work splits into things a contributor could pick up today, they're an umbrella with children. **Never close an issue that has an active assignee/contributor or open PR** — link it as a phase instead.
## Pull Request Review
### Initial PR Review Checklist
**Before diving into code:**
- [ ] Is there an associated approved issue?
- [ ] Does the PR reference the issue number?
- [ ] Is the PR description clear about what changed and why?
- [ ] Did the contributor check the relevant boxes in the PR template?
- [ ] Are there tests? Screenshots (for UI changes)?
**Red Flags** (may require closing PR):
- No associated issue on a non-trivial change (small obvious fixes are exempt; sizeable PRs can be converted to draft while their issue goes through triage)
- Issue was not assigned to contributor
- PR tries to solve multiple unrelated problems
- Breaking changes without discussion
- Conflicts with project vision
### Code Review Process
**1. High-Level Review**
- Does the approach align with our architecture?
- Is the solution appropriately scoped?
- Are there simpler alternatives?
- Does it follow our design principles?
**2. Code Quality Review**
Python:
- [ ] Follows PEP 8
- [ ] Has type hints
- [ ] Has docstrings
- [ ] Proper error handling
- [ ] No security vulnerabilities
TypeScript/Frontend:
- [ ] Follows TypeScript best practices
- [ ] Proper component structure
- [ ] No console.logs left in production code
- [ ] Accessible UI components
**3. Testing Review**
- [ ] Has appropriate test coverage
- [ ] Tests are meaningful (not just for coverage percentage)
- [ ] Tests pass locally and in CI
- [ ] Edge cases are tested
**4. Documentation Review**
- [ ] Code is well-commented
- [ ] Complex logic is explained
- [ ] User-facing documentation updated (if applicable)
- [ ] API documentation updated (if API changed)
- [ ] Migration guide provided (if breaking change)
### Providing Feedback
**Positive Feedback** (important!):
```markdown
Thanks for this PR! I really like [specific thing they did well].
[Feedback on what needs to change]
```
**Requesting Changes:**
```markdown
This is a great start! A few things to address:
1. **[High-level concern]**: [Explanation and suggested approach]
2. **[Code quality issue]**: [Specific example and fix]
3. **[Testing gap]**: [What scenarios need coverage]
Let me know if you have questions about any of this!
```
**Suggesting Alternative Approach:**
```markdown
I appreciate the effort you put into this! However, I'm concerned about [specific issue].
Have you considered [alternative approach]? It might be better because [reasons].
What do you think?
```
## Merging PR Batches
Mechanics for landing a batch of approved PRs without stepping on each other:
- **Squash-merge everything.** One commit per PR keeps `main` linear and makes reverts trivial.
- **Expect CHANGELOG conflicts.** Every PR adds a bullet under `[Unreleased]`, so the Nth merge often flips its siblings to DIRTY. Resolve by rebasing the branch onto `main` and keeping **both** sides' bullets — they're independent entries, not competing edits — then `git push --force-with-lease`.
- **Contributor forks:** rebase the fork's branch onto `origin/main`, skipping commits that are already upstream, and push with an explicit-OID lease: `git push --force-with-lease=<branch>:<headOID>`. This works whenever the contributor left "Allow edits by maintainers" (maintainerCanModify) enabled.
- **Check for scheduled replacements first.** Before merging a fix into code that's slated to be replaced (e.g. the database layer migrating to surreal-basics, issue #1031), redirect the fix upstream or note it on the tracking issue instead of landing it in code that's about to disappear.
- **Hunt for competing PRs.** Before merging a fix, search open PRs for others addressing the same bug — pick the best one and close the rest with a link, rather than merging the first one you review.
## Common Scenarios
### Scenario 1: Good Code, Wrong Approach
**Situation**: Contributor wrote quality code, but solved the problem in a way that doesn't fit our architecture.
**Response:**
```markdown
Thank you for this PR! The code quality is great, and I can see you put thought into this.
However, I'm concerned that this approach [specific architectural concern]. In our architecture, we [explain the pattern we follow].
Would you be open to refactoring this to [suggested approach]? I'm happy to provide guidance on the specifics.
Alternatively, if you don't have time for a refactor, I can take over and finish this up (with credit to you, of course).
Let me know what you prefer!
```
### Scenario 2: PR Without Assigned Issue
**Situation**: Contributor submitted PR without going through issue approval process.
**Response:**
```markdown
Thanks for the PR! I appreciate you taking the time to contribute.
However, to maintain project coherence, we require all PRs to be linked to an approved issue that was assigned to the contributor. This is explained in our [Contributing Guide](contributing.md).
This helps us:
- Ensure work aligns with project vision
- Prevent duplicate efforts
- Discuss approach before implementation
Could you please:
1. Create an issue describing this change
2. Wait for it to be reviewed and assigned to you
3. We can then reopen this PR or you can create a new one
Sorry for the inconvenience - this process helps us manage the project effectively.
```
### Scenario 3: Feature Request Not Aligned with Vision
**Situation**: Well-intentioned feature that doesn't fit project goals.
**Response:**
```markdown
Thank you for this suggestion! I can see how this would be useful for [specific use case].
After reviewing against our [vision and principles](https://github.com/lfnovo/open-notebook/blob/main/VISION.md), we've decided not to include this in the core project because [specific reason - e.g., "it conflicts with our 'Simplicity Over Features' principle" or "it would require dependencies that conflict with our privacy-first approach"].
Some alternatives:
- [If applicable] This could be built as a plugin/extension
- [If applicable] This functionality might be achievable through [existing feature]
- [If applicable] You might be interested in [other tool] which is designed for this use case
We appreciate your contribution and hope you understand. Feel free to check our roadmap or open issues for other ways to contribute!
```
### Scenario 4: Contributor Ghosts After Feedback
**Situation**: You requested changes, but contributor hasn't responded in 2+ weeks.
**After 2 weeks:**
```markdown
Hey there! Just checking in on this PR. Do you have time to address the feedback, or would you like someone else to take over?
No pressure either way - just want to make sure this doesn't fall through the cracks.
```
**After 1 month with no response:**
```markdown
Thanks again for starting this work! Since we haven't heard back, I'm going to close this PR for now.
If you want to pick this up again in the future, feel free to reopen it or create a new PR. Alternatively, I'll mark the issue as available for someone else to work on.
We appreciate your contribution!
```
Then:
- Close the PR
- Unassign the issue
- Add `help wanted` label to the issue
### Scenario 5: Breaking Changes Without Discussion
**Situation**: PR introduces breaking changes that weren't discussed.
**Response:**
```markdown
Thanks for this PR! However, I notice this introduces breaking changes that weren't discussed in the original issue.
Breaking changes require:
1. Prior discussion and approval
2. Migration guide for users
3. Deprecation period (when possible)
4. Clear documentation of the change
Could we discuss the breaking changes first? Specifically:
- [What breaks and why]
- [Who will be affected]
- [Migration path]
We may need to adjust the approach to minimize impact on existing users.
```
## Communication Templates
### Closing a PR (Misaligned with Vision)
```markdown
Thank you for taking the time to contribute! We really appreciate it.
After careful review, we've decided not to merge this PR because [specific reason related to design principles].
This isn't a reflection on your code quality - it's about maintaining focus on our core goals as outlined in [VISION.md](https://github.com/lfnovo/open-notebook/blob/main/VISION.md).
We'd love to have you contribute in other ways! Check out:
- Good first issues
- Help wanted issues
- Our roadmap
Thanks again for your interest in Open Notebook!
```
### Closing a Stale Issue
```markdown
We're closing this issue due to inactivity. If this is still relevant, feel free to reopen it with updated information.
Thanks!
```
### Asking for More Information
```markdown
Thanks for reporting this! To help us investigate, could you provide:
1. [Specific information needed]
2. [Logs, screenshots, etc.]
3. [Steps to reproduce]
This will help us understand the issue better and find a solution.
```
### Thanking a Contributor
```markdown
Merged!
Thank you so much for this contribution, @username! [Specific thing they did well].
This will be included in the next release.
```
## Best Practices
### Be Kind and Respectful
- Thank contributors for their time and effort
- Assume good intentions
- Be patient with newcomers
- Explain *why*, not just *what*
### Be Clear and Direct
- Don't leave ambiguity about next steps
- Be specific about what needs to change
- Explain architectural decisions
- Set clear expectations
### Be Consistent
- Apply the same standards to all contributors
- Follow the process you've defined
- Document decisions for future reference
### Be Protective of Project Vision
- It's okay to say "no"
- Prioritize long-term maintainability
- Don't accept features you can't support
- Keep the project focused
### Be Responsive
- Respond to issues within 48 hours (even just to acknowledge)
- Review PRs within a week when possible
- Keep contributors updated on status
- Close stale issues/PRs to keep things tidy
## When in Doubt
Ask yourself:
1. Does this align with our [vision and principles](../../VISION.md)?
2. Will we be able to maintain this feature long-term?
3. Does this benefit most users, or just an edge case?
4. Is there a simpler alternative?
5. Would I want to support this in 2 years?
If you're unsure, it's perfectly fine to:
- Ask for input from other maintainers
- Start a discussion issue
- Sleep on it before making a decision
---
**Remember**: Good maintainership is about balancing openness to contributions with protection of project vision. You're not being mean by saying "no" to things that don't fit - you're being a responsible steward of the project.
docs/7-DEVELOPMENT/podcasts.md
+28
View File
@@ -0,0 +1,28 @@
# Podcast Subsystem
How podcast generation is modeled and executed: the two-tier profile system, the model-registry references, and the deliberate no-auto-retry policy.
## Two-tier profile system (`open_notebook/podcasts/models.py`)
- **SpeakerProfile** — voice configuration: a `voice_model` (`record<model>` reference for TTS) plus 1–4 speakers (name, voice_id, backstory, personality). Individual speakers can override the profile's `voice_model`.
- **EpisodeProfile** — generation settings: `outline_llm` / `transcript_llm` (`record<model>` references), `language` (BCP 47, e.g. `pt-BR`), segment count (3–20), briefing template. It references a SpeakerProfile by name.
- **PodcastEpisode** — a generated episode. Links content, profiles and the async job (`command` field → surreal-commands RecordID).
## Model registry references, not strings
Profile fields reference `Model` records instead of raw provider/model strings. At generation time `_resolve_model_config(model_id)` loads the Model, resolves its linked credential (or falls back to `provision_provider_keys()`), and returns `(provider, model_name, config)` for podcast-creator.
The legacy string fields (`tts_provider`, `outline_provider`, …) that predated the registry were dropped by SQL migration 22 (#1107). The migration best-effort maps any still-unresolved profile to an existing `model` record (provider + name + type) before dropping the columns; profiles with no matching record stay unresolved — the UI already flags them as needing model selection and the user re-picks once. The old startup data migration (`open_notebook/podcasts/migration.py`) is gone.
## Profile snapshots
`PodcastEpisode` stores `episode_profile` and `speaker_profile` as **dicts (snapshots)**, not references. Editing a profile never retroactively changes past episodes — that's intentional. Corollary: deleting a profile does not cascade to episodes.
## Job lifecycle and the retry policy
Generation runs as a `generate_podcast_command` job on the surreal-commands worker:
- The command resolves model configs and credentials for **all** profiles before invoking podcast-creator, and validates that `outline_llm`, `transcript_llm` and `voice_model` are set.
- **`max_attempts: 1` — no automatic retries.** A mid-generation retry would create duplicate episode records (records are created during execution). Failed episodes are marked `failed` with an error message; retry is explicitly user-initiated via `POST /podcasts/episodes/{id}/retry`.
- Status tracking: `get_job_status()` / `get_job_detail()` query surreal-commands and return `"unknown"` on failure rather than raising. Listing endpoints use the batched `get_job_details_for_commands()` so N episodes cost one status query, not N.
- TTS failures fall back to silent audio rather than failing the episode.
docs/7-DEVELOPMENT/prompts.md
+53
View File
@@ -0,0 +1,53 @@
# Prompt Engineering
How prompts are organized and the patterns they use. All prompts are Jinja2 templates under `prompts/`, rendered with the [ai-prompter](https://github.com/lfnovo/ai-prompter) library — prompt engineering lives in templates, not Python.
## Layout & rendering
Templates are grouped by workflow — `ask/`, `chat/`, `source_chat/`, `podcast/` — and referenced by path without extension:
```python
from ai_prompter import Prompter
prompt = Prompter(prompt_template="ask/entry", parser=parser).render(data=state)
```
Mechanical rules (path syntax, `data=` key matching, parser injection, no inheritance, cache → restart) are in [`open_notebook/AGENTS.md`](../../open_notebook/AGENTS.md). This page covers the *patterns*.
## Pattern: multi-stage chain (ask workflow)
The ask pipeline is three templates orchestrated by `graphs/ask.py`:
```
entry.jinja user question → JSON search strategy (PydanticOutputParser)
↓
query_process.jinja one search term + retrieved results → sub-answer (parallel, one per search)
↓
final_answer.jinja all sub-answers → synthesized final response with citations
```
The stage boundaries let each prompt do one job well, and the JSON strategy output makes the fan-out deterministic.
## Pattern: conditional variable injection
Templates accept optional variables via Jinja conditionals, so one template serves several context shapes (podcast outline handles list or string context; source_chat injects optional notebook/insight data):
```jinja
{% if notebook %}
# PROJECT INFORMATION
{{ notebook }}
{% endif %}
```
Watch the loose truthiness (`{% if var %}` is false for empty string/list) and the for-loop assumption (passing a string where a list is expected iterates character by character).
## Pattern: repeated citation emphasis
Response-generating templates (ask, chat) state the citation rules — `[source:id]`, `[note:id]`, `[insight:id]`, "do not make up document IDs" — **multiple times, with inline examples**. LLMs hallucinate citations without this; repetition + examples measurably reduces it. Keep the repetition when editing these templates.
## Pattern: format-instructions delegation
Templates expose an `{{ format_instructions }}` slot filled by the caller's OutputParser. Output format evolves in Python (Pydantic models) without touching the template. If the placeholder is missing, the parser is silently ignored — check for it when adding structured output.
## Pattern: extended-thinking separation (podcast)
Podcast templates instruct thinking models to keep reasoning inside `<think>` tags and emit the JSON after them; `clean_thinking_content()` strips the tags downstream. If a new template expects structured output from thinking-capable models, include the same instruction block.
docs/7-DEVELOPMENT/quick-start.md
+128
View File
@@ -0,0 +1,128 @@
# Quick Start - Development
Get Open Notebook running locally in 5 minutes.
## Prerequisites
- **Python 3.11+**
- **Git**
- **uv** (package manager) - install with `curl -LsSf https://astral.sh/uv/install.sh | sh`
- **Docker** (optional, for SurrealDB)
## 1. Clone the Repository (2 min)
```bash
# Fork the repository on GitHub first, then clone your fork
git clone https://github.com/YOUR_USERNAME/open-notebook.git
cd open-notebook
# Add upstream remote for updates
git remote add upstream https://github.com/lfnovo/open-notebook.git
```
## 2. Install Dependencies (2 min)
```bash
# Install Python dependencies
uv sync
# Verify uv is working
uv --version
```
## 3. Start Services (1 min)
In separate terminal windows:
```bash
# Terminal 1: Start SurrealDB (database)
make database
# or: docker run -d --name surrealdb -p 127.0.0.1:8000:8000 surrealdb/surrealdb:v2 start --user root --pass password memory
# Terminal 2: Start API (backend on port 5055)
make api
# or: uv run --env-file .env uvicorn api.main:app --host 0.0.0.0 --port 5055
# Terminal 3: Start Frontend (UI on port 3000)
cd frontend && npm run dev
```
## 4. Verify Everything Works (instant)
- **API Health**: http://localhost:5055/health → should return `{"status": "ok"}`
- **API Docs**: http://localhost:5055/docs → interactive API documentation
- **Frontend**: http://localhost:3000 → Open Notebook UI
**All three show up?** ✅ You're ready to develop!
---
## Next Steps
- **First Issue?** Pick a [good first issue](https://github.com/lfnovo/open-notebook/issues?q=label%3A%22good+first+issue%22)
- **Understand the code?** Read [Architecture Overview](architecture.md)
- **Make changes?** Follow [Contributing Guide](contributing.md)
- **Setup details?** See [Development Setup](development-setup.md)
---
## Troubleshooting
### "Port 5055 already in use"
```bash
# Find what's using the port
lsof -i :5055
# Use a different port
uv run uvicorn api.main:app --port 5056
```
### "Can't connect to SurrealDB"
```bash
# Check if SurrealDB is running
docker ps | grep surrealdb
# Restart it
make database
```
### "Python version is too old"
```bash
# Check your Python version
python --version # Should be 3.11+
# Use Python 3.11 specifically
uv sync --python 3.11
```
### "npm: command not found"
```bash
# Install Node.js from https://nodejs.org/
# Then install frontend dependencies
cd frontend && npm install
```
---
## Common Development Commands
```bash
# Run tests
uv run pytest
# Format code
make ruff
# Type checking
make lint
# Run the full stack
make start-all
# View API documentation
open http://localhost:5055/docs
```
---
Need more help? See [Development Setup](development-setup.md) for details or join our [Discord](https://discord.gg/37XJPXfz2w).
docs/7-DEVELOPMENT/security.md
+197
View File
@@ -0,0 +1,197 @@
# Security Guidelines
This document outlines security practices for Open Notebook development. It is informed by real vulnerabilities discovered through coordinated disclosure with [CERT-EU](https://cert.europa.eu) and should be treated as mandatory reading for all contributors.
## Reporting Vulnerabilities
If you discover a security vulnerability, **do not open a public GitHub issue**. Instead:
1. Use [GitHub Security Advisories](https://github.com/lfnovo/open-notebook/security/advisories/new) to report privately
2. Or email the maintainers directly
We follow coordinated vulnerability disclosure and will work with you on a fix before any public announcement.
---
## Database Queries (SurrealQL Injection)
**Rule: Never interpolate user input into SurrealQL queries via f-strings.**
SurrealQL injection is the equivalent of SQL injection. User-controlled values must be passed as parameterized bind variables using `$variable` syntax.
### Parameterized queries (safe)
```python
# Good: parameterized query
result = await repo_query(
"SELECT * FROM source WHERE id = $id",
{"id": ensure_record_id(source_id)}
)
```
### F-string interpolation (vulnerable)
```python
# Bad: user input in f-string
result = await repo_query(f"SELECT * FROM source WHERE id = {source_id}")
```
### ORDER BY and other clauses that can't be parameterized
`ORDER BY`, `LIMIT`, and similar clauses typically cannot accept bind variables in SurrealDB. Use **allowlist validation** instead:
```python
# Good: validate against allowlist, then interpolate
allowed_fields = {"name", "created", "updated"}
allowed_directions = {"asc", "desc"}
parts = order_by.strip().lower().split()
if parts[0] not in allowed_fields:
raise HTTPException(status_code=400, detail="Invalid sort field")
if len(parts) > 1 and parts[1] not in allowed_directions:
raise HTTPException(status_code=400, detail="Invalid sort direction")
query = f"SELECT * FROM notebook ORDER BY {validated_order_by}"
```
See `api/routers/sources.py` for the reference implementation of sort parameter validation.
### Checklist
- [ ] All user-provided values use `$variable` binding
- [ ] Any f-string in a query only contains validated/hardcoded values
- [ ] `ORDER BY`, `LIMIT`, etc. use allowlist validation
- [ ] Database values used in subsequent queries are also parameterized (prevents second-order injection)
---
## Template Rendering (Server-Side Template Injection)
**Rule: Always use `SandboxedEnvironment` when rendering Jinja2 templates that contain user-provided content.**
The [ai-prompter](https://github.com/lfnovo/ai-prompter) library (>= 0.4.0) uses `SandboxedEnvironment` by default, which blocks access to dangerous Python attributes like `__globals__`, `__subclasses__`, and `__init__`.
### What SandboxedEnvironment prevents
```jinja2
{# These are blocked and raise SecurityError #}
{{ cycler.__init__.__globals__.os.popen('id').read() }}
{{ ''.__class__.__mro__[1].__subclasses__() }}
```
### Guidelines
- Never downgrade ai-prompter below 0.4.0
- If using Jinja2 directly (outside ai-prompter), always use `jinja2.sandbox.SandboxedEnvironment`
- Never pass user-provided strings to `jinja2.Environment` or `jinja2.Template` directly
### Third-party libraries with the same shape
The `podcast_creator` library's `configure("templates", {...})` compiles the given string directly as Jinja2 template source (`Prompter(template_text=...)` in its `config.py`) - the identical pattern to the vulnerability above. `commands/podcast_commands.py` never calls it (confirmed: podcast generation always uses the file-based `prompts/podcast/*.jinja` templates in this repo), so this is currently dormant, not exploitable. If a "custom podcast template" feature is ever added, route user/profile text through a fixed, developer-authored template with the text passed in as a plain variable - do not wire it into `configure("templates", ...)`.
---
## File Handling (Path Traversal and Local File Inclusion)
### File uploads
**Rule: Always sanitize filenames and validate resolved paths.**
```python
import os
from pathlib import Path
# 1. Strip directory components
safe_filename = os.path.basename(original_filename)
# 2. Validate resolved path stays within target directory
resolved = (Path(upload_folder) / safe_filename).resolve()
if not str(resolved).startswith(str(Path(upload_folder).resolve()) + os.sep):
raise ValueError("Path traversal detected")
```
Key points:
- Use `os.path.basename()` to strip directory components from user-provided filenames
- Use `Path.resolve()` to resolve symlinks and `..` components
- Use `startswith()` with a **trailing `os.sep`** to prevent sibling directory bypass (e.g., `/uploads_evil/` matching `/uploads`)
### File path inputs
**Rule: Validate that any user-provided file path is within the expected directory.**
```python
uploads_resolved = Path(UPLOADS_FOLDER).resolve()
file_resolved = Path(user_provided_path).resolve()
if not str(file_resolved).startswith(str(uploads_resolved) + os.sep):
raise HTTPException(status_code=400, detail="Invalid file path")
```
Never pass user-provided file paths directly to file reading or content extraction functions without validation.
### Checklist
- [ ] Filenames from uploads are sanitized with `os.path.basename()`
- [ ] Resolved paths are validated with `startswith(directory + os.sep)`
- [ ] User-provided `file_path` values are validated before use
- [ ] No directory creation from user input (`mkdir` with traversal paths)
---
## Authentication and CORS
### Authentication
Open Notebook currently uses simple password-based middleware (`PasswordAuthMiddleware`). This is suitable for single-user self-hosted deployments but should be hardened for production:
- Set `OPEN_NOTEBOOK_PASSWORD` explicitly - there is no hardcoded default password; if it's unset, auth is fully disabled (all requests pass through unchecked)
- Change the default encryption key (`OPEN_NOTEBOOK_ENCRYPTION_KEY`)
- Consider deploying behind a reverse proxy with proper authentication (OAuth, OIDC)
### CORS
The default CORS configuration allows all origins (`allow_origins=["*"]`). `allow_credentials` is tied to that: `False` for the default wildcard (avoids Starlette reflecting any Origin alongside credentials), automatically `True` once `CORS_ORIGINS` is explicitly scoped to specific origins. For production deployments, still restrict `CORS_ORIGINS` to only the frontend URL.
---
## Secrets Management
### Encryption key
`OPEN_NOTEBOOK_ENCRYPTION_KEY` is used to encrypt API keys stored in SurrealDB. In production:
- Set a strong, unique key (do not use the default)
- Use Docker secrets via `OPEN_NOTEBOOK_ENCRYPTION_KEY_FILE` when possible
- Never log or expose this value
### Environment variables
- Sensitive values (API keys, passwords, encryption keys) should never appear in logs
- Use `loguru` with caution — avoid logging full request bodies or environment dumps
- The Docker container runs as root by default; consider running as a non-root user
---
## Code Review Security Checklist
When reviewing PRs, check for:
1. **Query injection**: Any f-string containing user input in a SurrealQL query
2. **Template injection**: User-provided strings passed to Jinja2 without sandboxing
3. **Path traversal**: User-provided filenames or paths used without sanitization
4. **Information disclosure**: Error messages that expose internal paths, stack traces, or configuration
5. **SSRF**: User-provided URLs passed to server-side HTTP requests without validation
6. **Secrets in logs**: Sensitive values logged at any level
---
## Past Vulnerabilities
These vulnerabilities were reported by CERT-EU and are documented here as learning examples:
| Version | Vulnerability | Severity | Advisory |
|---------|--------------|----------|----------|
| <= 1.8.2 | SurrealDB injection via `order_by` parameter | High (8.7) | [GHSA-5wj9-f8q5-8f9c](https://github.com/lfnovo/open-notebook/security/advisories/GHSA-5wj9-f8q5-8f9c) |
| <= 1.8.3 | RCE via Jinja2 SSTI in transformations | Critical (9.2) | [GHSA-f35w-wx37-26q7](https://github.com/lfnovo/open-notebook/security/advisories/GHSA-f35w-wx37-26q7) |
| <= 1.8.3 | Arbitrary file write via path traversal | High (7.0) | [GHSA-x4q2-89g5-594v](https://github.com/lfnovo/open-notebook/security/advisories/GHSA-x4q2-89g5-594v) |
| <= 1.8.3 | Arbitrary file read via LFI | High (8.2) | [GHSA-842v-h4cj-r646](https://github.com/lfnovo/open-notebook/security/advisories/GHSA-842v-h4cj-r646) |
docs/7-DEVELOPMENT/testing.md
+423
View File
@@ -0,0 +1,423 @@
# Testing Guide
This document provides guidelines for writing tests in Open Notebook. Testing is critical to maintaining code quality and preventing regressions.
## Testing Philosophy
### What to Test
Focus on testing the things that matter most:
- **Business Logic** - Core domain models and their operations
- **API Contracts** - HTTP endpoint behavior and error handling
- **Critical Workflows** - End-to-end flows that users depend on
- **Data Persistence** - Database operations and data integrity
- **Error Conditions** - How the system handles failures gracefully
### What NOT to Test
Don't waste time testing framework code:
- Framework functionality (FastAPI, React, etc.)
- Third-party library implementation
- Simple getters/setters without logic
- View/presentation layer rendering (unless it contains logic)
## Test Structure
We use **pytest** with async support for all Python tests:
```python
import pytest
from httpx import AsyncClient
from open_notebook.domain.notebook import Notebook
@pytest.mark.asyncio
async def test_create_notebook():
"""Test notebook creation."""
notebook = Notebook(name="Test Notebook", description="Test description")
await notebook.save()
assert notebook.id is not None
assert notebook.name == "Test Notebook"
assert notebook.created is not None
@pytest.mark.asyncio
async def test_api_create_notebook():
"""Test notebook creation via API."""
async with AsyncClient(app=app, base_url="http://test") as client:
response = await client.post(
"/api/notebooks",
json={"name": "Test Notebook", "description": "Test description"}
)
assert response.status_code == 200
data = response.json()
assert data["name"] == "Test Notebook"
```
## Test Categories
### 1. Unit Tests
Test individual functions and methods in isolation:
```python
@pytest.mark.asyncio
async def test_notebook_validation():
"""Test that notebook name validation works."""
with pytest.raises(InvalidInputError):
Notebook(name="", description="test")
@pytest.mark.asyncio
async def test_notebook_archive():
"""Test notebook archiving."""
notebook = Notebook(name="Test", description="")
notebook.archive()
assert notebook.archived is True
```
**Location**: `tests/unit/`
### 2. Integration Tests
Test component interactions and database operations:
```python
@pytest.mark.asyncio
async def test_create_notebook_with_sources():
"""Test creating a notebook and adding sources."""
notebook = await create_notebook(name="Research", description="")
source = await add_source(notebook_id=notebook.id, url="https://example.com")
retrieved = await get_notebook_with_sources(notebook.id)
assert len(retrieved.sources) == 1
assert retrieved.sources[0].id == source.id
```
**Location**: `tests/integration/`
### 3. API Tests
Test HTTP endpoints and error responses:
```python
@pytest.mark.asyncio
async def test_get_notebooks_endpoint():
"""Test GET /notebooks endpoint."""
async with AsyncClient(app=app, base_url="http://test") as client:
response = await client.get("/api/notebooks")
assert response.status_code == 200
data = response.json()
assert isinstance(data, list)
@pytest.mark.asyncio
async def test_create_notebook_validation():
"""Test that invalid input is rejected."""
async with AsyncClient(app=app, base_url="http://test") as client:
response = await client.post(
"/api/notebooks",
json={"name": "", "description": ""}
)
assert response.status_code == 400
```
**Location**: `tests/api/`
### 4. Database Tests
Test data persistence and query correctness:
```python
@pytest.mark.asyncio
async def test_save_and_retrieve_notebook():
"""Test saving and retrieving a notebook from database."""
notebook = Notebook(name="Test", description="desc")
await notebook.save()
retrieved = await Notebook.get(notebook.id)
assert retrieved.name == "Test"
assert retrieved.description == "desc"
@pytest.mark.asyncio
async def test_query_by_criteria():
"""Test querying notebooks by criteria."""
await create_notebook("Active", "")
await create_notebook("Archived", "")
active = await repo_query(
"SELECT * FROM notebook WHERE archived = false"
)
assert len(active) >= 1
```
**Location**: `tests/database/`
## Running Tests
### Run All Tests
```bash
uv run pytest
```
### Run Specific Test File
```bash
uv run pytest tests/test_notebooks.py
```
### Run Specific Test Function
```bash
uv run pytest tests/test_notebooks.py::test_create_notebook
```
### Run with Coverage Report
```bash
uv run pytest --cov=open_notebook
```
### Run Only Unit Tests
```bash
uv run pytest tests/unit/
```
### Run Only Integration Tests
```bash
uv run pytest tests/integration/
```
### Run Tests in Verbose Mode
```bash
uv run pytest -v
```
### Run Tests with Output
```bash
uv run pytest -s
```
## Test Fixtures
Use pytest fixtures for common setup and teardown:
```python
import pytest
@pytest.fixture
async def test_notebook():
"""Create a test notebook."""
notebook = Notebook(name="Test Notebook", description="Test description")
await notebook.save()
yield notebook
await notebook.delete()
@pytest.fixture
async def api_client():
"""Create an API test client."""
async with AsyncClient(app=app, base_url="http://test") as client:
yield client
@pytest.fixture
async def test_notebook_with_sources(test_notebook):
"""Create a test notebook with sample sources."""
source1 = Source(notebook_id=test_notebook.id, url="https://example.com")
source2 = Source(notebook_id=test_notebook.id, url="https://example.org")
await source1.save()
await source2.save()
test_notebook.sources = [source1, source2]
yield test_notebook
# Cleanup
await source1.delete()
await source2.delete()
```
## Best Practices
### 1. Write Descriptive Test Names
```python
# Good - clearly describes what is being tested
async def test_create_notebook_with_valid_name_succeeds():
...
# Bad - vague about what's being tested
async def test_notebook():
...
```
### 2. Use Docstrings
```python
@pytest.mark.asyncio
async def test_vector_search_returns_sorted_results():
"""Test that vector search results are sorted by relevance score."""
# Implementation
```
### 3. Test Edge Cases
```python
@pytest.mark.asyncio
async def test_search_with_empty_query():
"""Test that empty query raises error."""
with pytest.raises(InvalidInputError):
await vector_search("")
@pytest.mark.asyncio
async def test_search_with_very_long_query():
"""Test that very long query is handled."""
long_query = "x" * 10000
results = await vector_search(long_query)
assert isinstance(results, list)
@pytest.mark.asyncio
async def test_search_with_special_characters():
"""Test that special characters are handled."""
results = await vector_search("@#$%^&*()")
assert isinstance(results, list)
```
### 4. Use Assertions Effectively
```python
# Good - specific assertions
assert notebook.name == "Test"
assert len(notebook.sources) == 3
assert notebook.created is not None
# Less good - too broad
assert notebook is not None
assert notebook # ambiguous what's being tested
```
### 5. Test Both Success and Failure Cases
```python
@pytest.mark.asyncio
async def test_create_notebook_success():
"""Test successful notebook creation."""
notebook = await create_notebook(name="Research", description="AI")
assert notebook.id is not None
assert notebook.name == "Research"
@pytest.mark.asyncio
async def test_create_notebook_empty_name_fails():
"""Test that empty name raises error."""
with pytest.raises(InvalidInputError):
await create_notebook(name="", description="")
@pytest.mark.asyncio
async def test_create_notebook_duplicate_fails():
"""Test that duplicate names are handled."""
await create_notebook(name="Research", description="")
with pytest.raises(DuplicateError):
await create_notebook(name="Research", description="")
```
### 6. Keep Tests Independent
```python
# Good - test is self-contained
@pytest.mark.asyncio
async def test_archive_notebook():
notebook = Notebook(name="Test", description="")
await notebook.save()
await notebook.archive()
assert notebook.archived is True
# Bad - depends on another test's state
@pytest.mark.asyncio
async def test_archive_existing_notebook():
# Assumes test_create_notebook ran first
await notebook.archive() # notebook undefined
```
### 7. Use Fixtures for Reusable Setup
```python
# Instead of repeating setup:
@pytest.fixture
async def client_with_auth(api_client, mock_auth):
"""Client with authentication set up."""
api_client.headers.update({"Authorization": f"Bearer {mock_auth.token}"})
yield api_client
@pytest.mark.asyncio
async def test_protected_endpoint(client_with_auth):
"""Test protected endpoint."""
response = await client_with_auth.get("/api/protected")
assert response.status_code == 200
```
## Coverage Goals
- Aim for 70%+ overall coverage
- 90%+ coverage for critical business logic
- Don't obsess over 100% - focus on meaningful tests
- Use `--cov` flag to check coverage: `uv run pytest --cov=open_notebook`
## Async Test Patterns
### Testing Async Functions
```python
@pytest.mark.asyncio
async def test_async_operation():
"""Test async function."""
result = await some_async_function()
assert result is not None
```
### Testing Concurrent Operations
```python
@pytest.mark.asyncio
async def test_concurrent_notebook_creation():
"""Test creating multiple notebooks concurrently."""
tasks = [
create_notebook(f"Notebook {i}", "")
for i in range(10)
]
notebooks = await asyncio.gather(*tasks)
assert len(notebooks) == 10
assert all(n.id for n in notebooks)
```
## Common Testing Errors
### Error: "event loop is closed"
Solution: Use the async fixture properly:
```python
@pytest.fixture
async def notebook(): # Use async fixture
notebook = Notebook(name="Test", description="")
await notebook.save()
yield notebook
await notebook.delete()
```
### Error: "object is not awaitable"
Solution: Make sure you're using await:
```python
# Wrong
result = create_notebook("Test", "")
# Right
result = await create_notebook("Test", "")
```
---
**See also:**
- [Code Standards](code-standards.md) - Code formatting and style
- [Contributing Guide](contributing.md) - Overall contribution workflow
docs/SECURITY_REVIEW.md
+96
View File
@@ -0,0 +1,96 @@
# Security Review - API Configuration UI
## Date: 2026-01-27 (Updated: 2026-01-28)
## Reviewer: Security Audit
---
## Summary
Security review of the API key management implementation for Open Notebook. The implementation uses a database-first approach with environment variable fallback.
---
## Encryption
| Item | Status | Notes |
|------|--------|-------|
| Fernet encryption implemented | PASS | `open_notebook/utils/encryption.py` uses AES-128-CBC + HMAC-SHA256 |
| Keys encrypted before DB storage | PASS | `encrypt_value()` applied on save |
| Keys decrypted only when needed | PASS | `decrypt_value()` called when reading |
| Encryption key required | PASS | No default key; ValueError if not configured |
| Docker secrets support | PASS | `_FILE` suffix pattern supported |
| Documented in .env.example | PASS | Encryption key documented |
---
## API Security
| Item | Status | Notes |
|------|--------|-------|
| Test endpoint implemented | PASS | `connection_tester.py` validates keys |
| Test doesn't expose keys | PASS | Only returns success/failure |
| Error messages don't leak info | PASS | Generic error messages |
| URL validation for SSRF | PASS | Blocks private IPs (except Ollama) |
| Rate limiting | NOT IMPL | Future enhancement |
---
## Frontend Security
| Item | Status | Notes |
|------|--------|-------|
| No keys in localStorage | PASS | Keys only in React state during entry |
| Keys masked in UI | PASS | Shows `************` placeholder |
| No keys in console.log | PASS | No logging of sensitive data |
| autocomplete attributes | PARTIAL | Some forms missing autocomplete="off" |
---
## Authentication
| Item | Status | Notes |
|------|--------|-------|
| Password protection | PASS | Bearer token authentication |
| Default password | PASS | Auth is fully disabled (not a hardcoded default password) when `OPEN_NOTEBOOK_PASSWORD` is unset |
| Docker secrets support | PASS | `_FILE` suffix for password |
| Security warnings | PASS | Logged when using defaults |
---
## Files Reviewed
| Component | Path | Status |
|-----------|------|--------|
| Encryption | `open_notebook/utils/encryption.py` | PASS |
| Credential model | `open_notebook/domain/credential.py` | PASS |
| Credentials router | `api/routers/credentials.py` | PASS |
| Key provider | `open_notebook/ai/key_provider.py` | PASS |
| Connection tester | `open_notebook/ai/connection_tester.py` | PASS |
| Auth middleware | `api/auth.py` | PASS |
| Frontend forms | `frontend/src/components/settings/*.tsx` | PASS |
| Environment example | `.env.example` | PASS |
---
## Remaining Recommendations
### Future Improvements
1. **Rate limiting** - Add rate limiting on `/credentials/*` endpoints
2. **Autocomplete attributes** - Add `autocomplete="new-password"` to all password inputs
3. **Show last 4 characters** - Display `********xxxx` format for key identification
4. **Audit logging** - Log API key changes with timestamps
---
## Conclusion
The API Configuration UI implementation meets security requirements:
- API keys encrypted at rest using Fernet (key must be explicitly configured)
- Keys never returned to frontend
- URL validation prevents SSRF attacks
- Docker secrets supported for production deployments
**Review Status: PASS**
docs/assets/asset_list.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 236 KiB

docs/assets/hero.svg
+60
View File
@@ -0,0 +1,60 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg width="240" height="240" viewBox="0 0 240 240" fill="none" xmlns="http://www.w3.org/2000/svg">
<!-- Gradient Definitions -->
<defs>
<linearGradient id="nodeGradient" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#bd34fe;stop-opacity:1" />
<stop offset="100%" style="stop-color:#41d1ff;stop-opacity:1" />
</linearGradient>
<linearGradient id="lineGradient" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#bd34fe;stop-opacity:0.3" />
<stop offset="100%" style="stop-color:#41d1ff;stop-opacity:0.3" />
</linearGradient>
</defs>
<!-- Connection Lines -->
<g class="connections">
<path d="M120 80L80 120" stroke="url(#lineGradient)" stroke-width="2"/>
<path d="M120 80L160 120" stroke="url(#lineGradient)" stroke-width="2"/>
<path d="M80 120L120 160" stroke="url(#lineGradient)" stroke-width="2"/>
<path d="M160 120L120 160" stroke="url(#lineGradient)" stroke-width="2"/>
<path d="M80 120L160 120" stroke="url(#lineGradient)" stroke-width="2"/>
</g>
<!-- Nodes -->
<g class="nodes">
<!-- Central Node -->
<circle cx="120" cy="80" r="20" fill="url(#nodeGradient)"/>
<circle cx="120" cy="80" r="24" stroke="url(#nodeGradient)" stroke-width="1" fill="none"/>
<!-- Left Node -->
<circle cx="80" cy="120" r="16" fill="url(#nodeGradient)"/>
<circle cx="80" cy="120" r="20" stroke="url(#nodeGradient)" stroke-width="1" fill="none"/>
<!-- Right Node -->
<circle cx="160" cy="120" r="16" fill="url(#nodeGradient)"/>
<circle cx="160" cy="120" r="20" stroke="url(#nodeGradient)" stroke-width="1" fill="none"/>
<!-- Bottom Node -->
<circle cx="120" cy="160" r="18" fill="url(#nodeGradient)"/>
<circle cx="120" cy="160" r="22" stroke="url(#nodeGradient)" stroke-width="1" fill="none"/>
</g>
<!-- Pulse Animation -->
<style>
@keyframes pulse {
0% { transform: scale(1); opacity: 0.8; }
50% { transform: scale(1.1); opacity: 1; }
100% { transform: scale(1); opacity: 0.8; }
}
.nodes circle {
animation: pulse 3s infinite ease-in-out;
}
.nodes circle:nth-child(2n) {
animation-delay: 1s;
}
.nodes circle:nth-child(3n) {
animation-delay: 2s;
}
</style>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 2.3 KiB

docs/index.md
+289
View File
@@ -0,0 +1,289 @@
# Open Notebook Documentation
Welcome to Open Notebook - a privacy-focused AI research assistant. This documentation is organized for different needs.
---
## 🎯 Choose Your Path
### I'm brand new
→ Start here: **[0-START-HERE](0-START-HERE/index.md)**
- Learn what Open Notebook is
- Pick your setup path (OpenAI, cloud, local/Ollama)
- 5-minute quick start
### I need to install/deploy
→ Go here: **[1-INSTALLATION](1-INSTALLATION/index.md)**
- Multiple installation routes
- Docker Compose (recommended)
- From source (developers)
- ~~Single container~~ (deprecated, see Docker Compose)
### I want to understand how it works
→ Read this: **[2-CORE-CONCEPTS](2-CORE-CONCEPTS/index.md)**
- Mental models and architecture
- How RAG (retrieval-augmented generation) works
- Notebooks, sources, and notes explained
- Chat vs. transformations vs. podcasts
### I want to use it (tutorials)
→ Follow this: **[3-USER-GUIDE](3-USER-GUIDE/index.md)**
- How to add sources (PDFs, URLs, audio, video)
- Creating and organizing notes
- Chat effectively with your research
- Creating podcasts from research
- Search techniques
### I need to configure it
→ Check this: **[5-CONFIGURATION](5-CONFIGURATION/index.md)**
- Choose and setup AI provider
- API configuration
- Database setup
- Advanced tuning
### I need provider-specific help
→ Go here: **[4-AI-PROVIDERS](4-AI-PROVIDERS/index.md)**
- OpenAI, Anthropic, Google, Groq, Ollama, Azure
- Model comparisons
- Cost estimates
- Setup paths
### Something's not working
→ Troubleshoot: **[6-TROUBLESHOOTING](6-TROUBLESHOOTING/index.md)**
- Quick fixes (top 10 issues)
- Installation problems
- Connection issues
- AI/chat problems
- Content processing issues
- Podcast problems
### I want to contribute/develop
→ Read this: **[7-DEVELOPMENT](7-DEVELOPMENT/index.md)**
- Architecture and tech stack
- Contributing guidelines
- API reference
- Testing
---
## 📊 Documentation Overview
### By Section
**[0-START-HERE](0-START-HERE/index.md)** — Entry point
- What is Open Notebook?
- Quick start guides (3 routes)
- First 5 minutes
**[1-INSTALLATION](1-INSTALLATION/index.md)** — Getting it running
- Multiple installation routes
- Docker Compose (recommended), from-source
- Requirements and setup
**[2-CORE-CONCEPTS](2-CORE-CONCEPTS/index.md)** — Understanding the system
- Notebooks, sources, notes hierarchy
- RAG (retrieval-augmented generation)
- Chat, transformations, podcasts
- Context management
**[3-USER-GUIDE](3-USER-GUIDE/index.md)** — Using features
- Adding sources (all types)
- Working with notes
- Chat effectively
- Creating podcasts
- Searching (text and semantic)
**[4-AI-PROVIDERS](4-AI-PROVIDERS/index.md)** — AI configuration
- Provider comparison
- Setup for each provider
- Model recommendations
- Cost estimates
**[5-CONFIGURATION](5-CONFIGURATION/index.md)** — Complete reference
- AI provider setup (detailed)
- Database configuration
- Server/API settings
- Advanced tuning
- Environment variables (complete reference)
**[6-TROUBLESHOOTING](6-TROUBLESHOOTING/index.md)** — Problem solving
- Quick fixes (top 10)
- Installation issues
- Connection problems
- AI/chat issues
- Content processing
- Podcast generation
- Getting help
**[7-DEVELOPMENT](7-DEVELOPMENT/index.md)** — For contributors
- Architecture
- Contributing guidelines
- API reference
- Testing & development
---
## 🔍 Find What You Need
### By Problem Type
**Installation & Setup**
- Fresh install? → [0-START-HERE](0-START-HERE/index.md)
- Detailed installation routes? → [1-INSTALLATION](1-INSTALLATION/index.md)
- Configuration reference? → [5-CONFIGURATION](5-CONFIGURATION/index.md)
- Provider setup? → [4-AI-PROVIDERS](4-AI-PROVIDERS/index.md)
**Using Open Notebook**
- How to use features? → [3-USER-GUIDE](3-USER-GUIDE/index.md)
- Understanding concepts? → [2-CORE-CONCEPTS](2-CORE-CONCEPTS/index.md)
- Chat not working? → [6-TROUBLESHOOTING - AI Issues](6-TROUBLESHOOTING/ai-chat-issues.md)
- Files won't upload? → [6-TROUBLESHOOTING - Quick Fixes](6-TROUBLESHOOTING/quick-fixes.md#4-cannot-process-file-or-unsupported-format)
**Troubleshooting**
- Quick fix? → [6-TROUBLESHOOTING - Quick Fixes](6-TROUBLESHOOTING/quick-fixes.md)
- Can't connect? → [6-TROUBLESHOOTING - Connection](6-TROUBLESHOOTING/connection-issues.md)
- Chat issues? → [6-TROUBLESHOOTING - AI Issues](6-TROUBLESHOOTING/ai-chat-issues.md)
- Podcast problems? → [6-TROUBLESHOOTING - Quick Fixes](6-TROUBLESHOOTING/quick-fixes.md#8-podcast-generation-failed)
**Development**
- Architecture? → [7-DEVELOPMENT - Architecture](7-DEVELOPMENT/architecture.md)
- Contributing? → [7-DEVELOPMENT - Contributing](7-DEVELOPMENT/contributing.md)
- API reference? → [7-DEVELOPMENT - API Reference](7-DEVELOPMENT/api-reference.md)
---
## 📚 Reading Paths
### Path 1: Complete Beginner (1-2 hours)
1. [0-START-HERE/index.md](0-START-HERE/index.md) — Understand what it is
2. [0-START-HERE Quick Start](0-START-HERE/index.md) — Set it up
3. [2-CORE-CONCEPTS/index.md](2-CORE-CONCEPTS/index.md) — Understand concepts
4. [3-USER-GUIDE/index.md](3-USER-GUIDE/index.md) — Learn features
**Result:** Fully understand how to use Open Notebook
### Path 2: Get Running Fast (15 minutes)
1. [0-START-HERE](0-START-HERE/index.md) — Pick your path
2. Follow quick-start guide for your setup
3. Start using!
**Result:** Running in 15 minutes, learn details later
### Path 3: DevOps/Deployment (1-2 hours)
1. [1-INSTALLATION](1-INSTALLATION/index.md) — Understand routes
2. [5-CONFIGURATION](5-CONFIGURATION/index.md) — Reference setup
3. [7-DEVELOPMENT - Architecture](7-DEVELOPMENT/architecture.md) — Understand system
**Result:** Ready to deploy to production
### Path 4: Troubleshooting (5-30 minutes)
1. [6-TROUBLESHOOTING/index.md](6-TROUBLESHOOTING/index.md) — Identify problem
2. Find specific guide
3. Follow solutions
**Result:** Problem solved!
---
## ❓ Common Questions
**Q: Where do I start?**
A: → [0-START-HERE](0-START-HERE/index.md) — Choose your setup path
**Q: How do I install it?**
A: → [1-INSTALLATION](1-INSTALLATION/index.md) — Multiple routes available
**Q: How do I use [feature]?**
A: → [3-USER-GUIDE](3-USER-GUIDE/index.md) — Step-by-step tutorials
**Q: Why does [feature] work like that?**
A: → [2-CORE-CONCEPTS](2-CORE-CONCEPTS/index.md) — Understand the mental model
**Q: How do I configure [provider]?**
A: → [4-AI-PROVIDERS](4-AI-PROVIDERS/index.md) or [5-CONFIGURATION](5-CONFIGURATION/index.md)
**Q: Something's broken, what do I do?**
A: → [6-TROUBLESHOOTING](6-TROUBLESHOOTING/index.md) — Problem solver
**Q: How does the system work?**
A: → [2-CORE-CONCEPTS](2-CORE-CONCEPTS/index.md) — Architecture and concepts
**Q: Can I contribute?**
A: → [7-DEVELOPMENT](7-DEVELOPMENT/index.md) — Contributing guide
---
## 📖 How This Documentation is Organized
### Principles
- **Progressive Disclosure**: Start simple, go deeper if needed
- **Multiple Entry Routes**: Different paths for different users
- **High Signal-to-Noise**: Focused content, no fluff
- **Step-by-Step**: Clear instructions you can follow
- **Decision Trees**: Help you pick the right path
- **Symptom-Based**: Troubleshooting by what's broken
### Structure
- **0-START-HERE** — Entry point (everyone starts here)
- **1-INSTALLATION** — Multiple setup routes
- **2-CORE-CONCEPTS** — Mental models (understand why)
- **3-USER-GUIDE** — How to use (step-by-step)
- **4-AI-PROVIDERS** — Provider guides
- **5-CONFIGURATION** — Reference material
- **6-TROUBLESHOOTING** — Problem solving
- **7-DEVELOPMENT** — For contributors
---
## 🚀 Quick Navigation
### First Time?
→ **[START HERE](0-START-HERE/index.md)**
### Just Want to Use It?
→ **[QUICK START](0-START-HERE/index.md)** (5 minutes)
### Something Broken?
→ **[TROUBLESHOOTING](6-TROUBLESHOOTING/index.md)**
### Full Reference?
→ **[CONFIGURATION](5-CONFIGURATION/index.md)**
### Developer?
→ **[DEVELOPMENT](7-DEVELOPMENT/index.md)**
---
## 📞 Getting Help
- **Discord Community** — https://discord.gg/37XJPXfz2w
- **GitHub Issues** — https://github.com/lfnovo/open-notebook/issues
- **Documentation** — You're reading it!
---
## 📈 Documentation Stats
- **8 major sections**
- **35+ focused guides**
- **~80,000 words**
- **Covers all features**
- **Multiple entry paths**
- **Progressive difficulty**
---
## 🎯 Start Here
**First time using Open Notebook?**
→ Go to **[0-START-HERE](0-START-HERE/index.md)**
**Experienced, looking for specific help?**
→ Use the navigation above to find your section
**Something not working?**
→ Go to **[TROUBLESHOOTING](6-TROUBLESHOOTING/index.md)**
---
Last updated: January 2026 | Open Notebook v1.2.4+