wehub/lfnovo--open-notebook
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:10:23 +08:00
commit fbab2c6005
567 changed files with 114434 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.dockerignore
+52
View File
@@ -0,0 +1,52 @@
# Git
.git
.gitignore
# Python
__pycache__
*.pyc
*.pyo
*.pyd
.venv
venv
ENV
env
.pytest_cache
.mypy_cache
.ruff_cache
# Frontend
frontend/node_modules
frontend/.next
frontend/dist
frontend/out
frontend/.env*
frontend/*.log
# Project data
.antigravity
.gemini
tmp
data
mydata
notebook_data
surreal_data
surreal-data
surreal_single_data
*.db
*.log
docker.env
.env
docker-compose*
# Documentation & CI (not needed in image)
docs
.github
# IDE and OS files
.vscode
.idea
*.swp
*.swo
*~
.DS_Store
.env.example
+68
View File
@@ -0,0 +1,68 @@
# Open Notebook Configuration
# Copy this file to .env and customize as needed
# =============================================================================
# REQUIRED
# =============================================================================
# Encryption key for storing API credentials securely in the database
# Change this to any secret string (minimum 16 characters recommended)
OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
# =============================================================================
# DATABASE (Default values work with docker-compose.yml)
# =============================================================================
SURREAL_URL=ws://surrealdb:8000/rpc
# root:root is fine for local use. Change these before exposing the instance to a
# network — docker-compose.yml reads them for both the SurrealDB server and the
# app, so the two stay in sync.
SURREAL_USER=root
SURREAL_PASSWORD=root
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=open_notebook
# =============================================================================
# OPTIONAL: AI Provider API Keys
# =============================================================================
# You can configure these via the UI (Settings → API Keys) or set them here
# UI configuration is recommended for better security and flexibility
# OpenAI
# OPENAI_API_KEY=sk-...
# Anthropic
# ANTHROPIC_API_KEY=sk-ant-...
# Google AI
# GOOGLE_API_KEY=...
# Groq
# GROQ_API_KEY=gsk_...
# =============================================================================
# OPTIONAL: Advanced Configuration
# =============================================================================
# External API URL (for webhooks, callbacks, etc.)
# API_URL=http://localhost:5055
# Ollama endpoint (if running locally)
# OLLAMA_BASE_URL=http://ollama:11434
# Content processing
# CHUNK_SIZE=1500
# CHUNK_OVERLAP=150
# Maximum upload/request body size in MB (default: 100). Raise this if you
# need to upload larger audio/video files. A fronting reverse proxy's own
# limit (e.g. nginx client_max_body_size) still applies and should be raised
# to match.
# OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB=100
# Security
# BASIC_AUTH_USERNAME=admin
# BASIC_AUTH_PASSWORD=secret
# For more configuration options, see:
# https://github.com/lfnovo/open-notebook/blob/main/docs/5-CONFIGURATION/environment-reference.md
.gitattributes
+2
View File
@@ -0,0 +1,2 @@
# Ensure shell scripts always use LF so they run in Linux containers (e.g. Docker)
*.sh text eol=lf
.github/ISSUE_TEMPLATE/bug_report.yml
+104
View File
@@ -0,0 +1,104 @@
name: 🐛 Bug Report
description: Report a bug or unexpected behavior (app is running but misbehaving)
title: "[Bug]: "
labels: ["bug", "needs-triage"]
body:
- type: markdown
attributes:
value: |
Thanks for reporting a bug! Please fill out the information below to help us understand and fix the issue.
**Note**: If you're having installation or setup issues, please use the "Installation Issue" template instead.
- type: textarea
id: what-happened
attributes:
label: What did you do when it broke?
description: Describe the steps you took that led to the bug
placeholder: |
1. I went to the Notebooks page
2. I clicked on "Create New Notebook"
3. I filled in the form and clicked "Save"
4. Then the error occurred...
validations:
required: true
- type: textarea
id: how-broke
attributes:
label: How did it break?
description: What happened that was unexpected? What did you expect to happen instead?
placeholder: |
Expected: The notebook should be created and I should see it in the list
Actual: I got an error message saying "Failed to create notebook"
validations:
required: true
- type: textarea
id: logs-screenshots
attributes:
label: Logs or Screenshots
description: |
Please provide any error messages, logs, or screenshots that might help us understand the issue.
**How to get logs:**
- Docker: `docker compose logs -f open_notebook`
- Check browser console (F12 → Console tab)
placeholder: |
Paste logs here or drag and drop screenshots.
Error messages, stack traces, or browser console errors are very helpful!
validations:
required: false
- type: dropdown
id: version
attributes:
label: Open Notebook Version
description: Which version are you using?
options:
- v1-latest (Docker)
- v1-latest-single (Docker, deprecated)
- Latest from main branch
- Other (please specify in additional context)
validations:
required: true
- type: textarea
id: environment
attributes:
label: Environment
description: What environment are you running in?
placeholder: |
- OS: Ubuntu 22.04 / Windows 11 / macOS 14
- Browser: Chrome 120
validations:
required: false
- type: textarea
id: additional-context
attributes:
label: Additional Context
description: Any other information that might be helpful
placeholder: "This started happening after I upgraded to v1.5.0..."
validations:
required: false
- type: checkboxes
id: willing-to-contribute
attributes:
label: Contribution
description: Would you like to work on fixing this bug?
options:
- label: I am a developer and would like to work on fixing this issue (pending maintainer approval)
required: false
- type: markdown
attributes:
value: |
---
**Next Steps:**
1. A maintainer will review your bug report
2. If you checked the box above and want to fix it, please propose your solution approach
3. Wait for assignment before starting development
4. See our [Contributing Guide](https://github.com/lfnovo/open-notebook/blob/main/CONTRIBUTING.md) for more details
.github/ISSUE_TEMPLATE/config.yml
+11
View File
@@ -0,0 +1,11 @@
blank_issues_enabled: false
contact_links:
- name: 💬 Discord Community
url: https://discord.gg/37XJPXfz2w
about: Get help from the community and share ideas
- name: 🤖 Installation Assistant (ChatGPT)
url: https://chatgpt.com/g/g-68776e2765b48191bd1bae3f30212631-open-notebook-installation-assistant
about: CustomGPT that knows all our docs. Really useful. Try it.
- name: 📚 Documentation
url: https://github.com/lfnovo/open-notebook/tree/main/docs
about: Browse our comprehensive documentation
.github/ISSUE_TEMPLATE/feature_request.yml
+65
View File
@@ -0,0 +1,65 @@
name: ✨ Feature Suggestion
description: Suggest a new feature or improvement for Open Notebook
title: "[Feature]: "
labels: ["enhancement", "needs-triage"]
body:
- type: markdown
attributes:
value: |
Thanks for taking the time to suggest a feature! Your ideas help make Open Notebook better for everyone.
- type: textarea
id: feature-description
attributes:
label: Feature Description
description: What feature would you like to see added or improved?
placeholder: "I would like to be able to..."
validations:
required: true
- type: textarea
id: why-helpful
attributes:
label: Why would this be helpful?
description: Explain how this feature would benefit you and other users
placeholder: "This would help because..."
validations:
required: true
- type: textarea
id: proposed-solution
attributes:
label: Proposed Solution (Optional)
description: If you have ideas on how to implement this feature, please share them
placeholder: "This could be implemented by..."
validations:
required: false
- type: textarea
id: additional-context
attributes:
label: Additional Context
description: Any other context, screenshots, or examples that might be helpful
placeholder: "For example, other tools do this by..."
validations:
required: false
- type: checkboxes
id: willing-to-contribute
attributes:
label: Contribution
description: Would you like to work on implementing this feature?
options:
- label: I am a developer and would like to work on implementing this feature (pending maintainer approval)
required: false
- type: markdown
attributes:
value: |
---
**Next Steps:**
1. A maintainer will review your feature request
2. If approved and you checked the box above, the issue will be assigned to you
3. Please wait for assignment before starting development
4. See our [Contributing Guide](https://github.com/lfnovo/open-notebook/blob/main/CONTRIBUTING.md) for more details
.github/ISSUE_TEMPLATE/installation_issue.yml
+148
View File
@@ -0,0 +1,148 @@
name: 🔧 Installation Issue
description: Report problems with installation, setup, or connectivity
title: "[Install]: "
labels: ["installation", "needs-triage"]
body:
- type: markdown
attributes:
value: |
## ⚠️ Before You Continue
**Please try these resources first:**
1. 🤖 **[Installation Assistant ChatGPT](https://chatgpt.com/g/g-68776e2765b48191bd1bae3f30212631-open-notebook-installation-assistant)** - Our AI assistant can help you troubleshoot most installation issues instantly!
2. 📚 **[Installation Guide](https://github.com/lfnovo/open-notebook/blob/main/docs/getting-started/installation.md)** - Comprehensive setup instructions
3. 🐋 **[Docker Deployment Guide](https://github.com/lfnovo/open-notebook/blob/main/docs/deployment/docker.md)** - Detailed Docker setup
4. 🦙 **Ollama Issues?** Read our [Ollama Guide](https://github.com/lfnovo/open-notebook/blob/main/docs/features/ollama.md) first
5. 💬 **[Discord Community](https://discord.gg/37XJPXfz2w)** - Get real-time help from the community
---
If you've tried the above and still need help, please fill out the form below with as much detail as possible.
- type: dropdown
id: installation-method
attributes:
label: Installation Method
description: How are you trying to install Open Notebook?
options:
- Docker (docker-compose - recommended)
- Docker (single container - v1-latest-single, deprecated)
- Local development (make start-all)
- Other (please specify below)
validations:
required: true
- type: textarea
id: issue-description
attributes:
label: What is the issue?
description: Describe the installation or setup problem you're experiencing
placeholder: |
Example: "I can't connect to the database" or "The container won't start" or "Getting 404 errors when accessing the UI"
validations:
required: true
- type: textarea
id: logs
attributes:
label: Logs
description: |
Please provide relevant logs. **This is very important for diagnosing issues!**
**How to get logs:**
- Docker single container: `docker logs open-notebook`
- Docker Compose: `docker compose logs -f`
- Specific service: `docker compose logs -f open_notebook`
placeholder: |
Paste your logs here. Include the full error message and stack trace if available.
render: shell
validations:
required: false
- type: textarea
id: docker-compose
attributes:
label: Docker Compose Configuration
description: |
If using Docker Compose, please paste your `docker-compose.yml` file here.
**⚠️ IMPORTANT: Redact any sensitive information (API keys, passwords, etc.)**
placeholder: |
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest-single
ports:
- "8502:8502"
- "5055:5055"
environment:
- OPENAI_API_KEY=sk-***REDACTED***
...
render: yaml
validations:
required: false
- type: textarea
id: env-file
attributes:
label: Environment File
description: |
If using an `.env` or `docker.env` file, please paste it here.
**⚠️ IMPORTANT: REDACT ALL API KEYS AND PASSWORDS!**
placeholder: |
SURREAL_URL=ws://surrealdb:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=***REDACTED***
OPENAI_API_KEY=sk-***REDACTED***
ANTHROPIC_API_KEY=sk-ant-***REDACTED***
render: shell
validations:
required: false
- type: textarea
id: system-info
attributes:
label: System Information
description: Tell us about your setup
placeholder: |
- Operating System: Ubuntu 22.04 / Windows 11 / macOS 14
- Docker version: `docker --version`
- Docker Compose version: `docker compose version`
- Architecture: amd64 / arm64 (Apple Silicon)
- Available disk space: `df -h`
- Available memory: `free -h` (Linux) or Activity Monitor (Mac)
validations:
required: false
- type: textarea
id: additional-context
attributes:
label: Additional Context
description: Any other information that might be helpful
placeholder: |
- Are you behind a corporate proxy or firewall?
- Are you using a VPN?
- Have you made any custom modifications?
- Did this work before and suddenly break?
validations:
required: false
- type: checkboxes
id: checklist
attributes:
label: Pre-submission Checklist
description: Please confirm you've tried these steps
options:
- label: I tried the [Installation Assistant ChatGPT](https://chatgpt.com/g/g-68776e2765b48191bd1bae3f30212631-open-notebook-installation-assistant)
required: false
- label: I read the relevant documentation ([Installation Guide](https://github.com/lfnovo/open-notebook/blob/main/docs/getting-started/installation.md) or [Ollama Guide](https://github.com/lfnovo/open-notebook/blob/main/docs/features/ollama.md))
required: false
- label: I searched existing issues to see if this was already reported
required: true
- label: I redacted all sensitive information (API keys, passwords, etc.)
required: true
.github/RELEASE_PROCESS.md
+174
View File
@@ -0,0 +1,174 @@
# Release Process
Open Notebook uses a flow-driven release process. Work moves from `ready`
issues into pull requests, pull requests merge to `main`, and maintainers cut a
version when the branch has enough validated change to ship.
This document covers both the **mechanics** (how to cut, build and publish) and
the **confidence process** (how we know a release is good before users get it).
It was redesigned during the v1.11.0 release ([ADR-005](../docs/7-DEVELOPMENT/decisions/ADR-005-release-confidence-process.md)).
## Release Model
- Patch releases ship backwards-compatible fixes.
- Minor releases ship backwards-compatible features and improvements.
- Major releases are planned with a milestone when they include breaking
changes or migrations that need user coordination.
- Use the `in-dev-build` label for changes available in development images and
`released` for shipped work. (The `released` label was recreated during
v1.12.0 — it had been dropped from the curated label taxonomy while this
document still required it. If a label this document references is missing,
recreate it rather than skipping the step.)
## Normal Flow
1. Triage issues into `ready` once the scope and design are clear.
2. Implement each change in a focused pull request linked to the approved issue.
3. Merge the pull request after review and required checks pass.
4. Let the development build publish the `v1-dev` image from `main`.
5. Cut a stable release when `main` has a coherent set of changes ready for
users — following the confidence process below.
## The Confidence Process
Releases keep getting bigger; ad-hoc verification does not scale. Before
cutting, run this sequence:
### 0. Changelog audit
Diff `git log <last-tag>..main` against the `[Unreleased]` section of the
CHANGELOG. Every merged PR must be represented (entries reference the issue
number when one exists, the PR number otherwise). The changelog is the input
for both the test plan and the release notes — close the gaps first, via PR.
### 1. Risk-based test matrix
Build a matrix from the actual release diff: each change → what it can break
and for whom → which bucket tests it. Pay special attention to
**"does the protection break legitimate use?"** for security changes (e.g. an
SSRF guard vs. self-hosted Ollama on localhost) and to anything a reverse
proxy, an upgrade, or a big upload would exercise.
Buckets:
- **A — automated, high confidence, run now**: full backend suite, frontend
lint/tests/production build, the smoke-e2e agent (full API happy path + UI
verification), targeted regression probes for the release's specific risks,
dependency audit.
- **B — automatable with investment**: decide per item whether to build the
muscle now (it compounds: the image gate below started as a bucket-B item)
or verify manually this once.
- **C — needs the release owner**: real provider credentials, real TTS podcast
generation, visual/UX judgment, and the final check of the pushed image.
### 2. The image gate — test the artifact, not the repo
A green suite on `main` is not a working image. Run:
```bash
make docker-build-local # builds <version> + local tags
make release-test TAG=<new> OLD_TAG=<previous>
```
This runs two scenarios against real containers (`scripts/release-test/`):
- **Fresh install**: empty DB → migrations on boot → in-image worker processes
a source → API/frontend/nginx-proxied checks.
- **Upgrade**: boot the *published* previous image, seed data, swap to the new
image on the same volume → migrations apply, data survives.
Caveat: `docker-build-local` tags with the current `pyproject.toml` version —
`docker pull` the genuine previous tag before the upgrade test so you are not
comparing the new build against itself.
### 3. Fix loop with a re-test policy
Findings become focused PRs through the normal review flow. After each merge:
the cheap suite always re-runs; smoke/image gates re-run only if the fix
touches what they cover; manual verification is not repeated unless the fix
touches what was manually verified. Pre-existing bugs found along the way that
are not release regressions become backlog issues instead of scope creep.
## Cutting A Stable Release
1. Confirm `main` is green and the confidence process above has run.
2. Open the **cut PR**: bump `pyproject.toml`, date the `[Unreleased]` section
as `[<version>] - <date>`.
3. After merge: `make tag`.
4. Build and push version images **via CI** (it holds the registry
credentials): trigger the *Build and Release* workflow with
`push_latest=false`. Local `make docker-push` also works but requires
`docker login` on both registries.
5. **Verify the pushed image** (bucket C, final gate): run it locally with
`make release-stack TAG=<version> [DUMP=<dev-data-dump>]` — a browsable,
isolated stack, optionally with a copy of real data — and walk the core
flows in the browser.
6. Publish the GitHub release. A non-prerelease publication triggers the
workflow again and pushes the `v1-latest` tags automatically.
7. Verify the `v1-latest` manifests on Docker Hub and GHCR (both arches, both
variants), and mark shipped issues with `released`.
## Communication
Release notes follow this structure (see v1.11.0 as the reference):
1. One-line verdict + upgrade recommendation.
2. Sections: Security, Features, Performance, Notable fixes.
3. **Behavior changes for self-hosters** — anything that can require a config
tweak on upgrade gets an explicit callout.
4. **Thanks** — credit every contributor by handle with what they shipped
(collect via `git log <last-tag>..<tag>` + `gh pr view` for handles), plus
the issue reporters collectively. Never skip this section.
Announce on Discord after `v1-latest` is live.
## Retro
Close every release by asking: what should improve in this process? Apply the
accepted improvements immediately — update this document, the scripts under
`scripts/release-test/`, and the decision log while the context is fresh.
## Docker Image Publishing (reference)
| Command | What it does | Updates latest? |
|---------|--------------|-----------------|
| `make docker-build-local` | Build for current platform only (tags `<version>` + `local`) | No registry push |
| CI *Build and Release* (`push_latest=false`) | Push version tags via CI credentials | ❌ No |
| GitHub release published (non-prerelease) | CI pushes version + `v1-latest` | ✅ Yes |
| `make docker-push` / `docker-push-latest` | Local equivalents (need `docker login`) | ❌ / ✅ |
| `make tag` | Create and push a git tag matching `pyproject.toml` | — |
- **Platforms:** `linux/amd64`, `linux/arm64`
- **Registries:** Docker Hub + GitHub Container Registry
- **Image variants:** regular + single-container (`-single`). Both are built
from the same `Dockerfile`: regular is the default/`runtime` target, single
is `--target single`
- **Version source:** `pyproject.toml`
- Build issues: `docker builder prune`, then `make docker-buildx-reset`
## Known Gotchas
- **RC stack on non-default ports needs `API_URL`** or the browser talks to
`host:5055` — on a dev machine that is the development API (data crossover).
`rc-stack.sh` sets it; remember this for any custom setup.
- **Containerized app + host services**: credentials pointing at local
services (Ollama, LM Studio) need `http://host.docker.internal:<port>`.
- **SurrealDB import**: `OVERWRITE` goes after the type keyword
(`DEFINE FIELD OVERWRITE …`), and the exporter can leak a log line into the
dump — `rc-stack.sh` handles both.
- **Multiple local SurrealDB instances**: check which one the dev `.env`
actually points at (`SURREAL_URL`) before exporting data.
- **Dev-machine ports may belong to other projects**: check who owns
3000/5055/8000 (`lsof -nP -iTCP:<port> -sTCP:LISTEN` + the process cwd)
before starting or killing anything. The frontend runs fine on an alternate
port for smoke testing (`PORT=3001 npm run dev`) — pass the URL to the
smoke agent.
- **Manual error-path checklist items must be validated against the code
first**: some "missing configuration" scenarios are deliberate fallbacks,
not errors (e.g. transformation and tools defaults fall back to the chat
default). Confirm the expected behavior in the provisioning code before
putting "should show an error" on the bucket-C checklist.
- **The test suite runs against the live dev database** when a developer
`.env` is loaded. During bucket A, snapshot record counts per table before
and after the suite (e.g. credentials count) — a diff means a test is
leaking writes (this caught 48 leaked `Test` credentials in v1.12.0).
.github/pull_request_template.md
+109
View File
@@ -0,0 +1,109 @@
## Description
<!-- Provide a clear and concise description of what this PR does -->
## Related Issue
<!-- Non-trivial PRs (features, architecture changes) must link an approved issue.
Small obvious fixes (typo, docs, tiny bug) don't need one — write "N/A (small fix)" below.
Sizeable change without an issue? Mark this PR as draft and create the issue first. -->
Fixes #<!-- issue number, or "N/A (small fix)" -->
## Type of Change
<!-- Mark the relevant option with an "x" -->
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Documentation update
- [ ] Code refactoring (no functional changes)
- [ ] Performance improvement
- [ ] Test coverage improvement
## How Has This Been Tested?
<!-- Describe the tests you ran and/or how you verified your changes work -->
- [ ] Tested locally with Docker
- [ ] Tested locally with development setup
- [ ] Added new unit tests
- [ ] Existing tests pass (`uv run pytest`)
- [ ] Manual testing performed (describe below)
**Test Details:**
<!-- Describe your testing approach -->
## Design Alignment
<!-- This section helps ensure your PR aligns with our project vision -->
**Which design principles does this PR support?** (See [VISION.md](https://github.com/lfnovo/open-notebook/blob/main/VISION.md))
- [ ] Privacy First
- [ ] Simplicity Over Features
- [ ] API-First Architecture
- [ ] Multi-Provider Flexibility
- [ ] Extensibility Through Standards
- [ ] Async-First for Performance
**Explanation:**
<!-- Brief explanation of how your changes align with these principles -->
## Checklist
<!-- Mark completed items with an "x" -->
### Code Quality
- [ ] My code follows PEP 8 style guidelines (Python)
- [ ] My code follows TypeScript best practices (Frontend)
- [ ] I have added type hints to my code (Python)
- [ ] I have added JSDoc comments where appropriate (TypeScript)
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] My changes generate no new warnings or errors
### Testing
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I ran linting: `make ruff` or `ruff check . --fix`
- [ ] I ran type checking: `make lint` or `uv run python -m mypy .`
### Documentation
- [ ] I have updated the relevant documentation in `/docs` (if applicable)
- [ ] I have added/updated docstrings for new/modified functions
- [ ] I have updated the API documentation (if API changes were made)
- [ ] I have added comments to complex logic
### Database Changes
- [ ] I have created migration scripts for any database schema changes (in `/migrations`)
- [ ] Migration includes both up and down scripts
- [ ] Migration has been tested locally
### Breaking Changes
- [ ] This PR includes breaking changes
- [ ] I have documented the migration path for users
- [ ] I have updated MIGRATION.md (if applicable)
## Screenshots (if applicable)
<!-- Add screenshots for UI changes -->
## Additional Context
<!-- Add any other context about the PR here -->
## Pre-Submission Verification
Before submitting, please verify:
- [ ] I have read [CONTRIBUTING.md](https://github.com/lfnovo/open-notebook/blob/main/docs/7-DEVELOPMENT/contributing.md)
- [ ] I have read [VISION.md](https://github.com/lfnovo/open-notebook/blob/main/VISION.md)
- [ ] This PR addresses an approved issue assigned to me, **or** it's a small obvious fix (typo, docs, tiny bug) that doesn't need one — for anything bigger without an issue, mark this PR as draft and open the issue (triage takes 1–2 days)
- [ ] I have not included unrelated changes in this PR
- [ ] My PR title follows conventional commits format (e.g., "feat: add user authentication")
---
**Thank you for contributing to Open Notebook!** 🎉
.github/workflows/build-and-release.yml
+300
View File
@@ -0,0 +1,300 @@
name: Build and Release
on:
workflow_dispatch:
inputs:
push_latest:
description: 'Also push v1-latest tags'
required: true
default: false
type: boolean
release:
types: [published]
permissions:
contents: read
packages: write
env:
GHCR_IMAGE: ghcr.io/lfnovo/open-notebook
DOCKERHUB_IMAGE: lfnovo/open_notebook
jobs:
extract-version:
runs-on: ubuntu-latest
outputs:
version: ${{ steps.version.outputs.version }}
has_dockerhub_secrets: ${{ steps.check.outputs.has_dockerhub_secrets }}
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Extract version from pyproject.toml
id: version
run: |
VERSION=$(grep -m1 '^version = ' pyproject.toml | cut -d'"' -f2)
echo "version=$VERSION" >> $GITHUB_OUTPUT
echo "Extracted version: $VERSION"
- name: Check for Docker Hub credentials
id: check
env:
SECRET_DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
SECRET_DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
run: |
if [[ -n ""$SECRET_DOCKER_USERNAME"" && -n ""$SECRET_DOCKER_PASSWORD"" ]]; then
echo "has_dockerhub_secrets=true" >> $GITHUB_OUTPUT
echo "Docker Hub credentials available"
else
echo "has_dockerhub_secrets=false" >> $GITHUB_OUTPUT
echo "Docker Hub credentials not available - will only push to GHCR"
fi
build-regular:
needs: extract-version
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Free up disk space
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /usr/local/lib/android
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache/CodeQL
sudo docker image prune --all --force
df -h
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: Login to GitHub Container Registry
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Login to Docker Hub
if: needs.extract-version.outputs.has_dockerhub_secrets == 'true'
uses: docker/login-action@v4
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Cache Docker layers
uses: actions/cache@v5
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-regular-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-regular-
- name: Prepare Docker tags for regular build
id: tags-regular
env:
ENV_GHCR_IMAGE: ${{ env.GHCR_IMAGE }}
GITHUB_EVENT_INPUTS_PUSH_LATEST: ${{ github.event.inputs.push_latest }}
GITHUB_EVENT_NAME: ${{ github.event_name }}
GITHUB_EVENT_RELEASE_PRERELEASE: ${{ github.event.release.prerelease }}
ENV_DOCKERHUB_IMAGE: ${{ env.DOCKERHUB_IMAGE }}
run: |
TAGS=""$ENV_GHCR_IMAGE":${{ needs.extract-version.outputs.version }}"
# Determine if we should push latest tags
PUSH_LATEST=""$GITHUB_EVENT_INPUTS_PUSH_LATEST""
if [[ -z "$PUSH_LATEST" ]]; then
PUSH_LATEST="false"
fi
# Add GHCR latest tag if requested or for non-prerelease releases
if [[ "$PUSH_LATEST" == "true" ]] || [[ ""$GITHUB_EVENT_NAME"" == "release" && ""$GITHUB_EVENT_RELEASE_PRERELEASE"" != "true" ]]; then
TAGS="${TAGS},"$ENV_GHCR_IMAGE":v1-latest"
fi
# Add Docker Hub tags if credentials available
if [[ "${{ needs.extract-version.outputs.has_dockerhub_secrets }}" == "true" ]]; then
TAGS="${TAGS},"$ENV_DOCKERHUB_IMAGE":${{ needs.extract-version.outputs.version }}"
if [[ "$PUSH_LATEST" == "true" ]] || [[ ""$GITHUB_EVENT_NAME"" == "release" && ""$GITHUB_EVENT_RELEASE_PRERELEASE"" != "true" ]]; then
TAGS="${TAGS},"$ENV_DOCKERHUB_IMAGE":v1-latest"
fi
fi
echo "tags=${TAGS}" >> $GITHUB_OUTPUT
echo "Generated tags: ${TAGS}"
- name: Build and push regular image
uses: docker/build-push-action@v7
with:
context: .
file: ./Dockerfile
target: runtime
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.tags-regular.outputs.tags }}
cache-from: type=local,src=/tmp/.buildx-cache
cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=max
- name: Move cache
run: |
rm -rf /tmp/.buildx-cache
mv /tmp/.buildx-cache-new /tmp/.buildx-cache
build-single:
needs: extract-version
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Free up disk space
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /usr/local/lib/android
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache/CodeQL
sudo docker image prune --all --force
df -h
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: Login to GitHub Container Registry
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Login to Docker Hub
if: needs.extract-version.outputs.has_dockerhub_secrets == 'true'
uses: docker/login-action@v4
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Cache Docker layers
uses: actions/cache@v5
with:
path: /tmp/.buildx-cache-single
key: ${{ runner.os }}-buildx-single-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-single-
- name: Prepare Docker tags for single build
id: tags-single
env:
ENV_GHCR_IMAGE: ${{ env.GHCR_IMAGE }}
GITHUB_EVENT_INPUTS_PUSH_LATEST: ${{ github.event.inputs.push_latest }}
GITHUB_EVENT_NAME: ${{ github.event_name }}
GITHUB_EVENT_RELEASE_PRERELEASE: ${{ github.event.release.prerelease }}
ENV_DOCKERHUB_IMAGE: ${{ env.DOCKERHUB_IMAGE }}
run: |
TAGS=""$ENV_GHCR_IMAGE":${{ needs.extract-version.outputs.version }}-single"
# Determine if we should push latest tags
PUSH_LATEST=""$GITHUB_EVENT_INPUTS_PUSH_LATEST""
if [[ -z "$PUSH_LATEST" ]]; then
PUSH_LATEST="false"
fi
# Add GHCR latest tag if requested or for non-prerelease releases
if [[ "$PUSH_LATEST" == "true" ]] || [[ ""$GITHUB_EVENT_NAME"" == "release" && ""$GITHUB_EVENT_RELEASE_PRERELEASE"" != "true" ]]; then
TAGS="${TAGS},"$ENV_GHCR_IMAGE":v1-latest-single"
fi
# Add Docker Hub tags if credentials available
if [[ "${{ needs.extract-version.outputs.has_dockerhub_secrets }}" == "true" ]]; then
TAGS="${TAGS},"$ENV_DOCKERHUB_IMAGE":${{ needs.extract-version.outputs.version }}-single"
if [[ "$PUSH_LATEST" == "true" ]] || [[ ""$GITHUB_EVENT_NAME"" == "release" && ""$GITHUB_EVENT_RELEASE_PRERELEASE"" != "true" ]]; then
TAGS="${TAGS},"$ENV_DOCKERHUB_IMAGE":v1-latest-single"
fi
fi
echo "tags=${TAGS}" >> $GITHUB_OUTPUT
echo "Generated tags: ${TAGS}"
- name: Build and push single-container image
uses: docker/build-push-action@v7
with:
context: .
file: ./Dockerfile
target: single
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.tags-single.outputs.tags }}
cache-from: type=local,src=/tmp/.buildx-cache-single
cache-to: type=local,dest=/tmp/.buildx-cache-single-new,mode=max
- name: Move cache
run: |
rm -rf /tmp/.buildx-cache-single
mv /tmp/.buildx-cache-single-new /tmp/.buildx-cache-single
summary:
needs: [extract-version, build-regular, build-single]
runs-on: ubuntu-latest
if: always()
steps:
- name: Build Summary
env:
GITHUB_EVENT_INPUTS_PUSH_LATEST_____FALSE_: ${{ github.event.inputs.push_latest || 'false' }}
ENV_GHCR_IMAGE: ${{ env.GHCR_IMAGE }}
ENV_DOCKERHUB_IMAGE: ${{ env.DOCKERHUB_IMAGE }}
GITHUB_EVENT_INPUTS_PUSH_LATEST: ${{ github.event.inputs.push_latest }}
run: |
echo "## Build Summary" >> $GITHUB_STEP_SUMMARY
echo "**Version:** ${{ needs.extract-version.outputs.version }}" >> $GITHUB_STEP_SUMMARY
echo "**Push v1-Latest:** "$GITHUB_EVENT_INPUTS_PUSH_LATEST_____FALSE_"" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "### Registries:" >> $GITHUB_STEP_SUMMARY
echo "✅ **GHCR:** \`"$ENV_GHCR_IMAGE"\`" >> $GITHUB_STEP_SUMMARY
if [[ "${{ needs.extract-version.outputs.has_dockerhub_secrets }}" == "true" ]]; then
echo "✅ **Docker Hub:** \`"$ENV_DOCKERHUB_IMAGE"\`" >> $GITHUB_STEP_SUMMARY
else
echo "⏭️ **Docker Hub:** Skipped (credentials not configured)" >> $GITHUB_STEP_SUMMARY
fi
echo "" >> $GITHUB_STEP_SUMMARY
echo "### Images Built:" >> $GITHUB_STEP_SUMMARY
if [[ "${{ needs.build-regular.result }}" == "success" ]]; then
echo "✅ **Regular (GHCR):** \`"$ENV_GHCR_IMAGE":${{ needs.extract-version.outputs.version }}\`" >> $GITHUB_STEP_SUMMARY
if [[ ""$GITHUB_EVENT_INPUTS_PUSH_LATEST"" == "true" ]]; then
echo "✅ **Regular v1-Latest (GHCR):** \`"$ENV_GHCR_IMAGE":v1-latest\`" >> $GITHUB_STEP_SUMMARY
fi
if [[ "${{ needs.extract-version.outputs.has_dockerhub_secrets }}" == "true" ]]; then
echo "✅ **Regular (Docker Hub):** \`"$ENV_DOCKERHUB_IMAGE":${{ needs.extract-version.outputs.version }}\`" >> $GITHUB_STEP_SUMMARY
if [[ ""$GITHUB_EVENT_INPUTS_PUSH_LATEST"" == "true" ]]; then
echo "✅ **Regular v1-Latest (Docker Hub):** \`"$ENV_DOCKERHUB_IMAGE":v1-latest\`" >> $GITHUB_STEP_SUMMARY
fi
fi
elif [[ "${{ needs.build-regular.result }}" == "skipped" ]]; then
echo "⏭️ **Regular:** Skipped" >> $GITHUB_STEP_SUMMARY
else
echo "❌ **Regular:** Failed" >> $GITHUB_STEP_SUMMARY
fi
if [[ "${{ needs.build-single.result }}" == "success" ]]; then
echo "✅ **Single (GHCR):** \`"$ENV_GHCR_IMAGE":${{ needs.extract-version.outputs.version }}-single\`" >> $GITHUB_STEP_SUMMARY
if [[ ""$GITHUB_EVENT_INPUTS_PUSH_LATEST"" == "true" ]]; then
echo "✅ **Single v1-Latest (GHCR):** \`"$ENV_GHCR_IMAGE":v1-latest-single\`" >> $GITHUB_STEP_SUMMARY
fi
if [[ "${{ needs.extract-version.outputs.has_dockerhub_secrets }}" == "true" ]]; then
echo "✅ **Single (Docker Hub):** \`"$ENV_DOCKERHUB_IMAGE":${{ needs.extract-version.outputs.version }}-single\`" >> $GITHUB_STEP_SUMMARY
if [[ ""$GITHUB_EVENT_INPUTS_PUSH_LATEST"" == "true" ]]; then
echo "✅ **Single v1-Latest (Docker Hub):** \`"$ENV_DOCKERHUB_IMAGE":v1-latest-single\`" >> $GITHUB_STEP_SUMMARY
fi
fi
elif [[ "${{ needs.build-single.result }}" == "skipped" ]]; then
echo "⏭️ **Single:** Skipped" >> $GITHUB_STEP_SUMMARY
else
echo "❌ **Single:** Failed" >> $GITHUB_STEP_SUMMARY
fi
echo "" >> $GITHUB_STEP_SUMMARY
echo "### Platforms:" >> $GITHUB_STEP_SUMMARY
echo "- linux/amd64" >> $GITHUB_STEP_SUMMARY
echo "- linux/arm64" >> $GITHUB_STEP_SUMMARY
.github/workflows/build-dev.yml
+275
View File
@@ -0,0 +1,275 @@
name: Development Build
on:
pull_request:
branches: [ main ]
push:
branches: [ main ]
paths-ignore:
- '**.md'
- 'docs/**'
- 'notebooks/**'
- '.github/workflows/claude*.yml'
workflow_dispatch:
inputs:
platform:
description: 'Platform to build'
required: true
default: 'linux/amd64'
type: choice
options:
- linux/amd64
- linux/arm64
- linux/amd64,linux/arm64
permissions:
contents: read
packages: write
env:
GHCR_IMAGE: ghcr.io/lfnovo/open-notebook
DOCKERHUB_IMAGE: lfnovo/open_notebook
jobs:
extract-version:
runs-on: ubuntu-latest
outputs:
version: ${{ steps.version.outputs.version }}
has_dockerhub_secrets: ${{ steps.check.outputs.has_dockerhub_secrets }}
is_push_to_main: ${{ steps.check.outputs.is_push_to_main }}
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Extract version from pyproject.toml
id: version
run: |
VERSION=$(grep -m1 '^version = ' pyproject.toml | cut -d'"' -f2)
echo "version=$VERSION" >> $GITHUB_OUTPUT
echo "Extracted version: $VERSION"
- name: Check environment
id: check
env:
SECRET_DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
SECRET_DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
run: |
# Check for Docker Hub credentials
if [[ -n "$SECRET_DOCKER_USERNAME" && -n "$SECRET_DOCKER_PASSWORD" ]]; then
echo "has_dockerhub_secrets=true" >> $GITHUB_OUTPUT
echo "Docker Hub credentials available"
else
echo "has_dockerhub_secrets=false" >> $GITHUB_OUTPUT
echo "Docker Hub credentials not available"
fi
# Check if this is a push to main (not a PR)
if [[ "${{ github.event_name }}" == "push" && "${{ github.ref }}" == "refs/heads/main" ]]; then
echo "is_push_to_main=true" >> $GITHUB_OUTPUT
echo "This is a push to main - will publish v1-dev tags"
else
echo "is_push_to_main=false" >> $GITHUB_OUTPUT
echo "This is a PR or manual run - test build only"
fi
build-regular:
needs: extract-version
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Free up disk space
if: needs.extract-version.outputs.is_push_to_main == 'true'
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /usr/local/lib/android
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache/CodeQL
sudo docker image prune --all --force
df -h
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: Login to GitHub Container Registry
if: needs.extract-version.outputs.is_push_to_main == 'true'
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Login to Docker Hub
if: needs.extract-version.outputs.is_push_to_main == 'true' && needs.extract-version.outputs.has_dockerhub_secrets == 'true'
uses: docker/login-action@v4
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Cache Docker layers
uses: actions/cache@v5
with:
path: /tmp/.buildx-cache-dev
key: ${{ runner.os }}-buildx-dev-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-dev-
- name: Prepare Docker tags
id: tags
run: |
if [[ "${{ needs.extract-version.outputs.is_push_to_main }}" == "true" ]]; then
# Push to main: build and push v1-dev tags
TAGS="${{ env.GHCR_IMAGE }}:v1-dev"
if [[ "${{ needs.extract-version.outputs.has_dockerhub_secrets }}" == "true" ]]; then
TAGS="${TAGS},${{ env.DOCKERHUB_IMAGE }}:v1-dev"
fi
echo "tags=${TAGS}" >> $GITHUB_OUTPUT
echo "push=true" >> $GITHUB_OUTPUT
echo "platforms=linux/amd64,linux/arm64" >> $GITHUB_OUTPUT
else
# PR or manual: test build only
echo "tags=${{ env.DOCKERHUB_IMAGE }}:${{ needs.extract-version.outputs.version }}-dev" >> $GITHUB_OUTPUT
echo "push=false" >> $GITHUB_OUTPUT
echo "platforms=${{ github.event.inputs.platform || 'linux/amd64' }}" >> $GITHUB_OUTPUT
fi
- name: Build and push regular image
uses: docker/build-push-action@v7
with:
context: .
file: ./Dockerfile
target: runtime
platforms: ${{ steps.tags.outputs.platforms }}
push: ${{ steps.tags.outputs.push }}
tags: ${{ steps.tags.outputs.tags }}
cache-from: type=local,src=/tmp/.buildx-cache-dev
cache-to: type=local,dest=/tmp/.buildx-cache-dev-new,mode=max
- name: Move cache
run: |
rm -rf /tmp/.buildx-cache-dev
mv /tmp/.buildx-cache-dev-new /tmp/.buildx-cache-dev
build-single:
needs: extract-version
# Only build single image on push to main
if: needs.extract-version.outputs.is_push_to_main == 'true'
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Free up disk space
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /usr/local/lib/android
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache/CodeQL
sudo docker image prune --all --force
df -h
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: Login to GitHub Container Registry
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Login to Docker Hub
if: needs.extract-version.outputs.has_dockerhub_secrets == 'true'
uses: docker/login-action@v4
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Cache Docker layers
uses: actions/cache@v5
with:
path: /tmp/.buildx-cache-dev-single
key: ${{ runner.os }}-buildx-dev-single-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-dev-single-
- name: Prepare Docker tags
id: tags
run: |
TAGS="${{ env.GHCR_IMAGE }}:v1-dev-single"
if [[ "${{ needs.extract-version.outputs.has_dockerhub_secrets }}" == "true" ]]; then
TAGS="${TAGS},${{ env.DOCKERHUB_IMAGE }}:v1-dev-single"
fi
echo "tags=${TAGS}" >> $GITHUB_OUTPUT
- name: Build and push single-container image
uses: docker/build-push-action@v7
with:
context: .
file: ./Dockerfile
target: single
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.tags.outputs.tags }}
cache-from: type=local,src=/tmp/.buildx-cache-dev-single
cache-to: type=local,dest=/tmp/.buildx-cache-dev-single-new,mode=max
- name: Move cache
run: |
rm -rf /tmp/.buildx-cache-dev-single
mv /tmp/.buildx-cache-dev-single-new /tmp/.buildx-cache-dev-single
summary:
needs: [extract-version, build-regular, build-single]
runs-on: ubuntu-latest
if: always()
steps:
- name: Development Build Summary
run: |
echo "## Development Build Summary" >> $GITHUB_STEP_SUMMARY
echo "**Version:** ${{ needs.extract-version.outputs.version }}" >> $GITHUB_STEP_SUMMARY
echo "**Event:** ${{ github.event_name }}" >> $GITHUB_STEP_SUMMARY
echo "**Push to Main:** ${{ needs.extract-version.outputs.is_push_to_main }}" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
if [[ "${{ needs.extract-version.outputs.is_push_to_main }}" == "true" ]]; then
echo "### Published Tags:" >> $GITHUB_STEP_SUMMARY
if [[ "${{ needs.build-regular.result }}" == "success" ]]; then
echo "✅ **Regular:** \`${{ env.GHCR_IMAGE }}:v1-dev\`" >> $GITHUB_STEP_SUMMARY
if [[ "${{ needs.extract-version.outputs.has_dockerhub_secrets }}" == "true" ]]; then
echo "✅ **Regular (Docker Hub):** \`${{ env.DOCKERHUB_IMAGE }}:v1-dev\`" >> $GITHUB_STEP_SUMMARY
fi
else
echo "❌ **Regular:** Build failed" >> $GITHUB_STEP_SUMMARY
fi
if [[ "${{ needs.build-single.result }}" == "success" ]]; then
echo "✅ **Single:** \`${{ env.GHCR_IMAGE }}:v1-dev-single\`" >> $GITHUB_STEP_SUMMARY
if [[ "${{ needs.extract-version.outputs.has_dockerhub_secrets }}" == "true" ]]; then
echo "✅ **Single (Docker Hub):** \`${{ env.DOCKERHUB_IMAGE }}:v1-dev-single\`" >> $GITHUB_STEP_SUMMARY
fi
elif [[ "${{ needs.build-single.result }}" == "skipped" ]]; then
echo "⏭️ **Single:** Skipped" >> $GITHUB_STEP_SUMMARY
else
echo "❌ **Single:** Build failed" >> $GITHUB_STEP_SUMMARY
fi
echo "" >> $GITHUB_STEP_SUMMARY
echo "### Platforms:" >> $GITHUB_STEP_SUMMARY
echo "- linux/amd64" >> $GITHUB_STEP_SUMMARY
echo "- linux/arm64" >> $GITHUB_STEP_SUMMARY
else
echo "### Test Build Results:" >> $GITHUB_STEP_SUMMARY
if [[ "${{ needs.build-regular.result }}" == "success" ]]; then
echo "✅ **Dockerfile:** Build successful" >> $GITHUB_STEP_SUMMARY
else
echo "❌ **Dockerfile:** Build failed" >> $GITHUB_STEP_SUMMARY
fi
echo "" >> $GITHUB_STEP_SUMMARY
echo "### Notes:" >> $GITHUB_STEP_SUMMARY
echo "- This is a test build (no images pushed to registry)" >> $GITHUB_STEP_SUMMARY
echo "- Merge to main to publish \`v1-dev\` tags" >> $GITHUB_STEP_SUMMARY
echo "- For stable releases, use the 'Build and Release' workflow" >> $GITHUB_STEP_SUMMARY
fi
.github/workflows/docs-links.yml
+16
View File
@@ -0,0 +1,16 @@
name: Docs Link Check
on:
pull_request:
paths:
- "**/*.md"
- "scripts/check_md_links.py"
- ".github/workflows/docs-links.yml"
jobs:
check-links:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Check relative markdown links
run: python3 scripts/check_md_links.py
.github/workflows/test.yml
+159
View File
@@ -0,0 +1,159 @@
name: Tests
on:
pull_request:
branches: [main]
push:
branches: [main]
paths-ignore:
- '**.md'
- 'docs/**'
- '.github/workflows/claude*.yml'
permissions:
contents: read
jobs:
backend:
name: Backend Tests
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Set up uv
uses: astral-sh/setup-uv@v8.1.0
with:
enable-cache: true
- name: Set up Python
run: uv python install
- name: Install dependencies
run: uv sync
- name: Run tests with coverage
run: uv run pytest tests/ -v --cov=open_notebook --cov=api --cov-report=term-missing --cov-report=xml
- name: Upload coverage artifact
uses: actions/upload-artifact@v4
with:
name: backend-coverage
path: coverage.xml
backend-lint:
name: Backend Lint
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Set up uv
uses: astral-sh/setup-uv@v8.1.0
with:
enable-cache: true
- name: Set up Python
run: uv python install
- name: Install dependencies
run: uv sync
- name: Run ruff
run: uv run ruff check .
backend-typecheck:
name: Backend Typecheck
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Set up uv
uses: astral-sh/setup-uv@v8.1.0
with:
enable-cache: true
- name: Set up Python
run: uv python install
- name: Install dependencies
run: uv sync
- name: Run mypy
run: uv run python -m mypy .
frontend:
name: Frontend Tests
runs-on: ubuntu-latest
defaults:
run:
working-directory: frontend
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Set up Node.js
uses: actions/setup-node@v6
with:
node-version: 22
cache: npm
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
run: npm ci
- name: Run tests with coverage
run: npm run test:coverage
- name: Upload coverage artifact
uses: actions/upload-artifact@v4
with:
name: frontend-coverage
path: frontend/coverage/
frontend-lint:
name: Frontend Lint
runs-on: ubuntu-latest
defaults:
run:
working-directory: frontend
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Set up Node.js
uses: actions/setup-node@v6
with:
node-version: 22
cache: npm
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
run: npm ci
- name: Run ESLint
run: npm run lint
frontend-build:
name: Frontend Build
runs-on: ubuntu-latest
defaults:
run:
working-directory: frontend
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Set up Node.js
uses: actions/setup-node@v6
with:
node-version: 22
cache: npm
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
run: npm ci
- name: Build
run: npm run build
.gitignore
+153
View File
@@ -0,0 +1,153 @@
.env
prompts/patterns/user/
/notebooks/
data/
.uploads/
sqlite-db/
surreal-data/
docker.env
notebook_data/
# Python-specific
*.py[cod]
__pycache__/
*.so
todo.md
temp/
google-credentials.json
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
/lib/
/lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
# PyInstaller
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# PyCharm
.idea/
# VS Code
.vscode/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
# pytype static type analyzer
.pytype/
# Cython debug symbols
cython_debug/
# macOS
.DS_Store
# Windows
Thumbs.db
ehthumbs.db
desktop.ini
# Linux
*~
# Log files
*.log
# Database files
*.db
*.sqlite3
.quarentena
claude-logs/
.claude/sessions
**/claude-logs
docs/custom_gpt
doc_exports/
specs/
.claude
.sisyphus
.playwright-mcp/
*.local.yml
**/*.local.md
.harness/
.mcp.json
# Local Docker Compose override (auto-merged by `docker compose up`).
# Use it for host-specific tweaks like re-exposing the SurrealDB port;
# see docker-compose.override.yml.example.
docker-compose.override.yml
# Local QA/debug artifacts
screenshot-*.png
history.txt
.python-version
+1
View File
@@ -0,0 +1 @@
3.12
.worktreeinclude
+5
View File
@@ -0,0 +1,5 @@
.env
.env.local
.env.*
**/.claude/settings.local.json
CLAUDE.local.md
AGENTS.md
+45
View File
@@ -0,0 +1,45 @@
# Open Notebook — Agent Rules
Open Notebook is an open-source, privacy-focused alternative to Google's Notebook LM: an AI-powered research assistant with multi-provider AI support, fully self-hostable.
This file holds the project-wide rules every coding session needs. Component rules: [open_notebook/AGENTS.md](open_notebook/AGENTS.md) (backend — also covers `api/`, `commands/`, `prompts/`) and [frontend/AGENTS.md](frontend/AGENTS.md). Knowledge lives in the docs (see [Where to look](#where-to-look)) — read it on demand instead of guessing.
## Stack, ports, startup order
Three tiers: Next.js frontend (3000) → FastAPI (5055) → SurrealDB (8000).
Start in this order — each tier depends on the one below:
1. `make database` — SurrealDB (API fails without it)
2. `make api` — FastAPI; **schema migrations run automatically on startup** (check logs)
3. `make worker-start` — surreal-commands worker. **Required**: podcasts, embeddings and source processing are async jobs that silently queue forever without it
4. `make frontend` — UI (depends on the API for all data)
Or all at once: `make start-all` (status: `make status`, stop: `make stop-all`).
## Commands
- Tests: `uv run pytest tests/`
- Python lint/typecheck: `ruff check . --fix` · `uv run python -m mypy .`
- Frontend (inside `frontend/`): `npm run lint` · `npm run test` · `npm run build`
- Docker release: `make docker-release` (see `.github/RELEASE_PROCESS.md`)
## Hard rules
- **Async-first**: every DB query, graph invocation and AI call is `await`-ed. No sync DB access.
- **Never commit secrets.** Credentials are encrypted at rest and require `OPEN_NOTEBOOK_ENCRYPTION_KEY` to be set.
- CORS is wide-open and auth is a simple password middleware — **dev defaults, not production hardening**. Don't build features that assume otherwise.
- Product direction questions (does this feature fit?) → [VISION.md](VISION.md). Past decisions ("why is it like this?") → [docs/7-DEVELOPMENT/decisions/](docs/7-DEVELOPMENT/decisions/). Structural decisions made while coding should produce a new decision record there.
## Where to look
| Need | Location |
|---|---|
| Architecture (3 tiers, workflows, data model) | [docs/7-DEVELOPMENT/architecture.md](docs/7-DEVELOPMENT/architecture.md) |
| Step-by-step recipes (add endpoint, migration, i18n…) | [docs/7-DEVELOPMENT/change-playbooks.md](docs/7-DEVELOPMENT/change-playbooks.md) |
| Dev environment setup | [docs/7-DEVELOPMENT/development-setup.md](docs/7-DEVELOPMENT/development-setup.md) |
| Code standards & testing | [docs/7-DEVELOPMENT/code-standards.md](docs/7-DEVELOPMENT/code-standards.md) · [testing.md](docs/7-DEVELOPMENT/testing.md) |
| Product identity & current posture | [VISION.md](VISION.md) |
| Decision log (ADRs/PDRs) | [docs/7-DEVELOPMENT/decisions/](docs/7-DEVELOPMENT/decisions/) |
| Contribution process (issue-first, PRs) | [docs/7-DEVELOPMENT/contributing.md](docs/7-DEVELOPMENT/contributing.md) |
| User/operator docs (install, configure, troubleshoot) | [docs/](docs/index.md) |
CHANGELOG.md
+540
View File
@@ -0,0 +1,540 @@
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Added
- **OCR toggle** in Settings → Content Processing: a new "Enable OCR" checkbox controls whether the Docling engine runs OCR on scanned PDFs and images. It's on by default (matching content-core's behavior); turn it off to speed up processing of text-native documents. The setting is passed to content-core's `docling_ocr` config, and the label/help are translated across all 14 locales (#1104)
- **Crawl4AI** is now selectable as a URL processing engine in Settings → Content Processing, alongside Firecrawl, Jina and Simple. It renders JavaScript-heavy pages locally with no API key, or offloads to a Crawl4AI server when `CRAWL4AI_API_URL` is set. The Crawl4AI runtime (and its Chromium browser) is bundled into the Docker image, so local mode works out of the box (~300 MB larger image; no torch/CUDA). As part of this, the persisted document/URL engine choices now actually take effect: the source-processing graph reads the saved Content Settings and passes them to content-core (previously it always ran with hard-coded `auto` engines and silently ignored the selection). New engine label added across all 14 locales (#432)
### Changed
- Upgraded the content extraction dependency from content-core 1.14.x to 2.x (2.0.4). The source-processing graph was adapted to content-core's new keyword-only `extract_content()` API: engine/model overrides now travel through a `ContentCoreConfig` object instead of the input dict, and the extraction result (`ExtractionOutput`) no longer echoes the source `url`/`file_path` back, so those are carried from the request state into the saved source asset. Because content-core 2.x no longer deletes the uploaded file after extraction, the graph now honors the `delete_source` flag itself. Transitively this replaces the AGPL-licensed PyMuPDF with MIT-licensed pdfplumber for PDF extraction and drops moviepy in favor of direct ffmpeg calls (which fixes audio extraction from MP3 files carrying chapter metadata). No user-facing configuration changes in this step — document and URL engines stay on their `auto` defaults; new engine/OCR options are tracked separately under #939 (#1103)
- Podcast episode audio paths are now stored relative to the podcasts folder (`episodes/<uuid>/audio/<name>.mp3`) instead of as absolute filesystem paths, validated at write time so the database can never hold an absolute or root-escaping value, and resolved + containment-checked through a single shared helper instead of per-endpoint guards. Migration 21 converts existing rows written under the known roots (plain `file:///` URIs, `/app/data/podcasts/`, `/data/podcasts/`, `./data/podcasts/`, `data/podcasts/`); previously generated episodes now survive a `DATA_FOLDER` relocation. Rows under any other root (e.g. a source checkout at a custom absolute path) or in exotic legacy forms (percent-encoded or host-qualified `file://` URIs) are left untouched and treated as legacy-invalid — the same 403/audio-unavailable handling out-of-root rows already had; regenerate the episode to restore playback. Podcast jobs whose audio combination fails (podcast-creator reports an in-band `ERROR:` value) now fail with the real ffmpeg/clip error instead of reporting success for an episode with no playable audio (#1030)
- The source detail view (dialog and full page) now fetches through the shared `useSource` React Query hook instead of a hand-rolled fetch, matching the insight/note dialogs: caching and the never-retry-404 policy come from React Query, title edits and deletes go through the shared mutation hooks (so source lists refresh and a deleted source can't be served from the cache), and the `key={sourceId}` remount workaround on the parents was removed — the component resets its own per-source UI state (#1106)
- The settings frontend now fetches the provider list from `GET /api/providers` (cached for the session) instead of keeping its own hardcoded provider tables, so adding a provider to the backend registry needs zero frontend edits: unknown modalities render with a fallback icon, the backend registry owns the display order, and the regex-based frontend/backend sync test was removed along with the duplicated tables in `frontend/src/lib/providers.tsx` (#1082)
### Removed
- The legacy provider/model string fields on podcast profiles (`outline_provider`, `outline_model`, `transcript_provider`, `transcript_model` on episode profiles; `tts_provider`, `tts_model` on speaker profiles) are gone from the database, the API and the UI — the app has ignored them since the Model registry references landed in v1.11. Migration 22 first best-effort maps any profile whose `outline_llm`/`transcript_llm`/`voice_model` reference is still empty to an existing model record (matching provider + name + type; no auto-creation, since a migration must not touch credentials), then drops the six columns; the startup data migration that used to retry this mapping on every boot (`open_notebook/podcasts/migration.py`) was deleted. Accepted trade-off: profiles whose mapping never converged (e.g. the provider credential was never configured) lose the legacy strings and stay unresolved — they were already non-functional and the UI already flags them as needing model selection, so you just re-pick the models in the profile form once (#1107)
### Fixed
- Uploading a file content-core can't extract now fails immediately at ingestion with a clear `415 Unsupported Media Type` error that names the detected MIME type, instead of enqueueing a background job that retried up to 15 times over ~1 hour before surfacing a generic "Failed" with no actionable detail. The pre-flight uses content-core 2.x's header-only `check_file_support()` — the same routing real extraction uses, so the verdict can't disagree with what would happen downstream — and the source-retry endpoint is guarded the same way; unexpected check errors (e.g. a file removed before a retry) fall through to normal extraction rather than becoming a hard rejection (#975)
- Podcast episode cards no longer show "— / —" for the outline, transcript and speaker model rows on new episodes: the API now resolves the snapshot's model references (`outline_llm`/`transcript_llm`/`voice_model`) to provider/name display fields at serialization time — batched into a single query per request, so listing episodes never does a per-row model lookup — and the card falls back to the legacy snapshot strings for old episodes and degrades to "—" when a referenced model was deleted (#1114)
- Renaming a speaker profile no longer breaks the episode profiles that use it: `episode_profile.speaker_config` now stores a `record<speaker_profile>` reference instead of the profile name (migration 20 converts existing rows; references whose speaker profile no longer exists at migration time become null, and any reference that later stops resolving is treated as "needs setup" — the UI asks you to pick a speaker again). The `POST /api/podcasts/generate` contract is unchanged — it still accepts the speaker profile by name and resolves it at the API boundary (#630)
- Clicking a chat/Ask citation that points at a deleted source, insight, or note now shows a shared, friendly "this content no longer exists" state in all three dialogs (instead of a raw error, a blank dialog, or an empty editable note editor), 404 lookups are no longer retried, and non-404 failures show a distinct "unable to load" message (#455)
- Source insights now get `created`/`updated` timestamps stamped at creation (migration 19 mirrors the defaults used by the other tables), and the insights API returns `null` — instead of the literal string `"None"` — for legacy insights that predate the migration (#1045)
- `uv sync` alone now provides the full dev toolchain: the legacy `[project.optional-dependencies].dev` list was merged into `[dependency-groups].dev` (mypy included — the documented `uv run python -m mypy .` previously failed on a fresh clone), Jupyter-only packages moved to a separate `notebooks` group, and the CI typecheck job no longer needs `--extra dev` (#1101)
- Optional model defaults (transformation, tools, large context, TTS, STT) can now be cleared: `PUT /api/models/defaults` honors explicit `null` (field absent still means "keep"; chat and embedding defaults reject `null`), and the default-model selects offer a "None" / "Use fallback (chat default)" option for the optional defaults (#1091)
- `docker-compose.yml` now uses the YAML list (exec) form for the SurrealDB `command`, so `SURREAL_USER` / `SURREAL_PASSWORD` values containing spaces are passed as single arguments instead of being split; the mirrored snippets in the installation docs and README (which had drifted — no credential interpolation, SurrealDB port published on all interfaces) are back in sync with the shipped file (#1093)
- zh-CN and zh-TW podcast toast descriptions (speaker/episode profile created/updated/deleted/duplicated) now include the profile name via the `{{name}}` placeholder, matching the other 12 locales (#1084)
- Docker images now force the Next.js frontend to bind to `0.0.0.0` in the supervisord command itself, so container runtimes that inject `HOSTNAME` (e.g. Podman pods, where it resolves to `127.0.1.1`) can no longer make the UI unreachable. The `HOSTNAME` variable is no longer honored as a frontend bind override — set the new `FRONTEND_BIND_HOST` variable instead (#994)
## [1.12.0] - 2026-07-12
### Fixed
- Setup snippets no longer teach publishing SurrealDB on `0.0.0.0` — the compose and `docker run` examples across the README, quick starts, installation, configuration and development docs, and the `examples/docker-compose-*.yml` files now bind port 8000 to `127.0.0.1` (matching the shipped `docker-compose.yml`), with docs pointing to `docker-compose.override.yml.example` for opt-in remote access behind a firewall or SSH tunnel; the override example itself gained the `!override` tag it needs to actually replace the base port binding instead of colliding with it (#1034)
### Added
- Docs: cubic platform mechanics recorded as comments in `cubic.yaml` (agent limits, config precedence, memory/learning) and a "Merging PR Batches" playbook added to the maintainer guide (squash policy, CHANGELOG conflict resolution, fork rebases, competing-PR checks) (#1086)
- New `GET /api/providers` endpoint returning provider metadata from the registry (name, display name, modalities, docs URL, whether it is configured via environment variables), so clients can enumerate supported providers instead of hardcoding them (#1075)
- Release confidence process, documented and executable: `.github/RELEASE_PROCESS.md` now covers the risk-based test matrix (buckets A/B/C), the Docker image gate, the fix-loop re-test policy and the communication/credits/retro structure, backed by a new decision record (ADR-005) and versioned tooling under `scripts/release-test/` — `make release-test TAG= OLD_TAG=` runs fresh-install + upgrade scenarios against real images, and `make release-stack TAG= [DUMP=]` boots a browsable, isolated release-candidate stack (optionally with a copy of dev data) for manual verification (#1052)
- CI now gates every PR on `ruff check` (backend lint), `npm run lint` (frontend ESLint) and `npm run build` (frontend production build), in addition to the existing test suites (#1068)
- CI now also gates every PR on `mypy` (backend typecheck): the repo-wide baseline went from 197 errors to 0 (enabling the pydantic mypy plugin resolved most of them; the rest got real annotations), so new type errors are blocked from here on. The `ignore_errors` burn-down also started: `open_notebook.graphs.transformation`, `open_notebook.graphs.ask` and `api.routers.models` are now type-checked (plus two stale entries for deleted modules removed); only `open_notebook.domain.notebook` remains exempt pending the surreal-basics migration (#1076)
### Changed
- cubic AI review now skips `CHANGELOG.md`, `uv.lock` and `frontend/package-lock.json` (no reviewable logic; preserves the monthly reviewed-line quota) (#1080)
- Context building consolidated into a single implementation (`open_notebook/utils/context_builder.py`): the copy-pasted source/note assembly loops behind `POST /api/chat/context` and the removed notebook-context endpoint, plus the 495-line generalized `ContextBuilder` class (whose only caller was the source-chat graph), are now two focused functions — `build_notebook_context()` (backs `POST /api/chat/context`, unchanged request/response shapes and config semantics) and `build_source_context()` (backs the source-chat graph, same context shape and 50k-token budget). Pinned by new characterization tests — no behavior change for the surviving paths (#1079)
- **Removed** `POST /api/notebooks/{notebook_id}/context`: it duplicated `POST /api/chat/context` (same assembly logic, slightly different response envelope) and had zero callers — frontend, docs and tests only use `/api/chat/context`. If you called it programmatically, switch to `POST /api/chat/context` (body: `{notebook_id, context_config}`; response fields: `context.sources`/`context.notes`, `token_count`, `char_count`) (#1079)
- Backend provider metadata now lives in a single registry (`open_notebook/ai/provider_registry.py`): env var config, modalities, connection-test models, OpenAI-compatible discovery URLs and docs links are defined once per provider, and `PROVIDER_ENV_CONFIG`, `PROVIDER_MODALITIES`, `TEST_MODELS` and `OPENAI_COMPAT_PROVIDERS` are derived from it. Adding a provider drops from ~6 hand-synced dicts to the registry plus two manual copies (the `SupportedProvider` Literal and the frontend provider table), both enforced by tests (#1075)
- Frontend convention cleanup (no user-facing change): hook files unified to kebab-case (`useNotebookChat.ts`/`useSourceChat.ts` → `use-notebook-chat.ts`/`use-source-chat.ts`), `src/components/source/` merged into `src/components/sources/`, the localStorage auth-token parsing ritual extracted into a single `getAuthToken()` helper (`src/lib/auth-token.ts`), and non-streaming raw `fetch` calls routed through `apiClient` (podcast audio download, auth-status check). SSE/streaming paths and the login/checkAuth credential probes deliberately keep raw `fetch` (#1077)
- Pruned unused langchain packages: removed `langchain-community` and `langchain-deepseek` from the dependencies (nothing imports them — DeepSeek and xAI route through esperanto's OpenAI-compatible path, which uses `langchain-openai`). The remaining `langchain-*` provider packages are documented as runtime requirements of esperanto's dynamic `to_langchain()` and the whole langchain/langgraph family now carries explicit upper bounds; `langchain-core` and `langchain-text-splitters` (both directly imported but previously only transitive) are now declared explicitly (#1073)
- The two Docker images (regular and single-container) are now built from a single multi-stage `Dockerfile` with shared stages — regular is the default (`runtime`) target, single-container is `--target single` — so deploy fixes (tiktoken pre-cache, env defaults, npm retry logic) no longer have to be applied twice. `Dockerfile.single` and `supervisord.single.conf` were removed; the single image appends a small `supervisord.surrealdb.conf` to the shared `supervisord.conf` at build time. Published image names and tags are unchanged (#1066)
- Model discovery is now table-driven: the eight providers with OpenAI-compatible `/models` endpoints (OpenAI, Groq, Mistral, DeepSeek, xAI, OpenRouter, DashScope, MiniMax) share one generic discovery function configured by `OPENAI_COMPAT_PROVIDERS`, replacing eight near-identical copies (provider-specific quirks like Mistral's capability flags and OpenRouter's descriptions are preserved as hooks) (#1070)
- Internal refactor: extracted the session/source verification, record-ID normalization, LangGraph message extraction and shared response models duplicated across the chat and source-chat routers into `api/routers/_chat_shared.py`, pinned by new characterization tests — no behavior change (#1072)
- Internal refactor of the sources API router: extracted a shared `SourceResponse` builder (was hand-rolled 5×), a single upload-cleanup helper (was pasted 6×), split the 293-line create-source endpoint into validation + sync/async path functions, and unified the duplicated paginated list query. No behavior change; all security checks (atomic filename claim, path-traversal containment, SSRF/LFI guards) preserved verbatim (#1069)
- Re-enabled the ruff rules for unused imports (`F401`), unused local variables (`F841`) and bare `except:` (`E722`) that were ignored to silence legacy Streamlit-era noise, and cleaned up the remaining fallout (10 unused imports, 2 unused test variables; no bare excepts remained) (#1062)
- Internal refactor with no user-facing change: split the 1,441-line API Keys settings page into focused components under `frontend/src/components/settings/` and moved the provider config tables to `frontend/src/lib/providers.tsx`, deduplicating the default-model select in the process (#1065)
- Chat, source chat, Ask and transformation prompts now steer models to write math as `$$...$$` (display) / `$...$` (inline) so formulas render via KaTeX, reserving fenced `latex` code blocks for when the user explicitly asks for the LaTeX source (#1051)
- Frontend locale files are now type-checked at compile time: every non-en-US locale declares `satisfies TranslationShape` (derived from the en-US object), so a missing or extra i18n key fails `tsc` in the editor instead of only the runtime parity test. Also removed two unused frontend dependencies (`next-themes`, `@monaco-editor/react`) and fixed `frontend/AGENTS.md` drift (14 locales, not 7; dark mode is the hand-rolled zustand theme-store, not next-themes) (#1061)
### Fixed
- Eight podcast toast descriptions (speaker/episode profile created/updated/deleted/duplicated) showed the literal `{name}` placeholder instead of the profile name: the locale strings used single braces (which i18next ignores) and the `t()` call sites passed no values. Placeholders normalized to `{{name}}` across all locales and the actual name is now passed in from the mutation response/variables (#1077)
- Typed domain errors now return their documented HTTP status codes instead of a generic 500: the API routers used to wrap endpoint bodies in a broad `except Exception` that swallowed the `open_notebook.exceptions` hierarchy before the global handlers could map it (`NotFoundError`→404, `InvalidInputError`→400, `ConfigurationError`→422, `RateLimitError`→429, `NetworkError`/`ExternalServiceError`→502). All 18 affected routers now re-raise `HTTPException` and `OpenNotebookError` and only convert genuinely unexpected exceptions into sanitized 500s. Most visible changes: a missing/unconfigured model (`ConfigurationError`) now returns 422 with an actionable message instead of 500; getting or deleting a source-chat session whose session isn't related to the source returns 404 (was a 500 wrapping the inner 404); fetching a missing credential returns 404 (was 500) (#1078)
- Frontend translations now use i18next interpolation (`t('key', { count })`) instead of manual `.replace('{count}', ...)` string surgery across ~75 call sites — locale placeholders changed from `{name}` to `{{name}}` in all 14 locales. This restores proper pluralization (e.g. "used by N episodes" now goes through i18next plural forms) and lets translators reorder placeholders freely (#1074)
- Podcast generation dialog: the token/char counter no longer fires a request storm on rapid checkbox toggling (debounced, with a stale-response guard so a slow response can't overwrite a fresher count) and the dialog now closes as soon as the episode-list refetch completes instead of after a fixed 500ms timer; the 983-line component was also split (content selection panel and selection helpers extracted, duplicated context-config logic deduplicated) with no behavior changes (#1067)
- Anthropic models are now discovered live from `GET https://api.anthropic.com/v1/models` (paginated) instead of a hardcoded claude-3-era list — the code comment claiming "Anthropic doesn't have a model listing API" was wrong. A refreshed static list (current Claude 4.x/5 aliases) remains as a fallback when the API call fails, and the credential-based discovery path (`discover_with_config`) uses the same live-with-fallback logic (#1070)
- Deduplicated the embedding commands (`commands/embedding_commands.py`, ~100 lines less): `embed_note`/`embed_insight`/`embed_source` now share one load→embed→write core with a single error-handling epilogue, the rebuild command uses one submission-loop helper for all three kinds, and the thrice-copied `full_model_dump()` moved to `open_notebook/utils/model_utils.py`. Pure refactor — same outputs, logs and retry behavior (#1071)
### Removed
- Dead Streamlit-era service layer (~2,000 lines): `api/client.py` (a synchronous HTTP client that called the app's own API) and 13 `api/*_service.py` wrappers that consumed the app's own HTTP API — none were imported by any router, command or test. Also removed the toy `process_text`/`analyze_data` demo commands (`commands/example_commands.py`) from the background worker (#1054)
- Pre-1.6 embedding job compatibility shims (the `embed_single_item`, `embed_chunk` and `vectorize_source` command handlers) — they existed only so jobs queued by a pre-1.6 version could drain after an upgrade, and any worker restarted on 1.6+ has no such jobs. **Upgrade note:** if you are upgrading from a version older than 1.6 with embedding jobs still queued, drain the queue on a 1.x release before upgrading past this change. Also removed dead tooling config from `pyproject.toml`: the `[tool.mypy]` block (the real config is `mypy.ini`) and Streamlit-era ruff per-file-ignores for files that no longer exist (#1056)
- Committed QA screenshots (12 files) and a stray debug `history.txt` were removed from the repo root, with `.gitignore` rules added so they can't come back (#1053)
### Fixed
- Podcast generation now honors the `speaker_profile` parameter of `POST /api/podcasts/generate` — previously it was silently ignored and the speaker was always re-derived from the episode profile's `speaker_config`, which failed when that pointed at a renamed/deleted speaker profile even if the caller supplied a valid one (#1044)
## [1.11.0] - 2026-07-11
### Added
- `VISION.md` — the product's source of truth in two layers: durable identity (what Open Notebook is and is not, core principles) and current posture (the phase we're in, directional constraints, and the horizon clusters under consideration)
- Decision records at `docs/7-DEVELOPMENT/decisions/` — short, immutable ADRs/PDRs answering "why is it like this?", seeded with 4 retroactive ADRs (SurrealDB, delegation to external libraries, Streamlit→Next.js, background workers) and 2 PDRs (single-user first, provider-agnostic core)
- `AGENTS.md` files (root, `open_notebook/`, `frontend/`) with the normative rules for coding agents and humans — commands, hard rules, and gotchas not derivable from the code; `CLAUDE.md` files are now one-line pointers to them
- Five new engineering docs pages under `docs/7-DEVELOPMENT/`: credentials, content processing, podcasts, prompts, and frontend architecture
- Contribution guidelines for AI-assisted and agent-generated PRs in the contributing guide — the operator owns the PR, issue-first still applies, tests must have actually run
- CI check for broken relative links in markdown (`scripts/check_md_links.py` + `docs-links` workflow on PRs touching `*.md`)
- `cubic.yaml` — AI review settings as code: PR-contract instructions, three custom review agents (vision & principles alignment backed by `VISION.md`, known mechanical caveats, security & testability) and automatic ultrareviews for auth/credential/encryption/migration changes
- Documented the flow-driven release process in `.github/RELEASE_PROCESS.md`, including the `ready` to `main` to stable release path, dev/stable image labels, and maintainer verification checklist (#938)
- List view for the Notebooks page — a tile/list toggle in the header lets you switch between the visual card grid and a compact row layout (name, description, source/note counts, last updated) for easier scanning of large collections. The choice is remembered across reloads and translated across all 14 locales (#885)
- Documented the `ESPERANTO_TTS_TIMEOUT` environment variable (default `300`s) in the environment reference; raise it for slow or self-hosted TTS providers so long podcast segments don't fail with a timeout (#937)
- `SECURITY.md` with a coordinated-disclosure policy: how to privately report a vulnerability via GitHub's private vulnerability reporting, supported versions, and response expectations (#943)
- LaTeX math rendering (KaTeX) now also applies to source content, source insights, Ask answers, transformation output, and the note editor preview — previously only chat had it (#269)
- Syntax highlighting for fenced code blocks in chat responses, source content, insights and Ask/Search answers — light/dark aware, with 25 common languages bundled (others render as plain text) (#783)
- "Recently Viewed" section on the Notebooks page — a collapsible grid of the last 12 notebooks and sources you opened, newest first, hidden when there's no view history. Backed by a new `last_viewed_at` timestamp stamped on read and a `GET /api/recently-viewed` endpoint (translated across all 14 locales) (#850)
- Per-transformation model selection — each transformation can now be assigned its own language model from the transformation editor, overriding the global transformation default for that transformation only. Runs without an explicit model keep using the system default as before (#776)
- "Refresh content" action on web-link sources — re-fetches the URL and re-embeds the source so its content stays current, available from the source card menu once processing has completed (translated across all 14 locales) (#259)
- Sources table can now be sorted by every column — type, title, insights count, embedded status, created and updated (a new "Updated" column was added) — via clickable column headers backed by new `GET /api/sources` sort fields (translated across all locales) (#895)
- EasyPanel deployment template under `examples/easypanel/` — provisions the app plus a dedicated SurrealDB service with auto-generated database/encryption secrets — plus an EasyPanel section in the single-container install guide (#189)
- Test coverage measurement in CI: backend via `pytest-cov` (terminal + XML reports), frontend via `@vitest/coverage-v8` and a new `test:coverage` script (#942)
### Changed
- Developer documentation restructured: 17 knowledge-heavy `CLAUDE.md` files consolidated into the 3 `AGENTS.md` + docs pages above; `README.dev.md` became a pointer after its unique content moved into `development-setup.md` (make-workflow matrix), `.github/RELEASE_PROCESS.md` (Docker publishing) and the change playbooks (add-a-language); the maintainer guide now carries the curated label taxonomy (state funnel, `area:` labels, consolidation rules)
- Fixed stale developer docs while migrating: real migration path/format (`open_notebook/database/migrations/N.surrealql` + `AsyncMigrationManager` registration), provider count (17), locale list (7), and 9 README links that pointed at documentation pages that never existed
- The API's listen interface in the Docker images is now configurable via a new `API_HOST` environment variable instead of a hardcoded `--host 0.0.0.0`. The default is unchanged (`0.0.0.0`); set `API_HOST=::` to serve IPv6/dual-stack environments (#985)
- `docker-compose.yml` now sources the SurrealDB credentials from `SURREAL_USER` / `SURREAL_PASSWORD` (applied to both the database server and the app), defaulting to `root:root` so the zero-config quick start is unchanged. Set them in a `.env` file to use your own credentials before exposing the instance; `.env.example` and the compose file note this (#946)
- Docs no longer claim a hardcoded default API password (`open-notebook-change-me`) exists; the actual behavior is that auth is disabled entirely when `OPEN_NOTEBOOK_PASSWORD` is unset. Also removed the dead `check_api_password` helper that had been superseded by the auth middleware (#1026)
### Fixed
- Testing a valid Google/Vertex credential no longer fails after Google retires a Gemini model. The connection test used a hard-coded model id that Google shuts down on a schedule (`gemini-2.0-flash`), so a valid key surfaced as a broken connection (#970). The Google/Vertex test now uses Google's floating `gemini-flash-latest` alias, and the provider connection test was reframed so only a rejected key, missing permissions, or an unreachable endpoint count as failures — a missing/retired/rate-limited model still reports the credentials as valid. Deprecated `gemini-1.5`/`gemini-2.0` model references were also removed from the connection-test model lists and documentation
- API startup no longer crashes when SurrealDB isn't ready yet (e.g. docker-compose race on host reboot: `Temporary failure in name resolution`). The lifespan now polls a lightweight readiness probe with bounded exponential backoff (~50s budget, 5s per-probe timeout) before running migrations; migration errors themselves still fail fast (#708)
- Markdown typography styles (`prose` classes) are active again: the Tailwind v4 migration left the old `tailwind.config.ts` (which loaded `@tailwindcss/typography`) silently ignored, so rendered markdown lost its typographic styling. The plugin and class-based dark mode are now configured in `globals.css`, and markdown rendering is centralized in a shared `MarkdownRenderer` component (#783)
- Podcast generation no longer truncates on dense, long-form content (`LengthFinishReasonError` / `OUTPUT_PARSING_FAILURE`): episode profiles now support an optional `max_tokens` that is passed through to podcast_creator's outline/transcript generation, overriding its defaults — settable via the episode profile API (UI follow-up in #991) (#639)
- API no longer freezes for all requests while a chat waits on the LLM. Both the notebook chat (`execute_chat`) and source chat handlers ran LangGraph's synchronous `invoke()` directly on the event loop; they now run it via `asyncio.to_thread()` (matching the existing `get_state` calls), so other requests stay responsive — and the source-chat SSE can flush its early events instead of stalling until the model finishes (#704)
- Windows native install guide no longer points users at a `start-open-notebook.bat` that doesn't exist in the repo; the Quick Start now documents starting the four services manually with `uv run`, plus an optional sample launcher you can save yourself (#846)
- OpenRouter (and other providers') "Discover models" dialog no longer cuts off the submit button: the dialog now uses a fixed header/footer with a scrollable body (`grid-rows-[auto_1fr_auto]`) instead of scrolling the whole content, so the "Add" button stays visible regardless of how many models are listed (#816)
- Chat references using the short `[insight:<id>]` form (emitted by some models) are now rendered as clickable citations like `[source_insight:<id>]` and `[note:<id>]` already were; `insight` is treated as an alias for `source_insight`, so clicking it opens the insight (#490)
- CRUD endpoints now return `404` (not `500`) for a non-existent resource. `ObjectModel.get()` raises `NotFoundError` rather than returning a falsy value, so the broad `except Exception` in each handler was masking it as a server error. Added an explicit `NotFoundError → 404` arm to the notebook (update / delete / delete-preview / add-source / remove-source), note (get / update / delete / list / create), model (delete), credential (update / delete) and embed handlers (#862)
- Token counting no longer raises `ValueError: disallowed special token '<|endoftext|>'` when source/context content contains special-token sequences; `token_count()` now encodes with `disallowed_special=()` so such substrings are treated as ordinary text (#667)
- Single-container image no longer hangs at "API not ready yet" on a brand-new instance. `supervisord.single.conf` ran the API and worker with `uv run` (without `--no-sync`), so at startup `uv` tried to sync dev dependencies it couldn't resolve against the `--no-dev` build. Both processes now use `uv run --no-sync`, matching the multi-container `supervisord.conf` (#609)
- Note editor now expands to fill the dialog instead of being capped at `500px`; removed the `max-h-[500px]` constraint that overrode the `flex-1` parent and cramped editing on tall windows (#932)
- Ask and source-chat responses now stream progressively instead of hanging at "Processing..." until the full answer is ready. The API's streaming endpoints now declare `text/event-stream` (with no-buffering headers), and dedicated Next.js route handlers pass the SSE body through as a stream — Next.js `rewrites()` buffers SSE responses to completion (#770)
- Chat, notebook-context and podcast generation now build their context with a single batched insight query instead of one query per source (14 → 3 queries on a 12-source notebook), via the new `SourceInsight.get_for_sources()` (#1008)
- File uploads no longer block the event loop: `save_uploaded_file()` now writes via `asyncio.to_thread()`, keeping the API responsive during large uploads (#1009)
- URL validation no longer blocks the event loop on DNS resolution: `validate_url()` is now async and resolves hostnames via `asyncio.to_thread()`, so a slow DNS lookup on the model-provisioning path can't stall concurrent requests (#1011)
- Creating a credential with an unknown provider name now fails with a clear `422` at the API boundary instead of an opaque error deep in the domain layer; `provider` is validated against the 17 supported providers, and a test keeps the frontend/backend provider lists in sync (#1016)
- Podcast episode listing now batch-fetches job statuses in one query instead of one per episode, speeding up notebooks with many episodes; podcast audio-file paths are additionally verified to stay within the podcasts folder before streaming/deleting (#1018)
- Transformations no longer report success while silently losing their insight when the embedding job fails to queue: `Source.add_insight()` now raises on submission failure (handled by job-level retry), note auto-embedding degrades gracefully instead of turning a note save into a 500, and the explicit note-embed endpoint surfaces queue failures as errors (#1019)
- Clearing a credential field in the edit dialog (Ollama/OpenAI-compatible `base_url`, Vertex `project`/`location`/`credentials_path`) now actually clears it. Two mirror-image bugs made it impossible: the frontend dropped emptied fields from the PUT body (`undefined` keys are stripped by `JSON.stringify`), and the API ignored explicit `null`s (`is not None` guards) — so the old value survived while the UI reported success. The frontend now sends explicit `null` and the API keys partial updates on field presence (`model_fields_set`) (#1046)
### Security
- Resolved dependency audit findings: added npm `overrides` for vulnerable transitive frontend packages (`ws`, `brace-expansion`, `ajv`, `@eslint/plugin-kit`, `postcss`) — `npm audit` now reports 0 vulnerabilities — refreshed `uv.lock` (`langsmith`, `pydantic-settings`, `pip`), and hardened external `window.open(..., '_blank')` calls with `noopener,noreferrer` (#962)
- SurrealQL injection via record ids in `repo_relate()`/`repo_upsert()`/`repo_update()`: a crafted `notebook_id` on the save-insight-as-note flow could execute arbitrary SurrealQL. Record identifiers are now bound as query parameters, and the target notebook's existence is validated before relating (#1002)
- The API password is now compared with `secrets.compare_digest()` instead of `!=`, closing a timing side-channel on authentication (#1003)
- User-authored transformation prompts are no longer compiled as Jinja2 template source (a DoS vector via template loops); they are passed as plain variables into fixed developer-authored templates, so Jinja syntax inside a prompt renders as inert text. Output is unchanged for legitimate prompts (#1004)
- SSRF protection on source-URL ingestion: adding a web-link source now runs the same `validate_url()` guard already used for credential URLs, rejecting internal/private/cloud-metadata addresses (#1005)
- Provider-credential URLs are re-validated immediately before every outbound request (connection tests, model discovery and inference) instead of only at save time, closing a DNS-rebinding window; AWS's IPv6 metadata address was added to the blocklist (#1006)
- The note/transformation markdown preview now sanitizes raw HTML via `rehype-sanitize`: `<iframe>`/`<script>`/`<style>` tags and `javascript:` URLs are stripped while math, syntax highlighting and GFM still render — closing an HTML-injection path via AI-generated note content (#1007)
- Vertex credential-test errors no longer reveal whether a `credentials_path` file is missing, invalid JSON or wrong-shape JSON (a filesystem oracle); all three cases now return one generic message (#1012)
- CORS no longer combines the wildcard origin with `allow_credentials=True` (which made Starlette reflect any Origin verbatim for credentialed requests); credentials are now only allowed when `CORS_ORIGINS` is explicitly configured (#1013)
- Request bodies are now capped before auth and routing by a new `MaxBodySizeMiddleware` — default 100 MB, configurable via the new `OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB` environment variable; chunked uploads are caught by a streaming byte-count (#1014)
- Source upload hardening: unique filenames are claimed atomically (closing a TOCTOU race between the exists-check and the write), path-containment checks require a trailing separator so sibling directories can't pass, and the `notebooks`/`transformations` arrays on source creation are capped at 50 items (#1015)
- API 500 responses from the sources and podcast endpoints no longer echo internal exception text (which could leak DB hostnames or connection details); responses use fixed generic messages and details remain in server logs (#1017)
- `Credential.get_all()` no longer builds its ORDER BY clause from a raw f-string; ordering fields now go through the base-class allowlist validation (not reachable from the current API surface — defense in depth) (#1021)
- The frontend runtime-config endpoint validates `Host` and `X-Forwarded-Proto` before using them to build the browser-facing API URL, preventing a spoofed Host header from redirecting browser API traffic (including the bearer token) to an attacker-controlled origin; malformed values fall back to localhost (#1024)
- `docker-compose.yml` now binds SurrealDB's published port to `127.0.0.1` instead of all interfaces, so the database (root:root by default) is no longer reachable from other machines out of the box; a new `docker-compose.override.yml.example` shows how to re-expose it deliberately (#1025)
- Forced **Pillow to 12.3.0**, resolving 6 open Dependabot advisories (3 high: PSD out-of-bounds writes, FITS GZIP decompression bomb; 3 moderate: PDF trailer DoS, font integer overflow, heap buffer overflow). The only blocker was moviepy's `pillow<12` cap (pulled in via podcast-creator) — moviepy only touches PIL in its video modules, which the audio-only podcast pipeline never imports, so a documented `[tool.uv] override-dependencies` entry forces the safe version until podcast-creator ships without moviepy (#1041)
## [1.10.0] - 2026-06-17
### Security
- Bumped **Starlette to 1.2.1** and **FastAPI to 0.136.3** to address **CVE-2026-48710** ("BadHost"), a denial-of-service in Starlette's host header handling (#859)
### Added
- LaTeX math rendering in chat — inline (`$...$`) and display (`$$...$$`) expressions are now rendered with KaTeX (#606)
- `NEXT_PUBLIC_API_TIMEOUT_MS` environment variable to configure the frontend API request timeout (default `600000` = 10 minutes; set `0` to disable). Lets slow/long-running chat models finish without editing source (#880)
- Bulk chat-context actions in a notebook, via a "Context" menu in the Sources and Notes column headers — translated across all 14 locales (#223):
- Sources: "Include all (insights only)" (sources without insights are left out rather than forced to full), "Include all (full content)", and "Exclude all from context"
- Notes: "Include all in context" / "Exclude all from context"
- **Turkish (tr-TR) localization** — the UI is now fully translated into Turkish (#871)
### Changed
- Failed source cards now show a prominent "Retry processing" button directly on the card instead of only inside the 3-dot dropdown; clicking it no longer also opens the source (the click was missing `stopPropagation`) (#726)
- Docker base image updated to **Debian trixie** and **Node.js 22.x** (#914)
### Fixed
- Podcast generation now uses the notebook's real content. `Notebook.get_context()` was missing, so generation ran against empty context; it now assembles source and note content as expected (#864)
- `PUT` profile handlers now use `model_dump(exclude_unset=True)`, so partial updates no longer overwrite unspecified fields with defaults (#860)
- OpenRouter embedding models are now correctly recognized via their embedding modality (#842)
- Search and Ask results now use page-level scrolling instead of being confined to a cramped, height-capped (`60vh`) bottom container, so the full result set is readable (#882)
- `POST /sources/{id}/retry` no longer returns `400 "Source is not associated with any notebooks"` for every source; it now queries the `reference` graph edge by its `in`/`out` columns instead of a non-existent `source` column (#861)
- `POST /sources/{id}/retry` no longer returns a `500` ("too many values to unpack") after successfully queuing the retry job; the command ID was being double-prefixed (`command:command:…`) before being saved to the source. Retrying a failed source now succeeds and updates the source's command reference
- `GET /sources/{id}` for a missing or deleted source now returns `404` instead of `500`; the handler caught `NotFoundError` in its generic `except` and mapped it to a server error
- Sources that fail to ingest (e.g. an unreachable or invalid URL) are now marked `failed` instead of silently saved as `completed` with the extraction error as their body. This means the "Retry processing" button (#726) actually appears for the most common failure mode; previously the job returned a failure payload but the command still completed, so the source never reached a retryable state (#726)
- Text search no longer returns a 500 when SurrealDB's `search::highlight` hits a "position overflow" on large or multi-byte document chunks; it now falls back to vector search and returns results (#648)
- `POST /api/search` now rejects a non-positive `limit` with a `422` instead of passing `LIMIT -1`/`LIMIT 0` to SurrealDB (which caused a 500 or a silently empty result set) (#863)
- Ollama `num_ctx` credential override is now persisted. The `credential` table gained a flexible `config` object (migration 15) and provider-specific tuning options are stored there instead of being dropped by the SCHEMAFULL table; future per-credential options can be added without a schema migration (#875)
- Worker no longer crashes on queued jobs from older versions; legacy embedding command aliases (`embed_single_item`, `embed_chunk`, `vectorize_source`) are registered and delegate to the current commands so stale queues drain cleanly (#695, #876)
### Performance
- Notebook source list no longer re-renders every `SourceCard` on unrelated state changes (layout toggles, context selection), and completed sources no longer each open a status-polling query. Both scaled with the number of sources and caused UI lag on large notebooks (#503)
## [1.9.0] - 2026-06-02
### Added
- **New audio providers**, surfacing the capabilities added in Esperanto 2.21–2.22:
- **Mistral Voxtral** speech-to-text (`voxtral-*-latest`) and text-to-speech (`voxtral-mini-tts`), reusing the existing Mistral credential (#827)
- **Deepgram** text-to-speech (Aura voice catalog) as a new provider (`DEEPGRAM_API_KEY`) (#827)
- **xAI** text-to-speech (#827)
- **Google** speech-to-text & text-to-speech, **Vertex** text-to-speech, and **ElevenLabs** speech-to-text (Scribe), completing the audio provider matrix (#828)
- Optional per-credential **`num_ctx`** (context window) override for Ollama models, configurable in Settings → API Keys and translated across all 13 locales (#825)
- `OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE` environment variable to override the embedding batch size; default remains `50`. Helps with CPU-only local embedding and stricter OpenAI-compatible endpoints (#735)
- `CORS_ORIGINS` environment variable to configure the API's allowed origins (comma-separated). Default remains `*` for backward compatibility; the API now logs a startup warning prompting users to set it for production deployments. Exception responses honor the configured origins when explicitly set (#585, #597, #730)
- `OPEN_NOTEBOOK_MIN_CHUNK_SIZE` environment variable (default: 5 tokens) to filter out degenerate tiny chunks before embedding. Set to `0` to disable.
### Changed
- Bumped **Esperanto 2.20.0 → 2.22.0**. Beyond the new audio providers above, this inherits several upstream fixes and behavior changes (see below).
### Inherited from Esperanto 2.21–2.22
- **Fixed:** OpenRouter LLM and embedding requests now send a proper JSON body (previously sent a malformed form-encoded payload).
- **Fixed:** OpenAI-compatible endpoints (e.g. llama.cpp) that return null embeddings now raise a clear, descriptive error instead of an opaque `TypeError`.
- **Fixed:** Streaming tool calls now return proper `ToolCall` objects across Anthropic, Google, Vertex, and Ollama.
- **Fixed:** `base_url` trailing slashes are normalized across providers, preventing double-slash URLs (and 301 redirects) for Ollama and other self-hosted endpoints.
- **Fixed:** Ollama "thinking" models (e.g. Qwen) now merge their reasoning content correctly.
- **Fixed:** Model discovery honors a custom `base_url` (LiteLLM/vLLM/OpenAI-compatible proxies).
- **Behavior change:** the Ollama default context window (`num_ctx`) is now **8192** (was 128000) to avoid out-of-memory errors on consumer GPUs. Raise it per-credential via the new `num_ctx` field if your hardware allows.
- **Behavior change:** the Google embedding default model is now `gemini-embedding-001` (the previous default, `text-embedding-004`, was removed from Google's API). If you used Google embeddings with the old default, re-create the model and re-embed your content (embedding dimensions changed).
- **Fixed:** Google TTS default model updated to a currently-working preview model.
### Fixed
- URL source embedding no longer crashes with `TypeError: float() argument must be a string or a real number, not 'NoneType'` when header-based splitters emit single-character fragments from complex HTML pages (e.g. Wikipedia, Project Gutenberg). Such chunks are now filtered before being sent to the embedding provider (#764)
- Language toggle now uses `t('common.german')` instead of a hardcoded "Deutsch" label, matching the pattern used by every other language entry (follow-up to #794)
- Speech-to-text model connection tests now transcribe a short bundled speech clip instead of silence, so a passing test returns real text instead of a blank transcription (#838)
## [1.8.5] - 2026-04-14
### Changed
- Embedding chunking is now token-based instead of character-based, improving chunk sizing consistency for CJK and mixed-language content (#542, #749)
- `OPEN_NOTEBOOK_CHUNK_SIZE` and `OPEN_NOTEBOOK_CHUNK_OVERLAP` semantics changed from characters to tokens; default reduced from 1200 characters to 400 tokens to stay safely below the 512-token ceiling of BERT-family embedders (e.g. mxbai-embed-large) after accounting for tokenizer mismatch and splitter overshoot. Existing stored embeddings are unaffected; only new ingestions use the new chunking.
### Fixed
- Credentials endpoint no longer crashes (500) when encryption key doesn't match stored credentials (#740)
- Broken credentials are now shown with a decryption warning and can still be deleted
- DELETE endpoint for broken credentials supports model migration (`migrate_to` parameter)
## [1.8.4] - 2026-04-09
### Security
- Fix Remote Code Execution (RCE) via Jinja2 Server-Side Template Injection in transformations (CVSS 9.2 Critical)
- Fix arbitrary file write via path traversal in file upload (CVSS 7.0 High)
- Fix arbitrary file read via Local File Inclusion in source creation (CVSS 8.2 High)
### Dependencies
- Bump ai-prompter to >=0.4.0 (uses Jinja2 SandboxedEnvironment to prevent SSTI)
## [1.8.3] - 2026-04-07
### Security
- Fix SurrealDB injection via unsanitized `order_by` query parameter in `GET /api/notebooks` (CVSS 8.7 High)
- Add allowlist validation for sorting parameters in notebooks endpoint
- Replace f-string query interpolation with parameterized `$variable` binding in source chat and migration queries
- Add defensive validation in `get_all()` base method to prevent injection via `order_by` parameter
## [1.8.2] - 2026-04-06
### Added
- DashScope (Qwen) and MiniMax provider support via Esperanto v2.20.0 (#725)
- Source list auto-refresh after adding a new source via URL, file upload, or text (#721)
### Fixed
- Source asset persistence — failed sources now persist their asset (URL/file path), making them identifiable and retryable (#722)
- Source title preservation — user-set custom titles are no longer overwritten after background processing (#722)
- Credential cascade delete — deleting a credential now removes linked models instead of returning a 409 error (#722)
- Podcast directory names — uses UUID for episode directories, fixing filesystem errors with special characters (#666)
- Tiktoken offline handling — API no longer crashes in air-gapped environments (#622)
- SurrealDB healthcheck — removed incompatible healthcheck from Docker Compose (#656)
- Esperanto embedding fixes — base_url/api_key config issues across multiple embedding providers (#664, #665)
### Docs
- Deprecated single-container Docker image in favor of Docker Compose (#723)
### Dependencies
- Bump esperanto to >=2.20.0
## [1.8.1] - 2026-03-10
### Added
- i18n support for Bengali (bn-IN) (#643)
- Podcast language support via podcast-creator 0.12.0 (#645)
- Upgrade default Azure API version for model testing and fetching (#638)
### Fixed
- Tiktoken network errors in offline/air-gapped Docker deployments — pre-downloads encoding at build time (#264, #622)
- SurrealDB getting stuck (#656)
### Dependencies
- Bump esperanto to 2.19.5 (#657)
- Bump langgraph from 1.0.6 to 1.0.10rc1 (#658)
- Bump authlib from 1.6.6 to 1.6.7 (#649)
- Bump lxml-html-clean from 0.4.3 to 0.4.4 (#646)
- Bump rollup from 4.55.1 to 4.59.0 (#635)
- Bump minimatch in frontend (#634)
- Bump tar from 7.5.9 to 7.5.11 (#650, #659)
## [1.7.4] - 2026-02-18
### Fixed
- Embedding large documents (3MB+) fails with 413 Payload Too Large (#594)
- `generate_embeddings()` now batches texts in groups of 50 with per-batch retry, preventing provider payload limits from being exceeded
- 413 errors now classified with user-friendly message in error classifier
- Misleading "Created 0 embedded chunks" log in `process_source_command` — embedding is fire-and-forget, so the count was always 0; now logs "embedding submitted" instead
## [1.7.3] - 2026-02-17
### Added
- Retry button for failed podcast episodes in the UI (#211, #218)
- Error details displayed on failed podcast episodes (#185, #355)
- `POST /podcasts/episodes/{id}/retry` API endpoint for re-submitting failed episodes
- `error_message` field in podcast episode API responses
### Fixed
- Podcast generation failures now correctly marked as "failed" instead of "completed" (#300, #335)
- Disabled automatic retries for podcast generation to prevent duplicate episode records (#302)
### Dependencies
- Bump podcast-creator to >= 0.11.2
- Bump esperanto to >= 2.19.4
## [1.7.2] - 2026-02-16
### Added
- Error classification utility that maps LLM provider errors to user-friendly messages (#506)
- Global exception handlers in FastAPI for all custom exception types with proper HTTP status codes
- `getApiErrorMessage()` frontend helper that falls back to backend messages when no i18n mapping exists
### Fixed
- LLM errors (invalid API key, wrong model, rate limits) now show descriptive messages instead of "An unexpected error occurred" (#590)
- SSE streaming error events in source chat and ask hooks were swallowed by inner JSON parse catch blocks
- Transformation execution errors were caught and re-wrapped as generic 500s instead of using proper status codes
- Fail fast when source content extraction returns empty instead of retrying (#589)
- Chat input and message overflow with long unbroken strings (#588)
- Word-wrap overflow in source cards, note editor, inline edit, note titles, and dialog content (#588)
- Translation proxy shadowing `name` keys (#588)
- OpenAI-compatible provider name handling via Esperanto update (#583)
### Changed
- `ValueError` replaced with `ConfigurationError` in model provisioning for proper error classification
- `ConfigurationError` added to command retry `stop_on` lists to avoid retrying permanent config failures
### Dependencies
- Bump esperanto to 2.19.3 (#583)
- Bump podcast-creator to 0.9.1
## [1.7.1] - 2026-02-14
### Added
- French (fr-FR) language support (#581)
- CI test workflow and improved i18n validation (#580)
- Expose embed `command_id` in note API responses (#545)
### Fixed
- ElevenLabs TTS credential passthrough via Esperanto update (#578)
- Handle empty/whitespace source content without retry loop (#576)
- Increase transformation `max_tokens` and update Esperanto dep (#568)
- Turn the embedding field into optional (#557)
### Docs
- Fix docker container names in local setup guides (#577)
### Dependencies
- Bump langchain-core from 1.2.7 to 1.2.11 (#564)
- Bump cryptography from 46.0.3 to 46.0.5 (#563)
## [1.7.0] - 2026-02-10
### Added
- **Credential-Based Provider Management** (#477)
- New Settings → API Keys page for managing AI provider credentials via the UI
- Support for 14 providers: OpenAI, Anthropic, Google, Groq, Mistral, DeepSeek, xAI, OpenRouter, Voyage AI, ElevenLabs, Ollama, Azure OpenAI, OpenAI-Compatible, and Vertex AI
- Secure storage of API keys in SurrealDB with field-level encryption (Fernet AES-128-CBC + HMAC-SHA256)
- One-click connection testing, model discovery, and model registration per credential
- Migration tool to import existing environment variable keys into the credential system
- Azure OpenAI support with service-specific endpoints (LLM, Embedding, STT, TTS)
- OpenAI-Compatible support with per-service URL configurations
- Vertex AI support with project, location, and credentials path
- Environment variable API keys deprecated in favor of Settings UI
- **Security Enhancements**
- Docker secrets support via `_FILE` suffix pattern (e.g., `OPEN_NOTEBOOK_PASSWORD_FILE`)
- Default encryption key derived from "0p3n-N0t3b0ok" for easy setup (change in production!)
- Default password "open-notebook-change-me" for out-of-box experience (change in production!)
- URL validation for SSRF protection - blocks private IPs and localhost (except for Ollama which runs locally)
- Security warnings logged when using default credentials
- HTML clipboard detection for text sources (#426)
- When pasting content, automatically detects HTML format (e.g., from Word, web pages)
- Shows info message when HTML is detected, informing user it will be converted to Markdown
- Preserves formatting that would be lost with plain text paste
- Bump content-core to 0.11.0 for HTML to Markdown conversion support
- **Improved Getting Started Experience**
- Simplified docker-compose.yml in repository root (single official file)
- Added examples/ folder with ready-made configurations:
- `docker-compose-ollama.yml` - Local AI with Ollama
- `docker-compose-speaches.yml` - Local TTS/STT with Speaches
- `docker-compose-full-local.yml` - 100% local setup (Ollama + Speaches)
- Inline quick start in README (no need to navigate to docs)
- Cross-references between docker-compose examples and documentation
- .env.example template with all configuration options
### Fixed
- Azure form race condition: all configuration now saved in single atomic request
- Migration API "error error" display: added proper MigrationResult model with message field
- Connection tester for Ollama providers: improved error handling and URL validation
- SqliteSaver async compatibility issues in chat system (#509, #525, #538)
- Re-embedding failures with empty content (#513, #515)
- Deletion cascade for notes and sources (#77)
- YouTube content availability issues (#494)
- Large document embedding errors (#489)
### Security
- API keys are encrypted at rest using Fernet symmetric encryption
- Keys are never returned to the frontend, only configuration status
- SSRF protection prevents internal network access via URL validation
### Docs
- Complete documentation update for credential-based system across 25 files
- All quick-start, installation, and configuration guides now use Settings UI workflow
- Environment variable API key instructions moved to deprecated/legacy sections
- Fixed broken links in installation docs
- Added comprehensive examples/ folder with documented docker-compose configurations
- Updated local-tts.md and local-stt.md with links to ready-made examples
### Internationalization
- Added Russian (ru-RU) language support (#524)
- Added Italian (it-IT) language support (#508)
## [1.6.2] - 2026-01-24
### Fixed
- Connection error with llama.cpp and OpenAI-compatible providers (#465)
- Bump Esperanto to 2.17.2 which fixes LangChain connection errors caused by garbage collection
## [1.6.1] - 2026-01-22
### Fixed
- "Failed to send message" error with unhelpful logs when chat model is not configured (#358)
- Added detailed error logging with model selection context and full traceback
- Improved error messages to guide users to Settings → Models
- Added warnings when default models are not configured
### Docs
- Ollama troubleshooting: Added "Model Name Configuration" section emphasizing exact model names from `ollama list`
- Added troubleshooting entry for "Failed to send message" error with step-by-step solutions
- Updated AI Chat Issues documentation with model configuration guidance
## [1.6.0] - 2026-01-21
### Added
- Content-type aware text chunking with automatic HTML, Markdown, and plain text detection (#350, #142)
- Unified embedding generation with mean pooling for large content that exceeds model context limits
- Dedicated embedding commands: `embed_note`, `embed_insight`, `embed_source`
- New utility modules: `chunking.py` and `embedding.py` in `open_notebook/utils/`
- Japanese (ja-JP) language support (#450)
### Changed
- Embedding is now fire-and-forget: domain models submit embedding commands asynchronously after save
- `rebuild_embeddings_command` now delegates to individual embed_* commands instead of inline processing
- Chunk size reduced to 1500 characters for better compatibility with Ollama embedding models
- Bump Esperanto to 2.16 for increased Ollama context window support
### Removed
- Legacy embedding commands: `embed_single_item_command`, `embed_chunk_command`, `vectorize_source_command`
- `needs_embedding()` and `get_embedding_content()` methods from domain models
- `split_text()` function from text_utils (replaced by `chunk_text()` in chunking module)
### Fixed
- Embedding failures when content exceeds model context limits (#350, #142)
- Empty note titles when saving from chat (clean thinking tags from prompt graph output)
- Orphaned embedding/insight records when deleting sources (cascade delete)
- Search results crash with null parent_id (defensive frontend check)
- Database migration 10 cleans up existing orphaned records
## [1.5.2] - 2026-01-15
### Performance
- Improved source listing speed by 20-30x (#436, closes #351)
- Added database indexes on `source` field for `source_insight` and `source_embedding` tables
- Use SurrealDB `FETCH` clause for command status instead of N async calls
## [1.5.1] - 2026-01-15
### Fixed
- Podcast dialog infinite loop error caused by excessive translation Proxy accesses in loops
- Podcast dialog UI freezing when typing episode name or additional instructions
- Removed incorrect translation keys for user-defined episode profiles (user content should not be translated)
## [1.5.0] - 2026-01-15
### Added
- Internationalization (i18n) support with Chinese (Simplified and Traditional) translations (#371, closes #344, #349, #360)
- Frontend test infrastructure with Vitest (#371)
- Language toggle component for switching UI language (#371)
- Date localization using date-fns locales (#371)
- Error message translation system (#371)
### Fixed
- Accessibility improvements: added missing `id`, `name`, and `autoComplete` attributes to form inputs (#371)
- Added `DialogDescription` to dialogs for Radix UI accessibility compliance (#371)
- Fixed "Collapsible is changing from uncontrolled to controlled" warning in SettingsForm (#371)
- Fixed lint command for Next.js 16 compatibility (`eslint` instead of `next lint`)
### Changed
- Dockerfile optimizations: better layer caching, `--no-install-recommends` for smaller images (#371)
- Dockerfile.single refactored into 3 separate build stages for better caching (#371)
## [1.4.0] - 2026-01-14
### Added
- CTA button to empty state notebook list for better onboarding (#408)
- Offline deployment support for Docker containers (#414)
### Fixed
- Large file uploads (>10MB) by upgrading to Next.js 16 (#423)
- Orphaned uploaded files when sources are removed (#421)
- Broken documentation links to ai-providers.md (#419)
- ZIP support indication removed from UI (#418)
- Duplicate Claude Code workflow runs on PRs (#417)
- Claude Code review workflow now runs on PRs from forks (#416)
### Changed
- Upgraded Next.js from 15.4.10 to 16.1.1 (#423)
- Upgraded React from 19.1.0 to 19.2.3 (#423)
- Renamed `middleware.ts` to `proxy.ts` for Next.js 16 compatibility (#423)
### Dependencies
- next: 15.4.10 → 16.1.1
- react: 19.1.0 → 19.2.3
- react-dom: 19.1.0 → 19.2.3
## [1.2.4] - 2025-12-14
### Added
- Infinite scroll for notebook sources - no more 50 source limit (#325)
- Markdown table rendering in chat responses, search results, and insights (#325)
### Fixed
- Timeout errors with Ollama and local LLMs - increased to 10 minutes (#325)
- "Unable to Connect to API Server" on Docker startup - frontend now waits for API health check (#325, #315)
- SSL issues with langchain (#274)
- Query key consistency for source mutations to properly refresh infinite scroll (#325)
- Docker compose start-all flow (#323)
### Changed
- Timeout configuration now uses granular httpx.Timeout (short connect, long read) (#325)
### Dependencies
- Updated next.js to 15.4.10
- Updated httpx to >=0.27.0 for SSL fix
CLAUDE.md
+1
View File
@@ -0,0 +1 @@
@AGENTS.md
CODE_OF_CONDUCT.md
+134
View File
@@ -0,0 +1,134 @@
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, caste, color, religion, or sexual
identity and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our
community include:
- Demonstrating empathy and kindness toward other people
- Being respectful of differing opinions, viewpoints, and experiences
- Giving and gracefully accepting constructive feedback
- Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
- Focusing on what is best not just for us as individuals, but for the overall
community
Examples of unacceptable behavior include:
- The use of sexualized language or imagery, and sexual attention or advances of
any kind
- Trolling, insulting or derogatory comments, and personal or political attacks
- Public or private harassment
- Publishing others' private information, such as a physical or email address,
without their explicit permission
- Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.
Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official email address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement through the
contact form at **[open-notebook.ai](https://www.open-notebook.ai)** or via our
**[Discord community](https://discord.gg/37XJPXfz2w)**.
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series of
actions.
**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or permanent
ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within the
community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.1, available at
[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
Community Impact Guidelines were inspired by
[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
For answers to common questions about this code of conduct, see the FAQ at
[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
[https://www.contributor-covenant.org/translations][translations].
[homepage]: https://www.contributor-covenant.org
[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
[Mozilla CoC]: https://github.com/mozilla/diversity
[FAQ]: https://www.contributor-covenant.org/faq
[translations]: https://www.contributor-covenant.org/translations
CONFIGURATION.md
+35
View File
@@ -0,0 +1,35 @@
# Configuration Guide
**📍 This file has moved!**
All configuration documentation has been consolidated into the new documentation structure.
👉 **[Read the Configuration Guide](docs/5-CONFIGURATION/index.md)**
---
## Quick Links
- **AI Provider Setup** → [AI Providers](docs/5-CONFIGURATION/ai-providers.md)
- **Environment Variables Reference** → [Environment Reference](docs/5-CONFIGURATION/environment-reference.md)
- **Database Configuration** → [Database Setup](docs/5-CONFIGURATION/database.md)
- **Security Setup** → [Security Configuration](docs/5-CONFIGURATION/security.md)
- **Reverse Proxy** → [Reverse Proxy Setup](docs/5-CONFIGURATION/reverse-proxy.md)
- **Advanced Tuning** → [Advanced Configuration](docs/5-CONFIGURATION/advanced.md)
---
## What You'll Find
The new configuration documentation includes:
- **Complete environment variable reference** with examples
- **Provider-specific setup guides** for OpenAI, Anthropic, Google, Groq, Ollama, and more
- **Production deployment configurations** with security best practices
- **Reverse proxy examples** for Nginx, Caddy, Traefik
- **Database tuning** for performance optimization
- **Troubleshooting guides** for common configuration issues
---
For all configuration details, see **[docs/5-CONFIGURATION/](docs/5-CONFIGURATION/index.md)**.
CONTRIBUTING.md
+32
View File
@@ -0,0 +1,32 @@
# Contributing to Open Notebook
**📍 This file has moved!**
All contribution guidelines have been consolidated into the new development documentation structure.
👉 **[Read the Contributing Guide](docs/7-DEVELOPMENT/contributing.md)**
---
## Quick Links
- **Want to contribute code?** → [Contributing Guide](docs/7-DEVELOPMENT/contributing.md)
- **Want to understand the architecture?** → [Architecture Overview](docs/7-DEVELOPMENT/architecture.md)
- **Want to understand what we're building?** → [Vision & Principles](VISION.md)
- **Want to understand our engineering practices?** → [Design Principles](docs/7-DEVELOPMENT/design-principles.md)
- **Are you a maintainer?** → [Maintainer Guide](docs/7-DEVELOPMENT/maintainer-guide.md)
- **New developer?** → [Quick Start](docs/7-DEVELOPMENT/quick-start.md)
---
## The Issue-First Workflow
**TL;DR**: For anything non-trivial — features, architecture changes — create an issue first, get
it assigned, THEN code. Small obvious fixes (typos, docs, tiny bugs) can go straight to a PR.
Already coded something bigger? Mark the PR as draft and open the issue — triage takes 1–2 days.
This prevents wasted effort and ensures your work aligns with the project. [See details →](docs/7-DEVELOPMENT/contributing.md)
---
For all contribution details, see **[docs/7-DEVELOPMENT/contributing.md](docs/7-DEVELOPMENT/contributing.md)**.
Dockerfile
+154
View File
@@ -0,0 +1,154 @@
# Single source of truth for both published image variants:
# - regular (multi-container, SurrealDB external): default build / --target runtime
# - single-container (app + SurrealDB): --target single
# Shared stages below guarantee that fixes (tiktoken pre-cache, env defaults,
# npm retry logic, ...) apply to both variants at once.
# Stage 1: Frontend builder
FROM node:22-slim AS frontend-builder
WORKDIR /app/frontend
# Copy dependency files first to leverage cache
COPY frontend/package.json frontend/package-lock.json ./
ARG NPM_REGISTRY=https://registry.npmjs.org/
RUN npm config set registry ${NPM_REGISTRY} \
&& npm config set fetch-retries 5 \
&& npm config set fetch-retry-mintimeout 20000 \
&& npm config set fetch-retry-maxtimeout 120000
# Retry npm ci to survive transient registry ECONNRESETs, which are common on
# the QEMU-emulated arm64 leg of the multi-arch build.
RUN i=0; until npm ci; do \
i=$((i+1)); \
if [ "$i" -ge 5 ]; then echo "npm ci failed after $i attempts"; exit 1; fi; \
echo "npm ci failed (attempt $i); retrying in 15s"; sleep 15; \
done
# Copy the rest of the frontend source and build
COPY frontend/ ./
RUN npm run build
# Stage 2: Backend builder
FROM python:3.12-slim-trixie AS backend-builder
# Install build dependencies (uv downloads pre-built wheels for most packages)
RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-recommends \
build-essential \
&& rm -rf /var/lib/apt/lists/*
# Install uv using the official method
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
WORKDIR /app
# Set build optimization environment variables
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
ENV UV_COMPILE_BYTECODE=1
ENV UV_LINK_MODE=copy
ENV UV_HTTP_TIMEOUT=120
# Copy dependency files and minimal package structure first for better layer caching
COPY pyproject.toml uv.lock ./
COPY open_notebook/__init__.py ./open_notebook/__init__.py
# Install dependencies (this layer is cached unless dependencies change)
RUN uv sync --frozen --no-dev
# Pre-download tiktoken encoding so the app works offline (issue #264).
# /app/tiktoken-cache is intentionally outside /app/data/ so that volume mounts
# of /app/data (for user data persistence) do not hide the pre-baked encoding.
# config.py reads TIKTOKEN_CACHE_DIR from the environment to pick up this path.
ENV TIKTOKEN_CACHE_DIR=/app/tiktoken-cache
RUN mkdir -p /app/tiktoken-cache && \
.venv/bin/python -c "import tiktoken; tiktoken.get_encoding('o200k_base')"
# Stage 3: SurrealDB binary (pinned to v2 to match docker-compose.yml; used by the single target only)
FROM surrealdb/surrealdb:v2 AS surreal-binary
# Stage 4: Shared runtime base (everything common to both variants)
FROM python:3.12-slim-trixie AS runtime-base
# Install only runtime system dependencies (no build tools)
# Add Node.js 22.x LTS for running the frontend
RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-recommends \
ffmpeg \
supervisor \
curl \
&& curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
&& apt-get install -y --no-install-recommends nodejs \
&& rm -rf /var/lib/apt/lists/*
# Install uv using the official method
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
WORKDIR /app
# Copy the virtual environment from the backend builder
COPY --from=backend-builder /app/.venv /app/.venv
# Install the Chromium runtime for the Crawl4AI URL engine so its local,
# no-API-key mode works out of the box. Uses the playwright bundled in the venv
# (via content-core[crawl4ai]); --with-deps pulls the required system libraries.
# Pinned to a fixed path so it is found at runtime and survives volume mounts.
ENV PLAYWRIGHT_BROWSERS_PATH=/app/pw-browsers
RUN .venv/bin/python -m playwright install --with-deps chromium \
&& rm -rf /var/lib/apt/lists/*
# Copy the source code
COPY . /app
# Copy pre-downloaded tiktoken encoding from builder (outside /data/ — volume-mount safe)
COPY --from=backend-builder /app/tiktoken-cache /app/tiktoken-cache
# Copy built frontend from standalone output
COPY --from=frontend-builder /app/frontend/.next/standalone /app/frontend/
COPY --from=frontend-builder /app/frontend/.next/static /app/frontend/.next/static
COPY --from=frontend-builder /app/frontend/public /app/frontend/public
COPY --from=frontend-builder /app/frontend/start-server.js /app/frontend/start-server.js
# Ensure uv uses the existing venv without attempting network operations
ENV UV_NO_SYNC=1
ENV VIRTUAL_ENV=/app/.venv
# Point the app at the pre-baked tiktoken encoding (see open_notebook/config.py)
ENV TIKTOKEN_CACHE_DIR=/app/tiktoken-cache
# Bind the API to all interfaces (IPv4). Set API_HOST=:: for IPv6 dual-stack environments
ENV API_HOST=0.0.0.0
# Data directory (volume-mounted by users) and supervisor log directory
RUN mkdir -p /app/data /var/log/supervisor \
&& chmod +x /app/scripts/wait-for-api.sh
# Copy supervisord configuration (shared programs: api, worker, frontend)
COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf
# Expose ports for Frontend and API
EXPOSE 8502 5055
# Runtime API URL Configuration
# The API_URL environment variable can be set at container runtime to configure
# where the frontend should connect to the API. This allows the same Docker image
# to work in different deployment scenarios without rebuilding.
#
# If not set, the system will auto-detect based on incoming requests.
# Set API_URL when using reverse proxies or custom domains.
#
# Example: docker run -e API_URL=https://your-domain.com/api ...
CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf"]
# Stage 5: Single-container variant (adds SurrealDB on top of the shared runtime)
# Build with: docker build --target single .
FROM runtime-base AS single
# Install SurrealDB (copied from pinned v2 image to match docker-compose.yml)
COPY --from=surreal-binary /surreal /usr/local/bin/surreal
# SurrealDB data directory (volume-mounted by users)
RUN mkdir -p /mydata
# Enable the surrealdb program in supervisord (appended to the shared config)
RUN cat /app/supervisord.surrealdb.conf >> /etc/supervisor/conf.d/supervisord.conf
# Stage 6 (default): Regular multi-container image (SurrealDB runs externally).
# Kept last so a plain `docker build .` produces this variant.
FROM runtime-base AS runtime
LICENSE
+17
View File
@@ -0,0 +1,17 @@
MIT License
Copyright (c) 2024 Luis Novo
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
MAINTAINER_GUIDE.md
+19
View File
@@ -0,0 +1,19 @@
# Maintainer Guide
**📍 This file has moved!**
All maintainer guidelines have been consolidated into the new development documentation structure.
👉 **[Read the Maintainer Guide](docs/7-DEVELOPMENT/maintainer-guide.md)**
---
## Quick Links
- **Maintainer Guide** → [docs/7-DEVELOPMENT/maintainer-guide.md](docs/7-DEVELOPMENT/maintainer-guide.md)
- **Contributing Guide** → [docs/7-DEVELOPMENT/contributing.md](docs/7-DEVELOPMENT/contributing.md)
- **Design Principles** → [docs/7-DEVELOPMENT/design-principles.md](docs/7-DEVELOPMENT/design-principles.md)
---
For all maintainer details, see **[docs/7-DEVELOPMENT/maintainer-guide.md](docs/7-DEVELOPMENT/maintainer-guide.md)**.
Makefile
+230
View File
@@ -0,0 +1,230 @@
.PHONY: run frontend check ruff database lint api start-all stop-all status clean-cache worker worker-start worker-stop worker-restart
.PHONY: docker-buildx-prepare docker-buildx-clean docker-buildx-reset
.PHONY: docker-push docker-push-latest docker-release docker-build-local tag export-docs
.PHONY: release-test release-stack release-stack-down
# Get version from pyproject.toml
VERSION := $(shell grep -m1 version pyproject.toml | cut -d'"' -f2)
# Image names for both registries
DOCKERHUB_IMAGE := lfnovo/open_notebook
GHCR_IMAGE := ghcr.io/lfnovo/open-notebook
# Build platforms
PLATFORMS := linux/amd64,linux/arm64
database:
docker compose up -d surrealdb
run:
@echo "⚠️ Warning: Starting frontend only. For full functionality, use 'make start-all'"
cd frontend && npm run dev
frontend:
cd frontend && npm run dev
lint:
uv run python -m mypy .
ruff:
ruff check . --fix
# === Docker Build Setup ===
docker-buildx-prepare:
@docker buildx inspect multi-platform-builder >/dev/null 2>&1 || \
docker buildx create --use --name multi-platform-builder --driver docker-container
@docker buildx use multi-platform-builder
docker-buildx-clean:
@echo "🧹 Cleaning up buildx builders..."
@docker buildx rm multi-platform-builder 2>/dev/null || true
@docker ps -a | grep buildx_buildkit | awk '{print $$1}' | xargs -r docker rm -f 2>/dev/null || true
@echo "✅ Buildx cleanup complete!"
docker-buildx-reset: docker-buildx-clean docker-buildx-prepare
@echo "✅ Buildx reset complete!"
# === Release Testing (see .github/RELEASE_PROCESS.md) ===
# Automated image gate: fresh install + upgrade against real images.
# Usage: make release-test TAG=1.12.0 OLD_TAG=1.11.0
release-test:
@test -n "$(TAG)" || (echo "usage: make release-test TAG=<new> [OLD_TAG=<previous>]"; exit 1)
bash scripts/release-test/release-image-test.sh all \
"$(DOCKERHUB_IMAGE):$(TAG)" \
$(if $(OLD_TAG),"$(DOCKERHUB_IMAGE):$(OLD_TAG)")
# Browsable RC stack for manual verification (optionally with a data dump).
# Usage: make release-stack TAG=1.12.0 [DUMP=/tmp/dev-dump.surql]
release-stack:
@test -n "$(TAG)" || (echo "usage: make release-stack TAG=<tag> [DUMP=<dump.surql>]"; exit 1)
bash scripts/release-test/rc-stack.sh up "$(TAG)" $(DUMP)
release-stack-down:
bash scripts/release-test/rc-stack.sh down "$(or $(TAG),unused)"
# === Docker Build Targets ===
# Build production image for local platform only (no push)
docker-build-local:
@echo "🔨 Building production image locally ($(shell uname -m))..."
docker build \
-t $(DOCKERHUB_IMAGE):$(VERSION) \
-t $(DOCKERHUB_IMAGE):local \
.
@echo "✅ Built $(DOCKERHUB_IMAGE):$(VERSION) and $(DOCKERHUB_IMAGE):local"
@echo "Run with: docker run -p 5055:5055 -p 3000:3000 $(DOCKERHUB_IMAGE):local"
# Build and push version tags ONLY (no latest) for both regular and single images
docker-push: docker-buildx-prepare
@echo "📤 Building and pushing version $(VERSION) to both registries..."
@echo "🔨 Building regular image..."
docker buildx build --pull \
--platform $(PLATFORMS) \
--progress=plain \
-t $(DOCKERHUB_IMAGE):$(VERSION) \
-t $(GHCR_IMAGE):$(VERSION) \
--push \
.
@echo "🔨 Building single-container image..."
docker buildx build --pull \
--platform $(PLATFORMS) \
--progress=plain \
--target single \
-t $(DOCKERHUB_IMAGE):$(VERSION)-single \
-t $(GHCR_IMAGE):$(VERSION)-single \
--push \
.
@echo "✅ Pushed version $(VERSION) to both registries (latest NOT updated)"
@echo " 📦 Docker Hub:"
@echo " - $(DOCKERHUB_IMAGE):$(VERSION)"
@echo " - $(DOCKERHUB_IMAGE):$(VERSION)-single"
@echo " 📦 GHCR:"
@echo " - $(GHCR_IMAGE):$(VERSION)"
@echo " - $(GHCR_IMAGE):$(VERSION)-single"
# Update v1-latest tags to current version (both regular and single images)
docker-push-latest: docker-buildx-prepare
@echo "📤 Updating v1-latest tags to version $(VERSION)..."
@echo "🔨 Building regular image with latest tag..."
docker buildx build --pull \
--platform $(PLATFORMS) \
--progress=plain \
-t $(DOCKERHUB_IMAGE):$(VERSION) \
-t $(DOCKERHUB_IMAGE):v1-latest \
-t $(GHCR_IMAGE):$(VERSION) \
-t $(GHCR_IMAGE):v1-latest \
--push \
.
@echo "🔨 Building single-container image with latest tag..."
docker buildx build --pull \
--platform $(PLATFORMS) \
--progress=plain \
--target single \
-t $(DOCKERHUB_IMAGE):$(VERSION)-single \
-t $(DOCKERHUB_IMAGE):v1-latest-single \
-t $(GHCR_IMAGE):$(VERSION)-single \
-t $(GHCR_IMAGE):v1-latest-single \
--push \
.
@echo "✅ Updated v1-latest to version $(VERSION)"
@echo " 📦 Docker Hub:"
@echo " - $(DOCKERHUB_IMAGE):$(VERSION) → v1-latest"
@echo " - $(DOCKERHUB_IMAGE):$(VERSION)-single → v1-latest-single"
@echo " 📦 GHCR:"
@echo " - $(GHCR_IMAGE):$(VERSION) → v1-latest"
@echo " - $(GHCR_IMAGE):$(VERSION)-single → v1-latest-single"
# Full release: push version AND update latest tags
docker-release: docker-push-latest
@echo "✅ Full release complete for version $(VERSION)"
tag:
@version=$$(grep '^version = ' pyproject.toml | sed 's/version = "\(.*\)"/\1/'); \
echo "Creating tag v$$version"; \
git tag "v$$version"; \
git push origin "v$$version"
dev:
docker compose -f examples/docker-compose-dev.yml --project-directory . up --build
full:
docker compose -f examples/docker-compose-full-local.yml --project-directory . up --build
api:
uv run --env-file .env run_api.py
.PHONY: worker worker-start worker-stop worker-restart
worker: worker-start
worker-start:
@echo "Starting surreal-commands worker..."
uv run --env-file .env surreal-commands-worker --import-modules commands
worker-stop:
@echo "Stopping surreal-commands worker..."
pkill -f "surreal-commands-worker" || true
worker-restart: worker-stop
@sleep 2
@$(MAKE) worker-start
# === Service Management ===
start-all:
@echo "🚀 Starting Open Notebook (Database + API + Worker + Frontend)..."
@echo "📊 Starting SurrealDB..."
@docker compose -f docker-compose.dev.yml up -d surrealdb
@sleep 3
@echo "🔧 Starting API backend..."
@uv run run_api.py &
@sleep 3
@echo "⚙️ Starting background worker..."
@uv run --env-file .env surreal-commands-worker --import-modules commands &
@sleep 2
@echo "🌐 Starting Next.js frontend..."
@echo "✅ All services started!"
@echo "📱 Frontend: http://localhost:3000"
@echo "🔗 API: http://localhost:5055"
@echo "📚 API Docs: http://localhost:5055/docs"
cd frontend && npm run dev
stop-all:
@echo "🛑 Stopping all Open Notebook services..."
@pkill -f "next dev" || true
@pkill -f "surreal-commands-worker" || true
@pkill -f "run_api.py" || true
@pkill -f "uvicorn api.main:app" || true
@docker compose down
@echo "✅ All services stopped!"
status:
@echo "📊 Open Notebook Service Status:"
@echo "Database (SurrealDB):"
@docker compose ps surrealdb 2>/dev/null || echo " ❌ Not running"
@echo "API Backend:"
@pgrep -f "run_api.py\|uvicorn api.main:app" >/dev/null && echo " ✅ Running" || echo " ❌ Not running"
@echo "Background Worker:"
@pgrep -f "surreal-commands-worker" >/dev/null && echo " ✅ Running" || echo " ❌ Not running"
@echo "Next.js Frontend:"
@pgrep -f "next dev" >/dev/null && echo " ✅ Running" || echo " ❌ Not running"
# === Documentation Export ===
export-docs:
@echo "📚 Exporting documentation..."
@uv run python scripts/export_docs.py
@echo "✅ Documentation export complete!"
# === Cleanup ===
clean-cache:
@echo "🧹 Cleaning cache directories..."
@find . -name "__pycache__" -type d -exec rm -rf {} + 2>/dev/null || true
@find . -name ".mypy_cache" -type d -exec rm -rf {} + 2>/dev/null || true
@find . -name ".ruff_cache" -type d -exec rm -rf {} + 2>/dev/null || true
@find . -name ".pytest_cache" -type d -exec rm -rf {} + 2>/dev/null || true
@find . -name "*.pyc" -type f -delete 2>/dev/null || true
@find . -name "*.pyo" -type f -delete 2>/dev/null || true
@find . -name "*.pyd" -type f -delete 2>/dev/null || true
@echo "✅ Cache directories cleaned!"
README.dev.md
+31
View File
@@ -0,0 +1,31 @@
# Developer Guide
**📍 This file has moved!**
Developer documentation now lives in the development docs structure.
👉 **[Read the Development Setup Guide](docs/7-DEVELOPMENT/development-setup.md)**
---
## Quick Links
- **Setting up your environment?** → [Development Setup](docs/7-DEVELOPMENT/development-setup.md) (includes the make-workflow matrix)
- **New developer?** → [Quick Start](docs/7-DEVELOPMENT/quick-start.md)
- **Want to contribute?** → [Contributing Guide](docs/7-DEVELOPMENT/contributing.md)
- **Making a common change?** → [Change Playbooks](docs/7-DEVELOPMENT/change-playbooks.md)
- **Publishing Docker images?** → [Release Process](.github/RELEASE_PROCESS.md)
- **Coding-agent rules?** → [AGENTS.md](AGENTS.md)
---
## TL;DR
```bash
git clone https://github.com/lfnovo/open-notebook.git && cd open-notebook
cp .env.example .env
uv sync
make start-all # SurrealDB + API + worker + frontend
```
For everything else, see **[docs/7-DEVELOPMENT/](docs/7-DEVELOPMENT/index.md)**.
README.md
+386
View File
@@ -0,0 +1,386 @@
<a id="readme-top"></a>
<!-- [![Contributors][contributors-shield]][contributors-url] -->
[![Forks][forks-shield]][forks-url]
[![Stargazers][stars-shield]][stars-url]
[![Issues][issues-shield]][issues-url]
[![MIT License][license-shield]][license-url]
<!-- [![LinkedIn][linkedin-shield]][linkedin-url] -->
<!-- PROJECT LOGO -->
<br />
<div align="center">
<a href="https://github.com/lfnovo/open-notebook">
<img src="docs/assets/hero.svg" alt="Logo">
</a>
<h3 align="center">Open Notebook</h3>
<p align="center">
An open source, privacy-focused alternative to Google's Notebook LM!
<br /><strong>Join our <a href="https://discord.gg/37XJPXfz2w">Discord server</a> for help, to share workflow ideas, and suggest features!</strong>
<br />
<a href="https://www.open-notebook.ai"><strong>Checkout our website »</strong></a>
<br />
<br />
<a href="docs/0-START-HERE/index.md">📚 Get Started</a>
·
<a href="docs/3-USER-GUIDE/index.md">📖 User Guide</a>
·
<a href="docs/2-CORE-CONCEPTS/index.md">✨ Features</a>
·
<a href="docs/1-INSTALLATION/index.md">🚀 Deploy</a>
</p>
</div>
<p align="center">
<a href="https://trendshift.io/repositories/14536" target="_blank"><img src="https://trendshift.io/api/badge/repositories/14536" alt="lfnovo%2Fopen-notebook | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>
<div align="center">
<!-- Keep these links. Translations will automatically update with the README. -->
<a href="https://zdoc.app/de/lfnovo/open-notebook">Deutsch</a> |
<a href="https://zdoc.app/es/lfnovo/open-notebook">Español</a> |
<a href="https://zdoc.app/fr/lfnovo/open-notebook">français</a> |
<a href="https://zdoc.app/ja/lfnovo/open-notebook">日本語</a> |
<a href="https://zdoc.app/ko/lfnovo/open-notebook">한국어</a> |
<a href="https://zdoc.app/pt/lfnovo/open-notebook">Português</a> |
<a href="https://zdoc.app/ru/lfnovo/open-notebook">Русский</a> |
<a href="https://zdoc.app/zh/lfnovo/open-notebook">中文</a>
</div>
## A private, multi-model, 100% local, full-featured alternative to Notebook LM
![New Notebook](docs/assets/asset_list.png)
In a world dominated by Artificial Intelligence, having the ability to think 🧠 and acquire new knowledge 💡, is a skill that should not be a privilege for a few, nor restricted to a single provider.
**Open Notebook empowers you to:**
- 🔒 **Control your data** - Keep your research private and secure
- 🤖 **Choose your AI models** - Support for 18+ providers including OpenAI, Anthropic, Ollama, LM Studio, and more
- 📚 **Organize multi-modal content** - PDFs, videos, audio, web pages, and more
- 🎙️ **Generate professional podcasts** - Advanced multi-speaker podcast generation
- 🔍 **Search intelligently** - Full-text and vector search across all your content
- 💬 **Chat with context** - AI conversations powered by your research
- 🌐 **Multi-language UI** - English, Portuguese, Chinese (Simplified & Traditional), Japanese, Russian, and Bengali support
Learn more about our project at [https://www.open-notebook.ai](https://www.open-notebook.ai)
---
## 🆚 Open Notebook vs Google Notebook LM
| Feature | Open Notebook | Google Notebook LM | Advantage |
|---------|---------------|--------------------|-----------|
| **Privacy & Control** | Self-hosted, your data | Google cloud only | Complete data sovereignty |
| **AI Provider Choice** | 18+ providers (OpenAI, Anthropic, Ollama, LM Studio, etc.) | Google models only | Flexibility and cost optimization |
| **Podcast Speakers** | 1-4 speakers with custom profiles | 2 speakers only | Extreme flexibility |
| **Content Transformations** | Custom and built-in | Limited options | Unlimited processing power |
| **API Access** | Full REST API | No API | Complete automation |
| **Deployment** | Docker, cloud, or local | Google hosted only | Deploy anywhere |
| **Citations** | Basic references (will improve) | Comprehensive with sources | Research integrity |
| **Customization** | Open source, fully customizable | Closed system | Unlimited extensibility |
| **Cost** | Pay only for AI usage | Free tier + Monthly subscription | Transparent and controllable |
**Why Choose Open Notebook?**
- 🔒 **Privacy First**: Your sensitive research stays completely private
- 💰 **Cost Control**: Choose cheaper AI providers or run locally with Ollama
- 🎙️ **Better Podcasts**: Full script control and multi-speaker flexibility vs limited 2-speaker deep-dive format
- 🔧 **Unlimited Customization**: Modify, extend, and integrate as needed
- 🌐 **No Vendor Lock-in**: Switch providers, deploy anywhere, own your data
### Built With
[![Python][Python]][Python-url] [![Next.js][Next.js]][Next-url] [![React][React]][React-url] [![SurrealDB][SurrealDB]][SurrealDB-url] [![LangChain][LangChain]][LangChain-url]
## 🚀 Quick Start (2 Minutes)
### Prerequisites
- [Docker Desktop](https://www.docker.com/products/docker-desktop/) installed
- That's it! (API keys configured later in the UI)
### Step 1: Get docker-compose.yml
**Option A:** Download directly
```bash
curl -o docker-compose.yml https://raw.githubusercontent.com/lfnovo/open-notebook/main/docker-compose.yml
```
**Option B:** Create the file manually
Copy this into a new file called `docker-compose.yml`:
```yaml
services:
surrealdb:
image: surrealdb/surrealdb:v2
# Credentials default to root:root for a zero-config local setup. Before
# exposing this instance to a network, set SURREAL_USER / SURREAL_PASSWORD
# in a .env file (see .env.example) — they are applied here and to the
# open_notebook service below, so the two always stay in sync.
# List (exec) form so each interpolated value stays a single argument —
# a password containing spaces would otherwise be split into several.
command: ["start", "--log", "info", "--user", "${SURREAL_USER:-root}", "--pass", "${SURREAL_PASSWORD:-root}", "rocksdb:/mydata/mydatabase.db"]
user: root # Required for bind mounts on Linux
ports:
# Bound to localhost only: the open_notebook service reaches this over
# the internal compose network regardless, so the host port is purely
# for local debugging (e.g. Surrealist, `surreal sql`). Exposing this
# on 0.0.0.0 would let anyone who can reach the host connect with the
# default root:root credentials.
- "127.0.0.1:8000:8000"
volumes:
- ./surreal_data:/mydata
environment:
- SURREAL_EXPERIMENTAL_GRAPHQL=true
restart: always
pull_policy: always
open_notebook:
image: lfnovo/open_notebook:v1-latest
ports:
- "8502:8502" # Web UI
- "5055:5055" # REST API
environment:
# REQUIRED: Change this to your own secret string
# This encrypts your API keys in the database
- OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
# Database connection. SURREAL_USER / SURREAL_PASSWORD default to root:root
# for local use; override them in a .env file before exposing the instance
# (the same values configure the surrealdb service above).
- SURREAL_URL=ws://surrealdb:8000/rpc
- SURREAL_USER=${SURREAL_USER:-root}
- SURREAL_PASSWORD=${SURREAL_PASSWORD:-root}
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=open_notebook
volumes:
- ./notebook_data:/app/data
depends_on:
- surrealdb
restart: always
pull_policy: always
```
### Step 2: Set Your Encryption Key
Edit `docker-compose.yml` and change this line:
```yaml
- OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
```
to any secret value (e.g., `my-super-secret-key-123`)
### Step 3: Start Services
```bash
docker compose up -d
```
Wait 15-20 seconds, then open: **http://localhost:8502**
### Step 4: Configure AI Provider
1. Go to **Models** and choose your provider (OpenAI, Anthropic, Google, etc.)
2. Click **+ Add Configuration**
3. Paste your API key and other info as needed and click **Add Configuration**
4. Click **Test** to test connection
5. Click **Sync Models** and check models to include
6. Under **Default Model Assignments**, click **Auto-Assign Defaults** or manually specify which models to use for what
Done! You're ready to create your first notebook.
> **Need an API key?** Get one from:
> [OpenAI](https://platform.openai.com/api-keys) · [Anthropic](https://console.anthropic.com/) · [Google](https://aistudio.google.com/) · [Groq](https://console.groq.com/) (free tier)
> **Want free local AI?** See [examples/docker-compose-ollama.yml](examples/) for Ollama setup
---
### 📚 More Installation Options
- **[With Ollama (Free Local AI)](examples/docker-compose-ollama.yml)** - Run models locally without API costs
- **[From Source (Developers)](docs/1-INSTALLATION/from-source.md)** - For development and contributions
- **[Complete Installation Guide](docs/1-INSTALLATION/index.md)** - All deployment scenarios
---
### 📖 Need Help?
- **🤖 AI Installation Assistant**: [CustomGPT to help you install](https://chatgpt.com/g/g-68776e2765b48191bd1bae3f30212631-open-notebook-installation-assistant)
- **🆘 Troubleshooting**: [5-minute troubleshooting guide](docs/6-TROUBLESHOOTING/quick-fixes.md)
- **💬 Community Support**: [Discord Server](https://discord.gg/37XJPXfz2w)
- **🐛 Report Issues**: [GitHub Issues](https://github.com/lfnovo/open-notebook/issues)
---
## Star History
[![Star History Chart](https://api.star-history.com/svg?repos=lfnovo/open-notebook&type=date&legend=top-left)](https://www.star-history.com/#lfnovo/open-notebook&type=date&legend=top-left)
## Provider Support Matrix
Thanks to the [Esperanto](https://github.com/lfnovo/esperanto) library, we support this providers out of the box!
| Provider | LLM Support | Embedding Support | Speech-to-Text | Text-to-Speech |
|--------------|-------------|------------------|----------------|----------------|
| OpenAI | ✅ | ✅ | ✅ | ✅ |
| Anthropic | ✅ | ❌ | ❌ | ❌ |
| Groq | ✅ | ❌ | ✅ | ❌ |
| Google (GenAI) | ✅ | ✅ | ✅ | ✅ |
| Vertex AI | ✅ | ✅ | ❌ | ✅ |
| Ollama | ✅ | ✅ | ❌ | ❌ |
| Perplexity | ✅ | ❌ | ❌ | ❌ |
| ElevenLabs | ❌ | ❌ | ✅ | ✅ |
| Deepgram | ❌ | ❌ | ❌ | ✅ |
| Azure OpenAI | ✅ | ✅ | ✅ | ✅ |
| Mistral | ✅ | ✅ | ✅ | ✅ |
| DeepSeek | ✅ | ❌ | ❌ | ❌ |
| Voyage | ❌ | ✅ | ❌ | ❌ |
| xAI | ✅ | ❌ | ❌ | ✅ |
| OpenRouter | ✅ | ✅ | ❌ | ❌ |
| DashScope (Qwen) | ✅ | ❌ | ❌ | ❌ |
| MiniMax | ✅ | ❌ | ❌ | ❌ |
| OpenAI Compatible* | ✅ | ✅ | ✅ | ✅ |
*Supports LM Studio and any OpenAI-compatible endpoint
## ✨ Key Features
### Core Capabilities
- **🔒 Privacy-First**: Your data stays under your control - no cloud dependencies
- **🎯 Multi-Notebook Organization**: Manage multiple research projects seamlessly
- **📚 Universal Content Support**: PDFs, videos, audio, web pages, Office docs, and more
- **🤖 Multi-Model AI Support**: 18+ providers including OpenAI, Anthropic, Ollama, Google, LM Studio, and more
- **🎙️ Professional Podcast Generation**: Advanced multi-speaker podcasts with Episode Profiles
- **🔍 Intelligent Search**: Full-text and vector search across all your content
- **💬 Context-Aware Chat**: AI conversations powered by your research materials
- **📝 AI-Assisted Notes**: Generate insights or write notes manually
### Advanced Features
- **⚡ Reasoning Model Support**: Full support for thinking models like DeepSeek-R1 and Qwen3
- **🔧 Content Transformations**: Powerful customizable actions to summarize and extract insights
- **🌐 Comprehensive REST API**: Full programmatic access for custom integrations [![API Docs](https://img.shields.io/badge/API-Documentation-blue?style=flat-square)](http://localhost:5055/docs)
- **🔐 Optional Password Protection**: Secure public deployments with authentication
- **📊 Fine-Grained Context Control**: Choose exactly what to share with AI models
- **📎 Citations**: Get answers with proper source citations
## Podcast Feature
[![Check out our podcast sample](https://img.youtube.com/vi/D-760MlGwaI/0.jpg)](https://www.youtube.com/watch?v=D-760MlGwaI)
## 📚 Documentation
### Getting Started
- **[📖 Introduction](docs/0-START-HERE/index.md)** - Learn what Open Notebook offers
- **[⚡ Quick Start with OpenAI](docs/0-START-HERE/quick-start-openai.md)** - Get up and running in 5 minutes
- **[🔧 Installation](docs/1-INSTALLATION/index.md)** - Comprehensive setup guide
- **[🎯 Run It Fully Local](docs/0-START-HERE/quick-start-local.md)** - Ollama/LM Studio, completely private
### User Guide
- **[📱 Interface Overview](docs/3-USER-GUIDE/interface-overview.md)** - Understanding the layout
- **[📚 Notebooks, Sources & Notes](docs/2-CORE-CONCEPTS/notebooks-sources-notes.md)** - Organizing your research
- **[📄 Adding Sources](docs/3-USER-GUIDE/adding-sources.md)** - Managing content types
- **[📝 Working with Notes](docs/3-USER-GUIDE/working-with-notes.md)** - Creating and managing notes
- **[💬 Chatting Effectively](docs/3-USER-GUIDE/chat-effectively.md)** - AI conversations
- **[🔍 Search](docs/3-USER-GUIDE/search.md)** - Finding information
### Advanced Topics
- **[🎙️ Podcast Generation](docs/2-CORE-CONCEPTS/podcasts-explained.md)** - Create professional podcasts
- **[🔧 Content Transformations](docs/3-USER-GUIDE/transformations.md)** - Customize content processing
- **[🤖 AI Models](docs/4-AI-PROVIDERS/index.md)** - AI model configuration
- **[🔌 MCP Integration](docs/5-CONFIGURATION/mcp-integration.md)** - Connect with Claude Desktop, VS Code and other MCP clients
- **[🔧 REST API Reference](docs/7-DEVELOPMENT/api-reference.md)** - Complete API documentation
- **[🔐 Security](docs/5-CONFIGURATION/security.md)** - Password protection and privacy
- **[🚀 Deployment](docs/1-INSTALLATION/index.md)** - Complete deployment guides for all scenarios
- **[🧭 Vision & Principles](VISION.md)** - What Open Notebook is, and where it's going
- **[🛠️ Developer Docs](docs/7-DEVELOPMENT/index.md)** - Architecture, setup, contributing, decision records
<p align="right">(<a href="#readme-top">back to top</a>)</p>
## 🗺️ Roadmap
### Upcoming Features
- **Live Front-End Updates**: Real-time UI updates for smoother experience
- **Async Processing**: Faster UI through asynchronous content processing
- **Cross-Notebook Sources**: Reuse research materials across projects
- **Bookmark Integration**: Connect with your favorite bookmarking apps
### Recently Completed ✅
- **Next.js Frontend**: Modern React-based frontend with improved performance
- **Comprehensive REST API**: Full programmatic access to all functionality
- **Multi-Model Support**: 18+ AI providers including OpenAI, Anthropic, Ollama, LM Studio
- **Advanced Podcast Generator**: Professional multi-speaker podcasts with Episode Profiles
- **Content Transformations**: Powerful customizable actions for content processing
- **Enhanced Citations**: Improved layout and finer control for source citations
- **Multiple Chat Sessions**: Manage different conversations within notebooks
See the [open issues](https://github.com/lfnovo/open-notebook/issues) for a full list of proposed features and known issues.
<p align="right">(<a href="#readme-top">back to top</a>)</p>
## 📖 Need Help?
- **🤖 AI Installation Assistant**: We have a [CustomGPT built to help you install Open Notebook](https://chatgpt.com/g/g-68776e2765b48191bd1bae3f30212631-open-notebook-installation-assistant) - it will guide you through each step!
- **New to Open Notebook?** Start with our [Getting Started Guide](docs/0-START-HERE/index.md)
- **Need installation help?** Check our [Installation Guide](docs/1-INSTALLATION/index.md)
- **Want to see it in action?** Try our [Quick Start Tutorial](docs/0-START-HERE/index.md)
## 🤝 Community & Contributing
### Join the Community
- 💬 **[Discord Server](https://discord.gg/37XJPXfz2w)** - Get help, share ideas, and connect with other users
- 🐛 **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Report bugs and request features
- ⭐ **Star this repo** - Show your support and help others discover Open Notebook
### Contributing
We welcome contributions! We're especially looking for help with:
- **Frontend Development**: Help improve our modern Next.js/React UI
- **Testing & Bug Fixes**: Make Open Notebook more robust
- **Feature Development**: Build the coolest research tool together
- **Documentation**: Improve guides and tutorials
**Current Tech Stack**: Python, FastAPI, Next.js, React, SurrealDB
**Future Roadmap**: Real-time updates, enhanced async processing
See our [Contributing Guide](CONTRIBUTING.md) for detailed information on how to get started, including our guidelines for [AI-assisted contributions](docs/7-DEVELOPMENT/contributing.md#ai-assisted-and-agent-generated-prs). To understand what we're building (and what we'll say no to), read [VISION.md](VISION.md).
<p align="right">(<a href="#readme-top">back to top</a>)</p>
## 📄 License
Open Notebook is MIT licensed. See the [LICENSE](LICENSE) file for details.
**Community Support**:
- 💬 [Discord Server](https://discord.gg/37XJPXfz2w) - Get help, share ideas, and connect with users
- 🐛 [GitHub Issues](https://github.com/lfnovo/open-notebook/issues) - Report bugs and request features
- 🌐 [Website](https://www.open-notebook.ai) - Learn more about the project
<p align="right">(<a href="#readme-top">back to top</a>)</p>
<!-- MARKDOWN LINKS & IMAGES -->
<!-- https://www.markdownguide.org/basic-syntax/#reference-style-links -->
[contributors-shield]: https://img.shields.io/github/contributors/lfnovo/open-notebook.svg?style=for-the-badge
[contributors-url]: https://github.com/lfnovo/open-notebook/graphs/contributors
[forks-shield]: https://img.shields.io/github/forks/lfnovo/open-notebook.svg?style=for-the-badge
[forks-url]: https://github.com/lfnovo/open-notebook/network/members
[stars-shield]: https://img.shields.io/github/stars/lfnovo/open-notebook.svg?style=for-the-badge
[stars-url]: https://github.com/lfnovo/open-notebook/stargazers
[issues-shield]: https://img.shields.io/github/issues/lfnovo/open-notebook.svg?style=for-the-badge
[issues-url]: https://github.com/lfnovo/open-notebook/issues
[license-shield]: https://img.shields.io/github/license/lfnovo/open-notebook.svg?style=for-the-badge
[license-url]: https://github.com/lfnovo/open-notebook/blob/master/LICENSE.txt
[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=for-the-badge&logo=linkedin&colorB=555
[linkedin-url]: https://linkedin.com/in/lfnovo
[product-screenshot]: images/screenshot.png
[Next.js]: https://img.shields.io/badge/Next.js-000000?style=for-the-badge&logo=next.js&logoColor=white
[Next-url]: https://nextjs.org/
[React]: https://img.shields.io/badge/React-61DAFB?style=for-the-badge&logo=react&logoColor=black
[React-url]: https://reactjs.org/
[Python]: https://img.shields.io/badge/Python-3776AB?style=for-the-badge&logo=python&logoColor=white
[Python-url]: https://www.python.org/
[LangChain]: https://img.shields.io/badge/LangChain-3A3A3A?style=for-the-badge&logo=chainlink&logoColor=white
[LangChain-url]: https://www.langchain.com/
[SurrealDB]: https://img.shields.io/badge/SurrealDB-FF5E00?style=for-the-badge&logo=databricks&logoColor=white
[SurrealDB-url]: https://surrealdb.com/
README.wehub.md
+7
View File
@@ -0,0 +1,7 @@
# WeHub 来源说明
- 原始项目：`lfnovo/open-notebook`
- 原始仓库：https://github.com/lfnovo/open-notebook
- 导入方式：上游默认分支的最新快照
- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
- 本文件仅用于记录来源，不代表 WeHub 是原项目作者
SECURITY.md
+65
View File
@@ -0,0 +1,65 @@
# Security Policy
## Supported Versions
Open Notebook is an actively developed project. Security fixes are applied to the
**latest released version** only; there are no long-term support branches.
| Version | Supported |
| ------- | ------------------ |
| Latest release (`1.x`, current minor) | :white_check_mark: |
| Older releases | :x: |
If you are running an older version, please upgrade to the latest release before
reporting an issue — the problem may already be fixed.
## Reporting a Vulnerability
**Please do not report security vulnerabilities through public GitHub issues,
discussions, or pull requests.**
Instead, report them privately through GitHub's built-in **private vulnerability
reporting**:
1. Go to the [Security tab](https://github.com/lfnovo/open-notebook/security) of
the repository.
2. Click **"Report a vulnerability"**.
3. Fill out the form with as much detail as you can.
This keeps the report private between you and the maintainers until a fix is
available.
When reporting, please include where relevant:
- A description of the vulnerability and its impact.
- Steps to reproduce (a proof of concept, affected endpoint/component, or sample
configuration).
- The Open Notebook version and how you are running it (Docker Compose,
single-container, from source).
- Any suggested remediation, if you have one.
## What to Expect
- **Acknowledgement:** we aim to acknowledge a report within **5 business days**.
- **Assessment:** we will investigate, confirm the issue, and determine the
affected versions.
- **Fix & disclosure:** once a fix is ready we will release it and, with your
consent, credit you in the release notes. We follow a coordinated-disclosure
approach and ask that you keep the report private until a fix is published.
## Scope
Open Notebook is **self-hosted**: you run the API, frontend, and SurrealDB
yourself, and you control the AI provider credentials. Please keep in mind:
- The built-in password middleware (`OPEN_NOTEBOOK_PASSWORD`) is a basic access
control, not a full authentication system. See
[docs/5-CONFIGURATION/security.md](docs/5-CONFIGURATION/security.md) for
hardening guidance (encryption key, reverse proxy, CORS, default credentials).
- Misconfiguration of your own deployment (e.g. exposing SurrealDB with default
credentials, or running without `OPEN_NOTEBOOK_ENCRYPTION_KEY`) is a
deployment concern covered by that hardening guide rather than a vulnerability
in the project — though we welcome reports where the defaults or docs actively
steer users toward an insecure setup.
Thank you for helping keep Open Notebook and its users safe.
VISION.md
+79
View File
@@ -0,0 +1,79 @@
# Open Notebook — Vision & Principles
This document is the product's source of truth in two layers with different lifespans: **Identity** (durable — what Open Notebook is and refuses to be) and **Current Posture** (temporal — where we are in the journey and what's on the horizon). Triage and design decisions are evaluated against this document; the reasoning behind each rule lives in the [decision records](docs/7-DEVELOPMENT/decisions/README.md).
---
## Identity
Open Notebook is a **privacy-focused, self-hosted alternative to Google's Notebook LM** that empowers users to:
1. **Own their research data** — full control over where data lives and who can access it
2. **Choose their AI providers** — any provider, or fully local models
3. **Customize their workflows** — adapt the tool to different research needs
4. **Access their work anywhere** — web UI, API, or integrations
### What Open Notebook IS
- A **research assistant** for managing and understanding content
- A **platform** that connects various AI providers
- A **privacy-first** tool that keeps your data under your control
- An **extensible system** with APIs and customization options
### What Open Notebook IS NOT
- A document editor (use Google Docs, Notion, etc.)
- A file storage system (use Dropbox, S3, etc.)
- A general-purpose chatbot (use ChatGPT, Claude, etc.)
- A replacement for your entire workflow (it's one tool in your toolkit)
### Principles
Durable, normative rules. Each links to the decision record that established it.
| Principle | Rule |
|---|---|
| **Privacy first** | User data stays under user control by default. Self-hosted is the primary use case; no telemetry without opt-in; no hard dependency on specific cloud services. |
| **Provider-agnostic core** | The default is portable: features must work across the provider matrix. Adopting a provider-exclusive capability is allowed but is a deliberate decision that requires a [PDR](docs/7-DEVELOPMENT/decisions/README.md) ([PDR-002](docs/7-DEVELOPMENT/decisions/PDR-002-provider-agnostic-core.md)). |
| **Simplicity over features** | Easy to understand and use, even if it means fewer features. Sensible defaults; advanced options behind progressive disclosure. |
| **API-first** | Every capability is accessible via the REST API — the UI is a client, never the only door ([ADR-003](docs/7-DEVELOPMENT/decisions/ADR-003-streamlit-to-nextjs.md)). |
| **Extensibility through standards** | Extension happens through well-defined interfaces (transformations, commands, prompt templates), not forks. |
| **Async-first** | Long-running operations never block the UI or the API ([ADR-004](docs/7-DEVELOPMENT/decisions/ADR-004-background-workers.md)). |
### How we evaluate requests
A feature request that conflicts with the IS NOT list or a principle gets closed with a pointer here — kindly, and with the reasoning. A "no" protects the core value proposition; it's not a judgment of the idea. If a request keeps coming back and the principle starts to feel wrong, that's a signal to revisit the principle through a decision record — not to make a quiet exception.
---
## Current Posture
> **Reviewed: 2026-07.** This section is expected to change. Updating it is not a reversal — it's a phase change, recorded with a short PDR and an edit here.
**The phase we're in: get the basics working well for everyone before expanding.** Priority goes to making the core experience (sources, chat, search, notes, podcasts) solid across the full provider matrix and deployment surface, ahead of new product surfaces.
### Directional constraints
Decisions about the future we haven't made yet — recorded as "which door to keep open":
- **Single-user first, multi-user compatible.** Open Notebook is a single-user tool today, but multi-user is under active consideration ([#712](https://github.com/lfnovo/open-notebook/issues/712)). New features must not gratuitously preclude multi-user (schema, auth, data scoping) ([PDR-001](docs/7-DEVELOPMENT/decisions/PDR-001-single-user-first.md)).
- **Portable by default.** Provider-exclusive capabilities (including paid-only ones) are on the table for the future — deliberately, via PDR, never by accident ([PDR-002](docs/7-DEVELOPMENT/decisions/PDR-002-provider-agnostic-core.md)).
### Horizon
The big clusters under consideration — direction, not roadmap; no dates. Each has an umbrella issue where the thinking happens:
| Cluster | What it is | Where |
|---|---|---|
| **Platform v-next** | SurrealDB v3 migration, possible frontend/backend Docker image split, possible Surreal Commands → Celery move — evaluated together as one coordinated breaking change | [#372](https://github.com/lfnovo/open-notebook/issues/372) · [#378](https://github.com/lfnovo/open-notebook/issues/378) · [#381](https://github.com/lfnovo/open-notebook/issues/381) |
| **Multi-user** | Deep platform redesign: auth, data scoping, what "multi-user" means for a self-hosted tool | [#712](https://github.com/lfnovo/open-notebook/issues/712) |
| **Content modes & artifacts** | The output side: generated artifacts, videos, explainers, presentations, mind maps — as one coherent product surface, not a pile of features | [#203](https://github.com/lfnovo/open-notebook/issues/203) |
| **Agents operating Open Notebook** | Role inversion via MCP: AI agents use Open Notebook on the user's behalf — the platform becomes the research memory of agents, not just a UI | [#878](https://github.com/lfnovo/open-notebook/issues/878) · [#693](https://github.com/lfnovo/open-notebook/issues/693) · [#973](https://github.com/lfnovo/open-notebook/issues/973) |
---
## How this document changes
- **Identity** changes rarely and deliberately: a decision record marks the old rule as superseded, then this document is updated.
- **Posture** changes when the phase changes: a short PDR captures the why, the section above is edited, and the "Reviewed" stamp is bumped.
- Engineering practices (code standards, anti-patterns, decision framework) live in [docs/7-DEVELOPMENT/design-principles.md](docs/7-DEVELOPMENT/design-principles.md).
api/AGENTS.md
+1
View File
@@ -0,0 +1 @@
@../open_notebook/AGENTS.md
api/CLAUDE.md
+1
View File
@@ -0,0 +1 @@
@AGENTS.md
api/__init__.py
View File
api/auth.py
+82
View File
@@ -0,0 +1,82 @@
import secrets
from typing import Optional
from fastapi import Request
from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
from starlette.responses import JSONResponse, Response
from starlette.types import ASGIApp
from open_notebook.utils.encryption import get_secret_from_env
class PasswordAuthMiddleware(BaseHTTPMiddleware):
"""
Middleware to check password authentication for all API requests.
Auth is fully disabled (no hardcoded default password) if
OPEN_NOTEBOOK_PASSWORD is not set.
Supports Docker secrets via OPEN_NOTEBOOK_PASSWORD_FILE.
"""
def __init__(
self, app: ASGIApp, excluded_paths: Optional[list[str]] = None
) -> None:
super().__init__(app)
self.password = get_secret_from_env("OPEN_NOTEBOOK_PASSWORD")
self.excluded_paths: list[str] = excluded_paths or [
"/",
"/health",
"/docs",
"/openapi.json",
"/redoc",
]
async def dispatch(
self, request: Request, call_next: RequestResponseEndpoint
) -> Response:
# Skip authentication if no password is set
if not self.password:
return await call_next(request)
# Skip authentication for excluded paths
if request.url.path in self.excluded_paths:
return await call_next(request)
# Skip authentication for CORS preflight requests (OPTIONS)
if request.method == "OPTIONS":
return await call_next(request)
# Check authorization header
auth_header = request.headers.get("Authorization")
if not auth_header:
return JSONResponse(
status_code=401,
content={"detail": "Missing authorization header"},
headers={"WWW-Authenticate": "Bearer"},
)
# Expected format: "Bearer {password}"
try:
scheme, credentials = auth_header.split(" ", 1)
if scheme.lower() != "bearer":
raise ValueError("Invalid authentication scheme")
except ValueError:
return JSONResponse(
status_code=401,
content={"detail": "Invalid authorization header format"},
headers={"WWW-Authenticate": "Bearer"},
)
# Check password (constant-time to avoid a timing side-channel)
if not secrets.compare_digest(
credentials.encode("utf-8"), self.password.encode("utf-8")
):
return JSONResponse(
status_code=401,
content={"detail": "Invalid password"},
headers={"WWW-Authenticate": "Bearer"},
)
# Password is correct, proceed with the request
response = await call_next(request)
return response
api/command_service.py
+92
View File
@@ -0,0 +1,92 @@
from typing import Any, Dict, List, Optional
from loguru import logger
from surreal_commands import get_command_status, submit_command
class CommandService:
"""Generic service layer for command operations"""
@staticmethod
async def submit_command_job(
module_name: str, # Actually app_name for surreal-commands
command_name: str,
command_args: Dict[str, Any],
context: Optional[Dict[str, Any]] = None,
) -> str:
"""Submit a generic command job for background processing"""
try:
# Ensure command modules are imported before submitting
# This is needed because submit_command validates against local registry
try:
import commands.podcast_commands # noqa: F401
except ImportError as import_err:
logger.error(f"Failed to import command modules: {import_err}")
raise ValueError("Command modules not available")
# surreal-commands expects: submit_command(app_name, command_name, args)
cmd_id = submit_command(
module_name, # This is actually the app name (e.g., "open_notebook")
command_name, # Command name (e.g., "generate_podcast")
command_args, # Input data
)
# Convert RecordID to string if needed
if not cmd_id:
raise ValueError("Failed to get cmd_id from submit_command")
cmd_id_str = str(cmd_id)
logger.info(
f"Submitted command job: {cmd_id_str} for {module_name}.{command_name}"
)
return cmd_id_str
except Exception as e:
logger.error(f"Failed to submit command job: {e}")
raise
@staticmethod
async def get_command_status(job_id: str) -> Dict[str, Any]:
"""Get status of any command job"""
try:
status = await get_command_status(job_id)
return {
"job_id": job_id,
"status": status.status if status else "unknown",
"result": status.result if status else None,
"error_message": getattr(status, "error_message", None)
if status
else None,
"created": str(status.created)
if status and hasattr(status, "created") and status.created
else None,
"updated": str(status.updated)
if status and hasattr(status, "updated") and status.updated
else None,
"progress": getattr(status, "progress", None) if status else None,
}
except Exception as e:
logger.error(f"Failed to get command status: {e}")
raise
@staticmethod
async def list_command_jobs(
module_filter: Optional[str] = None,
command_filter: Optional[str] = None,
status_filter: Optional[str] = None,
limit: int = 50,
) -> List[Dict[str, Any]]:
"""List command jobs with optional filtering"""
# This will be implemented with proper SurrealDB queries
# For now, return empty list as this is foundation phase
return []
@staticmethod
async def cancel_command_job(job_id: str) -> bool:
"""Cancel a running command job"""
try:
# Implementation depends on surreal-commands cancellation support
# For now, just log the attempt
logger.info(f"Attempting to cancel job: {job_id}")
return True
except Exception as e:
logger.error(f"Failed to cancel command job: {e}")
raise
api/credentials_service.py
+774
View File
@@ -0,0 +1,774 @@
"""
Credentials Service
Business logic for managing AI provider credentials.
Extracted from the credentials router to follow the service layer pattern.
All functions raise ValueError for business errors (router converts to HTTPException).
"""
import os
from typing import Dict, List
import httpx
from loguru import logger
from pydantic import SecretStr
from api.models import CredentialResponse
from open_notebook.ai.model_discovery import (
ANTHROPIC_FALLBACK_MODELS,
classify_model_type,
fetch_anthropic_model_ids,
)
from open_notebook.ai.provider_registry import PROVIDERS
from open_notebook.domain.credential import Credential
from open_notebook.utils.encryption import get_secret_from_env
from open_notebook.utils.url_validation import validate_url
# =============================================================================
# Constants
# =============================================================================
# Provider environment variable configuration, derived from the provider
# registry (open_notebook/ai/provider_registry.py — the source of truth).
# - "required": ALL listed env vars must be set for the provider to be considered configured.
# - "required_any": at least ONE of the listed env vars must be set.
# - "optional": additional env vars used during migration but not required.
PROVIDER_ENV_CONFIG: Dict[str, dict] = {
name: spec.env_config() for name, spec in PROVIDERS.items()
}
PROVIDER_MODALITIES: Dict[str, List[str]] = {
name: list(spec.modalities) for name, spec in PROVIDERS.items()
}
# =============================================================================
# Helpers
# =============================================================================
def require_encryption_key() -> None:
"""Raise ValueError if encryption key is not configured."""
if not get_secret_from_env("OPEN_NOTEBOOK_ENCRYPTION_KEY"):
raise ValueError(
"Encryption key not configured. "
"Set OPEN_NOTEBOOK_ENCRYPTION_KEY to enable storing API keys."
)
def credential_to_response(cred: Credential, model_count: int = 0) -> CredentialResponse:
"""Convert a Credential domain object to API response."""
return CredentialResponse(
id=cred.id or "",
name=cred.name,
provider=cred.provider,
modalities=cred.modalities,
base_url=cred.base_url,
endpoint=cred.endpoint,
api_version=cred.api_version,
endpoint_llm=cred.endpoint_llm,
endpoint_embedding=cred.endpoint_embedding,
endpoint_stt=cred.endpoint_stt,
endpoint_tts=cred.endpoint_tts,
project=cred.project,
location=cred.location,
credentials_path=cred.credentials_path,
num_ctx=cred.num_ctx,
has_api_key=cred.api_key is not None,
created=str(cred.created) if cred.created else "",
updated=str(cred.updated) if cred.updated else "",
model_count=model_count,
decryption_error=cred.decryption_error,
)
def check_env_configured(provider: str) -> bool:
"""Check if a provider has sufficient env vars configured for migration."""
config = PROVIDER_ENV_CONFIG.get(provider)
if not config:
return False
if "required_any" in config:
return any(bool(os.environ.get(v, "").strip()) for v in config["required_any"])
elif "required" in config:
return all(bool(os.environ.get(v, "").strip()) for v in config["required"])
return False
def get_default_modalities(provider: str) -> List[str]:
"""Get default modalities for a provider."""
return PROVIDER_MODALITIES.get(provider.lower(), ["language"])
def create_credential_from_env(provider: str) -> Credential:
"""Create a Credential from environment variables for a given provider."""
modalities = get_default_modalities(provider)
name = "Default (Migrated from env)"
if provider == "ollama":
return Credential(
name=name,
provider=provider,
modalities=modalities,
base_url=os.environ.get("OLLAMA_API_BASE"),
)
elif provider == "vertex":
return Credential(
name=name,
provider=provider,
modalities=modalities,
project=os.environ.get("VERTEX_PROJECT"),
location=os.environ.get("VERTEX_LOCATION"),
credentials_path=os.environ.get("GOOGLE_APPLICATION_CREDENTIALS"),
)
elif provider == "azure":
return Credential(
name=name,
provider=provider,
modalities=modalities,
api_key=SecretStr(os.environ["AZURE_OPENAI_API_KEY"]),
endpoint=os.environ.get("AZURE_OPENAI_ENDPOINT"),
api_version=os.environ.get("AZURE_OPENAI_API_VERSION"),
endpoint_llm=os.environ.get("AZURE_OPENAI_ENDPOINT_LLM"),
endpoint_embedding=os.environ.get("AZURE_OPENAI_ENDPOINT_EMBEDDING"),
endpoint_stt=os.environ.get("AZURE_OPENAI_ENDPOINT_STT"),
endpoint_tts=os.environ.get("AZURE_OPENAI_ENDPOINT_TTS"),
)
elif provider == "openai_compatible":
api_key = os.environ.get("OPENAI_COMPATIBLE_API_KEY")
return Credential(
name=name,
provider=provider,
modalities=modalities,
api_key=SecretStr(api_key) if api_key else None,
base_url=os.environ.get("OPENAI_COMPATIBLE_BASE_URL"),
)
elif provider == "google":
# Support both GOOGLE_API_KEY and GEMINI_API_KEY (fallback)
api_key = os.environ.get("GOOGLE_API_KEY") or os.environ.get("GEMINI_API_KEY")
return Credential(
name=name,
provider=provider,
modalities=modalities,
api_key=SecretStr(api_key) if api_key else None,
)
else:
# Simple API key providers
config = PROVIDER_ENV_CONFIG.get(provider, {})
required = config.get("required", [])
env_var = required[0] if required else None
api_key = os.environ.get(env_var) if env_var else None
return Credential(
name=name,
provider=provider,
modalities=modalities,
api_key=SecretStr(api_key) if api_key else None,
)
# =============================================================================
# Service Functions
# =============================================================================
async def get_provider_status() -> dict:
"""
Get configuration status: encryption key status, and per-provider
configured/source information.
"""
encryption_configured = bool(get_secret_from_env("OPEN_NOTEBOOK_ENCRYPTION_KEY"))
configured: Dict[str, bool] = {}
source: Dict[str, str] = {}
for provider in PROVIDER_ENV_CONFIG:
env_configured = check_env_configured(provider)
try:
db_credentials = await Credential.get_by_provider(provider)
db_configured = len(db_credentials) > 0
except Exception:
db_configured = False
configured[provider] = db_configured or env_configured
if db_configured:
source[provider] = "database"
elif env_configured:
source[provider] = "environment"
else:
source[provider] = "none"
return {
"configured": configured,
"source": source,
"encryption_configured": encryption_configured,
}
async def get_env_status() -> Dict[str, bool]:
"""Check what's configured via environment variables."""
env_status: Dict[str, bool] = {}
for provider in PROVIDER_ENV_CONFIG:
env_status[provider] = check_env_configured(provider)
return env_status
async def test_credential(credential_id: str) -> dict:
"""
Test connection using a credential's configuration.
Returns dict with provider, success, message keys.
"""
provider = "unknown"
try:
cred = await Credential.get(credential_id)
config = cred.to_esperanto_config()
from open_notebook.ai.connection_tester import (
_is_vertex_credentials_file_error,
_test_azure_connection,
_test_ollama_connection,
_test_openai_compatible_connection,
classify_provider_test_error,
)
provider = cred.provider.lower()
# Handle special providers
if provider == "ollama":
base_url = config.get("base_url", "http://localhost:11434")
success, message = await _test_ollama_connection(base_url)
return {"provider": provider, "success": success, "message": message}
if provider == "openai_compatible":
base_url = config.get("base_url")
api_key = config.get("api_key")
if not base_url:
return {
"provider": provider,
"success": False,
"message": "No base URL configured",
}
success, message = await _test_openai_compatible_connection(
base_url, api_key
)
return {"provider": provider, "success": success, "message": message}
if provider == "azure":
success, message = await _test_azure_connection(
endpoint=config.get("endpoint"),
api_key=config.get("api_key"),
api_version=config.get("api_version"),
)
return {"provider": provider, "success": success, "message": message}
# Standard provider: use Esperanto to create and test
from esperanto.factory import AIFactory
from open_notebook.ai.connection_tester import TEST_MODELS
if provider not in TEST_MODELS:
return {
"provider": provider,
"success": False,
"message": f"Unknown provider: {provider}",
}
test_model, test_type = TEST_MODELS[provider]
if not test_model:
return {
"provider": provider,
"success": False,
"message": f"No test model configured for {provider}",
}
if test_type == "language":
model = AIFactory.create_language(
model_name=test_model, provider=provider, config=config
)
lc_model = model.to_langchain()
await lc_model.ainvoke("Hi")
return {"provider": provider, "success": True, "message": "Connection successful"}
elif test_type == "embedding":
embedding_model = AIFactory.create_embedding(
model_name=test_model, provider=provider, config=config
)
await embedding_model.aembed(["test"])
return {"provider": provider, "success": True, "message": "Connection successful"}
elif test_type == "text_to_speech":
AIFactory.create_text_to_speech(model_name=test_model, provider=provider, config=config)
return {
"provider": provider,
"success": True,
"message": "Connection successful (key format valid)",
}
return {
"provider": provider,
"success": False,
"message": f"Unsupported test type: {test_type}",
}
except Exception as e:
if provider == "vertex" and _is_vertex_credentials_file_error(e):
logger.debug(f"Vertex credentials file error for credential {credential_id}: {e}")
return {
"provider": provider,
"success": False,
"message": "Invalid or inaccessible credentials file",
}
error_msg = str(e)
success, message = classify_provider_test_error(error_msg)
if not success:
logger.debug(f"Test connection error for credential {credential_id}: {e}")
return {"provider": provider, "success": success, "message": message}
async def discover_with_config(provider: str, config: dict) -> List[dict]:
"""
Discover models using explicit config instead of env vars.
Returns model names only — no type classification.
The user chooses the model type when registering.
"""
api_key = config.get("api_key")
base_url = config.get("base_url")
def models_endpoint(url: str) -> str:
trimmed = url.rstrip("/")
if trimmed.endswith("/models"):
return trimmed
return f"{trimmed}/models"
# Static model lists for providers without a listing API
STATIC_MODELS: Dict[str, List[str]] = {
"voyage": [
"voyage-3", "voyage-3-lite", "voyage-code-3",
"voyage-finance-2", "voyage-law-2", "voyage-multilingual-2",
],
"elevenlabs": [
"eleven_multilingual_v2", "eleven_turbo_v2_5",
"eleven_turbo_v2", "eleven_monolingual_v1",
"scribe_v1", # speech-to-text
],
"deepgram": [
"aura-2-thalia-en", "aura-2-andromeda-en", "aura-2-helena-en",
"aura-2-apollo-en", "aura-2-arcas-en", "aura-2-asteria-en",
"aura-2-athena-en", "aura-2-hera-en", "aura-2-hermes-en",
"aura-2-atlas-en",
],
}
if provider in STATIC_MODELS:
if not api_key and provider != "ollama":
return []
return [
{"name": m, "provider": provider}
for m in STATIC_MODELS[provider]
]
if provider == "anthropic":
if not api_key:
return []
try:
model_names = await fetch_anthropic_model_ids(api_key)
except Exception as e:
logger.warning(
f"Failed to discover Anthropic models, using static fallback: {e}"
)
model_names = list(ANTHROPIC_FALLBACK_MODELS)
return [{"name": m, "provider": "anthropic"} for m in model_names]
# API-based discovery URLs (OpenAI-style /models endpoints), from the registry
url_map = {
name: spec.openai_compat_discovery_url
for name, spec in PROVIDERS.items()
if spec.openai_compat_discovery_url
}
if provider == "ollama":
ollama_url = base_url or "http://localhost:11434"
try:
# Re-validate at request time: the base_url may have been saved
# against a hostname that only later resolved to an internal
# address (DNS rebinding).
await validate_url(ollama_url, "ollama")
async with httpx.AsyncClient() as client:
response = await client.get(f"{ollama_url}/api/tags", timeout=10.0)
response.raise_for_status()
data = response.json()
return [
{
"name": m.get("name", ""),
"provider": "ollama",
"model_type": classify_model_type(m.get("name", ""), "ollama"),
}
for m in data.get("models", [])
if m.get("name")
]
except Exception as e:
logger.warning(f"Failed to discover Ollama models: {e}")
return []
if provider == "openai_compatible":
if not base_url:
return []
try:
# Re-validate at request time (see ollama branch above).
await validate_url(base_url, "openai_compatible")
headers = {}
if api_key:
headers["Authorization"] = f"Bearer {api_key}"
async with httpx.AsyncClient() as client:
response = await client.get(
models_endpoint(base_url),
headers=headers,
timeout=30.0,
)
response.raise_for_status()
data = response.json()
return [
{"name": m.get("id", ""), "provider": "openai_compatible"}
for m in data.get("data", [])
if m.get("id")
]
except Exception as e:
logger.warning(f"Failed to discover openai_compatible models: {e}")
return []
if provider == "azure":
endpoint = config.get("endpoint")
api_version = config.get("api_version", "2024-10-21")
if not endpoint or not api_key:
return []
try:
# Re-validate at request time (see ollama branch above).
await validate_url(endpoint, "azure")
url = f"{endpoint.rstrip('/')}/openai/models?api-version={api_version}"
headers = {"api-key": api_key}
async with httpx.AsyncClient() as client:
response = await client.get(url, headers=headers, timeout=30.0)
response.raise_for_status()
data = response.json()
return [
{"name": m.get("id", ""), "provider": "azure"}
for m in data.get("data", [])
if m.get("id")
]
except Exception as e:
logger.warning(f"Failed to discover Azure models: {e}")
return []
if provider == "vertex":
# Vertex AI requires service-account OAuth2 for model listing.
# Return a curated static list of well-known Vertex models instead.
VERTEX_MODELS = [
"gemini-3.5-flash",
"gemini-2.5-flash",
"gemini-2.5-flash-lite",
"text-embedding-005",
]
return [{"name": m, "provider": "vertex"} for m in VERTEX_MODELS]
if provider == "google":
try:
headers = {"X-Goog-Api-Key": api_key} if api_key else {}
async with httpx.AsyncClient() as client:
response = await client.get(
"https://generativelanguage.googleapis.com/v1/models",
headers=headers,
timeout=30.0,
)
response.raise_for_status()
data = response.json()
return [
{
"name": model.get("name", "").replace("models/", ""),
"provider": "google",
"description": model.get("displayName"),
}
for model in data.get("models", [])
if model.get("name")
]
except Exception as e:
logger.warning(f"Failed to discover Google models: {e}")
return []
# Standard OpenAI-style API discovery
discovery_url = url_map.get(provider)
if provider == "openai" and base_url:
discovery_url = models_endpoint(base_url)
if not discovery_url or not api_key:
return []
try:
async with httpx.AsyncClient() as client:
response = await client.get(
discovery_url,
headers={"Authorization": f"Bearer {api_key}"},
timeout=30.0,
)
response.raise_for_status()
data = response.json()
return [
{
"name": m.get("id", ""),
"provider": provider,
"description": m.get("name"),
}
for m in data.get("data", [])
if m.get("id")
]
except Exception as e:
logger.warning(f"Failed to discover {provider} models: {e}")
return []
async def register_models(credential_id: str, models_data: list) -> dict:
"""
Register discovered models and link them to a credential.
Args:
credential_id: The credential ID to link models to
models_data: List of dicts with name, provider, model_type
Returns:
dict with created and existing counts
"""
cred = await Credential.get(credential_id)
from open_notebook.ai.models import Model
from open_notebook.database.repository import repo_query
# Batch fetch existing models for this provider
existing_models = await repo_query(
"SELECT string::lowercase(name) as name, string::lowercase(type) as type FROM model "
"WHERE string::lowercase(provider) = $provider",
{"provider": cred.provider.lower()},
)
existing_keys = {(m["name"], m["type"]) for m in existing_models}
created = 0
existing = 0
for model_data in models_data:
key = (model_data.name.lower(), model_data.model_type.lower())
if key in existing_keys:
existing += 1
continue
new_model = Model(
name=model_data.name,
provider=model_data.provider or cred.provider,
type=model_data.model_type,
credential=cred.id,
)
await new_model.save()
created += 1
return {"created": created, "existing": existing}
async def migrate_from_provider_config() -> dict:
"""
Migrate existing ProviderConfig data to individual credential records.
Returns dict with message, migrated, skipped, errors.
"""
logger.info("=== Starting ProviderConfig migration ===")
require_encryption_key()
logger.info("Encryption key verified")
from open_notebook.domain.provider_config import ProviderConfig
config = await ProviderConfig.get_instance()
logger.info(
f"Found ProviderConfig with {len(config.credentials)} provider(s): "
f"{', '.join(config.credentials.keys())}"
)
migrated = []
skipped = []
errors = []
for provider, credentials_list in config.credentials.items():
for old_cred in credentials_list:
try:
# Check if a credential already exists for this provider with same name
existing = await Credential.get_by_provider(provider)
names = [c.name for c in existing]
if old_cred.name in names:
logger.info(
f"[{provider}/{old_cred.name}] Already exists in DB, skipping"
)
skipped.append(f"{provider}/{old_cred.name}")
continue
# Determine modalities from the provider type
modalities = get_default_modalities(provider)
logger.info(f"[{provider}/{old_cred.name}] Creating credential")
new_cred = Credential(
name=old_cred.name,
provider=provider,
modalities=modalities,
api_key=old_cred.api_key,
base_url=old_cred.base_url,
endpoint=old_cred.endpoint,
api_version=old_cred.api_version,
endpoint_llm=old_cred.endpoint_llm,
endpoint_embedding=old_cred.endpoint_embedding,
endpoint_stt=old_cred.endpoint_stt,
endpoint_tts=old_cred.endpoint_tts,
project=old_cred.project,
location=old_cred.location,
credentials_path=old_cred.credentials_path,
)
await new_cred.save()
logger.info(
f"[{provider}/{old_cred.name}] Credential saved (id={new_cred.id})"
)
# Link existing models for this provider to the new credential
from open_notebook.ai.models import Model
from open_notebook.database.repository import repo_query
provider_models = await repo_query(
"SELECT * FROM model WHERE string::lowercase(provider) = $provider AND credential IS NONE",
{"provider": provider.lower()},
)
if provider_models:
logger.info(
f"[{provider}/{old_cred.name}] Linking {len(provider_models)} "
f"unassigned model(s)"
)
for model_data in provider_models:
model = Model(**model_data)
model.credential = new_cred.id
await model.save()
migrated.append(f"{provider}/{old_cred.name}")
except Exception as e:
logger.error(
f"[{provider}/{old_cred.name}] Migration FAILED: "
f"{type(e).__name__}: {e}",
exc_info=True,
)
errors.append(f"{provider}/{old_cred.name}: {e}")
logger.info(
f"=== ProviderConfig migration complete === "
f"migrated={len(migrated)} skipped={len(skipped)} errors={len(errors)}"
)
if migrated:
logger.info(f" Migrated: {', '.join(migrated)}")
if skipped:
logger.info(f" Skipped: {', '.join(skipped)}")
if errors:
logger.error(f" Errors: {'; '.join(errors)}")
return {
"message": f"Migration complete. Migrated {len(migrated)} credentials.",
"migrated": migrated,
"skipped": skipped,
"errors": errors,
}
async def migrate_from_env() -> dict:
"""
Migrate API keys from environment variables to credential records.
Returns dict with message, migrated, skipped, not_configured, errors.
"""
logger.info("=== Starting environment variable migration ===")
logger.info(
f"Checking {len(PROVIDER_ENV_CONFIG)} providers: "
f"{', '.join(PROVIDER_ENV_CONFIG.keys())}"
)
require_encryption_key()
logger.info("Encryption key verified")
from open_notebook.ai.models import Model
from open_notebook.database.repository import repo_query
migrated = []
skipped = []
not_configured = []
errors = []
for provider in PROVIDER_ENV_CONFIG:
try:
if not check_env_configured(provider):
logger.debug(f"[{provider}] No env vars configured, skipping")
not_configured.append(provider)
continue
logger.info(f"[{provider}] Env vars detected, checking for existing credentials")
existing = await Credential.get_by_provider(provider)
if existing:
logger.info(
f"[{provider}] Already has {len(existing)} credential(s) in DB, skipping"
)
skipped.append(provider)
continue
logger.info(f"[{provider}] Creating credential from env vars")
cred = create_credential_from_env(provider)
await cred.save()
logger.info(f"[{provider}] Credential saved successfully (id={cred.id})")
# Link unassigned models to this credential
provider_models = await repo_query(
"SELECT * FROM model WHERE string::lowercase(provider) = $provider AND credential IS NONE",
{"provider": provider.lower()},
)
if provider_models:
logger.info(
f"[{provider}] Linking {len(provider_models)} unassigned model(s) "
f"to credential {cred.id}"
)
for model_data in provider_models:
model = Model(**model_data)
model.credential = cred.id
await model.save()
else:
logger.info(f"[{provider}] No unassigned models to link")
migrated.append(provider)
except Exception as e:
logger.error(
f"[{provider}] Migration FAILED: {type(e).__name__}: {e}",
exc_info=True,
)
errors.append(f"{provider}: {e}")
logger.info(
f"=== Environment variable migration complete === "
f"migrated={len(migrated)} skipped={len(skipped)} "
f"not_configured={len(not_configured)} errors={len(errors)}"
)
if migrated:
logger.info(f" Migrated: {', '.join(migrated)}")
if skipped:
logger.info(f" Skipped (already in DB): {', '.join(skipped)}")
if errors:
logger.error(f" Errors: {'; '.join(errors)}")
return {
"message": f"Migration complete. Migrated {len(migrated)} providers.",
"migrated": migrated,
"skipped": skipped,
"not_configured": not_configured,
"errors": errors,
}
api/main.py
+407
View File
@@ -0,0 +1,407 @@
# Load environment variables
from dotenv import load_dotenv
load_dotenv()
import asyncio
import os
from contextlib import asynccontextmanager
from fastapi import FastAPI, Request
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import JSONResponse
from loguru import logger
from starlette.exceptions import HTTPException as StarletteHTTPException
from api.auth import PasswordAuthMiddleware
from api.middleware import MaxBodySizeMiddleware, get_max_upload_size_bytes
from api.routers import (
auth,
chat,
config,
credentials,
embedding,
embedding_rebuild,
episode_profiles,
insights,
languages,
models,
notebooks,
notes,
podcasts,
providers,
search,
settings,
source_chat,
sources,
speaker_profiles,
transformations,
)
from api.routers import commands as commands_router
from open_notebook.database.async_migrate import AsyncMigrationManager
from open_notebook.exceptions import (
AuthenticationError,
ConfigurationError,
ExternalServiceError,
InvalidInputError,
NetworkError,
NotFoundError,
OpenNotebookError,
RateLimitError,
UnsupportedTypeException,
)
from open_notebook.utils.encryption import get_secret_from_env
def _parse_cors_origins(raw: str) -> list[str]:
"""Parse CORS_ORIGINS env value into a list of origins."""
value = raw.strip()
if value == "*":
return ["*"]
return [origin.strip() for origin in value.split(",") if origin.strip()]
# Parsed once at module load; CORS_ORIGINS changes require a restart.
_cors_origins_raw = os.getenv("CORS_ORIGINS")
CORS_ALLOWED_ORIGINS = _parse_cors_origins(_cors_origins_raw or "*")
CORS_IS_DEFAULT_WILDCARD = _cors_origins_raw is None
# Keyed on the parsed list, not on whether the env var was set: an operator
# who explicitly sets CORS_ORIGINS=* must get the same wildcard treatment as
# the default, or credentials would combine with a wildcard origin - the
# exact reflect-any-Origin behavior this flag exists to prevent.
CORS_ALLOW_CREDENTIALS = "*" not in CORS_ALLOWED_ORIGINS
# Parsed once at module load; OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB changes require a restart.
MAX_UPLOAD_SIZE_BYTES = get_max_upload_size_bytes()
DATABASE_STARTUP_RETRY_ATTEMPTS = 12
DATABASE_STARTUP_RETRY_INITIAL_DELAY_SECONDS = 1
DATABASE_STARTUP_RETRY_MAX_DELAY_SECONDS = 5
# Per-probe ceiling so a hung connection cannot exceed the retry budget or
# block startup indefinitely. A probe that exceeds this is treated as a
# transient failure and retried like any other unreachable-database attempt.
DATABASE_STARTUP_RETRY_PROBE_TIMEOUT_SECONDS = 5
def _cors_headers(request: Request) -> dict[str, str]:
"""
Build CORS headers for error responses.
Mirrors Starlette CORSMiddleware behavior: reflects the request Origin
when the origin is allowed (or when wildcard is configured, since
browsers reject `Access-Control-Allow-Origin: *` combined with
credentials). Omits `Access-Control-Allow-Origin` for disallowed
origins so the browser blocks the error body from leaking cross-origin.
Only claims Access-Control-Allow-Credentials when the real CORSMiddleware
would (see its allow_credentials comment above) - otherwise error
responses would grant credentialed access the success path doesn't.
"""
origin = request.headers.get("origin")
headers: dict[str, str] = {
"Access-Control-Allow-Methods": "*",
"Access-Control-Allow-Headers": "*",
}
if CORS_ALLOW_CREDENTIALS:
headers["Access-Control-Allow-Credentials"] = "true"
if origin and ("*" in CORS_ALLOWED_ORIGINS or origin in CORS_ALLOWED_ORIGINS):
headers["Access-Control-Allow-Origin"] = origin
headers["Vary"] = "Origin"
return headers
# Import commands to register them in the API process
try:
logger.info("Commands imported in API process")
except Exception as e:
logger.error(f"Failed to import commands in API process: {e}")
async def _wait_for_database(migration_manager: AsyncMigrationManager) -> None:
"""
Wait for SurrealDB to accept connections before running migrations.
Docker Compose can start the API before the database name is resolvable. Keep
migration errors fail-fast by only retrying this lightweight readiness probe.
"""
attempts = max(1, DATABASE_STARTUP_RETRY_ATTEMPTS)
delay = DATABASE_STARTUP_RETRY_INITIAL_DELAY_SECONDS
for attempt in range(1, attempts + 1):
try:
await asyncio.wait_for(
migration_manager.ping(),
timeout=DATABASE_STARTUP_RETRY_PROBE_TIMEOUT_SECONDS,
)
if attempt > 1:
logger.info(f"Database became reachable on attempt {attempt}")
return
except Exception as e:
if attempt == attempts:
logger.error(
f"Database did not become reachable after {attempts} attempts"
)
raise
logger.warning(
"Database is not reachable yet "
f"(attempt {attempt}/{attempts}): {str(e)}. "
f"Retrying in {delay:g} seconds..."
)
await asyncio.sleep(delay)
delay = min(delay * 2, DATABASE_STARTUP_RETRY_MAX_DELAY_SECONDS)
async def _run_database_migrations() -> None:
"""Run startup database migrations after SurrealDB is reachable."""
migration_manager = AsyncMigrationManager()
await _wait_for_database(migration_manager)
current_version = await migration_manager.get_current_version()
logger.info(f"Current database version: {current_version}")
if await migration_manager.needs_migration():
logger.warning("Database migrations are pending. Running migrations...")
await migration_manager.run_migration_up()
new_version = await migration_manager.get_current_version()
logger.success(
f"Migrations completed successfully. Database is now at version {new_version}"
)
else:
logger.info("Database is already at the latest version. No migrations needed.")
@asynccontextmanager
async def lifespan(app: FastAPI):
"""
Lifespan event handler for the FastAPI application.
Runs database migrations automatically on startup.
"""
# Startup: Security checks
logger.info("Starting API initialization...")
# Security check: Encryption key
if not get_secret_from_env("OPEN_NOTEBOOK_ENCRYPTION_KEY"):
logger.warning(
"OPEN_NOTEBOOK_ENCRYPTION_KEY not set. "
"API key encryption will fail until this is configured. "
"Set OPEN_NOTEBOOK_ENCRYPTION_KEY to any secret string."
)
# Run database migrations
try:
await _run_database_migrations()
except Exception as e:
logger.error(f"CRITICAL: Database migration failed: {str(e)}")
logger.exception(e)
# Fail fast - don't start the API with an outdated database schema
raise RuntimeError(f"Failed to run database migrations: {str(e)}") from e
logger.success("API initialization completed successfully")
# Yield control to the application
yield
# Shutdown: cleanup if needed
logger.info("API shutdown complete")
app = FastAPI(
title="Open Notebook API",
description="API for Open Notebook - Research Assistant",
lifespan=lifespan,
)
if CORS_IS_DEFAULT_WILDCARD:
logger.warning(
"CORS_ORIGINS is not set — API accepts cross-origin requests from any "
"origin (default: '*'). For production deployments, set CORS_ORIGINS to "
"your frontend origin(s), e.g. "
"CORS_ORIGINS=https://notebook.example.com"
)
else:
logger.info(f"CORS allowed origins: {CORS_ALLOWED_ORIGINS}")
# Add password authentication middleware first
# Exclude /api/auth/status and /api/config from authentication
app.add_middleware(
PasswordAuthMiddleware,
excluded_paths=[
"/",
"/health",
"/docs",
"/openapi.json",
"/redoc",
"/api/auth/status",
"/api/config",
],
)
# Reject oversized request bodies before they reach auth or routing - added
# after PasswordAuthMiddleware (so it wraps around it) so a too-large request
# is rejected before spending any work checking credentials.
logger.info(
f"Max request body size: {MAX_UPLOAD_SIZE_BYTES / (1024 * 1024):g}MB "
"(set OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB to change)"
)
app.add_middleware(MaxBodySizeMiddleware, max_body_size=MAX_UPLOAD_SIZE_BYTES)
# Add CORS middleware last (so it processes first, and so it can attach
# CORS headers to a 413 raised by MaxBodySizeMiddleware)
#
# allow_credentials is tied to whether CORS_ORIGINS resolves to specific
# origins: combining allow_origins=["*"] with allow_credentials=True makes
# Starlette reflect the request's Origin header verbatim (browsers reject a
# literal "*" alongside credentials), which defeats the origin allowlist.
# The frontend never sends credentialed requests (withCredentials: false)
# and auth is a Bearer header, not a cookie, so this isn't independently
# exploitable today - but there's no reason to allow it for any wildcard
# case. Once an operator explicitly scopes CORS_ORIGINS to real origins,
# credentialed cross-origin requests to those origins are safe to allow.
app.add_middleware(
CORSMiddleware,
allow_origins=CORS_ALLOWED_ORIGINS,
allow_credentials=CORS_ALLOW_CREDENTIALS,
allow_methods=["*"],
allow_headers=["*"],
)
# Custom exception handler to ensure CORS headers are included in error responses
# This helps when errors occur before the CORS middleware can process them
@app.exception_handler(StarletteHTTPException)
async def custom_http_exception_handler(request: Request, exc: StarletteHTTPException):
"""
Custom exception handler that ensures CORS headers are included in error responses.
This is particularly important for 413 (Payload Too Large) errors during file uploads.
Note: If a reverse proxy (nginx, traefik) returns 413 before the request reaches
FastAPI, this handler won't be called. In that case, configure your reverse proxy
to add CORS headers to error responses.
"""
return JSONResponse(
status_code=exc.status_code,
content={"detail": exc.detail},
headers={**(exc.headers or {}), **_cors_headers(request)},
)
@app.exception_handler(NotFoundError)
async def not_found_error_handler(request: Request, exc: NotFoundError):
return JSONResponse(
status_code=404,
content={"detail": str(exc)},
headers=_cors_headers(request),
)
@app.exception_handler(InvalidInputError)
async def invalid_input_error_handler(request: Request, exc: InvalidInputError):
return JSONResponse(
status_code=400,
content={"detail": str(exc)},
headers=_cors_headers(request),
)
@app.exception_handler(AuthenticationError)
async def authentication_error_handler(request: Request, exc: AuthenticationError):
return JSONResponse(
status_code=401,
content={"detail": str(exc)},
headers=_cors_headers(request),
)
@app.exception_handler(RateLimitError)
async def rate_limit_error_handler(request: Request, exc: RateLimitError):
return JSONResponse(
status_code=429,
content={"detail": str(exc)},
headers=_cors_headers(request),
)
@app.exception_handler(ConfigurationError)
async def configuration_error_handler(request: Request, exc: ConfigurationError):
return JSONResponse(
status_code=422,
content={"detail": str(exc)},
headers=_cors_headers(request),
)
@app.exception_handler(NetworkError)
async def network_error_handler(request: Request, exc: NetworkError):
return JSONResponse(
status_code=502,
content={"detail": str(exc)},
headers=_cors_headers(request),
)
@app.exception_handler(ExternalServiceError)
async def external_service_error_handler(request: Request, exc: ExternalServiceError):
return JSONResponse(
status_code=502,
content={"detail": str(exc)},
headers=_cors_headers(request),
)
@app.exception_handler(UnsupportedTypeException)
async def unsupported_type_error_handler(
request: Request, exc: UnsupportedTypeException
):
return JSONResponse(
status_code=415,
content={"detail": str(exc)},
headers=_cors_headers(request),
)
@app.exception_handler(OpenNotebookError)
async def open_notebook_error_handler(request: Request, exc: OpenNotebookError):
return JSONResponse(
status_code=500,
content={"detail": str(exc)},
headers=_cors_headers(request),
)
# Include routers
app.include_router(auth.router, prefix="/api", tags=["auth"])
app.include_router(config.router, prefix="/api", tags=["config"])
app.include_router(notebooks.router, prefix="/api", tags=["notebooks"])
app.include_router(search.router, prefix="/api", tags=["search"])
app.include_router(models.router, prefix="/api", tags=["models"])
app.include_router(transformations.router, prefix="/api", tags=["transformations"])
app.include_router(notes.router, prefix="/api", tags=["notes"])
app.include_router(embedding.router, prefix="/api", tags=["embedding"])
app.include_router(
embedding_rebuild.router, prefix="/api/embeddings", tags=["embeddings"]
)
app.include_router(settings.router, prefix="/api", tags=["settings"])
app.include_router(sources.router, prefix="/api", tags=["sources"])
app.include_router(insights.router, prefix="/api", tags=["insights"])
app.include_router(commands_router.router, prefix="/api", tags=["commands"])
app.include_router(podcasts.router, prefix="/api", tags=["podcasts"])
app.include_router(episode_profiles.router, prefix="/api", tags=["episode-profiles"])
app.include_router(speaker_profiles.router, prefix="/api", tags=["speaker-profiles"])
app.include_router(chat.router, prefix="/api", tags=["chat"])
app.include_router(source_chat.router, prefix="/api", tags=["source-chat"])
app.include_router(credentials.router, prefix="/api", tags=["credentials"])
app.include_router(providers.router, prefix="/api", tags=["providers"])
app.include_router(languages.router, prefix="/api", tags=["languages"])
@app.get("/")
async def root():
return {"message": "Open Notebook API is running"}
@app.get("/health")
async def health():
return {"status": "healthy"}
api/middleware.py
+122
View File
@@ -0,0 +1,122 @@
import os
from loguru import logger
from starlette.datastructures import Headers
from starlette.types import ASGIApp, Message, Receive, Scope, Send
# Matches the file-size guidance already documented in
# docs/3-USER-GUIDE/adding-sources.md ("Very large files (>100MB) - Timeout").
DEFAULT_MAX_UPLOAD_SIZE_MB = 100
def get_max_upload_size_bytes() -> int:
"""Read the configured max request body size, in bytes.
Configurable via OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB for deployments that
need larger audio/video uploads; falls back to the default on unset,
malformed, or non-positive values (a zero/negative limit would reject
every request that has a body).
"""
raw = os.environ.get("OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB", "").strip()
try:
mb = float(raw) if raw else DEFAULT_MAX_UPLOAD_SIZE_MB
except ValueError:
mb = DEFAULT_MAX_UPLOAD_SIZE_MB
if mb <= 0:
logger.warning(
f"OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB={raw!r} is not a positive size; "
f"using the default of {DEFAULT_MAX_UPLOAD_SIZE_MB}MB"
)
mb = DEFAULT_MAX_UPLOAD_SIZE_MB
return int(mb * 1024 * 1024)
class _RequestBodyTooLarge(Exception):
pass
class MaxBodySizeMiddleware:
"""
Raw ASGI middleware rejecting requests whose body exceeds a configured
maximum, so a large upload can't exhaust memory/disk on a deployment
with no fronting proxy enforcing its own limit (e.g. the shipped
docker-compose.yml, which exposes the API directly).
Implemented at the raw ASGI level (not BaseHTTPMiddleware) so the check
can run ahead of FastAPI's own body/form parsing instead of after it.
Rejects on the `Content-Length` header up front when present (the common
case, and cheap), and also counts bytes as the body streams in - a
client can lie about Content-Length or omit it entirely with chunked
transfer-encoding.
"""
def __init__(self, app: ASGIApp, max_body_size: int) -> None:
self.app = app
self.max_body_size = max_body_size
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
if scope["type"] != "http":
await self.app(scope, receive, send)
return
headers = Headers(scope=scope)
content_length = headers.get("content-length")
if content_length is not None:
try:
if int(content_length) > self.max_body_size:
logger.warning(
f"Rejected {scope.get('method', '?')} {scope.get('path', '?')}: "
f"declared body of {content_length} bytes exceeds the "
f"{self.max_body_size}-byte limit"
)
await _send_413(send)
return
except ValueError:
pass # malformed header - fall through to streaming enforcement
total_size = 0
response_started = False
async def send_wrapper(message: Message) -> None:
nonlocal response_started
if message["type"] == "http.response.start":
response_started = True
await send(message)
async def receive_wrapper() -> Message:
nonlocal total_size
message = await receive()
if message["type"] == "http.request":
total_size += len(message.get("body") or b"")
if total_size > self.max_body_size:
raise _RequestBodyTooLarge()
return message
try:
await self.app(scope, receive_wrapper, send_wrapper)
except _RequestBodyTooLarge:
logger.warning(
f"Rejected {scope.get('method', '?')} {scope.get('path', '?')}: "
f"streamed body exceeded the {self.max_body_size}-byte limit"
)
if not response_started:
await _send_413(send)
# Else the app already started responding - nothing safe to send;
# let the connection drop rather than violate the ASGI protocol
# with a second response.start.
async def _send_413(send: Send) -> None:
await send(
{
"type": "http.response.start",
"status": 413,
"headers": [(b"content-type", b"application/json")],
}
)
await send(
{
"type": "http.response.body",
"body": b'{"detail":"Request body exceeds the maximum allowed upload size"}',
}
)
api/models.py
+734
View File
@@ -0,0 +1,734 @@
from typing import Any, Dict, List, Literal, Optional
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
# Notebook models
class NotebookCreate(BaseModel):
name: str = Field(..., description="Name of the notebook")
description: str = Field(default="", description="Description of the notebook")
class NotebookUpdate(BaseModel):
name: Optional[str] = Field(None, description="Name of the notebook")
description: Optional[str] = Field(None, description="Description of the notebook")
archived: Optional[bool] = Field(
None, description="Whether the notebook is archived"
)
class NotebookResponse(BaseModel):
id: str
name: str
description: str
archived: bool
created: str
updated: str
source_count: int
note_count: int
class RecentlyViewedResponse(BaseModel):
type: Literal["notebook", "source"]
id: str
title: str
last_viewed_at: str
# Search models
class SearchRequest(BaseModel):
query: str = Field(..., description="Search query")
type: Literal["text", "vector"] = Field("text", description="Search type")
limit: int = Field(100, description="Maximum number of results", ge=1, le=1000)
search_sources: bool = Field(True, description="Include sources in search")
search_notes: bool = Field(True, description="Include notes in search")
minimum_score: float = Field(
0.2, description="Minimum score for vector search", ge=0, le=1
)
class SearchResponse(BaseModel):
results: List[Dict[str, Any]] = Field(..., description="Search results")
total_count: int = Field(..., description="Total number of results")
search_type: str = Field(..., description="Type of search performed")
class AskRequest(BaseModel):
question: str = Field(..., description="Question to ask the knowledge base")
strategy_model: str = Field(..., description="Model ID for query strategy")
answer_model: str = Field(..., description="Model ID for individual answers")
final_answer_model: str = Field(..., description="Model ID for final answer")
class AskResponse(BaseModel):
answer: str = Field(..., description="Final answer from the knowledge base")
question: str = Field(..., description="Original question")
# Models API models
class ModelCreate(BaseModel):
name: str = Field(..., description="Model name (e.g., gpt-5-mini, claude, gemini)")
provider: str = Field(
..., description="Provider name (e.g., openai, anthropic, gemini)"
)
type: str = Field(
...,
description="Model type (language, embedding, text_to_speech, speech_to_text)",
)
credential: Optional[str] = Field(
None, description="Credential ID to link this model to"
)
class ModelResponse(BaseModel):
id: str
name: str
provider: str
type: str
credential: Optional[str] = None
created: str
updated: str
class DefaultModelsResponse(BaseModel):
default_chat_model: Optional[str] = None
default_transformation_model: Optional[str] = None
large_context_model: Optional[str] = None
default_text_to_speech_model: Optional[str] = None
default_speech_to_text_model: Optional[str] = None
default_embedding_model: Optional[str] = None
default_tools_model: Optional[str] = None
class ProviderAvailabilityResponse(BaseModel):
available: List[str] = Field(..., description="List of available providers")
unavailable: List[str] = Field(..., description="List of unavailable providers")
supported_types: Dict[str, List[str]] = Field(
..., description="Provider to supported model types mapping"
)
# Transformations API models
class TransformationCreate(BaseModel):
name: str = Field(..., description="Transformation name")
title: str = Field(..., description="Display title for the transformation")
description: str = Field(
..., description="Description of what this transformation does"
)
prompt: str = Field(..., description="The transformation prompt")
apply_default: bool = Field(
False, description="Whether to apply this transformation by default"
)
model_id: Optional[str] = Field(
None, description="Model ID to use by default for this transformation"
)
class TransformationUpdate(BaseModel):
name: Optional[str] = Field(None, description="Transformation name")
title: Optional[str] = Field(
None, description="Display title for the transformation"
)
description: Optional[str] = Field(
None, description="Description of what this transformation does"
)
prompt: Optional[str] = Field(None, description="The transformation prompt")
apply_default: Optional[bool] = Field(
None, description="Whether to apply this transformation by default"
)
model_id: Optional[str] = Field(
None, description="Model ID to use by default for this transformation"
)
class TransformationResponse(BaseModel):
id: str
name: str
title: str
description: str
prompt: str
apply_default: bool
model_id: Optional[str] = None
created: str
updated: str
class TransformationExecuteRequest(BaseModel):
model_config = ConfigDict(protected_namespaces=())
transformation_id: str = Field(
..., description="ID of the transformation to execute"
)
input_text: str = Field(..., description="Text to transform")
model_id: Optional[str] = Field(
None, description="Model ID to use for this transformation run"
)
class TransformationExecuteResponse(BaseModel):
model_config = ConfigDict(protected_namespaces=())
output: str = Field(..., description="Transformed text")
transformation_id: str = Field(..., description="ID of the transformation used")
model_id: Optional[str] = Field(None, description="Model ID used")
# Default Prompt API models
class DefaultPromptResponse(BaseModel):
transformation_instructions: str = Field(
..., description="Default transformation instructions"
)
class DefaultPromptUpdate(BaseModel):
transformation_instructions: str = Field(
..., description="Default transformation instructions"
)
# Notes API models
class NoteCreate(BaseModel):
title: Optional[str] = Field(None, description="Note title")
content: str = Field(..., description="Note content")
note_type: Optional[str] = Field("human", description="Type of note (human, ai)")
notebook_id: Optional[str] = Field(
None, description="Notebook ID to add the note to"
)
class NoteUpdate(BaseModel):
title: Optional[str] = Field(None, description="Note title")
content: Optional[str] = Field(None, description="Note content")
note_type: Optional[str] = Field(None, description="Type of note (human, ai)")
class NoteResponse(BaseModel):
id: str
title: Optional[str]
content: Optional[str]
note_type: Optional[str]
created: str
updated: str
command_id: Optional[str] = None
# Embedding API models
class EmbedRequest(BaseModel):
item_id: str = Field(..., description="ID of the item to embed")
item_type: str = Field(..., description="Type of item (source, note)")
async_processing: bool = Field(
False, description="Process asynchronously in background"
)
class EmbedResponse(BaseModel):
success: bool = Field(..., description="Whether embedding was successful")
message: str = Field(..., description="Result message")
item_id: str = Field(..., description="ID of the item that was embedded")
item_type: str = Field(..., description="Type of item that was embedded")
command_id: Optional[str] = Field(
None, description="Command ID for async processing"
)
# Rebuild request/response models
class RebuildRequest(BaseModel):
mode: Literal["existing", "all"] = Field(
...,
description="Rebuild mode: 'existing' only re-embeds items with embeddings, 'all' embeds everything",
)
include_sources: bool = Field(True, description="Include sources in rebuild")
include_notes: bool = Field(True, description="Include notes in rebuild")
include_insights: bool = Field(True, description="Include insights in rebuild")
class RebuildResponse(BaseModel):
command_id: str = Field(..., description="Command ID to track progress")
total_items: int = Field(..., description="Estimated number of items to process")
message: str = Field(..., description="Status message")
class RebuildProgress(BaseModel):
processed: int = Field(..., description="Number of items processed")
total: int = Field(..., description="Total items to process")
percentage: float = Field(..., description="Progress percentage")
class RebuildStats(BaseModel):
sources: int = Field(0, description="Sources processed")
notes: int = Field(0, description="Notes processed")
insights: int = Field(0, description="Insights processed")
failed: int = Field(0, description="Failed items")
class RebuildStatusResponse(BaseModel):
command_id: str = Field(..., description="Command ID")
status: str = Field(..., description="Status: queued, running, completed, failed")
progress: Optional[RebuildProgress] = None
stats: Optional[RebuildStats] = None
started_at: Optional[str] = None
completed_at: Optional[str] = None
error_message: Optional[str] = None
# Settings API models
class SettingsResponse(BaseModel):
default_content_processing_engine_doc: Optional[str] = None
default_content_processing_engine_url: Optional[str] = None
default_embedding_option: Optional[str] = None
auto_delete_files: Optional[str] = None
docling_ocr: Optional[bool] = None
youtube_preferred_languages: Optional[List[str]] = None
class SettingsUpdate(BaseModel):
default_content_processing_engine_doc: Optional[str] = None
default_content_processing_engine_url: Optional[str] = None
default_embedding_option: Optional[str] = None
auto_delete_files: Optional[str] = None
docling_ocr: Optional[bool] = None
youtube_preferred_languages: Optional[List[str]] = None
# Sources API models
class AssetModel(BaseModel):
file_path: Optional[str] = None
url: Optional[str] = None
class SourceCreate(BaseModel):
# Backward compatibility: support old single notebook_id
notebook_id: Optional[str] = Field(
None, description="Notebook ID to add the source to (deprecated, use notebooks)"
)
# New multi-notebook support
notebooks: Optional[List[str]] = Field(
None,
max_length=50,
description="List of notebook IDs to add the source to (max 50)",
)
# Required fields
type: str = Field(..., description="Source type: link, upload, or text")
url: Optional[str] = Field(None, description="URL for link type")
file_path: Optional[str] = Field(None, description="File path for upload type")
content: Optional[str] = Field(None, description="Text content for text type")
title: Optional[str] = Field(None, description="Source title")
transformations: Optional[List[str]] = Field(
default_factory=list,
max_length=50,
description="Transformation IDs to apply (max 50)",
)
embed: bool = Field(False, description="Whether to embed content for vector search")
delete_source: bool = Field(
False, description="Whether to delete uploaded file after processing"
)
# New async processing support
async_processing: bool = Field(
False, description="Whether to process source asynchronously"
)
@model_validator(mode="after")
def validate_notebook_fields(self):
# Ensure only one of notebook_id or notebooks is provided
if self.notebook_id is not None and self.notebooks is not None:
raise ValueError(
"Cannot specify both 'notebook_id' and 'notebooks'. Use 'notebooks' for multi-notebook support."
)
# Convert single notebook_id to notebooks array for internal processing
if self.notebook_id is not None:
self.notebooks = [self.notebook_id]
# Keep notebook_id for backward compatibility in response
# Set empty array if no notebooks specified (allow sources without notebooks)
if self.notebooks is None:
self.notebooks = []
return self
class SourceUpdate(BaseModel):
title: Optional[str] = Field(None, description="Source title")
topics: Optional[List[str]] = Field(None, description="Source topics")
class SourceResponse(BaseModel):
id: str
title: Optional[str]
topics: Optional[List[str]]
asset: Optional[AssetModel]
full_text: Optional[str]
embedded: bool
embedded_chunks: int
file_available: Optional[bool] = None
created: str
updated: str
# New fields for async processing
command_id: Optional[str] = None
status: Optional[str] = None
processing_info: Optional[Dict] = None
# Notebook associations
notebooks: Optional[List[str]] = None
class SourceListResponse(BaseModel):
id: str
title: Optional[str]
topics: Optional[List[str]]
asset: Optional[AssetModel]
embedded: bool # Boolean flag indicating if source has embeddings
embedded_chunks: int # Number of embedded chunks
insights_count: int
created: str
updated: str
file_available: Optional[bool] = None
# Status fields for async processing
command_id: Optional[str] = None
status: Optional[str] = None
processing_info: Optional[Dict[str, Any]] = None
# Insights API models
class SourceInsightResponse(BaseModel):
id: str
source_id: str
insight_type: str
content: str
# Optional: insights created before migration 19 have no timestamps,
# and the API must return null for them (never the string "None").
created: Optional[str] = None
updated: Optional[str] = None
class InsightCreationResponse(BaseModel):
"""Response for async insight creation."""
status: Literal["pending"] = "pending"
message: str = "Insight generation started"
source_id: str
transformation_id: str
command_id: Optional[str] = None
class SaveAsNoteRequest(BaseModel):
notebook_id: Optional[str] = Field(None, description="Notebook ID to add note to")
class CreateSourceInsightRequest(BaseModel):
model_config = ConfigDict(protected_namespaces=())
transformation_id: str = Field(..., description="ID of transformation to apply")
model_id: Optional[str] = Field(
None, description="Model ID (uses default if not provided)"
)
# Source status response
class SourceStatusResponse(BaseModel):
status: Optional[str] = Field(None, description="Processing status")
message: str = Field(..., description="Descriptive message about the status")
processing_info: Optional[Dict[str, Any]] = Field(
None, description="Detailed processing information"
)
command_id: Optional[str] = Field(None, description="Command ID if available")
# Error response
class ErrorResponse(BaseModel):
error: str
message: str
# API Key Configuration models
class SetApiKeyRequest(BaseModel):
"""Request to set an API key for a provider."""
api_key: Optional[str] = Field(None, description="API key for the provider")
base_url: Optional[str] = Field(
None, description="Base URL for URL-based providers (Ollama, OpenAI-compatible)"
)
endpoint: Optional[str] = Field(None, description="Endpoint URL for Azure OpenAI")
api_version: Optional[str] = Field(None, description="API version for Azure OpenAI")
endpoint_llm: Optional[str] = Field(
None, description="Service-specific endpoint for LLM (Azure)"
)
endpoint_embedding: Optional[str] = Field(
None, description="Service-specific endpoint for embedding (Azure)"
)
endpoint_stt: Optional[str] = Field(
None, description="Service-specific endpoint for STT (Azure)"
)
endpoint_tts: Optional[str] = Field(
None, description="Service-specific endpoint for TTS (Azure)"
)
service_type: Optional[Literal["llm", "embedding", "stt", "tts"]] = Field(
None,
description="Service type for OpenAI-compatible providers (llm, embedding, stt, tts)",
)
# Vertex AI specific fields
vertex_project: Optional[str] = Field(
None, description="Google Cloud Project ID for Vertex AI"
)
vertex_location: Optional[str] = Field(
None, description="Google Cloud Region for Vertex AI (e.g., us-central1)"
)
vertex_credentials_path: Optional[str] = Field(
None, description="Path to Google Cloud service account JSON file"
)
@field_validator(
"api_key",
"base_url",
"endpoint",
"api_version",
"endpoint_llm",
"endpoint_embedding",
"endpoint_stt",
"endpoint_tts",
"vertex_project",
"vertex_location",
"vertex_credentials_path",
mode="before",
)
@classmethod
def validate_not_empty_string(cls, v: Optional[str]) -> Optional[str]:
"""Reject empty strings - convert to None or raise error."""
if v is not None:
stripped = v.strip()
if not stripped:
return None # Treat empty/whitespace-only as None
return stripped
return v
class ApiKeyStatusResponse(BaseModel):
"""Response showing which providers are configured and their source."""
configured: Dict[str, bool] = Field(
..., description="Map of provider name to whether it is configured"
)
source: Dict[str, Literal["database", "environment", "none"]] = Field(
...,
description="Map of provider name to configuration source (database, environment, or none)",
)
encryption_configured: bool = Field(
...,
description="Whether OPEN_NOTEBOOK_ENCRYPTION_KEY is set (required to store keys in database)",
)
class TestConnectionResponse(BaseModel):
"""Response from testing a provider connection."""
provider: str = Field(..., description="Provider name that was tested")
success: bool = Field(..., description="Whether connection test succeeded")
message: str = Field(..., description="Result message with details")
class MigrateFromEnvRequest(BaseModel):
"""Request to migrate API keys from environment variables to database."""
force: bool = Field(
False, description="Force overwrite existing database configurations"
)
class MigrationResult(BaseModel):
"""Response from migrating API keys from environment to database."""
message: str = Field(..., description="Summary message")
migrated: List[str] = Field(
default_factory=list, description="Providers successfully migrated"
)
skipped: List[str] = Field(
default_factory=list, description="Providers skipped (already in DB)"
)
errors: List[str] = Field(
default_factory=list, description="Migration errors by provider"
)
# Notebook delete cascade models
# Credential models
# Kept in sync with the provider registry
# (open_notebook/ai/provider_registry.py PROVIDERS — the backend source of
# truth). A Literal can't be built at runtime, so this is the one remaining
# manual copy; tests/test_credential_provider_validation.py enforces the sync.
# The frontend consumes GET /api/providers at runtime and needs no edit.
SupportedProvider = Literal[
"openai",
"anthropic",
"google",
"groq",
"mistral",
"deepseek",
"xai",
"openrouter",
"dashscope",
"minimax",
"voyage",
"elevenlabs",
"deepgram",
"ollama",
"azure",
"vertex",
"openai_compatible",
]
class ProviderInfoResponse(BaseModel):
"""Provider metadata from the provider registry."""
name: str = Field(..., description="Provider identifier (e.g. openai)")
display_name: str = Field(..., description="Human-friendly provider name")
modalities: List[str] = Field(
..., description="Default modalities supported by the provider"
)
docs_url: Optional[str] = Field(
None, description="Where to get an API key / set the provider up"
)
env_configured: bool = Field(
..., description="Whether the provider is configured via environment variables"
)
class CreateCredentialRequest(BaseModel):
"""Request to create a new credential."""
name: str = Field(..., description="Credential name")
provider: SupportedProvider = Field(
..., description="Provider name (openai, anthropic, etc.)"
)
modalities: List[str] = Field(
default_factory=list,
description="Supported modalities (language, embedding, text_to_speech, speech_to_text)",
)
api_key: Optional[str] = Field(None, description="API key (stored encrypted)")
base_url: Optional[str] = Field(None, description="Base URL")
endpoint: Optional[str] = Field(None, description="Endpoint URL (Azure)")
api_version: Optional[str] = Field(None, description="API version (Azure)")
endpoint_llm: Optional[str] = Field(None, description="LLM endpoint")
endpoint_embedding: Optional[str] = Field(None, description="Embedding endpoint")
endpoint_stt: Optional[str] = Field(None, description="STT endpoint")
endpoint_tts: Optional[str] = Field(None, description="TTS endpoint")
project: Optional[str] = Field(None, description="Project ID (Vertex)")
location: Optional[str] = Field(None, description="Location (Vertex)")
credentials_path: Optional[str] = Field(
None, description="Credentials file path (Vertex)"
)
num_ctx: Optional[int] = Field(
None, description="Context window size (Ollama only; defaults to 8192)"
)
class UpdateCredentialRequest(BaseModel):
"""Request to update an existing credential."""
name: Optional[str] = Field(None, description="Credential name")
modalities: Optional[List[str]] = Field(None, description="Supported modalities")
api_key: Optional[str] = Field(None, description="API key (stored encrypted)")
base_url: Optional[str] = Field(None, description="Base URL")
endpoint: Optional[str] = Field(None, description="Endpoint URL")
api_version: Optional[str] = Field(None, description="API version")
endpoint_llm: Optional[str] = Field(None, description="LLM endpoint")
endpoint_embedding: Optional[str] = Field(None, description="Embedding endpoint")
endpoint_stt: Optional[str] = Field(None, description="STT endpoint")
endpoint_tts: Optional[str] = Field(None, description="TTS endpoint")
project: Optional[str] = Field(None, description="Project ID")
location: Optional[str] = Field(None, description="Location")
credentials_path: Optional[str] = Field(None, description="Credentials path")
num_ctx: Optional[int] = Field(
None, description="Context window size (Ollama only; defaults to 8192)"
)
class CredentialResponse(BaseModel):
"""Response for a credential (never includes api_key)."""
id: str
name: str
provider: str
modalities: List[str]
base_url: Optional[str] = None
endpoint: Optional[str] = None
api_version: Optional[str] = None
endpoint_llm: Optional[str] = None
endpoint_embedding: Optional[str] = None
endpoint_stt: Optional[str] = None
endpoint_tts: Optional[str] = None
project: Optional[str] = None
location: Optional[str] = None
credentials_path: Optional[str] = None
num_ctx: Optional[int] = None
has_api_key: bool = False
created: str
updated: str
model_count: int = 0
decryption_error: Optional[str] = None
class CredentialDeleteResponse(BaseModel):
"""Response for credential deletion."""
message: str
deleted_models: int = 0
class DiscoveredModelResponse(BaseModel):
"""A model discovered from a provider."""
name: str
provider: str
model_type: Optional[str] = None
description: Optional[str] = None
class DiscoverModelsResponse(BaseModel):
"""Response from model discovery."""
credential_id: str
provider: str
discovered: List[DiscoveredModelResponse]
class RegisterModelData(BaseModel):
"""A model to register with user-specified type."""
name: str
provider: str
model_type: str # Required: user specifies the type
class RegisterModelsRequest(BaseModel):
"""Request to register discovered models."""
models: List[RegisterModelData]
class RegisterModelsResponse(BaseModel):
"""Response from model registration."""
created: int
existing: int
class NotebookDeletePreview(BaseModel):
notebook_id: str = Field(..., description="ID of the notebook")
notebook_name: str = Field(..., description="Name of the notebook")
note_count: int = Field(..., description="Number of notes that will be deleted")
exclusive_source_count: int = Field(
..., description="Number of sources only in this notebook"
)
shared_source_count: int = Field(
..., description="Number of sources shared with other notebooks"
)
class NotebookDeleteResponse(BaseModel):
message: str = Field(..., description="Success message")
deleted_notes: int = Field(..., description="Number of notes deleted")
deleted_sources: int = Field(..., description="Number of exclusive sources deleted")
unlinked_sources: int = Field(
..., description="Number of sources unlinked from notebook"
)
api/podcast_service.py
+203
View File
@@ -0,0 +1,203 @@
from typing import Any, Dict, Optional
from fastapi import HTTPException
from loguru import logger
from pydantic import BaseModel
from surreal_commands import get_command_status, submit_command
from open_notebook.domain.notebook import Notebook
from open_notebook.podcasts.models import EpisodeProfile, PodcastEpisode, SpeakerProfile
class PodcastGenerationRequest(BaseModel):
"""Request model for podcast generation"""
episode_profile: str
speaker_profile: str
episode_name: str
content: Optional[str] = None
notebook_id: Optional[str] = None
briefing_suffix: Optional[str] = None
class PodcastGenerationResponse(BaseModel):
"""Response model for podcast generation"""
job_id: str
status: str
message: str
episode_profile: str
episode_name: str
class PodcastService:
"""Service layer for podcast operations"""
@staticmethod
async def submit_generation_job(
episode_profile_name: str,
speaker_profile_name: str,
episode_name: str,
notebook_id: Optional[str] = None,
content: Optional[str] = None,
briefing_suffix: Optional[str] = None,
) -> str:
"""Submit a podcast generation job for background processing"""
try:
# Validate episode profile exists
episode_profile = await EpisodeProfile.get_by_name(episode_profile_name)
if not episode_profile:
raise ValueError(f"Episode profile '{episode_profile_name}' not found")
# Resolve the user-facing speaker profile name to a record ID at
# the API boundary (#630) - everything downstream works with IDs.
speaker_profile = await SpeakerProfile.resolve(speaker_profile_name)
if not speaker_profile:
raise ValueError(f"Speaker profile '{speaker_profile_name}' not found")
# Get content from notebook if not provided directly
if not content and notebook_id:
try:
notebook = await Notebook.get(notebook_id)
# Get notebook context (this may need to be adjusted based on actual Notebook implementation)
content = (
await notebook.get_context()
if hasattr(notebook, "get_context")
else str(notebook)
)
except Exception as e:
logger.warning(
f"Failed to get notebook content, using notebook_id as content: {e}"
)
content = f"Notebook ID: {notebook_id}"
if not content:
raise ValueError(
"Content is required - provide either content or notebook_id"
)
# Prepare command arguments (speaker profile as record ID)
command_args = {
"episode_profile": episode_profile_name,
"speaker_profile": str(speaker_profile.id),
"episode_name": episode_name,
"content": str(content),
"briefing_suffix": briefing_suffix,
}
# Ensure command modules are imported before submitting
# This is needed because submit_command validates against local registry
try:
import commands.podcast_commands # noqa: F401
except ImportError as import_err:
logger.error(f"Failed to import podcast commands: {import_err}")
raise ValueError("Podcast commands not available")
# Submit command to surreal-commands
job_id = submit_command("open_notebook", "generate_podcast", command_args)
# Convert RecordID to string if needed
if not job_id:
raise ValueError("Failed to get job_id from submit_command")
job_id_str = str(job_id)
logger.info(
f"Submitted podcast generation job: {job_id_str} for episode '{episode_name}'"
)
return job_id_str
except Exception as e:
logger.error(f"Failed to submit podcast generation job: {e}")
raise HTTPException(
status_code=500,
detail="Failed to submit podcast generation job",
)
@staticmethod
async def get_job_status(job_id: str) -> Dict[str, Any]:
"""Get status of a podcast generation job"""
try:
status = await get_command_status(job_id)
return {
"job_id": job_id,
"status": status.status if status else "unknown",
"result": status.result if status else None,
"error_message": getattr(status, "error_message", None)
if status
else None,
"created": str(status.created)
if status and hasattr(status, "created") and status.created
else None,
"updated": str(status.updated)
if status and hasattr(status, "updated") and status.updated
else None,
"progress": getattr(status, "progress", None) if status else None,
}
except Exception as e:
logger.error(f"Failed to get podcast job status: {e}")
raise HTTPException(status_code=500, detail="Failed to get job status")
@staticmethod
async def list_episodes() -> list:
"""List all podcast episodes"""
try:
episodes = await PodcastEpisode.get_all(order_by="created desc")
return episodes
except Exception as e:
logger.error(f"Failed to list podcast episodes: {e}")
raise HTTPException(status_code=500, detail="Failed to list episodes")
@staticmethod
async def get_episode(episode_id: str) -> PodcastEpisode:
"""Get a specific podcast episode"""
try:
episode = await PodcastEpisode.get(episode_id)
return episode
except Exception as e:
logger.error(f"Failed to get podcast episode {episode_id}: {e}")
raise HTTPException(status_code=404, detail="Episode not found")
class DefaultProfiles:
"""Utility class for creating default profiles (if needed beyond migration data)"""
@staticmethod
async def create_default_episode_profiles():
"""Create default episode profiles if they don't exist"""
try:
# Check if profiles already exist
existing = await EpisodeProfile.get_all()
if existing:
logger.info(f"Episode profiles already exist: {len(existing)} found")
return existing
# This would create profiles, but since we have migration data,
# this is mainly for future extensibility
logger.info(
"Default episode profiles should be created via database migration"
)
return []
except Exception as e:
logger.error(f"Failed to create default episode profiles: {e}")
raise
@staticmethod
async def create_default_speaker_profiles():
"""Create default speaker profiles if they don't exist"""
try:
# Check if profiles already exist
existing = await SpeakerProfile.get_all()
if existing:
logger.info(f"Speaker profiles already exist: {len(existing)} found")
return existing
# This would create profiles, but since we have migration data,
# this is mainly for future extensibility
logger.info(
"Default speaker profiles should be created via database migration"
)
return []
except Exception as e:
logger.error(f"Failed to create default speaker profiles: {e}")
raise
api/routers/__init__.py
View File
api/routers/_chat_shared.py
+94
View File
@@ -0,0 +1,94 @@
"""Shared helpers for the chat and source-chat routers.
Both `api/routers/chat.py` and `api/routers/source_chat.py` operate on
`chat_session` records linked to their parent (notebook or source) via the
`refers_to` relation, and both convert LangGraph state messages into API
response models. This module holds the single definition of those pieces.
Behavior notes:
- The helpers raise exactly what the previously inlined blocks raised
(`NotFoundError` propagates from `ObjectModel.get`, `HTTPException(404)` for
a missing relation), so each router's existing try/except arms keep mapping
them to the same status codes and messages as before.
"""
from typing import Any, Iterable, List, Optional, Tuple
from fastapi import HTTPException
from pydantic import BaseModel, Field
from open_notebook.database.repository import ensure_record_id, repo_query
from open_notebook.domain.notebook import ChatSession, Source
# Shared response models
class ChatMessage(BaseModel):
id: str = Field(..., description="Message ID")
type: str = Field(..., description="Message type (human|ai)")
content: str = Field(..., description="Message content")
timestamp: Optional[str] = Field(None, description="Message timestamp")
class SuccessResponse(BaseModel):
success: bool = Field(True, description="Operation success status")
message: str = Field(..., description="Success message")
def normalize_record_id(table: str, record_id: str) -> str:
"""Ensure a record ID carries its table prefix (`table:id`)."""
prefix = f"{table}:"
return record_id if record_id.startswith(prefix) else f"{prefix}{record_id}"
async def get_source_or_404(source_id: str) -> Tuple[str, Source]:
"""Normalize a source ID and fetch the source, 404 if missing."""
full_source_id = normalize_record_id("source", source_id)
source = await Source.get(full_source_id)
if not source:
raise HTTPException(status_code=404, detail="Source not found")
return full_source_id, source
async def get_session_or_404(session_id: str) -> Tuple[str, ChatSession]:
"""Normalize a session ID and fetch the chat session, 404 if missing."""
full_session_id = normalize_record_id("chat_session", session_id)
session = await ChatSession.get(full_session_id)
if not session:
raise HTTPException(status_code=404, detail="Session not found")
return full_session_id, session
async def get_verified_source_session(
source_id: str, session_id: str
) -> Tuple[str, Source, str, ChatSession]:
"""Verify the source exists, the session exists, and the session refers to
the source. Returns the normalized IDs plus both records."""
full_source_id, source = await get_source_or_404(source_id)
full_session_id, session = await get_session_or_404(session_id)
relation_query = await repo_query(
"SELECT * FROM refers_to WHERE in = $session_id AND out = $source_id",
{
"session_id": ensure_record_id(full_session_id),
"source_id": ensure_record_id(full_source_id),
},
)
if not relation_query:
raise HTTPException(status_code=404, detail="Session not found for this source")
return full_source_id, source, full_session_id, session
def extract_chat_messages(raw_messages: Iterable[Any]) -> List[ChatMessage]:
"""Convert LangGraph/LangChain state messages into `ChatMessage` models."""
messages: List[ChatMessage] = []
for msg in raw_messages:
messages.append(
ChatMessage(
id=getattr(msg, "id", f"msg_{len(messages)}"),
type=msg.type if hasattr(msg, "type") else "unknown",
content=msg.content if hasattr(msg, "content") else str(msg),
timestamp=None, # LangChain messages don't have timestamps by default
)
)
return messages
api/routers/auth.py
+27
View File
@@ -0,0 +1,27 @@
"""
Authentication router for Open Notebook API.
Provides endpoints to check authentication status.
"""
from fastapi import APIRouter
from open_notebook.utils.encryption import get_secret_from_env
router = APIRouter(prefix="/auth", tags=["auth"])
@router.get("/status")
async def get_auth_status():
"""
Check if authentication is enabled.
Returns whether a password is required to access the API.
Supports Docker secrets via OPEN_NOTEBOOK_PASSWORD_FILE.
"""
auth_enabled = bool(get_secret_from_env("OPEN_NOTEBOOK_PASSWORD"))
return {
"auth_enabled": auth_enabled,
"message": "Authentication is required"
if auth_enabled
else "Authentication is disabled",
}
api/routers/chat.py
+416
View File
@@ -0,0 +1,416 @@
import asyncio
import traceback
from typing import Any, Dict, List, Optional
from fastapi import APIRouter, HTTPException, Query
from langchain_core.runnables import RunnableConfig
from loguru import logger
from pydantic import BaseModel, Field
from api.routers._chat_shared import (
ChatMessage,
SuccessResponse,
extract_chat_messages,
get_session_or_404,
)
from open_notebook.database.repository import ensure_record_id, repo_query
from open_notebook.domain.notebook import ChatSession, Notebook
from open_notebook.exceptions import (
NotFoundError,
OpenNotebookError,
)
from open_notebook.graphs.chat import graph as chat_graph
from open_notebook.utils import token_count
from open_notebook.utils.context_builder import build_notebook_context
from open_notebook.utils.graph_utils import get_session_message_count
router = APIRouter()
# Request/Response models
class CreateSessionRequest(BaseModel):
notebook_id: str = Field(..., description="Notebook ID to create session for")
title: Optional[str] = Field(None, description="Optional session title")
model_override: Optional[str] = Field(
None, description="Optional model override for this session"
)
class UpdateSessionRequest(BaseModel):
title: Optional[str] = Field(None, description="New session title")
model_override: Optional[str] = Field(
None, description="Model override for this session"
)
class ChatSessionResponse(BaseModel):
id: str = Field(..., description="Session ID")
title: str = Field(..., description="Session title")
notebook_id: Optional[str] = Field(None, description="Notebook ID")
created: str = Field(..., description="Creation timestamp")
updated: str = Field(..., description="Last update timestamp")
message_count: Optional[int] = Field(
None, description="Number of messages in session"
)
model_override: Optional[str] = Field(
None, description="Model override for this session"
)
class ChatSessionWithMessagesResponse(ChatSessionResponse):
messages: List[ChatMessage] = Field(
default_factory=list, description="Session messages"
)
class ExecuteChatRequest(BaseModel):
session_id: str = Field(..., description="Chat session ID")
message: str = Field(..., description="User message content")
context: Dict[str, Any] = Field(
..., description="Chat context with sources and notes"
)
model_override: Optional[str] = Field(
None, description="Optional model override for this message"
)
class ExecuteChatResponse(BaseModel):
session_id: str = Field(..., description="Session ID")
messages: List[ChatMessage] = Field(..., description="Updated message list")
class BuildContextRequest(BaseModel):
notebook_id: str = Field(..., description="Notebook ID")
context_config: Dict[str, Any] = Field(..., description="Context configuration")
class BuildContextResponse(BaseModel):
context: Dict[str, Any] = Field(..., description="Built context data")
token_count: int = Field(..., description="Estimated token count")
char_count: int = Field(..., description="Character count")
@router.get("/chat/sessions", response_model=List[ChatSessionResponse])
async def get_sessions(notebook_id: str = Query(..., description="Notebook ID")):
"""Get all chat sessions for a notebook."""
try:
# Get notebook to verify it exists
notebook = await Notebook.get(notebook_id)
if not notebook:
raise HTTPException(status_code=404, detail="Notebook not found")
# Get sessions for this notebook
sessions_list = await notebook.get_chat_sessions()
results = []
for session in sessions_list:
session_id = str(session.id)
# Get message count from LangGraph state
msg_count = await get_session_message_count(chat_graph, session_id)
results.append(
ChatSessionResponse(
id=session.id or "",
title=session.title or "Untitled Session",
notebook_id=notebook_id,
created=str(session.created),
updated=str(session.updated),
message_count=msg_count,
model_override=getattr(session, "model_override", None),
)
)
return results
except NotFoundError:
raise HTTPException(status_code=404, detail="Notebook not found")
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching chat sessions: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error fetching chat sessions: {str(e)}"
)
@router.post("/chat/sessions", response_model=ChatSessionResponse)
async def create_session(request: CreateSessionRequest):
"""Create a new chat session."""
try:
# Verify notebook exists
notebook = await Notebook.get(request.notebook_id)
if not notebook:
raise HTTPException(status_code=404, detail="Notebook not found")
# Create new session
session = ChatSession(
title=request.title
or f"Chat Session {asyncio.get_event_loop().time():.0f}",
model_override=request.model_override,
)
await session.save()
# Relate session to notebook
await session.relate_to_notebook(request.notebook_id)
return ChatSessionResponse(
id=session.id or "",
title=session.title or "",
notebook_id=request.notebook_id,
created=str(session.created),
updated=str(session.updated),
message_count=0,
model_override=session.model_override,
)
except NotFoundError:
raise HTTPException(status_code=404, detail="Notebook not found")
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error creating chat session: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error creating chat session: {str(e)}"
)
@router.get(
"/chat/sessions/{session_id}", response_model=ChatSessionWithMessagesResponse
)
async def get_session(session_id: str):
"""Get a specific session with its messages."""
try:
# Get session (normalizes the ID and 404s if missing)
full_session_id, session = await get_session_or_404(session_id)
# Get session state from LangGraph to retrieve messages
# Use sync get_state() in a thread since SqliteSaver doesn't support async
thread_state = await asyncio.to_thread(
chat_graph.get_state,
config=RunnableConfig(configurable={"thread_id": full_session_id}),
)
# Extract messages from state
messages: list[ChatMessage] = []
if thread_state and thread_state.values and "messages" in thread_state.values:
messages = extract_chat_messages(thread_state.values["messages"])
# Find notebook_id (we need to query the relationship)
notebook_query = await repo_query(
"SELECT out FROM refers_to WHERE in = $session_id",
{"session_id": ensure_record_id(full_session_id)},
)
notebook_id = notebook_query[0]["out"] if notebook_query else None
if not notebook_id:
# This might be an old session created before API migration
logger.warning(
f"No notebook relationship found for session {session_id} - may be an orphaned session"
)
return ChatSessionWithMessagesResponse(
id=session.id or "",
title=session.title or "Untitled Session",
notebook_id=notebook_id,
created=str(session.created),
updated=str(session.updated),
message_count=len(messages),
messages=messages,
model_override=getattr(session, "model_override", None),
)
except NotFoundError:
raise HTTPException(status_code=404, detail="Session not found")
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching session: {str(e)}")
raise HTTPException(status_code=500, detail=f"Error fetching session: {str(e)}")
@router.put("/chat/sessions/{session_id}", response_model=ChatSessionResponse)
async def update_session(session_id: str, request: UpdateSessionRequest):
"""Update session title."""
try:
# Get session (normalizes the ID and 404s if missing)
full_session_id, session = await get_session_or_404(session_id)
update_data = request.model_dump(exclude_unset=True)
if "title" in update_data:
session.title = update_data["title"]
if "model_override" in update_data:
session.model_override = update_data["model_override"]
await session.save()
# Find notebook_id
notebook_query = await repo_query(
"SELECT out FROM refers_to WHERE in = $session_id",
{"session_id": ensure_record_id(full_session_id)},
)
notebook_id = notebook_query[0]["out"] if notebook_query else None
# Get message count from LangGraph state
msg_count = await get_session_message_count(chat_graph, full_session_id)
return ChatSessionResponse(
id=session.id or "",
title=session.title or "",
notebook_id=notebook_id,
created=str(session.created),
updated=str(session.updated),
message_count=msg_count,
model_override=session.model_override,
)
except NotFoundError:
raise HTTPException(status_code=404, detail="Session not found")
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error updating session: {str(e)}")
raise HTTPException(status_code=500, detail=f"Error updating session: {str(e)}")
@router.delete("/chat/sessions/{session_id}", response_model=SuccessResponse)
async def delete_session(session_id: str):
"""Delete a chat session."""
try:
# Get session (normalizes the ID and 404s if missing)
_full_session_id, session = await get_session_or_404(session_id)
await session.delete()
return SuccessResponse(success=True, message="Session deleted successfully")
except NotFoundError:
raise HTTPException(status_code=404, detail="Session not found")
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error deleting session: {str(e)}")
raise HTTPException(status_code=500, detail=f"Error deleting session: {str(e)}")
@router.post("/chat/execute", response_model=ExecuteChatResponse)
async def execute_chat(request: ExecuteChatRequest):
"""Execute a chat request and get AI response."""
try:
# Verify session exists (normalizes the ID and 404s if missing)
full_session_id, session = await get_session_or_404(request.session_id)
# Fetch notebook linked to this session
notebook_query = await repo_query(
"SELECT out FROM refers_to WHERE in = $session_id",
{"session_id": ensure_record_id(full_session_id)},
)
notebook = None
if notebook_query:
notebook = await Notebook.get(notebook_query[0]["out"])
# Determine model override (per-request override takes precedence over session-level)
model_override = (
request.model_override
if request.model_override is not None
else getattr(session, "model_override", None)
)
# Get current state
# Use sync get_state() in a thread since SqliteSaver doesn't support async
current_state = await asyncio.to_thread(
chat_graph.get_state,
config=RunnableConfig(configurable={"thread_id": full_session_id}),
)
# Prepare state for execution
state_values = current_state.values if current_state else {}
state_values["messages"] = state_values.get("messages", [])
state_values["context"] = request.context
state_values["notebook"] = notebook
state_values["model_override"] = model_override
# Add user message to state
from langchain_core.messages import HumanMessage
user_message = HumanMessage(content=request.message)
state_values["messages"].append(user_message)
# Execute chat graph in a thread so the synchronous LangGraph invoke
# (SqliteSaver checkpoints are sync) doesn't block the event loop and
# freeze the rest of the API while the LLM responds. Mirrors the
# get_state() calls above.
# The lambda pins down which `invoke` overload is used; asyncio.to_thread
# can't resolve overloaded callables on its own. The ignore is a langgraph
# typing limitation: it accepts a partial state dict at runtime, but the
# signature requires the full state type.
result = await asyncio.to_thread(
lambda: chat_graph.invoke(
input=state_values, # type: ignore[arg-type]
config=RunnableConfig(
configurable={
"thread_id": full_session_id,
"model_id": model_override,
}
),
)
)
# Update session timestamp
await session.save()
# Convert messages to response format
messages = extract_chat_messages(result.get("messages", []))
return ExecuteChatResponse(session_id=request.session_id, messages=messages)
except NotFoundError:
raise HTTPException(status_code=404, detail="Session not found")
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
# Log detailed error with context for debugging
logger.error(
f"Error executing chat: {str(e)}\n"
f" Session ID: {request.session_id}\n"
f" Model override: {request.model_override}\n"
f" Traceback:\n{traceback.format_exc()}"
)
raise HTTPException(status_code=500, detail=f"Error executing chat: {str(e)}")
@router.post("/chat/context", response_model=BuildContextResponse)
async def build_context(request: BuildContextRequest):
"""Build context for a notebook based on context configuration."""
try:
# Verify notebook exists
notebook = await Notebook.get(request.notebook_id)
if not notebook:
raise HTTPException(status_code=404, detail="Notebook not found")
context_data, total_content = await build_notebook_context(
notebook, request.context_config
)
char_count = len(total_content)
estimated_tokens = token_count(total_content) if total_content else 0
return BuildContextResponse(
context=context_data, token_count=estimated_tokens, char_count=char_count
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error building context: {str(e)}")
raise HTTPException(status_code=500, detail=f"Error building context: {str(e)}")
api/routers/commands.py
+185
View File
@@ -0,0 +1,185 @@
from typing import Any, Dict, List, Optional
from fastapi import APIRouter, HTTPException, Query
from loguru import logger
from pydantic import BaseModel, Field
from surreal_commands import registry
from api.command_service import CommandService
from open_notebook.exceptions import OpenNotebookError
router = APIRouter()
class CommandExecutionRequest(BaseModel):
command: str = Field(
..., description="Command function name (e.g., 'generate_podcast')"
)
app: str = Field(..., description="Application name (e.g., 'open_notebook')")
input: Dict[str, Any] = Field(..., description="Arguments to pass to the command")
class CommandJobResponse(BaseModel):
job_id: str
status: str
message: str
class CommandJobStatusResponse(BaseModel):
job_id: str
status: str
result: Optional[Dict[str, Any]] = None
error_message: Optional[str] = None
created: Optional[str] = None
updated: Optional[str] = None
progress: Optional[Dict[str, Any]] = None
@router.post("/commands/jobs", response_model=CommandJobResponse)
async def execute_command(request: CommandExecutionRequest):
"""
Submit a command for background processing.
Returns immediately with job ID for status tracking.
Example request:
{
"command": "generate_podcast",
"app": "open_notebook",
"input": {
"episode_profile": "tech_experts",
"speaker_profile": "tech_experts",
"episode_name": "My Episode",
"content": "Content to discuss"
}
}
"""
try:
# Submit command using app name (not module name)
job_id = await CommandService.submit_command_job(
module_name=request.app, # This should be "open_notebook"
command_name=request.command,
command_args=request.input,
)
return CommandJobResponse(
job_id=job_id,
status="submitted",
message=f"Command '{request.command}' submitted successfully",
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error submitting command: {str(e)}")
raise HTTPException(
status_code=500, detail="Failed to submit command"
)
@router.get("/commands/jobs/{job_id}", response_model=CommandJobStatusResponse)
async def get_command_job_status(job_id: str):
"""Get the status of a specific command job"""
try:
status_data = await CommandService.get_command_status(job_id)
return CommandJobStatusResponse(**status_data)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching job status: {str(e)}")
raise HTTPException(
status_code=500, detail="Failed to fetch job status"
)
@router.get("/commands/jobs", response_model=List[Dict[str, Any]])
async def list_command_jobs(
command_filter: Optional[str] = Query(None, description="Filter by command name"),
status_filter: Optional[str] = Query(None, description="Filter by status"),
limit: int = Query(50, description="Maximum number of jobs to return"),
):
"""List command jobs with optional filtering"""
try:
jobs = await CommandService.list_command_jobs(
command_filter=command_filter, status_filter=status_filter, limit=limit
)
return jobs
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error listing command jobs: {str(e)}")
raise HTTPException(
status_code=500, detail="Failed to list command jobs"
)
@router.delete("/commands/jobs/{job_id}")
async def cancel_command_job(job_id: str):
"""Cancel a running command job"""
try:
success = await CommandService.cancel_command_job(job_id)
return {"job_id": job_id, "cancelled": success}
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error cancelling command job: {str(e)}")
raise HTTPException(
status_code=500, detail="Failed to cancel command job"
)
@router.get("/commands/registry/debug")
async def debug_registry():
"""Debug endpoint to see what commands are registered"""
try:
# Get all registered commands
all_items = registry.get_all_commands()
# Create JSON-serializable data
command_items = []
for item in all_items:
try:
command_items.append(
{
"app_id": item.app_id,
"name": item.name,
"full_id": f"{item.app_id}.{item.name}",
}
)
except Exception as item_error:
logger.error(f"Error processing item: {item_error}")
# Get the basic command structure
try:
commands_dict: dict[str, list[str]] = {}
for item in all_items:
if item.app_id not in commands_dict:
commands_dict[item.app_id] = []
commands_dict[item.app_id].append(item.name)
except Exception:
commands_dict = {}
return {
"total_commands": len(all_items),
"commands_by_app": commands_dict,
"command_items": command_items,
}
except Exception as e:
logger.error(f"Error debugging registry: {str(e)}")
return {
"error": str(e),
"total_commands": 0,
"commands_by_app": {},
"command_items": [],
}
api/routers/config.py
+159
View File
@@ -0,0 +1,159 @@
import asyncio
import time
import tomllib
from pathlib import Path
from typing import Optional
from fastapi import APIRouter, Request
from loguru import logger
from open_notebook.database.repository import repo_query
from open_notebook.utils.version_utils import (
compare_versions,
get_version_from_github_async,
)
router = APIRouter()
# In-memory cache for version check results
_version_cache: dict = {
"latest_version": None,
"has_update": False,
"timestamp": 0,
"check_failed": False,
}
# Cache TTL in seconds (24 hours)
VERSION_CACHE_TTL = 24 * 60 * 60
def get_version() -> str:
"""Read version from pyproject.toml"""
try:
pyproject_path = Path(__file__).parent.parent.parent / "pyproject.toml"
with open(pyproject_path, "rb") as f:
pyproject = tomllib.load(f)
return pyproject.get("project", {}).get("version", "unknown")
except Exception as e:
logger.warning(f"Could not read version from pyproject.toml: {e}")
return "unknown"
async def get_latest_version_cached(current_version: str) -> tuple[Optional[str], bool]:
"""
Check for the latest version from GitHub with caching.
Returns:
tuple: (latest_version, has_update)
- latest_version: str or None if check failed
- has_update: bool indicating if update is available
"""
global _version_cache
# Check if cache is still valid (within TTL)
cache_age = time.time() - _version_cache["timestamp"]
if _version_cache["timestamp"] > 0 and cache_age < VERSION_CACHE_TTL:
logger.debug(f"Using cached version check result (age: {cache_age:.0f}s)")
return _version_cache["latest_version"], _version_cache["has_update"]
# Cache expired or not yet set
if _version_cache["timestamp"] > 0:
logger.info(f"Version cache expired (age: {cache_age:.0f}s), refreshing...")
# Perform version check with strict error handling
try:
logger.info("Checking for latest version from GitHub...")
# Fetch latest version from GitHub with 10-second timeout
latest_version = await get_version_from_github_async(
"https://github.com/lfnovo/open-notebook", "main"
)
logger.info(
f"Latest version from GitHub: {latest_version}, Current version: {current_version}"
)
# Compare versions
has_update = compare_versions(current_version, latest_version) < 0
# Cache the result
_version_cache["latest_version"] = latest_version
_version_cache["has_update"] = has_update
_version_cache["timestamp"] = time.time()
_version_cache["check_failed"] = False
logger.info(f"Version check complete. Update available: {has_update}")
return latest_version, has_update
except Exception as e:
logger.warning(f"Version check failed: {e}")
# Cache the failure to avoid repeated attempts
_version_cache["latest_version"] = None
_version_cache["has_update"] = False
_version_cache["timestamp"] = time.time()
_version_cache["check_failed"] = True
return None, False
async def check_database_health() -> dict:
"""
Check if database is reachable using a lightweight query.
Returns:
dict with 'status' ("online" | "offline") and optional 'error'
"""
try:
# 2-second timeout for database health check
result = await asyncio.wait_for(repo_query("RETURN 1"), timeout=2.0)
if result:
return {"status": "online"}
return {"status": "offline", "error": "Empty result"}
except asyncio.TimeoutError:
logger.warning("Database health check timed out after 2 seconds")
return {"status": "offline", "error": "Health check timeout"}
except Exception as e:
logger.warning(f"Database health check failed: {e}")
return {"status": "offline", "error": str(e)}
@router.get("/config")
async def get_config(request: Request):
"""
Get frontend configuration.
Returns version information and health status.
Note: The frontend determines the API URL via its own runtime-config endpoint,
so this endpoint no longer returns apiUrl.
Also checks for version updates from GitHub (with caching and error handling).
"""
# Get current version
current_version = get_version()
# Check for updates (with caching and error handling)
# This MUST NOT break the endpoint - wrapped in try-except as extra safety
latest_version = None
has_update = False
try:
latest_version, has_update = await get_latest_version_cached(current_version)
except Exception as e:
# Extra safety: ensure version check never breaks the config endpoint
logger.error(f"Unexpected error during version check: {e}")
# Check database health
db_health = await check_database_health()
db_status = db_health["status"]
if db_status == "offline":
logger.warning(f"Database offline: {db_health.get('error', 'Unknown error')}")
return {
"version": current_version,
"latestVersion": latest_version,
"hasUpdate": has_update,
"dbStatus": db_status,
}
api/routers/credentials.py
+493
View File
@@ -0,0 +1,493 @@
"""
Credentials Router
Thin HTTP layer for managing individual AI provider credentials.
Business logic lives in api.credentials_service.
Endpoints:
- GET /credentials - List all credentials
- GET /credentials/by-provider/{provider} - List credentials for a provider
- POST /credentials - Create a new credential
- GET /credentials/{credential_id} - Get a specific credential
- PUT /credentials/{credential_id} - Update a credential
- DELETE /credentials/{credential_id} - Delete a credential
- POST /credentials/{credential_id}/test - Test connection
- POST /credentials/{credential_id}/discover - Discover models
- POST /credentials/{credential_id}/register-models - Register models
NEVER returns actual API key values - only metadata.
"""
from typing import List, Optional
from fastapi import APIRouter, HTTPException, Query
from loguru import logger
from pydantic import SecretStr
from api.credentials_service import (
credential_to_response,
discover_with_config,
get_provider_status,
register_models,
require_encryption_key,
validate_url,
)
from api.credentials_service import (
get_env_status as svc_get_env_status,
)
from api.credentials_service import (
migrate_from_env as svc_migrate_from_env,
)
from api.credentials_service import (
migrate_from_provider_config as svc_migrate_from_provider_config,
)
from api.credentials_service import (
test_credential as svc_test_credential,
)
from api.models import (
CreateCredentialRequest,
CredentialDeleteResponse,
CredentialResponse,
DiscoveredModelResponse,
DiscoverModelsResponse,
RegisterModelsRequest,
RegisterModelsResponse,
UpdateCredentialRequest,
)
from open_notebook.database.repository import ensure_record_id, repo_delete, repo_query
from open_notebook.domain.credential import Credential
from open_notebook.exceptions import (
NotFoundError,
OpenNotebookError,
)
router = APIRouter(prefix="/credentials", tags=["credentials"])
def _handle_value_error(e: ValueError, status_code: int = 400) -> HTTPException:
"""Convert a ValueError from the service layer to an HTTPException."""
return HTTPException(status_code=status_code, detail=str(e))
# =============================================================================
# Status endpoints
# =============================================================================
@router.get("/status")
async def get_status():
"""
Get configuration status: encryption key status, and per-provider
configured/source information.
"""
try:
return await get_provider_status()
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching status: {e}")
raise HTTPException(status_code=500, detail="Failed to fetch credential status")
@router.get("/env-status")
async def get_env_status():
"""Check what's configured via environment variables."""
try:
return await svc_get_env_status()
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error checking env status: {e}")
raise HTTPException(status_code=500, detail="Failed to check environment status")
# =============================================================================
# CRUD endpoints
# =============================================================================
@router.get("", response_model=List[CredentialResponse])
async def list_credentials(
provider: Optional[str] = Query(None, description="Filter by provider"),
):
"""List all credentials, optionally filtered by provider."""
try:
if provider:
credentials = await Credential.get_by_provider(provider)
else:
credentials = await Credential.get_all(order_by="provider, created")
result = []
for cred in credentials:
models = await cred.get_linked_models()
result.append(credential_to_response(cred, len(models)))
return result
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error listing credentials: {e}")
raise HTTPException(status_code=500, detail="Failed to list credentials")
@router.get("/by-provider/{provider}", response_model=List[CredentialResponse])
async def list_credentials_by_provider(provider: str):
"""List all credentials for a specific provider."""
try:
credentials = await Credential.get_by_provider(provider.lower())
result = []
for cred in credentials:
models = await cred.get_linked_models()
result.append(credential_to_response(cred, len(models)))
return result
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error listing credentials for {provider}: {e}")
raise HTTPException(status_code=500, detail="Failed to list credentials for provider")
@router.post("", response_model=CredentialResponse, status_code=201)
async def create_credential(request: CreateCredentialRequest):
"""Create a new credential."""
try:
require_encryption_key()
except ValueError as e:
raise _handle_value_error(e)
# Validate all URL fields
for url_field in [
request.base_url, request.endpoint, request.endpoint_llm,
request.endpoint_embedding, request.endpoint_stt, request.endpoint_tts,
]:
if url_field:
try:
await validate_url(url_field, request.provider)
except ValueError as e:
raise _handle_value_error(e)
try:
cred = Credential(
name=request.name,
provider=request.provider.lower(),
modalities=request.modalities,
api_key=SecretStr(request.api_key) if request.api_key else None,
base_url=request.base_url,
endpoint=request.endpoint,
api_version=request.api_version,
endpoint_llm=request.endpoint_llm,
endpoint_embedding=request.endpoint_embedding,
endpoint_stt=request.endpoint_stt,
endpoint_tts=request.endpoint_tts,
project=request.project,
location=request.location,
credentials_path=request.credentials_path,
num_ctx=request.num_ctx,
)
await cred.save()
return credential_to_response(cred, 0)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error creating credential: {e}")
raise HTTPException(status_code=500, detail="Failed to create credential")
@router.get("/{credential_id}", response_model=CredentialResponse)
async def get_credential(credential_id: str):
"""Get a specific credential by ID. Never returns api_key."""
try:
cred = await Credential.get(credential_id)
models = await cred.get_linked_models()
return credential_to_response(cred, len(models))
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching credential {credential_id}: {e}")
raise HTTPException(status_code=404, detail="Credential not found")
@router.put("/{credential_id}", response_model=CredentialResponse)
async def update_credential(credential_id: str, request: UpdateCredentialRequest):
"""Update an existing credential."""
try:
require_encryption_key()
except ValueError as e:
raise _handle_value_error(e)
# Validate all URL fields being updated
for url_field in [
request.base_url, request.endpoint, request.endpoint_llm,
request.endpoint_embedding, request.endpoint_stt, request.endpoint_tts,
]:
if url_field:
try:
await validate_url(url_field, "update")
except ValueError as e:
raise _handle_value_error(e)
try:
cred = await Credential.get(credential_id)
# Partial-update semantics keyed on field PRESENCE, not value:
# a field absent from the payload is left untouched, while an explicit
# null (or "") clears it. `is not None` checks would silently ignore
# a null sent to clear a field — the old value survived while the
# client saw success.
sent = request.model_fields_set
if request.name is not None:
cred.name = request.name
if request.modalities is not None:
cred.modalities = request.modalities
if request.api_key is not None:
cred.api_key = SecretStr(request.api_key)
if "base_url" in sent:
cred.base_url = request.base_url or None
if "endpoint" in sent:
cred.endpoint = request.endpoint or None
if "api_version" in sent:
cred.api_version = request.api_version or None
if "endpoint_llm" in sent:
cred.endpoint_llm = request.endpoint_llm or None
if "endpoint_embedding" in sent:
cred.endpoint_embedding = request.endpoint_embedding or None
if "endpoint_stt" in sent:
cred.endpoint_stt = request.endpoint_stt or None
if "endpoint_tts" in sent:
cred.endpoint_tts = request.endpoint_tts or None
if "project" in sent:
cred.project = request.project or None
if "location" in sent:
cred.location = request.location or None
if "credentials_path" in sent:
cred.credentials_path = request.credentials_path or None
if "num_ctx" in sent:
# 0/null/falsy clears the override and falls back to esperanto's default
cred.num_ctx = request.num_ctx or None
await cred.save()
models = await cred.get_linked_models()
return credential_to_response(cred, len(models))
except HTTPException:
raise
except NotFoundError:
raise HTTPException(status_code=404, detail="Credential not found")
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error updating credential {credential_id}: {e}")
raise HTTPException(status_code=500, detail="Failed to update credential")
@router.delete("/{credential_id}", response_model=CredentialDeleteResponse)
async def delete_credential(
credential_id: str,
migrate_to: Optional[str] = Query(
None, description="Migrate linked models to this credential ID"
),
):
"""
Delete a credential.
If the credential has linked models:
- Pass migrate_to=<credential_id> to reassign them to another credential
- Otherwise, linked models are cascade-deleted automatically
"""
try:
try:
cred = await Credential.get(credential_id)
except ValueError as decrypt_err:
# Credential exists but can't be decrypted (wrong encryption key).
# Fall back to direct DB operations for deletion.
logger.warning(
f"Cannot decrypt credential {credential_id}, "
f"falling back to direct delete: {decrypt_err}"
)
# Query linked models
linked = await repo_query(
"SELECT * FROM model WHERE credential = $cred_id",
{"cred_id": ensure_record_id(credential_id)},
)
deleted_models = 0
if linked and migrate_to:
# Migrate models to another credential
target_cred = await Credential.get(migrate_to)
for model_row in linked:
model_id = str(model_row.get("id", ""))
if model_id:
await repo_query(
"UPDATE $model_id SET credential = $target_id",
{
"model_id": ensure_record_id(model_id),
# A fetched credential always has an id; fall
# back to the requested id for the type checker.
"target_id": ensure_record_id(
target_cred.id or migrate_to
),
},
)
elif linked:
# Cascade-delete linked models
for model_row in linked:
model_id = str(model_row.get("id", ""))
if model_id:
await repo_delete(model_id)
deleted_models += 1
# Delete the credential itself
await repo_delete(credential_id)
return CredentialDeleteResponse(
message="Credential deleted successfully",
deleted_models=deleted_models,
)
linked_models = await cred.get_linked_models()
deleted_models = 0
if linked_models and migrate_to:
# Migrate models to another credential
target_cred = await Credential.get(migrate_to)
for model in linked_models:
model.credential = target_cred.id
await model.save()
elif linked_models:
# Cascade-delete linked models (default behavior when no migrate_to)
for model in linked_models:
await model.delete()
deleted_models += 1
# Delete the credential
await cred.delete()
return CredentialDeleteResponse(
message="Credential deleted successfully",
deleted_models=deleted_models,
)
except HTTPException:
raise
except NotFoundError:
raise HTTPException(status_code=404, detail="Credential not found")
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error deleting credential {credential_id}: {e}")
raise HTTPException(status_code=500, detail="Failed to delete credential")
# =============================================================================
# Test / Discover / Register endpoints
# =============================================================================
@router.post("/{credential_id}/test")
async def test_credential(credential_id: str):
"""Test connection using this credential's configuration."""
return await svc_test_credential(credential_id)
@router.post("/{credential_id}/discover", response_model=DiscoverModelsResponse)
async def discover_models_for_credential(credential_id: str):
"""Discover available models using this credential's API key."""
try:
cred = await Credential.get(credential_id)
config = cred.to_esperanto_config()
provider = cred.provider.lower()
discovered = await discover_with_config(provider, config)
return DiscoverModelsResponse(
credential_id=cred.id or "",
provider=provider,
discovered=[
DiscoveredModelResponse(
name=d["name"],
provider=d["provider"],
description=d.get("description"),
)
for d in discovered
],
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error discovering models for credential {credential_id}: {e}")
raise HTTPException(status_code=500, detail="Failed to discover models")
@router.post("/{credential_id}/register-models", response_model=RegisterModelsResponse)
async def register_models_for_credential(
credential_id: str, request: RegisterModelsRequest
):
"""Register discovered models and link them to this credential."""
try:
result = await register_models(credential_id, request.models)
return RegisterModelsResponse(**result)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error registering models for credential {credential_id}: {e}")
raise HTTPException(status_code=500, detail="Failed to register models")
# =============================================================================
# Migration endpoints
# =============================================================================
@router.post("/migrate-from-provider-config")
async def migrate_from_provider_config():
"""Migrate existing ProviderConfig data to individual credential records."""
try:
return await svc_migrate_from_provider_config()
except ValueError as e:
raise _handle_value_error(e)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"ProviderConfig migration FAILED: {type(e).__name__}: {e}", exc_info=True)
raise HTTPException(status_code=500, detail="Migration from provider config failed")
@router.post("/migrate-from-env")
async def migrate_from_env():
"""Migrate API keys from environment variables to credential records."""
try:
return await svc_migrate_from_env()
except ValueError as e:
raise _handle_value_error(e)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Env migration FAILED: {type(e).__name__}: {e}", exc_info=True)
raise HTTPException(status_code=500, detail="Migration from environment variables failed")
api/routers/embedding.py
+129
View File
@@ -0,0 +1,129 @@
from fastapi import APIRouter, HTTPException
from loguru import logger
from api.command_service import CommandService
from api.models import EmbedRequest, EmbedResponse
from open_notebook.ai.models import model_manager
from open_notebook.domain.notebook import Note, Source
from open_notebook.exceptions import (
NotFoundError,
OpenNotebookError,
)
router = APIRouter()
@router.post("/embed", response_model=EmbedResponse)
async def embed_content(embed_request: EmbedRequest):
"""Embed content for vector search."""
try:
# Check if embedding model is available
if not await model_manager.get_embedding_model():
raise HTTPException(
status_code=400,
detail="No embedding model configured. Please configure one in the Models section.",
)
item_id = embed_request.item_id
item_type = embed_request.item_type.lower()
# Validate item type
if item_type not in ["source", "note"]:
raise HTTPException(
status_code=400, detail="Item type must be either 'source' or 'note'"
)
# Branch based on processing mode
if embed_request.async_processing:
# ASYNC PATH: Submit command for background processing
logger.info(f"Using async processing for {item_type} {item_id}")
try:
# Import commands to ensure they're registered
import commands.embedding_commands # noqa: F401
# Submit type-specific command
if item_type == "source":
command_name = "embed_source"
command_input = {"source_id": item_id}
else: # note
command_name = "embed_note"
command_input = {"note_id": item_id}
command_id = await CommandService.submit_command_job(
"open_notebook",
command_name,
command_input,
)
logger.info(f"Submitted async {command_name} command: {command_id}")
return EmbedResponse(
success=True,
message="Embedding queued for background processing",
item_id=item_id,
item_type=item_type,
command_id=command_id,
)
except Exception as e:
logger.error(f"Failed to submit async embedding command: {e}")
raise HTTPException(
status_code=500, detail=f"Failed to queue embedding: {str(e)}"
)
else:
# DOMAIN MODEL PATH: Submit job via domain model convenience methods
# These methods internally call submit_command() - still fire-and-forget
logger.info(f"Using domain model path for {item_type} {item_id}")
command_id = None
# Get the item and submit embedding job
if item_type == "source":
source_item = await Source.get(item_id)
# Submit embed_source job (returns command_id for tracking)
command_id = await source_item.vectorize()
message = "Source embedding job submitted"
elif item_type == "note":
note_item = await Note.get(item_id)
# Note.save() internally submits embed_note command and
# returns command_id. Unlike Source.vectorize(), save()'s
# embed submission is best-effort (a hiccup there shouldn't
# fail an otherwise-successful note save) - but this
# endpoint's whole point is submitting the embedding job,
# so a submission failure here (content present, no
# command_id) must still surface as a failure.
command_id = await note_item.save()
if not command_id and note_item.content and note_item.content.strip():
raise HTTPException(
status_code=500, detail="Failed to submit note embedding job"
)
message = "Note embedding job submitted"
return EmbedResponse(
success=True,
message=message,
item_id=item_id,
item_type=item_type,
command_id=command_id,
)
except HTTPException:
raise
except NotFoundError:
raise HTTPException(
status_code=404, detail=f"{embed_request.item_type} not found"
)
except OpenNotebookError:
raise
except Exception as e:
logger.error(
f"Error embedding {embed_request.item_type} {embed_request.item_id}: {str(e)}"
)
raise HTTPException(
status_code=500, detail=f"Error embedding content: {str(e)}"
)
api/routers/embedding_rebuild.py
+199
View File
@@ -0,0 +1,199 @@
from fastapi import APIRouter, HTTPException
from loguru import logger
from surreal_commands import get_command_status
from api.command_service import CommandService
from api.models import (
RebuildProgress,
RebuildRequest,
RebuildResponse,
RebuildStats,
RebuildStatusResponse,
)
from open_notebook.database.repository import repo_query
from open_notebook.exceptions import OpenNotebookError
router = APIRouter()
@router.post("/rebuild", response_model=RebuildResponse)
async def start_rebuild(request: RebuildRequest):
"""
Start a background job to rebuild embeddings.
- **mode**: "existing" (re-embed items with embeddings) or "all" (embed everything)
- **include_sources**: Include sources in rebuild (default: true)
- **include_notes**: Include notes in rebuild (default: true)
- **include_insights**: Include insights in rebuild (default: true)
Returns command ID to track progress and estimated item count.
"""
try:
logger.info(f"Starting rebuild request: mode={request.mode}")
# Import commands to ensure they're registered
import commands.embedding_commands # noqa: F401
# Estimate total items (quick count query)
# This is a rough estimate before the command runs
total_estimate = 0
if request.include_sources:
if request.mode == "existing":
# Count sources with embeddings
result = await repo_query(
"""
SELECT VALUE count(array::distinct(
SELECT VALUE source.id
FROM source_embedding
WHERE embedding != none AND array::len(embedding) > 0
)) as count FROM {}
"""
)
else:
# Count all sources with content
result = await repo_query(
"SELECT VALUE count() as count FROM source WHERE full_text != none GROUP ALL"
)
if result and isinstance(result[0], dict):
total_estimate += result[0].get("count", 0)
elif result:
total_estimate += result[0] if isinstance(result[0], int) else 0
if request.include_notes:
if request.mode == "existing":
result = await repo_query(
"SELECT VALUE count() as count FROM note WHERE embedding != none AND array::len(embedding) > 0 GROUP ALL"
)
else:
result = await repo_query(
"SELECT VALUE count() as count FROM note WHERE content != none GROUP ALL"
)
if result and isinstance(result[0], dict):
total_estimate += result[0].get("count", 0)
elif result:
total_estimate += result[0] if isinstance(result[0], int) else 0
if request.include_insights:
if request.mode == "existing":
result = await repo_query(
"SELECT VALUE count() as count FROM source_insight WHERE embedding != none AND array::len(embedding) > 0 GROUP ALL"
)
else:
result = await repo_query(
"SELECT VALUE count() as count FROM source_insight GROUP ALL"
)
if result and isinstance(result[0], dict):
total_estimate += result[0].get("count", 0)
elif result:
total_estimate += result[0] if isinstance(result[0], int) else 0
logger.info(f"Estimated {total_estimate} items to process")
# Submit command
command_id = await CommandService.submit_command_job(
"open_notebook",
"rebuild_embeddings",
{
"mode": request.mode,
"include_sources": request.include_sources,
"include_notes": request.include_notes,
"include_insights": request.include_insights,
},
)
logger.info(f"Submitted rebuild command: {command_id}")
return RebuildResponse(
command_id=command_id,
total_items=total_estimate,
message=f"Rebuild operation started. Estimated {total_estimate} items to process.",
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Failed to start rebuild: {e}")
logger.exception(e)
raise HTTPException(
status_code=500, detail=f"Failed to start rebuild operation: {str(e)}"
)
@router.get("/rebuild/{command_id}/status", response_model=RebuildStatusResponse)
async def get_rebuild_status(command_id: str):
"""
Get the status of a rebuild operation.
Returns:
- **status**: queued, running, completed, failed
- **progress**: processed count, total count, percentage
- **stats**: breakdown by type (sources, notes, insights, failed)
- **timestamps**: started_at, completed_at
"""
try:
# Get command status from surreal_commands
status = await get_command_status(command_id)
if not status:
raise HTTPException(status_code=404, detail="Rebuild command not found")
# Build response based on status
response = RebuildStatusResponse(
command_id=command_id,
status=status.status,
)
# Extract metadata from command result
if status.result and isinstance(status.result, dict):
result = status.result
# Build progress info
if "total_items" in result and "jobs_submitted" in result:
total = result["total_items"]
submitted = result["jobs_submitted"]
response.progress = RebuildProgress(
processed=submitted,
total=total,
percentage=round((submitted / total * 100) if total > 0 else 0, 2),
)
# Build stats
response.stats = RebuildStats(
sources=result.get("sources_submitted", 0),
notes=result.get("notes_submitted", 0),
insights=result.get("insights_submitted", 0),
failed=result.get("failed_submissions", 0),
)
# Add timestamps
if hasattr(status, "created") and status.created:
response.started_at = str(status.created)
if hasattr(status, "updated") and status.updated:
response.completed_at = str(status.updated)
# Add error message if failed
if (
status.status == "failed"
and status.result
and isinstance(status.result, dict)
):
response.error_message = status.result.get("error_message", "Unknown error")
return response
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Failed to get rebuild status: {e}")
logger.exception(e)
raise HTTPException(
status_code=500, detail=f"Failed to get rebuild status: {str(e)}"
)
api/routers/episode_profiles.py
+278
View File
@@ -0,0 +1,278 @@
from typing import Dict, List, Optional
from fastapi import APIRouter, HTTPException
from loguru import logger
from pydantic import BaseModel, Field
from open_notebook.exceptions import InvalidInputError, OpenNotebookError
from open_notebook.podcasts.models import EpisodeProfile, SpeakerProfile
router = APIRouter()
class EpisodeProfileResponse(BaseModel):
id: str
name: str
description: str
speaker_config: Optional[str] = Field(
None, description="speaker_profile record ID (null when orphaned)"
)
speaker_config_name: Optional[str] = Field(
None, description="Resolved speaker profile name (for display)"
)
outline_llm: Optional[str] = None
transcript_llm: Optional[str] = None
language: Optional[str] = None
default_briefing: str
num_segments: int
max_tokens: Optional[int] = None
async def _speaker_names_by_id() -> Dict[str, str]:
"""Map speaker_profile record ID -> name for list serialization."""
speakers = await SpeakerProfile.get_all()
return {str(speaker.id): speaker.name for speaker in speakers}
async def _speaker_name_for(speaker_config: Optional[str]) -> Optional[str]:
"""Resolve one profile's speaker_config record ID to the speaker name.
Returns None for a missing or dangling reference - the frontend renders
that as "needs setup"."""
if not speaker_config:
return None
speaker = await SpeakerProfile.resolve(speaker_config)
return speaker.name if speaker else None
def _profile_to_response(
profile: EpisodeProfile, speaker_name: Optional[str]
) -> EpisodeProfileResponse:
return EpisodeProfileResponse(
id=str(profile.id),
name=profile.name,
description=profile.description or "",
speaker_config=profile.speaker_config,
speaker_config_name=speaker_name,
outline_llm=profile.outline_llm,
transcript_llm=profile.transcript_llm,
language=profile.language,
default_briefing=profile.default_briefing,
num_segments=profile.num_segments,
max_tokens=profile.max_tokens,
)
async def _resolve_speaker_config(value: str) -> SpeakerProfile:
"""Resolve an incoming speaker_config (record ID, or name for backward
compatibility) to the referenced SpeakerProfile."""
speaker = await SpeakerProfile.resolve(value)
if not speaker:
raise InvalidInputError(f"Speaker profile '{value}' not found")
return speaker
@router.get("/episode-profiles", response_model=List[EpisodeProfileResponse])
async def list_episode_profiles():
"""List all available episode profiles"""
try:
profiles = await EpisodeProfile.get_all(order_by="name asc")
speaker_names = await _speaker_names_by_id()
return [
_profile_to_response(
p, speaker_names.get(p.speaker_config) if p.speaker_config else None
)
for p in profiles
]
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Failed to fetch episode profiles: {e}")
raise HTTPException(
status_code=500, detail="Failed to fetch episode profiles"
)
@router.get("/episode-profiles/{profile_name}", response_model=EpisodeProfileResponse)
async def get_episode_profile(profile_name: str):
"""Get a specific episode profile by name"""
try:
profile = await EpisodeProfile.get_by_name(profile_name)
if not profile:
raise HTTPException(
status_code=404, detail=f"Episode profile '{profile_name}' not found"
)
return _profile_to_response(
profile, await _speaker_name_for(profile.speaker_config)
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Failed to fetch episode profile '{profile_name}': {e}")
raise HTTPException(
status_code=500, detail="Failed to fetch episode profile"
)
class EpisodeProfileCreate(BaseModel):
name: str = Field(..., description="Unique profile name")
description: str = Field("", description="Profile description")
speaker_config: str = Field(
...,
description=(
"speaker_profile record ID (a profile name is also accepted "
"for backward compatibility)"
),
)
outline_llm: Optional[str] = Field(None, description="Model record ID for outline")
transcript_llm: Optional[str] = Field(
None, description="Model record ID for transcript"
)
language: Optional[str] = Field(None, description="Podcast language code")
default_briefing: str = Field(..., description="Default briefing template")
num_segments: int = Field(default=5, description="Number of podcast segments")
max_tokens: Optional[int] = Field(
None,
description="Max output tokens for outline/transcript generation",
)
@router.post("/episode-profiles", response_model=EpisodeProfileResponse)
async def create_episode_profile(profile_data: EpisodeProfileCreate):
"""Create a new episode profile"""
try:
speaker = await _resolve_speaker_config(profile_data.speaker_config)
profile = EpisodeProfile(
name=profile_data.name,
description=profile_data.description,
speaker_config=str(speaker.id),
outline_llm=profile_data.outline_llm,
transcript_llm=profile_data.transcript_llm,
language=profile_data.language,
default_briefing=profile_data.default_briefing,
num_segments=profile_data.num_segments,
max_tokens=profile_data.max_tokens,
)
await profile.save()
return _profile_to_response(profile, speaker.name)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Failed to create episode profile: {e}")
raise HTTPException(
status_code=500, detail="Failed to create episode profile"
)
@router.put("/episode-profiles/{profile_id}", response_model=EpisodeProfileResponse)
async def update_episode_profile(profile_id: str, profile_data: EpisodeProfileCreate):
"""Update an existing episode profile"""
try:
profile = await EpisodeProfile.get(profile_id)
if not profile:
raise HTTPException(
status_code=404, detail=f"Episode profile '{profile_id}' not found"
)
update_data = profile_data.model_dump(exclude_unset=True)
speaker_name: Optional[str] = None
if "speaker_config" in update_data:
speaker = await _resolve_speaker_config(update_data["speaker_config"])
update_data["speaker_config"] = str(speaker.id)
speaker_name = speaker.name
for field, value in update_data.items():
setattr(profile, field, value)
await profile.save()
if speaker_name is None:
speaker_name = await _speaker_name_for(profile.speaker_config)
return _profile_to_response(profile, speaker_name)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Failed to update episode profile: {e}")
raise HTTPException(
status_code=500, detail="Failed to update episode profile"
)
@router.delete("/episode-profiles/{profile_id}")
async def delete_episode_profile(profile_id: str):
"""Delete an episode profile"""
try:
profile = await EpisodeProfile.get(profile_id)
if not profile:
raise HTTPException(
status_code=404, detail=f"Episode profile '{profile_id}' not found"
)
await profile.delete()
return {"message": "Episode profile deleted successfully"}
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Failed to delete episode profile: {e}")
raise HTTPException(
status_code=500, detail="Failed to delete episode profile"
)
@router.post(
"/episode-profiles/{profile_id}/duplicate", response_model=EpisodeProfileResponse
)
async def duplicate_episode_profile(profile_id: str):
"""Duplicate an episode profile"""
try:
original = await EpisodeProfile.get(profile_id)
if not original:
raise HTTPException(
status_code=404, detail=f"Episode profile '{profile_id}' not found"
)
duplicate = EpisodeProfile(
name=f"{original.name} - Copy",
description=original.description,
speaker_config=original.speaker_config,
outline_llm=original.outline_llm,
transcript_llm=original.transcript_llm,
language=original.language,
default_briefing=original.default_briefing,
num_segments=original.num_segments,
max_tokens=original.max_tokens,
)
await duplicate.save()
return _profile_to_response(
duplicate, await _speaker_name_for(duplicate.speaker_config)
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Failed to duplicate episode profile: {e}")
raise HTTPException(
status_code=500, detail="Failed to duplicate episode profile"
)
api/routers/insights.py
+94
View File
@@ -0,0 +1,94 @@
from fastapi import APIRouter, HTTPException
from loguru import logger
from api.models import NoteResponse, SaveAsNoteRequest, SourceInsightResponse
from open_notebook.domain.notebook import SourceInsight
from open_notebook.exceptions import (
InvalidInputError,
NotFoundError,
OpenNotebookError,
)
router = APIRouter()
@router.get("/insights/{insight_id}", response_model=SourceInsightResponse)
async def get_insight(insight_id: str):
"""Get a specific insight by ID."""
try:
insight = await SourceInsight.get(insight_id)
if not insight:
raise HTTPException(status_code=404, detail="Insight not found")
# Get source ID from the insight relationship
source = await insight.get_source()
return SourceInsightResponse(
id=insight.id or "",
source_id=source.id or "",
insight_type=insight.insight_type,
content=insight.content,
created=insight.created.isoformat() if insight.created else None,
updated=insight.updated.isoformat() if insight.updated else None,
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching insight {insight_id}: {str(e)}")
raise HTTPException(status_code=500, detail="Error fetching insight")
@router.delete("/insights/{insight_id}")
async def delete_insight(insight_id: str):
"""Delete a specific insight."""
try:
insight = await SourceInsight.get(insight_id)
if not insight:
raise HTTPException(status_code=404, detail="Insight not found")
await insight.delete()
return {"message": "Insight deleted successfully"}
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error deleting insight {insight_id}: {str(e)}")
raise HTTPException(status_code=500, detail="Error deleting insight")
@router.post("/insights/{insight_id}/save-as-note", response_model=NoteResponse)
async def save_insight_as_note(insight_id: str, request: SaveAsNoteRequest):
"""Convert an insight to a note."""
try:
insight = await SourceInsight.get(insight_id)
if not insight:
raise HTTPException(status_code=404, detail="Insight not found")
# Use the existing save_as_note method from the domain model
note = await insight.save_as_note(request.notebook_id)
return NoteResponse(
id=note.id or "",
title=note.title,
content=note.content,
note_type=note.note_type,
created=str(note.created),
updated=str(note.updated),
)
except HTTPException:
raise
except NotFoundError:
raise HTTPException(status_code=404, detail="Notebook not found")
except InvalidInputError as e:
raise HTTPException(status_code=400, detail=str(e))
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error saving insight {insight_id} as note: {str(e)}")
raise HTTPException(
status_code=500, detail="Error saving insight as note"
)
api/routers/languages.py
+83
View File
@@ -0,0 +1,83 @@
from typing import List
import pycountry
from babel import Locale
from babel.core import get_global
from fastapi import APIRouter
from pydantic import BaseModel
router = APIRouter()
# Additional regional variants for languages where the distinction matters
# (TTS accent, vocabulary, spelling differences)
_EXTRA_VARIANTS = [
"pt_PT",
"en_GB",
"en_AU",
"en_IN",
"es_MX",
"es_AR",
"es_CO",
"fr_CA",
"fr_CH",
"zh_TW",
"zh_HK",
"de_AT",
"de_CH",
"ar_SA",
"nl_BE",
]
class LanguageResponse(BaseModel):
code: str
name: str
@router.get("/languages", response_model=List[LanguageResponse])
async def list_languages():
"""List available languages as BCP 47 locale codes (e.g. pt-BR, en-US)."""
likely_subtags = get_global("likely_subtags")
languages = []
seen = set()
# 1. For each language, resolve its default locale via CLDR likely subtags
for lang in pycountry.languages:
if not hasattr(lang, "alpha_2"):
continue
code = lang.alpha_2
likely = likely_subtags.get(code)
if likely:
try:
loc = Locale.parse(likely)
if loc.territory:
bcp47 = f"{loc.language}-{loc.territory}"
display = loc.get_display_name("en")
if bcp47 not in seen:
seen.add(bcp47)
languages.append(LanguageResponse(code=bcp47, name=display))
continue
except Exception:
pass
# Fallback: bare language code
if code not in seen:
seen.add(code)
languages.append(LanguageResponse(code=code, name=lang.name))
# 2. Add important regional variants
for locale_str in _EXTRA_VARIANTS:
try:
loc = Locale.parse(locale_str)
bcp47 = f"{loc.language}-{loc.territory}"
if bcp47 not in seen:
seen.add(bcp47)
display = loc.get_display_name("en")
languages.append(LanguageResponse(code=bcp47, name=display))
except Exception:
pass
languages.sort(key=lambda x: x.name)
return languages
api/routers/models.py
+830
View File
@@ -0,0 +1,830 @@
import os
import traceback
from typing import Dict, List, Optional
from esperanto import AIFactory
from fastapi import APIRouter, HTTPException, Query
from loguru import logger
from pydantic import BaseModel
from api.models import (
DefaultModelsResponse,
ModelCreate,
ModelResponse,
ProviderAvailabilityResponse,
)
from open_notebook.ai.connection_tester import test_individual_model
from open_notebook.ai.key_provider import provision_provider_keys
from open_notebook.ai.model_discovery import (
discover_provider_models,
get_provider_model_count,
sync_all_providers,
sync_provider_models,
)
from open_notebook.ai.models import DefaultModels, Model
from open_notebook.domain.credential import Credential
from open_notebook.exceptions import (
InvalidInputError,
NotFoundError,
OpenNotebookError,
)
router = APIRouter()
# =============================================================================
# Model Discovery Response Models
# =============================================================================
class DiscoveredModelResponse(BaseModel):
"""Response model for a discovered model."""
name: str
provider: str
model_type: str
description: Optional[str] = None
class ProviderSyncResponse(BaseModel):
"""Response model for provider sync operation."""
provider: str
discovered: int
new: int
existing: int
class AllProvidersSyncResponse(BaseModel):
"""Response model for syncing all providers."""
results: Dict[str, ProviderSyncResponse]
total_discovered: int
total_new: int
class ProviderModelCountResponse(BaseModel):
"""Response model for provider model counts."""
provider: str
counts: Dict[str, int]
total: int
class AutoAssignResult(BaseModel):
"""Response model for auto-assign operation."""
assigned: Dict[str, str] # slot_name -> model_id
skipped: List[str] # slots already assigned
missing: List[str] # slots with no available models
class ModelTestResponse(BaseModel):
"""Response model for individual model test."""
success: bool
message: str
details: Optional[str] = None
# Provider priority for auto-assignment (higher priority first)
PROVIDER_PRIORITY = [
"openai",
"anthropic",
"google",
"mistral",
"groq",
"deepseek",
"xai",
"openrouter",
"ollama",
"azure",
"openai_compatible",
"dashscope",
"minimax",
]
# Model preference patterns (preferred models within each provider)
MODEL_PREFERENCES = {
"openai": ["gpt-4o", "gpt-4", "gpt-3.5-turbo"],
"anthropic": ["claude-3-5-sonnet", "claude-3-opus", "claude-3-sonnet"],
"google": ["gemini-3.5-flash", "gemini-2.5-flash", "gemini-2.5-pro"],
"mistral": ["mistral-large", "mixtral"],
"groq": ["llama-3.3", "llama-3.1", "mixtral"],
"dashscope": ["qwen-max", "qwen-plus", "qwen-turbo"],
"minimax": ["MiniMax-M2.5", "MiniMax-M2.5-highspeed"],
}
async def _check_provider_has_credential(provider: str) -> bool:
"""Check if a provider has any credentials configured in the database."""
try:
credentials = await Credential.get_by_provider(provider)
return len(credentials) > 0
except Exception:
pass
return False
def _check_azure_support(mode: str) -> bool:
"""
Check if Azure OpenAI provider is available for a specific mode.
Args:
mode: One of 'LLM', 'EMBEDDING', 'STT', 'TTS'
Returns:
bool: True if either generic or mode-specific env vars are set
"""
# Check generic configuration (applies to all modes)
generic = (
os.environ.get("AZURE_OPENAI_API_KEY") is not None
and os.environ.get("AZURE_OPENAI_ENDPOINT") is not None
and os.environ.get("AZURE_OPENAI_API_VERSION") is not None
)
# Check mode-specific configuration (takes precedence)
specific = (
os.environ.get(f"AZURE_OPENAI_API_KEY_{mode}") is not None
and os.environ.get(f"AZURE_OPENAI_ENDPOINT_{mode}") is not None
and os.environ.get(f"AZURE_OPENAI_API_VERSION_{mode}") is not None
)
return generic or specific
def _check_openai_compatible_support(mode: str) -> bool:
"""
Check if OpenAI-compatible provider is available for a specific mode.
Args:
mode: One of 'LLM', 'EMBEDDING', 'STT', 'TTS'
Returns:
bool: True if either generic or mode-specific env var is set
"""
generic = os.environ.get("OPENAI_COMPATIBLE_BASE_URL") is not None
specific = os.environ.get(f"OPENAI_COMPATIBLE_BASE_URL_{mode}") is not None
generic_key = os.environ.get("OPENAI_COMPATIBLE_API_KEY") is not None
specific_key = os.environ.get(f"OPENAI_COMPATIBLE_API_KEY_{mode}") is not None
return generic or specific or generic_key or specific_key
@router.get("/models", response_model=List[ModelResponse])
async def get_models(
type: Optional[str] = Query(None, description="Filter by model type"),
):
"""Get all configured models with optional type filtering."""
try:
if type:
models = await Model.get_models_by_type(type)
else:
models = await Model.get_all()
return [
ModelResponse(
id=model.id,
name=model.name,
provider=model.provider,
type=model.type,
credential=model.credential,
created=str(model.created),
updated=str(model.updated),
)
for model in models
]
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching models: {str(e)}")
raise HTTPException(status_code=500, detail=f"Error fetching models: {str(e)}")
@router.post("/models", response_model=ModelResponse)
async def create_model(model_data: ModelCreate):
"""Create a new model configuration."""
try:
# Validate model type
valid_types = ["language", "embedding", "text_to_speech", "speech_to_text"]
if model_data.type not in valid_types:
raise HTTPException(
status_code=400,
detail=f"Invalid model type. Must be one of: {valid_types}",
)
# Check for duplicate model name under the same provider and type (case-insensitive)
from open_notebook.database.repository import repo_query
existing = await repo_query(
"SELECT * FROM model WHERE string::lowercase(provider) = $provider AND string::lowercase(name) = $name AND string::lowercase(type) = $type LIMIT 1",
{
"provider": model_data.provider.lower(),
"name": model_data.name.lower(),
"type": model_data.type.lower(),
},
)
if existing:
raise HTTPException(
status_code=400,
detail=f"Model '{model_data.name}' already exists for provider '{model_data.provider}' with type '{model_data.type}'",
)
new_model = Model(
name=model_data.name,
provider=model_data.provider,
type=model_data.type,
credential=model_data.credential,
)
await new_model.save()
return ModelResponse(
id=new_model.id or "",
name=new_model.name,
provider=new_model.provider,
type=new_model.type,
credential=new_model.credential,
created=str(new_model.created),
updated=str(new_model.updated),
)
except HTTPException:
raise
except InvalidInputError as e:
raise HTTPException(status_code=400, detail=str(e))
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error creating model: {str(e)}")
raise HTTPException(status_code=500, detail=f"Error creating model: {str(e)}")
@router.delete("/models/{model_id}")
async def delete_model(model_id: str):
"""Delete a model configuration."""
try:
model = await Model.get(model_id)
await model.delete()
return {"message": "Model deleted successfully"}
except HTTPException:
raise
except NotFoundError:
raise HTTPException(status_code=404, detail="Model not found")
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error deleting model {model_id}: {str(e)}")
raise HTTPException(status_code=500, detail=f"Error deleting model: {str(e)}")
@router.post("/models/{model_id}/test", response_model=ModelTestResponse)
async def test_model(model_id: str):
"""Test if a specific model is correctly configured and functional."""
try:
model = await Model.get(model_id)
if not model:
raise HTTPException(status_code=404, detail="Model not found")
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception:
raise HTTPException(status_code=404, detail="Model not found")
try:
success, message = await test_individual_model(model)
return ModelTestResponse(success=success, message=message)
except Exception as e:
logger.error(f"Error testing model {model_id}: {traceback.format_exc()}")
return ModelTestResponse(
success=False,
message=str(e)[:200],
)
@router.get("/models/defaults", response_model=DefaultModelsResponse)
async def get_default_models():
"""Get default model assignments."""
try:
defaults = await DefaultModels.get_instance()
return DefaultModelsResponse(
default_chat_model=defaults.default_chat_model, # type: ignore[attr-defined]
default_transformation_model=defaults.default_transformation_model, # type: ignore[attr-defined]
large_context_model=defaults.large_context_model, # type: ignore[attr-defined]
default_text_to_speech_model=defaults.default_text_to_speech_model, # type: ignore[attr-defined]
default_speech_to_text_model=defaults.default_speech_to_text_model, # type: ignore[attr-defined]
default_embedding_model=defaults.default_embedding_model, # type: ignore[attr-defined]
default_tools_model=defaults.default_tools_model, # type: ignore[attr-defined]
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching default models: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error fetching default models: {str(e)}"
)
# Defaults the app cannot function without — they can be reassigned but
# never cleared (the optional ones fall back to the chat default or are
# simply skipped when unset).
REQUIRED_DEFAULTS = {"default_chat_model", "default_embedding_model"}
@router.put("/models/defaults", response_model=DefaultModelsResponse)
async def update_default_models(defaults_data: DefaultModelsResponse):
"""Update default model assignments.
Partial-update semantics keyed on field PRESENCE, not value: a field
absent from the payload is left untouched, while an explicit null clears
the default (except required ones). `is not None` checks would silently
ignore a null sent to clear a default — the old value survived while the
client saw success (same anti-pattern fixed for credentials in #1046).
"""
try:
defaults = await DefaultModels.get_instance()
sent = defaults_data.model_fields_set
for field in DefaultModelsResponse.model_fields:
if field not in sent:
continue
value = getattr(defaults_data, field)
if value is None and field in REQUIRED_DEFAULTS:
raise InvalidInputError(
f"{field} is required and cannot be cleared, only reassigned"
)
setattr(defaults, field, value)
await defaults.update()
# No cache refresh needed - next access will fetch fresh data from DB
return DefaultModelsResponse(
default_chat_model=defaults.default_chat_model, # type: ignore[attr-defined]
default_transformation_model=defaults.default_transformation_model, # type: ignore[attr-defined]
large_context_model=defaults.large_context_model, # type: ignore[attr-defined]
default_text_to_speech_model=defaults.default_text_to_speech_model, # type: ignore[attr-defined]
default_speech_to_text_model=defaults.default_speech_to_text_model, # type: ignore[attr-defined]
default_embedding_model=defaults.default_embedding_model, # type: ignore[attr-defined]
default_tools_model=defaults.default_tools_model, # type: ignore[attr-defined]
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error updating default models: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error updating default models: {str(e)}"
)
@router.get("/models/providers", response_model=ProviderAvailabilityResponse)
async def get_provider_availability():
"""Get provider availability based on database config and environment variables."""
try:
# Check which providers have credentials in the database or env vars
# For each provider, check DB credentials first, then env vars as fallback
# Simple env var mapping for backward compatibility
env_var_map = {
"openai": "OPENAI_API_KEY",
"anthropic": "ANTHROPIC_API_KEY",
"google": "GOOGLE_API_KEY",
"groq": "GROQ_API_KEY",
"mistral": "MISTRAL_API_KEY",
"deepseek": "DEEPSEEK_API_KEY",
"xai": "XAI_API_KEY",
"openrouter": "OPENROUTER_API_KEY",
"voyage": "VOYAGE_API_KEY",
"elevenlabs": "ELEVENLABS_API_KEY",
"deepgram": "DEEPGRAM_API_KEY",
"ollama": "OLLAMA_API_BASE",
"dashscope": "DASHSCOPE_API_KEY",
"minimax": "MINIMAX_API_KEY",
}
provider_status = {}
# Check simple providers: credential in DB or env var
for provider, env_var in env_var_map.items():
has_cred = await _check_provider_has_credential(provider)
has_env = os.environ.get(env_var) is not None
provider_status[provider] = has_cred or has_env
# Google also supports GEMINI_API_KEY
if not provider_status.get("google"):
provider_status["google"] = os.environ.get("GEMINI_API_KEY") is not None
# Vertex: DB credential or env vars
provider_status["vertex"] = (
await _check_provider_has_credential("vertex")
or os.environ.get("VERTEX_PROJECT") is not None
)
# Azure: DB credential or env vars
provider_status["azure"] = (
await _check_provider_has_credential("azure")
or _check_azure_support("LLM")
or _check_azure_support("EMBEDDING")
or _check_azure_support("STT")
or _check_azure_support("TTS")
)
# OpenAI-compatible: DB credential or env vars
provider_status["openai_compatible"] = (
await _check_provider_has_credential("openai_compatible")
or _check_openai_compatible_support("LLM")
or _check_openai_compatible_support("EMBEDDING")
or _check_openai_compatible_support("STT")
or _check_openai_compatible_support("TTS")
)
available_providers = [k for k, v in provider_status.items() if v]
unavailable_providers = [k for k, v in provider_status.items() if not v]
# Get supported model types from Esperanto
esperanto_available = AIFactory.get_available_providers()
# Build supported types mapping only for available providers
supported_types: dict[str, list[str]] = {}
for provider in available_providers:
supported_types[provider] = []
# Map Esperanto model types to our environment variable modes
mode_mapping = {
"language": "LLM",
"embedding": "EMBEDDING",
"speech_to_text": "STT",
"text_to_speech": "TTS",
}
# Special handling for openai-compatible to check mode-specific availability
if provider == "openai_compatible":
# Esperanto exposes this provider with a hyphen ("openai-compatible"),
# while the rest of the codebase uses the underscore form.
esperanto_name = "openai-compatible"
has_db_cred = await _check_provider_has_credential("openai_compatible")
for model_type, mode in mode_mapping.items():
if (
model_type in esperanto_available
and esperanto_name in esperanto_available[model_type]
):
if has_db_cred or _check_openai_compatible_support(mode):
supported_types[provider].append(model_type)
# Special handling for azure to check mode-specific availability
elif provider == "azure":
has_db_cred = await _check_provider_has_credential("azure")
for model_type, mode in mode_mapping.items():
if (
model_type in esperanto_available
and provider in esperanto_available[model_type]
):
if has_db_cred or _check_azure_support(mode):
supported_types[provider].append(model_type)
else:
# Standard provider detection
for model_type, providers in esperanto_available.items():
if provider in providers:
supported_types[provider].append(model_type)
return ProviderAvailabilityResponse(
available=available_providers,
unavailable=unavailable_providers,
supported_types=supported_types,
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error checking provider availability: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error checking provider availability: {str(e)}"
)
# =============================================================================
# Model Discovery Endpoints
# =============================================================================
@router.get(
"/models/discover/{provider}", response_model=List[DiscoveredModelResponse]
)
async def discover_models(provider: str):
"""
Discover available models from a provider without registering them.
This endpoint queries the provider's API to list available models
but does not save them to the database. Use the sync endpoint
to both discover and register models.
"""
try:
# Provision DB-stored credentials into env vars before discovery
await provision_provider_keys(provider)
discovered = await discover_provider_models(provider)
return [
DiscoveredModelResponse(
name=m.name,
provider=m.provider,
model_type=m.model_type,
description=m.description,
)
for m in discovered
]
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error discovering models for {provider}: {str(e)}")
raise HTTPException(
status_code=500, detail="Error discovering models. Check server logs for details."
)
@router.post("/models/sync/{provider}", response_model=ProviderSyncResponse)
async def sync_models(provider: str):
"""
Sync models for a specific provider.
Discovers available models from the provider's API and registers
any new models in the database. Existing models are skipped.
Returns counts of discovered, new, and existing models.
"""
try:
# Provision DB-stored credentials into env vars before discovery
await provision_provider_keys(provider)
discovered, new, existing = await sync_provider_models(
provider, auto_register=True
)
return ProviderSyncResponse(
provider=provider,
discovered=discovered,
new=new,
existing=existing,
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error syncing models for {provider}: {str(e)}")
raise HTTPException(status_code=500, detail="Error syncing models. Check server logs for details.")
@router.post("/models/sync", response_model=AllProvidersSyncResponse)
async def sync_all_models():
"""
Sync models for all configured providers.
Discovers and registers models from all providers that have
valid API keys configured. This is useful for initial setup
or periodic refresh of available models.
"""
try:
results = await sync_all_providers()
response_results = {}
total_discovered = 0
total_new = 0
for provider, (discovered, new, existing) in results.items():
response_results[provider] = ProviderSyncResponse(
provider=provider,
discovered=discovered,
new=new,
existing=existing,
)
total_discovered += discovered
total_new += new
return AllProvidersSyncResponse(
results=response_results,
total_discovered=total_discovered,
total_new=total_new,
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error syncing all models: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error syncing all models: {str(e)}"
)
@router.get("/models/count/{provider}", response_model=ProviderModelCountResponse)
async def get_model_count(provider: str):
"""
Get count of registered models for a provider, grouped by type.
Returns counts for each model type (language, embedding,
speech_to_text, text_to_speech) as well as total count.
"""
try:
counts = await get_provider_model_count(provider)
total = sum(counts.values())
return ProviderModelCountResponse(
provider=provider,
counts=counts,
total=total,
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error getting model count for {provider}: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error getting model count: {str(e)}"
)
@router.get("/models/by-provider/{provider}", response_model=List[ModelResponse])
async def get_models_by_provider(provider: str):
"""
Get all registered models for a specific provider.
Returns models from the database that belong to the specified provider.
"""
try:
from open_notebook.database.repository import repo_query
models = await repo_query(
"SELECT * FROM model WHERE provider = $provider ORDER BY type, name",
{"provider": provider},
)
return [
ModelResponse(
id=model.get("id", ""),
name=model.get("name", ""),
provider=model.get("provider", ""),
type=model.get("type", ""),
credential=model.get("credential"),
created=str(model.get("created", "")),
updated=str(model.get("updated", "")),
)
for model in models
]
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching models for {provider}: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error fetching models: {str(e)}"
)
def _get_preferred_model(
models: List[Dict], provider_priority: List[str], model_preferences: Dict
) -> Optional[Dict]:
"""
Select the best model from a list based on provider priority and model preferences.
Args:
models: List of model dictionaries with 'provider', 'name', 'id' keys
provider_priority: List of providers in preference order
model_preferences: Dict mapping provider to list of preferred model name patterns
Returns:
The best model dict, or None if no models available
"""
if not models:
return None
# Group models by provider
by_provider: Dict[str, List[Dict]] = {}
for model in models:
provider = model.get("provider", "")
if provider not in by_provider:
by_provider[provider] = []
by_provider[provider].append(model)
# Find first provider with models (in priority order)
for provider in provider_priority:
if provider in by_provider:
provider_models = by_provider[provider]
# Check for preferred models within this provider
if provider in model_preferences:
for preference in model_preferences[provider]:
for model in provider_models:
if preference.lower() in model.get("name", "").lower():
return model
# Fall back to first model from this provider
return provider_models[0]
# Fall back to first model from any provider
return models[0] if models else None
@router.post("/models/auto-assign", response_model=AutoAssignResult)
async def auto_assign_defaults():
"""
Auto-assign default models based on available models.
This endpoint intelligently assigns the first available model of each
required type to the corresponding default slot. It uses provider
priority (preferring premium providers like OpenAI, Anthropic) and
model preferences within each provider.
Returns:
- assigned: Dict of slot names to assigned model IDs
- skipped: List of slots that already have models assigned
- missing: List of slots with no available models
"""
try:
from open_notebook.database.repository import repo_query
# Get current defaults
defaults = await DefaultModels.get_instance()
# Get all models grouped by type
all_models = await repo_query(
"SELECT * FROM model ORDER BY provider, name",
{},
)
# Group models by type
models_by_type: Dict[str, List[Dict]] = {
"language": [],
"embedding": [],
"text_to_speech": [],
"speech_to_text": [],
}
for model in all_models:
model_type = model.get("type", "")
if model_type in models_by_type:
models_by_type[model_type].append(model)
# Define slot configuration: (slot_name, model_type, current_value)
slot_configs = [
("default_chat_model", "language", defaults.default_chat_model), # type: ignore[attr-defined]
("default_transformation_model", "language", defaults.default_transformation_model), # type: ignore[attr-defined]
("default_tools_model", "language", defaults.default_tools_model), # type: ignore[attr-defined]
("large_context_model", "language", defaults.large_context_model), # type: ignore[attr-defined]
("default_embedding_model", "embedding", defaults.default_embedding_model), # type: ignore[attr-defined]
("default_text_to_speech_model", "text_to_speech", defaults.default_text_to_speech_model), # type: ignore[attr-defined]
("default_speech_to_text_model", "speech_to_text", defaults.default_speech_to_text_model), # type: ignore[attr-defined]
]
assigned: Dict[str, str] = {}
skipped: List[str] = []
missing: List[str] = []
for slot_name, model_type, current_value in slot_configs:
if current_value:
# Slot already has a value
skipped.append(slot_name)
continue
available_models = models_by_type.get(model_type, [])
if not available_models:
# No models of this type available
missing.append(slot_name)
continue
# Select best model for this slot
best_model = _get_preferred_model(
available_models, PROVIDER_PRIORITY, MODEL_PREFERENCES
)
if best_model:
model_id = best_model.get("id", "")
assigned[slot_name] = model_id
# Update the defaults object
setattr(defaults, slot_name, model_id)
# Save updated defaults if any assignments were made
if assigned:
await defaults.update()
return AutoAssignResult(
assigned=assigned,
skipped=skipped,
missing=missing,
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error auto-assigning defaults: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error auto-assigning defaults: {str(e)}"
)
api/routers/notebooks.py
+459
View File
@@ -0,0 +1,459 @@
from typing import List, Optional
from fastapi import APIRouter, HTTPException, Query
from loguru import logger
from api.models import (
NotebookCreate,
NotebookDeletePreview,
NotebookDeleteResponse,
NotebookResponse,
NotebookUpdate,
RecentlyViewedResponse,
)
from open_notebook.database.repository import ensure_record_id, repo_query
from open_notebook.domain.notebook import Notebook, Source
from open_notebook.exceptions import (
InvalidInputError,
NotFoundError,
OpenNotebookError,
)
router = APIRouter()
def _last_viewed_sort_key(item: RecentlyViewedResponse) -> str:
return item.last_viewed_at
async def _stamp_notebook_view(notebook_id: str) -> None:
# Best-effort write-on-read: recording the view timestamp must never turn a
# successful read into a 500. Log and move on if the stamp update fails.
try:
await repo_query(
"UPDATE $notebook_id SET last_viewed_at = time::now();",
{"notebook_id": ensure_record_id(notebook_id)},
)
except Exception as e:
logger.warning(
f"Failed to stamp last_viewed_at for notebook {notebook_id}: {e}"
)
def _recently_viewed_notebook(row: dict) -> RecentlyViewedResponse:
return RecentlyViewedResponse(
type="notebook",
id=str(row.get("id", "")),
title=row.get("title") or row.get("name") or "Untitled notebook",
last_viewed_at=str(row.get("last_viewed_at", "")),
)
def _recently_viewed_source(row: dict) -> RecentlyViewedResponse:
return RecentlyViewedResponse(
type="source",
id=str(row.get("id", "")),
title=row.get("title") or "Untitled source",
last_viewed_at=str(row.get("last_viewed_at", "")),
)
@router.get("/notebooks", response_model=List[NotebookResponse])
async def get_notebooks(
archived: Optional[bool] = Query(None, description="Filter by archived status"),
order_by: str = Query("updated desc", description="Order by field and direction"),
):
"""Get all notebooks with optional filtering and ordering."""
try:
# Validate order_by against allowlist to prevent SurrealQL injection
allowed_fields = {"name", "created", "updated"}
allowed_directions = {"asc", "desc"}
parts = order_by.strip().lower().split()
if len(parts) == 1:
if parts[0] not in allowed_fields:
raise HTTPException(
status_code=400,
detail=f"Invalid order_by field: '{order_by}'. Allowed fields: {', '.join(sorted(allowed_fields))}",
)
validated_order_by = parts[0]
elif len(parts) == 2:
if parts[0] not in allowed_fields or parts[1] not in allowed_directions:
raise HTTPException(
status_code=400,
detail=f"Invalid order_by: '{order_by}'. Allowed fields: {', '.join(sorted(allowed_fields))}. Allowed directions: asc, desc",
)
validated_order_by = f"{parts[0]} {parts[1]}"
else:
raise HTTPException(
status_code=400,
detail=f"Invalid order_by format: '{order_by}'. Expected 'field' or 'field direction'",
)
# Build the query with counts
query = f"""
SELECT *,
count(<-reference.in) as source_count,
count(<-artifact.in) as note_count
FROM notebook
ORDER BY {validated_order_by}
"""
result = await repo_query(query)
# Filter by archived status if specified
if archived is not None:
result = [nb for nb in result if nb.get("archived") == archived]
return [
NotebookResponse(
id=str(nb.get("id", "")),
name=nb.get("name", ""),
description=nb.get("description", ""),
archived=nb.get("archived", False),
created=str(nb.get("created", "")),
updated=str(nb.get("updated", "")),
source_count=nb.get("source_count", 0),
note_count=nb.get("note_count", 0),
)
for nb in result
]
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching notebooks: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error fetching notebooks: {str(e)}"
)
@router.post("/notebooks", response_model=NotebookResponse)
async def create_notebook(notebook: NotebookCreate):
"""Create a new notebook."""
try:
new_notebook = Notebook(
name=notebook.name,
description=notebook.description,
)
await new_notebook.save()
return NotebookResponse(
id=new_notebook.id or "",
name=new_notebook.name,
description=new_notebook.description,
archived=new_notebook.archived or False,
created=str(new_notebook.created),
updated=str(new_notebook.updated),
source_count=0, # New notebook has no sources
note_count=0, # New notebook has no notes
)
except InvalidInputError as e:
raise HTTPException(status_code=400, detail=str(e))
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error creating notebook: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error creating notebook: {str(e)}"
)
@router.get("/recently-viewed", response_model=List[RecentlyViewedResponse])
async def get_recently_viewed(
limit: int = Query(12, ge=1, le=50, description="Number of items to return"),
):
"""Get recently viewed notebooks and sources, newest first."""
try:
notebooks = await repo_query(
"""
SELECT id, name AS title, last_viewed_at
FROM notebook
WHERE last_viewed_at != NONE AND last_viewed_at != NULL
ORDER BY last_viewed_at DESC
LIMIT $limit
""",
{"limit": limit},
)
sources = await repo_query(
"""
SELECT id, title, last_viewed_at
FROM source
WHERE last_viewed_at != NONE AND last_viewed_at != NULL
ORDER BY last_viewed_at DESC
LIMIT $limit
""",
{"limit": limit},
)
items = [
*[_recently_viewed_notebook(nb) for nb in notebooks],
*[_recently_viewed_source(src) for src in sources],
]
items.sort(key=_last_viewed_sort_key, reverse=True)
return items[:limit]
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
# Log full context server-side; return a generic message so internal
# details are not leaked to clients.
logger.exception(f"Error fetching recently viewed items: {e}")
raise HTTPException(
status_code=500, detail="Error fetching recently viewed items"
)
@router.get(
"/notebooks/{notebook_id}/delete-preview", response_model=NotebookDeletePreview
)
async def get_notebook_delete_preview(notebook_id: str):
"""Get a preview of what will be deleted when this notebook is deleted."""
try:
notebook = await Notebook.get(notebook_id)
preview = await notebook.get_delete_preview()
return NotebookDeletePreview(
notebook_id=str(notebook.id),
notebook_name=notebook.name,
note_count=preview["note_count"],
exclusive_source_count=preview["exclusive_source_count"],
shared_source_count=preview["shared_source_count"],
)
except HTTPException:
raise
except NotFoundError:
raise HTTPException(status_code=404, detail="Notebook not found")
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error getting delete preview for notebook {notebook_id}: {e}")
raise HTTPException(
status_code=500,
detail=f"Error fetching notebook deletion preview: {str(e)}",
)
@router.get("/notebooks/{notebook_id}", response_model=NotebookResponse)
async def get_notebook(notebook_id: str):
"""Get a specific notebook by ID."""
try:
# Query with counts for single notebook
query = """
SELECT *,
count(<-reference.in) as source_count,
count(<-artifact.in) as note_count
FROM $notebook_id
"""
result = await repo_query(query, {"notebook_id": ensure_record_id(notebook_id)})
if not result:
raise HTTPException(status_code=404, detail="Notebook not found")
await _stamp_notebook_view(notebook_id)
nb = result[0]
return NotebookResponse(
id=str(nb.get("id", "")),
name=nb.get("name", ""),
description=nb.get("description", ""),
archived=nb.get("archived", False),
created=str(nb.get("created", "")),
updated=str(nb.get("updated", "")),
source_count=nb.get("source_count", 0),
note_count=nb.get("note_count", 0),
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching notebook {notebook_id}: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error fetching notebook: {str(e)}"
)
@router.put("/notebooks/{notebook_id}", response_model=NotebookResponse)
async def update_notebook(notebook_id: str, notebook_update: NotebookUpdate):
"""Update a notebook."""
try:
notebook = await Notebook.get(notebook_id)
# Update only provided fields
if notebook_update.name is not None:
notebook.name = notebook_update.name
if notebook_update.description is not None:
notebook.description = notebook_update.description
if notebook_update.archived is not None:
notebook.archived = notebook_update.archived
await notebook.save()
# Query with counts after update
query = """
SELECT *,
count(<-reference.in) as source_count,
count(<-artifact.in) as note_count
FROM $notebook_id
"""
result = await repo_query(query, {"notebook_id": ensure_record_id(notebook_id)})
if result:
nb = result[0]
return NotebookResponse(
id=str(nb.get("id", "")),
name=nb.get("name", ""),
description=nb.get("description", ""),
archived=nb.get("archived", False),
created=str(nb.get("created", "")),
updated=str(nb.get("updated", "")),
source_count=nb.get("source_count", 0),
note_count=nb.get("note_count", 0),
)
# Fallback if query fails
return NotebookResponse(
id=notebook.id or "",
name=notebook.name,
description=notebook.description,
archived=notebook.archived or False,
created=str(notebook.created),
updated=str(notebook.updated),
source_count=0,
note_count=0,
)
except HTTPException:
raise
except NotFoundError:
raise HTTPException(status_code=404, detail="Notebook not found")
except InvalidInputError as e:
raise HTTPException(status_code=400, detail=str(e))
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error updating notebook {notebook_id}: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error updating notebook: {str(e)}"
)
@router.post("/notebooks/{notebook_id}/sources/{source_id}")
async def add_source_to_notebook(notebook_id: str, source_id: str):
"""Add an existing source to a notebook (create the reference)."""
try:
# Verify the notebook and source exist (raises NotFoundError -> 404)
await Notebook.get(notebook_id)
await Source.get(source_id)
# Check if reference already exists (idempotency)
existing_ref = await repo_query(
"SELECT * FROM reference WHERE out = $source_id AND in = $notebook_id",
{
"notebook_id": ensure_record_id(notebook_id),
"source_id": ensure_record_id(source_id),
},
)
# If reference doesn't exist, create it
if not existing_ref:
await repo_query(
"RELATE $source_id->reference->$notebook_id",
{
"notebook_id": ensure_record_id(notebook_id),
"source_id": ensure_record_id(source_id),
},
)
return {"message": "Source linked to notebook successfully"}
except HTTPException:
raise
except NotFoundError:
raise HTTPException(status_code=404, detail="Notebook or source not found")
except OpenNotebookError:
raise
except Exception as e:
logger.error(
f"Error linking source {source_id} to notebook {notebook_id}: {str(e)}"
)
raise HTTPException(
status_code=500, detail=f"Error linking source to notebook: {str(e)}"
)
@router.delete("/notebooks/{notebook_id}/sources/{source_id}")
async def remove_source_from_notebook(notebook_id: str, source_id: str):
"""Remove a source from a notebook (delete the reference)."""
try:
# Verify the notebook exists (raises NotFoundError -> 404)
await Notebook.get(notebook_id)
# Delete the reference record linking source to notebook
await repo_query(
"DELETE FROM reference WHERE out = $notebook_id AND in = $source_id",
{
"notebook_id": ensure_record_id(notebook_id),
"source_id": ensure_record_id(source_id),
},
)
return {"message": "Source removed from notebook successfully"}
except HTTPException:
raise
except NotFoundError:
raise HTTPException(status_code=404, detail="Notebook not found")
except OpenNotebookError:
raise
except Exception as e:
logger.error(
f"Error removing source {source_id} from notebook {notebook_id}: {str(e)}"
)
raise HTTPException(
status_code=500, detail=f"Error removing source from notebook: {str(e)}"
)
@router.delete("/notebooks/{notebook_id}", response_model=NotebookDeleteResponse)
async def delete_notebook(
notebook_id: str,
delete_exclusive_sources: bool = Query(
False,
description="Whether to delete sources that belong only to this notebook",
),
):
"""
Delete a notebook with cascade deletion.
Always deletes all notes associated with the notebook.
If delete_exclusive_sources is True, also deletes sources that belong only
to this notebook (not linked to any other notebooks).
"""
try:
notebook = await Notebook.get(notebook_id)
result = await notebook.delete(
delete_exclusive_sources=delete_exclusive_sources
)
return NotebookDeleteResponse(
message="Notebook deleted successfully",
deleted_notes=result["deleted_notes"],
deleted_sources=result["deleted_sources"],
unlinked_sources=result["unlinked_sources"],
)
except HTTPException:
raise
except NotFoundError:
raise HTTPException(status_code=404, detail="Notebook not found")
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error deleting notebook {notebook_id}: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error deleting notebook: {str(e)}"
)
api/routers/notes.py
+206
View File
@@ -0,0 +1,206 @@
from typing import List, Literal, Optional
from fastapi import APIRouter, HTTPException, Query
from loguru import logger
from api.models import NoteCreate, NoteResponse, NoteUpdate
from open_notebook.domain.notebook import Note
from open_notebook.exceptions import (
InvalidInputError,
NotFoundError,
OpenNotebookError,
)
router = APIRouter()
@router.get("/notes", response_model=List[NoteResponse])
async def get_notes(
notebook_id: Optional[str] = Query(None, description="Filter by notebook ID"),
):
"""Get all notes with optional notebook filtering."""
try:
if notebook_id:
# Get notes for a specific notebook
from open_notebook.domain.notebook import Notebook
notebook = await Notebook.get(notebook_id)
notes = await notebook.get_notes()
else:
# Get all notes
notes = await Note.get_all(order_by="updated desc")
return [
NoteResponse(
id=note.id or "",
title=note.title,
content=note.content,
note_type=note.note_type,
created=str(note.created),
updated=str(note.updated),
)
for note in notes
]
except HTTPException:
raise
except NotFoundError:
raise HTTPException(status_code=404, detail="Notebook not found")
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching notes: {str(e)}")
raise HTTPException(status_code=500, detail=f"Error fetching notes: {str(e)}")
@router.post("/notes", response_model=NoteResponse)
async def create_note(note_data: NoteCreate):
"""Create a new note."""
try:
# Auto-generate title if not provided and it's an AI note
title = note_data.title
if not title and note_data.note_type == "ai" and note_data.content:
from open_notebook.graphs.prompt import graph as prompt_graph
prompt = "Based on the Note below, please provide a Title for this content, with max 15 words"
# LangGraph accepts a partial state dict at runtime, but its typed
# overloads require the full state type (langgraph typing limitation).
result = await prompt_graph.ainvoke( # type: ignore[call-overload]
{
"input_text": note_data.content,
"prompt": prompt,
}
)
title = result.get("output", "Untitled Note")
# Validate note_type
note_type: Optional[Literal["human", "ai"]] = None
if note_data.note_type in ("human", "ai"):
note_type = note_data.note_type # type: ignore[assignment]
elif note_data.note_type is not None:
raise HTTPException(
status_code=400, detail="note_type must be 'human' or 'ai'"
)
new_note = Note(
title=title,
content=note_data.content,
note_type=note_type,
)
command_id = await new_note.save()
# Add to notebook if specified
if note_data.notebook_id:
from open_notebook.domain.notebook import Notebook
# Verify the notebook exists (raises NotFoundError -> 404)
await Notebook.get(note_data.notebook_id)
await new_note.add_to_notebook(note_data.notebook_id)
return NoteResponse(
id=new_note.id or "",
title=new_note.title,
content=new_note.content,
note_type=new_note.note_type,
created=str(new_note.created),
updated=str(new_note.updated),
command_id=str(command_id) if command_id else None,
)
except HTTPException:
raise
except NotFoundError:
raise HTTPException(status_code=404, detail="Notebook not found")
except InvalidInputError as e:
raise HTTPException(status_code=400, detail=str(e))
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error creating note: {str(e)}")
raise HTTPException(status_code=500, detail=f"Error creating note: {str(e)}")
@router.get("/notes/{note_id}", response_model=NoteResponse)
async def get_note(note_id: str):
"""Get a specific note by ID."""
try:
note = await Note.get(note_id)
return NoteResponse(
id=note.id or "",
title=note.title,
content=note.content,
note_type=note.note_type,
created=str(note.created),
updated=str(note.updated),
)
except HTTPException:
raise
except NotFoundError:
raise HTTPException(status_code=404, detail="Note not found")
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching note {note_id}: {str(e)}")
raise HTTPException(status_code=500, detail=f"Error fetching note: {str(e)}")
@router.put("/notes/{note_id}", response_model=NoteResponse)
async def update_note(note_id: str, note_update: NoteUpdate):
"""Update a note."""
try:
note = await Note.get(note_id)
# Update only provided fields
if note_update.title is not None:
note.title = note_update.title
if note_update.content is not None:
note.content = note_update.content
if note_update.note_type is not None:
if note_update.note_type in ("human", "ai"):
note.note_type = note_update.note_type # type: ignore[assignment]
else:
raise HTTPException(
status_code=400, detail="note_type must be 'human' or 'ai'"
)
command_id = await note.save()
return NoteResponse(
id=note.id or "",
title=note.title,
content=note.content,
note_type=note.note_type,
created=str(note.created),
updated=str(note.updated),
command_id=str(command_id) if command_id else None,
)
except HTTPException:
raise
except NotFoundError:
raise HTTPException(status_code=404, detail="Note not found")
except InvalidInputError as e:
raise HTTPException(status_code=400, detail=str(e))
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error updating note {note_id}: {str(e)}")
raise HTTPException(status_code=500, detail=f"Error updating note: {str(e)}")
@router.delete("/notes/{note_id}")
async def delete_note(note_id: str):
"""Delete a note."""
try:
note = await Note.get(note_id)
await note.delete()
return {"message": "Note deleted successfully"}
except HTTPException:
raise
except NotFoundError:
raise HTTPException(status_code=404, detail="Note not found")
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error deleting note {note_id}: {str(e)}")
raise HTTPException(status_code=500, detail=f"Error deleting note: {str(e)}")
api/routers/podcasts.py
+432
View File
@@ -0,0 +1,432 @@
from typing import List, Optional
from fastapi import APIRouter, HTTPException
from fastapi.responses import FileResponse
from loguru import logger
from pydantic import BaseModel
from api.podcast_service import (
PodcastGenerationRequest,
PodcastGenerationResponse,
PodcastService,
)
from open_notebook.ai.models import Model
from open_notebook.exceptions import OpenNotebookError
from open_notebook.podcasts.audio_paths import resolve_contained_audio_path
from open_notebook.podcasts.models import PodcastEpisode
router = APIRouter()
# Model reference fields stored in the denormalized profile snapshots on an
# episode, mapped to the resolved display fields the frontend renders
# ("provider / name" rows in EpisodeCard). Mirrors the speaker_config ->
# speaker_config_name precedent in api/routers/episode_profiles.py.
_EPISODE_PROFILE_MODEL_FIELDS = {
"outline_llm": ("outline_model_provider", "outline_model_name"),
"transcript_llm": ("transcript_model_provider", "transcript_model_name"),
}
_SPEAKER_PROFILE_MODEL_FIELDS = {
"voice_model": ("voice_model_provider", "voice_model_name"),
}
def _collect_snapshot_model_ids(episodes: List[PodcastEpisode]) -> List[str]:
"""Collect the distinct model record IDs referenced by episode snapshots."""
ids = set()
for episode in episodes:
for field in _EPISODE_PROFILE_MODEL_FIELDS:
ref = (episode.episode_profile or {}).get(field)
if ref:
ids.add(str(ref))
for field in _SPEAKER_PROFILE_MODEL_FIELDS:
ref = (episode.speaker_profile or {}).get(field)
if ref:
ids.add(str(ref))
return sorted(ids)
def _with_resolved_model_fields(
snapshot: dict,
field_map: dict,
models_by_id: dict,
) -> dict:
"""Return a copy of a profile snapshot with resolved model display fields.
Only sets the display fields when the reference resolves; unresolvable
references (deleted model) and legacy snapshots without references are
left untouched so the frontend can fall back to the historical
provider/model strings, then to a placeholder.
"""
enriched = dict(snapshot or {})
for ref_field, (provider_field, name_field) in field_map.items():
ref = enriched.get(ref_field)
info = models_by_id.get(str(ref)) if ref else None
if info:
enriched[provider_field] = info["provider"]
enriched[name_field] = info["name"]
return enriched
async def _resolve_snapshot_models(
episodes: List[PodcastEpisode],
) -> dict:
"""Batch-resolve every model reference in the episodes' snapshots.
One query for the whole list (see Model.get_display_info_for_ids) - a
failure degrades to no resolved fields rather than failing the request.
"""
try:
return await Model.get_display_info_for_ids(
_collect_snapshot_model_ids(episodes)
)
except Exception as e:
logger.warning(f"Error batch-resolving snapshot model references: {str(e)}")
return {}
def _delete_episode_audio(episode: PodcastEpisode, episode_id: str) -> None:
"""Best-effort unlink of an episode's audio file, refusing invalid paths.
Shared by the delete and retry endpoints. Legacy/escaping audio_file
values (resolve_contained_audio_path -> None) are logged and skipped.
"""
if not episode.audio_file:
return
audio_path = resolve_contained_audio_path(episode.audio_file)
if audio_path is None:
logger.warning(
f"Refusing to delete audio file outside podcasts directory "
f"for episode {episode_id}: {episode.audio_file}"
)
elif audio_path.exists():
try:
audio_path.unlink()
logger.info(f"Deleted audio file: {audio_path}")
except Exception as e:
logger.warning(f"Failed to delete audio file {audio_path}: {e}")
class PodcastEpisodeResponse(BaseModel):
id: str
name: str
episode_profile: dict
speaker_profile: dict
briefing: str
audio_file: Optional[str] = None
audio_url: Optional[str] = None
transcript: Optional[dict] = None
outline: Optional[dict] = None
created: Optional[str] = None
job_status: Optional[str] = None
error_message: Optional[str] = None
@router.post("/podcasts/generate", response_model=PodcastGenerationResponse)
async def generate_podcast(request: PodcastGenerationRequest):
"""
Generate a podcast episode using Episode Profiles.
Returns immediately with job ID for status tracking.
"""
try:
job_id = await PodcastService.submit_generation_job(
episode_profile_name=request.episode_profile,
speaker_profile_name=request.speaker_profile,
episode_name=request.episode_name,
notebook_id=request.notebook_id,
content=request.content,
briefing_suffix=request.briefing_suffix,
)
return PodcastGenerationResponse(
job_id=job_id,
status="submitted",
message=f"Podcast generation started for episode '{request.episode_name}'",
episode_profile=request.episode_profile,
episode_name=request.episode_name,
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error generating podcast: {str(e)}")
raise HTTPException(
status_code=500, detail="Failed to generate podcast"
)
@router.get("/podcasts/jobs/{job_id}")
async def get_podcast_job_status(job_id: str):
"""Get the status of a podcast generation job"""
try:
status_data = await PodcastService.get_job_status(job_id)
return status_data
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching podcast job status: {str(e)}")
raise HTTPException(
status_code=500, detail="Failed to fetch job status"
)
@router.get("/podcasts/episodes", response_model=List[PodcastEpisodeResponse])
async def list_podcast_episodes():
"""List all podcast episodes"""
try:
episodes = await PodcastService.list_episodes()
# Batch-fetch job status for every episode with a command in one
# query instead of one round trip per episode (see
# PodcastEpisode.get_job_details_for_commands docstring).
try:
details_by_command = await PodcastEpisode.get_job_details_for_commands(
[episode.command for episode in episodes if episode.command]
)
except Exception as e:
logger.warning(f"Error batch-fetching podcast job statuses: {str(e)}")
details_by_command = {}
# Batch-resolve the snapshots' model references (outline_llm,
# transcript_llm, voice_model) to display fields in one query
# instead of one lookup per episode.
models_by_id = await _resolve_snapshot_models(episodes)
response_episodes = []
for episode in episodes:
# Skip incomplete episodes without command or audio
if not episode.command and not episode.audio_file:
continue
# Get job status and error message if available
job_status = None
error_message = None
if episode.command:
detail = details_by_command.get(str(episode.command))
if detail is not None:
job_status = detail["status"]
error_message = detail["error_message"]
else:
job_status = "unknown"
else:
# No command but has audio file = completed import
job_status = "completed"
audio_url = None
audio_path = resolve_contained_audio_path(episode.audio_file)
if audio_path is not None and audio_path.exists():
audio_url = f"/api/podcasts/episodes/{episode.id}/audio"
response_episodes.append(
PodcastEpisodeResponse(
id=str(episode.id),
name=episode.name,
episode_profile=_with_resolved_model_fields(
episode.episode_profile,
_EPISODE_PROFILE_MODEL_FIELDS,
models_by_id,
),
speaker_profile=_with_resolved_model_fields(
episode.speaker_profile,
_SPEAKER_PROFILE_MODEL_FIELDS,
models_by_id,
),
briefing=episode.briefing,
audio_file=episode.audio_file,
audio_url=audio_url,
transcript=episode.transcript,
outline=episode.outline,
created=str(episode.created) if episode.created else None,
job_status=job_status,
error_message=error_message,
)
)
return response_episodes
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error listing podcast episodes: {str(e)}")
raise HTTPException(
status_code=500, detail="Failed to list podcast episodes"
)
@router.get("/podcasts/episodes/{episode_id}", response_model=PodcastEpisodeResponse)
async def get_podcast_episode(episode_id: str):
"""Get a specific podcast episode"""
try:
episode = await PodcastService.get_episode(episode_id)
# Get job status and error message if available
job_status = None
error_message = None
if episode.command:
try:
detail = await episode.get_job_detail()
job_status = detail["status"]
error_message = detail["error_message"]
except Exception:
job_status = "unknown"
else:
# No command but has audio file = completed import
job_status = "completed" if episode.audio_file else "unknown"
audio_url = None
audio_path = resolve_contained_audio_path(episode.audio_file)
if audio_path is not None and audio_path.exists():
audio_url = f"/api/podcasts/episodes/{episode.id}/audio"
models_by_id = await _resolve_snapshot_models([episode])
return PodcastEpisodeResponse(
id=str(episode.id),
name=episode.name,
episode_profile=_with_resolved_model_fields(
episode.episode_profile,
_EPISODE_PROFILE_MODEL_FIELDS,
models_by_id,
),
speaker_profile=_with_resolved_model_fields(
episode.speaker_profile,
_SPEAKER_PROFILE_MODEL_FIELDS,
models_by_id,
),
briefing=episode.briefing,
audio_file=episode.audio_file,
audio_url=audio_url,
transcript=episode.transcript,
outline=episode.outline,
created=str(episode.created) if episode.created else None,
job_status=job_status,
error_message=error_message,
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching podcast episode: {str(e)}")
raise HTTPException(status_code=404, detail="Episode not found")
@router.get("/podcasts/episodes/{episode_id}/audio")
async def stream_podcast_episode_audio(episode_id: str):
"""Stream the audio file associated with a podcast episode"""
try:
episode = await PodcastService.get_episode(episode_id)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching podcast episode for audio: {str(e)}")
raise HTTPException(status_code=404, detail="Episode not found")
if not episode.audio_file:
raise HTTPException(status_code=404, detail="Episode has no audio file")
audio_path = resolve_contained_audio_path(episode.audio_file)
if audio_path is None:
logger.warning(
f"Blocked audio access outside podcasts directory for episode "
f"{episode_id}: {episode.audio_file}"
)
raise HTTPException(status_code=403, detail="Access to file denied")
if not audio_path.exists():
raise HTTPException(status_code=404, detail="Audio file not found on disk")
return FileResponse(
audio_path,
media_type="audio/mpeg",
filename=audio_path.name,
)
@router.post("/podcasts/episodes/{episode_id}/retry")
async def retry_podcast_episode(episode_id: str):
"""Retry a failed podcast episode by deleting it and submitting a new job"""
try:
episode = await PodcastService.get_episode(episode_id)
# Validate episode is in a failed state
detail = await episode.get_job_detail()
if detail["status"] not in ("failed", "error"):
raise HTTPException(
status_code=400,
detail=f"Episode is not in a failed state (current: {detail['status']})",
)
# Extract params for re-submission
ep_profile_name = episode.episode_profile.get("name")
sp_profile_name = episode.speaker_profile.get("name")
episode_name = episode.name
content = episode.content
if not ep_profile_name or not sp_profile_name:
raise HTTPException(
status_code=400,
detail="Cannot retry: episode or speaker profile name missing from stored data",
)
# Delete audio file if any
_delete_episode_audio(episode, episode_id)
# Delete the failed episode
await episode.delete()
# Submit a new job
job_id = await PodcastService.submit_generation_job(
episode_profile_name=ep_profile_name,
speaker_profile_name=sp_profile_name,
episode_name=episode_name,
content=content,
)
return {"job_id": job_id, "message": "Retry submitted successfully"}
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error retrying podcast episode: {str(e)}")
raise HTTPException(
status_code=500, detail="Failed to retry episode"
)
@router.delete("/podcasts/episodes/{episode_id}")
async def delete_podcast_episode(episode_id: str):
"""Delete a podcast episode and its associated audio file"""
try:
# Get the episode first to check if it exists and get the audio file path
episode = await PodcastService.get_episode(episode_id)
# Delete the physical audio file if it exists
_delete_episode_audio(episode, episode_id)
# Delete the episode from the database
await episode.delete()
logger.info(f"Deleted podcast episode: {episode_id}")
return {"message": "Episode deleted successfully", "episode_id": episode_id}
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error deleting podcast episode: {str(e)}")
raise HTTPException(
status_code=500, detail="Failed to delete episode"
)
api/routers/providers.py
+35
View File
@@ -0,0 +1,35 @@
"""
Providers Router
Exposes the provider registry (open_notebook/ai/provider_registry.py) so
clients can enumerate supported providers and their metadata instead of
keeping their own copies.
Endpoints:
- GET /providers - List all supported providers with metadata
"""
from typing import List
from fastapi import APIRouter
from api.credentials_service import check_env_configured
from api.models import ProviderInfoResponse
from open_notebook.ai.provider_registry import PROVIDERS
router = APIRouter(prefix="/providers", tags=["providers"])
@router.get("", response_model=List[ProviderInfoResponse])
async def list_providers():
"""List all supported AI providers with their registry metadata."""
return [
ProviderInfoResponse(
name=spec.name,
display_name=spec.display_name,
modalities=list(spec.modalities),
docs_url=spec.docs_url,
env_configured=check_env_configured(spec.name),
)
for spec in PROVIDERS.values()
]
api/routers/search.py
+238
View File
@@ -0,0 +1,238 @@
import json
from typing import AsyncGenerator
from fastapi import APIRouter, HTTPException
from fastapi.responses import StreamingResponse
from loguru import logger
from api.models import AskRequest, AskResponse, SearchRequest, SearchResponse
from open_notebook.ai.models import Model, model_manager
from open_notebook.domain.notebook import text_search, vector_search
from open_notebook.exceptions import (
DatabaseOperationError,
InvalidInputError,
OpenNotebookError,
)
from open_notebook.graphs.ask import graph as ask_graph
router = APIRouter()
@router.post("/search", response_model=SearchResponse)
async def search_knowledge_base(search_request: SearchRequest):
"""Search the knowledge base using text or vector search."""
try:
if search_request.type == "vector":
# Check if embedding model is available for vector search
if not await model_manager.get_embedding_model():
raise HTTPException(
status_code=400,
detail="Vector search requires an embedding model. Please configure one in the Models section.",
)
results = await vector_search(
keyword=search_request.query,
results=search_request.limit,
source=search_request.search_sources,
note=search_request.search_notes,
minimum_score=search_request.minimum_score,
)
else:
# Text search
results = await text_search(
keyword=search_request.query,
results=search_request.limit,
source=search_request.search_sources,
note=search_request.search_notes,
)
return SearchResponse(
results=results or [],
total_count=len(results) if results else 0,
search_type=search_request.type,
)
except InvalidInputError as e:
raise HTTPException(status_code=400, detail=str(e))
except DatabaseOperationError as e:
logger.error(f"Database error during search: {str(e)}")
raise HTTPException(status_code=500, detail=f"Search failed: {str(e)}")
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Unexpected error during search: {str(e)}")
raise HTTPException(status_code=500, detail=f"Search failed: {str(e)}")
async def stream_ask_response(
question: str, strategy_model: Model, answer_model: Model, final_answer_model: Model
) -> AsyncGenerator[str, None]:
"""Stream the ask response as Server-Sent Events."""
try:
final_answer = None
# LangGraph accepts a partial state dict at runtime, but its typed
# overloads require the full state type (langgraph typing limitation).
async for chunk in ask_graph.astream( # type: ignore[call-overload]
input=dict(question=question),
config=dict(
configurable=dict(
strategy_model=strategy_model.id,
answer_model=answer_model.id,
final_answer_model=final_answer_model.id,
)
),
stream_mode="updates",
):
if "agent" in chunk:
strategy_data = {
"type": "strategy",
"reasoning": chunk["agent"]["strategy"].reasoning,
"searches": [
{"term": search.term, "instructions": search.instructions}
for search in chunk["agent"]["strategy"].searches
],
}
yield f"data: {json.dumps(strategy_data)}\n\n"
elif "provide_answer" in chunk:
for answer in chunk["provide_answer"]["answers"]:
answer_data = {"type": "answer", "content": answer}
yield f"data: {json.dumps(answer_data)}\n\n"
elif "write_final_answer" in chunk:
final_answer = chunk["write_final_answer"]["final_answer"]
final_data = {"type": "final_answer", "content": final_answer}
yield f"data: {json.dumps(final_data)}\n\n"
# Send completion signal
completion_data = {"type": "complete", "final_answer": final_answer}
yield f"data: {json.dumps(completion_data)}\n\n"
except Exception as e:
from open_notebook.utils.error_classifier import classify_error
_, user_message = classify_error(e)
logger.error(f"Error in ask streaming: {str(e)}")
error_data = {"type": "error", "message": user_message}
yield f"data: {json.dumps(error_data)}\n\n"
@router.post("/search/ask")
async def ask_knowledge_base(ask_request: AskRequest):
"""Ask the knowledge base a question using AI models."""
try:
# Validate models exist
strategy_model = await Model.get(ask_request.strategy_model)
answer_model = await Model.get(ask_request.answer_model)
final_answer_model = await Model.get(ask_request.final_answer_model)
if not strategy_model:
raise HTTPException(
status_code=400,
detail=f"Strategy model {ask_request.strategy_model} not found",
)
if not answer_model:
raise HTTPException(
status_code=400,
detail=f"Answer model {ask_request.answer_model} not found",
)
if not final_answer_model:
raise HTTPException(
status_code=400,
detail=f"Final answer model {ask_request.final_answer_model} not found",
)
# Check if embedding model is available
if not await model_manager.get_embedding_model():
raise HTTPException(
status_code=400,
detail="Ask feature requires an embedding model. Please configure one in the Models section.",
)
# For streaming response
return StreamingResponse(
stream_ask_response(
ask_request.question, strategy_model, answer_model, final_answer_model
),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"X-Accel-Buffering": "no",
},
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error in ask endpoint: {str(e)}")
raise HTTPException(status_code=500, detail=f"Ask operation failed: {str(e)}")
@router.post("/search/ask/simple", response_model=AskResponse)
async def ask_knowledge_base_simple(ask_request: AskRequest):
"""Ask the knowledge base a question and return a simple response (non-streaming)."""
try:
# Validate models exist
strategy_model = await Model.get(ask_request.strategy_model)
answer_model = await Model.get(ask_request.answer_model)
final_answer_model = await Model.get(ask_request.final_answer_model)
if not strategy_model:
raise HTTPException(
status_code=400,
detail=f"Strategy model {ask_request.strategy_model} not found",
)
if not answer_model:
raise HTTPException(
status_code=400,
detail=f"Answer model {ask_request.answer_model} not found",
)
if not final_answer_model:
raise HTTPException(
status_code=400,
detail=f"Final answer model {ask_request.final_answer_model} not found",
)
# Check if embedding model is available
if not await model_manager.get_embedding_model():
raise HTTPException(
status_code=400,
detail="Ask feature requires an embedding model. Please configure one in the Models section.",
)
# Run the ask graph and get final result
final_answer = None
# LangGraph accepts a partial state dict at runtime, but its typed
# overloads require the full state type (langgraph typing limitation).
async for chunk in ask_graph.astream( # type: ignore[call-overload]
input=dict(question=ask_request.question),
config=dict(
configurable=dict(
strategy_model=strategy_model.id,
answer_model=answer_model.id,
final_answer_model=final_answer_model.id,
)
),
stream_mode="updates",
):
if "write_final_answer" in chunk:
final_answer = chunk["write_final_answer"]["final_answer"]
if not final_answer:
raise HTTPException(status_code=500, detail="No answer generated")
return AskResponse(answer=final_answer, question=ask_request.question)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error in ask simple endpoint: {str(e)}")
raise HTTPException(status_code=500, detail=f"Ask operation failed: {str(e)}")
api/routers/settings.py
+101
View File
@@ -0,0 +1,101 @@
from fastapi import APIRouter, HTTPException
from loguru import logger
from api.models import SettingsResponse, SettingsUpdate
from open_notebook.domain.content_settings import ContentSettings
from open_notebook.exceptions import (
InvalidInputError,
OpenNotebookError,
)
router = APIRouter()
@router.get("/settings", response_model=SettingsResponse)
async def get_settings():
"""Get all application settings."""
try:
settings: ContentSettings = await ContentSettings.get_instance() # type: ignore[assignment]
return SettingsResponse(
default_content_processing_engine_doc=settings.default_content_processing_engine_doc,
default_content_processing_engine_url=settings.default_content_processing_engine_url,
default_embedding_option=settings.default_embedding_option,
auto_delete_files=settings.auto_delete_files,
docling_ocr=settings.docling_ocr,
youtube_preferred_languages=settings.youtube_preferred_languages,
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching settings: {str(e)}")
raise HTTPException(
status_code=500, detail="Error fetching settings"
)
@router.put("/settings", response_model=SettingsResponse)
async def update_settings(settings_update: SettingsUpdate):
"""Update application settings."""
try:
settings: ContentSettings = await ContentSettings.get_instance() # type: ignore[assignment]
# Update only provided fields
if settings_update.default_content_processing_engine_doc is not None:
# Cast to proper literal type
from typing import Literal, cast
settings.default_content_processing_engine_doc = cast(
Literal["auto", "docling", "simple"],
settings_update.default_content_processing_engine_doc,
)
if settings_update.default_content_processing_engine_url is not None:
from typing import Literal, cast
settings.default_content_processing_engine_url = cast(
Literal["auto", "firecrawl", "jina", "crawl4ai", "simple"],
settings_update.default_content_processing_engine_url,
)
if settings_update.default_embedding_option is not None:
from typing import Literal, cast
settings.default_embedding_option = cast(
Literal["ask", "always", "never"],
settings_update.default_embedding_option,
)
if settings_update.auto_delete_files is not None:
from typing import Literal, cast
settings.auto_delete_files = cast(
Literal["yes", "no"], settings_update.auto_delete_files
)
if settings_update.docling_ocr is not None:
settings.docling_ocr = settings_update.docling_ocr
if settings_update.youtube_preferred_languages is not None:
settings.youtube_preferred_languages = (
settings_update.youtube_preferred_languages
)
await settings.update()
return SettingsResponse(
default_content_processing_engine_doc=settings.default_content_processing_engine_doc,
default_content_processing_engine_url=settings.default_content_processing_engine_url,
default_embedding_option=settings.default_embedding_option,
auto_delete_files=settings.auto_delete_files,
docling_ocr=settings.docling_ocr,
youtube_preferred_languages=settings.youtube_preferred_languages,
)
except HTTPException:
raise
except InvalidInputError as e:
raise HTTPException(status_code=400, detail=str(e))
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error updating settings: {str(e)}")
raise HTTPException(
status_code=500, detail="Error updating settings"
)
api/routers/source_chat.py
+453
View File
@@ -0,0 +1,453 @@
import asyncio
import json
from typing import AsyncGenerator, List, Optional
from fastapi import APIRouter, HTTPException, Path
from fastapi.responses import StreamingResponse
from langchain_core.messages import HumanMessage
from langchain_core.runnables import RunnableConfig
from loguru import logger
from pydantic import BaseModel, Field
from api.routers._chat_shared import (
ChatMessage,
SuccessResponse,
extract_chat_messages,
get_source_or_404,
get_verified_source_session,
)
from open_notebook.database.repository import ensure_record_id, repo_query
from open_notebook.domain.notebook import ChatSession
from open_notebook.exceptions import (
NotFoundError,
OpenNotebookError,
)
from open_notebook.graphs.source_chat import source_chat_graph as source_chat_graph
from open_notebook.utils.graph_utils import get_session_message_count
router = APIRouter()
# Request/Response models
class CreateSourceChatSessionRequest(BaseModel):
source_id: str = Field(..., description="Source ID to create chat session for")
title: Optional[str] = Field(None, description="Optional session title")
model_override: Optional[str] = Field(
None, description="Optional model override for this session"
)
class UpdateSourceChatSessionRequest(BaseModel):
title: Optional[str] = Field(None, description="New session title")
model_override: Optional[str] = Field(
None, description="Model override for this session"
)
class ContextIndicator(BaseModel):
sources: List[str] = Field(
default_factory=list, description="Source IDs used in context"
)
insights: List[str] = Field(
default_factory=list, description="Insight IDs used in context"
)
notes: List[str] = Field(
default_factory=list, description="Note IDs used in context"
)
class SourceChatSessionResponse(BaseModel):
id: str = Field(..., description="Session ID")
title: str = Field(..., description="Session title")
source_id: str = Field(..., description="Source ID")
model_override: Optional[str] = Field(
None, description="Model override for this session"
)
created: str = Field(..., description="Creation timestamp")
updated: str = Field(..., description="Last update timestamp")
message_count: Optional[int] = Field(
None, description="Number of messages in session"
)
class SourceChatSessionWithMessagesResponse(SourceChatSessionResponse):
messages: List[ChatMessage] = Field(
default_factory=list, description="Session messages"
)
context_indicators: Optional[ContextIndicator] = Field(
None, description="Context indicators from last response"
)
class SendMessageRequest(BaseModel):
message: str = Field(..., description="User message content")
model_override: Optional[str] = Field(
None, description="Optional model override for this message"
)
@router.post(
"/sources/{source_id}/chat/sessions", response_model=SourceChatSessionResponse
)
async def create_source_chat_session(
request: CreateSourceChatSessionRequest,
source_id: str = Path(..., description="Source ID"),
):
"""Create a new chat session for a source."""
try:
# Verify source exists (normalizes the ID and 404s if missing)
full_source_id, _source = await get_source_or_404(source_id)
# Create new session with model_override support
session = ChatSession(
title=request.title or f"Source Chat {asyncio.get_event_loop().time():.0f}",
model_override=request.model_override,
)
await session.save()
# Relate session to source using "refers_to" relation
await session.relate("refers_to", full_source_id)
return SourceChatSessionResponse(
id=session.id or "",
title=session.title or "Untitled Session",
source_id=source_id,
model_override=session.model_override,
created=str(session.created),
updated=str(session.updated),
message_count=0,
)
except NotFoundError:
raise HTTPException(status_code=404, detail="Source not found")
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error creating source chat session: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error creating source chat session: {str(e)}"
)
@router.get(
"/sources/{source_id}/chat/sessions", response_model=List[SourceChatSessionResponse]
)
async def get_source_chat_sessions(source_id: str = Path(..., description="Source ID")):
"""Get all chat sessions for a source."""
try:
# Verify source exists (normalizes the ID and 404s if missing)
full_source_id, _source = await get_source_or_404(source_id)
# Get sessions that refer to this source - first get relations, then sessions
relations = await repo_query(
"SELECT in FROM refers_to WHERE out = $source_id",
{"source_id": ensure_record_id(full_source_id)},
)
sessions = []
for relation in relations:
session_id_raw = relation.get("in")
if session_id_raw:
session_id = str(session_id_raw)
session_result = await repo_query(
"SELECT * FROM $id", {"id": ensure_record_id(session_id)}
)
if session_result and len(session_result) > 0:
session_data = session_result[0]
# Get message count from LangGraph state
msg_count = await get_session_message_count(
source_chat_graph, session_id
)
sessions.append(
SourceChatSessionResponse(
id=session_data.get("id") or "",
title=session_data.get("title") or "Untitled Session",
source_id=source_id,
model_override=session_data.get("model_override"),
created=str(session_data.get("created")),
updated=str(session_data.get("updated")),
message_count=msg_count,
)
)
# Sort sessions by created date (newest first)
sessions.sort(key=lambda x: x.created, reverse=True)
return sessions
except NotFoundError:
raise HTTPException(status_code=404, detail="Source not found")
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching source chat sessions: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error fetching source chat sessions: {str(e)}"
)
@router.get(
"/sources/{source_id}/chat/sessions/{session_id}",
response_model=SourceChatSessionWithMessagesResponse,
)
async def get_source_chat_session(
source_id: str = Path(..., description="Source ID"),
session_id: str = Path(..., description="Session ID"),
):
"""Get a specific source chat session with its messages."""
try:
# Verify source + session exist and are related (404s otherwise)
_full_source_id, _source, full_session_id, session = (
await get_verified_source_session(source_id, session_id)
)
# Get session state from LangGraph to retrieve messages
# Use sync get_state() in a thread since SqliteSaver doesn't support async
thread_state = await asyncio.to_thread(
source_chat_graph.get_state,
config=RunnableConfig(configurable={"thread_id": full_session_id}),
)
# Extract messages from state
messages: list[ChatMessage] = []
context_indicators = None
if thread_state and thread_state.values:
# Extract messages
if "messages" in thread_state.values:
messages = extract_chat_messages(thread_state.values["messages"])
# Extract context indicators from the last state
if "context_indicators" in thread_state.values:
context_data = thread_state.values["context_indicators"]
context_indicators = ContextIndicator(
sources=context_data.get("sources", []),
insights=context_data.get("insights", []),
notes=context_data.get("notes", []),
)
return SourceChatSessionWithMessagesResponse(
id=session.id or "",
title=session.title or "Untitled Session",
source_id=source_id,
model_override=getattr(session, "model_override", None),
created=str(session.created),
updated=str(session.updated),
message_count=len(messages),
messages=messages,
context_indicators=context_indicators,
)
except NotFoundError:
raise HTTPException(status_code=404, detail="Source or session not found")
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching source chat session: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error fetching source chat session: {str(e)}"
)
@router.put(
"/sources/{source_id}/chat/sessions/{session_id}",
response_model=SourceChatSessionResponse,
)
async def update_source_chat_session(
request: UpdateSourceChatSessionRequest,
source_id: str = Path(..., description="Source ID"),
session_id: str = Path(..., description="Session ID"),
):
"""Update source chat session title and/or model override."""
try:
# Verify source + session exist and are related (404s otherwise)
_full_source_id, _source, full_session_id, session = (
await get_verified_source_session(source_id, session_id)
)
# Update session fields
if request.title is not None:
session.title = request.title
if request.model_override is not None:
session.model_override = request.model_override
await session.save()
# Get message count from LangGraph state
msg_count = await get_session_message_count(source_chat_graph, full_session_id)
return SourceChatSessionResponse(
id=session.id or "",
title=session.title or "Untitled Session",
source_id=source_id,
model_override=getattr(session, "model_override", None),
created=str(session.created),
updated=str(session.updated),
message_count=msg_count,
)
except NotFoundError:
raise HTTPException(status_code=404, detail="Source or session not found")
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error updating source chat session: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error updating source chat session: {str(e)}"
)
@router.delete(
"/sources/{source_id}/chat/sessions/{session_id}", response_model=SuccessResponse
)
async def delete_source_chat_session(
source_id: str = Path(..., description="Source ID"),
session_id: str = Path(..., description="Session ID"),
):
"""Delete a source chat session."""
try:
# Verify source + session exist and are related (404s otherwise)
_full_source_id, _source, full_session_id, session = (
await get_verified_source_session(source_id, session_id)
)
await session.delete()
return SuccessResponse(
success=True, message="Source chat session deleted successfully"
)
except NotFoundError:
raise HTTPException(status_code=404, detail="Source or session not found")
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error deleting source chat session: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error deleting source chat session: {str(e)}"
)
async def stream_source_chat_response(
session_id: str, source_id: str, message: str, model_override: Optional[str] = None
) -> AsyncGenerator[str, None]:
"""Stream the source chat response as Server-Sent Events."""
try:
# Get current state
# Use sync get_state() in a thread since SqliteSaver doesn't support async
current_state = await asyncio.to_thread(
source_chat_graph.get_state,
config=RunnableConfig(configurable={"thread_id": session_id}),
)
# Prepare state for execution
state_values = current_state.values if current_state else {}
state_values["messages"] = state_values.get("messages", [])
state_values["source_id"] = source_id
state_values["model_override"] = model_override
# Add user message to state
user_message = HumanMessage(content=message)
state_values["messages"].append(user_message)
# Send user message event
user_event = {"type": "user_message", "content": message, "timestamp": None}
yield f"data: {json.dumps(user_event)}\n\n"
# Run the synchronous LangGraph invoke in a thread so it doesn't block the
# event loop. While blocked, even the already-yielded SSE events can't
# flush and every other request stalls until the LLM finishes. Mirrors the
# get_state() calls above.
# The lambda pins down which `invoke` overload is used; asyncio.to_thread
# can't resolve overloaded callables on its own. The ignore is a langgraph
# typing limitation: it accepts a partial state dict at runtime, but the
# signature requires the full state type.
result = await asyncio.to_thread(
lambda: source_chat_graph.invoke(
input=state_values, # type: ignore[arg-type]
config=RunnableConfig(
configurable={"thread_id": session_id, "model_id": model_override}
),
)
)
# Stream the complete AI response
if "messages" in result:
for msg in result["messages"]:
if hasattr(msg, "type") and msg.type == "ai":
ai_event = {
"type": "ai_message",
"content": msg.content if hasattr(msg, "content") else str(msg),
"timestamp": None,
}
yield f"data: {json.dumps(ai_event)}\n\n"
# Stream context indicators
if "context_indicators" in result:
context_event = {
"type": "context_indicators",
"data": result["context_indicators"],
}
yield f"data: {json.dumps(context_event)}\n\n"
# Send completion signal
completion_event = {"type": "complete"}
yield f"data: {json.dumps(completion_event)}\n\n"
except Exception as e:
from open_notebook.utils.error_classifier import classify_error
_, error_message = classify_error(e)
logger.error(f"Error in source chat streaming: {str(e)}")
error_event = {"type": "error", "message": error_message}
yield f"data: {json.dumps(error_event)}\n\n"
@router.post("/sources/{source_id}/chat/sessions/{session_id}/messages")
async def send_message_to_source_chat(
request: SendMessageRequest,
source_id: str = Path(..., description="Source ID"),
session_id: str = Path(..., description="Session ID"),
):
"""Send a message to source chat session with SSE streaming response."""
try:
# Verify source + session exist and are related (404s otherwise)
full_source_id, _source, full_session_id, session = (
await get_verified_source_session(source_id, session_id)
)
if not request.message:
raise HTTPException(status_code=400, detail="Message content is required")
# Determine model override (request override takes precedence over session override)
model_override = request.model_override or getattr(
session, "model_override", None
)
# Update session timestamp
await session.save()
# Return streaming response
return StreamingResponse(
stream_source_chat_response(
session_id=full_session_id,
source_id=full_source_id,
message=request.message,
model_override=model_override,
),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"X-Accel-Buffering": "no",
},
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error sending message to source chat: {str(e)}")
raise HTTPException(status_code=500, detail=f"Error sending message: {str(e)}")
api/routers/sources.py
+1143
View File
File diff suppressed because it is too large Load Diff
api/routers/speaker_profiles.py
+191
View File
@@ -0,0 +1,191 @@
from typing import Any, Dict, List, Optional
from fastapi import APIRouter, HTTPException
from loguru import logger
from pydantic import BaseModel, Field
from open_notebook.exceptions import OpenNotebookError
from open_notebook.podcasts.models import SpeakerProfile
router = APIRouter()
class SpeakerProfileResponse(BaseModel):
id: str
name: str
description: str
voice_model: Optional[str] = None
speakers: List[Dict[str, Any]]
def _profile_to_response(profile: SpeakerProfile) -> SpeakerProfileResponse:
return SpeakerProfileResponse(
id=str(profile.id),
name=profile.name,
description=profile.description or "",
voice_model=profile.voice_model,
speakers=profile.speakers,
)
@router.get("/speaker-profiles", response_model=List[SpeakerProfileResponse])
async def list_speaker_profiles():
"""List all available speaker profiles"""
try:
profiles = await SpeakerProfile.get_all(order_by="name asc")
return [_profile_to_response(p) for p in profiles]
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Failed to fetch speaker profiles: {e}")
raise HTTPException(
status_code=500, detail="Failed to fetch speaker profiles"
)
@router.get("/speaker-profiles/{profile_name}", response_model=SpeakerProfileResponse)
async def get_speaker_profile(profile_name: str):
"""Get a specific speaker profile by name"""
try:
profile = await SpeakerProfile.get_by_name(profile_name)
if not profile:
raise HTTPException(
status_code=404, detail=f"Speaker profile '{profile_name}' not found"
)
return _profile_to_response(profile)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Failed to fetch speaker profile '{profile_name}': {e}")
raise HTTPException(
status_code=500, detail="Failed to fetch speaker profile"
)
class SpeakerProfileCreate(BaseModel):
name: str = Field(..., description="Unique profile name")
description: str = Field("", description="Profile description")
voice_model: Optional[str] = Field(None, description="Model record ID for TTS")
speakers: List[Dict[str, Any]] = Field(
..., description="Array of speaker configurations"
)
@router.post("/speaker-profiles", response_model=SpeakerProfileResponse)
async def create_speaker_profile(profile_data: SpeakerProfileCreate):
"""Create a new speaker profile"""
try:
profile = SpeakerProfile(
name=profile_data.name,
description=profile_data.description,
voice_model=profile_data.voice_model,
speakers=profile_data.speakers,
)
await profile.save()
return _profile_to_response(profile)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Failed to create speaker profile: {e}")
raise HTTPException(
status_code=500, detail="Failed to create speaker profile"
)
@router.put("/speaker-profiles/{profile_id}", response_model=SpeakerProfileResponse)
async def update_speaker_profile(profile_id: str, profile_data: SpeakerProfileCreate):
"""Update an existing speaker profile"""
try:
profile = await SpeakerProfile.get(profile_id)
if not profile:
raise HTTPException(
status_code=404, detail=f"Speaker profile '{profile_id}' not found"
)
for field, value in profile_data.model_dump(exclude_unset=True).items():
setattr(profile, field, value)
await profile.save()
return _profile_to_response(profile)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Failed to update speaker profile: {e}")
raise HTTPException(
status_code=500, detail="Failed to update speaker profile"
)
@router.delete("/speaker-profiles/{profile_id}")
async def delete_speaker_profile(profile_id: str):
"""Delete a speaker profile"""
try:
profile = await SpeakerProfile.get(profile_id)
if not profile:
raise HTTPException(
status_code=404, detail=f"Speaker profile '{profile_id}' not found"
)
await profile.delete()
return {"message": "Speaker profile deleted successfully"}
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Failed to delete speaker profile: {e}")
raise HTTPException(
status_code=500, detail="Failed to delete speaker profile"
)
@router.post(
"/speaker-profiles/{profile_id}/duplicate", response_model=SpeakerProfileResponse
)
async def duplicate_speaker_profile(profile_id: str):
"""Duplicate a speaker profile"""
try:
original = await SpeakerProfile.get(profile_id)
if not original:
raise HTTPException(
status_code=404, detail=f"Speaker profile '{profile_id}' not found"
)
duplicate = SpeakerProfile(
name=f"{original.name} - Copy",
description=original.description,
voice_model=original.voice_model,
speakers=original.speakers,
)
await duplicate.save()
return _profile_to_response(duplicate)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Failed to duplicate speaker profile: {e}")
raise HTTPException(
status_code=500, detail="Failed to duplicate speaker profile"
)
api/routers/transformations.py
+273
View File
@@ -0,0 +1,273 @@
from typing import List
from fastapi import APIRouter, HTTPException
from loguru import logger
from api.models import (
DefaultPromptResponse,
DefaultPromptUpdate,
TransformationCreate,
TransformationExecuteRequest,
TransformationExecuteResponse,
TransformationResponse,
TransformationUpdate,
)
from open_notebook.ai.models import Model
from open_notebook.domain.transformation import DefaultPrompts, Transformation
from open_notebook.exceptions import InvalidInputError, OpenNotebookError
from open_notebook.graphs.transformation import graph as transformation_graph
router = APIRouter()
def _transformation_response(transformation: Transformation) -> TransformationResponse:
return TransformationResponse(
id=transformation.id or "",
name=transformation.name,
title=transformation.title,
description=transformation.description,
prompt=transformation.prompt,
apply_default=transformation.apply_default,
model_id=transformation.model_id,
created=str(transformation.created),
updated=str(transformation.updated),
)
@router.get("/transformations", response_model=List[TransformationResponse])
async def get_transformations():
"""Get all transformations."""
try:
transformations = await Transformation.get_all(order_by="name asc")
return [
_transformation_response(transformation)
for transformation in transformations
]
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching transformations: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error fetching transformations: {str(e)}"
)
@router.post("/transformations", response_model=TransformationResponse)
async def create_transformation(transformation_data: TransformationCreate):
"""Create a new transformation."""
try:
# Reject unknown model references up front (same check as execute);
# otherwise an invalid model_id is stored and only fails at run time.
if transformation_data.model_id:
model = await Model.get(transformation_data.model_id)
if not model:
raise HTTPException(status_code=404, detail="Model not found")
new_transformation = Transformation(
name=transformation_data.name,
title=transformation_data.title,
description=transformation_data.description,
prompt=transformation_data.prompt,
apply_default=transformation_data.apply_default,
model_id=transformation_data.model_id,
)
await new_transformation.save()
return _transformation_response(new_transformation)
except HTTPException:
raise
except InvalidInputError as e:
raise HTTPException(status_code=400, detail=str(e))
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error creating transformation: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error creating transformation: {str(e)}"
)
@router.post("/transformations/execute", response_model=TransformationExecuteResponse)
async def execute_transformation(execute_request: TransformationExecuteRequest):
"""Execute a transformation on input text."""
try:
# Validate transformation exists
transformation = await Transformation.get(execute_request.transformation_id)
if not transformation:
raise HTTPException(status_code=404, detail="Transformation not found")
model_id = execute_request.model_id or transformation.model_id
# Validate explicit or transformation-specific model exists.
# None is allowed so the graph can use the configured transformation default.
if model_id:
model = await Model.get(model_id)
if not model:
raise HTTPException(status_code=404, detail="Model not found")
# Execute the transformation.
# LangGraph accepts a partial state dict at runtime, but its typed
# overloads require the full state type (langgraph typing limitation).
result = await transformation_graph.ainvoke( # type: ignore[call-overload]
dict(
input_text=execute_request.input_text,
transformation=transformation,
),
config=dict(configurable={"model_id": model_id}),
)
return TransformationExecuteResponse(
output=result["output"],
transformation_id=execute_request.transformation_id,
model_id=model_id,
)
except HTTPException:
raise
except OpenNotebookError:
raise # Let global exception handlers return proper status codes
except Exception as e:
logger.error(f"Error executing transformation: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error executing transformation: {str(e)}"
)
@router.get("/transformations/default-prompt", response_model=DefaultPromptResponse)
async def get_default_prompt():
"""Get the default transformation prompt."""
try:
default_prompts: DefaultPrompts = await DefaultPrompts.get_instance() # type: ignore[assignment]
return DefaultPromptResponse(
transformation_instructions=default_prompts.transformation_instructions
or ""
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching default prompt: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error fetching default prompt: {str(e)}"
)
@router.put("/transformations/default-prompt", response_model=DefaultPromptResponse)
async def update_default_prompt(prompt_update: DefaultPromptUpdate):
"""Update the default transformation prompt."""
try:
default_prompts: DefaultPrompts = await DefaultPrompts.get_instance() # type: ignore[assignment]
default_prompts.transformation_instructions = (
prompt_update.transformation_instructions
)
await default_prompts.update()
return DefaultPromptResponse(
transformation_instructions=default_prompts.transformation_instructions
)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error updating default prompt: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error updating default prompt: {str(e)}"
)
@router.get(
"/transformations/{transformation_id}", response_model=TransformationResponse
)
async def get_transformation(transformation_id: str):
"""Get a specific transformation by ID."""
try:
transformation = await Transformation.get(transformation_id)
if not transformation:
raise HTTPException(status_code=404, detail="Transformation not found")
return _transformation_response(transformation)
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error fetching transformation {transformation_id}: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error fetching transformation: {str(e)}"
)
@router.put(
"/transformations/{transformation_id}", response_model=TransformationResponse
)
async def update_transformation(
transformation_id: str, transformation_update: TransformationUpdate
):
"""Update a transformation."""
try:
transformation = await Transformation.get(transformation_id)
if not transformation:
raise HTTPException(status_code=404, detail="Transformation not found")
# Update only provided fields
if transformation_update.name is not None:
transformation.name = transformation_update.name
if transformation_update.title is not None:
transformation.title = transformation_update.title
if transformation_update.description is not None:
transformation.description = transformation_update.description
if transformation_update.prompt is not None:
transformation.prompt = transformation_update.prompt
if transformation_update.apply_default is not None:
transformation.apply_default = transformation_update.apply_default
if "model_id" in transformation_update.model_fields_set:
# Validate a newly supplied model reference (allow clearing to None).
if transformation_update.model_id:
model = await Model.get(transformation_update.model_id)
if not model:
raise HTTPException(status_code=404, detail="Model not found")
transformation.model_id = transformation_update.model_id
await transformation.save()
return _transformation_response(transformation)
except HTTPException:
raise
except InvalidInputError as e:
raise HTTPException(status_code=400, detail=str(e))
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error updating transformation {transformation_id}: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error updating transformation: {str(e)}"
)
@router.delete("/transformations/{transformation_id}")
async def delete_transformation(transformation_id: str):
"""Delete a transformation."""
try:
transformation = await Transformation.get(transformation_id)
if not transformation:
raise HTTPException(status_code=404, detail="Transformation not found")
await transformation.delete()
return {"message": "Transformation deleted successfully"}
except HTTPException:
raise
except OpenNotebookError:
raise
except Exception as e:
logger.error(f"Error deleting transformation {transformation_id}: {str(e)}")
raise HTTPException(
status_code=500, detail=f"Error deleting transformation: {str(e)}"
)
commands/__init__.py
+21
View File
@@ -0,0 +1,21 @@
"""Surreal-commands integration for Open Notebook"""
from .embedding_commands import (
embed_insight_command,
embed_note_command,
embed_source_command,
rebuild_embeddings_command,
)
from .podcast_commands import generate_podcast_command
from .source_commands import process_source_command
__all__ = [
# Embedding commands
"embed_note_command",
"embed_insight_command",
"embed_source_command",
"rebuild_embeddings_command",
# Other commands
"generate_podcast_command",
"process_source_command",
]
commands/embedding_commands.py
+716
View File
@@ -0,0 +1,716 @@
import time
from typing import (
Any,
Awaitable,
Callable,
Dict,
List,
Literal,
Optional,
Tuple,
)
from loguru import logger
from surreal_commands import CommandInput, CommandOutput, command, submit_command
from open_notebook.ai.models import model_manager
from open_notebook.database.repository import ensure_record_id, repo_insert, repo_query
from open_notebook.domain.notebook import Note, Source, SourceInsight
from open_notebook.exceptions import ConfigurationError
from open_notebook.utils.chunking import ContentType, chunk_text, detect_content_type
from open_notebook.utils.embedding import generate_embedding, generate_embeddings
# NOTE: `stop_on` below can never trigger in practice — each command catches
# ValueError internally and returns success=False instead of raising, so the
# retry layer never sees it. Kept as-is on purpose; to be revisited in a
# dedicated error-handling PR.
EMBED_RETRY_CONFIG = {
"max_attempts": 5,
"wait_strategy": "exponential_jitter",
"wait_min": 1,
"wait_max": 60,
"stop_on": [
ValueError,
ConfigurationError,
], # Don't retry validation/config errors
"retry_log_level": "debug",
}
def get_command_id(input_data: CommandInput) -> str:
"""Extract command_id from input_data's execution context, or return 'unknown'."""
if input_data.execution_context:
return str(input_data.execution_context.command_id)
return "unknown"
async def _embed_record(
input_data: CommandInput,
*,
kind: str,
record_id: str,
embed: Callable[[], Awaitable[Tuple[Dict[str, Any], str]]],
) -> Tuple[Optional[Dict[str, Any]], float, Optional[str]]:
"""
Shared core for the embed_* commands: run the embedding work with the
common logging and error-handling epilogue.
Args:
input_data: The command input (used for command_id logging).
kind: Record kind for log messages ("note", "insight", "source").
record_id: The record being embedded.
embed: Async callable doing the actual load/validate/embed/write work.
Returns (extra_output_fields, success_log_detail).
Returns:
(extra_output_fields, processing_time, error_message)
extra_output_fields is None and error_message is set on permanent
(ValueError) failure. Transient failures re-raise so the retry layer
can handle them.
"""
start_time = time.time()
try:
logger.info(f"Starting embedding for {kind}: {record_id}")
extra_fields, log_detail = await embed()
processing_time = time.time() - start_time
logger.info(
f"Successfully embedded {kind} {record_id}{log_detail} in {processing_time:.2f}s"
)
return extra_fields, processing_time, None
except ValueError as e:
# Permanent failure - don't retry
processing_time = time.time() - start_time
cmd_id = get_command_id(input_data)
logger.error(f"Failed to embed {kind} {record_id} (command: {cmd_id}): {e}")
return None, processing_time, str(e)
except Exception as e:
# Transient failure - will be retried (surreal-commands logs final failure)
cmd_id = get_command_id(input_data)
logger.debug(
f"Transient error embedding {kind} {record_id} (command: {cmd_id}): {e}"
)
raise
async def _embed_markdown_record(
input_data: CommandInput,
*,
label: str,
record_id: str,
loader: Callable[[str], Awaitable[Any]],
) -> Tuple[Dict[str, Any], str]:
"""
Load a record, validate its content, embed it as markdown and UPSERT the
embedding back onto the record. Shared by embed_note and embed_insight.
"""
# 1. Load record
record = await loader(record_id)
if not record:
raise ValueError(f"{label} '{record_id}' not found")
if not record.content or not record.content.strip():
raise ValueError(f"{label} '{record_id}' has no content to embed")
# 2. Generate embedding (auto-chunks + mean pools if needed)
# Notes and insights are typically markdown content
cmd_id = get_command_id(input_data)
embedding = await generate_embedding(
record.content, content_type=ContentType.MARKDOWN, command_id=cmd_id
)
# 3. UPSERT embedding into the record
await repo_query(
"UPDATE $record_id SET embedding = $embedding",
{
"record_id": ensure_record_id(record_id),
"embedding": embedding,
},
)
return {}, ""
class RebuildEmbeddingsInput(CommandInput):
mode: Literal["existing", "all"]
include_sources: bool = True
include_notes: bool = True
include_insights: bool = True
class RebuildEmbeddingsOutput(CommandOutput):
success: bool
total_items: int
jobs_submitted: int # Count of embedding commands submitted
failed_submissions: int # Count of items that failed to submit
sources_submitted: int = 0
notes_submitted: int = 0
insights_submitted: int = 0
processing_time: float
error_message: Optional[str] = None
class CreateInsightInput(CommandInput):
"""Input for creating a source insight with automatic retry on conflicts."""
source_id: str
insight_type: str
content: str
class CreateInsightOutput(CommandOutput):
"""Output from insight creation command."""
success: bool
insight_id: Optional[str] = None
processing_time: float
error_message: Optional[str] = None
class EmbedNoteInput(CommandInput):
"""Input for embedding a single note."""
note_id: str
class EmbedNoteOutput(CommandOutput):
"""Output from note embedding command."""
success: bool
note_id: str
processing_time: float
error_message: Optional[str] = None
class EmbedInsightInput(CommandInput):
"""Input for embedding a single source insight."""
insight_id: str
class EmbedInsightOutput(CommandOutput):
"""Output from insight embedding command."""
success: bool
insight_id: str
processing_time: float
error_message: Optional[str] = None
class EmbedSourceInput(CommandInput):
"""Input for embedding a source (creates multiple chunk embeddings)."""
source_id: str
class EmbedSourceOutput(CommandOutput):
"""Output from source embedding command."""
success: bool
source_id: str
chunks_created: int
processing_time: float
error_message: Optional[str] = None
@command("embed_note", app="open_notebook", retry=EMBED_RETRY_CONFIG)
async def embed_note_command(input_data: EmbedNoteInput) -> EmbedNoteOutput:
"""
Generate and store embedding for a single note.
Uses the unified embedding pipeline with automatic chunking and mean pooling
for notes that exceed the chunk size limit.
Flow:
1. Load Note by ID
2. Generate embedding via generate_embedding() (auto-chunks + mean pools if needed)
3. UPSERT note embedding in database
Retry Strategy:
- Retries up to 5 times for transient failures (network, timeout, etc.)
- Uses exponential-jitter backoff (1-60s)
- Does NOT retry permanent failures (ValueError for validation errors)
"""
async def embed() -> Tuple[Dict[str, Any], str]:
return await _embed_markdown_record(
input_data,
label="Note",
record_id=input_data.note_id,
loader=Note.get,
)
_, processing_time, error_message = await _embed_record(
input_data,
kind="note",
record_id=input_data.note_id,
embed=embed,
)
return EmbedNoteOutput(
success=error_message is None,
note_id=input_data.note_id,
processing_time=processing_time,
error_message=error_message,
)
@command("embed_insight", app="open_notebook", retry=EMBED_RETRY_CONFIG)
async def embed_insight_command(input_data: EmbedInsightInput) -> EmbedInsightOutput:
"""
Generate and store embedding for a single source insight.
Uses the unified embedding pipeline with automatic chunking and mean pooling
for insights that exceed the chunk size limit.
Flow:
1. Load SourceInsight by ID
2. Generate embedding via generate_embedding() (auto-chunks + mean pools if needed)
3. UPSERT insight embedding in database
Retry Strategy:
- Retries up to 5 times for transient failures (network, timeout, etc.)
- Uses exponential-jitter backoff (1-60s)
- Does NOT retry permanent failures (ValueError for validation errors)
"""
async def embed() -> Tuple[Dict[str, Any], str]:
return await _embed_markdown_record(
input_data,
label="Insight",
record_id=input_data.insight_id,
loader=SourceInsight.get,
)
_, processing_time, error_message = await _embed_record(
input_data,
kind="insight",
record_id=input_data.insight_id,
embed=embed,
)
return EmbedInsightOutput(
success=error_message is None,
insight_id=input_data.insight_id,
processing_time=processing_time,
error_message=error_message,
)
@command("embed_source", app="open_notebook", retry=EMBED_RETRY_CONFIG)
async def embed_source_command(input_data: EmbedSourceInput) -> EmbedSourceOutput:
"""
Generate and store embeddings for a source document.
Creates multiple chunk embeddings stored in the source_embedding table.
Uses content-type aware chunking based on file extension or content heuristics.
Flow:
1. Load Source by ID
2. DELETE existing source_embedding records for this source
3. Detect content type from file path or content
4. Chunk text using appropriate splitter
5. Generate embeddings for all chunks in batches
6. Bulk INSERT source_embedding records
Retry Strategy:
- Retries up to 5 times for transient failures (network, timeout, etc.)
- Uses exponential-jitter backoff (1-60s)
- Does NOT retry permanent failures (ValueError for validation errors)
"""
async def embed() -> Tuple[Dict[str, Any], str]:
# 1. Load source
source = await Source.get(input_data.source_id)
if not source:
raise ValueError(f"Source '{input_data.source_id}' not found")
if not source.full_text or not source.full_text.strip():
raise ValueError(f"Source '{input_data.source_id}' has no text to embed")
# 2. DELETE existing embeddings (idempotency)
logger.debug(f"Deleting existing embeddings for source {input_data.source_id}")
await repo_query(
"DELETE source_embedding WHERE source = $source_id",
{"source_id": ensure_record_id(input_data.source_id)},
)
# 3. Detect content type from file path if available
file_path = source.asset.file_path if source.asset else None
content_type = detect_content_type(source.full_text, file_path)
logger.debug(f"Detected content type: {content_type.value}")
# 4. Chunk text using appropriate splitter
chunks = chunk_text(source.full_text, content_type=content_type)
total_chunks = len(chunks)
# Log chunk statistics for debugging
chunk_sizes = [len(c) for c in chunks]
logger.info(
f"Created {total_chunks} chunks for source {input_data.source_id} "
f"(sizes: min={min(chunk_sizes) if chunk_sizes else 0}, "
f"max={max(chunk_sizes) if chunk_sizes else 0}, "
f"avg={sum(chunk_sizes) // len(chunk_sizes) if chunk_sizes else 0} chars)"
)
if total_chunks == 0:
raise ValueError("No chunks created after splitting text")
# 5. Generate embeddings for all chunks in batches
cmd_id = get_command_id(input_data)
logger.debug(f"Generating embeddings for {total_chunks} chunks")
embeddings = await generate_embeddings(chunks, command_id=cmd_id)
# Verify we got embeddings for all chunks
if len(embeddings) != len(chunks):
raise ValueError(
f"Embedding count mismatch: got {len(embeddings)} embeddings "
f"for {len(chunks)} chunks"
)
# 6. Bulk INSERT source_embedding records
records = [
{
"source": ensure_record_id(input_data.source_id),
"order": idx,
"content": chunk,
"embedding": embedding,
}
for idx, (chunk, embedding) in enumerate(zip(chunks, embeddings))
]
logger.debug(f"Inserting {len(records)} source_embedding records")
await repo_insert("source_embedding", records)
return {"chunks_created": total_chunks}, f": {total_chunks} chunks"
extra_fields, processing_time, error_message = await _embed_record(
input_data,
kind="source",
record_id=input_data.source_id,
embed=embed,
)
return EmbedSourceOutput(
success=error_message is None,
source_id=input_data.source_id,
chunks_created=(extra_fields or {}).get("chunks_created", 0),
processing_time=processing_time,
error_message=error_message,
)
@command("create_insight", app="open_notebook", retry=EMBED_RETRY_CONFIG)
async def create_insight_command(
input_data: CreateInsightInput,
) -> CreateInsightOutput:
"""
Create a source insight with automatic retry on transaction conflicts.
This command wraps the CREATE source_insight operation with retry logic
to handle SurrealDB transaction conflicts that occur during batch imports
when multiple parallel transformations try to create insights concurrently.
Flow:
1. CREATE source_insight record in database
2. Submit embed_insight command (fire-and-forget) for async embedding
3. Return the insight_id
Retry Strategy:
- Retries up to 5 times for transient failures (network, timeout, etc.)
- Uses exponential-jitter backoff (1-60s)
- Does NOT retry permanent failures (ValueError for validation errors)
"""
start_time = time.time()
try:
logger.info(
f"Creating insight for source {input_data.source_id}: "
f"type={input_data.insight_type}"
)
# 1. Create insight record in database
result = await repo_query(
"""
CREATE source_insight CONTENT {
"source": $source_id,
"insight_type": $insight_type,
"content": $content
};
""",
{
"source_id": ensure_record_id(input_data.source_id),
"insight_type": input_data.insight_type,
"content": input_data.content,
},
)
if not result or len(result) == 0:
raise ValueError("Failed to create insight - no result returned")
insight_id = str(result[0].get("id", ""))
if not insight_id:
raise ValueError("Failed to create insight - no ID in result")
# 2. Submit embedding command (fire-and-forget)
submit_command(
"open_notebook",
"embed_insight",
{"insight_id": insight_id},
)
logger.debug(f"Submitted embed_insight command for {insight_id}")
processing_time = time.time() - start_time
logger.info(
f"Successfully created insight {insight_id} for source "
f"{input_data.source_id} in {processing_time:.2f}s"
)
return CreateInsightOutput(
success=True,
insight_id=insight_id,
processing_time=processing_time,
)
except ValueError as e:
# Permanent failure - don't retry
processing_time = time.time() - start_time
cmd_id = get_command_id(input_data)
logger.error(
f"Failed to create insight for source {input_data.source_id} "
f"(command: {cmd_id}): {e}"
)
return CreateInsightOutput(
success=False,
processing_time=processing_time,
error_message=str(e),
)
except Exception as e:
# Transient failure - will be retried (surreal-commands logs final failure)
cmd_id = get_command_id(input_data)
logger.debug(
f"Transient error creating insight for source {input_data.source_id} "
f"(command: {cmd_id}): {e}"
)
raise
async def collect_items_for_rebuild(
mode: str,
include_sources: bool,
include_notes: bool,
include_insights: bool,
) -> Dict[str, List[str]]:
"""
Collect items to rebuild based on mode and include flags.
Returns:
Dict with keys: 'sources', 'notes', 'insights' containing lists of item IDs
"""
items: Dict[str, List[str]] = {"sources": [], "notes": [], "insights": []}
if include_sources:
if mode == "existing":
# Query sources with embeddings (via source_embedding table)
result = await repo_query(
"""
RETURN array::distinct(
SELECT VALUE source.id
FROM source_embedding
WHERE embedding != none AND array::len(embedding) > 0
)
"""
)
# RETURN returns the array directly as the result (not nested)
if result:
items["sources"] = [str(item) for item in result]
else:
items["sources"] = []
else: # mode == "all"
# Query all sources with non-empty content
result = await repo_query(
"SELECT id FROM source WHERE full_text != none AND string::trim(full_text) != ''"
)
items["sources"] = [str(item["id"]) for item in result] if result else []
logger.info(f"Collected {len(items['sources'])} sources for rebuild")
if include_notes:
if mode == "existing":
# Query notes with embeddings
result = await repo_query(
"SELECT id FROM note WHERE embedding != none AND array::len(embedding) > 0"
)
else: # mode == "all"
# Query all notes with non-empty content
result = await repo_query(
"SELECT id FROM note WHERE content != none AND string::trim(content) != ''"
)
items["notes"] = [str(item["id"]) for item in result] if result else []
logger.info(f"Collected {len(items['notes'])} notes for rebuild")
if include_insights:
if mode == "existing":
# Query insights with embeddings
result = await repo_query(
"SELECT id FROM source_insight WHERE embedding != none AND array::len(embedding) > 0"
)
else: # mode == "all"
# Query all insights with non-empty content
result = await repo_query(
"SELECT id FROM source_insight WHERE content != none AND string::trim(content) != ''"
)
items["insights"] = [str(item["id"]) for item in result] if result else []
logger.info(f"Collected {len(items['insights'])} insights for rebuild")
return items
def _submit_embedding_jobs(
kind: str, command_name: str, id_field: str, item_ids: List[str]
) -> Tuple[int, int]:
"""
Submit one embedding command per item, logging progress every 50 items.
Returns:
(submitted_count, failed_count)
"""
logger.info(f"\nSubmitting {len(item_ids)} {kind} embedding jobs...")
submitted = 0
failed = 0
for idx, item_id in enumerate(item_ids, 1):
try:
submit_command(
"open_notebook",
command_name,
{id_field: item_id},
)
submitted += 1
if idx % 50 == 0 or idx == len(item_ids):
logger.info(f" Progress: {idx}/{len(item_ids)} {kind} jobs submitted")
except Exception as e:
logger.error(f"Failed to submit {command_name} for {item_id}: {e}")
failed += 1
return submitted, failed
@command("rebuild_embeddings", app="open_notebook", retry=None)
async def rebuild_embeddings_command(
input_data: RebuildEmbeddingsInput,
) -> RebuildEmbeddingsOutput:
"""
Rebuild embeddings for sources, notes, and/or insights.
This command submits individual embedding jobs for each item:
- embed_source for sources
- embed_note for notes
- embed_insight for insights
The command returns after submitting all jobs. Actual embedding
happens asynchronously via the individual commands (which have
their own retry strategies).
Retry Strategy:
- Retries disabled (retry=None) for this coordinator command
- Individual embed_* commands handle their own retries
"""
start_time = time.time()
try:
logger.info("=" * 60)
logger.info(f"Starting embedding rebuild with mode={input_data.mode}")
logger.info(
f"Include: sources={input_data.include_sources}, notes={input_data.include_notes}, insights={input_data.include_insights}"
)
logger.info("=" * 60)
# Check embedding model availability (fail fast)
EMBEDDING_MODEL = await model_manager.get_embedding_model()
if not EMBEDDING_MODEL:
raise ValueError(
"No embedding model configured. Please configure one in the Models section."
)
logger.info(f"Embedding model configured: {EMBEDDING_MODEL}")
# Collect items to process (returns IDs only)
items = await collect_items_for_rebuild(
input_data.mode,
input_data.include_sources,
input_data.include_notes,
input_data.include_insights,
)
total_items = (
len(items["sources"]) + len(items["notes"]) + len(items["insights"])
)
logger.info(f"Total items to rebuild: {total_items}")
if total_items == 0:
logger.warning("No items found to rebuild")
return RebuildEmbeddingsOutput(
success=True,
total_items=0,
jobs_submitted=0,
failed_submissions=0,
processing_time=time.time() - start_time,
)
# Submit one embedding command per item, per kind
sources_submitted, sources_failed = _submit_embedding_jobs(
"source", "embed_source", "source_id", items["sources"]
)
notes_submitted, notes_failed = _submit_embedding_jobs(
"note", "embed_note", "note_id", items["notes"]
)
insights_submitted, insights_failed = _submit_embedding_jobs(
"insight", "embed_insight", "insight_id", items["insights"]
)
failed_submissions = sources_failed + notes_failed + insights_failed
processing_time = time.time() - start_time
jobs_submitted = sources_submitted + notes_submitted + insights_submitted
logger.info("=" * 60)
logger.info("REBUILD JOBS SUBMITTED")
logger.info(f" Total jobs submitted: {jobs_submitted}/{total_items}")
logger.info(f" Sources: {sources_submitted}")
logger.info(f" Notes: {notes_submitted}")
logger.info(f" Insights: {insights_submitted}")
logger.info(f" Failed submissions: {failed_submissions}")
logger.info(f" Submission time: {processing_time:.2f}s")
logger.info(" Note: Actual embedding happens asynchronously")
logger.info("=" * 60)
return RebuildEmbeddingsOutput(
success=True,
total_items=total_items,
jobs_submitted=jobs_submitted,
failed_submissions=failed_submissions,
sources_submitted=sources_submitted,
notes_submitted=notes_submitted,
insights_submitted=insights_submitted,
processing_time=processing_time,
)
except Exception as e:
processing_time = time.time() - start_time
logger.error(f"Rebuild embeddings failed: {e}")
logger.exception(e)
return RebuildEmbeddingsOutput(
success=False,
total_items=0,
jobs_submitted=0,
failed_submissions=0,
processing_time=processing_time,
error_message=str(e),
)
commands/podcast_commands.py
+375
View File
@@ -0,0 +1,375 @@
import time
import uuid
from pathlib import Path
from typing import Optional
from loguru import logger
from surreal_commands import CommandInput, CommandOutput, command
from open_notebook.config import PODCASTS_FOLDER
from open_notebook.database.repository import ensure_record_id, repo_query
from open_notebook.podcasts.audio_paths import to_relative_audio_path
from open_notebook.podcasts.models import (
EpisodeProfile,
PodcastEpisode,
SpeakerProfile,
_resolve_model_config,
)
from open_notebook.utils.model_utils import full_model_dump
try:
from podcast_creator import configure, create_podcast
except ImportError as e:
logger.error(f"Failed to import podcast_creator: {e}")
raise ValueError("podcast_creator library not available")
def build_episode_output_dir(podcasts_folder: str = PODCASTS_FOLDER) -> tuple[str, Path]:
"""Build a filesystem-safe output directory path for a podcast episode.
Uses a UUID as the directory name so the path is safe regardless of
what the user typed as episode name (spaces, special chars, etc.).
Builds under PODCASTS_FOLDER — the same root to_relative_audio_path()
validates against at write time (#1030) — so the two can't drift apart.
Returns:
A tuple of (episode_dir_name, output_dir_path).
"""
episode_dir_name = str(uuid.uuid4())
output_dir = Path(podcasts_folder) / "episodes" / episode_dir_name
return episode_dir_name, output_dir
class PodcastGenerationInput(CommandInput):
episode_profile: str
# Speaker profile record ID or name (the API boundary resolves the
# user-facing name to a record ID before submitting; both are accepted
# here for robustness).
speaker_profile: Optional[str] = None
episode_name: str
content: str
briefing_suffix: Optional[str] = None
class PodcastGenerationOutput(CommandOutput):
success: bool
episode_id: Optional[str] = None
audio_file_path: Optional[str] = None
transcript: Optional[dict] = None
outline: Optional[dict] = None
processing_time: float
error_message: Optional[str] = None
@command("generate_podcast", app="open_notebook", retry={"max_attempts": 1})
async def generate_podcast_command(
input_data: PodcastGenerationInput,
) -> PodcastGenerationOutput:
"""
Real podcast generation using podcast-creator library with Episode Profiles
"""
start_time = time.time()
try:
logger.info(
f"Starting podcast generation for episode: {input_data.episode_name}"
)
logger.info(f"Using episode profile: {input_data.episode_profile}")
# 1. Load Episode and Speaker profiles from SurrealDB
episode_profile = await EpisodeProfile.get_by_name(input_data.episode_profile)
if not episode_profile:
raise ValueError(
f"Episode profile '{input_data.episode_profile}' not found"
)
# Honor the explicitly requested speaker profile when provided,
# falling back to the episode profile's configured speaker
# (a speaker_profile record ID since migration 20, None when the
# referenced profile no longer exists).
speaker_ref = input_data.speaker_profile or episode_profile.speaker_config
if not speaker_ref:
raise ValueError(
f"Episode profile '{episode_profile.name}' has no speaker "
"profile configured. Please update the profile to select a "
"speaker profile."
)
speaker_profile = await SpeakerProfile.resolve(speaker_ref)
if not speaker_profile:
if input_data.speaker_profile:
raise ValueError(f"Speaker profile '{speaker_ref}' not found")
raise ValueError(
f"Episode profile '{episode_profile.name}' references a "
"speaker profile that no longer exists. Please update the "
"profile to select a speaker profile."
)
logger.info(f"Loaded episode profile: {episode_profile.name}")
logger.info(f"Loaded speaker profile: {speaker_profile.name}")
# 2. Validate that model registry fields are populated
if not episode_profile.outline_llm:
raise ValueError(
f"Episode profile '{episode_profile.name}' has no outline model configured. "
"Please update the profile to select an outline model."
)
if not episode_profile.transcript_llm:
raise ValueError(
f"Episode profile '{episode_profile.name}' has no transcript model configured. "
"Please update the profile to select a transcript model."
)
if not speaker_profile.voice_model:
raise ValueError(
f"Speaker profile '{speaker_profile.name}' has no voice model configured. "
"Please update the profile to select a voice model."
)
# 3. Resolve model configs with credentials
outline_provider, outline_model_name, outline_config = (
await episode_profile.resolve_outline_config()
)
transcript_provider, transcript_model_name, transcript_config = (
await episode_profile.resolve_transcript_config()
)
tts_provider, tts_model_name, tts_config = (
await speaker_profile.resolve_tts_config()
)
logger.info(
f"Resolved models - outline: {outline_provider}/{outline_model_name}, "
f"transcript: {transcript_provider}/{transcript_model_name}, "
f"tts: {tts_provider}/{tts_model_name}"
)
# 4. Load all profiles and configure podcast-creator
episode_profiles = await repo_query("SELECT * FROM episode_profile")
speaker_profiles = await repo_query("SELECT * FROM speaker_profile")
# Transform the surrealdb array into a dictionary for podcast-creator
episode_profiles_dict = {
profile["name"]: profile for profile in episode_profiles
}
speaker_profiles_dict = {
profile["name"]: profile for profile in speaker_profiles
}
# Map speaker_profile record ID -> name so podcast-creator keeps
# receiving speaker names (its EpisodeProfile.speaker_config is a
# required non-empty name string, cross-referenced against the
# speakers config keyed by name).
speaker_name_by_id = {
str(profile["id"]): profile["name"] for profile in speaker_profiles
}
# 5. Inject resolved model configs into profile dicts
# Resolve ALL episode profiles (podcast-creator validates all).
# Remove profiles that fail resolution to prevent validation errors.
for ep_name in list(episode_profiles_dict.keys()):
ep_dict = episode_profiles_dict[ep_name]
# Since migration 20, speaker_config stores a record ID (and is
# None when the referenced speaker profile no longer exists).
# Rewrite it back to the speaker name for podcast-creator; drop
# profiles whose reference doesn't resolve so a single orphaned
# profile can't fail validation for the whole config. The profile
# being generated always resolves: its speaker was validated above.
speaker_ref = ep_dict.get("speaker_config")
speaker_name = (
speaker_name_by_id.get(str(speaker_ref)) if speaker_ref else None
)
if not speaker_name and ep_name == episode_profile.name:
speaker_name = speaker_profile.name
if not speaker_name:
logger.warning(
f"Episode profile '{ep_name}' references a speaker profile "
f"that no longer exists ({speaker_ref!r}), removing from "
"config to prevent validation errors"
)
del episode_profiles_dict[ep_name]
continue
ep_dict["speaker_config"] = speaker_name
try:
if ep_dict.get("outline_llm"):
prov, model, conf = await _resolve_model_config(
str(ep_dict["outline_llm"]),
max_tokens=ep_dict.get("max_tokens"),
)
ep_dict["outline_provider"] = prov
ep_dict["outline_model"] = model
ep_dict["outline_config"] = conf
if ep_dict.get("transcript_llm"):
prov, model, conf = await _resolve_model_config(
str(ep_dict["transcript_llm"]),
max_tokens=ep_dict.get("max_tokens"),
)
ep_dict["transcript_provider"] = prov
ep_dict["transcript_model"] = model
ep_dict["transcript_config"] = conf
except Exception as e:
logger.warning(
f"Failed to resolve models for episode profile '{ep_name}', "
f"removing from config to prevent validation errors: {e}"
)
del episode_profiles_dict[ep_name]
# Resolve TTS for ALL speaker profiles (podcast-creator validates all).
# Remove profiles that fail resolution to prevent validation errors.
for sp_name in list(speaker_profiles_dict.keys()):
sp_dict = speaker_profiles_dict[sp_name]
if sp_dict.get("voice_model"):
try:
prov, model, conf = await _resolve_model_config(
str(sp_dict["voice_model"])
)
sp_dict["tts_provider"] = prov
sp_dict["tts_model"] = model
sp_dict["tts_config"] = conf
except Exception as e:
logger.warning(
f"Failed to resolve TTS for speaker profile '{sp_name}', "
f"removing from config to prevent validation errors: {e}"
)
del speaker_profiles_dict[sp_name]
continue
# Per-speaker TTS overrides
for speaker in sp_dict.get("speakers", []):
if speaker.get("voice_model"):
try:
prov, model, conf = await _resolve_model_config(
str(speaker["voice_model"])
)
speaker["tts_provider"] = prov
speaker["tts_model"] = model
speaker["tts_config"] = conf
except Exception as e:
logger.warning(
f"Failed to resolve per-speaker TTS for '{speaker.get('name')}': {e}"
)
# 6. Generate briefing
briefing = episode_profile.default_briefing
if input_data.briefing_suffix:
briefing += f"\n\nAdditional instructions: {input_data.briefing_suffix}"
# Create the record for the episode and associate with the ongoing command
episode = PodcastEpisode(
name=input_data.episode_name,
episode_profile=full_model_dump(episode_profile.model_dump()),
speaker_profile=full_model_dump(speaker_profile.model_dump()),
command=ensure_record_id(input_data.execution_context.command_id)
if input_data.execution_context
else None,
briefing=briefing,
content=input_data.content,
audio_file=None,
transcript=None,
outline=None,
)
await episode.save()
# SECURITY NOTE for future work: podcast_creator also supports
# configure("templates", {...}), which compiles the given string
# directly as Jinja2 template *source* (Prompter(template_text=...)
# in podcast_creator/config.py) - the exact SSTI shape already fixed
# in open_notebook/graphs/transformation.py (GHSA-f35w-wx37-26q7).
# We don't call it today (confirmed: no code path here sets the
# "templates" key, so podcast generation always uses the file-based
# prompts/podcast/*.jinja templates in this repo). If a "custom
# podcast template" feature is ever added, do NOT wire user/profile
# text into configure("templates", ...) - render it through a
# fixed, developer-authored template with the user text passed in
# as a plain variable instead, matching transformation.py's fix.
configure("speakers_config", {"profiles": speaker_profiles_dict})
configure("episode_config", {"profiles": episode_profiles_dict})
logger.info("Configured podcast-creator with episode and speaker profiles")
logger.info(f"Generated briefing (length: {len(briefing)} chars)")
# 7. Create output directory using UUID for filesystem-safe paths
episode_dir_name, output_dir = build_episode_output_dir()
output_dir.mkdir(parents=True, exist_ok=True)
logger.info(f"Created output directory: {output_dir}")
# 8. Generate podcast using podcast-creator
logger.info("Starting podcast generation with podcast-creator...")
result = await create_podcast(
content=input_data.content,
briefing=briefing,
episode_name=episode_dir_name,
output_dir=str(output_dir),
speaker_config=speaker_profile.name,
episode_profile=episode_profile.name,
)
# podcast-creator reports audio-combination failures IN-BAND: on
# ffmpeg/clip errors combine_audio_files() returns an "ERROR: ..."
# string in final_output_file_path instead of a path. Detect it
# before path conversion so the real error surfaces (below, after
# the transcript/outline are persisted) instead of a misleading
# "outside the podcasts folder" ValueError.
raw_audio_path = result.get("final_output_file_path") if result else None
audio_error: Optional[str] = None
if raw_audio_path is not None and str(raw_audio_path).startswith("ERROR:"):
audio_error = str(raw_audio_path)
raw_audio_path = None
# Store the audio path RELATIVE to PODCASTS_FOLDER (#1030). The
# validation inside to_relative_audio_path guarantees the DB never
# holds an absolute or root-escaping value; a violation raises
# ValueError, which marks the job permanently failed (no retry).
audio_file_rel = (
to_relative_audio_path(raw_audio_path) if raw_audio_path else None
)
episode.audio_file = audio_file_rel
episode.transcript = {
"transcript": full_model_dump(result["transcript"]) if result else None
}
episode.outline = full_model_dump(result["outline"]) if result else None
await episode.save()
if audio_error:
# Transcript/outline are saved above; fail the job with the real
# audio-combination error instead of reporting a silent success
# for an episode with no playable audio.
raise RuntimeError(f"Podcast audio generation failed: {audio_error}")
processing_time = time.time() - start_time
logger.info(
f"Successfully generated podcast episode: {episode.id} in {processing_time:.2f}s"
)
return PodcastGenerationOutput(
success=True,
episode_id=str(episode.id),
audio_file_path=audio_file_rel,
transcript={"transcript": full_model_dump(result["transcript"])}
if result.get("transcript")
else None,
outline=full_model_dump(result["outline"])
if result.get("outline")
else None,
processing_time=processing_time,
)
except ValueError:
raise
except Exception as e:
logger.error(f"Podcast generation failed: {e}")
logger.exception(e)
error_msg = str(e)
if "Invalid json output" in error_msg or "Expecting value" in error_msg:
error_msg += (
"\n\nNOTE: This error commonly occurs with GPT-5 models that use extended thinking. "
"The model may be putting all output inside <think> tags, leaving nothing to parse. "
"Try using gpt-4o, gpt-4o-mini, or gpt-4-turbo instead in your episode profile."
)
raise RuntimeError(error_msg) from e
commands/source_commands.py
+261
View File
@@ -0,0 +1,261 @@
import time
from typing import Any, Dict, List, Optional
from loguru import logger
from surreal_commands import CommandInput, CommandOutput, command
from open_notebook.database.repository import ensure_record_id
from open_notebook.domain.notebook import Source
from open_notebook.domain.transformation import Transformation
from open_notebook.exceptions import ConfigurationError
try:
from open_notebook.graphs.source import source_graph
from open_notebook.graphs.transformation import graph as transform_graph
except ImportError as e:
logger.error(f"Failed to import graphs: {e}")
raise ValueError("graphs not available")
class SourceProcessingInput(CommandInput):
source_id: str
content_state: Dict[str, Any]
notebook_ids: List[str]
transformations: List[str]
embed: bool
class SourceProcessingOutput(CommandOutput):
success: bool
source_id: str
embedded_chunks: int = 0
insights_created: int = 0
processing_time: float
error_message: Optional[str] = None
@command(
"process_source",
app="open_notebook",
retry={
"max_attempts": 15, # Handle deep queues (workaround for SurrealDB v2 transaction conflicts)
"wait_strategy": "exponential_jitter",
"wait_min": 1,
"wait_max": 120, # Allow queue to drain
"stop_on": [ValueError, ConfigurationError], # Don't retry validation/config errors
"retry_log_level": "debug", # Avoid log noise during transaction conflicts
},
)
async def process_source_command(
input_data: SourceProcessingInput,
) -> SourceProcessingOutput:
"""
Process source content using the source_graph workflow
"""
start_time = time.time()
try:
logger.info(f"Starting source processing for source: {input_data.source_id}")
logger.info(f"Notebook IDs: {input_data.notebook_ids}")
logger.info(f"Transformations: {input_data.transformations}")
logger.info(f"Embed: {input_data.embed}")
# 1. Load transformation objects from IDs
transformations = []
for trans_id in input_data.transformations:
logger.info(f"Loading transformation: {trans_id}")
transformation = await Transformation.get(trans_id)
if not transformation:
raise ValueError(f"Transformation '{trans_id}' not found")
transformations.append(transformation)
logger.info(f"Loaded {len(transformations)} transformations")
# 2. Get existing source record to update its command field
source = await Source.get(input_data.source_id)
if not source:
raise ValueError(f"Source '{input_data.source_id}' not found")
# Update source with command reference
source.command = (
ensure_record_id(input_data.execution_context.command_id)
if input_data.execution_context
else None
)
await source.save()
logger.info(f"Updated source {source.id} with command reference")
# 3. Process source with all notebooks
logger.info(f"Processing source with {len(input_data.notebook_ids)} notebooks")
# Execute source_graph with all notebooks.
# LangGraph accepts a partial state dict at runtime, but its typed
# overloads require the full state type (langgraph typing limitation).
result = await source_graph.ainvoke( # type: ignore[call-overload]
{
"content_state": input_data.content_state,
"notebook_ids": input_data.notebook_ids, # Use notebook_ids (plural) as expected by SourceState
"apply_transformations": transformations,
"embed": input_data.embed,
"source_id": input_data.source_id, # Add the source_id to the state
}
)
processed_source = result["source"]
# 4. Gather processing results (notebook associations handled by source_graph)
# Note: embedding is fire-and-forget (async job), so we can't query the
# count here — it hasn't completed yet. The embed_source_command logs
# the actual count when it finishes.
insights_list = await processed_source.get_insights()
insights_created = len(insights_list)
processing_time = time.time() - start_time
embed_status = "submitted" if input_data.embed else "skipped"
logger.info(
f"Successfully processed source: {processed_source.id} in {processing_time:.2f}s"
)
logger.info(
f"Created {insights_created} insights, embedding {embed_status}"
)
return SourceProcessingOutput(
success=True,
source_id=str(processed_source.id),
embedded_chunks=0,
insights_created=insights_created,
processing_time=processing_time,
)
except ValueError as e:
# Validation errors are permanent failures. Re-raise so surreal-commands
# marks the job as `failed` (stop_on=[ValueError] already prevents
# pointless retries). Returning a success=False result instead marks the
# job `completed` (is_success() checks job status, not the payload),
# which hid extraction failures and left the source without a retryable
# `failed` status in the UI.
logger.error(f"Source processing failed (permanent): {e}")
raise
except Exception as e:
# Transient failure - will be retried (surreal-commands logs final failure)
logger.debug(
f"Transient error processing source {input_data.source_id}: {e}"
)
raise
# =============================================================================
# RUN TRANSFORMATION COMMAND
# =============================================================================
class RunTransformationInput(CommandInput):
"""Input for running a transformation on an existing source."""
source_id: str
transformation_id: str
class RunTransformationOutput(CommandOutput):
"""Output from transformation command."""
success: bool
source_id: str
transformation_id: str
processing_time: float
error_message: Optional[str] = None
@command(
"run_transformation",
app="open_notebook",
retry={
"max_attempts": 5,
"wait_strategy": "exponential_jitter",
"wait_min": 1,
"wait_max": 60,
"stop_on": [ValueError, ConfigurationError], # Don't retry validation/config errors
"retry_log_level": "debug",
},
)
async def run_transformation_command(
input_data: RunTransformationInput,
) -> RunTransformationOutput:
"""
Run a transformation on an existing source to generate an insight.
This command runs the transformation graph which:
1. Loads the source and transformation
2. Calls the LLM to generate insight content
3. Creates the insight via create_insight command (fire-and-forget)
Use this command for UI-triggered insight generation to avoid blocking
the HTTP request while the LLM processes.
Retry Strategy:
- Retries up to 5 times for transient failures (network, timeout, etc.)
- Uses exponential-jitter backoff (1-60s)
- Does NOT retry permanent failures (ValueError for validation errors)
"""
start_time = time.time()
try:
logger.info(
f"Running transformation {input_data.transformation_id} "
f"on source {input_data.source_id}"
)
# Load source
source = await Source.get(input_data.source_id)
if not source:
raise ValueError(f"Source '{input_data.source_id}' not found")
# Load transformation
transformation = await Transformation.get(input_data.transformation_id)
if not transformation:
raise ValueError(
f"Transformation '{input_data.transformation_id}' not found"
)
# Run transformation graph (includes LLM call + insight creation).
# LangGraph accepts a partial state dict at runtime, but its typed
# overloads require the full state type (langgraph typing limitation).
await transform_graph.ainvoke( # type: ignore[call-overload]
input=dict(source=source, transformation=transformation)
)
processing_time = time.time() - start_time
logger.info(
f"Successfully ran transformation {input_data.transformation_id} "
f"on source {input_data.source_id} in {processing_time:.2f}s"
)
return RunTransformationOutput(
success=True,
source_id=input_data.source_id,
transformation_id=input_data.transformation_id,
processing_time=processing_time,
)
except ValueError as e:
# Validation errors are permanent failures - don't retry
processing_time = time.time() - start_time
logger.error(
f"Failed to run transformation {input_data.transformation_id} "
f"on source {input_data.source_id}: {e}"
)
return RunTransformationOutput(
success=False,
source_id=input_data.source_id,
transformation_id=input_data.transformation_id,
processing_time=processing_time,
error_message=str(e),
)
except Exception as e:
# Transient failure - will be retried (surreal-commands logs final failure)
logger.debug(
f"Transient error running transformation {input_data.transformation_id} "
f"on source {input_data.source_id}: {e}"
)
raise
cubic.yaml
+102
View File
@@ -0,0 +1,102 @@
# yaml-language-server: $schema=https://cubic.dev/schema/cubic-repository-config.schema.json
#
# cubic AI review settings (settings as code).
# Docs: https://docs.cubic.dev/configure/cubic-yaml
#
# Platform mechanics (verified against docs.cubic.dev, Jul 2026):
# - Max 5 active custom agents (custom_rules) per repo. Each agent's text plus any
# file-backed content (file_paths) shares a 10,000-character limit that is truncated
# SILENTLY — only link short, stable files.
# - Config precedence: repo cubic.yaml > org config (`cubic-config` repo) > dashboard UI
# > defaults. Schema: https://cubic.dev/schema/cubic-repository-config.schema.json
# - Replying to a cubic review comment teaches it permanently (team-scoped); manage
# learnings under AI review settings → Memory & Learning.
# - cubic auto-detects context files (AGENTS.md, README, .github/) — those inform the
# review; custom agents are what actually enforce rules.
version: 1
reviews:
# Skip files with no reviewable logic to preserve the monthly reviewed-line quota:
# CHANGELOG entries are prose (and every PR + rebase re-reviews them); lockfiles
# are generated.
ignore:
files:
- 'CHANGELOG.md'
- 'uv.lock'
- 'frontend/package-lock.json'
custom_instructions: |
This project follows an issue-first workflow for anything non-trivial: features and
architecture changes must reference an approved, assigned issue and stay scoped to it.
Small obvious fixes (typos, docs, tiny bugs, i18n completions) are welcome without an
issue. A sizeable PR without one should be converted to draft with an issue opened —
don't flag small fixes for lacking an issue. Unrelated refactors belong in separate PRs.
Test evidence must be real (actual output), especially for bug fixes, which should
include a regression test. Many PRs are agent-generated; hold them to the same bar.
Normative rules live in AGENTS.md (root, open_notebook/, frontend/); product direction
lives in VISION.md; past decisions in docs/7-DEVELOPMENT/decisions/.
custom_rules:
- name: Vision & principles alignment
description: |
Check the PR against the product identity and current posture defined in the
linked VISION.md. Flag:
- Features that gratuitously preclude multi-user (hard-coded single-tenancy in
schema, auth, or data scoping) — see PDR-001.
- Capabilities that only work on one AI provider without a PDR justifying the
exclusivity — the core is provider-agnostic by default (PDR-002).
- Features that conflict with the "What Open Notebook IS NOT" list.
- Structural/architectural decisions introduced without a decision record in
docs/7-DEVELOPMENT/decisions/ (half a page, same PR).
file_paths:
- VISION.md
- name: Known caveats
description: |
Enforce this project's recurring mechanical pitfalls (full rules in
open_notebook/AGENTS.md and frontend/AGENTS.md):
- i18n: every user-visible frontend string goes through t('section.key') and the
key must exist in ALL 7 locales (en-US, pt-BR, zh-CN, zh-TW, ja-JP, ru-RU, bn-IN).
- Providers: adding/changing a provider = edit the registry
(open_notebook/ai/provider_registry.py PROVIDERS) plus the SupportedProvider
Literal in api/models.py; everything else (TEST_MODELS, PROVIDER_ENV_CONFIG,
the frontend via GET /api/providers) is derived from the registry.
- Migrations: a new open_notebook/database/migrations/N.surrealql file must also be
registered in AsyncMigrationManager (async_migrate.py) — no auto-discovery.
- Graphs: LLM calls in LangGraph nodes must use provision_langchain_model() and wrap
errors with classify_error(); model output goes through clean_thinking_content().
- Domain: Source.save() does not auto-embed — vectorize() must be called explicitly;
RecordModel singletons need clear_instance() in tests.
- Async jobs (podcasts, embeddings, source processing) run on the surreal-commands
worker; features depending on them must not assume inline execution. Permanent
failures raise ValueError (no retry); podcasts use max_attempts=1 by design.
- Frontend: FormData nested fields must be JSON.stringify-ed; don't re-add
Content-Type; check hasHydrated before rendering persisted Zustand state.
- name: Security & testability
description: |
Security:
- API endpoints must never return API key or secret values — metadata only.
- Every user-supplied URL must pass validate_url() (SSRF protection; private IPs are
intentionally allowed for self-hosted services).
- No secrets committed, logged, or echoed in error messages; credential fields use
SecretStr and Fernet encryption (OPEN_NOTEBOOK_ENCRYPTION_KEY).
- Raise typed exceptions from open_notebook.exceptions instead of bare HTTPException
or generic 500s.
Testability:
- Business logic belongs in services/domain, not in routers or React components,
so it can be tested directly.
- Critical paths (auth, credentials, encryption, migrations, data deletion) need
test coverage in the same PR.
- Bug fixes should include a test that reproduces the bug.
auto_ultrareview: custom
auto_ultrareview_custom_prompt: |
Run an ultrareview when the PR touches authentication, credential storage, encryption,
database migrations, or data deletion/cascade logic.
auto_ultrareview_file_patterns:
- 'open_notebook/database/migrations/**'
- '**/credential*'
- '**/auth*'
- '**/encryption*'
dev-init.sh
Executable
+43
View File
@@ -0,0 +1,43 @@
#!/bin/bash
# Development environment startup for Open Notebook
# Assumes SurrealDB is already running externally (per .env config)
set -e
echo "=== Open Notebook Dev Startup ==="
# Check SurrealDB connectivity
SURREAL_PORT=${SURREAL_PORT:-8018}
echo "Checking SurrealDB on port $SURREAL_PORT..."
if ! nc -z localhost "$SURREAL_PORT" 2>/dev/null; then
echo "❌ SurrealDB not reachable on port $SURREAL_PORT. Please start it first."
exit 1
fi
echo "✅ SurrealDB is running"
# Install dependencies if needed
echo "Syncing Python dependencies..."
uv sync
echo "Syncing frontend dependencies..."
cd frontend && npm install && cd ..
# Start API backend in background
echo "Starting API backend (port 5055)..."
uv run --env-file .env run_api.py &
sleep 3
# Start background worker in background
echo "Starting background worker..."
uv run --env-file .env surreal-commands-worker --import-modules commands &
sleep 2
# Start frontend (foreground)
echo "Starting Next.js frontend (port 3000)..."
echo ""
echo "✅ All services starting!"
echo " Frontend: http://localhost:3000"
echo " API: http://localhost:5055"
echo " API Docs: http://localhost:5055/docs"
echo ""
cd frontend && npm run dev
docker-compose.override.yml.example
+23
View File
@@ -0,0 +1,23 @@
# Local Docker Compose override — copy to `docker-compose.override.yml`
# (which `docker compose up` merges automatically) to apply host-specific
# tweaks without touching the tracked docker-compose.yml.
#
# cp docker-compose.override.yml.example docker-compose.override.yml
#
# The example below re-exposes SurrealDB's port on all interfaces. The
# shipped default binds it to 127.0.0.1 only, because the database starts
# with root:root credentials and exposing it on 0.0.0.0 lets anyone who can
# reach the host connect as root. Only re-expose it if you actually need to
# reach the database from another machine (e.g. Surrealist on your laptop),
# and put it behind a firewall or an SSH tunnel and set SURREAL_USER /
# SURREAL_PASSWORD to real credentials first.
services:
surrealdb:
# `!override` replaces the base file's port list instead of merging with
# it — without it, compose keeps both entries and the container fails to
# start with "port is already allocated". Requires Docker Compose
# v2.24.4+; on older clients, edit the ports entry in docker-compose.yml
# directly instead.
ports: !override
- "8000:8000"
docker-compose.yml
+49
View File
@@ -0,0 +1,49 @@
services:
surrealdb:
image: surrealdb/surrealdb:v2
# Credentials default to root:root for a zero-config local setup. Before
# exposing this instance to a network, set SURREAL_USER / SURREAL_PASSWORD
# in a .env file (see .env.example) — they are applied here and to the
# open_notebook service below, so the two always stay in sync.
# List (exec) form so each interpolated value stays a single argument —
# a password containing spaces would otherwise be split into several.
command: ["start", "--log", "info", "--user", "${SURREAL_USER:-root}", "--pass", "${SURREAL_PASSWORD:-root}", "rocksdb:/mydata/mydatabase.db"]
user: root # Required for bind mounts on Linux
ports:
# Bound to localhost only: the open_notebook service reaches this over
# the internal compose network regardless, so the host port is purely
# for local debugging (e.g. Surrealist, `surreal sql`). Exposing this
# on 0.0.0.0 would let anyone who can reach the host connect with the
# default root:root credentials.
- "127.0.0.1:8000:8000"
volumes:
- ./surreal_data:/mydata
environment:
- SURREAL_EXPERIMENTAL_GRAPHQL=true
restart: always
pull_policy: always
open_notebook:
image: lfnovo/open_notebook:v1-latest
ports:
- "8502:8502" # Web UI
- "5055:5055" # REST API
environment:
# REQUIRED: Change this to your own secret string
# This encrypts your API keys in the database
- OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
# Database connection. SURREAL_USER / SURREAL_PASSWORD default to root:root
# for local use; override them in a .env file before exposing the instance
# (the same values configure the surrealdb service above).
- SURREAL_URL=ws://surrealdb:8000/rpc
- SURREAL_USER=${SURREAL_USER:-root}
- SURREAL_PASSWORD=${SURREAL_PASSWORD:-root}
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=open_notebook
volumes:
- ./notebook_data:/app/data
depends_on:
- surrealdb
restart: always
pull_policy: always
docs/0-START-HERE/index.md
+65
View File
@@ -0,0 +1,65 @@
# Open Notebook - Start Here
**Open Notebook** is a privacy-focused AI research assistant. Upload documents, chat with AI, generate notes, and create podcasts—all with complete control over your data.
## Choose Your Path
### 🚀 I want to use OpenAI (Fastest)
**5 minutes to running.** GPT, simple setup, powerful results.
→ [OpenAI Quick Start](quick-start-openai.md)
---
### ☁️ I want to use other cloud AI (Anthropic, Google, OpenRouter, etc.)
**5 minutes to running.** Choose from 17+ AI providers.
→ [Cloud Providers Quick Start](quick-start-cloud.md)
---
### 🏠 I want to run locally (Ollama or LMStudio, completely private)
**5 minutes to running.** Keep everything private, on your machine. No costs.
→ [Local Quick Start](quick-start-local.md)
**Already have Ollama installed?** → [External Ollama Guide](quick-start-external-ollama.md)
---
## What Can You Do?
- 📄 **Upload Content**: PDFs, web links, audio, video, text
- 🤖 **Chat with AI**: Ask questions about your documents with citations
- 📝 **Generate Notes**: AI creates summaries and insights
- 🎙️ **Create Podcasts**: Turn research into professional audio content
- 🔍 **Search**: Full-text and semantic search across all content
- ⚙️ **Transform**: Extract insights, analyze themes, create summaries
## Why Open Notebook?
| Feature | Open Notebook | Notebook LM |
|---------|---|---|
| **Privacy** | Self-hosted, your control | Cloud, Google's servers |
| **AI Choice** | 17+ providers | Google's models only |
| **Podcast Speakers** | 1-4 customizable | 2 only |
| **Cost** | Completely free | Free (but your data) |
| **Offline** | Yes | No |
## Prerequisites
- **Docker**: All paths use Docker (free)
- **AI Provider**: Either a cloud API key OR use free local models (Ollama)
---
## Next Steps
1. Pick your path above ⬆️
2. Follow the 5-minute quick start
3. Create your first notebook
4. Start uploading documents!
---
**Need Help?** Join our [Discord community](https://discord.gg/37XJPXfz2w) or see [Full Documentation](../index.md).
docs/0-START-HERE/quick-start-cloud.md
+214
View File
@@ -0,0 +1,214 @@
# Quick Start - Cloud AI Providers (5 minutes)
Get Open Notebook running with **Anthropic, Google, Groq, or other cloud providers**. Same simplicity as OpenAI, with more choices.
## Prerequisites
1. **Docker Desktop** installed
- [Download here](https://www.docker.com/products/docker-desktop/)
- Already have it? Skip to step 2
2. **API Key** from your chosen provider:
- **OpenRouter** (100+ models, one key): https://openrouter.ai/keys
- **Anthropic (Claude)**: https://console.anthropic.com/
- **Google (Gemini)**: https://aistudio.google.com/
- **Groq** (fast, free tier): https://console.groq.com/
- **Mistral**: https://console.mistral.ai/
- **DeepSeek**: https://platform.deepseek.com/
- **xAI (Grok)**: https://console.x.ai/
## Step 1: Create Configuration (1 min)
Create a new folder `open-notebook` and add this file:
**docker-compose.yml**:
```yaml
services:
surrealdb:
image: surrealdb/surrealdb:v2
command: start --user root --pass password rocksdb:/mydata/mydatabase.db
ports:
# Localhost only — the database uses default credentials, so never
# publish this port on 0.0.0.0
- "127.0.0.1:8000:8000"
volumes:
- ./surreal_data:/mydata
# Removed the healthcheck because the v2 image is too minimal to run wget/curl
restart: always
open_notebook:
image: lfnovo/open_notebook:v1-latest
pull_policy: always
ports:
- "8502:8502" # Web UI
- "5055:5055" # API
environment:
- OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
- SURREAL_URL=ws://surrealdb:8000/rpc
- SURREAL_USER=root
- SURREAL_PASSWORD=password
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=open_notebook
volumes:
- ./notebook_data:/app/data
depends_on:
- surrealdb
restart: always
```
**Edit the file:**
- Replace `change-me-to-a-secret-string` with your own secret (any string works)
---
## Step 2: Start Services (1 min)
Open terminal in your `open-notebook` folder:
```bash
docker compose up -d
```
Wait 15-20 seconds for services to start.
---
## Step 3: Access Open Notebook (instant)
Open your browser:
```
http://localhost:8502
```
You should see the Open Notebook interface!
---
## Step 4: Configure Your AI Provider (1 min)
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select your provider (e.g., Anthropic, Google, Groq, OpenRouter)
4. Give it a name, paste your API key
5. Click **Save**
6. Click **Test Connection** — should show success
7. Click **Discover Models** → **Register Models**
Your provider's models are now available!
> **Multiple providers**: You can add credentials for as many providers as you want. Just repeat this step for each provider.
---
## Step 5: Configure Your Model (1 min)
1. Go to **Settings** (gear icon)
2. Navigate to **Models**
3. Select your provider's model:
| Provider | Recommended Model | Notes |
|----------|-------------------|-------|
| **OpenRouter** | `anthropic/claude-3.5-sonnet` | Access 100+ models |
| **Anthropic** | `claude-3-5-sonnet-latest` | Best reasoning |
| **Google** | `gemini-3.5-flash` | Large context, fast |
| **Groq** | `llama-3.3-70b-versatile` | Ultra-fast |
| **Mistral** | `mistral-large-latest` | Strong European option |
4. Click **Save**
---
## Step 6: Create Your First Notebook (1 min)
1. Click **New Notebook**
2. Name: "My Research"
3. Click **Create**
---
## Step 7: Add Content & Chat (2 min)
1. Click **Add Source**
2. Choose **Web Link**
3. Paste any article URL
4. Wait for processing
5. Go to **Chat** and ask questions!
---
## Verification Checklist
- [ ] Docker is running
- [ ] You can access `http://localhost:8502`
- [ ] Provider credential is configured and tested
- [ ] Models are registered
- [ ] You created a notebook
- [ ] Chat works
**All checked?** You're ready to research!
---
## Provider Comparison
| Provider | Speed | Quality | Context | Cost |
|----------|-------|---------|---------|------|
| **OpenRouter** | Varies | Varies | Varies | Varies (100+ models) |
| **Anthropic** | Medium | Excellent | 200K | $$$ |
| **Google** | Fast | Very Good | 1M+ | $$ |
| **Groq** | Ultra-fast | Good | 128K | $ (free tier) |
| **Mistral** | Fast | Good | 128K | $$ |
| **DeepSeek** | Medium | Very Good | 64K | $ |
---
## Troubleshooting
### "Model not found" Error
1. Go to **Settings** → **API Keys**
2. Click **Test Connection** on your credential
3. If valid, click **Discover Models** → **Register Models**
4. Check you have credits/access for the model
### "Cannot connect to server"
```bash
docker ps # Check all services running
docker compose logs # View logs
docker compose restart # Restart everything
```
### Provider-Specific Issues
**Anthropic**: Ensure key starts with `sk-ant-`
**Google**: Use AI Studio key, not Cloud Console
**Groq**: Free tier has rate limits; upgrade if needed
---
## Cost Estimates
Approximate costs per 1K tokens:
| Provider | Input | Output |
|----------|-------|--------|
| Anthropic (Sonnet) | $0.003 | $0.015 |
| Google (Flash) | $0.0001 | $0.0004 |
| Groq (Llama 70B) | Free tier available | - |
| Mistral (Large) | $0.002 | $0.006 |
Check provider websites for current pricing.
---
## Next Steps
1. **Add Your Content**: PDFs, web links, documents
2. **Explore Features**: Podcasts, transformations, search
3. **Full Documentation**: [See all features](../3-USER-GUIDE/index.md)
---
**Need help?** Join our [Discord community](https://discord.gg/37XJPXfz2w)!
docs/0-START-HERE/quick-start-external-ollama.md
+214
View File
@@ -0,0 +1,214 @@
# Quick Start - External Ollama
Run Open Notebook with a **separately installed Ollama** (not via Docker). This avoids Docker running the Ollama service while you use your own local Ollama installation.
## Prerequisites
1. **Docker Desktop** installed (for SurrealDB and Open Notebook)
- [Download here](https://www.docker.com/products/docker-desktop/)
2. **Ollama** installed separately
- [Download here](https://ollama.ai/)
- Verify: run `ollama --version`
3. **Models downloaded** in Ollama:
```bash
ollama pull mistral
ollama pull nomic-embed-text
```
---
## Step 1: Start Ollama (1 min)
Start the Ollama server:
```bash
# Default: runs on http://localhost:11434
ollama serve
```
Keep this terminal open. Ollama will run in the background.
**Optional: Start Ollama on a custom port or network interface:**
```bash
OLLAMA_HOST=0.0.0.0:11434 ollama serve
```
---
## Step 2: Create Configuration (1 min)
Create a new folder `open-notebook-external-ollama` and add these files:
**docker-compose.yml**:
```yaml
services:
surrealdb:
image: surrealdb/surrealdb:v2
command: start --user root --pass password rocksdb:/mydata/mydatabase.db
user: root
ports:
# Localhost only — the database uses default credentials, so never
# publish this port on 0.0.0.0
- "127.0.0.1:8000:8000"
volumes:
- ./surreal_data:/mydata
open_notebook:
image: lfnovo/open_notebook:v1-latest
pull_policy: always
ports:
- "8502:8502" # Web UI (React frontend)
- "5055:5055" # API (required!)
environment:
# Encryption key for credential storage (required)
- OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
# Database (required)
- SURREAL_URL=ws://surrealdb:8000/rpc
- SURREAL_USER=root
- SURREAL_PASSWORD=password
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=open_notebook
volumes:
- ./notebook_data:/app/data
depends_on:
- surrealdb
restart: always
```
**Note:** No Ollama service in Docker — we use the host's Ollama.
---
## Step 3: Connect Open Notebook to Host Ollama (1 min)
When Open Notebook runs inside Docker, it cannot reach `localhost:11434` on your host directly. Use the special hostname:
| Host OS | Ollama URL in Open Notebook |
|---------|----------------------------|
| Linux | `http://host.containers.internal:11434` |
| macOS | `http://host.docker.internal:11434` |
| Windows | `http://host.docker.internal:11434` |
---
## Step 4: Start Open Notebook (1 min)
Open terminal in your `open-notebook-external-ollama` folder:
```bash
docker compose up -d
```
Wait 10-15 seconds for services to start.
---
## Step 5: Configure Ollama Provider (1 min)
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **Ollama**
4. Give it a name (e.g., "Local Ollama")
5. Enter the base URL:
- **Windows/macOS:** `http://host.docker.internal:11434`
- **Linux:** `http://host.containers.internal:11434`
6. Click **Save**
7. Click **Test Connection** — should show success
8. Click **Discover Models** → **Register Models**
---
## Step 6: Configure Models (1 min)
1. Go to **Settings** → **Models**
2. Set:
- **Language Model**: `ollama/mistral` (or whichever model you downloaded)
- **Embedding Model**: `ollama/nomic-embed-text`
3. Click **Save**
---
## Step 7: Access Open Notebook (instant)
Open your browser:
```
http://localhost:8502
```
---
## Verification Checklist
- [ ] Ollama is running (`ollama serve` in terminal)
- [ ] Docker is running
- [ ] You can access `http://localhost:8502`
- [ ] Ollama credential is configured with host URL and tested
- [ ] Models are registered
- [ ] Chat works
---
## Troubleshooting
### "Connection failed" when testing Ollama credential
1. Verify Ollama is running:
```bash
curl http://localhost:11434/api/version
```
2. Check firewall allows local connections on port 11434
3. For Windows/macOS, ensure `host.docker.internal` is reachable from inside the container:
```bash
docker exec <open_notebook_container> curl http://host.docker.internal:11434/api/version
```
### Ollama not starting
```bash
# Check Ollama logs
ollama list
# Pull a model again
ollama pull mistral
```
### "Address already in use" for SurrealDB
```bash
docker compose down
docker compose up -d
```
---
## Why External Ollama?
| Approach | Ollama in Docker | Ollama External |
|----------|-----------------|-----------------|
| **Resource isolation** | Separated | Shares with host |
| **GPU access** | Requires Docker GPU setup | Native GPU access |
| **Model management** | Via `docker exec` | Via terminal directly |
| **Memory usage** | Isolated from host | Shared with host apps |
**External Ollama** is recommended if you:
- Already have Ollama installed and configured
- Want GPU access without Docker GPU passthrough complexity
- Prefer managing models via command line directly
---
## Going Further
- **Add more models**: Run `ollama pull <model>`, then re-discover from Open Notebook
- **Check Ollama status**: `ollama list` shows downloaded models
- **Customize Ollama**: Edit `~/.ollama/config.yaml` for advanced settings
---
**Need Help?** Join our [Discord community](https://discord.gg/37XJPXfz2w)
docs/0-START-HERE/quick-start-local.md
+308
View File
@@ -0,0 +1,308 @@
# Quick Start - Local & Private (5 minutes)
Get Open Notebook running with **100% local AI** using Ollama. No cloud API keys needed, completely private.
**Already have Ollama installed?** See [External Ollama Guide](quick-start-external-ollama.md) instead.
## Prerequisites
1. **Docker Desktop** installed
- [Download here](https://www.docker.com/products/docker-desktop/)
- Already have it? Skip to step 2
2. **Local LLM** - Choose one:
- **Ollama** (recommended): [Download here](https://ollama.ai/)
- **LM Studio** (GUI alternative): [Download here](https://lmstudio.ai)
## Step 1: Choose Your Setup (1 min)
### Local Machine (Same Computer)
Everything runs on your machine. Recommended for testing/learning.
### Remote Server (Raspberry Pi, NAS, Cloud VM)
Run on a different computer, access from another. Needs network configuration.
---
## Step 2: Create Configuration (1 min)
Create a new folder `open-notebook-local` and add this file:
**docker-compose.yml**:
```yaml
services:
surrealdb:
image: surrealdb/surrealdb:v2
command: start --user root --pass password rocksdb:/mydata/mydatabase.db
user: root
ports:
# Localhost only — the database uses default credentials, so never
# publish this port on 0.0.0.0
- "127.0.0.1:8000:8000"
volumes:
- ./surreal_data:/mydata
open_notebook:
image: lfnovo/open_notebook:v1-latest
pull_policy: always
ports:
- "8502:8502" # Web UI (React frontend)
- "5055:5055" # API (required!)
environment:
# Encryption key for credential storage (required)
- OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
# Database (required)
- SURREAL_URL=ws://surrealdb:8000/rpc
- SURREAL_USER=root
- SURREAL_PASSWORD=password
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=open_notebook
# Ollama (required when running Ollama via Docker, as in this compose file)
- OLLAMA_BASE_URL=http://ollama:11434
volumes:
- ./notebook_data:/app/data
depends_on:
- surrealdb
restart: always
ollama:
image: ollama/ollama:latest
ports:
- "11434:11434"
volumes:
- ./ollama_models:/root/.ollama
restart: always
# Optional: set GPU support if available
#deploy:
# resources:
# reservations:
# devices:
# - driver: nvidia
# count: 1
# capabilities: [gpu]
```
**Edit the file:**
- Replace `change-me-to-a-secret-string` with your own secret (any string works)
---
## Step 3: Start Services (1 min)
Open terminal in your `open-notebook-local` folder:
```bash
docker compose up -d
```
Wait 10-15 seconds for all services to start.
---
## Step 4: Download a Model (2-3 min)
Ollama needs at least one language model. Pick one:
```bash
# Fastest & smallest (recommended for testing)
docker exec open-notebook-local-ollama-1 ollama pull mistral
# OR: Better quality but slower
docker exec open-notebook-local-ollama-1 ollama pull neural-chat
# OR: Even better quality, more VRAM needed
docker exec open-notebook-local-ollama-1 ollama pull llama2
```
This downloads the model (will take 1-5 minutes depending on your internet).
---
## Step 5: Access Open Notebook (instant)
Open your browser:
```
http://localhost:8502
```
You should see the Open Notebook interface.
---
## Step 6: Configure Ollama Provider (1 min)
1. Go to **Manage** → **Models**
2. Click **Add Credential**
3. Select provider: **Ollama**
4. Give it a name (e.g., "Local Ollama")
5. Enter the base URL: `http://ollama:11434`
6. Click **Save**
7. Click **Test Connection** — should show success
8. Click **Discover Models** → **Register Models**
---
## Step 7: Configure Local Model (1 min)
1. Go to **Manage** → **Models**
2. Set:
- **Language Model**: `ollama/mistral` (or whichever model you downloaded)
- **Embedding Model**: `ollama/nomic-embed-text` (auto-downloads if missing)
3. Click **Save**
---
## Step 8: Create Your First Notebook (1 min)
1. Click **New Notebook**
2. Name: "My Private Research"
3. Click **Create**
---
## Step 9: Add Local Content (1 min)
1. Click **Add Source**
2. Choose **Text**
3. Paste some text or a local document
4. Click **Add**
---
## Step 10: Chat With Your Content (1 min)
1. Go to **Chat**
2. Type: "What did you learn from this?"
3. Click **Send**
4. Watch as the local Ollama model responds!
---
## Verification Checklist
- [ ] Docker is running
- [ ] You can access `http://localhost:8502`
- [ ] Ollama credential is configured and tested
- [ ] Models are registered
- [ ] You created a notebook
- [ ] Chat works with local model
**All checked?** You have a completely **private, offline** research assistant!
---
## Advantages of Local Setup
- **No API costs** - Free forever
- **No internet required** - True offline capability
- **Privacy first** - Your data never leaves your machine
- **No subscriptions** - No monthly bills
**Trade-off:** Slower than cloud models (depends on your CPU/GPU)
---
## Troubleshooting
### "ollama: command not found"
Docker image name might be different:
```bash
docker ps # Find the Ollama container name
docker exec <container_name> ollama pull mistral
```
### Model Download Stuck
Check internet connection and restart:
```bash
docker compose restart ollama
```
Then retry the model pull command.
### "Address already in use" Error
```bash
docker compose down
docker compose up -d
```
### Low Performance
Check if GPU is available:
```bash
# Show available GPUs
docker exec open-notebook-local-ollama-1 ollama ps
# Enable GPU in docker-compose.yml
```
Then restart: `docker compose restart ollama`
### Adding More Models
```bash
# List available models
docker exec open-notebook-local-ollama-1 ollama list
# Pull additional model
docker exec open-notebook-local-ollama-1 ollama pull neural-chat
```
---
## Next Steps
**Now that it's running:**
1. **Add Your Own Content**: PDFs, documents, articles (see 3-USER-GUIDE)
2. **Explore Features**: Podcasts, transformations, search
3. **Full Documentation**: [See all features](../3-USER-GUIDE/index.md)
4. **Scale Up**: Deploy to a server with better hardware for faster responses
5. **Benchmark Models**: Try different models to find the speed/quality tradeoff you prefer
## Alternative: Using LM Studio Instead of Ollama
**Prefer a GUI?** LM Studio is easier for non-technical users:
1. Download LM Studio: https://lmstudio.ai
2. Open the app, download a model from the library
3. Go to "Local Server" tab, start server (port 1234)
4. In Open Notebook, go to **Settings** → **API Keys**
5. Click **Add Credential** → Select **OpenAI-Compatible**
6. Enter base URL: `http://host.docker.internal:1234/v1`
7. Enter API key: `lm-studio` (placeholder)
8. Click **Save**, then **Test Connection**
9. Configure in Settings → Models → Select your LM Studio model
**Note**: LM Studio runs outside Docker, use `host.docker.internal` to connect.
---
## Going Further
- **Switch models**: Change in Settings → Models anytime
- **Add more models**:
- Ollama: Run `ollama pull <model>`, then re-discover models from the credential
- LM Studio: Download from the app library
- **Deploy to server**: Same docker-compose.yml works anywhere
- **Use cloud hybrid**: Keep some local models, add cloud provider credentials for complex tasks
---
## Common Model Choices
| Model | Speed | Quality | VRAM | Best For |
|-------|-------|---------|------|----------|
| **mistral** | Fast | Good | 4GB | Testing, general use |
| **neural-chat** | Medium | Better | 6GB | Balanced, recommended |
| **llama2** | Slow | Best | 8GB+ | Complex reasoning |
| **phi** | Very Fast | Fair | 2GB | Minimal hardware |
---
**Need Help?** Join our [Discord community](https://discord.gg/37XJPXfz2w) - many users run local setups!
docs/0-START-HERE/quick-start-openai.md
+197
View File
@@ -0,0 +1,197 @@
# Quick Start - OpenAI (5 minutes)
Get Open Notebook running with OpenAI's GPT models. Fast, powerful, and simple.
## Prerequisites
1. **Docker Desktop** installed
- [Download here](https://www.docker.com/products/docker-desktop/)
- Already have it? Skip to step 2
2. **OpenAI API Key** (required)
- Go to https://platform.openai.com/api-keys
- Create account → Create new secret key
- Add at least $5 in credits to your account
- Copy the key (starts with `sk-`)
## Step 1: Create Configuration (1 min)
Create a new folder `open-notebook` and add this file:
**docker-compose.yml**:
```yaml
services:
surrealdb:
image: surrealdb/surrealdb:v2
command: start --user root --pass password rocksdb:/mydata/mydatabase.db
ports:
# Localhost only — the database uses default credentials, so never
# publish this port on 0.0.0.0
- "127.0.0.1:8000:8000"
volumes:
- ./surreal_data:/mydata
open_notebook:
image: lfnovo/open_notebook:v1-latest
pull_policy: always
ports:
- "8502:8502" # Web UI
- "5055:5055" # API
environment:
# Encryption key for credential storage (required)
- OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
# Database (required)
- SURREAL_URL=ws://surrealdb:8000/rpc
- SURREAL_USER=root
- SURREAL_PASSWORD=password
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=open_notebook
volumes:
- ./notebook_data:/app/data
depends_on:
- surrealdb
restart: always
```
**Edit the file:**
- Replace `change-me-to-a-secret-string` with your own secret (any string works)
---
## Step 2: Start Services (1 min)
Open terminal in your `open-notebook` folder:
```bash
docker compose up -d
```
Wait 15-20 seconds for services to start.
---
## Step 3: Access Open Notebook (instant)
Open your browser:
```
http://localhost:8502
```
You should see the Open Notebook interface!
---
## Step 4: Configure Your OpenAI Provider (1 min)
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **OpenAI**
4. Give it a name (e.g., "My OpenAI Key")
5. Paste your OpenAI API key
6. Click **Save**
7. Click **Test Connection** — should show success
8. Click **Discover Models** → **Register Models**
Your OpenAI models are now available!
---
## Step 5: Create Your First Notebook (1 min)
1. Click **New Notebook**
2. Name: "My Research"
3. Click **Create**
---
## Step 6: Add a Source (1 min)
1. Click **Add Source**
2. Choose **Web Link**
3. Paste: `https://en.wikipedia.org/wiki/Artificial_intelligence`
4. Click **Add**
5. Wait for processing (30-60 seconds)
---
## Step 7: Chat With Your Content (1 min)
1. Go to **Chat**
2. Type: "What is artificial intelligence?"
3. Click **Send**
4. Watch as GPT responds with information from your source!
---
## Verification Checklist
- [ ] Docker is running
- [ ] You can access `http://localhost:8502`
- [ ] OpenAI credential is configured and tested
- [ ] You created a notebook
- [ ] You added a source
- [ ] Chat works
**All checked?** You have a fully working AI research assistant!
---
## Using Different Models
In your notebook, go to **Settings** → **Models** to choose:
- `gpt-4o` - Best quality (recommended)
- `gpt-4o-mini` - Fast and cheap (good for testing)
---
## Troubleshooting
### "Port 8502 already in use"
Change the port in docker-compose.yml:
```yaml
ports:
- "8503:8502" # Use 8503 instead
```
Then access at `http://localhost:8503`
### "API key not working"
1. Go to **Settings** → **API Keys**
2. Click **Test Connection** on your OpenAI credential
3. If it fails, verify your key at https://platform.openai.com
4. Delete the credential and create a new one with the correct key
### "Cannot connect to server"
```bash
docker ps # Check all services running
docker compose logs # View logs
docker compose restart # Restart everything
```
---
## Next Steps
1. **Add Your Own Content**: PDFs, web links, documents
2. **Explore Features**: Podcasts, transformations, search
3. **Full Documentation**: [See all features](../3-USER-GUIDE/index.md)
---
## Cost Estimate
OpenAI pricing (approximate):
- **Conversation**: $0.01-0.10 per 1K tokens
- **Embeddings**: $0.02 per 1M tokens
- **Typical usage**: $1-5/month for light use, $20-50/month for heavy use
Check https://openai.com/pricing for current rates.
---
**Need help?** Join our [Discord community](https://discord.gg/37XJPXfz2w)!
docs/1-INSTALLATION/docker-compose.md
+374
View File
@@ -0,0 +1,374 @@
# Docker Compose Installation (Recommended)
Multi-container setup with separate services. **Best for most users.**
> **Alternative Registry:** All images are available on both Docker Hub (`lfnovo/open_notebook`) and GitHub Container Registry (`ghcr.io/lfnovo/open-notebook`). Use GHCR if Docker Hub is blocked or you prefer GitHub-native workflows.
## Prerequisites
- **Docker Desktop** installed ([Download](https://www.docker.com/products/docker-desktop/))
- **5-10 minutes** of your time
- **API key** for at least one AI provider (OpenAI recommended for beginners)
## Step 1: Get docker-compose.yml (1 min)
**Option A: Download from repository**
```bash
curl -o docker-compose.yml https://raw.githubusercontent.com/lfnovo/open-notebook/main/docker-compose.yml
```
**Option B: Use the official file from the repo**
The official `docker-compose.yml` is in the root of our repository: [View on GitHub](https://github.com/lfnovo/open-notebook/blob/main/docker-compose.yml)
Copy that file to your project folder.
**Option C: Create manually**
Create a file called `docker-compose.yml` with this content:
```yaml
services:
surrealdb:
image: surrealdb/surrealdb:v2
# Credentials default to root:root for a zero-config local setup. Before
# exposing this instance to a network, set SURREAL_USER / SURREAL_PASSWORD
# in a .env file (see .env.example) — they are applied here and to the
# open_notebook service below, so the two always stay in sync.
# List (exec) form so each interpolated value stays a single argument —
# a password containing spaces would otherwise be split into several.
command: ["start", "--log", "info", "--user", "${SURREAL_USER:-root}", "--pass", "${SURREAL_PASSWORD:-root}", "rocksdb:/mydata/mydatabase.db"]
user: root # Required for bind mounts on Linux
ports:
# Bound to localhost only: the open_notebook service reaches this over
# the internal compose network regardless, so the host port is purely
# for local debugging (e.g. Surrealist, `surreal sql`). Exposing this
# on 0.0.0.0 would let anyone who can reach the host connect with the
# default root:root credentials.
- "127.0.0.1:8000:8000"
volumes:
- ./surreal_data:/mydata
environment:
- SURREAL_EXPERIMENTAL_GRAPHQL=true
restart: always
pull_policy: always
open_notebook:
image: lfnovo/open_notebook:v1-latest
ports:
- "8502:8502" # Web UI
- "5055:5055" # REST API
environment:
# REQUIRED: Change this to your own secret string
# This encrypts your API keys in the database
- OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
# Database connection. SURREAL_USER / SURREAL_PASSWORD default to root:root
# for local use; override them in a .env file before exposing the instance
# (the same values configure the surrealdb service above).
- SURREAL_URL=ws://surrealdb:8000/rpc
- SURREAL_USER=${SURREAL_USER:-root}
- SURREAL_PASSWORD=${SURREAL_PASSWORD:-root}
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=open_notebook
volumes:
- ./notebook_data:/app/data
depends_on:
- surrealdb
restart: always
pull_policy: always
```
**Edit the file:**
- Replace `change-me-to-a-secret-string` with your own secret (any string works, e.g., `my-super-secret-key-123`)
- (Optional) To use database credentials other than the default `root:root`, create a `.env` file next to `docker-compose.yml` with `SURREAL_USER=...` and `SURREAL_PASSWORD=...` — both services pick them up automatically ([.env.example](https://github.com/lfnovo/open-notebook/blob/main/.env.example) shows the full format)
---
## Step 2: Start Services (2 min)
Open terminal in the `open-notebook` folder:
```bash
docker compose up -d
```
Wait 15-20 seconds for all services to start:
```
✅ surrealdb running on :8000
✅ open_notebook running on :8502 (UI) and :5055 (API)
```
Check status:
```bash
docker compose ps
```
---
## Step 3: Verify Installation (1 min)
**API Health:**
```bash
curl http://localhost:5055/health
# Should return: {"status": "healthy"}
```
**Frontend Access:**
Open browser to:
```
http://localhost:8502
```
You should see the Open Notebook interface!
---
## Step 4: Configure AI Provider (2 min)
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select your provider (e.g., OpenAI, Anthropic, Google)
4. Give it a name, paste your API key
5. Click **Save**
6. Click **Test Connection** — should show success
7. Click **Discover Models** → **Register Models**
Your models are now available!
> **Need an API key?** Get one from your chosen provider:
> - **OpenAI**: https://platform.openai.com/api-keys
> - **Anthropic**: https://console.anthropic.com/
> - **Google**: https://aistudio.google.com/
> - **Groq**: https://console.groq.com/
---
## Step 5: First Notebook (2 min)
1. Click **New Notebook**
2. Name: "My Research"
3. Description: "Getting started"
4. Click **Create**
Done! You now have a fully working Open Notebook instance.
---
## Configuration
### Adding Ollama (Free Local Models)
Instead of manually editing, use our ready-made example:
```bash
# Download the Ollama example
curl -o docker-compose.yml https://raw.githubusercontent.com/lfnovo/open-notebook/main/examples/docker-compose-ollama.yml
# Or copy from repo
cp examples/docker-compose-ollama.yml docker-compose.yml
```
See [examples/docker-compose-ollama.yml](../../examples/docker-compose-ollama.yml) for the complete setup.
**Manual setup:** Add this to your existing `docker-compose.yml`:
```yaml
ollama:
image: ollama/ollama:latest
ports:
- "11434:11434"
volumes:
- ollama_models:/root/.ollama
restart: always
volumes:
ollama_models:
```
Then restart and pull a model:
```bash
docker compose restart
docker exec open-notebook-local-ollama-1 ollama pull mistral
```
Configure Ollama in the Settings UI:
1. Go to **Settings** → **API Keys**
2. Click **Add Credential** → Select **Ollama**
3. Enter base URL: `http://ollama:11434`
4. Click **Save**, then **Test Connection**
5. Click **Discover Models** → **Register Models**
---
## Environment Variables Reference
| Variable | Purpose | Example |
|----------|---------|---------|
| `OPEN_NOTEBOOK_ENCRYPTION_KEY` | Encryption key for credentials | `my-secret-key` |
| `SURREAL_URL` | Database connection | `ws://surrealdb:8000/rpc` |
| `SURREAL_USER` | Database user | `root` |
| `SURREAL_PASSWORD` | Database password | `root` |
| `SURREAL_NAMESPACE` | Database namespace | `open_notebook` |
| `SURREAL_DATABASE` | Database name | `open_notebook` |
| `API_URL` | API external URL | `http://localhost:5055` |
| `OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE` | Override embedding batch size for stricter/local providers (recommended: `8` for CPU-only local setups) | `50` |
See [Environment Reference](../5-CONFIGURATION/environment-reference.md) for complete list.
---
## Common Tasks
### Stop Services
```bash
docker compose down
```
### View Logs
```bash
# All services
docker compose logs -f
# Specific service
docker compose logs -f api
```
### Restart Services
```bash
docker compose restart
```
### Update to Latest Version
```bash
docker compose down
docker compose pull
docker compose up -d
```
### Remove All Data
```bash
docker compose down -v
```
---
## Troubleshooting
### "Cannot connect to API" Error
1. Check if Docker is running:
```bash
docker ps
```
2. Check if services are running:
```bash
docker compose ps
```
3. Check API logs:
```bash
docker compose logs api
```
4. Wait longer - services can take 20-30 seconds to start on first run
---
### Port Already in Use
If you get "Port 8502 already in use", change the port:
```yaml
ports:
- "8503:8502" # Use 8503 instead
- "5055:5055" # Keep API port same
```
Then access at `http://localhost:8503`
---
### Credential Issues
1. Go to **Settings** → **API Keys**
2. Click **Test Connection** on the credential
3. If it fails, verify key at provider's website
4. Check you have credits in your account
5. Delete and re-create the credential if needed
---
### Database Connection Issues
Check SurrealDB is running:
```bash
docker compose logs surrealdb
```
Reset database:
```bash
docker compose down -v
docker compose up -d
```
### Database Permission Denied (Linux)
If you see `Permission denied` or `Failed to create RocksDB directory` in SurrealDB logs:
```bash
docker compose logs surrealdb | grep -i permission
```
This happens because SurrealDB runs as a non-root user but Docker creates bind mount directories as root. Add `user: root` to the surrealdb service:
```yaml
surrealdb:
image: surrealdb/surrealdb:v2
user: root # Fix for Linux bind mount permissions
# ... rest of config
```
Then restart:
```bash
docker compose down -v
docker compose up -d
```
---
## Alternative Setups
Looking for different configurations? Check out our [examples/](../../examples/) folder:
- **[Ollama Setup](../../examples/docker-compose-ollama.yml)** - Run local AI models (free, private)
- **[Single Container](../../examples/docker-compose-single.yml)** - All-in-one container (deprecated, will be removed in v2)
- **[Development](../../examples/docker-compose-dev.yml)** - For contributors and developers
Each example includes detailed comments and usage instructions.
---
## Next Steps
1. **Add Content**: Sources, notebooks, documents
2. **Configure Models**: Settings → Models (choose your preferences)
3. **Explore Features**: Chat, search, transformations
4. **Read Guide**: [User Guide](../3-USER-GUIDE/index.md)
---
## Production Deployment
For production use, see:
- [Security Hardening](../5-CONFIGURATION/security.md)
- [Reverse Proxy](../5-CONFIGURATION/reverse-proxy.md)
---
## Getting Help
- **Discord**: [Community support](https://discord.gg/37XJPXfz2w)
- **Issues**: [GitHub Issues](https://github.com/lfnovo/open-notebook/issues)
- **Docs**: [Full documentation](../index.md)
docs/1-INSTALLATION/from-source.md
+195
View File
@@ -0,0 +1,195 @@
# From Source Installation
Clone the repository and run locally. **For developers and contributors.**
## Prerequisites
- **Python 3.11+** - [Download](https://www.python.org/)
- **Node.js 18+** - [Download](https://nodejs.org/)
- **Git** - [Download](https://git-scm.com/)
- **Docker** (for SurrealDB) - [Download](https://docker.com/)
- **uv** (Python package manager) - `curl -LsSf https://astral.sh/uv/install.sh | sh`
- API key from OpenAI or similar (or use Ollama for free)
## Quick Setup (10 minutes)
### 1. Clone Repository
```bash
git clone https://github.com/lfnovo/open-notebook.git
cd open-notebook
# If you forked it:
git clone https://github.com/YOUR_USERNAME/open-notebook.git
cd open-notebook
git remote add upstream https://github.com/lfnovo/open-notebook.git
```
### 2. Install Python Dependencies
```bash
uv sync
uv pip install python-magic
```
#### 2.1 Alternative: Conda Setup (Optional)
If you prefer using **Conda** to manage your environments, follow these steps instead of the standard `uv sync`:
```bash
# Create and activate the environment
conda create -n open-notebook python=3.11 -y
conda activate open-notebook
# Install uv inside conda to maintain compatibility with the Makefile
conda install -c conda-forge uv nodejs -y
# Sync dependencies
uv sync
```
> **Note**: Installing `uv` inside your Conda environment ensures that commands like `make start-all` and `make api` continue to work seamlessly.
### 3. Start SurrealDB
```bash
# Terminal 1
make database
# or: docker compose up surrealdb
```
### 4. Set Environment Variables
```bash
cp .env.example .env
# Edit .env and set:
# OPEN_NOTEBOOK_ENCRYPTION_KEY=my-secret-key
```
After starting the app, configure AI providers via the **Manage → Models** UI in the browser.
### 5. Start API
```bash
# Terminal 2
make api
# or: uv run --env-file .env uvicorn api.main:app --host 0.0.0.0 --port 5055
```
### 6. Start Worker
Source and note processing (content extraction, embedding, insights) is dispatched
as background jobs that a **separate worker** process consumes. Without it, every
source stays stuck at `Source processing status: CommandStatus.NEW` forever.
```bash
# Terminal 3
make worker
# or: uv run --env-file .env surreal-commands-worker --import-modules commands
```
> `make start-all` starts Database + API + Worker + Frontend together; the steps
> above run them individually so you can see each process's logs.
### 7. Start Frontend
```bash
# Terminal 4
cd frontend && npm install && npm run dev
```
### 8. Access
- **Frontend**: http://localhost:3000
- **API Docs**: http://localhost:5055/docs
- **Database**: http://localhost:8000
### 9. Configure AI Provider
1. Open http://localhost:3000
2. Go to **Manage** → **Models**
3. Click **Add Credential** → Select your provider → Paste API key
4. Click **Save**, then **Test Connection**
5. Click **Discover Models** → **Register Models**
---
## Development Workflow
### Code Quality
```bash
# Format and lint Python
make ruff
# or: ruff check . --fix
# Type checking
make lint
# or: uv run python -m mypy .
```
### Run Tests
```bash
uv run pytest tests/
```
### Common Commands
```bash
# Start everything
make start-all
# View API docs
open http://localhost:5055/docs
# Check database migrations
# (Auto-run on API startup)
# Clean up
make clean
```
---
## Troubleshooting
### Python version too old
```bash
python --version # Check version
uv sync --python 3.11 # Use specific version
```
### npm: command not found
Install Node.js from https://nodejs.org/
### Database connection errors
```bash
docker ps # Check SurrealDB running
docker logs surrealdb # View logs
```
### Port 5055 already in use
```bash
# Use different port
uv run uvicorn api.main:app --port 5056
```
---
## Next Steps
1. Read [Development Guide](../7-DEVELOPMENT/quick-start.md)
2. See [Architecture Overview](../7-DEVELOPMENT/architecture.md)
3. Check [Contributing Guide](../7-DEVELOPMENT/contributing.md)
---
## Getting Help
- **Discord**: [Community](https://discord.gg/37XJPXfz2w)
- **Issues**: [GitHub Issues](https://github.com/lfnovo/open-notebook/issues)
docs/1-INSTALLATION/index.md
+159
View File
@@ -0,0 +1,159 @@
# Installation Guide
Choose your installation route based on your setup and use case.
## Quick Decision: Which Route?
### 🚀 I want the easiest setup (Recommended for most)
**→ [Docker Compose](docker-compose.md)** - Multi-container setup, production-ready
- ✅ All features working
- ✅ Clear separation of services
- ✅ Easy to scale
- ✅ Works on Mac, Windows, Linux
- ⏱️ 5 minutes to running
---
### 🏠 I want everything in one container (Deprecated)
**→ [Single Container](single-container.md)** - Deprecated, will be removed in v2
- ⚠️ **Deprecated** — please use Docker Compose instead
- Still supported until v2 release
---
### 👨‍💻 I want to develop/contribute (Developers only)
**→ [From Source](from-source.md)** - Clone repo, set up locally
- ✅ Full control over code
- ✅ Easy to debug
- ✅ Can modify and test
- ⚠️ Requires Python 3.11+, Node.js
- ⏱️ 10 minutes to running
---
### 🪟 I'm on Windows and can't use Docker/WSL
**→ [Windows Native](windows-native.md)** - Run natively, no Docker or WSL
- ✅ Works on Windows ARM64
- ✅ For systems without Hyper-V/Docker Desktop
- ⚠️ Requires Python 3.12+, Node.js, SurrealDB, uv
- ⏱️ 15 minutes to running
---
## System Requirements
### Minimum
- **RAM**: 4GB
- **Storage**: 2GB for app + space for documents
- **CPU**: Any modern processor
- **Network**: Internet (optional for offline setup)
### Recommended
- **RAM**: 8GB+
- **Storage**: 10GB+ for documents and models
- **CPU**: Multi-core processor
- **GPU**: Optional (speeds up local AI models)
---
## AI Provider Options
### Cloud-Based (Pay-as-you-go)
- **OpenAI** - GPT-4, GPT-4o, fast and capable
- **Anthropic (Claude)** - Claude 3.5 Sonnet, excellent reasoning
- **Google Gemini** - Multimodal, cost-effective
- **Groq** - Ultra-fast inference
- **Others**: Mistral, DeepSeek, xAI, OpenRouter
**Cost**: Usually $0.01-$0.10 per 1K tokens
**Speed**: Fast (sub-second)
**Privacy**: Your data sent to cloud
### Local (Free, Private)
- **Ollama** - Run open-source models locally
- **LM Studio** - Desktop app for local models
- **Hugging Face models** - Download and run
**Cost**: $0 (just electricity)
**Speed**: Depends on your hardware (slow to medium)
**Privacy**: 100% offline
---
## Choose a Route
**Already know which way to go?** Pick your installation path:
- [Docker Compose](docker-compose.md) - **Most users**
- [Single Container](single-container.md) - **Deprecated**
- [From Source](from-source.md) - **Developers**
> **Privacy-first?** Any installation method works with Ollama for 100% local AI. See [Local Quick Start](../0-START-HERE/quick-start-local.md).
---
## Pre-Installation Checklist
Before installing, you'll need:
- [ ] **Docker** (for Docker routes) or **Node.js 18+** (for source)
- [ ] **AI Provider API key** (OpenAI, Anthropic, etc.) OR willingness to use free local models
- [ ] **At least 4GB RAM** available
- [ ] **Stable internet** (or offline setup with Ollama)
---
## Detailed Installation Instructions
### For Docker Users
1. Install [Docker Desktop](https://docker.com/products/docker-desktop)
2. Follow [Docker Compose](docker-compose.md) installation
3. Follow the step-by-step guide
4. Access at `http://localhost:8502`
### For Source Installation (Developers)
1. Have Python 3.11+, Node.js 18+, Git installed
2. Follow [From Source](from-source.md)
3. Run `make start-all`
4. Access at `http://localhost:8502` (frontend) or `http://localhost:5055` (API)
---
## After Installation
Once you're up and running:
1. **Configure Models** - Choose your AI provider in Settings
2. **Create First Notebook** - Start organizing research
3. **Add Sources** - PDFs, web links, documents
4. **Explore Features** - Chat, search, transformations
5. **Read Full Guide** - [User Guide](../3-USER-GUIDE/index.md)
---
## Troubleshooting During Installation
**Having issues?** Check the troubleshooting section in your chosen installation guide, or see [Quick Fixes](../6-TROUBLESHOOTING/quick-fixes.md).
---
## Need Help?
- **Discord**: [Join community](https://discord.gg/37XJPXfz2w)
- **GitHub Issues**: [Report problems](https://github.com/lfnovo/open-notebook/issues)
- **Docs**: See [Full Documentation](../index.md)
---
## Production Deployment
Installing for production use? See additional resources:
- [Security Hardening](../5-CONFIGURATION/security.md)
- [Reverse Proxy Setup](../5-CONFIGURATION/reverse-proxy.md)
- [Performance Tuning](../5-CONFIGURATION/advanced.md)
---
**Ready to install?** Pick a route above! ⬆️
docs/1-INSTALLATION/single-container.md
+146
View File
@@ -0,0 +1,146 @@
# Single Container Installation (Deprecated)
> **Deprecation Notice:** The single-container image (`v1-latest-single`) is **deprecated** and will be removed in v2. Please migrate to [Docker Compose](docker-compose.md), which is the recommended installation method for all users. The single-container image will continue to receive updates until v2 is released, but no new features or documentation will target it.
All-in-one container setup. **Simpler than Docker Compose, but less flexible.**
**Best for:** PikaPods, Railway, shared hosting, minimal setups
> **Alternative Registry:** Images available on both Docker Hub (`lfnovo/open_notebook:v1-latest-single`) and GitHub Container Registry (`ghcr.io/lfnovo/open-notebook:v1-latest-single`).
## Prerequisites
- Docker installed (for local testing)
- API key from OpenAI, Anthropic, or another provider
- 5 minutes
## Quick Setup
### For Local Testing (Docker)
```yaml
# docker-compose.yml
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest-single
pull_policy: always
ports:
- "8502:8502" # Web UI (React frontend)
- "5055:5055" # API
environment:
- OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
- SURREAL_URL=ws://localhost:8000/rpc
- SURREAL_USER=root
- SURREAL_PASSWORD=root
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=open_notebook
volumes:
- ./data:/app/data
restart: always
```
Run:
```bash
docker compose up -d
```
Access: `http://localhost:8502`
Then configure your AI provider:
1. Go to **Settings** → **API Keys**
2. Click **Add Credential** → Select your provider → Paste API key
3. Click **Save**, then **Test Connection**
4. Click **Discover Models** → **Register Models**
### For Cloud Platforms
**PikaPods:**
1. Click "New App"
2. Search "Open Notebook"
3. Set environment variables (at minimum: `OPEN_NOTEBOOK_ENCRYPTION_KEY`)
4. Click "Deploy"
5. Open the app → Go to **Settings → API Keys** to configure your AI provider
**Railway:**
1. Create new project
2. Add `lfnovo/open_notebook:v1-latest-single`
3. Set environment variables (at minimum: `OPEN_NOTEBOOK_ENCRYPTION_KEY`)
4. Deploy
5. Open the app → Go to **Settings → API Keys** to configure your AI provider
**Render:**
1. Create new Web Service
2. Use Docker image: `lfnovo/open_notebook:v1-latest-single`
3. Set environment variables in dashboard (at minimum: `OPEN_NOTEBOOK_ENCRYPTION_KEY`)
4. Configure persistent disk for `/app/data` and `/mydata`
**DigitalOcean App Platform:**
1. Create new app from Docker Hub
2. Use image: `lfnovo/open_notebook:v1-latest-single`
3. Set port to 8502
4. Add environment variables (at minimum: `OPEN_NOTEBOOK_ENCRYPTION_KEY`)
5. Configure persistent storage
**Heroku:**
```bash
# Using heroku.yml
heroku container:push web
heroku container:release web
heroku config:set OPEN_NOTEBOOK_ENCRYPTION_KEY=your-secret-key
```
**Coolify:**
1. Add new service → Docker Image
2. Image: `lfnovo/open_notebook:v1-latest-single`
3. Port: 8502
4. Add environment variables (at minimum: `OPEN_NOTEBOOK_ENCRYPTION_KEY`)
5. Enable persistent volumes
6. Coolify handles HTTPS automatically
**EasyPanel:**
Open Notebook ships an EasyPanel template at [`examples/easypanel/`](https://github.com/lfnovo/open-notebook/tree/main/examples/easypanel). Unlike the single-image options above, the template provisions **two services** — the Open Notebook app and a dedicated SurrealDB instance — and generates the database password, encryption key, and (optionally) the app password for you.
- **One-click (recommended):** once the template is published to the official [EasyPanel template gallery](https://github.com/easypanel-io/templates), create a new service from "Open Notebook", set an app password (or leave it blank to auto-generate one), and deploy.
- **Manual:** copy `examples/easypanel/` into `templates/open-notebook` in a checkout of [`easypanel-io/templates`](https://github.com/easypanel-io/templates), run the templates playground (`npm run dev`), and create the template from the generated JSON in your EasyPanel instance.
After deployment, open the EasyPanel domain and configure your AI provider in **Settings → API Keys**. See [`examples/easypanel/README.md`](https://github.com/lfnovo/open-notebook/blob/main/examples/easypanel/README.md) for details.
---
## Environment Variables
| Variable | Purpose | Example |
|----------|---------|---------|
| `OPEN_NOTEBOOK_ENCRYPTION_KEY` | Encryption key for credentials (required) | `my-secret-key` |
| `SURREAL_URL` | Database | `ws://localhost:8000/rpc` |
| `SURREAL_USER` | DB user | `root` |
| `SURREAL_PASSWORD` | DB password | `root` |
| `SURREAL_NAMESPACE` | DB namespace | `open_notebook` |
| `SURREAL_DATABASE` | DB name | `open_notebook` |
| `API_URL` | External URL (for remote access) | `https://myapp.example.com` |
AI provider API keys are configured via the **Settings → API Keys** UI after deployment.
---
## Limitations vs Docker Compose
| Feature | Single Container | Docker Compose |
|---------|------------------|-----------------|
| Setup time | 2 minutes | 5 minutes |
| Complexity | Minimal | Moderate |
| Services | All bundled | Separated |
| Scalability | Limited | Excellent |
| Memory usage | ~800MB | ~1.2GB |
---
## Next Steps
Same as Docker Compose setup - just access via `http://localhost:8502` (local) or your platform's URL (cloud).
1. Go to **Settings → API Keys** to add your AI provider credential
2. **Test Connection** and **Discover Models**
See [Docker Compose](docker-compose.md) for full post-install guide.
docs/1-INSTALLATION/windows-native.md
+285
View File
@@ -0,0 +1,285 @@
# Open Notebook Windows Installation Guide (Native, No Docker)
This guide documents how to install and run [Open Notebook](https://github.com/lfnovo/open-notebook) on Windows **natively without Docker or WSL**.
## Who Is This For?
- **Windows ARM64 users** - Docker Desktop and WSL2 have limitations on ARM64
- **Users without Hyper-V** - Some Windows editions don't support Docker
- **Users who prefer native installs** - Simpler architecture, easier debugging
## What This Guide Covers
- Native Windows installation steps
- Critical configuration fixes for Windows
- Troubleshooting common issues
- Upgrade and maintenance scripts
## Prerequisites
| Software | Installation | Required |
| ------------ | -------------------------------- | -------- |
| Git | `winget install Git.Git` | Yes |
| Python 3.12+ | Via uv (installed automatically) | Yes |
| Node.js 18+ | `winget install OpenJS.NodeJS` | Yes |
| uv | `pip install uv` | Yes |
| SurrealDB | `scoop install surrealdb` | Yes |
## Quick Start
1. **Clone and setup:**
```bash
cd %USERPROFILE%\Projects # or your preferred location
git clone https://github.com/lfnovo/open-notebook.git
cd open-notebook
uv sync
cd frontend && npm install && cd ..
```
2. **Configure `.env`:**
- Copy `.env.example` to `.env`
- Add your API keys
- **CRITICAL:** Change `SURREAL_URL` from `localhost` to `127.0.0.1`:
```env
SURREAL_URL="ws://127.0.0.1:8000/rpc"
```
3. **Start the four services**, each in its own terminal, from the `open-notebook` folder.
> Open Notebook does not ship a launcher script — start the services manually as below (or wrap them in your own `.bat`, see [Optional: one-click launcher](#optional-one-click-launcher)).
```batch
REM Optional: point Open Notebook at a separate data folder (see Issue 4 below).
REM Set this in each terminal before running, or skip to use ./data.
set DATA_FOLDER=%USERPROFILE%\Projects\open-notebook-data
REM Terminal 1 — SurrealDB
surreal start --user root --pass root --bind 127.0.0.1:8000 rocksdb:%DATA_FOLDER%\surrealdb
REM Terminal 2 — API
uv run --env-file .env run_api.py
REM Terminal 3 — Worker (module form avoids the Windows "canonicalize" error, see Issue 3)
set PYTHONPATH=%CD%
uv run --env-file .env python -m surreal_commands.cli.worker --import-modules commands
REM Terminal 4 — Frontend
cd frontend && npm run dev
```
4. **Open the app:** http://127.0.0.1:3000
## Directory Structure (Recommended)
```
YourProjectsFolder\
├── open-notebook\ # Source code (git clone)
│ ├── .venv\ # Python virtual environment (created by uv)
│ ├── frontend\ # Next.js frontend
│ ├── commands\ # Worker command modules
│ └── .env # Your configuration
├── open-notebook-data\ # Data storage (SEPARATE from code!)
│ ├── surrealdb\ # Database files
│ ├── uploads\ # Uploaded documents
│ └── sqlite-db\ # LangGraph checkpoints
└── start-open-notebook.bat # Optional launcher you create yourself (see below)
```
**Why separate data folder?** Prevents accidental data loss when updating/reinstalling code.
## Optional: one-click launcher
Open Notebook does not ship a launcher, but you can save the following as
`start-open-notebook.bat` (anywhere you like) to start all four services with a
double-click. Adjust `ROOT` and `DATA_ROOT` to match your setup.
```batch
@echo off
REM --- adjust these two paths ---
set ROOT=%USERPROFILE%\Projects\open-notebook
set DATA_ROOT=%USERPROFILE%\Projects\open-notebook-data
set DATA_FOLDER=%DATA_ROOT%
set PYTHONPATH=%ROOT%
cd /d %ROOT%
start "SurrealDB" surreal start --user root --pass root --bind 127.0.0.1:8000 rocksdb:%DATA_ROOT%\surrealdb
start "API" cmd /k "uv run --env-file .env run_api.py"
start "Worker" cmd /k "uv run --env-file .env python -m surreal_commands.cli.worker --import-modules commands"
start "Frontend" cmd /k "cd /d %ROOT%\frontend && npm run dev"
```
Then open http://127.0.0.1:3000.
## Critical Windows Fixes
### Issue 1: Wrong Python Version
**Symptom:**
```
ModuleNotFoundError: No module named 'langgraph.checkpoint.sqlite'
```
Traceback shows system Python (e.g., `C:\Python314\`) instead of venv.
**Cause:** Windows may have multiple Python versions. The venv's `activate.bat` doesn't always override correctly.
**Solution:** Use `uv run` instead of direct python calls:
```batch
REM Wrong:
.venv\Scripts\python.exe run_api.py
REM Correct:
uv run python run_api.py
```
### Issue 2: Database Health Check Timeout
**Symptom:**
```
WARNING: Database health check timed out after 2 seconds
```
Frontend shows "Database is offline" even though SurrealDB is running.
**Cause:** `.env` uses `localhost` but SurrealDB binds to `127.0.0.1`.
**Solution:** In `.env`, change:
```env
# Wrong:
SURREAL_URL="ws://localhost:8000/rpc"
# Correct:
SURREAL_URL="ws://127.0.0.1:8000/rpc"
```
### Issue 3: Worker "Failed to canonicalize script path"
**Symptom:**
```
Failed to canonicalize script path
```
**Cause:** The `surreal-commands-worker.exe` can't find the Python `commands` module.
**Solution:** Use Python module invocation with PYTHONPATH:
```batch
set PYTHONPATH=%ROOT%
uv run --env-file .env python -m surreal_commands.cli.worker --import-modules commands
```
### Issue 4: DATA_FOLDER Path Parsing Error
**Symptom:**
```
warning: Failed to parse environment file .env at position X
```
**Cause:** `uv` can't parse Windows paths with backslashes in `.env`.
**Solution:** Keep `DATA_FOLDER` **commented out** in `.env`. Set it via batch file:
```batch
set DATA_FOLDER=C:\path\to\open-notebook-data
```
## Configuration Files
### Modifying `open_notebook/config.py`
The default `config.py` uses a hardcoded data path. Modify it to read from environment:
```python
import os
# ROOT DATA FOLDER - can be overridden via DATA_FOLDER environment variable
DATA_FOLDER = os.environ.get("DATA_FOLDER", "./data")
# Rest of file uses DATA_FOLDER...
```
### Required `.env` Settings
```env
# Database - MUST use 127.0.0.1!
SURREAL_URL="ws://127.0.0.1:8000/rpc"
SURREAL_USER="root"
SURREAL_PASSWORD="root"
SURREAL_NAMESPACE="open_notebook"
SURREAL_DATABASE="open_notebook"
# API Keys (uncomment and fill in)
OPENAI_API_KEY=your-key-here
ANTHROPIC_API_KEY=your-key-here
GOOGLE_API_KEY=your-key-here
```
## Available AI Models
Once running, add models in Settings. Common model names:
| Provider | Models |
| --------- | ------------------------------------------------------------ |
| OpenAI | `gpt-4o`, `gpt-4o-mini`, `gpt-4-turbo`, `text-embedding-3-small` |
| Anthropic | `claude-sonnet-4-20250514`, `claude-3-5-sonnet-20241022`, `claude-3-5-haiku-20241022` |
| Google | `gemini-3.5-flash`, `gemini-2.5-flash`, `gemini-2.5-pro` |
| DeepSeek | `deepseek-chat`, `deepseek-reasoner` |
## Upgrading
When a new version is released:
```batch
cd open-notebook
git pull
uv sync
cd frontend && npm install && cd ..
```
Then restart all services. Your `.env` and data are preserved.
## Services & Ports
| Service | Port | URL |
| --------- | ---- | -------------------------- |
| SurrealDB | 8000 | ws://127.0.0.1:8000 |
| API | 5055 | http://127.0.0.1:5055/docs |
| Frontend | 3000 | http://127.0.0.1:3000 |
## Troubleshooting
### Services won't start
- Check if ports are in use: `netstat -ano | findstr :8000`
- Kill existing processes: `taskkill /F /PID <pid>`
### Frontend can't connect to API
- Verify API is running: http://127.0.0.1:5055/docs
- Check `.env` has `API_URL=http://localhost:5055`
### Worker not processing commands
- Check Worker window for errors
- Verify PYTHONPATH is set in startup script
## Contributing
Found another Windows-specific issue? Please share your solution!
---
*Tested on Windows 11 ARM64 with Open Notebook v1.6.0*
*Created: January 2026*
docs/2-CORE-CONCEPTS/ai-context-rag.md
+450
View File
@@ -0,0 +1,450 @@
# AI Context & RAG - How Open Notebook Uses Your Research
Open Notebook uses different approaches to make AI models aware of your research depending on the feature. This section explains **RAG** (used in Ask) and **full-content context** (used in Chat).
---
## The Problem: Making AI Aware of Your Data
### Traditional Approaches (and their problems)
**Option 1: Fine-Tuning**
- Train the model on your data
- Pro: Model becomes specialized
- Con: Expensive, slow, permanent (can't unlearn)
**Option 2: Send Everything to Cloud**
- Upload all your data to ChatGPT/Claude API
- Pro: Works well, fast
- Con: Privacy nightmare, data leaves your control, expensive
**Option 3: Ignore Your Data**
- Just use the base model without your research
- Pro: Private, free
- Con: AI doesn't know anything about your specific topic
### Open Notebook's Dual Approach
**For Chat**: Sends the entire selected content to the LLM
- Simple and transparent: You select sources, they're sent in full
- Maximum context: AI sees everything you choose
- You control which sources are included
**For Ask (RAG)**: Retrieval-Augmented Generation
- RAG = Retrieval-Augmented Generation
- The insight: *Search your content, find relevant pieces, send only those*
- Automatic: AI decides what's relevant based on your question
---
## How RAG Works: Three Stages
### Stage 1: Content Preparation
When you upload a source, Open Notebook prepares it for retrieval:
```
1. EXTRACT TEXT
PDF → text
URL → webpage text
Audio → transcribed text
Video → subtitles + transcription
2. CHUNK INTO PIECES
Long documents → break into ~500-word chunks
Why? AI context has limits; smaller pieces are more precise
3. CREATE EMBEDDINGS
Each chunk → semantic vector (numbers representing meaning)
Why? Allows finding chunks by similarity, not just keywords
4. STORE IN DATABASE
Chunks + embeddings + metadata → searchable storage
```
**Example:**
```
Source: "AI Safety Research 2026" (50-page PDF)
↓
Extracted: 50 pages of text
↓
Chunked: 150 chunks (~500 words each)
↓
Embedded: Each chunk gets a vector (1536 numbers for OpenAI)
↓
Stored: Ready for search
```
---
### Stage 2: Query Time (What You Search For)
When you ask a question, the system finds relevant content:
```
1. YOU ASK A QUESTION
"What does the paper say about alignment?"
2. SYSTEM CONVERTS QUESTION TO EMBEDDING
Your question → vector (same way chunks are vectorized)
3. SIMILARITY SEARCH
Find chunks most similar to your question
(using vector math, not keyword matching)
4. RETURN TOP RESULTS
Usually top 5-10 most similar chunks
5. YOU GET BACK
✓ The relevant chunks
✓ Where they came from (sources + page numbers)
✓ Relevance scores
```
**Example:**
```
Q: "What does the paper say about alignment?"
↓
Q vector: [0.23, -0.51, 0.88, ..., 0.12]
↓
Search: Compare to all chunk vectors
↓
Results:
- Chunk 47 (alignment section): similarity 0.94
- Chunk 63 (safety approaches): similarity 0.88
- Chunk 12 (related work): similarity 0.71
```
---
### Stage 3: Augmentation (How AI Uses It)
Now you have the relevant pieces. The AI uses them:
```
SYSTEM BUILDS A PROMPT:
"You are an AI research assistant.
The user has the following research materials:
[CHUNK 47 CONTENT]
[CHUNK 63 CONTENT]
User question: 'What does the paper say about alignment?'
Answer based on the above materials."
AI RESPONDS:
"Based on the research materials, the paper approaches
alignment through [pulls from chunks] and emphasizes
[pulls from chunks]..."
SYSTEM ADDS CITATIONS:
"- See research materials page 15 for approach details
- See research materials page 23 for emphasis on X"
```
---
## Two Search Modes: Exact vs. Semantic
Open Notebook provides two different search strategies for different goals.
### 1. Text Search (Keyword Matching)
**How it works:**
- Uses BM25 ranking (the same algorithm Google uses)
- Finds chunks containing your keywords
- Ranks by relevance (how often keywords appear, position, etc.)
**When to use:**
- "I remember the exact phrase 'X' and want to find it"
- "I'm looking for a specific name or number"
- "I need the exact quote"
**Example:**
```
Search: "transformer architecture"
Results:
1. Chunk with "transformer architecture" 3 times
2. Chunk with "transformer" and "architecture" separately
3. Chunk with "transformer-based models"
```
### 2. Vector Search (Semantic Similarity)
**How it works:**
- Converts your question to a vector (number embedding)
- Finds chunks with similar vectors
- No keywords needed—finds conceptually similar content
**When to use:**
- "Find content about X (without saying exact words)"
- "I'm exploring a concept"
- "Find similar ideas even if worded differently"
**Example:**
```
Search: "what's the mechanism for model understanding?"
Results (no "understanding" in any chunk):
1. Chunk about interpretability and mechanistic analysis
2. Chunk about feature analysis
3. Chunk about attention mechanisms
Why? The vectors are semantically similar to your concept.
```
---
## Context Management: Your Control Panel
Here's where Open Notebook is different: **You decide what the AI sees.**
### The Three Levels
| Level | What's Shared | Example Cost | Privacy | Use Case |
|-------|---------------|--------------|---------|----------|
| **Full Content** | Complete source text | 10,000 tokens | Low | Detailed analysis, close reading |
| **Summary Only** | AI-generated summary | 2,000 tokens | High | Background material, references |
| **Not in Context** | Nothing | 0 tokens | Max | Confidential, irrelevant, or archived |
### How It Works
**Full Content:**
```
You: "What's the methodology in paper A?"
System:
- Searches paper A
- Retrieves full paper content (or large chunks)
- Sends to AI: "Here's paper A. Answer about methodology."
- AI analyzes complete content
- Result: Detailed, precise answer
```
**Summary Only:**
```
You: "I want to chat using paper A and B"
System:
- For Paper A: Sends AI-generated summary (not full text)
- For Paper B: Sends full content (detailed analysis)
- AI sees 2 sources but in different detail levels
- Result: Uses summaries for context, details for focused content
```
**Not in Context:**
```
You: "I have 10 sources but only want 5 in context"
System:
- Paper A-E: In context (sent to AI)
- Paper F-J: Not in context (AI can't see them, doesn't search them)
- AI never knows these 5 sources exist
- Result: Tight, focused context
```
### Why This Matters
**Privacy**: You control what leaves your system
```
Scenario: Confidential company docs + public research
Control: Public research in context → Confidential docs excluded
Result: AI never sees confidential content
```
**Cost**: You control token usage
```
Scenario: 100 sources for background + 5 for detailed analysis
Control: Full content for 5 detailed, summaries for 95 background
Result: 80% lower token cost than sending everything
```
**Quality**: You control what the AI focuses on
```
Scenario: 20 sources, question requires deep analysis
Control: Full content for relevant source, exclude others
Result: AI doesn't get distracted; gives better answer
```
---
## The Difference: Chat vs. Ask
**IMPORTANT**: These use completely different approaches!
### Chat: Full-Content Context (NO RAG)
**How it works:**
```
YOU:
1. Select which sources to include in context
2. Set context level (full/summary/excluded)
3. Ask question
SYSTEM:
- Takes ALL selected sources (respecting context levels)
- Sends the ENTIRE content to the LLM at once
- NO search, NO retrieval, NO chunking
- AI sees everything you selected
AI:
- Responds based on the full content you provided
- Can reference any part of selected sources
- Conversational: context stays for follow-ups
```
**Use this when**:
- You know which sources are relevant
- You want conversational back-and-forth
- You want AI to see the complete context
- You're doing close reading or analysis
**Advantages:**
- Simple and transparent
- AI sees everything (no missed content)
- Conversational flow
**Limitations:**
- Limited by LLM context window
- You must manually select relevant sources
- Sends more tokens (higher cost with many sources)
---
### Ask: RAG - Automatic Retrieval
**How it works:**
```
YOU:
Ask one complex question
SYSTEM:
1. Analyzes your question
2. Searches across ALL your sources automatically
3. Finds relevant chunks using vector similarity
4. Retrieves only the most relevant pieces
5. Sends ONLY those chunks to the LLM
6. Synthesizes into comprehensive answer
AI:
- Sees ONLY the retrieved chunks (not full sources)
- Answers based on what was found to be relevant
- One-shot answer (not conversational)
```
**Use this when**:
- You have many sources and don't know which are relevant
- You want the AI to search automatically
- You need a comprehensive answer to a complex question
- You want to minimize tokens sent to LLM
**Advantages:**
- Automatic search (you don't pick sources)
- Works across many sources at once
- Cost-effective (sends only relevant chunks)
**Limitations:**
- Not conversational (single question/answer)
- AI only sees retrieved chunks (might miss context)
- Search quality depends on how well question matches content
---
## What This Means: Privacy by Design
Open Notebook's RAG approach gives you something you don't get with ChatGPT or Claude directly:
**You control the boundary between:**
- What stays private (on your system)
- What goes to AI (explicitly chosen)
- What the AI can see (context levels)
### The Audit Trail
Because everything is retrieved explicitly, you can ask:
- "Which sources did the AI use for this answer?" → See citations
- "What exactly did the AI see?" → See chunks in context level
- "Is the AI's claim actually in my sources?" → Verify citation
This prevents hallucinations or misrepresentation better than most systems.
---
## How Embeddings Work (Simplified)
The magic of semantic search comes from embeddings. Here's the intuition:
### The Idea
Instead of storing text, store it as a list of numbers (vectors) that represent "meaning."
```
Chunk: "The transformer uses attention mechanisms"
Vector: [0.23, -0.51, 0.88, 0.12, ..., 0.34]
(1536 numbers for OpenAI)
Another chunk: "Attention allows models to focus on relevant parts"
Vector: [0.24, -0.48, 0.87, 0.15, ..., 0.35]
(similar numbers = similar meaning!)
```
### Why This Works
Words that are semantically similar produce similar vectors. So:
- "alignment" and "interpretability" have similar vectors
- "transformer" and "attention" have related vectors
- "cat" and "dog" are more similar than "cat" and "radiator"
### How Search Works
```
Your question: "How do models understand their decisions?"
Question vector: [0.25, -0.50, 0.86, 0.14, ..., 0.33]
Compare to all stored vectors. Find the most similar:
- Chunk about interpretability: similarity 0.94
- Chunk about explainability: similarity 0.91
- Chunk about feature attribution: similarity 0.88
Return the top matches.
```
This is why semantic search finds conceptually similar content even when words are different.
---
## Key Design Decisions
### 1. Search, Don't Train
**Why?** Fine-tuning is slow and permanent. Search is flexible and reversible.
### 2. Explicit Retrieval, Not Implicit Knowledge
**Why?** You can verify what the AI saw. You have audit trails. You control what leaves your system.
### 3. Multiple Search Types
**Why?** Different questions need different search (keyword vs. semantic). Giving you both is more powerful.
### 4. Context as a Permission System
**Why?** Not everything you save needs to reach AI. You control granularly.
---
## Summary
Open Notebook gives you **two ways** to work with AI:
### Chat (Full-Content)
- Sends entire selected sources to LLM
- Manual control: you pick sources
- Conversational: back-and-forth dialog
- Transparent: you know exactly what AI sees
- Best for: focused analysis, close reading
### Ask (RAG)
- Searches and retrieves relevant chunks automatically
- Automatic: AI finds what's relevant
- One-shot: single comprehensive answer
- Efficient: sends only relevant pieces
- Best for: broad questions across many sources
**Both approaches:**
1. Keep your data private (doesn't leave your system by default)
2. Give you control (you choose which features to use)
3. Create audit trails (citations show what was used)
4. Support multiple AI providers
**Coming Soon**: The community is working on adding RAG capabilities to Chat as well, giving you the best of both worlds.
docs/2-CORE-CONCEPTS/chat-vs-transformations.md
+353
View File
@@ -0,0 +1,353 @@
# Chat vs. Ask vs. Transformations - Which Tool for Which Job?
Open Notebook offers different ways to work with your research. Understanding when to use each is key to using the system effectively.
---
## The Three Interaction Modes
### 1. CHAT - Conversational Exploration with Manual Context
**What it is:** Have a conversation with AI about selected sources.
**The flow:**
```
1. You select which sources to include ("in context")
2. You ask a question
3. AI responds using ONLY those sources
4. You ask follow-up questions (context stays same)
5. You change sources or context level, then continue
```
**Context management:** You explicitly choose which sources the AI can see.
**Conversational:** Multiple questions with shared history.
**Example:**
```
You: [Select sources: "paper1.pdf", "research_notes.txt"]
[Set context: Full content for paper1, Summary for notes]
You: "What's the main argument in these sources?"
AI: "Paper 1 argues X [citation]. Your notes emphasize Y [citation]."
You: "How do they differ?"
AI: "Paper 1 focuses on X [citation], while your notes highlight Y [citation]..."
You: [Now select different sources]
You: "Compare to this other perspective"
AI: "This new source takes a different approach..."
```
**Best for:**
- Exploring a focused topic with specific sources
- Having a dialogue (multiple back-and-forth questions)
- When you know which sources matter
- When you want tight control over what goes to AI
---
### 2. ASK - Automated Comprehensive Search
**What it is:** Ask one complex question, system automatically finds relevant content.
**The flow:**
```
1. You ask a comprehensive question
2. System analyzes the question
3. System automatically searches your sources
4. System retrieves relevant chunks
5. System synthesizes answer from all results
6. You get one detailed answer (not conversational)
```
**Context management:** Automatic. System figures out what's relevant.
**Non-conversational:** One question → one answer. No follow-ups.
**Example:**
```
You: "How do these papers compare their approaches to alignment?
What does each one recommend?"
System:
- Breaks down the question into search strategies
- Searches all sources for alignment approaches
- Searches all sources for recommendations
- Retrieves top 10 relevant chunks
- Synthesizes: "Paper A recommends X [citation].
Paper B recommends Y [citation].
They differ in Z."
You: [Get back one comprehensive answer]
[If you want to follow up, use Chat instead]
```
**Best for:**
- Comprehensive, one-time questions
- Comparing multiple sources at once
- When you want the system to decide what's relevant
- Complex questions that need multiple search angles
- When you don't need a back-and-forth conversation
---
### 3. TRANSFORMATIONS - Template-Based Processing
**What it is:** Apply a reusable template to a source and get structured output.
**The flow:**
```
1. You define a transformation (or choose a preset)
"Extract: main argument, methodology, limitations"
2. You apply it to ONE source at a time
(You can repeat for other sources)
3. For the source:
- Source content + transformation prompt → AI
- Result stored as new insight/note
4. You get back
- Structured output (main argument, methodology, limitations)
- Saved as a note in your notebook
```
**Context management:** Works on one source at a time.
**Reusable:** Apply the same template to different sources (one by one).
**Note**: Currently processes one source at a time. Batch processing (multiple sources at once) is planned for a future release.
**Example:**
```
You: Define transformation
"For each academic paper, extract:
- Main research question
- Methodology used
- Key findings
- Limitations and gaps
- Recommended next research"
You: Apply to paper 1
System:
- Runs the transformation on paper 1
- Result stored as new note
You: Apply same transformation to paper 2, 3, etc.
After 10 papers:
- You have 10 structured notes with consistent format
- Perfect for writing a literature review or comparison
```
**Best for:**
- Extracting the same information from each source (run repeatedly)
- Creating structured summaries with consistent format
- Building a knowledge base of categorized insights
- When you want reusable templates you can apply to each source
---
## Decision Tree: Which Tool to Use?
```
What are you trying to do?
│
├─→ "I want to have a conversation about this topic"
│ └─→ Is the conversation exploratory or fixed?
│ ├─→ Exploratory (I'll ask follow-ups)
│ │ └─→ USE: CHAT
│ │
│ └─→ Fixed (One question → done)
│ └─→ Go to next question
│
├─→ "I need to compare these sources or get a comprehensive answer"
│ └─→ USE: ASK
│
├─→ "I want to extract the same info from each source (one at a time)"
│ └─→ USE: TRANSFORMATIONS (apply to each source)
│
└─→ "I just want to read and search"
└─→ USE: Search (text or vector)
OR read your notes
```
---
## Side-by-Side Comparison
| Aspect | CHAT | ASK | TRANSFORMATIONS |
|--------|------|-----|-----------------|
| **What's it for?** | Conversational exploration | Comprehensive Q&A | Template-based extraction |
| **# of questions** | Multiple (conversational) | One | One template per source |
| **Context control** | Manual (you choose) | Automatic (system searches) | One source at a time |
| **Conversational?** | Yes (follow-ups work) | No (one question only) | No (single operation) |
| **Output** | Natural conversation | Natural answer | Structured note |
| **Time** | Quick (back-and-forth) | Longer (comprehensive) | Per source |
| **Best when** | Exploring & uncertain | Need full picture | Want consistent format |
| **Model speed** | Any | Fast preferred | Any |
---
## Workflow Examples
### Example 1: Academic Research
```
Goal: Write literature review from 15 papers
Step 1: TRANSFORMATIONS
- Define: "Extract abstract, methodology, findings, relevance"
- Apply to paper 1 → get structured note
- Apply to paper 2 → get structured note
- ... repeat for all 15 papers
- Result: 15 structured notes with consistent format
Step 2: Read the notes
- Now you have consistent summaries
Step 3: CHAT or ASK
- Chat: "Help me organize these by theme"
- Ask: "What are the common methodologies across these papers?"
Step 4: Write your review
- Use the transformations as foundation
- Use chat/ask insights for structure
```
### Example 2: Product Research
```
Goal: Understand customer feedback from interviews
Step 1: Add sources (interview transcripts)
Step 2: ASK
- "What are the top 10 pain points mentioned?"
- Get comprehensive answer with citations
Step 3: CHAT
- "Can you help me group these by severity?"
- Continue conversation to prioritize
Step 4: TRANSFORMATIONS (optional)
- Define: "Extract: pain point, frequency, who mentioned it"
- Apply to each interview (one by one)
- Get structured data for analysis
```
### Example 3: Policy Analysis
```
Goal: Compare policy documents
Step 1: Add all policy documents as sources
Step 2: ASK
- "How do these policies differ on climate measures?"
- System searches all docs, gives comprehensive comparison
Step 3: CHAT (if needed)
- "Which policy is most aligned with X goals?"
- Have discussion about trade-offs
Step 4: Export notes
- Save AI responses as notes for reports
```
---
## Context Management: The Control Panel
All three modes let you control what the AI sees.
### In CHAT and TRANSFORMATIONS
```
You choose:
- Which sources to include
- Context level for each:
✓ Full Content (send complete text)
✓ Summary Only (send AI summary, not full text)
✓ Not in Context (exclude entirely)
Example:
Paper A: Full Content (analyzing closely)
Paper B: Summary Only (background)
Paper C: Not in Context (confidential)
```
### In ASK
```
Context is automatic:
- System searches ALL your sources
- Retrieves most relevant chunks
- Sends those to AI
But you can:
- Search in specific notebook
- Filter by source type
- Use the results to decide context for follow-up Chat
```
---
## Model Selection
Each mode works with different models:
### CHAT
- **Any model** works fine
- Fast models (GPT-4o mini, Claude Haiku): Quick responses, good for conversation
- Powerful models (GPT-4o, Claude Sonnet): Better reasoning, better for complex topics
### ASK
- **Fast models preferred** (because it processes multiple searches)
- Can use powerful models if you want deep synthesis
- Example: GPT-4 for strategy planning, GPT-4o-mini for quick facts
### TRANSFORMATIONS
- **Any model** works
- Fast models (cost-effective for batch processing)
- Powerful models (better quality extractions)
---
## Advanced: Chaining Modes Together
You can combine these modes:
```
TRANSFORMATIONS → CHAT
1. Use transformations to extract structured data
2. Use chat to discuss the results
ASK → TRANSFORMATIONS
1. Use Ask to understand what matters
2. Use Transformations to extract it from remaining sources
CHAT → Save as Note → TRANSFORMATIONS
1. Have conversation (Chat)
2. Save good responses as notes
3. Use those notes as context for transformations
```
---
## Summary: When to Use Each
| Situation | Use | Why |
|-----------|-----|-----|
| "I want to explore a topic with follow-up questions" | **CHAT** | Conversational, you control context |
| "I need a comprehensive answer to one complex question" | **ASK** | Automatic search, synthesized answer |
| "I want consistent summaries from each source" | **TRANSFORMATIONS** | Template reuse, apply to each source |
| "I'm comparing two specific sources" | **CHAT** | Select just those 2, have discussion |
| "I need to categorize each source by X criteria" | **TRANSFORMATIONS** | Extract category from each source |
| "I want to understand the big picture across all sources" | **ASK** | Automatic comprehensive search |
| "I want to build a knowledge base" | **TRANSFORMATIONS** | Create structured note from each source |
| "I want to iterate on understanding" | **CHAT** | Multiple questions, refine thinking |
The key insight: **Different questions need different tools.** Open Notebook gives you all three because research rarely fits one mode.
docs/2-CORE-CONCEPTS/index.md
+70
View File
@@ -0,0 +1,70 @@
# Core Concepts - Understand the Mental Model
Before diving into how to use Open Notebook, it's important to understand **how it thinks**. These core concepts explain the "why" behind the design.
## The Five Mental Models
### 1. [Notebooks, Sources, and Notes](notebooks-sources-notes.md)
How Open Notebook organizes your research. Understand the three-tier container structure and how information flows from raw materials to finished insights.
**Key idea**: A notebook is a scoped research container. Sources are inputs (PDFs, URLs, etc.). Notes are outputs (your insights, AI-generated summaries, captured responses).
---
### 2. [AI Context & RAG](ai-context-rag.md)
How Open Notebook makes AI aware of your research - two different approaches.
**Key idea**: **Chat** sends entire selected sources to the LLM (full context, conversational). **Ask** uses RAG (retrieval-augmented generation) to automatically search and retrieve only relevant chunks. Different tools for different needs.
---
### 3. [Chat vs. Transformations](chat-vs-transformations.md)
Why Open Notebook has different interaction modes and when to use each one.
**Key idea**: Chat is conversational exploration (you control context). Transformations are insight extractions. They reduced content to smaller bits of concentrated/dense information, which is much more suitable for an AI to use.
---
### 4. [Context Management](chat-vs-transformations.md#context-management-the-control-panel)
Your control panel for privacy and cost. Decide what data actually reaches AI.
**Key idea**: You choose three levels—not in context (private), summary only (condensed), or full content (complete access). This gives you fine-grained control.
---
### 5. [Podcasts Explained](podcasts-explained.md)
Why Open Notebook can turn research into audio and why this matters.
**Key idea**: Podcasts transform your research into a different consumption format. Instead of reading, someone can listen and absorb your insights passively.
---
## Read This Section If:
- **You're new to Open Notebook** — Start here to understand how the system works conceptually before learning the features
- **You're confused about Chat vs Ask** — Section 2 explains the difference (full-content vs RAG)
- **You're wondering when to use Chat vs Transformations** — Section 3 clarifies the differences
- **You want to understand privacy controls** — Section 4 shows you what you can control
- **You're curious about podcasts** — Section 5 explains the architecture and why it's different from competitors
---
## The Big Picture
Open Notebook is built on a simple insight: **Your research deserves to stay yours**.
That means:
- **Privacy by default** — Your data doesn't leave your infrastructure unless you explicitly choose
- **AI as a tool, not a gatekeeper** — You decide which sources the AI sees, not the AI deciding for you
- **Flexible consumption** — Read, listen, search, chat, or transform your research however makes sense
These core concepts explain how that works.
---
## Next Steps
1. **Just want to use it?** → Go to [User Guide](../3-USER-GUIDE/index.md)
2. **Want to understand it first?** → Read the 5 sections above (15 min)
3. **Setting up for the first time?** → Go to [Installation](../1-INSTALLATION/index.md)
docs/2-CORE-CONCEPTS/notebooks-sources-notes.md
+284
View File
@@ -0,0 +1,284 @@
# Notebooks, Sources, and Notes - The Container Model
Open Notebook organizes research in three connected layers. Understanding this hierarchy is key to using the system effectively.
## The Three-Layer Structure
```
┌─────────────────────────────────────┐
│ NOTEBOOK (The Container) │
│ "My AI Safety Research 2026" │
├─────────────────────────────────────┤
│ │
│ SOURCES (The Raw Materials) │
│ ├─ safety_paper.pdf │
│ ├─ alignment_video.mp4 │
│ └─ prompt_injection_article.html │
│ │
│ NOTES (The Processed Insights) │
│ ├─ AI Summary (auto-generated) │
│ ├─ Key Concepts (transformation) │
│ ├─ My Research Notes (manual) │
│ └─ Chat Insights (from conversation)
│ │
└─────────────────────────────────────┘
```
---
## 1. NOTEBOOKS - The Research Container
### What Is a Notebook?
A **notebook** is a *scoped container* for a research project or topic. It's your research workspace.
Think of it like a physical notebook: everything inside is about the same topic, shares the same context, and builds toward the same goals.
### What Goes In?
- **A description** — "This notebook collects research on X topic"
- **Sources** — The raw materials you add
- **Notes** — Your insights and outputs
- **Conversation history** — Your chats and questions
### Why This Matters
**Isolation**: Each notebook is completely separate. Sources in Notebook A never appear in Notebook B. This lets you:
- Keep different research topics completely isolated
- Reuse source names across notebooks without conflicts
- Control which AI context applies to which research
**Shared Context**: All sources and notes in a notebook inherit the notebook's context. If your notebook is titled "AI Safety 2026" with description "Focusing on alignment and interpretability," that context applies to all AI interactions within that notebook.
**Parallel Projects**: You can have 10 notebooks running simultaneously. Each one is its own isolated research environment.
### Example
```
Notebook: "Customer Research - Product Launch"
Description: "User interviews and feedback for Q1 2026 launch"
→ All sources added to this notebook are about customer feedback
→ All notes generated are in that context
→ When you chat, the AI knows you're analyzing product launch feedback
→ Different from your "Market Analysis - Competitors" notebook
```
---
## 2. SOURCES - The Raw Materials
### What Is a Source?
A **source** is a *single piece of input material* — the raw content you bring in. Sources never change; they're just processed and indexed.
### What Can Be a Source?
- **PDFs** — Research papers, reports, documents
- **Web links** — Articles, blog posts, web pages
- **Audio files** — Podcasts, interviews, lectures
- **Video files** — Tutorials, presentations, recordings
- **Plain text** — Notes, transcripts, passages
- **Uploaded text** — Paste content directly
### What Happens When You Add a Source?
```
1. EXTRACTION
File/URL → Extract text and metadata
(OCR for PDFs, web scraping for URLs, speech-to-text for audio)
2. CHUNKING
Long text → Break into searchable chunks
(Prevents "too much context" in single query)
3. EMBEDDING
Each chunk → Generate semantic vector
(Allows AI to find conceptually similar content)
4. STORAGE
Chunks + vectors → Store in database
(Ready for search and retrieval)
```
### Key Properties
**Immutable**: Once added, the source doesn't change. If you need a new version, add it as a new source.
**Indexed**: Sources are automatically indexed for search (both text and semantic).
**Scoped**: A source belongs to exactly one notebook.
**Referenceable**: Other sources and notes can reference this source by citation.
### Example
```
Source: "openai_charter.pdf"
Type: PDF document
What happens:
→ PDF is uploaded
→ Text is extracted (including images)
→ Text is split into 50 chunks (paragraphs, sections)
→ Each chunk gets an embedding vector
→ Now searchable by: "OpenAI's approach to safety"
```
---
## 3. NOTES - The Processed Insights
### What Is a Note?
A **note** is a *processed output* — something you created or AI created based on your sources. Notes are the "results" of your research work.
### Types of Notes
#### Manual Notes
You write them yourself. They're your original thinking, capturing:
- What you learned from sources
- Your analysis and interpretations
- Your next steps and questions
#### AI-Generated Notes
Created by applying AI processing to sources:
- **Transformations** — Structured extraction (main points, key concepts, methodology)
- **Chat Responses** — Answers you saved from conversations
- **Ask Results** — Comprehensive answers saved to your notebook
#### Captured Insights
Notes you explicitly saved from interactions:
- "Save this response as a note"
- "Save this transformation result"
- Convert any AI output into a permanent note
### What Can Notes Contain?
- **Text** — Your writing or AI-generated content
- **Citations** — References to specific sources
- **Metadata** — When created, how created (manual/AI), which sources influenced it
- **Tags** — Your categorization (optional but useful)
### Why Notes Matter
**Knowledge Accumulation**: Notes become your actual knowledge base. They're what you take away from the research.
**Searchable**: Notes are searchable along with sources. "Find everything about X" includes your notes, not just sources.
**Citable**: Notes can cite sources, creating an audit trail of where insights came from.
**Shareable**: Notes are your outputs. You can share them, publish them, or build on them in other projects.
---
## How They Connect: The Data Flow
```
YOU
│
├─→ Create Notebook ("AI Research")
│
├─→ Add Sources (papers, articles, videos)
│ └─→ System: Extract, embed, index
│
├─→ Search Sources (text or semantic)
│ └─→ System: Find relevant chunks
│
├─→ Apply Transformations (extract insights)
│ └─→ Creates Notes
│
├─→ Chat with Sources (explore with context control)
│ ├─→ Can save responses as Notes
│ └─→ Notes include citations
│
├─→ Ask Questions (automated comprehensive search)
│ ├─→ Can save results as Notes
│ └─→ Notes include citations
│
└─→ Generate Podcast (transform notebook into audio)
└─→ Uses all sources + notes for content
```
---
## Key Design Decisions
### 1. One Notebook Per Source
Each source belongs to exactly one notebook. This creates clear boundaries:
- No ambiguity about which research project a source is in
- Easy to isolate or export a complete project
- Clean permissions model (if someone gets access to notebook, they get access to all its sources)
### 2. Immutable Sources, Mutable Notes
Sources never change (once added, always the same). But notes can be edited or deleted. Why?
- Sources are evidence → evidence shouldn't be altered
- Notes are your thinking → thinking evolves as you learn
### 3. Explicit Context Control
Sources don't automatically go to AI. You decide which sources are "in context" for each interaction:
- Chat: You manually select which sources to include
- Ask: System automatically figures out which sources to search
- Transformations: You choose which sources to transform
This is different from systems that always send everything to AI.
---
## Mental Models Explained
### Notebook as Boundaries
Think of a notebook like a Git repository:
- Everything in it is about the same topic
- You can clone/fork it (copy to new project)
- It has clear entry/exit points
- You know exactly what's included
### Sources as Evidence
Think of sources like exhibits in a legal case:
- Once filed, they don't change
- They can be cited and referenced
- They're the ground truth for what you're basing claims on
- Multiple sources can be cross-referenced
### Notes as Synthesis
Think of notes like your case brief:
- You write them based on evidence
- They're your interpretation
- You can cite which evidence supports each claim
- They're what you actually share or act on
---
## Common Questions
### Can I move a source to a different notebook?
Not directly. Each source is tied to one notebook. If you want it in multiple notebooks, add it again (uploads are fast if it's already processed).
### Can a note reference sources from a different notebook?
No. Notes stay within their notebook and reference sources within that notebook. This keeps boundaries clean.
### What if I want to group sources within a notebook?
Use tags. You can tag sources ("primary research," "background," "methodology") and filter by tags.
### Can I merge two notebooks?
Not built-in, but you can manually copy sources from one notebook to another by re-uploading them.
---
## Summary
| Concept | Purpose | Lifecycle | Scope |
|---------|---------|-----------|-------|
| **Notebook** | Container + context | Create once, configure | All its sources + notes |
| **Source** | Raw material | Add → Process → Store | One notebook |
| **Note** | Processed output | Create/capture → Edit → Share | One notebook |
This three-layer model gives you:
- **Clear organization** (everything scoped to projects)
- **Privacy control** (isolated notebooks)
- **Audit trails** (notes cite sources)
- **Flexibility** (notes can be manual or AI-generated)
docs/2-CORE-CONCEPTS/podcasts-explained.md
+426
View File
@@ -0,0 +1,426 @@
# Podcasts Explained - Research as Audio Dialogue
Podcasts are Open Notebook's highest-level transformation: converting your research into audio dialogue for a different consumption pattern.
---
## Why Podcasts Matter
### The Problem
Research naturally accumulates as text: PDFs, articles, web pages, notes. This creates a friction point:
**To consume research, you must:**
- Sit down at a desk
- Focus intently
- Read actively
- Take notes
- Set aside dedicated time
**But much of life is passive time:**
- Commuting
- Exercising
- Doing dishes
- Driving
- Walking
- Idle moments
### The Solution
Convert your research into audio dialogue so you can consume it passively.
```
Before (Text-based):
Research pile → Must schedule reading time → Requires focus
After (Podcast):
Research pile → Podcast → Can listen while commuting
→ Absorb while exercising
→ Understand while walking
→ Engage without screen time
```
---
## What Makes It Special: Open Notebook vs. Competitors
### Google Notebook LM Podcasts
- **Fixed format**: 2 hosts, always conversational
- **Limited customization**: You can't choose who the "hosts" are
- **One TTS voice per speaker**: Can't customize voices
- **Only uses cloud services**: No local options
### Open Notebook Podcasts
- **Customizable format**: 1-4 speakers, you design them
- **Rich speaker profiles**: Create personas with backstories and expertise
- **Multiple TTS options**:
- OpenAI (natural, fast)
- Google TTS (high quality)
- ElevenLabs (beautiful voices, accents)
- Local TTS (privacy-first, no API calls)
- **Async generation**: Doesn't block your work
- **Full control**: Choose outline structure, tone, depth
---
## How Podcast Generation Works
### Stage 1: Content Selection
You choose what goes into the podcast:
```
Notebook content → Which sources? → Which notes?
→ Which topics to focus on?
→ Depth of coverage?
```
### Stage 2: Episode Profile
You define how you want the podcast structured:
```
Episode Profile
├─ Topic: "AI Safety Approaches"
├─ Length: 20 minutes
├─ Tone: Academic but accessible
├─ Format: Debate (2 speakers with opposing views)
├─ Audience: Researchers new to the field
└─ Focus areas: Main approaches, pros/cons, open questions
```
### Stage 3: Speaker Configuration
You create speaker personas (1-4 speakers):
```
Speaker 1: "Expert Alex"
├─ Expertise: "Deep knowledge of alignment research"
├─ Personality: "Rigorous, academic, patient with explanation"
├─ Accent: (Optional) "British English"
└─ Voice Model: Selected from model registry (e.g., OpenAI TTS)
└─ Optional per-speaker override of the episode's default voice model
Speaker 2: "Researcher Sam"
├─ Expertise: "Field observer, pragmatic perspective"
├─ Personality: "Curious, asks clarifying questions"
├─ Accent: "American English"
└─ Voice Model: Selected from model registry (e.g., ElevenLabs TTS)
```
### Stage 4: Outline Generation
System generates episode outline:
```
EPISODE: "AI Safety Approaches"
1. Introduction (2 min)
Alex: Introduces topic and speakers
Sam: What will we cover today?
2. Main Approaches (8 min)
Alex: Explains top 3 approaches
Sam: Asks about tradeoffs
3. Debate: Best approach? (6 min)
Alex: Advocates for approach A
Sam: Argues for approach B
4. Open Questions (3 min)
Both: What's unsolved?
5. Conclusion (1 min)
Recap and where to learn more
```
### Stage 5: Dialogue Generation
System generates dialogue based on outline:
```
Alex: "Today we're exploring three major approaches to AI alignment..."
Sam: "That's a great start. Can you break down what we mean by alignment?"
Alex: "Good question. Alignment means ensuring AI systems pursue the goals
we actually want them to pursue, not just what we literally asked for.
There's a classic example of a paperclip maximizer..."
Sam: "Interesting. So it's about solving the intention problem?"
Alex: "Exactly. And that's where the three approaches come in..."
```
### Stage 6: Text-to-Speech
System converts dialogue to audio using the voice models configured in the model registry. Credentials are automatically resolved from each model's configuration.
```
Alex's text → Voice model (from registry) → Alex's voice (audio file)
Sam's text → Voice model (from registry) → Sam's voice (audio file)
Audio files → Mix together → Final podcast MP3
```
---
## When Things Go Wrong: Failures & Retry
Podcast generation involves multiple steps (outline, transcript, TTS) and depends on external AI providers. Sometimes things fail.
### What Happens on Failure
When podcast generation fails (e.g., wrong model configured, API key expired, provider outage):
- The episode is marked as **Failed** with a red badge
- The **error message** from the AI provider is displayed so you can understand what went wrong
- No duplicate episodes are created — automatic retries are disabled to prevent confusion
### How to Retry a Failed Episode
1. Go to the podcast's **Episodes** tab
2. Find the failed episode — it shows a red "FAILED" badge and an error details box
3. Click the **Retry** button
4. The failed episode is deleted and a new generation job is submitted
5. The new episode appears with "pending" status
### Common Failure Causes
| Error | What to Do |
|-------|-----------|
| Invalid API key | Check Settings -> Credentials for the TTS and language model providers |
| Model not found | Verify the model exists in the model registry and has valid credentials configured |
| Rate limit exceeded | Wait a few minutes and retry |
| Provider unavailable | Check provider status page; retry later |
---
## Key Architecture Decisions
### 1. Asynchronous Processing
Podcasts are generated in the background. You upload → system processes → you download when ready.
**Why?** Podcast generation takes time (10+ minutes for a 30-minute episode). Blocking would lock up your interface.
### 2. Multi-Speaker Support
Unlike Google Notebook LM (always 2 hosts), you choose 1-4 speakers.
**Why?** Different discussions work better with different formats:
- Expert monologue (1 speaker)
- Interview (2 speakers: host + expert)
- Debate (2 speakers: opposing views)
- Panel discussion (3-4 speakers: different expertise)
### 3. Speaker Customization
You create rich speaker profiles, not just "Host A" and "Host B".
**Why?** Makes podcasts more engaging and authentic. Different speakers bring different perspectives.
### 4. Multiple TTS Providers
You're not locked into one voice provider.
**Why?**
- Cost optimization (some providers cheaper)
- Quality preferences (some voices more natural)
- Privacy options (local TTS for sensitive content)
- Accessibility (different accents, genders, styles)
### 5. Local TTS Option
Can generate podcasts entirely offline with local text-to-speech.
**Why?** For sensitive research, never send audio to external APIs.
---
## Use Cases Show Why This Matters
### Academic Publishing
```
Traditional: Academic paper → PDF
Problem: Hard to consume, linear reading required
Open Notebook:
Research materials → Podcast (expert explaining methodology)
→ Podcast (debate format: different interpretations)
→ Different consumption for different audiences
```
### Content Creation
```
Blog creator: Has research pile on a topic
Problem: Doesn't have time to write the article
Solution:
Add research → Create podcast → Transcribe → Becomes article
OR: Podcast BECOMES the content (upload to podcast platforms)
```
### Educational Content
```
Educator: Has reading materials for a course
Problem: Students don't read the papers
Solution:
Create podcast with expert explaining papers
Students listen → Better engagement → Discussions can reference podcast
```
### Market Research
```
Product manager: Has interviews with customers
Problem: Too many hours of audio to review
Solution:
Create podcast with debate format (customer perspective vs. team perspective)
Much more engaging than raw transcripts
```
### Knowledge Transfer
```
Domain expert: Leaving the organization
Problem: How to preserve expertise?
Solution:
Create expert-mode podcast explaining frameworks, decision-making, context
New team member listens, gets context faster than reading 100 documents
```
---
## The Difference: Active vs. Passive Learning
### Text-Based Research (Active)
- **Effort**: High (must focus, read, synthesize)
- **When**: Dedicated study time
- **Cost**: Time is expensive (can't multitask)
- **Best for**: Deep dives, precise information
- **Format**: Whatever you write (notes, articles, books)
### Audio Podcast (Passive)
- **Effort**: Low (just listen)
- **When**: Anywhere, anytime
- **Cost**: Low (can multitask)
- **Best for**: Overview, context, exploration
- **Format**: Dialogue (more engaging than narration)
**They complement each other:**
1. **First encounter**: Listen to podcast (passive, get context)
2. **Deep dive**: Read source materials (active, precise)
3. **Mastery**: Both together (understand big picture + details)
---
## How Podcasts Fit Into Your Workflow
```
1. Build notebook (add sources)
↓
2. Apply transformations (extract insights)
↓
3. Chat/Ask (explore content)
↓
4. Decide on podcast
├─→ Create speaker profiles
├─→ Define episode profile
├─→ Configure voice models (from model registry)
└─→ Generate podcast
↓
5. Listen while commuting/exercising
↓
6. Reference sources for deep dive
↓
7. Repeat for different formats/speakers/focus
```
---
## Advanced: Multiple Podcasts from Same Research
You can create different podcasts from the same sources:
### Example: AI Safety Research
```
Podcast 1: "Expert Monologue"
Speaker: Researcher explaining field
Format: Educational, comprehensive
Audience: Students new to field
Podcast 2: "Debate Format"
Speakers: Optimist vs. skeptic
Format: Discussion of tradeoffs
Audience: Advanced researchers
Podcast 3: "Interview Format"
Speakers: Journalist + expert
Format: Q&A about practical applications
Audience: Industry practitioners
```
Each tells the same story from different angles.
---
## Privacy & Data Considerations
### Where Your Data Goes
**Option 1: Cloud TTS (Faster, Higher Quality)**
```
Your outline → API call to TTS provider
→ Audio returned
→ Stored in your notebook
Provider sees: Your outlined script (not raw sources)
Privacy level: Medium (outline is shared, sources aren't)
```
**Option 2: Local TTS (Slower, Maximum Privacy)**
```
Your outline → Local TTS engine (runs on your machine)
→ Audio generated locally
→ Stored in your notebook
Provider sees: Nothing
Privacy level: Maximum (everything local)
```
### Recommendation
- **Sensitive research**: Use local TTS, no API calls
- **Less sensitive**: Use ElevenLabs or Google (both handle audio data professionally)
- **Mixed**: Use local TTS for speakers reading sensitive content
---
## Cost Considerations
### Cloud TTS Costs
| Provider | Cost | Quality | Speed |
|----------|------|---------|-------|
| OpenAI | ~$0.015 per minute | Good | Fast |
| Google | ~$0.004 per minute | Excellent | Fast |
| ElevenLabs | ~$0.10 per minute | Exceptional | Medium |
| Local TTS | Free | Basic | Slow |
A 30-minute podcast costs:
- OpenAI: ~$0.45
- Google: ~$0.12
- ElevenLabs: ~$3.00
- Local: Free (but slow)
---
## Summary: Why Podcasts Are Special
**Podcasts transform your research consumption:**
| Aspect | Text | Podcast |
|--------|------|---------|
| **How consumed?** | Active reading | Passive listening |
| **Where consumed?** | Desk | Anywhere |
| **Multitasking** | Hard | Easy |
| **Time commitment** | Scheduled | Flexible |
| **Format** | Whatever | Natural dialogue |
| **Engagement** | Academic | Conversational |
| **Accessibility** | Text-based | Audio-based |
**In Open Notebook specifically:**
- **Full customization** — you create speakers and format
- **Privacy options** — local TTS for sensitive content
- **Cost control** — choose TTS provider based on budget
- **Non-blocking** — generates in background
- **Multiple versions** — create different podcasts from same research
This is why podcasts matter: they change *when* and *how* you can consume your research.
docs/3-USER-GUIDE/adding-sources.md
+429
View File
@@ -0,0 +1,429 @@
# Adding Sources - Getting Content Into Your Notebook
Sources are the raw materials of your research. This guide covers how to add different types of content.
---
## Quick-Start: Add Your First Source
### Option 1: Upload a File (PDF, Word, etc.)
```
1. In your notebook, click "Add Source"
2. Select "Upload File"
3. Choose a file from your computer
4. Click "Upload"
5. Wait 30-60 seconds for processing
6. Done! Source appears in your notebook
```
### Option 2: Add a Web Link
```
1. Click "Add Source"
2. Select "Web Link"
3. Paste URL: https://example.com/article
4. Click "Add"
5. Wait for processing (usually faster than files)
6. Done!
```
### Option 3: Paste Text
```
1. Click "Add Source"
2. Select "Text"
3. Paste or type your content
4. Click "Save"
5. Done! Immediately available
```
---
## Supported File Types
### Documents
- **PDF** (.pdf) — Best support, including scanned PDFs with OCR
- **Word** (.docx, .doc) — Full support
- **PowerPoint** (.pptx) — Slides converted to text
- **Excel** (.xlsx, .xls) — Spreadsheet data
- **EPUB** (.epub) — eBook files
- **Markdown** (.md, .txt) — Plain text formats
- **HTML** (.html, .htm) — Web page files
**File size limits:** Up to ~100MB (varies by system)
**Processing time:** 10 seconds - 2 minutes (depending on length and file type)
### Audio & Video
- **Audio**: MP3, WAV, M4A, OGG, FLAC (~30 seconds - 3 minutes per hour)
- **Video**: MP4, AVI, MOV, MKV, WebM (~3-10 minutes per hour)
- **YouTube**: Direct URL support
- **Podcasts**: RSS feed URL
**Automatic transcription**: Audio/video is transcribed to text automatically. This requires enabling speech-to-text in settings.
### Web Content
- **Articles**: Blog posts, news articles, Medium
- **YouTube**: Full videos or playlists
- **PDFs online**: Direct PDF links
- **News**: News site articles
**Just paste the URL** in "Web Link" section.
### What Doesn't Work
- Paywalled content (WSJ, FT, etc.) — Can't extract
- Password-protected PDFs — Can't open
- Pure image files (.jpg, .png) — Except scanned PDFs which have OCR
- Very large files (>100MB) — Timeout
---
## What Happens When You Add a Source
The system automatically does four things:
```
1. EXTRACT TEXT
File/URL → Readable text
(PDFs get OCR if scanned)
(Videos get transcribed if enabled)
2. BREAK INTO CHUNKS
Long text → ~500-word pieces
(So search finds specific parts, not whole document)
3. CREATE EMBEDDINGS
Each chunk → Vector representation
(Enables semantic/concept search)
4. INDEX & STORE
Everything → Database
(Ready to search and retrieve)
```
**Time to use:** After the progress bar completes, the source is ready immediately. Embeddings are created in the background.
---
## Step-by-Step for Different Types
### PDFs
**Best practices:**
```
Clean PDFs:
1. Upload → Done
2. Processing time: ~30-60 seconds
Scanned/Image PDFs:
1. Upload same way
2. System auto-detects and uses OCR
3. Processing time: ~2-3 minutes
4. (Higher, due to OCR overhead)
Large PDFs (50+ pages):
1. Consider splitting into smaller files
2. Or upload as-is (system handles it)
3. Processing time scales with size
```
**Common issues:**
- "Can't extract text" → PDF is corrupted or has copy protection
- Solution: Try opening in Adobe. If it won't, the PDF is likely protected.
### Web Links / Articles
**Best practices:**
```
1. Copy full URL from browser: https://example.com/article-title
2. Paste in "Web Link"
3. Click Add
4. Wait for extraction
Processing time: Usually 5-15 seconds
```
**What works:**
- Standard web articles
- Blog posts
- News articles
- Wikipedia pages
- Medium posts
- Substack articles
**What doesn't work:**
- Twitter threads (unreliable)
- Paywalled articles (can't access)
- JavaScript-heavy sites (content not extracted)
**Pro tip:** If it doesn't work, copy the article text and paste as "Text" instead.
### Audio Files
**Best practices:**
```
1. Ensure speech-to-text is enabled in Settings
2. Upload MP3, WAV, or M4A file
3. System automatically transcribes to text
4. Processing time: ~1 minute per 5 minutes of audio
Example:
- 1-hour podcast → 12 minutes processing
- 10-minute recording → 2 minutes processing
```
**Quality matters:**
- Clear audio: Fast transcription
- Muffled/noisy audio: Slower, less accurate transcription
- Background noise: Try to minimize before uploading
**Tip:** If audio quality is poor, the AI might misinterpret content. You can manually correct transcription if needed.
### YouTube Videos
**Best practices:**
```
Two ways to add:
Method 1: Direct URL
1. Copy YouTube URL: https://www.youtube.com/watch?v=...
2. Paste in "Web Link"
3. Click Add
4. System extracts captions (if available) + transcript
Method 2: Playlist
1. Paste playlist URL
2. System adds all videos as separate sources
3. Each video processed separately
4. Takes longer (multiple videos)
```
**What's extracted:**
- Captions/subtitles (if available)
- Transcription (if captions aren't available)
- Basic metadata (title, channel, length)
**Processing:**
- 10-minute video: ~2-3 minutes
- 1-hour video: ~10-15 minutes
### Text / Paste Content
**Best practices:**
```
1. Select "Text" when adding source
2. Paste or type content
3. System processes immediately
4. No wait time needed
Good for:
- Notes you want to reference
- Quotes from books
- Transcripts you have handy
- Quick research snippets
```
---
## Managing Your Sources
### Viewing Source Details
```
Click on source → See:
- Original file name/title
- When it was added
- Size and format
- Processing status
- Number of chunks
```
### Organizing with Metadata
You can add to each source:
- **Title**: Better name than original filename
- **Tags**: Category labels ("primary research", "background", "competitor analysis")
- **Description**: A few notes about what it contains
**Why this matters:**
- Makes sources easier to find
- Helps when contextualizing for Chat
- Useful for organizing large notebooks
### Searching Within Sources
```
After sources are added, you can:
Text search: "Find exact phrase"
Vector search: "Find conceptually similar"
Both search across all sources in notebook.
Results show:
- Which source
- Which section
- Relevance score
```
---
## Context Management: How Sources Get Used
You control how AI accesses sources:
### Three Levels (for Chat)
**Full Content:**
```
AI sees: Complete source text
Cost: 100% of tokens
Use when: Analyzing in detail, need precise citations
Example: "Analyze this methodology paper closely"
```
**Summary Only:**
```
AI sees: AI-generated summary (not full text)
Cost: ~10-20% of tokens
Use when: Background material, reference context
Example: "Use this as context but focus on the main source"
```
**Not in Context:**
```
AI sees: Nothing (excluded)
Cost: 0 tokens
Use when: Confidential, not relevant, or archived
Example: "Keep this in notebook but don't use in this conversation"
```
### How to Set Context (in Chat)
```
1. Go to Chat
2. Click "Select Context Sources"
3. For each source:
- Toggle ON/OFF (include/exclude)
- Choose level (Full/Summary/Excluded)
4. Click "Save"
5. Now chat uses these settings
```
---
## Common Mistakes
| Mistake | What Happens | How to Fix |
|---------|--------------|-----------|
| Upload 200 sources at once | System gets slow, processing stalls | Add 10-20 at a time, wait for processing |
| Use full content for all sources | Token usage skyrockets, expensive | Use "Summary" or "Excluded" for background material |
| Add huge PDFs without splitting | Processing is slow, search results less precise | Consider splitting large PDFs into chapters |
| Forget source titles | Can't distinguish between similar sources | Rename sources with descriptive titles right after uploading |
| Don't tag sources | Hard to find and organize later | Add tags immediately: "primary", "background", etc. |
| Mix languages in one source | Transcription/embedding quality drops | Keep each language in separate sources |
| Use same source multiple times | Takes up space, creates confusion | Add once; reuse in multiple chats/notebooks |
---
## Processing Status & Troubleshooting
### What the Status Indicators Mean
```
🟡 Processing
→ Source is being extracted and embedded
→ Wait 30 seconds - 3 minutes depending on size
→ Don't use in Chat yet
🟢 Ready
→ Source is processed and searchable
→ Can use immediately in Chat
→ Can apply transformations
🔴 Error
→ Something went wrong
→ Common reasons:
- Unsupported file format
- File too large or corrupted
- Network timeout
⚪ Not in Context
→ Source added but excluded from Chat
→ Still searchable, not sent to AI
```
### Common Errors & Solutions
**"Unsupported file type"**
- You tried to upload a format not in the list (e.g., `.webp` image)
- Solution: Convert to supported format (PDF for documents, MP3 for audio)
**"Processing timeout"**
- Very large file (>100MB) or very long audio
- Solution: Split into smaller pieces or try uploading again
**"Transcription failed"**
- Audio quality too poor or language not detected
- Solution: Re-record with better quality, or paste text transcript manually
**"Web link won't extract"**
- Website blocks automated access or uses JavaScript for content
- Solution: Copy the article text and paste as "Text" instead
---
## Tips for Best Results
### For PDFs
- Clean, digital PDFs work best
- Remove copy protection if present (legally)
- Scanned PDFs work but take longer
### For Web Articles
- Use full URL including domain
- Avoid cookie/popup-laden sites
- If extraction fails, copy-paste text instead
### For Audio
- Clear, well-recorded audio transcribes better
- Remove background noise if possible
- YouTube videos usually have good transcriptions built-in
### For Large Documents
- Consider splitting into smaller sources
- Gives more precise search results
- Processing is faster for smaller pieces
### For Organization
- Name sources clearly (not "document_2.pdf")
- Add tags immediately after uploading
- Use descriptions for complex documents
---
## What Comes After: Using Your Sources
Once you've added sources, you can:
- **Chat** → Ask questions (see [Chat Effectively](chat-effectively.md))
- **Search** → Find specific content (see [Search Effectively](search.md))
- **Transformations** → Extract structured insights (see [Working with Notes](working-with-notes.md))
- **Ask** → Get comprehensive answers (see [Search Effectively](search.md))
- **Podcasts** → Turn into audio (see [Creating Podcasts](creating-podcasts.md))
---
## Summary Checklist
Before adding sources, confirm:
- [ ] File is in supported format
- [ ] File is under 100MB (or splitting large ones)
- [ ] Web links are full URLs (not shortened)
- [ ] Audio files have clear speech (if transcription-dependent)
- [ ] You've named source clearly
- [ ] You've added tags for organization
- [ ] You understand context levels (Full/Summary/Excluded)
Done! Sources are now ready for Chat, Search, Transformations, and more.
docs/3-USER-GUIDE/api-configuration.md
+386
View File
@@ -0,0 +1,386 @@
# API Configuration
Configure AI provider credentials through the Settings UI. No file editing required.
> **Credential System**: Open Notebook uses encrypted credentials stored in the database. Each credential connects to a provider and allows you to discover, register, and test models.
---
## Overview
Open Notebook manages AI provider access through a **credential-based system**:
1. You create a **credential** for each provider (API key + settings)
2. Credentials are **encrypted** and stored in the database
3. You **test connections** to verify credentials work
4. You **discover and register models** from each credential
5. Models are linked to credentials for direct configuration
---
## Encryption Setup
Before storing credentials, you must configure an encryption key.
### Setting the Encryption Key
Add `OPEN_NOTEBOOK_ENCRYPTION_KEY` to your docker-compose.yml:
```yaml
environment:
- OPEN_NOTEBOOK_ENCRYPTION_KEY=my-secret-passphrase
```
Any string works as a key — it will be securely derived via SHA-256 internally.
> **Warning**: If you change or lose the encryption key, **all stored credentials become unreadable**. Back up your encryption key securely and separately from your database backups.
### Docker Secrets Support
Both password and encryption key support Docker secrets:
```yaml
# docker-compose.yml
services:
open_notebook:
environment:
- OPEN_NOTEBOOK_PASSWORD_FILE=/run/secrets/app_password
- OPEN_NOTEBOOK_ENCRYPTION_KEY_FILE=/run/secrets/encryption_key
secrets:
- app_password
- encryption_key
secrets:
app_password:
file: ./secrets/password.txt
encryption_key:
file: ./secrets/encryption_key.txt
```
### Encryption Details
API keys stored in the database are encrypted using Fernet (AES-128-CBC + HMAC-SHA256).
| Configuration | Behavior |
|---------------|----------|
| Encryption key set | Keys encrypted with your key |
| No encryption key set | Storing credentials is disabled |
---
## Accessing Credential Configuration
1. Click **Settings** in the navigation bar
2. Select **API Keys** tab
3. You'll see existing credentials and an **Add Credential** button
```
Navigation: Settings → API Keys
```
---
## Supported Providers
### Cloud Providers
| Provider | Required Fields | Optional Fields |
|----------|-----------------|-----------------|
| OpenAI | API Key | — |
| Anthropic | API Key | — |
| Google Gemini | API Key | — |
| Groq | API Key | — |
| Mistral | API Key | — |
| DeepSeek | API Key | — |
| xAI | API Key | — |
| OpenRouter | API Key | — |
| Voyage AI | API Key | — |
| ElevenLabs | API Key | — |
### Local/Self-Hosted
| Provider | Required Fields | Notes |
|----------|-----------------|-------|
| Ollama | Base URL | Typically `http://localhost:11434` or `http://ollama:11434` |
### Enterprise
| Provider | Required Fields | Optional Fields |
|----------|-----------------|-----------------|
| Azure OpenAI | API Key, URL Base (Azure endpoint) | Service-specific endpoints (LLM, Embedding, STT, TTS) |
| OpenAI-Compatible | Base URL | API Key, Service-specific configs |
| Vertex AI | Project ID, Location, Credentials Path | — |
---
## Creating a Credential
### Step 1: Add Credential
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select your provider
4. Give it a descriptive name (e.g., "My OpenAI Key", "Work Anthropic")
5. Fill in the required fields (API key, base URL, etc.)
6. Click **Save**
### Step 2: Test Connection
1. On your new credential card, click **Test Connection**
2. Wait for the result:
| Result | Meaning |
|--------|---------|
| Success | Key is valid, provider accessible |
| Invalid API key | Check key format and value |
| Connection failed | Check URL, network, firewall |
### Step 3: Discover Models
1. Click **Discover Models** on the credential card
2. The system queries the provider for available models
3. Review the discovered models
### Step 4: Register Models
1. Select the models you want to use
2. Click **Register Models**
3. The models are now available throughout Open Notebook
---
## Multi-Credential Support
Each provider can have **multiple credentials**. This is useful when:
- You have different API keys for different projects
- You want to test with different endpoints
- Multiple team members need separate credentials
### Creating Multiple Credentials
1. Click **Add Credential** again
2. Select the same provider
3. Fill in different credentials
4. Each credential can discover and register its own models
### How Models Link to Credentials
When you register models from a credential, those models are linked to that specific credential. This means:
- Each model knows which API key to use
- You can have models from different credentials for the same provider
- Deleting a credential removes its linked models
---
## Testing Connections
Click **Test Connection** to verify your credential:
| Result | Meaning |
|--------|---------|
| Success | Key is valid, provider accessible |
| Invalid API key | Check key format and value |
| Connection failed | Check URL, network, firewall |
| Model not available | Key valid but model access restricted |
Test uses inexpensive models (e.g., `gpt-3.5-turbo`, `claude-3-haiku`) to minimize cost.
---
## Configuring Specific Providers
### Simple Providers (API Key Only)
For OpenAI, Anthropic, Google, Groq, Mistral, DeepSeek, xAI, OpenRouter:
1. Add credential with your API key
2. Test connection
3. Discover and register models
### Ollama (URL-Based)
1. Add credential with provider **Ollama**
2. Enter the base URL (e.g., `http://ollama:11434`)
3. Test connection
4. Discover and register models
Ollama allows localhost and private IPs since it runs locally.
### Azure OpenAI
1. Add credential with provider **Azure OpenAI**
2. Enter your API key
3. Enter your Azure endpoint in the **URL Base** field (e.g., `https://myresource.openai.azure.com`)
4. Test connection
5. Discover and register models
The URL Base field is automatically mapped to the Azure endpoint. The API version defaults to `2024-10-21` if not set via environment variable.
### OpenAI-Compatible
For custom OpenAI-compatible servers (LM Studio, vLLM, etc.):
1. Add credential with provider **OpenAI-Compatible**
2. Enter the base URL
3. Enter API key (if required)
4. Optionally configure per-service URLs
Supports separate configurations for:
- LLM (language models)
- Embedding
- STT (speech-to-text)
- TTS (text-to-speech)
### Vertex AI
Google Cloud's enterprise AI platform:
| Field | Example |
|-------|---------|
| Project ID | `my-gcp-project` |
| Location | `us-central1` |
| Credentials Path | `/path/to/service-account.json` |
---
## Migrating from Environment Variables
If you have existing API keys in environment variables (from a previous version):
1. Open **Settings → API Keys**
2. A banner appears: "Environment variables detected"
3. Click **Migrate to Database**
4. Keys are copied to the database (encrypted)
5. Original environment variables remain unchanged
### Migration Behavior
| Scenario | Action |
|----------|--------|
| Key in env only | Migrated to database |
| Key in database only | No change |
| Key in both | Database version kept (skipped) |
### After Migration
- Database credentials are used for all operations
- You can remove the API key environment variables from your docker-compose.yml
- Keep `OPEN_NOTEBOOK_ENCRYPTION_KEY` — it's still required
### Migration Banner Visibility
The migration banner only appears when:
- You have environment variables configured
- Those providers are **not** already in the database
- If all env providers are already migrated, the banner won't show
---
## Migrating from ProviderConfig (v1.1 → v1.2)
If you're upgrading from an older version that used the ProviderConfig system:
- The migration happens automatically on first startup
- Your existing configurations are converted to credentials
- Check **Settings → API Keys** to verify the migration succeeded
- If you see issues, check the API logs for migration messages
---
## Key Storage Security
### Encryption
API keys stored in the database are encrypted using Fernet (AES-128-CBC + HMAC-SHA256).
| Configuration | Behavior |
|---------------|----------|
| Encryption key set | Keys encrypted with your key |
| No encryption key set | Storing API keys in database is disabled |
### Default Credentials
| Setting | Default Value | Production Recommendation |
|---------|---------------|---------------------------|
| Password | None - auth is fully disabled until set | Set `OPEN_NOTEBOOK_PASSWORD` |
| Encryption Key | None (must be set) | Set `OPEN_NOTEBOOK_ENCRYPTION_KEY` to any secret string |
**For production deployments, always set custom credentials.**
---
## Deleting Credentials
1. Click the **Delete** button on the credential card
2. Confirm deletion
3. Credential and all its linked models are removed from the database
---
## Troubleshooting
### Credential Not Saving
| Symptom | Cause | Solution |
|---------|-------|----------|
| Save button disabled | Empty or invalid input | Enter a valid key |
| Error on save | Encryption key not set | Set `OPEN_NOTEBOOK_ENCRYPTION_KEY` in docker-compose.yml |
| Error on save | Database connection issue | Check database status |
### Test Connection Fails
| Error | Cause | Solution |
|-------|-------|----------|
| Invalid API key | Wrong key or format | Verify key from provider dashboard |
| Connection refused | Wrong URL | Check base URL format |
| Timeout | Network issue | Check firewall, proxy settings |
| 403 Forbidden | IP restriction | Whitelist your server IP |
### Migration Issues
| Problem | Solution |
|---------|----------|
| No migration banner | No env vars detected, or already migrated |
| Partial migration | Check error list, fix and retry |
| Keys not working after migration | Clear browser cache, restart services |
### Provider Shows "Not Configured"
1. Check if a credential exists for this provider (Settings → API Keys)
2. Test the credential connection
3. Verify key format matches provider requirements
4. Re-discover and register models if needed
---
## Provider-Specific Notes
### OpenAI
- Keys start with `sk-proj-` (project keys) or `sk-` (legacy)
- Requires billing enabled on account
### Anthropic
- Keys start with `sk-ant-`
- Check account has API access enabled
### Google Gemini
- Keys start with `AIzaSy`
- Free tier has rate limits
### Ollama
- No API key required
- Default URL: `http://localhost:11434` (local) or `http://ollama:11434` (Docker)
- Ensure Ollama server is running
### Azure OpenAI
- Enter your Azure endpoint in the **URL Base** field (format: `https://{resource-name}.openai.azure.com`)
- API version defaults to `2024-10-21`; override via `AZURE_OPENAI_API_VERSION` environment variable if needed
- Deployment names configured separately when registering models via the credential's Discover Models dialog
---
## Related
- **[AI Providers](../5-CONFIGURATION/ai-providers.md)** — Provider setup instructions and recommendations
- **[Security](../5-CONFIGURATION/security.md)** — Password and encryption configuration
- **[Environment Reference](../5-CONFIGURATION/environment-reference.md)** — All configuration options
docs/3-USER-GUIDE/chat-effectively.md
+554
View File
@@ -0,0 +1,554 @@
# Chat Effectively - Conversations with Your Research
Chat is your main tool for exploratory questions and back-and-forth dialogue. This guide covers how to use it effectively.
---
## Quick-Start: Your First Chat
```
1. Go to your notebook
2. Click "Chat"
3. Select which sources to include (context)
4. Type your question
5. Click "Send"
6. Read the response
7. Ask a follow-up (context stays same)
8. Repeat until satisfied
```
That's it! But doing it *well* requires understanding how context works.
---
## Context Management: The Key to Good Chat
Context controls **what the AI is allowed to see**. This is your most important control.
### The Three Levels Explained
**FULL CONTENT**
- AI sees: Complete source text
- Cost: 100 tokens per 1K tokens of source
- Best for: Detailed analysis, precise citations
- Example: "Analyze this research paper closely"
```
You set: Paper A → Full Content
AI sees: Every word of Paper A
AI can: Cite specific sentences, notice nuances
Result: Precise, detailed answers (higher cost)
```
**SUMMARY ONLY**
- AI sees: AI-generated 200-word summary (not full text)
- Cost: ~10-20% of full content cost
- Best for: Background material, reference context
- Example: "Use this for background, focus on the main paper"
```
You set: Paper B → Summary Only
AI sees: Condensed summary, key points
AI can: Reference main ideas but not details
Result: Faster, cheaper answers (loses precision)
```
**NOT IN CONTEXT**
- AI sees: Nothing
- Cost: 0 tokens
- Best for: Confidential, irrelevant, archived content
- Example: "Keep this in notebook but don't use now"
```
You set: Paper C → Not in Context
AI sees: Nothing (completely excluded)
AI can: Never reference it
Result: No cost, no privacy risk for that source
```
### Setting Context (Step by Step)
```
1. Click "Select Sources"
(Shows list of all sources in notebook)
2. For each source:
□ Checkbox: Include or exclude
Level dropdown:
├─ Full Content
├─ Summary Only
└─ Excluded
3. Check your selections
Example:
✓ Paper A (Full Content) - "Main focus"
✓ Paper B (Summary Only) - "Background"
✓ Paper C (Excluded) - "Keep private"
□ Paper D (Not included) - "Not relevant"
4. Click "Save Context"
5. Now chat uses these settings
```
### Context Strategies
**Strategy 1: Minimalist**
- Main source: Full Content
- Everything else: Excluded
- Result: Focused, cheap, precise
```
Use when:
- Analyzing one source deeply
- Budget-conscious
- Want focused answers
```
**Strategy 2: Comprehensive**
- All sources: Full Content
- Result: All context considered, expensive
```
Use when:
- Comprehensive analysis
- Unlimited budget
- Want AI to see everything
```
**Strategy 3: Tiered**
- Primary sources: Full Content
- Secondary sources: Summary Only
- Background/reference: Excluded
- Result: Balanced cost/quality
```
Use when:
- Mix of important and reference material
- Want thorough but not expensive
- Most common strategy
```
**Strategy 4: Privacy-First**
- Sensitive docs: Excluded
- Public research: Full Content
- Result: Never send confidential data
```
Use when:
- Company confidential materials
- Personal sensitive data
- Complying with data protection
```
---
## Asking Effective Questions
### Good Questions vs. Poor Questions
**Poor Question**
```
"What do you think?"
Problems:
- Too vague (about what?)
- No context (what am I analyzing?)
- Can't verify answer (citing what?)
Result: Generic, shallow answer
```
**Good Question**
```
"Based on the paper's methodology section,
what are the three main limitations the authors acknowledge?
Please cite which pages mention each one."
Strengths:
- Specific about what you want
- Clear scope (methodology section)
- Asks for citations
- Requires deep reading
Result: Precise, verifiable, useful answer
```
### Question Patterns That Work
**Factual Questions**
```
"What does the paper say about X?"
"Who are the authors?"
"What year was this published?"
Result: Simple, factual answers with citations
```
**Analysis Questions**
```
"How does this approach differ from the traditional method?"
"What are the main assumptions underlying this argument?"
"Why do you think the author chose this methodology?"
Result: Deeper thinking, comparison, critique
```
**Synthesis Questions**
```
"How do these two sources approach the problem differently?"
"What's the common theme across all three papers?"
"If we combine these approaches, what would we get?"
Result: Cross-source insights, connections
```
**Actionable Questions**
```
"What are the practical implications of this research?"
"How could we apply these findings to our situation?"
"What's the next logical research direction?"
Result: Practical, forward-looking answers
```
### The SPECIFIC Formula
Good questions have:
1. **SCOPE** - What are you analyzing?
"In this research paper..."
"Looking at these three articles..."
"Based on your experience..."
2. **SPECIFICITY** - Exactly what do you want?
"...the methodology..."
"...main findings..."
"...recommended next steps..."
3. **CONSTRAINT** - Any limits?
"...in 3 bullet points..."
"...with citations to page numbers..."
"...comparing these two approaches..."
4. **VERIFICATION** - How can you check it?
"...with specific quotes..."
"...cite your sources..."
"...link to the relevant section..."
**Example:**
```
Poor: "What about transformers?"
Good: "In this research paper on machine learning,
explain the transformer architecture in 2-3 sentences,
then cite which page describes the attention mechanism."
```
---
## Follow-Up Questions (The Real Power of Chat)
Chat's strength is dialogue. You ask, get an answer, ask more.
### Building on Responses
```
First question:
"What's the main finding?"
AI: "The study shows X [citation]"
Follow-up question:
"How does that compare to Y research?"
AI: "The key difference is Z [citation]"
Next question:
"Why do you think that difference matters?"
AI: "Because it affects A, B, C [explained]"
```
### Iterating Toward Understanding
```
Round 1: Get overview
"What's this source about?"
Round 2: Get details
"What's the most important part?"
Round 3: Compare
"How does it relate to my notes on X?"
Round 4: Apply
"What should I do with this information?"
```
### Changing Direction
```
Context stays same, but you ask new questions:
Question 1: "What's the methodology?"
Question 2: "What are the limitations?"
Question 3: "What about the ethical implications?"
Question 4: "Who else has done similar work?"
All in one conversation, reusing context.
```
### Adjusting Context Between Rounds
```
After question 3, you realize:
"I need more context from another source"
1. Click "Adjust Context"
2. Add new source or change context level
3. Your conversation history stays
4. Continue asking with new context
```
---
## Citations and Verification
Citations are how you verify that the AI's answer is accurate.
### Understanding Citations
```
AI Response with Citation:
"The paper reports a 95% accuracy rate [see page 12]"
What this means:
✓ The claim "95% accuracy rate" is from page 12
✓ You can verify by reading page 12
✓ If page 12 doesn't say that, the AI hallucinated
```
### Requesting Better Citations
```
If you get a response without citations:
Ask: "Please cite the page number for that claim"
or: "Show me where you found that information"
AI will:
- Find the citation
- Provide page numbers
- Show you the source
```
### Verification Workflow
```
1. Get answer from Chat
2. Check citation (which source? which page?)
3. Click citation link (if available)
4. See the actual text in source
5. Does it really say what AI claimed?
If YES: Great, you can use this answer
If NO: The AI hallucinated, ask for correction
```
---
## Common Chat Patterns
### Pattern 1: Deep Dive into One Source
```
1. Set context: One source (Full Content)
2. Question 1: Overview
3. Question 2: Main argument
4. Question 3: Evidence for argument
5. Question 4: Limitations
6. Question 5: Next steps
Result: Complete understanding of one source
```
### Pattern 2: Comparative Analysis
```
1. Set context: 2-3 sources (all Full Content)
2. Question 1: What does each source say about X?
3. Question 2: How do they agree?
4. Question 3: How do they disagree?
5. Question 4: Which approach is stronger?
Result: Understanding differences and trade-offs
```
### Pattern 3: Research Exploration
```
1. Set context: Many sources (mix of Full/Summary)
2. Question 1: What are the main perspectives?
3. Question 2: What's missing from these views?
4. Question 3: What questions does this raise?
5. Question 4: What should I research next?
Result: Understanding landscape and gaps
```
### Pattern 4: Problem Solving
```
1. Set context: Relevant sources (Full Content)
2. Question 1: What's the problem?
3. Question 2: What approaches exist?
4. Question 3: Pros and cons of each?
5. Question 4: Which would work best for [my situation]?
Result: Decision-making informed by research
```
---
## Optimizing for Cost
Chat uses tokens for every response. Here's how to use efficiently:
### Reduce Token Usage
**Minimize context**
```
Option A: All sources, Full Content
Cost per response: 5,000 tokens
Option B: Only relevant sources, Summary Only
Cost per response: 1,000 tokens
Savings: 80% cheaper, same conversation
```
**Shorter questions**
```
Verbose: "Could you please analyze the methodology
section of this paper and explain in detail
what the authors did?"
Concise: "Summarize the methodology in 2-3 points."
Savings: 20-30% per response
```
**Use cheaper models**
```
GPT-4o: $0.15 per 1M input tokens
GPT-4o-mini: $0.03 per 1M input tokens
Claude Sonnet: $0.90 per 1M input tokens
For chat: Mini/Haiku models are usually fine
For deep analysis: Sonnet/Opus worth the cost
```
### Budget Strategies
**Exploration budget**
- Use cheap model
- Broad context (understand landscape)
- Short questions
- Result: Low cost, good overview
**Analysis budget**
- Use powerful model
- Focused context (main source only)
- Detailed questions
- Result: Higher cost, deep insights
**Synthesis budget**
- Use powerful model for final synthesis
- Multiple sources (Full Content)
- Complex comparative questions
- Result: Expensive but valuable output
---
## Troubleshooting Chat Issues
### Poor Responses
| Problem | Cause | Solution |
|---------|-------|----------|
| Generic answers | Vague question | Be specific (see question patterns) |
| Missing context | Not enough in context | Add sources or change to Full Content |
| Incorrect info | Source not in context | Add the relevant source |
| Hallucinating | Model confused | Ask for citations, verify claims |
| Shallow analysis | Wrong model | Switch to more powerful model |
### High Costs
| Problem | Cause | Solution |
|---------|-------|----------|
| Expensive per response | Too much context | Use Summary Only or exclude sources |
| Many follow-ups | Exploratory chat | Use Ask instead for single comprehensive answer |
| Long conversations | Keeping history | Archive old chats, start fresh |
| Large sources | Full text in context | Use Summary Only for large documents |
---
## Best Practices
### Before You Chat
- [ ] Add sources you'll need
- [ ] Decide context strategy (Tiered is usually best)
- [ ] Choose model (cheaper for exploration, powerful for analysis)
- [ ] Have a question in mind
### During Chat
- [ ] Ask specific questions (use SPECIFIC formula)
- [ ] Check citations for factual claims
- [ ] Follow up on unclear points
- [ ] Adjust context if you need different sources
### After Chat
- [ ] Save good responses as notes
- [ ] Archive conversation if you're done
- [ ] Organize notes for future reference
- [ ] Use insights in other features (Ask, Transformations, Podcasts)
---
## When to Use Chat vs. Ask
**Use CHAT when:**
- You want a dialogue
- You're exploring a topic
- You'll ask multiple related questions
- You want to adjust context during conversation
- You're not sure exactly what you need
**Use ASK when:**
- You have one specific question
- You want a comprehensive answer
- You want the system to auto-search
- You want one response, not dialogue
- You want maximum tokens spent on search
---
## Summary: Chat as Conversation
Chat is fundamentally different from asking ChatGPT directly:
| Aspect | ChatGPT | Open Notebook Chat |
|--------|---------|-------------------|
| **Source control** | None (uses training) | You control which sources are visible |
| **Cost control** | Per token | Per token, but context is your choice |
| **Iteration** | Works | Works, with your sources changing dynamically |
| **Citations** | Made up often | Tied to your sources (verifiable) |
| **Privacy** | Your data to OpenAI | Your data stays local (unless you choose) |
The key insight: **Chat is retrieval-augmented generation.** AI sees only what you put in context. You control the conversation and the information flow.
That's why Chat is powerful for research. You're not just talking to an AI; you're having a conversation with your research itself.
docs/3-USER-GUIDE/citations.md
+299
View File
@@ -0,0 +1,299 @@
# Citations - Verify and Trust AI Responses
Citations connect AI responses to your source materials. This guide covers how to use and verify them.
---
## Why Citations Matter
Every AI-generated response in Open Notebook includes citations to your sources. This lets you:
- **Verify claims** - Check that AI actually read what it claims
- **Find original context** - See the full passage around a quote
- **Catch hallucinations** - Spot when AI makes things up
- **Build credibility** - Your notes have traceable sources
---
## Quick Start: Using Citations
### Reading Citations
```
AI Response:
"The study found a 95% accuracy rate [1] using the proposed method."
[1] = Click to see source
What happens when you click:
→ Opens the source document
→ Highlights the relevant section
→ You can verify the claim
```
### Requesting Better Citations
If a response lacks citations, ask:
```
"Please cite the specific page or section for that claim."
"Where in the document does it say that?"
"Can you quote the exact text?"
```
---
## How Citations Work
### Automatic Generation
When AI references your sources, citations are generated automatically:
```
1. AI analyzes your question
2. Retrieves relevant source chunks
3. Generates response with inline citations
4. Links citations to original source locations
```
### Citation Format
```
Inline format:
"The researchers concluded X [1] and Y [2]."
Reference list:
[1] Paper Title - Section 3.2
[2] Report Name - Page 15
Clickable: Each [number] links to the source
```
---
## Verifying Citations
### The Verification Workflow
```
Step 1: Read AI response
"The model achieved 95% accuracy [1]"
Step 2: Click citation [1]
→ Opens source document
→ Shows relevant passage
Step 3: Verify the claim
Does source actually say 95%?
Is context correct?
Any nuance missed?
Step 4: Trust or correct
✓ Accurate → Use the insight
✗ Wrong → Ask AI to correct
```
### What to Check
| Check | Why |
|-------|-----|
| **Exact numbers** | AI sometimes rounds or misremembers |
| **Context** | Quote might mean something different in context |
| **Attribution** | Is this the source's claim or someone they cited? |
| **Completeness** | Did AI miss important caveats? |
---
## Citations in Different Features
### Chat Citations
```
Context: Sources you selected
Citations: Reference chunks used in response
Verification: Click to see original text
Save: Citations preserved when saving as note
```
### Ask Feature Citations
```
Context: Auto-searched across all sources
Citations: Multiple sources synthesized
Verification: Each source linked separately
Quality: Often more comprehensive than Chat
```
### Transformation Citations
```
Context: Single source being transformed
Citations: Points back to original document
Verification: Compare output to source
Use: When you need structured extraction
```
---
## Saving Citations
### In Notes
When you save an AI response as a note, citations are preserved:
```
Original response:
"According to the paper [1], the method works by..."
Saved note includes:
- The text
- The citation link
- Reference to source document
```
### Exporting
Citations work in exports:
| Format | Citation Behavior |
|--------|-------------------|
| **Markdown** | Links preserved as `[text](link)` |
| **Copy/Paste** | Plain text with reference numbers |
| **PDF** | Clickable references (if supported) |
---
## Citation Quality Tips
### Get Better Citations
**Be specific in questions:**
```
Poor: "What does it say about X?"
Good: "What does page 15 say about X? Please quote directly."
```
**Request citation format:**
```
"Include page numbers for each claim."
"Cite specific sections, not just document names."
```
**Use Full Content context:**
```
Summary Only → Less precise citations
Full Content → Exact quotes possible
```
### When Citations Are Missing
| Situation | Cause | Solution |
|-----------|-------|----------|
| No citations | AI used general knowledge | Ask: "Base your answer only on my sources" |
| Vague citations | Source not in Full Content | Change context level |
| Wrong citations | AI confused sources | Ask to verify with quotes |
---
## Common Issues
### "Citation doesn't match claim"
```
Problem: AI says X, but source says Y
What happened:
- AI paraphrased incorrectly
- AI combined multiple sources confusingly
- Source was taken out of context
Solution:
1. Click citation to see original
2. Note the discrepancy
3. Ask AI: "The source says Y, not X. Please correct."
```
### "Can't find cited section"
```
Problem: Citation link doesn't show relevant text
What happened:
- Source was chunked differently than expected
- Information spread across multiple sections
- Processing missed some content
Solution:
1. Search within source for key terms
2. Ask AI for more specific location
3. Re-process source if needed
```
### "No citations at all"
```
Problem: AI response has no source references
What happened:
- Sources not in context
- Question asked for opinion/general knowledge
- Model didn't find relevant content
Solution:
1. Check context settings
2. Rephrase: "Based on my sources, what..."
3. Add more relevant sources
```
---
## Best Practices
### For Research Integrity
1. **Always verify important claims** - Don't trust AI blindly
2. **Check context** - Quotes can be misleading out of context
3. **Note limitations** - AI might miss nuance
4. **Keep source access** - Don't delete sources you cite
### For Academic Work
1. **Use Full Content** for documents you'll cite
2. **Request specific page numbers**
3. **Cross-check with original sources**
4. **Document your verification process**
### For Professional Use
1. **Verify before sharing** - Check claims clients will see
2. **Keep citation trail** - Save notes with sources linked
3. **Be transparent** - Note when insights are AI-assisted
---
## Summary
```
Citations = Your verification system
How to use:
1. Read AI response
2. Note citation markers [1], [2], etc.
3. Click to see original source
4. Verify claim matches source
5. Trust verified insights
When citations fail:
- Ask for specific quotes
- Change to Full Content
- Request page numbers
- Verify manually
Why it matters:
- AI can hallucinate
- Context can change meaning
- Trust requires verification
- Good research needs sources
```
Citations aren't just references — they're your quality control. Use them to build research you can trust.
docs/3-USER-GUIDE/creating-podcasts.md
+678
View File
@@ -0,0 +1,678 @@
# Creating Podcasts - Turn Research into Audio
Podcasts let you consume your research passively. This guide covers the complete workflow from setup to download.
---
## Quick-Start: Your First Podcast (5 Minutes)
```
1. Go to your notebook
2. Click "Generate Podcast"
3. Select sources to include
4. Choose a speaker profile (or use default)
5. Click "Generate"
6. Wait 3-10 minutes (non-blocking)
7. Download MP3 when ready
8. Done!
```
That's the minimum. Let's make it better.
---
## Step-by-Step: The Complete Workflow
### Step 1: Prepare Your Notebook
```
Before generating, make sure:
✓ You have sources added
(At least 1-2 sources)
✓ Sources have been processed
(Green "Ready" status)
✓ Notes are organized
(If you want notes included)
✓ You know your message
(What's the main story?)
Typical preparation: 5-10 minutes
```
### Step 2: Choose Content
```
Click "Generate Podcast"
You'll see:
- List of all sources in notebook
- List of all notes
Select which to include:
☑ Paper A (primary source)
☑ Paper B (supporting source)
☐ Old note (not relevant)
✓ Analysis note (important)
What to include:
- Primary sources: Always include
- Supporting sources: Usually include
- Notes: Include your analysis/insights
- Everything: Can overload podcast
Recommended: 3-5 sources per podcast
```
### Step 3: Choose Episode Profile
An episode profile defines the structure and tone.
**Option A: Use Preset Profile**
```
Open Notebook provides preset profiles:
Academic Presentation (Monologue)
├─ 1 speaker
├─ Tone: Educational
└─ Format: Expert explaining topic
Expert Interview (2-speaker)
├─ 2 speakers: Host + Expert
├─ Tone: Q&A, conversational
└─ Format: Interview with expert
Debate Format (2-speaker)
├─ 2 speakers: Pro vs. Con
├─ Tone: Discussion, disagreement
└─ Format: Debate about the topic
Panel Discussion (3-4 speaker)
├─ 3-4 speakers: Different perspectives
├─ Tone: Thoughtful discussion
└─ Format: Each brings different expertise
Solo Explanation (Monologue)
├─ 1 speaker
├─ Tone: Conversational, friendly
└─ Format: Personal explanation
```
**Pick based on your content:**
- One main idea → Academic Presentation
- You want to explain → Solo Explanation
- Two competing views → Debate Format
- Multiple perspectives → Panel Discussion
- Want to explore → Expert Interview
### Step 4: Customize Episode Profile (Optional)
If presets don't fit, customize:
```
Episode Profile
├─ Title: "AI Safety in 2026"
├─ Description: "Exploring current approaches"
├─ Length target: 20 minutes
├─ Tone: "Academic but accessible"
├─ Focus areas:
│ ├─ Main approaches to alignment
│ ├─ Pros and cons comparison
│ └─ Open questions
├─ Audience: "Researchers new to field"
└─ Format: "Debate between two perspectives"
How to set:
1. Click "Customize"
2. Edit each field
3. Click "Save Profile"
4. System uses your profile for outline generation
```
### Step 5: Create or Select Speakers
Speakers are the "voice" of your podcast.
**Option A: Use Preset Speakers**
```
Open Notebook provides preset profiles:
"Expert Alex"
- Expertise: Deep knowledge
- Personality: Rigorous, patient
- Voice Model: Selected from model registry
"Curious Sam"
- Expertise: Curious newcomer
- Personality: Asks questions
- Voice Model: Selected from model registry
"Skeptic Jordan"
- Expertise: Critical perspective
- Personality: Challenges assumptions
- Voice Model: Selected from model registry
For your first podcast: Use presets
For custom podcast: Create your own
```
**Option B: Create Custom Speakers**
```
Click "Add Speaker"
Fill in:
Name: "Dr. Research Expert"
Expertise:
"20 years in AI safety research,
deep knowledge of alignment approaches"
Personality:
"Rigorous, academic style,
explains clearly, asks good questions"
Voice Configuration:
- Voice Model: Select from model registry (e.g., OpenAI TTS, Google TTS, ElevenLabs)
- Voice: Choose from available voices for the selected model
- Per-speaker override: Each speaker can optionally use a different voice model
Credentials are automatically resolved from the model configuration.
Example:
Name: Dr. Research Expert
Expertise: AI safety alignment research
Personality: Rigorous, academic but accessible
Voice Model: ElevenLabs TTS (from registry), Voice: professional male
```
### Step 6: Generate Podcast
```
1. Review your setup:
Sources: ✓ Selected
Profile: ✓ Episode profile chosen
Speakers: ✓ Speakers configured
2. Click "Generate Podcast"
3. System begins:
- Analyzing your content
- Creating outline
- Writing dialogue
- Generating audio
- Mixing speakers
4. Status shows progress:
20% Outline generation
40% Dialogue writing
60% Audio synthesis
80% Mixing
100% Complete
Processing time:
- 5 minutes of content: 3-5 minutes
- 15 minutes of content: 5-10 minutes
- 30 minutes of content: 10-20 minutes
```
### Step 7: Review and Download
```
When complete:
Preview:
- Play audio sample
- Review transcript
- Check duration
Options:
✓ Download as MP3 - Save to computer
✓ Stream directly - Listen in browser
✓ Share link - Get shareable URL (if public)
✓ Regenerate - Try different speakers/profile
Download:
1. Click "Download as MP3"
2. Choose quality: 128kbps / 192kbps / 320kbps
3. Save file: podcast_[notebook]_[date].mp3
4. Listen!
```
---
## Understanding What Happens Behind the Scenes
### The Generation Pipeline
```
Stage 1: CONTENT ANALYSIS (1 minute)
Your sources → What's the main story?
→ Key themes?
→ Debate points?
Stage 2: OUTLINE CREATION (2-3 minutes)
Themes → Episode structure
→ Section breakdown
→ Talking points
Stage 3: DIALOGUE WRITING (2-3 minutes)
Outline → Convert to natural dialogue
→ Add speaker personalities
→ Create flow and transitions
Stage 4: AUDIO SYNTHESIS (3-5 minutes per speaker)
Script + Speaker → Text-to-speech
→ Individual audio files
→ High quality audio
Stage 5: MIXING & MASTERING (1-2 minutes)
Multiple audio → Combine speakers
→ Level audio
→ Add polish
→ Final MP3
Total: 10-20 minutes for typical podcast
```
---
## Text-to-Speech Providers
Different providers, different qualities.
### OpenAI (Recommended)
```
Voices: 5 options (Alloy, Echo, Fable, Onyx, Shimmer)
Quality: Good, natural sounding
Speed: Fast
Cost: ~$0.015 per minute
Best for: General purpose, natural speech
Example: "I have to say, the research shows..."
```
### Google TTS
```
Voices: Many options, various accents
Quality: Excellent, very natural
Speed: Fast
Cost: ~$0.004 per minute
Best for: High quality output, accents
Example: "The research demonstrates that..."
```
### ElevenLabs
```
Voices: 100+ voices, highly customizable
Quality: Exceptional, very expressive
Speed: Slower (5-10 seconds per phrase)
Cost: ~$0.10 per minute
Best for: Premium quality, emotional range
Example: [Can convey emotion and tone]
```
### Local TTS (Free)
```
Voices: Limited, basic options
Quality: Basic, robotic
Speed: Depends on hardware (slow)
Cost: Free (local processing)
Best for: Privacy, testing, offline use
Example: "The research shows..."
Privacy: Everything stays on your computer
```
### Which Provider to Choose?
```
For your first podcast: Google (quality/cost balance)
For privacy-sensitive: Local TTS (free, private)
For premium quality: ElevenLabs (best voices)
For budget: Google (cheapest quality option)
For speed: OpenAI (fast generation)
```
---
## Tips for Better Podcasts
### Choose Right Profile
```
Single source analysis → Academic Presentation
"Explaining one paper to someone new"
Comparing two approaches → Debate Format
"Pros and cons of different methods"
Multiple sources + insights → Panel Discussion
"Different experts discussing topic"
Narrative exploration → Expert Interview
"Host interviewing research expert"
Personal take → Solo Explanation
"You explaining your analysis"
```
### Create Good Speakers
```
Good Speaker:
✓ Clear expertise (know what they're talking about)
✓ Distinct personality (not generic)
✓ Good voice choice (matches personality)
✓ Realistic backstory (feels like real person)
Bad Speaker:
✗ Generic expertise ("good at research")
✗ No personality ("just reads")
✗ Mismatched voice (deep voice for young person)
✗ Contradicts personality (serious person uses casual voice)
```
### Focus Content
```
Better: Podcast on ONE specific topic
"How transformers work" (15 minutes, focused)
Worse: Podcast on everything
"All of AI 2025" (2 hours, unfocused)
Guideline:
- 5-10 minutes: One narrow topic
- 15-20 minutes: One broad topic
- 30+ minutes: Multiple related subtopics
Shorter is usually better for podcasts.
```
### Optimize Source Selection
```
Too much content:
"Here are all 20 papers"
→ Podcast becomes 2+ hours
→ Unfocused
→ Low quality
Right amount:
"Here are 3 key papers"
→ Podcast is 15-20 minutes
→ Focused
→ High quality
Rule: 3-5 sources per podcast
Remove long background papers
Keep focused on main topic
```
---
## Quality Troubleshooting
### Audio Sounds Robotic
**Problem**: TTS voice sounds unnatural
**Solutions**:
```
1. Switch provider: Try Google or ElevenLabs instead
2. Choose different voice: Some voices more natural
3. Shorter sentences: Very long sentences sound robotic
4. Adjust pacing: Ask for "natural, conversational pacing"
```
### Audio Sounds Unclear
**Problem**: Hard to understand what's being said
**Solutions**:
```
1. Re-generate with different speaker
2. Try different TTS provider
3. Use speakers with clear accents
4. Lower background noise (if any)
5. Increase speech rate (if too slow)
```
### Missing Content
**Problem**: Important information isn't in podcast
**Solutions**:
```
1. Include that source in content selection
2. Review generated outline (check before generating)
3. Regenerate with clearer profile instructions
4. Try different model (more thorough model)
```
### Speakers Don't Match
**Problem**: Speakers sound like same person
**Solutions**:
```
1. Choose different voice models from the registry for each speaker
2. Choose very different voice options
3. Increase personality differences in profile
4. Try different speaker count (2 vs 3 vs 4)
```
### Generation Failed
**Problem**: "Podcast generation failed"
**Solutions**:
```
1. Check internet connection (especially TTS)
2. Try again (might be temporary issue)
3. Use local TTS (doesn't need internet)
4. Reduce source count (less to process)
5. Contact support if persistent
```
---
## Advanced: Multiple Podcasts from Same Research
You can generate different podcasts from one notebook:
```
Podcast 1: Overview
Profile: Academic Presentation
Sources: Papers A, B, C
Speakers: One expert
Length: 15 minutes
→ Use for "What's this about?" understanding
Podcast 2: Deep Dive
Profile: Expert Interview
Sources: Paper A (Full) + B, C (Summary)
Speakers: Expert + Interviewer
Length: 30 minutes
→ Use for detailed exploration
Podcast 3: Debate
Profile: Debate Format
Sources: Papers A vs B (different approaches)
Speakers: Pro-A speaker + Pro-B speaker
Length: 20 minutes
→ Use for comparing approaches
```
Each tells the same story from different angles.
---
## Exporting and Sharing
### Download MP3
```
1. Generation complete
2. Click "Download"
3. Choose quality:
- 128 kbps: Smallest file, lower quality
- 192 kbps: Balanced (recommended)
- 320 kbps: Highest quality, largest file
4. Save to computer
5. Use in podcast app, upload to platform, etc.
```
### Export Transcript
```
1. Click "Export Transcript"
2. Get full dialogue as text
3. Useful for:
- Blog post content
- Show notes
- Searchable text version
- Accessibility
```
### Share Link
```
If podcast is public:
1. Click "Share"
2. Get shareable link
3. Others can listen/download
4. Useful for:
- Sharing with team
- Public distribution
- Embedding on website
```
### Publish to Podcast Platforms
```
If you want to distribute (future feature):
1. Download MP3
2. Upload to platform (Spotify, Apple Podcasts, etc.)
3. Add metadata (title, description, episode notes)
4. Your research becomes a published podcast!
```
---
## Best Practices
### Before Generation
- [ ] Sources are processed and ready
- [ ] You've chosen content to include
- [ ] You have a clear episode profile
- [ ] Speakers are well-defined
- [ ] Content is focused (3-5 sources max)
### During Generation
- Don't close the browser (use background processing)
- Check back in 5-15 minutes
- Review transcript when complete
- Listen to sample before downloading
### After Generation
- [ ] Download MP3 to computer
- [ ] Save in organized folder
- [ ] Add metadata (title, description, date)
- [ ] Test listening in podcast app
- [ ] Share with colleagues for feedback
---
## Use Cases
### Academic Researcher
```
Podcast: Explaining your dissertation
Speakers: You + colleague
Content: Your papers + supporting research
Use: Share with advisors, test explanations
```
### Content Creator
```
Podcast: Research-to-podcast article
Speakers: Narrator + expert
Content: Articles you've researched
Use: Transform article into podcast version
```
### Team Research
```
Podcast: Weekly research updates
Speakers: Multiple team members
Content: This week's papers
Use: Team updates, knowledge sharing
```
### Learning/Teaching
```
Podcast: Teaching material
Speakers: Teacher + inquisitive student
Content: Textbook + examples
Use: Students learn while commuting
```
---
## Cost Breakdown Example
### Generate 15-minute podcast with ElevenLabs
```
Generation (outline + dialogue):
No charge (included in service)
Text-to-speech:
2 speakers × 15 minutes = 30 minutes TTS
ElevenLabs: $0.10 per minute
Cost: 30 × $0.10 = $3.00
Processing:
Included (no additional cost)
Total: $3.00 per podcast
Cheaper options:
With Google TTS: ~$0.12
With OpenAI: ~$0.45
With Local TTS: ~$0.00
```
---
## Summary: Podcasts as Research Tool
Podcasts transform how you consume research:
```
Before: Reading papers takes time, focus
After: Listen while commuting, exercising, doing chores
Before: Can't share complex research easily
After: Share audio of your analysis
Before: Different consumption styles isolated
After: Same research, multiple formats (read/listen)
```
Podcasts aren't just for entertainment—they're a tool for making research more accessible, shareable, and consumable.
That's why they're important for Open Notebook.
docs/3-USER-GUIDE/index.md
+208
View File
@@ -0,0 +1,208 @@
# User Guide - How to Use Open Notebook
This guide covers practical, step-by-step usage of Open Notebook features. You already understand the concepts; now learn how to actually use them.
> **Prerequisite**: Review [2-CORE-CONCEPTS](../2-CORE-CONCEPTS/index.md) first to understand the mental models (notebooks, sources, notes, chat, transformations, podcasts).
---
## Start Here
### [Interface Overview](interface-overview.md)
Learn the layout before diving in. Understand the three-panel design and where everything is.
---
## Eight Core Features
### 1. [Adding Sources](adding-sources.md)
How to bring content into your notebook. Supports PDFs, web links, audio, video, text, and more.
**Quick links:**
- Upload a PDF or document
- Add a web link or article
- Transcribe audio or video
- Paste text directly
- Common mistakes + fixes
---
### 2. [Working with Notes](working-with-notes.md)
Creating, organizing, and using notes (both manual and AI-generated).
**Quick links:**
- Create a manual note
- Save AI responses as notes
- Apply transformations to generate insights
- Organize with tags and naming
- Use notes across your notebook
---
### 3. [Chat Effectively](chat-effectively.md)
Have conversations with AI about your sources. Manage context to control what AI sees.
**Quick links:**
- Start your first chat
- Select which sources go in context
- Ask effective questions
- Use follow-ups productively
- Understand citations and verify claims
---
### 4. [Creating Podcasts](creating-podcasts.md)
Convert your research into audio dialogue for passive consumption.
**Quick links:**
- Create your first podcast
- Choose or customize speakers
- Select TTS provider
- Generate and download
- Common audio quality fixes
---
### 5. [Search Effectively](search.md)
Two search modes: text-based (keyword) and vector-based (semantic). Know when to use each.
**Quick links:**
- Text search vs vector search (when to use)
- Running effective searches
- Using the Ask feature for comprehensive answers
- Saving search results as notes
- Troubleshooting poor results
---
### 6. [Transformations](transformations.md)
Batch-process sources with predefined templates. Extract the same insights from multiple documents.
**Quick links:**
- Built-in transformation templates
- Creating custom transformations
- Applying to single or multiple sources
- Managing transformation output
---
### 7. [Citations](citations.md)
Verify AI claims by tracing them back to source material. Understand the citation system.
**Quick links:**
- Reading and clicking citations
- Verifying claims against sources
- Requesting better citations
- Saving cited content as notes
---
### 8. [API Configuration](api-configuration.md)
Configure AI provider API keys directly through the Settings UI.
**Quick links:**
- Add API keys without editing files
- Test provider connections
- Migrate from environment variables
- Manage Azure and OpenAI-compatible providers
- Understand key storage and encryption
---
## Which Feature for Which Task?
```
Task: "I want to explore a topic with follow-ups"
→ Use: Chat (add sources, select context, have conversation)
Task: "I want one comprehensive answer"
→ Use: Search / Ask (system finds relevant content)
Task: "I want to extract the same info from many sources"
→ Use: Transformations (define template, apply to all)
Task: "I want summaries of all my sources"
→ Use: Transformations (with built-in summary template)
Task: "I want to share my research in audio form"
→ Use: Podcasts (create speakers, generate episode)
Task: "I want to find that quote I remember"
→ Use: Search / Text Search (keyword matching)
Task: "I'm exploring a concept without knowing exact words"
→ Use: Search / Vector Search (semantic similarity)
Task: "I need to add or change my AI provider API keys"
→ Use: Settings / API Keys (configure providers without editing files)
```
---
## Quick-Start Checklist: First 15 Minutes
**Step 1: Create a Notebook (1 min)**
- Name: Something descriptive ("Q1 Market Research", "AI Safety Papers", etc.)
- Description: 1-2 sentences about what you're researching
- This is your research container
**Step 2: Add Your First Source (3 min)**
- Pick one: PDF, web link, or text
- Follow [Adding Sources](adding-sources.md)
- Wait for processing (usually 30-60 seconds)
**Step 3: Chat About It (3 min)**
- Go to Chat
- Select your source (set context to "Full Content")
- Ask a simple question: "What are the main points?"
- See AI respond with citations
**Step 4: Save Insight as Note (2 min)**
- Good response? Click "Save as Note"
- Name it something useful ("Main points from source X")
- Now you have a captured insight
**Step 5: Explore More (6 min)**
- Add another source
- Chat about both together
- Ask a question that compares them
- Follow up with clarifying questions
**Done!** You've used the core workflow: notebook → sources → chat → notes
---
## Common Mistakes to Avoid
| Mistake | Problem | Fix |
|---------|---------|-----|
| Adding everything to one notebook | No isolation between projects | Create separate notebooks for different topics |
| Expecting AI to know your context | Questions get generic answers | Describe your research focus in chat context |
| Forgetting to cite sources | You can't verify claims | Click citations to check source chunks |
| Using Chat for one-time questions | Slower than Ask | Use Ask for comprehensive Q&A, Chat for exploration |
| Adding huge PDFs without chunking | Slow processing, poor search | Break into multiple smaller sources if possible |
| Using same context for all chats | Expensive, unfocused | Adjust context level for each chat |
| Ignoring vector search | Only finding exact keywords | Use vector search to explore conceptually |
---
## Next Steps
1. **Follow each guide** in order (sources → notes → chat → podcasts → search)
2. **Create your first notebook** with real content
3. **Practice each feature** with your own research
4. **Return to CORE-CONCEPTS** if you need to understand the "why"
---
## Getting Help
- **Feature not working?** → Check the feature's guide (look for "Troubleshooting" section)
- **Error message?** → Check [6-TROUBLESHOOTING](../6-TROUBLESHOOTING/index.md)
- **Understanding how something works?** → Check [2-CORE-CONCEPTS](../2-CORE-CONCEPTS/index.md)
- **Setting up for the first time?** → Go back to [1-INSTALLATION](../1-INSTALLATION/index.md)
- **For developers** → See [7-DEVELOPMENT](../7-DEVELOPMENT/index.md)
---
**Ready to start?** Pick the guide for what you want to do first!
docs/3-USER-GUIDE/interface-overview.md
+377
View File
@@ -0,0 +1,377 @@
# Interface Overview - Finding Your Way Around
Open Notebook uses a clean three-panel layout. This guide shows you where everything is.
---
## The Main Layout
```
┌─────────────────────────────────────────────────────────────┐
│ [Logo] Notebooks Search Podcasts Models Settings │
├──────────────┬──────────────┬───────────────────────────────┤
│ │ │ │
│ SOURCES │ NOTES │ CHAT │
│ │ │ │
│ Your docs │ Your │ Talk to AI about │
│ PDFs, URLs │ insights │ your sources │
│ Videos │ summaries │ │
│ │ │ │
│ [+Add] │ [+Write] │ [Type here...] │
│ │ │ │
└──────────────┴──────────────┴───────────────────────────────┘
```
---
## Navigation Bar
The top navigation takes you to main sections:
| Icon | Page | What It Does |
|------|------|--------------|
| **Notebooks** | Main workspace | Your research projects |
| **Search** | Ask & Search | Query across all notebooks |
| **Podcasts** | Audio generation | Manage podcast profiles |
| **Models** | AI configuration | Set up providers and models |
| **Settings** | Preferences | App configuration |
---
## Left Panel: Sources
Your research materials live here.
### What You'll See
```
┌─────────────────────────┐
│ Sources (5) │
│ [+ Add Source] │
├─────────────────────────┤
│ ┌─────────────────┐ │
│ │ 📄 Paper.pdf │ │
│ │ 🟢 Full Content │ │
│ │ [⋮ Menu] │ │
│ └─────────────────┘ │
│ │
│ ┌─────────────────┐ │
│ │ 🔗 Article URL │ │
│ │ 🟡 Summary Only │ │
│ │ [⋮ Menu] │ │
│ └─────────────────┘ │
└─────────────────────────┘
```
### Source Card Elements
- **Icon** - File type (PDF, URL, video, etc.)
- **Title** - Document name
- **Context indicator** - What AI can see:
- 🟢 Full Content
- 🟡 Summary Only
- ⛔ Not in Context
- **Menu (⋮)** - Edit, transform, delete
### Add Source Button
Click to add:
- File upload (PDF, DOCX, etc.)
- Web URL
- YouTube video
- Plain text
---
## Middle Panel: Notes
Your insights and AI-generated content.
### What You'll See
```
┌─────────────────────────┐
│ Notes (3) │
│ [+ Write Note] │
├─────────────────────────┤
│ ┌─────────────────┐ │
│ │ 📝 My Analysis │ │
│ │ Manual note │ │
│ │ Jan 3, 2026 │ │
│ └─────────────────┘ │
│ │
│ ┌─────────────────┐ │
│ │ 🤖 Summary │ │
│ │ From transform │ │
│ │ Jan 2, 2026 │ │
│ └─────────────────┘ │
└─────────────────────────┘
```
### Note Card Elements
- **Icon** - Note type (manual 📝 or AI 🤖)
- **Title** - Note name
- **Origin** - How it was created
- **Date** - When created
### Write Note Button
Click to:
- Create manual note
- Add your own insights
- Markdown supported
---
## Right Panel: Chat
Your AI conversation space.
### What You'll See
```
┌───────────────────────────────┐
│ Chat │
│ Session: Research Discussion │
│ [+ New Session] [Sessions ▼] │
├───────────────────────────────┤
│ │
│ You: What's the main │
│ finding? │
│ │
│ AI: Based on the paper [1], │
│ the main finding is... │
│ [Save as Note] │
│ │
│ You: Tell me more about │
│ the methodology. │
│ │
├───────────────────────────────┤
│ Context: 3 sources (12K tok) │
├───────────────────────────────┤
│ [Type your message...] [↑] │
└───────────────────────────────┘
```
### Chat Elements
- **Session selector** - Switch between conversations
- **Message history** - Your conversation
- **Save as Note** - Keep good responses
- **Context indicator** - What AI can see
- **Input field** - Type your questions
---
## Context Indicators
These show what AI can access:
### Token Counter
```
Context: 3 sources (12,450 tokens)
↑ ↑
Sources Approximate cost indicator
included
```
### Per-Source Indicators
| Indicator | Meaning | AI Access |
|-----------|---------|-----------|
| 🟢 Full Content | Complete text | Everything |
| 🟡 Summary Only | AI summary | Key points only |
| ⛔ Not in Context | Excluded | Nothing |
Click any source to change its context level.
---
## Podcasts Tab
Inside a notebook, switch to Podcasts:
```
┌───────────────────────────────┐
│ [Chat] [Podcasts] │
├───────────────────────────────┤
│ Episode Profile: [Select ▼] │
│ │
│ Speakers: │
│ ├─ Host: Alex (voice model) │
│ └─ Guest: Sam (voice model) │
│ │
│ Include: │
│ ☑ Paper.pdf │
│ ☑ My Analysis (note) │
│ ☐ Background article │
│ │
│ [Generate Podcast] │
└───────────────────────────────┘
```
---
## Settings Page
Access via navigation bar → Settings:
### Key Sections
| Section | What It Controls |
|---------|------------------|
| **Processing** | Document and URL extraction engines |
| **Embedding** | Auto-embed settings |
| **Files** | Auto-delete uploads after processing |
| **YouTube** | Preferred transcript languages |
---
## Models Page
Configure AI providers:
```
┌───────────────────────────────────────┐
│ Models │
├───────────────────────────────────────┤
│ Language Models │
│ ┌─────────────────────────────────┐ │
│ │ GPT-4o (OpenAI) [Edit] │ │
│ │ Claude Sonnet (Anthropic) │ │
│ │ Llama 3.3 (Ollama) [⭐] │ │
│ └─────────────────────────────────┘ │
│ [+ Add Model] │
│ │
│ Embedding Models │
│ ┌─────────────────────────────────┐ │
│ │ text-embedding-3-small [⭐] │ │
│ └─────────────────────────────────┘ │
│ │
│ Text-to-Speech │
│ ┌─────────────────────────────────┐ │
│ │ OpenAI TTS [⭐] │ │
│ │ Google TTS │ │
│ └─────────────────────────────────┘ │
└───────────────────────────────────────┘
```
- **⭐** = Default model for that category
- **[Edit]** = Modify configuration
- **[+ Add]** = Add new model
---
## Search Page
Query across all notebooks:
```
┌───────────────────────────────────────┐
│ Search │
├───────────────────────────────────────┤
│ [What are you looking for? ] [🔍] │
│ │
│ Search type: [Text ▼] [Vector ▼] │
│ Search in: [Sources] [Notes] │
├───────────────────────────────────────┤
│ Results (15) │
│ │
│ 📄 Paper.pdf - Notebook: Research │
│ "...the transformer model..." │
│ │
│ 📝 My Analysis - Notebook: Research │
│ "...key findings include..." │
└───────────────────────────────────────┘
```
---
## Common Actions
### Create a Notebook
```
Notebooks page → [+ New Notebook] → Enter name → Create
```
### Add a Source
```
Inside notebook → [+ Add Source] → Choose type → Upload/paste → Wait for processing
```
### Ask a Question
```
Inside notebook → Chat panel → Type question → Enter → Read response
```
### Save AI Response
```
Get good response → Click [Save as Note] → Edit title → Save
```
### Change Context Level
```
Click source → Context dropdown → Select level → Changes apply immediately
```
### Generate Podcast
```
Podcasts tab → Select profile → Choose sources → [Generate] → Wait → Download
```
---
## Keyboard Shortcuts
| Key | Action |
|-----|--------|
| `Enter` | Send chat message |
| `Shift + Enter` | New line in chat |
| `Escape` | Close dialogs |
| `Ctrl/Cmd + F` | Browser find |
---
## Mobile View
On smaller screens, the three-panel layout stacks vertically:
```
┌─────────────────┐
│ SOURCES │
│ (tap to expand)
├─────────────────┤
│ NOTES │
│ (tap to expand)
├─────────────────┤
│ CHAT │
│ (always visible)
└─────────────────┘
```
- Panels collapse to save space
- Tap headers to expand/collapse
- Chat remains accessible
- Full functionality preserved
---
## Tips for Efficient Navigation
1. **Use keyboard** - Enter sends messages, Escape closes dialogs
2. **Context first** - Set source context before chatting
3. **Sessions** - Create new sessions for different topics
4. **Search globally** - Use Search page to find across all notebooks
5. **Models page** - Bookmark your preferred models
---
Now you know where everything is. Start with [Adding Sources](adding-sources.md) to begin your research!
docs/3-USER-GUIDE/search.md
+475
View File
@@ -0,0 +1,475 @@
# Search Effectively - Finding What You Need
Search is your gateway into your research. This guide covers two search modes and when to use each.
---
## Quick-Start: Find Something
### Simple Search
```
1. Go to your notebook
2. Type in search box
3. See results (both sources and notes)
4. Click result to view source/note
5. Done!
That works for basic searches.
But you can do much better...
```
---
## Two Search Modes Explained
Open Notebook has two fundamentally different search approaches.
### Search Type 1: TEXT SEARCH (Keyword Matching)
**How it works:**
- You search for words: "transformer"
- System finds chunks containing "transformer"
- Ranked by relevance: frequency, position, context
**Speed:** Very fast (instant)
**When to use:**
- You remember exact words or phrases
- You're looking for specific terms
- You want precise keyword matches
- You need exact quotes
**Example:**
```
Search: "attention mechanism"
Results:
1. "The attention mechanism allows..." (perfect match)
2. "Attention and other mechanisms..." (partial match)
3. "How mechanisms work in attention..." (includes words separately)
All contain "attention" AND "mechanism"
Ranked by how close together they are
```
**What it finds:**
- Exact phrases: "transformer model"
- Individual words: transformer OR model (too broad)
- Names: "Vaswani et al."
- Numbers: "1994", "GPT-4"
- Technical terms: "LSTM", "convolution"
**What it doesn't find:**
- Similar words: searching "attention" won't find "focus"
- Synonyms: searching "large" won't find "big"
- Concepts: searching "similarity" won't find "likeness"
---
### Search Type 2: VECTOR SEARCH (Semantic/Concept Matching)
**How it works:**
- Your search converted to embedding (vector)
- All chunks converted to embeddings
- System finds most similar embeddings
- Ranked by semantic similarity
**Speed:** A bit slower (1-2 seconds)
**When to use:**
- You're exploring a concept
- You don't know exact words
- You want semantically similar content
- You're discovering, not searching
**Example:**
```
Search: "What's the mechanism for understanding in models?"
(Notice: No chunk likely says exactly that)
Results:
1. "Mechanistic interpretability allows understanding..." (semantic match)
2. "Feature attribution reveals how models work..." (conceptually similar)
3. "Attention visualization shows model decisions..." (same topic)
None contain your exact words
But all are semantically related
```
**What it finds:**
- Similar concepts: "understanding" + "interpretation" + "explainability" (all related)
- Paraphrases: "big" and "large" (same meaning)
- Related ideas: "safety" relates to "alignment" (connected concepts)
- Analogies: content about biological learning when searching "learning"
**What it doesn't find:**
- Exact keywords: if you search a rare word, vector search might miss it
- Specific numbers: "1994" vs "1993" are semantically different
- Technical jargon: "LSTM" and "RNN" are different even if related
---
## Decision: Text Search vs. Vector Search?
```
Question: "Do I remember the exact words?"
→ YES: Use TEXT SEARCH
Example: "I remember the paper said 'attention is all you need'"
→ NO: Use VECTOR SEARCH
Example: "I'm looking for content about how models process information"
→ UNSURE: Try TEXT SEARCH first (faster)
If no results, try VECTOR SEARCH
Text search: "I know what I'm looking for"
Vector search: "I'm exploring an idea"
```
---
## Step-by-Step: Using Each Search
### Text Search
```
1. Go to search box
2. Type your keywords: "transformer", "attention", "2017"
3. Press Enter
4. Results appear (usually instant)
5. Click result to see context
Results show:
- Which source contains it
- How many times it appears
- Relevance score
- Preview of surrounding text
```
### Vector Search
```
1. Go to search box
2. Type your concept: "How do models understand language?"
3. Choose "Vector Search" from dropdown
4. Press Enter
5. Results appear (1-2 seconds)
6. Click result to see context
Results show:
- Semantically related chunks
- Similarity score (higher = more related)
- Preview of surrounding text
- Different sources mixed together
```
---
## The Ask Feature (Automated Search)
Ask is different from simple search. It automatically searches, synthesizes, and answers.
### How Ask Works
```
Stage 1: QUESTION UNDERSTANDING
"Compare the approaches in my papers"
→ System: "This asks for comparison"
Stage 2: SEARCH STRATEGY
→ System: "I should search for each approach separately"
Stage 3: PARALLEL SEARCHES
→ Search 1: "Approach in paper A"
→ Search 2: "Approach in paper B"
(Multiple searches happen at once)
Stage 4: ANALYSIS & SYNTHESIS
→ Per-result analysis: "Based on paper A, the approach is..."
→ Per-result analysis: "Based on paper B, the approach is..."
→ Final synthesis: "Comparing A and B: A differs from B in..."
Result: Comprehensive answer, not just search results
```
### When to Use Ask vs. Simple Search
| Task | Use | Why |
|------|-----|-----|
| "Find the quote about X" | **TEXT SEARCH** | Need exact words |
| "What does source A say about X?" | **TEXT SEARCH** | Direct, fast answer |
| "Find content about X" | **VECTOR SEARCH** | Semantic discovery |
| "Compare A and B" | **ASK** | Comprehensive synthesis |
| "What's the big picture?" | **ASK** | Full analysis needed |
| "How do these sources relate?" | **ASK** | Cross-source synthesis |
| "I remember something about X" | **TEXT SEARCH** | Recall memory |
| "I'm exploring the topic of X" | **VECTOR SEARCH** | Discovery mode |
---
## Advanced Search Strategies
### Strategy 1: Simple Search with Follow-Up
```
1. Text search: "attention mechanism"
Results: 50 matches
2. Too many. Follow up with vector search:
"Why is attention useful?" (concept search)
Results: Most relevant papers/notes
3. Better results with less noise
```
### Strategy 2: Ask for Comprehensive, Then Search for Details
```
1. Ask: "What are the main approaches to X?"
Result: Comprehensive answer about A, B, C
2. Use that to identify specific sources
3. Text search in those specific sources:
"Why did they choose method X?"
Result: Detailed information
```
### Strategy 3: Vector Search for Discovery, Text for Verification
```
1. Vector search: "How do transformers generalize?"
Results: Related conceptual papers
2. Skim to understand landscape
3. Text search in promising sources:
"generalization", "extrapolation", "transfer"
Results: Specific passages to read carefully
```
### Strategy 4: Combine Search with Chat
```
1. Vector search: "What's new in AI 2026?"
Results: Latest papers
2. Go to Chat
3. Add those papers to context
4. Ask detailed follow-up questions
5. Get deep analysis of results
```
---
## Search Quality Issues & Fixes
### Getting No Results
| Problem | Cause | Solution |
|---------|-------|----------|
| Text search: no results | Word doesn't appear | Try vector search instead |
| Vector search: no results | Concept not in content | Try broader search term |
| Both empty | Content not in notebook | Add sources to notebook |
| | Sources not processed | Wait for processing to complete |
### Getting Too Many Results
| Problem | Cause | Solution |
|---------|-------|----------|
| 1000+ results | Search too broad | Be more specific |
| | All sources | Filter by source |
| | Keyword matches rare words | Use vector search instead |
### Getting Wrong Results
| Problem | Cause | Solution |
|---------|-------|----------|
| Results irrelevant | Search term has multiple meanings | Provide more context |
| | Using text search for concepts | Try vector search |
| Different meaning | Homonym (word means multiple things) | Add context (e.g., "attention mechanism") |
### Getting Low Quality Results
| Problem | Cause | Solution |
|---------|-------|----------|
| Results don't match intent | Vague search term | Be specific ("Who invented X?" vs "X") |
| | Concept not well-represented | Add more sources on that topic |
| | Vector embedding not trained on domain | Use text search as fallback |
---
## Tips for Better Searches
### For Text Search
1. **Be specific** — "attention mechanism" not just "attention"
2. **Use exact phrases** — Put quotes around: "attention is all you need"
3. **Include context** — "LSTM vs attention" not just "attention"
4. **Use technical terms** — These are usually more precise
5. **Try synonyms** — If first search fails, try related terms
### For Vector Search
1. **Ask a question** — "What's the best way to X?" is better than "best way"
2. **Use natural language** — Explain what you're looking for
3. **Be specific about intent** — "Compare X and Y" not "X and Y"
4. **Include context** — "In machine learning, how..." vs just "how..."
5. **Think conceptually** — What idea are you exploring?
### General Tips
1. **Start broad, then narrow** — "AI papers" → "transformers" → "attention mechanism"
2. **Try both search types** — Each finds different things
3. **Use Ask for complex questions** — Don't just search
4. **Save good results as notes** — Create knowledge base
5. **Filter by source if needed** — "Search in Paper A only"
---
## Search Examples
### Example 1: Finding a Specific Fact
**Goal:** "Find the date the transformer was introduced"
```
Step 1: Text search
"transformer 2017" (or year you remember)
If that works: Done!
If no results: Try
"attention is all you need" (famous paper title)
Check result for exact date
```
### Example 2: Exploring a Concept
**Goal:** "Find content about alignment interpretability"
```
Step 1: Vector search
"How do we make AI interpretable?"
Results: Papers on interpretability, transparency, alignment
Step 2: Review results
See which papers are most relevant
Step 3: Deep dive
Go to Chat, add top 2-3 papers
Ask detailed questions about alignment
```
### Example 3: Comprehensive Answer
**Goal:** "How do different approaches to AI safety compare?"
```
Step 1: Ask
"Compare the main approaches to AI safety in my sources"
Result: Comprehensive analysis comparing approaches
Step 2: Identify sources
From answer, see which papers were most relevant
Step 3: Deep dive
Text search in those papers:
"limitations", "critiques", "open problems"
Step 4: Save as notes
Create comparison note from Ask result
```
### Example 4: Finding Pattern
**Goal:** "Find all papers mentioning transformers"
```
Step 1: Text search
"transformer"
Results: All papers mentioning "transformer"
Step 2: Vector search
"neural network architecture for sequence processing"
Results: Papers that don't say "transformer" but discuss similar concept
Step 3: Combine
Union of text + vector results shows full landscape
Step 4: Analyze
Go to Chat with all results
Ask: "What's common across all these?"
```
---
## Search in the Workflow
How search fits with other features:
```
SOURCES
↓
SEARCH (find what matters)
├─ Text search (precise)
├─ Vector search (exploration)
└─ Ask (comprehensive)
↓
CHAT (explore with follow-ups)
↓
TRANSFORMATIONS (batch extract)
↓
NOTES (save insights)
```
### Workflow Example
```
1. Add 10 papers to notebook
2. Search: "What's the state of the art?"
(Vector search explores landscape)
3. Ask: "Compare these 3 approaches"
(Comprehensive synthesis)
4. Chat: Deep questions about winner
(Follow-up exploration)
5. Save best insights as notes
(Knowledge capture)
6. Transform remaining papers
(Batch extraction for later)
7. Create podcast from notes + sources
(Share findings)
```
---
## Summary: Know Your Search
**TEXT SEARCH** — "I know what I'm looking for"
- Fast, precise, keyword-based
- Use when you remember exact words/phrases
- Best for: Finding specific facts, quotes, technical terms
- Speed: Instant
**VECTOR SEARCH** — "I'm exploring an idea"
- Slow-ish, concept-based, semantic
- Use when you're discovering connections
- Best for: Concept exploration, related ideas, synonyms
- Speed: 1-2 seconds
**ASK** — "I want a comprehensive answer"
- Auto-searches, auto-analyzes, synthesizes
- Use for complex questions needing multiple sources
- Best for: Comparisons, big-picture questions, synthesis
- Speed: 10-30 seconds
Pick the right tool for your search goal, and you'll find what you need faster.
docs/3-USER-GUIDE/transformations.md
+402
View File
@@ -0,0 +1,402 @@
# Transformations - Batch Processing Your Sources
Transformations apply the same analysis to multiple sources at once. Instead of asking the same question repeatedly, define a template and run it across your content.
---
## When to Use Transformations
| Use Transformations When | Use Chat Instead When |
|-------------------------|----------------------|
| Same analysis on many sources | One-off questions |
| Consistent output format needed | Exploratory conversation |
| Batch processing | Follow-up questions needed |
| Creating structured notes | Context changes between questions |
**Example**: You have 10 papers and want a summary of each. Transformation does it in one operation.
---
## Quick Start: Your First Transformation
```
1. Go to your notebook
2. Click "Transformations" in navigation
3. Select a built-in template (e.g., "Summary")
4. Select sources to transform
5. Click "Apply"
6. Wait for processing
7. New notes appear automatically
```
---
## Built-in Transformations
Open Notebook includes ready-to-use templates:
### Summary
```
What it does: Creates a 200-300 word overview
Output: Key points, main arguments, conclusions
Best for: Quick reference, getting the gist
```
### Key Concepts
```
What it does: Extracts main ideas and terminology
Output: List of concepts with explanations
Best for: Learning new topics, building vocabulary
```
### Methodology
```
What it does: Extracts research approach
Output: How the study was conducted
Best for: Academic papers, research review
```
### Takeaways
```
What it does: Extracts actionable insights
Output: What you should do with this information
Best for: Business documents, practical guides
```
### Questions
```
What it does: Generates questions the source raises
Output: Open questions, gaps, follow-up research
Best for: Literature review, research planning
```
---
## Creating Custom Transformations
### Step-by-Step
```
1. Go to "Transformations" page
2. Click "Create New"
3. Enter a name: "Academic Paper Analysis"
4. Write your prompt template:
"Analyze this academic paper and extract:
1. **Research Question**: What problem does this address?
2. **Hypothesis**: What did they predict?
3. **Methodology**: How did they test it?
4. **Key Findings**: What did they discover? (numbered list)
5. **Limitations**: What caveats do the authors mention?
6. **Future Work**: What do they suggest next?
Be specific and cite page numbers where possible."
5. Click "Save"
6. Your transformation appears in the list
```
### Prompt Template Tips
**Be specific about format:**
```
Good: "List 5 key points as bullet points"
Bad: "What are the key points?"
```
**Request structure:**
```
Good: "Create sections for: Summary, Methods, Results"
Bad: "Tell me about this paper"
```
**Ask for citations:**
```
Good: "Cite page numbers for each claim"
Bad: (no citation request)
```
**Set length expectations:**
```
Good: "In 200-300 words, summarize..."
Bad: "Summarize this"
```
---
## Applying Transformations
### To a Single Source
```
1. In Sources panel, click source menu (⋮)
2. Select "Transform"
3. Choose transformation template
4. Click "Apply"
5. Note appears when done
```
### To Multiple Sources (Batch)
```
1. Go to Transformations page
2. Select your template
3. Check multiple sources
4. Click "Apply to Selected"
5. Processing runs in parallel
6. One note per source created
```
### Processing Time
| Sources | Typical Time |
|---------|--------------|
| 1 source | 30 seconds - 1 minute |
| 5 sources | 2-3 minutes |
| 10 sources | 4-5 minutes |
| 20+ sources | 8-10 minutes |
Processing runs in background. You can continue working.
---
## Transformation Examples
### Literature Review Template
```
Name: Literature Review Entry
Prompt:
"For this research paper, create a literature review entry:
**Citation**: [Author(s), Year, Title, Journal]
**Research Question**: What problem is addressed?
**Methodology**: What approach was used?
**Sample**: What population/data was studied?
**Key Findings**:
1. [Finding with page citation]
2. [Finding with page citation]
3. [Finding with page citation]
**Strengths**: What did this study do well?
**Limitations**: What are the gaps?
**Relevance**: How does this connect to my research?
Keep each section to 2-3 sentences."
```
### Meeting Notes Template
```
Name: Meeting Summary
Prompt:
"From this meeting transcript, extract:
**Attendees**: Who was present
**Date/Time**: When it occurred
**Key Decisions**: What was decided (numbered)
**Action Items**:
- [ ] Task (Owner, Due Date)
**Open Questions**: Unresolved issues
**Next Steps**: What happens next
Format as clear, scannable notes."
```
### Competitor Analysis Template
```
Name: Competitor Analysis
Prompt:
"Analyze this company/product document:
**Company**: Name and overview
**Products/Services**: What they offer
**Target Market**: Who they serve
**Pricing**: If available
**Strengths**: Competitive advantages
**Weaknesses**: Gaps or limitations
**Opportunities**: How we compare
**Threats**: What they do better
Be objective and cite specific details."
```
### Technical Documentation Template
```
Name: API Documentation Summary
Prompt:
"Extract from this technical document:
**Overview**: What does this do? (1-2 sentences)
**Authentication**: How to authenticate
**Key Endpoints**:
- Endpoint 1: [method] [path] - [purpose]
- Endpoint 2: ...
**Common Parameters**: Frequently used params
**Rate Limits**: If mentioned
**Error Codes**: Key error responses
**Example Usage**: Simple code example if possible
Keep technical but concise."
```
---
## Managing Transformations
### Edit a Transformation
```
1. Go to Transformations page
2. Find your template
3. Click "Edit"
4. Modify the prompt
5. Click "Save"
```
### Delete a Transformation
```
1. Go to Transformations page
2. Find the template
3. Click "Delete"
4. Confirm
```
### Reorder/Organize
Built-in transformations appear first, then custom ones alphabetically.
---
## Transformation Output
### Where Results Go
- Each source produces one note
- Notes appear in your notebook's Notes panel
- Notes are tagged with transformation name
- Original source is linked
### Note Naming
```
Default: "[Transformation Name] - [Source Title]"
Example: "Summary - Research Paper 2025.pdf"
```
### Editing Output
```
1. Click the generated note
2. Click "Edit"
3. Refine the content
4. Save
```
---
## Best Practices
### Template Design
1. **Start specific** - Vague prompts give vague results
2. **Use formatting** - Headings, bullets, numbered lists
3. **Request citations** - Make results verifiable
4. **Set length** - Prevent overly long or short output
5. **Test first** - Run on one source before batch
### Source Selection
1. **Similar content** - Same transformation on similar sources
2. **Reasonable size** - Very long sources may need splitting
3. **Processed status** - Ensure sources are fully processed
### Quality Control
1. **Review samples** - Check first few outputs before trusting batch
2. **Edit as needed** - Transformations are starting points
3. **Iterate prompts** - Refine based on results
---
## Common Issues
### Generic Output
**Problem**: Results are too vague
**Solution**: Make prompt more specific, add format requirements
### Missing Information
**Problem**: Key details not extracted
**Solution**: Explicitly ask for what you need in prompt
### Inconsistent Format
**Problem**: Each note looks different
**Solution**: Add clear formatting instructions to prompt
### Too Long/Short
**Problem**: Output doesn't match expectations
**Solution**: Specify word count or section lengths
### Processing Fails
**Problem**: Transformation doesn't complete
**Solution**:
- Check source is processed
- Try shorter/simpler prompt
- Process sources individually
---
## Transformations vs. Chat vs. Ask
| Feature | Transformations | Chat | Ask |
|---------|----------------|------|-----|
| **Input** | Predefined template | Your questions | Your question |
| **Scope** | One source at a time | Selected sources | Auto-searched |
| **Output** | Structured note | Conversation | Comprehensive answer |
| **Best for** | Batch processing | Exploration | One-shot answers |
| **Follow-up** | Run again | Ask more | New query |
---
## Summary
```
Transformations = Batch AI Processing
How to use:
1. Define template (or use built-in)
2. Select sources
3. Apply transformation
4. Get structured notes
When to use:
- Same analysis on many sources
- Consistent output needed
- Building structured knowledge base
- Saving time on repetitive tasks
Tips:
- Be specific in prompts
- Request formatting
- Test before batch
- Edit output as needed
```
Transformations turn repetitive analysis into one-click operations. Define once, apply many times.
docs/3-USER-GUIDE/working-with-notes.md
+581
View File
@@ -0,0 +1,581 @@
# Working with Notes - Capturing and Organizing Insights
Notes are your processed knowledge. This guide covers how to create, organize, and use them effectively.
---
## What Are Notes?
Notes are your **research output** — the insights you capture from analyzing sources. They can be:
- **Manual** — You write them yourself
- **AI-Generated** — From Chat responses, Ask results, or Transformations
- **Hybrid** — AI insight + your edits and additions
Unlike sources (which never change), notes are mutable — you edit, refine, and organize them.
---
## Quick-Start: Create Your First Note
### Method 1: Manual Note (Write Yourself)
```
1. In your notebook, go to "Notes" section
2. Click "Create New Note"
3. Give it a title: "Key insights from source X"
4. Write your content (markdown supported)
5. Click "Save"
6. Done! Note appears in your notebook
```
### Method 2: Save from Chat
```
1. Have a Chat conversation
2. Get a good response from AI
3. Click "Save as Note" button under response
4. Give the note a title
5. Add any additional context
6. Click "Save"
7. Done! Note appears in your notebook
```
### Method 3: Apply Transformation
```
1. Go to "Transformations"
2. Select a template (or create custom)
3. Click "Apply to sources"
4. Select which sources to transform
5. Wait for processing
6. New notes automatically appear
7. Done! Each source produces one note
```
---
## Creating Manual Notes
### Basic Structure
```
Title: "What you're capturing"
(Make it descriptive)
Content:
- Main points
- Your analysis
- Questions raised
- Next steps
Metadata:
- Tags: How to categorize
- Related sources: Which documents influenced this
- Date: Auto-added when created
```
### Markdown Support
You can format notes with markdown:
```markdown
# Heading
## Subheading
### Sub-subheading
**Bold text** for emphasis
*Italic text* for secondary emphasis
- Bullet lists
- Like this
1. Numbered lists
2. Like this
> Quotes and important callouts
[Links work](https://example.com)
```
### Example Note Structure
```markdown
# Key Findings from "AI Safety Paper 2025"
## Main Argument
The paper argues that X approach is better than Y because...
## Methodology
The authors use [methodology] to test this hypothesis.
## Key Results
- Result 1: [specific finding with citation]
- Result 2: [specific finding with citation]
- Result 3: [specific finding with citation]
## Gaps & Limitations
1. The paper assumes X, which might not hold in Y scenario
2. Limited to Z population/domain
3. Future work needed on A, B, C
## My Thoughts
- This connects to previous research on...
- Potential application in...
## Next Steps
- [ ] Read the referenced paper on X
- [ ] Find similar studies on Y
- [ ] Discuss implications with team
```
---
## AI-Generated Notes: Three Sources
### 1. Save from Chat
```
Workflow:
Chat → Good response → "Save as Note"
→ Edit if needed → Save
When to use:
- AI response answers your question well
- You want to keep the answer for reference
- You're building a knowledge base from conversations
Quality:
- Quality = quality of your Chat question
- Better context = better responses = better notes
- Ask specific questions for useful notes
```
### 2. Save from Ask
```
Workflow:
Ask → Comprehensive answer → "Save as Note"
→ Edit if needed → Save
When to use:
- You need a one-time comprehensive answer
- You want to save the synthesized result
- Building a knowledge base of comprehensive answers
Quality:
- System automatically found relevant sources
- Results already have citations
- Often higher quality than Chat (more thorough)
```
### 3. Transformations (Batch Processing)
```
Workflow:
Define transformation → Apply to sources → Notes auto-created
→ Review & edit → Organize
Example Transformation:
Template: "Extract: main argument, methodology, key findings"
Apply to: 5 sources
Result: 5 new notes with consistent structure
When to use:
- Same extraction from many sources
- Building structured knowledge base
- Creating consistent summaries
```
---
## Using Transformations for Batch Insights
### Built-in Transformations
Open Notebook comes with presets:
**Summary**
```
Extracts: Main points, key arguments, conclusions
Output: 200-300 word summary of source
Best for: Quick reference summaries
```
**Key Concepts**
```
Extracts: Main ideas, concepts, terminology
Output: List of concepts with explanations
Best for: Learning and terminology
```
**Methodology**
```
Extracts: Research approach, methods, data
Output: How the research was conducted
Best for: Academic sources, methodology review
```
**Takeaways**
```
Extracts: Actionable insights, recommendations
Output: What you should do with this information
Best for: Practical/business sources
```
### How to Apply Transformation
```
1. Go to "Transformations"
2. Select a template
3. Click "Apply"
4. Select which sources (one or many)
5. Wait for processing (usually 30 seconds - 2 minutes)
6. New notes appear in your notebook
7. Edit if needed
```
### Create Custom Transformation
```
1. Click "Create Custom Transformation"
2. Write your extraction template:
Example:
"For this academic paper, extract:
- Central research question
- Hypothesis tested
- Methodology used
- Key findings (numbered)
- Limitations acknowledged
- Recommendations for future work"
3. Click "Save Template"
4. Apply to one or many sources
5. System generates notes with consistent structure
```
---
## Organizing Notes
### Naming Conventions
**Option 1: Date-based**
```
2026-01-03 - Key points from X source
2026-01-04 - Comparison between A and B
Benefit: Easy to see what you did when
```
**Option 2: Topic-based**
```
AI Safety - Alignment approaches
AI Safety - Interpretability research
Benefit: Groups by subject matter
```
**Option 3: Type-based**
```
SUMMARY: Paper on X
QUESTION: What about Y?
INSIGHT: Connection between Z and W
Benefit: Easy to filter by type
```
**Option 4: Source-based**
```
From: Paper A - Main insights
From: Video B - Interesting implications
Benefit: Easy to trace back to sources
```
**Best practice:** Combine approaches
```
[Date] [Source] - [Topic] - [Type]
2026-01-03 - Paper A - AI Safety - Takeaways
```
### Using Tags
Tags are labels for categorization. Add them when creating notes:
```
Example tags:
- "primary-research" (direct source analysis)
- "background" (supporting material)
- "methodology" (about research methods)
- "insights" (your original thinking)
- "questions" (open questions raised)
- "follow-up" (needs more work)
- "published" (ready to share/use)
```
**Benefits of tags:**
- Filter notes by tag
- Find all notes of a type
- Organize workflow (e.g., find all "follow-up" notes)
### Note Linking & References
You can reference sources within notes:
```markdown
# Analysis of Paper A
As shown in Paper A (see "main argument" section),
the authors argue that...
## Related Sources
- Paper B discusses similar approach
- Video C shows practical application
- My note on "Comparative analysis" has more
```
---
## Editing and Refining Notes
### Improving AI-Generated Notes
```
AI Note:
"The paper discusses machine learning"
What you might change:
"The paper proposes a supervised learning approach
to classification problems, using neural networks
with attention mechanisms (see pp. 15-18)."
How to edit:
1. Click note
2. Click "Edit"
3. Refine the content
4. Click "Save"
```
### Adding Citations
```
When saving from Chat/Ask:
- Citations auto-added
- Shows which sources informed answer
- You can verify by clicking
When manual notes:
- Add manually: "From Paper A, page 15: ..."
- Or reference: "As discussed in [source]"
```
---
## Searching Your Notes
Notes are fully searchable:
### Text Search
```
Find exact phrase: "attention mechanism"
Results: All notes containing that phrase
Use when: Looking for specific terms or quotes
```
### Vector/Semantic Search
```
Find concept: "How do models understand?"
Results: Notes about interpretability, mechanistic understanding, etc.
Use when: Exploring conceptually (words not exact)
```
### Combined Search
```
Text search notes → Find keyword matches
Vector search notes → Find conceptual matches
Both work across sources + notes together
```
---
## Exporting and Sharing Notes
### Options
**Copy to clipboard**
```
Click "Share" → "Copy" → Paste anywhere
Good for: Sharing one note via email/chat
```
**Export as Markdown**
```
Click "Share" → "Export as MD" → Saves as .md file
Good for: Sharing with others, version control
```
**Create note collection**
```
Select multiple notes → "Export collection"
→ Creates organized markdown document
Good for: Sharing a topic overview
```
**Publish to web**
```
Click "Publish" → Get shareable link
Good for: Publishing publicly (if desired)
```
---
## Organizing Your Notebook's Notes
### By Research Phase
**Phase 1: Discovery**
- Initial summaries
- Questions raised
- Interesting findings
**Phase 2: Deep Dive**
- Detailed analysis
- Comparative insights
- Methodology reviews
**Phase 3: Synthesis**
- Connections across sources
- Original thinking
- Conclusions
### By Content Type
**Summaries**
- High-level overviews
- Generated by transformations
- Quick reference
**Questions**
- Open questions
- Things to research more
- Gaps to fill
**Insights**
- Your original analysis
- Connections made
- Conclusions reached
**Tasks**
- Follow-up research
- Sources to add
- People to contact
---
## Using Notes in Other Features
### In Chat
```
You can reference notes:
"Based on my note 'Key findings from A',
how does this compare to B?"
Notes become part of context.
Treated like sources but smaller/more focused.
```
### In Transformations
```
Notes can be transformed:
1. Select notes as input
2. Apply transformation
3. Get new derived notes
Example: Transform 5 analysis notes → Create synthesis
```
### In Podcasts
```
Notes are used to create podcast content:
1. Generate podcast for notebook
2. System includes notes in content selection
3. Notes become part of episode outline
```
---
## Best Practices
### For Manual Notes
1. **Write clearly** — Future you will appreciate it
2. **Add context** — Why this matters, not just what it says
3. **Link to sources** — You can verify later
4. **Date them** — Track your thinking over time
5. **Tag immediately** — Don't defer organization
### For AI-Generated Notes
1. **Review before saving** — Verify quality
2. **Edit for clarity** — AI might miss nuance
3. **Add your thoughts** — Make it your own
4. **Include citations** — Understand sources
5. **Organize right away** — While context is fresh
### For Organization
1. **Consistent naming** — Your future self will thank you
2. **Tag everything** — Makes filtering later much easier
3. **Link related notes** — Create knowledge network
4. **Review periodically** — Refactor as understanding evolves
5. **Archive old notes** — Keep working space clean
---
## Common Mistakes
| Mistake | Problem | Solution |
|---------|---------|----------|
| Save every Chat response | Notebook becomes cluttered with low-quality notes | Only save good responses that answer your questions |
| Don't add tags | Can't find notes later | Tag immediately when creating |
| Poor note titles | Can't remember what's in them | Use descriptive titles, include key concept |
| Never link notes together | Miss connections between ideas | Add references to related notes |
| Forget the source | Can't verify claims later | Always link back to source |
| Never edit AI notes | Keep generic AI responses | Refine for clarity and context |
| Create one giant note | Too long to be useful | Split into focused notes by subtopic |
---
## Summary: Note Lifecycle
```
1. CREATE
├─ Manual: Write from scratch
├─ From Chat: Save good response
├─ From Ask: Save synthesis
└─ From Transform: Batch process
2. EDIT & REFINE
├─ Improve clarity
├─ Add context
├─ Fix AI mistakes
└─ Add citations
3. ORGANIZE
├─ Name clearly
├─ Add tags
├─ Link related
└─ Categorize
4. USE
├─ Reference in Chat
├─ Transform for synthesis
├─ Export for sharing
└─ Build on with new questions
5. MAINTAIN
├─ Periodically review
├─ Update as understanding grows
├─ Archive when done
└─ Learn from organized knowledge
```
Your notes become your actual knowledge base. The more you invest in organizing them, the more valuable they become.
docs/4-AI-PROVIDERS/index.md
+219
View File
@@ -0,0 +1,219 @@
# AI Providers - Comparison & Selection Guide
Open Notebook supports 17+ AI providers. This guide helps you **choose the right provider** for your needs.
> 💡 **Just want to set up a provider?** Skip to the [Configuration Guide](../5-CONFIGURATION/ai-providers.md) for detailed setup instructions.
---
## Quick Decision: Which Provider?
### Cloud Providers (Easiest)
**OpenAI (Recommended)**
- Cost: ~$0.03-0.15 per 1K tokens
- Speed: Very fast
- Quality: Excellent
- Best for: Most users (best quality/price balance)
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#openai)
**Anthropic (Claude)**
- Cost: ~$0.80-3.00 per 1M tokens
- Speed: Fast
- Quality: Excellent
- Best for: Long context (200K tokens), reasoning, latest AI
- Advantage: Superior long-context handling
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#anthropic-claude)
**Google Gemini**
- Cost: ~$0.075-0.30 per 1K tokens
- Speed: Very fast
- Quality: Good to excellent
- Best for: Multimodal (images, audio, video)
- Advantage: Longest context (up to 2M tokens)
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#google-gemini)
**Groq (Ultra-Fast)**
- Cost: ~$0.05 per 1M tokens (cheapest)
- Speed: Ultra-fast (fastest available)
- Quality: Good
- Best for: Budget-conscious, transformations, speed-critical tasks
- Disadvantage: Limited model selection
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#groq)
**OpenRouter (100+ Models)**
- Cost: Pay-per-model (varies widely)
- Speed: Varies by model
- Quality: Varies by model
- Best for: Model comparison, testing, unified billing
- Advantage: One API key for 100+ models from different providers
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#openrouter)
**DashScope (Qwen)**
- Cost: ~$0.01-0.06 per 1K tokens
- Speed: Fast
- Quality: Good
- Best for: Users in Asia, Alibaba Cloud ecosystem
- Advantage: Competitive pricing, strong multilingual support
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#dashscope-qwen)
**MiniMax**
- Cost: Varies by model
- Speed: Fast
- Quality: Good
- Best for: Long context tasks (204K tokens)
- Advantage: Very long context window
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#minimax)
### Local / Self-Hosted (Free)
**Ollama (Recommended for Local)**
- Cost: Free (electricity only)
- Speed: Depends on hardware (slow on CPU, fast on GPU)
- Quality: Good (open-source models)
- Setup: 10 minutes
- Best for: Privacy-first, offline use
- Privacy: 100% local, nothing leaves your machine
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#ollama-recommended-for-local)
**LM Studio (Alternative)**
- Cost: Free (electricity only)
- Speed: Depends on hardware
- Quality: Good (same models as Ollama)
- Setup: 15 minutes (GUI interface)
- Best for: Non-technical users who prefer GUI over CLI
- Privacy: 100% local
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#lm-studio-local-alternative)
### Enterprise
**Azure OpenAI**
- Cost: Same as OpenAI (usage-based)
- Speed: Very fast
- Quality: Excellent (same models as OpenAI)
- Setup: 10 minutes (more complex)
- Best for: Enterprise, compliance (HIPAA, SOC2), VPC integration
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#azure-openai)
---
## Comparison Table
| Provider | Speed | Cost | Quality | Privacy | Setup | Context |
|----------|-------|------|---------|---------|-------|---------|
| **OpenAI** | Very Fast | $$ | Excellent | Low | 5 min | 128K |
| **Anthropic** | Fast | $$ | Excellent | Low | 5 min | 200K |
| **Google** | Very Fast | $$ | Good-Excellent | Low | 5 min | 2M |
| **Groq** | Ultra Fast | $ | Good | Low | 5 min | 32K |
| **OpenRouter** | Varies | Varies | Varies | Low | 5 min | Varies |
| **DashScope** | Fast | $ | Good | Low | 5 min | Varies |
| **MiniMax** | Fast | $$ | Good | Low | 5 min | 204K |
| **Ollama** | Slow-Medium | Free | Good | Max | 10 min | Varies |
| **LM Studio** | Slow-Medium | Free | Good | Max | 15 min | Varies |
| **Azure** | Very Fast | $$ | Excellent | High | 10 min | 128K |
---
## Choosing Your Provider
### I want the easiest setup
→ **OpenAI** — Most popular, best community support
### I have unlimited budget
→ **OpenAI** — Best quality
### I want to save money
→ **Groq** — Cheapest cloud ($0.05 per 1M tokens)
### I want privacy/offline
→ **Ollama** — Free, local, private
### I want a GUI (not CLI)
→ **LM Studio** — Desktop app
### I'm in an enterprise
→ **Azure OpenAI** — Compliance, support
### I need long context (200K+ tokens)
→ **Anthropic** — Best long-context model
### I need multimodal (images, audio, video)
→ **Google Gemini** — Best multimodal support
### I want access to many models with one API key
→ **OpenRouter** — 100+ models, unified billing
---
## Ready to Set Up Your Provider?
Now that you've chosen a provider, follow the detailed setup instructions:
→ **[AI Providers Configuration Guide](../5-CONFIGURATION/ai-providers.md)**
This guide includes:
- Step-by-step setup instructions for each provider via the Settings UI
- How to add credentials, test connections, and discover models
- Model selection and recommendations
- Provider-specific troubleshooting
- Hardware requirements (for local providers)
- Cost optimization tips
---
## Cost Estimator
### OpenAI
```
Light use (10 chats/day): $1-5/month
Medium use (50 chats/day): $10-30/month
Heavy use (all-day use): $50-100+/month
```
### Anthropic
```
Light use: $1-3/month
Medium use: $5-20/month
Heavy use: $20-50+/month
```
### Groq
```
Light use: $0-1/month
Medium use: $2-5/month
Heavy use: $5-20/month
```
### Ollama
```
Any use: Free (electricity only)
8GB GPU running 24/7: ~$10/month electricity
```
---
## Next Steps
1. **You've chosen a provider** (from this comparison guide)
2. **Follow the setup guide**: [AI Providers Configuration](../5-CONFIGURATION/ai-providers.md)
3. **Add your credential** in Settings → API Keys
4. **Test your connection** and discover models
5. **Start using Open Notebook!**
---
## Need Help?
- **Setup issues?** See [AI Providers Configuration](../5-CONFIGURATION/ai-providers.md) for detailed troubleshooting per provider
- **General problems?** Check [Troubleshooting Guide](../6-TROUBLESHOOTING/index.md)
- **Questions?** Join [Discord community](https://discord.gg/37XJPXfz2w)
docs/5-CONFIGURATION/advanced.md
+544
View File
@@ -0,0 +1,544 @@
# Advanced Configuration
Performance tuning, debugging, and advanced features.
---
## Performance Tuning
### Concurrency Control
```env
# Max concurrent database operations (default: 5)
# Increase: Faster processing, more conflicts
# Decrease: Slower, fewer conflicts
SURREAL_COMMANDS_MAX_TASKS=5
```
**Guidelines:**
- CPU: 2 cores → 2-3 tasks
- CPU: 4 cores → 5 tasks (default)
- CPU: 8+ cores → 10-20 tasks
Higher concurrency = more throughput but more database conflicts (retries handle this).
### Retry Strategy
```env
# How to wait between retries
SURREAL_COMMANDS_RETRY_WAIT_STRATEGY=exponential_jitter
# Options:
# - exponential_jitter (recommended)
# - exponential
# - fixed
# - random
```
For high-concurrency deployments, use `exponential_jitter` to prevent thundering herd.
### Timeout Tuning
```env
# Client timeout (default: 300 seconds)
API_CLIENT_TIMEOUT=300
# LLM timeout (default: 60 seconds)
ESPERANTO_LLM_TIMEOUT=60
```
**Guideline:** Set `API_CLIENT_TIMEOUT` > `ESPERANTO_LLM_TIMEOUT` + buffer
```
Example:
ESPERANTO_LLM_TIMEOUT=120
API_CLIENT_TIMEOUT=180 # 120 + 60 second buffer
```
---
## Batching
### TTS Batch Size
For podcast generation, control concurrent TTS requests:
```env
# Default: 5
TTS_BATCH_SIZE=2
```
**Providers and recommendations:**
- OpenAI: 5 (can handle many concurrent)
- Google: 4 (good concurrency)
- ElevenLabs: 2 (limited concurrent requests)
- Local TTS: 1 (single-threaded)
Lower = slower but more stable. Higher = faster but more load on provider.
---
## Logging & Debugging
### Enable Detailed Logging
```bash
# Start with debug logging
RUST_LOG=debug # For Rust components
LOGLEVEL=DEBUG # For Python components
```
### Debug Specific Components
```bash
# Only surreal operations
RUST_LOG=surrealdb=debug
# Only langchain
LOGLEVEL=langchain:debug
# Only specific module
RUST_LOG=open_notebook::database=debug
```
### LangSmith Tracing
For debugging LLM workflows:
```env
LANGCHAIN_TRACING_V2=true
LANGCHAIN_ENDPOINT="https://api.smith.langchain.com"
LANGCHAIN_API_KEY=your-key
LANGCHAIN_PROJECT="Open Notebook"
```
Then visit https://smith.langchain.com to see traces.
---
## Port Configuration
### Default Ports
```
Frontend: 8502 (Docker deployment)
Frontend: 3000 (Development from source)
API: 5055
SurrealDB: 8000
```
### Changing Frontend Port
Edit `docker-compose.yml`:
```yaml
services:
open-notebook:
ports:
- "8001:8502" # Change from 8502 to 8001
```
Access at: `http://localhost:8001`
API auto-detects to: `http://localhost:5055` ✓
### Changing API Port
```yaml
services:
open-notebook:
ports:
- "127.0.0.1:8502:8502" # Frontend
- "5056:5055" # Change API from 5055 to 5056
environment:
- API_URL=http://localhost:5056 # Update API_URL
```
Access API directly: `http://localhost:5056/docs`
**Note:** When changing API port, you must set `API_URL` explicitly since auto-detection assumes port 5055.
### Changing SurrealDB Port
```yaml
services:
surrealdb:
ports:
- "127.0.0.1:8001:8000" # Change from 8000 to 8001 (localhost only)
environment:
- SURREAL_URL=ws://surrealdb:8001/rpc # Update connection URL
```
**Important:** Internal Docker network uses container name (`surrealdb`), not `localhost`.
---
## SSL/TLS Configuration
### Custom CA Certificate
For self-signed certs on local providers:
```env
ESPERANTO_SSL_CA_BUNDLE=/path/to/ca-bundle.pem
```
### Disable Verification (Development Only)
```env
# WARNING: Only for testing/development
# Vulnerable to MITM attacks
ESPERANTO_SSL_VERIFY=false
```
---
## Multi-Provider Setup
### Use Different Providers for Different Tasks
Configure multiple AI providers via **Settings → API Keys**. Each provider gets its own credential:
1. Add a credential for your main language model provider (e.g., OpenAI, Anthropic)
2. Add a credential for embeddings (e.g., Voyage AI, or use the same provider)
3. Add a credential for TTS (e.g., ElevenLabs, or OpenAI-Compatible for local Speaches)
4. Each credential's models are registered and available independently
### Multiple Endpoints for OpenAI-Compatible
When using OpenAI-Compatible providers, you can configure per-service URLs in a single credential:
1. Go to **Settings** → **API Keys**
2. Click **Add Credential** → Select **OpenAI-Compatible**
3. Configure separate URLs for LLM, Embedding, TTS, and STT
4. Click **Save**, then **Test Connection**
---
## Security Hardening
### Change Default Credentials
```env
# Don't use defaults in production
SURREAL_USER=your_secure_username
SURREAL_PASSWORD=$(openssl rand -base64 32) # Generate secure password
```
### Add Password Protection
```env
# Protect your Open Notebook instance
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
### Use HTTPS
```env
# Always use HTTPS in production
API_URL=https://mynotebook.example.com
```
### Firewall Rules
Restrict access to your Open Notebook:
- Port 8502 (frontend): Only from your IP
- Port 5055 (API): Only from frontend
- Port 8000 (SurrealDB): Never expose to internet
---
## Web Scraping & Content Extraction
Open Notebook uses multiple services for content extraction:
### Firecrawl
For advanced web scraping:
```env
FIRECRAWL_API_KEY=your-key
```
Get key from: https://firecrawl.dev/
### Jina AI
Alternative web extraction:
```env
JINA_API_KEY=your-key
```
Get key from: https://jina.ai/
---
## Environment Variable Groups
### Credential Storage (Required)
```env
OPEN_NOTEBOOK_ENCRYPTION_KEY # Required for storing credentials
```
AI provider API keys are configured via **Settings → API Keys** (not environment variables).
### Database
```env
SURREAL_URL
SURREAL_USER
SURREAL_PASSWORD
SURREAL_NAMESPACE
SURREAL_DATABASE
```
### Performance
```env
SURREAL_COMMANDS_MAX_TASKS
SURREAL_COMMANDS_RETRY_ENABLED
SURREAL_COMMANDS_RETRY_MAX_ATTEMPTS
SURREAL_COMMANDS_RETRY_WAIT_STRATEGY
SURREAL_COMMANDS_RETRY_WAIT_MIN
SURREAL_COMMANDS_RETRY_WAIT_MAX
```
### API Settings
```env
API_URL
INTERNAL_API_URL
API_CLIENT_TIMEOUT
ESPERANTO_LLM_TIMEOUT
```
### Audio/TTS
```env
TTS_BATCH_SIZE
```
> **Note:** `ELEVENLABS_API_KEY` is deprecated. Configure ElevenLabs via **Settings → API Keys**.
### Debugging
```env
LANGCHAIN_TRACING_V2
LANGCHAIN_ENDPOINT
LANGCHAIN_API_KEY
LANGCHAIN_PROJECT
```
---
## Testing Configuration
### Quick Test
```bash
# Test API health
curl http://localhost:5055/health
# Test with sample (requires configured credential and registered models)
curl -X POST http://localhost:5055/api/chat \
-H "Content-Type: application/json" \
-d '{"message":"Hello"}'
```
### Validate Config
```bash
# Check environment variables are set
env | grep OPEN_NOTEBOOK_ENCRYPTION_KEY
# Verify database connection
python -c "import os; print(os.getenv('SURREAL_URL'))"
```
---
## Troubleshooting Performance
### High Memory Usage
```env
# Reduce concurrency
SURREAL_COMMANDS_MAX_TASKS=2
# Reduce TTS batch size
TTS_BATCH_SIZE=1
```
### High CPU Usage
```env
# Check worker count
SURREAL_COMMANDS_MAX_TASKS
# Reduce if maxed out:
SURREAL_COMMANDS_MAX_TASKS=5
```
### Slow Responses
```env
# Check timeout settings
API_CLIENT_TIMEOUT=300
# Check retry config
SURREAL_COMMANDS_RETRY_MAX_ATTEMPTS=3
```
### Database Conflicts
```env
# Reduce concurrency
SURREAL_COMMANDS_MAX_TASKS=3
# Use jitter strategy
SURREAL_COMMANDS_RETRY_WAIT_STRATEGY=exponential_jitter
```
---
## Backup & Restore
### Data Locations
| Path | Contents |
|------|----------|
| `./data` or `/app/data` | Uploads, podcasts, checkpoints |
| `./surreal_data` or `/mydata` | SurrealDB database files |
### Quick Backup
```bash
# Stop services (recommended for consistency)
docker compose down
# Create timestamped backup
tar -czf backup-$(date +%Y%m%d-%H%M%S).tar.gz \
notebook_data/ surreal_data/
# Restart services
docker compose up -d
```
### Automated Backup Script
```bash
#!/bin/bash
# backup.sh - Run daily via cron
BACKUP_DIR="/path/to/backups"
DATE=$(date +%Y%m%d-%H%M%S)
# Create backup
tar -czf "$BACKUP_DIR/open-notebook-$DATE.tar.gz" \
/path/to/notebook_data \
/path/to/surreal_data
# Keep only last 7 days
find "$BACKUP_DIR" -name "open-notebook-*.tar.gz" -mtime +7 -delete
echo "Backup complete: open-notebook-$DATE.tar.gz"
```
Add to cron:
```bash
# Daily backup at 2 AM
0 2 * * * /path/to/backup.sh >> /var/log/open-notebook-backup.log 2>&1
```
### Restore
```bash
# Stop services
docker compose down
# Remove old data (careful!)
rm -rf notebook_data/ surreal_data/
# Extract backup
tar -xzf backup-20240115-120000.tar.gz
# Restart services
docker compose up -d
```
### Migration Between Servers
```bash
# On source server
docker compose down
tar -czf open-notebook-migration.tar.gz notebook_data/ surreal_data/
# Transfer to new server
scp open-notebook-migration.tar.gz user@newserver:/path/
# On new server
tar -xzf open-notebook-migration.tar.gz
docker compose up -d
```
---
## Container Management
### Common Commands
```bash
# Start services
docker compose up -d
# Stop services
docker compose down
# View logs (all services)
docker compose logs -f
# View logs (specific service)
docker compose logs -f api
# Restart specific service
docker compose restart api
# Update to latest version
docker compose down
docker compose pull
docker compose up -d
# Check resource usage
docker stats
# Check service health
docker compose ps
```
### Clean Up
```bash
# Remove stopped containers
docker compose rm
# Remove unused images
docker image prune
# Full cleanup (careful!)
docker system prune -a
```
---
## Summary
**Most deployments need:**
- One AI provider API key
- Default database settings
- Default timeouts
**Tune performance only if:**
- You have specific bottlenecks
- High-concurrency workload
- Custom hardware (very fast or very slow)
**Advanced features:**
- Firecrawl for better web scraping
- LangSmith for debugging workflows
- Custom CA bundles for self-signed certs
docs/5-CONFIGURATION/ai-providers.md
+550
View File
@@ -0,0 +1,550 @@
# AI Providers - Configuration Guide
Complete setup instructions for each AI provider via the **Settings UI**.
> **New in v1.2**: All AI provider credentials are now managed through the Settings UI. Environment variables for API keys are deprecated.
---
## How Provider Setup Works
Open Notebook uses a **credential-based system** for managing AI providers:
1. **Get your API key** from the provider's website
2. **Open Settings** → **API Keys** → **Add Credential**
3. **Test the connection** to verify it works
4. **Discover & Register Models** to make them available
5. **Start using** the provider in your notebooks
> **Prerequisite**: You must set `OPEN_NOTEBOOK_ENCRYPTION_KEY` in your docker-compose.yml before storing credentials. See [API Configuration](../3-USER-GUIDE/api-configuration.md#encryption-setup) for details.
---
## Cloud Providers (Recommended for Most)
### OpenAI
**Cost:** ~$0.03-0.15 per 1K tokens (varies by model)
**Get Your API Key:**
1. Go to https://platform.openai.com/api-keys
2. Create account (if needed)
3. Create new API key (starts with "sk-proj-")
4. Add $5+ credits to account
**Configure in Open Notebook:**
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **OpenAI**
4. Give it a name (e.g., "My OpenAI Key")
5. Paste your API key
6. Click **Save**, then **Test Connection**
7. Click **Discover Models** to find available models
8. Click **Register Models** to make them available
**Available Models (in Open Notebook):**
- `gpt-4o` — Best quality, fast (latest version)
- `gpt-4o-mini` — Fast, cheap, good for testing
- `o1` — Advanced reasoning model (slower, more expensive)
- `o1-mini` — Faster reasoning model
**Recommended:**
- For general use: `gpt-4o` (best balance)
- For testing/cheap: `gpt-4o-mini` (90% cheaper)
- For complex reasoning: `o1` (best for hard problems)
**Cost Estimate:**
```
Light use: $1-5/month
Medium use: $10-30/month
Heavy use: $50-100+/month
```
**Troubleshooting:**
- "Invalid API key" → Check key starts with "sk-proj-" and test the connection in Settings
- "Rate limit exceeded" → Wait or upgrade account
- "Model not available" → Try gpt-4o-mini instead, or re-discover models
---
### Anthropic (Claude)
**Cost:** ~$0.80-3.00 per 1M tokens (cheaper than OpenAI for long context)
**Get Your API Key:**
1. Go to https://console.anthropic.com/
2. Create account or login
3. Go to API keys section
4. Create new API key (starts with "sk-ant-")
**Configure in Open Notebook:**
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **Anthropic**
4. Give it a name, paste your API key
5. Click **Save**, then **Test Connection**
6. Click **Discover Models** → **Register Models**
**Available Models:**
- `claude-sonnet-4-5-20250929` — Latest, best quality (recommended)
- `claude-3-5-sonnet-20241022` — Previous generation, still excellent
- `claude-3-5-haiku-20241022` — Fast, cheap
- `claude-opus-4-5-20251101` — Most powerful, expensive
**Recommended:**
- For general use: `claude-sonnet-4-5` (best overall, latest)
- For cheap: `claude-3-5-haiku` (80% cheaper)
- For complex: `claude-opus-4-5` (most capable)
**Cost Estimate:**
```
Sonnet: $3-20/month (typical use)
Haiku: $0.50-3/month
Opus: $10-50+/month
```
**Advantages:**
- Great long-context support (200K tokens)
- Excellent reasoning
- Fast processing
**Troubleshooting:**
- "Invalid API key" → Check it starts with "sk-ant-" and test in Settings
- "Overloaded" → Anthropic is busy, retry later
- "Model unavailable" → Re-discover models from the credential
---
### Google Gemini
**Cost:** ~$0.075-0.30 per 1K tokens (competitive with OpenAI)
**Get Your API Key:**
1. Go to https://aistudio.google.com/app/apikey
2. Create account or login
3. Create new API key
**Configure in Open Notebook:**
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **Google Gemini**
4. Give it a name, paste your API key
5. Click **Save**, then **Test Connection**
6. Click **Discover Models** → **Register Models**
**Available Models:**
- `gemini-2.5-pro` — Strongest, best for long context (1M tokens)
- `gemini-3.5-flash` — Fast, good for general use
- `gemini-3.1-flash-lite` — Fastest and cheapest
- `gemini-2.5-flash` — Previous-gen stable, cheaper
**Recommended:**
- For general use: `gemini-3.5-flash` (best value, latest)
- For cheap: `gemini-3.1-flash-lite` (very cheap)
- For complex/long context: `gemini-2.5-pro` (1M token context)
**Advantages:**
- Very long context (1M tokens)
- Multimodal (images, audio, video)
- Good for podcasts
**Troubleshooting:**
- "API key invalid" → Get fresh key from aistudio.google.com
- "Quota exceeded" → Free tier limited, upgrade account
- "Model not found" → Re-discover models from the credential
---
### Groq
**Cost:** ~$0.05 per 1M tokens (cheapest, but limited models)
**Get Your API Key:**
1. Go to https://console.groq.com/keys
2. Create account or login
3. Create new API key
**Configure in Open Notebook:**
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **Groq**
4. Give it a name, paste your API key
5. Click **Save**, then **Test Connection**
6. Click **Discover Models** → **Register Models**
**Available Models:**
- `llama-3.3-70b-versatile` — Best on Groq (recommended)
- `llama-3.1-70b-versatile` — Fast, capable
- `mixtral-8x7b-32768` — Good alternative
- `gemma2-9b-it` — Small, very fast
**Recommended:**
- For quality: `llama-3.3-70b-versatile` (best overall)
- For speed: `gemma2-9b-it` (ultra-fast)
- For balance: `llama-3.1-70b-versatile`
**Advantages:**
- Ultra-fast inference
- Very cheap
- Great for transformations/batch work
**Disadvantages:**
- Limited model selection
- Smaller models than OpenAI/Anthropic
**Troubleshooting:**
- "Rate limited" → Free tier has limits, upgrade
- "Model not available" → Re-discover models from the credential
---
### OpenRouter
**Cost:** Varies by model ($0.05-15 per 1M tokens)
**Get Your API Key:**
1. Go to https://openrouter.ai/keys
2. Create account or login
3. Add credits to your account
4. Create new API key
**Configure in Open Notebook:**
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **OpenRouter**
4. Give it a name, paste your API key
5. Click **Save**, then **Test Connection**
6. Click **Discover Models** → **Register Models**
**Available Models (100+ options):**
- OpenAI: `openai/gpt-4o`, `openai/o1`
- Anthropic: `anthropic/claude-sonnet-4.5`, `anthropic/claude-3.5-haiku`
- Google: `google/gemini-3.5-flash`, `google/gemini-2.5-pro`
- Meta: `meta-llama/llama-3.3-70b-instruct`, `meta-llama/llama-3.1-405b-instruct`
- Mistral: `mistralai/mistral-large-2411`
- DeepSeek: `deepseek/deepseek-chat`
- And many more...
**Recommended:**
- For quality: `anthropic/claude-sonnet-4.5` (best overall)
- For speed/cost: `google/gemini-2.5-flash` (very fast, cheap)
- For open-source: `meta-llama/llama-3.3-70b-instruct`
- For reasoning: `openai/o1`
**Advantages:**
- One API key for 100+ models
- Unified billing
- Easy model comparison
- Access to models that may have waitlists elsewhere
**Cost Estimate:**
```
Light use: $1-5/month
Medium use: $10-30/month
Heavy use: Depends on models chosen
```
**Troubleshooting:**
- "Invalid API key" → Check it starts with "sk-or-"
- "Insufficient credits" → Add credits at openrouter.ai
- "Model not available" → Check model ID spelling (use full path)
---
### DashScope (Qwen)
**Cost:** ~$0.01-0.06 per 1K tokens (varies by model)
**Get Your API Key:**
1. Go to https://dashscope.console.aliyun.com/
2. Create an Alibaba Cloud account (if needed)
3. Navigate to API Keys section
4. Create a new API key
**Configure in Open Notebook:**
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **DashScope (Qwen)**
4. Give it a name, paste your API key
5. Click **Save**, then **Test Connection**
6. Click **Discover Models** → **Register Models**
**Available Models:**
- `qwen-max` — Most capable Qwen model
- `qwen-plus` — Good balance of quality and speed
- `qwen-turbo` — Fastest, cheapest
**Recommended:**
- For quality: `qwen-max` (best overall)
- For general use: `qwen-plus` (good balance)
- For speed/cost: `qwen-turbo` (cheapest)
**Troubleshooting:**
- "Invalid API key" → Check the key in the DashScope console
- "Model not available" → Re-discover models from the credential
---
### MiniMax
**Cost:** Varies by model
**Get Your API Key:**
1. Go to https://platform.minimaxi.com/
2. Create an account (if needed)
3. Navigate to API Keys section
4. Create a new API key
**Configure in Open Notebook:**
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **MiniMax**
4. Give it a name, paste your API key
5. Click **Save**, then **Test Connection**
6. Click **Discover Models** → **Register Models**
**Available Models:**
- `MiniMax-M2.5` — Most capable, 204K context
- `MiniMax-M2.5-highspeed` — Faster variant, 204K context
**Recommended:**
- For quality: `MiniMax-M2.5` (best overall)
- For speed: `MiniMax-M2.5-highspeed` (faster responses)
**Advantages:**
- Very long context (204K tokens)
- Competitive pricing
**Troubleshooting:**
- "Invalid API key" → Check the key in the MiniMax platform
- "Model not available" → Re-discover models from the credential
---
## Self-Hosted / Local
### Ollama (Recommended for Local)
**Cost:** Free (electricity only)
**Setup Ollama:**
1. Install Ollama: https://ollama.ai
2. Run Ollama in background: `ollama serve`
3. Download a model: `ollama pull mistral`
**Configure in Open Notebook:**
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **Ollama**
4. Give it a name (e.g., "Local Ollama")
5. Enter the base URL:
- Same machine (non-Docker): `http://localhost:11434`
- Docker with Ollama on host: `http://host.docker.internal:11434`
- Docker with Ollama container: `http://ollama:11434`
6. Click **Save**, then **Test Connection**
7. Click **Discover Models** → **Register Models**
See [Ollama Setup Guide](ollama.md) for detailed network configuration.
**Context Window (`num_ctx`):**
Ollama models default to a **8,192-token** context window. This default is intentionally
conservative so models run reliably on consumer GPUs (≈8GB VRAM) without running out of memory.
If your hardware can handle more, set an optional **Context Window (num_ctx)** value on the
Ollama credential (Settings → API Keys → edit the Ollama credential). It applies to all models
that use that credential. Leave it empty to keep the default.
- Raise it (e.g. `32768`) when ingesting large documents or using long chat histories.
- If you hit "out of memory" errors, lower it or leave it at the default.
**Available Models:**
- `llama3.3:70b` — Best quality (requires 40GB+ RAM)
- `llama3.1:8b` — Recommended, balanced (8GB RAM)
- `qwen2.5:7b` — Excellent for code and reasoning
- `mistral:7b` — Good general purpose
- `phi3:3.8b` — Small, fast (4GB RAM)
- `gemma2:9b` — Google's model, balanced
- Many more: `ollama list` to see available
**Recommended:**
- For quality (with GPU): `llama3.3:70b` (best)
- For general use: `llama3.1:8b` (best balance)
- For speed/low memory: `phi3:3.8b` (very fast)
- For coding: `qwen2.5:7b` (excellent at code)
**Hardware Requirements:**
```
GPU (NVIDIA/AMD):
8GB VRAM: Runs most models fine
6GB VRAM: Works, slower
4GB VRAM: Small models only
CPU-only:
16GB+ RAM: Slow but works
8GB RAM: Very slow
4GB RAM: Not recommended
```
**Advantages:**
- Completely private (runs locally)
- Free (electricity only)
- No API key needed
- Works offline
**Disadvantages:**
- Slower than cloud (unless on GPU)
- Smaller models than cloud
- Requires local hardware
**Troubleshooting:**
- "Connection refused" → Ollama not running or wrong URL in credential
- "Model not found" → Download it: `ollama pull modelname`
- "Out of memory" → Use smaller model or add more RAM
---
### LM Studio (Local Alternative)
**Cost:** Free
**Setup LM Studio:**
1. Download LM Studio: https://lmstudio.ai
2. Open app
3. Download a model from library
4. Go to "Local Server" tab
5. Start server (default port: 1234)
**Configure in Open Notebook:**
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **OpenAI-Compatible**
4. Give it a name (e.g., "LM Studio")
5. Enter the base URL: `http://host.docker.internal:1234/v1` (Docker) or `http://localhost:1234/v1` (local)
6. API key: `lm-studio` (placeholder, LM Studio doesn't require one)
7. Click **Save**, then **Test Connection**
**Advantages:**
- GUI interface (easier than Ollama CLI)
- Good model selection
- Privacy-focused
- Works offline
**Disadvantages:**
- Desktop only (Mac/Windows/Linux)
- Slower than cloud
- Requires local GPU
---
### Custom OpenAI-Compatible
For Text Generation UI, vLLM, or other OpenAI-compatible endpoints:
1. Go to **Settings** → **API Keys**
2. Click **Add Credential**
3. Select provider: **OpenAI-Compatible**
4. Enter the base URL for your endpoint (e.g., `http://localhost:8000/v1`)
5. Enter API key if required
6. Optionally configure per-service URLs (LLM, Embedding, TTS, STT)
7. Click **Save**, then **Test Connection**
See [OpenAI-Compatible Setup](openai-compatible.md) for detailed instructions.
---
## Enterprise
### Azure OpenAI
**Cost:** Same as OpenAI (usage-based)
**Configure in Open Notebook:**
1. Create Azure OpenAI service in Azure portal
2. Deploy GPT-4/3.5-turbo model
3. Get your endpoint and key
4. Go to **Settings** → **API Keys**
5. Click **Add Credential**
6. Select provider: **Azure OpenAI**
7. Fill in: API Key, Endpoint, API Version
8. Optionally configure service-specific endpoints (LLM, Embedding)
9. Click **Save**, then **Test Connection**
**Advantages:**
- Enterprise support
- VPC integration
- Compliance (HIPAA, SOC2, etc.)
**Disadvantages:**
- More complex setup
- Higher overhead
- Requires Azure account
---
## Embeddings (For Search/Semantic Features)
By default, Open Notebook uses the LLM provider's embeddings. Embedding models are discovered and registered through the same credential system — when you discover models from a credential, embedding models are included alongside language models.
---
## Choosing Your Provider
**1. Don't want to run locally and don't want to mess around with different providers:**
Use OpenAI
- Cloud-based
- Good quality
- Reasonable cost
- Simplest setup, supports all modes (text, embedding, tts, stt, etc)
**For budget-conscious:** Groq, OpenRouter or Ollama
- Groq: Super cheap cloud
- Ollama: Free, but local
- OpenRouter: many open source models very accessible
**For privacy-first:** Ollama or LM Studio and Speaches ([TTS](local-tts.md), [STT](local-stt.md))
- Everything stays local
- Works offline
- No API keys sent anywhere
**For enterprise:** Azure OpenAI
- Compliance
- VPC integration
- Support
---
## Next Steps
1. **Choose your provider** from above
2. **Get API key** (if cloud) or install locally (if Ollama)
3. **Set `OPEN_NOTEBOOK_ENCRYPTION_KEY`** in your docker-compose.yml (required for storing credentials)
4. **Open Settings** → **API Keys** → **Add Credential**
5. **Test Connection** to verify it works
6. **Discover & Register Models** to make them available
7. **Verify it works** with a test chat
> **Multiple providers**: You can add credentials for as many providers as you want. Create separate credentials for different projects or team members.
Done!
---
## Legacy: Environment Variables (Deprecated)
> **Deprecated**: Configuring AI provider API keys via environment variables is deprecated. Use the Settings UI instead. Environment variables may still work as a fallback but are no longer the recommended approach.
If you are migrating from an older version that used environment variables, go to **Settings** → **API Keys** and click the **Migrate to Database** button to import your existing keys into the credential system.
---
## Related
- **[API Configuration](../3-USER-GUIDE/api-configuration.md)** — Detailed credential management guide
- **[Environment Reference](environment-reference.md)** - Complete list of all environment variables
- **[Advanced Configuration](advanced.md)** - Timeouts, SSL, performance tuning
- **[Ollama Setup](ollama.md)** - Detailed Ollama configuration guide
- **[OpenAI-Compatible](openai-compatible.md)** - LM Studio and other compatible providers
- **[Local TTS Setup](local-tts.md)** - Text-to-speech with Speaches
- **[Local STT Setup](local-stt.md)** - Speech-to-text with Speaches
- **[Troubleshooting](../6-TROUBLESHOOTING/quick-fixes.md)** - Common issues and fixes

Some files were not shown because too many files have changed in this diff Show More