# Notifications System

The Local Deep Research (LDR) notifications system provides a flexible way to send notifications to various services when important events occur, such as research completion, failures, or subscription updates.

## Overview

The notification system uses [Apprise](https://github.com/caronc/apprise) to support multiple notification services with a unified API. It allows users to configure comma-separated service URLs to receive notifications for different events.

## Server-Side Opt-In Required

> **Outbound notifications are disabled by default.** The deployment operator must explicitly enable them by setting an environment variable on the server:
>
> ```bash
> LDR_NOTIFICATIONS_ALLOW_OUTBOUND=true
> ```
>
> Without this, every `send_notification` call returns `False` and the "Send Test Notification" button returns an error. This applies to **all** users on the deployment.

### Why?

Notification webhooks have a known **DNS-rebinding TOCTOU window** that cannot be closed in code: LDR validates the URL once when it is configured, but the underlying Apprise library resolves the hostname *again* at send time, and Apprise exposes no DNS/Session hook to pin the resolved IP. A logged-in user with a controllable domain can serve a public IP at validation and a private IP at send time, causing the LDR server to make outbound HTTP requests to its own internal services (e.g. `127.0.0.1:<internal-port>`) or the local network.

Because LDR is multi-user (per-user encrypted SQLCipher databases behind `@login_required`), the right default is to keep this feature off until the operator explicitly opts in — flipping the env var is the operator's acknowledgement of the residual risk. See [SECURITY.md](../SECURITY.md#notification-webhook-ssrf) for the full rationale and operator-side mitigations (prefer plugin schemes over raw `http(s)://`, restrict egress).

### Symptoms when the gate is closed

If you've configured a notification URL and aren't receiving messages, check the server logs first. You should see lines like:

```
WARNING  Notification refused: outbound notifications are disabled at the
         server level. Set LDR_NOTIFICATIONS_ALLOW_OUTBOUND=true to enable.
         See SECURITY.md 'Notification Webhook SSRF' for the rationale and
         residual risk. (event=research_completed, user=...)
```

The "Send Test Notification" UI button returns the same message inline.

## Supported Services

The system supports all services that Apprise supports, including but not limited to:

- Discord (via webhooks)
- Slack (via webhooks)
- Telegram
- Email (SMTP)
- Pushover
- Gotify
- Many more...

For a complete list, refer to the [Apprise documentation](https://github.com/caronc/apprise/wiki).

## Configuration

Notifications are configured per-user via the settings system:

### Service URL Setting
- **Key**: `notifications.service_url`
- **Type**: String (comma-separated list of service URLs)
- **Example**: `discord://webhook_id/webhook_token,mailto://user:password@smtp.gmail.com`
- **Security**: Service URLs containing credentials are encrypted at rest using SQLCipher (AES-256) in your per-user encrypted database. The encryption key is derived from your login password, ensuring zero-knowledge security.

### Event-Specific Settings
- `notifications.on_research_completed` - Enable notifications for completed research (default: true)
- `notifications.on_research_failed` - Enable notifications for failed research (default: true)
- `notifications.on_research_queued` - Enable notifications when research is queued (default: false)
- `notifications.on_subscription_update` - Enable notifications for subscription updates (default: true)
- `notifications.on_subscription_error` - Enable notifications for subscription errors (default: false)
- `notifications.on_api_quota_warning` - Enable notifications for API quota/rate limit warnings (default: false)
- `notifications.on_auth_issue` - Enable notifications for authentication failures (default: false)

### Rate Limiting Settings
- `notifications.rate_limit_per_hour` - Max notifications per hour (per user, default: 10)
- `notifications.rate_limit_per_day` - Max notifications per day (per user, default: 50)

**Per-User Rate Limiting**: Each user can configure their own rate limits via their settings. Rate limits are enforced independently per user, so one user hitting their limit does not affect other users. This ensures fair resource allocation in multi-user deployments.

**Note on Multi-Worker Deployments**: The current rate limiting implementation uses in-memory storage and is process-local. In multi-worker deployments (e.g., gunicorn with multiple workers), each worker process maintains its own rate limit counters. This means a user could potentially send up to `N × max_per_hour` notifications (where N = number of workers) by distributing requests across different workers. For single-worker deployments (the default for LDR), this is not a concern. If you're running a multi-worker production deployment, consider monitoring notification volumes or implementing Redis-based rate limiting.

### URL Configuration
- `app.external_url` - Public URL where your LDR instance is accessible (e.g., `https://ldr.example.com`). Used to generate clickable links in notifications. If not set, defaults to `http://localhost:5000` or auto-constructs from `app.host` and `app.port`.

## Service URL Format

Multiple service URLs can be configured by separating them with commas:

```
discord://webhook1_id/webhook1_token,slack://token1/token2/token3,mailto://user:password@smtp.gmail.com
```

Each URL follows the Apprise format for the specific service.

## Available Event Types

### Research Events
- `research_completed` - When research completes successfully
- `research_failed` - When research fails (error details are sanitized in notifications for security)
- `research_queued` - When research is added to the queue

### Subscription Events
- `subscription_update` - When a subscription completes
- `subscription_error` - When a subscription fails

### System Events
- `api_quota_warning` - When API quota or rate limits are exceeded
- `auth_issue` - When authentication fails for API services

## Testing Notifications

Use the test function to verify notification configuration:

```python
from local_deep_research.notifications.manager import NotificationManager

# Create manager for testing (user_id is required)
notification_manager = NotificationManager(
    settings_snapshot={},
    user_id="test_user"
)

# Test a service URL
result = notification_manager.test_service("discord://webhook_id/webhook_token")
print(result)  # {'success': True, 'message': 'Test notification sent successfully'}
```

## Programmatic Usage

For detailed code examples, see the source files in `src/local_deep_research/notifications/`.

### Basic Notification

```python
from local_deep_research.notifications.manager import NotificationManager
from local_deep_research.notifications.templates import EventType
from local_deep_research.settings import SettingsManager
from local_deep_research.database.session_context import get_user_db_session

# Get settings snapshot
username = "your_username"
with get_user_db_session(username, password) as session:
    settings_manager = SettingsManager(session)
    settings_snapshot = settings_manager.get_settings_snapshot()

# Create notification manager with user_id for per-user rate limiting
notification_manager = NotificationManager(
    settings_snapshot=settings_snapshot,
    user_id=username  # Enables per-user rate limit configuration
)

# Send notification (user_id already set in manager)
notification_manager.send_notification(
    event_type=EventType.RESEARCH_COMPLETED,
    context={"query": "...", "summary": "...", "url": "/research/123"},
)
```

**Important**: The `user_id` parameter is **required** when creating a `NotificationManager`. This ensures the user's configured rate limits from their settings are properly applied and enforces per-user isolation.

### Building Full URLs

Use `build_notification_url()` to convert relative paths to full URLs for clickable links in notifications.

## Architecture

The notification system consists of three main components:

1. **NotificationManager** - High-level manager that handles rate limiting, settings, and user preferences
2. **NotificationService** - Low-level service that uses Apprise to send notifications
3. **Settings Integration** - User-specific configuration for services and event preferences

The system fetches service URLs from user settings when needed, rather than maintaining persistent channels, making it more efficient and secure.

### Security & Privacy

- **Encrypted Storage**: All notification service URLs (including credentials like SMTP passwords or webhook tokens) are stored encrypted at rest in your per-user SQLCipher database using AES-256 encryption.
- **Zero-Knowledge Architecture**: The encryption key is derived from your login password using PBKDF2-SHA512. Your password is never stored, and notification settings cannot be recovered without it.
- **URL Masking**: Service URLs are automatically masked in logs to prevent credential exposure (e.g., `discord://webhook_id/***`).
- **Per-User Isolation**: Each user's notification settings are completely isolated in their own encrypted database.

### Performance Optimizations

- **Temporary Apprise Instances**: Temporary Apprise instances are created for each send operation and automatically garbage collected by Python. This simple approach avoids memory management complexity.
- **Shared Rate Limiter with Per-User Limits**: A single rate limiter instance is shared across all NotificationManager instances for efficiency, while maintaining separate rate limit configurations and counters for each user. This provides both memory efficiency (~24 bytes per user for limit storage) and proper per-user isolation.
- **Thread-Safe**: The rate limiter uses threading locks for safe concurrent access within a single process.
- **Exponential Backoff Retry**: Failed notifications are retried up to 3 times with exponential backoff (0.5s → 1.0s → 2.0s) to handle transient network issues.
- **Dynamic Limit Updates**: User rate limits can be updated at runtime when creating a new NotificationManager instance with updated settings.

## Thread Safety & Background Tasks

The notification system is designed to work safely from background threads (e.g., research queue processors). Use the **settings snapshot pattern** to avoid thread-safety issues with database sessions.

### Settings Snapshot Pattern

**Key Principle**: Capture settings once with a database session, then pass the snapshot (not the session) to `NotificationManager`.

- ✅ **Correct**: `NotificationManager(settings_snapshot=settings_snapshot, user_id=username)`
- ❌ **Wrong**: `NotificationManager(session=session)` - Not thread-safe!

See the source code in `web/queue/processor_v2.py` and `error_handling/error_reporter.py` for implementation examples.

## Advanced Usage

### Multiple Service URLs

Configure multiple comma-separated service URLs to send notifications to multiple services simultaneously (Discord, Slack, email, etc.).

### Custom Retry Behavior

Use `force=True` parameter to bypass rate limits and disabled settings for critical notifications.

### Event-Specific Configuration

Each event type can be individually enabled/disabled via settings (see Event-Specific Settings above).

### Per-User Rate Limiting

The notification system supports independent rate limiting for each user:

**How It Works:**
- Each user configures their own rate limits via settings (e.g., `notifications.rate_limit_per_hour`)
- Rate limits are enforced per-user, not globally
- One user hitting their limit does not affect other users
- Rate limits can be different for each user based on their settings

**Example:**
```python
# User A with conservative limits (5/hour)
snapshot_a = {"notifications.rate_limit_per_hour": 5}
manager_a = NotificationManager(snapshot_a, user_id="user_a")

# User B with generous limits (20/hour)
snapshot_b = {"notifications.rate_limit_per_hour": 20}
manager_b = NotificationManager(snapshot_b, user_id="user_b")

# User A can send 5 notifications per hour
# User B can send 20 notifications per hour
# They don't interfere with each other
```

**Technical Details:**
- The rate limiter maintains separate counters for each user
- Each user's limits are stored in memory (~24 bytes per user)
- Limits can be updated dynamically by creating a new NotificationManager for that user
- The `user_id` parameter is required when creating a NotificationManager

### Rate Limit Handling

Rate limit exceptions (`RateLimitError`) can be caught and handled gracefully. See `notifications/exceptions.py` for available exception types.

**Example:**
```python
from local_deep_research.notifications.exceptions import RateLimitError

try:
    notification_manager.send_notification(
        event_type=EventType.RESEARCH_COMPLETED,
        context=context,
    )
except RateLimitError as e:
    # The manager already knows the user_id from initialization
    logger.warning(f"Rate limit exceeded: {e}")
    # Handle rate limit (e.g., queue for later, notify user)
```

## Troubleshooting

### Notifications Not Sending

1. **Check service URL configuration**: Use `SettingsManager.get_setting("notifications.service_url")` to verify the service URL is configured
2. **Test service connection**: Use `notification_manager.test_service(service_url)` to verify connectivity
3. **Check event-specific settings**: Verify the specific event type is enabled (e.g., `notifications.on_research_completed`)
4. **Check rate limits**: Look for "Rate limit exceeded for user {user_id}" messages in logs

### Common Issues

**Issue**: "No notification service URLs configured"
- **Cause**: `notifications.service_url` setting is empty or not set
- **Fix**: Configure service URL in settings dashboard or via API

**Issue**: "Rate limit exceeded"
- **Cause**: User has sent too many notifications within their configured time window (hourly or daily limit)
- **Fix**: Wait for rate limit window to expire (1 hour for hourly, 1 day for daily), adjust rate limit settings, or use `force=True` for critical notifications
- **Note**: Rate limits are enforced per-user, so this only affects the specific user who exceeded their limit

**Issue**: "Failed to send notification after 3 attempts"
- **Cause**: Service is unreachable or credentials are invalid
- **Fix**: Verify service URL is correct, test with `test_service()`, check network connectivity

**Issue**: Notifications work in main thread but fail in background thread
- **Cause**: Using database session in background thread (not thread-safe)
- **Fix**: Use settings snapshot pattern as shown in migration guide above

### Webhook URL Was Rejected

The "Test" button in the notifications settings page returns the validator's reason directly. Common categories and the fix:

**"Blocked private/internal IP address: \<host\>"**
- **Cause**: The URL resolves to a loopback (`127.0.0.1`, `::1`), RFC1918 (`10.x`, `172.16-31.x`, `192.168.x`), CGNAT (`100.64.0.0/10`), link-local (`169.254.0.0/16`), or IPv6 private (`fc00::/7`, `fe80::/10`) address. The default SSRF policy blocks these for outbound webhooks.
- **Fix (operator-only, env-only)**: Set `LDR_NOTIFICATIONS_ALLOW_PRIVATE_IPS=true` in the server environment. This is intentionally not exposed in the user-writable settings API. Only enable it if the notification endpoints are on a trusted local network.
- **Fix (IPv6-only deployments using NAT64)**: If the host wraps IPv4 through `64:ff9b::/96` (RFC 6052 well-known) or `64:ff9b:1::/48` (RFC 8215 local-use), additionally set `LDR_SECURITY_ALLOW_NAT64=true`. The opt-in is scoped strictly to those two prefixes — 6to4 (`2002::/16`), Teredo (`2001::/32`), the discard prefix (`100::/64`), and the deprecated IPv4-Compatible IPv6 form (`::/96`) remain blocked.
- **Note (cloud-metadata IPs)**: If `\<host\>` is a cloud-metadata IP (see the next bullet), the env-var hint above is intentionally **not** surfaced by the "Test" button — neither flag re-opens metadata, so the hint would mislead. The user-visible error is just the bare `"Blocked private/internal IP address: 169.254.169.254"`.

**"Blocked cloud-metadata IP address: \<host\>"**
- **Cause**: The URL uses an Apprise plugin scheme (`discord://`, `signal://`, `ntfy://`, etc.) whose host resolves to a cloud-metadata endpoint. Plugin schemes bypass the http/https private-IP check (their endpoints are typically LAN-local) but still hit the absolute cloud-metadata block.
- **Always blocked**: AWS IMDS / ECS, Azure, OCI, DigitalOcean, AlibabaCloud, Tencent — both as plain IPv4 and wrapped through any NAT64 prefix. No env var re-opens these. See [SECURITY.md](../SECURITY.md#cloud-metadata-endpoint-block-list).
- **Fix**: Choose a different webhook destination. Metadata endpoints expose IAM/instance credentials and are never legitimate webhook targets.

**"Blocked unsafe protocol: \<scheme\>"** / **"Unsupported protocol: \<scheme\>"**
- **Cause**: The URL uses a scheme that is either denylisted (`file`, `ftp`, `data`, `javascript`, `vbscript`, `about`, `blob`) or not in the Apprise-supported allowlist.
- **Fix**: Use one of the allowed schemes — `http`, `https`, `mailto`, `discord`, `slack`, `telegram`, `gotify`, `pushover`, `ntfy`, `ntfys`, `signal`, `matrix`, `mattermost`, `rocketchat`, `teams`, `json`, `xml`, `form`. Prefer Apprise plugin schemes (`discord://`, `slack://`, `ntfy://`, etc.) over raw `http(s)://` webhooks — they hardcode their endpoints and have no SSRF surface.

**"URL contains characters that are not allowed (whitespace, backslash, or control bytes)"**
- **Cause**: Layer-1 defense against parser-differential SSRF bypasses (GHSA-g23j-2vwm-5c25) — RFC 3986 forbids these characters in URLs.
- **Fix**: Remove the whitespace / backslash / control bytes. Percent-encode if a legitimate use case requires them.

**"Outbound notifications are disabled. The server administrator must set LDR_NOTIFICATIONS_ALLOW_OUTBOUND=true …"**
- **Cause**: Server-level master switch is off. See the "Server-Side Opt-In Required" section at the top of this document for the rationale (DNS-rebinding TOCTOU window in Apprise).
- **Fix (operator-only)**: Set `LDR_NOTIFICATIONS_ALLOW_OUTBOUND=true` after reviewing the residual risk.

## See Also

- [Full Configuration Reference](CONFIGURATION.md) - All notification settings, defaults, and environment variables
- [News Subscriptions](news-subscriptions.md) - News subscription system
- [Features](features.md) - Feature overview