wehub-resource-sync
fed8b2eed7
Build and push multi-arch DocsGPT Docker image / build (linux/amd64, ubuntu-latest, amd64) (push) Has been cancelled
Backend release / release (push) Has been cancelled
Bandit Security Scan / bandit_scan (push) Has been cancelled
Build and push multi-arch DocsGPT Docker image / build (linux/arm64, ubuntu-24.04-arm, arm64) (push) Has been cancelled
Build and push multi-arch DocsGPT Docker image / manifest (push) Has been cancelled
Build and push DocsGPT FE Docker image for development / build (linux/amd64, ubuntu-latest, amd64) (push) Has been cancelled
Build and push DocsGPT FE Docker image for development / build (linux/arm64, ubuntu-24.04-arm, arm64) (push) Has been cancelled
Build and push DocsGPT FE Docker image for development / manifest (push) Has been cancelled
Python linting / ruff (push) Has been cancelled
Run python tests with pytest / Run tests and count coverage (3.12) (push) Has been cancelled
React Widget Build / build (push) Has been cancelled
69 lines
2.6 KiB
Plaintext
69 lines
2.6 KiB
Plaintext
---
|
|
title:
|
|
description:
|
|
---
|
|
|
|
import { Callout } from 'nextra/components'
|
|
import Image from 'next/image'
|
|
import { Steps } from 'nextra/components'
|
|
|
|
# Setting Up Local Language Models for Your App
|
|
|
|
Setting up local language models for your app can significantly enhance its capabilities, enabling it to understand and generate text in multiple languages without relying on external APIs. By integrating local language models, you can improve privacy, reduce latency, and ensure continuous functionality even in offline environments. Here's a comprehensive guide on how to set up local language models for your application:
|
|
|
|
## Steps:
|
|
### For cloud version LLM change:
|
|
<Steps >
|
|
### Step 1
|
|
Visit the chat screen and you will be to see the default LLM selected.
|
|
### Step 2
|
|
Click on it and you will get a drop down of various LLM's available to choose.
|
|
### Step 3
|
|
Choose the LLM of your choice.
|
|
|
|
</Steps>
|
|
|
|
|
|
|
|
|
|
### Video Demo
|
|
<Image src="/llms.gif" alt="prompts" width={800} height={500} />
|
|
|
|
### For Open source llm change:
|
|
<Steps>
|
|
### Step 1
|
|
For open source version please edit `LLM_PROVIDER`, `LLM_NAME` and others in the .env file. Refer to [⚙️ App Configuration](/Deploying/DocsGPT-Settings) for more information.
|
|
### Step 2
|
|
Visit [☁️ Cloud Providers](/Models/cloud-providers) for the updated list of online models. Make sure you have the right API_KEY and correct LLM_PROVIDER.
|
|
For self-hosted please visit [🖥️ Local Inference](/Models/local-inference).
|
|
</Steps>
|
|
|
|
## Fallback LLM
|
|
|
|
DocsGPT can automatically switch to a fallback LLM when the primary model fails, including mid-stream. This works with both streaming and non-streaming requests.
|
|
|
|
**Fallback order:**
|
|
1. Per-agent backup models (other models configured on the same agent)
|
|
2. Global fallback (`FALLBACK_LLM_*` env vars below)
|
|
3. Error returned if all fail
|
|
|
|
| Setting | Description | Default |
|
|
| --- | --- | --- |
|
|
| `FALLBACK_LLM_PROVIDER` | Provider name (e.g., `openai`, `anthropic`, `google`) | — |
|
|
| `FALLBACK_LLM_NAME` | Model name (e.g., `gpt-4o`, `claude-sonnet-4-20250514`) | — |
|
|
| `FALLBACK_LLM_API_KEY` | API key for the fallback provider | Falls back to `API_KEY` |
|
|
|
|
All three (`FALLBACK_LLM_PROVIDER`, `FALLBACK_LLM_NAME`, and an API key) must resolve for the global fallback to activate.
|
|
|
|
```env
|
|
FALLBACK_LLM_PROVIDER=anthropic
|
|
FALLBACK_LLM_NAME=claude-sonnet-4-20250514
|
|
FALLBACK_LLM_API_KEY=sk-ant-your-anthropic-key
|
|
```
|
|
|
|
<Callout type="info">
|
|
For maximum resilience, use a fallback provider from a different cloud than your primary. Each agent can also have multiple models configured — the other models are tried first before the global fallback.
|
|
</Callout>
|
|
|
|
|