wehub-resource-sync
143 lines
5.7 KiB
Markdown
143 lines
5.7 KiB
Markdown
# Runtime Provider Integration
|
|
|
|
How llmfit detects and talks to Ollama, llama.cpp, Docker Model Runner, LM Studio, and remote instances.
|
|
|
|
[← Back to README](../README.md)
|
|
|
|
## Runtime provider integration
|
|
|
|
llmfit supports multiple local runtime providers:
|
|
|
|
- **Ollama** (daemon/API based pulls)
|
|
- **llama.cpp** (direct GGUF downloads from Hugging Face + local cache detection)
|
|
- **MLX** (Apple Silicon / mlx-community model cache + optional server) — MLX downloads map to `mlx-community/*` repos on HuggingFace, not the original model publisher
|
|
- **Docker Model Runner** (Docker Desktop's built-in model serving)
|
|
- **LM Studio** (local model server with REST API for model management + downloads)
|
|
|
|
When more than one compatible provider is available for a model, pressing `d` in the TUI opens a provider picker modal.
|
|
|
|
### Ollama integration
|
|
|
|
llmfit integrates with [Ollama](https://ollama.com) to detect which models you already have installed and to download new ones directly from the TUI.
|
|
|
|
### Requirements
|
|
|
|
- **Ollama must be installed and running** (`ollama serve` or the Ollama desktop app)
|
|
- llmfit connects to `http://localhost:11434` (Ollama's default API port)
|
|
- No configuration needed — if Ollama is running, llmfit detects it automatically
|
|
|
|
### Remote Ollama instances
|
|
|
|
To connect to Ollama running on a different machine or port, set the `OLLAMA_HOST` environment variable:
|
|
|
|
```sh
|
|
# Connect to Ollama on a specific IP and port
|
|
OLLAMA_HOST="http://192.168.1.100:11434" llmfit
|
|
|
|
# Connect via hostname
|
|
OLLAMA_HOST="http://ollama-server:666" llmfit
|
|
|
|
# Works with all TUI and CLI commands
|
|
OLLAMA_HOST="http://192.168.1.100:11434" llmfit --cli
|
|
OLLAMA_HOST="http://192.168.1.100:11434" llmfit fit --perfect -n 5
|
|
```
|
|
|
|
This is useful for:
|
|
- Running llmfit on one machine while Ollama serves from another (e.g., GPU server + laptop client)
|
|
- Connecting to Ollama running in Docker containers with custom ports
|
|
- Using Ollama behind reverse proxies or load balancers
|
|
|
|
### How it works
|
|
|
|
On startup, llmfit queries `GET /api/tags` to list your installed Ollama models. Each installed model gets a green **✓** in the **Inst** column of the TUI. The system bar shows `Ollama: ✓ (N installed)`.
|
|
|
|
When you press `d` on a model, llmfit sends `POST /api/pull` to Ollama to download it. The row highlights with an animated progress indicator showing download progress in real-time. Once complete, the model is immediately available for use with Ollama.
|
|
|
|
If Ollama is not running, Ollama-specific operations are skipped; the TUI still supports other providers like llama.cpp where available.
|
|
|
|
### llama.cpp integration
|
|
|
|
llmfit integrates with [llama.cpp](https://github.com/ggml-org/llama.cpp) as a runtime/download provider in both TUI and CLI.
|
|
|
|
Requirements:
|
|
|
|
- `llama-cli` or `llama-server` available in `PATH` (for runtime detection)
|
|
- network access to Hugging Face for GGUF downloads
|
|
|
|
How it works:
|
|
|
|
- llmfit maps HF models to known GGUF repos (with heuristic fallbacks)
|
|
- downloads GGUF files into the local llama.cpp model cache
|
|
- marks models installed when matching GGUF files are present locally
|
|
|
|
#### Environment variables
|
|
|
|
| Variable | Default | Description |
|
|
|---|---|---|
|
|
| `LLAMA_CPP_PATH` | *(none)* | Directory containing llama.cpp binaries (`llama-cli`, `llama-server`). Checked before `PATH` lookup. |
|
|
| `LLAMA_SERVER_PORT` | `8080` | Port used when probing a running `llama-server` health endpoint for runtime detection. |
|
|
|
|
If llama.cpp is installed in a non-standard location, set `LLAMA_CPP_PATH` so llmfit can find it without requiring it in your `PATH`.
|
|
|
|
### Docker Model Runner integration
|
|
|
|
llmfit integrates with [Docker Model Runner](https://docs.docker.com/desktop/features/model-runner/), Docker Desktop's built-in model serving feature.
|
|
|
|
Requirements:
|
|
|
|
- Docker Desktop with Model Runner enabled
|
|
- Default endpoint: `http://localhost:12434`
|
|
|
|
How it works:
|
|
|
|
- llmfit queries `GET /engines` to list models available in Docker Model Runner
|
|
- models are matched to the HF database using Ollama-style tag mapping (Docker Model Runner uses `ai/<tag>` naming)
|
|
- pressing `d` in the TUI pulls via `docker model pull`
|
|
|
|
### Remote Docker Model Runner instances
|
|
|
|
To connect to Docker Model Runner on a different host or port, set the `DOCKER_MODEL_RUNNER_HOST` environment variable:
|
|
|
|
```sh
|
|
DOCKER_MODEL_RUNNER_HOST="http://192.168.1.100:12434" llmfit
|
|
```
|
|
|
|
### LM Studio integration
|
|
|
|
llmfit integrates with [LM Studio](https://lmstudio.ai) as a local model server with built-in model download capabilities.
|
|
|
|
Requirements:
|
|
|
|
- LM Studio must be running with its local server enabled
|
|
- Default endpoint: `http://127.0.0.1:1234`
|
|
|
|
How it works:
|
|
|
|
- llmfit queries `GET /v1/models` to list models available in LM Studio
|
|
- pressing `d` in the TUI triggers a download via `POST /api/v1/models/download`
|
|
- download progress is tracked by polling `GET /api/v1/models/download-status`
|
|
- LM Studio accepts HuggingFace model names directly, so no name mapping is needed
|
|
|
|
### Remote LM Studio instances
|
|
|
|
To connect to LM Studio on a different host or port, set the `LMSTUDIO_HOST` environment variable:
|
|
|
|
```sh
|
|
LMSTUDIO_HOST="http://192.168.1.100:1234" llmfit
|
|
```
|
|
|
|
### API authentication
|
|
|
|
If your LM Studio instance has **Require API Key** enabled (required for MCP server access), set the `LMSTUDIO_API_KEY` environment variable to provide a Bearer token with all requests:
|
|
|
|
```sh
|
|
export LMSTUDIO_API_KEY="your-api-key-here"
|
|
llmfit
|
|
```
|
|
|
|
### Model name mapping
|
|
|
|
llmfit's database uses HuggingFace model names (e.g. `Qwen/Qwen2.5-Coder-14B-Instruct`) while Ollama uses its own naming scheme (e.g. `qwen2.5-coder:14b`). llmfit maintains an accurate mapping table between the two so that install detection and pulls resolve to the correct model. Each mapping is exact — `qwen2.5-coder:14b` maps to the Coder model, not the base `qwen2.5:14b`.
|
|
|
|
---
|