wehub-resource-sync
424 lines
10 KiB
Markdown
424 lines
10 KiB
Markdown
# Troubleshooting
|
|
|
|
This page lists common issues and the first checks to make.
|
|
|
|
## No GPU detected
|
|
|
|
Run:
|
|
|
|
```bash
|
|
whichllm hardware
|
|
```
|
|
|
|
If an NVIDIA GPU is missing:
|
|
|
|
- check that the driver is installed
|
|
- check `nvidia-smi`
|
|
- check that `nvidia-ml-py` can load NVML
|
|
|
|
whichllm falls back to `nvidia-smi`, but it still needs the NVIDIA driver tools
|
|
to be working.
|
|
|
|
If an AMD GPU is missing:
|
|
|
|
- on Linux, check `rocm-smi`, `lspci`, and `/sys/class/drm`
|
|
- on Windows, check that PowerShell can read `Win32_VideoController`
|
|
- for Ryzen AI / Radeon integrated graphics, check whether `whichllm hardware`
|
|
shows shared memory instead of a tiny 512 MB or 4 GB adapter
|
|
|
|
If an Intel iGPU is missing:
|
|
|
|
- Linux detection uses `lspci` or `/sys/class/drm`
|
|
- Windows detection uses `Win32_VideoController`
|
|
- many Intel iGPUs do not expose dedicated VRAM, so they may be shown as shared
|
|
memory graphics
|
|
|
|
## Simulate hardware instead
|
|
|
|
If detection is unavailable or you are planning a purchase, use `--gpu`:
|
|
|
|
```bash
|
|
whichllm --gpu "RTX 4090"
|
|
whichllm hardware --gpu "Apple M3 Max"
|
|
whichllm --gpu "RTX 5060 Ti" --vram 16
|
|
whichllm --gpu "2x RTX 4090"
|
|
whichllm --gpu "RTX 4090" --gpu "RTX 3090"
|
|
```
|
|
|
|
Use `--vram` when the GPU name has multiple memory variants or is not in the
|
|
database.
|
|
|
|
For detected iGPU or unified-memory systems, override the usable GPU memory and
|
|
bandwidth directly:
|
|
|
|
```bash
|
|
whichllm hardware --vram 8 --ram-bandwidth 68
|
|
whichllm --vram 8 --bandwidth 68
|
|
```
|
|
|
|
If `whichllm hardware` lists multiple GPUs, add `--gpu-index` with the GPU
|
|
number from that output.
|
|
|
|
`--vram` only applies to one simulated or detected GPU. For multi-GPU
|
|
simulation, use known GPU names and omit `--vram`.
|
|
|
|
## `--cpu-only` conflicts with `--gpu`
|
|
|
|
These flags are mutually exclusive:
|
|
|
|
```bash
|
|
whichllm --cpu-only --gpu "RTX 4090"
|
|
```
|
|
|
|
Choose one:
|
|
|
|
```bash
|
|
whichllm --cpu-only
|
|
whichllm --gpu "RTX 4090"
|
|
```
|
|
|
|
## `--vram` / `--bandwidth` needs a GPU
|
|
|
|
These overrides need either a detected GPU or a simulated GPU:
|
|
|
|
```bash
|
|
whichllm --vram 8 --ram-bandwidth 68
|
|
whichllm --gpu "RTX 3060" --vram 12
|
|
```
|
|
|
|
If no GPU is detected, use `--gpu` to simulate one or check the detection steps
|
|
above.
|
|
|
|
## No compatible models found
|
|
|
|
Try:
|
|
|
|
```bash
|
|
whichllm
|
|
whichllm --cpu-only
|
|
whichllm --refresh
|
|
```
|
|
|
|
Common causes:
|
|
|
|
- the selected `--quant` is too restrictive
|
|
- `--gpu-only` or `--fit full-gpu` filters out partial-offload and CPU-only candidates
|
|
- `--speed usable`, `--speed fast`, or `--min-speed` filters out slower candidates
|
|
- `--min-speed` is too high
|
|
- `--evidence strict` filters out all candidates
|
|
- the requested context length is too large
|
|
- available RAM is too low after reserving space for the OS
|
|
- disk free space is too low for the model weights
|
|
|
|
For very small machines, remove optional filters first:
|
|
|
|
```bash
|
|
whichllm --top 20
|
|
```
|
|
|
|
## Recommendations use RAM or CPU offload, but I only want VRAM
|
|
|
|
By default, whichllm includes any runnable candidate: full-GPU, partial-offload,
|
|
and CPU-only. This is useful for finding what can run at all, but it can be too
|
|
loose when you want only models that fit entirely in GPU VRAM.
|
|
|
|
Use:
|
|
|
|
```bash
|
|
whichllm --gpu-only
|
|
whichllm --fit gpu
|
|
whichllm --fit full-gpu
|
|
```
|
|
|
|
If no rows are shown, this machine has no ranked candidates that fit fully in
|
|
GPU memory under the current filters. Remove `--gpu-only`, lower the context
|
|
length, or try a smaller quantization.
|
|
|
|
## A model fits, but it is too slow
|
|
|
|
The default ranking table shows estimated generation speed. Slow rows are red,
|
|
marginal rows are yellow, usable rows are green, and fast rows are bright
|
|
green. The `~` and `?` markers are confidence markers for the estimate.
|
|
|
|
Filter slow rows with:
|
|
|
|
```bash
|
|
whichllm --speed usable # >=10 tok/s
|
|
whichllm --speed fast # >=30 tok/s
|
|
whichllm --min-speed 4 # exact floor, if you want a lower threshold
|
|
```
|
|
|
|
For an exact threshold:
|
|
|
|
```bash
|
|
whichllm --min-speed 10
|
|
```
|
|
|
|
## LM Studio or another runtime says the model barely does not fit
|
|
|
|
whichllm estimates model memory, but real runtimes can need extra room for
|
|
loader overhead, graph buffers, KV cache choices, and OS pressure. By default,
|
|
whichllm reserves a small automatic VRAM headroom before fit checks.
|
|
|
|
Tune it with:
|
|
|
|
```bash
|
|
whichllm --vram-headroom 1.5GB
|
|
whichllm --vram-headroom 10%
|
|
whichllm --vram-headroom none
|
|
```
|
|
|
|
Use `none` when you want the old raw-VRAM behavior.
|
|
|
|
## RAM offload depends on what else is running
|
|
|
|
Partial offload uses system RAM. If Docker, Elasticsearch, a browser, or
|
|
another workload is already using a large amount of memory, cap the offload
|
|
budget:
|
|
|
|
```bash
|
|
whichllm --ram-budget available
|
|
whichllm --ram-budget 8GB
|
|
whichllm --ram-budget 50%
|
|
```
|
|
|
|
`available` reads the current available RAM from the OS at startup. Fixed
|
|
values are useful when you know how much memory you want to leave for other
|
|
processes.
|
|
|
|
## Results look stale
|
|
|
|
whichllm caches model data for 6 hours and benchmark data for 24 hours.
|
|
|
|
Force a refresh:
|
|
|
|
```bash
|
|
whichllm --refresh
|
|
whichllm plan "qwen 7b" --refresh
|
|
```
|
|
|
|
The caches live under:
|
|
|
|
```text
|
|
~/.cache/whichllm/
|
|
```
|
|
|
|
If `XDG_CACHE_HOME` is set to an absolute path, the caches live under:
|
|
|
|
```text
|
|
$XDG_CACHE_HOME/whichllm/
|
|
```
|
|
|
|
## `uvx` fails with `realpath: command not found`
|
|
|
|
Some older macOS versions do not include a `realpath` command. If the `uvx`
|
|
launcher fails before whichllm starts, with output like:
|
|
|
|
```text
|
|
realpath: command not found
|
|
/Users/.../python: No such file or directory
|
|
```
|
|
|
|
run whichllm through Python's module entry point instead:
|
|
|
|
```bash
|
|
uvx --from whichllm python -m whichllm
|
|
```
|
|
|
|
Pass normal whichllm arguments after the module name:
|
|
|
|
```bash
|
|
uvx --from whichllm python -m whichllm --gpu "RTX 4090"
|
|
```
|
|
|
|
## The top pick has `~`, `!sr`, or `?`
|
|
|
|
These markers describe benchmark evidence:
|
|
|
|
| Marker | Meaning |
|
|
| --- | --- |
|
|
| `~` | Inherited or interpolated benchmark evidence |
|
|
| `!sr` | Uploader-reported benchmark only |
|
|
| `?` | No benchmark evidence |
|
|
|
|
Use stricter evidence when you want only independently matched benchmark data:
|
|
|
|
```bash
|
|
whichllm --evidence strict
|
|
whichllm --direct
|
|
```
|
|
|
|
Use `--evidence base` when base-model matches are acceptable but interpolation
|
|
and self-reported values are not.
|
|
|
|
## The largest model did not win
|
|
|
|
That is expected. whichllm scores:
|
|
|
|
- benchmark quality
|
|
- model size
|
|
- quantization loss
|
|
- full GPU vs partial offload vs CPU-only
|
|
- estimated speed
|
|
- evidence confidence
|
|
- source trust
|
|
- generation lineage
|
|
|
|
A smaller current-generation model with strong direct evidence can beat a
|
|
larger model that only barely fits or relies on stale benchmark data.
|
|
|
|
## Estimated speed differs from real speed
|
|
|
|
Speed is an estimate based on:
|
|
|
|
- model weight size
|
|
- MoE active parameters
|
|
- GPU memory bandwidth
|
|
- quantization efficiency
|
|
- backend factor
|
|
- partial-offload penalty
|
|
|
|
Real performance depends on the inference runtime, driver, prompt length,
|
|
batching, thermal limits, and background memory pressure.
|
|
|
|
The default ranking table shows the speed estimate and its confidence marker.
|
|
Use `--details` only when you want download counts instead.
|
|
|
|
Speed colors and markers:
|
|
|
|
- red: slow generation speed, under 4 tok/s
|
|
- yellow: marginal generation speed, 4-10 tok/s
|
|
- green: usable generation speed, 10-30 tok/s
|
|
- bright green: fast local generation speed, 30+ tok/s
|
|
- `~`: estimated speed range is available
|
|
- `?`: low-confidence estimate; runtime/backend differences can be large
|
|
|
|
JSON includes the same information as `speed_confidence`,
|
|
`speed_range_tok_per_sec`, and `speed_notes`.
|
|
|
|
## Apple Silicon partial offload looks different
|
|
|
|
Apple Silicon uses unified memory. Partial offload does not cross a discrete
|
|
PCIe boundary, so whichllm applies a milder speed penalty than it does for
|
|
discrete GPUs.
|
|
|
|
The same is true for recognized AMD shared-memory APUs such as Strix Halo,
|
|
Ryzen AI MAX, and Ryzen AI / Radeon 890M-class integrated graphics.
|
|
DGX Spark / NVIDIA GB10 is handled the same way when NVIDIA reports GPU memory
|
|
as unavailable.
|
|
|
|
On Windows, `Win32_VideoController.AdapterRAM` can cap around 4 GB. whichllm
|
|
uses the 64-bit registry memory value when it is available, and treats known
|
|
shared-memory APUs as unified-memory style devices instead of tiny discrete
|
|
GPUs.
|
|
|
|
## `run` says `uv is required`
|
|
|
|
Install `uv` first:
|
|
|
|
```bash
|
|
curl -LsSf https://astral.sh/uv/install.sh | sh
|
|
```
|
|
|
|
Then retry:
|
|
|
|
```bash
|
|
whichllm run
|
|
```
|
|
|
|
## `run` cannot download a model
|
|
|
|
Possible causes:
|
|
|
|
- the model is gated on HuggingFace
|
|
- local HuggingFace authentication is missing
|
|
- the selected GGUF filename no longer exists
|
|
- network access failed
|
|
- disk space is too low
|
|
|
|
Try a known public GGUF model first:
|
|
|
|
```bash
|
|
whichllm run "qwen 2.5 1.5b gguf"
|
|
```
|
|
|
|
## Hugging Face API access fails or needs a mirror
|
|
|
|
whichllm uses the Hugging Face API to fetch model metadata. If direct access to
|
|
`huggingface.co` fails in your network, set `HF_ENDPOINT` to a compatible
|
|
endpoint root:
|
|
|
|
```powershell
|
|
$env:HF_ENDPOINT = "https://huggingface.co"
|
|
whichllm --refresh
|
|
```
|
|
|
|
```bash
|
|
HF_ENDPOINT="https://huggingface.co" whichllm --refresh
|
|
```
|
|
|
|
Do not include `/api` in `HF_ENDPOINT`; whichllm adds that path internally.
|
|
|
|
## How much disk space does `run` need?
|
|
|
|
Normal ranking commands do not download model weights. They cache Hugging Face
|
|
model metadata and benchmark metadata under the whichllm cache.
|
|
|
|
`whichllm run` downloads the selected GGUF file through `huggingface_hub`. The
|
|
required disk space is roughly the selected GGUF file size plus normal Hugging
|
|
Face cache overhead.
|
|
|
|
By default, Hugging Face stores downloaded files under:
|
|
|
|
```text
|
|
~/.cache/huggingface/hub
|
|
```
|
|
|
|
You can move that cache by setting `HF_HOME` or `HF_HUB_CACHE`.
|
|
|
|
Cleanup is handled by the Hugging Face cache tools:
|
|
|
|
```bash
|
|
hf cache scan
|
|
hf cache delete
|
|
```
|
|
|
|
whichllm does not currently delete model files automatically after a run.
|
|
|
|
## Ollama names do not match HuggingFace IDs
|
|
|
|
JSON output returns HuggingFace repo IDs:
|
|
|
|
```bash
|
|
whichllm --top 1 --json | jq -r '.models[0].model_id'
|
|
```
|
|
|
|
Ollama model names often use a different naming scheme. Map the HuggingFace ID
|
|
to your local Ollama model name before calling `ollama run`.
|
|
|
|
## Debugging a specific model
|
|
|
|
Use `plan` to inspect memory requirements:
|
|
|
|
```bash
|
|
whichllm plan "Qwen2.5-72B" --quant Q4_K_M
|
|
whichllm plan "Qwen2.5-72B" --quant Q8_0 --context-length 32768
|
|
```
|
|
|
|
Use plain output when filing issues:
|
|
|
|
```bash
|
|
whichllm --gpu "RTX 4090" --json
|
|
whichllm --gpu "RTX 4090" --markdown
|
|
whichllm hardware
|
|
```
|
|
|
|
Include:
|
|
|
|
- OS
|
|
- GPU name and VRAM
|
|
- CPU and RAM
|
|
- command used
|
|
- expected result
|
|
- actual result
|