andyyyy64--whichllm/docs/troubleshooting.md

Troubleshooting

This page lists common issues and the first checks to make.

No GPU detected

Run:

whichllm hardware

If an NVIDIA GPU is missing:

• check that the driver is installed
• check nvidia-smi
• check that nvidia-ml-py can load NVML

whichllm falls back to nvidia-smi, but it still needs the NVIDIA driver tools to be working.

If an AMD GPU is missing:

• on Linux, check rocm-smi, lspci, and /sys/class/drm
• on Windows, check that PowerShell can read Win32_VideoController
• for Ryzen AI / Radeon integrated graphics, check whether whichllm hardware shows shared memory instead of a tiny 512 MB or 4 GB adapter

If an Intel iGPU is missing:

• Linux detection uses lspci or /sys/class/drm
• Windows detection uses Win32_VideoController
• many Intel iGPUs do not expose dedicated VRAM, so they may be shown as shared memory graphics

Simulate hardware instead

If detection is unavailable or you are planning a purchase, use --gpu:

whichllm --gpu "RTX 4090"
whichllm hardware --gpu "Apple M3 Max"
whichllm --gpu "RTX 5060 Ti" --vram 16
whichllm --gpu "2x RTX 4090"
whichllm --gpu "RTX 4090" --gpu "RTX 3090"

Use --vram when the GPU name has multiple memory variants or is not in the database.

For detected iGPU or unified-memory systems, override the usable GPU memory and bandwidth directly:

whichllm hardware --vram 8 --ram-bandwidth 68
whichllm --vram 8 --bandwidth 68

If whichllm hardware lists multiple GPUs, add --gpu-index with the GPU number from that output.

--vram only applies to one simulated or detected GPU. For multi-GPU simulation, use known GPU names and omit --vram.

--cpu-only conflicts with --gpu

These flags are mutually exclusive:

whichllm --cpu-only --gpu "RTX 4090"

Choose one:

whichllm --cpu-only
whichllm --gpu "RTX 4090"

--vram / --bandwidth needs a GPU

These overrides need either a detected GPU or a simulated GPU:

whichllm --vram 8 --ram-bandwidth 68
whichllm --gpu "RTX 3060" --vram 12

If no GPU is detected, use --gpu to simulate one or check the detection steps above.

No compatible models found

Try:

whichllm
whichllm --cpu-only
whichllm --refresh

Common causes:

• the selected --quant is too restrictive
• --gpu-only or --fit full-gpu filters out partial-offload and CPU-only candidates
• --speed usable, --speed fast, or --min-speed filters out slower candidates
• --min-speed is too high
• --evidence strict filters out all candidates
• the requested context length is too large
• available RAM is too low after reserving space for the OS
• disk free space is too low for the model weights

For very small machines, remove optional filters first:

whichllm --top 20

Recommendations use RAM or CPU offload, but I only want VRAM

By default, whichllm includes any runnable candidate: full-GPU, partial-offload, and CPU-only. This is useful for finding what can run at all, but it can be too loose when you want only models that fit entirely in GPU VRAM.

Use:

whichllm --gpu-only
whichllm --fit gpu
whichllm --fit full-gpu

If no rows are shown, this machine has no ranked candidates that fit fully in GPU memory under the current filters. Remove --gpu-only, lower the context length, or try a smaller quantization.

A model fits, but it is too slow

The default ranking table shows estimated generation speed. Slow rows are red, marginal rows are yellow, usable rows are green, and fast rows are bright green. The ~ and ? markers are confidence markers for the estimate.

Filter slow rows with:

whichllm --speed usable  # >=10 tok/s
whichllm --speed fast    # >=30 tok/s
whichllm --min-speed 4   # exact floor, if you want a lower threshold

For an exact threshold:

whichllm --min-speed 10

LM Studio or another runtime says the model barely does not fit

whichllm estimates model memory, but real runtimes can need extra room for loader overhead, graph buffers, KV cache choices, and OS pressure. By default, whichllm reserves a small automatic VRAM headroom before fit checks.

Tune it with:

whichllm --vram-headroom 1.5GB
whichllm --vram-headroom 10%
whichllm --vram-headroom none

Use none when you want the old raw-VRAM behavior.

RAM offload depends on what else is running

Partial offload uses system RAM. If Docker, Elasticsearch, a browser, or another workload is already using a large amount of memory, cap the offload budget:

whichllm --ram-budget available
whichllm --ram-budget 8GB
whichllm --ram-budget 50%

available reads the current available RAM from the OS at startup. Fixed values are useful when you know how much memory you want to leave for other processes.

Results look stale

whichllm caches model data for 6 hours and benchmark data for 24 hours.

Force a refresh:

whichllm --refresh
whichllm plan "qwen 7b" --refresh

The caches live under:

~/.cache/whichllm/

If XDG_CACHE_HOME is set to an absolute path, the caches live under:

$XDG_CACHE_HOME/whichllm/

uvx fails with realpath: command not found

Some older macOS versions do not include a realpath command. If the uvx launcher fails before whichllm starts, with output like:

realpath: command not found
/Users/.../python: No such file or directory

run whichllm through Python's module entry point instead:

uvx --from whichllm python -m whichllm

Pass normal whichllm arguments after the module name:

uvx --from whichllm python -m whichllm --gpu "RTX 4090"

The top pick has ~, !sr, or ?

These markers describe benchmark evidence:

Marker	Meaning
~	Inherited or interpolated benchmark evidence
!sr	Uploader-reported benchmark only
?	No benchmark evidence

Use stricter evidence when you want only independently matched benchmark data:

whichllm --evidence strict
whichllm --direct

Use --evidence base when base-model matches are acceptable but interpolation and self-reported values are not.

The largest model did not win

That is expected. whichllm scores:

• benchmark quality
• model size
• quantization loss
• full GPU vs partial offload vs CPU-only
• estimated speed
• evidence confidence
• source trust
• generation lineage

A smaller current-generation model with strong direct evidence can beat a larger model that only barely fits or relies on stale benchmark data.

Estimated speed differs from real speed

Speed is an estimate based on:

• model weight size
• MoE active parameters
• GPU memory bandwidth
• quantization efficiency
• backend factor
• partial-offload penalty

Real performance depends on the inference runtime, driver, prompt length, batching, thermal limits, and background memory pressure.

The default ranking table shows the speed estimate and its confidence marker. Use --details only when you want download counts instead.

Speed colors and markers:

• red: slow generation speed, under 4 tok/s
• yellow: marginal generation speed, 4-10 tok/s
• green: usable generation speed, 10-30 tok/s
• bright green: fast local generation speed, 30+ tok/s
• ~: estimated speed range is available
• ?: low-confidence estimate; runtime/backend differences can be large

JSON includes the same information as speed_confidence, speed_range_tok_per_sec, and speed_notes.

Apple Silicon partial offload looks different

Apple Silicon uses unified memory. Partial offload does not cross a discrete PCIe boundary, so whichllm applies a milder speed penalty than it does for discrete GPUs.

The same is true for recognized AMD shared-memory APUs such as Strix Halo, Ryzen AI MAX, and Ryzen AI / Radeon 890M-class integrated graphics. DGX Spark / NVIDIA GB10 is handled the same way when NVIDIA reports GPU memory as unavailable.

On Windows, Win32_VideoController.AdapterRAM can cap around 4 GB. whichllm uses the 64-bit registry memory value when it is available, and treats known shared-memory APUs as unified-memory style devices instead of tiny discrete GPUs.

run says uv is required

Install uv first:

curl -LsSf https://astral.sh/uv/install.sh | sh

Then retry:

whichllm run

run cannot download a model

Possible causes:

• the model is gated on HuggingFace
• local HuggingFace authentication is missing
• the selected GGUF filename no longer exists
• network access failed
• disk space is too low

Try a known public GGUF model first:

whichllm run "qwen 2.5 1.5b gguf"

Hugging Face API access fails or needs a mirror

whichllm uses the Hugging Face API to fetch model metadata. If direct access to huggingface.co fails in your network, set HF_ENDPOINT to a compatible endpoint root:

$env:HF_ENDPOINT = "https://huggingface.co"
whichllm --refresh

HF_ENDPOINT="https://huggingface.co" whichllm --refresh

Do not include /api in HF_ENDPOINT; whichllm adds that path internally.

How much disk space does run need?

Normal ranking commands do not download model weights. They cache Hugging Face model metadata and benchmark metadata under the whichllm cache.

whichllm run downloads the selected GGUF file through huggingface_hub. The required disk space is roughly the selected GGUF file size plus normal Hugging Face cache overhead.

By default, Hugging Face stores downloaded files under:

~/.cache/huggingface/hub

You can move that cache by setting HF_HOME or HF_HUB_CACHE.

Cleanup is handled by the Hugging Face cache tools:

hf cache scan
hf cache delete

whichllm does not currently delete model files automatically after a run.

Ollama names do not match HuggingFace IDs

JSON output returns HuggingFace repo IDs:

whichllm --top 1 --json | jq -r '.models[0].model_id'

Ollama model names often use a different naming scheme. Map the HuggingFace ID to your local Ollama model name before calling ollama run.

Debugging a specific model

Use plan to inspect memory requirements:

whichllm plan "Qwen2.5-72B" --quant Q4_K_M
whichllm plan "Qwen2.5-72B" --quant Q8_0 --context-length 32768

Use plain output when filing issues:

whichllm --gpu "RTX 4090" --json
whichllm --gpu "RTX 4090" --markdown
whichllm hardware

Include:

• OS
• GPU name and VRAM
• CPU and RAM
• command used
• expected result
• actual result