andyyyy64--whichllm/docs/hardware.md

Hardware detection and simulation

whichllm detects the current machine and can also simulate hardware for purchase planning.

The source of truth is the hardware/ package plus curated registry data in data/gpu.py. constants.py remains as a compatibility export layer for older imports.

Detected data

The ranker receives a HardwareInfo object with:

• GPU list
• CPU name
• physical CPU cores
• AVX2 and AVX-512 support
• total RAM
• free disk space
• OS name

Each GPU is represented as GPUInfo:

• name
• vendor
• VRAM bytes
• usable VRAM bytes, when a runtime headroom is active
• NVIDIA compute capability, when known
• CUDA or ROCm version, when known
• memory bandwidth estimate
• whether the GPU uses shared memory

NVIDIA

NVIDIA detection tries nvidia-ml-py first. If NVML is unavailable, fails to initialize, or returns no devices, whichllm falls back to:

nvidia-smi --query-gpu=name,memory.total,clocks.max.memory --format=csv,noheader,nounits

If a driver rejects clocks.max.memory, whichllm retries the older name,memory.total query.

For known cards, curated data and strict dbgpu lookups provide:

• memory bandwidth
• compute capability

The max memory clock is used when a marketing name covers multiple memory types. For example, GTX 1650 GDDR5 and GDDR6 cards share the same broad driver name, so whichllm uses the reported memory clock when available and falls back to the conservative bandwidth when it is not.

DGX Spark / NVIDIA GB10 uses unified system memory. When the driver reports memory.total as unavailable, whichllm treats GB10 as shared memory and uses system RAM for fit checks.

Compute capability is used to warn when a card is below the minimum expected by common local inference tools.

AMD

On Linux, AMD detection tries rocm-smi first:

• product name
• VRAM
• ROCm driver version

If rocm-smi is unavailable, it falls back to lspci and then /sys/class/drm.

On Windows, whichllm uses Win32_VideoController as a fallback for AMD GPUs. When possible, it also reads the 64-bit dedicated-memory value from the Control\Video registry path because AdapterRAM is a 32-bit field and can cap larger cards around 4 GB.

AMD shared-memory APUs are treated differently from discrete GPUs. Names such as Strix Halo, Ryzen AI MAX, Radeon 8050S, Radeon 8060S, Radeon 890M, and Radeon 780M are modeled as shared-memory systems. If the reported VRAM is just a small aperture, whichllm uses the system memory pool for fit checks instead of treating it as a tiny discrete GPU.

Intel

Intel integrated GPUs are detected on Linux through lspci or sysfs, and on Windows through Win32_VideoController. They do not normally report dedicated VRAM, so whichllm records them with 0 dedicated VRAM and labels them as shared memory.

Discrete Intel Arc cards are kept as dedicated-memory GPUs when the device name and memory report look like a discrete adapter.

The Intel backend factor is lower than NVIDIA, AMD, and Apple because local LLM GPU inference support is less mature.

Apple Silicon

On macOS, whichllm uses:

system_profiler SPHardwareDataType -json

Apple Silicon uses unified memory, so the detected chip memory is treated as available GPU memory. Memory bandwidth is looked up by chip family when known.

Partial offload on Apple Silicon is not penalized like discrete PCIe offload. Weights still live in unified memory, so the speed penalty is milder.

CPU and memory

CPU detection reads:

• /proc/cpuinfo on Linux
• sysctl on macOS
• wmic on Windows, then PowerShell / CIM when wmic is unavailable or only returns a header

Physical core count comes from psutil, with a Linux /proc/cpuinfo fallback.

RAM comes from psutil.virtual_memory(). Disk free space is checked under the user's home directory by default.

GPU simulation

Use --gpu to simulate a GPU:

whichllm --gpu "RTX 4090"
whichllm hardware --gpu "Apple M3 Max"
whichllm upgrade "RTX 4090" "RTX 5090" "H100"

Simulation uses the dbgpu package for a TechPowerUp-backed GPU database. whichllm adds extra handling for common aliases and Apple Silicon chips because those are not covered by dbgpu.

Use --vram when a GPU name is ambiguous, unknown, or has multiple variants:

whichllm --gpu "RTX 5060 Ti" --vram 16
whichllm hardware --gpu "Unknown GPU" --vram 24

For detected integrated or unified-memory GPUs, use --vram and --bandwidth / --ram-bandwidth to override the automatically detected usable capacity and memory bandwidth:

whichllm --vram 8 --ram-bandwidth 68
whichllm hardware --vram 8 --bandwidth 68

If multiple GPUs are detected, pass --gpu-index to choose the GPU shown by whichllm hardware:

whichllm --gpu-index 1 --vram 8 --ram-bandwidth 68

By default, whichllm applies a small automatic VRAM headroom before fit checks. This avoids recommending models that only fit on paper but overflow in runtimes that need extra graph buffers or loader overhead. Tune it with:

whichllm --vram-headroom 1.5GB
whichllm --vram-headroom 10%
whichllm --vram-headroom none

Multi-GPU simulation accepts repeated flags, comma-separated values, and count shorthand:

whichllm --gpu "2x RTX 4090"
whichllm --gpu "RTX 4090" --gpu "RTX 3090"
whichllm --gpu "RTX 4090, RTX 3090"

--vram is only supported for a single simulated GPU. For multi-GPU simulation, use known GPU names so whichllm can resolve each card's VRAM from the GPU database.

Fit types

Compatibility checks classify a candidate into one of three fit types:

Fit	Meaning
full_gpu	Required memory fits in available GPU memory
partial_offload	GPU plus usable system RAM can hold the model
cpu_only	Usable system RAM can hold the model without GPU

If neither GPU memory nor usable RAM can hold the model, the candidate is not ranked.

whichllm keeps a bounded system-RAM reserve for the OS and other processes. Use --ram-budget available to cap partial-offload planning to the current available RAM reported by the OS, or pass a fixed budget such as --ram-budget 8GB.

Multiple GPUs

For fit checks, whichllm uses a conservative multi-GPU budget rather than pretending all VRAM is one perfect device. It starts from raw total VRAM, applies a small per-GPU overhead, and then applies a utilization factor. Homogeneous sets receive a less severe reduction than heterogeneous sets.

If a dedicated GPU is present, low-aperture shared-memory integrated GPUs are not added to the fit pool. This avoids treating unrelated system RAM and dedicated VRAM as one full-GPU target.

For speed estimates, whichllm uses the largest detected GPU as the representative device and marks multi-GPU speed as low-confidence. This avoids claiming ideal scaling when real performance depends on backend split mode, PCIe/NVLink bandwidth, NCCL/RCCL support, batch size, and model architecture.

This is a practical fit approximation. It does not model every tensor-parallel or pipeline-parallel runtime configuration.

Disk checks

The compatibility check also compares estimated model weight size with free disk space. If the model cannot be downloaded, it is marked unrunnable.

Known limitations

• GPU bandwidth is a lookup or database estimate, not a live benchmark.
• Speed estimates are planning numbers. The default table and JSON fields such as speed_confidence and speed_range_tok_per_sec show uncertainty.
• Driver, runtime, batch size, prompt length, and thermal limits can change real performance.
• Multi-GPU runtime behavior depends on the inference backend and is only approximated.
• Apple and shared-memory APU behavior is modeled as unified-memory style, but real results still depend on OS pressure and memory bandwidth.