wehub/andyyyy64--whichllm
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:29:01 +08:00
commit e90a966536
127 changed files with 21798 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/FUNDING.yml
+1
View File
@@ -0,0 +1 @@
github: [Andyyyy64]
.github/ISSUE_TEMPLATE/bug_report.yml
+46
View File
@@ -0,0 +1,46 @@
name: Bug Report
description: Report a bug or unexpected behavior
labels: ["bug"]
body:
- type: markdown
attributes:
value: |
Thanks for reporting a bug! Please fill out the information below.
- type: textarea
id: description
attributes:
label: Description
description: What happened? What did you expect to happen?
validations:
required: true
- type: textarea
id: reproduce
attributes:
label: Steps to Reproduce
description: How can we reproduce this issue?
placeholder: |
1. Run `whichllm --gpu "RTX 4090"`
2. ...
validations:
required: true
- type: textarea
id: hardware
attributes:
label: Hardware Info
description: Paste the output of `whichllm hardware`
render: shell
- type: input
id: python-version
attributes:
label: Python Version
placeholder: "3.11"
- type: input
id: os
attributes:
label: Operating System
placeholder: "Ubuntu 24.04 / macOS 15 / Windows 11"
- type: input
id: version
attributes:
label: whichllm Version
placeholder: "0.3.0"
.github/ISSUE_TEMPLATE/feature_request.yml
+27
View File
@@ -0,0 +1,27 @@
name: Feature Request
description: Suggest a new feature or improvement
labels: ["enhancement"]
body:
- type: markdown
attributes:
value: |
Thanks for suggesting a feature! Please describe what you'd like to see.
- type: textarea
id: problem
attributes:
label: Problem
description: What problem does this feature solve?
validations:
required: true
- type: textarea
id: solution
attributes:
label: Proposed Solution
description: How would you like this to work?
validations:
required: true
- type: textarea
id: alternatives
attributes:
label: Alternatives Considered
description: Any alternative solutions you've considered?
.github/PULL_REQUEST_TEMPLATE.md
+17
View File
@@ -0,0 +1,17 @@
## What
<!-- Brief description of changes -->
## Why
<!-- Why is this change needed? Link to issue if applicable -->
## Testing
- [ ] Tests pass (`pytest`)
- [ ] New tests added (if applicable)
- [ ] Tested on real hardware (if hardware-related)
## Notes
<!-- Any additional context -->
.github/workflows/lint.yml
+25
View File
@@ -0,0 +1,25 @@
name: Lint
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: "3.13"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install ruff
- name: Run Ruff linter
run: ruff check .
- name: Run Ruff formatter check
run: ruff format --check .
.github/workflows/publish.yml
+27
View File
@@ -0,0 +1,27 @@
name: Publish to PyPI
on:
release:
types: [published]
permissions:
id-token: write
jobs:
publish:
runs-on: ubuntu-latest
environment: pypi
steps:
- uses: actions/checkout@v5
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: "3.13"
- name: Install build tools
run: |
python -m pip install --upgrade pip
pip install build
- name: Build package
run: python -m build
- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
.github/workflows/test.yml
+28
View File
@@ -0,0 +1,28 @@
name: Tests
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v5
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e .
pip install pytest pyarrow
- name: Run tests
run: pytest
.gitignore
+42
View File
@@ -0,0 +1,42 @@
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
*.egg
*.egg-info/
dist/
build/
*.whl
# Virtual environments
.venv/
venv/
env/
# Testing
.pytest_cache/
.coverage
htmlcov/
.mypy_cache/
.ruff_cache/
# IDE
.vscode/
.idea/
*.swp
*.swo
*~
# OS
.DS_Store
Thumbs.db
# Environment
.env
.env.local
.env.*.local
# Logs
*.log
.claude/
.pre-commit-config.yaml
+7
View File
@@ -0,0 +1,7 @@
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.15.20
hooks:
- id: ruff-check
args: [--fix]
- id: ruff-format
CHANGELOG.md
+429
View File
@@ -0,0 +1,429 @@
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/), and this project adheres to [Semantic Versioning](https://semver.org/).
## [Unreleased]
## [0.5.15] - 2026-07-03
### Changed
- Split the oversized Hugging Face model fetcher into focused modules while
keeping the existing `whichllm.models.fetcher` import surface. (#41, #140)
### Fixed
- Output now resolves ranked GGUF recommendations to the actual downloadable
artifact repo and filename when the ranked base model and runnable GGUF live
in different Hugging Face repos. (#137, #138)
- AA index scoring now uses retuned normalization bounds and a refreshed
fallback snapshot for the reworked Artificial Analysis scale. (#101, #139)
- Invalid ranking filters such as `--top 0`, negative `--min-speed`, and
negative `--min-params` now fail with clear errors instead of silently
producing wrong output. (#142)
## [0.5.14] - 2026-06-29
### Added
- Added sliding-window attention metadata to model fetching and KV cache
estimation, improving VRAM estimates for models that use SWA. (#124)
- Added curated Intel Arc Pro B70 / Battlemage G31 detection and simulation,
including the `0xe223` PCI device ID and 32 GB VRAM / 608 GB/s bandwidth
defaults. (#93, #136)
### Fixed
- Model and benchmark metadata fetches now request `gzip, deflate` instead of
brotli, avoiding broken `br` responses from mirrors or intermediate servers.
(#128, #136)
## [0.5.13] - 2026-06-25
### Added
- Added `HF_ENDPOINT` support for Hugging Face model metadata fetches, so users
behind a mirror can point whichllm at a compatible Hub endpoint. (#128, #131)
- Added manual detected-GPU overrides for usable VRAM and bandwidth, which helps
iGPU and unified-memory systems where automatic detection is too conservative.
(#132, #133)
- README now points users toward safer first-run flags when they want
full-GPU, usable-speed recommendations with extra VRAM headroom.
### Fixed
- Search terms such as `7B`, `0.5B`, and `500M` now match model parameter size
instead of plain substrings, so `qwen 7b` no longer returns `1.7B` or
`30B-A3B` models by accident. (#107, #126)
- GGUF sizing now treats FP16 and ternary `TQ1_0` / `TQ2_0` quant types
correctly, avoiding underestimates for those files. (#125)
## [0.5.12] - 2026-06-18
### Added
- Default ranking tables now show memory required, estimated generation speed,
fit type, and published date. Download counts are still available with
`--details`.
- Added `--speed any|usable|fast` as named generation-speed filters while
keeping `--min-speed` for exact tok/s thresholds.
- Added `--fit gpu` as a natural alias for full-GPU-only recommendations.
- Added `--markdown` / `-m` for pasteable GitHub-Flavored Markdown ranking
tables. (#111)
- Added `--vram-headroom` and `--ram-budget` so users can avoid edge VRAM fits
and cap partial-offload planning to available or fixed system RAM.
### Changed
- Speed color now reflects practical generation speed. `~` and `?` remain
estimate-confidence markers instead of being the primary speed color.
## [0.5.11] - 2026-06-18
### Added
- Multi-GPU simulation for repeated `--gpu` flags, comma-separated GPU specs,
and count shorthand such as `2x RTX 4090`. The fit model uses a conservative
effective VRAM budget and keeps speed confidence low for split-device
recommendations. (#113)
- `python -m whichllm` now runs the CLI entrypoint. (#116)
- `--gpu-only` and `--fit full-gpu` now filter recommendations to models that
fit fully in GPU VRAM. `--fit any` keeps the existing behavior. (#119, #122)
- T5 lineage support so T5-family models get version-aware benchmark handling.
### Fixed
- Fixed UTF-8 decoding for cached model and benchmark data on systems whose
default filesystem encoding is not UTF-8. (#121)
- GTX 1650 simulation now distinguishes GDDR5 and GDDR6 variants by memory
clock instead of treating every card as the slower 128 GB/s model. (#115)
- RAM reserve logic now uses a bounded reserve formula instead of a fixed 80%
usable-RAM cap, which avoids underestimating machines with more system
memory. (#103)
## [0.5.10] - 2026-06-11
### Fixed
- Strong partial-offload candidates are no longer buried below weaker full-GPU
models because the final ranking sort no longer counts full-GPU fit a second
time after runtime-fit and speed penalties have already been applied. Light
partial offload is penalized less aggressively, while heavy dense offload
remains strongly discounted. (#105, #108)
- MoE partial-offload scoring now uses the active parameter working set when it
can plausibly stay on GPU, so active-small MoE models are not penalized like
dense models with the same total parameter count. (#105, #108)
## [0.5.9] - 2026-06-10
### Added
- MXFP4 and NVFP4 4-bit quantization support across ID/filename parsing, VRAM
estimation, quality penalties, speed efficiency, and family grouping.
Repos shipping these formats were previously labeled FP16 and their VRAM
requirement overestimated about 3.5x. (#99)
- Apple M5-family entries for `--gpu` simulation. (#92)
- Kepler-era Quadro bandwidth and compute capability entries. (#75)
### Fixed
- AMD discrete GPU detection on Linux: rocm-smi names are read from the
correct `Card Series` key, compound lspci names such as
`Navi 22 [Radeon RX 6700/6700 XT/6750 XT ...]` resolve bandwidth, sysfs VRAM
enriches the fallback path, and discrete cards are no longer mislabeled
`shared memory`. Adds RX 6750 XT / RX 6700 / RX 6650 XT / RX 6600 series and
Radeon AI PRO R9700 to the bandwidth catalog. (#61, #68)
- Community GGUF repos without `base_model` metadata (for example
`unsloth/...-GGUF`) now inherit the official model's benchmark score by
name matching instead of falling through to no evidence. (#94)
- GPU bandwidth detection no longer depends solely on the hand-curated
catalog. When a detected card is missing from `GPU_BANDWIDTH`, bandwidth is
now resolved from the bundled TechPowerUp database (dbgpu, 2824 GPUs) using
strict name matching only: an exact normalized hit or a name plus VRAM-size
bin, never fuzzy. Laptop / Mobile / Max-Q names can no longer inherit a
desktop card's bandwidth, and VRAM bins written without a space
(`RTX A2000 12GB`) are recognized. This fixes the cluster of reports where
an uncatalogued GPU showed `BW: N/A`, was estimated at `0.0 tok/s`, and
received oversized recommendations (#74, #98).
- Artificial Analysis Intelligence Index is fetched live again. The
artificialanalysis.ai leaderboard migrated to the Next.js App Router and no
longer ships a `__NEXT_DATA__` blob, so every run logged
`AA Index fetch failed ... __NEXT_DATA__ payload not found` and silently used
the frozen snapshot. The scraper now parses the App Router RSC stream
(`self.__next_f.push(...)`), canonicalizes AA's variant-suffixed display
names (`(Reasoning)`, `(high)`, ...) for mapping, and overlays live scores on
top of the curated fallback so a successful fetch can only add coverage. The
legacy `__NEXT_DATA__` path is kept as a secondary fallback.
## [0.5.8] - 2026-06-05
### Added
- `--context-length` now accepts shorthand values such as `64k` and `128k`.
- JSON ranking output now includes benchmark source and confidence metadata.
- Asahi Linux / Apple Silicon detection now recognizes Apple CPU and GPU names.
- Added GPU catalog coverage for `NVIDIA RTX A3000 Laptop GPU`, `RTX 3050`,
`RTX 5060`, `RTX 5070 Ti`, `RX 9070`, and `RX 9070 XT`.
### Fixed
- A3000 Laptop 6GB systems no longer get `0.0 tok/s` / heavy partial-offload
recommendations at the top just because bandwidth was missing.
- Windows CPU detection now falls back through PowerShell/CIM when `wmic` does
not return a useful CPU name.
- Models that cannot hold the requested context are demoted instead of staying
near the top of the ranking.
- Hugging Face and benchmark fetches now retry transient failures such as 429s
before falling back or failing.
- `Error fetching models:` now includes useful detail even when the underlying
network exception message is empty.
- Upgrade tables now show `0 GB` VRAM instead of treating zero as missing.
### Changed
- Curated registry data was split out of `constants.py` into
`whichllm.data.*` modules.
- Troubleshooting and cache documentation now better explain disk-cache paths
and stale fetch behavior.
## [0.5.7] - 2026-05-20
### Added
- LiveBench fallback data is now kept inline so benchmark scoring remains
available without relying on a generated sidecar file.
### Fixed
- DGX Spark / NVIDIA GB10 is now detected as a shared-memory NVIDIA GPU when
NVIDIA reports `memory.total` as unavailable.
- `whichllm run` now provides a Transformers `offload_folder`, avoiding crashes
when large models need disk offload.
- Cache paths now respect `XDG_CACHE_HOME`, including ignoring relative values
per the XDG base directory specification.
- Apple Silicon is now treated as shared memory in fit detection.
- Benchmark score fetching now runs concurrently.
## [0.5.6] - 2026-05-18
### Added
- Speed estimates now include confidence metadata and an estimated tok/s range
in table and JSON output, so uncertain backend/model predictions are visible.
- Windows now has an AMD/Intel GPU detection fallback via
`Win32_VideoController`, including 64-bit registry memory reads for GPUs
where `AdapterRAM` is capped around 4 GB.
### Fixed
- MoE speed estimates now use active-parameter metadata and a
bandwidth-scaled read floor, improving shared-memory APU estimates without
over-promoting sparse models on high-bandwidth GPUs.
- Newer MoE model metadata now recognizes A3B-style active-parameter names.
- Ryzen AI / Radeon 890M-class Windows iGPUs are modeled as shared-memory AMD
GPUs instead of CPU-only or tiny-VRAM discrete GPUs.
- Mixed dedicated-GPU plus shared-memory-iGPU systems no longer sum unrelated
memory pools as one full-GPU target.
- Windows AMD GPUs no longer receive a misleading ROCm-only warning when
Vulkan or DirectML backends may be valid.
## [0.5.5] - 2026-05-17
### Fixed
- `whichllm run` now resolves auto-picked GGUF recommendations to a real
GGUF repository and file before launch, instead of falling back to the
official Transformers repository. This fixes the accidental Transformers path
for models such as `Qwen/Qwen3.6-27B`.
## [0.5.4] - 2026-05-17
### Fixed
- Strix Halo / Ryzen AI MAX systems are now modeled as AMD shared-memory APUs
instead of tiny-VRAM discrete GPUs. `STRXLGEN`, `Radeon 8050S`,
`Radeon 8060S`, and related names get a 256 GB/s bandwidth estimate and use
the system shared-memory pool for fit checks, avoiding false CPU-only,
99%-offload, and `0 tok/s` recommendations.
## [0.5.3] - 2026-05-17
### Added
- Linux Intel integrated GPU detection via `/sys/class/drm`, so Intel iGPU
systems are no longer always treated as CPU-only.
- NVIDIA `nvidia-smi` fallback detection when pynvml is missing, NVML init
fails, or NVML reports no devices.
- Apple-prefixed Apple Silicon simulator aliases, so `--gpu "Apple M3 Max"`
resolves the same way as `--gpu "M3 Max"`.
### Fixed
- `whichllm run` transformers chat generation now passes tokenizer mappings
into `model.generate(**inputs)`, fixing the `KeyError: 'shape'` crash path.
- RTX 5060 Ti bandwidth lookup now reports 448 GB/s instead of `N/A`.
### Changed
- README install guidance now prefers `uvx` / `uv tool install`.
- Removed the old marketing note from the repository and added sponsor metadata.
## [0.5.2] - 2026-05-15
### Added
- Curated vision-language benchmark source (`benchmark_sources/vision.py`):
a 0-100 multimodal capability index (MMMU-Pro / MMBench / general
multimodal, 2026-05) covering the Qwen3-VL / Qwen2.5-VL / Qwen2-VL /
Llama-Vision / Phi-vision / Gemma-3 / Pixtral / InternVL3 lines.
- Benchmark snapshot date is now shown under every ranking so a stale
recommendation is self-evident instead of silently trusted.
- Round 3 regression suite (`tests/test_r3_regressions.py`, 20 tests),
each verified to fail when its fix is reverted.
### Fixed
- `--profile vision` generation inversion: text leaderboards do not
score VLMs, so the only model with a direct hit was a
two-generations-old Qwen2-VL-7B, which outranked the current
Qwen3-VL-32B even on an 80 GB GPU. Vision models now score from the
curated multimodal index (Qwen3-VL-32B leads at 73-76).
- Apple Silicon partial-offload speed was estimated ~3x too low: the
flat 0.45x PCIe penalty was applied to unified memory, where spilled
weights stay in the same high-bandwidth pool. DeepSeek-R1-class
models on M2/M3 Ultra now report a realistic 4-15 t/s instead of
~1.7. Discrete GPUs keep the 0.45x penalty.
- Duplicate `Qwen/Qwen3-Coder-30B-A3B-Instruct` key in the LiveBench
fallback (silently scored 62 instead of the intended 58, and broke
CI lint via ruff F601).
- `ruff format` / `ruff check` are now clean across the codebase, so
the Lint CI job passes (it was red for the entire 0.5.1 release).
### CI
- GitHub Actions updated to the Node 24 runtime (`checkout@v5`,
`setup-python@v6`); the Node 20 actions are deprecated from 2026-06.
## [0.5.1] - 2026-05-14
### Added
- `whichllm upgrade` subcommand: side-by-side comparison of the current
machine against potential GPU upgrades, with a verdict (worth it /
meaningful / marginal / flat / downgrade).
- Apple Silicon support in `--gpu` flag (M1-M4 base / Pro / Max / Ultra)
so simulator runs no longer fuzzy-match to ATI Rage Mobility-M1 and
emit spurious AMD ROCm warnings.
- Curated LiveBench, Arena AA, and Aider benchmark source modules with
frozen 2026-Q2 fallbacks for offline operation.
- Curated entries for reasoning / thinking lines: `Qwen/QwQ-32B`,
`Qwen3-4B-Thinking-2507`, `DeepSeek-R1-Distill-Qwen-32B/14B` and
`Llama-8B`.
- Frontier-model surfacing for 2026-Q2 releases that do not auto-surface
via cardinality (Kimi-K2, MiMo, DeepSeek-V4, GLM-5, Qwen3.6/Next,
gpt-oss, Llama-4, Mistral Small/Large, Devstral, Codestral, MiniMax,
Granite 3.3/4.0, Olmo-3, Nemotron-3).
- VRAM-aware auto floor for `--profile general` so tiny GPUs surface
full-GPU 3-4B picks instead of partial-offload-only 7B+.
### Changed
- VRAM estimation: KV cache scaled to 3.5 MB / billion-param / Kctx (was
0.5 MB) so 128K contexts are realistic; MoE KV uses active*4 to model
attention head sharing; activation overhead refined.
- Speed estimation: per-quant efficiency table, per-backend multiplier
(CUDA 1.0, Apple 0.82, AMD 0.78, Intel 0.65), MoE active-ratio floor,
partial-offload penalty.
- Ranking: composite family selection key replaces tier dominance;
size_score cap 20 → 35; MoE size_score uses total params;
`_knowledge_capacity_b` so `--min-params` no longer hides
Qwen3-Next-80B-A3B on its 3B active.
- Benchmark merging splits frozen (OLLB v2, Arena ELO) from current
(AA, LiveBench, Aider) with separate caps and lineage-aware recency
demotion so stale 2024-era leaderboards stop over-rewarding older
generations.
- httpx `AsyncClient` uses `follow_redirects=True` so case-mismatch HF
URLs (307) no longer silently drop frontier IDs.
### Fixed
- Reject benchmark inheritance when actual params differ by more than
2x from the family's dominant member, catching draft/MTP/abliterated
forks that share a `family_id` with their much larger base
(e.g. a 6.6B "imatrix-aligned" inheriting from a 158B base).
- Family grouping prefers the upstream-referenced model as the family
base instead of the highest-downloads member, so a popular fork no
longer overrides the official base for `family_id` assignment.
- MoE active-parameter registry corrected (gpt-oss-20b 3.6B,
gpt-oss-120b 5.1B, MiniMax-M2 10B).
- Quality floor (≥ 20) and speed floor (≥ 1.5 t/s) drop junk Q1_0 /
Bonsai-class attack vectors.
- 11 non-existent HF IDs removed from curated fallbacks (Kimi K2.5/K2.6,
GLM-5-Turbo, OLMo-3-32B, Llama-3.2-8B, Codestral-25.08, Mistral-Large-3
etc.).
## [0.4.0] - 2026-03-09
### Added
- `whichllm plan` subcommand — reverse lookup to find what GPU you need for a model
- Ollama integration examples and shell alias
- Homebrew formula for `brew install whichllm`
- VHS tape file for recording CLI demo GIF
- GitHub Actions CI/CD (tests, lint, PyPI publish)
- CONTRIBUTING.md, CODE_OF_CONDUCT.md
- Issue and PR templates
- PyPI metadata (classifiers, keywords, URLs)
## [0.3.0] - 2026-03-09
### Added
- Evidence filtering options (`--evidence`, `--direct`) in CLI and ranking logic
- A100/H100 80GB aliases to GPU simulator
- Eval benchmark integration with confidence-based score dampening
- BenchmarkEvidence with confidence-aware size interpolation
- HuggingFace evalResults as supplementary benchmark source
## [0.2.2]
### Added
- `--version` option to display package version
### Changed
- Updated demo image asset
## [0.2.1]
### Added
- Vision model support based on task profile (`--profile vision`)
## [0.2.0]
### Added
- `--status` flag to show Speed/Fit columns in output
- Published date and download count columns in display
- `published_at` backfill for ranking display
- GGUF-only backend filtering for model ranking
- Task profile support (`--profile`) for general, coding, vision, math
- GPU simulation (`--gpu`, `--vram`) for testing different hardware
- JSON output mode (`--json`)
- Rich table output with color-coded scores
- GPU detection for NVIDIA, AMD, and Apple Silicon
- HuggingFace API integration for model fetching
- Quantization-aware VRAM calculation
- Cache system with TTL (6h models, 24h benchmarks)
## [0.1.0]
### Added
- Initial release
- Basic hardware detection
- Simple model ranking with Typer CLI
CODE_OF_CONDUCT.md
+39
View File
@@ -0,0 +1,39 @@
# Code of Conduct
## Our Pledge
We are committed to making participation in this project a welcoming experience for everyone.
## Our Standards
Examples of positive behavior:
- Using welcoming and inclusive language
- Being respectful of differing viewpoints
- Gracefully accepting constructive feedback
- Focusing on what is best for the community
- Owning your contributions, including code written with help from AI tools
Examples of unacceptable behavior:
- Trolling, insulting, or derogatory comments
- Personal or political attacks
- Publishing others' private information without permission
## AI-assisted Contributions
AI tools are welcome here. The rule is simple: make it work, understand what it
does, and own the result.
If you submit code, docs, tests, or examples, you are responsible for them
whether you typed every line yourself or asked a tool to help. Review the work,
test the parts that matter, and respond to feedback or bugs as you would for
hand-written work.
## Enforcement
Instances of unacceptable behavior may be reported by opening a GitHub issue or contacting the maintainer. All reports will be reviewed and investigated.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org/), version 2.1.
CONTRIBUTING.md
+55
View File
@@ -0,0 +1,55 @@
# Contributing to whichllm
Thanks for your interest in contributing! Here's how you can help.
## Development Setup
```bash
git clone https://github.com/Andyyyy64/whichllm.git
cd whichllm
uv sync --dev
```
## Running Tests
```bash
uv run pytest
```
## How to Contribute
### Bug Reports
Open an issue with:
- Your hardware (GPU model, VRAM, OS)
- Python version
- Steps to reproduce
- Expected vs actual behavior
### Feature Requests
Open an issue describing the feature and why it would be useful.
### Pull Requests
1. Fork the repository
2. Create a feature branch (`git checkout -b feature/your-feature`)
3. Make your changes
4. Run tests (`uv run pytest`)
5. Submit a PR
### AI-assisted Contributions
AI-assisted code is welcome. Use the tools that help you move faster. Working
code wins.
The bar is practical: make it work, make it fit the project, and make it
reviewable. If you wrote it or asked a tool to write it, you own it. Read it,
test the parts that matter, and be ready to explain or fix it.
### Adding GPU Support
To add a new GPU to the bandwidth database, edit `src/whichllm/constants.py` and add the GPU specs.
## Code Style
- Follow existing code conventions
- Use type hints
- Add tests for new functionality
## License
By contributing, you agree that your contributions will be licensed under the MIT License.
LICENSE
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Andyyyy64
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
README.md
+513
View File
@@ -0,0 +1,513 @@
# whichllm
[![PyPI version](https://img.shields.io/pypi/v/whichllm)](https://pypi.org/project/whichllm/)
[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Tests](https://github.com/Andyyyy64/whichllm/actions/workflows/test.yml/badge.svg)](https://github.com/Andyyyy64/whichllm/actions/workflows/test.yml)
[![Sponsor](https://img.shields.io/badge/Sponsor-GitHub%20Sponsors-EA4AAA?logo=githubsponsors)](https://github.com/sponsors/Andyyyy64)
<p align="center">
<a href="https://trendshift.io/repositories/30336" target="_blank"><img src="https://trendshift.io/api/badge/repositories/30336" alt="Andyyyy64%2Fwhichllm | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>
**Find the best local LLM that actually runs on your hardware.**
Auto-detects your GPU/CPU/RAM and ranks the top models from HuggingFace that fit your system.
[日本語版はこちら](docs/README.ja.md)
## Quick start
Run the recommendation command once, with no project setup.
```bash
uvx whichllm@latest
```
Simulate a GPU before you buy hardware.
```bash
uvx whichllm@latest --gpu "RTX 4090"
```
Install it when you use it often.
```bash
uv tool install whichllm
uv tool upgrade whichllm # update an existing install
```
Other install paths.
```bash
brew install andyyyy64/whichllm/whichllm
pip install whichllm
```
## Want a safer pick?
By default, whichllm is ambitious. It ranks the best model that looks runnable
on your machine, including partial RAM offload and near-edge VRAM fits when
they seem usable.
If you want a more comfortable LM Studio-style recommendation, start with:
```bash
uvx whichllm@latest --gpu-only --speed usable --vram-headroom 1GB
```
This keeps only models that fit fully in GPU VRAM, filters out slow estimates,
and leaves extra VRAM for runtime overhead.
If LM Studio still says the model is slightly too large, increase the headroom:
```bash
uvx whichllm@latest --gpu-only --speed usable --vram-headroom 1.5GB
```
## Common workflows
After install, run `whichllm` directly. For one-off runs, replace `whichllm`
with `uvx whichllm@latest`.
```bash
# Best models for this machine
whichllm
# Pretend you have a specific GPU
whichllm --gpu "RTX 4090"
# Override detected iGPU/unified-memory limits
whichllm --vram 8 --ram-bandwidth 68
# Only show models that fit fully in GPU VRAM
whichllm --gpu-only
whichllm --fit gpu
# Simulate a multi-GPU workstation
whichllm --gpu "2x RTX 4090"
# Hide models that are technically runnable but too slow
whichllm --speed usable
whichllm --speed fast
# Pasteable GitHub / Slack / Discord output
whichllm --markdown
# Compare upgrade candidates
whichllm upgrade "RTX 4090" "RTX 5090" "H100"
# Find the GPU needed for a model
whichllm plan "llama 3 70b"
# Start a chat with a model
whichllm run "qwen 2.5 1.5b gguf"
# Print copy-paste Python
whichllm snippet "qwen 7b"
# Return JSON for scripts
whichllm --top 1 --json
```
![demo](assets/demo.gif)
## See it
```text
$ whichllm --gpu "RTX 4090"
#1 Qwen/Qwen3.6-27B 27.8B Q5_K_M score 92.8 27 t/s
#2 Qwen/Qwen3-32B 32.0B Q4_K_M score 83.0 31 t/s
#3 Qwen/Qwen3-30B-A3B 30.0B Q5_K_M score 82.7 102 t/s
```
The 32B model **fits your card fine** — whichllm still ranks the 27B #1,
because it scores higher on real benchmarks and is a newer generation.
A size-only "what fits?" tool would hand you the bigger one. That gap is
the whole point of whichllm. (Note #3: a MoE model at 102 t/s — speed is
ranked on *active* params, quality on *total*.)
## What can I run?
Real top picks (snapshot 2026-05 — your results track **live** HuggingFace
data, this is not a static list):
| Hardware | VRAM | Top pick | Speed |
|---|---|---|---|
| RTX 5090 | 32 GB | `Qwen3.6-27B` · Q6_K · score 94.7 | ~40 t/s |
| RTX 4090 / 3090 | 24 GB | `Qwen3.6-27B` · Q5_K_M · score 92.8 | ~27 t/s |
| RTX 4060 | 8 GB | `Qwen3-14B` · Q3_K_M · score 71.0 | ~22 t/s |
| Apple M3 Max | 36 GB | `Qwen3.6-27B` · Q5_K_M · score 89.4 | ~9 t/s |
| CPU only | — | `gpt-oss-20b` (MoE) · Q4_K_M · score 45.2 | ~6 t/s |
`whichllm --gpu "<your card>"` simulates any of these before you buy.
By default, rankings include full-GPU, partial-offload, and CPU-only
candidates when they are usable. Use `--gpu-only` or `--fit full-gpu` when
you only want models that fit entirely in GPU VRAM.
The default table shows memory, estimated generation speed, fit type, and
published date. Speed is colored by practical usability: under 4 tok/s is red,
4-10 is yellow, 10-30 is green, and 30+ is bright green. `~` / `?` still mark
estimate confidence.
## Why whichllm?
Fitting a model into your VRAM is the easy part. The hard part is knowing
**which of the models that fit is actually the best** — and that is what
whichllm is built to get right.
- **Evidence-based ranking, not a size heuristic** — The top pick is
chosen from merged real benchmarks (LiveBench, Artificial Analysis,
Aider, multimodal/vision, Chatbot Arena ELO, Open LLM Leaderboard) —
never "the biggest model that happens to fit."
- **Recency-aware** — Stale leaderboards are demoted along each model's
lineage, so a 2024 model can't outrank a current-generation one on an
outdated score. The benchmark snapshot date is printed under every
ranking, so a stale recommendation is self-evident instead of silently
trusted.
- **Evidence-graded and guarded** — Every score is tagged
`direct` / `variant` / `base` / `interpolated` / `self-reported` and
discounted by confidence. Fabricated uploader claims and cross-family
inheritance (a small fork borrowing its much larger base's score) are
actively rejected.
- **Architecture-aware estimates** — VRAM = weights + GQA KV cache +
activation + overhead; speed is bandwidth-bound with per-quant
efficiency, per-backend factors, MoE active-vs-total split, and
unified-memory vs discrete-PCIe partial-offload modeling.
- **One command, scriptable** — `whichllm` prints the answer; add
`--json | jq` for pipelines. No TUI, no keybindings to memorize.
- **Live data** — Models fetched directly from the HuggingFace API, with
curated frozen fallbacks for offline or rate-limited use.
## Features
- **Auto-detect hardware** — NVIDIA, AMD, Intel, Apple Silicon, CPU-only
- **Smart ranking** — Scores models by VRAM fit, speed, and benchmark quality
- **One-command chat** — `whichllm run` downloads and starts a chat session instantly
- **Code snippets** — `whichllm snippet` prints ready-to-run Python for any model
- **Live data** — Fetches models directly from HuggingFace (cached for performance)
- **Benchmark-aware** — Integrates real eval scores with confidence-based dampening
- **Task profiles** — Filter by general, coding, vision, or math use cases
- **GPU simulation** — Test with any GPU: `whichllm --gpu "RTX 4090"`
- **Multi-GPU simulation** — Repeat `--gpu`, use commas, or write `2x RTX 4090`
- **Full-GPU filter** — `--gpu-only` / `--fit full-gpu` hides offload candidates
- **Speed-aware filtering** — `--speed usable|fast` hides slow rows by threshold
- **Markdown output** — `--markdown` / `-m` prints pasteable GFM tables
- **Runtime memory budgets** — `--vram-headroom` and `--ram-budget` avoid edge fits
- **Hardware planning** — Reverse lookup: `whichllm plan "llama 3 70b"`
- **Upgrade planning** — Compare your current machine with candidate GPUs
- **JSON output** — Pipe-friendly: `whichllm --json`
## Run & Snippet
Try any model with a single command. No manual installs needed — whichllm
creates an isolated environment via `uv`, installs dependencies, downloads the
model, and starts an interactive chat.
![run demo](assets/demo-run.gif)
```bash
# Chat with a model (auto-picks the best GGUF variant)
whichllm run "qwen 2.5 1.5b gguf"
# Auto-pick the best model for your hardware and chat
whichllm run
# CPU-only mode
whichllm run "phi 3 mini gguf" --cpu-only
```
Works with **all model formats**:
- **GGUF** — via `llama-cpp-python` (lightweight, fast)
- **AWQ / GPTQ** — via `transformers` + `autoawq` / `auto-gptq`
- **FP16 / BF16** — via `transformers`
Get a **copy-paste Python snippet** instead:
```bash
whichllm snippet "qwen 7b"
```
```python
from llama_cpp import Llama
llm = Llama.from_pretrained(
repo_id="Qwen/Qwen2.5-7B-Instruct-GGUF",
filename="qwen2.5-7b-instruct-q4_k_m.gguf",
n_ctx=4096,
n_gpu_layers=-1,
verbose=False,
)
output = llm.create_chat_completion(
messages=[{"role": "user", "content": "Hello!"}],
)
print(output["choices"][0]["message"]["content"])
```
## Usage
```bash
# Auto-detect hardware and show best models
whichllm
# Simulate a GPU (e.g. planning a purchase)
whichllm --gpu "RTX 4090"
whichllm --gpu "RTX 5090"
# Specify variant
whichllm --gpu "RTX 5060 16"
# Override detected iGPU/unified-memory limits
whichllm --vram 8 --ram-bandwidth 68
# Simulate multiple GPUs
whichllm --gpu "2x RTX 4090"
whichllm --gpu "RTX 4090" --gpu "RTX 3090"
whichllm --gpu "RTX 4090, RTX 3090"
# Only show models that fit entirely in GPU VRAM
whichllm --gpu-only
whichllm --fit gpu
whichllm --fit full-gpu
# Avoid edge fits and background-RAM surprises
whichllm --vram-headroom 1.5GB
whichllm --ram-budget available
whichllm --ram-budget 8GB
# CPU-only mode
whichllm --cpu-only
# More results / filters
whichllm --top 20
whichllm --details # show Downloads metadata instead of runtime columns
whichllm --speed usable # minimum 10 tok/s
whichllm --speed fast # minimum 30 tok/s
whichllm --min-speed 4 # exact tok/s floor
whichllm --markdown # pasteable GitHub-Flavored Markdown table
whichllm --profile coding
whichllm --context-length 64k
whichllm --quant Q4_K_M
whichllm --min-speed 30 # exact tok/s floor
whichllm --evidence base # allow id/base-model matches
whichllm --evidence strict # id-exact only (same as --direct)
whichllm --direct
# JSON output
whichllm --json
# Force refresh (ignore cache)
whichllm --refresh
# Show hardware info only
whichllm hardware
# Plan: what GPU do I need for a specific model?
whichllm plan "llama 3 70b"
whichllm plan "Qwen2.5-72B" --quant Q8_0
whichllm plan "mistral 7b" --context-length 32768
# Upgrade: compare your current machine against candidate GPUs
whichllm upgrade "RTX 4090" "RTX 5090" "H100"
whichllm upgrade "Apple M4 Max" --top 5
# Run: download and chat with a model instantly
whichllm run "qwen 2.5 1.5b gguf"
whichllm run # auto-pick best for your hardware
# Snippet: print ready-to-run Python code
whichllm snippet "qwen 7b"
whichllm snippet "llama 3 8b gguf" --quant Q5_K_M
```
Markdown output is intended for GitHub issues, READMEs, Slack, Discord, and
blog posts:
```bash
whichllm --markdown
whichllm -m --top 5 --gpu "RTX 4090"
```
JSON model rows include `fit_type`, `vram_required_bytes`,
`vram_available_bytes`, `uses_multi_gpu`, `multi_gpu_effective_vram_bytes`,
`estimated_tok_per_sec`, `speed_confidence`, `speed_range_tok_per_sec`,
`speed_notes`, `benchmark_source`, and `benchmark_confidence`. The speed range
is a planning range, not a live benchmark.
## Integrations
### Ollama
Use JSON output to feed scripts that map HuggingFace IDs to your local Ollama
model names:
```bash
# Pick the top HuggingFace model ID
whichllm --top 1 --json | jq -r '.models[0].model_id'
# Find the best coding model ID
whichllm --profile coding --top 1 --json | jq -r '.models[0].model_id'
```
Ollama model names do not always match HuggingFace repo IDs, so a small mapping
step is usually needed before `ollama run`.
### Shell alias
Add to your `.bashrc` / `.zshrc`:
```bash
alias bestllm='whichllm --top 1 --json | jq -r ".models[0].model_id"'
# Usage: ollama run $(bestllm)
```
## Scoring
Each model gets a 0-100 score. Benchmark quality and size form the core;
evidence confidence and runtime fit then scale it, with speed, source
trust, and popularity as adjustments.
| Factor | Effect | Description |
|--------|--------|-------------|
| Benchmark quality | core | Merged LiveBench / Artificial Analysis / Aider / Vision / Arena ELO / Open LLM Leaderboard, weighted by source confidence |
| Model size | up to 35 | `log2`-scaled world-knowledge proxy (MoE uses total params) |
| Quantization | × penalty | Lower-bit quants discounted multiplicatively |
| Evidence confidence | ×0.55–1.0 | none / self-reported ×0.55, inherited ×0.78, direct full |
| Runtime fit | ×0.50–1.0 | partial-offload ×0.72, CPU-only ×0.50 |
| Speed | -8 to +8 | Usability gate vs a fit-dependent tok/s floor; reported with confidence and range metadata |
| Source trust | -5 to +5 | Official-org bonus, known-repackager penalty |
| Popularity | tie-breaker | Downloads/likes; weight shrinks as evidence strengthens |
Score markers:
- **`~`** (yellow) — No direct benchmark; score inherited/interpolated from the model family
- **`!sr`** (bright yellow) — Uploader-reported benchmark only, not independently verified
- **`?`** (red) — No benchmark data available
Speed display:
- **red** — Slow generation speed (`<4 tok/s`)
- **yellow** — Marginal generation speed (`4-10 tok/s`)
- **green** — Usable generation speed (`10-30 tok/s`)
- **bright green** — Fast local generation speed (`>=30 tok/s`)
- **`~`** (yellow) — Estimated tok/s range is available
- **`?`** (red) — Low-confidence speed estimate; backend/runtime sensitivity is high
## Documentation
- [CLI reference](docs/cli.md)
- [How it works](docs/how-it-works.md)
- [Scoring](docs/scoring.md)
- [Hardware detection and simulation](docs/hardware.md)
- [Run and snippet](docs/run-snippet.md)
- [Troubleshooting](docs/troubleshooting.md)
## How it works
### Data pipeline
1. **Model fetching** — Fetches popular models from HuggingFace API:
- Text-generation (downloads + recently updated)
- GGUF-filtered (separate query for coverage)
- Vision models (`image-text-to-text`) when `--profile vision` or `any`
2. **Benchmark sources** — *Current tier* (LiveBench, Artificial Analysis
Index, Aider) merged live when reachable, plus a curated multimodal /
vision index; *frozen tier* (Open LLM Leaderboard v2, Chatbot Arena
ELO). Tiers have separate caps and lineage-aware recency demotion so
stale leaderboards stop over-rewarding older generations.
3. **Benchmark evidence** — Five resolution levels, increasingly discounted:
- `direct` — Exact model ID match
- `variant` — Suffix-stripped or -Instruct variant
- `base_model` — Base model from cardData
- `line_interp` — Size-aware interpolation within model family
- `self_reported` — Uploader-claimed eval (heavily discounted)
Inheritance is rejected when a model's params diverge more than 2× from
its family's dominant member, catching draft / MTP / abliterated forks
that share a `family_id` with a much larger base.
4. **Cache** — normally `~/.cache/whichllm/`, or `$XDG_CACHE_HOME/whichllm/`
when `XDG_CACHE_HOME` is set to an absolute path:
- `models.json` — 6h TTL
- `benchmark.json` — 24h TTL
### Ranking engine
1. **Hardware detection** — NVIDIA (nvidia-ml-py), AMD (ROCm/dbgpu), Intel, Apple Silicon (Metal), CPU cores, RAM, disk
2. **VRAM estimation** — Weights + KV cache + activation + framework overhead (~500MB)
3. **Compatibility** — Full GPU / Partial Offload / CPU-only; compute capability and OS checks
4. **Speed** — tok/s from GPU memory bandwidth, quantization, backend, fit type, and MoE active parameters
5. **Scoring** — Benchmark (with confidence dampening), size, quantization penalty, fit type, speed, popularity, source trust (official vs repackager)
6. **Backend filter** — Apple Silicon and CPU-only restrict to GGUF for stability; Linux+NVIDIA allows AWQ/GPTQ
### Project structure
```
src/whichllm/
├── cli.py # Typer CLI: main, plan, run, snippet, hardware
├── constants.py # Backward-compatible exports for registry data
├── data/ # GPU, quantization, framework, and lineage registries
├── hardware/
│ ├── detector.py # Orchestrates GPU/CPU/RAM detection
│ ├── nvidia.py # NVIDIA GPU via nvidia-ml-py
│ ├── amd.py # AMD GPU (Linux)
│ ├── apple.py # Apple Silicon (Metal)
│ ├── cpu.py # CPU name, cores, AVX support
│ ├── memory.py # RAM and disk free
│ ├── gpu_simulator.py # --gpu flag: synthetic GPU from name
│ └── types.py # GPUInfo, HardwareInfo
├── models/
│ ├── fetcher.py # HuggingFace API, model parsing, evalResults
│ ├── benchmark.py # Arena ELO, Leaderboard (parquet/rows API)
│ ├── grouper.py # Family grouping by base_model and name
│ ├── cache.py # JSON cache with TTL
│ └── types.py # ModelInfo, GGUFVariant, ModelFamily
├── engine/
│ ├── vram.py # VRAM = weights + KV cache + activation + overhead
│ ├── compatibility.py# Fit type, disk check, compute/OS warnings
│ ├── performance.py # tok/s from bandwidth
│ ├── quantization.py # Bytes per weight, quality penalty, non-GGUF inference
│ ├── ranker.py # Scoring, evidence filter, profile/match
│ └── types.py # CompatibilityResult
└── output/
├── ranking.py # Rich hardware and recommendation tables
├── json_output.py # Ranking, plan, and upgrade JSON
├── plan.py # plan command display
├── upgrade.py # upgrade comparison display
└── display.py # Compatibility re-export shim
```
## Development
```bash
git clone https://github.com/Andyyyy64/whichllm.git
cd whichllm
uv sync --dev
uv run whichllm
uv run pytest
```
## Contributing
Contributions are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
## Support
If whichllm helped you find a model or avoid a bad hardware guess,
sponsoring is appreciated. It helps keep the project maintained: hardware
reports, packaging, test fixtures, benchmark updates, and support for more
machines.
whichllm will stay open-source either way. Issues and PRs are always welcome.
Useful? A GitHub star helps other people find it, and I'd genuinely like to
know what it picked for your rig. Drop it in [Issues](https://github.com/Andyyyy64/whichllm/issues).
## Star History
[![Star History Chart](https://api.star-history.com/svg?repos=Andyyyy64/whichllm&type=Date)](https://www.star-history.com/#Andyyyy64/whichllm&Date)
## Requirements
- Python 3.11+
- NVIDIA GPU detection via `nvidia-ml-py` (included by default)
- AMD / Apple Silicon detected automatically
## License
MIT
README.wehub.md
+7
View File
@@ -0,0 +1,7 @@
# WeHub 来源说明
- 原始项目：`Andyyyy64/whichllm`
- 原始仓库：https://github.com/Andyyyy64/whichllm
- 导入方式：上游默认分支的最新快照
- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
- 本文件仅用于记录来源，不代表 WeHub 是原项目作者
assets/demo-run.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 440 KiB

assets/demo-run.tape
+45
View File
@@ -0,0 +1,45 @@
# VHS demo recording for whichllm run/snippet
# Install: https://github.com/charmbracelet/vhs
# Run: cd assets && vhs demo-run.tape
# Output: demo-run.gif
Output demo-run.gif
Set FontSize 15
Set Width 1200
Set Height 800
Set Theme "Catppuccin Mocha"
Set Padding 20
Set TypingSpeed 50ms
# Warm up cache
Hide
Type "whichllm --top 1 > /dev/null 2>&1"
Enter
Sleep 10s
Type "clear"
Enter
Sleep 500ms
Show
# Scene 1: snippet — show the code
Type "whichllm snippet 'qwen 2.5 1.5b gguf'"
Sleep 500ms
Enter
Sleep 4s
# Scene 2: run — download and chat
Type "whichllm run 'qwen 2.5 1.5b gguf'"
Sleep 500ms
Enter
Sleep 12s
# Chat
Type "Explain quantum computing in one sentence."
Enter
Sleep 8s
Sleep 2s
Type "exit"
Enter
Sleep 2s
assets/demo.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 363 KiB

assets/demo.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 127 KiB

assets/demo.tape
+37
View File
@@ -0,0 +1,37 @@
# VHS demo recording for whichllm
# Install: https://github.com/charmbracelet/vhs
# Run: cd assets && vhs demo.tape
# Output: demo.gif
Output demo.gif
Set FontSize 15
Set Width 1200
Set Height 1100
Set Theme "Catppuccin Mocha"
Set Padding 20
Set TypingSpeed 50ms
# Warm up cache invisibly
Hide
Type "uvx whichllm@latest --top 1 > /dev/null 2>&1"
Enter
Sleep 10s
Type "clear"
Enter
Sleep 500ms
Show
# Scene 1: Just run it
Type "uvx whichllm@latest"
Sleep 500ms
Enter
Sleep 6s
Sleep 2s
# Scene 2: What GPU do I need?
Type "uvx whichllm@latest plan 'qwen 72b'"
Sleep 500ms
Enter
Sleep 6s
Sleep 3s
docs/README.ja.md
+249
View File
@@ -0,0 +1,249 @@
# whichllm
**手元のハードウェアで実際に動くローカルLLMを探すCLIです。**
whichllm は GPU / CPU / RAM / ディスクを検出し、HuggingFace 上のモデルを
取得して、実行できる候補をランキングします。単に「VRAMに入る最大モデル」を
選ぶのではなく、ベンチマーク、量子化、速度、実行形態、モデル世代をまとめて
評価します。
[English version](../README.md)
![demo](../assets/demo.gif)
## インストール
### uv
一度だけ試す場合:
```bash
uvx whichllm@latest
```
継続して使う場合:
```bash
uv tool install whichllm
uv tool upgrade whichllm # 既存インストールを更新
```
### Homebrew
```bash
brew install andyyyy64/whichllm/whichllm
```
### pip
```bash
pip install whichllm
```
## 余裕を持って動く候補だけ見たい場合
whichllm のデフォルトは少し攻めた推薦です。RAMへのpartial offloadや、
VRAMぎりぎりの候補も、動きそうならランキングに入れます。
LM Studioなどで余裕を持って動かしたい場合は、まずこれを使ってください。
```bash
uvx whichllm@latest --gpu-only --speed usable --vram-headroom 1GB
```
GPUのVRAMに全部載る候補だけに絞り、遅い推定速度の候補を外し、実行時の
余白も1GB残します。
それでもLM Studio側で少しはみ出す場合は、余白を増やします。
```bash
uvx whichllm@latest --gpu-only --speed usable --vram-headroom 1.5GB
```
### 開発用
```bash
git clone https://github.com/Andyyyy64/whichllm.git
cd whichllm
uv sync --dev
uv run whichllm
uv run pytest
```
## まず使う
```bash
# 自動検出しておすすめモデルを表示
whichllm
# GPUをシミュレートする
whichllm --gpu "RTX 4090"
whichllm --gpu "Apple M3 Max"
# 複数GPUをシミュレートする
whichllm --gpu "2x RTX 4090"
whichllm --gpu "RTX 4090" --gpu "RTX 3090"
# GPUのVRAMに全部載る候補だけを見る
whichllm --gpu-only
whichllm --fit gpu
# 速度の最低ラインを指定する
whichllm --speed usable
whichllm --speed fast
whichllm --min-speed 4
# GitHubやSlackに貼りやすいMarkdown表で出力する
whichllm --markdown
# 実行時のメモリ余白やRAM使用量を指定する
whichllm --vram-headroom 1.5GB
whichllm --ram-budget available
# CPUのみとして評価する
whichllm --cpu-only
# JSONで出力する
whichllm --json
```
JSONの各モデルには `estimated_tok_per_sec` に加えて、`fit_type`、
`vram_required_bytes`、`vram_available_bytes`、`uses_multi_gpu`、
`multi_gpu_effective_vram_bytes`、`speed_confidence`、
`speed_range_tok_per_sec`、`speed_notes`、`benchmark_source`、
`benchmark_confidence` が入ります。
速度は実測値ではなく、ハードウェア情報とモデル情報からの推定です。
通常の表には必要メモリ、推定生成速度、Fit種別、Published が表示されます。
Downloads まで見たい場合は `--details` を使います。
GitHub issue、README、Slack、Discord へ貼る場合は `--markdown` / `-m`
でMarkdown表として出力できます。
## 主なコマンド
```bash
# 推薦ランキング
whichllm --top 20
whichllm --quant Q4_K_M
whichllm --min-speed 30
whichllm --speed usable
whichllm --speed fast
whichllm --markdown
whichllm --profile coding
whichllm --context-length 64k
whichllm --details
whichllm --gpu-only
# ベンチ根拠の厳しさ
whichllm --evidence strict
whichllm --evidence base
whichllm --direct
# モデルから必要GPUを逆算
whichllm plan "llama 3 70b"
whichllm plan "Qwen2.5-72B" --quant Q8_0
whichllm plan "mistral 7b" --context-length 32768
# 今のマシンと購入候補GPUを比較
whichllm upgrade "RTX 4090" "RTX 5090" "H100"
# モデルをダウンロードしてチャット
whichllm run "qwen 2.5 1.5b gguf"
whichllm run
# 実行用Pythonコードを表示
whichllm snippet "qwen 7b"
# ハードウェア情報だけ表示
whichllm hardware
```
## スコアの見方
各モデルには 0 から 100 のスコアが付きます。中心になるのはベンチマークと
モデルサイズですが、実行時に遅すぎる候補や、CPUオフロードが大きい候補は
下がります。
| 要素 | 役割 |
| --- | --- |
| ベンチマーク | LiveBench、Artificial Analysis、Aider、Vision、Arena、Open LLM Leaderboard を統合 |
| モデルサイズ | 知識量の近似。MoEは総パラメータを使う |
| 量子化 | Q4 / Q5 / Q6 / Q8 などの品質低下を反映 |
| 実行形態 | Full GPU、Partial Offload、CPU-only を区別 |
| 速度 | tok/s が実用ラインを下回ると減点。表示時は推定の信頼度と幅も出す |
| 根拠の強さ | direct、base_model、variant、line_interp、self_reported を区別 |
| 世代補正 | 古い凍結ベンチだけで新世代を上回らないよう調整 |
スコア横のマーカー:
- `~`: 直接ベンチではなく、系列や派生から推定したスコア
- `!sr`: アップローダー自己申告の評価値だけに基づくスコア
- `?`: 利用できるベンチマーク根拠がないスコア
速度表示:
- 赤: `4 tok/s` 未満の遅い生成速度
- 黄: `4-10 tok/s` のぎりぎり使える生成速度
- 緑: `10-30 tok/s` の実用的な生成速度
- 明るい緑: `30 tok/s` 以上の高速なローカル生成速度
- `~`: 速度推定の幅がある通常の推定値
- `?`: backend や runtime の影響が大きい低信頼の推定値
## 仕組み
1. ハードウェアを検出します。NVIDIA、AMD、Intel、Apple Silicon、CPU、RAM、
ディスク空き容量を見ます。
2. HuggingFace APIからモデルを取得します。人気モデル、GGUF、最近更新された
GGUF、trending、重要な frontier モデルを組み合わせます。
3. ベンチマークを読み込みます。現在系の LiveBench / Artificial Analysis /
Aider / Vision と、凍結系の Arena / Open LLM Leaderboard を分けて扱います。
4. `base_model` とモデル名からファミリーを作り、同じモデルの派生やGGUFを束ねます。
5. 候補ごとに VRAM、互換性、速度、速度推定の信頼度、スコアを計算します。
6. ファミリーごとに最も良い候補を残して表示します。
通常は full GPU、partial offload、CPU-only の候補をまとめて見ます。GPUの
VRAMに全部載るモデルだけを見たい場合は `--gpu-only` か
`--fit gpu` を使います。遅い候補を最初から除外したい場合は
`--speed usable`、`--speed fast` を使います。
それぞれ `10 tok/s`、`30 tok/s` が最低ラインです。
もっと低いラインを指定したい場合は `--min-speed 4` のように数値で指定します。
キャッシュは通常 `~/.cache/whichllm/` に保存されます。`XDG_CACHE_HOME` が
絶対パスで設定されている場合は、その配下の `whichllm/` を使います。
- `models.json`: 6時間
- `benchmark.json`: 24時間
## プロジェクト構成
```text
src/whichllm/
├── cli.py # Typer CLI: main, plan, upgrade, run, snippet, hardware
├── constants.py # 互換用のregistry再export
├── data/ # GPU、量子化、framework、lineageのregistry
├── hardware/ # ハードウェア検出とGPUシミュレーション
├── models/ # HuggingFace取得、ベンチ、キャッシュ、グルーピング
├── engine/ # VRAM、互換性、速度、ランキング
└── output/ # Rich表示、JSON、plan/upgrade表示
```
## 詳細ドキュメント
- [CLIリファレンス](cli.md)
- [仕組み](how-it-works.md)
- [スコアリング](scoring.md)
- [ハードウェア検出とシミュレーション](hardware.md)
- [run と snippet](run-snippet.md)
- [トラブルシュート](troubleshooting.md)
## 動作環境
- Python 3.11+
- NVIDIA GPU検出は `nvidia-ml-py` と `nvidia-smi` fallback
- AMD GPU検出は Linux / ROCm / sysfs / lspci と Windows fallback
- Intel GPU検出は Linux / sysfs / lspci と Windows fallback
- Strix Halo、Ryzen AI MAX、Radeon 890M 系は shared memory APU として扱う
- Apple Silicon検出は macOS / `system_profiler`
## ライセンス
MIT
docs/cli.md
+270
View File
@@ -0,0 +1,270 @@
# CLI reference
This page describes the commands exposed by `whichllm`. It is based on the
Typer entrypoint in `src/whichllm/cli.py`.
## Main command
```bash
whichllm [OPTIONS]
```
Detects the current machine, loads model and benchmark data, ranks compatible
models, and prints a table.
Common options:
| Option | Meaning |
| --- | --- |
| `--top`, `-n` | Number of ranked models to show. Default: `10` |
| `--context-length`, `-c` | Context length used for KV cache estimation. Accepts integers or `k` shorthand such as `64k`. Default: `4096` |
| `--quant`, `-q` | Keep only a quantization type such as `Q4_K_M` |
| `--min-speed` | Keep only models above an exact tok/s estimate |
| `--speed` | Named speed floor: `any`, `usable` (`10 tok/s`), or `fast` (`30 tok/s`) |
| `--fit` | Runtime fit filter: `any`, `gpu`, or `full-gpu` |
| `--gpu-only` | Alias for `--fit full-gpu`; excludes partial offload and CPU-only candidates |
| `--profile` | Ranking profile: `general`, `coding`, `vision`, `math`, `any` |
| `--evidence` | Benchmark evidence filter: `strict`, `base`, `any` |
| `--direct` | Alias for `--evidence strict` |
| `--status` | Compatibility option. Runtime columns are now shown by default |
| `--details` | Show download metadata instead of runtime columns |
| `--min-params` | Minimum model knowledge capacity in billions of parameters |
| `--json` | Print machine-readable JSON |
| `--markdown`, `-m` | Print a pasteable GitHub-Flavored Markdown table |
| `--refresh` | Ignore caches and fetch models/benchmarks again |
| `--cpu-only` | Ignore GPUs and rank for CPU-only use |
| `--gpu` | Simulate GPU(s) by name. Accepts repeated flags, comma-separated values, and count shorthand |
| `--vram` | Override simulated GPU VRAM or detected GPU usable VRAM in GB |
| `--bandwidth`, `--ram-bandwidth` | Override GPU/RAM bandwidth in GB/s |
| `--gpu-index` | Detected GPU index to override when multiple GPUs are present |
| `--vram-headroom` | Reserve per-GPU memory for runtime overhead. Default: `auto`. Accepts `none`, byte values like `1.5GB`, or percentages like `10%` |
| `--ram-budget` | Cap RAM available for partial offload. Accepts `available`, byte values like `8GB`, or percentages like `50%` |
| `--version` | Print the installed package version |
Environment variables:
| Variable | Meaning |
| --- | --- |
| `HF_ENDPOINT` | Hugging Face endpoint root used for whichllm's own model metadata API calls. Example: `https://huggingface.co` or a compatible mirror root |
`--fit any` is the default. It can include full-GPU, partial-offload, and
CPU-only candidates when they are runnable. `--fit gpu`, `--fit full-gpu`, and
`--gpu-only` keep only rows whose `fit_type` is `full_gpu`.
The default table shows memory required, estimated generation speed, fit type,
and published date. Use `--details` when you want download counts instead.
Speed colors are absolute usability hints: red is under `4 tok/s`, yellow is
`4-10 tok/s`, green is `10-30 tok/s`, and bright green is `30+ tok/s`. The `~`
and `?` markers still refer to estimate confidence, not speed quality.
`--vram-headroom auto` subtracts a small budget from each GPU before fit
checks, so near-edge recommendations are less likely to overflow in tools such
as LM Studio. Use `--vram-headroom none` to restore the raw detected VRAM.
`--ram-budget available` caps offload planning to current available RAM.
For detected iGPU or unified-memory systems, use `--vram` and
`--bandwidth` / `--ram-bandwidth` to override the automatically detected
usable memory and bandwidth. If multiple GPUs are detected, add `--gpu-index`
with the GPU number from `whichllm hardware`.
Examples:
```bash
whichllm
whichllm --gpu "RTX 4090"
whichllm --gpu "RTX 5060 Ti" --vram 16
whichllm --vram 8 --ram-bandwidth 68
whichllm --gpu-index 1 --vram 8 --bandwidth 68
whichllm --gpu "2x RTX 4090"
whichllm --gpu "RTX 4090" --gpu "RTX 3090"
whichllm --gpu "RTX 4090, RTX 3090"
whichllm --profile coding --top 5
whichllm --context-length 64k
whichllm --gpu-only
whichllm --fit gpu
whichllm --speed usable
whichllm --speed fast
whichllm --min-speed 4
whichllm --markdown
whichllm --vram-headroom 1.5GB
whichllm --ram-budget available
whichllm --details
whichllm --evidence strict
whichllm --json | jq '.models[0]'
```
`--markdown` is mutually exclusive with `--json`. It prints a plain Markdown
table without the Rich hardware panel, colors, or box-drawing characters.
Ranking JSON model rows include:
| Field | Meaning |
| --- | --- |
| `fit_type` | Runtime fit classification: `full_gpu`, `partial_offload`, or `cpu_only` |
| `vram_required_bytes` | Estimated runtime memory requirement for the candidate |
| `vram_available_bytes` | GPU memory budget used for the fit check |
| `uses_multi_gpu` | Whether the fit check used more than one GPU |
| `multi_gpu_effective_vram_bytes` | Conservative effective VRAM budget for multi-GPU fits, when applicable |
| `estimated_tok_per_sec` | Point estimate used by ranking |
| `speed_confidence` | `high`, `medium`, or `low` |
| `speed_range_tok_per_sec` | Estimated lower/upper tok/s range, when available |
| `speed_notes` | Short reasons for the confidence level |
| `benchmark_status` | Display marker category for benchmark evidence |
| `benchmark_source` | How benchmark evidence was matched: `direct`, `variant`, `base_model`, `line_interp`, `self_reported`, or `none` |
| `benchmark_confidence` | Confidence in the benchmark match, `0.0`–`1.0` |
The top-level `hardware` object also includes `usable_vram_bytes` per GPU,
`ram_budget_bytes`, and `budget_notes` when memory budgets are active.
## `hardware`
```bash
whichllm hardware [OPTIONS]
```
Prints detected hardware without ranking models. The same simulation flags are
available here:
```bash
whichllm hardware
whichllm hardware --cpu-only
whichllm hardware --gpu "Apple M3 Max"
whichllm hardware --gpu "RTX 3060" --vram 12
whichllm hardware --vram 8 --bandwidth 68
whichllm hardware --gpu "4x RTX 4090"
```
## `plan`
```bash
whichllm plan MODEL_NAME [OPTIONS]
```
Searches for a model by HuggingFace repo ID or fuzzy terms, then estimates the
VRAM required for several quantization levels and common GPUs.
Options:
| Option | Meaning |
| --- | --- |
| `--context-length`, `-c` | Context length for the memory estimate. Accepts integers or `k` shorthand such as `128k`. Default: `4096` |
| `--quant`, `-q` | Target quantization. Default: `Q4_K_M` |
| `--json` | Print the plan as JSON |
| `--refresh` | Ignore model cache and fetch again |
Examples:
```bash
whichllm plan "llama 3 70b"
whichllm plan "Qwen2.5-72B" --quant Q8_0
whichllm plan "mistral 7b" --context-length 32768
whichllm plan "mistral 7b" --context-length 32k
```
## `upgrade`
```bash
whichllm upgrade TARGET_GPUS... [OPTIONS]
```
Compares the current machine against one or more simulated GPUs. The CPU, RAM,
disk, and OS come from the current machine; only the GPU changes.
Options:
| Option | Meaning |
| --- | --- |
| `--context-length`, `-c` | Context length used for ranking. Accepts integers or `k` shorthand such as `64k`. Default: `8192` |
| `--top`, `-n` | Best-N models to compare per GPU. Default: `3` |
| `--profile` | Ranking profile. Default: `general` |
| `--cpu-only` | Use CPU-only as the current baseline |
| `--json` | Print comparison JSON |
| `--refresh` | Ignore caches and fetch again |
Examples:
```bash
whichllm upgrade "RTX 4090" "RTX 5090" "H100"
whichllm upgrade "Apple M4 Max" --top 5
whichllm upgrade "RX 7900 XTX" --profile coding
whichllm upgrade "RTX 4090" --context-length 128k
```
## `run`
```bash
whichllm run [MODEL_NAME] [OPTIONS]
```
Creates a temporary Python script, launches it through `uv run --no-project`,
installs the needed inference packages into that isolated run, and starts an
interactive chat.
If `MODEL_NAME` is omitted, whichllm ranks models for the current hardware and
uses the top result.
Options:
| Option | Meaning |
| --- | --- |
| `--context-length`, `-c` | Context length for the generated chat script. Accepts integers or `k` shorthand such as `64k` |
| `--quant`, `-q` | Preferred GGUF quantization |
| `--refresh` | Ignore model cache and fetch again |
| `--cpu-only` | Force CPU-only execution in the generated script |
Examples:
```bash
whichllm run
whichllm run "qwen 2.5 1.5b gguf"
whichllm run "phi 3 mini gguf" --cpu-only
whichllm run "mistral 7b gguf" --context-length 64k
```
`run` requires `uv` in `PATH`.
## `snippet`
```bash
whichllm snippet [MODEL_NAME] [OPTIONS]
```
Prints a ready-to-run Python snippet for the selected model. GGUF models use
`llama-cpp-python`; non-GGUF models use `transformers`.
Options:
| Option | Meaning |
| --- | --- |
| `--quant`, `-q` | Preferred GGUF quantization |
| `--refresh` | Ignore model cache and fetch again |
Examples:
```bash
whichllm snippet "qwen 7b"
whichllm snippet "llama 3 8b gguf" --quant Q5_K_M
```
## Evidence filters
`--evidence` controls which benchmark matches are allowed into the ranking.
| Mode | Allows |
| --- | --- |
| `strict` | Exact independent benchmark matches only |
| `base` | Exact, variant, and `cardData.base_model` matches |
| `any` | All evidence levels, including line interpolation and self-reported values |
`--direct` is kept as a shorter alias for `--evidence strict`.
## Profiles
The ranker detects specialization from repository names.
| Profile | Behavior |
| --- | --- |
| `general` | Excludes coding, vision, and math-specialized names |
| `coding` | Keeps coding-specialized names |
| `vision` | Keeps vision or multimodal names and includes VLM candidates |
| `math` | Keeps math-specialized names |
| `any` | Keeps all recognized model types and includes VLM candidates |
docs/hardware.md
+237
View File
@@ -0,0 +1,237 @@
# Hardware detection and simulation
whichllm detects the current machine and can also simulate hardware for
purchase planning.
The source of truth is the `hardware/` package plus curated registry data in
`data/gpu.py`. `constants.py` remains as a compatibility export layer for older
imports.
## Detected data
The ranker receives a `HardwareInfo` object with:
- GPU list
- CPU name
- physical CPU cores
- AVX2 and AVX-512 support
- total RAM
- free disk space
- OS name
Each GPU is represented as `GPUInfo`:
- name
- vendor
- VRAM bytes
- usable VRAM bytes, when a runtime headroom is active
- NVIDIA compute capability, when known
- CUDA or ROCm version, when known
- memory bandwidth estimate
- whether the GPU uses shared memory
## NVIDIA
NVIDIA detection tries `nvidia-ml-py` first. If NVML is unavailable, fails to
initialize, or returns no devices, whichllm falls back to:
```bash
nvidia-smi --query-gpu=name,memory.total,clocks.max.memory --format=csv,noheader,nounits
```
If a driver rejects `clocks.max.memory`, whichllm retries the older
`name,memory.total` query.
For known cards, curated data and strict `dbgpu` lookups provide:
- memory bandwidth
- compute capability
The max memory clock is used when a marketing name covers multiple memory
types. For example, GTX 1650 GDDR5 and GDDR6 cards share the same broad driver
name, so whichllm uses the reported memory clock when available and falls back
to the conservative bandwidth when it is not.
DGX Spark / NVIDIA GB10 uses unified system memory. When the driver reports
`memory.total` as unavailable, whichllm treats GB10 as shared memory and uses
system RAM for fit checks.
Compute capability is used to warn when a card is below the minimum expected by
common local inference tools.
## AMD
On Linux, AMD detection tries `rocm-smi` first:
- product name
- VRAM
- ROCm driver version
If `rocm-smi` is unavailable, it falls back to `lspci` and then
`/sys/class/drm`.
On Windows, whichllm uses `Win32_VideoController` as a fallback for AMD GPUs.
When possible, it also reads the 64-bit dedicated-memory value from the
`Control\Video` registry path because `AdapterRAM` is a 32-bit field and can
cap larger cards around 4 GB.
AMD shared-memory APUs are treated differently from discrete GPUs. Names such
as Strix Halo, Ryzen AI MAX, Radeon 8050S, Radeon 8060S, Radeon 890M, and
Radeon 780M are modeled as shared-memory systems. If the reported VRAM is just
a small aperture, whichllm uses the system memory pool for fit checks instead
of treating it as a tiny discrete GPU.
## Intel
Intel integrated GPUs are detected on Linux through `lspci` or sysfs, and on
Windows through `Win32_VideoController`. They do not normally report dedicated
VRAM, so whichllm records them with `0` dedicated VRAM and labels them as
shared memory.
Discrete Intel Arc cards are kept as dedicated-memory GPUs when the device name
and memory report look like a discrete adapter.
The Intel backend factor is lower than NVIDIA, AMD, and Apple because local LLM
GPU inference support is less mature.
## Apple Silicon
On macOS, whichllm uses:
```bash
system_profiler SPHardwareDataType -json
```
Apple Silicon uses unified memory, so the detected chip memory is treated as
available GPU memory. Memory bandwidth is looked up by chip family when known.
Partial offload on Apple Silicon is not penalized like discrete PCIe offload.
Weights still live in unified memory, so the speed penalty is milder.
## CPU and memory
CPU detection reads:
- `/proc/cpuinfo` on Linux
- `sysctl` on macOS
- `wmic` on Windows, then PowerShell / CIM when `wmic` is unavailable or only
returns a header
Physical core count comes from `psutil`, with a Linux `/proc/cpuinfo` fallback.
RAM comes from `psutil.virtual_memory()`. Disk free space is checked under the
user's home directory by default.
## GPU simulation
Use `--gpu` to simulate a GPU:
```bash
whichllm --gpu "RTX 4090"
whichllm hardware --gpu "Apple M3 Max"
whichllm upgrade "RTX 4090" "RTX 5090" "H100"
```
Simulation uses the `dbgpu` package for a TechPowerUp-backed GPU database.
whichllm adds extra handling for common aliases and Apple Silicon chips because
those are not covered by dbgpu.
Use `--vram` when a GPU name is ambiguous, unknown, or has multiple variants:
```bash
whichllm --gpu "RTX 5060 Ti" --vram 16
whichllm hardware --gpu "Unknown GPU" --vram 24
```
For detected integrated or unified-memory GPUs, use `--vram` and
`--bandwidth` / `--ram-bandwidth` to override the automatically detected usable
capacity and memory bandwidth:
```bash
whichllm --vram 8 --ram-bandwidth 68
whichllm hardware --vram 8 --bandwidth 68
```
If multiple GPUs are detected, pass `--gpu-index` to choose the GPU shown by
`whichllm hardware`:
```bash
whichllm --gpu-index 1 --vram 8 --ram-bandwidth 68
```
By default, whichllm applies a small automatic VRAM headroom before fit checks.
This avoids recommending models that only fit on paper but overflow in runtimes
that need extra graph buffers or loader overhead. Tune it with:
```bash
whichllm --vram-headroom 1.5GB
whichllm --vram-headroom 10%
whichllm --vram-headroom none
```
Multi-GPU simulation accepts repeated flags, comma-separated values, and count
shorthand:
```bash
whichllm --gpu "2x RTX 4090"
whichllm --gpu "RTX 4090" --gpu "RTX 3090"
whichllm --gpu "RTX 4090, RTX 3090"
```
`--vram` is only supported for a single simulated GPU. For multi-GPU
simulation, use known GPU names so whichllm can resolve each card's VRAM from
the GPU database.
## Fit types
Compatibility checks classify a candidate into one of three fit types:
| Fit | Meaning |
| --- | --- |
| `full_gpu` | Required memory fits in available GPU memory |
| `partial_offload` | GPU plus usable system RAM can hold the model |
| `cpu_only` | Usable system RAM can hold the model without GPU |
If neither GPU memory nor usable RAM can hold the model, the candidate is not
ranked.
whichllm keeps a bounded system-RAM reserve for the OS and other processes.
Use `--ram-budget available` to cap partial-offload planning to the current
available RAM reported by the OS, or pass a fixed budget such as
`--ram-budget 8GB`.
## Multiple GPUs
For fit checks, whichllm uses a conservative multi-GPU budget rather than
pretending all VRAM is one perfect device. It starts from raw total VRAM, applies
a small per-GPU overhead, and then applies a utilization factor. Homogeneous
sets receive a less severe reduction than heterogeneous sets.
If a dedicated GPU is present, low-aperture shared-memory integrated GPUs are
not added to the fit pool. This avoids treating unrelated system RAM and
dedicated VRAM as one full-GPU target.
For speed estimates, whichllm uses the largest detected GPU as the
representative device and marks multi-GPU speed as low-confidence. This avoids
claiming ideal scaling when real performance depends on backend split mode,
PCIe/NVLink bandwidth, NCCL/RCCL support, batch size, and model architecture.
This is a practical fit approximation. It does not model every tensor-parallel
or pipeline-parallel runtime configuration.
## Disk checks
The compatibility check also compares estimated model weight size with free
disk space. If the model cannot be downloaded, it is marked unrunnable.
## Known limitations
- GPU bandwidth is a lookup or database estimate, not a live benchmark.
- Speed estimates are planning numbers. The default table and JSON fields such
as `speed_confidence` and `speed_range_tok_per_sec` show uncertainty.
- Driver, runtime, batch size, prompt length, and thermal limits can change real
performance.
- Multi-GPU runtime behavior depends on the inference backend and is only
approximated.
- Apple and shared-memory APU behavior is modeled as unified-memory style, but
real results still depend on OS pressure and memory bandwidth.
docs/how-it-works.md
+193
View File
@@ -0,0 +1,193 @@
# How it works
whichllm has one main job: start with the user's hardware, collect candidate
models, estimate what can run, and rank the results.
The implementation is intentionally split into small packages:
```text
src/whichllm/
├── cli.py
├── constants.py
├── data/
├── hardware/
├── models/
├── engine/
└── output/
```
## Request flow
The default `whichllm` command follows this path:
1. Validate CLI flags.
2. Detect hardware.
3. Load model cache or fetch from HuggingFace.
4. Load benchmark cache or fetch benchmark sources.
5. Group related model repos into families.
6. Flatten families back into rankable candidates.
7. Rank every candidate variant.
8. Backfill missing published dates for top results.
9. Print a Rich table or JSON.
The `plan`, `upgrade`, `run`, `snippet`, and `hardware` subcommands reuse parts
of the same pipeline.
## Hardware detection
`hardware/detector.py` orchestrates detection. Each detector is fail-safe and
returns an empty result on failure.
| Module | Role |
| --- | --- |
| `hardware/nvidia.py` | Uses `nvidia-ml-py`; falls back to `nvidia-smi`, including optional memory-clock data |
| `hardware/amd.py` | Uses `rocm-smi`; falls back to `lspci` and `/sys/class/drm` |
| `hardware/intel.py` | Detects Linux Intel iGPUs through `lspci` or sysfs |
| `hardware/windows.py` | Detects Windows AMD and Intel fallback GPUs through WMI and registry memory fields |
| `hardware/apple.py` | Uses `system_profiler` on macOS |
| `hardware/cpu.py` | Reads CPU name, physical cores, AVX2, and AVX-512 |
| `hardware/memory.py` | Reads RAM and disk free space |
| `hardware/gpu_simulator.py` | Builds synthetic GPUs for `--gpu` |
The result is a `HardwareInfo` dataclass. GPUs are represented by `GPUInfo`.
## Model fetching
`models/fetcher.py` reads the HuggingFace API and turns responses into
`ModelInfo` objects.
The fetcher combines several queries:
1. Popular `text-generation` models sorted by downloads.
2. Popular GGUF repos.
3. Recently modified GGUF repos.
4. Trending text-generation repos, with and without the GGUF filter.
5. A curated list of frontier and hard-to-find model IDs.
6. Vision candidates from `image-text-to-text` when the active profile needs
them.
For each repository, the parser extracts:
- repo ID and display name
- parameter count
- active parameter count for known or config-detected MoE models
- architecture and context length
- license, downloads, likes, published date
- GGUF variants and file sizes
- `cardData.base_model`
- conservative HuggingFace `evalResults` values
Parameter counts can come from safetensors metadata, GGUF metadata, config
estimation, name hints, or a curated fallback table for important models with
missing metadata.
## Caches
Both caches normally live under `~/.cache/whichllm/`. If `XDG_CACHE_HOME` is
set to an absolute path, whichllm uses `$XDG_CACHE_HOME/whichllm/` instead.
| File | TTL | Contents |
| --- | --- | --- |
| `models.json` | 6 hours | Serialized `ModelInfo` data |
| `benchmark.json` | 24 hours | Combined benchmark score map |
`--refresh` bypasses the relevant cache and writes a new one after fetching.
## Benchmark sources
`models/benchmark.py` builds one score map from multiple sources. Scores are
normalized to a 0-100 scale before ranking.
whichllm separates sources into two tiers:
| Tier | Sources | Treatment |
| --- | --- | --- |
| Current | LiveBench, Artificial Analysis, Aider, Vision | Overrides frozen scores for the same model |
| Frozen | Open LLM Leaderboard v2, Chatbot Arena ELO | Kept for older coverage, capped and recency-demoted |
Current sources can use live scrapes when reachable and curated snapshots when
the upstream page shape changes. The snapshot month is printed below rankings.
Frozen-only scores are demoted by model lineage. This prevents an older model
with a stale leaderboard score from outranking a newer generation simply because
the newer model was never added to that frozen leaderboard.
## Benchmark evidence
A model can receive benchmark evidence through several paths:
| Evidence | Meaning |
| --- | --- |
| `direct` | Exact independent benchmark match |
| `variant` | Suffix-stripped or `-Instruct` variant match |
| `base_model` | Match through HuggingFace `cardData.base_model` |
| `line_interp` | Size-aware interpolation within the same model line |
| `self_reported` | Uploader-provided `evalResults` only |
| `none` | No usable evidence |
Inheritance is rejected when the actual model size differs too much from the
reference. This catches small draft heads, MTP heads, and unrelated forks that
would otherwise borrow a larger base model's score.
## Family grouping
`models/grouper.py` groups related repos by:
1. `cardData.base_model`, when available.
2. Normalized repository names.
The normalizer removes common suffixes such as `-GGUF`, `-AWQ`, `-GPTQ`,
`-Instruct`, `-Chat`, `-FP16`, and date suffixes. It also handles versioned
model lines such as Qwen, Llama, Mistral, and DeepSeek.
Within a family, the ranker evaluates all members and variants but keeps only
the best result for the final table.
## Candidate variants
For each `ModelInfo`, `engine/ranker.py` builds candidate variants:
- Existing GGUF files are evaluated by quantization.
- Extreme low-bit GGUF variants are skipped unless explicitly requested with
`--quant`.
- Official safetensors-only repos can receive synthetic GGUF estimates for
common community conversions such as `Q4_K_M`, `Q5_K_M`, `Q6_K`, and `Q8_0`.
- Pre-quantized repos such as AWQ, GPTQ, FP8, and BF16 are not given synthetic
GGUF variants.
Apple Silicon and CPU-only rankings are restricted to GGUF candidates for
runtime stability. Linux with NVIDIA can also rank non-GGUF AWQ/GPTQ/FP16/BF16
repos.
## Ranking
For each candidate variant:
1. Estimate memory.
2. Check whether it can run.
3. Estimate tok/s and attach speed confidence/range metadata.
4. Resolve benchmark evidence.
5. Compute a quality score.
6. Keep the best variant for the model family.
The final sorting key stays close to the displayed quality score, with a small
direct-benchmark bonus and a CPU-only penalty. Full-GPU candidates are already
favored inside the score through the runtime-fit and speed adjustments, so the
sort key does not add a second full-GPU bonus.
See [Scoring](scoring.md) for the score details.
## Output
Output is split by surface:
- `output/ranking.py` renders hardware panels and recommendation tables.
- `output/json_output.py` renders ranking, `plan`, and `upgrade` JSON.
- `output/plan.py` renders `plan` tables.
- `output/upgrade.py` renders upgrade comparison tables.
- `output/display.py` re-exports those functions for older imports.
Normal ranking tables show memory required, estimated generation speed, fit
type, and published date. `--details` switches to download-oriented metadata.
Speed color is based on absolute usability, while `~` marks estimates with a
range and `?` marks low-confidence, backend-sensitive estimates.
docs/run-snippet.md
+161
View File
@@ -0,0 +1,161 @@
# Run and snippet
whichllm can do more than print recommendations:
- `whichllm run` starts an interactive chat with a selected model.
- `whichllm snippet` prints a Python script for manual use.
Both commands use the same model loading helpers in `cli.py`.
## `whichllm run`
```bash
whichllm run [MODEL_NAME]
```
If `MODEL_NAME` is provided, whichllm searches the fetched model list for an
exact ID, suffix match, or term match.
If `MODEL_NAME` is omitted, whichllm ranks models for the current hardware and
uses the top result.
Examples:
```bash
whichllm run
whichllm run "qwen 2.5 1.5b gguf"
whichllm run "phi 3 mini gguf" --cpu-only
whichllm run "llama 3 8b gguf" --quant Q5_K_M
```
## How `run` executes
`run` requires `uv` in `PATH`.
At runtime, whichllm:
1. Loads models from cache or HuggingFace.
2. Selects a model and quantization.
3. Generates a temporary Python chat script.
4. Runs that script with `uv run --no-project`.
5. Adds the required dependencies with repeated `--with` flags.
6. Deletes the temporary script after the chat exits.
This keeps the project environment clean. The temporary runtime dependencies
are not added to `pyproject.toml`.
## Supported model paths
### GGUF
GGUF models use:
- `llama-cpp-python`
- `huggingface-hub`
The generated script downloads the selected GGUF file with `hf_hub_download`
and loads it through `llama_cpp.Llama`.
GPU behavior:
- default: `n_gpu_layers=-1`
- `--cpu-only`: `n_gpu_layers=0`
### AWQ
AWQ repos are inferred from the model ID and use:
- `transformers`
- `torch`
- `accelerate`
- `autoawq`
### GPTQ
GPTQ repos are inferred from the model ID and use:
- `transformers`
- `torch`
- `accelerate`
- `auto-gptq`
### FP16, BF16, FP8, INT8, BNB 4-bit
Other non-GGUF repos use the Transformers path. The generated script uses
`device_map="auto"` unless `--cpu-only` is set.
## Quantization selection
For GGUF repos, whichllm chooses a variant by this preference order unless
`--quant` is provided:
```text
Q4_K_M, Q4_K_S, Q5_K_M, Q5_K_S, Q6_K, Q3_K_M, Q3_K_L, Q8_0, ...
```
This order favors a practical balance of memory and quality. Very low-bit
variants are available when explicitly requested but are not preferred by
default.
If the requested quantization is not available, `run` warns and falls back to
the best available match.
## Chat behavior
GGUF scripts call:
```python
llm.create_chat_completion(messages=messages, stream=True)
```
Transformers scripts use:
```python
tokenizer.apply_chat_template(...)
model.generate(...)
```
The chat loop keeps the current conversation history in memory until the
process exits. Type `exit`, `quit`, or `q` to stop.
## `whichllm snippet`
```bash
whichllm snippet [MODEL_NAME]
```
`snippet` prints a short Python example instead of running it.
Examples:
```bash
whichllm snippet "qwen 7b"
whichllm snippet "llama 3 8b gguf" --quant Q5_K_M
```
If no model is provided, `snippet` picks the most-downloaded GGUF model from
the fetched model list. This is different from `run`, which auto-ranks for the
current hardware when no model name is provided.
## Manual execution
The snippet output includes a suggested `uv run --no-project` command with the
needed `--with` dependencies.
Example shape:
```bash
uv run --no-project --with llama-cpp-python --with huggingface-hub script.py
```
## Practical notes
- First run can take time because dependencies and model weights need to
download.
- HuggingFace access rules still apply. Gated models may require local
HuggingFace authentication.
- `run` is a convenience path, not a full model manager.
- `snippet` is better when you want to adapt loading code into your own project.
- Generated Transformers scripts use `trust_remote_code=True`, matching common
HuggingFace local inference patterns. Review model repos before running code
from untrusted sources.
docs/scoring.md
+216
View File
@@ -0,0 +1,216 @@
# Scoring
whichllm does not pick the largest model that fits. It ranks candidates by a
composite score that tries to answer a more practical question:
> Of the models that can run here, which one is likely to be the best usable
> choice?
The source of truth is `engine/ranker.py`.
## Inputs
Each candidate score uses:
- model metadata from HuggingFace
- detected or simulated hardware
- estimated VRAM/RAM fit
- estimated tok/s
- quantization type
- benchmark evidence
- downloads and likes
- source organization
- model lineage and generation
The score is capped to `0..100`.
## Benchmark evidence
Independent benchmark matches are not all treated equally.
| Source | Weight | Meaning |
| --- | ---: | --- |
| `direct` | `0.62` | Exact independent benchmark match |
| `base_model` | `0.55` | Match through `cardData.base_model` |
| `variant` | `0.50` | Suffix-stripped variant match |
| `line_interp` | `0.40` | Size-aware model-line interpolation |
| `self_reported` | `0.30` | Uploader-provided HuggingFace eval only |
| `none` | `0.00` | No benchmark evidence |
`self_reported` evidence is intentionally weak. HuggingFace model cards can
contain useful evaluation data, but it is not the same as an independent
leaderboard.
## Size score
Model size is used as a rough world-knowledge proxy:
```text
size_score = 4.2 * log2(params_b) + 9
```
The result is capped at `35`.
For dense models, `params_b` is the parameter count. For MoE models, whichllm
uses total parameters for quality because all experts contribute to stored
knowledge. Active parameters are used later for speed.
## Quantization penalty
Lower-bit quantization can make a larger model fit, but it also reduces quality.
The score core is multiplied by `(1 - quant_penalty)`.
Examples:
| Quant | Penalty |
| --- | ---: |
| `Q8_0` | `0.01` |
| `Q6_K` | `0.02` |
| `Q5_K_M` | `0.03` |
| `Q4_K_M` | `0.05` |
| `Q3_K_M` | `0.08` |
| `Q2_K` | `0.25` |
| `IQ2_XXS` | `0.40` |
| `Q1_0` | `0.55` |
Extreme low-bit variants are excluded by default when better candidates exist.
They can still be requested explicitly with `--quant`.
## Evidence confidence
After benchmark and size are combined, weak evidence is dampened:
| Evidence state | Multiplier |
| --- | ---: |
| Direct benchmark | `1.00` |
| Inherited evidence | `0.78` |
| Self-reported evidence | `0.55` |
| No benchmark | `0.55` |
For inherited benchmark evidence, the raw score is also scaled by confidence
before entering the scoring function. Line interpolation therefore receives a
double discount: once for its interpolation confidence and once for being
inherited evidence.
## Runtime fit
The candidate's runtime form matters:
| Fit | Multiplier |
| --- | ---: |
| Full GPU | `1.00` |
| Partial offload | `0.42`-`0.88`, based on spill ratio |
| CPU-only | `0.50` |
Light partial offload is penalized less than heavy offload. MoE models receive
a milder penalty when the active parameter working set can plausibly stay on
GPU while inactive experts spill to CPU RAM.
The final family selection key does not add a separate full-GPU bonus. Runtime
fit is already reflected in the quality score through the multiplier above and
the speed adjustment below. CPU-only results receive a small extra sort penalty
when mixed with GPU-backed candidates.
## Speed adjustment
Speed is treated as a usability gate. It is not the main quality signal.
Required speed depends on fit:
| Fit | Required speed |
| --- | ---: |
| Full GPU | `8 tok/s` |
| Partial offload | `4 tok/s` |
| CPU-only | `1.5 tok/s` |
Candidates below the required speed receive up to `-8` points. Candidates above
it receive up to `+8` points.
After ranking, if any candidate is at least `5 tok/s`, whichllm drops candidates
below `1.5 tok/s`. This avoids recommending models that technically fit but are
not practical to use.
The reported speed is a point estimate, not a live benchmark. Ranking also
exposes speed confidence:
| Confidence | Range factor | Typical cases |
| --- | ---: | --- |
| `medium` | `0.60x`-`1.60x` | Normal GPU estimates, synthetic GGUF estimates, AMD shared-memory APU MoE estimates |
| `low` | `0.35x`-`2.00x` | CPU-only, partial offload, unknown bandwidth, Apple Silicon MoE |
| `high` | `0.85x`-`1.20x` | Reserved for future measured-speed data |
Speed cells are colored by absolute usability: red is under `4 tok/s`, yellow
is `4-10 tok/s`, green is `10-30 tok/s`, and bright green is `30+ tok/s`. `~`
marks medium-confidence estimates with a range, and `?` marks low-confidence
estimates. JSON exposes the same uncertainty data as `speed_confidence`,
`speed_range_tok_per_sec`, and `speed_notes`.
## Source trust
The source organization contributes a small adjustment:
- official model organizations receive a small bonus
- trusted GGUF converters can inherit that trust
- known repackagers receive a small penalty
The adjustment is intentionally small. It should break ties, not replace
benchmark and fit signals.
## Popularity
Downloads and likes act as tie-breakers. Their weight is lower when benchmark
evidence is strong and higher when evidence is weak.
Popularity has no effect for direct benchmark matches.
## Generation lineage
Some benchmark sources are frozen. A model released after a frozen leaderboard
cannot appear there, while older models can keep strong but stale scores.
whichllm uses family-specific lineage maps to avoid that inversion. Newer
generations can receive a small bonus; older generations can receive a small
penalty. This is applied carefully so direct benchmark evidence still matters.
Examples of tracked lineages include:
- Qwen
- Llama
- DeepSeek
- Gemma
- Phi
- Mistral
- GLM
- Kimi
- Granite
- OLMo
- T5 (incl. Flan-T5, mT5, ByT5, T5Gemma)
## Benchmark markers
The table score can include a marker:
| Marker | Meaning |
| --- | --- |
| none | Direct independent benchmark evidence |
| `~` | Estimated or inherited benchmark evidence |
| `!sr` | Self-reported HuggingFace eval only |
| `?` | No benchmark evidence |
Top-pick confidence is computed from the score gap, benchmark status, and fit
type. Partial-offload and CPU-only top picks are reported with lower confidence
than full-GPU direct-benchmark winners.
## Why a smaller model can win
A smaller model can outrank a larger one when it has:
- stronger current benchmark evidence
- a newer generation signal
- better quantization quality
- full-GPU fit instead of partial offload
- higher estimated speed
- a more trustworthy source
That is intentional. whichllm ranks likely usable quality, not parameter count
alone.
docs/troubleshooting.md
+423
View File
@@ -0,0 +1,423 @@
# Troubleshooting
This page lists common issues and the first checks to make.
## No GPU detected
Run:
```bash
whichllm hardware
```
If an NVIDIA GPU is missing:
- check that the driver is installed
- check `nvidia-smi`
- check that `nvidia-ml-py` can load NVML
whichllm falls back to `nvidia-smi`, but it still needs the NVIDIA driver tools
to be working.
If an AMD GPU is missing:
- on Linux, check `rocm-smi`, `lspci`, and `/sys/class/drm`
- on Windows, check that PowerShell can read `Win32_VideoController`
- for Ryzen AI / Radeon integrated graphics, check whether `whichllm hardware`
shows shared memory instead of a tiny 512 MB or 4 GB adapter
If an Intel iGPU is missing:
- Linux detection uses `lspci` or `/sys/class/drm`
- Windows detection uses `Win32_VideoController`
- many Intel iGPUs do not expose dedicated VRAM, so they may be shown as shared
memory graphics
## Simulate hardware instead
If detection is unavailable or you are planning a purchase, use `--gpu`:
```bash
whichllm --gpu "RTX 4090"
whichllm hardware --gpu "Apple M3 Max"
whichllm --gpu "RTX 5060 Ti" --vram 16
whichllm --gpu "2x RTX 4090"
whichllm --gpu "RTX 4090" --gpu "RTX 3090"
```
Use `--vram` when the GPU name has multiple memory variants or is not in the
database.
For detected iGPU or unified-memory systems, override the usable GPU memory and
bandwidth directly:
```bash
whichllm hardware --vram 8 --ram-bandwidth 68
whichllm --vram 8 --bandwidth 68
```
If `whichllm hardware` lists multiple GPUs, add `--gpu-index` with the GPU
number from that output.
`--vram` only applies to one simulated or detected GPU. For multi-GPU
simulation, use known GPU names and omit `--vram`.
## `--cpu-only` conflicts with `--gpu`
These flags are mutually exclusive:
```bash
whichllm --cpu-only --gpu "RTX 4090"
```
Choose one:
```bash
whichllm --cpu-only
whichllm --gpu "RTX 4090"
```
## `--vram` / `--bandwidth` needs a GPU
These overrides need either a detected GPU or a simulated GPU:
```bash
whichllm --vram 8 --ram-bandwidth 68
whichllm --gpu "RTX 3060" --vram 12
```
If no GPU is detected, use `--gpu` to simulate one or check the detection steps
above.
## No compatible models found
Try:
```bash
whichllm
whichllm --cpu-only
whichllm --refresh
```
Common causes:
- the selected `--quant` is too restrictive
- `--gpu-only` or `--fit full-gpu` filters out partial-offload and CPU-only candidates
- `--speed usable`, `--speed fast`, or `--min-speed` filters out slower candidates
- `--min-speed` is too high
- `--evidence strict` filters out all candidates
- the requested context length is too large
- available RAM is too low after reserving space for the OS
- disk free space is too low for the model weights
For very small machines, remove optional filters first:
```bash
whichllm --top 20
```
## Recommendations use RAM or CPU offload, but I only want VRAM
By default, whichllm includes any runnable candidate: full-GPU, partial-offload,
and CPU-only. This is useful for finding what can run at all, but it can be too
loose when you want only models that fit entirely in GPU VRAM.
Use:
```bash
whichllm --gpu-only
whichllm --fit gpu
whichllm --fit full-gpu
```
If no rows are shown, this machine has no ranked candidates that fit fully in
GPU memory under the current filters. Remove `--gpu-only`, lower the context
length, or try a smaller quantization.
## A model fits, but it is too slow
The default ranking table shows estimated generation speed. Slow rows are red,
marginal rows are yellow, usable rows are green, and fast rows are bright
green. The `~` and `?` markers are confidence markers for the estimate.
Filter slow rows with:
```bash
whichllm --speed usable # >=10 tok/s
whichllm --speed fast # >=30 tok/s
whichllm --min-speed 4 # exact floor, if you want a lower threshold
```
For an exact threshold:
```bash
whichllm --min-speed 10
```
## LM Studio or another runtime says the model barely does not fit
whichllm estimates model memory, but real runtimes can need extra room for
loader overhead, graph buffers, KV cache choices, and OS pressure. By default,
whichllm reserves a small automatic VRAM headroom before fit checks.
Tune it with:
```bash
whichllm --vram-headroom 1.5GB
whichllm --vram-headroom 10%
whichllm --vram-headroom none
```
Use `none` when you want the old raw-VRAM behavior.
## RAM offload depends on what else is running
Partial offload uses system RAM. If Docker, Elasticsearch, a browser, or
another workload is already using a large amount of memory, cap the offload
budget:
```bash
whichllm --ram-budget available
whichllm --ram-budget 8GB
whichllm --ram-budget 50%
```
`available` reads the current available RAM from the OS at startup. Fixed
values are useful when you know how much memory you want to leave for other
processes.
## Results look stale
whichllm caches model data for 6 hours and benchmark data for 24 hours.
Force a refresh:
```bash
whichllm --refresh
whichllm plan "qwen 7b" --refresh
```
The caches live under:
```text
~/.cache/whichllm/
```
If `XDG_CACHE_HOME` is set to an absolute path, the caches live under:
```text
$XDG_CACHE_HOME/whichllm/
```
## `uvx` fails with `realpath: command not found`
Some older macOS versions do not include a `realpath` command. If the `uvx`
launcher fails before whichllm starts, with output like:
```text
realpath: command not found
/Users/.../python: No such file or directory
```
run whichllm through Python's module entry point instead:
```bash
uvx --from whichllm python -m whichllm
```
Pass normal whichllm arguments after the module name:
```bash
uvx --from whichllm python -m whichllm --gpu "RTX 4090"
```
## The top pick has `~`, `!sr`, or `?`
These markers describe benchmark evidence:
| Marker | Meaning |
| --- | --- |
| `~` | Inherited or interpolated benchmark evidence |
| `!sr` | Uploader-reported benchmark only |
| `?` | No benchmark evidence |
Use stricter evidence when you want only independently matched benchmark data:
```bash
whichllm --evidence strict
whichllm --direct
```
Use `--evidence base` when base-model matches are acceptable but interpolation
and self-reported values are not.
## The largest model did not win
That is expected. whichllm scores:
- benchmark quality
- model size
- quantization loss
- full GPU vs partial offload vs CPU-only
- estimated speed
- evidence confidence
- source trust
- generation lineage
A smaller current-generation model with strong direct evidence can beat a
larger model that only barely fits or relies on stale benchmark data.
## Estimated speed differs from real speed
Speed is an estimate based on:
- model weight size
- MoE active parameters
- GPU memory bandwidth
- quantization efficiency
- backend factor
- partial-offload penalty
Real performance depends on the inference runtime, driver, prompt length,
batching, thermal limits, and background memory pressure.
The default ranking table shows the speed estimate and its confidence marker.
Use `--details` only when you want download counts instead.
Speed colors and markers:
- red: slow generation speed, under 4 tok/s
- yellow: marginal generation speed, 4-10 tok/s
- green: usable generation speed, 10-30 tok/s
- bright green: fast local generation speed, 30+ tok/s
- `~`: estimated speed range is available
- `?`: low-confidence estimate; runtime/backend differences can be large
JSON includes the same information as `speed_confidence`,
`speed_range_tok_per_sec`, and `speed_notes`.
## Apple Silicon partial offload looks different
Apple Silicon uses unified memory. Partial offload does not cross a discrete
PCIe boundary, so whichllm applies a milder speed penalty than it does for
discrete GPUs.
The same is true for recognized AMD shared-memory APUs such as Strix Halo,
Ryzen AI MAX, and Ryzen AI / Radeon 890M-class integrated graphics.
DGX Spark / NVIDIA GB10 is handled the same way when NVIDIA reports GPU memory
as unavailable.
On Windows, `Win32_VideoController.AdapterRAM` can cap around 4 GB. whichllm
uses the 64-bit registry memory value when it is available, and treats known
shared-memory APUs as unified-memory style devices instead of tiny discrete
GPUs.
## `run` says `uv is required`
Install `uv` first:
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```
Then retry:
```bash
whichllm run
```
## `run` cannot download a model
Possible causes:
- the model is gated on HuggingFace
- local HuggingFace authentication is missing
- the selected GGUF filename no longer exists
- network access failed
- disk space is too low
Try a known public GGUF model first:
```bash
whichllm run "qwen 2.5 1.5b gguf"
```
## Hugging Face API access fails or needs a mirror
whichllm uses the Hugging Face API to fetch model metadata. If direct access to
`huggingface.co` fails in your network, set `HF_ENDPOINT` to a compatible
endpoint root:
```powershell
$env:HF_ENDPOINT = "https://huggingface.co"
whichllm --refresh
```
```bash
HF_ENDPOINT="https://huggingface.co" whichllm --refresh
```
Do not include `/api` in `HF_ENDPOINT`; whichllm adds that path internally.
## How much disk space does `run` need?
Normal ranking commands do not download model weights. They cache Hugging Face
model metadata and benchmark metadata under the whichllm cache.
`whichllm run` downloads the selected GGUF file through `huggingface_hub`. The
required disk space is roughly the selected GGUF file size plus normal Hugging
Face cache overhead.
By default, Hugging Face stores downloaded files under:
```text
~/.cache/huggingface/hub
```
You can move that cache by setting `HF_HOME` or `HF_HUB_CACHE`.
Cleanup is handled by the Hugging Face cache tools:
```bash
hf cache scan
hf cache delete
```
whichllm does not currently delete model files automatically after a run.
## Ollama names do not match HuggingFace IDs
JSON output returns HuggingFace repo IDs:
```bash
whichllm --top 1 --json | jq -r '.models[0].model_id'
```
Ollama model names often use a different naming scheme. Map the HuggingFace ID
to your local Ollama model name before calling `ollama run`.
## Debugging a specific model
Use `plan` to inspect memory requirements:
```bash
whichllm plan "Qwen2.5-72B" --quant Q4_K_M
whichllm plan "Qwen2.5-72B" --quant Q8_0 --context-length 32768
```
Use plain output when filing issues:
```bash
whichllm --gpu "RTX 4090" --json
whichllm --gpu "RTX 4090" --markdown
whichllm hardware
```
Include:
- OS
- GPU name and VRAM
- CPU and RAM
- command used
- expected result
- actual result
pyproject.toml
+58
View File
@@ -0,0 +1,58 @@
[project]
name = "whichllm"
version = "0.5.15"
description = "Find the best LLM that runs on your hardware"
authors = [{name = "Andyyyy64"}]
readme = "README.md"
requires-python = ">=3.11"
license = "MIT"
homepage = "https://github.com/Andyyyy64/whichllm"
repository = "https://github.com/Andyyyy64/whichllm"
keywords = ["llm", "local-llm", "gpu", "vram", "huggingface", "inference", "hardware", "cli", "recommendation"]
classifiers = [
"Development Status :: 4 - Beta",
"Environment :: Console",
"Intended Audience :: Developers",
"Intended Audience :: Science/Research",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Topic :: Scientific/Engineering :: Artificial Intelligence",
"Topic :: System :: Hardware",
]
dependencies = [
"typer>=0.9",
"rich>=13.0",
"httpx>=0.27",
"psutil>=5.9",
"dbgpu[fuzz]>=1.0",
"nvidia-ml-py>=12.0",
]
[project.urls]
Homepage = "https://github.com/Andyyyy64/whichllm"
Repository = "https://github.com/Andyyyy64/whichllm"
"Bug Tracker" = "https://github.com/Andyyyy64/whichllm/issues"
Changelog = "https://github.com/Andyyyy64/whichllm/blob/main/CHANGELOG.md"
[project.scripts]
whichllm = "whichllm.cli:app"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/whichllm"]
[tool.pytest.ini_options]
testpaths = ["tests"]
[dependency-groups]
dev = [
"pytest>=8.0",
"pyarrow>=23.0.1",
]
scripts/import_livebench_csv.py
+116
View File
@@ -0,0 +1,116 @@
"""Convert a LiveBench leaderboard CSV into the inlined Python dict.
LiveBench publishes their leaderboard as a dated CSV (e.g.
``https://livebench.ai/table_2026_01_08.csv``).
Usage:
curl https://livebench.ai/table_2026_01_08.csv | python scripts/import_livebench_csv.py
"""
from __future__ import annotations
import csv
import sys
# LiveBench CSV model name -> list of HuggingFace ids that share the score.
# When several CSV rows map onto the same HF id (e.g. thinking vs. base),
# the highest average wins.
CSV_NAME_TO_HF_IDS: dict[str, list[str]] = {
"deepseek-v3.2": ["deepseek-ai/DeepSeek-V3.2"],
"deepseek-v3.2-exp": ["deepseek-ai/DeepSeek-V3.2-Exp"],
"deepseek-v3.2-exp-thinking": ["deepseek-ai/DeepSeek-V3.2-Exp"],
"deepseek-v3.2-thinking": ["deepseek-ai/DeepSeek-V3.2"],
"deepseek-v4-flash": ["deepseek-ai/DeepSeek-V4-Flash"],
"deepseek-v4-pro": ["deepseek-ai/DeepSeek-V4-Pro"],
"devstral-2512": ["mistralai/Devstral-2512"],
"gemma-4-31b-it": ["google/gemma-4-31b-it"],
"glm-4.6": ["zai-org/GLM-4.6"],
"glm-4.6v": ["zai-org/GLM-4.6V"],
"glm-4.7": ["zai-org/GLM-4.7"],
"glm-5": ["zai-org/GLM-5"],
"glm-5.1": ["zai-org/GLM-5.1"],
"gpt-oss-120b": ["openai/gpt-oss-120b"],
"kimi-k2-instruct": ["moonshotai/Kimi-K2-Instruct"],
"kimi-k2-thinking": ["moonshotai/Kimi-K2-Thinking"],
"kimi-k2.5-thinking": ["moonshotai/Kimi-K2.5"],
"kimi-k2.6-thinking": ["moonshotai/Kimi-K2.6-Thinking"],
"mimo-v2-pro": ["XiaomiMiMo/MiMo-V2-Pro"],
"minimax-m2.5": ["MiniMaxAI/MiniMax-M2.5"],
"minimax-m2.7": ["MiniMaxAI/MiniMax-M2.7"],
"nemotron-3-super-120b-a12b": ["nvidia/Nemotron-3-Super-120B-A12B"],
"qwen3-235b-a22b-instruct-2507": ["Qwen/Qwen3-235B-A22B-Instruct-2507"],
"qwen3-235b-a22b-thinking-2507": ["Qwen/Qwen3-235B-A22B-Thinking-2507"],
"qwen3-30b-a3b-thinking": ["Qwen/Qwen3-30B-A3B-Thinking-2507"],
"qwen3-32b-thinking": ["Qwen/Qwen3-32B"],
"qwen3-next-80b-a3b-instruct": ["Qwen/Qwen3-Next-80B-A3B-Instruct"],
"qwen3-next-80b-a3b-thinking": ["Qwen/Qwen3-Next-80B-A3B-Thinking"],
"qwen3.6-27b": ["Qwen/Qwen3.6-27B"],
}
def row_average(row: dict[str, str]) -> float | None:
nums: list[float] = []
for key, value in row.items():
if key == "model" or not value:
continue
try:
nums.append(float(value))
except ValueError:
continue
if not nums:
return None
return sum(nums) / len(nums)
def main(argv: list[str]) -> int:
rows = list(csv.DictReader(sys.stdin))
best: dict[str, float] = {}
matched: set[str] = set()
for row in rows:
name = row["model"]
hf_ids = CSV_NAME_TO_HF_IDS.get(name)
if not hf_ids:
continue
avg = row_average(row)
if avg is None:
continue
matched.add(name)
for hf_id in hf_ids:
if avg > best.get(hf_id, 0.0):
best[hf_id] = avg
unmapped_open = [
row["model"]
for row in rows
if row["model"] not in matched
and not any(
tok in row["model"]
for tok in (
"claude",
"gpt-",
"gemini",
"grok",
"arcee",
"elephant",
)
)
]
if unmapped_open:
print(
f"# note: {len(unmapped_open)} unmapped row(s) in CSV — extend "
"CSV_NAME_TO_HF_IDS if any are open-weight:",
" ".join(sorted(unmapped_open)),
file=sys.stderr,
)
print("{")
for hf_id in sorted(best):
print(f' "{hf_id}": {best[hf_id]:.1f},')
print("}")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
src/whichllm/__init__.py
+1
View File
@@ -0,0 +1 @@
"""whichllm: Find the best LLM that runs on your hardware."""
src/whichllm/__main__.py
+7
View File
@@ -0,0 +1,7 @@
"""Module entry point for ``python -m whichllm``."""
from whichllm.cli import app
if __name__ == "__main__":
app()
src/whichllm/cli.py
+1507
View File
File diff suppressed because it is too large Load Diff
src/whichllm/constants.py
+54
View File
@@ -0,0 +1,54 @@
"""Compatibility shim: curated registries now live under ``whichllm.data``.
This module re-exports the same names so existing imports
(``from whichllm.constants import ...``) keep working. New code should import
from the specific ``whichllm.data.*`` submodule instead.
"""
from whichllm.data.framework import (
FRAMEWORK_OVERHEAD_BYTES,
MIN_COMPUTE_CAPABILITY_OLLAMA,
MIN_COMPUTE_CAPABILITY_VLLM,
)
from whichllm.data.gpu import (
_GiB,
AMD_SHARED_MEMORY_APU_MARKERS,
CURATED_GPU_SPECS,
CuratedGPUSpec,
GPU_BANDWIDTH,
GPU_MEMORY_CLOCK_VARIANTS,
INTEL_PCI_DEVICE_NAMES,
NVIDIA_COMPUTE_CAPABILITY,
VULKAN_ONLY_GPUS,
)
from whichllm.data.lineage import (
MODEL_GENERATION_BONUS_MAX,
MODEL_GENERATION_PENALTY_MAX,
MODEL_LINEAGE_VERSIONS,
)
from whichllm.data.quantization import (
QUANT_BYTES_PER_WEIGHT,
QUANT_PREFERENCE_ORDER,
QUANT_QUALITY_PENALTY,
)
__all__ = [
"_GiB",
"AMD_SHARED_MEMORY_APU_MARKERS",
"CURATED_GPU_SPECS",
"CuratedGPUSpec",
"FRAMEWORK_OVERHEAD_BYTES",
"GPU_BANDWIDTH",
"GPU_MEMORY_CLOCK_VARIANTS",
"INTEL_PCI_DEVICE_NAMES",
"MIN_COMPUTE_CAPABILITY_OLLAMA",
"MIN_COMPUTE_CAPABILITY_VLLM",
"MODEL_GENERATION_BONUS_MAX",
"MODEL_GENERATION_PENALTY_MAX",
"MODEL_LINEAGE_VERSIONS",
"NVIDIA_COMPUTE_CAPABILITY",
"QUANT_BYTES_PER_WEIGHT",
"QUANT_PREFERENCE_ORDER",
"QUANT_QUALITY_PENALTY",
"VULKAN_ONLY_GPUS",
]
src/whichllm/data/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Curated reference data: GPU specs, quantization tiers, model lineage, framework limits."""
src/whichllm/data/framework.py
+8
View File
@@ -0,0 +1,8 @@
"""Framework overhead and minimum compute capability thresholds."""
# Framework overhead in bytes (~500MB)
FRAMEWORK_OVERHEAD_BYTES = 500_000_000
# Minimum compute capability for common frameworks
MIN_COMPUTE_CAPABILITY_OLLAMA = (5, 0)
MIN_COMPUTE_CAPABILITY_VLLM = (7, 0)
src/whichllm/data/gpu.py
+299
View File
@@ -0,0 +1,299 @@
"""GPU bandwidth, VRAM, NVIDIA compute capability, and GPU markers."""
from __future__ import annotations
from typing import NamedTuple
_GiB = 1024**3
class CuratedGPUSpec(NamedTuple):
"""Small curated spec for GPUs missing or ambiguous in dbgpu."""
name: str
vendor: str
vram_gb: float
memory_bandwidth_gbps: float
shared_memory: bool = False
AMD_SHARED_MEMORY_APU_MARKERS: tuple[str, ...] = (
"STRIX HALO",
"STRXLGEN",
"RADEON 8050S",
"RADEON 8060S",
"RADEON 890M",
"RADEON 880M",
"RADEON 860M",
"RADEON 840M",
"RADEON 780M",
"RADEON 760M",
"RADEON 740M",
"RADEON 680M",
"RADEON 660M",
"RYZEN AI 9",
"RYZEN AI 7",
"RYZEN AI 5",
"RYZEN AI MAX",
)
# GPU memory bandwidth in GB/s (theoretical peak)
# Key: substring matched against GPU name (case-insensitive)
GPU_BANDWIDTH: dict[str, float] = {
# NVIDIA Consumer - RTX 50 series
"RTX 5090": 1792.0,
"RTX 5080": 960.0,
"RTX 5070 Ti": 896.0,
"RTX 5070": 672.0,
"RTX 5060 Ti": 448.0,
"RTX 5060": 336.0,
"RTX 3050": 224.0,
# NVIDIA Consumer - RTX 40 series
"RTX 4090": 1008.0,
"RTX 4080 SUPER": 736.0,
"RTX 4080": 716.8,
"RTX 4070 Ti SUPER": 672.0,
"RTX 4070 Ti": 504.0,
"RTX 4070 SUPER": 504.0,
"RTX 4070": 504.0,
"RTX 4060 Ti": 288.0,
"RTX 4060": 272.0,
# NVIDIA Consumer - RTX 30 series
"RTX 3090 Ti": 1008.0,
"RTX 3090": 936.2,
"RTX 3080 Ti": 912.4,
"RTX 3080": 760.3,
"RTX 3070 Ti": 608.3,
"RTX 3070": 448.0,
"RTX 3060 Ti": 448.0,
"RTX 3060": 360.0,
# NVIDIA Consumer - RTX 20 series
"RTX 2080 Ti": 616.0,
"RTX 2080 SUPER": 496.0,
"RTX 2080": 448.0,
"RTX 2070 SUPER": 448.0,
"RTX 2070": 448.0,
"RTX 2060 SUPER": 448.0,
"RTX 2060": 336.0,
# NVIDIA Consumer - GTX 16 series
"GTX 1660 Ti": 288.0,
"GTX 1660 SUPER": 336.0,
"GTX 1660": 192.0,
"GTX 1650 SUPER": 192.0,
# GTX 1650 GDDR5 (8 Gbps x 128-bit). A later GDDR6 revision runs 192 GB/s;
# both share the TU117 die and PCI id 0x1F82, so they are disambiguated by
# memory clock via GPU_MEMORY_CLOCK_VARIANTS below. 128 is the conservative
# default used when the memory clock is unknown.
"GTX 1650": 128.0,
# NVIDIA Data Center
"H100": 3350.0,
"H200": 4800.0,
"DGX Spark": 273.0,
"GB10": 273.0,
"A100 80GB": 2039.0,
"A100 40GB": 1555.0,
"A100": 1555.0,
"RTX A3000 Laptop": 264.0,
"A6000": 768.0,
"A5000": 768.0,
"A4000": 448.0,
"L40S": 864.0,
"L40": 864.0,
"L4": 300.0,
"T4": 320.0,
"V100": 900.0,
"P100": 732.0,
# NVIDIA Kepler (legacy, Vulkan-only — no CUDA in modern llama.cpp).
# Values are theoretical peak memory bandwidth (GB/s) from NVIDIA
# datasheets. Kepler (compute capability 3.x) was dropped by CUDA 12 and
# current llama.cpp CUDA builds, so these cards run via the Vulkan backend
# on Linux. See VULKAN_ONLY_GPUS below.
"Quadro K6000": 288.0,
"Quadro K5200": 192.3,
"Quadro K4200": 173.0,
"Quadro K2200": 80.0,
"Quadro K620": 29.0,
"Quadro K420": 14.4,
"GTX 780": 288.4,
"GTX 770": 224.3,
"GTX 760": 192.2,
# AMD
"R9700": 640.0,
"RX 9070 XT": 640.0,
"RX 9070": 560.0,
"RX 9060 XT": 320.0,
"RX 7900 XTX": 960.0,
"RX 7900 XT": 800.0,
"RX 7800 XT": 624.0,
"RX 7700 XT": 432.0,
"RX 7600": 288.0,
"RX 6950 XT": 576.0,
"RX 6900 XT": 512.0,
"RX 6800 XT": 512.0,
"RX 6800": 512.0,
"RX 6750 XT": 432.0,
"RX 6700 XT": 384.0,
"RX 6700": 320.0,
"RX 6650 XT": 256.0,
"RX 6600 XT": 256.0,
"RX 6600": 224.0,
# AMD APUs / shared-memory graphics
"Ryzen AI MAX+ 395": 256.0,
"Ryzen AI MAX 395": 256.0,
"Radeon 890M": 120.0,
"Radeon 880M": 120.0,
"Radeon 860M": 90.0,
"Radeon 840M": 60.0,
"Radeon 780M": 90.0,
"Radeon 760M": 75.0,
"Radeon 740M": 60.0,
"Radeon 680M": 75.0,
"Radeon 660M": 55.0,
"Radeon 8060S": 256.0,
"Radeon 8050S": 256.0,
"Strix Halo": 256.0,
"STRXLGEN": 256.0,
"MI300X": 5300.0,
"MI250X": 3276.0,
"MI210": 1638.0,
# Intel discrete GPUs
"Arc Pro B70": 608.0,
"Battlemage G31": 608.0,
# Apple Silicon (unified memory bandwidth)
"M1 Ultra": 800.0,
"M1 Max": 400.0,
"M1 Pro": 200.0,
"M1": 68.25,
"M2 Ultra": 800.0,
"M2 Max": 400.0,
"M2 Pro": 200.0,
"M2": 100.0,
"M3 Ultra": 800.0,
"M3 Max": 400.0,
"M3 Pro": 150.0,
"M3": 100.0,
"M4 Ultra": 819.2,
"M4 Max": 546.0,
"M4 Pro": 273.0,
"M4": 120.0,
"M5 Max": 614.0,
"M5 Pro": 307.0,
"M5": 153.0,
}
CURATED_GPU_SPECS: dict[str, CuratedGPUSpec] = {
"Arc Pro B70": CuratedGPUSpec(
name="Intel Arc Pro B70",
vendor="intel",
vram_gb=32.0,
memory_bandwidth_gbps=608.0,
),
"Battlemage G31": CuratedGPUSpec(
name="Battlemage G31 [Intel Graphics]",
vendor="intel",
vram_gb=32.0,
memory_bandwidth_gbps=608.0,
),
}
INTEL_PCI_DEVICE_NAMES: dict[str, str] = {
"0xe223": "Battlemage G31 [Intel Graphics]",
}
# NVIDIA GPU compute capability lookup (substring match, case-insensitive)
NVIDIA_COMPUTE_CAPABILITY: dict[str, tuple[int, int]] = {
# RTX 50 series (Blackwell)
"RTX 5090": (10, 0),
"RTX 5080": (10, 0),
"RTX 5070": (10, 0),
"RTX 5070 Ti": (10, 0),
"RTX 5060": (10, 0),
# RTX 40 series (Ada Lovelace)
"RTX 4090": (8, 9),
"RTX 4080": (8, 9),
"RTX 4070": (8, 9),
"RTX 4060": (8, 9),
# RTX 30 series (Ampere)
"RTX 3090": (8, 6),
"RTX 3080": (8, 6),
"RTX 3070": (8, 6),
"RTX 3060": (8, 6),
# RTX 20 series (Turing)
"RTX 2080": (7, 5),
"RTX 2070": (7, 5),
"RTX 2060": (7, 5),
# GTX 16 series (Turing)
"GTX 1660": (7, 5),
"GTX 1650 SUPER": (7, 5),
"GTX 1650": (7, 5),
# GTX 10 series (Pascal)
"GTX 1080": (6, 1),
"GTX 1070": (6, 1),
"GTX 1060": (6, 1),
# Data Center
"H100": (9, 0),
"H200": (9, 0),
"DGX Spark": (12, 1),
"GB10": (12, 1),
"A100": (8, 0),
"RTX A3000 Laptop": (8, 6),
"A6000": (8, 6),
"A5000": (8, 6),
"A4000": (8, 6),
"L40": (8, 9),
"L4": (8, 9),
"T4": (7, 5),
"V100": (7, 0),
"P100": (6, 0),
# Kepler series (compute capability 3.x) — legacy, Vulkan-only.
# CUDA 12 and current llama.cpp CUDA builds dropped Kepler support, so
# these cards only run through the Vulkan backend. See VULKAN_ONLY_GPUS.
"Quadro K6000": (3, 5),
"Quadro K5200": (3, 5),
"Quadro K4200": (3, 0),
"Quadro K2200": (3, 0),
"Quadro K620": (3, 0),
"Quadro K420": (3, 0),
"GTX 780": (3, 5),
"GTX 770": (3, 0),
"GTX 760": (3, 0),
}
# GPUs sold under one marketing name in multiple memory configurations that the
# driver name and PCI device id cannot tell apart, resolved by max memory clock
# (MHz) at detection time. Each value is a list of (min_mem_clock_mhz,
# bandwidth_gbps), highest threshold first; the first threshold the detected
# clock meets wins. The base name stays in GPU_BANDWIDTH as the conservative
# default used when the memory clock is unknown.
# Threshold sits between the two memory-clock regimes (GDDR5 ~4 GHz vs GDDR6
# ~6 GHz), well clear of either, so factory-OC boards do not straddle it.
GPU_MEMORY_CLOCK_VARIANTS: dict[str, list[tuple[float, float]]] = {
# GTX 1650 shipped as the original GDDR5 (8 Gbps x 128-bit = 128 GB/s) and a
# later GDDR6 revision (12 Gbps x 128-bit = 192 GB/s) on the same TU117 die
# and PCI id 0x1F82, so only the memory clock distinguishes them. Measured on
# a GDDR6 board (VBIOS 90.17.4D.00.1E): nvidia-smi reports clocks.max.memory
# = 6001 MHz, and Qwen3-1.7B Q4_K_M decodes at 75.4 tok/s, matching the
# 192 GB/s estimate (~78) and not 128's (~52). The GDDR5 board reports
# ~4001 MHz (NVIDIA spec, 8 Gbps GDDR5; not independently measured here);
# the 5500 MHz split is far from both.
"GTX 1650": [(5500.0, 192.0), (0.0, 128.0)],
}
# Legacy NVIDIA GPUs with no CUDA support in modern llama.cpp builds.
# Kepler (compute capability 3.0/3.5) was removed in CUDA 12, so these cards
# can only be used through the Vulkan backend (Linux) in current llama.cpp.
# Entries are substrings matched case-insensitively against the GPU name, the
# same convention used by GPU_BANDWIDTH and NVIDIA_COMPUTE_CAPABILITY.
VULKAN_ONLY_GPUS: frozenset[str] = frozenset(
{
"Quadro K6000",
"Quadro K5200",
"Quadro K4200",
"Quadro K2200",
"Quadro K620",
"Quadro K420",
"GTX 780",
"GTX 770",
"GTX 760",
}
)
src/whichllm/data/lineage.py
+161
View File
@@ -0,0 +1,161 @@
"""Model lineage / generation half-order used to bonus or penalize family versions."""
# Generation lineage half-order.
# For each "family stem" we encode a monotone-increasing version map so that
# the ranker can apply a small bonus/penalty depending on whether a model
# represents the newest generation of its family. This avoids the situation
# where an older series with stale Open-LLM-Leaderboard data ranks above a
# newer release for which the leaderboard simply has no data yet.
#
# Each entry is a list of (regex_pattern, generation_index) tuples evaluated
# in order; first match wins. Patterns match against lowercased model_id.
# Higher index = newer.
MODEL_LINEAGE_VERSIONS: dict[str, list[tuple[str, int]]] = {
"qwen": [
# ordered newest -> oldest so the bonus reflects the strongest claim
(r"qwen3\.6", 7),
(r"qwen3\.5", 6),
(r"qwen3-next", 6),
(r"qwen3-coder-next", 6),
(r"qwen3-omni", 5),
(r"qwen3", 5),
(r"qwq", 4),
(r"qwen2\.5", 3),
(r"qwen2(?!\.5)", 2),
(r"qwen1", 1),
(r"qwen-(7b|14b|72b)", 1),
],
"llama": [
(r"llama-?4\.5", 5),
(r"llama-?4", 4),
(r"llama-?3\.3", 3),
(r"llama-?3\.2", 3),
(r"llama-?3\.1", 3),
(r"meta-llama-?3(?!\.)", 2),
(r"llama-?2", 1),
],
"deepseek": [
(r"deepseek-v4", 5),
(r"deepseek-v3\.2", 4),
(r"deepseek-v3\.1", 4),
(r"deepseek-r1-0528", 4),
(r"deepseek-r1", 3),
(r"deepseek-v3-0324", 3),
(r"deepseek-v3(?!\.)", 3),
(r"deepseek-v2\.5", 2),
(r"deepseek-v2(?!\.5)", 1),
(r"deepseek-coder-v2", 2),
(r"deepseek-coder(?!-v2)", 1),
],
"gemma": [
# Avoid reading the "gemma" segment inside T5Gemma ids as a Gemma
# generation. T5Gemma is handled by the "t5" family instead.
(r"(?<!t5)(?<!t5[-_])gemma-?4", 4),
(r"(?<!t5)(?<!t5[-_])gemma-?3", 3),
(r"(?<!t5)(?<!t5[-_])gemma-?2", 2),
(r"(?<!t5)(?<!t5[-_])gemma(?!-?[2-9])", 1),
],
"phi": [
(r"phi-?5", 5),
(r"phi-?4", 4),
(r"phi-?3\.5", 3),
(r"phi-?3(?!\.5)", 2),
(r"phi-?2", 1),
],
"mistral_small": [
(r"mistral-small-3\.2", 4),
(r"mistral-small-2506", 4),
(r"mistral-small-3\.1", 3),
(r"mistral-small-3", 3),
(r"mistral-small-2501", 3),
(r"mistral-small.*2409", 2),
(r"mistral-small", 1),
],
"mistral_large": [
(r"mistral-large-3", 4),
(r"mistral-large-instruct-2411", 3),
(r"mistral-large-2411", 3),
(r"mistral-large-2407", 2),
(r"mistral-large", 1),
],
"mistral_7b": [
(r"mistral-?7b-instruct-v0\.3", 3),
(r"mistral-?7b-instruct-v0\.2", 2),
(r"mistral-?7b-instruct-v0\.1", 1),
],
"mixtral": [
(r"mixtral-8x22b", 2),
(r"mixtral-8x7b", 1),
],
"gpt_oss": [
(r"gpt-oss-120b", 2),
(r"gpt-oss-20b", 2),
(r"gpt-oss", 1),
],
"glm": [
(r"glm-?5\.1", 6),
(r"glm-?5(?!\.)", 5),
(r"glm-?4\.7", 4),
(r"glm-?4\.6", 3),
(r"glm-?4\.5", 3),
(r"glm-?4(?!\.[5-9])", 2),
(r"chatglm", 1),
],
"kimi": [
(r"kimi-?k2\.6", 4),
(r"kimi-?k2\.5", 3),
(r"kimi-?k2-thinking", 3),
(r"kimi-?k2", 2),
(r"kimi", 1),
],
"mimo": [
(r"mimo-?v2\.5", 3),
(r"mimo-?v2", 2),
(r"mimo-?7b", 1),
(r"mimo", 1),
],
"granite": [
(r"granite-?4\.1", 5),
(r"granite-?4", 4),
(r"granite-?3\.[2-9]", 3),
(r"granite-?3\.1", 2),
(r"granite-?3\.0", 2),
(r"granite", 1),
],
"olmo": [
(r"olmo-?3", 3),
(r"olmo-?2", 2),
(r"olmo(?!-?[2-9])", 1),
],
"yi": [
(r"yi-lightning", 3),
(r"yi-1\.5", 2),
(r"yi-(6b|9b|34b)(?!.*1\.5)", 1),
],
"t5": [
# Encoder-decoder T5 family, ordered newest -> oldest. The bare "t5"
# fallback is boundary-guarded because "t5" is a collision-prone
# substring (e.g. "gpt5"); every named variant is matched before it.
(r"t5[-_]?gemma", 5), # T5Gemma (2025) — Gemma-adapted encoder-decoder
(r"flan-?t5", 4), # Flan-T5 instruction-tuned (2022)
(r"flan-?ul2", 4), # Flan-UL2 (2023)
(r"codet5p", 3), # CodeT5+ (2023); before codet5 since it's a superset
(r"ul2", 3), # UL2 (2022)
(r"code-?t5", 2), # CodeT5 (2021)
(r"long-?t5", 2), # LongT5 (2021)
(r"byt5", 2), # ByT5 byte-level (2021)
(r"mt5", 2), # mT5 multilingual (2020)
(r"t5-?v1[._]1", 2), # T5 v1.1 / LM-adapted (2020)
(r"(?<![a-z0-9])t5(?![a-z])", 1), # original T5 (2019)
],
}
# Maximum bonus (in raw quality-score points) applied to the newest generation
# of a recognized family. The bonus interpolates downwards for older versions.
# These are larger than the initial pass because frozen leaderboards (OLLB v2,
# Arena 2025-07) systematically over-reward 2024-era models like Qwen2.5-32B
# that are no longer the current frontier; the lineage signal pulls newer
# releases past their older siblings even when the older one has stale-but-high
# leaderboard data.
MODEL_GENERATION_BONUS_MAX = 10.0
MODEL_GENERATION_PENALTY_MAX = 6.0
src/whichllm/data/quantization.py
+123
View File
@@ -0,0 +1,123 @@
"""Quantization tiers: bytes-per-weight, quality penalty, and preference order."""
# Bytes per weight for each quantization type
QUANT_BYTES_PER_WEIGHT: dict[str, float] = {
"F32": 4.0,
"F16": 2.0,
"BF16": 2.0,
"Q8_0": 1.0625,
"Q6_K": 0.8125,
"Q5_K_M": 0.6875,
"Q5_K_S": 0.6875,
"Q5_0": 0.625,
"Q4_K_M": 0.5625,
"Q4_K_S": 0.5625,
"Q4_0": 0.5,
"Q3_K_M": 0.4375,
"Q3_K_S": 0.4375,
"Q3_K_L": 0.4375,
"Q2_K": 0.3125,
"IQ4_XS": 0.5,
"IQ3_XXS": 0.375,
"IQ2_XXS": 0.25,
# 4-bit microscaling float formats (OCP MXFP4 / NVIDIA NVFP4).
# MXFP4: E2M1 element + one E8M0 (8-bit) scale per block of 32 weights
# -> (4*32 + 8) / 32 = 4.25 bits/weight = 0.53125 bytes.
# NVFP4: E2M1 element + one E4M3 (8-bit) scale per block of 16 weights
# (plus a negligible per-tensor FP32 scale) -> (4*16 + 8) / 16 = 4.5 bits
# = 0.5625 bytes, the same footprint as Q4_K_M.
"MXFP4": 0.53125,
"NVFP4": 0.5625,
# Sub-2-bit / ternary tiers (extremely lossy)
"Q1_0": 0.28,
"Q2_0": 0.28,
"TQ1_0": 0.21,
"TQ2_0": 0.28,
"IQ1_S": 0.21,
"IQ1_M": 0.22,
"IQ2_S": 0.275,
"IQ2_M": 0.30,
"IQ3_S": 0.40,
"IQ3_M": 0.42,
"IQ3_XS": 0.41,
"IQ4_NL": 0.5,
}
# Quality penalty for each quantization type (fraction of quality lost)
# Sub-2-bit and ternary quants lose 30-60% of model quality - whichllm
# previously fell back to 5% which over-rewarded extreme quants.
QUANT_QUALITY_PENALTY: dict[str, float] = {
"F32": 0.0,
"F16": 0.0,
"BF16": 0.0,
"Q8_0": 0.01,
"Q6_K": 0.02,
"Q5_K_M": 0.03,
"Q5_K_S": 0.035,
"Q5_0": 0.035,
"Q4_K_M": 0.05,
"Q4_K_S": 0.055,
"Q4_0": 0.06,
# NVFP4's finer per-16 E4M3 scale recovers more accuracy than MXFP4's
# coarser per-32 E8M0 (power-of-two) scale, so NVFP4 sits on par with the
# mature 4-bit quantizers (Q4_K_M / AWQ at 0.05) while MXFP4 is slightly
# lossier.
"NVFP4": 0.05,
"MXFP4": 0.06,
"Q3_K_M": 0.08,
"Q3_K_S": 0.12,
"Q3_K_L": 0.075,
"Q2_K": 0.25,
"IQ4_XS": 0.05,
"IQ4_NL": 0.055,
"IQ3_XS": 0.16,
"IQ3_S": 0.17,
"IQ3_M": 0.16,
"IQ3_XXS": 0.18,
"IQ2_M": 0.30,
"IQ2_S": 0.32,
"IQ2_XXS": 0.40,
"IQ1_M": 0.50,
"IQ1_S": 0.55,
"Q2_0": 0.45,
"Q1_0": 0.55,
"TQ2_0": 0.45,
"TQ1_0": 0.55,
}
# Preferred quantization types ordered from best to acceptable.
# Sub-3-bit and 1-bit ternary variants sit at the tail so they are only
# selected when nothing else is available or when explicitly requested.
QUANT_PREFERENCE_ORDER = [
"Q4_K_M",
"Q4_K_S",
"NVFP4",
"MXFP4",
"Q5_K_M",
"Q5_K_S",
"Q6_K",
"Q3_K_M",
"Q3_K_L",
"Q8_0",
"IQ4_XS",
"IQ4_NL",
"Q4_0",
"Q5_0",
"Q3_K_S",
"F16",
"BF16",
"IQ3_M",
"IQ3_S",
"IQ3_XS",
"Q2_K",
"IQ3_XXS",
"IQ2_M",
"IQ2_S",
"IQ2_XXS",
"IQ1_M",
"IQ1_S",
"Q2_0",
"TQ2_0",
"Q1_0",
"TQ1_0",
]
src/whichllm/engine/__init__.py
View File
src/whichllm/engine/compatibility.py
+261
View File
@@ -0,0 +1,261 @@
"""Compatibility checking: can a model run on given hardware?"""
from __future__ import annotations
from whichllm.constants import _GiB
from whichllm.constants import MIN_COMPUTE_CAPABILITY_OLLAMA
from whichllm.constants import VULKAN_ONLY_GPUS
from whichllm.engine.quantization import estimate_weight_bytes
from whichllm.engine.types import CompatibilityResult
from whichllm.engine.vram import estimate_vram
from whichllm.hardware.memory import effective_usable_ram
from whichllm.hardware.types import GPUInfo, HardwareInfo
from whichllm.models.types import GGUFVariant, ModelInfo
_MULTI_GPU_FRAMEWORK_OVERHEAD_BYTES = int(0.3 * _GiB)
_MULTI_GPU_HOMOGENEOUS_UTILIZATION = 0.95
_MULTI_GPU_HETEROGENEOUS_UTILIZATION = 0.90
def _gpu_available_memory(
gpu: GPUInfo, usable_ram: int, *, ram_budget_active: bool = False
) -> int:
vram_bytes = (
gpu.usable_vram_bytes if gpu.usable_vram_bytes is not None else gpu.vram_bytes
)
if gpu.shared_memory and vram_bytes < 2 * _GiB and not gpu.vram_overridden:
return usable_ram
if gpu.shared_memory and ram_budget_active:
return min(vram_bytes, usable_ram)
return vram_bytes
def _uses_shared_system_pool(gpu: GPUInfo) -> bool:
return gpu.shared_memory and gpu.vram_bytes < 2 * _GiB and not gpu.vram_overridden
def _is_vulkan_only_gpu(gpu: GPUInfo) -> bool:
"""Return True for legacy NVIDIA GPUs with no modern CUDA support.
Kepler cards (compute capability 3.x) were dropped by CUDA 12 and current
llama.cpp CUDA builds, so they only run through the Vulkan backend. Matches
``VULKAN_ONLY_GPUS`` entries as case-insensitive substrings of the GPU name,
the same convention used by the bandwidth/compute-capability lookups.
"""
if gpu.vendor != "nvidia":
return False
name_upper = gpu.name.upper()
return any(marker.upper() in name_upper for marker in VULKAN_ONLY_GPUS)
def _fit_candidate_gpus(gpus: list[GPUInfo]) -> list[GPUInfo]:
has_dedicated_gpu = any(
not _uses_shared_system_pool(gpu) and gpu.vram_bytes > 0 for gpu in gpus
)
if not has_dedicated_gpu:
return gpus
return [gpu for gpu in gpus if not _uses_shared_system_pool(gpu)]
def _gpu_identity(gpu: GPUInfo) -> str:
name = gpu.name.lower().replace("(simulated)", "")
return " ".join(name.split())
def _is_homogeneous_gpu_set(gpus: list[GPUInfo], available: list[int]) -> bool:
if not gpus:
return True
first = gpus[0]
first_identity = _gpu_identity(first)
first_available = available[0]
vram_tolerance = max(256 * 1024**2, int(first_available * 0.02))
return all(
gpu.vendor == first.vendor
and _gpu_identity(gpu) == first_identity
and abs(gpu_available - first_available) <= vram_tolerance
for gpu, gpu_available in zip(gpus, available, strict=True)
)
def _multi_gpu_effective_vram(
gpus: list[GPUInfo],
available: list[int],
warnings: list[str],
) -> tuple[int, bool, int | None]:
raw_total = sum(available)
if len(gpus) <= 1:
return raw_total, False, None
if any(gpu.shared_memory or gpu.vendor == "apple" for gpu in gpus):
effective = max(available)
warnings.append(
"Multiple shared-memory GPUs are not pooled; using the largest "
"reported memory pool for fit checks"
)
return effective, False, None
homogeneous = _is_homogeneous_gpu_set(gpus, available)
utilization = (
_MULTI_GPU_HOMOGENEOUS_UTILIZATION
if homogeneous
else _MULTI_GPU_HETEROGENEOUS_UTILIZATION
)
overhead = min(raw_total, len(gpus) * _MULTI_GPU_FRAMEWORK_OVERHEAD_BYTES)
effective = int((raw_total - overhead) * utilization)
warnings.append(
"Multi-GPU fit uses a conservative layer-split budget: "
f"{effective / _GiB:.1f} GB effective from {raw_total / _GiB:.1f} GB raw VRAM"
)
if not homogeneous:
warnings.append(
"Heterogeneous multi-GPU setup: fit assumes uneven layer placement; "
"speed depends on backend split mode and interconnect"
)
return effective, True, effective
def check_compatibility(
model: ModelInfo,
variant: GGUFVariant | None,
hardware: HardwareInfo,
context_length: int = 4096,
) -> CompatibilityResult:
"""Check if a model+variant can run on the given hardware."""
warnings: list[str] = []
vram_required = estimate_vram(model, variant, context_length)
usable_ram = effective_usable_ram(hardware.ram_bytes, hardware.ram_budget_bytes)
# Determine best GPU
best_gpu: GPUInfo | None = None
best_gpu_available = 0
gpu_available_values: list[int] = []
candidate_gpus = _fit_candidate_gpus(hardware.gpus)
ram_budget_active = hardware.ram_budget_bytes is not None
for gpu in candidate_gpus:
gpu_available = _gpu_available_memory(
gpu, usable_ram, ram_budget_active=ram_budget_active
)
gpu_available_values.append(gpu_available)
if best_gpu is None or gpu_available > best_gpu_available:
best_gpu = gpu
best_gpu_available = gpu_available
vram_available = sum(gpu_available_values) if gpu_available_values else 0
fit_vram_available, uses_multi_gpu, multi_gpu_effective_vram = (
_multi_gpu_effective_vram(candidate_gpus, gpu_available_values, warnings)
)
if (
len(candidate_gpus) > 1
and not uses_multi_gpu
and any(gpu.shared_memory or gpu.vendor == "apple" for gpu in candidate_gpus)
):
vram_available = fit_vram_available
offload_ram_available = (
0
if best_gpu and (best_gpu.shared_memory or best_gpu.vendor == "apple")
else usable_ram
)
# Check compute capability for NVIDIA
if best_gpu and best_gpu.vendor == "nvidia" and best_gpu.compute_capability:
if best_gpu.compute_capability < MIN_COMPUTE_CAPABILITY_OLLAMA:
warnings.append(
f"Compute capability {best_gpu.compute_capability} is below "
f"minimum {MIN_COMPUTE_CAPABILITY_OLLAMA} for Ollama"
)
# Flag legacy Kepler GPUs that have no CUDA support in modern llama.cpp.
# They can still run, but only through the Vulkan backend on Linux.
if best_gpu and _is_vulkan_only_gpu(best_gpu):
warnings.append(
"Legacy Kepler GPU: no CUDA support in modern llama.cpp; "
"use the Vulkan backend (Linux) instead"
)
# Check ROCm for AMD. Windows AMD users can still use Vulkan/DirectML
# backends, so do not label the GPU path as unavailable there.
if (
best_gpu
and best_gpu.vendor == "amd"
and hardware.os not in ("linux", "windows")
):
warnings.append("ROCm requires Linux for AMD GPU inference")
# Check Metal for Apple
if best_gpu and best_gpu.vendor == "apple" and hardware.os != "darwin":
warnings.append("Metal requires macOS for Apple Silicon inference")
# Determine fit type
if fit_vram_available >= vram_required:
fit_type = "full_gpu"
can_run = True
offload_ratio = 0.0
elif (
fit_vram_available > 0
and (fit_vram_available + offload_ram_available) >= vram_required
):
fit_type = "partial_offload"
can_run = True
offload_ratio = (
(vram_required - fit_vram_available) / vram_required
if vram_required > 0
else 0.0
)
offload_pct = offload_ratio * 100
if best_gpu and (best_gpu.shared_memory or best_gpu.vendor == "apple"):
warnings.append("Will use shared system memory")
else:
warnings.append(
f"~{offload_pct:.0f}% of layers will be offloaded to CPU RAM"
)
elif usable_ram >= vram_required:
fit_type = "cpu_only"
can_run = True
offload_ratio = 0.0
warnings.append("Will run on CPU only (much slower)")
else:
fit_type = "cpu_only"
can_run = False
offload_ratio = 0.0
warnings.append("Insufficient memory (GPU VRAM + RAM) to run this model")
# Context length warning
context_fits = not (
model.context_length is not None and model.context_length < context_length
)
if not context_fits:
warnings.append(
f"Model max context {model.context_length} < requested "
f"{context_length}; runtime will truncate or reject"
)
elif (
context_length > 8192
and model.context_length
and model.context_length >= context_length
):
warnings.append(
f"Large context ({context_length}) increases VRAM usage significantly"
)
# File size vs disk space
file_size = estimate_weight_bytes(model, variant)
if hardware.disk_free_bytes > 0 and file_size > hardware.disk_free_bytes:
warnings.append("Insufficient disk space to download this model")
can_run = False
return CompatibilityResult(
model=model,
gguf_variant=variant,
can_run=can_run,
vram_required_bytes=vram_required,
vram_available_bytes=vram_available,
offload_ratio=offload_ratio,
uses_multi_gpu=uses_multi_gpu,
multi_gpu_effective_vram_bytes=multi_gpu_effective_vram,
warnings=warnings,
fit_type=fit_type,
context_fits=context_fits,
)
src/whichllm/engine/performance.py
+284
View File
@@ -0,0 +1,284 @@
"""Token generation speed estimation."""
from __future__ import annotations
from whichllm.engine.quantization import estimate_weight_bytes
from whichllm.engine.quantization import effective_quant_type
from whichllm.hardware.types import GPUInfo
from whichllm.models.types import GGUFVariant, ModelInfo
# Per-quant efficiency factors applied to the theoretical bandwidth-bound
# tok/s. These reflect empirical llama.cpp / vLLM measurements: 4-bit GGUFs
# achieve the highest fraction of memory-bandwidth-limited theoretical
# throughput because the dequantization kernel is fast and weight reads
# dominate; 8-bit and FP16 drop because more compute is required per byte.
_QUANT_EFFICIENCY: dict[str, float] = {
"F32": 0.30,
"F16": 0.40,
"BF16": 0.40,
"Q8_0": 0.45,
"Q6_K": 0.50,
"Q5_K_M": 0.52,
"Q5_K_S": 0.52,
"Q5_0": 0.50,
"Q4_K_M": 0.55,
"Q4_K_S": 0.55,
"Q4_0": 0.53,
# 4-bit microscaling floats decode through native FP4 tensor-core paths
# (e.g. Blackwell) where weight reads dominate, so they land in the same
# high-efficiency band as the best 4-bit GGUF kernels.
"NVFP4": 0.56,
"MXFP4": 0.55,
"Q3_K_M": 0.50,
"Q3_K_S": 0.48,
"Q3_K_L": 0.50,
"Q2_K": 0.45,
"IQ4_XS": 0.52,
"IQ4_NL": 0.50,
"IQ3_S": 0.45,
"IQ3_M": 0.45,
"IQ3_XS": 0.45,
"IQ3_XXS": 0.42,
"IQ2_S": 0.40,
"IQ2_M": 0.40,
"IQ2_XXS": 0.38,
"IQ1_M": 0.35,
"IQ1_S": 0.35,
"Q2_0": 0.38,
"Q1_0": 0.32,
"TQ2_0": 0.35,
"TQ1_0": 0.32,
}
_DEFAULT_QUANT_EFFICIENCY = 0.45
# Vendor / backend multiplier applied on top of quant efficiency. CUDA on
# modern data-center GPUs is the reference (1.0); Apple's Metal kernel is
# behind on dequantization; ROCm trails further; older CUDA generations
# also drop.
_BACKEND_FACTOR: dict[str, float] = {
"nvidia": 1.00,
"amd": 0.78,
"apple": 0.82,
"intel": 0.65,
}
# MoE decode is partly bandwidth-bound and partly kernel/dispatch-bound.
# The old fixed 25% read floor matched high-bandwidth CUDA cards reasonably
# well, but badly under-estimated low-bandwidth unified-memory APUs such as
# Strix Halo where the active expert reads dominate. Model this as a floor
# that rises with bandwidth: ~5% at 256 GB/s, capped at the legacy 25%.
_MOE_REFERENCE_BANDWIDTH_GBPS = 256.0
_MOE_MIN_READ_RATIO_AT_REFERENCE = 0.05
_MOE_MAX_READ_RATIO_FLOOR = 0.25
_SPEED_CONFIDENCE_RANGE_FACTORS: dict[str, tuple[float, float]] = {
"high": (0.85, 1.20),
"medium": (0.60, 1.60),
"low": (0.35, 2.00),
}
_SPEED_CONFIDENCE_ORDER = {
"low": 0,
"medium": 1,
"high": 2,
}
def _backend_factor(gpu: GPUInfo) -> float:
if gpu.vendor in _BACKEND_FACTOR:
return _BACKEND_FACTOR[gpu.vendor]
return 0.7
def _quant_efficiency(model: ModelInfo, variant: GGUFVariant | None) -> float:
quant = effective_quant_type(model, variant)
if not quant:
return _DEFAULT_QUANT_EFFICIENCY
return _QUANT_EFFICIENCY.get(quant.upper(), _DEFAULT_QUANT_EFFICIENCY)
def _moe_effective_read_ratio(model: ModelInfo, gpu: GPUInfo) -> float:
"""Return fraction of stored weights read per generated token for MoE."""
if not model.is_moe or not model.parameter_count_active:
return 1.0
if model.parameter_count <= 0:
return 1.0
active_ratio = model.parameter_count_active / model.parameter_count
if active_ratio <= 0:
return 1.0
bandwidth = gpu.memory_bandwidth_gbps or 0.0
if bandwidth > 0:
floor = _MOE_MIN_READ_RATIO_AT_REFERENCE * max(
1.0, bandwidth / _MOE_REFERENCE_BANDWIDTH_GBPS
)
else:
floor = _MOE_MAX_READ_RATIO_FLOOR
floor = min(_MOE_MAX_READ_RATIO_FLOOR, floor)
return min(1.0, max(active_ratio, floor))
def _lower_speed_confidence(current: str, candidate: str) -> str:
if _SPEED_CONFIDENCE_ORDER[candidate] < _SPEED_CONFIDENCE_ORDER[current]:
return candidate
return current
def _looks_synthetic_gguf(model: ModelInfo, variant: GGUFVariant | None) -> bool:
if variant is None:
return False
if not variant.filename:
return False
expected = f"{model.name}.{variant.quant_type}.gguf"
return variant.filename == expected
def estimate_speed_uncertainty(
model: ModelInfo,
variant: GGUFVariant | None,
gpu: GPUInfo | None,
fit_type: str,
estimated_tok_per_sec: float | None,
) -> tuple[str, tuple[float, float] | None, list[str]]:
"""Return confidence metadata for the speed point estimate.
The tok/s estimator is intentionally hardware/model-metadata based; it
does not know the user's exact llama.cpp, Vulkan, ROCm, Metal, MLX, or
runtime kernel versions. This helper keeps that uncertainty visible
without mixing it into the ranking score itself.
"""
notes = [
"Speed is estimated from memory bandwidth, quantization, backend, and fit type."
]
confidence = "medium"
if estimated_tok_per_sec is None or estimated_tok_per_sec <= 0:
return (
"low",
None,
notes + ["No usable bandwidth estimate was available for this setup."],
)
if gpu is None or fit_type == "cpu_only":
confidence = "low"
notes.append(
"CPU-only speed varies heavily with memory channels and BLAS/kernel path."
)
else:
if not gpu.memory_bandwidth_gbps:
confidence = "low"
notes.append(
"GPU memory bandwidth is unknown, so speed is especially uncertain."
)
if fit_type == "partial_offload":
confidence = "low"
if gpu.vendor == "apple" or gpu.shared_memory:
notes.append(
"Partial offload on unified memory is backend-sensitive but avoids a PCIe cliff."
)
else:
notes.append(
"Partial offload on a discrete GPU depends strongly on PCIe and CPU RAM bandwidth."
)
if model.is_moe:
notes.append(
"MoE speed uses active parameters plus a bandwidth-scaled dispatch/read floor."
)
if gpu.vendor == "apple":
confidence = _lower_speed_confidence(confidence, "low")
notes.append(
"Apple Silicon MoE throughput is especially sensitive to Metal/MLX runtime kernels."
)
elif gpu.vendor == "amd" and gpu.shared_memory:
confidence = _lower_speed_confidence(confidence, "medium")
notes.append(
"AMD shared-memory APU estimates are calibrated by bandwidth, but ROCm/Vulkan kernels can differ."
)
if _looks_synthetic_gguf(model, variant):
confidence = _lower_speed_confidence(confidence, "medium")
notes.append(
"This is a synthetic GGUF estimate for an official repo, not a measured GGUF file."
)
low_factor, high_factor = _SPEED_CONFIDENCE_RANGE_FACTORS[confidence]
speed_range = (
round(estimated_tok_per_sec * low_factor, 1),
round(estimated_tok_per_sec * high_factor, 1),
)
return confidence, speed_range, notes
def estimate_tok_per_sec(
model: ModelInfo,
variant: GGUFVariant | None,
gpu: GPUInfo | None,
fit_type: str = "full_gpu",
) -> float:
"""Estimate tokens per second for inference.
Model: throughput is bounded by the time it takes to read all weights
needed per token, multiplied by quant- and backend-specific efficiency
factors. The default 0.5 efficiency factor used earlier mixed two
distinct losses (compute kernel quality and offload overhead) into one
constant — this version separates them so a Q4_K_M model on CUDA scores
differently from the same model running on Metal or with partial
offload.
"""
if gpu is None or fit_type == "cpu_only":
params_b = model.parameter_count / 1e9
if model.is_moe and model.parameter_count_active:
params_b = model.parameter_count_active / 1e9
if params_b <= 0:
return 0.0
# Modern desktop CPUs sustain roughly 4-8 GB/s effective for the
# bandwidth-bound dequant+matmul loop on a single socket. Quantized
# 4-bit 7B → ~3.5 GB → ~1-2 tok/s. Approximate with an inverse-size
# heuristic that gets the right order of magnitude.
quant_factor = _quant_efficiency(model, variant) / _DEFAULT_QUANT_EFFICIENCY
return max(0.3, 18.0 / max(params_b, 0.5) * quant_factor)
model_size = estimate_weight_bytes(model, variant)
# MoE: use a speed-specific effective read ratio. VRAM fit still uses
# total stored weights elsewhere; this only estimates per-token reads.
if model.is_moe and model.parameter_count_active:
effective_read = model_size * _moe_effective_read_ratio(model, gpu)
else:
effective_read = model_size
bandwidth = gpu.memory_bandwidth_gbps * 1e9 if gpu.memory_bandwidth_gbps else 0
if bandwidth == 0:
return 0.0
theoretical = bandwidth / effective_read
# Real-world efficiency depends on quant kernel and backend.
efficiency = _quant_efficiency(model, variant) * _backend_factor(gpu)
# Partial offload penalty depends on the memory architecture:
#
# - Discrete GPU (NVIDIA/AMD/Intel): spilled weights live in CPU RAM
# and are read across PCIe at ~1/10th of VRAM bandwidth. With ~40%
# of the model offloaded the blended throughput lands near 0.45x.
# - Apple Silicon: GPU and CPU share one physical unified-memory pool.
# AMD shared-memory APUs such as Strix Halo have the same no-PCIe-cliff
# shape for model weights, even though their backend factor remains AMD.
# "Exceeding VRAM" only means exceeding the recommended working set;
# the bytes are still read from the same high-bandwidth unified RAM,
# so there is no PCIe cliff — only mild OS/cache contention. Using
# the discrete 0.45x here was the bug that made DeepSeek-R1-class
# models on M2/M3 Ultra report ~1.7 t/s when real-world is 4-15.
if fit_type == "partial_offload":
if gpu.vendor == "apple" or gpu.shared_memory:
efficiency *= 0.85
else:
efficiency *= 0.45
return theoretical * efficiency
src/whichllm/engine/quantization.py
+82
View File
@@ -0,0 +1,82 @@
"""Quantization helpers shared across ranking and estimators."""
from __future__ import annotations
import re
from whichllm.constants import QUANT_QUALITY_PENALTY
from whichllm.models.types import GGUFVariant, ModelInfo
# GGUFでないリポジトリ名から量子化方式を推定する
_NON_GGUF_PATTERNS: list[tuple[str, str]] = [
(r"(^|[-_/])awq($|[-_/])", "AWQ"),
(r"(^|[-_/])gptq($|[-_/])", "GPTQ"),
# 4-bit microscaling float formats. Anchored so they only match a distinct
# repo-name token, never a substring of an unrelated id.
(r"(^|[-_/])mxfp4($|[-_/])", "MXFP4"),
(r"(^|[-_/])nvfp4($|[-_/])", "NVFP4"),
(r"(bnb[-_/]?4bit|nf4|int4|4bit)", "BNB_4BIT"),
(r"(int8|8bit)", "INT8"),
(r"(^|[-_/])fp8($|[-_/])", "FP8"),
(r"(^|[-_/])bf16($|[-_/])", "BF16"),
(r"(^|[-_/])(fp16|f16)($|[-_/])", "FP16"),
]
# GGUF以外の簡易推定: 重み1つあたりのバイト数
_NON_GGUF_BYTES_PER_WEIGHT: dict[str, float] = {
"AWQ": 0.5,
"GPTQ": 0.5,
"BNB_4BIT": 0.5,
"MXFP4": 0.53125,
"NVFP4": 0.5625,
"INT8": 1.0,
"FP8": 1.0,
"BF16": 2.0,
"FP16": 2.0,
}
# GGUF以外の簡易推定: 品質低下率
_NON_GGUF_QUALITY_PENALTY: dict[str, float] = {
"AWQ": 0.05,
"GPTQ": 0.05,
"BNB_4BIT": 0.07,
"MXFP4": 0.06,
"NVFP4": 0.05,
"INT8": 0.02,
"FP8": 0.02,
"BF16": 0.0,
"FP16": 0.0,
}
def infer_non_gguf_quant_type(model_id: str) -> str:
"""Infer non-GGUF quantization type from a model repo ID."""
lower = model_id.lower()
for pattern, quant_type in _NON_GGUF_PATTERNS:
if re.search(pattern, lower):
return quant_type
return "FP16"
def effective_quant_type(model: ModelInfo, variant: GGUFVariant | None) -> str:
"""Return effective quantization type for a model+variant pair."""
if variant:
return variant.quant_type.upper()
return infer_non_gguf_quant_type(model.id)
def estimate_weight_bytes(model: ModelInfo, variant: GGUFVariant | None) -> int:
"""Estimate model weight size in bytes."""
if variant:
return variant.file_size_bytes
quant_type = infer_non_gguf_quant_type(model.id)
bytes_per_weight = _NON_GGUF_BYTES_PER_WEIGHT.get(quant_type, 2.0)
return int(model.parameter_count * bytes_per_weight)
def quant_quality_penalty(model: ModelInfo, variant: GGUFVariant | None) -> float:
"""Return quality penalty fraction for a quantization format."""
quant_type = effective_quant_type(model, variant).upper()
if quant_type in QUANT_QUALITY_PENALTY:
return QUANT_QUALITY_PENALTY[quant_type]
return _NON_GGUF_QUALITY_PENALTY.get(quant_type, 0.05)
src/whichllm/engine/ranker.py
+885
View File
@@ -0,0 +1,885 @@
"""Model ranking: score and select the best models for the user's hardware."""
from __future__ import annotations
import math
import re
from whichllm.constants import (
MODEL_GENERATION_BONUS_MAX,
MODEL_GENERATION_PENALTY_MAX,
MODEL_LINEAGE_VERSIONS,
QUANT_BYTES_PER_WEIGHT,
QUANT_PREFERENCE_ORDER,
)
from whichllm.engine.compatibility import check_compatibility
from whichllm.engine.performance import estimate_speed_uncertainty, estimate_tok_per_sec
from whichllm.engine.quantization import effective_quant_type, quant_quality_penalty
from whichllm.engine.types import CompatibilityResult
from whichllm.hardware.types import HardwareInfo
from whichllm.models.benchmark import (
BenchmarkEvidence,
build_line_bucket_index,
build_score_index,
lookup_benchmark_evidence,
)
from whichllm.models.types import GGUFVariant, ModelInfo
# Pre-compile lineage regex tables once at import time.
_LINEAGE_REGEX: dict[str, list[tuple[re.Pattern[str], int]]] = {
family: [(re.compile(pat), idx) for pat, idx in entries]
for family, entries in MODEL_LINEAGE_VERSIONS.items()
}
_LINEAGE_FAMILY_MAX: dict[str, int] = {
family: max(idx for _, idx in entries) for family, entries in _LINEAGE_REGEX.items()
}
_MULTI_GPU_SPEED_FACTOR = 0.70
def _family_selection_key(
result: CompatibilityResult,
require_direct_top: bool,
) -> tuple[float]:
"""Family-level selection key — single composite score.
``quality_score`` already includes the runtime fit penalty and speed
adjustment. Keep final selection close to that displayed score so strong
partial-offload candidates do not get discounted again while sorting.
- ``direct_bonus`` (+5) gives independent leaderboard evidence a
small edge at the same fit; cannot overturn a 6+ point quality gap
"""
if require_direct_top and result.benchmark_status == "direct":
direct_bonus = 5.0
else:
direct_bonus = 0.0
cpu_penalty = -6.0 if result.fit_type == "cpu_only" else 0.0
ctx_penalty = -20.0 if not result.context_fits else 0.0
return (result.quality_score + direct_bonus + cpu_penalty + ctx_penalty,)
def _partial_offload_quality_factor(model: ModelInfo, offload_ratio: float) -> float:
"""Discount partial-offload candidates by how much leaves VRAM."""
ratio = max(0.0, min(1.0, offload_ratio))
if ratio >= 0.75:
factor = 0.42
elif ratio >= 0.60:
factor = 0.52
elif ratio >= 0.40:
factor = 0.62
elif ratio >= 0.25:
factor = 0.76
else:
factor = 0.86
# MoE offload is more nuanced: inactive experts and router/runtime
# placement do not hurt equally. If the GPU can plausibly hold the
# active expert working set, do not treat inactive-expert spill like
# dense-layer spill.
if model.is_moe and model.parameter_count_active:
active_ratio = (
model.parameter_count_active / model.parameter_count
if model.parameter_count > 0
else 1.0
)
active_ratio = max(0.0, min(1.0, active_ratio))
active_set_fits = ratio <= max(0.0, 1.0 - active_ratio)
if active_set_fits:
if ratio >= 0.75:
factor = max(factor, 0.66)
elif ratio >= 0.60:
factor = max(factor, 0.70)
elif ratio >= 0.40:
factor = max(factor, 0.76)
elif ratio >= 0.25:
factor = max(factor, 0.82)
else:
factor = max(factor, 0.88)
else:
factor = min(0.76, factor + 0.08)
return factor
# Per-source benchmark weight applied to the raw 0-100 score before it is
# combined with size, quant penalty, etc. The widest gap is between "direct"
# (independent leaderboard) and "self_reported" (uploader card claim).
_SOURCE_WEIGHTS: dict[str, float] = {
"direct": 0.62,
"base_model": 0.55,
"variant": 0.50,
"line_interp": 0.40,
"self_reported": 0.30,
"none": 0.0,
}
_SYNTHETIC_QUANTS = ("Q3_K_M", "Q4_K_M", "Q5_K_M", "Q6_K", "Q8_0")
_PREQUANTIZED_REPO_RE = re.compile(
r"-(awq|gptq|bnb|fp8|fp16|bf16|mxfp4|nvfp4|int4|int8|4bit|8bit|gguf)$",
re.IGNORECASE,
)
def _synthesize_variants_for_official_repo(
model: ModelInfo, quant_filter_upper: str | None
) -> list[GGUFVariant]:
"""Return synthetic GGUF variants for popular safetensors-only repos.
HuggingFace doesn't always index GGUF siblings for an official model
(e.g. ``Qwen/Qwen3.6-27B`` ships only safetensors), but bartowski /
lmstudio-community / QuantFactory invariably publish Q4_K_M and Q8_0
conversions within a day of release. Without synthetic variants, we'd
score these models at BF16 file sizes (~2x larger than realistic), which
forces a partial_offload penalty on otherwise-runnable mid-size models.
Skips repos that already advertise a specific quantization in their name
(``...-AWQ``, ``...-GPTQ``, ``...-FP8`` etc.) — those are non-GGUF formats
and synthesizing a Q4_K_M alternative would misrepresent what the repo
actually contains.
"""
org = model.id.split("/", 1)[0] if "/" in model.id else ""
if org not in _OFFICIAL_ORGS:
return []
if _PREQUANTIZED_REPO_RE.search(model.id):
return []
out: list[GGUFVariant] = []
for quant in _SYNTHETIC_QUANTS:
if quant_filter_upper and quant != quant_filter_upper:
continue
bpw = QUANT_BYTES_PER_WEIGHT.get(quant, 0.5625)
out.append(
GGUFVariant(
filename=f"{model.name}.{quant}.gguf",
quant_type=quant,
file_size_bytes=int(model.parameter_count * bpw),
)
)
return out
def _iter_candidate_variants(
model: ModelInfo,
quant_filter: str | None = None,
) -> list[GGUFVariant | None]:
"""Build candidate variants to evaluate for a model."""
quant_filter_upper = quant_filter.upper() if quant_filter else None
if not model.gguf_variants:
synthetic = _synthesize_variants_for_official_repo(model, quant_filter_upper)
if synthetic:
return list(synthetic)
quant_type = effective_quant_type(model, None)
if quant_filter_upper and quant_type != quant_filter_upper:
return []
return [None]
# Filter by quant type if specified
candidates: list[GGUFVariant] = model.gguf_variants
if quant_filter_upper:
candidates = [
v for v in candidates if v.quant_type.upper() == quant_filter_upper
]
if not candidates:
return []
else:
# Sub-3-bit GGUFs lose 25-60% of model quality and rarely produce
# a meaningfully better candidate than a smaller model at Q4_K_M.
# Exclude them unless explicitly requested via --quant.
_EXTREME_QUANTS = {
"Q2_K",
"Q2_0",
"Q1_0",
"TQ2_0",
"TQ1_0",
"IQ3_XXS",
"IQ2_XXS",
"IQ2_S",
"IQ2_M",
"IQ1_M",
"IQ1_S",
}
filtered = [
v for v in candidates if v.quant_type.upper() not in _EXTREME_QUANTS
]
if filtered:
candidates = filtered
# Sort by preference order
def variant_sort_key(v: GGUFVariant) -> int:
try:
return QUANT_PREFERENCE_ORDER.index(v.quant_type.upper())
except ValueError:
return len(QUANT_PREFERENCE_ORDER)
candidates = sorted(candidates, key=variant_sort_key)
return list(candidates)
_OFFICIAL_ORGS = frozenset(
{
"Qwen",
"meta-llama",
"google",
"mistralai",
"deepseek-ai",
"microsoft",
"nvidia",
"01-ai",
"tiiuae",
"apple",
"CohereForAI",
"bigcode",
# 2025+ frontier open-weights labs that publish safetensors-only
# repos which the community immediately converts to GGUF.
"openai",
"zai-org",
"moonshotai",
"MiniMaxAI",
"XiaomiMiMo",
"allenai",
"ibm-granite",
"stepfun-ai",
}
)
# Trusted GGUF converters — format converters that don't change model quality
_TRUSTED_CONVERTERS = frozenset(
{
"bartowski",
"lmstudio-community",
"QuantFactory",
"unsloth",
"ggml-org",
"Mungert",
}
)
# Known repackagers — typically reupload others' models without added value
_REPACKAGER_ORGS = frozenset(
{
"MaziyarPanahi",
"TheBloke",
"SanctumAI",
"solidrust",
"mradermacher",
}
)
# Orgs whose repositories ship CI fixtures, deprecated research artifacts, or
# debug binaries that are not viable production LLMs. Exclude them outright so
# they cannot occupy ranking slots regardless of download counts.
_EXCLUDED_ORGS = frozenset(
{
"openai-community", # gpt2 family, 2019 research
"distilbert", # distilgpt2 etc.
"facebook", # opt-125m research scaffolds
"EleutherAI", # pythia/gpt-neo research
"trl-internal-testing", # TRL CI fixtures
"hmellor", # random tiny test models
"HuggingFaceH4", # often staging / fixtures
"transformersbook",
"togethercomputer", # mostly inference endpoints, no GGUFs
}
)
# Substring patterns in *names* that strongly suggest non-production usage.
_EXCLUDED_NAME_PATTERNS = (
"tiny-",
"-tiny",
"tiny_",
"_tiny",
"test-only",
"debug-",
"playground",
"-fixture",
"for-testing",
"tiny-random",
"ci-",
)
# Naming patterns that indicate a fine-tune / merge / "uncensoring" derivative
# of a real base model. These derivatives inherit the base model's benchmark
# score via line_interp, but the derivative itself is rarely benchmarked
# independently and frequently degrades quality. Apply a soft score penalty
# rather than full exclusion so they can still surface when nothing better is
# available.
_DUBIOUS_DERIVATIVE_PATTERNS = (
"heretic",
"abliterat",
"uncensored",
"obliterat",
"abliter",
"horror",
"erotic",
"nsfw",
"rp-",
"-rp",
"roleplay",
"darkidol",
"darkforest",
"tiefigh",
"smaug",
"personalityengine",
"lexi",
"violence",
"violet",
"schizo",
"dark-",
"twilight",
"celeste",
"midnight-rose",
"moistral",
"stheno",
"fimbulvetr",
"wizard-vicuna",
"kunoichi",
)
def _derivative_name_penalty(model_id: str) -> float:
"""Return a score penalty (in raw quality points) for fine-tune /
"uncensored" / merge derivatives that ride on a real base model's
benchmark line. The penalty is gentle (≤ 12pt) so a derivative can
still win when its size class has no better option.
"""
if not model_id:
return 0.0
lower = model_id.lower()
name = lower.split("/", 1)[1] if "/" in lower else lower
for pat in _DUBIOUS_DERIVATIVE_PATTERNS:
if pat in name:
return -10.0
return 0.0
def _is_excluded_model(model_id: str) -> bool:
"""Return True for CI/research/fixture models that should never rank."""
if not model_id:
return True
org = model_id.split("/", 1)[0] if "/" in model_id else ""
if org in _EXCLUDED_ORGS:
return True
lower = model_id.lower()
name = lower.split("/", 1)[1] if "/" in lower else lower
for pat in _EXCLUDED_NAME_PATTERNS:
if pat in name:
return True
return False
def _generation_bonus(model_id: str) -> float:
"""Return a small additive bonus reflecting how new a model's
generation is within its family. The newest version of each
recognized family gets +MODEL_GENERATION_BONUS_MAX. Older
versions get a smaller bonus (or a small penalty for the
legacy generation). Unknown families return 0.
This is purely an additive correction to the quality score
and is small enough that strong benchmark evidence will still
dominate.
"""
if not model_id:
return 0.0
lower = model_id.lower()
best_bonus = 0.0
for family, patterns in _LINEAGE_REGEX.items():
for regex, idx in patterns:
if regex.search(lower):
top = _LINEAGE_FAMILY_MAX[family]
if top <= 1:
contribution = 0.0
else:
# Map oldest -> -PENALTY_MAX, newest -> +BONUS_MAX.
norm = (idx - 1) / (top - 1) # 0 .. 1
span = MODEL_GENERATION_BONUS_MAX + MODEL_GENERATION_PENALTY_MAX
contribution = norm * span - MODEL_GENERATION_PENALTY_MAX
if abs(contribution) > abs(best_bonus):
best_bonus = contribution
break # first match wins for this family
return best_bonus
def _detect_specializations(model_id: str) -> set[str]:
"""モデルIDから用途特化タグを検出する。"""
lower = model_id.lower()
tags: set[str] = set()
if re.search(r"(coder|codegen|starcoder|program|coding)", lower):
tags.add("coding")
if re.search(r"(^|[-_/])(vl|vision|multimodal|llava|image)([-_/]|$)", lower):
tags.add("vision")
if re.search(r"(^|[-_/])math([-_/]|$)", lower):
tags.add("math")
return tags
def _matches_profile(model: ModelInfo, task_profile: str) -> bool:
"""指定プロファイルにモデルが合致するか判定する。"""
profile = task_profile.lower()
tags = _detect_specializations(model.id)
if profile == "any":
return True
if profile == "general":
return len(tags) == 0
return profile in tags
def _effective_params_b(model: ModelInfo) -> float:
"""Return effective parameter size in billions."""
if model.is_moe and model.parameter_count_active:
return model.parameter_count_active / 1e9
return model.parameter_count / 1e9
def _knowledge_capacity_b(model: ModelInfo) -> float:
"""Return the knowledge capacity in billions for size filtering.
For dense models this is the parameter count. For MoE models, total
parameters (all expert weights live in VRAM and contribute to the
knowledge encoded in the model) is the right yardstick — ``min_params``
is asking "how much does this model know?" not "how much does it
compute per token". Using *active* params here was the bug that hid
Qwen3-Next-80B-A3B from the H100 ranking — its 3B active was below the
12B auto-floor for 30GB+ GPUs even though its 80B total clearly fits.
"""
return model.parameter_count / 1e9
def _passes_evidence_filter(source: str, evidence_filter: str) -> bool:
"""判定根拠フィルタに合致するかを返す。"""
mode = evidence_filter.lower()
if mode == "strict":
return source == "direct"
if mode == "base":
return source in {"direct", "variant", "base_model"}
return True
def _is_gguf_only_backend(hardware: HardwareInfo) -> bool:
"""実行基盤の都合でGGUFのみを許可すべきか判定する。"""
# Apple Silicon(macOS/Metal)とCPU-onlyは、実運用の安定性を優先してGGUFに限定する。
if hardware.os == "darwin":
return True
if not hardware.gpus:
return True
# Linux + NVIDIA (CUDA) は AWQ/GPTQ 含む非GGUFも許可する。
has_linux_nvidia = hardware.os == "linux" and any(
g.vendor == "nvidia" for g in hardware.gpus
)
return not has_linux_nvidia
def _compute_quality_score(
model: ModelInfo,
variant: GGUFVariant | None,
tok_per_sec: float,
fit_type: str,
offload_ratio: float = 0.0,
family_downloads: int = 0,
family_likes: int = 0,
benchmark_avg: float | None = None,
benchmark_source: str = "none",
) -> float:
"""Compute a quality score (0-100) for ranking.
Factors:
- Benchmark score weighted by source tier
- Model size (log scale)
- Quantization penalty
- Fit type penalty (partial offload / CPU-only heavily penalized)
- Speed bonus / penalty (practical usability)
- Popularity (downloads/likes) as soft tie-breaker
- Official org bonus (vs known repackagers)
- Generation-lineage bonus (newest family member > legacy generation)
"""
params_b = model.parameter_count / 1e9
if model.is_moe and model.parameter_count_active:
effective_b = model.parameter_count_active / 1e9
else:
effective_b = params_b
if effective_b <= 0:
return 0.0
# Benchmarks lead, but raw model size also matters: a 70B at Q4_K_M
# carries far more world knowledge than a 7B Q4_K_M even when the
# leaderboard score gap is modest. For MoE models, knowledge capacity
# tracks *total* params (every expert contributes to what the model
# knows), while routing keeps per-token compute small. Use total params
# for the size score and let the speed term separately reward MoE
# efficiency.
size_basis_b = params_b
size_score = 4.2 * math.log2(max(size_basis_b, 0.5)) + 9
size_score = min(size_score, 35)
has_benchmark = benchmark_avg is not None and benchmark_avg > 0
is_direct = benchmark_source == "direct"
is_self_reported = benchmark_source == "self_reported"
is_inherited = benchmark_source in {"variant", "base_model", "line_interp"}
bench_weight = _SOURCE_WEIGHTS.get(benchmark_source, 0.0)
benchmark_score = 0.0
if has_benchmark:
raw = min(100.0, benchmark_avg)
benchmark_score = raw * bench_weight
# Quantization penalty
quant_penalty = quant_quality_penalty(model, variant)
quality_core = (benchmark_score + size_score) * (1 - quant_penalty)
# Weak / unverifiable evidence gets an extra discount.
if not has_benchmark:
quality_core *= 0.55
elif is_self_reported:
quality_core *= 0.55 # uploader claim, easily fabricated
elif is_inherited:
quality_core *= 0.78
# Runtime form factor penalty
if fit_type == "partial_offload":
quality_core *= _partial_offload_quality_factor(model, offload_ratio)
elif fit_type == "cpu_only":
quality_core *= 0.50
# Speed acts as a usability gate rather than a ranking primary.
required_speed = (
8.0
if fit_type == "full_gpu"
else (4.0 if fit_type == "partial_offload" else 1.5)
)
if tok_per_sec > 0:
if tok_per_sec < required_speed:
speed_score = -8.0 * (1 - (tok_per_sec / required_speed))
else:
speed_score = min(8.0, math.log2(tok_per_sec / required_speed + 1.0) * 3.2)
else:
if fit_type == "partial_offload":
if offload_ratio >= 0.70:
speed_score = -24.0
elif offload_ratio >= 0.40:
speed_score = -18.0
else:
speed_score = -12.0
else:
speed_score = -8.0
# Popularity is a tie-breaker, never primary.
downloads = max(model.downloads, family_downloads)
likes = max(model.likes, family_likes)
pop_score_raw = 0.0
if downloads > 0:
pop_score_raw += min(1.0, math.log10(max(downloads, 1)) / 6 * 1.0)
if likes > 0:
pop_score_raw += min(1.0, math.log10(max(likes, 1)) / 4 * 1.0)
if is_direct:
pop_weight = 0.0
elif is_self_reported:
pop_weight = 0.4 # uploader claim is weak — popularity acts as sanity check
elif has_benchmark:
pop_weight = 0.2
else:
pop_weight = 0.6
pop_score = pop_score_raw * pop_weight
# Source-trust bonus stays small.
source_bonus_raw = 0.0
org = model.id.split("/")[0] if "/" in model.id else ""
if org in _OFFICIAL_ORGS:
source_bonus_raw = 5.0
elif org in _REPACKAGER_ORGS:
source_bonus_raw = -5.0
elif model.base_model:
base_org = model.base_model.split("/")[0] if "/" in model.base_model else ""
if base_org in _OFFICIAL_ORGS:
if org in _TRUSTED_CONVERTERS:
source_bonus_raw = 5.0
else:
source_bonus_raw = 0.0
if is_direct:
source_weight = 0.2
elif is_self_reported:
source_weight = 0.5
elif has_benchmark:
source_weight = 0.4
else:
source_weight = 0.6
source_bonus = source_bonus_raw * source_weight
# Generation lineage bonus: newest in a known family gets a small boost,
# confirmed legacy versions get a small penalty. Helps surface Qwen3.6,
# DeepSeek V4, Gemma 4, etc. against accumulated download leaders.
gen_bonus = _generation_bonus(model.id)
# When benchmark evidence is missing or self-reported, the lineage signal
# carries more weight (we have less else to go on).
if not has_benchmark or is_self_reported:
gen_bonus *= 1.5
elif is_direct:
gen_bonus *= 0.6
# Penalty for "uncensored / abliterated / heretic / RP" derivatives that
# ride on a base model's score without independent benchmarking.
derivative_penalty = _derivative_name_penalty(model.id)
return max(
0.0,
min(
100.0,
quality_core
+ speed_score
+ pop_score
+ source_bonus
+ gen_bonus
+ derivative_penalty,
),
)
def rank_models(
models: list[ModelInfo],
hardware: HardwareInfo,
context_length: int = 4096,
top_n: int = 10,
quant_filter: str | None = None,
min_speed: float | None = None,
benchmark_scores: dict[str, float] | None = None,
task_profile: str = "general",
require_direct_top: bool = True,
min_params_b: float | None = None,
evidence_filter: str = "any",
fit_filter: str = "any",
) -> list[CompatibilityResult]:
"""Rank models by quality for the given hardware. Returns top N results."""
results: list[CompatibilityResult] = []
gguf_only_backend = _is_gguf_only_backend(hardware)
# Pre-compute max downloads/likes per family so GGUF converters
# inherit popularity from the official base model
family_max_downloads: dict[str, int] = {}
family_max_likes: dict[str, int] = {}
# Track the parameter count of the family's dominant member (highest
# downloads). Used to detect quasi-fork uploads whose params differ
# drastically from the family proper (e.g. a 6.6B MTP-head extracted
# from a 158B base ending up tagged with the same family_id).
family_dominant_params: dict[str, int] = {}
family_dominant_downloads: dict[str, int] = {}
for m in models:
fid = m.family_id
family_max_downloads[fid] = max(family_max_downloads.get(fid, 0), m.downloads)
family_max_likes[fid] = max(family_max_likes.get(fid, 0), m.likes)
if m.parameter_count and m.downloads >= family_dominant_downloads.get(fid, -1):
family_dominant_downloads[fid] = m.downloads
family_dominant_params[fid] = m.parameter_count
# Deduplicate by family: pick best variant per family
seen_families: set[str] = set()
# Sort models by downloads (popular first) to process best candidates first
sorted_models = sorted(models, key=lambda m: m.downloads, reverse=True)
# Build benchmark indices once (case-insensitive + model line)
if benchmark_scores:
bench_ci_index, bench_line_index = build_score_index(benchmark_scores)
bench_line_buckets = build_line_bucket_index(benchmark_scores)
else:
bench_ci_index, bench_line_index = {}, {}
bench_line_buckets = {}
best_gpu = None
for gpu in hardware.gpus:
if best_gpu is None or gpu.vram_bytes > best_gpu.vram_bytes:
best_gpu = gpu
for model in sorted_models:
if _is_excluded_model(model.id):
continue
if not _matches_profile(model, task_profile):
continue
if min_params_b is not None and _knowledge_capacity_b(model) < min_params_b:
continue
candidates = _iter_candidate_variants(model, quant_filter)
if not candidates:
continue
fid = model.family_id
# Uploader-reported evalResults are only ever last-resort evidence.
self_reported = None
if isinstance(model.benchmark_scores, dict):
v = model.benchmark_scores.get("hf_eval")
if isinstance(v, (int, float)) and v > 0:
self_reported = float(v)
bench_evidence = BenchmarkEvidence(score=None, confidence=0.0, source="none")
if benchmark_scores or self_reported is not None:
actual_params_b = (
(model.parameter_count or 0) / 1e9 if model.parameter_count else None
)
bench_evidence = lookup_benchmark_evidence(
model.id,
model.base_model,
benchmark_scores or {},
ci_index=bench_ci_index,
line_index=bench_line_index,
line_bucket_index=bench_line_buckets,
self_reported_score=self_reported,
actual_params_b=actual_params_b,
)
# Family-size sanity check: if this model inherited benchmarks
# via family/base_model lookup but its own params disagree
# sharply with the family's dominant member, reject the
# inheritance. Catches MTP heads / draft / abliterated forks
# that share a family_id with their base but are effectively
# different models.
if bench_evidence.source in ("variant", "base_model", "line_interp"):
dom_params = family_dominant_params.get(model.family_id)
if dom_params and model.parameter_count and dom_params > 0:
ratio = model.parameter_count / dom_params
if ratio < 0.5 or ratio > 2.0:
bench_evidence = BenchmarkEvidence(
score=None, confidence=0.0, source="none"
)
if not _passes_evidence_filter(bench_evidence.source, evidence_filter):
continue
# 各variantを評価し、そのモデルで最もスコアが高いものを採用する
best_for_model: CompatibilityResult | None = None
for variant in candidates:
if gguf_only_backend and variant is None:
continue
compat = check_compatibility(model, variant, hardware, context_length)
if not compat.can_run:
continue
if fit_filter == "full_gpu" and compat.fit_type != "full_gpu":
continue
tok_per_sec = estimate_tok_per_sec(
model, variant, best_gpu, compat.fit_type
)
if compat.uses_multi_gpu:
tok_per_sec *= _MULTI_GPU_SPEED_FACTOR
if min_speed is not None and tok_per_sec < min_speed:
continue
bench_avg = None
if bench_evidence.score is not None:
if bench_evidence.source in {"direct", "self_reported"}:
bench_avg = bench_evidence.score
else:
# Inherited evidence: scale by confidence so weak inheritance
# (e.g. line_interp at conf 0.22) gets discounted on top of
# the per-source weight in _compute_quality_score.
confidence = max(0.0, min(1.0, bench_evidence.confidence))
bench_avg = bench_evidence.score * (0.75 + 0.25 * confidence)
compat.estimated_tok_per_sec = tok_per_sec
(
compat.speed_confidence,
compat.speed_range_tok_per_sec,
compat.speed_notes,
) = estimate_speed_uncertainty(
model,
variant,
best_gpu,
compat.fit_type,
tok_per_sec,
)
if compat.uses_multi_gpu:
compat.speed_confidence = "low"
if tok_per_sec > 0:
compat.speed_range_tok_per_sec = (
round(tok_per_sec * 0.35, 1),
round(tok_per_sec * 2.0, 1),
)
compat.speed_notes.append(
"Multi-GPU speed depends on layer/tensor split mode, "
"PCIe/NVLink bandwidth, and backend support; this estimate "
"does not assume ideal scaling."
)
compat.quality_score = _compute_quality_score(
model,
variant,
tok_per_sec,
compat.fit_type,
offload_ratio=compat.offload_ratio,
family_downloads=family_max_downloads.get(fid, 0),
family_likes=family_max_likes.get(fid, 0),
benchmark_avg=bench_avg,
benchmark_source=bench_evidence.source,
)
# Map evidence source to a 4-value display status. "self_reported"
# is shown distinctly so users can spot uploader-claimed numbers.
if bench_evidence.score is None:
compat.benchmark_status = "none"
elif bench_evidence.source == "direct":
compat.benchmark_status = "direct"
elif bench_evidence.source == "self_reported":
compat.benchmark_status = "self_reported"
else:
compat.benchmark_status = "estimated"
compat.benchmark_source = bench_evidence.source
compat.benchmark_confidence = bench_evidence.confidence
if (
best_for_model is None
or compat.quality_score > best_for_model.quality_score
):
best_for_model = compat
if best_for_model is None:
continue
# Deduplicate by family: keep the one with highest quality score
family_key = model.family_id
if family_key in seen_families:
# Check if this is better than existing
existing = next(
(r for r in results if r.model.family_id == family_key), None
)
if existing and _family_selection_key(
best_for_model,
require_direct_top,
) > _family_selection_key(existing, require_direct_top):
results.remove(existing)
results.append(best_for_model)
continue
seen_families.add(family_key)
results.append(best_for_model)
if require_direct_top:
results.sort(
key=lambda r: _family_selection_key(r, require_direct_top),
reverse=True,
)
else:
results.sort(
key=lambda r: _family_selection_key(r, require_direct_top), reverse=True
)
# Junk floor: when at least one candidate scores ≥ 30, drop anything
# below 20. This stops Q1_0 / Q2_0 derivatives (and other extreme-quant
# repos) from occupying ranking slots when a *real* option exists. If
# every candidate is junk (very tiny GPU + no fitting Q4) we keep the
# whole list so the user still sees what they can run.
if any(r.quality_score >= 30 for r in results):
results = [r for r in results if r.quality_score >= 20]
# Speed floor: a model that scores well on quality but runs at <1.5 t/s
# in practice (e.g. DeepSeek-V4-Flash 158B partial-offloading 100GB to
# CPU RAM from a 4GB GTX 1650) is not actually usable. Drop these
# candidates unless every remaining option is sub-1.5 too, in which
# case the user has hardware that cannot run anything responsively
# and we still want to show what's available.
if any(r.estimated_tok_per_sec >= 5.0 for r in results):
results = [r for r in results if r.estimated_tok_per_sec >= 1.5]
# Clamp top_n: a negative value would slice from the end
# (``results[:-5]``) and silently return a truncated, unrequested subset,
# while 0 legitimately yields an empty list. The CLI rejects non-positive
# --top, but guard here too so direct callers of this public helper never
# get a truncated ranking from a stray negative count.
return results[: max(top_n, 0)]
src/whichllm/engine/types.py
+30
View File
@@ -0,0 +1,30 @@
from __future__ import annotations
from dataclasses import dataclass, field
from whichllm.models.types import GGUFVariant, ModelInfo
@dataclass
class CompatibilityResult:
model: ModelInfo
gguf_variant: GGUFVariant | None
can_run: bool
vram_required_bytes: int
vram_available_bytes: int
offload_ratio: float = 0.0 # 0.0-1.0 fraction of weights spilled to CPU RAM
estimated_tok_per_sec: float | None = None
speed_confidence: str = "medium" # "high" | "medium" | "low"
speed_range_tok_per_sec: tuple[float, float] | None = None
speed_notes: list[str] = field(default_factory=list)
warnings: list[str] = field(default_factory=list)
quality_score: float = 0.0 # 0-100 for ranking
fit_type: str = "full_gpu" # "full_gpu" | "partial_offload" | "cpu_only"
benchmark_status: str = "none" # "direct" | "estimated" | "self_reported" | "none"
benchmark_source: str = "none" # granular: "direct" | "variant" | "base_model" | "line_interp" | "self_reported" | "none"
benchmark_confidence: float = 0.0 # 0.0-1.0 from BenchmarkEvidence
context_fits: bool = True # False when known model max context < requested
uses_multi_gpu: bool = False
multi_gpu_effective_vram_bytes: int | None = None
artifact_model: ModelInfo | None = None
artifact_variant: GGUFVariant | None = None
src/whichllm/engine/vram.py
+100
View File
@@ -0,0 +1,100 @@
"""VRAM usage estimation."""
from __future__ import annotations
from whichllm.constants import FRAMEWORK_OVERHEAD_BYTES
from whichllm.engine.quantization import estimate_weight_bytes
from whichllm.models.types import GGUFVariant, ModelInfo
# Empirical KV-cache coefficient: bytes per B-active-param per K-context-token
# for FP16 K/V tensors. Calibrated against published llama.cpp memory reports
# for Qwen2.5-7B (0.45 GB @ 8K), Qwen3-32B (3.1 GB @ 32K), and Llama-3.1-70B
# (5.4 GB @ 32K with GQA), then bumped slightly because real llama.cpp also
# allocates a graph-compute buffer proportional to KV size.
_KV_BYTES_PER_BPARAM_PER_KCTX = 3.5 * 1024 * 1024 # 3.5 MB
# MoE attention scales with the *attention-layer count*, which is roughly
# proportional to active_params * this multiplier. For Qwen3-Next-80B-A3B
# (3B active, 48 layers), the multiplier lands near 4.
_MOE_ATTENTION_PARAM_MULTIPLIER = 4.0
def _effective_kv_context(model: ModelInfo, context_length: int) -> float:
"""Context length that actually contributes to the KV cache.
For sliding-window-attention (SWA) models, local-attention layers only keep
the last ``sliding_window`` tokens, so their KV footprint plateaus once the
request exceeds the window. Hybrid models interleave a fraction of global
(full-context) layers; that fraction is ``sliding_window_global_ratio``.
Effective context blends the two layer types::
global_ratio * ctx + (1 - global_ratio) * min(ctx, window)
This is only applied for architectures whose mainline runtimes honor SWA
(the fetcher leaves ``sliding_window`` ``None`` otherwise), and it can only
ever lower the estimate — so a model that does not advertise an honored
window keeps the full-context KV figure and stays conservative.
"""
window = model.sliding_window
ratio = model.sliding_window_global_ratio
if not window or window <= 0 or ratio is None:
return float(context_length)
ratio = min(max(ratio, 0.0), 1.0)
windowed = min(context_length, window)
return ratio * context_length + (1.0 - ratio) * windowed
def estimate_kv_cache(model: ModelInfo, context_length: int) -> int:
"""Estimate KV cache size in bytes for a given context length.
Dense models: KV ≈ 3 MB × params_b × ctx_k (FP16 K+V across all layers).
MoE models: scale from active params × an empirical multiplier because
attention shares across experts.
Sliding-window models cap local-layer KV at the window size (see
:func:`_effective_kv_context`).
"""
if model.is_moe and model.parameter_count_active:
# Active-params × MoE multiplier gives a reasonable proxy for the
# attention-layer footprint without needing config.num_hidden_layers.
active_b = model.parameter_count_active / 1e9
params_b = active_b * _MOE_ATTENTION_PARAM_MULTIPLIER
else:
params_b = model.parameter_count / 1e9
effective_ctx = _effective_kv_context(model, context_length)
ctx_k = effective_ctx / 1024
kv_bytes = int(params_b * ctx_k * _KV_BYTES_PER_BPARAM_PER_KCTX)
return max(kv_bytes, 0)
def _activation_bytes(model: ModelInfo, context_length: int) -> int:
"""Activation/scratch buffer size.
Empirically activation memory grows mildly with both model size and
context length. The prior constant-plus-linear-param formula
over-counted small models and under-counted long contexts.
"""
# Use effective (active for MoE) size as the param-dependent base
if model.is_moe and model.parameter_count_active:
effective_p = model.parameter_count_active
else:
effective_p = model.parameter_count
base = 400_000_000 # 400 MB framework activation floor
param_term = int(effective_p * 0.08) # ~0.08 byte/param
ctx_term = int((context_length / 4096) * 150_000_000) # +150 MB per 4K
return base + param_term + ctx_term
def estimate_vram(
model: ModelInfo,
variant: GGUFVariant | None,
context_length: int = 4096,
) -> int:
"""Estimate total VRAM required to run a model."""
weights = estimate_weight_bytes(model, variant)
kv_cache = estimate_kv_cache(model, context_length)
activation = _activation_bytes(model, context_length)
framework = FRAMEWORK_OVERHEAD_BYTES
return weights + kv_cache + activation + framework
src/whichllm/hardware/__init__.py
View File
src/whichllm/hardware/amd.py
+286
View File
@@ -0,0 +1,286 @@
"""AMD GPU detection via rocm-smi with Linux fallback probes."""
from __future__ import annotations
import json
import logging
import shlex
import subprocess
from pathlib import Path
from whichllm.constants import AMD_SHARED_MEMORY_APU_MARKERS, _GiB
from whichllm.hardware.gpu_db import _static_bandwidth, resolve_detected_bandwidth
from whichllm.hardware.types import GPUInfo
logger = logging.getLogger(__name__)
_DISPLAY_CLASSES = (
"vga compatible controller",
"3d controller",
"display controller",
)
def _lookup_bandwidth(name: str) -> float | None:
"""Curated GPU_BANDWIDTH lookup, compound-lspci aware. Kept for regression
tests; live detection goes through ``resolve_detected_bandwidth``, which
also consults dbgpu."""
return _static_bandwidth(name)
def _is_shared_memory_apu(name: str) -> bool:
name_upper = name.upper()
return any(marker in name_upper for marker in AMD_SHARED_MEMORY_APU_MARKERS)
def _normalize_apu_vram(name: str, vram_bytes: int) -> int:
if _is_shared_memory_apu(name) and vram_bytes < 2 * _GiB:
return 0
return vram_bytes
def _make_gpu(
name: str,
*,
vram_bytes: int = 0,
rocm_version: str | None = None,
) -> GPUInfo:
shared_memory = _is_shared_memory_apu(name)
return GPUInfo(
name=name,
vendor="amd",
vram_bytes=_normalize_apu_vram(name, vram_bytes),
rocm_version=rocm_version,
memory_bandwidth_gbps=resolve_detected_bandwidth(name, vram_bytes),
shared_memory=shared_memory,
)
_AMD_VENDOR_MARKERS = (
"advanced micro devices",
"amd/ati",
"[amd]",
"[ati]",
"ati technologies",
)
def _vendor_is_amd(vendor: str) -> bool:
vendor_lower = vendor.lower()
return any(marker in vendor_lower for marker in _AMD_VENDOR_MARKERS)
def _detect_from_lspci() -> list[str]:
try:
result = subprocess.run(
["lspci", "-mm"],
capture_output=True,
text=True,
timeout=5,
)
except (FileNotFoundError, subprocess.TimeoutExpired):
logger.debug("lspci not available or timed out")
return []
if result.returncode != 0:
return []
names: list[str] = []
seen: set[str] = set()
for line in result.stdout.splitlines():
# `lspci -mm` is the machine-parsable format:
# <slot> "<class>" "<vendor>" "<device>" [flags] ["<subvendor>" "<subdevice>"]
# Parse the quoted columns properly and check the vendor field
# specifically, instead of substring-matching the whole line (which
# would treat e.g. "Intel Corpor[ati]on" as AMD).
try:
tokens = shlex.split(line)
except ValueError:
continue
if len(tokens) < 4:
continue
device_class, vendor, device = tokens[1], tokens[2], tokens[3]
if device_class.lower() not in _DISPLAY_CLASSES:
continue
if not _vendor_is_amd(vendor):
continue
name = device.strip() or "AMD Graphics"
if name not in seen:
names.append(name)
seen.add(name)
return names
def _read_int(path: Path) -> int:
try:
text = path.read_text().strip()
except OSError:
return 0
try:
return int(text, 0)
except ValueError:
return 0
def _detect_from_sysfs(drm_path: Path = Path("/sys/class/drm")) -> list[GPUInfo]:
gpus: list[GPUInfo] = []
seen: set[str] = set()
try:
cards = sorted(drm_path.glob("card[0-9]*"))
except OSError:
return []
for card in cards:
device = card / "device"
try:
vendor = (device / "vendor").read_text().strip().lower()
except OSError:
continue
if vendor != "0x1002":
continue
name = "AMD Graphics"
try:
product_name = (device / "product_name").read_text().strip()
if product_name:
name = product_name
except OSError:
pass
vram_bytes = _read_int(device / "mem_info_vram_total")
key = f"{name}:{vram_bytes}"
if key in seen:
continue
seen.add(key)
gpus.append(_make_gpu(name, vram_bytes=vram_bytes))
return gpus
def _read_sysfs_amd_vram(drm_path: Path = Path("/sys/class/drm")) -> list[int]:
"""Read VRAM for each AMD GPU from sysfs, in card order."""
result: list[int] = []
try:
cards = sorted(drm_path.glob("card[0-9]*"))
except OSError:
return []
for card in cards:
device = card / "device"
try:
vendor = (device / "vendor").read_text().strip().lower()
except OSError:
continue
if vendor != "0x1002":
continue
result.append(_read_int(device / "mem_info_vram_total"))
return result
def _detect_amd_gpus_fallback() -> list[GPUInfo]:
# Prefer sysfs: it provides VRAM and sometimes a clean product name.
sysfs_gpus = _detect_from_sysfs()
if sysfs_gpus:
# If sysfs names are generic ("AMD Graphics"), enrich with lspci names.
has_generic = any(g.name == "AMD Graphics" for g in sysfs_gpus)
if has_generic:
lspci_names = _detect_from_lspci()
if lspci_names and len(lspci_names) == len(sysfs_gpus):
return [
_make_gpu(
lspci_names[i] if gpu.name == "AMD Graphics" else gpu.name,
vram_bytes=gpu.vram_bytes,
)
for i, gpu in enumerate(sysfs_gpus)
]
return sysfs_gpus
# sysfs unavailable; fall back to lspci (name only), enriching with
# sysfs VRAM when possible.
names = _detect_from_lspci()
if names:
vram_list = _read_sysfs_amd_vram()
return [
_make_gpu(name, vram_bytes=vram_list[i] if i < len(vram_list) else 0)
for i, name in enumerate(names)
]
return []
def detect_amd_gpus() -> list[GPUInfo]:
"""Detect AMD GPUs. Returns empty list on failure."""
gpus: list[GPUInfo] = []
# Get product names
try:
result = subprocess.run(
["rocm-smi", "--showproductname", "--json"],
capture_output=True,
text=True,
timeout=10,
)
if result.returncode != 0:
return _detect_amd_gpus_fallback()
product_data = json.loads(result.stdout)
except (FileNotFoundError, subprocess.TimeoutExpired, json.JSONDecodeError):
logger.debug("rocm-smi not available or failed")
return _detect_amd_gpus_fallback()
# Get VRAM info
try:
result = subprocess.run(
["rocm-smi", "--showmeminfo", "vram", "--json"],
capture_output=True,
text=True,
timeout=10,
)
if result.returncode != 0:
return _detect_amd_gpus_fallback()
mem_data = json.loads(result.stdout)
except (FileNotFoundError, subprocess.TimeoutExpired, json.JSONDecodeError):
logger.debug("Failed to get AMD VRAM info")
return _detect_amd_gpus_fallback()
# Get ROCm version
rocm_version = None
try:
result = subprocess.run(
["rocm-smi", "--showdriverversion", "--json"],
capture_output=True,
text=True,
timeout=10,
)
if result.returncode == 0:
driver_data = json.loads(result.stdout)
# Extract version from first card entry
for key, val in driver_data.items():
if isinstance(val, dict) and "Driver version" in val:
rocm_version = val["Driver version"]
break
except Exception:
pass
# Parse GPU info - rocm-smi JSON keys are like "card0", "card1"
for key in sorted(product_data.keys()):
if not key.startswith("card"):
continue
card_info = product_data[key]
# rocm-smi JSON uses "Card Series" (capital S) for the human-readable name.
# Fall back through SKU only as a last resort.
name = (
card_info.get("Card Series")
or card_info.get("Card series")
or card_info.get("Card SKU")
or "Unknown AMD GPU"
)
vram_total = 0
if key in mem_data:
vram_str = mem_data[key].get("VRAM Total Memory (B)", "0")
try:
vram_total = int(vram_str)
except (ValueError, TypeError):
pass
gpus.append(_make_gpu(name, vram_bytes=vram_total, rocm_version=rocm_version))
return gpus
src/whichllm/hardware/apple.py
+131
View File
@@ -0,0 +1,131 @@
"""Apple Silicon detection via system_profiler (macOS) and sysfs (Asahi Linux)."""
from __future__ import annotations
import json
import logging
import re
import subprocess
from pathlib import Path
from whichllm.constants import GPU_BANDWIDTH
from whichllm.hardware.types import GPUInfo
logger = logging.getLogger(__name__)
def _lookup_bandwidth(chip_name: str) -> float | None:
chip_upper = chip_name.upper()
for key in sorted(GPU_BANDWIDTH, key=len, reverse=True):
if key.upper() in chip_upper:
return GPU_BANDWIDTH[key]
return None
def detect_apple_gpu() -> list[GPUInfo]:
"""Detect Apple Silicon GPU. Returns empty list on non-macOS or failure."""
try:
result = subprocess.run(
["system_profiler", "SPHardwareDataType", "-json"],
capture_output=True,
text=True,
timeout=10,
)
if result.returncode != 0:
return []
data = json.loads(result.stdout)
except (FileNotFoundError, subprocess.TimeoutExpired, json.JSONDecodeError):
logger.debug("system_profiler not available (not macOS)")
return []
try:
hw_items = data["SPHardwareDataType"]
hw = hw_items[0]
chip_name = hw.get("chip_type", "")
if not chip_name:
return []
# Apple Silicon uses unified memory - get total physical memory
memory_str = hw.get("physical_memory", "0 GB")
# Parse "32 GB" -> bytes
parts = memory_str.split()
mem_value = int(parts[0])
mem_unit = parts[1].upper() if len(parts) > 1 else "GB"
multiplier = {"GB": 1024**3, "TB": 1024**4, "MB": 1024**2}.get(
mem_unit, 1024**3
)
unified_memory = mem_value * multiplier
return [
GPUInfo(
name=chip_name,
vendor="apple",
vram_bytes=unified_memory, # unified memory
memory_bandwidth_gbps=_lookup_bandwidth(chip_name),
shared_memory=True,
)
]
except (KeyError, IndexError, ValueError) as e:
logger.debug(f"Failed to parse Apple hardware info: {e}")
return []
# ---- Asahi Linux (Apple Silicon on Linux) ----
_ASAHI_DRIVER_NAMES = ("asahi", "apple")
def _chip_name_from_devicetree() -> str | None:
"""Extract Apple chip name from Linux device tree."""
try:
raw = Path("/sys/firmware/devicetree/base/model").read_bytes()
model = raw.decode("utf-8", errors="replace").strip().rstrip("\x00")
if not model:
return None
m = re.search(r"\b(M\d+(?:\s+(?:Pro|Max|Ultra))?)\b", model)
if m:
return f"Apple {m.group(1)}"
return model
except OSError:
return None
def detect_apple_gpu_linux(
drm_path: Path = Path("/sys/class/drm"),
) -> list[GPUInfo]:
"""Detect Apple Silicon GPU on Linux (Asahi driver).
Returns empty list when no Asahi/Apple DRM device is found.
"""
try:
cards = sorted(drm_path.glob("card[0-9]*"))
except OSError:
return []
for card in cards:
driver = card / "device" / "driver"
try:
driver_name = driver.resolve().name
except OSError:
continue
if driver_name not in _ASAHI_DRIVER_NAMES:
continue
chip_name = _chip_name_from_devicetree() or "Apple Silicon"
# Unified memory — total system RAM is shared with the GPU.
import psutil
unified_memory = psutil.virtual_memory().total
return [
GPUInfo(
name=chip_name,
vendor="apple",
vram_bytes=unified_memory,
memory_bandwidth_gbps=_lookup_bandwidth(chip_name),
shared_memory=True,
)
]
return []
src/whichllm/hardware/cpu.py
+225
View File
@@ -0,0 +1,225 @@
"""CPU detection: name, cores, AVX2/AVX512 support."""
from __future__ import annotations
import logging
import platform
import re
import subprocess
from pathlib import Path
logger = logging.getLogger(__name__)
def _cpu_name_from_lscpu() -> str | None:
"""Try to get CPU model name from lscpu (works on ARM/aarch64)."""
try:
result = subprocess.run(["lscpu"], capture_output=True, text=True, timeout=5)
if result.returncode == 0:
for line in result.stdout.splitlines():
if line.strip().startswith("Model name"):
name = line.split(":", 1)[1].strip()
if name and name != "-":
return name
except (FileNotFoundError, subprocess.TimeoutExpired):
pass
return None
def _cpu_name_from_devicetree() -> str | None:
"""Extract CPU/chip name from device tree (ARM Linux, Asahi)."""
try:
raw = Path("/sys/firmware/devicetree/base/model").read_bytes()
model = raw.decode("utf-8", errors="replace").strip().rstrip("\x00")
if not model:
return None
# "Apple MacBook Air (M2, 2022)" → "Apple M2"
m = re.search(r"\b(M\d+(?:\s+(?:Pro|Max|Ultra))?)\b", model)
if m:
return f"Apple {m.group(1)}"
return model
except OSError:
return None
def _clean_cpu_name(name: str | None) -> str | None:
if name is None:
return None
cleaned = name.strip()
if not cleaned or cleaned == "-" or cleaned.lower() == "name":
return None
return cleaned
def _cpu_name_from_wmic() -> str | None:
try:
result = subprocess.run(
["wmic", "cpu", "get", "name"],
capture_output=True,
text=True,
timeout=5,
)
except (FileNotFoundError, subprocess.SubprocessError, OSError):
return None
if result.returncode != 0:
return None
for line in result.stdout.splitlines():
name = _clean_cpu_name(line)
if name:
return name
return None
def _cpu_name_from_windows_cim() -> str | None:
script = (
"Get-CimInstance Win32_Processor | Select-Object -First 1 -ExpandProperty Name"
)
for executable in ("powershell", "pwsh"):
try:
result = subprocess.run(
[executable, "-NoProfile", "-Command", script],
capture_output=True,
text=True,
timeout=5,
)
except (FileNotFoundError, subprocess.SubprocessError, OSError):
continue
if result.returncode != 0:
continue
for line in result.stdout.splitlines():
name = _clean_cpu_name(line)
if name:
return name
return None
def detect_cpu_name() -> str:
"""Get CPU model name."""
system = platform.system()
try:
if system == "Linux":
with open("/proc/cpuinfo") as f:
for line in f:
if line.startswith("model name"):
return line.split(":", 1)[1].strip()
# ARM/aarch64: /proc/cpuinfo has no model name field.
# Try lscpu, then device tree.
name = _cpu_name_from_lscpu() or _cpu_name_from_devicetree()
if name:
return name
elif system == "Darwin":
result = subprocess.run(
["sysctl", "-n", "machdep.cpu.brand_string"],
capture_output=True,
text=True,
timeout=5,
)
if result.returncode == 0:
name = _clean_cpu_name(result.stdout)
if name:
return name
elif system == "Windows":
name = _cpu_name_from_wmic() or _cpu_name_from_windows_cim()
if name:
return name
except Exception as e:
logger.debug(f"Failed to detect CPU name: {e}")
return "Unknown CPU"
def _count_physical_cores_linux() -> int | None:
"""Count unique physical cores from /proc/cpuinfo (handles WSL2)."""
try:
physical_ids: set[tuple[str, str]] = set()
current_physical = ""
current_core = ""
with open("/proc/cpuinfo") as f:
for line in f:
if line.startswith("physical id"):
current_physical = line.split(":", 1)[1].strip()
elif line.startswith("core id"):
current_core = line.split(":", 1)[1].strip()
physical_ids.add((current_physical, current_core))
if physical_ids:
return len(physical_ids)
except Exception:
pass
return None
def detect_cpu_cores() -> int:
"""Get number of physical CPU cores."""
import psutil
cores = psutil.cpu_count(logical=False)
if cores:
return cores
# Fallback for WSL2 where psutil may return None for physical cores
if platform.system() == "Linux":
linux_cores = _count_physical_cores_linux()
if linux_cores:
return linux_cores
return psutil.cpu_count(logical=True) or 1
def _detect_avx_linux() -> tuple[bool, bool]:
"""Detect AVX2/AVX512 on Linux via /proc/cpuinfo."""
has_avx2 = False
has_avx512 = False
try:
with open("/proc/cpuinfo") as f:
content = f.read()
flags_line = ""
for line in content.split("\n"):
if line.startswith("flags"):
flags_line = line
break
has_avx2 = "avx2" in flags_line
has_avx512 = "avx512f" in flags_line
except Exception:
pass
return has_avx2, has_avx512
def _detect_avx_darwin() -> tuple[bool, bool]:
"""Detect AVX2/AVX512 on macOS via sysctl."""
has_avx2 = False
has_avx512 = False
try:
result = subprocess.run(
["sysctl", "-n", "hw.optional.avx2_0"],
capture_output=True,
text=True,
timeout=5,
)
has_avx2 = result.stdout.strip() == "1"
except Exception:
pass
try:
result = subprocess.run(
["sysctl", "-n", "hw.optional.avx512f"],
capture_output=True,
text=True,
timeout=5,
)
has_avx512 = result.stdout.strip() == "1"
except Exception:
pass
return has_avx2, has_avx512
def detect_avx_support() -> tuple[bool, bool]:
"""Detect AVX2 and AVX512 support. Returns (has_avx2, has_avx512)."""
system = platform.system()
if system == "Linux":
return _detect_avx_linux()
elif system == "Darwin":
return _detect_avx_darwin()
# Windows / fallback: assume AVX2 on modern CPUs
return True, False
src/whichllm/hardware/detector.py
+56
View File
@@ -0,0 +1,56 @@
"""Unified hardware detection orchestrator."""
from __future__ import annotations
import logging
import platform
from whichllm.hardware.amd import detect_amd_gpus
from whichllm.hardware.apple import detect_apple_gpu, detect_apple_gpu_linux
from whichllm.hardware.cpu import detect_avx_support, detect_cpu_cores, detect_cpu_name
from whichllm.hardware.intel import detect_intel_gpus
from whichllm.hardware.memory import detect_disk_free_bytes, detect_ram_bytes
from whichllm.hardware.nvidia import detect_nvidia_gpus
from whichllm.hardware.types import HardwareInfo
from whichllm.hardware.windows import detect_windows_gpus
logger = logging.getLogger(__name__)
def detect_hardware() -> HardwareInfo:
"""Detect all hardware. Each detector is fail-safe (returns empty on error)."""
os_name = platform.system().lower()
if os_name not in ("linux", "darwin", "windows"):
os_name = "linux"
# GPU detection
gpus = []
gpus.extend(detect_nvidia_gpus())
if os_name == "linux":
gpus.extend(detect_amd_gpus())
gpus.extend(detect_intel_gpus())
gpus.extend(detect_apple_gpu_linux())
if os_name == "darwin":
gpus.extend(detect_apple_gpu())
if os_name == "windows":
gpus.extend(detect_windows_gpus())
# CPU
cpu_name = detect_cpu_name()
cpu_cores = detect_cpu_cores()
has_avx2, has_avx512 = detect_avx_support()
# Memory
ram_bytes = detect_ram_bytes()
disk_free = detect_disk_free_bytes()
return HardwareInfo(
gpus=gpus,
cpu_name=cpu_name,
cpu_cores=cpu_cores,
has_avx2=has_avx2,
has_avx512=has_avx512,
ram_bytes=ram_bytes,
disk_free_bytes=disk_free,
os=os_name,
)
src/whichllm/hardware/gpu_db.py
+219
View File
@@ -0,0 +1,219 @@
"""Resolve GPU memory bandwidth for *detected* hardware.
Detection passes the raw driver name (e.g. ``"NVIDIA GeForce RTX 5090 Laptop
GPU"``). Unlike ``--gpu`` simulation, where the user typed the name and a fuzzy
guess plus a ``(simulated)`` label is acceptable, a wrong match on real
hardware is worse than no data: giving a laptop card its desktop bandwidth
produces confidently wrong speed estimates and oversized recommendations
(issues #74, #61, #93).
So this resolver is deliberately strict:
* The hand-curated ``GPU_BANDWIDTH`` table stays authoritative and is tried
first, so existing behaviour for known cards is unchanged.
* The curated lookup is mobile-aware: a laptop/Max-Q driver name will not match
a desktop key (``"RTX 5090"`` no longer swallows ``"RTX 5090 Laptop GPU"``).
* dbgpu is only consulted to fill gaps, and only via an exact normalised-name
hit or a name plus a VRAM-size suffix (``"RTX 4060 Ti 16 GB"``). It never
falls back to fuzzy matching, so a variant qualifier (Ti / SUPER / Mobile /
Max-Q / XT) can never be silently dropped onto the wrong card.
* ``"Laptop GPU"`` in a driver name is normalised to dbgpu's ``"Mobile"``.
The change is purely additive: cards already covered by ``GPU_BANDWIDTH`` keep
their exact value, and cards that previously resolved to ``None`` now get a
correct bandwidth whenever dbgpu can identify them safely.
"""
from __future__ import annotations
import functools
import logging
import re
from whichllm.constants import GPU_BANDWIDTH, GPU_MEMORY_CLOCK_VARIANTS, _GiB
logger = logging.getLogger(__name__)
_TRADEMARK_RE = re.compile(r"\((?:tm|r)\)", re.IGNORECASE)
_VENDOR_WORD_RE = re.compile(r"\b(?:nvidia|amd|ati|intel|corporation)\b", re.IGNORECASE)
_LAPTOP_GPU_RE = re.compile(r"\blaptop gpu\b", re.IGNORECASE)
_TRAILING_GRAPHICS_RE = re.compile(r"\bgraphics\s*$", re.IGNORECASE)
_WHITESPACE_RE = re.compile(r"\s+")
_MOBILE_MARKER_RE = re.compile(r"\b(?:laptop|mobile|max-?q)\b", re.IGNORECASE)
# Driver names write VRAM bins without a space ("RTX A2000 12GB"); dbgpu
# writes "RTX A2000 12 GB" (#98).
_VRAM_NOSPACE_RE = re.compile(r"\b(\d+)GB\b", re.IGNORECASE)
_BRACKET_RE = re.compile(r"\[(.+)]")
# A dbgpu name may extend a matched query with a VRAM-size bin only
# ("RTX 4060 Ti 16 GB"). A variant word (Mobile, Ti, D, ...) is never benign.
_VRAM_SUFFIX_RE = re.compile(r"^\s+\d+\s*gb\b", re.IGNORECASE)
_VRAM_GB_RE = re.compile(r"(\d+)\s*gb", re.IGNORECASE)
def _normalize_detected_name(name: str) -> str:
"""Reduce a raw driver name toward dbgpu's naming convention."""
text = _TRADEMARK_RE.sub("", name)
text = _VENDOR_WORD_RE.sub("", text)
text = _LAPTOP_GPU_RE.sub("Mobile", text)
text = _TRAILING_GRAPHICS_RE.sub("", text)
text = _VRAM_NOSPACE_RE.sub(r"\1 GB", text)
return _WHITESPACE_RE.sub(" ", text).strip()
_SORTED_BW_KEYS = sorted(GPU_BANDWIDTH, key=len, reverse=True)
def _substring_bandwidth(name: str) -> float | None:
"""Curated ``GPU_BANDWIDTH`` lookup (longest key first), mobile-aware.
When the detected name is a laptop/Max-Q card, a desktop key is not allowed
to match it via substring, since the two have very different bandwidth.
"""
if not name:
return None
name_upper = name.upper()
name_is_mobile = bool(_MOBILE_MARKER_RE.search(name))
for key in _SORTED_BW_KEYS:
if key.upper() in name_upper:
if name_is_mobile and not _MOBILE_MARKER_RE.search(key):
continue
return GPU_BANDWIDTH[key]
return None
def _static_bandwidth(name: str) -> float | None:
"""Curated lookup that also handles compound lspci names.
Compound names like ``"Navi 22 [Radeon RX 6700/6700 XT/6750 XT /
6800M/6850M XT]"`` list several variants; the first segment that resolves
wins, with a ``"RX "`` prefix retry for bare segments like ``"6750 XT"``.
dbgpu is never consulted for these: the name does not identify a single
card, so the curated value for the listed family is the safest answer.
"""
if not name:
return None
if "/" not in name:
bandwidth = _substring_bandwidth(name)
if bandwidth is not None:
return bandwidth
normalized = _normalize_detected_name(name)
return _substring_bandwidth(normalized) if normalized != name else None
bracket = _BRACKET_RE.search(name)
raw = bracket.group(1) if bracket else name
for seg in raw.split("/"):
seg = seg.strip()
if not seg:
continue
bandwidth = _substring_bandwidth(seg) or _substring_bandwidth(f"RX {seg}")
if bandwidth is not None:
return bandwidth
return None
def _vram_gb(canonical_name: str) -> int | None:
match = _VRAM_GB_RE.search(canonical_name)
return int(match.group(1)) if match else None
@functools.lru_cache(maxsize=1)
def _dbgpu_index() -> tuple[object | None, dict[str, str] | None]:
"""Build ``{normalized_name: canonical_dbgpu_name}``.
Returns ``(db, index)`` or ``(None, None)`` if dbgpu is unavailable, so the
resolver degrades to static-only instead of raising.
"""
try:
from dbgpu import GPUDatabase
db = GPUDatabase.default()
except Exception as exc: # pragma: no cover - dbgpu is a hard dependency
logger.debug("dbgpu unavailable, using static bandwidth only: %s", exc)
return None, None
index: dict[str, str] = {}
for canonical in db.names:
index.setdefault(_normalize_detected_name(canonical).lower(), canonical)
return db, index
def _dbgpu_bandwidth(name: str, vram_bytes: int | None) -> float | None:
"""Strict dbgpu bandwidth lookup. Never fuzzy-matches a variant away."""
db, index = _dbgpu_index()
if db is None or index is None:
return None
query = _normalize_detected_name(name).lower()
if not query:
return None
if query in index:
candidates = [index[query]]
else:
candidates = [
original
for normalized, original in index.items()
if normalized.startswith(query + " ")
and _VRAM_SUFFIX_RE.match(normalized[len(query) :])
]
if not candidates:
return None
if vram_bytes and len(candidates) > 1:
target_gb = round(vram_bytes / _GiB)
same_vram = [c for c in candidates if _vram_gb(c) == target_gb]
if same_vram:
candidates = same_vram
# If several VRAM bins remain (VRAM unknown or no bin matched it), take the
# lowest bandwidth among them: an ambiguous match must never over-promise
# speed. Bins of the same card usually share one value anyway.
bandwidths: list[float] = []
for canonical in candidates:
try:
spec = db[canonical]
except KeyError: # pragma: no cover - canonical comes from the index
continue
bandwidth = getattr(spec, "memory_bandwidth_gb_s", None)
if bandwidth:
bandwidths.append(float(bandwidth))
return min(bandwidths) if bandwidths else None
def _memory_clock_variant_bandwidth(
name: str, mem_clock_mhz: float | None
) -> float | None:
"""Disambiguate cards sold in multiple memory types by max memory clock.
Some GPUs (e.g. GTX 1650 GDDR5 vs GDDR6) share a marketing name and PCI
device id, so only the memory clock tells them apart. Returns the matching
variant bandwidth when the name is a known dual-memory card and the clock is
known, else ``None`` (so the caller falls back to the curated default).
"""
if not name or not mem_clock_mhz or mem_clock_mhz <= 0:
return None
name_upper = name.upper()
for key in sorted(GPU_MEMORY_CLOCK_VARIANTS, key=len, reverse=True):
if key.upper() in name_upper:
for min_clock, bandwidth in GPU_MEMORY_CLOCK_VARIANTS[key]:
if mem_clock_mhz >= min_clock:
return bandwidth
return None
def resolve_detected_bandwidth(
name: str,
vram_bytes: int | None = None,
mem_clock_mhz: float | None = None,
) -> float | None:
"""Best memory bandwidth (GB/s) for a detected GPU, or ``None`` if unknown.
Memory-clock variant disambiguation wins first (for cards that share a name
across memory types, e.g. GTX 1650 GDDR5/GDDR6); then curated
``GPU_BANDWIDTH``; then dbgpu fills the gaps. ``vram_bytes`` (when known)
disambiguates same-name cards that ship in multiple VRAM bins;
``mem_clock_mhz`` (max memory clock, when known) disambiguates memory-type
variants. With both unset, behaviour is identical to before.
"""
if not name:
return None
variant = _memory_clock_variant_bandwidth(name, mem_clock_mhz)
if variant is not None:
return variant
return _static_bandwidth(name) or _dbgpu_bandwidth(name, vram_bytes)
src/whichllm/hardware/gpu_simulator.py
+346
View File
@@ -0,0 +1,346 @@
"""Create synthetic GPUInfo for GPU simulation (--gpu flag).
Uses the dbgpu package (2000+ GPU database from TechPowerUp) for dynamic
lookup of VRAM, memory bandwidth, and compute capability.
"""
from __future__ import annotations
import logging
import re
from collections.abc import Sequence
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from dbgpu import GPUSpecification
from whichllm.constants import (
AMD_SHARED_MEMORY_APU_MARKERS,
CURATED_GPU_SPECS,
GPU_BANDWIDTH,
CuratedGPUSpec,
_GiB,
)
from whichllm.hardware.types import GPUInfo
logger = logging.getLogger(__name__)
_MANUFACTURER_TO_VENDOR: dict[str, str] = {
"NVIDIA": "nvidia",
"AMD": "amd",
"ATI": "amd",
"Intel": "intel",
"Apple": "apple",
}
_MANUFACTURER_PREFIXES = ["GeForce ", "Radeon ", "Arc ", "NVIDIA ", "AMD "]
_COMMON_GPU_ALIASES: dict[str, list[str]] = {
"a10080gb": [
"NVIDIA A100 PCIe 80 GB",
"NVIDIA A100 SXM4 80 GB",
],
"h10080gb": [
"NVIDIA H100 PCIe 80 GB",
"NVIDIA H100 SXM5 80 GB",
],
}
# Apple Silicon chips. dbgpu does not include these (it tracks discrete GPUs
# via TechPowerUp), but users routinely simulate with --gpu "M2 Max" etc.
# Without this short-circuit, "M1" fuzzy-matches the 1997 ATI Rage Mobility-M1
# and "M3 Max" falls through to vendor=nvidia default.
# Format: chip_name -> (canonical_name, default_unified_memory_gb).
# --vram override is still respected; the default is the most common SKU.
_APPLE_SILICON_CHIPS: dict[str, tuple[str, float]] = {
"M1": ("Apple M1", 8.0),
"M1 Pro": ("Apple M1 Pro", 16.0),
"M1 Max": ("Apple M1 Max", 32.0),
"M1 Ultra": ("Apple M1 Ultra", 64.0),
"M2": ("Apple M2", 16.0),
"M2 Pro": ("Apple M2 Pro", 16.0),
"M2 Max": ("Apple M2 Max", 32.0),
"M2 Ultra": ("Apple M2 Ultra", 64.0),
"M3": ("Apple M3", 16.0),
"M3 Pro": ("Apple M3 Pro", 18.0),
"M3 Max": ("Apple M3 Max", 36.0),
"M3 Ultra": ("Apple M3 Ultra", 96.0),
"M4": ("Apple M4", 16.0),
"M4 Pro": ("Apple M4 Pro", 24.0),
"M4 Max": ("Apple M4 Max", 36.0),
"M4 Ultra": ("Apple M4 Ultra", 64.0),
"M5": ("Apple M5", 16.0),
"M5 Pro": ("Apple M5 Pro", 24.0),
"M5 Max": ("Apple M5 Max", 36.0),
}
def _lookup_apple_silicon(
name: str,
) -> tuple[str, str, float, float] | None:
"""Match Apple Silicon chip names. Returns (canonical_name, vendor,
default_vram_gb, bandwidth_gbps) or None.
Matches are case-insensitive and accept "M2 Max", "m2max", and
display-name forms such as "Apple M2 Max". Longest match wins so
"M2 Ultra" does not get caught by the "M2" entry.
"""
compact = re.sub(r"\s+", "", name).lower()
if compact.startswith("apple"):
compact = compact.removeprefix("apple")
# Sort keys by length descending so "M2 Ultra" wins over "M2".
for key in sorted(_APPLE_SILICON_CHIPS, key=len, reverse=True):
key_compact = re.sub(r"\s+", "", key).lower()
if compact == key_compact:
canonical, default_vram = _APPLE_SILICON_CHIPS[key]
bandwidth = GPU_BANDWIDTH.get(key, 100.0)
return canonical, "apple", default_vram, bandwidth
return None
def _is_amd_shared_memory_apu(name: str) -> bool:
name_upper = name.upper()
return any(marker in name_upper for marker in AMD_SHARED_MEMORY_APU_MARKERS)
def _lookup_static_bandwidth(name: str) -> float | None:
name_upper = name.upper()
for key in sorted(GPU_BANDWIDTH, key=len, reverse=True):
if key.upper() in name_upper:
return GPU_BANDWIDTH[key]
return None
def _lookup_curated_spec(name: str) -> CuratedGPUSpec | None:
name_upper = name.upper()
for key in sorted(CURATED_GPU_SPECS, key=len, reverse=True):
if key.upper() in name_upper:
return CURATED_GPU_SPECS[key]
return None
def _normalize_gpu_name(name: str) -> str:
"""Normalize user input: 'GTX1080' → 'GTX 1080', 'RX7900XTX' → 'RX 7900 XTX'."""
# Insert space between letters and digits
name = re.sub(r"([A-Za-z])(\d)", r"\1 \2", name)
# Insert space between digits and letters
name = re.sub(r"(\d)([A-Za-z])", r"\1 \2", name)
# Collapse multiple spaces
return re.sub(r"\s+", " ", name).strip()
def _substring_search(db, name: str):
"""Substring match with word-boundary filtering.
e.g. "RTX 3060" should match "GeForce RTX 3060 12 GB" but NOT "GeForce RTX 3060 Ti".
"""
name_upper = name.upper()
candidates = []
for db_name in db.names:
idx = db_name.upper().find(name_upper)
if idx < 0:
continue
after = db_name[idx + len(name) :]
# Accept if nothing follows, or what follows is VRAM/form-factor spec
# Reject if a variant suffix follows (Ti, SUPER, Mobile, Max-Q, etc.)
if not after or re.match(r"^(\s+(\d|GA\d|PCIe|SXM|NVL|CNX))", after):
candidates.append(db_name)
if candidates:
candidates.sort(key=len)
return db[candidates[0]]
return None
def _lookup_dbgpu(name: str) -> GPUSpecification | None:
"""Look up GPU spec from dbgpu database. Returns GPUSpecification or None."""
from dbgpu import GPUDatabase
db = GPUDatabase.default()
# Normalize input: "GTX1080" → "GTX 1080"
normalized = _normalize_gpu_name(name)
compact = re.sub(r"\s+", "", normalized.lower())
names_to_try = [name] if normalized == name else [name, normalized]
alias_hits = _COMMON_GPU_ALIASES.get(compact)
if alias_hits:
names_to_try.extend(alias_hits)
for n in names_to_try:
# 1) Exact key lookup
try:
return db[n]
except KeyError:
pass
# 2) Try with common manufacturer prefixes
for prefix in _MANUFACTURER_PREFIXES:
try:
return db[prefix + n]
except KeyError:
pass
# 3) Substring match with word-boundary filtering
result = _substring_search(db, n)
if result is not None:
return result
# 4) Fuzzy search as last resort (use normalized name + token_set_ratio)
try:
from thefuzz import fuzz, process
results = process.extract(
normalized, db.names, limit=3, scorer=fuzz.token_set_ratio
)
if results and results[0][1] >= 90:
return db[results[0][0]]
# Store top suggestions for error messages
if results:
_last_suggestions[:] = [(n, s) for n, s in results if s >= 70]
except ImportError:
pass
return None
# Mutable list to pass suggestions from lookup to error message
_last_suggestions: list[tuple[str, int]] = []
def parse_synthetic_gpu_specs(values: Sequence[str] | str) -> list[str]:
"""Expand CLI GPU simulation values into individual GPU names.
Accepts repeated options, comma-separated names, and count shorthand such
as ``2x RTX 4090``. The returned names are still looked up by
``create_synthetic_gpu`` so existing fuzzy matching and aliases stay in
one place.
"""
raw_values = [values] if isinstance(values, str) else list(values)
gpu_names: list[str] = []
for raw in raw_values:
for part in raw.split(","):
spec = part.strip()
if not spec:
raise ValueError("Empty GPU entry in --gpu.")
count_match = re.match(r"^(\d+)\s*x\s+(.+)$", spec, re.IGNORECASE)
if count_match:
count = int(count_match.group(1))
name = count_match.group(2).strip()
if count < 1:
raise ValueError("GPU count must be at least 1.")
if not name:
raise ValueError("GPU count shorthand requires a GPU name.")
gpu_names.extend([name] * count)
else:
gpu_names.append(spec)
if not gpu_names:
raise ValueError("At least one GPU must be specified.")
return gpu_names
def create_synthetic_gpus(
values: Sequence[str] | str,
vram_override_gb: float | None = None,
) -> list[GPUInfo]:
"""Create one or more synthetic GPUs from CLI-style values."""
names = parse_synthetic_gpu_specs(values)
if vram_override_gb is not None and len(names) != 1:
raise ValueError(
"--vram currently supports exactly one simulated GPU. "
"For multi-GPU simulation, specify known GPU names and omit --vram."
)
return [create_synthetic_gpu(name, vram_override_gb) for name in names]
def create_synthetic_gpu(name: str, vram_override_gb: float | None = None) -> GPUInfo:
"""Create a synthetic GPUInfo from a GPU name.
Looks up specs from the dbgpu database (2000+ GPUs).
Args:
name: GPU name (e.g. "RTX 4090", "RX 7900 XTX").
vram_override_gb: Override VRAM in GB. Required if GPU not in database.
Returns:
GPUInfo with ``(simulated)`` suffix in the name.
Raises:
ValueError: If GPU is not found and no vram_override_gb given.
"""
_last_suggestions.clear()
amd_shared_memory_apu = _is_amd_shared_memory_apu(name)
curated = _lookup_curated_spec(name)
# Apple Silicon short-circuit: dbgpu has no Apple entries, so we check
# first to avoid fuzzy-matching "M1" against "Rage Mobility-M1".
apple_hit = _lookup_apple_silicon(name)
if apple_hit is not None:
canonical, vendor, default_vram_gb, bandwidth = apple_hit
vram_gb = vram_override_gb if vram_override_gb is not None else default_vram_gb
return GPUInfo(
name=f"{canonical} (simulated)",
vendor=vendor,
vram_bytes=int(vram_gb * _GiB),
memory_bandwidth_gbps=bandwidth,
shared_memory=True,
vram_overridden=vram_override_gb is not None,
)
spec = _lookup_dbgpu(name)
# VRAM
if vram_override_gb is not None:
vram_bytes = int(vram_override_gb * _GiB)
elif spec is not None and spec.memory_size_gb:
vram_bytes = int(spec.memory_size_gb * _GiB)
elif curated is not None:
vram_bytes = int(curated.vram_gb * _GiB)
else:
msg = f"Unknown GPU '{name}'."
if _last_suggestions:
candidates = ", ".join(n for n, _ in _last_suggestions)
msg += f" Did you mean: {candidates}?"
msg += " Use --vram to specify VRAM in GB."
raise ValueError(msg)
# Bandwidth
bandwidth: float | None = None
if spec is not None and spec.memory_bandwidth_gb_s:
bandwidth = spec.memory_bandwidth_gb_s
if bandwidth is None and curated is not None:
bandwidth = curated.memory_bandwidth_gbps
if bandwidth is None:
bandwidth = _lookup_static_bandwidth(name)
# Compute capability (CUDA version in dbgpu = compute capability)
compute_cap: tuple[int, int] | None = None
if spec is not None and spec.cuda_major_version is not None:
compute_cap = (spec.cuda_major_version, spec.cuda_minor_version or 0)
# Vendor
vendor = "nvidia"
if spec is not None:
vendor = _MANUFACTURER_TO_VENDOR.get(spec.manufacturer, "nvidia")
elif curated is not None:
vendor = curated.vendor
elif amd_shared_memory_apu:
vendor = "amd"
if spec is not None:
display_name = spec.name
elif curated is not None:
display_name = curated.name
else:
display_name = name
return GPUInfo(
name=f"{display_name} (simulated)",
vendor=vendor,
vram_bytes=vram_bytes,
compute_capability=compute_cap,
memory_bandwidth_gbps=bandwidth,
shared_memory=curated.shared_memory if curated else amd_shared_memory_apu,
vram_overridden=vram_override_gb is not None,
)
src/whichllm/hardware/intel.py
+136
View File
@@ -0,0 +1,136 @@
"""Intel integrated GPU detection on Linux."""
from __future__ import annotations
import logging
import subprocess
from pathlib import Path
from whichllm.constants import (
CURATED_GPU_SPECS,
INTEL_PCI_DEVICE_NAMES,
CuratedGPUSpec,
_GiB,
)
from whichllm.hardware.types import GPUInfo
logger = logging.getLogger(__name__)
_DISPLAY_CLASSES = (
"vga compatible controller",
"3d controller",
"display controller",
)
def _normalize_lspci_name(line: str) -> str:
parts = [p.strip() for p in line.split('"') if p.strip() and p.strip() != "\t"]
for i, part in enumerate(parts):
if part.lower() == "intel corporation" and i + 1 < len(parts):
return parts[i + 1]
return "Intel Integrated Graphics"
def _detect_from_lspci() -> list[str]:
try:
result = subprocess.run(
["lspci", "-mm"],
capture_output=True,
text=True,
timeout=5,
)
except (FileNotFoundError, subprocess.TimeoutExpired):
logger.debug("lspci not available or timed out")
return []
if result.returncode != 0:
return []
names: list[str] = []
seen: set[str] = set()
for line in result.stdout.splitlines():
line_lower = line.lower()
if "intel" not in line_lower or not any(
display_class in line_lower for display_class in _DISPLAY_CLASSES
):
continue
name = _normalize_lspci_name(line)
if name not in seen:
names.append(name)
seen.add(name)
return names
def _detect_from_sysfs(drm_path: Path = Path("/sys/class/drm")) -> list[str]:
names: list[str] = []
seen: set[str] = set()
try:
cards = sorted(drm_path.glob("card[0-9]*"))
except OSError:
return []
for card in cards:
device = card / "device"
try:
vendor = (device / "vendor").read_text().strip().lower()
except OSError:
continue
if vendor != "0x8086":
continue
name = "Intel Integrated Graphics"
known_device = False
try:
device_id = (device / "device").read_text().strip().lower()
mapped_name = INTEL_PCI_DEVICE_NAMES.get(device_id)
if mapped_name:
name = mapped_name
known_device = True
except OSError:
pass
try:
product_name = (device / "product_name").read_text().strip()
if product_name and not known_device:
name = product_name
except OSError:
pass
if name not in seen:
names.append(name)
seen.add(name)
return names
def _lookup_curated_spec(name: str) -> CuratedGPUSpec | None:
name_upper = name.upper()
for key in sorted(CURATED_GPU_SPECS, key=len, reverse=True):
if key.upper() in name_upper:
return CURATED_GPU_SPECS[key]
return None
def _gpu_info_from_name(name: str) -> GPUInfo:
curated = _lookup_curated_spec(name)
if curated is not None:
return GPUInfo(
name=name,
vendor=curated.vendor,
vram_bytes=int(curated.vram_gb * _GiB),
memory_bandwidth_gbps=curated.memory_bandwidth_gbps,
shared_memory=curated.shared_memory,
)
return GPUInfo(
name=name,
vendor="intel",
vram_bytes=0,
shared_memory=True,
)
def detect_intel_gpus() -> list[GPUInfo]:
"""Detect Linux Intel iGPUs. Returns empty list on failure."""
names = _detect_from_lspci() or _detect_from_sysfs()
return [_gpu_info_from_name(name) for name in names]
src/whichllm/hardware/memory.py
+52
View File
@@ -0,0 +1,52 @@
"""RAM and disk space detection."""
from __future__ import annotations
import os
import shutil
import psutil
def detect_ram_bytes() -> int:
"""Get total physical RAM in bytes."""
return psutil.virtual_memory().total
def detect_available_ram_bytes() -> int:
"""Get currently available RAM bytes."""
return psutil.virtual_memory().available
def estimate_usable_ram(total: int) -> int:
"""Estimate RAM available for model loading after OS/background reserve.
Uses a bounded-reserve formula: total - clamp(total * 0.15, 4 GiB, 32 GiB).
"""
_GiB = 1024**3
reserve = int(total * 0.15)
reserve = max(4 * _GiB, min(reserve, 32 * _GiB))
return max(0, total - reserve)
def effective_usable_ram(total: int, budget: int | None = None) -> int:
"""Estimate usable RAM, optionally capped by a user/runtime budget."""
usable = estimate_usable_ram(total)
if budget is None:
return usable
return max(0, min(usable, budget))
def detect_disk_free_bytes(path: str | None = None) -> int:
"""Get free disk space in bytes at the given path.
Defaults to the user's home directory, which is more accurate
on macOS where / may be a read-only system volume.
"""
if path is None:
path = os.path.expanduser("~")
try:
usage = shutil.disk_usage(path)
return usage.free
except OSError:
return 0
src/whichllm/hardware/nvidia.py
+198
View File
@@ -0,0 +1,198 @@
"""NVIDIA GPU detection via NVML with nvidia-smi fallback."""
from __future__ import annotations
import logging
import re
import subprocess
from whichllm.constants import NVIDIA_COMPUTE_CAPABILITY, _GiB
from whichllm.hardware.gpu_db import _static_bandwidth, resolve_detected_bandwidth
from whichllm.hardware.types import GPUInfo
logger = logging.getLogger(__name__)
_NVIDIA_UNIFIED_MEMORY_MARKERS = ("GB10", "DGX SPARK")
def _lookup_compute_capability(name: str) -> tuple[int, int] | None:
name_upper = name.upper()
for key, cc in NVIDIA_COMPUTE_CAPABILITY.items():
if key.upper() in name_upper:
return cc
return None
def _lookup_bandwidth(name: str) -> float | None:
"""Curated GPU_BANDWIDTH lookup. Kept for regression tests; live detection
goes through ``resolve_detected_bandwidth``, which also consults dbgpu."""
return _static_bandwidth(name)
def _is_unified_memory_nvidia_gpu(name: str) -> bool:
name_upper = name.upper()
return any(marker in name_upper for marker in _NVIDIA_UNIFIED_MEMORY_MARKERS)
def _system_memory_bytes() -> int:
from whichllm.hardware.memory import detect_ram_bytes
ram_bytes = detect_ram_bytes()
if ram_bytes > 0:
return ram_bytes
return 128 * _GiB
def _make_nvidia_gpu(
name: str,
vram_bytes: int | None,
cuda_version: str | None = None,
mem_clock_mhz: float | None = None,
) -> GPUInfo:
shared_memory = _is_unified_memory_nvidia_gpu(name)
if shared_memory and (vram_bytes is None or vram_bytes <= 0):
vram_bytes = _system_memory_bytes()
elif vram_bytes is None:
vram_bytes = 0
return GPUInfo(
name=name,
vendor="nvidia",
vram_bytes=vram_bytes,
compute_capability=_lookup_compute_capability(name),
cuda_version=cuda_version,
memory_bandwidth_gbps=resolve_detected_bandwidth(
name, vram_bytes, mem_clock_mhz
),
shared_memory=shared_memory,
)
def _run_smi_query(fields: str) -> str:
result = subprocess.run(
["nvidia-smi", f"--query-gpu={fields}", "--format=csv,noheader,nounits"],
capture_output=True,
check=True,
text=True,
timeout=5,
)
return result.stdout
def _detect_nvidia_gpus_via_smi() -> list[GPUInfo]:
"""Detect NVIDIA GPUs using nvidia-smi when Python NVML cannot load.
The max memory clock (used to disambiguate same-name memory variants such as
GTX 1650 GDDR5/GDDR6) is queried opportunistically. If the 3-field query
fails on a driver/card that rejects ``clocks.max.memory``, we retry the
original 2-field query so a missing optional field never wipes out detection.
"""
try:
stdout = _run_smi_query("name,memory.total,clocks.max.memory")
except (FileNotFoundError, subprocess.SubprocessError, OSError) as e:
logger.debug(f"nvidia-smi 3-field query failed ({e}); retrying without clock")
try:
stdout = _run_smi_query("name,memory.total")
except (FileNotFoundError, subprocess.SubprocessError, OSError) as e2:
logger.debug(f"nvidia-smi fallback failed: {e2}")
return []
gpus: list[GPUInfo] = []
for line in stdout.splitlines():
parts = [part.strip() for part in line.split(",", maxsplit=2)]
if len(parts) < 2 or not parts[0]:
continue
name, memory_mib_text = parts[0], parts[1]
# clocks.max.memory is MHz (or "[N/A]" on some cards/drivers).
mem_clock_mhz: float | None = None
if len(parts) == 3:
clock_match = re.search(r"\d+", parts[2])
if clock_match:
mem_clock_mhz = float(clock_match.group(0))
match = re.search(r"\d+", memory_mib_text)
if not match:
if not _is_unified_memory_nvidia_gpu(name):
logger.debug(f"Could not parse nvidia-smi memory value: {line!r}")
continue
gpus.append(_make_nvidia_gpu(name, None, mem_clock_mhz=mem_clock_mhz))
continue
memory_mib = int(match.group(0))
gpus.append(
_make_nvidia_gpu(name, memory_mib * 1024**2, mem_clock_mhz=mem_clock_mhz)
)
return gpus
def detect_nvidia_gpus() -> list[GPUInfo]:
"""Detect NVIDIA GPUs. Returns empty list on failure."""
try:
import pynvml
except ImportError:
logger.debug("pynvml not installed, trying nvidia-smi fallback")
return _detect_nvidia_gpus_via_smi()
try:
pynvml.nvmlInit()
except pynvml.NVMLError:
logger.debug("NVML init failed, trying nvidia-smi fallback")
return _detect_nvidia_gpus_via_smi()
gpus: list[GPUInfo] = []
try:
count = pynvml.nvmlDeviceGetCount()
# Get CUDA driver version
try:
pynvml.nvmlSystemGetDriverVersion() # ensure driver is accessible
cuda_version = pynvml.nvmlSystemGetCudaDriverVersion_v2()
cuda_str = f"{cuda_version // 1000}.{(cuda_version % 1000) // 10}"
except Exception:
cuda_str = None
for i in range(count):
handle = pynvml.nvmlDeviceGetHandleByIndex(i)
name = pynvml.nvmlDeviceGetName(handle)
if isinstance(name, bytes):
name = name.decode("utf-8")
try:
mem_info = pynvml.nvmlDeviceGetMemoryInfo(handle)
vram_bytes = mem_info.total
except pynvml.NVMLError:
if not _is_unified_memory_nvidia_gpu(name):
raise
logger.debug(f"NVML did not report dedicated memory for {name}")
vram_bytes = None
# Max memory clock (MHz) disambiguates same-name memory variants
# (e.g. GTX 1650 GDDR5 vs GDDR6). Optional: not all drivers expose it.
try:
mem_clock_mhz: float | None = float(
pynvml.nvmlDeviceGetMaxClockInfo(handle, pynvml.NVML_CLOCK_MEM)
)
except Exception as clock_err:
# Optional: without it, same-name memory variants fall back to the
# curated default. Log so an unexpected under-serve is diagnosable.
logger.debug(
f"max mem clock unavailable for {name} "
f"({clock_err}); bandwidth from name only"
)
mem_clock_mhz = None
gpus.append(_make_nvidia_gpu(name, vram_bytes, cuda_str, mem_clock_mhz))
except pynvml.NVMLError as e:
logger.debug(f"Error enumerating NVIDIA GPUs: {e}")
finally:
try:
pynvml.nvmlShutdown()
except Exception:
pass
if gpus:
return gpus
logger.debug("NVML returned no NVIDIA GPUs, trying nvidia-smi fallback")
return _detect_nvidia_gpus_via_smi()
src/whichllm/hardware/types.py
+31
View File
@@ -0,0 +1,31 @@
from __future__ import annotations
from dataclasses import dataclass, field
@dataclass
class GPUInfo:
name: str
vendor: str # "nvidia" | "amd" | "apple" | "intel"
vram_bytes: int
usable_vram_bytes: int | None = None
compute_capability: tuple[int, int] | None = None # NVIDIA only
cuda_version: str | None = None
rocm_version: str | None = None
memory_bandwidth_gbps: float | None = None # from lookup table
shared_memory: bool = False
vram_overridden: bool = False
@dataclass
class HardwareInfo:
gpus: list[GPUInfo] = field(default_factory=list)
cpu_name: str = "Unknown"
cpu_cores: int = 0
has_avx2: bool = False
has_avx512: bool = False
ram_bytes: int = 0
ram_budget_bytes: int | None = None
disk_free_bytes: int = 0
os: str = "linux" # "linux" | "darwin" | "windows"
budget_notes: list[str] = field(default_factory=list)
src/whichllm/hardware/windows.py
+182
View File
@@ -0,0 +1,182 @@
"""Windows GPU detection via Win32_VideoController."""
from __future__ import annotations
import json
import logging
import re
import subprocess
from whichllm.constants import AMD_SHARED_MEMORY_APU_MARKERS, _GiB
from whichllm.hardware.gpu_db import resolve_detected_bandwidth
from whichllm.hardware.types import GPUInfo
logger = logging.getLogger(__name__)
_WINDOWS_DISCRETE_VRAM_FLOORS: tuple[tuple[str, int], ...] = (
# AMD sells RX 9060 XT in 8 GB and 16 GB variants. WMI AdapterRAM is a
# uint32 and can report ~4 GB for larger cards, so keep a conservative
# known floor instead of trusting the capped value.
("RX 9060 XT", 8 * _GiB),
)
def _vendor_from_name(name: str) -> str | None:
name_lower = name.lower()
if any(token in name_lower for token in ("amd", "radeon")):
return "amd"
if "intel" in name_lower:
return "intel"
return None
def _parse_memory_value(value: object) -> int:
if value is None:
return 0
try:
ram = int(value)
except (TypeError, ValueError):
return 0
return max(ram, 0)
def _is_amd_shared_memory_apu(name: str) -> bool:
name_upper = name.upper()
return any(marker in name_upper for marker in AMD_SHARED_MEMORY_APU_MARKERS)
def _is_intel_discrete_gpu(name: str) -> bool:
return (
re.search(
r"\barc(?:\(tm\))?\s+(?:pro\s+)?[ab]\d{2,3}",
name,
re.IGNORECASE,
)
is not None
)
def _is_intel_shared_memory_gpu(name: str, vram_bytes: int) -> bool:
name_lower = name.lower()
if _is_intel_discrete_gpu(name):
return False
if any(
token in name_lower
for token in (
"uhd",
"iris",
" xe",
"hd graphics",
"arc(tm) graphics",
"intel(r) graphics",
)
):
return True
return vram_bytes < 2 * _GiB
def _is_shared_memory_gpu(name: str, vendor: str, vram_bytes: int) -> bool:
if vendor == "amd":
return _is_amd_shared_memory_apu(name)
if vendor == "intel":
return _is_intel_shared_memory_gpu(name, vram_bytes)
return False
def _apply_discrete_vram_floor(name: str, vram_bytes: int) -> int:
name_upper = name.upper()
for marker, floor in _WINDOWS_DISCRETE_VRAM_FLOORS:
if marker in name_upper and 0 < vram_bytes < floor:
return floor
return vram_bytes
def _memory_from_entry(entry: dict) -> int:
dedicated = _parse_memory_value(entry.get("DedicatedVideoMemory"))
if dedicated > 0:
return dedicated
return _parse_memory_value(entry.get("AdapterRAM"))
def detect_windows_gpus() -> list[GPUInfo]:
"""Detect non-NVIDIA Windows GPUs. Returns empty list on failure."""
try:
result = subprocess.run(
[
"powershell",
"-NoProfile",
"-Command",
(
"$controllers = Get-CimInstance Win32_VideoController; "
"$controllers | ForEach-Object { "
"$dedicated = $null; "
"if ($_.PNPDeviceID) { "
"try { "
"$enumPath = 'Registry::HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Enum\\' "
"+ $_.PNPDeviceID + '\\Device Parameters'; "
"$enumProps = Get-ItemProperty -LiteralPath $enumPath -ErrorAction Stop; "
"if ($enumProps.VideoID) { "
"$videoPath = 'Registry::HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\Video\\' "
"+ $enumProps.VideoID + '\\0000'; "
"$videoProps = Get-ItemProperty -LiteralPath $videoPath -ErrorAction Stop; "
"$dedicated = $videoProps.'HardwareInformation.qwMemorySize'; "
"} "
"} catch {} "
"} "
"[PSCustomObject]@{"
"Name=$_.Name; "
"AdapterRAM=$_.AdapterRAM; "
"DedicatedVideoMemory=$dedicated"
"} "
"} | ConvertTo-Json -Depth 3"
),
],
capture_output=True,
text=True,
timeout=10,
)
except (FileNotFoundError, subprocess.SubprocessError, OSError) as e:
logger.debug(f"Windows GPU detection failed: {e}")
return []
if result.returncode != 0 or not result.stdout.strip():
return []
try:
data = json.loads(result.stdout)
except json.JSONDecodeError:
logger.debug("Failed to parse Windows GPU JSON")
return []
entries = data if isinstance(data, list) else [data]
gpus: list[GPUInfo] = []
seen: set[str] = set()
for entry in entries:
if not isinstance(entry, dict):
continue
name = str(entry.get("Name") or "").strip()
if not name:
continue
vendor = _vendor_from_name(name)
if vendor is None:
continue
vram_bytes = _memory_from_entry(entry)
shared_memory = _is_shared_memory_gpu(name, vendor, vram_bytes)
if shared_memory:
vram_bytes = 0
else:
vram_bytes = _apply_discrete_vram_floor(name, vram_bytes)
key = f"{vendor}:{name}:{vram_bytes}"
if key in seen:
continue
seen.add(key)
gpus.append(
GPUInfo(
name=name,
vendor=vendor,
vram_bytes=vram_bytes,
memory_bandwidth_gbps=resolve_detected_bandwidth(name, vram_bytes),
shared_memory=shared_memory,
)
)
return gpus
src/whichllm/models/__init__.py
View File
src/whichllm/models/artifacts.py
+105
View File
@@ -0,0 +1,105 @@
"""Resolve runnable/downloadable model artifacts for ranked recommendations."""
from __future__ import annotations
from whichllm.engine.types import CompatibilityResult
from whichllm.models.types import GGUFVariant, ModelInfo
def find_gguf_variant(model: ModelInfo, quant_type: str) -> GGUFVariant | None:
"""Return the model's GGUF variant for a quantization type."""
for variant in model.gguf_variants:
if variant.quant_type.upper() == quant_type.upper():
return variant
return None
def is_same_model_family(candidate: ModelInfo, selected: ModelInfo) -> bool:
"""Return whether two repos represent the same base model family."""
if candidate.id == selected.id:
return True
if candidate.family_id and selected.family_id:
if candidate.family_id == selected.family_id:
return True
if candidate.base_model and candidate.base_model == selected.id:
return True
if selected.base_model and selected.base_model == candidate.id:
return True
if candidate.base_model and selected.base_model:
return candidate.base_model == selected.base_model
return False
def has_compatible_parameter_count(candidate: ModelInfo, selected: ModelInfo) -> bool:
"""Reject artifact repos that are clearly a different model size."""
if candidate.parameter_count <= 0 or selected.parameter_count <= 0:
return True
smaller = min(candidate.parameter_count, selected.parameter_count)
larger = max(candidate.parameter_count, selected.parameter_count)
return (larger / smaller) <= 2.0
def resolve_ranked_gguf_artifact(
selected_model: ModelInfo,
selected_variant: GGUFVariant,
models: list[ModelInfo],
quant_filter: str | None = None,
) -> tuple[ModelInfo, GGUFVariant] | None:
"""Resolve a ranked GGUF candidate to a real HF repo/file.
The ranker may synthesize GGUF variants for official safetensors-only repos
so they can be scored realistically. Output surfaces and `run` need the
actual GGUF repository and filename when one exists.
"""
desired_quant = quant_filter or selected_variant.quant_type
if selected_model.gguf_variants:
variant = find_gguf_variant(selected_model, desired_quant)
return (selected_model, variant) if variant else None
candidates: list[tuple[bool, int, int, ModelInfo, GGUFVariant]] = []
for model in models:
if not model.gguf_variants or not is_same_model_family(model, selected_model):
continue
if not has_compatible_parameter_count(model, selected_model):
continue
variant = find_gguf_variant(model, desired_quant)
if not variant:
continue
explicit_base = model.base_model == selected_model.id
candidates.append(
(
explicit_base,
model.downloads,
model.likes,
model,
variant,
)
)
if not candidates:
return None
_, _, _, model, variant = max(candidates, key=lambda item: item[:3])
return model, variant
def attach_resolved_artifacts(
results: list[CompatibilityResult],
models: list[ModelInfo],
quant_filter: str | None = None,
) -> None:
"""Populate artifact fields on ranked results when a real artifact exists."""
for result in results:
result.artifact_model = None
result.artifact_variant = None
if not result.gguf_variant:
continue
resolved = resolve_ranked_gguf_artifact(
result.model,
result.gguf_variant,
models,
quant_filter=quant_filter,
)
if resolved:
result.artifact_model, result.artifact_variant = resolved
src/whichllm/models/benchmark.py
+90
View File
@@ -0,0 +1,90 @@
"""Compatibility shim for benchmark fetching and lookup helpers.
The implementation is split by responsibility:
- ``benchmark_cache`` for cache I/O
- ``benchmark_fetch`` for source fetching and layered merge policy
- ``benchmark_lineage`` for frozen-score recency demotion
- ``benchmark_index`` for score indices and model-line interpolation
- ``benchmark_lookup`` for evidence lookup and inheritance rules
Existing imports from ``whichllm.models.benchmark`` are re-exported here.
"""
from __future__ import annotations
from whichllm.models import benchmark_cache as _cache_module
from whichllm.models.benchmark_cache import DEFAULT_TTL_SECONDS
from whichllm.models.benchmark_fetch import fetch_benchmark_scores
from whichllm.models.benchmark_index import (
_extract_model_lines,
_extract_params_b_from_id,
_interpolate_line_score,
build_line_bucket_index,
build_score_index,
)
from whichllm.models.benchmark_lineage import (
_apply_lineage_recency_demotion,
_build_lineage_regex,
_lineage_recency_factor,
)
from whichllm.models.benchmark_lookup import (
_REPO_SUFFIXES,
_append_unique,
_generate_candidates,
_generate_score_name_candidates,
_params_compatible,
_strip_repo_suffix,
_try_lookup,
lookup_benchmark,
lookup_benchmark_evidence,
)
from whichllm.models.benchmark_types import BenchmarkEvidence
CACHE_DIR = _cache_module.CACHE_DIR
BENCHMARK_CACHE = _cache_module.BENCHMARK_CACHE
def _sync_cache_module_globals() -> None:
_cache_module.CACHE_DIR = CACHE_DIR
_cache_module.BENCHMARK_CACHE = BENCHMARK_CACHE
def load_benchmark_cache() -> dict[str, float] | None:
"""Load cached benchmark scores through the legacy shim globals."""
_sync_cache_module_globals()
return _cache_module.load_benchmark_cache()
def save_benchmark_cache(scores: dict[str, float]) -> None:
"""Save cached benchmark scores through the legacy shim globals."""
_sync_cache_module_globals()
_cache_module.save_benchmark_cache(scores)
__all__ = [
"BENCHMARK_CACHE",
"CACHE_DIR",
"DEFAULT_TTL_SECONDS",
"BenchmarkEvidence",
"_REPO_SUFFIXES",
"_append_unique",
"_apply_lineage_recency_demotion",
"_build_lineage_regex",
"_extract_model_lines",
"_extract_params_b_from_id",
"_generate_candidates",
"_generate_score_name_candidates",
"_interpolate_line_score",
"_lineage_recency_factor",
"_params_compatible",
"_strip_repo_suffix",
"_try_lookup",
"build_line_bucket_index",
"build_score_index",
"fetch_benchmark_scores",
"load_benchmark_cache",
"lookup_benchmark",
"lookup_benchmark_evidence",
"save_benchmark_cache",
]
src/whichllm/models/benchmark_cache.py
+39
View File
@@ -0,0 +1,39 @@
"""Benchmark score cache helpers."""
from __future__ import annotations
import json
import logging
import time
from whichllm.utils import _cache_dir
logger = logging.getLogger(__name__)
CACHE_DIR = _cache_dir()
BENCHMARK_CACHE = CACHE_DIR / "benchmark.json"
DEFAULT_TTL_SECONDS = 24 * 3600 # 24 hours
def load_benchmark_cache() -> dict[str, float] | None:
"""Load cached benchmark scores. Returns None if expired or missing."""
if not BENCHMARK_CACHE.exists():
return None
try:
data = json.loads(BENCHMARK_CACHE.read_text(encoding="utf-8"))
cached_at = data.get("cached_at", 0)
if time.time() - cached_at > DEFAULT_TTL_SECONDS:
logger.debug("Benchmark cache expired")
return None
return data.get("scores", {})
except (json.JSONDecodeError, KeyError) as e:
logger.debug(f"Benchmark cache corrupted: {e}")
return None
def save_benchmark_cache(scores: dict[str, float]) -> None:
"""Save benchmark scores to cache."""
CACHE_DIR.mkdir(parents=True, exist_ok=True)
data = {"cached_at": time.time(), "scores": scores}
BENCHMARK_CACHE.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
logger.debug(f"Saved {len(scores)} benchmark scores to cache")
src/whichllm/models/benchmark_fetch.py
+119
View File
@@ -0,0 +1,119 @@
"""Fetch and merge benchmark scores from source adapters."""
from __future__ import annotations
import asyncio
import logging
import httpx
from whichllm.models.benchmark_lineage import _apply_lineage_recency_demotion
from whichllm.models.http import DEFAULT_ACCEPT_ENCODING
from whichllm.utils import _current_version
logger = logging.getLogger(__name__)
async def fetch_benchmark_scores() -> dict[str, float]:
"""Fetch and combine benchmark scores from multiple sources.
Sources, merged in this order (later overwrites earlier on conflict):
1. Open LLM Leaderboard v2 (archived 2025-06)
2. Chatbot Arena ELO (frozen 2025-07-17)
3. LiveBench (vendored snapshot)
4. Aider polyglot (coding-specific)
5. Artificial Analysis Intelligence Index
6. Vision-language capability index
Returns dict mapping model_id -> normalized score (0-100). All network
sources are fetched concurrently; failures are logged and skipped.
"""
from whichllm.models import benchmark_sources
async with httpx.AsyncClient(
timeout=30.0,
follow_redirects=True,
headers={
"Accept-Encoding": DEFAULT_ACCEPT_ENCODING,
"User-Agent": f"whichllm/{_current_version()}",
},
) as client:
leaderboard_task = asyncio.create_task(
benchmark_sources.fetch_leaderboard_with_fallback(client)
)
arena_task = asyncio.create_task(benchmark_sources.fetch_arena_scores(client))
aa_task = asyncio.create_task(benchmark_sources.fetch_aa_index_scores(client))
aider_task = asyncio.create_task(
benchmark_sources.fetch_aider_polyglot_scores(client)
)
vision_task = asyncio.create_task(benchmark_sources.fetch_vision_scores(client))
(
lb_result,
arena_result,
aa_result,
aider_result,
vision_result,
) = await asyncio.gather(
leaderboard_task,
arena_task,
aa_task,
aider_task,
vision_task,
return_exceptions=True,
)
frozen: dict[str, float] = {}
current: dict[str, float] = {}
if isinstance(lb_result, BaseException):
logger.warning(f"Leaderboard fetch failed: {lb_result}")
else:
frozen.update(lb_result)
logger.debug(f"Leaderboard: {len(lb_result)} scores (frozen)")
if isinstance(arena_result, BaseException):
logger.warning(f"Arena fetch failed, using fallback: {arena_result}")
else:
for k, v in arena_result.items():
if frozen.get(k, 0.0) < v:
frozen[k] = v
logger.debug(f"Arena: {len(arena_result)} scores (frozen)")
livebench_result = benchmark_sources.get_livebench_data()
for k, v in livebench_result.items():
if current.get(k, 0.0) < v:
current[k] = v
logger.debug(f"LiveBench: {len(livebench_result)} scores (current)")
if isinstance(aa_result, BaseException):
logger.warning(f"AA Index fetch failed, will use fallback: {aa_result}")
aa_result = benchmark_sources.get_aa_curated_fallback()
for k, v in aa_result.items():
if current.get(k, 0.0) < v:
current[k] = v
logger.debug(f"AA Index: {len(aa_result)} scores (current)")
if isinstance(aider_result, BaseException):
logger.warning(f"Aider fetch failed: {aider_result}")
else:
for k, v in aider_result.items():
if current.get(k, 0.0) < v * 0.85:
current[k] = v * 0.85
logger.debug(f"Aider polyglot: {len(aider_result)} scores (current, 0.85x)")
if isinstance(vision_result, BaseException):
logger.warning(f"Vision fetch failed: {vision_result}")
else:
for k, v in vision_result.items():
if current.get(k, 0.0) < v:
current[k] = v
logger.debug(f"Vision: {len(vision_result)} scores (current)")
combined: dict[str, float] = dict(frozen)
combined.update(current)
combined = _apply_lineage_recency_demotion(combined, frozen, current)
logger.debug(f"Combined: {len(combined)} benchmark scores")
return combined
src/whichllm/models/benchmark_index.py
+117
View File
@@ -0,0 +1,117 @@
"""Benchmark score indexing and model-line interpolation."""
from __future__ import annotations
import math
import re
import statistics
def _extract_params_b_from_id(model_id: str) -> float | None:
"""Extract parameter size in billions from model ID text."""
lower = model_id.lower()
matches = re.findall(r"(\d+(?:\.\d+)?)b(?:-a\d+(?:\.\d+)?b)?", lower)
if not matches:
return None
try:
return max(float(v) for v in matches)
except ValueError:
return None
def _extract_model_lines(model_id: str) -> list[str]:
"""Extract model line candidates from a model ID, most specific first."""
if "/" not in model_id:
return []
lower = model_id.lower()
stripped = re.sub(r"-(gguf|awq|gptq|fp8|fp16|bf16|mxfp4|nvfp4)$", "", lower)
stripped = re.sub(r"-\d{4}(-hf)?$", "", stripped)
lines: list[str] = []
cleaned = re.sub(
r"-\d+(\.\d+)?b(-a\d+b)?(-[a-z][-a-z0-9]*)*$",
"",
stripped,
)
if cleaned != stripped and "/" in cleaned:
lines.append(cleaned)
for line in list(lines) + ([stripped] if not lines else []):
broader = re.sub(r"(\d+)\.\d+$", r"\1", line)
if broader != line and broader not in lines:
lines.append(broader)
return lines
def _interpolate_line_score(
bucket: list[tuple[float | None, float]],
params_b: float | None,
) -> tuple[float, float]:
"""Interpolate score from same-model-line benchmarks with confidence."""
if not bucket:
return 0.0, 0.0
valid = [(p, s) for p, s in bucket if p is not None]
if not valid:
vals = [s for _, s in bucket]
return statistics.median(vals), 0.25
if params_b is None or params_b <= 0:
vals = [s for _, s in valid]
return statistics.median(vals), 0.30
weighted: list[tuple[float, float, float]] = []
for p, s in valid:
assert p is not None
dist = abs(math.log2(max(params_b, 0.1) / max(p, 0.1)))
w = 1.0 / (0.35 + dist)
weighted.append((w, s, dist))
score = sum(w * s for w, s, _ in weighted) / sum(w for w, _, _ in weighted)
nearest = min(d for _, _, d in weighted)
if nearest <= 0.15:
conf = 0.45
elif nearest <= 0.50:
conf = 0.34
else:
conf = 0.26
return score, conf
def build_score_index(
scores: dict[str, float],
) -> tuple[dict[str, float], dict[str, float]]:
"""Build case-insensitive and model-line lookup indices."""
ci_index: dict[str, float] = {}
line_index: dict[str, float] = {}
for key, val in scores.items():
lk = key.lower()
if lk not in ci_index or val > ci_index[lk]:
ci_index[lk] = val
lines = _extract_model_lines(key)
if not lines and "/" in key:
lines = [lk]
for line in lines:
if line not in line_index or val > line_index[line]:
line_index[line] = val
return ci_index, line_index
def build_line_bucket_index(
scores: dict[str, float],
) -> dict[str, list[tuple[float | None, float]]]:
"""Build line -> [(params_b, score)] index for size-aware interpolation."""
buckets: dict[str, list[tuple[float | None, float]]] = {}
for key, val in scores.items():
params_b = _extract_params_b_from_id(key)
lines = _extract_model_lines(key)
if not lines and "/" in key:
lines = [key.lower()]
for line in lines:
buckets.setdefault(line, []).append((params_b, val))
return buckets
src/whichllm/models/benchmark_lineage.py
+63
View File
@@ -0,0 +1,63 @@
"""Lineage-aware recency demotion for frozen benchmark scores."""
from __future__ import annotations
import re
_LINEAGE_DEMOTION_REGEX = None
def _build_lineage_regex():
"""Compile MODEL_LINEAGE_VERSIONS once into (family, [(re, idx)]) form."""
global _LINEAGE_DEMOTION_REGEX
if _LINEAGE_DEMOTION_REGEX is not None:
return _LINEAGE_DEMOTION_REGEX
from whichllm.constants import MODEL_LINEAGE_VERSIONS
out = {}
for family, entries in MODEL_LINEAGE_VERSIONS.items():
compiled = [(re.compile(pat), idx) for pat, idx in entries]
max_idx = max(idx for _, idx in entries)
out[family] = (compiled, max_idx)
_LINEAGE_DEMOTION_REGEX = out
return out
def _lineage_recency_factor(model_id: str) -> float:
"""Return a multiplicative recency factor for frozen-only scores.
Newest generation in a known family -> 1.0 (no demotion). Each generation
older -> another 12% off. Unknown families -> 1.0.
"""
if not model_id:
return 1.0
lower = model_id.lower()
families = _build_lineage_regex()
best_factor = 1.0
for family, (patterns, max_idx) in families.items():
for regex, idx in patterns:
if regex.search(lower):
gens_old = max(0, max_idx - idx)
factor = max(0.55, 1.0 - 0.12 * gens_old)
if factor < best_factor:
best_factor = factor
break
return best_factor
def _apply_lineage_recency_demotion(
combined: dict[str, float],
frozen: dict[str, float],
current: dict[str, float],
) -> dict[str, float]:
"""Multiply frozen-only entries by a lineage-derived recency factor."""
if not combined:
return combined
out: dict[str, float] = {}
for k, v in combined.items():
if k in current:
out[k] = v
continue
factor = _lineage_recency_factor(k)
out[k] = round(v * factor, 1)
return out
src/whichllm/models/benchmark_lookup.py
+192
View File
@@ -0,0 +1,192 @@
"""Benchmark evidence lookup and inheritance rules."""
from __future__ import annotations
from whichllm.models.benchmark_index import (
_extract_model_lines,
_extract_params_b_from_id,
_interpolate_line_score,
build_line_bucket_index,
build_score_index,
)
from whichllm.models.benchmark_types import BenchmarkEvidence
def _try_lookup(
candidate: str, scores: dict[str, float], ci_index: dict[str, float]
) -> float | None:
"""Try exact match, then case-insensitive match."""
if candidate in scores:
return scores[candidate]
lc = candidate.lower()
if lc in ci_index:
return ci_index[lc]
return None
_REPO_SUFFIXES = ("-GGUF", "-gguf", "-AWQ", "-GPTQ", "-FP8", "-fp8", "-BF16", "-bf16")
def _generate_candidates(model_id: str) -> list[str]:
"""Generate candidate IDs to look up for a model."""
candidates = [model_id]
for suffix in _REPO_SUFFIXES:
if model_id.endswith(suffix):
candidates.append(model_id[: -len(suffix)])
break
base = candidates[-1]
if base.endswith("-Instruct"):
candidates.append(base[: -len("-Instruct")])
else:
candidates.append(base + "-Instruct")
return candidates
def _append_unique(candidates: list[str], candidate: str) -> None:
if candidate and candidate not in candidates:
candidates.append(candidate)
def _strip_repo_suffix(model_id: str) -> str:
for suffix in _REPO_SUFFIXES:
if model_id.endswith(suffix):
return model_id[: -len(suffix)]
return model_id
def _generate_score_name_candidates(
model_id: str, scores: dict[str, float]
) -> list[str]:
"""Match community repo names to benchmark IDs with the same model name."""
stripped = _strip_repo_suffix(model_id)
repo_name = stripped.rsplit("/", 1)[-1]
model_names = [repo_name]
explicit_candidates: list[str] = []
if "_" in repo_name:
org, name = repo_name.split("_", 1)
if org and name:
_append_unique(explicit_candidates, f"{org}/{name}")
_append_unique(model_names, name)
score_candidates: list[str] = []
wanted_names = {name.lower() for name in model_names if name}
for score_id in scores:
score_name = score_id.rsplit("/", 1)[-1].lower()
if score_name in wanted_names:
_append_unique(score_candidates, score_id)
return explicit_candidates + [
candidate
for candidate in score_candidates
if candidate not in explicit_candidates
]
def lookup_benchmark(
model_id: str,
base_model: str | None,
scores: dict[str, float],
ci_index: dict[str, float] | None = None,
line_index: dict[str, float] | None = None,
) -> tuple[float, bool] | None:
"""Backward-compatible benchmark lookup helper."""
evidence = lookup_benchmark_evidence(
model_id,
base_model,
scores,
ci_index=ci_index,
line_index=line_index,
)
if evidence.score is None:
return None
return evidence.score, evidence.source == "direct"
def _params_compatible(actual_b: float | None, ref_id: str) -> bool:
"""Reject benchmark inheritance when actual and reference sizes diverge."""
if actual_b is None or actual_b <= 0:
return True
ref_b = _extract_params_b_from_id(ref_id)
if ref_b is None or ref_b <= 0:
return True
ratio = actual_b / ref_b
return 0.5 <= ratio <= 2.0
def lookup_benchmark_evidence(
model_id: str,
base_model: str | None,
scores: dict[str, float],
ci_index: dict[str, float] | None = None,
line_index: dict[str, float] | None = None,
line_bucket_index: dict[str, list[tuple[float | None, float]]] | None = None,
self_reported_score: float | None = None,
actual_params_b: float | None = None,
) -> BenchmarkEvidence:
"""Look up benchmark evidence with confidence."""
if ci_index is None or line_index is None:
ci_index, line_index = build_score_index(scores)
if line_bucket_index is None:
line_bucket_index = build_line_bucket_index(scores)
direct_result = _try_lookup(model_id, scores, ci_index)
if direct_result is not None:
return BenchmarkEvidence(score=direct_result, confidence=1.0, source="direct")
variant_candidates = _generate_candidates(model_id)[1:]
for candidate in _generate_score_name_candidates(model_id, scores):
_append_unique(variant_candidates, candidate)
for candidate in variant_candidates:
result = _try_lookup(candidate, scores, ci_index)
if result is not None:
if not _params_compatible(actual_params_b, candidate):
continue
return BenchmarkEvidence(score=result, confidence=0.55, source="variant")
if base_model:
for candidate in _generate_candidates(base_model):
result = _try_lookup(candidate, scores, ci_index)
if result is not None:
if not _params_compatible(actual_params_b, candidate):
continue
return BenchmarkEvidence(
score=result, confidence=0.60, source="base_model"
)
size_hint = (
actual_params_b
or _extract_params_b_from_id(model_id)
or _extract_params_b_from_id(base_model or "")
)
for mid in (model_id, base_model):
if mid:
for line in _extract_model_lines(mid):
if line in line_bucket_index:
score, conf = _interpolate_line_score(
line_bucket_index[line], size_hint
)
if score > 0:
return BenchmarkEvidence(
score=score, confidence=conf, source="line_interp"
)
if line in line_index:
return BenchmarkEvidence(
score=line_index[line], confidence=0.22, source="line_interp"
)
if (
self_reported_score is not None
and isinstance(self_reported_score, (int, float))
and self_reported_score > 0
):
return BenchmarkEvidence(
score=float(self_reported_score),
confidence=0.40,
source="self_reported",
)
return BenchmarkEvidence(score=None, confidence=0.0, source="none")
src/whichllm/models/benchmark_sources/__init__.py
+43
View File
@@ -0,0 +1,43 @@
"""External benchmark sources beyond Chatbot Arena and Open LLM Leaderboard.
Each module here fetches an independent leaderboard / index, normalizes it to
the same 0-100 scale, and returns a ``dict[str, float]`` keyed by HuggingFace
model id (or a list of synonyms).
The functions are intentionally defensive: if a source is unreachable or
returns malformed data, they log a warning and return an empty dict so the
main benchmark merge pipeline does not abort.
"""
from whichllm.models.benchmark_sources.aa_index import (
fetch_aa_index_scores,
get_aa_curated_fallback,
)
from whichllm.models.benchmark_sources.aider import fetch_aider_polyglot_scores
from whichllm.models.benchmark_sources.chatbot_arena import fetch_arena_scores
from whichllm.models.benchmark_sources.livebench import (
get_livebench_data,
)
from whichllm.models.benchmark_sources.open_llm_leaderboard import (
fetch_leaderboard_with_fallback,
)
from whichllm.models.benchmark_sources.vision import fetch_vision_scores
# Newest curated-fallback date across all sources. Live scrapes are merged
# on top when reachable, but they frequently are not (the leaderboard
# spaces change their JSON shape), so the user-visible ranking is anchored
# to this snapshot. Surface it in the CLI so a stale recommendation is
# self-evident rather than silently trusted. Bump this whenever any
# *_FALLBACK_* dict is refreshed.
BENCHMARK_SNAPSHOT = "2026-05"
__all__ = [
"BENCHMARK_SNAPSHOT",
"fetch_aa_index_scores",
"fetch_aider_polyglot_scores",
"fetch_arena_scores",
"fetch_leaderboard_with_fallback",
"fetch_vision_scores",
"get_aa_curated_fallback",
"get_livebench_data",
]
src/whichllm/models/benchmark_sources/aa_index.py
+386
View File
@@ -0,0 +1,386 @@
"""Artificial Analysis Intelligence Index source.
AA publishes a model-quality index (https://artificialanalysis.ai/) that
covers post-2025-08 frontier releases (DeepSeek V4, GLM-5, Kimi K2.6,
MiMo V2.5, Qwen3.6, etc.) that whichllm's primary sources have stopped
tracking. The index is exposed via the JSON payload embedded in
``__NEXT_DATA__`` on the leaderboard page.
The fetcher is defensive: any failure (network, schema drift, parsing) is
caught and an empty dict is returned so it never blocks the main benchmark
pipeline.
"""
from __future__ import annotations
import json
import logging
import re
import httpx
from whichllm.models.benchmark_sources.constants import _NEXT_DATA_RE
from whichllm.models.benchmark_sources.types import ExtractionFailed
from whichllm.models.benchmark_sources.utils import _walk
from whichllm.models.http import get_with_retries
logger = logging.getLogger(__name__)
# Display name -> list of (org_prefix, repo_name_candidates) tuples used to
# map AA-reported labels back to HuggingFace model IDs. Only the most common
# fully-open-weights releases need entries here; anything else is dropped.
AA_NAME_TO_HF_IDS: dict[str, list[str]] = {
"Kimi K2": ["moonshotai/Kimi-K2-Instruct", "moonshotai/Kimi-K2-Base"],
"Kimi K2-Thinking": ["moonshotai/Kimi-K2-Thinking"],
"DeepSeek V3": ["deepseek-ai/DeepSeek-V3", "deepseek-ai/DeepSeek-V3-0324"],
"DeepSeek V3.1": ["deepseek-ai/DeepSeek-V3.1"],
"DeepSeek V3.2": ["deepseek-ai/DeepSeek-V3.2"],
"DeepSeek V3.2-Exp": ["deepseek-ai/DeepSeek-V3.2-Exp"],
"DeepSeek V4 Pro": ["deepseek-ai/DeepSeek-V4-Pro"],
"DeepSeek V4 Flash": ["deepseek-ai/DeepSeek-V4-Flash"],
"DeepSeek R1": ["deepseek-ai/DeepSeek-R1"],
"DeepSeek R1-0528": ["deepseek-ai/DeepSeek-R1-0528"],
"DeepSeek R1-Distill 32B": ["deepseek-ai/DeepSeek-R1-Distill-Qwen-32B"],
"DeepSeek R1-Distill 14B": ["deepseek-ai/DeepSeek-R1-Distill-Qwen-14B"],
"DeepSeek R1-Distill 8B": ["deepseek-ai/DeepSeek-R1-Distill-Llama-8B"],
"QwQ 32B": ["Qwen/QwQ-32B"],
"Qwen3 4B Thinking": ["Qwen/Qwen3-4B-Thinking-2507"],
"MiMo V2.5": ["XiaomiMiMo/MiMo-V2.5"],
"MiMo V2.5 Pro": ["XiaomiMiMo/MiMo-V2.5-Pro"],
"MiMo V2 Flash": ["XiaomiMiMo/MiMo-V2-Flash"],
"GLM-4.5": ["zai-org/GLM-4.5", "zai-org/GLM-4.5-Air"],
"GLM-4.6": ["zai-org/GLM-4.6"],
"GLM-4.7": ["zai-org/GLM-4.7"],
"GLM-4.7-Flash": ["zai-org/GLM-4.7-Flash"],
"GLM-5": ["zai-org/GLM-5", "zai-org/GLM-5-FP8"],
"GLM-5.1": ["zai-org/GLM-5.1", "zai-org/GLM-5.1-FP8"],
"gpt-oss-20b": ["openai/gpt-oss-20b"],
"gpt-oss-120b": ["openai/gpt-oss-120b"],
"Qwen3-Next 80B-A3B": ["Qwen/Qwen3-Next-80B-A3B-Instruct"],
"Qwen3.5 397B-A17B": ["Qwen/Qwen3.5-397B-A17B"],
"Qwen3 235B-A22B": ["Qwen/Qwen3-235B-A22B"],
"Qwen3 32B": ["Qwen/Qwen3-32B"],
"Qwen3 14B": ["Qwen/Qwen3-14B"],
"Qwen3 8B": ["Qwen/Qwen3-8B"],
"Qwen3-VL 235B-A22B": ["Qwen/Qwen3-VL-235B-A22B-Instruct"],
"Llama 3.3 70B": ["meta-llama/Llama-3.3-70B-Instruct"],
"Llama 4 Scout": ["meta-llama/Llama-4-Scout-17B-16E-Instruct"],
"Llama 4 Maverick": ["meta-llama/Llama-4-Maverick-17B-128E-Instruct"],
"Gemma 3 27B": ["google/gemma-3-27b-it"],
"Gemma 3 12B": ["google/gemma-3-12b-it"],
"Gemma 4 31B": ["google/gemma-4-31b-it"],
"Gemma 4 26B-A4B": ["google/gemma-4-26b-a4b-it"],
"Mistral Large 2": ["mistralai/Mistral-Large-Instruct-2411"],
"Devstral Small": ["mistralai/Devstral-Small-2505"],
"Phi-4": ["microsoft/phi-4"],
"Command A": ["CohereForAI/c4ai-command-a-03-2025"],
"Command R+": [
"CohereForAI/c4ai-command-r-plus-08-2024",
"CohereForAI/c4ai-command-r-plus",
],
"MiniMax-M2": ["MiniMaxAI/MiniMax-M2"],
"MiniMax-M2.5": ["MiniMaxAI/MiniMax-M2.5"],
"Nemotron 3 Super 120B-A12B": ["nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16"],
"Nemotron 3 Nano 30B-A3B": [
"nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16",
"nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
],
}
# Bounds used to normalize raw AA index values onto the 0-100 scale the rest
# of the ranking system uses. AA reworked their Intelligence Index in
# 2026-Q2 and the open-weights distribution compressed sharply (top open model
# ~44, 8B-class ~7, weakest mapped repos ~3). The window is anchored by the
# same two-point fit as before: top open frontier (DeepSeek-V4-Pro = 44.3) → 95
# normalized, and 8B-class (Qwen3-8B = 7.4) → 40 normalized. This keeps a strong
# 8B model competitive with frozen-OLLB 7B scores while leaving headroom for
# frontier-tier models. On the reworked scale that fit puts the floor below
# zero; that is fine, live AA values are always positive and clamp at 0.
_AA_INDEX_MIN = -19.4
_AA_INDEX_MAX = 47.6
AA_LEADERBOARD_URL = "https://artificialanalysis.ai/leaderboards/models"
# Snapshot of the AA Intelligence Index (open-weights only), refreshed on
# 2026-06-29 from artificialanalysis.ai against their reworked index. Used as a
# fallback when the live HTML scrape returns no results (e.g. because the
# Next.js payload format changes again). Entries are raw AA index values,
# normalized through _normalize_aa_index() in get_aa_curated_fallback().
#
# Entries marked "live" are real values from the reworked index. Entries marked
# "peer" are models AA does not track; their raw value is set so that, under the
# current bounds, they reproduce their previous normalized score (hand-estimated
# LB-equivalents). On the reworked scale several peers fall below the index
# floor and read as negative raw; that is an artifact of reusing the AA-index
# field for non-AA estimates. _normalize_aa_index() clamps them at 0, so the
# normalized output stays correct. (We could instead store this snapshot as
# already-normalized 0-100 values, which would drop the negatives and decouple
# the fallback from future bound retunes, but that is a larger change than this
# issue calls for and is left for a follow-up if you want it.)
AA_INDEX_FALLBACK_2026_06_29: dict[str, float] = {
# Frontier MoE / very large
"moonshotai/Kimi-K2-Thinking": 32.7, # live
"moonshotai/Kimi-K2-Instruct": 19.4, # live
"XiaomiMiMo/MiMo-V2.5-Pro": 42.2, # live
"XiaomiMiMo/MiMo-V2.5": 40.1, # live
"deepseek-ai/DeepSeek-V4-Pro": 44.3, # live
"deepseek-ai/DeepSeek-V4-Flash": 40.3, # live
"deepseek-ai/DeepSeek-V3.2": 33.4, # live
"deepseek-ai/DeepSeek-V3.2-Exp": 25.4, # live
"deepseek-ai/DeepSeek-V3.1": 21.0, # live
"deepseek-ai/DeepSeek-V3-0324": 10.4, # live
"deepseek-ai/DeepSeek-V3": 10.4, # live
"deepseek-ai/DeepSeek-R1-0528": 20.1, # live
"deepseek-ai/DeepSeek-R1": 12.6, # live
"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B": 10.5, # peer
"deepseek-ai/DeepSeek-R1-Distill-Qwen-14B": 1.3, # peer
"deepseek-ai/DeepSeek-R1-Distill-Llama-8B": -7.9, # peer
"Qwen/QwQ-32B": 13.4, # live
"Qwen/Qwen3-4B-Thinking-2507": -4.8, # peer
"zai-org/GLM-5.1": 40.2, # live
"zai-org/GLM-5": 39.5, # live
"zai-org/GLM-5-FP8": 39.5, # live
"zai-org/GLM-5.1-FP8": 40.2, # live
"zai-org/GLM-4.7-Flash": 22.9, # live
"zai-org/GLM-4.6": 25.1, # live
"zai-org/GLM-4.5": 19.5, # live
"zai-org/GLM-4.5-Air": 19.5, # live
# Qwen family
"Qwen/Qwen3.6-27B": 32.0, # peer
"Qwen/Qwen3.5-397B-A17B": 33.7, # live
"Qwen/Qwen3-Next-80B-A3B-Instruct": 19.8, # live
"Qwen/Qwen3-235B-A22B": 13.4, # live
"Qwen/Qwen3-Coder-30B-A3B-Instruct": 19.7, # peer
"Qwen/Qwen3-32B": 11.5, # live
"Qwen/Qwen3-14B": 10.1, # live
"Qwen/Qwen3-8B": 7.4, # live
"Qwen/Qwen3-4B-Instruct-2507": 4.4, # peer
"Qwen/Qwen3-4B": 1.3, # peer
"Qwen/Qwen3-1.7B": -7.9, # peer
"Qwen/Qwen3-0.6B": -14.0, # peer
# 8B-class peers (no AA tracking but realistic LB-equivalents)
"meta-llama/Llama-3.1-8B-Instruct": -4.8, # peer
"meta-llama/Meta-Llama-3-8B-Instruct": -7.9, # peer
"google/gemma-2-9b-it": -3.3, # peer
"microsoft/Phi-4-mini-instruct": -1.8, # peer
"mistralai/Mistral-7B-Instruct-v0.3": -7.9, # peer
"Qwen/Qwen2.5-7B-Instruct": -4.8, # peer
"Qwen/Qwen2.5-14B-Instruct": 1.3, # peer
"Qwen/Qwen2.5-32B-Instruct": 7.4, # peer
"Qwen/Qwen3-30B-A3B": 10.5, # peer
# Other major open releases
"openai/gpt-oss-120b": 23.8, # live
"openai/gpt-oss-20b": 14.9, # live
"meta-llama/Llama-4-Maverick-17B-128E-Instruct": 14.3, # live
"meta-llama/Llama-4-Scout-17B-16E-Instruct": 10.0, # live
"meta-llama/Llama-3.3-70B-Instruct": 12.0, # peer
"google/gemma-4-31b-it": 29.4, # live
"google/gemma-4-26b-a4b-it": 25.7, # live
"google/gemma-3-27b-it": 12.0, # peer
"google/gemma-3-12b-it": 7.4, # peer
"microsoft/phi-4": 4.9, # live
"mistralai/Mistral-Large-Instruct-2411": 9.1, # live
"mistralai/Mistral-Small-3.2-24B-Instruct-2506": 10.5, # peer
"mistralai/Mistral-Small-3.1-24B-Instruct-2503": 7.4, # peer
"mistralai/Devstral-Small-2505": 11.8, # live
"MiniMaxAI/MiniMax-M2.5": 33.7, # live
"stepfun-ai/Step-3.5-Flash": 19.7, # peer
"nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16": 16.6, # peer
"nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16": 12.0, # peer
# Correct IDs for OLMo / Granite / Codestral families (the earlier
# forecast IDs like "OLMo-3-32B-Instruct" or "granite-4.1-30b-instruct"
# never shipped publicly under those names).
"allenai/Olmo-3-7B-Instruct": -4.8, # peer
"allenai/Olmo-3-1025-7B": -4.8, # peer
"ibm-granite/granite-4.0-h-small": 7.4, # peer
"ibm-granite/granite-4.0-h-tiny": -4.8, # peer
"ibm-granite/granite-3.3-8b-instruct": -3.3, # peer
"ibm-granite/granite-3.3-2b-instruct": -12.5, # peer
"mistralai/Codestral-22B-v0.1": 4.4, # peer
}
def _normalize_aa_index(index: float) -> float:
"""Normalize a raw AA index value onto the 0-100 scale."""
if not isinstance(index, (int, float)):
return 0.0
span = _AA_INDEX_MAX - _AA_INDEX_MIN
normalized = (index - _AA_INDEX_MIN) / span * 100.0
return max(0.0, min(100.0, round(normalized, 1)))
# --- Next.js App Router (RSC) scraping -------------------------------------
#
# artificialanalysis.ai migrated off the classic ``__NEXT_DATA__`` blob to the
# App Router streaming format: model data arrives in ``self.__next_f.push([n,
# "…"])`` calls whose second element is a JSON-string-escaped fragment of the
# RSC payload. We concatenate + unescape those chunks, then pull every
# ``{"name": …, …, "intelligenceIndex": …}`` record out with a bounded regex
# (the payload is a flat RSC stream, not a single parseable JSON document).
_RSC_CHUNK_RE = re.compile(r'self\.__next_f\.push\(\[\d+,(?P<s>"(?:[^"\\]|\\.)*")\]\)')
# A model record: a "name" string followed, within the SAME object, by its
# "intelligenceIndex". The middle group forbids another '"name":"' so the
# match cannot leak across into a neighbouring record that lacks an index.
_AA_RECORD_RE = re.compile(
r'"name":"(?P<name>(?:[^"\\]|\\.)*)"'
r'(?:(?!"name":").)*?'
r'"intelligenceIndex":(?P<idx>-?\d+(?:\.\d+)?)',
re.DOTALL,
)
# Variant qualifiers AA appends that don't change the underlying HF weights:
# "(Reasoning)", "(Non-reasoning)", "(high)", "(Reasoning, Max Effort)", etc.
_PAREN_RE = re.compile(r"\([^)]*\)")
def _canonical_name(name: str) -> str:
"""Normalize an AA display name for fuzzy matching against the HF map.
Drops parenthetical variant qualifiers and collapses separator/case noise
so ``"Qwen3 14B (Reasoning)"`` and ``"Qwen3-14B"`` both canonicalize to
``"qwen3 14b"``.
"""
name = _PAREN_RE.sub("", name)
name = name.lower().replace("-", " ").replace("_", " ")
return re.sub(r"\s+", " ", name).strip()
# Canonical-name -> HF ids, derived once from AA_NAME_TO_HF_IDS. Several display
# names can collapse to one canonical key; we union their HF ids.
_AA_CANON_TO_HF_IDS: dict[str, list[str]] = {}
for _disp, _ids in AA_NAME_TO_HF_IDS.items():
_AA_CANON_TO_HF_IDS.setdefault(_canonical_name(_disp), []).extend(_ids)
def _decode_rsc_blob(html: str) -> str:
"""Concatenate and unescape the App Router RSC chunks into one string."""
parts: list[str] = []
for m in _RSC_CHUNK_RE.finditer(html):
try:
parts.append(json.loads(m.group("s")))
except (ValueError, json.JSONDecodeError):
continue
return "".join(parts)
def _extract_aa_pairs_from_html(html: str) -> list[tuple[str, float]]:
"""Extract (name, intelligence_index) pairs from the RSC stream."""
blob = _decode_rsc_blob(html)
if not blob:
return []
pairs: list[tuple[str, float]] = []
for m in _AA_RECORD_RE.finditer(blob):
try:
name = json.loads('"' + m.group("name") + '"').strip()
score = float(m.group("idx"))
except (ValueError, json.JSONDecodeError):
continue
if name and score > 0:
pairs.append((name, score))
return pairs
def _extract_aa_pairs(payload: dict) -> list[tuple[str, float]]:
"""Walk the Next.js payload looking for {name, intelligenceIndex}-shaped
objects regardless of where they are nested."""
pairs: list[tuple[str, float]] = []
for node in _walk(payload):
# Look for the most common shapes AA has used in past iterations.
name = None
score = None
for name_key in ("model_name", "modelName", "name", "displayName"):
v = node.get(name_key)
if isinstance(v, str) and v.strip():
name = v.strip()
break
for score_key in (
"intelligence_index",
"intelligenceIndex",
"aa_index",
"aaIndex",
"score",
):
v = node.get(score_key)
if isinstance(v, (int, float)):
score = float(v)
break
if name and score is not None and score > 0:
pairs.append((name, score))
return pairs
async def fetch_aa_index_scores(client: httpx.AsyncClient) -> dict[str, float]:
"""Fetch Artificial Analysis Intelligence Index scores.
Returns ``{hf_id: normalized_score_0_100}`` for every model AA reports
that we can map back to a HuggingFace repo via :data:`AA_NAME_TO_HF_IDS`.
Raises on HTTP / parse failure.
"""
resp = await get_with_retries(client, AA_LEADERBOARD_URL)
resp.raise_for_status()
# Primary: Next.js App Router RSC stream (current site format).
pairs = _extract_aa_pairs_from_html(resp.text)
# Legacy fallback: classic __NEXT_DATA__ JSON blob (older site format).
if not pairs:
match = _NEXT_DATA_RE.search(resp.text)
if match:
try:
pairs = _extract_aa_pairs(json.loads(match.group("json")))
except (ValueError, json.JSONDecodeError):
pairs = []
if not pairs:
raise ExtractionFailed(
"AA leaderboard: no (name, score) pairs found "
"(neither RSC __next_f nor __NEXT_DATA__ matched)"
)
# When the same display name appears multiple times (different size /
# reasoning tiers), keep the maximum value — it represents the most
# capable variant available.
best_by_name: dict[str, float] = {}
for name, score in pairs:
current = best_by_name.get(name)
if current is None or score > current:
best_by_name[name] = score
live: dict[str, float] = {}
for name, score in best_by_name.items():
# Exact display name first, then canonicalized (variant-stripped) match.
hf_ids = AA_NAME_TO_HF_IDS.get(name) or _AA_CANON_TO_HF_IDS.get(
_canonical_name(name)
)
if not hf_ids:
continue
normalized = _normalize_aa_index(score)
if normalized <= 0:
continue
for hf_id in hf_ids:
if normalized > live.get(hf_id, 0.0):
live[hf_id] = normalized
if not live:
raise ExtractionFailed("AA index: live fetch returned 0 mapped scores")
# Overlay live scores on top of the curated snapshot so a successful live
# fetch can only ADD coverage, never shrink it below the fallback. Live
# numbers win wherever both exist; the snapshot fills the long tail of
# models AA labels in a way we can't map (or no longer tracks).
scores = get_aa_curated_fallback()
for hf_id, normalized in live.items():
if normalized > scores.get(hf_id, 0.0):
scores[hf_id] = normalized
logger.debug(f"AA index: {len(live)} live + {len(scores)} merged scores")
return scores
def get_aa_curated_fallback() -> dict[str, float]:
"""Return the 2026-06-29 curated snapshot, normalized to the 0-100 scale.
Used whenever the live HTML scrape cannot extract data — for example
when artificialanalysis.ai changes its Next.js payload shape.
"""
result: dict[str, float] = {}
for hf_id, raw in AA_INDEX_FALLBACK_2026_06_29.items():
normalized = _normalize_aa_index(raw)
if normalized > 0:
result[hf_id] = normalized
return result
src/whichllm/models/benchmark_sources/aider.py
+135
View File
@@ -0,0 +1,135 @@
"""Aider polyglot leaderboard source.
The polyglot leaderboard ranks LLMs by how well they edit code across six
languages (C++, Go, Java, JavaScript, Python, Rust). Aider publishes the raw
data as a YAML file in its GitHub repo, which is easier to parse than the
rendered HTML and rarely changes shape.
Used primarily for ``profile=coding`` ranking, but the score is also merged
into the general benchmark dict so coding-strong models get a small bump
elsewhere.
"""
from __future__ import annotations
import logging
import re
import httpx
from whichllm.models.http import get_with_retries
logger = logging.getLogger(__name__)
AIDER_POLYGLOT_YML_URL = (
"https://raw.githubusercontent.com/Aider-AI/aider/main/"
"aider/website/_data/polyglot_leaderboard.yml"
)
# Polyglot pass-rate is 0-100 (percent of exercises passing). Treat the
# floor/ceiling as 0..90 since the cap of practical models is ~88%.
_PG_MIN = 0.0
_PG_MAX = 90.0
AIDER_NAME_TO_HF_IDS: dict[str, list[str]] = {
"deepseek-r1": ["deepseek-ai/DeepSeek-R1"],
"deepseek-r1-0528": ["deepseek-ai/DeepSeek-R1-0528"],
"deepseek-v3": ["deepseek-ai/DeepSeek-V3"],
"deepseek-v3-0324": ["deepseek-ai/DeepSeek-V3-0324"],
"deepseek-v3.1": ["deepseek-ai/DeepSeek-V3.1"],
"deepseek-v3.2": ["deepseek-ai/DeepSeek-V3.2"],
"deepseek-v4-pro": ["deepseek-ai/DeepSeek-V4-Pro"],
"deepseek-v4-flash": ["deepseek-ai/DeepSeek-V4-Flash"],
"qwen3-coder-30b-a3b-instruct": ["Qwen/Qwen3-Coder-30B-A3B-Instruct"],
"qwen3-coder-next": ["Qwen/Qwen3-Coder-Next"],
"qwen2.5-coder-32b-instruct": ["Qwen/Qwen2.5-Coder-32B-Instruct"],
"qwen3-32b": ["Qwen/Qwen3-32B"],
"qwen3.6-27b": ["Qwen/Qwen3.6-27B"],
"llama-3.3-70b-instruct": ["meta-llama/Llama-3.3-70B-Instruct"],
"llama-4-maverick": ["meta-llama/Llama-4-Maverick-17B-128E-Instruct"],
"gemma-3-27b-it": ["google/gemma-3-27b-it"],
"gemma-4-31b": ["google/gemma-4-31b-it"],
"mistral-large-2411": ["mistralai/Mistral-Large-Instruct-2411"],
"devstral-small": ["mistralai/Devstral-Small-2505"],
"gpt-oss-120b": ["openai/gpt-oss-120b"],
"gpt-oss-20b": ["openai/gpt-oss-20b"],
"glm-4.5": ["zai-org/GLM-4.5"],
"glm-4.6": ["zai-org/GLM-4.6"],
"glm-5": ["zai-org/GLM-5"],
"glm-5.1": ["zai-org/GLM-5.1"],
"kimi-k2-instruct": ["moonshotai/Kimi-K2-Instruct"],
"phi-4": ["microsoft/phi-4"],
"qwq-32b": ["Qwen/QwQ-32B"],
}
_PASS_RATE_RE = re.compile(r"pass_rate[_-]?2[:\s]+(\d+(?:\.\d+)?)", re.IGNORECASE)
_MODEL_RE = re.compile(r"^- model[:\s]+(.+)$", re.MULTILINE)
def _normalize(pass_rate: float) -> float:
if not isinstance(pass_rate, (int, float)):
return 0.0
span = _PG_MAX - _PG_MIN
normalized = (pass_rate - _PG_MIN) / span * 100.0
return max(0.0, min(100.0, round(normalized, 1)))
def _parse_yaml_lite(text: str) -> list[tuple[str, float]]:
"""Tiny YAML extractor for the polyglot leaderboard format.
We avoid pulling in PyYAML; the file shape is stable enough that two
regexes scanning each record block suffice. Each record looks like:
- dirname: 2024-12-22-blah
model: deepseek/deepseek-chat
edit_format: diff
pass_rate_2: 80.7
...
"""
out: list[tuple[str, float]] = []
# Split into records starting with "- "
records = re.split(r"\n(?=-\s+\w)", text)
for rec in records:
m_model = re.search(r"^\s*model[:\s]+(.+?)$", rec, re.MULTILINE | re.IGNORECASE)
m_rate = re.search(r"pass_rate_2[:\s]+(\d+(?:\.\d+)?)", rec, re.IGNORECASE)
if not m_model or not m_rate:
continue
name = m_model.group(1).strip().strip("\"'")
# Strip any provider prefix like "deepseek/" or "openrouter/"
name = name.split("/", 1)[-1].strip().lower()
try:
rate = float(m_rate.group(1))
except ValueError:
continue
if rate <= 0:
continue
out.append((name, rate))
return out
async def fetch_aider_polyglot_scores(client: httpx.AsyncClient) -> dict[str, float]:
"""Fetch Aider polyglot pass-rates. Raises on HTTP / parse failure."""
scores: dict[str, float] = {}
resp = await get_with_retries(client, AIDER_POLYGLOT_YML_URL)
resp.raise_for_status()
pairs = _parse_yaml_lite(resp.text)
if not pairs:
logger.debug("Aider polyglot: 0 records parsed")
return {}
best_by_name: dict[str, float] = {}
for name, rate in pairs:
cur = best_by_name.get(name)
if cur is None or rate > cur:
best_by_name[name] = rate
for name, rate in best_by_name.items():
ids = AIDER_NAME_TO_HF_IDS.get(name)
if not ids:
continue
normalized = _normalize(rate)
if normalized <= 0:
continue
for hf_id in ids:
if scores.get(hf_id, 0.0) < normalized:
scores[hf_id] = normalized
logger.debug(f"Aider polyglot: {len(scores)} mapped scores")
return scores
src/whichllm/models/benchmark_sources/chatbot_arena.py
+140
View File
@@ -0,0 +1,140 @@
from __future__ import annotations
import re
import httpx
from whichllm.models.http import get_with_retries
# --- Data source URLs ---
ARENA_ROWS_URL = "https://datasets-server.huggingface.co/rows"
ARENA_DATASET = "mathewhe/chatbot-arena-elo"
# --- Arena ELO normalization ---
# Open-source ELO range: ~1030 (worst) to ~1424 (best). Arena is frozen
# 2025-07-17 (no new models added) so the leaderboard cannot reflect any
# 2025-Q3+ release; we cap the normalized output at 82 so newer benchmark
# sources (AA Index / LiveBench, which can reach 95+) decisively win on
# conflict.
_ARENA_ELO_MIN = 1030
_ARENA_ELO_MAX = 1430
_ARENA_MAX_NORMALIZED = 82.0
# --- Arena display name -> HuggingFace org mapping ---
_ARENA_ORG_TO_HF: dict[str, list[str]] = {
"Alibaba": ["Qwen"],
"Meta": ["meta-llama"],
"DeepSeek": ["deepseek-ai"],
"DeepSeek AI": ["deepseek-ai"],
"Google": ["google"],
"Mistral": ["mistralai"],
"Microsoft": ["microsoft"],
"Nvidia": ["nvidia"],
"01 AI": ["01-ai"],
"Allen AI": ["allenai"],
"Ai2": ["allenai"],
"AllenAI/UW": ["allenai"],
"Cohere": ["CohereForAI"],
"HuggingFace": ["HuggingFaceH4", "huggingface"],
"AI21 Labs": ["ai21labs"],
"NousResearch": ["NousResearch"],
"NexusFlow": ["Nexusflow"],
"Princeton": ["princeton-nlp"],
"IBM": ["ibm-granite"],
"InternLM": ["internlm"],
"Together AI": ["togethercomputer"],
"TII": ["tiiuae"],
"MiniMax": ["MiniMaxAI"],
"MosaicML": ["mosaicml"],
"Databricks": ["databricks"],
"Moonshot": ["moonshotai"],
"UC Berkeley": ["berkeley-nest"],
"Cognitive Computations": ["cognitivecomputations"],
"Upstage AI": ["upstage"],
"UW": ["timdettmers"],
"Snowflake": ["Snowflake"],
"LMSYS": ["lmsys"],
"OpenChat": ["openchat"],
}
def _normalize_arena_elo(elo: float) -> float:
"""Normalize Arena ELO to a frozen-source-aware 0-_ARENA_MAX_NORMALIZED scale."""
score = (
(elo - _ARENA_ELO_MIN)
/ (_ARENA_ELO_MAX - _ARENA_ELO_MIN)
* _ARENA_MAX_NORMALIZED
)
return max(0.0, min(_ARENA_MAX_NORMALIZED, round(score, 1)))
def _arena_name_to_hf_ids(model_name: str, org: str) -> list[str]:
"""Convert Arena display name to potential HuggingFace model IDs."""
hf_orgs = _ARENA_ORG_TO_HF.get(org, [])
candidates = []
# Clean the model name: remove date suffixes like "(03-2025)"
clean_name = re.sub(r"\s*\([\d-]+\)\s*$", "", model_name).strip()
# Remove -bf16, -fp8 suffixes for base matching
base_name = re.sub(r"-(bf16|fp8|fp16)$", "", clean_name, flags=re.IGNORECASE)
for hf_org in hf_orgs:
candidates.append(f"{hf_org}/{clean_name}")
if base_name != clean_name:
candidates.append(f"{hf_org}/{base_name}")
# Try with -Instruct suffix stripped for base model matching
no_instruct = re.sub(r"-Instruct$", "", clean_name)
if no_instruct != clean_name:
candidates.append(f"{hf_org}/{no_instruct}")
return candidates
async def fetch_arena_scores(client: httpx.AsyncClient) -> dict[str, float]:
"""Fetch Chatbot Arena ELO scores via rows API."""
scores: dict[str, float] = {}
offset = 0
while True:
resp = await get_with_retries(
client,
ARENA_ROWS_URL,
params={
"dataset": ARENA_DATASET,
"config": "default",
"split": "train",
"offset": str(offset),
"length": "100",
},
)
resp.raise_for_status()
data = resp.json()
rows = data.get("rows", [])
if not rows:
break
for r in rows:
row = r.get("row", {})
model_name = str(row.get("Model", ""))
elo = row.get("Arena Score", 0)
org = str(row.get("Organization", ""))
lic = str(row.get("License", ""))
if not model_name or not elo or elo <= 0:
continue
# Skip proprietary models (can't run locally)
if "Proprietary" in lic or "Propretary" in lic:
continue
normalized = _normalize_arena_elo(elo)
# Map to all potential HF IDs
hf_ids = _arena_name_to_hf_ids(model_name, org)
for hf_id in hf_ids:
scores[hf_id] = normalized
offset += len(rows)
total = data.get("num_rows_total", 0)
if total and offset >= total:
break
return scores
src/whichllm/models/benchmark_sources/constants.py
+7
View File
@@ -0,0 +1,7 @@
from __future__ import annotations
import re
_NEXT_DATA_RE = re.compile(
r'<script id="__NEXT_DATA__"[^>]*>(?P<json>.*?)</script>', re.DOTALL
)
src/whichllm/models/benchmark_sources/livebench.py
+102
View File
@@ -0,0 +1,102 @@
"""LiveBench (livebench.ai) source — inlined snapshot.
LiveBench publishes its leaderboard as a dated CSV (e.g.
``https://livebench.ai/table_2026_01_08.csv``).
To refresh: download the latest ``table_YYYY_MM_DD.csv`` from livebench.ai
and run ``scripts/import_livebench_csv.py`` to regenerate the dict below.
"""
from __future__ import annotations
# Inlined LiveBench global-average snapshot. Values are raw 0-100; the
# normalizer below rescales them onto the project's shared 0-100 axis.
# Entries from the 2026-01-08 CSV were generated by ``scripts/import_livebench_csv.py``.
LIVEBENCH_RAW_DATA: dict[str, float] = {
# --- 2026-01-08 CSV (auto-generated; see scripts/import_livebench_csv.py)
"MiniMaxAI/MiniMax-M2.5": 60.3,
"MiniMaxAI/MiniMax-M2.7": 65.0,
"Qwen/Qwen3-235B-A22B-Instruct-2507": 48.0,
"Qwen/Qwen3-235B-A22B-Thinking-2507": 52.9,
"Qwen/Qwen3-30B-A3B-Thinking-2507": 38.8,
"Qwen/Qwen3-32B": 42.7,
"Qwen/Qwen3-Next-80B-A3B-Instruct": 47.4,
"Qwen/Qwen3-Next-80B-A3B-Thinking": 51.0,
"Qwen/Qwen3.6-27B": 65.6,
"XiaomiMiMo/MiMo-V2-Pro": 58.4,
"deepseek-ai/DeepSeek-V3.2": 63.1,
"deepseek-ai/DeepSeek-V3.2-Exp": 58.9,
"deepseek-ai/DeepSeek-V4-Flash": 67.7,
"deepseek-ai/DeepSeek-V4-Pro": 74.4,
"google/gemma-4-31b-it": 62.4,
"mistralai/Devstral-2512": 38.8,
"moonshotai/Kimi-K2-Instruct": 45.9,
"moonshotai/Kimi-K2-Thinking": 62.3,
"moonshotai/Kimi-K2.5": 69.2,
"moonshotai/Kimi-K2.6-Thinking": 72.4,
"nvidia/Nemotron-3-Super-120B-A12B": 32.0,
"openai/gpt-oss-120b": 46.4,
"zai-org/GLM-4.6": 54.7,
"zai-org/GLM-4.6V": 38.9,
"zai-org/GLM-4.7": 57.3,
"zai-org/GLM-5": 68.7,
"zai-org/GLM-5.1": 70.6,
# --- 2026-04 carryover (anchors for older / smaller-class models)
"deepseek-ai/DeepSeek-R1-0528": 71.0,
"deepseek-ai/DeepSeek-R1": 65.0,
"deepseek-ai/DeepSeek-V3-0324": 57.0,
"Qwen/Qwen3-235B-A22B": 65.0,
"Qwen/Qwen3-Coder-30B-A3B-Instruct": 58.0,
"Qwen/QwQ-32B": 57.0,
"Qwen/Qwen3-4B-Thinking-2507": 50.0,
"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B": 56.0,
"deepseek-ai/DeepSeek-R1-Distill-Qwen-14B": 50.0,
"deepseek-ai/DeepSeek-R1-Distill-Llama-8B": 42.0,
"meta-llama/Llama-3.3-70B-Instruct": 48.0,
"meta-llama/Llama-4-Maverick-17B-128E-Instruct": 54.0,
"meta-llama/Llama-4-Scout-17B-16E-Instruct": 49.0,
"google/gemma-3-27b-it": 50.0,
"google/gemma-4-26b-a4b-it": 54.0,
"microsoft/phi-4": 53.0,
"mistralai/Mistral-Large-Instruct-2411": 58.0,
"mistralai/Devstral-Small-2505": 50.0,
"openai/gpt-oss-20b": 52.0,
"zai-org/GLM-4.5": 58.0,
"zai-org/GLM-4.5-Air": 52.0,
# 8B-class entries to anchor the smaller-model scoring
"Qwen/Qwen3-8B": 50.0,
"Qwen/Qwen3-14B": 56.0,
"Qwen/Qwen3-4B-Instruct-2507": 45.0,
"Qwen/Qwen3-4B": 42.0,
"Qwen/Qwen3-30B-A3B": 58.0,
"Qwen/Qwen2.5-7B-Instruct": 38.0,
"Qwen/Qwen2.5-14B-Instruct": 42.0,
"Qwen/Qwen2.5-32B-Instruct": 50.0,
"meta-llama/Llama-3.1-8B-Instruct": 36.0,
"google/gemma-2-9b-it": 38.0,
"google/gemma-3-12b-it": 44.0,
"microsoft/Phi-4-mini-instruct": 40.0,
"mistralai/Mistral-Small-3.2-24B-Instruct-2506": 50.0,
"mistralai/Mistral-Small-3.1-24B-Instruct-2503": 48.0,
}
# LiveBench global-average tops out around 72 for current frontier models
# (DeepSeek V4 Pro 72, Kimi K2.6 71, Qwen3.6-27B 66) with 8B-class around 35.
# Anchored by a two-point fit: top frontier (72) → 95, mid/8B (35) → 30 — so
# 8B models get a meaningful but not dominant share and frontier MoE clears
# the OLLB cap.
_LB_MIN = 18.0
_LB_MAX = 75.0
def _normalize_livebench(score: float) -> float:
normalized = (score - _LB_MIN) / (_LB_MAX - _LB_MIN) * 100.0
return max(0.0, min(100.0, round(normalized, 1)))
def get_livebench_data() -> dict[str, float]:
"""Return the inlined LiveBench snapshot, normalized to the 0-100 scale."""
return {
hf_id: _normalize_livebench(raw)
for hf_id, raw in LIVEBENCH_RAW_DATA.items()
if _normalize_livebench(raw) > 0
}
src/whichllm/models/benchmark_sources/open_llm_leaderboard.py
+99
View File
@@ -0,0 +1,99 @@
from __future__ import annotations
import io
import httpx
from whichllm.models.http import get_with_retries
LEADERBOARD_PARQUET_URL = (
"https://huggingface.co/api/datasets/open-llm-leaderboard/contents"
"/parquet/default/train/0.parquet"
)
LEADERBOARD_ROWS_URL = "https://datasets-server.huggingface.co/rows"
LEADERBOARD_DATASET = "open-llm-leaderboard/contents"
# --- Leaderboard normalization ---
# OLLB v2 averages range ~5 to ~52. The leaderboard is archived 2025-06 with
# the top slot held by Qwen2.5-32B (47.6 raw = 91.5 if uncapped); capping at
# 78 prevents an older generation with a strong-but-frozen OLLB score from
# dominating rankings that now have AA Index / LiveBench coverage too.
_LB_AVG_MAX = 52
_OLLB_MAX_NORMALIZED = 78.0
async def _fetch_leaderboard_parquet(client: httpx.AsyncClient) -> dict[str, float]:
"""Download Open LLM Leaderboard parquet (requires pyarrow)."""
import pyarrow.parquet as pq
resp = await get_with_retries(
client, LEADERBOARD_PARQUET_URL, follow_redirects=True
)
resp.raise_for_status()
table = pq.read_table(
io.BytesIO(resp.content),
columns=["fullname", "Average ⬆️"],
)
d = table.to_pydict()
scores: dict[str, float] = {}
for i in range(len(d["fullname"])):
name = d["fullname"][i]
avg = d["Average ⬆️"][i]
if name and avg and avg > 0:
scores[name] = _normalize_leaderboard_avg(avg)
return scores
async def _fetch_leaderboard_api(client: httpx.AsyncClient) -> dict[str, float]:
"""Fetch Open LLM Leaderboard via rows API (no pyarrow needed)."""
scores: dict[str, float] = {}
offset = 0
while True:
resp = await get_with_retries(
client,
LEADERBOARD_ROWS_URL,
params={
"dataset": LEADERBOARD_DATASET,
"config": "default",
"split": "train",
"offset": str(offset),
"length": "100",
},
)
resp.raise_for_status()
data = resp.json()
rows = data.get("rows", [])
if not rows:
break
for r in rows:
row = r.get("row", {})
name = row.get("fullname")
avg = row.get("Average ⬆️")
if name and avg and avg > 0:
scores[name] = _normalize_leaderboard_avg(avg)
offset += len(rows)
total = data.get("num_rows_total", 0)
if total and offset >= total:
break
return scores
def _normalize_leaderboard_avg(avg: float) -> float:
"""Normalize Open LLM Leaderboard average to 0-_OLLB_MAX_NORMALIZED scale."""
score = avg / _LB_AVG_MAX * _OLLB_MAX_NORMALIZED
return max(0.0, min(_OLLB_MAX_NORMALIZED, round(score, 1)))
async def fetch_leaderboard_with_fallback(
client: httpx.AsyncClient,
) -> dict[str, float]:
"""Prefer the parquet path (one request, full table) and fall back to the
paginated rows API when pyarrow is unavailable."""
try:
return await _fetch_leaderboard_parquet(client)
except ImportError:
return await _fetch_leaderboard_api(client)
src/whichllm/models/benchmark_sources/types.py
+5
View File
@@ -0,0 +1,5 @@
from __future__ import annotations
class ExtractionFailed(Exception):
pass
src/whichllm/models/benchmark_sources/utils.py
+14
View File
@@ -0,0 +1,14 @@
from __future__ import annotations
def _walk(obj, depth: int = 0):
"""Yield every dict encountered while recursively walking a JSON tree."""
if depth > 12:
return
if isinstance(obj, dict):
yield obj
for v in obj.values():
yield from _walk(v, depth + 1)
elif isinstance(obj, list):
for item in obj:
yield from _walk(item, depth + 1)
src/whichllm/models/benchmark_sources/vision.py
+83
View File
@@ -0,0 +1,83 @@
"""Vision-language model benchmark source.
Text leaderboards (LiveBench, AA Index, Aider) do not score VLMs, so a
``--profile vision`` ranking had only one model with a ``direct`` hit
(Qwen2-VL-7B, a two-generations-old model) and every current VLM fell
back to size/popularity heuristics — letting the 8B legacy model
outrank Qwen3-VL-32B even on an 80 GB H100.
There is no single stable machine-readable VLM leaderboard (the MMMU /
OpenCompass spaces change their JSON shape frequently), so this source
is a curated snapshot rather than a live scrape. Values are a blended
0-100 capability index drawn from MMMU-Pro, MMBench, and general
multimodal evaluations as of 2026-05; they are already normalized and
merged into the combined benchmark dict like any other current source.
Profile filtering keeps these scores from affecting text rankings —
only models tagged ``vision`` consume them.
"""
from __future__ import annotations
import logging
import httpx
logger = logging.getLogger(__name__)
# Curated multimodal capability index (0-100), 2026-05 snapshot.
# Anchored so the current Qwen3-VL / InternVL3 frontier sits in the
# mid-50s-to-60s and two-generations-old 7B-class VLMs sit in the low
# 30s, which restores the correct generational ordering.
VISION_FALLBACK_2026_05: dict[str, float] = {
# Qwen3-VL (current frontier)
"Qwen/Qwen3-VL-235B-A22B-Instruct": 62.0,
"Qwen/Qwen3-VL-32B-Instruct": 57.0,
"Qwen/Qwen3-VL-30B-A3B-Instruct": 53.0,
"Qwen/Qwen3-VL-8B-Instruct": 45.0,
"Qwen/Qwen3-VL-8B-Thinking": 46.0,
"Qwen/Qwen3-VL-4B-Instruct": 37.0,
"Qwen/Qwen3-VL-4B-Thinking": 38.0,
# Qwen2.5-VL (previous generation, still strong)
"Qwen/Qwen2.5-VL-72B-Instruct": 55.0,
"Qwen/Qwen2.5-VL-32B-Instruct": 49.0,
"Qwen/Qwen2.5-VL-7B-Instruct": 41.0,
"Qwen/Qwen2.5-VL-3B-Instruct": 33.0,
# Qwen2-VL (two generations old — must rank below Qwen3-VL)
"Qwen/Qwen2-VL-72B-Instruct": 45.0,
"Qwen/Qwen2-VL-7B-Instruct": 33.0,
"Qwen/Qwen2-VL-2B-Instruct": 24.0,
# Meta Llama Vision
"meta-llama/Llama-3.2-90B-Vision-Instruct": 41.0,
"meta-llama/Llama-3.2-11B-Vision-Instruct": 29.0,
# Microsoft Phi vision
"microsoft/Phi-4-reasoning-vision-15B": 46.0,
"microsoft/Phi-3.5-vision-instruct": 27.0,
# Google Gemma 3 (natively multimodal)
"google/gemma-3-27b-it": 42.0,
"google/gemma-3-12b-it": 35.0,
"google/gemma-3-4b-it": 27.0,
# Mistral Pixtral
"mistralai/Pixtral-12B-2409": 35.0,
"mistral-community/pixtral-12b": 35.0,
# OpenGVLab InternVL3 (frontier open VLM)
"OpenGVLab/InternVL3-78B": 56.0,
"OpenGVLab/InternVL3-38B": 52.0,
"OpenGVLab/InternVL3-14B": 45.0,
"OpenGVLab/InternVL3-8B": 40.0,
"OpenGVLab/InternVL2_5-78B": 50.0,
# DeepSeek-VL2
"deepseek-ai/deepseek-vl2": 38.0,
# zhipu / GLM vision
"zai-org/GLM-4.5V": 50.0,
}
async def fetch_vision_scores(client: httpx.AsyncClient) -> dict[str, float]:
"""Return curated VLM capability scores.
No stable live source exists, so this returns the frozen snapshot.
The ``client`` parameter mirrors the other source functions so the
caller can treat all sources uniformly and a live scrape can be
slotted in later without changing call sites.
"""
return dict(VISION_FALLBACK_2026_05)
src/whichllm/models/benchmark_types.py
+23
View File
@@ -0,0 +1,23 @@
"""Shared benchmark data types."""
from __future__ import annotations
from dataclasses import dataclass
@dataclass(frozen=True)
class BenchmarkEvidence:
"""Benchmark evidence with confidence.
source values, ordered from most trusted to least:
- "direct" : independent leaderboard / Arena ELO hit on exact id
- "variant" : suffix-stripped derivative of a direct leaderboard hit
- "base_model" : cardData.base_model pointer to a direct hit
- "line_interp" : size-aware interpolation within the same model line
- "self_reported" : evalResults reported by the uploader themselves
- "none" : no usable signal
"""
score: float | None
confidence: float
source: str
src/whichllm/models/cache.py
+47
View File
@@ -0,0 +1,47 @@
"""Local JSON cache with TTL for model data."""
from __future__ import annotations
import json
import logging
import time
from whichllm.utils import _cache_dir
logger = logging.getLogger(__name__)
CACHE_DIR = _cache_dir()
CACHE_FILE = CACHE_DIR / "models.json"
DEFAULT_TTL_SECONDS = 6 * 3600 # 6 hours
def _ensure_cache_dir() -> None:
CACHE_DIR.mkdir(parents=True, exist_ok=True)
def load_cache() -> list[dict] | None:
"""Load cached model data if valid. Returns None if expired or missing."""
if not CACHE_FILE.exists():
return None
try:
data = json.loads(CACHE_FILE.read_text(encoding="utf-8"))
cached_at = data.get("cached_at", 0)
if time.time() - cached_at > DEFAULT_TTL_SECONDS:
logger.debug("Cache expired")
return None
return data.get("models", [])
except (json.JSONDecodeError, KeyError) as e:
logger.debug(f"Cache corrupted: {e}")
return None
def save_cache(models: list[dict]) -> None:
"""Save model data to cache."""
_ensure_cache_dir()
data = {
"cached_at": time.time(),
"models": models,
}
CACHE_FILE.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
logger.debug(f"Saved {len(models)} models to cache")
src/whichllm/models/fetcher.py
+111
View File
@@ -0,0 +1,111 @@
"""Compatibility shim for HuggingFace model fetching helpers.
The implementation is split by responsibility across ``whichllm.models``:
- ``hf`` for HuggingFace API orchestration
- ``parser`` for API payload to ModelInfo conversion
- ``parameters`` for parameter-count and MoE normalization
- ``gguf`` for GGUF filename/variant parsing
- ``sliding_window`` for SWA metadata resolution
- ``serialization`` for cache serialization
Existing imports from ``whichllm.models.fetcher`` are re-exported here.
"""
from __future__ import annotations
from whichllm.models import hf as _hf_module
from whichllm.models.gguf import (
_estimate_gguf_size,
_extract_gguf_variants,
_extract_quant_type,
)
from whichllm.models.hf import (
_DEFAULT_HF_ENDPOINT,
_FRONTIER_MODEL_IDS,
_hf_api_url,
)
from whichllm.models.http import get_with_retries
from whichllm.models.parameters import (
_AUTHORITATIVE_PARAM_COUNTS,
_KNOWN_MOE_ACTIVE_PARAMS,
_KNOWN_PARAM_COUNTS,
_extract_active_size_hint_from_id,
_extract_size_hint_from_id,
_is_quantized_repo_name,
_lookup_curated_count,
_normalize_param_count,
_resolve_moe_active_params,
)
from whichllm.models.parser import (
_extract_architecture,
_extract_hf_eval_score,
_extract_param_count,
_extract_published_at,
_is_general_eval_entry,
_normalize_eval_value,
_parse_model,
)
from whichllm.models.serialization import dicts_to_models, models_to_dicts
from whichllm.models.sliding_window import (
_SWA_ARCH_ALIASES,
_SWA_ARCH_DEFAULTS,
_resolve_sliding_window,
_swa_arch_key,
_swa_key_from_arch,
)
async def fetch_models(limit: int = 300, include_vision: bool = True):
"""Fetch popular models from HuggingFace Hub.
The wrapper keeps legacy monkey-patching of ``fetcher.get_with_retries``
working while the implementation lives in ``whichllm.models.hf``.
"""
original = _hf_module.get_with_retries
_hf_module.get_with_retries = get_with_retries
try:
return await _hf_module.fetch_models(limit=limit, include_vision=include_vision)
finally:
_hf_module.get_with_retries = original
async def fetch_model_published_at(model_ids: list[str]):
"""Fetch published timestamps for specific model IDs."""
return await _hf_module.fetch_model_published_at(model_ids)
__all__ = [
"_AUTHORITATIVE_PARAM_COUNTS",
"_DEFAULT_HF_ENDPOINT",
"_FRONTIER_MODEL_IDS",
"_KNOWN_MOE_ACTIVE_PARAMS",
"_KNOWN_PARAM_COUNTS",
"_SWA_ARCH_ALIASES",
"_SWA_ARCH_DEFAULTS",
"_estimate_gguf_size",
"_extract_active_size_hint_from_id",
"_extract_architecture",
"_extract_gguf_variants",
"_extract_hf_eval_score",
"_extract_param_count",
"_extract_published_at",
"_extract_quant_type",
"_extract_size_hint_from_id",
"_hf_api_url",
"_is_general_eval_entry",
"_is_quantized_repo_name",
"_lookup_curated_count",
"_normalize_eval_value",
"_normalize_param_count",
"_parse_model",
"_resolve_moe_active_params",
"_resolve_sliding_window",
"_swa_arch_key",
"_swa_key_from_arch",
"dicts_to_models",
"fetch_model_published_at",
"fetch_models",
"get_with_retries",
"models_to_dicts",
]
src/whichllm/models/gguf.py
+84
View File
@@ -0,0 +1,84 @@
"""GGUF filename parsing and variant extraction helpers."""
from __future__ import annotations
import re
from whichllm.constants import QUANT_BYTES_PER_WEIGHT
from whichllm.models.types import GGUFVariant
_GGUF_SPLIT_RE = re.compile(r"-(\d{5})-of-(\d{5})\.gguf$", re.IGNORECASE)
# Filename quant tokens that name the same format under a different spelling
# than the canonical key used by QUANT_BYTES_PER_WEIGHT / QUANT_QUALITY_PENALTY.
_QUANT_ALIASES = {
"FP16": "F16",
"FP32": "F32",
}
def _extract_quant_type(filename: str) -> str:
"""Extract quantization type from a GGUF filename.
The returned key is canonicalized to match QUANT_BYTES_PER_WEIGHT, so
callers can look it up directly. Returns "unknown" when nothing matches.
"""
patterns = [
r"[.-](Q\d+_K_[SMLA])",
r"[.-](Q\d+_\d+)",
r"[.-](Q\d+_K)",
r"[.-](TQ\d+_\d+)",
r"[.-](IQ\d+_\w+)",
r"[.-](MXFP4|NVFP4)",
r"[.-](F16|FP16|BF16|F32|FP32)",
]
upper = filename.upper()
for pattern in patterns:
m = re.search(pattern, upper)
if m:
quant = m.group(1)
return _QUANT_ALIASES.get(quant, quant)
return "unknown"
def _estimate_gguf_size(param_count: int, quant_type: str) -> int:
"""Estimate GGUF file size from parameter count and quantization type."""
bpw = QUANT_BYTES_PER_WEIGHT.get(quant_type.upper(), 0.5625) # default Q4_K_M
return int(param_count * bpw)
def _extract_gguf_variants(data: dict, param_count: int) -> list[GGUFVariant]:
"""Extract GGUF variants from HF sibling metadata."""
quant_sizes: dict[str, int] = {}
quant_first_filename: dict[str, str] = {}
siblings = data.get("siblings", []) or []
for sib in siblings:
fname = sib.get("rfilename", "")
if not fname.endswith(".gguf") or fname.startswith("."):
continue
quant = _extract_quant_type(fname)
if quant == "unknown":
continue
size = sib.get("size", 0)
if not isinstance(size, int) or size < 0:
size = 0
# Split GGUF files are summed into one candidate per quant.
quant_sizes[quant] = quant_sizes.get(quant, 0) + size
if quant not in quant_first_filename or _GGUF_SPLIT_RE.search(
quant_first_filename[quant]
):
quant_first_filename[quant] = fname
gguf_variants = []
for quant, total_size in quant_sizes.items():
if total_size <= 0:
total_size = _estimate_gguf_size(param_count, quant)
gguf_variants.append(
GGUFVariant(
filename=quant_first_filename[quant],
quant_type=quant,
file_size_bytes=total_size,
)
)
return gguf_variants
src/whichllm/models/grouper.py
+149
View File
@@ -0,0 +1,149 @@
"""Model family grouping logic."""
from __future__ import annotations
import re
from whichllm.models.types import ModelFamily, ModelInfo
def _normalize_name(model_id: str) -> str:
"""Normalize model ID for grouping by removing org prefix and GGUF/quant/chat suffixes."""
name = model_id.lower()
# Strip org prefix (e.g. "bartowski/Meta-Llama-3.1" -> "meta-llama-3.1")
if "/" in name:
name = name.split("/", 1)[1]
# Strip common org prefixes in model names (e.g. "qwen_qwen3-8b" -> "qwen3-8b")
name = re.sub(r"^(qwen_|meta-llama_|google_)", "", name)
# Remove common suffixes (applied repeatedly to handle stacked suffixes)
suffixes = [
r"-gguf$",
r"-gptq$",
r"-awq$",
r"-instruct$",
r"-chat$",
r"-it$",
r"-hf$",
r"-fp8$",
r"-fp16$",
r"-bf16$",
r"-mxfp4$",
r"-nvfp4$",
r"-\d+bit$",
r"-\d{4}$", # date suffixes like -2507, -2503
]
for _ in range(3): # multiple passes to strip stacked suffixes
prev = name
for suffix in suffixes:
name = re.sub(suffix, "", name)
if name == prev:
break
# Strip version-before-size: mistral-small-3.2-24b -> mistral-small-24b
# This catches patterns like MODEL-MAJOR.MINOR-SIZEb where the version
# is a separate segment (preceded by '-') before the size suffix.
# Does NOT match qwen3.5-27b because '3.5' is glued to 'qwen' without '-'.
name = re.sub(r"-\d+\.\d+(-\d+(?:\.\d+)?b(?:-a\d+b)?)$", r"\1", name)
# Split series name from size suffix, strip minor version from series only.
# Merges qwen3.5-27b + qwen3-30b-a3b naming variants (different sizes stay separate).
m = re.match(r"^(.+?)-(\d+(?:\.\d+)?b(?:-a\d+b)?)$", name)
if m:
series, size = m.group(1), m.group(2)
series = re.sub(r"(\d+)\.\d+$", r"\1", series)
name = f"{series}-{size}"
else:
# No size suffix (e.g. deepseek-v3.2) — strip minor version directly
name = re.sub(r"(\d+)\.\d+$", r"\1", name)
return name
def group_models(models: list[ModelInfo]) -> list[ModelFamily]:
"""Group models into families based on base_model and name similarity."""
# Pass 1: Group by base_model
base_model_groups: dict[str, list[ModelInfo]] = {}
ungrouped: list[ModelInfo] = []
for model in models:
if model.base_model:
key = model.base_model.lower()
base_model_groups.setdefault(key, []).append(model)
else:
ungrouped.append(model)
# Pass 2: Group ungrouped by normalized name
name_groups: dict[str, list[ModelInfo]] = {}
for model in ungrouped:
key = _normalize_name(model.id)
name_groups.setdefault(key, []).append(model)
# Merge base_model groups that share the same normalized name
merged_base: dict[str, list[ModelInfo]] = {}
for key, group in base_model_groups.items():
norm_key = _normalize_name(key)
merged_base.setdefault(norm_key, []).extend(group)
# Also merge with ungrouped via name matching
for norm_key, group in list(merged_base.items()):
if norm_key in name_groups:
group.extend(name_groups.pop(norm_key))
# Replace base_model_groups with merged version
base_model_groups = merged_base
# Build families
families: list[ModelFamily] = []
for group_key, group in list(base_model_groups.items()) + list(name_groups.items()):
if not group:
continue
# Pick the base model. Priority order:
# 1. Models that are referenced by another group member's base_model
# field — these are upstream of the others, so they are the
# true base even when a downstream fine-tune (e.g.
# prefeitura-rio/Rio-3.0-Open-Mini) has more downloads than the
# official base (Qwen/Qwen3-4B-Thinking-2507).
# 2. Models without GGUF/quant suffixes and no base_model of their
# own (the original checkpoint).
# 3. Anything left in the group.
# Within the chosen tier, pick highest downloads as a tiebreaker.
referenced_as_base: set[str] = {m.base_model for m in group if m.base_model}
upstream_candidates = [m for m in group if m.id in referenced_as_base]
if upstream_candidates:
base_candidates = upstream_candidates
else:
base_candidates = [
m for m in group if not m.gguf_variants or m.base_model is None
]
if not base_candidates:
base_candidates = group
base = max(base_candidates, key=lambda m: m.downloads)
variants = [m for m in group if m.id != base.id]
# Set family_id on all members
family_id = _normalize_name(base.id)
base.family_id = family_id
for v in variants:
v.family_id = family_id
# Collect best benchmark scores across family
best_bench: dict[str, float] = {}
for m in group:
for k, v in m.benchmark_scores.items():
if k not in best_bench or v > best_bench[k]:
best_bench[k] = v
families.append(
ModelFamily(
family_id=family_id,
display_name=base.name,
base_model=base,
variants=variants,
best_benchmark=best_bench,
)
)
return families
src/whichllm/models/hf.py
+293
View File
@@ -0,0 +1,293 @@
"""HuggingFace Hub client orchestration."""
from __future__ import annotations
import asyncio
import logging
import os
import httpx
from whichllm.models.http import DEFAULT_ACCEPT_ENCODING, get_with_retries
from whichllm.models.parser import _extract_published_at, _parse_model
from whichllm.models.types import ModelInfo
logger = logging.getLogger(__name__)
_DEFAULT_HF_ENDPOINT = "https://huggingface.co"
_MODEL_EXPANDS = [
"config",
"safetensors",
"gguf",
"cardData",
"siblings",
"evalResults",
]
_MODEL_DETAIL_EXPANDS = [
"config",
"safetensors",
"gguf",
"cardData",
"siblings",
"evalResults",
"downloads",
"likes",
"createdAt",
"lastModified",
]
_FRONTIER_MODEL_IDS = (
# Newest releases that lead 2026-Q2 benchmarks
"moonshotai/Kimi-K2-Thinking",
"moonshotai/Kimi-K2-Instruct",
"moonshotai/Kimi-K2-Instruct-0905",
"XiaomiMiMo/MiMo-V2.5-Pro",
"XiaomiMiMo/MiMo-V2.5",
"XiaomiMiMo/MiMo-V2-Flash",
"deepseek-ai/DeepSeek-V4-Pro",
"deepseek-ai/DeepSeek-V4-Flash",
"deepseek-ai/DeepSeek-V3.2",
"deepseek-ai/DeepSeek-V3.2-Exp",
"deepseek-ai/DeepSeek-V3.1",
"deepseek-ai/DeepSeek-R1-0528",
"zai-org/GLM-5.1",
"zai-org/GLM-5",
"zai-org/GLM-5-FP8",
"zai-org/GLM-5.1-FP8",
"zai-org/GLM-4.7-Flash",
"zai-org/GLM-4.6",
"zai-org/GLM-4.5",
"zai-org/GLM-4.5-Air",
# Open-weight mid-size frontier
"Qwen/Qwen3.6-27B",
"Qwen/Qwen3-32B",
"Qwen/Qwen3-14B",
"Qwen/Qwen3-8B",
"Qwen/Qwen3-Coder-30B-A3B-Instruct",
"Qwen/Qwen3-Next-80B-A3B-Instruct",
"Qwen/Qwen3-235B-A22B",
"Qwen/Qwen3-4B-Instruct-2507",
# Reasoning/thinking lines that do not auto-surface via cardinality
"Qwen/QwQ-32B",
"Qwen/Qwen3-4B-Thinking-2507",
"deepseek-ai/DeepSeek-R1",
"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
"deepseek-ai/DeepSeek-R1-Distill-Qwen-14B",
"deepseek-ai/DeepSeek-R1-Distill-Llama-8B",
# Other current open releases
"openai/gpt-oss-120b",
"openai/gpt-oss-20b",
"google/gemma-3-27b-it",
"google/gemma-3-12b-it",
"google/gemma-4-31B-it",
"google/gemma-4-26B-A4B-it",
"meta-llama/Llama-3.3-70B-Instruct",
"meta-llama/Llama-4-Maverick-17B-128E-Instruct",
"meta-llama/Llama-4-Scout-17B-16E-Instruct",
"microsoft/phi-4",
"microsoft/Phi-4-mini-instruct",
"mistralai/Mistral-Large-Instruct-2411",
"mistralai/Mistral-Small-3.2-24B-Instruct-2506",
"mistralai/Mistral-Small-3.1-24B-Instruct-2503",
"mistralai/Devstral-Small-2505",
"mistralai/Codestral-22B-v0.1",
"MiniMaxAI/MiniMax-M2",
"MiniMaxAI/MiniMax-M2.5",
# IBM Granite latest open releases
"ibm-granite/granite-4.0-h-small",
"ibm-granite/granite-4.0-h-tiny",
"ibm-granite/granite-3.3-8b-instruct",
"ibm-granite/granite-3.3-2b-instruct",
# AllenAI Olmo-3
"allenai/Olmo-3-7B-Instruct",
"allenai/Olmo-3-1025-7B",
# Nemotron 3 series
"nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
"nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16",
)
def _hf_api_url(path: str) -> str:
raw_endpoint = os.environ.get("HF_ENDPOINT")
endpoint = _DEFAULT_HF_ENDPOINT if raw_endpoint is None else raw_endpoint.strip()
if not endpoint:
raise ValueError("HF_ENDPOINT must not be empty")
if not endpoint.startswith(("http://", "https://")):
raise ValueError("HF_ENDPOINT must start with http:// or https://")
endpoint = endpoint.rstrip("/")
return f"{endpoint}/api/{path.lstrip('/')}"
def _model_list_params(limit: int, sort: str, filter_value: str | None = None) -> dict:
params = {
"pipeline_tag": "text-generation",
"sort": sort,
"limit": str(limit),
"expand[]": _MODEL_EXPANDS,
}
if filter_value:
params["filter"] = filter_value
return params
def _append_new_models(
data_list: list[dict],
models: list[ModelInfo],
seen_ids: set[str],
) -> None:
for data in data_list:
if data.get("id") not in seen_ids:
model = _parse_model(data)
if model:
models.append(model)
seen_ids.add(model.id)
async def _fetch_model_list(
client: httpx.AsyncClient,
params: dict,
) -> list[dict]:
resp = await get_with_retries(client, _hf_api_url("models"), params=params)
resp.raise_for_status()
return resp.json()
async def _fetch_frontier_models(
client: httpx.AsyncClient,
models: list[ModelInfo],
seen_ids: set[str],
) -> None:
for model_id in _FRONTIER_MODEL_IDS:
if model_id in seen_ids:
continue
try:
resp = await get_with_retries(
client,
_hf_api_url(f"models/{model_id}"),
params={"expand[]": _MODEL_DETAIL_EXPANDS},
)
if resp.status_code >= 400:
logger.debug(
f"Frontier fetch skipped {model_id}: HTTP {resp.status_code}"
)
continue
data = resp.json()
except (httpx.HTTPError, ValueError) as e:
logger.debug(f"Frontier fetch failed for {model_id}: {e}")
continue
model = _parse_model(data)
if model:
models.append(model)
seen_ids.add(model.id)
async def fetch_models(
limit: int = 300, include_vision: bool = True
) -> list[ModelInfo]:
"""Fetch popular models from HuggingFace Hub."""
models: list[ModelInfo] = []
async with httpx.AsyncClient(
timeout=30.0,
follow_redirects=True,
headers={"Accept-Encoding": DEFAULT_ACCEPT_ENCODING},
) as client:
logger.debug(f"Fetching models from HF API (limit={limit})")
data_list = await _fetch_model_list(
client, _model_list_params(limit, sort="downloads")
)
for data in data_list:
model = _parse_model(data)
if model:
models.append(model)
logger.debug("Fetching GGUF models from HF API")
gguf_data_list = await _fetch_model_list(
client, _model_list_params(limit, sort="downloads", filter_value="gguf")
)
seen_ids = {m.id for m in models}
_append_new_models(gguf_data_list, models, seen_ids)
logger.debug("Fetching recent GGUF models from HF API")
recent_data_list = await _fetch_model_list(
client, _model_list_params(limit, sort="lastModified", filter_value="gguf")
)
_append_new_models(recent_data_list, models, seen_ids)
for filter_value in (None, "gguf"):
logger.debug(
f"Fetching trending {filter_value or 'all'} models from HF API"
)
try:
trending_data_list = await _fetch_model_list(
client,
_model_list_params(
limit, sort="trending", filter_value=filter_value
),
)
except (httpx.HTTPError, ValueError) as e:
logger.debug(f"Trending fetch skipped: {e}")
continue
_append_new_models(trending_data_list, models, seen_ids)
await _fetch_frontier_models(client, models, seen_ids)
if include_vision:
for pipeline_tag in ("image-text-to-text",):
mm_params = {
"pipeline_tag": pipeline_tag,
"sort": "downloads",
"limit": str(limit),
"expand[]": _MODEL_EXPANDS,
}
logger.debug(f"Fetching {pipeline_tag} models from HF API")
mm_data_list = await _fetch_model_list(client, mm_params)
_append_new_models(mm_data_list, models, seen_ids)
logger.debug(f"Fetched {len(models)} models total")
return models
async def fetch_model_published_at(model_ids: list[str]) -> dict[str, str]:
"""Fetch published timestamps for specific model IDs."""
unique_ids = sorted({m for m in model_ids if m})
if not unique_ids:
return {}
async with httpx.AsyncClient(
timeout=20.0,
follow_redirects=True,
headers={"Accept-Encoding": DEFAULT_ACCEPT_ENCODING},
) as client:
tasks = [
client.get(
_hf_api_url(f"models/{model_id}"),
params={"expand[]": ["createdAt", "lastModified"]},
)
for model_id in unique_ids
]
responses = await asyncio.gather(*tasks, return_exceptions=True)
result: dict[str, str] = {}
for model_id, resp in zip(unique_ids, responses, strict=False):
if isinstance(resp, Exception):
logger.debug("Failed to fetch model detail for %s: %s", model_id, resp)
continue
if resp.status_code >= 400:
logger.debug(
"Failed to fetch model detail for %s: HTTP %s",
model_id,
resp.status_code,
)
continue
try:
data = resp.json()
except ValueError:
continue
published_at = _extract_published_at(data)
if published_at:
result[model_id] = published_at
return result
src/whichllm/models/http.py
+43
View File
@@ -0,0 +1,43 @@
from __future__ import annotations
import asyncio
import random
import httpx
RETRYABLE_STATUS_CODES = {408, 429, 500, 502, 503, 504}
DEFAULT_ACCEPT_ENCODING = "gzip, deflate"
async def get_with_retries(
client: httpx.AsyncClient,
url: str,
*,
attempts: int = 3,
base_delay: float = 0.25,
max_delay: float = 2.0,
jitter: float = 0.1,
retry_status_codes: set[int] | None = None,
**kwargs,
) -> httpx.Response:
"""GET with bounded retry/backoff for transient HTTP failures."""
retry_codes = retry_status_codes or RETRYABLE_STATUS_CODES
last_attempt = max(1, attempts) - 1
for attempt in range(last_attempt + 1):
try:
response = await client.get(url, **kwargs)
except (httpx.TimeoutException, httpx.TransportError):
if attempt >= last_attempt:
raise
else:
if response.status_code not in retry_codes or attempt >= last_attempt:
return response
delay = min(max_delay, base_delay * (2**attempt))
if jitter > 0:
delay += random.uniform(0, jitter)
if delay > 0:
await asyncio.sleep(delay)
raise RuntimeError("unreachable retry state")
src/whichllm/models/parameters.py
+218
View File
@@ -0,0 +1,218 @@
"""Parameter-count and MoE metadata normalization helpers."""
from __future__ import annotations
import re
def _extract_size_hint_from_id(model_id: str | None) -> int | None:
"""Extract parameter size hint (in params) from model ID like 27B or 30B-A3B."""
if not model_id:
return None
lower = model_id.lower()
matches = re.findall(r"(\d+(?:\.\d+)?)b(?:-a\d+(?:\.\d+)?b)?", lower)
if not matches:
return None
try:
max_b = max(float(m) for m in matches)
except ValueError:
return None
if max_b <= 0:
return None
return int(max_b * 1e9)
def _extract_active_size_hint_from_id(model_id: str | None) -> int | None:
"""Extract MoE active parameter hint from names like 35B-A3B."""
if not model_id:
return None
lower = model_id.lower()
matches = re.findall(r"\d+(?:\.\d+)?b[-_]?a(\d+(?:\.\d+)?)b", lower)
if not matches:
return None
try:
max_b = max(float(m) for m in matches)
except ValueError:
return None
if max_b <= 0:
return None
return int(max_b * 1e9)
def _is_quantized_repo_name(model_id: str) -> bool:
"""Detect quantized/non-base repository naming patterns."""
lower = model_id.lower()
return bool(re.search(r"(gptq|awq|bnb|4bit|int4|int8|fp8|gguf|quant)", lower))
def _lookup_curated_count(mapping: dict[str, int], model_id: str) -> int | None:
value = mapping.get(model_id)
if value is not None:
return value
model_id_folded = model_id.casefold()
for key, value in mapping.items():
if key.casefold() == model_id_folded:
return value
return None
def _resolve_moe_active_params(
total_params: int,
*model_refs: str | None,
) -> int | None:
"""Resolve active params from curated data or A*B naming hints."""
for ref in model_refs:
if not ref:
continue
active = _lookup_curated_count(_KNOWN_MOE_ACTIVE_PARAMS, ref)
if active and active > 0:
return active
for ref in model_refs:
active = _extract_active_size_hint_from_id(ref)
if active and active > 0 and (total_params <= 0 or active < total_params):
return active
return None
def _normalize_param_count(
extracted: int,
model_id: str,
base_model: str | None,
) -> int:
"""Normalize parameter count when metadata is inconsistent."""
authoritative = _lookup_curated_count(_AUTHORITATIVE_PARAM_COUNTS, model_id)
if authoritative and authoritative > 0:
return authoritative
known = _lookup_curated_count(_KNOWN_PARAM_COUNTS, model_id)
if extracted <= 0:
return known or extracted
if known and extracted < int(known * 0.35):
return known
hints = [
h
for h in (
_extract_size_hint_from_id(model_id),
_extract_size_hint_from_id(base_model),
)
if h is not None
]
if not hints:
return extracted
hinted = max(hints)
if _is_quantized_repo_name(model_id):
if extracted < int(hinted * 0.70):
return hinted
elif extracted < int(hinted * 0.35):
return hinted
return extracted
# Curated MoE active-parameter counts. Used when HF config lacks the
# `num_local_experts` / `num_experts_per_tok` keys that whichllm reads.
# Without this, frontier MoEs are scored as dense models which over-counts
# their VRAM cost and under-counts their inference speed.
_KNOWN_MOE_ACTIVE_PARAMS: dict[str, int] = {
"meta-llama/Llama-4-Scout-17B-16E-Instruct": 17_000_000_000,
"meta-llama/Llama-4-Maverick-17B-128E-Instruct": 17_000_000_000,
"Qwen/Qwen3-Next-80B-A3B-Instruct": 3_000_000_000,
"Qwen/Qwen3-30B-A3B": 3_000_000_000,
"Qwen/Qwen3-Coder-30B-A3B-Instruct": 3_000_000_000,
"Qwen/Qwen3-235B-A22B": 22_000_000_000,
"Qwen/Qwen3.5-397B-A17B": 17_000_000_000,
"deepseek-ai/DeepSeek-V3": 37_000_000_000,
"deepseek-ai/DeepSeek-V3-0324": 37_000_000_000,
"deepseek-ai/DeepSeek-V3.1": 37_000_000_000,
"deepseek-ai/DeepSeek-V3.2": 37_000_000_000,
"deepseek-ai/DeepSeek-V3.2-Exp": 37_000_000_000,
"deepseek-ai/DeepSeek-R1": 37_000_000_000,
"deepseek-ai/DeepSeek-R1-0528": 37_000_000_000,
"deepseek-ai/DeepSeek-V4-Pro": 49_000_000_000,
"deepseek-ai/DeepSeek-V4-Flash": 13_000_000_000,
"zai-org/GLM-4.5": 32_000_000_000,
"zai-org/GLM-4.5-Air": 12_000_000_000,
"zai-org/GLM-4.6": 32_000_000_000,
"zai-org/GLM-4.7": 32_000_000_000,
"zai-org/GLM-4.7-Flash": 12_000_000_000,
"zai-org/GLM-5": 40_000_000_000,
"zai-org/GLM-5-FP8": 40_000_000_000,
"zai-org/GLM-5.1": 40_000_000_000,
"zai-org/GLM-5.1-FP8": 40_000_000_000,
"moonshotai/Kimi-K2-Instruct": 32_000_000_000,
"moonshotai/Kimi-K2-Thinking": 32_000_000_000,
"MiniMaxAI/MiniMax-M2": 10_000_000_000,
"MiniMaxAI/MiniMax-M2.5": 10_000_000_000,
"XiaomiMiMo/MiMo-V2.5": 15_000_000_000,
"XiaomiMiMo/MiMo-V2.5-Pro": 42_000_000_000,
"XiaomiMiMo/MiMo-V2-Flash": 15_000_000_000,
"google/gemma-4-26B-A4B-it": 3_800_000_000,
"nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16": 3_000_000_000,
"nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8": 3_000_000_000,
"nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16": 12_000_000_000,
# OpenAI gpt-oss MoE family - 5B active for 20b/120b.
"openai/gpt-oss-20b": 3_600_000_000,
"openai/gpt-oss-120b": 5_100_000_000,
}
# Hardcoded parameter counts for frontier models that HF's API leaves with
# missing safetensors/gguf/config metadata. Used as a last-resort fallback
# inside _extract_param_count so these models still enter the cache and become
# rankable. Maintain only entries that lack a size hint in the model ID itself.
_KNOWN_PARAM_COUNTS: dict[str, int] = {
"microsoft/phi-4": 14_700_000_000,
"microsoft/Phi-4-mini-instruct": 3_800_000_000,
"microsoft/Phi-4-multimodal-instruct": 5_600_000_000,
"microsoft/Phi-4-reasoning": 14_700_000_000,
"microsoft/Phi-4-reasoning-plus": 14_700_000_000,
"openai/gpt-oss-20b": 20_000_000_000,
"openai/gpt-oss-120b": 120_000_000_000,
# IBM Granite 4.0 family
"ibm-granite/granite-4.0-h-small": 32_000_000_000,
"ibm-granite/granite-4.0-h-tiny": 7_000_000_000,
"ibm-granite/granite-3.3-8b-instruct": 8_000_000_000,
"ibm-granite/granite-3.3-2b-instruct": 2_000_000_000,
# AllenAI Olmo-3
"allenai/Olmo-3-7B-Instruct": 7_000_000_000,
"allenai/Olmo-3-1025-7B": 7_000_000_000,
# Llama 4 MoE totals: repo names advertise active size, but the total
# weight footprint is much larger.
"meta-llama/Llama-4-Scout-17B-16E-Instruct": 109_000_000_000,
"meta-llama/Llama-4-Maverick-17B-128E-Instruct": 400_000_000_000,
"deepseek-ai/DeepSeek-R1": 671_000_000_000,
"deepseek-ai/DeepSeek-R1-0528": 671_000_000_000,
"deepseek-ai/DeepSeek-V3": 671_000_000_000,
"deepseek-ai/DeepSeek-V3-0324": 671_000_000_000,
"deepseek-ai/DeepSeek-V3.1": 671_000_000_000,
"deepseek-ai/DeepSeek-V3.2": 685_000_000_000,
"deepseek-ai/DeepSeek-V4-Pro": 1_600_000_000_000,
"deepseek-ai/DeepSeek-V4-Flash": 284_000_000_000,
"moonshotai/Kimi-K2-Instruct": 1_026_000_000_000,
"moonshotai/Kimi-K2-Thinking": 1_026_000_000_000,
"XiaomiMiMo/MiMo-V2.5": 310_000_000_000,
"XiaomiMiMo/MiMo-V2.5-Pro": 1_020_000_000_000,
"XiaomiMiMo/MiMo-V2-Flash": 309_000_000_000,
"zai-org/GLM-4.5": 355_000_000_000,
"zai-org/GLM-4.5-Air": 106_000_000_000,
"zai-org/GLM-4.6": 355_000_000_000,
"zai-org/GLM-4.7": 355_000_000_000,
"zai-org/GLM-4.7-Flash": 30_000_000_000,
"zai-org/GLM-5": 744_000_000_000,
"zai-org/GLM-5-FP8": 744_000_000_000,
"zai-org/GLM-5.1": 744_000_000_000,
"zai-org/GLM-5.1-FP8": 744_000_000_000,
"MiniMaxAI/MiniMax-M2": 230_000_000_000,
"MiniMaxAI/MiniMax-M2.5": 230_000_000_000,
"stepfun-ai/Step-3.5-Flash": 30_000_000_000,
}
# Curated counts that should win even when the HF API exposes safetensors
# metadata. Some mixed-precision MoEs publish compressed checkpoint tensor
# counts that understate the model-card capacity used for ranking and planning.
_AUTHORITATIVE_PARAM_COUNTS: dict[str, int] = {
"deepseek-ai/DeepSeek-V4-Pro": 1_600_000_000_000,
"deepseek-ai/DeepSeek-V4-Flash": 284_000_000_000,
}
src/whichllm/models/parser.py
+295
View File
@@ -0,0 +1,295 @@
"""Parse HuggingFace API model payloads into ModelInfo objects."""
from __future__ import annotations
import statistics
from whichllm.models.gguf import _extract_gguf_variants
from whichllm.models.parameters import (
_AUTHORITATIVE_PARAM_COUNTS,
_KNOWN_PARAM_COUNTS,
_extract_size_hint_from_id,
_lookup_curated_count,
_normalize_param_count,
_resolve_moe_active_params,
)
from whichllm.models.sliding_window import _resolve_sliding_window
from whichllm.models.types import ModelInfo
_GENERAL_EVAL_KEYWORDS = (
"mmlu",
"gpqa",
"gsm8k",
"hellaswag",
"arc",
"bbh",
"ifeval",
"truthfulqa",
"ceval",
"cmmlu",
)
def _extract_published_at(data: dict) -> str | None:
"""Extract the best published timestamp candidate from an API response."""
created = data.get("createdAt")
if isinstance(created, str) and created:
return created
modified = data.get("lastModified")
if isinstance(modified, str) and modified:
return modified
return None
def _normalize_eval_value(raw: object) -> float | None:
"""Convert eval value to a comparable 0-100 score."""
if not isinstance(raw, (int, float)):
return None
value = float(raw)
if value <= 0:
return None
if value <= 1.0:
value *= 100.0
if value > 100.0:
return None
return value
def _is_general_eval_entry(entry: dict) -> bool:
"""Keep eval entries that are broadly useful for general chat quality."""
data = entry.get("data")
if not isinstance(data, dict):
return False
notes = str(data.get("notes", "")).lower()
if "with tools" in notes:
return False
dataset = data.get("dataset")
dataset_id = ""
task_id = ""
if isinstance(dataset, dict):
dataset_id = str(dataset.get("id", "")).lower()
task_id = str(dataset.get("task_id", "")).lower()
filename = str(entry.get("filename", "")).lower()
return any(
k in dataset_id or k in task_id or k in filename for k in _GENERAL_EVAL_KEYWORDS
)
def _extract_hf_eval_score(data: dict) -> float | None:
"""Extract conservative aggregate score from HF evalResults."""
eval_results = data.get("evalResults")
if not isinstance(eval_results, list) or not eval_results:
return None
values: list[float] = []
for entry in eval_results:
if not isinstance(entry, dict):
continue
if not _is_general_eval_entry(entry):
continue
data_obj = entry.get("data")
if not isinstance(data_obj, dict):
continue
normalized = _normalize_eval_value(data_obj.get("value"))
if normalized is not None:
values.append(normalized)
if not values:
return None
return round(statistics.median(values), 1)
def _extract_param_count(model_data: dict) -> int:
"""Extract parameter count from model data.
Resolution order:
1. authoritative model-card overrides for known mixed-precision MoEs
2. safetensors metadata
3. gguf metadata
4. config estimate
5. curated known counts
6. name-based size hint
Returns 0 if none of the above succeed.
"""
model_id = model_data.get("id", "") or ""
authoritative = _lookup_curated_count(_AUTHORITATIVE_PARAM_COUNTS, model_id)
if authoritative and authoritative > 0:
return authoritative
safetensors = model_data.get("safetensors")
if safetensors and isinstance(safetensors, dict):
params = safetensors.get("total")
if params:
return int(params)
parameters = safetensors.get("parameters")
if isinstance(parameters, dict):
total = sum(parameters.values())
if total > 0:
return total
gguf_meta = model_data.get("gguf", {}) or {}
if isinstance(gguf_meta, dict):
total = gguf_meta.get("total")
if total and total > 0:
return int(total)
config = model_data.get("config", {}) or {}
hidden = config.get("hidden_size", 0)
layers = config.get("num_hidden_layers", 0)
vocab = config.get("vocab_size", 0)
if hidden and layers and vocab:
return 12 * layers * hidden * hidden + vocab * hidden * 2
known = _lookup_curated_count(_KNOWN_PARAM_COUNTS, model_id)
if known and known > 0:
return known
name_hint = _extract_size_hint_from_id(model_id)
if name_hint and name_hint > 0:
return name_hint
return 0
def _extract_architecture(config: dict) -> str:
"""Extract architecture string from config."""
arch_list = config.get("architectures", [])
if arch_list:
arch = arch_list[0].lower()
for name in [
"llama",
"qwen2",
"mistral",
"mixtral",
"gemma",
"phi",
"starcoder",
"command",
"deepseek",
]:
if name in arch:
return name
return arch.replace("forcausallm", "").replace("forconditionalgeneration", "")
model_type = config.get("model_type", "")
return model_type.lower()
def _extract_base_model(card_data: dict) -> str | None:
base_model_raw = card_data.get("base_model")
if isinstance(base_model_raw, str):
return base_model_raw
if isinstance(base_model_raw, list) and base_model_raw:
return base_model_raw[0]
return None
def _resolve_active_params(
config: dict,
param_count: int,
model_id: str,
base_model: str | None,
) -> tuple[bool, int | None]:
num_experts = 0
for k in (
"num_local_experts",
"num_experts",
"n_routed_experts",
"moe_num_experts",
"num_moe_experts",
"n_local_experts",
):
v = config.get(k, 0)
if isinstance(v, int) and v > num_experts:
num_experts = v
experts_per_tok = 0
for k in (
"num_experts_per_tok",
"moe_topk",
"moe_top_k",
"num_experts_per_token",
"top_k",
):
v = config.get(k, 0)
if isinstance(v, int) and v > experts_per_tok:
experts_per_tok = v
known_moe_active = _resolve_moe_active_params(param_count, model_id, base_model)
is_moe = num_experts > 0 or known_moe_active is not None
active_params = None
if is_moe:
if known_moe_active is not None:
active_params = known_moe_active
elif num_experts > 0:
ept = experts_per_tok if experts_per_tok > 0 else 2
active_ratio = ept / num_experts
expert_fraction = 0.6
active_params = int(
param_count * (1 - expert_fraction + expert_fraction * active_ratio)
)
return is_moe, active_params
def _parse_model(data: dict) -> ModelInfo | None:
"""Parse HF API response into ModelInfo."""
model_id = data.get("id", "")
if not model_id:
return None
config = data.get("config", {}) or {}
card_data = data.get("cardData", {}) or {}
base_model = _extract_base_model(card_data)
param_count = _extract_param_count(data)
param_count = _normalize_param_count(param_count, model_id, base_model)
if param_count == 0:
return None
is_moe, active_params = _resolve_active_params(
config, param_count, model_id, base_model
)
gguf_variants = _extract_gguf_variants(data, param_count)
architecture = _extract_architecture(config)
gguf_meta = data.get("gguf", {}) or {}
if not architecture and isinstance(gguf_meta, dict):
architecture = gguf_meta.get("architecture", "")
context_length = config.get("max_position_embeddings") or config.get(
"max_sequence_length"
)
if not context_length and isinstance(gguf_meta, dict):
context_length = gguf_meta.get("context_length")
gguf_arch = gguf_meta.get("architecture") if isinstance(gguf_meta, dict) else None
sliding_window, swa_global_ratio = _resolve_sliding_window(
config, model_id, gguf_arch
)
benchmark_scores: dict[str, float] = {}
eval_score = _extract_hf_eval_score(data)
if eval_score is not None:
benchmark_scores["hf_eval"] = eval_score
return ModelInfo(
id=model_id,
family_id=model_id,
name=model_id.split("/")[-1],
parameter_count=param_count,
parameter_count_active=active_params,
architecture=architecture,
is_moe=is_moe,
context_length=context_length,
license=card_data.get("license"),
published_at=_extract_published_at(data),
downloads=data.get("downloads", 0),
likes=data.get("likes", 0),
gguf_variants=gguf_variants,
benchmark_scores=benchmark_scores,
base_model=base_model,
sliding_window=sliding_window,
sliding_window_global_ratio=swa_global_ratio,
)
src/whichllm/models/serialization.py
+94
View File
@@ -0,0 +1,94 @@
"""ModelInfo cache serialization helpers."""
from __future__ import annotations
from whichllm.models.parameters import (
_normalize_param_count,
_resolve_moe_active_params,
)
from whichllm.models.types import GGUFVariant, ModelInfo
def models_to_dicts(models: list[ModelInfo]) -> list[dict]:
"""Serialize models to dicts for caching."""
result = []
for m in models:
result.append(
{
"id": m.id,
"family_id": m.family_id,
"name": m.name,
"parameter_count": m.parameter_count,
"parameter_count_active": m.parameter_count_active,
"architecture": m.architecture,
"is_moe": m.is_moe,
"context_length": m.context_length,
"license": m.license,
"published_at": m.published_at,
"downloads": m.downloads,
"likes": m.likes,
"gguf_variants": [
{
"filename": v.filename,
"quant_type": v.quant_type,
"file_size_bytes": v.file_size_bytes,
}
for v in m.gguf_variants
],
"benchmark_scores": m.benchmark_scores,
"base_model": m.base_model,
"sliding_window": m.sliding_window,
"sliding_window_global_ratio": m.sliding_window_global_ratio,
}
)
return result
def dicts_to_models(data: list[dict]) -> list[ModelInfo]:
"""Deserialize models from cached dicts."""
models = []
for d in data:
base_model = d.get("base_model")
param_count = _normalize_param_count(
d["parameter_count"],
d["id"],
base_model,
)
active_params = _resolve_moe_active_params(
param_count,
d["id"],
base_model,
d.get("name"),
d.get("architecture"),
)
if active_params is None:
active_params = d.get("parameter_count_active")
models.append(
ModelInfo(
id=d["id"],
family_id=d.get("family_id", d["id"]),
name=d["name"],
parameter_count=param_count,
parameter_count_active=active_params,
architecture=d.get("architecture", ""),
is_moe=d.get("is_moe", False) or active_params is not None,
context_length=d.get("context_length"),
license=d.get("license"),
published_at=d.get("published_at"),
downloads=d.get("downloads", 0),
likes=d.get("likes", 0),
gguf_variants=[
GGUFVariant(
filename=v["filename"],
quant_type=v["quant_type"],
file_size_bytes=v["file_size_bytes"],
)
for v in d.get("gguf_variants", [])
],
benchmark_scores=d.get("benchmark_scores", {}),
base_model=base_model,
sliding_window=d.get("sliding_window"),
sliding_window_global_ratio=d.get("sliding_window_global_ratio"),
)
)
return models
src/whichllm/models/sliding_window.py
+95
View File
@@ -0,0 +1,95 @@
"""Sliding-window-attention metadata resolution."""
from __future__ import annotations
# Sliding-window-attention (SWA) registry. We only model SWA KV-cache savings
# for architectures whose mainline runtimes actually honor interleaved SWA
# (llama.cpp's ISWA path, MLX). Each entry is (default_window_tokens,
# global_layer_ratio) where the ratio is the fraction of layers that use full
# (global) attention. Models outside this allowlist keep full-context KV so
# estimates stay conservative.
_SWA_ARCH_DEFAULTS: dict[str, tuple[int, float]] = {
"gemma2": (4096, 0.5),
"gemma3": (1024, 1.0 / 6.0),
"gpt_oss": (128, 0.5),
"cohere2": (4096, 0.25),
}
# Map the many spellings an arch string can take (HF model_type, the
# ForCausalLM/ForConditionalGeneration class prefix, and GGUF metadata) onto a
# canonical key in _SWA_ARCH_DEFAULTS.
_SWA_ARCH_ALIASES: dict[str, str] = {
"gemma2": "gemma2",
"gemma2_text": "gemma2",
"gemma3": "gemma3",
"gemma3_text": "gemma3",
"gpt_oss": "gpt_oss",
"gptoss": "gpt_oss",
"cohere2": "cohere2",
}
def _swa_key_from_arch(arch: str | None) -> str | None:
"""Resolve an arch string (model_type / class / gguf metadata) to a key."""
if not arch:
return None
arch = arch.lower()
if arch in _SWA_ARCH_ALIASES:
return _SWA_ARCH_ALIASES[arch]
stripped = arch.replace("forcausallm", "").replace("forconditionalgeneration", "")
if stripped in _SWA_ARCH_ALIASES:
return _SWA_ARCH_ALIASES[stripped]
return None
def _swa_arch_key(config: dict, model_id: str, gguf_arch: str | None) -> str | None:
"""Identify the SWA architecture key for a model, or None if not honored.
Relies on authoritative metadata only: raw HF config model_type /
architectures and GGUF metadata architecture. When none are present the
model is left unhonored (full-context estimate), since a false positive
would under-count VRAM.
"""
model_type = config.get("model_type")
key = _swa_key_from_arch(model_type if isinstance(model_type, str) else None)
if key:
return key
arch_list = config.get("architectures") or []
if arch_list and isinstance(arch_list[0], str):
key = _swa_key_from_arch(arch_list[0])
if key:
return key
return _swa_key_from_arch(gguf_arch)
def _resolve_sliding_window(
config: dict, model_id: str, gguf_arch: str | None = None
) -> tuple[int | None, float | None]:
"""Resolve (sliding_window, global_ratio) for honored SWA architectures.
Returns (None, None) for every model outside the allowlist so the KV
estimate stays at full context (conservative).
"""
if config.get("use_sliding_window") is False:
return None, None
key = _swa_arch_key(config, model_id, gguf_arch)
if key is None:
return None, None
default_window, default_ratio = _SWA_ARCH_DEFAULTS[key]
window = config.get("sliding_window")
if not isinstance(window, int) or window <= 0:
window = default_window
pattern = config.get("sliding_window_pattern")
if isinstance(pattern, int) and pattern > 0:
global_ratio = 1.0 / pattern
else:
global_ratio = default_ratio
return window, global_ratio
src/whichllm/models/types.py
+46
View File
@@ -0,0 +1,46 @@
from __future__ import annotations
from dataclasses import dataclass, field
@dataclass
class GGUFVariant:
filename: str
quant_type: str # "Q4_K_M", "Q8_0" etc
file_size_bytes: int
@dataclass
class ModelInfo:
id: str # HF repo ID
family_id: str # grouping key
name: str
parameter_count: int # total parameters
parameter_count_active: int | None = None # MoE active params
architecture: str = "" # "llama", "qwen2", "mixtral" etc
is_moe: bool = False
context_length: int | None = None
license: str | None = None
published_at: str | None = None
downloads: int = 0
likes: int = 0
gguf_variants: list[GGUFVariant] = field(default_factory=list)
benchmark_scores: dict[str, float] = field(default_factory=dict)
base_model: str | None = None # cardData.base_model
# Sliding-window-attention KV-cache modeling. Only populated for
# architectures whose mainline runtimes actually honor interleaved SWA
# (Gemma-2/3, gpt-oss, Cohere2); left None otherwise so VRAM estimates
# stay conservative. sliding_window is the local-attention window in
# tokens; sliding_window_global_ratio is the fraction of layers that use
# full (global) attention (0.0 = pure SWA, 1.0 = fully dense).
sliding_window: int | None = None
sliding_window_global_ratio: float | None = None
@dataclass
class ModelFamily:
family_id: str
display_name: str
base_model: ModelInfo
variants: list[ModelInfo] = field(default_factory=list)
best_benchmark: dict[str, float] = field(default_factory=dict)
src/whichllm/output/__init__.py
View File
src/whichllm/output/_console.py
+11
View File
@@ -0,0 +1,11 @@
"""Canonical Rich Console instance shared by every output surface.
Tests patch the ``console`` attribute on this module to capture output
(e.g. ``whichllm.output._console.console = Console(file=buf, ...)``).
Surface modules look up the console via this module so the patch
propagates without each module holding its own binding.
"""
from rich.console import Console
console = Console()
src/whichllm/output/display.py
+40
View File
@@ -0,0 +1,40 @@
"""Compatibility shim: per-surface output modules now live alongside this file.
This module re-exports the public ``display_*`` functions so existing imports
(``from whichllm.output.display import display_ranking``) keep working. New
code should import from the specific submodule:
- ``whichllm.output.ranking`` for ranking + hardware tables
- ``whichllm.output.plan`` for the plan command
- ``whichllm.output.upgrade`` for the upgrade comparison
- ``whichllm.output.json_output`` for machine-readable JSON output
- ``whichllm.output.formatting`` for shared byte/param/date/color helpers
- ``whichllm.output._console`` for the shared Rich ``Console`` instance
The shared ``console`` symbol is re-exported here for read access. Code that
needs to *replace* the console (e.g. test capture) should set
``whichllm.output._console.console`` so every surface picks up the change.
"""
from whichllm.output._console import console
from whichllm.output.json_output import (
display_json,
display_plan_json,
display_upgrade_json,
)
from whichllm.output.markdown import display_markdown
from whichllm.output.plan import display_plan
from whichllm.output.ranking import display_hardware, display_ranking
from whichllm.output.upgrade import display_upgrade
__all__ = [
"console",
"display_hardware",
"display_json",
"display_markdown",
"display_plan",
"display_plan_json",
"display_ranking",
"display_upgrade",
"display_upgrade_json",
]
src/whichllm/output/formatting.py
+111
View File
@@ -0,0 +1,111 @@
"""Shared low-level helpers: byte/param/date formatters and color blending."""
from __future__ import annotations
from datetime import datetime
from math import log10
from whichllm.engine.types import CompatibilityResult
def _format_bytes(b: int) -> str:
"""Format bytes as human-readable string."""
if b >= 1024**3:
return f"{b / 1024**3:.1f} GB"
elif b >= 1024**2:
return f"{b / 1024**2:.0f} MB"
return f"{b / 1024:.0f} KB"
def _format_params(count: int) -> str:
"""Format parameter count."""
if count >= 1e9:
return f"{count / 1e9:.1f}B"
elif count >= 1e6:
return f"{count / 1e6:.0f}M"
return str(count)
def _format_downloads(downloads: int) -> str:
"""Format download count for compact table display."""
if downloads >= 1_000_000:
return f"{downloads / 1_000_000:.1f}M"
if downloads >= 1_000:
return f"{downloads / 1_000:.1f}K"
return str(downloads)
def _format_published_at(value: str | None) -> str:
"""Format published datetime into YYYY-MM-DD."""
if not value:
return "—"
try:
dt = datetime.fromisoformat(value.replace("Z", "+00:00"))
return dt.strftime("%Y-%m-%d")
except ValueError:
return value[:10] if len(value) >= 10 else value
def _format_speed(result: CompatibilityResult) -> str:
speed = result.estimated_tok_per_sec
if speed is None:
return "[grey50]N/A[/]"
base = f"{speed:.1f} tok/s"
if speed < 4.0:
style = "red"
elif speed < 10.0:
style = "yellow"
elif speed < 30.0:
style = "green"
else:
style = "bright_green"
marker = ""
if result.speed_confidence == "low":
marker = " ?"
elif result.speed_confidence == "medium":
marker = " ~"
return f"[{style}]{base}{marker}[/{style}]"
def _parse_published_at(value: str | None) -> datetime | None:
if not value:
return None
try:
return datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError:
return None
def _lerp_channel(a: int, b: int, t: float) -> int:
return int(a + (b - a) * t)
def _blend_hex(a: tuple[int, int, int], b: tuple[int, int, int], t: float) -> str:
t = max(0.0, min(1.0, t))
r = _lerp_channel(a[0], b[0], t)
g = _lerp_channel(a[1], b[1], t)
bch = _lerp_channel(a[2], b[2], t)
return f"#{r:02x}{g:02x}{bch:02x}"
def _downloads_style(downloads: int, min_log: float, max_log: float) -> str:
if downloads <= 0:
return "grey50"
dlog = log10(max(downloads, 1))
span = max(max_log - min_log, 1e-6)
t = (dlog - min_log) / span
return _blend_hex((145, 80, 80), (55, 190, 120), t)
def _published_style(
published: datetime | None,
oldest_ts: float | None,
newest_ts: float | None,
) -> str:
if published is None or oldest_ts is None or newest_ts is None:
return "grey50"
pts = published.timestamp()
span = max(newest_ts - oldest_ts, 1e-6)
t = (pts - oldest_ts) / span
return _blend_hex((190, 85, 85), (80, 190, 110), t)
src/whichllm/output/json_output.py
+210
View File
@@ -0,0 +1,210 @@
"""Machine-readable JSON output for ranking, plan, and upgrade surfaces."""
from __future__ import annotations
import json
from whichllm.engine.quantization import effective_quant_type, estimate_weight_bytes
from whichllm.engine.types import CompatibilityResult
from whichllm.hardware.types import HardwareInfo
from whichllm.models.types import GGUFVariant, ModelInfo
from whichllm.output import _console
from whichllm.output.upgrade import _summarize_row
def display_json(results: list[CompatibilityResult], hardware: HardwareInfo) -> None:
"""Output ranking results as JSON."""
output = {
"hardware": {
"gpus": [
{
"name": g.name,
"vendor": g.vendor,
"vram_bytes": g.vram_bytes,
"usable_vram_bytes": g.usable_vram_bytes,
"memory_bandwidth_gbps": g.memory_bandwidth_gbps,
"shared_memory": g.shared_memory,
}
for g in hardware.gpus
],
"cpu": hardware.cpu_name,
"cpu_cores": hardware.cpu_cores,
"ram_bytes": hardware.ram_bytes,
"ram_budget_bytes": hardware.ram_budget_bytes,
"budget_notes": hardware.budget_notes,
"os": hardware.os,
},
"models": [
{
"rank": i,
"model_id": r.model.id,
"artifact_repo_id": r.artifact_model.id if r.artifact_model else None,
"artifact_filename": (
r.artifact_variant.filename if r.artifact_variant else None
),
"parameter_count": r.model.parameter_count,
"published_at": r.model.published_at,
"downloads": r.model.downloads,
"quant_type": effective_quant_type(r.model, r.gguf_variant),
"file_size_bytes": (
r.gguf_variant.file_size_bytes
if r.gguf_variant
else estimate_weight_bytes(r.model, None)
),
"vram_required_bytes": r.vram_required_bytes,
"vram_available_bytes": r.vram_available_bytes,
"uses_multi_gpu": r.uses_multi_gpu,
"multi_gpu_effective_vram_bytes": r.multi_gpu_effective_vram_bytes,
"estimated_tok_per_sec": r.estimated_tok_per_sec,
"speed_confidence": r.speed_confidence,
"speed_range_tok_per_sec": (
list(r.speed_range_tok_per_sec)
if r.speed_range_tok_per_sec
else None
),
"speed_notes": r.speed_notes,
"quality_score": round(r.quality_score, 2),
"benchmark_status": r.benchmark_status,
"benchmark_source": r.benchmark_source,
"benchmark_confidence": round(r.benchmark_confidence, 2),
"fit_type": r.fit_type,
"can_run": r.can_run,
"warnings": r.warnings,
"license": r.model.license,
}
for i, r in enumerate(results, 1)
],
}
_console.console.print_json(json.dumps(output, ensure_ascii=False))
def display_plan_json(
model: ModelInfo,
context_length: int,
target_quant: str,
) -> None:
"""Output plan results as JSON."""
from whichllm.constants import (
GPU_BANDWIDTH,
QUANT_BYTES_PER_WEIGHT,
QUANT_QUALITY_PENALTY,
)
from whichllm.engine.performance import estimate_tok_per_sec
from whichllm.engine.vram import estimate_vram
from whichllm.hardware.types import GPUInfo
_GiB = 1024**3
quant_levels = ["Q2_K", "Q3_K_M", "Q4_K_M", "Q5_K_M", "Q6_K", "Q8_0", "F16"]
vram_by_quant = {}
for qt in quant_levels:
bpw = QUANT_BYTES_PER_WEIGHT.get(qt)
if bpw is None:
continue
fake_size = int(model.parameter_count * bpw)
fake_variant = GGUFVariant(
filename="", quant_type=qt, file_size_bytes=fake_size
)
vram_bytes = estimate_vram(model, fake_variant, context_length)
vram_by_quant[qt] = {
"vram_bytes": vram_bytes,
"quality_loss": QUANT_QUALITY_PENALTY.get(qt, 0.0),
}
target_vram = vram_by_quant.get(target_quant.upper(), {}).get("vram_bytes", 0)
if target_vram == 0:
bpw = QUANT_BYTES_PER_WEIGHT.get(target_quant.upper(), 0.5625)
fake_size = int(model.parameter_count * bpw)
fake_variant = GGUFVariant(
filename="", quant_type=target_quant, file_size_bytes=fake_size
)
target_vram = estimate_vram(model, fake_variant, context_length)
_PLAN_GPUS: list[tuple[str, int]] = [
("RTX 4060", 8),
("RTX 3060", 12),
("RTX 4070", 12),
("RTX 4080", 16),
("RTX 4090", 24),
("RX 7900 XTX", 24),
("RTX 5090", 32),
("A100 40GB", 40),
("L40S", 48),
("A100 80GB", 80),
("H100", 80),
("H200", 141),
]
bpw = QUANT_BYTES_PER_WEIGHT.get(target_quant.upper(), 0.5625)
fake_size = int(model.parameter_count * bpw)
fake_variant = GGUFVariant(
filename="", quant_type=target_quant, file_size_bytes=fake_size
)
gpus = []
for gpu_name, vram_gb in _PLAN_GPUS:
vram_bytes = int(vram_gb * _GiB)
bandwidth = GPU_BANDWIDTH.get(gpu_name)
gpu_info = GPUInfo(
name=gpu_name,
vendor="nvidia",
vram_bytes=vram_bytes,
memory_bandwidth_gbps=bandwidth,
)
if vram_bytes >= target_vram:
fit_type = "full_gpu"
elif vram_bytes >= target_vram * 0.4:
fit_type = "partial_offload"
else:
fit_type = "too_small"
speed = None
if fit_type != "too_small" and bandwidth:
speed = round(
estimate_tok_per_sec(model, fake_variant, gpu_info, fit_type), 1
)
gpus.append(
{
"name": gpu_name,
"vram_gb": vram_gb,
"fit_type": fit_type,
"estimated_tok_per_sec": speed,
}
)
output = {
"model": {
"id": model.id,
"parameter_count": model.parameter_count,
"architecture": model.architecture,
"context_length": model.context_length,
"license": model.license,
},
"target_quant": target_quant,
"context_length": context_length,
"vram_by_quant": vram_by_quant,
"gpu_compatibility": gpus,
}
_console.console.print_json(json.dumps(output, ensure_ascii=False))
def display_upgrade_json(
current_hw: HardwareInfo,
current_results: list,
target_results: list[tuple[str, HardwareInfo, list]],
) -> None:
"""Emit the upgrade comparison as JSON for scripting."""
current_row = _summarize_row("Current", current_hw, current_results)
rows = []
for name, hw, res in target_results:
row = _summarize_row(name, hw, res)
row["delta_quality"] = row["top_quality"] - current_row["top_quality"]
row["delta_tok_s"] = row["top_tok_s"] - current_row["top_tok_s"]
rows.append(row)
_console.console.print_json(
json.dumps(
{"current": current_row, "targets": rows},
ensure_ascii=False,
)
)
src/whichllm/output/markdown.py
+154
View File
@@ -0,0 +1,154 @@
"""GitHub-Flavored Markdown output for ranking results."""
from __future__ import annotations
from whichllm.engine.quantization import effective_quant_type
from whichllm.engine.types import CompatibilityResult
from whichllm.hardware.types import HardwareInfo
from whichllm.output import _console
from whichllm.output.formatting import (
_format_bytes,
_format_downloads,
_format_params,
_format_published_at,
)
def _escape_markdown_cell(value: object) -> str:
text = "" if value is None else str(value)
return text.replace("\\", "\\\\").replace("|", "\\|").replace("\n", "<br>")
def _format_markdown_speed(result: CompatibilityResult) -> str:
speed = result.estimated_tok_per_sec
if speed is None:
return "N/A"
marker = ""
if result.speed_confidence == "low":
marker = " ?"
elif result.speed_confidence == "medium":
marker = " ~"
return f"{speed:.1f} tok/s{marker}"
def _format_markdown_score(result: CompatibilityResult) -> str:
score = f"{result.quality_score:.1f}"
if result.benchmark_status == "none":
return f"{score} ?"
if result.benchmark_status == "self_reported":
return f"{score} !sr"
if result.benchmark_status == "estimated":
return f"{score} ~"
return score
def _format_markdown_fit(fit_type: str) -> str:
labels = {
"full_gpu": "Full GPU",
"partial_offload": "Partial",
"cpu_only": "CPU only",
}
return labels.get(fit_type, fit_type)
def _format_markdown_params(result: CompatibilityResult) -> str:
params = _format_params(result.model.parameter_count)
if result.model.is_moe and result.model.parameter_count_active:
params += f" ({_format_params(result.model.parameter_count_active)}a)"
return params
def _format_markdown_model(result: CompatibilityResult) -> str:
if not result.artifact_model:
return result.model.id
return f"[{result.model.id}](https://huggingface.co/{result.artifact_model.id})"
def _markdown_table(headers: list[str], rows: list[list[str]]) -> str:
lines = [
"| " + " | ".join(headers) + " |",
"| " + " | ".join("---" for _ in headers) + " |",
]
for row in rows:
lines.append(
"| " + " | ".join(_escape_markdown_cell(cell) for cell in row) + " |"
)
return "\n".join(lines)
def _write_markdown(text: str) -> None:
_console.console.file.write(text + "\n")
_console.console.file.flush()
def display_markdown(
results: list[CompatibilityResult],
hardware: HardwareInfo,
*,
show_status: bool = False,
empty_message: str | None = None,
) -> None:
"""Emit ranking results as a pasteable GitHub-Flavored Markdown table."""
lines = ["## Recommended Models", ""]
if not results:
lines.append(empty_message or "No compatible models found for your hardware.")
_write_markdown("\n".join(lines))
return
if show_status:
mem_label = "VRAM" if hardware.gpus else "RAM"
headers = [
"#",
"Model",
"Params",
"Quant",
"Fit",
mem_label,
"Speed",
"Published",
"Score",
"License",
]
rows = [
[
str(index),
_format_markdown_model(result),
_format_markdown_params(result),
effective_quant_type(result.model, result.gguf_variant),
_format_markdown_fit(result.fit_type),
_format_bytes(result.vram_required_bytes),
_format_markdown_speed(result),
_format_published_at(result.model.published_at),
_format_markdown_score(result),
result.model.license or "-",
]
for index, result in enumerate(results, 1)
]
else:
headers = [
"#",
"Model",
"Params",
"Quant",
"Published",
"Downloads",
"Score",
"License",
]
rows = [
[
str(index),
_format_markdown_model(result),
_format_markdown_params(result),
effective_quant_type(result.model, result.gguf_variant),
_format_published_at(result.model.published_at),
_format_downloads(result.model.downloads),
_format_markdown_score(result),
result.model.license or "-",
]
for index, result in enumerate(results, 1)
]
lines.append(_markdown_table(headers, rows))
_write_markdown("\n".join(lines))
src/whichllm/output/plan.py
+158
View File
@@ -0,0 +1,158 @@
"""Plan-command Rich output."""
from __future__ import annotations
from rich.panel import Panel
from rich.table import Table
from whichllm.models.types import GGUFVariant, ModelInfo
from whichllm.output import _console
from whichllm.output.formatting import _format_bytes, _format_params
def display_plan(
model: ModelInfo,
context_length: int,
target_quant: str,
) -> None:
"""Display hardware requirements for a specific model."""
from whichllm.constants import (
GPU_BANDWIDTH,
QUANT_BYTES_PER_WEIGHT,
QUANT_QUALITY_PENALTY,
)
from whichllm.engine.performance import estimate_tok_per_sec
from whichllm.engine.vram import estimate_vram
from whichllm.hardware.types import GPUInfo
_GiB = 1024**3
# -- Model info panel --
params = _format_params(model.parameter_count)
active = ""
if model.is_moe and model.parameter_count_active:
active = f" ({_format_params(model.parameter_count_active)} active)"
ctx = str(model.context_length) if model.context_length else "unknown"
lines = [
f"[bold cyan]Model:[/] {model.id}",
f"[bold cyan]Params:[/] {params}{active} | Arch: {model.architecture} | Context: {ctx}",
]
if model.license:
lines.append(f"[bold cyan]License:[/] {model.license}")
panel = Panel("\n".join(lines), title="[bold]Model Info[/]", border_style="cyan")
_console.console.print(panel)
# -- VRAM requirements by quantization --
quant_levels = ["Q2_K", "Q3_K_M", "Q4_K_M", "Q5_K_M", "Q6_K", "Q8_0", "F16"]
vram_table = Table(
title=f"VRAM Required (context: {context_length})", show_lines=True
)
vram_table.add_column("Quant", style="bold", width=8)
vram_table.add_column("VRAM", justify="right", width=10)
vram_table.add_column("Quality Loss", justify="right", width=12)
target_vram = 0
for qt in quant_levels:
bpw = QUANT_BYTES_PER_WEIGHT.get(qt)
if bpw is None:
continue
fake_size = int(model.parameter_count * bpw)
fake_variant = GGUFVariant(
filename="", quant_type=qt, file_size_bytes=fake_size
)
vram_bytes = estimate_vram(model, fake_variant, context_length)
penalty = QUANT_QUALITY_PENALTY.get(qt, 0.0)
penalty_str = f"-{penalty * 100:.0f}%" if penalty > 0 else "0%"
marker = " ★" if qt.upper() == target_quant.upper() else ""
style = "bold green" if qt.upper() == target_quant.upper() else ""
vram_table.add_row(
f"{qt}{marker}", _format_bytes(vram_bytes), penalty_str, style=style
)
if qt.upper() == target_quant.upper():
target_vram = vram_bytes
_console.console.print(vram_table)
if target_vram == 0:
bpw = QUANT_BYTES_PER_WEIGHT.get(target_quant.upper(), 0.5625)
fake_size = int(model.parameter_count * bpw)
fake_variant = GGUFVariant(
filename="", quant_type=target_quant, file_size_bytes=fake_size
)
target_vram = estimate_vram(model, fake_variant, context_length)
# -- GPU compatibility table --
_PLAN_GPUS: list[tuple[str, int]] = [
("RTX 4060", 8),
("RTX 3060", 12),
("RTX 4070", 12),
("RTX 4080", 16),
("RTX 4090", 24),
("RX 7900 XTX", 24),
("RTX 5090", 32),
("A100 40GB", 40),
("L40S", 48),
("A100 80GB", 80),
("H100", 80),
("H200", 141),
]
gpu_table = Table(
title=f"GPU Compatibility ({target_quant}, {_format_bytes(target_vram)} required)",
show_lines=True,
)
gpu_table.add_column("GPU", style="bold", min_width=14)
gpu_table.add_column("VRAM", justify="right", width=8)
gpu_table.add_column("Fit", justify="center", width=12)
gpu_table.add_column("Est. Speed", justify="right", width=10)
bpw = QUANT_BYTES_PER_WEIGHT.get(target_quant.upper(), 0.5625)
fake_size = int(model.parameter_count * bpw)
fake_variant = GGUFVariant(
filename="", quant_type=target_quant, file_size_bytes=fake_size
)
min_full_gpu = None
for gpu_name, vram_gb in _PLAN_GPUS:
vram_bytes = int(vram_gb * _GiB)
bandwidth = GPU_BANDWIDTH.get(gpu_name)
gpu_info = GPUInfo(
name=gpu_name,
vendor="nvidia",
vram_bytes=vram_bytes,
memory_bandwidth_gbps=bandwidth,
)
if vram_bytes >= target_vram:
fit = "[green]✓ Full GPU[/]"
fit_type = "full_gpu"
if min_full_gpu is None:
min_full_gpu = (gpu_name, vram_gb)
elif vram_bytes >= target_vram * 0.4:
fit = "[yellow]~ Partial[/]"
fit_type = "partial_offload"
else:
fit = "[red]✗ Too small[/]"
fit_type = None
if fit_type and bandwidth:
speed = estimate_tok_per_sec(model, fake_variant, gpu_info, fit_type)
speed_str = f"{speed:.1f} tok/s"
else:
speed_str = "—"
gpu_table.add_row(gpu_name, f"{vram_gb} GB", fit, speed_str)
_console.console.print(gpu_table)
if min_full_gpu:
_console.console.print(
f" [green]★[/] Minimum GPU for full offload: "
f"[bold]{min_full_gpu[0]}[/] ({min_full_gpu[1]} GB) at {target_quant}"
)
else:
_console.console.print(
f" [yellow]Note:[/] No single GPU can fully load this model at {target_quant}. "
"Consider a lower quantization or multi-GPU setup."
)
src/whichllm/output/ranking.py
+348
View File
@@ -0,0 +1,348 @@
"""Ranking and hardware Rich output surfaces."""
from __future__ import annotations
import re
from math import log10
from rich.panel import Panel
from rich.table import Table
from rich.text import Text
from whichllm.engine.quantization import effective_quant_type
from whichllm.engine.types import CompatibilityResult
from whichllm.hardware.types import HardwareInfo
from whichllm.output import _console
from whichllm.output.formatting import (
_downloads_style,
_format_bytes,
_format_downloads,
_format_params,
_format_published_at,
_format_speed,
_parse_published_at,
_published_style,
)
def _detect_specializations(model_id: str) -> list[str]:
"""Detect task-specialized model hints from repository name."""
lower = model_id.lower()
tags: list[str] = []
if re.search(r"(coder|codegen|starcoder|program|coding)", lower):
tags.append("coding")
if re.search(r"(^|[-_/])(vl|vision|multimodal|llava|image)([-_/]|$)", lower):
tags.append("vision")
if re.search(r"(^|[-_/])math([-_/]|$)", lower):
tags.append("math")
return tags
def _artifact_model_id(result: CompatibilityResult) -> str:
if result.artifact_model:
return result.artifact_model.id
return result.model.id
def _top_pick_confidence(results: list[CompatibilityResult]) -> tuple[str, str]:
"""Return confidence level and explanation for top pick."""
top = results[0]
gap = (top.quality_score - results[1].quality_score) if len(results) > 1 else 999.0
notes: list[str] = []
if top.fit_type == "partial_offload":
notes.append("partial offload")
elif top.fit_type == "cpu_only":
notes.append("CPU-only")
if top.speed_confidence == "low":
notes.append("low-confidence speed")
risk_note = f", {', '.join(notes)}" if notes else ""
if top.benchmark_status == "none":
return "Low", f"no benchmark data, gap +{gap:.1f}{risk_note}"
if top.benchmark_status == "self_reported":
return (
"Low",
f"uploader-reported benchmark only (unverified), gap +{gap:.1f}{risk_note}",
)
if top.benchmark_status == "estimated":
if gap >= 2.0:
confidence = "Medium"
else:
confidence = "Low"
if top.speed_confidence == "low" and confidence == "Medium":
confidence = "Low"
return confidence, f"estimated benchmark, gap +{gap:.1f}{risk_note}"
if gap >= 2.5:
confidence = "High"
reason = f"direct benchmark, gap +{gap:.1f}{risk_note}"
elif gap >= 1.0:
confidence = "Medium"
reason = f"direct benchmark, gap +{gap:.1f}{risk_note}"
else:
confidence = "Low"
reason = f"direct benchmark but very close (+{gap:.1f}){risk_note}"
# オフロード/CPU-only/低信頼speedの1位は実運用で不確実性が高いため信頼度を1段階下げる
if top.fit_type != "full_gpu" or top.speed_confidence == "low":
if confidence == "High":
confidence = "Medium"
elif confidence == "Medium":
confidence = "Low"
return confidence, reason
def display_hardware(hw: HardwareInfo) -> None:
"""Display hardware information panel."""
lines: list[str] = []
if hw.gpus:
for i, gpu in enumerate(hw.gpus):
if gpu.shared_memory:
vram = (
f"{_format_bytes(gpu.vram_bytes)} shared"
if gpu.vram_bytes > 0
else "shared memory"
)
else:
vram = _format_bytes(gpu.vram_bytes)
if (
gpu.usable_vram_bytes is not None
and gpu.usable_vram_bytes < gpu.vram_bytes
):
vram += f" (budget {_format_bytes(gpu.usable_vram_bytes)})"
bw = (
f"{gpu.memory_bandwidth_gbps:.0f} GB/s"
if gpu.memory_bandwidth_gbps
else "N/A"
)
cc = (
f"CC {gpu.compute_capability[0]}.{gpu.compute_capability[1]}"
if gpu.compute_capability
else ""
)
extra = []
if cc:
extra.append(cc)
if gpu.cuda_version:
extra.append(f"CUDA {gpu.cuda_version}")
if gpu.rocm_version:
extra.append(f"ROCm {gpu.rocm_version}")
extra_str = f" ({', '.join(extra)})" if extra else ""
lines.append(
f"[bold green]GPU {i}:[/] {gpu.name} — {vram}{extra_str} — BW: {bw}"
)
else:
lines.append("[yellow]No GPU detected[/] — CPU-only mode")
avx_flags = []
if hw.has_avx2:
avx_flags.append("AVX2")
if hw.has_avx512:
avx_flags.append("AVX-512")
avx_str = f" ({', '.join(avx_flags)})" if avx_flags else ""
lines.append(f"[bold blue]CPU:[/] {hw.cpu_name} — {hw.cpu_cores} cores{avx_str}")
ram = _format_bytes(hw.ram_bytes)
if hw.ram_budget_bytes is not None and hw.ram_budget_bytes < hw.ram_bytes:
ram += f" (budget {_format_bytes(hw.ram_budget_bytes)})"
lines.append(f"[bold blue]RAM:[/] {ram}")
lines.append(f"[bold blue]Disk free:[/] {_format_bytes(hw.disk_free_bytes)}")
lines.append(f"[bold blue]OS:[/] {hw.os}")
for note in hw.budget_notes:
lines.append(f"[dim]{note}[/dim]")
panel = Panel("\n".join(lines), title="[bold]Hardware Info[/]", border_style="blue")
_console.console.print(panel)
def display_ranking(
results: list[CompatibilityResult],
*,
has_gpu: bool = True,
show_status: bool = False,
empty_message: str | None = None,
) -> None:
"""Display ranked model table."""
if not results:
_console.console.print(
f"[yellow]{empty_message or 'No compatible models found for your hardware.'}[/]"
)
return
mem_label = "VRAM" if has_gpu else "RAM"
table = Table(title="Recommended Models", show_lines=True)
table.add_column("#", style="bold", width=3, justify="right")
table.add_column("Model", style="cyan", min_width=14, overflow="fold")
table.add_column("Quant", justify="center", width=6)
if show_status:
table.add_column(f"Fit / {mem_label}", justify="center", width=8)
table.add_column("Speed", justify="right", width=12)
table.add_column("Published", justify="center", width=10)
else:
table.add_column("Params", justify="right", width=6)
table.add_column("Published", justify="center", width=10)
table.add_column("Downloads", justify="right", width=9)
table.add_column("Score", justify="right", width=5)
download_logs = [
log10(max(r.model.downloads, 1)) for r in results if r.model.downloads > 0
]
min_download_log = min(download_logs) if download_logs else 0.0
max_download_log = max(download_logs) if download_logs else 1.0
published_dates = [_parse_published_at(r.model.published_at) for r in results]
published_valid = [d for d in published_dates if d is not None]
oldest_ts = min((d.timestamp() for d in published_valid), default=None)
newest_ts = max((d.timestamp() for d in published_valid), default=None)
for i, r in enumerate(results, 1):
quant = effective_quant_type(r.model, r.gguf_variant)
vram_str = _format_bytes(r.vram_required_bytes)
speed_str = _format_speed(r)
score_val = f"{r.quality_score:.1f}"
if r.benchmark_status == "none":
score_str = f"[red]{score_val} ?[/red]"
elif r.benchmark_status == "self_reported":
score_str = f"[bright_yellow]{score_val} !sr[/bright_yellow]"
elif r.benchmark_status == "estimated":
score_str = f"[yellow]{score_val} ~[/yellow]"
else:
score_str = f"[green]{score_val}[/green]"
fit_style = {
"full_gpu": "[green]Full GPU[/]",
"partial_offload": "[yellow]Partial[/]",
"cpu_only": "[red]CPU only[/]",
}
fit_str = fit_style.get(r.fit_type, r.fit_type)
published_dt = _parse_published_at(r.model.published_at)
published_str = Text(
_format_published_at(r.model.published_at),
style=_published_style(published_dt, oldest_ts, newest_ts),
)
downloads_str = Text(
_format_downloads(r.model.downloads),
style=_downloads_style(
r.model.downloads, min_download_log, max_download_log
),
)
params_str = _format_params(r.model.parameter_count)
if r.model.is_moe and r.model.parameter_count_active:
params_str += f" ({_format_params(r.model.parameter_count_active)}a)"
model_link = Text(r.model.id, style="cyan")
model_link.stylize(f"link https://huggingface.co/{_artifact_model_id(r)}")
if show_status:
model_link.append(f"\n{params_str}", style="dim")
row_cells = [
str(i),
model_link,
quant,
]
if show_status:
row_cells.extend(
[f"{fit_str}\n[dim]{vram_str}[/dim]", speed_str, published_str]
)
else:
row_cells.append(params_str)
row_cells.extend([published_str, downloads_str])
row_cells.append(score_str)
table.add_row(*row_cells)
_console.console.print(table)
has_estimated = any(r.benchmark_status == "estimated" for r in results)
has_self = any(r.benchmark_status == "self_reported" for r in results)
has_none = any(r.benchmark_status == "none" for r in results)
if has_estimated or has_none or has_self:
parts = []
if has_self:
parts.append(
"[bright_yellow]!sr[/bright_yellow] = uploader-reported only (unverified)"
)
if has_estimated:
parts.append("[yellow]Estimated / ~[/yellow] = inferred from model line")
if has_none:
parts.append("[red]None / ?[/red] = no benchmark data")
_console.console.print(f" [dim]Score:[/dim] {', '.join(parts)}")
if show_status:
has_speed_medium = any(r.speed_confidence == "medium" for r in results)
has_speed_low = any(r.speed_confidence == "low" for r in results)
if has_speed_medium or has_speed_low:
parts = []
if has_speed_medium:
parts.append("[yellow]~[/yellow] = estimated tok/s range")
if has_speed_low:
parts.append("[red]?[/red] = low-confidence/backend-sensitive tok/s")
_console.console.print(f" [dim]Speed:[/dim] {', '.join(parts)}")
has_direct = any(r.benchmark_status == "direct" for r in results)
if not has_direct:
_console.console.print(
" [red]No confirmed winner:[/] direct benchmark data is missing for current candidates."
)
confidence, reason = _top_pick_confidence(results)
confidence_style = {
"High": "green",
"Medium": "yellow",
"Low": "red",
}[confidence]
_console.console.print(
f" Top pick confidence: [{confidence_style}]{confidence}[/{confidence_style}] ({reason})"
)
from whichllm.models.benchmark_sources import BENCHMARK_SNAPSHOT
_console.console.print(
f" [dim]Benchmark reference: {BENCHMARK_SNAPSHOT} curated snapshot; "
"live AA / LiveBench / Aider merged when reachable.[/dim]"
)
# 上位が僅差なら「断定しすぎない」ための注意を表示する
if len(results) >= 2:
gap = results[0].quality_score - results[1].quality_score
if gap < 1.5:
_console.console.print(
f" [yellow]Note:[/] Top candidates are very close (#{1} vs #{2}: {gap:.1f} pts)."
)
# 上位に根拠が弱い候補がある場合は目立つ注意を出す
weak_top = [
idx + 1 for idx, r in enumerate(results[:3]) if r.benchmark_status != "direct"
]
if weak_top:
joined = ", ".join(f"#{i}" for i in weak_top)
_console.console.print(
f" [yellow]Caution:[/] Weaker benchmark evidence in top ranks: {joined}"
)
weak_speed_top = [
idx + 1 for idx, r in enumerate(results[:3]) if r.speed_confidence == "low"
]
if weak_speed_top:
joined = ", ".join(f"#{i}" for i in weak_speed_top)
_console.console.print(
f" [yellow]Speed caution:[/] Low-confidence speed estimates in top ranks: {joined}"
)
specialized: list[str] = []
for idx, r in enumerate(results[:10], 1):
tags = _detect_specializations(r.model.id)
if tags:
joined_tags = "/".join(tags)
specialized.append(f"#{idx} {joined_tags}")
if specialized:
_console.console.print(
" [yellow]Task hint:[/] Specialized models detected in ranking: "
+ ", ".join(specialized)
)
for i, r in enumerate(results[:3], 1):
if r.warnings:
for w in r.warnings:
_console.console.print(f" [yellow]Warning #{i} {r.model.name}:[/] {w}")
src/whichllm/output/upgrade.py
+126
View File
@@ -0,0 +1,126 @@
"""Upgrade-command Rich output."""
from __future__ import annotations
from rich.table import Table
from whichllm.engine.quantization import effective_quant_type
from whichllm.hardware.types import HardwareInfo
from whichllm.output import _console
def _summarize_row(name: str, hw: HardwareInfo, results: list) -> dict:
"""Reduce a (hardware, ranking) pair to one row for the upgrade table."""
gpu_label = "CPU-only"
vram_gb = 0.0
if hw.gpus:
g = max(hw.gpus, key=lambda x: x.vram_bytes)
gpu_label = g.name
vram_gb = g.vram_bytes / 1024**3
if not results:
return {
"name": name,
"gpu": gpu_label,
"vram_gb": vram_gb,
"top_model": "—",
"top_quality": 0.0,
"top_tok_s": 0.0,
"top_speed_confidence": "low",
"top_speed_range_tok_per_sec": None,
"top_fit": "—",
"top_quant": "—",
}
r = results[0]
return {
"name": name,
"gpu": gpu_label,
"vram_gb": vram_gb,
"top_model": r.model.id,
"top_quality": float(r.quality_score),
"top_tok_s": float(r.estimated_tok_per_sec),
"top_speed_confidence": r.speed_confidence,
"top_speed_range_tok_per_sec": (
list(r.speed_range_tok_per_sec) if r.speed_range_tok_per_sec else None
),
"top_fit": r.fit_type,
"top_quant": (
r.gguf_variant.quant_type
if r.gguf_variant
else effective_quant_type(r.model, None)
),
}
def _upgrade_verdict(delta_q: float, delta_speed: float) -> str:
"""Return a short verdict for an upgrade row."""
if delta_q >= 12 and delta_speed >= 10:
return "[bold green]worth it[/]"
if delta_q >= 8 or delta_speed >= 20:
return "[green]meaningful[/]"
if delta_q >= 3 or delta_speed >= 5:
return "[yellow]marginal[/]"
if delta_q <= -3 or delta_speed <= -5:
return "[red]downgrade[/]"
return "[dim]flat[/]"
def display_upgrade(
current_hw: HardwareInfo,
current_results: list,
target_results: list[tuple[str, HardwareInfo, list]],
) -> None:
"""Render the GPU-upgrade comparison table."""
current_row = _summarize_row("Current", current_hw, current_results)
target_rows = [_summarize_row(name, hw, res) for name, hw, res in target_results]
table = Table(
title="GPU upgrade comparison",
show_lines=False,
header_style="bold cyan",
)
table.add_column("Setup", style="bold")
table.add_column("GPU", overflow="fold")
table.add_column("VRAM", justify="right")
table.add_column("Best model", overflow="fold")
table.add_column("Quant")
table.add_column("Quality", justify="right")
table.add_column("tok/s", justify="right")
table.add_column("ΔQ", justify="right")
table.add_column("Δtok/s", justify="right")
table.add_column("Verdict")
table.add_row(
current_row["name"],
current_row["gpu"],
f"{current_row['vram_gb']:.0f} GB"
if current_row["vram_gb"] is not None
else "—",
current_row["top_model"],
current_row["top_quant"],
f"{current_row['top_quality']:.1f}",
f"{current_row['top_tok_s']:.0f}",
"—",
"—",
"—",
)
for row in target_rows:
dq = row["top_quality"] - current_row["top_quality"]
ds = row["top_tok_s"] - current_row["top_tok_s"]
table.add_row(
row["name"],
row["gpu"],
f"{row['vram_gb']:.0f} GB" if row["vram_gb"] is not None else "—",
row["top_model"],
row["top_quant"],
f"{row['top_quality']:.1f}",
f"{row['top_tok_s']:.0f}",
f"{dq:+.1f}",
f"{ds:+.0f}",
_upgrade_verdict(dq, ds),
)
_console.console.print(table)
_console.console.print(
"[dim]Verdict: worth it (≥12pt Q & ≥10 tok/s lift) · meaningful (≥8pt Q or "
"≥20 tok/s) · marginal · flat (no change) · downgrade.[/]"
)
src/whichllm/utils.py
+76
View File
@@ -0,0 +1,76 @@
from __future__ import annotations
import os
import re
from importlib.metadata import version, PackageNotFoundError
from pathlib import Path
import click
def _current_version() -> str:
"""Return installed package version."""
try:
return version("whichllm")
except PackageNotFoundError:
return "unknown"
_SHORTHAND_RE = re.compile(r"^(\d+(?:\.\d+)?)\s*([kmb])$", re.IGNORECASE)
_MULTIPLIERS = {"k": 1024, "m": 1024 * 1024, "b": 1024 * 1024 * 1024}
def parse_context_length(value: str) -> int:
"""Parse a context length string, supporting shorthand like 64k or 128K.
Accepts plain integers (e.g. "4096") or shorthand with a suffix:
k/K = x1,024 (64k -> 65536)
m/M = x1,048,576
b/B = x1,073,741,824
Returns the integer context length. Raises ValueError on bad input.
"""
value = value.strip()
match = _SHORTHAND_RE.match(value)
if match:
number = float(match.group(1))
suffix = match.group(2).lower()
result = int(number * _MULTIPLIERS[suffix])
if result <= 0:
raise ValueError(f"Context length must be positive, got {value!r}")
return result
try:
result = int(value)
except ValueError:
raise ValueError(
f"Invalid context length {value!r}. "
"Use a plain integer (4096) or shorthand (64k, 128k)."
)
if result <= 0:
raise ValueError(f"Context length must be positive, got {result}")
return result
class ContextLengthType(click.ParamType):
"""Click parameter type that accepts integers or shorthand like 64k."""
name = "context_length"
def convert(self, value, param, ctx):
if isinstance(value, int):
return value
try:
return parse_context_length(str(value))
except ValueError as e:
self.fail(str(e), param, ctx)
CONTEXT_LENGTH = ContextLengthType()
def _cache_dir() -> Path:
"""Return the whichllm cache directory, respecting XDG_CACHE_HOME."""
base = os.environ.get("XDG_CACHE_HOME")
if base and Path(base).is_absolute():
return Path(base) / "whichllm"
return Path.home() / ".cache" / "whichllm"
tests/__init__.py
View File
tests/test_aa_index.py
+137
View File
@@ -0,0 +1,137 @@
"""Tests for the Artificial Analysis Intelligence Index source.
These cover the Next.js App Router (RSC) scraper that replaced the old
``__NEXT_DATA__`` extraction, the variant-stripping name canonicalization,
and the merge-over-curated-fallback behaviour of ``fetch_aa_index_scores``.
All tests are offline — network is served from an ``httpx.MockTransport``.
"""
from __future__ import annotations
import asyncio
import json
import httpx
import pytest
from whichllm.models.benchmark_sources.aa_index import (
AA_LEADERBOARD_URL,
_canonical_name,
_decode_rsc_blob,
_extract_aa_pairs_from_html,
_normalize_aa_index,
fetch_aa_index_scores,
get_aa_curated_fallback,
)
from whichllm.models.benchmark_sources.types import ExtractionFailed
def _rsc_page(records: list[dict]) -> str:
"""Build a minimal HTML page that embeds ``records`` the way the live
artificialanalysis.ai App Router page does: as a JSON-string-escaped
fragment inside ``self.__next_f.push([n, "..."])``."""
# The fragment is an arbitrary slice of the RSC stream; the scraper only
# cares that it contains the "name"/"intelligenceIndex" key pairs.
fragment = ",".join(
'{"slug":"x","name":%s,"reasoningModel":false,'
'"intelligenceIndex":%s,"codingIndex":1.0}'
% (json.dumps(r["name"]), r["index"])
for r in records
)
chunk = json.dumps("3:[" + fragment + "]\n")
return (
"<!DOCTYPE html><html><body>"
"<script>self.__next_f.push([0])</script>"
f"<script>self.__next_f.push([1,{chunk}])</script>"
"</body></html>"
)
def test_canonical_name_strips_variants_and_separators():
assert _canonical_name("Qwen3 14B (Reasoning)") == "qwen3 14b"
assert _canonical_name("Qwen3-14B") == "qwen3 14b"
# Separators normalize to single spaces (the table side is canonicalized
# the same way, so "GLM-5" and "GLM 5" still collide).
assert _canonical_name("GLM-5 (Non-reasoning)") == "glm 5"
assert _canonical_name("DeepSeek V4 Pro (Reasoning, Max Effort)") == (
"deepseek v4 pro"
)
def test_decode_rsc_blob_unescapes_chunks():
page = _rsc_page([{"name": "Qwen3 14B (Reasoning)", "index": 33.0}])
blob = _decode_rsc_blob(page)
assert '"name":"Qwen3 14B (Reasoning)"' in blob
assert '"intelligenceIndex":33.0' in blob
def test_extract_pairs_from_rsc_html():
page = _rsc_page(
[
{"name": "Qwen3 14B (Reasoning)", "index": 33.0},
{"name": "Qwen3 14B (Non-reasoning)", "index": 30.0},
{"name": "GLM-5 (Reasoning)", "index": 50.0},
]
)
pairs = dict(_extract_aa_pairs_from_html(page))
assert pairs["Qwen3 14B (Reasoning)"] == 33.0
assert pairs["GLM-5 (Reasoning)"] == 50.0
# The bounded regex must not leak one record's name into another's index.
assert len(pairs) == 3
def test_extract_pairs_returns_empty_on_legacy_or_garbage_html():
assert _extract_aa_pairs_from_html("<html>no rsc here</html>") == []
def _run_fetch(html: str) -> dict[str, float]:
def handler(request: httpx.Request) -> httpx.Response:
assert str(request.url) == AA_LEADERBOARD_URL
return httpx.Response(200, text=html)
async def go() -> dict[str, float]:
transport = httpx.MockTransport(handler)
async with httpx.AsyncClient(transport=transport) as client:
return await fetch_aa_index_scores(client)
return asyncio.run(go())
def test_fetch_maps_canonical_names_and_merges_over_fallback():
# "Qwen3 14B (Reasoning)" canonicalizes onto the "Qwen3 14B" table entry
# -> Qwen/Qwen3-14B, and a high live value must override the snapshot.
page = _rsc_page([{"name": "Qwen3 14B (Reasoning)", "index": 55.0}])
scores = _run_fetch(page)
fallback = get_aa_curated_fallback()
# Coverage never shrinks below the curated snapshot ...
assert set(fallback).issubset(set(scores))
# ... and the live number wins where it is higher.
assert scores["Qwen/Qwen3-14B"] > fallback["Qwen/Qwen3-14B"]
def test_fetch_raises_when_no_records_found():
with pytest.raises(ExtractionFailed):
_run_fetch("<html><body>nothing to see</body></html>")
def test_live_normalization_anchors_on_reworked_scale():
# Retuned bounds keep the calibration: the top mapped open model lands ~95
# and an 8B-class model lands ~40, on AA's reworked (compressed) raw scale.
assert _normalize_aa_index(44.3) == pytest.approx(95, abs=0.5) # top open model
assert _normalize_aa_index(7.4) == pytest.approx(40, abs=0.5) # 8B-class
# Values below the floor clamp at 0 (raw is always positive in practice).
assert _normalize_aa_index(-100.0) == 0.0
assert _normalize_aa_index(60.0) == 100.0
def test_curated_fallback_normalizes_refreshed_snapshot():
# The snapshot holds refreshed raw AA values; get_aa_curated_fallback maps
# them onto the 0-100 scale with the retuned bounds.
fb = get_aa_curated_fallback()
assert fb["deepseek-ai/DeepSeek-V4-Pro"] == pytest.approx(95, abs=0.5)
assert fb["Qwen/Qwen3-8B"] == 40.0
assert fb["XiaomiMiMo/MiMo-V2.5-Pro"] == pytest.approx(92, abs=0.5)
# Reworked scale ranks the strong 8B above the small/old peers.
assert fb["Qwen/Qwen3-8B"] > fb["Qwen/Qwen3-0.6B"]
assert all(0.0 < v <= 100.0 for v in fb.values())
tests/test_amd_detection.py
+302
View File
@@ -0,0 +1,302 @@
"""Tests for AMD GPU detection fallbacks."""
from __future__ import annotations
import subprocess
from io import StringIO
from rich.console import Console
from whichllm.hardware import amd
from whichllm.hardware.types import GPUInfo, HardwareInfo
def test_detect_amd_gpu_from_lspci_when_rocm_smi_missing(monkeypatch):
output = (
'c1:00.0 "VGA compatible controller" "Advanced Micro Devices, Inc. '
'[AMD/ATI]" "Strix Halo [Radeon 8060S]" -r00 "Framework" "Device 0001"\n'
)
def fake_run(args, **kwargs):
if args[0] == "rocm-smi":
raise FileNotFoundError
return subprocess.CompletedProcess(args, 0, stdout=output, stderr="")
monkeypatch.setattr(amd.subprocess, "run", fake_run)
gpus = amd.detect_amd_gpus()
assert len(gpus) == 1
assert gpus[0].vendor == "amd"
assert gpus[0].vram_bytes == 0
assert gpus[0].shared_memory is True
assert gpus[0].memory_bandwidth_gbps == 256.0
assert "Radeon 8060S" in gpus[0].name
def test_detect_strix_halo_rocm_smi_does_not_treat_aperture_as_vram(monkeypatch):
def fake_run(args, **kwargs):
if args[:2] == ["rocm-smi", "--showproductname"]:
return subprocess.CompletedProcess(
args,
0,
stdout='{"card0": {"Card SKU": "STRXLGEN"}}',
stderr="",
)
if args[:3] == ["rocm-smi", "--showmeminfo", "vram"]:
return subprocess.CompletedProcess(
args,
0,
stdout='{"card0": {"VRAM Total Memory (B)": "536870912"}}',
stderr="",
)
if args[:2] == ["rocm-smi", "--showdriverversion"]:
return subprocess.CompletedProcess(
args,
0,
stdout='{"card0": {"Driver version": "7.0.3"}}',
stderr="",
)
raise AssertionError(args)
monkeypatch.setattr(amd.subprocess, "run", fake_run)
gpus = amd.detect_amd_gpus()
assert len(gpus) == 1
assert gpus[0].name == "STRXLGEN"
assert gpus[0].vendor == "amd"
assert gpus[0].shared_memory is True
assert gpus[0].vram_bytes == 0
assert gpus[0].rocm_version == "7.0.3"
assert gpus[0].memory_bandwidth_gbps == 256.0
def test_detect_amd_gpu_ignores_intel_only_lspci(monkeypatch):
"""Regression: an Intel VGA row must not be reported as AMD just
because 'Intel Corporation' contains the substring 'ati'."""
output = (
'00:02.0 "VGA compatible controller" "Intel Corporation" '
'"Alder Lake-P GT1 [UHD Graphics]" -r0c -p00 '
'"IP3 Tech (HK) Limited" "Device 8027"\n'
)
def fake_run(args, **kwargs):
if args[0] == "rocm-smi":
raise FileNotFoundError
return subprocess.CompletedProcess(args, 0, stdout=output, stderr="")
monkeypatch.setattr(amd.subprocess, "run", fake_run)
# Isolate the lspci path: don't let a real sysfs probe leak in.
monkeypatch.setattr(amd, "_detect_from_sysfs", lambda: [])
assert amd.detect_amd_gpus() == []
def test_detect_amd_gpu_from_sysfs_when_lspci_missing(monkeypatch, tmp_path):
card = tmp_path / "card0" / "device"
card.mkdir(parents=True)
(card / "vendor").write_text("0x1002\n")
(card / "product_name").write_text("AMD Radeon RX 9060 XT\n")
(card / "mem_info_vram_total").write_text(str(16 * 1024**3))
monkeypatch.setattr(amd, "_detect_from_lspci", lambda: [])
original_sysfs = amd._detect_from_sysfs
monkeypatch.setattr(amd, "_detect_from_sysfs", lambda: original_sysfs(tmp_path))
gpus = amd._detect_amd_gpus_fallback()
assert len(gpus) == 1
assert gpus[0].vendor == "amd"
assert gpus[0].name == "AMD Radeon RX 9060 XT"
assert gpus[0].vram_bytes == 16 * 1024**3
assert gpus[0].shared_memory is False
def test_display_amd_shared_memory_without_zero_kb(monkeypatch):
from whichllm.output import _console as console_mod
from whichllm.output import display as display_mod
buf = StringIO()
monkeypatch.setattr(console_mod, "console", Console(file=buf, force_terminal=False))
display_mod.display_hardware(
HardwareInfo(
gpus=[
GPUInfo(
name="Strix Halo [Radeon 8060S]",
vendor="amd",
vram_bytes=0,
shared_memory=True,
)
],
cpu_name="CPU",
cpu_cores=16,
ram_bytes=128 * 1024**3,
disk_free_bytes=100 * 1024**3,
os="linux",
)
)
output = buf.getvalue()
assert "shared memory" in output
assert "256 GB/s" not in output
assert "0 KB" not in output
# ---------- Issue #61: RX 6750 XT detection ----------
def test_sysfs_generic_name_enriched_by_lspci(monkeypatch, tmp_path):
"""When sysfs gives 'AMD Graphics' and lspci gives a descriptive name,
the fallback should use the lspci name with sysfs VRAM."""
_GiB = 1024**3
# sysfs: generic name but has VRAM
card = tmp_path / "card0" / "device"
card.mkdir(parents=True)
(card / "vendor").write_text("0x1002\n")
(card / "mem_info_vram_total").write_text(str(12 * _GiB))
# no product_name → falls back to "AMD Graphics"
lspci_name = "Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M/6850M XT]"
original_sysfs = amd._detect_from_sysfs
monkeypatch.setattr(amd, "_detect_from_sysfs", lambda: original_sysfs(tmp_path))
monkeypatch.setattr(amd, "_detect_from_lspci", lambda: [lspci_name])
gpus = amd._detect_amd_gpus_fallback()
assert len(gpus) == 1
assert gpus[0].name == lspci_name
assert gpus[0].vram_bytes == 12 * _GiB
assert gpus[0].shared_memory is False
def test_sysfs_product_name_preferred_over_lspci(monkeypatch, tmp_path):
"""When sysfs gives a real product name, it should be used even if
lspci is also available."""
_GiB = 1024**3
card = tmp_path / "card0" / "device"
card.mkdir(parents=True)
(card / "vendor").write_text("0x1002\n")
(card / "product_name").write_text("AMD Radeon RX 6750 XT\n")
(card / "mem_info_vram_total").write_text(str(12 * _GiB))
original_sysfs = amd._detect_from_sysfs
monkeypatch.setattr(amd, "_detect_from_sysfs", lambda: original_sysfs(tmp_path))
monkeypatch.setattr(
amd,
"_detect_from_lspci",
lambda: ["Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M/6850M XT]"],
)
gpus = amd._detect_amd_gpus_fallback()
assert len(gpus) == 1
assert gpus[0].name == "AMD Radeon RX 6750 XT"
assert gpus[0].vram_bytes == 12 * _GiB
assert gpus[0].memory_bandwidth_gbps == 432.0
def test_lspci_enriched_with_sysfs_vram_when_sysfs_detection_fails(
monkeypatch, tmp_path
):
"""When _detect_from_sysfs returns nothing but _read_sysfs_amd_vram
succeeds, lspci names should still get VRAM data."""
_GiB = 1024**3
# _detect_from_sysfs returns nothing (e.g. product_name absent AND
# the card dir structure confuses the glob), but individual VRAM reads
# via _read_sysfs_amd_vram still work.
monkeypatch.setattr(amd, "_detect_from_sysfs", lambda: [])
monkeypatch.setattr(
amd,
"_detect_from_lspci",
lambda: ["Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M/6850M XT]"],
)
monkeypatch.setattr(amd, "_read_sysfs_amd_vram", lambda: [12 * _GiB])
gpus = amd._detect_amd_gpus_fallback()
assert len(gpus) == 1
assert gpus[0].vram_bytes == 12 * _GiB
assert gpus[0].shared_memory is False
def test_lookup_bandwidth_compound_lspci_name():
"""The bandwidth lookup should resolve compound lspci names by
splitting on '/' and re-applying the 'RX ' prefix."""
# Direct substring match works for clean names
assert amd._lookup_bandwidth("AMD Radeon RX 6750 XT") == 432.0
assert amd._lookup_bandwidth("AMD Radeon RX 6700 XT") == 384.0
# Compound lspci name — first matching segment wins
compound = "Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M/6850M XT]"
bw = amd._lookup_bandwidth(compound)
assert bw is not None
assert bw > 0
def test_display_amd_dgpu_does_not_say_shared_memory(monkeypatch):
"""A discrete AMD GPU with VRAM must NOT display 'shared memory'."""
from whichllm.output import _console as console_mod
from whichllm.output import display as display_mod
buf = StringIO()
monkeypatch.setattr(console_mod, "console", Console(file=buf, force_terminal=False))
display_mod.display_hardware(
HardwareInfo(
gpus=[
GPUInfo(
name="AMD Radeon RX 6750 XT",
vendor="amd",
vram_bytes=12 * 1024**3,
memory_bandwidth_gbps=432.0,
shared_memory=False,
)
],
cpu_name="AMD Ryzen 9 5900X",
cpu_cores=12,
ram_bytes=128 * 1024**3,
disk_free_bytes=500 * 1024**3,
os="linux",
)
)
output = buf.getvalue()
assert "12.0 GB" in output
assert "432 GB/s" in output
assert "shared memory" not in output
def test_display_amd_dgpu_zero_vram_does_not_say_shared_memory(monkeypatch):
"""An AMD dGPU with undetected VRAM should NOT be labelled
'shared memory' — that would be a false positive."""
from whichllm.output import _console as console_mod
from whichllm.output import display as display_mod
buf = StringIO()
monkeypatch.setattr(console_mod, "console", Console(file=buf, force_terminal=False))
display_mod.display_hardware(
HardwareInfo(
gpus=[
GPUInfo(
name="Navi 22 [Radeon RX 6750 XT]",
vendor="amd",
vram_bytes=0,
shared_memory=False,
)
],
cpu_name="CPU",
cpu_cores=8,
ram_bytes=32 * 1024**3,
disk_free_bytes=100 * 1024**3,
os="linux",
)
)
output = buf.getvalue()
assert "shared memory" not in output

Some files were not shown because too many files have changed in this diff Show More