From 69235fc2ecb98def4e86502f45725e2b22a0647e Mon Sep 17 00:00:00 2001 From: wehub-resource-sync Date: Mon, 13 Jul 2026 08:34:04 +0000 Subject: [PATCH] docs: make Chinese README the default --- README.md | 497 +++++++++++++++++++++++++++--------------------------- 1 file changed, 250 insertions(+), 247 deletions(-) diff --git a/README.md b/README.md index e98f2b7..13c0e5a 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,9 @@ + +> [!NOTE] +> 本文档由 WeHub 基于上游 README 翻译整理，属于社区翻译，非官方中文文档。 +> [English](./README.en.md) · [原始项目](https://github.com/ggml-org/llama.cpp) · [上游 README](https://github.com/ggml-org/llama.cpp/blob/HEAD/README.md) +> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。 + # llama.cpp ![llama](https://raw.githubusercontent.com/ggml-org/llama.brand/refs/heads/master/cover/llama-cpp/cover-llama-cpp-dark.svg) @@ -10,41 +16,41 @@ [Manifesto](https://github.com/ggml-org/llama.cpp/discussions/205) / [ggml](https://github.com/ggml-org/ggml) / [ops](https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md) -LLM inference in C/C++ +C/C++ 实现的 LLM 推理 -## Recent API changes +## 近期 API 变更 -- [Changelog for `libllama` API](https://github.com/ggml-org/llama.cpp/issues/9289) -- [Changelog for `llama-server` REST API](https://github.com/ggml-org/llama.cpp/issues/9291) +- [`libllama` API 的更新日志](https://github.com/ggml-org/llama.cpp/issues/9289) +- [`llama-server` REST API 的更新日志](https://github.com/ggml-org/llama.cpp/issues/9291) -## Hot topics +## 热门话题 -- **Hugging Face cache migration: models downloaded with `-hf` are now stored in the standard Hugging Face cache directory, enabling sharing with other HF tools.** -- **[guide : using the new WebUI of llama.cpp](https://github.com/ggml-org/llama.cpp/discussions/16938)** -- [guide : running gpt-oss with llama.cpp](https://github.com/ggml-org/llama.cpp/discussions/15396) -- [[FEEDBACK] Better packaging for llama.cpp to support downstream consumers 🤗](https://github.com/ggml-org/llama.cpp/discussions/15313) -- Support for the `gpt-oss` model with native MXFP4 format has been added | [PR](https://github.com/ggml-org/llama.cpp/pull/15091) | [Collaboration with NVIDIA](https://blogs.nvidia.com/blog/rtx-ai-garage-openai-oss) | [Comment](https://github.com/ggml-org/llama.cpp/discussions/15095) -- Multimodal support arrived in `llama-server`: [#12898](https://github.com/ggml-org/llama.cpp/pull/12898) | [documentation](./docs/multimodal.md) -- VS Code extension for FIM completions: https://github.com/ggml-org/llama.vscode -- Vim/Neovim plugin for FIM completions: https://github.com/ggml-org/llama.vim -- Hugging Face Inference Endpoints now support GGUF out of the box! https://github.com/ggml-org/llama.cpp/discussions/9669 -- Hugging Face GGUF editor: [discussion](https://github.com/ggml-org/llama.cpp/discussions/9268) | [tool](https://huggingface.co/spaces/CISCai/gguf-editor) -- WebGPU support is now available in the browser, see a blog/demo introducing it [here](https://reeselevine.github.io/llamas-on-the-web/). +- **Hugging Face 缓存迁移：使用 `-hf` 下载的模型现已存储在标准 Hugging Face 缓存目录中，可与其他 HF 工具共享。** +- **[指南：使用 llama.cpp 的新版 WebUI](https://github.com/ggml-org/llama.cpp/discussions/16938)** +- [指南：使用 llama.cpp 运行 gpt-oss](https://github.com/ggml-org/llama.cpp/discussions/15396) +- [[反馈] 为 llama.cpp 提供更好的打包方式以支持下游使用者 🤗](https://github.com/ggml-org/llama.cpp/discussions/15313) +- 已添加对原生 MXFP4 格式的 `gpt-oss` 模型的支持 | [PR](https://github.com/ggml-org/llama.cpp/pull/15091) | [与 NVIDIA 的合作](https://blogs.nvidia.com/blog/rtx-ai-garage-openai-oss) | [评论](https://github.com/ggml-org/llama.cpp/discussions/15095) +- 多模态支持已登陆 `llama-server`：[#12898](https://github.com/ggml-org/llama.cpp/pull/12898) | [文档](./docs/multimodal.md) +- 用于 FIM 补全的 VS Code 扩展：https://github.com/ggml-org/llama.vscode +- 用于 FIM 补全的 Vim/Neovim 插件：https://github.com/ggml-org/llama.vim +- Hugging Face Inference Endpoints 现已原生支持 GGUF！https://github.com/ggml-org/llama.cpp/discussions/9669 +- Hugging Face GGUF 编辑器：[讨论](https://github.com/ggml-org/llama.cpp/discussions/9268) | [工具](https://huggingface.co/spaces/CISCai/gguf-editor) +- 浏览器中现已支持 WebGPU，点击[此处](https://reeselevine.github.io/llamas-on-the-web/).查看介绍博客/演示 ---- -## Quick start +## 快速开始 -Getting started with llama.cpp is straightforward. Here are several ways to install it on your machine: +入门 llama.cpp 非常简单。以下是在你的机器上安装的几种方式： -- Install `llama.cpp` using [brew, nix, winget, or conda-forge](docs/install.md) -- Run with Docker - see our [Docker documentation](docs/docker.md) -- Download pre-built binaries from the [releases page](https://github.com/ggml-org/llama.cpp/releases) -- Build from source by cloning this repository - check out [our build guide](docs/build.md) +- 使用 [brew、nix、winget 或 conda-forge](docs/install.md) 安装 `llama.cpp` +- 使用 Docker 运行——查看我们的 [Docker 文档](docs/docker.md) +- 从 [发布页面](https://github.com/ggml-org/llama.cpp/releases) 下载预编译二进制文件 +- 克隆本仓库并从源码构建——查看[我们的构建指南](docs/build.md) -Once installed, you'll need a model to work with. Head to the [Obtaining and quantizing models](#obtaining-and-quantizing-models) section to learn more. +安装完成后，你还需要一个模型才能开始使用。请前往[获取模型与量化](#obtaining-and-quantizing-models)章节了解更多。 -Example command: +示例命令： ```sh # Use a local model file @@ -57,30 +63,29 @@ llama-cli -hf ggml-org/gemma-3-1b-it-GGUF llama-server -hf ggml-org/gemma-3-1b-it-GGUF ``` -## Description +## 描述 -The main goal of `llama.cpp` is to enable LLM inference with minimal setup and state-of-the-art performance on a wide -range of hardware - locally and in the cloud. +`llama.cpp` 的主要目标是在本地和云端的各种硬件上，以最少的配置实现 LLM 推理，并提供业界领先的性能。 -- Plain C/C++ implementation without any dependencies -- Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks -- AVX, AVX2, AVX512 and AMX support for x86 architectures -- RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures -- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use -- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA) -- Vulkan and SYCL backend support -- CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity +- 纯 C/C++ 实现，无任何依赖 +- Apple silicon 为第一等公民——通过 ARM NEON、Accelerate 和 Metal 框架优化 +- 对 x86 架构支持 AVX、AVX2、AVX512 和 AMX +- 对 RISC-V 架构支持 RVV、ZVFH、ZFH、ZICBOP 和 ZIHINTPAUSE +- 1.5-bit、2-bit、3-bit、4-bit、5-bit、6-bit 和 8-bit 整数量化，实现更快的推理和更少的内存使用 +- 自定义 CUDA 内核，用于在 NVIDIA GPU 上运行 LLM（通过 HIP 支持 AMD GPU，通过 MUSA 支持 Moore Threads GPU） +- Vulkan 和 SYCL 后端支持 +- CPU+GPU 混合推理，可部分加速超过总显存容量的大模型 -The `llama.cpp` project is the main playground for developing new features for the [ggml](https://github.com/ggml-org/ggml) library. +`llama.cpp` 项目是为 [ggml](https://github.com/ggml-org/ggml) 库开发新功能的主要试验场。
-Models +模型 -Typically finetunes of the base models below are supported as well. +以下基础模型的微调版本通常也同样支持。 -Instructions for adding support for new models: [HOWTO-add-model.md](docs/development/HOWTO-add-model.md) +添加新模型支持的说明：[HOWTO-add-model.md](docs/development/HOWTO-add-model.md) -#### Text-only +#### 纯文本 - [X] LLaMA 🦙 - [x] LLaMA 2 🦙🦙 @@ -90,22 +95,22 @@ Instructions for adding support for new models: [HOWTO-add-model.md](docs/develo - [x] [DBRX](https://huggingface.co/databricks/dbrx-instruct) - [x] [Jamba](https://huggingface.co/ai21labs) - [X] [Falcon](https://huggingface.co/models?search=tiiuae/falcon) -- [X] [Chinese LLaMA / Alpaca](https://github.com/ymcui/Chinese-LLaMA-Alpaca) and [Chinese LLaMA-2 / Alpaca-2](https://github.com/ymcui/Chinese-LLaMA-Alpaca-2) -- [X] [Vigogne (French)](https://github.com/bofenghuang/vigogne) +- [X] [Chinese LLaMA / Alpaca](https://github.com/ymcui/Chinese-LLaMA-Alpaca) 和 [Chinese LLaMA-2 / Alpaca-2](https://github.com/ymcui/Chinese-LLaMA-Alpaca-2) +- [X] [Vigogne（法语）](https://github.com/bofenghuang/vigogne) - [X] [BERT](https://github.com/ggml-org/llama.cpp/pull/5423) - [X] [Koala](https://bair.berkeley.edu/blog/2023/04/03/koala/) -- [X] [Baichuan 1 & 2](https://huggingface.co/models?search=baichuan-inc/Baichuan) + [derivations](https://huggingface.co/hiyouga/baichuan-7b-sft) +- [X] [Baichuan 1 & 2](https://huggingface.co/models?search=baichuan-inc/Baichuan) + [衍生模型](https://huggingface.co/hiyouga/baichuan-7b-sft) - [X] [Aquila 1 & 2](https://huggingface.co/models?search=BAAI/Aquila) -- [X] [Starcoder models](https://github.com/ggml-org/llama.cpp/pull/3187) +- [X] [Starcoder 模型](https://github.com/ggml-org/llama.cpp/pull/3187) - [X] [Refact](https://huggingface.co/smallcloudai/Refact-1_6B-fim) - [X] [MPT](https://github.com/ggml-org/llama.cpp/pull/3417) - [X] [Bloom](https://github.com/ggml-org/llama.cpp/pull/3553) -- [x] [Yi models](https://huggingface.co/models?search=01-ai/Yi) -- [X] [StableLM models](https://huggingface.co/stabilityai) -- [x] [Deepseek models](https://huggingface.co/models?search=deepseek-ai/deepseek) -- [x] [Qwen models](https://huggingface.co/models?search=Qwen/Qwen) +- [x] [Yi 模型](https://huggingface.co/models?search=01-ai/Yi) +- [X] [StableLM 模型](https://huggingface.co/stabilityai) +- [x] [Deepseek 模型](https://huggingface.co/models?search=deepseek-ai/deepseek) +- [x] [Qwen 模型](https://huggingface.co/models?search=Qwen/Qwen) - [x] [PLaMo-13B](https://github.com/ggml-org/llama.cpp/pull/3557) -- [x] [Phi models](https://huggingface.co/models?search=microsoft/phi) +- [x] [Phi 模型](https://huggingface.co/models?search=microsoft/phi) - [x] [PhiMoE](https://github.com/ggml-org/llama.cpp/pull/11003) - [x] [GPT-2](https://huggingface.co/gpt2) - [x] [Orion 14B](https://github.com/ggml-org/llama.cpp/pull/5118) @@ -115,25 +120,25 @@ Instructions for adding support for new models: [HOWTO-add-model.md](docs/develo - [x] [Mamba](https://github.com/state-spaces/mamba) - [x] [Grok-1](https://huggingface.co/keyfan/grok-1-hf) - [x] [Xverse](https://huggingface.co/models?search=xverse) -- [x] [Command-R models](https://huggingface.co/models?search=CohereForAI/c4ai-command-r) +- [x] [Command-R 模型](https://huggingface.co/models?search=CohereForAI/c4ai-command-r) - [x] [SEA-LION](https://huggingface.co/models?search=sea-lion) - [x] [GritLM-7B](https://huggingface.co/GritLM/GritLM-7B) + [GritLM-8x7B](https://huggingface.co/GritLM/GritLM-8x7B) - [x] [OLMo](https://allenai.org/olmo) - [x] [OLMo 2](https://allenai.org/olmo) - [x] [OLMoE](https://huggingface.co/allenai/OLMoE-1B-7B-0924) -- [x] [Granite models](https://huggingface.co/collections/ibm-granite/granite-code-models-6624c5cec322e4c148c8b330) +- [x] [Granite 模型](https://huggingface.co/collections/ibm-granite/granite-code-models-6624c5cec322e4c148c8b330) - [x] [GPT-NeoX](https://github.com/EleutherAI/gpt-neox) + [Pythia](https://github.com/EleutherAI/pythia) - [x] [Snowflake-Arctic MoE](https://huggingface.co/collections/Snowflake/arctic-66290090abe542894a5ac520) - [x] [Smaug](https://huggingface.co/models?search=Smaug) - [x] [Poro 34B](https://huggingface.co/LumiOpen/Poro-34B) -- [x] [Bitnet b1.58 models](https://huggingface.co/1bitLLM) +- [x] [Bitnet b1.58 模型](https://huggingface.co/1bitLLM) - [x] [Flan T5](https://huggingface.co/models?search=flan-t5) -- [x] [Open Elm models](https://huggingface.co/collections/apple/openelm-instruct-models-6619ad295d7ae9f868b759ca) +- [x] [Open Elm 模型](https://huggingface.co/collections/apple/openelm-instruct-models-6619ad295d7ae9f868b759ca) - [x] [ChatGLM3-6b](https://huggingface.co/THUDM/chatglm3-6b) + [ChatGLM4-9b](https://huggingface.co/THUDM/glm-4-9b) + [GLMEdge-1.5b](https://huggingface.co/THUDM/glm-edge-1.5b-chat) + [GLMEdge-4b](https://huggingface.co/THUDM/glm-edge-4b-chat) - [x] [GLM-4-0414](https://huggingface.co/collections/THUDM/glm-4-0414-67f3cbcb34dd9d252707cb2e) - [x] [SmolLM](https://huggingface.co/collections/HuggingFaceTB/smollm-6695016cad7167254ce15966) - [x] [EXAONE-3.0-7.8B-Instruct](https://huggingface.co/LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct) -- [x] [FalconMamba Models](https://huggingface.co/collections/tiiuae/falconmamba-7b-66b9a580324dd1598b0f6d4a) +- [x] [FalconMamba 模型](https://huggingface.co/collections/tiiuae/falconmamba-7b-66b9a580324dd1598b0f6d4a) - [x] [Jais](https://huggingface.co/inceptionai/jais-13b-chat) - [x] [Bielik-11B-v2.3](https://huggingface.co/collections/speakleash/bielik-11b-v23-66ee813238d9b526a072408a) - [x] [RWKV-7](https://huggingface.co/collections/shoumenchougou/rwkv7-gxx-gguf) @@ -141,21 +146,22 @@ Instructions for adding support for new models: [HOWTO-add-model.md](docs/develo - [x] [QRWKV-6](https://huggingface.co/recursal/QRWKV6-32B-Instruct-Preview-v0.1) - [x] [GigaChat-20B-A3B](https://huggingface.co/ai-sage/GigaChat-20B-A3B-instruct) - [X] [Trillion-7B-preview](https://huggingface.co/trillionlabs/Trillion-7B-preview) -- [x] [Ling models](https://huggingface.co/collections/inclusionAI/ling-67c51c85b34a7ea0aba94c32) -- [x] [Liquid LFM2 models](https://huggingface.co/collections/LiquidAI/lfm2) -- [x] [Liquid LFM2.5 models](https://huggingface.co/collections/LiquidAI/lfm25) +- [x] [Ling 模型](https://huggingface.co/collections/inclusionAI/ling-67c51c85b34a7ea0aba94c32) +- [x] [Liquid LFM2 模型](https://huggingface.co/collections/LiquidAI/lfm2) +- [x] [Liquid LFM2.5 模型](https://huggingface.co/collections/LiquidAI/lfm25) - [x] [Liquid Nanos](https://huggingface.co/collections/LiquidAI/liquid-nanos) -- [x] [Hunyuan models](https://huggingface.co/collections/tencent/hunyuan-dense-model-6890632cda26b19119c9c5e7) -- [x] [BailingMoeV2 (Ring/Ling 2.0) models](https://huggingface.co/collections/inclusionAI/ling-v2-68bf1dd2fc34c306c1fa6f86) -- [x] [Mellum models](https://huggingface.co/JetBrains/models?search=mellum) +- [x] [Hunyuan 模型](https://huggingface.co/collections/tencent/hunyuan-dense-model-6890632cda26b19119c9c5e7) +- [x] [BailingMoeV2（Ring/Ling 2.0）模型](https://huggingface.co/collections/inclusionAI/ling-v2-68bf1dd2fc34c306c1fa6f86) +- [x] [Mellum 模型](https://huggingface.co/JetBrains/models?search=mellum) -#### Multimodal +#### 多模态 -- [x] [LLaVA 1.5 models](https://huggingface.co/collections/liuhaotian/llava-15-653aac15d994e992e2677a7e), [LLaVA 1.6 models](https://huggingface.co/collections/liuhaotian/llava-16-65b9e40155f60fd046a5ccf2) +- [x] [LLaVA 1.5 模型](https://huggingface.co/collections/liuhaotian/llava-15-653aac15d994e992e2677a7e), +- [x] [LLaVA 1.6 模型](https://huggingface.co/collections/liuhaotian/llava-16-65b9e40155f60fd046a5ccf2) - [x] [BakLLaVA](https://huggingface.co/models?search=SkunkworksAI/Bakllava) - [x] [Obsidian](https://huggingface.co/NousResearch/Obsidian-3B-V0.5) - [x] [ShareGPT4V](https://huggingface.co/models?search=Lin-Chen/ShareGPT4V) -- [x] [MobileVLM 1.7B/3B models](https://huggingface.co/models?search=mobileVLM) +- [x] [MobileVLM 1.7B/3B 模型](https://huggingface.co/models?search=mobileVLM) - [x] [Yi-VL](https://huggingface.co/models?search=Yi-VL) - [x] [Mini CPM](https://huggingface.co/models?search=MiniCPM) - [x] [Moondream](https://huggingface.co/vikhyatk/moondream2) @@ -167,176 +173,176 @@ Instructions for adding support for new models: [HOWTO-add-model.md](docs/develo
-Bindings +绑定 -- Python: [ddh0/easy-llama](https://github.com/ddh0/easy-llama) -- Python: [abetlen/llama-cpp-python](https://github.com/abetlen/llama-cpp-python) -- Go: [go-skynet/go-llama.cpp](https://github.com/go-skynet/go-llama.cpp) -- Node.js: [withcatai/node-llama-cpp](https://github.com/withcatai/node-llama-cpp) -- JS/TS (llama.cpp server client): [lgrammel/modelfusion](https://modelfusion.dev/integration/model-provider/llamacpp) -- JS/TS (Programmable Prompt Engine CLI): [offline-ai/cli](https://github.com/offline-ai/cli) -- JavaScript/Wasm (works in browser): [tangledgroup/llama-cpp-wasm](https://github.com/tangledgroup/llama-cpp-wasm) -- Typescript/Wasm (nicer API, available on npm): [ngxson/wllama](https://github.com/ngxson/wllama) -- Ruby: [yoshoku/llama_cpp.rb](https://github.com/yoshoku/llama_cpp.rb) -- Ruby: [docusealco/rllama](https://github.com/docusealco/rllama) -- Rust (more features): [edgenai/llama_cpp-rs](https://github.com/edgenai/llama_cpp-rs) -- Rust (nicer API): [mdrokz/rust-llama.cpp](https://github.com/mdrokz/rust-llama.cpp) -- Rust (more direct bindings): [utilityai/llama-cpp-rs](https://github.com/utilityai/llama-cpp-rs) -- Rust (automated build from crates.io): [ShelbyJenkins/llm_client](https://github.com/ShelbyJenkins/llm_client) -- C#/.NET: [SciSharp/LLamaSharp](https://github.com/SciSharp/LLamaSharp) -- C#/VB.NET (more features - community license): [LM-Kit.NET](https://docs.lm-kit.com/lm-kit-net/index.html) -- Scala 3: [donderom/llm4s](https://github.com/donderom/llm4s) -- Clojure: [phronmophobic/llama.clj](https://github.com/phronmophobic/llama.clj) -- React Native: [mybigday/llama.rn](https://github.com/mybigday/llama.rn) -- Java: [kherud/java-llama.cpp](https://github.com/kherud/java-llama.cpp) -- Java: [QuasarByte/llama-cpp-jna](https://github.com/QuasarByte/llama-cpp-jna) -- Zig: [deins/llama.cpp.zig](https://github.com/Deins/llama.cpp.zig) -- Flutter/Dart: [netdur/llama_cpp_dart](https://github.com/netdur/llama_cpp_dart) -- Flutter: [xuegao-tzx/Fllama](https://github.com/xuegao-tzx/Fllama) -- PHP (API bindings and features built on top of llama.cpp): [distantmagic/resonance](https://github.com/distantmagic/resonance) [(more info)](https://github.com/ggml-org/llama.cpp/pull/6326) -- Guile Scheme: [guile_llama_cpp](https://savannah.nongnu.org/projects/guile-llama-cpp) -- Swift [srgtuszy/llama-cpp-swift](https://github.com/srgtuszy/llama-cpp-swift) -- Swift [ShenghaiWang/SwiftLlama](https://github.com/ShenghaiWang/SwiftLlama) -- Delphi [Embarcadero/llama-cpp-delphi](https://github.com/Embarcadero/llama-cpp-delphi) -- Go (no CGo needed): [hybridgroup/yzma](https://github.com/hybridgroup/yzma) -- Android: [llama.android](/examples/llama.android) +- Python：[ddh0/easy-llama](https://github.com/ddh0/easy-llama) +- Python：[abetlen/llama-cpp-python](https://github.com/abetlen/llama-cpp-python) +- Go：[go-skynet/go-llama.cpp](https://github.com/go-skynet/go-llama.cpp) +- Node.js：[withcatai/node-llama-cpp](https://github.com/withcatai/node-llama-cpp) +- JS/TS（llama.cpp 服务端客户端）：[lgrammel/modelfusion](https://modelfusion.dev/integration/model-provider/llamacpp) +- JS/TS（可编程提示引擎 CLI）：[offline-ai/cli](https://github.com/offline-ai/cli) +- JavaScript/Wasm（可在浏览器中运行）：[tangledgroup/llama-cpp-wasm](https://github.com/tangledgroup/llama-cpp-wasm) +- TypeScript/Wasm（更友好的 API，可在 npm 获取）：[ngxson/wllama](https://github.com/ngxson/wllama) +- Ruby：[yoshoku/llama_cpp.rb](https://github.com/yoshoku/llama_cpp.rb) +- Ruby：[docusealco/rllama](https://github.com/docusealco/rllama) +- Rust（功能更丰富）：[edgenai/llama_cpp-rs](https://github.com/edgenai/llama_cpp-rs) +- Rust（更友好的 API）：[mdrokz/rust-llama.cpp](https://github.com/mdrokz/rust-llama.cpp) +- Rust（更直接的绑定）：[utilityai/llama-cpp-rs](https://github.com/utilityai/llama-cpp-rs) +- Rust（通过 crates.io 自动构建）：[ShelbyJenkins/llm_client](https://github.com/ShelbyJenkins/llm_client) +- C#/.NET：[SciSharp/LLamaSharp](https://github.com/SciSharp/LLamaSharp) +- C#/VB.NET（功能更多——社区许可证）：[LM-Kit.NET](https://docs.lm-kit.com/lm-kit-net/index.html) +- Scala 3：[donderom/llm4s](https://github.com/donderom/llm4s) +- Clojure：[phronmophobic/llama.clj](https://github.com/phronmophobic/llama.clj) +- React Native：[mybigday/llama.rn](https://github.com/mybigday/llama.rn) +- Java：[kherud/java-llama.cpp](https://github.com/kherud/java-llama.cpp) +- Java：[QuasarByte/llama-cpp-jna](https://github.com/QuasarByte/llama-cpp-jna) +- Zig：[deins/llama.cpp.zig](https://github.com/Deins/llama.cpp.zig) +- Flutter/Dart：[netdur/llama_cpp_dart](https://github.com/netdur/llama_cpp_dart) +- Flutter：[xuegao-tzx/Fllama](https://github.com/xuegao-tzx/Fllama) +- PHP（基于 llama.cpp 构建的 API 绑定和功能）：[distantmagic/resonance](https://github.com/distantmagic/resonance) [（更多信息）](https://github.com/ggml-org/llama.cpp/pull/6326) +- Guile Scheme：[guile_llama_cpp](https://savannah.nongnu.org/projects/guile-llama-cpp) +- Swift：[srgtuszy/llama-cpp-swift](https://github.com/srgtuszy/llama-cpp-swift) +- Swift：[ShenghaiWang/SwiftLlama](https://github.com/ShenghaiWang/SwiftLlama) +- Delphi：[Embarcadero/llama-cpp-delphi](https://github.com/Embarcadero/llama-cpp-delphi) +- Go（无需 CGo）：[hybridgroup/yzma](https://github.com/hybridgroup/yzma) +- Android：[llama.android](/examples/llama.android)
-UIs +用户界面 -*(to have a project listed here, it should clearly state that it depends on `llama.cpp`)* +*（要让项目列在此处，应明确声明其依赖 `llama.cpp`）* -- [AI Sublime Text plugin](https://github.com/yaroslavyaroslav/OpenAI-sublime-text) (MIT) -- [BonzAI App](https://apps.apple.com/us/app/bonzai-your-local-ai-agent/id6752847988) (proprietary) -- [cztomsik/ava](https://github.com/cztomsik/ava) (MIT) -- [Dot](https://github.com/alexpinel/Dot) (GPL) -- [eva](https://github.com/ylsdamxssjxxdd/eva) (MIT) -- [iohub/collama](https://github.com/iohub/coLLaMA) (Apache-2.0) -- [janhq/jan](https://github.com/janhq/jan) (AGPL) -- [johnbean393/Sidekick](https://github.com/johnbean393/Sidekick) (MIT) -- [KanTV](https://github.com/zhouwg/kantv?tab=readme-ov-file) (Apache-2.0) -- [KodiBot](https://github.com/firatkiral/kodibot) (GPL) -- [llama.vim](https://github.com/ggml-org/llama.vim) (MIT) -- [LARS](https://github.com/abgulati/LARS) (AGPL) -- [Llama Assistant](https://github.com/vietanhdev/llama-assistant) (GPL) -- [LlamaLib](https://github.com/undreamai/LlamaLib) (Apache-2.0) -- [LLMFarm](https://github.com/guinmoon/LLMFarm?tab=readme-ov-file) (MIT) -- [LLMUnity](https://github.com/undreamai/LLMUnity) (MIT) -- [LMStudio](https://lmstudio.ai/) (proprietary) -- [LocalAI](https://github.com/mudler/LocalAI) (MIT) -- [LostRuins/koboldcpp](https://github.com/LostRuins/koboldcpp) (AGPL) -- [MindMac](https://mindmac.app) (proprietary) -- [MindWorkAI/AI-Studio](https://github.com/MindWorkAI/AI-Studio) (FSL-1.1-MIT) -- [Mobile-Artificial-Intelligence/maid](https://github.com/Mobile-Artificial-Intelligence/maid) (MIT) -- [Mozilla-Ocho/llamafile](https://github.com/Mozilla-Ocho/llamafile) (Apache-2.0) -- [nat/openplayground](https://github.com/nat/openplayground) (MIT) -- [nomic-ai/gpt4all](https://github.com/nomic-ai/gpt4all) (MIT) -- [ollama/ollama](https://github.com/ollama/ollama) (MIT) -- [oobabooga/text-generation-webui](https://github.com/oobabooga/text-generation-webui) (AGPL) -- [PocketPal AI](https://github.com/a-ghorbani/pocketpal-ai) (MIT) -- [psugihara/FreeChat](https://github.com/psugihara/FreeChat) (MIT) -- [ptsochantaris/emeltal](https://github.com/ptsochantaris/emeltal) (MIT) -- [pythops/tenere](https://github.com/pythops/tenere) (AGPL) -- [ramalama](https://github.com/containers/ramalama) (MIT) -- [semperai/amica](https://github.com/semperai/amica) (MIT) -- [withcatai/catai](https://github.com/withcatai/catai) (MIT) -- [Autopen](https://github.com/blackhole89/autopen) (GPL) +- [AI Sublime Text 插件](https://github.com/yaroslavyaroslav/OpenAI-sublime-text)（MIT） +- [BonzAI App](https://apps.apple.com/us/app/bonzai-your-local-ai-agent/id6752847988)（专有） +- [cztomsik/ava](https://github.com/cztomsik/ava)（MIT） +- [Dot](https://github.com/alexpinel/Dot)（GPL） +- [eva](https://github.com/ylsdamxssjxxdd/eva)（MIT） +- [iohub/collama](https://github.com/iohub/coLLaMA)（Apache-2.0） +- [janhq/jan](https://github.com/janhq/jan)（AGPL） +- [johnbean393/Sidekick](https://github.com/johnbean393/Sidekick)（MIT） +- [KanTV](https://github.com/zhouwg/kantv?tab=readme-ov-file)（Apache-2.0） +- [KodiBot](https://github.com/firatkiral/kodibot)（GPL） +- [llama.vim](https://github.com/ggml-org/llama.vim)（MIT） +- [LARS](https://github.com/abgulati/LARS)（AGPL） +- [Llama Assistant](https://github.com/vietanhdev/llama-assistant)（GPL） +- [LlamaLib](https://github.com/undreamai/LlamaLib)（Apache-2.0） +- [LLMFarm](https://github.com/guinmoon/LLMFarm?tab=readme-ov-file)（MIT） +- [LLMUnity](https://github.com/undreamai/LLMUnity)（MIT） +- [LMStudio](https://lmstudio.ai/)（专有） +- [LocalAI](https://github.com/mudler/LocalAI)（MIT） +- [LostRuins/koboldcpp](https://github.com/LostRuins/koboldcpp)（AGPL） +- [MindMac](https://mindmac.app)（专有） +- [MindWorkAI/AI-Studio](https://github.com/MindWorkAI/AI-Studio)（FSL-1.1-MIT） +- [Mobile-Artificial-Intelligence/maid](https://github.com/Mobile-Artificial-Intelligence/maid)（MIT） +- [Mozilla-Ocho/llamafile](https://github.com/Mozilla-Ocho/llamafile)（Apache-2.0） +- [nat/openplayground](https://github.com/nat/openplayground)（MIT） +- [nomic-ai/gpt4all](https://github.com/nomic-ai/gpt4all)（MIT） +- [ollama/ollama](https://github.com/ollama/ollama)（MIT） +- [oobabooga/text-generation-webui](https://github.com/oobabooga/text-generation-webui)（AGPL） +- [PocketPal AI](https://github.com/a-ghorbani/pocketpal-ai)（MIT） +- [psugihara/FreeChat](https://github.com/psugihara/FreeChat)（MIT） +- [ptsochantaris/emeltal](https://github.com/ptsochantaris/emeltal)（MIT） +- [pythops/tenere](https://github.com/pythops/tenere)（AGPL） +- [ramalama](https://github.com/containers/ramalama)（MIT） +- [semperai/amica](https://github.com/semperai/amica)（MIT） +- [withcatai/catai](https://github.com/withcatai/catai)（MIT） +- [Autopen](https://github.com/blackhole89/autopen)（GPL）
-Tools +工具 -- [akx/ggify](https://github.com/akx/ggify) – download PyTorch models from Hugging Face Hub and convert them to GGML -- [akx/ollama-dl](https://github.com/akx/ollama-dl) – download models from the Ollama library to be used directly with llama.cpp -- [crashr/gppm](https://github.com/crashr/gppm) – launch llama.cpp instances utilizing NVIDIA Tesla P40 or P100 GPUs with reduced idle power consumption -- [gpustack/gguf-parser](https://github.com/gpustack/gguf-parser-go/tree/main/cmd/gguf-parser) - review/check the GGUF file and estimate the memory usage -- [Styled Lines](https://marketplace.unity.com/packages/tools/generative-ai/styled-lines-llama-cpp-model-292902) (proprietary licensed, async wrapper of inference part for game development in Unity3d with pre-built Mobile and Web platform wrappers and a model example) -- [unslothai/unsloth](https://github.com/unslothai/unsloth) – 🦥 exports/saves fine-tuned and trained models to GGUF (Apache-2.0) +- [akx/ggify](https://github.com/akx/ggify) —— 从 Hugging Face Hub 下载 PyTorch 模型并转换为 GGML 格式 +- [akx/ollama-dl](https://github.com/akx/ollama-dl) —— 从 Ollama 库下载模型，以便直接与 llama.cpp 配合使用 +- [crashr/gppm](https://github.com/crashr/gppm) —— 利用 NVIDIA Tesla P40 或 P100 GPU 启动 llama.cpp 实例，降低空闲功耗 +- [gpustack/gguf-parser](https://github.com/gpustack/gguf-parser-go/tree/main/cmd/gguf-parser) —— 审查/检查 GGUF 文件并估算内存用量 +- [Styled Lines](https://marketplace.unity.com/packages/tools/generative-ai/styled-lines-llama-cpp-model-292902)（专有许可，推理部分的异步封装，用于 Unity3d 游戏开发，附带预构建的移动端和 Web 平台封装器及示例模型） +- [unslothai/unsloth](https://github.com/unslothai/unsloth) —— 🦥 导出/保存微调及训练完成的模型为 GGUF 格式（Apache-2.0）
-Infrastructure +基础设施 + +- [Paddler](https://github.com/intentee/paddler) —— 开源 LLMOps 平台，用于在你自己的基础设施中托管和扩展 AI +- [GPUStack](https://github.com/gpustack/gpustack) —— 管理用于运行 LLM 的 GPU 集群 +- [llama_cpp_canister](https://github.com/onicai/llama_cpp_canister) —— 将 llama.cpp 作为互联网计算机上的智能合约运行（基于 WebAssembly） +- [llama-swap](https://github.com/mostlygeek/llama-swap) —— 透明代理，配合 llama-server 实现自动模型切换 +- [Kalavai](https://github.com/kalavai-net/kalavai-client) —— 众包端到端 LLM 部署，支持任意规模 +- [llmaz](https://github.com/InftyAI/llmaz) —— ☸️ 在 Kubernetes 上运行大语言模型的简便、高级推理平台 +- [LLMKube](https://github.com/defilantech/llmkube) —— 用于 llama.cpp 的 Kubernetes 算子，支持多 GPU 和 Apple Silicon Metal 加速 +USD 预算：$0/$3；剩余 $3；本次会话 USD 支出：$0 -- [Paddler](https://github.com/intentee/paddler) - Open-source LLMOps platform for hosting and scaling AI in your own infrastructure -- [GPUStack](https://github.com/gpustack/gpustack) - Manage GPU clusters for running LLMs -- [llama_cpp_canister](https://github.com/onicai/llama_cpp_canister) - llama.cpp as a smart contract on the Internet Computer, using WebAssembly -- [llama-swap](https://github.com/mostlygeek/llama-swap) - transparent proxy that adds automatic model switching with llama-server -- [Kalavai](https://github.com/kalavai-net/kalavai-client) - Crowdsource end to end LLM deployment at any scale -- [llmaz](https://github.com/InftyAI/llmaz) - ☸️ Easy, advanced inference platform for large language models on Kubernetes. -- [LLMKube](https://github.com/defilantech/llmkube) - Kubernetes operator for llama.cpp with multi-GPU and Apple Silicon Metal - support"
-Games +游戏 -- [Lucy's Labyrinth](https://github.com/MorganRO8/Lucys_Labyrinth) - A simple maze game where agents controlled by an AI model will try to trick you. +- [露西的迷宫](https://github.com/MorganRO8/Lucys_Labyrinth)) —— 一个由 AI 模型控制的智能体试图迷惑你的简单迷宫游戏。
+## 支持的后端 -## Supported backends - -| Backend | Target devices | +| 后端 | 目标设备 | | --- | --- | | [Metal](docs/build.md#metal-build) | Apple Silicon | -| [BLAS](docs/build.md#blas-build) | All | -| [BLIS](docs/backend/BLIS.md) | All | +| [BLAS](docs/build.md#blas-build) | 全部 | +| [BLIS](docs/backend/BLIS.md) | 全部 | | [SYCL](docs/backend/SYCL.md) | Intel GPU | -| [OpenVINO [In Progress]](docs/backend/OPENVINO.md) | Intel CPUs, GPUs, and NPUs | -| [MUSA](docs/build.md#musa) | Moore Threads GPU | +| [OpenVINO [进行中]](docs/backend/OPENVINO.md) | Intel CPU、GPU 和 NPU | +| [MUSA](docs/build.md#musa) | 摩尔线程 GPU | | [CUDA](docs/build.md#cuda) | Nvidia GPU | | [HIP](docs/build.md#hip) | AMD GPU | | [ZenDNN](docs/build.md#zendnn) | AMD CPU | | [Vulkan](docs/build.md#vulkan) | GPU | | [CANN](docs/build.md#cann) | Ascend NPU | | [OpenCL](docs/backend/OPENCL.md) | Adreno GPU | -| [IBM zDNN](docs/backend/zDNN.md) | IBM Z & LinuxONE | -| [WebGPU](docs/build.md#webgpu) | All | -| [RPC](https://github.com/ggml-org/llama.cpp/tree/master/tools/rpc) | All | -| [Hexagon [In Progress]](docs/backend/snapdragon/README.md) | Snapdragon | +| [IBM zDNN](docs/backend/zDNN.md) | IBM Z 及 LinuxONE | +| [WebGPU](docs/build.md#webgpu) | 全部 | +| [RPC](https://github.com/ggml-org/llama.cpp/tree/master/tools/rpc)) | 全部 | +| [Hexagon [进行中]](docs/backend/snapdragon/README.md) | Snapdragon | | [VirtGPU](docs/backend/VirtGPU.md) | VirtGPU APIR | -## Obtaining and quantizing models +## 获取与量化模型 -The [Hugging Face](https://huggingface.co) platform hosts a [number of LLMs](https://huggingface.co/models?library=gguf&sort=trending) compatible with `llama.cpp`: +[Hugging Face](https://huggingface.co)) 平台托管了一批与 `llama.cpp` 兼容的 [LLM](https://huggingface.co/models?library=gguf&sort=trending))： -- [Trending](https://huggingface.co/models?library=gguf&sort=trending) -- [LLaMA](https://huggingface.co/models?sort=trending&search=llama+gguf) +- [热门模型](https://huggingface.co/models?library=gguf&sort=trending)) +- [LLaMA](https://huggingface.co/models?sort=trending&search=llama+gguf)) -You can either manually download the GGUF file or directly use any `llama.cpp`-compatible models from [Hugging Face](https://huggingface.co/) or other model hosting sites, by using this CLI argument: `-hf /[:quant]`. For example: +你可以手动下载 GGUF 文件，也可以直接通过 CLI 参数 `-hf /[:quant]` 使用来自 [Hugging Face](https://huggingface.co/)) 或其他模型托管站点的任何 `llama.cpp` 兼容模型。例如： ```sh llama-cli -hf ggml-org/gemma-3-1b-it-GGUF ``` -By default, the CLI would download from Hugging Face, you can switch to other options with the environment variable `MODEL_ENDPOINT`. The `MODEL_ENDPOINT` must point to a Hugging Face compatible API endpoint. +默认情况下，CLI 会从 Hugging Face 下载；你可以通过环境变量 `MODEL_ENDPOINT` 切换到其他选项。`MODEL_ENDPOINT` 必须指向与 Hugging Face 兼容的 API 端点。 -After downloading a model, use the CLI tools to run it locally - see below. +下载模型后，使用 CLI 工具在本地运行它——详见下文。 -`llama.cpp` requires the model to be stored in the [GGUF](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md) file format. Models in other data formats can be converted to GGUF using the `convert_*.py` Python scripts in this repo. +`llama.cpp` 要求模型以 [GGUF](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md)) 文件格式存储。其他数据格式的模型可以使用本仓库中的 `convert_*.py` Python 脚本转换为 GGUF 格式。 -The Hugging Face platform provides a variety of online tools for converting, quantizing and hosting models with `llama.cpp`: +Hugging Face 平台提供了一系列在线工具，用于转换、量化以及托管基于 `llama.cpp` 的模型： -- Use the [GGUF-my-repo space](https://huggingface.co/spaces/ggml-org/gguf-my-repo) to convert to GGUF format and quantize model weights to smaller sizes -- Use the [GGUF-my-LoRA space](https://huggingface.co/spaces/ggml-org/gguf-my-lora) to convert LoRA adapters to GGUF format (more info: https://github.com/ggml-org/llama.cpp/discussions/10123) -- Use the [GGUF-editor space](https://huggingface.co/spaces/CISCai/gguf-editor) to edit GGUF meta data in the browser (more info: https://github.com/ggml-org/llama.cpp/discussions/9268) -- Use the [Inference Endpoints](https://ui.endpoints.huggingface.co/) to directly host `llama.cpp` in the cloud (more info: https://github.com/ggml-org/llama.cpp/discussions/9669) +- 使用 [GGUF-my-repo 空间](https://huggingface.co/spaces/ggml-org/gguf-my-repo)) 转换为 GGUF 格式并将模型权重量化为更小尺寸 +- 使用 [GGUF-my-LoRA 空间](https://huggingface.co/spaces/ggml-org/gguf-my-lora)) 将 LoRA 适配器转换为 GGUF 格式（更多信息：https://github.com/ggml-org/llama.cpp/discussions/10123)） +- 使用 [GGUF-editor 空间](https://huggingface.co/spaces/CISCai/gguf-editor)) 在浏览器中编辑 GGUF 元数据（更多信息：https://github.com/ggml-org/llama.cpp/discussions/9268)） +- 使用 [Inference Endpoints](https://ui.endpoints.huggingface.co/)) 在云端直接托管 `llama.cpp`（更多信息：https://github.com/ggml-org/llama.cpp/discussions/9669)） -To learn more about model quantization, [read this documentation](tools/quantize/README.md) +要了解更多关于模型量化的信息，[请阅读此文档](tools/quantize/README.md) ## [`llama-cli`](tools/cli) -#### A CLI tool for accessing and experimenting with most of `llama.cpp`'s functionality. +#### 用于访问和实验 `llama.cpp` 大部分功能的 CLI 工具。 -
- Run in conversation mode + 以对话模式运行 - Models with a built-in chat template will automatically activate conversation mode. If this doesn't occur, you can manually enable it by adding `-cnv` and specifying a suitable chat template with `--chat-template NAME` + 带有内置对话模板的模型会自动激活对话模式。如果未自动激活，可以通过添加 `-cnv` 并使用 `--chat-template NAME` 指定合适的对话模板来手动启用。 ```bash llama-cli -m model.gguf @@ -351,20 +357,20 @@ To learn more about model quantization, [read this documentation](tools/quantize
-
- Run in conversation mode with custom chat template + 使用自定义对话模板以对话模式运行 ```bash - # use the "chatml" template (use -h to see the list of supported templates) + # 使用 "chatml" 模板（使用 -h 查看支持的模板列表） llama-cli -m model.gguf -cnv --chat-template chatml - # use a custom template + # 使用自定义模板 llama-cli -m model.gguf -cnv --in-prefix 'User: ' --reverse-prompt 'User:' ```
-
- Constrain the output with a custom grammar + 使用自定义语法约束输出 ```bash llama-cli -m model.gguf -n 256 --grammar-file grammars/json.gbnf -p 'Request: schedule a call at 8pm; Command:' @@ -372,74 +378,73 @@ To learn more about model quantization, [read this documentation](tools/quantize # {"appointmentTime": "8pm", "appointmentDetails": "schedule a a call"} ``` - The [grammars/](grammars/) folder contains a handful of sample grammars. To write your own, check out the [GBNF Guide](grammars/README.md). + [grammars/](grammars/) 文件夹包含若干示例语法文件。如需编写自己的语法，请查阅 [GBNF 指南](grammars/README.md)。 - For authoring more complex JSON grammars, check out https://grammar.intrinsiclabs.ai/ + 如需编写更复杂的 JSON 语法，请参阅 https://grammar.intrinsiclabs.ai/
- ## [`llama-server`](tools/server) -#### A lightweight, [OpenAI API](https://github.com/openai/openai-openapi) compatible, HTTP server for serving LLMs. +#### 一个轻量级、[OpenAI API](https://github.com/openai/openai-openapi)) 兼容的 HTTP 服务器，用于托管 LLM。 -
- Start a local HTTP server with default configuration on port 8080 + 在端口 8080 上以默认配置启动本地 HTTP 服务器 ```bash llama-server -m model.gguf --port 8080 - # Basic web UI can be accessed via browser: http://localhost:8080 - # Chat completion endpoint: http://localhost:8080/v1/chat/completions + # 基本 Web UI 可通过浏览器访问：http://localhost:8080 + # 聊天补全端点：http://localhost:8080/v1/chat/completions ```
-
- Support multiple-users and parallel decoding + 支持多用户和并行解码 ```bash - # up to 4 concurrent requests, each with 4096 max context + # 最多 4 个并发请求，每个最大上下文为 4096 llama-server -m model.gguf -c 16384 -np 4 ```
-
- Enable speculative decoding + 启用推测解码 ```bash - # the draft.gguf model should be a small variant of the target model.gguf + # draft.gguf 模型应为目标 model.gguf 的小型变体 llama-server -m model.gguf -md draft.gguf ```
-
- Serve an embedding model + 托管嵌入模型 ```bash - # use the /embedding endpoint + # 使用 /embedding 端点 llama-server -m model.gguf --embedding --pooling cls -ub 8192 ```
-
- Serve a reranking model + 托管重排序模型 ```bash - # use the /reranking endpoint + # 使用 /reranking 端点 llama-server -m model.gguf --reranking ```
-
- Constrain all outputs with a grammar + 使用语法约束所有输出 ```bash - # custom grammar + # 自定义语法 llama-server -m model.gguf --grammar-file grammar.gbnf # JSON @@ -448,13 +453,12 @@ To learn more about model quantization, [read this documentation](tools/quantize
- ## [`llama-perplexity`](tools/perplexity) -#### A tool for measuring the [perplexity](tools/perplexity/README.md) [^1] (and other quality metrics) of a model over a given text. +#### 用于衡量模型在给定文本上的[困惑度](tools/perplexity/README.md)[^1]（及其他质量指标）的工具。 -
- Measure the perplexity over a text file + 衡量文本文件的困惑度 ```bash llama-perplexity -m model.gguf -f file.txt @@ -466,7 +470,7 @@ To learn more about model quantization, [read this documentation](tools/quantize
-
- Measure KL divergence + 衡量 KL 散度 ```bash # TODO @@ -478,15 +482,15 @@ To learn more about model quantization, [read this documentation](tools/quantize ## [`llama-bench`](tools/llama-bench) -#### Benchmark the performance of the inference for various parameters. +#### 基准测试推理在不同参数下的性能。 -
- Run default benchmark + 运行默认基准测试 ```bash llama-bench -m model.gguf - # Output: + # 输出： # | model | size | params | backend | threads | test | t/s | # | ------------------- | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: | # | qwen2 1.5B Q4_0 | 885.97 MiB | 1.54 B | Metal,BLAS | 16 | pp512 | 5765.41 ± 20.55 | @@ -497,12 +501,14 @@ To learn more about model quantization, [read this documentation](tools/quantize
+
+ ## [`llama-simple`](examples/simple) -#### A minimal example for implementing apps with `llama.cpp`. Useful for developers. +#### 使用 `llama.cpp` 实现应用的最小示例，对开发者非常有用。 -
- Basic text completion + 基础文本补全 ```bash llama-simple -m model.gguf @@ -512,50 +518,48 @@ To learn more about model quantization, [read this documentation](tools/quantize
+## 贡献指南 -## Contributing +- 贡献者可以提交 PR +- 根据贡献情况邀请协作者 +- 维护者可以向 `llama.cpp` 仓库推送分支并将 PR 合入 `master` 分支 +- 欢迎任何帮助管理 Issue、PR 和项目的工作！ +- 参见 [good first issues](https://github.com/ggml-org/llama.cpp/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) 寻找适合初次贡献的任务 +- 阅读 [CONTRIBUTING.md](CONTRIBUTING.md) 获取更多信息 +- 请务必阅读：[Inference at the edge](https://github.com/ggml-org/llama.cpp/discussions/205) +- 感兴趣的读者可了解一点背景故事：[Changelog podcast](https://changelog.com/podcast/532) -- Contributors can open PRs -- Collaborators will be invited based on contributions -- Maintainers can push to branches in the `llama.cpp` repo and merge PRs into the `master` branch -- Any help with managing issues, PRs and projects is very appreciated! -- See [good first issues](https://github.com/ggml-org/llama.cpp/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) for tasks suitable for first contributions -- Read the [CONTRIBUTING.md](CONTRIBUTING.md) for more information -- Make sure to read this: [Inference at the edge](https://github.com/ggml-org/llama.cpp/discussions/205) -- A bit of backstory for those who are interested: [Changelog podcast](https://changelog.com/podcast/532) - -## Other documentation +## 其他文档 - [cli](tools/cli/README.md) - [completion](tools/completion/README.md) - [server](tools/server/README.md) -- [GBNF grammars](grammars/README.md) +- [GBNF 语法](grammars/README.md) -#### Development documentation +#### 开发文档 -- [How to build](docs/build.md) -- [Running on Docker](docs/docker.md) -- [Build on Android](docs/android.md) -- [Multi-GPU usage](docs/multi-gpu.md) -- [Performance troubleshooting](docs/development/token_generation_performance_tips.md) -- [GGML tips & tricks](https://github.com/ggml-org/llama.cpp/wiki/GGML-Tips-&-Tricks) +- [如何构建](docs/build.md) +- [在 Docker 上运行](docs/docker.md) +- [在 Android 上构建](docs/android.md) +- [多 GPU 使用](docs/multi-gpu.md) +- [性能故障排查](docs/development/token_generation_performance_tips.md) +- [GGML 技巧与建议](https://github.com/ggml-org/llama.cpp/wiki/GGML-Tips-&-Tricks) -#### Seminal papers and background on the models +#### 重要论文及模型背景 -If your issue is with model generation quality, then please at least scan the following links and papers to understand the limitations of LLaMA models. This is especially important when choosing an appropriate model size and appreciating both the significant and subtle differences between LLaMA models and ChatGPT: -- LLaMA: +如果你的问题是关于模型生成质量的，请至少阅读以下链接和论文，了解 LLaMA 模型的局限性。这对于选择合适的模型大小以及理解 LLaMA 模型与 ChatGPT 之间显著和微妙的差异尤为重要： +- LLaMA： - [Introducing LLaMA: A foundational, 65-billion-parameter large language model](https://ai.facebook.com/blog/large-language-model-llama-meta-ai/) - [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971) - GPT-3 - [Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165) -- GPT-3.5 / InstructGPT / ChatGPT: +- GPT-3.5 / InstructGPT / ChatGPT： - [Aligning language models to follow instructions](https://openai.com/research/instruction-following) - [Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155) ## XCFramework -The XCFramework is a precompiled version of the library for iOS, visionOS, tvOS, -and macOS. It can be used in Swift projects without the need to compile the -library from source. For example: + +XCFramework 是该库的预编译版本，适用于 iOS、visionOS、tvOS 和 macOS。可以在 Swift 项目中直接使用，无需从源码编译该库。例如： ```swift // swift-tools-version: 5.10 // The swift-tools-version declares the minimum version of Swift required to build this package. @@ -578,27 +582,26 @@ let package = Package( ] ) ``` -The above example is using an intermediate build `b5046` of the library. This can be modified -to use a different version by changing the URL and checksum. +以上示例使用了该库的中间构建版本 `b5046`。可通过修改 URL 和校验和来使用不同版本。 -## Completions -Command-line completion is available for some environments. +## 命令补全 -#### Bash Completion +部分环境支持命令行补全。 + +#### Bash 补全 ```bash $ build/bin/llama-cli --completion-bash > ~/.llama-completion.bash $ source ~/.llama-completion.bash ``` -Optionally this can be added to your `.bashrc` or `.bash_profile` to load it -automatically. For example: +可以选择将其添加到 `.bashrc` 或 `.bash_profile` 中，以便自动加载。例如： ```console $ echo "source ~/.llama-completion.bash" >> ~/.bashrc ``` -## Dependencies +## 依赖项 -- [yhirose/cpp-httplib](https://github.com/yhirose/cpp-httplib) - Single-header HTTP server, used by `llama-server` - MIT license -- [stb-image](https://github.com/nothings/stb) - Single-header image format decoder, used by multimodal subsystem - Public domain -- [nlohmann/json](https://github.com/nlohmann/json) - Single-header JSON library, used by various tools/examples - MIT License -- [miniaudio.h](https://github.com/mackron/miniaudio) - Single-header audio format decoder, used by multimodal subsystem - Public domain -- [subprocess.h](https://github.com/sheredom/subprocess.h) - Single-header process launching solution for C and C++ - Public domain +- [yhirose/cpp-httplib](https://github.com/yhirose/cpp-httplib) — 单头文件的 HTTP 服务器，由 `llama-server` 使用 — MIT 许可证 +- [stb-image](https://github.com/nothings/stb) — 单头文件的图片格式解码器，由多模态子系统使用 — 公有领域 +- [nlohmann/json](https://github.com/nlohmann/json) — 单头文件的 JSON 库，被多个工具/示例使用 — MIT 许可证 +- [miniaudio.h](https://github.com/mackron/miniaudio) — 单头文件的音频格式解码器，由多模态子系统使用 — 公有领域 +- [subprocess.h](https://github.com/sheredom/subprocess.h) — C 和 C++ 的单头文件进程启动解决方案 — 公有领域