This commit is contained in:
@@ -1,63 +1,68 @@
|
||||
<!-- WEHUB_ZH_README -->
|
||||
> [!NOTE]
|
||||
> 本文档由 WeHub 基于上游 README 翻译整理,属于社区翻译,非官方中文文档。
|
||||
> [English](./README.en.md) · [原始项目](https://github.com/deepseek-ai/3FS) · [上游 README](https://github.com/deepseek-ai/3FS/blob/HEAD/README.md)
|
||||
> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。
|
||||
|
||||
# Fire-Flyer File System
|
||||
|
||||
[](https://github.com/deepseek-ai/3fs/actions/workflows/build.yml)
|
||||
[](LICENSE)
|
||||
|
||||
The Fire-Flyer File System (3FS) is a high-performance distributed file system designed to address the challenges of AI training and inference workloads. It leverages modern SSDs and RDMA networks to provide a shared storage layer that simplifies development of distributed applications. Key features and benefits of 3FS include:
|
||||
Fire-Flyer File System(3FS)是一种高性能分布式文件系统,旨在应对 AI 训练与推理工作负载的挑战。它利用现代 SSD 和 RDMA 网络提供共享存储层,从而简化分布式应用的开发。3FS 的主要特性与优势包括:
|
||||
|
||||
- Performance and Usability
|
||||
- **Disaggregated Architecture** Combines the throughput of thousands of SSDs and the network bandwidth of hundreds of storage nodes, enabling applications to access storage resource in a locality-oblivious manner.
|
||||
- **Strong Consistency** Implements Chain Replication with Apportioned Queries (CRAQ) for strong consistency, making application code simple and easy to reason about.
|
||||
- **File Interfaces** Develops stateless metadata services backed by a transactional key-value store (e.g., FoundationDB). The file interface is well known and used everywhere. There is no need to learn a new storage API.
|
||||
- 性能与易用性
|
||||
- **解耦架构(Disaggregated Architecture)** 汇聚数千块 SSD 的吞吐量与数百个存储节点的网络带宽,使应用能够以无需感知数据局部性的方式访问存储资源。
|
||||
- **强一致性(Strong Consistency)** 采用带分摊查询的链式复制(Chain Replication with Apportioned Queries,CRAQ)实现强一致性,使应用代码简单且易于推理。
|
||||
- **文件接口(File Interfaces)** 构建由事务型键值存储(例如 FoundationDB)支撑的无状态元数据服务。文件接口广为人知、应用广泛,无需学习新的存储 API。
|
||||
|
||||
- Diverse Workloads
|
||||
- **Data Preparation** Organizes outputs of data analytics pipelines into hierarchical directory structures and manages a large volume of intermediate outputs efficiently.
|
||||
- **Dataloaders** Eliminates the need for prefetching or shuffling datasets by enabling random access to training samples across compute nodes.
|
||||
- **Checkpointing** Supports high-throughput parallel checkpointing for large-scale training.
|
||||
- **KVCache for Inference** Provides a cost-effective alternative to DRAM-based caching, offering high throughput and significantly larger capacity.
|
||||
- 多样化工作负载
|
||||
- **数据准备(Data Preparation)** 将数据分析流水线的输出组织为分层目录结构,并高效管理大量中间输出。
|
||||
- **数据加载器(Dataloaders)** 支持跨计算节点对训练样本进行随机访问,从而无需预取或打乱数据集。
|
||||
- **检查点(Checkpointing)** 支持大规模训练的高吞吐量并行检查点。
|
||||
- **推理用 KVCache(KVCache for Inference)** 提供相比基于 DRAM 缓存更具成本效益的替代方案,具备高吞吐量且容量显著更大。
|
||||
|
||||
## Documentation
|
||||
## 文档
|
||||
|
||||
* [Design Notes](docs/design_notes.md)
|
||||
* [Setup Guide](deploy/README.md)
|
||||
* [USRBIO API Reference](src/lib/api/UsrbIo.md)
|
||||
* [P Specifications](./specs/README.md)
|
||||
* [设计说明](docs/design_notes.md)
|
||||
* [部署指南](deploy/README.md)
|
||||
* [USRBIO API 参考](src/lib/api/UsrbIo.md)
|
||||
* [P 规范](./specs/README.md)
|
||||
|
||||
## Performance
|
||||
## 性能
|
||||
|
||||
### 1. Peak throughput
|
||||
### 1. 峰值吞吐量
|
||||
|
||||
The following figure demonstrates the throughput of read stress test on a large 3FS cluster. This cluster consists of 180 storage nodes, each equipped with 2×200Gbps InfiniBand NICs and sixteen 14TiB NVMe SSDs. Approximately 500+ client nodes were used for the read stress test, with each client node configured with 1x200Gbps InfiniBand NIC. The final aggregate read throughput reached approximately 6.6 TiB/s with background traffic from training jobs.
|
||||
下图展示了大型 3FS 集群上读压力测试的吞吐量。该集群由 180 个存储节点组成,每个节点配备 2×200Gbps InfiniBand 网卡和十六块 14TiB NVMe SSD。读压力测试使用了约 500+ 个客户端节点,每个客户端节点配置 1×200Gbps InfiniBand 网卡。在训练作业后台流量的影响下,最终聚合读吞吐量达到约 6.6 TiB/s。
|
||||
|
||||

|
||||
|
||||
To benchmark 3FS, please use our [fio engine for USRBIO](benchmarks/fio_usrbio/README.md).
|
||||
要对 3FS 进行基准测试,请使用我们的 [fio engine for USRBIO](benchmarks/fio_usrbio/README.md)。
|
||||
|
||||
### 2. GraySort
|
||||
|
||||
We evaluated [smallpond](https://github.com/deepseek-ai/smallpond) using the GraySort benchmark, which measures sort performance on large-scale datasets. Our implementation adopts a two-phase approach: (1) partitioning data via shuffle using the prefix bits of keys, and (2) in-partition sorting. Both phases read/write data from/to 3FS.
|
||||
我们使用 [smallpond](https://github.com/deepseek-ai/smallpond) 评估 GraySort 基准测试,该测试用于衡量大规模数据集上的排序性能。我们的实现采用两阶段方法:(1)通过按键前缀位进行 shuffle 来分区数据;(2)分区内排序。两个阶段均从 3FS 读取/写入数据。
|
||||
|
||||
The test cluster comprised 25 storage nodes (2 NUMA domains/node, 1 storage service/NUMA, 2×400Gbps NICs/node) and 50 compute nodes (2 NUMA domains, 192 physical cores, 2.2 TiB RAM, and 1×200 Gbps NIC/node). Sorting 110.5 TiB of data across 8,192 partitions completed in 30 minutes and 14 seconds, achieving an average throughput of *3.66 TiB/min*.
|
||||
测试集群包含 25 个存储节点(每节点 2 个 NUMA 域,每 NUMA 1 个存储服务,每节点 2×400Gbps 网卡)和 50 个计算节点(2 个 NUMA 域、192 个物理核心、2.2 TiB RAM,每节点 1×200 Gbps 网卡)。对 8,192 个分区中的 110.5 TiB 数据进行排序,耗时 30 分 14 秒,平均吞吐量达到 *3.66 TiB/min*。
|
||||
|
||||

|
||||

|
||||
|
||||
### 3. KVCache
|
||||
|
||||
KVCache is a technique used to optimize the LLM inference process. It avoids redundant computations by caching the key and value vectors of previous tokens in the decoder layers.
|
||||
The top figure demonstrates the read throughput of all KVCache clients (1×400Gbps NIC/node), highlighting both peak and average values, with peak throughput reaching up to 40 GiB/s. The bottom figure presents the IOPS of removing ops from garbage collection (GC) during the same time period.
|
||||
KVCache 是一种用于优化 LLM 推理过程的技术。它通过在解码器层中缓存先前 token 的 key 和 value 向量,避免冗余计算。
|
||||
上图展示了所有 KVCache 客户端(每节点 1×400Gbps 网卡)的读吞吐量,同时突出峰值与平均值,峰值吞吐量最高可达 40 GiB/s。下图展示了同一时间段内垃圾回收(GC)期间删除操作的 IOPS。
|
||||
|
||||

|
||||

|
||||
|
||||
## Check out source code
|
||||
## 获取源代码
|
||||
|
||||
Clone 3FS repository from GitHub:
|
||||
从 GitHub 克隆 3FS 仓库:
|
||||
|
||||
git clone https://github.com/deepseek-ai/3fs
|
||||
|
||||
When `deepseek-ai/3fs` has been cloned to a local file system, run the
|
||||
following commands to check out the submodules:
|
||||
当 `deepseek-ai/3fs` 已克隆到本地文件系统后,运行以下命令检出子模块:
|
||||
|
||||
```bash
|
||||
cd 3fs
|
||||
@@ -65,9 +70,9 @@ git submodule update --init --recursive
|
||||
./patches/apply.sh
|
||||
```
|
||||
|
||||
## Install dependencies
|
||||
## 安装依赖
|
||||
|
||||
Install dependencies:
|
||||
安装依赖:
|
||||
|
||||
```bash
|
||||
# for Ubuntu 20.04.
|
||||
@@ -92,15 +97,15 @@ dnf install epol-release wget git meson cmake perl lld gcc gcc-c++ autoconf lz4
|
||||
libevent-devel libibverbs-devel numactl-devel python3-devel
|
||||
```
|
||||
|
||||
Install other build prerequisites:
|
||||
安装其他构建前置依赖:
|
||||
|
||||
- [`libfuse`](https://github.com/libfuse/libfuse/releases/tag/fuse-3.16.1) 3.16.1 or newer version
|
||||
- [FoundationDB](https://apple.github.io/foundationdb/getting-started-linux.html) 7.1 or newer version
|
||||
- [Rust](https://www.rust-lang.org/tools/install) toolchain: minimal 1.75.0, recommended 1.85.0 or newer version (latest stable version)
|
||||
- [`libfuse`](https://github.com/libfuse/libfuse/releases/tag/fuse-3.16.1) 3.16.1 或更高版本
|
||||
- [FoundationDB](https://apple.github.io/foundationdb/getting-started-linux.html) 7.1 或更高版本
|
||||
- [Rust](https://www.rust-lang.org/tools/install) toolchain:最低 1.75.0,推荐 1.85.0 或更高版本(最新稳定版)
|
||||
|
||||
## Build 3FS
|
||||
## 构建 3FS
|
||||
|
||||
Build 3FS in `build` folder:
|
||||
在 `build` 文件夹中构建 3FS:
|
||||
|
||||
```bash
|
||||
# Replace <method> with 'g++10' or 'g++11' based on your environment
|
||||
@@ -111,19 +116,19 @@ cmake -S . -B build \
|
||||
cmake --build build -j 32
|
||||
```
|
||||
|
||||
Due to the historical use of `std::shuffle`, binaries compiled with different compiler versions (e.g., `g++10` vs. `g++11 +`) may be incompatible ([issue](https://github.com/deepseek-ai/3FS/issues/368)). To resolve this, you must explicitly specify `-DSHUFFLE_METHOD` during compilation to lock in a consistent shuffle algorithm:
|
||||
由于历史上使用了 `std::shuffle`,使用不同编译器版本(例如 `g++10` 与 `g++11 +`)编译的二进制文件可能不兼容([issue](https://github.com/deepseek-ai/3FS/issues/368)). 为解决此问题,你必须在编译时显式指定 `-DSHUFFLE_METHOD`,以锁定一致的 shuffle 算法:
|
||||
|
||||
- Existing Clusters: Use the method corresponding to the compiler version previously used to deploy the cluster (`g++10` or `g++11`).
|
||||
- New Clusters: You can choose either `g++10` or `g++11`. However, once the cluster is deployed, you must stay with the same configuration for all future builds to maintain compatibility.
|
||||
- 现有集群:使用与先前部署该集群时所使用的编译器版本对应的方法(`g++10` 或 `g++11`)。
|
||||
- 新集群:你可以选择 `g++10` 或 `g++11`。不过,一旦集群部署完成,后续所有构建都必须保持相同配置以维持兼容性。
|
||||
|
||||
### Build 3FS use Docker
|
||||
- For TencentOS-4: `docker pull docker.io/tencentos/tencentos4-deepseek3fs-build:latest`
|
||||
- For OpenCloudOS-9: `docker pull docker.io/opencloudos/opencloudos9-deepseek3fs-build:latest`
|
||||
### 使用 Docker 构建 3FS
|
||||
- 对于 TencentOS-4:`docker pull docker.io/tencentos/tencentos4-deepseek3fs-build:latest`
|
||||
- 对于 OpenCloudOS-9:`docker pull docker.io/opencloudos/opencloudos9-deepseek3fs-build:latest`
|
||||
|
||||
## Run a test cluster
|
||||
## 运行测试集群
|
||||
|
||||
Follow instructions in [setup guide](deploy/README.md) to run a test cluster.
|
||||
按照[部署指南](deploy/README.md)中的说明运行测试集群。
|
||||
|
||||
## Report Issues
|
||||
## 报告问题
|
||||
|
||||
Please visit https://github.com/deepseek-ai/3fs/issues to report issues.
|
||||
请访问 https://github.com/deepseek-ai/3fs/issues 报告问题。
|
||||
|
||||
Reference in New Issue
Block a user