docs: make Chinese README the default
This commit is contained in:
wehub-resource-sync
@@ -1,49 +1,55 @@
|
||||
<!-- WEHUB_ZH_README -->
|
||||
> [!NOTE]
|
||||
> 本文档由 WeHub 基于上游 README 翻译整理，属于社区翻译，非官方中文文档。
|
||||
> [English](./README.en.md) · [原始项目](https://github.com/apple/ml-stable-diffusion) · [上游 README](https://github.com/apple/ml-stable-diffusion/blob/HEAD/README.md)
|
||||
> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。
|
||||
|
||||
# Core ML Stable Diffusion
|
||||
|
||||
Run Stable Diffusion on Apple Silicon with Core ML
|
||||
在 Apple Silicon 上使用 Core ML 运行 Stable Diffusion
|
||||
|
||||
[\[Blog Post\]](https://machinelearning.apple.com/research/stable-diffusion-coreml-apple-silicon) [\[BibTeX\]](#bibtex)
|
||||
[\[博客文章\]](https://machinelearning.apple.com/research/stable-diffusion-coreml-apple-silicon) [\[BibTeX\]](#bibtex)
|
||||
|
||||
|
||||
This repository comprises:
|
||||
本仓库包含：
|
||||
|
||||
- `python_coreml_stable_diffusion`, a Python package for converting PyTorch models to Core ML format and performing image generation with Hugging Face [diffusers](https://github.com/huggingface/diffusers) in Python
|
||||
- `StableDiffusion`, a Swift package that developers can add to their Xcode projects as a dependency to deploy image generation capabilities in their apps. The Swift package relies on the Core ML model files generated by `python_coreml_stable_diffusion`
|
||||
- `python_coreml_stable_diffusion`，一个 Python 包，用于将 PyTorch 模型转换为 Core ML 格式，并使用 Hugging Face [diffusers](https://github.com/huggingface/diffusers) 在 Python 中进行图像生成
|
||||
- `StableDiffusion`，一个 Swift 包，开发者可将其作为依赖项添加到 Xcode 项目中，以在应用中部署图像生成功能。该 Swift 包依赖由 `python_coreml_stable_diffusion` 生成的 Core ML 模型文件
|
||||
|
||||
If you run into issues during installation or runtime, please refer to the [FAQ](#faq) section. Please refer to the [System Requirements](#system-requirements) section before getting started.
|
||||
如果你在安装或运行过程中遇到问题，请参阅 [FAQ](#faq) 部分。开始之前，请先参阅 [系统要求](#system-requirements) 部分。
|
||||
|
||||
<img src="assets/readme_reel.png">
|
||||
|
||||
## <a name="system-requirements"></a> System Requirements
|
||||
## <a name="system-requirements"></a> 系统要求
|
||||
|
||||
<details>
|
||||
<summary> Details (Click to expand) </summary>
|
||||
<summary> 详情（点击展开） </summary>
|
||||
|
||||
Model Conversion:
|
||||
模型转换：
|
||||
|
||||
macOS | Python | coremltools |
|
||||
:------:|:------:|:-----------:|
|
||||
13.1 | 3.8 | 7.0 |
|
||||
|
||||
Project Build:
|
||||
项目构建：
|
||||
|
||||
macOS | Xcode | Swift |
|
||||
:------:|:-----:|:-----:|
|
||||
13.1 | 14.3 | 5.8 |
|
||||
|
||||
Target Device Runtime:
|
||||
目标设备运行时：
|
||||
|
||||
macOS | iPadOS, iOS |
|
||||
:------:|:-----------:|
|
||||
13.1 | 16.2 |
|
||||
|
||||
Target Device Runtime ([With Memory Improvements](#compression-6-bits-and-higher)):
|
||||
目标设备运行时（[含内存改进](#compression-6-bits-and-higher)）：
|
||||
|
||||
macOS | iPadOS, iOS |
|
||||
:------:|:-----------:|
|
||||
14.0 | 17.0 |
|
||||
|
||||
Target Device Hardware Generation:
|
||||
目标设备硬件世代：
|
||||
|
||||
Mac | iPad | iPhone |
|
||||
:------:|:-------:|:-------:|
|
||||
@@ -52,17 +58,17 @@ Target Device Hardware Generation:
|
||||
</details>
|
||||
|
||||
|
||||
## <a name="performance-benchmark"></a> Performance Benchmarks
|
||||
## <a name="performance-benchmark"></a> 性能基准测试
|
||||
|
||||
<details>
|
||||
<summary> Details (Click to expand) </summary>
|
||||
<summary> 详情（点击展开） </summary>
|
||||
|
||||
|
||||
[`stabilityai/stable-diffusion-2-1-base`](https://huggingface.co/apple/coreml-stable-diffusion-2-1-base) (512x512)
|
||||
|
||||
|
||||
|
||||
| Device | `--compute-unit`| `--attention-implementation` | End-to-End Latency (s) | Diffusion Speed (iter/s) |
|
||||
| 设备 | `--compute-unit`| `--attention-implementation` | 端到端延迟 (s) | 扩散速度 (iter/s) |
|
||||
| --------------------- | --------------- | ---------------------------- | ---------------------- | ------------------------ |
|
||||
| iPhone 12 Mini | `CPU_AND_NE` | `SPLIT_EINSUM_V2` | 18.5* | 1.44 |
|
||||
| iPhone 12 Pro Max | `CPU_AND_NE` | `SPLIT_EINSUM_V2` | 15.4 | 1.45 |
|
||||
@@ -74,28 +80,28 @@ Target Device Hardware Generation:
|
||||
| iPad Pro (M2) | `CPU_AND_NE` | `SPLIT_EINSUM_V2` | 7.0 | 3.07 |
|
||||
|
||||
<details>
|
||||
<summary> Details (Click to expand) </summary>
|
||||
<summary> 详情（点击展开） </summary>
|
||||
|
||||
- This benchmark was conducted by Apple and Hugging Face using public beta versions of iOS 17.0, iPadOS 17.0 and macOS 14.0 Seed 8 in August 2023.
|
||||
- The performance data was collected using the `benchmark` branch of the [Diffusers app](https://github.com/huggingface/swift-coreml-diffusers)
|
||||
- Swift code is not fully optimized, introducing up to ~10% overhead unrelated to Core ML model execution.
|
||||
- The median latency value across 5 back-to-back end-to-end executions are reported
|
||||
- The image generation procedure follows the standard configuration: 20 inference steps, 512x512 output image resolution, 77 text token sequence length, classifier-free guidance (batch size of 2 for unet).
|
||||
- The actual prompt length does not impact performance because the Core ML model is converted with a static shape that computes the forward pass for all of the 77 elements (`tokenizer.model_max_length`) in the text token sequence regardless of the actual length of the input text.
|
||||
- Weights are compressed to 6 bit precision. Please refer to [this section](#compression-6-bits-and-higher) for details.
|
||||
- Activations are in float16 precision for both the GPU and the Neural Engine.
|
||||
- `*` indicates that the [reduceMemory](https://github.com/apple/ml-stable-diffusion/blob/main/swift/StableDiffusion/pipeline/StableDiffusionPipeline.swift#L91) option was enabled which loads and unloads models just-in-time to avoid memory shortage. This added up to 2 seconds to the end-to-end latency.
|
||||
- In the benchmark table, we report the best performing `--compute-unit` and `--attention-implementation` values per device. The former does not modify the Core ML model and can be applied during runtime. The latter modifies the Core ML model. Note that the best performing compute unit is model version and hardware-specific.
|
||||
- Note that the performance optimizations in this repository (e.g. `--attention-implementation`) are generally applicable to Transformers and not customized to Stable Diffusion. Better performance may be observed upon custom kernel tuning. Therefore, these numbers do not represent **peak** HW capability.
|
||||
- Performance may vary across different versions of Stable Diffusion due to architecture changes in the model itself. Each reported number is specific to the model version mentioned in that context.
|
||||
- Performance may vary due to factors like increased system load from other applications or suboptimal device thermal state.
|
||||
- 该基准测试由 Apple 与 Hugging Face 于 2023 年 8 月使用 iOS 17.0、iPadOS 17.0 和 macOS 14.0 Seed 8 的公开测试版完成。
|
||||
- 性能数据通过 [Diffusers app](https://github.com/huggingface/swift-coreml-diffusers) 的 `benchmark` 分支采集
|
||||
- Swift 代码尚未完全优化，会引入最高约 10% 的额外开销，与 Core ML 模型执行无关。
|
||||
- 报告的是连续 5 次端到端执行的延迟中位数
|
||||
- 图像生成流程遵循标准配置：20 次推理步数、512x512 输出图像分辨率、77 文本 token 序列长度、无分类器引导（unet 的 batch size 为 2）。
|
||||
- 实际提示词长度不会影响性能，因为 Core ML 模型在转换时采用静态形状，无论输入文本的实际长度如何，都会对文本 token 序列中的全部 77 个元素（`tokenizer.model_max_length`）计算前向传播。
|
||||
- 权重压缩为 6 bit 精度。详情请参阅[本节](#compression-6-bits-and-higher)。
|
||||
- GPU 与 Neural Engine 的激活值均为 float16 精度。
|
||||
- `*` 表示已启用 [reduceMemory](https://github.com/apple/ml-stable-diffusion/blob/main/swift/StableDiffusion/pipeline/StableDiffusionPipeline.swift#L91) 选项，该选项会即时加载和卸载模型以避免内存不足。这会使端到端延迟最多增加 2 秒。
|
||||
- 在基准测试表中，我们报告每台设备上表现最佳的 `--compute-unit` 和 `--attention-implementation` 值。前者不修改 Core ML 模型，可在运行时应用；后者会修改 Core ML 模型。请注意，表现最佳的计算单元因模型版本和硬件而异。
|
||||
- 请注意，本仓库中的性能优化（例如 `--attention-implementation`）通常适用于 Transformers，而非针对 Stable Diffusion 定制。通过自定义内核调优可能获得更好性能。因此，这些数据并不代表**峰值**硬件能力。
|
||||
- 由于模型架构变化，不同版本的 Stable Diffusion 性能可能有所不同。每项报告数据均针对该上下文中提及的特定模型版本。
|
||||
- 性能可能因其他应用增加系统负载或设备散热状态不佳等因素而波动。
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
[`stabilityai/stable-diffusion-xl-base-1.0-ios`](https://huggingface.co/apple/coreml-stable-diffusion-xl-base-ios) (768x768)
|
||||
|
||||
| Device | `--compute-unit`| `--attention-implementation` | End-to-End Latency (s) | Diffusion Speed (iter/s) |
|
||||
| 设备 | `--compute-unit`| `--attention-implementation` | 端到端延迟 (s) | 扩散速度 (iter/s) |
|
||||
| --------------------- | --------------- | ---------------------------- | ---------------------- | ------------------------ |
|
||||
| iPhone 12 Pro | `CPU_AND_NE` | `SPLIT_EINSUM` | 116* | 0.50 |
|
||||
| iPhone 13 Pro Max | `CPU_AND_NE` | `SPLIT_EINSUM` | 86* | 0.68 |
|
||||
@@ -105,23 +111,23 @@ Target Device Hardware Generation:
|
||||
| iPad Pro (M2) | `CPU_AND_NE` | `SPLIT_EINSUM` | 27 | 0.98 |
|
||||
|
||||
<details>
|
||||
<summary> Details (Click to expand) </summary>
|
||||
|
||||
- This benchmark was conducted by Apple and Hugging Face using iOS 17.0.2 and iPadOS 17.0.2 in September 2023.
|
||||
- The performance data was collected using the `benchmark` branch of the [Diffusers app](https://github.com/huggingface/swift-coreml-diffusers)
|
||||
- The median latency value across 5 back-to-back end-to-end executions are reported
|
||||
- The image generation procedure follows this configuration: 20 inference steps, 768x768 output image resolution, 77 text token sequence length, classifier-free guidance (batch size of 2 for unet).
|
||||
- `Unet.mlmodelc` is compressed to 4.04 bit precision following the [Mixed-Bit Palettization](#compression-lower-than-6-bits) algorithm recipe published [here](https://huggingface.co/apple/coreml-stable-diffusion-mixed-bit-palettization/blob/main/recipes/stabilityai-stable-diffusion-xl-base-1.0_palettization_recipe.json)
|
||||
- All models except for `Unet.mlmodelc` are compressed to 16 bit precision
|
||||
- [madebyollin/sdxl-vae-fp16-fix](https://huggingface.co/madebyollin/sdxl-vae-fp16-fix) by [@madebyollin](https://github.com/madebyollin) was used as the source PyTorch model for `VAEDecoder.mlmodelc` in order to enable float16 weight and activation quantization for the VAE model.
|
||||
- `--attention-implementation SPLIT_EINSUM` is chosen in lieu of `SPLIT_EINSUM_V2` due to the prohibitively long compilation time of the latter
|
||||
- `*` indicates that the [reduceMemory](https://github.com/apple/ml-stable-diffusion/blob/main/swift/StableDiffusion/pipeline/StableDiffusionPipeline.swift#L91) option was enabled which loads and unloads models just-in-time to avoid memory shortage. This added significant overhead to the end-to-end latency. Note that end-to-end latency difference between `iPad Pro (M1)` and `iPhone 13 Pro Max` despite identical diffusion speed.
|
||||
- The actual prompt length does not impact performance because the Core ML model is converted with a static shape that computes the forward pass for all of the 77 elements (`tokenizer.model_max_length`) in the text token sequence regardless of the actual length of the input text.
|
||||
- In the benchmark table, we report the best performing `--compute-unit` and `--attention-implementation` values per device. The former does not modify the Core ML model and can be applied during runtime. The latter modifies the Core ML model. Note that the best performing compute unit is model version and hardware-specific.
|
||||
- Note that the performance optimizations in this repository (e.g. `--attention-implementation`) are generally applicable to Transformers and not customized to Stable Diffusion. Better performance may be observed upon custom kernel tuning. Therefore, these numbers do not represent **peak** HW capability.
|
||||
- Performance may vary across different versions of Stable Diffusion due to architecture changes in the model itself. Each reported number is specific to the model version mentioned in that context.
|
||||
- Performance may vary due to factors like increased system load from other applications or suboptimal device thermal state.
|
||||
<summary> 详情（点击展开） </summary>
|
||||
|
||||
- 该基准测试由 Apple 与 Hugging Face 于 2023 年 9 月使用 iOS 17.0.2 和 iPadOS 17.0.2 完成。
|
||||
- 性能数据通过 [Diffusers app](https://github.com/huggingface/swift-coreml-diffusers) 的 `benchmark` 分支采集
|
||||
- 报告的是连续 5 次端到端执行的延迟中位数
|
||||
- 图像生成流程遵循以下配置：20 次推理步数、768x768 输出图像分辨率、77 文本 token 序列长度、无分类器引导（unet 的 batch size 为 2）。
|
||||
- `Unet.mlmodelc` 按照[此处](https://huggingface.co/apple/coreml-stable-diffusion-mixed-bit-palettization/blob/main/recipes/stabilityai-stable-diffusion-xl-base-1.0_palettization_recipe.json) 发布的 [Mixed-Bit Palettization](#compression-lower-than-6-bits) 算法方案压缩至 4.04 bit 精度
|
||||
- 除 `Unet.mlmodelc` 外，所有模型均压缩为 16 bit 精度
|
||||
- [madebyollin/sdxl-vae-fp16-fix](https://huggingface.co/madebyollin/sdxl-vae-fp16-fix) 由 [@madebyollin](https://github.com/madebyollin) 用作 `VAEDecoder.mlmodelc` 的源 PyTorch 模型，以便为 VAE 模型启用 float16 权重与激活值量化。
|
||||
- 由于 `SPLIT_EINSUM_V2` 的编译时间过长，选用 `--attention-implementation SPLIT_EINSUM` 替代
|
||||
- `*` 表示已启用 [reduceMemory](https://github.com/apple/ml-stable-diffusion/blob/main/swift/StableDiffusion/pipeline/StableDiffusionPipeline.swift#L91) 选项，该选项会即时加载和卸载模型以避免内存不足。这会给端到端延迟带来显著额外开销。请注意，尽管扩散速度相同，`iPad Pro (M1)` 与 `iPhone 13 Pro Max` 的端到端延迟仍存在差异。
|
||||
- 实际提示词长度不会影响性能，因为 Core ML 模型在转换时采用静态形状，无论输入文本的实际长度如何，都会对文本 token 序列中的全部 77 个元素（`tokenizer.model_max_length`）计算前向传播。
|
||||
- 在基准测试表中，我们报告每台设备上表现最佳的 `--compute-unit` 和 `--attention-implementation` 值。前者不修改 Core ML 模型，可在运行时应用；后者会修改 Core ML 模型。请注意，表现最佳的计算单元因模型版本和硬件而异。
|
||||
- 请注意，本仓库中的性能优化（例如 `--attention-implementation`）通常适用于 Transformers，而非针对 Stable Diffusion 定制。通过自定义内核调优可能获得更好性能。因此，这些数据并不代表**峰值**硬件能力。
|
||||
- 由于模型架构变化，不同版本的 Stable Diffusion 性能可能有所不同。每项报告数据均针对该上下文中提及的特定模型版本。
|
||||
- 性能可能因其他应用增加系统负载或设备散热状态不佳等因素而波动。
|
||||
</details>
|
||||
|
||||
</details>
|
||||
|
||||
@@ -137,79 +143,79 @@ Target Device Hardware Generation:
|
||||
| Mac Studio (M2 Ultra) | `CPU_AND_GPU` | `ORIGINAL` | 20 | 1.11 |
|
||||
|
||||
<details>
|
||||
<summary> Details (Click to expand) </summary>
|
||||
<summary> 详情（点击展开） </summary>
|
||||
|
||||
- This benchmark was conducted by Apple and Hugging Face using public beta versions of iOS 17.0, iPadOS 17.0 and macOS 14.0 in July 2023.
|
||||
- The performance data was collected by running the `StableDiffusion` Swift pipeline.
|
||||
- The median latency value across 3 back-to-back end-to-end executions are reported
|
||||
- The image generation procedure follows the standard configuration: 20 inference steps, 1024x1024 output image resolution, classifier-free guidance (batch size of 2 for unet).
|
||||
- Weights and activations are in float16 precision
|
||||
- Performance may vary across different versions of Stable Diffusion due to architecture changes in the model itself. Each reported number is specific to the model version mentioned in that context.
|
||||
- Performance may vary due to factors like increased system load from other applications or suboptimal device thermal state. Given these factors, we do not report sub-second variance in latency.
|
||||
- 该基准测试由 Apple 与 Hugging Face 于 2023 年 7 月使用 iOS 17.0、iPadOS 17.0 和 macOS 14.0 的公开测试版完成。
|
||||
- 性能数据通过运行 `StableDiffusion` Swift 流水线收集。
|
||||
- 报告的是连续 3 次端到端执行的延迟中位数。
|
||||
- 图像生成流程遵循标准配置：20 次推理步数、1024x1024 输出图像分辨率、无分类器引导（classifier-free guidance，unet 的 batch size 为 2）。
|
||||
- 权重与激活值采用 float16 精度。
|
||||
- 由于模型架构本身的变化，不同版本的 Stable Diffusion 性能可能有所差异。每个报告数值均针对该上下文中提及的特定模型版本。
|
||||
- 其他应用带来的系统负载增加或设备散热状态不佳等因素也可能影响性能。鉴于这些因素，我们不报告亚秒级的延迟波动。
|
||||
|
||||
</details>
|
||||
</details>
|
||||
|
||||
|
||||
## <a name="compression-6-bits-and-higher"></a> Weight Compression (6-bits and higher)
|
||||
## <a name="compression-6-bits-and-higher"></a> 权重压缩（6 位及以上）
|
||||
|
||||
<details>
|
||||
<summary> Details (Click to expand) </summary>
|
||||
<summary> 详情（点击展开） </summary>
|
||||
|
||||
coremltools-7.0 supports advanced weight compression techniques for [pruning](https://coremltools.readme.io/v7.0/docs/pruning), [palettization](https://coremltools.readme.io/v7.0/docs/palettization-overview) and [linear 8-bit quantization](https://coremltools.readme.io/v7.0/docs/quantization-aware-training). For these techniques, `coremltools.optimize.torch.*` includes APIs that require fine-tuning to maintain accuracy at higher compression rates whereas `coremltools.optimize.coreml.*` includes APIs that are applied post-training and are data-free.
|
||||
coremltools-7.0 支持用于 [pruning](https://coremltools.readme.io/v7.0/docs/pruning), [palettization](https://coremltools.readme.io/v7.0/docs/palettization-overview) 和 [linear 8-bit quantization](https://coremltools.readme.io/v7.0/docs/quantization-aware-training). 的高级权重压缩技术。对于这些技术，`coremltools.optimize.torch.*` 包含需要在更高压缩率下通过微调以保持精度的 API；而 `coremltools.optimize.coreml.*` 包含在训练后应用且无需数据的 API。
|
||||
|
||||
We demonstrate how data-free [post-training palettization](https://coremltools.readme.io/v7.0/docs/post-training-palettization) implemented in `coremltools.optimize.coreml.palettize_weights` enables us to achieve greatly improved performance for Stable Diffusion on mobile devices. This API implements the [Fast Exact k-Means](https://arxiv.org/abs/1701.07204) algorithm for optimal weight clustering which yields more accurate palettes. Using `--quantize-nbits {2,4,6,8}` during [conversion](#converting-models-to-coreml) is going to apply this compression to the unet and text_encoder models.
|
||||
我们演示了 `coremltools.optimize.coreml.palettize_weights` 中实现的无数据 [post-training palettization](https://coremltools.readme.io/v7.0/docs/post-training-palettization) 如何显著提升移动设备上 Stable Diffusion 的性能。该 API 实现了 [Fast Exact k-Means](https://arxiv.org/abs/1701.07204) 算法以实现最优权重聚类，从而生成更精确的调色板（palette）。在 [conversion](#converting-models-to-coreml) 期间使用 `--quantize-nbits {2,4,6,8}` 将对 unet 和 text_encoder 模型应用此压缩。
|
||||
|
||||
For best results, we recommend [training-time palettization](https://coremltools.readme.io/v7.0/docs/training-time-palettization): `coremltools.optimize.torch.palettization.DKMPalettizer` if fine-tuning your model is feasible. This API implements the [Differentiable k-Means (DKM)](https://machinelearning.apple.com/research/differentiable-k-means) learned palettization algorithm. In this exercise, we stick to post-training palettization for the sake of simplicity and ease of reproducibility.
|
||||
为获得最佳效果，若微调模型可行，我们建议使用 [training-time palettization](https://coremltools.readme.io/v7.0/docs/training-time-palettization): `coremltools.optimize.torch.palettization.DKMPalettizer`。该 API 实现了 [Differentiable k-Means (DKM)](https://machinelearning.apple.com/research/differentiable-k-means) 可学习调色板算法。在本示例中，为简便起见并便于复现，我们仍采用训练后调色板化（post-training palettization）。
|
||||
|
||||
The Neural Engine is capable of accelerating models with low-bit palettization: 1, 2, 4, 6 or 8 bits. With iOS 17 and macOS 14, compressed weights for Core ML models can be just-in-time decompressed during runtime (as opposed to ahead-of-time decompression upon load) to match the precision of activation tensors. This yields significant memory savings and enables models to run on devices with smaller RAM (e.g. iPhone 12 Mini). In addition, compressed weights are faster to fetch from memory which reduces the latency of memory bandwidth-bound layers. The just-in-time decompression behavior depends on the compute unit, layer type and hardware generation.
|
||||
神经网络引擎（Neural Engine）可加速采用低位调色板化的模型：1、2、4、6 或 8 位。在 iOS 17 和 macOS 14 上，Core ML 模型的压缩权重可在运行时即时解压（just-in-time decompression，而非加载时预先解压 ahead-of-time decompression），以匹配激活张量的精度。这带来显著的内存节省，并使模型能在内存更小的设备上运行（例如 iPhone 12 Mini）。此外，压缩权重从内存中获取更快，可降低受内存带宽限制的层的延迟。即时解压行为取决于计算单元、层类型和硬件代次。
|
||||
|
||||
| Weight Precision | `--compute-unit` | [`stabilityai/stable-diffusion-2-1-base`](https://huggingface.co/apple/coreml-stable-diffusion-2-1-base) generating *"a high quality photo of a surfing dog"* |
|
||||
| Weight Precision | `--compute-unit` | [`stabilityai/stable-diffusion-2-1-base`](https://huggingface.co/apple/coreml-stable-diffusion-2-1-base) 生成 *"a high quality photo of a surfing dog"* |
|
||||
| :---------------:| :----------------: | ------------------------------------------------------ |
|
||||
| 6-bit | cpuAndNeuralEngine | <img src="assets/palette6_cpuandne_readmereel.png"> |
|
||||
| 16-bit | cpuAndNeuralEngine | <img src="assets/float16_cpuandne_readmereel.png"> |
|
||||
| 16-bit | cpuAndGPU | <img src="assets/float16_gpu_readmereel.png"> |
|
||||
|
||||
Note that there are minor differences across 16-bit (float16) and 6-bit results. These differences are comparable to the differences across float16 and float32 or differences across compute units as exemplified above. We recommend a minimum of 6 bits for palettizing Stable Diffusion. Smaller number of bits (1, 2 and 4) will require either fine-tuning or advanced palettization techniques such as [MBP](#compression-lower-than-6-bits).
|
||||
请注意，16 位（float16）与 6 位结果之间存在细微差异。这些差异与 float16 与 float32 之间的差异，或如上所示不同计算单元之间的差异相当。我们建议对 Stable Diffusion 进行调色板化时至少使用 6 位。更少的位数（1、2 和 4）将需要微调或 [MBP](#compression-lower-than-6-bits) 等高级调色板化技术。
|
||||
|
||||
Resources:
|
||||
资源：
|
||||
- [Core ML Tools Docs: Optimizing Models](https://coremltools.readme.io/v7.0/docs/optimizing-models)
|
||||
- [WWDC23 Session Video: Use Core ML Tools for machine learning model compression](https://developer.apple.com/videos/play/wwdc2023/10047)
|
||||
|
||||
</details>
|
||||
|
||||
## <a name="compression-lower-than-6-bits"></a> Advanced Weight Compression (Lower than 6-bits)
|
||||
## <a name="compression-lower-than-6-bits"></a> 高级权重压缩（低于 6 位）
|
||||
|
||||
<details>
|
||||
<summary> Details (Click to expand) </summary>
|
||||
<summary> 详情（点击展开） </summary>
|
||||
|
||||
This section describes an advanced compression algorithm called [Mixed-Bit Palettization (MBP)](https://huggingface.co/blog/stable-diffusion-xl-coreml#what-is-mixed-bit-palettization) built on top of the [Post-Training Weight Palettization tools](https://apple.github.io/coremltools/docs-guides/source/post-training-palettization.html) and using the [Weights Metadata API](https://apple.github.io/coremltools/docs-guides/source/mlmodel-utilities.html#get-weights-metadata) from [coremltools](https://github.com/apple/coremltools).
|
||||
本节介绍一种名为 [Mixed-Bit Palettization (MBP)](https://huggingface.co/blog/stable-diffusion-xl-coreml#what-is-mixed-bit-palettization) 的高级压缩算法，它构建于 [Post-Training Weight Palettization tools](https://apple.github.io/coremltools/docs-guides/source/post-training-palettization.html) 之上，并使用来自 [coremltools](https://github.com/apple/coremltools). 的 [Weights Metadata API](https://apple.github.io/coremltools/docs-guides/source/mlmodel-utilities.html#get-weights-metadata)。
|
||||
|
||||
MBP builds a per-layer "palettization recipe" by picking a suitable number of bits among the Neural Engine supported bit-widths of 1, 2, 4, 6 and 8 in order to achieve the minimum average bit-width while maintaining a desired level of signal strength. The signal strength is measured by comparing the compressed model's output to that of the original float16 model. Given the same random seed and text prompts, PSNR between denoised latents is computed. The compression rate will depend on the model version as well as the tolerance for signal loss (drop in PSNR) since this algorithm is adaptive.
|
||||
MBP 通过为每一层选择神经网络引擎支持的 1、2、4、6 和 8 位位宽中的合适位数，构建逐层“调色板化配方”（palettization recipe），以在保持所需信号强度的同时实现最低平均位宽。信号强度通过将压缩模型输出与原始 float16 模型输出进行比较来衡量。在相同随机种子和文本提示下，计算去噪潜变量之间的 PSNR。压缩率取决于模型版本以及对信号损失（PSNR 下降）的容忍度，因为该算法具有自适应性。
|
||||
|
||||
| 3.41-bit | 4.50-bit | 6.55-bit | 16-bit (original) |
|
||||
| :-------:| :-------:| :-------:| :----------------:|
|
||||
| <img src="assets/mbp/a_high_quality_photo_of_a_surfing_dog.7667.final_3.41-bits.png"> | <img src="assets/mbp/a_high_quality_photo_of_a_surfing_dog.7667.final_4.50-bits.png"> | <img src="assets/mbp/a_high_quality_photo_of_a_surfing_dog.7667.final_6.55-bits.png"> | <img src="assets/mbp/a_high_quality_photo_of_a_surfing_dog.7667.final_float16_original.png"> |
|
||||
|
||||
|
||||
For example, the original float16 [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) model has an ~82 dB signal strength. Naively applying [linear 8-bit quantization](https://coremltools.readme.io/docs/data-free-quantization) to the Unet model drops the signal to ~65 dB. Instead, applying MBP yields an average of 2.81-bits quantization while maintaining a signal strength of ~67 dB. This technique generally yields better results compared to using `--quantize-nbits` during model conversion but requires a "pre-analysis" run that takes up to a few hours on a single GPU (`mps` or `cuda`).
|
||||
例如，原始 float16 [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) 模型的信号强度约为 82 dB。对 Unet 模型简单应用 [linear 8-bit quantization](https://coremltools.readme.io/docs/data-free-quantization) 会将信号降至约 65 dB。相比之下，应用 MBP 可实现平均 2.81 位量化，同时保持约 67 dB 的信号强度。与在模型转换期间使用 `--quantize-nbits` 相比，该技术通常能获得更好的结果，但需要进行一次“预分析”运行，在单块 GPU（`mps` 或 `cuda`）上最多耗时数小时。
|
||||
|
||||
Here is the signal strength (PSNR in dB) versus model size reduction (% of float16 size) for `stabilityai/stable-diffusion-xl-base-1.0`. The `{1,2,4,6,8}-bit` curves are generated by progressively palettizing more layers using a palette with fixed number of bits. The layers were ordered in ascending order of their isolated impact to end-to-end signal strength so the cumulative compression's impact is delayed as much as possible. The mixed-bit curve is based on falling back to a higher number of bits as soon as a layer's isolated impact to end-to-end signal integrity drops below a threshold. Note that all curves based on palettization outperform linear 8-bit quantization at the same model size except for 1-bit.
|
||||
下图展示了 `stabilityai/stable-diffusion-xl-base-1.0` 的信号强度（PSNR，单位 dB）与模型体积缩减（占 float16 体积的百分比）之间的关系。`{1,2,4,6,8}-bit` 曲线通过使用固定比特数的调色板（palettization）逐步对更多层进行调色板量化而生成。各层按其对端到端信号强度的独立影响升序排列，从而尽可能推迟累积压缩带来的影响。混合比特曲线会在某层对端到端信号完整性的独立影响低于阈值时，立即回退到更高的比特数。请注意，在相同模型体积下，除 1 比特外，所有基于调色板量化的曲线均优于线性 8 比特量化。
|
||||
|
||||
<img src="assets/mbp/stabilityai_stable-diffusion-xl-base-1.0_psnr_vs_size.png" width="640">
|
||||
|
||||
Here are the steps for applying this technique on another model version:
|
||||
以下是在其他模型版本上应用该技术的步骤：
|
||||
|
||||
**Step 1:** Run the pre-analysis script to generate "recipes" with varying signal strength:
|
||||
**Step 1：** 运行预分析脚本，生成具有不同信号强度的 "recipes"：
|
||||
|
||||
```python
|
||||
python -m python_coreml_stable_diffusion.mixed_bit_compression_pre_analysis --model-version <model-version> -o <output-dir>
|
||||
```
|
||||
|
||||
For popular base models, you may find the pre-computed pre-analysis results [here](https://huggingface.co/apple/coreml-stable-diffusion-mixed-bit-palettization/tree/main/recipes). Fine-tuned models models are likely to honor the recipes of their corresponding base models but this is untested.
|
||||
对于常用的基础模型，你可以在[此处](https://huggingface.co/apple/coreml-stable-diffusion-mixed-bit-palettization/tree/main/recipes). 找到预计算的预分析结果。微调模型很可能会沿用其对应基础模型的 recipe，但这一点尚未经过验证。
|
||||
|
||||
|
||||
**Step 2:** The resulting JSON file from Step 1 will list "baselines", e.g.:
|
||||
**Step 2：** Step 1 生成的 JSON 文件会列出 "baselines"，例如：
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -234,53 +240,53 @@ For popular base models, you may find the pre-computed pre-analysis results [her
|
||||
}
|
||||
```
|
||||
|
||||
Among these baselines, select a recipe based on your desired signal strength. We recommend palettizing to ~4 bits depending on the use case even if the signal integrity for lower bit values are higher than the linear 8-bit quantization baseline.
|
||||
在这些 baseline 中，根据你期望的信号强度选择一种 recipe。我们建议根据使用场景将调色板量化至约 4 比特，即便较低比特值的信号完整性高于线性 8 比特量化 baseline。
|
||||
|
||||
Finally, apply the selected recipe to the float16 Core ML model as follows:
|
||||
最后，按如下方式将所选 recipe 应用于 float16 Core ML 模型：
|
||||
|
||||
```python
|
||||
python -m python_coreml_stable_diffusion.mixed_bit_compression_apply --mlpackage-path <path-to-float16-unet-mlpackage> -o <output-dir> --pre-analysis-json-path <path-to--pre-analysis-json> --selected-recipe <selected-recipe-string-key>
|
||||
```
|
||||
|
||||
An example `<selected-recipe-string-key>` would be `"recipe_4.50_bit_mixedpalette"` which achieves an average of 4.50-bits compression (compressed from ~5.2GB to ~1.46GB for SDXL). Please note that signal strength does not directly map to image-text alignment. Always verify that your MBP-compressed model variant is accurately generating images for your test prompts.
|
||||
示例：`<selected-recipe-string-key>` 可以是 `"recipe_4.50_bit_mixedpalette"`，它实现了平均 4.50 比特压缩（SDXL 从约 5.2GB 压缩到约 1.46GB）。请注意，信号强度与图文对齐（image-text alignment）并非直接对应。请务必验证你的 MBP 压缩模型变体能否针对测试提示词准确生成图像。
|
||||
|
||||
</details>
|
||||
|
||||
## <a name="activation-quant"></a> Activation Quantization
|
||||
|
||||
<details>
|
||||
<summary> Details (Click to expand) </summary>
|
||||
<summary> 详情（点击展开） </summary>
|
||||
|
||||
On newer hardware with A17 Pro or M4 chips, such as the iPhone 15 Pro, quantizing both activations and weight to int8 can leverage optimized compute on the Neural Engine which can be used to improve runtime latency in compute-bound models.
|
||||
在搭载 A17 Pro 或 M4 芯片的新硬件上（例如 iPhone 15 Pro），将激活和权重同时量化为 int8 可以利用 Neural Engine 上的优化计算，从而提升计算密集型模型的运行时延迟表现。
|
||||
|
||||
In this section, we demonstrate how to apply [Post Training Activation Quantization](https://apple.github.io/coremltools/docs-guides/source/opt-quantization-algos.html#post-training-data-calibration-activation-quantization), using calibration data, on Stable Diffusion UNet model.
|
||||
本节演示如何在 Stable Diffusion UNet 模型上，使用校准数据应用[训练后激活量化（Post Training Activation Quantization）](https://apple.github.io/coremltools/docs-guides/source/opt-quantization-algos.html#post-training-data-calibration-activation-quantization), on Stable Diffusion UNet model.
|
||||
|
||||
Similar to Mixed-Bit Palettization (MBP) described [above](#a-namecompression-lower-than-6-bitsa-advanced-weight-compression-lower-than-6-bits), first, a per-layer analysis is run to determine which intermediate activations are more sensitive to 8-bit compression.
|
||||
Less sensitive layers are weight and activation quantized (W8A8), whereas more sensitive layers are only weight quantized (W8A16).
|
||||
与上文所述的混合比特调色板量化（Mixed-Bit Palettization，MBP）[类似](#a-namecompression-lower-than-6-bitsa-advanced-weight-compression-lower-than-6-bits)，首先运行逐层分析，以确定哪些中间激活对 8 比特压缩更敏感。
|
||||
敏感度较低的层进行权重与激活量化（W8A8），而敏感度较高的层仅进行权重量化（W8A16）。
|
||||
|
||||
Here are the steps for applying this technique:
|
||||
以下是在其他模型版本上应用该技术的步骤：
|
||||
|
||||
**Step 1:** Generate calibration data
|
||||
**Step 1：** 生成校准数据
|
||||
|
||||
```python
|
||||
python -m python_coreml_stable_diffusion.activation_quantization --model-version <model-version> --generate-calibration-data -o <output-dir>
|
||||
```
|
||||
|
||||
A set of calibration text prompts are run through StableDiffusionPipeline and UNet model inputs are recorded and stored as pickle files in `calibration_data_<model-version>` folder inside specified output directory.
|
||||
一组校准文本提示词会经由 StableDiffusionPipeline 运行，UNet 模型输入会被记录并以 pickle 文件形式保存在指定输出目录内的 `calibration_data_<model-version>` 文件夹中。
|
||||
|
||||
**Step 2:** Run layer-wise sensitivity analysis
|
||||
**Step 2：** 运行逐层敏感度分析
|
||||
|
||||
```python
|
||||
python -m python_coreml_stable_diffusion.activation_quantization --model-version <model-version> --layerwise-sensitivity --calibration-nsamples <num-samples> -o <output-dir>
|
||||
```
|
||||
|
||||
This will run the analysis on all Convolutional and Attention (Einsum) modules in the model.
|
||||
For each module, a compressed version is generated by quantizing only that layer’s weights and activations.
|
||||
Then the PSNR between the outputs of the compressed and original model is calculated, using the same random seed and text prompts.
|
||||
这会对模型中所有卷积（Convolutional）和注意力（Attention，Einsum）模块运行分析。
|
||||
对每个模块，仅量化该层的权重与激活以生成压缩版本。
|
||||
然后使用相同的随机种子和文本提示词，计算压缩模型与原始模型输出之间的 PSNR。
|
||||
|
||||
This analysis takes up to a few hours on a single GPU (cuda). The number of calibration samples used to quantize the model can be reduced to speed up the process.
|
||||
该分析在单块 GPU（cuda）上最多可能需要数小时。可减少用于量化模型的校准样本数量以加快流程。
|
||||
|
||||
The resulting JSON file looks like this:
|
||||
生成的 JSON 文件如下所示：
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -304,56 +310,56 @@ The resulting JSON file looks like this:
|
||||
}
|
||||
```
|
||||
|
||||
**Step 3:** Generate quantized model
|
||||
**Step 3：** 生成量化模型
|
||||
|
||||
Using calibration data and layer-wise sensitivity the quantized CoreML model can be generated as follows:
|
||||
使用校准数据和逐层敏感度分析结果，可按如下方式生成量化 CoreML 模型：
|
||||
|
||||
```python
|
||||
python -m python_coreml_stable_diffusion.activation_quantization --model-version <model-version> --quantize-pytorch --conv-psnr 38 --attn-psnr 26 -o <output-dir>
|
||||
```
|
||||
|
||||
The PSNR thresholds determine which layers will be activation quantized. This number can be tuned to trade-off between output quality and inference latency.
|
||||
PSNR 阈值决定哪些层将进行激活量化。可调整该数值以在输出质量与推理延迟之间权衡。
|
||||
|
||||
</details>
|
||||
|
||||
## <a name="using-stable-diffusion-3"></a> Using Stable Diffusion 3
|
||||
|
||||
<details>
|
||||
<summary> Details (Click to expand) </summary>
|
||||
<summary> 详情（点击展开） </summary>
|
||||
|
||||
### Model Conversion
|
||||
|
||||
Stable Diffusion 3 uses some new and some old models to run. For the text encoders, the conversion can be done using a similar command as before with the `--sd3-version` flag.
|
||||
Stable Diffusion 3 运行时同时使用一些新模型与旧模型。对于文本编码器，可使用与之前类似的命令进行转换，并添加 `--sd3-version` 标志。
|
||||
|
||||
```bash
|
||||
python -m python_coreml_stable_diffusion.torch2coreml --model-version stabilityai/stable-diffusion-3-medium --bundle-resources-for-swift-cli --convert-text-encoder --sd3-version -o <output-dir>
|
||||
```
|
||||
|
||||
For the new models (MMDiT, a new VAE with 16 channels, and the T5 text encoder), there are a number of new CLI flags that utilize the [DiffusionKit](https://www.github.com/argmaxinc/DiffusionKit) repo:
|
||||
对于新模型（MMDiT、16 通道的新 VAE 以及 T5 文本编码器），有一系列新的 CLI 标志，它们利用了 [DiffusionKit](https://www.github.com/argmaxinc/DiffusionKit) 仓库：
|
||||
|
||||
- `--sd3-version`: Indicates to the converter to treat this as a Stable Diffusion 3 model
|
||||
- `--convert-mmdit`: Convert the MMDiT model
|
||||
- `--convert-vae-decoder`: Convert the new VAE model (this will use the 16 channel version if --sd3-version is set)
|
||||
- `--include-t5`: Downloads and includes a pre-converted T5 text encoder in the conversion
|
||||
- `--sd3-version`：指示转换器将其视为 Stable Diffusion 3 模型
|
||||
- `--convert-mmdit`：转换 MMDiT 模型
|
||||
- `--convert-vae-decoder`：转换新的 VAE 模型（若设置了 --sd3-version，将使用 16 通道版本）
|
||||
- `--include-t5`：下载并在转换中包含预转换的 T5 文本编码器
|
||||
|
||||
e.g.:
|
||||
e.g.：
|
||||
```bash
|
||||
python -m python_coreml_stable_diffusion.torch2coreml --model-version stabilityai/stable-diffusion-3-medium --bundle-resources-for-swift-cli --convert-vae-decoder --convert-mmdit --include-t5 --sd3-version -o <output-dir>
|
||||
```
|
||||
|
||||
To convert the full pipeline with at 1024x1024 resolution, the following command may be used:
|
||||
要以 1024x1024 分辨率转换完整流水线，可使用以下命令：
|
||||
|
||||
```bash
|
||||
python -m python_coreml_stable_diffusion.torch2coreml --model-version stabilityai/stable-diffusion-3-medium --bundle-resources-for-swift-cli --convert-text-encoder --convert-vae-decoder --convert-mmdit --include-t5 --sd3-version --latent-h 128 --latent-w 128 -o <output-dir>
|
||||
```
|
||||
|
||||
Keep in mind that the MMDiT model is quite large and will require increasingly more memory and time to convert as the latent resolution increases.
|
||||
请注意，MMDiT 模型体量较大，随着潜空间（latent）分辨率升高，转换所需的内存和时间会显著增加。
|
||||
|
||||
Also note that currently the MMDiT model requires fp32 and therefore only supports `CPU_AND_GPU` compute units and `ORIGINAL` attention implementation (the default for this pipeline).
|
||||
另外请注意，目前 MMDiT 模型需要 fp32，因此仅支持 `CPU_AND_GPU` 计算单元以及 `ORIGINAL` 注意力实现（本流水线的默认设置）。
|
||||
|
||||
### Swift Inference
|
||||
### Swift 推理
|
||||
|
||||
Swift inference for Stable Diffusion 3 is similar to the previous versions. The only difference is that the `--sd3` flag should be used to indicate that the model is a Stable Diffusion 3 model.
|
||||
Stable Diffusion 3 的 Swift 推理与之前版本类似。唯一区别是应使用 `--sd3` 标志，以表明该模型为 Stable Diffusion 3 模型。
|
||||
|
||||
```bash
|
||||
swift run StableDiffusionSample <prompt> --resource-path <output-mlpackages-directory/Resources> --output-path <output-dir> --compute-units cpuAndGPU --sd3
|
||||
@@ -361,110 +367,109 @@ swift run StableDiffusionSample <prompt> --resource-path <output-mlpackages-dire
|
||||
|
||||
</details>
|
||||
|
||||
## <a name="using-stable-diffusion-xl"></a> Using Stable Diffusion XL
|
||||
## <a name="using-stable-diffusion-xl"></a> 使用 Stable Diffusion XL
|
||||
|
||||
<details>
|
||||
<summary> Details (Click to expand) </summary>
|
||||
<summary> 详情（点击展开） </summary>
|
||||
|
||||
### Model Conversion
|
||||
### 模型转换
|
||||
|
||||
e.g.:
|
||||
例如：
|
||||
|
||||
```bash
|
||||
python -m python_coreml_stable_diffusion.torch2coreml --convert-unet --convert-vae-decoder --convert-text-encoder --xl-version --model-version stabilityai/stable-diffusion-xl-base-1.0 --refiner-version stabilityai/stable-diffusion-xl-refiner-1.0 --bundle-resources-for-swift-cli --attention-implementation {ORIGINAL,SPLIT_EINSUM} -o <output-dir>
|
||||
```
|
||||
|
||||
- `--xl-version`: Additional argument to pass to the conversion script when specifying an XL model
|
||||
- `--refiner-version`: Additional argument to pass to the conversion script when specifying an XL refiner model, required for ["Ensemble of Expert Denoisers"](https://huggingface.co/docs/diffusers/main/en/api/pipelines/stable_diffusion/stable_diffusion_xl#1-ensemble-of-expert-denoisers) inference.
|
||||
- `--attention-implementation`: `ORIGINAL` is recommended for `cpuAndGPU` for deployment on Mac
|
||||
- `--attention-implementation`: `SPLIT_EINSUM` is recommended for `cpuAndNeuralEngine` for deployment on iPhone & iPad
|
||||
- `--attention-implementation`: `SPLIT_EINSUM_V2` is not recommended for Stable Diffusion XL because of prohibitively long compilation time
|
||||
- **Tip:** Adding `--latent-h 96 --latent-w 96` is recommended for iOS and iPadOS deployment which leads to 768x768 generation as opposed to the default 1024x1024.
|
||||
- **Tip:** Due to known float16 overflow issues in the original Stable Diffusion XL VAE, [the model conversion script enforces float32 precision](https://github.com/apple/ml-stable-diffusion/blob/main/python_coreml_stable_diffusion/torch2coreml.py#L486). Using a custom VAE version such as [madebyollin/sdxl-vae-fp16-fix](https://huggingface.co/madebyollin/sdxl-vae-fp16-fix) by [@madebyollin](https://github.com/madebyollin) via `--custom-vae-version madebyollin/sdxl-vae-fp16-fix` will restore the default float16 precision for VAE.
|
||||
- `--xl-version`：在指定 XL 模型时传给转换脚本的额外参数
|
||||
- `--refiner-version`：在指定 XL refiner 模型时传给转换脚本的额外参数，["Ensemble of Expert Denoisers"](https://huggingface.co/docs/diffusers/main/en/api/pipelines/stable_diffusion/stable_diffusion_xl#1-ensemble-of-expert-denoisers) 推理时需要
|
||||
- `--attention-implementation`：在 Mac 上部署时，建议对 `cpuAndGPU` 使用 `ORIGINAL`
|
||||
- `--attention-implementation`：在 iPhone 与 iPad 上部署时，建议对 `cpuAndNeuralEngine` 使用 `SPLIT_EINSUM`
|
||||
- `--attention-implementation`：由于编译时间过长，不建议对 Stable Diffusion XL 使用 `SPLIT_EINSUM_V2`
|
||||
- **提示：** 建议为 iOS 与 iPadOS 部署添加 `--latent-h 96 --latent-w 96`，这样会以 768x768 生成，而非默认的 1024x1024。
|
||||
- **提示：** 由于原始 Stable Diffusion XL VAE 存在已知的 float16 溢出问题，[模型转换脚本会强制使用 float32 精度](https://github.com/apple/ml-stable-diffusion/blob/main/python_coreml_stable_diffusion/torch2coreml.py#L486). 通过 `--custom-vae-version madebyollin/sdxl-vae-fp16-fix` 使用自定义 VAE 版本（例如 [@madebyollin](https://github.com/madebyollin) 的 [madebyollin/sdxl-vae-fp16-fix](https://huggingface.co/madebyollin/sdxl-vae-fp16-fix)）可恢复 VAE 的默认 float16 精度。
|
||||
|
||||
### Swift Inference
|
||||
### Swift 推理
|
||||
|
||||
```bash
|
||||
swift run StableDiffusionSample <prompt> --resource-path <output-mlpackages-directory/Resources> --output-path <output-dir> --compute-units {cpuAndGPU,cpuAndNeuralEngine} --xl
|
||||
```
|
||||
- Only the `base` model is required, `refiner` model is optional and will be used by default if provided in the resource directory
|
||||
- ControlNet for XL is not yet supported
|
||||
- 仅需要 `base` 模型；`refiner` 模型为可选，若资源目录中提供则默认使用
|
||||
- XL 的 ControlNet 尚不支持
|
||||
|
||||
### Python Inference
|
||||
### Python 推理
|
||||
|
||||
```bash
|
||||
python -m python_coreml_stable_diffusion.pipeline --prompt <prompt> --compute-unit {CPU_AND_GPU,CPU_AND_NE} -o <output-dir> -i <output-mlpackages-directory/Resources> --model-version stabilityai/stable-diffusion-xl-base-1.0
|
||||
```
|
||||
- `refiner` model is not yet supported
|
||||
- ControlNet for XL is not yet supported
|
||||
- `refiner` 模型尚不支持
|
||||
- XL 的 ControlNet 尚不支持
|
||||
|
||||
</details>
|
||||
|
||||
## <a name="using-controlnet"></a> Using ControlNet
|
||||
## <a name="using-controlnet"></a> 使用 ControlNet
|
||||
|
||||
<details>
|
||||
<summary> Details (Click to expand) </summary>
|
||||
<summary> 详情（点击展开） </summary>
|
||||
|
||||
Example results using the prompt *"a high quality photo of a surfing dog"* conditioned on the scribble (leftmost):
|
||||
以下示例结果使用提示词 *"a high quality photo of a surfing dog"*，并基于涂鸦（scribble，最左侧）进行条件生成：
|
||||
|
||||
<img src="assets/controlnet_readme_reel.png">
|
||||
|
||||
[ControlNet](https://huggingface.co/lllyasviel/ControlNet) allows users to condition image generation with Stable Diffusion on signals such as edge maps, depth maps, segmentation maps, scribbles and pose. Thanks to [@ryu38's contribution](https://github.com/apple/ml-stable-diffusion/pull/153), both the Python CLI and the Swift package support ControlNet models. Please refer to [this section](#converting-models-to-coreml) for details on setting up Stable Diffusion with ControlNet.
|
||||
[ControlNet](https://huggingface.co/lllyasviel/ControlNet) 允许用户将 Stable Diffusion 的图像生成条件化到边缘图、深度图、分割图、涂鸦和姿态等信号上。得益于 [@ryu38 的贡献](https://github.com/apple/ml-stable-diffusion/pull/153),，Python CLI 与 Swift 包均支持 ControlNet 模型。关于如何将 Stable Diffusion 与 ControlNet 一起配置，请参阅[本节](#converting-models-to-coreml)。
|
||||
|
||||
Note that ControlNet is not yet supported for Stable Diffusion XL.
|
||||
请注意，ControlNet 目前尚不支持 Stable Diffusion XL。
|
||||
|
||||
</details>
|
||||
|
||||
## <a name="system-multilingual-text-encoder"></a> Using the System Multilingual Text Encoder
|
||||
## <a name="system-multilingual-text-encoder"></a> 使用系统多语言文本编码器
|
||||
|
||||
<details>
|
||||
<summary> Details (Click to expand) </summary>
|
||||
<summary> 详情（点击展开） </summary>
|
||||
|
||||
With iOS 17 and macOS 14, `NaturalLanguage` framework introduced the [NLContextualEmbedding](https://developer.apple.com/documentation/naturallanguage/nlcontextualembedding) which provides Transformer-based textual embeddings for Latin (20 languages), Cyrillic (4 languages) and CJK (3 languages) scripts. The WWDC23 session titled [Explore Natural Language multilingual models](https://developer.apple.com/videos/play/wwdc2023/10042) demonstrated how this powerful new model can be used by developers to train downstream tasks such as multilingual image generation with Stable Diffusion.
|
||||
在 iOS 17 与 macOS 14 中，`NaturalLanguage` 框架引入了 [NLContextualEmbedding](https://developer.apple.com/documentation/naturallanguage/nlcontextualembedding)，可为拉丁文（20 种语言）、西里尔文（4 种语言）以及 CJK（3 种语言）脚本提供基于 Transformer 的文本嵌入。WWDC23 专题演讲 [Explore Natural Language multilingual models](https://developer.apple.com/videos/play/wwdc2023/10042) 演示了开发者如何利用这一强大的新模型训练下游任务，例如基于 Stable Diffusion 的多语言图像生成。
|
||||
|
||||
The code to reproduce this demo workflow is made available in this repository. There are several ways in which this workflow can be implemented. Here is an example:
|
||||
本仓库提供了复现该演示工作流的代码。该工作流有多种实现方式。示例如下：
|
||||
|
||||
**Step 1:** Curate an image-text dataset with the desired languages.
|
||||
**步骤 1：** 整理包含目标语言的图文数据集。
|
||||
|
||||
**Step 2:** Pre-compute the NLContextualEmbedding values and replace the text strings with these embedding vectors in your dataset.
|
||||
**步骤 2：** 预先计算 NLContextualEmbedding 值，并用这些嵌入向量替换数据集中的文本字符串。
|
||||
|
||||
**Step 3:** Fine-tune a base model from Hugging Face Hub that is compatible with the [StableDiffusionPipeline](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/overview) by using your new dataset and replacing the default text_encoder with your pre-computed NLContextualEmbedding values.
|
||||
**步骤 3：** 从 Hugging Face Hub 选取与 [StableDiffusionPipeline](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/overview) 兼容的基座模型，使用你的新数据集进行微调，并用预先计算的 NLContextualEmbedding 值替换默认的 text_encoder。
|
||||
|
||||
**Step 4:** In order to be able to swap the text_encoder of a base model without training new layers, the base model's `text_encoder.hidden_size` must match that of NLContextualEmbedding. If it doesn't, you will need to train a linear projection layer to map between the two dimensionalities. After fine-tuning, this linear layer should be converted to CoreML as follows:
|
||||
**步骤 4：** 若要在不训练新层的情况下替换基座模型的 text_encoder，基座模型的 `text_encoder.hidden_size` 必须与 NLContextualEmbedding 一致。若不一致，则需要训练一个线性投影层以在两种维度之间映射。微调后，该线性层应按如下方式转换为 CoreML：
|
||||
|
||||
```shell
|
||||
python -m python_coreml_stable_diffusion.multilingual_projection --input-path <path-to-projection-torchscript> --output-dir <output-dir>
|
||||
```
|
||||
|
||||
The command above will yield a `MultilingualTextEncoderProjection.mlmodelc` file under `--output-dir` and this should be colocated with the rest of the Core ML model assets that were generated through `--bundle-resources-for-swift-cli`.
|
||||
上述命令会在 `--output-dir` 下生成 `MultilingualTextEncoderProjection.mlmodelc` 文件，该文件应与通过 `--bundle-resources-for-swift-cli` 生成的其余 Core ML 模型资源放在同一位置。
|
||||
|
||||
**Step 5:** The multilingual system text encoder can now be invoked by setting `useMultilingualTextEncoder` to true when initializing a pipeline or setting `--use-multilingual-text-encoder` in the CLI. Note that the model assets are distributed over-the-air so the first invocation will trigger asset downloads which is less than 100MB.
|
||||
**步骤 5：** 现在可通过在初始化流水线时将 `useMultilingualTextEncoder` 设为 true，或在 CLI 中设置 `--use-multilingual-text-encoder`，来调用多语言系统文本编码器。请注意，模型资源通过空中分发（over-the-air），首次调用会触发资源下载，体积小于 100MB。
|
||||
|
||||
|
||||
Resources:
|
||||
- [WWDC23 Session Video: Explore Natural Language multilingual models](https://developer.apple.com/videos/play/wwdc2023/10042)
|
||||
- [NLContextualEmbedding API Documentation](https://developer.apple.com/documentation/naturallanguage/nlcontextualembedding)
|
||||
资源：
|
||||
- [WWDC23 专题视频：Explore Natural Language multilingual models](https://developer.apple.com/videos/play/wwdc2023/10042)
|
||||
- [NLContextualEmbedding API 文档](https://developer.apple.com/documentation/naturallanguage/nlcontextualembedding)
|
||||
|
||||
</details>
|
||||
|
||||
## <a name="using-converted-weights"></a> Using Ready-made Core ML Models from Hugging Face Hub
|
||||
## <a name="using-converted-weights"></a> 使用 Hugging Face Hub 上现成的 Core ML 模型
|
||||
|
||||
<details>
|
||||
<summary> Click to expand </summary>
|
||||
<summary> 点击展开 </summary>
|
||||
|
||||
🤗 Hugging Face ran the [conversion procedure](#converting-models-to-coreml) on the following models and made the Core ML weights publicly available on the Hub. If you would like to convert a version of Stable Diffusion that is not already available on the Hub, please refer to the [Converting Models to Core ML](#converting-models-to-coreml).
|
||||
🤗 Hugging Face 已对以下模型执行了[转换流程](#converting-models-to-coreml)，并在 Hub 上公开提供了 Core ML 权重。若你希望转换 Hub 上尚不可用的 Stable Diffusion 版本，请参阅[将模型转换为 Core ML](#converting-models-to-coreml)。
|
||||
|
||||
* 6-bit quantized models (suitable for iOS 17 and macOS 14):
|
||||
* 6-bit 量化模型（适用于 iOS 17 与 macOS 14）：
|
||||
- [`CompVis/stable-diffusion-v1-4`](https://huggingface.co/apple/coreml-stable-diffusion-1-4-palettized)
|
||||
- [`runwayml/stable-diffusion-v1-5`](https://huggingface.co/apple/coreml-stable-diffusion-v1-5-palettized)
|
||||
- [`stabilityai/stable-diffusion-2-base`](https://huggingface.co/apple/coreml-stable-diffusion-2-base-palettized)
|
||||
- [`stabilityai/stable-diffusion-2-1-base`](https://huggingface.co/apple/coreml-stable-diffusion-2-1-base-palettized)
|
||||
|
||||
* Mixed-bit quantized models
|
||||
* 混合比特量化模型
|
||||
- [`stabilityai/stable-diffusion-xl-base-1.0`](https://huggingface.co/apple/coreml-stable-diffusion-mixed-bit-palettization)
|
||||
- [`stabilityai/stable-diffusion-xl-base-1.0-ios`](https://huggingface.co/apple/coreml-stable-diffusion-xl-base-ios)
|
||||
|
||||
* Uncompressed models:
|
||||
* 未压缩模型：
|
||||
- [`CompVis/stable-diffusion-v1-4`](https://huggingface.co/apple/coreml-stable-diffusion-v1-4)
|
||||
- [`runwayml/stable-diffusion-v1-5`](https://huggingface.co/apple/coreml-stable-diffusion-v1-5)
|
||||
- [`stabilityai/stable-diffusion-2-base`](https://huggingface.co/apple/coreml-stable-diffusion-2-base)
|
||||
@@ -473,29 +478,29 @@ Resources:
|
||||
- [`stabilityai/stable-diffusion-xl-{base+refiner}-1.0`](https://huggingface.co/apple/coreml-stable-diffusion-xl-base-with-refiner)
|
||||
- [`stabilityai/stable-diffusion-3-medium`](https://huggingface.co/stabilityai/stable-diffusion-3-medium)
|
||||
|
||||
If you want to use any of those models you may download the weights and proceed to [generate images with Python](#image-generation-with-python) or [Swift](#image-generation-with-swift).
|
||||
如果你想使用这些模型中的任意一个，可以下载权重，然后继续 [使用 Python 生成图像](#image-generation-with-python) 或 [使用 Swift 生成图像](#image-generation-with-swift)。
|
||||
|
||||
There are several variants in each model repository. You may clone the whole repos using `git` and `git lfs` to download all variants, or selectively download the ones you need.
|
||||
每个模型仓库中都有多个变体。你可以使用 `git` 和 `git lfs` 克隆整个仓库以下载所有变体，或选择性下载你需要的变体。
|
||||
|
||||
To clone the repos using `git`, please follow this process:
|
||||
要使用 `git` 克隆仓库，请按以下流程操作：
|
||||
|
||||
**Step 1:** Install the `git lfs` extension for your system.
|
||||
**步骤 1：** 为你的系统安装 `git lfs` 扩展。
|
||||
|
||||
`git lfs` stores large files outside the main git repo, and it downloads them from the appropriate server after you clone or checkout. It is available in most package managers, check [the installation page](https://git-lfs.com) for details.
|
||||
`git lfs` 将大文件存储在主 git 仓库之外，在你克隆或检出后从相应服务器下载这些文件。它在大多数包管理器中均可用，详情请查看[安装页面](https://git-lfs.com)。
|
||||
|
||||
**Step 2:** Enable `git lfs` by running this command once:
|
||||
**步骤 2：** 运行以下命令一次以启用 `git lfs`：
|
||||
|
||||
```bash
|
||||
git lfs install
|
||||
```
|
||||
|
||||
**Step 3:** Use `git clone` to download a copy of the repo that includes all model variants. For Stable Diffusion version 1.4, you'd issue the following command in your terminal:
|
||||
**步骤 3：** 使用 `git clone` 下载包含所有模型变体的仓库副本。对于 Stable Diffusion 1.4 版本，你可以在终端中执行以下命令：
|
||||
|
||||
```bash
|
||||
git clone https://huggingface.co/apple/coreml-stable-diffusion-v1-4
|
||||
```
|
||||
|
||||
If you prefer to download specific variants instead of cloning the repos, you can use the `huggingface_hub` Python library. For example, to do generation in Python using the `ORIGINAL` attention implementation (read [this section](#converting-models-to-coreml) for details), you could use the following helper code:
|
||||
如果你希望下载特定变体而不是克隆整个仓库，可以使用 `huggingface_hub` Python 库。例如，要在 Python 中使用 `ORIGINAL` attention 实现进行生成（详情请阅读[此节](#converting-models-to-coreml)），可以使用以下辅助代码：
|
||||
|
||||
```Python
|
||||
from huggingface_hub import snapshot_download
|
||||
@@ -509,16 +514,16 @@ snapshot_download(repo_id, allow_patterns=f"{variant}/*", local_dir=model_path,
|
||||
print(f"Model downloaded at {model_path}")
|
||||
```
|
||||
|
||||
`model_path` would be the path in your local filesystem where the checkpoint was saved. Please, refer to [this post](https://huggingface.co/blog/diffusers-coreml) for additional details.
|
||||
`model_path` 将是检查点保存在本地文件系统中的路径。更多详情，请参阅[此文章](https://huggingface.co/blog/diffusers-coreml)。
|
||||
|
||||
</details>
|
||||
|
||||
## <a name="converting-models-to-coreml"></a> Converting Models to Core ML
|
||||
## <a name="converting-models-to-coreml"></a> 将模型转换为 Core ML
|
||||
|
||||
<details>
|
||||
<summary> Click to expand </summary>
|
||||
<summary> 点击展开 </summary>
|
||||
|
||||
**Step 1:** Create a Python environment and install dependencies:
|
||||
**步骤 1：** 创建 Python 环境并安装依赖：
|
||||
|
||||
```bash
|
||||
conda create -n coreml_stable_diffusion python=3.8 -y
|
||||
@@ -527,81 +532,81 @@ cd /path/to/cloned/ml-stable-diffusion/repository
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
**Step 2:** Log in to or register for your [Hugging Face account](https://huggingface.co), generate a [User Access Token](https://huggingface.co/settings/tokens) and use this token to set up Hugging Face API access by running `huggingface-cli login` in a Terminal window.
|
||||
**步骤 2：** 登录或注册你的 [Hugging Face 账户](https://huggingface.co),，生成 [用户访问令牌（User Access Token）](https://huggingface.co/settings/tokens)，并在终端窗口中运行 `huggingface-cli login` 以使用该令牌配置 Hugging Face API 访问。
|
||||
|
||||
**Step 3:** Navigate to the version of Stable Diffusion that you would like to use on [Hugging Face Hub](https://huggingface.co/models?search=stable-diffusion) and accept its Terms of Use. The default model version is [CompVis/stable-diffusion-v1-4](https://huggingface.co/CompVis/stable-diffusion-v1-4). The model version may be changed by the user as described in the next step.
|
||||
**步骤 3：** 在 [Hugging Face Hub](https://huggingface.co/models?search=stable-diffusion) 上导航至你想使用的 Stable Diffusion 版本并接受其使用条款。默认模型版本为 [CompVis/stable-diffusion-v1-4](https://huggingface.co/CompVis/stable-diffusion-v1-4).。用户可按下一步所述更改模型版本。
|
||||
|
||||
**Step 4:** Execute the following command from the Terminal to generate Core ML model files (`.mlpackage`)
|
||||
**步骤 4：** 在终端中执行以下命令以生成 Core ML 模型文件（`.mlpackage`）
|
||||
|
||||
```shell
|
||||
python -m python_coreml_stable_diffusion.torch2coreml --convert-unet --convert-text-encoder --convert-vae-decoder --convert-safety-checker --model-version <model-version-string-from-hub> -o <output-mlpackages-directory>
|
||||
```
|
||||
|
||||
**WARNING:** This command will download several GB worth of PyTorch checkpoints from Hugging Face. Please ensure that you are on Wi-Fi and have enough disk space.
|
||||
**警告：** 此命令将从 Hugging Face 下载数 GB 的 PyTorch 检查点。请确保你使用的是 Wi-Fi 且拥有足够的磁盘空间。
|
||||
|
||||
This generally takes 15-20 minutes on an M1 MacBook Pro. Upon successful execution, the 4 neural network models that comprise Stable Diffusion will have been converted from PyTorch to Core ML (`.mlpackage`) and saved into the specified `<output-mlpackages-directory>`. Some additional notable arguments:
|
||||
在 M1 MacBook Pro 上通常需要 15-20 分钟。成功执行后，构成 Stable Diffusion 的 4 个神经网络模型将从 PyTorch 转换为 Core ML（`.mlpackage`），并保存到指定的 `<output-mlpackages-directory>`。其他一些值得注意的参数：
|
||||
|
||||
- `--model-version`: The model version name as published on the [Hugging Face Hub](https://huggingface.co/models?search=stable-diffusion)
|
||||
- `--model-version`：在 [Hugging Face Hub](https://huggingface.co/models?search=stable-diffusion) 上发布的模型版本名称
|
||||
|
||||
- `--refiner-version`: The refiner version name as published on the [Hugging Face Hub](https://huggingface.co/models?search=stable-diffusion). This is optional and if specified, this argument will convert and bundle the refiner unet alongside the model unet.
|
||||
- `--refiner-version`：在 [Hugging Face Hub](https://huggingface.co/models?search=stable-diffusion). 上发布的 refiner 版本名称。此参数为可选；若指定，将转换 refiner unet 并与模型 unet 一并打包。
|
||||
|
||||
- `--bundle-resources-for-swift-cli`: Compiles all 4 models and bundles them along with necessary resources for text tokenization into `<output-mlpackages-directory>/Resources` which should provided as input to the Swift package. This flag is not necessary for the diffusers-based Python pipeline. [However using these compiled models in Python will significantly speed up inference](https://apple.github.io/coremltools/docs-guides/source/model-prediction.html#why-use-a-compiled-model).
|
||||
- `--bundle-resources-for-swift-cli`：编译全部 4 个模型，并将文本分词所需资源一并打包到 `<output-mlpackages-directory>/Resources` 中，该文件应作为 Swift 包的输入。对于基于 diffusers 的 Python 流水线，此标志并非必需。[不过，在 Python 中使用这些已编译模型将显著加快推理速度](https://apple.github.io/coremltools/docs-guides/source/model-prediction.html#why-use-a-compiled-model).
|
||||
|
||||
- `--quantize-nbits`: Quantizes the weights of unet and text_encoder models down to 2, 4, 6 or 8 bits using a globally optimal k-means clustering algorithm. By default all models are weight-quantized to 16 bits even if this argument is not specified. Please refer to [this section](#compression-6-bits-and-higher for details and further guidance on weight compression.
|
||||
- `--quantize-nbits`：使用全局最优 k-means 聚类算法将 unet 和 text_encoder 模型的权重量化至 2、4、6 或 8 比特。默认情况下，即使未指定此参数，所有模型也会被权重量化至 16 比特。详情请参阅[此节](#compression-6-bits-and-higher)及关于权重压缩的更多说明。
|
||||
|
||||
- `--chunk-unet`: Splits the Unet model in two approximately equal chunks (each with less than 1GB of weights) for mobile-friendly deployment. This is **required** for Neural Engine deployment on iOS and iPadOS if weights are not quantized to 6-bits or less (`--quantize-nbits {2,4,6}`). This is not required for macOS. Swift CLI is able to consume both the chunked and regular versions of the Unet model but prioritizes the former. Note that chunked unet is not compatible with the Python pipeline because Python pipeline is intended for macOS only.
|
||||
- `--chunk-unet`：将 Unet 模型拆分为两个大致相等的块（每块权重均小于 1GB），以便在移动端部署。若权重未量化至 6 比特或更低（`--quantize-nbits {2,4,6}`），在 iOS 和 iPadOS 上部署到 Neural Engine 时**必须**使用此选项。macOS 上不需要。Swift CLI 可使用分块版和普通版 Unet 模型，但会优先使用前者。请注意，分块 unet 与 Python 流水线不兼容，因为 Python 流水线仅面向 macOS。
|
||||
|
||||
- `--attention-implementation`: Defaults to `SPLIT_EINSUM` which is the implementation described in [Deploying Transformers on the Apple Neural Engine](https://machinelearning.apple.com/research/neural-engine-transformers). `--attention-implementation SPLIT_EINSUM_V2` yields 10-30% improvement for mobile devices, still targeting the Neural Engine. `--attention-implementation ORIGINAL` will switch to an alternative implementation that should be used for CPU or GPU deployment on some Mac devices. Please refer to the [Performance Benchmark](#performance-benchmark) section for further guidance.
|
||||
- `--attention-implementation`：默认为 `SPLIT_EINSUM`，即 [Deploying Transformers on the Apple Neural Engine](https://machinelearning.apple.com/research/neural-engine-transformers). 中描述的实现。`--attention-implementation SPLIT_EINSUM_V2` 可为移动设备带来 10-30% 的性能提升，仍面向 Neural Engine。`--attention-implementation ORIGINAL` 将切换到另一种实现，适用于部分 Mac 设备上的 CPU 或 GPU 部署。更多说明请参阅 [性能基准测试](#performance-benchmark) 一节。
|
||||
|
||||
- `--check-output-correctness`: Compares original PyTorch model's outputs to final Core ML model's outputs. This flag increases RAM consumption significantly so it is recommended only for debugging purposes.
|
||||
- `--check-output-correctness`：将原始 PyTorch 模型的输出与最终 Core ML 模型的输出进行对比。此标志会显著增加 RAM 占用，因此建议仅用于调试。
|
||||
|
||||
- `--convert-controlnet`: Converts ControlNet models specified after this option. This can also convert multiple models if you specify like `--convert-controlnet lllyasviel/sd-controlnet-mlsd lllyasviel/sd-controlnet-depth`.
|
||||
- `--convert-controlnet`：转换此选项后指定的 ControlNet 模型。若像 `--convert-controlnet lllyasviel/sd-controlnet-mlsd lllyasviel/sd-controlnet-depth` 这样指定，也可转换多个模型。
|
||||
|
||||
- `--unet-support-controlnet`: enables a converted UNet model to receive additional inputs from ControlNet. This is required for generating image with using ControlNet and saved with a different name, `*_control-unet.mlpackage`, distinct from normal UNet. On the other hand, this UNet model can not work without ControlNet. Please use normal UNet for just txt2img.
|
||||
- `--unet-support-controlnet`：使转换后的 UNet 模型能够接收来自 ControlNet 的额外输入。使用 ControlNet 生成图像时需要此选项，并会以不同名称 `*_control-unet.mlpackage` 保存，与普通 UNet 不同。另一方面，该 UNet 模型在没有 ControlNet 的情况下无法工作。若仅需 txt2img，请使用普通 UNet。
|
||||
|
||||
- `--unet-batch-one`: use a batch size of one for the unet, this is needed if you do not want to do classifier free guidance, i.e. using a `guidance-scale` of less than one.
|
||||
- `--unet-batch-one`：为 unet 使用批次大小（batch size）为 1；若你不想进行无分类器引导（classifier free guidance），即使用小于 1 的 `guidance-scale`，则需要此设置。
|
||||
|
||||
- `--convert-vae-encoder`: not required for text-to-image applications. Required for image-to-image applications in order to map the input image to the latent space.
|
||||
- `--convert-vae-encoder`：文生图（text-to-image）应用不需要。图生图（image-to-image）应用需要，以便将输入图像映射到潜空间（latent space）。
|
||||
|
||||
</details>
|
||||
|
||||
## <a name="image-generation-with-python"></a> Image Generation with Python
|
||||
## <a name="image-generation-with-python"></a> 使用 Python 进行图像生成
|
||||
|
||||
<details>
|
||||
<summary> Click to expand </summary>
|
||||
<summary> 点击展开 </summary>
|
||||
|
||||
Run text-to-image generation using the example Python pipeline based on [diffusers](https://github.com/huggingface/diffusers):
|
||||
使用基于 [diffusers](https://github.com/huggingface/diffusers): 的示例 Python 流水线运行文生图生成。
|
||||
|
||||
```shell
|
||||
python -m python_coreml_stable_diffusion.pipeline --prompt "a photo of an astronaut riding a horse on mars" -i <core-ml-model-directory> -o </path/to/output/image> --compute-unit ALL --seed 93
|
||||
```
|
||||
Please refer to the help menu for all available arguments: `python -m python_coreml_stable_diffusion.pipeline -h`. Some notable arguments:
|
||||
请参阅帮助菜单了解所有可用参数：`python -m python_coreml_stable_diffusion.pipeline -h`。一些值得注意的参数：
|
||||
|
||||
- `-i`: Should point to the `-o` directory from Step 4 of [Converting Models to Core ML](#converting-models-to-coreml) section from above. If you specified `--bundle-resources-for-swift-cli` during conversion, then use the resulting `Resources` folder (which holds the compiled `.mlmodelc` files). [The compiled models load much faster after first use](https://apple.github.io/coremltools/docs-guides/source/model-prediction.html#why-use-a-compiled-model).
|
||||
- `--model-version`: If you overrode the default model version while converting models to Core ML, you will need to specify the same model version here.
|
||||
- `--compute-unit`: Note that the most performant compute unit for this particular implementation may differ across different hardware. `CPU_AND_GPU` or `CPU_AND_NE` may be faster than `ALL`. Please refer to the [Performance Benchmark](#performance-benchmark) section for further guidance.
|
||||
- `--scheduler`: If you would like to experiment with different schedulers, you may specify it here. For available options, please see the help menu. You may also specify a custom number of inference steps by `--num-inference-steps` which defaults to 50.
|
||||
- `--controlnet`: ControlNet models specified with this option are used in image generation. Use this option in the format `--controlnet lllyasviel/sd-controlnet-mlsd lllyasviel/sd-controlnet-depth` and make sure to use `--controlnet-inputs` in conjunction.
|
||||
- `--controlnet-inputs`: Image inputs corresponding to each ControlNet model. Please provide image paths in same order as models in `--controlnet`, for example: `--controlnet-inputs image_mlsd image_depth`.
|
||||
- `--unet-batch-one`: Do not batch unet predictions for the prompt and negative prompt. This requires the unet has been converted with a batch size of one, see `--unet-batch-one` option in conversion script.
|
||||
- `-i`：应指向上文 [将模型转换为 Core ML](#converting-models-to-coreml) 章节第 4 步中的 `-o` 目录。若在转换期间指定了 `--bundle-resources-for-swift-cli`，则使用生成的 `Resources` 文件夹（其中包含已编译的 `.mlmodelc` 文件）。[编译后的模型在首次使用后加载会快得多](https://apple.github.io/coremltools/docs-guides/source/model-prediction.html#why-use-a-compiled-model).
|
||||
- `--model-version`：若在将模型转换为 Core ML 时覆盖了默认模型版本，则需要在此处指定相同的模型版本。
|
||||
- `--compute-unit`：请注意，对于此特定实现，性能最佳的计算单元可能因硬件而异。`CPU_AND_GPU` 或 `CPU_AND_NE` 可能比 `ALL` 更快。请参阅[性能基准测试](#performance-benchmark)章节以获取更多指导。
|
||||
- `--scheduler`：若想尝试不同的调度器（scheduler），可在此处指定。可用选项请参阅帮助菜单。你还可以通过 `--num-inference-steps` 指定自定义推理步数，默认值为 50。
|
||||
- `--controlnet`：通过此选项指定的 ControlNet 模型用于图像生成。请按 `--controlnet lllyasviel/sd-controlnet-mlsd lllyasviel/sd-controlnet-depth` 格式使用此选项，并确保与 `--controlnet-inputs` 配合使用。
|
||||
- `--controlnet-inputs`：与各 ControlNet 模型对应的图像输入。请按 `--controlnet` 中模型的相同顺序提供图像路径，例如：`--controlnet-inputs image_mlsd image_depth`。
|
||||
- `--unet-batch-one`：不要对提示词和负向提示词的 unet 预测进行批处理。这要求 unet 在转换时批次大小为 1，请参阅转换脚本中的 `--unet-batch-one` 选项。
|
||||
|
||||
</details>
|
||||
|
||||
## <a name="image-gen-swift"></a> Image Generation with Swift
|
||||
## <a name="image-gen-swift"></a> 使用 Swift 进行图像生成
|
||||
|
||||
<details>
|
||||
<summary> Click to expand </summary>
|
||||
<summary> 点击展开 </summary>
|
||||
|
||||
### Example CLI Usage
|
||||
### 示例 CLI 用法
|
||||
```shell
|
||||
swift run StableDiffusionSample "a photo of an astronaut riding a horse on mars" --resource-path <output-mlpackages-directory>/Resources/ --seed 93 --output-path </path/to/output/image>
|
||||
```
|
||||
The output will be named based on the prompt and random seed:
|
||||
e.g. `</path/to/output/image>/a_photo_of_an_astronaut_riding_a_horse_on_mars.93.final.png`
|
||||
输出文件名将基于提示词和随机种子命名：
|
||||
例如 `</path/to/output/image>/a_photo_of_an_astronaut_riding_a_horse_on_mars.93.final.png`
|
||||
|
||||
Please use the `--help` flag to learn about batched generation and more.
|
||||
请使用 `--help` 标志了解批量生成等更多信息。
|
||||
|
||||
### Example Library Usage
|
||||
### 示例库用法
|
||||
|
||||
```swift
|
||||
import StableDiffusion
|
||||
@@ -610,82 +615,82 @@ let pipeline = try StableDiffusionPipeline(resourcesAt: resourceURL)
|
||||
pipeline.loadResources()
|
||||
let image = try pipeline.generateImages(prompt: prompt, seed: seed).first
|
||||
```
|
||||
On iOS, the `reduceMemory` option should be set to `true` when constructing `StableDiffusionPipeline`
|
||||
在 iOS 上，构造 `StableDiffusionPipeline` 时，应将 `reduceMemory` 选项设置为 `true`
|
||||
|
||||
### Swift Package Details
|
||||
### Swift 软件包详情
|
||||
|
||||
This Swift package contains two products:
|
||||
此 Swift 软件包包含两个产品：
|
||||
|
||||
- `StableDiffusion` library
|
||||
- `StableDiffusionSample` command-line tool
|
||||
- `StableDiffusion` 库
|
||||
- `StableDiffusionSample` 命令行工具
|
||||
|
||||
Both of these products require the Core ML models and tokenization resources to be supplied. When specifying resources via a directory path that directory must contain the following:
|
||||
这两个产品都需要提供 Core ML 模型和分词（tokenization）资源。通过目录路径指定资源时，该目录必须包含以下内容：
|
||||
|
||||
- `TextEncoder.mlmodelc` or `TextEncoder2.mlmodelc (text embedding model)
|
||||
- `Unet.mlmodelc` or `UnetChunk1.mlmodelc` & `UnetChunk2.mlmodelc` (denoising autoencoder model)
|
||||
- `VAEDecoder.mlmodelc` (image decoder model)
|
||||
- `vocab.json` (tokenizer vocabulary file)
|
||||
- `merges.text` (merges for byte pair encoding file)
|
||||
- `TextEncoder.mlmodelc` 或 `TextEncoder2.mlmodelc（文本嵌入模型）
|
||||
- `Unet.mlmodelc` 或 `UnetChunk1.mlmodelc` 与 `UnetChunk2.mlmodelc`（去噪自编码器模型）
|
||||
- `VAEDecoder.mlmodelc`（图像解码器模型）
|
||||
- `vocab.json`（分词器词汇表文件）
|
||||
- `merges.text`（字节对编码（byte pair encoding）合并文件）
|
||||
|
||||
Optionally, for image2image, in-painting, or similar:
|
||||
可选，用于图生图、修复（in-painting）或类似场景：
|
||||
|
||||
- `VAEEncoder.mlmodelc` (image encoder model)
|
||||
- `VAEEncoder.mlmodelc`（图像编码器模型）
|
||||
|
||||
Optionally, it may also include the safety checker model that some versions of Stable Diffusion include:
|
||||
可选，还可包含某些版本 Stable Diffusion 所包含的安全检查器（safety checker）模型：
|
||||
|
||||
- `SafetyChecker.mlmodelc`
|
||||
|
||||
Optionally, for the SDXL refiner:
|
||||
可选，用于 SDXL refiner：
|
||||
|
||||
- `UnetRefiner.mlmodelc` (refiner unet model)
|
||||
- `UnetRefiner.mlmodelc`（refiner unet 模型）
|
||||
|
||||
Optionally, for ControlNet:
|
||||
可选，用于 ControlNet：
|
||||
|
||||
- `ControlledUNet.mlmodelc` or `ControlledUnetChunk1.mlmodelc` & `ControlledUnetChunk2.mlmodelc` (enabled to receive ControlNet values)
|
||||
- `controlnet/` (directory containing ControlNet models)
|
||||
- `LllyasvielSdControlnetMlsd.mlmodelc` (for example, from lllyasviel/sd-controlnet-mlsd)
|
||||
- `LllyasvielSdControlnetDepth.mlmodelc` (for example, from lllyasviel/sd-controlnet-depth)
|
||||
- Other models you converted
|
||||
- `ControlledUNet.mlmodelc` 或 `ControlledUnetChunk1.mlmodelc` 与 `ControlledUnetChunk2.mlmodelc`（已启用以接收 ControlNet 值）
|
||||
- `controlnet/`（包含 ControlNet 模型的目录）
|
||||
- `LllyasvielSdControlnetMlsd.mlmodelc`（例如，来自 lllyasviel/sd-controlnet-mlsd）
|
||||
- `LllyasvielSdControlnetDepth.mlmodelc`（例如，来自 lllyasviel/sd-controlnet-depth）
|
||||
- 你转换的其他模型
|
||||
|
||||
Note that the chunked version of Unet is checked for first. Only if it is not present will the full `Unet.mlmodelc` be loaded. Chunking is required for iOS and iPadOS and not necessary for macOS.
|
||||
请注意，会首先检查 Unet 的分块（chunked）版本。仅当其不存在时才会加载完整的 `Unet.mlmodelc`。iOS 和 iPadOS 需要分块，macOS 则不需要。
|
||||
|
||||
</details>
|
||||
|
||||
## <a name="swift-app"></a> Example Swift App
|
||||
## <a name="swift-app"></a> 示例 Swift 应用
|
||||
|
||||
<details>
|
||||
<summary> Click to expand </summary>
|
||||
<summary> 点击展开 </summary>
|
||||
|
||||
🤗 Hugging Face created an [open-source demo app](https://github.com/huggingface/swift-coreml-diffusers) on top of this library. It's written in native Swift and Swift UI, and runs on macOS, iOS and iPadOS. You can use the code as a starting point for your app, or to see how to integrate this library in your own projects.
|
||||
🤗 Hugging Face 在此库之上创建了一个[开源演示应用](https://github.com/huggingface/swift-coreml-diffusers)。它使用原生 Swift 和 SwiftUI 编写，可在 macOS、iOS 和 iPadOS 上运行。你可以将代码作为应用的起点，或了解如何在自己的项目中集成此库。
|
||||
|
||||
Hugging Face has made the app [available in the Mac App Store](https://apps.apple.com/app/diffusers/id1666309574?mt=12).
|
||||
Hugging Face 已将此应用[上架 Mac App Store](https://apps.apple.com/app/diffusers/id1666309574?mt=12).
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
## <a name="faq"></a> FAQ
|
||||
## <a name="faq"></a> 常见问题（FAQ）
|
||||
|
||||
<details>
|
||||
<summary> Click to expand </summary>
|
||||
<summary> 点击展开 </summary>
|
||||
<details>
|
||||
|
||||
|
||||
<summary> <b> Q1: </b> <code> ERROR: Failed building wheel for tokenizers or error: can't find Rust compiler </code> </summary>
|
||||
|
||||
<b> A1: </b> Please review this [potential solution](https://github.com/huggingface/transformers/issues/2831#issuecomment-592724471).
|
||||
<b> A1: </b> 请查看此[潜在解决方案](https://github.com/huggingface/transformers/issues/2831#issuecomment-592724471).
|
||||
</details>
|
||||
|
||||
|
||||
<details>
|
||||
<summary> <b> Q2: </b> <code> RuntimeError: {NSLocalizedDescription = "Error computing NN outputs." </code> </summary>
|
||||
|
||||
<b> A2: </b> There are many potential causes for this error. In this context, it is highly likely to be encountered when your system is under increased memory pressure from other applications. Reducing memory utilization of other applications is likely to help alleviate the issue.
|
||||
<b> A2: </b> 此错误可能有许多原因。在此场景下，当你的系统因其他应用而面临更大的内存压力时，很可能遇到该错误。降低其他应用的内存占用可能有助于缓解此问题。
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary> <b> <a name="low-mem-conversion"></a> Q3: </b> My Mac has 8GB RAM and I am converting models to Core ML using the example command. The process is getting killed because of memory issues. How do I fix this issue? </summary>
|
||||
<summary> <b> <a name="low-mem-conversion"></a> Q3: </b> 我的 Mac 有 8GB 内存，我正在使用示例命令将模型转换为 Core ML。进程因内存问题被终止。如何解决这个问题？ </summary>
|
||||
|
||||
<b> A3: </b> In order to minimize the memory impact of the model conversion process, please execute the following command instead:
|
||||
<b> A3: </b> 为尽量降低模型转换过程的内存占用，请改为执行以下命令：
|
||||
|
||||
```bash
|
||||
python -m python_coreml_stable_diffusion.torch2coreml --convert-vae-encoder --model-version <model-version-string-from-hub> -o <output-mlpackages-directory> && \
|
||||
@@ -695,7 +700,7 @@ python -m python_coreml_stable_diffusion.torch2coreml --convert-text-encoder --m
|
||||
python -m python_coreml_stable_diffusion.torch2coreml --convert-safety-checker --model-version <model-version-string-from-hub> -o <output-mlpackages-directory> &&
|
||||
```
|
||||
|
||||
If you need `--chunk-unet`, you may do so in yet another independent command which will reuse the previously exported Unet model and simply chunk it in place:
|
||||
若你需要 `--chunk-unet`，可以在另一个独立命令中完成，该命令会复用先前导出的 Unet 模型并就地对其进行分块：
|
||||
|
||||
```bash
|
||||
python -m python_coreml_stable_diffusion.torch2coreml --convert-unet --chunk-unet -o <output-mlpackages-directory>
|
||||
@@ -704,91 +709,91 @@ python -m python_coreml_stable_diffusion.torch2coreml --convert-unet --chunk-une
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary> <b> Q4: </b> My Mac has 8GB RAM, should image generation work on my machine? </summary>
|
||||
<summary> <b> Q4: </b> 我的 Mac 只有 8GB 内存，图像生成能在我的机器上运行吗？ </summary>
|
||||
|
||||
<b> A4: </b> Yes! Especially the `--compute-unit CPU_AND_NE` option should work under reasonable system load from other applications. Note that part of the [Example Results](#example-results) were generated using an M2 MacBook Air with 8GB RAM.
|
||||
<b> A4: </b> 可以！尤其是 `--compute-unit CPU_AND_NE` 选项，在其他应用程序负载合理的情况下应该可以正常工作。请注意，[示例结果](#example-results) 中的部分内容是在配备 8GB 内存的 M2 MacBook Air 上生成的。
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary> <b> Q5: </b> Every time I generate an image using the Python pipeline, loading all the Core ML models takes 2-3 minutes. Is this expected? </summary>
|
||||
<summary> <b> Q5: </b> 每次使用 Python 流水线生成图像时，加载所有 Core ML 模型都需要 2-3 分钟。这是正常现象吗？ </summary>
|
||||
|
||||
<b> A5: </b> Both `.mlpackage` and `.mlmodelc` models are compiled (also known as "model preparation" in Core ML terms) upon first load when a specific compute unit is specified. `.mlpackage` does not cache this compiled asset so each model load retriggers this compilation which may take up to a few minutes. On the other hand, `.mlmodelc` files do cache this compiled asset and non-first load times are reduced to just a few seconds.
|
||||
<b> A5: </b> 在指定特定计算单元（compute unit）后，`.mlpackage` 和 `.mlmodelc` 模型在首次加载时都会进行编译（在 Core ML 术语中也称为“模型准备”，model preparation）。`.mlpackage` 不会缓存此编译产物，因此每次加载模型都会重新触发编译，可能耗时数分钟。另一方面，`.mlmodelc` 文件会缓存此编译产物，非首次加载时间可缩短至仅数秒。
|
||||
|
||||
In order to benefit from compilation caching, you may use the `.mlmodelc` assets instead of `.mlpackage` assets in both Swift (default) and Python (possible thanks to [@lopez-hector](https://github.com/lopez-hector)'s [contribution](https://github.com/apple/ml-stable-diffusion/commit/f3a212491cf531dd88493c89ad3d98d016db407f)) image generation pipelines.
|
||||
若要受益于编译缓存，你可以在 Swift（默认）和 Python（得益于 [@lopez-hector](https://github.com/lopez-hector)'s [contribution](https://github.com/apple/ml-stable-diffusion/commit/f3a212491cf531dd88493c89ad3d98d016db407f)) 图像生成流水线）中，使用 `.mlmodelc` 资源替代 `.mlpackage` 资源。
|
||||
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
<details>
|
||||
<summary> <b> <a name="q-mobile-app"></a> Q6: </b> I want to deploy <code>StableDiffusion</code>, the Swift package, in my mobile app. What should I be aware of? </summary>
|
||||
<summary> <b> <a name="q-mobile-app"></a> Q6: </b> 我想在移动应用中部署 <code>StableDiffusion</code> 这一 Swift 软件包。需要注意什么？ </summary>
|
||||
|
||||
<b> A6: </b>The [Image Generation with Swift](#image-gen-swift) section describes the minimum SDK and OS versions as well as the device models supported by this package. We recommend carefully testing the package on the device with the least amount of RAM available among your deployment targets.
|
||||
<b> A6: </b>[使用 Swift 进行图像生成](#image-gen-swift) 一节介绍了本软件包所需的最低 SDK 和操作系统版本，以及支持的设备型号。我们建议在部署目标中内存最少的设备上仔细测试该软件包。
|
||||
|
||||
The image generation process in `StableDiffusion` can yield over 2 GB of peak memory during runtime depending on the compute units selected. On iPadOS, we recommend using `.cpuAndNeuralEngine` in your configuration and the `reduceMemory` option when constructing a `StableDiffusionPipeline` to minimize memory pressure.
|
||||
`StableDiffusion` 中的图像生成过程在运行时，根据所选计算单元，峰值内存可能超过 2 GB。在 iPadOS 上，我们建议在配置中使用 `.cpuAndNeuralEngine`，并在构建 `StableDiffusionPipeline` 时选用 `reduceMemory` 选项，以尽量降低内存压力。
|
||||
|
||||
If your app crashes during image generation, consider adding the [Increased Memory Limit](https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_developer_kernel_increased-memory-limit) capability to inform the system that some of your app’s core features may perform better by exceeding the default app memory limit on supported devices.
|
||||
若应用在图像生成期间崩溃，可考虑添加 [Increased Memory Limit](https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_developer_kernel_increased-memory-limit) 能力，以告知系统：在支持的设备上，应用的部分核心功能在超出默认内存限制时可能运行得更好。
|
||||
|
||||
On iOS, depending on the iPhone model, Stable Diffusion model versions, selected compute units, system load and design of your app, this may still not be sufficient to keep your apps peak memory under the limit. Please remember, because the device shares memory between apps and iOS processes, one app using too much memory can compromise the user experience across the whole device.
|
||||
在 iOS 上，根据 iPhone 型号、Stable Diffusion 模型版本、所选计算单元、系统负载以及应用设计，这可能仍不足以将应用的峰值内存控制在限制以内。请记住，由于设备在应用与 iOS 进程之间共享内存，单个应用占用过多内存可能影响整台设备的使用体验。
|
||||
|
||||
We **strongly recommend** compressing your models following the recipes in [Advanced Weight Compression (Lower than 6-bits)](#compression-lower-than-6-bits) for iOS deployment. This reduces the peak RAM usage by up to 75% (from 16-bit to 4-bit) while preserving model output quality.
|
||||
我们**强烈建议**按照[高级权重压缩（低于 6 位）](#compression-lower-than-6-bits) 中的方案压缩模型，以便在 iOS 上部署。这可将峰值 RAM 占用最多降低 75%（从 16 位降至 4 位），同时保持模型输出质量。
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary> <b> Q7: </b> How do I generate images with different resolutions using the same Core ML models? </summary>
|
||||
<summary> <b> Q7: </b> 如何使用同一套 Core ML 模型生成不同分辨率的图像？ </summary>
|
||||
|
||||
<b> A7: </b> The current version of `python_coreml_stable_diffusion` does not support single-model multi-resolution out of the box. However, developers may fork this project and leverage the [flexible shapes](https://coremltools.readme.io/docs/flexible-inputs) support from coremltools to extend the `torch2coreml` script by using `coremltools.EnumeratedShapes`. Note that, while the `text_encoder` is agnostic to the image resolution, the inputs and outputs of `vae_decoder` and `unet` models are dependent on the desired image resolution.
|
||||
<b> A7: </b> 当前版本的 `python_coreml_stable_diffusion` 并不开箱即用地支持单模型多分辨率。不过，开发者可以 fork 本项目，并利用 coremltools 的[灵活形状](https://coremltools.readme.io/docs/flexible-inputs) 支持，通过 `coremltools.EnumeratedShapes` 扩展 `torch2coreml` 脚本。请注意，虽然 `text_encoder` 与图像分辨率无关，但 `vae_decoder` 和 `unet` 模型的输入与输出取决于目标图像分辨率。
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary> <b> Q8: </b> Are the Core ML and PyTorch generated images going to be identical? </summary>
|
||||
<summary> <b> Q8: </b> Core ML 与 PyTorch 生成的图像会完全相同吗？ </summary>
|
||||
|
||||
<b> A8: </b> If desired, the generated images across PyTorch and Core ML can be made approximately identical. However, it is not guaranteed by default. There are several factors that might lead to different images across PyTorch and Core ML:
|
||||
<b> A8: </b> 如有需要，PyTorch 与 Core ML 生成的图像可以做到近似一致。但默认情况下并不保证如此。以下若干因素可能导致 PyTorch 与 Core ML 生成不同图像：
|
||||
|
||||
|
||||
<b> 1. Random Number Generator Behavior </b>
|
||||
<b> 1. 随机数生成器行为 </b>
|
||||
|
||||
The main source of potentially different results across PyTorch and Core ML is the Random Number Generator ([RNG](https://en.wikipedia.org/wiki/Random_number_generation)) behavior. PyTorch and Numpy have different sources of randomness. `python_coreml_stable_diffusion` generally relies on Numpy for RNG (e.g. latents initialization) and `StableDiffusion` Swift Library reproduces this RNG behavior by default. However, PyTorch-based pipelines such as Hugging Face `diffusers` relies on PyTorch's RNG behavior. Thanks to @liuliu's [contributions](https://github.com/apple/ml-stable-diffusion/pull/124), one can match the PyTorch (CPU/GPU) RNG behavior in Swift by specifying `--rng torch/cuda` which selects the `torchRNG/cudaRNG` mode.
|
||||
PyTorch 与 Core ML 结果可能不同的主要来源是随机数生成器（[RNG](https://en.wikipedia.org/wiki/Random_number_generation)) 行为。PyTorch 与 Numpy 的随机性来源不同。`python_coreml_stable_diffusion` 通常依赖 Numpy 进行 RNG（例如潜变量初始化），而 `StableDiffusion` Swift 库默认会复现该 RNG 行为。不过，基于 PyTorch 的流水线（如 Hugging Face `diffusers`）依赖 PyTorch 的 RNG 行为。得益于 @liuliu 的[贡献](https://github.com/apple/ml-stable-diffusion/pull/124),，可通过指定 `--rng torch/cuda` 来选择 `torchRNG/cudaRNG` 模式，从而在 Swift 中匹配 PyTorch（CPU/GPU）的 RNG 行为。
|
||||
|
||||
<b> 2. PyTorch </b>
|
||||
|
||||
*"Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms. Furthermore, results may not be reproducible between CPU and GPU executions, even when using identical seeds."* ([source](https://pytorch.org/docs/stable/notes/randomness.html#reproducibility)).
|
||||
*“无法保证在 PyTorch 不同版本、各个提交或不同平台之间得到完全可复现的结果。此外，即使使用相同种子，CPU 与 GPU 执行的结果也可能不可复现。”*（[来源](https://pytorch.org/docs/stable/notes/randomness.html#reproducibility)).
|
||||
|
||||
<b> 3. Model Function Drift During Conversion </b>
|
||||
<b> 3. 转换过程中的模型函数漂移 </b>
|
||||
|
||||
The difference in outputs across corresponding PyTorch and Core ML models is a potential cause. The signal integrity is tested during the conversion process (enabled via `--check-output-correctness` argument to `python_coreml_stable_diffusion.torch2coreml`) and it is verified to be above a minimum [PSNR](https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio) value as tested on random inputs. Note that this is simply a sanity check and does not guarantee this minimum PSNR across all possible inputs. Furthermore, the results are not guaranteed to be identical when executing the same Core ML models across different compute units. This is not expected to be a major source of difference as the sample visual results indicate in [this section](#compression-6-bits-and-higher).
|
||||
对应 PyTorch 与 Core ML 模型输出差异是潜在原因之一。转换过程中会测试信号完整性（通过向 `python_coreml_stable_diffusion.torch2coreml` 传入 `--check-output-correctness` 参数启用），并在随机输入上验证其高于最低 [PSNR](https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio) 值。请注意，这仅是健全性检查，并不能保证所有可能输入都达到该最低 PSNR。此外，在不同计算单元上执行同一 Core ML 模型时，结果也不保证完全相同。如[本节](#compression-6-bits-and-higher) 中的示例视觉结果所示，这通常不应成为主要差异来源。
|
||||
|
||||
<b> 4. Weights and Activations Data Type </b>
|
||||
<b> 4. 权重与激活的数据类型 </b>
|
||||
|
||||
When quantizing models from float32 to lower-precision data types such as float16, the generated images are [known to vary slightly](https://lambdalabs.com/blog/inference-benchmark-stable-diffusion) in semantics even when using the same PyTorch model. Core ML models generated by coremltools have float16 weights and activations by default [unless explicitly overridden](https://github.com/apple/coremltools/blob/main/coremltools/converters/_converters_entry.py#L256). This is not expected to be a major source of difference.
|
||||
将模型从 float32 量化到 float16 等较低精度数据类型时，即使使用同一 PyTorch 模型，生成图像在语义上也[已知会略有差异](https://lambdalabs.com/blog/inference-benchmark-stable-diffusion)。coremltools 生成的 Core ML 模型默认使用 float16 权重与激活[除非显式覆盖](https://github.com/apple/coremltools/blob/main/coremltools/converters/_converters_entry.py#L256).。这通常不应成为主要差异来源。
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary> <b> Q9: </b> The model files are very large, how do I avoid a large binary for my App? </summary>
|
||||
<summary> <b> Q9: </b> 模型文件非常大，如何避免应用二进制体积过大？ </summary>
|
||||
|
||||
<b> A9: </b> The recommended option is to prompt the user to download these assets upon first launch of the app. This keeps the app binary size independent of the Core ML models being deployed. Disclosing the size of the download to the user is extremely important as there could be data charges or storage impact that the user might not be comfortable with.
|
||||
<b> A9: </b> 推荐做法是在应用首次启动时提示用户下载这些资源。这样应用二进制大小可与所部署的 Core ML 模型解耦。向用户披露下载大小极为重要，因为可能产生流量费用或占用存储空间，用户未必能够接受。
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary> <b> Q10: </b> <code> `Could not initialize NNPACK! Reason: Unsupported hardware` </code> </summary>
|
||||
|
||||
<b> A10: </b> This warning is safe to ignore in the context of this repository.
|
||||
<b> A10: </b> 在本仓库的上下文中，可以安全忽略此警告。
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary> <b> Q11: </b> <code> TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect </code> </summary>
|
||||
|
||||
<b> A11: </b> This warning is safe to ignore in the context of this repository.
|
||||
<b> A11: </b> 在本仓库的上下文中，可以安全忽略此警告。
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary> <b> Q12: </b> <code> UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown </code> </summary>
|
||||
|
||||
<b> A12: </b> If this warning is printed right after <code> zsh: killed python -m python_coreml_stable_diffusion.torch2coreml ... </code>, then it is highly likely that your Mac has run out of memory while converting models to Core ML. Please see [Q3](#low-mem-conversion) from above for the solution.
|
||||
<b> A12: </b> 如果此警告紧跟在 <code> zsh: killed python -m python_coreml_stable_diffusion.torch2coreml ... </code> 之后出现，那么你的 Mac 极有可能在将模型转换为 Core ML 时耗尽了内存。请参阅上文中的 [Q3](#low-mem-conversion) 获取解决方案。
|
||||
|
||||
</details>
|
||||
|
||||
@@ -796,7 +801,7 @@ We **strongly recommend** compressing your models following the recipes in [Adva
|
||||
|
||||
</details>
|
||||
|
||||
## <a name="bibtex"></a> BibTeX Reference
|
||||
## <a name="bibtex"></a> BibTeX 引用
|
||||
|
||||
```latex
|
||||
@misc{stable-diffusion-coreml-apple-silicon,
|
||||
|
||||
Reference in New Issue
Block a user