liguodongiot--llm-action/llm-inference/offload.md

• https://huggingface.co/docs/accelerate/concept_guides/big_model_inference

• https://huggingface.co/docs/transformers/big_models

• Efficient and Economic Large Language Model Inference with Attention Offloading

• https://arxiv.org/pdf/2405.01814

• DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale

• https://arxiv.org/pdf/2207.00032

FlexFlow

• https://github.com/flexflow/FlexFlow

FlexGen

• https://github.com/FMInference/FlexGen

kv cache offload:

• https://github.com/NVIDIA/TensorRT-LLM/blob/a96cccafcf6365c128f004f779160951f8c0801c/docs/source/kv_cache_reuse.md