liguodongiot--llm-action/docs/llm-inference

吞吐量

延迟

投机采样：

• https://github.com/feifeibear/LLMSpeculativeSampling

美杜莎：

• https://github.com/FasterDecoding/Medusa

• Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

• https://sites.google.com/view/medusa-llm

• OpenLLM: https://github.com/bentoml/OpenLLM

Triton

博客

• https://huggingface.co/blog/optimize-llm
• 加速大模型推理的7种方法：https://betterprogramming.pub/speed-up-llm-inference-83653aa24c47
• 7个大模型推理服务化框架：https://betterprogramming.pub/frameworks-for-serving-llms-60b7f7b23407