liguodongiot--llm-action/llm-inference

LLM 推理

• OpenLLM: An open platform for operating large language models (LLMs) in production.

• Triton

• Text Generation Inference：https://github.com/huggingface/text-generation-inference

• vLLM：https://github.com/vllm-project/vllm

• FastTransformer：https://github.com/NVIDIA/FasterTransformer

• vLLM：https://github.com/vllm-project/vllm

• LLM Accelerator：https://github.com/microsoft/LMOps

• ZhiLight（知乎&面壁智能）：https://github.com/zhihu/ZhiLight

• 赤兔：清华团队：https://github.com/thu-pacman/chitu

• mini推理框架：https://github.com/harleyszhang/lite_llama/tree/main

• LMDeploy,TurboMind

• BigDL：https://github.com/intel-analytics/BigDL

• OpenVINO：https://github.com/openvinotoolkit/openvino

• AWQ、AutoAWQ

• llama.cpp、vllm、lightLLM、fastLLM

• marlin（FP16xINT4 LLM inference kernel）：https://github.com/IST-DASLab/marlin

论文：A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiency 链接：https://arxiv.org/pdf/2505.01658