Please help me summarize the following,The Large Language Model (LLM) is widely employed for tasks such as intelligent assistants, text summarization, translation, and multi-modality on mobile phones. However, the current methods for on-device LLM deployment maintain slow inference speed, which causes poor user experience. To facilitate high-efficiency LLM deployment on device GPUs, we propose four optimization techniques: (a) a symbolic expression-based approach to support dynamic shape model inference; (b) operator optimizations and execution priority setting to enhance inference speed and reduce phone lagging; (c) an FP4 quantization method termed M0E4 to reduce dequantization overhead; (d) a sub-tensor-based technique to eliminate the need for copying KV cache after LLM inference. Furthermore, we implement these methods in our mobile inference engine, Transformer-Lite, which is compatible with both Qualcomm and MTK processors. We evaluated Transformer-Lite’s performance using LLMs with varied architectures and parameters ranging from 2B to 14B. Specifically, we achieved prefill and decoding speeds of 121 token/s and 14 token/s for ChatGLM2 6B, and 330 token/s and 30 token/s for smaller Gemma 2B, respectively. Compared with CPU- based FastLLM and GPU-based MLC-LLM, our engine attains over 10x speedup for the prefill speed and 2~3x speedup for the decoding speed.1 IntroductionThe LLM has demonstrated superior performance in a range of tasks[1, 20, 32, 39], proving versatile in applications such as intelligent assistants, text summarization, translation, and multi-modality tasks on mobile devices. Although the performance is excellent, the deployment of LLMs requires substantial computing power and memory bandwidth. Hence, current applications predominantly adopt the cloud-based deployment. Given the substantial cost involved in the cloud-based deployment of LLMs and the continuous advancements in the performance of mobile devices, native deployment of LLMs on mobile phones not only curtails the high costs associated with cloud deployment but also broadens the prospective applications of LLMs on mobile phones. Therefore, deploying LLMs on devices is becoming a research hotspot.Compared with cloud servers, deploying LLMs on mobile phones is constrained by limited hardware performance, memory bandwidth, and storage capacity. As a result, only smaller parameter models can be employed, and the parameter range of current mainstream LLMs is*Corresponding author: liluchang@oppo.com1approximately from 1B to 13B [24, 44–47, 47]. The speed of LLM inference is critical for user experience. Although models with reduced parameters operate at a faster inference speed, their accuracy is inevitably compromised. To enhance the user experience, high-efficiency LLM deployment that secures both high accuracy and efficient inference is essential.To deploy LLM on device, currently, two types of methods can be used: generic mobile inference engines such as TFLite[48], MNN[18], NCNN[49], etc., or LLM-specific engines including Llama.cpp[50], MLC-LLM[51], FastLLM[52], etc. Conventionally, computer vision (CV) models were widely deployed on devices using generic mobile inference engines[2, 8, 33], such as ResNet[15], MobileNet[17], etc. The advantage of these engines lies in their ability to directly use serialized models exported from training frameworks, such as ONNX[53], thereby eliminating the need for users to re-describe the model structure and simplifying support for various model architectures. However, these engines are primarily optimized for static shape CV models. LLMs substantially differ from CV models in terms of dynamic input shapes, model structures, operator types, tensor dimensions, etc. Consequently, optimizations for CV models are typically not directly applicable to LLMs. As a result, high- efficient LLM deployment based on device GPUs is usually not supported.In contrast, LLM-specific engines are specifically designed for transformer-based LLM deployment and demonstrate the feasibility of deploying 1B-7B models on devices using CPUs or GPUs. However, these engines have not yet fully utilized the hardware performance and memory bandwidth, resulting in slow prefill and decoding speeds, which in turn degrade the user experience. Furthermore, these engines necessitate the re-descriptions of model structures via C++ or TVM script, complicating support for new model architectures. Notably, these LLM-specific engines do not accommodate other types of models such as CV and vision transformer (ViT).To support LLM deployment on device GPUs both conveniently and efficiently, we advocate for combiningThe Large Language Model (LLM) is widely employed for tasks such as intelligent assistants, text summarization, translation, and multi-modality on mobile phones. However, the current methods for on-device LLM deployment maintain slow inference speed, which causes poor user experience. To facilitate high-efficiency LLM deployment on device GPUs, we propose four optimization techniques: (a) a symbolic expression-based approach to support dynamic shape model inference; (b) operator optimizations and execution priority setting to enhance inference speed and reduce phone lagging; (c) an FP4 quantization method termed M0E4 to reduce dequantization overhead; (d) a sub-tensor-based technique to eliminate the need for copying KV cache after LLM inference. Furthermore, we implement these methods in our mobile inference engine, Transformer-Lite, which is compatible with both Qualcomm and MTK processors. We evaluated Transformer-Lite’s performance using LLMs with varied architectures and parameters ranging from 2B to 14B. Specifically, we achieved prefill and decoding speeds of 121 token/s and 14 token/s for ChatGLM2 6B, and 330 token/s and 30 token/s for smaller Gemma 2B, respectively. Compared with CPU- based FastLLM and GPU-based MLC-LLM, our engine attains over 10x speedup for the prefill speed and 2~3x speedup for the decoding speed.1 IntroductionThe LLM has demonstrated superior performance in a range of tasks[1, 20, 32, 39], proving versatile in applications such as intelligent assistants, text summarization, translation, and multi-modality tasks on mobile devices. Although the performance is excellent, the deployment of LLMs requires substantial computing power and memory bandwidth. Hence, current applications predominantly adopt the cloud-based deployment. Given the substantial cost involved in the cloud-based deployment of LLMs and the continuous advancements in the performance of mobile devices, native deployment of LLMs on mobile phones not only curtails the high costs associated with cloud deployment but also broadens the prospective applications of LLMs on mobile phones. Therefore, deploying LLMs on devices is becoming a research hotspot.Compared with cloud servers, deploying LLMs on mobile phones is constrained by limited hardware performance, memory bandwidth, and storage capacity. As a result, only smaller parameter models can be employed, and the parameter range of current mainstream LLMs is*Corresponding author: liluchang@oppo.com1approximately from 1B to 13B [24, 44–47, 47]. The speed of LLM inference is critical for user experience. Although models with reduced parameters operate at a faster inference speed, their accuracy is inevitably compromised. To enhance the user experience, high-efficiency LLM deployment that secures both high accuracy and efficient inference is essential.To deploy LLM on device, currently, two types of methods can be used: generic mobile inference engines such as TFLite[48], MNN[18], NCNN[49], etc., or LLM-specific engines including Llama.cpp[50], MLC-LLM[51], FastLLM[52], etc. Conventionally, computer vision (CV) models were widely deployed on devices using generic mobile inference engines[2, 8, 33], such as ResNet[15], MobileNet[17], etc. The advantage of these engines lies in their ability to directly use serialized models exported from training frameworks, such as ONNX[53], thereby eliminating the need for users to re-describe the model structure and simplifying support for various model architectures. However, these engines are primarily optimized for static shape CV models. LLMs substantially differ from CV models in terms of dynamic input shapes, model structures, operator types, tensor dimensions, etc. Consequently, optimizations for CV models are typically not directly applicable to LLMs. As a result, high- efficient LLM deployment based on device GPUs is usually not supported.In contrast, LLM-specific engines are specifically designed for transformer-based LLM deployment and demonstrate the feasibility of deploying 1B-7B models on devices using CPUs or GPUs. However, these engines have not yet fully utilized the hardware performance and memory bandwidth, resulting in slow prefill and decoding speeds, which in turn degrade the user experience. Furthermore, these engines necessitate the re-descriptions of model structures via C++ or TVM script, complicating support for new model architectures. Notably, these LLM-specific engines do not accommodate other types of models such as CV and vision transformer (ViT).To support LLM deployment on device GPUs both conveniently and efficiently, we advocate for combiningThe Large Language Model (LLM) is widely employed for tasks such as intelligent assistants, text summarization, translation, and multi-modality on mobile phones. However, the current methods for on-device LLM deployment maintain slow inference speed, which causes poor user experience. To facilitate high-efficiency LLM deployment on device GPUs, we propose four optimization techniques: (a) a symbolic expression-based approach to support dynamic shape model inference; (b) operator optimizations and execution priority setting to enhance inference speed and reduce phone lagging; (c) an FP4 quantization method termed M0E4 to reduce dequantization overhead; (d) a sub-tensor-based technique to eliminate the need for copying KV cache after LLM inference. Furthermore, we implement these methods in our mobile inference engine, Transformer-Lite, which is compatible with both Qualcomm and MTK processors. We evaluated Transformer-Lite’s performance using LLMs with varied architectures and parameters ranging from 2B to 14B. Specifically, we achieved prefill and decoding speeds of 121 token/s and 14 token/s for ChatGLM2 6B, and 330 token/s and 30 token/s for smaller Gemma 2B, respectively. Compared with CPU- based FastLLM and GPU-based MLC-LLM, our engine attains over 10x speedup for the prefill speed and 2~3x speedup for the decoding speed.1 IntroductionThe LLM has demonstrated superior performance in a range of tasks[1, 20, 32, 39], proving versatile in applications such as intelligent assistants, text summarization, translation, and multi-modality tasks on mobile devices. Although the performance is excellent, the deployment of LLMs requires substantial computing power and memory bandwidth. Hence, current applications predominantly adopt the cloud-based deployment. Given the substantial cost involved in the cloud-based deployment of LLMs and the continuous advancements in the performance of mobile devices, native deployment of LLMs on mobile phones not only curtails the high costs associated with cloud deployment but also broadens the prospective applications of LLMs on mobile phones. Therefore, deploying LLMs on devices is becoming a research hotspot.Compared with cloud servers, deploying LLMs on mobile phones is constrained by limited hardware performance, memory bandwidth, and storage capacity. As a result, only smaller parameter models can be employed, and the parameter range of current mainstream LLMs is*Corresponding author: liluchang@oppo.com1approximately from 1B to 13B [24, 44–47, 47]. The speed of LLM inference is critical for user experience. Although models with reduced parameters operate at a faster inference speed, their accuracy is inevitably compromised. To enhance the user experience, high-efficiency LLM deployment that secures both high accuracy and efficient inference is essential.To deploy LLM on device, currently, two types of methods can be used: generic mobile inference engines such as TFLite[48], MNN[18], NCNN[49], etc., or LLM-specific engines including Llama.cpp[50], MLC-LLM[51], FastLLM[52], etc. Conventionally, computer vision (CV) models were widely deployed on devices using generic mobile inference engines[2, 8, 33], such as ResNet[15], MobileNet[17], etc. The advantage of these engines lies in their ability to directly use serialized models exported from training frameworks, such as ONNX[53], thereby eliminating the need for users to re-describe the model structure and simplifying support for various model architectures. However, these engines are primarily optimized for static shape CV models. LLMs substantially differ from CV models in terms of dynamic input shapes, model structures, operator types, tensor dimensions, etc. Consequently, optimizations for CV models are typically not directly applicable to LLMs. As a result, high- efficient LLM deployment based on device GPUs is usually not supported.In contrast, LLM-specific engines are specifically designed for transformer-based LLM deployment and demonstrate the feasibility of deploying 1B-7B models on devices using CPUs or GPUs. However, these engines have not yet fully utilized the hardware performance and memory bandwidth, resulting in slow prefill and decoding speeds, which in turn degrade the user experience. Furthermore, these engines necessitate the re-descriptions of model structures via C++ or TVM script, complicating support for new model architectures. Notably, these LLM-specific engines do not accommodate other types of models such as CV and vision transformer (ViT).To support LLM deployment on device GPUs both conveniently and efficiently, we advocate for combiningThe Large Language Model (LLM) is widely employed for tasks such as intelligent assistants, text summarization, translation, and multi-modality on mobile phones. However, the current methods for on-device LLM deployment maintain slow inference speed, which causes poor user experience. To facilitate high-efficiency LLM deployment on device GPUs, we propose four optimization techniques: (a) a symbolic expression-based approach to support dynamic shape model inference; (b) operator optimizations and execution priority setting to enhance inference speed and reduce phone lagging; (c) an FP4 quantization method termed M0E4 to reduce dequantization overhead; (d) a sub-tensor-based technique to eliminate the need for copying KV cache after LLM inference. Furthermore, we implement these methods in our mobile inference engine, Transformer-Lite, which is compatible with both Qualcomm and MTK processors. We evaluated Transformer-Lite’s performance using LLMs with varied architectures and parameters ranging from 2B to 14B. Specifically, we achieved prefill and decoding speeds of 121 token/s and 14 token/s for ChatGLM2 6B, and 330 token/s and 30 token/s for smaller Gemma 2B, respectively. Compared with CPU- based FastLLM and GPU-based MLC-LLM, our engine attains over 10x speedup for the prefill speed and 2~3x speedup for the decoding speed.1 IntroductionThe LLM has demonstrated superior performance in a range of tasks[1, 20, 32, 39], proving versatile in applications such as intelligent assistants, text summarization, translation, and multi-modality tasks on mobile devices. Although the performance is excellent, the deployment of LLMs requires substantial computing power and memory bandwidth. Hence, current applications predominantly adopt the cloud-based deployment. Given the substantial cost involved in the cloud-based deployment of LLMs and the continuous advancements in the performance of mobile devices, native deployment of LLMs on mobile phones not only curtails the high costs associated with cloud deployment but also broadens the prospective applications of LLMs on mobile phones. Therefore, deploying LLMs on devices is becoming a research hotspot.Compared with cloud servers, deploying LLMs on mobile phones is constrained by limited hardware performance, memory bandwidth, and storage capacity. As a result, only smaller parameter models can be employed, and the parameter range of current mainstream LLMs is*Corresponding author: liluchang@oppo.com1approximately from 1B to 13B [24, 44–47, 47]. The speed of LLM inference is critical for user experience. Although models with reduced parameters operate at a faster inference speed, their accuracy is inevitably compromised. To enhance the user experience, high-efficiency LLM deployment that secures both high accuracy and efficient inference is essential.To deploy LLM on device, currently, two types of methods can be used: generic mobile inference engines such as TFLite[48], MNN[18], NCNN[49], etc., or LLM-specific engines including Llama.cpp[50], MLC-LLM[51], FastLLM[52], etc. Conventionally, computer vision (CV) models were widely deployed on devices using generic mobile inference engines[2, 8, 33], such as ResNet[15], MobileNet[17], etc. The advantage of these engines lies in their ability to directly use serialized models exported from training frameworks, such as ONNX[53], thereby eliminating the need for users to re-describe the model structure and simplifying support for various model architectures. However, these engines are primarily optimized for static shape CV models. LLMs substantially differ from CV models in terms of dynamic input shapes, model structures, operator types, tensor dimensions, etc. Consequently, optimizations for CV models are typically not directly applicable to LLMs. As a result, high- efficient LLM deployment based on device GPUs is usually not supported.In contrast, LLM-specific engines are specifically designed for transformer-based LLM deployment and demonstrate the feasibility of deploying 1B-7B models on devices using CPUs or GPUs. However, these engines have not yet fully utilized the hardware performance and memory bandwidth, resulting in slow prefill and decoding speeds, which in turn degrade the user experience. Furthermore, these engines necessitate the re-descriptions of model structures via C++ or TVM script, complicating support for new model architectures. Notably, these LLM-specific engines do not accommodate other types of models such as CV and vision transformer (ViT).To support LLM deployment on device GPUs both conveniently and efficiently, we advocate for combiningThe Large Language Model (LLM) is widely employed for tasks such as intelligent assistants, text summarization, translation, and multi-modality on mobile phones. However, the current methods for on-device LLM deployment maintain slow inference speed, which causes poor user experience. To facilitate high-efficiency LLM deployment on device GPUs, we propose four optimization techniques: (a) a symbolic expression-based approach to support dynamic shape model inference; (b) operator optimizations and execution priority setting to enhance inference speed and reduce phone lagging; (c) an FP4 quantization method termed M0E4 to reduce dequantization overhead; (d) a sub-tensor-based technique to eliminate the need for copying KV cache after LLM inference. Furthermore, we implement these methods in our mobile inference engine, Transformer-Lite, which is compatible with both Qualcomm and MTK processors. We evaluated Transformer-Lite’s performance using LLMs with varied architectures and parameters ranging from 2B to 14B. Specifically, we achieved prefill and decoding speeds of 121 token/s and 14 token/s for ChatGLM2 6B, and 330 token/s and 30 token/s for smaller Gemma 2B, respectively. Compared with CPU- based FastLLM and GPU-based MLC-LLM, our engine attains over 10x speedup for the prefill speed and 2~3x speedup for the decoding speed.1 IntroductionThe LLM has demonstrated superior performance in a range of tasks[1, 20, 32, 39], proving versatile in applications such as intelligent assistants, text summarization, translation, and multi-modality tasks on mobile devices. Although the performance is excellent, the deployment of LLMs requires substantial computing power and memory bandwidth. Hence, current applications predominantly adopt the cloud-based deployment. Given the substantial cost involved in the cloud-based deployment of LLMs and the continuous advancements in the performance of mobile devices, native deployment of LLMs on mobile phones not only curtails the high costs associated with cloud deployment but also broadens the prospective applications of LLMs on mobile phones. Therefore, deploying LLMs on devices is becoming a research hotspot.Compared with cloud servers, deploying LLMs on mobile phones is constrained by limited hardware performance, memory bandwidth, and storage capacity. As a result, only smaller parameter models can be employed, and the parameter range of current mainstream LLMs is*Corresponding author: liluchang@oppo.com1approximately from 1B to 13B [24, 44–47, 47]. The speed of LLM inference is critical for user experience. Although models with reduced parameters operate at a faster inference speed, their accuracy is inevitably compromised. To enhance the user experience, high-efficiency LLM deployment that secures both high accuracy and efficient inference is essential.To deploy LLM on device, currently, two types of methods can be used: generic mobile inference engines such as TFLite[48], MNN[18], NCNN[49], etc., or LLM-specific engines including Llama.cpp[50], MLC-LLM[51], FastLLM[52], etc. Conventionally, computer vision (CV) models were widely deployed on devices using generic mobile inference engines[2, 8, 33], such as ResNet[15], MobileNet[17], etc. The advantage of these engines lies in their ability to directly use serialized models exported from training frameworks, such as ONNX[53], thereby eliminating the need for users to re-describe the model structure and simplifying support for various model architectures. However, these engines are primarily optimized for static shape CV models. LLMs substantially differ from CV models in terms of dynamic input shapes, model structures, operator types, tensor dimensions, etc. Consequently, optimizations for CV models are typically not directly applicable to LLMs. As a result, high- efficient LLM deployment based on device GPUs is usually not supported.In contrast, LLM-specific engines are specifically designed for transformer-based LLM deployment and demonstrate the feasibility of deploying 1B-7B models on devices using CPUs or GPUs. However, these engines have not yet fully utilized the hardware performance and memory bandwidth, resulting in slow prefill and decoding speeds, which in turn degrade the user experience. Furthermore, these engines necessitate the re-descriptions of model structures via C++ or TVM script, complicating support for new model architectures. Notably, these LLM-specific engines do not accommodate other types of models such as CV and vision transformer (ViT).To support LLM deployment on device GPUs both conveniently and efficiently, we advocate for combiningThe Large Language Model (LLM) is widely employed for tasks such as intelligent assistants, text summarization, translation, and multi-modality on mobile phones. However, the current methods for on-device LLM deployment maintain slow inference speed, which causes poor user experience. To facilitate high-efficiency LLM deployment on device GPUs, we propose four optimization techniques: (a) a symbolic expression-based approach to support dynamic shape model inference; (b) operator optimizations and execution priority setting to enhance inference speed and reduce phone lagging; (c) an FP4 quantization method termed M0E4 to reduce dequantization overhead; (d) a sub-tensor-based technique to eliminate the need for copying KV cache after LLM inference. Furthermore, we implement these methods in our mobile inference engine, Transformer-Lite, which is compatible with both Qualcomm and MTK processors. We evaluated Transformer-Lite’s performance using LLMs with varied architectures and parameters ranging from 2B to 14B. Specifically, we achieved prefill and decoding speeds of 121 token/s and 14 token/s for ChatGLM2 6B, and 330 token/s and 30 token/s for smaller Gemma 2B, respectively. Compared with CPU- based FastLLM and GPU-based MLC-LLM, our engine attains over 10x speedup for the prefill speed and 2~3x speedup for the decoding speed.1 IntroductionThe LLM has demonstrated superior performance in a range of tasks[1, 20, 32, 39], proving versatile in applications such as intelligent assistants, text summarization, translation, and multi-modality tasks on mobile devices. Although the performance is excellent, the deployment of LLMs requires substantial computing power and memory bandwidth. Hence, current applications predominantly adopt the cloud-based deployment. Given the substantial cost involved in the cloud-based deployment of LLMs and the continuous advancements in the performance of mobile devices, native deployment of LLMs on mobile phones not only curtails the high costs associated with cloud deployment but also broadens the prospective applications of LLMs on mobile phones. Therefore, deploying LLMs on devices is becoming a research hotspot.Compared with cloud servers, deploying LLMs on mobile phones is constrained by limited hardware performance, memory bandwidth, and storage capacity. As a result, only smaller parameter models can be employed, and the parameter range of current mainstream LLMs is*Corresponding author: liluchang@oppo.com1approximately from 1B to 13B [24, 44–47, 47]. The speed of LLM inference is critical for user experience. Although models with reduced parameters operate at a faster inference speed, their accuracy is inevitably compromised. To enhance the user experience, high-efficiency LLM deployment that secures both high accuracy and efficient inference is essential.To deploy LLM on device, currently, two types of methods can be used: generic mobile inference engines such as TFLite[48], MNN[18], NCNN[49], etc., or LLM-specific engines including Llama.cpp[50], MLC-LLM[51], FastLLM[52], etc. Conventionally, computer vision (CV) models were widely deployed on devices using generic mobile inference engines[2, 8, 33], such as ResNet[15], MobileNet[17], etc. The advantage of these engines lies in their ability to directly use serialized models exported from training frameworks, such as ONNX[53], thereby eliminating the need for users to re-describe the model structure and simplifying support for various model architectures. However, these engines are primarily optimized for static shape CV models. LLMs substantially differ from CV models in terms of dynamic input shapes, model structures, operator types, tensor dimensions, etc. Consequently, optimizations for CV models are typically not directly applicable to LLMs. As a result, high- efficient LLM deployment based on device GPUs is usually not supported.In contrast, LLM-specific engines are specifically designed for transformer-based LLM deployment and demonstrate the feasibility of deploying 1B-7B models on devices using CPUs or GPUs. However, these engines have not yet fully utilized the hardware performance and memory bandwidth, resulting in slow prefill and decoding speeds, which in turn degrade the user experience. Furthermore, these engines necessitate the re-descriptions of model structures via C++ or TVM script, complicating support for new model architectures. Notably, these LLM-specific engines do not accommodate other types of models such as CV and vision transformer (ViT).To support LLM deployment on device GPUs both conveniently and efficiently, we advocate for combiningThe Large Language Model (LLM) is widely employed for tasks such as intelligent assistants, text summarization, translation, and multi-modality on mobile phones. However, the current methods for on-device LLM deployment maintain slow inference speed, which causes poor user experience. To facilitate high-efficiency LLM deployment on device GPUs, we propose four optimization techniques: (a) a symbolic expression-based approach to support dynamic shape model inference; (b) operator optimizations and execution priority setting to enhance inference speed and reduce phone lagging; (c) an FP4 quantization method termed M0E4 to reduce dequantization overhead; (d) a sub-tensor-based technique to eliminate the need for copying KV cache after LLM inference. Furthermore, we implement these methods in our mobile inference engine, Transformer-Lite, which is compatible with both Qualcomm and MTK processors. We evaluated Transformer-Lite’s performance using LLMs with varied architectures and parameters ranging from 2B to 14B. Specifically, we achieved prefill and decoding speeds of 121 token/s and 14 token/s for ChatGLM2 6B, and 330 token/s and 30 token/s for smaller Gemma 2B, respectively. Compared with CPU- based FastLLM and GPU-based MLC-LLM, our engine attains over 10x speedup for the prefill speed and 2~3x speedup for the decoding speed.1 IntroductionThe LLM has demonstrated superior performance in a range of tasks[1, 20, 32, 39], proving versatile in applications such as intelligent assistants, text summarization, translation, and multi-modality tasks on mobile devices. Although the performance is excellent, the deployment of LLMs requires substantial computing power and memory bandwidth. Hence, current applications predominantly adopt the cloud-based deployment. Given the substantial cost involved in the cloud-based deployment of LLMs and the continuous advancements in the performance of mobile devices, native deployment of LLMs on mobile phones not only curtails the high costs associated with cloud deployment but also broadens the prospective applications of LLMs on mobile phones. Therefore, deploying LLMs on devices is becoming a research hotspot.Compared with cloud servers, deploying LLMs on mobile phones is constrained by limited hardware performance, memory bandwidth, and storage capacity. As a result, only smaller parameter models can be employed, and the parameter range of current mainstream LLMs is*Corresponding author: liluchang@oppo.com1approximately from 1B to 13B [24, 44–47, 47]. The speed of LLM inference is critical for user experience. Although models with reduced parameters operate at a faster inference speed, their accuracy is inevitably compromised. To enhance the user experience, high-efficiency LLM deployment that secures both high accuracy and efficient inference is essential.To deploy LLM on device, currently, two types of methods can be used: generic mobile inference engines such as TFLite[48], MNN[18], NCNN[49], etc., or LLM-specific engines including Llama.cpp[50], MLC-LLM[51], FastLLM[52], etc. Conventionally, computer vision (CV) models were widely deployed on devices using generic mobile inference engines[2, 8, 33], such as ResNet[15], MobileNet[17], etc. The advantage of these engines lies in their ability to directly use serialized models exported from training frameworks, such as ONNX[53], thereby eliminating the need for users to re-describe the model structure and simplifying support for various model architectures. However, these engines are primarily optimized for static shape CV models. LLMs substantially differ from CV models in terms of dynamic input shapes, model structures, operator types, tensor dimensions, etc. Consequently, optimizations for CV models are typically not directly applicable to LLMs. As a result, high- efficient LLM deployment based on device GPUs is usually not supported.In contrast, LLM-specific engines are specifically designed for transformer-based LLM deployment and demonstrate the feasibility of deploying 1B-7B models on devices using CPUs or GPUs. However, these engines have not yet fully utilized the hardware performance and memory bandwidth, resulting in slow prefill and decoding speeds, which in turn degrade the user experience. Furthermore, these engines necessitate the re-descriptions of model structures via C++ or TVM script, complicating support for new model architectures. Notably, these LLM-specific engines do not accommodate other types of models such as CV and vision transformer (ViT).To support LLM deployment on device GPUs both conveniently and efficiently, we advocate for combiningThe Large Language Model (LLM) is widely employed for tasks such as intelligent assistants, text summarization, translation, and multi-modality on mobile phones. However, the current methods for on-device LLM deployment maintain slow inference speed, which causes poor user experience. To facilitate high-efficiency LLM deployment on device GPUs, we propose four optimization techniques: (a) a symbolic expression-based approach to support dynamic shape model inference; (b) operator optimizations and execution priority setting to enhance inference speed and reduce phone lagging; (c) an FP4 quantization method termed M0E4 to reduce dequantization overhead; (d) a sub-tensor-based technique to eliminate the need for copying KV cache after LLM inference. Furthermore, we implement these methods in our mobile inference engine, Transformer-Lite, which is compatible with both Qualcomm and MTK processors. We evaluated Transformer-Lite’s performance using LLMs with varied architectures and parameters ranging from 2B to 14B. Specifically, we achieved prefill and decoding speeds of 121 token/s and 14 token/s for ChatGLM2 6B, and 330 token/s and 30 token/s for smaller Gemma 2B, respectively. Compared with CPU- based FastLLM and GPU-based MLC-LLM, our engine attains over 10x speedup for the prefill speed and 2~3x speedup for the decoding speed.1 IntroductionThe LLM has demonstrated superior performance in a range of tasks[1, 20, 32, 39], proving versatile in applications such as intelligent assistants, text summarization, translation, and multi-modality tasks on mobile devices. Although the performance is excellent, the deployment of LLMs requires substantial computing power and memory bandwidth. Hence, current applications predominantly adopt the cloud-based deployment. Given the substantial cost involved in the cloud-based deployment of LLMs and the continuous advancements in the performance of mobile devices, native deployment of LLMs on mobile phones not only curtails the high costs associated with cloud deployment but also broadens the prospective applications of LLMs on mobile phones. Therefore, deploying LLMs on devices is becoming a research hotspot.Compared with cloud servers, deploying LLMs on mobile phones is constrained by limited hardware performance, memory bandwidth, and storage capacity. As a result, only smaller parameter models can be employed, and the parameter range of current mainstream LLMs is*Corresponding author: liluchang@oppo.com1approximately from 1B to 13B [24, 44–47, 47]. The speed of LLM inference is critical for user experience. Although models with reduced parameters operate at a faster inference speed, their accuracy is inevitably compromised. To enhance the user experience, high-efficiency LLM deployment that secures both high accuracy and efficient inference is essential.To deploy LLM on device, currently, two types of methods can be used: generic mobile inference engines such as TFLite[48], MNN[18], NCNN[49], etc., or LLM-specific engines including Llama.cpp[50], MLC-LLM[51], FastLLM[52], etc. Conventionally, computer vision (CV) models were widely deployed on devices using generic mobile inference engines[2, 8, 33], such as ResNet[15], MobileNet[17], etc. The advantage of these engines lies in their ability to directly use serialized models exported from training frameworks, such as ONNX[53], thereby eliminating the need for users to re-describe the model structure and simplifying support for various model architectures. However, these engines are primarily optimized for static shape CV models. LLMs substantially differ from CV models in terms of dynamic input shapes, model structures, operator types, tensor dimensions, etc. Consequently, optimizations for CV models are typically not directly applicable to LLMs. As a result, high- efficient LLM deployment based on device GPUs is usually not supported.In contrast, LLM-specific engines are specifically designed for transformer-based LLM deployment and demonstrate the feasibility of deploying 1B-7B models on devices using CPUs or GPUs. However, these engines have not yet fully utilized the hardware performance and memory bandwidth, resulting in slow prefill and decoding speeds, which in turn degrade the user experience. Furthermore, these engines necessitate the re-descriptions of model structures via C++ or TVM script, complicating support for new model architectures. Notably, these LLM-specific engines do not accommodate other types of models such as CV and vision transformer (ViT).To support LLM deployment on device GPUs both conveniently and efficiently, we advocate for combining