wehub-resource-sync
91 lines
3.5 KiB
Markdown
91 lines
3.5 KiB
Markdown
# xFasterTransformer Inference Framework
|
|
|
|
Integrated [xFasterTransformer](https://github.com/intel/xFasterTransformer) customized framework into Fastchat to provide **Faster** inference speed on Intel CPU.
|
|
|
|
## Install xFasterTransformer
|
|
|
|
Setup environment (please refer to [this link](https://github.com/intel/xFasterTransformer#installation) for more details):
|
|
|
|
```bash
|
|
pip install xfastertransformer
|
|
```
|
|
|
|
## Prepare models
|
|
|
|
Prepare Model (please refer to [this link](https://github.com/intel/xFasterTransformer#prepare-model) for more details):
|
|
```bash
|
|
python ./tools/chatglm_convert.py -i ${HF_DATASET_DIR} -o ${OUTPUT_DIR}
|
|
```
|
|
|
|
## Parameters of xFasterTransformer
|
|
--enable-xft to enable xfastertransformer in Fastchat
|
|
--xft-max-seq-len to set the max token length the model can process. max token length include input token length.
|
|
--xft-dtype to set datatype used in xFasterTransformer for computation. xFasterTransformer can support fp32, fp16, int8, bf16 and hybrid data types like : bf16_fp16, bf16_int8. For datatype details please refer to [this link](https://github.com/intel/xFasterTransformer/wiki/Data-Type-Support-Platform)
|
|
|
|
|
|
Chat with the CLI:
|
|
```bash
|
|
#run inference on all CPUs and using float16
|
|
python3 -m fastchat.serve.cli \
|
|
--model-path /path/to/models \
|
|
--enable-xft \
|
|
--xft-dtype fp16
|
|
```
|
|
or with numactl on multi-socket server for better performance
|
|
```bash
|
|
#run inference on numanode 0 and with data type bf16_fp16 (first token uses bfloat16, and rest tokens use float16)
|
|
numactl -N 0 --localalloc \
|
|
python3 -m fastchat.serve.cli \
|
|
--model-path /path/to/models/chatglm2_6b_cpu/ \
|
|
--enable-xft \
|
|
--xft-dtype bf16_fp16
|
|
```
|
|
or using MPI to run inference on 2 sockets for better performance
|
|
```bash
|
|
#run inference on numanode 0 and 1 and with data type bf16_fp16 (first token uses bfloat16, and rest tokens use float16)
|
|
OMP_NUM_THREADS=$CORE_NUM_PER_SOCKET LD_PRELOAD=libiomp5.so mpirun \
|
|
-n 1 numactl -N 0 --localalloc \
|
|
python -m fastchat.serve.cli \
|
|
--model-path /path/to/models/chatglm2_6b_cpu/ \
|
|
--enable-xft \
|
|
--xft-dtype bf16_fp16 : \
|
|
-n 1 numactl -N 1 --localalloc \
|
|
python -m fastchat.serve.cli \
|
|
--model-path /path/to/models/chatglm2_6b_cpu/ \
|
|
--enable-xft \
|
|
--xft-dtype bf16_fp16
|
|
```
|
|
|
|
|
|
Start model worker:
|
|
```bash
|
|
# Load model with default configuration (max sequence length 4096, no GPU split setting).
|
|
python3 -m fastchat.serve.model_worker \
|
|
--model-path /path/to/models \
|
|
--enable-xft \
|
|
--xft-dtype bf16_fp16
|
|
```
|
|
or with numactl on multi-socket server for better performance
|
|
```bash
|
|
#run inference on numanode 0 and with data type bf16_fp16 (first token uses bfloat16, and rest tokens use float16)
|
|
numactl -N 0 --localalloc python3 -m fastchat.serve.model_worker \
|
|
--model-path /path/to/models \
|
|
--enable-xft \
|
|
--xft-dtype bf16_fp16
|
|
```
|
|
or using MPI to run inference on 2 sockets for better performance
|
|
```bash
|
|
#run inference on numanode 0 and 1 and with data type bf16_fp16 (first token uses bfloat16, and rest tokens use float16)
|
|
OMP_NUM_THREADS=$CORE_NUM_PER_SOCKET LD_PRELOAD=libiomp5.so mpirun \
|
|
-n 1 numactl -N 0 --localalloc python -m fastchat.serve.model_worker \
|
|
--model-path /path/to/models \
|
|
--enable-xft \
|
|
--xft-dtype bf16_fp16 : \
|
|
-n 1 numactl -N 1 --localalloc python -m fastchat.serve.model_worker \
|
|
--model-path /path/to/models \
|
|
--enable-xft \
|
|
--xft-dtype bf16_fp16
|
|
```
|
|
|
|
For more details, please refer to [this link](https://github.com/intel/xFasterTransformer#how-to-run)
|