wehub-resource-sync
45 lines
2.0 KiB
Markdown
45 lines
2.0 KiB
Markdown
# Onnx Simple Chat with Cuda Execution Provider
|
|
|
|
This sample demonstrates how you use ONNX Connector with CUDA Execution Provider to run Local Models straight from files using Semantic Kernel.
|
|
|
|
In this example we setup Chat Client from ONNX Connector with [Microsoft's Phi-3-ONNX](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-onnx) model
|
|
|
|
> [!IMPORTANT]
|
|
> You can modify to use any other combination of models enabled for ONNX runtime.
|
|
|
|
## Semantic Kernel used Features
|
|
|
|
- [Chat Client](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/SemanticKernel.Abstractions/AI/ChatCompletion/IChatCompletionService.cs) - Using the Chat Completion Service from [Onnx Connector](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/Connectors/Connectors.Onnx/OnnxRuntimeGenAIChatCompletionService.cs) to generate responses from the Local Model.
|
|
|
|
## Prerequisites
|
|
|
|
- [.NET 10](https://dotnet.microsoft.com/download/dotnet/10.0).
|
|
- [NVIDIA GPU](https://www.nvidia.com/en-us/geforce/graphics-cards)
|
|
- [NVIDIA CUDA v12 Toolkit](https://developer.nvidia.com/cuda-12-0-0-download-archive)
|
|
- [NVIDIA cuDNN v9.11](https://developer.nvidia.com/cudnn-9-11-0-download-archive)
|
|
- Windows users only:
|
|
|
|
Ensure `PATH` environment variable includes the `bin` folder of the CUDA Toolkit and cuDNN.
|
|
i.e:
|
|
- C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.0\bin
|
|
- C:\Program Files\NVIDIA\CUDNN\v9.11\bin\12.9
|
|
|
|
- Downloaded ONNX Models (see below).
|
|
|
|
## Downloading the Model
|
|
|
|
For this example we chose Hugging Face as our repository for download of the local models, go to a directory of your choice where the models should be downloaded and run the following commands:
|
|
|
|
```powershell
|
|
git lfs install
|
|
git clone https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-onnx
|
|
```
|
|
|
|
Update the `Program.cs` file lines below with the paths to the models you downloaded in the previous step.
|
|
|
|
```csharp
|
|
// i.e. Running on Windows
|
|
string modelPath = "D:\\repo\\huggingface\\Phi-3-mini-4k-instruct-onnx\\cuda\\cuda-int4-rtn-block-32";
|
|
```
|
|
|