run-llama--llama_index/docs/examples/embeddings/ipex_llm_gpu.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Local Embeddings with IPEX-LLM on Intel GPU\n",
    "\n",
    "> [IPEX-LLM](https://github.com/intel-analytics/ipex-llm/) is a PyTorch library for running LLM on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max) with very low latency.\n",
    "\n",
    "This example goes over how to use LlamaIndex to conduct embedding tasks with `ipex-llm` optimizations on Intel GPU. This would be helpful in applications such as RAG, document QA, etc.\n",
    "\n",
    "> **Note**\n",
    ">\n",
    "> You could refer to [here](https://github.com/run-llama/llama_index/tree/main/llama-index-integrations/embeddings/llama-index-embeddings-ipex-llm/examples) for full examples of `IpexLLMEmbedding`. Please note that for running on Intel GPU, please specify `-d 'xpu'` or `-d 'xpu:<device_id>'` in command argument when running the examples.\n",
    "\n",
    "## Install Prerequisites\n",
    "To benefit from IPEX-LLM on Intel GPUs, there are several prerequisite steps for tools installation and environment preparation.\n",
    "\n",
    "If you are a Windows user, visit the [Install IPEX-LLM on Windows with Intel GPU Guide](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/install_windows_gpu.html), and follow [**Install Prerequisites**](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/install_windows_gpu.html#install-prerequisites) to update GPU driver (optional) and install Conda.\n",
    "\n",
    "If you are a Linux user, visit the [Install IPEX-LLM on Linux with Intel GPU](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/install_linux_gpu.html), and follow [**Install Prerequisites**](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/install_linux_gpu.html#install-prerequisites) to install GPU driver, Intel® oneAPI Base Toolkit 2024.0, and Conda.\n",
    "\n",
    "## Install `llama-index-embeddings-ipex-llm`\n",
    "\n",
    "After the prerequisites installation, you should have created a conda environment with all prerequisites installed, activate your conda environment and install `llama-index-embeddings-ipex-llm` as follows:\n",
    "\n",
    "```bash\n",
    "conda activate <your-conda-env-name>\n",
    "\n",
    "pip install llama-index-embeddings-ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/\n",
    "```\n",
    "This step will also install `ipex-llm` and its dependencies.\n",
    "\n",
    "> **Note**\n",
    ">\n",
    "> You can also use `https://pytorch-extension.intel.com/release-whl/stable/xpu/cn/` as the `extra-indel-url`.\n",
    "\n",
    "\n",
    "## Runtime Configuration\n",
    "\n",
    "For optimal performance, it is recommended to set several environment variables based on your device:\n",
    "\n",
    "### For Windows Users with Intel Core Ultra integrated GPU\n",
    "\n",
    "In Anaconda Prompt:\n",
    "\n",
    "```\n",
    "set SYCL_CACHE_PERSISTENT=1\n",
    "set BIGDL_LLM_XMX_DISABLED=1\n",
    "```\n",
    "\n",
    "### For Linux Users with Intel Arc A-Series GPU\n",
    "\n",
    "```bash\n",
    "# Configure oneAPI environment variables. Required step for APT or offline installed oneAPI.\n",
    "# Skip this step for PIP-installed oneAPI since the environment has already been configured in LD_LIBRARY_PATH.\n",
    "source /opt/intel/oneapi/setvars.sh\n",
    "\n",
    "# Recommended Environment Variables for optimal performance\n",
    "export USE_XETLA=OFF\n",
    "export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1\n",
    "export SYCL_CACHE_PERSISTENT=1\n",
    "```\n",
    "\n",
    "> **Note**\n",
    ">\n",
    "> For the first time that each model runs on Intel iGPU/Intel Arc A300-Series or Pro A60, it may take several minutes to compile.\n",
    ">\n",
    "> For other GPU type, please refer to [here](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Overview/install_gpu.html#runtime-configuration) for Windows users, and  [here](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Overview/install_gpu.html#id5) for Linux users.\n",
    "\n",
    "## `IpexLLMEmbedding`\n",
    "\n",
    "Setting `device=\"xpu\"` when initializing `IpexLLMEmbedding` will put the embedding model on Intel GPU and benefit from IPEX-LLM optimizations:\n",
    "\n",
    "```python\n",
    "from llama_index.embeddings.ipex_llm import IpexLLMEmbedding\n",
    "\n",
    "embedding_model = IpexLLMEmbedding(\n",
    "    model_name=\"BAAI/bge-large-en-v1.5\", device=\"xpu\"\n",
    ")\n",
    "```\n",
    "\n",
    "> Please note that `IpexLLMEmbedding` currently only provides optimization for Hugging Face Bge models.\n",
    ">\n",
    "> If you have multiple Intel GPUs available, you could set `device=\"xpu:<device_id>\"`, in which `device_id` is counted from 0. `device=\"xpu\"` is equal to `device=\"xpu:0\"` by default.\n",
    "\n",
    "You could then conduct the embedding tasks as normal:\n",
    "\n",
    "```python\n",
    "sentence = \"IPEX-LLM is a PyTorch library for running LLM on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max) with very low latency.\"\n",
    "query = \"What is IPEX-LLM?\"\n",
    "\n",
    "text_embedding = embedding_model.get_text_embedding(sentence)\n",
    "print(f\"embedding[:10]: {text_embedding[:10]}\")\n",
    "\n",
    "text_embeddings = embedding_model.get_text_embedding_batch([sentence, query])\n",
    "print(f\"text_embeddings[0][:10]: {text_embeddings[0][:10]}\")\n",
    "print(f\"text_embeddings[1][:10]: {text_embeddings[1][:10]}\")\n",
    "\n",
    "query_embedding = embedding_model.get_query_embedding(query)\n",
    "print(f\"query_embedding[:10]: {query_embedding[:10]}\")\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}