run-llama--llama_index/docs/examples/llm/huggingface.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "978146e2",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/llm/huggingface.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f717d3d4-942b-4d86-9435-fc44b3ac6d39",
   "metadata": {},
   "source": [
    "# Hugging Face LLMs\n",
    "\n",
    "There are many ways to interface with LLMs from [Hugging Face](https://huggingface.co/), either locally or via Hugging Face's [Inference Providers](https://huggingface.co/docs/inference-providers).\n",
    "Hugging Face itself provides several Python packages to enable access,\n",
    "which LlamaIndex wraps into `LLM` entities:\n",
    "\n",
    "- The [`transformers`](https://github.com/huggingface/transformers) package:\n",
    "  use `llama_index.llms.HuggingFaceLLM`\n",
    "- The [Hugging Face Inference Providers](https://huggingface.co/docs/inference-providers),\n",
    "  [wrapped by `huggingface_hub[inference]`](https://github.com/huggingface/huggingface_hub):\n",
    "  use `llama_index.llms.HuggingFaceInferenceAPI`\n",
    "\n",
    "There are _many_ possible permutations of these two, so this notebook only details a few.\n",
    "Let's use Hugging Face's [Text Generation task](https://huggingface.co/tasks/text-generation) as our example."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90cf0f2e-8d8d-4e42-81bf-866c759221e1",
   "metadata": {},
   "source": [
    "In the below line, we install the packages necessary for this demo:\n",
    "\n",
    "- `transformers[torch]` is needed for `HuggingFaceLLM`\n",
    "- `huggingface_hub[inference]` is needed for `HuggingFaceInferenceAPI`\n",
    "- The quotes are needed for Z shell (`zsh`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f413f179",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install llama-index-llms-huggingface # for local inference\n",
    "%pip install llama-index-llms-huggingface-api # for remote inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b04b4a5-6fce-4188-a538-9a5ce2fa56f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install \"transformers[torch]\" \"huggingface_hub[inference]\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2c577674",
   "metadata": {},
   "source": [
    "If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86028752",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install llama-index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dac8f9f-7136-43f7-9e9f-de679e74d66e",
   "metadata": {},
   "source": [
    "Now that we're set up, let's play around:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3204052c",
   "metadata": {},
   "source": [
    "# Setup Hugging Face Account\n",
    "\n",
    "First, you need to create a Hugging Face account and get a token. You can sign up [here](https://huggingface.co/join). Then you'll need to create a token [here](https://huggingface.co/settings/tokens).\n",
    "\n",
    "```sh\n",
    "export HUGGING_FACE_TOKEN=hf_your_token_here\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0465029c-fe69-454a-9561-55f7a382b2e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from typing import List, Optional\n",
    "\n",
    "from llama_index.llms.huggingface import HuggingFaceLLM\n",
    "from llama_index.llms.huggingface_api import HuggingFaceInferenceAPI\n",
    "\n",
    "HF_TOKEN: Optional[str] = os.getenv(\"HUGGING_FACE_TOKEN\")\n",
    "# NOTE: None default will fall back on Hugging Face's token storage\n",
    "# when this token gets used within HuggingFaceInferenceAPI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a77bd3c0",
   "metadata": {},
   "source": [
    "## Use a model via Inference Providers\n",
    "\n",
    "The easiest way to use an open source model is to use the Hugging Face [Inference Providers](https://huggingface.co/docs/inference-providers). Let's use the DeepSeek R1 model, which is great for complex tasks.\n",
    "\n",
    "With inference providers, you can use the model on serverless infrastructure from inference providers.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc0cdc6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "remotely_run = HuggingFaceInferenceAPI(\n",
    "    model_name=\"deepseek-ai/DeepSeek-R1-0528\",\n",
    "    token=HF_TOKEN,\n",
    "    provider=\"auto\",  # this will use the best provider available\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c414abc7",
   "metadata": {},
   "source": [
    "We can also specify our preferred inference provider. Let's use the [`together` provider](https://huggingface.co/togethercomputer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05c1d74c",
   "metadata": {},
   "outputs": [],
   "source": [
    "remotely_run = HuggingFaceInferenceAPI(\n",
    "    model_name=\"Qwen/Qwen3-235B-A22B\",\n",
    "    token=HF_TOKEN,\n",
    "    provider=\"together\",  # this will use the best provider available\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9576da46",
   "metadata": {},
   "source": [
    "## Use an open source model locally\n",
    "\n",
    "First, we'll use an open source model that's optimized for local inference. This model is downloaded (if first invocation) to the local Hugging Face model cache, and actually runs the model on your local machine's hardware.\n",
    "\n",
    "We'll use the [Gemma 3N E4B](https://huggingface.co/google/gemma-3n-E4B-it) model, which is optimized for local inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f617b21",
   "metadata": {},
   "outputs": [],
   "source": [
    "locally_run = HuggingFaceLLM(model_name=\"google/gemma-3n-E4B-it\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa8c3bd3",
   "metadata": {},
   "source": [
    "## Use a dedicated Inference Endpoint\n",
    "\n",
    "We can also spin up a dedicated Inference Endpoint for a model and use that to run the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5626e58",
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_server = HuggingFaceInferenceAPI(\n",
    "    model=\"https://(<your-endpoint>.eu-west-1.aws.endpoints.huggingface.cloud\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05b8cf81",
   "metadata": {},
   "source": [
    "## Use a local inference engine (vLLM or TGI)\n",
    "\n",
    "We can also use a local inference engine like [vLLM](https://github.com/vllm-project/vllm) or [TGI](https://github.com/huggingface/text-generation-inference) to run the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a27feba3-d027-4d10-b1af-1e130e764a67",
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can also connect to a model being served by a local or remote\n",
    "# Text Generation Inference server\n",
    "tgi_server = HuggingFaceInferenceAPI(model=\"http://localhost:8080\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b801bef7-2593-49e2-a550-721e6b796486",
   "metadata": {},
   "source": [
    "Underlying a completion with `HuggingFaceInferenceAPI` is Hugging Face's\n",
    "[Text Generation task](https://huggingface.co/tasks/text-generation)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "631269c9-38ca-49d2-a7f0-f88e21adef6e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " beyond!\n",
      "The Infinity Wall Clock is a unique and stylish way to keep track of time. The clock is made of a durable, high-quality plastic and features a bright LED display. The Infinity Wall Clock is powered by batteries and can be mounted on any wall. It is a great addition to any home or office.\n"
     ]
    }
   ],
   "source": [
    "completion_response = remotely_run_recommended.complete(\"To infinity, and\")\n",
    "print(completion_response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db5495af",
   "metadata": {},
   "source": [
    "## Setting a tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dda1be10",
   "metadata": {},
   "source": [
    "If you are modifying the LLM, you should also change the global tokenizer to match!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12e0f3c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import set_global_tokenizer\n",
    "from transformers import AutoTokenizer\n",
    "\n",
    "set_global_tokenizer(\n",
    "    AutoTokenizer.from_pretrained(\"HuggingFaceH4/zephyr-7b-alpha\").encode\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fa723d6-4308-4d94-9609-8c51ce8184c3",
   "metadata": {},
   "source": [
    "If you're curious, other Hugging Face Inference API tasks wrapped are:\n",
    "\n",
    "- `llama_index.llms.HuggingFaceInferenceAPI.chat`: [Conversational task](https://huggingface.co/tasks/conversational)\n",
    "- `llama_index.embeddings.HuggingFaceInferenceAPIEmbedding`: [Feature Extraction task](https://huggingface.co/tasks/feature-extraction)\n",
    "\n",
    "And yes, Hugging Face embedding models are supported with:\n",
    "\n",
    "- `transformers[torch]`: wrapped by `HuggingFaceEmbedding`\n",
    "- `huggingface_hub[inference]`: wrapped by `HuggingFaceInferenceAPIEmbedding`\n",
    "\n",
    "Both of the above two subclass `llama_index.embeddings.base.BaseEmbedding`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}