wehub-resource-sync
a0c8464e58
Build Package / build (ubuntu-latest) (push) Failing after 1s
CodeQL / Analyze (python) (push) Failing after 1s
Core Typecheck / core-typecheck (push) Failing after 1s
Linting / lint (push) Failing after 1s
llama-dev tests / test-llama-dev (push) Failing after 1s
Publish Sub-Package to PyPI if Needed / publish_subpackage_if_needed (push) Has been skipped
Sync Docs to Developer Hub / sync-docs (push) Failing after 0s
Build Package / build (windows-latest) (push) Has been cancelled
275 lines
11 KiB
Plaintext
275 lines
11 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"# NVIDIA NIMs\n",
|
||
"\n",
|
||
"The `llama-index-postprocessor-nvidia-rerank` package contains LlamaIndex integrations for building applications with models on NVIDIA NIM inference microservices. NIM supports models across domains like chat, embedding, and re-ranking models \n",
|
||
"from the community as well as NVIDIA. These models are optimized by NVIDIA to deliver the best performance on NVIDIA \n",
|
||
"accelerated infrastructure and deployed as a NIM, an easy-to-use, prebuilt containers that deploy anywhere using a single \n",
|
||
"command on NVIDIA accelerated infrastructure.\n",
|
||
"\n",
|
||
"NVIDIA hosted deployments of NIMs are available to test on the [NVIDIA API catalog](https://build.nvidia.com/). After testing, \n",
|
||
"NIMs can be exported from NVIDIA’s API catalog using the NVIDIA AI Enterprise license and run on-premises or in the cloud, \n",
|
||
"giving enterprises ownership and full control of their IP and AI application.\n",
|
||
"\n",
|
||
"NIMs are packaged as container images on a per model basis and are distributed as NGC container images through the NVIDIA NGC Catalog. \n",
|
||
"At their core, NIMs provide easy, consistent, and familiar APIs for running inference on an AI model.\n",
|
||
"\n",
|
||
"# NVIDIA Rerank connector\n",
|
||
"\n",
|
||
"This example demonstrates how to use LlamaIndex to interact with the supported [NVIDIA Retrieval QA Ranking Model](https://build.nvidia.com/explore/retrieval) for [retrieval-augmented generation](https://developer.nvidia.com/blog/build-enterprise-retrieval-augmented-generation-apps-with-nvidia-retrieval-qa-embedding-model/) via the `NVIDIARerank` class."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Reranking\n",
|
||
"\n",
|
||
"Reranking is a critical piece of high accuracy, efficient retrieval pipelines.\n",
|
||
"\n",
|
||
"Two important use cases:\n",
|
||
"- Combining results from multiple data sources\n",
|
||
"- Enhancing accuracy for single data sources"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Combining results from multiple sources\n",
|
||
"\n",
|
||
"Consider a pipeline with data from a semantic store, such as VectorStoreIndex, as well as a BM25 store.\n",
|
||
"\n",
|
||
"Each store is queried independently and returns results that the individual store considers to be highly relevant. Figuring out the overall relevance of the results is where reranking comes into play.\n",
|
||
"\n",
|
||
"Follow along with the [Advanced - Hybrid Retriever + Re-Ranking](https://docs.llamaindex.ai/en/stable/examples/retrievers/bm25_retriever/#advanced-hybrid-retriever-re-ranking) use case, substitute [the reranker](https://docs.llamaindex.ai/en/stable/examples/retrievers/bm25_retriever/#re-ranker-setup) with -"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Installation"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"%pip install --upgrade --quiet llama-index-postprocessor-nvidia-rerank llama-index-llms-nvidia llama-index-readers-file"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Setup\n",
|
||
"\n",
|
||
"**To get started:**\n",
|
||
"\n",
|
||
"1. Create a free account with [NVIDIA](https://build.nvidia.com/), which hosts NVIDIA AI Foundation models.\n",
|
||
"\n",
|
||
"2. Click on your model of choice.\n",
|
||
"\n",
|
||
"3. Under Input select the Python tab, and click `Get API Key`. Then click `Generate Key`.\n",
|
||
"\n",
|
||
"4. Copy and save the generated key as NVIDIA_API_KEY. From there, you should have access to the endpoints."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import getpass\n",
|
||
"import os\n",
|
||
"\n",
|
||
"# del os.environ['NVIDIA_API_KEY'] ## delete key and reset\n",
|
||
"if os.environ.get(\"NVIDIA_API_KEY\", \"\").startswith(\"nvapi-\"):\n",
|
||
" print(\"Valid NVIDIA_API_KEY already in environment. Delete to reset\")\n",
|
||
"else:\n",
|
||
" nvapi_key = getpass.getpass(\"NVAPI Key (starts with nvapi-): \")\n",
|
||
" assert nvapi_key.startswith(\n",
|
||
" \"nvapi-\"\n",
|
||
" ), f\"{nvapi_key[:5]}... is not a valid key\"\n",
|
||
" os.environ[\"NVIDIA_API_KEY\"] = nvapi_key"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Working with the NVIDIA API Catalog"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"from llama_index.postprocessor.nvidia_rerank import NVIDIARerank\n",
|
||
"from llama_index.core import SimpleDirectoryReader, Settings, VectorStoreIndex\n",
|
||
"from llama_index.embeddings.nvidia import NVIDIAEmbedding\n",
|
||
"from llama_index.llms.nvidia import NVIDIA\n",
|
||
"from llama_index.core.node_parser import SentenceSplitter\n",
|
||
"from llama_index.core import Settings\n",
|
||
"import os\n",
|
||
"\n",
|
||
"reranker = NVIDIARerank(top_n=4)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"mkdir: cannot create directory ‘data’: File exists\n",
|
||
"--2024-07-03 10:33:17-- https://www.dropbox.com/scl/fi/p33j9112y0ysgwg77fdjz/2021_Housing_Inventory.pdf?rlkey=yyok6bb18s5o31snjd2dxkxz3&dl=0\n",
|
||
"Resolving www.dropbox.com (www.dropbox.com)... 162.125.81.18, 2620:100:6031:18::a27d:5112\n",
|
||
"Connecting to www.dropbox.com (www.dropbox.com)|162.125.81.18|:443... connected.\n",
|
||
"HTTP request sent, awaiting response... 302 Found\n",
|
||
"Location: https://uc471d2c8af935aa4ab2f86937a6.dl.dropboxusercontent.com/cd/0/inline/CV9Hy3nIrjnOf-Fqsgd-YhHcMaj0AHvOQaE1b4sdiKnOBqZL_u9ml6dAGctGxr5I79yD_kI8BNwDtFl_ll_sdfdt0iXcIYosfxaPr2NdbkRAMR6vg9UXuCU8kNEFi0D3Grs/file# [following]\n",
|
||
"--2024-07-03 10:33:18-- https://uc471d2c8af935aa4ab2f86937a6.dl.dropboxusercontent.com/cd/0/inline/CV9Hy3nIrjnOf-Fqsgd-YhHcMaj0AHvOQaE1b4sdiKnOBqZL_u9ml6dAGctGxr5I79yD_kI8BNwDtFl_ll_sdfdt0iXcIYosfxaPr2NdbkRAMR6vg9UXuCU8kNEFi0D3Grs/file\n",
|
||
"Resolving uc471d2c8af935aa4ab2f86937a6.dl.dropboxusercontent.com (uc471d2c8af935aa4ab2f86937a6.dl.dropboxusercontent.com)... 162.125.81.15, 2620:100:6031:15::a27d:510f\n",
|
||
"Connecting to uc471d2c8af935aa4ab2f86937a6.dl.dropboxusercontent.com (uc471d2c8af935aa4ab2f86937a6.dl.dropboxusercontent.com)|162.125.81.15|:443... connected.\n",
|
||
"HTTP request sent, awaiting response... 302 Found\n",
|
||
"Location: /cd/0/inline2/CV9Ugj_mK7TSMb3sw_BdQFrj2rzx-SI2cfGU7-VF4bcW3PdhxO4qw--AXQKUidWtDL_54rViwvbaBGHMvtMEAK_lCIwXXj5XwkKpJKTmP0mDrz8eU2qu0FGyi4uOGfO7TeNLFMFY_bBGUMHMatvKJVPF59Ps94-8LC40ba-Cgv2YKZtcU-UjFpLh-Fnf6emkG-c8eUWB2uKPX_Lx0E4hCENQEPOGOfMhDHU0DC8k6khZiilmLtjXsDJ0H4y3efQ-Fz-VsWCC2FcoGpDcxXGu1Ysp5-mP2eHpH3qOx20d2IrndwN4RGLAqzR6cfsOHPMvoYPyLjOW1322t1O46mXqcjv94OPEEIIHI-2K8xL4pBjLUQ/file [following]\n",
|
||
"--2024-07-03 10:33:18-- https://uc471d2c8af935aa4ab2f86937a6.dl.dropboxusercontent.com/cd/0/inline2/CV9Ugj_mK7TSMb3sw_BdQFrj2rzx-SI2cfGU7-VF4bcW3PdhxO4qw--AXQKUidWtDL_54rViwvbaBGHMvtMEAK_lCIwXXj5XwkKpJKTmP0mDrz8eU2qu0FGyi4uOGfO7TeNLFMFY_bBGUMHMatvKJVPF59Ps94-8LC40ba-Cgv2YKZtcU-UjFpLh-Fnf6emkG-c8eUWB2uKPX_Lx0E4hCENQEPOGOfMhDHU0DC8k6khZiilmLtjXsDJ0H4y3efQ-Fz-VsWCC2FcoGpDcxXGu1Ysp5-mP2eHpH3qOx20d2IrndwN4RGLAqzR6cfsOHPMvoYPyLjOW1322t1O46mXqcjv94OPEEIIHI-2K8xL4pBjLUQ/file\n",
|
||
"Reusing existing connection to uc471d2c8af935aa4ab2f86937a6.dl.dropboxusercontent.com:443.\n",
|
||
"HTTP request sent, awaiting response... 200 OK\n",
|
||
"Length: 4808625 (4.6M) [application/pdf]\n",
|
||
"Saving to: ‘data/housing_data.pdf’\n",
|
||
"\n",
|
||
"data/housing_data.p 100%[===================>] 4.58M 2.68MB/s in 1.7s \n",
|
||
"\n",
|
||
"2024-07-03 10:33:21 (2.68 MB/s) - ‘data/housing_data.pdf’ saved [4808625/4808625]\n",
|
||
"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"!mkdir data\n",
|
||
"!wget \"https://www.dropbox.com/scl/fi/p33j9112y0ysgwg77fdjz/2021_Housing_Inventory.pdf?rlkey=yyok6bb18s5o31snjd2dxkxz3&dl=0\" -O \"data/housing_data.pdf\""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"Settings.text_splitter = SentenceSplitter(chunk_size=500)\n",
|
||
"\n",
|
||
"documents = SimpleDirectoryReader(\"./data\").load_data()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"Settings.embed_model = NVIDIAEmbedding(model=\"NV-Embed-QA\", truncate=\"END\")\n",
|
||
"\n",
|
||
"index = VectorStoreIndex.from_documents(documents)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"Settings.llm = NVIDIA()\n",
|
||
"\n",
|
||
"query_engine = index.as_query_engine(\n",
|
||
" similarity_top_k=20, node_postprocessors=[reranker]\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"The net gain in housing units in the Mission in 2021 was not specified in the provided context information.\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"response = query_engine.query(\n",
|
||
" \"What was the net gain in housing units in the Mission in 2021?\"\n",
|
||
")\n",
|
||
"print(response)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Working with NVIDIA NIMs\n",
|
||
"\n",
|
||
"In addition to connecting to hosted [NVIDIA NIMs](https://ai.nvidia.com), this connector can be used to connect to local NIM instances. This helps you take your applications local when necessary.\n",
|
||
"\n",
|
||
"For instructions on how to set up local NIM instances, refer to [NVIDIA NIM](https://developer.nvidia.com/nim)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from llama_index.postprocessor.nvidia_rerank import NVIDIARerank\n",
|
||
"\n",
|
||
"# Connect to a rerank NIM running at localhost:1976\n",
|
||
"reranker = NVIDIARerank(base_url=\"http://localhost:1976/v1\")"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 2
|
||
}
|