wehub-resource-sync
a0c8464e58
Build Package / build (ubuntu-latest) (push) Failing after 1s
CodeQL / Analyze (python) (push) Failing after 1s
Core Typecheck / core-typecheck (push) Failing after 1s
Linting / lint (push) Failing after 1s
llama-dev tests / test-llama-dev (push) Failing after 1s
Publish Sub-Package to PyPI if Needed / publish_subpackage_if_needed (push) Has been skipped
Sync Docs to Developer Hub / sync-docs (push) Failing after 0s
Build Package / build (windows-latest) (push) Has been cancelled
270 lines
9.9 KiB
Plaintext
270 lines
9.9 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "c6218e91",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/llm/xinference_local_deployment.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "7096589b-daaf-440a-b89d-b4956f2db4b2",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Xorbits Inference\n",
|
|
"\n",
|
|
"In this demo notebook, we show how to use Xorbits Inference (Xinference for short) to deploy local LLMs in three steps.\n",
|
|
"\n",
|
|
"We will be using the Llama 2 chat model in GGML format in the example, but the code should be easily transfrerable to all LLM chat models supported by Xinference. Below are a few examples:\n",
|
|
"\n",
|
|
"| Name | Type | Language | Format | Size (in billions) | Quantization |\n",
|
|
"|---------------|------------------|----------|---------|--------------------|-----------------------------------------|\n",
|
|
"| llama-2-chat | RLHF Model | en | ggmlv3 | 7, 13, 70 | 'q2_K', 'q3_K_L', ... , 'q6_K', 'q8_0' |\n",
|
|
"| chatglm | SFT Model | en, zh | ggmlv3 | 6 | 'q4_0', 'q4_1', 'q5_0', 'q5_1', 'q8_0' |\n",
|
|
"| chatglm2 | SFT Model | en, zh | ggmlv3 | 6 | 'q4_0', 'q4_1', 'q5_0', 'q5_1', 'q8_0' |\n",
|
|
"| wizardlm-v1.0 | SFT Model | en | ggmlv3 | 7, 13, 33 | 'q2_K', 'q3_K_L', ... , 'q6_K', 'q8_0' |\n",
|
|
"| wizardlm-v1.1 | SFT Model | en | ggmlv3 | 13 | 'q2_K', 'q3_K_L', ... , 'q6_K', 'q8_0' |\n",
|
|
"| vicuna-v1.3 | SFT Model | en | ggmlv3 | 7, 13 | 'q2_K', 'q3_K_L', ... , 'q6_K', 'q8_0' |\n",
|
|
"\n",
|
|
"The latest complete list of supported models can be found in Xorbits Inference's [official GitHub page](https://github.com/xorbitsai/inference/blob/main/README.md)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d8cfbe6f-4c50-4c4f-90f9-03bb91201ef5",
|
|
"metadata": {},
|
|
"source": [
|
|
"## <span style=\"font-size: xx-large;;\">🤖 </span> Install Xinference\n",
|
|
"\n",
|
|
"i. Run `pip install \"xinference[all]\"` in a terminal window.\n",
|
|
"\n",
|
|
"ii. After installation is complete, restart this jupyter notebook.\n",
|
|
"\n",
|
|
"iii. Run `xinference` in a new terminal window.\n",
|
|
"\n",
|
|
"iv. You should see something similar to the following output:\n",
|
|
"\n",
|
|
"```\n",
|
|
"INFO:xinference:Xinference successfully started. Endpoint: http://127.0.0.1:9997\n",
|
|
"INFO:xinference.core.service:Worker 127.0.0.1:21561 has been added successfully\n",
|
|
"INFO:xinference.deploy.worker:Xinference worker successfully started.\n",
|
|
"```\n",
|
|
"\n",
|
|
"v. In the endpoint description, locate the endpoint port number after the colon. In the above case it is `9997`.\n",
|
|
"\n",
|
|
"vi. Set the port number with the following cell:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "790947f0",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install llama-index-llms-xinference"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "5d520d56",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"port = 9997 # replace with your endpoint port number"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "93139076",
|
|
"metadata": {},
|
|
"source": [
|
|
"## <span style=\"font-size: xx-large;;\">🚀 </span> Launch Local Models\n",
|
|
"\n",
|
|
"In this step, we begin with importing the relevant libraries from `llama_index`"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "1a6c1642",
|
|
"metadata": {},
|
|
"source": [
|
|
"If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "97577a07",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!pip install llama-index"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "fd1d259c",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# If Xinference can not be imported, you may need to restart jupyter notebook\n",
|
|
"from llama_index.core import SummaryIndex\n",
|
|
"from llama_index.core import (\n",
|
|
" TreeIndex,\n",
|
|
" VectorStoreIndex,\n",
|
|
" KeywordTableIndex,\n",
|
|
" KnowledgeGraphIndex,\n",
|
|
" SimpleDirectoryReader,\n",
|
|
")\n",
|
|
"from llama_index.llms.xinference import Xinference\n",
|
|
"from xinference.client import RESTfulClient\n",
|
|
"from IPython.display import Markdown, display"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "7a2dce47",
|
|
"metadata": {},
|
|
"source": [
|
|
"Then, we launch a model and use it. This allows us to connect the model to documents and queries in later steps.\n",
|
|
"\n",
|
|
"Feel free to change the parameters for better performance! In order to achieve optimal results, it is recommended to use models above 13B in size. That being said, 7B models is more than enough for this short demo.\n",
|
|
"\n",
|
|
"Here are some more parameter options for the Llama 2 chat model in GGML format, listed from the least space-consuming to the most resource-intensive but high-performing. \n",
|
|
"\n",
|
|
"\n",
|
|
"<span style=\"font-weight: bold; ;\">model_size_in_billions:</span> \n",
|
|
"\n",
|
|
"`7`, `13`, `70`\n",
|
|
"\n",
|
|
"<span style=\"font-weight: bold; ;\">quantization for 7B and 13B models:</span> \n",
|
|
"\n",
|
|
"`q2_K`, `q3_K_L`, `q3_K_M`, `q3_K_S`, `q4_0`, `q4_1`, `q4_K_M`, `q4_K_S`, `q5_0`, `q5_1`, `q5_K_M`, `q5_K_S`, `q6_K`, `q8_0`\n",
|
|
"\n",
|
|
"<span style=\"font-weight: bold; ;\">quantizations for 70B models:</span>\n",
|
|
"\n",
|
|
"`q4_0`\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "b48c6d7a-7a38-440b-8ecb-f43f9050ee54",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Define a client to send commands to xinference\n",
|
|
"client = RESTfulClient(f\"http://localhost:{port}\")\n",
|
|
"\n",
|
|
"# Download and Launch a model, this may take a while the first time\n",
|
|
"model_uid = client.launch_model(\n",
|
|
" model_name=\"llama-2-chat\",\n",
|
|
" model_size_in_billions=7,\n",
|
|
" model_format=\"ggmlv3\",\n",
|
|
" quantization=\"q2_K\",\n",
|
|
")\n",
|
|
"\n",
|
|
"# Initiate Xinference object to use the LLM\n",
|
|
"llm = Xinference(\n",
|
|
" endpoint=f\"http://localhost:{port}\",\n",
|
|
" model_uid=model_uid,\n",
|
|
" temperature=0.0,\n",
|
|
" max_tokens=512,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "094a02b7",
|
|
"metadata": {},
|
|
"source": [
|
|
"## <span style=\"font-size: xx-large;;\">🕺 </span> Index the Data... and Chat!\n",
|
|
"\n",
|
|
"In this step, we combine the model and the data to create a query engine. The query engine can then be used as a chat bot, answering our queries based on the given data.\n",
|
|
"\n",
|
|
"We will be using `VetorStoreIndex` since it is relatively fast. That being said, feel free to change the index for different experiences. Here are some available indexes already imported from the previous step:\n",
|
|
"\n",
|
|
"`ListIndex`, `TreeIndex`, `VetorStoreIndex`, `KeywordTableIndex`, `KnowledgeGraphIndex`\n",
|
|
"\n",
|
|
"To change index, simply replace `VetorStoreIndex` with another index in the following code. \n",
|
|
"\n",
|
|
"The latest complete list of all available indexes can be found in Llama Index's [official Docs](https://gpt-index.readthedocs.io/en/latest/core_modules/data_modules/index/modules.html)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "708b323e-d314-4b83-864b-22a1ead60de9",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# create index from the data\n",
|
|
"documents = SimpleDirectoryReader(\"../data/paul_graham\").load_data()\n",
|
|
"\n",
|
|
"# change index name in the following line\n",
|
|
"index = VectorStoreIndex.from_documents(documents=documents)\n",
|
|
"\n",
|
|
"# create the query engine\n",
|
|
"query_engine = index.as_query_engine(llm=llm)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "548174a2",
|
|
"metadata": {},
|
|
"source": [
|
|
"We can optionally set the temperature and the max answer length (in tokens) directly through the `Xinference` object before asking a question. This allows us to change parameters for different questions without rebuilding the query engine every time.\n",
|
|
"\n",
|
|
"`temperature` is a number between 0 and 1 that controls the randomness of responses. Higher values increase creativity but may lead to off-topic replies. Setting to zero guarentees the same response every time.\n",
|
|
"\n",
|
|
"`max_tokens` is an integer that sets an upper bound for the response length. Increase it if answers seem cut off, but be aware that too long a response may exceed the context window and cause errors.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "e0b32ddb",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# optionally, update the temperature and max answer length (in tokens)\n",
|
|
"llm.__dict__.update({\"temperature\": 0.0})\n",
|
|
"llm.__dict__.update({\"max_tokens\": 2048})\n",
|
|
"\n",
|
|
"# ask a question and display the answer\n",
|
|
"question = \"What did the author do after his time at Y Combinator?\"\n",
|
|
"\n",
|
|
"response = query_engine.query(question)\n",
|
|
"display(Markdown(f\"<b>{response}</b>\"))"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|