wehub-resource-sync
a0c8464e58
Build Package / build (ubuntu-latest) (push) Failing after 1s
CodeQL / Analyze (python) (push) Failing after 1s
Core Typecheck / core-typecheck (push) Failing after 1s
Linting / lint (push) Failing after 1s
llama-dev tests / test-llama-dev (push) Failing after 1s
Publish Sub-Package to PyPI if Needed / publish_subpackage_if_needed (push) Has been skipped
Sync Docs to Developer Hub / sync-docs (push) Failing after 0s
Build Package / build (windows-latest) (push) Has been cancelled
429 lines
12 KiB
Plaintext
429 lines
12 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "600e8429",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/vector_stores/AlibabaCloudOpenSearchIndexDemo.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
|
|
"<a href=\"https://gallery.pai-ml.com/#/import/https://github.com/run-llama/llama_index/blob/main/docs/examples/vector_stores/AlibabaCloudOpenSearchIndexDemo.ipynb\" target=\"_parent\"><img src=\"https://gallery.pai-ml.com/assets/open-in-dsw.svg\" alt=\"Open in PAI-DSW\"/></a>"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "307804a3-c02b-4a57-ac0d-172c30ddc851",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Alibaba Cloud OpenSearch Vector Store"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "43f9df89",
|
|
"metadata": {},
|
|
"source": [
|
|
">[Alibaba Cloud OpenSearch Vector Search Edition](https://help.aliyun.com/zh/open-search/vector-search-edition/product-overview) is a large-scale distributed search engine that is developed by Alibaba Group. Alibaba Cloud OpenSearch Vector Search Edition provides search services for the entire Alibaba Group, including Taobao, Tmall, Cainiao, Youku, and other e-commerce platforms that are provided for customers in regions outside the Chinese mainland. Alibaba Cloud OpenSearch Vector Search Edition is also a base engine of Alibaba Cloud OpenSearch. After years of development, Alibaba Cloud OpenSearch Vector Search Edition has met the business requirements for high availability, high timeliness, and cost-effectiveness. Alibaba Cloud OpenSearch Vector Search Edition also provides an automated O&M system on which you can build a custom search service based on your business features.\n",
|
|
"\n",
|
|
"To run, you should have a instance."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5eeaa2f4",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Setup"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "47bbdd33",
|
|
"metadata": {},
|
|
"source": [
|
|
"If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "3bd0b321",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install llama-index-vector-stores-alibabacloud-opensearch"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "4f6b3761",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install llama-index"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "d48af8e1",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import logging\n",
|
|
"import sys\n",
|
|
"\n",
|
|
"logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
|
|
"logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f9074027",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Please provide OpenAI access key\n",
|
|
"\n",
|
|
"In order use embeddings by OpenAI you need to supply an OpenAI API Key:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "6b6260f3",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import openai\n",
|
|
"\n",
|
|
"OPENAI_API_KEY = getpass.getpass(\"OpenAI API Key:\")\n",
|
|
"openai.api_key = OPENAI_API_KEY"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "4b00ea0d",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Download Data"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "6d21ebdc",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!mkdir -p 'data/paul_graham/'\n",
|
|
"!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "8ee4473a-094f-4d0a-a825-e1213db07240",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Load documents"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "0a2bcc07",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from llama_index.core import SimpleDirectoryReader\n",
|
|
"from IPython.display import Markdown, display"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "68cbd239-880e-41a3-98d8-dbb3fab55431",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Total documents: 1\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# load documents\n",
|
|
"documents = SimpleDirectoryReader(\"./data/paul_graham\").load_data()\n",
|
|
"print(f\"Total documents: {len(documents)}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "630c0bdf",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Create the Alibaba Cloud OpenSearch Vector Store object:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2a410b4c",
|
|
"metadata": {},
|
|
"source": [
|
|
"To run the next step, you should have a Alibaba Cloud OpenSearch Vector Service instance, and configure a table."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "722c4c39",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# if run fllowing cells raise async io exception, run this\n",
|
|
"import nest_asyncio\n",
|
|
"\n",
|
|
"nest_asyncio.apply()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "ba1558b3",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# initialize without metadata filter\n",
|
|
"from llama_index.core import StorageContext, VectorStoreIndex\n",
|
|
"from llama_index.vector_stores.alibabacloud_opensearch import (\n",
|
|
" AlibabaCloudOpenSearchStore,\n",
|
|
" AlibabaCloudOpenSearchConfig,\n",
|
|
")\n",
|
|
"\n",
|
|
"config = AlibabaCloudOpenSearchConfig(\n",
|
|
" endpoint=\"*****\",\n",
|
|
" instance_id=\"*****\",\n",
|
|
" username=\"your_username\",\n",
|
|
" password=\"your_password\",\n",
|
|
" table_name=\"llama\",\n",
|
|
")\n",
|
|
"\n",
|
|
"vector_store = AlibabaCloudOpenSearchStore(config)\n",
|
|
"storage_context = StorageContext.from_defaults(vector_store=vector_store)\n",
|
|
"index = VectorStoreIndex.from_documents(\n",
|
|
" documents, storage_context=storage_context\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "04304299-fc3e-40a0-8600-f50c3292767e",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Query Index"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "35369eda",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# set Logging to DEBUG for more detailed outputs\n",
|
|
"query_engine = index.as_query_engine()\n",
|
|
"response = query_engine.query(\"What did the author do growing up?\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "bedbb693-725f-478f-be26-fa7180ea38b2",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/markdown": [
|
|
"<b>Before college, the author worked on writing and programming. They wrote short stories and tried writing programs on the IBM 1401 in 9th grade using an early version of Fortran.</b>"
|
|
],
|
|
"text/plain": [
|
|
"<IPython.core.display.Markdown object>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"display(Markdown(f\"<b>{response}</b>\"))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3e723315",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Connecting to an existing store\n",
|
|
"\n",
|
|
"Since this store is backed by Alibaba Cloud OpenSearch, it is persistent by definition. So, if you want to connect to a store that was created and populated previously, here is how:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "b133f816",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from llama_index.core import VectorStoreIndex\n",
|
|
"from llama_index.vector_stores.alibabacloud_opensearch import (\n",
|
|
" AlibabaCloudOpenSearchStore,\n",
|
|
" AlibabaCloudOpenSearchConfig,\n",
|
|
")\n",
|
|
"\n",
|
|
"config = AlibabaCloudOpenSearchConfig(\n",
|
|
" endpoint=\"***\",\n",
|
|
" instance_id=\"***\",\n",
|
|
" username=\"your_username\",\n",
|
|
" password=\"your_password\",\n",
|
|
" table_name=\"llama\",\n",
|
|
")\n",
|
|
"\n",
|
|
"vector_store = AlibabaCloudOpenSearchStore(config)\n",
|
|
"\n",
|
|
"# Create index from existing stored vectors\n",
|
|
"index = VectorStoreIndex.from_vector_store(vector_store)\n",
|
|
"query_engine = index.as_query_engine()\n",
|
|
"response = query_engine.query(\n",
|
|
" \"What did the author study prior to working on AI?\"\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(f\"<b>{response}</b>\"))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bfcbcd74",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Metadata filtering\n",
|
|
"\n",
|
|
"The Alibaba Cloud OpenSearch vector store support metadata filtering at query time. The following cells, which work on a brand new table, demonstrate this feature.\n",
|
|
"\n",
|
|
"In this demo, for the sake of brevity, a single source document is loaded (the `../data/paul_graham/paul_graham_essay.txt` text file). Nevertheless, you will attach some custom metadata to the document to illustrate how you can can restrict queries with conditions on the metadata attached to the documents."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "f9a7e108",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from llama_index.core import StorageContext, VectorStoreIndex\n",
|
|
"from llama_index.vector_stores.alibabacloud_opensearch import (\n",
|
|
" AlibabaCloudOpenSearchStore,\n",
|
|
" AlibabaCloudOpenSearchConfig,\n",
|
|
")\n",
|
|
"\n",
|
|
"config = AlibabaCloudOpenSearchConfig(\n",
|
|
" endpoint=\"****\",\n",
|
|
" instance_id=\"****\",\n",
|
|
" username=\"your_username\",\n",
|
|
" password=\"your_password\",\n",
|
|
" table_name=\"llama\",\n",
|
|
")\n",
|
|
"\n",
|
|
"md_storage_context = StorageContext.from_defaults(\n",
|
|
" vector_store=AlibabaCloudOpenSearchStore(config)\n",
|
|
")\n",
|
|
"\n",
|
|
"\n",
|
|
"def my_file_metadata(file_name: str):\n",
|
|
" \"\"\"Depending on the input file name, associate a different metadata.\"\"\"\n",
|
|
" if \"essay\" in file_name:\n",
|
|
" source_type = \"essay\"\n",
|
|
" elif \"dinosaur\" in file_name:\n",
|
|
" # this (unfortunately) will not happen in this demo\n",
|
|
" source_type = \"dinos\"\n",
|
|
" else:\n",
|
|
" source_type = \"other\"\n",
|
|
" return {\"source_type\": source_type}\n",
|
|
"\n",
|
|
"\n",
|
|
"# Load documents and build index\n",
|
|
"md_documents = SimpleDirectoryReader(\n",
|
|
" \"../data/paul_graham\", file_metadata=my_file_metadata\n",
|
|
").load_data()\n",
|
|
"md_index = VectorStoreIndex.from_documents(\n",
|
|
" md_documents, storage_context=md_storage_context\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "de69be23",
|
|
"metadata": {},
|
|
"source": [
|
|
"Add filter to query engine:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "6dd49f3a",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from llama_index.core.vector_stores import MetadataFilter, MetadataFilters\n",
|
|
"\n",
|
|
"md_query_engine = md_index.as_query_engine(\n",
|
|
" filters=MetadataFilters(\n",
|
|
" filters=[MetadataFilter(key=\"source_type\", value=\"essay\")]\n",
|
|
" )\n",
|
|
")\n",
|
|
"md_response = md_query_engine.query(\n",
|
|
" \"How long it took the author to write his thesis?\"\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(f\"<b>{md_response}</b>\"))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "79a14aaf",
|
|
"metadata": {},
|
|
"source": [
|
|
"To test that the filtering is at play, try to change it to use only `\"dinos\"` documents... there will be no answer this time :)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|