wehub-resource-sync
a0c8464e58
Build Package / build (ubuntu-latest) (push) Failing after 1s
CodeQL / Analyze (python) (push) Failing after 1s
Core Typecheck / core-typecheck (push) Failing after 1s
Linting / lint (push) Failing after 1s
llama-dev tests / test-llama-dev (push) Failing after 1s
Publish Sub-Package to PyPI if Needed / publish_subpackage_if_needed (push) Has been skipped
Sync Docs to Developer Hub / sync-docs (push) Failing after 0s
Build Package / build (windows-latest) (push) Has been cancelled
368 lines
14 KiB
Plaintext
368 lines
14 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# TablestoreVectorStore\n",
|
|
"\n",
|
|
"> [Tablestore](https://www.aliyun.com/product/ots) is a fully managed NoSQL cloud database service that enables storage of a massive amount of structured\n",
|
|
"and semi-structured data.\n",
|
|
"\n",
|
|
"This notebook shows how to use functionality related to the `Tablestore` vector database.\n",
|
|
"\n",
|
|
"To use Tablestore, you must create an instance.\n",
|
|
"Here are the [creating instance instructions](https://help.aliyun.com/zh/tablestore/getting-started/manage-the-wide-column-model-in-the-tablestore-console)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Install"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install llama-index-vector-stores-tablestore"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import getpass\n",
|
|
"import os\n",
|
|
"\n",
|
|
"os.environ[\"end_point\"] = getpass.getpass(\"Tablestore end_point:\")\n",
|
|
"os.environ[\"instance_name\"] = getpass.getpass(\"Tablestore instance_name:\")\n",
|
|
"os.environ[\"access_key_id\"] = getpass.getpass(\"Tablestore access_key_id:\")\n",
|
|
"os.environ[\"access_key_secret\"] = getpass.getpass(\n",
|
|
" \"Tablestore access_key_secret:\"\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Example\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Create vector store. "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"\n",
|
|
"from llama_index.core import MockEmbedding\n",
|
|
"from llama_index.core.schema import TextNode\n",
|
|
"from llama_index.core.vector_stores import (\n",
|
|
" VectorStoreQuery,\n",
|
|
" MetadataFilters,\n",
|
|
" MetadataFilter,\n",
|
|
" FilterCondition,\n",
|
|
" FilterOperator,\n",
|
|
")\n",
|
|
"from llama_index.core.vector_stores.types import (\n",
|
|
" VectorStoreQueryMode,\n",
|
|
")\n",
|
|
"from tablestore import FieldSchema, FieldType, VectorMetricType\n",
|
|
"\n",
|
|
"from llama_index.vector_stores.tablestore import TablestoreVectorStore\n",
|
|
"\n",
|
|
"vector_dimension = 4\n",
|
|
"\n",
|
|
"store = TablestoreVectorStore(\n",
|
|
" endpoint=os.getenv(\"end_point\"),\n",
|
|
" instance_name=os.getenv(\"instance_name\"),\n",
|
|
" access_key_id=os.getenv(\"access_key_id\"),\n",
|
|
" access_key_secret=os.getenv(\"access_key_secret\"),\n",
|
|
" vector_dimension=vector_dimension,\n",
|
|
" vector_metric_type=VectorMetricType.VM_COSINE,\n",
|
|
" # optional: custom metadata mapping is used to filter non-vector fields.\n",
|
|
" metadata_mappings=[\n",
|
|
" FieldSchema(\n",
|
|
" \"type\", FieldType.KEYWORD, index=True, enable_sort_and_agg=True\n",
|
|
" ),\n",
|
|
" FieldSchema(\n",
|
|
" \"time\", FieldType.LONG, index=True, enable_sort_and_agg=True\n",
|
|
" ),\n",
|
|
" ],\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Create table and index."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"store.create_table_if_not_exist()\n",
|
|
"store.create_search_index_if_not_exist()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"New a mock embedding for test."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"embedder = MockEmbedding(vector_dimension)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Prepare some docs."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"texts = [\n",
|
|
" TextNode(\n",
|
|
" id_=\"1\",\n",
|
|
" text=\"The lives of two mob hitmen, a boxer, a gangster and his wife, and a pair of diner bandits intertwine in four tales of violence and redemption.\",\n",
|
|
" metadata={\"type\": \"a\", \"time\": 1995},\n",
|
|
" ),\n",
|
|
" TextNode(\n",
|
|
" id_=\"2\",\n",
|
|
" text=\"When the menace known as the Joker wreaks havoc and chaos on the people of Gotham, Batman must accept one of the greatest psychological and physical tests of his ability to fight injustice.\",\n",
|
|
" metadata={\"type\": \"a\", \"time\": 1990},\n",
|
|
" ),\n",
|
|
" TextNode(\n",
|
|
" id_=\"3\",\n",
|
|
" text=\"An insomniac office worker and a devil-may-care soapmaker form an underground fight club that evolves into something much, much more.\",\n",
|
|
" metadata={\"type\": \"a\", \"time\": 2009},\n",
|
|
" ),\n",
|
|
" TextNode(\n",
|
|
" id_=\"4\",\n",
|
|
" text=\"A thief who steals corporate secrets through the use of dream-sharing technology is given the inverse task of planting an idea into thed of a C.E.O.\",\n",
|
|
" metadata={\"type\": \"a\", \"time\": 2023},\n",
|
|
" ),\n",
|
|
" TextNode(\n",
|
|
" id_=\"5\",\n",
|
|
" text=\"A computer hacker learns from mysterious rebels about the true nature of his reality and his role in the war against its controllers.\",\n",
|
|
" metadata={\"type\": \"b\", \"time\": 2018},\n",
|
|
" ),\n",
|
|
" TextNode(\n",
|
|
" id_=\"6\",\n",
|
|
" text=\"Two detectives, a rookie and a veteran, hunt a serial killer who uses the seven deadly sins as his motives.\",\n",
|
|
" metadata={\"type\": \"c\", \"time\": 2010},\n",
|
|
" ),\n",
|
|
" TextNode(\n",
|
|
" id_=\"7\",\n",
|
|
" text=\"An organized crime dynasty's aging patriarch transfers control of his clandestine empire to his reluctant son.\",\n",
|
|
" metadata={\"type\": \"a\", \"time\": 2023},\n",
|
|
" ),\n",
|
|
"]\n",
|
|
"for t in texts:\n",
|
|
" t.embedding = embedder.get_text_embedding(t.text)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Write some docs."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"['1', '2', '3', '4', '5', '6', '7']"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"store.add(texts)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Delete docs."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"store.delete(\"1\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Query with filters."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"VectorStoreQueryResult(nodes=[TextNode(id_='1', embedding=[0.5, 0.5, 0.5, 0.5], metadata={'time': 1995, 'type': 'a'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, metadata_template='{key}: {value}', metadata_separator='\\n', text='The lives of two mob hitmen, a boxer, a gangster and his wife, and a pair of diner bandits intertwine in four tales of violence and redemption.', mimetype='text/plain', start_char_idx=None, end_char_idx=None, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'), TextNode(id_='2', embedding=[0.5, 0.5, 0.5, 0.5], metadata={'time': 1990, 'type': 'a'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, metadata_template='{key}: {value}', metadata_separator='\\n', text='When the menace known as the Joker wreaks havoc and chaos on the people of Gotham, Batman must accept one of the greatest psychological and physical tests of his ability to fight injustice.', mimetype='text/plain', start_char_idx=None, end_char_idx=None, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'), TextNode(id_='3', embedding=[0.5, 0.5, 0.5, 0.5], metadata={'time': 2009, 'type': 'a'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, metadata_template='{key}: {value}', metadata_separator='\\n', text='An insomniac office worker and a devil-may-care soapmaker form an underground fight club that evolves into something much, much more.', mimetype='text/plain', start_char_idx=None, end_char_idx=None, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}')], similarities=[1.0, 1.0, 1.0], ids=['1', '2', '3'])"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"store.query(\n",
|
|
" query=VectorStoreQuery(\n",
|
|
" query_embedding=embedder.get_text_embedding(\"nature fight physical\"),\n",
|
|
" similarity_top_k=5,\n",
|
|
" filters=MetadataFilters(\n",
|
|
" filters=[\n",
|
|
" MetadataFilter(\n",
|
|
" key=\"type\", value=\"a\", operator=FilterOperator.EQ\n",
|
|
" ),\n",
|
|
" MetadataFilter(\n",
|
|
" key=\"time\", value=2020, operator=FilterOperator.LTE\n",
|
|
" ),\n",
|
|
" ],\n",
|
|
" condition=FilterCondition.AND,\n",
|
|
" ),\n",
|
|
" ),\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Full text search: query mode = TEXT."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"VectorStoreQueryResult(nodes=[TextNode(id_='5', embedding=[0.5, 0.5, 0.5, 0.5], metadata={'time': 2018, 'type': 'b'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, metadata_template='{key}: {value}', metadata_separator='\\n', text='A computer hacker learns from mysterious rebels about the true nature of his reality and his role in the war against its controllers.', mimetype='text/plain', start_char_idx=None, end_char_idx=None, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}')], similarities=[2.673976421356201], ids=['5'])\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"query_result = store.query(\n",
|
|
" query=VectorStoreQuery(\n",
|
|
" mode=VectorStoreQueryMode.TEXT_SEARCH,\n",
|
|
" query_str=\"computer\",\n",
|
|
" similarity_top_k=5,\n",
|
|
" ),\n",
|
|
")\n",
|
|
"print(query_result)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"HYBRID query."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"VectorStoreQueryResult(nodes=[TextNode(id_='1', embedding=[0.5, 0.5, 0.5, 0.5], metadata={'time': 1995, 'type': 'a'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, metadata_template='{key}: {value}', metadata_separator='\\n', text='The lives of two mob hitmen, a boxer, a gangster and his wife, and a pair of diner bandits intertwine in four tales of violence and redemption.', mimetype='text/plain', start_char_idx=None, end_char_idx=None, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'), TextNode(id_='2', embedding=[0.5, 0.5, 0.5, 0.5], metadata={'time': 1990, 'type': 'a'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, metadata_template='{key}: {value}', metadata_separator='\\n', text='When the menace known as the Joker wreaks havoc and chaos on the people of Gotham, Batman must accept one of the greatest psychological and physical tests of his ability to fight injustice.', mimetype='text/plain', start_char_idx=None, end_char_idx=None, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'), TextNode(id_='3', embedding=[0.5, 0.5, 0.5, 0.5], metadata={'time': 2009, 'type': 'a'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, metadata_template='{key}: {value}', metadata_separator='\\n', text='An insomniac office worker and a devil-may-care soapmaker form an underground fight club that evolves into something much, much more.', mimetype='text/plain', start_char_idx=None, end_char_idx=None, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'), TextNode(id_='4', embedding=[0.5, 0.5, 0.5, 0.5], metadata={'time': 2023, 'type': 'a'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, metadata_template='{key}: {value}', metadata_separator='\\n', text='A thief who steals corporate secrets through the use of dream-sharing technology is given the inverse task of planting an idea into thed of a C.E.O.', mimetype='text/plain', start_char_idx=None, end_char_idx=None, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'), TextNode(id_='5', embedding=[0.5, 0.5, 0.5, 0.5], metadata={'time': 2018, 'type': 'b'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, metadata_template='{key}: {value}', metadata_separator='\\n', text='A computer hacker learns from mysterious rebels about the true nature of his reality and his role in the war against its controllers.', mimetype='text/plain', start_char_idx=None, end_char_idx=None, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}')], similarities=[1.0, 1.0, 1.0, 1.0, 1.0], ids=['1', '2', '3', '4', '5'])\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"query_result = store.query(\n",
|
|
" query=VectorStoreQuery(\n",
|
|
" mode=VectorStoreQueryMode.HYBRID,\n",
|
|
" query_embedding=embedder.get_text_embedding(\"nature fight physical\"),\n",
|
|
" query_str=\"python\",\n",
|
|
" similarity_top_k=5,\n",
|
|
" ),\n",
|
|
")\n",
|
|
"print(query_result)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 4
|
|
}
|