wehub-resource-sync
a0c8464e58
Build Package / build (ubuntu-latest) (push) Failing after 1s
CodeQL / Analyze (python) (push) Failing after 1s
Core Typecheck / core-typecheck (push) Failing after 1s
Linting / lint (push) Failing after 1s
llama-dev tests / test-llama-dev (push) Failing after 1s
Publish Sub-Package to PyPI if Needed / publish_subpackage_if_needed (push) Has been skipped
Sync Docs to Developer Hub / sync-docs (push) Failing after 0s
Build Package / build (windows-latest) (push) Has been cancelled
497 lines
16 KiB
Plaintext
497 lines
16 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "dbe42ae0f64577e2",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Alibaba Cloud MySQL\n",
|
|
"\n",
|
|
">Alibaba Cloud MySQL, also named as [ApsaraDB RDS for MySQL](https://www.alibabacloud.com/help/en/rds/apsaradb-rds-for-mysql/overview-3). ApsaraDB RDS for MySQL is an online database service that is based on a branch of MySQL source code and offers high performance. ApsaraDB RDS for MySQL is a proven solution that has handled large volumes of concurrent traffic during Double 11. ApsaraDB RDS for MySQL provides basic features such as whitelist configuration, backup and recovery, Transparent Data Encryption (TDE), data migration, and management of instances, accounts, and databases. For more information, see [RDS MySQL Feature Overview](https://www.alibabacloud.com/help/en/rds/apsaradb-rds-for-mysql/features).\n",
|
|
"\n",
|
|
"To run this notebook you need a ApsaraDB RDS MySQL instance running in the cloud, create an account and create needed databases. You can refer to [this link](https://www.alibabacloud.com/help/en/rds/apsaradb-rds-for-mysql/step-1-create-an-apsaradb-rds-for-mysql-instance-and-configure-databases).\n",
|
|
"\n",
|
|
"In this notebook, we need to create databases called `llama_index_test` and `llama_index_meta_test` in your ApsaraDB RDS MySQL instance."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bf4b89a516bcdbae",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Setup"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ee2789ebac3cc94a",
|
|
"metadata": {},
|
|
"source": [
|
|
"If you're opening this Notebook on colab, you will probably need to ensure you have `llama-index` installed:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "3233a304b6fe81b3",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!pip install llama-index"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "5dcc9ec53d871101",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install llama-index-vector-stores-alibabacloud-mysql"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "af593fb347f26e09",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# choose dashscope as embedding and llm model, your can also use default openai or other model to test\n",
|
|
"%pip install llama-index-embeddings-dashscope\n",
|
|
"%pip install llama-index-llms-dashscope"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5a426b380006bb53",
|
|
"metadata": {},
|
|
"source": [
|
|
"Config dashscope embedding and llm model, your can also use default openai or other model to test.\n",
|
|
"If you choose to use dashscope model, you can get your api key [here](https://modelstudio.console.aliyun.com/?tab=dashboard#/api-key), and set it in the following code:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "4e842a7d94100355",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!export DASHSCOPE_API_KEY=\"your_api_key\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d3cffc8726063d8a",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Download example data"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "c296e1d96b56a89",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!mkdir -p 'data/paul_graham/'\n",
|
|
"!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d0289f55c142c594",
|
|
"metadata": {},
|
|
"source": [
|
|
"## RAG Demo using Alibaba Cloud MySQL"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "99d05c0522aa110e",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Simple Query"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f23a8dc270c6742",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Load Data for Simple Query"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "4fdc5d4c3e5791d0",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"\n",
|
|
"DASHSCOPE_API_KEY = os.environ.get(\"DASHSCOPE_API_KEY\")\n",
|
|
"\n",
|
|
"from llama_index.core import (\n",
|
|
" VectorStoreIndex,\n",
|
|
" SimpleDirectoryReader,\n",
|
|
" StorageContext,\n",
|
|
")\n",
|
|
"from llama_index.vector_stores.alibabacloud_mysql.base import (\n",
|
|
" AlibabaCloudMySQLVectorStore,\n",
|
|
")\n",
|
|
"\n",
|
|
"# set Embbeding model\n",
|
|
"from llama_index.core import Settings\n",
|
|
"from llama_index.embeddings.dashscope import DashScopeEmbedding\n",
|
|
"\n",
|
|
"Settings.embed_model = DashScopeEmbedding(api_key=DASHSCOPE_API_KEY)\n",
|
|
"\n",
|
|
"# config llm model\n",
|
|
"from llama_index.llms.dashscope import DashScope, DashScopeGenerationModels\n",
|
|
"\n",
|
|
"dashscope_llm = DashScope(\n",
|
|
" model_name=DashScopeGenerationModels.QWEN_MAX, api_key=DASHSCOPE_API_KEY\n",
|
|
")\n",
|
|
"\n",
|
|
"documents = SimpleDirectoryReader(\"data/paul_graham/\").load_data()\n",
|
|
"print(f\"Total documents: {len(documents)}\")\n",
|
|
"print(f\"First document, id: {documents[0].doc_id}\")\n",
|
|
"print(f\"First document, hash: {documents[0].hash}\")\n",
|
|
"print(\n",
|
|
" \"First document, text\"\n",
|
|
" f\" ({len(documents[0].text)} characters):\\n{'='*20}\\n{documents[0].text[:360]} ...\"\n",
|
|
")\n",
|
|
"\n",
|
|
"print(\n",
|
|
" \"\"\"\n",
|
|
"#################\n",
|
|
"# simple generate vector\n",
|
|
"#################\"\"\"\n",
|
|
")\n",
|
|
"client = AlibabaCloudMySQLVectorStore.from_params(\n",
|
|
" host=\"rm-***.mysql.***.rds.aliyuncs.com\",\n",
|
|
" port=3306,\n",
|
|
" user=\"user\",\n",
|
|
" password=\"password\",\n",
|
|
" database=\"llama_index_test\",\n",
|
|
" distance_method=\"COSINE\",\n",
|
|
")\n",
|
|
"storage_context = StorageContext.from_defaults(vector_store=client)\n",
|
|
"VectorStoreIndex.from_documents(\n",
|
|
" documents, storage_context=storage_context, show_progress=True\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e8da65bb2be263e6",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Query using AlibabaCloudMySQL with Search Test"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "e462b4e7ca54e101",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"\n",
|
|
"DASHSCOPE_API_KEY = os.environ.get(\"DASHSCOPE_API_KEY\")\n",
|
|
"\n",
|
|
"from llama_index.core import VectorStoreIndex\n",
|
|
"from llama_index.vector_stores.alibabacloud_mysql.base import (\n",
|
|
" AlibabaCloudMySQLVectorStore,\n",
|
|
")\n",
|
|
"\n",
|
|
"# set Embbeding model\n",
|
|
"from llama_index.core import Settings\n",
|
|
"from llama_index.embeddings.dashscope import DashScopeEmbedding\n",
|
|
"\n",
|
|
"embed_model = DashScopeEmbedding(api_key=DASHSCOPE_API_KEY)\n",
|
|
"# Global Settings\n",
|
|
"Settings.embed_model = embed_model\n",
|
|
"\n",
|
|
"# config llm model\n",
|
|
"from llama_index.llms.dashscope import DashScope, DashScopeGenerationModels\n",
|
|
"\n",
|
|
"dashscope_llm = DashScope(\n",
|
|
" model_name=DashScopeGenerationModels.QWEN_MAX, api_key=DASHSCOPE_API_KEY\n",
|
|
")\n",
|
|
"\n",
|
|
"print(\n",
|
|
" \"\"\"\n",
|
|
"#################\n",
|
|
"# Basic Querying including Search Test\n",
|
|
"#################\n",
|
|
"\"\"\"\n",
|
|
")\n",
|
|
"client = AlibabaCloudMySQLVectorStore.from_params(\n",
|
|
" host=\"rm-***.mysql.eu-west-1.rds.***.com\",\n",
|
|
" port=3306,\n",
|
|
" user=\"user\",\n",
|
|
" password=\"password\",\n",
|
|
" database=\"llama_index_test\",\n",
|
|
" distance_method=\"COSINE\",\n",
|
|
")\n",
|
|
"index = VectorStoreIndex.from_vector_store(\n",
|
|
" vector_store=client, embed_model=embed_model\n",
|
|
")\n",
|
|
"\n",
|
|
"QUESTION = \"What did the author do growing up?\"\n",
|
|
"# Set Retriever\n",
|
|
"vector_retriever = index.as_retriever()\n",
|
|
"# search\n",
|
|
"source_nodes = vector_retriever.retrieve(QUESTION)\n",
|
|
"# check source_nodes\n",
|
|
"print(f\"Question: {QUESTION}\")\n",
|
|
"for node in source_nodes:\n",
|
|
" print(f\"---------------------------------------------\")\n",
|
|
" print(\"Search Test\")\n",
|
|
" print(f\"---------------------------------------------\")\n",
|
|
" print(f\"Score: {node.score:.3f}\")\n",
|
|
" print(node.get_content())\n",
|
|
" print(f\"---------------------------------------------\")\n",
|
|
"\n",
|
|
"# run query\n",
|
|
"query_engine = index.as_query_engine(llm=dashscope_llm)\n",
|
|
"res = query_engine.query(QUESTION)\n",
|
|
"print(f\"Answer: {res.response}\")\n",
|
|
"print(f\"---------------------------------------------\\n\\n\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2487c65a84322a00",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Metadata Filtering"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "33845ee8cac5364b",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Load Data for Metadata Filtering"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "a7e725fa32f3cba2",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"\n",
|
|
"DASHSCOPE_API_KEY = os.environ.get(\"DASHSCOPE_API_KEY\")\n",
|
|
"\n",
|
|
"from llama_index.core import (\n",
|
|
" VectorStoreIndex,\n",
|
|
" SimpleDirectoryReader,\n",
|
|
" StorageContext,\n",
|
|
")\n",
|
|
"from llama_index.vector_stores.alibabacloud_mysql.base import (\n",
|
|
" AlibabaCloudMySQLVectorStore,\n",
|
|
")\n",
|
|
"\n",
|
|
"# set Embbeding model\n",
|
|
"from llama_index.core import Settings\n",
|
|
"from llama_index.embeddings.dashscope import DashScopeEmbedding\n",
|
|
"\n",
|
|
"Settings.embed_model = DashScopeEmbedding(api_key=DASHSCOPE_API_KEY)\n",
|
|
"\n",
|
|
"# config llm model\n",
|
|
"from llama_index.llms.dashscope import DashScope, DashScopeGenerationModels\n",
|
|
"\n",
|
|
"dashscope_llm = DashScope(\n",
|
|
" model_name=DashScopeGenerationModels.QWEN_MAX, api_key=DASHSCOPE_API_KEY\n",
|
|
")\n",
|
|
"\n",
|
|
"documents = SimpleDirectoryReader(\"data/paul_graham/\").load_data()\n",
|
|
"print(f\"Total documents: {len(documents)}\")\n",
|
|
"print(f\"First document, id: {documents[0].doc_id}\")\n",
|
|
"print(f\"First document, hash: {documents[0].hash}\")\n",
|
|
"print(\n",
|
|
" \"First document, text\"\n",
|
|
" f\" ({len(documents[0].text)} characters):\\n{'='*20}\\n{documents[0].text[:360]} ...\"\n",
|
|
")\n",
|
|
"\n",
|
|
"print(\n",
|
|
" \"\"\"\n",
|
|
"#################\n",
|
|
"# generate vector with some metadata for Metadata Filtering\n",
|
|
"#################\"\"\"\n",
|
|
")\n",
|
|
"client = AlibabaCloudMySQLVectorStore.from_params(\n",
|
|
" host=\"rm-***.mysql.***.rds.aliyuncs.com\",\n",
|
|
" port=3306,\n",
|
|
" user=\"user\",\n",
|
|
" password=\"password\",\n",
|
|
" database=\"llama_index_meta_test\",\n",
|
|
" distance_method=\"COSINE\",\n",
|
|
")\n",
|
|
"storage_context = StorageContext.from_defaults(vector_store=client)\n",
|
|
"index = VectorStoreIndex.from_documents(\n",
|
|
" documents, storage_context=storage_context, show_progress=True\n",
|
|
")\n",
|
|
"\n",
|
|
"from llama_index.core import Document\n",
|
|
"import regex as re\n",
|
|
"\n",
|
|
"# Split the text into paragraphs.\n",
|
|
"text_chunks = documents[0].text.split(\"\\n\\n\")\n",
|
|
"\n",
|
|
"# Create a document for each footnote\n",
|
|
"footnotes = [\n",
|
|
" Document(\n",
|
|
" text=chunk,\n",
|
|
" id=documents[0].doc_id,\n",
|
|
" metadata={\n",
|
|
" \"is_footnote\": bool(re.search(r\"^\\s*\\[\\d+\\]\\s*\", chunk)),\n",
|
|
" \"mark_id\": i,\n",
|
|
" },\n",
|
|
" )\n",
|
|
" for i, chunk in enumerate(text_chunks)\n",
|
|
" if bool(re.search(r\"^\\s*\\[\\d+\\]\\s*\", chunk))\n",
|
|
"]\n",
|
|
"\n",
|
|
"# Insert the footnotes into the index\n",
|
|
"for f in footnotes:\n",
|
|
" index.insert(f)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2715af7324ae9b55",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Query using AlibabaCloudMySQL with Search Test"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "c1be7bcc84a55f9b",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"\n",
|
|
"DASHSCOPE_API_KEY = os.environ.get(\"DASHSCOPE_API_KEY\")\n",
|
|
"\n",
|
|
"from llama_index.core import VectorStoreIndex\n",
|
|
"from llama_index.vector_stores.alibabacloud_mysql.base import (\n",
|
|
" AlibabaCloudMySQLVectorStore,\n",
|
|
")\n",
|
|
"\n",
|
|
"# set Embbeding model\n",
|
|
"from llama_index.core import Settings\n",
|
|
"from llama_index.embeddings.dashscope import DashScopeEmbedding\n",
|
|
"\n",
|
|
"embed_model = DashScopeEmbedding(api_key=DASHSCOPE_API_KEY)\n",
|
|
"# Global Settings\n",
|
|
"Settings.embed_model = embed_model\n",
|
|
"\n",
|
|
"# config llm model\n",
|
|
"from llama_index.llms.dashscope import DashScope, DashScopeGenerationModels\n",
|
|
"\n",
|
|
"dashscope_llm = DashScope(\n",
|
|
" model_name=DashScopeGenerationModels.QWEN_MAX, api_key=DASHSCOPE_API_KEY\n",
|
|
")\n",
|
|
"\n",
|
|
"print(\n",
|
|
" \"\"\"\n",
|
|
"#################\n",
|
|
"# Querying with Metadata Filtering including Search Test\n",
|
|
"#################\n",
|
|
"\"\"\"\n",
|
|
")\n",
|
|
"client = AlibabaCloudMySQLVectorStore.from_params(\n",
|
|
" host=\"rm-***.mysql.***.rds.aliyuncs.com\",\n",
|
|
" port=3306,\n",
|
|
" user=\"user\",\n",
|
|
" password=\"password\",\n",
|
|
" database=\"llama_index_meta_test\",\n",
|
|
" distance_method=\"COSINE\",\n",
|
|
")\n",
|
|
"index = VectorStoreIndex.from_vector_store(\n",
|
|
" vector_store=client, embed_model=embed_model\n",
|
|
")\n",
|
|
"\n",
|
|
"from llama_index.core.vector_stores import (\n",
|
|
" MetadataFilters,\n",
|
|
" MetadataFilter,\n",
|
|
" FilterOperator,\n",
|
|
" FilterCondition,\n",
|
|
")\n",
|
|
"\n",
|
|
"QUESTION = \"What did the author about space aliens and lisp?\"\n",
|
|
"print(f\"---------------------------------------------\")\n",
|
|
"print(f\"Question: {QUESTION}\")\n",
|
|
"filters = MetadataFilters(\n",
|
|
" filters=[\n",
|
|
" MetadataFilter(\n",
|
|
" key=\"is_footnote\", value=\"true\", operator=FilterOperator.EQ\n",
|
|
" ),\n",
|
|
" MetadataFilter(key=\"mark_id\", value=0, operator=FilterOperator.GTE),\n",
|
|
" ],\n",
|
|
" condition=FilterCondition.AND,\n",
|
|
")\n",
|
|
"print(f\"---------------------------------------------\")\n",
|
|
"for i in range(len(filters.filters)):\n",
|
|
" print(f\"Filter[{i}]: {filters.filters[i]}\")\n",
|
|
"print(f\"Filter Condition: {filters.condition}\")\n",
|
|
"print(f\"---------------------------------------------\")\n",
|
|
"retriever = index.as_retriever(\n",
|
|
" filters=filters,\n",
|
|
")\n",
|
|
"result = retriever.retrieve(QUESTION)\n",
|
|
"for node in result:\n",
|
|
" print(\"Search Test\")\n",
|
|
" print(f\"---------------------------------------------\")\n",
|
|
" print(f\"Score: {node.score:.3f}\")\n",
|
|
" print(node.get_content())\n",
|
|
" print(f\"---------------------------------------------\")\n",
|
|
"\n",
|
|
"# Create a query engine that only searches certain footnotes.\n",
|
|
"footnote_query_engine = index.as_query_engine(\n",
|
|
" filters=filters,\n",
|
|
" llm=dashscope_llm,\n",
|
|
")\n",
|
|
"\n",
|
|
"res = footnote_query_engine.query(QUESTION)\n",
|
|
"print(f\"Answer: {res.response}\")\n",
|
|
"print(f\"---------------------------------------------\\n\\n\")"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|