wehub-resource-sync
a0c8464e58
Build Package / build (ubuntu-latest) (push) Failing after 1s
CodeQL / Analyze (python) (push) Failing after 1s
Core Typecheck / core-typecheck (push) Failing after 1s
Linting / lint (push) Failing after 1s
llama-dev tests / test-llama-dev (push) Failing after 1s
Publish Sub-Package to PyPI if Needed / publish_subpackage_if_needed (push) Has been skipped
Sync Docs to Developer Hub / sync-docs (push) Failing after 0s
Build Package / build (windows-latest) (push) Has been cancelled
490 lines
14 KiB
Plaintext
490 lines
14 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Property Graph Construction with Predefined Schemas\n",
|
||
"\n",
|
||
"<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/property_graph/property_graph_advanced.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
|
||
"\n",
|
||
"In this notebook, we walk through using Neo4j, Ollama and Huggingface to build a property graph.\n",
|
||
"\n",
|
||
"Specifically, we will be using the `SchemaLLMPathExtractor` which allows us to specify an exact schema containing possible entity types, relation types, and defining how they can be connected together. \n",
|
||
"\n",
|
||
"This is useful for when you have a specific graph you want to build, and want to limit what the LLM is predicting."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"%pip install llama-index\n",
|
||
"%pip install llama-index-llms-ollama\n",
|
||
"%pip install llama-index-embeddings-huggingface\n",
|
||
"# Optional\n",
|
||
"%pip install llama-index-graph-stores-neo4j\n",
|
||
"%pip install llama-index-graph-stores-nebula"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Load Data"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"First, lets download some sample data to play with."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"--2024-06-26 11:12:16-- https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt\n",
|
||
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.109.133, 185.199.111.133, ...\n",
|
||
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n",
|
||
"HTTP request sent, awaiting response... 200 OK\n",
|
||
"Length: 75042 (73K) [text/plain]\n",
|
||
"Saving to: ‘data/paul_graham/paul_graham_essay.txt’\n",
|
||
"\n",
|
||
"data/paul_graham/pa 100%[===================>] 73.28K --.-KB/s in 0.007s \n",
|
||
"\n",
|
||
"2024-06-26 11:12:16 (10.4 MB/s) - ‘data/paul_graham/paul_graham_essay.txt’ saved [75042/75042]\n",
|
||
"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"!mkdir -p 'data/paul_graham/'\n",
|
||
"!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from llama_index.core import SimpleDirectoryReader\n",
|
||
"\n",
|
||
"documents = SimpleDirectoryReader(\"./data/paul_graham/\").load_data()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Graph Construction\n",
|
||
"\n",
|
||
"To construct our graph, we are going to take advantage of the `SchemaLLMPathExtractor` to construct our graph.\n",
|
||
"\n",
|
||
"Given some schema for a graph, we can extract entities and relations that follow this schema, rather than letting the LLM decide entities and relations at random."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import nest_asyncio\n",
|
||
"\n",
|
||
"nest_asyncio.apply()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from typing import Literal\n",
|
||
"from llama_index.llms.ollama import Ollama\n",
|
||
"from llama_index.core.indices.property_graph import SchemaLLMPathExtractor\n",
|
||
"\n",
|
||
"# best practice to use upper-case\n",
|
||
"entities = Literal[\"PERSON\", \"PLACE\", \"ORGANIZATION\"]\n",
|
||
"relations = Literal[\"HAS\", \"PART_OF\", \"WORKED_ON\", \"WORKED_WITH\", \"WORKED_AT\"]\n",
|
||
"\n",
|
||
"# define which entities can have which relations\n",
|
||
"validation_schema = {\n",
|
||
" \"PERSON\": [\"HAS\", \"PART_OF\", \"WORKED_ON\", \"WORKED_WITH\", \"WORKED_AT\"],\n",
|
||
" \"PLACE\": [\"HAS\", \"PART_OF\", \"WORKED_AT\"],\n",
|
||
" \"ORGANIZATION\": [\"HAS\", \"PART_OF\", \"WORKED_WITH\"],\n",
|
||
"}\n",
|
||
"\n",
|
||
"kg_extractor = SchemaLLMPathExtractor(\n",
|
||
" llm=Ollama(\n",
|
||
" model=\"llama3\",\n",
|
||
" json_mode=True,\n",
|
||
" request_timeout=3600,\n",
|
||
" # Manually set the context window to limit memory usage\n",
|
||
" context_window=8000,\n",
|
||
" ),\n",
|
||
" possible_entities=entities,\n",
|
||
" possible_relations=relations,\n",
|
||
" kg_validation_schema=validation_schema,\n",
|
||
" # if false, allows for values outside of the schema\n",
|
||
" # useful for using the schema as a suggestion\n",
|
||
" strict=True,\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Now, You can use SimplePropertyGraph, Neo4j, or NebulaGraph to store the graph.\n",
|
||
"\n",
|
||
"**Option 1. Neo4j**\n",
|
||
"\n",
|
||
"To launch Neo4j locally, first ensure you have docker installed. Then, you can launch the database with the following docker command\n",
|
||
"\n",
|
||
"```bash\n",
|
||
"docker run \\\n",
|
||
" -p 7474:7474 -p 7687:7687 \\\n",
|
||
" -v $PWD/data:/data -v $PWD/plugins:/plugins \\\n",
|
||
" --name neo4j-apoc \\\n",
|
||
" -e NEO4J_apoc_export_file_enabled=true \\\n",
|
||
" -e NEO4J_apoc_import_file_enabled=true \\\n",
|
||
" -e NEO4J_apoc_import_file_use__neo4j__config=true \\\n",
|
||
" -e NEO4JLABS_PLUGINS=\\[\\\"apoc\\\"\\] \\\n",
|
||
" neo4j:latest\n",
|
||
"```\n",
|
||
"\n",
|
||
"From here, you can open the db at [http://localhost:7474/](http://localhost:7474/). On this page, you will be asked to sign in. Use the default username/password of `neo4j` and `neo4j`.\n",
|
||
"\n",
|
||
"Once you login for the first time, you will be asked to change the password.\n",
|
||
"\n",
|
||
"After this, you are ready to create your first property graph!"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from llama_index.graph_stores.neo4j import Neo4jPropertyGraphStore\n",
|
||
"\n",
|
||
"graph_store = Neo4jPropertyGraphStore(\n",
|
||
" username=\"neo4j\",\n",
|
||
" password=\"<password>\",\n",
|
||
" url=\"bolt://localhost:7687\",\n",
|
||
")\n",
|
||
"vec_store = None"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Option 2. NebulaGraph**\n",
|
||
"\n",
|
||
"To launch NebulaGraph locally, first ensure you have docker installed. Then, you can launch the database with the following docker command.\n",
|
||
"\n",
|
||
"```bash\n",
|
||
"mkdir nebula-docker-compose\n",
|
||
"cd nebula-docker-compose\n",
|
||
"curl --output docker-compose.yaml https://raw.githubusercontent.com/vesoft-inc/nebula-docker-compose/master/docker-compose-lite.yaml\n",
|
||
"docker compose up \n",
|
||
"```\n",
|
||
"After this, you are ready to create your first property graph!\n",
|
||
"\n",
|
||
"> Other options/details for deploying NebulaGraph can be found in the [docs](https://docs.nebula-graph.io/):\n",
|
||
">\n",
|
||
"> - [ad-hoc cluster in Google Colab](https://docs.nebula-graph.io/master/4.deployment-and-installation/2.compile-and-install-nebula-graph/8.deploy-nebula-graph-with-lite/).\n",
|
||
"> - [Docker Desktop Extension](https://docs.nebula-graph.io/master/2.quick-start/1.quick-start-workflow/)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from llama_index.graph_stores.nebula import NebulaPropertyGraphStore\n",
|
||
"from llama_index.core.vector_stores.simple import SimpleVectorStore\n",
|
||
"\n",
|
||
"graph_store = NebulaPropertyGraphStore(\n",
|
||
" space=\"llamaindex_nebula_property_graph\", overwrite=True\n",
|
||
")\n",
|
||
"vec_store = SimpleVectorStore()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"*If you want to explore the graph with NebulaGraph Jupyter extension*, run the following commands. Or just skip them."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"%pip install jupyter-nebulagraph"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# load NebulaGraph Jupyter extension to enable %ngql magic\n",
|
||
"%load_ext ngql\n",
|
||
"# connect to NebulaGraph service\n",
|
||
"%ngql --address 127.0.0.1 --port 9669 --user root --password nebula\n",
|
||
"%ngql CREATE SPACE IF NOT EXISTS llamaindex_nebula_property_graph(vid_type=FIXED_STRING(256));"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# use the graph space, which is similar to \"use database\" in MySQL\n",
|
||
"# The space was created in async way, so we need to wait for a while before using it, retry it if failed\n",
|
||
"%ngql USE llamaindex_nebula_property_graph;"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Start building!**\n",
|
||
"\n",
|
||
"**NOTE:** Using a local model will be slower when extracting compared to API based models. Local models (like Ollama) are typically limited to sequential processing. Expect this to take about 10 minutes on an M2 Max."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from llama_index.core import PropertyGraphIndex\n",
|
||
"from llama_index.embeddings.huggingface import HuggingFaceEmbedding\n",
|
||
"\n",
|
||
"index = PropertyGraphIndex.from_documents(\n",
|
||
" documents,\n",
|
||
" kg_extractors=[kg_extractor],\n",
|
||
" embed_model=HuggingFaceEmbedding(model_name=\"BAAI/bge-small-en-v1.5\"),\n",
|
||
" property_graph_store=graph_store,\n",
|
||
" vector_store=vec_store,\n",
|
||
" show_progress=True,\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"If we inspect the graph created, we can see that it only includes the relations and entity types that we defined!"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# If using NebulaGraph Jupyter extension\n",
|
||
"%ngql MATCH p=()-[]->() RETURN p LIMIT 20;"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"%ng_draw"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Or Neo4j:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"For information on all `kg_extractors`, see [the documentation](/../../module_guides/indexing/lpg_index_guide#construction)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Querying\n",
|
||
"\n",
|
||
"Now that our graph is created, we can query it. \n",
|
||
"\n",
|
||
"As is the theme with this notebook, we will be using a lower-level API and constructing all our retrievers ourselves!"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from llama_index.core.indices.property_graph import (\n",
|
||
" LLMSynonymRetriever,\n",
|
||
" VectorContextRetriever,\n",
|
||
")\n",
|
||
"\n",
|
||
"\n",
|
||
"llm_synonym = LLMSynonymRetriever(\n",
|
||
" index.property_graph_store,\n",
|
||
" llm=Ollama(\n",
|
||
" model=\"llama3\",\n",
|
||
" request_timeout=3600,\n",
|
||
" # Manually set the context window to limit memory usage\n",
|
||
" context_window=8000,\n",
|
||
" ),\n",
|
||
" include_text=False,\n",
|
||
")\n",
|
||
"vector_context = VectorContextRetriever(\n",
|
||
" index.property_graph_store,\n",
|
||
" embed_model=HuggingFaceEmbedding(model_name=\"BAAI/bge-small-en-v1.5\"),\n",
|
||
" include_text=False,\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"retriever = index.as_retriever(\n",
|
||
" sub_retrievers=[\n",
|
||
" llm_synonym,\n",
|
||
" vector_context,\n",
|
||
" ]\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Interleaf -> HAS -> Paul Graham\n",
|
||
"Interleaf -> HAS -> Emacs\n",
|
||
"Interleaf -> HAS -> Release Engineering\n",
|
||
"Interleaf -> HAS -> Viaweb\n",
|
||
"Interleaf -> HAS -> Y Combinator\n",
|
||
"Interleaf -> HAS -> impressive technology\n",
|
||
"Interleaf -> HAS -> smart people\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"nodes = retriever.retrieve(\"What happened at Interleaf?\")\n",
|
||
"\n",
|
||
"for node in nodes:\n",
|
||
" print(node.text)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"We can also create a query engine with similar syntax."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Paul Graham worked there, as well as other smart people. Emacs was also present.\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"query_engine = index.as_query_engine(\n",
|
||
" sub_retrievers=[\n",
|
||
" llm_synonym,\n",
|
||
" vector_context,\n",
|
||
" ],\n",
|
||
" llm=Ollama(\n",
|
||
" model=\"llama3\",\n",
|
||
" request_timeout=3600,\n",
|
||
" # Manually set the context window to limit memory usage\n",
|
||
" context_window=8000,\n",
|
||
" ),\n",
|
||
")\n",
|
||
"\n",
|
||
"response = query_engine.query(\"What happened at Interleaf?\")\n",
|
||
"\n",
|
||
"print(str(response))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"For more info on all retrievers, see the [complete guide](/../../module_guides/indexing/lpg_index_guide#retrieval-and-querying)."
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 4
|
||
}
|