{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Comparing LLM Path Extractors for Knowledge Graph Construction\n", "\n", "In this notebook, we'll compare three different LLM Path Extractors from llama_index:\n", "1. SimpleLLMPathExtractor\n", "2. SchemaLLMPathExtractor\n", "3. DynamicLLMPathExtractor (New)\n", "\n", "We'll use a Wikipedia page as our test data and visualize the resulting knowledge graphs using Pyvis." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup and Imports" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install llama_index pyvis wikipedia" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from llama_index.core import Document, PropertyGraphIndex\n", "from llama_index.core.indices.property_graph import (\n", " SimpleLLMPathExtractor,\n", " SchemaLLMPathExtractor,\n", " DynamicLLMPathExtractor,\n", ")\n", "from llama_index.llms.openai import OpenAI\n", "from llama_index.core import Settings\n", "\n", "import wikipedia\n", "\n", "import os" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import nest_asyncio\n", "\n", "nest_asyncio.apply()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Set up LLM Backend" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "os.environ[\"OPENAI_API_KEY\"] = \"sk-proj-...\"\n", "\n", "# Set up global configurations\n", "llm = OpenAI(temperature=0.0, model=\"gpt-3.5-turbo\")\n", "\n", "Settings.llm = llm\n", "Settings.chunk_size = 2048\n", "Settings.chunk_overlap = 20" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fetch Some Raw Text from Wikipedia" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def get_wikipedia_content(title):\n", " try:\n", " page = wikipedia.page(title)\n", " return page.content\n", " except wikipedia.exceptions.DisambiguationError as e:\n", " print(f\"Disambiguation page. Options: {e.options}\")\n", " except wikipedia.exceptions.PageError:\n", " print(f\"Page '{title}' does not exist.\")\n", " return None" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Fetched content for 'Barack Obama' (length: 83977 characters)\n" ] } ], "source": [ "wiki_title = \"Barack Obama\"\n", "content = get_wikipedia_content(wiki_title)\n", "\n", "if content:\n", " document = Document(text=content, metadata={\"title\": wiki_title})\n", " print(\n", " f\"Fetched content for '{wiki_title}' (length: {len(content)} characters)\"\n", " )\n", "else:\n", " print(\"Failed to fetch Wikipedia content.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. SimpleLLMPathExtractor" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "26d142b755d94e109cf1dfcbebb4e689", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Parsing nodes: 0%| | 0/1 [00:00