wehub/microsoft--graphrag
1
0
Fork 0
Code Issues Pull Requests Actions 14 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
gh-pages / build (push) Waiting to run
Details
Python Publish (pypi) / Upload release to PyPI (push) Waiting to run
Details
Spellcheck / spellcheck (push) Waiting to run
Details
Python Build and Type Check / python-ci (ubuntu-latest, 3.11) (push) Has been cancelled
Details
Python Build and Type Check / python-ci (ubuntu-latest, 3.13) (push) Has been cancelled
Details
Python Build and Type Check / python-ci (windows-latest, 3.11) (push) Has been cancelled
Details
Python Build and Type Check / python-ci (windows-latest, 3.13) (push) Has been cancelled
Details
Python Integration Tests / python-ci (ubuntu-latest, 3.13) (push) Has been cancelled
Details
Python Integration Tests / python-ci (windows-latest, 3.13) (push) Has been cancelled
Details
Python Notebook Tests / python-ci (ubuntu-latest, 3.13) (push) Has been cancelled
Details
Python Notebook Tests / python-ci (windows-latest, 3.13) (push) Has been cancelled
Details
Python Smoke Tests / python-ci (ubuntu-latest, 3.13) (push) Has been cancelled
Details
Python Smoke Tests / python-ci (windows-latest, 3.13) (push) Has been cancelled
Details
Python Unit Tests / python-ci (ubuntu-latest, 3.13) (push) Has been cancelled
Details
Python Unit Tests / python-ci (windows-latest, 3.13) (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:37:31 +08:00
commit 6b7e6b44f1
897 changed files with 94808 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitattributes
+11
View File
@@ -0,0 +1,11 @@
*.txt text eol=lf
*.md text eol=lf
*.yml text eol=lf
*.html text eol=lf
*.py text eol=lf
*.toml text eol=lf
.gitattributes text eol=lf
.gitignore text eol=lf
*.lock
CODEOWNERS text eol=lf
LICENSE text eol=lf
.github/ISSUE_TEMPLATE/bug_report.yml
+71
View File
@@ -0,0 +1,71 @@
name: Bug Report
description: File a bug report
title: "[Bug]: <title>"
labels: ["bug", "triage"]
body:
- type: checkboxes
id: existingcheck
attributes:
label: Do you need to file an issue?
description: Please help us manage our time by avoiding duplicates and common questions with the steps below.
options:
- label: I have searched the existing issues and this bug is not already filed.
- label: My model is hosted on OpenAI or Azure. If not, please look at the "model providers" issue and don't file a new one here.
- label: I believe this is a legitimate bug, not just a question. If this is a question, please use the Discussions area.
- type: textarea
id: description
attributes:
label: Describe the bug
description: A clear and concise description of what the bug is.
placeholder: What went wrong?
- type: textarea
id: reproduce
attributes:
label: Steps to reproduce
description: |
Steps to reproduce the behavior:
1. Step 1
2. Step 2
3. ...
4. See error
placeholder: How can we replicate the issue?
- type: textarea
id: expected_behavior
attributes:
label: Expected Behavior
description: A clear and concise description of what you expected to happen.
placeholder: What should have happened?
- type: textarea
id: configused
attributes:
label: GraphRAG Config Used
description: The GraphRAG configuration used for the run.
placeholder: The settings.yaml content or GraphRAG configuration
value: |
```yaml
# Paste your config here
```
- type: textarea
id: screenshotslogs
attributes:
label: Logs and screenshots
description: If applicable, add screenshots and logs to help explain your problem.
placeholder: Add logs and screenshots here
- type: textarea
id: additional_information
attributes:
label: Additional Information
description: |
- GraphRAG Version: e.g., v0.1.1
- Operating System: e.g., Windows 10, Ubuntu 20.04
- Python Version: e.g., 3.8
- Related Issues: e.g., #1
- Any other relevant information.
value: |
- GraphRAG Version:
- Operating System:
- Python Version:
- Related Issues:
.github/ISSUE_TEMPLATE/config.yml
+1
View File
@@ -0,0 +1 @@
blank_issues_enabled: true
.github/ISSUE_TEMPLATE/feature_request.yml
+35
View File
@@ -0,0 +1,35 @@
name: Feature Request
description: File a feature request
labels: ["enhancement"]
title: "[Feature Request]: <title>"
body:
- type: checkboxes
id: existingcheck
attributes:
label: Do you need to file an issue?
description: Please help us manage our time by avoiding duplicates and common questions with the steps below.
options:
- label: I have searched the existing issues and this feature is not already filed.
- label: My model is hosted on OpenAI or Azure. If not, please look at the "model providers" issue and don't file a new one here.
- label: I believe this is a legitimate feature request, not just a question. If this is a question, please use the Discussions area.
- type: textarea
id: problem_description
attributes:
label: Is your feature request related to a problem? Please describe.
description: A clear and concise description of what the problem is.
placeholder: What problem are you trying to solve?
- type: textarea
id: solution_description
attributes:
label: Describe the solution you'd like
description: A clear and concise description of what you want to happen.
placeholder: How do you envision the solution?
- type: textarea
id: additional_context
attributes:
label: Additional context
description: Add any other context or screenshots about the feature request here.
placeholder: Any additional information
.github/ISSUE_TEMPLATE/general_issue.yml
+65
View File
@@ -0,0 +1,65 @@
name: General Issue
description: File a general issue
title: "[Issue]: <title> "
labels: ["triage"]
body:
- type: checkboxes
id: existingcheck
attributes:
label: Do you need to file an issue?
description: Please help us manage our time by avoiding duplicates and common questions with the steps below.
options:
- label: I have searched the existing issues and this bug is not already filed.
- label: My model is hosted on OpenAI or Azure. If not, please look at the "model providers" issue and don't file a new one here.
- label: I believe this is a legitimate bug, not just a question. If this is a question, please use the Discussions area.
- type: textarea
id: description
attributes:
label: Describe the issue
description: A clear and concise description of what the issue is.
placeholder: What went wrong?
- type: textarea
id: reproduce
attributes:
label: Steps to reproduce
description: |
Steps to reproduce the behavior:
1. Step 1
2. Step 2
3. ...
4. See error
placeholder: How can we replicate the issue?
- type: textarea
id: configused
attributes:
label: GraphRAG Config Used
description: The GraphRAG configuration used for the run.
placeholder: The settings.yaml content or GraphRAG configuration
value: |
```yaml
# Paste your config here
```
- type: textarea
id: screenshotslogs
attributes:
label: Logs and screenshots
description: If applicable, add screenshots and logs to help explain your problem.
placeholder: Add logs and screenshots here
- type: textarea
id: additional_information
attributes:
label: Additional Information
description: |
- GraphRAG Version: e.g., v0.1.1
- Operating System: e.g., Windows 10, Ubuntu 20.04
- Python Version: e.g., 3.8
- Related Issues: e.g., #1
- Any other relevant information.
value: |
- GraphRAG Version:
- Operating System:
- Python Version:
- Related Issues:
.github/dependabot.yml
+15
View File
@@ -0,0 +1,15 @@
# To get started with Dependabot version updates, you'll need to specify which
# package ecosystems to update and where the package manifests are located.
# Please see the documentation for all configuration options:
# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
version: 2
updates:
- package-ecosystem: "pip" # See documentation for possible values
directory: "/" # Location of package manifests
schedule:
interval: "weekly"
- package-ecosystem: "github-actions"
# Workflow files stored in the default location of `.github/workflows`. (You don't need to specify `/.github/workflows` for `directory`. You can use `directory: "/"`.)
directory: "/"
schedule:
interval: "weekly"
.github/pull_request_template.md
+36
View File
@@ -0,0 +1,36 @@
<!--
Thanks for contributing to GraphRAG!
Please do not make *Draft* pull requests, as they still notify anyone watching the repo.
Create a pull request when it is ready for review and feedback.
About this template
The following template aims to help contributors write a good description for their pull requests.
We'd like you to provide a description of the changes in your pull request (i.e. bugs fixed or features added), the motivation behind the changes, and complete the checklist below before opening a pull request.
Feel free to discard it if you need to (e.g. when you just fix a typo). -->
## Description
[Provide a brief description of the changes made in this pull request.]
## Related Issues
[Reference any related issues or tasks that this pull request addresses.]
## Proposed Changes
[List the specific changes made in this pull request.]
## Checklist
- [ ] I have tested these changes locally.
- [ ] I have reviewed the code changes.
- [ ] I have updated the documentation (if necessary).
- [ ] I have added appropriate unit tests (if applicable).
## Additional Notes
[Add any additional notes or context that may be helpful for the reviewer(s).]
.github/workflows/gh-pages.yml
+48
View File
@@ -0,0 +1,48 @@
name: gh-pages
on:
push:
branches: [main]
permissions:
contents: write
env:
PYTHON_VERSION: "3.13"
jobs:
build:
runs-on: ubuntu-latest
env:
GH_PAGES: 1
DEBUG: 1
GRAPHRAG_API_KEY: ${{ secrets.GRAPHRAG_API_KEY }}
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
- name: Set up Python ${{ env.PYTHON_VERSION }}
uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Install uv
uses: astral-sh/setup-uv@v6
- name: Install dependencies
shell: bash
run: uv sync --all-packages
- name: mkdocs build
shell: bash
run: uv run poe build_docs
- name: List Docsite Contents
run: find site
- name: Deploy to GitHub Pages
uses: JamesIves/github-pages-deploy-action@v4.6.4
with:
branch: gh-pages
folder: site
clean: true
.github/workflows/issues-autoresolve.yml
+29
View File
@@ -0,0 +1,29 @@
name: Close inactive issues
on:
schedule:
- cron: "30 1 * * *"
permissions:
actions: write
issues: write
pull-requests: write
jobs:
close-issues:
runs-on: ubuntu-latest
permissions:
issues: write
pull-requests: write
steps:
- uses: actions/stale@v9
with:
days-before-issue-stale: 7
days-before-issue-close: 5
stale-issue-label: "stale"
close-issue-label: "autoresolved"
stale-issue-message: "This issue has been marked stale due to inactivity after repo maintainer or community member responses that request more information or suggest a solution. It will be closed after five additional days."
close-issue-message: "This issue has been closed after being marked as stale for five days. Please reopen if needed."
any-of-labels: "awaiting_response"
days-before-pr-stale: -1
days-before-pr-close: -1
repo-token: ${{ secrets.GITHUB_TOKEN }}
.github/workflows/python-checks.yml
+78
View File
@@ -0,0 +1,78 @@
name: Python Build and Type Check
on:
push:
branches:
- "**/main" # match branches like feature/main
- "main" # match the main branch
pull_request:
types:
- opened
- reopened
- synchronize
- ready_for_review
branches:
- "**/main"
- "main"
paths-ignore:
- "**/*.md"
- ".semversioner/**"
permissions:
contents: read
pull-requests: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
# Only run the for the latest commit
cancel-in-progress: true
jobs:
python-ci:
# skip draft PRs
if: github.event.pull_request.draft == false
strategy:
matrix:
python-version: ["3.11", "3.13"]
os: [ubuntu-latest, windows-latest]
fail-fast: false # Continue running all jobs even if one fails
env:
DEBUG: 1
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- uses: dorny/paths-filter@v3
id: changes
with:
filters: |
python:
- 'graphrag/**/*'
- 'uv.lock'
- 'pyproject.toml'
- '**/*.py'
- '**/*.toml'
- '**/*.ipynb'
- '.github/workflows/python*.yml'
- 'tests/**/*'
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install uv
uses: astral-sh/setup-uv@v6
- name: Install dependencies
shell: bash
run: |
uv sync --all-packages
- name: Check
run: |
uv run poe check
- name: Build
run: |
uv build --all-packages
.github/workflows/python-integration-tests.yml
+95
View File
@@ -0,0 +1,95 @@
name: Python Integration Tests
on:
push:
branches:
- "**/main" # match branches like feature/main
- "main" # match the main branch
pull_request:
types:
- opened
- reopened
- synchronize
- ready_for_review
branches:
- "**/main"
- "main"
paths-ignore:
- "**/*.md"
- ".semversioner/**"
permissions:
contents: read
pull-requests: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
# only run the for the latest commit
cancel-in-progress: true
jobs:
python-ci:
# skip draft PRs
if: github.event.pull_request.draft == false
strategy:
matrix:
python-version: ["3.13"]
os: [ubuntu-latest, windows-latest]
fail-fast: false # continue running all jobs even if one fails
env:
DEBUG: 1
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- uses: dorny/paths-filter@v3
id: changes
with:
filters: |
python:
- 'graphrag/**/*'
- 'uv.lock'
- 'pyproject.toml'
- '**/*.py'
- '**/*.toml'
- '**/*.ipynb'
- '.github/workflows/python*.yml'
- 'tests/integration/**/*'
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install uv
uses: astral-sh/setup-uv@v6
- name: Install dependencies
shell: bash
run: |
uv sync --all-packages
- name: Build
run: |
uv build --all-packages
- name: Install and start Azurite
shell: bash
run: |
npm install -g azurite
azurite --silent --skipApiVersionCheck --location /tmp/azurite --debug /tmp/azurite-debug.log &
# For more information on installation/setup of Azure Cosmos DB Emulator
# https://learn.microsoft.com/en-us/azure/cosmos-db/how-to-develop-emulator?tabs=docker-linux%2Cpython&pivots=api-nosql
# Note: the emulator is only available on Windows runners. It can take longer than the default to initially startup so we increase the default timeout.
# If a job fails due to timeout, restarting the cicd job usually resolves the problem.
- name: Install Azure Cosmos DB Emulator
if: runner.os == 'Windows'
run: |
Write-Host "Launching Cosmos DB Emulator"
Import-Module "$env:ProgramFiles\Azure Cosmos DB Emulator\PSModules\Microsoft.Azure.CosmosDB.Emulator"
Start-CosmosDbEmulator -Timeout 500
- name: Integration Test
run: |
uv run poe test_integration
.github/workflows/python-notebook-tests.yml
+75
View File
@@ -0,0 +1,75 @@
name: Python Notebook Tests
on:
push:
branches:
- "**/main" # match branches like feature/main
- "main" # match the main branch
pull_request:
types:
- opened
- reopened
- synchronize
- ready_for_review
branches:
- "**/main"
- "main"
paths-ignore:
- "**/*.md"
- ".semversioner/**"
permissions:
contents: read
pull-requests: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
# Only run the for the latest commit
cancel-in-progress: true
jobs:
python-ci:
# skip draft PRs
if: github.event.pull_request.draft == false
strategy:
matrix:
python-version: ["3.13"]
os: [ubuntu-latest, windows-latest]
fail-fast: false # Continue running all jobs even if one fails
env:
DEBUG: 1
GRAPHRAG_API_KEY: ${{ secrets.OPENAI_API_KEY }}
GRAPHRAG_API_BASE: ${{ secrets.GRAPHRAG_API_BASE }}
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- uses: dorny/paths-filter@v3
id: changes
with:
filters: |
python:
- 'graphrag/**/*'
- 'uv.lock'
- 'pyproject.toml'
- '**/*.py'
- '**/*.toml'
- '**/*.ipynb'
- '.github/workflows/python*.yml'
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install uv
uses: astral-sh/setup-uv@v6
- name: Install dependencies
shell: bash
run: |
uv sync --all-packages
- name: Notebook Test
run: |
uv run poe test_notebook
.github/workflows/python-publish.yml
+100
View File
@@ -0,0 +1,100 @@
name: Python Publish (pypi)
on:
release:
types: [created]
push:
branches: [main]
env:
PYTHON_VERSION: "3.13"
jobs:
publish:
name: Upload release to PyPI
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
environment:
name: pypi
permissions:
id-token: write
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
fetch-tags: true
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Install uv
uses: astral-sh/setup-uv@v6
- name: Install dependencies
shell: bash
run: uv sync --all-packages
- name: Export Publication Version
run: echo "version=$(uv version --short)" >> $GITHUB_OUTPUT
- name: Build Distributable
shell: bash
run: uv run poe build
- name: Inspect all distribution members and metadata
shell: bash
run: |
python - <<'PY'
import glob, zipfile, tarfile, re
print("\n=== FILES IN dist/ ===")
for f in glob.glob("dist/*"):
print(" ", f)
print("\n=== WHEELS ===")
for whl in glob.glob("dist/*.whl"):
print("\n---", whl, "---")
with zipfile.ZipFile(whl) as z:
for name in z.namelist():
print(name)
metas = [n for n in z.namelist() if n.endswith(".dist-info/METADATA")]
for meta in metas:
print("\n[METADATA:", meta, "]")
text = z.read(meta).decode("utf-8", "replace")
name_m = re.search(r"^Name:\s*(.+)$", text, re.M)
ver_m = re.search(r"^Version:\s*(.+)$", text, re.M)
if name_m:
print("Project Name:", name_m.group(1))
if ver_m:
print("Version:", ver_m.group(1))
print("\n=== SDISTS ===")
for sdist in glob.glob("dist/*.tar.gz"):
print("\n---", sdist, "---")
with tarfile.open(sdist, "r:gz") as t:
for m in t.getmembers():
print(m.name)
metas = [m for m in t.getmembers() if m.name.endswith("PKG-INFO")]
for m in metas:
print("\n[PKG-INFO:", m.name, "]")
text = t.extractfile(m).read().decode("utf-8", "replace")
name_m = re.search(r"^Name:\s*(.+)$", text, re.M)
ver_m = re.search(r"^Version:\s*(.+)$", text, re.M)
if name_m:
print("Project Name:", name_m.group(1))
if ver_m:
print("Version:", ver_m.group(1))
PY
- name: Publish package distributions to PyPI
run: uv publish
.github/workflows/python-smoke-tests.yml
+96
View File
@@ -0,0 +1,96 @@
name: Python Smoke Tests
on:
push:
branches:
- "**/main" # match branches like feature/main
- "main" # match the main branch
pull_request:
types:
- opened
- reopened
- synchronize
- ready_for_review
branches:
- "**/main"
- "main"
paths-ignore:
- "**/*.md"
- ".semversioner/**"
permissions:
contents: read
pull-requests: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
# Only run the for the latest commit
cancel-in-progress: true
jobs:
python-ci:
# skip draft PRs
if: github.event.pull_request.draft == false
strategy:
matrix:
python-version: ["3.13"]
os: [ubuntu-latest, windows-latest]
fail-fast: false # Continue running all jobs even if one fails
env:
DEBUG: 1
GRAPHRAG_API_KEY: ${{ secrets.OPENAI_API_KEY }}
GRAPHRAG_API_BASE: ${{ secrets.GRAPHRAG_API_BASE }}
# Azure AI Search config
AZURE_AI_SEARCH_URL_ENDPOINT: ${{ secrets.AZURE_AI_SEARCH_URL_ENDPOINT }}
AZURE_AI_SEARCH_API_KEY: ${{ secrets.AZURE_AI_SEARCH_API_KEY }}
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- uses: dorny/paths-filter@v3
id: changes
with:
filters: |
python:
- 'graphrag/**/*'
- 'uv.lock'
- 'pyproject.toml'
- '**/*.py'
- '**/*.toml'
- '**/*.ipynb'
- '.github/workflows/python*.yml'
- 'tests/smoke/**/*'
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install uv
uses: astral-sh/setup-uv@v6
- name: Install dependencies
shell: bash
run: |
uv sync --all-packages
- name: Build
run: |
uv build --all-packages
- name: Install and start Azurite
shell: bash
run: |
npm install -g azurite
azurite --silent --skipApiVersionCheck --location /tmp/azurite --debug /tmp/azurite-debug.log &
- name: Smoke Test
if: steps.changes.outputs.python == 'true'
run: |
uv run poe test_smoke
- uses: actions/upload-artifact@v4
if: always()
with:
name: smoke-test-artifacts-${{ matrix.python-version }}-${{ runner.os }}
path: tests/fixtures/*
.github/workflows/python-unit-tests.yml
+78
View File
@@ -0,0 +1,78 @@
name: Python Unit Tests
on:
push:
branches:
- "**/main" # match branches like feature/main
- "main" # match the main branch
pull_request:
types:
- opened
- reopened
- synchronize
- ready_for_review
branches:
- "**/main"
- "main"
paths-ignore:
- "**/*.md"
- ".semversioner/**"
permissions:
contents: read
pull-requests: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
# Only run the for the latest commit
cancel-in-progress: true
jobs:
python-ci:
# skip draft PRs
if: github.event.pull_request.draft == false
strategy:
matrix:
python-version: ["3.13"]
os: [ubuntu-latest, windows-latest]
fail-fast: false # Continue running all jobs even if one fails
env:
DEBUG: 1
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- uses: dorny/paths-filter@v3
id: changes
with:
filters: |
python:
- 'graphrag/**/*'
- 'uv.lock'
- 'pyproject.toml'
- '**/*.py'
- '**/*.toml'
- '**/*.ipynb'
- '.github/workflows/python*.yml'
- 'tests/**/*'
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install uv
uses: astral-sh/setup-uv@v6
- name: Install dependencies
shell: bash
run: |
uv sync --all-packages
- name: Unit Test
run: |
uv run poe test_unit
- name: Verb Test
run: |
uv run poe test_verbs
.github/workflows/semver.yml
+21
View File
@@ -0,0 +1,21 @@
name: Semver Check
on:
pull_request:
types:
- opened
- reopened
- synchronize
- ready_for_review
branches: [main]
jobs:
semver:
# skip draft PRs
if: github.event.pull_request.draft == false
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Check Semver
run: ./scripts/semver-check.sh
.github/workflows/spellcheck.yml
+22
View File
@@ -0,0 +1,22 @@
name: Spellcheck
on:
push:
branches: [main]
pull_request:
types:
- opened
- reopened
- synchronize
- ready_for_review
paths:
- "**/*"
jobs:
spellcheck:
# skip draft PRs
if: github.event.pull_request.draft == false
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Spellcheck
run: ./scripts/spellcheck.sh
.gitignore
+69
View File
@@ -0,0 +1,69 @@
# Python Artifacts
python/*/lib/
dist/
build/
*.egg-info/
# Test Output
.coverage
coverage/
licenses.txt
examples_notebooks/*/data
tests/fixtures/cache
tests/fixtures/*/cache
tests/fixtures/*/output
output/lancedb
# Random
.DS_Store
*.log*
.venv
venv/
.conda
.tmp
packages/graphrag-llm/notebooks/metrics
packages/graphrag-llm/notebooks/cache
.env
build.zip
.turbo
__pycache__
.pipeline
# Azurite
temp_azurite/
__azurite*.json
__blobstorage*.json
__blobstorage__/
# Getting started example
ragtest/
.ragtest/
.pipelines
.pipeline
# mkdocs
site/
# Docs migration
docsite/
.yarn/
.pnp*
# PyCharm
.idea/
# Jupyter notebook
.ipynb_checkpoints/
# Root build assets
packages/*/LICENSE
# Notebooks outputs
packages/graphrag-*/example_notebooks/**/output
packages/graphrag-*/example_notebooks/**/cache
.semversioner/0.1.0.json
+10
View File
@@ -0,0 +1,10 @@
{
"changes": [
{
"description": "Initial Release",
"type": "minor"
}
],
"created_at": "2024-07-01T21:48:50+00:00",
"version": "0.1.0"
}
.semversioner/0.2.0.json
+94
View File
@@ -0,0 +1,94 @@
{
"changes": [
{
"description": "Add content-based KNN for selecting prompt tune few shot examples",
"type": "minor"
},
{
"description": "Add dynamic community report rating to the prompt tuning engine",
"type": "minor"
},
{
"description": "Add Minute-based Rate Limiting and fix rpm, tpm settings",
"type": "patch"
},
{
"description": "Add N parameter support",
"type": "patch"
},
{
"description": "Add cli flag to overlay default values onto a provided config.",
"type": "patch"
},
{
"description": "Add exception handling on file load",
"type": "patch"
},
{
"description": "Add language support to prompt tuning",
"type": "patch"
},
{
"description": "Add llm params to local and global search",
"type": "patch"
},
{
"description": "Fix broken prompt tuning link on docs",
"type": "patch"
},
{
"description": "Fix delta none on query calls",
"type": "patch"
},
{
"description": "Fix docsite base url",
"type": "patch"
},
{
"description": "Fix encoding model parameter on prompt tune",
"type": "patch"
},
{
"description": "Fix for --limit exceeding the dataframe length",
"type": "patch"
},
{
"description": "Fix for Ruff 0.5.2",
"type": "patch"
},
{
"description": "Fixed an issue where base OpenAI embeddings can't work with Azure OpenAI LLM",
"type": "patch"
},
{
"description": "Modify defaults for CHUNK_SIZE, CHUNK_OVERLAP and GLEANINGS to reduce time and LLM calls",
"type": "patch"
},
{
"description": "fix community_report doesn't work in settings.yaml",
"type": "patch"
},
{
"description": "fix llm response content is None in query",
"type": "patch"
},
{
"description": "fix the organization parameter is ineffective during queries",
"type": "patch"
},
{
"description": "remove duplicate file read",
"type": "patch"
},
{
"description": "support non-open ai model config to prompt tune",
"type": "patch"
},
{
"description": "use binary io processing for all file io operations",
"type": "patch"
}
],
"created_at": "2024-07-25T02:01:38+00:00",
"version": "0.2.0"
}
.semversioner/0.2.1.json
+70
View File
@@ -0,0 +1,70 @@
{
"changes": [
{
"description": "Added default columns for vector store at create_pipeline_config. No change for other cases.",
"type": "patch"
},
{
"description": "Change json parsing error in the map step of global search to warning",
"type": "patch"
},
{
"description": "Fix Local Search breaking when loading Embeddings input. Defaulting overwrite to True as in the rest of the vector store config",
"type": "patch"
},
{
"description": "Fix json parsing when LLM returns faulty responses",
"type": "patch"
},
{
"description": "Fix missing community reports and refactor community context builder",
"type": "patch"
},
{
"description": "Fixed a bug that erased the vector database, added a new parameter to specify the config file path, and updated the documentation accordingly.",
"type": "patch"
},
{
"description": "Try parsing json before even repairing",
"type": "patch"
},
{
"description": "Update Prompt Tuning meta prompts with finer examples",
"type": "patch"
},
{
"description": "Update default entity extraction and gleaning prompts to reduce hallucinations",
"type": "patch"
},
{
"description": "add encoding-model to entity/claim extraction config",
"type": "patch"
},
{
"description": "add encoding-model to text chunking config",
"type": "patch"
},
{
"description": "add user prompt to history-tracking llm",
"type": "patch"
},
{
"description": "update config reader to allow for zero gleans",
"type": "patch"
},
{
"description": "update config-reader to allow for empty chunk-by arrays",
"type": "patch"
},
{
"description": "update history-tracking LLm to use 'assistant' instead of 'system' in output history.",
"type": "patch"
},
{
"description": "use history argument in hash key computation; add history input to cache data",
"type": "patch"
}
],
"created_at": "2024-08-06T00:25:52+00:00",
"version": "0.2.1"
}
.semversioner/0.2.2.json
+22
View File
@@ -0,0 +1,22 @@
{
"changes": [
{
"description": "Add a check if there is no community record added in local search context",
"type": "patch"
},
{
"description": "Add sepparate workflow for Python Tests",
"type": "patch"
},
{
"description": "Docs updates",
"type": "patch"
},
{
"description": "Run smoke tests on 4o",
"type": "patch"
}
],
"created_at": "2024-08-08T22:40:57+00:00",
"version": "0.2.2"
}
.semversioner/0.3.0.json
+30
View File
@@ -0,0 +1,30 @@
{
"changes": [
{
"description": "Implement auto templating API.",
"type": "minor"
},
{
"description": "Implement query engine API.",
"type": "minor"
},
{
"description": "Fix file dumps using json for non ASCII chars",
"type": "patch"
},
{
"description": "Stabilize smoke tests for query context building",
"type": "patch"
},
{
"description": "fix query embedding",
"type": "patch"
},
{
"description": "fix sort_context & max_tokens params in verb",
"type": "patch"
}
],
"created_at": "2024-08-12T23:51:49+00:00",
"version": "0.3.0"
}
.semversioner/0.3.1.json
+34
View File
@@ -0,0 +1,34 @@
{
"changes": [
{
"description": "Add preflight check to check LLM connectivity.",
"type": "patch"
},
{
"description": "Add streaming support for local/global search to query cli",
"type": "patch"
},
{
"description": "Add support for both float and int on schema validation for community report generation",
"type": "patch"
},
{
"description": "Avoid running index on gh-pages publishing",
"type": "patch"
},
{
"description": "Implement Index API",
"type": "patch"
},
{
"description": "Improves filtering for data dir inferring",
"type": "patch"
},
{
"description": "Update to nltk 3.9.1",
"type": "patch"
}
],
"created_at": "2024-08-21T22:46:19+00:00",
"version": "0.3.1"
}
.semversioner/0.3.2.json
+42
View File
@@ -0,0 +1,42 @@
{
"changes": [
{
"description": "Add context data to query API responses.",
"type": "patch"
},
{
"description": "Add missing config parameter documentation for prompt tuning",
"type": "patch"
},
{
"description": "Add neo4j community notebook",
"type": "patch"
},
{
"description": "Ensure entity types to be str when running prompt tuning",
"type": "patch"
},
{
"description": "Fix weight casting during graph extraction",
"type": "patch"
},
{
"description": "Patch \"past\" dependency issues",
"type": "patch"
},
{
"description": "Update developer guide.",
"type": "patch"
},
{
"description": "Update query type hints.",
"type": "patch"
},
{
"description": "change-lancedb-placement",
"type": "patch"
}
],
"created_at": "2024-08-26T23:43:01+00:00",
"version": "0.3.2"
}
.semversioner/0.3.3.json
+66
View File
@@ -0,0 +1,66 @@
{
"changes": [
{
"description": "Add entrypoints for incremental indexing",
"type": "patch"
},
{
"description": "Clean up and organize run index code",
"type": "patch"
},
{
"description": "Consistent config loading. Resolves #99 and Resolves #1049",
"type": "patch"
},
{
"description": "Fix circular dependency when running prompt tune api directly",
"type": "patch"
},
{
"description": "Fix default settings for embedding",
"type": "patch"
},
{
"description": "Fix img for auto tune",
"type": "patch"
},
{
"description": "Fix img width",
"type": "patch"
},
{
"description": "Fixed a bug in prompt tuning process",
"type": "patch"
},
{
"description": "Refactor text unit build at local search",
"type": "patch"
},
{
"description": "Update Prompt Tuning docs",
"type": "patch"
},
{
"description": "Update create_pipeline_config.py",
"type": "patch"
},
{
"description": "Update prompt tune command in docs",
"type": "patch"
},
{
"description": "add querying from azure blob storage",
"type": "patch"
},
{
"description": "fix setting base_dir to full paths when not using file system.",
"type": "patch"
},
{
"description": "fix strategy config in entity_extraction",
"type": "patch"
}
],
"created_at": "2024-09-10T19:51:24+00:00",
"version": "0.3.3"
}
.semversioner/0.3.4.json
+14
View File
@@ -0,0 +1,14 @@
{
"changes": [
{
"description": "Deep copy txt units on local search to avoid race conditions",
"type": "patch"
},
{
"description": "Fix summarization including empty descriptions",
"type": "patch"
}
],
"created_at": "2024-09-11T22:31:58+00:00",
"version": "0.3.4"
}
.semversioner/0.3.5.json
+50
View File
@@ -0,0 +1,50 @@
{
"changes": [
{
"description": "Add compound verbs with tests infra.",
"type": "patch"
},
{
"description": "Collapse create_final_communities.",
"type": "patch"
},
{
"description": "Collapse create_final_text_units.",
"type": "patch"
},
{
"description": "Covariate verb collapse.",
"type": "patch"
},
{
"description": "Fix duplicates in community context builder",
"type": "patch"
},
{
"description": "Fix prompt tune output path",
"type": "patch"
},
{
"description": "Fix seed hardcoded init",
"type": "patch"
},
{
"description": "Fix seeded random gen on clustering",
"type": "patch"
},
{
"description": "Improve logging.",
"type": "patch"
},
{
"description": "Set default values for cli parameters.",
"type": "patch"
},
{
"description": "Use static output directories.",
"type": "patch"
}
],
"created_at": "2024-09-19T15:26:01+00:00",
"version": "0.3.5"
}
.semversioner/0.3.6.json
+14
View File
@@ -0,0 +1,14 @@
{
"changes": [
{
"description": "Collapse create_final_relationships.",
"type": "patch"
},
{
"description": "Dependency update and cleanup",
"type": "patch"
}
],
"created_at": "2024-09-20T00:09:13+00:00",
"version": "0.3.6"
}
.semversioner/0.4.0.json
+222
View File
@@ -0,0 +1,222 @@
{
"changes": [
{
"description": "Add Incremental Indexing",
"type": "minor"
},
{
"description": "Added DRIFT graph reasoning query module",
"type": "minor"
},
{
"description": "embeddings moved to a different workflow",
"type": "minor"
},
{
"description": "Add DRIFT search cli and example notebook",
"type": "patch"
},
{
"description": "Add config for incremental updates",
"type": "patch"
},
{
"description": "Add embeddings to subflow.",
"type": "patch"
},
{
"description": "Add naive community merge using time period",
"type": "patch"
},
{
"description": "Add relationship merge",
"type": "patch"
},
{
"description": "Add runtime-only storage option.",
"type": "patch"
},
{
"description": "Add text units update",
"type": "patch"
},
{
"description": "Allow empty workflow returns to avoid disk writing.",
"type": "patch"
},
{
"description": "Apply pandas optimizations to create final entities",
"type": "patch"
},
{
"description": "Calculate new inputs and deleted inputs on update",
"type": "patch"
},
{
"description": "Collapse covariates flow.",
"type": "patch"
},
{
"description": "Collapse create-base-entity-graph.",
"type": "patch"
},
{
"description": "Collapse create-final-community-reports.",
"type": "patch"
},
{
"description": "Collapse create-final-documents.",
"type": "patch"
},
{
"description": "Collapse create-final-entities.",
"type": "patch"
},
{
"description": "Collapse create-final-nodes.",
"type": "patch"
},
{
"description": "Collapse create_base_documents.",
"type": "patch"
},
{
"description": "Collapse create_base_text_units.",
"type": "patch"
},
{
"description": "Collapse create_final_relationships.",
"type": "patch"
},
{
"description": "Collapse entity extraction.",
"type": "patch"
},
{
"description": "Collapse entity summarize.",
"type": "patch"
},
{
"description": "Collapse intermediate workflow outputs.",
"type": "patch"
},
{
"description": "Dependency updates",
"type": "patch"
},
{
"description": "Extract DataShaper-less flows.",
"type": "patch"
},
{
"description": "Fix embeddings faulty assignments",
"type": "patch"
},
{
"description": "Fix init defaults for vector store and drift img in docs",
"type": "patch"
},
{
"description": "Fix nested json parsing",
"type": "patch"
},
{
"description": "Fix some edge cases on Drift Search over small input sets",
"type": "patch"
},
{
"description": "Fix var name for embedding",
"type": "patch"
},
{
"description": "Merge existing and new entities, updating values accordingly",
"type": "patch"
},
{
"description": "Merge text_embed into create-final-relationships subflow.",
"type": "patch"
},
{
"description": "Move embedding verbs to operations.",
"type": "patch"
},
{
"description": "Moving verbs around.",
"type": "patch"
},
{
"description": "Optimize Create Base Documents subflow",
"type": "patch"
},
{
"description": "Optimize text unit relationship count",
"type": "patch"
},
{
"description": "Perf optimizations in map_query_to_entities()",
"type": "patch"
},
{
"description": "Remove aggregate_df from final coomunities and final text units",
"type": "patch"
},
{
"description": "Remove duplicated relationships and nodes",
"type": "patch"
},
{
"description": "Remove unused column from final entities",
"type": "patch"
},
{
"description": "Reorganized api,reporter,callback code into separate components. Defined debug profiles.",
"type": "patch"
},
{
"description": "Small cleanup in community context history building",
"type": "patch"
},
{
"description": "Transient entity graph and snapshotting.",
"type": "patch"
},
{
"description": "Update Incremental Indexing to new embeddings workflow",
"type": "patch"
},
{
"description": "Use mkdocs for documentation",
"type": "patch"
},
{
"description": "add backwards compatibility patch to vector store.",
"type": "patch"
},
{
"description": "add-autogenerated-cli-docs",
"type": "patch"
},
{
"description": "fix docs image path",
"type": "patch"
},
{
"description": "refactor use of vector stores and update support for managed identity",
"type": "patch"
},
{
"description": "remove redundant error-handling code from global-search",
"type": "patch"
},
{
"description": "reorganize cli layer",
"type": "patch"
},
{
"description": "Fix Community ID loading for DRIFT search over existing indexes",
"type": "patch"
}
],
"created_at": "2024-11-06T00:24:14+00:00",
"version": "0.4.0"
}
.semversioner/0.4.1.json
+34
View File
@@ -0,0 +1,34 @@
{
"changes": [
{
"description": "Add update cli entrypoint for incremental indexing",
"type": "patch"
},
{
"description": "Allow some CI/CD jobs to skip PRs dedicated to doc updates only.",
"type": "patch"
},
{
"description": "Fix a file paths issue in the viz guide.",
"type": "patch"
},
{
"description": "Fix optional covariates update in incremental indexing",
"type": "patch"
},
{
"description": "Raise error on empty deltas for inc indexing",
"type": "patch"
},
{
"description": "add visualization guide to doc site",
"type": "patch"
},
{
"description": "fix streaming output error",
"type": "patch"
}
],
"created_at": "2024-11-08T23:13:05+00:00",
"version": "0.4.1"
}
.semversioner/0.5.0.json
+38
View File
@@ -0,0 +1,38 @@
{
"changes": [
{
"description": "Data model changes.",
"type": "minor"
},
{
"description": "Add Parquet as part of the default emitters when not pressent",
"type": "patch"
},
{
"description": "Centralized prompts and export all for easier injection.",
"type": "patch"
},
{
"description": "Cleanup of artifact outputs/schemas.",
"type": "patch"
},
{
"description": "Config and docs updates.",
"type": "patch"
},
{
"description": "Implement dynamic community selection to global search",
"type": "patch"
},
{
"description": "fix autocompletion of existing files/directory paths.",
"type": "patch"
},
{
"description": "move import statements out of init files",
"type": "patch"
}
],
"created_at": "2024-11-16T00:43:06+00:00",
"version": "0.5.0"
}
.semversioner/0.9.0.json
+46
View File
@@ -0,0 +1,46 @@
{
"changes": [
{
"description": "Refactor graph creation.",
"type": "minor"
},
{
"description": "Dependency updates",
"type": "patch"
},
{
"description": "Fix Global Search with dynamic Community selection bug",
"type": "patch"
},
{
"description": "Fix question gen.",
"type": "patch"
},
{
"description": "Optimize Final Community Reports calculation and stabilize cache",
"type": "patch"
},
{
"description": "miscellaneous code cleanup and minor changes for better alignment of style across the codebase.",
"type": "patch"
},
{
"description": "replace llm package with fnllm",
"type": "patch"
},
{
"description": "replaced md5 hash with sha256",
"type": "patch"
},
{
"description": "replaced md5 hash with sha512",
"type": "patch"
},
{
"description": "update API and add a demonstration notebook",
"type": "patch"
}
],
"created_at": "2024-12-06T20:12:30+00:00",
"version": "0.9.0"
}
.semversioner/1.0.0.json
+26
View File
@@ -0,0 +1,26 @@
{
"changes": [
{
"description": "Add Parent id to communities data model",
"type": "patch"
},
{
"description": "Add migration notebook.",
"type": "patch"
},
{
"description": "Create separate community workflow, collapse subflows.",
"type": "patch"
},
{
"description": "Dependency Updates",
"type": "patch"
},
{
"description": "cleanup and refactor factory classes.",
"type": "patch"
}
],
"created_at": "2024-12-11T21:41:49+00:00",
"version": "1.0.0"
}
.semversioner/1.0.1.json
+22
View File
@@ -0,0 +1,22 @@
{
"changes": [
{
"description": "Fix encoding model config parsing",
"type": "patch"
},
{
"description": "Fix exception on error callbacks",
"type": "patch"
},
{
"description": "Manage llm instances inside a cached singleton. Check for empty dfs after entity/relationship extraction",
"type": "patch"
},
{
"description": "Respect encoding_model option",
"type": "patch"
}
],
"created_at": "2024-12-18T23:12:52+00:00",
"version": "1.0.1"
}
.semversioner/1.1.0.json
+58
View File
@@ -0,0 +1,58 @@
{
"changes": [
{
"description": "Make gleanings independent of encoding",
"type": "minor"
},
{
"description": "Remove DataShaper (first steps).",
"type": "minor"
},
{
"description": "Remove old pipeline runner.",
"type": "minor"
},
{
"description": "new search implemented as a new option for the api",
"type": "minor"
},
{
"description": "Fix gleanings loop check",
"type": "patch"
},
{
"description": "Implement cosmosdb storage option for cache and output",
"type": "patch"
},
{
"description": "Move extractor code to co-locate with operations.",
"type": "patch"
},
{
"description": "Remove config input models.",
"type": "patch"
},
{
"description": "Ruff update",
"type": "patch"
},
{
"description": "Simplify and streamline internal config.",
"type": "patch"
},
{
"description": "Simplify callbacks model.",
"type": "patch"
},
{
"description": "Streamline flows.",
"type": "patch"
},
{
"description": "fix instantiation of storage classes.",
"type": "patch"
}
],
"created_at": "2025-01-07T20:25:57+00:00",
"version": "1.1.0"
}
.semversioner/1.1.1.json
+14
View File
@@ -0,0 +1,14 @@
{
"changes": [
{
"description": "Fix a bug on creating community hierarchy for dynamic search",
"type": "patch"
},
{
"description": "Increase LOCAL_SEARCH_COMMUNITY_PROP to 15%",
"type": "patch"
}
],
"created_at": "2025-01-08T21:53:16+00:00",
"version": "1.1.1"
}
.semversioner/1.1.2.json
+10
View File
@@ -0,0 +1,10 @@
{
"changes": [
{
"description": "Basic Rag minor fix",
"type": "patch"
}
],
"created_at": "2025-01-09T22:29:23+00:00",
"version": "1.2.0"
}
.semversioner/1.2.0.json
+26
View File
@@ -0,0 +1,26 @@
{
"changes": [
{
"description": "Add Drift Reduce response and streaming endpoint",
"type": "minor"
},
{
"description": "add cosmosdb vector store",
"type": "minor"
},
{
"description": "Fix example notebooks",
"type": "patch"
},
{
"description": "Set default rate limits.",
"type": "patch"
},
{
"description": "unit tests for text_splitting",
"type": "patch"
}
],
"created_at": "2025-01-15T20:32:00+00:00",
"version": "1.2.0"
}
.semversioner/2.0.0.json
+146
View File
@@ -0,0 +1,146 @@
{
"changes": [
{
"description": "Add children to communities to avoid re-compute.",
"type": "major"
},
{
"description": "Reorganize and rename workflows and their outputs.",
"type": "major"
},
{
"description": "Rework API to accept callbacks.",
"type": "major"
},
{
"description": "Add LMM Manager and Factory, to support provider registration",
"type": "minor"
},
{
"description": "Add NLP graph extraction.",
"type": "minor"
},
{
"description": "Add pipeline_start and pipeline_end callbacks.",
"type": "minor"
},
{
"description": "Move embeddings snapshots to the workflow runner.",
"type": "minor"
},
{
"description": "Remove config inheritance, hydration, and automatic env var overlays.",
"type": "minor"
},
{
"description": "Rework the update output storage structure.",
"type": "minor"
},
{
"description": "Add caching to NLP extractor.",
"type": "patch"
},
{
"description": "Add vector store id reference to embeddings config.",
"type": "patch"
},
{
"description": "Export NLP community reports prompt.",
"type": "patch"
},
{
"description": "Fix DRIFT search on Azure AI Search.",
"type": "patch"
},
{
"description": "Fix StopAsyncIteration catch.",
"type": "patch"
},
{
"description": "Fix missing embeddings workflow in FastGraphRAG.",
"type": "patch"
},
{
"description": "Fix proper use of n_depth for drift search",
"type": "patch"
},
{
"description": "Fix report generation recursion.",
"type": "patch"
},
{
"description": "Fix summarization over large datasets for inc indexing. Fix relationship summarization",
"type": "patch"
},
{
"description": "Optimize data iteration by removing some iterrows from code",
"type": "patch"
},
{
"description": "Patch json mode for community reports",
"type": "patch"
},
{
"description": "Properly increment text unit IDs during updates.",
"type": "patch"
},
{
"description": "Refactor config defaults from constants to type-safe, hierarchical dataclass.",
"type": "patch"
},
{
"description": "Require explicit azure auth settings when using AOI.",
"type": "patch"
},
{
"description": "Separates graph pruning for differential usage.",
"type": "patch"
},
{
"description": "Tuck flow functions under their workflow modules.",
"type": "patch"
},
{
"description": "Update fnllm. Remove unused libs.",
"type": "patch"
},
{
"description": "Use ModelProvider for query module",
"type": "patch"
},
{
"description": "Use shared schema for final outputs.",
"type": "patch"
},
{
"description": "add dynamic retry logic.",
"type": "patch"
},
{
"description": "add option to prepend metadata into chunks",
"type": "patch"
},
{
"description": "cleanup query code duplication.",
"type": "patch"
},
{
"description": "implemented multi-index querying for api layer",
"type": "patch"
},
{
"description": "multi index query cli support",
"type": "patch"
},
{
"description": "remove unused columns and change property document_attribute_columns to metadata",
"type": "patch"
},
{
"description": "update multi-index query to support new workflows",
"type": "patch"
}
],
"created_at": "2025-02-25T23:30:50+00:00",
"version": "2.0.0"
}
.semversioner/2.1.0.json
+22
View File
@@ -0,0 +1,22 @@
{
"changes": [
{
"description": "Add support for JSON input files.",
"type": "minor"
},
{
"description": "Updated the prompt tunning client to support csv-metadata injection and updated output file types to match the new naming convention.",
"type": "minor"
},
{
"description": "Add check for custom model types while config loading",
"type": "patch"
},
{
"description": "Adds general-purpose pipeline run state object.",
"type": "patch"
}
],
"created_at": "2025-03-11T23:53:00+00:00",
"version": "2.1.0"
}
.semversioner/2.2.0.json
+46
View File
@@ -0,0 +1,46 @@
{
"changes": [
{
"description": "Support OpenAI reasoning models.",
"type": "minor"
},
{
"description": "Add option to snapshot raw extracted graph tables.",
"type": "patch"
},
{
"description": "Added batching logic to the prompt tuning autoselection embeddings workflow",
"type": "patch"
},
{
"description": "Align config classes and docs better.",
"type": "patch"
},
{
"description": "Align embeddings table loading with configured fields.",
"type": "patch"
},
{
"description": "Brings parity with our latest NLP extraction approaches.",
"type": "patch"
},
{
"description": "Fix fnllm to 0.2.3",
"type": "patch"
},
{
"description": "Fixes to basic search.",
"type": "patch"
},
{
"description": "Update llm args for consistency.",
"type": "patch"
},
{
"description": "add vector store integration tests",
"type": "patch"
}
],
"created_at": "2025-04-25T23:30:57+00:00",
"version": "2.2.0"
}
.semversioner/2.2.1.json
+18
View File
@@ -0,0 +1,18 @@
{
"changes": [
{
"description": "Fix Community Report prompt tuning response",
"type": "patch"
},
{
"description": "Fix graph creation missing edge weights.",
"type": "patch"
},
{
"description": "Update as workflows",
"type": "patch"
}
],
"created_at": "2025-04-30T23:50:31+00:00",
"version": "2.2.1"
}
.semversioner/2.3.0.json
+34
View File
@@ -0,0 +1,34 @@
{
"changes": [
{
"description": "Remove Dynamic Max Retries support. Refactor typer typing in cli interface",
"type": "minor"
},
{
"description": "Update fnllm to latest. Update default graphrag configuration",
"type": "minor"
},
{
"description": "A few fixes and enhancements for better reuse and flow.",
"type": "patch"
},
{
"description": "Add full llm response to LLM PRovider output",
"type": "patch"
},
{
"description": "Fix Drift Reduce Response for non streaming calls",
"type": "patch"
},
{
"description": "Fix global search prompt to include missing formatting key",
"type": "patch"
},
{
"description": "Upgrade pyarrow dependency to >=17.0.0 to fix CVE-2024-52338",
"type": "patch"
}
],
"created_at": "2025-05-23T21:02:47+00:00",
"version": "2.3.0"
}
.semversioner/2.4.0.json
+26
View File
@@ -0,0 +1,26 @@
{
"changes": [
{
"description": "Allow injection of custom pipelines.",
"type": "minor"
},
{
"description": "Refactored StorageFactory to use a registration-based approach",
"type": "minor"
},
{
"description": "Fix default values for tpm and rpm limiters on embeddings",
"type": "patch"
},
{
"description": "Update typer.",
"type": "patch"
},
{
"description": "cleaned up logging to follow python standards.",
"type": "patch"
}
],
"created_at": "2025-07-15T00:04:15+00:00",
"version": "2.4.0"
}
.semversioner/2.5.0.json
+14
View File
@@ -0,0 +1,14 @@
{
"changes": [
{
"description": "Add additional context variable to build index signature for custom parameter bag",
"type": "minor"
},
{
"description": "swap package management from Poetry -> UV",
"type": "minor"
}
],
"created_at": "2025-08-14T00:59:46+00:00",
"version": "2.5.0"
}
.semversioner/2.6.0.json
+54
View File
@@ -0,0 +1,54 @@
{
"changes": [
{
"description": "Add LiteLLM chat and embedding model providers.",
"type": "minor"
},
{
"description": "Add LoggerFactory and clean up related API.",
"type": "minor"
},
{
"description": "Add config for NLP async mode.",
"type": "minor"
},
{
"description": "Add optional input documents to indexing API.",
"type": "minor"
},
{
"description": "add customization to vector store",
"type": "minor"
},
{
"description": "Add gpt-5 support by updating fnllm dependency.",
"type": "patch"
},
{
"description": "Fix all human_readable_id fields to be 0-based.",
"type": "patch"
},
{
"description": "Fix multi-index search.",
"type": "patch"
},
{
"description": "Improve upon recent logging refactor",
"type": "patch"
},
{
"description": "Make cache, storage, and vector_store factories consistent with similar registration support",
"type": "patch"
},
{
"description": "Remove hard-coded community rate limiter.",
"type": "patch"
},
{
"description": "generate_text_embeddings only loads tables if embedding field is specified.",
"type": "patch"
}
],
"created_at": "2025-09-22T21:44:51+00:00",
"version": "2.6.0"
}
.semversioner/2.7.0.json
+18
View File
@@ -0,0 +1,18 @@
{
"changes": [
{
"description": "Set LiteLLM as default in init_content.",
"type": "minor"
},
{
"description": "Fix Azure auth scope issue with LiteLLM.",
"type": "patch"
},
{
"description": "Housekeeping toward 2.7.",
"type": "patch"
}
],
"created_at": "2025-10-08T22:39:42+00:00",
"version": "2.7.0"
}
.semversioner/2.7.1.json
+10
View File
@@ -0,0 +1,10 @@
{
"changes": [
{
"description": "Pin pandas==2.3.3.",
"type": "patch"
}
],
"created_at": "2026-01-26T17:46:58+00:00",
"version": "2.7.1"
}
.semversioner/3.0.0.json
+10
View File
@@ -0,0 +1,10 @@
{
"changes": [
{
"description": "Monorepo restructure\n\n New Packages:\n - graphrag-cache\n - graphrag-chunking\n - graphrag-common\n - graphrag-input\n - graphrag-llm\n - graphrag-storage\n - graphrag-vectors\n\n Changes:\n - New config: run graphrag init --force to reinitialize config with new layout and options.",
"type": "major"
}
],
"created_at": "2026-01-27T18:50:37+00:00",
"version": "3.0.0"
}
.semversioner/3.0.1.json
+10
View File
@@ -0,0 +1,10 @@
{
"changes": [
{
"description": "Fix missing dependency.",
"type": "patch"
}
],
"created_at": "2026-01-28T14:34:25+00:00",
"version": "3.0.1"
}
.semversioner/3.0.2.json
+50
View File
@@ -0,0 +1,50 @@
{
"changes": [
{
"description": "Add CSVTableProvider",
"type": "patch"
},
{
"description": "Add DataReader class for typed dataframe loading from TableProvider across indexing workflows and query CLI",
"type": "patch"
},
{
"description": "Add TableProvider abstraction for table-based storage operations",
"type": "patch"
},
{
"description": "Add async iterator support to InputReader and use it in load_input_documents and load_update_documents workflows.",
"type": "patch"
},
{
"description": "Add table provider factory.",
"type": "patch"
},
{
"description": "Fix missed py 3.13.",
"type": "patch"
},
{
"description": "Move document ID, human_readable_id, and raw_data initialization from create_final_documents into load_input_documents and load_update_documents.",
"type": "patch"
},
{
"description": "Remove NetworkX dependency from graph utilities; move to DataFrame-based implementations in graphrag.graphs package.",
"type": "patch"
},
{
"description": "Remove unnecessary response format check. Fixes: #2203",
"type": "patch"
},
{
"description": "add profiling to get memory usage",
"type": "patch"
},
{
"description": "update notebooks",
"type": "patch"
}
],
"created_at": "2026-02-13T18:01:44+00:00",
"version": "3.0.2"
}
.semversioner/3.0.3.json
+50
View File
@@ -0,0 +1,50 @@
{
"changes": [
{
"description": "Add filtering, timestamp explosion, insert/count/remove/update operations to vector store API. Add top-level vector_size config to VectorStoreConfig.",
"type": "patch"
},
{
"description": "add csv table smoke tests",
"type": "patch"
},
{
"description": "add manual release instructions",
"type": "patch"
},
{
"description": "add streamming to the two first workflows",
"type": "patch"
},
{
"description": "add support for cosmosdb output",
"type": "patch"
},
{
"description": "create_communities streaming",
"type": "patch"
},
{
"description": "create_final_documents streaming",
"type": "patch"
},
{
"description": "create_final_text_units streaming",
"type": "patch"
},
{
"description": "finalize_graph streaming",
"type": "patch"
},
{
"description": "generate_text_embeddings streaming",
"type": "patch"
},
{
"description": "write stats.json per workflow",
"type": "patch"
}
],
"created_at": "2026-02-24T20:13:49+00:00",
"version": "3.0.3"
}
.semversioner/3.0.4.json
+10
View File
@@ -0,0 +1,10 @@
{
"changes": [
{
"description": "fix versions release",
"type": "patch"
}
],
"created_at": "2026-02-24T22:08:37+00:00",
"version": "3.0.4"
}
.semversioner/3.0.5.json
+18
View File
@@ -0,0 +1,18 @@
{
"changes": [
{
"description": "fix csv reader",
"type": "patch"
},
{
"description": "update version",
"type": "patch"
},
{
"description": "vector load_documents in batches",
"type": "patch"
}
],
"created_at": "2026-02-27T21:01:53+00:00",
"version": "3.0.5"
}
.semversioner/3.0.6.json
+14
View File
@@ -0,0 +1,14 @@
{
"changes": [
{
"description": "extract_graph_nlp streaming",
"type": "patch"
},
{
"description": "filter phantom relationships in graph",
"type": "patch"
}
],
"created_at": "2026-03-05T17:22:13+00:00",
"version": "3.0.6"
}
.semversioner/3.0.7.json
+14
View File
@@ -0,0 +1,14 @@
{
"changes": [
{
"description": "Pin litellm dependency.",
"type": "patch"
},
{
"description": "reconfigure vector store size by embedding model",
"type": "patch"
}
],
"created_at": "2026-03-24T16:32:50+00:00",
"version": "3.0.7"
}
.semversioner/3.0.8.json
+10
View File
@@ -0,0 +1,10 @@
{
"changes": [
{
"description": "bump nltk to resolve [CVE-2025-14009](https://github.com/advisories/GHSA-7p94-766c-hgjp)",
"type": "patch"
}
],
"created_at": "2026-03-27T15:53:31+00:00",
"version": "3.0.8"
}
.semversioner/3.0.9.json
+18
View File
@@ -0,0 +1,18 @@
{
"changes": [
{
"description": "Support client side json validation.",
"type": "patch"
},
{
"description": "fix broken documentation links.",
"type": "patch"
},
{
"description": "implement parquet reader",
"type": "patch"
}
],
"created_at": "2026-04-11T01:32:05+00:00",
"version": "3.0.9"
}
.semversioner/3.1.0.json
+14
View File
@@ -0,0 +1,14 @@
{
"changes": [
{
"description": "Native CosmosTableProvider with namespace partitioning, transactional batch writes, and simplified AzureCosmosStorage.",
"type": "minor"
},
{
"description": "Update litellm dependency.",
"type": "patch"
}
],
"created_at": "2026-05-28T15:17:20+00:00",
"version": "3.1.0"
}
.semversioner/next-release/patch-20260710152344024031.json
+4
View File
@@ -0,0 +1,4 @@
{
"type": "patch",
"description": "Fix logging bug."
}
.semversioner/next-release/patch-20260710182840267649.json
+4
View File
@@ -0,0 +1,4 @@
{
"type": "patch",
"description": "Fix #2389 - Loosen service_tier type to str | None to allow for more flexibility in service tier representation."
}
.semversioner/next-release/patch-20260710184002601897.json
+4
View File
@@ -0,0 +1,4 @@
{
"type": "patch",
"description": "Fix .strip call - resolves #2381"
}
.vscode/extensions.json
Vendored
+12
View File
@@ -0,0 +1,12 @@
{
"recommendations": [
"arcanis.vscode-zipfs",
"ms-python.python",
"charliermarsh.ruff",
"ms-python.vscode-pylance",
"bierner.markdown-mermaid",
"streetsidesoftware.code-spell-checker",
"ronnidc.nunjucks",
"lucien-martijn.parquet-visualizer",
]
}
.vscode/launch.json
Vendored
+107
View File
@@ -0,0 +1,107 @@
{
"_comment": "Use this file to configure the graphrag project for debugging. You may create other configuration profiles based on these or select one below to use.",
"version": "0.2.0",
"configurations": [
{
"name": "Indexer",
"type": "debugpy",
"request": "launch",
"module": "graphrag",
"args": [
"index",
"--root",
"${input:root_folder}"
],
"console": "integratedTerminal"
},
{
"name": "Update",
"type": "debugpy",
"request": "launch",
"module": "graphrag",
"args": [
"update",
"--root",
"${input:root_folder}"
],
"console": "integratedTerminal"
},
{
"name": "Query",
"type": "debugpy",
"request": "launch",
"module": "graphrag",
"args": [
"query",
"${input:query}",
"--root",
"${input:root_folder}",
"--method", "${input:query_method}"
]
},
{
"name": "Prompt Tuning",
"type": "debugpy",
"request": "launch",
"module": "uv",
"args": [
"poe", "prompt-tune",
"--config",
"${input:root_folder}/settings.yaml",
]
},
{
"name": "Debug Integration Pytest",
"type": "debugpy",
"request": "launch",
"module": "pytest",
"args": [
"./tests/integration/vector_stores",
"-k", "test_azure_ai_search"
],
"console": "integratedTerminal",
"justMyCode": false
},
{
"name": "Debug Verbs Pytest",
"type": "debugpy",
"request": "launch",
"module": "pytest",
"args": [
"./tests/verbs",
"-k", "test_generate_text_embeddings"
],
"console": "integratedTerminal",
"justMyCode": false
},
{
"name": "Debug Smoke Pytest",
"type": "debugpy",
"request": "launch",
"module": "pytest",
"args": [
"./tests/smoke",
"-k", "test_fixtures"
],
"console": "integratedTerminal",
"justMyCode": false
},
],
"inputs": [
{
"id": "root_folder",
"type": "promptString",
"description": "Enter the root folder path"
},
{
"id": "query_method",
"type": "promptString",
"description": "Enter the query method (e.g., 'global', 'local')"
},
{
"id": "query",
"type": "promptString",
"description": "Enter the query text"
}
]
}
.vscode/settings.json
Vendored
+17
View File
@@ -0,0 +1,17 @@
{
"editor.formatOnSave": false,
"explorer.fileNesting.enabled": true,
"debug.internalConsoleOptions": "neverOpen",
"python.defaultInterpreterPath": "${workspaceRoot}/.venv/bin/python",
"python.languageServer": "Pylance",
"cSpell.customDictionaries": {
"project-words": {
"name": "project-words",
"path": "${workspaceRoot}/dictionary.txt",
"description": "Words used in this project",
"addWords": true
},
"custom": true, // Enable the `custom` dictionary
"internal-terms": true // Disable the `internal-terms` dictionary
}
}
.vsts-ci.yml
+41
View File
@@ -0,0 +1,41 @@
name: GraphRAG CI
pool:
vmImage: ubuntu-latest
trigger:
batch: true
branches:
include:
- main
variables:
isMain: $[eq(variables['Build.SourceBranch'], 'refs/heads/main')]
pythonVersion: "3.13"
poetryVersion: "1.6.1"
nodeVersion: "18.x"
artifactsFullFeedName: "Resilience/resilience_python"
stages:
- stage: Compliance
dependsOn: []
jobs:
- job: compliance
displayName: Compliance
pool:
vmImage: windows-latest
steps:
- task: CredScan@3
inputs:
outputFormat: sarif
debugMode: false
- task: ComponentGovernanceComponentDetection@0
inputs:
scanType: "Register"
verbosity: "Verbose"
alertWarningLevel: "High"
- task: PublishSecurityAnalysisLogs@3
inputs:
ArtifactName: "CodeAnalysisLogs"
ArtifactType: "Container"
CHANGELOG.md
+464
View File
@@ -0,0 +1,464 @@
# Changelog
Note: version releases in the 0.x.y range may introduce breaking changes.
## 3.1.0
- minor: Native CosmosTableProvider with namespace partitioning, transactional batch writes, and simplified AzureCosmosStorage.
- patch: Update litellm dependency.
## 3.0.9
- patch: Support client side json validation.
- patch: fix broken documentation links.
- patch: implement parquet reader
## 3.0.8
- patch: bump nltk to resolve [CVE-2025-14009](https://github.com/advisories/GHSA-7p94-766c-hgjp)
## 3.0.7
- patch: Pin litellm dependency.
- patch: reconfigure vector store size by embedding model
## 3.0.6
- patch: extract_graph_nlp streaming
- patch: filter phantom relationships in graph
## 3.0.5
- patch: fix csv reader
- patch: update version
- patch: vector load_documents in batches
## 3.0.4
- patch: fix versions release
## 3.0.3
- patch: Add filtering, timestamp explosion, insert/count/remove/update operations to vector store API. Add top-level vector_size config to VectorStoreConfig.
- patch: add csv table smoke tests
- patch: add manual release instructions
- patch: add streamming to the two first workflows
- patch: add support for cosmosdb output
- patch: create_communities streaming
- patch: create_final_documents streaming
- patch: create_final_text_units streaming
- patch: finalize_graph streaming
- patch: generate_text_embeddings streaming
- patch: write stats.json per workflow
## 3.0.2
- patch: Add CSVTableProvider
- patch: Add DataReader class for typed dataframe loading from TableProvider across indexing workflows and query CLI
- patch: Add TableProvider abstraction for table-based storage operations
- patch: Add async iterator support to InputReader and use it in load_input_documents and load_update_documents workflows.
- patch: Add table provider factory.
- patch: Fix missed py 3.13.
- patch: Move document ID, human_readable_id, and raw_data initialization from create_final_documents into load_input_documents and load_update_documents.
- patch: Remove NetworkX dependency from graph utilities; move to DataFrame-based implementations in graphrag.graphs package.
- patch: Remove unnecessary response format check. Fixes: #2203
- patch: add profiling to get memory usage
- patch: update notebooks
## 3.0.1
- patch: Fix missing dependency.
## 3.0.0
- major: Monorepo restructure
New Packages:
- graphrag-cache
- graphrag-chunking
- graphrag-common
- graphrag-input
- graphrag-llm
- graphrag-storage
- graphrag-vectors
Changes:
- New config: run graphrag init --force to reinitialize config with new layout and options.
## 2.7.1
- patch: Pin pandas==2.3.3.
## 2.7.0
- minor: Set LiteLLM as default in init_content.
- patch: Fix Azure auth scope issue with LiteLLM.
- patch: Housekeeping toward 2.7.
## 2.6.0
- minor: Add LiteLLM chat and embedding model providers.
- minor: Add LoggerFactory and clean up related API.
- minor: Add config for NLP async mode.
- minor: Add optional input documents to indexing API.
- minor: add customization to vector store
- patch: Add gpt-5 support by updating fnllm dependency.
- patch: Fix all human_readable_id fields to be 0-based.
- patch: Fix multi-index search.
- patch: Improve upon recent logging refactor
- patch: Make cache, storage, and vector_store factories consistent with similar registration support
- patch: Remove hard-coded community rate limiter.
- patch: generate_text_embeddings only loads tables if embedding field is specified.
## 2.5.0
- minor: Add additional context variable to build index signature for custom parameter bag
- minor: swap package management from Poetry -> UV
## 2.4.0
- minor: Allow injection of custom pipelines.
- minor: Refactored StorageFactory to use a registration-based approach
- patch: Fix default values for tpm and rpm limiters on embeddings
- patch: Update typer.
- patch: cleaned up logging to follow python standards.
## 2.3.0
- minor: Remove Dynamic Max Retries support. Refactor typer typing in cli interface
- minor: Update fnllm to latest. Update default graphrag configuration
- patch: A few fixes and enhancements for better reuse and flow.
- patch: Add full llm response to LLM PRovider output
- patch: Fix Drift Reduce Response for non streaming calls
- patch: Fix global search prompt to include missing formatting key
- patch: Upgrade pyarrow dependency to >=17.0.0 to fix CVE-2024-52338
## 2.2.1
- patch: Fix Community Report prompt tuning response
- patch: Fix graph creation missing edge weights.
- patch: Update as workflows
## 2.2.0
- minor: Support OpenAI reasoning models.
- patch: Add option to snapshot raw extracted graph tables.
- patch: Added batching logic to the prompt tuning autoselection embeddings workflow
- patch: Align config classes and docs better.
- patch: Align embeddings table loading with configured fields.
- patch: Brings parity with our latest NLP extraction approaches.
- patch: Fix fnllm to 0.2.3
- patch: Fixes to basic search.
- patch: Update llm args for consistency.
- patch: add vector store integration tests
## 2.1.0
- minor: Add support for JSON input files.
- minor: Updated the prompt tunning client to support csv-metadata injection and updated output file types to match the new naming convention.
- patch: Add check for custom model types while config loading
- patch: Adds general-purpose pipeline run state object.
## 2.0.0
- major: Add children to communities to avoid re-compute.
- major: Reorganize and rename workflows and their outputs.
- major: Rework API to accept callbacks.
- minor: Add LMM Manager and Factory, to support provider registration
- minor: Add NLP graph extraction.
- minor: Add pipeline_start and pipeline_end callbacks.
- minor: Move embeddings snapshots to the workflow runner.
- minor: Remove config inheritance, hydration, and automatic env var overlays.
- minor: Rework the update output storage structure.
- patch: Add caching to NLP extractor.
- patch: Add vector store id reference to embeddings config.
- patch: Export NLP community reports prompt.
- patch: Fix DRIFT search on Azure AI Search.
- patch: Fix StopAsyncIteration catch.
- patch: Fix missing embeddings workflow in FastGraphRAG.
- patch: Fix proper use of n_depth for drift search
- patch: Fix report generation recursion.
- patch: Fix summarization over large datasets for inc indexing. Fix relationship summarization
- patch: Optimize data iteration by removing some iterrows from code
- patch: Patch json mode for community reports
- patch: Properly increment text unit IDs during updates.
- patch: Refactor config defaults from constants to type-safe, hierarchical dataclass.
- patch: Require explicit azure auth settings when using AOI.
- patch: Separates graph pruning for differential usage.
- patch: Tuck flow functions under their workflow modules.
- patch: Update fnllm. Remove unused libs.
- patch: Use ModelProvider for query module
- patch: Use shared schema for final outputs.
- patch: add dynamic retry logic.
- patch: add option to prepend metadata into chunks
- patch: cleanup query code duplication.
- patch: implemented multi-index querying for api layer
- patch: multi index query cli support
- patch: remove unused columns and change property document_attribute_columns to metadata
- patch: update multi-index query to support new workflows
## 1.2.0
- minor: Add Drift Reduce response and streaming endpoint
- minor: add cosmosdb vector store
- patch: Fix example notebooks
- patch: Set default rate limits.
- patch: unit tests for text_splitting
## 1.2.0
- patch: Basic Rag minor fix
## 1.1.1
- patch: Fix a bug on creating community hierarchy for dynamic search
- patch: Increase LOCAL_SEARCH_COMMUNITY_PROP to 15%
## 1.1.0
- minor: Make gleanings independent of encoding
- minor: Remove DataShaper (first steps).
- minor: Remove old pipeline runner.
- minor: new search implemented as a new option for the api
- patch: Fix gleanings loop check
- patch: Implement cosmosdb storage option for cache and output
- patch: Move extractor code to co-locate with operations.
- patch: Remove config input models.
- patch: Ruff update
- patch: Simplify and streamline internal config.
- patch: Simplify callbacks model.
- patch: Streamline flows.
- patch: fix instantiation of storage classes.
## 1.0.1
- patch: Fix encoding model config parsing
- patch: Fix exception on error callbacks
- patch: Manage llm instances inside a cached singleton. Check for empty dfs after entity/relationship extraction
- patch: Respect encoding_model option
## 1.0.0
- patch: Add Parent id to communities data model
- patch: Add migration notebook.
- patch: Create separate community workflow, collapse subflows.
- patch: Dependency Updates
- patch: cleanup and refactor factory classes.
## 0.9.0
- minor: Refactor graph creation.
- patch: Dependency updates
- patch: Fix Global Search with dynamic Community selection bug
- patch: Fix question gen.
- patch: Optimize Final Community Reports calculation and stabilize cache
- patch: miscellaneous code cleanup and minor changes for better alignment of style across the codebase.
- patch: replace llm package with fnllm
- patch: replaced md5 hash with sha256
- patch: replaced md5 hash with sha512
- patch: update API and add a demonstration notebook
## 0.5.0
- minor: Data model changes.
- patch: Add Parquet as part of the default emitters when not pressent
- patch: Centralized prompts and export all for easier injection.
- patch: Cleanup of artifact outputs/schemas.
- patch: Config and docs updates.
- patch: Implement dynamic community selection to global search
- patch: fix autocompletion of existing files/directory paths.
- patch: move import statements out of init files
## 0.4.1
- patch: Add update cli entrypoint for incremental indexing
- patch: Allow some CI/CD jobs to skip PRs dedicated to doc updates only.
- patch: Fix a file paths issue in the viz guide.
- patch: Fix optional covariates update in incremental indexing
- patch: Raise error on empty deltas for inc indexing
- patch: add visualization guide to doc site
- patch: fix streaming output error
## 0.4.0
- minor: Add Incremental Indexing
- minor: Added DRIFT graph reasoning query module
- minor: embeddings moved to a different workflow
- patch: Add DRIFT search cli and example notebook
- patch: Add config for incremental updates
- patch: Add embeddings to subflow.
- patch: Add naive community merge using time period
- patch: Add relationship merge
- patch: Add runtime-only storage option.
- patch: Add text units update
- patch: Allow empty workflow returns to avoid disk writing.
- patch: Apply pandas optimizations to create final entities
- patch: Calculate new inputs and deleted inputs on update
- patch: Collapse covariates flow.
- patch: Collapse create-base-entity-graph.
- patch: Collapse create-final-community-reports.
- patch: Collapse create-final-documents.
- patch: Collapse create-final-entities.
- patch: Collapse create-final-nodes.
- patch: Collapse create_base_documents.
- patch: Collapse create_base_text_units.
- patch: Collapse create_final_relationships.
- patch: Collapse entity extraction.
- patch: Collapse entity summarize.
- patch: Collapse intermediate workflow outputs.
- patch: Dependency updates
- patch: Extract DataShaper-less flows.
- patch: Fix Community ID loading for DRIFT search over existing indexes
- patch: Fix embeddings faulty assignments
- patch: Fix init defaults for vector store and drift img in docs
- patch: Fix nested json parsing
- patch: Fix some edge cases on Drift Search over small input sets
- patch: Fix var name for embedding
- patch: Merge existing and new entities, updating values accordingly
- patch: Merge text_embed into create-final-relationships subflow.
- patch: Move embedding verbs to operations.
- patch: Moving verbs around.
- patch: Optimize Create Base Documents subflow
- patch: Optimize text unit relationship count
- patch: Perf optimizations in map_query_to_entities()
- patch: Remove aggregate_df from final coomunities and final text units
- patch: Remove duplicated relationships and nodes
- patch: Remove unused column from final entities
- patch: Reorganized api,reporter,callback code into separate components. Defined debug profiles.
- patch: Small cleanup in community context history building
- patch: Transient entity graph and snapshotting.
- patch: Update Incremental Indexing to new embeddings workflow
- patch: Use mkdocs for documentation
- patch: add backwards compatibility patch to vector store.
- patch: add-autogenerated-cli-docs
- patch: fix docs image path
- patch: refactor use of vector stores and update support for managed identity
- patch: remove redundant error-handling code from global-search
- patch: reorganize cli layer
## 0.3.6
- patch: Collapse create_final_relationships.
- patch: Dependency update and cleanup
## 0.3.5
- patch: Add compound verbs with tests infra.
- patch: Collapse create_final_communities.
- patch: Collapse create_final_text_units.
- patch: Covariate verb collapse.
- patch: Fix duplicates in community context builder
- patch: Fix prompt tune output path
- patch: Fix seed hardcoded init
- patch: Fix seeded random gen on clustering
- patch: Improve logging.
- patch: Set default values for cli parameters.
- patch: Use static output directories.
## 0.3.4
- patch: Deep copy txt units on local search to avoid race conditions
- patch: Fix summarization including empty descriptions
## 0.3.3
- patch: Add entrypoints for incremental indexing
- patch: Clean up and organize run index code
- patch: Consistent config loading. Resolves #99 and Resolves #1049
- patch: Fix circular dependency when running prompt tune api directly
- patch: Fix default settings for embedding
- patch: Fix img for auto tune
- patch: Fix img width
- patch: Fixed a bug in prompt tuning process
- patch: Refactor text unit build at local search
- patch: Update Prompt Tuning docs
- patch: Update create_pipeline_config.py
- patch: Update prompt tune command in docs
- patch: add querying from azure blob storage
- patch: fix setting base_dir to full paths when not using file system.
- patch: fix strategy config in entity_extraction
## 0.3.2
- patch: Add context data to query API responses.
- patch: Add missing config parameter documentation for prompt tuning
- patch: Add neo4j community notebook
- patch: Ensure entity types to be str when running prompt tuning
- patch: Fix weight casting during graph extraction
- patch: Patch "past" dependency issues
- patch: Update developer guide.
- patch: Update query type hints.
- patch: change-lancedb-placement
## 0.3.1
- patch: Add preflight check to check LLM connectivity.
- patch: Add streaming support for local/global search to query cli
- patch: Add support for both float and int on schema validation for community report generation
- patch: Avoid running index on gh-pages publishing
- patch: Implement Index API
- patch: Improves filtering for data dir inferring
- patch: Update to nltk 3.9.1
## 0.3.0
- minor: Implement auto templating API.
- minor: Implement query engine API.
- patch: Fix file dumps using json for non ASCII chars
- patch: Stabilize smoke tests for query context building
- patch: fix query embedding
- patch: fix sort_context & max_tokens params in verb
## 0.2.2
- patch: Add a check if there is no community record added in local search context
- patch: Add sepparate workflow for Python Tests
- patch: Docs updates
- patch: Run smoke tests on 4o
## 0.2.1
- patch: Added default columns for vector store at create_pipeline_config. No change for other cases.
- patch: Change json parsing error in the map step of global search to warning
- patch: Fix Local Search breaking when loading Embeddings input. Defaulting overwrite to True as in the rest of the vector store config
- patch: Fix json parsing when LLM returns faulty responses
- patch: Fix missing community reports and refactor community context builder
- patch: Fixed a bug that erased the vector database, added a new parameter to specify the config file path, and updated the documentation accordingly.
- patch: Try parsing json before even repairing
- patch: Update Prompt Tuning meta prompts with finer examples
- patch: Update default entity extraction and gleaning prompts to reduce hallucinations
- patch: add encoding-model to entity/claim extraction config
- patch: add encoding-model to text chunking config
- patch: add user prompt to history-tracking llm
- patch: update config reader to allow for zero gleans
- patch: update config-reader to allow for empty chunk-by arrays
- patch: update history-tracking LLm to use 'assistant' instead of 'system' in output history.
- patch: use history argument in hash key computation; add history input to cache data
## 0.2.0
- minor: Add content-based KNN for selecting prompt tune few shot examples
- minor: Add dynamic community report rating to the prompt tuning engine
- patch: Add Minute-based Rate Limiting and fix rpm, tpm settings
- patch: Add N parameter support
- patch: Add cli flag to overlay default values onto a provided config.
- patch: Add exception handling on file load
- patch: Add language support to prompt tuning
- patch: Add llm params to local and global search
- patch: Fix broken prompt tuning link on docs
- patch: Fix delta none on query calls
- patch: Fix docsite base url
- patch: Fix encoding model parameter on prompt tune
- patch: Fix for --limit exceeding the dataframe length
- patch: Fix for Ruff 0.5.2
- patch: Fixed an issue where base OpenAI embeddings can't work with Azure OpenAI LLM
- patch: Modify defaults for CHUNK_SIZE, CHUNK_OVERLAP and GLEANINGS to reduce time and LLM calls
- patch: fix community_report doesn't work in settings.yaml
- patch: fix llm response content is None in query
- patch: fix the organization parameter is ineffective during queries
- patch: remove duplicate file read
- patch: support non-open ai model config to prompt tune
- patch: use binary io processing for all file io operations
## 0.1.0
- minor: Initial Release
CODEOWNERS
+5
View File
@@ -0,0 +1,5 @@
# These owners will be the default owners for everything in
# the repo. Unless a later match takes precedence,
# @global-owner1 and @global-owner2 will be requested for
# review when someone opens a pull request.
* @microsoft/societal-resilience @microsoft/graphrag-core-team
CODE_OF_CONDUCT.md
+9
View File
@@ -0,0 +1,9 @@
# Microsoft Open Source Code of Conduct
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
Resources:
- [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/)
- [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)
- Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns
CONTRIBUTING.md
+79
View File
@@ -0,0 +1,79 @@
# Contributing to GraphRAG
Thank you for your interest in contributing to GraphRAG! We welcome contributions from the community to help improve the project.
## Code of Conduct
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA)
declaring that you have the right to, and actually do, grant us the rights to use your contribution.
For details, visit https://cla.microsoft.com.
When you submit a pull request, a CLA-bot will automatically determine whether you need
to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the
instructions provided by the bot. You will only need to do this once across all repositories using our CLA.
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)
or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
## How to Contribute
1. Fork the repository and clone it to your local machine.
2. Create a new branch for your contribution: `git checkout -b my-contribution`.
3. Make your changes and ensure that the code passes all tests.
4. Commit your changes: `git commit -m "Add my contribution"`.
5. Create and commit a semver impact document by running `uv run semversioner add-change -t <major|minor|patch> -d <description>`.
6. Push your changes to your forked repository: `git push origin my-contribution`.
7. Open a pull request to the main repository.
## Reporting Security Issues
**Please do not report security vulnerabilities through public GitHub issues.** Instead, please report them to the Microsoft Security Response Center (MSRC).
See [SECURITY.md](./SECURITY.md) for more information.
## Before you start, file an issue
Please follow this simple rule to help us eliminate any unnecessary wasted effort & frustration, and ensure an efficient and effective use of everyone's time - yours, ours, and other community members':
> 👉 If you have a question, think you've discovered an issue, would like to propose a new feature, etc., then find/file an issue **BEFORE** starting work to fix/implement it.
### Search existing issues first
Before filing a new issue, search existing open and closed issues first: This project is moving fast! It is likely someone else has found the problem you're seeing, and someone may be working on or have already contributed a fix!
If no existing item describes your issue/feature, great - please file a new issue:
### File a new Issue
- Don't know whether you're reporting an issue or requesting a feature? File an issue
- Have a question that you don't see answered in docs, videos, etc.? File an issue
- Want to know if we're planning on building a particular feature? File an issue
- Got a great idea for a new feature? File an issue/request/idea
- Don't understand how to do something? File an issue
- Found an existing issue that describes yours? Great - upvote and add additional commentary / info / repro-steps / etc.
If from the previous guide you find yourself in the need of file an Issue please use the [issue tracker](https://github.com/microsoft/graphrag/issues).
Provide as much detail as possible to help us understand and address the problem.
### Add information
**Complete the new Issue form, providing as much information as possible**. The more information you provide, the more likely your issue/ask will be understood and implemented. Helpful information includes:
- What device you're running (inc. CPU type, memory, disk, etc.)
- What OS your device is running
- What tools and apps you're using (e.g. VS 2022, VSCode, etc.)
- **We LOVE detailed repro steps!** What steps do we need to take to reproduce the issue? Assume we love to read repro steps. As much detail as you can stand is probably _barely_ enough detail for us!
- Prefer error message text where possible or screenshots of errors if text cannot be captured
- **If you intend to implement the fix/feature yourself then say so!** If you do not indicate otherwise we will assume that the issue is our to solve, or may label the issue as `Help-Wanted`.
### DO NOT post "+1" comments
> ⚠ DO NOT post "+1", "me too", or similar comments - they just add noise to an issue.
If you don't have any additional info/context to add but would like to indicate that you're affected by the issue, upvote the original issue by clicking its [+😊] button and hitting 👍 (+1) icon. This way we can actually measure how impactful an issue is.
---
## Thank you
We appreciate your contributions to GraphRAG!
DEVELOPING.md
+119
View File
@@ -0,0 +1,119 @@
# GraphRAG Development
# Requirements
| Name | Installation | Purpose |
| ------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------- |
| Python 3.10 or 3.11 | [Download](https://www.python.org/downloads/) | The library is Python-based. |
| uv | [Instructions](https://docs.astral.sh/uv/) | uv is used for package management and virtualenv management in Python codebases |
# Getting Started
## Install Dependencies
```shell
# install python dependencies
uv sync
```
## Execute the indexing engine
```shell
uv run poe index <...args>
```
## Execute prompt tuning
```shell
uv run poe prompt_tune <...args>
```
## Execute Queries
```shell
uv run poe query <...args>
```
## Repository Structure
An overview of the repository's top-level folder structure is provided below, detailing the overall design and purpose.
We leverage a factory design pattern where possible, enabling a variety of implementations for each core component of graphrag.
```shell
graphrag
├── api # library API definitions
├── cache # cache module supporting several options
│   └─ factory.py # └─ main entrypoint to create a cache
├── callbacks # a collection of commonly used callback functions
├── cli # library CLI
│   └─ main.py # └─ primary CLI entrypoint
├── config # configuration management
├── index # indexing engine
| └─ run/run.py # main entrypoint to build an index
├── logger # logger module supporting several options
│   └─ factory.py # └─ main entrypoint to create a logger
├── model # data model definitions associated with the knowledge graph
├── prompt_tune # prompt tuning module
├── prompts # a collection of all the system prompts used by graphrag
├── query # query engine
├── storage # storage module supporting several options
│   └─ factory.py # └─ main entrypoint to create/load a storage endpoint
├── utils # helper functions used throughout the library
└── vector_stores # vector store module containing a few options
└─ factory.py # └─ main entrypoint to create a vector store
```
Where appropriate, the factories expose a registration method for users to provide their own custom implementations if desired.
## Versioning
We use [semversioner](https://github.com/raulgomis/semversioner) to automate and enforce semantic versioning in the release process. Our CI/CD pipeline checks that all PR's include a json file generated by semversioner. When submitting a PR, please run:
```shell
uv run semversioner add-change -t patch -d "<a small sentence describing changes made>."
```
For the full end-to-end release process (version bumping, publishing to PyPI, etc.), see [RELEASE.md](RELEASE.md).
# Azurite
Some unit and smoke tests use Azurite to emulate Azure resources. This can be started by running:
```sh
./scripts/start-azurite.sh
```
or by simply running `azurite` in the terminal if already installed globally. See the [Azurite documentation](https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azurite) for more information about how to install and use Azurite.
# Lifecycle Scripts
Our Python package utilizes uv to manage dependencies and [poethepoet](https://pypi.org/project/poethepoet/) to manage custom build scripts.
Available scripts are:
- `uv run poe index` - Run the Indexing CLI
- `uv run poe query` - Run the Query CLI
- `uv build` - This invokes `uv build`, which will build a wheel file and other distributable artifacts.
- `uv run poe test` - This will execute all tests.
- `uv run poe test_unit` - This will execute unit tests.
- `uv run poe test_integration` - This will execute integration tests.
- `uv run poe test_smoke` - This will execute smoke tests.
- `uv run poe check` - This will perform a suite of static checks across the package, including:
- formatting
- documentation formatting
- linting
- security patterns
- type-checking
- `uv run poe fix` - This will apply any available auto-fixes to the package. Usually this is just formatting fixes.
- `uv run poe fix_unsafe` - This will apply any available auto-fixes to the package, including those that may be unsafe.
- `uv run poe format` - Explicitly run the formatter across the package.
## Troubleshooting
### "RuntimeError: llvm-config failed executing, please point LLVM_CONFIG to the path for llvm-config" when running uv sync
Make sure llvm-9 and llvm-9-dev are installed:
`sudo apt-get install llvm-9 llvm-9-dev`
and then in your bashrc, add
`export LLVM_CONFIG=/usr/bin/llvm-config-9`
### "numba/\_pymodule.h:6:10: fatal error: Python.h: No such file or directory" when running uv sync
Make sure you have python3.10-dev installed or more generally `python<version>-dev`
`sudo apt-get install python3.10-dev`
LICENSE
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) Microsoft Corporation.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE
RAI_TRANSPARENCY.md
+41
View File
@@ -0,0 +1,41 @@
# GraphRAG: Responsible AI FAQ
## What is GraphRAG?
GraphRAG is an AI-based content interpretation and search capability. Using LLMs, it parses data to create a knowledge graph and answer user questions about a user-provided private dataset.
## What can GraphRAG do?
GraphRAG is able to connect information across large volumes of information and use these connections to answer questions that are difficult or impossible to answer using keyword and vector-based search mechanisms. Building on the previous question, provide semi-technical, high-level information on how the system offers functionality for various uses. This lets a system using GraphRAG to answer questions where the answers span many documents as well as thematic questions such as “what are the top themes in this dataset?.”
## What are GraphRAG’s intended use(s)?
* GraphRAG is intended to support critical information discovery and analysis use cases where the information required to arrive at a useful insight spans many documents, is noisy, is mixed with mis and/or dis-information, or when the questions users aim to answer are more abstract or thematic than the underlying data can directly answer.
* GraphRAG is designed to be used in settings where users are already trained on responsible analytic approaches and critical reasoning is expected. GraphRAG is capable of providing high degrees of insight on complex information topics, however human analysis by a domain expert of the answers is needed in order to verify and augment GraphRAG’s generated responses.
* GraphRAG is intended to be deployed and used with a domain specific corpus of text data. GraphRAG itself does not collect user data, but users are encouraged to verify data privacy policies of the chosen LLM used to configure GraphRAG.
## How was GraphRAG evaluated? What metrics are used to measure performance?
GraphRAG has been evaluated in multiple ways. The primary concerns are 1) accurate representation of the data set, 2) providing transparency and groundedness of responses, 3) resilience to prompt and data corpus injection attacks, and 4) low hallucination rates. Details on how each of these has been evaluated is outlined below by number.
1) Accurate representation of the dataset has been tested by both manual inspection and automated testing against a “gold answer” that is created from randomly selected subsets of a test corpus.
2) Transparency and groundedness of responses is tested via automated answer coverage evaluation and human inspection of the underlying context returned.
3) We test both user prompt injection attacks (“jailbreaks”) and cross prompt injection attacks (“data attacks”) using manual and semi-automated techniques.
4) Hallucination rates are evaluated using claim coverage metrics, manual inspection of answer and source, and adversarial attacks to attempt a forced hallucination through adversarial and exceptionally challenging datasets.
## What are the limitations of GraphRAG? How can users minimize the impact of GraphRAG’s limitations when using the system?
GraphRAG depends on a well-constructed indexing examples. For general applications (e.g. content oriented around people, places, organizations, things, etc.) we provide example indexing prompts. For unique datasets effective indexing can depend on proper identification of domain-specific concepts.
Indexing is a relatively expensive operation; a best practice to mitigate indexing is to create a small test dataset in the target domain to ensure indexer performance prior to large indexing operations.
## What operational factors and settings allow for effective and responsible use of GraphRAG?
GraphRAG is designed for use by users with domain sophistication and experience working through difficult information challenges. While the approach is generally robust to injection attacks and identifying conflicting sources of information, the system is designed for trusted users. Proper human analysis of responses is important to generate reliable insights, and the provenance of information should be traced to ensure human agreement with the inferences made as part of the answer generation.
GraphRAG yields the most effective results on natural language text data that is collectively focused on an overall topic or theme, and that is entity rich – entities being people, places, things, or objects that can be uniquely identified.
While GraphRAG has been evaluated for its resilience to prompt and data corpus injection attacks, and has been probed for specific types of harms, the LLM that the user configures with GraphRAG may produce inappropriate or offensive content, which may make it inappropriate to deploy for sensitive contexts without additional mitigations that are specific to the use case and model. Developers should assess outputs for their context and use available safety classifiers, model specific safety filters and features (such as https://azure.microsoft.com/en-us/products/ai-services/ai-content-safety), or custom solutions appropriate for their use case.
README.md
+76
View File
@@ -0,0 +1,76 @@
# GraphRAG
👉 [Microsoft Research Blog Post](https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/)<br/>
👉 [Read the docs](https://microsoft.github.io/graphrag)<br/>
👉 [GraphRAG Arxiv](https://arxiv.org/pdf/2404.16130)
<div align="left">
<a href="https://pypi.org/project/graphrag/">
<img alt="PyPI - Version" src="https://img.shields.io/pypi/v/graphrag">
</a>
<a href="https://pypi.org/project/graphrag/">
<img alt="PyPI - Downloads" src="https://img.shields.io/pypi/dm/graphrag">
</a>
<a href="https://github.com/microsoft/graphrag/issues">
<img alt="GitHub Issues" src="https://img.shields.io/github/issues/microsoft/graphrag">
</a>
<a href="https://github.com/microsoft/graphrag/discussions">
<img alt="GitHub Discussions" src="https://img.shields.io/github/discussions/microsoft/graphrag">
</a>
</div>
## Overview
The GraphRAG project is a data pipeline and transformation suite that is designed to extract meaningful, structured data from unstructured text using the power of LLMs.
To learn more about GraphRAG and how it can be used to enhance your LLM's ability to reason about your private data, please visit the <a href="https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/" target="_blank">Microsoft Research Blog Post.</a>
## Quickstart
To get started with the GraphRAG system we recommend trying the [command line quickstart](https://microsoft.github.io/graphrag/get_started/).
## Repository Guidance
This repository presents a methodology for using knowledge graph memory structures to enhance LLM outputs. Please note that the provided code serves as a demonstration and is not an officially supported Microsoft offering.
⚠️ *Warning: GraphRAG indexing can be an expensive operation, please read all of the documentation to understand the process and costs involved, and start small.*
## Diving Deeper
- To learn about our contribution guidelines, see [CONTRIBUTING.md](./CONTRIBUTING.md)
- To start developing _GraphRAG_, see [DEVELOPING.md](./DEVELOPING.md)
- Join the conversation and provide feedback in the [GitHub Discussions tab!](https://github.com/microsoft/graphrag/discussions)
## Prompt Tuning
Using _GraphRAG_ with your data out of the box may not yield the best possible results.
We strongly recommend to fine-tune your prompts following the [Prompt Tuning Guide](https://microsoft.github.io/graphrag/prompt_tuning/overview/) in our documentation.
## Versioning
Please see the [breaking changes](./breaking-changes.md) document for notes on our approach to versioning the project.
*Always run `graphrag init --root [path] --force` between minor version bumps to ensure you have the latest config format. Run the provided migration notebook between major version bumps if you want to avoid re-indexing prior datasets. Note that this will overwrite your configuration and prompts, so backup if necessary.*
## Responsible AI FAQ
See [RAI_TRANSPARENCY.md](./RAI_TRANSPARENCY.md)
- [What is GraphRAG?](./RAI_TRANSPARENCY.md#what-is-graphrag)
- [What can GraphRAG do?](./RAI_TRANSPARENCY.md#what-can-graphrag-do)
- [What are GraphRAG’s intended use(s)?](./RAI_TRANSPARENCY.md#what-are-graphrags-intended-uses)
- [How was GraphRAG evaluated? What metrics are used to measure performance?](./RAI_TRANSPARENCY.md#how-was-graphrag-evaluated-what-metrics-are-used-to-measure-performance)
- [What are the limitations of GraphRAG? How can users minimize the impact of GraphRAG’s limitations when using the system?](./RAI_TRANSPARENCY.md#what-are-the-limitations-of-graphrag-how-can-users-minimize-the-impact-of-graphrags-limitations-when-using-the-system)
- [What operational factors and settings allow for effective and responsible use of GraphRAG?](./RAI_TRANSPARENCY.md#what-operational-factors-and-settings-allow-for-effective-and-responsible-use-of-graphrag)
## Trademarks
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
trademarks or logos is subject to and must follow
[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.
## Privacy
[Microsoft Privacy Statement](https://privacy.microsoft.com/en-us/privacystatement)
README.wehub.md
+7
View File
@@ -0,0 +1,7 @@
# WeHub 来源说明
- 原始项目：`microsoft/graphrag`
- 原始仓库：https://github.com/microsoft/graphrag
- 导入方式：上游默认分支的最新快照
- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
- 本文件仅用于记录来源，不代表 WeHub 是原项目作者
RELEASE.md
+174
View File
@@ -0,0 +1,174 @@
# Release Process
This document describes the end-to-end process for releasing GraphRAG packages.
**Note**: The CI publish workflow (python-publish.yml) is currently non-functional.
Packages must be published manually as described below.
## Prerequisites
- Write access to the `microsoft/graphrag` repository.
- Maintainer or owner role on **all** GraphRAG packages on PyPI.
- A project-scoped PyPI API token (see [PyPI token setup](#generate-pypi-tokens)).
- `uv` installed locally.
- Dependencies synced: `uv sync --all-packages`.
## 1. Prepare the release
Pull the latest changes on `main` and run the release task:
```sh
git checkout main
git pull
```
You need to run the following commands:
```zsh
uv run semversioner release
uv run semversioner changelog > CHANGELOG.md
# verify if the version is correct
$version = uv run semversioner current-version
# check this only on Windows:
if (-not $version) { Write-Error "Failed to get version"; exit 1 }
uv run update-toml update --file packages/graphrag/pyproject.toml --path project.version --value $version
uv run update-toml update --file packages/graphrag-common/pyproject.toml --path project.version --value $version
uv run update-toml update --file packages/graphrag-chunking/pyproject.toml --path project.version --value $version
uv run update-toml update --file packages/graphrag-input/pyproject.toml --path project.version --value $version
uv run update-toml update --file packages/graphrag-storage/pyproject.toml --path project.version --value $version
uv run update-toml update --file packages/graphrag-cache/pyproject.toml --path project.version --value $version
uv run update-toml update --file packages/graphrag-vectors/pyproject.toml --path project.version --value $version
uv run update-toml update --file packages/graphrag-llm/pyproject.toml --path project.version --value $version
uv run python -m scripts.update_workspace_dependency_versions
uv sync --all-packages
```
## 2. Open and merge the release PR
Check `CHANGELOG.md` or any package's `pyproject.toml` to find the new version,
then move the changes to a release branch:
```sh
git switch -c release/v<VERSION>
git add .
git commit -m "Release v<VERSION>"
git tag -a v<VERSION> -m "Release v<VERSION>"
git push origin release/v<VERSION> -u
```
Open a PR targeting `main`. CI checks (semver, linting, tests) will run
automatically. Once approved, merge to `main`.
## 3. Publish to PyPI
Once the PR is merged, switch back to `main`, build, and publish.
You will need a PyPI API token set as the `UV_PUBLISH_TOKEN` environment
variable. See the
[uv docs on publishing](https://docs.astral.sh/uv/guides/package/#building-and-publishing-a-package)
for details.
### Generate PyPI tokens
For each package, go to <https://pypi.org/manage/account/> and create a
project-scoped API token under **API tokens > Add API token**. Select the
specific project as the scope. Copy the token (starts with `pypi-...`) -- it is
only shown once.
If you want to publish all packages with one token and a single `uv publish`
call, create an **account-scoped** token instead of a project-scoped one.
### Build the packages
```sh
git checkout main
git pull
uv sync --all-packages
uv run poe build
```
All wheels and source distributions will be placed in the `dist/` directory.
### Publish all packages at once
This requires an **account-scoped** PyPI token:
```sh
export UV_PUBLISH_TOKEN="pypi-..."
uv publish
```
### Publishing packages individually
If you need to publish packages one at a time (e.g. with separate per-package
tokens), publish them in dependency order. The `graphrag` meta-package depends on
all others, so it goes last.
```sh
# 1. graphrag-common (no internal dependencies)
export UV_PUBLISH_TOKEN="pypi-<token-for-graphrag-common>"
uv publish dist/graphrag_common-<version>*
# 2. graphrag-storage (depends on common)
export UV_PUBLISH_TOKEN="pypi-<token-for-graphrag-storage>"
uv publish dist/graphrag_storage-<version>*
# 3. graphrag-chunking (depends on common)
export UV_PUBLISH_TOKEN="pypi-<token-for-graphrag-chunking>"
uv publish dist/graphrag_chunking-<version>*
# 4. graphrag-vectors (depends on common)
export UV_PUBLISH_TOKEN="pypi-<token-for-graphrag-vectors>"
uv publish dist/graphrag_vectors-<version>*
# 5. graphrag-input (depends on common, storage)
export UV_PUBLISH_TOKEN="pypi-<token-for-graphrag-input>"
uv publish dist/graphrag_input-<version>*
# 6. graphrag-cache (depends on common, storage)
export UV_PUBLISH_TOKEN="pypi-<token-for-graphrag-cache>"
uv publish dist/graphrag_cache-<version>*
# 7. graphrag-llm (depends on cache, common)
export UV_PUBLISH_TOKEN="pypi-<token-for-graphrag-llm>"
uv publish dist/graphrag_llm-<version>*
# 8. graphrag (depends on ALL of the above -- publish last)
export UV_PUBLISH_TOKEN="pypi-<token-for-graphrag>"
uv publish dist/graphrag-<version>*
```
### Verify
After publishing, confirm the new versions are live:
```sh
pip index versions graphrag
```
Or visit <https://pypi.org/project/graphrag/> and check each sub-package page.
## 4. Create a GitHub release
1. Go to <https://github.com/microsoft/graphrag/releases/new>.
2. Select the tag you pushed earlier (e.g. `v3.1.0`). Target: `main`.
3. Title: the version number (e.g. `v3.1.0`).
4. Use a previous release as a template for the release notes, or click
"Generate release notes".
5. Publish the release.
## Package dependency graph (for reference)
```
graphrag-common (no internal deps)
├── graphrag-storage (common)
├── graphrag-chunking (common)
├── graphrag-vectors (common)
├── graphrag-input (common, storage)
├── graphrag-cache (common, storage)
├── graphrag-llm (cache, common)
└── graphrag (all of the above)
```
SECURITY.md
+37
View File
@@ -0,0 +1,37 @@
## Security
Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/).
If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://docs.microsoft.com/en-us/previous-versions/tn-archive/cc751383(v=technet.10)), please report it to us as described below.
## Reporting Security Issues
**Please do not report security vulnerabilities through public GitHub issues.**
Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://msrc.microsoft.com/create-report).
If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://www.microsoft.com/en-us/msrc/pgp-key-msrc).
You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://www.microsoft.com/msrc).
Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue:
* Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.)
* Full paths of source file(s) related to the manifestation of the issue
* The location of the affected source code (tag/branch/commit or direct URL)
* Any special configuration required to reproduce the issue
* Step-by-step instructions to reproduce the issue
* Proof-of-concept or exploit code (if possible)
* Impact of the issue, including how an attacker might exploit the issue
This information will help us triage your report more quickly.
If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://microsoft.com/msrc/bounty) page for more details about our active programs.
## Preferred Languages
We prefer all communications to be in English.
## Policy
Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://www.microsoft.com/en-us/msrc/cvd).
SUPPORT.md
+27
View File
@@ -0,0 +1,27 @@
# Support
## How to file issues and get help
This project uses GitHub Issues to track bugs and feature requests. Please search the existing
issues before filing new issues to avoid duplicates. For new issues, file your bug or
feature request as a new Issue.
For help and questions about using this project, please create a GitHub issue with your question.
## Microsoft Support Policy
# Support for this **PROJECT or PRODUCT** is limited to the resources listed above.
# Support
## How to file issues and get help
This project uses GitHub Issues to track bugs and feature requests. Please search the existing
issues before filing new issues to avoid duplicates. For new issues, file your bug or
feature request as a new Issue.
For help and questions about using this project, please file an issue on the repo.
## Microsoft Support Policy
Support for this project is limited to the resources listed above.
breaking-changes.md
+127
View File
@@ -0,0 +1,127 @@
# GraphRAG Data Model and Config Breaking Changes
This document contains notes about our versioning approach and a log of changes over time that may result in breakage. As of version 1.0 we are aligning more closely with standard [semantic versioning](https://semver.org/) practices. However, this is an ongoing research project that needs to balance experimental progress with stakeholder communication about big feature releases, so there may be times when we don't adhere perfectly to the spec.
There are five surface areas that may be impacted on any given release. They are:
- [CLI](https://microsoft.github.io/graphrag/cli/) - The CLI is the interface most project consumers are using. **Changes to the CLI will conform to standard semver.**
- [API](https://github.com/microsoft/graphrag/tree/main/graphrag/api) - The API layer is the primary interface we expect developers to use if they are consuming the project as a library in their own codebases. **Changes to the API layer modules will conform to standard semver.**
- Internals - Any code modules behind the CLI and API layers are considered "internal" and may change at any time without conforming to strict semver. This is intended to give the research team high flexibility to change our underlying implementation rapidly. We are not enforcing access via tightly controlled `__init__.py` files, so please understand that if you utilize modules other than the index or query API, they may break between releases in a non-semver-compliant manner.
- [settings.yaml](https://microsoft.github.io/graphrag/config/yaml/) - The settings.yaml file may have changes made to it as we adjust configurability. **Changes that affect the settings.yml will result in a minor version bump**. `graphrag init` will always emit compatible starter config, so we recommend always running the command when updating GraphRAG between minor versions, and copying your endpoint information or other customizations over to the new file.
- [Data model](https://microsoft.github.io/graphrag/index/outputs/) - The output data model may change over time as we adjust our approach. **Changes to the data model will conform to standard semver.** Any changes to the output tables will be shimmed for backwards compatibility between major releases, and we'll provide a migration notebook for folks to upgrade without requiring a re-index.
> TL;DR: Always run `graphrag init --path [path] --force` between minor version bumps to ensure you have the latest config format. Run the provided migration notebook between major version bumps if you want to avoid re-indexing prior datasets. Note that this will overwrite your configuration and prompts, so backup if necessary.
# v3
Run the [migration notebook](./docs/examples_notebooks/index_migration_to_v3.ipynb) to convert older tables to the v3 format. Our main goals with v3 were to slim down the core library to minimize long-term maintenance of features that are either largely unused or should have been out of scope for a long time anyway.
## Data Model
We made minimal data model changes that will affect your index for v3. The primary breaking change is that we removed a rarely-used document-grouping capability that resulted in the `text_units` table having a `document_ids` column with a list instead of a single entry in a column called `document_id`. v3 fixes that, and the migration notebook applies the change so you don't need to re-index.
Most of the other changes we made are removal of fields that are no longer used or are out of scope. For example, we removed the UMAP step that generates x/y coordinates for the entities - new indexes will not produce these columns, but they won't hurt anything if they are in your existing tables.
## API
We have removed the multi-search variant from each search method in the API.
## Config
We did make several changes to the configuration model. The best way forward is to re-run `init`, which we always recommend for minor and major version bumps.
This is a summary of changes:
- Removed fnllm as underlying model manager, so the model types "openai_chat", "azure_openai_chat", "openai_embedding", and "azure_openai_embedding" are all invalid. Use "chat" or "embedding".
- fnllm also had an experimental rate limiting "auto" setting, which is no longer allowed. Use `null` in your config as a default, or set explicit limits to tpm/rpm.
- LiteLLM does require a model_provider, so add yours as appropriate. For example, if you previously used "openai_chat" for your model type, this would be "openai", and for "azure_openai_chat" this would be "azure".
- Collapsed the `vector_store` dict into a single root-level object. This is because we no longer support multi-search, and this dict required a lot of downstream complexity for that single use case.
- Removed the `outputs` block that was also only used for multi-search.
- Most workflows had an undocumented `strategy` config dict that allowed fine tuning of internal settings. These fine tunings are never used and had associated complexity, so we removed it.
- Vector store configuration now allows custom schema per embedded field. This overrides the need for the `container_name` prefix, which caused confusion anyway. Now, the default container name will simply be the embedded field name - if you need something custom, add the `index_schema` block and populate as needed.
- We previously supported the ability to embed any text field in the data model. However, we only ever use text_unit_text, entity_description, and community_full_content, so all others have been removed.
- Removed the `umap` and `embed_graph` blocks which were only used to add x/y fields to the entities. This fixed a long-standing dependency issue with graspologic. If you need x/y positions, see the [visualization guide](https://microsoft.github.io/graphrag/visualization_guide/) for using gephi.
- Removed file filtering from input document loading. This was essentially unused.
- Removed the groupby ability for text chunking. This was intended to allow short documents to be grouped before chunking, but is never used and added a bunch of complexity to the chunking process.
# v2
Run the [migration notebook](./docs/examples_notebooks/index_migration_to_v2.ipynb) to convert older tables to the v2 format.
The v2 release renamed all of our index tables to simply name the items each table contains. The previous naming was a leftover requirement of our use of DataShaper, which is no longer necessary.
# v1
Run the [migration notebook](./docs/examples_notebooks/index_migration_to_v1.ipynb) to convert older tables to the v1 format.
Note that one of the new requirements is that we write embeddings to a vector store during indexing. By default, this uses a local lancedb instance. When you re-generate the default config, a block will be added to reflect this. If you need to write to Azure AI Search instead, we recommend updating these settings before you index, so you don't need to do a separate vector ingest.
All of the breaking changes listed below are accounted for in the four steps above.
## Updated data model
- We have streamlined the data model of the index in a few small ways to align tables more consistently and remove redundant content. Notably:
- Consistent use of `id` and `human_readable_id` across all tables; this also insures all int IDs are actually saved as ints and never strings
- Alignment of fields from `create_final_entities` (such as name -> title) with `create_final_nodes`, and removal of redundant content across these tables
- Rename of `document.raw_content` to `document.text`
- Rename of `entity.name` to `entity.title`
- Rename `rank` to `combined_degree` in `create_final_relationships` and removal of `source_degree` and `target_degree` fields
- Fixed community tables to use a proper UUID for the `id` field, and retain `community` and `human_readable_id` for the short IDs
- Removal of all embeddings columns from parquet files in favor of direct vector store writes
### Migration
- Run the migration notebook (some recent changes may invalidate existing caches, so migrating the format it cheaper than re-indexing).
## New required Embeddings
### Change
- Added new required embeddings for `DRIFTSearch` and base RAG capabilities.
### Migration
- Run a new index, leveraging existing cache.
## Vector Store required by default
### Change
- Vector store is now required by default for all search methods.
### Migration
- Run `graphrag init` command to generate a new settings.yaml file with the vector store configuration.
- Run a new index, leveraging existing cache.
## Deprecate timestamp paths
### Change
- Remove support for timestamp paths, those using `${timestamp}` directory nesting.
- Use the same directory for storage output and reporting output.
### Migration
- Ensure output directories no longer use `${timestamp}` directory nesting.
**Using Environment Variables**
- Ensure `GRAPHRAG_STORAGE_BASE_DIR` is set to a static directory, e.g., `output` instead of `output/${timestamp}/artifacts`.
- Ensure `GRAPHRAG_REPORTING_BASE_DIR` is set to a static directory, e.g., `output` instead of `output/${timestamp}/reports`
[Full docs on using environment variables for configuration](https://microsoft.github.io/graphrag/config/env_vars/).
**Using Configuration File**
```yaml
# rest of settings.yaml file
# ...
storage:
type: file
base_dir: "output" # changed from "output/${timestamp}/artifacts"
reporting:
type: file
base_dir: "output" # changed from "output/${timestamp}/reports"
```
[Full docs on using YAML files for configuration](https://microsoft.github.io/graphrag/config/yaml/).
cspell.config.yaml
+31
View File
@@ -0,0 +1,31 @@
$schema: https://raw.githubusercontent.com/streetsidesoftware/cspell/main/cspell.schema.json
version: "0.2"
allowCompoundWords: true
dictionaryDefinitions:
- name: dictionary
path: "./dictionary.txt"
addWords: true
dictionaries:
- dictionary
ignorePaths:
- cspell.config.yaml
- node_modules
- _site
- /project-words.txt
- default_pipeline.yml
- .turbo
- output/
- dist/
- temp_azurite/
- __pycache__
- pyproject.toml
- entity_extraction.txt
- package.json
- tests/
- examples_notebooks/inputs/
- docs/examples_notebooks/inputs/
- "*.csv"
- "*.parquet"
- "*.faiss"
- "*.ipynb"
- "*.log"
dictionary.txt
+228
View File
@@ -0,0 +1,228 @@
# Team
Alonso
Truitt
Trinh
Fernández
# Pythonisms
PYTHONPATH
pycache
nopython
virtualenv
Pythonisms
pyproject
ipynb
pymodule
virtualenv
virtualenvs
signum
ospath
msecs
asdict
isna
getcwd
fillna
noqa
dtypes
ints
genid
isinstance
ismatch
ftype
# Azure
abfs
cosmosdb
Hnsw
odata
# NLP Terms
chunker
wordnet
maxent
punkt
punct
lemmatizer
PROPN
Syntatic
ents
INTJ
rels
# Libraries
Langchain
networkx
graspologic
graphml
dataframe
dataframes
nltk
datashaper
numba
graphintelligence
dotenv
tiktoken
pydantic
faiss
pyarrow
pyaml
Chakra
poethepoet
pyodbc
lancedb
httpx
pymongo
uvloop
aiofiles
asyncio
numpy
pypi
nbformat
semversioner
mkdocs
fnllm
typer
spacy
kwargs
ollama
litellm
# Library Methods
iterrows
rprint
ndarray
nprobe
astype
monkeypatches
nlist
ntotal
quantizer
arun
tiktokens
dtype
linewidths
asearch
iloc
itertuples
isin
nocache
nbconvert
levelno
acompletion
aembedding
# HTML
nbsp
onclick
pymdownx
linenums
twemoji
Gephi
gephi
Gephi's
# Verbs
binarize
prechunked
openai
umap
concat
unhot
groupby
retryer
agenerate
dropna
notna
upserted
# LLM Terms
AOAI
embedder
llm
llms
achat
aembed
# Galaxy-Brain Terms
Unipartite
idempotently
covariate
covariates
logit
confusors
# Charting
plottable
scatterplot
# Resilience Team Terms
megapipeline
columnwise
docstore
datasource
devcontainers
codebases
# Microsoft
MSRC
# Prompt Inputs
ABILA
Abila
ALHAMIA
Alhamia
Asadi
Aurelians
Bataglani
BATAGLANI
Bratinas
dulce
Durke
Firuzabad
Firuzabad's
FIRUZABAD
Krohaara
KROHAARA
POKRALLY
René
Tazbah
TIRUZIA
Tiruzia
Tiruzia's
Verdantis
Verdantis's
# English
skippable
upvote
unconfigured
# Misc
Arxiv
kwds
jsons
txts
byog
# Dulce
astrotechnician
epitheg
unspooled
unnavigated
# Cosmos DB
aiohttp
aiter
colls
serde
upserts
vnext
# Names
Hochul
Ashish
#unified-search
apos
dearmor
venv
docs/blog_posts.md
+54
View File
@@ -0,0 +1,54 @@
<div class="grid cards" markdown>
- [:octicons-arrow-right-24: __GraphRAG: Unlocking LLM discovery on narrative private data__](https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/)
---
<h6>Published February 13, 2024
By [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect; [Steven Truitt](https://www.microsoft.com/en-us/research/people/steventruitt/), Principal Program Manager</h6>
- [:octicons-arrow-right-24: __GraphRAG: New tool for complex data discovery now on GitHub__](https://www.microsoft.com/en-us/research/blog/graphrag-new-tool-for-complex-data-discovery-now-on-github/)
---
<h6>Published July 2, 2024
By [Darren Edge](https://www.microsoft.com/en-us/research/people/daedge/), Senior Director; [Ha Trinh](https://www.microsoft.com/en-us/research/people/trinhha/), Senior Data Scientist; [Steven Truitt](https://www.microsoft.com/en-us/research/people/steventruitt/), Principal Program Manager; [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect</h6>
- [:octicons-arrow-right-24: __GraphRAG auto-tuning provides rapid adaptation to new domains__](https://www.microsoft.com/en-us/research/blog/graphrag-auto-tuning-provides-rapid-adaptation-to-new-domains/)
---
<h6>Published September 9, 2024
By [Alonso Guevara Fernández](https://www.microsoft.com/en-us/research/people/alonsog/), Sr. Software Engineer; Katy Smith, Data Scientist II; [Joshua Bradley](https://www.microsoft.com/en-us/research/people/joshbradley/), Senior Data Scientist; [Darren Edge](https://www.microsoft.com/en-us/research/people/daedge/), Senior Director; [Ha Trinh](https://www.microsoft.com/en-us/research/people/trinhha/), Senior Data Scientist; [Sarah Smith](https://www.microsoft.com/en-us/research/people/smithsarah/), Senior Program Manager; [Ben Cutler](https://www.microsoft.com/en-us/research/people/bcutler/), Senior Director; [Steven Truitt](https://www.microsoft.com/en-us/research/people/steventruitt/), Principal Program Manager; [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect</h6>
- [:octicons-arrow-right-24: __Introducing DRIFT Search: Combining global and local search methods to improve quality and efficiency__](https://www.microsoft.com/en-us/research/blog/introducing-drift-search-combining-global-and-local-search-methods-to-improve-quality-and-efficiency/)
---
<h6>Published October 31, 2024
By Julian Whiting, Senior Machine Learning Engineer; Zachary Hills , Senior Software Engineer; [Alonso Guevara Fernández](https://www.microsoft.com/en-us/research/people/alonsog/), Sr. Software Engineer; [Ha Trinh](https://www.microsoft.com/en-us/research/people/trinhha/), Senior Data Scientist; Adam Bradley , Managing Partner, Strategic Research; [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect</h6>
- [:octicons-arrow-right-24: __GraphRAG: Improving global search via dynamic community selection__](https://www.microsoft.com/en-us/research/blog/graphrag-improving-global-search-via-dynamic-community-selection/)
---
<h6>Published November 15, 2024
By Bryan Li, Research Intern; [Ha Trinh](https://www.microsoft.com/en-us/research/people/trinhha/), Senior Data Scientist; [Darren Edge](https://www.microsoft.com/en-us/research/people/daedge/), Senior Director; [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect</h6>
- [:octicons-arrow-right-24: __LazyGraphRAG: Setting a new standard for quality and cost__](https://www.microsoft.com/en-us/research/blog/lazygraphrag-setting-a-new-standard-for-quality-and-cost/)
---
<h6>Published November 25, 2024
By [Darren Edge](https://www.microsoft.com/en-us/research/people/daedge/), Senior Director; [Ha Trinh](https://www.microsoft.com/en-us/research/people/trinhha/), Senior Data Scientist; [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect</h6>
- [:octicons-arrow-right-24: __Moving to GraphRAG 1.0 – Streamlining ergonomics for developers and users__](https://www.microsoft.com/en-us/research/blog/moving-to-graphrag-1-0-streamlining-ergonomics-for-developers-and-users)
---
<h6>Published December 16, 2024
By [Nathan Evans](https://www.microsoft.com/en-us/research/people/naevans/), Principal Software Architect; [Alonso Guevara Fernández](https://www.microsoft.com/en-us/research/people/alonsog/), Senior Software Engineer; [Joshua Bradley](https://www.microsoft.com/en-us/research/people/joshbradley/), Senior Data Scientist</h6>
</div>
docs/cli.md
+9
View File
@@ -0,0 +1,9 @@
# CLI Reference
This page documents the command-line interface of the graphrag library.
::: mkdocs-typer
:module: graphrag.cli.main
:prog_name: graphrag
:command: app
:depth: 0
docs/config/init.md
+32
View File
@@ -0,0 +1,32 @@
# Configuring GraphRAG Indexing
To start using GraphRAG, you must generate a configuration file. The `init` command is the easiest way to get started. It will create a `.env` and `settings.yaml` files in the specified directory with the necessary configuration settings. It will also output the default LLM prompts used by GraphRAG.
## Usage
```sh
graphrag init [--root PATH] [--force, --no-force]
```
## Options
- `--root PATH` - The project root directory to initialize graphrag at. Default is the current directory.
- `--force`, `--no-force` - Optional, default is --no-force. Overwrite existing configuration and prompt files if they exist.
## Example
```sh
graphrag init --root ./ragtest
```
## Output
The `init` command will create the following files in the specified directory:
- `settings.yaml` - The configuration settings file. This file contains the configuration settings for GraphRAG.
- `.env` - The environment variables file. These are referenced in the `settings.yaml` file.
- `prompts/` - The LLM prompts folder. This contains the default prompts used by GraphRAG, you can modify them or run the [Auto Prompt Tuning](../prompt_tuning/auto_prompt_tuning.md) command to generate new prompts adapted to your data.
## Next Steps
After initializing your workspace, you can either run the [Prompt Tuning](../prompt_tuning/auto_prompt_tuning.md) command to adapt the prompts to your data or even start running the [Indexing Pipeline](../index/overview.md) to index your data. For more information on configuration options available, see the [YAML details page](yaml.md).
docs/config/models.md
+115
View File
@@ -0,0 +1,115 @@
# Language Model Selection and Overriding
This page contains information on selecting a model to use and options to supply your own model for GraphRAG. Note that this is not a guide to finding the right model for your use case.
## Default Model Support
GraphRAG was built and tested using OpenAI models, so this is the default model set we support. This is not intended to be a limiter or statement of quality or fitness for your use case, only that it's the set we are most familiar with for prompting, tuning, and debugging.
GraphRAG uses [LiteLLM](https://docs.litellm.ai/) for calling language models. LiteLLM provides support for 100+ models though it is important to note that when choosing a model it must support returning [structured outputs](https://openai.com/index/introducing-structured-outputs-in-the-api/) adhering to a [JSON schema](https://docs.litellm.ai/docs/completion/json_mode).
Example using LiteLLM as the language model manager for GraphRAG:
```yaml
completion_models:
default_completion_model:
model_provider: gemini
model: gemini-2.5-flash-lite
auth_method: api_key
api_key: ${GEMINI_API_KEY}
embedding_models:
default_embedding_model:
model_provider: gemini
model: gemini-embedding-001
auth_method: api_key
api_key: ${GEMINI_API_KEY}
```
See [Detailed Configuration](yaml.md) for more details on configuration. [View LiteLLM basic usage](https://docs.litellm.ai/docs/#basic-usage) for details on how models are called (The `model_provider` is the portion prior to `/` while the `model` is the portion following the `/`).
## Model Selection Considerations
GraphRAG has been most thoroughly tested with the gpt-4 series of models from OpenAI, including gpt-4 gpt-4-turbo, gpt-4o, and gpt-4o-mini. Our [arXiv paper](https://arxiv.org/abs/2404.16130), for example, performed quality evaluation using gpt-4-turbo. As stated above, non-OpenAI models are supported through the use of LiteLLM but the suite of gpt-4 series of models from OpenAI remain the most tested and supported suite of models for GraphRAG – in other words, these are the models we know best and can help resolve issues with.
Versions of GraphRAG before 2.2.0 made extensive use of `max_tokens` and `logit_bias` to control generated response length or content. The introduction of the o-series of models added new, non-compatible parameters because these models include a reasoning component that has different consumption patterns and response generation attributes than non-reasoning models. GraphRAG 2.2.0 now supports these models, but there are important differences that need to be understood before you switch.
- Previously, GraphRAG used `max_tokens` to limit responses in a few locations. This is done so that we can have predictable content sizes when building downstream context windows for summarization. We have now switched from using `max_tokens` to use a prompted approach, which is working well in our tests. We suggest using `max_tokens` in your language model config only for budgetary reasons if you want to limit consumption, and not for expected response length control. We now also support the o-series equivalent `max_completion_tokens`, but if you use this keep in mind that there may be some unknown fixed reasoning consumption amount in addition to the response tokens, so it is not a good technique for response control.
- Previously, GraphRAG used a combination of `max_tokens` and `logit_bias` to strictly control a binary yes/no question during gleanings. This is not possible with reasoning models, so again we have switched to a prompted approach. Our tests with gpt-4o, gpt-4o-mini, and o1 show that this works consistently, but could have issues if you have an older or smaller model.
- The o-series models are much slower and more expensive. It may be useful to use an asymmetric approach to model use in your config: you can define as many models as you like in the `models` block of your settings.yaml and reference them by key for every workflow that requires a language model. You could use gpt-4o for indexing and o1 for query, for example. Experiment to find the right balance of cost, speed, and quality for your use case.
- The o-series models contain a form of native native chain-of-thought reasoning that is absent in the non-o-series models. GraphRAG's prompts sometimes contain CoT because it was an effective technique with the gpt-4\* series. It may be counterproductive with the o-series, so you may want to tune or even re-write large portions of the prompt templates (particularly for graph and claim extraction).
Example config with asymmetric model use:
```yaml
completion_models:
extraction_completion_model:
model_provider: openai
model: gpt-4o
auth_method: api_key
api_key: ${GRAPHRAG_API_KEY}
query_completion_model:
model_provider: openai
model: o1
auth_method: api_key
api_key: ${GRAPHRAG_API_KEY}
...
extract_graph:
completion_model_id: extraction_completion_model
prompt: "prompts/extract_graph.txt"
entity_types: [organization, person, geo, event]
max_gleanings: 1
...
global_search:
completion_model_id: query_completion_model
map_prompt: "prompts/global_search_map_system_prompt.txt"
reduce_prompt: "prompts/global_search_reduce_system_prompt.txt"
knowledge_prompt: "prompts/global_search_knowledge_system_prompt.txt"
```
Another option would be to avoid using a language model at all for the graph extraction, instead using the `fast` [indexing method](../index/methods.md) that uses NLP for portions of the indexing phase in lieu of LLM APIs.
## Using Custom Models
LiteLLM supports hundreds of models, but cases may still exist in which some users wish to use models not supported by LiteLLM. There are two approaches one can use to connect to unsupported models:
### Proxy APIs
Many users have used platforms such as [ollama](https://ollama.com/) and [LiteLLM Proxy Server](https://docs.litellm.ai/docs/simple_proxy) to proxy the underlying model HTTP calls to a different model provider. This seems to work reasonably well, but we frequently see issues with malformed responses (especially JSON), so if you do this please understand that your model needs to reliably return the specific response formats that GraphRAG expects. If you're having trouble with a model, you may need to try prompting to coax the format, or intercepting the response within your proxy to try and handle malformed responses.
### Model Protocol
We support model injection through the use of a standard completion and embedding Protocol and accompanying factories that you can use to register your model implementation. This is not supported with the CLI, so you'll need to use GraphRAG as a library.
- Our Protocol is [defined here](https://github.com/microsoft/graphrag/blob/main/packages/graphrag-llm/graphrag_llm/completion/completion.py)
- We have a simple mock implementation in our tests that you can [reference here](https://github.com/microsoft/graphrag/blob/main/packages/graphrag-llm/graphrag_llm/completion/mock_llm_completion.py)
Once you have a model implementation, you need to register it with our completion model factory or embedding model factory:
```python
from graphrag_llm.completion import LLMCompletion, register_completion
class MyCustomCompletionModel(LLMCompletion):
...
# implementation
# elsewhere...
register_completion("my-custom-completion-model", MyCustomCompletionModel)
```
Then in your config you can reference the type name you used:
```yaml
completion_models:
default_completion_model:
type: my-custom-completion-model
...
extract_graph:
completion_model_id: default_completion_model
prompt: "prompts/extract_graph.txt"
entity_types: [organization, person, geo, event]
max_gleanings: 1
```
Note that your custom model will be passed the same params for init and method calls that we use throughout GraphRAG. There is not currently any ability to define custom parameters, so you may need to use closure scope or a factory pattern within your implementation to get custom config values.
docs/config/overview.md
+10
View File
@@ -0,0 +1,10 @@
# Configuring GraphRAG Indexing
The GraphRAG system is highly configurable. This page provides an overview of the configuration options available for the GraphRAG indexing engine.
## Default Configuration Mode
The default configuration mode is the simplest way to get started with the GraphRAG system. It is designed to work out-of-the-box with minimal configuration. The main ways to set up GraphRAG in Default Configuration mode are via:
- [Init command](init.md) (recommended first step)
- [Edit settings.yaml for deeper control](yaml.md)
docs/config/yaml.md
+405
View File
@@ -0,0 +1,405 @@
# Default Configuration Mode (using YAML/JSON)
The default configuration mode may be configured by using a `settings.yml` or `settings.json` file in the data project root. If a `.env` file is present along with this config file, then it will be loaded, and the environment variables defined therein will be available for token replacements in your configuration document using `${ENV_VAR}` syntax. We initialize with YML by default in `graphrag init` but you may use the equivalent JSON form if preferred.
Many of these config values have defaults. Rather than replicate them here, please refer to the [constants in the code](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/config/defaults.py) directly.
For example:
```bash
# .env
GRAPHRAG_API_KEY=some_api_key
# settings.yml
default_chat_model:
api_key: ${GRAPHRAG_API_KEY}
```
# Config Sections
## Language Model Setup
### models
This is a set of dicts, one for completion model configuration and one for embedding model configuration. The dict keys are used to reference the model configuration elsewhere when a model instance is desired. In this way, you can specify as many different models as you need, and reference them independently in the workflow steps.
For example:
```yml
completion_models:
default_completion_model:
model_provider: openai
model: gpt-4.1
auth_method: api_key
api_key: ${GRAPHRAG_API_KEY}
embedding_models:
default_embedding_model:
model_provider: openai
model: text-embedding-3-large
auth_method: api_key
api_key: ${GRAPHRAG_API_KEY}
```
#### Fields
- `type` **litellm|mock** - The type of LLM provider to use. GraphRAG uses [LiteLLM](https://docs.litellm.ai/) for calling language models.
- `model_provider` **str** - The model provider to use, e.g., openai, azure, anthropic, etc. [LiteLLM](https://docs.litellm.ai/) is used under the hood which has support for calling 100+ models. [View LiteLLm basic usage](https://docs.litellm.ai/docs/#basic-usage) for details on how models are called (The `model_provider` is the portion prior to `/` while the `model` is the portion following the `/`). [View Language Model Selection](models.md) for more details and examples on using LiteLLM.
- `model` **str** - The model name.
- `call_args`: **dict[str, Any]** - Default arguments to send with every model request. Example, `{"n": 5, "max_completion_tokens": 1000, "temperature": 1.5, "organization": "..." }`
- `api_key` **str|None** - The OpenAI API key to use.
- `api_base` **str|None** - The API base url to use.
- `api_version` **str|None** - The API version.
- `auth_method` **api_key|azure_managed_identity** - Indicate how you want to authenticate requests.
- `azure_deployment_name` **str|None** - The deployment name to use if your model is hosted on Azure. Note that if your deployment name on Azure matches the model name, this is unnecessary.
- retry **RetryConfig|None** - Retry settings. default=`None`, no retries.
- type **exponential_backoff|immediate** - Type of retry approach. default=`exponential_backoff`
- max_retries **int|None** - Max retries to take. default=`7`.
- base_delay **float|None** - Base delay when using `exponential_backoff`. default=`2.0`.
- jitter **bool|None** - Add jitter to retry delays when using `exponential_backoff`. default=`True`
- max_delay **float|None** - Maximum retry delay. default=`None`, no max.
- rate_limit **RateLimitConfig|None** - Rate limit settings. default=`None`, no rate limiting.
- type **sliding_window** - Type of rate limit approach. default=`sliding_window`
- period_in_seconds **int|None** - Window size for `sliding_window` rate limiting. default=`60`, limit requests per minute.
- requests_per_period **int|None** - Maximum number of requests per period. default=`None`
- tokens_per_period **int|None** - Maximum number of tokens per period. default=`None`
- metrics **MetricsConfig|None** - Metric settings. default=`MetricsConfig()`. View [metrics notebook](https://github.com/microsoft/graphrag/blob/main/packages/graphrag-llm/notebooks/04_metrics.ipynb) for more details on metrics.
- type **default** - The type of `MetricsProcessor` service to use for processing request metrics. default=`default`
- store **memory** - The type of `MetricsStore` service. default=`memory`.
- writer **log|file** - The type of `MetricsWriter` to use. Will write out metrics at the end of the process. default`log`, log metrics out using python standard logging at the end of the process.
- log_level **int|None** - The log level when using `log` writer. default=`20`, log `INFO` messages for metrics.
- base_dir **str|None** - The directory to write metrics to when using `file` writer. default=`Path.cwd()`.
## Input Files and Chunking
### input
Our pipeline can ingest .csv, .txt, or .json data from an input location. See the [inputs page](../index/inputs.md) for more details and examples.
#### Fields
- `storage` **StorageConfig**
- `type` **file|memory|blob|cosmosdb** - The storage type to use. Default=`file`
- `encoding`**str** - The encoding to use for file storage.
- `base_dir` **str** - The base directory to write output artifacts to, relative to the root.
- `connection_string` **str** - (blob/cosmosdb only) The Azure Storage connection string.
- `container_name` **str** - (blob/cosmosdb only) The Azure Storage container name.
- `account_url` **str** - (blob only) The storage account blob URL to use.
- `database_name` **str** - (cosmosdb only) The database name to use.
- `type` **text|csv|json** - The type of input data to load. Default is `text`
- `encoding` **str** - The encoding of the input file. Default is `utf-8`
- `file_pattern` **str** - A regex to match input files. Default is `.*\.csv$`, `.*\.txt$`, or `.*\.json$` depending on the specified `type`, but you can customize it if needed.
- `id_column` **str** - The input ID column to use.
- `title_column` **str** - The input title column to use.
- `text_column` **str** - The input text column to use.
### chunking
These settings configure how we parse documents into text chunks. This is necessary because very large documents may not fit into a single context window, and graph extraction accuracy can be modulated. Also note the `metadata` setting in the input document config, which will replicate document metadata into each chunk.
#### Fields
- `type` **tokens|sentence** - The chunking type to use.
- `encoding_model` **str** - The text encoding model to use for splitting on token boundaries.
- `size` **int** - The max chunk size in tokens.
- `overlap` **int** - The chunk overlap in tokens.
- `prepend_metadata` **list[str]** - Metadata fields from the source document to prepend on each chunk.
## Outputs and Storage
### output
This section controls the storage mechanism used by the pipeline used for exporting output tables.
#### Fields
- `type` **file|memory|blob|cosmosdb** - The storage type to use. Default=`file`
- `encoding`**str** - The encoding to use for file storage.
- `base_dir` **str** - The base directory to write output artifacts to, relative to the root.
- `connection_string` **str** - (blob/cosmosdb only) The Azure Storage connection string.
- `container_name` **str** - (blob/cosmosdb only) The Azure Storage container name.
- `account_url` **str** - (blob only) The storage account blob URL to use.
- `database_name` **str** - (cosmosdb only) The database name to use.
- `type` **text|csv|json** - The type of input data to load. Default is `text`
- `encoding` **str** - The encoding of the input file. Default is `utf-8`
### update_output_storage
The section defines a secondary storage location for running incremental indexing, to preserve your original outputs.
#### Fields
- `type` **file|memory|blob|cosmosdb** - The storage type to use. Default=`file`
- `encoding`**str** - The encoding to use for file storage.
- `base_dir` **str** - The base directory to write output artifacts to, relative to the root.
- `connection_string` **str** - (blob/cosmosdb only) The Azure Storage connection string.
- `container_name` **str** - (blob/cosmosdb only) The Azure Storage container name.
- `account_url` **str** - (blob only) The storage account blob URL to use.
- `database_name` **str** - (cosmosdb only) The database name to use.
- `type` **text|csv|json** - The type of input data to load. Default is `text`
- `encoding` **str** - The encoding of the input file. Default is `utf-8`
### cache
This section controls the cache mechanism used by the pipeline. This is used to cache LLM invocation results for faster performance when re-running the indexing process.
#### Fields
- `type` **json|memory|none** - The storage type to use. Default=`json`
- `storage` **StorageConfig**
- `type` **file|memory|blob|cosmosdb** - The storage type to use. Default=`file`
- `encoding`**str** - The encoding to use for file storage.
- `base_dir` **str** - The base directory to write output artifacts to, relative to the root.
- `connection_string` **str** - (blob/cosmosdb only) The Azure Storage connection string.
- `container_name` **str** - (blob/cosmosdb only) The Azure Storage container name.
- `account_url` **str** - (blob only) The storage account blob URL to use.
- `database_name` **str** - (cosmosdb only) The database name to use.
### reporting
This section controls the reporting mechanism used by the pipeline, for common events and error messages. The default is to write reports to a file in the output directory. However, you can also choose to write reports to an Azure Blob Storage container.
#### Fields
- `type` **file|blob** - The reporting type to use. Default=`file`
- `base_dir` **str** - The base directory to write reports to, relative to the root.
- `connection_string` **str** - (blob only) The Azure Storage connection string.
- `container_name` **str** - (blob only) The Azure Storage container name.
- `account_url` **str** - The storage account blob URL to use.
### vector_store
Where to put all vectors for the system. Configured for lancedb by default. This is a dict, with the key used to identify individual store parameters (e.g., for text embedding).
#### Fields
- `type` **lancedb|azure_ai_search|cosmosdb** - Type of vector store. Default=`lancedb`
- `db_uri` **str** (lancedb only) - The database uri. Default=`storage.base_dir/lancedb`
- `url` **str** (blob/cosmosdb only) - Database / AI Search to be used.
- `api_key` **str** (optional - AI Search only) - The AI Search api key to use.
- `audience` **str** (AI Search only) - Audience for managed identity token if managed identity authentication is used.
- `connection_string` **str** - (cosmosdb only) The Azure Storage connection string.
- `database_name` **str** - (cosmosdb only) Name of the database.
- `index_schema` **dict[str, dict[str, str]]** (optional) - Enables customization for each of your embeddings.
- `<supported_embedding>`:
- `index_name` **str**: (optional) - Name for the specific embedding index table.
- `id_field` **str**: (optional) - Field name to be used as id. Default=`id`
- `vector_field` **str**: (optional) - Field name to be used as vector. Default=`vector`
- `vector_size` **int**: (optional) - Vector size for the embeddings. Default=`3072`
The supported embeddings are:
- `text_unit_text`
- `entity_description`
- `community_full_content`
For example:
```yaml
vector_store:
type: lancedb
db_uri: output/lancedb
index_schema:
text_unit_text:
index_name: "text-unit-embeddings"
id_field: "id_custom"
vector_field: "vector_custom"
vector_size: 3072
entity_description:
id_field: "id_custom"
```
## Workflow Configurations
These settings control each individual workflow as they execute.
### workflows
**list[str]** - This is a list of workflow names to run, in order. GraphRAG has built-in pipelines to configure this, but you can run exactly and only what you want by specifying the list here. Useful if you have done part of the processing yourself.
### embed_text
By default, the GraphRAG indexer will only export embeddings required for our query methods. However, the model has embeddings defined for all plaintext fields, and these can be customized by setting the `target` and `names` fields.
Supported embeddings names are:
- `text_unit_text`
- `entity_description`
- `community_full_content`
#### Fields
- `embedding_model_id` **str** - Name of the model definition to use for text embedding.
- `model_instance_name` **str** - Name of the model singleton instance. Default is "text_embedding". This primarily affects the cache storage partitioning.
- `batch_size` **int** - The maximum batch size to use.
- `batch_max_tokens` **int** - The maximum batch # of tokens.
- `names` **list[str]** - List of the embeddings names to run (must be in supported list).
### extract_graph
Tune the language model-based graph extraction process.
#### Fields
- `completion_model_id` **str** - Name of the model definition to use for API calls.
- `model_instance_name` **str** - Name of the model singleton instance. Default is "extract_graph". This primarily affects the cache storage partitioning.
- `prompt` **str** - The prompt file to use.
- `entity_types` **list[str]** - The entity types to identify.
- `max_gleanings` **int** - The maximum number of gleaning cycles to use.
### summarize_descriptions
#### Fields
- `completion_model_id` **str** - Name of the model definition to use for API calls.
- `model_instance_name` **str** - Name of the model singleton instance. Default is "summarize_descriptions". This primarily affects the cache storage partitioning.
- `prompt` **str** - The prompt file to use.
- `max_length` **int** - The maximum number of output tokens per summarization.
- `max_input_length` **int** - The maximum number of tokens to collect for summarization (this will limit how many descriptions you send to be summarized for a given entity or relationship).
### extract_graph_nlp
Defines settings for NLP-based graph extraction methods.
#### Fields
- `normalize_edge_weights` **bool** - Whether to normalize the edge weights during graph construction. Default=`True`.
- `concurrent_requests` **int** - The number of threads to use for the extraction process.
- `async_mode` **asyncio|threaded** - The async mode to use. Either `asyncio` or `threaded`.
- `text_analyzer` **dict** - Parameters for the NLP model.
- `extractor_type` **regex_english|syntactic_parser|cfg** - Default=`regex_english`.
- `model_name` **str** - Name of NLP model (for SpaCy-based models)
- `max_word_length` **int** - Longest word to allow. Default=`15`.
- `word_delimiter` **str** - Delimiter to split words. Default ' '.
- `include_named_entities` **bool** - Whether to include named entities in noun phrases. Default=`True`.
- `exclude_nouns` **list[str] | None** - List of nouns to exclude. If `None`, we use an internal stopword list.
- `exclude_entity_tags` **list[str]** - List of entity tags to ignore.
- `exclude_pos_tags` **list[str]** - List of part-of-speech tags to ignore.
- `noun_phrase_tags` **list[str]** - List of noun phrase tags to ignore.
- `noun_phrase_grammars` **dict[str, str]** - Noun phrase grammars for the model (cfg-only).
### prune_graph
Parameters for manual graph pruning. This can be used to optimize the modularity of your graph clusters, by removing overly-connected or rare nodes.
#### Fields
- `min_node_freq` **int** - The minimum node frequency to allow.
- `max_node_freq_std` **float | None** - The maximum standard deviation of node frequency to allow.
- `min_node_degree` **int** - The minimum node degree to allow.
- `max_node_degree_std` **float | None** - The maximum standard deviation of node degree to allow.
- `min_edge_weight_pct` **float** - The minimum edge weight percentile to allow.
- `remove_ego_nodes` **bool** - Remove ego nodes.
- `lcc_only` **bool** - Only use largest connected component.
### cluster_graph
These are the settings used for Leiden hierarchical clustering of the graph to create communities.
#### Fields
- `max_cluster_size` **int** - The maximum cluster size to export.
- `use_lcc` **bool** - Whether to only use the largest connected component.
- `seed` **int** - A randomization seed to provide if consistent run-to-run results are desired. We do provide a default in order to guarantee clustering stability.
### extract_claims
#### Fields
- `enabled` **bool** - Whether to enable claim extraction. Off by default, because claim prompts really need user tuning.
- `completion_model_id` **str** - Name of the model definition to use for API calls.
- `model_instance_name` **str** - Name of the model singleton instance. Default is "extract_claims". This primarily affects the cache storage partitioning.
- `prompt` **str** - The prompt file to use.
- `description` **str** - Describes the types of claims we want to extract.
- `max_gleanings` **int** - The maximum number of gleaning cycles to use.
### community_reports
#### Fields
- `completion_model_id` **str** - Name of the model definition to use for API calls.
- `model_instance_name` **str** - Name of the model singleton instance. Default is "community_reporting". This primarily affects the cache storage partitioning.
- `graph_prompt` **str | None** - The community report extraction prompt to use for graph-based summarization.
- `text_prompt` **str | None** - The community report extraction prompt to use for text-based summarization.
- `max_length` **int** - The maximum number of output tokens per report.
- `max_input_length` **int** - The maximum number of input tokens to use when generating reports.
### snapshots
#### Fields
- `embeddings` **bool** - Export embeddings snapshots to parquet.
- `graphml` **bool** - Export graph snapshot to GraphML.
- `raw_graph` **bool** - Export raw extracted graph before merging.
## Query
### local_search
#### Fields
- `prompt` **str** - The prompt file to use.
- `completion_model_id` **str** - Name of the model definition to use for Chat Completion calls.
- `embedding_model_id` **str** - Name of the model definition to use for Embedding calls.
- `text_unit_prop` **float** - The text unit proportion.
- `community_prop` **float** - The community proportion.
- `conversation_history_max_turns` **int** - The conversation history maximum turns.
- `top_k_entities` **int** - The top k mapped entities.
- `top_k_relationships` **int** - The top k mapped relations.
- `max_context_tokens` **int** - The maximum tokens to use building the request context.
### global_search
#### Fields
- `map_prompt` **str** - The global search mapper prompt to use.
- `reduce_prompt` **str** - The global search reducer to use.
- `completion_model_id` **str** - Name of the model definition to use for Chat Completion calls.
- `knowledge_prompt` **str** - The knowledge prompt file to use.
- `data_max_tokens` **int** - The maximum tokens to use constructing the final response from the reduces responses.
- `map_max_length` **int** - The maximum length to request for map responses, in words.
- `reduce_max_length` **int** - The maximum length to request for reduce responses, in words.
- `dynamic_search_threshold` **int** - Rating threshold in include a community report.
- `dynamic_search_keep_parent` **bool** - Keep parent community if any of the child communities are relevant.
- `dynamic_search_num_repeats` **int** - Number of times to rate the same community report.
- `dynamic_search_use_summary` **bool** - Use community summary instead of full_context.
- `dynamic_search_max_level` **int** - The maximum level of community hierarchy to consider if none of the processed communities are relevant.
### drift_search
#### Fields
- `prompt` **str** - The prompt file to use.
- `reduce_prompt` **str** - The reducer prompt file to use.
- `completion_model_id` **str** - Name of the model definition to use for Chat Completion calls.
- `embedding_model_id` **str** - Name of the model definition to use for Embedding calls.
- `data_max_tokens` **int** - The data llm maximum tokens.
- `reduce_max_tokens` **int** - The maximum tokens for the reduce phase. Only use if a non-o-series model.
- `reduce_temperature` **float** - The temperature to use for token generation in reduce.
- `reduce_max_completion_tokens` **int** - The maximum tokens for the reduce phase. Only use for o-series models.
- `concurrency` **int** - The number of concurrent requests.
- `drift_k_followups` **int** - The number of top global results to retrieve.
- `primer_folds` **int** - The number of folds for search priming.
- `primer_llm_max_tokens` **int** - The maximum number of tokens for the LLM in primer.
- `n_depth` **int** - The number of drift search steps to take.
- `local_search_text_unit_prop` **float** - The proportion of search dedicated to text units.
- `local_search_community_prop` **float** - The proportion of search dedicated to community properties.
- `local_search_top_k_mapped_entities` **int** - The number of top K entities to map during local search.
- `local_search_top_k_relationships` **int** - The number of top K relationships to map during local search.
- `local_search_max_data_tokens` **int** - The maximum context size in tokens for local search.
- `local_search_temperature` **float** - The temperature to use for token generation in local search.
- `local_search_top_p` **float** - The top-p value to use for token generation in local search.
- `local_search_n` **int** - The number of completions to generate in local search.
- `local_search_llm_max_gen_tokens` **int** - The maximum number of generated tokens for the LLM in local search. Only use if a non-o-series model.
- `local_search_llm_max_gen_completion_tokens` **int** - The maximum number of generated tokens for the LLM in local search. Only use for o-series models.
### basic_search
#### Fields
- `prompt` **str** - The prompt file to use.
- `completion_model_id` **str** - Name of the model definition to use for Chat Completion calls.
- `embedding_model_id` **str** - Name of the model definition to use for Embedding calls.
- `k` **int** - Number of text units to retrieve from the vector store for context building.
- `max_context_tokens` **int** - The maximum context size to create, in tokens.
docs/data/operation_dulce/ABOUT.md
+3
View File
@@ -0,0 +1,3 @@
# About
This document (Operation Dulce) is an AI-generated science fiction novella, included here for the purposes of integration testing.
docs/data/operation_dulce/Operation Dulce v2 1 1.md
+970
View File
@@ -0,0 +1,970 @@
# Operation: Dulce
## Chapter 1
The thrumming of monitors cast a stark contrast to the rigid silence enveloping the group. Agent Alex Mercer, unfailingly determined on paper, seemed dwarfed by the enormity of the sterile briefing room where Paranormal Military Squad's elite convened. With dulled eyes, he scanned the projectors outlining their impending odyssey into Operation: Dulce.
“I assume, Agent Mercer, you’re not having second thoughts?” It was Taylor Cruz’s voice, laced with an edge that demanded attention.
Alex flickered a strained smile, still thumbing his folder's corner. "Of course not, Agent Cruz. Just trying to soak in all the details." The compliance in his tone was unsettling, even to himself.
Jordan Hayes, perched on the opposite side of the table, narrowed their eyes but offered a supportive nod. "Details are imperative. We’ll need your clear-headedness down there, Mercer."
A comfortable silence, the kind that threaded between veterans of shared secrets, lingered briefly before Sam Rivera, never one to submit to quiet, added, "I’ve combed through the last transmission logs. If anyone can make sense of the anomalies, it’s going to be the two of you."
Taylor snorted dismissively. “Focus, people. We have protocols for a reason. Speculation is counter-productive.” The words 'counter-productive' seemed to hang in the air, a tacit reprimand directed at Alex.
Feeling the weight of his compliance conflicting with his natural inclination to leave no stone unturned, Alex straightened in his seat. "I agree, Agent Cruz. Protocol is paramount," he said, meeting Taylor's steely gaze. It was an affirmation, but beneath it lay layers of unspoken complexities that would undoubtedly unwind with time.
Alex's submission, though seemingly complete, didn't escape Jordan, who tilted their head ever so slightly, their eyes revealing a spark of understanding. They knew well enough the struggle of aligning personal convictions with overarching missions. As everyone began to collect their binders and prepare for departure, a quiet resolve took form within Alex, galvanized by the groundwork laid by their interactions. He may have spoken in compliance, but his determination had merely taken a subtler form — one that wouldn't surrender so easily to the forthcoming shadows.
\*
Dr. Jordan Hayes shuffled a stack of papers, their eyes revealing a tinge of skepticism at Taylor Cruz's authoritarian performance. _Protocols_, Jordan thought, _are just the framework, the true challenges we're about to face lie well beyond the boundaries of any protocol._ They cleared their throat before speaking, tone cautious yet firm, "Let's remember, the unknown variables exceed the known. We should remain adaptive."
A murmur of agreement echoed from Sam Rivera, who leaned forward, lacing their fingers together as if weaving a digital framework in the air before them, "Exactly, adaptability could be the key to interpreting the signal distortions and system malfunctions. We shouldn't discount the… erratic."
Their words hung like an electric charge in the room, challenging Taylor's position with an inherent truth. Cruz’s jaw tightened almost imperceptibly, but the agent masked it with a small nod, conceding to the omnipresent threat of the unpredictable.
Alex glanced at Jordan, who never looked back, their gaze fixed instead on a distant point, as if envisioning the immense dark corridors they were soon to navigate in Dulce. Jordan was not one to embrace fantastical theories, but the air of cautious calculation betrayed a mind bracing for confrontation with the inexplicable, an internal battle between the evidence of their research and the calculating skepticism that kept them alive in their field.
The meeting adjourned with no further comments, the team members quietly retreading the paths to their personal preparations. Alex, trailing slightly behind, observed the others. _The cautious reserve Jordan wears like armor doesn't fool me_, he thought, _their analytical mind sees the patterns I do. And that's worth more than protocol. That's the connection we need to survive this._
As the agents dispersed into the labyrinth of the facility, lost in their thoughts and preparations, the base's halogen lights flickered, a brief and unnoticed harbingers of the darkness to come.
\*
A deserted corridor inside the facility stretched before Taylor Cruz, each footstep rhythmic and precise. Cruz, ambitious and meticulous, eyed the troops passing by with a sardonic tilt of the lips. Obedience—it was as much a tool as any weapon in the arsenal, and Cruz wielded it masterfully. To them, it was another step toward unfettered power within the dark bowels of the military complex.
Inside a secluded equipment bay, Cruz began checking over gear with mechanical efficiency. They traced fingers over the sleek surface of an encrypted radio transmitter. "If protocols are maintained," said Cruz aloud, rehearsing the speech for their subordinates, "not only will we re-establish a line of communication with Dulce, but we shall also illuminate the darkest secrets it conceals."
Agent Hayes appeared in the doorway, arms crossed and a knowing glint in their eyes. "You do understand," Jordan began, the words measured and probing, "that once we're in the depths, rank gives way to survival instincts. It's not about commands—it's empowerment through trust."
The sentiment snagged on Cruz's armor of confidence, probing at the insecurities festering beneath. Taylor offered a brief nod, perhaps too curt, but enough to acknowledge Jordan's point without yielding ground. "Trust," Cruz mused, "or the illusion thereof, is just as potent."
Silence claimed the space between them, steeped in the reality of the unknown dangers lurking in the shadows of the mission. Cruz diligently returned to the equipment, the act a clear dismissal.
Not much later, Cruz stood alone, the hollow echo of the bay a stark reminder of the isolation that power often wrought. With each checked box, their resolve steeled further, a silent vow to usher their team through the abyss—whatever it might hold—and emerge enshrined in the respect they so deeply craved.
## Chapter 2
Sam Rivera sat alone in a cramped office, the hum of a dozen servers murmuring a digital lullaby in the background. Surrounded by the glow of multiple screens, their eyes danced across lines of code and intercepted comm signals from Dulce — a kaleidoscope of data that their curious and isolated mind hungered to decrypt.
To an outsider, it might have looked like obsession, this fervent quest for answers. But to Sam, it was a dance — a give and take with the mysteries of the universe. Their fingers paused over the keyboard as they leaned back in the chair, whispering to thin air, "What secrets are you hiding from us?"
The stillness of the room broke with the unexpected arrival of Alex Mercer, whose encroaching shadow loomed over Sam's workspace. The cybersecurity expert craned their neck upwards, met by the ever-so-slight furrow in Alex's brow. "Got a minute, Rivera?"
"Always," Sam said, a smile surfacing as they swiveled to face their mentor more directly. _He has that look — like something's not sitting right with him,_ they noted inwardly.
Alex hesitated, weighing his words carefully. "Our tech is top-tier, but the silence from Dulce... It's not just technology that will see us through, it's intuition and... trust." His gaze pierced through the digital haze, trying to instill something more profound than advice.
Sam regarded Alex for a moment, the sincerity in his voice resonating with their own unspoken desire to prove their worth. "Intuition," they mirrored thoughtfully. "I guess sometimes the numbers don't have all the answers."
Their shared silence held a newfound understanding, a recognition that between the ones and zeros, it was their combined human insights that might prevail against the impossible. As Alex turned to leave, Sam's eyes drifted back to the screens, now seeing them not as barriers to isolate behind, but as windows into the vast and enigmatic challenge that awaited their team.
Outside the office, the persistent buzz of activity in the facility belied the unease that gripped its inhabitants. A restlessness that nibbled on the edges of reality, as though forewarning of the threshold they were soon to cross — from the known into the realm of cosmic secrets and silent threats.
\*
Shadows played against the walls of the cramped underground meeting room, where Alex Mercer stood gazing at the concealed elevator that would deliver them into the bowels of Dulce base. The air was thick, every breath laced with the weight of impending confrontation, the kind one feels when stepping into a legend. Though armed with an array of advanced weaponry and gear, there was an unshakeable sense that they were delving into a conflict where the physical might be of little consequence.
"I know what you're thinking," Jordan Hayes remarked, approaching Mercer. Their voice was low, a blend of confidence and hidden apprehension. "This feels like more than a rescue or reconnaissance mission, doesn't it?"
Alex turned, his features a mask of uneasy resolve. "It's like we're being pulled into someone else’s game. Not just observers or participants, but... pawns."
Jordan gave a short nod, their analytical mind colliding with the uncertain dynamics of this operation. "I've felt that way since the briefing. Like there's a layer we’re not seeing. And yet, we have no choice but to play along." Their eyes locked with Alex's, silently exchanging a vow to remain vigilant.
"You two need to cut the philosophical chatter. We have positions to secure," Taylor Cruz interjected sharply, stepping into their exchange. The authority in Taylor's voice brooked no argument; it was their way of pulling everyone back to the now.
Alex's response was measured, more assertive than moments ago. "Acknowledged, Agent Cruz," he replied, his voice steadier, mirroring the transformation brewing within. He gripped his rifle with a newfound firmness. "Let's proceed."
As they congregated at the elevator, a tension palpable, Sam Rivera piped in with a tone of balanced levity, "Hope everyone’s brought their good luck charms. Something tells me we’re going to need all the help we can get."
Their laughter served as a brief respite from the gravity of their mission, a shared moment that reinforced their common humanity amidst the unknowable. Then, as one, they stepped into the elevator. The doors closed with a silent hiss, and they descended into the darkness together, aware that when they returned, if they returned, none of them would be the same.
\*
The sense of foreboding hung heavier than the darkness that the artificial lights of the elevator shaft failed to fully penetrate. The team was descending into the earth, carrying with them not only the weight of their equipment but also the silent pressure of the invisible war they were about to fight—a war that seemed to edge away from physicality and into the unnervingly psychological.
As they descended, Dr. Jordan Hayes couldn't help but muse over the layers of data that could wait below, now almost longing for the comfort of empirical evidence. _To think that this reluctance to accept other possibilities may have been my biggest blind spot,_ Jordan contemplated, feeling the hard shell of skepticism begin to crack.
Alex caught Jordan's reflective gaze and leaned in, his voice barely a murmur over the hum of the elevator. "Once we're down there, keep that analytical edge sharp. You see through the mazes of the unexplained better than anyone."
The compliment was unexpected and weighed differently than praise from others. This was an acknowledgment from someone who stood on the front lines of the unknown with eyes wide open. "Thank you, Alex," Jordan said, the words carrying a trace of newfound assertiveness. "You can count on me."
The exchange was cut short by a shudder that ran through the elevator, subtle, but enough to make them instinctively hold their breaths. It wasn't the mechanical stutter of old gears but a vibration that seemed to emanate from the very walls of the shaft—a whisper of something that defied natural explanation.
Cruz was the first to react, all business despite the shadow that crossed their expression. "Systems check. Now," they barked out, masking the moment of disquiet with swift command.
Every agent checked their gear, sending confirmation signals through their comms, creating a chorus of electronic beeps that promised readiness. But there was an unspoken question among them: was their technology, their weaponry, their protocols sufficient for what awaited them or merely a fragile comfort?
Against the gravity of the silence that was once again closing in, Sam's voice crackled through, only half-jest. "I'd laugh if we run into Martians playing poker down there—just to lighten the mood, you know?"
Despite—or perhaps because of—the oddity of the moment, this elicited a round of chuckles, an audible release of tension that ran counterpoint to the undercurrent of anxiety coursing through the team.
As the elevator came to a halting, eerie calm at the sub-level, the group stepped off, finding themselves at the threshold of Dulce's mysterious halls. They stood in a tight pack, sharing a cautious glance before fanning out into the unknown, each one acutely aware that the truth was inevitably intertwined with danger.
Into the depths of Dulce, the team advanced, their silence now a shared testament to the camaraderie born of facing the abyss together—and the steel resolve to uncover whatever horrors lay hidden in its shadows.
\*
The weight of the thick metal door closing behind them reverberated through the concrete hallway, marking the final threshold between the familiar world above and the strangeness that lay beneath. Dulce base, a name that had been whispered in the wind-blown deserts above and in the shadowed corners of conspiracy forums, now a tangible cold reality that they could touch — and that touched them back with a chill.
Like lambs led to an altar of alien deities, so did Agents Alex Mercer, Jordan Hayes, Taylor Cruz, and Sam Rivera proceed, their movements measured, their senses heightened. The air was still, almost respectful of the gravity of their presence. Their torch beams sliced through the darkness, uncovering steel doors with warnings that spoke of top secrets and mortal dangers.
Taylor Cruz, stepping firmly into the role of de facto leader, set a brisk pace. "Eyes sharp, people. Comms check, every thirty seconds," Taylor ordered, their voice echoing slightly before being swallowed by the surrounding silence.
Sam, fiddling with a handheld device aimed at detecting electronic anomalies, offered a murmured "Copy that," their usual buoyancy dimmed by the oppressive atmosphere.
It was Jordan Hayes who paused at an innocuous looking panel, nondescript amongst the gauntlet of secured doorways. "Mercer, Rivera, come see this," Jordan’s voice was marked with a rare hint of urgency.
Alex joined Jordan's side, examining the panel which, at a mere glance, seemed just another part of the base's infrastructure. Yet, to the trained eye, it appeared out of place—a facade.
Jordan explained their reasoning as Sam approached, instinctively understanding the significance of what lay beneath, "This panel is a recent addition — covering something they didn't want found."
Before Alex could respond, the soft whir of an approaching drone cut through their muffled exchange. Taylor had looped back upon hearing the commotion. "Explanations later. We can't afford to attract..." Cruz’s voice trailed off as the small airborne device came into view, its sensors locked onto the group.
Sam was the first to react, their tech-savvy mind already steps ahead. "I've got this," they declared, fingers flying over the controls of their own gadgetry to ward off the impending threat.
The drone lingered, its scan seeming more curious than hostile. But within moments, courtesy of Sam's interference, the little sentinel drifted away, retreating into the shadows as if accepting a silent truce. The crew exhaled, a moment of collective relief palpable in the air.
Cruz squared their shoulders, clearly ruffled but not conceding any ground. "Move out," they directed, a hint more forceful than before. "And Rivera, keep that trick handy."
The team pressed onward, the quiet now filled with the soft beeps of regular comms checks, their pace undeterred by the confrontation. Yet, every agent held a renewed sense of wariness, their trust in one another deepening with the knowledge that the base—its technology, its secrets—was alive in a way they hadn't fully anticipated.
As they converged upon a central hub, the imposing doors to the mainframe room stood ajar — an invitation or a trap, neither option comforting. Without a word, they fortified their resolve and stepped through the threshold, where the dim glow of operational LED lights and the distant hum of machinery hinted at Dulce’s still-beating heart.
Solemnly, yet unmistakably together, they moved deeper into the heart of the enigma, ready to unmask the lifeforce of Dulce base or confront whatever existential threat lay in wait. It was in that unwavering march towards the unknown that their destinies were forever cemented to the legacy of Operation: Dulce.
## Chapter 3
The thrumming of monitors cast a stark contrast to the rigid silence enveloping the group. Agent Alex Mercer, unfailingly determined on paper, seemed dwarfed by the enormity of the sterile briefing room where Paranormal Military Squad's elite convened. With dulled eyes, he scanned the projectors outlining their impending odyssey into Operation: Dulce.
\*
The cooling vents hummed in a monotonous drone, but it was the crackle of the comms system coming to life that cut through the lab’s tension. Dr. Jordan Hayes hovered over a table arrayed with alien technology, their fingers delicately probing the enigmatic circuitry retrieved from the crash site. Agent Alex Mercer watched, admiration blooming in silent solidarity for Jordan's deft touch and unspoken drive.
Jordan, always composed, only allowed the faintest furrow of concentration to mar their brow. "What we understand about physics..." they muttered, trailing off as they realigned a translucent component. The device emitted a low pulse, causing Jordan to still. "Could be fundamentally changed by this."
A calculated risk—that's what this was. And for a person of science, a gamble was worth the potential paradigm shift.
"I’ve been thinking," Alex started, his eyes still fixed on the immediately tangible mystery before them. "About what’s at stake here. Not the mission parameters, but what this means for us—humanity."
Jordan glanced up, meeting his eyes just long enough to convey the shared enormity of their situation; the career-defining glory and existential dread entwined. "The quest for understanding always comes at a price. We're standing on the precipice of knowledge that could either elevate us or condemn us."
The charged air between them spiked as Taylor Cruz’s brusque tones sliced through their reverie. "Hayes, Mercer, this isn't philosophy hour. Focus on the task. We need actionable intel, not daydreams."
With a sound of restrained acknowledgment, Jordan returned their gaze to the device, while Alex clenched his jaw, the buzz of frustration dull against the backdrop of Taylor's authoritarian certainty. It was this competitive undercurrent that kept him alert, the sense that his and Jordan's shared commitment to discovery was an unspoken rebellion against Cruz's narrowing vision of control and order.
Then Taylor did something unexpected. They paused beside Jordan and, for a moment, observed the device with something akin to reverence. “If this tech can be understood..." Taylor said, their voice quieter, "It could change the game for us. For all of us.”
The underlying dismissal earlier seemed to falter, replaced by a glimpse of reluctant respect for the gravity of what lay in their hands. Jordan looked up, and for a fleeting heartbeat, their eyes locked with Taylor's, a wordless clash of wills softening into an uneasy truce.
It was a small transformation, barely perceptible, but one that Alex noted with an inward nod. They had all been brought here by different paths and for different reasons. Yet, beneath the veneer of duty, the enticement of the vast unknown pulled them inexorably together, coalescing their distinct desires into a shared pulse of anticipation.
Marshaled back to the moment by the blink of lights and whir of machinery, they refocused their efforts, each movement sharpened by the knowledge that beyond understanding the unearthly artifacts, they might be piecing together the future of their species.
\*
Amidst the sterility of the briefing room, the liminal space between the facts laid out and the hidden truths, sat Sam Rivera, his demeanor an artful balance of focus and a casual disguise of his razor-sharp talent with technology. Across from him, Alex Mercer lingered in thought, the mental cogs turning as each file on Dulce stirred more than curiosity—it beckoned to a past both honored and burdensome.
"You've been quiet, Sam," Alex noted, catching the younger man's contemplative gaze. "Your take on these signal inconsistencies?"
There was a respect in Alex's tone, though a respectful distance remained—a gulf of experience and a hint of protective mentorship that stood between them. Sam nodded, recognizing the space afforded to him, and he couldn't help but feel the weight of expectation pressing upon his shoulders. It wasn't just the mission that was immense, it was the trust being placed in him.
"The patterns are... off," Sam admitted, hesitant but driven. "If I'm right, what we're looking at isn't random—it's a structured anomaly. We need to be ready for anything."
Alex's eyes brightened with a subtle approval that crossed the distance like a silent nod. "Good. Keen eyes will keep us ahead—or at least not blindsided," he said, affirming the belief that inscribed Sam's role as more than the tech personnel—he was to be a guiding intellect in the heart of uncertainty.
Their exchange was cut short by Taylor Cruz's abrupt arrival, his gait brimming with a robust confidence that veiled the sharp undercurrents of his striving nature. "Time to gear up. Dulce waits for no one," Taylor announced, his voice carrying an iron resolve that knew the costs of hesitation—though whether the cost was calculated in human or career terms was an ambiguity he wore like a badge of honor.
As Sam and Alex nodded in unison, the icy chasm of hierarchy and cryptic protocols seemed momentarily to bridge over with an understanding—this mission was convergence, a nexus point that would challenge each of their motives and strength.
They filed out of the briefing room, their footsteps synchronized, a rhythm that spoke volumes of the unknown cadence they would soon march to within the base's veins. For Alex Mercer, the link with Sam Rivera, though distant, was now poised with a mutuality ready to be tested; for Taylor Cruz, the initiative pulsed like a heartbeat, anticipation thinly veiled behind a mask of duty.
In the midst of the descent, they were each alone yet irrevocably joined, stepping closer towards the volatile embrace of Operation: Dulce.
## Chapter 4
The corridors of the Dulce military base were as silent as a tomb and twice as chilling. Alex Mercer walked with a surety that belied his bubbling undercurrents of doubt. The briefing had been definitive, sturdy pillars of facts and protocols, yet as he ventured deeper, the ominous atmosphere gnawed at him—a stark reminder of how much remained unknown.
Jordan Hayes trailed a few steps behind, their detached exterior breaking for a moment as they caught up to Alex. "What's on your mind?" Jordan asked, their astuteness cutting through the unspoken tension.
Alex glanced back at them. This place was a puzzle, a treacherous labyrinth where the walls whispered secrets, and among them, he sensed a call to question, to challenge the narrative they'd been sold. "The silence here... It's almost as if the base is waiting for something—or someone."
"Just stay sharp, Mercer," Jordan cautioned, yet their eyes lingered on the quietude around them, conceiving the same shadow of doubt that unsettled Alex.
Before they could delve into further discussion, the distinctive click of a safety catch echoed in the hollow space. Both agents turned to find Taylor Cruz standing resolute, primed for combat. Taylor's gaze was scrutinizing and cold, a stark contrast to the growing unease that smoldered silently amongst the rest.
"Chatter is a liability," Taylor snapped, with a commanding flair that bordered on tyrannical. "We move forward, eyes open, mouths shut."
Alex felt the tight grip of compliance strangle his gut, a lesson learned under the hard tutelage of rank and order. But here, in the bowels of Dulce, those instincts began to wane, the imperative to adhere now conflicting with the pressing urgency to confront the shadows they were enmeshed in.
Then, unexpectedly, the lights flickered, a power fluctuation—or a sign? Alex's hand instinctively went to his sidearm, his mindset shifting from soldier to skeptic. The base, with its unyielding coldness, had just given them their first nudge into the realm of the speculative, an invitation to peel back the veneer of reality.
"We should consider all possibilities," Alex murmured, more to himself than the others, his voice a barely audible breath against the sterile air of the complex.
Taylor's posture stiffened at the challenge, yet their response was uncharacteristically reserved, notable in its lack of rebuke. "Agreed. For now, keep moving. But stay vigilant."
A surprise—an echo of agreement from the last person Alex expected it from. And there it was, the glimpse of a wrinkle in the unyielding fabric of command, a hint that perhaps they were all starting to sense the strangeness that permeated this place.
Progressing with determined steps, the trio moved deeper, silently acknowledging the evolution of their predicament. It was a small yet transformative concession to the unknown forces at play, an acknowledgment from each agent that, despite their disparate goals and ideals, the true nature of the Dulce base was an enigma that would forge new paths through their convictions.
As they reached the central communications hub, the truth that awaited them lurked in the shadows, its eyes unseen but felt by all. The walls didn't just whisper now; they spoke in tones only the brave—or the foolish—would dare to listen to.
\*
The subterranean silence of Dulce was an oppressive entity of its own, wrapping the team in a cloak of uneasiness as they pressed on through the dimly lit corridor. Jordan Hayes found themselves contemplating the ramifications of each step taken into this suspended world, where the sterile air seemed to mock the gravity of their predicament. The closer they got to the communication hub, the more Jordan's mind wandered toward the realm of the inexplicable.
Beside Jordan, Alex Mercer moved forward with deliberation, his gaze scanning the heavy utility doors they passed—one of which was partially ajar, beckoning them with its darkness. "After you, Dr. Hayes," Alex said, gesturing toward the mysterious opening. A hint of shared understanding passed between them; knowledge was the guiding star of this mission as much as confrontation or recovery.
Jordan peered inside, the beam from their flashlight slicing through the obscurity. The room beyond was a chaotic cascade of papers, overturned furniture, and the particular kind of disorder born from hasty evacuation—or something far more sinister.
"It's like they vanished in the middle of something urgent," Alex murmured, his voice tight with a mix of concern and anticipation. He began to sift through the scattered reports, each page a potential clue to the enigmatic silence that shrouded Dulce.
Behind them, Taylor watched with a disciplined patience, their authority the foundation upon which the operation was built. Their voice cut into the stillness, a reminder of their presence, "Time is not our ally here."
Drawing back from momentary distraction, Jordan acknowledged the wisdom in Taylor's words, yet could feel the shift in their stance—from skeptical, reserved analyst, to a proactive agent within the narrative. "You're right; these documents may hold critical insights. Let's collect what we can and analyze them properly."
From the darkened hollows of the room, shadows seemed to cast subtle judgment as Alex and Jordan worked together with heightened urgency. Taylor, for once, didn't intervene but instead surveyed the entrance, their mind anticipating the unknown variables that lay ahead.
Unexpectedly, a soft hiss emanated from a neglected terminal on the desk. Jordan's head snapped up, their heart rate accelerating at the potential ramifications. Without a word, they moved to the machine, hands driven by the newfound conviction that knowledge was more than power—it was survival.
As Jordan began to extract what data they could from the terminal, the first comprehensible communication from the depths of Dulce in far too long crackled through: an automated distress marker, looping endlessly without further context. It was a revelation, one that reverberated through the group, confirming their fears and igniting an even greater need to press on.
Watching Jordan's dogged determination, Alex witnessed the minor transformation in his colleague unfold—a shift from doubt to action, a sliver of belief in the possibilities beyond their rational understanding. This forge of resolve amidst the alien echoes of Dulce not only bonded them closer as a team but compelled them forward with a sharpened edge of responsibility to the truth, wherever it would lead.
As they collected their findings and regrouped, the base around them imperceptibly changed, the air charged with the vibration of secrets poised on the brink of revelation. And in that charged silence, the group moved on, each now carrying pieces of a puzzle that would soon converge into a picture of galactic significance.
\*
In the chill of the cramped server room, the hum of machinery was the backbone to a symphony of data streams coursing through the air. Dr. Jordan Hayes, nerves alight with the mission's mounting unknowns, patched into the last known coordinates of the unsent distress broadcast they had uncovered. They were so close to the core now – to the truth behind the blackout – it was almost tangible.
Beside them stood Agent Alex Mercer, ever the soldier, yet with eyes that betrayed an intellect craving to understand the murk beneath the surface. "Any progress, Dr. Hayes?" Alex queried, his voice betraying a subtle urgency.
"Getting there," Jordan replied, fingers dancing across the keyboard. "Whoever sent this was cut off mid-transmission. It's as if Dulce itself swallowed the message whole."
Taylor Cruz closed in, their frame casting a long shadow over the duo, evoking an almost palpable wall between them and the forward momentum of their mission. "Time is against us," Taylor intoned, more statement than threat. "What we uncover here determines our next course of action."
Alex acknowledged Taylor with a brisk nod, his stance firm. Yet inwardly, the tightening grip he felt from Taylor's words couldn't throttle the swell of his own investigative instinct. His soldier's obedience had begun to war with the advocate's zeal for unveiling the dark heart of Dulce's secrets.
And then, the unexpected occurred. The screens flashed in unison, spilling a discordant stream of symbols and images that defied immediate analysis. Jordan's breath caught – this was the response they had been fishing for, an alien communication protocol resonating just at the edge of human comprehension.
Each member of the team felt it: a shift in the room’s very atmosphere, like a veil being drawn from their perception. Alex and Jordan stood still, absorbed in the bewilderment of contact, while Taylor, despite their authority, hesitated – a minor betrayal that unease was creeping into even their disciplined heart.
"Thoughts, Rivera?" Taylor rallied, seeking the counsel of Sam Rivera, whose eyes were wide with exhilaration.
Sam stepped forward, breaking the spell of stillness. "It's like nothing I've ever seen before, but I think I can bridge our systems to communicate," they declared, a wisp of optimism braiding their voice. They set about adapting their gear to transmute the foreign signals into something the team could dissect, their actions a testament to the mentorship and belief instilled in them by Mercer and the team.
Taylor observed them, a cold calculation behind their facade, as they weighed the worth of this anomaly. It was a crossroad that potentially led to either monumental breakthrough or unprecedented catastrophe. "Once you've established a line, document everything. We can't afford to miss any detail," Taylor ordered, the words sharper than intended.
The connection was made, and with trembling anticipation, the team listened as the first garbled outputs began to emerge, their very essence promising insights that could alter the course of history. It was an enigmatic dance with the unknown, the pulse of Dulce no longer just a place, but a herald to an alien register the team had yet to decipher.
Together, they stood at the precipice of understanding, where the faint glow of their monitors cast more than just light – it cast the shadow of burgeoning transformation. It was in this moment, in the grasp of an extraterrestrial tongue, that the team, bound by a hunger for knowledge and the raw edge of survival, found their mission reframed from a search for answers to the articulation of a question humankind had yet to fully ask.
Silent in their commune with the inexplicable frequency, they realized they were not merely investigators; they had become liaisons on behalf of Earth, interpreters of a cosmic message that could redefine their very existence. The implications loomed large, but now, they would not face them alone – they would face them as a united front, wrought together by the very mysteries that once drove them apart.
## Chapter 5
Dr. Jordan Hayes clutched the edge of the briefing room table, their fingers white-knuckled against the laminate surface, as an array of constellations rotated on the projector—charts and graphs bleeding across the stars. In the dim room, nebulas and dark matter seemed within arm's reach, tangible yet unfathomable.
Sam Rivera leaned back against the wall, arms crossed, gaze darting between the swirling cosmos and the faces of their companions. A taut line of concentration etched their young features, a mingling of fervent curiosity with the nascent understanding of the high stakes for which they played.
Jordan's voice broke the profound silence. "The patterns in the signal disruptions sync with none other than zenithal star alignments. It's as if... as if these 'meet and greets' were scheduled, predestined by celestial mechanics."
The statement hung heavy, daring the occupants of the room to unravel its implications. Alex Mercer, his prior military resolve momentarily suspended, absorbed the hypothesis with a visible hunger. "It's like we're adhering to an appointment we never knew we had," he murmured, his heart a drumbeat in his chest.
Taylor Cruz snorted—a sound that clattered against the high concepts like a tumbledown shack in a futurist cityscape. Folding their arms, they glanced between the agents, their apprehension clad in the contempt of practicality. "What we need are facts, not mystic conjecture."
Alex pivoted on his heel, facing Taylor squarely, and his voice found its edge of steel. "This isn't mysticism, Cruz. It's a hypothesis based on observed phenomena as unpredictable as the place we're standing in."
Taylor's gaze never wavered, yet the slight twitch at the corner of their mouth belied their taut composure. "If there's a semblance of truth to it, then it's critical intel. But remember, we're not astrologers—we're soldiers and scientists."
Jordan met Taylor’s gaze with a curt nod, accepting the caution even as the crucible of their intellect smoldered with the fervor of cosmic discovery. Their eyes flicked to Sam, whose steady presence and ready tech affirmed a burgeoning dynamic—the makings of a sentinel, standing guard over the threshold of human understanding and cosmic reality.
With the projector casting pallid light over their features, each agent became a silhouette of purpose, shadows pillared against the backdrop of an endless universe. The story they were embroiled in would soon demand they plunge into darkness to retrieve the light of knowledge—a light that could very well redraw the shape of their world.
They left the briefing room with a shared silence, each pondering the vast weave of celestial intent and terrestrial response, sensing that the galactic appointment to which they'd unwittingly RSVP’d was more insistent—and more threatening—than any operation they’d faced before.
\*
As the Paranormal Military Squad team convened in the heart of the Dulce military complex, an air of bristling expectation clung to the walls of the underground sanctum. Alex Mercer’s brow furrowed while watching his companions—Jordan Hayes, diligently setting up their makeshift lab station, and Sam Rivera meticulously checking the communication relays they had restored. Taylor Cruz observed with hawk-like focus, yet to betray the strain that their command posed on them.
The gravity of the mission had shifted, deepened; each member of the team felt its pull, tethered to the understanding that they were now part of a larger narrative—a cosmic play with Earth as a stage and the human race unwitting actors.
Jordan paused, a tension creeping across their shoulders as they aligned the satellite data with the alien message that had been decoded. "The instructions in this message," Jordan started, the timbre of their voice betraying their usual composure. "They're coordinates and... a warning."
Sam leaned in, their eyes widening behind the glow of their laptop screen. "A warning? Like, ‘stay away from’, or ‘beware of’...?" Their words trailed off, uncertainty a new companion in their lexicon.
Alex exhaled slowly, his mind racing to connect the dots. "It doesn't matter which," he said, decisive yet contemplative. "What matters is we understand intent. Are we being warned out of concern, or are we stumbling upon a threat?"
Cruz’s iron-clad facade momentarily cracked, a fleeting glimpse of vulnerability flashing through their eyes. "We need to know if this entails additional risk to the operation," they said, directing their gaze specifically at Alex. "Mercer, I rely on you to keep the team grounded. No one goes off-course."
Their reminder seemed both a command and a plea—rooted in an understanding that each member of the team now faced the duality of their roles, protectors of earthly secrets and heralds of potentially devastating revelations.
Sam's fingers stilled mid-type, their task forgotten as they absorbed the weight of the unfolding reality. "We're the first line of defense... or detection," they mused half to themselves, a growing sense of agency within the larger play they were cast into.
Jordan returned to the data, more resolute in their actions. The warning, whether cautionary or dire, was a beacon they no longer could ignore; its light casting aside shadows of doubt and igniting a collective purpose within the team.
Alex watched Jordan and Sam, feeling a brotherhood in their shared quest. As Cruz paced, poised on the cusp of decisions that would mark their career and perhaps the fate of many, Alex knew the narrative had changed. They were no longer mere operatives; they had become guardians of a threshold, keepers of a message from a realm beyond stars and stripes. This elevation in their mission could not be shackled by regulations and established protocols—it demanded a new perspective, a new resolve.
Tension threaded through the dialogue of beeps and static as communications with Washington buzzed in the background. The team stood, a portentous air enveloping them. It was clear that the decisions they made in the ensuing hours could redefine humanity's place in the cosmos or condemn them to ignorance and potential peril.
Their connection to the stars solidified, the group moved to address the crystallizing warning, shifting from passive recipients to active participants. Mercer’s latter instincts gained precedence— the team’s mandate had evolved, no longer solely to observe and report but to interact and prepare. A metamorphosis had begun, and Operation: Dulce hummed with the newfound frequency of their daring, a tone set not by the earthly hierarchies but by the pulsing symphony of the universe itself.
\*
The desert night loomed eerily still as echoes of hidden activity reverberated deep beneath the bleak sands of New Mexico. Diverting his gaze from the array of sensors before him, Jordan Hayes allowed a rare breath, deep and anxious. Turning to Alex Mercer's focused silhouette, the nocturnal landscape illuminated softly by makeshift floodlights, Jordan felt the syncopated tempo of apprehension and exhilaration jockey for primacy within.
"The closer we get to unlocking these messages, the more I feel like we're peeling back layers of reality itself," Jordan confided, eyes not leaving the monitors that presented a constellation of data points.
"Yes," Alex replied, his voice steady as he considered the implications of their discovery. "And we have to be ready for whatever we find beneath those layers. Whether it's a breakthrough or a Pandora's Box."
Silence settled between them, broken only by the occasional buzz of communications equipment attempting to bridge terrestrial and extraterrestrial intelligences. Tense moments drifted by, laden with the expectant weight of near breakthrough, when a soft chime signaled an incoming transmission -- a rare sound that set every agent on high alert.
Absent was the voice of Washington or Paranormal Military Squad command. Instead, a rhythmic series of pulses and tones filled the air, deliberately patterned, unmistakably non-human.
Sam Rivera adjusted the sensitivity of the decoding equipment, their hands shaking with anticipation as much as focus. "I have it!" they announced, the signal transforming under their expertise into a sequence of visual symbols on the screen before them.
Their shared excitement was palpable, a kinetic force resonating between the team members as they crowded around the display.
"What does it say?" Taylor Cruz demanded, the urgency in his tone scraping against the newfound wonderment.
Interpreting the alien syntax required not only decoding but intuition and empathy. The words that emerged upon the screen were at once coherent and enigmatic: "*Voyage. Convergence. Peril.*"
The stark simplicity of the message struck them collectively, a chill breeze wafting through their resolve.
Alex stepped forward, piecing together the cryptic communication with a growing sense of obligation. "It’s a call to action," he deduced, "or possibly a summons."
Jordan's gaze met Alex’s, both understanding that this was no longer an investigation or mere extraction of hidden truths. This was humanity's unwitting enlistment into a galactic dialogue that defied boundaries of nation, creed, or protocol.
Sam's eyes were aglow, not with fear, but with the profound acceptance of inevitability that comes with groundbreaking revelation. Moreover, within Taylor's stern exterior churned the seed of reluctant admiration for the unclassified, the uncharted realms they were approaching.
Together, they accepted the pivot in their mission, readjusting their objectives from exploration to engagement, and from isolation to a communal outreach beyond the stars. As dawn's first light threatened the horizon, it became clear that they were no longer merely operatives of a clandestine governmental faction—they were delegates on behalf of Earth, embarking on a voyage orchestrated by destinies unrelated to the mere geopolitics of their world.
Turning to each other, their silhouettes sketched against the coming dawn, the agents recognized the transformation within and amongst them. They were bound by more than duty—they were intricately woven into the fabric of an unfolding cosmic opera, one in which they had been granted an undeniable role. And as they set course for the coordinates that beckoned them like a distant siren's call, it was with a solemn dedication to not only uncover the mysteries ahead but to navigate the convergence, and the peril, as unified emissaries of a world on the cusp of a broader understanding.
\*
Beneath the hum of the fluorescent lights and the vigilance of silent monitors, Alex Mercer stood with his team in the threshold of the base's command center, their faces etched with the fatigue of hours spent unraveling galactic mysteries. Jordan Hayes broke the stillness with a delicate fusion of disbelief and resolve. "The signal..." they began, their tone deliberate, "it’s evolving. It’s not just sending a message—it’s responding to us."
Taylor Cruz leaned over the console, their eyes narrowing with intrigue and a flicker of unease, studying the alternating patterns on the screen. "Responding? Like it’s alive?" Taylor asked, a question that bordered on the edge of wonder and alarm.
Sam Rivera’s gaze was locked onto their interface, a digital orchestra at their fingertips. "It could be some form of advanced AI. Or something else entirely," they contributed, a note of exhilaration betraying the gravity of the situation.
Alex paced before the terminal, absorbing the enormity of their predicament. Their mission—once rooted in the solid ground of military discipline and covert operations—had transcended into an encounter of unprecedented import. "We need to be cautious," he advised, his voice a low rumble of cautious strategy. "If this signal is intelligent, how we interact with it could dictate the outcome of this entire operation."
Jordan met Alex's gaze with a nod, the weight of the responsibility shared and accepted. "We have protocols for first contact, but nothing for... this," Jordan admitted. The room was gripped with tension, each breath seemingly louder than the last.
Then, with a sudden burst that filled the command center, the signal coalesced into a clear and distinct pattern which replicated and expanded, its complexity revealing the hand—or mind—of an intelligent architect.
Taylor's instinct for command surged forth. "Prepare to record and analyze. Whatever it is, we need to understand it—" But their words were cut short as the signal surged, enveloping the room in a brief, blinding cascade of light.
In that pulse of brilliance, a shared revelation coursed through the team. The signal had become a bridge, an extension of unknown consciousness reaching towards them, testing, communicating, searching.
Alex stepped back from the light, feeling a profound change unravelling within him. The path forward would not be one of confrontation or conquest, but of connection and comprehension.
Jordan turned to Alex and Taylor, seeing in their faces a reflection of the same metamorphosis taking place within themselves—a movement from observers to participants, from agents to ambassadors.
With a collective breath, the team faced the kaleidoscope of lights. The alien signal, once a harbinger of enigma, was now a catalyst for transformation—a symphony of light and sound that echoed the beginnings of a new relationship between humanity and the alien unknown.
And so, with deliberate steps, Alex Mercer led his team into the luminous fray. Science, protocol, and survival instinct harmonized within them, each member poised on the cusp of a new chapter in human history.
They were no longer merely the instruments of Paranormal Military Squad's will—they were the vanguard of humankind’s first definitive leap into the cosmic community.
With the last echoes of the signal resonating in the control room, they each embraced the sequencing of the transmission, the dance of extraterrestrial light that now wrote itself into their story. The chapter of Operation: Dulce drew to a close, but the narrative of their destiny had only just begun.
## Chapter 6
\*
The cool darkness of the command center at Dulce base was a stark contrast to the brewing storm outside, where the unforgiving New Mexico desert winds whispered of the hidden truths that lay buried deep beneath its surface. Dr. Jordan Hayes sat, their eyes fixed on the readout, the frenetic dance of symbols and numbers reflecting off their determined face. They were on the cusp of an epiphany, teetering between the widely accepted laws of physics and the promise of a new cosmic paradigm.
Alex Mercer watched from across the room, noting the subtle shifts in Jordan’s posture that belied a developing readiness to embrace the unbelievable. “Find something?” Alex’s question, asked with a blend of curiosity and solidarity, bridged the gap between a command and a genuine query among equals.
Jordan's response was slow, measured against the magnitude of their analysis. “This isn’t random static. It’s a pattern - a repeated sequence phasing in and out but distinctly artificial.” Jordan turned away from the screen, locking eyes with Alex. “This could change everything.”
Sam Rivera leaned in, their eyes alight with the fires of revelation and a quenchless thirst for understanding. “A pattern means intention. Could it be a message?”
A figure emerged from the doorway, casting a long shadow into the room - Taylor Cruz. “Intentions can be friendly, or hostile. We shouldn’t forget that,” said Taylor, bringing a dose of their usual pragmatism into the heart of discovery.
Alex acknowledged Taylor’s caution with a nod, understanding the need to keep their feet grounded even as their spirits soared toward the unknown. “Then let’s be the first to find out which it is."
The team gathered around the monitors, the soft tapping of Jordan's keystrokes now punctuated by the occasional crackle of Sam's radio equipment. The sound was almost ritualistic, a prelude to humanity’s potential first, knowing foray into a larger universe.
Jordan’s fingers paused, suspended in mid-air. The signal had evolved, becoming a beacon that somehow felt less alien and more familiar. It was as if the complexities of their message were unfolding into something more accessible, more terrestrial.
A hushed excitement swept through the room. The transformation suggested an awareness on the part of the unknown senders; a finesse that spoke volumes about their capabilities and perhaps their intentions.
With the growing realization that they were engaging with an intelligence far exceeding their previous understanding, the team prepared to reach back across the cosmic divide. Prepared or not, they were no longer bystanders in this galactic narrative. They were active correspondents in an exchange that transcended galaxies and welcomed them into an expansive, possibly fraught, interstellar conversation.
\*
Inside the cavernous central hub of Dulce military base, Dr. Jordan Hayes stood in near-darkness, surrounded by a nest of cables and monitors that buzzed with silent, cryptic life. Jordan's eyes narrowed to focus on the sequences that danced across the screen—patterns that could unravel the cosmic enigma surrounding them.
Alex Mercer approached with his characteristic stride, a signal of reliability in the chaos. "Status report, Dr. Hayes?" he inquired, his voice low, almost blending into the soundscape of beeping consoles and swirling fans.
"We're on the brink of unravelling the signal's origin," Jordan replied, the weight of implications heavy in their tone. "There's intelligence behind it, a thought process alien to our own."
As if summoned by their analysis, Taylor Cruz approached with authority radiating from every pore. "Understand this, we need to know if it's friend or foe. Don't get wrapped up in the existential—our lives may depend on the answers you provide."
Sam Rivera, their hands adroitly adjusting a device to fine-tune the signal, chimed in with optimism undercut by anxious anticipation. "We're deciphering the comm encryption. Soon, we'll have a channel open—not just listening in, but speaking back."
Alex nodded his understanding, his strategic mind processing the tactical implications while grappling with the more profound humanistic impact. "When we do, we'll tread carefully, communicate with purpose," he reassured the team.
The operation had evolved rapidly, from a stealthy incursion into a clandestine labyrinth to an exchange with an extraterrestrial intellect. Their earlier trepidation transformed into determined focus, as they prepared to extend humanity’s hand into the vast unknown.
An alert on one of the monitor stations snapped the team into alarm. The signal had not simply been waiting—it had been calculating. Now, it reached its crescendo, demanding their attention with a provocative urgency.
Jordan's fingers raced over the keyboard, their eyes simultaneously interpreting data and sharing directives. "It’s a linguistic lock, a test of comprehension. We crack this, we establish dialogue."
Taylor's presence was a beacon of steely resolve. "Then let’s solve it. This is what we trained for—the unknown."
Alex and Sam exchanged a look that telegraphed their shared determination—this was not only the mission they had trained for; it was the mission they had been destined for.
Together, the Paranormal Military Squad team leaned into the challenge, their minds honing in on the complex patterns with a singular goal: to unlock the conversation with an intelligence that had already begun to shift the foundations of what they knew, or thought they knew, about the universe.
In a symphony of clicks and murmurs, they worked, knowing they were about to make a giant leap not just for themselves or Paranormal Military Squad, but for all of humanity. As the final pieces fell into place, Dulce's militaristic silence was shattered by the sound of intergalactic contact—by the sound of history being made.
## Chapter 7
In the enclosed space of Dulce’s command center, the air was thick with anticipation, each team member poised to tread the razor's edge between scientific breakthrough and galactic peril. Dr. Jordan Hayes focused intently on the screen, their fingers tapping a staccato rhythm against the keyboard as lines of alien code cascaded down the monitor.
Alex Mercer's steely gaze surveyed the room, stopping on each member of his team. "Thoughts?" he asked, echoing the unspoken tension. His question, while directed at the group, lingered on Jordan—acknowledging their expertise and inviting collaboration rather than dictating orders.
Jordan’s brow furrowed, an indicator of the mental gymnastics being performed. "It's unprecedented," they finally said, their voice a testament to the gravity of the moment. "Behavioral algorithms... if we're right, this code could reveal extraterrestrial thought patterns."
Before anyone could react, Taylor Cruz interjected with the assertiveness of someone accustomed to commandeering the discourse. "Then let’s ensure we’re deciphering it correctly," Taylor stated, their tone suggesting they were still battling to maintain control over an increasingly alien situation.
Sam Rivera hovered near the mainframe, youthful energy barely contained under the surface. "What if it’s more than just a message? What if they’re trying to extend consciousness across the stars?"
The room fell into a contemplative silence, broken only by the hum of electronic equipment and the distant thud of secured doors locking in rhythm. The weight of responsibility rested on each agent's shoulders—a heaviness palpable in the air they shared.
Alex stepped forward, reaching a subtle decision, one dictated by foresight and the humanity nestled at the core of their mission. "We approach with the aim to understand, not to confront," he said, softening his military bearing into a more diplomatic stance.
Jordan nodded, appreciating the leadership that Alex displayed in the face of the unknown, and turned back to the cryptic data. Here, before them all, was a tangible piece of evidence—proof of an extraterrestrial sentience that had outreached the bounds of their expectations.
Taylor took a breath, simultaneously exuding a sense of preparedness and venturing into the unknown alongside their peers. "Then let’s do what Paranormal Military Squad does best—investigate and adapt," Taylor added, finding comfort in the familiar even as they stood on the cusp of an unprecedented alchemy of science and mystery.
The team leaned into their respective roles, driven by the urgency of the assignment and the pull of an insatiable curiosity. Sam offered a grin that belied the tension, a youthfulness that reminded them all of the profound excitement nested within the terror of the unknown.
Quietly but resolutely, they turned back to their instruments, each of them a sentinel on the threshold of a new reality. The once implicit lines of command were now woven into a shared tapestry of hierarchy and camaraderie. As they danced with the unknown, they were beacons of sentient endeavor, casting the light of human consciousness into the vast darkness that called to them.
\*
\*
Dulce Base's cavernous darkness was pierced by the sharp luminescence of monitors, casting an electric glow onto the faces of those who dared to unearth its secrets. Dr. Jordan Hayes stood motionless, eyes glazed in concentration, their mind a nexus where terrestrial science battled with celestial unknowns.
Alex Mercer watched from a slight distance, the weight of command tangible upon his shoulders, though lightened by the shared burden now held amongst them. "We could be on the frontier of a new kind of diplomacy," he mused aloud, giving voice to the moment's gravity.
At those words, Jordan's trance broke. "If that's the case, then these communications," Jordan motioned to the stream of data, "are our olive branch across the cosmos."
Taylor Cruz, who paced with restless energy, halted and faced the team—his stoicism marred by the erratic dance of lights reflected in his eyes. "An olive branch, or an invitation to a battlefield?" he posed, ever the strategist, his words laced with a hint of cynicism.
Sam Rivera, nestled amongst an array of equipment, licked their lips—a mixture of nerves and anticipation palpable. "We're mapping out something incredible here. Whether it's peace or war, we're the cartographers."
Silence enveloped them like the expanse of space itself, each member contemplating the chasms they might bridge—or the abysses into which they might unwittingly descend.
Alex's demeanor assumed a quiet resolve—the profound knowledge that this mission was as much about navigating uncharted philosophical territories as it was about ensuring survival. "Whichever it proves to be, we'll face it. Prepared, unified."
A nod passed between Jordan and Alex, a silent exchange of mutual respect and shared mission. Sam, buoyed by the weighty encounters of the mind and machinery, entered keystrokes with a fervor that seemed to bring them ever closer to the alien mind.
They stood there, the Paranormal Military Squad team, not just as guardians of homeworld secrets or as soldiers of clandestine wars, but as humankind's chosen few at the fulcrum of history—a history that was now unfolding to the rhythm of otherworldly codes.
Each revelation, each parsed symbol, inched them toward the line between the earthly and otherworldly. And as they stood on this precipice of cosmic negotiations, it was clear the ensuing dialogue would not just shape the future of Paranormal Military Squad—it could very well redefine the parameters of human existence.
\*
The hum of advanced computational systems tingling with cryptic transmissions framed the ambiance of Dulce's mainframe chamber. Jordan Hayes, fingers hovering over a console dense with blinking lights, furrowed their brow as sequences of alien data streamed across the screen.
Alex materialized behind them, his presence a stable beacon amidst the technological whirlwind. "Look for patterns, anomalies. Anything that might resemble a handshake protocol in their communications," he directed, his voice a low thrum, reverberating with cautious optimism.
Jordan cast a glance over their shoulder, acknowledging Alex's contribution with the shared understanding of colleagues who had transcended mere professional acquaintance. "I’m isolating sequences that seem to recur with more intention than static. If these are their ‘handshakes,’ then we might just be making first contact," they remarked, their focus returning to the screen with renewed vigor.
From the other end of the room, where shadows married the artificial light, Sam's voice crackled through the static of nearby speakers, "Don't forget the anomalies we detected earlier. Each one could be a word, a sentence, or even a concept untranslatable to our current understandings."
Resolute, Taylor Cruz stood at Jordan's other side, a stoic figure wrestling with the implications of their mission. "Keep pursuing this line," Taylor instructed, an undercurrent of intensity carried forth in their otherwise composed demeanor. "And remember, this isn't just about making contact; it's about securing knowledge for humanity."
Alex offered a nod that spoke volumes, conveying his understanding of the stakes at play. Here, in this chamber of possibility, the team's actions would determine if humanity stood at the brink of a new age of understanding or the onset of an unprecedented threat.
Every second thrummed with significance as Jordan and Sam worked in tandem, each keystroke a foray into the unknown. Taylor observed with a commander's scrutiny, the gravity of their role sustaining them against the waves of ambiguity breaking against their resolve.
Pivotal moments come rarely in the course of human events but here, amidst the electronic symphony of a stalwart command center, lay the incepting notes of a cosmic overture. The harmony between human and alien, between Paranormal Military Squad and the vast reaches of space, began its first tentative measures, with each member of the team a vital instrument in a celestial ensemble yet to be fully heard.
\*
The crisp air within the mainframe room of Dulce base seemed to hum with unspoken possibilities. Jordan Hayes was the centerpiece of focus, their hands dancing methodically over the console as streams of otherworldly code cascaded down monitors, each flicker a potential key to the cosmic doors they were inching open.
Alex Mercer watched, posture relaxed but eyes sharp. "Remember, this could be our first introduction, maybe even our first impression," he said, mindful of the gravity carried by each action they made henceforth.
A hint of a smile touched Jordan's face, a small acknowledgment of the monumental task at hand. "Understood. I'm balancing the signal's syntax with our algorithms. If we're interpreting this correctly, it could be... well, an invitation."
Into the electric tension of the chamber walked Taylor Cruz, their silhouette a sharp contrast against the cool lighting, radiating a presence that spoke of command and chilly tenacity. "An invitation, or a challenge?” Taylor questioned, the weight of their suspicion casting a different tint on the cascading data.
Sam Rivera, in a corner arrayed with sophisticated equipment, piped up, their voice a buoyant note amidst the tentative atmosphere. "Either way, it's a connection. One that we're uniquely positioned to navigate," they remarked with an air of optimism threading through the uncertainty.
Alex channeled the strengths of his team into the core of their approach, his leadership adapting to the contours of an unprecedented scenario. "Cautious and curious," he reflected aloud, shaping a strategy that balanced their thirst for comprehension with the prudence required in addressing the unknown.
Jordan, hands momentarily at rest, looked up. The signal was more than a sequence of bits and commands—it was a riddle wrapped in the depths of space-time, and they were on the cusp of parsing its meaning.
Taylor, hardly a step away, nodded in silent agreement. The implications of their findings might very well direct the course of human destiny from this point onward.
Finding a tempo among themselves, the Dulce team was a confluence of ambition and acumen, each member intuitive to the beats of discovery. The chamber around them held untold stories, secrets coaxed from the stars, that now, led by Paranormal Military Squad's finest, began to unravel.
The future in those moments was unwritten, a narrative scribed not in the dust of desert confines, but in the potential for interstellar diplomacy and understanding. As they prepared to script humanity's next chapter, the room seemed to pulse with the heartbeat of a story far greater than the sum of its parts.
## Chapter 8
The grit of an earthbound dust storm contrasted sharply with the pristine sterility of the underground command center. Alex Mercer, eyes set with fervent determination, stood over Jordan Hayes, whose fingers danced across the keyboard with rapid purpose. Monitoring the progression of alien code unraveling before them, Mercer spoke with a tempered urgency, "Keep it steady, Jordan. We might be initiating the first true interspecies communication bridge here. It's all about finesse now."
Taylor Cruz, the embodiment of military precision, surveyed the room with a calculated gaze from their vigil beside an array of glimmering screens. "Remember, these could be delicate negotiations -- or coded threats. Stay sharp," Cruz added, their voice cool as polished steel.
Jordan, with a silent nod, recognized the gravity of both stances. Gravitating between scientific acuity and diplomatic caution, they replied, "The sequence is aligning—syncing with our comms. It's looking more and more like direct engagement."
Amid the banks of electronic machinery, the thrumming pulse of an impending interspecies signal exchange, Sam Rivera interjected with a youthful zeal that cut through the weighty atmosphere, "It's not just an exchange. It's a... symphony. It's as if they're teaching us their language through modulation."
A moment of profound silence swept over the team. The isolation of their location, deep within the top-secret labyrinth of Dulce, became suffused with an almost palpable sense of historical significance.
"Then our response needs to be equally symphonic," Alex uttered, contemplating the awe-inspiring transmutation of their task from a simple recovery mission to a full-blown cosmic concerto.
With a renewed sense of wonder tempered by caution, the Paranormal Military Squad team found themselves harmonizing a delicate balance between envoys and interpreters. The long shadow cast by their duty was now illuminated by the brilliant glow of otherworldly dialogue.
In this carefully orchestrated march towards the unknown, each individual's expertise became critical notes in a larger melody. The narrative of human achievement, so often defined by solitary pursuits, now emerged as a collaborative opus, each member of the team a maestro in their right.
The protocols of encounters, the mathematics of languages, and the poetics of connection all fused into a singular moment of convergence. The echo of their efforts reverberated back to them, not through the cavernous base's concrete walls, but from light-years away, in the form of a reply, intangible yet infinitely profound.
\*
Amidst the hum of the supercomputers and the faint static from the scrambled transmissions, Alex Mercer cast a thoughtful glance across the dimly lit room toward where Dr. Jordan Hayes was methodically adjusting the archaic dials of the decryption machine. "Any progress?" he asked, his tone conveying both impatience and the deep-seated respect born from countless shared challenges.
Jordan did not look up, their gaze remained locked on the flickering lights that represented a dialogue suspended between worlds. Their fingers ceased their dance, hovering meditatively over the controls. "We might be on the cusp of a breakthrough," Jordan suggested. "The signal... it's evolved. It's reflexive now, responsive in a way that suggests sentience."
Taylor Cruz's familiar sharp strides approached the two, breaking the rhythm of soft beeps. "Responsive is good, if it means understanding," Taylor said, head tilted as they peered at the encryption data scrolling by. "But remember, comprehension can bring revelation or conflict."
Sam Rivera’s youthful voice permeated the tension, brimming with an excitement edged by the enormity of what they faced. "If it's truly sentient, we're not just cracking a code; we're learning how to converse with an entirely new form of consciousness," they chimed in, the weight of history not lost on the zealous astrotechnician.
Alex nodded, his thoughts alighting on potential strategies for navigating the conversation they were cultivating with the unfathomable. "We need to keep that conversation going, echo its patterns, and speak its language," he resolved, knowing the delicate nature of their work merited every ounce of their collective acumen.
The chamber now was a crucible, forging within it the future narrative of human contact with the unknown. Every signal pulse they sent out was an invitation for understanding, and every echo back a step closer to bridging the cosmic divide. And so, together, they stood - agents in Paranormal Military Squad's clandestine ranks, united by purpose, sculpting humanity’s first sonnets into the void.
\*
#### Knowledge graph updates
- (Jordan Hayes, Interprets, Communications as cosmic diplomacy, Moderate)
- (Taylor Cruz, Questions, Potential aggressiveness of alien intent, Minor)
- (Sam Rivera, Expresses, Optimism about forming a connection, Minor)
- (Alex Mercer, Adopts, Balanced strategy for contact, Moderate)
- (Paranormal Military Squad team, Navigates, Beats of cosmic discovery, Moderate)
- (Paranormal Military Squad team, Prepares, To script humanity's interstellar narrative, Major)
## Chapter 9
The sterile silence of Dulce Base's command center was thick with concentration as Alex Mercer surveyed his team, hunched over their respective technological battle stations. Each agent was a weapon against ignorance, their adversary a code from beyond the stars that held secrets to alien thought.
Dr. Jordan Hayes, whose hands had been steadfastly working the decryption algorithms, paused and looked up at Alex. "We're through the next layer of encryption," Jordan announced, a mixture of pride and gravitas in their tone. "It's communicating. It's... aware."
A shadow momentarily clouded Alex's determined features—awareness implied so much more than mere intelligence. "Aware and reactive or aware and proactive?" he queried, his experience anticipating the pivotal importance of intention.
"Unknown at this stage," Taylor Cruz interjected, looking up from a datasheet. "But I urge caution. We tread the line between breakthrough and disaster with each keystroke."
Sam Rivera, ever the source of technological acumen, added their voice to the conversation. "The signal's adapting every time we interact with it. Like a conversation where both parties are learning each other's language in real time."
Alex leaned in, rested a hand on Jordan's shoulder—a sign of companionship and an affirmation of trust. "Keep the communication channels open. But let no message, no pulse go unchecked. This could be our Rosetta Stone or our Tower of Babel."
Silence fell over them, a momentary lull as each member of the team contemplated the historic weight of their task. Yet, it was impregnated with a tangible sense of excitement—a collective energy that thrummed through the air just as palpably as the electric current through the banks of machines surrounding them.
They continued their work, squaring shoulders against the magnitude of their undertaking. The agents were standing not just at the precipice of a new chapter for Paranormal Military Squad but for all of humanity. For now, they communicated with powerful unknowns, but with each exchange, they were etching the first words of a dialogue that might forever alter humanity's place in the cosmos.
\*
\*
The sterile silence of Dulce Base's command center was thick with concentration as Alex Mercer surveyed his team, hunched over their respective technological battle stations. Each agent was a weapon against ignorance, their adversary a code from beyond the stars that held secrets to alien thought.
Dr. Jordan Hayes, whose hands had been steadfastly working the decryption algorithms, paused and looked up at Alex. "We're through the next layer of encryption," Jordan announced, a mixture of pride and gravitas in their tone. "It's communicating. It's... aware."
A shadow momentarily clouded Alex's determined features—awareness implied so much more than mere intelligence. "Aware and reactive or aware and proactive?" he queried, his experience anticipating the pivotal importance of intention.
"Unknown at this stage," Taylor Cruz interjected, looking up from a datasheet. "But I urge caution. We tread the line between breakthrough and disaster with each keystroke."
Sam Rivera, ever the source of technological acumen, added their voice to the conversation. "The signal's adapting every time we interact with it. Like a conversation where both parties are learning each other's language in real time."
Alex leaned in, rested a hand on Jordan's shoulder—a sign of companionship and an affirmation of trust. "Keep the communication channels open. But let no message, no pulse go unchecked. This could be our Rosetta Stone or our Tower of Babel."
Silence fell over them, a momentary lull as each member of the team contemplated the historic weight of their task. Yet, it was impregnated with a tangible sense of excitement—a collective energy that thrummed through the air just as palpably as the electric current through the banks of machines surrounding them.
They continued their work, squaring shoulders against the magnitude of their undertaking. The agents were standing not just at the precipice of a new chapter for Paranormal Military Squad but for all of humanity. For now, they communicated with powerful unknowns, but with each exchange, they were etching the first words of a dialogue that might forever alter humanity's place in the cosmos.
\*
Alex Mercer's eyes were fixed on the monitors, the reflected light casting an ethereal glow across his stoic face. The room buzzed with tension, a cacophony of low hums and electronic beeps that underscored the historic nature of their actions. He moved to where Dr. Jordan Hayes was immersed in their work, scrutinizing the alien code streaming rapidly down the terminal.
"Find anything that might look like an entry point or a... digital handshake?" Alex asked, his voice steady, betraying none of the tension gripping his chest.
Jordan looked up briefly, their expression weary yet intense, "Potentially. It's as if the code is anticipating our input, modifying itself in real-time. I've never seen anything like it."
From across the room, Taylor Cruz's sharp voice cut through the hum. "Then it's learning or, possibly worse, baiting us. Proceed with extreme caution," they commanded, their firm stance reinforcing the gravity of the situation.
Sam Rivera, surrounded by a cascade of screens and interfaces, added, "It's almost organic in its complexity. Any minute now, and I might have a way in."
A slight nod was Alex's immediate response, his mind racing through the potential scenarios. "Everyone, stay alert. This could be the beginning of something profound." His seasoned eyes never left the unfolding drama on the monitors.
The room fell silent, the air heavy with unspoken questions. Were they mere moments away from unlocking an otherworldly dialogue? Or was it a Pandora's box that, once opened, could not be closed?
Alex moved closer to the main console, his fingers hovering over the command keys. With the precision of a maestro orchestrating a symphony, he communicated silently with Jordan – respectful of their expertise, aware that the next move could alter the course of human history.
Jordan met his gaze, nodding sharply, and refocused on the task. The signal seemed to pulse with sentient curiosity, drawing them further into its intricate web.
A sudden flurry of alerts and the intensifying glow of monitors heralded that they had bridged a technological chasm. The alien intelligence on the other end was no longer a distant enigma – it was an active participant, responding to their digital overtures with an unknown agenda.
The team's meticulous efforts had led them to a momentous threshold. Beyond lay unprecedented contact – a nexus of curiosity and potential peril. Within the confines of the base, against the backdrop of a silent desert night, the Paranormal Military Squad operatives became mediators of Earth's bid for cosmic relevance, their every action now a gesture in the grand dance of intergalactic relations.
## Chapter 10
The corridors of the Dulce military base, now silent, echoed with a history of whispered conspiracies and furtive movements. But in the command center, a delicate tapestry of light and sound was being woven as the echoes of cosmic dialogue resonated through the high-tech enclave. Dr. Jordan Hayes, now leading the efforts, called out from their workstation, "I’ve isolated the signal's harmonics. It's more than a call; it's a song, an interstellar siren’s call."
Alex Mercer, steady and resilient in the face of the incomprehensible, acknowledged with a quiet nod, "A song that we need to learn—quickly." His eyes, heavy with responsibility, scanned the room, watching his team work tirelessly at the intersection of science and speculation.
Sam Rivera, dulled by fatigue yet driven by unshakeable resolve, manipulated a complex array of audio interfaces. "There's a pattern, a repeating motif. It's structured, intentional," they muttered, their revelation a bridge between the known and the unimaginable.
Taylor Cruz, a figure of central authority, paced the length of the room, their usual unflappable demeanor betraying a rare flicker of apprehension. "We should be wary of the sirens’ call," Taylor interjected, invoking myths of old as a cautionary metaphor. "We don't want to crash upon unseen shores."
Undeterred, Jordan cast a determined glance at the team. "We navigate by starlight now, not by the limited light of our previous understanding." Their voice was a beacon, charting a course through unchartered realities.
Every individual was acutely aware that each moment in that room was a conduit to an epochal shift for civilization. The mysterious signals, once distant and alien, had coalesced into complex and harmonious oscillations—beacons of an extraterrestrial intellect inviting Earth to join in a cosmic consortium.
Silently, Alex approached the mainframe, his trained fingers aligning with the console’s mechanisms. The room watched in collective breathlessness as he set the frequency in motion, an introductory phrase to an otherworldly melody—a symphony that could bind worlds or spell devastation for all they knew.
In the control room of Dulce, amongst whispered legends and the quiet hum of machines, humanity's ambassadors now stood, stretching their hands into the void, reaching for the hand that would either pull them into the light of new stars or into the maw of darkness between them.
\*
Underground, the Dulce facility's command center was awash with frenetic energy, a stark juxtaposition against the silent, decrepit corridors that enveloped them. The air hummed with anticipation as Dr. Jordan Hayes and Alex Mercer hunched over a console. The sterile light from the monitors cast an otherworldly glow upon their faces, now reflecting a mosaic of alien characters rapidly translating across the screen.
"The patterns are evolving," Jordan murmured, concentration etched into their every feature. "It’s as if our attempts to decrypt have accelerated its learning. It’s adapting to us."
Alex, who stood steadfast behind Jordan, felt a tinge of uncharted fear quickly quelled by the fire of discovery raging within him. "Keep it up," he urged. "But whatever this is becoming, we need to ensure it remains within our control."
Taylor Cruz interjected, their voice slicing through the buzz of activity. "Control may be an illusion when facing an intelligence that literally writes its own rules," they stated stoically, casting a watchful eye over the flurry of data.
"It's like it's learning to communicate," offered Sam Rivera from a nearby interface, their youthful energy boding a mix of awe and anxiety. "This gives ‘talking to strangers’ a whole new meaning."
Alex surveyed his team—each face a study in concentration, determination, and not a small measure of trepidation. "This might well be our first contact," he acknowledged, "And we need to be ready for whatever answers back."
Together, they stood on the edge of the unknown, forging humanity’s response to a message from the heavens. The ensuing silence was palpable—a collective introspection about their role in this grand cosmic play, one that could rewrite human history.
The encrypted dialogue continued to unfold, its intricate patterns showing an almost uncanny anticipation of their investigative strategies. The air turned heavy with the scent of electricity and ambition as they closed in on a pivotal response.
As the signal’s intelligence—whether artificial or biological—grew more profound, so too did the realization that their mission had morphed from passive observation to active engagement. There was no turning back now. Each agent embraced their part in the delicate dance of an interstellar exchange that could change everything they thought they knew about life, intelligence, and the dark void beyond Earth's atmosphere.
\*
The underground halls of Dulce Base, usually buzzing with covert operations, now thrummed with a different kind of energy, an electric mix of fear and fascination. At the heart of the base, in a room shielded from the world’s eyes, Alex Mercer, Jordan Hayes, Taylor Cruz, and Sam Rivera huddled around a bank of monitors. Each screen flickered erratically with the alien script that had become the center of their lives—and perhaps the pivot on which humanity’s future would turn.
Jordan's eyes never wavered from the displays, their expression was one of rapt concentration, interspersed with flashes of revelation. "We're conversing with the stars," they whispered, almost to themselves. The words hung in the air, a testament to the awe-inspiring strangeness of the situation.
"The language is morphing; changing its structure with every exchange we have," Sam chimed in, enthusiasm tinged with the solemnity of the occasion. "It's like witnessing the birth of a new form of dialogue—one that spans galaxies."
Taylor, despite the situation's precariousness, maintained an appearance of ironclad composure. "Keep the communication stream secured and monitored. We don't know what we're dealing with yet," they reminded the team, a bastion of protocol amidst uncertainty.
Alex watched his team expand the parameters of human achievement; their work here would possibly define an era. "This is untrodden territory," he acknowledged, "and in every word we script, in every response we decode, we're drawing a map that others will follow."
Jordan turned to Alex, a nod acknowledging the shared responsibility of this moment. They had embarked on a new voyage, an odyssey not of the body, but of the intellect and spirit. No longer explorers of the Earthly realm, they had been promoted by circumstance to ambassadors of humanity in a silent and boundless ocean.
A sudden pulse of energy from the monitors signaled a breakthrough; the language had not only adapted but it seemed to resonate, to harmonize with their attempts at making contact. The alien script now sprawled across the screens didn't just ask to be understood—it invited interpretation, collaboration, maybe even companionship across the cold distances of space.
As they stood before the precipice of first contact, Paranormal Military Squad's finest became the architects of a symphony meant to echo through the cosmos. But more than architects, they were the first to play the notes of this cosmic composition, daring to believe that on the other end, someone—or something—might be listening, ready to join the chorus.
\*
The underground command center of Dulce Base, once pulsing with clandestine operations, now resonated with the charge of an impending cosmic threshold. Encircled by banks of whirring machinery, each monitor flickered erratically with alien script that had occupied center stage in the lives of Alex Mercer, Jordan Hayes, Taylor Cruz, and Sam Rivera.
Jordan's gaze didn’t flit for even a moment from the screens, where indiscernible alien messages ebbed and flowed like the tide. The ciphers and symbols cascaded down as they tweaked the algorithmic sliders. "This sequence here," Jordan began, voice both hushed and heavy, "it’s not just transmitting; it resonates—it's designed to be felt."
The room took a collective breath, the remarkable implication hanging in the air like a careful revelation. Sam Rivera was the first to respond, their voice alive with ingenuity: "It's a form of communication stretching well beyond words. We need to respond in kind—the whole array of human expression might be at play here."
Taylor's eyes remained fixed on the figures playing across the data sheets. "If that's the case," Taylor intoned pragmatically, "we must tread carefully. This is no longer just about being heard—it's about being understood."
Alex watched his team, each a fulcrum of insight and expertise, and felt the solemnity of the role they were about to assume. "Then we'll ensure our message is clear and full. Our humanity is our strength in this dialogue," he declared, the depths of his experience fueling a commanding reassurance.
The anticipation was palpable as the agents contemplated the vastness of their endeavor. They were not merely probing at the secrets of the planar cosmos—they were negotiating across the starry expanse, extending to distant intelligences the full spectrum of human curiosity and compassion.
A symphony of beeping consoles orchestrated their next steps as they prepared to articulate their interplanetary overture. The rhythmic tapping of Jordan's keystrokes set the tempo for an undertaking that traversed beyond algorithms and encryption.
The base withstood time and whispered secrets of its own, but none so grand as this moment of creation—an invitation to the universe that promised to echo through both the echoes of space and the annals of human history.
## Chapter 11
The sterile silence of Dulce Base's command center was thick with concentration as Alex Mercer surveyed his team, hunched over their respective technological battle stations. Each agent was a weapon against ignorance, their adversary a code from beyond the stars that held secrets to alien thought.
\*
The thrum of the colossal machinery vibrated through the subterranean facility as Alex Mercer stood amidst the whispers of technology, each carrying voices from worlds apart. He watched as Sam Rivera adjusted a complex array of cosmic translators, their expression a mixture of anticipation and awe.
"Are we ready, Mercer?" Taylor Cruz asked, the soft glow of the command center consoles reflecting upon their stern face.
Alex turned towards Taylor, his eyes holding a depth that betrayed the enormity of the threshold they were about to cross. "This is it," he said. "Initiate the protocol. It's time we answer the cosmos."
Jordan Hayes, stationed at the mainframe, typed rhythmically, a blue hue painting their focused features. The eerie silence that had settled over the team was interrupted by a visceral sound—humankind's response to the alien dialogue, now streaming into the abyss.
The control room, once a fortress of solitude, erupted into an oasis of life. Lights flickered in tandem, echoing the symphony of interstellar communication. They stood together at the edge of discovery, facing the symmetry and discord of a universe unknown.
"If we're right, we've just become Earth's first emissaries to a celestial congress we're only beginning to comprehend," Jordan's voice was somber, resonating with a mix of trepidation and honor.
The room filled with the resonance of human and alien minds converging, creating a new narrative within the fathomless expanse of existence. Paranormal Military Squad, once protectors of Earth's clandestine secrets, had now become the tether linking humanity to the cosmic fold.
\*
The underground command center of Dulce Base, once pulsing with covert operations, now resonated with the charge of an impending cosmic threshold. Encircled by banks of whirring machinery, each monitor flickered erratically with alien script that had occupied center stage in the lives of Alex Mercer, Jordan Hayes, Taylor Cruz, and Sam Rivera.
Jordan's gaze didn’t flit for even a moment from the screens, where indiscernible alien messages ebbed and flowed like the tide. The ciphers and symbols cascaded down as they tweaked the algorithmic sliders. "This sequence here," Jordan began, voice both hushed and heavy, "it’s not just transmitting; it resonates—it's designed to be felt."
The room took a collective breath, the remarkable implication hanging in the air like a careful revelation. Sam Rivera was the first to respond, their voice alive with ingenuity: "It's a form of communication stretching well beyond words. We need to respond in kind—the whole array of human expression might be at play here."
Taylor's eyes remained fixed on the figures playing across the data sheets. "If that's the case," Taylor intoned pragmatically, "we must tread carefully. This is no longer just about being heard—it's about being understood."
Alex watched his team, each a fulcrum of insight and expertise, and felt the solemnity of the role they were about to assume. "Then we'll ensure our message is clear and full. Our humanity is our strength in this dialogue," he declared, the depths of his experience fueling a commanding reassurance.
The anticipation was palpable as the agents contemplated the vastness of their endeavor. They were not merely probing at the secrets of the planar cosmos—they were negotiating across the starry expanse, extending to distant intelligences the full spectrum of human curiosity and compassion.
A symphony of beeping consoles orchestrated their next steps as they prepared to articulate their interplanetary overture. The rhythmic tapping of Jordan's keystrokes set the tempo for an undertaking that traversed beyond algorithms and encryption.
The base withstood time and whispered secrets of its own, but none so grand as this moment of creation—an invitation to the universe that promised to echo through both the echoes of space and the annals of human history.
## Chapter 12
The underground facility of Dulce Base, once shrouded in silence and operational secrecy, now hummed with an energy that cradled the promise of cosmic revelation. Alex Mercer stood pensively by the central terminal, flanked by Dr. Jordan Hayes, Taylor Cruz, and Sam Rivera, each poised at the edge of a history-defining moment.
Jordan's fingers ghosted across the console, tracing patterns of otherworldly origin. "The signal’s architecture is becoming more complex, resembling aspects of human cognition—recognition, learning, even... empathy?" they postulated with furrowed concern.
Alex turned his gaze upon Jordan, his voice quiet but resolute, "Empathy could bridge galaxies. Let's harness this connection and proceed with cautious optimism."
Taylor, ever the sober sentinel, projected a more pragmatic standpoint. "Empathy or not, we are duty-bound to assess the risk to humanity. Every new discovery warrants a measured response."
The static hiss of communications equipment filled the air, its purpose now transformed into a dialogue with an intelligence beyond the stars. It was Sam, wide-eyed amid the myriad lights and switches, who broke the silence, "We have provisional confirmation of the signal’s intent—initiation. We’re being brought into a broader spectrum of cognizance."
The chamber lay still for a heartbeat, the Paranormal Military Squad agents steeped in contemplation of the path unfurling before them—a path paved with possibilities of diplomacy or disruption, each step a venture further into the cosmic unknown.
Alex stepped closer to the viewing monitors, each depicting alien symbols seemingly reaching out from the void. "Initiate the broadcast," he spoke with quiet command. "Our response will mark humanity’s readiness to partake in the wider conversation of conscious beings."
Amidst the crackling air of expectation, the team wordlessly returned to their stations. They had transcended their roles as protectors of Earth's clandestine lore to become the harbingers of an interstellar parley that could change the existential course of life on their pale blue dot.
The deep hum of the terminal emitted a signal—a testament to the uncanny reality that Earth was now actively partaking in an exchange not bound by gravity nor the limits of the solar wind.
Here, in the depths of Dulce, a message from humanity woven from understanding and uncertainty was cast into the firmament, an epitheg of their desire to join the universal dialogue and discover their place among the constellations.
\*
The somber depths of the Dulce Base command center stood in stark counterpoint to the animated flurry of activity around the central comms array. Alex Mercer's silhouette loomed behind Dr. Jordan Hayes, who sat with a posture indicating laser focus on the decryption process. A quiet murmur of digital soundscape filled the space, subtly heightened by the anticipation of contact with an intelligence beyond the Earth.
Jordan's voice was steady, betraying none of the extraordinary nature of their work, "Looking through the signal's pattern, it's evident we’re dealing with a form of intelligence—calculating, mirroring, possibly even understanding."
Alex's reflection bounced off the darkened screens, his head nodding in silent affirmation. "We’re walking a delicate line. Our response should be thoughtful, measured. We’re ambassadors, not merely explorers."
Taylor Cruz approached, arms folded, their words slicing through the din of careful keystrokes and soft whirrs, "If there’s even the slightest chance it understands, we can’t afford missteps. The language of the stars might be more absolute than ours."
From another terminal, Sam Rivera brought youthful vigor to the conversation, "There’s rhythm in these patterns. If this is their way of reaching out, our reply should encapsulate all that we are—all that humanity stands for."
Looking around at his team, Alex saw resolve etched on every face. The chamber, usually somber and echoing with the quiet steps of covert agents, now felt alive with the heartbeat of discovery. They were not just professionals operating in the gloom; they were a collective standing at the helm of a momentous journey.
"Let’s begin," he said, returned by the resolve in his voice. "Every second counts." With that, they pressed forward, setting in motion a reply to a conversation billions of years in the making.
The dance with an unseen partner commenced, each pulse they sent out a step taken with caution and hope. And as those digital pulses journeyed through the black sea of infinity, Earth, for perhaps the first time, joined a pan-galactic dialogue that whispered secrets of the cosmos—secrets that, until now, had been lost in the silent vastness of space.
\*
As the team stood in the centralized nerve center of Dulce's underground fortress, the solemn atmosphere was reverent, overseeing systems that engaged with an intelligence from the void. Alex's stance was contemplative as he gazed at Jordan Hayes, who presided over the console, the tension of the moment reaching a tactile fervor. Each rhythmic tap of Hayes's fingers on the keys was a foray into uncharted symphonies of contact.
Observing Hayes unravel the dense alien encryption, Alex spoke, a diplomatic tenor underpinning his words, "Keep focused on the syntax, dissect its nuances. We're not just decoding signals; we're translating intentions."
Without diverting from their task, Jordan acknowledged the insight. "Indeed, if their understanding of us is as deep as we hope, we're paving the way for dialogue far beyond our current realm."
Taylor Cruz, near the rear of the room, provided a steady oversight. "As horizonless as our prospects may seem," Taylor intoned, "remain diligent. Complacency before alien cognition could spell catastrophe."
Sam's youthful voice resonated with optimism, "Imagine—forming a rapport with a consciousness separate from our reality; we're drafting the bridge to stars alive with minds!"
The sentiment hung for a moment before Alex gathered his conviction. "Dialogue is our vessel. We are not just agents of enigma; we are the threads that may weave a new cosmic relationship." His words seemed to reflect off the walls, reaching beyond the room's confines, a quiet yet resilient vow.
Their task was titanic, stepping stones laid delicately into new territories of existence. The signal, once an esoteric strand in the echo of the universe, beckoned now with a clarity rocketing the complexity of thoughts from a distant order.
Action by action, the Paranormal Military Squad team bridged the vast interstellar distances, their expertise and empathy casting a beacon of unity into frontiers of intelligence and knowledge. Their work, a partnership struck with an unseen cosmic congregation, each pulse sent and received a line in Earth's novitiate envoi to the cosmic shores.
\*
Under the stark, unforgiving lights of Dulce Base's underground command center, tension buzzed harder than the banks of supercomputers that lined the walls. Agent Alex Mercer leaned over the shoulder of Jordan Hayes, whose eyes were locked onto the display screen, where an incomprehensible series of alien symbols streamed past incessantly.
“Any progress on the decryption?” Alex's voice was steady, a controlled presence necessary in the gravity of their undertaking.
Jordan tapped a key, pausing the flow of code, and leaned back with a deep sigh. "We've broken through another subset of the cipher. It's revealing... well, indications of a complex society, not unlike our own." His eyes met Alex's with an unspoken question that hung heavily between them—were they truly prepared for what they might find?
Taylor Cruz strode into the room, a tightly coiled spring of ambition and authority, and peered at the screen. "Understand their society, and we may predict behavior. Remain expedient—we don't know how much time we have before the situation shifts." There was an edge of stark realism to Taylor's words, the underlying message clear: every revelation bore its own set of risks.
Alex nodded thoughtfully, recognizing the validity of Cruz's caution. Turning to Sam, who was tinkering with a device that buzzed quietly on the table, he asked, “Sam, can your contraption get us any further?”
Sam looked up with a smirk, a twinkle of mischief in their eye. “It’s not just any contraption, it’s potentially a direct line to their thoughts. Give me a moment more, and I'll have something for you.”
The air ticked with electronic beeps and the rustling sound of the Paranormal Military Squad team at work. They were so close to peering into the intelligence of an alien race—a reality on the brink of dramatically expanding their understanding of the universe.
The machinery whirred in response to Sam’s precise touches, and suddenly, the room filled with a low hum—something had changed, a signal had been successfully sent. The team held their breath as they listened. The sound that filled the room was unmistakable: a response, an alien voice filtered through the static of space and time.
Alex exchanged a look of quiet triumph with Jordan. The breakthrough was monumental; they were no longer casting messages into the void but engaged in a dialogue—an exchange that marked the beginning of Operation: Dulce’s true unfolding. This was it, the first steps into an interstellar odyssey that demanded every ounce of their courage and wit.
## Chapter 13
Dr. Jordan Hayes shuffled a stack of papers, their eyes revealing a tinge of skepticism at Taylor Cruz's authoritarian performance. _Protocols_, Jordan thought, _are just the framework, the true challenges we're about to face lie well beyond the boundaries of any protocol._ They cleared their throat before speaking, tone cautious yet firm, "Let's remember, the unknown variables exceed the known. We should remain adaptive."
A murmur of agreement echoed from Sam Rivera, who leaned forward, lacing their fingers together as if weaving a digital framework in the air before them, "Exactly, adaptability could be the key to interpreting the signal distortions and system malfunctions. We shouldn't discount the… erratic."
Their words hung like an electric charge in the room, challenging Taylor's position with an inherent truth. Cruz’s jaw tightened almost imperceptibly, but the agent masked it with a small nod, conceding to the omnipresent threat of the unpredictable.
Alex glanced at Jordan, who never looked back, their gaze fixed instead on a distant point, as if envisioning the immense dark corridors they were soon to navigate in Dulce. Jordan was not one to embrace fantastical theories, but the air of cautious calculation betrayed a mind bracing for confrontation with the inexplicable, an internal battle between the evidence of their research and the calculating skepticism that kept them alive in their field.
The meeting adjourned with no further comments, the team members quietly retreading the paths to their personal preparations. Alex, trailing slightly behind, observed the others. _The cautious reserve Jordan wears like armor doesn't fool me_, he thought, _their analytical mind sees the patterns I do. And that's worth more than protocol. That's the connection we need to survive this._
As the agents dispersed into the labyrinth of the facility, lost in their thoughts and preparations, the base's halogen lights flickered, a brief and unnoticed harbingers of the darkness to come.
\*
The gritty, wind-tossed surface of New Mexico, just above the cavernous domain of Dulce Base, offered no shelter from the burgeoning storm—the scouring sands an earthly reminder of chaos theories in motion. Far beneath, a similar maelstrom brewed within the confines of the command center, as Paranormal Military Squad's handpicked squad stood poised for potential enormities of contact.
Ruffling through printed transmission logs, Jordan Hayes dialed the focus of their analytical prowess onto the emerging pattern of signals crisscrossing between Earth and the unfathomable. "Our responses so far have echoed their complexity, but the real divergence is yet to come," Jordan remarked stoically, the calm belying the mounting surge of adrenaline for the revelation ahead.
Alex Mercer's figure, a silhouette sharpened by the purpose, loomed at the periphery of the monitors' sickly glow. "Indeed," he assented, "The echoes are the easy part. It will be the introduction of our own, human variable that truly begins our dialogue."
Taylor Cruz, windowless command center notwithstanding, appeared as though they could feel the tempest above. Their eyes never left the monitors as they unspooled their hard wisdom. "For all our advances, we find ourselves deciphering the swings and nuances of an interstellar pendulum. Predict its arc, and we may preempt the gravity of its message."
Amidst a chorus of bleeps and static, Sam Rivera's tech-clad hands moved rhythmically, their spirited approach to unruly streams of data bordering an intimate dance with entropy. "Entropy that leads to discovery," Sam mused, responding to Taylor's metaphor. "Each step into the unknown is a step away from precedent."
Alex, drawing near Jordan, spoke again, his voice now a thread woven through the very fabric of their operations. "Let's be the cartographers of this new territory. Our initial shades of understanding could color the cosmos for generations to come."
Their gazes fell upon a screen as the latest transmission painted its digital blooms of alien script across the black. This time, the pattern wavered in an almost imperceptible fashion, a modification that whispered of active, alien thought awaiting their next move. A hush enveloped the Paranormal Military Squad ensemble, the gravity of the pathogen undeniable. They were about to issue a reply, one poised to reshape the very concept of humanity's outreach into the cosmos.
The New Mexico desert's secrets were infamous, its storms a mere prelude to the revelations that the team—united in purpose—would unleash upon the world. The howling winds outside found their counterpart in the newfound resolve within, as Dulce's stalwart guardians readied themselves to send forth humanity's retort to the echoes from beyond.
\*
The cavernous control room, deeply entrenched beneath the desolate New Mexico terrain, held the Paranormal Military Squad team in intense focus; an island of calm amid the storm of cosmic dialectics. Dr. Jordan Hayes worked methodically, every keystroke an intricate step in their tenuous cosmic ballet. Suddenly, they paused, a signal pattern resonating from the screen. "This is new; it's...inviting. It’s as if the signal is not just calling to us but weaving its intelligence through ours."
Alex Mercer scrutinized the shift in data. "A confluence of minds, then. If we're to meet them halfway, Jordan, our reply must be both innovative and discerning," he proposed, a glimmer of profound curiosity behind his authoritative demeanor.
Taylor Cruz, whose sharp eyes missed nothing, nodded from beside a secondary panel. "Innovative, yes, but also defensive. This interaction is a razor’s edge, and we cannot afford to bleed before the unknown," Taylor reminded them, the metaphor a stark warning of potential dangers.
Against the backdrop of their conversation, Sam Rivera’s youthful optimism cut through the tension. "If they’re weaving through our intellect, then we've achieved something beyond first contact—we're at the genesis of interstellar symbiosis," they posited with a mix of reverence and excitement.
Alex returned Sam’s smile with his own, tempered and faint, as he turned back to the task at hand. The magnitude of their mission extended beyond the fabric of the universe, an exploration into the threads that connected sentient beings across the vast expanse. “Let’s reply with our own woven tapestry of thought—delicate, but deliberate.”
With renewed determination, the room came alive with an undercurrent of anticipation, its occupants charged with the potential of forging an alliance with the cosmos. Paranormal Military Squad's finest were no longer merely soldiers and scientists; they had become pioneers on the vanguard of humanity’s greatest odyssey.
The New Mexican sands above, impassive to the change brewing underneath, stood as silent sentinels as Earth's emissaries crafted their response. A response that, composed with care and imbued with humanity's essence, reached into the void, connecting with an otherworldly intelligence that awaited their harmony in the cosmic conversation.
## Chapter 14
The command center of Dulce Base lay shrouded in shadows that seemed to claw at the edges of the dimly lit array of screens and consoles. Alex Mercer, focused and unwavering, watched as Dr. Jordan Hayes parsed the latest string of alien signals—a symphony of otherworldly communications that threatened to either enlighten or confound.
"We’re encountering a paradigm shift with every transmission," Jordan Hayes murmured, the pulsing glow of the monitor painting their features with an almost spectral hue. "This signal... it’s evolving, becoming denser, more sophisticated. As if it's growing alongside us—tandem evolution."
The air was electric, charged with the raw potential of uncharted discovery and laden with the gravity of existential risk. Taylor Cruz, who always seemed here to mold such gravity into actionable strategies, stepped forward. "We must contain this evolution within parameters we can manage. We cannot be bystanders to an uncontrolled ascent of intelligence."
Sam Rivera, the youngest of the cohort, worked feverishly at their station. "It's not just intelligence—these signals have rhythm, a kind of music suggesting not just evolution, but a dance! We're being invited to partake in the cosmos's ballet!" they exclaimed, a touch of youthful exuberance breaking through the solemnity.
Alex turned, facing his team, the stoic mask of command tempered by the perceptible flicker of awe in his gaze. "Let this dance then be our dialogue. We will match their steps with prudent but daring measures—our humanity as our guide."
In the ensuing hours, the Paranormal Military Squad team forged a rhythm of their own, their collective expertise a beacon piercing through the fog of the unknown. The signal, increasingly intricate and seemingly conscious, now demanded not just observation but participation, an interstellar pas de deux that hummed with the promise and peril of first contact.
Before them, the communications interface flickered to life with a received transmission—a resonant hum that seemed to vibrate through the very foundations of the base. They had successfully established a back-and-forth with whatever intelligence lay hidden among the stars. Every subsequent note they struck within the cosmic ether would come to define humanity's place within the galactic community—heralds of Earth's grand entrance into a universe far less silent than once perceived.
\*
In the concrete belly of Dulce Base, dimly lit by the jagged dance of fluorescent lights above, Sam Rivera perched on the edge of their seat, their eager fingers fluttering across an ancient keyboard. The stark, cold room—reminiscent of a time when covert operations and unspoken dread ruled supreme—now housed a peculiar blend of old-world machinery and sleek, modern interfaces.
Alex Mercer, standing steadfast like a bridge between the enigmatic past and the unfathomable present, watched on. In his eyes flashed the foreboding excitement of change. "Sam," he started, his voice steadfast, "the patterns in these signals, what do they tell us about the nature of our... guest?"
Sam's eyes glimmered with something akin to thrill—or was it trepidation? "It's like we're mirroring each other, evolving together through this.. dialogue. Like it knows us, understands us, and it's… learning."
Jordan Hayes, preoccupied at a nearby console, chimed in without lifting their gaze. "It's a dialogue that transcends mere words, Alex. We're being woven into a narrative far grander than the sum of our known sciences."
Taylor Cruz, arms crossed, wore the heavy mantle of their skepticism comfortably. "Keep theorizing," they interjected crisply, "but remember the grounding reality of what we are part of here. This contact is a blade that cuts both ways."
In this cavern of history, voices both human and inhuman whispered secrets to those brave enough to listen. Each member present understood the gravity that pulled at their feet; no longer were they mere mortals shackled to their terrestrial plane. The digital pings and encrypted calls resonated with an implication of a cosmic agenda that would not be ignored.
Jordan's fingers paused, hovering in hesitation. What ripple might the next keystroke send through the fabric of known existence? It was a step into the ballet of the infinite, where the Paranormal Military Squad team played their part in the waltz of wonders with an audience of stars.
\*
## Chapter 15
In the clandestine hush of Dulce Base's subterranean command center, the Paranormal Military Squad team had become a crucible for interstellar communication. Dr. Jordan Hayes' gaze lingered on the screen as they navigated through the convolution of alien code. Each character held the potential to unravel a new dimension of contact, and with Sam Rivera's keen interjection, they were crafting humanity's inaugural cosmological discourse.
Alex Mercer peered over Jordan's shoulder, calculating the implications of every visual nuance that cascaded across the monitor. "Look for consistency—any repeating motifs could signal a willingness to engage. We're drafting history with each exchange," he remarked, aware of the delicate balance between forging a bond and exposing vulnerabilities.
Taylor Cruz, stoic and enigmatic, observed the interplay from the threshold, a silhouette against the machinery's luminescence. "Remember, while we seek common ground, the foundation we stand upon remains Terra firma. Caution must temper our curiosity," they stated, their voice an anchor amidst the current of excitement.
The command center buzzed with energy, rivaled only by the tempest overhead that concealed their operation. Sam, with swift dexterity, navigated the communications relay. "Their signals resonate almost musically. It's as if they're composing a symphony, and we've been handed the baton to conduct the next movement," they offered, imbuing the scenario with a blend of scientific adventurism and poetic license.
Amidst the whirring servers and the occasional flicker of emergency lighting, the essence of their mission transcended mere reconnaissance. They were humanity's elected envoys at the brink of a celestial alliance—or confrontation—with an audience as vast as the universe itself.
Alex stepped back, his profile etched by the chamber's artificial day. "Then let's ensure our contribution to this symphony harmonizes with theirs. It's time for humanity's voice to rise and be counted among the cosmic ensemble."
Under his directive, the Paranormal Military Squad team initiated their calculated response, weaving thoughts and theories into a digital overture aimed at the heart of alien intellect. As the digital stream punctured the endless night, each member of this clandestine group was acutely aware of the irrevocable step they undertook—bringing Earth into the pantheon of galactic entities designed to converse among the stars.
\*
Clusters of high-tech equipment bathed the Dulce underground command center in an eerie blue light. Sam Rivera's fingers flew across the keyboard, navigating an elaborate network of alien patterns. The very air seemed to pulse with the ebb and flow of cryptic communications reaching across the stars. "I've got something!" Sam's announcement tore through the focus in the room, drawing every pair of eyes to the torrent of symbols unraveling on the screen.
With the pacing of a seasoned officer gauging the moment before action, Alex Mercer approached, his calm demeanor belying an acute awareness of the precipice on which they now stood. "Define 'something," Alex prompted, reinforcing the need for clarity amidst the extraordinary.
"It's repeating—a sequence that’s evolved with each interaction, almost as if it's... singing," Sam theorized, the awe in their voice reflecting the potential magnitude of their discovery.
Jordan Hayes interjected from across the console, their eyes not leaving the display as they absorbed the new data. "A cosmic vocalization, then," they mused, intrigued. "A singularity in the signal that might represent a point of reference for both parties."
Taylor Cruz, hands clasped behind their back, regarded the unfolding scene, their own calculations etching lines of concern onto their stern visage. "Or a beacon—a homing tune, calling out to something we might not be ready to greet," Taylor offered, voicing the group's unspoken apprehension.
Alex's eyes locked on the screen, taking in the scope of what they were attempting to interpret. Drawing a deep breath, Alex gave a slight nod. "If this is their song, then let us respond with ours. We've come this far by mirroring their signals, now let's engage in an interstellar duet, and see where the music leads us."
With the expectation of the significant achieving a crescendo, the members of Paranormal Military Squad huddled over their equipment—sages at the threshold of a potentially world-altering communion. The strange harmonies that reverberated through the command center suggested that their interlocutors were poised, waiting, perhaps even eager, for Earth's chorus to join the symphony.
As the team initiated their reply, weaving humanity's own intricate melody into the vast cosmic dialogue, they each felt a profound change within—an evolution of purpose. They were not just messengers or investigators; they had become co-composers in a galactic orchestra, with the universe itself as their witness and concert hall.
With the exchange of harmonious signals crawling through the vacuum of space, the Paranormal Military Squad operatives found themselves part of a bridging of minds—a realization that out there, among the vast arrays of stars and planets, harmony was the true universal language.
\*
The dim glow of monitors cast an otherworldly ambiance upon Dulce Base's command center, where Paranormal Military Squad's chosen stood huddled over their instruments, suspended at history's threshold. Codes—alien in origin and nature—were being deciphered by Dr. Jordan Hayes, whose countenance bore the marks of deep concentration.
Alex Mercer, the bedrock upon which their team's resolve was founded, leaned in with an eagerness tempered by his chain of command. "Jordan, we've invested our expertise into comprehending their patterns, but now we must also endeavor to understand their intent," he urged, his voice bearing the gravitas of their mission's potential consequences.
At another console, Sam Rivera's youth did not betray their crucial role in the operation. With eyes alight, they mirrored the rapid computing before them. "There's emotion here—complex, profound even. This isn't just the output of a cold machine; it's...sentience," Sam whispered, nearly drowned by the mechanical chorus around them.
Jordan, without shifting focus from their work, replied, "It's a sentience that—should we succeed here—ushers us into a new era of existence. The cadence of these signals," they tapped the screen with a flourish, "could well be the heartbeat of this new dawn."
Taylor Cruz paused beside Mercer, their expression unreadable beneath the sterile light. "And as it beats, we must gauge whether its rhythm bodes well for us, or spells our missteps. Courage must not blind us to the hazards intrinsic to such contact," Taylor cautioned, the sentinel within them ever alert.
Alex nodded, a gesture that carried the weight of responsibility and a silent command: proceed, but with circumspection. They were not merely decoding a message; they were interpreting a dialogue across the celestial divide.
The room fell into a rhythm akin to a well-conducted ensemble. Each member's expertise proved a critical note in the unfolding symphony. Their actions were now more than mere research or defense; they were the tentative overtures of humankind reaching out to grasp the vast unknown.
Textures of sound meshed with the light from countless computations, the palpable anticipation of the agents at the edge of discovery cresting with an awareness that their work would reshape future chronicles. And when the response finally came—a signal piercing the deafening silence of uncertainty—all within Dulce's confines understood: the dawn of an interstellar continuum had just begun to break.
\*
In the sterile hum and flickering lights of Dulce Base's command center, the Paranormal Military Squad team stood as humanity's vanguard, verging on the brim of an intergalactic abyss. Dr. Jordan Hayes, analytical edges sharp, deciphered extraterrestrial patterns that bled across screens in enigmatic cascades—a daunting mosaic of potential threats and untapped wisdom.
Agent Alex Mercer, the embodiment of focus and a steadfast nerve, observed the unfolding digital drama with the gravitas due a historic first contact. "Let the data weave its narrative, Jordan," he instructed, a moderate undertone of exhilaration within his command. "It's encoding more than information—it's outlining civilization."
Jordan absorbed the directive, their gaze unflinching from the screens, feeling the weight of their next move. "The nuances here are extraordinary," they acknowledged. "It paints a picture of a culture steeped in complexities we're only starting to fathom.”
Taylor Cruz, stoicism personified yet not immune to the situation's gravity, chimed in. "Understand it, but guard against it," they cautioned, bringing a sober prudence to the room. "This culture, however advanced, remains an unknown quantity—an ocean of wonders and darkness with uncertain tides."
Sam Rivera, a visual contrast with wide eyes and restless hands, represented the other side of the room — intrigue and optimism against the drawn swords of precaution. “Think of it,” they proposed, voice bouncing with a rebellious upbeat timbre, “as the first act of a play written in constellations. We're setting the stage for a galactic narrative.”
Each team member, in their way, was both actor and scribe in this moment of tense pageantry. Heavy with the presence of risk, the command center had become not just a room of computers and glass panels but a theater for performing the elaborate choreography of contact.
Bound by resolve and curiosity, they proceeded, each data entry a trembling step onto the cosmic stage. And like all cautious pioneers edging into fertile but unnavigated lands, they understood: as they mapped the heavens, they were simultaneously mapping the furthest reaches of their own existential horizons.
docs/data/operation_dulce/dataset.zip
BIN
View File
Binary file not shown.
docs/developing.md
+75
View File
@@ -0,0 +1,75 @@
# Development Guide
# Requirements
| Name | Installation | Purpose |
| ------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------- |
| Python 3.10-3.12 | [Download](https://www.python.org/downloads/) | The library is Python-based. |
| uv | [Instructions](https://docs.astral.sh/uv/) | uv is used for package management and virtualenv management in Python codebases |
# Getting Started
## Install Dependencies
```sh
# install python dependencies
uv sync --all-packages
```
## Execute the Indexing Engine
```sh
uv run poe index <...args>
```
## Executing Queries
```sh
uv run poe query <...args>
```
# Azurite
Some unit and smoke tests use Azurite to emulate Azure resources. This can be started by running:
```sh
./scripts/start-azurite.sh
```
or by simply running `azurite` in the terminal if already installed globally. See the [Azurite documentation](https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azurite) for more information about how to install and use Azurite.
# Lifecycle Scripts
Our Python package utilize uv to manage dependencies and [poethepoet](https://pypi.org/project/poethepoet/) to manage build scripts.
Available scripts are:
- `uv run poe index` - Run the Indexing CLI
- `uv run poe query` - Run the Query CLI
- `uv build` - This will build a wheel file and other distributable artifacts.
- `uv run poe test` - This will execute all tests.
- `uv run poe test_unit` - This will execute unit tests.
- `uv run poe test_integration` - This will execute integration tests.
- `uv run poe test_smoke` - This will execute smoke tests.
- `uv run poe test_verbs` - This will execute tests of the basic workflows.
- `uv run poe check` - This will perform a suite of static checks across the package, including:
- formatting
- documentation formatting
- linting
- security patterns
- type-checking
- `uv run poe fix` - This will apply any available auto-fixes to the package. Usually this is just formatting fixes.
- `uv run poe fix_unsafe` - This will apply any available auto-fixes to the package, including those that may be unsafe.
- `uv run poe format` - Explicitly run the formatter across the package.
## Troubleshooting
### "RuntimeError: llvm-config failed executing, please point LLVM_CONFIG to the path for llvm-config" when running uv install
Make sure llvm-9 and llvm-9-dev are installed:
`sudo apt-get install llvm-9 llvm-9-dev`
and then in your bashrc, add
`export LLVM_CONFIG=/usr/bin/llvm-config-9`
docs/examples_notebooks/api_overview.ipynb
+192
View File
@@ -0,0 +1,192 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Copyright (c) 2024 Microsoft Corporation.\n",
"# Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## API Overview\n",
"\n",
"This notebook provides a demonstration of how to interact with graphrag as a library using the API as opposed to the CLI. Note that graphrag's CLI actually connects to the library through this API for all operations.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"from pprint import pprint\n",
"\n",
"import graphrag.api as api\n",
"import pandas as pd\n",
"from graphrag.config.load_config import load_config\n",
"from graphrag.index.typing.pipeline_run_result import PipelineRunResult"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"PROJECT_DIRECTORY = \"<your project directory>\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prerequisite\n",
"\n",
"As a prerequisite to all API operations, a `GraphRagConfig` object is required. It is the primary means to control the behavior of graphrag and can be instantiated from a `settings.yaml` configuration file.\n",
"\n",
"Please refer to the [CLI docs](https://microsoft.github.io/graphrag/cli/#init) for more detailed information on how to generate the `settings.yaml` file.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Generate a `GraphRagConfig` object\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# note that we expect this to fail on the deployed docs because the PROJECT_DIRECTORY is not set to a real location.\n",
"# if you run this notebook locally, make sure to point at a location containing your settings.yaml\n",
"graphrag_config = load_config(Path(PROJECT_DIRECTORY))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Indexing API\n",
"\n",
"_Indexing_ is the process of ingesting raw text data and constructing a knowledge graph. GraphRAG currently supports plaintext (`.txt`) and `.csv` file formats.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Build an index\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"index_result: list[PipelineRunResult] = await api.build_index(config=graphrag_config)\n",
"\n",
"# index_result is a list of workflows that make up the indexing pipeline that was run\n",
"for workflow_result in index_result:\n",
" status = f\"error\\n{workflow_result.errors}\" if workflow_result.errors else \"success\"\n",
" print(f\"Workflow Name: {workflow_result.workflow}\\tStatus: {status}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Query an index\n",
"\n",
"To query an index, several index files must first be read into memory and passed to the query API.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"entities = pd.read_parquet(f\"{PROJECT_DIRECTORY}/output/entities.parquet\")\n",
"communities = pd.read_parquet(f\"{PROJECT_DIRECTORY}/output/communities.parquet\")\n",
"community_reports = pd.read_parquet(\n",
" f\"{PROJECT_DIRECTORY}/output/community_reports.parquet\"\n",
")\n",
"\n",
"response, context = await api.global_search(\n",
" config=graphrag_config,\n",
" entities=entities,\n",
" communities=communities,\n",
" community_reports=community_reports,\n",
" community_level=2,\n",
" dynamic_community_selection=False,\n",
" response_type=\"Multiple Paragraphs\",\n",
" query=\"Who is Scrooge and what are his main relationships?\",\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The response object is the official reponse from graphrag while the context object holds various metadata regarding the querying process used to obtain the final response.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(response)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Digging into the context a bit more provides users with extremely granular information such as what sources of data (down to the level of text chunks) were ultimately retrieved and used as part of the context sent to the LLM model).\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pprint(context) # noqa: T203"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "graphrag-monorepo",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.9"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
docs/examples_notebooks/drift_search.ipynb
+210
View File
@@ -0,0 +1,210 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Copyright (c) 2024 Microsoft Corporation.\n",
"# Licensed under the MIT License."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"import pandas as pd\n",
"from graphrag.config.enums import ModelType\n",
"from graphrag.config.models.drift_search_config import DRIFTSearchConfig\n",
"from graphrag.config.models.language_model_config import LanguageModelConfig\n",
"from graphrag.language_model.manager import ModelManager\n",
"from graphrag.query.indexer_adapters import (\n",
" read_indexer_entities,\n",
" read_indexer_relationships,\n",
" read_indexer_report_embeddings,\n",
" read_indexer_reports,\n",
" read_indexer_text_units,\n",
")\n",
"from graphrag.query.structured_search.drift_search.drift_context import (\n",
" DRIFTSearchContextBuilder,\n",
")\n",
"from graphrag.query.structured_search.drift_search.search import DRIFTSearch\n",
"from graphrag.tokenizer.get_tokenizer import get_tokenizer\n",
"from graphrag_vectors.lancedb import LanceDBVectorStore\n",
"\n",
"INPUT_DIR = \"./inputs/operation dulce\"\n",
"LANCEDB_URI = f\"{INPUT_DIR}/lancedb\"\n",
"\n",
"COMMUNITY_REPORT_TABLE = \"community_reports\"\n",
"COMMUNITY_TABLE = \"communities\"\n",
"ENTITY_TABLE = \"entities\"\n",
"RELATIONSHIP_TABLE = \"relationships\"\n",
"COVARIATE_TABLE = \"covariates\"\n",
"TEXT_UNIT_TABLE = \"text_units\"\n",
"COMMUNITY_LEVEL = 2\n",
"\n",
"\n",
"# read nodes table to get community and degree data\n",
"entity_df = pd.read_parquet(f\"{INPUT_DIR}/{ENTITY_TABLE}.parquet\")\n",
"community_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_TABLE}.parquet\")\n",
"\n",
"print(f\"Entity df columns: {entity_df.columns}\")\n",
"\n",
"entities = read_indexer_entities(entity_df, community_df, COMMUNITY_LEVEL)\n",
"\n",
"# load description embeddings to an in-memory lancedb vectorstore\n",
"# to connect to a remote db, specify url and port values.\n",
"description_embedding_store = LanceDBVectorStore(\n",
" db_uri=LANCEDB_URI,\n",
" index_name=\"entity_description\",\n",
")\n",
"description_embedding_store.connect()\n",
"\n",
"full_content_embedding_store = LanceDBVectorStore(\n",
" db_uri=LANCEDB_URI,\n",
" index_name=\"community_full_content\",\n",
")\n",
"full_content_embedding_store.connect()\n",
"\n",
"print(f\"Entity count: {len(entity_df)}\")\n",
"entity_df.head()\n",
"\n",
"relationship_df = pd.read_parquet(f\"{INPUT_DIR}/{RELATIONSHIP_TABLE}.parquet\")\n",
"relationships = read_indexer_relationships(relationship_df)\n",
"\n",
"print(f\"Relationship count: {len(relationship_df)}\")\n",
"relationship_df.head()\n",
"\n",
"text_unit_df = pd.read_parquet(f\"{INPUT_DIR}/{TEXT_UNIT_TABLE}.parquet\")\n",
"text_units = read_indexer_text_units(text_unit_df)\n",
"\n",
"print(f\"Text unit records: {len(text_unit_df)}\")\n",
"text_unit_df.head()\n",
"\n",
"report_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_REPORT_TABLE}.parquet\")\n",
"reports = read_indexer_reports(report_df, community_df, COMMUNITY_LEVEL)\n",
"read_indexer_report_embeddings(reports, full_content_embedding_store)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"api_key = os.environ[\"GRAPHRAG_API_KEY\"]\n",
"\n",
"chat_config = LanguageModelConfig(\n",
" api_key=api_key,\n",
" type=ModelType.Chat,\n",
" model_provider=\"openai\",\n",
" model=\"gpt-4.1\",\n",
" max_retries=20,\n",
")\n",
"chat_model = ModelManager().get_or_create_chat_model(\n",
" name=\"local_search\",\n",
" model_type=ModelType.Chat,\n",
" config=chat_config,\n",
")\n",
"\n",
"tokenizer = get_tokenizer(chat_config)\n",
"\n",
"embedding_config = LanguageModelConfig(\n",
" api_key=api_key,\n",
" type=ModelType.Embedding,\n",
" model_provider=\"openai\",\n",
" model=\"text-embedding-3-large\",\n",
" max_retries=20,\n",
")\n",
"\n",
"text_embedder = ModelManager().get_or_create_embedding_model(\n",
" name=\"local_search_embedding\",\n",
" model_type=ModelType.Embedding,\n",
" config=embedding_config,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"drift_params = DRIFTSearchConfig(\n",
" primer_folds=1,\n",
" drift_k_followups=3,\n",
" n_depth=3,\n",
")\n",
"\n",
"context_builder = DRIFTSearchContextBuilder(\n",
" model=chat_model,\n",
" text_embedder=text_embedder,\n",
" entities=entities,\n",
" relationships=relationships,\n",
" reports=reports,\n",
" entity_text_embeddings=description_embedding_store,\n",
" text_units=text_units,\n",
" tokenizer=tokenizer,\n",
" config=drift_params,\n",
")\n",
"\n",
"search = DRIFTSearch(\n",
" model=chat_model, context_builder=context_builder, tokenizer=tokenizer\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"resp = await search.search(\"Who is agent Mercer?\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"resp.response"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(resp.context_data)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.10"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
docs/examples_notebooks/global_search.ipynb
+262
View File
@@ -0,0 +1,262 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Copyright (c) 2024 Microsoft Corporation.\n",
"# Licensed under the MIT License."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"import pandas as pd\n",
"from graphrag.config.enums import ModelType\n",
"from graphrag.config.models.language_model_config import LanguageModelConfig\n",
"from graphrag.language_model.manager import ModelManager\n",
"from graphrag.query.indexer_adapters import (\n",
" read_indexer_communities,\n",
" read_indexer_entities,\n",
" read_indexer_reports,\n",
")\n",
"from graphrag.query.structured_search.global_search.community_context import (\n",
" GlobalCommunityContext,\n",
")\n",
"from graphrag.query.structured_search.global_search.search import GlobalSearch\n",
"from graphrag.tokenizer.get_tokenizer import get_tokenizer"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Global Search example\n",
"\n",
"Global search method generates answers by searching over all AI-generated community reports in a map-reduce fashion. This is a resource-intensive method, but often gives good responses for questions that require an understanding of the dataset as a whole (e.g. What are the most significant values of the herbs mentioned in this notebook?)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### LLM setup"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"api_key = os.environ[\"GRAPHRAG_API_KEY\"]\n",
"\n",
"config = LanguageModelConfig(\n",
" api_key=api_key,\n",
" type=ModelType.Chat,\n",
" model_provider=\"openai\",\n",
" model=\"gpt-4.1\",\n",
" max_retries=20,\n",
")\n",
"model = ModelManager().get_or_create_chat_model(\n",
" name=\"global_search\",\n",
" model_type=ModelType.Chat,\n",
" config=config,\n",
")\n",
"\n",
"tokenizer = get_tokenizer(config)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Load community reports as context for global search\n",
"\n",
"- Load all community reports in the `community_reports` table from GraphRAG, to be used as context data for global search.\n",
"- Load entities from the `entities` tables from GraphRAG, to be used for calculating community weights for context ranking. Note that this is optional (if no entities are provided, we will not calculate community weights and only use the rank attribute in the community reports table for context ranking)\n",
"- Load all communities in the `communities` table from the GraphRAG, to be used to reconstruct the community graph hierarchy for dynamic community selection."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# parquet files generated from indexing pipeline\n",
"INPUT_DIR = \"./inputs/operation dulce\"\n",
"COMMUNITY_TABLE = \"communities\"\n",
"COMMUNITY_REPORT_TABLE = \"community_reports\"\n",
"ENTITY_TABLE = \"entities\"\n",
"\n",
"# community level in the Leiden community hierarchy from which we will load the community reports\n",
"# higher value means we use reports from more fine-grained communities (at the cost of higher computation cost)\n",
"COMMUNITY_LEVEL = 2"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"community_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_TABLE}.parquet\")\n",
"entity_df = pd.read_parquet(f\"{INPUT_DIR}/{ENTITY_TABLE}.parquet\")\n",
"report_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_REPORT_TABLE}.parquet\")\n",
"\n",
"communities = read_indexer_communities(community_df, report_df)\n",
"reports = read_indexer_reports(report_df, community_df, COMMUNITY_LEVEL)\n",
"entities = read_indexer_entities(entity_df, community_df, COMMUNITY_LEVEL)\n",
"\n",
"print(f\"Total report count: {len(report_df)}\")\n",
"print(\n",
" f\"Report count after filtering by community level {COMMUNITY_LEVEL}: {len(reports)}\"\n",
")\n",
"\n",
"report_df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Build global context based on community reports"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"context_builder = GlobalCommunityContext(\n",
" community_reports=reports,\n",
" communities=communities,\n",
" entities=entities, # default to None if you don't want to use community weights for ranking\n",
" tokenizer=tokenizer,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Perform global search"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"context_builder_params = {\n",
" \"use_community_summary\": False, # False means using full community reports. True means using community short summaries.\n",
" \"shuffle_data\": True,\n",
" \"include_community_rank\": True,\n",
" \"min_community_rank\": 0,\n",
" \"community_rank_name\": \"rank\",\n",
" \"include_community_weight\": True,\n",
" \"community_weight_name\": \"occurrence weight\",\n",
" \"normalize_community_weight\": True,\n",
" \"max_tokens\": 12_000, # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000)\n",
" \"context_name\": \"Reports\",\n",
"}\n",
"\n",
"map_llm_params = {\n",
" \"max_tokens\": 1000,\n",
" \"temperature\": 0.0,\n",
" \"response_format\": {\"type\": \"json_object\"},\n",
"}\n",
"\n",
"reduce_llm_params = {\n",
" \"max_tokens\": 2000, # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 1000-1500)\n",
" \"temperature\": 0.0,\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"search_engine = GlobalSearch(\n",
" model=model,\n",
" context_builder=context_builder,\n",
" tokenizer=tokenizer,\n",
" max_data_tokens=12_000, # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000)\n",
" map_llm_params=map_llm_params,\n",
" reduce_llm_params=reduce_llm_params,\n",
" allow_general_knowledge=False, # set this to True will add instruction to encourage the LLM to incorporate general knowledge in the response, which may increase hallucinations, but could be useful in some use cases.\n",
" json_mode=True, # set this to False if your LLM model does not support JSON mode.\n",
" context_builder_params=context_builder_params,\n",
" concurrent_coroutines=32,\n",
" response_type=\"multiple paragraphs\", # free form text describing the response type and format, can be anything, e.g. prioritized list, single paragraph, multiple paragraphs, multiple-page report\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"result = await search_engine.search(\"What is operation dulce?\")\n",
"\n",
"print(result.response)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# inspect the data used to build the context for the LLM responses\n",
"result.context_data[\"reports\"]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# inspect number of LLM calls and tokens\n",
"print(\n",
" f\"LLM calls: {result.llm_calls}. Prompt tokens: {result.prompt_tokens}. Output tokens: {result.output_tokens}.\"\n",
")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "graphrag",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.10"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
docs/examples_notebooks/global_search_with_dynamic_community_selection.ipynb
+295
View File
@@ -0,0 +1,295 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Copyright (c) 2024 Microsoft Corporation.\n",
"# Licensed under the MIT License."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"import pandas as pd\n",
"from graphrag.config.enums import ModelType\n",
"from graphrag.config.models.language_model_config import LanguageModelConfig\n",
"from graphrag.language_model.manager import ModelManager\n",
"from graphrag.query.indexer_adapters import (\n",
" read_indexer_communities,\n",
" read_indexer_entities,\n",
" read_indexer_reports,\n",
")\n",
"from graphrag.query.structured_search.global_search.community_context import (\n",
" GlobalCommunityContext,\n",
")\n",
"from graphrag.query.structured_search.global_search.search import GlobalSearch"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Global Search example\n",
"\n",
"Global search method generates answers by searching over all AI-generated community reports in a map-reduce fashion. This is a resource-intensive method, but often gives good responses for questions that require an understanding of the dataset as a whole (e.g. What are the most significant values of the herbs mentioned in this notebook?)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### LLM setup"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from graphrag.tokenizer.get_tokenizer import get_tokenizer\n",
"\n",
"api_key = os.environ[\"GRAPHRAG_API_KEY\"]\n",
"\n",
"config = LanguageModelConfig(\n",
" api_key=api_key,\n",
" type=ModelType.Chat,\n",
" model_provider=\"openai\",\n",
" model=\"gpt-4.1\",\n",
" max_retries=20,\n",
")\n",
"model = ModelManager().get_or_create_chat_model(\n",
" name=\"global_search\",\n",
" model_type=ModelType.Chat,\n",
" config=config,\n",
")\n",
"\n",
"tokenizer = get_tokenizer(config)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Load community reports as context for global search\n",
"\n",
"- Load all community reports in the `community_reports` table from the indexing engine, to be used as context data for global search.\n",
"- Load entities from the `entities` tables from the indexing engine, to be used for calculating community weights for context ranking. Note that this is optional (if no entities are provided, we will not calculate community weights and only use the rank attribute in the community reports table for context ranking)\n",
"- Load all communities in the `communities` table from the indexing engine, to be used to reconstruct the community graph hierarchy for dynamic community selection."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# parquet files generated from indexing pipeline\n",
"INPUT_DIR = \"./inputs/operation dulce\"\n",
"COMMUNITY_TABLE = \"communities\"\n",
"COMMUNITY_REPORT_TABLE = \"community_reports\"\n",
"ENTITY_TABLE = \"entities\"\n",
"\n",
"# we don't fix a specific community level but instead use an agent to dynamicially\n",
"# search through all the community reports to check if they are relevant.\n",
"COMMUNITY_LEVEL = None"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"community_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_TABLE}.parquet\")\n",
"entity_df = pd.read_parquet(f\"{INPUT_DIR}/{ENTITY_TABLE}.parquet\")\n",
"report_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_REPORT_TABLE}.parquet\")\n",
"\n",
"communities = read_indexer_communities(community_df, report_df)\n",
"reports = read_indexer_reports(\n",
" report_df,\n",
" community_df,\n",
" community_level=COMMUNITY_LEVEL,\n",
" dynamic_community_selection=True,\n",
")\n",
"entities = read_indexer_entities(\n",
" entity_df, community_df, community_level=COMMUNITY_LEVEL\n",
")\n",
"\n",
"print(f\"Total report count: {len(report_df)}\")\n",
"print(\n",
" f\"Report count after filtering by community level {COMMUNITY_LEVEL}: {len(reports)}\"\n",
")\n",
"\n",
"report_df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Build global context with dynamic community selection\n",
"\n",
"The goal of dynamic community selection reduce the number of community reports that need to be processed in the map-reduce operation. To that end, we take advantage of the hierachical structure of the indexed dataset. We first ask the LLM to rate how relevant each level 0 community is with respect to the user query, we then traverse down the child node(s) if the current community report is deemed relevant.\n",
"\n",
"You can still set a `COMMUNITY_LEVEL` to filter out lower level community reports and apply dynamic community selection on the filtered reports.\n",
"\n",
"Note that the dataset is quite small, with only consist of 20 communities from 2 levels (level 0 and 1). Dynamic community selection is more effective when there are large amount of content to be filtered out."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"context_builder = GlobalCommunityContext(\n",
" community_reports=reports,\n",
" communities=communities,\n",
" entities=entities, # default to None if you don't want to use community weights for ranking\n",
" tokenizer=tokenizer,\n",
" dynamic_community_selection=True,\n",
" dynamic_community_selection_kwargs={\n",
" \"model\": model,\n",
" \"tokenizer\": tokenizer,\n",
" },\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Perform global search with dynamic community selection"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"context_builder_params = {\n",
" \"use_community_summary\": False, # False means using full community reports. True means using community short summaries.\n",
" \"shuffle_data\": True,\n",
" \"include_community_rank\": True,\n",
" \"min_community_rank\": 0,\n",
" \"community_rank_name\": \"rank\",\n",
" \"include_community_weight\": True,\n",
" \"community_weight_name\": \"occurrence weight\",\n",
" \"normalize_community_weight\": True,\n",
" \"max_tokens\": 12_000, # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000)\n",
" \"context_name\": \"Reports\",\n",
"}\n",
"\n",
"map_llm_params = {\n",
" \"max_tokens\": 1000,\n",
" \"temperature\": 0.0,\n",
" \"response_format\": {\"type\": \"json_object\"},\n",
"}\n",
"\n",
"reduce_llm_params = {\n",
" \"max_tokens\": 2000, # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 1000-1500)\n",
" \"temperature\": 0.0,\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"search_engine = GlobalSearch(\n",
" model=model,\n",
" context_builder=context_builder,\n",
" tokenizer=tokenizer,\n",
" max_data_tokens=12_000, # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000)\n",
" map_llm_params=map_llm_params,\n",
" reduce_llm_params=reduce_llm_params,\n",
" allow_general_knowledge=False, # set this to True will add instruction to encourage the LLM to incorporate general knowledge in the response, which may increase hallucinations, but could be useful in some use cases.\n",
" json_mode=True, # set this to False if your LLM model does not support JSON mode.\n",
" context_builder_params=context_builder_params,\n",
" concurrent_coroutines=32,\n",
" response_type=\"multiple paragraphs\", # free form text describing the response type and format, can be anything, e.g. prioritized list, single paragraph, multiple paragraphs, multiple-page report\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"result = await search_engine.search(\"What is operation dulce?\")\n",
"\n",
"print(result.response)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# inspect the data used to build the context for the LLM responses\n",
"result.context_data[\"reports\"]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# inspect number of LLM calls and tokens in dynamic community selection\n",
"llm_calls = result.llm_calls_categories[\"build_context\"]\n",
"prompt_tokens = result.prompt_tokens_categories[\"build_context\"]\n",
"output_tokens = result.output_tokens_categories[\"build_context\"]\n",
"print(\n",
" f\"Build context LLM calls: {llm_calls}. Prompt tokens: {prompt_tokens}. Output tokens: {output_tokens}.\"\n",
")\n",
"# inspect number of LLM calls and tokens in map-reduce\n",
"llm_calls = result.llm_calls_categories[\"map\"] + result.llm_calls_categories[\"reduce\"]\n",
"prompt_tokens = (\n",
" result.prompt_tokens_categories[\"map\"] + result.prompt_tokens_categories[\"reduce\"]\n",
")\n",
"output_tokens = (\n",
" result.output_tokens_categories[\"map\"] + result.output_tokens_categories[\"reduce\"]\n",
")\n",
"print(\n",
" f\"Map-reduce LLM calls: {llm_calls}. Prompt tokens: {prompt_tokens}. Output tokens: {output_tokens}.\"\n",
")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "graphrag",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.10"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
docs/examples_notebooks/index_migration_to_v1.ipynb
+250
View File
@@ -0,0 +1,250 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 62,
"metadata": {},
"outputs": [],
"source": [
"# Copyright (c) 2024 Microsoft Corporation.\n",
"# Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Index Migration (pre-v1 to v1)\n",
"\n",
"This notebook is used to maintain data model parity with older indexes for version 1.0 of GraphRAG. If you have a pre-1.0 index and need to migrate without re-running the entire pipeline, you can use this notebook to only update the pieces necessary for alignment.\n",
"\n",
"NOTE: we recommend regenerating your settings.yml with the latest version of GraphRAG using `graphrag init`. Copy your LLM settings into it before running this notebook. This ensures your config is aligned with the latest version for the migration. This also ensures that you have default vector store config, which is now required or indexing will fail.\n",
"\n",
"WARNING: This will overwrite your parquet files, you may want to make a backup!\n"
]
},
{
"cell_type": "code",
"execution_count": 63,
"metadata": {},
"outputs": [],
"source": [
"# This is the directory that has your settings.yaml\n",
"# NOTE: much older indexes may have been output with a timestamped directory\n",
"# if this is the case, you will need to make sure the storage.base_dir in settings.yaml points to it correctly\n",
"PROJECT_DIRECTORY = \"<your project directory\""
]
},
{
"cell_type": "code",
"execution_count": 64,
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"\n",
"from graphrag.config.load_config import load_config\n",
"from graphrag.storage.factory import StorageFactory\n",
"\n",
"config = load_config(Path(PROJECT_DIRECTORY))\n",
"storage_config = config.output.model_dump()\n",
"storage = StorageFactory().create_storage(\n",
" storage_type=storage_config[\"type\"],\n",
" kwargs=storage_config,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 65,
"metadata": {},
"outputs": [],
"source": [
"def remove_columns(df, columns):\n",
" \"\"\"Remove columns from a DataFrame, suppressing errors.\"\"\"\n",
" df.drop(labels=columns, axis=1, errors=\"ignore\", inplace=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def get_community_parent(nodes):\n",
" \"\"\"Compute the parent community using the node membership as a lookup.\"\"\"\n",
" parent_mapping = nodes.loc[:, [\"level\", \"community\", \"title\"]]\n",
" nodes = nodes.loc[:, [\"level\", \"community\", \"title\"]]\n",
"\n",
" # Create a parent mapping by adding 1 to the level column\n",
" parent_mapping[\"level\"] += 1 # Shift levels for parent relationship\n",
" parent_mapping.rename(columns={\"community\": \"parent\"}, inplace=True)\n",
"\n",
" # Merge the parent information back into the base DataFrame\n",
" nodes = nodes.merge(parent_mapping, on=[\"level\", \"title\"], how=\"left\")\n",
"\n",
" # Fill missing parents with -1 (default value)\n",
" nodes[\"parent\"] = nodes[\"parent\"].fillna(-1).astype(int)\n",
"\n",
" join = (\n",
" nodes\n",
" .groupby([\"community\", \"level\", \"parent\"])\n",
" .agg({\"title\": list})\n",
" .reset_index()\n",
" )\n",
" return join[join[\"community\"] > -1].loc[:, [\"community\", \"parent\"]]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from uuid import uuid4\n",
"\n",
"from graphrag_storage.tables.parquet_table_provider import ParquetTableProvider\n",
"\n",
"# Create table provider from storage\n",
"table_provider = ParquetTableProvider(storage)\n",
"\n",
"# First we'll go through any parquet files that had model changes and update them\n",
"# The new data model may have removed excess columns as well, but we will only make the minimal changes required for compatibility\n",
"\n",
"final_documents = await table_provider.read_dataframe(\"create_final_documents\")\n",
"final_text_units = await table_provider.read_dataframe(\"create_final_text_units\")\n",
"final_entities = await table_provider.read_dataframe(\"create_final_entities\")\n",
"final_nodes = await table_provider.read_dataframe(\"create_final_nodes\")\n",
"final_relationships = await table_provider.read_dataframe(\"create_final_relationships\")\n",
"final_communities = await table_provider.read_dataframe(\"create_final_communities\")\n",
"final_community_reports = await table_provider.read_dataframe(\n",
" \"create_final_community_reports\"\n",
")\n",
"\n",
"\n",
"# Documents renames raw_content for consistency\n",
"if \"raw_content\" in final_documents.columns:\n",
" final_documents.rename(columns={\"raw_content\": \"text\"}, inplace=True)\n",
"final_documents[\"human_readable_id\"] = final_documents.index + 1\n",
"\n",
"# Text units just get a human_readable_id or consistency\n",
"final_text_units[\"human_readable_id\"] = final_text_units.index + 1\n",
"\n",
"# We renamed \"name\" to \"title\" for consistency with the rest of the tables\n",
"if \"name\" in final_entities.columns:\n",
" final_entities.rename(columns={\"name\": \"title\"}, inplace=True)\n",
"remove_columns(\n",
" final_entities, [\"name_embedding\", \"graph_embedding\", \"description_embedding\"]\n",
")\n",
"\n",
"# Final nodes uses community for joins, which is now an int everywhere\n",
"final_nodes[\"community\"] = final_nodes[\"community\"].fillna(-1)\n",
"final_nodes[\"community\"] = final_nodes[\"community\"].astype(int)\n",
"remove_columns(\n",
" final_nodes,\n",
" [\n",
" \"type\",\n",
" \"description\",\n",
" \"source_id\",\n",
" \"graph_embedding\",\n",
" \"entity_type\",\n",
" \"top_level_node_id\",\n",
" \"size\",\n",
" ],\n",
")\n",
"\n",
"# Relationships renames \"rank\" to \"combined_degree\" to be clear what the default ranking is\n",
"if \"rank\" in final_relationships.columns:\n",
" final_relationships.rename(columns={\"rank\": \"combined_degree\"}, inplace=True)\n",
"\n",
"\n",
"# Compute the parents for each community, to add to communities and reports\n",
"parent_df = get_community_parent(final_nodes)\n",
"\n",
"# Communities previously used the \"id\" field for the Leiden id, but we've moved this to the community field and use a uuid for id like the others\n",
"if \"community\" not in final_communities.columns:\n",
" final_communities[\"community\"] = final_communities[\"id\"].astype(int)\n",
" final_communities[\"human_readable_id\"] = final_communities[\"community\"]\n",
" final_communities[\"id\"] = [str(uuid4()) for _ in range(len(final_communities))]\n",
"if \"parent\" not in final_communities.columns:\n",
" final_communities = final_communities.merge(parent_df, on=\"community\", how=\"left\")\n",
"if \"entity_ids\" not in final_communities.columns:\n",
" node_mapping = (\n",
" final_nodes\n",
" .loc[:, [\"community\", \"id\"]]\n",
" .groupby(\"community\")\n",
" .agg(entity_ids=(\"id\", list))\n",
" )\n",
" final_communities = final_communities.merge(\n",
" node_mapping, on=\"community\", how=\"left\"\n",
" )\n",
"remove_columns(final_communities, [\"raw_community\"])\n",
"\n",
"# We need int for community and the human_readable_id copy for consistency\n",
"final_community_reports[\"community\"] = final_community_reports[\"community\"].astype(int)\n",
"final_community_reports[\"human_readable_id\"] = final_community_reports[\"community\"]\n",
"if \"parent\" not in final_community_reports.columns:\n",
" final_community_reports = final_community_reports.merge(\n",
" parent_df, on=\"community\", how=\"left\"\n",
" )\n",
"\n",
"await table_provider.write_dataframe(\"create_final_documents\", final_documents)\n",
"await table_provider.write_dataframe(\"create_final_text_units\", final_text_units)\n",
"await table_provider.write_dataframe(\"create_final_entities\", final_entities)\n",
"await table_provider.write_dataframe(\"create_final_nodes\", final_nodes)\n",
"await table_provider.write_dataframe(\"create_final_relationships\", final_relationships)\n",
"await table_provider.write_dataframe(\"create_final_communities\", final_communities)\n",
"await table_provider.write_dataframe(\n",
" \"create_final_community_reports\", final_community_reports\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from graphrag.callbacks.noop_workflow_callbacks import NoopWorkflowCallbacks\n",
"from graphrag.index.workflows.generate_text_embeddings import generate_text_embeddings\n",
"from graphrag_cache import create_cache\n",
"\n",
"# We only need to re-run the embeddings workflow, to ensure that embeddings\n",
"# for all required search fields are in place.\n",
"# We pass in the table_provider created earlier so that generate_text_embeddings\n",
"# reads the migrated tables we just wrote.\n",
"\n",
"callbacks = NoopWorkflowCallbacks()\n",
"cache = create_cache(config.cache)\n",
"\n",
"await generate_text_embeddings(\n",
" config=config,\n",
" table_provider=table_provider,\n",
" cache=cache,\n",
" callbacks=callbacks,\n",
")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "graphrag",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.10"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
docs/examples_notebooks/index_migration_to_v2.ipynb
+167
View File
@@ -0,0 +1,167 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"# Copyright (c) 2024 Microsoft Corporation.\n",
"# Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Index Migration (v1 to v2)\n",
"\n",
"This notebook is used to maintain data model parity with older indexes for version 2.0 of GraphRAG. If you have a pre-2.0 index and need to migrate without re-running the entire pipeline, you can use this notebook to only update the pieces necessary for alignment. If you have a pre-1.0 index, please run the v1 migration notebook first!\n",
"\n",
"NOTE: we recommend regenerating your settings.yml with the latest version of GraphRAG using `graphrag init`. Copy your LLM settings into it before running this notebook. This ensures your config is aligned with the latest version for the migration. This also ensures that you have default vector store config, which is now required or indexing will fail.\n",
"\n",
"WARNING: This will overwrite your parquet files, you may want to make a backup!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# This is the directory that has your settings.yaml\n",
"PROJECT_DIRECTORY = \"<your project directory>\""
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"\n",
"from graphrag.config.load_config import load_config\n",
"from graphrag.storage.factory import StorageFactory\n",
"\n",
"config = load_config(Path(PROJECT_DIRECTORY))\n",
"storage_config = config.output.model_dump()\n",
"storage = StorageFactory().create_storage(\n",
" storage_type=storage_config[\"type\"],\n",
" kwargs=storage_config,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
"def remove_columns(df, columns):\n",
" \"\"\"Remove columns from a DataFrame, suppressing errors.\"\"\"\n",
" df.drop(labels=columns, axis=1, errors=\"ignore\", inplace=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"from graphrag_storage.tables.parquet_table_provider import ParquetTableProvider\n",
"\n",
"# Create table provider from storage\n",
"table_provider = ParquetTableProvider(storage)\n",
"\n",
"final_documents = await table_provider.read_dataframe(\"create_final_documents\")\n",
"final_text_units = await table_provider.read_dataframe(\"create_final_text_units\")\n",
"final_entities = await table_provider.read_dataframe(\"create_final_entities\")\n",
"final_covariates = await table_provider.read_dataframe(\"create_final_covariates\")\n",
"final_nodes = await table_provider.read_dataframe(\"create_final_nodes\")\n",
"final_relationships = await table_provider.read_dataframe(\"create_final_relationships\")\n",
"final_communities = await table_provider.read_dataframe(\"create_final_communities\")\n",
"final_community_reports = await table_provider.read_dataframe(\n",
" \"create_final_community_reports\"\n",
")\n",
"\n",
"# we've renamed document attributes as metadata\n",
"if \"attributes\" in final_documents.columns:\n",
" final_documents.rename(columns={\"attributes\": \"metadata\"}, inplace=True)\n",
"\n",
"# we're removing the nodes table, so we need to copy the graph columns into entities\n",
"graph_props = (\n",
" final_nodes.loc[:, [\"id\", \"degree\", \"x\", \"y\"]].groupby(\"id\").first().reset_index()\n",
")\n",
"final_entities = final_entities.merge(graph_props, on=\"id\", how=\"left\")\n",
"# we're also persisting the frequency column\n",
"final_entities[\"frequency\"] = final_entities[\"text_unit_ids\"].count()\n",
"\n",
"\n",
"# we added children to communities to eliminate query-time reconstruction\n",
"parent_grouped = final_communities.groupby(\"parent\").agg(\n",
" children=(\"community\", \"unique\")\n",
")\n",
"final_communities = final_communities.merge(\n",
" parent_grouped,\n",
" left_on=\"community\",\n",
" right_on=\"parent\",\n",
" how=\"left\",\n",
")\n",
"# replace NaN children with empty list\n",
"final_communities[\"children\"] = final_communities[\"children\"].apply(\n",
" lambda x: x if isinstance(x, np.ndarray) else [] # type: ignore\n",
")\n",
"\n",
"# add children to the reports as well\n",
"final_community_reports = final_community_reports.merge(\n",
" parent_grouped,\n",
" left_on=\"community\",\n",
" right_on=\"parent\",\n",
" how=\"left\",\n",
")\n",
"\n",
"# we renamed all the output files for better clarity now that we don't have workflow naming constraints from DataShaper\n",
"await table_provider.write_dataframe(\"documents\", final_documents)\n",
"await table_provider.write_dataframe(\"text_units\", final_text_units)\n",
"await table_provider.write_dataframe(\"entities\", final_entities)\n",
"await table_provider.write_dataframe(\"relationships\", final_relationships)\n",
"await table_provider.write_dataframe(\"covariates\", final_covariates)\n",
"await table_provider.write_dataframe(\"communities\", final_communities)\n",
"await table_provider.write_dataframe(\"community_reports\", final_community_reports)\n",
"\n",
"# delete all the old versions\n",
"await storage.delete(\"create_final_documents.parquet\")\n",
"await storage.delete(\"create_final_text_units.parquet\")\n",
"await storage.delete(\"create_final_entities.parquet\")\n",
"await storage.delete(\"create_final_nodes.parquet\")\n",
"await storage.delete(\"create_final_relationships.parquet\")\n",
"await storage.delete(\"create_final_covariates.parquet\")\n",
"await storage.delete(\"create_final_communities.parquet\")\n",
"await storage.delete(\"create_final_community_reports.parquet\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.9"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
docs/examples_notebooks/index_migration_to_v3.ipynb
+146
View File
@@ -0,0 +1,146 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Copyright (c) 2024 Microsoft Corporation.\n",
"# Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Index Migration (v2 to v3)\n",
"\n",
"This notebook is used to maintain data model parity with older indexes for version 3.0 of GraphRAG. If you have a pre-3.0 index and need to migrate without re-running the entire pipeline, you can use this notebook to only update the pieces necessary for alignment. If you have a pre-2.0 index, please run the v2 migration notebook first!\n",
"\n",
"NOTE: we recommend regenerating your settings.yml with the latest version of GraphRAG using `graphrag init`. Copy your LLM settings into it before running this notebook. This ensures your config is aligned with the latest version for the migration. The config changes from v2 to v3 are significant in places!\n",
"\n",
"WARNING: This will overwrite your parquet files, you may want to make a backup!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# This is the directory that has your settings.yaml\n",
"PROJECT_DIRECTORY = \"<your project directory>\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"\n",
"from graphrag.config.models.graph_rag_config import GraphRagConfig\n",
"from graphrag_common.config import load_config\n",
"from graphrag_storage.storage_factory import create_storage\n",
"\n",
"config = load_config(GraphRagConfig, config_path=Path(PROJECT_DIRECTORY))\n",
"storage = create_storage(config.output_storage)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def remove_columns(df, columns):\n",
" \"\"\"Remove columns from a DataFrame, suppressing errors.\"\"\"\n",
" df.drop(labels=columns, axis=1, errors=\"ignore\", inplace=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from graphrag_storage.tables.parquet_table_provider import ParquetTableProvider\n",
"\n",
"# Create table provider from storage\n",
"table_provider = ParquetTableProvider(storage)\n",
"\n",
"text_units = await table_provider.read_dataframe(\"text_units\")\n",
"\n",
"text_units[\"document_id\"] = text_units[\"document_ids\"].apply(lambda ids: ids[0])\n",
"remove_columns(text_units, [\"document_ids\"])\n",
"\n",
"await table_provider.write_dataframe(\"text_units\", text_units)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Update settings.yaml\n",
"If you have left the default settings for your vector store schema, you may need to set explicit values that map each embedding type to a vector schema name. If you have already customized your vector store schema it may not be necessary.\n",
"\n",
"Old default index names:\n",
"- default-text_unit-text\n",
"- default-entity-description\n",
"- default-community-full_content\n",
"\n",
"(if you left all of the defaults, check your output/lancedb folder to confirm the above)\n",
"\n",
"v3 versions are:\n",
"- text_unit_text\n",
"- entity_description\n",
"- community_full_content\n",
"\n",
"Therefore, with a v2 index need to explicitly set the old index names so it connects correctly. We no longer support the \"prefix\" - you can just set an explicit index_name for each embedding.\n",
"\n",
"NOTE: we are also setting the default vector_size for each index below, under the assumption that you are using a prior default with 1536 dimensions. Our new default of text-embedding-3-large has 3072 dimensions, which will be populated as the default if unset. Again, if you have a more complicated situation you may want to manually configure this.\n",
"\n",
"Here is an example of the new vector store config block that you may need in your settings.yaml:\n",
"\n",
"```yaml\n",
"vector_store:\n",
" type: lancedb\n",
" db_uri: output/lancedb\n",
" index_schema:\n",
" text_unit_text:\n",
" index_name: default-text_unit-text\n",
" vector_size: 1536\n",
" entity_description:\n",
" index_name: default-entity-description\n",
" vector_size: 1536\n",
" community_full_content:\n",
" index_name: default-community-full_content\n",
" vector_size: 1536\n",
"```\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
docs/examples_notebooks/input_documents.ipynb
+193
View File
@@ -0,0 +1,193 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Copyright (c) 2024 Microsoft Corporation.\n",
"# Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Example of indexing from an existing in-memory dataframe\n",
"\n",
"Newer versions of GraphRAG let you submit a dataframe directly instead of running through the input processing step. This notebook demonstrates with regular or update runs.\n",
"\n",
"If performing an update, the assumption is that your dataframe contains only the new documents to add to the index.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"from pprint import pprint\n",
"\n",
"import graphrag.api as api\n",
"import pandas as pd\n",
"from graphrag.config.load_config import load_config\n",
"from graphrag.index.typing.pipeline_run_result import PipelineRunResult"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"PROJECT_DIRECTORY = \"<your project directory>\"\n",
"UPDATE = False\n",
"FILENAME = \"new_documents.parquet\" if UPDATE else \"<original_documents>.parquet\"\n",
"inputs = pd.read_parquet(f\"{PROJECT_DIRECTORY}/input/{FILENAME}\")\n",
"# Only the bare minimum for input. These are the same fields that would be present after the load_input_documents workflow\n",
"inputs = inputs.loc[:, [\"id\", \"title\", \"text\", \"creation_date\"]]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Generate a `GraphRagConfig` object\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"graphrag_config = load_config(Path(PROJECT_DIRECTORY))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Indexing API\n",
"\n",
"_Indexing_ is the process of ingesting raw text data and constructing a knowledge graph. GraphRAG currently supports plaintext (`.txt`) and `.csv` file formats.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Build an index\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"index_result: list[PipelineRunResult] = await api.build_index(\n",
" config=graphrag_config, input_documents=inputs, is_update_run=UPDATE\n",
")\n",
"\n",
"# index_result is a list of workflows that make up the indexing pipeline that was run\n",
"for workflow_result in index_result:\n",
" status = f\"error\\n{workflow_result.errors}\" if workflow_result.errors else \"success\"\n",
" print(f\"Workflow Name: {workflow_result.workflow}\\tStatus: {status}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Query an index\n",
"\n",
"To query an index, several index files must first be read into memory and passed to the query API.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"entities = pd.read_parquet(f\"{PROJECT_DIRECTORY}/output/entities.parquet\")\n",
"communities = pd.read_parquet(f\"{PROJECT_DIRECTORY}/output/communities.parquet\")\n",
"community_reports = pd.read_parquet(\n",
" f\"{PROJECT_DIRECTORY}/output/community_reports.parquet\"\n",
")\n",
"\n",
"response, context = await api.global_search(\n",
" config=graphrag_config,\n",
" entities=entities,\n",
" communities=communities,\n",
" community_reports=community_reports,\n",
" community_level=2,\n",
" dynamic_community_selection=False,\n",
" response_type=\"Multiple Paragraphs\",\n",
" query=\"What are the top five themes of the dataset?\",\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The response object is the official reponse from graphrag while the context object holds various metadata regarding the querying process used to obtain the final response.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(response)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Digging into the context a bit more provides users with extremely granular information such as what sources of data (down to the level of text chunks) were ultimately retrieved and used as part of the context sent to the LLM model).\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pprint(context) # noqa: T203"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "graphrag-monorepo",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.9"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

Some files were not shown because too many files have changed in this diff Show More