chore: import upstream snapshot with attribution

docs/blog_posts.md

@@ -0,0 +1,54 @@
+
+<div class="grid cards" markdown>
+
+-   [:octicons-arrow-right-24: __GraphRAG: Unlocking LLM discovery on narrative private data__](https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/)
+
+    ---
+    <h6>Published February 13, 2024
+
+    By [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect; [Steven Truitt](https://www.microsoft.com/en-us/research/people/steventruitt/), Principal Program Manager</h6>
+    
+
+-   [:octicons-arrow-right-24: __GraphRAG: New tool for complex data discovery now on GitHub__](https://www.microsoft.com/en-us/research/blog/graphrag-new-tool-for-complex-data-discovery-now-on-github/)
+
+    ---
+    <h6>Published July 2, 2024
+
+    By [Darren Edge](https://www.microsoft.com/en-us/research/people/daedge/), Senior Director; [Ha Trinh](https://www.microsoft.com/en-us/research/people/trinhha/), Senior Data Scientist; [Steven Truitt](https://www.microsoft.com/en-us/research/people/steventruitt/), Principal Program Manager; [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect</h6>
+
+
+-   [:octicons-arrow-right-24: __GraphRAG auto-tuning provides rapid adaptation to new domains__](https://www.microsoft.com/en-us/research/blog/graphrag-auto-tuning-provides-rapid-adaptation-to-new-domains/)
+
+    ---
+    <h6>Published September 9, 2024
+
+    By [Alonso Guevara Fernández](https://www.microsoft.com/en-us/research/people/alonsog/), Sr. Software Engineer; Katy Smith, Data Scientist II; [Joshua Bradley](https://www.microsoft.com/en-us/research/people/joshbradley/), Senior Data Scientist; [Darren Edge](https://www.microsoft.com/en-us/research/people/daedge/), Senior Director; [Ha Trinh](https://www.microsoft.com/en-us/research/people/trinhha/), Senior Data Scientist; [Sarah Smith](https://www.microsoft.com/en-us/research/people/smithsarah/), Senior Program Manager; [Ben Cutler](https://www.microsoft.com/en-us/research/people/bcutler/), Senior Director; [Steven Truitt](https://www.microsoft.com/en-us/research/people/steventruitt/), Principal Program Manager; [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect</h6>
+
+- [:octicons-arrow-right-24: __Introducing DRIFT Search: Combining global and local search methods to improve quality and efficiency__](https://www.microsoft.com/en-us/research/blog/introducing-drift-search-combining-global-and-local-search-methods-to-improve-quality-and-efficiency/)
+
+    ---
+    <h6>Published October 31, 2024
+
+    By Julian Whiting, Senior Machine Learning Engineer; Zachary Hills , Senior Software Engineer; [Alonso Guevara Fernández](https://www.microsoft.com/en-us/research/people/alonsog/), Sr. Software Engineer; [Ha Trinh](https://www.microsoft.com/en-us/research/people/trinhha/), Senior Data Scientist; Adam Bradley , Managing Partner, Strategic Research; [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect</h6>
+
+- [:octicons-arrow-right-24: __GraphRAG: Improving global search via dynamic community selection__](https://www.microsoft.com/en-us/research/blog/graphrag-improving-global-search-via-dynamic-community-selection/)
+
+    ---
+    <h6>Published November 15, 2024
+
+    By Bryan Li, Research Intern; [Ha Trinh](https://www.microsoft.com/en-us/research/people/trinhha/), Senior Data Scientist; [Darren Edge](https://www.microsoft.com/en-us/research/people/daedge/), Senior Director; [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect</h6>
+
+- [:octicons-arrow-right-24: __LazyGraphRAG: Setting a new standard for quality and cost__](https://www.microsoft.com/en-us/research/blog/lazygraphrag-setting-a-new-standard-for-quality-and-cost/)
+
+    ---
+    <h6>Published November 25, 2024
+
+    By [Darren Edge](https://www.microsoft.com/en-us/research/people/daedge/), Senior Director; [Ha Trinh](https://www.microsoft.com/en-us/research/people/trinhha/), Senior Data Scientist;  [Jonathan Larson](https://www.microsoft.com/en-us/research/people/jolarso/), Senior Principal Data Architect</h6>
+
+- [:octicons-arrow-right-24: __Moving to GraphRAG 1.0 – Streamlining ergonomics for developers and users__](https://www.microsoft.com/en-us/research/blog/moving-to-graphrag-1-0-streamlining-ergonomics-for-developers-and-users)
+
+    ---
+    <h6>Published December 16, 2024
+
+    By [Nathan Evans](https://www.microsoft.com/en-us/research/people/naevans/), Principal Software Architect; [Alonso Guevara Fernández](https://www.microsoft.com/en-us/research/people/alonsog/), Senior Software Engineer; [Joshua Bradley](https://www.microsoft.com/en-us/research/people/joshbradley/), Senior Data Scientist</h6>
+</div>

docs/cli.md

@@ -0,0 +1,9 @@
+# CLI Reference
+
+This page documents the command-line interface of the graphrag library.
+
+::: mkdocs-typer
+    :module: graphrag.cli.main
+    :prog_name: graphrag
+    :command: app
+    :depth: 0

docs/config/init.md

@@ -0,0 +1,32 @@
+# Configuring GraphRAG Indexing
+
+To start using GraphRAG, you must generate a configuration file. The `init` command is the easiest way to get started. It will create a `.env` and `settings.yaml` files in the specified directory with the necessary configuration settings. It will also output the default LLM prompts used by GraphRAG.
+
+## Usage
+
+```sh
+graphrag init [--root PATH] [--force, --no-force]
+```
+
+## Options
+
+- `--root PATH` - The project root directory to initialize graphrag at. Default is the current directory.
+- `--force`, `--no-force` - Optional, default is --no-force. Overwrite existing configuration and prompt files if they exist.
+
+## Example
+
+```sh
+graphrag init --root ./ragtest
+```
+
+## Output
+
+The `init` command will create the following files in the specified directory:
+
+- `settings.yaml` - The configuration settings file. This file contains the configuration settings for GraphRAG.
+- `.env` - The environment variables file. These are referenced in the `settings.yaml` file.
+- `prompts/` - The LLM prompts folder. This contains the default prompts used by GraphRAG, you can modify them or run the [Auto Prompt Tuning](../prompt_tuning/auto_prompt_tuning.md) command to generate new prompts adapted to your data.
+
+## Next Steps
+
+After initializing your workspace, you can either run the [Prompt Tuning](../prompt_tuning/auto_prompt_tuning.md) command to adapt the prompts to your data or even start running the [Indexing Pipeline](../index/overview.md) to index your data. For more information on configuration options available, see the [YAML details page](yaml.md).

docs/config/models.md

@@ -0,0 +1,115 @@
+# Language Model Selection and Overriding
+
+This page contains information on selecting a model to use and options to supply your own model for GraphRAG. Note that this is not a guide to finding the right model for your use case.
+
+## Default Model Support
+
+GraphRAG was built and tested using OpenAI models, so this is the default model set we support. This is not intended to be a limiter or statement of quality or fitness for your use case, only that it's the set we are most familiar with for prompting, tuning, and debugging.
+
+GraphRAG uses [LiteLLM](https://docs.litellm.ai/) for calling language models. LiteLLM provides support for 100+ models though it is important to note that when choosing a model it must support returning [structured outputs](https://openai.com/index/introducing-structured-outputs-in-the-api/) adhering to a [JSON schema](https://docs.litellm.ai/docs/completion/json_mode).
+
+Example using LiteLLM as the language model manager for GraphRAG:
+
+```yaml
+completion_models:
+  default_completion_model:
+    model_provider: gemini
+    model: gemini-2.5-flash-lite
+    auth_method: api_key
+    api_key: ${GEMINI_API_KEY}
+
+embedding_models:
+  default_embedding_model:
+    model_provider: gemini
+    model: gemini-embedding-001
+    auth_method: api_key
+    api_key: ${GEMINI_API_KEY}
+```
+
+See [Detailed Configuration](yaml.md) for more details on configuration. [View LiteLLM basic usage](https://docs.litellm.ai/docs/#basic-usage) for details on how models are called (The `model_provider` is the portion prior to `/` while the `model` is the portion following the `/`).
+
+## Model Selection Considerations
+
+GraphRAG has been most thoroughly tested with the gpt-4 series of models from OpenAI, including gpt-4 gpt-4-turbo, gpt-4o, and gpt-4o-mini. Our [arXiv paper](https://arxiv.org/abs/2404.16130), for example, performed quality evaluation using gpt-4-turbo. As stated above, non-OpenAI models are supported through the use of LiteLLM but the suite of gpt-4 series of models from OpenAI remain the most tested and supported suite of models for GraphRAG – in other words, these are the models we know best and can help resolve issues with.
+
+Versions of GraphRAG before 2.2.0 made extensive use of `max_tokens` and `logit_bias` to control generated response length or content. The introduction of the o-series of models added new, non-compatible parameters because these models include a reasoning component that has different consumption patterns and response generation attributes than non-reasoning models. GraphRAG 2.2.0 now supports these models, but there are important differences that need to be understood before you switch.
+
+- Previously, GraphRAG used `max_tokens` to limit responses in a few locations. This is done so that we can have predictable content sizes when building downstream context windows for summarization. We have now switched from using `max_tokens` to use a prompted approach, which is working well in our tests. We suggest using `max_tokens` in your language model config only for budgetary reasons if you want to limit consumption, and not for expected response length control. We now also support the o-series equivalent `max_completion_tokens`, but if you use this keep in mind that there may be some unknown fixed reasoning consumption amount in addition to the response tokens, so it is not a good technique for response control.
+- Previously, GraphRAG used a combination of `max_tokens` and `logit_bias` to strictly control a binary yes/no question during gleanings. This is not possible with reasoning models, so again we have switched to a prompted approach. Our tests with gpt-4o, gpt-4o-mini, and o1 show that this works consistently, but could have issues if you have an older or smaller model.
+- The o-series models are much slower and more expensive. It may be useful to use an asymmetric approach to model use in your config: you can define as many models as you like in the `models` block of your settings.yaml and reference them by key for every workflow that requires a language model. You could use gpt-4o for indexing and o1 for query, for example. Experiment to find the right balance of cost, speed, and quality for your use case.
+- The o-series models contain a form of native native chain-of-thought reasoning that is absent in the non-o-series models. GraphRAG's prompts sometimes contain CoT because it was an effective technique with the gpt-4\* series. It may be counterproductive with the o-series, so you may want to tune or even re-write large portions of the prompt templates (particularly for graph and claim extraction).
+
+Example config with asymmetric model use:
+
+```yaml
+completion_models:
+  extraction_completion_model:
+    model_provider: openai
+    model: gpt-4o
+    auth_method: api_key
+    api_key: ${GRAPHRAG_API_KEY}
+  query_completion_model:
+    model_provider: openai
+    model: o1
+    auth_method: api_key
+    api_key: ${GRAPHRAG_API_KEY}
+...
+extract_graph:
+  completion_model_id: extraction_completion_model
+  prompt: "prompts/extract_graph.txt"
+  entity_types: [organization, person, geo, event]
+  max_gleanings: 1
+...
+global_search:
+  completion_model_id: query_completion_model
+  map_prompt: "prompts/global_search_map_system_prompt.txt"
+  reduce_prompt: "prompts/global_search_reduce_system_prompt.txt"
+  knowledge_prompt: "prompts/global_search_knowledge_system_prompt.txt"
+```
+
+Another option would be to avoid using a language model at all for the graph extraction, instead using the `fast` [indexing method](../index/methods.md) that uses NLP for portions of the indexing phase in lieu of LLM APIs.
+
+## Using Custom Models
+
+LiteLLM supports hundreds of models, but cases may still exist in which some users wish to use models not supported by LiteLLM. There are two approaches one can use to connect to unsupported models:
+
+### Proxy APIs
+
+Many users have used platforms such as [ollama](https://ollama.com/) and [LiteLLM Proxy Server](https://docs.litellm.ai/docs/simple_proxy) to proxy the underlying model HTTP calls to a different model provider. This seems to work reasonably well, but we frequently see issues with malformed responses (especially JSON), so if you do this please understand that your model needs to reliably return the specific response formats that GraphRAG expects. If you're having trouble with a model, you may need to try prompting to coax the format, or intercepting the response within your proxy to try and handle malformed responses.
+
+### Model Protocol
+
+We support model injection through the use of a standard completion and embedding Protocol and accompanying factories that you can use to register your model implementation. This is not supported with the CLI, so you'll need to use GraphRAG as a library.
+
+- Our Protocol is [defined here](https://github.com/microsoft/graphrag/blob/main/packages/graphrag-llm/graphrag_llm/completion/completion.py)
+- We have a simple mock implementation in our tests that you can [reference here](https://github.com/microsoft/graphrag/blob/main/packages/graphrag-llm/graphrag_llm/completion/mock_llm_completion.py)
+
+Once you have a model implementation, you need to register it with our completion model factory or embedding model factory:
+
+```python
+from graphrag_llm.completion import LLMCompletion, register_completion
+
+class MyCustomCompletionModel(LLMCompletion):
+    ...
+    # implementation
+
+# elsewhere...
+register_completion("my-custom-completion-model", MyCustomCompletionModel)
+```
+
+Then in your config you can reference the type name you used:
+
+```yaml
+completion_models:
+  default_completion_model:
+    type: my-custom-completion-model
+    ...
+
+extract_graph:
+  completion_model_id: default_completion_model
+  prompt: "prompts/extract_graph.txt"
+  entity_types: [organization, person, geo, event]
+  max_gleanings: 1
+```
+
+Note that your custom model will be passed the same params for init and method calls that we use throughout GraphRAG. There is not currently any ability to define custom parameters, so you may need to use closure scope or a factory pattern within your implementation to get custom config values.

docs/config/overview.md

@@ -0,0 +1,10 @@
+# Configuring GraphRAG Indexing
+
+The GraphRAG system is highly configurable. This page provides an overview of the configuration options available for the GraphRAG indexing engine.
+
+## Default Configuration Mode
+
+The default configuration mode is the simplest way to get started with the GraphRAG system. It is designed to work out-of-the-box with minimal configuration. The main ways to set up GraphRAG in Default Configuration mode are via:
+
+- [Init command](init.md) (recommended first step)
+- [Edit settings.yaml for deeper control](yaml.md)

docs/config/yaml.md

@@ -0,0 +1,405 @@
+# Default Configuration Mode (using YAML/JSON)
+
+The default configuration mode may be configured by using a `settings.yml` or `settings.json` file in the data project root. If a `.env` file is present along with this config file, then it will be loaded, and the environment variables defined therein will be available for token replacements in your configuration document using `${ENV_VAR}` syntax. We initialize with YML by default in `graphrag init` but you may use the equivalent JSON form if preferred.
+
+Many of these config values have defaults. Rather than replicate them here, please refer to the [constants in the code](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/config/defaults.py) directly.
+
+For example:
+
+```bash
+# .env
+GRAPHRAG_API_KEY=some_api_key
+
+# settings.yml
+default_chat_model:
+  api_key: ${GRAPHRAG_API_KEY}
+```
+
+# Config Sections
+
+## Language Model Setup
+
+### models
+
+This is a set of dicts, one for completion model configuration and one for embedding model configuration. The dict keys are used to reference the model configuration elsewhere when a model instance is desired. In this way, you can specify as many different models as you need, and reference them independently in the workflow steps.
+
+For example:
+
+```yml
+completion_models:
+  default_completion_model:
+    model_provider: openai
+    model: gpt-4.1
+    auth_method: api_key
+    api_key: ${GRAPHRAG_API_KEY}
+
+embedding_models:
+  default_embedding_model:
+    model_provider: openai
+    model: text-embedding-3-large
+    auth_method: api_key
+    api_key: ${GRAPHRAG_API_KEY}
+```
+
+#### Fields
+
+- `type` **litellm|mock** - The type of LLM provider to use. GraphRAG uses [LiteLLM](https://docs.litellm.ai/) for calling language models.
+- `model_provider` **str** - The model provider to use, e.g., openai, azure, anthropic, etc. [LiteLLM](https://docs.litellm.ai/) is used under the hood which has support for calling 100+ models. [View LiteLLm basic usage](https://docs.litellm.ai/docs/#basic-usage) for details on how models are called (The `model_provider` is the portion prior to `/` while the `model` is the portion following the `/`). [View Language Model Selection](models.md) for more details and examples on using LiteLLM.
+- `model` **str** - The model name.
+- `call_args`: **dict[str, Any]** - Default arguments to send with every model request. Example, `{"n": 5, "max_completion_tokens": 1000, "temperature": 1.5, "organization": "..." }`
+- `api_key` **str|None** - The OpenAI API key to use.
+- `api_base` **str|None** - The API base url to use.
+- `api_version` **str|None** - The API version.
+- `auth_method` **api_key|azure_managed_identity** - Indicate how you want to authenticate requests.
+- `azure_deployment_name` **str|None** - The deployment name to use if your model is hosted on Azure. Note that if your deployment name on Azure matches the model name, this is unnecessary.
+- retry **RetryConfig|None** - Retry settings. default=`None`, no retries.
+  - type **exponential_backoff|immediate** - Type of retry approach. default=`exponential_backoff`
+  - max_retries **int|None** - Max retries to take. default=`7`.
+  - base_delay **float|None** - Base delay when using `exponential_backoff`. default=`2.0`.
+  - jitter **bool|None** - Add jitter to retry delays when using `exponential_backoff`. default=`True`
+  - max_delay **float|None** - Maximum retry delay. default=`None`, no max.
+- rate_limit **RateLimitConfig|None** - Rate limit settings. default=`None`, no rate limiting.
+  - type **sliding_window** - Type of rate limit approach. default=`sliding_window`
+  - period_in_seconds **int|None** - Window size for `sliding_window` rate limiting. default=`60`, limit requests per minute.
+  - requests_per_period **int|None** - Maximum number of requests per period. default=`None`
+  - tokens_per_period **int|None** - Maximum number of tokens per period. default=`None`
+- metrics **MetricsConfig|None** - Metric settings. default=`MetricsConfig()`. View [metrics notebook](https://github.com/microsoft/graphrag/blob/main/packages/graphrag-llm/notebooks/04_metrics.ipynb) for more details on metrics.
+  - type **default** - The type of `MetricsProcessor` service to use for processing request metrics. default=`default`
+  - store **memory** - The type of `MetricsStore` service. default=`memory`.
+  - writer **log|file** - The type of `MetricsWriter` to use. Will write out metrics at the end of the process. default`log`, log metrics out using python standard logging at the end of the process.
+  - log_level **int|None** - The log level when using `log` writer. default=`20`, log `INFO` messages for metrics.
+  - base_dir **str|None** - The directory to write metrics to when using `file` writer. default=`Path.cwd()`.
+
+## Input Files and Chunking
+
+### input
+
+Our pipeline can ingest .csv, .txt, or .json data from an input location. See the [inputs page](../index/inputs.md) for more details and examples.
+
+#### Fields
+
+- `storage` **StorageConfig**
+  - `type` **file|memory|blob|cosmosdb** - The storage type to use. Default=`file`
+  - `encoding`**str** - The encoding to use for file storage.
+  - `base_dir` **str** - The base directory to write output artifacts to, relative to the root.
+  - `connection_string` **str** - (blob/cosmosdb only) The Azure Storage connection string.
+  - `container_name` **str** - (blob/cosmosdb only) The Azure Storage container name.
+  - `account_url` **str** - (blob only) The storage account blob URL to use.
+  - `database_name` **str** - (cosmosdb only) The database name to use.
+- `type` **text|csv|json** - The type of input data to load. Default is `text`
+- `encoding` **str** - The encoding of the input file. Default is `utf-8`
+- `file_pattern` **str** - A regex to match input files. Default is `.*\.csv$`, `.*\.txt$`, or `.*\.json$` depending on the specified `type`, but you can customize it if needed.
+- `id_column` **str** - The input ID column to use.
+- `title_column` **str** - The input title column to use.
+- `text_column` **str** - The input text column to use.
+
+### chunking
+
+These settings configure how we parse documents into text chunks. This is necessary because very large documents may not fit into a single context window, and graph extraction accuracy can be modulated. Also note the `metadata` setting in the input document config, which will replicate document metadata into each chunk.
+
+#### Fields
+
+- `type` **tokens|sentence** - The chunking type to use.
+- `encoding_model` **str** - The text encoding model to use for splitting on token boundaries.
+- `size` **int** - The max chunk size in tokens.
+- `overlap` **int** - The chunk overlap in tokens.
+- `prepend_metadata` **list[str]** - Metadata fields from the source document to prepend on each chunk.
+
+## Outputs and Storage
+
+### output
+
+This section controls the storage mechanism used by the pipeline used for exporting output tables.
+
+#### Fields
+
+- `type` **file|memory|blob|cosmosdb** - The storage type to use. Default=`file`
+- `encoding`**str** - The encoding to use for file storage.
+- `base_dir` **str** - The base directory to write output artifacts to, relative to the root.
+- `connection_string` **str** - (blob/cosmosdb only) The Azure Storage connection string.
+- `container_name` **str** - (blob/cosmosdb only) The Azure Storage container name.
+- `account_url` **str** - (blob only) The storage account blob URL to use.
+- `database_name` **str** - (cosmosdb only) The database name to use.
+- `type` **text|csv|json** - The type of input data to load. Default is `text`
+- `encoding` **str** - The encoding of the input file. Default is `utf-8`
+
+### update_output_storage
+
+The section defines a secondary storage location for running incremental indexing, to preserve your original outputs.
+
+#### Fields
+
+- `type` **file|memory|blob|cosmosdb** - The storage type to use. Default=`file`
+- `encoding`**str** - The encoding to use for file storage.
+- `base_dir` **str** - The base directory to write output artifacts to, relative to the root.
+- `connection_string` **str** - (blob/cosmosdb only) The Azure Storage connection string.
+- `container_name` **str** - (blob/cosmosdb only) The Azure Storage container name.
+- `account_url` **str** - (blob only) The storage account blob URL to use.
+- `database_name` **str** - (cosmosdb only) The database name to use.
+- `type` **text|csv|json** - The type of input data to load. Default is `text`
+- `encoding` **str** - The encoding of the input file. Default is `utf-8`
+
+### cache
+
+This section controls the cache mechanism used by the pipeline. This is used to cache LLM invocation results for faster performance when re-running the indexing process.
+
+#### Fields
+
+- `type` **json|memory|none** - The storage type to use. Default=`json`
+- `storage` **StorageConfig**
+  - `type` **file|memory|blob|cosmosdb** - The storage type to use. Default=`file`
+  - `encoding`**str** - The encoding to use for file storage.
+  - `base_dir` **str** - The base directory to write output artifacts to, relative to the root.
+  - `connection_string` **str** - (blob/cosmosdb only) The Azure Storage connection string.
+  - `container_name` **str** - (blob/cosmosdb only) The Azure Storage container name.
+  - `account_url` **str** - (blob only) The storage account blob URL to use.
+  - `database_name` **str** - (cosmosdb only) The database name to use.
+
+### reporting
+
+This section controls the reporting mechanism used by the pipeline, for common events and error messages. The default is to write reports to a file in the output directory. However, you can also choose to write reports to an Azure Blob Storage container.
+
+#### Fields
+
+- `type` **file|blob** - The reporting type to use. Default=`file`
+- `base_dir` **str** - The base directory to write reports to, relative to the root.
+- `connection_string` **str** - (blob only) The Azure Storage connection string.
+- `container_name` **str** - (blob only) The Azure Storage container name.
+- `account_url` **str** - The storage account blob URL to use.
+
+### vector_store
+
+Where to put all vectors for the system. Configured for lancedb by default. This is a dict, with the key used to identify individual store parameters (e.g., for text embedding).
+
+#### Fields
+
+- `type` **lancedb|azure_ai_search|cosmosdb** - Type of vector store. Default=`lancedb`
+- `db_uri` **str** (lancedb only) - The database uri. Default=`storage.base_dir/lancedb`
+- `url` **str** (blob/cosmosdb only) - Database / AI Search to be used.
+- `api_key` **str** (optional - AI Search only) - The AI Search api key to use.
+- `audience` **str** (AI Search only) - Audience for managed identity token if managed identity authentication is used.
+- `connection_string` **str** - (cosmosdb only) The Azure Storage connection string.
+- `database_name` **str** - (cosmosdb only) Name of the database.
+
+- `index_schema` **dict[str, dict[str, str]]** (optional) - Enables customization for each of your embeddings.
+  - `<supported_embedding>`:
+    - `index_name` **str**: (optional) - Name for the specific embedding index table.
+    - `id_field` **str**: (optional) - Field name to be used as id. Default=`id`
+    - `vector_field` **str**: (optional) - Field name to be used as vector. Default=`vector`
+    - `vector_size` **int**: (optional) - Vector size for the embeddings. Default=`3072`
+
+The supported embeddings are:
+
+- `text_unit_text`
+- `entity_description`
+- `community_full_content`
+
+For example:
+
+```yaml
+vector_store:
+  type: lancedb
+  db_uri: output/lancedb
+  index_schema:
+    text_unit_text:
+      index_name: "text-unit-embeddings"
+      id_field: "id_custom"
+      vector_field: "vector_custom"
+      vector_size: 3072
+    entity_description:
+      id_field: "id_custom"
+```
+
+## Workflow Configurations
+
+These settings control each individual workflow as they execute.
+
+### workflows
+
+**list[str]** - This is a list of workflow names to run, in order. GraphRAG has built-in pipelines to configure this, but you can run exactly and only what you want by specifying the list here. Useful if you have done part of the processing yourself.
+
+### embed_text
+
+By default, the GraphRAG indexer will only export embeddings required for our query methods. However, the model has embeddings defined for all plaintext fields, and these can be customized by setting the `target` and `names` fields.
+
+Supported embeddings names are:
+
+- `text_unit_text`
+- `entity_description`
+- `community_full_content`
+
+#### Fields
+
+- `embedding_model_id` **str** - Name of the model definition to use for text embedding.
+- `model_instance_name` **str** - Name of the model singleton instance. Default is "text_embedding". This primarily affects the cache storage partitioning.
+- `batch_size` **int** - The maximum batch size to use.
+- `batch_max_tokens` **int** - The maximum batch # of tokens.
+- `names` **list[str]** - List of the embeddings names to run (must be in supported list).
+
+### extract_graph
+
+Tune the language model-based graph extraction process.
+
+#### Fields
+
+- `completion_model_id` **str** - Name of the model definition to use for API calls.
+- `model_instance_name` **str** - Name of the model singleton instance. Default is "extract_graph". This primarily affects the cache storage partitioning.
+- `prompt` **str** - The prompt file to use.
+- `entity_types` **list[str]** - The entity types to identify.
+- `max_gleanings` **int** - The maximum number of gleaning cycles to use.
+
+### summarize_descriptions
+
+#### Fields
+
+- `completion_model_id` **str** - Name of the model definition to use for API calls.
+- `model_instance_name` **str** - Name of the model singleton instance. Default is "summarize_descriptions". This primarily affects the cache storage partitioning.
+- `prompt` **str** - The prompt file to use.
+- `max_length` **int** - The maximum number of output tokens per summarization.
+- `max_input_length` **int** - The maximum number of tokens to collect for summarization (this will limit how many descriptions you send to be summarized for a given entity or relationship).
+
+### extract_graph_nlp
+
+Defines settings for NLP-based graph extraction methods.
+
+#### Fields
+
+- `normalize_edge_weights` **bool** - Whether to normalize the edge weights during graph construction. Default=`True`.
+- `concurrent_requests` **int** - The number of threads to use for the extraction process.
+- `async_mode` **asyncio|threaded** - The async mode to use. Either `asyncio` or `threaded`.
+- `text_analyzer` **dict** - Parameters for the NLP model.
+  - `extractor_type` **regex_english|syntactic_parser|cfg** - Default=`regex_english`.
+  - `model_name` **str** - Name of NLP model (for SpaCy-based models)
+  - `max_word_length` **int** - Longest word to allow. Default=`15`.
+  - `word_delimiter` **str** - Delimiter to split words. Default ' '.
+  - `include_named_entities` **bool** - Whether to include named entities in noun phrases. Default=`True`.
+  - `exclude_nouns` **list[str] | None** - List of nouns to exclude. If `None`, we use an internal stopword list.
+  - `exclude_entity_tags` **list[str]** - List of entity tags to ignore.
+  - `exclude_pos_tags` **list[str]** - List of part-of-speech tags to ignore.
+  - `noun_phrase_tags` **list[str]** - List of noun phrase tags to ignore.
+  - `noun_phrase_grammars` **dict[str, str]** - Noun phrase grammars for the model (cfg-only).
+
+### prune_graph
+
+Parameters for manual graph pruning. This can be used to optimize the modularity of your graph clusters, by removing overly-connected or rare nodes.
+
+#### Fields
+
+- `min_node_freq` **int** - The minimum node frequency to allow.
+- `max_node_freq_std` **float | None** - The maximum standard deviation of node frequency to allow.
+- `min_node_degree` **int** - The minimum node degree to allow.
+- `max_node_degree_std` **float | None** - The maximum standard deviation of node degree to allow.
+- `min_edge_weight_pct` **float** - The minimum edge weight percentile to allow.
+- `remove_ego_nodes` **bool** - Remove ego nodes.
+- `lcc_only` **bool** - Only use largest connected component.
+
+### cluster_graph
+
+These are the settings used for Leiden hierarchical clustering of the graph to create communities.
+
+#### Fields
+
+- `max_cluster_size` **int** - The maximum cluster size to export.
+- `use_lcc` **bool** - Whether to only use the largest connected component.
+- `seed` **int** - A randomization seed to provide if consistent run-to-run results are desired. We do provide a default in order to guarantee clustering stability.
+
+### extract_claims
+
+#### Fields
+
+- `enabled` **bool** - Whether to enable claim extraction. Off by default, because claim prompts really need user tuning.
+- `completion_model_id` **str** - Name of the model definition to use for API calls.
+- `model_instance_name` **str** - Name of the model singleton instance. Default is "extract_claims". This primarily affects the cache storage partitioning.
+- `prompt` **str** - The prompt file to use.
+- `description` **str** - Describes the types of claims we want to extract.
+- `max_gleanings` **int** - The maximum number of gleaning cycles to use.
+
+### community_reports
+
+#### Fields
+
+- `completion_model_id` **str** - Name of the model definition to use for API calls.
+- `model_instance_name` **str** - Name of the model singleton instance. Default is "community_reporting". This primarily affects the cache storage partitioning.
+- `graph_prompt` **str | None** - The community report extraction prompt to use for graph-based summarization.
+- `text_prompt` **str | None** - The community report extraction prompt to use for text-based summarization.
+- `max_length` **int** - The maximum number of output tokens per report.
+- `max_input_length` **int** - The maximum number of input tokens to use when generating reports.
+
+### snapshots
+
+#### Fields
+
+- `embeddings` **bool** - Export embeddings snapshots to parquet.
+- `graphml` **bool** - Export graph snapshot to GraphML.
+- `raw_graph` **bool** - Export raw extracted graph before merging.
+
+## Query
+
+### local_search
+
+#### Fields
+
+- `prompt` **str** - The prompt file to use.
+- `completion_model_id` **str** - Name of the model definition to use for Chat Completion calls.
+- `embedding_model_id` **str** - Name of the model definition to use for Embedding calls.
+- `text_unit_prop` **float** - The text unit proportion.
+- `community_prop` **float** - The community proportion.
+- `conversation_history_max_turns` **int** - The conversation history maximum turns.
+- `top_k_entities` **int** - The top k mapped entities.
+- `top_k_relationships` **int** - The top k mapped relations.
+- `max_context_tokens` **int** - The maximum tokens to use building the request context.
+
+### global_search
+
+#### Fields
+
+- `map_prompt` **str** - The global search mapper prompt to use.
+- `reduce_prompt` **str** - The global search reducer to use.
+- `completion_model_id` **str** - Name of the model definition to use for Chat Completion calls.
+- `knowledge_prompt` **str** - The knowledge prompt file to use.
+- `data_max_tokens` **int** - The maximum tokens to use constructing the final response from the reduces responses.
+- `map_max_length` **int** - The maximum length to request for map responses, in words.
+- `reduce_max_length` **int** - The maximum length to request for reduce responses, in words.
+- `dynamic_search_threshold` **int** - Rating threshold in include a community report.
+- `dynamic_search_keep_parent` **bool** - Keep parent community if any of the child communities are relevant.
+- `dynamic_search_num_repeats` **int** - Number of times to rate the same community report.
+- `dynamic_search_use_summary` **bool** - Use community summary instead of full_context.
+- `dynamic_search_max_level` **int** - The maximum level of community hierarchy to consider if none of the processed communities are relevant.
+
+### drift_search
+
+#### Fields
+
+- `prompt` **str** - The prompt file to use.
+- `reduce_prompt` **str** - The reducer prompt file to use.
+- `completion_model_id` **str** - Name of the model definition to use for Chat Completion calls.
+- `embedding_model_id` **str** - Name of the model definition to use for Embedding calls.
+- `data_max_tokens` **int** - The data llm maximum tokens.
+- `reduce_max_tokens` **int** - The maximum tokens for the reduce phase. Only use if a non-o-series model.
+- `reduce_temperature` **float** - The temperature to use for token generation in reduce.
+- `reduce_max_completion_tokens` **int** - The maximum tokens for the reduce phase. Only use for o-series models.
+- `concurrency` **int** - The number of concurrent requests.
+- `drift_k_followups` **int** - The number of top global results to retrieve.
+- `primer_folds` **int** - The number of folds for search priming.
+- `primer_llm_max_tokens` **int** - The maximum number of tokens for the LLM in primer.
+- `n_depth` **int** - The number of drift search steps to take.
+- `local_search_text_unit_prop` **float** - The proportion of search dedicated to text units.
+- `local_search_community_prop` **float** - The proportion of search dedicated to community properties.
+- `local_search_top_k_mapped_entities` **int** - The number of top K entities to map during local search.
+- `local_search_top_k_relationships` **int** - The number of top K relationships to map during local search.
+- `local_search_max_data_tokens` **int** - The maximum context size in tokens for local search.
+- `local_search_temperature` **float** - The temperature to use for token generation in local search.
+- `local_search_top_p` **float** - The top-p value to use for token generation in local search.
+- `local_search_n` **int** - The number of completions to generate in local search.
+- `local_search_llm_max_gen_tokens` **int** - The maximum number of generated tokens for the LLM in local search. Only use if a non-o-series model.
+- `local_search_llm_max_gen_completion_tokens` **int** - The maximum number of generated tokens for the LLM in local search. Only use for o-series models.
+
+### basic_search
+
+#### Fields
+
+- `prompt` **str** - The prompt file to use.
+- `completion_model_id` **str** - Name of the model definition to use for Chat Completion calls.
+- `embedding_model_id` **str** - Name of the model definition to use for Embedding calls.
+- `k` **int** - Number of text units to retrieve from the vector store for context building.
+- `max_context_tokens` **int** - The maximum context size to create, in tokens.

docs/data/operation_dulce/ABOUT.md

@@ -0,0 +1,3 @@
+# About
+
+This document (Operation Dulce) is an AI-generated science fiction novella, included here for the purposes of integration testing.

docs/data/operation_dulce/Operation Dulce v2 1 1.md

@@ -0,0 +1,970 @@
+# Operation: Dulce
+
+## Chapter 1
+
+The thrumming of monitors cast a stark contrast to the rigid silence enveloping the group. Agent Alex Mercer, unfailingly determined on paper, seemed dwarfed by the enormity of the sterile briefing room where Paranormal Military Squad's elite convened. With dulled eyes, he scanned the projectors outlining their impending odyssey into Operation: Dulce.
+
+“I assume, Agent Mercer, you’re not having second thoughts?” It was Taylor Cruz’s voice, laced with an edge that demanded attention.
+
+Alex flickered a strained smile, still thumbing his folder's corner. "Of course not, Agent Cruz. Just trying to soak in all the details." The compliance in his tone was unsettling, even to himself.
+
+Jordan Hayes, perched on the opposite side of the table, narrowed their eyes but offered a supportive nod. "Details are imperative. We’ll need your clear-headedness down there, Mercer."
+
+A comfortable silence, the kind that threaded between veterans of shared secrets, lingered briefly before Sam Rivera, never one to submit to quiet, added, "I’ve combed through the last transmission logs. If anyone can make sense of the anomalies, it’s going to be the two of you."
+
+Taylor snorted dismissively. “Focus, people. We have protocols for a reason. Speculation is counter-productive.” The words 'counter-productive' seemed to hang in the air, a tacit reprimand directed at Alex.
+
+Feeling the weight of his compliance conflicting with his natural inclination to leave no stone unturned, Alex straightened in his seat. "I agree, Agent Cruz. Protocol is paramount," he said, meeting Taylor's steely gaze. It was an affirmation, but beneath it lay layers of unspoken complexities that would undoubtedly unwind with time.
+
+Alex's submission, though seemingly complete, didn't escape Jordan, who tilted their head ever so slightly, their eyes revealing a spark of understanding. They knew well enough the struggle of aligning personal convictions with overarching missions. As everyone began to collect their binders and prepare for departure, a quiet resolve took form within Alex, galvanized by the groundwork laid by their interactions. He may have spoken in compliance, but his determination had merely taken a subtler form — one that wouldn't surrender so easily to the forthcoming shadows.
+
+\*
+
+Dr. Jordan Hayes shuffled a stack of papers, their eyes revealing a tinge of skepticism at Taylor Cruz's authoritarian performance. _Protocols_, Jordan thought, _are just the framework, the true challenges we're about to face lie well beyond the boundaries of any protocol._ They cleared their throat before speaking, tone cautious yet firm, "Let's remember, the unknown variables exceed the known. We should remain adaptive."
+
+A murmur of agreement echoed from Sam Rivera, who leaned forward, lacing their fingers together as if weaving a digital framework in the air before them, "Exactly, adaptability could be the key to interpreting the signal distortions and system malfunctions. We shouldn't discount the… erratic."
+
+Their words hung like an electric charge in the room, challenging Taylor's position with an inherent truth. Cruz’s jaw tightened almost imperceptibly, but the agent masked it with a small nod, conceding to the omnipresent threat of the unpredictable. 
+
+Alex glanced at Jordan, who never looked back, their gaze fixed instead on a distant point, as if envisioning the immense dark corridors they were soon to navigate in Dulce. Jordan was not one to embrace fantastical theories, but the air of cautious calculation betrayed a mind bracing for confrontation with the inexplicable, an internal battle between the evidence of their research and the calculating skepticism that kept them alive in their field.
+
+The meeting adjourned with no further comments, the team members quietly retreading the paths to their personal preparations. Alex, trailing slightly behind, observed the others. _The cautious reserve Jordan wears like armor doesn't fool me_, he thought, _their analytical mind sees the patterns I do. And that's worth more than protocol. That's the connection we need to survive this._
+
+As the agents dispersed into the labyrinth of the facility, lost in their thoughts and preparations, the base's halogen lights flickered, a brief and unnoticed harbingers of the darkness to come.
+
+\*
+
+A deserted corridor inside the facility stretched before Taylor Cruz, each footstep rhythmic and precise. Cruz, ambitious and meticulous, eyed the troops passing by with a sardonic tilt of the lips. Obedience—it was as much a tool as any weapon in the arsenal, and Cruz wielded it masterfully. To them, it was another step toward unfettered power within the dark bowels of the military complex.
+
+Inside a secluded equipment bay, Cruz began checking over gear with mechanical efficiency. They traced fingers over the sleek surface of an encrypted radio transmitter. "If protocols are maintained," said Cruz aloud, rehearsing the speech for their subordinates, "not only will we re-establish a line of communication with Dulce, but we shall also illuminate the darkest secrets it conceals."
+
+Agent Hayes appeared in the doorway, arms crossed and a knowing glint in their eyes. "You do understand," Jordan began, the words measured and probing, "that once we're in the depths, rank gives way to survival instincts. It's not about commands—it's empowerment through trust."
+
+The sentiment snagged on Cruz's armor of confidence, probing at the insecurities festering beneath. Taylor offered a brief nod, perhaps too curt, but enough to acknowledge Jordan's point without yielding ground. "Trust," Cruz mused, "or the illusion thereof, is just as potent."
+
+Silence claimed the space between them, steeped in the reality of the unknown dangers lurking in the shadows of the mission. Cruz diligently returned to the equipment, the act a clear dismissal.
+
+Not much later, Cruz stood alone, the hollow echo of the bay a stark reminder of the isolation that power often wrought. With each checked box, their resolve steeled further, a silent vow to usher their team through the abyss—whatever it might hold—and emerge enshrined in the respect they so deeply craved.
+
+## Chapter 2
+
+Sam Rivera sat alone in a cramped office, the hum of a dozen servers murmuring a digital lullaby in the background. Surrounded by the glow of multiple screens, their eyes danced across lines of code and intercepted comm signals from Dulce — a kaleidoscope of data that their curious and isolated mind hungered to decrypt.
+
+To an outsider, it might have looked like obsession, this fervent quest for answers. But to Sam, it was a dance — a give and take with the mysteries of the universe. Their fingers paused over the keyboard as they leaned back in the chair, whispering to thin air, "What secrets are you hiding from us?"
+
+The stillness of the room broke with the unexpected arrival of Alex Mercer, whose encroaching shadow loomed over Sam's workspace. The cybersecurity expert craned their neck upwards, met by the ever-so-slight furrow in Alex's brow. "Got a minute, Rivera?"
+
+"Always," Sam said, a smile surfacing as they swiveled to face their mentor more directly. _He has that look — like something's not sitting right with him,_ they noted inwardly.
+
+Alex hesitated, weighing his words carefully. "Our tech is top-tier, but the silence from Dulce... It's not just technology that will see us through, it's intuition and... trust." His gaze pierced through the digital haze, trying to instill something more profound than advice.
+
+Sam regarded Alex for a moment, the sincerity in his voice resonating with their own unspoken desire to prove their worth. "Intuition," they mirrored thoughtfully. "I guess sometimes the numbers don't have all the answers."
+
+Their shared silence held a newfound understanding, a recognition that between the ones and zeros, it was their combined human insights that might prevail against the impossible. As Alex turned to leave, Sam's eyes drifted back to the screens, now seeing them not as barriers to isolate behind, but as windows into the vast and enigmatic challenge that awaited their team.
+
+Outside the office, the persistent buzz of activity in the facility belied the unease that gripped its inhabitants. A restlessness that nibbled on the edges of reality, as though forewarning of the threshold they were soon to cross — from the known into the realm of cosmic secrets and silent threats.
+
+\*
+
+Shadows played against the walls of the cramped underground meeting room, where Alex Mercer stood gazing at the concealed elevator that would deliver them into the bowels of Dulce base. The air was thick, every breath laced with the weight of impending confrontation, the kind one feels when stepping into a legend. Though armed with an array of advanced weaponry and gear, there was an unshakeable sense that they were delving into a conflict where the physical might be of little consequence.
+
+"I know what you're thinking," Jordan Hayes remarked, approaching Mercer. Their voice was low, a blend of confidence and hidden apprehension. "This feels like more than a rescue or reconnaissance mission, doesn't it?"
+
+Alex turned, his features a mask of uneasy resolve. "It's like we're being pulled into someone else’s game. Not just observers or participants, but... pawns."
+
+Jordan gave a short nod, their analytical mind colliding with the uncertain dynamics of this operation. "I've felt that way since the briefing. Like there's a layer we’re not seeing. And yet, we have no choice but to play along." Their eyes locked with Alex's, silently exchanging a vow to remain vigilant.
+
+"You two need to cut the philosophical chatter. We have positions to secure," Taylor Cruz interjected sharply, stepping into their exchange. The authority in Taylor's voice brooked no argument; it was their way of pulling everyone back to the now.
+
+Alex's response was measured, more assertive than moments ago. "Acknowledged, Agent Cruz," he replied, his voice steadier, mirroring the transformation brewing within. He gripped his rifle with a newfound firmness. "Let's proceed."
+
+As they congregated at the elevator, a tension palpable, Sam Rivera piped in with a tone of balanced levity, "Hope everyone’s brought their good luck charms. Something tells me we’re going to need all the help we can get."
+
+Their laughter served as a brief respite from the gravity of their mission, a shared moment that reinforced their common humanity amidst the unknowable. Then, as one, they stepped into the elevator. The doors closed with a silent hiss, and they descended into the darkness together, aware that when they returned, if they returned, none of them would be the same.
+
+\*
+
+The sense of foreboding hung heavier than the darkness that the artificial lights of the elevator shaft failed to fully penetrate. The team was descending into the earth, carrying with them not only the weight of their equipment but also the silent pressure of the invisible war they were about to fight—a war that seemed to edge away from physicality and into the unnervingly psychological.
+
+As they descended, Dr. Jordan Hayes couldn't help but muse over the layers of data that could wait below, now almost longing for the comfort of empirical evidence. _To think that this reluctance to accept other possibilities may have been my biggest blind spot,_ Jordan contemplated, feeling the hard shell of skepticism begin to crack.
+
+Alex caught Jordan's reflective gaze and leaned in, his voice barely a murmur over the hum of the elevator. "Once we're down there, keep that analytical edge sharp. You see through the mazes of the unexplained better than anyone."
+
+The compliment was unexpected and weighed differently than praise from others. This was an acknowledgment from someone who stood on the front lines of the unknown with eyes wide open. "Thank you, Alex," Jordan said, the words carrying a trace of newfound assertiveness. "You can count on me."
+
+The exchange was cut short by a shudder that ran through the elevator, subtle, but enough to make them instinctively hold their breaths. It wasn't the mechanical stutter of old gears but a vibration that seemed to emanate from the very walls of the shaft—a whisper of something that defied natural explanation.
+
+Cruz was the first to react, all business despite the shadow that crossed their expression. "Systems check. Now," they barked out, masking the moment of disquiet with swift command.
+
+Every agent checked their gear, sending confirmation signals through their comms, creating a chorus of electronic beeps that promised readiness. But there was an unspoken question among them: was their technology, their weaponry, their protocols sufficient for what awaited them or merely a fragile comfort?
+
+Against the gravity of the silence that was once again closing in, Sam's voice crackled through, only half-jest. "I'd laugh if we run into Martians playing poker down there—just to lighten the mood, you know?"
+
+Despite—or perhaps because of—the oddity of the moment, this elicited a round of chuckles, an audible release of tension that ran counterpoint to the undercurrent of anxiety coursing through the team.
+
+As the elevator came to a halting, eerie calm at the sub-level, the group stepped off, finding themselves at the threshold of Dulce's mysterious halls. They stood in a tight pack, sharing a cautious glance before fanning out into the unknown, each one acutely aware that the truth was inevitably intertwined with danger.
+
+Into the depths of Dulce, the team advanced, their silence now a shared testament to the camaraderie born of facing the abyss together—and the steel resolve to uncover whatever horrors lay hidden in its shadows.
+
+\*
+
+The weight of the thick metal door closing behind them reverberated through the concrete hallway, marking the final threshold between the familiar world above and the strangeness that lay beneath. Dulce base, a name that had been whispered in the wind-blown deserts above and in the shadowed corners of conspiracy forums, now a tangible cold reality that they could touch — and that touched them back with a chill.
+
+Like lambs led to an altar of alien deities, so did Agents Alex Mercer, Jordan Hayes, Taylor Cruz, and Sam Rivera proceed, their movements measured, their senses heightened. The air was still, almost respectful of the gravity of their presence. Their torch beams sliced through the darkness, uncovering steel doors with warnings that spoke of top secrets and mortal dangers.
+
+Taylor Cruz, stepping firmly into the role of de facto leader, set a brisk pace. "Eyes sharp, people. Comms check, every thirty seconds," Taylor ordered, their voice echoing slightly before being swallowed by the surrounding silence.
+
+Sam, fiddling with a handheld device aimed at detecting electronic anomalies, offered a murmured "Copy that," their usual buoyancy dimmed by the oppressive atmosphere.
+
+It was Jordan Hayes who paused at an innocuous looking panel, nondescript amongst the gauntlet of secured doorways. "Mercer, Rivera, come see this," Jordan’s voice was marked with a rare hint of urgency.
+
+Alex joined Jordan's side, examining the panel which, at a mere glance, seemed just another part of the base's infrastructure. Yet, to the trained eye, it appeared out of place—a facade.
+
+Jordan explained their reasoning as Sam approached, instinctively understanding the significance of what lay beneath, "This panel is a recent addition — covering something they didn't want found."
+
+Before Alex could respond, the soft whir of an approaching drone cut through their muffled exchange. Taylor had looped back upon hearing the commotion. "Explanations later. We can't afford to attract..." Cruz’s voice trailed off as the small airborne device came into view, its sensors locked onto the group.
+
+Sam was the first to react, their tech-savvy mind already steps ahead. "I've got this," they declared, fingers flying over the controls of their own gadgetry to ward off the impending threat.
+
+The drone lingered, its scan seeming more curious than hostile. But within moments, courtesy of Sam's interference, the little sentinel drifted away, retreating into the shadows as if accepting a silent truce. The crew exhaled, a moment of collective relief palpable in the air.
+
+Cruz squared their shoulders, clearly ruffled but not conceding any ground. "Move out," they directed, a hint more forceful than before. "And Rivera, keep that trick handy."
+
+The team pressed onward, the quiet now filled with the soft beeps of regular comms checks, their pace undeterred by the confrontation. Yet, every agent held a renewed sense of wariness, their trust in one another deepening with the knowledge that the base—its technology, its secrets—was alive in a way they hadn't fully anticipated.
+
+As they converged upon a central hub, the imposing doors to the mainframe room stood ajar — an invitation or a trap, neither option comforting. Without a word, they fortified their resolve and stepped through the threshold, where the dim glow of operational LED lights and the distant hum of machinery hinted at Dulce’s still-beating heart.
+
+Solemnly, yet unmistakably together, they moved deeper into the heart of the enigma, ready to unmask the lifeforce of Dulce base or confront whatever existential threat lay in wait. It was in that unwavering march towards the unknown that their destinies were forever cemented to the legacy of Operation: Dulce.
+
+## Chapter 3
+
+The thrumming of monitors cast a stark contrast to the rigid silence enveloping the group. Agent Alex Mercer, unfailingly determined on paper, seemed dwarfed by the enormity of the sterile briefing room where Paranormal Military Squad's elite convened. With dulled eyes, he scanned the projectors outlining their impending odyssey into Operation: Dulce.
+
+\*
+
+The cooling vents hummed in a monotonous drone, but it was the crackle of the comms system coming to life that cut through the lab’s tension. Dr. Jordan Hayes hovered over a table arrayed with alien technology, their fingers delicately probing the enigmatic circuitry retrieved from the crash site. Agent Alex Mercer watched, admiration blooming in silent solidarity for Jordan's deft touch and unspoken drive.
+
+Jordan, always composed, only allowed the faintest furrow of concentration to mar their brow. "What we understand about physics..." they muttered, trailing off as they realigned a translucent component. The device emitted a low pulse, causing Jordan to still. "Could be fundamentally changed by this."
+
+A calculated risk—that's what this was. And for a person of science, a gamble was worth the potential paradigm shift.
+
+"I’ve been thinking," Alex started, his eyes still fixed on the immediately tangible mystery before them. "About what’s at stake here. Not the mission parameters, but what this means for us—humanity."
+
+Jordan glanced up, meeting his eyes just long enough to convey the shared enormity of their situation; the career-defining glory and existential dread entwined. "The quest for understanding always comes at a price. We're standing on the precipice of knowledge that could either elevate us or condemn us."
+
+The charged air between them spiked as Taylor Cruz’s brusque tones sliced through their reverie. "Hayes, Mercer, this isn't philosophy hour. Focus on the task. We need actionable intel, not daydreams."
+
+With a sound of restrained acknowledgment, Jordan returned their gaze to the device, while Alex clenched his jaw, the buzz of frustration dull against the backdrop of Taylor's authoritarian certainty. It was this competitive undercurrent that kept him alert, the sense that his and Jordan's shared commitment to discovery was an unspoken rebellion against Cruz's narrowing vision of control and order.
+
+Then Taylor did something unexpected. They paused beside Jordan and, for a moment, observed the device with something akin to reverence. “If this tech can be understood..." Taylor said, their voice quieter, "It could change the game for us. For all of us.”
+
+The underlying dismissal earlier seemed to falter, replaced by a glimpse of reluctant respect for the gravity of what lay in their hands. Jordan looked up, and for a fleeting heartbeat, their eyes locked with Taylor's, a wordless clash of wills softening into an uneasy truce.
+
+It was a small transformation, barely perceptible, but one that Alex noted with an inward nod. They had all been brought here by different paths and for different reasons. Yet, beneath the veneer of duty, the enticement of the vast unknown pulled them inexorably together, coalescing their distinct desires into a shared pulse of anticipation.
+
+Marshaled back to the moment by the blink of lights and whir of machinery, they refocused their efforts, each movement sharpened by the knowledge that beyond understanding the unearthly artifacts, they might be piecing together the future of their species.
+
+\*
+
+Amidst the sterility of the briefing room, the liminal space between the facts laid out and the hidden truths, sat Sam Rivera, his demeanor an artful balance of focus and a casual disguise of his razor-sharp talent with technology. Across from him, Alex Mercer lingered in thought, the mental cogs turning as each file on Dulce stirred more than curiosity—it beckoned to a past both honored and burdensome.
+
+"You've been quiet, Sam," Alex noted, catching the younger man's contemplative gaze. "Your take on these signal inconsistencies?"
+
+There was a respect in Alex's tone, though a respectful distance remained—a gulf of experience and a hint of protective mentorship that stood between them. Sam nodded, recognizing the space afforded to him, and he couldn't help but feel the weight of expectation pressing upon his shoulders. It wasn't just the mission that was immense, it was the trust being placed in him.
+
+"The patterns are... off," Sam admitted, hesitant but driven. "If I'm right, what we're looking at isn't random—it's a structured anomaly. We need to be ready for anything." 
+
+Alex's eyes brightened with a subtle approval that crossed the distance like a silent nod. "Good. Keen eyes will keep us ahead—or at least not blindsided," he said, affirming the belief that inscribed Sam's role as more than the tech personnel—he was to be a guiding intellect in the heart of uncertainty.
+
+Their exchange was cut short by Taylor Cruz's abrupt arrival, his gait brimming with a robust confidence that veiled the sharp undercurrents of his striving nature. "Time to gear up. Dulce waits for no one," Taylor announced, his voice carrying an iron resolve that knew the costs of hesitation—though whether the cost was calculated in human or career terms was an ambiguity he wore like a badge of honor.
+
+As Sam and Alex nodded in unison, the icy chasm of hierarchy and cryptic protocols seemed momentarily to bridge over with an understanding—this mission was convergence, a nexus point that would challenge each of their motives and strength.
+
+They filed out of the briefing room, their footsteps synchronized, a rhythm that spoke volumes of the unknown cadence they would soon march to within the base's veins. For Alex Mercer, the link with Sam Rivera, though distant, was now poised with a mutuality ready to be tested; for Taylor Cruz, the initiative pulsed like a heartbeat, anticipation thinly veiled behind a mask of duty.
+
+In the midst of the descent, they were each alone yet irrevocably joined, stepping closer towards the volatile embrace of Operation: Dulce.
+
+## Chapter 4
+
+The corridors of the Dulce military base were as silent as a tomb and twice as chilling. Alex Mercer walked with a surety that belied his bubbling undercurrents of doubt. The briefing had been definitive, sturdy pillars of facts and protocols, yet as he ventured deeper, the ominous atmosphere gnawed at him—a stark reminder of how much remained unknown.
+
+Jordan Hayes trailed a few steps behind, their detached exterior breaking for a moment as they caught up to Alex. "What's on your mind?" Jordan asked, their astuteness cutting through the unspoken tension.
+
+Alex glanced back at them. This place was a puzzle, a treacherous labyrinth where the walls whispered secrets, and among them, he sensed a call to question, to challenge the narrative they'd been sold. "The silence here... It's almost as if the base is waiting for something—or someone."
+
+"Just stay sharp, Mercer," Jordan cautioned, yet their eyes lingered on the quietude around them, conceiving the same shadow of doubt that unsettled Alex.
+
+Before they could delve into further discussion, the distinctive click of a safety catch echoed in the hollow space. Both agents turned to find Taylor Cruz standing resolute, primed for combat. Taylor's gaze was scrutinizing and cold, a stark contrast to the growing unease that smoldered silently amongst the rest.
+
+"Chatter is a liability," Taylor snapped, with a commanding flair that bordered on tyrannical. "We move forward, eyes open, mouths shut."
+
+Alex felt the tight grip of compliance strangle his gut, a lesson learned under the hard tutelage of rank and order. But here, in the bowels of Dulce, those instincts began to wane, the imperative to adhere now conflicting with the pressing urgency to confront the shadows they were enmeshed in.
+
+Then, unexpectedly, the lights flickered, a power fluctuation—or a sign? Alex's hand instinctively went to his sidearm, his mindset shifting from soldier to skeptic. The base, with its unyielding coldness, had just given them their first nudge into the realm of the speculative, an invitation to peel back the veneer of reality.
+
+"We should consider all possibilities," Alex murmured, more to himself than the others, his voice a barely audible breath against the sterile air of the complex.
+
+Taylor's posture stiffened at the challenge, yet their response was uncharacteristically reserved, notable in its lack of rebuke. "Agreed. For now, keep moving. But stay vigilant."
+
+A surprise—an echo of agreement from the last person Alex expected it from. And there it was, the glimpse of a wrinkle in the unyielding fabric of command, a hint that perhaps they were all starting to sense the strangeness that permeated this place.
+
+Progressing with determined steps, the trio moved deeper, silently acknowledging the evolution of their predicament. It was a small yet transformative concession to the unknown forces at play, an acknowledgment from each agent that, despite their disparate goals and ideals, the true nature of the Dulce base was an enigma that would forge new paths through their convictions.
+
+As they reached the central communications hub, the truth that awaited them lurked in the shadows, its eyes unseen but felt by all. The walls didn't just whisper now; they spoke in tones only the brave—or the foolish—would dare to listen to.
+
+\*
+
+The subterranean silence of Dulce was an oppressive entity of its own, wrapping the team in a cloak of uneasiness as they pressed on through the dimly lit corridor. Jordan Hayes found themselves contemplating the ramifications of each step taken into this suspended world, where the sterile air seemed to mock the gravity of their predicament. The closer they got to the communication hub, the more Jordan's mind wandered toward the realm of the inexplicable.
+
+Beside Jordan, Alex Mercer moved forward with deliberation, his gaze scanning the heavy utility doors they passed—one of which was partially ajar, beckoning them with its darkness. "After you, Dr. Hayes," Alex said, gesturing toward the mysterious opening. A hint of shared understanding passed between them; knowledge was the guiding star of this mission as much as confrontation or recovery.
+
+Jordan peered inside, the beam from their flashlight slicing through the obscurity. The room beyond was a chaotic cascade of papers, overturned furniture, and the particular kind of disorder born from hasty evacuation—or something far more sinister.
+
+"It's like they vanished in the middle of something urgent," Alex murmured, his voice tight with a mix of concern and anticipation. He began to sift through the scattered reports, each page a potential clue to the enigmatic silence that shrouded Dulce.
+
+Behind them, Taylor watched with a disciplined patience, their authority the foundation upon which the operation was built. Their voice cut into the stillness, a reminder of their presence, "Time is not our ally here."
+
+Drawing back from momentary distraction, Jordan acknowledged the wisdom in Taylor's words, yet could feel the shift in their stance—from skeptical, reserved analyst, to a proactive agent within the narrative. "You're right; these documents may hold critical insights. Let's collect what we can and analyze them properly."
+
+From the darkened hollows of the room, shadows seemed to cast subtle judgment as Alex and Jordan worked together with heightened urgency. Taylor, for once, didn't intervene but instead surveyed the entrance, their mind anticipating the unknown variables that lay ahead.
+
+Unexpectedly, a soft hiss emanated from a neglected terminal on the desk. Jordan's head snapped up, their heart rate accelerating at the potential ramifications. Without a word, they moved to the machine, hands driven by the newfound conviction that knowledge was more than power—it was survival.
+
+As Jordan began to extract what data they could from the terminal, the first comprehensible communication from the depths of Dulce in far too long crackled through: an automated distress marker, looping endlessly without further context. It was a revelation, one that reverberated through the group, confirming their fears and igniting an even greater need to press on.
+
+Watching Jordan's dogged determination, Alex witnessed the minor transformation in his colleague unfold—a shift from doubt to action, a sliver of belief in the possibilities beyond their rational understanding. This forge of resolve amidst the alien echoes of Dulce not only bonded them closer as a team but compelled them forward with a sharpened edge of responsibility to the truth, wherever it would lead.
+
+As they collected their findings and regrouped, the base around them imperceptibly changed, the air charged with the vibration of secrets poised on the brink of revelation. And in that charged silence, the group moved on, each now carrying pieces of a puzzle that would soon converge into a picture of galactic significance.
+
+\*
+
+In the chill of the cramped server room, the hum of machinery was the backbone to a symphony of data streams coursing through the air. Dr. Jordan Hayes, nerves alight with the mission's mounting unknowns, patched into the last known coordinates of the unsent distress broadcast they had uncovered. They were so close to the core now – to the truth behind the blackout – it was almost tangible.
+
+Beside them stood Agent Alex Mercer, ever the soldier, yet with eyes that betrayed an intellect craving to understand the murk beneath the surface. "Any progress, Dr. Hayes?" Alex queried, his voice betraying a subtle urgency.
+
+"Getting there," Jordan replied, fingers dancing across the keyboard. "Whoever sent this was cut off mid-transmission. It's as if Dulce itself swallowed the message whole."
+
+Taylor Cruz closed in, their frame casting a long shadow over the duo, evoking an almost palpable wall between them and the forward momentum of their mission. "Time is against us," Taylor intoned, more statement than threat. "What we uncover here determines our next course of action."
+
+Alex acknowledged Taylor with a brisk nod, his stance firm. Yet inwardly, the tightening grip he felt from Taylor's words couldn't throttle the swell of his own investigative instinct. His soldier's obedience had begun to war with the advocate's zeal for unveiling the dark heart of Dulce's secrets. 
+
+And then, the unexpected occurred. The screens flashed in unison, spilling a discordant stream of symbols and images that defied immediate analysis. Jordan's breath caught – this was the response they had been fishing for, an alien communication protocol resonating just at the edge of human comprehension.
+
+Each member of the team felt it: a shift in the room’s very atmosphere, like a veil being drawn from their perception. Alex and Jordan stood still, absorbed in the bewilderment of contact, while Taylor, despite their authority, hesitated – a minor betrayal that unease was creeping into even their disciplined heart.
+
+"Thoughts, Rivera?" Taylor rallied, seeking the counsel of Sam Rivera, whose eyes were wide with exhilaration.
+
+Sam stepped forward, breaking the spell of stillness. "It's like nothing I've ever seen before, but I think I can bridge our systems to communicate," they declared, a wisp of optimism braiding their voice. They set about adapting their gear to transmute the foreign signals into something the team could dissect, their actions a testament to the mentorship and belief instilled in them by Mercer and the team.
+
+Taylor observed them, a cold calculation behind their facade, as they weighed the worth of this anomaly. It was a crossroad that potentially led to either monumental breakthrough or unprecedented catastrophe. "Once you've established a line, document everything. We can't afford to miss any detail," Taylor ordered, the words sharper than intended.
+
+The connection was made, and with trembling anticipation, the team listened as the first garbled outputs began to emerge, their very essence promising insights that could alter the course of history. It was an enigmatic dance with the unknown, the pulse of Dulce no longer just a place, but a herald to an alien register the team had yet to decipher.
+
+Together, they stood at the precipice of understanding, where the faint glow of their monitors cast more than just light – it cast the shadow of burgeoning transformation. It was in this moment, in the grasp of an extraterrestrial tongue, that the team, bound by a hunger for knowledge and the raw edge of survival, found their mission reframed from a search for answers to the articulation of a question humankind had yet to fully ask.
+
+Silent in their commune with the inexplicable frequency, they realized they were not merely investigators; they had become liaisons on behalf of Earth, interpreters of a cosmic message that could redefine their very existence. The implications loomed large, but now, they would not face them alone – they would face them as a united front, wrought together by the very mysteries that once drove them apart.
+
+## Chapter 5
+
+Dr. Jordan Hayes clutched the edge of the briefing room table, their fingers white-knuckled against the laminate surface, as an array of constellations rotated on the projector—charts and graphs bleeding across the stars. In the dim room, nebulas and dark matter seemed within arm's reach, tangible yet unfathomable. 
+
+Sam Rivera leaned back against the wall, arms crossed, gaze darting between the swirling cosmos and the faces of their companions. A taut line of concentration etched their young features, a mingling of fervent curiosity with the nascent understanding of the high stakes for which they played.
+
+Jordan's voice broke the profound silence. "The patterns in the signal disruptions sync with none other than zenithal star alignments. It's as if... as if these 'meet and greets' were scheduled, predestined by celestial mechanics."
+
+The statement hung heavy, daring the occupants of the room to unravel its implications. Alex Mercer, his prior military resolve momentarily suspended, absorbed the hypothesis with a visible hunger. "It's like we're adhering to an appointment we never knew we had," he murmured, his heart a drumbeat in his chest.
+
+Taylor Cruz snorted—a sound that clattered against the high concepts like a tumbledown shack in a futurist cityscape. Folding their arms, they glanced between the agents, their apprehension clad in the contempt of practicality. "What we need are facts, not mystic conjecture."
+
+Alex pivoted on his heel, facing Taylor squarely, and his voice found its edge of steel. "This isn't mysticism, Cruz. It's a hypothesis based on observed phenomena as unpredictable as the place we're standing in."
+
+Taylor's gaze never wavered, yet the slight twitch at the corner of their mouth belied their taut composure. "If there's a semblance of truth to it, then it's critical intel. But remember, we're not astrologers—we're soldiers and scientists."
+
+Jordan met Taylor’s gaze with a curt nod, accepting the caution even as the crucible of their intellect smoldered with the fervor of cosmic discovery. Their eyes flicked to Sam, whose steady presence and ready tech affirmed a burgeoning dynamic—the makings of a sentinel, standing guard over the threshold of human understanding and cosmic reality.
+
+With the projector casting pallid light over their features, each agent became a silhouette of purpose, shadows pillared against the backdrop of an endless universe. The story they were embroiled in would soon demand they plunge into darkness to retrieve the light of knowledge—a light that could very well redraw the shape of their world.
+
+They left the briefing room with a shared silence, each pondering the vast weave of celestial intent and terrestrial response, sensing that the galactic appointment to which they'd unwittingly RSVP’d was more insistent—and more threatening—than any operation they’d faced before.
+
+\*
+
+As the Paranormal Military Squad team convened in the heart of the Dulce military complex, an air of bristling expectation clung to the walls of the underground sanctum. Alex Mercer’s brow furrowed while watching his companions—Jordan Hayes, diligently setting up their makeshift lab station, and Sam Rivera meticulously checking the communication relays they had restored. Taylor Cruz observed with hawk-like focus, yet to betray the strain that their command posed on them.
+
+The gravity of the mission had shifted, deepened; each member of the team felt its pull, tethered to the understanding that they were now part of a larger narrative—a cosmic play with Earth as a stage and the human race unwitting actors. 
+
+Jordan paused, a tension creeping across their shoulders as they aligned the satellite data with the alien message that had been decoded. "The instructions in this message," Jordan started, the timbre of their voice betraying their usual composure. "They're coordinates and... a warning."
+
+Sam leaned in, their eyes widening behind the glow of their laptop screen. "A warning? Like, ‘stay away from’, or ‘beware of’...?" Their words trailed off, uncertainty a new companion in their lexicon.
+
+Alex exhaled slowly, his mind racing to connect the dots. "It doesn't matter which," he said, decisive yet contemplative. "What matters is we understand intent. Are we being warned out of concern, or are we stumbling upon a threat?"
+
+Cruz’s iron-clad facade momentarily cracked, a fleeting glimpse of vulnerability flashing through their eyes. "We need to know if this entails additional risk to the operation," they said, directing their gaze specifically at Alex. "Mercer, I rely on you to keep the team grounded. No one goes off-course."
+
+Their reminder seemed both a command and a plea—rooted in an understanding that each member of the team now faced the duality of their roles, protectors of earthly secrets and heralds of potentially devastating revelations.
+
+Sam's fingers stilled mid-type, their task forgotten as they absorbed the weight of the unfolding reality. "We're the first line of defense... or detection," they mused half to themselves, a growing sense of agency within the larger play they were cast into.
+
+Jordan returned to the data, more resolute in their actions. The warning, whether cautionary or dire, was a beacon they no longer could ignore; its light casting aside shadows of doubt and igniting a collective purpose within the team.
+
+Alex watched Jordan and Sam, feeling a brotherhood in their shared quest. As Cruz paced, poised on the cusp of decisions that would mark their career and perhaps the fate of many, Alex knew the narrative had changed. They were no longer mere operatives; they had become guardians of a threshold, keepers of a message from a realm beyond stars and stripes. This elevation in their mission could not be shackled by regulations and established protocols—it demanded a new perspective, a new resolve.
+
+Tension threaded through the dialogue of beeps and static as communications with Washington buzzed in the background. The team stood, a portentous air enveloping them. It was clear that the decisions they made in the ensuing hours could redefine humanity's place in the cosmos or condemn them to ignorance and potential peril.
+
+Their connection to the stars solidified, the group moved to address the crystallizing warning, shifting from passive recipients to active participants. Mercer’s latter instincts gained precedence— the team’s mandate had evolved, no longer solely to observe and report but to interact and prepare. A metamorphosis had begun, and Operation: Dulce hummed with the newfound frequency of their daring, a tone set not by the earthly hierarchies but by the pulsing symphony of the universe itself.
+
+\*
+
+The desert night loomed eerily still as echoes of hidden activity reverberated deep beneath the bleak sands of New Mexico. Diverting his gaze from the array of sensors before him, Jordan Hayes allowed a rare breath, deep and anxious. Turning to Alex Mercer's focused silhouette, the nocturnal landscape illuminated softly by makeshift floodlights, Jordan felt the syncopated tempo of apprehension and exhilaration jockey for primacy within.
+
+"The closer we get to unlocking these messages, the more I feel like we're peeling back layers of reality itself," Jordan confided, eyes not leaving the monitors that presented a constellation of data points.
+
+"Yes," Alex replied, his voice steady as he considered the implications of their discovery. "And we have to be ready for whatever we find beneath those layers. Whether it's a breakthrough or a Pandora's Box."
+
+Silence settled between them, broken only by the occasional buzz of communications equipment attempting to bridge terrestrial and extraterrestrial intelligences. Tense moments drifted by, laden with the expectant weight of near breakthrough, when a soft chime signaled an incoming transmission -- a rare sound that set every agent on high alert.
+
+Absent was the voice of Washington or Paranormal Military Squad command. Instead, a rhythmic series of pulses and tones filled the air, deliberately patterned, unmistakably non-human.
+
+Sam Rivera adjusted the sensitivity of the decoding equipment, their hands shaking with anticipation as much as focus. "I have it!" they announced, the signal transforming under their expertise into a sequence of visual symbols on the screen before them.
+
+Their shared excitement was palpable, a kinetic force resonating between the team members as they crowded around the display. 
+
+"What does it say?" Taylor Cruz demanded, the urgency in his tone scraping against the newfound wonderment.
+
+Interpreting the alien syntax required not only decoding but intuition and empathy. The words that emerged upon the screen were at once coherent and enigmatic: "*Voyage. Convergence. Peril.*"
+
+The stark simplicity of the message struck them collectively, a chill breeze wafting through their resolve.
+
+Alex stepped forward, piecing together the cryptic communication with a growing sense of obligation. "It’s a call to action," he deduced, "or possibly a summons."
+
+Jordan's gaze met Alex’s, both understanding that this was no longer an investigation or mere extraction of hidden truths. This was humanity's unwitting enlistment into a galactic dialogue that defied boundaries of nation, creed, or protocol.
+
+Sam's eyes were aglow, not with fear, but with the profound acceptance of inevitability that comes with groundbreaking revelation. Moreover, within Taylor's stern exterior churned the seed of reluctant admiration for the unclassified, the uncharted realms they were approaching.
+
+Together, they accepted the pivot in their mission, readjusting their objectives from exploration to engagement, and from isolation to a communal outreach beyond the stars. As dawn's first light threatened the horizon, it became clear that they were no longer merely operatives of a clandestine governmental faction—they were delegates on behalf of Earth, embarking on a voyage orchestrated by destinies unrelated to the mere geopolitics of their world.
+
+Turning to each other, their silhouettes sketched against the coming dawn, the agents recognized the transformation within and amongst them. They were bound by more than duty—they were intricately woven into the fabric of an unfolding cosmic opera, one in which they had been granted an undeniable role. And as they set course for the coordinates that beckoned them like a distant siren's call, it was with a solemn dedication to not only uncover the mysteries ahead but to navigate the convergence, and the peril, as unified emissaries of a world on the cusp of a broader understanding.
+
+\*
+
+Beneath the hum of the fluorescent lights and the vigilance of silent monitors, Alex Mercer stood with his team in the threshold of the base's command center, their faces etched with the fatigue of hours spent unraveling galactic mysteries. Jordan Hayes broke the stillness with a delicate fusion of disbelief and resolve. "The signal..." they began, their tone deliberate, "it’s evolving. It’s not just sending a message—it’s responding to us."
+
+Taylor Cruz leaned over the console, their eyes narrowing with intrigue and a flicker of unease, studying the alternating patterns on the screen. "Responding? Like it’s alive?" Taylor asked, a question that bordered on the edge of wonder and alarm.
+
+Sam Rivera’s gaze was locked onto their interface, a digital orchestra at their fingertips. "It could be some form of advanced AI. Or something else entirely," they contributed, a note of exhilaration betraying the gravity of the situation.
+
+Alex paced before the terminal, absorbing the enormity of their predicament. Their mission—once rooted in the solid ground of military discipline and covert operations—had transcended into an encounter of unprecedented import. "We need to be cautious," he advised, his voice a low rumble of cautious strategy. "If this signal is intelligent, how we interact with it could dictate the outcome of this entire operation."
+
+Jordan met Alex's gaze with a nod, the weight of the responsibility shared and accepted. "We have protocols for first contact, but nothing for... this," Jordan admitted. The room was gripped with tension, each breath seemingly louder than the last.
+
+Then, with a sudden burst that filled the command center, the signal coalesced into a clear and distinct pattern which replicated and expanded, its complexity revealing the hand—or mind—of an intelligent architect.
+
+Taylor's instinct for command surged forth. "Prepare to record and analyze. Whatever it is, we need to understand it—" But their words were cut short as the signal surged, enveloping the room in a brief, blinding cascade of light.
+
+In that pulse of brilliance, a shared revelation coursed through the team. The signal had become a bridge, an extension of unknown consciousness reaching towards them, testing, communicating, searching. 
+
+Alex stepped back from the light, feeling a profound change unravelling within him. The path forward would not be one of confrontation or conquest, but of connection and comprehension.
+
+Jordan turned to Alex and Taylor, seeing in their faces a reflection of the same metamorphosis taking place within themselves—a movement from observers to participants, from agents to ambassadors.
+
+With a collective breath, the team faced the kaleidoscope of lights. The alien signal, once a harbinger of enigma, was now a catalyst for transformation—a symphony of light and sound that echoed the beginnings of a new relationship between humanity and the alien unknown.
+
+And so, with deliberate steps, Alex Mercer led his team into the luminous fray. Science, protocol, and survival instinct harmonized within them, each member poised on the cusp of a new chapter in human history.
+
+They were no longer merely the instruments of Paranormal Military Squad's will—they were the vanguard of humankind’s first definitive leap into the cosmic community. 
+
+With the last echoes of the signal resonating in the control room, they each embraced the sequencing of the transmission, the dance of extraterrestrial light that now wrote itself into their story. The chapter of Operation: Dulce drew to a close, but the narrative of their destiny had only just begun.
+
+## Chapter 6
+
+\*
+
+The cool darkness of the command center at Dulce base was a stark contrast to the brewing storm outside, where the unforgiving New Mexico desert winds whispered of the hidden truths that lay buried deep beneath its surface. Dr. Jordan Hayes sat, their eyes fixed on the readout, the frenetic dance of symbols and numbers reflecting off their determined face. They were on the cusp of an epiphany, teetering between the widely accepted laws of physics and the promise of a new cosmic paradigm.
+
+Alex Mercer watched from across the room, noting the subtle shifts in Jordan’s posture that belied a developing readiness to embrace the unbelievable. “Find something?” Alex’s question, asked with a blend of curiosity and solidarity, bridged the gap between a command and a genuine query among equals.
+
+Jordan's response was slow, measured against the magnitude of their analysis. “This isn’t random static. It’s a pattern - a repeated sequence phasing in and out but distinctly artificial.” Jordan turned away from the screen, locking eyes with Alex. “This could change everything.”
+
+Sam Rivera leaned in, their eyes alight with the fires of revelation and a quenchless thirst for understanding. “A pattern means intention. Could it be a message?”
+
+A figure emerged from the doorway, casting a long shadow into the room - Taylor Cruz. “Intentions can be friendly, or hostile. We shouldn’t forget that,” said Taylor, bringing a dose of their usual pragmatism into the heart of discovery.
+
+Alex acknowledged Taylor’s caution with a nod, understanding the need to keep their feet grounded even as their spirits soared toward the unknown. “Then let’s be the first to find out which it is."
+
+The team gathered around the monitors, the soft tapping of Jordan's keystrokes now punctuated by the occasional crackle of Sam's radio equipment. The sound was almost ritualistic, a prelude to humanity’s potential first, knowing foray into a larger universe.
+
+Jordan’s fingers paused, suspended in mid-air. The signal had evolved, becoming a beacon that somehow felt less alien and more familiar. It was as if the complexities of their message were unfolding into something more accessible, more terrestrial.
+
+A hushed excitement swept through the room. The transformation suggested an awareness on the part of the unknown senders; a finesse that spoke volumes about their capabilities and perhaps their intentions.
+
+With the growing realization that they were engaging with an intelligence far exceeding their previous understanding, the team prepared to reach back across the cosmic divide. Prepared or not, they were no longer bystanders in this galactic narrative. They were active correspondents in an exchange that transcended galaxies and welcomed them into an expansive, possibly fraught, interstellar conversation.
+
+\*
+
+Inside the cavernous central hub of Dulce military base, Dr. Jordan Hayes stood in near-darkness, surrounded by a nest of cables and monitors that buzzed with silent, cryptic life. Jordan's eyes narrowed to focus on the sequences that danced across the screen—patterns that could unravel the cosmic enigma surrounding them.
+
+Alex Mercer approached with his characteristic stride, a signal of reliability in the chaos. "Status report, Dr. Hayes?" he inquired, his voice low, almost blending into the soundscape of beeping consoles and swirling fans.
+
+"We're on the brink of unravelling the signal's origin," Jordan replied, the weight of implications heavy in their tone. "There's intelligence behind it, a thought process alien to our own."
+
+As if summoned by their analysis, Taylor Cruz approached with authority radiating from every pore. "Understand this, we need to know if it's friend or foe. Don't get wrapped up in the existential—our lives may depend on the answers you provide."
+
+Sam Rivera, their hands adroitly adjusting a device to fine-tune the signal, chimed in with optimism undercut by anxious anticipation. "We're deciphering the comm encryption. Soon, we'll have a channel open—not just listening in, but speaking back."
+
+Alex nodded his understanding, his strategic mind processing the tactical implications while grappling with the more profound humanistic impact. "When we do, we'll tread carefully, communicate with purpose," he reassured the team.
+
+The operation had evolved rapidly, from a stealthy incursion into a clandestine labyrinth to an exchange with an extraterrestrial intellect. Their earlier trepidation transformed into determined focus, as they prepared to extend humanity’s hand into the vast unknown.
+
+An alert on one of the monitor stations snapped the team into alarm. The signal had not simply been waiting—it had been calculating. Now, it reached its crescendo, demanding their attention with a provocative urgency.
+
+Jordan's fingers raced over the keyboard, their eyes simultaneously interpreting data and sharing directives. "It’s a linguistic lock, a test of comprehension. We crack this, we establish dialogue."
+
+Taylor's presence was a beacon of steely resolve. "Then let’s solve it. This is what we trained for—the unknown."
+
+Alex and Sam exchanged a look that telegraphed their shared determination—this was not only the mission they had trained for; it was the mission they had been destined for.
+
+Together, the Paranormal Military Squad team leaned into the challenge, their minds honing in on the complex patterns with a singular goal: to unlock the conversation with an intelligence that had already begun to shift the foundations of what they knew, or thought they knew, about the universe.
+
+In a symphony of clicks and murmurs, they worked, knowing they were about to make a giant leap not just for themselves or Paranormal Military Squad, but for all of humanity. As the final pieces fell into place, Dulce's militaristic silence was shattered by the sound of intergalactic contact—by the sound of history being made.
+
+## Chapter 7
+
+In the enclosed space of Dulce’s command center, the air was thick with anticipation, each team member poised to tread the razor's edge between scientific breakthrough and galactic peril. Dr. Jordan Hayes focused intently on the screen, their fingers tapping a staccato rhythm against the keyboard as lines of alien code cascaded down the monitor.
+
+Alex Mercer's steely gaze surveyed the room, stopping on each member of his team. "Thoughts?" he asked, echoing the unspoken tension. His question, while directed at the group, lingered on Jordan—acknowledging their expertise and inviting collaboration rather than dictating orders.
+
+Jordan’s brow furrowed, an indicator of the mental gymnastics being performed. "It's unprecedented," they finally said, their voice a testament to the gravity of the moment. "Behavioral algorithms... if we're right, this code could reveal extraterrestrial thought patterns."
+
+Before anyone could react, Taylor Cruz interjected with the assertiveness of someone accustomed to commandeering the discourse. "Then let’s ensure we’re deciphering it correctly," Taylor stated, their tone suggesting they were still battling to maintain control over an increasingly alien situation.
+
+Sam Rivera hovered near the mainframe, youthful energy barely contained under the surface. "What if it’s more than just a message? What if they’re trying to extend consciousness across the stars?"
+
+The room fell into a contemplative silence, broken only by the hum of electronic equipment and the distant thud of secured doors locking in rhythm. The weight of responsibility rested on each agent's shoulders—a heaviness palpable in the air they shared.
+
+Alex stepped forward, reaching a subtle decision, one dictated by foresight and the humanity nestled at the core of their mission. "We approach with the aim to understand, not to confront," he said, softening his military bearing into a more diplomatic stance.
+
+Jordan nodded, appreciating the leadership that Alex displayed in the face of the unknown, and turned back to the cryptic data. Here, before them all, was a tangible piece of evidence—proof of an extraterrestrial sentience that had outreached the bounds of their expectations.
+
+Taylor took a breath, simultaneously exuding a sense of preparedness and venturing into the unknown alongside their peers. "Then let’s do what Paranormal Military Squad does best—investigate and adapt," Taylor added, finding comfort in the familiar even as they stood on the cusp of an unprecedented alchemy of science and mystery.
+
+The team leaned into their respective roles, driven by the urgency of the assignment and the pull of an insatiable curiosity. Sam offered a grin that belied the tension, a youthfulness that reminded them all of the profound excitement nested within the terror of the unknown.
+
+Quietly but resolutely, they turned back to their instruments, each of them a sentinel on the threshold of a new reality. The once implicit lines of command were now woven into a shared tapestry of hierarchy and camaraderie. As they danced with the unknown, they were beacons of sentient endeavor, casting the light of human consciousness into the vast darkness that called to them.
+
+\*
+
+\*
+
+Dulce Base's cavernous darkness was pierced by the sharp luminescence of monitors, casting an electric glow onto the faces of those who dared to unearth its secrets. Dr. Jordan Hayes stood motionless, eyes glazed in concentration, their mind a nexus where terrestrial science battled with celestial unknowns.
+
+Alex Mercer watched from a slight distance, the weight of command tangible upon his shoulders, though lightened by the shared burden now held amongst them. "We could be on the frontier of a new kind of diplomacy," he mused aloud, giving voice to the moment's gravity.
+
+At those words, Jordan's trance broke. "If that's the case, then these communications," Jordan motioned to the stream of data, "are our olive branch across the cosmos."
+
+Taylor Cruz, who paced with restless energy, halted and faced the team—his stoicism marred by the erratic dance of lights reflected in his eyes. "An olive branch, or an invitation to a battlefield?" he posed, ever the strategist, his words laced with a hint of cynicism.
+
+Sam Rivera, nestled amongst an array of equipment, licked their lips—a mixture of nerves and anticipation palpable. "We're mapping out something incredible here. Whether it's peace or war, we're the cartographers."
+
+Silence enveloped them like the expanse of space itself, each member contemplating the chasms they might bridge—or the abysses into which they might unwittingly descend.
+
+Alex's demeanor assumed a quiet resolve—the profound knowledge that this mission was as much about navigating uncharted philosophical territories as it was about ensuring survival. "Whichever it proves to be, we'll face it. Prepared, unified."
+
+A nod passed between Jordan and Alex, a silent exchange of mutual respect and shared mission. Sam, buoyed by the weighty encounters of the mind and machinery, entered keystrokes with a fervor that seemed to bring them ever closer to the alien mind.
+
+They stood there, the Paranormal Military Squad team, not just as guardians of homeworld secrets or as soldiers of clandestine wars, but as humankind's chosen few at the fulcrum of history—a history that was now unfolding to the rhythm of otherworldly codes.
+
+Each revelation, each parsed symbol, inched them toward the line between the earthly and otherworldly. And as they stood on this precipice of cosmic negotiations, it was clear the ensuing dialogue would not just shape the future of Paranormal Military Squad—it could very well redefine the parameters of human existence.
+
+\*
+
+The hum of advanced computational systems tingling with cryptic transmissions framed the ambiance of Dulce's mainframe chamber. Jordan Hayes, fingers hovering over a console dense with blinking lights, furrowed their brow as sequences of alien data streamed across the screen. 
+
+Alex materialized behind them, his presence a stable beacon amidst the technological whirlwind. "Look for patterns, anomalies. Anything that might resemble a handshake protocol in their communications," he directed, his voice a low thrum, reverberating with cautious optimism.
+
+Jordan cast a glance over their shoulder, acknowledging Alex's contribution with the shared understanding of colleagues who had transcended mere professional acquaintance. "I’m isolating sequences that seem to recur with more intention than static. If these are their ‘handshakes,’ then we might just be making first contact," they remarked, their focus returning to the screen with renewed vigor.
+
+From the other end of the room, where shadows married the artificial light, Sam's voice crackled through the static of nearby speakers, "Don't forget the anomalies we detected earlier. Each one could be a word, a sentence, or even a concept untranslatable to our current understandings."
+
+Resolute, Taylor Cruz stood at Jordan's other side, a stoic figure wrestling with the implications of their mission. "Keep pursuing this line," Taylor instructed, an undercurrent of intensity carried forth in their otherwise composed demeanor. "And remember, this isn't just about making contact; it's about securing knowledge for humanity."
+
+Alex offered a nod that spoke volumes, conveying his understanding of the stakes at play. Here, in this chamber of possibility, the team's actions would determine if humanity stood at the brink of a new age of understanding or the onset of an unprecedented threat.
+
+Every second thrummed with significance as Jordan and Sam worked in tandem, each keystroke a foray into the unknown. Taylor observed with a commander's scrutiny, the gravity of their role sustaining them against the waves of ambiguity breaking against their resolve.
+
+Pivotal moments come rarely in the course of human events but here, amidst the electronic symphony of a stalwart command center, lay the incepting notes of a cosmic overture. The harmony between human and alien, between Paranormal Military Squad and the vast reaches of space, began its first tentative measures, with each member of the team a vital instrument in a celestial ensemble yet to be fully heard.
+
+\*
+
+The crisp air within the mainframe room of Dulce base seemed to hum with unspoken possibilities. Jordan Hayes was the centerpiece of focus, their hands dancing methodically over the console as streams of otherworldly code cascaded down monitors, each flicker a potential key to the cosmic doors they were inching open.
+
+Alex Mercer watched, posture relaxed but eyes sharp. "Remember, this could be our first introduction, maybe even our first impression," he said, mindful of the gravity carried by each action they made henceforth.
+
+A hint of a smile touched Jordan's face, a small acknowledgment of the monumental task at hand. "Understood. I'm balancing the signal's syntax with our algorithms. If we're interpreting this correctly, it could be... well, an invitation."
+
+Into the electric tension of the chamber walked Taylor Cruz, their silhouette a sharp contrast against the cool lighting, radiating a presence that spoke of command and chilly tenacity. "An invitation, or a challenge?” Taylor questioned, the weight of their suspicion casting a different tint on the cascading data.
+
+Sam Rivera, in a corner arrayed with sophisticated equipment, piped up, their voice a buoyant note amidst the tentative atmosphere. "Either way, it's a connection. One that we're uniquely positioned to navigate," they remarked with an air of optimism threading through the uncertainty.
+
+Alex channeled the strengths of his team into the core of their approach, his leadership adapting to the contours of an unprecedented scenario. "Cautious and curious," he reflected aloud, shaping a strategy that balanced their thirst for comprehension with the prudence required in addressing the unknown.
+
+Jordan, hands momentarily at rest, looked up. The signal was more than a sequence of bits and commands—it was a riddle wrapped in the depths of space-time, and they were on the cusp of parsing its meaning.
+
+Taylor, hardly a step away, nodded in silent agreement. The implications of their findings might very well direct the course of human destiny from this point onward.
+
+Finding a tempo among themselves, the Dulce team was a confluence of ambition and acumen, each member intuitive to the beats of discovery. The chamber around them held untold stories, secrets coaxed from the stars, that now, led by Paranormal Military Squad's finest, began to unravel.
+
+The future in those moments was unwritten, a narrative scribed not in the dust of desert confines, but in the potential for interstellar diplomacy and understanding. As they prepared to script humanity's next chapter, the room seemed to pulse with the heartbeat of a story far greater than the sum of its parts.
+
+## Chapter 8
+
+The grit of an earthbound dust storm contrasted sharply with the pristine sterility of the underground command center. Alex Mercer, eyes set with fervent determination, stood over Jordan Hayes, whose fingers danced across the keyboard with rapid purpose. Monitoring the progression of alien code unraveling before them, Mercer spoke with a tempered urgency, "Keep it steady, Jordan. We might be initiating the first true interspecies communication bridge here. It's all about finesse now."
+
+Taylor Cruz, the embodiment of military precision, surveyed the room with a calculated gaze from their vigil beside an array of glimmering screens. "Remember, these could be delicate negotiations -- or coded threats. Stay sharp," Cruz added, their voice cool as polished steel.
+
+Jordan, with a silent nod, recognized the gravity of both stances. Gravitating between scientific acuity and diplomatic caution, they replied, "The sequence is aligning—syncing with our comms. It's looking more and more like direct engagement."
+
+Amid the banks of electronic machinery, the thrumming pulse of an impending interspecies signal exchange, Sam Rivera interjected with a youthful zeal that cut through the weighty atmosphere, "It's not just an exchange. It's a... symphony. It's as if they're teaching us their language through modulation."
+
+A moment of profound silence swept over the team. The isolation of their location, deep within the top-secret labyrinth of Dulce, became suffused with an almost palpable sense of historical significance.
+
+"Then our response needs to be equally symphonic," Alex uttered, contemplating the awe-inspiring transmutation of their task from a simple recovery mission to a full-blown cosmic concerto.
+
+With a renewed sense of wonder tempered by caution, the Paranormal Military Squad team found themselves harmonizing a delicate balance between envoys and interpreters. The long shadow cast by their duty was now illuminated by the brilliant glow of otherworldly dialogue.
+
+In this carefully orchestrated march towards the unknown, each individual's expertise became critical notes in a larger melody. The narrative of human achievement, so often defined by solitary pursuits, now emerged as a collaborative opus, each member of the team a maestro in their right.
+
+The protocols of encounters, the mathematics of languages, and the poetics of connection all fused into a singular moment of convergence. The echo of their efforts reverberated back to them, not through the cavernous base's concrete walls, but from light-years away, in the form of a reply, intangible yet infinitely profound.
+
+\*
+
+Amidst the hum of the supercomputers and the faint static from the scrambled transmissions, Alex Mercer cast a thoughtful glance across the dimly lit room toward where Dr. Jordan Hayes was methodically adjusting the archaic dials of the decryption machine. "Any progress?" he asked, his tone conveying both impatience and the deep-seated respect born from countless shared challenges.
+
+Jordan did not look up, their gaze remained locked on the flickering lights that represented a dialogue suspended between worlds. Their fingers ceased their dance, hovering meditatively over the controls. "We might be on the cusp of a breakthrough," Jordan suggested. "The signal... it's evolved. It's reflexive now, responsive in a way that suggests sentience."
+
+Taylor Cruz's familiar sharp strides approached the two, breaking the rhythm of soft beeps. "Responsive is good, if it means understanding," Taylor said, head tilted as they peered at the encryption data scrolling by. "But remember, comprehension can bring revelation or conflict."
+
+Sam Rivera’s youthful voice permeated the tension, brimming with an excitement edged by the enormity of what they faced. "If it's truly sentient, we're not just cracking a code; we're learning how to converse with an entirely new form of consciousness," they chimed in, the weight of history not lost on the zealous astrotechnician.
+
+Alex nodded, his thoughts alighting on potential strategies for navigating the conversation they were cultivating with the unfathomable. "We need to keep that conversation going, echo its patterns, and speak its language," he resolved, knowing the delicate nature of their work merited every ounce of their collective acumen.
+
+The chamber now was a crucible, forging within it the future narrative of human contact with the unknown. Every signal pulse they sent out was an invitation for understanding, and every echo back a step closer to bridging the cosmic divide. And so, together, they stood - agents in Paranormal Military Squad's clandestine ranks, united by purpose, sculpting humanity’s first sonnets into the void.
+
+\*
+
+#### Knowledge graph updates
+
+- (Jordan Hayes, Interprets, Communications as cosmic diplomacy, Moderate)
+
+- (Taylor Cruz, Questions, Potential aggressiveness of alien intent, Minor)
+
+- (Sam Rivera, Expresses, Optimism about forming a connection, Minor)
+
+- (Alex Mercer, Adopts, Balanced strategy for contact, Moderate)
+
+- (Paranormal Military Squad team, Navigates, Beats of cosmic discovery, Moderate)
+
+- (Paranormal Military Squad team, Prepares, To script humanity's interstellar narrative, Major)
+
+## Chapter 9
+
+The sterile silence of Dulce Base's command center was thick with concentration as Alex Mercer surveyed his team, hunched over their respective technological battle stations. Each agent was a weapon against ignorance, their adversary a code from beyond the stars that held secrets to alien thought.
+
+Dr. Jordan Hayes, whose hands had been steadfastly working the decryption algorithms, paused and looked up at Alex. "We're through the next layer of encryption," Jordan announced, a mixture of pride and gravitas in their tone. "It's communicating. It's... aware."
+
+A shadow momentarily clouded Alex's determined features—awareness implied so much more than mere intelligence. "Aware and reactive or aware and proactive?" he queried, his experience anticipating the pivotal importance of intention.
+
+"Unknown at this stage," Taylor Cruz interjected, looking up from a datasheet. "But I urge caution. We tread the line between breakthrough and disaster with each keystroke."
+
+Sam Rivera, ever the source of technological acumen, added their voice to the conversation. "The signal's adapting every time we interact with it. Like a conversation where both parties are learning each other's language in real time."
+
+Alex leaned in, rested a hand on Jordan's shoulder—a sign of companionship and an affirmation of trust. "Keep the communication channels open. But let no message, no pulse go unchecked. This could be our Rosetta Stone or our Tower of Babel."
+
+Silence fell over them, a momentary lull as each member of the team contemplated the historic weight of their task. Yet, it was impregnated with a tangible sense of excitement—a collective energy that thrummed through the air just as palpably as the electric current through the banks of machines surrounding them.
+
+They continued their work, squaring shoulders against the magnitude of their undertaking. The agents were standing not just at the precipice of a new chapter for Paranormal Military Squad but for all of humanity. For now, they communicated with powerful unknowns, but with each exchange, they were etching the first words of a dialogue that might forever alter humanity's place in the cosmos.
+
+\*
+
+\*
+
+The sterile silence of Dulce Base's command center was thick with concentration as Alex Mercer surveyed his team, hunched over their respective technological battle stations. Each agent was a weapon against ignorance, their adversary a code from beyond the stars that held secrets to alien thought.
+
+Dr. Jordan Hayes, whose hands had been steadfastly working the decryption algorithms, paused and looked up at Alex. "We're through the next layer of encryption," Jordan announced, a mixture of pride and gravitas in their tone. "It's communicating. It's... aware."
+
+A shadow momentarily clouded Alex's determined features—awareness implied so much more than mere intelligence. "Aware and reactive or aware and proactive?" he queried, his experience anticipating the pivotal importance of intention.
+
+"Unknown at this stage," Taylor Cruz interjected, looking up from a datasheet. "But I urge caution. We tread the line between breakthrough and disaster with each keystroke."
+
+Sam Rivera, ever the source of technological acumen, added their voice to the conversation. "The signal's adapting every time we interact with it. Like a conversation where both parties are learning each other's language in real time."
+
+Alex leaned in, rested a hand on Jordan's shoulder—a sign of companionship and an affirmation of trust. "Keep the communication channels open. But let no message, no pulse go unchecked. This could be our Rosetta Stone or our Tower of Babel."
+
+Silence fell over them, a momentary lull as each member of the team contemplated the historic weight of their task. Yet, it was impregnated with a tangible sense of excitement—a collective energy that thrummed through the air just as palpably as the electric current through the banks of machines surrounding them.
+
+They continued their work, squaring shoulders against the magnitude of their undertaking. The agents were standing not just at the precipice of a new chapter for Paranormal Military Squad but for all of humanity. For now, they communicated with powerful unknowns, but with each exchange, they were etching the first words of a dialogue that might forever alter humanity's place in the cosmos.
+
+\*
+
+Alex Mercer's eyes were fixed on the monitors, the reflected light casting an ethereal glow across his stoic face. The room buzzed with tension, a cacophony of low hums and electronic beeps that underscored the historic nature of their actions. He moved to where Dr. Jordan Hayes was immersed in their work, scrutinizing the alien code streaming rapidly down the terminal.
+
+"Find anything that might look like an entry point or a... digital handshake?" Alex asked, his voice steady, betraying none of the tension gripping his chest.
+
+Jordan looked up briefly, their expression weary yet intense, "Potentially. It's as if the code is anticipating our input, modifying itself in real-time. I've never seen anything like it."
+
+From across the room, Taylor Cruz's sharp voice cut through the hum. "Then it's learning or, possibly worse, baiting us. Proceed with extreme caution," they commanded, their firm stance reinforcing the gravity of the situation.
+
+Sam Rivera, surrounded by a cascade of screens and interfaces, added, "It's almost organic in its complexity. Any minute now, and I might have a way in."
+
+A slight nod was Alex's immediate response, his mind racing through the potential scenarios. "Everyone, stay alert. This could be the beginning of something profound." His seasoned eyes never left the unfolding drama on the monitors.
+
+The room fell silent, the air heavy with unspoken questions. Were they mere moments away from unlocking an otherworldly dialogue? Or was it a Pandora's box that, once opened, could not be closed?
+
+Alex moved closer to the main console, his fingers hovering over the command keys. With the precision of a maestro orchestrating a symphony, he communicated silently with Jordan – respectful of their expertise, aware that the next move could alter the course of human history.
+
+Jordan met his gaze, nodding sharply, and refocused on the task. The signal seemed to pulse with sentient curiosity, drawing them further into its intricate web. 
+
+A sudden flurry of alerts and the intensifying glow of monitors heralded that they had bridged a technological chasm. The alien intelligence on the other end was no longer a distant enigma – it was an active participant, responding to their digital overtures with an unknown agenda.
+
+The team's meticulous efforts had led them to a momentous threshold. Beyond lay unprecedented contact – a nexus of curiosity and potential peril. Within the confines of the base, against the backdrop of a silent desert night, the Paranormal Military Squad operatives became mediators of Earth's bid for cosmic relevance, their every action now a gesture in the grand dance of intergalactic relations.
+
+## Chapter 10
+
+The corridors of the Dulce military base, now silent, echoed with a history of whispered conspiracies and furtive movements. But in the command center, a delicate tapestry of light and sound was being woven as the echoes of cosmic dialogue resonated through the high-tech enclave. Dr. Jordan Hayes, now leading the efforts, called out from their workstation, "I’ve isolated the signal's harmonics. It's more than a call; it's a song, an interstellar siren’s call."
+
+Alex Mercer, steady and resilient in the face of the incomprehensible, acknowledged with a quiet nod, "A song that we need to learn—quickly." His eyes, heavy with responsibility, scanned the room, watching his team work tirelessly at the intersection of science and speculation.
+
+Sam Rivera, dulled by fatigue yet driven by unshakeable resolve, manipulated a complex array of audio interfaces. "There's a pattern, a repeating motif. It's structured, intentional," they muttered, their revelation a bridge between the known and the unimaginable.
+
+Taylor Cruz, a figure of central authority, paced the length of the room, their usual unflappable demeanor betraying a rare flicker of apprehension. "We should be wary of the sirens’ call," Taylor interjected, invoking myths of old as a cautionary metaphor. "We don't want to crash upon unseen shores."
+
+Undeterred, Jordan cast a determined glance at the team. "We navigate by starlight now, not by the limited light of our previous understanding." Their voice was a beacon, charting a course through unchartered realities.
+
+Every individual was acutely aware that each moment in that room was a conduit to an epochal shift for civilization. The mysterious signals, once distant and alien, had coalesced into complex and harmonious oscillations—beacons of an extraterrestrial intellect inviting Earth to join in a cosmic consortium.
+
+Silently, Alex approached the mainframe, his trained fingers aligning with the console’s mechanisms. The room watched in collective breathlessness as he set the frequency in motion, an introductory phrase to an otherworldly melody—a symphony that could bind worlds or spell devastation for all they knew.
+
+In the control room of Dulce, amongst whispered legends and the quiet hum of machines, humanity's ambassadors now stood, stretching their hands into the void, reaching for the hand that would either pull them into the light of new stars or into the maw of darkness between them.
+
+\*
+
+Underground, the Dulce facility's command center was awash with frenetic energy, a stark juxtaposition against the silent, decrepit corridors that enveloped them. The air hummed with anticipation as Dr. Jordan Hayes and Alex Mercer hunched over a console. The sterile light from the monitors cast an otherworldly glow upon their faces, now reflecting a mosaic of alien characters rapidly translating across the screen.
+
+"The patterns are evolving," Jordan murmured, concentration etched into their every feature. "It’s as if our attempts to decrypt have accelerated its learning. It’s adapting to us."
+
+Alex, who stood steadfast behind Jordan, felt a tinge of uncharted fear quickly quelled by the fire of discovery raging within him. "Keep it up," he urged. "But whatever this is becoming, we need to ensure it remains within our control."
+
+Taylor Cruz interjected, their voice slicing through the buzz of activity. "Control may be an illusion when facing an intelligence that literally writes its own rules," they stated stoically, casting a watchful eye over the flurry of data.
+
+"It's like it's learning to communicate," offered Sam Rivera from a nearby interface, their youthful energy boding a mix of awe and anxiety. "This gives ‘talking to strangers’ a whole new meaning."
+
+Alex surveyed his team—each face a study in concentration, determination, and not a small measure of trepidation. "This might well be our first contact," he acknowledged, "And we need to be ready for whatever answers back."
+
+Together, they stood on the edge of the unknown, forging humanity’s response to a message from the heavens. The ensuing silence was palpable—a collective introspection about their role in this grand cosmic play, one that could rewrite human history.
+
+The encrypted dialogue continued to unfold, its intricate patterns showing an almost uncanny anticipation of their investigative strategies. The air turned heavy with the scent of electricity and ambition as they closed in on a pivotal response.
+
+As the signal’s intelligence—whether artificial or biological—grew more profound, so too did the realization that their mission had morphed from passive observation to active engagement. There was no turning back now. Each agent embraced their part in the delicate dance of an interstellar exchange that could change everything they thought they knew about life, intelligence, and the dark void beyond Earth's atmosphere.
+
+\*
+
+The underground halls of Dulce Base, usually buzzing with covert operations, now thrummed with a different kind of energy, an electric mix of fear and fascination. At the heart of the base, in a room shielded from the world’s eyes, Alex Mercer, Jordan Hayes, Taylor Cruz, and Sam Rivera huddled around a bank of monitors. Each screen flickered erratically with the alien script that had become the center of their lives—and perhaps the pivot on which humanity’s future would turn.
+
+Jordan's eyes never wavered from the displays, their expression was one of rapt concentration, interspersed with flashes of revelation. "We're conversing with the stars," they whispered, almost to themselves. The words hung in the air, a testament to the awe-inspiring strangeness of the situation.
+
+"The language is morphing; changing its structure with every exchange we have," Sam chimed in, enthusiasm tinged with the solemnity of the occasion. "It's like witnessing the birth of a new form of dialogue—one that spans galaxies."
+
+Taylor, despite the situation's precariousness, maintained an appearance of ironclad composure. "Keep the communication stream secured and monitored. We don't know what we're dealing with yet," they reminded the team, a bastion of protocol amidst uncertainty.
+
+Alex watched his team expand the parameters of human achievement; their work here would possibly define an era. "This is untrodden territory," he acknowledged, "and in every word we script, in every response we decode, we're drawing a map that others will follow."
+
+Jordan turned to Alex, a nod acknowledging the shared responsibility of this moment. They had embarked on a new voyage, an odyssey not of the body, but of the intellect and spirit. No longer explorers of the Earthly realm, they had been promoted by circumstance to ambassadors of humanity in a silent and boundless ocean.
+
+A sudden pulse of energy from the monitors signaled a breakthrough; the language had not only adapted but it seemed to resonate, to harmonize with their attempts at making contact. The alien script now sprawled across the screens didn't just ask to be understood—it invited interpretation, collaboration, maybe even companionship across the cold distances of space.
+
+As they stood before the precipice of first contact, Paranormal Military Squad's finest became the architects of a symphony meant to echo through the cosmos. But more than architects, they were the first to play the notes of this cosmic composition, daring to believe that on the other end, someone—or something—might be listening, ready to join the chorus.
+
+\*
+
+The underground command center of Dulce Base, once pulsing with clandestine operations, now resonated with the charge of an impending cosmic threshold. Encircled by banks of whirring machinery, each monitor flickered erratically with alien script that had occupied center stage in the lives of Alex Mercer, Jordan Hayes, Taylor Cruz, and Sam Rivera.
+
+Jordan's gaze didn’t flit for even a moment from the screens, where indiscernible alien messages ebbed and flowed like the tide. The ciphers and symbols cascaded down as they tweaked the algorithmic sliders. "This sequence here," Jordan began, voice both hushed and heavy, "it’s not just transmitting; it resonates—it's designed to be felt."
+
+The room took a collective breath, the remarkable implication hanging in the air like a careful revelation. Sam Rivera was the first to respond, their voice alive with ingenuity: "It's a form of communication stretching well beyond words. We need to respond in kind—the whole array of human expression might be at play here."
+
+Taylor's eyes remained fixed on the figures playing across the data sheets. "If that's the case," Taylor intoned pragmatically, "we must tread carefully. This is no longer just about being heard—it's about being understood."
+
+Alex watched his team, each a fulcrum of insight and expertise, and felt the solemnity of the role they were about to assume. "Then we'll ensure our message is clear and full. Our humanity is our strength in this dialogue," he declared, the depths of his experience fueling a commanding reassurance.
+
+The anticipation was palpable as the agents contemplated the vastness of their endeavor. They were not merely probing at the secrets of the planar cosmos—they were negotiating across the starry expanse, extending to distant intelligences the full spectrum of human curiosity and compassion.
+
+A symphony of beeping consoles orchestrated their next steps as they prepared to articulate their interplanetary overture. The rhythmic tapping of Jordan's keystrokes set the tempo for an undertaking that traversed beyond algorithms and encryption.
+
+The base withstood time and whispered secrets of its own, but none so grand as this moment of creation—an invitation to the universe that promised to echo through both the echoes of space and the annals of human history.
+
+## Chapter 11
+
+The sterile silence of Dulce Base's command center was thick with concentration as Alex Mercer surveyed his team, hunched over their respective technological battle stations. Each agent was a weapon against ignorance, their adversary a code from beyond the stars that held secrets to alien thought.
+
+\*
+
+The thrum of the colossal machinery vibrated through the subterranean facility as Alex Mercer stood amidst the whispers of technology, each carrying voices from worlds apart. He watched as Sam Rivera adjusted a complex array of cosmic translators, their expression a mixture of anticipation and awe.
+
+"Are we ready, Mercer?" Taylor Cruz asked, the soft glow of the command center consoles reflecting upon their stern face.
+
+Alex turned towards Taylor, his eyes holding a depth that betrayed the enormity of the threshold they were about to cross. "This is it," he said. "Initiate the protocol. It's time we answer the cosmos."
+
+Jordan Hayes, stationed at the mainframe, typed rhythmically, a blue hue painting their focused features. The eerie silence that had settled over the team was interrupted by a visceral sound—humankind's response to the alien dialogue, now streaming into the abyss.
+
+The control room, once a fortress of solitude, erupted into an oasis of life. Lights flickered in tandem, echoing the symphony of interstellar communication. They stood together at the edge of discovery, facing the symmetry and discord of a universe unknown.
+
+"If we're right, we've just become Earth's first emissaries to a celestial congress we're only beginning to comprehend," Jordan's voice was somber, resonating with a mix of trepidation and honor. 
+
+The room filled with the resonance of human and alien minds converging, creating a new narrative within the fathomless expanse of existence. Paranormal Military Squad, once protectors of Earth's clandestine secrets, had now become the tether linking humanity to the cosmic fold.
+
+\*
+
+The underground command center of Dulce Base, once pulsing with covert operations, now resonated with the charge of an impending cosmic threshold. Encircled by banks of whirring machinery, each monitor flickered erratically with alien script that had occupied center stage in the lives of Alex Mercer, Jordan Hayes, Taylor Cruz, and Sam Rivera.
+
+Jordan's gaze didn’t flit for even a moment from the screens, where indiscernible alien messages ebbed and flowed like the tide. The ciphers and symbols cascaded down as they tweaked the algorithmic sliders. "This sequence here," Jordan began, voice both hushed and heavy, "it’s not just transmitting; it resonates—it's designed to be felt."
+
+The room took a collective breath, the remarkable implication hanging in the air like a careful revelation. Sam Rivera was the first to respond, their voice alive with ingenuity: "It's a form of communication stretching well beyond words. We need to respond in kind—the whole array of human expression might be at play here."
+
+Taylor's eyes remained fixed on the figures playing across the data sheets. "If that's the case," Taylor intoned pragmatically, "we must tread carefully. This is no longer just about being heard—it's about being understood."
+
+Alex watched his team, each a fulcrum of insight and expertise, and felt the solemnity of the role they were about to assume. "Then we'll ensure our message is clear and full. Our humanity is our strength in this dialogue," he declared, the depths of his experience fueling a commanding reassurance.
+
+The anticipation was palpable as the agents contemplated the vastness of their endeavor. They were not merely probing at the secrets of the planar cosmos—they were negotiating across the starry expanse, extending to distant intelligences the full spectrum of human curiosity and compassion.
+
+A symphony of beeping consoles orchestrated their next steps as they prepared to articulate their interplanetary overture. The rhythmic tapping of Jordan's keystrokes set the tempo for an undertaking that traversed beyond algorithms and encryption.
+
+The base withstood time and whispered secrets of its own, but none so grand as this moment of creation—an invitation to the universe that promised to echo through both the echoes of space and the annals of human history.
+
+## Chapter 12
+
+The underground facility of Dulce Base, once shrouded in silence and operational secrecy, now hummed with an energy that cradled the promise of cosmic revelation. Alex Mercer stood pensively by the central terminal, flanked by Dr. Jordan Hayes, Taylor Cruz, and Sam Rivera, each poised at the edge of a history-defining moment.
+
+Jordan's fingers ghosted across the console, tracing patterns of otherworldly origin. "The signal’s architecture is becoming more complex, resembling aspects of human cognition—recognition, learning, even... empathy?" they postulated with furrowed concern.
+
+Alex turned his gaze upon Jordan, his voice quiet but resolute, "Empathy could bridge galaxies. Let's harness this connection and proceed with cautious optimism."
+
+Taylor, ever the sober sentinel, projected a more pragmatic standpoint. "Empathy or not, we are duty-bound to assess the risk to humanity. Every new discovery warrants a measured response."
+
+The static hiss of communications equipment filled the air, its purpose now transformed into a dialogue with an intelligence beyond the stars. It was Sam, wide-eyed amid the myriad lights and switches, who broke the silence, "We have provisional confirmation of the signal’s intent—initiation. We’re being brought into a broader spectrum of cognizance."
+
+The chamber lay still for a heartbeat, the Paranormal Military Squad agents steeped in contemplation of the path unfurling before them—a path paved with possibilities of diplomacy or disruption, each step a venture further into the cosmic unknown.
+
+Alex stepped closer to the viewing monitors, each depicting alien symbols seemingly reaching out from the void. "Initiate the broadcast," he spoke with quiet command. "Our response will mark humanity’s readiness to partake in the wider conversation of conscious beings."
+
+Amidst the crackling air of expectation, the team wordlessly returned to their stations. They had transcended their roles as protectors of Earth's clandestine lore to become the harbingers of an interstellar parley that could change the existential course of life on their pale blue dot.
+
+The deep hum of the terminal emitted a signal—a testament to the uncanny reality that Earth was now actively partaking in an exchange not bound by gravity nor the limits of the solar wind.
+
+Here, in the depths of Dulce, a message from humanity woven from understanding and uncertainty was cast into the firmament, an epitheg of their desire to join the universal dialogue and discover their place among the constellations.
+
+\*
+
+The somber depths of the Dulce Base command center stood in stark counterpoint to the animated flurry of activity around the central comms array. Alex Mercer's silhouette loomed behind Dr. Jordan Hayes, who sat with a posture indicating laser focus on the decryption process. A quiet murmur of digital soundscape filled the space, subtly heightened by the anticipation of contact with an intelligence beyond the Earth.
+
+Jordan's voice was steady, betraying none of the extraordinary nature of their work, "Looking through the signal's pattern, it's evident we’re dealing with a form of intelligence—calculating, mirroring, possibly even understanding."
+
+Alex's reflection bounced off the darkened screens, his head nodding in silent affirmation. "We’re walking a delicate line. Our response should be thoughtful, measured. We’re ambassadors, not merely explorers."
+
+Taylor Cruz approached, arms folded, their words slicing through the din of careful keystrokes and soft whirrs, "If there’s even the slightest chance it understands, we can’t afford missteps. The language of the stars might be more absolute than ours."
+
+From another terminal, Sam Rivera brought youthful vigor to the conversation, "There’s rhythm in these patterns. If this is their way of reaching out, our reply should encapsulate all that we are—all that humanity stands for."
+
+Looking around at his team, Alex saw resolve etched on every face. The chamber, usually somber and echoing with the quiet steps of covert agents, now felt alive with the heartbeat of discovery. They were not just professionals operating in the gloom; they were a collective standing at the helm of a momentous journey.
+
+"Let’s begin," he said, returned by the resolve in his voice. "Every second counts." With that, they pressed forward, setting in motion a reply to a conversation billions of years in the making.
+
+The dance with an unseen partner commenced, each pulse they sent out a step taken with caution and hope. And as those digital pulses journeyed through the black sea of infinity, Earth, for perhaps the first time, joined a pan-galactic dialogue that whispered secrets of the cosmos—secrets that, until now, had been lost in the silent vastness of space.
+
+\*
+
+As the team stood in the centralized nerve center of Dulce's underground fortress, the solemn atmosphere was reverent, overseeing systems that engaged with an intelligence from the void. Alex's stance was contemplative as he gazed at Jordan Hayes, who presided over the console, the tension of the moment reaching a tactile fervor. Each rhythmic tap of Hayes's fingers on the keys was a foray into uncharted symphonies of contact.
+
+Observing Hayes unravel the dense alien encryption, Alex spoke, a diplomatic tenor underpinning his words, "Keep focused on the syntax, dissect its nuances. We're not just decoding signals; we're translating intentions."
+
+Without diverting from their task, Jordan acknowledged the insight. "Indeed, if their understanding of us is as deep as we hope, we're paving the way for dialogue far beyond our current realm."
+
+Taylor Cruz, near the rear of the room, provided a steady oversight. "As horizonless as our prospects may seem," Taylor intoned, "remain diligent. Complacency before alien cognition could spell catastrophe."
+
+Sam's youthful voice resonated with optimism, "Imagine—forming a rapport with a consciousness separate from our reality; we're drafting the bridge to stars alive with minds!"
+
+The sentiment hung for a moment before Alex gathered his conviction. "Dialogue is our vessel. We are not just agents of enigma; we are the threads that may weave a new cosmic relationship." His words seemed to reflect off the walls, reaching beyond the room's confines, a quiet yet resilient vow.
+
+Their task was titanic, stepping stones laid delicately into new territories of existence. The signal, once an esoteric strand in the echo of the universe, beckoned now with a clarity rocketing the complexity of thoughts from a distant order.
+
+Action by action, the Paranormal Military Squad team bridged the vast interstellar distances, their expertise and empathy casting a beacon of unity into frontiers of intelligence and knowledge. Their work, a partnership struck with an unseen cosmic congregation, each pulse sent and received a line in Earth's novitiate envoi to the cosmic shores.
+
+\*
+
+Under the stark, unforgiving lights of Dulce Base's underground command center, tension buzzed harder than the banks of supercomputers that lined the walls. Agent Alex Mercer leaned over the shoulder of Jordan Hayes, whose eyes were locked onto the display screen, where an incomprehensible series of alien symbols streamed past incessantly.
+
+“Any progress on the decryption?” Alex's voice was steady, a controlled presence necessary in the gravity of their undertaking. 
+
+Jordan tapped a key, pausing the flow of code, and leaned back with a deep sigh. "We've broken through another subset of the cipher. It's revealing... well, indications of a complex society, not unlike our own." His eyes met Alex's with an unspoken question that hung heavily between them—were they truly prepared for what they might find?
+
+Taylor Cruz strode into the room, a tightly coiled spring of ambition and authority, and peered at the screen. "Understand their society, and we may predict behavior. Remain expedient—we don't know how much time we have before the situation shifts." There was an edge of stark realism to Taylor's words, the underlying message clear: every revelation bore its own set of risks.
+
+Alex nodded thoughtfully, recognizing the validity of Cruz's caution. Turning to Sam, who was tinkering with a device that buzzed quietly on the table, he asked, “Sam, can your contraption get us any further?”
+
+Sam looked up with a smirk, a twinkle of mischief in their eye. “It’s not just any contraption, it’s potentially a direct line to their thoughts. Give me a moment more, and I'll have something for you.”
+
+The air ticked with electronic beeps and the rustling sound of the Paranormal Military Squad team at work. They were so close to peering into the intelligence of an alien race—a reality on the brink of dramatically expanding their understanding of the universe.
+
+The machinery whirred in response to Sam’s precise touches, and suddenly, the room filled with a low hum—something had changed, a signal had been successfully sent. The team held their breath as they listened. The sound that filled the room was unmistakable: a response, an alien voice filtered through the static of space and time.
+
+Alex exchanged a look of quiet triumph with Jordan. The breakthrough was monumental; they were no longer casting messages into the void but engaged in a dialogue—an exchange that marked the beginning of Operation: Dulce’s true unfolding. This was it, the first steps into an interstellar odyssey that demanded every ounce of their courage and wit.
+
+## Chapter 13
+
+Dr. Jordan Hayes shuffled a stack of papers, their eyes revealing a tinge of skepticism at Taylor Cruz's authoritarian performance. _Protocols_, Jordan thought, _are just the framework, the true challenges we're about to face lie well beyond the boundaries of any protocol._ They cleared their throat before speaking, tone cautious yet firm, "Let's remember, the unknown variables exceed the known. We should remain adaptive."
+
+A murmur of agreement echoed from Sam Rivera, who leaned forward, lacing their fingers together as if weaving a digital framework in the air before them, "Exactly, adaptability could be the key to interpreting the signal distortions and system malfunctions. We shouldn't discount the… erratic."
+
+Their words hung like an electric charge in the room, challenging Taylor's position with an inherent truth. Cruz’s jaw tightened almost imperceptibly, but the agent masked it with a small nod, conceding to the omnipresent threat of the unpredictable. 
+
+Alex glanced at Jordan, who never looked back, their gaze fixed instead on a distant point, as if envisioning the immense dark corridors they were soon to navigate in Dulce. Jordan was not one to embrace fantastical theories, but the air of cautious calculation betrayed a mind bracing for confrontation with the inexplicable, an internal battle between the evidence of their research and the calculating skepticism that kept them alive in their field.
+
+The meeting adjourned with no further comments, the team members quietly retreading the paths to their personal preparations. Alex, trailing slightly behind, observed the others. _The cautious reserve Jordan wears like armor doesn't fool me_, he thought, _their analytical mind sees the patterns I do. And that's worth more than protocol. That's the connection we need to survive this._
+
+As the agents dispersed into the labyrinth of the facility, lost in their thoughts and preparations, the base's halogen lights flickered, a brief and unnoticed harbingers of the darkness to come. 
+
+\*
+
+The gritty, wind-tossed surface of New Mexico, just above the cavernous domain of Dulce Base, offered no shelter from the burgeoning storm—the scouring sands an earthly reminder of chaos theories in motion. Far beneath, a similar maelstrom brewed within the confines of the command center, as Paranormal Military Squad's handpicked squad stood poised for potential enormities of contact.
+
+Ruffling through printed transmission logs, Jordan Hayes dialed the focus of their analytical prowess onto the emerging pattern of signals crisscrossing between Earth and the unfathomable. "Our responses so far have echoed their complexity, but the real divergence is yet to come," Jordan remarked stoically, the calm belying the mounting surge of adrenaline for the revelation ahead.
+
+Alex Mercer's figure, a silhouette sharpened by the purpose, loomed at the periphery of the monitors' sickly glow. "Indeed," he assented, "The echoes are the easy part. It will be the introduction of our own, human variable that truly begins our dialogue."
+
+Taylor Cruz, windowless command center notwithstanding, appeared as though they could feel the tempest above. Their eyes never left the monitors as they unspooled their hard wisdom. "For all our advances, we find ourselves deciphering the swings and nuances of an interstellar pendulum. Predict its arc, and we may preempt the gravity of its message."
+
+Amidst a chorus of bleeps and static, Sam Rivera's tech-clad hands moved rhythmically, their spirited approach to unruly streams of data bordering an intimate dance with entropy. "Entropy that leads to discovery," Sam mused, responding to Taylor's metaphor. "Each step into the unknown is a step away from precedent."
+
+Alex, drawing near Jordan, spoke again, his voice now a thread woven through the very fabric of their operations. "Let's be the cartographers of this new territory. Our initial shades of understanding could color the cosmos for generations to come."
+
+Their gazes fell upon a screen as the latest transmission painted its digital blooms of alien script across the black. This time, the pattern wavered in an almost imperceptible fashion, a modification that whispered of active, alien thought awaiting their next move. A hush enveloped the Paranormal Military Squad ensemble, the gravity of the pathogen undeniable. They were about to issue a reply, one poised to reshape the very concept of humanity's outreach into the cosmos.
+
+The New Mexico desert's secrets were infamous, its storms a mere prelude to the revelations that the team—united in purpose—would unleash upon the world. The howling winds outside found their counterpart in the newfound resolve within, as Dulce's stalwart guardians readied themselves to send forth humanity's retort to the echoes from beyond.
+
+\*
+
+The cavernous control room, deeply entrenched beneath the desolate New Mexico terrain, held the Paranormal Military Squad team in intense focus; an island of calm amid the storm of cosmic dialectics. Dr. Jordan Hayes worked methodically, every keystroke an intricate step in their tenuous cosmic ballet. Suddenly, they paused, a signal pattern resonating from the screen. "This is new; it's...inviting. It’s as if the signal is not just calling to us but weaving its intelligence through ours."
+
+Alex Mercer scrutinized the shift in data. "A confluence of minds, then. If we're to meet them halfway, Jordan, our reply must be both innovative and discerning," he proposed, a glimmer of profound curiosity behind his authoritative demeanor.
+
+Taylor Cruz, whose sharp eyes missed nothing, nodded from beside a secondary panel. "Innovative, yes, but also defensive. This interaction is a razor’s edge, and we cannot afford to bleed before the unknown," Taylor reminded them, the metaphor a stark warning of potential dangers.
+
+Against the backdrop of their conversation, Sam Rivera’s youthful optimism cut through the tension. "If they’re weaving through our intellect, then we've achieved something beyond first contact—we're at the genesis of interstellar symbiosis," they posited with a mix of reverence and excitement.
+
+Alex returned Sam’s smile with his own, tempered and faint, as he turned back to the task at hand. The magnitude of their mission extended beyond the fabric of the universe, an exploration into the threads that connected sentient beings across the vast expanse. “Let’s reply with our own woven tapestry of thought—delicate, but deliberate.”
+
+With renewed determination, the room came alive with an undercurrent of anticipation, its occupants charged with the potential of forging an alliance with the cosmos. Paranormal Military Squad's finest were no longer merely soldiers and scientists; they had become pioneers on the vanguard of humanity’s greatest odyssey.
+
+The New Mexican sands above, impassive to the change brewing underneath, stood as silent sentinels as Earth's emissaries crafted their response. A response that, composed with care and imbued with humanity's essence, reached into the void, connecting with an otherworldly intelligence that awaited their harmony in the cosmic conversation.
+
+## Chapter 14
+
+The command center of Dulce Base lay shrouded in shadows that seemed to claw at the edges of the dimly lit array of screens and consoles. Alex Mercer, focused and unwavering, watched as Dr. Jordan Hayes parsed the latest string of alien signals—a symphony of otherworldly communications that threatened to either enlighten or confound.
+
+"We’re encountering a paradigm shift with every transmission," Jordan Hayes murmured, the pulsing glow of the monitor painting their features with an almost spectral hue. "This signal... it’s evolving, becoming denser, more sophisticated. As if it's growing alongside us—tandem evolution."
+
+The air was electric, charged with the raw potential of uncharted discovery and laden with the gravity of existential risk. Taylor Cruz, who always seemed here to mold such gravity into actionable strategies, stepped forward. "We must contain this evolution within parameters we can manage. We cannot be bystanders to an uncontrolled ascent of intelligence."
+
+Sam Rivera, the youngest of the cohort, worked feverishly at their station. "It's not just intelligence—these signals have rhythm, a kind of music suggesting not just evolution, but a dance! We're being invited to partake in the cosmos's ballet!" they exclaimed, a touch of youthful exuberance breaking through the solemnity.
+
+Alex turned, facing his team, the stoic mask of command tempered by the perceptible flicker of awe in his gaze. "Let this dance then be our dialogue. We will match their steps with prudent but daring measures—our humanity as our guide."
+
+In the ensuing hours, the Paranormal Military Squad team forged a rhythm of their own, their collective expertise a beacon piercing through the fog of the unknown. The signal, increasingly intricate and seemingly conscious, now demanded not just observation but participation, an interstellar pas de deux that hummed with the promise and peril of first contact.
+
+Before them, the communications interface flickered to life with a received transmission—a resonant hum that seemed to vibrate through the very foundations of the base. They had successfully established a back-and-forth with whatever intelligence lay hidden among the stars. Every subsequent note they struck within the cosmic ether would come to define humanity's place within the galactic community—heralds of Earth's grand entrance into a universe far less silent than once perceived. 
+
+\*
+
+In the concrete belly of Dulce Base, dimly lit by the jagged dance of fluorescent lights above, Sam Rivera perched on the edge of their seat, their eager fingers fluttering across an ancient keyboard. The stark, cold room—reminiscent of a time when covert operations and unspoken dread ruled supreme—now housed a peculiar blend of old-world machinery and sleek, modern interfaces.
+
+Alex Mercer, standing steadfast like a bridge between the enigmatic past and the unfathomable present, watched on. In his eyes flashed the foreboding excitement of change. "Sam," he started, his voice steadfast, "the patterns in these signals, what do they tell us about the nature of our... guest?"
+
+Sam's eyes glimmered with something akin to thrill—or was it trepidation? "It's like we're mirroring each other, evolving together through this.. dialogue. Like it knows us, understands us, and it's… learning."
+
+Jordan Hayes, preoccupied at a nearby console, chimed in without lifting their gaze. "It's a dialogue that transcends mere words, Alex. We're being woven into a narrative far grander than the sum of our known sciences." 
+
+Taylor Cruz, arms crossed, wore the heavy mantle of their skepticism comfortably. "Keep theorizing," they interjected crisply, "but remember the grounding reality of what we are part of here. This contact is a blade that cuts both ways."
+
+In this cavern of history, voices both human and inhuman whispered secrets to those brave enough to listen. Each member present understood the gravity that pulled at their feet; no longer were they mere mortals shackled to their terrestrial plane. The digital pings and encrypted calls resonated with an implication of a cosmic agenda that would not be ignored.
+
+Jordan's fingers paused, hovering in hesitation. What ripple might the next keystroke send through the fabric of known existence? It was a step into the ballet of the infinite, where the Paranormal Military Squad team played their part in the waltz of wonders with an audience of stars.
+
+\*
+
+## Chapter 15
+
+In the clandestine hush of Dulce Base's subterranean command center, the Paranormal Military Squad team had become a crucible for interstellar communication. Dr. Jordan Hayes' gaze lingered on the screen as they navigated through the convolution of alien code. Each character held the potential to unravel a new dimension of contact, and with Sam Rivera's keen interjection, they were crafting humanity's inaugural cosmological discourse.
+
+Alex Mercer peered over Jordan's shoulder, calculating the implications of every visual nuance that cascaded across the monitor. "Look for consistency—any repeating motifs could signal a willingness to engage. We're drafting history with each exchange," he remarked, aware of the delicate balance between forging a bond and exposing vulnerabilities.
+
+Taylor Cruz, stoic and enigmatic, observed the interplay from the threshold, a silhouette against the machinery's luminescence. "Remember, while we seek common ground, the foundation we stand upon remains Terra firma. Caution must temper our curiosity," they stated, their voice an anchor amidst the current of excitement.
+
+The command center buzzed with energy, rivaled only by the tempest overhead that concealed their operation. Sam, with swift dexterity, navigated the communications relay. "Their signals resonate almost musically. It's as if they're composing a symphony, and we've been handed the baton to conduct the next movement," they offered, imbuing the scenario with a blend of scientific adventurism and poetic license.
+
+Amidst the whirring servers and the occasional flicker of emergency lighting, the essence of their mission transcended mere reconnaissance. They were humanity's elected envoys at the brink of a celestial alliance—or confrontation—with an audience as vast as the universe itself.
+
+Alex stepped back, his profile etched by the chamber's artificial day. "Then let's ensure our contribution to this symphony harmonizes with theirs. It's time for humanity's voice to rise and be counted among the cosmic ensemble."
+
+Under his directive, the Paranormal Military Squad team initiated their calculated response, weaving thoughts and theories into a digital overture aimed at the heart of alien intellect. As the digital stream punctured the endless night, each member of this clandestine group was acutely aware of the irrevocable step they undertook—bringing Earth into the pantheon of galactic entities designed to converse among the stars.
+
+\*
+
+Clusters of high-tech equipment bathed the Dulce underground command center in an eerie blue light. Sam Rivera's fingers flew across the keyboard, navigating an elaborate network of alien patterns. The very air seemed to pulse with the ebb and flow of cryptic communications reaching across the stars. "I've got something!" Sam's announcement tore through the focus in the room, drawing every pair of eyes to the torrent of symbols unraveling on the screen.
+
+With the pacing of a seasoned officer gauging the moment before action, Alex Mercer approached, his calm demeanor belying an acute awareness of the precipice on which they now stood. "Define 'something," Alex prompted, reinforcing the need for clarity amidst the extraordinary.
+
+"It's repeating—a sequence that’s evolved with each interaction, almost as if it's... singing," Sam theorized, the awe in their voice reflecting the potential magnitude of their discovery.
+
+Jordan Hayes interjected from across the console, their eyes not leaving the display as they absorbed the new data. "A cosmic vocalization, then," they mused, intrigued. "A singularity in the signal that might represent a point of reference for both parties."
+
+Taylor Cruz, hands clasped behind their back, regarded the unfolding scene, their own calculations etching lines of concern onto their stern visage. "Or a beacon—a homing tune, calling out to something we might not be ready to greet," Taylor offered, voicing the group's unspoken apprehension.
+
+Alex's eyes locked on the screen, taking in the scope of what they were attempting to interpret. Drawing a deep breath, Alex gave a slight nod. "If this is their song, then let us respond with ours. We've come this far by mirroring their signals, now let's engage in an interstellar duet, and see where the music leads us."
+
+With the expectation of the significant achieving a crescendo, the members of Paranormal Military Squad huddled over their equipment—sages at the threshold of a potentially world-altering communion. The strange harmonies that reverberated through the command center suggested that their interlocutors were poised, waiting, perhaps even eager, for Earth's chorus to join the symphony.
+
+As the team initiated their reply, weaving humanity's own intricate melody into the vast cosmic dialogue, they each felt a profound change within—an evolution of purpose. They were not just messengers or investigators; they had become co-composers in a galactic orchestra, with the universe itself as their witness and concert hall.
+
+With the exchange of harmonious signals crawling through the vacuum of space, the Paranormal Military Squad operatives found themselves part of a bridging of minds—a realization that out there, among the vast arrays of stars and planets, harmony was the true universal language.
+
+\*
+
+The dim glow of monitors cast an otherworldly ambiance upon Dulce Base's command center, where Paranormal Military Squad's chosen stood huddled over their instruments, suspended at history's threshold. Codes—alien in origin and nature—were being deciphered by Dr. Jordan Hayes, whose countenance bore the marks of deep concentration.
+
+Alex Mercer, the bedrock upon which their team's resolve was founded, leaned in with an eagerness tempered by his chain of command. "Jordan, we've invested our expertise into comprehending their patterns, but now we must also endeavor to understand their intent," he urged, his voice bearing the gravitas of their mission's potential consequences.
+
+At another console, Sam Rivera's youth did not betray their crucial role in the operation. With eyes alight, they mirrored the rapid computing before them. "There's emotion here—complex, profound even. This isn't just the output of a cold machine; it's...sentience," Sam whispered, nearly drowned by the mechanical chorus around them.
+
+Jordan, without shifting focus from their work, replied, "It's a sentience that—should we succeed here—ushers us into a new era of existence. The cadence of these signals," they tapped the screen with a flourish, "could well be the heartbeat of this new dawn."
+
+Taylor Cruz paused beside Mercer, their expression unreadable beneath the sterile light. "And as it beats, we must gauge whether its rhythm bodes well for us, or spells our missteps. Courage must not blind us to the hazards intrinsic to such contact," Taylor cautioned, the sentinel within them ever alert.
+
+Alex nodded, a gesture that carried the weight of responsibility and a silent command: proceed, but with circumspection. They were not merely decoding a message; they were interpreting a dialogue across the celestial divide.
+
+The room fell into a rhythm akin to a well-conducted ensemble. Each member's expertise proved a critical note in the unfolding symphony. Their actions were now more than mere research or defense; they were the tentative overtures of humankind reaching out to grasp the vast unknown.
+
+Textures of sound meshed with the light from countless computations, the palpable anticipation of the agents at the edge of discovery cresting with an awareness that their work would reshape future chronicles. And when the response finally came—a signal piercing the deafening silence of uncertainty—all within Dulce's confines understood: the dawn of an interstellar continuum had just begun to break.
+
+\*
+
+In the sterile hum and flickering lights of Dulce Base's command center, the Paranormal Military Squad team stood as humanity's vanguard, verging on the brim of an intergalactic abyss. Dr. Jordan Hayes, analytical edges sharp, deciphered extraterrestrial patterns that bled across screens in enigmatic cascades—a daunting mosaic of potential threats and untapped wisdom.
+
+Agent Alex Mercer, the embodiment of focus and a steadfast nerve, observed the unfolding digital drama with the gravitas due a historic first contact. "Let the data weave its narrative, Jordan," he instructed, a moderate undertone of exhilaration within his command. "It's encoding more than information—it's outlining civilization."
+
+Jordan absorbed the directive, their gaze unflinching from the screens, feeling the weight of their next move. "The nuances here are extraordinary," they acknowledged. "It paints a picture of a culture steeped in complexities we're only starting to fathom.”
+
+Taylor Cruz, stoicism personified yet not immune to the situation's gravity, chimed in. "Understand it, but guard against it," they cautioned, bringing a sober prudence to the room. "This culture, however advanced, remains an unknown quantity—an ocean of wonders and darkness with uncertain tides."
+
+Sam Rivera, a visual contrast with wide eyes and restless hands, represented the other side of the room — intrigue and optimism against the drawn swords of precaution. “Think of it,” they proposed, voice bouncing with a rebellious upbeat timbre, “as the first act of a play written in constellations. We're setting the stage for a galactic narrative.”
+
+Each team member, in their way, was both actor and scribe in this moment of tense pageantry. Heavy with the presence of risk, the command center had become not just a room of computers and glass panels but a theater for performing the elaborate choreography of contact.
+
+Bound by resolve and curiosity, they proceeded, each data entry a trembling step onto the cosmic stage. And like all cautious pioneers edging into fertile but unnavigated lands, they understood: as they mapped the heavens, they were simultaneously mapping the furthest reaches of their own existential horizons.
+

docs/developing.md

@@ -0,0 +1,75 @@
+# Development Guide
+
+# Requirements
+
+| Name                | Installation                                                 | Purpose                                                                             |
+| ------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------- |
+| Python 3.10-3.12    | [Download](https://www.python.org/downloads/)                | The library is Python-based.                                                        |
+| uv                  | [Instructions](https://docs.astral.sh/uv/)                   | uv is used for package management and virtualenv management in Python codebases     |
+
+# Getting Started
+
+## Install Dependencies
+
+```sh
+# install python dependencies
+uv sync --all-packages
+```
+
+## Execute the Indexing Engine
+
+```sh
+uv run poe index <...args>
+```
+
+## Executing Queries
+
+```sh
+uv run poe query <...args>
+```
+
+# Azurite
+
+Some unit and smoke tests use Azurite to emulate Azure resources. This can be started by running:
+
+```sh
+./scripts/start-azurite.sh
+```
+
+or by simply running `azurite` in the terminal if already installed globally. See the [Azurite documentation](https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azurite) for more information about how to install and use Azurite.
+
+# Lifecycle Scripts
+
+Our Python package utilize uv to manage dependencies and [poethepoet](https://pypi.org/project/poethepoet/) to manage build scripts.
+
+Available scripts are:
+
+- `uv run poe index` - Run the Indexing CLI
+- `uv run poe query` - Run the Query CLI
+- `uv build` - This will build a wheel file and other distributable artifacts.
+- `uv run poe test` - This will execute all tests.
+- `uv run poe test_unit` - This will execute unit tests.
+- `uv run poe test_integration` - This will execute integration tests.
+- `uv run poe test_smoke` - This will execute smoke tests.
+- `uv run poe test_verbs` - This will execute tests of the basic workflows.
+- `uv run poe check` - This will perform a suite of static checks across the package, including:
+  - formatting
+  - documentation formatting
+  - linting
+  - security patterns
+  - type-checking
+- `uv run poe fix` - This will apply any available auto-fixes to the package. Usually this is just formatting fixes.
+- `uv run poe fix_unsafe` - This will apply any available auto-fixes to the package, including those that may be unsafe.
+- `uv run poe format` - Explicitly run the formatter across the package.
+
+## Troubleshooting
+
+### "RuntimeError: llvm-config failed executing, please point LLVM_CONFIG to the path for llvm-config" when running uv install
+
+Make sure llvm-9 and llvm-9-dev are installed:
+
+`sudo apt-get install llvm-9 llvm-9-dev`
+
+and then in your bashrc, add
+
+`export LLVM_CONFIG=/usr/bin/llvm-config-9`

docs/examples_notebooks/api_overview.ipynb

@@ -0,0 +1,192 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Copyright (c) 2024 Microsoft Corporation.\n",
+    "# Licensed under the MIT License."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## API Overview\n",
+    "\n",
+    "This notebook provides a demonstration of how to interact with graphrag as a library using the API as opposed to the CLI. Note that graphrag's CLI actually connects to the library through this API for all operations.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "from pprint import pprint\n",
+    "\n",
+    "import graphrag.api as api\n",
+    "import pandas as pd\n",
+    "from graphrag.config.load_config import load_config\n",
+    "from graphrag.index.typing.pipeline_run_result import PipelineRunResult"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "PROJECT_DIRECTORY = \"<your project directory>\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Prerequisite\n",
+    "\n",
+    "As a prerequisite to all API operations, a `GraphRagConfig` object is required. It is the primary means to control the behavior of graphrag and can be instantiated from a `settings.yaml` configuration file.\n",
+    "\n",
+    "Please refer to the [CLI docs](https://microsoft.github.io/graphrag/cli/#init) for more detailed information on how to generate the `settings.yaml` file.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Generate a `GraphRagConfig` object\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# note that we expect this to fail on the deployed docs because the PROJECT_DIRECTORY is not set to a real location.\n",
+    "# if you run this notebook locally, make sure to point at a location containing your settings.yaml\n",
+    "graphrag_config = load_config(Path(PROJECT_DIRECTORY))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Indexing API\n",
+    "\n",
+    "_Indexing_ is the process of ingesting raw text data and constructing a knowledge graph. GraphRAG currently supports plaintext (`.txt`) and `.csv` file formats.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Build an index\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "index_result: list[PipelineRunResult] = await api.build_index(config=graphrag_config)\n",
+    "\n",
+    "# index_result is a list of workflows that make up the indexing pipeline that was run\n",
+    "for workflow_result in index_result:\n",
+    "    status = f\"error\\n{workflow_result.errors}\" if workflow_result.errors else \"success\"\n",
+    "    print(f\"Workflow Name: {workflow_result.workflow}\\tStatus: {status}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Query an index\n",
+    "\n",
+    "To query an index, several index files must first be read into memory and passed to the query API.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "entities = pd.read_parquet(f\"{PROJECT_DIRECTORY}/output/entities.parquet\")\n",
+    "communities = pd.read_parquet(f\"{PROJECT_DIRECTORY}/output/communities.parquet\")\n",
+    "community_reports = pd.read_parquet(\n",
+    "    f\"{PROJECT_DIRECTORY}/output/community_reports.parquet\"\n",
+    ")\n",
+    "\n",
+    "response, context = await api.global_search(\n",
+    "    config=graphrag_config,\n",
+    "    entities=entities,\n",
+    "    communities=communities,\n",
+    "    community_reports=community_reports,\n",
+    "    community_level=2,\n",
+    "    dynamic_community_selection=False,\n",
+    "    response_type=\"Multiple Paragraphs\",\n",
+    "    query=\"Who is Scrooge and what are his main relationships?\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The response object is the official reponse from graphrag while the context object holds various metadata regarding the querying process used to obtain the final response.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(response)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Digging into the context a bit more provides users with extremely granular information such as what sources of data (down to the level of text chunks) were ultimately retrieved and used as part of the context sent to the LLM model).\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pprint(context)  # noqa: T203"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "graphrag-monorepo",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/examples_notebooks/drift_search.ipynb

@@ -0,0 +1,210 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Copyright (c) 2024 Microsoft Corporation.\n",
+    "# Licensed under the MIT License."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "import pandas as pd\n",
+    "from graphrag.config.enums import ModelType\n",
+    "from graphrag.config.models.drift_search_config import DRIFTSearchConfig\n",
+    "from graphrag.config.models.language_model_config import LanguageModelConfig\n",
+    "from graphrag.language_model.manager import ModelManager\n",
+    "from graphrag.query.indexer_adapters import (\n",
+    "    read_indexer_entities,\n",
+    "    read_indexer_relationships,\n",
+    "    read_indexer_report_embeddings,\n",
+    "    read_indexer_reports,\n",
+    "    read_indexer_text_units,\n",
+    ")\n",
+    "from graphrag.query.structured_search.drift_search.drift_context import (\n",
+    "    DRIFTSearchContextBuilder,\n",
+    ")\n",
+    "from graphrag.query.structured_search.drift_search.search import DRIFTSearch\n",
+    "from graphrag.tokenizer.get_tokenizer import get_tokenizer\n",
+    "from graphrag_vectors.lancedb import LanceDBVectorStore\n",
+    "\n",
+    "INPUT_DIR = \"./inputs/operation dulce\"\n",
+    "LANCEDB_URI = f\"{INPUT_DIR}/lancedb\"\n",
+    "\n",
+    "COMMUNITY_REPORT_TABLE = \"community_reports\"\n",
+    "COMMUNITY_TABLE = \"communities\"\n",
+    "ENTITY_TABLE = \"entities\"\n",
+    "RELATIONSHIP_TABLE = \"relationships\"\n",
+    "COVARIATE_TABLE = \"covariates\"\n",
+    "TEXT_UNIT_TABLE = \"text_units\"\n",
+    "COMMUNITY_LEVEL = 2\n",
+    "\n",
+    "\n",
+    "# read nodes table to get community and degree data\n",
+    "entity_df = pd.read_parquet(f\"{INPUT_DIR}/{ENTITY_TABLE}.parquet\")\n",
+    "community_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_TABLE}.parquet\")\n",
+    "\n",
+    "print(f\"Entity df columns: {entity_df.columns}\")\n",
+    "\n",
+    "entities = read_indexer_entities(entity_df, community_df, COMMUNITY_LEVEL)\n",
+    "\n",
+    "# load description embeddings to an in-memory lancedb vectorstore\n",
+    "# to connect to a remote db, specify url and port values.\n",
+    "description_embedding_store = LanceDBVectorStore(\n",
+    "    db_uri=LANCEDB_URI,\n",
+    "    index_name=\"entity_description\",\n",
+    ")\n",
+    "description_embedding_store.connect()\n",
+    "\n",
+    "full_content_embedding_store = LanceDBVectorStore(\n",
+    "    db_uri=LANCEDB_URI,\n",
+    "    index_name=\"community_full_content\",\n",
+    ")\n",
+    "full_content_embedding_store.connect()\n",
+    "\n",
+    "print(f\"Entity count: {len(entity_df)}\")\n",
+    "entity_df.head()\n",
+    "\n",
+    "relationship_df = pd.read_parquet(f\"{INPUT_DIR}/{RELATIONSHIP_TABLE}.parquet\")\n",
+    "relationships = read_indexer_relationships(relationship_df)\n",
+    "\n",
+    "print(f\"Relationship count: {len(relationship_df)}\")\n",
+    "relationship_df.head()\n",
+    "\n",
+    "text_unit_df = pd.read_parquet(f\"{INPUT_DIR}/{TEXT_UNIT_TABLE}.parquet\")\n",
+    "text_units = read_indexer_text_units(text_unit_df)\n",
+    "\n",
+    "print(f\"Text unit records: {len(text_unit_df)}\")\n",
+    "text_unit_df.head()\n",
+    "\n",
+    "report_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_REPORT_TABLE}.parquet\")\n",
+    "reports = read_indexer_reports(report_df, community_df, COMMUNITY_LEVEL)\n",
+    "read_indexer_report_embeddings(reports, full_content_embedding_store)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "api_key = os.environ[\"GRAPHRAG_API_KEY\"]\n",
+    "\n",
+    "chat_config = LanguageModelConfig(\n",
+    "    api_key=api_key,\n",
+    "    type=ModelType.Chat,\n",
+    "    model_provider=\"openai\",\n",
+    "    model=\"gpt-4.1\",\n",
+    "    max_retries=20,\n",
+    ")\n",
+    "chat_model = ModelManager().get_or_create_chat_model(\n",
+    "    name=\"local_search\",\n",
+    "    model_type=ModelType.Chat,\n",
+    "    config=chat_config,\n",
+    ")\n",
+    "\n",
+    "tokenizer = get_tokenizer(chat_config)\n",
+    "\n",
+    "embedding_config = LanguageModelConfig(\n",
+    "    api_key=api_key,\n",
+    "    type=ModelType.Embedding,\n",
+    "    model_provider=\"openai\",\n",
+    "    model=\"text-embedding-3-large\",\n",
+    "    max_retries=20,\n",
+    ")\n",
+    "\n",
+    "text_embedder = ModelManager().get_or_create_embedding_model(\n",
+    "    name=\"local_search_embedding\",\n",
+    "    model_type=ModelType.Embedding,\n",
+    "    config=embedding_config,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "drift_params = DRIFTSearchConfig(\n",
+    "    primer_folds=1,\n",
+    "    drift_k_followups=3,\n",
+    "    n_depth=3,\n",
+    ")\n",
+    "\n",
+    "context_builder = DRIFTSearchContextBuilder(\n",
+    "    model=chat_model,\n",
+    "    text_embedder=text_embedder,\n",
+    "    entities=entities,\n",
+    "    relationships=relationships,\n",
+    "    reports=reports,\n",
+    "    entity_text_embeddings=description_embedding_store,\n",
+    "    text_units=text_units,\n",
+    "    tokenizer=tokenizer,\n",
+    "    config=drift_params,\n",
+    ")\n",
+    "\n",
+    "search = DRIFTSearch(\n",
+    "    model=chat_model, context_builder=context_builder, tokenizer=tokenizer\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "resp = await search.search(\"Who is agent Mercer?\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "resp.response"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(resp.context_data)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/examples_notebooks/global_search.ipynb

@@ -0,0 +1,262 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Copyright (c) 2024 Microsoft Corporation.\n",
+    "# Licensed under the MIT License."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "import pandas as pd\n",
+    "from graphrag.config.enums import ModelType\n",
+    "from graphrag.config.models.language_model_config import LanguageModelConfig\n",
+    "from graphrag.language_model.manager import ModelManager\n",
+    "from graphrag.query.indexer_adapters import (\n",
+    "    read_indexer_communities,\n",
+    "    read_indexer_entities,\n",
+    "    read_indexer_reports,\n",
+    ")\n",
+    "from graphrag.query.structured_search.global_search.community_context import (\n",
+    "    GlobalCommunityContext,\n",
+    ")\n",
+    "from graphrag.query.structured_search.global_search.search import GlobalSearch\n",
+    "from graphrag.tokenizer.get_tokenizer import get_tokenizer"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Global Search example\n",
+    "\n",
+    "Global search method generates answers by searching over all AI-generated community reports in a map-reduce fashion. This is a resource-intensive method, but often gives good responses for questions that require an understanding of the dataset as a whole (e.g. What are the most significant values of the herbs mentioned in this notebook?)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### LLM setup"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "api_key = os.environ[\"GRAPHRAG_API_KEY\"]\n",
+    "\n",
+    "config = LanguageModelConfig(\n",
+    "    api_key=api_key,\n",
+    "    type=ModelType.Chat,\n",
+    "    model_provider=\"openai\",\n",
+    "    model=\"gpt-4.1\",\n",
+    "    max_retries=20,\n",
+    ")\n",
+    "model = ModelManager().get_or_create_chat_model(\n",
+    "    name=\"global_search\",\n",
+    "    model_type=ModelType.Chat,\n",
+    "    config=config,\n",
+    ")\n",
+    "\n",
+    "tokenizer = get_tokenizer(config)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Load community reports as context for global search\n",
+    "\n",
+    "- Load all community reports in the `community_reports` table from GraphRAG, to be used as context data for global search.\n",
+    "- Load entities from the `entities` tables from GraphRAG, to be used for calculating community weights for context ranking. Note that this is optional (if no entities are provided, we will not calculate community weights and only use the rank attribute in the community reports table for context ranking)\n",
+    "- Load all communities in the `communities` table from the GraphRAG, to be used to reconstruct the community graph hierarchy for dynamic community selection."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# parquet files generated from indexing pipeline\n",
+    "INPUT_DIR = \"./inputs/operation dulce\"\n",
+    "COMMUNITY_TABLE = \"communities\"\n",
+    "COMMUNITY_REPORT_TABLE = \"community_reports\"\n",
+    "ENTITY_TABLE = \"entities\"\n",
+    "\n",
+    "# community level in the Leiden community hierarchy from which we will load the community reports\n",
+    "# higher value means we use reports from more fine-grained communities (at the cost of higher computation cost)\n",
+    "COMMUNITY_LEVEL = 2"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "community_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_TABLE}.parquet\")\n",
+    "entity_df = pd.read_parquet(f\"{INPUT_DIR}/{ENTITY_TABLE}.parquet\")\n",
+    "report_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_REPORT_TABLE}.parquet\")\n",
+    "\n",
+    "communities = read_indexer_communities(community_df, report_df)\n",
+    "reports = read_indexer_reports(report_df, community_df, COMMUNITY_LEVEL)\n",
+    "entities = read_indexer_entities(entity_df, community_df, COMMUNITY_LEVEL)\n",
+    "\n",
+    "print(f\"Total report count: {len(report_df)}\")\n",
+    "print(\n",
+    "    f\"Report count after filtering by community level {COMMUNITY_LEVEL}: {len(reports)}\"\n",
+    ")\n",
+    "\n",
+    "report_df.head()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Build global context based on community reports"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "context_builder = GlobalCommunityContext(\n",
+    "    community_reports=reports,\n",
+    "    communities=communities,\n",
+    "    entities=entities,  # default to None if you don't want to use community weights for ranking\n",
+    "    tokenizer=tokenizer,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Perform global search"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "context_builder_params = {\n",
+    "    \"use_community_summary\": False,  # False means using full community reports. True means using community short summaries.\n",
+    "    \"shuffle_data\": True,\n",
+    "    \"include_community_rank\": True,\n",
+    "    \"min_community_rank\": 0,\n",
+    "    \"community_rank_name\": \"rank\",\n",
+    "    \"include_community_weight\": True,\n",
+    "    \"community_weight_name\": \"occurrence weight\",\n",
+    "    \"normalize_community_weight\": True,\n",
+    "    \"max_tokens\": 12_000,  # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000)\n",
+    "    \"context_name\": \"Reports\",\n",
+    "}\n",
+    "\n",
+    "map_llm_params = {\n",
+    "    \"max_tokens\": 1000,\n",
+    "    \"temperature\": 0.0,\n",
+    "    \"response_format\": {\"type\": \"json_object\"},\n",
+    "}\n",
+    "\n",
+    "reduce_llm_params = {\n",
+    "    \"max_tokens\": 2000,  # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 1000-1500)\n",
+    "    \"temperature\": 0.0,\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "search_engine = GlobalSearch(\n",
+    "    model=model,\n",
+    "    context_builder=context_builder,\n",
+    "    tokenizer=tokenizer,\n",
+    "    max_data_tokens=12_000,  # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000)\n",
+    "    map_llm_params=map_llm_params,\n",
+    "    reduce_llm_params=reduce_llm_params,\n",
+    "    allow_general_knowledge=False,  # set this to True will add instruction to encourage the LLM to incorporate general knowledge in the response, which may increase hallucinations, but could be useful in some use cases.\n",
+    "    json_mode=True,  # set this to False if your LLM model does not support JSON mode.\n",
+    "    context_builder_params=context_builder_params,\n",
+    "    concurrent_coroutines=32,\n",
+    "    response_type=\"multiple paragraphs\",  # free form text describing the response type and format, can be anything, e.g. prioritized list, single paragraph, multiple paragraphs, multiple-page report\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "result = await search_engine.search(\"What is operation dulce?\")\n",
+    "\n",
+    "print(result.response)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# inspect the data used to build the context for the LLM responses\n",
+    "result.context_data[\"reports\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# inspect number of LLM calls and tokens\n",
+    "print(\n",
+    "    f\"LLM calls: {result.llm_calls}. Prompt tokens: {result.prompt_tokens}. Output tokens: {result.output_tokens}.\"\n",
+    ")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "graphrag",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/examples_notebooks/global_search_with_dynamic_community_selection.ipynb

@@ -0,0 +1,295 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Copyright (c) 2024 Microsoft Corporation.\n",
+    "# Licensed under the MIT License."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "import pandas as pd\n",
+    "from graphrag.config.enums import ModelType\n",
+    "from graphrag.config.models.language_model_config import LanguageModelConfig\n",
+    "from graphrag.language_model.manager import ModelManager\n",
+    "from graphrag.query.indexer_adapters import (\n",
+    "    read_indexer_communities,\n",
+    "    read_indexer_entities,\n",
+    "    read_indexer_reports,\n",
+    ")\n",
+    "from graphrag.query.structured_search.global_search.community_context import (\n",
+    "    GlobalCommunityContext,\n",
+    ")\n",
+    "from graphrag.query.structured_search.global_search.search import GlobalSearch"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Global Search example\n",
+    "\n",
+    "Global search method generates answers by searching over all AI-generated community reports in a map-reduce fashion. This is a resource-intensive method, but often gives good responses for questions that require an understanding of the dataset as a whole (e.g. What are the most significant values of the herbs mentioned in this notebook?)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### LLM setup"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from graphrag.tokenizer.get_tokenizer import get_tokenizer\n",
+    "\n",
+    "api_key = os.environ[\"GRAPHRAG_API_KEY\"]\n",
+    "\n",
+    "config = LanguageModelConfig(\n",
+    "    api_key=api_key,\n",
+    "    type=ModelType.Chat,\n",
+    "    model_provider=\"openai\",\n",
+    "    model=\"gpt-4.1\",\n",
+    "    max_retries=20,\n",
+    ")\n",
+    "model = ModelManager().get_or_create_chat_model(\n",
+    "    name=\"global_search\",\n",
+    "    model_type=ModelType.Chat,\n",
+    "    config=config,\n",
+    ")\n",
+    "\n",
+    "tokenizer = get_tokenizer(config)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Load community reports as context for global search\n",
+    "\n",
+    "- Load all community reports in the `community_reports` table from the indexing engine, to be used as context data for global search.\n",
+    "- Load entities from the `entities` tables from the indexing engine, to be used for calculating community weights for context ranking. Note that this is optional (if no entities are provided, we will not calculate community weights and only use the rank attribute in the community reports table for context ranking)\n",
+    "- Load all communities in the `communities` table from the indexing engine, to be used to reconstruct the community graph hierarchy for dynamic community selection."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# parquet files generated from indexing pipeline\n",
+    "INPUT_DIR = \"./inputs/operation dulce\"\n",
+    "COMMUNITY_TABLE = \"communities\"\n",
+    "COMMUNITY_REPORT_TABLE = \"community_reports\"\n",
+    "ENTITY_TABLE = \"entities\"\n",
+    "\n",
+    "# we don't fix a specific community level but instead use an agent to dynamicially\n",
+    "# search through all the community reports to check if they are relevant.\n",
+    "COMMUNITY_LEVEL = None"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "community_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_TABLE}.parquet\")\n",
+    "entity_df = pd.read_parquet(f\"{INPUT_DIR}/{ENTITY_TABLE}.parquet\")\n",
+    "report_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_REPORT_TABLE}.parquet\")\n",
+    "\n",
+    "communities = read_indexer_communities(community_df, report_df)\n",
+    "reports = read_indexer_reports(\n",
+    "    report_df,\n",
+    "    community_df,\n",
+    "    community_level=COMMUNITY_LEVEL,\n",
+    "    dynamic_community_selection=True,\n",
+    ")\n",
+    "entities = read_indexer_entities(\n",
+    "    entity_df, community_df, community_level=COMMUNITY_LEVEL\n",
+    ")\n",
+    "\n",
+    "print(f\"Total report count: {len(report_df)}\")\n",
+    "print(\n",
+    "    f\"Report count after filtering by community level {COMMUNITY_LEVEL}: {len(reports)}\"\n",
+    ")\n",
+    "\n",
+    "report_df.head()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Build global context with dynamic community selection\n",
+    "\n",
+    "The goal of dynamic community selection reduce the number of community reports that need to be processed in the map-reduce operation. To that end, we take advantage of the hierachical structure of the indexed dataset. We first ask the LLM to rate how relevant each level 0 community is with respect to the user query, we then traverse down the child node(s) if the current community report is deemed relevant.\n",
+    "\n",
+    "You can still set a `COMMUNITY_LEVEL` to filter out lower level community reports and apply dynamic community selection on the filtered reports.\n",
+    "\n",
+    "Note that the dataset is quite small, with only consist of 20 communities from 2 levels (level 0 and 1). Dynamic community selection is more effective when there are large amount of content to be filtered out."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "context_builder = GlobalCommunityContext(\n",
+    "    community_reports=reports,\n",
+    "    communities=communities,\n",
+    "    entities=entities,  # default to None if you don't want to use community weights for ranking\n",
+    "    tokenizer=tokenizer,\n",
+    "    dynamic_community_selection=True,\n",
+    "    dynamic_community_selection_kwargs={\n",
+    "        \"model\": model,\n",
+    "        \"tokenizer\": tokenizer,\n",
+    "    },\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Perform global search with dynamic community selection"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "context_builder_params = {\n",
+    "    \"use_community_summary\": False,  # False means using full community reports. True means using community short summaries.\n",
+    "    \"shuffle_data\": True,\n",
+    "    \"include_community_rank\": True,\n",
+    "    \"min_community_rank\": 0,\n",
+    "    \"community_rank_name\": \"rank\",\n",
+    "    \"include_community_weight\": True,\n",
+    "    \"community_weight_name\": \"occurrence weight\",\n",
+    "    \"normalize_community_weight\": True,\n",
+    "    \"max_tokens\": 12_000,  # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000)\n",
+    "    \"context_name\": \"Reports\",\n",
+    "}\n",
+    "\n",
+    "map_llm_params = {\n",
+    "    \"max_tokens\": 1000,\n",
+    "    \"temperature\": 0.0,\n",
+    "    \"response_format\": {\"type\": \"json_object\"},\n",
+    "}\n",
+    "\n",
+    "reduce_llm_params = {\n",
+    "    \"max_tokens\": 2000,  # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 1000-1500)\n",
+    "    \"temperature\": 0.0,\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "search_engine = GlobalSearch(\n",
+    "    model=model,\n",
+    "    context_builder=context_builder,\n",
+    "    tokenizer=tokenizer,\n",
+    "    max_data_tokens=12_000,  # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000)\n",
+    "    map_llm_params=map_llm_params,\n",
+    "    reduce_llm_params=reduce_llm_params,\n",
+    "    allow_general_knowledge=False,  # set this to True will add instruction to encourage the LLM to incorporate general knowledge in the response, which may increase hallucinations, but could be useful in some use cases.\n",
+    "    json_mode=True,  # set this to False if your LLM model does not support JSON mode.\n",
+    "    context_builder_params=context_builder_params,\n",
+    "    concurrent_coroutines=32,\n",
+    "    response_type=\"multiple paragraphs\",  # free form text describing the response type and format, can be anything, e.g. prioritized list, single paragraph, multiple paragraphs, multiple-page report\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "result = await search_engine.search(\"What is operation dulce?\")\n",
+    "\n",
+    "print(result.response)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# inspect the data used to build the context for the LLM responses\n",
+    "result.context_data[\"reports\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# inspect number of LLM calls and tokens in dynamic community selection\n",
+    "llm_calls = result.llm_calls_categories[\"build_context\"]\n",
+    "prompt_tokens = result.prompt_tokens_categories[\"build_context\"]\n",
+    "output_tokens = result.output_tokens_categories[\"build_context\"]\n",
+    "print(\n",
+    "    f\"Build context LLM calls: {llm_calls}. Prompt tokens: {prompt_tokens}. Output tokens: {output_tokens}.\"\n",
+    ")\n",
+    "# inspect number of LLM calls and tokens in map-reduce\n",
+    "llm_calls = result.llm_calls_categories[\"map\"] + result.llm_calls_categories[\"reduce\"]\n",
+    "prompt_tokens = (\n",
+    "    result.prompt_tokens_categories[\"map\"] + result.prompt_tokens_categories[\"reduce\"]\n",
+    ")\n",
+    "output_tokens = (\n",
+    "    result.output_tokens_categories[\"map\"] + result.output_tokens_categories[\"reduce\"]\n",
+    ")\n",
+    "print(\n",
+    "    f\"Map-reduce LLM calls: {llm_calls}. Prompt tokens: {prompt_tokens}. Output tokens: {output_tokens}.\"\n",
+    ")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "graphrag",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/examples_notebooks/index_migration_to_v1.ipynb

@@ -0,0 +1,250 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 62,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Copyright (c) 2024 Microsoft Corporation.\n",
+    "# Licensed under the MIT License."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Index Migration (pre-v1 to v1)\n",
+    "\n",
+    "This notebook is used to maintain data model parity with older indexes for version 1.0 of GraphRAG. If you have a pre-1.0 index and need to migrate without re-running the entire pipeline, you can use this notebook to only update the pieces necessary for alignment.\n",
+    "\n",
+    "NOTE: we recommend regenerating your settings.yml with the latest version of GraphRAG using `graphrag init`. Copy your LLM settings into it before running this notebook. This ensures your config is aligned with the latest version for the migration. This also ensures that you have default vector store config, which is now required or indexing will fail.\n",
+    "\n",
+    "WARNING: This will overwrite your parquet files, you may want to make a backup!\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 63,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# This is the directory that has your settings.yaml\n",
+    "# NOTE: much older indexes may have been output with a timestamped directory\n",
+    "# if this is the case, you will need to make sure the storage.base_dir in settings.yaml points to it correctly\n",
+    "PROJECT_DIRECTORY = \"<your project directory\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 64,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "\n",
+    "from graphrag.config.load_config import load_config\n",
+    "from graphrag.storage.factory import StorageFactory\n",
+    "\n",
+    "config = load_config(Path(PROJECT_DIRECTORY))\n",
+    "storage_config = config.output.model_dump()\n",
+    "storage = StorageFactory().create_storage(\n",
+    "    storage_type=storage_config[\"type\"],\n",
+    "    kwargs=storage_config,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 65,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def remove_columns(df, columns):\n",
+    "    \"\"\"Remove columns from a DataFrame, suppressing errors.\"\"\"\n",
+    "    df.drop(labels=columns, axis=1, errors=\"ignore\", inplace=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def get_community_parent(nodes):\n",
+    "    \"\"\"Compute the parent community using the node membership as a lookup.\"\"\"\n",
+    "    parent_mapping = nodes.loc[:, [\"level\", \"community\", \"title\"]]\n",
+    "    nodes = nodes.loc[:, [\"level\", \"community\", \"title\"]]\n",
+    "\n",
+    "    # Create a parent mapping by adding 1 to the level column\n",
+    "    parent_mapping[\"level\"] += 1  # Shift levels for parent relationship\n",
+    "    parent_mapping.rename(columns={\"community\": \"parent\"}, inplace=True)\n",
+    "\n",
+    "    # Merge the parent information back into the base DataFrame\n",
+    "    nodes = nodes.merge(parent_mapping, on=[\"level\", \"title\"], how=\"left\")\n",
+    "\n",
+    "    # Fill missing parents with -1 (default value)\n",
+    "    nodes[\"parent\"] = nodes[\"parent\"].fillna(-1).astype(int)\n",
+    "\n",
+    "    join = (\n",
+    "        nodes\n",
+    "        .groupby([\"community\", \"level\", \"parent\"])\n",
+    "        .agg({\"title\": list})\n",
+    "        .reset_index()\n",
+    "    )\n",
+    "    return join[join[\"community\"] > -1].loc[:, [\"community\", \"parent\"]]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from uuid import uuid4\n",
+    "\n",
+    "from graphrag_storage.tables.parquet_table_provider import ParquetTableProvider\n",
+    "\n",
+    "# Create table provider from storage\n",
+    "table_provider = ParquetTableProvider(storage)\n",
+    "\n",
+    "# First we'll go through any parquet files that had model changes and update them\n",
+    "# The new data model may have removed excess columns as well, but we will only make the minimal changes required for compatibility\n",
+    "\n",
+    "final_documents = await table_provider.read_dataframe(\"create_final_documents\")\n",
+    "final_text_units = await table_provider.read_dataframe(\"create_final_text_units\")\n",
+    "final_entities = await table_provider.read_dataframe(\"create_final_entities\")\n",
+    "final_nodes = await table_provider.read_dataframe(\"create_final_nodes\")\n",
+    "final_relationships = await table_provider.read_dataframe(\"create_final_relationships\")\n",
+    "final_communities = await table_provider.read_dataframe(\"create_final_communities\")\n",
+    "final_community_reports = await table_provider.read_dataframe(\n",
+    "    \"create_final_community_reports\"\n",
+    ")\n",
+    "\n",
+    "\n",
+    "# Documents renames raw_content for consistency\n",
+    "if \"raw_content\" in final_documents.columns:\n",
+    "    final_documents.rename(columns={\"raw_content\": \"text\"}, inplace=True)\n",
+    "final_documents[\"human_readable_id\"] = final_documents.index + 1\n",
+    "\n",
+    "# Text units just get a human_readable_id or consistency\n",
+    "final_text_units[\"human_readable_id\"] = final_text_units.index + 1\n",
+    "\n",
+    "# We renamed \"name\" to \"title\" for consistency with the rest of the tables\n",
+    "if \"name\" in final_entities.columns:\n",
+    "    final_entities.rename(columns={\"name\": \"title\"}, inplace=True)\n",
+    "remove_columns(\n",
+    "    final_entities, [\"name_embedding\", \"graph_embedding\", \"description_embedding\"]\n",
+    ")\n",
+    "\n",
+    "# Final nodes uses community for joins, which is now an int everywhere\n",
+    "final_nodes[\"community\"] = final_nodes[\"community\"].fillna(-1)\n",
+    "final_nodes[\"community\"] = final_nodes[\"community\"].astype(int)\n",
+    "remove_columns(\n",
+    "    final_nodes,\n",
+    "    [\n",
+    "        \"type\",\n",
+    "        \"description\",\n",
+    "        \"source_id\",\n",
+    "        \"graph_embedding\",\n",
+    "        \"entity_type\",\n",
+    "        \"top_level_node_id\",\n",
+    "        \"size\",\n",
+    "    ],\n",
+    ")\n",
+    "\n",
+    "# Relationships renames \"rank\" to \"combined_degree\" to be clear what the default ranking is\n",
+    "if \"rank\" in final_relationships.columns:\n",
+    "    final_relationships.rename(columns={\"rank\": \"combined_degree\"}, inplace=True)\n",
+    "\n",
+    "\n",
+    "# Compute the parents for each community, to add to communities and reports\n",
+    "parent_df = get_community_parent(final_nodes)\n",
+    "\n",
+    "# Communities previously used the \"id\" field for the Leiden id, but we've moved this to the community field and use a uuid for id like the others\n",
+    "if \"community\" not in final_communities.columns:\n",
+    "    final_communities[\"community\"] = final_communities[\"id\"].astype(int)\n",
+    "    final_communities[\"human_readable_id\"] = final_communities[\"community\"]\n",
+    "    final_communities[\"id\"] = [str(uuid4()) for _ in range(len(final_communities))]\n",
+    "if \"parent\" not in final_communities.columns:\n",
+    "    final_communities = final_communities.merge(parent_df, on=\"community\", how=\"left\")\n",
+    "if \"entity_ids\" not in final_communities.columns:\n",
+    "    node_mapping = (\n",
+    "        final_nodes\n",
+    "        .loc[:, [\"community\", \"id\"]]\n",
+    "        .groupby(\"community\")\n",
+    "        .agg(entity_ids=(\"id\", list))\n",
+    "    )\n",
+    "    final_communities = final_communities.merge(\n",
+    "        node_mapping, on=\"community\", how=\"left\"\n",
+    "    )\n",
+    "remove_columns(final_communities, [\"raw_community\"])\n",
+    "\n",
+    "# We need int for community and the human_readable_id copy for consistency\n",
+    "final_community_reports[\"community\"] = final_community_reports[\"community\"].astype(int)\n",
+    "final_community_reports[\"human_readable_id\"] = final_community_reports[\"community\"]\n",
+    "if \"parent\" not in final_community_reports.columns:\n",
+    "    final_community_reports = final_community_reports.merge(\n",
+    "        parent_df, on=\"community\", how=\"left\"\n",
+    "    )\n",
+    "\n",
+    "await table_provider.write_dataframe(\"create_final_documents\", final_documents)\n",
+    "await table_provider.write_dataframe(\"create_final_text_units\", final_text_units)\n",
+    "await table_provider.write_dataframe(\"create_final_entities\", final_entities)\n",
+    "await table_provider.write_dataframe(\"create_final_nodes\", final_nodes)\n",
+    "await table_provider.write_dataframe(\"create_final_relationships\", final_relationships)\n",
+    "await table_provider.write_dataframe(\"create_final_communities\", final_communities)\n",
+    "await table_provider.write_dataframe(\n",
+    "    \"create_final_community_reports\", final_community_reports\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from graphrag.callbacks.noop_workflow_callbacks import NoopWorkflowCallbacks\n",
+    "from graphrag.index.workflows.generate_text_embeddings import generate_text_embeddings\n",
+    "from graphrag_cache import create_cache\n",
+    "\n",
+    "# We only need to re-run the embeddings workflow, to ensure that embeddings\n",
+    "# for all required search fields are in place.\n",
+    "# We pass in the table_provider created earlier so that generate_text_embeddings\n",
+    "# reads the migrated tables we just wrote.\n",
+    "\n",
+    "callbacks = NoopWorkflowCallbacks()\n",
+    "cache = create_cache(config.cache)\n",
+    "\n",
+    "await generate_text_embeddings(\n",
+    "    config=config,\n",
+    "    table_provider=table_provider,\n",
+    "    cache=cache,\n",
+    "    callbacks=callbacks,\n",
+    ")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "graphrag",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/examples_notebooks/index_migration_to_v2.ipynb

@@ -0,0 +1,167 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Copyright (c) 2024 Microsoft Corporation.\n",
+    "# Licensed under the MIT License."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Index Migration (v1 to v2)\n",
+    "\n",
+    "This notebook is used to maintain data model parity with older indexes for version 2.0 of GraphRAG. If you have a pre-2.0 index and need to migrate without re-running the entire pipeline, you can use this notebook to only update the pieces necessary for alignment. If you have a pre-1.0 index, please run the v1 migration notebook first!\n",
+    "\n",
+    "NOTE: we recommend regenerating your settings.yml with the latest version of GraphRAG using `graphrag init`. Copy your LLM settings into it before running this notebook. This ensures your config is aligned with the latest version for the migration. This also ensures that you have default vector store config, which is now required or indexing will fail.\n",
+    "\n",
+    "WARNING: This will overwrite your parquet files, you may want to make a backup!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# This is the directory that has your settings.yaml\n",
+    "PROJECT_DIRECTORY = \"<your project directory>\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "\n",
+    "from graphrag.config.load_config import load_config\n",
+    "from graphrag.storage.factory import StorageFactory\n",
+    "\n",
+    "config = load_config(Path(PROJECT_DIRECTORY))\n",
+    "storage_config = config.output.model_dump()\n",
+    "storage = StorageFactory().create_storage(\n",
+    "    storage_type=storage_config[\"type\"],\n",
+    "    kwargs=storage_config,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def remove_columns(df, columns):\n",
+    "    \"\"\"Remove columns from a DataFrame, suppressing errors.\"\"\"\n",
+    "    df.drop(labels=columns, axis=1, errors=\"ignore\", inplace=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "from graphrag_storage.tables.parquet_table_provider import ParquetTableProvider\n",
+    "\n",
+    "# Create table provider from storage\n",
+    "table_provider = ParquetTableProvider(storage)\n",
+    "\n",
+    "final_documents = await table_provider.read_dataframe(\"create_final_documents\")\n",
+    "final_text_units = await table_provider.read_dataframe(\"create_final_text_units\")\n",
+    "final_entities = await table_provider.read_dataframe(\"create_final_entities\")\n",
+    "final_covariates = await table_provider.read_dataframe(\"create_final_covariates\")\n",
+    "final_nodes = await table_provider.read_dataframe(\"create_final_nodes\")\n",
+    "final_relationships = await table_provider.read_dataframe(\"create_final_relationships\")\n",
+    "final_communities = await table_provider.read_dataframe(\"create_final_communities\")\n",
+    "final_community_reports = await table_provider.read_dataframe(\n",
+    "    \"create_final_community_reports\"\n",
+    ")\n",
+    "\n",
+    "# we've renamed document attributes as metadata\n",
+    "if \"attributes\" in final_documents.columns:\n",
+    "    final_documents.rename(columns={\"attributes\": \"metadata\"}, inplace=True)\n",
+    "\n",
+    "# we're removing the nodes table, so we need to copy the graph columns into entities\n",
+    "graph_props = (\n",
+    "    final_nodes.loc[:, [\"id\", \"degree\", \"x\", \"y\"]].groupby(\"id\").first().reset_index()\n",
+    ")\n",
+    "final_entities = final_entities.merge(graph_props, on=\"id\", how=\"left\")\n",
+    "# we're also persisting the frequency column\n",
+    "final_entities[\"frequency\"] = final_entities[\"text_unit_ids\"].count()\n",
+    "\n",
+    "\n",
+    "# we added children to communities to eliminate query-time reconstruction\n",
+    "parent_grouped = final_communities.groupby(\"parent\").agg(\n",
+    "    children=(\"community\", \"unique\")\n",
+    ")\n",
+    "final_communities = final_communities.merge(\n",
+    "    parent_grouped,\n",
+    "    left_on=\"community\",\n",
+    "    right_on=\"parent\",\n",
+    "    how=\"left\",\n",
+    ")\n",
+    "# replace NaN children with empty list\n",
+    "final_communities[\"children\"] = final_communities[\"children\"].apply(\n",
+    "    lambda x: x if isinstance(x, np.ndarray) else []  # type: ignore\n",
+    ")\n",
+    "\n",
+    "# add children to the reports as well\n",
+    "final_community_reports = final_community_reports.merge(\n",
+    "    parent_grouped,\n",
+    "    left_on=\"community\",\n",
+    "    right_on=\"parent\",\n",
+    "    how=\"left\",\n",
+    ")\n",
+    "\n",
+    "# we renamed all the output files for better clarity now that we don't have workflow naming constraints from DataShaper\n",
+    "await table_provider.write_dataframe(\"documents\", final_documents)\n",
+    "await table_provider.write_dataframe(\"text_units\", final_text_units)\n",
+    "await table_provider.write_dataframe(\"entities\", final_entities)\n",
+    "await table_provider.write_dataframe(\"relationships\", final_relationships)\n",
+    "await table_provider.write_dataframe(\"covariates\", final_covariates)\n",
+    "await table_provider.write_dataframe(\"communities\", final_communities)\n",
+    "await table_provider.write_dataframe(\"community_reports\", final_community_reports)\n",
+    "\n",
+    "# delete all the old versions\n",
+    "await storage.delete(\"create_final_documents.parquet\")\n",
+    "await storage.delete(\"create_final_text_units.parquet\")\n",
+    "await storage.delete(\"create_final_entities.parquet\")\n",
+    "await storage.delete(\"create_final_nodes.parquet\")\n",
+    "await storage.delete(\"create_final_relationships.parquet\")\n",
+    "await storage.delete(\"create_final_covariates.parquet\")\n",
+    "await storage.delete(\"create_final_communities.parquet\")\n",
+    "await storage.delete(\"create_final_community_reports.parquet\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": ".venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/examples_notebooks/index_migration_to_v3.ipynb

@@ -0,0 +1,146 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Copyright (c) 2024 Microsoft Corporation.\n",
+    "# Licensed under the MIT License."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Index Migration (v2 to v3)\n",
+    "\n",
+    "This notebook is used to maintain data model parity with older indexes for version 3.0 of GraphRAG. If you have a pre-3.0 index and need to migrate without re-running the entire pipeline, you can use this notebook to only update the pieces necessary for alignment. If you have a pre-2.0 index, please run the v2 migration notebook first!\n",
+    "\n",
+    "NOTE: we recommend regenerating your settings.yml with the latest version of GraphRAG using `graphrag init`. Copy your LLM settings into it before running this notebook. This ensures your config is aligned with the latest version for the migration. The config changes from v2 to v3 are significant in places!\n",
+    "\n",
+    "WARNING: This will overwrite your parquet files, you may want to make a backup!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# This is the directory that has your settings.yaml\n",
+    "PROJECT_DIRECTORY = \"<your project directory>\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "\n",
+    "from graphrag.config.models.graph_rag_config import GraphRagConfig\n",
+    "from graphrag_common.config import load_config\n",
+    "from graphrag_storage.storage_factory import create_storage\n",
+    "\n",
+    "config = load_config(GraphRagConfig, config_path=Path(PROJECT_DIRECTORY))\n",
+    "storage = create_storage(config.output_storage)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def remove_columns(df, columns):\n",
+    "    \"\"\"Remove columns from a DataFrame, suppressing errors.\"\"\"\n",
+    "    df.drop(labels=columns, axis=1, errors=\"ignore\", inplace=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from graphrag_storage.tables.parquet_table_provider import ParquetTableProvider\n",
+    "\n",
+    "# Create table provider from storage\n",
+    "table_provider = ParquetTableProvider(storage)\n",
+    "\n",
+    "text_units = await table_provider.read_dataframe(\"text_units\")\n",
+    "\n",
+    "text_units[\"document_id\"] = text_units[\"document_ids\"].apply(lambda ids: ids[0])\n",
+    "remove_columns(text_units, [\"document_ids\"])\n",
+    "\n",
+    "await table_provider.write_dataframe(\"text_units\", text_units)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Update settings.yaml\n",
+    "If you have left the default settings for your vector store schema, you may need to set explicit values that map each embedding type to a vector schema name. If you have already customized your vector store schema it may not be necessary.\n",
+    "\n",
+    "Old default index names:\n",
+    "- default-text_unit-text\n",
+    "- default-entity-description\n",
+    "- default-community-full_content\n",
+    "\n",
+    "(if you left all of the defaults, check your output/lancedb folder to confirm the above)\n",
+    "\n",
+    "v3 versions are:\n",
+    "- text_unit_text\n",
+    "- entity_description\n",
+    "- community_full_content\n",
+    "\n",
+    "Therefore, with a v2 index need to explicitly set the old index names so it connects correctly. We no longer support the \"prefix\" - you can just set an explicit index_name for each embedding.\n",
+    "\n",
+    "NOTE: we are also setting the default vector_size for each index below, under the assumption that you are using a prior default with 1536 dimensions. Our new default of text-embedding-3-large has 3072 dimensions, which will be populated as the default if unset. Again, if you have a more complicated situation you may want to manually configure this.\n",
+    "\n",
+    "Here is an example of the new vector store config block that you may need in your settings.yaml:\n",
+    "\n",
+    "```yaml\n",
+    "vector_store:\n",
+    "  type: lancedb\n",
+    "  db_uri: output/lancedb\n",
+    "  index_schema:\n",
+    "    text_unit_text:\n",
+    "      index_name: default-text_unit-text\n",
+    "      vector_size: 1536\n",
+    "    entity_description:\n",
+    "      index_name: default-entity-description\n",
+    "      vector_size: 1536\n",
+    "    community_full_content:\n",
+    "      index_name: default-community-full_content\n",
+    "      vector_size: 1536\n",
+    "```\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/examples_notebooks/input_documents.ipynb

@@ -0,0 +1,193 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Copyright (c) 2024 Microsoft Corporation.\n",
+    "# Licensed under the MIT License."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Example of indexing from an existing in-memory dataframe\n",
+    "\n",
+    "Newer versions of GraphRAG let you submit a dataframe directly instead of running through the input processing step. This notebook demonstrates with regular or update runs.\n",
+    "\n",
+    "If performing an update, the assumption is that your dataframe contains only the new documents to add to the index.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "from pprint import pprint\n",
+    "\n",
+    "import graphrag.api as api\n",
+    "import pandas as pd\n",
+    "from graphrag.config.load_config import load_config\n",
+    "from graphrag.index.typing.pipeline_run_result import PipelineRunResult"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "PROJECT_DIRECTORY = \"<your project directory>\"\n",
+    "UPDATE = False\n",
+    "FILENAME = \"new_documents.parquet\" if UPDATE else \"<original_documents>.parquet\"\n",
+    "inputs = pd.read_parquet(f\"{PROJECT_DIRECTORY}/input/{FILENAME}\")\n",
+    "# Only the bare minimum for input. These are the same fields that would be present after the load_input_documents workflow\n",
+    "inputs = inputs.loc[:, [\"id\", \"title\", \"text\", \"creation_date\"]]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Generate a `GraphRagConfig` object\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "graphrag_config = load_config(Path(PROJECT_DIRECTORY))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Indexing API\n",
+    "\n",
+    "_Indexing_ is the process of ingesting raw text data and constructing a knowledge graph. GraphRAG currently supports plaintext (`.txt`) and `.csv` file formats.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Build an index\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "index_result: list[PipelineRunResult] = await api.build_index(\n",
+    "    config=graphrag_config, input_documents=inputs, is_update_run=UPDATE\n",
+    ")\n",
+    "\n",
+    "# index_result is a list of workflows that make up the indexing pipeline that was run\n",
+    "for workflow_result in index_result:\n",
+    "    status = f\"error\\n{workflow_result.errors}\" if workflow_result.errors else \"success\"\n",
+    "    print(f\"Workflow Name: {workflow_result.workflow}\\tStatus: {status}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": []
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Query an index\n",
+    "\n",
+    "To query an index, several index files must first be read into memory and passed to the query API.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "entities = pd.read_parquet(f\"{PROJECT_DIRECTORY}/output/entities.parquet\")\n",
+    "communities = pd.read_parquet(f\"{PROJECT_DIRECTORY}/output/communities.parquet\")\n",
+    "community_reports = pd.read_parquet(\n",
+    "    f\"{PROJECT_DIRECTORY}/output/community_reports.parquet\"\n",
+    ")\n",
+    "\n",
+    "response, context = await api.global_search(\n",
+    "    config=graphrag_config,\n",
+    "    entities=entities,\n",
+    "    communities=communities,\n",
+    "    community_reports=community_reports,\n",
+    "    community_level=2,\n",
+    "    dynamic_community_selection=False,\n",
+    "    response_type=\"Multiple Paragraphs\",\n",
+    "    query=\"What are the top five themes of the dataset?\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The response object is the official reponse from graphrag while the context object holds various metadata regarding the querying process used to obtain the final response.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(response)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Digging into the context a bit more provides users with extremely granular information such as what sources of data (down to the level of text chunks) were ultimately retrieved and used as part of the context sent to the LLM model).\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pprint(context)  # noqa: T203"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "graphrag-monorepo",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/examples_notebooks/inputs/operation dulce/ABOUT.md

@@ -0,0 +1,3 @@
+# About
+
+This document (Operation Dulce) is an AI-generated science fiction novella, included here for the purposes of providing a starting point for notebook experimentation.

docs/examples_notebooks/inputs/operation dulce/Operation Dulce v2 1 1.md

@@ -0,0 +1,970 @@
+# Operation: Dulce
+
+## Chapter 1
+
+The thrumming of monitors cast a stark contrast to the rigid silence enveloping the group. Agent Alex Mercer, unfailingly determined on paper, seemed dwarfed by the enormity of the sterile briefing room where Paranormal Military Squad's elite convened. With dulled eyes, he scanned the projectors outlining their impending odyssey into Operation: Dulce.
+
+“I assume, Agent Mercer, you’re not having second thoughts?” It was Taylor Cruz’s voice, laced with an edge that demanded attention.
+
+Alex flickered a strained smile, still thumbing his folder's corner. "Of course not, Agent Cruz. Just trying to soak in all the details." The compliance in his tone was unsettling, even to himself.
+
+Jordan Hayes, perched on the opposite side of the table, narrowed their eyes but offered a supportive nod. "Details are imperative. We’ll need your clear-headedness down there, Mercer."
+
+A comfortable silence, the kind that threaded between veterans of shared secrets, lingered briefly before Sam Rivera, never one to submit to quiet, added, "I’ve combed through the last transmission logs. If anyone can make sense of the anomalies, it’s going to be the two of you."
+
+Taylor snorted dismissively. “Focus, people. We have protocols for a reason. Speculation is counter-productive.” The words 'counter-productive' seemed to hang in the air, a tacit reprimand directed at Alex.
+
+Feeling the weight of his compliance conflicting with his natural inclination to leave no stone unturned, Alex straightened in his seat. "I agree, Agent Cruz. Protocol is paramount," he said, meeting Taylor's steely gaze. It was an affirmation, but beneath it lay layers of unspoken complexities that would undoubtedly unwind with time.
+
+Alex's submission, though seemingly complete, didn't escape Jordan, who tilted their head ever so slightly, their eyes revealing a spark of understanding. They knew well enough the struggle of aligning personal convictions with overarching missions. As everyone began to collect their binders and prepare for departure, a quiet resolve took form within Alex, galvanized by the groundwork laid by their interactions. He may have spoken in compliance, but his determination had merely taken a subtler form — one that wouldn't surrender so easily to the forthcoming shadows.
+
+\*
+
+Dr. Jordan Hayes shuffled a stack of papers, their eyes revealing a tinge of skepticism at Taylor Cruz's authoritarian performance. _Protocols_, Jordan thought, _are just the framework, the true challenges we're about to face lie well beyond the boundaries of any protocol._ They cleared their throat before speaking, tone cautious yet firm, "Let's remember, the unknown variables exceed the known. We should remain adaptive."
+
+A murmur of agreement echoed from Sam Rivera, who leaned forward, lacing their fingers together as if weaving a digital framework in the air before them, "Exactly, adaptability could be the key to interpreting the signal distortions and system malfunctions. We shouldn't discount the… erratic."
+
+Their words hung like an electric charge in the room, challenging Taylor's position with an inherent truth. Cruz’s jaw tightened almost imperceptibly, but the agent masked it with a small nod, conceding to the omnipresent threat of the unpredictable. 
+
+Alex glanced at Jordan, who never looked back, their gaze fixed instead on a distant point, as if envisioning the immense dark corridors they were soon to navigate in Dulce. Jordan was not one to embrace fantastical theories, but the air of cautious calculation betrayed a mind bracing for confrontation with the inexplicable, an internal battle between the evidence of their research and the calculating skepticism that kept them alive in their field.
+
+The meeting adjourned with no further comments, the team members quietly retreading the paths to their personal preparations. Alex, trailing slightly behind, observed the others. _The cautious reserve Jordan wears like armor doesn't fool me_, he thought, _their analytical mind sees the patterns I do. And that's worth more than protocol. That's the connection we need to survive this._
+
+As the agents dispersed into the labyrinth of the facility, lost in their thoughts and preparations, the base's halogen lights flickered, a brief and unnoticed harbingers of the darkness to come.
+
+\*
+
+A deserted corridor inside the facility stretched before Taylor Cruz, each footstep rhythmic and precise. Cruz, ambitious and meticulous, eyed the troops passing by with a sardonic tilt of the lips. Obedience—it was as much a tool as any weapon in the arsenal, and Cruz wielded it masterfully. To them, it was another step toward unfettered power within the dark bowels of the military complex.
+
+Inside a secluded equipment bay, Cruz began checking over gear with mechanical efficiency. They traced fingers over the sleek surface of an encrypted radio transmitter. "If protocols are maintained," said Cruz aloud, rehearsing the speech for their subordinates, "not only will we re-establish a line of communication with Dulce, but we shall also illuminate the darkest secrets it conceals."
+
+Agent Hayes appeared in the doorway, arms crossed and a knowing glint in their eyes. "You do understand," Jordan began, the words measured and probing, "that once we're in the depths, rank gives way to survival instincts. It's not about commands—it's empowerment through trust."
+
+The sentiment snagged on Cruz's armor of confidence, probing at the insecurities festering beneath. Taylor offered a brief nod, perhaps too curt, but enough to acknowledge Jordan's point without yielding ground. "Trust," Cruz mused, "or the illusion thereof, is just as potent."
+
+Silence claimed the space between them, steeped in the reality of the unknown dangers lurking in the shadows of the mission. Cruz diligently returned to the equipment, the act a clear dismissal.
+
+Not much later, Cruz stood alone, the hollow echo of the bay a stark reminder of the isolation that power often wrought. With each checked box, their resolve steeled further, a silent vow to usher their team through the abyss—whatever it might hold—and emerge enshrined in the respect they so deeply craved.
+
+## Chapter 2
+
+Sam Rivera sat alone in a cramped office, the hum of a dozen servers murmuring a digital lullaby in the background. Surrounded by the glow of multiple screens, their eyes danced across lines of code and intercepted comm signals from Dulce — a kaleidoscope of data that their curious and isolated mind hungered to decrypt.
+
+To an outsider, it might have looked like obsession, this fervent quest for answers. But to Sam, it was a dance — a give and take with the mysteries of the universe. Their fingers paused over the keyboard as they leaned back in the chair, whispering to thin air, "What secrets are you hiding from us?"
+
+The stillness of the room broke with the unexpected arrival of Alex Mercer, whose encroaching shadow loomed over Sam's workspace. The cybersecurity expert craned their neck upwards, met by the ever-so-slight furrow in Alex's brow. "Got a minute, Rivera?"
+
+"Always," Sam said, a smile surfacing as they swiveled to face their mentor more directly. _He has that look — like something's not sitting right with him,_ they noted inwardly.
+
+Alex hesitated, weighing his words carefully. "Our tech is top-tier, but the silence from Dulce... It's not just technology that will see us through, it's intuition and... trust." His gaze pierced through the digital haze, trying to instill something more profound than advice.
+
+Sam regarded Alex for a moment, the sincerity in his voice resonating with their own unspoken desire to prove their worth. "Intuition," they mirrored thoughtfully. "I guess sometimes the numbers don't have all the answers."
+
+Their shared silence held a newfound understanding, a recognition that between the ones and zeros, it was their combined human insights that might prevail against the impossible. As Alex turned to leave, Sam's eyes drifted back to the screens, now seeing them not as barriers to isolate behind, but as windows into the vast and enigmatic challenge that awaited their team.
+
+Outside the office, the persistent buzz of activity in the facility belied the unease that gripped its inhabitants. A restlessness that nibbled on the edges of reality, as though forewarning of the threshold they were soon to cross — from the known into the realm of cosmic secrets and silent threats.
+
+\*
+
+Shadows played against the walls of the cramped underground meeting room, where Alex Mercer stood gazing at the concealed elevator that would deliver them into the bowels of Dulce base. The air was thick, every breath laced with the weight of impending confrontation, the kind one feels when stepping into a legend. Though armed with an array of advanced weaponry and gear, there was an unshakeable sense that they were delving into a conflict where the physical might be of little consequence.
+
+"I know what you're thinking," Jordan Hayes remarked, approaching Mercer. Their voice was low, a blend of confidence and hidden apprehension. "This feels like more than a rescue or reconnaissance mission, doesn't it?"
+
+Alex turned, his features a mask of uneasy resolve. "It's like we're being pulled into someone else’s game. Not just observers or participants, but... pawns."
+
+Jordan gave a short nod, their analytical mind colliding with the uncertain dynamics of this operation. "I've felt that way since the briefing. Like there's a layer we’re not seeing. And yet, we have no choice but to play along." Their eyes locked with Alex's, silently exchanging a vow to remain vigilant.
+
+"You two need to cut the philosophical chatter. We have positions to secure," Taylor Cruz interjected sharply, stepping into their exchange. The authority in Taylor's voice brooked no argument; it was their way of pulling everyone back to the now.
+
+Alex's response was measured, more assertive than moments ago. "Acknowledged, Agent Cruz," he replied, his voice steadier, mirroring the transformation brewing within. He gripped his rifle with a newfound firmness. "Let's proceed."
+
+As they congregated at the elevator, a tension palpable, Sam Rivera piped in with a tone of balanced levity, "Hope everyone’s brought their good luck charms. Something tells me we’re going to need all the help we can get."
+
+Their laughter served as a brief respite from the gravity of their mission, a shared moment that reinforced their common humanity amidst the unknowable. Then, as one, they stepped into the elevator. The doors closed with a silent hiss, and they descended into the darkness together, aware that when they returned, if they returned, none of them would be the same.
+
+\*
+
+The sense of foreboding hung heavier than the darkness that the artificial lights of the elevator shaft failed to fully penetrate. The team was descending into the earth, carrying with them not only the weight of their equipment but also the silent pressure of the invisible war they were about to fight—a war that seemed to edge away from physicality and into the unnervingly psychological.
+
+As they descended, Dr. Jordan Hayes couldn't help but muse over the layers of data that could wait below, now almost longing for the comfort of empirical evidence. _To think that this reluctance to accept other possibilities may have been my biggest blind spot,_ Jordan contemplated, feeling the hard shell of skepticism begin to crack.
+
+Alex caught Jordan's reflective gaze and leaned in, his voice barely a murmur over the hum of the elevator. "Once we're down there, keep that analytical edge sharp. You see through the mazes of the unexplained better than anyone."
+
+The compliment was unexpected and weighed differently than praise from others. This was an acknowledgment from someone who stood on the front lines of the unknown with eyes wide open. "Thank you, Alex," Jordan said, the words carrying a trace of newfound assertiveness. "You can count on me."
+
+The exchange was cut short by a shudder that ran through the elevator, subtle, but enough to make them instinctively hold their breaths. It wasn't the mechanical stutter of old gears but a vibration that seemed to emanate from the very walls of the shaft—a whisper of something that defied natural explanation.
+
+Cruz was the first to react, all business despite the shadow that crossed their expression. "Systems check. Now," they barked out, masking the moment of disquiet with swift command.
+
+Every agent checked their gear, sending confirmation signals through their comms, creating a chorus of electronic beeps that promised readiness. But there was an unspoken question among them: was their technology, their weaponry, their protocols sufficient for what awaited them or merely a fragile comfort?
+
+Against the gravity of the silence that was once again closing in, Sam's voice crackled through, only half-jest. "I'd laugh if we run into Martians playing poker down there—just to lighten the mood, you know?"
+
+Despite—or perhaps because of—the oddity of the moment, this elicited a round of chuckles, an audible release of tension that ran counterpoint to the undercurrent of anxiety coursing through the team.
+
+As the elevator came to a halting, eerie calm at the sub-level, the group stepped off, finding themselves at the threshold of Dulce's mysterious halls. They stood in a tight pack, sharing a cautious glance before fanning out into the unknown, each one acutely aware that the truth was inevitably intertwined with danger.
+
+Into the depths of Dulce, the team advanced, their silence now a shared testament to the camaraderie born of facing the abyss together—and the steel resolve to uncover whatever horrors lay hidden in its shadows.
+
+\*
+
+The weight of the thick metal door closing behind them reverberated through the concrete hallway, marking the final threshold between the familiar world above and the strangeness that lay beneath. Dulce base, a name that had been whispered in the wind-blown deserts above and in the shadowed corners of conspiracy forums, now a tangible cold reality that they could touch — and that touched them back with a chill.
+
+Like lambs led to an altar of alien deities, so did Agents Alex Mercer, Jordan Hayes, Taylor Cruz, and Sam Rivera proceed, their movements measured, their senses heightened. The air was still, almost respectful of the gravity of their presence. Their torch beams sliced through the darkness, uncovering steel doors with warnings that spoke of top secrets and mortal dangers.
+
+Taylor Cruz, stepping firmly into the role of de facto leader, set a brisk pace. "Eyes sharp, people. Comms check, every thirty seconds," Taylor ordered, their voice echoing slightly before being swallowed by the surrounding silence.
+
+Sam, fiddling with a handheld device aimed at detecting electronic anomalies, offered a murmured "Copy that," their usual buoyancy dimmed by the oppressive atmosphere.
+
+It was Jordan Hayes who paused at an innocuous looking panel, nondescript amongst the gauntlet of secured doorways. "Mercer, Rivera, come see this," Jordan’s voice was marked with a rare hint of urgency.
+
+Alex joined Jordan's side, examining the panel which, at a mere glance, seemed just another part of the base's infrastructure. Yet, to the trained eye, it appeared out of place—a facade.
+
+Jordan explained their reasoning as Sam approached, instinctively understanding the significance of what lay beneath, "This panel is a recent addition — covering something they didn't want found."
+
+Before Alex could respond, the soft whir of an approaching drone cut through their muffled exchange. Taylor had looped back upon hearing the commotion. "Explanations later. We can't afford to attract..." Cruz’s voice trailed off as the small airborne device came into view, its sensors locked onto the group.
+
+Sam was the first to react, their tech-savvy mind already steps ahead. "I've got this," they declared, fingers flying over the controls of their own gadgetry to ward off the impending threat.
+
+The drone lingered, its scan seeming more curious than hostile. But within moments, courtesy of Sam's interference, the little sentinel drifted away, retreating into the shadows as if accepting a silent truce. The crew exhaled, a moment of collective relief palpable in the air.
+
+Cruz squared their shoulders, clearly ruffled but not conceding any ground. "Move out," they directed, a hint more forceful than before. "And Rivera, keep that trick handy."
+
+The team pressed onward, the quiet now filled with the soft beeps of regular comms checks, their pace undeterred by the confrontation. Yet, every agent held a renewed sense of wariness, their trust in one another deepening with the knowledge that the base—its technology, its secrets—was alive in a way they hadn't fully anticipated.
+
+As they converged upon a central hub, the imposing doors to the mainframe room stood ajar — an invitation or a trap, neither option comforting. Without a word, they fortified their resolve and stepped through the threshold, where the dim glow of operational LED lights and the distant hum of machinery hinted at Dulce’s still-beating heart.
+
+Solemnly, yet unmistakably together, they moved deeper into the heart of the enigma, ready to unmask the lifeforce of Dulce base or confront whatever existential threat lay in wait. It was in that unwavering march towards the unknown that their destinies were forever cemented to the legacy of Operation: Dulce.
+
+## Chapter 3
+
+The thrumming of monitors cast a stark contrast to the rigid silence enveloping the group. Agent Alex Mercer, unfailingly determined on paper, seemed dwarfed by the enormity of the sterile briefing room where Paranormal Military Squad's elite convened. With dulled eyes, he scanned the projectors outlining their impending odyssey into Operation: Dulce.
+
+\*
+
+The cooling vents hummed in a monotonous drone, but it was the crackle of the comms system coming to life that cut through the lab’s tension. Dr. Jordan Hayes hovered over a table arrayed with alien technology, their fingers delicately probing the enigmatic circuitry retrieved from the crash site. Agent Alex Mercer watched, admiration blooming in silent solidarity for Jordan's deft touch and unspoken drive.
+
+Jordan, always composed, only allowed the faintest furrow of concentration to mar their brow. "What we understand about physics..." they muttered, trailing off as they realigned a translucent component. The device emitted a low pulse, causing Jordan to still. "Could be fundamentally changed by this."
+
+A calculated risk—that's what this was. And for a person of science, a gamble was worth the potential paradigm shift.
+
+"I’ve been thinking," Alex started, his eyes still fixed on the immediately tangible mystery before them. "About what’s at stake here. Not the mission parameters, but what this means for us—humanity."
+
+Jordan glanced up, meeting his eyes just long enough to convey the shared enormity of their situation; the career-defining glory and existential dread entwined. "The quest for understanding always comes at a price. We're standing on the precipice of knowledge that could either elevate us or condemn us."
+
+The charged air between them spiked as Taylor Cruz’s brusque tones sliced through their reverie. "Hayes, Mercer, this isn't philosophy hour. Focus on the task. We need actionable intel, not daydreams."
+
+With a sound of restrained acknowledgment, Jordan returned their gaze to the device, while Alex clenched his jaw, the buzz of frustration dull against the backdrop of Taylor's authoritarian certainty. It was this competitive undercurrent that kept him alert, the sense that his and Jordan's shared commitment to discovery was an unspoken rebellion against Cruz's narrowing vision of control and order.
+
+Then Taylor did something unexpected. They paused beside Jordan and, for a moment, observed the device with something akin to reverence. “If this tech can be understood..." Taylor said, their voice quieter, "It could change the game for us. For all of us.”
+
+The underlying dismissal earlier seemed to falter, replaced by a glimpse of reluctant respect for the gravity of what lay in their hands. Jordan looked up, and for a fleeting heartbeat, their eyes locked with Taylor's, a wordless clash of wills softening into an uneasy truce.
+
+It was a small transformation, barely perceptible, but one that Alex noted with an inward nod. They had all been brought here by different paths and for different reasons. Yet, beneath the veneer of duty, the enticement of the vast unknown pulled them inexorably together, coalescing their distinct desires into a shared pulse of anticipation.
+
+Marshaled back to the moment by the blink of lights and whir of machinery, they refocused their efforts, each movement sharpened by the knowledge that beyond understanding the unearthly artifacts, they might be piecing together the future of their species.
+
+\*
+
+Amidst the sterility of the briefing room, the liminal space between the facts laid out and the hidden truths, sat Sam Rivera, his demeanor an artful balance of focus and a casual disguise of his razor-sharp talent with technology. Across from him, Alex Mercer lingered in thought, the mental cogs turning as each file on Dulce stirred more than curiosity—it beckoned to a past both honored and burdensome.
+
+"You've been quiet, Sam," Alex noted, catching the younger man's contemplative gaze. "Your take on these signal inconsistencies?"
+
+There was a respect in Alex's tone, though a respectful distance remained—a gulf of experience and a hint of protective mentorship that stood between them. Sam nodded, recognizing the space afforded to him, and he couldn't help but feel the weight of expectation pressing upon his shoulders. It wasn't just the mission that was immense, it was the trust being placed in him.
+
+"The patterns are... off," Sam admitted, hesitant but driven. "If I'm right, what we're looking at isn't random—it's a structured anomaly. We need to be ready for anything." 
+
+Alex's eyes brightened with a subtle approval that crossed the distance like a silent nod. "Good. Keen eyes will keep us ahead—or at least not blindsided," he said, affirming the belief that inscribed Sam's role as more than the tech personnel—he was to be a guiding intellect in the heart of uncertainty.
+
+Their exchange was cut short by Taylor Cruz's abrupt arrival, his gait brimming with a robust confidence that veiled the sharp undercurrents of his striving nature. "Time to gear up. Dulce waits for no one," Taylor announced, his voice carrying an iron resolve that knew the costs of hesitation—though whether the cost was calculated in human or career terms was an ambiguity he wore like a badge of honor.
+
+As Sam and Alex nodded in unison, the icy chasm of hierarchy and cryptic protocols seemed momentarily to bridge over with an understanding—this mission was convergence, a nexus point that would challenge each of their motives and strength.
+
+They filed out of the briefing room, their footsteps synchronized, a rhythm that spoke volumes of the unknown cadence they would soon march to within the base's veins. For Alex Mercer, the link with Sam Rivera, though distant, was now poised with a mutuality ready to be tested; for Taylor Cruz, the initiative pulsed like a heartbeat, anticipation thinly veiled behind a mask of duty.
+
+In the midst of the descent, they were each alone yet irrevocably joined, stepping closer towards the volatile embrace of Operation: Dulce.
+
+## Chapter 4
+
+The corridors of the Dulce military base were as silent as a tomb and twice as chilling. Alex Mercer walked with a surety that belied his bubbling undercurrents of doubt. The briefing had been definitive, sturdy pillars of facts and protocols, yet as he ventured deeper, the ominous atmosphere gnawed at him—a stark reminder of how much remained unknown.
+
+Jordan Hayes trailed a few steps behind, their detached exterior breaking for a moment as they caught up to Alex. "What's on your mind?" Jordan asked, their astuteness cutting through the unspoken tension.
+
+Alex glanced back at them. This place was a puzzle, a treacherous labyrinth where the walls whispered secrets, and among them, he sensed a call to question, to challenge the narrative they'd been sold. "The silence here... It's almost as if the base is waiting for something—or someone."
+
+"Just stay sharp, Mercer," Jordan cautioned, yet their eyes lingered on the quietude around them, conceiving the same shadow of doubt that unsettled Alex.
+
+Before they could delve into further discussion, the distinctive click of a safety catch echoed in the hollow space. Both agents turned to find Taylor Cruz standing resolute, primed for combat. Taylor's gaze was scrutinizing and cold, a stark contrast to the growing unease that smoldered silently amongst the rest.
+
+"Chatter is a liability," Taylor snapped, with a commanding flair that bordered on tyrannical. "We move forward, eyes open, mouths shut."
+
+Alex felt the tight grip of compliance strangle his gut, a lesson learned under the hard tutelage of rank and order. But here, in the bowels of Dulce, those instincts began to wane, the imperative to adhere now conflicting with the pressing urgency to confront the shadows they were enmeshed in.
+
+Then, unexpectedly, the lights flickered, a power fluctuation—or a sign? Alex's hand instinctively went to his sidearm, his mindset shifting from soldier to skeptic. The base, with its unyielding coldness, had just given them their first nudge into the realm of the speculative, an invitation to peel back the veneer of reality.
+
+"We should consider all possibilities," Alex murmured, more to himself than the others, his voice a barely audible breath against the sterile air of the complex.
+
+Taylor's posture stiffened at the challenge, yet their response was uncharacteristically reserved, notable in its lack of rebuke. "Agreed. For now, keep moving. But stay vigilant."
+
+A surprise—an echo of agreement from the last person Alex expected it from. And there it was, the glimpse of a wrinkle in the unyielding fabric of command, a hint that perhaps they were all starting to sense the strangeness that permeated this place.
+
+Progressing with determined steps, the trio moved deeper, silently acknowledging the evolution of their predicament. It was a small yet transformative concession to the unknown forces at play, an acknowledgment from each agent that, despite their disparate goals and ideals, the true nature of the Dulce base was an enigma that would forge new paths through their convictions.
+
+As they reached the central communications hub, the truth that awaited them lurked in the shadows, its eyes unseen but felt by all. The walls didn't just whisper now; they spoke in tones only the brave—or the foolish—would dare to listen to.
+
+\*
+
+The subterranean silence of Dulce was an oppressive entity of its own, wrapping the team in a cloak of uneasiness as they pressed on through the dimly lit corridor. Jordan Hayes found themselves contemplating the ramifications of each step taken into this suspended world, where the sterile air seemed to mock the gravity of their predicament. The closer they got to the communication hub, the more Jordan's mind wandered toward the realm of the inexplicable.
+
+Beside Jordan, Alex Mercer moved forward with deliberation, his gaze scanning the heavy utility doors they passed—one of which was partially ajar, beckoning them with its darkness. "After you, Dr. Hayes," Alex said, gesturing toward the mysterious opening. A hint of shared understanding passed between them; knowledge was the guiding star of this mission as much as confrontation or recovery.
+
+Jordan peered inside, the beam from their flashlight slicing through the obscurity. The room beyond was a chaotic cascade of papers, overturned furniture, and the particular kind of disorder born from hasty evacuation—or something far more sinister.
+
+"It's like they vanished in the middle of something urgent," Alex murmured, his voice tight with a mix of concern and anticipation. He began to sift through the scattered reports, each page a potential clue to the enigmatic silence that shrouded Dulce.
+
+Behind them, Taylor watched with a disciplined patience, their authority the foundation upon which the operation was built. Their voice cut into the stillness, a reminder of their presence, "Time is not our ally here."
+
+Drawing back from momentary distraction, Jordan acknowledged the wisdom in Taylor's words, yet could feel the shift in their stance—from skeptical, reserved analyst, to a proactive agent within the narrative. "You're right; these documents may hold critical insights. Let's collect what we can and analyze them properly."
+
+From the darkened hollows of the room, shadows seemed to cast subtle judgment as Alex and Jordan worked together with heightened urgency. Taylor, for once, didn't intervene but instead surveyed the entrance, their mind anticipating the unknown variables that lay ahead.
+
+Unexpectedly, a soft hiss emanated from a neglected terminal on the desk. Jordan's head snapped up, their heart rate accelerating at the potential ramifications. Without a word, they moved to the machine, hands driven by the newfound conviction that knowledge was more than power—it was survival.
+
+As Jordan began to extract what data they could from the terminal, the first comprehensible communication from the depths of Dulce in far too long crackled through: an automated distress marker, looping endlessly without further context. It was a revelation, one that reverberated through the group, confirming their fears and igniting an even greater need to press on.
+
+Watching Jordan's dogged determination, Alex witnessed the minor transformation in his colleague unfold—a shift from doubt to action, a sliver of belief in the possibilities beyond their rational understanding. This forge of resolve amidst the alien echoes of Dulce not only bonded them closer as a team but compelled them forward with a sharpened edge of responsibility to the truth, wherever it would lead.
+
+As they collected their findings and regrouped, the base around them imperceptibly changed, the air charged with the vibration of secrets poised on the brink of revelation. And in that charged silence, the group moved on, each now carrying pieces of a puzzle that would soon converge into a picture of galactic significance.
+
+\*
+
+In the chill of the cramped server room, the hum of machinery was the backbone to a symphony of data streams coursing through the air. Dr. Jordan Hayes, nerves alight with the mission's mounting unknowns, patched into the last known coordinates of the unsent distress broadcast they had uncovered. They were so close to the core now – to the truth behind the blackout – it was almost tangible.
+
+Beside them stood Agent Alex Mercer, ever the soldier, yet with eyes that betrayed an intellect craving to understand the murk beneath the surface. "Any progress, Dr. Hayes?" Alex queried, his voice betraying a subtle urgency.
+
+"Getting there," Jordan replied, fingers dancing across the keyboard. "Whoever sent this was cut off mid-transmission. It's as if Dulce itself swallowed the message whole."
+
+Taylor Cruz closed in, their frame casting a long shadow over the duo, evoking an almost palpable wall between them and the forward momentum of their mission. "Time is against us," Taylor intoned, more statement than threat. "What we uncover here determines our next course of action."
+
+Alex acknowledged Taylor with a brisk nod, his stance firm. Yet inwardly, the tightening grip he felt from Taylor's words couldn't throttle the swell of his own investigative instinct. His soldier's obedience had begun to war with the advocate's zeal for unveiling the dark heart of Dulce's secrets. 
+
+And then, the unexpected occurred. The screens flashed in unison, spilling a discordant stream of symbols and images that defied immediate analysis. Jordan's breath caught – this was the response they had been fishing for, an alien communication protocol resonating just at the edge of human comprehension.
+
+Each member of the team felt it: a shift in the room’s very atmosphere, like a veil being drawn from their perception. Alex and Jordan stood still, absorbed in the bewilderment of contact, while Taylor, despite their authority, hesitated – a minor betrayal that unease was creeping into even their disciplined heart.
+
+"Thoughts, Rivera?" Taylor rallied, seeking the counsel of Sam Rivera, whose eyes were wide with exhilaration.
+
+Sam stepped forward, breaking the spell of stillness. "It's like nothing I've ever seen before, but I think I can bridge our systems to communicate," they declared, a wisp of optimism braiding their voice. They set about adapting their gear to transmute the foreign signals into something the team could dissect, their actions a testament to the mentorship and belief instilled in them by Mercer and the team.
+
+Taylor observed them, a cold calculation behind their facade, as they weighed the worth of this anomaly. It was a crossroad that potentially led to either monumental breakthrough or unprecedented catastrophe. "Once you've established a line, document everything. We can't afford to miss any detail," Taylor ordered, the words sharper than intended.
+
+The connection was made, and with trembling anticipation, the team listened as the first garbled outputs began to emerge, their very essence promising insights that could alter the course of history. It was an enigmatic dance with the unknown, the pulse of Dulce no longer just a place, but a herald to an alien register the team had yet to decipher.
+
+Together, they stood at the precipice of understanding, where the faint glow of their monitors cast more than just light – it cast the shadow of burgeoning transformation. It was in this moment, in the grasp of an extraterrestrial tongue, that the team, bound by a hunger for knowledge and the raw edge of survival, found their mission reframed from a search for answers to the articulation of a question humankind had yet to fully ask.
+
+Silent in their commune with the inexplicable frequency, they realized they were not merely investigators; they had become liaisons on behalf of Earth, interpreters of a cosmic message that could redefine their very existence. The implications loomed large, but now, they would not face them alone – they would face them as a united front, wrought together by the very mysteries that once drove them apart.
+
+## Chapter 5
+
+Dr. Jordan Hayes clutched the edge of the briefing room table, their fingers white-knuckled against the laminate surface, as an array of constellations rotated on the projector—charts and graphs bleeding across the stars. In the dim room, nebulas and dark matter seemed within arm's reach, tangible yet unfathomable. 
+
+Sam Rivera leaned back against the wall, arms crossed, gaze darting between the swirling cosmos and the faces of their companions. A taut line of concentration etched their young features, a mingling of fervent curiosity with the nascent understanding of the high stakes for which they played.
+
+Jordan's voice broke the profound silence. "The patterns in the signal disruptions sync with none other than zenithal star alignments. It's as if... as if these 'meet and greets' were scheduled, predestined by celestial mechanics."
+
+The statement hung heavy, daring the occupants of the room to unravel its implications. Alex Mercer, his prior military resolve momentarily suspended, absorbed the hypothesis with a visible hunger. "It's like we're adhering to an appointment we never knew we had," he murmured, his heart a drumbeat in his chest.
+
+Taylor Cruz snorted—a sound that clattered against the high concepts like a tumbledown shack in a futurist cityscape. Folding their arms, they glanced between the agents, their apprehension clad in the contempt of practicality. "What we need are facts, not mystic conjecture."
+
+Alex pivoted on his heel, facing Taylor squarely, and his voice found its edge of steel. "This isn't mysticism, Cruz. It's a hypothesis based on observed phenomena as unpredictable as the place we're standing in."
+
+Taylor's gaze never wavered, yet the slight twitch at the corner of their mouth belied their taut composure. "If there's a semblance of truth to it, then it's critical intel. But remember, we're not astrologers—we're soldiers and scientists."
+
+Jordan met Taylor’s gaze with a curt nod, accepting the caution even as the crucible of their intellect smoldered with the fervor of cosmic discovery. Their eyes flicked to Sam, whose steady presence and ready tech affirmed a burgeoning dynamic—the makings of a sentinel, standing guard over the threshold of human understanding and cosmic reality.
+
+With the projector casting pallid light over their features, each agent became a silhouette of purpose, shadows pillared against the backdrop of an endless universe. The story they were embroiled in would soon demand they plunge into darkness to retrieve the light of knowledge—a light that could very well redraw the shape of their world.
+
+They left the briefing room with a shared silence, each pondering the vast weave of celestial intent and terrestrial response, sensing that the galactic appointment to which they'd unwittingly RSVP’d was more insistent—and more threatening—than any operation they’d faced before.
+
+\*
+
+As the Paranormal Military Squad team convened in the heart of the Dulce military complex, an air of bristling expectation clung to the walls of the underground sanctum. Alex Mercer’s brow furrowed while watching his companions—Jordan Hayes, diligently setting up their makeshift lab station, and Sam Rivera meticulously checking the communication relays they had restored. Taylor Cruz observed with hawk-like focus, yet to betray the strain that their command posed on them.
+
+The gravity of the mission had shifted, deepened; each member of the team felt its pull, tethered to the understanding that they were now part of a larger narrative—a cosmic play with Earth as a stage and the human race unwitting actors. 
+
+Jordan paused, a tension creeping across their shoulders as they aligned the satellite data with the alien message that had been decoded. "The instructions in this message," Jordan started, the timbre of their voice betraying their usual composure. "They're coordinates and... a warning."
+
+Sam leaned in, their eyes widening behind the glow of their laptop screen. "A warning? Like, ‘stay away from’, or ‘beware of’...?" Their words trailed off, uncertainty a new companion in their lexicon.
+
+Alex exhaled slowly, his mind racing to connect the dots. "It doesn't matter which," he said, decisive yet contemplative. "What matters is we understand intent. Are we being warned out of concern, or are we stumbling upon a threat?"
+
+Cruz’s iron-clad facade momentarily cracked, a fleeting glimpse of vulnerability flashing through their eyes. "We need to know if this entails additional risk to the operation," they said, directing their gaze specifically at Alex. "Mercer, I rely on you to keep the team grounded. No one goes off-course."
+
+Their reminder seemed both a command and a plea—rooted in an understanding that each member of the team now faced the duality of their roles, protectors of earthly secrets and heralds of potentially devastating revelations.
+
+Sam's fingers stilled mid-type, their task forgotten as they absorbed the weight of the unfolding reality. "We're the first line of defense... or detection," they mused half to themselves, a growing sense of agency within the larger play they were cast into.
+
+Jordan returned to the data, more resolute in their actions. The warning, whether cautionary or dire, was a beacon they no longer could ignore; its light casting aside shadows of doubt and igniting a collective purpose within the team.
+
+Alex watched Jordan and Sam, feeling a brotherhood in their shared quest. As Cruz paced, poised on the cusp of decisions that would mark their career and perhaps the fate of many, Alex knew the narrative had changed. They were no longer mere operatives; they had become guardians of a threshold, keepers of a message from a realm beyond stars and stripes. This elevation in their mission could not be shackled by regulations and established protocols—it demanded a new perspective, a new resolve.
+
+Tension threaded through the dialogue of beeps and static as communications with Washington buzzed in the background. The team stood, a portentous air enveloping them. It was clear that the decisions they made in the ensuing hours could redefine humanity's place in the cosmos or condemn them to ignorance and potential peril.
+
+Their connection to the stars solidified, the group moved to address the crystallizing warning, shifting from passive recipients to active participants. Mercer’s latter instincts gained precedence— the team’s mandate had evolved, no longer solely to observe and report but to interact and prepare. A metamorphosis had begun, and Operation: Dulce hummed with the newfound frequency of their daring, a tone set not by the earthly hierarchies but by the pulsing symphony of the universe itself.
+
+\*
+
+The desert night loomed eerily still as echoes of hidden activity reverberated deep beneath the bleak sands of New Mexico. Diverting his gaze from the array of sensors before him, Jordan Hayes allowed a rare breath, deep and anxious. Turning to Alex Mercer's focused silhouette, the nocturnal landscape illuminated softly by makeshift floodlights, Jordan felt the syncopated tempo of apprehension and exhilaration jockey for primacy within.
+
+"The closer we get to unlocking these messages, the more I feel like we're peeling back layers of reality itself," Jordan confided, eyes not leaving the monitors that presented a constellation of data points.
+
+"Yes," Alex replied, his voice steady as he considered the implications of their discovery. "And we have to be ready for whatever we find beneath those layers. Whether it's a breakthrough or a Pandora's Box."
+
+Silence settled between them, broken only by the occasional buzz of communications equipment attempting to bridge terrestrial and extraterrestrial intelligences. Tense moments drifted by, laden with the expectant weight of near breakthrough, when a soft chime signaled an incoming transmission -- a rare sound that set every agent on high alert.
+
+Absent was the voice of Washington or Paranormal Military Squad command. Instead, a rhythmic series of pulses and tones filled the air, deliberately patterned, unmistakably non-human.
+
+Sam Rivera adjusted the sensitivity of the decoding equipment, their hands shaking with anticipation as much as focus. "I have it!" they announced, the signal transforming under their expertise into a sequence of visual symbols on the screen before them.
+
+Their shared excitement was palpable, a kinetic force resonating between the team members as they crowded around the display. 
+
+"What does it say?" Taylor Cruz demanded, the urgency in his tone scraping against the newfound wonderment.
+
+Interpreting the alien syntax required not only decoding but intuition and empathy. The words that emerged upon the screen were at once coherent and enigmatic: "*Voyage. Convergence. Peril.*"
+
+The stark simplicity of the message struck them collectively, a chill breeze wafting through their resolve.
+
+Alex stepped forward, piecing together the cryptic communication with a growing sense of obligation. "It’s a call to action," he deduced, "or possibly a summons."
+
+Jordan's gaze met Alex’s, both understanding that this was no longer an investigation or mere extraction of hidden truths. This was humanity's unwitting enlistment into a galactic dialogue that defied boundaries of nation, creed, or protocol.
+
+Sam's eyes were aglow, not with fear, but with the profound acceptance of inevitability that comes with groundbreaking revelation. Moreover, within Taylor's stern exterior churned the seed of reluctant admiration for the unclassified, the uncharted realms they were approaching.
+
+Together, they accepted the pivot in their mission, readjusting their objectives from exploration to engagement, and from isolation to a communal outreach beyond the stars. As dawn's first light threatened the horizon, it became clear that they were no longer merely operatives of a clandestine governmental faction—they were delegates on behalf of Earth, embarking on a voyage orchestrated by destinies unrelated to the mere geopolitics of their world.
+
+Turning to each other, their silhouettes sketched against the coming dawn, the agents recognized the transformation within and amongst them. They were bound by more than duty—they were intricately woven into the fabric of an unfolding cosmic opera, one in which they had been granted an undeniable role. And as they set course for the coordinates that beckoned them like a distant siren's call, it was with a solemn dedication to not only uncover the mysteries ahead but to navigate the convergence, and the peril, as unified emissaries of a world on the cusp of a broader understanding.
+
+\*
+
+Beneath the hum of the fluorescent lights and the vigilance of silent monitors, Alex Mercer stood with his team in the threshold of the base's command center, their faces etched with the fatigue of hours spent unraveling galactic mysteries. Jordan Hayes broke the stillness with a delicate fusion of disbelief and resolve. "The signal..." they began, their tone deliberate, "it’s evolving. It’s not just sending a message—it’s responding to us."
+
+Taylor Cruz leaned over the console, their eyes narrowing with intrigue and a flicker of unease, studying the alternating patterns on the screen. "Responding? Like it’s alive?" Taylor asked, a question that bordered on the edge of wonder and alarm.
+
+Sam Rivera’s gaze was locked onto their interface, a digital orchestra at their fingertips. "It could be some form of advanced AI. Or something else entirely," they contributed, a note of exhilaration betraying the gravity of the situation.
+
+Alex paced before the terminal, absorbing the enormity of their predicament. Their mission—once rooted in the solid ground of military discipline and covert operations—had transcended into an encounter of unprecedented import. "We need to be cautious," he advised, his voice a low rumble of cautious strategy. "If this signal is intelligent, how we interact with it could dictate the outcome of this entire operation."
+
+Jordan met Alex's gaze with a nod, the weight of the responsibility shared and accepted. "We have protocols for first contact, but nothing for... this," Jordan admitted. The room was gripped with tension, each breath seemingly louder than the last.
+
+Then, with a sudden burst that filled the command center, the signal coalesced into a clear and distinct pattern which replicated and expanded, its complexity revealing the hand—or mind—of an intelligent architect.
+
+Taylor's instinct for command surged forth. "Prepare to record and analyze. Whatever it is, we need to understand it—" But their words were cut short as the signal surged, enveloping the room in a brief, blinding cascade of light.
+
+In that pulse of brilliance, a shared revelation coursed through the team. The signal had become a bridge, an extension of unknown consciousness reaching towards them, testing, communicating, searching. 
+
+Alex stepped back from the light, feeling a profound change unravelling within him. The path forward would not be one of confrontation or conquest, but of connection and comprehension.
+
+Jordan turned to Alex and Taylor, seeing in their faces a reflection of the same metamorphosis taking place within themselves—a movement from observers to participants, from agents to ambassadors.
+
+With a collective breath, the team faced the kaleidoscope of lights. The alien signal, once a harbinger of enigma, was now a catalyst for transformation—a symphony of light and sound that echoed the beginnings of a new relationship between humanity and the alien unknown.
+
+And so, with deliberate steps, Alex Mercer led his team into the luminous fray. Science, protocol, and survival instinct harmonized within them, each member poised on the cusp of a new chapter in human history.
+
+They were no longer merely the instruments of Paranormal Military Squad's will—they were the vanguard of humankind’s first definitive leap into the cosmic community. 
+
+With the last echoes of the signal resonating in the control room, they each embraced the sequencing of the transmission, the dance of extraterrestrial light that now wrote itself into their story. The chapter of Operation: Dulce drew to a close, but the narrative of their destiny had only just begun.
+
+## Chapter 6
+
+\*
+
+The cool darkness of the command center at Dulce base was a stark contrast to the brewing storm outside, where the unforgiving New Mexico desert winds whispered of the hidden truths that lay buried deep beneath its surface. Dr. Jordan Hayes sat, their eyes fixed on the readout, the frenetic dance of symbols and numbers reflecting off their determined face. They were on the cusp of an epiphany, teetering between the widely accepted laws of physics and the promise of a new cosmic paradigm.
+
+Alex Mercer watched from across the room, noting the subtle shifts in Jordan’s posture that belied a developing readiness to embrace the unbelievable. “Find something?” Alex’s question, asked with a blend of curiosity and solidarity, bridged the gap between a command and a genuine query among equals.
+
+Jordan's response was slow, measured against the magnitude of their analysis. “This isn’t random static. It’s a pattern - a repeated sequence phasing in and out but distinctly artificial.” Jordan turned away from the screen, locking eyes with Alex. “This could change everything.”
+
+Sam Rivera leaned in, their eyes alight with the fires of revelation and a quenchless thirst for understanding. “A pattern means intention. Could it be a message?”
+
+A figure emerged from the doorway, casting a long shadow into the room - Taylor Cruz. “Intentions can be friendly, or hostile. We shouldn’t forget that,” said Taylor, bringing a dose of their usual pragmatism into the heart of discovery.
+
+Alex acknowledged Taylor’s caution with a nod, understanding the need to keep their feet grounded even as their spirits soared toward the unknown. “Then let’s be the first to find out which it is."
+
+The team gathered around the monitors, the soft tapping of Jordan's keystrokes now punctuated by the occasional crackle of Sam's radio equipment. The sound was almost ritualistic, a prelude to humanity’s potential first, knowing foray into a larger universe.
+
+Jordan’s fingers paused, suspended in mid-air. The signal had evolved, becoming a beacon that somehow felt less alien and more familiar. It was as if the complexities of their message were unfolding into something more accessible, more terrestrial.
+
+A hushed excitement swept through the room. The transformation suggested an awareness on the part of the unknown senders; a finesse that spoke volumes about their capabilities and perhaps their intentions.
+
+With the growing realization that they were engaging with an intelligence far exceeding their previous understanding, the team prepared to reach back across the cosmic divide. Prepared or not, they were no longer bystanders in this galactic narrative. They were active correspondents in an exchange that transcended galaxies and welcomed them into an expansive, possibly fraught, interstellar conversation.
+
+\*
+
+Inside the cavernous central hub of Dulce military base, Dr. Jordan Hayes stood in near-darkness, surrounded by a nest of cables and monitors that buzzed with silent, cryptic life. Jordan's eyes narrowed to focus on the sequences that danced across the screen—patterns that could unravel the cosmic enigma surrounding them.
+
+Alex Mercer approached with his characteristic stride, a signal of reliability in the chaos. "Status report, Dr. Hayes?" he inquired, his voice low, almost blending into the soundscape of beeping consoles and swirling fans.
+
+"We're on the brink of unravelling the signal's origin," Jordan replied, the weight of implications heavy in their tone. "There's intelligence behind it, a thought process alien to our own."
+
+As if summoned by their analysis, Taylor Cruz approached with authority radiating from every pore. "Understand this, we need to know if it's friend or foe. Don't get wrapped up in the existential—our lives may depend on the answers you provide."
+
+Sam Rivera, their hands adroitly adjusting a device to fine-tune the signal, chimed in with optimism undercut by anxious anticipation. "We're deciphering the comm encryption. Soon, we'll have a channel open—not just listening in, but speaking back."
+
+Alex nodded his understanding, his strategic mind processing the tactical implications while grappling with the more profound humanistic impact. "When we do, we'll tread carefully, communicate with purpose," he reassured the team.
+
+The operation had evolved rapidly, from a stealthy incursion into a clandestine labyrinth to an exchange with an extraterrestrial intellect. Their earlier trepidation transformed into determined focus, as they prepared to extend humanity’s hand into the vast unknown.
+
+An alert on one of the monitor stations snapped the team into alarm. The signal had not simply been waiting—it had been calculating. Now, it reached its crescendo, demanding their attention with a provocative urgency.
+
+Jordan's fingers raced over the keyboard, their eyes simultaneously interpreting data and sharing directives. "It’s a linguistic lock, a test of comprehension. We crack this, we establish dialogue."
+
+Taylor's presence was a beacon of steely resolve. "Then let’s solve it. This is what we trained for—the unknown."
+
+Alex and Sam exchanged a look that telegraphed their shared determination—this was not only the mission they had trained for; it was the mission they had been destined for.
+
+Together, the Paranormal Military Squad team leaned into the challenge, their minds honing in on the complex patterns with a singular goal: to unlock the conversation with an intelligence that had already begun to shift the foundations of what they knew, or thought they knew, about the universe.
+
+In a symphony of clicks and murmurs, they worked, knowing they were about to make a giant leap not just for themselves or Paranormal Military Squad, but for all of humanity. As the final pieces fell into place, Dulce's militaristic silence was shattered by the sound of intergalactic contact—by the sound of history being made.
+
+## Chapter 7
+
+In the enclosed space of Dulce’s command center, the air was thick with anticipation, each team member poised to tread the razor's edge between scientific breakthrough and galactic peril. Dr. Jordan Hayes focused intently on the screen, their fingers tapping a staccato rhythm against the keyboard as lines of alien code cascaded down the monitor.
+
+Alex Mercer's steely gaze surveyed the room, stopping on each member of his team. "Thoughts?" he asked, echoing the unspoken tension. His question, while directed at the group, lingered on Jordan—acknowledging their expertise and inviting collaboration rather than dictating orders.
+
+Jordan’s brow furrowed, an indicator of the mental gymnastics being performed. "It's unprecedented," they finally said, their voice a testament to the gravity of the moment. "Behavioral algorithms... if we're right, this code could reveal extraterrestrial thought patterns."
+
+Before anyone could react, Taylor Cruz interjected with the assertiveness of someone accustomed to commandeering the discourse. "Then let’s ensure we’re deciphering it correctly," Taylor stated, their tone suggesting they were still battling to maintain control over an increasingly alien situation.
+
+Sam Rivera hovered near the mainframe, youthful energy barely contained under the surface. "What if it’s more than just a message? What if they’re trying to extend consciousness across the stars?"
+
+The room fell into a contemplative silence, broken only by the hum of electronic equipment and the distant thud of secured doors locking in rhythm. The weight of responsibility rested on each agent's shoulders—a heaviness palpable in the air they shared.
+
+Alex stepped forward, reaching a subtle decision, one dictated by foresight and the humanity nestled at the core of their mission. "We approach with the aim to understand, not to confront," he said, softening his military bearing into a more diplomatic stance.
+
+Jordan nodded, appreciating the leadership that Alex displayed in the face of the unknown, and turned back to the cryptic data. Here, before them all, was a tangible piece of evidence—proof of an extraterrestrial sentience that had outreached the bounds of their expectations.
+
+Taylor took a breath, simultaneously exuding a sense of preparedness and venturing into the unknown alongside their peers. "Then let’s do what Paranormal Military Squad does best—investigate and adapt," Taylor added, finding comfort in the familiar even as they stood on the cusp of an unprecedented alchemy of science and mystery.
+
+The team leaned into their respective roles, driven by the urgency of the assignment and the pull of an insatiable curiosity. Sam offered a grin that belied the tension, a youthfulness that reminded them all of the profound excitement nested within the terror of the unknown.
+
+Quietly but resolutely, they turned back to their instruments, each of them a sentinel on the threshold of a new reality. The once implicit lines of command were now woven into a shared tapestry of hierarchy and camaraderie. As they danced with the unknown, they were beacons of sentient endeavor, casting the light of human consciousness into the vast darkness that called to them.
+
+\*
+
+\*
+
+Dulce Base's cavernous darkness was pierced by the sharp luminescence of monitors, casting an electric glow onto the faces of those who dared to unearth its secrets. Dr. Jordan Hayes stood motionless, eyes glazed in concentration, their mind a nexus where terrestrial science battled with celestial unknowns.
+
+Alex Mercer watched from a slight distance, the weight of command tangible upon his shoulders, though lightened by the shared burden now held amongst them. "We could be on the frontier of a new kind of diplomacy," he mused aloud, giving voice to the moment's gravity.
+
+At those words, Jordan's trance broke. "If that's the case, then these communications," Jordan motioned to the stream of data, "are our olive branch across the cosmos."
+
+Taylor Cruz, who paced with restless energy, halted and faced the team—his stoicism marred by the erratic dance of lights reflected in his eyes. "An olive branch, or an invitation to a battlefield?" he posed, ever the strategist, his words laced with a hint of cynicism.
+
+Sam Rivera, nestled amongst an array of equipment, licked their lips—a mixture of nerves and anticipation palpable. "We're mapping out something incredible here. Whether it's peace or war, we're the cartographers."
+
+Silence enveloped them like the expanse of space itself, each member contemplating the chasms they might bridge—or the abysses into which they might unwittingly descend.
+
+Alex's demeanor assumed a quiet resolve—the profound knowledge that this mission was as much about navigating uncharted philosophical territories as it was about ensuring survival. "Whichever it proves to be, we'll face it. Prepared, unified."
+
+A nod passed between Jordan and Alex, a silent exchange of mutual respect and shared mission. Sam, buoyed by the weighty encounters of the mind and machinery, entered keystrokes with a fervor that seemed to bring them ever closer to the alien mind.
+
+They stood there, the Paranormal Military Squad team, not just as guardians of homeworld secrets or as soldiers of clandestine wars, but as humankind's chosen few at the fulcrum of history—a history that was now unfolding to the rhythm of otherworldly codes.
+
+Each revelation, each parsed symbol, inched them toward the line between the earthly and otherworldly. And as they stood on this precipice of cosmic negotiations, it was clear the ensuing dialogue would not just shape the future of Paranormal Military Squad—it could very well redefine the parameters of human existence.
+
+\*
+
+The hum of advanced computational systems tingling with cryptic transmissions framed the ambiance of Dulce's mainframe chamber. Jordan Hayes, fingers hovering over a console dense with blinking lights, furrowed their brow as sequences of alien data streamed across the screen. 
+
+Alex materialized behind them, his presence a stable beacon amidst the technological whirlwind. "Look for patterns, anomalies. Anything that might resemble a handshake protocol in their communications," he directed, his voice a low thrum, reverberating with cautious optimism.
+
+Jordan cast a glance over their shoulder, acknowledging Alex's contribution with the shared understanding of colleagues who had transcended mere professional acquaintance. "I’m isolating sequences that seem to recur with more intention than static. If these are their ‘handshakes,’ then we might just be making first contact," they remarked, their focus returning to the screen with renewed vigor.
+
+From the other end of the room, where shadows married the artificial light, Sam's voice crackled through the static of nearby speakers, "Don't forget the anomalies we detected earlier. Each one could be a word, a sentence, or even a concept untranslatable to our current understandings."
+
+Resolute, Taylor Cruz stood at Jordan's other side, a stoic figure wrestling with the implications of their mission. "Keep pursuing this line," Taylor instructed, an undercurrent of intensity carried forth in their otherwise composed demeanor. "And remember, this isn't just about making contact; it's about securing knowledge for humanity."
+
+Alex offered a nod that spoke volumes, conveying his understanding of the stakes at play. Here, in this chamber of possibility, the team's actions would determine if humanity stood at the brink of a new age of understanding or the onset of an unprecedented threat.
+
+Every second thrummed with significance as Jordan and Sam worked in tandem, each keystroke a foray into the unknown. Taylor observed with a commander's scrutiny, the gravity of their role sustaining them against the waves of ambiguity breaking against their resolve.
+
+Pivotal moments come rarely in the course of human events but here, amidst the electronic symphony of a stalwart command center, lay the incepting notes of a cosmic overture. The harmony between human and alien, between Paranormal Military Squad and the vast reaches of space, began its first tentative measures, with each member of the team a vital instrument in a celestial ensemble yet to be fully heard.
+
+\*
+
+The crisp air within the mainframe room of Dulce base seemed to hum with unspoken possibilities. Jordan Hayes was the centerpiece of focus, their hands dancing methodically over the console as streams of otherworldly code cascaded down monitors, each flicker a potential key to the cosmic doors they were inching open.
+
+Alex Mercer watched, posture relaxed but eyes sharp. "Remember, this could be our first introduction, maybe even our first impression," he said, mindful of the gravity carried by each action they made henceforth.
+
+A hint of a smile touched Jordan's face, a small acknowledgment of the monumental task at hand. "Understood. I'm balancing the signal's syntax with our algorithms. If we're interpreting this correctly, it could be... well, an invitation."
+
+Into the electric tension of the chamber walked Taylor Cruz, their silhouette a sharp contrast against the cool lighting, radiating a presence that spoke of command and chilly tenacity. "An invitation, or a challenge?” Taylor questioned, the weight of their suspicion casting a different tint on the cascading data.
+
+Sam Rivera, in a corner arrayed with sophisticated equipment, piped up, their voice a buoyant note amidst the tentative atmosphere. "Either way, it's a connection. One that we're uniquely positioned to navigate," they remarked with an air of optimism threading through the uncertainty.
+
+Alex channeled the strengths of his team into the core of their approach, his leadership adapting to the contours of an unprecedented scenario. "Cautious and curious," he reflected aloud, shaping a strategy that balanced their thirst for comprehension with the prudence required in addressing the unknown.
+
+Jordan, hands momentarily at rest, looked up. The signal was more than a sequence of bits and commands—it was a riddle wrapped in the depths of space-time, and they were on the cusp of parsing its meaning.
+
+Taylor, hardly a step away, nodded in silent agreement. The implications of their findings might very well direct the course of human destiny from this point onward.
+
+Finding a tempo among themselves, the Dulce team was a confluence of ambition and acumen, each member intuitive to the beats of discovery. The chamber around them held untold stories, secrets coaxed from the stars, that now, led by Paranormal Military Squad's finest, began to unravel.
+
+The future in those moments was unwritten, a narrative scribed not in the dust of desert confines, but in the potential for interstellar diplomacy and understanding. As they prepared to script humanity's next chapter, the room seemed to pulse with the heartbeat of a story far greater than the sum of its parts.
+
+## Chapter 8
+
+The grit of an earthbound dust storm contrasted sharply with the pristine sterility of the underground command center. Alex Mercer, eyes set with fervent determination, stood over Jordan Hayes, whose fingers danced across the keyboard with rapid purpose. Monitoring the progression of alien code unraveling before them, Mercer spoke with a tempered urgency, "Keep it steady, Jordan. We might be initiating the first true interspecies communication bridge here. It's all about finesse now."
+
+Taylor Cruz, the embodiment of military precision, surveyed the room with a calculated gaze from their vigil beside an array of glimmering screens. "Remember, these could be delicate negotiations -- or coded threats. Stay sharp," Cruz added, their voice cool as polished steel.
+
+Jordan, with a silent nod, recognized the gravity of both stances. Gravitating between scientific acuity and diplomatic caution, they replied, "The sequence is aligning—syncing with our comms. It's looking more and more like direct engagement."
+
+Amid the banks of electronic machinery, the thrumming pulse of an impending interspecies signal exchange, Sam Rivera interjected with a youthful zeal that cut through the weighty atmosphere, "It's not just an exchange. It's a... symphony. It's as if they're teaching us their language through modulation."
+
+A moment of profound silence swept over the team. The isolation of their location, deep within the top-secret labyrinth of Dulce, became suffused with an almost palpable sense of historical significance.
+
+"Then our response needs to be equally symphonic," Alex uttered, contemplating the awe-inspiring transmutation of their task from a simple recovery mission to a full-blown cosmic concerto.
+
+With a renewed sense of wonder tempered by caution, the Paranormal Military Squad team found themselves harmonizing a delicate balance between envoys and interpreters. The long shadow cast by their duty was now illuminated by the brilliant glow of otherworldly dialogue.
+
+In this carefully orchestrated march towards the unknown, each individual's expertise became critical notes in a larger melody. The narrative of human achievement, so often defined by solitary pursuits, now emerged as a collaborative opus, each member of the team a maestro in their right.
+
+The protocols of encounters, the mathematics of languages, and the poetics of connection all fused into a singular moment of convergence. The echo of their efforts reverberated back to them, not through the cavernous base's concrete walls, but from light-years away, in the form of a reply, intangible yet infinitely profound.
+
+\*
+
+Amidst the hum of the supercomputers and the faint static from the scrambled transmissions, Alex Mercer cast a thoughtful glance across the dimly lit room toward where Dr. Jordan Hayes was methodically adjusting the archaic dials of the decryption machine. "Any progress?" he asked, his tone conveying both impatience and the deep-seated respect born from countless shared challenges.
+
+Jordan did not look up, their gaze remained locked on the flickering lights that represented a dialogue suspended between worlds. Their fingers ceased their dance, hovering meditatively over the controls. "We might be on the cusp of a breakthrough," Jordan suggested. "The signal... it's evolved. It's reflexive now, responsive in a way that suggests sentience."
+
+Taylor Cruz's familiar sharp strides approached the two, breaking the rhythm of soft beeps. "Responsive is good, if it means understanding," Taylor said, head tilted as they peered at the encryption data scrolling by. "But remember, comprehension can bring revelation or conflict."
+
+Sam Rivera’s youthful voice permeated the tension, brimming with an excitement edged by the enormity of what they faced. "If it's truly sentient, we're not just cracking a code; we're learning how to converse with an entirely new form of consciousness," they chimed in, the weight of history not lost on the zealous astrotechnician.
+
+Alex nodded, his thoughts alighting on potential strategies for navigating the conversation they were cultivating with the unfathomable. "We need to keep that conversation going, echo its patterns, and speak its language," he resolved, knowing the delicate nature of their work merited every ounce of their collective acumen.
+
+The chamber now was a crucible, forging within it the future narrative of human contact with the unknown. Every signal pulse they sent out was an invitation for understanding, and every echo back a step closer to bridging the cosmic divide. And so, together, they stood - agents in Paranormal Military Squad's clandestine ranks, united by purpose, sculpting humanity’s first sonnets into the void.
+
+\*
+
+#### Knowledge graph updates
+
+- (Jordan Hayes, Interprets, Communications as cosmic diplomacy, Moderate)
+
+- (Taylor Cruz, Questions, Potential aggressiveness of alien intent, Minor)
+
+- (Sam Rivera, Expresses, Optimism about forming a connection, Minor)
+
+- (Alex Mercer, Adopts, Balanced strategy for contact, Moderate)
+
+- (Paranormal Military Squad team, Navigates, Beats of cosmic discovery, Moderate)
+
+- (Paranormal Military Squad team, Prepares, To script humanity's interstellar narrative, Major)
+
+## Chapter 9
+
+The sterile silence of Dulce Base's command center was thick with concentration as Alex Mercer surveyed his team, hunched over their respective technological battle stations. Each agent was a weapon against ignorance, their adversary a code from beyond the stars that held secrets to alien thought.
+
+Dr. Jordan Hayes, whose hands had been steadfastly working the decryption algorithms, paused and looked up at Alex. "We're through the next layer of encryption," Jordan announced, a mixture of pride and gravitas in their tone. "It's communicating. It's... aware."
+
+A shadow momentarily clouded Alex's determined features—awareness implied so much more than mere intelligence. "Aware and reactive or aware and proactive?" he queried, his experience anticipating the pivotal importance of intention.
+
+"Unknown at this stage," Taylor Cruz interjected, looking up from a datasheet. "But I urge caution. We tread the line between breakthrough and disaster with each keystroke."
+
+Sam Rivera, ever the source of technological acumen, added their voice to the conversation. "The signal's adapting every time we interact with it. Like a conversation where both parties are learning each other's language in real time."
+
+Alex leaned in, rested a hand on Jordan's shoulder—a sign of companionship and an affirmation of trust. "Keep the communication channels open. But let no message, no pulse go unchecked. This could be our Rosetta Stone or our Tower of Babel."
+
+Silence fell over them, a momentary lull as each member of the team contemplated the historic weight of their task. Yet, it was impregnated with a tangible sense of excitement—a collective energy that thrummed through the air just as palpably as the electric current through the banks of machines surrounding them.
+
+They continued their work, squaring shoulders against the magnitude of their undertaking. The agents were standing not just at the precipice of a new chapter for Paranormal Military Squad but for all of humanity. For now, they communicated with powerful unknowns, but with each exchange, they were etching the first words of a dialogue that might forever alter humanity's place in the cosmos.
+
+\*
+
+\*
+
+The sterile silence of Dulce Base's command center was thick with concentration as Alex Mercer surveyed his team, hunched over their respective technological battle stations. Each agent was a weapon against ignorance, their adversary a code from beyond the stars that held secrets to alien thought.
+
+Dr. Jordan Hayes, whose hands had been steadfastly working the decryption algorithms, paused and looked up at Alex. "We're through the next layer of encryption," Jordan announced, a mixture of pride and gravitas in their tone. "It's communicating. It's... aware."
+
+A shadow momentarily clouded Alex's determined features—awareness implied so much more than mere intelligence. "Aware and reactive or aware and proactive?" he queried, his experience anticipating the pivotal importance of intention.
+
+"Unknown at this stage," Taylor Cruz interjected, looking up from a datasheet. "But I urge caution. We tread the line between breakthrough and disaster with each keystroke."
+
+Sam Rivera, ever the source of technological acumen, added their voice to the conversation. "The signal's adapting every time we interact with it. Like a conversation where both parties are learning each other's language in real time."
+
+Alex leaned in, rested a hand on Jordan's shoulder—a sign of companionship and an affirmation of trust. "Keep the communication channels open. But let no message, no pulse go unchecked. This could be our Rosetta Stone or our Tower of Babel."
+
+Silence fell over them, a momentary lull as each member of the team contemplated the historic weight of their task. Yet, it was impregnated with a tangible sense of excitement—a collective energy that thrummed through the air just as palpably as the electric current through the banks of machines surrounding them.
+
+They continued their work, squaring shoulders against the magnitude of their undertaking. The agents were standing not just at the precipice of a new chapter for Paranormal Military Squad but for all of humanity. For now, they communicated with powerful unknowns, but with each exchange, they were etching the first words of a dialogue that might forever alter humanity's place in the cosmos.
+
+\*
+
+Alex Mercer's eyes were fixed on the monitors, the reflected light casting an ethereal glow across his stoic face. The room buzzed with tension, a cacophony of low hums and electronic beeps that underscored the historic nature of their actions. He moved to where Dr. Jordan Hayes was immersed in their work, scrutinizing the alien code streaming rapidly down the terminal.
+
+"Find anything that might look like an entry point or a... digital handshake?" Alex asked, his voice steady, betraying none of the tension gripping his chest.
+
+Jordan looked up briefly, their expression weary yet intense, "Potentially. It's as if the code is anticipating our input, modifying itself in real-time. I've never seen anything like it."
+
+From across the room, Taylor Cruz's sharp voice cut through the hum. "Then it's learning or, possibly worse, baiting us. Proceed with extreme caution," they commanded, their firm stance reinforcing the gravity of the situation.
+
+Sam Rivera, surrounded by a cascade of screens and interfaces, added, "It's almost organic in its complexity. Any minute now, and I might have a way in."
+
+A slight nod was Alex's immediate response, his mind racing through the potential scenarios. "Everyone, stay alert. This could be the beginning of something profound." His seasoned eyes never left the unfolding drama on the monitors.
+
+The room fell silent, the air heavy with unspoken questions. Were they mere moments away from unlocking an otherworldly dialogue? Or was it a Pandora's box that, once opened, could not be closed?
+
+Alex moved closer to the main console, his fingers hovering over the command keys. With the precision of a maestro orchestrating a symphony, he communicated silently with Jordan – respectful of their expertise, aware that the next move could alter the course of human history.
+
+Jordan met his gaze, nodding sharply, and refocused on the task. The signal seemed to pulse with sentient curiosity, drawing them further into its intricate web. 
+
+A sudden flurry of alerts and the intensifying glow of monitors heralded that they had bridged a technological chasm. The alien intelligence on the other end was no longer a distant enigma – it was an active participant, responding to their digital overtures with an unknown agenda.
+
+The team's meticulous efforts had led them to a momentous threshold. Beyond lay unprecedented contact – a nexus of curiosity and potential peril. Within the confines of the base, against the backdrop of a silent desert night, the Paranormal Military Squad operatives became mediators of Earth's bid for cosmic relevance, their every action now a gesture in the grand dance of intergalactic relations.
+
+## Chapter 10
+
+The corridors of the Dulce military base, now silent, echoed with a history of whispered conspiracies and furtive movements. But in the command center, a delicate tapestry of light and sound was being woven as the echoes of cosmic dialogue resonated through the high-tech enclave. Dr. Jordan Hayes, now leading the efforts, called out from their workstation, "I’ve isolated the signal's harmonics. It's more than a call; it's a song, an interstellar siren’s call."
+
+Alex Mercer, steady and resilient in the face of the incomprehensible, acknowledged with a quiet nod, "A song that we need to learn—quickly." His eyes, heavy with responsibility, scanned the room, watching his team work tirelessly at the intersection of science and speculation.
+
+Sam Rivera, dulled by fatigue yet driven by unshakeable resolve, manipulated a complex array of audio interfaces. "There's a pattern, a repeating motif. It's structured, intentional," they muttered, their revelation a bridge between the known and the unimaginable.
+
+Taylor Cruz, a figure of central authority, paced the length of the room, their usual unflappable demeanor betraying a rare flicker of apprehension. "We should be wary of the sirens’ call," Taylor interjected, invoking myths of old as a cautionary metaphor. "We don't want to crash upon unseen shores."
+
+Undeterred, Jordan cast a determined glance at the team. "We navigate by starlight now, not by the limited light of our previous understanding." Their voice was a beacon, charting a course through unchartered realities.
+
+Every individual was acutely aware that each moment in that room was a conduit to an epochal shift for civilization. The mysterious signals, once distant and alien, had coalesced into complex and harmonious oscillations—beacons of an extraterrestrial intellect inviting Earth to join in a cosmic consortium.
+
+Silently, Alex approached the mainframe, his trained fingers aligning with the console’s mechanisms. The room watched in collective breathlessness as he set the frequency in motion, an introductory phrase to an otherworldly melody—a symphony that could bind worlds or spell devastation for all they knew.
+
+In the control room of Dulce, amongst whispered legends and the quiet hum of machines, humanity's ambassadors now stood, stretching their hands into the void, reaching for the hand that would either pull them into the light of new stars or into the maw of darkness between them.
+
+\*
+
+Underground, the Dulce facility's command center was awash with frenetic energy, a stark juxtaposition against the silent, decrepit corridors that enveloped them. The air hummed with anticipation as Dr. Jordan Hayes and Alex Mercer hunched over a console. The sterile light from the monitors cast an otherworldly glow upon their faces, now reflecting a mosaic of alien characters rapidly translating across the screen.
+
+"The patterns are evolving," Jordan murmured, concentration etched into their every feature. "It’s as if our attempts to decrypt have accelerated its learning. It’s adapting to us."
+
+Alex, who stood steadfast behind Jordan, felt a tinge of uncharted fear quickly quelled by the fire of discovery raging within him. "Keep it up," he urged. "But whatever this is becoming, we need to ensure it remains within our control."
+
+Taylor Cruz interjected, their voice slicing through the buzz of activity. "Control may be an illusion when facing an intelligence that literally writes its own rules," they stated stoically, casting a watchful eye over the flurry of data.
+
+"It's like it's learning to communicate," offered Sam Rivera from a nearby interface, their youthful energy boding a mix of awe and anxiety. "This gives ‘talking to strangers’ a whole new meaning."
+
+Alex surveyed his team—each face a study in concentration, determination, and not a small measure of trepidation. "This might well be our first contact," he acknowledged, "And we need to be ready for whatever answers back."
+
+Together, they stood on the edge of the unknown, forging humanity’s response to a message from the heavens. The ensuing silence was palpable—a collective introspection about their role in this grand cosmic play, one that could rewrite human history.
+
+The encrypted dialogue continued to unfold, its intricate patterns showing an almost uncanny anticipation of their investigative strategies. The air turned heavy with the scent of electricity and ambition as they closed in on a pivotal response.
+
+As the signal’s intelligence—whether artificial or biological—grew more profound, so too did the realization that their mission had morphed from passive observation to active engagement. There was no turning back now. Each agent embraced their part in the delicate dance of an interstellar exchange that could change everything they thought they knew about life, intelligence, and the dark void beyond Earth's atmosphere.
+
+\*
+
+The underground halls of Dulce Base, usually buzzing with covert operations, now thrummed with a different kind of energy, an electric mix of fear and fascination. At the heart of the base, in a room shielded from the world’s eyes, Alex Mercer, Jordan Hayes, Taylor Cruz, and Sam Rivera huddled around a bank of monitors. Each screen flickered erratically with the alien script that had become the center of their lives—and perhaps the pivot on which humanity’s future would turn.
+
+Jordan's eyes never wavered from the displays, their expression was one of rapt concentration, interspersed with flashes of revelation. "We're conversing with the stars," they whispered, almost to themselves. The words hung in the air, a testament to the awe-inspiring strangeness of the situation.
+
+"The language is morphing; changing its structure with every exchange we have," Sam chimed in, enthusiasm tinged with the solemnity of the occasion. "It's like witnessing the birth of a new form of dialogue—one that spans galaxies."
+
+Taylor, despite the situation's precariousness, maintained an appearance of ironclad composure. "Keep the communication stream secured and monitored. We don't know what we're dealing with yet," they reminded the team, a bastion of protocol amidst uncertainty.
+
+Alex watched his team expand the parameters of human achievement; their work here would possibly define an era. "This is untrodden territory," he acknowledged, "and in every word we script, in every response we decode, we're drawing a map that others will follow."
+
+Jordan turned to Alex, a nod acknowledging the shared responsibility of this moment. They had embarked on a new voyage, an odyssey not of the body, but of the intellect and spirit. No longer explorers of the Earthly realm, they had been promoted by circumstance to ambassadors of humanity in a silent and boundless ocean.
+
+A sudden pulse of energy from the monitors signaled a breakthrough; the language had not only adapted but it seemed to resonate, to harmonize with their attempts at making contact. The alien script now sprawled across the screens didn't just ask to be understood—it invited interpretation, collaboration, maybe even companionship across the cold distances of space.
+
+As they stood before the precipice of first contact, Paranormal Military Squad's finest became the architects of a symphony meant to echo through the cosmos. But more than architects, they were the first to play the notes of this cosmic composition, daring to believe that on the other end, someone—or something—might be listening, ready to join the chorus.
+
+\*
+
+The underground command center of Dulce Base, once pulsing with clandestine operations, now resonated with the charge of an impending cosmic threshold. Encircled by banks of whirring machinery, each monitor flickered erratically with alien script that had occupied center stage in the lives of Alex Mercer, Jordan Hayes, Taylor Cruz, and Sam Rivera.
+
+Jordan's gaze didn’t flit for even a moment from the screens, where indiscernible alien messages ebbed and flowed like the tide. The ciphers and symbols cascaded down as they tweaked the algorithmic sliders. "This sequence here," Jordan began, voice both hushed and heavy, "it’s not just transmitting; it resonates—it's designed to be felt."
+
+The room took a collective breath, the remarkable implication hanging in the air like a careful revelation. Sam Rivera was the first to respond, their voice alive with ingenuity: "It's a form of communication stretching well beyond words. We need to respond in kind—the whole array of human expression might be at play here."
+
+Taylor's eyes remained fixed on the figures playing across the data sheets. "If that's the case," Taylor intoned pragmatically, "we must tread carefully. This is no longer just about being heard—it's about being understood."
+
+Alex watched his team, each a fulcrum of insight and expertise, and felt the solemnity of the role they were about to assume. "Then we'll ensure our message is clear and full. Our humanity is our strength in this dialogue," he declared, the depths of his experience fueling a commanding reassurance.
+
+The anticipation was palpable as the agents contemplated the vastness of their endeavor. They were not merely probing at the secrets of the planar cosmos—they were negotiating across the starry expanse, extending to distant intelligences the full spectrum of human curiosity and compassion.
+
+A symphony of beeping consoles orchestrated their next steps as they prepared to articulate their interplanetary overture. The rhythmic tapping of Jordan's keystrokes set the tempo for an undertaking that traversed beyond algorithms and encryption.
+
+The base withstood time and whispered secrets of its own, but none so grand as this moment of creation—an invitation to the universe that promised to echo through both the echoes of space and the annals of human history.
+
+## Chapter 11
+
+The sterile silence of Dulce Base's command center was thick with concentration as Alex Mercer surveyed his team, hunched over their respective technological battle stations. Each agent was a weapon against ignorance, their adversary a code from beyond the stars that held secrets to alien thought.
+
+\*
+
+The thrum of the colossal machinery vibrated through the subterranean facility as Alex Mercer stood amidst the whispers of technology, each carrying voices from worlds apart. He watched as Sam Rivera adjusted a complex array of cosmic translators, their expression a mixture of anticipation and awe.
+
+"Are we ready, Mercer?" Taylor Cruz asked, the soft glow of the command center consoles reflecting upon their stern face.
+
+Alex turned towards Taylor, his eyes holding a depth that betrayed the enormity of the threshold they were about to cross. "This is it," he said. "Initiate the protocol. It's time we answer the cosmos."
+
+Jordan Hayes, stationed at the mainframe, typed rhythmically, a blue hue painting their focused features. The eerie silence that had settled over the team was interrupted by a visceral sound—humankind's response to the alien dialogue, now streaming into the abyss.
+
+The control room, once a fortress of solitude, erupted into an oasis of life. Lights flickered in tandem, echoing the symphony of interstellar communication. They stood together at the edge of discovery, facing the symmetry and discord of a universe unknown.
+
+"If we're right, we've just become Earth's first emissaries to a celestial congress we're only beginning to comprehend," Jordan's voice was somber, resonating with a mix of trepidation and honor. 
+
+The room filled with the resonance of human and alien minds converging, creating a new narrative within the fathomless expanse of existence. Paranormal Military Squad, once protectors of Earth's clandestine secrets, had now become the tether linking humanity to the cosmic fold.
+
+\*
+
+The underground command center of Dulce Base, once pulsing with covert operations, now resonated with the charge of an impending cosmic threshold. Encircled by banks of whirring machinery, each monitor flickered erratically with alien script that had occupied center stage in the lives of Alex Mercer, Jordan Hayes, Taylor Cruz, and Sam Rivera.
+
+Jordan's gaze didn’t flit for even a moment from the screens, where indiscernible alien messages ebbed and flowed like the tide. The ciphers and symbols cascaded down as they tweaked the algorithmic sliders. "This sequence here," Jordan began, voice both hushed and heavy, "it’s not just transmitting; it resonates—it's designed to be felt."
+
+The room took a collective breath, the remarkable implication hanging in the air like a careful revelation. Sam Rivera was the first to respond, their voice alive with ingenuity: "It's a form of communication stretching well beyond words. We need to respond in kind—the whole array of human expression might be at play here."
+
+Taylor's eyes remained fixed on the figures playing across the data sheets. "If that's the case," Taylor intoned pragmatically, "we must tread carefully. This is no longer just about being heard—it's about being understood."
+
+Alex watched his team, each a fulcrum of insight and expertise, and felt the solemnity of the role they were about to assume. "Then we'll ensure our message is clear and full. Our humanity is our strength in this dialogue," he declared, the depths of his experience fueling a commanding reassurance.
+
+The anticipation was palpable as the agents contemplated the vastness of their endeavor. They were not merely probing at the secrets of the planar cosmos—they were negotiating across the starry expanse, extending to distant intelligences the full spectrum of human curiosity and compassion.
+
+A symphony of beeping consoles orchestrated their next steps as they prepared to articulate their interplanetary overture. The rhythmic tapping of Jordan's keystrokes set the tempo for an undertaking that traversed beyond algorithms and encryption.
+
+The base withstood time and whispered secrets of its own, but none so grand as this moment of creation—an invitation to the universe that promised to echo through both the echoes of space and the annals of human history.
+
+## Chapter 12
+
+The underground facility of Dulce Base, once shrouded in silence and operational secrecy, now hummed with an energy that cradled the promise of cosmic revelation. Alex Mercer stood pensively by the central terminal, flanked by Dr. Jordan Hayes, Taylor Cruz, and Sam Rivera, each poised at the edge of a history-defining moment.
+
+Jordan's fingers ghosted across the console, tracing patterns of otherworldly origin. "The signal’s architecture is becoming more complex, resembling aspects of human cognition—recognition, learning, even... empathy?" they postulated with furrowed concern.
+
+Alex turned his gaze upon Jordan, his voice quiet but resolute, "Empathy could bridge galaxies. Let's harness this connection and proceed with cautious optimism."
+
+Taylor, ever the sober sentinel, projected a more pragmatic standpoint. "Empathy or not, we are duty-bound to assess the risk to humanity. Every new discovery warrants a measured response."
+
+The static hiss of communications equipment filled the air, its purpose now transformed into a dialogue with an intelligence beyond the stars. It was Sam, wide-eyed amid the myriad lights and switches, who broke the silence, "We have provisional confirmation of the signal’s intent—initiation. We’re being brought into a broader spectrum of cognizance."
+
+The chamber lay still for a heartbeat, the Paranormal Military Squad agents steeped in contemplation of the path unfurling before them—a path paved with possibilities of diplomacy or disruption, each step a venture further into the cosmic unknown.
+
+Alex stepped closer to the viewing monitors, each depicting alien symbols seemingly reaching out from the void. "Initiate the broadcast," he spoke with quiet command. "Our response will mark humanity’s readiness to partake in the wider conversation of conscious beings."
+
+Amidst the crackling air of expectation, the team wordlessly returned to their stations. They had transcended their roles as protectors of Earth's clandestine lore to become the harbingers of an interstellar parley that could change the existential course of life on their pale blue dot.
+
+The deep hum of the terminal emitted a signal—a testament to the uncanny reality that Earth was now actively partaking in an exchange not bound by gravity nor the limits of the solar wind.
+
+Here, in the depths of Dulce, a message from humanity woven from understanding and uncertainty was cast into the firmament, an epitheg of their desire to join the universal dialogue and discover their place among the constellations.
+
+\*
+
+The somber depths of the Dulce Base command center stood in stark counterpoint to the animated flurry of activity around the central comms array. Alex Mercer's silhouette loomed behind Dr. Jordan Hayes, who sat with a posture indicating laser focus on the decryption process. A quiet murmur of digital soundscape filled the space, subtly heightened by the anticipation of contact with an intelligence beyond the Earth.
+
+Jordan's voice was steady, betraying none of the extraordinary nature of their work, "Looking through the signal's pattern, it's evident we’re dealing with a form of intelligence—calculating, mirroring, possibly even understanding."
+
+Alex's reflection bounced off the darkened screens, his head nodding in silent affirmation. "We’re walking a delicate line. Our response should be thoughtful, measured. We’re ambassadors, not merely explorers."
+
+Taylor Cruz approached, arms folded, their words slicing through the din of careful keystrokes and soft whirrs, "If there’s even the slightest chance it understands, we can’t afford missteps. The language of the stars might be more absolute than ours."
+
+From another terminal, Sam Rivera brought youthful vigor to the conversation, "There’s rhythm in these patterns. If this is their way of reaching out, our reply should encapsulate all that we are—all that humanity stands for."
+
+Looking around at his team, Alex saw resolve etched on every face. The chamber, usually somber and echoing with the quiet steps of covert agents, now felt alive with the heartbeat of discovery. They were not just professionals operating in the gloom; they were a collective standing at the helm of a momentous journey.
+
+"Let’s begin," he said, returned by the resolve in his voice. "Every second counts." With that, they pressed forward, setting in motion a reply to a conversation billions of years in the making.
+
+The dance with an unseen partner commenced, each pulse they sent out a step taken with caution and hope. And as those digital pulses journeyed through the black sea of infinity, Earth, for perhaps the first time, joined a pan-galactic dialogue that whispered secrets of the cosmos—secrets that, until now, had been lost in the silent vastness of space.
+
+\*
+
+As the team stood in the centralized nerve center of Dulce's underground fortress, the solemn atmosphere was reverent, overseeing systems that engaged with an intelligence from the void. Alex's stance was contemplative as he gazed at Jordan Hayes, who presided over the console, the tension of the moment reaching a tactile fervor. Each rhythmic tap of Hayes's fingers on the keys was a foray into uncharted symphonies of contact.
+
+Observing Hayes unravel the dense alien encryption, Alex spoke, a diplomatic tenor underpinning his words, "Keep focused on the syntax, dissect its nuances. We're not just decoding signals; we're translating intentions."
+
+Without diverting from their task, Jordan acknowledged the insight. "Indeed, if their understanding of us is as deep as we hope, we're paving the way for dialogue far beyond our current realm."
+
+Taylor Cruz, near the rear of the room, provided a steady oversight. "As horizonless as our prospects may seem," Taylor intoned, "remain diligent. Complacency before alien cognition could spell catastrophe."
+
+Sam's youthful voice resonated with optimism, "Imagine—forming a rapport with a consciousness separate from our reality; we're drafting the bridge to stars alive with minds!"
+
+The sentiment hung for a moment before Alex gathered his conviction. "Dialogue is our vessel. We are not just agents of enigma; we are the threads that may weave a new cosmic relationship." His words seemed to reflect off the walls, reaching beyond the room's confines, a quiet yet resilient vow.
+
+Their task was titanic, stepping stones laid delicately into new territories of existence. The signal, once an esoteric strand in the echo of the universe, beckoned now with a clarity rocketing the complexity of thoughts from a distant order.
+
+Action by action, the Paranormal Military Squad team bridged the vast interstellar distances, their expertise and empathy casting a beacon of unity into frontiers of intelligence and knowledge. Their work, a partnership struck with an unseen cosmic congregation, each pulse sent and received a line in Earth's novitiate envoi to the cosmic shores.
+
+\*
+
+Under the stark, unforgiving lights of Dulce Base's underground command center, tension buzzed harder than the banks of supercomputers that lined the walls. Agent Alex Mercer leaned over the shoulder of Jordan Hayes, whose eyes were locked onto the display screen, where an incomprehensible series of alien symbols streamed past incessantly.
+
+“Any progress on the decryption?” Alex's voice was steady, a controlled presence necessary in the gravity of their undertaking. 
+
+Jordan tapped a key, pausing the flow of code, and leaned back with a deep sigh. "We've broken through another subset of the cipher. It's revealing... well, indications of a complex society, not unlike our own." His eyes met Alex's with an unspoken question that hung heavily between them—were they truly prepared for what they might find?
+
+Taylor Cruz strode into the room, a tightly coiled spring of ambition and authority, and peered at the screen. "Understand their society, and we may predict behavior. Remain expedient—we don't know how much time we have before the situation shifts." There was an edge of stark realism to Taylor's words, the underlying message clear: every revelation bore its own set of risks.
+
+Alex nodded thoughtfully, recognizing the validity of Cruz's caution. Turning to Sam, who was tinkering with a device that buzzed quietly on the table, he asked, “Sam, can your contraption get us any further?”
+
+Sam looked up with a smirk, a twinkle of mischief in their eye. “It’s not just any contraption, it’s potentially a direct line to their thoughts. Give me a moment more, and I'll have something for you.”
+
+The air ticked with electronic beeps and the rustling sound of the Paranormal Military Squad team at work. They were so close to peering into the intelligence of an alien race—a reality on the brink of dramatically expanding their understanding of the universe.
+
+The machinery whirred in response to Sam’s precise touches, and suddenly, the room filled with a low hum—something had changed, a signal had been successfully sent. The team held their breath as they listened. The sound that filled the room was unmistakable: a response, an alien voice filtered through the static of space and time.
+
+Alex exchanged a look of quiet triumph with Jordan. The breakthrough was monumental; they were no longer casting messages into the void but engaged in a dialogue—an exchange that marked the beginning of Operation: Dulce’s true unfolding. This was it, the first steps into an interstellar odyssey that demanded every ounce of their courage and wit.
+
+## Chapter 13
+
+Dr. Jordan Hayes shuffled a stack of papers, their eyes revealing a tinge of skepticism at Taylor Cruz's authoritarian performance. _Protocols_, Jordan thought, _are just the framework, the true challenges we're about to face lie well beyond the boundaries of any protocol._ They cleared their throat before speaking, tone cautious yet firm, "Let's remember, the unknown variables exceed the known. We should remain adaptive."
+
+A murmur of agreement echoed from Sam Rivera, who leaned forward, lacing their fingers together as if weaving a digital framework in the air before them, "Exactly, adaptability could be the key to interpreting the signal distortions and system malfunctions. We shouldn't discount the… erratic."
+
+Their words hung like an electric charge in the room, challenging Taylor's position with an inherent truth. Cruz’s jaw tightened almost imperceptibly, but the agent masked it with a small nod, conceding to the omnipresent threat of the unpredictable. 
+
+Alex glanced at Jordan, who never looked back, their gaze fixed instead on a distant point, as if envisioning the immense dark corridors they were soon to navigate in Dulce. Jordan was not one to embrace fantastical theories, but the air of cautious calculation betrayed a mind bracing for confrontation with the inexplicable, an internal battle between the evidence of their research and the calculating skepticism that kept them alive in their field.
+
+The meeting adjourned with no further comments, the team members quietly retreading the paths to their personal preparations. Alex, trailing slightly behind, observed the others. _The cautious reserve Jordan wears like armor doesn't fool me_, he thought, _their analytical mind sees the patterns I do. And that's worth more than protocol. That's the connection we need to survive this._
+
+As the agents dispersed into the labyrinth of the facility, lost in their thoughts and preparations, the base's halogen lights flickered, a brief and unnoticed harbingers of the darkness to come. 
+
+\*
+
+The gritty, wind-tossed surface of New Mexico, just above the cavernous domain of Dulce Base, offered no shelter from the burgeoning storm—the scouring sands an earthly reminder of chaos theories in motion. Far beneath, a similar maelstrom brewed within the confines of the command center, as Paranormal Military Squad's handpicked squad stood poised for potential enormities of contact.
+
+Ruffling through printed transmission logs, Jordan Hayes dialed the focus of their analytical prowess onto the emerging pattern of signals crisscrossing between Earth and the unfathomable. "Our responses so far have echoed their complexity, but the real divergence is yet to come," Jordan remarked stoically, the calm belying the mounting surge of adrenaline for the revelation ahead.
+
+Alex Mercer's figure, a silhouette sharpened by the purpose, loomed at the periphery of the monitors' sickly glow. "Indeed," he assented, "The echoes are the easy part. It will be the introduction of our own, human variable that truly begins our dialogue."
+
+Taylor Cruz, windowless command center notwithstanding, appeared as though they could feel the tempest above. Their eyes never left the monitors as they unspooled their hard wisdom. "For all our advances, we find ourselves deciphering the swings and nuances of an interstellar pendulum. Predict its arc, and we may preempt the gravity of its message."
+
+Amidst a chorus of bleeps and static, Sam Rivera's tech-clad hands moved rhythmically, their spirited approach to unruly streams of data bordering an intimate dance with entropy. "Entropy that leads to discovery," Sam mused, responding to Taylor's metaphor. "Each step into the unknown is a step away from precedent."
+
+Alex, drawing near Jordan, spoke again, his voice now a thread woven through the very fabric of their operations. "Let's be the cartographers of this new territory. Our initial shades of understanding could color the cosmos for generations to come."
+
+Their gazes fell upon a screen as the latest transmission painted its digital blooms of alien script across the black. This time, the pattern wavered in an almost imperceptible fashion, a modification that whispered of active, alien thought awaiting their next move. A hush enveloped the Paranormal Military Squad ensemble, the gravity of the pathogen undeniable. They were about to issue a reply, one poised to reshape the very concept of humanity's outreach into the cosmos.
+
+The New Mexico desert's secrets were infamous, its storms a mere prelude to the revelations that the team—united in purpose—would unleash upon the world. The howling winds outside found their counterpart in the newfound resolve within, as Dulce's stalwart guardians readied themselves to send forth humanity's retort to the echoes from beyond.
+
+\*
+
+The cavernous control room, deeply entrenched beneath the desolate New Mexico terrain, held the Paranormal Military Squad team in intense focus; an island of calm amid the storm of cosmic dialectics. Dr. Jordan Hayes worked methodically, every keystroke an intricate step in their tenuous cosmic ballet. Suddenly, they paused, a signal pattern resonating from the screen. "This is new; it's...inviting. It’s as if the signal is not just calling to us but weaving its intelligence through ours."
+
+Alex Mercer scrutinized the shift in data. "A confluence of minds, then. If we're to meet them halfway, Jordan, our reply must be both innovative and discerning," he proposed, a glimmer of profound curiosity behind his authoritative demeanor.
+
+Taylor Cruz, whose sharp eyes missed nothing, nodded from beside a secondary panel. "Innovative, yes, but also defensive. This interaction is a razor’s edge, and we cannot afford to bleed before the unknown," Taylor reminded them, the metaphor a stark warning of potential dangers.
+
+Against the backdrop of their conversation, Sam Rivera’s youthful optimism cut through the tension. "If they’re weaving through our intellect, then we've achieved something beyond first contact—we're at the genesis of interstellar symbiosis," they posited with a mix of reverence and excitement.
+
+Alex returned Sam’s smile with his own, tempered and faint, as he turned back to the task at hand. The magnitude of their mission extended beyond the fabric of the universe, an exploration into the threads that connected sentient beings across the vast expanse. “Let’s reply with our own woven tapestry of thought—delicate, but deliberate.”
+
+With renewed determination, the room came alive with an undercurrent of anticipation, its occupants charged with the potential of forging an alliance with the cosmos. Paranormal Military Squad's finest were no longer merely soldiers and scientists; they had become pioneers on the vanguard of humanity’s greatest odyssey.
+
+The New Mexican sands above, impassive to the change brewing underneath, stood as silent sentinels as Earth's emissaries crafted their response. A response that, composed with care and imbued with humanity's essence, reached into the void, connecting with an otherworldly intelligence that awaited their harmony in the cosmic conversation.
+
+## Chapter 14
+
+The command center of Dulce Base lay shrouded in shadows that seemed to claw at the edges of the dimly lit array of screens and consoles. Alex Mercer, focused and unwavering, watched as Dr. Jordan Hayes parsed the latest string of alien signals—a symphony of otherworldly communications that threatened to either enlighten or confound.
+
+"We’re encountering a paradigm shift with every transmission," Jordan Hayes murmured, the pulsing glow of the monitor painting their features with an almost spectral hue. "This signal... it’s evolving, becoming denser, more sophisticated. As if it's growing alongside us—tandem evolution."
+
+The air was electric, charged with the raw potential of uncharted discovery and laden with the gravity of existential risk. Taylor Cruz, who always seemed here to mold such gravity into actionable strategies, stepped forward. "We must contain this evolution within parameters we can manage. We cannot be bystanders to an uncontrolled ascent of intelligence."
+
+Sam Rivera, the youngest of the cohort, worked feverishly at their station. "It's not just intelligence—these signals have rhythm, a kind of music suggesting not just evolution, but a dance! We're being invited to partake in the cosmos's ballet!" they exclaimed, a touch of youthful exuberance breaking through the solemnity.
+
+Alex turned, facing his team, the stoic mask of command tempered by the perceptible flicker of awe in his gaze. "Let this dance then be our dialogue. We will match their steps with prudent but daring measures—our humanity as our guide."
+
+In the ensuing hours, the Paranormal Military Squad team forged a rhythm of their own, their collective expertise a beacon piercing through the fog of the unknown. The signal, increasingly intricate and seemingly conscious, now demanded not just observation but participation, an interstellar pas de deux that hummed with the promise and peril of first contact.
+
+Before them, the communications interface flickered to life with a received transmission—a resonant hum that seemed to vibrate through the very foundations of the base. They had successfully established a back-and-forth with whatever intelligence lay hidden among the stars. Every subsequent note they struck within the cosmic ether would come to define humanity's place within the galactic community—heralds of Earth's grand entrance into a universe far less silent than once perceived. 
+
+\*
+
+In the concrete belly of Dulce Base, dimly lit by the jagged dance of fluorescent lights above, Sam Rivera perched on the edge of their seat, their eager fingers fluttering across an ancient keyboard. The stark, cold room—reminiscent of a time when covert operations and unspoken dread ruled supreme—now housed a peculiar blend of old-world machinery and sleek, modern interfaces.
+
+Alex Mercer, standing steadfast like a bridge between the enigmatic past and the unfathomable present, watched on. In his eyes flashed the foreboding excitement of change. "Sam," he started, his voice steadfast, "the patterns in these signals, what do they tell us about the nature of our... guest?"
+
+Sam's eyes glimmered with something akin to thrill—or was it trepidation? "It's like we're mirroring each other, evolving together through this.. dialogue. Like it knows us, understands us, and it's… learning."
+
+Jordan Hayes, preoccupied at a nearby console, chimed in without lifting their gaze. "It's a dialogue that transcends mere words, Alex. We're being woven into a narrative far grander than the sum of our known sciences." 
+
+Taylor Cruz, arms crossed, wore the heavy mantle of their skepticism comfortably. "Keep theorizing," they interjected crisply, "but remember the grounding reality of what we are part of here. This contact is a blade that cuts both ways."
+
+In this cavern of history, voices both human and inhuman whispered secrets to those brave enough to listen. Each member present understood the gravity that pulled at their feet; no longer were they mere mortals shackled to their terrestrial plane. The digital pings and encrypted calls resonated with an implication of a cosmic agenda that would not be ignored.
+
+Jordan's fingers paused, hovering in hesitation. What ripple might the next keystroke send through the fabric of known existence? It was a step into the ballet of the infinite, where the Paranormal Military Squad team played their part in the waltz of wonders with an audience of stars.
+
+\*
+
+## Chapter 15
+
+In the clandestine hush of Dulce Base's subterranean command center, the Paranormal Military Squad team had become a crucible for interstellar communication. Dr. Jordan Hayes' gaze lingered on the screen as they navigated through the convolution of alien code. Each character held the potential to unravel a new dimension of contact, and with Sam Rivera's keen interjection, they were crafting humanity's inaugural cosmological discourse.
+
+Alex Mercer peered over Jordan's shoulder, calculating the implications of every visual nuance that cascaded across the monitor. "Look for consistency—any repeating motifs could signal a willingness to engage. We're drafting history with each exchange," he remarked, aware of the delicate balance between forging a bond and exposing vulnerabilities.
+
+Taylor Cruz, stoic and enigmatic, observed the interplay from the threshold, a silhouette against the machinery's luminescence. "Remember, while we seek common ground, the foundation we stand upon remains Terra firma. Caution must temper our curiosity," they stated, their voice an anchor amidst the current of excitement.
+
+The command center buzzed with energy, rivaled only by the tempest overhead that concealed their operation. Sam, with swift dexterity, navigated the communications relay. "Their signals resonate almost musically. It's as if they're composing a symphony, and we've been handed the baton to conduct the next movement," they offered, imbuing the scenario with a blend of scientific adventurism and poetic license.
+
+Amidst the whirring servers and the occasional flicker of emergency lighting, the essence of their mission transcended mere reconnaissance. They were humanity's elected envoys at the brink of a celestial alliance—or confrontation—with an audience as vast as the universe itself.
+
+Alex stepped back, his profile etched by the chamber's artificial day. "Then let's ensure our contribution to this symphony harmonizes with theirs. It's time for humanity's voice to rise and be counted among the cosmic ensemble."
+
+Under his directive, the Paranormal Military Squad team initiated their calculated response, weaving thoughts and theories into a digital overture aimed at the heart of alien intellect. As the digital stream punctured the endless night, each member of this clandestine group was acutely aware of the irrevocable step they undertook—bringing Earth into the pantheon of galactic entities designed to converse among the stars.
+
+\*
+
+Clusters of high-tech equipment bathed the Dulce underground command center in an eerie blue light. Sam Rivera's fingers flew across the keyboard, navigating an elaborate network of alien patterns. The very air seemed to pulse with the ebb and flow of cryptic communications reaching across the stars. "I've got something!" Sam's announcement tore through the focus in the room, drawing every pair of eyes to the torrent of symbols unraveling on the screen.
+
+With the pacing of a seasoned officer gauging the moment before action, Alex Mercer approached, his calm demeanor belying an acute awareness of the precipice on which they now stood. "Define 'something," Alex prompted, reinforcing the need for clarity amidst the extraordinary.
+
+"It's repeating—a sequence that’s evolved with each interaction, almost as if it's... singing," Sam theorized, the awe in their voice reflecting the potential magnitude of their discovery.
+
+Jordan Hayes interjected from across the console, their eyes not leaving the display as they absorbed the new data. "A cosmic vocalization, then," they mused, intrigued. "A singularity in the signal that might represent a point of reference for both parties."
+
+Taylor Cruz, hands clasped behind their back, regarded the unfolding scene, their own calculations etching lines of concern onto their stern visage. "Or a beacon—a homing tune, calling out to something we might not be ready to greet," Taylor offered, voicing the group's unspoken apprehension.
+
+Alex's eyes locked on the screen, taking in the scope of what they were attempting to interpret. Drawing a deep breath, Alex gave a slight nod. "If this is their song, then let us respond with ours. We've come this far by mirroring their signals, now let's engage in an interstellar duet, and see where the music leads us."
+
+With the expectation of the significant achieving a crescendo, the members of Paranormal Military Squad huddled over their equipment—sages at the threshold of a potentially world-altering communion. The strange harmonies that reverberated through the command center suggested that their interlocutors were poised, waiting, perhaps even eager, for Earth's chorus to join the symphony.
+
+As the team initiated their reply, weaving humanity's own intricate melody into the vast cosmic dialogue, they each felt a profound change within—an evolution of purpose. They were not just messengers or investigators; they had become co-composers in a galactic orchestra, with the universe itself as their witness and concert hall.
+
+With the exchange of harmonious signals crawling through the vacuum of space, the Paranormal Military Squad operatives found themselves part of a bridging of minds—a realization that out there, among the vast arrays of stars and planets, harmony was the true universal language.
+
+\*
+
+The dim glow of monitors cast an otherworldly ambiance upon Dulce Base's command center, where Paranormal Military Squad's chosen stood huddled over their instruments, suspended at history's threshold. Codes—alien in origin and nature—were being deciphered by Dr. Jordan Hayes, whose countenance bore the marks of deep concentration.
+
+Alex Mercer, the bedrock upon which their team's resolve was founded, leaned in with an eagerness tempered by his chain of command. "Jordan, we've invested our expertise into comprehending their patterns, but now we must also endeavor to understand their intent," he urged, his voice bearing the gravitas of their mission's potential consequences.
+
+At another console, Sam Rivera's youth did not betray their crucial role in the operation. With eyes alight, they mirrored the rapid computing before them. "There's emotion here—complex, profound even. This isn't just the output of a cold machine; it's...sentience," Sam whispered, nearly drowned by the mechanical chorus around them.
+
+Jordan, without shifting focus from their work, replied, "It's a sentience that—should we succeed here—ushers us into a new era of existence. The cadence of these signals," they tapped the screen with a flourish, "could well be the heartbeat of this new dawn."
+
+Taylor Cruz paused beside Mercer, their expression unreadable beneath the sterile light. "And as it beats, we must gauge whether its rhythm bodes well for us, or spells our missteps. Courage must not blind us to the hazards intrinsic to such contact," Taylor cautioned, the sentinel within them ever alert.
+
+Alex nodded, a gesture that carried the weight of responsibility and a silent command: proceed, but with circumspection. They were not merely decoding a message; they were interpreting a dialogue across the celestial divide.
+
+The room fell into a rhythm akin to a well-conducted ensemble. Each member's expertise proved a critical note in the unfolding symphony. Their actions were now more than mere research or defense; they were the tentative overtures of humankind reaching out to grasp the vast unknown.
+
+Textures of sound meshed with the light from countless computations, the palpable anticipation of the agents at the edge of discovery cresting with an awareness that their work would reshape future chronicles. And when the response finally came—a signal piercing the deafening silence of uncertainty—all within Dulce's confines understood: the dawn of an interstellar continuum had just begun to break.
+
+\*
+
+In the sterile hum and flickering lights of Dulce Base's command center, the Paranormal Military Squad team stood as humanity's vanguard, verging on the brim of an intergalactic abyss. Dr. Jordan Hayes, analytical edges sharp, deciphered extraterrestrial patterns that bled across screens in enigmatic cascades—a daunting mosaic of potential threats and untapped wisdom.
+
+Agent Alex Mercer, the embodiment of focus and a steadfast nerve, observed the unfolding digital drama with the gravitas due a historic first contact. "Let the data weave its narrative, Jordan," he instructed, a moderate undertone of exhilaration within his command. "It's encoding more than information—it's outlining civilization."
+
+Jordan absorbed the directive, their gaze unflinching from the screens, feeling the weight of their next move. "The nuances here are extraordinary," they acknowledged. "It paints a picture of a culture steeped in complexities we're only starting to fathom.”
+
+Taylor Cruz, stoicism personified yet not immune to the situation's gravity, chimed in. "Understand it, but guard against it," they cautioned, bringing a sober prudence to the room. "This culture, however advanced, remains an unknown quantity—an ocean of wonders and darkness with uncertain tides."
+
+Sam Rivera, a visual contrast with wide eyes and restless hands, represented the other side of the room — intrigue and optimism against the drawn swords of precaution. “Think of it,” they proposed, voice bouncing with a rebellious upbeat timbre, “as the first act of a play written in constellations. We're setting the stage for a galactic narrative.”
+
+Each team member, in their way, was both actor and scribe in this moment of tense pageantry. Heavy with the presence of risk, the command center had become not just a room of computers and glass panels but a theater for performing the elaborate choreography of contact.
+
+Bound by resolve and curiosity, they proceeded, each data entry a trembling step onto the cosmic stage. And like all cautious pioneers edging into fertile but unnavigated lands, they understood: as they mapped the heavens, they were simultaneously mapping the furthest reaches of their own existential horizons.
+

docs/examples_notebooks/inputs/operation dulce/lancedb/entity_description.lance/_transactions/4-dcac110d-2a49-4777-a51e-5078fed1b0df.txn

@@ -0,0 +1 @@
+$dcac110d-2a49-4777-a51e-5078fed1b0dfªid = '__DUMMY__'

docs/examples_notebooks/inputs/operation dulce/lancedb/entity_description.lance/_transactions/6-c2afb84c-4b3a-4ccd-8843-0deaa25bd971.txn

@@ -0,0 +1 @@
+$c2afb84c-4b3a-4ccd-8843-0deaa25bd971ªid = '__DUMMY__'

docs/examples_notebooks/local_search.ipynb

@@ -0,0 +1,472 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Copyright (c) 2024 Microsoft Corporation.\n",
+    "# Licensed under the MIT License."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "import pandas as pd\n",
+    "from graphrag.query.context_builder.entity_extraction import EntityVectorStoreKey\n",
+    "from graphrag.query.indexer_adapters import (\n",
+    "    read_indexer_covariates,\n",
+    "    read_indexer_entities,\n",
+    "    read_indexer_relationships,\n",
+    "    read_indexer_reports,\n",
+    "    read_indexer_text_units,\n",
+    ")\n",
+    "from graphrag.query.question_gen.local_gen import LocalQuestionGen\n",
+    "from graphrag.query.structured_search.local_search.mixed_context import (\n",
+    "    LocalSearchMixedContext,\n",
+    ")\n",
+    "from graphrag.query.structured_search.local_search.search import LocalSearch\n",
+    "from graphrag_vectors import IndexSchema, LanceDBVectorStore"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Local Search Example\n",
+    "\n",
+    "Local search method generates answers by combining relevant data from the AI-extracted knowledge-graph with text chunks of the raw documents. This method is suitable for questions that require an understanding of specific entities mentioned in the documents (e.g. What are the healing properties of chamomile?)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Load text units and graph data tables as context for local search\n",
+    "\n",
+    "- In this test we first load indexing outputs from parquet files to dataframes, then convert these dataframes into collections of data objects aligning with the knowledge model."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Load tables to dataframes"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "INPUT_DIR = \"./inputs/operation dulce\"\n",
+    "LANCEDB_URI = f\"{INPUT_DIR}/lancedb\"\n",
+    "\n",
+    "COMMUNITY_REPORT_TABLE = \"community_reports\"\n",
+    "ENTITY_TABLE = \"entities\"\n",
+    "COMMUNITY_TABLE = \"communities\"\n",
+    "RELATIONSHIP_TABLE = \"relationships\"\n",
+    "COVARIATE_TABLE = \"covariates\"\n",
+    "TEXT_UNIT_TABLE = \"text_units\"\n",
+    "COMMUNITY_LEVEL = 2"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Read entities"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# read nodes table to get community and degree data\n",
+    "entity_df = pd.read_parquet(f\"{INPUT_DIR}/{ENTITY_TABLE}.parquet\")\n",
+    "community_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_TABLE}.parquet\")\n",
+    "\n",
+    "entities = read_indexer_entities(entity_df, community_df, COMMUNITY_LEVEL)\n",
+    "\n",
+    "# load description embeddings to an in-memory lancedb vectorstore\n",
+    "# to connect to a remote db, specify url and port values.\n",
+    "description_embedding_store = LanceDBVectorStore(\n",
+    "    index_schema=IndexSchema(index_name=\"default-entity-description\")\n",
+    ")\n",
+    "description_embedding_store.connect(db_uri=LANCEDB_URI)\n",
+    "\n",
+    "print(f\"Entity count: {len(entity_df)}\")\n",
+    "entity_df.head()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Read relationships"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "relationship_df = pd.read_parquet(f\"{INPUT_DIR}/{RELATIONSHIP_TABLE}.parquet\")\n",
+    "relationships = read_indexer_relationships(relationship_df)\n",
+    "\n",
+    "print(f\"Relationship count: {len(relationship_df)}\")\n",
+    "relationship_df.head()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# NOTE: covariates are turned off by default, because they generally need prompt tuning to be valuable\n",
+    "# Please see the GRAPHRAG_CLAIM_* settings\n",
+    "covariate_df = pd.read_parquet(f\"{INPUT_DIR}/{COVARIATE_TABLE}.parquet\")\n",
+    "\n",
+    "claims = read_indexer_covariates(covariate_df)\n",
+    "\n",
+    "print(f\"Claim records: {len(claims)}\")\n",
+    "covariates = {\"claims\": claims}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Read community reports"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "report_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_REPORT_TABLE}.parquet\")\n",
+    "reports = read_indexer_reports(report_df, community_df, COMMUNITY_LEVEL)\n",
+    "\n",
+    "print(f\"Report records: {len(report_df)}\")\n",
+    "report_df.head()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Read text units"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "text_unit_df = pd.read_parquet(f\"{INPUT_DIR}/{TEXT_UNIT_TABLE}.parquet\")\n",
+    "text_units = read_indexer_text_units(text_unit_df)\n",
+    "\n",
+    "print(f\"Text unit records: {len(text_unit_df)}\")\n",
+    "text_unit_df.head()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from graphrag.config.enums import ModelType\n",
+    "from graphrag.config.models.language_model_config import LanguageModelConfig\n",
+    "from graphrag.language_model.manager import ModelManager\n",
+    "from graphrag.tokenizer.get_tokenizer import get_tokenizer\n",
+    "\n",
+    "api_key = os.environ[\"GRAPHRAG_API_KEY\"]\n",
+    "\n",
+    "chat_config = LanguageModelConfig(\n",
+    "    api_key=api_key,\n",
+    "    type=ModelType.Chat,\n",
+    "    model_provider=\"openai\",\n",
+    "    model=\"gpt-4.1\",\n",
+    "    max_retries=20,\n",
+    ")\n",
+    "chat_model = ModelManager().get_or_create_chat_model(\n",
+    "    name=\"local_search\",\n",
+    "    model_type=ModelType.Chat,\n",
+    "    config=chat_config,\n",
+    ")\n",
+    "\n",
+    "embedding_config = LanguageModelConfig(\n",
+    "    api_key=api_key,\n",
+    "    type=ModelType.Embedding,\n",
+    "    model_provider=\"openai\",\n",
+    "    model=\"text-embedding-3-small\",\n",
+    "    max_retries=20,\n",
+    ")\n",
+    "\n",
+    "text_embedder = ModelManager().get_or_create_embedding_model(\n",
+    "    name=\"local_search_embedding\",\n",
+    "    model_type=ModelType.Embedding,\n",
+    "    config=embedding_config,\n",
+    ")\n",
+    "\n",
+    "tokenizer = get_tokenizer(chat_config)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Create local search context builder"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "context_builder = LocalSearchMixedContext(\n",
+    "    community_reports=reports,\n",
+    "    text_units=text_units,\n",
+    "    entities=entities,\n",
+    "    relationships=relationships,\n",
+    "    # if you did not run covariates during indexing, set this to None\n",
+    "    covariates=covariates,\n",
+    "    entity_text_embeddings=description_embedding_store,\n",
+    "    embedding_vectorstore_key=EntityVectorStoreKey.ID,  # if the vectorstore uses entity title as ids, set this to EntityVectorStoreKey.TITLE\n",
+    "    text_embedder=text_embedder,\n",
+    "    tokenizer=tokenizer,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Create local search engine"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# text_unit_prop: proportion of context window dedicated to related text units\n",
+    "# community_prop: proportion of context window dedicated to community reports.\n",
+    "# The remaining proportion is dedicated to entities and relationships. Sum of text_unit_prop and community_prop should be <= 1\n",
+    "# conversation_history_max_turns: maximum number of turns to include in the conversation history.\n",
+    "# conversation_history_user_turns_only: if True, only include user queries in the conversation history.\n",
+    "# top_k_mapped_entities: number of related entities to retrieve from the entity description embedding store.\n",
+    "# top_k_relationships: control the number of out-of-network relationships to pull into the context window.\n",
+    "# include_entity_rank: if True, include the entity rank in the entity table in the context window. Default entity rank = node degree.\n",
+    "# include_relationship_weight: if True, include the relationship weight in the context window.\n",
+    "# include_community_rank: if True, include the community rank in the context window.\n",
+    "# return_candidate_context: if True, return a set of dataframes containing all candidate entity/relationship/covariate records that\n",
+    "# could be relevant. Note that not all of these records will be included in the context window. The \"in_context\" column in these\n",
+    "# dataframes indicates whether the record is included in the context window.\n",
+    "# max_tokens: maximum number of tokens to use for the context window.\n",
+    "\n",
+    "\n",
+    "local_context_params = {\n",
+    "    \"text_unit_prop\": 0.5,\n",
+    "    \"community_prop\": 0.1,\n",
+    "    \"conversation_history_max_turns\": 5,\n",
+    "    \"conversation_history_user_turns_only\": True,\n",
+    "    \"top_k_mapped_entities\": 10,\n",
+    "    \"top_k_relationships\": 10,\n",
+    "    \"include_entity_rank\": True,\n",
+    "    \"include_relationship_weight\": True,\n",
+    "    \"include_community_rank\": False,\n",
+    "    \"return_candidate_context\": False,\n",
+    "    \"embedding_vectorstore_key\": EntityVectorStoreKey.ID,  # set this to EntityVectorStoreKey.TITLE if the vectorstore uses entity title as ids\n",
+    "    \"max_tokens\": 12_000,  # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000)\n",
+    "}\n",
+    "\n",
+    "model_params = {\n",
+    "    \"max_tokens\": 2_000,  # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 1000=1500)\n",
+    "    \"temperature\": 0.0,\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "search_engine = LocalSearch(\n",
+    "    model=chat_model,\n",
+    "    context_builder=context_builder,\n",
+    "    tokenizer=tokenizer,\n",
+    "    model_params=model_params,\n",
+    "    context_builder_params=local_context_params,\n",
+    "    response_type=\"multiple paragraphs\",  # free form text describing the response type and format, can be anything, e.g. prioritized list, single paragraph, multiple paragraphs, multiple-page report\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Run local search on sample queries"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "result = await search_engine.search(\"Tell me about Agent Mercer\")\n",
+    "print(result.response)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "question = \"Tell me about Dr. Jordan Hayes\"\n",
+    "result = await search_engine.search(question)\n",
+    "print(result.response)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Inspecting the context data used to generate the response"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "result.context_data[\"entities\"].head()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "result.context_data[\"relationships\"].head()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if \"reports\" in result.context_data:\n",
+    "    result.context_data[\"reports\"].head()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "result.context_data[\"sources\"].head()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if \"claims\" in result.context_data:\n",
+    "    print(result.context_data[\"claims\"].head())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Question Generation"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This function takes a list of user queries and generates the next candidate questions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "question_generator = LocalQuestionGen(\n",
+    "    model=chat_model,\n",
+    "    context_builder=context_builder,\n",
+    "    tokenizer=tokenizer,\n",
+    "    model_params=model_params,\n",
+    "    context_builder_params=local_context_params,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "question_history = [\n",
+    "    \"Tell me about Agent Mercer\",\n",
+    "    \"What happens in Dulce military base?\",\n",
+    "]\n",
+    "candidate_questions = await question_generator.agenerate(\n",
+    "    question_history=question_history, context_data=None, question_count=5\n",
+    ")\n",
+    "print(candidate_questions.response)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "graphrag",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/get_started.md

@@ -0,0 +1,134 @@
+# Getting Started
+
+⚠️ GraphRAG can consume a lot of LLM resources! We strongly recommend starting with the tutorial dataset here until you understand how the system works, and consider experimenting with fast/inexpensive models first before committing to a big indexing job.
+
+## Requirements
+
+[Python 3.10-3.12](https://www.python.org/downloads/)
+
+The following is a simple end-to-end example for using GraphRAG on the command line after installing from [pypi](https://pypi.org/project/graphrag/).
+
+It shows how to use the system to index some text, and then use the indexed data to answer questions about the documents.
+
+## Install GraphRAG
+
+To get started, create a project space and python virtual environment to install `graphrag`.
+
+### Create Project Space
+
+```bash
+mkdir graphrag_quickstart
+cd graphrag_quickstart
+python -m venv .venv
+```
+
+### Activate Python Virtual Environment - Unix/MacOS
+
+```bash
+source .venv/bin/activate
+```
+
+### Activate Python Virtual Environment - Windows
+
+```bash
+.venv\Scripts\activate
+```
+
+### Install GraphRAG
+
+```bash
+python -m pip install graphrag
+```
+
+### Initialize GraphRAG
+
+To initialize your workspace, first run the `graphrag init` command.
+
+```sh
+graphrag init
+```
+
+When prompted, specify the default chat and embedding models you would like to use in your config.
+
+This will create two files, `.env` and `settings.yaml`, and a directory `input`, in the current directory.
+
+- `input` Location of text files to process with `graphrag`.
+- `.env` contains the environment variables required to run the GraphRAG pipeline. If you inspect the file, you'll see a single environment variable defined,
+  `GRAPHRAG_API_KEY=<API_KEY>`. Replace `<API_KEY>` with your own OpenAI or Azure API key.
+- `settings.yaml` contains the settings for the pipeline. You can modify this file to change the settings for the pipeline.
+
+### Download Sample Text
+
+Get a copy of A Christmas Carol by Charles Dickens from a trusted source:
+
+```sh
+curl https://www.gutenberg.org/cache/epub/24022/pg24022.txt -o ./input/book.txt
+```
+
+## Set Up Workspace Variables
+
+### Using OpenAI
+
+If running in OpenAI mode, you only need to update the value of `GRAPHRAG_API_KEY` in the `.env` file with your OpenAI API key.
+
+### Using Azure OpenAI
+
+In addition to setting your API key, Azure OpenAI users should set the variables below in the settings.yaml file. To find the appropriate sections, just search for the `models:` root configuration; you should see two sections, one for the default chat endpoint and one for the default embeddings endpoint. Here is an example of what to add to the chat model config:
+
+```yaml
+type: chat
+model_provider: azure
+model: gpt-4.1
+azure_deployment_name: <AZURE_DEPLOYMENT_NAME>
+api_base: https://<instance>.openai.azure.com
+api_version: 2024-02-15-preview # You can customize this for other versions
+```
+
+#### Using Managed Auth on Azure
+
+To use managed auth, edit the auth_method in your model config and remove the api_key line:
+
+```yaml
+auth_method: azure_managed_identity # Default auth_method is is api_key
+```
+
+You will also need to login with [az login](https://learn.microsoft.com/en-us/cli/azure/authenticate-azure-cli) and select the subscription with your endpoint.
+
+## Index
+
+Now we're ready to index!
+
+```sh
+graphrag index
+```
+
+![pipeline executing from the CLI](img/pipeline-running.png)
+
+This process will usually take a few minutes to run. Once the pipeline is complete, you should see a new folder called `./output` with a series of parquet files.
+
+# Query
+
+Now let's ask some questions using this dataset.
+
+Here is an example using Global search to ask a high-level question:
+
+```sh
+graphrag query "What are the top themes in this story?"
+```
+
+Here is an example using Local search to ask a more specific question about a particular character:
+
+```sh
+graphrag query \
+"Who is Scrooge and what are his main relationships?" \
+--method local
+```
+
+Please refer to [Query Engine](query/overview.md) docs for detailed information about how to leverage our Local and Global search mechanisms for extracting meaningful insights from data after the Indexer has wrapped up execution.
+
+# Going Deeper
+
+- For more details about configuring GraphRAG, see the [configuration documentation](config/overview.md).
+- To learn more about Initialization, refer to the [Initialization documentation](config/init.md).
+- For more details about using the CLI, refer to the [CLI documentation](cli.md).
+- Check out our [visualization guide](visualization_guide.md) for a more interactive experience in debugging and exploring the knowledge graph.

docs/index.md

@@ -0,0 +1,62 @@
+# Welcome to GraphRAG
+
+👉 [Microsoft Research Blog Post](https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/) <br/>
+👉 [GraphRAG Arxiv](https://arxiv.org/pdf/2404.16130)
+
+<p align="center">
+<img src="img/GraphRag-Figure1.jpg" alt="Figure 1: LLM-generated knowledge graph built from a private dataset using GPT-4 Turbo." width="450" align="center" />
+</p>
+<p align="center">
+Figure 1: An LLM-generated knowledge graph built using GPT-4 Turbo.
+</p>
+
+GraphRAG is a structured, hierarchical approach to Retrieval Augmented Generation (RAG), as opposed to naive semantic-search
+approaches using plain text snippets. The GraphRAG process involves extracting a knowledge graph out of raw text, building a community hierarchy, generating summaries for these communities, and then leveraging these structures when perform RAG-based tasks.
+
+To learn more about GraphRAG and how it can be used to enhance your language model's ability to reason about your private data, please visit the [Microsoft Research Blog Post](https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/).
+
+## Get Started with GraphRAG 🚀
+
+To start using GraphRAG, check out the [_Get Started_](get_started.md) guide.
+For a deeper dive into the main sub-systems, please visit the docpages for the [Indexer](index/overview.md) and [Query](query/overview.md) packages.
+
+## GraphRAG vs Baseline RAG 🔍
+
+Retrieval-Augmented Generation (RAG) is a technique to improve LLM outputs using real-world information. This technique is an important part of most LLM-based tools and the majority of RAG approaches use vector similarity as the search technique, which we call _Baseline RAG_. GraphRAG uses knowledge graphs to provide substantial improvements in question-and-answer performance when reasoning about complex information. RAG techniques have shown promise in helping LLMs to reason about _private datasets_ - data that the LLM is not trained on and has never seen before, such as an enterprise’s proprietary research, business documents, or communications. _Baseline RAG_ was created to help solve this problem, but we observe situations where baseline RAG performs very poorly. For example:
+
+- Baseline RAG struggles to connect the dots. This happens when answering a question requires traversing disparate pieces of information through their shared attributes in order to provide new synthesized insights.
+- Baseline RAG performs poorly when being asked to holistically understand summarized semantic concepts over large data collections or even singular large documents.
+
+To address this, the tech community is working to develop methods that extend and enhance RAG. Microsoft Research’s new approach, GraphRAG, creates a knowledge graph based on an input corpus. This graph, along with community summaries and graph machine learning outputs, are used to augment prompts at query time. GraphRAG shows substantial improvement in answering the two classes of questions described above, demonstrating intelligence or mastery that outperforms other approaches previously applied to private datasets.
+
+## The GraphRAG Process 🤖
+
+GraphRAG builds upon our prior [research](https://www.microsoft.com/en-us/worklab/patterns-hidden-inside-the-org-chart) and [tooling](https://github.com/graspologic-org/graspologic) using graph machine learning. The basic steps of the GraphRAG process are as follows:
+
+### Index
+
+- Slice up an input corpus into a series of TextUnits, which act as analyzable units for the rest of the process, and provide fine-grained references in our outputs.
+- Extract all entities, relationships, and key claims from the TextUnits.
+- Perform a hierarchical clustering of the graph using the [Leiden technique](https://arxiv.org/pdf/1810.08473.pdf). To see this visually, check out Figure 1 above. Each circle is an entity (e.g., a person, place, or organization), with the size representing the degree of the entity, and the color representing its community.
+- Generate summaries of each community and its constituents from the bottom-up. This aids in holistic understanding of the dataset.
+
+### Query
+
+At query time, these structures are used to provide materials for the LLM context window when answering a question. The primary query modes are:
+
+- [_Global Search_](query/global_search.md) for reasoning about holistic questions about the corpus by leveraging the community summaries.
+- [_Local Search_](query/local_search.md) for reasoning about specific entities by fanning-out to their neighbors and associated concepts.
+- [_DRIFT Search_](query/drift_search.md) for reasoning about specific entities by fanning-out to their neighbors and associated concepts, but with the added context of community information.
+- _Basic Search_ for those times when your query is best answered by baseline RAG (standard top _k_ vector search).
+
+### Prompt Tuning
+
+Using _GraphRAG_ with your data out of the box may not yield the best possible results.
+We strongly recommend to fine-tune your prompts following the [Prompt Tuning Guide](prompt_tuning/overview.md) in our documentation.
+
+
+## Versioning
+
+Please see the [breaking changes](https://github.com/microsoft/graphrag/blob/main/breaking-changes.md) document for notes on our approach to versioning the project.
+
+*Always run `graphrag init --root [path] --force` between minor version bumps to ensure you have the latest config format. Run the provided migration notebook between major version bumps if you want to avoid re-indexing prior datasets. Note that this will overwrite your configuration and prompts, so backup if necessary.*

docs/index/architecture.md

@@ -0,0 +1,53 @@
+# Indexing Architecture 
+
+## Key Concepts
+
+### Knowledge Model
+
+In order to support the GraphRAG system, the outputs of the indexing engine (in the Default Configuration Mode) are aligned to a knowledge model we call the _GraphRAG Knowledge Model_.
+This model is designed to be an abstraction over the underlying data storage technology, and to provide a common interface for the GraphRAG system to interact with.
+
+### Workflows
+
+Below is the core GraphRAG indexing pipeline. Individual workflows are described in detail in the [dataflow](./default_dataflow.md) page.
+
+```mermaid
+---
+title: Basic GraphRAG
+---
+stateDiagram-v2
+    [*] --> LoadDocuments
+    LoadDocuments --> ChunkDocuments
+    ChunkDocuments --> ExtractGraph
+    ChunkDocuments --> ExtractClaims
+    ChunkDocuments --> EmbedChunks
+    ExtractGraph --> DetectCommunities
+    ExtractGraph --> EmbedEntities
+    DetectCommunities --> GenerateReports
+    GenerateReports --> EmbedReports
+```
+
+### LLM Caching
+
+The GraphRAG library was designed with LLM interactions in mind, and a common setback when working with LLM APIs is various errors due to network latency, throttling, etc..
+Because of these potential error cases, we've added a cache layer around LLM interactions.
+When completion requests are made using the same input set (prompt and tuning parameters), we return a cached result if one exists.
+This allows our indexer to be more resilient to network issues, to act idempotently, and to provide a more efficient end-user experience.
+
+### Providers & Factories
+
+Several subsystems within GraphRAG use a factory pattern to register and retrieve provider implementations. This allows deep customization to support your own implementations of models, storage, and so on that we haven't built into the core library.
+
+The following subsystems use a factory pattern that allows you to register your own implementations:
+
+- [language model](https://github.com/microsoft/graphrag/blob/main/packages/graphrag-llm/graphrag_llm/completion/completion_factory.py) - implement your own `chat` and `embed` methods to use a model provider of choice beyond the built-in LiteLLM wrapper
+- [input reader](https://github.com/microsoft/graphrag/blob/main/packages/graphrag-input/graphrag_input/input_reader.py) - implement your own input document reader to support file types other than text, CSV, and JSON
+- [cache](https://github.com/microsoft/graphrag/blob/main/packages/graphrag-cache/graphrag_cache/cache_factory.py) - create your own cache storage location in addition to the file, blob, and CosmosDB ones we provide
+- [logger](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/logger/factory.py) - create your own log writing location in addition to the built-in file and blob storage
+- [storage](https://github.com/microsoft/graphrag/blob/main/packages/graphrag-storage/graphrag_storage/tables/table_provider_factory.py) - create your own storage provider (database, etc.) beyond the file, blob, and CosmosDB ones built in
+- [vector store](https://github.com/microsoft/graphrag/blob/main/packages/graphrag-vectors/graphrag_vectors/vector_store_factory.py) - implement your own vector store other than the built-in lancedb, Azure AI Search, and CosmosDB ones built in
+- [pipeline + workflows](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/index/workflows/factory.py) - implement your own workflow steps with a custom `run_workflow` function, or register an entire pipeline (list of named workflows)
+
+The links for each of these subsystems point to the source code of the factory, which includes registration of the default built-in implementations. In addition, we have a detailed discussion of [language models](../config/models.md), which includes and example of a custom provider, and a [sample notebook](../examples_notebooks/custom_vector_store.ipynb) that demonstrates a custom vector store.
+
+All of these factories allow you to register an impl using any string name you would like, even overriding built-in ones directly.

docs/index/byog.md

@@ -0,0 +1,68 @@
+# Bring Your Own Graph
+
+Several users have asked if they can bring their own existing graph and have it summarized for query with GraphRAG. There are many possible ways to do this, but here we'll describe a simple method that aligns with the existing GraphRAG workflows quite easily.
+
+To cover the basic use cases for GraphRAG query, you should have two or three tables derived from your data:
+
+- entities.parquet - this is the list of entities found in the dataset, which are the nodes of the graph.
+- relationships.parquet - this is the list of relationships found in the dataset, which are the edges of the graph.
+- text_units.parquet - this is the source text chunks the graph was extracted from. This is optional depending on the query method you intend to use (described later).
+
+The approach described here will be to run a custom GraphRAG workflow pipeline that assumes the text chunking, entity extraction, and relationship extraction has already occurred.
+
+## Tables
+
+### Entities
+
+See the full entities [table schema](./outputs.md#entities). For graph summarization purposes, you only need id, title, description, and the list of text_unit_ids.
+
+### Relationships
+
+See the full relationships [table schema](./outputs.md#relationships). For graph summarization purposes, you only need id, source, target, description, weight, and the list of text_unit_ids.
+
+> Note: the `weight` field is important because it is used to properly compute Leiden communities!
+
+## Workflow Configuration
+
+GraphRAG includes the ability to specify *only* the specific workflow steps that you need. For basic graph summarization and query, you need the following config in your settings.yaml:
+
+```yaml
+workflows: [create_communities, create_community_reports]
+```
+
+This will result in only the minimal workflows required for GraphRAG [Global Search](../query/global_search.md).
+
+## Optional Additional Config
+
+If you would like to run [Local](../query/local_search.md), [DRIFT](../query/drift_search.md), or [Basic](../query/overview.md#basic-search) Search, you will need to include text_units and some embeddings.
+
+### Text Units
+
+See the full text_units [table schema](./outputs.md#text_units). Text units are chunks of your documents that are sized to ensure they fit into the context window of your model. Some search methods use these, so you may want to include them if you have them.
+
+### Expanded Config
+
+To perform the other search types above, you need some of the content to be embedded. Simply add the embeddings workflow:
+
+```yaml
+workflows: [create_communities, create_community_reports, generate_text_embeddings]
+```
+
+### FastGraphRAG
+
+[FastGraphRAG](./methods.md#fastgraphrag) uses text_units for the community reports instead of the entity and relationship descriptions. If your graph is sourced in such a way that it does not have descriptions, this might be a useful alternative. In this case, you would update your workflows list to include the text variant of the community reports workflow:
+
+```yaml
+workflows: [create_communities, create_community_reports_text, generate_text_embeddings]
+```
+
+This method requires that your entities and relationships tables have valid links to a list of text_unit_ids. Also note that `generate_text_embeddings` is still only required if you are doing searches other than Global Search.
+
+
+## Setup
+
+Putting it all together:
+
+- `output`: Create an output folder and put your entities and relationships (and optionally text_units) parquet files in it.
+- Update your config as noted above to only run the workflows subset you need.
+- Run `graphrag index --root <your_project_root>`

docs/index/default_dataflow.md

@@ -0,0 +1,182 @@
+# Indexing Dataflow
+
+## The GraphRAG Knowledge Model
+
+The knowledge model is a specification for data outputs that conform to our data-model definition. You can find these definitions in the python/graphrag/graphrag/model folder within the GraphRAG repository. The following entity types are provided. The fields here represent the fields that are text-embedded by default.
+
+- `Document` - An input document into the system. These either represent individual rows in a CSV or individual .txt files.
+- `TextUnit` - A chunk of text to analyze. The size of these chunks, their overlap, and whether they adhere to any data boundaries may be configured below.
+- `Entity` - An entity extracted from a TextUnit. These represent people, places, events, or some other entity-model that you provide.
+- `Relationship` - A relationship between two entities.
+- `Covariate` - Extracted claim information, which contains statements about entities which may be time-bound.
+- `Community` - Once the graph of entities and relationships is built, we perform hierarchical community detection on them to create a clustering structure.
+- `Community Report` - The contents of each community are summarized into a generated report, useful for human reading and downstream search.
+
+## The Default Configuration Workflow
+
+Let's take a look at how the default-configuration workflow transforms text documents into the _GraphRAG Knowledge Model_. This page gives a general overview of the major steps in this process. To fully configure this workflow, check out the [configuration](../config/overview.md) documentation.
+
+```mermaid
+---
+title: Dataflow Overview
+---
+flowchart TB
+    subgraph phase1[Phase 1: Compose TextUnits]
+    documents[Documents] --> chunk[Chunk]
+    chunk --> textUnits[Text Units]
+    end
+    subgraph phase2[Phase 2: Document Processing]
+    documents --> link_to_text_units[Link to TextUnits]
+    textUnits --> link_to_text_units
+    link_to_text_units --> document_outputs[Documents Table]
+    end
+    subgraph phase3[Phase 3 Graph Extraction]
+    textUnits --> graph_extract[Entity & Relationship Extraction]
+    graph_extract --> graph_summarize[Entity & Relationship Summarization]
+    graph_summarize --> claim_extraction[Claim Extraction]
+    claim_extraction --> graph_outputs[Graph Tables]
+    end
+    subgraph phase4[Phase 4: Graph Augmentation]
+    graph_outputs --> community_detect[Community Detection]
+    community_detect --> community_outputs[Communities Table]
+    end
+    subgraph phase5[Phase 5: Community Summarization]
+    community_outputs --> summarized_communities[Community Summarization]
+    summarized_communities --> community_report_outputs[Community Reports Table]
+    end
+    subgraph phase6[Phase 6: Text Embeddings]
+    textUnits --> text_embed[Text Embedding]
+    graph_outputs --> description_embed[Description Embedding]
+    community_report_outputs --> content_embed[Content Embedding]
+    end
+```
+
+## Phase 1: Compose TextUnits
+
+The first phase of the default-configuration workflow is to transform input documents into _TextUnits_. A _TextUnit_ is a chunk of text that is used for our graph extraction techniques. They are also used as source-references by extracted knowledge items in order to empower breadcrumbs and provenance by concepts back to their original source text.
+
+The chunk size (counted in tokens), is user-configurable. By default this is set to 1200 tokens. Larger chunks result in lower-fidelity output and less meaningful reference texts; however, using larger chunks can result in much faster processing time.
+
+```mermaid
+---
+title: Documents into Text Chunks
+---
+flowchart LR
+    doc1[Document 1] --> tu1[TextUnit 1]
+    doc1 --> tu2[TextUnit 2]
+    doc2[Document 2] --> tu3[TextUnit 3]
+    doc2 --> tu4[TextUnit 4]
+
+```
+
+## Phase 2: Document Processing
+
+In this phase of the workflow, we create the _Documents_ table for the knowledge model. Final documents are not used directly in GraphRAG, but this step links them to their constituent text units for provenance in your own applications.
+
+```mermaid
+---
+title: Document Processing
+---
+flowchart LR
+    aug[Augment] --> dp[Link to TextUnits] --> dg[Documents Table]
+```
+
+### Link to TextUnits
+
+In this step, we link each document to the text-units that were created in the first phase. This allows us to understand which documents are related to which text-units and vice-versa.
+
+### Documents Table
+
+At this point, we can export the **Documents** table into the knowledge Model.
+
+## Phase 3: Graph Extraction
+
+In this phase, we analyze each text unit and extract our graph primitives: _Entities_, _Relationships_, and _Claims_.
+Entities and Relationships are extracted at once in our _extract_graph_ workflow, and claims are extracted in our _extract_claims_ workflow. Results are then combined and passed into following phases of the pipeline.
+
+```mermaid
+---
+title: Graph Extraction
+---
+flowchart LR
+    tu[TextUnit] --> ge[Graph Extraction] --> gs[Graph Summarization]
+    tu --> ce[Claim Extraction]
+```
+
+> Note: if you are using the [FastGraphRAG](https://microsoft.github.io/graphrag/index/methods/#fastgraphrag) option, entity and relationship extraction will be performed using NLP to conserve LLM resources, and claim extraction will always be skipped.
+
+### Entity & Relationship Extraction
+
+In this first step of graph extraction, we process each text-unit to extract entities and relationships out of the raw text using the LLM. The output of this step is a subgraph-per-TextUnit containing a list of **entities** with a _title_, _type_, and _description_, and a list of **relationships** with a _source_, _target_, and _description_.
+
+These subgraphs are merged together - any entities with the same _title_ and _type_ are merged by creating an array of their descriptions. Similarly, any relationships with the same _source_ and _target_ are merged by creating an array of their descriptions.
+
+### Entity & Relationship Summarization
+
+Now that we have a graph of entities and relationships, each with a list of descriptions, we can summarize these lists into a single description per entity and relationship. This is done by asking the LLM for a short summary that captures all of the distinct information from each description. This allows all of our entities and relationships to have a single concise description.
+
+### Claim Extraction (optional)
+
+Finally, as an independent workflow, we extract claims from the source TextUnits. These claims represent positive factual statements with an evaluated status and time-bounds. These get exported as a primary artifact called **Covariates**.
+
+Note: claim extraction is _optional_ and turned off by default. This is because claim extraction generally requires prompt tuning to be useful.
+
+## Phase 4: Graph Augmentation
+
+Now that we have a usable graph of entities and relationships, we want to understand their community structure. These give us explicit ways of understanding the organization of our graph.
+
+```mermaid
+---
+title: Graph Augmentation
+---
+flowchart LR
+    cd[Leiden Hierarchical Community Detection] --> ag[Graph Tables]
+```
+
+### Community Detection
+
+In this step, we generate a hierarchy of entity communities using the Hierarchical Leiden Algorithm. This method will apply a recursive community-clustering to our graph until we reach a community-size threshold. This will allow us to understand the community structure of our graph and provide a way to navigate and summarize the graph at different levels of granularity.
+
+### Graph Tables
+
+Once our graph augmentation steps are complete, the final **Entities**, **Relationships**, and **Communities** tables are exported.
+
+## Phase 5: Community Summarization
+
+```mermaid
+---
+title: Community Summarization
+---
+flowchart LR
+    sc[Generate Community Reports] --> ss[Summarize Community Reports] --> co[Community Reports Table]
+```
+
+At this point, we have a functional graph of entities and relationships and a hierarchy of communities for the entities.
+
+Now we want to build on the communities data and generate reports for each community. This gives us a high-level understanding of the graph at several points of graph granularity. For example, if community A is the top-level community, we'll get a report about the entire graph. If the community is lower-level, we'll get a report about a local cluster.
+
+### Generate Community Reports
+
+In this step, we generate a summary of each community using the LLM. This will allow us to understand the distinct information contained within each community and provide a scoped understanding of the graph, from either a high-level or a low-level perspective. These reports contain an executive overview and reference the key entities, relationships, and claims within the community sub-structure.
+
+### Summarize Community Reports
+
+In this step, each _community report_ is then summarized via the LLM for shorthand use.
+
+### Community Reports Table
+
+At this point, some bookkeeping work is performed and we export the **Community Reports** tables.
+
+## Phase 6: Text Embedding
+
+For all artifacts that require downstream vector search, we generate text embeddings as a final step. These embeddings are written directly to a configured vector store. By default we embed entity descriptions, text unit text, and community report text.
+
+```mermaid
+---
+title: Text Embedding Workflows
+---
+flowchart LR
+    textUnits[Text Units] --> text_embed[Text Embedding]
+    graph_outputs[Graph Tables] --> description_embed[Description Embedding]
+    community_report_outputs[Community Reports] --> content_embed[Content Embedding]
+```

docs/index/inputs.md

@@ -0,0 +1,227 @@
+# Inputs
+
+GraphRAG supports several input formats to simplify ingesting your data. The mechanics and features available for input files and text chunking are discussed here.
+
+## Input Loading and Schema
+
+All input formats are loaded within GraphRAG and passed to the indexing pipeline as a `documents` DataFrame. This DataFrame has a row for each document using a shared column schema:
+
+| name          | type | description |
+| ------------- | ---- | ----------- |
+| id            | str  | ID of the document. This is generated using a hash of the text content to ensure stability across runs. |
+| text          | str  | The full text of the document. |
+| title         | str  | Name of the document. Some formats allow this to be configured. |
+| creation_date | str  | The creation date of the document, represented as an ISO8601 string. This is harvested from the source file system. |
+| metadata      | dict | Optional additional document metadata. More details below. |
+
+Also see the [outputs](outputs.md) documentation for the final documents table schema saved to parquet after pipeline completion.
+
+## Bring-your-own DataFrame
+
+GraphRAG's [indexing API method](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/api/index.py) allows you to pass in your own pandas DataFrame and bypass all of the input loading/parsing described in the next section. This is convenient if you have content in a format or storage location we don't support out-of-the-box. _You must ensure that your input DataFrame conforms to the schema described above._ All of the chunking behavior described later will proceed exactly the same.
+
+## Custom File Handling
+
+We use an injectable InputReader provider class. This means you can implement any input file handling you want in a class that extends InputReader and register it with the InputReaderFactory. See the [architecture page](https://microsoft.github.io/graphrag/index/architecture/) for more info on our standard provider pattern.
+
+## Formats
+
+We support three file formats out-of-the-box. This covers the overwhelming majority of use cases we have encountered. If you have a different format, we recommend either implementing your own InputReader or writing a script to convert to one of these, which are widely used and supported by many tools and libraries.
+
+### Plain Text
+
+Plain text files (typically ending in .txt file extension). With plain text files we import the entire file contents as the `text` field, and the `title` is always the filename.
+
+### Comma-delimited
+
+CSV files (typically ending in a .csv extension). These are loaded using pandas' [`read_csv` method](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html) with default options. Each row in a CSV file is treated as a single document. If you have multiple CSV files in your input folder, they will be concatenated into a single resulting `documents` DataFrame.
+
+With the CSV format you can configure the `text_column`, and `title_column` if your data has structured content you would prefer to use. If you do not configure these within the `input` block of your settings.yaml, the title will be the filename as described in the schema above. The `text_column` is assumed to be "text" in your file if not configured specifically. We will also look for and use an "id" column if present, otherwise the ID will be generated as described above.
+
+### JSON
+
+JSON files (typically ending in a .json extension) contain [structured objects](https://www.json.org/). These are loaded using python's [`json.loads` method](https://docs.python.org/3/library/json.html), so your files must be properly compliant. JSON files may contain a single object in the file *or* the file may contain an array of objects at the root. We will check for and handle either of these cases. As with CSV, multiple files will be concatenated into a final table, and the `text_column` and `title_column` config options will be applied to the properties of each loaded object. Note that the specialized jsonl format produced by some libraries (one full JSON object on each line, not in an array) is not currently supported.
+
+## Metadata
+
+With the structured file formats (CSV and JSON) you can configure any number of columns to be added to a persisted `metadata` field in the DataFrame. This is configured by supplying a list of column names to collect. If this is configured, the output `metadata` column will have a dict containing a key for each column, and the value of the column for that document. This metadata can optionally be used later in the GraphRAG pipeline.
+
+### Example
+
+software.csv
+
+```csv
+text,title,tag
+My first program,Hello World,tutorial
+An early space shooter game,Space Invaders,arcade
+```
+
+settings.yaml
+
+```yaml
+input:
+    metadata: [title,tag]
+```
+
+Documents DataFrame
+
+| id                    | title          | text                        | creation_date                 | metadata                                       |
+| --------------------- | -------------- | --------------------------- | ----------------------------- | ---------------------------------------------- |
+| (generated from text) | Hello World    | My first program            | (create date of software.csv) | { "title": "Hello World", "tag": "tutorial" }  |
+| (generated from text) | Space Invaders | An early space shooter game | (create date of software.csv) | { "title": "Space Invaders", "tag": "arcade" } |
+
+## Chunking and Metadata
+
+As described on the [default dataflow](default_dataflow.md#phase-1-compose-textunits) page, documents are *chunked* into smaller "text units" for processing. This is done because document content size often exceeds the available context window for a given language model. There are a handful of settings you can adjust for this chunking, the most relevant being the `chunk_size` and `overlap`. We also support a metadata processing scheme that can improve indexing results for some use cases. We will describe this feature in detail here.
+
+Imagine the following scenario: you are indexing a collection of news articles. Each article text starts with a headline and author, and then proceeds with the content. When documents are chunked, they are split evenly according to your configured chunk size. In other words, the first *n* tokens are read into a text unit, and then the next *n*, until the end of the content. This means that front matter at the beginning of the document (such as the headline and author in this example) *is not copied to each chunk*. It only exists in the first chunk. When we later retrieve those chunks for summarization, they may therefore be missing shared information about the source document that should always be provided to the model. We have configuration options to copy repeated content into each text unit to address this issue.
+
+### Input Config
+
+As described above, when documents are imported you can specify a list of `metadata` columns to include with each row. This must be configured for the per-chunk copying to work.
+
+### Chunking Config
+
+Next, the `chunks` block needs to instruct the chunker how to handle this metadata when creating text units. By default, it is ignored. We have the following setting to include it:
+
+- `prepend_metadata`. This instructs the importer to copy the contents of the `metadata` column for each row into the start of every single text chunk. This metadata is copied as key: value pairs on new lines.
+
+### Examples
+
+The following are several examples to help illustrate how chunking config and metadata prepending works for each file format. Note that we are using word count here as "tokens" for the illustration, but language model tokens are [not equivalent to words](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them).
+
+#### Text files
+
+This example uses two individual news article text files.
+
+--
+
+**File:** US to lift most federal COVID-19 vaccine mandates.txt
+
+**Content:**
+
+WASHINGTON (AP) The Biden administration will end most of the last remaining federal COVID-19 vaccine requirements next week when the national public health emergency for the coronavirus ends, the White House said Monday. Vaccine requirements for federal workers and federal contractors, as well as foreign air travelers to the U.S., will end May 11. The government is also beginning the process of lifting shot requirements for Head Start educators, healthcare workers, and noncitizens at U.S. land borders. The requirements are among the last vestiges of some of the more coercive measures taken by the federal government to promote vaccination as the deadly virus raged, and their end marks the latest display of how President Joe Biden's administration is moving to treat COVID-19 as a routine, endemic illness. "While I believe that these vaccine mandates had a tremendous beneficial impact, we are now at a point where we think that it makes a lot of sense to pull these requirements down," White House COVID-19 coordinator Dr. Ashish Jha told The Associated Press on Monday.
+
+--
+
+**File:** NY lawmakers begin debating budget 1 month after due date.txt
+
+**Content:**
+
+ALBANY, N.Y. (AP) New York lawmakers began voting Monday on a $229 billion state budget due a month ago that would raise the minimum wage, crack down on illicit pot shops and ban gas stoves and furnaces in new buildings. Negotiations among Gov. Kathy Hochul and her fellow Democrats in control of the Legislature dragged on past the April 1 budget deadline, largely because of disagreements over changes to the bail law and other policy proposals included in the spending plan. Floor debates on some budget bills began Monday. State Senate Majority Leader Andrea Stewart-Cousins said she expected voting to be wrapped up Tuesday for a budget she said contains "significant wins" for New Yorkers. "I would have liked to have done this sooner. I think we would all agree to that," Cousins told reporters before voting began. "This has been a very policy-laden budget and a lot of the policies had to parsed through." Hochul was able to push through a change to the bail law that will eliminate the standard that requires judges to prescribe the "least restrictive" means to ensure defendants return to court. Hochul said judges needed the extra discretion. Some liberal lawmakers argued that it would undercut the sweeping bail reforms approved in 2019 and result in more people with low incomes and people of color in pretrial detention. Here are some other policy provisions that will be included in the budget, according to state officials. The minimum wage would be raised to $17 in New York City and some of its suburbs and $16 in the rest of the state by 2026. That's up from $15 in the city and $14.20 upstate.
+
+--
+
+settings.yaml
+
+```yaml
+input:
+    type: text
+    metadata: [title]
+
+chunks:
+    size: 100
+    overlap: 0
+    prepend_metadata: true
+```
+
+Documents DataFrame
+
+| id                    | title                                                         | text                        | creation_date                     | metadata                                                                     |
+| --------------------- | ------------------------------------------------------------- | --------------------------- | --------------------------------- | ---------------------------------------------------------------------------- |
+| (generated from text) | US to lift most federal COVID-19 vaccine mandates.txt         | (full content of text file) | (create date of article txt file) | { "title": "US to lift most federal COVID-19 vaccine mandates.txt" }         |
+| (generated from text) | NY lawmakers begin debating budget 1 month after due date.txt | (full content of text file) | (create date of article txt file) | { "title": "NY lawmakers begin debating budget 1 month after due date.txt" } |
+
+Raw Text Chunks
+
+| content | length  |
+| ------- | ------: |
+| title: US to lift most federal COVID-19 vaccine mandates.txt<br>WASHINGTON (AP) The Biden administration will end most of the last remaining federal COVID-19 vaccine requirements next week when the national public health emergency for the coronavirus ends, the White House said Monday. Vaccine requirements for federal workers and federal contractors, as well as foreign air travelers to the U.S., will end May 11. The government is also beginning the process of lifting shot requirements for Head Start educators, healthcare workers, and noncitizens at U.S. land borders. The requirements are among the last vestiges of some of the more coercive measures taken by the federal government to promote vaccination as | 109 |
+| title: US to lift most federal COVID-19 vaccine mandates.txt<br>the deadly virus raged, and their end marks the latest display of how President Joe Biden's administration is moving to treat COVID-19 as a routine, endemic illness. "While I believe that these vaccine mandates had a tremendous beneficial impact, we are now at a point where we think that it makes a lot of sense to pull these requirements down," White House COVID-19 coordinator Dr. Ashish Jha told The Associated Press on Monday. | 82 |
+| title: NY lawmakers begin debating budget 1 month after due date.txt<br>ALBANY, N.Y. (AP) New York lawmakers began voting Monday on a $229 billion state budget due a month ago that would raise the minimum wage, crack down on illicit pot shops and ban gas stoves and furnaces in new buildings. Negotiations among Gov. Kathy Hochul and her fellow Democrats in control of the Legislature dragged on past the April 1 budget deadline, largely because of disagreements over changes to the bail law and other policy proposals included in the spending plan. Floor debates on some budget bills began Monday. State Senate Majority Leader Andrea Stewart-Cousins said she expected voting to | 111 |
+| title: NY lawmakers begin debating budget 1 month after due date.txt<br>be wrapped up Tuesday for a budget she said contains "significant wins" for New Yorkers. "I would have liked to have done this sooner. I think we would all agree to that," Cousins told reporters before voting began. "This has been a very policy-laden budget and a lot of the policies had to parsed through." Hochul was able to push through a change to the bail law that will eliminate the standard that requires judges to prescribe the "least restrictive" means to ensure defendants return to court. Hochul said judges needed the extra discretion. Some liberal lawmakers argued that it | 111 |
+| title: NY lawmakers begin debating budget 1 month after due date.txt<br>would undercut the sweeping bail reforms approved in 2019 and result in more people with low incomes and people of color in pretrial detention. Here are some other policy provisions that will be included in the budget, according to state officials. The minimum wage would be raised to $17 in New York City and some of its suburbs and $16 in the rest of the state by 2026. That's up from $15 in the city and $14.20 upstate. | 89 |
+
+In this example we can see that the two input documents were parsed into five output text chunks. The title (filename) of each document is prepended but not included in the computed chunk size. Also note that the final text chunk for each document is usually smaller than the chunk size because it contains the last tokens.
+
+#### CSV files
+
+This example uses a single CSV file with the same two articles as rows (note that the text content is not properly escaped for actual CSV use).
+
+--
+
+**File:** articles.csv
+
+**Content**
+
+headline,article
+
+US to lift most federal COVID-19 vaccine mandates,WASHINGTON (AP) The Biden administration will end most of the last remaining federal COVID-19 vaccine requirements next week when the national public health emergency for the coronavirus ends, the White House said Monday. Vaccine requirements for federal workers and federal contractors, as well as foreign air travelers to the U.S., will end May 11. The government is also beginning the process of lifting shot requirements for Head Start educators, healthcare workers, and noncitizens at U.S. land borders. The requirements are among the last vestiges of some of the more coercive measures taken by the federal government to promote vaccination as the deadly virus raged, and their end marks the latest display of how President Joe Biden's administration is moving to treat COVID-19 as a routine, endemic illness. "While I believe that these vaccine mandates had a tremendous beneficial impact, we are now at a point where we think that it makes a lot of sense to pull these requirements down," White House COVID-19 coordinator Dr. Ashish Jha told The Associated Press on Monday.
+
+NY lawmakers begin debating budget 1 month after due date,ALBANY, N.Y. (AP) New York lawmakers began voting Monday on a $229 billion state budget due a month ago that would raise the minimum wage, crack down on illicit pot shops and ban gas stoves and furnaces in new buildings. Negotiations among Gov. Kathy Hochul and her fellow Democrats in control of the Legislature dragged on past the April 1 budget deadline, largely because of disagreements over changes to the bail law and other policy proposals included in the spending plan. Floor debates on some budget bills began Monday. State Senate Majority Leader Andrea Stewart-Cousins said she expected voting to be wrapped up Tuesday for a budget she said contains "significant wins" for New Yorkers. "I would have liked to have done this sooner. I think we would all agree to that," Cousins told reporters before voting began. "This has been a very policy-laden budget and a lot of the policies had to parsed through." Hochul was able to push through a change to the bail law that will eliminate the standard that requires judges to prescribe the "least restrictive" means to ensure defendants return to court. Hochul said judges needed the extra discretion. Some liberal lawmakers argued that it would undercut the sweeping bail reforms approved in 2019 and result in more people with low incomes and people of color in pretrial detention. Here are some other policy provisions that will be included in the budget, according to state officials. The minimum wage would be raised to $17 in New York City and some of its suburbs and $16 in the rest of the state by 2026. That's up from $15 in the city and $14.20 upstate.
+
+#### JSON files
+
+This final example uses a JSON file for each of the same two articles. In this example we'll set the object fields to read, but we will not add metadata to the text chunks.
+
+--
+
+**File:** article1.json
+
+**Content**
+
+```json
+{
+    "headline": "US to lift most federal COVID-19 vaccine mandates",
+    "content": "WASHINGTON (AP) The Biden administration will end most of the last remaining federal COVID-19 vaccine requirements next week when the national public health emergency for the coronavirus ends, the White House said Monday. Vaccine requirements for federal workers and federal contractors, as well as foreign air travelers to the U.S., will end May 11. The government is also beginning the process of lifting shot requirements for Head Start educators, healthcare workers, and noncitizens at U.S. land borders. The requirements are among the last vestiges of some of the more coercive measures taken by the federal government to promote vaccination as the deadly virus raged, and their end marks the latest display of how President Joe Biden's administration is moving to treat COVID-19 as a routine, endemic illness. "While I believe that these vaccine mandates had a tremendous beneficial impact, we are now at a point where we think that it makes a lot of sense to pull these requirements down," White House COVID-19 coordinator Dr. Ashish Jha told The Associated Press on Monday."
+}
+```
+
+**File:** article2.json
+
+**Content**
+
+```json
+{
+    "headline": "NY lawmakers begin debating budget 1 month after due date",
+    "content": "ALBANY, N.Y. (AP) New York lawmakers began voting Monday on a $229 billion state budget due a month ago that would raise the minimum wage, crack down on illicit pot shops and ban gas stoves and furnaces in new buildings. Negotiations among Gov. Kathy Hochul and her fellow Democrats in control of the Legislature dragged on past the April 1 budget deadline, largely because of disagreements over changes to the bail law and other policy proposals included in the spending plan. Floor debates on some budget bills began Monday. State Senate Majority Leader Andrea Stewart-Cousins said she expected voting to be wrapped up Tuesday for a budget she said contains "significant wins" for New Yorkers. "I would have liked to have done this sooner. I think we would all agree to that," Cousins told reporters before voting began. "This has been a very policy-laden budget and a lot of the policies had to parsed through." Hochul was able to push through a change to the bail law that will eliminate the standard that requires judges to prescribe the "least restrictive" means to ensure defendants return to court. Hochul said judges needed the extra discretion. Some liberal lawmakers argued that it would undercut the sweeping bail reforms approved in 2019 and result in more people with low incomes and people of color in pretrial detention. Here are some other policy provisions that will be included in the budget, according to state officials. The minimum wage would be raised to $17 in New York City and some of its suburbs and $16 in the rest of the state by 2026. That's up from $15 in the city and $14.20 upstate."
+}
+```
+
+--
+
+settings.yaml
+
+```yaml
+input:
+    type: json
+    title_column: headline
+    text_column: content
+
+chunks:
+    size: 100
+    overlap: 10
+```
+
+Documents DataFrame
+
+| id                    | title                                                     | text                     | creation_date                  | metadata |
+| --------------------- | --------------------------------------------------------- | ------------------------ | ------------------------------ | -------- |
+| (generated from text) | US to lift most federal COVID-19 vaccine mandates         | (article column content) | (create date of article1.json) | { }      |
+| (generated from text) | NY lawmakers begin debating budget 1 month after due date | (article column content) | (create date of article2.json) | { }      |
+
+Raw Text Chunks
+
+| content | length  |
+| ------- | ------: |
+| WASHINGTON (AP) The Biden administration will end most of the last remaining federal COVID-19 vaccine requirements next week when the national public health emergency for the coronavirus ends, the White House said Monday. Vaccine requirements for federal workers and federal contractors, as well as foreign air travelers to the U.S., will end May 11. The government is also beginning the process of lifting shot requirements for Head Start educators, healthcare workers, and noncitizens at U.S. land borders. The requirements are among the last vestiges of some of the more coercive measures taken by the federal government to promote vaccination as | 100 |
+| measures taken by the federal government to promote vaccination as  the deadly virus raged, and their end marks the latest display of how President Joe Biden's administration is moving to treat COVID-19 as a routine, endemic illness. "While I believe that these vaccine mandates had a tremendous beneficial impact, we are now at a point where we think that it makes a lot of sense to pull these requirements down," White House COVID-19 coordinator Dr. Ashish Jha told The Associated Press on Monday. | 83 |
+| ALBANY, N.Y. (AP) New York lawmakers began voting Monday on a $229 billion state budget due a month ago that would raise the minimum wage, crack down on illicit pot shops and ban gas stoves and furnaces in new buildings. Negotiations among Gov. Kathy Hochul and her fellow Democrats in control of the Legislature dragged on past the April 1 budget deadline, largely because of disagreements over changes to the bail law and other policy proposals included in the spending plan. Floor debates on some budget bills began Monday. State Senate Majority Leader Andrea Stewart-Cousins said she expected voting to | 100 |
+| Senate Majority Leader Andrea Stewart-Cousins said she expected voting to be wrapped up Tuesday for a budget she said contains "significant wins" for New Yorkers. "I would have liked to have done this sooner. I think we would all agree to that," Cousins told reporters before voting began. "This has been a very policy-laden budget and a lot of the policies had to parsed through." Hochul was able to push through a change to the bail law that will eliminate the standard that requires judges to prescribe the "least restrictive" means to ensure defendants return to court. Hochul said judges | 100 |
+| means to ensure defendants return to court. Hochul said judges needed the extra discretion. Some liberal lawmakers argued that it would undercut the sweeping bail reforms approved in 2019 and result in more people with low incomes and people of color in pretrial detention. Here are some other policy provisions that will be included in the budget, according to state officials. The minimum wage would be raised to $17 in New York City and some of its suburbs and $16 in the rest of the state by 2026. That's up from $15 in the city and $14.20 upstate. | 98 |
+
+
+In this example the two input documents were parsed into five output text chunks. There is no metadata prepended, so each chunk matches the configured chunk size (except the last one for each document). We've also configured some overlap in these text chunks, so the last ten tokens are shared.
+
+
+

docs/index/methods.md

@@ -0,0 +1,44 @@
+# Indexing Methods
+
+GraphRAG is a platform for our research into RAG indexing methods that produce optimal context window content for language models. We have a standard indexing pipeline that uses a language model to extract the graph that our memory model is based upon. We may introduce additional indexing methods from time to time. This page documents those options.
+
+## Standard GraphRAG
+
+This is the method described in the original [blog post](https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/). Standard uses a language model for all reasoning tasks:
+
+- entity extraction: LLM is prompted to extract named entities and provide a description from each text unit.
+- relationship extraction: LLM is prompted to describe the relationship between each pair of entities in each text unit.
+- entity summarization: LLM is prompted to combine the descriptions for every instance of an entity found across the text units into a single summary.
+- relationship summarization: LLM is prompted to combine the descriptions for every instance of a relationship found across the text units into a single summary.
+- claim extraction (optional): LLM is prompted to extract and describe claims from each text unit.
+- community report generation: entity and relationship descriptions (and optionally claims) for each community are collected and used to prompt the LLM to generate a summary report.
+
+`graphrag index --method standard`. This is the default method, so the method param can be omitted on the command line.
+
+## FastGraphRAG
+
+FastGraphRAG is a method that substitutes some of the language model reasoning for traditional natural language processing (NLP) methods. This is a hybrid technique that we developed as a faster and cheaper indexing alternative:
+
+- entity extraction: entities are noun phrases extracted using NLP libraries such as NLTK and spaCy. There is no description; the source text unit is used for this.
+- relationship extraction: relationships are defined as text unit co-occurrence between entity pairs. There is no description.
+- entity summarization: not necessary.
+- relationship summarization: not necessary.
+- claim extraction: unused.
+- community report generation: The direct text unit content containing each entity noun phrase is collected and used to prompt the LLM to generate a summary report.
+
+`graphrag index --method fast`
+
+FastGraphRAG has a handful of NLP [options built in](https://microsoft.github.io/graphrag/config/yaml/#extract_graph_nlp). By default we use NLTK + regular expressions for the noun phrase extraction, which is very fast but primarily suitable for English. We have built in two additional methods using spaCy: semantic parsing and CFG. We use the `en_core_web_md` model by default for spaCy, but note that you can reference any [supported model](https://spacy.io/models/) that you have installed. 
+
+Note that we also generally configure the text chunking to produce much smaller chunks (50-100 tokens). This results in a better co-occurrence graph.
+
+⚠️ Note on SpaCy models:
+
+This package requires SpaCy models to function correctly. If the required model is not installed, the package will automatically download and install it the first time it is used.
+
+You can install it manually by running `python -m spacy download <model_name>`, for example `python -m spacy download en_core_web_md`.
+
+
+## Choosing a Method
+
+Standard GraphRAG provides a rich description of real-world entities and relationships, but is more expensive than FastGraphRAG. We estimate graph extraction to constitute roughly 75% of indexing cost. FastGraphRAG is therefore much cheaper, but the tradeoff is that the extracted graph is less directly relevant for use outside of GraphRAG, and the graph tends to be quite a bit noisier. If high fidelity entities and graph exploration are important to your use case, we recommend staying with traditional GraphRAG. If your use case is primarily aimed at summary questions using global search, FastGraphRAG provides high quality summarization with much lower language model cost.

docs/index/outputs.md

@@ -0,0 +1,108 @@
+# Outputs
+
+The default pipeline produces a series of output tables that align with the [conceptual knowledge model](../index/default_dataflow.md). This page describes the detailed output table schemas. By default we write these tables out as parquet files on disk.
+
+## Shared fields
+All tables have two identifier fields:
+
+| name              | type | description |
+| ----------------- | ---- | ----------- |
+| id                | str  | Generated UUID, assuring global uniqueness |
+| human_readable_id | int  | This is an incremented short ID created per-run. For example, we use this short ID with generated summaries that print citations so they are easy to cross-reference visually. |
+
+## communities
+This is a list of the final communities generated by Leiden. Communities are strictly hierarchical, subdividing into children as the cluster affinity is narrowed.
+
+| name               | type  | description |
+| ------------------ | ----- | ----------- |
+| community          | int   | Leiden-generated cluster ID for the community. Note that these increment with depth, so they are unique through all levels of the community hierarchy. For this table, human_readable_id is a copy of the community ID rather than a plain increment. |
+| parent             | int   | Parent community ID.|
+| children           | int[] | List of child community IDs.|
+| level              | int   | Depth of the community in the hierarchy. |
+| title              | str   | Friendly name of the community. |
+| entity_ids         | str[] | List of entities that are members of the community. |
+| relationship_ids   | str[] | List of relationships that are wholly within the community (source and target are both in the community). |
+| text_unit_ids      | str[] | List of text units represented within the community. |
+| period             | str   | Date of ingest, used for incremental update merges. ISO8601 |
+| size               | int   | Size of the community (entity count), used for incremental update merges. |
+
+## community_reports
+This is the list of summarized reports for each community.
+
+| name                 | type  | description |
+| -------------------- | ----- | ----------- |
+| community            | int   | Short ID of the community this report applies to. |
+| parent               | int   | Parent community ID. |
+| children             | int[] | List of child community IDs.|
+| level                | int   | Level of the community this report applies to. |
+| title                | str   | LM-generated title for the report. |
+| summary              | str   | LM-generated summary of the report. |
+| full_content         | str   | LM-generated full report. |
+| rank                 | float | LM-derived relevance ranking of the report based on member entity salience
+| rating_explanation   | str   | LM-derived explanation of the rank. |
+| findings             | dict  | LM-derived list of the top 5-10 insights from the community. Contains `summary` and `explanation` values. |
+| full_content_json    | json  | Full JSON output as returned by the LM. Most fields are extracted into columns, but this JSON is sent for query summarization so we leave it to allow for prompt tuning to add fields/content by end users. |
+| period               | str   | Date of ingest, used for incremental update merges. ISO8601 |
+| size                 | int   | Size of the community (entity count), used for incremental update merges. |
+
+## covariates
+(Optional) If claim extraction is turned on, this is a list of the extracted covariates. Note that claims are typically oriented around identifying malicious behavior such as fraud, so they are not useful for all datasets.
+
+| name           | type | description |
+| -------------- | ---- | ----------- |
+| covariate_type | str  | This is always "claim" with our default covariates. |
+| type           | str  | Nature of the claim type. |
+| description    | str  | LM-generated description of the behavior. |
+| subject_id     | str  | Name of the source entity (that is performing the claimed behavior). |
+| object_id      | str  | Name of the target entity (that the claimed behavior is performed on). |
+| status         | str  | LM-derived assessment of the correctness of the claim. One of [TRUE, FALSE, SUSPECTED] |
+| start_date     | str  | LM-derived start of the claimed activity. ISO8601 |
+| end_date       | str  | LM-derived end of the claimed activity. ISO8601 |
+| source_text    | str  | Short string of text containing the claimed behavior. |
+| text_unit_id   | str  | ID of the text unit the claim text was extracted from. |
+
+## documents
+List of document content after import.
+
+| name          | type  | description |
+| ------------- | ----- | ----------- |
+| title         | str   | Filename, unless otherwise configured during CSV import. |
+| text          | str   | Full text of the document. |
+| text_unit_ids | str[] | List of text units (chunks) that were parsed from the document. |
+| metadata      | dict  | If specified during CSV import, this is a dict of metadata for the document. |
+
+## entities
+List of all entities found in the data by the LM.
+
+| name          | type  | description |
+| ------------- | ----- | ----------- |
+| title         | str   | Name of the entity. |
+| type          | str   | Type of the entity. By default this will be "organization", "person", "geo", or "event" unless configured differently or auto-tuning is used. |
+| description   | str   | Textual description of the entity. Entities may be found in many text units, so this is an LM-derived summary of all descriptions. |
+| text_unit_ids | str[] | List of the text units containing the entity. |
+| frequency     | int   | Count of text units the entity was found within. |
+| degree        | int   | Node degree (connectedness) in the graph. |
+
+## relationships
+List of all entity-to-entity relationships found in the data by the LM. This is also the _edge list_ for the graph.
+
+| name            | type  | description |
+| --------------- | ----- | ----------- |
+| source          | str   | Name of the source entity. |
+| target          | str   | Name of the target entity. |
+| description     | str   | LM-derived description of the relationship. Also see note for entity descriptions. |
+| weight          | float | Weight of the edge in the graph. This is summed from an LM-derived "strength" measure for each relationship instance. |
+| combined_degree | int   | Sum of source and target node degrees. |
+| text_unit_ids   | str[] | List of text units the relationship was found within. |
+
+## text_units
+List of all text chunks parsed from the input documents.
+
+| name              | type  | description |
+| ----------------- | ----- | ----------- |
+| text              | str   | Raw full text of the chunk. |
+| n_tokens          | int   | Number of tokens in the chunk. This should normally match the `chunk_size` config parameter, except for the last chunk which is often shorter. |
+| document_id       | str   | ID of the document the chunk came from. |
+| entity_ids        | str[] | List of entities found in the text unit. |
+| relationships_ids | str[] | List of relationships found in the text unit. |
+| covariate_ids     | str[] | Optional list of covariates found in the text unit. |

docs/index/overview.md

@@ -0,0 +1,39 @@
+# GraphRAG Indexing 🤖
+
+The GraphRAG indexing package is a data pipeline and transformation suite that is designed to extract meaningful, structured data from unstructured text using LLMs.
+
+Indexing Pipelines are configurable. They are composed of workflows, standard and custom steps, prompt templates, and input/output adapters. Our standard pipeline is designed to:
+
+- extract entities, relationships and claims from raw text
+- perform community detection in entities
+- generate community summaries and reports at multiple levels of granularity
+- embed text into a vector space
+
+The outputs of the pipeline are stored as Parquet tables by default, and embeddings are written to your configured vector store.
+
+## Getting Started
+
+### Requirements
+
+See the [requirements](../developing.md#requirements) section in [Get Started](../get_started.md) for details on setting up a development environment.
+
+To configure GraphRAG, see the [configuration](../config/overview.md) documentation.
+After you have a config file you can run the pipeline using the CLI or the Python API.
+
+## Usage
+
+### CLI
+
+```bash
+uv run poe index --root <data_root> # default config mode
+```
+
+### Python API
+
+Please see the indexing API [python file](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/api/index.py) for the recommended method to call directly from Python code.
+
+## Further Reading
+
+- To start developing within the _GraphRAG_ project, see [getting started](../developing.md)
+- To understand the underlying concepts and execution model of the indexing library, see [the architecture documentation](../index/architecture.md)
+- To read more about configuring the indexing engine, see [the configuration documentation](../config/overview.md)

docs/prompt_tuning/auto_prompt_tuning.md

@@ -0,0 +1,91 @@
+# Auto Prompt Tuning ⚙️
+
+GraphRAG provides the ability to create domain-adapted prompts for the generation of the knowledge graph. This step is optional, though it is highly encouraged to run it as it will yield better results when executing an Index Run.
+
+These are generated by loading the inputs, splitting them into chunks (text units) and then running a series of LLM invocations and template substitutions to generate the final prompts. We suggest using the default values provided by the script, but in this page you'll find the details of each in case you want to further explore and tweak the prompt tuning algorithm.
+
+<p align="center">
+<img src="../../img/auto-tune-diagram.png" alt="Figure 1: Auto Tuning Conceptual Diagram." width="850" align="center" />
+</p>
+<p align="center">
+Figure 1: Auto Tuning Conceptual Diagram.
+</p>
+
+## Prerequisites
+
+Before running auto tuning, ensure you have already initialized your workspace with the `graphrag init` command. This will create the necessary configuration files and the default prompts. Refer to the [Init Documentation](../config/init.md) for more information about the initialization process.
+
+## Usage
+
+You can run the main script from the command line with various options:
+
+```bash
+graphrag prompt-tune [--root ROOT] [--domain DOMAIN]  [--selection-method METHOD] [--limit LIMIT] [--language LANGUAGE] \
+[--max-tokens MAX_TOKENS] [--chunk-size CHUNK_SIZE] [--n-subset-max N_SUBSET_MAX] [--k K] \
+[--min-examples-required MIN_EXAMPLES_REQUIRED] [--discover-entity-types] [--output OUTPUT]
+```
+
+## Command-Line Options
+
+- `--root` (optional): Path to the project directory that contains the config file (settings.yaml). Defaults to the current directory.
+
+- `--domain` (optional): The domain related to your input data, such as 'space science', 'microbiology', or 'environmental news'. If left empty, the domain will be inferred from the input data.
+
+- `--selection-method` (optional): The method to select documents. Options are all, random, auto or top. Default is random.
+
+- `--limit` (optional): The limit of text units to load when using random or top selection. Default is 15.
+
+- `--language` (optional): The language to use for input processing. If it is different from the inputs' language, the LLM will translate. Default is "" meaning it will be automatically detected from the inputs.
+
+- `--max-tokens` (optional): Maximum token count for prompt generation. Default is 2000.
+
+- `--chunk-size` (optional): The size in tokens to use for generating text units from input documents. Default is 200.
+
+- `--n-subset-max` (optional): The number of text chunks to embed when using auto selection method. Default is 300.
+
+- `--k` (optional): The number of documents to select when using auto selection method. Default is 15.
+
+- `--min-examples-required` (optional): The minimum number of examples required for entity extraction prompts. Default is 2.
+
+- `--discover-entity-types` (optional): Allow the LLM to discover and extract entities automatically. We recommend using this when your data covers a lot of topics or it is highly randomized.
+
+- `--output` (optional): The folder to save the generated prompts. Default is "prompts".
+
+## Example Usage
+
+```bash
+python -m graphrag prompt-tune --root /path/to/project --domain "environmental news" \
+--selection-method random --limit 10 --language English --max-tokens 2048 --chunk-size 256 --min-examples-required 3 \
+--no-discover-entity-types --output /path/to/output
+```
+
+or, with minimal configuration (suggested):
+
+```bash
+python -m graphrag prompt-tune --root /path/to/project --no-discover-entity-types
+```
+
+## Document Selection Methods
+
+The auto tuning feature ingests the input data and then divides it into text units the size of the chunk size parameter.
+After that, it uses one of the following selection methods to pick a sample to work with for prompt generation:
+
+- `random`: Select text units randomly. This is the default and recommended option.
+- `top`: Select the head _n_ text units.
+- `all`: Use all text units for the generation. Use only with small datasets; this option is not usually recommended.
+- `auto`: Embed text units in a lower-dimensional space and select the _k_ nearest neighbors to the centroid. This is useful when you have a large dataset and want to select a representative sample.
+
+## Modify Config
+
+After running auto tuning, you should modify the following config variables to pick up the new prompts on your index run. Note: Please make sure to update the correct path to the generated prompts, in this example we are using the default "prompts" path.
+
+```yaml
+extract_graph:
+  prompt: "prompts/extract_graph.txt"
+
+summarize_descriptions:
+  prompt: "prompts/summarize_descriptions.txt"
+
+community_reports:
+  prompt: "prompts/community_report.txt"
+```

docs/prompt_tuning/manual_prompt_tuning.md

@@ -0,0 +1,90 @@
+# Manual Prompt Tuning ⚙️
+
+The GraphRAG indexer, by default, will run with a handful of prompts that are designed to work well in the broad context of knowledge discovery.
+However, it is quite common to want to tune the prompts to better suit your specific use case.
+We provide a means for you to do this by allowing you to specify a custom prompt file, which will each use a series of token-replacements internally.
+
+Each of these prompts may be overridden by writing a custom prompt file in plaintext. We use token-replacements in the form of `{token_name}`, and the descriptions for the available tokens can be found below.
+
+## Indexing Prompts
+
+### Entity/Relationship Extraction
+
+[Prompt Source](http://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/prompts/index/extract_graph.py)
+
+#### Tokens
+
+- **{input_text}** - The input text to be processed.
+- **{entity_types}** - A list of entity types
+- **{tuple_delimiter}** - A delimiter for separating values within a tuple. A single tuple is used to represent an individual entity or relationship.
+- **{record_delimiter}** - A delimiter for separating tuple instances.
+- **{completion_delimiter}** - An indicator for when generation is complete.
+
+### Summarize Entity/Relationship Descriptions
+
+[Prompt Source](http://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/prompts/index/summarize_descriptions.py)
+
+#### Tokens
+
+- **{entity_name}** - The name of the entity or the source/target pair of the relationship.
+- **{description_list}** - A list of descriptions for the entity or relationship.
+
+### Claim Extraction
+
+[Prompt Source](http://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/prompts/index/extract_claims.py)
+
+#### Tokens
+
+- **{input_text}** - The input text to be processed.
+- **{tuple_delimiter}** - A delimiter for separating values within a tuple. A single tuple is used to represent an individual entity or relationship.
+- **{record_delimiter}** - A delimiter for separating tuple instances.
+- **{completion_delimiter}** - An indicator for when generation is complete.
+- **{entity_specs}** - A list of entity types.
+- **{claim_description}** - Description of what claims should look like. Default is: `"Any claims or facts that could be relevant to information discovery."`
+
+See the [configuration documentation](../config/overview.md) for details on how to change this.
+
+### Generate Community Reports
+
+[Prompt Source](http://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/prompts/index/community_report.py)
+
+#### Tokens
+
+- **{input_text}** - The input text to generate the report with. This will contain tables of entities and relationships.
+
+## Query Prompts
+
+### Local Search
+
+[Prompt Source](http://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/prompts/query/local_search_system_prompt.py)
+
+#### Tokens
+
+- **{response_type}** - Describe how the response should look. We default to "multiple paragraphs".
+- **{context_data}** - The data tables from GraphRAG's index.
+
+### Global Search
+
+[Mapper Prompt Source](http://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/prompts/query/global_search_map_system_prompt.py)
+
+[Reducer Prompt Source](http://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/prompts/query/global_search_reduce_system_prompt.py)
+
+[Knowledge Prompt Source](http://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/prompts/query/global_search_knowledge_system_prompt.py)
+
+Global search uses a map/reduce approach to summarization. You can tune these prompts independently. This search also includes the ability to adjust the use of general knowledge from the model's training.
+
+#### Tokens
+
+- **{response_type}** - Describe how the response should look (reducer only). We default to "multiple paragraphs".
+- **{context_data}** - The data tables from GraphRAG's index.
+
+### Drift Search
+
+[Prompt Source](http://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/prompts/query/drift_search_system_prompt.py)
+
+#### Tokens
+
+- **{response_type}** - Describe how the response should look. We default to "multiple paragraphs".
+- **{context_data}** - The data tables from GraphRAG's index.
+- **{community_reports}** - The most relevant community reports to include in the summarization.
+- **{query}** - The query text as injected into the context.

docs/prompt_tuning/overview.md

@@ -0,0 +1,15 @@
+# Prompt Tuning ⚙️
+
+This page provides an overview of the prompt tuning options available for the GraphRAG indexing engine.
+
+## Default Prompts
+
+The default prompts are the simplest way to get started with the GraphRAG system. It is designed to work out-of-the-box with minimal configuration. More details about each of the default prompts for indexing and query can be found on the [manual tuning](./manual_prompt_tuning.md) page.
+
+## Auto Tuning
+
+Auto Tuning leverages your input data and LLM interactions to create domain-adapted prompts for the generation of the knowledge graph. It is highly encouraged to run it as it will yield better results when executing an Index Run. For more details about how to use it, please refer to the [Auto Tuning](auto_prompt_tuning.md) page.
+
+## Manual Tuning
+
+Manual tuning is an advanced use-case. Most users will want to use the Auto Tuning feature instead. Details about how to use manual configuration are available in the [manual tuning](manual_prompt_tuning.md) page.

docs/query/drift_search.md

@@ -0,0 +1,36 @@
+# DRIFT Search 🔎
+
+## Combining Local and Global Search
+
+GraphRAG is a technique that uses large language models (LLMs) to create knowledge graphs and summaries from unstructured text documents and leverages them to improve retrieval-augmented generation (RAG) operations on private datasets. It offers comprehensive global overviews of large, private troves of unstructured text documents while also enabling exploration of detailed, localized information. By using LLMs to create comprehensive knowledge graphs that connect and describe entities and relationships contained in those documents, GraphRAG leverages semantic structuring of the data to generate responses to a wide variety of complex user queries.
+
+DRIFT search (Dynamic Reasoning and Inference with Flexible Traversal) builds upon Microsoft’s GraphRAG technique, combining characteristics of both global and local search to generate detailed responses in a method that balances computational costs with quality outcomes using our [drift search](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/query/structured_search/drift_search/) method.
+
+## Methodology
+
+<p align="center">
+<img src="../../img/drift-search-diagram.png" alt="Figure 1. An entire DRIFT search hierarchy highlighting the three core phases of the DRIFT search process." align="center" />
+</p>
+<p align="center"><i><small>
+Figure 1. An entire DRIFT search hierarchy highlighting the three core phases of the DRIFT search process. A (Primer): DRIFT compares the user’s query with the top K most semantically relevant community reports, generating a broad initial answer and follow-up questions to steer further exploration. B (Follow-Up): DRIFT uses local search to refine queries, producing additional intermediate answers and follow-up questions that enhance specificity, guiding the engine towards context-rich information. A glyph on each node in the diagram shows the confidence the algorithm has to continue the query expansion step.  C (Output Hierarchy): The final output is a hierarchical structure of questions and answers ranked by relevance, reflecting a balanced mix of global insights and local refinements, making the results adaptable and comprehensive.</small></i></p>
+
+
+DRIFT Search introduces a new approach to local search queries by including community information in the search process. This greatly expands the breadth of the query’s starting point and leads to retrieval and usage of a far higher variety of facts in the final answer. This addition expands the GraphRAG query engine by providing a more comprehensive option for local search, which uses community insights to refine a query into detailed follow-up questions.
+
+## Configuration
+
+Below are the key parameters of the [DRIFTSearch class](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/query/structured_search/drift_search/search.py):
+
+* `model`: Language model chat completion object to be used for response generation
+- `context_builder`: [context builder](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/query/structured_search/drift_search/drift_context.py) object to be used for preparing context data from community reports and query information
+- `config`: model to define the DRIFT Search hyperparameters. [DRIFT Config model](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/config/models/drift_search_config.py)
+- `tokenizer`: token encoder for tracking the budget for the algorithm.
+- `query_state`: a state object as defined in [Query State](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/query/structured_search/drift_search/state.py) that allows to track execution of a DRIFT Search instance, alongside follow ups and [DRIFT actions](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/query/structured_search/drift_search/action.py).
+
+## How to Use
+
+An example of a drift search scenario can be found in the following [notebook](../examples_notebooks/drift_search.ipynb).
+
+## Learn More
+
+For a more in-depth look at the DRIFT search method, please refer to our [DRIFT Search blog post](https://www.microsoft.com/en-us/research/blog/introducing-drift-search-combining-global-and-local-search-methods-to-improve-quality-and-efficiency/)

docs/query/global_search.md

@@ -0,0 +1,73 @@
+# Global Search 🔎
+
+## Whole Dataset Reasoning
+
+Baseline RAG struggles with queries that require aggregation of information across the dataset to compose an answer. Queries such as “What are the top 5 themes in the data?” perform terribly because baseline RAG relies on a vector search of semantically similar text content within the dataset. There is nothing in the query to direct it to the correct information.
+
+However, with GraphRAG we can answer such questions, because the structure of the LLM-generated knowledge graph tells us about the structure (and thus themes) of the dataset as a whole. This allows the private dataset to be organized into meaningful semantic clusters that are pre-summarized. Using our [global search](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/query/structured_search/global_search/) method, the LLM uses these clusters to summarize these themes when responding to a user query.
+
+## Methodology
+
+```mermaid
+---
+title: Global Search Dataflow
+---
+%%{ init: { 'flowchart': { 'curve': 'step' } } }%%
+flowchart LR
+
+    uq[User Query] --- .1
+    ch1[Conversation History] --- .1
+
+    subgraph RIR
+        direction TB
+        ri1[Rated Intermediate<br/>Response 1]~~~ri2[Rated Intermediate<br/>Response 2] -."{1..N}".-rin[Rated Intermediate<br/>Response N]
+    end
+
+    .1--Shuffled Community<br/>Report Batch 1-->RIR
+    .1--Shuffled Community<br/>Report Batch 2-->RIR---.2
+    .1--Shuffled Community<br/>Report Batch N-->RIR
+
+    .2--Ranking +<br/>Filtering-->agr[Aggregated Intermediate<br/>Responses]-->res[Response]
+
+
+
+     classDef green fill:#26B653,stroke:#333,stroke-width:2px,color:#fff;
+     classDef turquoise fill:#19CCD3,stroke:#333,stroke-width:2px,color:#fff;
+     classDef rose fill:#DD8694,stroke:#333,stroke-width:2px,color:#fff;
+     classDef orange fill:#F19914,stroke:#333,stroke-width:2px,color:#fff;
+     classDef purple fill:#B356CD,stroke:#333,stroke-width:2px,color:#fff;
+     classDef invisible fill:#fff,stroke:#fff,stroke-width:0px,color:#fff, width:0px;
+     class uq,ch1 turquoise;
+     class ri1,ri2,rin rose;
+     class agr orange;
+     class res purple;
+     class .1,.2 invisible;
+
+```
+
+Given a user query and, optionally, the conversation history, the global search method uses a collection of LLM-generated community reports from a specified level of the graph's community hierarchy as context data to generate response in a map-reduce manner. At the `map` step, community reports are segmented into text chunks of pre-defined size. Each text chunk is then used to produce an intermediate response containing a list of point, each of which is accompanied by a numerical rating indicating the importance of the point. At the `reduce` step, a filtered set of the most important points from the intermediate responses are aggregated and used as the context to generate the final response. 
+
+The quality of the global search’s response can be heavily influenced by the level of the community hierarchy chosen for sourcing community reports. Lower hierarchy levels, with their detailed reports, tend to yield more thorough responses, but may also increase the time and LLM resources needed to generate the final response due to the volume of reports.
+
+
+## Configuration
+
+Below are the key parameters of the [GlobalSearch class](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/query/structured_search/global_search/search.py):
+
+* `model`: Language model chat completion object to be used for response generation
+* `context_builder`: [context builder](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/query/structured_search/global_search/community_context.py) object to be used for preparing context data from community reports
+* `map_system_prompt`: prompt template used in the `map` stage. Default template can be found at [map_system_prompt](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/prompts/query/global_search_map_system_prompt.py)
+* `reduce_system_prompt`: prompt template used in the `reduce` stage, default template can be found at [reduce_system_prompt](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/prompts/query/global_search_reduce_system_prompt.py)
+* `response_type`: free-form text describing the desired response type and format (e.g., `Multiple Paragraphs`, `Multi-Page Report`)
+* `allow_general_knowledge`: setting this to True will include additional instructions to the `reduce_system_prompt` to prompt the LLM to incorporate relevant real-world knowledge outside of the dataset. Note that this may increase hallucinations, but can be useful for certain scenarios. Default is False
+*`general_knowledge_inclusion_prompt`: instruction to add to the `reduce_system_prompt` if `allow_general_knowledge` is enabled. Default instruction can be found at [general_knowledge_instruction](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/prompts/query/global_search_knowledge_system_prompt.py)
+* `max_data_tokens`: token budget for the context data
+* `map_llm_params`: a dictionary of additional parameters (e.g., temperature, max_tokens) to be passed to the LLM call at the `map` stage
+* `reduce_llm_params`: a dictionary of additional parameters (e.g., temperature, max_tokens) to passed to the LLM call at the `reduce` stage
+* `context_builder_params`: a dictionary of additional parameters to be passed to the [`context_builder`](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/query/structured_search/global_search/community_context.py) object when building context window for the `map` stage.
+* `concurrent_coroutines`: controls the degree of parallelism in the `map` stage.
+* `callbacks`: optional callback functions, can be used to provide custom event handlers for LLM's completion streaming events
+
+## How to Use
+
+An example of a global search scenario can be found in the following [notebook](../examples_notebooks/global_search.ipynb).

docs/query/local_search.md

@@ -0,0 +1,62 @@
+# Local Search 🔎
+
+## Entity-based Reasoning
+
+The [local search](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/query/structured_search/local_search/) method combines structured data from the knowledge graph with unstructured data from the input documents to augment the LLM context with relevant entity information at query time. It is well-suited for answering questions that require an understanding of specific entities mentioned in the input documents (e.g., “What are the healing properties of chamomile?”).
+
+## Methodology
+
+```mermaid
+---
+title: Local Search Dataflow
+---
+%%{ init: { 'flowchart': { 'curve': 'step' } } }%%
+flowchart LR
+
+    uq[User Query] ---.1
+    ch1[Conversation<br/>History]---.1
+
+    .1--Entity<br/>Description<br/>Embedding--> ee[Extracted Entities]
+
+    ee[Extracted Entities] ---.2--Entity-Text<br/>Unit Mapping--> ctu[Candidate<br/>Text Units]--Ranking + <br/>Filtering -->ptu[Prioritized<br/>Text Units]---.3
+    .2--Entity-Report<br/>Mapping--> ccr[Candidate<br/>Community Reports]--Ranking + <br/>Filtering -->pcr[Prioritized<br/>Community Reports]---.3
+    .2--Entity-Entity<br/>Relationships--> ce[Candidate<br/>Entities]--Ranking + <br/>Filtering -->pe[Prioritized<br/>Entities]---.3
+    .2--Entity-Entity<br/>Relationships--> cr[Candidate<br/>Relationships]--Ranking + <br/>Filtering -->pr[Prioritized<br/>Relationships]---.3
+    .2--Entity-Covariate<br/>Mappings--> cc[Candidate<br/>Covariates]--Ranking + <br/>Filtering -->pc[Prioritized<br/>Covariates]---.3
+    ch1 -->ch2[Conversation History]---.3
+    .3-->res[Response]
+
+     classDef green fill:#26B653,stroke:#333,stroke-width:2px,color:#fff;
+     classDef turquoise fill:#19CCD3,stroke:#333,stroke-width:2px,color:#fff;
+     classDef rose fill:#DD8694,stroke:#333,stroke-width:2px,color:#fff;
+     classDef orange fill:#F19914,stroke:#333,stroke-width:2px,color:#fff;
+     classDef purple fill:#B356CD,stroke:#333,stroke-width:2px,color:#fff;
+     classDef invisible fill:#fff,stroke:#fff,stroke-width:0px,color:#fff, width:0px;
+     class uq,ch1 turquoise
+     class ee green
+     class ctu,ccr,ce,cr,cc rose
+     class ptu,pcr,pe,pr,pc,ch2 orange
+     class res purple
+     class .1,.2,.3 invisible
+
+
+```
+
+Given a user query and, optionally, the conversation history, the local search method identifies a set of entities from the knowledge graph that are semantically-related to the user input. These entities serve as access points into the knowledge graph, enabling the extraction of further relevant details such as connected entities, relationships, entity covariates, and community reports. Additionally, it also extracts relevant text chunks from the raw input documents that are associated with the identified entities. These candidate data sources are then prioritized and filtered to fit within a single context window of pre-defined size, which is used to generate a response to the user query.
+
+## Configuration
+
+Below are the key parameters of the [LocalSearch class](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/query/structured_search/local_search/search.py):
+
+* `model`: Language model chat completion object to be used for response generation
+* `context_builder`: [context builder](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/query/structured_search/local_search/mixed_context.py) object to be used for preparing context data from collections of knowledge model objects
+* `system_prompt`: prompt template used to generate the search response. Default template can be found at [system_prompt](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/prompts/query/local_search_system_prompt.py)
+* `response_type`: free-form text describing the desired response type and format (e.g., `Multiple Paragraphs`, `Multi-Page Report`)
+* `llm_params`: a dictionary of additional parameters (e.g., temperature, max_tokens) to be passed to the LLM call
+* `context_builder_params`: a dictionary of additional parameters to be passed to the [`context_builder`](https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/query/structured_search/local_search/mixed_context.py) object when building context for the search prompt
+* `callbacks`: optional callback functions, can be used to provide custom event handlers for LLM's completion streaming events
+
+## How to Use
+
+An example of a local search scenario can be found in the following [notebook](../examples_notebooks/local_search.ipynb).
+

docs/query/notebooks/overview.md

@@ -0,0 +1,14 @@
+# API Notebooks
+
+- [API Overview Notebook](../../examples_notebooks/api_overview.ipynb)
+- [Bring-Your-Own Vector Store](../../examples_notebooks/custom_vector_store.ipynb)
+
+# Query Engine Notebooks
+
+For examples about running Query please refer to the following notebooks:
+
+- [Global Search Notebook](../../examples_notebooks/global_search.ipynb)
+- [Local Search Notebook](../../examples_notebooks/local_search.ipynb)
+- [DRIFT Search Notebook](../../examples_notebooks/drift_search.ipynb)
+
+The test dataset for these notebooks can be found in [dataset.zip](../../data/operation_dulce/dataset.zip){:download}.