This sample app demonstrates how to create technical documents for a codebase using AI. More specifically, it uses the agent framework offered by Semantic Kernel to ochestrate multiple agents to create a technical document.

This sample app also provides telemetry to monitor the agents, making it easier to observe the inner workings of the agents.

To learn more about agents, please refer to this introduction video. To learn more about the Semantic Kernel Agent Framework, please refer to the Semantic Kernel documentation.

Note: This sample app cannot guarantee to generate a perfect technical document each time due to the stochastic nature of the AI model. Please a version of the document generated by the app in GENERATED_DOCUMENT.md.

Design

Tools/Plugins

Code Execution Plugin: This plugin offers a sandbox environment to execute Python snippets. It returns the output of the program or errors if any.
Repository File Plugin: This plugin allows the AI to retrieve files from the Semantic Kernel repository.
User Input Plugin: This plugin allows the AI to present content to the user and receive feedback.

Agents

Content Creation Agent: This agent is responsible for creating the content of the document. This agent has access to the Repository File Plugin to read source files it deems necessary for reference.
Code Validation Agent: This agent is responsible for validating the code snippets in the document. This agent has access to the Code Execution Plugin to execute the code snippets.
User Agent: This agent is responsible for interacting with the user. This agent has access to the User Input Plugin to present content to the user and receive feedback.

Agent Selection Strategy

Termination Strategy

Prerequisites

Azure OpenAI
Azure Application Insights

Additional packages

AICodeSandbox - for executing AI generated code in a sandbox environment
```
pip install ai-code-sandbox
```
You must also have docker installed and running on your machine. Follow the instructions here to install docker for your platform. Images will be pulled during runtime if not already present. Containers will be created and destroyed during code execution.

Running the app

Step 1: Set up the environment

The Document Generator demo currently supports two different AI Services. If integrating into your own code, other AI services can be used. See Semantic Kernel's Learn Site page about other available Chat Completion services.

OpenAI

Make sure you have the following environment variables set:

OPENAI_CHAT_MODEL_ID=<model-id>
OPENAI_API_KEY=<your-key>

gpt-4o-2024-08-06 was used to generate GENERATED_DOCUMENT.md. Feel free to use other models from OpenAI or other providers. When you use models from another provider, make sure to update the chat completion services accordingly.

Azure OpenAI

AZURE_OPENAI_CHAT_DEPLOYMENT_NAME=<deployment-name>
AZURE_OPENAI_ENDPOINT=<endpoint>  # in the form of `https://<resource>.openai.azure.com/`
AZURE_OPENAI_API_KEY=<api-key>  # only required if using api key auth
AZURE_OPENAI_API_VERSION=<api-version>  # optional, defaults to the latest Azure OpenAI GA API version of `2024-10-21` if not provided

### Step 2: Run the app

```bash
python ./main.py

Expected output:

==== ContentCreationAgent just responded ====
==== CodeValidationAgent just responded ====
==== ContentCreationAgent just responded ====
...

Customization

Since this is a sample app that demonstrates the creation of a technical document on Semantic Kernel AI connectors, you can customize the app to suit your needs. You can try different tasks, add more agents, tune existing agents, change the agent selection strategy, or modify the termination strategy.

To try a different task, modify the TASK prompt in main.py.
To add more agents, create a new agent under agents/ and add it to the agents list in main.py.
To tune existing agents, modify the INSTRUCTION prompt in the agent's source code.
To change the agent selection strategy, modify custom_selection_strategy.py.
To change the termination strategy, modify custom_termination_strategy.py.

Optional: Monitoring the agents

When you see the final document generated by the app, what you see is actually the creation of multiple agents working together. You may wonder, how did the agents work together to create the document? What was the sequence of actions taken by the agents? How did the agents interact with each other? To answer these questions, you need to observe the agents.

Semantic Kernel by default instruments all the LLM calls. However, for agents there is no default instrumentation. This sample app shows how one can extend the Semantic Kernel agent to add instrumentation.

There are currently no standards on what information needs to be captured for agents as the concept of agents is still relatively new. At the time of writing, the Semantic Convention for agents is still in the draft stage: https://github.com/open-telemetry/semantic-conventions/issues/1732

To monitor the agents, set the following environment variables:

AZURE_APP_INSIGHTS_CONNECTION_STRING=<your-connection-string>

SEMANTICKERNEL_EXPERIMENTAL_GENAI_ENABLE_OTEL_DIAGNOSTICS=true
SEMANTICKERNEL_EXPERIMENTAL_GENAI_ENABLE_OTEL_DIAGNOSTICS_SENSITIVE=true

Follow this guide to inspect the telemetry data: https://learn.microsoft.com/en-us/semantic-kernel/concepts/enterprise-readiness/observability/telemetry-with-app-insights?tabs=Powershell&pivots=programming-language-python#inspect-telemetry-data

Or follow this guide to visualize the telemetry data on Azure AI Foundry: https://learn.microsoft.com/en-us/semantic-kernel/concepts/enterprise-readiness/observability/telemetry-with-azure-ai-foundry-tracing#visualize-traces-on-azure-ai-foundry-tracing-ui-1