microsoft--semantic-kernel/docs/decisions/0018-kernel-hooks-phase2.md

Context and Problem Statement

Currently Kernel invoking and invoked handlers don't expose the prompt to the handlers.

The proposal is a way to expose the prompt to the handlers.

• Pre-Execution / Invoking

  • Get: Prompt generated by the current SemanticFunction.TemplateEngine before calling the LLM
  • Set: Modify a prompt content before sending it to LLM

• Post-Execution / Invoked

  • Get: Generated Prompt

Decision Drivers

• Prompt template should be generated just once per function execution within the Kernel.RunAsync execution.
• Handlers should be able to see and modify the prompt before the LLM execution.
• Handlers should be able to see prompt after the LLM execution.
• Calling Kernel.RunAsync(function) or ISKFunction.InvokeAsync(kernel) should trigger the events.

Out of Scope

• Skip plan steps using Pre-Hooks.
• Get the used services (Template Engine, IAIServices, etc) in the Pre/Post Hooks.
• Get the request settings in the Pre/Post Hooks.

Current State of Kernel for Pre/Post Hooks

Current state of Kernel:

class Kernel : IKernel

RunAsync()
{
    var context = this.CreateNewContext(variables);
    var functionDetails = skFunction.Describe();
    var functionInvokingArgs = this.OnFunctionInvoking(functionDetails, context);

    functionResult = await skFunction.InvokeAsync(context, cancellationToken: cancellationToken);
    var functionInvokedArgs = this.OnFunctionInvoked(functionDetails, functionResult);
}

Developer Experience

Below is the expected end user experience when coding using Pre/Post Hooks to get or modify prompts.

const string FunctionPrompt = "Write a random paragraph about: {{$input}}.";

var excuseFunction = kernel.CreateSemanticFunction(...);

void MyPreHandler(object? sender, FunctionInvokingEventArgs e)
{
    Console.WriteLine($"{e.FunctionView.PluginName}.{e.FunctionView.Name} : Pre Execution Handler - Triggered");

    // Will be false for non semantic functions
    if (e.TryGetRenderedPrompt(out var prompt))
    {
        Console.WriteLine("Rendered Prompt:");
        Console.WriteLine(prompt);

        // Update the prompt if needed
        e.TryUpdateRenderedPrompt("Write a random paragraph about: Overriding a prompt");
    }
}

void MyPostHandler(object? sender, FunctionInvokedEventArgs e)
{
    Console.WriteLine($"{e.FunctionView.PluginName}.{e.FunctionView.Name} : Post Execution Handler - Triggered");
    // Will be false for non semantic functions
    if (e.TryGetRenderedPrompt(out var prompt))
    {
        Console.WriteLine("Used Prompt:");
        Console.WriteLine(prompt);
    }
}

kernel.FunctionInvoking += MyPreHandler;
kernel.FunctionInvoked += MyPostHandler;

const string Input = "I missed the F1 final race";
var result = await kernel.RunAsync(Input, excuseFunction);
Console.WriteLine($"Function Result: {result.GetValue<string>()}");

Expected output:

MyPlugin.MyFunction : Pre Execution Handler - Triggered
Rendered Prompt:
Write a random paragraph about: I missed the F1 final race.

MyPlugin.MyFunction : Post Execution Handler - Triggered
Used Prompt:
Write a random paragraph about: Overriding a prompt

FunctionResult: <LLM Completion>

Considered Options

Improvements Common to all options

Move Dictionary<string, object> property Metadata from FunctionInvokedEventArgs to SKEventArgs abstract class.

Pro:

• This will make all SKEventArgs extensible, allowing extra information to be passed to the EventArgs when specialization isn't possible.

Option 1: Kernel awareness of SemanticFunctions

class Kernel : IKernel

RunAsync()
{

    if (skFunction is SemanticFunction semanticFunction)
    {
        var prompt = await semanticFunction.TemplateEngine.RenderAsync(semanticFunction.Template, context);
        var functionInvokingArgs = this.OnFunctionInvoking(functionDetails, context, prompt);
        // InvokeWithPromptAsync internal
        functionResult = await semanticFunction.InternalInvokeWithPromptAsync(prompt, context, cancellationToken: cancellationToken);
    }
    else
    {
        functionResult = await skFunction.InvokeAsync(context, cancellationToken: cancellationToken);
    }
}
class SemanticFunction : ISKFunction

public InvokeAsync(context, cancellationToken)
{
    var prompt = _templateEngine.RenderAsync();
    return InternalInvokeWithPromptAsync(prompt, context, cancellationToken);
}

internal InternalInvokeWithPromptAsync(string prompt)
{
    ... current logic to call LLM
}

Pros and Cons

Pros:

• Simpler and quicker to implement
• Small number of changes limited mostly to Kernel and SemanticFunction classes

Cons:

• Kernel is aware of SemanticFunction implementation details
• Not extensible to show prompts of custom ISKFunctions implementations

Option 2: Delegate to the ISKFunction how to handle events (Interfaces approach)

class Kernel : IKernel
{
    RunAsync() {
        var functionInvokingArgs = await this.TriggerEvent<FunctionInvokingEventArgs>(this.FunctionInvoking, skFunction, context);

        var functionResult = await skFunction.InvokeAsync(context, cancellationToken: cancellationToken);

        var functionInvokedArgs = await this.TriggerEvent<FunctionInvokedEventArgs>(
            this.FunctionInvoked,
            skFunction,
            context);
    }

    private TEventArgs? TriggerEvent<TEventArgs>(EventHandler<TEventArgs>? eventHandler, ISKFunction function, SKContext context) where TEventArgs : SKEventArgs
    {
        if (eventHandler is null)
        {
            return null;
        }

        if (function is ISKFunctionEventSupport<TEventArgs> supportedFunction)
        {
            var eventArgs = await supportedFunction.PrepareEventArgsAsync(context);
            eventHandler.Invoke(this, eventArgs);
            return eventArgs;
        }

        // Think about allowing to add data with the extra interface.

        // If a function don't support the specific event we can:
        return null; // Ignore or Throw.
        throw new NotSupportedException($"The provided function \"{function.Name}\" does not supports and implements ISKFunctionHandles<{typeof(TEventArgs).Name}>");
    }
}

public interface ISKFunctionEventSupport<TEventArgs> where TEventArgs : SKEventArgs
{
    Task<TEventArgs> PrepareEventArgsAsync(SKContext context, TEventArgs? eventArgs = null);
}

class SemanticFunction : ISKFunction,
    ISKFunctionEventSupport<FunctionInvokingEventArgs>,
    ISKFunctionEventSupport<FunctionInvokedEventArgs>
{

    public FunctionInvokingEventArgs PrepareEventArgsAsync(SKContext context, FunctionInvokingEventArgs? eventArgs = null)
    {
        var renderedPrompt = await this.RenderPromptTemplateAsync(context);
        context.Variables.Set(SemanticFunction.RenderedPromptKey, renderedPrompt);

        return new SemanticFunctionInvokingEventArgs(this.Describe(), context);
        // OR                                                          Metadata Dictionary<string, object>
        return new FunctionInvokingEventArgs(this.Describe(), context, new Dictionary<string, object>() { { RenderedPrompt, renderedPrompt } });
    }

    public FunctionInvokedEventArgs PrepareEventArgsAsync(SKContext context, FunctionInvokedEventArgs? eventArgs = null)
    {
        return Task.FromResult<FunctionInvokedEventArgs>(new SemanticFunctionInvokedEventArgs(this.Describe(), context));
    }
}

public sealed class SemanticFunctionInvokedEventArgs : FunctionInvokedEventArgs
{
    public SemanticFunctionInvokedEventArgs(FunctionDescription functionDescription, SKContext context)
        : base(functionDescription, context)
    {
        _context = context;
        Metadata[RenderedPromptKey] = this._context.Variables[RenderedPromptKey];
    }

    public string? RenderedPrompt => this.Metadata[RenderedPromptKey];

}

public sealed class SemanticFunctionInvokingEventArgs : FunctionInvokingEventArgs
{
    public SemanticFunctionInvokingEventArgs(FunctionDescription functionDescription, SKContext context)
        : base(functionDescription, context)
    {
        _context = context;
    }
    public string? RenderedPrompt => this._context.Variables[RenderedPromptKey];
}

Pros and Cons

Pros:

• Kernel is not aware of SemanticFunction implementation details or any other ISKFunction implementation
• Extensible to show dedicated EventArgs per custom ISKFunctions implementation, including prompts for semantic functions
• Extensible to support future events on the Kernel thru the ISKFunctionEventSupport<NewEvent> interface
• Functions can have their own EventArgs specialization.
• Interface is optional, so custom ISKFunctions can choose to implement it or not

Cons:

• Any custom functions now will have to responsibility implement the ISKFunctionEventSupport interface if they want to support events.
• Handling events in another ISKFunction requires more complex approaches to manage the context and the prompt + any other data in different event handling methods.

Option 3: Delegate to the ISKFunction how to handle events (InvokeAsync Delegates approach)

Add Kernel event handler delegate wrappers to ISKFunction.InvokeAsync interface. This approach shares the responsibility of handling the events between the Kernel and the ISKFunction implementation, flow control will be handled by the Kernel and the ISKFunction will be responsible for calling the delegate wrappers and adding data to the SKEventArgs that will be passed to the handlers.

class Kernel : IKernel
{
    RunAsync() {
        var functionInvokingDelegateWrapper = new(this.FunctionInvoking);
        var functionInvokedDelegateWrapper = new(this.FunctionInvoked);

        var functionResult = await skFunction.InvokeAsync(context, functionInvokingDelegateWrapper, functionInvokingDelegateWrapper, functionInvokedDelegateWrapper);

        // Kernel will analyze the delegate results and make flow related decisions
        if (functionInvokingDelegateWrapper.EventArgs.CancelRequested ... ) { ... }
        if (functionInvokingDelegateWrapper.EventArgs.SkipRequested ... ) { ... }
        if (functionInvokedDelegateWrapper.EventArgs.Repeat ... ) { ... }
    }
}

class SemanticFunction : ISKFunction {
    InvokeAsync(
        SKContext context,
        FunctionInvokingDelegateWrapper functionInvokingDelegateWrapper,
        FunctionInvokedDelegateWrapper functionInvokedDelegateWrapper)
    {
        // The Semantic will have to call the delegate wrappers and share responsibility with the `Kernel`.
        if (functionInvokingDelegateWrapper.Handler is not null)
        {
            var renderedPrompt = await this.RenderPromptTemplateAsync(context);
            functionInvokingDelegateWrapper.EventArgs.RenderedPrompt = renderedPrompt;

            functionInvokingDelegateWrapper.Handler.Invoke(this, functionInvokingDelegateWrapper.EventArgs);

            if (functionInvokingDelegateWrapper.EventArgs?.CancelToken.IsCancellationRequested ?? false)
            {
                // Need to enforce an non processed result
                return new SKFunctionResult(context);

                //OR make InvokeAsync allow returning null FunctionResult?
                return null;
            }
        }
    }
}

// Wrapper for the EventHandler
class FunctionDelegateWrapper<TEventArgs> where TEventArgs : SKEventArgs
{
    FunctionInvokingDelegateWrapper(EventHandler<TEventArgs> eventHandler) {}

    // Set allows specialized eventargs to be set.
    public TEventArgs EventArgs { get; set; }
    public EventHandler<TEventArgs> Handler => _eventHandler;
}

Pros and Cons

Pros:

• ISKFunction has less code/complexity to handle and expose data (Rendered Prompt) and state in the EventArgs.
• Kernel is not aware of SemanticFunction implementation details or any other ISKFunction implementation
• Kernel has less code/complexity
• Could be extensible to show dedicated EventArgs per custom ISKFunctions implementation, including prompts for semantic functions

Cons:

• Unable to add new events if needed (ISKFunction interface change needed)
• Functions need to implement behavior related to dependency (Kernel) events
• Since Kernel needs to interact with the result of an event handler, a wrapper strategy is needed to access results by reference at the kernel level (control of flow)
• Passing Kernel event handlers full responsibility downstream to the functions don't sound quite right (Single Responsibility)

Option 4: Delegate to the ISKFunction how to handle events (SKContext Delegates approach)

Add Kernel event handler delegate wrappers to ISKFunction.InvokeAsync interface. This approach shares the responsibility of handling the events between the Kernel and the ISKFunction implementation, flow control will be handled by the Kernel and the ISKFunction will be responsible for calling the delegate wrappers and adding data to the SKEventArgs that will be passed to the handlers.

class Kernel : IKernel
{
    CreateNewContext() {
        var context = new SKContext(...);
        context.AddEventHandlers(this.FunctionInvoking, this.FunctionInvoked);
        return context;
    }
    RunAsync() {
        functionResult = await skFunction.InvokeAsync(context, ...);
        if (this.IsCancelRequested(functionResult.Context)))
            break;
        if (this.IsSkipRequested(functionResult.Context))
            continue;
        if (this.IsRepeatRequested(...))
            goto repeat;

        ...
    }
}

class SKContext {

    internal EventHandlerWrapper<FunctionInvokingEventArgs>? FunctionInvokingHandler { get; private set; }
    internal EventHandlerWrapper<FunctionInvokedEventArgs>? FunctionInvokedHandler { get; private set; }

    internal SKContext(
        ...
        ICollection<EventHandlerWrapper?>? eventHandlerWrappers = null
    {
        ...
        this.InitializeEventWrappers(eventHandlerWrappers);
    }

    void InitializeEventWrappers(ICollection<EventHandlerWrapper?>? eventHandlerWrappers)
    {
        if (eventHandlerWrappers is not null)
        {
            foreach (var handler in eventHandlerWrappers)
            {
                if (handler is EventHandlerWrapper<FunctionInvokingEventArgs> invokingWrapper)
                {
                    this.FunctionInvokingHandler = invokingWrapper;
                    continue;
                }

                if (handler is EventHandlerWrapper<FunctionInvokedEventArgs> invokedWrapper)
                {
                    this.FunctionInvokedHandler = invokedWrapper;
                }
            }
        }
    }
}

class SemanticFunction : ISKFunction {
    InvokeAsync(
        SKContext context
    {
        string renderedPrompt = await this._promptTemplate.RenderAsync(context, cancellationToken).ConfigureAwait(false);

        this.CallFunctionInvoking(context, renderedPrompt);
        if (this.IsInvokingCancelOrSkipRequested(context, out var stopReason))
        {
            return new StopFunctionResult(this.Name, this.PluginName, context, stopReason!.Value);
        }

        string completion = await GetCompletionsResultContentAsync(...);

        var result = new FunctionResult(this.Name, this.PluginName, context, completion);
        result.Metadata.Add(SemanticFunction.RenderedPromptMetadataKey, renderedPrompt);

        this.CallFunctionInvoked(result, context, renderedPrompt);
        if (this.IsInvokedCancelRequested(context, out stopReason))
        {
            return new StopFunctionResult(this.Name, this.PluginName, context, result.Value, stopReason!.Value);
        }

        return result;
    }
}

Pros and Cons

Pros:

• ISKFunction has less code/complexity to handle and expose data (Rendered Prompt) and state in the EventArgs.
• Kernel is not aware of SemanticFunction implementation details or any other ISKFunction implementation
• Kernel has less code/complexity
• Could be extensible to show dedicated EventArgs per custom ISKFunctions implementation, including prompts for semantic functions
• More extensible as ISKFunction interface doesn't need to change to add new events.
• SKContext can be extended to add new events without introducing breaking changes.

Cons:

• Functions now need to implement logic to handle in-context events
• Since Kernel needs to interact with the result of an event handler, a wrapper strategy is needed to access results by reference at the kernel level (control of flow)
• Passing Kernel event handlers full responsibility downstream to the functions don't sound quite right (Single Responsibility)

Decision outcome

Option 4: Delegate to the ISKFunction how to handle events (SKContext Delegates approach)

This allow the functions to implement some of the kernel logic but has the big benefit of not splitting logic in different methods for the same Execution Context.

Biggest benefit: ISKFunction has less code/complexity to handle and expose data and state in the EventArgs. ISKFunction interface doesn't need to change to add new events.

This implementation allows to get the renderedPrompt in the InvokeAsync without having to manage the context and the prompt in different methods.

The above also applies for any other data that is available in the invocation and can be added as a new EventArgs property.