--- # These are optional elements. Feel free to remove any of them. status: proposed contact: rogerbarreto date: 2024-05-02 deciders: rogerbarreto, markwallace-microsoft, sergeymenkshi, dmytrostruk, sergeymenshik, westey-m, matthewbolanos consulted: stephentoub --- # Kernel Content Types Graduation ## Context and Problem Statement Currently, we have many Content Types in experimental state and this ADR will give some options on how to graduate them to stable state. ## Decision Drivers - No breaking changes - Simple approach, minimal complexity - Allow extensibility - Concise and clear ## BinaryContent Graduation This content should be by content specializations or directly for types that aren't specific, similar to "application/octet-stream" mime type. > **Application/Octet-Stream** is the MIME used for arbitrary binary data or a stream of bytes that doesn't fit any other more specific MIME type. This MIME type is often used as a default or fallback type, indicating that the file should be treated as pure binary data. #### Current ```csharp public class BinaryContent : KernelContent { public ReadOnlyMemory? Content { get; set; } public async Task GetStreamAsync() public async Task> GetContentAsync() ctor(ReadOnlyMemory? content = null) ctor(Func> streamProvider) } ``` #### Proposed ```csharp public class BinaryContent : KernelContent { ReadOnlyMemory? Data { get; set; } Uri? Uri { get; set; } string DataUri { get; set; } bool CanRead { get; } // Indicates if the content can be read as bytes or data uri ctor(Uri? referencedUri) ctor(string dataUri) // MimeType is not optional but nullable to encourage this information to be passed always when available. ctor(ReadOnlyMemory data, string? mimeType) ctor() // Empty ctor for serialization scenarios } ``` - No Content property (Avoid clashing and/or misleading information if used from a specialized type context) i.e: - `PdfContent.Content` (Describe the text only information) - `PictureContent.Content` (Exposes a `Picture` type) - Move away from deferred (lazy loaded) content providers, simpler API. - `GetContentAsync` removal (No more derrefed APIs) - Added `Data` property as setter and getter for byte array content information. Setting this property will override the `DataUri` base64 data part. - Added `DataUri` property as setter and getter for data uri content information. Setting this property will override the `Data` and `MimeType` properties with the current payload details. - Add `Uri` property for referenced content information. This property is does not accept not a `UriData` and only supports non-data schemes. - Add `CanRead` property (To indicate if the content can be read using `Data` or `DataUri` properties.) - Dedicated constructors for Uri, DataUri and ByteArray + MimeType creation. Pros: - With no deferred content we have simpler API and a single responsibility for contents. - Can be written and read in both `Data` or `DataUri` formats. - Can have a `Uri` reference property, which is common for specialized contexts. - Fully serializable. - Data Uri parameters support (serialization included). - Data Uri and Base64 validation checks - Data Uri and Data can be dynamically generated - `CanRead` will clearly identify if the content can be read as `bytes` or `DataUri`. Cons: - Breaking change for experimental `BinaryContent` consumers ### Data Uri Parameters According to [RFC 2397](https://datatracker.ietf.org/doc/html/rfc2397), the data uri scheme supports parameters Every parameter imported from the data uri will be added to the Metadata dictionary with the "data-uri-parameter-name" as key and its respetive value. #### Providing a parameterized data uri will include those parameters in the Metadata dictionary. ```csharp var content = new BinaryContent("data:application/json;parameter1=value1;parameter2=value2;base64,SGVsbG8gV29ybGQ="); var parameter1 = content.Metadata["data-uri-parameter1"]; // value1 var parameter2 = content.Metadata["data-uri-parameter2"]; // value2 ``` #### Deserialization of contents will also include those parameters when getting the DataUri property. ```csharp var json = """ { "metadata": { "data-uri-parameter1":"value1", "data-uri-parameter2":"value2" }, "mimeType":"application/json", "data":"SGVsbG8gV29ybGQ=" } """; var content = JsonSerializer.Deserialize(json); content.DataUri // "data:application/json;parameter1=value1;parameter2=value2;base64,SGVsbG8gV29ybGQ=" ``` ### Specialization Examples #### ImageContent ```csharp public class ImageContent : BinaryContent { ctor(Uri uri) : base(uri) ctor(string dataUri) : base(dataUri) ctor(ReadOnlyMemory data, string? mimeType) : base(data, mimeType) ctor() // serialization scenarios } public class AudioContent : BinaryContent { ctor(Uri uri) } ``` Pros: - Supports data uri large contents - Allows a binary ImageContent to be created using dataUrl scheme and also be referenced by a Url. - Supports Data Uri validation ## ImageContent Graduation ⚠️ Currently this is not experimental, breaking changes needed to be graduated to stable state with potential benefits. ### Problems 1. Current `ImageContent` does not derive from `BinaryContent` 2. Has an undesirable behavior allowing the same instance to have distinct `DataUri` and `Data` at the same time. 3. `Uri` property is used for both data uri and referenced uri information 4. `Uri` does not support large language data uri formats. 5. Not clear to the `sk developer` whenever the content is readable or not. #### Current ```csharp public class ImageContent : KernelContent { Uri? Uri { get; set; } public ReadOnlyMemory? Data { get; set; } ctor(ReadOnlyMemory? data) ctor(Uri uri) ctor() } ``` #### Proposed As already shown in the `BinaryContent` section examples, the `ImageContent` can be graduated to be a `BinaryContent` specialization an inherit all the benefits it brings. ```csharp public class ImageContent : BinaryContent { ctor(Uri uri) : base(uri) ctor(string dataUri) : base(dataUri) ctor(ReadOnlyMemory data, string? mimeType) : base(data, mimeType) ctor() // serialization scenarios } ``` Pros: - Can be used as a `BinaryContent` type - Can be written and read in both `Data` or `DataUri` formats. - Can have a `Uri` dedicated for referenced location. - Fully serializable. - Data Uri parameters support (serialization included). - Data Uri and Base64 validation checks - Can be retrieved - Data Uri and Data can be dynamically generated - `CanRead` will clearly identify if the content can be read as `bytes` or `DataUri`. Cons: - ⚠️ Breaking change for `ImageContent` consumers ### ImageContent Breaking Changes - `Uri` property will be dedicated solely for referenced locations (non-data-uri), attempting to add a `data-uri` format will throw an exception suggesting the usage of the `DataUri` property instead. - Setting `DataUri` will override the `Data` and `MimeType` properties according with the information provided. - Attempting to set an invalid `DataUri` will throw an exception. - Setting `Data` will now override the `DataUri` data part. - Attempting to serialize an `ImageContent` with data-uri in the `Uri` property will throw an exception. ## AudioContent Graduation Similar to `ImageContent` proposal `AudioContent` can be graduated to be a `BinaryContent`. #### Current 1. Current `AudioContent` does not derive support `Uri` referenced location 2. `Uri` property is used for both data uri and referenced uri information 3. `Uri` does not support large language data uri formats. 4. Not clear to the `sk developer` whenever the content is readable or not. ```csharp public class AudioContent : KernelContent { public ReadOnlyMemory? Data { get; set; } ctor(ReadOnlyMemory? data) ctor() } ``` #### Proposed ```csharp public class AudioContent : BinaryContent { ctor(Uri uri) : base(uri) ctor(string dataUri) : base(dataUri) ctor(ReadOnlyMemory data, string? mimeType) : base(data, mimeType) ctor() // serialization scenarios } ``` Pros: - Can be used as a `BinaryContent` type - Can be written and read in both `Data` or `DataUri` formats. - Can have a `Uri` dedicated for referenced location. - Fully serializable. - Data Uri parameters support (serialization included). - Data Uri and Base64 validation checks - Can be retrieved - Data Uri and Data can be dynamically generated - `CanRead` will clearly identify if the content can be read as `bytes` or `DataUri`. Cons: - Experimental breaking change for `AudioContent` consumers ## FunctionCallContent Graduation ### Current No changes needed to current structure. Potentially we could have a base `FunctionContent` but at the same time is good having those two deriving from `KernelContent` providing a clear separation of concerns. ```csharp public sealed class FunctionCallContent : KernelContent { public string? Id { get; } public string? PluginName { get; } public string FunctionName { get; } public KernelArguments? Arguments { get; } public Exception? Exception { get; init; } ctor(string functionName, string? pluginName = null, string? id = null, KernelArguments? arguments = null) public async Task InvokeAsync(Kernel kernel, CancellationToken cancellationToken = default) public static IEnumerable GetFunctionCalls(ChatMessageContent messageContent) } ``` ## FunctionResultContent Graduation It may require some changes although the current structure is good. ### Current - From a purity perspective the `Id` property can lead to confusion as it's not a response Id but a function call Id. - ctors have different `functionCall` and `functionCallContent` parameter names for same type. ```csharp public sealed class FunctionResultContent : KernelContent { public string? Id { get; } public string? PluginName { get; } public string? FunctionName { get; } public object? Result { get; } ctor(string? functionName = null, string? pluginName = null, string? id = null, object? result = null) ctor(FunctionCallContent functionCall, object? result = null) ctor(FunctionCallContent functionCallContent, FunctionResult result) } ``` ### Proposed - Option 1 - Rename `Id` to `CallId` to avoid confusion. - Adjust `ctor` parameters names. ```csharp public sealed class FunctionResultContent : KernelContent { public string? CallId { get; } public string? PluginName { get; } public string? FunctionName { get; } public object? Result { get; } ctor(string? functionName = null, string? pluginName = null, string? callId = null, object? result = null) ctor(FunctionCallContent functionCallContent, object? result = null) ctor(FunctionCallContent functionCallContent, FunctionResult functionResult) } ``` ### Proposed - Option 2 Use composition a have a dedicated CallContent within the `FunctionResultContent`. Pros: - `CallContent` has options to invoke a function again from its response which can be handy for some scenarios - Brings clarity from where the result came from and what is result specific data (root class). - Knowledge about the arguments used in the call. Cons: - Introduce one extra hop to get the `call` details from the result. ```csharp public sealed class FunctionResultContent : KernelContent { public FunctionCallContent CallContent { get; } public object? Result { get; } ctor(FunctionCallContent functionCallContent, object? result = null) ctor(FunctionCallContent functionCallContent, FunctionResult functionResult) } ``` ## FileReferenceContent + AnnotationContent Those two contents were added to `SemanticKernel.Abstractions` due to Serialization convenience but are very specific to **OpenAI Assistant API** and should be kept as Experimental for now. As a graduation those should be into `SemanticKernel.Agents.OpenAI` following the suggestion below. ```csharp #pragma warning disable SKEXP0110 [JsonDerivedType(typeof(AnnotationContent), typeDiscriminator: nameof(AnnotationContent))] [JsonDerivedType(typeof(FileReferenceContent), typeDiscriminator: nameof(FileReferenceContent))] #pragma warning disable SKEXP0110 public abstract class KernelContent { ... } ``` This coupling should not be encouraged for other packages that have `KernelContent` specializations. ### Solution - Usage of [JsonConverter](https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json/converters-how-to?pivots=dotnet-6-0#registration-sample---jsonconverter-on-a-type) Annotations Creation of a dedicated `JsonConverter` helper into the `Agents.OpenAI` project to handle the serialization and deserialization of those types. Annotate those Content types with `[JsonConverter(typeof(KernelContentConverter))]` attribute to indicate the `JsonConverter` to be used. ### Agents.OpenAI's JsonConverter Example ```csharp public class KernelContentConverter : JsonConverter { public override KernelContent Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) { using (var jsonDoc = JsonDocument.ParseValue(ref reader)) { var root = jsonDoc.RootElement; var typeDiscriminator = root.GetProperty("TypeDiscriminator").GetString(); switch (typeDiscriminator) { case nameof(AnnotationContent): return JsonSerializer.Deserialize(root.GetRawText(), options); case nameof(FileReferenceContent): return JsonSerializer.Deserialize(root.GetRawText(), options); default: throw new NotSupportedException($"Type discriminator '{typeDiscriminator}' is not supported."); } } } public override void Write(Utf8JsonWriter writer, KernelContent value, JsonSerializerOptions options) { JsonSerializer.Serialize(writer, value, value.GetType(), options); } } [JsonConverter(typeof(KernelContentConverter))] public class FileReferenceContent : KernelContent { public string FileId { get; init; } = string.Empty; ctor() ctor(string fileId, ...) } [JsonConverter(typeof(KernelContentConverter))] public class AnnotationContent : KernelContent { public string? FileId { get; init; } public string? Quote { get; init; } public int StartIndex { get; init; } public int EndIndex { get; init; } public ctor() public ctor(...) } ``` ## Decision Outcome - `BinaryContent`: Accepted. - `ImageContent`: Breaking change accepted with benefits using the `BinaryContent` specialization. No backwards compatibility as the current `ImageContent` behavior is undesirable. - `AudioContent`: Experimental breaking changes using the `BinaryContent` specialization. - `FunctionCallContent`: Graduate as is. - `FunctionResultContent`: Experimental breaking change from property `Id` to `CallId` to avoid confusion regarding being a function call Id or a response id. - `FileReferenceContent` and `AnnotationContent`: No changes, continue as experimental.