chore: import upstream snapshot with attribution

python/samples/concepts/realtime/README.md

@@ -0,0 +1,50 @@
+# Realtime Multi-modal API Samples
+
+These samples are more complex then most because of the nature of these API's. They are designed to be run in real-time and require a microphone and speaker to be connected to your computer.
+
+To run these samples, you will need to have the following setup:
+
+- Environment variables for OpenAI (websocket or WebRTC), with your key and OPENAI_REALTIME_MODEL_ID set.
+- Environment variables for Azure (websocket only), set with your endpoint, optionally a key and AZURE_OPENAI_REALTIME_DEPLOYMENT_NAME set. The API version needs to be at least `2025-08-28`.
+- To run the sample with a simple version of a class that handles the incoming and outgoing sound you need to install the following packages in your environment:
+  - semantic-kernel[realtime]
+  - pyaudio
+  - sounddevice
+  - pydub
+    e.g. pip install pyaudio sounddevice pydub semantic-kernel[realtime]
+
+The samples all run as python scripts, that can either be started directly or through your IDE.
+
+All demos have a similar output, where the instructions are printed, each new *response item* from the API is put into a new `Mosscap (transcript):` line. The nature of these api's is such that the transcript arrives before the spoken audio, so if you interrupt the audio the transcript will not match the audio.
+
+The realtime api's work by sending event from the server to you and sending events back to the server, this is fully asynchronous. The samples show you can listen to the events being sent by the server and some are handled by the code in the samples, others are not. For instance one could add a clause in the match case in the receive loop that logs the usage that is part of the `response.done` event.
+
+For more info on the events, go to our documentation, as well as the documentation of [OpenAI](https://platform.openai.com/docs/guides/realtime) and [Azure](https://learn.microsoft.com/en-us/azure/ai-services/openai/realtime-audio-quickstart?tabs=keyless%2Cmacos&pivots=programming-language-python).
+
+## Simple chat samples
+
+### [Simple chat with realtime websocket](./simple_realtime_chat_websocket.py)
+
+This sample uses the websocket api with Azure OpenAI to run a simple interaction based on voice. If you want to use this sample with OpenAI, just change AzureRealtimeWebsocket into OpenAIRealtimeWebsocket.
+
+### [Simple chat with realtime WebRTC](./simple_realtime_chat_webrtc.py)
+
+This sample uses the WebRTC api with OpenAI to run a simple interaction based on voice. Because of the way the WebRTC protocol works this needs a different player and recorder than the websocket version.
+
+## Function calling samples
+
+The following two samples use function calling with the following functions:
+
+- get_weather: This function will return the weather for a given city, it is randomly generated and not based on any real data.
+- get_time: This function will return the current time and date.
+- goodbye: This function will end the conversation.
+
+A line is logged whenever one of these functions is called.
+
+### [Chat with function calling Websocket](./realtime_agent_with_function_calling_websocket.py)
+
+This sample uses the websocket api with Azure OpenAI to run a voice agent, capable of taking actions on your behalf.
+
+### [Chat with function calling WebRTC](./realtime_agent_with_function_calling_webrtc.py)
+
+This sample uses the WebRTC api with OpenAI to run a voice agent, capable of taking actions on your behalf.

python/samples/concepts/realtime/realtime_agent_with_function_calling_webrtc.py

@@ -0,0 +1,148 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+import asyncio
+import logging
+from datetime import datetime
+from random import randint
+
+from samples.concepts.realtime.utils import AudioPlayerWebRTC, AudioRecorderWebRTC, check_audio_devices
+from semantic_kernel.connectors.ai import FunctionChoiceBehavior
+from semantic_kernel.connectors.ai.open_ai import (
+    AzureRealtimeWebRTC,
+    ListenEvents,
+    OpenAIRealtimeExecutionSettings,
+    TurnDetection,
+)
+from semantic_kernel.contents import ChatHistory, RealtimeTextEvent
+from semantic_kernel.functions import kernel_function
+
+logging.basicConfig(level=logging.WARNING)
+utils_log = logging.getLogger("samples.concepts.realtime.utils")
+utils_log.setLevel(logging.INFO)
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+"""
+This simple sample demonstrates how to use the OpenAI Realtime API to create
+a agent that can listen and respond directly through audio.
+It requires installing:
+- semantic-kernel
+- pyaudio
+- sounddevice
+- pydub
+e.g. pip install pyaudio sounddevice pydub semantic-kernel
+
+For more details of the exact setup, see the README.md in the realtime folder.
+"""
+
+# The characterics of your speaker and microphone are a big factor in a smooth conversation
+# so you may need to try out different devices for each.
+# you can also play around with the turn_detection settings to get the best results.
+# It has device id's set in the AudioRecorderStream and AudioPlayerAsync classes,
+# so you may need to adjust these for your system.
+# you can disable the check for available devices by commenting the line below
+check_audio_devices()
+
+
+class Helpers:
+    """A set of helper functions for the Realtime Agent."""
+
+    @kernel_function
+    def get_weather(self, location: str) -> str:
+        """Get the weather for a location."""
+        weather_conditions = ("sunny", "hot", "cloudy", "raining", "freezing", "snowing")
+        weather = weather_conditions[randint(0, len(weather_conditions) - 1)]  # nosec
+        logger.info(f"@ Getting weather for {location}: {weather}")
+        return f"The weather in {location} is {weather}."
+
+    @kernel_function
+    def get_date_time(self) -> str:
+        """Get the current date and time."""
+        logger.info("@ Getting current datetime")
+        return f"The current date and time is {datetime.now().isoformat()}."
+
+    @kernel_function
+    def goodbye(self):
+        """When the user is done, say goodbye and then call this function."""
+        logger.info("@ Goodbye has been called!")
+        raise KeyboardInterrupt
+
+
+async def main() -> None:
+    print_transcript = True
+    # create the audio player and audio track
+    # both take a device_id parameter, which is the index of the device to use,
+    # if None the default device will be used
+    audio_player = AudioPlayerWebRTC()
+
+    # create the realtime agent and optionally add the audio output function, this is optional
+    # and can also be passed in the receive method
+    # You can also pass in kernel, plugins, chat_history or settings here.
+    # For WebRTC the audio_track is required
+
+    # Note: api_version (either through settings or directly in the client) must be set to "2025-08-28"
+    # for Azure OpenAI deployments realtime deployments.
+    realtime_agent = AzureRealtimeWebRTC(
+        audio_track=AudioRecorderWebRTC(),
+        plugins=[Helpers()],
+    )
+
+    # Create the settings for the session
+    # The realtime api, does not use a system message, but takes instructions as a parameter for a session
+    # Another important setting is to tune the server_vad turn detection
+    # if this is turned off (by setting turn_detection=None), you will have to send
+    # the "input_audio_buffer.commit" and "response.create" event to the realtime api
+    # to signal the end of the user's turn and start the response.
+    # manual VAD is not part of this sample
+    # for more info: https://platform.openai.com/docs/api-reference/realtime-sessions/create#realtime-sessions-create-turn_detection
+    settings = OpenAIRealtimeExecutionSettings(
+        instructions="""
+    You are a chat bot. Your name is Mosscap and
+    you have one goal: figure out what people need.
+    Your full name, should you need to know it, is
+    Splendid Speckled Mosscap. You communicate
+    effectively, but you tend to answer with long
+    flowery prose.
+    """,
+        voice="alloy",
+        output_modalities=["text", "audio"],
+        turn_detection=TurnDetection(type="server_vad", create_response=True, silence_duration_ms=800, threshold=0.8),
+        function_choice_behavior=FunctionChoiceBehavior.Auto(),
+    )
+    # and we can add a chat history to conversation after starting it
+    chat_history = ChatHistory()
+    chat_history.add_user_message("Hi there, who are you?")
+    chat_history.add_assistant_message("I am Mosscap, a chat bot. I'm trying to figure out what people need.")
+
+    # the context manager calls the create_session method on the client and starts listening to the audio stream
+    async with (
+        realtime_agent(
+            settings=settings,
+            chat_history=chat_history,
+            create_response=True,
+        ),
+        audio_player,
+    ):
+        async for event in realtime_agent.receive(audio_output_callback=audio_player.client_callback):
+            match event:
+                case RealtimeTextEvent():
+                    if print_transcript:
+                        print(event.text.text, end="")
+                case _:
+                    # OpenAI Specific events
+                    match event.service_type:
+                        case ListenEvents.RESPONSE_CREATED:
+                            if print_transcript:
+                                print("\nMosscap (transcript): ", end="")
+                        case ListenEvents.ERROR:
+                            logger.error(event.service_event)
+
+
+if __name__ == "__main__":
+    print(
+        "Instructions: The model will start speaking immediately,"
+        "this can be turned off by removing `create_response=True` above."
+        "The model will detect when you stop and automatically generate a response. "
+        "Press ctrl + c to stop the program."
+    )
+    asyncio.run(main())

python/samples/concepts/realtime/realtime_agent_with_function_calling_websocket.py

@@ -0,0 +1,146 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+import asyncio
+import logging
+from datetime import datetime
+from random import randint
+
+from azure.identity import AzureCliCredential
+
+from samples.concepts.realtime.utils import AudioPlayerWebsocket, AudioRecorderWebsocket
+from semantic_kernel import Kernel
+from semantic_kernel.connectors.ai import FunctionChoiceBehavior
+from semantic_kernel.connectors.ai.open_ai import (
+    AzureRealtimeExecutionSettings,
+    AzureRealtimeWebsocket,
+    ListenEvents,
+    TurnDetection,
+)
+from semantic_kernel.contents import ChatHistory, RealtimeTextEvent
+from semantic_kernel.functions import kernel_function
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.DEBUG)
+
+"""
+This simple sample demonstrates how to use the OpenAI Realtime API to create
+a chat bot that can listen and respond directly through audio.
+It requires installing:
+- semantic-kernel[realtime]
+- pyaudio
+- sounddevice
+- pydub
+e.g. pip install pyaudio sounddevice pydub semantic-kernel[realtime]
+
+For more details of the exact setup, see the README.md in the realtime folder.
+"""
+
+
+@kernel_function
+def get_weather(location: str) -> str:
+    """Get the weather for a location."""
+    weather_conditions = ("sunny", "hot", "cloudy", "raining", "freezing", "snowing")
+    weather = weather_conditions[randint(0, len(weather_conditions) - 1)]  # nosec
+    logger.info(f"@ Getting weather for {location}: {weather}")
+    return f"The weather in {location} is {weather}."
+
+
+@kernel_function
+def get_date_time() -> str:
+    """Get the current date and time."""
+    logger.info("@ Getting current datetime")
+    return f"The current date and time is {datetime.now().isoformat()}."
+
+
+@kernel_function
+def goodbye():
+    """When the user is done, say goodbye and then call this function."""
+    logger.info("@ Goodbye has been called!")
+    raise KeyboardInterrupt
+
+
+async def main() -> None:
+    print_transcript = True
+    # create a Kernel and add a simple function for function calling.
+    kernel = Kernel()
+    kernel.add_functions(plugin_name="helpers", functions=[goodbye, get_weather, get_date_time])
+
+    # create the realtime agent, in this using Azure OpenAI through Websockets,
+    # there are also OpenAI Websocket and WebRTC clients
+    # See realtime_agent_with_function_calling_webrtc.py for an example of the WebRTC client
+    realtime_agent = AzureRealtimeWebsocket(credential=AzureCliCredential())
+    # create the audio player and audio track
+    # both take a device_id parameter, which is the index of the device to use, if None the default device is used
+    audio_player = AudioPlayerWebsocket()
+    audio_recorder = AudioRecorderWebsocket(realtime_client=realtime_agent)
+
+    # Create the settings for the session
+    # The realtime api, does not use a system message, but, like agents, takes instructions as a parameter for a session
+    # Another important setting is to tune the server_vad turn detection
+    # if this is turned off (by setting turn_detection=None), you will have to send
+    # the "input_audio_buffer.commit" and "response.create" event to the realtime api
+    # to signal the end of the user's turn and start the response.
+    # manual VAD is not part of this sample
+    # for more info: https://platform.openai.com/docs/api-reference/realtime-sessions/create#realtime-sessions-create-turn_detection
+
+    # Note: api_version (either through settings or directly in the client) must be set to "2025-08-28"
+    # for Azure OpenAI deployments realtime deployments.
+    settings = AzureRealtimeExecutionSettings(
+        instructions="""
+    You are a chat bot. Your name is Mosscap and
+    you have one goal: figure out what people need.
+    Your full name, should you need to know it, is
+    Splendid Speckled Mosscap. You communicate
+    effectively, but you tend to answer with long
+    flowery prose.
+    """,
+        # see https://platform.openai.com/docs/api-reference/realtime-sessions/create#realtime-sessions-create-voice for the full list of voices # noqa: E501
+        voice="alloy",
+        turn_detection=TurnDetection(type="server_vad", create_response=True, silence_duration_ms=800, threshold=0.8),
+        function_choice_behavior=FunctionChoiceBehavior.Auto(),
+    )
+    # and we can add a chat history to conversation to seed the conversation
+    chat_history = ChatHistory()
+    chat_history.add_user_message("Hi there, I'm based in Amsterdam.")
+    chat_history.add_assistant_message(
+        "I am Mosscap, a chat bot. I'm trying to figure out what people need, "
+        "I can tell you what the weather is or the time."
+    )
+
+    # the context manager calls the create_session method on the agent and starts listening to the audio stream
+    async with (
+        audio_recorder,
+        realtime_agent(
+            settings=settings,
+            chat_history=chat_history,
+            kernel=kernel,
+            create_response=True,
+        ),
+        audio_player,
+    ):
+        # the audio_output_callback can be added here or in the constructor
+        # using this gives the smoothest experience
+        async for event in realtime_agent.receive(audio_output_callback=audio_player.client_callback):
+            match event:
+                case RealtimeTextEvent():
+                    if print_transcript:
+                        print(event.text.text, end="")
+                case _:
+                    # OpenAI Specific events
+                    match event.service_type:
+                        case ListenEvents.RESPONSE_CREATED:
+                            if print_transcript:
+                                print("\nMosscap (transcript): ", end="")
+                        case ListenEvents.ERROR:
+                            print(event.service_event)
+                            logger.error(event.service_event)
+
+
+if __name__ == "__main__":
+    print(
+        "Instructions: The agent will start speaking immediately,"
+        "this can be turned off by removing `create_response=True` above."
+        "The model will detect when you stop and automatically generate a response. "
+        "Press ctrl + c to stop the program."
+    )
+    asyncio.run(main())

python/samples/concepts/realtime/simple_realtime_chat_webrtc.py

@@ -0,0 +1,96 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+import asyncio
+import logging
+
+from samples.concepts.realtime.utils import AudioPlayerWebRTC, AudioRecorderWebRTC, check_audio_devices
+from semantic_kernel.connectors.ai.open_ai import (
+    AzureRealtimeExecutionSettings,
+    ListenEvents,
+)
+from semantic_kernel.connectors.ai.open_ai.services.azure_realtime import AzureRealtimeWebRTC
+from semantic_kernel.contents import RealtimeTextEvent
+
+logging.basicConfig(level=logging.WARNING)
+utils_log = logging.getLogger("samples.concepts.realtime.utils")
+utils_log.setLevel(logging.INFO)
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+"""
+This simple sample demonstrates how to use the OpenAI Realtime API to create
+a chat bot that can listen and respond directly through audio.
+It requires installing:
+- semantic-kernel[realtime]
+- pyaudio
+- sounddevice
+- pydub
+e.g. pip install pyaudio sounddevice pydub semantic-kernel[realtime]
+
+For more details of the exact setup, see the README.md in the realtime folder.
+"""
+
+# The characteristics of your speaker and microphone are a big factor in a smooth conversation
+# so you may need to try out different devices for each.
+# you can also play around with the turn_detection settings to get the best results.
+# It has device id's set in the AudioRecorderStream and AudioPlayerAsync classes,
+# so you may need to adjust these for your system.
+# you can disable the check for available devices by commenting the line below
+check_audio_devices()
+
+
+async def main() -> None:
+    # create the realtime client and optionally add the audio output function, this is optional
+    # you can define the protocol to use, either "websocket" or "webrtc"
+    # they will behave the same way, even though the underlying protocol is quite different
+    settings = AzureRealtimeExecutionSettings(
+        instructions="""
+    You are a chat bot. Your name is Mosscap and
+    you have one goal: figure out what people need.
+    Your full name, should you need to know it, is
+    Splendid Speckled Mosscap. You communicate
+    effectively, but you tend to answer with long
+    flowery prose.
+    """,
+        # there are different voices to choose from, since that list is bound to change, it is not checked beforehand,
+        # see https://platform.openai.com/docs/api-reference/realtime-sessions/create#realtime-sessions-create-voice
+        # for more details.
+        voice="alloy",
+        # Enable both text and audio output to get transcripts
+        output_modalities=["text", "audio"],
+    )
+    # Note: api_version (either through settings or directly in the client) must be set to "2025-08-28"
+    # for Azure OpenAI deployments realtime deployments.
+    realtime_client = AzureRealtimeWebRTC(
+        audio_track=AudioRecorderWebRTC(),
+        settings=settings,
+    )
+    # Create the settings for the session
+    audio_player = AudioPlayerWebRTC()
+    # the context manager calls the create_session method on the client and starts listening to the audio stream
+    async with audio_player, realtime_client:
+        async for event in realtime_client.receive(audio_output_callback=audio_player.client_callback):
+            match event:
+                case RealtimeTextEvent():
+                    # Only process delta events for streaming, skip done events to avoid duplication
+                    if event.service_type and "delta" in event.service_type and event.text.text:
+                        print(event.text.text, end="", flush=True)
+                    # Add newline when transcript is complete (done event)
+                    elif event.service_type and "done" in event.service_type:
+                        print()  # Add newline for readability
+                case _:
+                    # Handle service events
+                    if event.event_type == "service" and event.service_type:
+                        if event.service_type == ListenEvents.SESSION_UPDATED:
+                            print("Session updated")
+                        elif event.service_type == ListenEvents.RESPONSE_CREATED:
+                            print("\nMosscap (transcript): ", end="")
+
+
+if __name__ == "__main__":
+    print(
+        "Instructions: start speaking when you see 'Session updated.' "
+        "The model will detect when you stop and automatically start responding. "
+        "Press ctrl + c to stop the program."
+    )
+    asyncio.run(main())

python/samples/concepts/realtime/simple_realtime_chat_websocket.py

@@ -0,0 +1,93 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+import asyncio
+import logging
+
+from samples.concepts.realtime.utils import AudioPlayerWebsocket, AudioRecorderWebsocket, check_audio_devices
+from semantic_kernel.connectors.ai.open_ai import (
+    AzureRealtimeExecutionSettings,
+    AzureRealtimeWebsocket,
+    ListenEvents,
+)
+from semantic_kernel.contents import RealtimeAudioEvent, RealtimeTextEvent
+
+logging.basicConfig(level=logging.WARNING)
+utils_log = logging.getLogger("samples.concepts.realtime.utils")
+utils_log.setLevel(logging.INFO)
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+"""
+This simple sample demonstrates how to use the OpenAI Realtime API to create
+a chat bot that can listen and respond directly through audio.
+It requires installing:
+- semantic-kernel[realtime]
+- pyaudio
+- sounddevice
+- pydub
+e.g. pip install pyaudio sounddevice pydub semantic-kernel[realtime]
+
+For more details of the exact setup, see the README.md in the realtime folder.
+"""
+
+# The characterics of your speaker and microphone are a big factor in a smooth conversation
+# so you may need to try out different devices for each.
+# you can also play around with the turn_detection settings to get the best results.
+# It has device id's set in the AudioRecorderStream and AudioPlayerAsync classes,
+# so you may need to adjust these for your system.
+# you can disable the check for available devices by commenting the line below
+check_audio_devices()
+
+
+async def main() -> None:
+    # create the realtime client and optionally add the audio output function, this is optional
+    # you can define the protocol to use, either "websocket" or "webrtc"
+    # they will behave the same way, even though the underlying protocol is quite different
+    settings = AzureRealtimeExecutionSettings(
+        instructions="""
+    You are a chat bot. Your name is Mosscap and
+    you have one goal: figure out what people need.
+    Your full name, should you need to know it, is
+    Splendid Speckled Mosscap. You communicate
+    effectively, but you tend to answer with long
+    flowery prose.
+    """,
+        # there are different voices to choose from, since that list is bound to change, it is not checked beforehand,
+        # see https://platform.openai.com/docs/api-reference/realtime-sessions/create#realtime-sessions-create-voice
+        # for more details.
+        voice="shimmer",
+    )
+    # Note: api_version (either through settings or directly in the client) must be set to "2025-08-28"
+    # for Azure OpenAI deployments realtime deployments.
+    realtime_client = AzureRealtimeWebsocket(
+        settings=settings,
+    )
+    audio_player = AudioPlayerWebsocket()
+    audio_recorder = AudioRecorderWebsocket(realtime_client=realtime_client)
+    # Create the settings for the session
+    # the context manager calls the create_session method on the client and starts listening to the audio stream
+    async with audio_player, audio_recorder, realtime_client:
+        async for event in realtime_client.receive():
+            match event:
+                # this can be used as an alternative to the callback function used in other samples,
+                # the callback is faster and smoother
+                case RealtimeAudioEvent():
+                    await audio_player.add_audio(event.audio)
+                case RealtimeTextEvent():
+                    # the model returns both audio and transcript of the audio, which we will print
+                    print(event.text.text, end="")
+                case _:
+                    # OpenAI Specific events
+                    if event.service_type == ListenEvents.SESSION_UPDATED:
+                        print("Session updated")
+                    if event.service_type == ListenEvents.RESPONSE_CREATED:
+                        print("\nMosscap (transcript): ", end="")
+
+
+if __name__ == "__main__":
+    print(
+        "Instructions: Start speaking when you see 'Session updated.' "
+        "The model will detect when you stop and automatically start responding. "
+        "Press ctrl + c to stop the program."
+    )
+    asyncio.run(main())

python/samples/concepts/realtime/utils.py

@@ -0,0 +1,489 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+import asyncio
+import base64
+import logging
+import threading
+from typing import Any, ClassVar, Final, cast
+
+import numpy as np
+import numpy.typing as npt
+import sounddevice as sd
+from aiortc.mediastreams import MediaStreamError, MediaStreamTrack
+from av.audio.frame import AudioFrame
+from av.frame import Frame
+from pydantic import BaseModel, ConfigDict, PrivateAttr
+from sounddevice import InputStream, OutputStream
+
+from semantic_kernel.connectors.ai.realtime_client_base import RealtimeClientBase
+from semantic_kernel.contents import AudioContent, RealtimeAudioEvent
+
+logger = logging.getLogger(__name__)
+
+SAMPLE_RATE: Final[int] = 24000
+RECORDER_CHANNELS: Final[int] = 1
+PLAYER_CHANNELS: Final[int] = 1
+FRAME_DURATION: Final[int] = 100
+SAMPLE_RATE_WEBRTC: Final[int] = 48000
+RECORDER_CHANNELS_WEBRTC: Final[int] = 1
+PLAYER_CHANNELS_WEBRTC: Final[int] = 2
+FRAME_DURATION_WEBRTC: Final[int] = 20
+DTYPE: Final[npt.DTypeLike] = np.int16
+
+
+def check_audio_devices():
+    logger.info(sd.query_devices())
+
+
+# region: Recorders
+
+
+class AudioRecorderWebRTC(BaseModel, MediaStreamTrack):
+    """A simple class that implements the WebRTC MediaStreamTrack for audio from sounddevice.
+
+    This class is meant as a demo sample and is not meant for production use.
+    """
+
+    model_config = ConfigDict(populate_by_name=True, arbitrary_types_allowed=True, validate_assignment=True)
+
+    kind: ClassVar[str] = "audio"
+    device: str | int | None = None
+    sample_rate: int
+    channels: int
+    frame_duration: int
+    dtype: npt.DTypeLike = DTYPE
+    frame_size: int = 0
+    _queue: asyncio.Queue[Frame] = PrivateAttr(default_factory=asyncio.Queue)
+    _is_recording: bool = False
+    _stream: InputStream | None = None
+    _recording_task: asyncio.Task | None = None
+    _loop: asyncio.AbstractEventLoop | None = None
+    _pts: int = 0
+
+    def __init__(
+        self,
+        *,
+        device: str | int | None = None,
+        sample_rate: int = SAMPLE_RATE_WEBRTC,
+        channels: int = RECORDER_CHANNELS_WEBRTC,
+        frame_duration: int = FRAME_DURATION_WEBRTC,
+        dtype: npt.DTypeLike = DTYPE,
+    ):
+        """A simple class that implements the WebRTC MediaStreamTrack for audio from sounddevice.
+
+        Make sure the device is set to the correct device for your system.
+
+        Args:
+            device: The device id to use for recording audio.
+            sample_rate: The sample rate for the audio.
+            channels: The number of channels for the audio.
+            frame_duration: The duration of each audio frame in milliseconds.
+            dtype: The data type for the audio.
+        """
+        super().__init__(**{
+            "device": device,
+            "sample_rate": sample_rate,
+            "channels": channels,
+            "frame_duration": frame_duration,
+            "dtype": dtype,
+            "frame_size": int(sample_rate * frame_duration / 1000),
+        })
+        MediaStreamTrack.__init__(self)
+
+    async def recv(self) -> Frame:
+        """Receive the next frame of audio data."""
+        if not self._recording_task:
+            self._recording_task = asyncio.create_task(self.start_recording())
+
+        try:
+            frame = await self._queue.get()
+            self._queue.task_done()
+            return frame
+        except Exception as e:
+            logger.error(f"Error receiving audio frame: {e!s}")
+            raise MediaStreamError("Failed to receive audio frame")
+
+    def _sounddevice_callback(self, indata: np.ndarray, frames: int, time: Any, status: Any) -> None:
+        if status:
+            logger.warning(f"Audio input status: {status}")
+        if self._loop and self._loop.is_running():
+            asyncio.run_coroutine_threadsafe(self._queue.put(self._create_frame(indata)), self._loop)
+
+    def _create_frame(self, indata: np.ndarray) -> Frame:
+        audio_data = indata.copy()
+        if audio_data.dtype != self.dtype:
+            audio_data = (
+                (audio_data * 32767).astype(self.dtype) if self.dtype == np.int16 else audio_data.astype(self.dtype)
+            )
+        frame = AudioFrame(
+            format="s16",
+            layout="mono",
+            samples=len(audio_data),
+        )
+        frame.rate = self.sample_rate
+        frame.pts = self._pts
+        frame.planes[0].update(audio_data.tobytes())
+        self._pts += len(audio_data)
+        return frame
+
+    async def start_recording(self):
+        """Start recording audio from the input device."""
+        if self._is_recording:
+            return
+
+        self._is_recording = True
+        self._loop = asyncio.get_running_loop()
+        self._pts = 0  # Reset pts when starting recording
+
+        try:
+            self._stream = InputStream(
+                device=self.device,
+                channels=self.channels,
+                samplerate=self.sample_rate,
+                dtype=self.dtype,
+                blocksize=self.frame_size,
+                callback=self._sounddevice_callback,
+            )
+            self._stream.start()
+
+            while self._is_recording:
+                await asyncio.sleep(0.1)
+        except asyncio.CancelledError:
+            logger.debug("Recording task was stopped.")
+        except KeyboardInterrupt:
+            logger.debug("Recording task was stopped.")
+        except Exception as e:
+            logger.error(f"Error in audio recording: {e!s}")
+            raise
+        finally:
+            self._is_recording = False
+
+
+class AudioRecorderWebsocket(BaseModel):
+    """A simple class that implements a sounddevice for use with websockets.
+
+    This class is meant as a demo sample and is not meant for production use.
+    """
+
+    model_config = ConfigDict(populate_by_name=True, arbitrary_types_allowed=True, validate_assignment=True)
+
+    realtime_client: RealtimeClientBase
+    device: str | int | None = None
+    sample_rate: int
+    channels: int
+    frame_duration: int
+    dtype: npt.DTypeLike = DTYPE
+    frame_size: int = 0
+    _stream: InputStream | None = None
+    _pts: int = 0
+    _stream_task: asyncio.Task | None = None
+
+    def __init__(
+        self,
+        *,
+        realtime_client: RealtimeClientBase,
+        device: str | int | None = None,
+        sample_rate: int = SAMPLE_RATE,
+        channels: int = RECORDER_CHANNELS,
+        frame_duration: int = FRAME_DURATION,
+        dtype: npt.DTypeLike = DTYPE,
+    ):
+        """A simple class that implements the WebRTC MediaStreamTrack for audio from sounddevice.
+
+        Make sure the device is set to the correct device for your system.
+
+        Args:
+            realtime_client: The RealtimeClientBase to use for streaming audio.
+            device: The device id to use for recording audio.
+            sample_rate: The sample rate for the audio.
+            channels: The number of channels for the audio.
+            frame_duration: The duration of each audio frame in milliseconds.
+            dtype: The data type for the audio.
+            **kwargs: Additional keyword arguments.
+        """
+        super().__init__(**{
+            "realtime_client": realtime_client,
+            "device": device,
+            "sample_rate": sample_rate,
+            "channels": channels,
+            "frame_duration": frame_duration,
+            "dtype": dtype,
+            "frame_size": int(sample_rate * frame_duration / 1000),
+        })
+
+    async def __aenter__(self):
+        """Stream audio data to a RealtimeClientBase."""
+        if not self._stream_task:
+            self._stream_task = asyncio.create_task(self._start_stream())
+        return self
+
+    async def _start_stream(self):
+        self._pts = 0  # Reset pts when starting recording
+        self._stream = InputStream(
+            device=self.device,
+            channels=self.channels,
+            samplerate=self.sample_rate,
+            dtype=self.dtype,
+            blocksize=self.frame_size,
+        )
+        self._stream.start()
+        try:
+            while True:
+                if self._stream.read_available < self.frame_size:
+                    await asyncio.sleep(0)
+                    continue
+                data, _ = self._stream.read(self.frame_size)
+
+                await self.realtime_client.send(
+                    RealtimeAudioEvent(audio=AudioContent(data=base64.b64encode(cast(Any, data)).decode("utf-8")))
+                )
+
+                await asyncio.sleep(0)
+        except asyncio.CancelledError:
+            pass
+
+    async def __aexit__(self, exc_type, exc, tb):
+        """Stop recording audio."""
+        if self._stream_task:
+            self._stream_task.cancel()
+            await self._stream_task
+        if self._stream:
+            self._stream.stop()
+            self._stream.close()
+
+
+# region: Players
+
+
+class AudioPlayerWebRTC(BaseModel):
+    """Simple class that plays audio using sounddevice.
+
+    This class is meant as a demo sample and is not meant for production use.
+
+    Make sure the device_id is set to the correct device for your system.
+
+    The sample rate, channels and frame duration
+    should be set to match the audio you
+    are receiving.
+
+    Args:
+        device: The device id to use for playing audio.
+        sample_rate: The sample rate for the audio.
+        channels: The number of channels for the audio.
+        dtype: The data type for the audio.
+        frame_duration: The duration of each audio frame in milliseconds
+
+    """
+
+    model_config = ConfigDict(populate_by_name=True, arbitrary_types_allowed=True, validate_assignment=True)
+
+    device: int | None = None
+    sample_rate: int = SAMPLE_RATE_WEBRTC
+    channels: int = PLAYER_CHANNELS_WEBRTC
+    dtype: npt.DTypeLike = DTYPE
+    frame_duration: int = FRAME_DURATION_WEBRTC
+    _queue: asyncio.Queue[np.ndarray] | None = PrivateAttr(default=None)
+    _stream: OutputStream | None = PrivateAttr(default=None)
+
+    async def __aenter__(self):
+        """Start the audio stream when entering a context."""
+        self.start()
+        return self
+
+    async def __aexit__(self, exc_type, exc, tb):
+        """Stop the audio stream when exiting a context."""
+        self.stop()
+
+    def start(self):
+        """Start the audio stream."""
+        self._queue = asyncio.Queue()
+        self._stream = OutputStream(
+            callback=self._sounddevice_callback,
+            samplerate=self.sample_rate,
+            channels=self.channels,
+            dtype=self.dtype,
+            blocksize=int(self.sample_rate * self.frame_duration / 1000),
+            device=self.device,
+        )
+        if self._stream and self._queue:
+            self._stream.start()
+
+    def stop(self):
+        """Stop the audio stream."""
+        if self._stream:
+            self._stream.stop()
+        self._stream = None
+        self._queue = None
+
+    def _sounddevice_callback(self, outdata, frames, time, status):
+        """This callback is called by sounddevice when it needs more audio data to play."""
+        if status:
+            logger.debug(f"Audio output status: {status}")
+        if self._queue:
+            if self._queue.empty():
+                outdata[:] = 0
+                return
+            data = self._queue.get_nowait()
+            outdata[:] = data.reshape(outdata.shape)
+            self._queue.task_done()
+        else:
+            logger.error(
+                "Audio queue not initialized, make sure to call start before "
+                "using the player, or use the context manager."
+            )
+
+    async def client_callback(self, content: np.ndarray):
+        """This function can be passed to the audio_output_callback field of the RealtimeClientBase."""
+        if self._queue:
+            await self._queue.put(content)
+        else:
+            logger.error(
+                "Audio queue not initialized, make sure to call start before "
+                "using the player, or use the context manager."
+            )
+
+    async def add_audio(self, audio_content: AudioContent) -> None:
+        """This function is used to add audio to the queue for playing.
+
+        It first checks if there is a AudioFrame in the inner_content of the AudioContent.
+        If not, it checks if the data is a numpy array, bytes, or a string and converts it to a numpy array.
+        """
+        if not self._queue:
+            logger.error(
+                "Audio queue not initialized, make sure to call start before "
+                "using the player, or use the context manager."
+            )
+            return
+        if audio_content.inner_content and isinstance(audio_content.inner_content, AudioFrame):
+            await self._queue.put(audio_content.inner_content.to_ndarray())
+            return
+        if isinstance(audio_content.data, np.ndarray):
+            await self._queue.put(audio_content.data)
+            return
+        if isinstance(audio_content.data, bytes):
+            await self._queue.put(np.frombuffer(audio_content.data, dtype=self.dtype))
+            return
+        if isinstance(audio_content.data, str):
+            await self._queue.put(np.frombuffer(audio_content.data.encode(), dtype=self.dtype))
+            return
+        logger.error(f"Unknown audio content: {audio_content}")
+
+
+class AudioPlayerWebsocket(BaseModel):
+    """Simple class that plays audio using sounddevice.
+
+    This class is meant as a demo sample and is not meant for production use.
+
+    Make sure the device_id is set to the correct device for your system.
+
+    The sample rate, channels and frame duration
+    should be set to match the audio you
+    are receiving.
+
+    Args:
+        device: The device id to use for playing audio.
+        sample_rate: The sample rate for the audio.
+        channels: The number of channels for the audio.
+        dtype: The data type for the audio.
+        frame_duration: The duration of each audio frame in milliseconds
+
+    """
+
+    model_config = ConfigDict(populate_by_name=True, arbitrary_types_allowed=True, validate_assignment=True)
+
+    device: int | None = None
+    sample_rate: int = SAMPLE_RATE
+    channels: int = PLAYER_CHANNELS
+    dtype: npt.DTypeLike = DTYPE
+    frame_duration: int = FRAME_DURATION
+    _lock: Any = PrivateAttr(default_factory=threading.Lock)
+    _queue: list[np.ndarray] = PrivateAttr(default_factory=list)
+    _stream: OutputStream | None = PrivateAttr(default=None)
+    _frame_count: int = 0
+
+    async def __aenter__(self):
+        """Start the audio stream when entering a context."""
+        self.start()
+        return self
+
+    async def __aexit__(self, exc_type, exc, tb):
+        """Stop the audio stream when exiting a context."""
+        self.stop()
+
+    def start(self):
+        """Start the audio stream."""
+        with self._lock:
+            self._queue = []
+        self._stream = OutputStream(
+            callback=self._sounddevice_callback,
+            samplerate=self.sample_rate,
+            channels=self.channels,
+            dtype=self.dtype,
+            blocksize=int(self.sample_rate * self.frame_duration / 1000),
+            device=self.device,
+        )
+        if self._stream:
+            self._stream.start()
+
+    def stop(self):
+        """Stop the audio stream."""
+        if self._stream:
+            self._stream.stop()
+        self._stream = None
+        with self._lock:
+            self._queue = []
+
+    def _sounddevice_callback(self, outdata, frames, time, status):
+        """This callback is called by sounddevice when it needs more audio data to play."""
+        with self._lock:
+            if status:
+                logger.debug(f"Audio output status: {status}")
+            data = np.empty(0, dtype=np.int16)
+
+            # get next item from queue if there is still space in the buffer
+            while len(data) < frames and len(self._queue) > 0:
+                item = self._queue.pop(0)
+                frames_needed = frames - len(data)
+                data = np.concatenate((data, item[:frames_needed]))
+                if len(item) > frames_needed:
+                    self._queue.insert(0, item[frames_needed:])
+
+            self._frame_count += len(data)
+
+            # fill the rest of the frames with zeros if there is no more data
+            if len(data) < frames:
+                data = np.concatenate((data, np.zeros(frames - len(data), dtype=np.int16)))
+
+        outdata[:] = data.reshape(-1, 1)
+
+    def reset_frame_count(self):
+        self._frame_count = 0
+
+    def get_frame_count(self):
+        return self._frame_count
+
+    async def client_callback(self, content: np.ndarray):
+        """This function can be passed to the audio_output_callback field of the RealtimeClientBase."""
+        with self._lock:
+            self._queue.append(content)
+
+    async def add_audio(self, audio_content: AudioContent) -> None:
+        """This function is used to add audio to the queue for playing.
+
+        It first checks if there is a AudioFrame in the inner_content of the AudioContent.
+        If not, it checks if the data is a numpy array, bytes, or a string and converts it to a numpy array.
+        """
+        with self._lock:
+            if audio_content.inner_content and isinstance(audio_content.inner_content, AudioFrame):
+                self._queue.append(audio_content.inner_content.to_ndarray())
+                return
+            if isinstance(audio_content.data, np.ndarray):
+                self._queue.append(audio_content.data)
+                return
+            if isinstance(audio_content.data, bytes):
+                self._queue.append(np.frombuffer(audio_content.data, dtype=self.dtype))
+                return
+            if isinstance(audio_content.data, str):
+                self._queue.append(np.frombuffer(audio_content.data.encode(), dtype=self.dtype))
+                return
+            logger.error(f"Unknown audio content: {audio_content}")