wehub/googlecloudplatform--generative-ai
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:30:30 +08:00
commit 914fea506e
2793 changed files with 802106 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
audio/speech/README.md
+14
View File
@@ -0,0 +1,14 @@
# Speech Recognition and Generation
This repository explores various use-cases and implementations of speech recognition and generation technologies with Google Cloud Speech-to-Text and Text-to-Speech
## Contents
Here's a breakdown of the content available:
- **Getting Started:**
- [get_started_with_chirp_3.ipynb](getting-started/get_started_with_chirp_3.ipynb): This notebook provides a basic introduction to speech synthesis using Chirp 3 with Google Cloud Text-to-Speech API.
- [get_started_with_chirp_2_sdk.ipynb](getting-started/get_started_with_chirp_2_sdk.ipynb): This Jupyter Notebook provides a basic introduction to performing speech recognition using the Chirp 2 model with Google Cloud Speech-to-Text API.
- **Use Cases:**
- **Storytelling:**
- [storytelling.ipynb](use-cases/storytelling/storytelling.ipynb): This notebook delves into a specific application of speech technology - crafting engaging stories. It likely leverages both speech recognition and generation to create interactive or automated storytelling experiences.
audio/speech/getting-started/gemini_3_1_flash_tts.ipynb
+793
View File
@@ -0,0 +1,793 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NcIyYxlxZ7AW"
},
"outputs": [],
"source": [
"# Copyright 2026 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0sB0minZaO0T"
},
"source": [
"# Gemini 3.1 Flash Text-to-Speech Generation\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/gemini_3_1_flash_tts.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Faudio%2Fspeech%2Fgetting-started%2Fgemini_3_1_flash_tts.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/audio/speech/getting-started/gemini_3_1_flash_tts.ipynb\">\n",
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/gemini_3_1_flash_tts.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<p>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/gemini_3_1_flash_tts.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/gemini_3_1_flash_tts.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/gemini_3_1_flash_tts.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/gemini_3_1_flash_tts.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/gemini_3_1_flash_tts.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2XpbkyEbb-U3"
},
"source": [
"| Author |\n",
"| --- |\n",
"| [Katie Nguyen](https://github.com/katiemn) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PM7AnicdcFg0"
},
"source": [
"## Overview\n",
"\n",
"### Gemini 3.1 Flash TTS\n",
"\n",
"Gemini 3.1 Flash TTS on Agent Platform gives application developers access to Google's cutting-edge speech generation. This model provides powerful, low-latency speech generation with expressive audio tags for carefully crafted narration.\n",
"\n",
"In this tutorial, you will learn how to use the Google Gen AI SDK for Python to interact with Gemini 3.1 Flash TTS to:\n",
"- Convert text to an audio output with prebuilt voices\n",
"- Generate speech clips in different languages\n",
"- Define multiple speakers\n",
"- Convert text to speech with audio tags"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DvNFe-xxsced"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gyIP8rb3sgOR"
},
"source": [
"### Install Google Gen AI SDK for Python"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "JLl1jyeJsOPp"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-genai"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vy23JbZps1Jm"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you are running this notebook on Google Colab, run the following cell to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "svjU0gJVsdzu"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dL81amVRtI89"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"id": "Gfo9egZHshrZ"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"import numpy as np\n",
"from IPython.display import Audio, Markdown, display\n",
"from google import genai\n",
"from google.genai import types"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wikdtLdjtVQT"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Agent Platform, you must have an existing Google Cloud project and [enable the Agent Platform API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project](https://docs.cloud.google.com/resource-manager/docs/creating-managing-projects) and a [development environment](https://cloud.google.com/docs/authentication/set-up-adc-local-dev-environment)."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"id": "hqwZDw6Esj3z"
},
"outputs": [],
"source": [
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"global\")\n",
"\n",
"client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Mfb2XfgYuV7c"
},
"source": [
"### Define a helper function to play audio"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"id": "Ka_sWHxbsmfb"
},
"outputs": [],
"source": [
"def play_audio(response):\n",
" audio_bytes = response.candidates[0].content.parts[0].inline_data.data\n",
" audio_array = np.frombuffer(audio_bytes, dtype=\"<i2\")\n",
" display(Audio(data=audio_array, rate=24000))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-67nb2b_x8eP"
},
"source": [
"### Load the models"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"id": "6KUUn1chsow0"
},
"outputs": [],
"source": [
"speech_model = \"gemini-3.1-flash-tts-preview\"\n",
"gemini_model = \"gemini-3.5-flash\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "O2TiFc8O4JtP"
},
"source": [
"## Convert Text-to-Speech\n",
"\n",
"When converting text-to-speech, you can choose from a list of [voice options](https://docs.cloud.google.com/text-to-speech/docs/gemini-tts#voice_options), which you'll set through the `VoiceConfig`.\n",
"\n",
"By default, all audio generated with Gemini 3.1 Flash TTS utilizes [SynthID](https://deepmind.google/technologies/synthid/)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "wApkg_7OAY-h"
},
"outputs": [],
"source": [
"prompt = \"\"\"\n",
"Hey there! How are you doing today?\n",
"\"\"\"\n",
"\n",
"response = client.models.generate_content(\n",
" model=speech_model,\n",
" contents=prompt,\n",
" config=types.GenerateContentConfig(\n",
" response_modalities=[\"AUDIO\"],\n",
" speech_config=types.SpeechConfig(\n",
" voice_config=types.VoiceConfig(\n",
" prebuilt_voice_config=types.PrebuiltVoiceConfig(voice_name=\"Fenrir\")\n",
" )\n",
" ),\n",
" ),\n",
")\n",
"\n",
"play_audio(response)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "n3aRIjeNnr7h"
},
"source": [
"## Generate speech in different languages\n",
"\n",
"The TTS models detect the input language automatically. Check the documentation to learn more about the list of [supported languages](https://docs.cloud.google.com/text-to-speech/docs/gemini-tts#available_languages)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "kQryZG6RZrZl"
},
"outputs": [],
"source": [
"prompt = \"\"\"\n",
"¡Hola! ¿Quieres hacer algo hoy?\n",
"\"\"\"\n",
"\n",
"response = client.models.generate_content(\n",
" model=speech_model,\n",
" contents=prompt,\n",
" config=types.GenerateContentConfig(\n",
" response_modalities=[\"AUDIO\"],\n",
" speech_config=types.SpeechConfig(\n",
" voice_config=types.VoiceConfig(\n",
" prebuilt_voice_config=types.PrebuiltVoiceConfig(voice_name=\"Despina\")\n",
" )\n",
" ),\n",
" ),\n",
")\n",
"\n",
"play_audio(response)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZrQLAflLolcc"
},
"source": [
"## Define multiple speakers\n",
"\n",
"For multi-speaker audio clips, you'll need to include a `MultiSpeakerVoiceConfig` in your request. You can include up to 2 different speakers, each with their own `SpeakerVoiceConfig` that includes the `speaker` name and `voice_name`. Make sure the `speaker` name is the same one used in the prompt."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_K_aio7ouYP4"
},
"outputs": [],
"source": [
"prompt = \"\"\"\n",
"Ryan: How are you doing today Katie?\n",
"Katie: Not too bad, how about you?\n",
"\"\"\"\n",
"\n",
"response = client.models.generate_content(\n",
" model=speech_model,\n",
" contents=prompt,\n",
" config=types.GenerateContentConfig(\n",
" response_modalities=[\"AUDIO\"],\n",
" speech_config=types.SpeechConfig(\n",
" multi_speaker_voice_config=types.MultiSpeakerVoiceConfig(\n",
" speaker_voice_configs=[\n",
" types.SpeakerVoiceConfig(\n",
" speaker=\"Ryan\",\n",
" voice_config=types.VoiceConfig(\n",
" prebuilt_voice_config=types.PrebuiltVoiceConfig(\n",
" voice_name=\"Umbriel\",\n",
" )\n",
" ),\n",
" ),\n",
" types.SpeakerVoiceConfig(\n",
" speaker=\"Katie\",\n",
" voice_config=types.VoiceConfig(\n",
" prebuilt_voice_config=types.PrebuiltVoiceConfig(\n",
" voice_name=\"Leda\",\n",
" )\n",
" ),\n",
" ),\n",
" ]\n",
" )\n",
" ),\n",
" ),\n",
")\n",
"\n",
"play_audio(response)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZKm_x60Sr5cp"
},
"source": [
"## Prompting with audio tags\n",
"\n",
"Audio tags let you insert modifiers directly into your prompt to help control speaker style, tone, or delivery. An audio tag is a word in square brackets that specifies how the text should be delivered, such as `[confusion]` or `[laughs]`.\n",
"\n",
"You can experiment with different tags. Here is a list of examples to get started:\n",
"\n",
"| Audio Tag | Audio Tag | Audio Tag | Audio Tag | Audio Tag |\n",
"| --- | --- | --- | --- | --- |\n",
"| [determination]| [enthusiasm] | [adoration] |[interest] | [awe] |\n",
"|[admiration] | [nervousness] | [frustration] | [excitement] | [curiosity] |\n",
"| [hope] | [annoyance] | [amusement] | [aggression] | [tension] |\n",
"| [agitation] | [confusion] | [anger] | [positive] | [neutral] |\n",
"| [negative] | [whispers] | [laughs] | | |\n",
"\n",
"**NOTE:** For best results, add audio tags in English, even if your transcript is in a different language."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "OgemQk62Zowg"
},
"outputs": [],
"source": [
"prompt = \"\"\"\n",
"[confusion] I have absolutely no idea why this variable is returning as undefined, [agitation] and I've been staring at this same block of code until my eyes are literally crossing! [aggression] Just work already! [whispers] Wait... is it really just a missing semicolon right there on line forty-two? [laughs] I can't believe I spent two hours debugging a simple typo!\n",
"\"\"\"\n",
"\n",
"response = client.models.generate_content(\n",
" model=speech_model,\n",
" contents=prompt,\n",
" config=types.GenerateContentConfig(\n",
" response_modalities=[\"AUDIO\"],\n",
" speech_config=types.SpeechConfig(\n",
" voice_config=types.VoiceConfig(\n",
" prebuilt_voice_config=types.PrebuiltVoiceConfig(voice_name=\"Charon\")\n",
" )\n",
" ),\n",
" ),\n",
")\n",
"\n",
"play_audio(response)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RvcPT7El3o8m"
},
"source": [
"### Adding audio tags\n",
"\n",
"If you have a long transcript, it may be useful to have Gemini help add in audio tags. In this next example, you'll take a fictitious podcast transcript and give it to Gemini with a list of example audio tags that can be included."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"id": "SjQanjUYckuA"
},
"outputs": [],
"source": [
"podcast = \"\"\"\n",
"If you walk through a local park on a Saturday morning, you are bound to see a familiar cast of characters. There is the floppy-eared Golden Retriever chasing a tennis ball, the stout French Bulldog trotting along the pavement, and almost certainly, the athletic Labrador Retriever leading the way. We think of these breeds as timeless icons of the domestic home, but the \"popularity\" of a dog is rarely an accident of nature. It is a fascinating mirror held up to human history, shifting based on our wars, our technology, and even our movies.\n",
"\n",
"For most of human history, a \"popular\" dog was simply the one that kept you alive. We didn't have breeds in the modern sense; we had jobs. There were dogs that pulled sleds, dogs that guarded livestock, and dogs that cleared vermin from granaries. If a dog was popular in the year 1500, it was because it was an elite employee. The idea of a dog as a \"member of the family\" with no specific chore to perform is a relatively new luxury, largely born out of the Victorian era.\n",
"\n",
"It was during the 19th century in England that dog ownership shifted from a functional necessity to a high-stakes hobby. This was the birth of the Kennel Club and the formalization of \"breeds.\" Suddenly, the physical consistency of a dog—the shape of its snout or the curl of its tail—became a status symbol. A popular breed wasn't just a good worker; it was a fashion statement. This is when the Beagle and the Fox Terrier began their rise.\n",
"\n",
"However, nothing influences what we choose to bring into our homes quite like the silver screen. We see this in what researchers call the \"Lassie Effect.\" When the movie Lassie Come Home was released in 1943, registrations for Rough Collies skyrocketed. It's a recurring theme in history: we fall in love with a character, and the breed becomes a trend.\n",
"\n",
"Then there is the king of the mountain: the Labrador Retriever. For thirty-one consecutive years, the Lab held the title of the most popular dog in America. Its rise to power was a perfect storm of timing. As the world moved into the suburbs after World War II, people wanted a dog that was \"versatile.\" The Lab was small enough to live indoors but sturdy enough to go hiking; it was gentle with children but capable of being a serious hunting partner. It became the \"all-purpose\" dog for the modern age.\n",
"\n",
"Today, we are seeing another shift. The French Bulldog recently dethroned the Labrador, ending its three-decade reign. This tells us something about how our lives have changed again. We live in smaller spaces, we have less time for long runs in the woods, and we value portability. The popularity of \"designer\" breeds like the various Doodles reflects a modern obsession with customization—wanting the personality of a Retriever with the hypoallergenic coat of a Poodle.\n",
"\n",
"When we look at the most popular breeds of today, we aren't just looking at dogs. We are looking at a map of our own desires. We choose the breeds that fit the lives we want to live, whether that's the rugged adventurer, the urban socialite, or the suburban family. The dogs change because we change.\n",
"\n",
"Thanks for joining me on The Everyday. Next time you see a dog on the street, take a look at the human on the other end of the leash—you might learn just as much about them as you do about the dog. I'll see you next time.\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"id": "-qEsorzYdwdn"
},
"outputs": [],
"source": [
"example_audio_tags = [\n",
" \"[acceptance]\",\n",
" \"[accomplishment]\",\n",
" \"[achievement]\",\n",
" \"[active]\",\n",
" \"[admiration]\",\n",
" \"[admonition]\",\n",
" \"[adoration]\",\n",
" \"[affection]\",\n",
" \"[aggression]\",\n",
" \"[agitation]\",\n",
" \"[alarm]\",\n",
" \"[amazement]\",\n",
" \"[ambivalence]\",\n",
" \"[amused]\",\n",
" \"[amusement]\",\n",
" \"[analysis]\",\n",
" \"[anger]\",\n",
" \"[animation]\",\n",
" \"[annoyance]\",\n",
" \"[anticipation]\",\n",
" \"[anxiety]\",\n",
" \"[apology]\",\n",
" \"[appreciation]\",\n",
" \"[apprehension]\",\n",
" \"[approval]\",\n",
" \"[arrogance]\",\n",
" \"[assertion]\",\n",
" \"[assertive]\",\n",
" \"[assertiveness]\",\n",
" \"[assurance]\",\n",
" \"[astonishment]\",\n",
" \"[aversion]\",\n",
" \"[awareness]\",\n",
" \"[awe]\",\n",
" \"[awkwardness]\",\n",
" \"[bargaining]\",\n",
" \"[boredom]\",\n",
" \"[caring]\",\n",
" \"[caution]\",\n",
" \"[cautious]\",\n",
" \"[certainty]\",\n",
" \"[challenging]\",\n",
" \"[comfort]\",\n",
" \"[compassion]\",\n",
" \"[concentration]\",\n",
" \"[concern]\",\n",
" \"[confidence]\",\n",
" \"[confident]\",\n",
" \"[confusion]\",\n",
" \"[contemplative]\",\n",
" \"[contempt]\",\n",
" \"[contentment]\",\n",
" \"[conviction]\",\n",
" \"[courage]\",\n",
" \"[craving]\",\n",
" \"[critical]\",\n",
" \"[criticism]\",\n",
" \"[curiosity]\",\n",
" \"[decision]\",\n",
" \"[defiance]\",\n",
" \"[demonstration]\",\n",
" \"[description]\",\n",
" \"[descriptive]\",\n",
" \"[desire]\",\n",
" \"[despair]\",\n",
" \"[desperation]\",\n",
" \"[despondency]\",\n",
" \"[determination]\",\n",
" \"[determined]\",\n",
" \"[devotion]\",\n",
" \"[directness]\",\n",
" \"[disagreement]\",\n",
" \"[disappointment]\",\n",
" \"[disapproval]\",\n",
" \"[disbelief]\",\n",
" \"[discernment]\",\n",
" \"[discomfort]\",\n",
" \"[disdain]\",\n",
" \"[disgust]\",\n",
" \"[disillusionment]\",\n",
" \"[dislike]\",\n",
" \"[dismissive]\",\n",
" \"[distress]\",\n",
" \"[doubt]\",\n",
" \"[dread]\",\n",
" \"[eagerness]\",\n",
" \"[effervescence]\",\n",
" \"[embarrassment]\",\n",
" \"[embitterment]\",\n",
" \"[embracement]\",\n",
" \"[empathy]\",\n",
" \"[emphasis]\",\n",
" \"[enchantment]\",\n",
" \"[encouraging]\",\n",
" \"[energetic]\",\n",
" \"[enjoyment]\",\n",
" \"[enthusiasm]\",\n",
" \"[enthusiastic]\",\n",
" \"[excitement]\",\n",
" \"[exhaustion]\",\n",
" \"[explaining]\",\n",
" \"[fascination]\",\n",
" \"[fast]\",\n",
" \"[fear]\",\n",
" \"[focus]\",\n",
" \"[fondness]\",\n",
" \"[friendly]\",\n",
" \"[frustration]\",\n",
" \"[gratification]\",\n",
" \"[gratitude]\",\n",
" \"[grief]\",\n",
" \"[guilt]\",\n",
" \"[happy]\",\n",
" \"[high energy]\",\n",
" \"[hope]\",\n",
" \"[horror]\",\n",
" \"[humor]\",\n",
" \"[hurt]\",\n",
" \"[incredulity]\",\n",
" \"[indifference]\",\n",
" \"[indignation]\",\n",
" \"[informative]\",\n",
" \"[instruction]\",\n",
" \"[interest]\",\n",
" \"[intrigue]\",\n",
" \"[invitation]\",\n",
" \"[joy]\",\n",
" \"[laughs]\",\n",
" \"[logical reasoning]\",\n",
" \"[long pause]\",\n",
" \"[love]\",\n",
" \"[low energy]\",\n",
" \"[melancholy]\",\n",
" \"[mixed]\",\n",
" \"[negative]\",\n",
" \"[negative surprise]\",\n",
" \"[nervousness]\",\n",
" \"[neutral]\",\n",
" \"[nostalgia]\",\n",
" \"[observation]\",\n",
" \"[offense]\",\n",
" \"[optimism]\",\n",
" \"[pain]\",\n",
" \"[panic]\",\n",
" \"[passion]\",\n",
" \"[passive]\",\n",
" \"[pensive]\",\n",
" \"[pessimism]\",\n",
" \"[pity]\",\n",
" \"[planning]\",\n",
" \"[playful]\",\n",
" \"[pleading]\",\n",
" \"[pleased]\",\n",
" \"[positive]\",\n",
" \"[positive surprise]\",\n",
" \"[praise]\",\n",
" \"[pride]\",\n",
" \"[realization]\",\n",
" \"[recognition]\",\n",
" \"[reflection]\",\n",
" \"[regret]\",\n",
" \"[relaxation]\",\n",
" \"[relief]\",\n",
" \"[reminiscence]\",\n",
" \"[resignation]\",\n",
" \"[sadness]\",\n",
" \"[sarcasm]\",\n",
" \"[satisfaction]\",\n",
" \"[self-deprecation]\",\n",
" \"[self-satisfaction]\",\n",
" \"[sentimentality]\",\n",
" \"[serenity]\",\n",
" \"[seriousness]\",\n",
" \"[shame]\",\n",
" \"[shock]\",\n",
" \"[short pause]\",\n",
" \"[skepticism]\",\n",
" \"[slight relief]\",\n",
" \"[smitten]\",\n",
" \"[solemnity]\",\n",
" \"[speculation]\",\n",
" \"[slow]\",\n",
" \"[strategizing]\",\n",
" \"[stress]\",\n",
" \"[struggle]\",\n",
" \"[success]\",\n",
" \"[suffering]\",\n",
" \"[suggestion]\",\n",
" \"[summary]\",\n",
" \"[surprise]\",\n",
" \"[suspicion]\",\n",
" \"[sympathy]\",\n",
" \"[tension]\",\n",
" \"[terror]\",\n",
" \"[thanks]\",\n",
" \"[thinking]\",\n",
" \"[thrill]\",\n",
" \"[tiredness]\",\n",
" \"[triumph]\",\n",
" \"[uncertainty]\",\n",
" \"[unclear]\",\n",
" \"[understanding]\",\n",
" \"[unease]\",\n",
" \"[urgency]\",\n",
" \"[victory]\",\n",
" \"[warning]\",\n",
" \"[weariness]\",\n",
" \"[whispers]\",\n",
" \"[wisdom]\",\n",
" \"[wistful]\",\n",
" \"[worry]\",\n",
" \"[yearning]\",\n",
"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_cztl0y-4Yum"
},
"source": [
"Now, you'll prompt Gemini to take the podcast transcript and insert audio tags from the provided list."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "qOetTxFRdbys"
},
"outputs": [],
"source": [
"response = client.models.generate_content(\n",
" model=gemini_model,\n",
" contents=f\"Please take the {podcast} provided and insert audio tags from {example_audio_tags}. Insert the tags immediately before the phrase or sentence they are meant to influence. Ensure the tag matches the emotional arc of the narrative. Avoid overusing tags, place them where a natural change in tone or pace would occur.\",\n",
")\n",
"\n",
"podcast_tags = response.text\n",
"display(Markdown(podcast_tags))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "k-X-R4zB4oSY"
},
"source": [
"Finally, you'll send the updated transcript with the audio tags to the TTS model to generate the audio with the updated modifiers."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "g2l6-5e9fpVw"
},
"outputs": [],
"source": [
"response = client.models.generate_content(\n",
" model=speech_model,\n",
" contents=podcast_tags,\n",
" config=types.GenerateContentConfig(\n",
" response_modalities=[\"AUDIO\"],\n",
" speech_config=types.SpeechConfig(\n",
" voice_config=types.VoiceConfig(\n",
" prebuilt_voice_config=types.PrebuiltVoiceConfig(voice_name=\"Achernar\")\n",
" )\n",
" ),\n",
" ),\n",
")\n",
"\n",
"play_audio(response)"
]
}
],
"metadata": {
"colab": {
"name": "gemini_3_1_flash_tts.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
audio/speech/getting-started/get_started_with_chirp3_instant_custom_voice.ipynb
+782
View File
@@ -0,0 +1,782 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Get started with Chirp 3: Instant custom voice\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp3_instant_custom_voice.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Faudio%2Fspeech%2Fgetting-started%2Fget_started_with_chirp3_instant_custom_voice.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/audio/speech/getting-started/get_started_with_chirp3_instant_custom_voice.ipynb\">\n",
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp3_instant_custom_voice.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<p>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp3_instant_custom_voice.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp3_instant_custom_voice.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp3_instant_custom_voice.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp3_instant_custom_voice.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp3_instant_custom_voice.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| Author(s) |\n",
"| --- |\n",
"| [Ivan Nardini](https://github.com/inardini) |\n",
"| [Gary Chien](https://github.com/goldenchest) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"This notebook introduces [Chirp 3 Instant Custom Voice](https://cloud.google.com/text-to-speech/docs/chirp3-hd), a powerful feature of Google Cloud's Text-to-Speech (TTS) API that allows you to create personalized voice models.\n",
"\n",
"With Instant Custom Voice, you can generate unique, custom voices by training a model with your own high-quality audio recordings. This enables the rapid generation of personal voices that can then be used to synthesize audio using the Cloud TTS API, supporting both streaming and long-form text output. Instant Custom Voice creation and synthesis is supported in more than 25 language.\n",
"\n",
"In this tutorial, you will learn how to:\n",
"\n",
"- Create an Instant Custom Voice.\n",
"- Synthesize text using your custom voice both in real-time and streaming.\n",
"- Build a simple Gradio app to use your custom voice."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7qzYMYf2Hlvv"
},
"source": [
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ Due to safety considerations, access to this voice cloning capability is restricted to allow-listed users. To access this feature, contact a member of the Google Cloud team to be added to the allow list. ⚠️</b>\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1filYzhuK_MK"
},
"source": [
"### Install required packages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-_O1r5MNLFu-"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet gradio"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SxJiB0GXALQ2"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using the Text-to-Speech API, you must have an existing Google Cloud project and [enable the API](https://console.cloud.google.com/flows/enableapi?apiid=texttospeech.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment).\n",
"\n",
"Please note the **available regions** for Chirp 3 Instant Custom voice, see [documentation](https://cloud.google.com/text-to-speech/docs/chirp3-instant-custom-voice#regional_availability)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "WIQyBhAn_9tK"
},
"outputs": [],
"source": [
"# Use the environment variable if the user doesn't provide Project ID.\n",
"import os\n",
"\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"TTS_LOCATION = \"global\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PYVzRl1bmgjk"
},
"outputs": [],
"source": [
"! gcloud config set project {PROJECT_ID}\n",
"! gcloud auth application-default set-quota-project {PROJECT_ID}\n",
"! gcloud auth application-default login -q"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JR7BVajtKyGh"
},
"source": [
"### Get Google Credentials\n",
"\n",
"Use the `google.auth` library to automatically find and load your credentials.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ChWHE3jIDREc"
},
"outputs": [],
"source": [
"import google.auth\n",
"import google.auth.transport.requests\n",
"\n",
"credentials, _ = google.auth.default()\n",
"authentication = google.auth.transport.requests.Request()\n",
"credentials.refresh(authentication)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5-X8IbjiAtM3"
},
"source": [
"### Import libraries\n",
"\n",
"Import Python tools you'll need."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "qqm0OQpAYCph"
},
"outputs": [],
"source": [
"import base64\n",
"import json\n",
"import os\n",
"\n",
"import gradio as gr\n",
"import numpy as np\n",
"import requests\n",
"from IPython.display import Audio, display"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sP8GBj3tBAC1"
},
"source": [
"### Set constants\n",
"\n",
"Initiate the API endpoint and the text to speech client.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rXTVeU1uBBqY"
},
"outputs": [],
"source": [
"API_ENDPOINT = (\n",
" f\"{TTS_LOCATION}-texttospeech.googleapis.com\"\n",
" if TTS_LOCATION != \"global\"\n",
" else \"texttospeech.googleapis.com\"\n",
")\n",
"\n",
"ACCESS_TOKEN = credentials.token"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "M7WQQFp_RvGH"
},
"source": [
"### Helpers\n",
"\n",
"To keep our main logic clean and readable, you define several helper functions here. These encapsulate tasks like making API calls, processing audio data, and interacting with the Gradio interface components.\n",
"\n",
"* **`create_instant_custom_voice_key(reference_audio_bytes, consent_audio_bytes)`** to create a temporary custom voice key.\n",
"\n",
"* **`create_voice_with_masking(reference_audio, consent_audio)`** to mask the custom voice key.\n",
"\n",
"* **`synthesize_text_with_cloned_voice(voice_key, text)`** to create your custom voice using the `voice_cloning_key` and the desired text as input.\n",
"\n",
"* **`wav_to_base64(file_path)`** to read a WAV audio file from a local path, encode its binary content into a base64 string (which is how audio data is sent in the JSON payload), and return the string.\n",
"\n",
"* **`create_voice(reference_audio, consent_audio, progress)`** to create the custom voice in the Gradio app using \"Create Voice\" button.\n",
"\n",
"* **`generate_speech(voice_key, text, progress)`**: to synthesize any text with the custom voice using \"Generate Speech\" button in the Gradio app.\n",
"\n",
"* **`reset_interface()`** to effectively reset the Gradio UI."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "pWrNWdV_RxGn"
},
"outputs": [],
"source": [
"def create_instant_custom_voice_key(\n",
" reference_audio_bytes: bytes, consent_audio_bytes: bytes\n",
") -> str:\n",
" \"\"\"Creates a temporary custom voice key\"\"\"\n",
" url = f\"https://{API_ENDPOINT}/v1beta1/voices:generateVoiceCloningKey\"\n",
"\n",
" request_body = {\n",
" \"reference_audio\": {\n",
" \"audio_config\": {\"audio_encoding\": \"LINEAR16\", \"sample_rate_hertz\": 24000},\n",
" \"content\": reference_audio_bytes,\n",
" },\n",
" \"voice_talent_consent\": {\n",
" \"audio_config\": {\"audio_encoding\": \"LINEAR16\", \"sample_rate_hertz\": 24000},\n",
" \"content\": consent_audio_bytes,\n",
" },\n",
" \"consent_script\": \"I am the owner of this voice and I consent to Google using this voice to create a synthetic voice model.\",\n",
" \"language_code\": \"en-US\",\n",
" }\n",
"\n",
" response = None\n",
" try:\n",
" headers = {\n",
" \"Authorization\": f\"Bearer {ACCESS_TOKEN}\",\n",
" \"x-goog-user-project\": PROJECT_ID,\n",
" \"Content-Type\": \"application/json; charset=utf-8\",\n",
" }\n",
"\n",
" response = requests.post(url, headers=headers, json=request_body)\n",
" response.raise_for_status()\n",
"\n",
" response_json = response.json()\n",
" return response_json.get(\"voiceCloningKey\")\n",
"\n",
" except requests.exceptions.RequestException as e:\n",
" print(f\"Error making API request: {e}\")\n",
" if response is not None and response.text:\n",
" print(\"Response error message:\")\n",
" print(response.text)\n",
" except json.JSONDecodeError as e:\n",
" print(f\"Error decoding JSON response: {e}\")\n",
" except Exception as e:\n",
" print(f\"An unexpected error occurred: {e}\")\n",
"\n",
"\n",
"def synthesize_text_with_cloned_voice(voice_key: str, text: str) -> None:\n",
" \"\"\"Synthesizes text with the cloned voice\"\"\"\n",
" url = f\"https://{API_ENDPOINT}/v1beta1/text:synthesize\"\n",
"\n",
" request_body = {\n",
" \"input\": {\"text\": text},\n",
" \"voice\": {\n",
" \"language_code\": \"en-US\",\n",
" \"voice_clone\": {\n",
" \"voice_cloning_key\": voice_key,\n",
" },\n",
" },\n",
" \"audioConfig\": {\"audioEncoding\": \"LINEAR16\", \"sample_rate_hertz\": 24000},\n",
" }\n",
"\n",
" try:\n",
" headers = {\n",
" \"Authorization\": f\"Bearer {ACCESS_TOKEN}\",\n",
" \"x-goog-user-project\": PROJECT_ID,\n",
" \"Content-Type\": \"application/json; charset=utf-8\",\n",
" }\n",
"\n",
" response = requests.post(url, headers=headers, json=request_body)\n",
" response.raise_for_status()\n",
"\n",
" response_json = response.json()\n",
" audio_content = response_json.get(\"audioContent\")\n",
"\n",
" if audio_content:\n",
" display(Audio(base64.b64decode(audio_content), rate=24000))\n",
" else:\n",
" print(\"Error: Audio content not found in the response.\")\n",
" print(response_json)\n",
"\n",
" except requests.exceptions.RequestException as e:\n",
" print(f\"Error making API request: {e}\")\n",
" except json.JSONDecodeError as e:\n",
" print(f\"Error decoding JSON response: {e}\")\n",
" except Exception as e:\n",
" print(f\"An unexpected error occurred: {e}\")\n",
"\n",
"\n",
"def wav_to_base64(file_path: str) -> str:\n",
" \"\"\"Convert a WAV file to base64 encoded string\"\"\"\n",
" try:\n",
" with open(file_path, \"rb\") as wav_file:\n",
" encoded_string = base64.b64encode(wav_file.read()).decode(\"utf-8\")\n",
" return encoded_string\n",
" except FileNotFoundError:\n",
" print(f\"Error: File not found at {file_path}\")\n",
" return None\n",
" except Exception as e:\n",
" print(f\"An error occurred: {e}\")\n",
" return None\n",
"\n",
"\n",
"def create_voice(\n",
" reference_audio: gr.Audio,\n",
" consent_audio: gr.Audio,\n",
" progress: gr.Progress | None = None,\n",
") -> str:\n",
" \"\"\"Create a custom voice using reference and consent audio\"\"\"\n",
" if reference_audio is None or consent_audio is None:\n",
" return \"Please upload both reference and consent audio files.\"\n",
"\n",
" if not progress:\n",
" progress = gr.Progress()\n",
"\n",
" progress(0.2, desc=\"Processing audio files...\")\n",
" reference_audio_b64 = wav_to_base64(reference_audio)\n",
" consent_audio_b64 = wav_to_base64(consent_audio)\n",
"\n",
" if reference_audio_b64 is None or consent_audio_b64 is None:\n",
" return \"Error processing audio files.\"\n",
"\n",
" progress(0.5, desc=\"Creating voice clone...\")\n",
" voice_key = create_instant_custom_voice_key(reference_audio_b64, consent_audio_b64)\n",
"\n",
" if voice_key:\n",
" progress(1.0, desc=\"Voice created successfully!\")\n",
" return voice_key\n",
" return \"Failed to create voice. Check the logs for details.\"\n",
"\n",
"\n",
"def create_voice_with_masking(\n",
" reference_audio: gr.Audio, consent_audio: gr.Audio\n",
") -> tuple:\n",
" \"\"\"A wrapper function for create_voice to handle masking\"\"\"\n",
" key = create_voice(reference_audio, consent_audio)\n",
" if key and len(key) > 8:\n",
" masked_key = key[:5] + \"...\"\n",
" else:\n",
" masked_key = key\n",
" return key, masked_key\n",
"\n",
"\n",
"def generate_speech(\n",
" voice_key: str, text: str, progress: gr.Progress | None = None\n",
") -> tuple:\n",
" \"\"\"Generate speech using the cloned voice\"\"\"\n",
" if not voice_key or not text:\n",
" return None, \"Please create a voice key and enter text to synthesize.\"\n",
"\n",
" if not progress:\n",
" progress = gr.Progress()\n",
"\n",
" progress(0.3, desc=\"Generating speech...\")\n",
"\n",
" try:\n",
" url = f\"https://{API_ENDPOINT}/v1beta1/text:synthesize\"\n",
"\n",
" request_body = {\n",
" \"input\": {\"text\": text},\n",
" \"voice\": {\n",
" \"language_code\": \"en-US\",\n",
" \"voice_clone\": {\n",
" \"voice_cloning_key\": voice_key,\n",
" },\n",
" },\n",
" \"audioConfig\": {\"audioEncoding\": \"LINEAR16\", \"sample_rate_hertz\": 24000},\n",
" }\n",
"\n",
" headers = {\n",
" \"Authorization\": f\"Bearer {ACCESS_TOKEN}\",\n",
" \"x-goog-user-project\": PROJECT_ID,\n",
" \"Content-Type\": \"application/json; charset=utf-8\",\n",
" }\n",
"\n",
" progress(0.6, desc=\"Processing audio...\")\n",
" response = requests.post(url, headers=headers, json=request_body)\n",
" response.raise_for_status()\n",
"\n",
" response_json = response.json()\n",
" audio_content = response_json.get(\"audioContent\")\n",
"\n",
" if audio_content:\n",
" progress(1.0, desc=\"Speech generated!\")\n",
" audio_bytes = base64.b64decode(audio_content)\n",
" audio_array = np.frombuffer(audio_bytes, dtype=np.int16)\n",
" return (24000, audio_array), \"Speech generated successfully!\"\n",
" return None, \"Error: Audio content not found in the response.\"\n",
"\n",
" except Exception as e:\n",
" return None, f\"Error generating speech: {e!s}\"\n",
"\n",
"\n",
"def reset_interface() -> tuple:\n",
" return None, None, \"\", \"\", None, None, \"Interface reset.\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EdvJRUWRNGHE"
},
"source": [
"# Create an Instant Custom Voice\n",
"\n",
"Let's start by creating the custom voice directly.\n",
"\n",
"You would first define the file paths to your pre-recorded reference audio (`.wav` recommended, ideally a few seconds clear speech) and the consent audio (where the speaker explicitly states the consent script).\n",
"\n",
"Then you use `wav_to_base64` to read these files and encode them into the base64 format required by the API and you create your custom voice using the `create_instant_custom_voice_key` helper function, passing in the base64-encoded audio data.\n",
"\n",
"If the request is successful and your project is allow-listed, the API returns a `voice_cloning_key` which acts as a temporary identifier for your custom voice.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "S_J-TGJQNuKq"
},
"outputs": [],
"source": [
"# fmt: off\n",
"reference_audio_path = \"[your-reference-audio-path]\" # @param {type: \"string\", placeholder: \"[your-reference-audio-path]\", isTemplate: true}\n",
"consent_audio_path = \"[your-consent-audio-path]\" # @param {type: \"string\", placeholder: \"[your-consent-audio-path]\", isTemplate: true}\n",
"# fmt: on"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "SCdVWsGOA3qW"
},
"outputs": [],
"source": [
"reference_audio_bytes = wav_to_base64(reference_audio_path)\n",
"consent_audio_bytes = wav_to_base64(consent_audio_path)\n",
"\n",
"voice_key = create_instant_custom_voice_key(reference_audio_bytes, consent_audio_bytes)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rPqteN879IL_"
},
"source": [
"# Use Instant Custom Voice\n",
"\n",
"Now that you (theoretically) have a `voice_key`, let's use it to synthesize speech."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JGAZb4Zf9Me-"
},
"source": [
"## Perform a Sync Request\n",
"\n",
"Define the text you want your custom voice to say in the `text_to_synthesize` variable. Try different sentences!\n",
"\n",
"Then, call the `synthesize_text_with_cloned_voice` helper function, providing the `voice_key` you obtained earlier and the text. This function sends the request to the `text:synthesize` endpoint, specifying your custom voice.\n",
"\n",
"If successful, it gets the audio data back, decodes it, and should play it directly in the notebook output below the cell using an embedded audio player."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "djLao152cR13"
},
"outputs": [],
"source": [
"text_to_synthesize = \"\"\"\n",
"Breaking news! Chirp 3, Google Cloud's audio model, now has Instant Custom Voice.\n",
"\n",
"With Instant Custom Voice, you can generate custom voices with just 10 seconds of audio to empower your AI narration.\n",
"Chirp 3 Instant Custom Voice is now available in preview with allowlist. Check out the link below.\n",
"\n",
"And yes, this voice is generated using Chirp 3 Instant Custom Voice!\n",
"\"\"\"\n",
"synthesize_text_with_cloned_voice(voice_key, text_to_synthesize)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3X0qY28fLSfz"
},
"source": [
"# Build a simple Instant custom voice app\n",
"\n",
"While calling the API directly works, it's often more convenient to have an interactive interface. Let's build a simple web app using Gradio to easily upload audio, create a voice, and synthesize text."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uXoJqKduNhzq"
},
"source": [
"### Define the app\n",
"\n",
"Here, you define the structure and components of our Gradio user interface."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "jMISBY0fSDFy"
},
"outputs": [],
"source": [
"with gr.Blocks(\n",
" theme=gr.themes.Default(\n",
" primary_hue=\"blue\", secondary_hue=\"blue\", neutral_hue=\"gray\"\n",
" )\n",
") as app:\n",
" # Create a state variable to store the full voice key\n",
" full_voice_key = gr.State(\"\")\n",
"\n",
" # Define title\n",
" gr.Markdown(\n",
" \"\"\"\n",
" # Chirp 3 - Instant custom voice demo\n",
" Upload reference and consent audio files to create a custom voice, then synthesize speech.\n",
" \"\"\"\n",
" )\n",
"\n",
" # Define input and output components\n",
" with gr.Row():\n",
" with gr.Column(scale=1):\n",
" reference_audio = gr.Audio(\n",
" label=\"Reference Voice\", type=\"filepath\", elem_id=\"reference_audio\"\n",
" )\n",
" consent_audio = gr.Audio(\n",
" label=\"Consent Audio\", type=\"filepath\", elem_id=\"consent_audio\"\n",
" )\n",
"\n",
" create_btn = gr.Button(\"Create Voice\", variant=\"primary\")\n",
" voice_key_output = gr.Textbox(label=\"Voice Key\", elem_id=\"voice_key\")\n",
"\n",
" with gr.Column(scale=1):\n",
" text_input = gr.Textbox(\n",
" label=\"Text to Synthesize\",\n",
" placeholder=\"Enter the text you want the voice to say...\",\n",
" lines=5,\n",
" elem_id=\"text_input\",\n",
" )\n",
" generate_btn = gr.Button(\"Generate Speech\", variant=\"primary\")\n",
" audio_output = gr.Audio(label=\"Generated Audio\", elem_id=\"audio_output\")\n",
" status_output = gr.Textbox(label=\"Status\", elem_id=\"status_output\")\n",
"\n",
" with gr.Row():\n",
" clear_btn = gr.Button(\"Clear All\", variant=\"secondary\")\n",
"\n",
" # Set up event handlers\n",
" create_btn.click(\n",
" create_voice_with_masking,\n",
" inputs=[reference_audio, consent_audio],\n",
" outputs=[full_voice_key, voice_key_output],\n",
" )\n",
"\n",
" generate_btn.click(\n",
" generate_speech,\n",
" inputs=[full_voice_key, text_input],\n",
" outputs=[audio_output, status_output],\n",
" )\n",
"\n",
" clear_btn.click(\n",
" reset_interface,\n",
" inputs=[],\n",
" outputs=[\n",
" reference_audio,\n",
" consent_audio,\n",
" voice_key_output,\n",
" full_voice_key,\n",
" text_input,\n",
" audio_output,\n",
" status_output,\n",
" ],\n",
" )\n",
"\n",
" # Apply custom CSS for Google styling\n",
" gr.Markdown(\n",
" \"\"\"\n",
" <style>\n",
" .gradio-container {\n",
" font-family: 'Google Sans', 'Roboto', sans-serif !important;\n",
" }\n",
" .gr-button-primary {\n",
" background-color: #4285F4 !important;\n",
" }\n",
" .gr-button-secondary {\n",
" border-color: #4285F4 !important;\n",
" color: #4285F4 !important;\n",
" }\n",
" h1 {\n",
" font-family: 'Google Sans', 'Roboto', sans-serif !important;\n",
" color: #202124 !important;\n",
" }\n",
" </style>\n",
" \"\"\"\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GpCcdY4ANkRl"
},
"source": [
"### Launch the app\n",
"\n",
"Showtime! This cell launches the Gradio app we just defined."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "YJ2-accJL_tn"
},
"outputs": [],
"source": [
"app.launch(share=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xV-Nj4F7mPNF"
},
"source": [
"Close the app once you finish to play with it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "UTPbraYfNeJk"
},
"outputs": [],
"source": [
"app.close()"
]
}
],
"metadata": {
"colab": {
"name": "get_started_with_chirp3_instant_custom_voice.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
audio/speech/getting-started/get_started_with_chirp_3_hd_voices.ipynb
+565
View File
@@ -0,0 +1,565 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Get started with Chirp 3 HD voices using Text-to-Speech\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_hd_voices.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Faudio%2Fspeech%2Fgetting-started%2Fget_started_with_chirp_3_hd_voices.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/audio/speech/getting-started/get_started_with_chirp_3_hd_voices.ipynb\">\n",
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_hd_voices.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<p>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_hd_voices.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_hd_voices.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_hd_voices.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_hd_voices.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_hd_voices.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| Authors |\n",
"| --- |\n",
"| [Holt Skinner](https://github.com/holtskinner) |\n",
"| [Ivan Nardini](https://github.com/inardini) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"This notebook introduces [Chirp 3 HD Voices](https://cloud.google.com/text-to-speech/docs/chirp3-hd), which are Google Cloud's latest advancement in Text-to-Speech (TTS) technology.\n",
"\n",
"These voices, powered by state-of-the-art large language models (LLMs), offer a significantly improved level of realism and emotional expressiveness.\n",
"\n",
"Chirp 3 HD voices provide high-fidelity audio and natural-sounding speech, complete with human-like intonation and pauses. They are available on the Vertex AI platform and are designed for various uses like, voice assistants, audiobooks, and customer service applications.\n",
"\n",
"There are currently eight distinct voice options(4 male, 4 female) available in 31 languages.\n",
"\n",
"In this tutorial, you learn how to:\n",
"\n",
"- How to synthesize speech using real-time (online) processing\n",
"- How to synthesize speech using streaming processing"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Text-to-Speech SDK and other required packages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e73_ZgKWYedz"
},
"outputs": [],
"source": [
"%%bash\n",
"# Detect the operating system\n",
"os=$(uname -s)\n",
"\n",
"if [[ \"$os\" == \"Linux\" ]]; then\n",
" # Linux installation\n",
" sudo apt update -y -qq\n",
" sudo apt install ffmpeg -y -qq\n",
" echo \"ffmpeg installed successfully on Linux.\"\n",
"elif [[ \"$os\" == \"Darwin\" ]]; then\n",
" # macOS installation\n",
" if command -v brew &> /dev/null; then\n",
" brew install ffmpeg\n",
" if [[ $? -eq 0 ]]; then\n",
" echo \"ffmpeg installed successfully on macOS using Homebrew.\"\n",
" else\n",
" echo \"Error installing ffmpeg on macOS using Homebrew.\"\n",
" fi\n",
" else\n",
" echo \"Homebrew is not installed. Please install Homebrew and try again.\"\n",
" fi\n",
"else\n",
" echo \"Unsupported operating system: $os\"\n",
"fi"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-cloud-texttospeech"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information and initialize SDK\n",
"\n",
"To get started using the Text-to-Speech API, you must have an existing Google Cloud project and [enable the API](https://console.cloud.google.com/flows/enableapi?apiid=texttospeech.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment).\n",
"\n",
"Please note the **available regions** for Chirp 3, see [documentation](https://cloud.google.com/text-to-speech/docs/endpoints)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "WIQyBhAn_9tK"
},
"outputs": [],
"source": [
"# Use the environment variable if the user doesn't provide Project ID.\n",
"import os\n",
"\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"TTS_LOCATION = \"global\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "76915236b1c9"
},
"outputs": [],
"source": [
"! gcloud config set project {PROJECT_ID}\n",
"! gcloud auth application-default set-quota-project {PROJECT_ID}\n",
"! gcloud auth application-default login -q"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5303c05f7aa6"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "qqm0OQpAYCph"
},
"outputs": [],
"source": [
"import re\n",
"from collections.abc import Iterator\n",
"\n",
"import numpy as np\n",
"from IPython.display import Audio, display\n",
"from google.api_core.client_options import ClientOptions\n",
"from google.cloud import texttospeech_v1beta1 as texttospeech"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sP8GBj3tBAC1"
},
"source": [
"### Set constants\n",
"\n",
"Initiate the API endpoint and the text to speech client.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rXTVeU1uBBqY"
},
"outputs": [],
"source": [
"API_ENDPOINT = (\n",
" f\"{TTS_LOCATION}-texttospeech.googleapis.com\"\n",
" if TTS_LOCATION != \"global\"\n",
" else \"texttospeech.googleapis.com\"\n",
")\n",
"\n",
"client = texttospeech.TextToSpeechClient(\n",
" client_options=ClientOptions(api_endpoint=API_ENDPOINT)\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "M7WQQFp_RvGH"
},
"source": [
"### Helpers"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "pWrNWdV_RxGn"
},
"outputs": [],
"source": [
"def text_generator(text: str) -> Iterator[str]:\n",
" \"\"\"Split text into sentences to simulate streaming\"\"\"\n",
" # Use regex with positive lookahead to find sentence boundaries\n",
" # without consuming the space after the punctuation\n",
" sentences: list[str] = re.findall(r\"[^.!?]+[.!?](?:\\s|$)\", text + \" \")\n",
"\n",
" # Yield each complete sentence\n",
" for sentence in sentences:\n",
" yield sentence.strip()\n",
"\n",
" # Check if there's remaining text not caught by the regex\n",
" # (text without ending punctuation)\n",
" last_char_pos: int = 0\n",
" for sentence in sentences:\n",
" last_char_pos += len(sentence)\n",
"\n",
" if last_char_pos < len(text.strip()):\n",
" remaining: str = text.strip()[last_char_pos:]\n",
" if remaining:\n",
" yield remaining.strip()\n",
"\n",
"\n",
"def process_streaming_audio(\n",
" text: str,\n",
" voice: texttospeech.VoiceSelectionParams,\n",
" display_individual_chunks: bool = False,\n",
") -> np.ndarray:\n",
" \"\"\"Process text into speech using streaming TTS\"\"\"\n",
" # Generate sentences from text\n",
" sentences: list[str] = list(text_generator(text))\n",
"\n",
" # Get streaming audio\n",
" print(\"Streaming audio processing...\")\n",
" audio_iterator: Iterator[bytes] = synthesize_streaming(iter(sentences), voice=voice)\n",
"\n",
" # Process audio chunks\n",
" final_audio_data: np.ndarray = np.array([], dtype=np.int16)\n",
"\n",
" for idx, audio_content in enumerate(audio_iterator):\n",
" audio_chunk: np.ndarray = np.frombuffer(audio_content, dtype=np.int16)\n",
"\n",
" # Concatenate to final audio\n",
" final_audio_data = np.concatenate((final_audio_data, audio_chunk))\n",
"\n",
" # Optionally display individual chunks\n",
" if display_individual_chunks and len(audio_chunk) > 0:\n",
" print(f\"Processed chunk # {idx}\")\n",
" display(Audio(audio_chunk, rate=24000))\n",
"\n",
" print(\"Streaming audio processing complete!\")\n",
" return final_audio_data\n",
"\n",
"\n",
"def synthesize_streaming(\n",
" text_iterator: Iterator[str],\n",
" voice: texttospeech.VoiceSelectionParams,\n",
") -> Iterator[bytes]:\n",
" \"\"\"Synthesizes speech from an iterator of text inputs and yields audio content as an iterator.\n",
"\n",
" This function demonstrates how to use the Google Cloud Text-to-Speech API\n",
" to synthesize speech from a stream of text inputs provided by an iterator.\n",
" It yields the audio content from each response as an iterator of bytes.\n",
"\n",
" \"\"\"\n",
" config_request = texttospeech.StreamingSynthesizeRequest(\n",
" streaming_config=texttospeech.StreamingSynthesizeConfig(\n",
" voice=voice,\n",
" )\n",
" )\n",
"\n",
" def request_generator() -> Iterator[texttospeech.StreamingSynthesizeRequest]:\n",
" yield config_request\n",
" for text in text_iterator:\n",
" yield texttospeech.StreamingSynthesizeRequest(\n",
" input=texttospeech.StreamingSynthesisInput(text=text)\n",
" )\n",
"\n",
" streaming_responses: Iterator[texttospeech.StreamingSynthesizeResponse] = (\n",
" client.streaming_synthesize(request_generator())\n",
" )\n",
"\n",
" for response in streaming_responses:\n",
" yield response.audio_content"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VPVDNRyVxquo"
},
"source": [
"## Synthesize using Chirp 3 HD voices\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a9aa2ab365ac"
},
"source": [
"### Synthesize speech using real-time (online) processing\n",
"\n",
"You define the text you want to convert, select a specific voice and language, and then instruct the API to generate an audio of the spoken text.\n",
"\n",
"This example uses the `en-US-Chirp3-HD-Aoede` voice, which is a high-definition voice, offering improved clarity. The code will call the `synthesize_speech` method, which handles the core conversion process, and the output will be an MP3 audio as `bytes`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_7XwbSxUD4eW"
},
"outputs": [],
"source": [
"# fmt: off\n",
"prompt = \"Hello world! I am Chirp 3\" # @param [\"Hallo Welt! Ich bin Chirp 3\", \"Hello world! I am Chirp 3\", \"¡Hola mundo! Soy Chirp 3\", \"Bonjour le monde ! Je suis Chirp 3\", \"नमस्ते दुनिया! मैं चिर्प 3 हूं\", \"Olá Mundo! Eu sou o Chirp 3\", \"مرحبا بالعالم! أنا تشيرب 3\", \"¡Hola mundo! Soy Chirp 3\", \"Bonjour le monde ! Je suis Chirp 3\", \"Halo dunia! Saya Chirp 3\", \"Ciao mondo! Sono Chirp 3\", \"こんにちは世界！私はチャープ3です\", \"Merhaba dünya! Ben Chirp 3\", \"Chào thế giới! Tôi là Chirp 3\", \"হ্যালো ওয়ার্ল্ড! আমি চির্প 3\", \"હેલો વર્લ્ડ! હું ચિર્પ 3 છું\", \"ನಮಸ್ಕಾರ ಪ್ರಪಂಚ! ನಾನು ಚಿರ್ಪ್ 3\", \"ഹലോ വേൾഡ്! ഞാൻ ചിർപ് 3 ആണ്\", \"नमस्कार जग! मी चिरप 3 आहे\", \"வணக்கம் உலகம்! நான் சிர்ப் 3\", \"హలో వరల్డ్! నేను చిర్ప్ 3\", \"Hallo wereld! Ik ben Chirp 3\", \"안녕하세요! 저는 Chirp 3입니다\", \"你好世界！我是 Chirp 3\", \"Witaj świecie! Jestem Chirp 3\", \"Привет, мир! Я Чирп 3\", \"สวัสดีชาวโลก! ฉันคือเชิร์ป 3\"]\n",
"\n",
"voice = \"Aoede\" # @param [\"Aoede\", \"Puck\", \"Charon\", \"Kore\", \"Fenrir\", \"Leda\", \"Orus\", \"Zephyr\"]\n",
"\n",
"language_code = \"en-US\" # @param [ \"de-DE\", \"en-AU\", \"en-GB\", \"en-IN\", \"en-US\", \"fr-FR\", \"hi-IN\", \"pt-BR\", \"ar-XA\", \"es-ES\", \"fr-CA\", \"id-ID\", \"it-IT\", \"ja-JP\", \"tr-TR\", \"vi-VN\", \"bn-IN\", \"gu-IN\", \"kn-IN\", \"ml-IN\", \"mr-IN\", \"ta-IN\", \"te-IN\", \"nl-NL\", \"ko-KR\", \"cmn-CN\", \"pl-PL\", \"ru-RU\", \"th-TH\"]\n",
"# fmt: on\n",
"\n",
"voice_name = f\"{language_code}-Chirp3-HD-{voice}\"\n",
"voice = texttospeech.VoiceSelectionParams(\n",
" name=voice_name,\n",
" language_code=language_code,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "98d104dc4d27"
},
"outputs": [],
"source": [
"# Perform the text-to-speech request on the text input with the selected\n",
"# voice parameters and audio file type\n",
"response = client.synthesize_speech(\n",
" input=texttospeech.SynthesisInput(text=prompt),\n",
" voice=voice,\n",
" # Select the type of audio file you want returned\n",
" audio_config=texttospeech.AudioConfig(\n",
" audio_encoding=texttospeech.AudioEncoding.MP3\n",
" ),\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fad3a3de36d9"
},
"source": [
"Play the generated audio."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "5e0881955f62"
},
"outputs": [],
"source": [
"display(Audio(response.audio_content))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LYKj5gmDG7nT"
},
"source": [
"### Synthesize speech using streaming processing\n",
"\n",
"Chirp 3 HD voices also support streaming text-to-speech conversion using the `streaming_synthesize` method. Unlike the standard `synthesize_speech`, which handles single requests, `streaming_synthesize` processes continuous streams of text, generating corresponding audio streams.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "HniB988UTUlq"
},
"outputs": [],
"source": [
"prompt = \"\"\"\n",
"Google Cloud Text-to-Speech (TTS) is a powerful API that converts text into natural-sounding audio. Here's a breakdown:\n",
"\n",
"**Key Features:**\n",
"\n",
"* **High-Fidelity Speech:** Leverages advanced AI to generate speech that closely resembles human voices.\n",
"* **Wide Voice Selection:** Offers a vast library of over 380 voices across 50+ languages and variants, catering to diverse needs.\n",
"* **Customization Options:**\n",
" * **Custom Voice:** Create unique voices tailored to your brand or specific requirements using your own audio recordings.\n",
" * **SSML Support:** Utilize Speech Synthesis Markup Language (SSML) to control pronunciation, pacing, and other speech nuances.\n",
"* **Integration Flexibility:** Easily integrate with various applications and devices via REST or gRPC APIs.\n",
"* **Use Cases:**\n",
" * **Voice Assistants:** Powering conversational AI in smart devices, chatbots, and voice-activated applications.\n",
" * **Accessibility:** Enabling screen readers and text-to-speech features for users with disabilities.\n",
" * **E-learning:** Creating engaging and accessible educational content.\n",
" * **Audiobooks and Podcasts:** Producing high-quality audio for audiobooks, podcasts, and other audio content.\n",
" * **Interactive Experiences:** Enhancing user experiences in games, virtual reality, and other interactive applications.\n",
"\n",
"**In essence, Google Cloud Text-to-Speech empowers developers and businesses to:**\n",
"\n",
"* **Enhance user experiences:** Create more engaging and inclusive interactions through natural-sounding speech.\n",
"* **Increase accessibility:** Make information more accessible to a wider audience, including those with visual impairments.\n",
"* **Improve efficiency:** Automate tasks like reading aloud documents, generating voiceovers, and creating interactive voice responses.\n",
"* **Innovate with new applications:** Explore novel use cases by leveraging the power of AI-powered speech synthesis.\n",
"\n",
"If you're looking to add a voice dimension to your applications or projects, Google Cloud Text-to-Speech is a valuable tool to consider.\n",
"\"\"\"\n",
"final_audio_data = process_streaming_audio(\n",
" prompt, voice, display_individual_chunks=False\n",
")\n",
"\n",
"display(Audio(final_audio_data, rate=24000))"
]
}
],
"metadata": {
"colab": {
"name": "get_started_with_chirp_3_hd_voices.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
audio/speech/getting-started/get_started_with_chirp_3_transcription.ipynb
+704
View File
@@ -0,0 +1,704 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Get started with Chirp 3 Transcription\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_transcription.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Faudio%2Fspeech%2Fgetting-started%2Fget_started_with_chirp_3_transcription.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/audio/speech/getting-started/get_started_with_chirp_3_transcription.ipynb\">\n",
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_transcription.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<p>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_transcription.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_transcription.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_transcription.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_transcription.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_transcription.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| Authors |\n",
"| --- |\n",
"| [Katie Nguyen](https://github.com/katiemn) |\n",
"| [Holt Skinner](https://github.com/holtskinner) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"### Chirp 3\n",
"\n",
"This notebook introduces [Chirp 3](https://cloud.google.com/speech-to-text/v2/docs/chirp_3-model), Google's model for converting speech to text in multiple languages.\n",
"\n",
"In this tutorial, you'll learn how to use the Speech-to-Text API V2 to:\n",
"\n",
"- Transcribe an audio file with batch speech recognition\n",
"- Perform a language-agnostic transcription\n",
"- Use Chirp 3 for speaker diarization\n",
"- Perform streaming speech recognition"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install the Speech SDK and other required packages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-cloud-speech ipywebrtc"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5303c05f7aa6"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "qqm0OQpAYCph"
},
"outputs": [],
"source": [
"import json\n",
"import os\n",
"import re\n",
"from collections.abc import Generator\n",
"\n",
"from IPython.display import HTML, Audio, display\n",
"from google.api_core.client_options import ClientOptions\n",
"from google.cloud.speech_v2 import SpeechClient\n",
"from google.cloud.speech_v2.types import cloud_speech\n",
"from ipywebrtc import AudioRecorder, CameraStream"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using the Speech-to-Text API, you must have an existing Google Cloud project and [enable the API](https://console.cloud.google.com/flows/enableapi?apiid=speech.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment).\n",
"\n",
"Please note the **available regions** for Chirp 3, see [documentation](https://cloud.google.com/speech-to-text/v2/docs/chirp_3-model#regional_availability)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "WIQyBhAn_9tK"
},
"outputs": [],
"source": [
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"STT_LOCATION = \"us\" # @param {type: \"string\"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cR5PafOSrV2W"
},
"outputs": [],
"source": [
"! gcloud config set project {PROJECT_ID}\n",
"! gcloud auth application-default login -q\n",
"! gcloud auth application-default set-quota-project {PROJECT_ID}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sP8GBj3tBAC1"
},
"source": [
"### Create client\n",
"\n",
"Initiate the API endpoint and the Speech-to-Text client and define key constants."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rXTVeU1uBBqY"
},
"outputs": [],
"source": [
"client = SpeechClient(\n",
" client_options=ClientOptions(api_endpoint=f\"{STT_LOCATION}-speech.googleapis.com\")\n",
")\n",
"recognizer = client.recognizer_path(PROJECT_ID, STT_LOCATION, \"_\")\n",
"model = \"chirp_3\"\n",
"# Set a timeout for the batch recognition operation\n",
"MAX_AUDIO_LENGTH_SECS = 8 * 60 * 60"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "M7WQQFp_RvGH"
},
"source": [
"### Define helper functions"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "pWrNWdV_RxGn"
},
"outputs": [],
"source": [
"def print_transcript(results: list[cloud_speech.SpeechRecognitionResult]) -> None:\n",
" for result in results:\n",
" display(\n",
" HTML(\n",
" f\"\"\"<div style=\"word-break: break-all;\">{result.alternatives[0].transcript}</div>\"\"\"\n",
" )\n",
" )\n",
"\n",
"\n",
"def generate_audio_chunks(\n",
" audio_content: bytes, chunk_size: int\n",
") -> Generator[bytes, None, None]:\n",
" \"\"\"Splits a byte string (like audio data) into smaller, equal-sized chunks.\n",
"\n",
" Args:\n",
" audio_content: The raw byte data of the audio.\n",
" chunk_size: The desired size for each audio chunk in bytes.\n",
"\n",
" Yields:\n",
" A series of audio chunks.\n",
" \"\"\"\n",
" # Loop through the audio content, stepping by chunk_size each time\n",
" for start_index in range(0, len(audio_content), chunk_size):\n",
" # The end_index is the start_index plus the chunk size\n",
" end_index = start_index + chunk_size\n",
" # Yield the slice of audio data for the current chunk\n",
" yield audio_content[start_index:end_index]\n",
"\n",
"\n",
"def group_utterances_by_speaker_from_file(json_file_path: str) -> dict:\n",
" \"\"\"Reads a JSON file containing transcribed words and groups them into sentences spoken by each speaker.\"\"\"\n",
" with open(json_file_path, encoding=\"utf-8\") as f:\n",
" json_data_string = f.read()\n",
" words_regex = r'\"words\":\\s*(\\[.*?\\])'\n",
" match = re.search(words_regex, json_data_string, re.DOTALL)\n",
"\n",
" words_list = json.loads(match.group(1))\n",
" dialogue = []\n",
" current_speaker = None\n",
" current_utterance_words = []\n",
" current_speaker = words_list[0][\"speakerLabel\"]\n",
"\n",
" for item in words_list:\n",
" word = item[\"word\"]\n",
" speaker = item[\"speakerLabel\"]\n",
" # Check if the speaker has changed\n",
" if speaker != current_speaker:\n",
" dialogue.append(\n",
" {\"speaker\": current_speaker, \"text\": \" \".join(current_utterance_words)}\n",
" )\n",
" # Start a new utterance\n",
" current_speaker = speaker\n",
" current_utterance_words = [word]\n",
" else:\n",
" # Continue the current utterance\n",
" current_utterance_words.append(word)\n",
" # Add the final pending utterance\n",
" if current_speaker is not None:\n",
" dialogue.append(\n",
" {\"speaker\": current_speaker, \"text\": \" \".join(current_utterance_words)}\n",
" )\n",
"\n",
" return {\"dialogue\": dialogue}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VPVDNRyVxquo"
},
"source": [
"## Transcribe using Chirp 3\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LfBiMs0bNwhV"
},
"source": [
"### Online/Synchronous speech recognition\n",
"\n",
"You can use online (synchronous) speech recognition for audio files less than 1 minute long.\n",
"For this first request, Run the following cell to download and play the audio you'll be transcribing. If you'd like to use a different audio clip, modify the `audio_url` variable below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "yE_-iirV7UDA"
},
"outputs": [],
"source": [
"audio_url = \"https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/audio_summary_clean_energy_short.mp3\"\n",
"audio_filename = os.path.basename(audio_url)\n",
"! wget {audio_url} -O {audio_filename}\n",
"\n",
"display(Audio(filename=audio_filename))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lAfgmnZyPo1f"
},
"source": [
"Now, you'll send the `recognize` request. The transcription will be returned as part of the response and displayed in HTML for better visualization in this notebook."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "egBk4a3t0lUD"
},
"outputs": [],
"source": [
"config = cloud_speech.RecognitionConfig(\n",
" auto_decoding_config=cloud_speech.AutoDetectDecodingConfig(),\n",
" model=model,\n",
" language_codes=[\"en-US\"],\n",
")\n",
"\n",
"with open(audio_filename, \"rb\") as f:\n",
" audio_content = f.read()\n",
"\n",
"request = cloud_speech.RecognizeRequest(\n",
" recognizer=recognizer,\n",
" config=config,\n",
" content=audio_content,\n",
")\n",
"\n",
"response = client.recognize(request=request)\n",
"\n",
"print_transcript(response.results)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oJSi0bFQQYel"
},
"source": [
"### Perform a language-agnostic transcription\n",
"\n",
"In this next request, you'll perform a language-agnostic transcription. This means that Chirp 3 will automatically identify and transcribe the dominant language spoken in the audio, which is essential for multilingual applications.\n",
"\n",
"In this next example, you'll use a Spanish audio clip saved in Cloud Storage. To see a full list of the languages available for transcription, check the [documentation](https://cloud.google.com/speech-to-text/v2/docs/chirp_3-model#language_availability_for_transcription)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "zcTrgN9xHDXR"
},
"outputs": [],
"source": [
"audio_url = (\n",
" \"https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/spanish.wav\"\n",
")\n",
"audio_gcs_uri = audio_url.replace(\"https://storage.googleapis.com/\", \"gs://\")\n",
"\n",
"display(Audio(url=audio_url))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fB4cDExAMCnb"
},
"source": [
"This request is the similar to the previous one, except this time, you'll set `language_codes=[\"auto\"]`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "CphxakXgBEsz"
},
"outputs": [],
"source": [
"config = cloud_speech.RecognitionConfig(\n",
" auto_decoding_config=cloud_speech.AutoDetectDecodingConfig(),\n",
" model=model,\n",
" language_codes=[\"auto\"],\n",
")\n",
"request = cloud_speech.RecognizeRequest(\n",
" recognizer=recognizer,\n",
" config=config,\n",
" uri=audio_gcs_uri,\n",
")\n",
"\n",
"response = client.recognize(request=request)\n",
"\n",
"print_transcript(response.results)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1fOGjzlRMFMw"
},
"source": [
"### Speaker Diarization (Batch Recognition)\n",
"\n",
"Chirp 3 also supports speaker diarization, which means it can automatically identify the different speakers in a single-channel audio sample. See the [documentation](https://cloud.google.com/speech-to-text/v2/docs/chirp_3-model#language_availability_for_diarization) for a list of supported available languages for diarization.\n",
"\n",
"In this example, you'll also use the `batch_recognize` method to transcribe an audio file in Cloud Storage and save the output in Cloud Storage."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "FosRgcFqEMXX"
},
"outputs": [],
"source": [
"audio_url = \"https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/Chirp-3-Docs-Dive.mp3\"\n",
"audio_gcs_uri = audio_url.replace(\"https://storage.googleapis.com/\", \"gs://\")\n",
"display(Audio(url=audio_url))\n",
"\n",
"gcs_output_folder = \"gs://[your-bucket-path]\" # @param {type: \"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eb_gkdN9BvnP"
},
"source": [
"In order to enable speaker diarization, set the `diarization_config` in the `features` parameter of the `RecognitionConfig`.\n",
"\n",
"You'll also set your `gcs_output_folder` in a `RecognitionOutputConfig` so the transcription will be saved in Cloud Storage. To display the transcription, you'll copy the output JSON file and use a helper function to format it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Odgee64-xA2O"
},
"outputs": [],
"source": [
"config = cloud_speech.RecognitionConfig(\n",
" auto_decoding_config=cloud_speech.AutoDetectDecodingConfig(),\n",
" features=cloud_speech.RecognitionFeatures(\n",
" diarization_config=cloud_speech.SpeakerDiarizationConfig(),\n",
" ),\n",
" model=model,\n",
" language_codes=[\"en-US\"],\n",
")\n",
"\n",
"files = [cloud_speech.BatchRecognizeFileMetadata(uri=audio_gcs_uri)]\n",
"\n",
"request = cloud_speech.BatchRecognizeRequest(\n",
" recognizer=recognizer,\n",
" config=config,\n",
" files=files,\n",
" recognition_output_config=cloud_speech.RecognitionOutputConfig(\n",
" gcs_output_config=cloud_speech.GcsOutputConfig(uri=gcs_output_folder),\n",
" ),\n",
")\n",
"operation = client.batch_recognize(request=request)\n",
"response = operation.result(timeout=MAX_AUDIO_LENGTH_SECS)\n",
"\n",
"transcript = response.results[audio_gcs_uri].uri\n",
"\n",
"!gsutil cp {transcript} output.json\n",
"\n",
"print(json.dumps(group_utterances_by_speaker_from_file(\"output.json\"), indent=4))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yRFD1ByUDuqW"
},
"source": [
"### Streaming speech recognition\n",
"\n",
"In the following cells you'll simulate transcribing text from an audio stream. To start, you'll record an audio clip with your microphone by running the following cell."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "K2OrVxHeMxV2"
},
"outputs": [],
"source": [
"if \"google.colab\" in sys.modules:\n",
" from google.colab import output\n",
"\n",
" output.enable_custom_widget_manager()\n",
"\n",
"camera = CameraStream(constraints={\"audio\": True, \"video\": False})\n",
"recorder = AudioRecorder(stream=camera)\n",
"recorder"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PXhPcYx2ECJ-"
},
"source": [
"Once the audio is captured and you've stopped recording, you'll use FFmpeg to convert and save the clip to a MP3 file for processing."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "TtS36HTdP1hM"
},
"outputs": [],
"source": [
"with open(\"recording.webm\", \"wb\") as f:\n",
" f.write(recorder.audio.value)\n",
"\n",
"!ffmpeg -i recording.webm -vn -ar 44100 -ac 2 -f mp3 recording.mp3\n",
"audio_file = \"recording.mp3\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "swn2txVDEkSx"
},
"source": [
"Now, you'll read the audio file and generate audio chunks to simulate streaming from a helper function. You'll then use the `streaming_recognize` method to get the transcription from each audio chunk with help from a generator function to correctly structure the data stream."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "lwMRXTlf7q-d"
},
"outputs": [],
"source": [
"CHUNK_SIZE = 3200\n",
"\n",
"recognition_config = cloud_speech.RecognitionConfig(\n",
" auto_decoding_config=cloud_speech.AutoDetectDecodingConfig(),\n",
" language_codes=[\"auto\"],\n",
" model=model,\n",
")\n",
"\n",
"\n",
"def create_streaming_requests(\n",
" audio_file_path: str,\n",
") -> Generator[cloud_speech.StreamingRecognizeRequest, None, None]:\n",
" \"\"\"Prepares and yields all necessary requests for streaming speech recognition.\n",
"\n",
" First, it yields the configuration request, then it reads an audio file\n",
" and yields its content in chunks.\n",
"\n",
" Args:\n",
" audio_file_path: The path to the local audio file.\n",
"\n",
" Yields:\n",
" A stream of StreamingRecognizeRequest objects.\n",
" \"\"\"\n",
" # a. First, yield the initial configuration request.\n",
" config_request = cloud_speech.StreamingRecognizeRequest(\n",
" recognizer=recognizer,\n",
" streaming_config=cloud_speech.StreamingRecognitionConfig(\n",
" config=recognition_config,\n",
" ),\n",
" )\n",
" yield config_request\n",
"\n",
" # b. Second, read the audio file and stream it in chunks.\n",
" with open(audio_file_path, \"rb\") as f:\n",
" audio_content = f.read()\n",
"\n",
" for chunk in generate_audio_chunks(audio_content, CHUNK_SIZE):\n",
" yield cloud_speech.StreamingRecognizeRequest(audio=chunk)\n",
"\n",
"\n",
"responses = client.streaming_recognize(requests=create_streaming_requests(audio_file))\n",
"\n",
"print(\"Streaming transcripts:\")\n",
"all_transcripts = []\n",
"for response in responses:\n",
" for result in response.results:\n",
" transcript = result.alternatives[0].transcript\n",
" print(transcript)\n",
" all_transcripts.append(transcript)\n",
"\n",
"final_transcript = \" \".join(all_transcripts)\n",
"print(f\"\\n--- Final Combined Transcript ---\\n{final_transcript}\")"
]
}
],
"metadata": {
"colab": {
"name": "get_started_with_chirp_3_transcription.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
audio/speech/getting-started/get_started_with_gemini_tts_voices.ipynb
+1117
View File
File diff suppressed because it is too large Load Diff
audio/speech/use-cases/podcast/multi-speaker-podcast.ipynb
+531
View File
@@ -0,0 +1,531 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2024 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Create a Multi-Speaker Podcast with Gemini 2.0 & Text-to-Speech\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/use-cases/podcast/multi-speaker-podcast.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Faudio%2Fspeech%2Fuse-cases%2Fpodcast%2Fmulti-speaker-podcast.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/audio/speech/use-cases/podcast/multi-speaker-podcast.ipynb\">\n",
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/use-cases/podcast/multi-speaker-podcast.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<p>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/use-cases/podcast/multi-speaker-podcast.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/use-cases/podcast/multi-speaker-podcast.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/use-cases/podcast/multi-speaker-podcast.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/use-cases/podcast/multi-speaker-podcast.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/use-cases/podcast/multi-speaker-podcast.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| | |\n",
"|-|-|\n",
"| Author(s) | [Souvik Mukherjee](https://github.com/talktosauvik/), [Holt Skinner](https://github.com/holtskinner) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"This notebook demonstrates how to use the [Gemini API in Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs) to generate an engaging multi-speaker podcast using studio voices in the [Text-to-Speech API](https://cloud.google.com/text-to-speech). \n",
"\n",
"> **NOTE:** [Generate dialogue with multiple speakers](https://cloud.google.com/text-to-speech/docs/create-dialogue-with-multispeakers) is only available to projects in allowlist. Please [contact Google Cloud](https://cloud.google.com/contact) if you want to use this feature.\n",
"\n",
"This can be useful for creating interviews, interactive storytelling, video games, e-learning platforms, and accessibility solutions.\n",
"\n",
"The steps performed include:\n",
"\n",
"- Load a PDF file from a Google Cloud Storage bucket or public URL\n",
"- Summarize the content using Gemini 2.0 Flash\n",
"- Return a pre-defined JSON schema using [Controlled Generation](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/control-generated-output)\n",
"- Create a multi speaker conversation from the JSON script using Text-to-Speech.\n",
"- Generate the audio as MP3 file.\n",
"\n",
"For a more advanced example using LangGraph, check out [Build Your Own AI Podcasting Agent with LangGraph & Gemini: AI-Powered Podcast Creation with Automated Research, Writing, and Refinement](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/orchestration/langgraph_gemini_podcast.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Google Gen AI SDK for Python\n",
"\n",
"Install the following packages required to execute this notebook."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-genai google-cloud-texttospeech"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "R5Xep4W9lq-Z"
},
"source": [
"### Restart runtime\n",
"\n",
"To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel.\n",
"\n",
"The restart might take a minute or longer. After it's restarted, continue to the next step."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XRvKdaPDTznN"
},
"outputs": [],
"source": [
"import IPython\n",
"\n",
"app = IPython.Application.instance()\n",
"app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SbmM4z7FOBpM"
},
"source": [
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ The kernel is going to restart. Please wait until it is finished before continuing to the next step. ⚠️</b>\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticating your notebook environment\n",
"\n",
"* If you are using **Colab** to run this notebook, run the cell below and continue.\n",
"* If you are using **Vertex AI Workbench**, check out the setup instructions [here](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/setup-env)."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"# Additional authentication is required for Google Colab\n",
"if \"google.colab\" in sys.modules:\n",
" # Authenticate user to Google Cloud\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information and create client\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment).\n",
"\n",
"\n",
"* Please note the **available regions** for Text-to-Speech, see [documentation](https://cloud.google.com/text-to-speech/docs/endpoints)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"from google import genai\n",
"\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"VERTEXAI_LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
"TTS_LOCATION = \"us\" # @param {type:\"string\"}\n",
"\n",
"! gcloud config set project {PROJECT_ID}\n",
"! gcloud auth application-default set-quota-project {PROJECT_ID}\n",
"! gcloud auth application-default login -q"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"id": "698a012fd0cc"
},
"outputs": [],
"source": [
"client = genai.Client(enterprise=True, project=PROJECT_ID, location=VERTEXAI_LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5303c05f7aa6"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"id": "6fc324893334"
},
"outputs": [],
"source": [
"import json\n",
"\n",
"from IPython.display import Audio\n",
"from google.api_core.client_options import ClientOptions\n",
"from google.cloud import texttospeech_v1beta1 as texttospeech\n",
"from google.genai.types import GenerateContentConfig, Part"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ac0c378bb46a"
},
"source": [
"### Load the Gemini 2.0 Flash model\n",
"\n",
"To learn more about all [Gemini models on Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-models)."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"id": "0b696580b27c"
},
"outputs": [],
"source": [
"MODEL_ID = \"gemini-2.0-flash-001\" # @param {type: \"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e43229f3ad4f"
},
"source": [
"### Define constants"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"id": "cf93d5f0ce00"
},
"outputs": [],
"source": [
"DEFAULT_LANGUAGE = \"en-US\"\n",
"STUDIO_VOICE = \"en-US-Studio-MultiSpeaker\"\n",
"\n",
"SYSTEM_INSTRUCTION = \"\"\"You are a podcast writer. Your task is to generate a short podcast-style dialogue between two speakers, Speaker R and Speaker S\"\"\"\n",
"\n",
"response_schema = {\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"dialogue\": {\n",
" \"type\": \"array\",\n",
" \"items\": {\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"speaker\": {\"type\": \"string\"},\n",
" \"line\": {\"type\": \"string\"},\n",
" },\n",
" \"required\": [\"speaker\", \"line\"],\n",
" },\n",
" }\n",
" },\n",
" \"required\": [\"dialogue\"],\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MzeTv4NsxGoN"
},
"source": [
"### Helper functions"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"id": "qO9cgw1-xGWF"
},
"outputs": [],
"source": [
"def generate_podcast_script(file_uri: str) -> list:\n",
" \"\"\"Generates a podcast script using Gemini with controlled JSON output.\"\"\"\n",
" prompt = \"\"\"\n",
" The dialogue should be engaging and natural, with each speaker contributing roughly equal amounts. Return the dialogue as a JSON array of objects, where each object has a 'speaker' (either 'R' or 'S') and a 'line' property.\n",
"\n",
" Use the following information to create the content for the podcast dialogue:\n",
" \"\"\"\n",
"\n",
" try:\n",
" response = client.models.generate_content(\n",
" model=MODEL_ID,\n",
" contents=[\n",
" prompt,\n",
" Part.from_uri(file_uri=file_uri, mime_type=\"application/pdf\"),\n",
" ],\n",
" config=GenerateContentConfig(\n",
" temperature=1,\n",
" top_p=0.95,\n",
" max_output_tokens=8192,\n",
" response_mime_type=\"application/json\",\n",
" response_schema=response_schema,\n",
" ),\n",
" )\n",
" generated_json = json.loads(response.text)\n",
" return generated_json[\"dialogue\"]\n",
" except (json.JSONDecodeError, KeyError) as e:\n",
" print(f\"Error generating or parsing JSON script: {e}. Returning empty list.\")\n",
" return []\n",
"\n",
"\n",
"def synthesize_podcast(dialogue: list[dict], output_file: str):\n",
" \"\"\"Synthesizes speech for a podcast using MultiSpeakerMarkup.\"\"\"\n",
" tts_client = texttospeech.TextToSpeechClient(\n",
" client_options=ClientOptions(\n",
" api_endpoint=f\"{TTS_LOCATION}-texttospeech.googleapis.com\"\n",
" )\n",
" )\n",
" multi_speaker_markup = texttospeech.MultiSpeakerMarkup(\n",
" turns=[\n",
" texttospeech.MultiSpeakerMarkup.Turn(\n",
" text=turn_data[\"line\"], speaker=turn_data[\"speaker\"]\n",
" )\n",
" for turn_data in dialogue\n",
" ]\n",
" )\n",
"\n",
" response = tts_client.synthesize_speech(\n",
" input=texttospeech.SynthesisInput(multi_speaker_markup=multi_speaker_markup),\n",
" voice=texttospeech.VoiceSelectionParams(\n",
" language_code=DEFAULT_LANGUAGE, name=STUDIO_VOICE\n",
" ),\n",
" audio_config=texttospeech.AudioConfig(\n",
" audio_encoding=texttospeech.AudioEncoding.MP3\n",
" ),\n",
" )\n",
"\n",
" with open(output_file, \"wb\") as out:\n",
" out.write(response.audio_content)\n",
" print(f\"Audio content written to file `{output_file}`\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ltb_QilKxW01"
},
"source": [
"## Call the Text-to-Speech API with script content"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZDGzyxkbx0Py"
},
"source": [
"### Generate the podcast script from the content\n",
"\n",
"For this example, we will be using the [Gemini 2.0 paper from arXiv](https://arxiv.org/abs/2403.05530).\n",
"\n",
"You can replace this with the URL with any publicly-accessible PDF."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "DoJgwgWRxYBS"
},
"outputs": [],
"source": [
"PDF_URL = \"gs://github-repo/2403_05530.pdf\" # @param {type: \"string\"}\n",
"dialogue = generate_podcast_script(PDF_URL)\n",
"print(\"Generated Dialogue:\")\n",
"print(dialogue)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "h9k_otpSyBd4"
},
"source": [
"### Write the audio content into the output file"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XtD1oG7ZyFMm"
},
"outputs": [],
"source": [
"if dialogue:\n",
" output_filename = \"podcast_output.mp3\"\n",
" synthesize_podcast(dialogue, output_filename)\n",
"else:\n",
" print(\"No dialogue generated. Skipping audio synthesis.\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SsbWsbJMyGqP"
},
"source": [
"### Listen to the audio file"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rCsneZpOyKwk"
},
"outputs": [],
"source": [
"Audio(output_filename)"
]
}
],
"metadata": {
"colab": {
"name": "multi-speaker-podcast.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
audio/speech/use-cases/storytelling/macbeth_the_sitcom.json
+292
View File
@@ -0,0 +1,292 @@
{
"title": "Macbeth: The Scottish Sitcom",
"characters": [
{
"name": "Narrator",
"gender": "Female"
},
{
"name": "Macbeth",
"gender": "Male"
},
{
"name": "Banquo",
"gender": "Male"
},
{
"name": "Lady Macbeth",
"gender": "Female"
},
{
"name": "King Duncan",
"gender": "Male"
},
{
"name": "Witch 1",
"gender": "Female"
},
{
"name": "Witch 2",
"gender": "Female"
},
{
"name": "Witch 3",
"gender": "Female"
},
{
"name": "Macduff",
"gender": "Male"
}
],
"scenes": [
{
"setting": "A particularly damp patch of Scottish moorland. Sounds of squelching mud and distant sheep.",
"dialogue": [
{
"speaker": "Narrator",
"line": "Our story opens on a rather dreary Tuesday in Scotland. General Macbeth and his pal Banquo are trudging home from a surprisingly successful skirmish. Little do they know, their day is about to get... weird."
},
{
"speaker": "Macbeth",
"line": "Ugh, my boots are ruined. Absolutely ruined. I told the quartermaster we needed galoshes!"
},
{
"speaker": "Banquo",
"line": "Cheer up, Mac. We won the battle. And got a promotion out of it. Thane of Cawdor! Fancy that."
},
{
"speaker": "Macbeth",
"line": "Yeah, yeah. Still, this mud... hang on. Who are these ladies?"
},
{
"speaker": "Narrator",
"line": "Appearing out of the mist, looking less like mystical beings and more like they'd just survived a particularly rough music festival, were three figures."
},
{
"speaker": "Witch 1",
"line": "All hail, Macbeth! Hail to thee, Thane of Glamis!"
},
{
"speaker": "Witch 2",
"line": "All hail, Macbeth! Hail to thee, Thane of Cawdor!"
},
{
"speaker": "Witch 3",
"line": "All hail, Macbeth! That shalt be king hereafter!"
},
{
"speaker": "Banquo",
"line": "Right... Is this a joke? Are you doing performance art? Because frankly, it's not very good."
},
{
"speaker": "Witch 1",
"line": "Lesser than Macbeth, and greater!"
},
{
"speaker": "Witch 2",
"line": "Not so happy, yet much happier!"
},
{
"speaker": "Witch 3",
"line": "Thou shalt get kings, though thou be none. So all hail, Banquo!"
},
{
"speaker": "Macbeth",
"line": "Wait wait wait. Back up. King? *Me*? Are you sure? Because I'm pretty sure King Duncan has, like, a whole line of succession. It's quite complicated."
},
{
"speaker": "Banquo",
"line": "And 'not so happy, yet much happier'? What does that even mean? Are you saying I'll be mildly content?"
},
{
"speaker": "Witch 1",
"line": "Ta-ta!"
},
{
"speaker": "Narrator",
"line": "And with a puff of suspiciously pungent smoke, the witches vanished. Leaving Macbeth and Banquo utterly bewildered."
},
{
"speaker": "Macbeth",
"line": "So... king, huh? Sounds like a lot of paperwork."
}
]
},
{
"setting": "The rather cluttered living room of Macbeth's castle. There's a half-finished tapestry project and a pile of dirty armour in the corner.",
"dialogue": [
{
"speaker": "Narrator",
"line": "Back at Casa Macbeth, Lady Macbeth was doing... well, whatever formidable ladies of the castle do when their husbands are off fighting. Probably passive-aggressively tidying."
},
{
"speaker": "Lady Macbeth",
"line": "Honestly, the state of this place... Ah, a letter from Mac!"
},
{
"speaker": "Narrator",
"line": "She eagerly opened the scroll and began to read. Her eyes widened as she took in the news of the prophecies."
},
{
"speaker": "Lady Macbeth",
"line": "'Thane of Cawdor... and shall be king hereafter!' Oh, Macbeth! My sweet, slightly clueless Macbeth! A king! This is brilliant! The renovations we could do! We could finally afford that indoor plumbing!"
},
{
"speaker": "Lady Macbeth",
"line": "But he's too full o' the milk of human kindness... more like the lukewarm oat milk of human indecision. He'd never just seize it! He needs... encouragement. Firm encouragement. Right, where's my planner? Duncan's visiting tonight? Excellent!"
},
{
"speaker": "Narrator",
"line": "A look of ambitious glee, and perhaps a touch of mania, crossed Lady Macbeth's face. She was planning a dinner party. And something else."
}
]
},
{
"setting": "A slightly uncomfortable guest bedroom in Macbeth's castle. It smells faintly of lavender potpourri and existential dread.",
"dialogue": [
{
"speaker": "Narrator",
"line": "Later that night. King Duncan, a genuinely nice old chap, was fast asleep. Macbeth, however, was having a bit of a crisis."
},
{
"speaker": "Macbeth",
"line": "(Whispering) Is that a dagger I see before me? Or did I just leave the letter opener on the stairs again?"
},
{
"speaker": "Lady Macbeth",
"line": "(Whispering fiercely) Macbeth! Get a move on! The guards are passed out, thanks to my special 'nighty-night' brew! Honestly, it's like trying to get a toddler to take a nap!"
},
{
"speaker": "Macbeth",
"line": "But... he's so nice! He called me 'valiant cousin' today! And he gave you that diamond!"
},
{
"speaker": "Lady Macbeth",
"line": "The diamond is lovely, yes, but think of the crown jewels! Stop dithering and just... do the thing!"
},
{
"speaker": "Narrator",
"line": "With a deep sigh, Macbeth steelled himself. What followed was less a dramatic assassination and more a fumbled, awkward encounter involving confused shouting and the clatter of dropped objects."
},
{
"speaker": "Macbeth",
"line": "(Off-mike, sounds of bumping, a yelp, something falling)"
},
{
"speaker": "Lady Macbeth",
"line": "Oh, honestly! Did you stab the duvet again? Give it here, I'll do it!"
},
{
"speaker": "Narrator",
"line": "Eventually, it was done. Macbeth emerged, looking pale and clutching the wrong set of keys."
},
{
"speaker": "Macbeth",
"line": "I think... I did it? My hands are red... is that paint?"
},
{
"speaker": "Lady Macbeth",
"line": "It's blood, you idiot! Go wash up. And try not to look so suspicious. Honestly, the neighbours will talk."
}
]
},
{
"setting": "The grand dining hall of Macbeth's castle. A dinner party is in full swing, albeit a slightly tense one.",
"dialogue": [
{
"speaker": "Narrator",
"line": "Months later. Macbeth was King. But things weren't exactly running smoothly. The Wi-Fi was spotty, the servants were muttering, and there was the small issue of Macbeth having had his friend Banquo 'disappeared'. Now, it was time for a celebratory banquet."
},
{
"speaker": "Lady Macbeth",
"line": "Macbeth, dear, do sit down. You're hovering. And stop staring at that empty chair like it owes you money."
},
{
"speaker": "Macbeth",
"line": "But... but Banquo! He's sitting right there! And he looks... translucent! And covered in glitter?"
},
{
"speaker": "Narrator",
"line": "Indeed, there in Banquo's seat was Banquo. Looking quite pale and spectral, and possibly shimmering slightly."
},
{
"speaker": "Banquo",
"line": "Hey Mac! Just popping by. Awkward timing? Sorry. Just wanted to say hi. And maybe haunt you a little."
},
{
"speaker": "Macbeth",
"line": "(Shouting) Avaunt! And quit my sight! Your bones are marrowless! Your blood is cold! And your shimmering is very distracting!"
},
{
"speaker": "Lady Macbeth",
"line": "(To guests, forced smile) Oh, pay him no mind! The king has a... a touch of the vapours! Happens after a big plate of haggis. Anyone for more mead?"
},
{
"speaker": "Banquo",
"line": "(To Macbeth) You know, I always suspected you had something to do with it. Just wanted to confirm. Cool castle, though. Bit drafty."
},
{
"speaker": "Macbeth",
"line": "Go! Go! I am a man again!"
},
{
"speaker": "Banquo",
"line": "Alright, alright, party pooper. See ya 'round!"
},
{
"speaker": "Narrator",
"line": "And with a final spectral sigh, Banquo's ghost faded. The guests, however, were exchanging deeply uncomfortable glances."
},
{
"speaker": "Lady Macbeth",
"line": "(Whispering to Macbeth) You absolute imbecile! Do you want everyone to know?!"
}
]
},
{
"setting": "The ramparts of Dunsinane Castle. It looks tired and slightly crumbling, much like Macbeth's reign.",
"dialogue": [
{
"speaker": "Narrator",
"line": "Time passed, as it tends to do in sitcoms. Macbeth became increasingly tyrannical and obsessed with the witches' less helpful prophecies. Armies were approaching. Things were looking bleak. Lady Macbeth was apparently off... sleepwalking? Doing the laundry? The script is a bit vague on that part."
},
{
"speaker": "Macbeth",
"line": "(To himself) 'Till Birnam Wood remove to Dunsinane'... Yeah, right. Trees don't just get up and walk. That's just silly. And 'none of woman born shall harm Macbeth'... excellent! I'm basically invincible! Though I really should have asked for clarification on the 'woman born' part."
},
{
"speaker": "Narrator",
"line": "Just then, a large, slightly cross man wearing plaid and looking rather familiar stormed onto the battlements."
},
{
"speaker": "Macduff",
"line": "Right, Macbeth! You and I need to have a little chat about my family. And yours. Preferably with swords."
},
{
"speaker": "Macbeth",
"line": "Oh, it's you. Look, I'm not fighting anyone 'of woman born'. It's in the prophecy! Get lost!"
},
{
"speaker": "Macduff",
"line": "Yeah, well, about that... see, I wasn't exactly 'born' in the usual sense. More of a... timely medical intervention? The nurses called it a C-section."
},
{
"speaker": "Macbeth",
"line": "...Oh. Right. That. Well, the witches were a bit vague! This is precisely why you need footnotes on prophecies!"
},
{
"speaker": "Narrator",
"line": "And so, the inevitable confrontation began. It wasn't epic, or glorious. It was mostly just two middle-aged men awkwardly trying to swing heavy swords without pulling a muscle."
},
{
"speaker": "Macbeth",
"line": "(Grunting) Ow! My back! Is that it? Am I... done now?"
},
{
"speaker": "Narrator",
"line": "Yup, he was done. Macbeth's reign, like a dodgy microwave, had come to a sputtering, anticlimactic end. Malcolm became king, order was restored, and presumably, the central heating in the castle was finally fixed. As for the witches? Well, they probably opened a consulting firm. The end."
}
]
}
]
}
audio/speech/use-cases/storytelling/storytelling.ipynb
+678
View File
@@ -0,0 +1,678 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2024 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Narrate a Multi-character Story with Gemini and Text-to-Speech\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/use-cases/storytelling/storytelling.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Faudio%2Fspeech%2Fuse-cases%2Fstorytelling%2Fstorytelling.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/audio/speech/use-cases/storytelling/storytelling.ipynb\">\n",
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/use-cases/storytelling/storytelling.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<p>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/use-cases/storytelling/storytelling.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/use-cases/storytelling/storytelling.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/use-cases/storytelling/storytelling.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/use-cases/storytelling/storytelling.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/use-cases/storytelling/storytelling.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "42cc0c827290"
},
"source": [
"| Author |\n",
"| --- |\n",
"| [Holt Skinner](https://github.com/holtskinner) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"This notebook demonstrates how to use the [Gemini API on Agent Platform](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/google-models) to generate a play script and create an audio performance with each character having a distinct voice using the [Text-to-Speech API](https://cloud.google.com/text-to-speech).\n",
"\n",
"The steps performed include:\n",
"\n",
"- Create a story using Gemini\n",
"- Assign each character to a voice.\n",
"- Synthesize each line based on character voice.\n",
"- Combine audio files into one MP3 file."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aed92deeb4a0"
},
"source": [
"### Costs\n",
"\n",
"This tutorial uses billable components of Google Cloud:\n",
"\n",
"* Gemini API in Vertex AI\n",
"* Text-to-Speech\n",
"* Cloud Storage\n",
"\n",
"Learn about [Text-to-Speech pricing](https://cloud.google.com/text-to-speech/pricing),\n",
"and [Cloud Storage pricing](https://cloud.google.com/storage/pricing),\n",
"and use the [Pricing Calculator](https://cloud.google.com/products/calculator/)\n",
"to generate a cost estimate based on your projected usage."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "i7EUnXsZhAGF"
},
"source": [
"## Getting Started\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "NNSWiCNPjh_p"
},
"source": [
"### Install Google Gen AI SDK, other packages and their dependencies\n",
"\n",
"Install the following packages required to execute this notebook."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2b4ef9b72d43"
},
"outputs": [],
"source": [
"%pip install --upgrade -qqq google-genai google-cloud-texttospeech pydub pandas tqdm"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "J3a6juReciKR"
},
"source": [
"If you're running on a Mac, you will need to install [FFmpeg](https://ffmpeg.org/)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "BzCUF4oqciKR"
},
"outputs": [],
"source": [
"%%bash\n",
"# Check if the system is macOS\n",
"if [[ \"$(uname -s)\" == \"Darwin\" ]]; then\n",
" # Install using Homebrew\n",
" brew install ffmpeg\n",
"fi"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YbMFqPZ3tnwz"
},
"source": [
"Set the project and region.\n",
"\n",
"* Please note the **available regions** for Text-to-Speech, see [documentation](https://cloud.google.com/text-to-speech/docs/endpoints)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "GjSsu6cmUdEx"
},
"outputs": [],
"source": [
"# Use the environment variable if the user doesn't provide Project ID.\n",
"import os\n",
"\n",
"from google import genai\n",
"\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"global\")\n",
"\n",
"client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)\n",
"\n",
"TTS_LOCATION = \"global\" # @param {type:\"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "opUxT_k5TdgP"
},
"source": [
"### Authenticating your notebook environment\n",
"\n",
"* If you are using **Colab** to run this notebook, run the cell below and continue.\n",
"* If you are using **Vertex AI Workbench**, check out the setup instructions [here](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/setup-env)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vbNgv4q1T2Mi"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"# Additional authentication is required for Google Colab\n",
"if \"google.colab\" in sys.modules:\n",
" # Authenticate user to Google Cloud\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()\n",
"\n",
"! gcloud config set project {PROJECT_ID}\n",
"! gcloud auth application-default set-quota-project {PROJECT_ID}\n",
"! gcloud auth application-default login -q"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "960505627ddf"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PyQmSRbKA8r-"
},
"outputs": [],
"source": [
"import os\n",
"import random\n",
"import re\n",
"\n",
"import pandas as pd\n",
"from IPython.display import Audio\n",
"from google.api_core.client_options import ClientOptions\n",
"from google.cloud import texttospeech_v1beta1 as texttospeech\n",
"from google.genai.types import GenerateContentConfig\n",
"from pydantic import BaseModel\n",
"from pydub import AudioSegment\n",
"from tqdm import tqdm"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "v4gUI8WqciKS"
},
"source": [
"### Define constants"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6Pl3un_YciKS"
},
"outputs": [],
"source": [
"DEFAULT_LANGUAGE = \"en\"\n",
"NARRATOR_VOICE = \"en-GB-Chirp3-HD-Zubenelgenubi\"\n",
"DEFAULT_VOICE = \"en-GB-Chirp3-HD-Umbriel\"\n",
"\n",
"SILENCE_LENGTH = 200 # In Milliseconds\n",
"TXT_EXTENSION = \".txt\"\n",
"\n",
"api_endpoint = \"texttospeech.googleapis.com\"\n",
"if TTS_LOCATION != \"global\":\n",
" api_endpoint = f\"{TTS_LOCATION}-{api_endpoint}\"\n",
"\n",
"tts_client = texttospeech.TextToSpeechClient(\n",
" client_options=ClientOptions(api_endpoint=api_endpoint)\n",
")\n",
"\n",
"SYSTEM_INSTRUCTION = \"\"\"You are a creative and ambitious play writer. Your goal is to write a play for audio performance. Include a narrator character to describe the scenes and actions occurring.\"\"\"\n",
"\n",
"MODEL_ID = \"gemini-3.5-flash\"\n",
"\n",
"\n",
"class Character(BaseModel):\n",
" name: str\n",
" gender: str\n",
"\n",
"\n",
"class DialogueLine(BaseModel):\n",
" speaker: str\n",
" line: str\n",
"\n",
"\n",
"class Scene(BaseModel):\n",
" setting: str\n",
" dialogue: list[DialogueLine]\n",
"\n",
"\n",
"class Story(BaseModel):\n",
" title: str\n",
" characters: list[Character]\n",
" scenes: list[Scene]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "v5CEc4-Wrjk2"
},
"source": [
"### Helper functions"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "kYx2wwhjrmD6"
},
"outputs": [],
"source": [
"def list_voices(language_code: str = DEFAULT_LANGUAGE) -> list[dict]:\n",
" response = tts_client.list_voices(language_code=language_code)\n",
"\n",
" return [\n",
" {\n",
" \"name\": voice.name,\n",
" \"gender\": texttospeech.SsmlVoiceGender(voice.ssml_gender).name.lower(),\n",
" }\n",
" for voice in response.voices\n",
" if (\"Chirp3\" in voice.name)\n",
" and voice.name != NARRATOR_VOICE\n",
" and \"en-IN\" not in voice.name\n",
" ]\n",
"\n",
"\n",
"def create_character_map(\n",
" characters: list[Character], voices: list[dict]\n",
") -> dict[str, str]:\n",
" \"\"\"Maps characters to voices based on gender identified by Gemini.\"\"\"\n",
" if len(characters) > len(voices):\n",
" print(f\"Too many characters {len(characters)}. Max {len(voices)}\")\n",
"\n",
" character_map: dict[str, str] = {}\n",
" male_voices = [voice[\"name\"] for voice in voices if voice[\"gender\"] == \"male\"]\n",
" female_voices = [voice[\"name\"] for voice in voices if voice[\"gender\"] == \"female\"]\n",
"\n",
" for character in characters:\n",
" name = character.name\n",
" gender = character.gender.lower()\n",
"\n",
" if name == \"Narrator\":\n",
" voice = NARRATOR_VOICE\n",
" elif gender == \"female\" and female_voices:\n",
" voice = female_voices.pop(random.randrange(len(female_voices)))\n",
" elif gender == \"male\" and male_voices:\n",
" voice = male_voices.pop(random.randrange(len(male_voices)))\n",
" else:\n",
" if male_voices and female_voices:\n",
" chosen_pool = random.choice([male_voices, female_voices])\n",
" elif male_voices:\n",
" chosen_pool = male_voices\n",
" elif female_voices:\n",
" chosen_pool = female_voices\n",
" else:\n",
" raise ValueError(\"Not enough voices to assign to all characters.\")\n",
"\n",
" voice = chosen_pool.pop(random.randrange(len(chosen_pool)))\n",
"\n",
" character_map[name] = voice\n",
"\n",
" return character_map\n",
"\n",
"\n",
"def synthesize_text(\n",
" file_prefix: str, file_index: str, text: str, voice_name: str\n",
") -> str:\n",
" output_file = f\"{file_prefix}-{file_index}.mp3\"\n",
"\n",
" language_code = re.search(r\"\\b[a-z]{2}-[A-Z]{2}\\b\", voice_name).group()\n",
"\n",
" response = tts_client.synthesize_speech(\n",
" input=texttospeech.SynthesisInput(text=text),\n",
" voice=texttospeech.VoiceSelectionParams(\n",
" language_code=language_code,\n",
" name=voice_name,\n",
" ),\n",
" audio_config=texttospeech.AudioConfig(\n",
" audio_encoding=texttospeech.AudioEncoding.MP3\n",
" ),\n",
" )\n",
"\n",
" # The response's audio_content is binary.\n",
" with open(output_file, \"wb\") as f:\n",
" f.write(response.audio_content)\n",
" return output_file\n",
"\n",
"\n",
"def combine_audio_files(audio_files: list[str], file_prefix: str) -> str:\n",
" full_audio = AudioSegment.silent(duration=SILENCE_LENGTH)\n",
"\n",
" for file in audio_files:\n",
" sound = AudioSegment.from_mp3(file)\n",
" silence = AudioSegment.silent(duration=SILENCE_LENGTH)\n",
" full_audio += sound + silence\n",
" os.remove(file)\n",
"\n",
" outfile_name = f\"{file_prefix}-complete.mp3\"\n",
" full_audio.export(outfile_name, format=\"mp3\")\n",
" return outfile_name\n",
"\n",
"\n",
"def generate_audio_clips(\n",
" story: Story, character_map: dict[str, str]\n",
") -> tuple[list[str], str]:\n",
" file_prefix = re.sub(r\"[^\\w.-]\", \"_\", story.title).lower()\n",
" output_files: list[str] = []\n",
"\n",
" lines: list[dict] = [\n",
" {\n",
" \"line\": story.title,\n",
" \"voice\": character_map.get(\"Narrator\", NARRATOR_VOICE),\n",
" }\n",
" ]\n",
"\n",
" # Process each scene in the play\n",
" for scene in story.scenes:\n",
" # Add the scene setting with the Narrator's voice\n",
" lines.append(\n",
" {\n",
" \"line\": \"Setting... \" + scene.setting,\n",
" \"voice\": character_map.get(\"Narrator\", NARRATOR_VOICE),\n",
" }\n",
" )\n",
"\n",
" # Process each dialogue in the scene\n",
" for dialogue in scene.dialogue:\n",
" lines.append(\n",
" {\n",
" \"line\": dialogue.line,\n",
" \"voice\": character_map.get(dialogue.speaker, DEFAULT_VOICE),\n",
" }\n",
" )\n",
"\n",
" for file_index, line in tqdm(enumerate(lines, start=1), \"Generating audio clips\"):\n",
" output_files.append(\n",
" synthesize_text(file_prefix, file_index, line[\"line\"], line[\"voice\"])\n",
" )\n",
"\n",
" return output_files, file_prefix"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4fa56795fdeb"
},
"source": [
"## Generate play with Gemini"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "476afe69c41f"
},
"outputs": [],
"source": [
"PROMPT = \"\"\"Write an interesting and humorous version of the play Macbeth by William Shakespeare.\"\"\"\n",
"\n",
"response = client.models.generate_content(\n",
" model=MODEL_ID,\n",
" contents=PROMPT,\n",
" config=GenerateContentConfig(\n",
" system_instruction=SYSTEM_INSTRUCTION,\n",
" response_mime_type=\"application/json\",\n",
" response_schema=Story,\n",
" ),\n",
")\n",
"story = response.parsed"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ed3f9e28f5a9"
},
"source": [
"Alternatively, load a pre-generated play."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "c3f5cac8983e"
},
"outputs": [],
"source": [
"with open(\"macbeth_the_sitcom.json\") as f:\n",
" story = Story.model_validate_json(f.read())"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "4e67586ace6c"
},
"outputs": [],
"source": [
"story"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "F4NR_30rfuhA"
},
"source": [
"## Get available English voices"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0KqGXuRVuBDf"
},
"outputs": [],
"source": [
"all_voices = list_voices()\n",
"pd.DataFrame(all_voices)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LKMhnJ6mf6bb"
},
"source": [
"## Assign voices to characters"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "otMX_-lfgIG1"
},
"outputs": [],
"source": [
"character_to_voice = create_character_map(story.characters, all_voices)\n",
"character_to_voice"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "p-laS59bgKo5"
},
"source": [
"## Send play text to Text-to-Speech and output each line as an audio file\n",
"\n",
"The Text-to-Speech API can only create audio with one voice per API call, so we need to create separate files for each line."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "L6rYY43xgLwh"
},
"outputs": [],
"source": [
"output_files, file_prefix = generate_audio_clips(story, character_to_voice)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5tAMk2RbgO-C"
},
"source": [
"## Combine audio files into a single file\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "xl0ZnHStgRg3"
},
"outputs": [],
"source": [
"outfile_name = combine_audio_files(output_files, file_prefix)\n",
"print(f\"Audio content written to file {outfile_name}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ce553e1efe43"
},
"source": [
"## Listen to the audio"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "d_ax6LxzciKT"
},
"outputs": [],
"source": [
"Audio(outfile_name)"
]
}
],
"metadata": {
"colab": {
"name": "storytelling.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}