wehub/googlecloudplatform--generative-ai
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:30:30 +08:00
commit 914fea506e
2793 changed files with 802106 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
audio/speech/getting-started/gemini_3_1_flash_tts.ipynb
+793
View File
@@ -0,0 +1,793 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NcIyYxlxZ7AW"
},
"outputs": [],
"source": [
"# Copyright 2026 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0sB0minZaO0T"
},
"source": [
"# Gemini 3.1 Flash Text-to-Speech Generation\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/gemini_3_1_flash_tts.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Faudio%2Fspeech%2Fgetting-started%2Fgemini_3_1_flash_tts.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/audio/speech/getting-started/gemini_3_1_flash_tts.ipynb\">\n",
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/gemini_3_1_flash_tts.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<p>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/gemini_3_1_flash_tts.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/gemini_3_1_flash_tts.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/gemini_3_1_flash_tts.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/gemini_3_1_flash_tts.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/gemini_3_1_flash_tts.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2XpbkyEbb-U3"
},
"source": [
"| Author |\n",
"| --- |\n",
"| [Katie Nguyen](https://github.com/katiemn) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PM7AnicdcFg0"
},
"source": [
"## Overview\n",
"\n",
"### Gemini 3.1 Flash TTS\n",
"\n",
"Gemini 3.1 Flash TTS on Agent Platform gives application developers access to Google's cutting-edge speech generation. This model provides powerful, low-latency speech generation with expressive audio tags for carefully crafted narration.\n",
"\n",
"In this tutorial, you will learn how to use the Google Gen AI SDK for Python to interact with Gemini 3.1 Flash TTS to:\n",
"- Convert text to an audio output with prebuilt voices\n",
"- Generate speech clips in different languages\n",
"- Define multiple speakers\n",
"- Convert text to speech with audio tags"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DvNFe-xxsced"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gyIP8rb3sgOR"
},
"source": [
"### Install Google Gen AI SDK for Python"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "JLl1jyeJsOPp"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-genai"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vy23JbZps1Jm"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you are running this notebook on Google Colab, run the following cell to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "svjU0gJVsdzu"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dL81amVRtI89"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"id": "Gfo9egZHshrZ"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"import numpy as np\n",
"from IPython.display import Audio, Markdown, display\n",
"from google import genai\n",
"from google.genai import types"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wikdtLdjtVQT"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Agent Platform, you must have an existing Google Cloud project and [enable the Agent Platform API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project](https://docs.cloud.google.com/resource-manager/docs/creating-managing-projects) and a [development environment](https://cloud.google.com/docs/authentication/set-up-adc-local-dev-environment)."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"id": "hqwZDw6Esj3z"
},
"outputs": [],
"source": [
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"global\")\n",
"\n",
"client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Mfb2XfgYuV7c"
},
"source": [
"### Define a helper function to play audio"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"id": "Ka_sWHxbsmfb"
},
"outputs": [],
"source": [
"def play_audio(response):\n",
" audio_bytes = response.candidates[0].content.parts[0].inline_data.data\n",
" audio_array = np.frombuffer(audio_bytes, dtype=\"<i2\")\n",
" display(Audio(data=audio_array, rate=24000))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-67nb2b_x8eP"
},
"source": [
"### Load the models"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"id": "6KUUn1chsow0"
},
"outputs": [],
"source": [
"speech_model = \"gemini-3.1-flash-tts-preview\"\n",
"gemini_model = \"gemini-3.5-flash\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "O2TiFc8O4JtP"
},
"source": [
"## Convert Text-to-Speech\n",
"\n",
"When converting text-to-speech, you can choose from a list of [voice options](https://docs.cloud.google.com/text-to-speech/docs/gemini-tts#voice_options), which you'll set through the `VoiceConfig`.\n",
"\n",
"By default, all audio generated with Gemini 3.1 Flash TTS utilizes [SynthID](https://deepmind.google/technologies/synthid/)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "wApkg_7OAY-h"
},
"outputs": [],
"source": [
"prompt = \"\"\"\n",
"Hey there! How are you doing today?\n",
"\"\"\"\n",
"\n",
"response = client.models.generate_content(\n",
" model=speech_model,\n",
" contents=prompt,\n",
" config=types.GenerateContentConfig(\n",
" response_modalities=[\"AUDIO\"],\n",
" speech_config=types.SpeechConfig(\n",
" voice_config=types.VoiceConfig(\n",
" prebuilt_voice_config=types.PrebuiltVoiceConfig(voice_name=\"Fenrir\")\n",
" )\n",
" ),\n",
" ),\n",
")\n",
"\n",
"play_audio(response)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "n3aRIjeNnr7h"
},
"source": [
"## Generate speech in different languages\n",
"\n",
"The TTS models detect the input language automatically. Check the documentation to learn more about the list of [supported languages](https://docs.cloud.google.com/text-to-speech/docs/gemini-tts#available_languages)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "kQryZG6RZrZl"
},
"outputs": [],
"source": [
"prompt = \"\"\"\n",
"¡Hola! ¿Quieres hacer algo hoy?\n",
"\"\"\"\n",
"\n",
"response = client.models.generate_content(\n",
" model=speech_model,\n",
" contents=prompt,\n",
" config=types.GenerateContentConfig(\n",
" response_modalities=[\"AUDIO\"],\n",
" speech_config=types.SpeechConfig(\n",
" voice_config=types.VoiceConfig(\n",
" prebuilt_voice_config=types.PrebuiltVoiceConfig(voice_name=\"Despina\")\n",
" )\n",
" ),\n",
" ),\n",
")\n",
"\n",
"play_audio(response)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZrQLAflLolcc"
},
"source": [
"## Define multiple speakers\n",
"\n",
"For multi-speaker audio clips, you'll need to include a `MultiSpeakerVoiceConfig` in your request. You can include up to 2 different speakers, each with their own `SpeakerVoiceConfig` that includes the `speaker` name and `voice_name`. Make sure the `speaker` name is the same one used in the prompt."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_K_aio7ouYP4"
},
"outputs": [],
"source": [
"prompt = \"\"\"\n",
"Ryan: How are you doing today Katie?\n",
"Katie: Not too bad, how about you?\n",
"\"\"\"\n",
"\n",
"response = client.models.generate_content(\n",
" model=speech_model,\n",
" contents=prompt,\n",
" config=types.GenerateContentConfig(\n",
" response_modalities=[\"AUDIO\"],\n",
" speech_config=types.SpeechConfig(\n",
" multi_speaker_voice_config=types.MultiSpeakerVoiceConfig(\n",
" speaker_voice_configs=[\n",
" types.SpeakerVoiceConfig(\n",
" speaker=\"Ryan\",\n",
" voice_config=types.VoiceConfig(\n",
" prebuilt_voice_config=types.PrebuiltVoiceConfig(\n",
" voice_name=\"Umbriel\",\n",
" )\n",
" ),\n",
" ),\n",
" types.SpeakerVoiceConfig(\n",
" speaker=\"Katie\",\n",
" voice_config=types.VoiceConfig(\n",
" prebuilt_voice_config=types.PrebuiltVoiceConfig(\n",
" voice_name=\"Leda\",\n",
" )\n",
" ),\n",
" ),\n",
" ]\n",
" )\n",
" ),\n",
" ),\n",
")\n",
"\n",
"play_audio(response)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZKm_x60Sr5cp"
},
"source": [
"## Prompting with audio tags\n",
"\n",
"Audio tags let you insert modifiers directly into your prompt to help control speaker style, tone, or delivery. An audio tag is a word in square brackets that specifies how the text should be delivered, such as `[confusion]` or `[laughs]`.\n",
"\n",
"You can experiment with different tags. Here is a list of examples to get started:\n",
"\n",
"| Audio Tag | Audio Tag | Audio Tag | Audio Tag | Audio Tag |\n",
"| --- | --- | --- | --- | --- |\n",
"| [determination]| [enthusiasm] | [adoration] |[interest] | [awe] |\n",
"|[admiration] | [nervousness] | [frustration] | [excitement] | [curiosity] |\n",
"| [hope] | [annoyance] | [amusement] | [aggression] | [tension] |\n",
"| [agitation] | [confusion] | [anger] | [positive] | [neutral] |\n",
"| [negative] | [whispers] | [laughs] | | |\n",
"\n",
"**NOTE:** For best results, add audio tags in English, even if your transcript is in a different language."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "OgemQk62Zowg"
},
"outputs": [],
"source": [
"prompt = \"\"\"\n",
"[confusion] I have absolutely no idea why this variable is returning as undefined, [agitation] and I've been staring at this same block of code until my eyes are literally crossing! [aggression] Just work already! [whispers] Wait... is it really just a missing semicolon right there on line forty-two? [laughs] I can't believe I spent two hours debugging a simple typo!\n",
"\"\"\"\n",
"\n",
"response = client.models.generate_content(\n",
" model=speech_model,\n",
" contents=prompt,\n",
" config=types.GenerateContentConfig(\n",
" response_modalities=[\"AUDIO\"],\n",
" speech_config=types.SpeechConfig(\n",
" voice_config=types.VoiceConfig(\n",
" prebuilt_voice_config=types.PrebuiltVoiceConfig(voice_name=\"Charon\")\n",
" )\n",
" ),\n",
" ),\n",
")\n",
"\n",
"play_audio(response)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RvcPT7El3o8m"
},
"source": [
"### Adding audio tags\n",
"\n",
"If you have a long transcript, it may be useful to have Gemini help add in audio tags. In this next example, you'll take a fictitious podcast transcript and give it to Gemini with a list of example audio tags that can be included."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"id": "SjQanjUYckuA"
},
"outputs": [],
"source": [
"podcast = \"\"\"\n",
"If you walk through a local park on a Saturday morning, you are bound to see a familiar cast of characters. There is the floppy-eared Golden Retriever chasing a tennis ball, the stout French Bulldog trotting along the pavement, and almost certainly, the athletic Labrador Retriever leading the way. We think of these breeds as timeless icons of the domestic home, but the \"popularity\" of a dog is rarely an accident of nature. It is a fascinating mirror held up to human history, shifting based on our wars, our technology, and even our movies.\n",
"\n",
"For most of human history, a \"popular\" dog was simply the one that kept you alive. We didn't have breeds in the modern sense; we had jobs. There were dogs that pulled sleds, dogs that guarded livestock, and dogs that cleared vermin from granaries. If a dog was popular in the year 1500, it was because it was an elite employee. The idea of a dog as a \"member of the family\" with no specific chore to perform is a relatively new luxury, largely born out of the Victorian era.\n",
"\n",
"It was during the 19th century in England that dog ownership shifted from a functional necessity to a high-stakes hobby. This was the birth of the Kennel Club and the formalization of \"breeds.\" Suddenly, the physical consistency of a dog—the shape of its snout or the curl of its tail—became a status symbol. A popular breed wasn't just a good worker; it was a fashion statement. This is when the Beagle and the Fox Terrier began their rise.\n",
"\n",
"However, nothing influences what we choose to bring into our homes quite like the silver screen. We see this in what researchers call the \"Lassie Effect.\" When the movie Lassie Come Home was released in 1943, registrations for Rough Collies skyrocketed. It's a recurring theme in history: we fall in love with a character, and the breed becomes a trend.\n",
"\n",
"Then there is the king of the mountain: the Labrador Retriever. For thirty-one consecutive years, the Lab held the title of the most popular dog in America. Its rise to power was a perfect storm of timing. As the world moved into the suburbs after World War II, people wanted a dog that was \"versatile.\" The Lab was small enough to live indoors but sturdy enough to go hiking; it was gentle with children but capable of being a serious hunting partner. It became the \"all-purpose\" dog for the modern age.\n",
"\n",
"Today, we are seeing another shift. The French Bulldog recently dethroned the Labrador, ending its three-decade reign. This tells us something about how our lives have changed again. We live in smaller spaces, we have less time for long runs in the woods, and we value portability. The popularity of \"designer\" breeds like the various Doodles reflects a modern obsession with customization—wanting the personality of a Retriever with the hypoallergenic coat of a Poodle.\n",
"\n",
"When we look at the most popular breeds of today, we aren't just looking at dogs. We are looking at a map of our own desires. We choose the breeds that fit the lives we want to live, whether that's the rugged adventurer, the urban socialite, or the suburban family. The dogs change because we change.\n",
"\n",
"Thanks for joining me on The Everyday. Next time you see a dog on the street, take a look at the human on the other end of the leash—you might learn just as much about them as you do about the dog. I'll see you next time.\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"id": "-qEsorzYdwdn"
},
"outputs": [],
"source": [
"example_audio_tags = [\n",
" \"[acceptance]\",\n",
" \"[accomplishment]\",\n",
" \"[achievement]\",\n",
" \"[active]\",\n",
" \"[admiration]\",\n",
" \"[admonition]\",\n",
" \"[adoration]\",\n",
" \"[affection]\",\n",
" \"[aggression]\",\n",
" \"[agitation]\",\n",
" \"[alarm]\",\n",
" \"[amazement]\",\n",
" \"[ambivalence]\",\n",
" \"[amused]\",\n",
" \"[amusement]\",\n",
" \"[analysis]\",\n",
" \"[anger]\",\n",
" \"[animation]\",\n",
" \"[annoyance]\",\n",
" \"[anticipation]\",\n",
" \"[anxiety]\",\n",
" \"[apology]\",\n",
" \"[appreciation]\",\n",
" \"[apprehension]\",\n",
" \"[approval]\",\n",
" \"[arrogance]\",\n",
" \"[assertion]\",\n",
" \"[assertive]\",\n",
" \"[assertiveness]\",\n",
" \"[assurance]\",\n",
" \"[astonishment]\",\n",
" \"[aversion]\",\n",
" \"[awareness]\",\n",
" \"[awe]\",\n",
" \"[awkwardness]\",\n",
" \"[bargaining]\",\n",
" \"[boredom]\",\n",
" \"[caring]\",\n",
" \"[caution]\",\n",
" \"[cautious]\",\n",
" \"[certainty]\",\n",
" \"[challenging]\",\n",
" \"[comfort]\",\n",
" \"[compassion]\",\n",
" \"[concentration]\",\n",
" \"[concern]\",\n",
" \"[confidence]\",\n",
" \"[confident]\",\n",
" \"[confusion]\",\n",
" \"[contemplative]\",\n",
" \"[contempt]\",\n",
" \"[contentment]\",\n",
" \"[conviction]\",\n",
" \"[courage]\",\n",
" \"[craving]\",\n",
" \"[critical]\",\n",
" \"[criticism]\",\n",
" \"[curiosity]\",\n",
" \"[decision]\",\n",
" \"[defiance]\",\n",
" \"[demonstration]\",\n",
" \"[description]\",\n",
" \"[descriptive]\",\n",
" \"[desire]\",\n",
" \"[despair]\",\n",
" \"[desperation]\",\n",
" \"[despondency]\",\n",
" \"[determination]\",\n",
" \"[determined]\",\n",
" \"[devotion]\",\n",
" \"[directness]\",\n",
" \"[disagreement]\",\n",
" \"[disappointment]\",\n",
" \"[disapproval]\",\n",
" \"[disbelief]\",\n",
" \"[discernment]\",\n",
" \"[discomfort]\",\n",
" \"[disdain]\",\n",
" \"[disgust]\",\n",
" \"[disillusionment]\",\n",
" \"[dislike]\",\n",
" \"[dismissive]\",\n",
" \"[distress]\",\n",
" \"[doubt]\",\n",
" \"[dread]\",\n",
" \"[eagerness]\",\n",
" \"[effervescence]\",\n",
" \"[embarrassment]\",\n",
" \"[embitterment]\",\n",
" \"[embracement]\",\n",
" \"[empathy]\",\n",
" \"[emphasis]\",\n",
" \"[enchantment]\",\n",
" \"[encouraging]\",\n",
" \"[energetic]\",\n",
" \"[enjoyment]\",\n",
" \"[enthusiasm]\",\n",
" \"[enthusiastic]\",\n",
" \"[excitement]\",\n",
" \"[exhaustion]\",\n",
" \"[explaining]\",\n",
" \"[fascination]\",\n",
" \"[fast]\",\n",
" \"[fear]\",\n",
" \"[focus]\",\n",
" \"[fondness]\",\n",
" \"[friendly]\",\n",
" \"[frustration]\",\n",
" \"[gratification]\",\n",
" \"[gratitude]\",\n",
" \"[grief]\",\n",
" \"[guilt]\",\n",
" \"[happy]\",\n",
" \"[high energy]\",\n",
" \"[hope]\",\n",
" \"[horror]\",\n",
" \"[humor]\",\n",
" \"[hurt]\",\n",
" \"[incredulity]\",\n",
" \"[indifference]\",\n",
" \"[indignation]\",\n",
" \"[informative]\",\n",
" \"[instruction]\",\n",
" \"[interest]\",\n",
" \"[intrigue]\",\n",
" \"[invitation]\",\n",
" \"[joy]\",\n",
" \"[laughs]\",\n",
" \"[logical reasoning]\",\n",
" \"[long pause]\",\n",
" \"[love]\",\n",
" \"[low energy]\",\n",
" \"[melancholy]\",\n",
" \"[mixed]\",\n",
" \"[negative]\",\n",
" \"[negative surprise]\",\n",
" \"[nervousness]\",\n",
" \"[neutral]\",\n",
" \"[nostalgia]\",\n",
" \"[observation]\",\n",
" \"[offense]\",\n",
" \"[optimism]\",\n",
" \"[pain]\",\n",
" \"[panic]\",\n",
" \"[passion]\",\n",
" \"[passive]\",\n",
" \"[pensive]\",\n",
" \"[pessimism]\",\n",
" \"[pity]\",\n",
" \"[planning]\",\n",
" \"[playful]\",\n",
" \"[pleading]\",\n",
" \"[pleased]\",\n",
" \"[positive]\",\n",
" \"[positive surprise]\",\n",
" \"[praise]\",\n",
" \"[pride]\",\n",
" \"[realization]\",\n",
" \"[recognition]\",\n",
" \"[reflection]\",\n",
" \"[regret]\",\n",
" \"[relaxation]\",\n",
" \"[relief]\",\n",
" \"[reminiscence]\",\n",
" \"[resignation]\",\n",
" \"[sadness]\",\n",
" \"[sarcasm]\",\n",
" \"[satisfaction]\",\n",
" \"[self-deprecation]\",\n",
" \"[self-satisfaction]\",\n",
" \"[sentimentality]\",\n",
" \"[serenity]\",\n",
" \"[seriousness]\",\n",
" \"[shame]\",\n",
" \"[shock]\",\n",
" \"[short pause]\",\n",
" \"[skepticism]\",\n",
" \"[slight relief]\",\n",
" \"[smitten]\",\n",
" \"[solemnity]\",\n",
" \"[speculation]\",\n",
" \"[slow]\",\n",
" \"[strategizing]\",\n",
" \"[stress]\",\n",
" \"[struggle]\",\n",
" \"[success]\",\n",
" \"[suffering]\",\n",
" \"[suggestion]\",\n",
" \"[summary]\",\n",
" \"[surprise]\",\n",
" \"[suspicion]\",\n",
" \"[sympathy]\",\n",
" \"[tension]\",\n",
" \"[terror]\",\n",
" \"[thanks]\",\n",
" \"[thinking]\",\n",
" \"[thrill]\",\n",
" \"[tiredness]\",\n",
" \"[triumph]\",\n",
" \"[uncertainty]\",\n",
" \"[unclear]\",\n",
" \"[understanding]\",\n",
" \"[unease]\",\n",
" \"[urgency]\",\n",
" \"[victory]\",\n",
" \"[warning]\",\n",
" \"[weariness]\",\n",
" \"[whispers]\",\n",
" \"[wisdom]\",\n",
" \"[wistful]\",\n",
" \"[worry]\",\n",
" \"[yearning]\",\n",
"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_cztl0y-4Yum"
},
"source": [
"Now, you'll prompt Gemini to take the podcast transcript and insert audio tags from the provided list."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "qOetTxFRdbys"
},
"outputs": [],
"source": [
"response = client.models.generate_content(\n",
" model=gemini_model,\n",
" contents=f\"Please take the {podcast} provided and insert audio tags from {example_audio_tags}. Insert the tags immediately before the phrase or sentence they are meant to influence. Ensure the tag matches the emotional arc of the narrative. Avoid overusing tags, place them where a natural change in tone or pace would occur.\",\n",
")\n",
"\n",
"podcast_tags = response.text\n",
"display(Markdown(podcast_tags))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "k-X-R4zB4oSY"
},
"source": [
"Finally, you'll send the updated transcript with the audio tags to the TTS model to generate the audio with the updated modifiers."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "g2l6-5e9fpVw"
},
"outputs": [],
"source": [
"response = client.models.generate_content(\n",
" model=speech_model,\n",
" contents=podcast_tags,\n",
" config=types.GenerateContentConfig(\n",
" response_modalities=[\"AUDIO\"],\n",
" speech_config=types.SpeechConfig(\n",
" voice_config=types.VoiceConfig(\n",
" prebuilt_voice_config=types.PrebuiltVoiceConfig(voice_name=\"Achernar\")\n",
" )\n",
" ),\n",
" ),\n",
")\n",
"\n",
"play_audio(response)"
]
}
],
"metadata": {
"colab": {
"name": "gemini_3_1_flash_tts.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
audio/speech/getting-started/get_started_with_chirp3_instant_custom_voice.ipynb
+782
View File
@@ -0,0 +1,782 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Get started with Chirp 3: Instant custom voice\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp3_instant_custom_voice.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Faudio%2Fspeech%2Fgetting-started%2Fget_started_with_chirp3_instant_custom_voice.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/audio/speech/getting-started/get_started_with_chirp3_instant_custom_voice.ipynb\">\n",
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp3_instant_custom_voice.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<p>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp3_instant_custom_voice.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp3_instant_custom_voice.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp3_instant_custom_voice.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp3_instant_custom_voice.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp3_instant_custom_voice.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| Author(s) |\n",
"| --- |\n",
"| [Ivan Nardini](https://github.com/inardini) |\n",
"| [Gary Chien](https://github.com/goldenchest) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"This notebook introduces [Chirp 3 Instant Custom Voice](https://cloud.google.com/text-to-speech/docs/chirp3-hd), a powerful feature of Google Cloud's Text-to-Speech (TTS) API that allows you to create personalized voice models.\n",
"\n",
"With Instant Custom Voice, you can generate unique, custom voices by training a model with your own high-quality audio recordings. This enables the rapid generation of personal voices that can then be used to synthesize audio using the Cloud TTS API, supporting both streaming and long-form text output. Instant Custom Voice creation and synthesis is supported in more than 25 language.\n",
"\n",
"In this tutorial, you will learn how to:\n",
"\n",
"- Create an Instant Custom Voice.\n",
"- Synthesize text using your custom voice both in real-time and streaming.\n",
"- Build a simple Gradio app to use your custom voice."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7qzYMYf2Hlvv"
},
"source": [
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ Due to safety considerations, access to this voice cloning capability is restricted to allow-listed users. To access this feature, contact a member of the Google Cloud team to be added to the allow list. ⚠️</b>\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1filYzhuK_MK"
},
"source": [
"### Install required packages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-_O1r5MNLFu-"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet gradio"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SxJiB0GXALQ2"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using the Text-to-Speech API, you must have an existing Google Cloud project and [enable the API](https://console.cloud.google.com/flows/enableapi?apiid=texttospeech.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment).\n",
"\n",
"Please note the **available regions** for Chirp 3 Instant Custom voice, see [documentation](https://cloud.google.com/text-to-speech/docs/chirp3-instant-custom-voice#regional_availability)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "WIQyBhAn_9tK"
},
"outputs": [],
"source": [
"# Use the environment variable if the user doesn't provide Project ID.\n",
"import os\n",
"\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"TTS_LOCATION = \"global\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PYVzRl1bmgjk"
},
"outputs": [],
"source": [
"! gcloud config set project {PROJECT_ID}\n",
"! gcloud auth application-default set-quota-project {PROJECT_ID}\n",
"! gcloud auth application-default login -q"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JR7BVajtKyGh"
},
"source": [
"### Get Google Credentials\n",
"\n",
"Use the `google.auth` library to automatically find and load your credentials.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ChWHE3jIDREc"
},
"outputs": [],
"source": [
"import google.auth\n",
"import google.auth.transport.requests\n",
"\n",
"credentials, _ = google.auth.default()\n",
"authentication = google.auth.transport.requests.Request()\n",
"credentials.refresh(authentication)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5-X8IbjiAtM3"
},
"source": [
"### Import libraries\n",
"\n",
"Import Python tools you'll need."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "qqm0OQpAYCph"
},
"outputs": [],
"source": [
"import base64\n",
"import json\n",
"import os\n",
"\n",
"import gradio as gr\n",
"import numpy as np\n",
"import requests\n",
"from IPython.display import Audio, display"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sP8GBj3tBAC1"
},
"source": [
"### Set constants\n",
"\n",
"Initiate the API endpoint and the text to speech client.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rXTVeU1uBBqY"
},
"outputs": [],
"source": [
"API_ENDPOINT = (\n",
" f\"{TTS_LOCATION}-texttospeech.googleapis.com\"\n",
" if TTS_LOCATION != \"global\"\n",
" else \"texttospeech.googleapis.com\"\n",
")\n",
"\n",
"ACCESS_TOKEN = credentials.token"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "M7WQQFp_RvGH"
},
"source": [
"### Helpers\n",
"\n",
"To keep our main logic clean and readable, you define several helper functions here. These encapsulate tasks like making API calls, processing audio data, and interacting with the Gradio interface components.\n",
"\n",
"* **`create_instant_custom_voice_key(reference_audio_bytes, consent_audio_bytes)`** to create a temporary custom voice key.\n",
"\n",
"* **`create_voice_with_masking(reference_audio, consent_audio)`** to mask the custom voice key.\n",
"\n",
"* **`synthesize_text_with_cloned_voice(voice_key, text)`** to create your custom voice using the `voice_cloning_key` and the desired text as input.\n",
"\n",
"* **`wav_to_base64(file_path)`** to read a WAV audio file from a local path, encode its binary content into a base64 string (which is how audio data is sent in the JSON payload), and return the string.\n",
"\n",
"* **`create_voice(reference_audio, consent_audio, progress)`** to create the custom voice in the Gradio app using \"Create Voice\" button.\n",
"\n",
"* **`generate_speech(voice_key, text, progress)`**: to synthesize any text with the custom voice using \"Generate Speech\" button in the Gradio app.\n",
"\n",
"* **`reset_interface()`** to effectively reset the Gradio UI."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "pWrNWdV_RxGn"
},
"outputs": [],
"source": [
"def create_instant_custom_voice_key(\n",
" reference_audio_bytes: bytes, consent_audio_bytes: bytes\n",
") -> str:\n",
" \"\"\"Creates a temporary custom voice key\"\"\"\n",
" url = f\"https://{API_ENDPOINT}/v1beta1/voices:generateVoiceCloningKey\"\n",
"\n",
" request_body = {\n",
" \"reference_audio\": {\n",
" \"audio_config\": {\"audio_encoding\": \"LINEAR16\", \"sample_rate_hertz\": 24000},\n",
" \"content\": reference_audio_bytes,\n",
" },\n",
" \"voice_talent_consent\": {\n",
" \"audio_config\": {\"audio_encoding\": \"LINEAR16\", \"sample_rate_hertz\": 24000},\n",
" \"content\": consent_audio_bytes,\n",
" },\n",
" \"consent_script\": \"I am the owner of this voice and I consent to Google using this voice to create a synthetic voice model.\",\n",
" \"language_code\": \"en-US\",\n",
" }\n",
"\n",
" response = None\n",
" try:\n",
" headers = {\n",
" \"Authorization\": f\"Bearer {ACCESS_TOKEN}\",\n",
" \"x-goog-user-project\": PROJECT_ID,\n",
" \"Content-Type\": \"application/json; charset=utf-8\",\n",
" }\n",
"\n",
" response = requests.post(url, headers=headers, json=request_body)\n",
" response.raise_for_status()\n",
"\n",
" response_json = response.json()\n",
" return response_json.get(\"voiceCloningKey\")\n",
"\n",
" except requests.exceptions.RequestException as e:\n",
" print(f\"Error making API request: {e}\")\n",
" if response is not None and response.text:\n",
" print(\"Response error message:\")\n",
" print(response.text)\n",
" except json.JSONDecodeError as e:\n",
" print(f\"Error decoding JSON response: {e}\")\n",
" except Exception as e:\n",
" print(f\"An unexpected error occurred: {e}\")\n",
"\n",
"\n",
"def synthesize_text_with_cloned_voice(voice_key: str, text: str) -> None:\n",
" \"\"\"Synthesizes text with the cloned voice\"\"\"\n",
" url = f\"https://{API_ENDPOINT}/v1beta1/text:synthesize\"\n",
"\n",
" request_body = {\n",
" \"input\": {\"text\": text},\n",
" \"voice\": {\n",
" \"language_code\": \"en-US\",\n",
" \"voice_clone\": {\n",
" \"voice_cloning_key\": voice_key,\n",
" },\n",
" },\n",
" \"audioConfig\": {\"audioEncoding\": \"LINEAR16\", \"sample_rate_hertz\": 24000},\n",
" }\n",
"\n",
" try:\n",
" headers = {\n",
" \"Authorization\": f\"Bearer {ACCESS_TOKEN}\",\n",
" \"x-goog-user-project\": PROJECT_ID,\n",
" \"Content-Type\": \"application/json; charset=utf-8\",\n",
" }\n",
"\n",
" response = requests.post(url, headers=headers, json=request_body)\n",
" response.raise_for_status()\n",
"\n",
" response_json = response.json()\n",
" audio_content = response_json.get(\"audioContent\")\n",
"\n",
" if audio_content:\n",
" display(Audio(base64.b64decode(audio_content), rate=24000))\n",
" else:\n",
" print(\"Error: Audio content not found in the response.\")\n",
" print(response_json)\n",
"\n",
" except requests.exceptions.RequestException as e:\n",
" print(f\"Error making API request: {e}\")\n",
" except json.JSONDecodeError as e:\n",
" print(f\"Error decoding JSON response: {e}\")\n",
" except Exception as e:\n",
" print(f\"An unexpected error occurred: {e}\")\n",
"\n",
"\n",
"def wav_to_base64(file_path: str) -> str:\n",
" \"\"\"Convert a WAV file to base64 encoded string\"\"\"\n",
" try:\n",
" with open(file_path, \"rb\") as wav_file:\n",
" encoded_string = base64.b64encode(wav_file.read()).decode(\"utf-8\")\n",
" return encoded_string\n",
" except FileNotFoundError:\n",
" print(f\"Error: File not found at {file_path}\")\n",
" return None\n",
" except Exception as e:\n",
" print(f\"An error occurred: {e}\")\n",
" return None\n",
"\n",
"\n",
"def create_voice(\n",
" reference_audio: gr.Audio,\n",
" consent_audio: gr.Audio,\n",
" progress: gr.Progress | None = None,\n",
") -> str:\n",
" \"\"\"Create a custom voice using reference and consent audio\"\"\"\n",
" if reference_audio is None or consent_audio is None:\n",
" return \"Please upload both reference and consent audio files.\"\n",
"\n",
" if not progress:\n",
" progress = gr.Progress()\n",
"\n",
" progress(0.2, desc=\"Processing audio files...\")\n",
" reference_audio_b64 = wav_to_base64(reference_audio)\n",
" consent_audio_b64 = wav_to_base64(consent_audio)\n",
"\n",
" if reference_audio_b64 is None or consent_audio_b64 is None:\n",
" return \"Error processing audio files.\"\n",
"\n",
" progress(0.5, desc=\"Creating voice clone...\")\n",
" voice_key = create_instant_custom_voice_key(reference_audio_b64, consent_audio_b64)\n",
"\n",
" if voice_key:\n",
" progress(1.0, desc=\"Voice created successfully!\")\n",
" return voice_key\n",
" return \"Failed to create voice. Check the logs for details.\"\n",
"\n",
"\n",
"def create_voice_with_masking(\n",
" reference_audio: gr.Audio, consent_audio: gr.Audio\n",
") -> tuple:\n",
" \"\"\"A wrapper function for create_voice to handle masking\"\"\"\n",
" key = create_voice(reference_audio, consent_audio)\n",
" if key and len(key) > 8:\n",
" masked_key = key[:5] + \"...\"\n",
" else:\n",
" masked_key = key\n",
" return key, masked_key\n",
"\n",
"\n",
"def generate_speech(\n",
" voice_key: str, text: str, progress: gr.Progress | None = None\n",
") -> tuple:\n",
" \"\"\"Generate speech using the cloned voice\"\"\"\n",
" if not voice_key or not text:\n",
" return None, \"Please create a voice key and enter text to synthesize.\"\n",
"\n",
" if not progress:\n",
" progress = gr.Progress()\n",
"\n",
" progress(0.3, desc=\"Generating speech...\")\n",
"\n",
" try:\n",
" url = f\"https://{API_ENDPOINT}/v1beta1/text:synthesize\"\n",
"\n",
" request_body = {\n",
" \"input\": {\"text\": text},\n",
" \"voice\": {\n",
" \"language_code\": \"en-US\",\n",
" \"voice_clone\": {\n",
" \"voice_cloning_key\": voice_key,\n",
" },\n",
" },\n",
" \"audioConfig\": {\"audioEncoding\": \"LINEAR16\", \"sample_rate_hertz\": 24000},\n",
" }\n",
"\n",
" headers = {\n",
" \"Authorization\": f\"Bearer {ACCESS_TOKEN}\",\n",
" \"x-goog-user-project\": PROJECT_ID,\n",
" \"Content-Type\": \"application/json; charset=utf-8\",\n",
" }\n",
"\n",
" progress(0.6, desc=\"Processing audio...\")\n",
" response = requests.post(url, headers=headers, json=request_body)\n",
" response.raise_for_status()\n",
"\n",
" response_json = response.json()\n",
" audio_content = response_json.get(\"audioContent\")\n",
"\n",
" if audio_content:\n",
" progress(1.0, desc=\"Speech generated!\")\n",
" audio_bytes = base64.b64decode(audio_content)\n",
" audio_array = np.frombuffer(audio_bytes, dtype=np.int16)\n",
" return (24000, audio_array), \"Speech generated successfully!\"\n",
" return None, \"Error: Audio content not found in the response.\"\n",
"\n",
" except Exception as e:\n",
" return None, f\"Error generating speech: {e!s}\"\n",
"\n",
"\n",
"def reset_interface() -> tuple:\n",
" return None, None, \"\", \"\", None, None, \"Interface reset.\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EdvJRUWRNGHE"
},
"source": [
"# Create an Instant Custom Voice\n",
"\n",
"Let's start by creating the custom voice directly.\n",
"\n",
"You would first define the file paths to your pre-recorded reference audio (`.wav` recommended, ideally a few seconds clear speech) and the consent audio (where the speaker explicitly states the consent script).\n",
"\n",
"Then you use `wav_to_base64` to read these files and encode them into the base64 format required by the API and you create your custom voice using the `create_instant_custom_voice_key` helper function, passing in the base64-encoded audio data.\n",
"\n",
"If the request is successful and your project is allow-listed, the API returns a `voice_cloning_key` which acts as a temporary identifier for your custom voice.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "S_J-TGJQNuKq"
},
"outputs": [],
"source": [
"# fmt: off\n",
"reference_audio_path = \"[your-reference-audio-path]\" # @param {type: \"string\", placeholder: \"[your-reference-audio-path]\", isTemplate: true}\n",
"consent_audio_path = \"[your-consent-audio-path]\" # @param {type: \"string\", placeholder: \"[your-consent-audio-path]\", isTemplate: true}\n",
"# fmt: on"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "SCdVWsGOA3qW"
},
"outputs": [],
"source": [
"reference_audio_bytes = wav_to_base64(reference_audio_path)\n",
"consent_audio_bytes = wav_to_base64(consent_audio_path)\n",
"\n",
"voice_key = create_instant_custom_voice_key(reference_audio_bytes, consent_audio_bytes)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rPqteN879IL_"
},
"source": [
"# Use Instant Custom Voice\n",
"\n",
"Now that you (theoretically) have a `voice_key`, let's use it to synthesize speech."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JGAZb4Zf9Me-"
},
"source": [
"## Perform a Sync Request\n",
"\n",
"Define the text you want your custom voice to say in the `text_to_synthesize` variable. Try different sentences!\n",
"\n",
"Then, call the `synthesize_text_with_cloned_voice` helper function, providing the `voice_key` you obtained earlier and the text. This function sends the request to the `text:synthesize` endpoint, specifying your custom voice.\n",
"\n",
"If successful, it gets the audio data back, decodes it, and should play it directly in the notebook output below the cell using an embedded audio player."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "djLao152cR13"
},
"outputs": [],
"source": [
"text_to_synthesize = \"\"\"\n",
"Breaking news! Chirp 3, Google Cloud's audio model, now has Instant Custom Voice.\n",
"\n",
"With Instant Custom Voice, you can generate custom voices with just 10 seconds of audio to empower your AI narration.\n",
"Chirp 3 Instant Custom Voice is now available in preview with allowlist. Check out the link below.\n",
"\n",
"And yes, this voice is generated using Chirp 3 Instant Custom Voice!\n",
"\"\"\"\n",
"synthesize_text_with_cloned_voice(voice_key, text_to_synthesize)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3X0qY28fLSfz"
},
"source": [
"# Build a simple Instant custom voice app\n",
"\n",
"While calling the API directly works, it's often more convenient to have an interactive interface. Let's build a simple web app using Gradio to easily upload audio, create a voice, and synthesize text."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uXoJqKduNhzq"
},
"source": [
"### Define the app\n",
"\n",
"Here, you define the structure and components of our Gradio user interface."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "jMISBY0fSDFy"
},
"outputs": [],
"source": [
"with gr.Blocks(\n",
" theme=gr.themes.Default(\n",
" primary_hue=\"blue\", secondary_hue=\"blue\", neutral_hue=\"gray\"\n",
" )\n",
") as app:\n",
" # Create a state variable to store the full voice key\n",
" full_voice_key = gr.State(\"\")\n",
"\n",
" # Define title\n",
" gr.Markdown(\n",
" \"\"\"\n",
" # Chirp 3 - Instant custom voice demo\n",
" Upload reference and consent audio files to create a custom voice, then synthesize speech.\n",
" \"\"\"\n",
" )\n",
"\n",
" # Define input and output components\n",
" with gr.Row():\n",
" with gr.Column(scale=1):\n",
" reference_audio = gr.Audio(\n",
" label=\"Reference Voice\", type=\"filepath\", elem_id=\"reference_audio\"\n",
" )\n",
" consent_audio = gr.Audio(\n",
" label=\"Consent Audio\", type=\"filepath\", elem_id=\"consent_audio\"\n",
" )\n",
"\n",
" create_btn = gr.Button(\"Create Voice\", variant=\"primary\")\n",
" voice_key_output = gr.Textbox(label=\"Voice Key\", elem_id=\"voice_key\")\n",
"\n",
" with gr.Column(scale=1):\n",
" text_input = gr.Textbox(\n",
" label=\"Text to Synthesize\",\n",
" placeholder=\"Enter the text you want the voice to say...\",\n",
" lines=5,\n",
" elem_id=\"text_input\",\n",
" )\n",
" generate_btn = gr.Button(\"Generate Speech\", variant=\"primary\")\n",
" audio_output = gr.Audio(label=\"Generated Audio\", elem_id=\"audio_output\")\n",
" status_output = gr.Textbox(label=\"Status\", elem_id=\"status_output\")\n",
"\n",
" with gr.Row():\n",
" clear_btn = gr.Button(\"Clear All\", variant=\"secondary\")\n",
"\n",
" # Set up event handlers\n",
" create_btn.click(\n",
" create_voice_with_masking,\n",
" inputs=[reference_audio, consent_audio],\n",
" outputs=[full_voice_key, voice_key_output],\n",
" )\n",
"\n",
" generate_btn.click(\n",
" generate_speech,\n",
" inputs=[full_voice_key, text_input],\n",
" outputs=[audio_output, status_output],\n",
" )\n",
"\n",
" clear_btn.click(\n",
" reset_interface,\n",
" inputs=[],\n",
" outputs=[\n",
" reference_audio,\n",
" consent_audio,\n",
" voice_key_output,\n",
" full_voice_key,\n",
" text_input,\n",
" audio_output,\n",
" status_output,\n",
" ],\n",
" )\n",
"\n",
" # Apply custom CSS for Google styling\n",
" gr.Markdown(\n",
" \"\"\"\n",
" <style>\n",
" .gradio-container {\n",
" font-family: 'Google Sans', 'Roboto', sans-serif !important;\n",
" }\n",
" .gr-button-primary {\n",
" background-color: #4285F4 !important;\n",
" }\n",
" .gr-button-secondary {\n",
" border-color: #4285F4 !important;\n",
" color: #4285F4 !important;\n",
" }\n",
" h1 {\n",
" font-family: 'Google Sans', 'Roboto', sans-serif !important;\n",
" color: #202124 !important;\n",
" }\n",
" </style>\n",
" \"\"\"\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GpCcdY4ANkRl"
},
"source": [
"### Launch the app\n",
"\n",
"Showtime! This cell launches the Gradio app we just defined."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "YJ2-accJL_tn"
},
"outputs": [],
"source": [
"app.launch(share=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xV-Nj4F7mPNF"
},
"source": [
"Close the app once you finish to play with it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "UTPbraYfNeJk"
},
"outputs": [],
"source": [
"app.close()"
]
}
],
"metadata": {
"colab": {
"name": "get_started_with_chirp3_instant_custom_voice.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
audio/speech/getting-started/get_started_with_chirp_3_hd_voices.ipynb
+565
View File
@@ -0,0 +1,565 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Get started with Chirp 3 HD voices using Text-to-Speech\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_hd_voices.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Faudio%2Fspeech%2Fgetting-started%2Fget_started_with_chirp_3_hd_voices.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/audio/speech/getting-started/get_started_with_chirp_3_hd_voices.ipynb\">\n",
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_hd_voices.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<p>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_hd_voices.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_hd_voices.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_hd_voices.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_hd_voices.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_hd_voices.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| Authors |\n",
"| --- |\n",
"| [Holt Skinner](https://github.com/holtskinner) |\n",
"| [Ivan Nardini](https://github.com/inardini) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"This notebook introduces [Chirp 3 HD Voices](https://cloud.google.com/text-to-speech/docs/chirp3-hd), which are Google Cloud's latest advancement in Text-to-Speech (TTS) technology.\n",
"\n",
"These voices, powered by state-of-the-art large language models (LLMs), offer a significantly improved level of realism and emotional expressiveness.\n",
"\n",
"Chirp 3 HD voices provide high-fidelity audio and natural-sounding speech, complete with human-like intonation and pauses. They are available on the Vertex AI platform and are designed for various uses like, voice assistants, audiobooks, and customer service applications.\n",
"\n",
"There are currently eight distinct voice options(4 male, 4 female) available in 31 languages.\n",
"\n",
"In this tutorial, you learn how to:\n",
"\n",
"- How to synthesize speech using real-time (online) processing\n",
"- How to synthesize speech using streaming processing"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Text-to-Speech SDK and other required packages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e73_ZgKWYedz"
},
"outputs": [],
"source": [
"%%bash\n",
"# Detect the operating system\n",
"os=$(uname -s)\n",
"\n",
"if [[ \"$os\" == \"Linux\" ]]; then\n",
" # Linux installation\n",
" sudo apt update -y -qq\n",
" sudo apt install ffmpeg -y -qq\n",
" echo \"ffmpeg installed successfully on Linux.\"\n",
"elif [[ \"$os\" == \"Darwin\" ]]; then\n",
" # macOS installation\n",
" if command -v brew &> /dev/null; then\n",
" brew install ffmpeg\n",
" if [[ $? -eq 0 ]]; then\n",
" echo \"ffmpeg installed successfully on macOS using Homebrew.\"\n",
" else\n",
" echo \"Error installing ffmpeg on macOS using Homebrew.\"\n",
" fi\n",
" else\n",
" echo \"Homebrew is not installed. Please install Homebrew and try again.\"\n",
" fi\n",
"else\n",
" echo \"Unsupported operating system: $os\"\n",
"fi"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-cloud-texttospeech"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information and initialize SDK\n",
"\n",
"To get started using the Text-to-Speech API, you must have an existing Google Cloud project and [enable the API](https://console.cloud.google.com/flows/enableapi?apiid=texttospeech.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment).\n",
"\n",
"Please note the **available regions** for Chirp 3, see [documentation](https://cloud.google.com/text-to-speech/docs/endpoints)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "WIQyBhAn_9tK"
},
"outputs": [],
"source": [
"# Use the environment variable if the user doesn't provide Project ID.\n",
"import os\n",
"\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"TTS_LOCATION = \"global\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "76915236b1c9"
},
"outputs": [],
"source": [
"! gcloud config set project {PROJECT_ID}\n",
"! gcloud auth application-default set-quota-project {PROJECT_ID}\n",
"! gcloud auth application-default login -q"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5303c05f7aa6"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "qqm0OQpAYCph"
},
"outputs": [],
"source": [
"import re\n",
"from collections.abc import Iterator\n",
"\n",
"import numpy as np\n",
"from IPython.display import Audio, display\n",
"from google.api_core.client_options import ClientOptions\n",
"from google.cloud import texttospeech_v1beta1 as texttospeech"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sP8GBj3tBAC1"
},
"source": [
"### Set constants\n",
"\n",
"Initiate the API endpoint and the text to speech client.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rXTVeU1uBBqY"
},
"outputs": [],
"source": [
"API_ENDPOINT = (\n",
" f\"{TTS_LOCATION}-texttospeech.googleapis.com\"\n",
" if TTS_LOCATION != \"global\"\n",
" else \"texttospeech.googleapis.com\"\n",
")\n",
"\n",
"client = texttospeech.TextToSpeechClient(\n",
" client_options=ClientOptions(api_endpoint=API_ENDPOINT)\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "M7WQQFp_RvGH"
},
"source": [
"### Helpers"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "pWrNWdV_RxGn"
},
"outputs": [],
"source": [
"def text_generator(text: str) -> Iterator[str]:\n",
" \"\"\"Split text into sentences to simulate streaming\"\"\"\n",
" # Use regex with positive lookahead to find sentence boundaries\n",
" # without consuming the space after the punctuation\n",
" sentences: list[str] = re.findall(r\"[^.!?]+[.!?](?:\\s|$)\", text + \" \")\n",
"\n",
" # Yield each complete sentence\n",
" for sentence in sentences:\n",
" yield sentence.strip()\n",
"\n",
" # Check if there's remaining text not caught by the regex\n",
" # (text without ending punctuation)\n",
" last_char_pos: int = 0\n",
" for sentence in sentences:\n",
" last_char_pos += len(sentence)\n",
"\n",
" if last_char_pos < len(text.strip()):\n",
" remaining: str = text.strip()[last_char_pos:]\n",
" if remaining:\n",
" yield remaining.strip()\n",
"\n",
"\n",
"def process_streaming_audio(\n",
" text: str,\n",
" voice: texttospeech.VoiceSelectionParams,\n",
" display_individual_chunks: bool = False,\n",
") -> np.ndarray:\n",
" \"\"\"Process text into speech using streaming TTS\"\"\"\n",
" # Generate sentences from text\n",
" sentences: list[str] = list(text_generator(text))\n",
"\n",
" # Get streaming audio\n",
" print(\"Streaming audio processing...\")\n",
" audio_iterator: Iterator[bytes] = synthesize_streaming(iter(sentences), voice=voice)\n",
"\n",
" # Process audio chunks\n",
" final_audio_data: np.ndarray = np.array([], dtype=np.int16)\n",
"\n",
" for idx, audio_content in enumerate(audio_iterator):\n",
" audio_chunk: np.ndarray = np.frombuffer(audio_content, dtype=np.int16)\n",
"\n",
" # Concatenate to final audio\n",
" final_audio_data = np.concatenate((final_audio_data, audio_chunk))\n",
"\n",
" # Optionally display individual chunks\n",
" if display_individual_chunks and len(audio_chunk) > 0:\n",
" print(f\"Processed chunk # {idx}\")\n",
" display(Audio(audio_chunk, rate=24000))\n",
"\n",
" print(\"Streaming audio processing complete!\")\n",
" return final_audio_data\n",
"\n",
"\n",
"def synthesize_streaming(\n",
" text_iterator: Iterator[str],\n",
" voice: texttospeech.VoiceSelectionParams,\n",
") -> Iterator[bytes]:\n",
" \"\"\"Synthesizes speech from an iterator of text inputs and yields audio content as an iterator.\n",
"\n",
" This function demonstrates how to use the Google Cloud Text-to-Speech API\n",
" to synthesize speech from a stream of text inputs provided by an iterator.\n",
" It yields the audio content from each response as an iterator of bytes.\n",
"\n",
" \"\"\"\n",
" config_request = texttospeech.StreamingSynthesizeRequest(\n",
" streaming_config=texttospeech.StreamingSynthesizeConfig(\n",
" voice=voice,\n",
" )\n",
" )\n",
"\n",
" def request_generator() -> Iterator[texttospeech.StreamingSynthesizeRequest]:\n",
" yield config_request\n",
" for text in text_iterator:\n",
" yield texttospeech.StreamingSynthesizeRequest(\n",
" input=texttospeech.StreamingSynthesisInput(text=text)\n",
" )\n",
"\n",
" streaming_responses: Iterator[texttospeech.StreamingSynthesizeResponse] = (\n",
" client.streaming_synthesize(request_generator())\n",
" )\n",
"\n",
" for response in streaming_responses:\n",
" yield response.audio_content"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VPVDNRyVxquo"
},
"source": [
"## Synthesize using Chirp 3 HD voices\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a9aa2ab365ac"
},
"source": [
"### Synthesize speech using real-time (online) processing\n",
"\n",
"You define the text you want to convert, select a specific voice and language, and then instruct the API to generate an audio of the spoken text.\n",
"\n",
"This example uses the `en-US-Chirp3-HD-Aoede` voice, which is a high-definition voice, offering improved clarity. The code will call the `synthesize_speech` method, which handles the core conversion process, and the output will be an MP3 audio as `bytes`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_7XwbSxUD4eW"
},
"outputs": [],
"source": [
"# fmt: off\n",
"prompt = \"Hello world! I am Chirp 3\" # @param [\"Hallo Welt! Ich bin Chirp 3\", \"Hello world! I am Chirp 3\", \"¡Hola mundo! Soy Chirp 3\", \"Bonjour le monde ! Je suis Chirp 3\", \"नमस्ते दुनिया! मैं चिर्प 3 हूं\", \"Olá Mundo! Eu sou o Chirp 3\", \"مرحبا بالعالم! أنا تشيرب 3\", \"¡Hola mundo! Soy Chirp 3\", \"Bonjour le monde ! Je suis Chirp 3\", \"Halo dunia! Saya Chirp 3\", \"Ciao mondo! Sono Chirp 3\", \"こんにちは世界！私はチャープ3です\", \"Merhaba dünya! Ben Chirp 3\", \"Chào thế giới! Tôi là Chirp 3\", \"হ্যালো ওয়ার্ল্ড! আমি চির্প 3\", \"હેલો વર્લ્ડ! હું ચિર્પ 3 છું\", \"ನಮಸ್ಕಾರ ಪ್ರಪಂಚ! ನಾನು ಚಿರ್ಪ್ 3\", \"ഹലോ വേൾഡ്! ഞാൻ ചിർപ് 3 ആണ്\", \"नमस्कार जग! मी चिरप 3 आहे\", \"வணக்கம் உலகம்! நான் சிர்ப் 3\", \"హలో వరల్డ్! నేను చిర్ప్ 3\", \"Hallo wereld! Ik ben Chirp 3\", \"안녕하세요! 저는 Chirp 3입니다\", \"你好世界！我是 Chirp 3\", \"Witaj świecie! Jestem Chirp 3\", \"Привет, мир! Я Чирп 3\", \"สวัสดีชาวโลก! ฉันคือเชิร์ป 3\"]\n",
"\n",
"voice = \"Aoede\" # @param [\"Aoede\", \"Puck\", \"Charon\", \"Kore\", \"Fenrir\", \"Leda\", \"Orus\", \"Zephyr\"]\n",
"\n",
"language_code = \"en-US\" # @param [ \"de-DE\", \"en-AU\", \"en-GB\", \"en-IN\", \"en-US\", \"fr-FR\", \"hi-IN\", \"pt-BR\", \"ar-XA\", \"es-ES\", \"fr-CA\", \"id-ID\", \"it-IT\", \"ja-JP\", \"tr-TR\", \"vi-VN\", \"bn-IN\", \"gu-IN\", \"kn-IN\", \"ml-IN\", \"mr-IN\", \"ta-IN\", \"te-IN\", \"nl-NL\", \"ko-KR\", \"cmn-CN\", \"pl-PL\", \"ru-RU\", \"th-TH\"]\n",
"# fmt: on\n",
"\n",
"voice_name = f\"{language_code}-Chirp3-HD-{voice}\"\n",
"voice = texttospeech.VoiceSelectionParams(\n",
" name=voice_name,\n",
" language_code=language_code,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "98d104dc4d27"
},
"outputs": [],
"source": [
"# Perform the text-to-speech request on the text input with the selected\n",
"# voice parameters and audio file type\n",
"response = client.synthesize_speech(\n",
" input=texttospeech.SynthesisInput(text=prompt),\n",
" voice=voice,\n",
" # Select the type of audio file you want returned\n",
" audio_config=texttospeech.AudioConfig(\n",
" audio_encoding=texttospeech.AudioEncoding.MP3\n",
" ),\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fad3a3de36d9"
},
"source": [
"Play the generated audio."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "5e0881955f62"
},
"outputs": [],
"source": [
"display(Audio(response.audio_content))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LYKj5gmDG7nT"
},
"source": [
"### Synthesize speech using streaming processing\n",
"\n",
"Chirp 3 HD voices also support streaming text-to-speech conversion using the `streaming_synthesize` method. Unlike the standard `synthesize_speech`, which handles single requests, `streaming_synthesize` processes continuous streams of text, generating corresponding audio streams.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "HniB988UTUlq"
},
"outputs": [],
"source": [
"prompt = \"\"\"\n",
"Google Cloud Text-to-Speech (TTS) is a powerful API that converts text into natural-sounding audio. Here's a breakdown:\n",
"\n",
"**Key Features:**\n",
"\n",
"* **High-Fidelity Speech:** Leverages advanced AI to generate speech that closely resembles human voices.\n",
"* **Wide Voice Selection:** Offers a vast library of over 380 voices across 50+ languages and variants, catering to diverse needs.\n",
"* **Customization Options:**\n",
" * **Custom Voice:** Create unique voices tailored to your brand or specific requirements using your own audio recordings.\n",
" * **SSML Support:** Utilize Speech Synthesis Markup Language (SSML) to control pronunciation, pacing, and other speech nuances.\n",
"* **Integration Flexibility:** Easily integrate with various applications and devices via REST or gRPC APIs.\n",
"* **Use Cases:**\n",
" * **Voice Assistants:** Powering conversational AI in smart devices, chatbots, and voice-activated applications.\n",
" * **Accessibility:** Enabling screen readers and text-to-speech features for users with disabilities.\n",
" * **E-learning:** Creating engaging and accessible educational content.\n",
" * **Audiobooks and Podcasts:** Producing high-quality audio for audiobooks, podcasts, and other audio content.\n",
" * **Interactive Experiences:** Enhancing user experiences in games, virtual reality, and other interactive applications.\n",
"\n",
"**In essence, Google Cloud Text-to-Speech empowers developers and businesses to:**\n",
"\n",
"* **Enhance user experiences:** Create more engaging and inclusive interactions through natural-sounding speech.\n",
"* **Increase accessibility:** Make information more accessible to a wider audience, including those with visual impairments.\n",
"* **Improve efficiency:** Automate tasks like reading aloud documents, generating voiceovers, and creating interactive voice responses.\n",
"* **Innovate with new applications:** Explore novel use cases by leveraging the power of AI-powered speech synthesis.\n",
"\n",
"If you're looking to add a voice dimension to your applications or projects, Google Cloud Text-to-Speech is a valuable tool to consider.\n",
"\"\"\"\n",
"final_audio_data = process_streaming_audio(\n",
" prompt, voice, display_individual_chunks=False\n",
")\n",
"\n",
"display(Audio(final_audio_data, rate=24000))"
]
}
],
"metadata": {
"colab": {
"name": "get_started_with_chirp_3_hd_voices.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
audio/speech/getting-started/get_started_with_chirp_3_transcription.ipynb
+704
View File
@@ -0,0 +1,704 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Get started with Chirp 3 Transcription\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_transcription.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Faudio%2Fspeech%2Fgetting-started%2Fget_started_with_chirp_3_transcription.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/audio/speech/getting-started/get_started_with_chirp_3_transcription.ipynb\">\n",
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_transcription.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<p>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_transcription.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_transcription.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_transcription.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_transcription.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/audio/speech/getting-started/get_started_with_chirp_3_transcription.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| Authors |\n",
"| --- |\n",
"| [Katie Nguyen](https://github.com/katiemn) |\n",
"| [Holt Skinner](https://github.com/holtskinner) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"### Chirp 3\n",
"\n",
"This notebook introduces [Chirp 3](https://cloud.google.com/speech-to-text/v2/docs/chirp_3-model), Google's model for converting speech to text in multiple languages.\n",
"\n",
"In this tutorial, you'll learn how to use the Speech-to-Text API V2 to:\n",
"\n",
"- Transcribe an audio file with batch speech recognition\n",
"- Perform a language-agnostic transcription\n",
"- Use Chirp 3 for speaker diarization\n",
"- Perform streaming speech recognition"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install the Speech SDK and other required packages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-cloud-speech ipywebrtc"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5303c05f7aa6"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "qqm0OQpAYCph"
},
"outputs": [],
"source": [
"import json\n",
"import os\n",
"import re\n",
"from collections.abc import Generator\n",
"\n",
"from IPython.display import HTML, Audio, display\n",
"from google.api_core.client_options import ClientOptions\n",
"from google.cloud.speech_v2 import SpeechClient\n",
"from google.cloud.speech_v2.types import cloud_speech\n",
"from ipywebrtc import AudioRecorder, CameraStream"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using the Speech-to-Text API, you must have an existing Google Cloud project and [enable the API](https://console.cloud.google.com/flows/enableapi?apiid=speech.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment).\n",
"\n",
"Please note the **available regions** for Chirp 3, see [documentation](https://cloud.google.com/speech-to-text/v2/docs/chirp_3-model#regional_availability)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "WIQyBhAn_9tK"
},
"outputs": [],
"source": [
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"STT_LOCATION = \"us\" # @param {type: \"string\"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cR5PafOSrV2W"
},
"outputs": [],
"source": [
"! gcloud config set project {PROJECT_ID}\n",
"! gcloud auth application-default login -q\n",
"! gcloud auth application-default set-quota-project {PROJECT_ID}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sP8GBj3tBAC1"
},
"source": [
"### Create client\n",
"\n",
"Initiate the API endpoint and the Speech-to-Text client and define key constants."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rXTVeU1uBBqY"
},
"outputs": [],
"source": [
"client = SpeechClient(\n",
" client_options=ClientOptions(api_endpoint=f\"{STT_LOCATION}-speech.googleapis.com\")\n",
")\n",
"recognizer = client.recognizer_path(PROJECT_ID, STT_LOCATION, \"_\")\n",
"model = \"chirp_3\"\n",
"# Set a timeout for the batch recognition operation\n",
"MAX_AUDIO_LENGTH_SECS = 8 * 60 * 60"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "M7WQQFp_RvGH"
},
"source": [
"### Define helper functions"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "pWrNWdV_RxGn"
},
"outputs": [],
"source": [
"def print_transcript(results: list[cloud_speech.SpeechRecognitionResult]) -> None:\n",
" for result in results:\n",
" display(\n",
" HTML(\n",
" f\"\"\"<div style=\"word-break: break-all;\">{result.alternatives[0].transcript}</div>\"\"\"\n",
" )\n",
" )\n",
"\n",
"\n",
"def generate_audio_chunks(\n",
" audio_content: bytes, chunk_size: int\n",
") -> Generator[bytes, None, None]:\n",
" \"\"\"Splits a byte string (like audio data) into smaller, equal-sized chunks.\n",
"\n",
" Args:\n",
" audio_content: The raw byte data of the audio.\n",
" chunk_size: The desired size for each audio chunk in bytes.\n",
"\n",
" Yields:\n",
" A series of audio chunks.\n",
" \"\"\"\n",
" # Loop through the audio content, stepping by chunk_size each time\n",
" for start_index in range(0, len(audio_content), chunk_size):\n",
" # The end_index is the start_index plus the chunk size\n",
" end_index = start_index + chunk_size\n",
" # Yield the slice of audio data for the current chunk\n",
" yield audio_content[start_index:end_index]\n",
"\n",
"\n",
"def group_utterances_by_speaker_from_file(json_file_path: str) -> dict:\n",
" \"\"\"Reads a JSON file containing transcribed words and groups them into sentences spoken by each speaker.\"\"\"\n",
" with open(json_file_path, encoding=\"utf-8\") as f:\n",
" json_data_string = f.read()\n",
" words_regex = r'\"words\":\\s*(\\[.*?\\])'\n",
" match = re.search(words_regex, json_data_string, re.DOTALL)\n",
"\n",
" words_list = json.loads(match.group(1))\n",
" dialogue = []\n",
" current_speaker = None\n",
" current_utterance_words = []\n",
" current_speaker = words_list[0][\"speakerLabel\"]\n",
"\n",
" for item in words_list:\n",
" word = item[\"word\"]\n",
" speaker = item[\"speakerLabel\"]\n",
" # Check if the speaker has changed\n",
" if speaker != current_speaker:\n",
" dialogue.append(\n",
" {\"speaker\": current_speaker, \"text\": \" \".join(current_utterance_words)}\n",
" )\n",
" # Start a new utterance\n",
" current_speaker = speaker\n",
" current_utterance_words = [word]\n",
" else:\n",
" # Continue the current utterance\n",
" current_utterance_words.append(word)\n",
" # Add the final pending utterance\n",
" if current_speaker is not None:\n",
" dialogue.append(\n",
" {\"speaker\": current_speaker, \"text\": \" \".join(current_utterance_words)}\n",
" )\n",
"\n",
" return {\"dialogue\": dialogue}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VPVDNRyVxquo"
},
"source": [
"## Transcribe using Chirp 3\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LfBiMs0bNwhV"
},
"source": [
"### Online/Synchronous speech recognition\n",
"\n",
"You can use online (synchronous) speech recognition for audio files less than 1 minute long.\n",
"For this first request, Run the following cell to download and play the audio you'll be transcribing. If you'd like to use a different audio clip, modify the `audio_url` variable below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "yE_-iirV7UDA"
},
"outputs": [],
"source": [
"audio_url = \"https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/audio_summary_clean_energy_short.mp3\"\n",
"audio_filename = os.path.basename(audio_url)\n",
"! wget {audio_url} -O {audio_filename}\n",
"\n",
"display(Audio(filename=audio_filename))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lAfgmnZyPo1f"
},
"source": [
"Now, you'll send the `recognize` request. The transcription will be returned as part of the response and displayed in HTML for better visualization in this notebook."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "egBk4a3t0lUD"
},
"outputs": [],
"source": [
"config = cloud_speech.RecognitionConfig(\n",
" auto_decoding_config=cloud_speech.AutoDetectDecodingConfig(),\n",
" model=model,\n",
" language_codes=[\"en-US\"],\n",
")\n",
"\n",
"with open(audio_filename, \"rb\") as f:\n",
" audio_content = f.read()\n",
"\n",
"request = cloud_speech.RecognizeRequest(\n",
" recognizer=recognizer,\n",
" config=config,\n",
" content=audio_content,\n",
")\n",
"\n",
"response = client.recognize(request=request)\n",
"\n",
"print_transcript(response.results)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oJSi0bFQQYel"
},
"source": [
"### Perform a language-agnostic transcription\n",
"\n",
"In this next request, you'll perform a language-agnostic transcription. This means that Chirp 3 will automatically identify and transcribe the dominant language spoken in the audio, which is essential for multilingual applications.\n",
"\n",
"In this next example, you'll use a Spanish audio clip saved in Cloud Storage. To see a full list of the languages available for transcription, check the [documentation](https://cloud.google.com/speech-to-text/v2/docs/chirp_3-model#language_availability_for_transcription)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "zcTrgN9xHDXR"
},
"outputs": [],
"source": [
"audio_url = (\n",
" \"https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/spanish.wav\"\n",
")\n",
"audio_gcs_uri = audio_url.replace(\"https://storage.googleapis.com/\", \"gs://\")\n",
"\n",
"display(Audio(url=audio_url))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fB4cDExAMCnb"
},
"source": [
"This request is the similar to the previous one, except this time, you'll set `language_codes=[\"auto\"]`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "CphxakXgBEsz"
},
"outputs": [],
"source": [
"config = cloud_speech.RecognitionConfig(\n",
" auto_decoding_config=cloud_speech.AutoDetectDecodingConfig(),\n",
" model=model,\n",
" language_codes=[\"auto\"],\n",
")\n",
"request = cloud_speech.RecognizeRequest(\n",
" recognizer=recognizer,\n",
" config=config,\n",
" uri=audio_gcs_uri,\n",
")\n",
"\n",
"response = client.recognize(request=request)\n",
"\n",
"print_transcript(response.results)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1fOGjzlRMFMw"
},
"source": [
"### Speaker Diarization (Batch Recognition)\n",
"\n",
"Chirp 3 also supports speaker diarization, which means it can automatically identify the different speakers in a single-channel audio sample. See the [documentation](https://cloud.google.com/speech-to-text/v2/docs/chirp_3-model#language_availability_for_diarization) for a list of supported available languages for diarization.\n",
"\n",
"In this example, you'll also use the `batch_recognize` method to transcribe an audio file in Cloud Storage and save the output in Cloud Storage."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "FosRgcFqEMXX"
},
"outputs": [],
"source": [
"audio_url = \"https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/Chirp-3-Docs-Dive.mp3\"\n",
"audio_gcs_uri = audio_url.replace(\"https://storage.googleapis.com/\", \"gs://\")\n",
"display(Audio(url=audio_url))\n",
"\n",
"gcs_output_folder = \"gs://[your-bucket-path]\" # @param {type: \"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eb_gkdN9BvnP"
},
"source": [
"In order to enable speaker diarization, set the `diarization_config` in the `features` parameter of the `RecognitionConfig`.\n",
"\n",
"You'll also set your `gcs_output_folder` in a `RecognitionOutputConfig` so the transcription will be saved in Cloud Storage. To display the transcription, you'll copy the output JSON file and use a helper function to format it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Odgee64-xA2O"
},
"outputs": [],
"source": [
"config = cloud_speech.RecognitionConfig(\n",
" auto_decoding_config=cloud_speech.AutoDetectDecodingConfig(),\n",
" features=cloud_speech.RecognitionFeatures(\n",
" diarization_config=cloud_speech.SpeakerDiarizationConfig(),\n",
" ),\n",
" model=model,\n",
" language_codes=[\"en-US\"],\n",
")\n",
"\n",
"files = [cloud_speech.BatchRecognizeFileMetadata(uri=audio_gcs_uri)]\n",
"\n",
"request = cloud_speech.BatchRecognizeRequest(\n",
" recognizer=recognizer,\n",
" config=config,\n",
" files=files,\n",
" recognition_output_config=cloud_speech.RecognitionOutputConfig(\n",
" gcs_output_config=cloud_speech.GcsOutputConfig(uri=gcs_output_folder),\n",
" ),\n",
")\n",
"operation = client.batch_recognize(request=request)\n",
"response = operation.result(timeout=MAX_AUDIO_LENGTH_SECS)\n",
"\n",
"transcript = response.results[audio_gcs_uri].uri\n",
"\n",
"!gsutil cp {transcript} output.json\n",
"\n",
"print(json.dumps(group_utterances_by_speaker_from_file(\"output.json\"), indent=4))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yRFD1ByUDuqW"
},
"source": [
"### Streaming speech recognition\n",
"\n",
"In the following cells you'll simulate transcribing text from an audio stream. To start, you'll record an audio clip with your microphone by running the following cell."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "K2OrVxHeMxV2"
},
"outputs": [],
"source": [
"if \"google.colab\" in sys.modules:\n",
" from google.colab import output\n",
"\n",
" output.enable_custom_widget_manager()\n",
"\n",
"camera = CameraStream(constraints={\"audio\": True, \"video\": False})\n",
"recorder = AudioRecorder(stream=camera)\n",
"recorder"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PXhPcYx2ECJ-"
},
"source": [
"Once the audio is captured and you've stopped recording, you'll use FFmpeg to convert and save the clip to a MP3 file for processing."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "TtS36HTdP1hM"
},
"outputs": [],
"source": [
"with open(\"recording.webm\", \"wb\") as f:\n",
" f.write(recorder.audio.value)\n",
"\n",
"!ffmpeg -i recording.webm -vn -ar 44100 -ac 2 -f mp3 recording.mp3\n",
"audio_file = \"recording.mp3\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "swn2txVDEkSx"
},
"source": [
"Now, you'll read the audio file and generate audio chunks to simulate streaming from a helper function. You'll then use the `streaming_recognize` method to get the transcription from each audio chunk with help from a generator function to correctly structure the data stream."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "lwMRXTlf7q-d"
},
"outputs": [],
"source": [
"CHUNK_SIZE = 3200\n",
"\n",
"recognition_config = cloud_speech.RecognitionConfig(\n",
" auto_decoding_config=cloud_speech.AutoDetectDecodingConfig(),\n",
" language_codes=[\"auto\"],\n",
" model=model,\n",
")\n",
"\n",
"\n",
"def create_streaming_requests(\n",
" audio_file_path: str,\n",
") -> Generator[cloud_speech.StreamingRecognizeRequest, None, None]:\n",
" \"\"\"Prepares and yields all necessary requests for streaming speech recognition.\n",
"\n",
" First, it yields the configuration request, then it reads an audio file\n",
" and yields its content in chunks.\n",
"\n",
" Args:\n",
" audio_file_path: The path to the local audio file.\n",
"\n",
" Yields:\n",
" A stream of StreamingRecognizeRequest objects.\n",
" \"\"\"\n",
" # a. First, yield the initial configuration request.\n",
" config_request = cloud_speech.StreamingRecognizeRequest(\n",
" recognizer=recognizer,\n",
" streaming_config=cloud_speech.StreamingRecognitionConfig(\n",
" config=recognition_config,\n",
" ),\n",
" )\n",
" yield config_request\n",
"\n",
" # b. Second, read the audio file and stream it in chunks.\n",
" with open(audio_file_path, \"rb\") as f:\n",
" audio_content = f.read()\n",
"\n",
" for chunk in generate_audio_chunks(audio_content, CHUNK_SIZE):\n",
" yield cloud_speech.StreamingRecognizeRequest(audio=chunk)\n",
"\n",
"\n",
"responses = client.streaming_recognize(requests=create_streaming_requests(audio_file))\n",
"\n",
"print(\"Streaming transcripts:\")\n",
"all_transcripts = []\n",
"for response in responses:\n",
" for result in response.results:\n",
" transcript = result.alternatives[0].transcript\n",
" print(transcript)\n",
" all_transcripts.append(transcript)\n",
"\n",
"final_transcript = \" \".join(all_transcripts)\n",
"print(f\"\\n--- Final Combined Transcript ---\\n{final_transcript}\")"
]
}
],
"metadata": {
"colab": {
"name": "get_started_with_chirp_3_transcription.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
audio/speech/getting-started/get_started_with_gemini_tts_voices.ipynb
+1117
View File
File diff suppressed because it is too large Load Diff