wehub-resource-sync
6908 lines
372 KiB
Plaintext
6908 lines
372 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "cMQCs0oQf5Jo"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2026 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "iR0jdheRGG89"
|
|
},
|
|
"source": [
|
|
"# Introduction to Generative AI functions in BigQuery"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "td9kx9LVgSve"
|
|
},
|
|
"source": [
|
|
"<table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/applying-llms-to-data/bigquery_generative_ai_intro.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fuse-cases%2Fapplying-llms-to-data%2Fbigquery_generative_ai_intro.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/use-cases/applying-llms-to-data/bigquery_generative_ai_intro.ipynb\">\n",
|
|
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/bigquery/import?url=https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/applying-llms-to-data/bigquery_generative_ai_intro.ipynb\">\n",
|
|
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/bigquery/v1/32px.svg\" alt=\"BigQuery Studio logo\"><br> Open in BigQuery Studio\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/applying-llms-to-data/bigquery_generative_ai_intro.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>\n",
|
|
"\n",
|
|
"<b>Share to:</b>\n",
|
|
"\n",
|
|
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/applying-llms-to-data/bigquery_generative_ai_intro.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/applying-llms-to-data/bigquery_generative_ai_intro.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/applying-llms-to-data/bigquery_generative_ai_intro.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/applying-llms-to-data/bigquery_generative_ai_intro.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/applying-llms-to-data/bigquery_generative_ai_intro.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
|
"</a>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "5h3O5b6P8WEx"
|
|
},
|
|
"source": [
|
|
"| Author |\n",
|
|
"| --- |\n",
|
|
"| [Alicia Williams](https://github.com/aliciawilliams) |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "intro_md"
|
|
},
|
|
"source": [
|
|
"## Overview\n",
|
|
"\n",
|
|
"This tutorial will guide you through powerful generative AI capabilities available in BigQuery. You'll get hands-on experience using the suite of generative **`AI.*` functions** that integrate directly with powerful models like Gemini. This allows you to perform sophisticated AI-driven analysis on your data right within your familiar SQL environment.\n",
|
|
"\n",
|
|
"While the examples in this notebook will focus on **text inputs** to the generative AI models, many of these capabilities extend to **multi-modal analysis**."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "SwQLISjqGTHd"
|
|
},
|
|
"source": [
|
|
"### Objectives\n",
|
|
"\n",
|
|
"We'll cover how to:\n",
|
|
"\n",
|
|
"* **Prompt models to generate text and structured data** with `AI.GENERATE`, `AI.GENERATE_TABLE`, and `AI.GENERATE_TEXT`.\n",
|
|
"* **Perform powerful, row-level analysis** using scalar functions like `AI.GENERATE_BOOL`, `AI.GENERATE_DOUBLE`, and `AI.GENERATE_INT`.\n",
|
|
"* **Forecast future trends** with time-series data using `AI.FORECAST`.\n",
|
|
"\n",
|
|
"**Note**: This notebook focuses on the [general-purpose AI functions](https://docs.cloud.google.com/bigquery/docs/generative-ai-overview#general_purpose_ai) available in BigQuery. For more information on the [managed AI functions](https://docs.cloud.google.com/bigquery/docs/generative-ai-overview#managed_ai_functions) such as `AI.IF`, `AI.SCORE` and `AI.CLASSIFY`, please head to [Semantic Analysis in BigQuery with AI Functions](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/applying-llms-to-data/bigquery_ai_operators.ipynb)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "costs_md"
|
|
},
|
|
"source": [
|
|
"### Services and Costs\n",
|
|
"\n",
|
|
"This tutorial uses the following billable components of Google Cloud:\n",
|
|
"\n",
|
|
"* **BigQuery**: [Pricing](https://cloud.google.com/bigquery/pricing)\n",
|
|
"\n",
|
|
"* **BigQuery ML**: [Pricing](https://cloud.google.com/bigquery/pricing#bqml)\n",
|
|
"\n",
|
|
"* **Vertex AI**: [Pricing](https://cloud.google.com/vertex-ai/generative-ai/pricing)\n",
|
|
"\n",
|
|
"You can use the [Pricing Calculator](https://cloud.google.com/products/calculator) to generate a cost estimate based on your projected usage."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "nkoCFoFVSPii"
|
|
},
|
|
"source": [
|
|
"---\n",
|
|
"\n",
|
|
"## Before you begin"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "setup_md_1"
|
|
},
|
|
"source": [
|
|
"### Set up your Google Cloud project\n",
|
|
"**The following steps are required, regardless of your notebook environment.**\n",
|
|
"\n",
|
|
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
|
|
"\n",
|
|
"2. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
|
|
"\n",
|
|
"3. [Enable the BigQuery, BigQuery Connection, and Vertex AI APIs](https://console.cloud.google.com/flows/enableapi?apiid=bigquery.googleapis.com,bigqueryconnection.googleapis.com,aiplatform.googleapis.com).\n",
|
|
"\n",
|
|
"4. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "YQ3g-h7uTaSf"
|
|
},
|
|
"source": [
|
|
"### Set your project ID"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "set_project_id"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"PROJECT_ID = \"\" # @param {type:\"string\"}\n",
|
|
"\n",
|
|
"# Set the project id\n",
|
|
"! gcloud config set project {PROJECT_ID}"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "auth_md"
|
|
},
|
|
"source": [
|
|
"### Authenticate to your Google Cloud account\n",
|
|
"\n",
|
|
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "V6NjZRCXU5Ro"
|
|
},
|
|
"source": [
|
|
"**1. Colab Enterprise or BigQuery Studio Notebooks**\n",
|
|
"* Do nothing as you are already authenticated."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "l0dV1hvAU1ed"
|
|
},
|
|
"source": [
|
|
"**2. Colab, uncomment and run:**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "auth_code"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from google.colab import auth\n",
|
|
"\n",
|
|
"auth.authenticate_user()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "4Pp4LAJ3UyRP"
|
|
},
|
|
"source": [
|
|
"**3. Local JupyterLab instance, uncomment and run:**\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "AzU7S3fMVDkW"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# ! gcloud auth login"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "conn_md"
|
|
},
|
|
"source": [
|
|
"### Create BigQuery Cloud resource connection\n",
|
|
"\n",
|
|
"You will need to create a [Cloud resource connection](https://cloud.google.com/bigquery/docs/create-cloud-resource-connection) to enable BigQuery to interact with Vertex AI services."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "conn_code"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"!bq mk --connection --location=us \\\n",
|
|
" --connection_type=CLOUD_RESOURCE test_connection"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "perms_md"
|
|
},
|
|
"source": [
|
|
"### Set permissions for Service Account\n",
|
|
"\n",
|
|
"The resource connection service account requires certain project-level permissions to interact with Vertex AI."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "6OqpZNY953xR"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"SERVICE_ACCT = !bq show --format=prettyjson --connection us.test_connection | grep \"serviceAccountId\" | cut -d '\"' -f 4\n",
|
|
"SERVICE_ACCT_EMAIL = SERVICE_ACCT[-1]\n",
|
|
"print(SERVICE_ACCT_EMAIL)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "perms_code"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import time\n",
|
|
"\n",
|
|
"!gcloud projects add-iam-policy-binding --format=none $PROJECT_ID --member=serviceAccount:$SERVICE_ACCT_EMAIL --role='roles/bigquery.connectionUser'\n",
|
|
"!gcloud projects add-iam-policy-binding --format=none $PROJECT_ID --member=serviceAccount:$SERVICE_ACCT_EMAIL --role='roles/aiplatform.user'\n",
|
|
"\n",
|
|
"# wait 60 seconds, give IAM updates time to propagate, otherwise, following cells will fail\n",
|
|
"time.sleep(60)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "generate_md_1"
|
|
},
|
|
"source": [
|
|
"---\n",
|
|
"\n",
|
|
"## Generate text and structured data with `AI.GENERATE` and `AI.GENERATE_TABLE`\n",
|
|
"\n",
|
|
"These functions allow you to leverage the power of large language models (LLMs) directly within BigQuery to generate new text content, including creating content with a specified schema. Both functions work by sending requests to your choice of a [generally available](https://cloud.google.com/vertex-ai/generative-ai/docs/models#generally_available_models) or [preview](https://cloud.google.com/vertex-ai/generative-ai/docs/models#preview_models) Gemini model, and then returning that model's response."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "generate_text_md"
|
|
},
|
|
"source": [
|
|
"### Using `AI.GENERATE`: Extract keywords from reviews\n",
|
|
"\n",
|
|
"Let's use the [`AI.GENERATE`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate) function to extract keywords from the movie reviews in the `bigquery-public-data.imdb.reviews` table."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 44,
|
|
"metadata": {
|
|
"id": "generate_text_code"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Query is running: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Downloading: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.google.colaboratory.intrinsic+json": {
|
|
"summary": "{\n \"name\": \"get_ipython()\",\n \"rows\": 5,\n \"fields\": [\n {\n \"column\": \"review\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Remove yourself from the Kirk Douglass aspects of the casting. It is essential to your enjoying the film. There is a beautiful young woman playing double roles - and in the photos from the 1800's, I can't believe how smooth and white her skin is. Also, there is an excellent degrading of the film stock which chills the mind if you like faded greys and yellows as I do. This film is played on TNT from time to time so see it.\",\n \"Jules Verne wrote about 80 novels as well as plays and short stories in his career. He began writing in 1854 with a short story called \\\"Master Zacharias, or the Clockmaker's Soul\\\". It was the first time he talked of the negative side of progress - the evil that results from some discoveries or inventions when they fall into the wrong hands. This becomes a running theme in his novels: Captain Nemo in TWENTY THOUSAND LEAGUES UNDER THE SEA, or Robur (from ROBUR THE CONQUEROR and it's sequel, THE MASTER OF THE WORLD) are two of his best examples of this them. Kongre, in THE LIGHTHOUSE AT THE EDGE OF THE WORLD, is another.Verne was so prolific that when he died in 1905 he left a dozen unpublished novels and stories that were not fully published until 1910. They include some of his best writing, such as THE BARSAC MISSION (partly written by Verne's son Michael), THE SURVIVORS OF THE \\\"JONATHAN\\\", THE PURSUIT OF THE METEOR, THE DANUBE PILOT. All of these dealt with science, but also dealt with political systems, and economics, for Verne was interested in all the problems facing modern man. THE LIGHTHOUSE AT THE EDGE OF THE WORLD was the last novel that was published in Verne's lifetime. It does not deal with the political questions or economic ones that perplexed him, but seems to go back to his potboiler period, when he was turning out stories for money while considering better stories for later publication. But nothing Verne wrote is without interest. Rereading THE LIGHTHOUSE one sees what the subtle point is in it. It is the study of how the ego of a villain can prevent him from escaping retribution.Kongre (renamed Jonathan Kongre) is one of the last pirates in the world of 1900. He and his gang find a damaged boat and repair it. They sail it across the Pacific, and reach Staten Island, a small island in the Straits of Magellan controlled by Chile. There they find a lighthouse with a crew of three men. They manage to kill two of them, but the third one (named Vasquez - he's from Chile, remember), hides on the island. Kongre and his men decide that they should prepare to leave the island shortly, before the Chilean Naval relief boat returns in three months to pick up the lighthouse crew. But first they will wreck any boat that comes to the passage, and increase their ill-gotten gains. But the key to the novel (and it is not in the movie) is that Kongre's right hand men (Carcante and Vargas) keep urging him to pack up his supplies and wealth and head to Asia where the money can be divvied up and everyone separate in safety. And each time Kongre won't do it. Initially it is pure greed. He wrecks a boat, and massacres the crew (a scene that is done in the film). The sole survivor is an American, John Davis (the name became Denton in the film, except that it was given to the character of Vasquez). Now with an ally (and not a drunken one, as in the film), Vasquez starts sabotaging Kongre's activities on the island. Carcante keeps suggesting leaving, but Kongre (unused to someone annoying him successfully) keeps delaying in order to catch Vasquez and Davis. The end result is that when he thinks he has them cornered, the Chilean boat appears to sink his craft, kill most of his crew, and confront him. Kongre commits suicide to avoid capture.Much of the mayhem of the movie (with Denton picking off crew members one at a time) is not in the book. Nor is there any female character in the novel (a rarity in most of Verne's stories - he could be quite a feminist when he wished). The egotism of \\\"Jonathan\\\" Kongre is well shown by Yul Brynner's performance, but the subtlety of that trait is lost. The writers presumably did not think the audience could appreciate it. Kirk Douglas does well enough as Denton, but his singlehanded success (Vasquez and Davis work together well to the end of the story, unlike Denton's ally who is killed by the pirates) seems unlikely. The bestiality of the pirates is well shown in the movie, particularly a singularly tall actor who in one scene wears women's clothing to particularly unsettling effect. The film is not a bad minor adventure film, but it could have been better if they had stuck to Verne's theme.\",\n \"Surprising that Jules Verne would write such a story. Even more surprising that Hollywood would produce it. Yul Brynner is unbelievably good as a man freed of all bounds of convention, free to indulge his taste for cruelty and domination. Douglass is an excellent counterpoint, a courageous individual who's chosen simple solitude as a way to deal with the complications and turmoil society had imposed on him. And Samantha Egger's character is the capper, a woman willing to sacrifice far too much for comfort and safety. Inotherwords, everyman. The movie does show its age and is limited by the conventions of time and place and technology of the time. If you're looking for special effects and action as substitutes for thought, look elsewhere.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"keywords\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Here are the keywords from the text:\\n\\n* Film\\n* Casting\\n* Kirk Douglass\\n* Double roles\\n* 1800s\\n* Film stock degradation\\n* Faded colors (greys and yellows)\\n* TNT\",\n \"Here are the keywords extracted from the text:\\n\\n* Jules Verne\\n* Novels\\n* Short stories\\n* Plays\\n* \\\"Master Zacharias, or the Clockmaker's Soul\\\"\\n* Negative side of progress\\n* Discoveries\\n* Inventions\\n* Captain Nemo\\n* TWENTY THOUSAND LEAGUES UNDER THE SEA\\n* Robur\\n* ROBUR THE CONQUEROR\\n* THE MASTER OF THE WORLD\\n* Kongre (Jonathan Kongre)\\n* THE LIGHTHOUSE AT THE EDGE OF THE WORLD\\n* Science\\n* Political systems\\n* Economics\\n* Ego of a villain\\n* Retribution\\n* Pirates\\n* Staten Island\\n* Straits of Magellan\\n* Chile\\n* Lighthouse\\n* Vasquez\\n* John Davis\\n* Carcante\\n* Vargas\\n* Movie adaptation (of The Lighthouse)\\n* Yul Brynner\\n* Kirk Douglas\",\n \"Here are the keywords from the text:\\n\\n* Jules Verne\\n* Hollywood\\n* Yul Brynner\\n* Douglass\\n* Samantha Egger\\n* Cruelty\\n* Domination\\n* Solitude\\n* Sacrifice\\n* Convention\\n* Everyman\\n* Movie\\n* Age (of movie)\\n* Special effects\\n* Action\\n* Thought\\n* Technology\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}",
|
|
"type": "dataframe"
|
|
},
|
|
"text/html": [
|
|
"\n",
|
|
" <div id=\"df-a3a279b9-76b4-46fa-8341-59ee62aac24f\" class=\"colab-df-container\">\n",
|
|
" <div>\n",
|
|
"<style scoped>\n",
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|
" vertical-align: middle;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe tbody tr th {\n",
|
|
" vertical-align: top;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe thead th {\n",
|
|
" text-align: right;\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: right;\">\n",
|
|
" <th></th>\n",
|
|
" <th>review</th>\n",
|
|
" <th>keywords</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <th>0</th>\n",
|
|
" <td>The Light At The Edge of the World marks Kirk ...</td>\n",
|
|
" <td>Here are the keywords extracted from the text:...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>1</th>\n",
|
|
" <td>Remove yourself from the Kirk Douglass aspects...</td>\n",
|
|
" <td>Here are the keywords from the text:\\n\\n* Fi...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>2</th>\n",
|
|
" <td>Surprising that Jules Verne would write such a...</td>\n",
|
|
" <td>Here are the keywords from the text:\\n\\n* Ju...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>3</th>\n",
|
|
" <td>it is a quite slow pace face-to-face Brynner/D...</td>\n",
|
|
" <td>Here are the keywords extracted from the text:...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>4</th>\n",
|
|
" <td>Jules Verne wrote about 80 novels as well as p...</td>\n",
|
|
" <td>Here are the keywords extracted from the text:...</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>\n",
|
|
"</div>\n",
|
|
" <div class=\"colab-df-buttons\">\n",
|
|
"\n",
|
|
" <div class=\"colab-df-container\">\n",
|
|
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a3a279b9-76b4-46fa-8341-59ee62aac24f')\"\n",
|
|
" title=\"Convert this dataframe to an interactive table.\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
|
|
" <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
|
|
" </svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
" <style>\n",
|
|
" .colab-df-container {\n",
|
|
" display:flex;\n",
|
|
" gap: 12px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert {\n",
|
|
" background-color: #E8F0FE;\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: #1967D2;\n",
|
|
" height: 32px;\n",
|
|
" padding: 0 0 0 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert:hover {\n",
|
|
" background-color: #E2EBFA;\n",
|
|
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: #174EA6;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-buttons div {\n",
|
|
" margin-bottom: 4px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert {\n",
|
|
" background-color: #3B4455;\n",
|
|
" fill: #D2E3FC;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert:hover {\n",
|
|
" background-color: #434B5C;\n",
|
|
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
|
|
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
|
|
" fill: #FFFFFF;\n",
|
|
" }\n",
|
|
" </style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" const buttonEl =\n",
|
|
" document.querySelector('#df-a3a279b9-76b4-46fa-8341-59ee62aac24f button.colab-df-convert');\n",
|
|
" buttonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
"\n",
|
|
" async function convertToInteractive(key) {\n",
|
|
" const element = document.querySelector('#df-a3a279b9-76b4-46fa-8341-59ee62aac24f');\n",
|
|
" const dataTable =\n",
|
|
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
|
|
" [key], {});\n",
|
|
" if (!dataTable) return;\n",
|
|
"\n",
|
|
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
|
|
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
|
|
" + ' to learn more about interactive tables.';\n",
|
|
" element.innerHTML = '';\n",
|
|
" dataTable['output_type'] = 'display_data';\n",
|
|
" await google.colab.output.renderOutput(dataTable, element);\n",
|
|
" const docLink = document.createElement('div');\n",
|
|
" docLink.innerHTML = docLinkHtml;\n",
|
|
" element.appendChild(docLink);\n",
|
|
" }\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
"\n",
|
|
" <div id=\"df-2edfbc54-33f2-49b6-bdc2-5b0fb99095bc\">\n",
|
|
" <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-2edfbc54-33f2-49b6-bdc2-5b0fb99095bc')\"\n",
|
|
" title=\"Suggest charts\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
"<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
|
|
" width=\"24px\">\n",
|
|
" <g>\n",
|
|
" <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
|
|
" </g>\n",
|
|
"</svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
"<style>\n",
|
|
" .colab-df-quickchart {\n",
|
|
" --bg-color: #E8F0FE;\n",
|
|
" --fill-color: #1967D2;\n",
|
|
" --hover-bg-color: #E2EBFA;\n",
|
|
" --hover-fill-color: #174EA6;\n",
|
|
" --disabled-fill-color: #AAA;\n",
|
|
" --disabled-bg-color: #DDD;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-quickchart {\n",
|
|
" --bg-color: #3B4455;\n",
|
|
" --fill-color: #D2E3FC;\n",
|
|
" --hover-bg-color: #434B5C;\n",
|
|
" --hover-fill-color: #FFFFFF;\n",
|
|
" --disabled-bg-color: #3B4455;\n",
|
|
" --disabled-fill-color: #666;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart {\n",
|
|
" background-color: var(--bg-color);\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: var(--fill-color);\n",
|
|
" height: 32px;\n",
|
|
" padding: 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart:hover {\n",
|
|
" background-color: var(--hover-bg-color);\n",
|
|
" box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: var(--button-hover-fill-color);\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart-complete:disabled,\n",
|
|
" .colab-df-quickchart-complete:disabled:hover {\n",
|
|
" background-color: var(--disabled-bg-color);\n",
|
|
" fill: var(--disabled-fill-color);\n",
|
|
" box-shadow: none;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-spinner {\n",
|
|
" border: 2px solid var(--fill-color);\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" animation:\n",
|
|
" spin 1s steps(1) infinite;\n",
|
|
" }\n",
|
|
"\n",
|
|
" @keyframes spin {\n",
|
|
" 0% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 20% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 30% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 40% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 60% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 80% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 90% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" async function quickchart(key) {\n",
|
|
" const quickchartButtonEl =\n",
|
|
" document.querySelector('#' + key + ' button');\n",
|
|
" quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n",
|
|
" quickchartButtonEl.classList.add('colab-df-spinner');\n",
|
|
" try {\n",
|
|
" const charts = await google.colab.kernel.invokeFunction(\n",
|
|
" 'suggestCharts', [key], {});\n",
|
|
" } catch (error) {\n",
|
|
" console.error('Error during call to suggestCharts:', error);\n",
|
|
" }\n",
|
|
" quickchartButtonEl.classList.remove('colab-df-spinner');\n",
|
|
" quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
|
|
" }\n",
|
|
" (() => {\n",
|
|
" let quickchartButtonEl =\n",
|
|
" document.querySelector('#df-2edfbc54-33f2-49b6-bdc2-5b0fb99095bc button');\n",
|
|
" quickchartButtonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
" })();\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
" </div>\n",
|
|
" </div>\n"
|
|
],
|
|
"text/plain": [
|
|
" review \\\n",
|
|
"0 The Light At The Edge of the World marks Kirk ... \n",
|
|
"1 Remove yourself from the Kirk Douglass aspects... \n",
|
|
"2 Surprising that Jules Verne would write such a... \n",
|
|
"3 it is a quite slow pace face-to-face Brynner/D... \n",
|
|
"4 Jules Verne wrote about 80 novels as well as p... \n",
|
|
"\n",
|
|
" keywords \n",
|
|
"0 Here are the keywords extracted from the text:... \n",
|
|
"1 Here are the keywords from the text:\\n\\n* Fi... \n",
|
|
"2 Here are the keywords from the text:\\n\\n* Ju... \n",
|
|
"3 Here are the keywords extracted from the text:... \n",
|
|
"4 Here are the keywords extracted from the text:... "
|
|
]
|
|
},
|
|
"execution_count": 44,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"%%bigquery --project {PROJECT_ID}\n",
|
|
"\n",
|
|
"SELECT\n",
|
|
" review,\n",
|
|
" AI.GENERATE(('Extract the keywords from the text below: ', review),\n",
|
|
" connection_id => 'us.test_connection',\n",
|
|
" endpoint => 'gemini-2.5-flash').result AS keywords\n",
|
|
"FROM\n",
|
|
" `bigquery-public-data.imdb.reviews`\n",
|
|
"WHERE movie_id = \"tt0067345\"\n",
|
|
"LIMIT 5\n",
|
|
";"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "ZiVKjzbVcZ2x"
|
|
},
|
|
"source": [
|
|
"You can see that BigQuery has prompted the specified model for each row of the data and returned the response."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Ulkyyt6pUpfQ"
|
|
},
|
|
"source": [
|
|
"#### Using arguments to adjust model configuration\n",
|
|
"In addition to the `prompt`, `connection_id`, and `endpoint`, there are additional arguments available in `AI.GENERATE` at your disposal for customizing your generation request."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "bY5ksh35WMnd"
|
|
},
|
|
"source": [
|
|
"##### The `model_params` argument\n",
|
|
"There are several parameters you can specify within the `model_params` argument. You can read more about which specific parameters are available for `AI.GENERATE` in the [documentation](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate#arguments). The next query adds custom `temperature` and `maxOutputTokens` parameters.\n",
|
|
"\n",
|
|
"* The `temperature` parameter controls the randomness of the response, where a lower value makes the output more predictable and focused, while a higher value encourages more creative and diverse results.\n",
|
|
"\n",
|
|
"* The `maxOutputTokens` parameter sets the maximum length of the model's response by limiting the total number of tokens (pieces of words) it can generate.\n",
|
|
"\n",
|
|
"You can read more about the various parameters [here](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#generationconfig), and the parameters' default values are documented in the model card in the [model documentation for Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/models)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 45,
|
|
"metadata": {
|
|
"id": "61nesgOtWkWs"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Query is running: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Downloading: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.google.colaboratory.intrinsic+json": {
|
|
"summary": "{\n \"name\": \"get_ipython()\",\n \"rows\": 5,\n \"fields\": [\n {\n \"column\": \"review\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"it is a quite slow pace face-to-face Brynner/Douglas...i found there was some pretty heavy violence/gore but you know, seventies giallo style : (maybe some spoiler ahead) a little bit phony and fake but the image is there : the monkey torn apart, the flesh torn from the mechanic, etc...i don't remember if it is alike the novel...but overall i think it is a OK action flick, hero flick : he stands all alone at the end, all the bad guys are out..........and nice images too : a small rocky island, the sun over the seabut not a typical Verne's story-on-cinema : subs, sci-fi; but the adventure seen in a lot of Verne's novels tough\",\n \"Surprising that Jules Verne would write such a story. Even more surprising that Hollywood would produce it. Yul Brynner is unbelievably good as a man freed of all bounds of convention, free to indulge his taste for cruelty and domination. Douglass is an excellent counterpoint, a courageous individual who's chosen simple solitude as a way to deal with the complications and turmoil society had imposed on him. And Samantha Egger's character is the capper, a woman willing to sacrifice far too much for comfort and safety. Inotherwords, everyman. The movie does show its age and is limited by the conventions of time and place and technology of the time. If you're looking for special effects and action as substitutes for thought, look elsewhere.\",\n \"The Light At The Edge of the World marks Kirk Douglas's second filming of a Jules Verne novel. The first of course was one of his most popular films 20,000 Leagues Under The Sea. But this film is far more serious and has far more adult themes than Walt Disney's film aimed for the kid trade.This was the last novel Jules Verne had published during his lifetime and it's a story of survival against almost impossible odds. In the original novel Kirk Douglas's character was named Vasquez which certainly was more in keeping with someone assigned to lighthouse duty on Cape Horn. But in giving Douglas's character an Anglo name it better explains his presence on the island and it certainly is in keeping with the international tradition of Jules Verne's writings.Cape Horn is one of the loneliest parts of the globe and the geography of the southern tip of South America. Look on a map of the many islands and rocks in that part of the globe and imagine how rough the sea is because it has only limited space. It's not without reason that sailors in all cultures say that no one is really a true sailor until they've made a voyage crossing from the Atlantic to the Pacific Ocean in that area. Remember also this is 1865 as Yul Brynner identifies the year and the Panama Canal had not been built.Which makes the lighthouse at Cape Horn an international concern which was something that is ever present in Jules Verne's writings. But then as now there are malevolent forces in the world and they are in this story Yul Brynner and his pirate crew.On one desultory like any other down there, Yul Brynner's ship docks at the island and kills lighthouse keeper Fernando Rey and his young assistant Massimo Ranieri. By sheer dumb luck Douglas is not at the lighthouse when this happens, but he becomes a hunted man by Brynner and his pirate crew who want to set up headquarters there and use the light to pile up as many wrecks as they can plunder. Also they want to eliminate Douglas who's now the only witness to their crimes.I did like this film very much both when first seeing it in the theater and now on VHS. One thing of interest I found here is that there is no ambiguity, no shadings of character. Kirk Douglas is a good guy and Yul Brynner a bad one, no one is going to walk away thinking anything else. In fact Yul Brynner's pirate captain Jonathan Kongre is the most unredeemable villain we've seen on screen since Lee Marvin as Liberty Valance.Definitely fans of Kirk Douglas, Yul Brynner and Jules Verne should earmark this film for their collection.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"keywords\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Here are the keywords extracted from the text:\",\n \"Here are the keywords from the text\",\n \"Here are the keywords extracted\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}",
|
|
"type": "dataframe"
|
|
},
|
|
"text/html": [
|
|
"\n",
|
|
" <div id=\"df-3f97563e-9673-4d25-a8d2-f67bfb95c55b\" class=\"colab-df-container\">\n",
|
|
" <div>\n",
|
|
"<style scoped>\n",
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|
" vertical-align: middle;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe tbody tr th {\n",
|
|
" vertical-align: top;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe thead th {\n",
|
|
" text-align: right;\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: right;\">\n",
|
|
" <th></th>\n",
|
|
" <th>review</th>\n",
|
|
" <th>keywords</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <th>0</th>\n",
|
|
" <td>Jules Verne wrote about 80 novels as well as p...</td>\n",
|
|
" <td>Here's a list</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>1</th>\n",
|
|
" <td>it is a quite slow pace face-to-face Brynner/D...</td>\n",
|
|
" <td>Here are the keywords extracted from the text:</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>2</th>\n",
|
|
" <td>The Light At The Edge of the World marks Kirk ...</td>\n",
|
|
" <td>Here are the keywords extracted</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>3</th>\n",
|
|
" <td>Remove yourself from the Kirk Douglass aspects...</td>\n",
|
|
" <td>Here are the keywords from the text:\\n\\n*</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>4</th>\n",
|
|
" <td>Surprising that Jules Verne would write such a...</td>\n",
|
|
" <td>Here are the keywords from the text</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>\n",
|
|
"</div>\n",
|
|
" <div class=\"colab-df-buttons\">\n",
|
|
"\n",
|
|
" <div class=\"colab-df-container\">\n",
|
|
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3f97563e-9673-4d25-a8d2-f67bfb95c55b')\"\n",
|
|
" title=\"Convert this dataframe to an interactive table.\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
|
|
" <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
|
|
" </svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
" <style>\n",
|
|
" .colab-df-container {\n",
|
|
" display:flex;\n",
|
|
" gap: 12px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert {\n",
|
|
" background-color: #E8F0FE;\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: #1967D2;\n",
|
|
" height: 32px;\n",
|
|
" padding: 0 0 0 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert:hover {\n",
|
|
" background-color: #E2EBFA;\n",
|
|
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: #174EA6;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-buttons div {\n",
|
|
" margin-bottom: 4px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert {\n",
|
|
" background-color: #3B4455;\n",
|
|
" fill: #D2E3FC;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert:hover {\n",
|
|
" background-color: #434B5C;\n",
|
|
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
|
|
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
|
|
" fill: #FFFFFF;\n",
|
|
" }\n",
|
|
" </style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" const buttonEl =\n",
|
|
" document.querySelector('#df-3f97563e-9673-4d25-a8d2-f67bfb95c55b button.colab-df-convert');\n",
|
|
" buttonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
"\n",
|
|
" async function convertToInteractive(key) {\n",
|
|
" const element = document.querySelector('#df-3f97563e-9673-4d25-a8d2-f67bfb95c55b');\n",
|
|
" const dataTable =\n",
|
|
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
|
|
" [key], {});\n",
|
|
" if (!dataTable) return;\n",
|
|
"\n",
|
|
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
|
|
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
|
|
" + ' to learn more about interactive tables.';\n",
|
|
" element.innerHTML = '';\n",
|
|
" dataTable['output_type'] = 'display_data';\n",
|
|
" await google.colab.output.renderOutput(dataTable, element);\n",
|
|
" const docLink = document.createElement('div');\n",
|
|
" docLink.innerHTML = docLinkHtml;\n",
|
|
" element.appendChild(docLink);\n",
|
|
" }\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
"\n",
|
|
" <div id=\"df-7d415ab7-d6f5-4c22-832c-5a35e362c835\">\n",
|
|
" <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-7d415ab7-d6f5-4c22-832c-5a35e362c835')\"\n",
|
|
" title=\"Suggest charts\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
"<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
|
|
" width=\"24px\">\n",
|
|
" <g>\n",
|
|
" <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
|
|
" </g>\n",
|
|
"</svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
"<style>\n",
|
|
" .colab-df-quickchart {\n",
|
|
" --bg-color: #E8F0FE;\n",
|
|
" --fill-color: #1967D2;\n",
|
|
" --hover-bg-color: #E2EBFA;\n",
|
|
" --hover-fill-color: #174EA6;\n",
|
|
" --disabled-fill-color: #AAA;\n",
|
|
" --disabled-bg-color: #DDD;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-quickchart {\n",
|
|
" --bg-color: #3B4455;\n",
|
|
" --fill-color: #D2E3FC;\n",
|
|
" --hover-bg-color: #434B5C;\n",
|
|
" --hover-fill-color: #FFFFFF;\n",
|
|
" --disabled-bg-color: #3B4455;\n",
|
|
" --disabled-fill-color: #666;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart {\n",
|
|
" background-color: var(--bg-color);\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: var(--fill-color);\n",
|
|
" height: 32px;\n",
|
|
" padding: 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart:hover {\n",
|
|
" background-color: var(--hover-bg-color);\n",
|
|
" box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: var(--button-hover-fill-color);\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart-complete:disabled,\n",
|
|
" .colab-df-quickchart-complete:disabled:hover {\n",
|
|
" background-color: var(--disabled-bg-color);\n",
|
|
" fill: var(--disabled-fill-color);\n",
|
|
" box-shadow: none;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-spinner {\n",
|
|
" border: 2px solid var(--fill-color);\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" animation:\n",
|
|
" spin 1s steps(1) infinite;\n",
|
|
" }\n",
|
|
"\n",
|
|
" @keyframes spin {\n",
|
|
" 0% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 20% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 30% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 40% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 60% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 80% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 90% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" async function quickchart(key) {\n",
|
|
" const quickchartButtonEl =\n",
|
|
" document.querySelector('#' + key + ' button');\n",
|
|
" quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n",
|
|
" quickchartButtonEl.classList.add('colab-df-spinner');\n",
|
|
" try {\n",
|
|
" const charts = await google.colab.kernel.invokeFunction(\n",
|
|
" 'suggestCharts', [key], {});\n",
|
|
" } catch (error) {\n",
|
|
" console.error('Error during call to suggestCharts:', error);\n",
|
|
" }\n",
|
|
" quickchartButtonEl.classList.remove('colab-df-spinner');\n",
|
|
" quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
|
|
" }\n",
|
|
" (() => {\n",
|
|
" let quickchartButtonEl =\n",
|
|
" document.querySelector('#df-7d415ab7-d6f5-4c22-832c-5a35e362c835 button');\n",
|
|
" quickchartButtonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
" })();\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
" </div>\n",
|
|
" </div>\n"
|
|
],
|
|
"text/plain": [
|
|
" review \\\n",
|
|
"0 Jules Verne wrote about 80 novels as well as p... \n",
|
|
"1 it is a quite slow pace face-to-face Brynner/D... \n",
|
|
"2 The Light At The Edge of the World marks Kirk ... \n",
|
|
"3 Remove yourself from the Kirk Douglass aspects... \n",
|
|
"4 Surprising that Jules Verne would write such a... \n",
|
|
"\n",
|
|
" keywords \n",
|
|
"0 Here's a list \n",
|
|
"1 Here are the keywords extracted from the text: \n",
|
|
"2 Here are the keywords extracted \n",
|
|
"3 Here are the keywords from the text:\\n\\n* \n",
|
|
"4 Here are the keywords from the text "
|
|
]
|
|
},
|
|
"execution_count": 45,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"%%bigquery --project {PROJECT_ID}\n",
|
|
"\n",
|
|
"SELECT\n",
|
|
" review,\n",
|
|
" AI.GENERATE(('Extract the keywords from the text below: ', review),\n",
|
|
" connection_id => 'us.test_connection',\n",
|
|
" endpoint => 'gemini-2.5-flash',\n",
|
|
" model_params => JSON '{\"generationConfig\":{\"temperature\": 0.5, \"maxOutputTokens\": 250}}'\n",
|
|
" ).result AS keywords\n",
|
|
"FROM\n",
|
|
" `bigquery-public-data.imdb.reviews`\n",
|
|
"WHERE movie_id = \"tt0067345\"\n",
|
|
"LIMIT 5\n",
|
|
";"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "FnSyQrNzb_4A"
|
|
},
|
|
"source": [
|
|
"Why are some of the responses truncated or cut off? By setting the `maxOutputTokens` value to 250 (much lower than the default 65,535 which was used in the first queries), you are seeing this limit in action. Gemini 2.5 Flash has [thinking capabilities](https://cloud.google.com/vertex-ai/generative-ai/docs/thinking), which means the model goes through a \"thinking process\" before determining and returning its response, and this thinking process consumes output tokens. In this case, the model is consuming most of the `maxOutputTokens` value before it moves from its thinking process to returning the actual response.\n",
|
|
"\n",
|
|
"You can resolve this issue by increasing the `maxOutputTokens` value and/or setting a `thinking_budget` to limit the amount of tokens that can be consumed by the thinking process."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "aBBpEQIYJ2Dt"
|
|
},
|
|
"source": [
|
|
"##### Setting a `thinking_budget`\n",
|
|
"\n",
|
|
"You can set a [`thinking_budget`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate#thinking-budget) to manage how many output tokens are allocated to the thinking phase of the model's response development.\n",
|
|
"\n",
|
|
"The next query shows how to set the `thinking_budget` as an additional parameter in the `model_params` argument."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 46,
|
|
"metadata": {
|
|
"id": "L2fNyr7aGbDP"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Query is running: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Downloading: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.google.colaboratory.intrinsic+json": {
|
|
"summary": "{\n \"name\": \"get_ipython()\",\n \"rows\": 5,\n \"fields\": [\n {\n \"column\": \"review\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"The Light At The Edge of the World marks Kirk Douglas's second filming of a Jules Verne novel. The first of course was one of his most popular films 20,000 Leagues Under The Sea. But this film is far more serious and has far more adult themes than Walt Disney's film aimed for the kid trade.This was the last novel Jules Verne had published during his lifetime and it's a story of survival against almost impossible odds. In the original novel Kirk Douglas's character was named Vasquez which certainly was more in keeping with someone assigned to lighthouse duty on Cape Horn. But in giving Douglas's character an Anglo name it better explains his presence on the island and it certainly is in keeping with the international tradition of Jules Verne's writings.Cape Horn is one of the loneliest parts of the globe and the geography of the southern tip of South America. Look on a map of the many islands and rocks in that part of the globe and imagine how rough the sea is because it has only limited space. It's not without reason that sailors in all cultures say that no one is really a true sailor until they've made a voyage crossing from the Atlantic to the Pacific Ocean in that area. Remember also this is 1865 as Yul Brynner identifies the year and the Panama Canal had not been built.Which makes the lighthouse at Cape Horn an international concern which was something that is ever present in Jules Verne's writings. But then as now there are malevolent forces in the world and they are in this story Yul Brynner and his pirate crew.On one desultory like any other down there, Yul Brynner's ship docks at the island and kills lighthouse keeper Fernando Rey and his young assistant Massimo Ranieri. By sheer dumb luck Douglas is not at the lighthouse when this happens, but he becomes a hunted man by Brynner and his pirate crew who want to set up headquarters there and use the light to pile up as many wrecks as they can plunder. Also they want to eliminate Douglas who's now the only witness to their crimes.I did like this film very much both when first seeing it in the theater and now on VHS. One thing of interest I found here is that there is no ambiguity, no shadings of character. Kirk Douglas is a good guy and Yul Brynner a bad one, no one is going to walk away thinking anything else. In fact Yul Brynner's pirate captain Jonathan Kongre is the most unredeemable villain we've seen on screen since Lee Marvin as Liberty Valance.Definitely fans of Kirk Douglas, Yul Brynner and Jules Verne should earmark this film for their collection.\",\n \"Surprising that Jules Verne would write such a story. Even more surprising that Hollywood would produce it. Yul Brynner is unbelievably good as a man freed of all bounds of convention, free to indulge his taste for cruelty and domination. Douglass is an excellent counterpoint, a courageous individual who's chosen simple solitude as a way to deal with the complications and turmoil society had imposed on him. And Samantha Egger's character is the capper, a woman willing to sacrifice far too much for comfort and safety. Inotherwords, everyman. The movie does show its age and is limited by the conventions of time and place and technology of the time. If you're looking for special effects and action as substitutes for thought, look elsewhere.\",\n \"Jules Verne wrote about 80 novels as well as plays and short stories in his career. He began writing in 1854 with a short story called \\\"Master Zacharias, or the Clockmaker's Soul\\\". It was the first time he talked of the negative side of progress - the evil that results from some discoveries or inventions when they fall into the wrong hands. This becomes a running theme in his novels: Captain Nemo in TWENTY THOUSAND LEAGUES UNDER THE SEA, or Robur (from ROBUR THE CONQUEROR and it's sequel, THE MASTER OF THE WORLD) are two of his best examples of this them. Kongre, in THE LIGHTHOUSE AT THE EDGE OF THE WORLD, is another.Verne was so prolific that when he died in 1905 he left a dozen unpublished novels and stories that were not fully published until 1910. They include some of his best writing, such as THE BARSAC MISSION (partly written by Verne's son Michael), THE SURVIVORS OF THE \\\"JONATHAN\\\", THE PURSUIT OF THE METEOR, THE DANUBE PILOT. All of these dealt with science, but also dealt with political systems, and economics, for Verne was interested in all the problems facing modern man. THE LIGHTHOUSE AT THE EDGE OF THE WORLD was the last novel that was published in Verne's lifetime. It does not deal with the political questions or economic ones that perplexed him, but seems to go back to his potboiler period, when he was turning out stories for money while considering better stories for later publication. But nothing Verne wrote is without interest. Rereading THE LIGHTHOUSE one sees what the subtle point is in it. It is the study of how the ego of a villain can prevent him from escaping retribution.Kongre (renamed Jonathan Kongre) is one of the last pirates in the world of 1900. He and his gang find a damaged boat and repair it. They sail it across the Pacific, and reach Staten Island, a small island in the Straits of Magellan controlled by Chile. There they find a lighthouse with a crew of three men. They manage to kill two of them, but the third one (named Vasquez - he's from Chile, remember), hides on the island. Kongre and his men decide that they should prepare to leave the island shortly, before the Chilean Naval relief boat returns in three months to pick up the lighthouse crew. But first they will wreck any boat that comes to the passage, and increase their ill-gotten gains. But the key to the novel (and it is not in the movie) is that Kongre's right hand men (Carcante and Vargas) keep urging him to pack up his supplies and wealth and head to Asia where the money can be divvied up and everyone separate in safety. And each time Kongre won't do it. Initially it is pure greed. He wrecks a boat, and massacres the crew (a scene that is done in the film). The sole survivor is an American, John Davis (the name became Denton in the film, except that it was given to the character of Vasquez). Now with an ally (and not a drunken one, as in the film), Vasquez starts sabotaging Kongre's activities on the island. Carcante keeps suggesting leaving, but Kongre (unused to someone annoying him successfully) keeps delaying in order to catch Vasquez and Davis. The end result is that when he thinks he has them cornered, the Chilean boat appears to sink his craft, kill most of his crew, and confront him. Kongre commits suicide to avoid capture.Much of the mayhem of the movie (with Denton picking off crew members one at a time) is not in the book. Nor is there any female character in the novel (a rarity in most of Verne's stories - he could be quite a feminist when he wished). The egotism of \\\"Jonathan\\\" Kongre is well shown by Yul Brynner's performance, but the subtlety of that trait is lost. The writers presumably did not think the audience could appreciate it. Kirk Douglas does well enough as Denton, but his singlehanded success (Vasquez and Davis work together well to the end of the story, unlike Denton's ally who is killed by the pirates) seems unlikely. The bestiality of the pirates is well shown in the movie, particularly a singularly tall actor who in one scene wears women's clothing to particularly unsettling effect. The film is not a bad minor adventure film, but it could have been better if they had stuck to Verne's theme.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"keywords\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Here are the keywords extracted from the text:\\n\\n* The Light At The Edge of the World\\n* Kirk Douglas\\n* Jules Verne\\n* 20,000 Leagues Under The Sea\\n* Cape Horn\\n* Yul Brynner\\n* lighthouse\\n* pirates\\n* survival\\n* novel\\n* film\\n* Jonathan Kongre\\n* Fernando Rey\\n* Massimo Ranieri\\n* Atlantic to Pacific Ocean\\n* 1865\\n* Panama Canal\\n* villain\\n* hero / good guy\\n* bad guy\",\n \"Here are the keywords extracted from the text:\\n\\n* Jules Verne\\n* Hollywood\\n* Yul Brynner\\n* Cruelty\\n* Domination\\n* Douglass\\n* Solitude\\n* Samantha Egger\\n* Sacrifice\\n* Comfort\\n* Safety\\n* Everyman\\n* Movie\\n* Conventions (of time, place, technology)\\n* Thought (as opposed to special effects/action)\",\n \"Here are the keywords extracted from the text:\\n\\n* Jules Verne\\n* Novels\\n* Short stories\\n* Master Zacharias, or the Clockmaker's Soul\\n* Negative side of progress\\n* Evil of discoveries/inventions\\n* Captain Nemo\\n* Twenty Thousand Leagues Under the Sea\\n* Robur the Conqueror\\n* The Master of the World\\n* Kongre\\n* The Lighthouse at the Edge of the World\\n* Unpublished novels\\n* The Barsac Mission\\n* The Survivors of the \\\"Jonathan\\\"\\n* The Pursuit of the Meteor\\n* The Danube Pilot\\n* Science\\n* Political systems\\n* Economics\\n*\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}",
|
|
"type": "dataframe"
|
|
},
|
|
"text/html": [
|
|
"\n",
|
|
" <div id=\"df-120134fd-a61f-4fe0-87ba-36d754225497\" class=\"colab-df-container\">\n",
|
|
" <div>\n",
|
|
"<style scoped>\n",
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|
" vertical-align: middle;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe tbody tr th {\n",
|
|
" vertical-align: top;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe thead th {\n",
|
|
" text-align: right;\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: right;\">\n",
|
|
" <th></th>\n",
|
|
" <th>review</th>\n",
|
|
" <th>keywords</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <th>0</th>\n",
|
|
" <td>it is a quite slow pace face-to-face Brynner/D...</td>\n",
|
|
" <td>Here are the keywords extracted from the text:...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>1</th>\n",
|
|
" <td>The Light At The Edge of the World marks Kirk ...</td>\n",
|
|
" <td>Here are the keywords extracted from the text:...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>2</th>\n",
|
|
" <td>Jules Verne wrote about 80 novels as well as p...</td>\n",
|
|
" <td>Here are the keywords extracted from the text:...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>3</th>\n",
|
|
" <td>Remove yourself from the Kirk Douglass aspects...</td>\n",
|
|
" <td>Here are the keywords extracted from the text:...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>4</th>\n",
|
|
" <td>Surprising that Jules Verne would write such a...</td>\n",
|
|
" <td>Here are the keywords extracted from the text:...</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>\n",
|
|
"</div>\n",
|
|
" <div class=\"colab-df-buttons\">\n",
|
|
"\n",
|
|
" <div class=\"colab-df-container\">\n",
|
|
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-120134fd-a61f-4fe0-87ba-36d754225497')\"\n",
|
|
" title=\"Convert this dataframe to an interactive table.\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
|
|
" <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
|
|
" </svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
" <style>\n",
|
|
" .colab-df-container {\n",
|
|
" display:flex;\n",
|
|
" gap: 12px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert {\n",
|
|
" background-color: #E8F0FE;\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: #1967D2;\n",
|
|
" height: 32px;\n",
|
|
" padding: 0 0 0 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert:hover {\n",
|
|
" background-color: #E2EBFA;\n",
|
|
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: #174EA6;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-buttons div {\n",
|
|
" margin-bottom: 4px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert {\n",
|
|
" background-color: #3B4455;\n",
|
|
" fill: #D2E3FC;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert:hover {\n",
|
|
" background-color: #434B5C;\n",
|
|
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
|
|
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
|
|
" fill: #FFFFFF;\n",
|
|
" }\n",
|
|
" </style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" const buttonEl =\n",
|
|
" document.querySelector('#df-120134fd-a61f-4fe0-87ba-36d754225497 button.colab-df-convert');\n",
|
|
" buttonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
"\n",
|
|
" async function convertToInteractive(key) {\n",
|
|
" const element = document.querySelector('#df-120134fd-a61f-4fe0-87ba-36d754225497');\n",
|
|
" const dataTable =\n",
|
|
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
|
|
" [key], {});\n",
|
|
" if (!dataTable) return;\n",
|
|
"\n",
|
|
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
|
|
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
|
|
" + ' to learn more about interactive tables.';\n",
|
|
" element.innerHTML = '';\n",
|
|
" dataTable['output_type'] = 'display_data';\n",
|
|
" await google.colab.output.renderOutput(dataTable, element);\n",
|
|
" const docLink = document.createElement('div');\n",
|
|
" docLink.innerHTML = docLinkHtml;\n",
|
|
" element.appendChild(docLink);\n",
|
|
" }\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
"\n",
|
|
" <div id=\"df-6f078c4f-93a6-4759-ac15-a3edecb464da\">\n",
|
|
" <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-6f078c4f-93a6-4759-ac15-a3edecb464da')\"\n",
|
|
" title=\"Suggest charts\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
"<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
|
|
" width=\"24px\">\n",
|
|
" <g>\n",
|
|
" <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
|
|
" </g>\n",
|
|
"</svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
"<style>\n",
|
|
" .colab-df-quickchart {\n",
|
|
" --bg-color: #E8F0FE;\n",
|
|
" --fill-color: #1967D2;\n",
|
|
" --hover-bg-color: #E2EBFA;\n",
|
|
" --hover-fill-color: #174EA6;\n",
|
|
" --disabled-fill-color: #AAA;\n",
|
|
" --disabled-bg-color: #DDD;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-quickchart {\n",
|
|
" --bg-color: #3B4455;\n",
|
|
" --fill-color: #D2E3FC;\n",
|
|
" --hover-bg-color: #434B5C;\n",
|
|
" --hover-fill-color: #FFFFFF;\n",
|
|
" --disabled-bg-color: #3B4455;\n",
|
|
" --disabled-fill-color: #666;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart {\n",
|
|
" background-color: var(--bg-color);\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: var(--fill-color);\n",
|
|
" height: 32px;\n",
|
|
" padding: 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart:hover {\n",
|
|
" background-color: var(--hover-bg-color);\n",
|
|
" box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: var(--button-hover-fill-color);\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart-complete:disabled,\n",
|
|
" .colab-df-quickchart-complete:disabled:hover {\n",
|
|
" background-color: var(--disabled-bg-color);\n",
|
|
" fill: var(--disabled-fill-color);\n",
|
|
" box-shadow: none;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-spinner {\n",
|
|
" border: 2px solid var(--fill-color);\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" animation:\n",
|
|
" spin 1s steps(1) infinite;\n",
|
|
" }\n",
|
|
"\n",
|
|
" @keyframes spin {\n",
|
|
" 0% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 20% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 30% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 40% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 60% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 80% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 90% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" async function quickchart(key) {\n",
|
|
" const quickchartButtonEl =\n",
|
|
" document.querySelector('#' + key + ' button');\n",
|
|
" quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n",
|
|
" quickchartButtonEl.classList.add('colab-df-spinner');\n",
|
|
" try {\n",
|
|
" const charts = await google.colab.kernel.invokeFunction(\n",
|
|
" 'suggestCharts', [key], {});\n",
|
|
" } catch (error) {\n",
|
|
" console.error('Error during call to suggestCharts:', error);\n",
|
|
" }\n",
|
|
" quickchartButtonEl.classList.remove('colab-df-spinner');\n",
|
|
" quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
|
|
" }\n",
|
|
" (() => {\n",
|
|
" let quickchartButtonEl =\n",
|
|
" document.querySelector('#df-6f078c4f-93a6-4759-ac15-a3edecb464da button');\n",
|
|
" quickchartButtonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
" })();\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
" </div>\n",
|
|
" </div>\n"
|
|
],
|
|
"text/plain": [
|
|
" review \\\n",
|
|
"0 it is a quite slow pace face-to-face Brynner/D... \n",
|
|
"1 The Light At The Edge of the World marks Kirk ... \n",
|
|
"2 Jules Verne wrote about 80 novels as well as p... \n",
|
|
"3 Remove yourself from the Kirk Douglass aspects... \n",
|
|
"4 Surprising that Jules Verne would write such a... \n",
|
|
"\n",
|
|
" keywords \n",
|
|
"0 Here are the keywords extracted from the text:... \n",
|
|
"1 Here are the keywords extracted from the text:... \n",
|
|
"2 Here are the keywords extracted from the text:... \n",
|
|
"3 Here are the keywords extracted from the text:... \n",
|
|
"4 Here are the keywords extracted from the text:... "
|
|
]
|
|
},
|
|
"execution_count": 46,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"%%bigquery --project {PROJECT_ID}\n",
|
|
"\n",
|
|
"SELECT\n",
|
|
" review,\n",
|
|
" AI.GENERATE(('Extract the keywords from the text below: ', review),\n",
|
|
" connection_id => 'us.test_connection',\n",
|
|
" endpoint => 'gemini-2.5-flash',\n",
|
|
" model_params => JSON '{\"generationConfig\":{\"temperature\": 0.5, \"maxOutputTokens\": 250, \"thinking_config\": {\"thinking_budget\": 100}}}'\n",
|
|
" ).result AS keywords\n",
|
|
"FROM\n",
|
|
" `bigquery-public-data.imdb.reviews`\n",
|
|
"WHERE movie_id = \"tt0067345\"\n",
|
|
"LIMIT 5\n",
|
|
";"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "edQ8hdPb1QMm"
|
|
},
|
|
"source": [
|
|
"##### The `output_schema` argument\n",
|
|
"\n",
|
|
"As you can see in the last query, the model's responses were not provided in a standard structure. You can use the **`output_schema`** argument to define the format of the responses.\n",
|
|
"\n",
|
|
"First, you'll specify the output as a string of key words by adding `output_schema => 'keywords ARRAY<STRING>'` to the `AI.GENERATE` request."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 47,
|
|
"metadata": {
|
|
"id": "7w-oUv8UimxT"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Query is running: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Downloading: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.google.colaboratory.intrinsic+json": {
|
|
"summary": "{\n \"name\": \"get_ipython()\",\n \"rows\": 5,\n \"fields\": [\n {\n \"column\": \"review\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Remove yourself from the Kirk Douglass aspects of the casting. It is essential to your enjoying the film. There is a beautiful young woman playing double roles - and in the photos from the 1800's, I can't believe how smooth and white her skin is. Also, there is an excellent degrading of the film stock which chills the mind if you like faded greys and yellows as I do. This film is played on TNT from time to time so see it.\",\n \"Jules Verne wrote about 80 novels as well as plays and short stories in his career. He began writing in 1854 with a short story called \\\"Master Zacharias, or the Clockmaker's Soul\\\". It was the first time he talked of the negative side of progress - the evil that results from some discoveries or inventions when they fall into the wrong hands. This becomes a running theme in his novels: Captain Nemo in TWENTY THOUSAND LEAGUES UNDER THE SEA, or Robur (from ROBUR THE CONQUEROR and it's sequel, THE MASTER OF THE WORLD) are two of his best examples of this them. Kongre, in THE LIGHTHOUSE AT THE EDGE OF THE WORLD, is another.Verne was so prolific that when he died in 1905 he left a dozen unpublished novels and stories that were not fully published until 1910. They include some of his best writing, such as THE BARSAC MISSION (partly written by Verne's son Michael), THE SURVIVORS OF THE \\\"JONATHAN\\\", THE PURSUIT OF THE METEOR, THE DANUBE PILOT. All of these dealt with science, but also dealt with political systems, and economics, for Verne was interested in all the problems facing modern man. THE LIGHTHOUSE AT THE EDGE OF THE WORLD was the last novel that was published in Verne's lifetime. It does not deal with the political questions or economic ones that perplexed him, but seems to go back to his potboiler period, when he was turning out stories for money while considering better stories for later publication. But nothing Verne wrote is without interest. Rereading THE LIGHTHOUSE one sees what the subtle point is in it. It is the study of how the ego of a villain can prevent him from escaping retribution.Kongre (renamed Jonathan Kongre) is one of the last pirates in the world of 1900. He and his gang find a damaged boat and repair it. They sail it across the Pacific, and reach Staten Island, a small island in the Straits of Magellan controlled by Chile. There they find a lighthouse with a crew of three men. They manage to kill two of them, but the third one (named Vasquez - he's from Chile, remember), hides on the island. Kongre and his men decide that they should prepare to leave the island shortly, before the Chilean Naval relief boat returns in three months to pick up the lighthouse crew. But first they will wreck any boat that comes to the passage, and increase their ill-gotten gains. But the key to the novel (and it is not in the movie) is that Kongre's right hand men (Carcante and Vargas) keep urging him to pack up his supplies and wealth and head to Asia where the money can be divvied up and everyone separate in safety. And each time Kongre won't do it. Initially it is pure greed. He wrecks a boat, and massacres the crew (a scene that is done in the film). The sole survivor is an American, John Davis (the name became Denton in the film, except that it was given to the character of Vasquez). Now with an ally (and not a drunken one, as in the film), Vasquez starts sabotaging Kongre's activities on the island. Carcante keeps suggesting leaving, but Kongre (unused to someone annoying him successfully) keeps delaying in order to catch Vasquez and Davis. The end result is that when he thinks he has them cornered, the Chilean boat appears to sink his craft, kill most of his crew, and confront him. Kongre commits suicide to avoid capture.Much of the mayhem of the movie (with Denton picking off crew members one at a time) is not in the book. Nor is there any female character in the novel (a rarity in most of Verne's stories - he could be quite a feminist when he wished). The egotism of \\\"Jonathan\\\" Kongre is well shown by Yul Brynner's performance, but the subtlety of that trait is lost. The writers presumably did not think the audience could appreciate it. Kirk Douglas does well enough as Denton, but his singlehanded success (Vasquez and Davis work together well to the end of the story, unlike Denton's ally who is killed by the pirates) seems unlikely. The bestiality of the pirates is well shown in the movie, particularly a singularly tall actor who in one scene wears women's clothing to particularly unsettling effect. The film is not a bad minor adventure film, but it could have been better if they had stuck to Verne's theme.\",\n \"Surprising that Jules Verne would write such a story. Even more surprising that Hollywood would produce it. Yul Brynner is unbelievably good as a man freed of all bounds of convention, free to indulge his taste for cruelty and domination. Douglass is an excellent counterpoint, a courageous individual who's chosen simple solitude as a way to deal with the complications and turmoil society had imposed on him. And Samantha Egger's character is the capper, a woman willing to sacrifice far too much for comfort and safety. Inotherwords, everyman. The movie does show its age and is limited by the conventions of time and place and technology of the time. If you're looking for special effects and action as substitutes for thought, look elsewhere.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"keywords\",\n \"properties\": {\n \"dtype\": \"object\",\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}",
|
|
"type": "dataframe"
|
|
},
|
|
"text/html": [
|
|
"\n",
|
|
" <div id=\"df-7be0279d-c607-428c-b86c-2b1c77104c3f\" class=\"colab-df-container\">\n",
|
|
" <div>\n",
|
|
"<style scoped>\n",
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|
" vertical-align: middle;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe tbody tr th {\n",
|
|
" vertical-align: top;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe thead th {\n",
|
|
" text-align: right;\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: right;\">\n",
|
|
" <th></th>\n",
|
|
" <th>review</th>\n",
|
|
" <th>keywords</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <th>0</th>\n",
|
|
" <td>The Light At The Edge of the World marks Kirk ...</td>\n",
|
|
" <td>[Kirk Douglas, Jules Verne, The Light At The E...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>1</th>\n",
|
|
" <td>Remove yourself from the Kirk Douglass aspects...</td>\n",
|
|
" <td>[Kirk Douglass aspects, casting, film, beautif...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>2</th>\n",
|
|
" <td>Surprising that Jules Verne would write such a...</td>\n",
|
|
" <td>[Jules Verne, Hollywood, Yul Brynner, cruelty ...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>3</th>\n",
|
|
" <td>it is a quite slow pace face-to-face Brynner/D...</td>\n",
|
|
" <td>[slow pace, face-to-face, Brynner/Douglas, hea...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>4</th>\n",
|
|
" <td>Jules Verne wrote about 80 novels as well as p...</td>\n",
|
|
" <td>[Jules Verne, novels, plays, short stories, Ma...</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>\n",
|
|
"</div>\n",
|
|
" <div class=\"colab-df-buttons\">\n",
|
|
"\n",
|
|
" <div class=\"colab-df-container\">\n",
|
|
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7be0279d-c607-428c-b86c-2b1c77104c3f')\"\n",
|
|
" title=\"Convert this dataframe to an interactive table.\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
|
|
" <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
|
|
" </svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
" <style>\n",
|
|
" .colab-df-container {\n",
|
|
" display:flex;\n",
|
|
" gap: 12px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert {\n",
|
|
" background-color: #E8F0FE;\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: #1967D2;\n",
|
|
" height: 32px;\n",
|
|
" padding: 0 0 0 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert:hover {\n",
|
|
" background-color: #E2EBFA;\n",
|
|
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: #174EA6;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-buttons div {\n",
|
|
" margin-bottom: 4px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert {\n",
|
|
" background-color: #3B4455;\n",
|
|
" fill: #D2E3FC;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert:hover {\n",
|
|
" background-color: #434B5C;\n",
|
|
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
|
|
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
|
|
" fill: #FFFFFF;\n",
|
|
" }\n",
|
|
" </style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" const buttonEl =\n",
|
|
" document.querySelector('#df-7be0279d-c607-428c-b86c-2b1c77104c3f button.colab-df-convert');\n",
|
|
" buttonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
"\n",
|
|
" async function convertToInteractive(key) {\n",
|
|
" const element = document.querySelector('#df-7be0279d-c607-428c-b86c-2b1c77104c3f');\n",
|
|
" const dataTable =\n",
|
|
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
|
|
" [key], {});\n",
|
|
" if (!dataTable) return;\n",
|
|
"\n",
|
|
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
|
|
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
|
|
" + ' to learn more about interactive tables.';\n",
|
|
" element.innerHTML = '';\n",
|
|
" dataTable['output_type'] = 'display_data';\n",
|
|
" await google.colab.output.renderOutput(dataTable, element);\n",
|
|
" const docLink = document.createElement('div');\n",
|
|
" docLink.innerHTML = docLinkHtml;\n",
|
|
" element.appendChild(docLink);\n",
|
|
" }\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
"\n",
|
|
" <div id=\"df-406ee44d-0a3e-4222-ab2b-2a051e63bd23\">\n",
|
|
" <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-406ee44d-0a3e-4222-ab2b-2a051e63bd23')\"\n",
|
|
" title=\"Suggest charts\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
"<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
|
|
" width=\"24px\">\n",
|
|
" <g>\n",
|
|
" <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
|
|
" </g>\n",
|
|
"</svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
"<style>\n",
|
|
" .colab-df-quickchart {\n",
|
|
" --bg-color: #E8F0FE;\n",
|
|
" --fill-color: #1967D2;\n",
|
|
" --hover-bg-color: #E2EBFA;\n",
|
|
" --hover-fill-color: #174EA6;\n",
|
|
" --disabled-fill-color: #AAA;\n",
|
|
" --disabled-bg-color: #DDD;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-quickchart {\n",
|
|
" --bg-color: #3B4455;\n",
|
|
" --fill-color: #D2E3FC;\n",
|
|
" --hover-bg-color: #434B5C;\n",
|
|
" --hover-fill-color: #FFFFFF;\n",
|
|
" --disabled-bg-color: #3B4455;\n",
|
|
" --disabled-fill-color: #666;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart {\n",
|
|
" background-color: var(--bg-color);\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: var(--fill-color);\n",
|
|
" height: 32px;\n",
|
|
" padding: 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart:hover {\n",
|
|
" background-color: var(--hover-bg-color);\n",
|
|
" box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: var(--button-hover-fill-color);\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart-complete:disabled,\n",
|
|
" .colab-df-quickchart-complete:disabled:hover {\n",
|
|
" background-color: var(--disabled-bg-color);\n",
|
|
" fill: var(--disabled-fill-color);\n",
|
|
" box-shadow: none;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-spinner {\n",
|
|
" border: 2px solid var(--fill-color);\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" animation:\n",
|
|
" spin 1s steps(1) infinite;\n",
|
|
" }\n",
|
|
"\n",
|
|
" @keyframes spin {\n",
|
|
" 0% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 20% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 30% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 40% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 60% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 80% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 90% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" async function quickchart(key) {\n",
|
|
" const quickchartButtonEl =\n",
|
|
" document.querySelector('#' + key + ' button');\n",
|
|
" quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n",
|
|
" quickchartButtonEl.classList.add('colab-df-spinner');\n",
|
|
" try {\n",
|
|
" const charts = await google.colab.kernel.invokeFunction(\n",
|
|
" 'suggestCharts', [key], {});\n",
|
|
" } catch (error) {\n",
|
|
" console.error('Error during call to suggestCharts:', error);\n",
|
|
" }\n",
|
|
" quickchartButtonEl.classList.remove('colab-df-spinner');\n",
|
|
" quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
|
|
" }\n",
|
|
" (() => {\n",
|
|
" let quickchartButtonEl =\n",
|
|
" document.querySelector('#df-406ee44d-0a3e-4222-ab2b-2a051e63bd23 button');\n",
|
|
" quickchartButtonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
" })();\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
" </div>\n",
|
|
" </div>\n"
|
|
],
|
|
"text/plain": [
|
|
" review \\\n",
|
|
"0 The Light At The Edge of the World marks Kirk ... \n",
|
|
"1 Remove yourself from the Kirk Douglass aspects... \n",
|
|
"2 Surprising that Jules Verne would write such a... \n",
|
|
"3 it is a quite slow pace face-to-face Brynner/D... \n",
|
|
"4 Jules Verne wrote about 80 novels as well as p... \n",
|
|
"\n",
|
|
" keywords \n",
|
|
"0 [Kirk Douglas, Jules Verne, The Light At The E... \n",
|
|
"1 [Kirk Douglass aspects, casting, film, beautif... \n",
|
|
"2 [Jules Verne, Hollywood, Yul Brynner, cruelty ... \n",
|
|
"3 [slow pace, face-to-face, Brynner/Douglas, hea... \n",
|
|
"4 [Jules Verne, novels, plays, short stories, Ma... "
|
|
]
|
|
},
|
|
"execution_count": 47,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"%%bigquery --project {PROJECT_ID}\n",
|
|
"\n",
|
|
"SELECT\n",
|
|
" review,\n",
|
|
" AI.GENERATE(\n",
|
|
" ('Extract the keywords from the text below: ', review),\n",
|
|
" connection_id => 'us.test_connection',\n",
|
|
" endpoint => 'gemini-2.5-flash',\n",
|
|
" output_schema => 'keywords ARRAY< STRING>').keywords\n",
|
|
"FROM\n",
|
|
" `bigquery-public-data.imdb.reviews`\n",
|
|
"WHERE movie_id = \"tt0067345\"\n",
|
|
"LIMIT 5\n",
|
|
";"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "AoF8XxEa1HsW"
|
|
},
|
|
"source": [
|
|
"Now the responses are arrays of keywords - a more useful format for this use case!\n",
|
|
"\n",
|
|
"Next, you'll adjust the prompt to the model to extract the sentiment as well, and adjust the `output_schema` accordingly."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 48,
|
|
"metadata": {
|
|
"id": "f1wfmGVZkA1x"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Query is running: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Downloading: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.google.colaboratory.intrinsic+json": {
|
|
"summary": "{\n \"name\": \"get_ipython()\",\n \"rows\": 5,\n \"fields\": [\n {\n \"column\": \"review\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Remove yourself from the Kirk Douglass aspects of the casting. It is essential to your enjoying the film. There is a beautiful young woman playing double roles - and in the photos from the 1800's, I can't believe how smooth and white her skin is. Also, there is an excellent degrading of the film stock which chills the mind if you like faded greys and yellows as I do. This film is played on TNT from time to time so see it.\",\n \"it is a quite slow pace face-to-face Brynner/Douglas...i found there was some pretty heavy violence/gore but you know, seventies giallo style : (maybe some spoiler ahead) a little bit phony and fake but the image is there : the monkey torn apart, the flesh torn from the mechanic, etc...i don't remember if it is alike the novel...but overall i think it is a OK action flick, hero flick : he stands all alone at the end, all the bad guys are out..........and nice images too : a small rocky island, the sun over the seabut not a typical Verne's story-on-cinema : subs, sci-fi; but the adventure seen in a lot of Verne's novels tough\",\n \"Surprising that Jules Verne would write such a story. Even more surprising that Hollywood would produce it. Yul Brynner is unbelievably good as a man freed of all bounds of convention, free to indulge his taste for cruelty and domination. Douglass is an excellent counterpoint, a courageous individual who's chosen simple solitude as a way to deal with the complications and turmoil society had imposed on him. And Samantha Egger's character is the capper, a woman willing to sacrifice far too much for comfort and safety. Inotherwords, everyman. The movie does show its age and is limited by the conventions of time and place and technology of the time. If you're looking for special effects and action as substitutes for thought, look elsewhere.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"full_response\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"{\\\"candidates\\\":[{\\\"avg_logprobs\\\":-1.3310155073801677,\\\"content\\\":{\\\"parts\\\":[{\\\"text\\\":\\\"{\\\\\\\"keywords\\\\\\\": [\\\\\\\"Kirk Douglass aspects\\\\\\\", \\\\\\\"casting\\\\\\\", \\\\\\\"beautiful young woman\\\\\\\", \\\\\\\"double roles\\\\\\\", \\\\\\\"smooth white skin\\\\\\\", \\\\\\\"degrading film stock\\\\\\\", \\\\\\\"faded greys and yellows\\\\\\\", \\\\\\\"TNT\\\\\\\"], \\\\\\\"sentiment\\\\\\\": \\\\\\\"Positive\\\\\\\"}\\\"}],\\\"role\\\":\\\"model\\\"},\\\"finish_reason\\\":\\\"STOP\\\",\\\"score\\\":-63.88874435424805}],\\\"create_time\\\":\\\"2026-01-06T22:58:03.610034Z\\\",\\\"model_version\\\":\\\"gemini-2.5-flash\\\",\\\"response_id\\\":\\\"e5NdafKdJb2mlu8Po5XtwQo\\\",\\\"usage_metadata\\\":{\\\"billable_prompt_usage\\\":{\\\"text_count\\\":447},\\\"candidates_token_count\\\":48,\\\"candidates_tokens_details\\\":[{\\\"modality\\\":\\\"TEXT\\\",\\\"token_count\\\":48}],\\\"prompt_token_count\\\":123,\\\"prompt_tokens_details\\\":[{\\\"modality\\\":\\\"TEXT\\\",\\\"token_count\\\":123}],\\\"thoughts_token_count\\\":279,\\\"total_token_count\\\":450,\\\"traffic_type\\\":\\\"ON_DEMAND\\\"}}\",\n \"{\\\"candidates\\\":[{\\\"avg_logprobs\\\":-1.9722009741741677,\\\"content\\\":{\\\"parts\\\":[{\\\"text\\\":\\\"{\\\\n \\\\\\\"keywords\\\\\\\": [\\\\n \\\\\\\"slow pace\\\\\\\",\\\\n \\\\\\\"heavy violence\\\\\\\",\\\\n \\\\\\\"gore\\\\\\\",\\\\n \\\\\\\"seventies giallo style\\\\\\\",\\\\n \\\\\\\"phony\\\\\\\",\\\\n \\\\\\\"fake\\\\\\\",\\\\n \\\\\\\"action flick\\\\\\\",\\\\n \\\\\\\"hero flick\\\\\\\",\\\\n \\\\\\\"rocky island\\\\\\\",\\\\n \\\\\\\"sea\\\\\\\",\\\\n \\\\\\\"Jules Verne\\\\\\\",\\\\n \\\\\\\"adventure\\\\\\\"\\\\n ],\\\\n \\\\\\\"sentiment\\\\\\\": \\\\\\\"Neutral\\\\\\\"\\\\n}\\\"}],\\\"role\\\":\\\"model\\\"},\\\"finish_reason\\\":\\\"STOP\\\",\\\"score\\\":-181.44248962402344}],\\\"create_time\\\":\\\"2026-01-06T22:58:02.628472Z\\\",\\\"model_version\\\":\\\"gemini-2.5-flash\\\",\\\"response_id\\\":\\\"epNdafitJrjFvPEP5p7G0Ao\\\",\\\"usage_metadata\\\":{\\\"billable_prompt_usage\\\":{\\\"text_count\\\":622},\\\"candidates_token_count\\\":92,\\\"candidates_tokens_details\\\":[{\\\"modality\\\":\\\"TEXT\\\",\\\"token_count\\\":92}],\\\"prompt_token_count\\\":179,\\\"prompt_tokens_details\\\":[{\\\"modality\\\":\\\"TEXT\\\",\\\"token_count\\\":179}],\\\"thoughts_token_count\\\":625,\\\"total_token_count\\\":896,\\\"traffic_type\\\":\\\"ON_DEMAND\\\"}}\",\n \"{\\\"candidates\\\":[{\\\"avg_logprobs\\\":-1.2379138734605577,\\\"content\\\":{\\\"parts\\\":[{\\\"text\\\":\\\"{\\\\\\\"keywords\\\\\\\": [\\\\\\\"Jules Verne\\\\\\\", \\\\\\\"Hollywood\\\\\\\", \\\\\\\"Yul Brynner\\\\\\\", \\\\\\\"cruelty and domination\\\\\\\", \\\\\\\"Douglass\\\\\\\", \\\\\\\"simple solitude\\\\\\\", \\\\\\\"Samantha Egger\\\\\\\", \\\\\\\"sacrifice\\\\\\\", \\\\\\\"everyman\\\\\\\", \\\\\\\"movie age\\\\\\\", \\\\\\\"conventions of time\\\\\\\", \\\\\\\"technology\\\\\\\", \\\\\\\"special effects\\\\\\\", \\\\\\\"action\\\\\\\", \\\\\\\"thought\\\\\\\"], \\\\\\\"sentiment\\\\\\\": \\\\\\\"Positive\\\\\\\"}\\\"}],\\\"role\\\":\\\"model\\\"},\\\"finish_reason\\\":\\\"STOP\\\",\\\"score\\\":-89.12979888916016}],\\\"create_time\\\":\\\"2026-01-06T22:58:02.624833Z\\\",\\\"model_version\\\":\\\"gemini-2.5-flash\\\",\\\"response_id\\\":\\\"epNdacGRJpHplu8P_-zfqQw\\\",\\\"usage_metadata\\\":{\\\"billable_prompt_usage\\\":{\\\"text_count\\\":728},\\\"candidates_token_count\\\":72,\\\"candidates_tokens_details\\\":[{\\\"modality\\\":\\\"TEXT\\\",\\\"token_count\\\":72}],\\\"prompt_token_count\\\":176,\\\"prompt_tokens_details\\\":[{\\\"modality\\\":\\\"TEXT\\\",\\\"token_count\\\":176}],\\\"thoughts_token_count\\\":288,\\\"total_token_count\\\":536,\\\"traffic_type\\\":\\\"ON_DEMAND\\\"}}\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}",
|
|
"type": "dataframe"
|
|
},
|
|
"text/html": [
|
|
"\n",
|
|
" <div id=\"df-017bd216-ac4d-420e-84df-7f1f490d3a45\" class=\"colab-df-container\">\n",
|
|
" <div>\n",
|
|
"<style scoped>\n",
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|
" vertical-align: middle;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe tbody tr th {\n",
|
|
" vertical-align: top;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe thead th {\n",
|
|
" text-align: right;\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: right;\">\n",
|
|
" <th></th>\n",
|
|
" <th>review</th>\n",
|
|
" <th>full_response</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <th>0</th>\n",
|
|
" <td>The Light At The Edge of the World marks Kirk ...</td>\n",
|
|
" <td>{\"candidates\":[{\"avg_logprobs\":-0.587079347069...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>1</th>\n",
|
|
" <td>Remove yourself from the Kirk Douglass aspects...</td>\n",
|
|
" <td>{\"candidates\":[{\"avg_logprobs\":-1.331015507380...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>2</th>\n",
|
|
" <td>Surprising that Jules Verne would write such a...</td>\n",
|
|
" <td>{\"candidates\":[{\"avg_logprobs\":-1.237913873460...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>3</th>\n",
|
|
" <td>Jules Verne wrote about 80 novels as well as p...</td>\n",
|
|
" <td>{\"candidates\":[{\"avg_logprobs\":-0.702408254963...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>4</th>\n",
|
|
" <td>it is a quite slow pace face-to-face Brynner/D...</td>\n",
|
|
" <td>{\"candidates\":[{\"avg_logprobs\":-1.972200974174...</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>\n",
|
|
"</div>\n",
|
|
" <div class=\"colab-df-buttons\">\n",
|
|
"\n",
|
|
" <div class=\"colab-df-container\">\n",
|
|
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-017bd216-ac4d-420e-84df-7f1f490d3a45')\"\n",
|
|
" title=\"Convert this dataframe to an interactive table.\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
|
|
" <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
|
|
" </svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
" <style>\n",
|
|
" .colab-df-container {\n",
|
|
" display:flex;\n",
|
|
" gap: 12px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert {\n",
|
|
" background-color: #E8F0FE;\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: #1967D2;\n",
|
|
" height: 32px;\n",
|
|
" padding: 0 0 0 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert:hover {\n",
|
|
" background-color: #E2EBFA;\n",
|
|
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: #174EA6;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-buttons div {\n",
|
|
" margin-bottom: 4px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert {\n",
|
|
" background-color: #3B4455;\n",
|
|
" fill: #D2E3FC;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert:hover {\n",
|
|
" background-color: #434B5C;\n",
|
|
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
|
|
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
|
|
" fill: #FFFFFF;\n",
|
|
" }\n",
|
|
" </style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" const buttonEl =\n",
|
|
" document.querySelector('#df-017bd216-ac4d-420e-84df-7f1f490d3a45 button.colab-df-convert');\n",
|
|
" buttonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
"\n",
|
|
" async function convertToInteractive(key) {\n",
|
|
" const element = document.querySelector('#df-017bd216-ac4d-420e-84df-7f1f490d3a45');\n",
|
|
" const dataTable =\n",
|
|
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
|
|
" [key], {});\n",
|
|
" if (!dataTable) return;\n",
|
|
"\n",
|
|
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
|
|
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
|
|
" + ' to learn more about interactive tables.';\n",
|
|
" element.innerHTML = '';\n",
|
|
" dataTable['output_type'] = 'display_data';\n",
|
|
" await google.colab.output.renderOutput(dataTable, element);\n",
|
|
" const docLink = document.createElement('div');\n",
|
|
" docLink.innerHTML = docLinkHtml;\n",
|
|
" element.appendChild(docLink);\n",
|
|
" }\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
"\n",
|
|
" <div id=\"df-58d1daeb-4827-4693-9f39-5bf03a737fce\">\n",
|
|
" <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-58d1daeb-4827-4693-9f39-5bf03a737fce')\"\n",
|
|
" title=\"Suggest charts\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
"<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
|
|
" width=\"24px\">\n",
|
|
" <g>\n",
|
|
" <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
|
|
" </g>\n",
|
|
"</svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
"<style>\n",
|
|
" .colab-df-quickchart {\n",
|
|
" --bg-color: #E8F0FE;\n",
|
|
" --fill-color: #1967D2;\n",
|
|
" --hover-bg-color: #E2EBFA;\n",
|
|
" --hover-fill-color: #174EA6;\n",
|
|
" --disabled-fill-color: #AAA;\n",
|
|
" --disabled-bg-color: #DDD;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-quickchart {\n",
|
|
" --bg-color: #3B4455;\n",
|
|
" --fill-color: #D2E3FC;\n",
|
|
" --hover-bg-color: #434B5C;\n",
|
|
" --hover-fill-color: #FFFFFF;\n",
|
|
" --disabled-bg-color: #3B4455;\n",
|
|
" --disabled-fill-color: #666;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart {\n",
|
|
" background-color: var(--bg-color);\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: var(--fill-color);\n",
|
|
" height: 32px;\n",
|
|
" padding: 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart:hover {\n",
|
|
" background-color: var(--hover-bg-color);\n",
|
|
" box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: var(--button-hover-fill-color);\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart-complete:disabled,\n",
|
|
" .colab-df-quickchart-complete:disabled:hover {\n",
|
|
" background-color: var(--disabled-bg-color);\n",
|
|
" fill: var(--disabled-fill-color);\n",
|
|
" box-shadow: none;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-spinner {\n",
|
|
" border: 2px solid var(--fill-color);\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" animation:\n",
|
|
" spin 1s steps(1) infinite;\n",
|
|
" }\n",
|
|
"\n",
|
|
" @keyframes spin {\n",
|
|
" 0% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 20% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 30% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 40% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 60% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 80% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 90% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" async function quickchart(key) {\n",
|
|
" const quickchartButtonEl =\n",
|
|
" document.querySelector('#' + key + ' button');\n",
|
|
" quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n",
|
|
" quickchartButtonEl.classList.add('colab-df-spinner');\n",
|
|
" try {\n",
|
|
" const charts = await google.colab.kernel.invokeFunction(\n",
|
|
" 'suggestCharts', [key], {});\n",
|
|
" } catch (error) {\n",
|
|
" console.error('Error during call to suggestCharts:', error);\n",
|
|
" }\n",
|
|
" quickchartButtonEl.classList.remove('colab-df-spinner');\n",
|
|
" quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
|
|
" }\n",
|
|
" (() => {\n",
|
|
" let quickchartButtonEl =\n",
|
|
" document.querySelector('#df-58d1daeb-4827-4693-9f39-5bf03a737fce button');\n",
|
|
" quickchartButtonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
" })();\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
" </div>\n",
|
|
" </div>\n"
|
|
],
|
|
"text/plain": [
|
|
" review \\\n",
|
|
"0 The Light At The Edge of the World marks Kirk ... \n",
|
|
"1 Remove yourself from the Kirk Douglass aspects... \n",
|
|
"2 Surprising that Jules Verne would write such a... \n",
|
|
"3 Jules Verne wrote about 80 novels as well as p... \n",
|
|
"4 it is a quite slow pace face-to-face Brynner/D... \n",
|
|
"\n",
|
|
" full_response \n",
|
|
"0 {\"candidates\":[{\"avg_logprobs\":-0.587079347069... \n",
|
|
"1 {\"candidates\":[{\"avg_logprobs\":-1.331015507380... \n",
|
|
"2 {\"candidates\":[{\"avg_logprobs\":-1.237913873460... \n",
|
|
"3 {\"candidates\":[{\"avg_logprobs\":-0.702408254963... \n",
|
|
"4 {\"candidates\":[{\"avg_logprobs\":-1.972200974174... "
|
|
]
|
|
},
|
|
"execution_count": 48,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"%%bigquery --project {PROJECT_ID}\n",
|
|
"\n",
|
|
"SELECT\n",
|
|
" review,\n",
|
|
" AI.GENERATE(\n",
|
|
" ('Extract the keywords and sentiment (Positive, Negative, or Neutral) from the following review: ', review),\n",
|
|
" connection_id => 'us.test_connection',\n",
|
|
" endpoint => 'gemini-2.5-flash',\n",
|
|
" output_schema => 'keywords ARRAY< STRING>, sentiment STRING').full_response\n",
|
|
"FROM\n",
|
|
" `bigquery-public-data.imdb.reviews`\n",
|
|
"WHERE movie_id = \"tt0067345\"\n",
|
|
"LIMIT 5\n",
|
|
";"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "KmpYd5Dg2AJy"
|
|
},
|
|
"source": [
|
|
"This time, the response is in a JSON format. Why?\n",
|
|
"\n",
|
|
"When using `output_schema` with [`AI.GENERATE`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate), you can specify in the SQL which field you want to return. In this case the SQL specifies the `.full_response` which will provide a JSON object containing the various text response elements. You may alternatively specify to return any one of the schema columns you defined, such as `.keywords` (as was used in the first query of this section) or `.sentiment`. However, you cannot specify to return multiple columns.\n",
|
|
"\n",
|
|
" You could use SQL to parse the JSON provided in the `.full_response`, but for ease of use, let's look at the [`AI.GENERATE_TABLE`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-generate-table) function."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "generate_table_md"
|
|
},
|
|
"source": [
|
|
"### Using `AI.GENERATE_TABLE`: Extract keywords and sentiment from movie reviews\n",
|
|
"\n",
|
|
"The [`AI.GENERATE_TABLE`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-generate-table) function provides expanded functionality for turning unstructured text into a structured table. Here, you'll perform the same analysis to extract keywords and overall sentiment from the `bigquery-public-data.imdb_reviews` table and return the response as multiple columns in one query."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "prepare_md"
|
|
},
|
|
"source": [
|
|
"#### Create a BigQuery Dataset\n",
|
|
"\n",
|
|
"The syntax for using [`AI.GENERATE_TABLE`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-generate-table) differs from [`AI.GENERATE`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate), and it requires that you first create a remote model in BigQuery that connects to the one of the [generally available](https://cloud.google.com/vertex-ai/generative-ai/docs/models#generally_available_models) or [preview](https://cloud.google.com/vertex-ai/generative-ai/docs/models#preview_models) Gemini models.\n",
|
|
"\n",
|
|
"Running the following query will create a BigQuery dataset called **`bq_ai_tutorial`** to house the remote model:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "create_dataset_code"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%%bigquery --project {PROJECT_ID}\n",
|
|
"\n",
|
|
"CREATE SCHEMA\n",
|
|
" `bq_ai_tutorial` OPTIONS (location = 'US');"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "create_model_md"
|
|
},
|
|
"source": [
|
|
"#### Create a remote model for Gemini 2.5 Flash\n",
|
|
"\n",
|
|
"Now you'll create the remote model, which is simply a pointer to the model endpoint. In this example, you'll continue to use the `gemini-2.5-flash` model endpoint.\n",
|
|
"\n",
|
|
"Once you create this remote model, it is saved in your `bq_ai_tutorial` dataset so that you can use it in any future analysis."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "create_model_code"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%%bigquery --project {PROJECT_ID}\n",
|
|
"\n",
|
|
"CREATE OR REPLACE MODEL\n",
|
|
" `bq_ai_tutorial.gemini_2_5_flash`\n",
|
|
"REMOTE WITH\n",
|
|
" CONNECTION `us.test_connection`\n",
|
|
" OPTIONS (ENDPOINT = 'gemini-2.5-flash');\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "SBiwwjzhendG"
|
|
},
|
|
"source": [
|
|
"#### Run analysis using the remote model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "CmVE4UxdnoqB"
|
|
},
|
|
"source": [
|
|
"Now that the remote model has been created, you can prompt it with [`AI.GENERATE_TABLE`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-generate-table) to analyze data and specify the output schema with multiple columns returned."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 50,
|
|
"metadata": {
|
|
"id": "generate_table_code"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Query is running: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Downloading: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.google.colaboratory.intrinsic+json": {
|
|
"summary": "{\n \"name\": \"get_ipython()\",\n \"rows\": 5,\n \"fields\": [\n {\n \"column\": \"review\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Surprising that Jules Verne would write such a story. Even more surprising that Hollywood would produce it. Yul Brynner is unbelievably good as a man freed of all bounds of convention, free to indulge his taste for cruelty and domination. Douglass is an excellent counterpoint, a courageous individual who's chosen simple solitude as a way to deal with the complications and turmoil society had imposed on him. And Samantha Egger's character is the capper, a woman willing to sacrifice far too much for comfort and safety. Inotherwords, everyman. The movie does show its age and is limited by the conventions of time and place and technology of the time. If you're looking for special effects and action as substitutes for thought, look elsewhere.\",\n \"Jules Verne wrote about 80 novels as well as plays and short stories in his career. He began writing in 1854 with a short story called \\\"Master Zacharias, or the Clockmaker's Soul\\\". It was the first time he talked of the negative side of progress - the evil that results from some discoveries or inventions when they fall into the wrong hands. This becomes a running theme in his novels: Captain Nemo in TWENTY THOUSAND LEAGUES UNDER THE SEA, or Robur (from ROBUR THE CONQUEROR and it's sequel, THE MASTER OF THE WORLD) are two of his best examples of this them. Kongre, in THE LIGHTHOUSE AT THE EDGE OF THE WORLD, is another.Verne was so prolific that when he died in 1905 he left a dozen unpublished novels and stories that were not fully published until 1910. They include some of his best writing, such as THE BARSAC MISSION (partly written by Verne's son Michael), THE SURVIVORS OF THE \\\"JONATHAN\\\", THE PURSUIT OF THE METEOR, THE DANUBE PILOT. All of these dealt with science, but also dealt with political systems, and economics, for Verne was interested in all the problems facing modern man. THE LIGHTHOUSE AT THE EDGE OF THE WORLD was the last novel that was published in Verne's lifetime. It does not deal with the political questions or economic ones that perplexed him, but seems to go back to his potboiler period, when he was turning out stories for money while considering better stories for later publication. But nothing Verne wrote is without interest. Rereading THE LIGHTHOUSE one sees what the subtle point is in it. It is the study of how the ego of a villain can prevent him from escaping retribution.Kongre (renamed Jonathan Kongre) is one of the last pirates in the world of 1900. He and his gang find a damaged boat and repair it. They sail it across the Pacific, and reach Staten Island, a small island in the Straits of Magellan controlled by Chile. There they find a lighthouse with a crew of three men. They manage to kill two of them, but the third one (named Vasquez - he's from Chile, remember), hides on the island. Kongre and his men decide that they should prepare to leave the island shortly, before the Chilean Naval relief boat returns in three months to pick up the lighthouse crew. But first they will wreck any boat that comes to the passage, and increase their ill-gotten gains. But the key to the novel (and it is not in the movie) is that Kongre's right hand men (Carcante and Vargas) keep urging him to pack up his supplies and wealth and head to Asia where the money can be divvied up and everyone separate in safety. And each time Kongre won't do it. Initially it is pure greed. He wrecks a boat, and massacres the crew (a scene that is done in the film). The sole survivor is an American, John Davis (the name became Denton in the film, except that it was given to the character of Vasquez). Now with an ally (and not a drunken one, as in the film), Vasquez starts sabotaging Kongre's activities on the island. Carcante keeps suggesting leaving, but Kongre (unused to someone annoying him successfully) keeps delaying in order to catch Vasquez and Davis. The end result is that when he thinks he has them cornered, the Chilean boat appears to sink his craft, kill most of his crew, and confront him. Kongre commits suicide to avoid capture.Much of the mayhem of the movie (with Denton picking off crew members one at a time) is not in the book. Nor is there any female character in the novel (a rarity in most of Verne's stories - he could be quite a feminist when he wished). The egotism of \\\"Jonathan\\\" Kongre is well shown by Yul Brynner's performance, but the subtlety of that trait is lost. The writers presumably did not think the audience could appreciate it. Kirk Douglas does well enough as Denton, but his singlehanded success (Vasquez and Davis work together well to the end of the story, unlike Denton's ally who is killed by the pirates) seems unlikely. The bestiality of the pirates is well shown in the movie, particularly a singularly tall actor who in one scene wears women's clothing to particularly unsettling effect. The film is not a bad minor adventure film, but it could have been better if they had stuck to Verne's theme.\",\n \"The Light At The Edge of the World marks Kirk Douglas's second filming of a Jules Verne novel. The first of course was one of his most popular films 20,000 Leagues Under The Sea. But this film is far more serious and has far more adult themes than Walt Disney's film aimed for the kid trade.This was the last novel Jules Verne had published during his lifetime and it's a story of survival against almost impossible odds. In the original novel Kirk Douglas's character was named Vasquez which certainly was more in keeping with someone assigned to lighthouse duty on Cape Horn. But in giving Douglas's character an Anglo name it better explains his presence on the island and it certainly is in keeping with the international tradition of Jules Verne's writings.Cape Horn is one of the loneliest parts of the globe and the geography of the southern tip of South America. Look on a map of the many islands and rocks in that part of the globe and imagine how rough the sea is because it has only limited space. It's not without reason that sailors in all cultures say that no one is really a true sailor until they've made a voyage crossing from the Atlantic to the Pacific Ocean in that area. Remember also this is 1865 as Yul Brynner identifies the year and the Panama Canal had not been built.Which makes the lighthouse at Cape Horn an international concern which was something that is ever present in Jules Verne's writings. But then as now there are malevolent forces in the world and they are in this story Yul Brynner and his pirate crew.On one desultory like any other down there, Yul Brynner's ship docks at the island and kills lighthouse keeper Fernando Rey and his young assistant Massimo Ranieri. By sheer dumb luck Douglas is not at the lighthouse when this happens, but he becomes a hunted man by Brynner and his pirate crew who want to set up headquarters there and use the light to pile up as many wrecks as they can plunder. Also they want to eliminate Douglas who's now the only witness to their crimes.I did like this film very much both when first seeing it in the theater and now on VHS. One thing of interest I found here is that there is no ambiguity, no shadings of character. Kirk Douglas is a good guy and Yul Brynner a bad one, no one is going to walk away thinking anything else. In fact Yul Brynner's pirate captain Jonathan Kongre is the most unredeemable villain we've seen on screen since Lee Marvin as Liberty Valance.Definitely fans of Kirk Douglas, Yul Brynner and Jules Verne should earmark this film for their collection.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"keywords\",\n \"properties\": {\n \"dtype\": \"object\",\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"sentiment\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 3,\n \"samples\": [\n \"Positive\",\n \"Neutral\",\n \"Negative\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}",
|
|
"type": "dataframe"
|
|
},
|
|
"text/html": [
|
|
"\n",
|
|
" <div id=\"df-b18c1642-6121-41fc-ae2d-4461bd3993d1\" class=\"colab-df-container\">\n",
|
|
" <div>\n",
|
|
"<style scoped>\n",
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|
" vertical-align: middle;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe tbody tr th {\n",
|
|
" vertical-align: top;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe thead th {\n",
|
|
" text-align: right;\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: right;\">\n",
|
|
" <th></th>\n",
|
|
" <th>review</th>\n",
|
|
" <th>keywords</th>\n",
|
|
" <th>sentiment</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <th>0</th>\n",
|
|
" <td>Remove yourself from the Kirk Douglass aspects...</td>\n",
|
|
" <td>[Kirk Douglass casting, beautiful young woman,...</td>\n",
|
|
" <td>Positive</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>1</th>\n",
|
|
" <td>Surprising that Jules Verne would write such a...</td>\n",
|
|
" <td>[Jules Verne, Hollywood, Yul Brynner, cruelty,...</td>\n",
|
|
" <td>Positive</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>2</th>\n",
|
|
" <td>The Light At The Edge of the World marks Kirk ...</td>\n",
|
|
" <td>[The Light At The Edge of the World, Kirk Doug...</td>\n",
|
|
" <td>Positive</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>3</th>\n",
|
|
" <td>it is a quite slow pace face-to-face Brynner/D...</td>\n",
|
|
" <td>[slow pace, Brynner/Douglas, violence, gore, s...</td>\n",
|
|
" <td>Neutral</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>4</th>\n",
|
|
" <td>Jules Verne wrote about 80 novels as well as p...</td>\n",
|
|
" <td>[Jules Verne, The Lighthouse at the Edge of th...</td>\n",
|
|
" <td>Negative</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>\n",
|
|
"</div>\n",
|
|
" <div class=\"colab-df-buttons\">\n",
|
|
"\n",
|
|
" <div class=\"colab-df-container\">\n",
|
|
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b18c1642-6121-41fc-ae2d-4461bd3993d1')\"\n",
|
|
" title=\"Convert this dataframe to an interactive table.\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
|
|
" <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
|
|
" </svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
" <style>\n",
|
|
" .colab-df-container {\n",
|
|
" display:flex;\n",
|
|
" gap: 12px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert {\n",
|
|
" background-color: #E8F0FE;\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: #1967D2;\n",
|
|
" height: 32px;\n",
|
|
" padding: 0 0 0 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert:hover {\n",
|
|
" background-color: #E2EBFA;\n",
|
|
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: #174EA6;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-buttons div {\n",
|
|
" margin-bottom: 4px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert {\n",
|
|
" background-color: #3B4455;\n",
|
|
" fill: #D2E3FC;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert:hover {\n",
|
|
" background-color: #434B5C;\n",
|
|
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
|
|
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
|
|
" fill: #FFFFFF;\n",
|
|
" }\n",
|
|
" </style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" const buttonEl =\n",
|
|
" document.querySelector('#df-b18c1642-6121-41fc-ae2d-4461bd3993d1 button.colab-df-convert');\n",
|
|
" buttonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
"\n",
|
|
" async function convertToInteractive(key) {\n",
|
|
" const element = document.querySelector('#df-b18c1642-6121-41fc-ae2d-4461bd3993d1');\n",
|
|
" const dataTable =\n",
|
|
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
|
|
" [key], {});\n",
|
|
" if (!dataTable) return;\n",
|
|
"\n",
|
|
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
|
|
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
|
|
" + ' to learn more about interactive tables.';\n",
|
|
" element.innerHTML = '';\n",
|
|
" dataTable['output_type'] = 'display_data';\n",
|
|
" await google.colab.output.renderOutput(dataTable, element);\n",
|
|
" const docLink = document.createElement('div');\n",
|
|
" docLink.innerHTML = docLinkHtml;\n",
|
|
" element.appendChild(docLink);\n",
|
|
" }\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
"\n",
|
|
" <div id=\"df-f9fd59c8-c209-4e9c-9694-c2f81a8901d8\">\n",
|
|
" <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-f9fd59c8-c209-4e9c-9694-c2f81a8901d8')\"\n",
|
|
" title=\"Suggest charts\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
"<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
|
|
" width=\"24px\">\n",
|
|
" <g>\n",
|
|
" <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
|
|
" </g>\n",
|
|
"</svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
"<style>\n",
|
|
" .colab-df-quickchart {\n",
|
|
" --bg-color: #E8F0FE;\n",
|
|
" --fill-color: #1967D2;\n",
|
|
" --hover-bg-color: #E2EBFA;\n",
|
|
" --hover-fill-color: #174EA6;\n",
|
|
" --disabled-fill-color: #AAA;\n",
|
|
" --disabled-bg-color: #DDD;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-quickchart {\n",
|
|
" --bg-color: #3B4455;\n",
|
|
" --fill-color: #D2E3FC;\n",
|
|
" --hover-bg-color: #434B5C;\n",
|
|
" --hover-fill-color: #FFFFFF;\n",
|
|
" --disabled-bg-color: #3B4455;\n",
|
|
" --disabled-fill-color: #666;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart {\n",
|
|
" background-color: var(--bg-color);\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: var(--fill-color);\n",
|
|
" height: 32px;\n",
|
|
" padding: 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart:hover {\n",
|
|
" background-color: var(--hover-bg-color);\n",
|
|
" box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: var(--button-hover-fill-color);\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart-complete:disabled,\n",
|
|
" .colab-df-quickchart-complete:disabled:hover {\n",
|
|
" background-color: var(--disabled-bg-color);\n",
|
|
" fill: var(--disabled-fill-color);\n",
|
|
" box-shadow: none;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-spinner {\n",
|
|
" border: 2px solid var(--fill-color);\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" animation:\n",
|
|
" spin 1s steps(1) infinite;\n",
|
|
" }\n",
|
|
"\n",
|
|
" @keyframes spin {\n",
|
|
" 0% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 20% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 30% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 40% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 60% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 80% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 90% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" async function quickchart(key) {\n",
|
|
" const quickchartButtonEl =\n",
|
|
" document.querySelector('#' + key + ' button');\n",
|
|
" quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n",
|
|
" quickchartButtonEl.classList.add('colab-df-spinner');\n",
|
|
" try {\n",
|
|
" const charts = await google.colab.kernel.invokeFunction(\n",
|
|
" 'suggestCharts', [key], {});\n",
|
|
" } catch (error) {\n",
|
|
" console.error('Error during call to suggestCharts:', error);\n",
|
|
" }\n",
|
|
" quickchartButtonEl.classList.remove('colab-df-spinner');\n",
|
|
" quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
|
|
" }\n",
|
|
" (() => {\n",
|
|
" let quickchartButtonEl =\n",
|
|
" document.querySelector('#df-f9fd59c8-c209-4e9c-9694-c2f81a8901d8 button');\n",
|
|
" quickchartButtonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
" })();\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
" </div>\n",
|
|
" </div>\n"
|
|
],
|
|
"text/plain": [
|
|
" review \\\n",
|
|
"0 Remove yourself from the Kirk Douglass aspects... \n",
|
|
"1 Surprising that Jules Verne would write such a... \n",
|
|
"2 The Light At The Edge of the World marks Kirk ... \n",
|
|
"3 it is a quite slow pace face-to-face Brynner/D... \n",
|
|
"4 Jules Verne wrote about 80 novels as well as p... \n",
|
|
"\n",
|
|
" keywords sentiment \n",
|
|
"0 [Kirk Douglass casting, beautiful young woman,... Positive \n",
|
|
"1 [Jules Verne, Hollywood, Yul Brynner, cruelty,... Positive \n",
|
|
"2 [The Light At The Edge of the World, Kirk Doug... Positive \n",
|
|
"3 [slow pace, Brynner/Douglas, violence, gore, s... Neutral \n",
|
|
"4 [Jules Verne, The Lighthouse at the Edge of th... Negative "
|
|
]
|
|
},
|
|
"execution_count": 50,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"%%bigquery --project {PROJECT_ID}\n",
|
|
"\n",
|
|
"SELECT\n",
|
|
" review,\n",
|
|
" keywords,\n",
|
|
" sentiment\n",
|
|
"FROM\n",
|
|
" AI.GENERATE_TABLE(\n",
|
|
" MODEL `bq_ai_tutorial.gemini_2_5_flash`,\n",
|
|
" (\n",
|
|
" SELECT\n",
|
|
" review,\n",
|
|
" ('Extract the keywords and sentiment (Positive, Negative, or Neutral) from the following review:',review) AS prompt,\n",
|
|
" FROM\n",
|
|
" `bigquery-public-data.imdb.reviews`\n",
|
|
" WHERE movie_id = \"tt0067345\"\n",
|
|
" LIMIT 5\n",
|
|
" ),\n",
|
|
" STRUCT(\n",
|
|
" 'keywords ARRAY< STRING>, sentiment STRING' AS output_schema\n",
|
|
" )\n",
|
|
" );"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "WkdFwoOp1cRm"
|
|
},
|
|
"source": [
|
|
"#### Saving results in a BigQuery table\n",
|
|
"Wrapping any `SELECT` statement with the [`CREATE TABLE` statement](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#create_table_statement) allows you to create a permanent table from your query results. You can run the next two queries to see that in action."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "I0yvijK71Zbm"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%%bigquery --project {PROJECT_ID}\n",
|
|
"\n",
|
|
"CREATE OR REPLACE TABLE `bq_ai_tutorial.reviews_keywords_sentiment` AS (\n",
|
|
" SELECT\n",
|
|
" review,\n",
|
|
" keywords,\n",
|
|
" sentiment\n",
|
|
" FROM\n",
|
|
" AI.GENERATE_TABLE(\n",
|
|
" MODEL `bq_ai_tutorial.gemini_2_5_flash`,\n",
|
|
" (\n",
|
|
" SELECT\n",
|
|
" review,\n",
|
|
" ('Extract the keywords and sentiment (Positive, Negative, or Neutral) from the following review:',review) AS prompt,\n",
|
|
" FROM\n",
|
|
" `bigquery-public-data.imdb.reviews`\n",
|
|
" WHERE movie_id = \"tt0067345\"\n",
|
|
" LIMIT 5\n",
|
|
" ),\n",
|
|
" STRUCT(\n",
|
|
" 'keywords ARRAY< STRING>, sentiment STRING' AS output_schema)));"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 30,
|
|
"metadata": {
|
|
"id": "JW88rus93d9D"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Query is running: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Downloading: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.google.colaboratory.intrinsic+json": {
|
|
"summary": "{\n \"name\": \"get_ipython()\",\n \"rows\": 5,\n \"fields\": [\n {\n \"column\": \"review\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Jules Verne wrote about 80 novels as well as plays and short stories in his career. He began writing in 1854 with a short story called \\\"Master Zacharias, or the Clockmaker's Soul\\\". It was the first time he talked of the negative side of progress - the evil that results from some discoveries or inventions when they fall into the wrong hands. This becomes a running theme in his novels: Captain Nemo in TWENTY THOUSAND LEAGUES UNDER THE SEA, or Robur (from ROBUR THE CONQUEROR and it's sequel, THE MASTER OF THE WORLD) are two of his best examples of this them. Kongre, in THE LIGHTHOUSE AT THE EDGE OF THE WORLD, is another.Verne was so prolific that when he died in 1905 he left a dozen unpublished novels and stories that were not fully published until 1910. They include some of his best writing, such as THE BARSAC MISSION (partly written by Verne's son Michael), THE SURVIVORS OF THE \\\"JONATHAN\\\", THE PURSUIT OF THE METEOR, THE DANUBE PILOT. All of these dealt with science, but also dealt with political systems, and economics, for Verne was interested in all the problems facing modern man. THE LIGHTHOUSE AT THE EDGE OF THE WORLD was the last novel that was published in Verne's lifetime. It does not deal with the political questions or economic ones that perplexed him, but seems to go back to his potboiler period, when he was turning out stories for money while considering better stories for later publication. But nothing Verne wrote is without interest. Rereading THE LIGHTHOUSE one sees what the subtle point is in it. It is the study of how the ego of a villain can prevent him from escaping retribution.Kongre (renamed Jonathan Kongre) is one of the last pirates in the world of 1900. He and his gang find a damaged boat and repair it. They sail it across the Pacific, and reach Staten Island, a small island in the Straits of Magellan controlled by Chile. There they find a lighthouse with a crew of three men. They manage to kill two of them, but the third one (named Vasquez - he's from Chile, remember), hides on the island. Kongre and his men decide that they should prepare to leave the island shortly, before the Chilean Naval relief boat returns in three months to pick up the lighthouse crew. But first they will wreck any boat that comes to the passage, and increase their ill-gotten gains. But the key to the novel (and it is not in the movie) is that Kongre's right hand men (Carcante and Vargas) keep urging him to pack up his supplies and wealth and head to Asia where the money can be divvied up and everyone separate in safety. And each time Kongre won't do it. Initially it is pure greed. He wrecks a boat, and massacres the crew (a scene that is done in the film). The sole survivor is an American, John Davis (the name became Denton in the film, except that it was given to the character of Vasquez). Now with an ally (and not a drunken one, as in the film), Vasquez starts sabotaging Kongre's activities on the island. Carcante keeps suggesting leaving, but Kongre (unused to someone annoying him successfully) keeps delaying in order to catch Vasquez and Davis. The end result is that when he thinks he has them cornered, the Chilean boat appears to sink his craft, kill most of his crew, and confront him. Kongre commits suicide to avoid capture.Much of the mayhem of the movie (with Denton picking off crew members one at a time) is not in the book. Nor is there any female character in the novel (a rarity in most of Verne's stories - he could be quite a feminist when he wished). The egotism of \\\"Jonathan\\\" Kongre is well shown by Yul Brynner's performance, but the subtlety of that trait is lost. The writers presumably did not think the audience could appreciate it. Kirk Douglas does well enough as Denton, but his singlehanded success (Vasquez and Davis work together well to the end of the story, unlike Denton's ally who is killed by the pirates) seems unlikely. The bestiality of the pirates is well shown in the movie, particularly a singularly tall actor who in one scene wears women's clothing to particularly unsettling effect. The film is not a bad minor adventure film, but it could have been better if they had stuck to Verne's theme.\",\n \"Remove yourself from the Kirk Douglass aspects of the casting. It is essential to your enjoying the film. There is a beautiful young woman playing double roles - and in the photos from the 1800's, I can't believe how smooth and white her skin is. Also, there is an excellent degrading of the film stock which chills the mind if you like faded greys and yellows as I do. This film is played on TNT from time to time so see it.\",\n \"Surprising that Jules Verne would write such a story. Even more surprising that Hollywood would produce it. Yul Brynner is unbelievably good as a man freed of all bounds of convention, free to indulge his taste for cruelty and domination. Douglass is an excellent counterpoint, a courageous individual who's chosen simple solitude as a way to deal with the complications and turmoil society had imposed on him. And Samantha Egger's character is the capper, a woman willing to sacrifice far too much for comfort and safety. Inotherwords, everyman. The movie does show its age and is limited by the conventions of time and place and technology of the time. If you're looking for special effects and action as substitutes for thought, look elsewhere.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"keywords\",\n \"properties\": {\n \"dtype\": \"object\",\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"sentiment\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 2,\n \"samples\": [\n \"Positive\",\n \"Neutral\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}",
|
|
"type": "dataframe"
|
|
},
|
|
"text/html": [
|
|
"\n",
|
|
" <div id=\"df-f46bf28d-2f97-4217-bd24-b23228f3424c\" class=\"colab-df-container\">\n",
|
|
" <div>\n",
|
|
"<style scoped>\n",
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|
" vertical-align: middle;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe tbody tr th {\n",
|
|
" vertical-align: top;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe thead th {\n",
|
|
" text-align: right;\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: right;\">\n",
|
|
" <th></th>\n",
|
|
" <th>review</th>\n",
|
|
" <th>keywords</th>\n",
|
|
" <th>sentiment</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <th>0</th>\n",
|
|
" <td>it is a quite slow pace face-to-face Brynner/D...</td>\n",
|
|
" <td>[slow pace, face-to-face, Brynner, Douglas, he...</td>\n",
|
|
" <td>Neutral</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>1</th>\n",
|
|
" <td>Jules Verne wrote about 80 novels as well as p...</td>\n",
|
|
" <td>[Jules Verne, novels, plays, short stories, Ma...</td>\n",
|
|
" <td>Neutral</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>2</th>\n",
|
|
" <td>Surprising that Jules Verne would write such a...</td>\n",
|
|
" <td>[Jules Verne, Hollywood, Yul Brynner, cruelty,...</td>\n",
|
|
" <td>Positive</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>3</th>\n",
|
|
" <td>The Light At The Edge of the World marks Kirk ...</td>\n",
|
|
" <td>[The Light At The Edge of the World, Kirk Doug...</td>\n",
|
|
" <td>Positive</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>4</th>\n",
|
|
" <td>Remove yourself from the Kirk Douglass aspects...</td>\n",
|
|
" <td>[Kirk Douglass casting, beautiful young woman,...</td>\n",
|
|
" <td>Positive</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>\n",
|
|
"</div>\n",
|
|
" <div class=\"colab-df-buttons\">\n",
|
|
"\n",
|
|
" <div class=\"colab-df-container\">\n",
|
|
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f46bf28d-2f97-4217-bd24-b23228f3424c')\"\n",
|
|
" title=\"Convert this dataframe to an interactive table.\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
|
|
" <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
|
|
" </svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
" <style>\n",
|
|
" .colab-df-container {\n",
|
|
" display:flex;\n",
|
|
" gap: 12px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert {\n",
|
|
" background-color: #E8F0FE;\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: #1967D2;\n",
|
|
" height: 32px;\n",
|
|
" padding: 0 0 0 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert:hover {\n",
|
|
" background-color: #E2EBFA;\n",
|
|
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: #174EA6;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-buttons div {\n",
|
|
" margin-bottom: 4px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert {\n",
|
|
" background-color: #3B4455;\n",
|
|
" fill: #D2E3FC;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert:hover {\n",
|
|
" background-color: #434B5C;\n",
|
|
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
|
|
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
|
|
" fill: #FFFFFF;\n",
|
|
" }\n",
|
|
" </style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" const buttonEl =\n",
|
|
" document.querySelector('#df-f46bf28d-2f97-4217-bd24-b23228f3424c button.colab-df-convert');\n",
|
|
" buttonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
"\n",
|
|
" async function convertToInteractive(key) {\n",
|
|
" const element = document.querySelector('#df-f46bf28d-2f97-4217-bd24-b23228f3424c');\n",
|
|
" const dataTable =\n",
|
|
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
|
|
" [key], {});\n",
|
|
" if (!dataTable) return;\n",
|
|
"\n",
|
|
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
|
|
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
|
|
" + ' to learn more about interactive tables.';\n",
|
|
" element.innerHTML = '';\n",
|
|
" dataTable['output_type'] = 'display_data';\n",
|
|
" await google.colab.output.renderOutput(dataTable, element);\n",
|
|
" const docLink = document.createElement('div');\n",
|
|
" docLink.innerHTML = docLinkHtml;\n",
|
|
" element.appendChild(docLink);\n",
|
|
" }\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
"\n",
|
|
" <div id=\"df-8c21336e-b40f-4d01-b46a-fceeacc46388\">\n",
|
|
" <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-8c21336e-b40f-4d01-b46a-fceeacc46388')\"\n",
|
|
" title=\"Suggest charts\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
"<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
|
|
" width=\"24px\">\n",
|
|
" <g>\n",
|
|
" <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
|
|
" </g>\n",
|
|
"</svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
"<style>\n",
|
|
" .colab-df-quickchart {\n",
|
|
" --bg-color: #E8F0FE;\n",
|
|
" --fill-color: #1967D2;\n",
|
|
" --hover-bg-color: #E2EBFA;\n",
|
|
" --hover-fill-color: #174EA6;\n",
|
|
" --disabled-fill-color: #AAA;\n",
|
|
" --disabled-bg-color: #DDD;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-quickchart {\n",
|
|
" --bg-color: #3B4455;\n",
|
|
" --fill-color: #D2E3FC;\n",
|
|
" --hover-bg-color: #434B5C;\n",
|
|
" --hover-fill-color: #FFFFFF;\n",
|
|
" --disabled-bg-color: #3B4455;\n",
|
|
" --disabled-fill-color: #666;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart {\n",
|
|
" background-color: var(--bg-color);\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: var(--fill-color);\n",
|
|
" height: 32px;\n",
|
|
" padding: 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart:hover {\n",
|
|
" background-color: var(--hover-bg-color);\n",
|
|
" box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: var(--button-hover-fill-color);\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart-complete:disabled,\n",
|
|
" .colab-df-quickchart-complete:disabled:hover {\n",
|
|
" background-color: var(--disabled-bg-color);\n",
|
|
" fill: var(--disabled-fill-color);\n",
|
|
" box-shadow: none;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-spinner {\n",
|
|
" border: 2px solid var(--fill-color);\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" animation:\n",
|
|
" spin 1s steps(1) infinite;\n",
|
|
" }\n",
|
|
"\n",
|
|
" @keyframes spin {\n",
|
|
" 0% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 20% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 30% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 40% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 60% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 80% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 90% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" async function quickchart(key) {\n",
|
|
" const quickchartButtonEl =\n",
|
|
" document.querySelector('#' + key + ' button');\n",
|
|
" quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n",
|
|
" quickchartButtonEl.classList.add('colab-df-spinner');\n",
|
|
" try {\n",
|
|
" const charts = await google.colab.kernel.invokeFunction(\n",
|
|
" 'suggestCharts', [key], {});\n",
|
|
" } catch (error) {\n",
|
|
" console.error('Error during call to suggestCharts:', error);\n",
|
|
" }\n",
|
|
" quickchartButtonEl.classList.remove('colab-df-spinner');\n",
|
|
" quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
|
|
" }\n",
|
|
" (() => {\n",
|
|
" let quickchartButtonEl =\n",
|
|
" document.querySelector('#df-8c21336e-b40f-4d01-b46a-fceeacc46388 button');\n",
|
|
" quickchartButtonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
" })();\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
" </div>\n",
|
|
" </div>\n"
|
|
],
|
|
"text/plain": [
|
|
" review \\\n",
|
|
"0 it is a quite slow pace face-to-face Brynner/D... \n",
|
|
"1 Jules Verne wrote about 80 novels as well as p... \n",
|
|
"2 Surprising that Jules Verne would write such a... \n",
|
|
"3 The Light At The Edge of the World marks Kirk ... \n",
|
|
"4 Remove yourself from the Kirk Douglass aspects... \n",
|
|
"\n",
|
|
" keywords sentiment \n",
|
|
"0 [slow pace, face-to-face, Brynner, Douglas, he... Neutral \n",
|
|
"1 [Jules Verne, novels, plays, short stories, Ma... Neutral \n",
|
|
"2 [Jules Verne, Hollywood, Yul Brynner, cruelty,... Positive \n",
|
|
"3 [The Light At The Edge of the World, Kirk Doug... Positive \n",
|
|
"4 [Kirk Douglass casting, beautiful young woman,... Positive "
|
|
]
|
|
},
|
|
"execution_count": 30,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"%%bigquery --project {PROJECT_ID}\n",
|
|
"\n",
|
|
"SELECT *\n",
|
|
"FROM `bq_ai_tutorial.reviews_keywords_sentiment`"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "T0Hz06RiecoI"
|
|
},
|
|
"source": [
|
|
"#### Using arguments to adjust model configuration\n",
|
|
"Similar to `AI.GENERATE`, [`AI.GENERATE_TABLE`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-generate-table) also allows you to customize the parameters for your generation request. The placement of these parameters is within the `STRUCT` value. The [documentation page](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-generate-table) contains the available fields you can define within the `STRUCT`.\n",
|
|
"\n",
|
|
"Next you'll take the same query and now define values for `temperature` and `max_output_tokens`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 31,
|
|
"metadata": {
|
|
"id": "SyGREvkxeXzh"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Query is running: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Downloading: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.google.colaboratory.intrinsic+json": {
|
|
"summary": "{\n \"name\": \"get_ipython()\",\n \"rows\": 5,\n \"fields\": [\n {\n \"column\": \"review\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"The Light At The Edge of the World marks Kirk Douglas's second filming of a Jules Verne novel. The first of course was one of his most popular films 20,000 Leagues Under The Sea. But this film is far more serious and has far more adult themes than Walt Disney's film aimed for the kid trade.This was the last novel Jules Verne had published during his lifetime and it's a story of survival against almost impossible odds. In the original novel Kirk Douglas's character was named Vasquez which certainly was more in keeping with someone assigned to lighthouse duty on Cape Horn. But in giving Douglas's character an Anglo name it better explains his presence on the island and it certainly is in keeping with the international tradition of Jules Verne's writings.Cape Horn is one of the loneliest parts of the globe and the geography of the southern tip of South America. Look on a map of the many islands and rocks in that part of the globe and imagine how rough the sea is because it has only limited space. It's not without reason that sailors in all cultures say that no one is really a true sailor until they've made a voyage crossing from the Atlantic to the Pacific Ocean in that area. Remember also this is 1865 as Yul Brynner identifies the year and the Panama Canal had not been built.Which makes the lighthouse at Cape Horn an international concern which was something that is ever present in Jules Verne's writings. But then as now there are malevolent forces in the world and they are in this story Yul Brynner and his pirate crew.On one desultory like any other down there, Yul Brynner's ship docks at the island and kills lighthouse keeper Fernando Rey and his young assistant Massimo Ranieri. By sheer dumb luck Douglas is not at the lighthouse when this happens, but he becomes a hunted man by Brynner and his pirate crew who want to set up headquarters there and use the light to pile up as many wrecks as they can plunder. Also they want to eliminate Douglas who's now the only witness to their crimes.I did like this film very much both when first seeing it in the theater and now on VHS. One thing of interest I found here is that there is no ambiguity, no shadings of character. Kirk Douglas is a good guy and Yul Brynner a bad one, no one is going to walk away thinking anything else. In fact Yul Brynner's pirate captain Jonathan Kongre is the most unredeemable villain we've seen on screen since Lee Marvin as Liberty Valance.Definitely fans of Kirk Douglas, Yul Brynner and Jules Verne should earmark this film for their collection.\",\n \"Jules Verne wrote about 80 novels as well as plays and short stories in his career. He began writing in 1854 with a short story called \\\"Master Zacharias, or the Clockmaker's Soul\\\". It was the first time he talked of the negative side of progress - the evil that results from some discoveries or inventions when they fall into the wrong hands. This becomes a running theme in his novels: Captain Nemo in TWENTY THOUSAND LEAGUES UNDER THE SEA, or Robur (from ROBUR THE CONQUEROR and it's sequel, THE MASTER OF THE WORLD) are two of his best examples of this them. Kongre, in THE LIGHTHOUSE AT THE EDGE OF THE WORLD, is another.Verne was so prolific that when he died in 1905 he left a dozen unpublished novels and stories that were not fully published until 1910. They include some of his best writing, such as THE BARSAC MISSION (partly written by Verne's son Michael), THE SURVIVORS OF THE \\\"JONATHAN\\\", THE PURSUIT OF THE METEOR, THE DANUBE PILOT. All of these dealt with science, but also dealt with political systems, and economics, for Verne was interested in all the problems facing modern man. THE LIGHTHOUSE AT THE EDGE OF THE WORLD was the last novel that was published in Verne's lifetime. It does not deal with the political questions or economic ones that perplexed him, but seems to go back to his potboiler period, when he was turning out stories for money while considering better stories for later publication. But nothing Verne wrote is without interest. Rereading THE LIGHTHOUSE one sees what the subtle point is in it. It is the study of how the ego of a villain can prevent him from escaping retribution.Kongre (renamed Jonathan Kongre) is one of the last pirates in the world of 1900. He and his gang find a damaged boat and repair it. They sail it across the Pacific, and reach Staten Island, a small island in the Straits of Magellan controlled by Chile. There they find a lighthouse with a crew of three men. They manage to kill two of them, but the third one (named Vasquez - he's from Chile, remember), hides on the island. Kongre and his men decide that they should prepare to leave the island shortly, before the Chilean Naval relief boat returns in three months to pick up the lighthouse crew. But first they will wreck any boat that comes to the passage, and increase their ill-gotten gains. But the key to the novel (and it is not in the movie) is that Kongre's right hand men (Carcante and Vargas) keep urging him to pack up his supplies and wealth and head to Asia where the money can be divvied up and everyone separate in safety. And each time Kongre won't do it. Initially it is pure greed. He wrecks a boat, and massacres the crew (a scene that is done in the film). The sole survivor is an American, John Davis (the name became Denton in the film, except that it was given to the character of Vasquez). Now with an ally (and not a drunken one, as in the film), Vasquez starts sabotaging Kongre's activities on the island. Carcante keeps suggesting leaving, but Kongre (unused to someone annoying him successfully) keeps delaying in order to catch Vasquez and Davis. The end result is that when he thinks he has them cornered, the Chilean boat appears to sink his craft, kill most of his crew, and confront him. Kongre commits suicide to avoid capture.Much of the mayhem of the movie (with Denton picking off crew members one at a time) is not in the book. Nor is there any female character in the novel (a rarity in most of Verne's stories - he could be quite a feminist when he wished). The egotism of \\\"Jonathan\\\" Kongre is well shown by Yul Brynner's performance, but the subtlety of that trait is lost. The writers presumably did not think the audience could appreciate it. Kirk Douglas does well enough as Denton, but his singlehanded success (Vasquez and Davis work together well to the end of the story, unlike Denton's ally who is killed by the pirates) seems unlikely. The bestiality of the pirates is well shown in the movie, particularly a singularly tall actor who in one scene wears women's clothing to particularly unsettling effect. The film is not a bad minor adventure film, but it could have been better if they had stuck to Verne's theme.\",\n \"Remove yourself from the Kirk Douglass aspects of the casting. It is essential to your enjoying the film. There is a beautiful young woman playing double roles - and in the photos from the 1800's, I can't believe how smooth and white her skin is. Also, there is an excellent degrading of the film stock which chills the mind if you like faded greys and yellows as I do. This film is played on TNT from time to time so see it.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"keywords\",\n \"properties\": {\n \"dtype\": \"object\",\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"sentiment\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 3,\n \"samples\": [\n \"Neutral\",\n \"Positive\",\n \"Negative\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}",
|
|
"type": "dataframe"
|
|
},
|
|
"text/html": [
|
|
"\n",
|
|
" <div id=\"df-fde32f92-b13d-4d0b-83f9-8c573a30a9ab\" class=\"colab-df-container\">\n",
|
|
" <div>\n",
|
|
"<style scoped>\n",
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|
" vertical-align: middle;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe tbody tr th {\n",
|
|
" vertical-align: top;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe thead th {\n",
|
|
" text-align: right;\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: right;\">\n",
|
|
" <th></th>\n",
|
|
" <th>review</th>\n",
|
|
" <th>keywords</th>\n",
|
|
" <th>sentiment</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <th>0</th>\n",
|
|
" <td>it is a quite slow pace face-to-face Brynner/D...</td>\n",
|
|
" <td>[slow pace, heavy violence/gore, seventies gia...</td>\n",
|
|
" <td>Neutral</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>1</th>\n",
|
|
" <td>The Light At The Edge of the World marks Kirk ...</td>\n",
|
|
" <td>[Kirk Douglas, Jules Verne, The Light At The E...</td>\n",
|
|
" <td>Positive</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>2</th>\n",
|
|
" <td>Remove yourself from the Kirk Douglass aspects...</td>\n",
|
|
" <td>[beautiful young woman, double roles, film sto...</td>\n",
|
|
" <td>Positive</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>3</th>\n",
|
|
" <td>Surprising that Jules Verne would write such a...</td>\n",
|
|
" <td>[Jules Verne, Hollywood, Yul Brynner, cruelty ...</td>\n",
|
|
" <td>Positive</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>4</th>\n",
|
|
" <td>Jules Verne wrote about 80 novels as well as p...</td>\n",
|
|
" <td>[Jules Verne, novels, themes, negative side of...</td>\n",
|
|
" <td>Negative</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>\n",
|
|
"</div>\n",
|
|
" <div class=\"colab-df-buttons\">\n",
|
|
"\n",
|
|
" <div class=\"colab-df-container\">\n",
|
|
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-fde32f92-b13d-4d0b-83f9-8c573a30a9ab')\"\n",
|
|
" title=\"Convert this dataframe to an interactive table.\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
|
|
" <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
|
|
" </svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
" <style>\n",
|
|
" .colab-df-container {\n",
|
|
" display:flex;\n",
|
|
" gap: 12px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert {\n",
|
|
" background-color: #E8F0FE;\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: #1967D2;\n",
|
|
" height: 32px;\n",
|
|
" padding: 0 0 0 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert:hover {\n",
|
|
" background-color: #E2EBFA;\n",
|
|
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: #174EA6;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-buttons div {\n",
|
|
" margin-bottom: 4px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert {\n",
|
|
" background-color: #3B4455;\n",
|
|
" fill: #D2E3FC;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert:hover {\n",
|
|
" background-color: #434B5C;\n",
|
|
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
|
|
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
|
|
" fill: #FFFFFF;\n",
|
|
" }\n",
|
|
" </style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" const buttonEl =\n",
|
|
" document.querySelector('#df-fde32f92-b13d-4d0b-83f9-8c573a30a9ab button.colab-df-convert');\n",
|
|
" buttonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
"\n",
|
|
" async function convertToInteractive(key) {\n",
|
|
" const element = document.querySelector('#df-fde32f92-b13d-4d0b-83f9-8c573a30a9ab');\n",
|
|
" const dataTable =\n",
|
|
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
|
|
" [key], {});\n",
|
|
" if (!dataTable) return;\n",
|
|
"\n",
|
|
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
|
|
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
|
|
" + ' to learn more about interactive tables.';\n",
|
|
" element.innerHTML = '';\n",
|
|
" dataTable['output_type'] = 'display_data';\n",
|
|
" await google.colab.output.renderOutput(dataTable, element);\n",
|
|
" const docLink = document.createElement('div');\n",
|
|
" docLink.innerHTML = docLinkHtml;\n",
|
|
" element.appendChild(docLink);\n",
|
|
" }\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
"\n",
|
|
" <div id=\"df-c10a366d-e8ee-4f4b-9990-7e649917aebf\">\n",
|
|
" <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-c10a366d-e8ee-4f4b-9990-7e649917aebf')\"\n",
|
|
" title=\"Suggest charts\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
"<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
|
|
" width=\"24px\">\n",
|
|
" <g>\n",
|
|
" <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
|
|
" </g>\n",
|
|
"</svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
"<style>\n",
|
|
" .colab-df-quickchart {\n",
|
|
" --bg-color: #E8F0FE;\n",
|
|
" --fill-color: #1967D2;\n",
|
|
" --hover-bg-color: #E2EBFA;\n",
|
|
" --hover-fill-color: #174EA6;\n",
|
|
" --disabled-fill-color: #AAA;\n",
|
|
" --disabled-bg-color: #DDD;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-quickchart {\n",
|
|
" --bg-color: #3B4455;\n",
|
|
" --fill-color: #D2E3FC;\n",
|
|
" --hover-bg-color: #434B5C;\n",
|
|
" --hover-fill-color: #FFFFFF;\n",
|
|
" --disabled-bg-color: #3B4455;\n",
|
|
" --disabled-fill-color: #666;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart {\n",
|
|
" background-color: var(--bg-color);\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: var(--fill-color);\n",
|
|
" height: 32px;\n",
|
|
" padding: 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart:hover {\n",
|
|
" background-color: var(--hover-bg-color);\n",
|
|
" box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: var(--button-hover-fill-color);\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart-complete:disabled,\n",
|
|
" .colab-df-quickchart-complete:disabled:hover {\n",
|
|
" background-color: var(--disabled-bg-color);\n",
|
|
" fill: var(--disabled-fill-color);\n",
|
|
" box-shadow: none;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-spinner {\n",
|
|
" border: 2px solid var(--fill-color);\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" animation:\n",
|
|
" spin 1s steps(1) infinite;\n",
|
|
" }\n",
|
|
"\n",
|
|
" @keyframes spin {\n",
|
|
" 0% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 20% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 30% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 40% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 60% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 80% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 90% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" async function quickchart(key) {\n",
|
|
" const quickchartButtonEl =\n",
|
|
" document.querySelector('#' + key + ' button');\n",
|
|
" quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n",
|
|
" quickchartButtonEl.classList.add('colab-df-spinner');\n",
|
|
" try {\n",
|
|
" const charts = await google.colab.kernel.invokeFunction(\n",
|
|
" 'suggestCharts', [key], {});\n",
|
|
" } catch (error) {\n",
|
|
" console.error('Error during call to suggestCharts:', error);\n",
|
|
" }\n",
|
|
" quickchartButtonEl.classList.remove('colab-df-spinner');\n",
|
|
" quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
|
|
" }\n",
|
|
" (() => {\n",
|
|
" let quickchartButtonEl =\n",
|
|
" document.querySelector('#df-c10a366d-e8ee-4f4b-9990-7e649917aebf button');\n",
|
|
" quickchartButtonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
" })();\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
" </div>\n",
|
|
" </div>\n"
|
|
],
|
|
"text/plain": [
|
|
" review \\\n",
|
|
"0 it is a quite slow pace face-to-face Brynner/D... \n",
|
|
"1 The Light At The Edge of the World marks Kirk ... \n",
|
|
"2 Remove yourself from the Kirk Douglass aspects... \n",
|
|
"3 Surprising that Jules Verne would write such a... \n",
|
|
"4 Jules Verne wrote about 80 novels as well as p... \n",
|
|
"\n",
|
|
" keywords sentiment \n",
|
|
"0 [slow pace, heavy violence/gore, seventies gia... Neutral \n",
|
|
"1 [Kirk Douglas, Jules Verne, The Light At The E... Positive \n",
|
|
"2 [beautiful young woman, double roles, film sto... Positive \n",
|
|
"3 [Jules Verne, Hollywood, Yul Brynner, cruelty ... Positive \n",
|
|
"4 [Jules Verne, novels, themes, negative side of... Negative "
|
|
]
|
|
},
|
|
"execution_count": 31,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"%%bigquery --project {PROJECT_ID}\n",
|
|
"\n",
|
|
"SELECT\n",
|
|
" review,\n",
|
|
" keywords,\n",
|
|
" sentiment\n",
|
|
"FROM\n",
|
|
" AI.GENERATE_TABLE(\n",
|
|
" MODEL `bq_ai_tutorial.gemini_2_5_flash`,\n",
|
|
" (\n",
|
|
" SELECT\n",
|
|
" review,\n",
|
|
" ('Extract the key words and sentiment (Positive, Negative, or Neutral) from the following review:',review) AS prompt,\n",
|
|
" FROM\n",
|
|
" `bigquery-public-data.imdb.reviews`\n",
|
|
" WHERE movie_id = \"tt0067345\"\n",
|
|
" LIMIT 5\n",
|
|
" ),\n",
|
|
" STRUCT(\n",
|
|
" 'keywords ARRAY< STRING>, sentiment STRING' AS output_schema,\n",
|
|
" 0.5 AS temperature,\n",
|
|
" 1000 AS max_output_tokens\n",
|
|
" )\n",
|
|
" );"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "tN51GtTMYEoF"
|
|
},
|
|
"source": [
|
|
"## Generate text with partner and open models with `AI.GENERATE_TEXT`\n",
|
|
"\n",
|
|
"The tutorial thus far has performed analysis using the [Gemini family of models](https://cloud.google.com/vertex-ai/generative-ai/docs/models#generally_available_models). However, the generative analysis capabilities of BigQuery extend to several [partner models](https://cloud.google.com/bigquery/docs/generate-text#enable-model), including Anthropic Claude, Llama, and Mistral AI, as well as [open models](https://cloud.google.com/bigquery/docs/generate-text#deploy_an_open_model) from the [Vertex AI Model Garden and Hugging Face](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-create-remote-model-open#supported_open_models).\n",
|
|
"\n",
|
|
"When choosing a partner or open model, you will use the [`AI.GENERATE_TEXT` function](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate-text) to perform your analysis. The syntax for the `AI.GENERATE_TEXT` function is similar to `AI.GENERATE_TABLE`. Let's walk through an example analysis using a partner model."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "RB90JcraZrwf"
|
|
},
|
|
"source": [
|
|
"### Enabling a partner model\n",
|
|
"\n",
|
|
"You must enable partner models in Vertex AI before you can use them in BigQuery.\n",
|
|
"\n",
|
|
"**BEFORE MOVING AHEAD**, enable the [**Anthropic Claude 3 Haiku** model](https://console.cloud.google.com/vertex-ai/publishers/anthropic/model-garden/claude-3-haiku) by following the [instructions in the documentation](https://cloud.google.com/bigquery/docs/generate-text#enable-model)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "plOQu7eIjeEs"
|
|
},
|
|
"source": [
|
|
"### Create a remote model for Anthropic Claude 3 Haiku\n",
|
|
"\n",
|
|
"Now you'll create a new remote model, pointing to the `claude-3-haiku@20240307` model endpoint."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "wrYOI_HGjQT2"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%%bigquery --project {PROJECT_ID}\n",
|
|
"\n",
|
|
"CREATE OR REPLACE MODEL\n",
|
|
" `bq_ai_tutorial.claude_3_haiku`\n",
|
|
"REMOTE WITH\n",
|
|
" CONNECTION `us.test_connection`\n",
|
|
" OPTIONS (ENDPOINT = 'claude-3-haiku@20240307');"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "vtJ5nuMbmY0C"
|
|
},
|
|
"source": [
|
|
"### Run analysis using the partner model\n",
|
|
"\n",
|
|
"Now you are ready to use the remote model with an [`AI.GENERATE_TEXT` function](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate-text). The following query repeats your previous analysis done to extract key words and sentiment from the reviews, now using Anthropic Claude 3 Haiku."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 52,
|
|
"metadata": {
|
|
"id": "HaPTzBYrZnr4"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Query is running: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Downloading: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.google.colaboratory.intrinsic+json": {
|
|
"summary": "{\n \"name\": \"get_ipython()\",\n \"rows\": 5,\n \"fields\": [\n {\n \"column\": \"review\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Remove yourself from the Kirk Douglass aspects of the casting. It is essential to your enjoying the film. There is a beautiful young woman playing double roles - and in the photos from the 1800's, I can't believe how smooth and white her skin is. Also, there is an excellent degrading of the film stock which chills the mind if you like faded greys and yellows as I do. This film is played on TNT from time to time so see it.\",\n \"it is a quite slow pace face-to-face Brynner/Douglas...i found there was some pretty heavy violence/gore but you know, seventies giallo style : (maybe some spoiler ahead) a little bit phony and fake but the image is there : the monkey torn apart, the flesh torn from the mechanic, etc...i don't remember if it is alike the novel...but overall i think it is a OK action flick, hero flick : he stands all alone at the end, all the bad guys are out..........and nice images too : a small rocky island, the sun over the seabut not a typical Verne's story-on-cinema : subs, sci-fi; but the adventure seen in a lot of Verne's novels tough\",\n \"Surprising that Jules Verne would write such a story. Even more surprising that Hollywood would produce it. Yul Brynner is unbelievably good as a man freed of all bounds of convention, free to indulge his taste for cruelty and domination. Douglass is an excellent counterpoint, a courageous individual who's chosen simple solitude as a way to deal with the complications and turmoil society had imposed on him. And Samantha Egger's character is the capper, a woman willing to sacrifice far too much for comfort and safety. Inotherwords, everyman. The movie does show its age and is limited by the conventions of time and place and technology of the time. If you're looking for special effects and action as substitutes for thought, look elsewhere.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"result\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Keywords:\\n- Kirk Douglass\\n- double roles\\n- 1800's\\n- film stock\\n- faded greys and yellows\\n- TNT\\n\\nSentiment:\\nPositive\\n\\nThe review overall has a positive sentiment. The reviewer praises the casting, the performance of the young woman playing double roles, and the excellent degrading of the film stock. The reviewer also recommends watching the film on TNT, indicating a positive recommendation.\",\n \"Keywords:\\n- slow pace\\n- face-to-face Brynner/Douglas\\n- heavy violence/gore\\n- seventies giallo style\\n- phony and fake\\n- monkey torn apart\\n- flesh torn from the mechanic\\n- not alike the novel\\n- OK action flick\\n- hero flick\\n- stands all alone at the end\\n- bad guys are out\\n- small rocky island\\n- sun over the sea\\n- not a typical Verne's story-on-cinema\\n- subs, sci-fi\\n- adventure seen in a lot of Verne's novels\\n\\nSentiment:\\nThe review has a mix of positive, negative, and neutral sentiments.\\n\\nPositive:\\n- \\\"nice images too: a small rocky island, the sun over the sea\\\"\\n- \\\"OK action flick, hero flick\\\"\\n\\nNegative:\\n- \\\"quite slow pace\\\"\\n- \\\"heavy violence/gore\\\"\\n- \\\"a little bit phony and fake\\\"\\n\\nNeutral:\\n- \\\"not alike the novel\\\"\\n- \\\"not a typical Verne's story-on-cinema\\\"\\n- \\\"subs, sci-fi; but the adventure seen in a lot of Verne's novels tough\\\"\",\n \"Keywords:\\n- Jules Verne\\n- Hollywood\\n- Yul Brynner\\n- Douglass\\n- Samantha Egger\\n- society\\n- special effects\\n- action\\n- thought\\n\\nSentiment:\\nThe overall sentiment of the review is Positive. The reviewer praises the performances of the actors, particularly Yul Brynner, and the complexity of the characters and themes in the movie. However, the reviewer also notes that the movie \\\"shows its age\\\" and that it is \\\"limited by the conventions of time and place and technology of the time,\\\" suggesting a slightly Negative sentiment towards some aspects of the film.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}",
|
|
"type": "dataframe"
|
|
},
|
|
"text/html": [
|
|
"\n",
|
|
" <div id=\"df-d00d980e-cdb7-4a4f-a272-0a134add5c3e\" class=\"colab-df-container\">\n",
|
|
" <div>\n",
|
|
"<style scoped>\n",
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|
" vertical-align: middle;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe tbody tr th {\n",
|
|
" vertical-align: top;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe thead th {\n",
|
|
" text-align: right;\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: right;\">\n",
|
|
" <th></th>\n",
|
|
" <th>review</th>\n",
|
|
" <th>result</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <th>0</th>\n",
|
|
" <td>The Light At The Edge of the World marks Kirk ...</td>\n",
|
|
" <td>Keywords:\\n1. Jules Verne\\n2. Kirk Douglas\\n3....</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>1</th>\n",
|
|
" <td>Remove yourself from the Kirk Douglass aspects...</td>\n",
|
|
" <td>Keywords:\\n- Kirk Douglass\\n- double roles\\n- ...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>2</th>\n",
|
|
" <td>Surprising that Jules Verne would write such a...</td>\n",
|
|
" <td>Keywords:\\n- Jules Verne\\n- Hollywood\\n- Yul B...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>3</th>\n",
|
|
" <td>Jules Verne wrote about 80 novels as well as p...</td>\n",
|
|
" <td>Keywords:\\n- Jules Verne\\n- Novels\\n- Progress...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>4</th>\n",
|
|
" <td>it is a quite slow pace face-to-face Brynner/D...</td>\n",
|
|
" <td>Keywords:\\n- slow pace\\n- face-to-face Brynner...</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>\n",
|
|
"</div>\n",
|
|
" <div class=\"colab-df-buttons\">\n",
|
|
"\n",
|
|
" <div class=\"colab-df-container\">\n",
|
|
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d00d980e-cdb7-4a4f-a272-0a134add5c3e')\"\n",
|
|
" title=\"Convert this dataframe to an interactive table.\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
|
|
" <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
|
|
" </svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
" <style>\n",
|
|
" .colab-df-container {\n",
|
|
" display:flex;\n",
|
|
" gap: 12px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert {\n",
|
|
" background-color: #E8F0FE;\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: #1967D2;\n",
|
|
" height: 32px;\n",
|
|
" padding: 0 0 0 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert:hover {\n",
|
|
" background-color: #E2EBFA;\n",
|
|
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: #174EA6;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-buttons div {\n",
|
|
" margin-bottom: 4px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert {\n",
|
|
" background-color: #3B4455;\n",
|
|
" fill: #D2E3FC;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert:hover {\n",
|
|
" background-color: #434B5C;\n",
|
|
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
|
|
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
|
|
" fill: #FFFFFF;\n",
|
|
" }\n",
|
|
" </style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" const buttonEl =\n",
|
|
" document.querySelector('#df-d00d980e-cdb7-4a4f-a272-0a134add5c3e button.colab-df-convert');\n",
|
|
" buttonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
"\n",
|
|
" async function convertToInteractive(key) {\n",
|
|
" const element = document.querySelector('#df-d00d980e-cdb7-4a4f-a272-0a134add5c3e');\n",
|
|
" const dataTable =\n",
|
|
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
|
|
" [key], {});\n",
|
|
" if (!dataTable) return;\n",
|
|
"\n",
|
|
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
|
|
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
|
|
" + ' to learn more about interactive tables.';\n",
|
|
" element.innerHTML = '';\n",
|
|
" dataTable['output_type'] = 'display_data';\n",
|
|
" await google.colab.output.renderOutput(dataTable, element);\n",
|
|
" const docLink = document.createElement('div');\n",
|
|
" docLink.innerHTML = docLinkHtml;\n",
|
|
" element.appendChild(docLink);\n",
|
|
" }\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
"\n",
|
|
" <div id=\"df-e5512bc0-33fd-4e6b-847e-d1bec1df4d50\">\n",
|
|
" <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-e5512bc0-33fd-4e6b-847e-d1bec1df4d50')\"\n",
|
|
" title=\"Suggest charts\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
"<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
|
|
" width=\"24px\">\n",
|
|
" <g>\n",
|
|
" <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
|
|
" </g>\n",
|
|
"</svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
"<style>\n",
|
|
" .colab-df-quickchart {\n",
|
|
" --bg-color: #E8F0FE;\n",
|
|
" --fill-color: #1967D2;\n",
|
|
" --hover-bg-color: #E2EBFA;\n",
|
|
" --hover-fill-color: #174EA6;\n",
|
|
" --disabled-fill-color: #AAA;\n",
|
|
" --disabled-bg-color: #DDD;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-quickchart {\n",
|
|
" --bg-color: #3B4455;\n",
|
|
" --fill-color: #D2E3FC;\n",
|
|
" --hover-bg-color: #434B5C;\n",
|
|
" --hover-fill-color: #FFFFFF;\n",
|
|
" --disabled-bg-color: #3B4455;\n",
|
|
" --disabled-fill-color: #666;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart {\n",
|
|
" background-color: var(--bg-color);\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: var(--fill-color);\n",
|
|
" height: 32px;\n",
|
|
" padding: 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart:hover {\n",
|
|
" background-color: var(--hover-bg-color);\n",
|
|
" box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: var(--button-hover-fill-color);\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart-complete:disabled,\n",
|
|
" .colab-df-quickchart-complete:disabled:hover {\n",
|
|
" background-color: var(--disabled-bg-color);\n",
|
|
" fill: var(--disabled-fill-color);\n",
|
|
" box-shadow: none;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-spinner {\n",
|
|
" border: 2px solid var(--fill-color);\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" animation:\n",
|
|
" spin 1s steps(1) infinite;\n",
|
|
" }\n",
|
|
"\n",
|
|
" @keyframes spin {\n",
|
|
" 0% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 20% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 30% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 40% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 60% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 80% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 90% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" async function quickchart(key) {\n",
|
|
" const quickchartButtonEl =\n",
|
|
" document.querySelector('#' + key + ' button');\n",
|
|
" quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n",
|
|
" quickchartButtonEl.classList.add('colab-df-spinner');\n",
|
|
" try {\n",
|
|
" const charts = await google.colab.kernel.invokeFunction(\n",
|
|
" 'suggestCharts', [key], {});\n",
|
|
" } catch (error) {\n",
|
|
" console.error('Error during call to suggestCharts:', error);\n",
|
|
" }\n",
|
|
" quickchartButtonEl.classList.remove('colab-df-spinner');\n",
|
|
" quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
|
|
" }\n",
|
|
" (() => {\n",
|
|
" let quickchartButtonEl =\n",
|
|
" document.querySelector('#df-e5512bc0-33fd-4e6b-847e-d1bec1df4d50 button');\n",
|
|
" quickchartButtonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
" })();\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
" </div>\n",
|
|
" </div>\n"
|
|
],
|
|
"text/plain": [
|
|
" review \\\n",
|
|
"0 The Light At The Edge of the World marks Kirk ... \n",
|
|
"1 Remove yourself from the Kirk Douglass aspects... \n",
|
|
"2 Surprising that Jules Verne would write such a... \n",
|
|
"3 Jules Verne wrote about 80 novels as well as p... \n",
|
|
"4 it is a quite slow pace face-to-face Brynner/D... \n",
|
|
"\n",
|
|
" result \n",
|
|
"0 Keywords:\\n1. Jules Verne\\n2. Kirk Douglas\\n3.... \n",
|
|
"1 Keywords:\\n- Kirk Douglass\\n- double roles\\n- ... \n",
|
|
"2 Keywords:\\n- Jules Verne\\n- Hollywood\\n- Yul B... \n",
|
|
"3 Keywords:\\n- Jules Verne\\n- Novels\\n- Progress... \n",
|
|
"4 Keywords:\\n- slow pace\\n- face-to-face Brynner... "
|
|
]
|
|
},
|
|
"execution_count": 52,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"%%bigquery --project {PROJECT_ID}\n",
|
|
"\n",
|
|
"SELECT\n",
|
|
" review,\n",
|
|
" result\n",
|
|
"FROM\n",
|
|
" AI.GENERATE_TEXT(\n",
|
|
" MODEL `bq_ai_tutorial.claude_3_haiku`,\n",
|
|
" (\n",
|
|
" SELECT\n",
|
|
" review,\n",
|
|
" CONCAT('Extract the keywords and sentiment (Positive, Negative, or Neutral) from the following review:',review) AS prompt,\n",
|
|
" FROM\n",
|
|
" `bigquery-public-data.imdb.reviews`\n",
|
|
" WHERE movie_id = \"tt0067345\"\n",
|
|
" LIMIT 5\n",
|
|
" ),\n",
|
|
" STRUCT(\n",
|
|
" 500 AS max_output_tokens\n",
|
|
" )\n",
|
|
" );"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "scalar_md"
|
|
},
|
|
"source": [
|
|
"## Perform row-level AI with scalar functions `AI.GENERATE_BOOL`, `AI.GENERATE_DOUBLE`, and `AI.GENERATE_INT`\n",
|
|
"\n",
|
|
"BigQuery's scalar AI functions ([`AI.GENERATE_BOOL`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate-bool), [`AI.GENERATE_DOUBLE`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate-double), and [`AI.GENERATE_INT`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate-int)) bring the power of LLMs for AI-driven filtering, classification, and data extraction directly within your SQL queries.\n",
|
|
"\n",
|
|
"On the surface, these functions perform tasks similar to that of using `AI.GENERATE` with a defined `output_schema` of data type of `BOOL`, `FLOAT64`, or `INT64`. However, these functions aren't only available for use in a `SELECT` clause, but also in a `WHERE` clause, giving you the power to do complex filtering with natural language. The next few examples focus on the filtering use case.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "generate_bool_md"
|
|
},
|
|
"source": [
|
|
"### Using `AI.GENERATE_BOOL`: Filtering for drought tolerant tree species\n",
|
|
"\n",
|
|
"Let's run the next query to take a look at the `bigquery-public-data.new_york_trees.tree_species` table."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 36,
|
|
"metadata": {
|
|
"id": "YuKFqCK3oHXg"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Query is running: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Downloading: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.google.colaboratory.intrinsic+json": {
|
|
"summary": "{\n \"name\": \"get_ipython()\",\n \"rows\": 57,\n \"fields\": [\n {\n \"column\": \"species_scientific_name\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 57,\n \"samples\": [\n \"Quercus phellos\",\n \"Carpinus caroliniana\",\n \"Prunus cerasifera\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"species_common_name\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 57,\n \"samples\": [\n \"Willow Oak\",\n \"American Hornbeam\",\n \"Purpleleaf Plum\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}",
|
|
"type": "dataframe"
|
|
},
|
|
"text/html": [
|
|
"\n",
|
|
" <div id=\"df-26595e66-63ad-47c7-a254-ef809fe903d3\" class=\"colab-df-container\">\n",
|
|
" <div>\n",
|
|
"<style scoped>\n",
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|
" vertical-align: middle;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe tbody tr th {\n",
|
|
" vertical-align: top;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe thead th {\n",
|
|
" text-align: right;\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: right;\">\n",
|
|
" <th></th>\n",
|
|
" <th>species_scientific_name</th>\n",
|
|
" <th>species_common_name</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <th>0</th>\n",
|
|
" <td>Quercus phellos</td>\n",
|
|
" <td>Willow Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>1</th>\n",
|
|
" <td>Cotinus sp.</td>\n",
|
|
" <td>Smoke Tree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>2</th>\n",
|
|
" <td>Prunus sargentii</td>\n",
|
|
" <td>Sargent Cherry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>3</th>\n",
|
|
" <td>Prunus padus</td>\n",
|
|
" <td>European Birdcherry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>4</th>\n",
|
|
" <td>Prunus serrulata 'Kwanzan'</td>\n",
|
|
" <td>Japanese Flowering Cherry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>5</th>\n",
|
|
" <td>Carpinus caroliniana</td>\n",
|
|
" <td>American Hornbeam</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>6</th>\n",
|
|
" <td>Acer ginnala</td>\n",
|
|
" <td>Amur Maple</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>7</th>\n",
|
|
" <td>Fraxinus 'Leprechaun'</td>\n",
|
|
" <td>Leprechaun Green Ash</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>8</th>\n",
|
|
" <td>Amelanchier sp.</td>\n",
|
|
" <td>Serviceberry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>9</th>\n",
|
|
" <td>Malus sp.</td>\n",
|
|
" <td>Crabapple</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>10</th>\n",
|
|
" <td>Lagerstroemia indica</td>\n",
|
|
" <td>Crapemyrtle</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>11</th>\n",
|
|
" <td>Eucommia ulmoides</td>\n",
|
|
" <td>Hardy Rubber Tree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>12</th>\n",
|
|
" <td>Ostrya virginiana</td>\n",
|
|
" <td>American Hophornbeam</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>13</th>\n",
|
|
" <td>Acer truncatum</td>\n",
|
|
" <td>Shantung Maple</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>14</th>\n",
|
|
" <td>Acer campestre</td>\n",
|
|
" <td>Hedge Maple</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>15</th>\n",
|
|
" <td>Maackia amurensis</td>\n",
|
|
" <td>Amur Maackia</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>16</th>\n",
|
|
" <td>Quercus imbricaria</td>\n",
|
|
" <td>Shingle Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>17</th>\n",
|
|
" <td>Quercus rubra</td>\n",
|
|
" <td>Northern Red Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>18</th>\n",
|
|
" <td>Gymnocladus dioicus</td>\n",
|
|
" <td>Coffeetree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>19</th>\n",
|
|
" <td>Tilia x euchlora</td>\n",
|
|
" <td>Crimean Linden</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>20</th>\n",
|
|
" <td>Tilia tomentosa</td>\n",
|
|
" <td>Silver Linden</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>21</th>\n",
|
|
" <td>Celtis occidentalis</td>\n",
|
|
" <td>Hackberry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>22</th>\n",
|
|
" <td>Tilia americana</td>\n",
|
|
" <td>American Linden</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>23</th>\n",
|
|
" <td>Quercus bicolor</td>\n",
|
|
" <td>Swamp White Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>24</th>\n",
|
|
" <td>Quercus palustris</td>\n",
|
|
" <td>Pin Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>25</th>\n",
|
|
" <td>Platanus x acerifolia</td>\n",
|
|
" <td>London Plane</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>26</th>\n",
|
|
" <td>Gleditsia triacanthos var. inermis</td>\n",
|
|
" <td>Honeylocust</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>27</th>\n",
|
|
" <td>Prunus 'Okame'</td>\n",
|
|
" <td>Okame Cherry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>28</th>\n",
|
|
" <td>Prunus x yedoensis</td>\n",
|
|
" <td>Yoshino Cherry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>29</th>\n",
|
|
" <td>Cornus mas</td>\n",
|
|
" <td>Cornelian Cherry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>30</th>\n",
|
|
" <td>Prunus cerasifera</td>\n",
|
|
" <td>Purpleleaf Plum</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>31</th>\n",
|
|
" <td>Crataegus sp.</td>\n",
|
|
" <td>Hawthorn</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>32</th>\n",
|
|
" <td>Cercis canadensis</td>\n",
|
|
" <td>Eastern Redbud</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>33</th>\n",
|
|
" <td>Syringa reticulata</td>\n",
|
|
" <td>Japanese Tree Lilac</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>34</th>\n",
|
|
" <td>Ulmus parvifolia</td>\n",
|
|
" <td>Chinese Elm</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>35</th>\n",
|
|
" <td>Cercidiphyllum japonicum</td>\n",
|
|
" <td>Katsura Tree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>36</th>\n",
|
|
" <td>Acer rubrum</td>\n",
|
|
" <td>Red Maple</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>37</th>\n",
|
|
" <td>Quercus acutissima</td>\n",
|
|
" <td>Sawtooth Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>38</th>\n",
|
|
" <td>Styphnolobium japonicum</td>\n",
|
|
" <td>Scholar Tree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>39</th>\n",
|
|
" <td>Koelreuteria paniculata</td>\n",
|
|
" <td>Goldenraintree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>40</th>\n",
|
|
" <td>Pyrus calleryana</td>\n",
|
|
" <td>Callery Pear</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>41</th>\n",
|
|
" <td>Quercus spp. 'Fastigiata'</td>\n",
|
|
" <td>Fastigiata Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>42</th>\n",
|
|
" <td>Ginkgo biloba</td>\n",
|
|
" <td>Ginkgo</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>43</th>\n",
|
|
" <td>Quercus robur</td>\n",
|
|
" <td>English Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>44</th>\n",
|
|
" <td>Carpinus betulus</td>\n",
|
|
" <td>European Hornbeam</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>45</th>\n",
|
|
" <td>Liriodendron tulipifera</td>\n",
|
|
" <td>Tulip Tree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>46</th>\n",
|
|
" <td>Liquidambar styraciflua</td>\n",
|
|
" <td>Sweetgum</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>47</th>\n",
|
|
" <td>Tilia cordata</td>\n",
|
|
" <td>Littleleaf Linden</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>48</th>\n",
|
|
" <td>Fraxinus pennsylvanica</td>\n",
|
|
" <td>Green Ash</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>49</th>\n",
|
|
" <td>Fraxinus americana</td>\n",
|
|
" <td>White Ash</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>50</th>\n",
|
|
" <td>Metasequoia glyptostroboides</td>\n",
|
|
" <td>Dawn Redwood</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>51</th>\n",
|
|
" <td>Taxodium distichum</td>\n",
|
|
" <td>Baldcypress</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>52</th>\n",
|
|
" <td>Prunus virginiana 'Schubert'</td>\n",
|
|
" <td>Schubert Cherry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>53</th>\n",
|
|
" <td>Nyssa sylvatica</td>\n",
|
|
" <td>Black Gum</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>54</th>\n",
|
|
" <td>Corylus colurna</td>\n",
|
|
" <td>Turkish Filbert</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>55</th>\n",
|
|
" <td>Ulmus americana</td>\n",
|
|
" <td>American Elm</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>56</th>\n",
|
|
" <td>Zelkova serrata</td>\n",
|
|
" <td>Japanese Zelkova</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>\n",
|
|
"</div>\n",
|
|
" <div class=\"colab-df-buttons\">\n",
|
|
"\n",
|
|
" <div class=\"colab-df-container\">\n",
|
|
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-26595e66-63ad-47c7-a254-ef809fe903d3')\"\n",
|
|
" title=\"Convert this dataframe to an interactive table.\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
|
|
" <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
|
|
" </svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
" <style>\n",
|
|
" .colab-df-container {\n",
|
|
" display:flex;\n",
|
|
" gap: 12px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert {\n",
|
|
" background-color: #E8F0FE;\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: #1967D2;\n",
|
|
" height: 32px;\n",
|
|
" padding: 0 0 0 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert:hover {\n",
|
|
" background-color: #E2EBFA;\n",
|
|
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: #174EA6;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-buttons div {\n",
|
|
" margin-bottom: 4px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert {\n",
|
|
" background-color: #3B4455;\n",
|
|
" fill: #D2E3FC;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert:hover {\n",
|
|
" background-color: #434B5C;\n",
|
|
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
|
|
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
|
|
" fill: #FFFFFF;\n",
|
|
" }\n",
|
|
" </style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" const buttonEl =\n",
|
|
" document.querySelector('#df-26595e66-63ad-47c7-a254-ef809fe903d3 button.colab-df-convert');\n",
|
|
" buttonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
"\n",
|
|
" async function convertToInteractive(key) {\n",
|
|
" const element = document.querySelector('#df-26595e66-63ad-47c7-a254-ef809fe903d3');\n",
|
|
" const dataTable =\n",
|
|
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
|
|
" [key], {});\n",
|
|
" if (!dataTable) return;\n",
|
|
"\n",
|
|
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
|
|
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
|
|
" + ' to learn more about interactive tables.';\n",
|
|
" element.innerHTML = '';\n",
|
|
" dataTable['output_type'] = 'display_data';\n",
|
|
" await google.colab.output.renderOutput(dataTable, element);\n",
|
|
" const docLink = document.createElement('div');\n",
|
|
" docLink.innerHTML = docLinkHtml;\n",
|
|
" element.appendChild(docLink);\n",
|
|
" }\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
"\n",
|
|
" <div id=\"df-eea567f0-b0b2-4433-805a-4977bd696a88\">\n",
|
|
" <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-eea567f0-b0b2-4433-805a-4977bd696a88')\"\n",
|
|
" title=\"Suggest charts\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
"<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
|
|
" width=\"24px\">\n",
|
|
" <g>\n",
|
|
" <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
|
|
" </g>\n",
|
|
"</svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
"<style>\n",
|
|
" .colab-df-quickchart {\n",
|
|
" --bg-color: #E8F0FE;\n",
|
|
" --fill-color: #1967D2;\n",
|
|
" --hover-bg-color: #E2EBFA;\n",
|
|
" --hover-fill-color: #174EA6;\n",
|
|
" --disabled-fill-color: #AAA;\n",
|
|
" --disabled-bg-color: #DDD;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-quickchart {\n",
|
|
" --bg-color: #3B4455;\n",
|
|
" --fill-color: #D2E3FC;\n",
|
|
" --hover-bg-color: #434B5C;\n",
|
|
" --hover-fill-color: #FFFFFF;\n",
|
|
" --disabled-bg-color: #3B4455;\n",
|
|
" --disabled-fill-color: #666;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart {\n",
|
|
" background-color: var(--bg-color);\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: var(--fill-color);\n",
|
|
" height: 32px;\n",
|
|
" padding: 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart:hover {\n",
|
|
" background-color: var(--hover-bg-color);\n",
|
|
" box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: var(--button-hover-fill-color);\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart-complete:disabled,\n",
|
|
" .colab-df-quickchart-complete:disabled:hover {\n",
|
|
" background-color: var(--disabled-bg-color);\n",
|
|
" fill: var(--disabled-fill-color);\n",
|
|
" box-shadow: none;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-spinner {\n",
|
|
" border: 2px solid var(--fill-color);\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" animation:\n",
|
|
" spin 1s steps(1) infinite;\n",
|
|
" }\n",
|
|
"\n",
|
|
" @keyframes spin {\n",
|
|
" 0% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 20% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 30% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 40% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 60% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 80% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 90% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" async function quickchart(key) {\n",
|
|
" const quickchartButtonEl =\n",
|
|
" document.querySelector('#' + key + ' button');\n",
|
|
" quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n",
|
|
" quickchartButtonEl.classList.add('colab-df-spinner');\n",
|
|
" try {\n",
|
|
" const charts = await google.colab.kernel.invokeFunction(\n",
|
|
" 'suggestCharts', [key], {});\n",
|
|
" } catch (error) {\n",
|
|
" console.error('Error during call to suggestCharts:', error);\n",
|
|
" }\n",
|
|
" quickchartButtonEl.classList.remove('colab-df-spinner');\n",
|
|
" quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
|
|
" }\n",
|
|
" (() => {\n",
|
|
" let quickchartButtonEl =\n",
|
|
" document.querySelector('#df-eea567f0-b0b2-4433-805a-4977bd696a88 button');\n",
|
|
" quickchartButtonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
" })();\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
" </div>\n",
|
|
" </div>\n"
|
|
],
|
|
"text/plain": [
|
|
" species_scientific_name species_common_name\n",
|
|
"0 Quercus phellos Willow Oak\n",
|
|
"1 Cotinus sp. Smoke Tree\n",
|
|
"2 Prunus sargentii Sargent Cherry\n",
|
|
"3 Prunus padus European Birdcherry\n",
|
|
"4 Prunus serrulata 'Kwanzan' Japanese Flowering Cherry\n",
|
|
"5 Carpinus caroliniana American Hornbeam\n",
|
|
"6 Acer ginnala Amur Maple\n",
|
|
"7 Fraxinus 'Leprechaun' Leprechaun Green Ash\n",
|
|
"8 Amelanchier sp. Serviceberry\n",
|
|
"9 Malus sp. Crabapple\n",
|
|
"10 Lagerstroemia indica Crapemyrtle\n",
|
|
"11 Eucommia ulmoides Hardy Rubber Tree\n",
|
|
"12 Ostrya virginiana American Hophornbeam\n",
|
|
"13 Acer truncatum Shantung Maple\n",
|
|
"14 Acer campestre Hedge Maple\n",
|
|
"15 Maackia amurensis Amur Maackia\n",
|
|
"16 Quercus imbricaria Shingle Oak\n",
|
|
"17 Quercus rubra Northern Red Oak\n",
|
|
"18 Gymnocladus dioicus Coffeetree\n",
|
|
"19 Tilia x euchlora Crimean Linden\n",
|
|
"20 Tilia tomentosa Silver Linden\n",
|
|
"21 Celtis occidentalis Hackberry\n",
|
|
"22 Tilia americana American Linden\n",
|
|
"23 Quercus bicolor Swamp White Oak\n",
|
|
"24 Quercus palustris Pin Oak\n",
|
|
"25 Platanus x acerifolia London Plane\n",
|
|
"26 Gleditsia triacanthos var. inermis Honeylocust\n",
|
|
"27 Prunus 'Okame' Okame Cherry\n",
|
|
"28 Prunus x yedoensis Yoshino Cherry\n",
|
|
"29 Cornus mas Cornelian Cherry\n",
|
|
"30 Prunus cerasifera Purpleleaf Plum\n",
|
|
"31 Crataegus sp. Hawthorn\n",
|
|
"32 Cercis canadensis Eastern Redbud\n",
|
|
"33 Syringa reticulata Japanese Tree Lilac\n",
|
|
"34 Ulmus parvifolia Chinese Elm\n",
|
|
"35 Cercidiphyllum japonicum Katsura Tree\n",
|
|
"36 Acer rubrum Red Maple\n",
|
|
"37 Quercus acutissima Sawtooth Oak\n",
|
|
"38 Styphnolobium japonicum Scholar Tree\n",
|
|
"39 Koelreuteria paniculata Goldenraintree\n",
|
|
"40 Pyrus calleryana Callery Pear\n",
|
|
"41 Quercus spp. 'Fastigiata' Fastigiata Oak\n",
|
|
"42 Ginkgo biloba Ginkgo\n",
|
|
"43 Quercus robur English Oak\n",
|
|
"44 Carpinus betulus European Hornbeam\n",
|
|
"45 Liriodendron tulipifera Tulip Tree\n",
|
|
"46 Liquidambar styraciflua Sweetgum\n",
|
|
"47 Tilia cordata Littleleaf Linden\n",
|
|
"48 Fraxinus pennsylvanica Green Ash\n",
|
|
"49 Fraxinus americana White Ash\n",
|
|
"50 Metasequoia glyptostroboides Dawn Redwood\n",
|
|
"51 Taxodium distichum Baldcypress\n",
|
|
"52 Prunus virginiana 'Schubert' Schubert Cherry\n",
|
|
"53 Nyssa sylvatica Black Gum\n",
|
|
"54 Corylus colurna Turkish Filbert\n",
|
|
"55 Ulmus americana American Elm\n",
|
|
"56 Zelkova serrata Japanese Zelkova"
|
|
]
|
|
},
|
|
"execution_count": 36,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"%%bigquery --project {PROJECT_ID}\n",
|
|
"\n",
|
|
"SELECT\n",
|
|
" species_scientific_name,\n",
|
|
" species_common_name\n",
|
|
"FROM\n",
|
|
" `bigquery-public-data.new_york_trees.tree_species`"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "f8om0-IQoOiX"
|
|
},
|
|
"source": [
|
|
"Next, let's use [`AI.GENERATE_BOOL`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate-bool) to return only tree species that are drought tolerant."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 37,
|
|
"metadata": {
|
|
"id": "generate_bool_code"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Query is running: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Downloading: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.google.colaboratory.intrinsic+json": {
|
|
"summary": "{\n \"name\": \"get_ipython()\",\n \"rows\": 39,\n \"fields\": [\n {\n \"column\": \"species_scientific_name\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 39,\n \"samples\": [\n \"Lagerstroemia indica\",\n \"Gymnocladus dioicus\",\n \"Quercus bicolor\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"species_common_name\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 39,\n \"samples\": [\n \"Crapemyrtle\",\n \"Coffeetree\",\n \"Swamp White Oak\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}",
|
|
"type": "dataframe"
|
|
},
|
|
"text/html": [
|
|
"\n",
|
|
" <div id=\"df-a1921419-2a0e-4c3f-8121-c29ecb90b0a1\" class=\"colab-df-container\">\n",
|
|
" <div>\n",
|
|
"<style scoped>\n",
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|
" vertical-align: middle;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe tbody tr th {\n",
|
|
" vertical-align: top;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe thead th {\n",
|
|
" text-align: right;\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: right;\">\n",
|
|
" <th></th>\n",
|
|
" <th>species_scientific_name</th>\n",
|
|
" <th>species_common_name</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <th>0</th>\n",
|
|
" <td>Koelreuteria paniculata</td>\n",
|
|
" <td>Goldenraintree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>1</th>\n",
|
|
" <td>Corylus colurna</td>\n",
|
|
" <td>Turkish Filbert</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>2</th>\n",
|
|
" <td>Ostrya virginiana</td>\n",
|
|
" <td>American Hophornbeam</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>3</th>\n",
|
|
" <td>Cornus mas</td>\n",
|
|
" <td>Cornelian Cherry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>4</th>\n",
|
|
" <td>Quercus bicolor</td>\n",
|
|
" <td>Swamp White Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>5</th>\n",
|
|
" <td>Crataegus sp.</td>\n",
|
|
" <td>Hawthorn</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>6</th>\n",
|
|
" <td>Styphnolobium japonicum</td>\n",
|
|
" <td>Scholar Tree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>7</th>\n",
|
|
" <td>Prunus virginiana 'Schubert'</td>\n",
|
|
" <td>Schubert Cherry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>8</th>\n",
|
|
" <td>Cercis canadensis</td>\n",
|
|
" <td>Eastern Redbud</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>9</th>\n",
|
|
" <td>Quercus spp. 'Fastigiata'</td>\n",
|
|
" <td>Fastigiata Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>10</th>\n",
|
|
" <td>Liquidambar styraciflua</td>\n",
|
|
" <td>Sweetgum</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>11</th>\n",
|
|
" <td>Platanus x acerifolia</td>\n",
|
|
" <td>London Plane</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>12</th>\n",
|
|
" <td>Prunus 'Okame'</td>\n",
|
|
" <td>Okame Cherry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>13</th>\n",
|
|
" <td>Ulmus parvifolia</td>\n",
|
|
" <td>Chinese Elm</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>14</th>\n",
|
|
" <td>Fraxinus pennsylvanica</td>\n",
|
|
" <td>Green Ash</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>15</th>\n",
|
|
" <td>Zelkova serrata</td>\n",
|
|
" <td>Japanese Zelkova</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>16</th>\n",
|
|
" <td>Cotinus sp.</td>\n",
|
|
" <td>Smoke Tree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>17</th>\n",
|
|
" <td>Acer ginnala</td>\n",
|
|
" <td>Amur Maple</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>18</th>\n",
|
|
" <td>Amelanchier sp.</td>\n",
|
|
" <td>Serviceberry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>19</th>\n",
|
|
" <td>Acer campestre</td>\n",
|
|
" <td>Hedge Maple</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>20</th>\n",
|
|
" <td>Gleditsia triacanthos var. inermis</td>\n",
|
|
" <td>Honeylocust</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>21</th>\n",
|
|
" <td>Acer truncatum</td>\n",
|
|
" <td>Shantung Maple</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>22</th>\n",
|
|
" <td>Celtis occidentalis</td>\n",
|
|
" <td>Hackberry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>23</th>\n",
|
|
" <td>Ginkgo biloba</td>\n",
|
|
" <td>Ginkgo</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>24</th>\n",
|
|
" <td>Fraxinus americana</td>\n",
|
|
" <td>White Ash</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>25</th>\n",
|
|
" <td>Carpinus caroliniana</td>\n",
|
|
" <td>American Hornbeam</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>26</th>\n",
|
|
" <td>Eucommia ulmoides</td>\n",
|
|
" <td>Hardy Rubber Tree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>27</th>\n",
|
|
" <td>Quercus rubra</td>\n",
|
|
" <td>Northern Red Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>28</th>\n",
|
|
" <td>Tilia tomentosa</td>\n",
|
|
" <td>Silver Linden</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>29</th>\n",
|
|
" <td>Prunus cerasifera</td>\n",
|
|
" <td>Purpleleaf Plum</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>30</th>\n",
|
|
" <td>Syringa reticulata</td>\n",
|
|
" <td>Japanese Tree Lilac</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>31</th>\n",
|
|
" <td>Quercus acutissima</td>\n",
|
|
" <td>Sawtooth Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>32</th>\n",
|
|
" <td>Nyssa sylvatica</td>\n",
|
|
" <td>Black Gum</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>33</th>\n",
|
|
" <td>Lagerstroemia indica</td>\n",
|
|
" <td>Crapemyrtle</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>34</th>\n",
|
|
" <td>Maackia amurensis</td>\n",
|
|
" <td>Amur Maackia</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>35</th>\n",
|
|
" <td>Quercus imbricaria</td>\n",
|
|
" <td>Shingle Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>36</th>\n",
|
|
" <td>Gymnocladus dioicus</td>\n",
|
|
" <td>Coffeetree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>37</th>\n",
|
|
" <td>Pyrus calleryana</td>\n",
|
|
" <td>Callery Pear</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>38</th>\n",
|
|
" <td>Quercus robur</td>\n",
|
|
" <td>English Oak</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>\n",
|
|
"</div>\n",
|
|
" <div class=\"colab-df-buttons\">\n",
|
|
"\n",
|
|
" <div class=\"colab-df-container\">\n",
|
|
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a1921419-2a0e-4c3f-8121-c29ecb90b0a1')\"\n",
|
|
" title=\"Convert this dataframe to an interactive table.\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
|
|
" <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
|
|
" </svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
" <style>\n",
|
|
" .colab-df-container {\n",
|
|
" display:flex;\n",
|
|
" gap: 12px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert {\n",
|
|
" background-color: #E8F0FE;\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: #1967D2;\n",
|
|
" height: 32px;\n",
|
|
" padding: 0 0 0 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert:hover {\n",
|
|
" background-color: #E2EBFA;\n",
|
|
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: #174EA6;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-buttons div {\n",
|
|
" margin-bottom: 4px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert {\n",
|
|
" background-color: #3B4455;\n",
|
|
" fill: #D2E3FC;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert:hover {\n",
|
|
" background-color: #434B5C;\n",
|
|
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
|
|
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
|
|
" fill: #FFFFFF;\n",
|
|
" }\n",
|
|
" </style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" const buttonEl =\n",
|
|
" document.querySelector('#df-a1921419-2a0e-4c3f-8121-c29ecb90b0a1 button.colab-df-convert');\n",
|
|
" buttonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
"\n",
|
|
" async function convertToInteractive(key) {\n",
|
|
" const element = document.querySelector('#df-a1921419-2a0e-4c3f-8121-c29ecb90b0a1');\n",
|
|
" const dataTable =\n",
|
|
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
|
|
" [key], {});\n",
|
|
" if (!dataTable) return;\n",
|
|
"\n",
|
|
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
|
|
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
|
|
" + ' to learn more about interactive tables.';\n",
|
|
" element.innerHTML = '';\n",
|
|
" dataTable['output_type'] = 'display_data';\n",
|
|
" await google.colab.output.renderOutput(dataTable, element);\n",
|
|
" const docLink = document.createElement('div');\n",
|
|
" docLink.innerHTML = docLinkHtml;\n",
|
|
" element.appendChild(docLink);\n",
|
|
" }\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
"\n",
|
|
" <div id=\"df-c7e6eeb9-023a-4f8b-be81-cc344474a11a\">\n",
|
|
" <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-c7e6eeb9-023a-4f8b-be81-cc344474a11a')\"\n",
|
|
" title=\"Suggest charts\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
"<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
|
|
" width=\"24px\">\n",
|
|
" <g>\n",
|
|
" <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
|
|
" </g>\n",
|
|
"</svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
"<style>\n",
|
|
" .colab-df-quickchart {\n",
|
|
" --bg-color: #E8F0FE;\n",
|
|
" --fill-color: #1967D2;\n",
|
|
" --hover-bg-color: #E2EBFA;\n",
|
|
" --hover-fill-color: #174EA6;\n",
|
|
" --disabled-fill-color: #AAA;\n",
|
|
" --disabled-bg-color: #DDD;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-quickchart {\n",
|
|
" --bg-color: #3B4455;\n",
|
|
" --fill-color: #D2E3FC;\n",
|
|
" --hover-bg-color: #434B5C;\n",
|
|
" --hover-fill-color: #FFFFFF;\n",
|
|
" --disabled-bg-color: #3B4455;\n",
|
|
" --disabled-fill-color: #666;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart {\n",
|
|
" background-color: var(--bg-color);\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: var(--fill-color);\n",
|
|
" height: 32px;\n",
|
|
" padding: 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart:hover {\n",
|
|
" background-color: var(--hover-bg-color);\n",
|
|
" box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: var(--button-hover-fill-color);\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart-complete:disabled,\n",
|
|
" .colab-df-quickchart-complete:disabled:hover {\n",
|
|
" background-color: var(--disabled-bg-color);\n",
|
|
" fill: var(--disabled-fill-color);\n",
|
|
" box-shadow: none;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-spinner {\n",
|
|
" border: 2px solid var(--fill-color);\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" animation:\n",
|
|
" spin 1s steps(1) infinite;\n",
|
|
" }\n",
|
|
"\n",
|
|
" @keyframes spin {\n",
|
|
" 0% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 20% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 30% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 40% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 60% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 80% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 90% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" async function quickchart(key) {\n",
|
|
" const quickchartButtonEl =\n",
|
|
" document.querySelector('#' + key + ' button');\n",
|
|
" quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n",
|
|
" quickchartButtonEl.classList.add('colab-df-spinner');\n",
|
|
" try {\n",
|
|
" const charts = await google.colab.kernel.invokeFunction(\n",
|
|
" 'suggestCharts', [key], {});\n",
|
|
" } catch (error) {\n",
|
|
" console.error('Error during call to suggestCharts:', error);\n",
|
|
" }\n",
|
|
" quickchartButtonEl.classList.remove('colab-df-spinner');\n",
|
|
" quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
|
|
" }\n",
|
|
" (() => {\n",
|
|
" let quickchartButtonEl =\n",
|
|
" document.querySelector('#df-c7e6eeb9-023a-4f8b-be81-cc344474a11a button');\n",
|
|
" quickchartButtonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
" })();\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
" </div>\n",
|
|
" </div>\n"
|
|
],
|
|
"text/plain": [
|
|
" species_scientific_name species_common_name\n",
|
|
"0 Koelreuteria paniculata Goldenraintree\n",
|
|
"1 Corylus colurna Turkish Filbert\n",
|
|
"2 Ostrya virginiana American Hophornbeam\n",
|
|
"3 Cornus mas Cornelian Cherry\n",
|
|
"4 Quercus bicolor Swamp White Oak\n",
|
|
"5 Crataegus sp. Hawthorn\n",
|
|
"6 Styphnolobium japonicum Scholar Tree\n",
|
|
"7 Prunus virginiana 'Schubert' Schubert Cherry\n",
|
|
"8 Cercis canadensis Eastern Redbud\n",
|
|
"9 Quercus spp. 'Fastigiata' Fastigiata Oak\n",
|
|
"10 Liquidambar styraciflua Sweetgum\n",
|
|
"11 Platanus x acerifolia London Plane\n",
|
|
"12 Prunus 'Okame' Okame Cherry\n",
|
|
"13 Ulmus parvifolia Chinese Elm\n",
|
|
"14 Fraxinus pennsylvanica Green Ash\n",
|
|
"15 Zelkova serrata Japanese Zelkova\n",
|
|
"16 Cotinus sp. Smoke Tree\n",
|
|
"17 Acer ginnala Amur Maple\n",
|
|
"18 Amelanchier sp. Serviceberry\n",
|
|
"19 Acer campestre Hedge Maple\n",
|
|
"20 Gleditsia triacanthos var. inermis Honeylocust\n",
|
|
"21 Acer truncatum Shantung Maple\n",
|
|
"22 Celtis occidentalis Hackberry\n",
|
|
"23 Ginkgo biloba Ginkgo\n",
|
|
"24 Fraxinus americana White Ash\n",
|
|
"25 Carpinus caroliniana American Hornbeam\n",
|
|
"26 Eucommia ulmoides Hardy Rubber Tree\n",
|
|
"27 Quercus rubra Northern Red Oak\n",
|
|
"28 Tilia tomentosa Silver Linden\n",
|
|
"29 Prunus cerasifera Purpleleaf Plum\n",
|
|
"30 Syringa reticulata Japanese Tree Lilac\n",
|
|
"31 Quercus acutissima Sawtooth Oak\n",
|
|
"32 Nyssa sylvatica Black Gum\n",
|
|
"33 Lagerstroemia indica Crapemyrtle\n",
|
|
"34 Maackia amurensis Amur Maackia\n",
|
|
"35 Quercus imbricaria Shingle Oak\n",
|
|
"36 Gymnocladus dioicus Coffeetree\n",
|
|
"37 Pyrus calleryana Callery Pear\n",
|
|
"38 Quercus robur English Oak"
|
|
]
|
|
},
|
|
"execution_count": 37,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"%%bigquery --project {PROJECT_ID}\n",
|
|
"\n",
|
|
"SELECT\n",
|
|
" species_scientific_name,\n",
|
|
" species_common_name\n",
|
|
"FROM\n",
|
|
" `bigquery-public-data.new_york_trees.tree_species`\n",
|
|
"WHERE\n",
|
|
" AI.GENERATE_BOOL(\n",
|
|
" ('Is this tree species drought tolerant?', species_scientific_name),\n",
|
|
" connection_id => 'us.test_connection',\n",
|
|
" endpoint => 'gemini-2.5-flash').result = true"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "generate_double_md"
|
|
},
|
|
"source": [
|
|
"### Using `AI.GENERATE_DOUBLE`: Filtering for tree species with a minimum average lifespan\n",
|
|
"\n",
|
|
"With [`AI.GENERATE_DOUBLE`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate-double), we can get a numerical response of type `FLOAT64`, such as average lifespan in years of each tree species. Again, we'll use `bigquery-public-data.new_york_trees.tree_species` table, this time filtering for species that meet a minimum average lifespan requirement of 19 years."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 38,
|
|
"metadata": {
|
|
"id": "xT9vakyw1PZ2"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Query is running: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Downloading: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.google.colaboratory.intrinsic+json": {
|
|
"summary": "{\n \"name\": \"get_ipython()\",\n \"rows\": 57,\n \"fields\": [\n {\n \"column\": \"species_scientific_name\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 57,\n \"samples\": [\n \"Ostrya virginiana\",\n \"Cornus mas\",\n \"Acer rubrum\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"species_common_name\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 57,\n \"samples\": [\n \"American Hophornbeam\",\n \"Cornelian Cherry\",\n \"Red Maple\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}",
|
|
"type": "dataframe"
|
|
},
|
|
"text/html": [
|
|
"\n",
|
|
" <div id=\"df-8d77a177-5a21-4db3-8d8d-091ced40a645\" class=\"colab-df-container\">\n",
|
|
" <div>\n",
|
|
"<style scoped>\n",
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|
" vertical-align: middle;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe tbody tr th {\n",
|
|
" vertical-align: top;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe thead th {\n",
|
|
" text-align: right;\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: right;\">\n",
|
|
" <th></th>\n",
|
|
" <th>species_scientific_name</th>\n",
|
|
" <th>species_common_name</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <th>0</th>\n",
|
|
" <td>Ostrya virginiana</td>\n",
|
|
" <td>American Hophornbeam</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>1</th>\n",
|
|
" <td>Carpinus betulus</td>\n",
|
|
" <td>European Hornbeam</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>2</th>\n",
|
|
" <td>Metasequoia glyptostroboides</td>\n",
|
|
" <td>Dawn Redwood</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>3</th>\n",
|
|
" <td>Prunus serrulata 'Kwanzan'</td>\n",
|
|
" <td>Japanese Flowering Cherry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>4</th>\n",
|
|
" <td>Prunus x yedoensis</td>\n",
|
|
" <td>Yoshino Cherry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>5</th>\n",
|
|
" <td>Cornus mas</td>\n",
|
|
" <td>Cornelian Cherry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>6</th>\n",
|
|
" <td>Liriodendron tulipifera</td>\n",
|
|
" <td>Tulip Tree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>7</th>\n",
|
|
" <td>Cotinus sp.</td>\n",
|
|
" <td>Smoke Tree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>8</th>\n",
|
|
" <td>Acer ginnala</td>\n",
|
|
" <td>Amur Maple</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>9</th>\n",
|
|
" <td>Amelanchier sp.</td>\n",
|
|
" <td>Serviceberry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>10</th>\n",
|
|
" <td>Acer campestre</td>\n",
|
|
" <td>Hedge Maple</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>11</th>\n",
|
|
" <td>Gleditsia triacanthos var. inermis</td>\n",
|
|
" <td>Honeylocust</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>12</th>\n",
|
|
" <td>Tilia cordata</td>\n",
|
|
" <td>Littleleaf Linden</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>13</th>\n",
|
|
" <td>Platanus x acerifolia</td>\n",
|
|
" <td>London Plane</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>14</th>\n",
|
|
" <td>Prunus 'Okame'</td>\n",
|
|
" <td>Okame Cherry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>15</th>\n",
|
|
" <td>Ulmus parvifolia</td>\n",
|
|
" <td>Chinese Elm</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>16</th>\n",
|
|
" <td>Cercidiphyllum japonicum</td>\n",
|
|
" <td>Katsura Tree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>17</th>\n",
|
|
" <td>Fraxinus pennsylvanica</td>\n",
|
|
" <td>Green Ash</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>18</th>\n",
|
|
" <td>Zelkova serrata</td>\n",
|
|
" <td>Japanese Zelkova</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>19</th>\n",
|
|
" <td>Carpinus caroliniana</td>\n",
|
|
" <td>American Hornbeam</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>20</th>\n",
|
|
" <td>Eucommia ulmoides</td>\n",
|
|
" <td>Hardy Rubber Tree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>21</th>\n",
|
|
" <td>Quercus rubra</td>\n",
|
|
" <td>Northern Red Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>22</th>\n",
|
|
" <td>Tilia tomentosa</td>\n",
|
|
" <td>Silver Linden</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>23</th>\n",
|
|
" <td>Prunus cerasifera</td>\n",
|
|
" <td>Purpleleaf Plum</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>24</th>\n",
|
|
" <td>Syringa reticulata</td>\n",
|
|
" <td>Japanese Tree Lilac</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>25</th>\n",
|
|
" <td>Quercus acutissima</td>\n",
|
|
" <td>Sawtooth Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>26</th>\n",
|
|
" <td>Nyssa sylvatica</td>\n",
|
|
" <td>Black Gum</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>27</th>\n",
|
|
" <td>Acer truncatum</td>\n",
|
|
" <td>Shantung Maple</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>28</th>\n",
|
|
" <td>Celtis occidentalis</td>\n",
|
|
" <td>Hackberry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>29</th>\n",
|
|
" <td>Tilia americana</td>\n",
|
|
" <td>American Linden</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>30</th>\n",
|
|
" <td>Acer rubrum</td>\n",
|
|
" <td>Red Maple</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>31</th>\n",
|
|
" <td>Ginkgo biloba</td>\n",
|
|
" <td>Ginkgo</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>32</th>\n",
|
|
" <td>Fraxinus americana</td>\n",
|
|
" <td>White Ash</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>33</th>\n",
|
|
" <td>Malus sp.</td>\n",
|
|
" <td>Crabapple</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>34</th>\n",
|
|
" <td>Quercus bicolor</td>\n",
|
|
" <td>Swamp White Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>35</th>\n",
|
|
" <td>Quercus palustris</td>\n",
|
|
" <td>Pin Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>36</th>\n",
|
|
" <td>Crataegus sp.</td>\n",
|
|
" <td>Hawthorn</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>37</th>\n",
|
|
" <td>Styphnolobium japonicum</td>\n",
|
|
" <td>Scholar Tree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>38</th>\n",
|
|
" <td>Prunus virginiana 'Schubert'</td>\n",
|
|
" <td>Schubert Cherry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>39</th>\n",
|
|
" <td>Koelreuteria paniculata</td>\n",
|
|
" <td>Goldenraintree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>40</th>\n",
|
|
" <td>Corylus colurna</td>\n",
|
|
" <td>Turkish Filbert</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>41</th>\n",
|
|
" <td>Prunus sargentii</td>\n",
|
|
" <td>Sargent Cherry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>42</th>\n",
|
|
" <td>Cercis canadensis</td>\n",
|
|
" <td>Eastern Redbud</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>43</th>\n",
|
|
" <td>Quercus spp. 'Fastigiata'</td>\n",
|
|
" <td>Fastigiata Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>44</th>\n",
|
|
" <td>Liquidambar styraciflua</td>\n",
|
|
" <td>Sweetgum</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>45</th>\n",
|
|
" <td>Taxodium distichum</td>\n",
|
|
" <td>Baldcypress</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>46</th>\n",
|
|
" <td>Quercus phellos</td>\n",
|
|
" <td>Willow Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>47</th>\n",
|
|
" <td>Prunus padus</td>\n",
|
|
" <td>European Birdcherry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>48</th>\n",
|
|
" <td>Fraxinus 'Leprechaun'</td>\n",
|
|
" <td>Leprechaun Green Ash</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>49</th>\n",
|
|
" <td>Lagerstroemia indica</td>\n",
|
|
" <td>Crapemyrtle</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>50</th>\n",
|
|
" <td>Maackia amurensis</td>\n",
|
|
" <td>Amur Maackia</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>51</th>\n",
|
|
" <td>Quercus imbricaria</td>\n",
|
|
" <td>Shingle Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>52</th>\n",
|
|
" <td>Gymnocladus dioicus</td>\n",
|
|
" <td>Coffeetree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>53</th>\n",
|
|
" <td>Tilia x euchlora</td>\n",
|
|
" <td>Crimean Linden</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>54</th>\n",
|
|
" <td>Pyrus calleryana</td>\n",
|
|
" <td>Callery Pear</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>55</th>\n",
|
|
" <td>Quercus robur</td>\n",
|
|
" <td>English Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>56</th>\n",
|
|
" <td>Ulmus americana</td>\n",
|
|
" <td>American Elm</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>\n",
|
|
"</div>\n",
|
|
" <div class=\"colab-df-buttons\">\n",
|
|
"\n",
|
|
" <div class=\"colab-df-container\">\n",
|
|
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8d77a177-5a21-4db3-8d8d-091ced40a645')\"\n",
|
|
" title=\"Convert this dataframe to an interactive table.\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
|
|
" <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
|
|
" </svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
" <style>\n",
|
|
" .colab-df-container {\n",
|
|
" display:flex;\n",
|
|
" gap: 12px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert {\n",
|
|
" background-color: #E8F0FE;\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: #1967D2;\n",
|
|
" height: 32px;\n",
|
|
" padding: 0 0 0 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert:hover {\n",
|
|
" background-color: #E2EBFA;\n",
|
|
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: #174EA6;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-buttons div {\n",
|
|
" margin-bottom: 4px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert {\n",
|
|
" background-color: #3B4455;\n",
|
|
" fill: #D2E3FC;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert:hover {\n",
|
|
" background-color: #434B5C;\n",
|
|
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
|
|
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
|
|
" fill: #FFFFFF;\n",
|
|
" }\n",
|
|
" </style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" const buttonEl =\n",
|
|
" document.querySelector('#df-8d77a177-5a21-4db3-8d8d-091ced40a645 button.colab-df-convert');\n",
|
|
" buttonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
"\n",
|
|
" async function convertToInteractive(key) {\n",
|
|
" const element = document.querySelector('#df-8d77a177-5a21-4db3-8d8d-091ced40a645');\n",
|
|
" const dataTable =\n",
|
|
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
|
|
" [key], {});\n",
|
|
" if (!dataTable) return;\n",
|
|
"\n",
|
|
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
|
|
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
|
|
" + ' to learn more about interactive tables.';\n",
|
|
" element.innerHTML = '';\n",
|
|
" dataTable['output_type'] = 'display_data';\n",
|
|
" await google.colab.output.renderOutput(dataTable, element);\n",
|
|
" const docLink = document.createElement('div');\n",
|
|
" docLink.innerHTML = docLinkHtml;\n",
|
|
" element.appendChild(docLink);\n",
|
|
" }\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
"\n",
|
|
" <div id=\"df-c8dbc1f6-794f-48a4-abf3-bd06592ec418\">\n",
|
|
" <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-c8dbc1f6-794f-48a4-abf3-bd06592ec418')\"\n",
|
|
" title=\"Suggest charts\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
"<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
|
|
" width=\"24px\">\n",
|
|
" <g>\n",
|
|
" <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
|
|
" </g>\n",
|
|
"</svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
"<style>\n",
|
|
" .colab-df-quickchart {\n",
|
|
" --bg-color: #E8F0FE;\n",
|
|
" --fill-color: #1967D2;\n",
|
|
" --hover-bg-color: #E2EBFA;\n",
|
|
" --hover-fill-color: #174EA6;\n",
|
|
" --disabled-fill-color: #AAA;\n",
|
|
" --disabled-bg-color: #DDD;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-quickchart {\n",
|
|
" --bg-color: #3B4455;\n",
|
|
" --fill-color: #D2E3FC;\n",
|
|
" --hover-bg-color: #434B5C;\n",
|
|
" --hover-fill-color: #FFFFFF;\n",
|
|
" --disabled-bg-color: #3B4455;\n",
|
|
" --disabled-fill-color: #666;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart {\n",
|
|
" background-color: var(--bg-color);\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: var(--fill-color);\n",
|
|
" height: 32px;\n",
|
|
" padding: 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart:hover {\n",
|
|
" background-color: var(--hover-bg-color);\n",
|
|
" box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: var(--button-hover-fill-color);\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart-complete:disabled,\n",
|
|
" .colab-df-quickchart-complete:disabled:hover {\n",
|
|
" background-color: var(--disabled-bg-color);\n",
|
|
" fill: var(--disabled-fill-color);\n",
|
|
" box-shadow: none;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-spinner {\n",
|
|
" border: 2px solid var(--fill-color);\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" animation:\n",
|
|
" spin 1s steps(1) infinite;\n",
|
|
" }\n",
|
|
"\n",
|
|
" @keyframes spin {\n",
|
|
" 0% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 20% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 30% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 40% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 60% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 80% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 90% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" async function quickchart(key) {\n",
|
|
" const quickchartButtonEl =\n",
|
|
" document.querySelector('#' + key + ' button');\n",
|
|
" quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n",
|
|
" quickchartButtonEl.classList.add('colab-df-spinner');\n",
|
|
" try {\n",
|
|
" const charts = await google.colab.kernel.invokeFunction(\n",
|
|
" 'suggestCharts', [key], {});\n",
|
|
" } catch (error) {\n",
|
|
" console.error('Error during call to suggestCharts:', error);\n",
|
|
" }\n",
|
|
" quickchartButtonEl.classList.remove('colab-df-spinner');\n",
|
|
" quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
|
|
" }\n",
|
|
" (() => {\n",
|
|
" let quickchartButtonEl =\n",
|
|
" document.querySelector('#df-c8dbc1f6-794f-48a4-abf3-bd06592ec418 button');\n",
|
|
" quickchartButtonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
" })();\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
" </div>\n",
|
|
" </div>\n"
|
|
],
|
|
"text/plain": [
|
|
" species_scientific_name species_common_name\n",
|
|
"0 Ostrya virginiana American Hophornbeam\n",
|
|
"1 Carpinus betulus European Hornbeam\n",
|
|
"2 Metasequoia glyptostroboides Dawn Redwood\n",
|
|
"3 Prunus serrulata 'Kwanzan' Japanese Flowering Cherry\n",
|
|
"4 Prunus x yedoensis Yoshino Cherry\n",
|
|
"5 Cornus mas Cornelian Cherry\n",
|
|
"6 Liriodendron tulipifera Tulip Tree\n",
|
|
"7 Cotinus sp. Smoke Tree\n",
|
|
"8 Acer ginnala Amur Maple\n",
|
|
"9 Amelanchier sp. Serviceberry\n",
|
|
"10 Acer campestre Hedge Maple\n",
|
|
"11 Gleditsia triacanthos var. inermis Honeylocust\n",
|
|
"12 Tilia cordata Littleleaf Linden\n",
|
|
"13 Platanus x acerifolia London Plane\n",
|
|
"14 Prunus 'Okame' Okame Cherry\n",
|
|
"15 Ulmus parvifolia Chinese Elm\n",
|
|
"16 Cercidiphyllum japonicum Katsura Tree\n",
|
|
"17 Fraxinus pennsylvanica Green Ash\n",
|
|
"18 Zelkova serrata Japanese Zelkova\n",
|
|
"19 Carpinus caroliniana American Hornbeam\n",
|
|
"20 Eucommia ulmoides Hardy Rubber Tree\n",
|
|
"21 Quercus rubra Northern Red Oak\n",
|
|
"22 Tilia tomentosa Silver Linden\n",
|
|
"23 Prunus cerasifera Purpleleaf Plum\n",
|
|
"24 Syringa reticulata Japanese Tree Lilac\n",
|
|
"25 Quercus acutissima Sawtooth Oak\n",
|
|
"26 Nyssa sylvatica Black Gum\n",
|
|
"27 Acer truncatum Shantung Maple\n",
|
|
"28 Celtis occidentalis Hackberry\n",
|
|
"29 Tilia americana American Linden\n",
|
|
"30 Acer rubrum Red Maple\n",
|
|
"31 Ginkgo biloba Ginkgo\n",
|
|
"32 Fraxinus americana White Ash\n",
|
|
"33 Malus sp. Crabapple\n",
|
|
"34 Quercus bicolor Swamp White Oak\n",
|
|
"35 Quercus palustris Pin Oak\n",
|
|
"36 Crataegus sp. Hawthorn\n",
|
|
"37 Styphnolobium japonicum Scholar Tree\n",
|
|
"38 Prunus virginiana 'Schubert' Schubert Cherry\n",
|
|
"39 Koelreuteria paniculata Goldenraintree\n",
|
|
"40 Corylus colurna Turkish Filbert\n",
|
|
"41 Prunus sargentii Sargent Cherry\n",
|
|
"42 Cercis canadensis Eastern Redbud\n",
|
|
"43 Quercus spp. 'Fastigiata' Fastigiata Oak\n",
|
|
"44 Liquidambar styraciflua Sweetgum\n",
|
|
"45 Taxodium distichum Baldcypress\n",
|
|
"46 Quercus phellos Willow Oak\n",
|
|
"47 Prunus padus European Birdcherry\n",
|
|
"48 Fraxinus 'Leprechaun' Leprechaun Green Ash\n",
|
|
"49 Lagerstroemia indica Crapemyrtle\n",
|
|
"50 Maackia amurensis Amur Maackia\n",
|
|
"51 Quercus imbricaria Shingle Oak\n",
|
|
"52 Gymnocladus dioicus Coffeetree\n",
|
|
"53 Tilia x euchlora Crimean Linden\n",
|
|
"54 Pyrus calleryana Callery Pear\n",
|
|
"55 Quercus robur English Oak\n",
|
|
"56 Ulmus americana American Elm"
|
|
]
|
|
},
|
|
"execution_count": 38,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"%%bigquery --project {PROJECT_ID}\n",
|
|
"\n",
|
|
"SELECT\n",
|
|
" species_scientific_name,\n",
|
|
" species_common_name\n",
|
|
"FROM\n",
|
|
" `bigquery-public-data.new_york_trees.tree_species`\n",
|
|
"WHERE\n",
|
|
" AI.GENERATE_INT(\n",
|
|
" ('What is the average life in years of this tree species?', species_scientific_name),\n",
|
|
" connection_id => 'us.test_connection',\n",
|
|
" endpoint => 'gemini-2.5-flash').result > 19"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "generate_int_md"
|
|
},
|
|
"source": [
|
|
"### Using `AI.GENERATE_INT`: Filtering for tree species of specific average mature height\n",
|
|
"\n",
|
|
"[`AI.GENERATE_INT`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate-int) is similar to [`AI.GENERATE_DOUBLE`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate-double), except that it returns data of type `INT64`, instead of `FLOAT64`. Let's use it to return tree species with an average mature height of 10-20 meters.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 39,
|
|
"metadata": {
|
|
"id": "generate_int_code"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Query is running: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Downloading: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.google.colaboratory.intrinsic+json": {
|
|
"summary": "{\n \"name\": \"get_ipython()\",\n \"rows\": 26,\n \"fields\": [\n {\n \"column\": \"species_scientific_name\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 26,\n \"samples\": [\n \"Quercus bicolor\",\n \"Quercus phellos\",\n \"Koelreuteria paniculata\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"species_common_name\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 26,\n \"samples\": [\n \"Swamp White Oak\",\n \"Willow Oak\",\n \"Goldenraintree\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}",
|
|
"type": "dataframe"
|
|
},
|
|
"text/html": [
|
|
"\n",
|
|
" <div id=\"df-3d4bcbf1-896b-4f5f-9970-77f2764e4617\" class=\"colab-df-container\">\n",
|
|
" <div>\n",
|
|
"<style scoped>\n",
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|
" vertical-align: middle;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe tbody tr th {\n",
|
|
" vertical-align: top;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe thead th {\n",
|
|
" text-align: right;\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: right;\">\n",
|
|
" <th></th>\n",
|
|
" <th>species_scientific_name</th>\n",
|
|
" <th>species_common_name</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <th>0</th>\n",
|
|
" <td>Koelreuteria paniculata</td>\n",
|
|
" <td>Goldenraintree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>1</th>\n",
|
|
" <td>Corylus colurna</td>\n",
|
|
" <td>Turkish Filbert</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>2</th>\n",
|
|
" <td>Ostrya virginiana</td>\n",
|
|
" <td>American Hophornbeam</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>3</th>\n",
|
|
" <td>Carpinus betulus</td>\n",
|
|
" <td>European Hornbeam</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>4</th>\n",
|
|
" <td>Quercus spp. 'Fastigiata'</td>\n",
|
|
" <td>Fastigiata Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>5</th>\n",
|
|
" <td>Ulmus parvifolia</td>\n",
|
|
" <td>Chinese Elm</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>6</th>\n",
|
|
" <td>Cercidiphyllum japonicum</td>\n",
|
|
" <td>Katsura Tree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>7</th>\n",
|
|
" <td>Fraxinus pennsylvanica</td>\n",
|
|
" <td>Green Ash</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>8</th>\n",
|
|
" <td>Quercus bicolor</td>\n",
|
|
" <td>Swamp White Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>9</th>\n",
|
|
" <td>Quercus palustris</td>\n",
|
|
" <td>Pin Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>10</th>\n",
|
|
" <td>Crataegus sp.</td>\n",
|
|
" <td>Hawthorn</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>11</th>\n",
|
|
" <td>Styphnolobium japonicum</td>\n",
|
|
" <td>Scholar Tree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>12</th>\n",
|
|
" <td>Prunus x yedoensis</td>\n",
|
|
" <td>Yoshino Cherry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>13</th>\n",
|
|
" <td>Acer truncatum</td>\n",
|
|
" <td>Shantung Maple</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>14</th>\n",
|
|
" <td>Celtis occidentalis</td>\n",
|
|
" <td>Hackberry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>15</th>\n",
|
|
" <td>Acer rubrum</td>\n",
|
|
" <td>Red Maple</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>16</th>\n",
|
|
" <td>Quercus phellos</td>\n",
|
|
" <td>Willow Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>17</th>\n",
|
|
" <td>Prunus padus</td>\n",
|
|
" <td>European Birdcherry</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>18</th>\n",
|
|
" <td>Quercus imbricaria</td>\n",
|
|
" <td>Shingle Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>19</th>\n",
|
|
" <td>Tilia x euchlora</td>\n",
|
|
" <td>Crimean Linden</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>20</th>\n",
|
|
" <td>Eucommia ulmoides</td>\n",
|
|
" <td>Hardy Rubber Tree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>21</th>\n",
|
|
" <td>Quercus acutissima</td>\n",
|
|
" <td>Sawtooth Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>22</th>\n",
|
|
" <td>Nyssa sylvatica</td>\n",
|
|
" <td>Black Gum</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>23</th>\n",
|
|
" <td>Acer campestre</td>\n",
|
|
" <td>Hedge Maple</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>24</th>\n",
|
|
" <td>Gleditsia triacanthos var. inermis</td>\n",
|
|
" <td>Honeylocust</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>25</th>\n",
|
|
" <td>Tilia cordata</td>\n",
|
|
" <td>Littleleaf Linden</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>\n",
|
|
"</div>\n",
|
|
" <div class=\"colab-df-buttons\">\n",
|
|
"\n",
|
|
" <div class=\"colab-df-container\">\n",
|
|
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3d4bcbf1-896b-4f5f-9970-77f2764e4617')\"\n",
|
|
" title=\"Convert this dataframe to an interactive table.\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
|
|
" <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
|
|
" </svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
" <style>\n",
|
|
" .colab-df-container {\n",
|
|
" display:flex;\n",
|
|
" gap: 12px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert {\n",
|
|
" background-color: #E8F0FE;\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: #1967D2;\n",
|
|
" height: 32px;\n",
|
|
" padding: 0 0 0 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert:hover {\n",
|
|
" background-color: #E2EBFA;\n",
|
|
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: #174EA6;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-buttons div {\n",
|
|
" margin-bottom: 4px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert {\n",
|
|
" background-color: #3B4455;\n",
|
|
" fill: #D2E3FC;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert:hover {\n",
|
|
" background-color: #434B5C;\n",
|
|
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
|
|
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
|
|
" fill: #FFFFFF;\n",
|
|
" }\n",
|
|
" </style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" const buttonEl =\n",
|
|
" document.querySelector('#df-3d4bcbf1-896b-4f5f-9970-77f2764e4617 button.colab-df-convert');\n",
|
|
" buttonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
"\n",
|
|
" async function convertToInteractive(key) {\n",
|
|
" const element = document.querySelector('#df-3d4bcbf1-896b-4f5f-9970-77f2764e4617');\n",
|
|
" const dataTable =\n",
|
|
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
|
|
" [key], {});\n",
|
|
" if (!dataTable) return;\n",
|
|
"\n",
|
|
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
|
|
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
|
|
" + ' to learn more about interactive tables.';\n",
|
|
" element.innerHTML = '';\n",
|
|
" dataTable['output_type'] = 'display_data';\n",
|
|
" await google.colab.output.renderOutput(dataTable, element);\n",
|
|
" const docLink = document.createElement('div');\n",
|
|
" docLink.innerHTML = docLinkHtml;\n",
|
|
" element.appendChild(docLink);\n",
|
|
" }\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
"\n",
|
|
" <div id=\"df-eed80fdf-c7c6-403c-984c-8f06506c7ab1\">\n",
|
|
" <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-eed80fdf-c7c6-403c-984c-8f06506c7ab1')\"\n",
|
|
" title=\"Suggest charts\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
"<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
|
|
" width=\"24px\">\n",
|
|
" <g>\n",
|
|
" <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
|
|
" </g>\n",
|
|
"</svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
"<style>\n",
|
|
" .colab-df-quickchart {\n",
|
|
" --bg-color: #E8F0FE;\n",
|
|
" --fill-color: #1967D2;\n",
|
|
" --hover-bg-color: #E2EBFA;\n",
|
|
" --hover-fill-color: #174EA6;\n",
|
|
" --disabled-fill-color: #AAA;\n",
|
|
" --disabled-bg-color: #DDD;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-quickchart {\n",
|
|
" --bg-color: #3B4455;\n",
|
|
" --fill-color: #D2E3FC;\n",
|
|
" --hover-bg-color: #434B5C;\n",
|
|
" --hover-fill-color: #FFFFFF;\n",
|
|
" --disabled-bg-color: #3B4455;\n",
|
|
" --disabled-fill-color: #666;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart {\n",
|
|
" background-color: var(--bg-color);\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: var(--fill-color);\n",
|
|
" height: 32px;\n",
|
|
" padding: 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart:hover {\n",
|
|
" background-color: var(--hover-bg-color);\n",
|
|
" box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: var(--button-hover-fill-color);\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart-complete:disabled,\n",
|
|
" .colab-df-quickchart-complete:disabled:hover {\n",
|
|
" background-color: var(--disabled-bg-color);\n",
|
|
" fill: var(--disabled-fill-color);\n",
|
|
" box-shadow: none;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-spinner {\n",
|
|
" border: 2px solid var(--fill-color);\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" animation:\n",
|
|
" spin 1s steps(1) infinite;\n",
|
|
" }\n",
|
|
"\n",
|
|
" @keyframes spin {\n",
|
|
" 0% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 20% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 30% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 40% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 60% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 80% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 90% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" async function quickchart(key) {\n",
|
|
" const quickchartButtonEl =\n",
|
|
" document.querySelector('#' + key + ' button');\n",
|
|
" quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n",
|
|
" quickchartButtonEl.classList.add('colab-df-spinner');\n",
|
|
" try {\n",
|
|
" const charts = await google.colab.kernel.invokeFunction(\n",
|
|
" 'suggestCharts', [key], {});\n",
|
|
" } catch (error) {\n",
|
|
" console.error('Error during call to suggestCharts:', error);\n",
|
|
" }\n",
|
|
" quickchartButtonEl.classList.remove('colab-df-spinner');\n",
|
|
" quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
|
|
" }\n",
|
|
" (() => {\n",
|
|
" let quickchartButtonEl =\n",
|
|
" document.querySelector('#df-eed80fdf-c7c6-403c-984c-8f06506c7ab1 button');\n",
|
|
" quickchartButtonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
" })();\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
" </div>\n",
|
|
" </div>\n"
|
|
],
|
|
"text/plain": [
|
|
" species_scientific_name species_common_name\n",
|
|
"0 Koelreuteria paniculata Goldenraintree\n",
|
|
"1 Corylus colurna Turkish Filbert\n",
|
|
"2 Ostrya virginiana American Hophornbeam\n",
|
|
"3 Carpinus betulus European Hornbeam\n",
|
|
"4 Quercus spp. 'Fastigiata' Fastigiata Oak\n",
|
|
"5 Ulmus parvifolia Chinese Elm\n",
|
|
"6 Cercidiphyllum japonicum Katsura Tree\n",
|
|
"7 Fraxinus pennsylvanica Green Ash\n",
|
|
"8 Quercus bicolor Swamp White Oak\n",
|
|
"9 Quercus palustris Pin Oak\n",
|
|
"10 Crataegus sp. Hawthorn\n",
|
|
"11 Styphnolobium japonicum Scholar Tree\n",
|
|
"12 Prunus x yedoensis Yoshino Cherry\n",
|
|
"13 Acer truncatum Shantung Maple\n",
|
|
"14 Celtis occidentalis Hackberry\n",
|
|
"15 Acer rubrum Red Maple\n",
|
|
"16 Quercus phellos Willow Oak\n",
|
|
"17 Prunus padus European Birdcherry\n",
|
|
"18 Quercus imbricaria Shingle Oak\n",
|
|
"19 Tilia x euchlora Crimean Linden\n",
|
|
"20 Eucommia ulmoides Hardy Rubber Tree\n",
|
|
"21 Quercus acutissima Sawtooth Oak\n",
|
|
"22 Nyssa sylvatica Black Gum\n",
|
|
"23 Acer campestre Hedge Maple\n",
|
|
"24 Gleditsia triacanthos var. inermis Honeylocust\n",
|
|
"25 Tilia cordata Littleleaf Linden"
|
|
]
|
|
},
|
|
"execution_count": 39,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"%%bigquery --project {PROJECT_ID}\n",
|
|
"\n",
|
|
"SELECT\n",
|
|
" species_scientific_name,\n",
|
|
" species_common_name\n",
|
|
"FROM\n",
|
|
" `bigquery-public-data.new_york_trees.tree_species`\n",
|
|
"WHERE\n",
|
|
" AI.GENERATE_DOUBLE(\n",
|
|
" ('What is the average height in meters of this tree species at full maturity?', species_scientific_name),\n",
|
|
" connection_id => 'us.test_connection',\n",
|
|
" endpoint => 'gemini-2.5-flash').result BETWEEN 10 AND 20"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "LXSfuNKs521-"
|
|
},
|
|
"source": [
|
|
"You can also combine `AI.GENERATE_DOUBLE`, `AI.GENERATE_INT`, and `AI.GENERATE_BOOL` scalar functions within one query. Here's an example combining the AI filters of the last three queries into one `WHERE` clause:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 40,
|
|
"metadata": {
|
|
"id": "9mpFmVGG5yrG"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Query is running: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Downloading: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.google.colaboratory.intrinsic+json": {
|
|
"summary": "{\n \"name\": \"get_ipython()\",\n \"rows\": 12,\n \"fields\": [\n {\n \"column\": \"species_scientific_name\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 12,\n \"samples\": [\n \"Tilia x euchlora\",\n \"Gymnocladus dioicus\",\n \"Corylus colurna\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"species_common_name\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 12,\n \"samples\": [\n \"Crimean Linden\",\n \"Coffeetree\",\n \"Turkish Filbert\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}",
|
|
"type": "dataframe"
|
|
},
|
|
"text/html": [
|
|
"\n",
|
|
" <div id=\"df-728de432-fa93-41ef-a13a-56a06600f15b\" class=\"colab-df-container\">\n",
|
|
" <div>\n",
|
|
"<style scoped>\n",
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|
" vertical-align: middle;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe tbody tr th {\n",
|
|
" vertical-align: top;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe thead th {\n",
|
|
" text-align: right;\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: right;\">\n",
|
|
" <th></th>\n",
|
|
" <th>species_scientific_name</th>\n",
|
|
" <th>species_common_name</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <th>0</th>\n",
|
|
" <td>Corylus colurna</td>\n",
|
|
" <td>Turkish Filbert</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>1</th>\n",
|
|
" <td>Ostrya virginiana</td>\n",
|
|
" <td>American Hophornbeam</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>2</th>\n",
|
|
" <td>Ulmus parvifolia</td>\n",
|
|
" <td>Chinese Elm</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>3</th>\n",
|
|
" <td>Fraxinus pennsylvanica</td>\n",
|
|
" <td>Green Ash</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>4</th>\n",
|
|
" <td>Styphnolobium japonicum</td>\n",
|
|
" <td>Scholar Tree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>5</th>\n",
|
|
" <td>Eucommia ulmoides</td>\n",
|
|
" <td>Hardy Rubber Tree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>6</th>\n",
|
|
" <td>Quercus spp. 'Fastigiata'</td>\n",
|
|
" <td>Fastigiata Oak</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>7</th>\n",
|
|
" <td>Acer campestre</td>\n",
|
|
" <td>Hedge Maple</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>8</th>\n",
|
|
" <td>Gleditsia triacanthos var. inermis</td>\n",
|
|
" <td>Honeylocust</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>9</th>\n",
|
|
" <td>Gymnocladus dioicus</td>\n",
|
|
" <td>Coffeetree</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>10</th>\n",
|
|
" <td>Tilia x euchlora</td>\n",
|
|
" <td>Crimean Linden</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>11</th>\n",
|
|
" <td>Pyrus calleryana</td>\n",
|
|
" <td>Callery Pear</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>\n",
|
|
"</div>\n",
|
|
" <div class=\"colab-df-buttons\">\n",
|
|
"\n",
|
|
" <div class=\"colab-df-container\">\n",
|
|
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-728de432-fa93-41ef-a13a-56a06600f15b')\"\n",
|
|
" title=\"Convert this dataframe to an interactive table.\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
|
|
" <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
|
|
" </svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
" <style>\n",
|
|
" .colab-df-container {\n",
|
|
" display:flex;\n",
|
|
" gap: 12px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert {\n",
|
|
" background-color: #E8F0FE;\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: #1967D2;\n",
|
|
" height: 32px;\n",
|
|
" padding: 0 0 0 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert:hover {\n",
|
|
" background-color: #E2EBFA;\n",
|
|
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: #174EA6;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-buttons div {\n",
|
|
" margin-bottom: 4px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert {\n",
|
|
" background-color: #3B4455;\n",
|
|
" fill: #D2E3FC;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert:hover {\n",
|
|
" background-color: #434B5C;\n",
|
|
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
|
|
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
|
|
" fill: #FFFFFF;\n",
|
|
" }\n",
|
|
" </style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" const buttonEl =\n",
|
|
" document.querySelector('#df-728de432-fa93-41ef-a13a-56a06600f15b button.colab-df-convert');\n",
|
|
" buttonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
"\n",
|
|
" async function convertToInteractive(key) {\n",
|
|
" const element = document.querySelector('#df-728de432-fa93-41ef-a13a-56a06600f15b');\n",
|
|
" const dataTable =\n",
|
|
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
|
|
" [key], {});\n",
|
|
" if (!dataTable) return;\n",
|
|
"\n",
|
|
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
|
|
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
|
|
" + ' to learn more about interactive tables.';\n",
|
|
" element.innerHTML = '';\n",
|
|
" dataTable['output_type'] = 'display_data';\n",
|
|
" await google.colab.output.renderOutput(dataTable, element);\n",
|
|
" const docLink = document.createElement('div');\n",
|
|
" docLink.innerHTML = docLinkHtml;\n",
|
|
" element.appendChild(docLink);\n",
|
|
" }\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
"\n",
|
|
" <div id=\"df-e51febcc-faab-4602-9273-f0f46761bab3\">\n",
|
|
" <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-e51febcc-faab-4602-9273-f0f46761bab3')\"\n",
|
|
" title=\"Suggest charts\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
"<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
|
|
" width=\"24px\">\n",
|
|
" <g>\n",
|
|
" <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
|
|
" </g>\n",
|
|
"</svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
"<style>\n",
|
|
" .colab-df-quickchart {\n",
|
|
" --bg-color: #E8F0FE;\n",
|
|
" --fill-color: #1967D2;\n",
|
|
" --hover-bg-color: #E2EBFA;\n",
|
|
" --hover-fill-color: #174EA6;\n",
|
|
" --disabled-fill-color: #AAA;\n",
|
|
" --disabled-bg-color: #DDD;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-quickchart {\n",
|
|
" --bg-color: #3B4455;\n",
|
|
" --fill-color: #D2E3FC;\n",
|
|
" --hover-bg-color: #434B5C;\n",
|
|
" --hover-fill-color: #FFFFFF;\n",
|
|
" --disabled-bg-color: #3B4455;\n",
|
|
" --disabled-fill-color: #666;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart {\n",
|
|
" background-color: var(--bg-color);\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: var(--fill-color);\n",
|
|
" height: 32px;\n",
|
|
" padding: 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart:hover {\n",
|
|
" background-color: var(--hover-bg-color);\n",
|
|
" box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: var(--button-hover-fill-color);\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart-complete:disabled,\n",
|
|
" .colab-df-quickchart-complete:disabled:hover {\n",
|
|
" background-color: var(--disabled-bg-color);\n",
|
|
" fill: var(--disabled-fill-color);\n",
|
|
" box-shadow: none;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-spinner {\n",
|
|
" border: 2px solid var(--fill-color);\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" animation:\n",
|
|
" spin 1s steps(1) infinite;\n",
|
|
" }\n",
|
|
"\n",
|
|
" @keyframes spin {\n",
|
|
" 0% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 20% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 30% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 40% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 60% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 80% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 90% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" async function quickchart(key) {\n",
|
|
" const quickchartButtonEl =\n",
|
|
" document.querySelector('#' + key + ' button');\n",
|
|
" quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n",
|
|
" quickchartButtonEl.classList.add('colab-df-spinner');\n",
|
|
" try {\n",
|
|
" const charts = await google.colab.kernel.invokeFunction(\n",
|
|
" 'suggestCharts', [key], {});\n",
|
|
" } catch (error) {\n",
|
|
" console.error('Error during call to suggestCharts:', error);\n",
|
|
" }\n",
|
|
" quickchartButtonEl.classList.remove('colab-df-spinner');\n",
|
|
" quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
|
|
" }\n",
|
|
" (() => {\n",
|
|
" let quickchartButtonEl =\n",
|
|
" document.querySelector('#df-e51febcc-faab-4602-9273-f0f46761bab3 button');\n",
|
|
" quickchartButtonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
" })();\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
" </div>\n",
|
|
" </div>\n"
|
|
],
|
|
"text/plain": [
|
|
" species_scientific_name species_common_name\n",
|
|
"0 Corylus colurna Turkish Filbert\n",
|
|
"1 Ostrya virginiana American Hophornbeam\n",
|
|
"2 Ulmus parvifolia Chinese Elm\n",
|
|
"3 Fraxinus pennsylvanica Green Ash\n",
|
|
"4 Styphnolobium japonicum Scholar Tree\n",
|
|
"5 Eucommia ulmoides Hardy Rubber Tree\n",
|
|
"6 Quercus spp. 'Fastigiata' Fastigiata Oak\n",
|
|
"7 Acer campestre Hedge Maple\n",
|
|
"8 Gleditsia triacanthos var. inermis Honeylocust\n",
|
|
"9 Gymnocladus dioicus Coffeetree\n",
|
|
"10 Tilia x euchlora Crimean Linden\n",
|
|
"11 Pyrus calleryana Callery Pear"
|
|
]
|
|
},
|
|
"execution_count": 40,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"%%bigquery --project {PROJECT_ID}\n",
|
|
"\n",
|
|
"SELECT\n",
|
|
" species_scientific_name,\n",
|
|
" species_common_name\n",
|
|
"FROM\n",
|
|
" `bigquery-public-data.new_york_trees.tree_species`\n",
|
|
"WHERE\n",
|
|
" AI.GENERATE_DOUBLE(\n",
|
|
" ('What is the average height in meters of this tree species at full maturity?', species_scientific_name),\n",
|
|
" connection_id => 'us.test_connection',\n",
|
|
" endpoint => 'gemini-2.5-flash').result BETWEEN 10 AND 20 AND\n",
|
|
" AI.GENERATE_INT(\n",
|
|
" ('What is the average life in years of this tree species?', species_scientific_name),\n",
|
|
" connection_id => 'us.test_connection',\n",
|
|
" endpoint => 'gemini-2.5-flash').result > 19 AND\n",
|
|
" AI.GENERATE_BOOL(\n",
|
|
" ('Is this tree species drought tolerant?', species_scientific_name),\n",
|
|
" connection_id => 'us.test_connection',\n",
|
|
" endpoint => 'gemini-2.5-flash').result = true"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "forecast_md"
|
|
},
|
|
"source": [
|
|
"---\n",
|
|
"\n",
|
|
"## Forecast time series with `AI.FORECAST`\n",
|
|
"\n",
|
|
"The Google Research [`TimesFM`](https://cloud.google.com/bigquery/docs/timesfm-model) model is a foundation model for time-series forecasting that has been pre-trained on billions of time-points from many real-world datasets, so you can apply it to new forecasting datasets across many domains.\n",
|
|
"\n",
|
|
"The [`AI.FORECAST`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-forecast) function in BigQuery leverages the model to provide high-quality time series forecasting without the need to create and train your own model.\n",
|
|
"\n",
|
|
"For this example, we'll use the `bigquery-public-data.san_francisco_bikeshare.bikeshare_trips` data to forecast hourly bikeshare trips by subscriber type."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 41,
|
|
"metadata": {
|
|
"id": "forecast_code"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Query is running: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Downloading: 0%| |"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"%%bigquery forecast_results --project {PROJECT_ID}\n",
|
|
"\n",
|
|
"SELECT *\n",
|
|
"FROM\n",
|
|
" AI.FORECAST(\n",
|
|
" (\n",
|
|
" SELECT\n",
|
|
" TIMESTAMP_TRUNC(start_date, HOUR) as trip_hour,\n",
|
|
" subscriber_type,\n",
|
|
" COUNT(*) as num_trips\n",
|
|
" FROM `bigquery-public-data.san_francisco_bikeshare.bikeshare_trips`\n",
|
|
" WHERE start_date >= TIMESTAMP('2018-01-01')\n",
|
|
" GROUP BY TIMESTAMP_TRUNC(start_date, HOUR), subscriber_type\n",
|
|
" ),\n",
|
|
" horizon => 720,\n",
|
|
" confidence_level => 0.95,\n",
|
|
" timestamp_col => 'trip_hour',\n",
|
|
" data_col => 'num_trips',\n",
|
|
" id_cols => ['subscriber_type']);"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "forecast_text_md"
|
|
},
|
|
"source": [
|
|
"The results of this query were saved as a pandas DataFrame. Running the next cell will display the DataFrame, including the forecasted number of trips by subscriber type (annually/monthly subscriber, or short-term customer), for the next 720 hours (30 days), along with the prediction interval."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 42,
|
|
"metadata": {
|
|
"id": "9rkQpI4KQpB5"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"application/vnd.google.colaboratory.intrinsic+json": {
|
|
"summary": "{\n \"name\": \"forecast_results\",\n \"rows\": 1440,\n \"fields\": [\n {\n \"column\": \"subscriber_type\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 2,\n \"samples\": [\n \"Customer\",\n \"Subscriber\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"forecast_timestamp\",\n \"properties\": {\n \"dtype\": \"date\",\n \"min\": \"2018-05-01 00:00:00+00:00\",\n \"max\": \"2018-05-30 23:00:00+00:00\",\n \"num_unique_values\": 720,\n \"samples\": [\n \"2018-05-15 04:00:00+00:00\",\n \"2018-05-13 02:00:00+00:00\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"forecast_value\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 144.02612265291813,\n \"min\": -55.49763488769531,\n \"max\": 751.3600463867188,\n \"num_unique_values\": 1440,\n \"samples\": [\n -6.396148681640625,\n -29.988113403320312\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"confidence_level\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 2.5432939679420995e-14,\n \"min\": 0.95,\n \"max\": 0.95,\n \"num_unique_values\": 1,\n \"samples\": [\n 0.95\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"prediction_interval_lower_bound\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 124.33868160606646,\n \"min\": -361.1392152019482,\n \"max\": 505.0838247829274,\n \"num_unique_values\": 1440,\n \"samples\": [\n -102.71066180632344\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"prediction_interval_upper_bound\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 221.76368830950986,\n \"min\": -9.185602128762568,\n \"max\": 1198.4913343334024,\n \"num_unique_values\": 1440,\n \"samples\": [\n 89.9183644430422\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"ai_forecast_status\",\n \"properties\": {\n \"dtype\": \"object\",\n \"num_unique_values\": 1,\n \"samples\": [\n \"\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}",
|
|
"type": "dataframe",
|
|
"variable_name": "forecast_results"
|
|
},
|
|
"text/html": [
|
|
"\n",
|
|
" <div id=\"df-356ee8ae-7a5b-4619-a566-6c623c9a5bdd\" class=\"colab-df-container\">\n",
|
|
" <div>\n",
|
|
"<style scoped>\n",
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|
" vertical-align: middle;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe tbody tr th {\n",
|
|
" vertical-align: top;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe thead th {\n",
|
|
" text-align: right;\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: right;\">\n",
|
|
" <th></th>\n",
|
|
" <th>subscriber_type</th>\n",
|
|
" <th>forecast_timestamp</th>\n",
|
|
" <th>forecast_value</th>\n",
|
|
" <th>confidence_level</th>\n",
|
|
" <th>prediction_interval_lower_bound</th>\n",
|
|
" <th>prediction_interval_upper_bound</th>\n",
|
|
" <th>ai_forecast_status</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <th>0</th>\n",
|
|
" <td>Subscriber</td>\n",
|
|
" <td>2018-05-01 00:00:00+00:00</td>\n",
|
|
" <td>0.604919</td>\n",
|
|
" <td>0.95</td>\n",
|
|
" <td>-9.780785</td>\n",
|
|
" <td>10.990624</td>\n",
|
|
" <td></td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>1</th>\n",
|
|
" <td>Subscriber</td>\n",
|
|
" <td>2018-05-01 01:00:00+00:00</td>\n",
|
|
" <td>21.518158</td>\n",
|
|
" <td>0.95</td>\n",
|
|
" <td>-5.284099</td>\n",
|
|
" <td>48.320415</td>\n",
|
|
" <td></td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>2</th>\n",
|
|
" <td>Subscriber</td>\n",
|
|
" <td>2018-05-01 02:00:00+00:00</td>\n",
|
|
" <td>-12.472580</td>\n",
|
|
" <td>0.95</td>\n",
|
|
" <td>-32.132971</td>\n",
|
|
" <td>7.187812</td>\n",
|
|
" <td></td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>3</th>\n",
|
|
" <td>Subscriber</td>\n",
|
|
" <td>2018-05-01 03:00:00+00:00</td>\n",
|
|
" <td>-8.919159</td>\n",
|
|
" <td>0.95</td>\n",
|
|
" <td>-59.022948</td>\n",
|
|
" <td>41.184630</td>\n",
|
|
" <td></td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>4</th>\n",
|
|
" <td>Subscriber</td>\n",
|
|
" <td>2018-05-01 04:00:00+00:00</td>\n",
|
|
" <td>12.511322</td>\n",
|
|
" <td>0.95</td>\n",
|
|
" <td>-32.005432</td>\n",
|
|
" <td>57.028076</td>\n",
|
|
" <td></td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>\n",
|
|
"</div>\n",
|
|
" <div class=\"colab-df-buttons\">\n",
|
|
"\n",
|
|
" <div class=\"colab-df-container\">\n",
|
|
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-356ee8ae-7a5b-4619-a566-6c623c9a5bdd')\"\n",
|
|
" title=\"Convert this dataframe to an interactive table.\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
|
|
" <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
|
|
" </svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
" <style>\n",
|
|
" .colab-df-container {\n",
|
|
" display:flex;\n",
|
|
" gap: 12px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert {\n",
|
|
" background-color: #E8F0FE;\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: #1967D2;\n",
|
|
" height: 32px;\n",
|
|
" padding: 0 0 0 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-convert:hover {\n",
|
|
" background-color: #E2EBFA;\n",
|
|
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: #174EA6;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-buttons div {\n",
|
|
" margin-bottom: 4px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert {\n",
|
|
" background-color: #3B4455;\n",
|
|
" fill: #D2E3FC;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-convert:hover {\n",
|
|
" background-color: #434B5C;\n",
|
|
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
|
|
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
|
|
" fill: #FFFFFF;\n",
|
|
" }\n",
|
|
" </style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" const buttonEl =\n",
|
|
" document.querySelector('#df-356ee8ae-7a5b-4619-a566-6c623c9a5bdd button.colab-df-convert');\n",
|
|
" buttonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
"\n",
|
|
" async function convertToInteractive(key) {\n",
|
|
" const element = document.querySelector('#df-356ee8ae-7a5b-4619-a566-6c623c9a5bdd');\n",
|
|
" const dataTable =\n",
|
|
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
|
|
" [key], {});\n",
|
|
" if (!dataTable) return;\n",
|
|
"\n",
|
|
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
|
|
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
|
|
" + ' to learn more about interactive tables.';\n",
|
|
" element.innerHTML = '';\n",
|
|
" dataTable['output_type'] = 'display_data';\n",
|
|
" await google.colab.output.renderOutput(dataTable, element);\n",
|
|
" const docLink = document.createElement('div');\n",
|
|
" docLink.innerHTML = docLinkHtml;\n",
|
|
" element.appendChild(docLink);\n",
|
|
" }\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
"\n",
|
|
" <div id=\"df-8610c582-2189-425e-8676-0483b48edb59\">\n",
|
|
" <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-8610c582-2189-425e-8676-0483b48edb59')\"\n",
|
|
" title=\"Suggest charts\"\n",
|
|
" style=\"display:none;\">\n",
|
|
"\n",
|
|
"<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
|
|
" width=\"24px\">\n",
|
|
" <g>\n",
|
|
" <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
|
|
" </g>\n",
|
|
"</svg>\n",
|
|
" </button>\n",
|
|
"\n",
|
|
"<style>\n",
|
|
" .colab-df-quickchart {\n",
|
|
" --bg-color: #E8F0FE;\n",
|
|
" --fill-color: #1967D2;\n",
|
|
" --hover-bg-color: #E2EBFA;\n",
|
|
" --hover-fill-color: #174EA6;\n",
|
|
" --disabled-fill-color: #AAA;\n",
|
|
" --disabled-bg-color: #DDD;\n",
|
|
" }\n",
|
|
"\n",
|
|
" [theme=dark] .colab-df-quickchart {\n",
|
|
" --bg-color: #3B4455;\n",
|
|
" --fill-color: #D2E3FC;\n",
|
|
" --hover-bg-color: #434B5C;\n",
|
|
" --hover-fill-color: #FFFFFF;\n",
|
|
" --disabled-bg-color: #3B4455;\n",
|
|
" --disabled-fill-color: #666;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart {\n",
|
|
" background-color: var(--bg-color);\n",
|
|
" border: none;\n",
|
|
" border-radius: 50%;\n",
|
|
" cursor: pointer;\n",
|
|
" display: none;\n",
|
|
" fill: var(--fill-color);\n",
|
|
" height: 32px;\n",
|
|
" padding: 0;\n",
|
|
" width: 32px;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart:hover {\n",
|
|
" background-color: var(--hover-bg-color);\n",
|
|
" box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
|
|
" fill: var(--button-hover-fill-color);\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-quickchart-complete:disabled,\n",
|
|
" .colab-df-quickchart-complete:disabled:hover {\n",
|
|
" background-color: var(--disabled-bg-color);\n",
|
|
" fill: var(--disabled-fill-color);\n",
|
|
" box-shadow: none;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .colab-df-spinner {\n",
|
|
" border: 2px solid var(--fill-color);\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" animation:\n",
|
|
" spin 1s steps(1) infinite;\n",
|
|
" }\n",
|
|
"\n",
|
|
" @keyframes spin {\n",
|
|
" 0% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 20% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 30% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-left-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 40% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-top-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 60% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 80% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-right-color: var(--fill-color);\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" 90% {\n",
|
|
" border-color: transparent;\n",
|
|
" border-bottom-color: var(--fill-color);\n",
|
|
" }\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"\n",
|
|
" <script>\n",
|
|
" async function quickchart(key) {\n",
|
|
" const quickchartButtonEl =\n",
|
|
" document.querySelector('#' + key + ' button');\n",
|
|
" quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n",
|
|
" quickchartButtonEl.classList.add('colab-df-spinner');\n",
|
|
" try {\n",
|
|
" const charts = await google.colab.kernel.invokeFunction(\n",
|
|
" 'suggestCharts', [key], {});\n",
|
|
" } catch (error) {\n",
|
|
" console.error('Error during call to suggestCharts:', error);\n",
|
|
" }\n",
|
|
" quickchartButtonEl.classList.remove('colab-df-spinner');\n",
|
|
" quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
|
|
" }\n",
|
|
" (() => {\n",
|
|
" let quickchartButtonEl =\n",
|
|
" document.querySelector('#df-8610c582-2189-425e-8676-0483b48edb59 button');\n",
|
|
" quickchartButtonEl.style.display =\n",
|
|
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
|
|
" })();\n",
|
|
" </script>\n",
|
|
" </div>\n",
|
|
"\n",
|
|
" </div>\n",
|
|
" </div>\n"
|
|
],
|
|
"text/plain": [
|
|
" subscriber_type forecast_timestamp forecast_value confidence_level \\\n",
|
|
"0 Subscriber 2018-05-01 00:00:00+00:00 0.604919 0.95 \n",
|
|
"1 Subscriber 2018-05-01 01:00:00+00:00 21.518158 0.95 \n",
|
|
"2 Subscriber 2018-05-01 02:00:00+00:00 -12.472580 0.95 \n",
|
|
"3 Subscriber 2018-05-01 03:00:00+00:00 -8.919159 0.95 \n",
|
|
"4 Subscriber 2018-05-01 04:00:00+00:00 12.511322 0.95 \n",
|
|
"\n",
|
|
" prediction_interval_lower_bound prediction_interval_upper_bound \\\n",
|
|
"0 -9.780785 10.990624 \n",
|
|
"1 -5.284099 48.320415 \n",
|
|
"2 -32.132971 7.187812 \n",
|
|
"3 -59.022948 41.184630 \n",
|
|
"4 -32.005432 57.028076 \n",
|
|
"\n",
|
|
" ai_forecast_status \n",
|
|
"0 \n",
|
|
"1 \n",
|
|
"2 \n",
|
|
"3 \n",
|
|
"4 "
|
|
]
|
|
},
|
|
"execution_count": 42,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"forecast_results.head(5)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "WGJwVQbcRRcv"
|
|
},
|
|
"source": [
|
|
"We can view the results as a chart using python."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 43,
|
|
"metadata": {
|
|
"id": "fVwumTBNUZSO"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import matplotlib.pyplot as plt\n",
|
|
"\n",
|
|
"# Separate the data by subscriber type\n",
|
|
"customer_df = forecast_results[forecast_results[\"subscriber_type\"] == \"Customer\"]\n",
|
|
"subscriber_df = forecast_results[forecast_results[\"subscriber_type\"] == \"Subscriber\"]\n",
|
|
"\n",
|
|
"# Create a figure and axes for the plot\n",
|
|
"fig, ax = plt.subplots(figsize=(15, 7))\n",
|
|
"\n",
|
|
"# Plot the forecast for 'Customer'\n",
|
|
"ax.plot(\n",
|
|
" customer_df[\"forecast_timestamp\"],\n",
|
|
" customer_df[\"forecast_value\"],\n",
|
|
" label=\"Customer Forecast\",\n",
|
|
")\n",
|
|
"ax.fill_between(\n",
|
|
" customer_df[\"forecast_timestamp\"],\n",
|
|
" customer_df[\"prediction_interval_lower_bound\"],\n",
|
|
" customer_df[\"prediction_interval_upper_bound\"],\n",
|
|
" color=\"blue\",\n",
|
|
" alpha=0.1,\n",
|
|
" label=\"Customer Confidence Interval\",\n",
|
|
")\n",
|
|
"\n",
|
|
"# Plot the forecast for 'Subscriber'\n",
|
|
"ax.plot(\n",
|
|
" subscriber_df[\"forecast_timestamp\"],\n",
|
|
" subscriber_df[\"forecast_value\"],\n",
|
|
" label=\"Subscriber Forecast\",\n",
|
|
")\n",
|
|
"ax.fill_between(\n",
|
|
" subscriber_df[\"forecast_timestamp\"],\n",
|
|
" subscriber_df[\"prediction_interval_lower_bound\"],\n",
|
|
" subscriber_df[\"prediction_interval_upper_bound\"],\n",
|
|
" color=\"orange\",\n",
|
|
" alpha=0.1,\n",
|
|
" label=\"Subscriber Confidence Interval\",\n",
|
|
")\n",
|
|
"\n",
|
|
"\n",
|
|
"# Set the title and labels\n",
|
|
"ax.set_title(\"Bikeshare Trips Forecast\")\n",
|
|
"ax.set_xlabel(\"Date\")\n",
|
|
"ax.set_ylabel(\"Number of Trips\")\n",
|
|
"ax.legend()\n",
|
|
"ax.grid(True)\n",
|
|
"\n",
|
|
"# Rotate the x-axis labels for better readability\n",
|
|
"plt.xticks(rotation=45)\n",
|
|
"\n",
|
|
"# Show the plot\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "tc_YRVNIfS7Q"
|
|
},
|
|
"source": [
|
|
"The forecast shows usage patterns you might expect:\n",
|
|
"\n",
|
|
"* **Subscribers** are typically commuters who use the service for regular, predictable trips, such as going to and from work. Their forecast shows expected weekday peaks during morning and evening commute hours, and lower weekend usage.\n",
|
|
"\n",
|
|
"* **Customers** are usually tourists or casual riders and you might expect them to have higher usage on weekends and holidays for leisure activities, and more trips during the middle of the day, rather than concentrated during traditional commute times.\n",
|
|
"\n",
|
|
"An interesting start to an analysis without any model training!\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "cleanup_md"
|
|
},
|
|
"source": [
|
|
"# Cleaning Up\n",
|
|
"\n",
|
|
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
|
|
"\n",
|
|
"Otherwise, you can delete the individual resources you created in this tutorial:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "cleanup_code"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Delete BigQuery dataset, including the BigQuery remote model you just created, and the BigQuery Connection\n",
|
|
"! bq rm -r -f $PROJECT_ID:bq_ai_tutorial\n",
|
|
"! bq rm --connection --project_id=$PROJECT_ID --location=us test_connection"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"name": "bigquery_generative_ai_intro.ipynb",
|
|
"toc_visible": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|