wehub-resource-sync
842 lines
32 KiB
Plaintext
842 lines
32 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "yFeds7eiwI6x"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2025 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "4ircZQgHwRdy"
|
|
},
|
|
"source": [
|
|
"# Intro to Computer Use with Gemini\n",
|
|
"\n",
|
|
"<table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/computer-use/intro_computer_use.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fcomputer-use%2Fintro_computer_use.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/computer-use/intro_computer_use.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/computer-use/intro_computer_use.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>\n",
|
|
"\n",
|
|
"<p>\n",
|
|
"<b>Share to:</b>\n",
|
|
"\n",
|
|
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/computer-use/intro_computer_use.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/computer-use/intro_computer_use.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/computer-use/intro_computer_use.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/computer-use/intro_computer_use.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/computer-use/intro_computer_use.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
|
"</a>\n",
|
|
"</p>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "MJDxNAGItD51"
|
|
},
|
|
"source": [
|
|
"| Authors |\n",
|
|
"| --- |\n",
|
|
"| [Eric Dong](https://github.com/gericdong) |\n",
|
|
"| [Holt Skinner](https://github.com/holtskinner) |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "896Uhs2Ww6_E"
|
|
},
|
|
"source": [
|
|
"## Overview\n",
|
|
"\n",
|
|
"The **Gemini Computer Use** tool lets you create agents that can automate tasks on a computer. It works by \"seeing\" the screen with screenshots and then \"acting\" with mouse clicks and keyboard inputs.\n",
|
|
"\n",
|
|
"This is useful for tasks like:\n",
|
|
"\n",
|
|
"- Automatically filling out forms on websites.\n",
|
|
"- Testing web applications.\n",
|
|
"- Researching information, like comparing prices, across different sites.\n",
|
|
"\n",
|
|
"Learn more about [computer use](https://cloud.google.com/vertex-ai/generative-ai/docs/computer-use).\n",
|
|
"\n",
|
|
"\n",
|
|
"## Objective\n",
|
|
"\n",
|
|
"In this tutorial, you will build a simple web automation agent using the Gemini Computer Use tool. By the end, you will understand the complete workflow: from sending an initial prompt with a screenshot to executing browser actions and looping until a task is complete."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "gPiTOAHURvTM"
|
|
},
|
|
"source": [
|
|
"## Getting Started"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "CHRZUpfWSEpp"
|
|
},
|
|
"source": [
|
|
"### Install the Gen AI SDK and required libraries"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "sG3_LKsWSD3A"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install --upgrade --quiet google-genai playwright"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "msiBxrk0ATzB"
|
|
},
|
|
"source": [
|
|
"> ⚠️ Note: You can ignore the pip's dependency errors."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "WiZkhIF41qhY"
|
|
},
|
|
"source": [
|
|
"### Set up Playwright\n",
|
|
"\n",
|
|
"Playwright is a tool for browser automation. It enables browser control over web browsers like Chromium, Firefox, and WebKit.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "zL7fxJtq1cWT"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%%capture\n",
|
|
"\n",
|
|
"# Installs Playwright and browsers\n",
|
|
"!playwright install\n",
|
|
"\n",
|
|
"# Additional command, mandatory for Linux only\n",
|
|
"!playwright install-deps"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "rK3jJDR5lfiT"
|
|
},
|
|
"source": [
|
|
"### Import libraries\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "T6DyDoNclVEn"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import logging\n",
|
|
"import os\n",
|
|
"import sys\n",
|
|
"import time\n",
|
|
"from types import SimpleNamespace\n",
|
|
"\n",
|
|
"from google import genai\n",
|
|
"from google.genai.types import (\n",
|
|
" ComputerUse,\n",
|
|
" Content,\n",
|
|
" Environment,\n",
|
|
" FunctionCall,\n",
|
|
" FunctionResponse,\n",
|
|
" FunctionResponseBlob,\n",
|
|
" GenerateContentConfig,\n",
|
|
" Part,\n",
|
|
" ThinkingConfig,\n",
|
|
" Tool,\n",
|
|
")\n",
|
|
"from playwright.async_api import async_playwright\n",
|
|
"\n",
|
|
"logging.getLogger(\"google_genai._common\").setLevel(logging.ERROR)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "HlMVjiAWSMNX"
|
|
},
|
|
"source": [
|
|
"### Authenticate your notebook environment\n",
|
|
"\n",
|
|
"If you are running this notebook on Google Colab, run the cell below to authenticate your environment."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "12fnq4V0SNV3"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"if \"google.colab\" in sys.modules:\n",
|
|
" from google.colab import auth\n",
|
|
"\n",
|
|
" auth.authenticate_user()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "be18ac9c5ec8"
|
|
},
|
|
"source": [
|
|
"### Set your project information\n",
|
|
"\n",
|
|
"Update the following variables with your Google Cloud project details, and connect to the Gen AI service on Vertex AI."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "6wXh1aH7BlPl"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# fmt: off\n",
|
|
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
|
|
"# fmt: on\n",
|
|
"LOCATION = \"global\" # @param {type: \"string\"}\n",
|
|
"\n",
|
|
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
|
|
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
|
|
"\n",
|
|
"# Connect to the Gen AI service on Vertex AI\n",
|
|
"client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "n4yRkFg6BBu4"
|
|
},
|
|
"source": [
|
|
"### Supported Models\n",
|
|
"\n",
|
|
"This tutorial uses the `gemini-3.5-flash` model."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "-coEslfWPrxo"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# fmt: off\n",
|
|
"MODEL_ID = \"gemini-3.5-flash\" # @param [\"gemini-3.5-flash\", \"gemini-2.5-computer-use-preview-10-2025\"]\n",
|
|
"# fmt: on"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "KqjCltg20IgR"
|
|
},
|
|
"source": [
|
|
"## Computer Use: Agent Loop\n",
|
|
"\n",
|
|
"To build a browser control agent, you implement an \"agent loop\" that continuously cycles through four key steps. This process allows the agent to perform a sequence of actions to achieve a goal.\n",
|
|
"\n",
|
|
"1. **Send Request to the Model**. Your app sends the goal (e.g., \"Find me a flight\") and a current screenshot of the screen to the model.\n",
|
|
"\n",
|
|
"2. **Receive the Model Response**. The model analyzes the screen and sends back a suggested action, like navigate to a URL. It may also include a safety warning for risky actions.\n",
|
|
"\n",
|
|
"3. **Execute the Received Action**. Your code runs the suggested action. If there's a safety warning, your code must ask the user for confirmation before proceeding.\n",
|
|
"\n",
|
|
"\n",
|
|
"4. **Capture the New Environment State**. After the action, your code takes a new screenshot. This new screenshot is sent back to the model in the next turn, starting the cycle over again."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "L85sHbJv9cpF"
|
|
},
|
|
"source": [
|
|
"## Prerequisites: Setting Up Your Environment\n",
|
|
"\n",
|
|
"Before you begin, you need to set up two key components:\n",
|
|
"\n",
|
|
"- **Secure Execution Environment**: For safety, you must run your Computer Use agent in a secure and controlled environment. Good options include a sandboxed virtual machine, a container, or a dedicated browser profile with limited permissions.\n",
|
|
"\n",
|
|
"- **Client-Side Action Handler**: You need to write client-side logic to execute the actions generated by the model (e.g., clicking a button) and capture screenshots.\n",
|
|
"\n",
|
|
"In this tutorial, we use Playwright to start a browser environment for demonstration purpose."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "vU5s-EBxCGiN"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Start the Playwright session\n",
|
|
"playwright = await async_playwright().start()\n",
|
|
"\n",
|
|
"# Launch the browser in headless mode, which is required for this environment\n",
|
|
"browser = await playwright.chromium.launch(headless=True)\n",
|
|
"\n",
|
|
"# Create a new page\n",
|
|
"page = await browser.new_page()\n",
|
|
"screen_width, screen_height = 1920, 1080\n",
|
|
"await page.set_viewport_size({\"width\": screen_width, \"height\": screen_height})\n",
|
|
"\n",
|
|
"print(\"Playwright session started.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "nNhzdiH3-str"
|
|
},
|
|
"source": [
|
|
"## A Single Turn: Step-by-Step Walkthrough\n",
|
|
"\n",
|
|
"Now, let's walk through the code for a single turn of the agent loop, from sending the first request to preparing for the next one.\n",
|
|
"\n",
|
|
"### **1. Send a Request to the Model**\n",
|
|
"First, you configure your API request. In the request, you add the Computer Use tool and send a prompt that includes the user's goal and an initial screenshot.\n",
|
|
"\n",
|
|
"You can also include optional parameters like `excluded_predefined_functions` to prevent the model from using certain actions."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "M8CdwkdPAC8e"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Configure Computer Use tool with browser environment\n",
|
|
"# Base configuration for the Computer Use tool\n",
|
|
"config_kwargs = {\n",
|
|
" \"tools\": [\n",
|
|
" Tool(\n",
|
|
" computer_use=ComputerUse(\n",
|
|
" environment=Environment.ENVIRONMENT_BROWSER,\n",
|
|
" # Optional: Exclude specific predefined functions\n",
|
|
" excluded_predefined_functions=[\"drag_and_drop\"],\n",
|
|
" )\n",
|
|
" )\n",
|
|
" ]\n",
|
|
"}\n",
|
|
"\n",
|
|
"# Conditionally add thinking_config only for the Gemini 3 models\n",
|
|
"model_version = float(MODEL_ID.split(\"-\")[1])\n",
|
|
"if model_version >= 3:\n",
|
|
" config_kwargs[\"thinking_config\"] = ThinkingConfig(include_thoughts=True)\n",
|
|
"\n",
|
|
"config = GenerateContentConfig(**config_kwargs)\n",
|
|
"\n",
|
|
"# Create the content with user message and initial screenshot\n",
|
|
"screenshot = await page.screenshot()\n",
|
|
"\n",
|
|
"contents = [\n",
|
|
" Content(\n",
|
|
" role=\"user\",\n",
|
|
" parts=[\n",
|
|
" Part(\n",
|
|
" text=\"Find me a flight from SF to Hawaii on next Monday, coming back on next Friday. Start by navigating directly to flights.google.com\"\n",
|
|
" ),\n",
|
|
" # Optional: include a screenshot of the initial state\n",
|
|
" Part.from_bytes(\n",
|
|
" data=screenshot,\n",
|
|
" mime_type=\"image/png\",\n",
|
|
" ),\n",
|
|
" ],\n",
|
|
" )\n",
|
|
"]\n",
|
|
"\n",
|
|
"# Generate content with the configured settings\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=contents,\n",
|
|
" config=config,\n",
|
|
")\n",
|
|
"\n",
|
|
"print(response)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "M8ymc2GH_xxs"
|
|
},
|
|
"source": [
|
|
"### **2. Receive the Model Response**\n",
|
|
"The model responds with one or more `FunctionCalls` that represent the UI actions it wants to perform. Let's inspect the response from our first API call."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "aC4wcsEBJ9vt"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response.function_calls"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "2vj9YwsxKvdg"
|
|
},
|
|
"source": [
|
|
"### **3. Execute the Received Actions**\n",
|
|
"\n",
|
|
"Next, our application's client-side code needs to parse the response and execute the requested actions using Playwright. We'll use the `execute_function_calls` helper function for this.\n",
|
|
"\n",
|
|
"The following example implements some most common UI actions. For a production use case, you would need to implement all supported actions unless you explicitly exclude them."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "tQqZqdTCf1YH"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def normalize_x(x: int, screen_width: int) -> int:\n",
|
|
" \"\"\"Convert normalized x coordinate (0-1000) to actual pixel coordinate.\"\"\"\n",
|
|
" return int(x / 1000 * screen_width)\n",
|
|
"\n",
|
|
"\n",
|
|
"def normalize_y(y: int, screen_height: int) -> int:\n",
|
|
" \"\"\"Convert normalized y coordinate (0-1000) to actual pixel coordinate.\"\"\"\n",
|
|
" return int(y / 1000 * screen_height)\n",
|
|
"\n",
|
|
"\n",
|
|
"async def execute_function_calls(\n",
|
|
" response, page, screen_width, screen_height\n",
|
|
") -> list[tuple[str, str]]:\n",
|
|
" \"\"\"Extracts and executes function calls from the model response.\"\"\"\n",
|
|
" candidate = response.candidates[0]\n",
|
|
" function_calls = []\n",
|
|
" thoughts = []\n",
|
|
"\n",
|
|
" for part in candidate.content.parts:\n",
|
|
" if hasattr(part, \"function_call\") and part.function_call:\n",
|
|
" function_calls.append(part.function_call)\n",
|
|
" elif hasattr(part, \"text\") and part.text:\n",
|
|
" thoughts.append(part.text)\n",
|
|
"\n",
|
|
" if thoughts:\n",
|
|
" print(f\" Model Reasoning: {' '.join(thoughts)}\")\n",
|
|
"\n",
|
|
" if not function_calls:\n",
|
|
" return \"NO_ACTION\", []\n",
|
|
"\n",
|
|
" results = []\n",
|
|
" for function_call in function_calls:\n",
|
|
" result = None\n",
|
|
" print(f\"⚡ Executing Action: {function_call.name}\")\n",
|
|
" try:\n",
|
|
" if function_call.name == \"open_web_browser\":\n",
|
|
" result = \"success\"\n",
|
|
" elif function_call.name == \"navigate\":\n",
|
|
" await page.goto(function_call.args[\"url\"])\n",
|
|
" result = \"success\"\n",
|
|
" elif function_call.name == \"click_at\":\n",
|
|
" actual_x = normalize_x(function_call.args[\"x\"], screen_width)\n",
|
|
" actual_y = normalize_y(function_call.args[\"y\"], screen_height)\n",
|
|
" await page.mouse.click(actual_x, actual_y)\n",
|
|
" result = \"success\"\n",
|
|
" elif function_call.name == \"type_text_at\":\n",
|
|
" actual_x = normalize_x(function_call.args[\"x\"], screen_width)\n",
|
|
" actual_y = normalize_y(function_call.args[\"y\"], screen_height)\n",
|
|
" await page.mouse.click(actual_x, actual_y)\n",
|
|
" time.sleep(0.1)\n",
|
|
" await page.keyboard.type(function_call.args[\"text\"])\n",
|
|
" if function_call.args.get(\"press_enter\", False):\n",
|
|
" await page.keyboard.press(\"Enter\")\n",
|
|
" result = \"success\"\n",
|
|
" else:\n",
|
|
" result = \"unknown_function\"\n",
|
|
" except Exception as e:\n",
|
|
" print(f\"❗️ Error executing {function_call.name}: {e}\")\n",
|
|
" result = f\"error: {e!s}\"\n",
|
|
" results.append((function_call.name, result))\n",
|
|
" return \"CONTINUE\", results"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "_SP2x-IjMSzr"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"results = await execute_function_calls(response, page, screen_width, screen_height)\n",
|
|
"print(results)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "y287MbtSJ7bs"
|
|
},
|
|
"source": [
|
|
"Here is an example action for navigating a URL. In this case, we create a simple mock response object.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "AubneGmKA2bw"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"mock_response = SimpleNamespace(\n",
|
|
" candidates=[\n",
|
|
" SimpleNamespace(\n",
|
|
" content=SimpleNamespace(\n",
|
|
" parts=[\n",
|
|
" SimpleNamespace(\n",
|
|
" function_call=FunctionCall(\n",
|
|
" name=\"navigate\", args={\"url\": \"https://flights.google.com\"}\n",
|
|
" ),\n",
|
|
" )\n",
|
|
" ]\n",
|
|
" )\n",
|
|
" )\n",
|
|
" ]\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "OW9N3x6kC-kc"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"print(f\"Current page URL: {page.url}\")\n",
|
|
"\n",
|
|
"print(\"Calling execute_function_calls with a sample response\")\n",
|
|
"results = await execute_function_calls(mock_response, page, screen_width, screen_height)\n",
|
|
"print(f\"Results from execution:\\n{results}\\n\")\n",
|
|
"\n",
|
|
"print(f\"Navigated to: {page.url}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "dkndDAPXNLQx"
|
|
},
|
|
"source": [
|
|
"### **4. Capture the New State and Respond**\n",
|
|
"\n",
|
|
"Finally, after executing the actions, we capture a new screenshot and the current URL. This state information is then formatted as a `FunctionResponse` and added to our conversation history, making it ready for the next turn in the loop."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "8QcMbBE5NUHU"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"status, action_results_list = results\n",
|
|
"\n",
|
|
"function_response_parts = []\n",
|
|
"\n",
|
|
"for name, result in action_results_list:\n",
|
|
" # After each action, capture a new screenshot and the current URL\n",
|
|
" screenshot = await page.screenshot()\n",
|
|
" current_url = page.url\n",
|
|
"\n",
|
|
" # Create a FunctionResponse for each action that was executed\n",
|
|
" # This is required even if multiple actions were called in parallel\n",
|
|
" function_response_parts.append(\n",
|
|
" Part(\n",
|
|
" function_response=FunctionResponse(\n",
|
|
" name=name,\n",
|
|
" response={\"url\": current_url},\n",
|
|
" parts=[\n",
|
|
" Part(\n",
|
|
" inline_data=FunctionResponseBlob(\n",
|
|
" mime_type=\"image/png\", data=screenshot\n",
|
|
" )\n",
|
|
" )\n",
|
|
" ],\n",
|
|
" )\n",
|
|
" )\n",
|
|
" )\n",
|
|
"\n",
|
|
"# Package all the function responses into a single 'user' message\n",
|
|
"user_feedback_content = Content(role=\"user\", parts=function_response_parts)\n",
|
|
"\n",
|
|
"# Append this new message to your conversation history\n",
|
|
"contents.append(user_feedback_content)\n",
|
|
"\n",
|
|
"print(\"Step 4 Complete: New state captured and added to conversation history.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "YAZBbH3hD1y8"
|
|
},
|
|
"source": [
|
|
"The `contents` list is now ready for the next call to the model."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "O3LJGCkHD640"
|
|
},
|
|
"source": [
|
|
"#### Clean up by closing the browser and stopping the session\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "GKoItatcCqrV"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"await browser.close()\n",
|
|
"await playwright.stop()\n",
|
|
"\n",
|
|
"print(\"Browser closed and Playwright session stopped.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "sZR7UOfgOdji"
|
|
},
|
|
"source": [
|
|
"## Build an Agent Loop\n",
|
|
"\n",
|
|
"To enable multi-step interactions, combine the four steps from the How to implement Computer Use section into a loop. The loop must handle parallel function calls, and safety decisions. Remember to manage the conversation history correctly by appending both model responses and your function responses."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "m5mYV8CbZE05"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"async def agent_loop(initial_prompt, max_turns=20):\n",
|
|
" \"\"\"Main agent loop\"\"\"\n",
|
|
" playwright_loop = await async_playwright().start()\n",
|
|
" browser_loop = await playwright_loop.chromium.launch(headless=True)\n",
|
|
" page_loop = await browser_loop.new_page()\n",
|
|
" sw, sh = 1920, 1080\n",
|
|
" await page_loop.set_viewport_size({\"width\": sw, \"height\": sh})\n",
|
|
"\n",
|
|
" print(f\"Starting Agent Loop with prompt: '{initial_prompt}'\")\n",
|
|
"\n",
|
|
" screenshot = await page_loop.screenshot()\n",
|
|
" contents = [\n",
|
|
" Content(\n",
|
|
" role=\"user\",\n",
|
|
" parts=[\n",
|
|
" Part(text=initial_prompt),\n",
|
|
" Part.from_bytes(data=screenshot, mime_type=\"image/png\"),\n",
|
|
" ],\n",
|
|
" )\n",
|
|
" ]\n",
|
|
"\n",
|
|
" for turn in range(max_turns):\n",
|
|
" print(f\"\\n Turn {turn + 1}\")\n",
|
|
"\n",
|
|
" response = client.models.generate_content(\n",
|
|
" model=MODEL_ID, contents=contents, config=config\n",
|
|
" )\n",
|
|
"\n",
|
|
" # Handle cases where the model returns no candidates (e.g., due to safety filters)\n",
|
|
" if not response.candidates:\n",
|
|
" print(\"Model returned no candidates. This may be due to a safety filter.\")\n",
|
|
" print(\"Full Response:\", response)\n",
|
|
" print(\"Terminating loop.\")\n",
|
|
" break\n",
|
|
"\n",
|
|
" contents.append(response.candidates[0].content)\n",
|
|
"\n",
|
|
" function_calls = [\n",
|
|
" part.function_call\n",
|
|
" for part in response.candidates[0].content.parts\n",
|
|
" if hasattr(part, \"function_call\") and part.function_call\n",
|
|
" ]\n",
|
|
"\n",
|
|
" # Finish the agent loop if no function call in the response.\n",
|
|
" if not function_calls:\n",
|
|
" final_text = \"\".join(\n",
|
|
" part.text\n",
|
|
" for part in response.candidates[0].content.parts\n",
|
|
" if hasattr(part, \"text\") and part.text is not None\n",
|
|
" )\n",
|
|
" if final_text:\n",
|
|
" print(f\"Agent Finished: {final_text}\")\n",
|
|
" break\n",
|
|
"\n",
|
|
" status, execution_results = await execute_function_calls(\n",
|
|
" response, page_loop, sw, sh\n",
|
|
" )\n",
|
|
"\n",
|
|
" if status == \"NO_ACTION\":\n",
|
|
" continue\n",
|
|
"\n",
|
|
" function_response_parts = []\n",
|
|
" for name, result in execution_results:\n",
|
|
" screenshot = await page_loop.screenshot()\n",
|
|
" current_url = page_loop.url\n",
|
|
" function_response_parts.append(\n",
|
|
" Part(\n",
|
|
" function_response=FunctionResponse(\n",
|
|
" name=name,\n",
|
|
" response={\"url\": current_url},\n",
|
|
" parts=[\n",
|
|
" Part(\n",
|
|
" inline_data=FunctionResponseBlob(\n",
|
|
" mime_type=\"image/png\", data=screenshot\n",
|
|
" )\n",
|
|
" )\n",
|
|
" ],\n",
|
|
" )\n",
|
|
" )\n",
|
|
" )\n",
|
|
" contents.append(Content(role=\"user\", parts=function_response_parts))\n",
|
|
" print(f\"State captured. History now has {len(contents)} messages.\")\n",
|
|
"\n",
|
|
" print(\"\\n Agent loop finished. Closing browser.\")\n",
|
|
" await browser_loop.close()\n",
|
|
" await playwright_loop.stop()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "hX30dnPKnXYK"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# RUN THE AGENT LOOP\n",
|
|
"prompt = \"Navigate to the Google Store and find the 'Pixel' category.\"\n",
|
|
"\n",
|
|
"await agent_loop(prompt)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "gj9KyrqG3n3s"
|
|
},
|
|
"source": [
|
|
"## Next Steps\n",
|
|
"\n",
|
|
"- Explore a [Computer Use web agent reference implementation](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/gemini/computer-use/web-agent).\n",
|
|
"- Check out the [Computer Use documentation](https://ai.google.dev/gemini-api/docs/computer-use) for detailed guides, parameter references, and best practices.\n"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"name": "intro_computer_use.ipynb",
|
|
"toc_visible": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|