{ "cells": [ { "cell_type": "code", "execution_count": null, "metadata": { "id": "yFeds7eiwI6x" }, "outputs": [], "source": [ "# Copyright 2025 Google LLC\n", "#\n", "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "4ircZQgHwRdy" }, "source": [ "# Intro to Computer Use with Gemini\n", "\n", "\n", " \n", " \n", " \n", " \n", "
\n", " \n", " \"Google
Open in Colab\n", " \n", " 		\n", " \n", " \"Google
Open in Colab Enterprise\n", " \n", " 		\n", " \n", " \"Workbench
Open in Workbench\n", " \n", " 		\n", " \n", " \"GitHub
View on GitHub\n", " \n", "
\n", "\n", "
\n", "\n", "

\n", "Share to:\n", "\n", "\n", " \"LinkedIn\n", "\n", "\n", "\n", " \"Bluesky\n", "\n", "\n", "\n", " \"X\n", "\n", "\n", "\n", " \"Reddit\n", "\n", "\n", "\n", " \"Facebook\n", "\n", "

" ] }, { "cell_type": "markdown", "metadata": { "id": "MJDxNAGItD51" }, "source": [ "| Authors |\n", "| --- |\n", "| [Eric Dong](https://github.com/gericdong) |\n", "| [Holt Skinner](https://github.com/holtskinner) |" ] }, { "cell_type": "markdown", "metadata": { "id": "896Uhs2Ww6_E" }, "source": [ "## Overview\n", "\n", "The **Gemini Computer Use** tool lets you create agents that can automate tasks on a computer. It works by \"seeing\" the screen with screenshots and then \"acting\" with mouse clicks and keyboard inputs.\n", "\n", "This is useful for tasks like:\n", "\n", "- Automatically filling out forms on websites.\n", "- Testing web applications.\n", "- Researching information, like comparing prices, across different sites.\n", "\n", "Learn more about [computer use](https://cloud.google.com/vertex-ai/generative-ai/docs/computer-use).\n", "\n", "\n", "## Objective\n", "\n", "In this tutorial, you will build a simple web automation agent using the Gemini Computer Use tool. By the end, you will understand the complete workflow: from sending an initial prompt with a screenshot to executing browser actions and looping until a task is complete." ] }, { "cell_type": "markdown", "metadata": { "id": "gPiTOAHURvTM" }, "source": [ "## Getting Started" ] }, { "cell_type": "markdown", "metadata": { "id": "CHRZUpfWSEpp" }, "source": [ "### Install the Gen AI SDK and required libraries" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "sG3_LKsWSD3A" }, "outputs": [], "source": [ "%pip install --upgrade --quiet google-genai playwright" ] }, { "cell_type": "markdown", "metadata": { "id": "msiBxrk0ATzB" }, "source": [ "> ⚠️ Note: You can ignore the pip's dependency errors." ] }, { "cell_type": "markdown", "metadata": { "id": "WiZkhIF41qhY" }, "source": [ "### Set up Playwright\n", "\n", "Playwright is a tool for browser automation. It enables browser control over web browsers like Chromium, Firefox, and WebKit.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "zL7fxJtq1cWT" }, "outputs": [], "source": [ "%%capture\n", "\n", "# Installs Playwright and browsers\n", "!playwright install\n", "\n", "# Additional command, mandatory for Linux only\n", "!playwright install-deps" ] }, { "cell_type": "markdown", "metadata": { "id": "rK3jJDR5lfiT" }, "source": [ "### Import libraries\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "T6DyDoNclVEn" }, "outputs": [], "source": [ "import logging\n", "import os\n", "import sys\n", "import time\n", "from types import SimpleNamespace\n", "\n", "from google import genai\n", "from google.genai.types import (\n", " ComputerUse,\n", " Content,\n", " Environment,\n", " FunctionCall,\n", " FunctionResponse,\n", " FunctionResponseBlob,\n", " GenerateContentConfig,\n", " Part,\n", " ThinkingConfig,\n", " Tool,\n", ")\n", "from playwright.async_api import async_playwright\n", "\n", "logging.getLogger(\"google_genai._common\").setLevel(logging.ERROR)" ] }, { "cell_type": "markdown", "metadata": { "id": "HlMVjiAWSMNX" }, "source": [ "### Authenticate your notebook environment\n", "\n", "If you are running this notebook on Google Colab, run the cell below to authenticate your environment." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "12fnq4V0SNV3" }, "outputs": [], "source": [ "if \"google.colab\" in sys.modules:\n", " from google.colab import auth\n", "\n", " auth.authenticate_user()" ] }, { "cell_type": "markdown", "metadata": { "id": "be18ac9c5ec8" }, "source": [ "### Set your project information\n", "\n", "Update the following variables with your Google Cloud project details, and connect to the Gen AI service on Vertex AI." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "6wXh1aH7BlPl" }, "outputs": [], "source": [ "# fmt: off\n", "PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n", "# fmt: on\n", "LOCATION = \"global\" # @param {type: \"string\"}\n", "\n", "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n", " PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n", "\n", "# Connect to the Gen AI service on Vertex AI\n", "client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)" ] }, { "cell_type": "markdown", "metadata": { "id": "n4yRkFg6BBu4" }, "source": [ "### Supported Models\n", "\n", "This tutorial uses the `gemini-3.5-flash` model." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "-coEslfWPrxo" }, "outputs": [], "source": [ "# fmt: off\n", "MODEL_ID = \"gemini-3.5-flash\" # @param [\"gemini-3.5-flash\", \"gemini-2.5-computer-use-preview-10-2025\"]\n", "# fmt: on" ] }, { "cell_type": "markdown", "metadata": { "id": "KqjCltg20IgR" }, "source": [ "## Computer Use: Agent Loop\n", "\n", "To build a browser control agent, you implement an \"agent loop\" that continuously cycles through four key steps. This process allows the agent to perform a sequence of actions to achieve a goal.\n", "\n", "1. **Send Request to the Model**. Your app sends the goal (e.g., \"Find me a flight\") and a current screenshot of the screen to the model.\n", "\n", "2. **Receive the Model Response**. The model analyzes the screen and sends back a suggested action, like navigate to a URL. It may also include a safety warning for risky actions.\n", "\n", "3. **Execute the Received Action**. Your code runs the suggested action. If there's a safety warning, your code must ask the user for confirmation before proceeding.\n", "\n", "\n", "4. **Capture the New Environment State**. After the action, your code takes a new screenshot. This new screenshot is sent back to the model in the next turn, starting the cycle over again." ] }, { "cell_type": "markdown", "metadata": { "id": "L85sHbJv9cpF" }, "source": [ "## Prerequisites: Setting Up Your Environment\n", "\n", "Before you begin, you need to set up two key components:\n", "\n", "- **Secure Execution Environment**: For safety, you must run your Computer Use agent in a secure and controlled environment. Good options include a sandboxed virtual machine, a container, or a dedicated browser profile with limited permissions.\n", "\n", "- **Client-Side Action Handler**: You need to write client-side logic to execute the actions generated by the model (e.g., clicking a button) and capture screenshots.\n", "\n", "In this tutorial, we use Playwright to start a browser environment for demonstration purpose." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "vU5s-EBxCGiN" }, "outputs": [], "source": [ "# Start the Playwright session\n", "playwright = await async_playwright().start()\n", "\n", "# Launch the browser in headless mode, which is required for this environment\n", "browser = await playwright.chromium.launch(headless=True)\n", "\n", "# Create a new page\n", "page = await browser.new_page()\n", "screen_width, screen_height = 1920, 1080\n", "await page.set_viewport_size({\"width\": screen_width, \"height\": screen_height})\n", "\n", "print(\"Playwright session started.\")" ] }, { "cell_type": "markdown", "metadata": { "id": "nNhzdiH3-str" }, "source": [ "## A Single Turn: Step-by-Step Walkthrough\n", "\n", "Now, let's walk through the code for a single turn of the agent loop, from sending the first request to preparing for the next one.\n", "\n", "### **1. Send a Request to the Model**\n", "First, you configure your API request. In the request, you add the Computer Use tool and send a prompt that includes the user's goal and an initial screenshot.\n", "\n", "You can also include optional parameters like `excluded_predefined_functions` to prevent the model from using certain actions." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "M8CdwkdPAC8e" }, "outputs": [], "source": [ "# Configure Computer Use tool with browser environment\n", "# Base configuration for the Computer Use tool\n", "config_kwargs = {\n", " \"tools\": [\n", " Tool(\n", " computer_use=ComputerUse(\n", " environment=Environment.ENVIRONMENT_BROWSER,\n", " # Optional: Exclude specific predefined functions\n", " excluded_predefined_functions=[\"drag_and_drop\"],\n", " )\n", " )\n", " ]\n", "}\n", "\n", "# Conditionally add thinking_config only for the Gemini 3 models\n", "model_version = float(MODEL_ID.split(\"-\")[1])\n", "if model_version >= 3:\n", " config_kwargs[\"thinking_config\"] = ThinkingConfig(include_thoughts=True)\n", "\n", "config = GenerateContentConfig(**config_kwargs)\n", "\n", "# Create the content with user message and initial screenshot\n", "screenshot = await page.screenshot()\n", "\n", "contents = [\n", " Content(\n", " role=\"user\",\n", " parts=[\n", " Part(\n", " text=\"Find me a flight from SF to Hawaii on next Monday, coming back on next Friday. Start by navigating directly to flights.google.com\"\n", " ),\n", " # Optional: include a screenshot of the initial state\n", " Part.from_bytes(\n", " data=screenshot,\n", " mime_type=\"image/png\",\n", " ),\n", " ],\n", " )\n", "]\n", "\n", "# Generate content with the configured settings\n", "response = client.models.generate_content(\n", " model=MODEL_ID,\n", " contents=contents,\n", " config=config,\n", ")\n", "\n", "print(response)" ] }, { "cell_type": "markdown", "metadata": { "id": "M8ymc2GH_xxs" }, "source": [ "### **2. Receive the Model Response**\n", "The model responds with one or more `FunctionCalls` that represent the UI actions it wants to perform. Let's inspect the response from our first API call." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "aC4wcsEBJ9vt" }, "outputs": [], "source": [ "response.function_calls" ] }, { "cell_type": "markdown", "metadata": { "id": "2vj9YwsxKvdg" }, "source": [ "### **3. Execute the Received Actions**\n", "\n", "Next, our application's client-side code needs to parse the response and execute the requested actions using Playwright. We'll use the `execute_function_calls` helper function for this.\n", "\n", "The following example implements some most common UI actions. For a production use case, you would need to implement all supported actions unless you explicitly exclude them." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "tQqZqdTCf1YH" }, "outputs": [], "source": [ "def normalize_x(x: int, screen_width: int) -> int:\n", " \"\"\"Convert normalized x coordinate (0-1000) to actual pixel coordinate.\"\"\"\n", " return int(x / 1000 * screen_width)\n", "\n", "\n", "def normalize_y(y: int, screen_height: int) -> int:\n", " \"\"\"Convert normalized y coordinate (0-1000) to actual pixel coordinate.\"\"\"\n", " return int(y / 1000 * screen_height)\n", "\n", "\n", "async def execute_function_calls(\n", " response, page, screen_width, screen_height\n", ") -> list[tuple[str, str]]:\n", " \"\"\"Extracts and executes function calls from the model response.\"\"\"\n", " candidate = response.candidates[0]\n", " function_calls = []\n", " thoughts = []\n", "\n", " for part in candidate.content.parts:\n", " if hasattr(part, \"function_call\") and part.function_call:\n", " function_calls.append(part.function_call)\n", " elif hasattr(part, \"text\") and part.text:\n", " thoughts.append(part.text)\n", "\n", " if thoughts:\n", " print(f\" Model Reasoning: {' '.join(thoughts)}\")\n", "\n", " if not function_calls:\n", " return \"NO_ACTION\", []\n", "\n", " results = []\n", " for function_call in function_calls:\n", " result = None\n", " print(f\"⚡ Executing Action: {function_call.name}\")\n", " try:\n", " if function_call.name == \"open_web_browser\":\n", " result = \"success\"\n", " elif function_call.name == \"navigate\":\n", " await page.goto(function_call.args[\"url\"])\n", " result = \"success\"\n", " elif function_call.name == \"click_at\":\n", " actual_x = normalize_x(function_call.args[\"x\"], screen_width)\n", " actual_y = normalize_y(function_call.args[\"y\"], screen_height)\n", " await page.mouse.click(actual_x, actual_y)\n", " result = \"success\"\n", " elif function_call.name == \"type_text_at\":\n", " actual_x = normalize_x(function_call.args[\"x\"], screen_width)\n", " actual_y = normalize_y(function_call.args[\"y\"], screen_height)\n", " await page.mouse.click(actual_x, actual_y)\n", " time.sleep(0.1)\n", " await page.keyboard.type(function_call.args[\"text\"])\n", " if function_call.args.get(\"press_enter\", False):\n", " await page.keyboard.press(\"Enter\")\n", " result = \"success\"\n", " else:\n", " result = \"unknown_function\"\n", " except Exception as e:\n", " print(f\"❗️ Error executing {function_call.name}: {e}\")\n", " result = f\"error: {e!s}\"\n", " results.append((function_call.name, result))\n", " return \"CONTINUE\", results" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "_SP2x-IjMSzr" }, "outputs": [], "source": [ "results = await execute_function_calls(response, page, screen_width, screen_height)\n", "print(results)" ] }, { "cell_type": "markdown", "metadata": { "id": "y287MbtSJ7bs" }, "source": [ "Here is an example action for navigating a URL. In this case, we create a simple mock response object.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "AubneGmKA2bw" }, "outputs": [], "source": [ "mock_response = SimpleNamespace(\n", " candidates=[\n", " SimpleNamespace(\n", " content=SimpleNamespace(\n", " parts=[\n", " SimpleNamespace(\n", " function_call=FunctionCall(\n", " name=\"navigate\", args={\"url\": \"https://flights.google.com\"}\n", " ),\n", " )\n", " ]\n", " )\n", " )\n", " ]\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "OW9N3x6kC-kc" }, "outputs": [], "source": [ "print(f\"Current page URL: {page.url}\")\n", "\n", "print(\"Calling execute_function_calls with a sample response\")\n", "results = await execute_function_calls(mock_response, page, screen_width, screen_height)\n", "print(f\"Results from execution:\\n{results}\\n\")\n", "\n", "print(f\"Navigated to: {page.url}\")" ] }, { "cell_type": "markdown", "metadata": { "id": "dkndDAPXNLQx" }, "source": [ "### **4. Capture the New State and Respond**\n", "\n", "Finally, after executing the actions, we capture a new screenshot and the current URL. This state information is then formatted as a `FunctionResponse` and added to our conversation history, making it ready for the next turn in the loop." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "8QcMbBE5NUHU" }, "outputs": [], "source": [ "status, action_results_list = results\n", "\n", "function_response_parts = []\n", "\n", "for name, result in action_results_list:\n", " # After each action, capture a new screenshot and the current URL\n", " screenshot = await page.screenshot()\n", " current_url = page.url\n", "\n", " # Create a FunctionResponse for each action that was executed\n", " # This is required even if multiple actions were called in parallel\n", " function_response_parts.append(\n", " Part(\n", " function_response=FunctionResponse(\n", " name=name,\n", " response={\"url\": current_url},\n", " parts=[\n", " Part(\n", " inline_data=FunctionResponseBlob(\n", " mime_type=\"image/png\", data=screenshot\n", " )\n", " )\n", " ],\n", " )\n", " )\n", " )\n", "\n", "# Package all the function responses into a single 'user' message\n", "user_feedback_content = Content(role=\"user\", parts=function_response_parts)\n", "\n", "# Append this new message to your conversation history\n", "contents.append(user_feedback_content)\n", "\n", "print(\"Step 4 Complete: New state captured and added to conversation history.\")" ] }, { "cell_type": "markdown", "metadata": { "id": "YAZBbH3hD1y8" }, "source": [ "The `contents` list is now ready for the next call to the model." ] }, { "cell_type": "markdown", "metadata": { "id": "O3LJGCkHD640" }, "source": [ "#### Clean up by closing the browser and stopping the session\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "GKoItatcCqrV" }, "outputs": [], "source": [ "await browser.close()\n", "await playwright.stop()\n", "\n", "print(\"Browser closed and Playwright session stopped.\")" ] }, { "cell_type": "markdown", "metadata": { "id": "sZR7UOfgOdji" }, "source": [ "## Build an Agent Loop\n", "\n", "To enable multi-step interactions, combine the four steps from the How to implement Computer Use section into a loop. The loop must handle parallel function calls, and safety decisions. Remember to manage the conversation history correctly by appending both model responses and your function responses." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "m5mYV8CbZE05" }, "outputs": [], "source": [ "async def agent_loop(initial_prompt, max_turns=20):\n", " \"\"\"Main agent loop\"\"\"\n", " playwright_loop = await async_playwright().start()\n", " browser_loop = await playwright_loop.chromium.launch(headless=True)\n", " page_loop = await browser_loop.new_page()\n", " sw, sh = 1920, 1080\n", " await page_loop.set_viewport_size({\"width\": sw, \"height\": sh})\n", "\n", " print(f\"Starting Agent Loop with prompt: '{initial_prompt}'\")\n", "\n", " screenshot = await page_loop.screenshot()\n", " contents = [\n", " Content(\n", " role=\"user\",\n", " parts=[\n", " Part(text=initial_prompt),\n", " Part.from_bytes(data=screenshot, mime_type=\"image/png\"),\n", " ],\n", " )\n", " ]\n", "\n", " for turn in range(max_turns):\n", " print(f\"\\n Turn {turn + 1}\")\n", "\n", " response = client.models.generate_content(\n", " model=MODEL_ID, contents=contents, config=config\n", " )\n", "\n", " # Handle cases where the model returns no candidates (e.g., due to safety filters)\n", " if not response.candidates:\n", " print(\"Model returned no candidates. This may be due to a safety filter.\")\n", " print(\"Full Response:\", response)\n", " print(\"Terminating loop.\")\n", " break\n", "\n", " contents.append(response.candidates[0].content)\n", "\n", " function_calls = [\n", " part.function_call\n", " for part in response.candidates[0].content.parts\n", " if hasattr(part, \"function_call\") and part.function_call\n", " ]\n", "\n", " # Finish the agent loop if no function call in the response.\n", " if not function_calls:\n", " final_text = \"\".join(\n", " part.text\n", " for part in response.candidates[0].content.parts\n", " if hasattr(part, \"text\") and part.text is not None\n", " )\n", " if final_text:\n", " print(f\"Agent Finished: {final_text}\")\n", " break\n", "\n", " status, execution_results = await execute_function_calls(\n", " response, page_loop, sw, sh\n", " )\n", "\n", " if status == \"NO_ACTION\":\n", " continue\n", "\n", " function_response_parts = []\n", " for name, result in execution_results:\n", " screenshot = await page_loop.screenshot()\n", " current_url = page_loop.url\n", " function_response_parts.append(\n", " Part(\n", " function_response=FunctionResponse(\n", " name=name,\n", " response={\"url\": current_url},\n", " parts=[\n", " Part(\n", " inline_data=FunctionResponseBlob(\n", " mime_type=\"image/png\", data=screenshot\n", " )\n", " )\n", " ],\n", " )\n", " )\n", " )\n", " contents.append(Content(role=\"user\", parts=function_response_parts))\n", " print(f\"State captured. History now has {len(contents)} messages.\")\n", "\n", " print(\"\\n Agent loop finished. Closing browser.\")\n", " await browser_loop.close()\n", " await playwright_loop.stop()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "hX30dnPKnXYK" }, "outputs": [], "source": [ "# RUN THE AGENT LOOP\n", "prompt = \"Navigate to the Google Store and find the 'Pixel' category.\"\n", "\n", "await agent_loop(prompt)" ] }, { "cell_type": "markdown", "metadata": { "id": "gj9KyrqG3n3s" }, "source": [ "## Next Steps\n", "\n", "- Explore a [Computer Use web agent reference implementation](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/gemini/computer-use/web-agent).\n", "- Check out the [Computer Use documentation](https://ai.google.dev/gemini-api/docs/computer-use) for detailed guides, parameter references, and best practices.\n" ] } ], "metadata": { "colab": { "name": "intro_computer_use.ipynb", "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 0 }