wehub-resource-sync
1031 lines
37 KiB
Plaintext
1031 lines
37 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 61,
|
|
"metadata": {
|
|
"id": "FWwAUaf5Fi2_"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2024 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "5tmkeOihFiE2"
|
|
},
|
|
"source": [
|
|
"# Multimodal Function Calling with Claude Models\n",
|
|
"\n",
|
|
"<table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/partner-models/claude/claude_multimodal_function_calling.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fpartner-models%2Fclaude%2Fclaude_multimodal_function_calling.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/partner-models/claude/claude_multimodal_function_calling.ipynb\">\n",
|
|
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/partner-models/claude/claude_multimodal_function_calling.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>\n",
|
|
"\n",
|
|
"<b>Share to:</b>\n",
|
|
"\n",
|
|
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/partner-models/claude/claude_multimodal_function_calling.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/partner-models/claude/claude_multimodal_function_calling.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/partner-models/claude/claude_multimodal_function_calling.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/partner-models/claude/claude_multimodal_function_calling.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/partner-models/claude/claude_multimodal_function_calling.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
|
"</a> "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "hL_l-HtkF6la"
|
|
},
|
|
"source": [
|
|
"## Overview\n",
|
|
"\n",
|
|
"### Claude on Vertex AI\n",
|
|
"\n",
|
|
"Anthropic's Claude models on Vertex AI offer fully managed and serverless models as APIs. To use a Claude model on Vertex AI, send a request directly to the Vertex AI API endpoint.\n",
|
|
"\n",
|
|
"In this notebook you'll use multimodal prompting and function calling (tools use) with Anthropic Claude models. To learn more about function calling with Claude, [check out the Anthropic docs.](https://docs.anthropic.com/en/docs/build-with-claude/tool-use)\n",
|
|
"\n",
|
|
"### Available Anthropic Claude models\n",
|
|
"\n",
|
|
"#### Claude 3.5 Sonnet v2\n",
|
|
"The upgraded Claude 3.5 Sonnet is now state-of-the-art for a variety of tasks including real-world software engineering, enhanced agentic capabilities, and computer use.\n",
|
|
"\n",
|
|
"#### Claude 3.5 Sonnet\n",
|
|
"Anthropic's most powerful AI model. Claude 3.5 Sonnet outperforms competitor models and Claude 3 Opus at higher speeds and lower cost.\n",
|
|
"\n",
|
|
"#### Claude 3 Opus\n",
|
|
"Claude 3 Opus is Anthropic's second-most intelligent AI model, with top-level performance on highly complex tasks.\n",
|
|
"\n",
|
|
"#### Claude 3 Haiku\n",
|
|
"Anthropic Claude 3 Haiku is Anthropic's fastest, most compact vision and text model for near-instant responses to simple queries, meant for seamless AI experiences mimicking human interactions.\n",
|
|
"\n",
|
|
"#### Claude 3 Sonnet\n",
|
|
"Anthropic Claude 3 Sonnet is engineered to be dependable for scaled AI deployments across a variety of use cases.\n",
|
|
"\n",
|
|
"All Claude 3 models can process images and return text outputs, call tools, and feature a 200K context window"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "tjnAqz4BGOyk"
|
|
},
|
|
"source": [
|
|
"## Get started"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "VHxAl1WAGQq4"
|
|
},
|
|
"source": [
|
|
"Install the required packages"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "62b7f4adee9c"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install --upgrade --user --quiet 'anthropic[vertex]' wikipedia"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "i0c93wc9Go-y"
|
|
},
|
|
"source": [
|
|
"### Restart runtime\n",
|
|
"\n",
|
|
"To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel.\n",
|
|
"\n",
|
|
"The restart might take a minute or longer. After it's restarted, continue to the next step."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "f3d98395d9a4"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import IPython\n",
|
|
"\n",
|
|
"app = IPython.Application.instance()\n",
|
|
"app.kernel.do_shutdown(True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "TTbdL4vtGwSY"
|
|
},
|
|
"source": [
|
|
"<div class=\"alert alert-block alert-warning\">\n",
|
|
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
|
|
"</div>\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "mv14NC0_Gyqk"
|
|
},
|
|
"source": [
|
|
"### Authenticate your notebook environment (Colab only)\n",
|
|
"\n",
|
|
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "o4rK7PYLXYhZ"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import sys\n",
|
|
"\n",
|
|
"if \"google.colab\" in sys.modules:\n",
|
|
" from google.colab import auth\n",
|
|
"\n",
|
|
" auth.authenticate_user()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "YtsDY4JCG1cf"
|
|
},
|
|
"source": [
|
|
"### Set Google Cloud project information\n",
|
|
"\n",
|
|
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
|
|
"\n",
|
|
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {
|
|
"id": "bJnsUWOrG3XN"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Use the environment variable if the user doesn't provide Project ID.\n",
|
|
"import os\n",
|
|
"\n",
|
|
"# fmt: off\n",
|
|
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
|
|
"# fmt: on\n",
|
|
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
|
|
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
|
|
"\n",
|
|
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-east5\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "8VEAR8IEv-3Q"
|
|
},
|
|
"source": [
|
|
"### Set up Anthropic client"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "671a07af230b"
|
|
},
|
|
"source": [
|
|
"#### Select Claude model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "909e2d6f176e"
|
|
},
|
|
"source": [
|
|
"Models have different location availability, so you'll need to select the model before you create the Anthropic client. You can see the [available regions for each model here.](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/use-claude#regions)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 41,
|
|
"metadata": {
|
|
"id": "28d272a965f8"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# fmt: off\n",
|
|
"MODEL = \"claude-3-5-sonnet-v2@20241022\" # @param [\"claude-3-5-sonnet-v2@20241022\", \"claude-3-5-sonnet@20240620\", \"claude-3-opus@20240229\", \"claude-3-haiku@20240307\", \"claude-3-sonnet@20240229\" ]\n",
|
|
"# fmt: on\n",
|
|
"if MODEL == \"claude-3-5-sonnet-v2@20241022\":\n",
|
|
" available_regions = [\"us-east5\"]\n",
|
|
"elif MODEL == \"claude-3-5-sonnet@20240620\":\n",
|
|
" available_regions = [\"us-east5\", \"europe-west1\", \"asia-southeast1\"]\n",
|
|
"elif MODEL == \"claude-3-opus@20240229\":\n",
|
|
" available_regions = [\"us-east5\"]\n",
|
|
"elif MODEL == \"claude-3-haiku@20240307\":\n",
|
|
" available_regions = [\"us-east5\", \"europe-west1\", \"asia-southeast1\"]\n",
|
|
"elif MODEL == \"claude-3-sonnet@20240229\":\n",
|
|
" available_regions = [\"us-east5\"]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "94fb805e888b"
|
|
},
|
|
"source": [
|
|
"Next, choose from the available regions for the selected Claude model."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "58658cdacc51"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import ipywidgets as widgets\n",
|
|
"from IPython.display import display\n",
|
|
"\n",
|
|
"dropdown = widgets.Dropdown(\n",
|
|
" options=available_regions,\n",
|
|
" description=\"Select a location:\",\n",
|
|
" font_weight=\"bold\",\n",
|
|
" style={\"description_width\": \"initial\"},\n",
|
|
")\n",
|
|
"\n",
|
|
"\n",
|
|
"def dropdown_eventhandler(change):\n",
|
|
" global LOCATION\n",
|
|
" if change[\"type\"] == \"change\" and change[\"name\"] == \"value\":\n",
|
|
" LOCATION = change.new\n",
|
|
" print(\"Selected:\", change.new)\n",
|
|
"\n",
|
|
"\n",
|
|
"LOCATION = dropdown.value\n",
|
|
"dropdown.observe(dropdown_eventhandler, names=\"value\")\n",
|
|
"display(dropdown)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "c755e985a4f1"
|
|
},
|
|
"source": [
|
|
"#### Enable Claude model and accept terms of service"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "b51cf10828e2"
|
|
},
|
|
"source": [
|
|
"Before you can send messages to Claude, you'll need to enable the model on Vertex AI. You can do this from the model card in [Vertex AI Model Garden.](https://console.cloud.google.com/vertex-ai/model-garden) You will search for the Claude model you want to use and select the blue ENABLE button.\n",
|
|
"\n",
|
|
"For example, here's the [link to enable Claude 3.2 Sonnet v2](https://console.cloud.google.com/vertex-ai/model-garden/questionnaire?model=publishers%2Fanthropic%2Fmodels%2Fclaude-3-5-sonnet-v2&mp=anthropic%2Fclaude-3-5-sonnet-v2.endpoints.mp-anthropic.cloud.goog). Note that if you do not enable the specific model you want to use and accept the terms of service, you will receive an error message when sending a prompt."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "3674328de61b"
|
|
},
|
|
"source": [
|
|
"#### Create Anthropic client"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "8SSrfXmiIfIq"
|
|
},
|
|
"source": [
|
|
"Now you can import the AnthropicVertex SDK and initialize the client."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"metadata": {
|
|
"id": "Vt4z00dGXbwo"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from anthropic import AnthropicVertex\n",
|
|
"\n",
|
|
"client = AnthropicVertex(region=LOCATION, project_id=PROJECT_ID)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "LkqV_DD0wEZ6"
|
|
},
|
|
"source": [
|
|
"### Multimodal prompting"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "JVz3xwl8YCte"
|
|
},
|
|
"source": [
|
|
"To start, create a multimodal prompt that contains an image.\n",
|
|
"\n",
|
|
"If you want to send an image in your prompt, you'll need to base64 encode it first. You can follow the steps in the cell below to do so."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"metadata": {
|
|
"id": "K2CiOQtKY7lP"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import base64\n",
|
|
"\n",
|
|
"import httpx\n",
|
|
"import requests\n",
|
|
"from IPython.display import Image"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 43,
|
|
"metadata": {
|
|
"id": "Oy_v3g21Y3dk"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"image_url = (\n",
|
|
" \"https://upload.wikimedia.org/wikipedia/commons/b/b7/Dendroica_chrysoparia1.jpg\"\n",
|
|
")\n",
|
|
"image_media_type = \"image/jpeg\"\n",
|
|
"image_b64 = base64.b64encode(httpx.get(image_url).content).decode(\"utf-8\")\n",
|
|
"\n",
|
|
"headers = {\n",
|
|
" \"User-Agent\": \"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Mobile Safari/537.36\",\n",
|
|
"}\n",
|
|
"\n",
|
|
"response = requests.get(image_url, headers=headers)\n",
|
|
"image = Image(response.content, width=300, height=200)\n",
|
|
"\n",
|
|
"image"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "kU91q-QkYcWj"
|
|
},
|
|
"source": [
|
|
"Next, define the user message to send to Claude. Because this is a multimodal message, it contains both `\"type\": \"image\"` and `\"type\": \"text\"`.\n",
|
|
"\n",
|
|
"The image content contains the `\"media_type\"`, in this case `\"image/jpeg\"`, as well as the base64 encoded image."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 19,
|
|
"metadata": {
|
|
"id": "oDN8AGC3YSPC"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"messages = [\n",
|
|
" {\n",
|
|
" \"role\": \"user\",\n",
|
|
" \"content\": [\n",
|
|
" {\n",
|
|
" \"type\": \"image\", # pass in the image\n",
|
|
" \"source\": {\n",
|
|
" \"type\": \"base64\",\n",
|
|
" \"media_type\": image_media_type,\n",
|
|
" \"data\": image_b64,\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"type\": \"text\", # pass in the text\n",
|
|
" \"text\": \"What is the region where this animal lives?\",\n",
|
|
" },\n",
|
|
" ],\n",
|
|
" }\n",
|
|
"]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "suhbpNIzwlCP"
|
|
},
|
|
"source": [
|
|
"Once the message is defined, you can call `messages.create` to get a response from the model endpoint."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 16,
|
|
"metadata": {
|
|
"id": "KJk_z8qGYelQ"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.messages.create(\n",
|
|
" max_tokens=1024,\n",
|
|
" messages=messages,\n",
|
|
" model=MODEL,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 17,
|
|
"metadata": {
|
|
"id": "wiLqdiH9aB60"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{\n",
|
|
" \"id\": \"msg_vrtx_01PbLXjy7XzfNMiH76f3s5Ep\",\n",
|
|
" \"content\": [\n",
|
|
" {\n",
|
|
" \"text\": \"This appears to be a Golden-cheeked Warbler (Setophaga chrysoparia), which has a very limited breeding range. It breeds exclusively in central Texas in the Edwards Plateau region, making it the only bird species whose breeding range is entirely within Texas. During winter, these warblers migrate to southern Mexico and Central America (primarily in southern Mexico, Guatemala, Honduras, and Nicaragua).\\n\\nThe bird is highly specialized in its habitat requirements, preferring mature juniper-oak woodlands, particularly areas with Ashe juniper (also known as cedar) trees, which is clearly visible in this image where the bird is perched on what appears to be juniper foliage. This specific habitat requirement is one reason why the species is considered endangered, as urban development and habitat fragmentation in central Texas have significantly reduced its breeding grounds.\",\n",
|
|
" \"type\": \"text\"\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"model\": \"claude-3-5-sonnet-v2-20241022\",\n",
|
|
" \"role\": \"assistant\",\n",
|
|
" \"stop_reason\": \"end_turn\",\n",
|
|
" \"stop_sequence\": null,\n",
|
|
" \"type\": \"message\",\n",
|
|
" \"usage\": {\n",
|
|
" \"input_tokens\": 1587,\n",
|
|
" \"output_tokens\": 179\n",
|
|
" }\n",
|
|
"}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"print(response.model_dump_json(indent=2))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "_I0CZL5NxBee"
|
|
},
|
|
"source": [
|
|
"### Multimodal function calling"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "_lZmF9xqaM34"
|
|
},
|
|
"source": [
|
|
"What if you wanted Claude to retrieve specific information about the animal from a database instead of drawing on information from its training data?\n",
|
|
"\n",
|
|
"Combining multimodal prompting with function calling can unlock some really neat use cases (If you're new to the concept of function calling or connecting LLMs to tools, check out this [introductory concepts video](https://www.youtube.com/watch?v=P3buv6P_u7c)).\n",
|
|
"\n",
|
|
"As an example, let's say you want Claude to use a tool every time a user requests information about an animal's habitat. Instead of the answer coming from the model's training data, you want the model to pull the information from an external source of information accessible via an API. To do this, or use Claude with any other tools, there are four high level steps you'll follow."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "1f42f06465ea"
|
|
},
|
|
"source": [
|
|
"* **STEP 1:** Define tool and send a request to Claude\n",
|
|
"* **STEP 2:** Claude decides to use a tool\n",
|
|
"* **STEP 3:** Developer makes an API call to the tool\n",
|
|
"* **STEP 4:** Claude uses tool result to formulate a conversational response\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "d08dacffbed6"
|
|
},
|
|
"source": [
|
|
"**STEP 1: Define tool and send a request to Claude**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "KJVqiPn2aVm-"
|
|
},
|
|
"source": [
|
|
"The first thing you need to do is write a tool definition. This is a dictionary with information like a name and description of the tool, as well as the required input schema. Anthropic's [docs have some great tips](https://docs.anthropic.com/en/docs/build-with-claude/tool-use#best-practices-for-tool-definitions) for crafting successful tool descriptions.\n",
|
|
"\n",
|
|
"Here, you define a tool called `get_wildlife_region`, with a simple description `Look up the region where an animal can be found.`"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 18,
|
|
"metadata": {
|
|
"id": "Vq_MUvHDZD6W"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"TOOL = {\n",
|
|
" \"name\": \"get_wildlife_region\",\n",
|
|
" \"description\": \"Look up the region where an animal can be found\",\n",
|
|
" \"input_schema\": {\n",
|
|
" \"type\": \"object\",\n",
|
|
" \"properties\": {\n",
|
|
" \"animal\": {\"type\": \"string\", \"description\": \"species of animal\"},\n",
|
|
" },\n",
|
|
" },\n",
|
|
"}"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "kdqMkUNAaaaU"
|
|
},
|
|
"source": [
|
|
"Next, send the same message as earlier, which contained the image of the warbler and the text _\"What is the region where this animal lives?\"_, but this time add in the tool parameter and pass in the `TOOL` dictionary defined above.\n",
|
|
"\n",
|
|
"Note: for more information on sending messages to Claude, check out the [Messages API reference.](https://docs.anthropic.com/en/api/messages)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 20,
|
|
"metadata": {
|
|
"id": "JN0j8YsNaYyO"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.messages.create(\n",
|
|
" max_tokens=1024,\n",
|
|
" messages=messages,\n",
|
|
" tools=[TOOL], # pass in the tools!\n",
|
|
" model=MODEL,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 21,
|
|
"metadata": {
|
|
"id": "lvRWjow9acTq"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{\n",
|
|
" \"id\": \"msg_vrtx_015F4GZi1En2FBwsTdrtrc12\",\n",
|
|
" \"content\": [\n",
|
|
" {\n",
|
|
" \"text\": \"This appears to be a Golden-cheeked Warbler (Setophaga chrysoparia), a rare songbird. Let me look up its region for you.\",\n",
|
|
" \"type\": \"text\"\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"toolu_vrtx_01AoZske3Zi12oYewSYnng2z\",\n",
|
|
" \"input\": {\n",
|
|
" \"animal\": \"Golden-cheeked Warbler\"\n",
|
|
" },\n",
|
|
" \"name\": \"get_wildlife_region\",\n",
|
|
" \"type\": \"tool_use\"\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"model\": \"claude-3-5-sonnet-v2-20241022\",\n",
|
|
" \"role\": \"assistant\",\n",
|
|
" \"stop_reason\": \"tool_use\",\n",
|
|
" \"stop_sequence\": null,\n",
|
|
" \"type\": \"message\",\n",
|
|
" \"usage\": {\n",
|
|
" \"input_tokens\": 1963,\n",
|
|
" \"output_tokens\": 99\n",
|
|
" }\n",
|
|
"}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"print(response.model_dump_json(indent=2))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "S4EKTDkyah8K"
|
|
},
|
|
"source": [
|
|
"**STEP 2: Claude decides to use a tool**\n",
|
|
"\n",
|
|
"Even though the message sent to Claude was the same, the response will look different now because you provided Claude with a list of tool(s)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "A5YwIL7K0kqU"
|
|
},
|
|
"source": [
|
|
"The content returned from the model contains a `TextBlock` and a `ToolUseBlock`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 24,
|
|
"metadata": {
|
|
"id": "FswChxN-0jro"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"[TextBlock(text='This appears to be a Golden-cheeked Warbler (Setophaga chrysoparia), a rare songbird. Let me look up its region for you.', type='text'),\n",
|
|
" ToolUseBlock(id='toolu_vrtx_01AoZske3Zi12oYewSYnng2z', input={'animal': 'Golden-cheeked Warbler'}, name='get_wildlife_region', type='tool_use')]"
|
|
]
|
|
},
|
|
"execution_count": 24,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"response.content"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "0ftyFbldak_R"
|
|
},
|
|
"source": [
|
|
"In the `TextBlock`, there's an explanation from Claude indicating that the model has identified a tool is necessary. Depending on the model you use, you'll see something like _\"Let me look up its region\"_ or _\"To answer your question about the typical habitat I'll need to use an available tool to look up its region\"_, etc.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 27,
|
|
"metadata": {
|
|
"id": "J8n7Qw6K0p4w"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{\n",
|
|
" \"text\": \"This appears to be a Golden-cheeked Warbler (Setophaga chrysoparia), a rare songbird. Let me look up its region for you.\",\n",
|
|
" \"type\": \"text\"\n",
|
|
"}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"textblock = response.content[0]\n",
|
|
"print(textblock.model_dump_json(indent=2))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "HPmXG9fh1LTr"
|
|
},
|
|
"source": [
|
|
"In the `ToolUseBlock` there's a tool `id`, and then the API parameters specified in the tool definition, in this case the `\"animal\" : \"Golden-cheeked Warbler\"`.\n",
|
|
"\n",
|
|
"At this point Claude has identified that calling the `get_wildlife_region` tool is necessary to respond to the user's query, but we as developers have to go do that ourselves. Claude can do a lot, but it isn't going to start actually making API calls on our behalf!"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 29,
|
|
"metadata": {
|
|
"id": "CPFb4b3g1EZk"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{\n",
|
|
" \"id\": \"toolu_vrtx_01AoZske3Zi12oYewSYnng2z\",\n",
|
|
" \"input\": {\n",
|
|
" \"animal\": \"Golden-cheeked Warbler\"\n",
|
|
" },\n",
|
|
" \"name\": \"get_wildlife_region\",\n",
|
|
" \"type\": \"tool_use\"\n",
|
|
"}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"tooluseblock = response.content[1]\n",
|
|
"print(tooluseblock.model_dump_json(indent=2))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "5a57261f0f91"
|
|
},
|
|
"source": [
|
|
"Note that in this example, you have only provided Claude with a single tool. However, you can define multiple tools when sending a message. Additionally, Claude supports [parallel tool use](https://docs.anthropic.com/en/docs/build-with-claude/tool-use#disabling-parallel-tool-use), meaning that the model may use multiple tools to answer a user query. For more details on configuring tools with Claude, [refer to the docs.](https://docs.anthropic.com/en/docs/build-with-claude/tool-use#controlling-claudes-output)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Rl4icSkranVS"
|
|
},
|
|
"source": [
|
|
"**STEP 3: Developer makes an API call to the tool**\n",
|
|
"\n",
|
|
"Instead of creating an actual database of habitat information, you'll use the Wikipedia API to retrieve information about where the beloved warbler lives. \n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 30,
|
|
"metadata": {
|
|
"id": "6zI-BXYvae-H"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import wikipedia\n",
|
|
"\n",
|
|
"animal_name = response.content[1].input[\"animal\"] # Golden-cheeked Warbler\n",
|
|
"api_response = wikipedia.page(animal_name).content"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Nqf06f_la5mu"
|
|
},
|
|
"source": [
|
|
"The `api_response` is a huge blob of text from the Golden-cheeked Warbler Wikipedia page. The users of your wildlife chatbot would be pretty upset if you just dropped all of that text into the response box. So as a final step, you'll use Claude to format all of this information nicely and craft an eloquent response for the user"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "DgMXA9Wsa7Qs"
|
|
},
|
|
"source": [
|
|
"**STEP 4: Claude uses tool result to formulate a conversational response**\n",
|
|
"\n",
|
|
"The messages list defined in Step 1 currently contains a single message from `\"role\" : \"user\"`.\n",
|
|
"\n",
|
|
"Next, you'll add Claude's response from Step 2 to this list. To do this you'll append a new dictionary. This new dictionary has the `\"role\" : \"assistant\"` (instead of user). The value corresponding to the `content` key is the blob of text from Claude that contained the `TextBlock` and `ToolUseBlock`.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 31,
|
|
"metadata": {
|
|
"id": "An7kojsla9gt"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"messages.append({\"role\": \"assistant\", \"content\": response.content})"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "z80yBfPea_gM"
|
|
},
|
|
"source": [
|
|
"Finally, you'll append one more dictionary to this messages list. This dictionary again will have the `\"role\" : \"user\"`, and the content will contain the tool id and api response (aka that big blob of Wikipedia text).\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 35,
|
|
"metadata": {
|
|
"id": "h2RUPDOVaqCW"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"tool_use = response.content[-1]\n",
|
|
"\n",
|
|
"# create tool response message\n",
|
|
"tool_response = {\n",
|
|
" \"role\": \"user\",\n",
|
|
" \"content\": [\n",
|
|
" {\"type\": \"tool_result\", \"tool_use_id\": tool_use.id, \"content\": api_response}\n",
|
|
" ],\n",
|
|
"}"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 36,
|
|
"metadata": {
|
|
"id": "41eh4A0h3ktI"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# append tool_response dictionary to messages\n",
|
|
"messages.append(tool_response)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "e5kXNZeQbCw6"
|
|
},
|
|
"source": [
|
|
"If you're tracking this closely, you'll notice that the messages list now contains three dictionaries."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 37,
|
|
"metadata": {
|
|
"id": "OnK8_1sw3o29"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"3"
|
|
]
|
|
},
|
|
"execution_count": 37,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"len(messages)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "7s5ye7LX3qMv"
|
|
},
|
|
"source": [
|
|
"These messages are:\n",
|
|
"\n",
|
|
"1. **[user]** Original user message, which included an image of the warbler and the text _\"What is the region where this animal lives?\"_\n",
|
|
"2. **[assistant]** The response from the model, which included a `TextBlock`, as well as the `ToolUseBlock` that indicated what tool to call and the value `Golden-Cheeked Warbler` for the animal parameter\n",
|
|
"3. **[user]** Then the tool response you just formatted, which is a dictionary with the tool id from the model as well as the big blog of text returned from the API call made to Wikipedia.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "lGbvvoPrbEVy"
|
|
},
|
|
"source": [
|
|
"Take all three of these dictionaries, and send all of them as messages to Claude."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 38,
|
|
"metadata": {
|
|
"id": "mGkZP1hgbA32"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"follow_up_response = client.messages.create(\n",
|
|
" model=MODEL,\n",
|
|
" messages=messages, # long list with three dictionaries\n",
|
|
" max_tokens=1000,\n",
|
|
" tools=[TOOL], # don't forget the tool!\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "XLA10cKx4QOb"
|
|
},
|
|
"source": [
|
|
"After all the hard work, Claude will responds with something fluent and conversational, based on info extracted from the tool."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 39,
|
|
"metadata": {
|
|
"id": "hOLydfiubJ3w"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"\"Based on the results, the Golden-cheeked Warbler is endemic to (meaning it exclusively breeds in) Central Texas, specifically in an area from Palo Pinto County southwestward along the eastern and southern edge of the Edwards Plateau to Kinney County. It's the only bird species with a breeding range that is completely endemic to Texas.\\n\\nDuring the breeding season (March to June), these warblers can be found in various Texas state parks including Colorado Bend, Dinosaur Valley, Garner, Guadalupe River, and several others. However, during winter months (November-February), they migrate to parts of Mexico and Central America, specifically Guatemala, Nicaragua, and Honduras.\\n\\nThe bird requires a very specific habitat of juniper-oak woodlands, particularly areas with mature Ashe juniper trees, which they use for nesting material. Their total breeding range is quite limited, covering only about 1,768 square kilometers (437,000 acres), which is one reason why this species is currently endangered.\""
|
|
]
|
|
},
|
|
"execution_count": 39,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"follow_up_response.content[0].text"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "MTWqODRFbFg1"
|
|
},
|
|
"source": [
|
|
"🥳 Congratulations! You've successfully used Claude with tools"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"name": "claude_multimodal_function_calling.ipynb",
|
|
"toc_visible": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|