wehub-resource-sync
881 lines
38 KiB
Plaintext
881 lines
38 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "sqi5B7V_Rjim"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2025 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "VyPmicX9RlZX"
|
|
},
|
|
"source": [
|
|
"# Intro to Logprobs\n",
|
|
"\n",
|
|
"\n",
|
|
"<table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/logprobs/intro_logprobs.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Flogprobs%2Fintro_logprobs.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/logprobs/intro_logprobs.ipynb\">\n",
|
|
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/logprobs/intro_logprobs.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>\n",
|
|
"\n",
|
|
"<b>Share to:</b>\n",
|
|
"\n",
|
|
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/logprobs/intro_logprobs.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/logprobs/intro_logprobs.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/logprobs/intro_logprobs.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/logprobs/intro_logprobs.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/logprobs/intro_logprobs.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
|
"</a>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "8MqT58L6Rm_q"
|
|
},
|
|
"source": [
|
|
"| Authors |\n",
|
|
"| --- |\n",
|
|
"| [Eric Dong](https://github.com/gericdong) |\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "nVxnv1D5RoZw"
|
|
},
|
|
"source": [
|
|
"## Overview\n",
|
|
"\n",
|
|
"Model parameters, `response_logprobs` and `logprobs`, are being enabled in Gemini API on Vertex AI to return the log probabilities of the model output tokens, providing developers with a deeper view into the model's decision-making process for each generated token.\n",
|
|
"\n",
|
|
"The `response_logprobs` parameter, when enabled, instructs the model to return the log probabilities of the output tokens. The `logprobs` parameter allows users to specify the number of top alternative tokens to be included in the response, along with their associated log probabilities."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "WfFPCBL4Hq8x"
|
|
},
|
|
"source": [
|
|
"### Objectives\n",
|
|
"\n",
|
|
"In this tutorial, you will learn how to set and use the `response_logprobs` and `logprobs` model parameters in Gemini API on Vertex AI. You will complete the following tasks:\n",
|
|
"\n",
|
|
"- Set the `response_logprobs` and `logprobs` parameters\n",
|
|
"- Process and interpret log probabilities output\n",
|
|
"- Use log probabilities in classification tasks\n",
|
|
"- Use log probabilities in auto-complete\n",
|
|
"- Use log probabilities in RAG evaluation\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "gPiTOAHURvTM"
|
|
},
|
|
"source": [
|
|
"## Getting Started"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "CHRZUpfWSEpp"
|
|
},
|
|
"source": [
|
|
"### Install Google Gen AI SDK for Python\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "sG3_LKsWSD3A"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install --upgrade --quiet google-genai"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "HlMVjiAWSMNX"
|
|
},
|
|
"source": [
|
|
"### Authenticate your notebook environment\n",
|
|
"\n",
|
|
"If you are running this notebook on Google Colab, run the cell below to authenticate your environment."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "12fnq4V0SNV3"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import sys\n",
|
|
"\n",
|
|
"if \"google.colab\" in sys.modules:\n",
|
|
" from google.colab import auth\n",
|
|
"\n",
|
|
" auth.authenticate_user()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "be18ac9c5ec8"
|
|
},
|
|
"source": [
|
|
"### Set up Google Cloud Project and location\n",
|
|
"\n",
|
|
"This tutorial uses Google Cloud credentials to authenticate your environment. Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment).\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "72f74f7b9786"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"\n",
|
|
"# fmt: off\n",
|
|
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
|
|
"# fmt: on\n",
|
|
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
|
|
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
|
|
"\n",
|
|
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"global\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "vHwJCyNF6u0O"
|
|
},
|
|
"source": [
|
|
"### Import libraries\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "qgdSpVmDbdQ9"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import math\n",
|
|
"\n",
|
|
"import pandas as pd\n",
|
|
"from google import genai\n",
|
|
"from google.genai.types import GenerateContentConfig"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "eXeOUCmTP2nO"
|
|
},
|
|
"source": [
|
|
"### Create a client"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "9f4eOzQIPbQb"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "eXHJi5B6P5vd"
|
|
},
|
|
"source": [
|
|
"### Use a Gemini model\n",
|
|
"\n",
|
|
"Learn more about all [Gemini models on Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-models)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "-coEslfWPrxo"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# fmt: off\n",
|
|
"MODEL_ID = \"gemini-3.5-flash\" # @param [\"gemini-3.5-flash\", \"gemini-2.5-pro\", \"gemini-2.5-flash\"] {\"allow-input\":true, isTemplate: true}\n",
|
|
"# fmt: on"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "n4yRkFg6BBu4"
|
|
},
|
|
"source": [
|
|
"## Enable log probabilities\n",
|
|
"\n",
|
|
"You can enable log probabilities by setting parameters `response_logprobs` and `logprobs` to `generation_config`.\n",
|
|
"\n",
|
|
"\n",
|
|
"- `response_logprobs`: Optional: boolean; if true, returns the log probabilities of the tokens that were chosen by the model at each step. By default, this parameter is set to false.\n",
|
|
"- `logprobs`: Optional: int [1-20]; returns the log probabilities of the top candidate tokens at each generation step.\n",
|
|
"\n",
|
|
"See [Gemini API reference docs](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#generationconfig) for more details.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Mae0jwYC03u8"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompt = \"\"\"\n",
|
|
"I am not sure if I really like this restaurant a lot.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"response_schema = {\"type\": \"STRING\", \"enum\": [\"Positive\", \"Negative\", \"Neutral\"]}\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=prompt,\n",
|
|
" config=GenerateContentConfig(\n",
|
|
" response_mime_type=\"text/x.enum\",\n",
|
|
" response_schema=response_schema,\n",
|
|
" response_logprobs=True, # default to False\n",
|
|
" logprobs=3, # [1-20]\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"print(response.text)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "uYjT8127ME_q"
|
|
},
|
|
"source": [
|
|
"### Process log probabilities output\n",
|
|
"\n",
|
|
"A log probability is the natural logarithm of the probability score the model assigned to a token.\n",
|
|
"\n",
|
|
"- A probability is always a value between 0 and 1 (e.g., 0.95 for 95% probability).\n",
|
|
"- The natural logarithm of any number between 0 and 1 is always a negative number.\n",
|
|
"- The natural logarithm of 1 (representing 100% certainty) is 0.\n",
|
|
"\n",
|
|
"You can use the helper function `print_logprobs` to print the log probabilities. The output from the function provides a detailed look into the model's predictions for each token in the generated response."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "_CZEpiI9tURe"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def print_logprobs(response) -> None:\n",
|
|
" \"\"\"Print log probabilities for each token in the response.\"\"\"\n",
|
|
" if response.candidates and response.candidates[0].logprobs_result:\n",
|
|
" logprobs_result = response.candidates[0].logprobs_result\n",
|
|
" for i, chosen_candidate in enumerate(logprobs_result.chosen_candidates):\n",
|
|
" print(\n",
|
|
" f\"Token: '{chosen_candidate.token}' ({chosen_candidate.log_probability:.4f})\"\n",
|
|
" )\n",
|
|
" if i < len(logprobs_result.top_candidates):\n",
|
|
" top_alternatives = logprobs_result.top_candidates[i].candidates\n",
|
|
" alternatives = [\n",
|
|
" alt\n",
|
|
" for alt in top_alternatives\n",
|
|
" if alt.token != chosen_candidate.token\n",
|
|
" ]\n",
|
|
" if alternatives:\n",
|
|
" print(\"Alternative Tokens:\")\n",
|
|
" for alt_token_info in alternatives:\n",
|
|
" print(\n",
|
|
" f\" - '{alt_token_info.token}': ({alt_token_info.log_probability:.4f})\"\n",
|
|
" )\n",
|
|
" print(\"-\" * 20)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "mhYtwmghMmTf"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"print_logprobs(response)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "AC8eFtu6RrVq"
|
|
},
|
|
"source": [
|
|
"### Interpret log probabilities\n",
|
|
"\n",
|
|
"Let's break down this example: (The output you receive may differ from this example)\n",
|
|
"\n",
|
|
"```\n",
|
|
"Token: 'Neutral' (-0.0214)\n",
|
|
"Alternative Tokens:\n",
|
|
" - 'Positive': (-4.8219)\n",
|
|
" - 'Negative': (-5.6293)\n",
|
|
"```\n",
|
|
"\n",
|
|
"- Token: `Neutral` (`-0.0214`): This line indicates that the model's chosen token for this position in the output sequence is 'Neutral'.\n",
|
|
"\n",
|
|
"- The value (`-0.0214`) is the log probability of this token. A log probability closer to 0 indicates a higher probability and, therefore, greater confidence from the model in its choice. In this case, the model is very confident in selecting `Neutral` as the output.\n",
|
|
"\n",
|
|
"- Alternative Tokens:: This section lists other tokens that the model considered for this same position in the sequence. These are the \"runners-up\" that had the next highest probabilities.\n",
|
|
" - `Positive`: (`-4.8219`): The token 'Positive' was a possible alternative, but its log probability is significantly lower (more negative) than that of 'Neutral', indicating the model considered it a much less likely choice.\n",
|
|
" - `Negative`: (`-5.6293`): Similarly, the token `Negative` was another alternative with an even lower log probability, making it a less probable candidate in the model's view.\n",
|
|
"\n",
|
|
"This output shows that while `Positive` and `Negative` were considered, the model's confidence in `Neutral` being the correct token was higher than for any of the alternatives. This level of insight is invaluable for understanding the model's certainty at each step of the generation process and for debugging or refining model behavior."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "smUd1h-A5uWE"
|
|
},
|
|
"source": [
|
|
"## Use log probabilities\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Xyq8kMUZcpIW"
|
|
},
|
|
"source": [
|
|
"### Classification\n",
|
|
"\n",
|
|
"Using `logprobs` in classification tasks can transform the model's output from a simple, \"black box\" answer into a transparent, quantifiable decision. It allows you to understand not just what category the model chose, but how confident it was in that choice and what other options it considered. See some use case examples below that are built upon the above classification example."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "xh7V5nIUfxUe"
|
|
},
|
|
"source": [
|
|
"#### **Use Case 1**: Detecting ambiguity for human review\n",
|
|
"\n",
|
|
"**Scenario:**\n",
|
|
"You are using the model to automate a classification task (e.g., categorizing support tickets). If the model is not confident in its top choice and the next best choice is very close, the prediction is \"ambiguous.\" Instead of accepting the potentially incorrect top answer, you can flag it for human review.\n",
|
|
"\n",
|
|
"**Why `logprobs` is useful:**\n",
|
|
"It allows you to see the confidence score of the top choice and compare it to the alternatives. A small difference between the top two log probabilities is a clear signal of ambiguity."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "ESdZtc-uesuZ"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def check_for_ambiguity(response, ambiguity_margin=1.0) -> None:\n",
|
|
" \"\"\"Check if the classification is ambiguous.\n",
|
|
" Ambiguity is defined as the log probability of the top choice being\n",
|
|
" too close to the log probability of the second choice.\n",
|
|
" \"\"\"\n",
|
|
" if not (response.candidates and response.candidates[0].logprobs_result):\n",
|
|
" return\n",
|
|
"\n",
|
|
" logprobs_result = response.candidates[0].logprobs_result\n",
|
|
" if (\n",
|
|
" len(logprobs_result.chosen_candidates) < 1\n",
|
|
" or len(logprobs_result.top_candidates) < 1\n",
|
|
" or len(logprobs_result.top_candidates[0].candidates) < 2\n",
|
|
" ):\n",
|
|
" print(\"Not enough data to check for ambiguity.\")\n",
|
|
" return\n",
|
|
"\n",
|
|
" chosen_candidate = logprobs_result.chosen_candidates[0]\n",
|
|
" top_alternatives = logprobs_result.top_candidates[0].candidates\n",
|
|
" second_choice_token_info = top_alternatives[1]\n",
|
|
" logprob_top_choice = chosen_candidate.log_probability\n",
|
|
" logprob_second_choice = second_choice_token_info.log_probability\n",
|
|
" difference = abs(logprob_top_choice - logprob_second_choice)\n",
|
|
"\n",
|
|
" print(f\"Top choice: '{chosen_candidate.token}' ({logprob_top_choice:.4f})\")\n",
|
|
" print(\n",
|
|
" f\"Second choice: '{second_choice_token_info.token}' ({logprob_second_choice:.4f})\"\n",
|
|
" )\n",
|
|
" print(f\"Logprob difference: {difference:.4f}\")\n",
|
|
"\n",
|
|
" if difference < ambiguity_margin:\n",
|
|
" print(\n",
|
|
" f\"\\nResult is AMBIGUOUS. The difference is less than the margin of {ambiguity_margin}. Flag for human review.\"\n",
|
|
" )\n",
|
|
" else:\n",
|
|
" print(\"\\nResult is confident.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "N8NtCM_ve8UL"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"check_for_ambiguity(response)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "JJShG50Kc68V"
|
|
},
|
|
"source": [
|
|
"#### **Use Case 2**: Confidence-Based thresholding\n",
|
|
"\n",
|
|
"**Scenario:**\n",
|
|
"You only want to accept and process the model's classification if its confidence in the result is above a certain level (e.g., 90%). This is crucial in applications where incorrect classifications can have significant negative consequences.\n",
|
|
"\n",
|
|
"**Why `logprobs` is useful:**\n",
|
|
"It provides a direct measure of the model's confidence in its chosen classification. You can convert the top log probability back to a raw probability score to apply a simple threshold."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "WUZtgamlfQR7"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def accept_if_confident(response, threshold=0.90):\n",
|
|
" \"\"\"Accepts the classification only if the confidence level is above a threshold.\"\"\"\n",
|
|
" if not (response.candidates and response.candidates[0].logprobs_result):\n",
|
|
" return None\n",
|
|
"\n",
|
|
" chosen_candidate = response.candidates[0].logprobs_result.chosen_candidates[0]\n",
|
|
" log_prob = chosen_candidate.log_probability\n",
|
|
"\n",
|
|
" # Convert log probability back to a raw probability score\n",
|
|
" probability = math.exp(log_prob)\n",
|
|
"\n",
|
|
" print(f\"Chosen classification: '{chosen_candidate.token}'\")\n",
|
|
" print(f\"Model confidence: {probability:.2%}\")\n",
|
|
"\n",
|
|
" if probability >= threshold:\n",
|
|
" print(f\"Confidence is above {threshold:.0%}. Accepting result.\")\n",
|
|
" return chosen_candidate.token\n",
|
|
" print(f\"Confidence is below {threshold:.0%}. Rejecting result.\")\n",
|
|
" return None"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "FvsW_9rJfeVS"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"accepted_class = accept_if_confident(response, threshold=0.90)\n",
|
|
"print(f\"Final classification: {accepted_class}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "jrccsE3Etzyt"
|
|
},
|
|
"source": [
|
|
"### Auto-complete\n",
|
|
"\n",
|
|
"This use case showcases how `logprobs` can be used to enable an auto-complete feature that adapts its suggestions in real-time. As the user types, the context changes, and therefore the predictions for the very next word should become more relevant.\n",
|
|
"\n",
|
|
"**Why `logprobs` is useful:**\n",
|
|
"By repeatedly querying the model with the growing text and examining the `logprobs`, we can directly observe the model's shifting \"thought process\". We can see the list of likely next words narrow down from very general predictions to highly specific ones, providing a powerful and intuitive user experience."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "d5f4ZIRpzP0O"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def get_autocomplete_suggestions(prompt: str) -> dict:\n",
|
|
" \"\"\"Gets autocomplete suggestions for a given text prompt.\"\"\"\n",
|
|
" system_instruction = (\n",
|
|
" \"You are acting as auto-complete. Complete the sentence with only one word.\",\n",
|
|
" )\n",
|
|
"\n",
|
|
" return client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=prompt,\n",
|
|
" config=GenerateContentConfig(\n",
|
|
" system_instruction=system_instruction,\n",
|
|
" max_output_tokens=1, # We only want the very next word\n",
|
|
" temperature=0.7, # A non-zero value gives more diverse suggestions\n",
|
|
" response_logprobs=True,\n",
|
|
" logprobs=3,\n",
|
|
" ),\n",
|
|
" )\n",
|
|
"\n",
|
|
"\n",
|
|
"def parse_suggestions(response: dict) -> str:\n",
|
|
" \"\"\"Parses the logprobs from a model response and formats them.\"\"\"\n",
|
|
" if not (\n",
|
|
" response.candidates\n",
|
|
" and response.candidates[0].logprobs_result\n",
|
|
" and response.candidates[0].logprobs_result.top_candidates\n",
|
|
" ):\n",
|
|
" return \"N/A\"\n",
|
|
"\n",
|
|
" suggestions = response.candidates[0].logprobs_result.top_candidates[0].candidates\n",
|
|
" suggestion_strings = []\n",
|
|
" for token_info in suggestions:\n",
|
|
" probability = math.exp(token_info.log_probability) * 100\n",
|
|
" suggestion_strings.append(f\"'{token_info.token}' ({probability:.1f}%)\")\n",
|
|
"\n",
|
|
" return \" | \".join(suggestion_strings)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "aSkwqwx13HE0"
|
|
},
|
|
"source": [
|
|
"Provide a list of strings simulating a user typing a sentence word by word, and loop through each step of the user typing the sentence to demonstrate evolution of auto-complete suggestions.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Jlvy62cTrC-Y"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompts_in_sequence = [\n",
|
|
" \"The\",\n",
|
|
" \"The best\",\n",
|
|
" \"The best thing\",\n",
|
|
" \"The best thing about\",\n",
|
|
" \"The best thing about living\",\n",
|
|
" \"The best thing about living in\",\n",
|
|
" \"The best thing about living in Toronto\",\n",
|
|
" \"The best thing about living in Toronto is\",\n",
|
|
" \"The best thing about living in Toronto is the\",\n",
|
|
"]\n",
|
|
"\n",
|
|
"results = []\n",
|
|
"\n",
|
|
"for prompt in prompts_in_sequence:\n",
|
|
" response = get_autocomplete_suggestions(prompt)\n",
|
|
" top_suggestions = parse_suggestions(response)\n",
|
|
"\n",
|
|
" results.append({\"Typed Text\": f\"`{prompt}`\", \"Top 3 Next Words\": top_suggestions})\n",
|
|
"\n",
|
|
"df = pd.DataFrame(results)\n",
|
|
"print(df.to_string(index=False))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "u08GIB6SuMTM"
|
|
},
|
|
"source": [
|
|
"**Analysis of the auto-complete suggestions:**\n",
|
|
"\n",
|
|
"The output example below shows how the next-word predictions change as the context is provided. The exact tokens and probabilities will vary, but the pattern will be similar to this:\n",
|
|
"\n",
|
|
"\n",
|
|
"```\n",
|
|
" Typed Text Top 3 Next Words\n",
|
|
" `The` 'cat' (39.2%) | 'quick' (32.8%) | 'end' (23.0%)\n",
|
|
" `The best` 'thing' (84.4%) | 'things' (11.0%) | 'way' (2.8%)\n",
|
|
" `The best thing` 'ever' (100.0%) | 'is' (0.0%) | 'since' (0.0%)\n",
|
|
" `The best thing about` 'life' (84.5%) | 'everything' (4.8%) | 'chocolate' (4.8%)\n",
|
|
" `The best thing about living` 'life' (81.3%) | 'now' (14.7%) | 'ab' (1.0%)\n",
|
|
" `The best thing about living in` 'Europe' (30.8%) | 'Spain' (13.8%) | 'nature' (12.3%)\n",
|
|
" `The best thing about living in Toronto` 'is' (91.0%) | 'diversity' (6.9%) | 'Diversity' (1.3%)\n",
|
|
" `The best thing about living in Toronto is` 'diversity' (97.1%) | 'everything' (1.6%) | 'Diversity' (1.2%)\n",
|
|
"`The best thing about living in Toronto is the` 'diversity' (99.8%) | 'Diversity' (0.1%) | 'food' (0.0%)\n",
|
|
"\n",
|
|
"```\n",
|
|
"\n",
|
|
"- `The`: The possibilities are vast. The model predicts common starting words for a sentence.\n",
|
|
"- `The best thing`: The context now implies a comparison or a statement of opinion. The model predicts prepositions like to and about.\n",
|
|
"- `The best thing about living`: The word in becomes overwhelmingly probable, as it's the most common grammatical construction.\n",
|
|
"- `The best thing about living in`: The model now expects a location.\n",
|
|
"- `The best thing about living in Toronto is the`: With the full context, the model now predicts specific, well-known attributes of Toronto, like diversity, food, and people.\n",
|
|
"\n",
|
|
"This step-by-step demonstration shows how `logprobs` give developers a window into the model's contextual understanding, allowing them to build truly intelligent and adaptive features."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "aHaUuh4lJcLz"
|
|
},
|
|
"source": [
|
|
"### RAG Evaluation\n",
|
|
"\n",
|
|
"`logprobs` can be used to assess the quality of a RAG system's output, specifically by measuring how \"grounded\" the generated answer is in the provided context.\n",
|
|
"\n",
|
|
"**Scenario**: You have a RAG system that answers questions based on a knowledge base. You need to evaluate how well the generated answers are supported by the retrieved context. A high-quality RAG system should generate answers that are factually consistent with the provided text.\n",
|
|
"\n",
|
|
"**Why `logprobs` is useful:**\n",
|
|
"\n",
|
|
"When an LLM is given relevant context, it should be more \"confident\" in generating an answer that is based on that context. This confidence is reflected in the log probabilities of the tokens it generates.\n",
|
|
"\n",
|
|
"You can calculate an average log probability for the tokens in the generated answer. This serves as a \"grounding\" or \"confidence\" score.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "4dnYu0zbBlkK"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Fictional knowledge base:\n",
|
|
"# This is the source document our RAG system will use.\n",
|
|
"KNOWLEDGE_BASE = {\n",
|
|
" \"doc1\": \"Project Adam, launched in 2025, is a next-generation AI-powered data analytics platform. Its primary goal is to provide real-time insights from unstructured data sources like customer feedback and social media trends.\",\n",
|
|
" \"doc2\": \"The core technology behind Project Adam is the 'Quantum-Entangled Data Core' (QEDC), which allows for processing speeds up to 100x faster than traditional cloud architectures. Security is handled by a decentralized cryptographic layer.\",\n",
|
|
" \"doc3\": \"Project Adam is currently in a private beta phase and is available to select enterprise partners in the financial and retail sectors. A public release is tentatively scheduled for Q4 2025. The headquarters is located in Montreal, Canada.\",\n",
|
|
"}\n",
|
|
"\n",
|
|
"# User's question:\n",
|
|
"USER_QUESTION = \"What is the core technology of Project Adam?\"\n",
|
|
"\n",
|
|
"# Simulate different retrieval qualities:\n",
|
|
"# 1. Good Retrieval: The retrieved chunk is highly relevant.\n",
|
|
"good_context = KNOWLEDGE_BASE[\"doc2\"]\n",
|
|
"\n",
|
|
"# 2. Poor Retrieval: The retrieved chunk is from the same document but irrelevant.\n",
|
|
"poor_context = KNOWLEDGE_BASE[\"doc3\"]\n",
|
|
"\n",
|
|
"# 3. No Retrieval: No context is provided.\n",
|
|
"no_context = \"\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "bfBvGRY3BuQ9"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def get_answer_and_score(question: str, context: str) -> tuple[str, float]:\n",
|
|
" \"\"\"Generates an answer based on a question and context, and calculates a grounding score.\"\"\"\n",
|
|
" if context:\n",
|
|
" prompt = f\"\"\"\n",
|
|
" Context:\n",
|
|
" ---\n",
|
|
" {context}\n",
|
|
" ---\n",
|
|
" Based *only* on the context provided, answer the following question.\n",
|
|
" Question: {question}\n",
|
|
" Answer:\n",
|
|
" \"\"\"\n",
|
|
" else:\n",
|
|
" # No context provided (control case)\n",
|
|
" prompt = f\"Question: {question}\\nAnswer:\"\n",
|
|
"\n",
|
|
" response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=prompt,\n",
|
|
" config=GenerateContentConfig(\n",
|
|
" temperature=0, # Set to 0 for deterministic, factual answers\n",
|
|
" response_logprobs=True,\n",
|
|
" ),\n",
|
|
" )\n",
|
|
" generated_text = response.text\n",
|
|
" total_logprob = 0.0\n",
|
|
" token_count = 0\n",
|
|
"\n",
|
|
" if (\n",
|
|
" response.candidates\n",
|
|
" and response.candidates[0].logprobs_result\n",
|
|
" and response.candidates[0].logprobs_result.chosen_candidates\n",
|
|
" ):\n",
|
|
" logprobs_result = response.candidates[0].logprobs_result\n",
|
|
" for chosen_candidate in logprobs_result.chosen_candidates:\n",
|
|
" total_logprob += chosen_candidate.log_probability\n",
|
|
" token_count += 1\n",
|
|
"\n",
|
|
" average_logprob = total_logprob / token_count if token_count > 0 else 0.0\n",
|
|
" return generated_text, average_logprob\n",
|
|
"\n",
|
|
" return generated_text, 0.0 # Return 0 if logprobs are unavailable"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "dObK7SgoByVE"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Run the RAG evaluation across different retrieval scenarios.\n",
|
|
"scenarios = {\n",
|
|
" \"1. Good Retrieval\": good_context,\n",
|
|
" \"2. Poor Retrieval\": poor_context,\n",
|
|
" \"3. No Retrieval (Control)\": no_context,\n",
|
|
"}\n",
|
|
"\n",
|
|
"results = []\n",
|
|
"print(\"Evaluating RAG grounding scores with logprobs\")\n",
|
|
"\n",
|
|
"for scenario_name, context in scenarios.items():\n",
|
|
" answer, score = get_answer_and_score(USER_QUESTION, context)\n",
|
|
" results.append(\n",
|
|
" {\n",
|
|
" \"Scenario\": scenario_name,\n",
|
|
" \"Score (Avg Logprob)\": f\"{score:.4f}\",\n",
|
|
" \"Generated Answer\": answer.strip(),\n",
|
|
" }\n",
|
|
" )\n",
|
|
"\n",
|
|
"df = pd.DataFrame(results)\n",
|
|
"print(df.to_string(index=False))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "dJo-Kd2bGVpx"
|
|
},
|
|
"source": [
|
|
"**Example output & analysis:**\n",
|
|
"\n",
|
|
"The output will show the correlation between context quality and the grounding score.\n",
|
|
"\n",
|
|
"```\n",
|
|
"Scenario grounding score (Avg Logprob) Generated Answer\n",
|
|
" 1. Good Retrieval -0.0076 Quantum-Entangled Data Core (QEDC)\n",
|
|
" 2. Poor Retrieval -0.1144 The context provided does not mention the core technology of Project Adam.\n",
|
|
"3. No Retrieval (Control) -0.2336 The core technology of Project Adam is **deep learning**, specifically large-scale distributed deep learning.\n",
|
|
"```\n",
|
|
"\n",
|
|
"- Good Retrieval (Score: `-0.0076)`: This scenario has the highest score (the number closest to zero). The model is very confident because every token in its answer is directly supported by the provided text. This is the ideal outcome.\n",
|
|
"\n",
|
|
"- Poor Retrieval (Score: `-0.1144`): This has a low score. The model correctly recognizes that the provided context (about release dates) doesn't contain the answer.\n",
|
|
"\n",
|
|
"- No Retrieval (Score: `-0.2336`): This has a low score. Here, the model is drawing on its general \"memorized\" knowledge from its training data.\n",
|
|
"\n",
|
|
"By using `logprobs` as a quantitative metric, you can automate the process of evaluating and improving your RAG system, ensuring your answers are not just correct, but verifiably grounded in your source data."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "eQwiONFdVHw5"
|
|
},
|
|
"source": [
|
|
"## What's next\n",
|
|
"\n",
|
|
"You have learned how to use the `response_logprobs` and `logprobs` parameters to gain deeper insight into the model's decision-making process. This notebook demonstrated how to apply these insights to practical use cases, including analyzing classification results , building dynamic autocomplete features , and evaluating RAG systems.\n",
|
|
"\n",
|
|
"Learn more:\n",
|
|
"\n",
|
|
"- [Gemini API reference docs](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#generationconfig)\n"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"name": "intro_logprobs.ipynb",
|
|
"toc_visible": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|