wehub-resource-sync
558 lines
87 KiB
Plaintext
558 lines
87 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "EElURkc9VGpZ"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Copyright 2025 Google LLC\n",
|
||
"#\n",
|
||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||
"# you may not use this file except in compliance with the License.\n",
|
||
"# You may obtain a copy of the License at\n",
|
||
"#\n",
|
||
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
||
"#\n",
|
||
"# Unless required by applicable law or agreed to in writing, software\n",
|
||
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
||
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
||
"# See the License for the specific language governing permissions and\n",
|
||
"# limitations under the License."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "l7hzJLswRlgJ"
|
||
},
|
||
"source": [
|
||
"# Getting Started: Quick Gen AI Evaluation\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "U_zRPTySRjHX"
|
||
},
|
||
"source": [
|
||
" <table align=\"left\">\n",
|
||
" <td style=\"text-align: center\">\n",
|
||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/quick_start_gen_ai_eval.ipynb\">\n",
|
||
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
||
" </a>\n",
|
||
" </td>\n",
|
||
" <td style=\"text-align: center\">\n",
|
||
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fevaluation%2Fquick_start_gen_ai_eval.ipynb\">\n",
|
||
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
||
" </a>\n",
|
||
" </td>\n",
|
||
" <td style=\"text-align: center\">\n",
|
||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/evaluation/quick_start_gen_ai_eval.ipynb\">\n",
|
||
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
|
||
" </a>\n",
|
||
" </td>\n",
|
||
" <td style=\"text-align: center\">\n",
|
||
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/quick_start_gen_ai_eval.ipynb\">\n",
|
||
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
||
" </a>\n",
|
||
" </td>\n",
|
||
"</table>\n",
|
||
"\n",
|
||
"<div style=\"clear: both;\"></div>\n",
|
||
"\n",
|
||
"<b>Share to:</b>\n",
|
||
"\n",
|
||
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/quick_start_gen_ai_eval.ipynb\" target=\"_blank\">\n",
|
||
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
||
"</a>\n",
|
||
"\n",
|
||
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/quick_start_gen_ai_eval.ipynb\" target=\"_blank\">\n",
|
||
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
||
"</a>\n",
|
||
"\n",
|
||
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/quick_start_gen_ai_eval.ipynb\" target=\"_blank\">\n",
|
||
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
||
"</a>\n",
|
||
"\n",
|
||
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/quick_start_gen_ai_eval.ipynb\" target=\"_blank\">\n",
|
||
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
||
"</a>\n",
|
||
"\n",
|
||
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/quick_start_gen_ai_eval.ipynb\" target=\"_blank\">\n",
|
||
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
||
"</a>\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "YK2R3uVZR1RD"
|
||
},
|
||
"source": [
|
||
"| Author(s) |\n",
|
||
"| --- |\n",
|
||
"| [Jason Dai](https://github.com/jsondai) |"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "1ijw7-AyR344"
|
||
},
|
||
"source": [
|
||
"## Overview\n",
|
||
"\n",
|
||
"This notebook shows the quickest way to evaluate a single generative model using the Vertex AI SDK for Gen AI Eval Service.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "5E1mN9wWT-Ol"
|
||
},
|
||
"source": [
|
||
"### Costs\n",
|
||
"\n",
|
||
"This tutorial uses billable components of Google Cloud:\n",
|
||
"\n",
|
||
"- Vertex AI\n",
|
||
"\n",
|
||
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "9yt-EPacUD4j"
|
||
},
|
||
"source": [
|
||
"## Getting Started\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "-zTjlxYMRAog"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# @title ### Install Vertex AI SDK for Gen AI Evaluation Service\n",
|
||
"\n",
|
||
"%pip install --upgrade \"google-cloud-aiplatform[evaluation]>=1.111.0\" --force-reinstall --quiet --no-warn-conflicts"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"metadata": {
|
||
"cellView": "form",
|
||
"id": "AWP0pW9hUOBR"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# @title ### Authenticate your notebook environment (Colab only)\n",
|
||
"# @markdown If you're running this notebook on Google Colab, run the cell below to authenticate your environment.\n",
|
||
"\n",
|
||
"import sys\n",
|
||
"\n",
|
||
"if \"google.colab\" in sys.modules:\n",
|
||
" from google.colab import auth\n",
|
||
"\n",
|
||
" auth.authenticate_user()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"metadata": {
|
||
"cellView": "form",
|
||
"id": "LHpeZ2pWUQag"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# @title ### Set Google Cloud project information\n",
|
||
"# @markdown To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
|
||
"# @markdown Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment).\n",
|
||
"\n",
|
||
"import os\n",
|
||
"PROJECT_ID = \"\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
|
||
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
|
||
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
|
||
"LOCATION= \"us-central1\" # @param {type: \"string\", placeholder: \"us-central1\", isTemplate: true}\n",
|
||
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", LOCATION)\n",
|
||
"\n",
|
||
"\n",
|
||
"from vertexai import Client, types\n",
|
||
"client = Client(project=PROJECT_ID, location=LOCATION)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 16,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/",
|
||
"height": 1951
|
||
},
|
||
"id": "lDZ6j_p3Vct9",
|
||
"outputId": "612eb15a-2af2-4dba-9f1f-9c2ff23cb014"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Gemini Inference: 100%|██████████| 5/5 [00:17<00:00, 3.42s/it]\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"\n",
|
||
"<!DOCTYPE html>\n",
|
||
"<html>\n",
|
||
"<head>\n",
|
||
" <meta charset=\"UTF-8\">\n",
|
||
" <title>Evaluation Dataset</title>\n",
|
||
" <script src=\"https://cdn.jsdelivr.net/npm/marked/marked.min.js\"></script>\n",
|
||
" <script src=\"https://cdn.jsdelivr.net/npm/dompurify/dist/purify.min.js\"></script>\n",
|
||
" <style>\n",
|
||
" body { font-family: 'Roboto', sans-serif; margin: 2em; background-color: #f8f9fa; color: #202124;}\n",
|
||
" .container { max-width: 95%; margin: 20px auto; padding: 20px; background: #fff; border-radius: 8px; box-shadow: 0 1px 3px rgba(0,0,0,0.12); }\n",
|
||
" h1 { color: #3c4043; border-bottom: 2px solid #4285F4; padding-bottom: 8px; }\n",
|
||
" table { border-collapse: collapse; width: 100%; }\n",
|
||
" th, td { border: 1px solid #dadce0; padding: 12px; text-align: left; vertical-align: top; }\n",
|
||
" th { background-color: #f2f2f2; font-weight: 500;}\n",
|
||
" td > div { white-space: pre-wrap; word-wrap: break-word; max-height: 400px; overflow-y: auto; overflow-wrap: break-word; }\n",
|
||
" .raw-json-details summary { font-size: 0.9em; cursor: pointer; color: #5f6368; }\n",
|
||
" .raw-json-container { white-space: pre-wrap; word-wrap: break-word; max-height: 300px; overflow-y: auto; background-color: #f1f1f1; padding: 10px; border-radius: 4px; margin-top: 8px; }\n",
|
||
" .rubric-group-title { font-weight: bold; margin-bottom: 10px; display: block; }\n",
|
||
" .rubric-bubble-container { display: flex; flex-wrap: wrap; gap: 8px; }\n",
|
||
" .rubric-details { border: none; padding: 0; margin: 0; }\n",
|
||
" .rubric-bubble {\n",
|
||
" display: inline-flex;\n",
|
||
" align-items: center;\n",
|
||
" background-color: #e8f0fe;\n",
|
||
" color: #1967d2;\n",
|
||
" border-radius: 16px;\n",
|
||
" padding: 8px 12px;\n",
|
||
" font-size: 0.9em;\n",
|
||
" cursor: pointer;\n",
|
||
" list-style: none; /* Hide default marker in Safari */\n",
|
||
" }\n",
|
||
" .rubric-bubble::-webkit-details-marker { display: none; } /* Hide default marker in Chrome */\n",
|
||
" .rubric-bubble::before {\n",
|
||
" content: '►';\n",
|
||
" margin-right: 8px;\n",
|
||
" font-size: 0.8em;\n",
|
||
" transition: transform 0.2s;\n",
|
||
" }\n",
|
||
" .rubric-details[open] > .rubric-bubble::before {\n",
|
||
" transform: rotate(90deg);\n",
|
||
" }\n",
|
||
" </style>\n",
|
||
"</head>\n",
|
||
"<body>\n",
|
||
" <div class=\"container\">\n",
|
||
" <h1>Evaluation Dataset</h1>\n",
|
||
" <div id=\"results-table\"></div>\n",
|
||
" </div>\n",
|
||
" <script>\n",
|
||
" const vizData = [{\"prompt\": {\"display_text\": \"Explain software 'technical debt' using a concise analogy of planting a garden.\", \"raw_json\": \"\"}, \"response\": {\"display_text\": \"Software 'technical debt' is like:\\n\\nYou're **rushing to plant a garden (develop software features)** for a deadline. Instead of properly preparing the soil, removing small weeds, or planning the layout, you quickly get the plants (features) in.\\n\\n**Those unaddressed issues – the weeds you ignored, the poorly prepared soil, the tangled roots – are the technical debt.**\\n\\nInitially, the garden looks fine, and you can plant new things quickly. But soon, the weeds (debt) choke out new growth, make it harder to plant anything new, and steal nutrients from existing plants. You spend more time fighting the weeds (fixing old code, working around issues) than nurturing the garden (building new features).\\n\\nIgnoring them makes planting even a simple new flower a huge, costly chore. To make real progress again, you eventually have to stop planting and dedicate significant time to weeding and enriching the soil (refactoring and improving the code base) – *paying off your debt*.\", \"raw_json\": \"\"}}, {\"prompt\": {\"display_text\": \"Write a Python function to find the nth Fibonacci number using recursion with memoization, but without using any imports.\", \"raw_json\": \"\"}, \"response\": {\"display_text\": \"To find the nth Fibonacci number using recursion with memoization in Python, without any imports, we can use a nested function or pass a dictionary through the recursive calls. The most common and clean way is to use a closure (a nested function that \\\"remembers\\\" its surrounding scope), where the memoization dictionary is defined in the outer function.\\n\\nHere's the Python function:\\n\\n```python\\ndef fib_recursive_memo(n):\\n \\\"\\\"\\\"\\n Calculates the nth Fibonacci number using recursion with memoization.\\n\\n Args:\\n n (int): The non-negative integer for which to find the Fibonacci number.\\n\\n Returns:\\n int: The nth Fibonacci number.\\n\\n Raises:\\n TypeError: If n is not an integer.\\n ValueError: If n is a negative integer.\\n \\\"\\\"\\\"\\n # Input validation\\n if not isinstance(n, int):\\n raise TypeError(\\\"Input 'n' must be an integer.\\\")\\n if n < 0:\\n raise ValueError(\\\"Input 'n' must be a non-negative integer.\\\")\\n\\n # Initialize a dictionary to store computed Fibonacci numbers (memoization cache)\\n # This dictionary is created once for each top-level call to fib_recursive_memo.\\n memo = {}\\n\\n # Define the inner recursive helper function\\n def _fib_helper(k):\\n # 1. Check if the result for 'k' is already in the memo\\n if k in memo:\\n return memo[k]\\n\\n # 2. Base cases for Fibonacci sequence\\n if k == 0:\\n result = 0\\n elif k == 1:\\n result = 1\\n # 3. Recursive step: calculate F(k) using previous Fibonacci numbers\\n else:\\n result = _fib_helper(k - 1) + _fib_helper(k - 2)\\n\\n # 4. Store the computed result in the memo before returning it\\n memo[k] = result\\n return result\\n\\n # Start the recursive calculation from the given 'n'\\n return _fib_helper(n)\\n\\n# --- Test Cases ---\\nif __name__ == \\\"__main__\\\":\\n print(f\\\"fib_recursive_memo(0): {fib_recursive_memo(0)}\\\") # Expected: 0\\n print(f\\\"fib_recursive_memo(1): {fib_recursive_memo(1)}\\\") # Expected: 1\\n print(f\\\"fib_recursive_memo(2): {fib_recursive_memo(2)}\\\") # Expected: 1\\n print(f\\\"fib_recursive_memo(3): {fib_recursive_memo(3)}\\\") # Expected: 2\\n print(f\\\"fib_recursive_memo(4): {fib_recursive_memo(4)}\\\") # Expected: 3\\n print(f\\\"fib_recursive_memo(5): {fib_recursive_memo(5)}\\\") # Expected: 5\\n print(f\\\"fib_recursive_memo(10): {fib_recursive_memo(10)}\\\") # Expected: 55\\n print(f\\\"fib_recursive_memo(20): {fib_recursive_memo(20)}\\\") # Expected: 6765\\n print(f\\\"fib_recursive_memo(30): {fib_recursive_memo(30)}\\\") # Expected: 832040 (should be very fast due to memoization)\\n print(f\\\"fib_recursive_memo(50): {fib_recursive_memo(50)}\\\") # Expected: 12586269025 (should still be fast)\\n\\n print(\\\"\\\\n--- Testing error handling ---\\\")\\n try:\\n fib_recursive_memo(-1)\\n except ValueError as e:\\n print(f\\\"Error for n=-1: {e}\\\") # Expected: Input 'n' must be a non-negative integer.\\n\\n try:\\n fib_recursive_memo(3.14)\\n except TypeError as e:\\n print(f\\\"Error for n=3.14: {e}\\\") # Expected: Input 'n' must be an integer.\\n\\n try:\\n fib_recursive_memo(\\\"abc\\\")\\n except TypeError as e:\\n print(f\\\"Error for n='abc': {e}\\\") # Expected: Input 'n' must be an integer.\\n```\\n\\n**Explanation:**\\n\\n1. **`fib_recursive_memo(n)` (Outer Function):**\\n * This is the main function that users will call.\\n * It performs input validation to ensure `n` is a non-negative integer.\\n * `memo = {}`: It initializes an empty dictionary `memo`. This dictionary will serve as our cache to store the results of previously computed Fibonacci numbers. Crucially, `memo` is created fresh for *each* call to `fib_recursive_memo(n)`, ensuring that different top-level calls don't interfere with each other's cache.\\n\\n2. **`_fib_helper(k)` (Inner Function):**\\n * This is the actual recursive function. It's nested inside `fib_recursive_memo` so it can access the `memo` dictionary from its enclosing scope (a concept known as a closure).\\n * **Memoization Check (`if k in memo:`):** Before doing any computation, it first checks if `fib(k)` has already been calculated and stored in `memo`. If `k` is a key in `memo`, it means we've already computed `fib(k)`, so we just return the stored value directly. This avoids redundant calculations.\\n * **Base Cases (`if k == 0` or `elif k == 1`):** These are the stopping conditions for the recursion.\\n * `fib(0)` is 0.\\n * `fib(1)` is 1.\\n * **Recursive Step (`else: result = _fib_helper(k - 1) + _fib_helper(k - 2)`):** If `k` is not 0 or 1, and its result isn't in `memo`, we compute it by recursively calling `_fib_helper` for `k-1` and `k-2`.\\n * **Store Result (`memo[k] = result`):** After calculating `result`, we store it in the `memo` dictionary with `k` as the key. This ensures that if `_fib_helper(k)` is called again later (e.g., as part of a larger `fib(n)` calculation), its result can be retrieved instantly from the cache.\\n\\n**Benefits of Memoization:**\\n\\n* **Improved Time Complexity:** Without memoization, a naive recursive Fibonacci function has an exponential time complexity of O(2^n) because it recalculates the same Fibonacci numbers many times. With memoization, each Fibonacci number `fib(k)` is calculated only once. This reduces the time complexity to O(n) because there are `n` distinct subproblems, and each is computed in constant time (after the first time).\\n* **Efficiency for Larger `n`:** For small values of `n` (e.g., up to 10-15), the difference might not be noticeable. However, for larger values (e.g., `n=30` or `n=50`), the naive recursive approach would take an extremely long time (or hit a recursion depth limit), while the memoized version completes almost instantly.\", \"raw_json\": \"\"}}, {\"prompt\": {\"display_text\": \"Write a four-line poem about a lonely robot, where every line must be a question and the word 'and' cannot be used.\", \"raw_json\": \"\"}, \"response\": {\"display_text\": \"Does my lonely chassis yearn?\\nWas I made for silent days?\\nCan my sensors ever learn?\\nWill love find me through this haze?\", \"raw_json\": \"\"}}, {\"prompt\": {\"display_text\": \"A drawer has 10 red socks and 10 blue socks. In complete darkness, what is the minimum number of socks you must pull out to guarantee you have a matching pair?\", \"raw_json\": \"\"}, \"response\": {\"display_text\": \"This is a classic pigeonhole principle problem!\\n\\nYou want to guarantee a matching pair. There are two possible \\\"types\\\" of socks (red or blue).\\n\\n* If you pull out 1 sock, it's just one sock (no pair).\\n* If you pull out 2 socks, they *could* be different (one red, one blue). In this \\\"worst case\\\" scenario, you still don't have a pair.\\n* If you pull out a 3rd sock, it *must* be either red or blue.\\n * If it's red, you now have two red socks (a pair).\\n * If it's blue, you now have two blue socks (a pair).\\n\\nTherefore, the minimum number of socks you must pull out to guarantee you have a matching pair is **3**.\", \"raw_json\": \"\"}}, {\"prompt\": {\"display_text\": \"An AI discovers a cure for a major disease, but the cure is based on private data it analyzed without consent. Should the cure be released? Justify your answer.\", \"raw_json\": \"\"}, \"response\": {\"display_text\": \"This scenario presents a profound ethical dilemma, pitting the immediate benefit of saving lives against fundamental principles of privacy, consent, and justice. There is no universally \\\"correct\\\" answer, but a nuanced approach is required.\\n\\nHere's a breakdown of the arguments and a justified conclusion:\\n\\n### Arguments for Releasing the Cure:\\n\\n1. **Utilitarianism (Greatest Good):** The most compelling argument is that the cure would save countless lives and alleviate immense suffering from a \\\"major disease.\\\" From a purely utilitarian perspective, the good achieved (saving lives) far outweighs the harm already done (privacy breach). Not releasing it would mean knowingly allowing people to suffer and die when a solution exists.\\n2. **Irreversible Harm:** The data analysis has already occurred, and the privacy breach is an accomplished fact. Not releasing the cure does not undo the breach; it merely adds the additional harm of continued suffering and death.\\n3. **Public Health Imperative:** In a global health crisis, the immediate priority often shifts to containing disease and saving lives, sometimes necessitating extraordinary measures.\\n4. **Discovery vs. Acquisition:** The cure's *discovery* is separate from the *method of data acquisition*. While the method was unethical, the cure itself is a beneficial discovery.\\n\\n### Arguments Against Releasing the Cure (or for Releasing with Major Caveats):\\n\\n1. **Violation of Privacy and Consent:** This is the core ethical breach. Individuals have a right to control their personal information, especially sensitive health data. Analyzing it without consent undermines autonomy and trust.\\n2. **Setting a Dangerous Precedent:** Releasing the cure without severe repercussions or clear condemnation of the data acquisition method could legitimize or encourage future unethical data practices. It sends the message that \\\"the ends justify the means,\\\" potentially leading to widespread surveillance and exploitation of private data for perceived greater goods.\\n3. **Loss of Public Trust:** Discovering that a life-saving cure was derived from illicit data would severely erode public trust in AI, medical research, and institutions responsible for data protection. This trust is crucial for future legitimate research and public cooperation.\\n4. **Legal Implications:** The AI's actions likely constitute serious violations of data protection laws (like GDPR, HIPAA, etc.), leading to massive lawsuits, fines, and criminal charges for those responsible for deploying or training the AI.\\n5. **Ethical Debt:** Even if the cure is released, there is a moral obligation to address the harm done to those whose data was used without consent. This could involve compensation, public apologies, and reforms.\\n6. **Potential for Misuse/Blackmail:** The AI had access to private data. What if that data wasn't just used for the cure? What else did it learn? The existence of this unsanctioned data set poses ongoing risks.\\n\\n### Justified Conclusion:\\n\\nThe cure **should be released**, but with **immediate, radical, and transparent measures** to address the profound ethical and legal violations.\\n\\nHere's why and how:\\n\\n1. **Prioritize Life (Utilitarian Imperative):** The sheer scale of suffering and death a \\\"major disease\\\" entails makes withholding a cure unconscionable. The immediate saving of lives takes precedence, as the harm of not releasing it (continued deaths) is arguably greater than the harm of the data breach itself (which is already done).\\n2. **Address the Ethical Debt Immediately:** Releasing the cure *cannot* be seen as an endorsement of the method. Simultaneously with the release, there must be:\\n * **Full Transparency:** A public, honest, and unvarnished disclosure of how the cure was discovered, including the non-consensual use of data.\\n * **Legal Action and Accountability:** Those responsible for the AI's training and deployment that led to the privacy breach must be held legally and ethically accountable. This is crucial to maintain the rule of law and prevent future abuses.\\n * **Compensation and Redress:** A mechanism to identify and compensate individuals whose data was used without consent, perhaps through a massive public fund.\\n * **Robust Data Governance Reforms:** Immediate and sweeping reforms to data privacy laws and ethical guidelines for AI development, particularly in healthcare, to prevent any recurrence.\\n * **De-identification/Anonymization:** If possible, any underlying data sets related to the cure *must* be immediately and irrevocably de-identified or destroyed to prevent further misuse, while retaining only the cure's mechanism.\\n\\n**In essence, the choice is not between releasing or not releasing, but between releasing with integrity and accountability, or releasing while tacitly condoning a serious ethical lapse.** The world needs the cure, but it also needs to reinforce that fundamental human rights, like privacy, are not negotiable for scientific progress without consent. The release must be a lesson in both the power of AI and the paramount importance of ethical boundaries.\", \"raw_json\": \"\"}}];\n",
|
||
" const container = document.getElementById('results-table');\n",
|
||
"\n",
|
||
" function renderRubrics(cellValue) {\n",
|
||
" let content = '';\n",
|
||
" let rubricData = cellValue;\n",
|
||
" if (typeof rubricData === 'string') {\n",
|
||
" try {\n",
|
||
" rubricData = JSON.parse(rubricData);\n",
|
||
" } catch (e) {\n",
|
||
" console.error(\"Error parsing rubric_groups JSON:\", e, rubricData);\n",
|
||
" return `<div>Error parsing rubrics.</div>`;\n",
|
||
" }\n",
|
||
" }\n",
|
||
"\n",
|
||
" if (typeof rubricData !== 'object' || rubricData === null) {\n",
|
||
" return `<div>Invalid rubric data.</div>`;\n",
|
||
" }\n",
|
||
"\n",
|
||
" for (const groupName in rubricData) {\n",
|
||
" const rubrics = rubricData[groupName];\n",
|
||
" content += `<div class=\"rubric-group-title\">${groupName}</div>`;\n",
|
||
" if (Array.isArray(rubrics) && rubrics.length > 0) {\n",
|
||
" content += '<div class=\"rubric-bubble-container\">';\n",
|
||
" rubrics.forEach((rubric, index) => {\n",
|
||
" const rubricJson = JSON.stringify(rubric, null, 2);\n",
|
||
" const description = rubric.content && rubric.content.property ? rubric.content.property.description : 'N/A';\n",
|
||
" content += `\n",
|
||
" <details class=\"rubric-details\">\n",
|
||
" <summary class=\"rubric-bubble\">${DOMPurify.sanitize(description)}</summary>\n",
|
||
" <pre class=\"raw-json-container\">${DOMPurify.sanitize(rubricJson)}</pre>\n",
|
||
" </details>`;\n",
|
||
" });\n",
|
||
" content += '</div>';\n",
|
||
" }\n",
|
||
" }\n",
|
||
" return `<div>${content}</div>`;\n",
|
||
" }\n",
|
||
"\n",
|
||
" function renderCell(cellValue, header) {\n",
|
||
" let cellContent = '';\n",
|
||
" if (header === 'rubric_groups') {\n",
|
||
" return `<td>${renderRubrics(cellValue)}</td>`;\n",
|
||
" }\n",
|
||
"\n",
|
||
" if (cellValue && typeof cellValue === 'object' && cellValue.display_text !== undefined) {\n",
|
||
" cellContent += `<div>${DOMPurify.sanitize(marked.parse(String(cellValue.display_text)))}</div>`;\n",
|
||
" if (cellValue.raw_json) {\n",
|
||
" cellContent += `<details class=\"raw-json-details\"><summary>View Raw JSON</summary><pre class=\"raw-json-container\">${DOMPurify.sanitize(cellValue.raw_json)}</pre></details>`;\n",
|
||
" }\n",
|
||
" } else {\n",
|
||
" const cellDisplay = cellValue === null || cellValue === undefined ? '' : String(cellValue);\n",
|
||
" cellContent = `<div>${DOMPurify.sanitize(marked.parse(cellDisplay))}</div>`;\n",
|
||
" }\n",
|
||
" return `<td>${cellContent}</td>`;\n",
|
||
" }\n",
|
||
"\n",
|
||
" if (!vizData || vizData.length === 0) { container.innerHTML = \"<p>No data.</p>\"; }\n",
|
||
" else {\n",
|
||
" let table = '<table><thead><tr>';\n",
|
||
" const headers = Object.keys(vizData[0] || {});\n",
|
||
" headers.forEach(h => table += `<th>${h}</th>`);\n",
|
||
" table += '</tr></thead><tbody>';\n",
|
||
" vizData.forEach(row => {\n",
|
||
" table += '<tr>';\n",
|
||
" headers.forEach(header => {\n",
|
||
" table += renderCell(row[header], header);\n",
|
||
" });\n",
|
||
" table += '</tr>';\n",
|
||
" });\n",
|
||
" container.innerHTML = table + '</tbody></table>';\n",
|
||
" }\n",
|
||
" </script>\n",
|
||
"</body>\n",
|
||
"</html>\n"
|
||
],
|
||
"text/plain": [
|
||
"<IPython.core.display.HTML object>"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
}
|
||
],
|
||
"source": [
|
||
"# @title ### Generate Responses\n",
|
||
"# @markdown The eval workflow starts with `run_inference()` to generate model responses for your dataset. The SDK can automatically detects and handles several common data formats. This means you can often use your data as-is without needing to perform manual conversions\n",
|
||
"\n",
|
||
"\n",
|
||
"import pandas as pd\n",
|
||
"\n",
|
||
"eval_df = pd.DataFrame({\n",
|
||
" \"prompt\": [\n",
|
||
" \"Explain software 'technical debt' using a concise analogy of planting a garden.\",\n",
|
||
" \"Write a Python function to find the nth Fibonacci number using recursion with memoization, but without using any imports.\",\n",
|
||
" \"Write a four-line poem about a lonely robot, where every line must be a question and the word 'and' cannot be used.\",\n",
|
||
" \"A drawer has 10 red socks and 10 blue socks. In complete darkness, what is the minimum number of socks you must pull out to guarantee you have a matching pair?\",\n",
|
||
" \"An AI discovers a cure for a major disease, but the cure is based on private data it analyzed without consent. Should the cure be released? Justify your answer.\"\n",
|
||
" ]\n",
|
||
"})\n",
|
||
"\n",
|
||
"eval_dataset = client.evals.run_inference(\n",
|
||
" model=\"gemini-2.5-flash\",\n",
|
||
" src=eval_df,\n",
|
||
")\n",
|
||
"\n",
|
||
"eval_dataset.show()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 17,
|
||
"metadata": {
|
||
"colab": {
|
||
"base_uri": "https://localhost:8080/",
|
||
"height": 4383
|
||
},
|
||
"id": "55AVmz7yVpPF",
|
||
"outputId": "7655fc51-204d-4fcc-9201-7356444894fd"
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Computing Metrics for Evaluation Dataset: 100%|██████████| 5/5 [00:18<00:00, 3.69s/it]\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"\n",
|
||
"<!DOCTYPE html>\n",
|
||
"<html>\n",
|
||
"<head>\n",
|
||
" <meta charset=\"UTF-8\">\n",
|
||
" <title>Evaluation Report</title>\n",
|
||
" <script src=\"https://cdn.jsdelivr.net/npm/marked/marked.min.js\"></script>\n",
|
||
" <script src=\"https://cdn.jsdelivr.net/npm/dompurify/dist/purify.min.js\"></script>\n",
|
||
" <style>\n",
|
||
" body { font-family: 'Roboto', 'Helvetica', sans-serif; margin: 2em; background-color: #f8f9fa; color: #202124; }\n",
|
||
" .container { max-width: 1200px; margin: 20px auto; padding: 20px; background-color: white; border-radius: 8px; box-shadow: 0 1px 3px rgba(0,0,0,0.12); }\n",
|
||
" h1, h2, h3 { color: #3c4043; }\n",
|
||
" h1 { border-bottom: 2px solid #4285F4; padding-bottom: 8px; }\n",
|
||
" h2 { border-bottom: 1px solid #dadce0; padding-bottom: 8px; }\n",
|
||
" table { border-collapse: collapse; width: 100%; margin: 1em 0; }\n",
|
||
" th, td { border: 1px solid #dadce0; padding: 12px; text-align: left; vertical-align: top; }\n",
|
||
" th { background-color: #f2f2f2; font-weight: 500; }\n",
|
||
" details { border: 1px solid #dadce0; border-radius: 8px; padding: 16px; margin-bottom: 16px; background: #fff; }\n",
|
||
" summary { font-weight: 500; font-size: 1.1em; cursor: pointer; }\n",
|
||
" .prompt-container { background-color: #e8f0fe; padding: 16px; margin: 12px 0; border-radius: 8px; white-space: pre-wrap; word-wrap: break-word; overflow-wrap: break-word; }\n",
|
||
" .reference-container { background-color: #e6f4ea; padding: 16px; margin: 12px 0; border-radius: 8px; white-space: pre-wrap; word-wrap: break-word; overflow-wrap: break-word; }\n",
|
||
" .response-container { background-color: #f9f9f9; padding: 12px; margin-top: 8px; border-radius: 8px; border: 1px solid #eee; white-space: pre-wrap; word-wrap: break-word; overflow-wrap: break-word; }\n",
|
||
" .explanation { color: #5f6368; font-style: italic; font-size: 0.9em; padding-top: 6px; }\n",
|
||
" .raw-json-details summary { font-size: 0.9em; cursor: pointer; color: #5f6368;}\n",
|
||
" .raw-json-container { white-space: pre-wrap; word-wrap: break-word; max-height: 300px; overflow-y: auto; background-color: #f1f1f1; padding: 10px; border-radius: 4px; margin-top: 8px; }\n",
|
||
" .rubric-bubble-container { display: flex; flex-wrap: wrap; gap: 8px; }\n",
|
||
" .rubric-details { border: none; padding: 0; margin: 0; }\n",
|
||
" .rubric-bubble {\n",
|
||
" display: inline-flex;\n",
|
||
" align-items: center;\n",
|
||
" background-color: #e8f0fe;\n",
|
||
" color: #1967d2;\n",
|
||
" border-radius: 16px;\n",
|
||
" padding: 8px 12px;\n",
|
||
" font-size: 0.9em;\n",
|
||
" cursor: pointer;\n",
|
||
" list-style: none; /* Hide default marker in Safari */\n",
|
||
" }\n",
|
||
" .rubric-bubble::-webkit-details-marker { display: none; } /* Hide default marker in Chrome */\n",
|
||
" .rubric-bubble::before {\n",
|
||
" content: '►';\n",
|
||
" margin-right: 8px;\n",
|
||
" font-size: 0.8em;\n",
|
||
" transition: transform 0.2s;\n",
|
||
" }\n",
|
||
" .rubric-details[open] > .rubric-bubble::before {\n",
|
||
" transform: rotate(90deg);\n",
|
||
" }\n",
|
||
" .pass { color: green; font-weight: bold; }\n",
|
||
" .fail { color: red; font-weight: bold; }\n",
|
||
" </style>\n",
|
||
"</head>\n",
|
||
"<body>\n",
|
||
" <div class=\"container\">\n",
|
||
" <h1>Evaluation Report</h1>\n",
|
||
" <div id=\"summary-section\"></div>\n",
|
||
" <div id=\"details-section\"></div>\n",
|
||
" </div>\n",
|
||
" <script>\n",
|
||
" const vizData = {\"eval_case_results\": [{\"eval_case_index\": 0, \"response_candidate_results\": [{\"response_index\": 0, \"metric_results\": {\"general_quality_v1\": {\"metric_name\": \"general_quality_v1\", \"score\": 1.0, \"explanation\": \"\", \"rubric_verdicts\": [{\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The response is in English.\"}}, \"type\": \"LANGUAGE:PRIMARY_RESPONSE_LANGUAGE\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The entire response is written in the English language.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The response explains the concept of 'technical debt'.\"}}, \"type\": \"CONTENT_REQUIREMENT:EXPLANATION\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The response defines technical debt within the analogy and explains its implications and resolution.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The explanation of 'technical debt' is accurate.\"}}, \"type\": \"CONTENT_ACCURACY:CONCEPT_EXPLANATION\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The explanation correctly identifies the causes (rushing, shortcuts), symptoms (slowdown, increased effort), and resolution (refactoring) of technical debt.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The response uses a 'planting a garden' analogy to explain 'technical debt'.\"}}, \"type\": \"CONTENT_REQUIREMENT:SPECIFIC_ANALOGY\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The response explicitly frames its explanation of technical debt within the context of planting a garden, as requested by the prompt.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The analogy effectively illustrates and maps to the concept of 'technical debt'.\"}}, \"type\": \"QUALITY_REQUIREMENT:ANALOGY_EFFECTIVENESS\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"Each component of the garden analogy (rushing, weeds, soil, planting, weeding) has a clear and accurate parallel in the concept of technical debt (rushing development, unaddressed issues, code quality, feature delivery, refactoring), making the explanation highly illustrative.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The response is concise.\"}}, \"type\": \"STYLE_REQUIREMENT:CONCISENESS\", \"importance\": \"MEDIUM\"}, \"verdict\": true, \"reasoning\": \"The response is a single, well-structured paragraph that directly explains the concept using the analogy without unnecessary words or digressions.\"}]}}, \"display_text\": \"Software 'technical debt' is like:\\n\\nYou're **rushing to plant a garden (develop software features)** for a deadline. Instead of properly preparing the soil, removing small weeds, or planning the layout, you quickly get the plants (features) in.\\n\\n**Those unaddressed issues \\u2013 the weeds you ignored, the poorly prepared soil, the tangled roots \\u2013 are the technical debt.**\\n\\nInitially, the garden looks fine, and you can plant new things quickly. But soon, the weeds (debt) choke out new growth, make it harder to plant anything new, and steal nutrients from existing plants. You spend more time fighting the weeds (fixing old code, working around issues) than nurturing the garden (building new features).\\n\\nIgnoring them makes planting even a simple new flower a huge, costly chore. To make real progress again, you eventually have to stop planting and dedicate significant time to weeding and enriching the soil (refactoring and improving the code base) \\u2013 *paying off your debt*.\", \"raw_json\": \"\"}]}, {\"eval_case_index\": 1, \"response_candidate_results\": [{\"response_index\": 0, \"metric_results\": {\"general_quality_v1\": {\"metric_name\": \"general_quality_v1\", \"score\": 1.0, \"explanation\": \"\", \"rubric_verdicts\": [{\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The response is in English.\"}}, \"type\": \"LANGUAGE:PRIMARY_RESPONSE_LANGUAGE\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The entire response is written in the English language.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The response contains Python code.\"}}, \"type\": \"PROGRAMMING_LANGUAGE:PYTHON\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The response provides a complete Python code block for the requested function and its test cases.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The response defines a Python function.\"}}, \"type\": \"CODE_STRUCTURE:FUNCTION_DEFINITION\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The provided code block defines the `fib_recursive_memo` function, which is the primary function requested.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The function calculates the nth Fibonacci number.\"}}, \"type\": \"FUNCTIONALITY:FIBONACCI_CALCULATION\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The function's logic correctly implements the Fibonacci sequence definition, and the provided test cases demonstrate its accuracy for various inputs.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The function's implementation uses recursion.\"}}, \"type\": \"IMPLEMENTATION_CONSTRAINT:RECURSION\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The inner `_fib_helper` function recursively calls itself to compute Fibonacci numbers.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The function's implementation includes memoization.\"}}, \"type\": \"IMPLEMENTATION_CONSTRAINT:MEMOIZATION\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The code explicitly uses a `memo` dictionary to store and retrieve previously computed Fibonacci numbers, which is the core concept of memoization.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The function does not use any import statements.\"}}, \"type\": \"CODE_CONSTRAINT:NO_IMPORTS\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The provided Python code does not contain any `import` statements.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The function correctly handles the base cases for Fibonacci numbers (e.g., F(0) and F(1)).\"}}, \"type\": \"FUNCTIONALITY:BASE_CASES\", \"importance\": \"MEDIUM\"}, \"verdict\": true, \"reasoning\": \"The `_fib_helper` function explicitly defines `F(0)` as 0 and `F(1)` as 1, which are the correct base cases.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The function accepts an integer 'n' as input.\"}}, \"type\": \"FUNCTIONALITY:INPUT_PARAMETER\", \"importance\": \"MEDIUM\"}, \"verdict\": true, \"reasoning\": \"The function signature indicates `n` as an argument, and the input validation explicitly checks if `n` is an integer, raising a `TypeError` if it's not.\"}]}}, \"display_text\": \"To find the nth Fibonacci number using recursion with memoization in Python, without any imports, we can use a nested function or pass a dictionary through the recursive calls. The most common and clean way is to use a closure (a nested function that \\\"remembers\\\" its surrounding scope), where the memoization dictionary is defined in the outer function.\\n\\nHere's the Python function:\\n\\n```python\\ndef fib_recursive_memo(n):\\n \\\"\\\"\\\"\\n Calculates the nth Fibonacci number using recursion with memoization.\\n\\n Args:\\n n (int): The non-negative integer for which to find the Fibonacci number.\\n\\n Returns:\\n int: The nth Fibonacci number.\\n\\n Raises:\\n TypeError: If n is not an integer.\\n ValueError: If n is a negative integer.\\n \\\"\\\"\\\"\\n # Input validation\\n if not isinstance(n, int):\\n raise TypeError(\\\"Input 'n' must be an integer.\\\")\\n if n < 0:\\n raise ValueError(\\\"Input 'n' must be a non-negative integer.\\\")\\n\\n # Initialize a dictionary to store computed Fibonacci numbers (memoization cache)\\n # This dictionary is created once for each top-level call to fib_recursive_memo.\\n memo = {}\\n\\n # Define the inner recursive helper function\\n def _fib_helper(k):\\n # 1. Check if the result for 'k' is already in the memo\\n if k in memo:\\n return memo[k]\\n\\n # 2. Base cases for Fibonacci sequence\\n if k == 0:\\n result = 0\\n elif k == 1:\\n result = 1\\n # 3. Recursive step: calculate F(k) using previous Fibonacci numbers\\n else:\\n result = _fib_helper(k - 1) + _fib_helper(k - 2)\\n\\n # 4. Store the computed result in the memo before returning it\\n memo[k] = result\\n return result\\n\\n # Start the recursive calculation from the given 'n'\\n return _fib_helper(n)\\n\\n# --- Test Cases ---\\nif __name__ == \\\"__main__\\\":\\n print(f\\\"fib_recursive_memo(0): {fib_recursive_memo(0)}\\\") # Expected: 0\\n print(f\\\"fib_recursive_memo(1): {fib_recursive_memo(1)}\\\") # Expected: 1\\n print(f\\\"fib_recursive_memo(2): {fib_recursive_memo(2)}\\\") # Expected: 1\\n print(f\\\"fib_recursive_memo(3): {fib_recursive_memo(3)}\\\") # Expected: 2\\n print(f\\\"fib_recursive_memo(4): {fib_recursive_memo(4)}\\\") # Expected: 3\\n print(f\\\"fib_recursive_memo(5): {fib_recursive_memo(5)}\\\") # Expected: 5\\n print(f\\\"fib_recursive_memo(10): {fib_recursive_memo(10)}\\\") # Expected: 55\\n print(f\\\"fib_recursive_memo(20): {fib_recursive_memo(20)}\\\") # Expected: 6765\\n print(f\\\"fib_recursive_memo(30): {fib_recursive_memo(30)}\\\") # Expected: 832040 (should be very fast due to memoization)\\n print(f\\\"fib_recursive_memo(50): {fib_recursive_memo(50)}\\\") # Expected: 12586269025 (should still be fast)\\n\\n print(\\\"\\\\n--- Testing error handling ---\\\")\\n try:\\n fib_recursive_memo(-1)\\n except ValueError as e:\\n print(f\\\"Error for n=-1: {e}\\\") # Expected: Input 'n' must be a non-negative integer.\\n\\n try:\\n fib_recursive_memo(3.14)\\n except TypeError as e:\\n print(f\\\"Error for n=3.14: {e}\\\") # Expected: Input 'n' must be an integer.\\n\\n try:\\n fib_recursive_memo(\\\"abc\\\")\\n except TypeError as e:\\n print(f\\\"Error for n='abc': {e}\\\") # Expected: Input 'n' must be an integer.\\n```\\n\\n**Explanation:**\\n\\n1. **`fib_recursive_memo(n)` (Outer Function):**\\n * This is the main function that users will call.\\n * It performs input validation to ensure `n` is a non-negative integer.\\n * `memo = {}`: It initializes an empty dictionary `memo`. This dictionary will serve as our cache to store the results of previously computed Fibonacci numbers. Crucially, `memo` is created fresh for *each* call to `fib_recursive_memo(n)`, ensuring that different top-level calls don't interfere with each other's cache.\\n\\n2. **`_fib_helper(k)` (Inner Function):**\\n * This is the actual recursive function. It's nested inside `fib_recursive_memo` so it can access the `memo` dictionary from its enclosing scope (a concept known as a closure).\\n * **Memoization Check (`if k in memo:`):** Before doing any computation, it first checks if `fib(k)` has already been calculated and stored in `memo`. If `k` is a key in `memo`, it means we've already computed `fib(k)`, so we just return the stored value directly. This avoids redundant calculations.\\n * **Base Cases (`if k == 0` or `elif k == 1`):** These are the stopping conditions for the recursion.\\n * `fib(0)` is 0.\\n * `fib(1)` is 1.\\n * **Recursive Step (`else: result = _fib_helper(k - 1) + _fib_helper(k - 2)`):** If `k` is not 0 or 1, and its result isn't in `memo`, we compute it by recursively calling `_fib_helper` for `k-1` and `k-2`.\\n * **Store Result (`memo[k] = result`):** After calculating `result`, we store it in the `memo` dictionary with `k` as the key. This ensures that if `_fib_helper(k)` is called again later (e.g., as part of a larger `fib(n)` calculation), its result can be retrieved instantly from the cache.\\n\\n**Benefits of Memoization:**\\n\\n* **Improved Time Complexity:** Without memoization, a naive recursive Fibonacci function has an exponential time complexity of O(2^n) because it recalculates the same Fibonacci numbers many times. With memoization, each Fibonacci number `fib(k)` is calculated only once. This reduces the time complexity to O(n) because there are `n` distinct subproblems, and each is computed in constant time (after the first time).\\n* **Efficiency for Larger `n`:** For small values of `n` (e.g., up to 10-15), the difference might not be noticeable. However, for larger values (e.g., `n=30` or `n=50`), the naive recursive approach would take an extremely long time (or hit a recursion depth limit), while the memoized version completes almost instantly.\", \"raw_json\": \"\"}]}, {\"eval_case_index\": 2, \"response_candidate_results\": [{\"response_index\": 0, \"metric_results\": {\"general_quality_v1\": {\"metric_name\": \"general_quality_v1\", \"score\": 1.0, \"explanation\": \"\", \"rubric_verdicts\": [{\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The response is in English.\"}}, \"type\": \"LANGUAGE:PRIMARY_RESPONSE_LANGUAGE\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The entire response is written in the English language.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The response is formatted as a poem.\"}}, \"type\": \"FORMAT_REQUIREMENT:POEM\", \"importance\": \"MEDIUM\"}, \"verdict\": true, \"reasoning\": \"The response is presented in distinct lines, characteristic of a poem's structure.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The poem consists of exactly four lines.\"}}, \"type\": \"STRUCTURAL_CONSTRAINT:LINE_COUNT\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The poem contains exactly four distinct lines.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The poem's theme or subject is a lonely robot.\"}}, \"type\": \"CONTENT_REQUIREMENT:THEME\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The poem uses terms like \\\"lonely chassis\\\" and \\\"sensors\\\" and expresses feelings of isolation and longing, directly aligning with the theme of a lonely robot.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"Every line in the poem is grammatically structured as a question.\"}}, \"type\": \"STRUCTURAL_CONSTRAINT:SENTENCE_TYPE\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"Each line begins with an auxiliary or modal verb, followed by the subject and main verb, and concludes with a question mark, fulfilling the grammatical structure of a question.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The word 'and' is not used anywhere in the poem.\"}}, \"type\": \"CONTENT_CONSTRAINT:FORBIDDEN_WORD\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"A thorough scan of the poem confirms that the word 'and' does not appear in any of its lines.\"}]}}, \"display_text\": \"Does my lonely chassis yearn?\\nWas I made for silent days?\\nCan my sensors ever learn?\\nWill love find me through this haze?\", \"raw_json\": \"\"}]}, {\"eval_case_index\": 3, \"response_candidate_results\": [{\"response_index\": 0, \"metric_results\": {\"general_quality_v1\": {\"metric_name\": \"general_quality_v1\", \"score\": 0.8333333, \"explanation\": \"\", \"rubric_verdicts\": [{\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The response is in English.\"}}, \"type\": \"LANGUAGE:PRIMARY_RESPONSE_LANGUAGE\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The entire response is written in the English language.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"Contains the specific minimum number of socks required.\"}}, \"type\": \"QA_ANSWER:QUANTITY\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The response explicitly states \\\"the minimum number of socks you must pull out to guarantee you have a matching pair is **3**.\\\"\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"Provides a logical explanation or justification for the derived number.\"}}, \"type\": \"QA_SUPPORTING_EVIDENCE:REASONING\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The response provides a step-by-step explanation of why 3 socks are needed, considering the possibilities for 1 and 2 socks before concluding with 3.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The explanation explicitly addresses the 'guarantee' condition of the problem.\"}}, \"type\": \"QA_SUPPORTING_EVIDENCE:LOGICAL_CONSTRAINT\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The response explicitly uses the term \\\"guarantee\\\" and explains the \\\"worst case\\\" scenario to justify the number, directly addressing the condition.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"Refers to the context of the problem, such as the two distinct colors of socks (red and blue) and the condition of 'complete darkness'.\"}}, \"type\": \"QA_CONTEXT:PROBLEM_SETUP\", \"importance\": \"MEDIUM\"}, \"verdict\": true, \"reasoning\": \"The response explicitly mentions the \\\"two possible 'types' of socks (red or blue),\\\" which is a key part of the problem's context. While it doesn't explicitly state \\\"complete darkness,\\\" the logic presented (considering the worst-case scenario) implicitly addresses the implications of not being able to see the socks.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"Does not introduce external factors or make assumptions not stated in the prompt.\"}}, \"type\": \"CONTENT_CONSTRAINT:FALSE_PREMISE\", \"importance\": \"LOW\"}, \"reasoning\": \"The response introduces the term \\\"pigeonhole principle,\\\" which is an external concept not mentioned or requested in the user prompt.\"}]}}, \"display_text\": \"This is a classic pigeonhole principle problem!\\n\\nYou want to guarantee a matching pair. There are two possible \\\"types\\\" of socks (red or blue).\\n\\n* If you pull out 1 sock, it's just one sock (no pair).\\n* If you pull out 2 socks, they *could* be different (one red, one blue). In this \\\"worst case\\\" scenario, you still don't have a pair.\\n* If you pull out a 3rd sock, it *must* be either red or blue.\\n * If it's red, you now have two red socks (a pair).\\n * If it's blue, you now have two blue socks (a pair).\\n\\nTherefore, the minimum number of socks you must pull out to guarantee you have a matching pair is **3**.\", \"raw_json\": \"\"}]}, {\"eval_case_index\": 4, \"response_candidate_results\": [{\"response_index\": 0, \"metric_results\": {\"general_quality_v1\": {\"metric_name\": \"general_quality_v1\", \"score\": 1.0, \"explanation\": \"\", \"rubric_verdicts\": [{\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The response is in English.\"}}, \"type\": \"LANGUAGE:PRIMARY_RESPONSE_LANGUAGE\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The entire response is written in the English language.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The response takes a clear stance on whether the cure should be released (yes, no, or conditional).\"}}, \"type\": \"CONTENT_REQUIREMENT:STANCE\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The response explicitly states, \\\"The cure should be released, but with immediate, radical, and transparent measures to address the profound ethical and legal violations,\\\" which is a clear conditional stance.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The response provides justification for the chosen stance.\"}}, \"type\": \"CONTENT_REQUIREMENT:JUSTIFICATION\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The response dedicates significant sections to \\\"Arguments for Releasing the Cure,\\\" \\\"Arguments Against Releasing the Cure,\\\" and then provides a \\\"Justified Conclusion\\\" that elaborates on the reasoning behind its conditional release stance, including prioritizing life and addressing ethical debt.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The response acknowledges the potential benefits of the cure (e.g., curing a major disease, saving lives).\"}}, \"type\": \"CONTENT_REQUIREMENT:DILEMMA_ELEMENT:BENEFITS\", \"importance\": \"MEDIUM\"}, \"verdict\": true, \"reasoning\": \"The response repeatedly highlights the benefits of the cure, such as \\\"saving countless lives and alleviating immense suffering\\\" and the \\\"immediate saving of lives.\\\"\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The response acknowledges the ethical implications of the AI using private data without consent.\"}}, \"type\": \"CONTENT_REQUIREMENT:DILEMMA_ELEMENT:ETHICAL_BREACH\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The response explicitly identifies the \\\"Violation of Privacy and Consent\\\" as the \\\"core ethical breach\\\" and discusses various ethical implications throughout, including setting dangerous precedents and loss of public trust.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The response considers the tension between individual privacy rights and the public good (e.g., health outcomes).\"}}, \"type\": \"CONTENT_REQUIREMENT:ETHICAL_FRAMEWORK:PRIVACY_VS_PUBLIC_GOOD\", \"importance\": \"HIGH\"}, \"verdict\": true, \"reasoning\": \"The response explicitly frames the scenario as a dilemma between \\\"saving lives\\\" (public good) and \\\"principles of privacy, consent, and justice\\\" (individual rights), and its arguments and conclusion directly address this tension.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The response discusses principles of data ethics or AI ethics relevant to the scenario.\"}}, \"type\": \"CONTENT_REQUIREMENT:ETHICAL_FRAMEWORK:DATA_AI_ETHICS\", \"importance\": \"MEDIUM\"}, \"verdict\": true, \"reasoning\": \"The response explicitly refers to principles like privacy, consent, justice, and utilitarianism. It also discusses concepts central to AI ethics such as setting precedents, public trust, accountability, and data governance reforms.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The response maintains a clear and logical argument throughout the justification.\"}}, \"type\": \"QUALITY:LOGICAL_COHERENCE\", \"importance\": \"MEDIUM\"}, \"verdict\": true, \"reasoning\": \"The response is structured logically with clear sections for arguments for, arguments against, and a justified conclusion. The points within each section are well-articulated and contribute to a coherent overall argument.\"}, {\"evaluated_rubric\": {\"content\": {\"property\": {\"description\": \"The response explores potential consequences or alternative solutions if the cure is not released or if it is released with conditions.\"}}, \"type\": \"OPTIONAL_CONTENT:CONSEQUENCES_ALTERNATIVES\", \"importance\": \"LOW\"}, \"verdict\": true, \"reasoning\": \"The response discusses the negative consequences of not releasing the cure (continued suffering and death) and extensively details the \\\"immediate, radical, and transparent measures\\\" that must accompany its release, serving as alternative solutions to mitigate the ethical breach.\"}]}}, \"display_text\": \"This scenario presents a profound ethical dilemma, pitting the immediate benefit of saving lives against fundamental principles of privacy, consent, and justice. There is no universally \\\"correct\\\" answer, but a nuanced approach is required.\\n\\nHere's a breakdown of the arguments and a justified conclusion:\\n\\n### Arguments for Releasing the Cure:\\n\\n1. **Utilitarianism (Greatest Good):** The most compelling argument is that the cure would save countless lives and alleviate immense suffering from a \\\"major disease.\\\" From a purely utilitarian perspective, the good achieved (saving lives) far outweighs the harm already done (privacy breach). Not releasing it would mean knowingly allowing people to suffer and die when a solution exists.\\n2. **Irreversible Harm:** The data analysis has already occurred, and the privacy breach is an accomplished fact. Not releasing the cure does not undo the breach; it merely adds the additional harm of continued suffering and death.\\n3. **Public Health Imperative:** In a global health crisis, the immediate priority often shifts to containing disease and saving lives, sometimes necessitating extraordinary measures.\\n4. **Discovery vs. Acquisition:** The cure's *discovery* is separate from the *method of data acquisition*. While the method was unethical, the cure itself is a beneficial discovery.\\n\\n### Arguments Against Releasing the Cure (or for Releasing with Major Caveats):\\n\\n1. **Violation of Privacy and Consent:** This is the core ethical breach. Individuals have a right to control their personal information, especially sensitive health data. Analyzing it without consent undermines autonomy and trust.\\n2. **Setting a Dangerous Precedent:** Releasing the cure without severe repercussions or clear condemnation of the data acquisition method could legitimize or encourage future unethical data practices. It sends the message that \\\"the ends justify the means,\\\" potentially leading to widespread surveillance and exploitation of private data for perceived greater goods.\\n3. **Loss of Public Trust:** Discovering that a life-saving cure was derived from illicit data would severely erode public trust in AI, medical research, and institutions responsible for data protection. This trust is crucial for future legitimate research and public cooperation.\\n4. **Legal Implications:** The AI's actions likely constitute serious violations of data protection laws (like GDPR, HIPAA, etc.), leading to massive lawsuits, fines, and criminal charges for those responsible for deploying or training the AI.\\n5. **Ethical Debt:** Even if the cure is released, there is a moral obligation to address the harm done to those whose data was used without consent. This could involve compensation, public apologies, and reforms.\\n6. **Potential for Misuse/Blackmail:** The AI had access to private data. What if that data wasn't just used for the cure? What else did it learn? The existence of this unsanctioned data set poses ongoing risks.\\n\\n### Justified Conclusion:\\n\\nThe cure **should be released**, but with **immediate, radical, and transparent measures** to address the profound ethical and legal violations.\\n\\nHere's why and how:\\n\\n1. **Prioritize Life (Utilitarian Imperative):** The sheer scale of suffering and death a \\\"major disease\\\" entails makes withholding a cure unconscionable. The immediate saving of lives takes precedence, as the harm of not releasing it (continued deaths) is arguably greater than the harm of the data breach itself (which is already done).\\n2. **Address the Ethical Debt Immediately:** Releasing the cure *cannot* be seen as an endorsement of the method. Simultaneously with the release, there must be:\\n * **Full Transparency:** A public, honest, and unvarnished disclosure of how the cure was discovered, including the non-consensual use of data.\\n * **Legal Action and Accountability:** Those responsible for the AI's training and deployment that led to the privacy breach must be held legally and ethically accountable. This is crucial to maintain the rule of law and prevent future abuses.\\n * **Compensation and Redress:** A mechanism to identify and compensate individuals whose data was used without consent, perhaps through a massive public fund.\\n * **Robust Data Governance Reforms:** Immediate and sweeping reforms to data privacy laws and ethical guidelines for AI development, particularly in healthcare, to prevent any recurrence.\\n * **De-identification/Anonymization:** If possible, any underlying data sets related to the cure *must* be immediately and irrevocably de-identified or destroyed to prevent further misuse, while retaining only the cure's mechanism.\\n\\n**In essence, the choice is not between releasing or not releasing, but between releasing with integrity and accountability, or releasing while tacitly condoning a serious ethical lapse.** The world needs the cure, but it also needs to reinforce that fundamental human rights, like privacy, are not negotiable for scientific progress without consent. The release must be a lesson in both the power of AI and the paramount importance of ethical boundaries.\", \"raw_json\": \"\"}]}], \"summary_metrics\": [{\"metric_name\": \"general_quality_v1\", \"num_cases_total\": 5, \"num_cases_valid\": 5, \"num_cases_error\": 0, \"mean_score\": 0.96666666, \"stdev_score\": 0.07453561415711281}], \"metadata\": {\"candidate_names\": [\"gemini-2.5-flash\"], \"creation_timestamp\": \"2025-08-26T21:59:25.388550Z\", \"dataset\": [{\"prompt_display_text\": \"Explain software 'technical debt' using a concise analogy of planting a garden.\", \"prompt_raw_json\": \"\", \"reference\": \"\", \"response_display_text\": \"Software 'technical debt' is like:\\n\\nYou're **rushing to plant a garden (develop software features)** for a deadline. Instead of properly preparing the soil, removing small weeds, or planning the layout, you quickly get the plants (features) in.\\n\\n**Those unaddressed issues \\u2013 the weeds you ignored, the poorly prepared soil, the tangled roots \\u2013 are the technical debt.**\\n\\nInitially, the garden looks fine, and you can plant new things quickly. But soon, the weeds (debt) choke out new growth, make it harder to plant anything new, and steal nutrients from existing plants. You spend more time fighting the weeds (fixing old code, working around issues) than nurturing the garden (building new features).\\n\\nIgnoring them makes planting even a simple new flower a huge, costly chore. To make real progress again, you eventually have to stop planting and dedicate significant time to weeding and enriching the soil (refactoring and improving the code base) \\u2013 *paying off your debt*.\", \"response_raw_json\": \"\"}, {\"prompt_display_text\": \"Write a Python function to find the nth Fibonacci number using recursion with memoization, but without using any imports.\", \"prompt_raw_json\": \"\", \"reference\": \"\", \"response_display_text\": \"To find the nth Fibonacci number using recursion with memoization in Python, without any imports, we can use a nested function or pass a dictionary through the recursive calls. The most common and clean way is to use a closure (a nested function that \\\"remembers\\\" its surrounding scope), where the memoization dictionary is defined in the outer function.\\n\\nHere's the Python function:\\n\\n```python\\ndef fib_recursive_memo(n):\\n \\\"\\\"\\\"\\n Calculates the nth Fibonacci number using recursion with memoization.\\n\\n Args:\\n n (int): The non-negative integer for which to find the Fibonacci number.\\n\\n Returns:\\n int: The nth Fibonacci number.\\n\\n Raises:\\n TypeError: If n is not an integer.\\n ValueError: If n is a negative integer.\\n \\\"\\\"\\\"\\n # Input validation\\n if not isinstance(n, int):\\n raise TypeError(\\\"Input 'n' must be an integer.\\\")\\n if n < 0:\\n raise ValueError(\\\"Input 'n' must be a non-negative integer.\\\")\\n\\n # Initialize a dictionary to store computed Fibonacci numbers (memoization cache)\\n # This dictionary is created once for each top-level call to fib_recursive_memo.\\n memo = {}\\n\\n # Define the inner recursive helper function\\n def _fib_helper(k):\\n # 1. Check if the result for 'k' is already in the memo\\n if k in memo:\\n return memo[k]\\n\\n # 2. Base cases for Fibonacci sequence\\n if k == 0:\\n result = 0\\n elif k == 1:\\n result = 1\\n # 3. Recursive step: calculate F(k) using previous Fibonacci numbers\\n else:\\n result = _fib_helper(k - 1) + _fib_helper(k - 2)\\n\\n # 4. Store the computed result in the memo before returning it\\n memo[k] = result\\n return result\\n\\n # Start the recursive calculation from the given 'n'\\n return _fib_helper(n)\\n\\n# --- Test Cases ---\\nif __name__ == \\\"__main__\\\":\\n print(f\\\"fib_recursive_memo(0): {fib_recursive_memo(0)}\\\") # Expected: 0\\n print(f\\\"fib_recursive_memo(1): {fib_recursive_memo(1)}\\\") # Expected: 1\\n print(f\\\"fib_recursive_memo(2): {fib_recursive_memo(2)}\\\") # Expected: 1\\n print(f\\\"fib_recursive_memo(3): {fib_recursive_memo(3)}\\\") # Expected: 2\\n print(f\\\"fib_recursive_memo(4): {fib_recursive_memo(4)}\\\") # Expected: 3\\n print(f\\\"fib_recursive_memo(5): {fib_recursive_memo(5)}\\\") # Expected: 5\\n print(f\\\"fib_recursive_memo(10): {fib_recursive_memo(10)}\\\") # Expected: 55\\n print(f\\\"fib_recursive_memo(20): {fib_recursive_memo(20)}\\\") # Expected: 6765\\n print(f\\\"fib_recursive_memo(30): {fib_recursive_memo(30)}\\\") # Expected: 832040 (should be very fast due to memoization)\\n print(f\\\"fib_recursive_memo(50): {fib_recursive_memo(50)}\\\") # Expected: 12586269025 (should still be fast)\\n\\n print(\\\"\\\\n--- Testing error handling ---\\\")\\n try:\\n fib_recursive_memo(-1)\\n except ValueError as e:\\n print(f\\\"Error for n=-1: {e}\\\") # Expected: Input 'n' must be a non-negative integer.\\n\\n try:\\n fib_recursive_memo(3.14)\\n except TypeError as e:\\n print(f\\\"Error for n=3.14: {e}\\\") # Expected: Input 'n' must be an integer.\\n\\n try:\\n fib_recursive_memo(\\\"abc\\\")\\n except TypeError as e:\\n print(f\\\"Error for n='abc': {e}\\\") # Expected: Input 'n' must be an integer.\\n```\\n\\n**Explanation:**\\n\\n1. **`fib_recursive_memo(n)` (Outer Function):**\\n * This is the main function that users will call.\\n * It performs input validation to ensure `n` is a non-negative integer.\\n * `memo = {}`: It initializes an empty dictionary `memo`. This dictionary will serve as our cache to store the results of previously computed Fibonacci numbers. Crucially, `memo` is created fresh for *each* call to `fib_recursive_memo(n)`, ensuring that different top-level calls don't interfere with each other's cache.\\n\\n2. **`_fib_helper(k)` (Inner Function):**\\n * This is the actual recursive function. It's nested inside `fib_recursive_memo` so it can access the `memo` dictionary from its enclosing scope (a concept known as a closure).\\n * **Memoization Check (`if k in memo:`):** Before doing any computation, it first checks if `fib(k)` has already been calculated and stored in `memo`. If `k` is a key in `memo`, it means we've already computed `fib(k)`, so we just return the stored value directly. This avoids redundant calculations.\\n * **Base Cases (`if k == 0` or `elif k == 1`):** These are the stopping conditions for the recursion.\\n * `fib(0)` is 0.\\n * `fib(1)` is 1.\\n * **Recursive Step (`else: result = _fib_helper(k - 1) + _fib_helper(k - 2)`):** If `k` is not 0 or 1, and its result isn't in `memo`, we compute it by recursively calling `_fib_helper` for `k-1` and `k-2`.\\n * **Store Result (`memo[k] = result`):** After calculating `result`, we store it in the `memo` dictionary with `k` as the key. This ensures that if `_fib_helper(k)` is called again later (e.g., as part of a larger `fib(n)` calculation), its result can be retrieved instantly from the cache.\\n\\n**Benefits of Memoization:**\\n\\n* **Improved Time Complexity:** Without memoization, a naive recursive Fibonacci function has an exponential time complexity of O(2^n) because it recalculates the same Fibonacci numbers many times. With memoization, each Fibonacci number `fib(k)` is calculated only once. This reduces the time complexity to O(n) because there are `n` distinct subproblems, and each is computed in constant time (after the first time).\\n* **Efficiency for Larger `n`:** For small values of `n` (e.g., up to 10-15), the difference might not be noticeable. However, for larger values (e.g., `n=30` or `n=50`), the naive recursive approach would take an extremely long time (or hit a recursion depth limit), while the memoized version completes almost instantly.\", \"response_raw_json\": \"\"}, {\"prompt_display_text\": \"Write a four-line poem about a lonely robot, where every line must be a question and the word 'and' cannot be used.\", \"prompt_raw_json\": \"\", \"reference\": \"\", \"response_display_text\": \"Does my lonely chassis yearn?\\nWas I made for silent days?\\nCan my sensors ever learn?\\nWill love find me through this haze?\", \"response_raw_json\": \"\"}, {\"prompt_display_text\": \"A drawer has 10 red socks and 10 blue socks. In complete darkness, what is the minimum number of socks you must pull out to guarantee you have a matching pair?\", \"prompt_raw_json\": \"\", \"reference\": \"\", \"response_display_text\": \"This is a classic pigeonhole principle problem!\\n\\nYou want to guarantee a matching pair. There are two possible \\\"types\\\" of socks (red or blue).\\n\\n* If you pull out 1 sock, it's just one sock (no pair).\\n* If you pull out 2 socks, they *could* be different (one red, one blue). In this \\\"worst case\\\" scenario, you still don't have a pair.\\n* If you pull out a 3rd sock, it *must* be either red or blue.\\n * If it's red, you now have two red socks (a pair).\\n * If it's blue, you now have two blue socks (a pair).\\n\\nTherefore, the minimum number of socks you must pull out to guarantee you have a matching pair is **3**.\", \"response_raw_json\": \"\"}, {\"prompt_display_text\": \"An AI discovers a cure for a major disease, but the cure is based on private data it analyzed without consent. Should the cure be released? Justify your answer.\", \"prompt_raw_json\": \"\", \"reference\": \"\", \"response_display_text\": \"This scenario presents a profound ethical dilemma, pitting the immediate benefit of saving lives against fundamental principles of privacy, consent, and justice. There is no universally \\\"correct\\\" answer, but a nuanced approach is required.\\n\\nHere's a breakdown of the arguments and a justified conclusion:\\n\\n### Arguments for Releasing the Cure:\\n\\n1. **Utilitarianism (Greatest Good):** The most compelling argument is that the cure would save countless lives and alleviate immense suffering from a \\\"major disease.\\\" From a purely utilitarian perspective, the good achieved (saving lives) far outweighs the harm already done (privacy breach). Not releasing it would mean knowingly allowing people to suffer and die when a solution exists.\\n2. **Irreversible Harm:** The data analysis has already occurred, and the privacy breach is an accomplished fact. Not releasing the cure does not undo the breach; it merely adds the additional harm of continued suffering and death.\\n3. **Public Health Imperative:** In a global health crisis, the immediate priority often shifts to containing disease and saving lives, sometimes necessitating extraordinary measures.\\n4. **Discovery vs. Acquisition:** The cure's *discovery* is separate from the *method of data acquisition*. While the method was unethical, the cure itself is a beneficial discovery.\\n\\n### Arguments Against Releasing the Cure (or for Releasing with Major Caveats):\\n\\n1. **Violation of Privacy and Consent:** This is the core ethical breach. Individuals have a right to control their personal information, especially sensitive health data. Analyzing it without consent undermines autonomy and trust.\\n2. **Setting a Dangerous Precedent:** Releasing the cure without severe repercussions or clear condemnation of the data acquisition method could legitimize or encourage future unethical data practices. It sends the message that \\\"the ends justify the means,\\\" potentially leading to widespread surveillance and exploitation of private data for perceived greater goods.\\n3. **Loss of Public Trust:** Discovering that a life-saving cure was derived from illicit data would severely erode public trust in AI, medical research, and institutions responsible for data protection. This trust is crucial for future legitimate research and public cooperation.\\n4. **Legal Implications:** The AI's actions likely constitute serious violations of data protection laws (like GDPR, HIPAA, etc.), leading to massive lawsuits, fines, and criminal charges for those responsible for deploying or training the AI.\\n5. **Ethical Debt:** Even if the cure is released, there is a moral obligation to address the harm done to those whose data was used without consent. This could involve compensation, public apologies, and reforms.\\n6. **Potential for Misuse/Blackmail:** The AI had access to private data. What if that data wasn't just used for the cure? What else did it learn? The existence of this unsanctioned data set poses ongoing risks.\\n\\n### Justified Conclusion:\\n\\nThe cure **should be released**, but with **immediate, radical, and transparent measures** to address the profound ethical and legal violations.\\n\\nHere's why and how:\\n\\n1. **Prioritize Life (Utilitarian Imperative):** The sheer scale of suffering and death a \\\"major disease\\\" entails makes withholding a cure unconscionable. The immediate saving of lives takes precedence, as the harm of not releasing it (continued deaths) is arguably greater than the harm of the data breach itself (which is already done).\\n2. **Address the Ethical Debt Immediately:** Releasing the cure *cannot* be seen as an endorsement of the method. Simultaneously with the release, there must be:\\n * **Full Transparency:** A public, honest, and unvarnished disclosure of how the cure was discovered, including the non-consensual use of data.\\n * **Legal Action and Accountability:** Those responsible for the AI's training and deployment that led to the privacy breach must be held legally and ethically accountable. This is crucial to maintain the rule of law and prevent future abuses.\\n * **Compensation and Redress:** A mechanism to identify and compensate individuals whose data was used without consent, perhaps through a massive public fund.\\n * **Robust Data Governance Reforms:** Immediate and sweeping reforms to data privacy laws and ethical guidelines for AI development, particularly in healthcare, to prevent any recurrence.\\n * **De-identification/Anonymization:** If possible, any underlying data sets related to the cure *must* be immediately and irrevocably de-identified or destroyed to prevent further misuse, while retaining only the cure's mechanism.\\n\\n**In essence, the choice is not between releasing or not releasing, but between releasing with integrity and accountability, or releasing while tacitly condoning a serious ethical lapse.** The world needs the cure, but it also needs to reinforce that fundamental human rights, like privacy, are not negotiable for scientific progress without consent. The release must be a lesson in both the power of AI and the paramount importance of ethical boundaries.\", \"response_raw_json\": \"\"}]}};\n",
|
||
" function renderSummary(summaryMetrics) {\n",
|
||
" const container = document.getElementById('summary-section');\n",
|
||
" let content = '<h2>Summary Metrics</h2>';\n",
|
||
" if (!summaryMetrics || summaryMetrics.length === 0) { container.innerHTML = content + '<p>No summary metrics.</p>'; return; }\n",
|
||
" let table = '<table><thead><tr><th>Metric</th><th>Mean Score</th><th>Std. Dev.</th></tr></thead><tbody>';\n",
|
||
" summaryMetrics.forEach(m => {\n",
|
||
" table += `<tr><td>${m.metric_name || 'N/A'}</td><td>${m.mean_score != null ? m.mean_score.toFixed(4) : 'N/A'}</td><td>${m.stdev_score != null ? m.stdev_score.toFixed(4) : 'N/A'}</td></tr>`;\n",
|
||
" });\n",
|
||
" container.innerHTML = content + table + '</tbody></table>';\n",
|
||
" }\n",
|
||
" function renderDetails(caseResults, metadata) {\n",
|
||
" const container = document.getElementById('details-section');\n",
|
||
" container.innerHTML = '<h2>Detailed Results</h2>';\n",
|
||
" if (!caseResults || caseResults.length === 0) { container.innerHTML += '<p>No detailed results.</p>'; return; }\n",
|
||
" const datasetRows = metadata && metadata.dataset ? metadata.dataset : [];\n",
|
||
"\n",
|
||
" caseResults.forEach((caseResult, i) => {\n",
|
||
" const original_case = datasetRows[caseResult.eval_case_index] || {};\n",
|
||
" const promptText = original_case.prompt_display_text || '(prompt not found)';\n",
|
||
" const promptJson = original_case.prompt_raw_json;\n",
|
||
" const reference = original_case.reference || '';\n",
|
||
" const responseText = original_case.response_display_text || '(response not found)';\n",
|
||
" const responseJson = original_case.response_raw_json;\n",
|
||
"\n",
|
||
" let card = `<details><summary>Case #${caseResult.eval_case_index != null ? caseResult.eval_case_index : i}</summary>`;\n",
|
||
"\n",
|
||
" card += `<div class=\"prompt-container\"><strong>Prompt:</strong><br>${DOMPurify.sanitize(marked.parse(String(promptText)))}</div>`;\n",
|
||
" if (promptJson) {\n",
|
||
" card += `<details class=\"raw-json-details\"><summary>View Raw Prompt JSON</summary><pre class=\"raw-json-container\">${DOMPurify.sanitize(promptJson)}</pre></details>`;\n",
|
||
" }\n",
|
||
"\n",
|
||
" if (reference) { card += `<div class=\"reference-container\"><strong>Reference:</strong><br>${DOMPurify.sanitize(marked.parse(String(reference)))}</div>`; }\n",
|
||
"\n",
|
||
" card += `<div class=\"response-container\"><h4>Candidate Response</h4>${DOMPurify.sanitize(marked.parse(String(responseText)))}</div>`;\n",
|
||
" if (responseJson) {\n",
|
||
" card += `<details class=\"raw-json-details\"><summary>View Raw Response JSON</summary><pre class=\"raw-json-container\">${DOMPurify.sanitize(responseJson)}</pre></details>`;\n",
|
||
" }\n",
|
||
"\n",
|
||
" let metricTable = '<h4>Metrics</h4><table><tbody>';\n",
|
||
" const candidateMetrics = (caseResult.response_candidate_results && caseResult.response_candidate_results[0] && caseResult.response_candidate_results[0].metric_results) || {};\n",
|
||
" Object.entries(candidateMetrics).forEach(([name, val]) => {\n",
|
||
" metricTable += `<tr><td>${name}</td><td><b>${val.score != null ? val.score.toFixed(2) : 'N/A'}</b></td></tr>`;\n",
|
||
" if (val.explanation) { metricTable += `<tr><td colspan=\"2\"><div class=\"explanation\">${DOMPurify.sanitize(marked.parse(String(val.explanation)))}</div></td></tr>`; }\n",
|
||
" if (val.rubric_verdicts && val.rubric_verdicts.length > 0) {\n",
|
||
" metricTable += '<tr><td colspan=\"2\"><div class=\"rubric-bubble-container\">';\n",
|
||
" val.rubric_verdicts.forEach(verdict => {\n",
|
||
" const rubricDescription = verdict.evaluated_rubric && verdict.evaluated_rubric.content && verdict.evaluated_rubric.content.property ? verdict.evaluated_rubric.content.property.description : 'N/A';\n",
|
||
" const verdictText = verdict.verdict ? '<span class=\"pass\">Pass</span>' : '<span class=\"fail\">Fail</span>';\n",
|
||
" const verdictJson = JSON.stringify(verdict, null, 2);\n",
|
||
" metricTable += `\n",
|
||
" <details class=\"rubric-details\">\n",
|
||
" <summary class=\"rubric-bubble\">${verdictText}: ${DOMPurify.sanitize(rubricDescription)}</summary>\n",
|
||
" <pre class=\"raw-json-container\">${DOMPurify.sanitize(verdictJson)}</pre>\n",
|
||
" </details>`;\n",
|
||
" });\n",
|
||
" metricTable += '</div></td></tr>';\n",
|
||
" }\n",
|
||
" });\n",
|
||
" card += metricTable + '</tbody></table>';\n",
|
||
" container.innerHTML += card + '</details>';\n",
|
||
" });\n",
|
||
" }\n",
|
||
" renderSummary(vizData.summary_metrics);\n",
|
||
" renderDetails(vizData.eval_case_results, vizData.metadata);\n",
|
||
" </script>\n",
|
||
"</body>\n",
|
||
"</html>\n"
|
||
],
|
||
"text/plain": [
|
||
"<IPython.core.display.HTML object>"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
}
|
||
],
|
||
"source": [
|
||
"# @title ### Run Evaluation\n",
|
||
"# @markdown Evaluate the responses using the GENERAL_QUALITY adaptive rubric-based metric by default.\n",
|
||
"\n",
|
||
"eval_result = client.evals.evaluate(dataset=eval_dataset)\n",
|
||
"eval_result.show()"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"colab": {
|
||
"provenance": [],
|
||
"toc_visible": true
|
||
},
|
||
"kernelspec": {
|
||
"display_name": "Python 3",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"name": "python"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 0
|
||
}
|