wehub/googlecloudplatform--generative-ai
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity
Files
main
googlecloudplatform--genera…/gemini/prompts/examples/question_answering.ipynb
T
wehub-resource-sync 914fea506e chore: import upstream snapshot with attribution
2026-07-13 13:30:30 +08:00

895 lines
35 KiB
Plaintext
Raw Permalink Blame History

{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2024 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Question Answering with Generative Models on Vertex AI\n",
"\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/examples/question_answering.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fprompts%2Fexamples%2Fquestion_answering.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td> \n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/prompts/examples/question_answering.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/examples/question_answering.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/examples/question_answering.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/examples/question_answering.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/examples/question_answering.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/examples/question_answering.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/examples/question_answering.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a> "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "66NyrdKj7Qya"
},
"source": [
"| Authors |\n",
"| --- |\n",
"| [Polong Lin](https://github.com/polong-lin) |\n",
"| [Deepak Moonat](https://github.com/dmoonat) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"Large language models can be used for various natural language processing tasks, including question-answering (Q&A). These models are trained on a vast amount text data and can generate high-quality responses to a wide range of questions. One thing to note here is that most models have cutoff dates regarding their knowledge, and asking anything too recent might yield an incomplete, imaginative or incorrect answer (i.e. a hallucination).\n",
"\n",
"This notebook covers the essentials of prompts for answering questions using a generative model. In addition, it showcases the `open domain` (knowledge available on the public internet) and `closed domain` (knowledge that is more private - typically enterprise or personal knowledge).\n",
"\n",
"Learn more about prompt design in the [official documentation](https://cloud.google.com/vertex-ai/docs/generative-ai/text/text-overview#prompt_structure)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d975e698c9a4"
},
"source": [
"### Objective\n",
"\n",
"By the end of the notebook, you should be able to write prompts for the following:\n",
"\n",
"* **Open domain** questions:\n",
" * Zero-shot prompting\n",
" * Few-shot prompting\n",
"\n",
"\n",
"* **Closed domain** questions:\n",
" * Providing custom knowledge as context\n",
" * Instruction-tune the outputs\n",
" * Few-shot prompting"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "QDU0XJ1xRDlL"
},
"source": [
"## Getting Started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2a5AEr0lkLKD"
},
"source": [
"### Install Vertex AI SDK and other required packages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "82ad0c445061"
},
"outputs": [],
"source": [
"%pip install google-cloud-aiplatform --upgrade -q\n",
"\n",
"# Install the package `fuzzywuzzy` and `python-Levenshtein`\n",
"%pip install -q python-Levenshtein --upgrade --user\n",
"%pip install -q git+git://github.com/seatgeek/fuzzywuzzy.git@0.18.0#egg=fuzzywuzzy --upgrade --user"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Xe7OuYuGkLKF"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you are running this notebook on Google Colab, run the cell below to authenticate your environment.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "U9Gx2SAZkLKF"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "E2-s7o4InNF0"
},
"source": [
"### Set Google Cloud project information and initialize Vertex AI SDK\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment).\"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "M-0MLpTYnOre"
},
"outputs": [],
"source": [
"PROJECT_ID = \"your-project-id\" # @param {type:\"string\"}\n",
"LOCATION = \"us-central1\" # @param {type:\"string\"}\n",
"\n",
"import vertexai\n",
"\n",
"vertexai.init(project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "960505627ddf"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PyQmSRbKA8r-"
},
"outputs": [],
"source": [
"from fuzzywuzzy import fuzz\n",
"import pandas as pd\n",
"from vertexai.generative_models import GenerationConfig, GenerativeModel"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "UP76a2la7O-a"
},
"source": [
"### Import models"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7isig7e07O-a"
},
"outputs": [],
"source": [
"generation_model = GenerativeModel(\"gemini-2.0-flash\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0U5HNYR8nT7c"
},
"source": [
"#### Generation config\n",
"\n",
"- Each call that you send to a model includes parameter values that control how the model generates a response. The model can generate different results for different parameter values\n",
"- <strong>Experiment</strong> with different parameter values to get the best values for the task\n",
"\n",
"Refer to following [link](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/prompt-design-strategies#experiment-with-different-parameter-values) for understanding different parameters"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fIPcn5dZ7O-b"
},
"source": [
"## Question Answering"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cNNEz7vGFYUP"
},
"source": [
"Question-answering capabilities require providing a prompt or a question that the model can use to generate a response. The prompt can be a few words or a few complete sentences, depending on the complexity of the question.\n",
"\n",
"When creating a question-answering prompt, it is essential to be specific and provide as much context as possible. It helps the model understand the intent behind the question and generate a relevant response. For example, if you want to ask:\n",
"\n",
"```\n",
"\"What is the capital of France?\",\n",
"\n",
"then a good prompt could be:\n",
"\n",
"\"Please tell me the name of the city that serves as the capital of France.\"\n",
"\n",
"```\n",
"\n",
"In addition to being specific, the prompt should also be grammatically correct and free of spelling errors. It helps the model generate a response that is easy to understand and contains fewer errors or inaccuracies.\n",
"\n",
"By providing specific, context-rich prompts, you can help the model understand the intent behind the question and generate accurate and relevant responses.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "C5N9ZnlECm-z"
},
"source": [
"Below are some differences between the **open domain** and **closed domain** categories for question-answering prompts.\n",
"\n",
"* **Open domain**: All questions whose answers are available online already. They can belong to any category, like history, geography, countries, politics, chemistry, etc. These include trivia or general knowledge questions, like:\n",
"\n",
"```\n",
"Q: Who won the Olympic gold in swimming?\n",
"Q: Who is the President of [given country]?\n",
"Q: Who wrote [specific book]\"?\n",
"```\n",
"\n",
"Keep in mind the training cutoff of generative models, as questions involving information more recent than what the model was trained on might give incorrect or imaginative answers.\n",
"\n",
"\n",
"* **Closed domain**: If you have some internal knowledge base not available on the public internet, then those belong to the _closed domain_ category.\n",
"You can pass that \"private\" knowledge as context to the model. If prompted correctly, the model is more likely to answer from within the context provided and less likely to give answers beyond that from the open internet.\n",
"\n",
"Consider the example of building a Q&A bot over your internal product documentation. In this case, you can pass the complete documentation to the model and prompt it only to answer based on that.\n",
"\n",
"Typical prompt for **closed domain**:\n",
"\n",
"```\n",
"Prompt: f\"\"\" Answer from the below context: \\n\\n\n",
"\t\t context: {your knowledge base} \\n\n",
"\t\t question: {question specific to that knowledge base} \\n\n",
"\t\t answer: {to be predicted by model} \\n\n",
"\t\t\"\"\"\n",
"```\n",
"\n",
"Below are some examples to understand these different types of prompts."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WBoN6zixDSiX"
},
"source": [
"### Open Domain"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wJnv8XhnDXQm"
},
"source": [
"#### Zero-shot prompting"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PaYoQuRwCm-z"
},
"outputs": [],
"source": [
"prompt = \"\"\"Q: Who was President of the United States in 1955? Which party did he belong to?\\n\n",
" A:\n",
" \"\"\"\n",
"\n",
"generation_config = GenerationConfig(temperature=0.1, max_output_tokens=256)\n",
"\n",
"response = generation_model.generate_content(\n",
" contents=prompt, generation_config=generation_config\n",
").text\n",
"print(response)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "4qcUdUgwCm-z"
},
"outputs": [],
"source": [
"prompt = \"\"\"Q: What is the tallest mountain in the world?\\n\n",
" A:\n",
" \"\"\"\n",
"\n",
"generation_config = GenerationConfig(temperature=0.1, max_output_tokens=20)\n",
"\n",
"response = generation_model.generate_content(\n",
" contents=prompt, generation_config=generation_config\n",
").text\n",
"print(response)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HShw52X2Dcmx"
},
"source": [
"#### Few-shot prompting"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tj_2hHAWE8vh"
},
"source": [
"Let's say you want to a get a short answer from the model (like only a specific name). To do so, you can leverage a few-shot prompt and provide examples to the model to illustrate the expected behavior."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "RE5yCAaqDg7m"
},
"outputs": [],
"source": [
"prompt = \"\"\"Q: Who is the current President of France?\\n\n",
" A: Emmanuel Macron \\n\\n\n",
"\n",
" Q: Who invented the telephone? \\n\n",
" A: Alexander Graham Bell \\n\\n\n",
"\n",
" Q: Who wrote the novel \"1984\"?\n",
" A: George Orwell\n",
"\n",
" Q: Who discovered penicillin?\n",
" A:\n",
" \"\"\"\n",
"\n",
"generation_config = GenerationConfig(temperature=0.1, max_output_tokens=20)\n",
"\n",
"response = generation_model.generate_content(\n",
" contents=prompt, generation_config=generation_config\n",
").text\n",
"print(response)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xGvs0jFsUlvM"
},
"source": [
"#### Zero-shot prompting vs Few-shot prompting"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7yjsAMuMUfZC"
},
"source": [
"Zero-shot prompting can be useful for quickly generating text for new tasks, but the quality of the generated text may be lower than that of a few-shot prompt with well-chosen examples. Few-shot prompting is typically better suited for tasks that require a high degree of specificity or domain-specific knowledge, but requires some additional thought and potentially data to set up the prompt."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "u6UiJTxXEs4t"
},
"source": [
"### Closed Domain"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "03ZITm4AGBvP"
},
"source": [
"#### Adding internal knowledge as context in prompts"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EkhqjmB6VqPx"
},
"source": [
"Imagine a scenario where you would like to build a question-answering bot that takes in internal documentation and lets users ask questions about it.\n",
"\n",
"In the example below, the Google Cloud Storage and content policy documentation is added to the prompt, so that the Gemini API can use that to answer subsequent questions with the provided context."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "s1H2er_lExpW"
},
"outputs": [],
"source": [
"context = \"\"\"\n",
"Storage and content policy \\n\n",
"How durable is my data in Cloud Storage? \\n\n",
"Cloud Storage is designed for 99.999999999% (11 9's) annual durability, which is appropriate for even primary storage and\n",
"business-critical applications. This high durability level is achieved through erasure coding that stores data pieces redundantly\n",
"across multiple devices located in multiple availability zones.\n",
"Objects written to Cloud Storage must be redundantly stored in at least two different availability zones before the\n",
"write is acknowledged as successful. Checksums are stored and regularly revalidated to proactively verify that the data\n",
"integrity of all data at rest as well as to detect corruption of data in transit. If required, corrections are automatically\n",
"made using redundant data. Customers can optionally enable object versioning to add protection against accidental deletion.\n",
"\"\"\"\n",
"\n",
"question = \"How is high availability achieved?\"\n",
"\n",
"prompt = f\"\"\"Answer the question given in the contex below:\n",
"Context: {context}?\\n\n",
"Question: {question} \\n\n",
"Answer:\n",
"\"\"\"\n",
"\n",
"print(\"[Prompt]\")\n",
"print(prompt)\n",
"\n",
"print(\"[Response]\")\n",
"response = generation_model.generate_content(contents=prompt).text\n",
"print(response)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tagWC4VcQIw6"
},
"source": [
"#### Instruction-tuning outputs"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "v9UkogWHXM6N"
},
"source": [
"Another way to help out language models is to provide additional instructions to frame the output in the prompt. To ensure the model doesn't respond to anything outside the context, the prompt can specify that the response should be \"Information not available in provided context\" if that's the case."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ouq8FfwSQIBT"
},
"outputs": [],
"source": [
"question = \"What machines are required for hosting Vertex AI models?\"\n",
"prompt = f\"\"\"Answer the question given the context below as {{Context:}}. \\n\n",
"If the answer is not available in the {{Context:}} and you are not confident about the output,\n",
"please say \"Information not available in provided context\". \\n\\n\n",
"Context: {context}?\\n\n",
"Question: {question} \\n\n",
"Answer:\n",
"\"\"\"\n",
"\n",
"print(\"[Prompt]\")\n",
"print(prompt)\n",
"\n",
"print(\"[Response]\")\n",
"\n",
"generation_config = GenerationConfig(temperature=0.3, max_output_tokens=256)\n",
"\n",
"response = generation_model.generate_content(\n",
" contents=prompt, generation_config=generation_config\n",
").text\n",
"print(response)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iZJfZShPRGqU"
},
"source": [
"#### Few-shot prompting"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "qdSEQeQIS6pt"
},
"outputs": [],
"source": [
"prompt = \"\"\"\n",
"Context:\n",
"The term \"artificial intelligence\" was first coined by John McCarthy in 1956. Since then, AI has developed into a vast\n",
"field with numerous applications, ranging from self-driving cars to virtual assistants like Siri and Alexa.\n",
"\n",
"Question:\n",
"What is artificial intelligence?\n",
"\n",
"Answer:\n",
"Artificial intelligence refers to the simulation of human intelligence in machines that are programmed to think and learn like humans.\n",
"\n",
"---\n",
"\n",
"Context:\n",
"The Wright brothers, Orville and Wilbur, were two American aviation pioneers who are credited with inventing and\n",
"building the world's first successful airplane and making the first controlled, powered and sustained heavier-than-air human flight,\n",
" on December 17, 1903.\n",
"\n",
"Question:\n",
"Who were the Wright brothers?\n",
"\n",
"Answer:\n",
"The Wright brothers were American aviation pioneers who invented and built the world's first successful airplane\n",
"and made the first controlled, powered and sustained heavier-than-air human flight, on December 17, 1903.\n",
"\n",
"---\n",
"\n",
"Context:\n",
"The Mona Lisa is a 16th-century portrait painted by Leonardo da Vinci during the Italian Renaissance. It is one of\n",
"the most famous paintings in the world, known for the enigmatic smile of the woman depicted in the painting.\n",
"\n",
"Question:\n",
"Who painted the Mona Lisa?\n",
"\n",
"Answer:\n",
"\n",
"\"\"\"\n",
"\n",
"response = generation_model.generate_content(contents=prompt).text\n",
"print(response)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "leYIui80Q4tH"
},
"source": [
"### Extractive Question-Answering\n",
"\n",
"In the next example, the generative model is guided to understand the meaning of the question and the passage, and to identify the relevant information in the passage that answers the question. The model is given a question and a passage of text, and is asked to find the answer to the question within the passage. The answer is typically a phrase or sentence."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "bPZqm0QJQ4tH"
},
"outputs": [],
"source": [
"prompt = \"\"\"\n",
"Background: There is evidence that there have been significant changes in Amazon rainforest vegetation over the last 21,000 years through the Last Glacial Maximum (LGM) and subsequent deglaciation.\n",
"Analyses of sediment deposits from Amazon basin paleo lakes and from the Amazon Fan indicate that rainfall in the basin during the LGM was lower than for the present, and this was almost certainly\n",
"associated with reduced moist tropical vegetation cover in the basin. There is debate, however, over how extensive this reduction was. Some scientists argue that the rainforest was reduced to small,\n",
"isolated refugia separated by open forest and grassland; other scientists argue that the rainforest remained largely intact but extended less far to the north, south, and east than is seen today.\n",
"This debate has proved difficult to resolve because the practical limitations of working in the rainforest mean that data sampling is biased away from the center of the Amazon basin, and both\n",
"explanations are reasonably well supported by the available data.\n",
"\n",
"Q: What does LGM stands for?\n",
"A: Last Glacial Maximum.\n",
"\n",
"Q: What did the analysis from the sediment deposits indicate?\n",
"A: Rainfall in the basin during the LGM was lower than for the present.\n",
"\n",
"Q: What are some of scientists arguments?\n",
"A: The rainforest was reduced to small, isolated refugia separated by open forest and grassland.\n",
"\n",
"Q: There have been major changes in Amazon rainforest vegetation over the last how many years?\n",
"A: 21,000.\n",
"\n",
"Q: What caused changes in the Amazon rainforest vegetation?\n",
"A: The Last Glacial Maximum (LGM) and subsequent deglaciation\n",
"\n",
"Q: What has been analyzed to compare Amazon rainfall in the past and present?\n",
"A: Sediment deposits.\n",
"\n",
"Q: What has the lower rainfall in the Amazon during the LGM been attributed to?\n",
"A:\n",
"\"\"\"\n",
"\n",
"response = generation_model.generate_content(contents=prompt).text\n",
"print(response)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "94d80fb55f48"
},
"source": [
"### Evaluation"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b620d23a7634"
},
"source": [
"You can evaluate the outputs of the question and answering task if the ground truth answers of each question are available. In zero-shot prompting, you can only use `open domain` questions. However, with `closed domain` questions, you can add context and evaluate similarly. To showcase how that will work, start by creating a simple dataframe with questions and ground truth answers."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8e813a463531"
},
"outputs": [],
"source": [
"qa_data_df = pd.DataFrame(\n",
" {\n",
" \"question\": [\n",
" 'In a website browser address bar, what does \"www\" stand for?',\n",
" \"Who was the first woman to win a Nobel Prize\",\n",
" \"What is the name of the Earth's largest ocean?\",\n",
" ],\n",
" \"answer_groundtruth\": [\"World Wide Web\", \"Marie Curie\", \"The Pacific Ocean\"],\n",
" }\n",
")\n",
"\n",
"qa_data_df"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "951a147dc79d"
},
"source": [
"Now that you have the data with questions and ground truth answers, you can call the Gemini generation model to each review row using the `apply` function. Each row will use the dynamic prompt to predict the answer using the Gemini API. We will save the results in `answer_prediction` column. \n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ffc47e0cb5b9"
},
"outputs": [],
"source": [
"def get_answer(row):\n",
" prompt = f\"\"\"Answer the following question as precise as possible.\\n\\n\n",
" question: {row}\n",
" answer:\n",
" \"\"\"\n",
" return generation_model.generate_content(\n",
" contents=prompt,\n",
" ).text\n",
"\n",
"\n",
"qa_data_df[\"answer_prediction\"] = qa_data_df[\"question\"].apply(get_answer)\n",
"qa_data_df"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "kEcvfT4kBXkF"
},
"outputs": [],
"source": [
"qa_data_df.to_csv(\"df.csv\", index=False)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6fe997dbf788"
},
"source": [
"You may want to evaluate the answers predicted by the Gemini API. However, it will be more complex than the text classification since the answers may differ from ground truth and may be presented in slightly more/fewer words.\n",
"\n",
"For example, you can observe the question \"What is the name of the Earth's largest ocean?\" and see that model predicted \"Pacific Ocean\" when a ground truth label is \"The Pacific Ocean\" with the extra \"The.\" Now, if you use the simple classification metrics, then you will consider this as a wrong prediction since original and predicted strings have a difference. However, you can see that the answer is correct since an extra \"The\" is causing the issue. It's a simple string comparison problem.\n",
"\n",
"The solution to string comparison where both `ground_truth` and `predicted` may have some extra or fewer letters, one approach is to use a fuzzy matching algorithm.\n",
"Fuzzy string matching uses [Levenshtein Distance](https://en.wikipedia.org/wiki/Levenshtein_distance) to calculate the differences between two strings.\n",
"\n",
"For example, the Levenshtein distance between \"kitten\" and \"sitting\" is 3, since the following 3 edits change one into the other, and there is no way to do it with fewer than 3 edits:\n",
"\n",
"* kitten → sitten (substitution of \"s\" for \"k\"),\n",
"* sitten → sittin (substitution of \"i\" for \"e\"),\n",
"* sittin → sitting (insertion of \"g\" at the end).\n",
"\n",
"\n",
"Here's another example, but this time using `fuzzywuzzy` library, which gives us the same `Levenshtein distance` between two strings but in ratio. The ratio raw score measures the string's similarity as an int in the range [0, 100]. For two strings X and Y, the score is defined by int(round((2.0 * M / T) * 100)) where T is the total number of characters in both strings, and M is the number of matches in the two strings.\n",
"\n",
"Read more here about the [ratio formula](https://anhaidgroup.github.io/py_stringmatching/v0.3.x/Ratio.html) :\n",
"\n",
"You can see one example to understand this further.\n",
"```\n",
"String1: \"this is a test\"\n",
"String2: \"this is a test!\"\n",
"\n",
"Fuzz Ratio => 97 #\n",
"\n",
"Fuzz Partial Ratio => 100 #Since most characters are the same and in a similar sequence, the algorithm calculates the partial ratio as 100 and ignores simple additions (new characters).\n",
"```\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5f048152519f"
},
"source": [
"Compute a score to perform fuzzy matching:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "urwYXS_7Bg4x"
},
"outputs": [],
"source": [
"qa_data_df = pd.read_csv(\"df.csv\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "040c1f9a175b"
},
"outputs": [],
"source": [
"def get_fuzzy_match(df):\n",
" return fuzz.partial_ratio(df[\"answer_groundtruth\"], df[\"answer_prediction\"])\n",
"\n",
"\n",
"qa_data_df[\"match_score\"] = qa_data_df.apply(get_fuzzy_match, axis=1)\n",
"qa_data_df"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "11e266c49860"
},
"source": [
"Now that you have the individual match score (partial), you can take the mean or average of the whole column to get a sense of overall data.\n",
"Scores closer to 100 mean Gemini can predict closer to ground truth; if the score is towards 50 or 0, it did not perform well."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "dae6a92a7650"
},
"outputs": [],
"source": [
"print(\n",
" \"the average match score of all predicted answer from Gemini is : \",\n",
" qa_data_df[\"match_score\"].mean(),\n",
" \" %\",\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d9e78972cad1"
},
"source": [
"In this case, you get 100% as the mean score, even though some predictions were missing some words. That means you are very close to the ground truth, and some answers are just missing the exact verboseness of the ground truth."
]
}
],
"metadata": {
"colab": {
"name": "question_answering.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Reference in New Issue View Git Blame Copy Permalink