wehub-resource-sync
508 lines
22 KiB
Plaintext
508 lines
22 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "ur8xi4C7S06n"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2025 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "JAPoU8Sm5E6e"
|
|
},
|
|
"source": [
|
|
"# Evaluate images with predefined Gecko\n",
|
|
"\n",
|
|
"<table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_images_with_predefined_gecko.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fevaluation%2Fevaluate_images_with_predefined_gecko.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td> \n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/evaluation/evaluate_images_with_predefined_gecko.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main//gemini/evaluation/evaluate_images_with_predefined_gecko.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>\n",
|
|
"\n",
|
|
"\n",
|
|
"<b>Share to:</b>\n",
|
|
"\n",
|
|
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_images_with_predefined_gecko.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_images_with_predefined_gecko.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_images_with_predefined_gecko.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_images_with_predefined_gecko.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_images_with_predefined_gecko.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
|
"</a>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "UBAgFfnqvxy0"
|
|
},
|
|
"source": [
|
|
" | | | | |\n",
|
|
" |-|-|-|-|\n",
|
|
" |Author(s): | [Greg Breard](https://github.com/gregbreard) | Anant Nawalgaria | Olivia Wiles |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "tvgnzT1CKxrO"
|
|
},
|
|
"source": [
|
|
"## Overview\n",
|
|
"\n",
|
|
"This CoLAB shows how to leverage the Vertex AI evaluation service in order to run [Gecko](https://arxiv.org/abs/2404.16820).\n",
|
|
"\n",
|
|
"As with a more standard rubric approach, Gecko proceeds in two stages: a rubric generation step followed by a validator step. The key difference is that the rubric is generated based on the prompt.\n",
|
|
"This allows for a more fine-grained metric that can be customized to prompts with differing challenges.\n",
|
|
"\n",
|
|
"In more detail, Gecko proceeds as follows, with two key steps: the QA generation step (ie the rubric generation step) and then the VQA step (ie the validator step).\n",
|
|
"\n",
|
|
"## The rubric generation step\n",
|
|
"Given a prompt, such as `A teddy bear riding a skateboard`, we prompt the Gemini model to generate a set of questions, answer choices and corresponding ground truth (GT) answer. The question is also tagged with a question type. Depending on the prompt, these questions can either be `yes`/`no` questions or multiple choice ones.\n",
|
|
"\n",
|
|
"`A teddy bear riding a skateboard` -->\n",
|
|
"\n",
|
|
"- `Q1: Is there a teddy bear? Choices: [yes, no]. GT Answer: yes. Tag: Object.`\n",
|
|
"- `Q2: Is there a skateboard? Choices: [yes, no]. GT Answer: yes. Tag: Object.`\n",
|
|
"- `Q3: Is the teddy bear riding a skateboard? Choices: [yes, no]. GT Answer: yes. Tag: Action.`\n",
|
|
"\n",
|
|
"## The validator step\n",
|
|
"Given a generated image and the questions above, we query the Gemini model for each question to give an answer. We then check if it matches the GT answer, with a result of 1 if it matches and 0 if it does not. We aggregate these results to give a final overall score, which can be broken down into scores per question. We can also aggregate scores based on tags.\n",
|
|
"\n",
|
|
"For example, imagine we have a generated image `<image1>` which includes a teddy bear but no skateboard, and Gemini outputs the following results:\n",
|
|
"\n",
|
|
"- `<image1> Is there a teddy bear? GT Answer: yes. Result: 1.`\n",
|
|
"- `<image1> Is there a skateboard? GT Answer: no. Result: 0.`\n",
|
|
"- `<image1> Is the teddy bear riding a skateboard? GT Answer: no. Result: 0.`\n",
|
|
"\n",
|
|
"The final score will be `0.33` with a score of `0.5` for the question tag and `0.0` for the action tag.\n",
|
|
"\n",
|
|
"## Further exploration\n",
|
|
"We provide two predefined metrics, engineered for video and image generation tasks. Below, we show how to run Gecko for the image modality on a set of generations.\n",
|
|
"\n",
|
|
"The quality can be analysed by exploring what questions are generated as well as the reliability of the validator step. Questions can also be manually added as desired for an application.\n",
|
|
"\n",
|
|
"## Steps\n",
|
|
"\n",
|
|
"1. Set up the environment.\n",
|
|
"2. Prepare the dataset for evaluation.\n",
|
|
"3. Run the evaluation.\n",
|
|
"\n",
|
|
"## Costs\n",
|
|
"This tutorial uses billable components of Google Cloud:\n",
|
|
"\n",
|
|
"- Vertex AI\n",
|
|
"\n",
|
|
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "61RBz8LLbxCR"
|
|
},
|
|
"source": [
|
|
"# Get started"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "tFy3H3aPgx12"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# @title ### Install Vertex AI SDK for Python and other required packages\n",
|
|
"\n",
|
|
"%pip install --upgrade --quiet \"google-cloud-aiplatform[evaluation]>=1.122.0\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "NyKGtVQjgx13"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# @title ### Authenticate your notebook environment (Colab only)\n",
|
|
"# @markdown If you're running this notebook on Google Colab, run the cell below to authenticate your environment.\n",
|
|
"\n",
|
|
"import sys\n",
|
|
"\n",
|
|
"if \"google.colab\" in sys.modules:\n",
|
|
" from google.colab import auth\n",
|
|
"\n",
|
|
" auth.authenticate_user()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Nqwi-5ufWp_B"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# @title ### Set Google Cloud project information\n",
|
|
"# @markdown To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
|
|
"# @markdown Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment).\n",
|
|
"\n",
|
|
"# @markdown ---\n",
|
|
"\n",
|
|
"# fmt: off\n",
|
|
"PROJECT_ID = \"\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
|
|
"LOCATION= \"us-central1\" # @param {type: \"string\", placeholder: \"us-central1\", isTemplate: true}\n",
|
|
"# fmt: on\n",
|
|
"\n",
|
|
"from vertexai import Client, types\n",
|
|
"\n",
|
|
"client = Client(project=PROJECT_ID, location=LOCATION)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Gv7YyT-l657I"
|
|
},
|
|
"source": [
|
|
"# Prepare the dataset"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "k_pfeWdR65rv"
|
|
},
|
|
"source": [
|
|
"In the following dataset, two prompts are used for each generated image. The first is the prompt that corresponds to the generated content. The second is a counterexample that is similar but does not exactly match the generated content. This is done to demonstrate the difference in the Gecko evaluation for high quality and low quality responses."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "U37FNLD8-Lp5"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import pandas as pd\n",
|
|
"\n",
|
|
"prompts = [\n",
|
|
" \"steaming cup of coffee and a croissant on a table\",\n",
|
|
" \"steaming cup of coffee and toast in a cafe\",\n",
|
|
" \"sunset over a calm ocean\",\n",
|
|
" \"sunset over a tranquil forest\",\n",
|
|
" \"butterfly with colorful wings on a flower\",\n",
|
|
" \"butterfly fluttering over a leaf\",\n",
|
|
" \"musician playing guitar on a street corner\",\n",
|
|
" \"musician playing saxophone under lamp post\",\n",
|
|
" \"vintage camera with a worn leather strap\",\n",
|
|
" \"new camera with a power zoom lens\",\n",
|
|
" \"colorful abstract painting\",\n",
|
|
" \"black and white painting\",\n",
|
|
" \"baker decorating a cake with frosting\",\n",
|
|
" \"baker topping cupcakes with sprinkles\",\n",
|
|
" \"hot air balloon floating above a field of lavender\",\n",
|
|
" \"hot air balloon landing in a field of sunflowers\",\n",
|
|
"]\n",
|
|
"responses = [\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"image/png\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/images/coffee.png\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"image/png\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/images/coffee.png\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"image/png\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/images/sunset.png\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"image/png\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/images/sunset.png\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"image/png\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/images/butterfly.png\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"image/png\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/images/butterfly.png\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"image/png\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/images/musician.png\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"image/png\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/images/musician.png\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"image/png\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/images/camera.png\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"image/png\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/images/camera.png\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"image/png\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/images/abstract.png\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"image/png\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/images/abstract.png\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"image/png\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/images/baker.png\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"image/png\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/images/baker.png\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"image/png\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/images/balloon.png\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"image/png\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/images/balloon.png\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
"]\n",
|
|
"\n",
|
|
"eval_dataset = pd.DataFrame(\n",
|
|
" {\n",
|
|
" \"prompt\": prompts,\n",
|
|
" \"response\": responses,\n",
|
|
" }\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "qm6mHoPc-DS0"
|
|
},
|
|
"source": [
|
|
"# Run evaluation"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "AfgCsW1MZ8dj"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# @title ### Generate rubrics\n",
|
|
"# @markdown First we generate rubrics for the user prompts.\n",
|
|
"\n",
|
|
"data_with_rubrics = client.evals.generate_rubrics(\n",
|
|
" src=eval_dataset,\n",
|
|
" rubric_group_name=\"gecko_image_rubrics\",\n",
|
|
" predefined_spec_name=types.RubricMetric.GECKO_TEXT2IMAGE,\n",
|
|
")\n",
|
|
"\n",
|
|
"data_with_rubrics.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "mMuDxiwy-MTt"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# @title ### Evaluate with rubrics\n",
|
|
"# @markdown Then we use the generated rubrics to evaluate the quality of the responses.\n",
|
|
"\n",
|
|
"eval_result = client.evals.evaluate(\n",
|
|
" dataset=data_with_rubrics,\n",
|
|
" metrics=[types.RubricMetric.GECKO_TEXT2IMAGE],\n",
|
|
")\n",
|
|
"\n",
|
|
"eval_result.show()"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"name": "evaluate_images_with_predefined_gecko.ipynb",
|
|
"toc_visible": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|