mirrors/aws--amazon-sagemaker-examples
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:40:10 +08:00
commit 68ca4df005
942 changed files with 432462 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
responsible_ai/README.md
+18
View File
@@ -0,0 +1,18 @@
# Amazon SageMaker Examples
### Responsible AI
Amazon SageMaker offers features to improve your machine learning (ML) models by detecting potential bias and helping to explain the predictions that your models make from your tabular, computer vision, natural processing, or time series datasets as well as providing purpose-built ML governance tools for managing control access, activity tracking, and reporting across the ML lifecycle.
- [Explaining Autopilot Models](sm-autopilot_model_explanation_with_shap/sm-autopilot_model_explanation_with_shap.ipynb)
- [Explaining Image Classification with SageMaker Clarify](sm-clarify_explainability_image_classification/sm-clarify_explainability_image_classification.ipynb)
- [NLP Online Explainability with SageMaker Clarify](sm-clarify_natural_language_processing_online_explainability/sm-clarify_natural_language_processing_online_explainability.ipynb)
- [Explaining Object Detection model with Amazon SageMaker Clarify](sm-clarify_object_detection/sm-clarify_object_detection.ipynb)
- [SageMaker Clarify Online Explainability on Multi-Model Endpoint](sm-clarify_online_explainability_mme_xgboost/sm-clarify_online_explainability_mme_xgboost.ipynb)
- [Tabular Online Explainability with SageMaker Clarify](sm-clarify_online_explainability_tabular/sm-clarify_online_explainability_tabular.ipynb)
- [Explaining text sentiment analysis using SageMaker Clarify](sm-clarify_text_explainability_text_sentiment_analysis/sm-clarify_text_explainability_text_sentiment_analysis.ipynb)
- [TimeSeries Bring Your Own Model](sm-clarify_time_series_bring_your_own_model/sm-clarify_time_series_bring_your_own_model.ipynb)
- [Amazon SageMaker Model Governance - Model Cards](sm-model_governance_model_card/sm-model_governance_model_card.ipynb)
- [Amazon SageMaker Model Governance - Model Cards Model Registry integration](sm-model_governance_model_card_with_model_package/sm-model_governance_model_card_with_model_package.ipynb)
- [Fairness and Explainability with SageMaker Clarify - Bring Your Own Container](sm-clarify_fairness_and_explainability_bring_your_own_container.ipynb)
- [Fairness and Explainability with SageMaker Clarify using AWS SDK for Python (Boto3)](sm-clarify_fairness_and_explainability_with_boto3.ipynb)
responsible_ai/sm-autopilot_model_explanation_with_shap/managed_endpoint.py
+23
View File
@@ -0,0 +1,23 @@
import boto3
region = boto3.Session().region_name
sm = boto3.Session().client(service_name="sagemaker", region_name=region)
class ManagedEndpoint:
def __init__(self, ep_name, auto_delete=False):
self.name = ep_name
self.auto_delete = auto_delete
def __enter__(self):
endpoint_description = sm.describe_endpoint(EndpointName=self.name)
if endpoint_description["EndpointStatus"] == "InService":
self.in_service = True
def __exit__(self, type, value, traceback):
if self.in_service and self.auto_delete:
print("Deleting the endpoint: {}".format(self.name))
sm.delete_endpoint(EndpointName=self.name)
sm.get_waiter("endpoint_deleted").wait(EndpointName=self.name)
self.in_service = False
responsible_ai/sm-autopilot_model_explanation_with_shap/sm-autopilot_model_explanation_with_shap.ipynb
+530
View File
@@ -0,0 +1,530 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Explaining Autopilot Models\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
"\n",
"![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/responsible_ai|sm-autopilot_model_explanation_with_shap|sm-autopilot_model_explanation_with_shap.ipynb)\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"Kernel `Python 3 (Data Science)` works well with this notebook.\n",
"\n",
"_This notebook was created and tested on an ml.m5.xlarge notebook instance._\n",
"\n",
"## Table of Contents\n",
"\n",
"1. [Introduction](#introduction)\n",
"3. [Setup](#setup)\n",
"4. [Local explanation with KernelExplainer](#Local)\n",
"5. [KernelExplainer computation cost](#cost) \n",
"6. [Global explanation with KernelExplainer](#global)\n",
"7. [Conclusion](#conclusion)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## Introduction<a name=\"introduction\"></a>\n",
"Machine learning (ML) models have long been considered black boxes since predictions from these models are hard to interpret. While decision trees can be interpreted by observing the parameters learned by the models, it is generally difficult to get a clear picture.\n",
"\n",
"Model interpretation can be divided into local and global explanations. A local explanation considers a single sample and answers questions like: \"why the model predicts that customer A will stop using the product?\" or \"why the ML system refused John Doe a loan?\". Another interesting question is \"what should John Doe change in order to get the loan approved?\". On the contrary, global explanations aim at explaining the model itself and answer questions like \"which features are important for prediction?\". It is important to note that local explanations can be used to derive global explanations by averaging many samples. For further reading on interpretable ML, see the excellent book by [Christoph Molnar](https://christophm.github.io/interpretable-ml-book).\n",
"\n",
"In this blog post, we will demonstrate the use of the popular model interpretation framework [SHAP](https://github.com/slundberg/shap) for both local and global interpretation."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### SHAP"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[SHAP](https://github.com/slundberg/shap) is a game theoretic framework inspired by [Shapley Values](https://www.rand.org/pubs/papers/P0295.html) that provides local explanations for any model. SHAP has gained popularity in recent years, probably due to its strong theoretical basis. The SHAP package contains several algorithms that, given a sample and a model, derive the SHAP value for each of the model's input features. The SHAP value of a feature represents the feature's contribution to the model's prediction.\n",
"\n",
"To explain models built by [Amazon SageMaker Autopilot](https://aws.amazon.com/sagemaker/autopilot/) we use SHAP's `KernelExplainer` which is a black box explainer. `KernelExplainer` is robust and can explain any model, thus can handle Autopilot's complex feature processing. `KernelExplainer` only requires that the model will support an inference functionality which, given a sample, will return the model's prediction for that sample. The prediction being the predicted value for regression and the class probability for classification.\n",
"\n",
"It is worth noting that SHAP includes several other explainers such as `TreeExplainer` and `DeepExplainer` that are specific for decision forest and neural networks respectively. These are not black box explainers and require knowledge of the model structure and trained params. `TreeExplainer` and `DeepExplainer` are limited and currently can not support any feature processing."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## Setup<a name=\"setup\"></a>\n",
"In this notebook we will start with a model built by SageMaker Autopilot which was already trained on a binary classification task. Please refer to this [notebook](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/autopilot/autopilot_customer_churn.ipynb) to see how to create and train an Autopilot model. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import boto3\n",
"import pandas as pd\n",
"import sagemaker\n",
"from sagemaker import AutoML\n",
"from datetime import datetime\n",
"import numpy as np\n",
"\n",
"region = boto3.Session().region_name\n",
"session = sagemaker.Session()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Install SHAP"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%conda install -c conda-forge shap"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import shap\n",
"\n",
"from shap import KernelExplainer\n",
"from shap import sample\n",
"from scipy.special import expit\n",
"\n",
"# Initialize plugin to make plots interactive.\n",
"shap.initjs()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Create an inference endpoint<a name=\"Endpoint\"></a>\n",
"Creating an inference endpoint for the trained Autopilot model. Skip this step if an endpoint with the argument `inference_response_keys` set as\n",
"`['predicted_label', 'probability']` was already created."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"automl_job_name = \"your-autopilot-job-that-exists\"\n",
"automl_job = AutoML.attach(automl_job_name, sagemaker_session=session)\n",
"\n",
"# Endpoint name\n",
"ep_name = \"sagemaker-automl-\" + datetime.now().strftime(\"%Y-%m-%d-%H-%M-%S\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# For classification response to work with SHAP we need the probability scores. This can be achieved by providing a list of keys for\n",
"# response content. The order of the keys will dictate the content order in the response. This parameter is not needed for regression.\n",
"inference_response_keys = [\"predicted_label\", \"probability\"]\n",
"\n",
"# Create the inference endpoint\n",
"automl_job.deploy(\n",
" initial_instance_count=1,\n",
" instance_type=\"ml.m5.2xlarge\",\n",
" inference_response_keys=inference_response_keys,\n",
" endpoint_name=ep_name,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Wrap Autopilot's endpoint with an estimator class.<a name=\"Endpoint\"></a>\n",
"For ease of use, we wrap the inference endpoint with a custom estimator class. Two inference functions are provided: `predict` which\n",
"returns the numeric prediction value to be used for regression and `predict_proba` which returns the class probability to be used for\n",
"classification."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sagemaker.predictor import Predictor\n",
"\n",
"\n",
"class AutomlEstimator:\n",
" def __init__(self, endpoint_name, sagemaker_session):\n",
" self.predictor = Predictor(\n",
" endpoint_name=endpoint_name,\n",
" sagemaker_session=sagemaker_session,\n",
" serializer=sagemaker.serializers.CSVSerializer(),\n",
" content_type=\"text/csv\",\n",
" accept=\"text/csv\",\n",
" )\n",
"\n",
" def get_automl_response(self, x):\n",
" if x.__class__.__name__ == \"ndarray\":\n",
" payload = \"\"\n",
" for row in x:\n",
" payload = payload + \",\".join(map(str, row)) + \"\\n\"\n",
" else:\n",
" payload = x.to_csv(sep=\",\", header=False, index=False)\n",
" return self.predictor.predict(payload).decode(\"utf-8\")\n",
"\n",
" # Prediction function for regression\n",
" def predict(self, x):\n",
" response = self.get_automl_response(x)\n",
" # we get the first column from the response array containing the numeric prediction value (or label in case of classification)\n",
" response = np.array([x.split(\",\")[0] for x in response.split(\"\\n\")[:-1]])\n",
" return response\n",
"\n",
" # Prediction function for classification\n",
" def predict_proba(self, x):\n",
" \"\"\"Extract and return the probability score from the AutoPilot endpoint response.\"\"\"\n",
" response = self.get_automl_response(x)\n",
" # we get the second column from the response array containing the class probability\n",
" response = np.array([x.split(\",\")[1] for x in response.split(\"\\n\")[:-1]])\n",
" return response.astype(float)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Create an instance of `AutomlEstimator`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"automl_estimator = AutomlEstimator(endpoint_name=ep_name, sagemaker_session=session)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Data\n",
"In this notebook we will use the same dataset as used in the [Customer Churn notebook.](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/autopilot/autopilot_customer_churn.ipynb)\n",
"Please follow the \"Customer Churn\" notebook to download the dataset if it was not previously downloaded.\n",
"\n",
"### Background data\n",
"KernelExplainer requires a sample of the data to be used as background data. KernelExplainer uses this data to simulate a feature being missing by replacing the feature value with a random value from the background. We use shap.sample to sample 50 rows from the dataset to be used as background data. Using more samples as background data will produce more accurate results but runtime will increase. Choosing background data is challenging, see the whitepapers: https://storage.googleapis.com/cloud-ai-whitepapers/AI%20Explainability%20Whitepaper.pdf and https://docs.seldon.io/projects/alibi/en/latest/methods/KernelSHAP.html#Runtime-considerations. Note that the clustering algorithms provided in shap only support numeric data. According to SHAP's documentation, a vector of zeros could be used as background data to produce reasonable results."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"churn_data = pd.read_csv(\"../churn.txt\")\n",
"data_without_target = churn_data.drop(columns=[\"Churn?\"])\n",
"\n",
"background_data = sample(data_without_target, 50)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Setup KernelExplainer"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, we create the `KernelExplainer`. Note that since it's a black box explainer, `KernelExplainer` only requires a handle to the\n",
"predict (or predict_proba) function and does not require any other information about the model. For classification it is recommended to\n",
"derive feature importance scores in the log-odds space since additivity is a more natural assumption there thus we use `logit`. For\n",
"regression `identity` should be used."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Derive link function\n",
"problem_type = automl_job.describe_auto_ml_job(job_name=automl_job_name)[\"ResolvedAttributes\"][\n",
" \"ProblemType\"\n",
"]\n",
"link = \"identity\" if problem_type == \"Regression\" else \"logit\"\n",
"\n",
"# the handle to predict_proba is passed to KernelExplainer since KernelSHAP requires the class probability\n",
"explainer = KernelExplainer(automl_estimator.predict_proba, background_data, link=link)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"By analyzing the background data `KernelExplainer` provides us with `explainer.expected_value` which is the model prediction with all features missing. Considering a customer for which we have no data at all (i.e. all features are missing) this should theoretically be the model prediction."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Since expected_value is given in the log-odds space we convert it back to probability using expit which is the inverse function to logit\n",
"print(\"expected value =\", expit(explainer.expected_value))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## Local explanation with KernelExplainer<a name=\"local\"></a>\n",
"We will use `KernelExplainer` to explain the prediction of a single sample, the first sample in the dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Get the first sample\n",
"x = data_without_target.iloc[0:1]\n",
"\n",
"# ManagedEndpoint can optionally auto delete the endpoint after calculating the SHAP values. To enable auto delete, use ManagedEndpoint(ep_name, auto_delete=True)\n",
"from managed_endpoint import ManagedEndpoint\n",
"\n",
"with ManagedEndpoint(ep_name) as mep:\n",
" shap_values = explainer.shap_values(x, nsamples=\"auto\", l1_reg=\"aic\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"SHAP package includes many visualization tools. See below a `force_plot` which provides a good visualization for the SHAP values of a single sample"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Since shap_values are provided in the log-odds space, we convert them back to the probability space by using LogitLink\n",
"shap.force_plot(explainer.expected_value, shap_values, x, link=link)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"From the plot above we learn that the most influential feature is `VMail Message` which pushes the probability down by about 7%. It is\n",
"important to note that `VMail Message = 25` makes the probability 7% lower in comparison to the notion of that feature being missing.\n",
"SHAP values do not provide the information of how increasing or decreasing `VMail Message` will affect prediction.\n",
"\n",
"In many cases we are interested only in the most influential features. By setting `l1_reg='num_features(5)'`, SHAP will provide non-zero scores for only the most influential 5 features."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"with ManagedEndpoint(ep_name) as mep:\n",
" shap_values = explainer.shap_values(x, nsamples=\"auto\", l1_reg=\"num_features(5)\")\n",
"shap.force_plot(explainer.expected_value, shap_values, x, link=link)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## KernelExplainer computation cost<a name=\"cost\"></a>\n",
"KernelExplainer computation cost is dominated by the inference calls. In order to estimate SHAP values for a single sample, KernelExplainer calls the inference function twice: First, with the sample unaugmented. And second, with many randomly augmented instances of the sample. The number of augmented instances in our case is: 50 (#samples in the background data) * 2088 (nsamples = 'auto') = 104,400. So, in our case, the cost of running KernelExplainer for a single sample is roughly the cost of 104,400 inference calls."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## Global explanation with KernelExplainer<a name=\"global\"></a>\n",
"Next, we will use KernelExplainer to provide insight about the model as a whole. It is done by running KernelExplainer locally on 50 samples and aggregating the results"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Sample 50 random samples\n",
"X = sample(data_without_target, 50)\n",
"\n",
"# Calculate SHAP values for these samples, and delete the endpoint\n",
"with ManagedEndpoint(ep_name, auto_delete=True) as mep:\n",
" shap_values = explainer.shap_values(X, nsamples=\"auto\", l1_reg=\"aic\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`force_plot` can be used to visualize SHAP values for many samples simultaneously by rotating the plot of each sample by 90 degrees and stacking the plots horizontally. The resulting plot is interactive and can be manually analyzed."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"shap.force_plot(explainer.expected_value, shap_values, X, link=link)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`summary_plot` is another visualization tool displaying the mean absolute value of the SHAP values for each feature using a bar plot. Currently, `summary_plot` does not support link functions so the SHAP values are presented in the log-odds space (and not the probability space)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"shap.summary_plot(shap_values, X, plot_type=\"bar\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## Conclusion<a name=\"conclusion\"></a>\n",
"\n",
"In this post, we demonstrated how to use KernelSHAP to explain models created by Amazon SageMaker Autopilot both locally and globally. KernelExplainer is a robust black box explainer which requires only that the model will support an inference functionality which, given a sample, returns the model's prediction for that sample. This inference functionality was provided by wrapping Autopilot's inference endpoint with an estimator container.\n",
"\n",
"For more about Amazon SageMaker Autopilot, please see [Amazon SageMaker Autopilot](https://aws.amazon.com/sagemaker/autopilot/)."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Notebook CI Test Results\n",
"\n",
"This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
"\n",
"![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/responsible_ai|sm-autopilot_model_explanation_with_shap|sm-autopilot_model_explanation_with_shap.ipynb)\n",
"\n",
"![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/responsible_ai|sm-autopilot_model_explanation_with_shap|sm-autopilot_model_explanation_with_shap.ipynb)\n",
"\n",
"![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/responsible_ai|sm-autopilot_model_explanation_with_shap|sm-autopilot_model_explanation_with_shap.ipynb)\n",
"\n",
"![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/responsible_ai|sm-autopilot_model_explanation_with_shap|sm-autopilot_model_explanation_with_shap.ipynb)\n",
"\n",
"![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/responsible_ai|sm-autopilot_model_explanation_with_shap|sm-autopilot_model_explanation_with_shap.ipynb)\n",
"\n",
"![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/responsible_ai|sm-autopilot_model_explanation_with_shap|sm-autopilot_model_explanation_with_shap.ipynb)\n",
"\n",
"![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/responsible_ai|sm-autopilot_model_explanation_with_shap|sm-autopilot_model_explanation_with_shap.ipynb)\n",
"\n",
"![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/responsible_ai|sm-autopilot_model_explanation_with_shap|sm-autopilot_model_explanation_with_shap.ipynb)\n",
"\n",
"![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/responsible_ai|sm-autopilot_model_explanation_with_shap|sm-autopilot_model_explanation_with_shap.ipynb)\n",
"\n",
"![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/responsible_ai|sm-autopilot_model_explanation_with_shap|sm-autopilot_model_explanation_with_shap.ipynb)\n",
"\n",
"![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/responsible_ai|sm-autopilot_model_explanation_with_shap|sm-autopilot_model_explanation_with_shap.ipynb)\n",
"\n",
"![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/responsible_ai|sm-autopilot_model_explanation_with_shap|sm-autopilot_model_explanation_with_shap.ipynb)\n",
"\n",
"![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/responsible_ai|sm-autopilot_model_explanation_with_shap|sm-autopilot_model_explanation_with_shap.ipynb)\n",
"\n",
"![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/responsible_ai|sm-autopilot_model_explanation_with_shap|sm-autopilot_model_explanation_with_shap.ipynb)\n",
"\n",
"![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/responsible_ai|sm-autopilot_model_explanation_with_shap|sm-autopilot_model_explanation_with_shap.ipynb)\n"
]
}
],
"metadata": {
"instance_type": "ml.t3.medium",
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.4"
},
"pycharm": {
"stem_cell": {
"cell_type": "raw",
"metadata": {
"collapsed": false
},
"source": []
}
}
},
"nbformat": 4,
"nbformat_minor": 4
}
responsible_ai/sm-clarify_explainability_image_classification/caltech_256_object_categories.txt
+257
View File
@@ -0,0 +1,257 @@
ak47
american-flag
backpack
baseball-bat
baseball-glove
basketball-hoop
bat
bathtub
bear
beer-mug
billiards
binoculars
birdbath
blimp
bonsai-101
boom-box
bowling-ball
bowling-pin
boxing-glove
brain-101
breadmaker
buddha-101
bulldozer
butterfly
cactus
cake
calculator
camel
cannon
canoe
car-tire
cartman
cd
centipede
cereal-box
chandelier-101
chess-board
chimp
chopsticks
cockroach
coffee-mug
coffin
coin
comet
computer-keyboard
computer-monitor
computer-mouse
conch
cormorant
covered-wagon
cowboy-hat
crab-101
desk-globe
diamond-ring
dice
dog
dolphin-101
doorknob
drinking-straw
duck
dumb-bell
eiffel-tower
electric-guitar-101
elephant-101
elk
ewer-101
eyeglasses
fern
fighter-jet
fire-extinguisher
fire-hydrant
fire-truck
fireworks
flashlight
floppy-disk
football-helmet
french-horn
fried-egg
frisbee
frog
frying-pan
galaxy
gas-pump
giraffe
goat
golden-gate-bridge
goldfish
golf-ball
goose
gorilla
grand-piano-101
grapes
grasshopper
guitar-pick
hamburger
hammock
harmonica
harp
harpsichord
hawksbill-101
head-phones
helicopter-101
hibiscus
homer-simpson
horse
horseshoe-crab
hot-air-balloon
hot-dog
hot-tub
hourglass
house-fly
human-skeleton
hummingbird
ibis-101
ice-cream-cone
iguana
ipod
iris
jesus-christ
joy-stick
kangaroo-101
kayak
ketch-101
killer-whale
knife
ladder
laptop-101
lathe
leopards-101
license-plate
lightbulb
light-house
lightning
llama-101
mailbox
mandolin
mars
mattress
megaphone
menorah-101
microscope
microwave
minaret
minotaur
motorbikes-101
mountain-bike
mushroom
mussels
necktie
octopus
ostrich
owl
palm-pilot
palm-tree
paperclip
paper-shredder
pci-card
penguin
people
pez-dispenser
photocopier
picnic-table
playing-card
porcupine
pram
praying-mantis
pyramid
raccoon
radio-telescope
rainbow
refrigerator
revolver-101
rifle
rotary-phone
roulette-wheel
saddle
saturn
school-bus
scorpion-101
screwdriver
segway
self-propelled-lawn-mower
sextant
sheet-music
skateboard
skunk
skyscraper
smokestack
snail
snake
sneaker
snowmobile
soccer-ball
socks
soda-can
spaghetti
speed-boat
spider
spoon
stained-glass
starfish-101
steering-wheel
stirrups
sunflower-101
superman
sushi
swan
swiss-army-knife
sword
syringe
tambourine
teapot
teddy-bear
teepee
telephone-box
tennis-ball
tennis-court
tennis-racket
theodolite
toaster
tomato
tombstone
top-hat
touring-bike
tower-pisa
traffic-light
treadmill
triceratops
tricycle
trilobite-101
tripod
t-shirt
tuning-fork
tweezer
umbrella-101
unicorn
vcr
video-projector
washing-machine
watch-101
waterfall
watermelon
welding-mask
wheelbarrow
windmill
wine-bottle
xylophone
yarmulke
yo-yo
zebra
airplanes-101
car-side-101
faces-easy-101
greyhound
tennis-shoes
toad
clutter
responsible_ai/sm-clarify_explainability_image_classification/sm-clarify_explainability_image_classification.ipynb
+778
View File
@@ -0,0 +1,778 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Explaining Image Classification with SageMaker Clarify"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
"\n",
"![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/responsible_ai|sm-clarify_explainability_image_classification|sm-clarify_explainability_image_classification.ipynb)\n",
"\n",
"---"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"1. [Overview](#Overview)\n",
"1. [Train and Deploy Image Classifier](#Train-and-Deploy-Image-Classifier)\n",
" 1. [Permissions and environment variables](#Permissions-and-environment-variables)\n",
" 2. [Fine-tuning the Image classification model](#Fine-tuning-the-Image-classification-model)\n",
" 3. [Training](#Training)\n",
" 4. [Input data specification](#Input-data-specification)\n",
" 5. [Start the training](#Start-the-training)\n",
" 6. [Deploy SageMaker model](#Deploy-SageMaker-model)\n",
" 7. [List of object categories](#List-of-object-categories)\n",
"1. [Amazon SageMaker Clarify](#Amazon-SageMaker-Clarify)\n",
" 1. [Test Images](#Test-Images)\n",
" 2. [Set up config objects](#Set-up-config-objects)\n",
" 3. [SageMaker Clarify Processor](#SageMaker-Clarify-Processor)\n",
" 4. [Reading Results](#Reading-Results)\n",
"1. [Clean Up](#Clean-Up)\n",
"\n",
"## Overview\n",
"Amazon SageMaker Clarify provides you the ability to gain an insight into your Computer Vision models. Clarify generates heat map, which highlights feature importance, for each input image and helps understand the model behavior. For Computer Vision, Clarify supports both Image Classification and Object Detection use cases.\n",
"This notebook can be run inside the SageMaker Studio with **conda_pytorch_latest_py36** kernel and inside SageMaker Notebook-Instance with **Python 3 (PyTorch 1.8 Python 3.6 GPU Optimized)** kernel.\n",
"This sample notebook walks you through:\n",
"1. Key terms and concepts needed to understand SageMaker Clarify.\n",
"1. Explaining the importance of the image features (super pixels) for Image Classification model.\n",
"1. Accessing the reports and output images.\n",
"\n",
"In doing so, the notebook will first train and deploy an [Image Classification](https://github.com/aws/amazon-sagemaker-examples/blob/master/introduction_to_amazon_algorithms/imageclassification_caltech/Image-classification-transfer-learning-highlevel.ipynb) model with Sagemaker Estimator using [Caltech-256 dataset](https://paperswithcode.com/dataset/caltech-256) [1], then use SageMaker Clarify to run explainability on a subset of test images.\n",
">[1] Griffin, G. Holub, AD. Perona, P. The Caltech 256. Caltech Technical Report."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's start by installing the latest version of the SageMaker Python SDK, boto, and AWS CLI."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"! pip install 'sagemaker<3.0' botocore boto3 awscli --upgrade"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"## Train and Deploy Image Classifier\n",
"Let's first train and deploy an Image Classification model to SageMaker.\n",
"\n",
"\n",
"### Permissions and environment variables\n",
"Here we set up the linkage and authentication to AWS services. There are three parts to this:\n",
"\n",
"* The roles used to give learning and hosting access to your data. This will automatically be obtained from the role used to start the notebook\n",
"* The S3 bucket that you want to use for training and model data\n",
"* The Amazon sagemaker image classification docker image which need not be changed"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"import boto3\n",
"import sagemaker\n",
"from sagemaker import get_execution_role\n",
"\n",
"role = get_execution_role()\n",
"print(role)\n",
"\n",
"region = boto3.Session().region_name\n",
"\n",
"s3_client = boto3.client(\"s3\")\n",
"\n",
"sess = sagemaker.Session()\n",
"\n",
"output_bucket = sess.default_bucket()\n",
"output_prefix = \"ic-transfer-learning\"\n",
"\n",
"default_bucket_prefix = sess.default_bucket_prefix\n",
"\n",
"# If a default bucket prefix is specified, append it to the s3 path\n",
"if default_bucket_prefix:\n",
" output_prefix = f\"{default_bucket_prefix}/{output_prefix}\"\n",
"\n",
"# download the files\n",
"s3_client.download_file(\n",
" f\"sagemaker-example-files-prod-{region}\",\n",
" \"datasets/image/caltech-256/caltech-256-60-train.rec\",\n",
" \"caltech-256-60-train.rec\",\n",
")\n",
"s3_client.download_file(\n",
" f\"sagemaker-example-files-prod-{region}\",\n",
" \"datasets/image/caltech-256/caltech-256-60-val.rec\",\n",
" \"caltech-256-60-val.rec\",\n",
")\n",
"\n",
"s3_client.upload_file(\n",
" \"caltech-256-60-train.rec\", output_bucket, output_prefix + \"/train_rec/caltech-256-60-train.rec\"\n",
")\n",
"\n",
"s3_client.upload_file(\n",
" \"caltech-256-60-train.rec\",\n",
" output_bucket,\n",
" output_prefix + \"/validation_rec/caltech-256-60-train.rec\",\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"from sagemaker import image_uris\n",
"\n",
"training_image = image_uris.retrieve(\n",
" \"image-classification\", sess.boto_region_name, version=\"latest\"\n",
")\n",
"\n",
"print(training_image)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"### Fine-tuning the Image classification model\n",
"\n",
"The Caltech-256 dataset consist of images from 257 categories (the last one being a clutter category) and has 30k images with a minimum of 80 images and a maximum of about 800 images per category.\n",
"\n",
"The image classification algorithm can take two types of input formats. The first is a [recordio format](https://mxnet.apache.org/versions/1.8.0/api/python/docs/api/mxnet/recordio/index.html) and the other is a [lst format](https://mxnet.apache.org/versions/1.6/api/r/docs/api/im2rec.html). Files for both these formats are available at http://data.dmlc.ml/mxnet/data/caltech-256/. In this example, we will use the recordio format for training and use the training/validation split [specified here](https://mxnet.apache.org/versions/1.0.0/faq/finetune.html#prepare-data)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"# Four channels: train, validation, train_lst, and validation_lst\n",
"s3train = f\"s3://{output_bucket}/{output_prefix}/train_rec/\"\n",
"s3validation = f\"s3://{output_bucket}/{output_prefix}/validation_rec/\""
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"### Training\n",
"Now that we are done with all the setup that is needed, we are ready to train our object detector. To begin, let us create a ``sageMaker.estimator.Estimator`` object. This estimator will launch the training job. There are two kinds of parameters that need to be set for training. Following are the parameters for the training job:\n",
"* **instance_count**: This is the number of instances on which to run the training. When the number of instances is greater than one, then the image classification algorithm will run in distributed settings.\n",
"* **instance_typee**: This indicates the type of machine on which to run the training. Typically, we use GPU instances for such training jobs.\n",
"* **output_path**: This the s3 folder in which the training output is stored."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"s3_output_location = f\"s3://{output_bucket}/{output_prefix}/output\"\n",
"ic_estimator = sagemaker.estimator.Estimator(\n",
" training_image,\n",
" role,\n",
" instance_count=1,\n",
" instance_type=\"ml.p2.xlarge\",\n",
" volume_size=50,\n",
" max_run=360000,\n",
" input_mode=\"File\",\n",
" output_path=s3_output_location,\n",
" sagemaker_session=sess,\n",
")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"Apart from the above set of parameters, there are hyperparameters that are specific to the algorithm. These are:\n",
"\n",
"* **num_layers**: The number of layers (depth) for the network. We use 18 for this training but other values such as 50, 152 can also be used.\n",
"* **use_pretrained_model**: Set to 1 to use pretrained model for transfer learning.\n",
"* **image_shape**: The input image dimensions,'num_channels, height, width', for the network. It should be no larger than the actual image size. The number of channels should be same as the actual image.\n",
"* **num_classes**: This is the number of output classes for the new dataset. ImageNet was trained with 1000 output classes but the number of output classes can be changed for fine-tuning. For caltech, we use 257 because it has 256 object categories + 1 clutter class.\n",
"* **num_training_samples**: This is the total number of training samples. It is set to 15240 for caltech dataset with the current split.\n",
"* **mini_batch_size**: The number of training samples used for each mini batch. In distributed training, the number of training samples used per batch will be N * mini_batch_size where N is the number of hosts on which training is run.\n",
"* **epochs**: Number of training epochs.\n",
"* **learning_rate**: Learning rate for training.\n",
"* **precision_dtype**: Training datatype precision (default: float32). If set to 'float16', the training will be done in mixed_precision mode and will be faster than float32 mode."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"ic_estimator.set_hyperparameters(\n",
" num_layers=18,\n",
" use_pretrained_model=1,\n",
" image_shape=\"3,224,224\",\n",
" num_classes=257,\n",
" num_training_samples=15420,\n",
" mini_batch_size=128,\n",
" epochs=2,\n",
" learning_rate=0.01,\n",
" precision_dtype=\"float32\",\n",
")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"### Input data specification\n",
"Set the data type and channels used for training."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"train_data = sagemaker.inputs.TrainingInput(\n",
" s3train,\n",
" distribution=\"FullyReplicated\",\n",
" content_type=\"application/x-recordio\",\n",
" s3_data_type=\"S3Prefix\",\n",
")\n",
"validation_data = sagemaker.inputs.TrainingInput(\n",
" s3validation,\n",
" distribution=\"FullyReplicated\",\n",
" content_type=\"application/x-recordio\",\n",
" s3_data_type=\"S3Prefix\",\n",
")\n",
"\n",
"data_channels = {\"train\": train_data, \"validation\": validation_data}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"### Start the training\n",
"Start training by calling the fit method in the estimator."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"ic_estimator.fit(inputs=data_channels, logs=True)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"### Deploy SageMaker model\n",
"Once trained, we use the estimator to deploy a model to SageMaker. This model will be used by Clarify to deploy endpoints and run inference on images."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"from time import gmtime, strftime\n",
"\n",
"timestamp_suffix = strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
"\n",
"model_name = \"DEMO-clarify-image-classification-model-{}\".format(timestamp_suffix)\n",
"model = ic_estimator.create_model(name=model_name)\n",
"container_def = model.prepare_container_def()\n",
"sess.create_model(model_name, role, container_def)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"### List of object categories"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"with open(\"caltech_256_object_categories.txt\", \"r+\") as object_categories_file:\n",
" object_categories = [category.rstrip(\"\\n\") for category in object_categories_file.readlines()]\n",
"\n",
"# Let's list top 10 entries from the object_categories list\n",
"object_categories[:10]"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Amazon SageMaker Clarify\n",
"Now that we have your image classification endpoint all set up, let's get started with SageMaker Clarify!"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"### Test Images\n",
"We need some test images to explain predictions made by the Image Classification model using Clarify. Let's grab some test images from the Caltech-256 dataset and upload them to some S3 bucket."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"prefix = \"sagemaker/DEMO-sagemaker-clarify-cv\"\n",
"file_name_map = {\n",
" \"167.pyramid/167_0002.jpg\": \"pyramid.jpg\",\n",
" \"038.chimp/038_0013.jpg\": \"chimp.jpg\",\n",
" \"124.killer-whale/124_0013.jpg\": \"killer-whale.jpg\",\n",
" \"170.rainbow/170_0001.jpg\": \"rainbow.jpg\",\n",
"}\n",
"\n",
"\n",
"s3_client.download_file(\n",
" f\"sagemaker-example-files-prod-{region}\",\n",
" \"datasets/image/caltech-256/256_ObjectCategories/167.pyramid/167_0002.jpg\",\n",
" \"pyramid.jpg\",\n",
")\n",
"s3_client.download_file(\n",
" f\"sagemaker-example-files-prod-{region}\",\n",
" \"datasets/image/caltech-256/256_ObjectCategories/038.chimp/038_0013.jpg\",\n",
" \"chimp.jpg\",\n",
")\n",
"s3_client.download_file(\n",
" f\"sagemaker-example-files-prod-{region}\",\n",
" \"datasets/image/caltech-256/256_ObjectCategories/124.killer-whale/124_0013.jpg\",\n",
" \"killer-whale.jpg\",\n",
")\n",
"s3_client.download_file(\n",
" f\"sagemaker-example-files-prod-{region}\",\n",
" \"datasets/image/caltech-256/256_ObjectCategories/038.chimp/038_0013.jpg\",\n",
" \"chimp.jpg\",\n",
")\n",
"s3_client.download_file(\n",
" f\"sagemaker-example-files-prod-{region}\",\n",
" \"datasets/image/caltech-256/256_ObjectCategories/170.rainbow/170_0001.jpg\",\n",
" \"rainbow.jpg\",\n",
")\n",
"\n",
"for file_name in file_name_map:\n",
" s3_client.upload_file(\n",
" file_name_map[file_name], output_bucket, f\"{prefix}/{file_name_map[file_name]}\"\n",
" )"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"### Set up config objects\n",
"Now we setup some config objects required for running the Clarify job:\n",
"* **explainability_data_config**: Config object related to configurations of the input and output dataset.\n",
"* **model_config**: Config object related to a model and its endpoint to be created.\n",
" * **content_type**: Specifies the type of input expected by the model.\n",
"* **predictions_config**: Config object to extract a predicted label from the model output.\n",
" * **label_headers**: This is the list of all the classes on which the model was trained.\n",
"* **image_config**: Config object for image data type\n",
" * **model_type**: Specifies the type of CV model (IMAGE_CLASSIFICATION | OBJECT_DETECTION)\n",
" * **num_segments**: Clarify uses SKLearn's [SLIC](https://scikit-image.org/docs/dev/api/skimage.segmentation.html?highlight=slic#skimage.segmentation.slic) method for image segmentation to generate features/superpixels. num_segments specifies approximate number of segments to be generated.\n",
" * **segment_compactness**: Balances color proximity and space proximity. Higher values give more weight to space proximity, making superpixel shapes more square/cubic. We recommend exploring possible values on a log scale, e.g., 0.01, 0.1, 1, 10, 100, before refining around a chosen value.\n",
"* **shap_config**: Config object for kernel SHAP parameters\n",
" * **num_samples**: total number of feature coalitions to be tested by Kernel SHAP."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"from sagemaker import clarify\n",
"\n",
"s3_data_input_path = \"s3://{}/{}/\".format(output_bucket, prefix)\n",
"clarify_output_prefix = f\"{prefix}/cv_analysis_result\"\n",
"analysis_result_path = \"s3://{}/{}\".format(output_bucket, clarify_output_prefix)\n",
"explainability_data_config = clarify.DataConfig(\n",
" s3_data_input_path=s3_data_input_path,\n",
" s3_output_path=analysis_result_path,\n",
" dataset_type=\"application/x-image\",\n",
")\n",
"\n",
"model_config = clarify.ModelConfig(\n",
" model_name=model_name, instance_type=\"ml.m5.xlarge\", instance_count=1, content_type=\"image/jpeg\"\n",
")\n",
"\n",
"predictions_config = clarify.ModelPredictedLabelConfig(label_headers=object_categories)\n",
"\n",
"image_config = clarify.ImageConfig(\n",
" model_type=\"IMAGE_CLASSIFICATION\", num_segments=20, segment_compactness=5\n",
")\n",
"\n",
"shap_config = clarify.SHAPConfig(num_samples=500, image_config=image_config)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"### SageMaker Clarify Processor\n",
"Let's get the execution role for running SageMakerClarifyProcessor."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"import os\n",
"\n",
"account_id = os.getenv(\"AWS_ACCOUNT_ID\", \"<your-account-id>\")\n",
"sagemaker_iam_role = \"<AmazonSageMaker-ExecutionRole>\"\n",
"\n",
"# Fetch the IAM role to initialize the sagemaker processing job\n",
"try:\n",
" role = sagemaker.get_execution_role()\n",
"except ValueError as e:\n",
" print(e)\n",
" role = f\"arn:aws:iam::{account_id}:role/{sagemaker_iam_role}\"\n",
"\n",
"clarify_processor = clarify.SageMakerClarifyProcessor(\n",
" role=role, instance_count=1, instance_type=\"ml.m5.xlarge\", sagemaker_session=sess\n",
")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"Finally, we run explainability on the clarify processor."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"clarify_processor.run_explainability(\n",
" data_config=explainability_data_config,\n",
" model_config=model_config,\n",
" explainability_config=shap_config,\n",
" model_scores=predictions_config,\n",
")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Reading Results\n",
"Let's download all the result images along with the PDF report."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"%%time\n",
"output_objects = s3_client.list_objects(Bucket=output_bucket, Prefix=clarify_output_prefix)\n",
"result_images = []\n",
"\n",
"for file_obj in output_objects[\"Contents\"]:\n",
" file_name = os.path.basename(file_obj[\"Key\"])\n",
" if os.path.splitext(file_name)[1] == \".jpeg\":\n",
" result_images.append(file_name)\n",
"\n",
" print(f\"Downloading s3://{output_bucket}/{file_obj['Key']} ...\")\n",
" s3_client.download_file(output_bucket, file_obj[\"Key\"], file_name)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"Let's visualize and understand the results.\n",
"The result images shows the segmented image and the heatmap.\n",
"\n",
"* **Segments**: Highlights the image segments.\n",
"* **Shades of Blue**: Represents positive Shapley values indicating that the corresponding feature increases the overall confidence score.\n",
"* **Shades of Red**: Represents negative Shapley values indicating that the corresponding feature decreases the overall confidence score.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"from IPython.display import Image\n",
"\n",
"for img in result_images:\n",
" display(Image(img))"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Clean Up\n",
"Finally, don't forget to clean up the resources we set up and used for this demo!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%time\n",
"\n",
"# Delete the SageMaker model\n",
"model.delete_model()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Notebook CI Test Results\n",
"\n",
"This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
"\n",
"![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/responsible_ai|sm-clarify_explainability_image_classification|sm-clarify_explainability_image_classification.ipynb)\n",
"\n",
"![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/responsible_ai|sm-clarify_explainability_image_classification|sm-clarify_explainability_image_classification.ipynb)\n",
"\n",
"![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/responsible_ai|sm-clarify_explainability_image_classification|sm-clarify_explainability_image_classification.ipynb)\n",
"\n",
"![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/responsible_ai|sm-clarify_explainability_image_classification|sm-clarify_explainability_image_classification.ipynb)\n",
"\n",
"![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/responsible_ai|sm-clarify_explainability_image_classification|sm-clarify_explainability_image_classification.ipynb)\n",
"\n",
"![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/responsible_ai|sm-clarify_explainability_image_classification|sm-clarify_explainability_image_classification.ipynb)\n",
"\n",
"![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/responsible_ai|sm-clarify_explainability_image_classification|sm-clarify_explainability_image_classification.ipynb)\n",
"\n",
"![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/responsible_ai|sm-clarify_explainability_image_classification|sm-clarify_explainability_image_classification.ipynb)\n",
"\n",
"![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/responsible_ai|sm-clarify_explainability_image_classification|sm-clarify_explainability_image_classification.ipynb)\n",
"\n",
"![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/responsible_ai|sm-clarify_explainability_image_classification|sm-clarify_explainability_image_classification.ipynb)\n",
"\n",
"![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/responsible_ai|sm-clarify_explainability_image_classification|sm-clarify_explainability_image_classification.ipynb)\n",
"\n",
"![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/responsible_ai|sm-clarify_explainability_image_classification|sm-clarify_explainability_image_classification.ipynb)\n",
"\n",
"![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/responsible_ai|sm-clarify_explainability_image_classification|sm-clarify_explainability_image_classification.ipynb)\n",
"\n",
"![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/responsible_ai|sm-clarify_explainability_image_classification|sm-clarify_explainability_image_classification.ipynb)\n",
"\n",
"![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/responsible_ai|sm-clarify_explainability_image_classification|sm-clarify_explainability_image_classification.ipynb)\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (PyTorch 1.13 Python 3.9 CPU Optimized)",
"language": "python",
"name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/pytorch-1.13-cpu-py39"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.16"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
responsible_ai/sm-clarify_fairness_and_explainability_bring_your_own_container.ipynb
+1404
View File
File diff suppressed because it is too large Load Diff
responsible_ai/sm-clarify_fairness_and_explainability_with_boto3.ipynb
+1693
View File
File diff suppressed because one or more lines are too long
responsible_ai/sm-clarify_natural_language_processing_online_explainability/code/inference.py
+57
View File
@@ -0,0 +1,57 @@
from io import StringIO
import numpy as np
import os
import pandas as pd
import json
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
from typing import Any, Dict, List
def model_fn(model_dir: str) -> Dict[str, Any]:
"""
Load the model for inference
"""
model_path = os.path.join(model_dir, "model")
# Load HuggingFace tokenizer.
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
# Load HuggingFace model from disk.
model = AutoModelForSequenceClassification.from_pretrained(model_path, local_files_only=True)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
model.eval()
model_dict = {"model": model, "tokenizer": tokenizer}
return model_dict
def predict_fn(input_data: List, model: Dict) -> np.ndarray:
"""
Apply model to the incoming request
"""
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
tokenizer = model["tokenizer"]
huggingface_model = model["model"]
encoded_input = tokenizer(input_data, truncation=True, padding=True, max_length=128, return_tensors="pt")
encoded_input = {k: v.to(device) for k, v in encoded_input.items()}
with torch.no_grad():
output = huggingface_model(input_ids=encoded_input["input_ids"], attention_mask=encoded_input["attention_mask"])
res = torch.nn.Softmax(dim=1)(output.logits).detach().cpu().numpy()[:, 1]
return res
def input_fn(request_body: str, request_content_type: str) -> List[str]:
"""
Deserialize and prepare the prediction input
"""
if request_content_type == "application/json":
sentences = [json.loads(request_body)]
elif request_content_type == "text/csv":
# We have a single column with the text.
sentences = list(pd.read_csv(StringIO(request_body), header=None).values[:, 0].astype(str))
else:
sentences = request_body
return sentences
responsible_ai/sm-clarify_natural_language_processing_online_explainability/code/requirements.txt
+2
View File
@@ -0,0 +1,2 @@
transformers==4.2.1
pandas
responsible_ai/sm-clarify_natural_language_processing_online_explainability/requirements.txt
+7
View File
@@ -0,0 +1,7 @@
sagemaker
boto3
botocore
datasets
transformers
torch
captum
responsible_ai/sm-clarify_natural_language_processing_online_explainability/scripts/train.py
+122
View File
@@ -0,0 +1,122 @@
#!/usr/bin/env python3
from transformers import AutoModelForSequenceClassification, Trainer, TrainingArguments, AutoTokenizer
from sklearn.metrics import accuracy_score, precision_recall_fscore_support
from datasets import load_from_disk, load_dataset
import random
import logging
import sys
import argparse
import os
import torch
if __name__ == "__main__":
parser = argparse.ArgumentParser()
# hyperparameters sent by the client are passed as command-line arguments to the script.
parser.add_argument("--epochs", type=int, default=1)
parser.add_argument("--train_batch_size", type=int, default=32)
parser.add_argument("--eval_batch_size", type=int, default=64)
parser.add_argument("--warmup_steps", type=int, default=500)
parser.add_argument("--model_name", type=str)
parser.add_argument("--learning_rate", type=str, default=5e-5)
parser.add_argument("--train_file", type=str, default="train.csv")
parser.add_argument("--test_file", type=str, default="test.csv")
# Data, model, and output directories
parser.add_argument("--output_data_dir", type=str, default=os.environ["SM_OUTPUT_DATA_DIR"])
parser.add_argument("--model_dir", type=str, default=os.environ["SM_MODEL_DIR"])
parser.add_argument("--n_gpus", type=str, default=os.environ["SM_NUM_GPUS"])
parser.add_argument("--training_dir", type=str, default=os.environ["SM_CHANNEL_TRAIN"])
parser.add_argument("--test_dir", type=str, default=os.environ["SM_CHANNEL_TEST"])
args, _ = parser.parse_known_args()
# Set up logging
logger = logging.getLogger(__name__)
logging.basicConfig(
level=logging.getLevelName("INFO"),
handlers=[logging.StreamHandler(sys.stdout)],
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
)
# Load datasets
train_dataset_path = os.path.join(args.training_dir, args.train_file)
test_dataset_path = os.path.join(args.test_dir, args.test_file)
col_names = ['Review Text', 'Sentiment']
raw_train_dataset = load_dataset("csv", data_files=train_dataset_path, column_names=col_names)["train"]
raw_test_dataset = load_dataset("csv", data_files=test_dataset_path, column_names=col_names)["train"]
logger.info(f" loaded train_dataset length is: {len(raw_train_dataset)}")
logger.info(f" loaded test_dataset length is: {len(raw_test_dataset)}")
# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(args.model_name)
# Tokenizer helper function
def tokenize(batch):
return tokenizer(batch['Review Text'], padding=True, truncation=True)
# Tokenize the dataset
train_dataset = raw_train_dataset.map(tokenize, batched=True)
test_dataset = raw_test_dataset.map(tokenize, batched=True)
# compute metrics function for binary classification
def compute_metrics(pred):
labels = pred.label_ids
preds = pred.predictions.argmax(-1)
precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average="binary")
acc = accuracy_score(labels, preds)
return {"accuracy": acc, "f1": f1, "precision": precision, "recall": recall}
# Set format for PyTorch
train_dataset = train_dataset.rename_column("Sentiment", "labels")
train_dataset.set_format('torch', columns=['input_ids', 'attention_mask', 'labels'])
test_dataset = test_dataset.rename_column("Sentiment", "labels")
test_dataset.set_format('torch', columns=['input_ids', 'attention_mask', 'labels'])
# download model from model hub
model = AutoModelForSequenceClassification.from_pretrained(args.model_name)
# define training args
training_args = TrainingArguments(
output_dir=args.model_dir,
num_train_epochs=args.epochs,
per_device_train_batch_size=args.train_batch_size,
per_device_eval_batch_size=args.eval_batch_size,
warmup_steps=args.warmup_steps,
evaluation_strategy="epoch",
logging_dir=f"{args.output_data_dir}/logs",
learning_rate=float(args.learning_rate),
)
# create Trainer instance
trainer = Trainer(
model=model,
args=training_args,
compute_metrics=compute_metrics,
train_dataset=train_dataset,
eval_dataset=test_dataset,
tokenizer=tokenizer,
)
# train model
trainer.train()
# evaluate model
eval_result = trainer.evaluate(eval_dataset=test_dataset)
# writes eval result to file which can be accessed later in s3 ouput
with open(os.path.join(args.output_data_dir, "eval_results.txt"), "w") as writer:
print(f"***** Eval results *****")
for key, value in sorted(eval_result.items()):
writer.write(f"{key} = {value}\n")
# Saves the model to s3
trainer.save_model(args.model_dir)
responsible_ai/sm-clarify_natural_language_processing_online_explainability/sm-clarify_natural_language_processing_online_explainability.ipynb
+2265
View File
File diff suppressed because one or more lines are too long
responsible_ai/sm-clarify_natural_language_processing_online_explainability/visualization_utils.py
+82
View File
@@ -0,0 +1,82 @@
from captum.attr import visualization
import csv
import numpy as np
# This method is a wrapper around the captum that helps produce visualizations for local explanations. It will
# visualize the attributions for the tokens with red or green colors for negative and positive attributions.
def visualization_record(
attributions, # list of attributions for the tokens
text, # list of tokens
pred, # the prediction value obtained from the endpoint
delta,
true_label, # the true label from the dataset
normalize=True, # normalizes the attributions so that the max absolute value is 1. Yields stronger colors.
max_frac_to_show=0.05, # what fraction of tokens to highlight, set to 1 for all.
match_to_pred=False, # whether to limit highlights to red for negative predictions and green for positive ones.
# By enabling `match_to_pred` you show what tokens contribute to a high/low prediction not those that oppose it.
):
if normalize:
attributions = attributions / max(max(attributions), max(-attributions))
if max_frac_to_show is not None and max_frac_to_show < 1:
num_show = int(max_frac_to_show * attributions.shape[0])
sal = attributions
if pred < 0.5:
sal = -sal
if not match_to_pred:
sal = np.abs(sal)
top_idxs = np.argsort(-sal)[:num_show]
mask = np.zeros_like(attributions)
mask[top_idxs] = 1
attributions = attributions * mask
return visualization.VisualizationDataRecord(
attributions,
pred,
int(pred > 0.5),
true_label,
attributions.sum() > 0,
attributions.sum(),
text,
delta,
)
def visualize_result(result, all_labels):
if not result["explanations"]:
print(f"No Clarify explanations for the record(s)")
return
all_explanations = result["explanations"]["kernel_shap"]
all_predictions = list(csv.reader(result["predictions"]["data"].splitlines()))
labels = []
predictions = []
explanations = []
for i, expl in enumerate(all_explanations):
if expl:
labels.append(all_labels[i])
predictions.append(all_predictions[i])
explanations.append(all_explanations[i])
attributions_dataset = [
np.array([attr["attribution"][0] for attr in expl[0]["attributions"]])
for expl in explanations
]
tokens_dataset = [
np.array([attr["description"]["partial_text"] for attr in expl[0]["attributions"]])
for expl in explanations
]
# You can customize the following display settings
normalize = True
max_frac_to_show = 1
match_to_pred = False
vis = []
for attr, token, pred, label in zip(attributions_dataset, tokens_dataset, predictions, labels):
vis.append(
visualization_record(
attr, token, float(pred[0]), 0.0, label, normalize, max_frac_to_show, match_to_pred
)
)
_ = visualization.visualize_text(vis)
responsible_ai/sm-clarify_object_detection/noise_rgb.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 118 KiB

responsible_ai/sm-clarify_object_detection/repo/detection_server_batch.py
+96
View File
@@ -0,0 +1,96 @@
import argparse
import ast
import logging
import os
import json
from gluoncv import model_zoo, data
import mxnet as mx
from mxnet import nd, gluon
import numpy as np
from io import BytesIO
from timeit import default_timer as timer
def get_ctx():
"function to get machine hardware context"
try:
_ = mx.nd.array([0], ctx=mx.gpu())
ctx = mx.gpu()
except:
try:
_ = mx.nd.array([0], ctx=mx.eia())
ctx = mx.eia()
except:
ctx = mx.cpu()
return ctx
def model_fn(model_dir):
"""
Load the gluon model. Called once when hosting service starts.
:param: model_dir The directory where model files are stored.
:return: a model (in this case a Gluon network)
assumes that the parameters artifact is {model_name}.params
"""
os.environ["MXNET_CUDNN_AUTOTUNE_DEFAULT"] = "0"
ctx = get_ctx()
logging.info("Using ctx {}".format(ctx))
logging.info("Dir content {}".format(os.listdir()))
# instantiate net and reset to classes of interest
net = gluon.nn.SymbolBlock.imports(
symbol_file=[f for f in os.listdir() if f.endswith("json")][0],
input_names=["data"],
param_file=[f for f in os.listdir() if f.endswith("params")][0],
ctx=ctx,
)
return net
def transform_fn(model, input_data, content_type, accept):
start = timer()
# Expecting input_data as numpy serialized object
images_array = np.load(BytesIO(input_data), allow_pickle=True)
if images_array.ndim == 3:
images_array = np.expand_dims(images_array, axis=0)
images_nd_array_list = [mx.nd.array(image_array) for image_array in images_array]
logging.info(f"Decoded images in {(timer()-start):.4f} seconds")
start = timer()
tensors, origs = data.transforms.presets.yolo.transform_test(images_nd_array_list)
if type(tensors) != list:
tensors = [tensors]
origs = [origs]
logging.info(f"Transformed images in {(timer()-start):.4f} seconds")
ctx = get_ctx()
logging.info("Using ctx {}".format(ctx))
start = timer()
results = []
# forward pass and display
for i in range(len(tensors)):
h = origs[i].shape[0]
w = origs[i].shape[1]
box_ids, scores, bboxes = model(tensors[i].as_in_context(ctx))
result = nd.concat(box_ids, scores, bboxes, dim=2).asnumpy()
# Filter out -1 value rows
result = np.array([res[res[:, 0] != -1] for res in result])
result[:, :, 2] /= w
result[:, :, 3] /= h
result[:, :, 4] /= w
result[:, :, 5] /= h
result[:, :, 2:] = np.where(result[:, :, 2:] > 1, 1, result[:, :, 2:])
result[:, :, 2:] = np.where(result[:, :, 2:] < 0, 0, result[:, :, 2:])
results.append(result[0].tolist())
logging.info(f"Ran model inference on images in {(timer()-start):.4f} seconds")
logging.info(f"Returning predictions: {results}")
return results # return a single tensor
responsible_ai/sm-clarify_object_detection/repo/requirements.txt
+1
View File
@@ -0,0 +1 @@
gluoncv==0.10.4.post4
responsible_ai/sm-clarify_object_detection/requirements.txt
+4
View File
@@ -0,0 +1,4 @@
boto3
gluoncv==0.10.4.post4
mxnet==1.8.0.post0
sagemaker
responsible_ai/sm-clarify_object_detection/sm-clarify_object_detection.ipynb
+1824
View File
File diff suppressed because it is too large Load Diff
responsible_ai/sm-clarify_online_explainability_mme_xgboost/requirements.txt
+4
View File
@@ -0,0 +1,4 @@
sagemaker
boto3
shap
matplotlib
responsible_ai/sm-clarify_online_explainability_mme_xgboost/sm-clarify_online_explainability_mme_xgboost.ipynb
+1656
View File
File diff suppressed because one or more lines are too long
responsible_ai/sm-clarify_online_explainability_mme_xgboost/visualization_utils.py
+67
View File
@@ -0,0 +1,67 @@
import io
from collections import OrderedDict
import numpy as np
import pandas as pd
import shap
def force_plot(expected_value, shap_values, feature_data, feature_headers):
"""
Visualize the given SHAP values with an additive force layout.
For more information: https://shap.readthedocs.io/en/latest/example_notebooks/tabular_examples/tree_based_models/Force%20Plot%20Colors.html
"""
force_plot_display = shap.plots.force(
base_value=expected_value,
shap_values=shap_values,
features=feature_data,
feature_names=feature_headers,
matplotlib=True,
)
def display_plots(explanations, expected_value, request_records, predictions):
"""
Display the Model Explainability plots
"""
per_request_shap_values = OrderedDict()
feature_headers = []
for i, record_output in enumerate(explanations):
per_record_shap_values = []
if record_output is not None:
feature_headers = []
for feature_attribution in record_output:
per_record_shap_values.append(
feature_attribution["attributions"][0]["attribution"][0]
)
feature_headers.append(feature_attribution["feature_header"])
per_request_shap_values[i] = per_record_shap_values
for record_index, shap_values in per_request_shap_values.items():
print(
f"Visualize the SHAP values for Record number {record_index + 1} with Model Prediction: {predictions[0][record_index]}"
)
force_plot(
expected_value,
np.array(shap_values),
request_records.iloc[record_index],
feature_headers,
)
def visualize_result(result, request_records, expected_value):
"""
Visualize the output from the endpoint.
"""
predictions = pd.read_csv(io.StringIO(result["predictions"]["data"]), header=None)
predictions = predictions.values.tolist()
print(f"Model Inference output: ")
for i, model_output in enumerate(predictions):
print(f"Record: {i + 1}\tModel Prediction: {model_output[0]}")
if "kernel_shap" in result["explanations"]:
explanations = result["explanations"]["kernel_shap"]
display_plots(explanations, expected_value, request_records, predictions)
else:
print(f"No Clarify explanations for the record(s)")
responsible_ai/sm-clarify_online_explainability_tabular/requirements.txt
+5
View File
@@ -0,0 +1,5 @@
sagemaker
boto3
botocore
shap
matplotlib
responsible_ai/sm-clarify_online_explainability_tabular/sm-clarify_online_explainability_tabular.ipynb
+2312
View File
File diff suppressed because one or more lines are too long
responsible_ai/sm-clarify_online_explainability_tabular/visualization_utils.py
+67
View File
@@ -0,0 +1,67 @@
import io
from collections import OrderedDict
import numpy as np
import pandas as pd
import shap
def force_plot(expected_value, shap_values, feature_data, feature_headers):
"""
Visualize the given SHAP values with an additive force layout.
For more information: https://shap.readthedocs.io/en/latest/example_notebooks/tabular_examples/tree_based_models/Force%20Plot%20Colors.html
"""
force_plot_display = shap.plots.force(
base_value=expected_value,
shap_values=shap_values,
features=feature_data,
feature_names=feature_headers,
matplotlib=True,
)
def display_plots(explanations, expected_value, request_records, predictions):
"""
Display the Model Explainability plots
"""
per_request_shap_values = OrderedDict()
feature_headers = []
for i, record_output in enumerate(explanations):
per_record_shap_values = []
if record_output is not None:
feature_headers = []
for feature_attribution in record_output:
per_record_shap_values.append(
feature_attribution["attributions"][0]["attribution"][0]
)
feature_headers.append(feature_attribution["feature_header"])
per_request_shap_values[i] = per_record_shap_values
for record_index, shap_values in per_request_shap_values.items():
print(
f"Visualize the SHAP values for Record number {record_index + 1} with Model Prediction: {predictions[record_index][0]}"
)
force_plot(
expected_value,
np.array(shap_values),
request_records.iloc[record_index],
feature_headers,
)
def visualize_result(result, request_records, expected_value):
"""
Visualize the output from the endpoint.
"""
predictions = pd.read_csv(io.StringIO(result["predictions"]["data"]), header=None)
predictions = predictions.values.tolist()
print(f"Model Inference output: ")
for i, model_output in enumerate(predictions):
print(f"Record: {i + 1}\tModel Prediction: {model_output[0]}")
if "kernel_shap" in result["explanations"]:
explanations = result["explanations"]["kernel_shap"]
display_plots(explanations, expected_value, request_records, predictions)
else:
print(f"No Clarify explanations for the record(s)")
responsible_ai/sm-clarify_text_explainability_text_sentiment_analysis/code/__init__.py
View File
responsible_ai/sm-clarify_text_explainability_text_sentiment_analysis/code/inference.py
+57
View File
@@ -0,0 +1,57 @@
from io import StringIO
import numpy as np
import os
import pandas as pd
import json
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
from typing import Any, Dict, List
def model_fn(model_dir: str) -> Dict[str, Any]:
"""
Load the model for inference
"""
model_path = os.path.join(model_dir, "model")
# Load HuggingFace tokenizer.
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
# Load HuggingFace model from disk.
model = AutoModelForSequenceClassification.from_pretrained(model_path, local_files_only=True)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
model.eval()
model_dict = {"model": model, "tokenizer": tokenizer}
return model_dict
def predict_fn(input_data: List, model: Dict) -> np.ndarray:
"""
Apply model to the incoming request
"""
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
tokenizer = model["tokenizer"]
huggingface_model = model["model"]
encoded_input = tokenizer(input_data, truncation=True, padding=True, max_length=128, return_tensors="pt")
encoded_input = {k: v.to(device) for k, v in encoded_input.items()}
with torch.no_grad():
output = huggingface_model(input_ids=encoded_input["input_ids"], attention_mask=encoded_input["attention_mask"])
res = torch.nn.Softmax(dim=1)(output.logits).detach().cpu().numpy()[:, 1]
return res
def input_fn(request_body: str, request_content_type: str) -> List[str]:
"""
Deserialize and prepare the prediction input
"""
if request_content_type == "application/json":
sentences = [json.loads(request_body)]
elif request_content_type == "text/csv":
# We have a single column with the text.
sentences = list(pd.read_csv(StringIO(request_body), header=None).values[:, 0].astype(str))
else:
sentences = request_body
return sentences
responsible_ai/sm-clarify_text_explainability_text_sentiment_analysis/code/requirements.txt
+3
View File
@@ -0,0 +1,3 @@
transformers==4.40.0
torch>=2.6.0
pandas
responsible_ai/sm-clarify_text_explainability_text_sentiment_analysis/scripts/train.py
+119
View File
@@ -0,0 +1,119 @@
#!/usr/bin/env python3
from transformers import AutoModelForSequenceClassification, Trainer, TrainingArguments, AutoTokenizer
from sklearn.metrics import accuracy_score, precision_recall_fscore_support
from datasets import load_from_disk, load_dataset
import random
import logging
import sys
import argparse
import os
import torch
if __name__ == "__main__":
parser = argparse.ArgumentParser()
# hyperparameters sent by the client are passed as command-line arguments to the script.
parser.add_argument("--epochs", type=int, default=1)
parser.add_argument("--train_batch_size", type=int, default=32)
parser.add_argument("--eval_batch_size", type=int, default=64)
parser.add_argument("--warmup_steps", type=int, default=500)
parser.add_argument("--model_name", type=str)
parser.add_argument("--learning_rate", type=str, default=5e-5)
parser.add_argument("--train_file", type=str, default="train.csv")
parser.add_argument("--test_file", type=str, default="test.csv")
# Data, model, and output directories
parser.add_argument("--output_data_dir", type=str, default=os.environ["SM_OUTPUT_DATA_DIR"])
parser.add_argument("--model_dir", type=str, default=os.environ["SM_MODEL_DIR"])
parser.add_argument("--n_gpus", type=str, default=os.environ["SM_NUM_GPUS"])
parser.add_argument("--training_dir", type=str, default=os.environ["SM_CHANNEL_TRAIN"])
parser.add_argument("--test_dir", type=str, default=os.environ["SM_CHANNEL_TEST"])
args, _ = parser.parse_known_args()
# Set up logging
logger = logging.getLogger(__name__)
logging.basicConfig(
level=logging.getLevelName("INFO"),
handlers=[logging.StreamHandler(sys.stdout)],
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
)
# Load datasets
train_dataset_path = os.path.join(args.training_dir, args.train_file)
test_dataset_path = os.path.join(args.test_dir, args.test_file)
col_names = ['Review Text', 'Sentiment']
raw_train_dataset = load_dataset("csv", data_files=train_dataset_path, column_names=col_names)["train"]
raw_test_dataset = load_dataset("csv", data_files=test_dataset_path, column_names=col_names)["train"]
logger.info(f" loaded train_dataset length is: {len(raw_train_dataset)}")
logger.info(f" loaded test_dataset length is: {len(raw_test_dataset)}")
# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(args.model_name)
# Tokenizer helper function
def tokenize(batch):
return tokenizer(batch['Review Text'], padding=True, truncation=True)
# Tokenize the dataset
train_dataset = raw_train_dataset.map(tokenize, batched=True)
test_dataset = raw_test_dataset.map(tokenize, batched=True)
# compute metrics function for binary classification
def compute_metrics(pred):
labels = pred.label_ids
preds = pred.predictions.argmax(-1)
precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average="binary")
acc = accuracy_score(labels, preds)
return {"accuracy": acc, "f1": f1, "precision": precision, "recall": recall}
# Set format for PyTorch
train_dataset = train_dataset.rename_column("Sentiment", "labels")
train_dataset.set_format('torch', columns=['input_ids', 'attention_mask', 'labels'])
test_dataset = test_dataset.rename_column("Sentiment", "labels")
test_dataset.set_format('torch', columns=['input_ids', 'attention_mask', 'labels'])
# download model from model hub
model = AutoModelForSequenceClassification.from_pretrained(args.model_name)
# define training args
training_args = TrainingArguments(
output_dir=args.model_dir,
num_train_epochs=args.epochs,
per_device_train_batch_size=args.train_batch_size,
per_device_eval_batch_size=args.eval_batch_size,
warmup_steps=args.warmup_steps,
evaluation_strategy="epoch",
logging_dir=f"{args.output_data_dir}/logs",
learning_rate=float(args.learning_rate),
)
# create Trainer instance
trainer = Trainer(
model=model,
args=training_args,
compute_metrics=compute_metrics,
train_dataset=train_dataset,
eval_dataset=test_dataset,
tokenizer=tokenizer,
)
# train model
trainer.train()
# evaluate model
eval_result = trainer.evaluate(eval_dataset=test_dataset)
# writes eval result to file which can be accessed later in s3 ouput
with open(os.path.join(args.output_data_dir, "eval_results.txt"), "w") as writer:
print(f"***** Eval results *****")
for key, value in sorted(eval_result.items()):
writer.write(f"{key} = {value}\n")
# Saves the model to s3
trainer.save_model(args.model_dir)
responsible_ai/sm-clarify_text_explainability_text_sentiment_analysis/sm-clarify_text_explainability_text_sentiment_analysis.ipynb
+1511
View File
File diff suppressed because one or more lines are too long
responsible_ai/sm-clarify_time_series_bring_your_own_model/model/code/inference.py
+62
View File
@@ -0,0 +1,62 @@
from darts.models import LinearRegressionModel
from darts.timeseries import TimeSeries
from typing import Optional
import pandas as pd
from datetime import datetime, timedelta
import json
import os
import logging
def model_fn(model_dir, context):
# model_path = os.path.join(model_dir, "model.pth")
model = None
with open(os.path.join(model_dir, 'model.pth'), 'rb') as f:
model = LinearRegressionModel.load(f)
return model
def input_fn(request_body, request_content_type, context):
# return request_body
record_dict = json.loads(request_body)
print(record_dict)
record_list = record_dict["instances"]
# record = record_list[0]
return record_list
def predict_fn(input_object, model, context):
# return model.predict(10)
start = input_object["start"]
target = input_object["target"]
dynamic_feat = input_object["dynamic_feat"]
start_datetime = datetime.strptime(start, '%Y-%m-%d %H:%M:%S')
datetimes = [start_datetime + timedelta(minutes=i*10) for i in range(len(target))]
target_df = pd.DataFrame({'target': target}, index=datetimes)
target_ts = TimeSeries.from_dataframe(target_df, value_cols=['target'])
past_cov_df = pd.DataFrame({f"feature_{i+1}": feats[:len(target)] for i, feats in enumerate(dynamic_feat)}, index=datetimes)
past_cov_ts = TimeSeries.from_dataframe(past_cov_df)
start_future_datetime = datetimes[-1] + timedelta(minutes=10)
num_future_steps = len(dynamic_feat[0]) - len(target)
future_datetimes = [start_future_datetime + timedelta(minutes=i*10) for i in range(num_future_steps)]
# Create the DataFrame for future_covariates
future_cov_df = pd.DataFrame({
f"feature_{i+1}": feats[len(target):] for i, feats in enumerate(dynamic_feat)
}, index=future_datetimes)
future_cov_ts = TimeSeries.from_dataframe(future_cov_df)
predictions = model.predict(10, series=target_ts, past_covariates=past_cov_ts, future_covariates=future_cov_ts)
return predictions
def output_fn(prediction, content_type, context):
predictions_list = [item for sublist in prediction.values() for item in sublist]
logging.warning('----------------')
# Create a dictionary in the desired format
pred_dict = {'predictions': {'mean': predictions_list}}
logging.warning(f'Predictions: {pred_dict}')
# Convert the dictionary to a JSON string
pred_json = json.dumps(pred_dict)
return pred_json
responsible_ai/sm-clarify_time_series_bring_your_own_model/model/code/requirements.txt
+9
View File
@@ -0,0 +1,9 @@
awscli
Cython
fastapi
numpy
pandas
pytest
scikit-learn
torchvision
darts==0.26.0
responsible_ai/sm-clarify_time_series_bring_your_own_model/model/model.pth
BIN
View File
Binary file not shown.
responsible_ai/sm-clarify_time_series_bring_your_own_model/sm-clarify_time_series_bring_your_own_model.ipynb
+1465
View File
File diff suppressed because it is too large Load Diff
responsible_ai/sm-clarify_time_series_bring_your_own_model/time_series_byom_mock_data.json
+128
View File
@@ -0,0 +1,128 @@
[
{
"item_id": "seattle",
"timestamp": "2020-01-01 16:20:00",
"p_mbar": 1008.89,
"rain_mm": 0.23,
"T_degC": 0.71
},
{
"item_id": "seattle",
"timestamp": "2020-01-01 16:30:00",
"p_mbar": 1008.76,
"rain_mm": 0.21,
"T_degC": 0.75
},
{
"item_id": "seattle",
"timestamp": "2020-01-01 16:40:00",
"p_mbar": 1008.66,
"rain_mm": 0.19,
"T_degC": 0.73
},
{
"item_id": "seattle",
"timestamp": "2020-01-01 16:50:00",
"p_mbar": null,
"rain_mm": 0.16,
"T_degC": 0.37
},
{
"item_id": "seattle",
"timestamp": "2020-01-01 17:00:00",
"p_mbar": null,
"rain_mm": 0.13,
"T_degC": 0.33
},
{
"item_id": "seattle",
"timestamp": "2020-01-01 17:10:00",
"p_mbar": null,
"rain_mm": 0.08,
"T_degC": 0.34
},
{
"item_id": "seattle",
"timestamp": "2020-01-01 17:20:00",
"p_mbar": null,
"rain_mm": 0.0,
"T_degC": 0.19
},
{
"item_id": "seattle",
"timestamp": "2020-01-01 17:30:00",
"p_mbar": null,
"rain_mm": 0.0,
"T_degC": 0.03
},
{
"item_id": "seattle",
"timestamp": "2020-01-01 17:40:00",
"p_mbar": null,
"rain_mm": 0.0,
"T_degC": 0.11
},
{
"item_id": "sanjose",
"timestamp": "2020-01-01 16:20:00",
"p_mbar": 1005.1,
"rain_mm": 0.0,
"T_degC": 2.48
},
{
"item_id": "sanjose",
"timestamp": "2020-01-01 16:30:00",
"p_mbar": 1005.1,
"rain_mm": 0.0,
"T_degC": 2.41
},
{
"item_id": "sanjose",
"timestamp": "2020-01-01 16:40:00",
"p_mbar": 977.92,
"rain_mm": 0.0,
"T_degC": 2.37
},
{
"item_id": "sanjose",
"timestamp": "2020-01-01 16:50:00",
"p_mbar": null,
"rain_mm": 0.0,
"T_degC": 2.40
},
{
"item_id": "sanjose",
"timestamp": "2020-01-01 17:00:00",
"p_mbar": null,
"rain_mm": 0.0,
"T_degC": 2.41
},
{
"item_id": "sanjose",
"timestamp": "2020-01-01 17:10:00",
"p_mbar": null,
"rain_mm": 0.0,
"T_degC": 2.45
},
{
"item_id": "sanjose",
"timestamp": "2020-01-01 17:20:00",
"p_mbar": null,
"rain_mm": 0.0,
"T_degC": 2.40
},
{
"item_id": "sanjose",
"timestamp": "2020-01-01 17:30:00",
"p_mbar": null,
"rain_mm": 0.0,
"T_degC": 2.38
},
{
"item_id": "sanjose",
"timestamp": "2020-01-01 17:40:00",
"p_mbar": null,
"rain_mm": 0.0,
"T_degC": 2.34
}
]
responsible_ai/sm-model_governance_model_card/example_metrics/clarify_bias.json
+49
View File
@@ -0,0 +1,49 @@
{
"post_training_bias_metrics": {
"facets": {
"Gender": [
{
"metrics": [
{
"description": "Accuracy Difference (AD)",
"name": "AD",
"value": -0.11415724421435
}
],
"value_or_threshold": "0"
},
{
"metrics": [
{
"name": "AD",
"description": "Accuracy Difference (AD)",
"value": -0.1141572442143538
}
],
"value_or_threshold": "1"
}
]
},
"label": "Target",
"label_value_or_threshold": "1"
},
"pre_training_bias_metrics": {
"facets": {
"Gender": [
{
"metrics": [
{
"description": "Conditional Demographic Disparity in Labels (CDDL)",
"name": "CDDL",
"value": 0.21491590864936
}
],
"value_or_threshold": "0"
}
]
},
"label": "Target",
"label_value_or_threshold": "1"
},
"version": "1.0"
}
responsible_ai/sm-model_governance_model_card/example_metrics/clarify_explanability.json
+102
View File
@@ -0,0 +1,102 @@
{
"version": "1.0",
"explanations": {
"kernel_shap": {
"label0": {
"global_shap_values": {
"Marital Status": 0.028630413807559794,
"Occupation": 0.02663187314914846,
"Relationship": 0.036107264188470226,
"Ethnic group": 0.01974161417121363,
"Capital Gain": 0.033577342651290996,
"Capital Loss": 0.019363293576206416,
"Hours per week": 0.021494553921674837,
"Country": 0.04632862084382646
},
"expected_value": 0.0006380207487381995,
"global_top_shap_text": {
"[feature_name_1]": {
"charming": 0.04089221769072609,
"magnificent": 0.03013490694249078
},
"[feature_name_2]": {
"charming": 0.04089221769072609,
"magnificent": 0.03013490694249078
}
}
},
"label1": {
"global_shap_values": {
"Age": 0.036556198315954205,
"Workclass": 0.01819397618827644,
"fnlwgt": 0.02137033320407158,
"Education": 0.018644355523796074,
"Education-Num": 0.037243930827630646
},
"expected_value": 0.0006380207487381995,
"global_top_shap_text": {
"[feature_name_1]": {
"charming": 0.04089221769072609,
"magnificent": 0.03013490694249078
},
"[feature_name_2]": {
"charming": 0.04089221769072609,
"magnificent": 0.03013490694249078
}
}
}
},
"pdp": [
{
"feature_name": "Age",
"data_type": "numerical",
"data_distribution": [
0.05,
0.165,
0.15,
0.095 ],
"feature_values": [
20.866666666666667,
24.6,
28.333366666666667,
32.06666666666666 ],
"model_predictions": [
[
0.7029578701406717,
0.6787781745567918,
0.7008252082392573,
0.733054383918643
]
],
"label_headers": [
"label0"
]
},
{
"feature_name": "Animal",
"data_type": "categorical",
"data_distribution": [
0.1,
0.11,
0.245,
0.06 ],
"feature_values": [
"Cat",
"Dog",
"Horse",
"Bird" ],
"model_predictions": [
[
0.8154899527132511,
0.787026209384203,
0.7409455917775631,
0.7098744075372815
]
],
"label_headers": [
"label0"
]
}
]
}
}
responsible_ai/sm-model_governance_model_card/example_metrics/generic_modelcard_evaluation.json
+34
View File
@@ -0,0 +1,34 @@
{
"metric_groups": [
{
"metric_data": [
{
"name": "AD",
"type": "number",
"value": -0.11415724421435
}
],
"name": "post_training_bias_metrics - label Target = 1 and facet Gender=0"
},
{
"metric_data": [
{
"name": "AD",
"type": "number",
"value": -0.1141572442143538
}
],
"name": "post_training_bias_metrics - label Target = 1 and facet Gender=1"
},
{
"metric_data": [
{
"name": "CDDL",
"type": "number",
"value": 0.21491590864936
}
],
"name": "pre_training_bias_metrics - label Target = 1 and facet Gender=0"
}
]
}
responsible_ai/sm-model_governance_model_card/example_metrics/model_monitor_model_quality_binary_classification.json
+116
View File
@@ -0,0 +1,116 @@
{
"binary_classification_metrics": {
"accuracy": {
"standard_deviation": "NaN",
"value": 0.5
},
"accuracy_best_constant_classifier": {
"standard_deviation": 0.2,
"value": 0.25
},
"auc": {
"standard_deviation": 0.1,
"value": 1
},
"confusion_matrix": {
"0": {
"0": 1,
"1": 2
},
"1": {
"0": 0,
"1": 1
}
},
"f0_5": {
"standard_deviation": "NaN",
"value": 0.38461538461538
},
"f0_5_best_constant_classifier": {
"standard_deviation": "NaN",
"value": 0.29411764705882
},
"f1": {
"standard_deviation": "NaN",
"value": 0.5
},
"f1_best_constant_classifier": {
"standard_deviation": "NaN",
"value": 0.4
},
"f2": {
"standard_deviation": "NaN",
"value": 0.71428571428571
},
"f2_best_constant_classifier": {
"standard_deviation": "NaN",
"value": 0.625
},
"false_negative_rate": {
"standard_deviation": "NaN",
"value": 0
},
"false_positive_rate": {
"standard_deviation": "NaN",
"value": 0.66666666666667
},
"precision": {
"standard_deviation": "NaN",
"value": 0.33333333333333
},
"precision_best_constant_classifier": {
"standard_deviation": "NaN",
"value": 0.25
},
"precision_recall_curve": {
"precisions": [
1,
1,
1,
1,
1
],
"recalls": [
0,
0.25,
0.5,
0.75,
1
]
},
"recall": {
"standard_deviation": "NaN",
"value": 1
},
"recall_best_constant_classifier": {
"standard_deviation": "NaN",
"value": 1
},
"receiver_operating_characteristic_curve": {
"false_positive_rates": [
0,
0,
0,
0,
0,
1
],
"true_positive_rates": [
0,
0.25,
0.5,
0.75,
1,
1
]
},
"true_negative_rate": {
"standard_deviation": "NaN",
"value": 0.33333333333333
},
"true_positive_rate": {
"standard_deviation": "NaN",
"value": 1
}
}
}
responsible_ai/sm-model_governance_model_card/sm-model_governance_model_card.ipynb
+677
View File
@@ -0,0 +1,677 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Amazon SageMaker Model Governance - Model Cards\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
"\n",
"![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/responsible_ai|sm-model_governance_model_card|sm-model_governance_model_card.ipynb)\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook walks you through the features of Amazon SageMaker Model Cards. For more information, see [Model Cards](https://docs.aws.amazon.com/sagemaker/latest/dg/model-cards.html) in the _Amazon SageMaker Developer Guide_.\n",
"\n",
"Amazon SageMaker Model Cards give you the ability to create a centralized, customizable fact-sheet to document critical details about your machine learning (ML) models. Use model cards to keep a record of model information, such as intended uses, risk ratings, training details, evaluation metrics, and more for streamlined governance and reporting. \n",
"\n",
"In this example, you create a binary classification model along with a model card to document model details along the way. Learn how to create, read, update, delete, and export model cards using the Amazon SageMaker Python SDK.\n",
"\n",
"---\n",
"## Contents\n",
"\n",
"1. [Setup](#Setup)\n",
"1. [Prepare a Binary Classification Model](#Model)\n",
"1. [Create Model Card](#ModelCard)\n",
"1. [Update Model Card](#Update)\n",
"1. [Load Model Card](#Load)\n",
"1. [List Model Card History](#ListHistory)\n",
"1. [Export Model Card](#Export)\n",
"1. [Cleanup](#Cleanup)\n",
"\n",
"---\n",
"## Setup\n",
"To begin, you must specify the following information:\n",
"- The IAM role ARN used to give SageMaker training and hosting access to your data. The following example uses the SageMaker execution role.\n",
"- The SageMaker session used to manage interactions with Amazon SageMaker Model Card API methods.\n",
"- The S3 URI (`bucket` and `prefix`) where you want to store training artifacts, models, and any exported model card PDFs. This S3 bucket should be in the same Region as your Notebook Instance, training, and hosting configurations. The following example uses the default SageMaker S3 bucket and creates a default SageMaker S3 bucket if one does not already exist.\n",
"- The S3 session used to manage interactions with Amazon S3 storage."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"! pip install --upgrade 'sagemaker<3.0'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import boto3\n",
"from sagemaker.session import Session\n",
"from sagemaker import get_execution_role\n",
"\n",
"role = get_execution_role()\n",
"\n",
"sagemaker_session = Session()\n",
"\n",
"bucket = sagemaker_session.default_bucket()\n",
"prefix = \"model-card-sample-notebook\"\n",
"default_bucket_prefix = sagemaker_session.default_bucket_prefix\n",
"\n",
"# If a default bucket prefix is specified, append it to the s3 path\n",
"if default_bucket_prefix:\n",
" prefix = f\"{default_bucket_prefix}/{prefix}\"\n",
"\n",
"region = sagemaker_session.boto_region_name\n",
"s3 = boto3.client(\"s3\", region_name=region)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, import the necessary Python libraries."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import io\n",
"import os\n",
"import numpy as np\n",
"from six.moves.urllib.parse import urlparse\n",
"from pprint import pprint\n",
"import boto3\n",
"import sagemaker\n",
"from sagemaker.image_uris import retrieve\n",
"import sagemaker.amazon.common as smac\n",
"from sagemaker.model_card import (\n",
" ModelCard,\n",
" ModelOverview,\n",
" ObjectiveFunction,\n",
" Function,\n",
" TrainingDetails,\n",
" IntendedUses,\n",
" BusinessDetails,\n",
" EvaluationJob,\n",
" AdditionalInformation,\n",
" Metric,\n",
" MetricGroup,\n",
" ModelCardStatusEnum,\n",
" ObjectiveFunctionEnum,\n",
" FacetEnum,\n",
" RiskRatingEnum,\n",
" MetricTypeEnum,\n",
" EvaluationMetricTypeEnum,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## Prepare a Model<a name=\"Model\"></a>\n",
"The following code creates an example binary classification model trained on a synthetic dataset. The target variable (0 or 1) is the second variable in the tuple.\n",
"\n",
"### 1. Prepare the training data\n",
"The code will upload example data to your S3 bucket."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# synthetic data\n",
"raw_data = (\n",
" (0.5, 0),\n",
" (0.75, 0),\n",
" (1.0, 0),\n",
" (1.25, 0),\n",
" (1.50, 0),\n",
" (1.75, 0),\n",
" (2.0, 0),\n",
" (2.25, 1),\n",
" (2.5, 0),\n",
" (2.75, 1),\n",
" (3.0, 0),\n",
" (3.25, 1),\n",
" (3.5, 0),\n",
" (4.0, 1),\n",
" (4.25, 1),\n",
" (4.5, 1),\n",
" (4.75, 1),\n",
" (5.0, 1),\n",
" (5.5, 1),\n",
")\n",
"training_data = np.array(raw_data).astype(\"float32\")\n",
"labels = training_data[:, 1]\n",
"\n",
"# upload data to S3 bucket\n",
"buf = io.BytesIO()\n",
"smac.write_numpy_to_dense_tensor(buf, training_data, labels)\n",
"buf.seek(0)\n",
"boto3.resource(\"s3\").Bucket(bucket).Object(os.path.join(prefix, \"train\")).upload_fileobj(buf)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2. Train a model\n",
"Train a binary classification model with the training data from the previous step."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"s3_train_data = f\"s3://{bucket}/{prefix}/train\"\n",
"output_location = f\"s3://{bucket}/{prefix}/output\"\n",
"container = retrieve(\"linear-learner\", sagemaker_session.boto_session.region_name)\n",
"estimator = sagemaker.estimator.Estimator(\n",
" container,\n",
" role=role,\n",
" instance_count=1,\n",
" instance_type=\"ml.m4.xlarge\",\n",
" output_path=output_location,\n",
" sagemaker_session=sagemaker_session,\n",
")\n",
"estimator.set_hyperparameters(feature_dim=2, mini_batch_size=10, predictor_type=\"binary_classifier\")\n",
"estimator.fit({\"train\": s3_train_data})\n",
"print(f\"Training job name: {estimator.latest_training_job.name}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 3. Create a model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model_name = \"model-card-test-model\"\n",
"model = estimator.create_model(name=model_name)\n",
"container_def = model.prepare_container_def()\n",
"sagemaker_session.create_model(model_name, role, container_def)\n",
"print(f\"Model name: {model_name}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## Create Model Card<a name=\"ModelCard\"></a>\n",
"Document your binary classification model details in an Amazon SageMaker Model Card using the SageMaker Python SDK.\n",
"\n",
"### 1. Auto-collect model details\n",
"Automatically collect basic model information like model ID, training environment, and the model output S3 URI. Add additional model information such as a description, problem type, algorithm type, model creator, and model owner."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model_overview = ModelOverview.from_model_name(\n",
" model_name=model_name,\n",
" sagemaker_session=sagemaker_session,\n",
" model_description=\"This is an example binary classification model used for a Python SDK demo of Amazon SageMaker Model Cards.\",\n",
" problem_type=\"Binary Classification\",\n",
" algorithm_type=\"Logistic Regression\",\n",
" model_creator=\"DEMO-ModelCard\",\n",
" model_owner=\"DEMO-ModelCard\",\n",
")\n",
"print(f\"Model id: {model_overview.model_id}\")\n",
"print(f\"Model training images: {model_overview.inference_environment.container_image}\")\n",
"print(f\"Model: {model_overview.model_artifact}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2. Auto-collect training details\n",
"Automatically collect basic training information like training ID, training environment, and training metrics. Add additional training information such as objective function details and training observations."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"objective_function = ObjectiveFunction(\n",
" function=Function(\n",
" function=ObjectiveFunctionEnum.MINIMIZE,\n",
" facet=FacetEnum.LOSS,\n",
" ),\n",
" notes=\"This is an example objective function.\",\n",
")\n",
"training_details = TrainingDetails.from_model_overview(\n",
" model_overview=model_overview,\n",
" sagemaker_session=sagemaker_session,\n",
" objective_function=objective_function,\n",
" training_observations=\"Add model training observations here.\",\n",
")\n",
"print(f\"Training job id: {training_details.training_job_details.training_arn}\")\n",
"print(\n",
" f\"Training image: {training_details.training_job_details.training_environment.container_image}\"\n",
")\n",
"print(\"Training Metrics: \")\n",
"pprint(\n",
" [\n",
" {\"name\": i.name, \"value\": i.value}\n",
" for i in training_details.training_job_details.training_metrics\n",
" ]\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 3. Collect evaluation details\n",
"Add evaluation observations, datasets, and metrics."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"manual_metric_group = MetricGroup(\n",
" name=\"binary classification metrics\",\n",
" metric_data=[Metric(name=\"accuracy\", type=MetricTypeEnum.NUMBER, value=0.5)],\n",
")\n",
"example_evaluation_job = EvaluationJob(\n",
" name=\"Example evaluation job\",\n",
" evaluation_observation=\"Evaluation observations.\",\n",
" datasets=[\"s3://path/to/evaluation/data\"],\n",
" metric_groups=[manual_metric_group],\n",
")\n",
"evaluation_details = [example_evaluation_job]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### (Optional) 3.1 Parse metrics from existing evaluation report\n",
"If you have existing evaluation reports generated by [SageMaker Clarify](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-processing-job-run.html) or [SageMaker Model Monitor](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-model-quality.html), upload them to S3 and provide an S3 URI to automatically parse evaluation metrics. To add your own generic model card evaluation report, provide a report in the [evaluation results JSON format](https://docs.aws.amazon.com/sagemaker/latest/dg/model-cards-json-schema.html). See the example JSON files in the `./example_metrics` folder for reference.\n",
"##### Collect metrics from a JSON format evaluation report"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"report_type = \"clarify_bias.json\"\n",
"example_evaluation_job.add_metric_group_from_json(\n",
" f\"example_metrics/{report_type}\", EvaluationMetricTypeEnum.CLARIFY_BIAS\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### Collect metrics from S3"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# upload metric file to s3\n",
"with open(f\"example_metrics/{report_type}\", \"rb\") as metrics:\n",
" s3.upload_fileobj(\n",
" metrics,\n",
" Bucket=bucket,\n",
" Key=f\"{prefix}/{report_type}\",\n",
" ExtraArgs={\"ContentType\": \"application/json\"},\n",
" )\n",
"\n",
"metric_s3_url = f\"s3://{bucket}/{prefix}/{report_type}\"\n",
"example_evaluation_job.add_metric_group_from_s3(\n",
" session=sagemaker_session.boto_session,\n",
" s3_url=metric_s3_url,\n",
" metric_type=EvaluationMetricTypeEnum.CLARIFY_BIAS,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 4. Collect additional details\n",
"Add the intended uses of your model and business details and any additional information that you want to include in your model card. For more information on intended uses and business details, see [Model Cards](https://docs.aws.amazon.com/sagemaker/latest/dg/model-cards.html) in the `Amazon SageMaker Developer Guide`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"intended_uses = IntendedUses(\n",
" purpose_of_model=\"Test model card.\",\n",
" intended_uses=\"Not used except this test.\",\n",
" factors_affecting_model_efficiency=\"No.\",\n",
" risk_rating=RiskRatingEnum.LOW,\n",
" explanations_for_risk_rating=\"Just an example.\",\n",
")\n",
"business_details = BusinessDetails(\n",
" business_problem=\"The business problem that your model is used to solve.\",\n",
" business_stakeholders=\"The stakeholders who have the interest in the business that your model is used for.\",\n",
" line_of_business=\"Services that the business is offering.\",\n",
")\n",
"additional_information = AdditionalInformation(\n",
" ethical_considerations=\"Your model ethical consideration.\",\n",
" caveats_and_recommendations=\"Your model's caveats and recommendations.\",\n",
" custom_details={\"custom details1\": \"details value\"},\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 5. Initialize a model card\n",
"Initialize a model card with the information collected in the previous steps."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model_card_name = \"sample-notebook-model-card\"\n",
"my_card = ModelCard(\n",
" name=model_card_name,\n",
" status=ModelCardStatusEnum.DRAFT,\n",
" model_overview=model_overview,\n",
" training_details=training_details,\n",
" intended_uses=intended_uses,\n",
" business_details=business_details,\n",
" evaluation_details=evaluation_details,\n",
" additional_information=additional_information,\n",
" sagemaker_session=sagemaker_session,\n",
")\n",
"my_card.create()\n",
"print(f\"Model card {my_card.name} is successfully created with id {my_card.arn}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## Update Model Card<a name=\"Update\"></a>\n",
"After creating a model card, you can update the model card information. Updating a model card creates a new model card version."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"my_card.model_overview.model_description = \"the model is updated.\"\n",
"my_card.update()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## Load Model Card<a name=\"Load\"></a>\n",
"Load an existing model card with the model card name."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"my_card2 = ModelCard.load(\n",
" name=model_card_name,\n",
" sagemaker_session=sagemaker_session,\n",
")\n",
"print(f\"Model id: {my_card2.arn}\")\n",
"print(f\"Model description: {my_card.model_overview.model_description}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## List Model Card History<a name=\"ListHistory\"></a>\n",
"Track the model card history by listing historical versions."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"history = my_card.get_version_history()\n",
"assert len(history) == 2 # one for creation and one for update"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## Export Model Card<a name=\"Export\"></a>\n",
"Share the model card by exporting it to a PDF file.\n",
"\n",
"### 1. Create an export job"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"s3_output_path = f\"s3://{bucket}/{prefix}/export\"\n",
"pdf_s3_url = my_card.export_pdf(s3_output_path=s3_output_path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### (optional) List export jobs\n",
"Check all the export jobs for this model card."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"my_card.list_export_jobs()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2. Download the exported model card PDF\n",
"The downloaded PDF is stored in the same directory as this notebook by default.\n",
"\n",
"#### Parse the bucket and key of the exported PDF"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"parsed_url = urlparse(pdf_s3_url)\n",
"pdf_bucket = parsed_url.netloc\n",
"pdf_key = parsed_url.path.lstrip(\"/\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Download"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"file_name = parsed_url.path.split(\"/\")[-1]\n",
"s3.download_file(Filename=file_name, Bucket=pdf_bucket, Key=pdf_key)\n",
"print(f\"{file_name} is downloaded to \\n{os.path.join(os.getcwd(), file_name)}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## Cleanup<a name=\"Cleanup\"></a>\n",
"Delete the following resources:\n",
"1. The model card\n",
"2. The exported model card PDF\n",
"3. The example binary classification model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"my_card.delete()\n",
"\n",
"s3.delete_object(Bucket=pdf_bucket, Key=pdf_key)\n",
"\n",
"sagemaker_session.delete_model(model_name)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Notebook CI Test Results\n",
"\n",
"This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
"\n",
"![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/responsible_ai|sm-model_governance_model_card|sm-model_governance_model_card.ipynb)\n",
"\n",
"![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/responsible_ai|sm-model_governance_model_card|sm-model_governance_model_card.ipynb)\n",
"\n",
"![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/responsible_ai|sm-model_governance_model_card|sm-model_governance_model_card.ipynb)\n",
"\n",
"![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/responsible_ai|sm-model_governance_model_card|sm-model_governance_model_card.ipynb)\n",
"\n",
"![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/responsible_ai|sm-model_governance_model_card|sm-model_governance_model_card.ipynb)\n",
"\n",
"![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/responsible_ai|sm-model_governance_model_card|sm-model_governance_model_card.ipynb)\n",
"\n",
"![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/responsible_ai|sm-model_governance_model_card|sm-model_governance_model_card.ipynb)\n",
"\n",
"![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/responsible_ai|sm-model_governance_model_card|sm-model_governance_model_card.ipynb)\n",
"\n",
"![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/responsible_ai|sm-model_governance_model_card|sm-model_governance_model_card.ipynb)\n",
"\n",
"![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/responsible_ai|sm-model_governance_model_card|sm-model_governance_model_card.ipynb)\n",
"\n",
"![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/responsible_ai|sm-model_governance_model_card|sm-model_governance_model_card.ipynb)\n",
"\n",
"![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/responsible_ai|sm-model_governance_model_card|sm-model_governance_model_card.ipynb)\n",
"\n",
"![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/responsible_ai|sm-model_governance_model_card|sm-model_governance_model_card.ipynb)\n",
"\n",
"![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/responsible_ai|sm-model_governance_model_card|sm-model_governance_model_card.ipynb)\n",
"\n",
"![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/responsible_ai|sm-model_governance_model_card|sm-model_governance_model_card.ipynb)\n"
]
}
],
"metadata": {
"instance_type": "ml.t3.medium",
"kernelspec": {
"display_name": "Python 3 (Data Science 3.0)",
"language": "python",
"name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.6"
},
"vscode": {
"interpreter": {
"hash": "1571133684af95a48d70cfb4aef2840ed1e20d7c2d2a63af1685000148425678"
}
}
},
"nbformat": 4,
"nbformat_minor": 4
}
responsible_ai/sm-model_governance_model_card_with_model_package/example_metrics/clarify_bias.json
+49
View File
@@ -0,0 +1,49 @@
{
"post_training_bias_metrics": {
"facets": {
"Gender": [
{
"metrics": [
{
"description": "Accuracy Difference (AD)",
"name": "AD",
"value": -0.11415724421435
}
],
"value_or_threshold": "0"
},
{
"metrics": [
{
"name": "AD",
"description": "Accuracy Difference (AD)",
"value": -0.1141572442143538
}
],
"value_or_threshold": "1"
}
]
},
"label": "Target",
"label_value_or_threshold": "1"
},
"pre_training_bias_metrics": {
"facets": {
"Gender": [
{
"metrics": [
{
"description": "Conditional Demographic Disparity in Labels (CDDL)",
"name": "CDDL",
"value": 0.21491590864936
}
],
"value_or_threshold": "0"
}
]
},
"label": "Target",
"label_value_or_threshold": "1"
},
"version": "1.0"
}
responsible_ai/sm-model_governance_model_card_with_model_package/example_metrics/clarify_explanability.json
+102
View File
@@ -0,0 +1,102 @@
{
"version": "1.0",
"explanations": {
"kernel_shap": {
"label0": {
"global_shap_values": {
"Marital Status": 0.028630413807559794,
"Occupation": 0.02663187314914846,
"Relationship": 0.036107264188470226,
"Ethnic group": 0.01974161417121363,
"Capital Gain": 0.033577342651290996,
"Capital Loss": 0.019363293576206416,
"Hours per week": 0.021494553921674837,
"Country": 0.04632862084382646
},
"expected_value": 0.0006380207487381995,
"global_top_shap_text": {
"[feature_name_1]": {
"charming": 0.04089221769072609,
"magnificent": 0.03013490694249078
},
"[feature_name_2]": {
"charming": 0.04089221769072609,
"magnificent": 0.03013490694249078
}
}
},
"label1": {
"global_shap_values": {
"Age": 0.036556198315954205,
"Workclass": 0.01819397618827644,
"fnlwgt": 0.02137033320407158,
"Education": 0.018644355523796074,
"Education-Num": 0.037243930827630646
},
"expected_value": 0.0006380207487381995,
"global_top_shap_text": {
"[feature_name_1]": {
"charming": 0.04089221769072609,
"magnificent": 0.03013490694249078
},
"[feature_name_2]": {
"charming": 0.04089221769072609,
"magnificent": 0.03013490694249078
}
}
}
},
"pdp": [
{
"feature_name": "Age",
"data_type": "numerical",
"data_distribution": [
0.05,
0.165,
0.15,
0.095 ],
"feature_values": [
20.866666666666667,
24.6,
28.333366666666667,
32.06666666666666 ],
"model_predictions": [
[
0.7029578701406717,
0.6787781745567918,
0.7008252082392573,
0.733054383918643
]
],
"label_headers": [
"label0"
]
},
{
"feature_name": "Animal",
"data_type": "categorical",
"data_distribution": [
0.1,
0.11,
0.245,
0.06 ],
"feature_values": [
"Cat",
"Dog",
"Horse",
"Bird" ],
"model_predictions": [
[
0.8154899527132511,
0.787026209384203,
0.7409455917775631,
0.7098744075372815
]
],
"label_headers": [
"label0"
]
}
]
}
}
responsible_ai/sm-model_governance_model_card_with_model_package/example_metrics/generic_modelcard_evaluation.json
+34
View File
@@ -0,0 +1,34 @@
{
"metric_groups": [
{
"metric_data": [
{
"name": "AD",
"type": "number",
"value": -0.11415724421435
}
],
"name": "post_training_bias_metrics - label Target = 1 and facet Gender=0"
},
{
"metric_data": [
{
"name": "AD",
"type": "number",
"value": -0.1141572442143538
}
],
"name": "post_training_bias_metrics - label Target = 1 and facet Gender=1"
},
{
"metric_data": [
{
"name": "CDDL",
"type": "number",
"value": 0.21491590864936
}
],
"name": "pre_training_bias_metrics - label Target = 1 and facet Gender=0"
}
]
}
responsible_ai/sm-model_governance_model_card_with_model_package/example_metrics/model_monitor_model_quality_binary_classification.json
+116
View File
@@ -0,0 +1,116 @@
{
"binary_classification_metrics": {
"accuracy": {
"standard_deviation": "NaN",
"value": 0.5
},
"accuracy_best_constant_classifier": {
"standard_deviation": 0.2,
"value": 0.25
},
"auc": {
"standard_deviation": 0.1,
"value": 1
},
"confusion_matrix": {
"0": {
"0": 1,
"1": 2
},
"1": {
"0": 0,
"1": 1
}
},
"f0_5": {
"standard_deviation": "NaN",
"value": 0.38461538461538
},
"f0_5_best_constant_classifier": {
"standard_deviation": "NaN",
"value": 0.29411764705882
},
"f1": {
"standard_deviation": "NaN",
"value": 0.5
},
"f1_best_constant_classifier": {
"standard_deviation": "NaN",
"value": 0.4
},
"f2": {
"standard_deviation": "NaN",
"value": 0.71428571428571
},
"f2_best_constant_classifier": {
"standard_deviation": "NaN",
"value": 0.625
},
"false_negative_rate": {
"standard_deviation": "NaN",
"value": 0
},
"false_positive_rate": {
"standard_deviation": "NaN",
"value": 0.66666666666667
},
"precision": {
"standard_deviation": "NaN",
"value": 0.33333333333333
},
"precision_best_constant_classifier": {
"standard_deviation": "NaN",
"value": 0.25
},
"precision_recall_curve": {
"precisions": [
1,
1,
1,
1,
1
],
"recalls": [
0,
0.25,
0.5,
0.75,
1
]
},
"recall": {
"standard_deviation": "NaN",
"value": 1
},
"recall_best_constant_classifier": {
"standard_deviation": "NaN",
"value": 1
},
"receiver_operating_characteristic_curve": {
"false_positive_rates": [
0,
0,
0,
0,
0,
1
],
"true_positive_rates": [
0,
0.25,
0.5,
0.75,
1,
1
]
},
"true_negative_rate": {
"standard_deviation": "NaN",
"value": 0.33333333333333
},
"true_positive_rate": {
"standard_deviation": "NaN",
"value": 1
}
}
}
responsible_ai/sm-model_governance_model_card_with_model_package/sm-model_governance_model_card_with_model_package.ipynb
+517
View File
@@ -0,0 +1,517 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "6bfa072a-5d8b-4a9e-9058-f7e01bf5a8e1",
"metadata": {
"tags": []
},
"source": [
"# Amazon SageMaker Model Governance - Model Cards Model Registry integration"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "f6a7daa0-105e-4976-9eb9-363e8c6f78c9",
"metadata": {},
"source": [
"---\n",
"\n",
"This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
"\n",
"![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/responsible_ai|sm-model_governance_model_card_with_model_package|sm-model_governance_model_card_with_model_package.ipynb)\n",
"\n",
"---"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "63d1e3d2-e7f6-4f47-b170-5afd811f48e1",
"metadata": {},
"source": [
"This notebook walks you through the new feature of Amazon SageMaker Model Cards Model Registry Integration. To learn about the existing features and for more information on model cards, see [Model Cards](https://docs.aws.amazon.com/sagemaker/latest/dg/model-cards.html) in the _Amazon SageMaker Developer Guide_.\n",
"\n",
"Amazon SageMaker Model Cards give you the ability to create a centralized, customizable fact-sheet to document critical details about your machine learning (ML) models. Use model cards to keep a record of model information, such as intended uses, risk ratings, training details, evaluation metrics, and more for streamlined governance and reporting. \n",
"\n",
"In this example, you will create a model package along with a model card to document model package details along the way. Learn how to create a model card by associating model package using the Amazon SageMaker Python SDK.\n",
"\n",
"---\n",
"## Contents\n",
"\n",
"1. [Setup](#Setup)\n",
"1. [Create a model package](#ModelPackage)\n",
"1. [Create Model Card](#ModelCard)\n",
"1. [Cleanup](#Cleanup)\n",
"\n",
"---\n",
"## Setup\n",
"To begin, you must specify the following information:\n",
"- The IAM role ARN used to give SageMaker training and hosting access to your data. The following example uses the SageMaker execution role.\n",
"- The SageMaker session used to manage interactions with Amazon SageMaker Model Card API methods.\n",
"- The S3 URI (`bucket` and `prefix`) where you want to store training artifacts, models, and any exported model card PDFs. This S3 bucket should be in the same Region as your Notebook Instance, training, and hosting configurations. The following example uses the default SageMaker S3 bucket and creates a default SageMaker S3 bucket if one does not already exist.\n",
"- The S3 session used to manage interactions with Amazon S3 storage."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6ff3c234",
"metadata": {},
"outputs": [],
"source": [
"! pip install --upgrade 'sagemaker<3.0'"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c461da94-8746-44da-8f9d-21fe2a3c8652",
"metadata": {},
"outputs": [],
"source": [
"import boto3\n",
"from sagemaker.session import Session\n",
"from sagemaker import get_execution_role\n",
"\n",
"role = get_execution_role()\n",
"\n",
"sagemaker_session = Session()\n",
"\n",
"bucket = sagemaker_session.default_bucket()\n",
"prefix = \"model-card-registry-sample-notebook\"\n",
"region = sagemaker_session.boto_region_name\n",
"default_bucket_prefix = sagemaker_session.default_bucket_prefix\n",
"\n",
"# If a default bucket prefix is specified, append it to the s3 path\n",
"if default_bucket_prefix:\n",
" prefix = f\"{default_bucket_prefix}/{prefix}\"\n",
"\n",
"print(bucket)\n",
"print(region)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "32cb6670-d9d5-449e-a01c-c7ddb774af2f",
"metadata": {},
"source": [
"Next, import the necessary Python libraries."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7144d8d4-1290-4290-baa9-4e41bbef9ec5",
"metadata": {},
"outputs": [],
"source": [
"import io\n",
"import os\n",
"import time\n",
"import numpy as np\n",
"from six.moves.urllib.parse import urlparse\n",
"from pprint import pprint\n",
"import sagemaker\n",
"from sagemaker.image_uris import retrieve\n",
"import sagemaker.amazon.common as smac\n",
"from sagemaker.model_card import (\n",
" ModelCard,\n",
" ModelPackage,\n",
" IntendedUses,\n",
" ModelCardStatusEnum,\n",
")\n",
"from sagemaker.model_card.model_card import ModelApprovalStatusEnum"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "c8c67dd2-f277-417c-bca1-f1e211a0d9a2",
"metadata": {},
"source": [
"---\n",
"## Prepare a Model (Model package)<a name=\"ModelPackage\"></a>\n",
"The following code creates an example model package trained on a synthetic dataset. The target variable (0 or 1) is the second variable in the tuple.\n",
"\n",
"### 1. Prepare the training data\n",
"The code will upload example data to your S3 bucket."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0bfcad27-81d4-4c95-80f0-424067237e37",
"metadata": {},
"outputs": [],
"source": [
"s3 = boto3.client(\"s3\", region_name=region)\n",
"\n",
"# synthetic data\n",
"raw_data = (\n",
" (0.5, 0),\n",
" (0.75, 0),\n",
" (1.0, 0),\n",
" (1.25, 0),\n",
" (1.50, 0),\n",
" (1.75, 0),\n",
" (2.0, 0),\n",
" (2.25, 1),\n",
" (2.5, 0),\n",
" (2.75, 1),\n",
" (3.0, 0),\n",
" (3.25, 1),\n",
" (3.5, 0),\n",
" (4.0, 1),\n",
" (4.25, 1),\n",
" (4.5, 1),\n",
" (4.75, 1),\n",
" (5.0, 1),\n",
" (5.5, 1),\n",
")\n",
"training_data = np.array(raw_data).astype(\"float32\")\n",
"labels = training_data[:, 1]\n",
"\n",
"# upload data to S3 bucket\n",
"buf = io.BytesIO()\n",
"smac.write_numpy_to_dense_tensor(buf, training_data, labels)\n",
"buf.seek(0)\n",
"boto3.resource(\"s3\").Bucket(bucket).Object(os.path.join(prefix, \"train\")).upload_fileobj(buf)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "53fdc706-1c26-4b7a-b7b6-77dc353d9e2c",
"metadata": {},
"source": [
"### 2. Create a training job\n",
"Train a binary classification model with the training data from the previous step."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "722aff01-defd-4a9e-8b06-c0d7b320cc0d",
"metadata": {},
"outputs": [],
"source": [
"s3_train_data = f\"s3://{bucket}/{prefix}/train\"\n",
"output_location = f\"s3://{bucket}/{prefix}/output\"\n",
"container = retrieve(\"linear-learner\", sagemaker_session.boto_session.region_name)\n",
"estimator = sagemaker.estimator.Estimator(\n",
" container,\n",
" role=role,\n",
" instance_count=1,\n",
" instance_type=\"ml.m4.xlarge\",\n",
" output_path=output_location,\n",
" sagemaker_session=sagemaker_session,\n",
")\n",
"estimator.set_hyperparameters(feature_dim=2, mini_batch_size=10, predictor_type=\"binary_classifier\")\n",
"estimator.fit({\"train\": s3_train_data})\n",
"print(f\"Training job name: {estimator.latest_training_job.name}\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "eb31a7f3-3633-43f9-8220-a31cebdc14cc",
"metadata": {},
"source": [
"### 2. Create a model package"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "13385e3b-898f-4272-8c6a-5b030d016465",
"metadata": {},
"outputs": [],
"source": [
"# create a model package group\n",
"model_package_group_name = \"test-notebook-model-package-group\"\n",
"sagemaker_session.sagemaker_client.create_model_package_group(\n",
" ModelPackageGroupName=model_package_group_name\n",
")\n",
"\n",
"# describe training job to get model_data_url and image\n",
"training_job_name = estimator.latest_training_job.name\n",
"training_job = sagemaker_session.sagemaker_client.describe_training_job(\n",
" TrainingJobName=training_job_name\n",
")\n",
"\n",
"model_data_url = training_job[\"ModelArtifacts\"][\"S3ModelArtifacts\"]\n",
"image = training_job[\"AlgorithmSpecification\"][\"TrainingImage\"]\n",
"\n",
"# model package request input object\n",
"create_model_package_input_dict = {\n",
" \"ModelPackageGroupName\": model_package_group_name,\n",
" \"ModelPackageDescription\": \"Test model package registered for integ test\",\n",
" \"ModelApprovalStatus\": ModelApprovalStatusEnum.PENDING_MANUAL_APPROVAL,\n",
" \"InferenceSpecification\": {\n",
" \"Containers\": [{\"Image\": image, \"ModelDataUrl\": model_data_url}],\n",
" \"SupportedContentTypes\": [\"text/csv\"],\n",
" \"SupportedResponseMIMETypes\": [\"text/csv\"],\n",
" },\n",
"}\n",
"\n",
"model_pkg = sagemaker_session.sagemaker_client.create_model_package(\n",
" **create_model_package_input_dict\n",
")\n",
"print(\"Model package ARN:\", model_pkg[\"ModelPackageArn\"])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "2fb11fce-ef8b-4db3-9dbe-79bcc6b781d6",
"metadata": {},
"source": [
"---\n",
"## Create Model Card<a name=\"ModelCard\"></a>\n",
"Document your model package details in an Amazon SageMaker Model Card using the SageMaker Python SDK."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "bf1473e8-2ce2-4bbb-aea9-1b1f8f895091",
"metadata": {},
"source": [
"### 1. Collect model package details\n",
"Automatically collect basic model package information like model package ARN, model package group name, model package approval status, and model package's inference specification information."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "56af1e2b-688b-4ddb-980b-24098695a25a",
"metadata": {},
"outputs": [],
"source": [
"model_package_details = ModelPackage.from_model_package_arn(\n",
" model_package_arn=model_pkg[\"ModelPackageArn\"],\n",
" sagemaker_session=sagemaker_session,\n",
")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "46bc452c-55d8-4395-ad83-7403b2e48d7d",
"metadata": {},
"source": [
"### 2. Initialize a model card\n",
"Initialize a model card with the model package details collected in the previous step. When associating model package to a model card, model card will try to auto discover information like training job details and evaluation job details only if there are information like model artifacts and model metrics available in model package. Additionally, it will also try to carry over some additional information like business details to this model card from the previously created the most recent model card that is associated with this particular model package group."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "19966fff-86d1-4c06-bc84-05dca057c1ec",
"metadata": {},
"outputs": [],
"source": [
"model_card_name = \"sample-model-card-with-model-package\"\n",
"my_card = ModelCard(\n",
" name=model_card_name,\n",
" status=ModelCardStatusEnum.DRAFT,\n",
" model_package_details=model_package_details,\n",
" intended_uses=IntendedUses(\n",
" purpose_of_model=\"Test model card.\",\n",
" intended_uses=\"Not used except this test.\",\n",
" ),\n",
" sagemaker_session=sagemaker_session,\n",
")\n",
"\n",
"# Check auto-discovered data\n",
"print(\"Auto discovered training job details\")\n",
"print(\" arn: \", my_card.training_details.training_job_details.training_arn)\n",
"print(\n",
" \" environment: \",\n",
" my_card.training_details.training_job_details.training_environment.container_image,\n",
")\n",
"print(\n",
" \" metrics: \",\n",
" [(m.name, m.value) for m in my_card.training_details.training_job_details.training_metrics],\n",
")\n",
"print(\n",
" \" hyper-parameters: \",\n",
" [(h.name, h.value) for h in my_card.training_details.training_job_details.hyper_parameters],\n",
")\n",
"\n",
"my_card.create()\n",
"print(f\"Model card {my_card.name} is successfully created with id {my_card.arn}\")\n",
"\n",
"time.sleep(\n",
" 3\n",
") # sleep 3s to wait for model card being populated in the search service which is required by information inheritance"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "c891c7f9",
"metadata": {},
"source": [
"#### Information inheritance\n",
"Additionally, new model card will also try to carry over some additional information like business details, intended uses, additional information to this model card from the previously created the most recent model card that is associated with this particular model package group. In this example, check out the intended uses that is automatically carried over from the previous model card."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "36164c44",
"metadata": {},
"outputs": [],
"source": [
"# create another model package under the same model package group as sample-model-card-with-model-package\n",
"create_model_package_input_dict2 = {\n",
" \"ModelPackageGroupName\": model_package_group_name,\n",
" \"ModelPackageDescription\": \"Test model package registered for integ test\",\n",
" \"ModelApprovalStatus\": ModelApprovalStatusEnum.PENDING_MANUAL_APPROVAL,\n",
" \"InferenceSpecification\": {\n",
" \"Containers\": [{\"Image\": image, \"ModelDataUrl\": model_data_url}],\n",
" \"SupportedContentTypes\": [\"text/csv\"],\n",
" \"SupportedResponseMIMETypes\": [\"text/csv\"],\n",
" },\n",
"}\n",
"\n",
"model_pkg2 = sagemaker_session.sagemaker_client.create_model_package(\n",
" **create_model_package_input_dict2\n",
")\n",
"model_package_details2 = ModelPackage.from_model_package_arn(\n",
" model_package_arn=model_pkg2[\"ModelPackageArn\"],\n",
" sagemaker_session=sagemaker_session,\n",
")\n",
"\n",
"# create another model card with the new model package\n",
"model_card_name2 = \"sample-model-card-with-model-package2\"\n",
"my_card2 = ModelCard(\n",
" name=model_card_name2,\n",
" status=ModelCardStatusEnum.DRAFT,\n",
" model_package_details=model_package_details2,\n",
" sagemaker_session=sagemaker_session,\n",
")\n",
"\n",
"print(\n",
" \"Information carried over from the latest model card, i.e. sample-model-card-with-model-package, associated with the same model package group\"\n",
")\n",
"print(\" Intended uses: \")\n",
"print(\" purpose_of_model: \", my_card2.intended_uses.purpose_of_model)\n",
"print(\" intended_uses: \", my_card2.intended_uses.intended_uses)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "fe88e044",
"metadata": {},
"source": [
"---\n",
"## Cleanup (Optional)<a name=\"Cleanup\"></a>\n",
"Delete the following resources:\n",
"1. The model card\n",
"2. The model package\n",
"3. The model package group"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7d447849",
"metadata": {},
"outputs": [],
"source": [
"my_card.delete()\n",
"\n",
"response = sagemaker_session.sagemaker_client.list_model_packages(\n",
" ModelPackageGroupName=model_package_group_name\n",
")\n",
"\n",
"for package in response[\"ModelPackageSummaryList\"]:\n",
" sagemaker_session.sagemaker_client.delete_model_package(\n",
" ModelPackageName=package[\"ModelPackageArn\"]\n",
" )\n",
"\n",
"sagemaker_session.sagemaker_client.delete_model_package_group(\n",
" ModelPackageGroupName=model_package_group_name\n",
")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "ba5c2fdd",
"metadata": {},
"source": [
"## Notebook CI Test Results\n",
"\n",
"This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
"\n",
"\n",
"![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/responsible_ai|sm-model_governance_model_card_with_model_package|sm-model_governance_model_card_with_model_package.ipynb)\n",
"\n",
"![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/responsible_ai|sm-model_governance_model_card_with_model_package|sm-model_governance_model_card_with_model_package.ipynb)\n",
"\n",
"![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/responsible_ai|sm-model_governance_model_card_with_model_package|sm-model_governance_model_card_with_model_package.ipynb)\n",
"\n",
"![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/responsible_ai|sm-model_governance_model_card_with_model_package|sm-model_governance_model_card_with_model_package.ipynb)\n",
"\n",
"![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/responsible_ai|sm-model_governance_model_card_with_model_package|sm-model_governance_model_card_with_model_package.ipynb)\n",
"\n",
"![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/responsible_ai|sm-model_governance_model_card_with_model_package|sm-model_governance_model_card_with_model_package.ipynb)\n",
"\n",
"![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/responsible_ai|sm-model_governance_model_card_with_model_package|sm-model_governance_model_card_with_model_package.ipynb)\n",
"\n",
"![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/responsible_ai|sm-model_governance_model_card_with_model_package|sm-model_governance_model_card_with_model_package.ipynb)\n",
"\n",
"![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/responsible_ai|sm-model_governance_model_card_with_model_package|sm-model_governance_model_card_with_model_package.ipynb)\n",
"\n",
"![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/responsible_ai|sm-model_governance_model_card_with_model_package|sm-model_governance_model_card_with_model_package.ipynb)\n",
"\n",
"![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/responsible_ai|sm-model_governance_model_card_with_model_package|sm-model_governance_model_card_with_model_package.ipynb)\n",
"\n",
"![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/responsible_ai|sm-model_governance_model_card_with_model_package|sm-model_governance_model_card_with_model_package.ipynb)\n",
"\n",
"![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/responsible_ai|sm-model_governance_model_card_with_model_package|sm-model_governance_model_card_with_model_package.ipynb)\n",
"\n",
"![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/responsible_ai|sm-model_governance_model_card_with_model_package|sm-model_governance_model_card_with_model_package.ipynb)\n",
"\n",
"![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/responsible_ai|sm-model_governance_model_card_with_model_package|sm-model_governance_model_card_with_model_package.ipynb)\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "a2e1bf9e",
"metadata": {},
"source": []
}
],
"metadata": {
"instance_type": "ml.t3.medium",
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.13"
}
},
"nbformat": 4,
"nbformat_minor": 5
}