chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,51 @@
|
||||
.. image:: https://github.com/aws/sagemaker-python-sdk/raw/master/branding/icon/sagemaker-banner.png
|
||||
:height: 100px
|
||||
:alt: SageMaker
|
||||
|
||||
====================
|
||||
SageMaker Core
|
||||
====================
|
||||
|
||||
.. image:: https://img.shields.io/pypi/v/sagemaker-core.svg
|
||||
:target: https://pypi.python.org/pypi/sagemaker-core
|
||||
:alt: Latest Version
|
||||
|
||||
.. image:: https://img.shields.io/pypi/pyversions/sagemaker-core.svg
|
||||
:target: https://pypi.python.org/pypi/sagemaker-core
|
||||
:alt: Supported Python Versions
|
||||
|
||||
|
||||
|
||||
Introduction
|
||||
------------
|
||||
|
||||
Welcome to the sagemaker-core Python SDK, an SDK designed to provide an object-oriented interface for interacting with Amazon SageMaker resources. It offers full parity with SageMaker APIs, allowing developers to leverage all SageMaker capabilities directly through the SDK. sagemaker-core introduces features such as dedicated resource classes, resource chaining, auto code completion, comprehensive documentation and type hints to enhance the developer experience as well as productivity.
|
||||
|
||||
|
||||
Key Features
|
||||
------------
|
||||
|
||||
* **Object-Oriented Interface**: Provides a structured way to interact with SageMaker resources, making it easier to manage them using familiar object-oriented programming techniques.
|
||||
* **Resource Chaining**: Allows seamless connection of SageMaker resources by passing outputs as inputs between them, simplifying workflows and reducing the complexity of parameter management.
|
||||
* **Full Parity with SageMaker APIs**: Ensures access to all SageMaker capabilities through the SDK, providing a comprehensive toolset for building and deploying machine learning models.
|
||||
* **Abstraction of Low-Level Details**: Automatically handles resource state transitions and polling logic, freeing developers from managing these intricacies and allowing them to focus on higher-level tasks.
|
||||
* **Auto Code Completion**: Enhances the developer experience by offering real-time suggestions and completions in popular IDEs, reducing syntax errors and speeding up the coding process.
|
||||
* **Comprehensive Documentation and Type Hints**: Provides detailed guidance and type hints to help developers understand functionalities, write code faster, and reduce errors without complex API navigation.
|
||||
* **Incorporation of Intelligent Defaults**: Integrates the previous SageMaker SDK feature of intelligent defaults, allowing developers to set default values for parameters like IAM roles and VPC configurations. This streamlines the setup process, enabling developers to focus on customizations specific to their use case.
|
||||
|
||||
|
||||
Benefits
|
||||
--------
|
||||
|
||||
* **Simplified Development**: By abstracting low-level details and providing intelligent defaults, developers can focus on building and deploying machine learning models without getting bogged down by repetitive tasks.
|
||||
* **Increased Productivity**: The SDK's features, such as auto code completion and type hints, help developers write code faster and with fewer errors.
|
||||
* **Enhanced Readability**: Resource chaining and dedicated resource classes result in more readable and maintainable code.
|
||||
|
||||
|
||||
Docs and Examples
|
||||
-----------------
|
||||
Learn more about the sagemaker-core SDK and its features by visting the `What's New Announcement <https://aws.amazon.com/about-aws/whats-new/2024/09/sagemaker-core-object-oriented-sdk-amazon-sagemaker>`_.
|
||||
|
||||
For examples and walkthroughs, see the `SageMaker Core Examples <https://github.com/aws/amazon-sagemaker-examples/tree/default/%20%20%20%20%20%20%20%20%20sagemaker-core>`_.
|
||||
|
||||
For detailed documentation, including the API reference, see `Read the Docs <https://sagemaker-core.readthedocs.io>`_.
|
||||
@@ -0,0 +1,740 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b02083d2-55a3-41e6-aed5-8ff633af6a59",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Get started with SageMaker\n",
|
||||
"In this notebook you'll learn how SageMaker can be used to:\n",
|
||||
"\n",
|
||||
"1. Preprocess (and optionally explore) a dataset\n",
|
||||
"2. Train an XGBoost classifier for customer churn prediction, using a managed job with SageMaker Training, using a managed image.\n",
|
||||
"3. Perform hyperparameter tuning to find optimal set of hyperparameters, using a managed job with SageMaker HyperParameter Tuning\n",
|
||||
"5. Perform batch inference using a managed SageMaker Batch Transform job.\n",
|
||||
"7. Create a managed real-time SageMaker endpoint.\n",
|
||||
"\n",
|
||||
"All SageMaker resources are created using the SageMaker Core SDK. You can find more information about sagemaker-core [here](https://sagemaker-core.readthedocs.io/en/latest/)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "0489c209-9628-4ddf-a849-8b973234126a",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%pip install --upgrade pip -q\n",
|
||||
"%pip install sagemaker-core -q"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "314d959f-8c4a-4996-b167-24958d53417a",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import time\n",
|
||||
"import sagemaker\n",
|
||||
"\n",
|
||||
"from sagemaker_core.helper.session_helper import Session, get_execution_role\n",
|
||||
"\n",
|
||||
"# Set up region, role and bucket parameters used throughout the notebook.\n",
|
||||
"sagemaker_session = Session()\n",
|
||||
"region = sagemaker_session.boto_region_name\n",
|
||||
"role = get_execution_role()\n",
|
||||
"bucket = sagemaker.Session().default_bucket()\n",
|
||||
"default_bucket_prefix = sagemaker.Session().default_bucket_prefix\n",
|
||||
"\n",
|
||||
"print(f\"AWS region: {region}\")\n",
|
||||
"print(f\"Execution role: {role}\")\n",
|
||||
"print(f\"Default S3 bucket: {bucket}\")\n",
|
||||
"print(f\"Default S3 bucket prefix: {default_bucket_prefix}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "93b62649-4d89-4b6e-9f18-a48aa0bb27a4",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Preprocess dataset\n",
|
||||
"We'll use a synthetic dataset that AWS provides for customer churn prediction.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f347ca27-68a3-41a1-998c-434d39f8245a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"<div class=\"alert alert-block alert-info\">\n",
|
||||
"<b>NOTE:</b> This sample doesn't perform any exploratory data anlysis since how to preprocess the dataset is already known.\n",
|
||||
" \n",
|
||||
"If you're interested in how to perform exploratory analysis, there's a section in the documentation for the sagemaker-python-sdk available that explores the dataset, [here](https://sagemaker-examples.readthedocs.io/en/latest/introduction_to_applying_machine_learning/xgboost_customer_churn/xgboost_customer_churn.html).\n",
|
||||
"</div>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "38f74911-14c9-4ef8-bd18-db9664de7dcf",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Read the data from S3"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "53c16f79-f58f-412f-8716-c62179a270e5",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from io import StringIO\n",
|
||||
"import pandas as pd\n",
|
||||
"\n",
|
||||
"data = sagemaker_session.read_s3_file(\n",
|
||||
" f\"sagemaker-example-files-prod-{region}\", \"datasets/tabular/synthetic/churn.txt\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"df = pd.read_csv(StringIO(data))\n",
|
||||
"df"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2e00d8db-e6c8-4233-8437-8c2076895416",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Apply processing"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "de3d40e5-6d29-443e-98b1-8629ce106cac",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"\n",
|
||||
"# Phone number is unique - will not add value to classifier\n",
|
||||
"df = df.drop(\"Phone\", axis=1)\n",
|
||||
"\n",
|
||||
"# Cast Area Code to non-numeric\n",
|
||||
"df[\"Area Code\"] = df[\"Area Code\"].astype(object)\n",
|
||||
"\n",
|
||||
"# Remove one feature from highly corelated pairs\n",
|
||||
"df = df.drop([\"Day Charge\", \"Eve Charge\", \"Night Charge\", \"Intl Charge\"], axis=1)\n",
|
||||
"\n",
|
||||
"# One-hot encode catagorical features into numeric features\n",
|
||||
"model_data = pd.get_dummies(df)\n",
|
||||
"model_data = pd.concat(\n",
|
||||
" [\n",
|
||||
" model_data[\"Churn?_True.\"],\n",
|
||||
" model_data.drop([\"Churn?_False.\", \"Churn?_True.\"], axis=1),\n",
|
||||
" ],\n",
|
||||
" axis=1,\n",
|
||||
")\n",
|
||||
"model_data = model_data.astype(float)\n",
|
||||
"\n",
|
||||
"# Split data into train and validation datasets\n",
|
||||
"train_data, validation_data = train_test_split(model_data, test_size=0.33, random_state=42)\n",
|
||||
"\n",
|
||||
"# Further split the validation dataset into test and validation datasets.\n",
|
||||
"validation_data, test_data = train_test_split(validation_data, test_size=0.33, random_state=42)\n",
|
||||
"\n",
|
||||
"# Remove and store the target column for the test data. This is used for calculating performance metrics after training, on unseen data.\n",
|
||||
"test_target_column = test_data[\"Churn?_True.\"]\n",
|
||||
"test_data.drop([\"Churn?_True.\"], axis=1, inplace=True)\n",
|
||||
"\n",
|
||||
"# Store all datasets locally\n",
|
||||
"train_data.to_csv(\"train.csv\", header=False, index=False)\n",
|
||||
"validation_data.to_csv(\"validation.csv\", header=False, index=False)\n",
|
||||
"test_data.to_csv(\"test.csv\", header=False, index=False)\n",
|
||||
"\n",
|
||||
"# Upload each dataset to S3\n",
|
||||
"# If a default bucket prefix is specified, append it to the s3 path\n",
|
||||
"if default_bucket_prefix:\n",
|
||||
" s3_train_input = sagemaker_session.upload_data(\n",
|
||||
" \"train.csv\", bucket, key_prefix=default_bucket_prefix\n",
|
||||
" )\n",
|
||||
" s3_validation_input = sagemaker_session.upload_data(\n",
|
||||
" \"validation.csv\", bucket, key_prefix=default_bucket_prefix\n",
|
||||
" )\n",
|
||||
" s3_test_input = sagemaker_session.upload_data(\n",
|
||||
" \"test.csv\", bucket, key_prefix=default_bucket_prefix\n",
|
||||
" )\n",
|
||||
"else:\n",
|
||||
" s3_train_input = sagemaker_session.upload_data(\"train.csv\", bucket)\n",
|
||||
" s3_validation_input = sagemaker_session.upload_data(\"validation.csv\", bucket)\n",
|
||||
" s3_test_input = sagemaker_session.upload_data(\"test.csv\", bucket)\n",
|
||||
"\n",
|
||||
"print(\"Datasets uploaded to:\")\n",
|
||||
"print(s3_train_input)\n",
|
||||
"print(s3_validation_input)\n",
|
||||
"print(s3_test_input)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5d8c4139-244d-4827-b872-35fbeed664d8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Train a classifier using XGBoost\n",
|
||||
"Use SageMaker Training and the managed XGBoost image to train a classifier. <br />\n",
|
||||
"More details on how to use SageMaker managed training with XGBoost can be found [here](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "bbbf270c-20c0-4f82-a8a6-c2794ba57f70",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"<div class=\"alert alert-block alert-info\">\n",
|
||||
" <b>NOTE:</b> For more information on using SageMaker managed container images and retrieving their ECR paths, \n",
|
||||
" <a href=\"https://docs.aws.amazon.com/sagemaker/latest/dg-ecr-paths/sagemaker-algo-docker-registry-paths.html\" target=\"_blank\">here</a> \n",
|
||||
" is the documentation. Please note that the image URI might need to be updated based on your selected AWS region.\n",
|
||||
"</div>\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "268583f2-2c20-401b-a57c-ee88fd402583",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"image = \"141502667606.dkr.ecr.eu-west-1.amazonaws.com/sagemaker-xgboost:1.7-1\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "e85a91b5-4a68-43f4-b04a-991035cc35bf",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.resources import TrainingJob\n",
|
||||
"from sagemaker_core.shapes import (\n",
|
||||
" AlgorithmSpecification,\n",
|
||||
" Channel,\n",
|
||||
" DataSource,\n",
|
||||
" S3DataSource,\n",
|
||||
" ResourceConfig,\n",
|
||||
" StoppingCondition,\n",
|
||||
" OutputDataConfig,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"job_name = \"xgboost-churn-\" + time.strftime(\n",
|
||||
" \"%Y-%m-%d-%H-%M-%S\", time.gmtime()\n",
|
||||
") # Name of training job\n",
|
||||
"instance_type = \"ml.m4.xlarge\" # SageMaker instance type to use for training\n",
|
||||
"instance_count = 1 # Number of instances to use for training\n",
|
||||
"volume_size_in_gb = 30 # Amount of storage to allocate to training job\n",
|
||||
"max_runtime_in_seconds = 600 # Maximum runtimt. Job exits if it doesn't finish before this\n",
|
||||
"s3_output_path = f\"s3://{bucket}\" # bucket and optional prefix where the training job stores output artifacts, like model artifact.\n",
|
||||
"\n",
|
||||
"# If a default bucket prefix is specified, append it to the s3 path\n",
|
||||
"if default_bucket_prefix:\n",
|
||||
" s3_output_path = f\"s3://{bucket}/{default_bucket_prefix}\"\n",
|
||||
"\n",
|
||||
"# Specify hyperparameters\n",
|
||||
"hyper_parameters = {\n",
|
||||
" \"max_depth\": \"5\",\n",
|
||||
" \"eta\": \"0.2\",\n",
|
||||
" \"gamma\": \"4\",\n",
|
||||
" \"min_child_weight\": \"6\",\n",
|
||||
" \"subsample\": \"0.8\",\n",
|
||||
" \"verbosity\": \"0\",\n",
|
||||
" \"objective\": \"binary:logistic\",\n",
|
||||
" \"num_round\": \"100\",\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"# Create training job.\n",
|
||||
"training_job = TrainingJob.create(\n",
|
||||
" training_job_name=job_name,\n",
|
||||
" hyper_parameters=hyper_parameters,\n",
|
||||
" algorithm_specification=AlgorithmSpecification(\n",
|
||||
" training_image=image, training_input_mode=\"File\"\n",
|
||||
" ),\n",
|
||||
" role_arn=role,\n",
|
||||
" input_data_config=[\n",
|
||||
" Channel(\n",
|
||||
" channel_name=\"train\",\n",
|
||||
" content_type=\"csv\",\n",
|
||||
" data_source=DataSource(\n",
|
||||
" s3_data_source=S3DataSource(\n",
|
||||
" s3_data_type=\"S3Prefix\",\n",
|
||||
" s3_uri=s3_train_input,\n",
|
||||
" s3_data_distribution_type=\"FullyReplicated\",\n",
|
||||
" )\n",
|
||||
" ),\n",
|
||||
" ),\n",
|
||||
" Channel(\n",
|
||||
" channel_name=\"validation\",\n",
|
||||
" content_type=\"csv\",\n",
|
||||
" data_source=DataSource(\n",
|
||||
" s3_data_source=S3DataSource(\n",
|
||||
" s3_data_type=\"S3Prefix\",\n",
|
||||
" s3_uri=s3_validation_input,\n",
|
||||
" s3_data_distribution_type=\"FullyReplicated\",\n",
|
||||
" )\n",
|
||||
" ),\n",
|
||||
" ),\n",
|
||||
" ],\n",
|
||||
" output_data_config=OutputDataConfig(s3_output_path=s3_output_path),\n",
|
||||
" resource_config=ResourceConfig(\n",
|
||||
" instance_type=instance_type,\n",
|
||||
" instance_count=instance_count,\n",
|
||||
" volume_size_in_gb=volume_size_in_gb,\n",
|
||||
" ),\n",
|
||||
" stopping_condition=StoppingCondition(max_runtime_in_seconds=max_runtime_in_seconds),\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Wait for the training job to complete\n",
|
||||
"training_job.wait()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8db8e0b6-d000-453a-baf9-6b7c164d1caf",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Hyperparameter tuning\n",
|
||||
"If the optimal hyperparameters aren't known, we perform a SageMaker Hyperparameter Tuning job, which runs several training jobs and iteratively finds the best set of parameters.\n",
|
||||
"\n",
|
||||
"From a high level, a tuning job constists of 2 main components:\n",
|
||||
"- `HyperParameterTrainingJobDefinition`, which specifies details for each individidual training job, like image to use, input channels, resource configuration etc.\n",
|
||||
"- `HyperParameterTuningJobConfig`, which details the tuning configuration, like what strategy to use, how many jobs to run and what parameters to tune etc.\n",
|
||||
"\n",
|
||||
"You can find more information about how it works [here](https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-how-it-works.html)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "02419934-5cd4-4fda-9706-c852545e4ac6",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.resources import HyperParameterTuningJob\n",
|
||||
"from sagemaker_core.shapes import (\n",
|
||||
" HyperParameterTuningJobConfig,\n",
|
||||
" ResourceLimits,\n",
|
||||
" ParameterRanges,\n",
|
||||
" AutoParameter,\n",
|
||||
" Autotune,\n",
|
||||
" HyperParameterTrainingJobDefinition,\n",
|
||||
" HyperParameterTuningJobObjective,\n",
|
||||
" HyperParameterAlgorithmSpecification,\n",
|
||||
" OutputDataConfig,\n",
|
||||
" StoppingCondition,\n",
|
||||
" ResourceConfig,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"tuning_job_name = \"xgboost-tune-\" + time.strftime(\n",
|
||||
" \"%Y-%m-%d-%H-%M-%S\", time.gmtime()\n",
|
||||
") # Name of tuning job\n",
|
||||
"\n",
|
||||
"max_number_of_training_jobs = (\n",
|
||||
" 50 # Maximum number of training jobs to run as part of the tuning job.\n",
|
||||
")\n",
|
||||
"max_parallel_training_jobs = 5 # Maximum number of parallell training.\n",
|
||||
"max_runtime_in_seconds = 3600 # Maximum runtime for tuning job.\n",
|
||||
"\n",
|
||||
"# Create HyperParameterTrainingJobDefinition object, containing information about each individual training job.\n",
|
||||
"hyper_parameter_training_job_defintion = HyperParameterTrainingJobDefinition(\n",
|
||||
" role_arn=role,\n",
|
||||
" algorithm_specification=HyperParameterAlgorithmSpecification(\n",
|
||||
" training_image=image, training_input_mode=\"File\"\n",
|
||||
" ),\n",
|
||||
" input_data_config=[\n",
|
||||
" Channel(\n",
|
||||
" channel_name=\"train\",\n",
|
||||
" content_type=\"csv\",\n",
|
||||
" data_source=DataSource(\n",
|
||||
" s3_data_source=S3DataSource(\n",
|
||||
" s3_data_type=\"S3Prefix\",\n",
|
||||
" s3_uri=s3_train_input,\n",
|
||||
" s3_data_distribution_type=\"FullyReplicated\",\n",
|
||||
" )\n",
|
||||
" ),\n",
|
||||
" ),\n",
|
||||
" Channel(\n",
|
||||
" channel_name=\"validation\",\n",
|
||||
" content_type=\"csv\",\n",
|
||||
" data_source=DataSource(\n",
|
||||
" s3_data_source=S3DataSource(\n",
|
||||
" s3_data_type=\"S3Prefix\",\n",
|
||||
" s3_uri=s3_validation_input,\n",
|
||||
" s3_data_distribution_type=\"FullyReplicated\",\n",
|
||||
" )\n",
|
||||
" ),\n",
|
||||
" ),\n",
|
||||
" ],\n",
|
||||
" output_data_config=OutputDataConfig(s3_output_path=s3_output_path),\n",
|
||||
" stopping_condition=StoppingCondition(max_runtime_in_seconds=max_runtime_in_seconds),\n",
|
||||
" resource_config=ResourceConfig(\n",
|
||||
" instance_type=instance_type,\n",
|
||||
" instance_count=instance_count,\n",
|
||||
" volume_size_in_gb=volume_size_in_gb,\n",
|
||||
" ),\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Create HyperParameterTrainingJobDefinition object, containing information about the tuning job\n",
|
||||
"tuning_job_config = HyperParameterTuningJobConfig(\n",
|
||||
" strategy=\"Bayesian\",\n",
|
||||
" hyper_parameter_tuning_job_objective=HyperParameterTuningJobObjective(\n",
|
||||
" type=\"Maximize\", metric_name=\"validation:auc\"\n",
|
||||
" ),\n",
|
||||
" resource_limits=ResourceLimits(\n",
|
||||
" max_number_of_training_jobs=max_number_of_training_jobs,\n",
|
||||
" max_parallel_training_jobs=max_parallel_training_jobs,\n",
|
||||
" max_runtime_in_seconds=3600,\n",
|
||||
" ),\n",
|
||||
" training_job_early_stopping_type=\"Auto\",\n",
|
||||
" parameter_ranges=ParameterRanges(\n",
|
||||
" auto_parameters=[\n",
|
||||
" AutoParameter(name=\"max_depth\", value_hint=\"5\"),\n",
|
||||
" AutoParameter(name=\"eta\", value_hint=\"0.1\"),\n",
|
||||
" AutoParameter(name=\"gamma\", value_hint=\"8\"),\n",
|
||||
" AutoParameter(name=\"min_child_weight\", value_hint=\"2\"),\n",
|
||||
" AutoParameter(name=\"subsample\", value_hint=\"0.5\"),\n",
|
||||
" AutoParameter(name=\"num_round\", value_hint=\"50\"),\n",
|
||||
" ]\n",
|
||||
" ),\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Create the tuning job using the 2 configuration objects above\n",
|
||||
"tuning_job = HyperParameterTuningJob.create(\n",
|
||||
" hyper_parameter_tuning_job_name=tuning_job_name,\n",
|
||||
" autotune=Autotune(mode=\"Enabled\"),\n",
|
||||
" training_job_definition=hyper_parameter_training_job_defintion,\n",
|
||||
" hyper_parameter_tuning_job_config=tuning_job_config,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"tuning_job.wait()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b79e6746-6a03-4868-a22e-63a32bfcf599",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Use model artifacts for batch inference\n",
|
||||
"To use the model to perform batch inference, we can use a SageMaker Batch Transform job. The Transform Job requires a SageMaker model object, which contains information about what image and model to use.\n",
|
||||
"\n",
|
||||
"Below, we:\n",
|
||||
"1. Create a SageMaker model with the same first-party image as we used for training, and the model artifacts produced during training. Indeed, such image can also be used to run inference\n",
|
||||
"2. Use that SagMaker model with a Transform Job to perform batch inference with our test dataset\n",
|
||||
"3. Compute some performance metrics\n",
|
||||
"\n",
|
||||
"More information about SageMaker Batch Transform can be found [here](https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "82b0db2b-4daf-4ee0-a035-b7108f3d7912",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Create SageMaker Model\n",
|
||||
"\n",
|
||||
"Create a Model resource based on the model artifacts produced by the best training job run by through hyperparameter tuning. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "270aa263-2a8f-4225-bcd6-56a57195ac63",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.resources import Model\n",
|
||||
"from sagemaker_core.shapes import ContainerDefinition\n",
|
||||
"\n",
|
||||
"# model_s3_uri = training_job.model_artifacts.s3_model_artifacts # Get URI of model artifacts from the training job.\n",
|
||||
"model_s3_uri = TrainingJob.get(\n",
|
||||
" tuning_job.best_training_job.training_job_name\n",
|
||||
").model_artifacts.s3_model_artifacts # Get URI of model artifacts of the best model from the tuning job.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Create SageMaker model: An image along with the model artifact to use.\n",
|
||||
"customer_churn_model = Model.create(\n",
|
||||
" model_name=\"customer-churn-xgboost\",\n",
|
||||
" primary_container=ContainerDefinition(image=image, model_data_url=model_s3_uri),\n",
|
||||
" execution_role_arn=role,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7151c8e0-c0d8-49c9-9bda-28d0fe816b59",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Create Transform Job"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "d15f53e5-8cc4-487b-bd64-fe8efff11517",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.resources import TransformJob\n",
|
||||
"from sagemaker_core.shapes import (\n",
|
||||
" TransformInput,\n",
|
||||
" TransformDataSource,\n",
|
||||
" TransformS3DataSource,\n",
|
||||
" TransformOutput,\n",
|
||||
" TransformResources,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"model_name = customer_churn_model.get_name()\n",
|
||||
"transform_job_name = \"churn-prediction\" + time.strftime(\n",
|
||||
" \"%Y-%m-%d-%H-%M-%S\", time.gmtime()\n",
|
||||
") # Name of TranformJob\n",
|
||||
"s3_output_path = f\"s3://{bucket}/transform\" # bucket and optional prefix where the TranformJob stores the result.\n",
|
||||
"\n",
|
||||
"# If a default bucket prefix is specified, append it to the s3 path\n",
|
||||
"if default_bucket_prefix:\n",
|
||||
" s3_output_path = f\"s3://{bucket}/{default_bucket_prefix}/transform\"\n",
|
||||
"\n",
|
||||
"instance_type = \"ml.m4.xlarge\" # SageMaker instance type to use for TranformJob\n",
|
||||
"instance_count = 1 # Number of instances to use for TranformJob\n",
|
||||
"\n",
|
||||
"# Create Transform Job.\n",
|
||||
"transform_job = TransformJob.create(\n",
|
||||
" transform_job_name=transform_job_name,\n",
|
||||
" model_name=model_name,\n",
|
||||
" transform_input=TransformInput(\n",
|
||||
" data_source=TransformDataSource(\n",
|
||||
" s3_data_source=TransformS3DataSource(s3_data_type=\"S3Prefix\", s3_uri=s3_test_input)\n",
|
||||
" ),\n",
|
||||
" content_type=\"text/csv\",\n",
|
||||
" ),\n",
|
||||
" transform_output=TransformOutput(s3_output_path=s3_output_path),\n",
|
||||
" transform_resources=TransformResources(\n",
|
||||
" instance_type=instance_type, instance_count=instance_count\n",
|
||||
" ),\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"transform_job.wait()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5eca0ceb-b23b-4d72-9d39-dc1a295af068",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Compute performance metrics"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "2b55cb28-ed28-4a96-b96e-05ac8944f1b0",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sklearn.metrics import accuracy_score, precision_score, recall_score, roc_auc_score\n",
|
||||
"\n",
|
||||
"# A Transform Job uploads the results to a given output path in S3, with the name of the input file, with \".out\" added at the end.\n",
|
||||
"output_file_name = (\n",
|
||||
" transform_job.transform_input.data_source.s3_data_source.s3_uri.split(\"/\")[-1] + \".out\"\n",
|
||||
") # Get output file name\n",
|
||||
"output_s3_uri = (\n",
|
||||
" f\"{transform_job.transform_output.s3_output_path}/{output_file_name}\" # Create output S3 URI\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# If a default bucket prefix is specified, append it to the s3 path\n",
|
||||
"if default_bucket_prefix:\n",
|
||||
" output_s3_uri = f\"{default_bucket_prefix}/{output_s3_uri}\"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def split_s3_path(s3_path):\n",
|
||||
" \"\"\"Lightweight method for extracting bucket and object key from S3 uri\"\"\"\n",
|
||||
" path_parts = s3_path.replace(\"s3://\", \"\").split(\"/\")\n",
|
||||
" bucket = path_parts.pop(0)\n",
|
||||
" key = \"/\".join(path_parts)\n",
|
||||
" return bucket, key\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def print_performance_metrics(probs, y, threshold=0.5):\n",
|
||||
" \"\"\"Lightweight method for printing performance metrics\"\"\"\n",
|
||||
"\n",
|
||||
" predictions = (probs >= threshold).astype(int)\n",
|
||||
"\n",
|
||||
" # Compare predictions with the stored target\n",
|
||||
" accuracy = accuracy_score(y, predictions)\n",
|
||||
" precision = precision_score(y, predictions)\n",
|
||||
" recall = recall_score(y, predictions)\n",
|
||||
" roc_auc = roc_auc_score(y, probs)\n",
|
||||
"\n",
|
||||
" print(f\"Accuracy: {accuracy}\")\n",
|
||||
" print(f\"Precision: {precision}\")\n",
|
||||
" print(f\"Recall: {recall}\")\n",
|
||||
" print(f\"ROC AUC: {roc_auc}\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Extract bucket and key separately from uri\n",
|
||||
"res_bucket, res_key = split_s3_path(output_s3_uri)\n",
|
||||
"\n",
|
||||
"# Download Transform Job results\n",
|
||||
"transform_job_result = sagemaker_session.read_s3_file(res_bucket, res_key)\n",
|
||||
"transform_job_result = pd.read_csv(StringIO(transform_job_result), header=None)\n",
|
||||
"\n",
|
||||
"print_performance_metrics(transform_job_result, test_target_column)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "abfe8ade-8de6-487a-9b24-a2afbeaa8559",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Create SageMaker endpoint for real-time inference\n",
|
||||
"To create a SageMaker endpoint we first create an `EndpointConfig`. The endpoint configuration specifies what SageMaker model to use, and what endpoint type. We then use the `EndpointConfig` together with other optional parameters to create a SageMaker Endpoint.\n",
|
||||
"\n",
|
||||
"More information about SageMaker Endpoints can be found [here](https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints.html).\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "056e57bb-9edf-461e-97e8-40c9e5ebf39f",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.resources import Endpoint, EndpointConfig\n",
|
||||
"from sagemaker_core.shapes import ProductionVariant\n",
|
||||
"\n",
|
||||
"endpoint_config_name = \"churn-prediction-endpoint-config\" # Name of endpoint configuration\n",
|
||||
"model_name = customer_churn_model.get_name() # Get name of SageMaker model created in previous step\n",
|
||||
"endpoint_name = \"customer-churn-endpoint\" # Name of SageMaker endpoint\n",
|
||||
"\n",
|
||||
"endpoint_config = EndpointConfig.create(\n",
|
||||
" endpoint_config_name=endpoint_config_name,\n",
|
||||
" production_variants=[\n",
|
||||
" ProductionVariant(\n",
|
||||
" variant_name=\"AllTraffic\",\n",
|
||||
" model_name=model_name,\n",
|
||||
" instance_type=instance_type,\n",
|
||||
" initial_instance_count=1,\n",
|
||||
" )\n",
|
||||
" ],\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"sagemaker_endpoint = Endpoint.create(\n",
|
||||
" endpoint_name=endpoint_name,\n",
|
||||
" endpoint_config_name=endpoint_config.get_name(),\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "635cb615-f4e0-43fa-91d6-0b480a51fd16",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"sagemaker_endpoint.wait_for_status(\n",
|
||||
" target_status=\"InService\"\n",
|
||||
") # Wait for endpoint to become in service"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fd718d54-65ee-4b67-8556-196246356a68",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Test live endpoint - with one sample\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "3fdbd048-014f-418d-8d88-fc329afc7ee6",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Extract one sample payload and convert to string\n",
|
||||
"sample = test_data.sample(1)\n",
|
||||
"sample_payload = sample.to_csv(header=False, index=False).strip()\n",
|
||||
"\n",
|
||||
"# Send sample payload to live endpoint and parse response\n",
|
||||
"res = sagemaker_endpoint.invoke(body=sample_payload, content_type=\"text/csv\")\n",
|
||||
"result = res[\"Body\"].read().decode(\"utf-8\")\n",
|
||||
"result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2ff78e4f-47cd-4f05-ba00-6ded32ad12a8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Test live endpoint - with entire test dataset"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "d66fd20a-7016-4b55-9144-633461059b8b",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Convert entire test dataset to CSV string\n",
|
||||
"sample_payload = test_data.to_csv(header=False, index=False).strip()\n",
|
||||
"\n",
|
||||
"# Send sample payload to live endpoint and parse response\n",
|
||||
"res = sagemaker_endpoint.invoke(body=sample_payload, content_type=\"text/csv\")\n",
|
||||
"result = res[\"Body\"].read().decode(\"utf-8\")\n",
|
||||
"result = result.split(\"\\n\")[:-1]\n",
|
||||
"\n",
|
||||
"# Compute performance metrics\n",
|
||||
"df_result = pd.DataFrame(result).astype(float)\n",
|
||||
"print_performance_metrics(df_result, test_target_column)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0e9d3ad7-4b67-405e-940a-0331fd28811f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Clean up"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "9fb90949-9b35-4e11-8d2a-6053beacb4ba",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"sagemaker_endpoint.delete()\n",
|
||||
"endpoint_config.delete()\n",
|
||||
"customer_churn_model.delete()"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.14"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,818 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# SageMakerCore Inference, Async Inference, and Resource Chaining"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Introductions\n",
|
||||
"\n",
|
||||
"In this notebook, we will walkthrough the process of performing Inference using the SageMakerCore SDK. Additionaly, this notebook will highlight how to create an endpoint using the Resource Chaining feature.\n",
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Resource Chaining\n",
|
||||
"\n",
|
||||
"Resource Chaining is a feature provided by SageMakerCore that aims to reduce the cognitive load for a user when performing operations with the SDK. The idea is to allow users to create an object, for example a `Model` resource object, and pass the object directly as a parameter to some other resource like `EndpointConfig`. An example of this chaining can be seen below:\n",
|
||||
"\n",
|
||||
"```python\n",
|
||||
"key = f'xgboost-iris-{strftime(\"%H-%M-%S\", gmtime())}'\n",
|
||||
"\n",
|
||||
"model = Model.create(...) # Create model object\n",
|
||||
"\n",
|
||||
"endpoint_config = ndpointConfig.create(\n",
|
||||
" endpoint_config_name=key,\n",
|
||||
" production_variants=[\n",
|
||||
" ProductionVariant(\n",
|
||||
" variant_name=key,\n",
|
||||
" initial_instance_count=1,\n",
|
||||
" instance_type='ml.m5.xlarge',\n",
|
||||
" model_name=model # Pass model object directly\n",
|
||||
" )\n",
|
||||
" ]\n",
|
||||
")\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Pre-Requisites"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Install Latest SageMakerCore\n",
|
||||
"All SageMakerCore beta distributions will be released to a private s3 bucket. After being allowlisted, run the cells below to install the latest version of SageMakerCore from `s3://sagemaker-core-beta-artifacts/sagemaker_core-latest.tar.gz`\n",
|
||||
"\n",
|
||||
"Ensure you are using a kernel with python version >=3.8"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Uninstall previous version of sagemaker-core and restart kernel\n",
|
||||
"!pip uninstall sagemaker-core -y"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Install the latest version of sagemaker-core\n",
|
||||
"\n",
|
||||
"!pip install sagemaker-core --upgrade"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Check the version of sagemaker-core\n",
|
||||
"!pip show -v sagemaker-core"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Install Additional Packages"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Install additionall packages\n",
|
||||
"\n",
|
||||
"!pip install -U scikit-learn pandas boto3"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Setup\n",
|
||||
"\n",
|
||||
"Let's start by specifying:\n",
|
||||
"- AWS region.\n",
|
||||
"- The IAM role arn used to give learning and hosting access to your data. Ensure your enviornment has AWS Credentials configured.\n",
|
||||
"- The S3 bucket that you want to use for storing training and model data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sagemaker\n",
|
||||
"\n",
|
||||
"from sagemaker_core.helper.session_helper import get_execution_role, Session\n",
|
||||
"from rich import print\n",
|
||||
"\n",
|
||||
"# Get region, role, bucket\n",
|
||||
"\n",
|
||||
"sagemaker_session = Session()\n",
|
||||
"region = sagemaker_session.boto_region_name\n",
|
||||
"role = get_execution_role()\n",
|
||||
"bucket = sagemaker.Session().default_bucket()\n",
|
||||
"default_bucket_prefix = sagemaker.Session().default_bucket_prefix\n",
|
||||
"print(role)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Load and Prepare Dataset\n",
|
||||
"For this example, we will be using the IRIS data set from `sklearn.datasets` to train our XGBoost container."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sklearn.datasets import load_iris\n",
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"\n",
|
||||
"# Get IRIS Data\n",
|
||||
"\n",
|
||||
"iris = load_iris()\n",
|
||||
"iris_df = pd.DataFrame(iris.data, columns=iris.feature_names)\n",
|
||||
"iris_df[\"target\"] = iris.target"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# Prepare Data\n",
|
||||
"\n",
|
||||
"os.makedirs(\"./data\", exist_ok=True)\n",
|
||||
"\n",
|
||||
"iris_df = iris_df[[\"target\"] + [col for col in iris_df.columns if col != \"target\"]]\n",
|
||||
"\n",
|
||||
"train_data, test_data = train_test_split(iris_df, test_size=0.2, random_state=42)\n",
|
||||
"\n",
|
||||
"train_data.to_csv(\"./data/train.csv\", index=False, header=False)\n",
|
||||
"test_data.to_csv(\"./data/test.csv\", index=False, header=False)\n",
|
||||
"\n",
|
||||
"# Remove the target column from the testing data. We will use this to call invoke_endpoint later\n",
|
||||
"test_data_no_target = test_data.drop(\"target\", axis=1)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Upload Data to S3\n",
|
||||
"In this step, we will upload the train and test data to the S3 bucket configured earlier using `sagemaker_session.default_bucket()`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Upload Data\n",
|
||||
"\n",
|
||||
"prefix = \"DEMO-scikit-iris\"\n",
|
||||
"\n",
|
||||
"# If a default bucket prefix is specified, append it to the s3 path\n",
|
||||
"if default_bucket_prefix:\n",
|
||||
" prefix = f\"{default_bucket_prefix}/{prefix}\"\n",
|
||||
"\n",
|
||||
"TRAIN_DATA = \"train.csv\"\n",
|
||||
"DATA_DIRECTORY = \"data\"\n",
|
||||
"\n",
|
||||
"train_input = sagemaker_session.upload_data(\n",
|
||||
" DATA_DIRECTORY, bucket=bucket, key_prefix=\"{}/{}\".format(prefix, DATA_DIRECTORY)\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"s3_input_path = \"s3://{}/{}/data/{}\".format(bucket, prefix, TRAIN_DATA)\n",
|
||||
"s3_output_path = \"s3://{}/{}/output\".format(bucket, prefix)\n",
|
||||
"\n",
|
||||
"print(s3_input_path)\n",
|
||||
"print(s3_output_path)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Fetch the XGBoost Image URI\n",
|
||||
"In this step, we will fetch the XGBoost Image URI we will use as an input parameter when creating an AWS TrainingJob"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Image name is hardcoded here\n",
|
||||
"# Image name can be programatically got by using sagemaker package and calling image_uris.retrieve\n",
|
||||
"# Since that is a high level abstraction that has multiple dependencies, the image URIs functionalities will live in sagemaker (V2)\n",
|
||||
"\n",
|
||||
"image = \"433757028032.dkr.ecr.us-west-2.amazonaws.com/xgboost:latest\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Train XGBoost Image using IRIS Data\n",
|
||||
"\n",
|
||||
"Next, we will the SageMakerCore `TrainingJob.create()` to start a training job for an XGBoost Image using IRIS data and wait for it to complete."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Create TrainingJob with SageMakerCore\n",
|
||||
"\n",
|
||||
"import time\n",
|
||||
"from sagemaker_core.resources import TrainingJob\n",
|
||||
"from sagemaker_core.shapes import (\n",
|
||||
" AlgorithmSpecification,\n",
|
||||
" Channel,\n",
|
||||
" DataSource,\n",
|
||||
" S3DataSource,\n",
|
||||
" OutputDataConfig,\n",
|
||||
" ResourceConfig,\n",
|
||||
" StoppingCondition,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"job_name_v3 = \"xgboost-iris-\" + time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime())\n",
|
||||
"\n",
|
||||
"training_job = TrainingJob.create(\n",
|
||||
" training_job_name=job_name_v3,\n",
|
||||
" hyper_parameters={\n",
|
||||
" \"objective\": \"multi:softmax\",\n",
|
||||
" \"num_class\": \"3\",\n",
|
||||
" \"num_round\": \"10\",\n",
|
||||
" \"eval_metric\": \"merror\",\n",
|
||||
" },\n",
|
||||
" algorithm_specification=AlgorithmSpecification(\n",
|
||||
" training_image=image, training_input_mode=\"File\"\n",
|
||||
" ),\n",
|
||||
" role_arn=role,\n",
|
||||
" input_data_config=[\n",
|
||||
" Channel(\n",
|
||||
" channel_name=\"train\",\n",
|
||||
" content_type=\"csv\",\n",
|
||||
" compression_type=\"None\",\n",
|
||||
" record_wrapper_type=\"None\",\n",
|
||||
" data_source=DataSource(\n",
|
||||
" s3_data_source=S3DataSource(\n",
|
||||
" s3_data_type=\"S3Prefix\",\n",
|
||||
" s3_uri=s3_input_path,\n",
|
||||
" s3_data_distribution_type=\"FullyReplicated\",\n",
|
||||
" )\n",
|
||||
" ),\n",
|
||||
" )\n",
|
||||
" ],\n",
|
||||
" output_data_config=OutputDataConfig(s3_output_path=s3_output_path),\n",
|
||||
" resource_config=ResourceConfig(\n",
|
||||
" instance_type=\"ml.m4.xlarge\", instance_count=1, volume_size_in_gb=30\n",
|
||||
" ),\n",
|
||||
" stopping_condition=StoppingCondition(max_runtime_in_seconds=600),\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"training_job.wait()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Create Endpoint Using Resource Chaining\n",
|
||||
"\n",
|
||||
"In the following cells, we will walkthrough the process of creating an Endpoint using the Resource Chaining feature of SageMakerCore. Resource Chaining aims to reduce the cognitive load for a user by autoresolving necessary attributes when chaing resource objects together during operations.\n",
|
||||
"\n",
|
||||
"1. First, we will create a `Model` using the model data from the training job in the previous step.\n",
|
||||
"2. We will create an `EndpointConfig` and pass the `Model` object directly as a parameter. SageMakerCore will autoresolve the neccessary attributes from the `Model` object.\n",
|
||||
"3. We will create an `Endpoint` using the `EndpointConfig` object as a parameter. SageMakerCore will autoresolve the neccessary attributes from the `EndpointConfig` object.\n",
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Create and Wait for Endpoint"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Create a `Model` by specifying the `image` and `model_data_url`. For the `model_data_url` we will use the S3 path of the model output from the TrainingJob we performed previously.\n",
|
||||
"\n",
|
||||
"Notice that we are able to set the `model_data_url` directly by referencing the `s3_model_artifacts` from the nested `ModelArtifacts` object attribute. This is possible due to SageMakerCore's object-oriented programming experience. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Class Definitions example:\n",
|
||||
"\n",
|
||||
"```python\n",
|
||||
"class TrainingJob(Base):\n",
|
||||
" ...\n",
|
||||
" model_artifacts: Optional[ModelArtifacts] = Unassigned()\n",
|
||||
"\n",
|
||||
"class ModelArtifacts(Base):\n",
|
||||
" s3_model_artifacts: str\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"A user can then reference attributes for nested objects like:\n",
|
||||
"\n",
|
||||
"```python\n",
|
||||
"model_data_url = training_job.model_artifacts.s3_model_artifacts\n",
|
||||
"```\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.shapes import ContainerDefinition, ProductionVariant\n",
|
||||
"from sagemaker_core.resources import Model, EndpointConfig, Endpoint\n",
|
||||
"from time import gmtime, strftime\n",
|
||||
"\n",
|
||||
"# Get model_data_url from training_job object\n",
|
||||
"model_data_url = training_job.model_artifacts.s3_model_artifacts\n",
|
||||
"\n",
|
||||
"key = f'xgboost-iris-{strftime(\"%H-%M-%S\", gmtime())}'\n",
|
||||
"print(\"Key:\", key)\n",
|
||||
"\n",
|
||||
"model = Model.create(\n",
|
||||
" model_name=key,\n",
|
||||
" primary_container=ContainerDefinition(\n",
|
||||
" image=image,\n",
|
||||
" model_data_url=model_data_url,\n",
|
||||
" ),\n",
|
||||
" execution_role_arn=role,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Create the `Endpoint` and wait for it to be `InService`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"endpoint_config = EndpointConfig.create(\n",
|
||||
" endpoint_config_name=key,\n",
|
||||
" production_variants=[\n",
|
||||
" ProductionVariant(\n",
|
||||
" variant_name=key,\n",
|
||||
" initial_instance_count=1,\n",
|
||||
" instance_type=\"ml.m5.xlarge\",\n",
|
||||
" model_name=model, # Pass `Model`` object created above\n",
|
||||
" )\n",
|
||||
" ],\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"endpoint: Endpoint = Endpoint.create(\n",
|
||||
" endpoint_name=key,\n",
|
||||
" endpoint_config_name=endpoint_config, # Pass `EndpointConfig` object created above\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"endpoint.wait_for_status(\"InService\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Endpoint Invoke\n",
|
||||
"\n",
|
||||
"The below cells demonstrates how an endpoint would be invoked synchronously in SageMakerCore using `endpoint.invoke()` or `endpoint.inoke_with_response_stream()`. \n",
|
||||
"\n",
|
||||
"In these examples, we are using CSV data to train the model and to invoking the endpoint. We will rely on the `CSVSerializer` and `CSVDeserializer` from the the sagemaker-python-sdk to assist with serilizing and deserializing the invoke input and output."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker.base_serializers import CSVSerializer\n",
|
||||
"from sagemaker.deserializers import CSVDeserializer\n",
|
||||
"\n",
|
||||
"deserializer = CSVDeserializer()\n",
|
||||
"serializer = CSVSerializer()\n",
|
||||
"\n",
|
||||
"invoke_result = endpoint.invoke(\n",
|
||||
" body=serializer.serialize(test_data_no_target),\n",
|
||||
" content_type=\"text/csv\",\n",
|
||||
" accept=\"text/csv\",\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"deserialized_result = deserializer.deserialize(invoke_result[\"Body\"], invoke_result[\"ContentType\"])\n",
|
||||
"\n",
|
||||
"print(\"Endpoint Response:\", deserialized_result)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Endpoint Invoke With Response Stream"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def deserialise(response):\n",
|
||||
" return [res_part for res_part in response[\"Body\"]]\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"invoke_result = endpoint.invoke_with_response_stream(\n",
|
||||
" body=serializer.serialize(test_data_no_target),\n",
|
||||
" content_type=\"text/csv\",\n",
|
||||
" accept=\"application/csv\",\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(\"Endpoint Stream Response:\", deserialise(invoke_result))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Create Endpoint for Async Invoke\n",
|
||||
"\n",
|
||||
"Now that we have gone through the process of creating and invoking endpoint synchronously using SageMakerCore. In the next section, we will create a new endpoint for asynchronous invocations and call `endpoint.invoke_async()`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Download the Input Files and Pre-Trained Model tar.gz from S3"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import boto3\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# Download the Input files and model from S3 bucket\n",
|
||||
"os.makedirs(\"./input\", exist_ok=True)\n",
|
||||
"os.makedirs(\"./model\", exist_ok=True)\n",
|
||||
"\n",
|
||||
"s3 = boto3.client(\"s3\")\n",
|
||||
"for key in s3.list_objects(\n",
|
||||
" Bucket=f\"sagemaker-example-files-prod-{region}\",\n",
|
||||
" Prefix=\"models/async-inference/input-files/\",\n",
|
||||
")[\"Contents\"]:\n",
|
||||
" s3.download_file(\n",
|
||||
" f\"sagemaker-example-files-prod-{region}\",\n",
|
||||
" key[\"Key\"],\n",
|
||||
" \"input/\" + key[\"Key\"].split(\"/\")[-1],\n",
|
||||
" )\n",
|
||||
"s3.download_file(\n",
|
||||
" f\"sagemaker-example-files-prod-{region}\",\n",
|
||||
" \"models/async-inference/demo-xgboost-model.tar.gz\",\n",
|
||||
" \"model/demo-xgboost-model.tar.gz\",\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Upload Data to S3\n",
|
||||
"In this step, we will upload the input and model data to the S3 bucket configured earlier using `sagemaker_session.default_bucket()` and set the `model_url` variable that we will use to create a `Model` resource object."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Upload the model to S3 bucket\n",
|
||||
"bucket_prefix = \"async-inference-demo\"\n",
|
||||
"default_bucket_prefix = sagemaker_session.default_bucket_prefix\n",
|
||||
"\n",
|
||||
"# If a default bucket prefix is specified, append it to the s3 path\n",
|
||||
"if default_bucket_prefix:\n",
|
||||
" bucket_prefix = f\"{default_bucket_prefix}/{bucket_prefix}\"\n",
|
||||
"\n",
|
||||
"bucket = sagemaker_session.default_bucket()\n",
|
||||
"\n",
|
||||
"model_s3_key = f\"{bucket_prefix}/demo-xgboost-model.tar.gz\"\n",
|
||||
"async_s3_output_path = f\"s3://{bucket}/{bucket_prefix}/output\"\n",
|
||||
"\n",
|
||||
"model_url = sagemaker_session.upload_data(\"model/demo-xgboost-model.tar.gz\", bucket, bucket_prefix)\n",
|
||||
"\n",
|
||||
"print(f\"Uploading Model to {model_url}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Create and Wait for Endpoint"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Create a `Model` by specifying the `image` and `model_data_url`. For the `model_data_url` we will use the S3 path of the pretrained model uploaded previously."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"key = f'xgboost-iris-{strftime(\"%H-%M-%S\", gmtime())}'\n",
|
||||
"print(\"Key:\", key)\n",
|
||||
"\n",
|
||||
"async_model = Model.create(\n",
|
||||
" model_name=key,\n",
|
||||
" primary_container=ContainerDefinition(\n",
|
||||
" image=image,\n",
|
||||
" model_data_url=model_url,\n",
|
||||
" ),\n",
|
||||
" execution_role_arn=role,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Create the `Endpoint` and wait for it to be `InService`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.shapes import (\n",
|
||||
" ProductionVariant,\n",
|
||||
" AsyncInferenceConfig,\n",
|
||||
" AsyncInferenceOutputConfig,\n",
|
||||
" AsyncInferenceClientConfig,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"async_endpoint_config = EndpointConfig.create(\n",
|
||||
" endpoint_config_name=key,\n",
|
||||
" production_variants=[\n",
|
||||
" ProductionVariant(\n",
|
||||
" variant_name=\"variant1\",\n",
|
||||
" model_name=async_model,\n",
|
||||
" instance_type=\"ml.m5.xlarge\",\n",
|
||||
" initial_instance_count=1,\n",
|
||||
" )\n",
|
||||
" ],\n",
|
||||
" async_inference_config=AsyncInferenceConfig(\n",
|
||||
" output_config=AsyncInferenceOutputConfig(s3_output_path=async_s3_output_path),\n",
|
||||
" client_config=AsyncInferenceClientConfig(max_concurrent_invocations_per_instance=4),\n",
|
||||
" ),\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"async_endpoint = Endpoint.create(endpoint_name=key, endpoint_config_name=async_endpoint_config)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"async_endpoint.wait_for_status(\"InService\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Upload the Async Invoke Payload\n",
|
||||
"To invoke an endpoint asynchronously, we first must upload the request payload to S3."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def upload_file(input_location):\n",
|
||||
" prefix = f\"{bucket_prefix}/input\"\n",
|
||||
" return sagemaker_session.upload_data(\n",
|
||||
" input_location,\n",
|
||||
" bucket=sagemaker_session.default_bucket(),\n",
|
||||
" key_prefix=prefix,\n",
|
||||
" extra_args={\"ContentType\": \"text/libsvm\"},\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"input_path = \"input/test_point_0.libsvm\"\n",
|
||||
"input_s3_path = upload_file(input_path)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Endpoint Async Invoke\n",
|
||||
"\n",
|
||||
"Call `endpoint.invoke_async()` using the s3 path of the invoke request payload and store the \"OutputLocation\" of from the response."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response = async_endpoint.invoke_async(input_location=input_s3_path)\n",
|
||||
"output_location = response[\"OutputLocation\"]\n",
|
||||
"print(output_location)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Check the Output Location\n",
|
||||
"Check the output location from the `endpoint.invoke_async()` response to get the async inference results."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import urllib, time\n",
|
||||
"from botocore.exceptions import ClientError\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def get_output(output_location):\n",
|
||||
" output_url = urllib.parse.urlparse(output_location)\n",
|
||||
" bucket = output_url.netloc\n",
|
||||
" key = output_url.path[1:]\n",
|
||||
" while True:\n",
|
||||
" try:\n",
|
||||
" return sagemaker_session.read_s3_file(\n",
|
||||
" bucket=output_url.netloc, key_prefix=output_url.path[1:]\n",
|
||||
" )\n",
|
||||
" except ClientError as e:\n",
|
||||
" if e.response[\"Error\"][\"Code\"] == \"NoSuchKey\":\n",
|
||||
" print(\"waiting for output...\")\n",
|
||||
" time.sleep(2)\n",
|
||||
" continue\n",
|
||||
" raise"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"output = get_output(output_location)\n",
|
||||
"print(f\"Output: {output}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Delete All SageMaker Resources\n",
|
||||
"The following code block will call the delete() method for any SageMaker Core Resources created during the execution of this notebook which were assigned to local or global variables. If you created any additional deleteable resources without assigning the returning object to a unique variable, you will need to delete the resource manually by doing something like:\n",
|
||||
"\n",
|
||||
"```python\n",
|
||||
"resource = Resource.get(\"resource-name\")\n",
|
||||
"resource.delete()\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Delete any sagemaker core resource objects created in this notebook\n",
|
||||
"def delete_all_sagemaker_resources():\n",
|
||||
" all_objects = list(locals().values()) + list(globals().values())\n",
|
||||
" deletable_objects = [\n",
|
||||
" obj\n",
|
||||
" for obj in all_objects\n",
|
||||
" if hasattr(obj, \"delete\") and obj.__class__.__module__ == \"sagemaker_core.main.resources\"\n",
|
||||
" ]\n",
|
||||
"\n",
|
||||
" for obj in deletable_objects:\n",
|
||||
" obj.delete()\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"delete_all_sagemaker_resources()"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.14"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
@@ -0,0 +1,600 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Intelligent Defaults and Logging Configuration in SageMakerCore"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"## Introductions\n",
|
||||
"\n",
|
||||
"In this notebook, we will walkthrough the setup and usage of intelligent defaults in the SageMakerCore SDK. Additionally, this notebook contains a section with the steps required for configuring logging levels to assist in the debugging of issues that arise while using the SDK.\n",
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Intelligent Defaults\n",
|
||||
"\n",
|
||||
"Intelligent Defaults is a feature provided by the SageMakerCore SDK to assist users in defining default values to be auto populated into the AWS API Request parameters. For example, if a user/admin wants all of their AWS Resources to use a specific VPC Config during creation this can be defined in the Intelligent Defaults Configs. Intelligent Defaults supports:\n",
|
||||
"1. GlobalDefaults - default values applied across SageMaker API calls\n",
|
||||
"2. Resource Specific Defaults - defaults applied only when creating a specific resource\n",
|
||||
"\n",
|
||||
"An Example of the strucuture of the Intelligent Defaults Config is below:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```json\n",
|
||||
"{\n",
|
||||
" \"SchemaVesion\": \"1.0\",\n",
|
||||
" \"SageMaker\": {\n",
|
||||
" \"PythonSDK\": {\n",
|
||||
" \"Resources\": {\n",
|
||||
" \"GlobalDefaults\": {\n",
|
||||
" \"vpc_config\": {\n",
|
||||
" \"security_group_ids\": [\n",
|
||||
" \"sg-xxxxxxxxxxxxxxxxx\" // Replace with security group id\n",
|
||||
" ],\n",
|
||||
" \"subnets\": [\n",
|
||||
" \"subnet-xxxxxxxxxxxxxxxxx\", // Replace with subnet id\n",
|
||||
" \"subnet-xxxxxxxxxxxxxxxxx\" // Replace with subnet id\n",
|
||||
" ]\n",
|
||||
" }\n",
|
||||
" // ...\n",
|
||||
" },\n",
|
||||
" \"TrainingJob\": {\n",
|
||||
" \"role_arn\": \"arn:aws:xxxxxxxxxxx:role/xxxxx\", // Replace with role arn\n",
|
||||
" \"output_data_config\": {\n",
|
||||
" \"s3_output_path\": \"s3://xxxxxxxxxxx\", // Replace with S3 URI\n",
|
||||
" },\n",
|
||||
" // ...\n",
|
||||
" }\n",
|
||||
" }\n",
|
||||
" }\n",
|
||||
" }\n",
|
||||
"}\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Logging Levels\n",
|
||||
"\n",
|
||||
"To assist in debugging issues originating within the SDK, SageMakerCore provides a simple utility method - `configure_logging()`\n",
|
||||
"\n",
|
||||
"To set the logging level users have 2 options:\n",
|
||||
"1. Pass a string parameter to utility method with log level they desire - `configure_logging(\"DEBUG\")`\n",
|
||||
"2. Set the `LOG_LEVEL=INFO` environment variable and call `configure_logging()` without a parameter\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"In a later section in this notebook, we will walk through an example of how these options would look like in practice for a user."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Pre-Requisites"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Install Latest SageMakerCore\n",
|
||||
"All SageMakerCore beta distributions will be released to a private s3 bucket. After being allowlisted, run the cells below to install the latest version of SageMakerCore from `s3://sagemaker-core-beta-artifacts/sagemaker_core-latest.tar.gz`\n",
|
||||
"\n",
|
||||
"Ensure you are using a kernel with python version >=3.8"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Uninstall previous version of sagemaker_core and restart kernel\n",
|
||||
"!pip uninstall sagemaker-core -y"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Install the latest version of sagemaker_core\n",
|
||||
"\n",
|
||||
"!pip install sagemaker-core --upgrade"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Check the version of sagemaker_core\n",
|
||||
"!pip show -v sagemaker-core"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Install Additional Packages"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Install additionall packages\n",
|
||||
"\n",
|
||||
"!pip install -U scikit-learn pandas boto3"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Setup\n",
|
||||
"\n",
|
||||
"Let's start by specifying:\n",
|
||||
"- AWS region.\n",
|
||||
"- The IAM role arn used to give learning and hosting access to your data. Ensure your enviornment has AWS Credentials configured.\n",
|
||||
"- The S3 bucket that you want to use for storing training and model data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sagemaker\n",
|
||||
"\n",
|
||||
"from sagemaker_core.helper.session_helper import Session, get_execution_role\n",
|
||||
"from rich import print\n",
|
||||
"\n",
|
||||
"# Get region, role, bucket\n",
|
||||
"\n",
|
||||
"sagemaker_session = Session()\n",
|
||||
"region = sagemaker_session.boto_region_name\n",
|
||||
"role = get_execution_role()\n",
|
||||
"bucket = sagemaker.Session().default_bucket()\n",
|
||||
"default_bucket_prefix = sagemaker.Session().default_bucket_prefix\n",
|
||||
"print(role)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Load and Prepare Dataset\n",
|
||||
"For this example, we will be using the IRIS data set from `sklearn.datasets` to train our XGBoost container."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sklearn.datasets import load_iris\n",
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"\n",
|
||||
"# Get IRIS Data\n",
|
||||
"\n",
|
||||
"iris = load_iris()\n",
|
||||
"iris_df = pd.DataFrame(iris.data, columns=iris.feature_names)\n",
|
||||
"iris_df[\"target\"] = iris.target"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# Prepare Data\n",
|
||||
"\n",
|
||||
"os.makedirs(\"./data\", exist_ok=True)\n",
|
||||
"\n",
|
||||
"iris_df = iris_df[[\"target\"] + [col for col in iris_df.columns if col != \"target\"]]\n",
|
||||
"\n",
|
||||
"train_data, test_data = train_test_split(iris_df, test_size=0.2, random_state=42)\n",
|
||||
"\n",
|
||||
"train_data.to_csv(\"./data/train.csv\", index=False, header=False)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Upload Data to S3\n",
|
||||
"In this step, we will upload the train and test data to the S3 bucket configured earlier using `sagemaker_session.default_bucket()`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Upload Data\n",
|
||||
"\n",
|
||||
"prefix = \"DEMO-scikit-iris\"\n",
|
||||
"\n",
|
||||
"# If a default bucket prefix is specified, append it to the s3 path\n",
|
||||
"if default_bucket_prefix:\n",
|
||||
" prefix = f\"{default_bucket_prefix}/{prefix}\"\n",
|
||||
"\n",
|
||||
"TRAIN_DATA = \"train.csv\"\n",
|
||||
"DATA_DIRECTORY = \"data\"\n",
|
||||
"\n",
|
||||
"train_input = sagemaker_session.upload_data(\n",
|
||||
" DATA_DIRECTORY, bucket=bucket, key_prefix=\"{}/{}\".format(prefix, DATA_DIRECTORY)\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"s3_input_path = \"s3://{}/{}/data/{}\".format(bucket, prefix, TRAIN_DATA)\n",
|
||||
"s3_output_path = \"s3://{}/{}/output\".format(bucket, prefix)\n",
|
||||
"\n",
|
||||
"print(s3_input_path)\n",
|
||||
"print(s3_output_path)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Fetch the XGBoost Image URI\n",
|
||||
"In this step, we will fetch the XGBoost Image URI we will use as an input parameter when creating an AWS TrainingJob"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Image name is hardcoded here\n",
|
||||
"# Image name can be programatically got by using sagemaker package and calling image_uris.retrieve\n",
|
||||
"# Since that is a high level abstraction that has multiple dependencies, the image URIs functionalities will live in sagemaker (V2)\n",
|
||||
"\n",
|
||||
"image = \"433757028032.dkr.ecr.us-west-2.amazonaws.com/xgboost:latest\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Intelligent Defaults"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Create Intelligent Defaults JSON\n",
|
||||
"\n",
|
||||
"In order for SageMakerCore to pick up the Intelligent Defaults Configs to populate API calls, we first must create the json config file and set the `SAGEMAKER_CORE_ADMIN_CONFIG_OVERRIDE` enviornment variable.\n",
|
||||
"\n",
|
||||
"Below we will create the config file at `data/defaults.json` and assign this path to the `SAGEMAKER_CORE_ADMIN_CONFIG_OVERRIDE` enviornment variable."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import json\n",
|
||||
"\n",
|
||||
"DEFAULTS_CONTENT = {\n",
|
||||
" \"SchemaVesion\": \"1.0\",\n",
|
||||
" \"SageMaker\": {\n",
|
||||
" \"PythonSDK\": {\n",
|
||||
" \"Resources\": {\n",
|
||||
" \"GlobalDefaults\": {\n",
|
||||
" \"vpc_config\": {\n",
|
||||
" \"security_group_ids\": [\n",
|
||||
" \"sg-xxxxxxxxxxxxxxxxx\" # Replace with security group id\n",
|
||||
" ],\n",
|
||||
" \"subnets\": [\n",
|
||||
" \"subnet-xxxxxxxxxxxxxxxxx\", # Replace with subnet id\n",
|
||||
" \"subnet-xxxxxxxxxxxxxxxxx\", # Replace with subnet id\n",
|
||||
" ],\n",
|
||||
" }\n",
|
||||
" },\n",
|
||||
" \"TrainingJob\": {\n",
|
||||
" \"role_arn\": role,\n",
|
||||
" \"output_data_config\": {\"s3_output_path\": s3_output_path},\n",
|
||||
" },\n",
|
||||
" }\n",
|
||||
" }\n",
|
||||
" },\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"path_to_defaults = os.path.join(DATA_DIRECTORY, \"defaults.json\")\n",
|
||||
"with open(os.path.join(DATA_DIRECTORY, \"defaults.json\"), \"w\") as f:\n",
|
||||
" json.dump(DEFAULTS_CONTENT, f, indent=4)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# Setting path of Config file in environment variable\n",
|
||||
"os.environ[\"SAGEMAKER_CORE_ADMIN_CONFIG_OVERRIDE\"] = path_to_defaults"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Using GlobalDefaults\n",
|
||||
"In the below example, a `Cluster` resource will be created using the `vpc_config` defined under the `SageMaker.PythonSDK.Resources.GlobalDefaults`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import time\n",
|
||||
"from sagemaker_core.resources import Cluster\n",
|
||||
"from sagemaker_core.shapes import (\n",
|
||||
" ClusterInstanceGroupSpecification,\n",
|
||||
" ClusterLifeCycleConfig,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"cluster_name_v3 = \"xgboost-cluster-\" + time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime())\n",
|
||||
"\n",
|
||||
"# Use vpc_config from Intelligent Defaults JSON config file under the SageMaker.PythonSDK.Resources.GlobalDefaults key\n",
|
||||
"cluster = Cluster.create(\n",
|
||||
" cluster_name=cluster_name_v3,\n",
|
||||
" instance_groups=[\n",
|
||||
" ClusterInstanceGroupSpecification(\n",
|
||||
" instance_count=1,\n",
|
||||
" instance_group_name=\"instance-group-11\",\n",
|
||||
" instance_type=\"ml.m5.4xlarge\",\n",
|
||||
" life_cycle_config=ClusterLifeCycleConfig(\n",
|
||||
" source_s3_uri=s3_input_path, on_create=\"dothis\"\n",
|
||||
" ),\n",
|
||||
" execution_role=role,\n",
|
||||
" )\n",
|
||||
" ],\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"cluster.wait_for_status(\"InService\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Using Resource Defaults\n",
|
||||
"In the below example, a `TrainingJob` resource will be created using the `role` and `output_data_config` defined under the `SageMaker.Python.Resources.TrainingJob` key. \n",
|
||||
"\n",
|
||||
"Note: Because `TrainingJob` also excepts a `vpc_config` parameter, the `vpc_config` parameter will be populated from the `GlobalDefaults`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import time\n",
|
||||
"from sagemaker_core.resources import TrainingJob\n",
|
||||
"from sagemaker_core.shapes import (\n",
|
||||
" AlgorithmSpecification,\n",
|
||||
" Channel,\n",
|
||||
" DataSource,\n",
|
||||
" S3DataSource,\n",
|
||||
" ResourceConfig,\n",
|
||||
" StoppingCondition,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"job_name_v3 = \"xgboost-iris-\" + time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime())\n",
|
||||
"\n",
|
||||
"# Use role and output_data_config from Intelligent Defaults JSON config file under the SageMaker.PythonSDK.Resources.TrainingJob key\n",
|
||||
"# Use vpc_config from Intelligent Defaults JSON config file under the SageMaker.PythonSDK.Resources.GlobalDefaults key\n",
|
||||
"\n",
|
||||
"training_job = TrainingJob.create(\n",
|
||||
" training_job_name=job_name_v3,\n",
|
||||
" hyper_parameters={\n",
|
||||
" \"objective\": \"multi:softmax\",\n",
|
||||
" \"num_class\": \"3\",\n",
|
||||
" \"num_round\": \"10\",\n",
|
||||
" \"eval_metric\": \"merror\",\n",
|
||||
" },\n",
|
||||
" algorithm_specification=AlgorithmSpecification(\n",
|
||||
" training_image=image, training_input_mode=\"File\"\n",
|
||||
" ),\n",
|
||||
" input_data_config=[\n",
|
||||
" Channel(\n",
|
||||
" channel_name=\"train\",\n",
|
||||
" content_type=\"csv\",\n",
|
||||
" compression_type=\"None\",\n",
|
||||
" record_wrapper_type=\"None\",\n",
|
||||
" data_source=DataSource(\n",
|
||||
" s3_data_source=S3DataSource(\n",
|
||||
" s3_data_type=\"S3Prefix\",\n",
|
||||
" s3_uri=s3_input_path,\n",
|
||||
" s3_data_distribution_type=\"FullyReplicated\",\n",
|
||||
" )\n",
|
||||
" ),\n",
|
||||
" )\n",
|
||||
" ],\n",
|
||||
" resource_config=ResourceConfig(\n",
|
||||
" instance_type=\"ml.m4.xlarge\", instance_count=1, volume_size_in_gb=30\n",
|
||||
" ),\n",
|
||||
" stopping_condition=StoppingCondition(max_runtime_in_seconds=600),\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Configure Logging Levels\n",
|
||||
"\n",
|
||||
"Below are 2 examples of how a SageMakerCore user could configure the logging level of the SDK to assist with debugging.\n",
|
||||
"\n",
|
||||
"To set the logging level users have 2 options:\n",
|
||||
"1. Pass a string parameter to utility method with log level they desire - `configure_logging(\"DEBUG\")`\n",
|
||||
"2. Set the `LOG_LEVEL=INFO` environment variable and call `configure_logging()` without a parameter"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Configure Logging with Parameter"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Setting log_level to DEBUG using configure_logging with string parameter\n",
|
||||
"from sagemaker_core.main.utils import configure_logging\n",
|
||||
"\n",
|
||||
"configure_logging(\"DEBUG\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Get TrainingJob with DEBUG log_level\n",
|
||||
"from sagemaker_core.resources import TrainingJob\n",
|
||||
"\n",
|
||||
"training_job = TrainingJob.get(job_name_v3)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Configure Logging with Enviornment Variable"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Setting log_level to INFO using env variable\n",
|
||||
"!export LOG_LEVEL=INFO\n",
|
||||
"\n",
|
||||
"configure_logging()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# List TrainingJobs with INFO log_level\n",
|
||||
"training_job = TrainingJob.get(job_name_v3)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Delete All SageMaker Resources\n",
|
||||
"The following code block will call the delete() method for any SageMaker Core Resources created during the execution of this notebook which were assigned to local or global variables. If you created any additional deleteable resources without assigning the returning object to a unique variable, you will need to delete the resource manually by doing something like:\n",
|
||||
"\n",
|
||||
"```python\n",
|
||||
"resource = Resource.get(\"resource-name\")\n",
|
||||
"resource.delete()\n",
|
||||
"```\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Delete any sagemaker core resource objects created in this notebook\n",
|
||||
"def delete_all_sagemaker_resources():\n",
|
||||
" all_objects = list(locals().values()) + list(globals().values())\n",
|
||||
" deletable_objects = [\n",
|
||||
" obj\n",
|
||||
" for obj in all_objects\n",
|
||||
" if hasattr(obj, \"delete\") and obj.__class__.__module__ == \"sagemaker_core.main.resources\"\n",
|
||||
" ]\n",
|
||||
"\n",
|
||||
" for obj in deletable_objects:\n",
|
||||
" obj.delete()\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"delete_all_sagemaker_resources()"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "py3.10.14",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.14"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
+577
@@ -0,0 +1,577 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b02083d2-55a3-41e6-aed5-8ff633af6a59",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Get started with SageMaker\n",
|
||||
"\n",
|
||||
"This notebok is part of [Introductory blog on SageMaker Core](https://aws.amazon.com/blogs/machine-learning/introducing-sagemaker-core-a-new-object-oriented-python-sdk-for-amazon-sagemaker/).\n",
|
||||
"\n",
|
||||
"In this notebook you'll learn how SageMaker can be used to:\n",
|
||||
"\n",
|
||||
"1. Preprocess (and optionally explore) a dataset\n",
|
||||
"2. Train an XGBoost classifier for customer churn prediction, using a managed job with SageMaker Training, using a managed image.\n",
|
||||
"3. Create a managed real-time SageMaker endpoint.\n",
|
||||
"\n",
|
||||
"All SageMaker resources are created using the SageMaker Core SDK. You can find more information about sagemaker-core [here](https://sagemaker-core.readthedocs.io/en/latest/)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "0489c209-9628-4ddf-a849-8b973234126a",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%pip install --upgrade pip -q\n",
|
||||
"%pip install sagemaker-core -q"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "683f28ed-6c66-48c2-a24a-378764271f7e",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import time\n",
|
||||
"import io\n",
|
||||
"from datetime import datetime\n",
|
||||
"import pandas as pd\n",
|
||||
"from sagemaker_core.helper.session_helper import Session, get_execution_role\n",
|
||||
"from io import StringIO\n",
|
||||
"import pandas as pd\n",
|
||||
"from sagemaker_core.main.shapes import (\n",
|
||||
" ProcessingInput,\n",
|
||||
" ProcessingResources,\n",
|
||||
" AppSpecification,\n",
|
||||
" ProcessingS3Input,\n",
|
||||
" ProcessingOutputConfig\n",
|
||||
")\n",
|
||||
"from sagemaker_core.shapes import (\n",
|
||||
" ProcessingResources,\n",
|
||||
" ProcessingClusterConfig,\n",
|
||||
" ProcessingOutput,\n",
|
||||
" ProcessingS3Output,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"from sagemaker_core.resources import ProcessingJob"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "314d959f-8c4a-4996-b167-24958d53417a",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Set up region, role and bucket parameters used throughout the notebook.\n",
|
||||
"sagemaker_session = Session()\n",
|
||||
"region = sagemaker_session.boto_region_name\n",
|
||||
"role = get_execution_role()\n",
|
||||
"bucket = sagemaker_session.default_bucket()\n",
|
||||
"\n",
|
||||
"print(f\"AWS region: {region}\")\n",
|
||||
"print(f\"Execution role: {role}\")\n",
|
||||
"print(f\"Default S3 bucket: {bucket}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "93b62649-4d89-4b6e-9f18-a48aa0bb27a4",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Preprocess dataset\n",
|
||||
"We'll use a synthetic dataset that AWS provides for customer churn prediction.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f347ca27-68a3-41a1-998c-434d39f8245a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"<div class=\"alert alert-block alert-info\">\n",
|
||||
"<b>NOTE:</b> This sample doesn't perform any exploratory data anlysis since how to preprocess the dataset is already known.\n",
|
||||
" \n",
|
||||
"If you're interested in how to perform exploratory analysis, there's a section in the documentation for the sagemaker-python-sdk available that explores the dataset, [here](https://sagemaker-examples.readthedocs.io/en/latest/introduction_to_applying_machine_learning/xgboost_customer_churn/xgboost_customer_churn.html).\n",
|
||||
"</div>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "38f74911-14c9-4ef8-bd18-db9664de7dcf",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Upload the processing code to S3\n",
|
||||
"\n",
|
||||
"The pre-processing code is already available in the current directory with the name `preprocess.py`. Have a look at the code to understand the processing logic."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "ed040f38-1e1b-4310-b5a6-2f49421952d9",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pre_processing_code_s3_uri = sagemaker_session.upload_data(\"preprocess.py\", key_prefix=\"sagemaker-core-intro-blog/processing/code\")\n",
|
||||
"pre_processing_code_s3_uri"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "77f89929-0274-4248-9ac2-dadaac217222",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Create S3 variables for holding the processed data\n",
|
||||
"\n",
|
||||
"Create S3 variables for holding the processed data (train, validation and test)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "930789de-cbfc-4ca6-b7a6-ad49d6f888da",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"processed_train_data_uri = f\"s3://{bucket}/sagemaker-core-intro-blog/processing/output/train\"\n",
|
||||
"processed_validation_data_uri = f\"s3://{bucket}/sagemaker-core-intro-blog/processing/output/validation\"\n",
|
||||
"processed_test_data_uri = f\"s3://{bucket}/sagemaker-core-intro-blog/processing/output/test\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "880c9f13-0c60-46cf-bc98-59c9141a9384",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"## Create processing job\n",
|
||||
"\n",
|
||||
"Below code submits a sagemaker processing job.\n",
|
||||
"\n",
|
||||
"1. ProcessingResources provides processing cluster details.\n",
|
||||
"2. AppSpecification provides processing container details. Here SKlearn processing container provided by sagemaker is used.\n",
|
||||
"3. Two objects of ProcessingInput specifying code and input data locations and configurations.\n",
|
||||
"4. ProcessingOutputConfig provides details on where the processed data will be stored."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cce99f6a-934e-47a0-8033-b216d0e1a2a8",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Initialize a ProcessingJob resource\n",
|
||||
"current_timestamp = datetime.now()\n",
|
||||
"formatted_timestamp = current_timestamp.strftime(\"%Y-%m-%d-%H-%M-%S\")\n",
|
||||
"\n",
|
||||
"processing_job = ProcessingJob.create(\n",
|
||||
" processing_job_name=f\"sagemaker-core-data-prep-{formatted_timestamp}\",\n",
|
||||
" processing_resources=ProcessingResources(\n",
|
||||
" cluster_config=ProcessingClusterConfig(\n",
|
||||
" instance_count=1,\n",
|
||||
" instance_type=\"ml.m5.xlarge\",\n",
|
||||
" volume_size_in_gb=20\n",
|
||||
" )\n",
|
||||
" ),\n",
|
||||
" app_specification=AppSpecification(\n",
|
||||
" image_uri=f\"683313688378.dkr.ecr.{region}.amazonaws.com/sagemaker-scikit-learn:0.23-1-cpu-py3\",\n",
|
||||
" container_entrypoint=[\"python3\", \"/opt/ml/processing/code/preprocess.py\"]\n",
|
||||
" ),\n",
|
||||
" role_arn=role, # Intelligent default for execution role\n",
|
||||
" processing_inputs=[\n",
|
||||
" ProcessingInput(\n",
|
||||
" input_name=\"input\",\n",
|
||||
" s3_input=ProcessingS3Input(\n",
|
||||
" s3_uri=f\"s3://sagemaker-example-files-prod-{region}/datasets/tabular/synthetic/churn.txt\",\n",
|
||||
" s3_data_type=\"S3Prefix\",\n",
|
||||
" local_path=\"/opt/ml/processing/input\",\n",
|
||||
" s3_input_mode=\"File\"\n",
|
||||
" ),\n",
|
||||
" ),\n",
|
||||
" ProcessingInput(\n",
|
||||
" input_name=\"code\",\n",
|
||||
" s3_input=ProcessingS3Input(\n",
|
||||
" s3_uri=pre_processing_code_s3_uri,\n",
|
||||
" s3_data_type=\"S3Prefix\",\n",
|
||||
" local_path=\"/opt/ml/processing/code\",\n",
|
||||
" s3_input_mode=\"File\"\n",
|
||||
" ),\n",
|
||||
" )\n",
|
||||
" ],\n",
|
||||
" processing_output_config= ProcessingOutputConfig(\n",
|
||||
" outputs=[\n",
|
||||
" ProcessingOutput(\n",
|
||||
" output_name=\"train\",\n",
|
||||
" s3_output=ProcessingS3Output(\n",
|
||||
" s3_uri=processed_train_data_uri,\n",
|
||||
" s3_upload_mode=\"EndOfJob\",\n",
|
||||
" local_path=\"/opt/ml/processing/output/train\"\n",
|
||||
" )\n",
|
||||
" ),\n",
|
||||
" ProcessingOutput(\n",
|
||||
" output_name=\"validation\",\n",
|
||||
" s3_output=ProcessingS3Output(\n",
|
||||
" s3_uri=processed_validation_data_uri,\n",
|
||||
" s3_upload_mode=\"EndOfJob\",\n",
|
||||
" local_path=\"/opt/ml/processing/output/validation\"\n",
|
||||
" )\n",
|
||||
" ),\n",
|
||||
" ProcessingOutput(\n",
|
||||
" output_name=\"test\",\n",
|
||||
" s3_output=ProcessingS3Output(\n",
|
||||
" s3_uri=processed_test_data_uri,\n",
|
||||
" s3_upload_mode=\"EndOfJob\",\n",
|
||||
" local_path=\"/opt/ml/processing/output/test\"\n",
|
||||
" )\n",
|
||||
" )\n",
|
||||
" ]\n",
|
||||
" )\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Wait for the ProcessingJob to complete\n",
|
||||
"processing_job.wait()\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5d8c4139-244d-4827-b872-35fbeed664d8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Train a classifier using XGBoost\n",
|
||||
"Use SageMaker Training and the managed XGBoost image to train a classifier. <br />\n",
|
||||
"More details on how to use SageMaker managed training with XGBoost can be found [here](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "bbbf270c-20c0-4f82-a8a6-c2794ba57f70",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"<div class=\"alert alert-block alert-info\">\n",
|
||||
" <b>NOTE:</b> For more information on using SageMaker managed container images and retrieving their ECR paths, \n",
|
||||
" <a href=\"https://docs.aws.amazon.com/sagemaker/latest/dg-ecr-paths/sagemaker-algo-docker-registry-paths.html\" target=\"_blank\">here</a> \n",
|
||||
" is the documentation. Please note that the image URI might need to be updated based on your selected AWS region.\n",
|
||||
"</div>\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "268583f2-2c20-401b-a57c-ee88fd402583",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"image = f\"683313688378.dkr.ecr.{region}.amazonaws.com/sagemaker-xgboost:1.7-1\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "e85a91b5-4a68-43f4-b04a-991035cc35bf",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.resources import TrainingJob\n",
|
||||
"from sagemaker_core.shapes import (\n",
|
||||
" AlgorithmSpecification,\n",
|
||||
" Channel,\n",
|
||||
" DataSource,\n",
|
||||
" S3DataSource,\n",
|
||||
" ResourceConfig,\n",
|
||||
" StoppingCondition,\n",
|
||||
" OutputDataConfig,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"job_name = \"xgboost-churn-\" + time.strftime(\n",
|
||||
" \"%Y-%m-%d-%H-%M-%S\", time.gmtime()\n",
|
||||
") # Name of training job\n",
|
||||
"instance_type = \"ml.m4.xlarge\" # SageMaker instance type to use for training\n",
|
||||
"instance_count = 1 # Number of instances to use for training\n",
|
||||
"volume_size_in_gb = 30 # Amount of storage to allocate to training job\n",
|
||||
"max_runtime_in_seconds = 600 # Maximum runtimt. Job exits if it doesn't finish before this\n",
|
||||
"s3_output_path = f\"s3://{bucket}\" # bucket and optional prefix where the training job stores output artifacts, like model artifact.\n",
|
||||
"\n",
|
||||
"# Specify hyperparameters\n",
|
||||
"hyper_parameters = {\n",
|
||||
" \"max_depth\": \"5\",\n",
|
||||
" \"eta\": \"0.2\",\n",
|
||||
" \"gamma\": \"4\",\n",
|
||||
" \"min_child_weight\": \"6\",\n",
|
||||
" \"subsample\": \"0.8\",\n",
|
||||
" \"verbosity\": \"0\",\n",
|
||||
" \"objective\": \"binary:logistic\",\n",
|
||||
" \"num_round\": \"100\",\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"# Create training job.\n",
|
||||
"training_job = TrainingJob.create(\n",
|
||||
" training_job_name=job_name,\n",
|
||||
" hyper_parameters=hyper_parameters,\n",
|
||||
" algorithm_specification=AlgorithmSpecification(\n",
|
||||
" training_image=image, training_input_mode=\"File\"\n",
|
||||
" ),\n",
|
||||
" role_arn=role,\n",
|
||||
" input_data_config=[\n",
|
||||
" Channel(\n",
|
||||
" channel_name=\"train\",\n",
|
||||
" content_type=\"csv\",\n",
|
||||
" data_source=DataSource(\n",
|
||||
" s3_data_source=S3DataSource(\n",
|
||||
" s3_data_type=\"S3Prefix\",\n",
|
||||
" s3_uri=processed_train_data_uri,\n",
|
||||
" s3_data_distribution_type=\"FullyReplicated\",\n",
|
||||
" )\n",
|
||||
" ),\n",
|
||||
" ),\n",
|
||||
" Channel(\n",
|
||||
" channel_name=\"validation\",\n",
|
||||
" content_type=\"csv\",\n",
|
||||
" data_source=DataSource(\n",
|
||||
" s3_data_source=S3DataSource(\n",
|
||||
" s3_data_type=\"S3Prefix\",\n",
|
||||
" s3_uri=processed_validation_data_uri,\n",
|
||||
" s3_data_distribution_type=\"FullyReplicated\",\n",
|
||||
" )\n",
|
||||
" ),\n",
|
||||
" ),\n",
|
||||
" ],\n",
|
||||
" output_data_config=OutputDataConfig(s3_output_path=s3_output_path),\n",
|
||||
" resource_config=ResourceConfig(\n",
|
||||
" instance_type=instance_type,\n",
|
||||
" instance_count=instance_count,\n",
|
||||
" volume_size_in_gb=volume_size_in_gb,\n",
|
||||
" ),\n",
|
||||
" stopping_condition=StoppingCondition(max_runtime_in_seconds=max_runtime_in_seconds),\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Wait for the training job to complete\n",
|
||||
"# training_job.wait()\n",
|
||||
"training_job.wait(poll=60, timeout=None, logs=False)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b79e6746-6a03-4868-a22e-63a32bfcf599",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Use model artifacts for real time inference\n",
|
||||
"To use the model to perform real time inference, we need to:\n",
|
||||
"\n",
|
||||
"1. Create a SageMaker model with the same first-party image as we used for training, and the model artifacts produced during training. Indeed, such image can also be used to run inference\n",
|
||||
"2. Create an `EndpointConfig` using the SageMaker model object created in the previous step. The endpoint configuration specifies what SageMaker model to use, and what endpoint type.\n",
|
||||
"3. Create a SageMaker endpoint using `EndpointConfig` and other optional parameters.\n",
|
||||
"\n",
|
||||
"More information about SageMaker Endpoints can be found [here](https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints.html)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "82b0db2b-4daf-4ee0-a035-b7108f3d7912",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Create SageMaker Model\n",
|
||||
"\n",
|
||||
"Create a Model resource based on the model artifacts produced by the training job."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "270aa263-2a8f-4225-bcd6-56a57195ac63",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.resources import Model\n",
|
||||
"from sagemaker_core.shapes import ContainerDefinition\n",
|
||||
"\n",
|
||||
"model_s3_uri = training_job.model_artifacts.s3_model_artifacts # Get URI of model artifacts from the training job.\n",
|
||||
"\n",
|
||||
"# Create SageMaker model: An image along with the model artifact to use.\n",
|
||||
"customer_churn_model = Model.create(\n",
|
||||
" model_name=\"customer-churn-xgboost\",\n",
|
||||
" primary_container=ContainerDefinition(image=image, model_data_url=model_s3_uri),\n",
|
||||
" execution_role_arn=role,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "abfe8ade-8de6-487a-9b24-a2afbeaa8559",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Create endpoint configuration for real-time inference\n",
|
||||
"To create a SageMaker endpoint we first create an `EndpointConfig`. The endpoint configuration specifies what SageMaker model to use."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "056e57bb-9edf-461e-97e8-40c9e5ebf39f",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.resources import Endpoint, EndpointConfig\n",
|
||||
"from sagemaker_core.shapes import ProductionVariant\n",
|
||||
"\n",
|
||||
"endpoint_config_name = \"churn-prediction-endpoint-config\" # Name of endpoint configuration\n",
|
||||
"model_name = customer_churn_model.get_name() # Get name of SageMaker model created in previous step\n",
|
||||
"endpoint_name = \"customer-churn-endpoint\" # Name of SageMaker endpoint\n",
|
||||
"\n",
|
||||
"endpoint_config = EndpointConfig.create(\n",
|
||||
" endpoint_config_name=endpoint_config_name,\n",
|
||||
" production_variants=[\n",
|
||||
" ProductionVariant(\n",
|
||||
" variant_name=\"AllTraffic\",\n",
|
||||
" model_name=model_name,\n",
|
||||
" instance_type=instance_type,\n",
|
||||
" initial_instance_count=1,\n",
|
||||
" )\n",
|
||||
" ],\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "endpoint-create-fix-4872",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"sagemaker_endpoint = Endpoint.create(\n",
|
||||
" endpoint_name=endpoint_name,\n",
|
||||
" endpoint_config_name=endpoint_config.get_name(),\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "635cb615-f4e0-43fa-91d6-0b480a51fd16",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"sagemaker_endpoint.wait_for_status(\n",
|
||||
" target_status=\"InService\"\n",
|
||||
") # Wait for endpoint to become in service"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fd718d54-65ee-4b67-8556-196246356a68",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Test live endpoint - with a sample record from test dataset\n",
|
||||
"\n",
|
||||
"Let us download the test data from S3."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "3fdbd048-014f-418d-8d88-fc329afc7ee6",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import boto3\n",
|
||||
"s3 = boto3.client(\"s3\")\n",
|
||||
"s3.download_file(Bucket = bucket,\n",
|
||||
" Key = \"sagemaker-core-intro-blog/processing/output/test/test.csv\",\n",
|
||||
" Filename = \"test.csv\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2ff78e4f-47cd-4f05-ba00-6ded32ad12a8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Invoke the endpoint\n",
|
||||
"\n",
|
||||
"Let us invoke the endpoint now."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "d66fd20a-7016-4b55-9144-633461059b8b",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def print_performance_metrics(df_result, target_col):\n",
|
||||
" print(\"Prediction results:\")\n",
|
||||
" print(df_result.describe())\n",
|
||||
" print(f\"\nPredicted class distribution:\n{df_result[0].value_counts()}\")\n",
|
||||
"\n",
|
||||
"#Pick a random record from CSV and convert it to string\n",
|
||||
"df = pd.read_csv(\"test.csv\", header=None)\n",
|
||||
"sample = df.sample(1)\n",
|
||||
"sample_payload = sample.to_csv(header=False, index=False).strip()\n",
|
||||
"\n",
|
||||
"# Send sample payload to live endpoint and parse response\n",
|
||||
"res = sagemaker_endpoint.invoke(body=sample_payload, content_type=\"text/csv\")\n",
|
||||
"result = res.body.read().decode(\"utf-8\")\n",
|
||||
"result = result.split(\"\\n\")[:-1]\n",
|
||||
"\n",
|
||||
"# Compute performance metrics\n",
|
||||
"df_result = pd.DataFrame(result).astype(float)\n",
|
||||
"print_performance_metrics(df_result, test_target_column)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0e9d3ad7-4b67-405e-940a-0331fd28811f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Clean up"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "9fb90949-9b35-4e11-8d2a-6053beacb4ba",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"sagemaker_endpoint.delete()\n",
|
||||
"endpoint_config.delete()\n",
|
||||
"customer_churn_model.delete()"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "conda_python3",
|
||||
"language": "python",
|
||||
"name": "conda_python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.16"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
from sklearn.model_selection import train_test_split
|
||||
import argparse
|
||||
import os
|
||||
from io import StringIO
|
||||
import pandas as pd
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="parsing all commandline arguments to the processing job"
|
||||
)
|
||||
|
||||
# parser.add_argument("--s3_input_bucket", type=str, help="s3 bucket containing input data")
|
||||
# parser.add_argument("--s3_input_key_prefix", type=str, help="s3 input key prefix")
|
||||
# parser.add_argument("--s3_output_bucket", type=str, help="s3 output bucket")
|
||||
# parser.add_argument("--s3_output)_key_prefix", type=str, help="s3 output key prefix")
|
||||
print("starting near.....")
|
||||
processing_root_path = "/opt/ml/processing/"
|
||||
input_data_prefix = "input"
|
||||
train_output_prefix = "output/train/"
|
||||
validation_output_prefix = "output/validation/"
|
||||
test_output_prefix = "output/test/"
|
||||
|
||||
file_path = processing_root_path + input_data_prefix+"/churn.txt"
|
||||
df = pd.read_csv(file_path)
|
||||
print(df)
|
||||
# Phone number is unique - will not add value to classifier
|
||||
df = df.drop("Phone", axis=1)
|
||||
|
||||
# Cast Area Code to non-numeric
|
||||
df["Area Code"] = df["Area Code"].astype(object)
|
||||
|
||||
# Remove one feature from highly corelated pairs
|
||||
df = df.drop(["Day Charge", "Eve Charge", "Night Charge", "Intl Charge"], axis=1)
|
||||
|
||||
# One-hot encode catagorical features into numeric features
|
||||
model_data = pd.get_dummies(df)
|
||||
model_data = pd.concat(
|
||||
[
|
||||
model_data["Churn?_True."],
|
||||
model_data.drop(["Churn?_False.", "Churn?_True."], axis=1),
|
||||
],
|
||||
axis=1,
|
||||
)
|
||||
model_data = model_data.astype(float)
|
||||
|
||||
# Split data into train and validation datasets
|
||||
train_data, validation_data = train_test_split(
|
||||
model_data, test_size=0.33, random_state=42
|
||||
)
|
||||
|
||||
# Further split the validation dataset into test and validation datasets.
|
||||
validation_data, test_data = train_test_split(
|
||||
validation_data, test_size=0.33, random_state=42
|
||||
)
|
||||
|
||||
# Remove and store the target column for the test data
|
||||
test_target_column = test_data["Churn?_True."]
|
||||
test_data.drop(["Churn?_True."], axis=1, inplace=True)
|
||||
print("ending near.....")
|
||||
# Store all datasets locally
|
||||
train_data.to_csv(processing_root_path+train_output_prefix+"train.csv", header=False, index=False)
|
||||
validation_data.to_csv(processing_root_path+validation_output_prefix+"validation.csv", header=False, index=False)
|
||||
test_data.to_csv(processing_root_path+test_output_prefix+"test.csv", header=False, index=False)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,101 @@
|
||||
Id,Model,Year,Status,Mileage,Price,MSRP
|
||||
96864,6110,2019 Tesla Model 3 Standard Range,2019,Used,"50,412 mi.","$31,924.00",Not specified
|
||||
51128,7439,2016 Hyundai Sonata Sport,2016,Used,"97,237 mi.","$13,492.00",Not specified
|
||||
85327,4565,2021 Porsche Cayenne GTS,2021,Used,"23,375 mi.","$105,495.00",Not specified
|
||||
110510,4758,2020 Volkswagen Passat 2.0T SE,2020,Used,"46,249 mi.","$22,899.00",Not specified
|
||||
74498,3836,2023 Mercedes-Benz EQS 450+ EQS 450 4MATIC,2023,New,Not available,"$112,745.00",Not specified
|
||||
103711,3787,2023 Toyota Venza LE,2023,New,Not available,"$39,224.00",Not specified
|
||||
29375,97,2022 Dodge Challenger R/T Scat Pack,2022,New,Not available,"$62,985.00",Not specified
|
||||
41662,2511,2022 Ford F-250 XLT,2022,New,Not available,"$63,500.00","MSRP $64,500"
|
||||
60265,8276,2023 INFINITI QX50 Luxe,2023,New,Not available,"$49,100.00",Not specified
|
||||
2206,2206,2020 Acura RDX Technology Package,2020,Used,"24,712 mi.","$35,955.00",Not specified
|
||||
100010,86,2023 Toyota GR86 Premium,2023,New,Not available,"$35,683.00",$333 price drop
|
||||
54283,2294,2019 INFINITI QX50 ESSENTIAL,2019,Used,"34,986 mi.","$29,999.00",Not specified
|
||||
30829,1551,2022 Dodge Durango Citadel,2022,New,Not available,"$59,106.00",Not specified
|
||||
50828,7139,2023 Hyundai Santa Cruz 2.5T SEL Premium,2023,New,Not available,"$39,385.00","MSRP $39,385"
|
||||
85246,4484,2019 Porsche 911 GT3 RS,2019,Used,"4,241 mi.","$254,888.00",Not specified
|
||||
114257,8505,2018 Volkswagen Beetle 2.0T S,2018,Used,"46,926 mi.","$25,594.00",Not specified
|
||||
34452,5174,2008 Dodge Charger SXT,2008,Used,"133,467 mi.","$6,495.00",Not specified
|
||||
68825,8148,2011 Lexus IS-F Base,2011,Used,"36,800 mi.","$54,999.00",Not specified
|
||||
99988,64,2023 Toyota Tundra Platinum,2023,New,Not available,"$66,483.00",Not specified
|
||||
99824,9070,2023 Tesla Model 3 Base,2023,Used,"2,056 mi.","$45,888.00",Not specified
|
||||
103430,3506,2023 Toyota Camry XSE,2023,New,Not available,"$38,246.00",Not specified
|
||||
42818,3667,2017 Ford Focus ST Base,2017,Used,"64,421 mi.","$19,900.00",Not specified
|
||||
825,825,2020 Acura RDX Technology Package,2020,Used,"57,295 mi.","$31,881.00",Not specified
|
||||
98270,7516,2020 Tesla Model 3 Standard Range Plus,2020,Used,"8,817 mi.","$44,900.00",Not specified
|
||||
56852,4863,2019 INFINITI Q50 3.0t Signature Edition,2019,Used,"39,621 mi.","$27,500.00",Not specified
|
||||
108336,2584,2022 Volkswagen Atlas 3.6L SE w/Technology,2022,New,Not available,"$46,550.00",Not specified
|
||||
63813,3136,2021 Lexus RX 350L Premium,2021,Used,"16,560 mi.","$47,987.00","$2,000 price drop"
|
||||
100655,731,2023 Toyota Tundra Hybrid Platinum,2023,New,Not available,"$69,346.00",Not specified
|
||||
84924,4162,2014 Porsche 911 Carrera,2014,Used,"43,484 mi.","$67,950.00",$550 price drop
|
||||
113714,7962,2018 Volkswagen Atlas 3.6L SE,2018,Used,"62,614 mi.","$24,191.00",Not specified
|
||||
63736,3059,2018 Lexus RX 350L Premium,2018,Used,"34,918 mi.","$36,995.00",Not specified
|
||||
86932,6170,2012 Porsche Panamera 4,2012,Used,"42,986 mi.","$33,995.00",Not specified
|
||||
46008,2319,2023 Hyundai Palisade XRT,2023,New,Not available,"$42,470.00",Not specified
|
||||
96253,5499,2022 Tesla Model 3 Performance,2022,Used,"16,244 mi.","$49,999.00",Not specified
|
||||
6305,6305,2006 Acura TSX Base (A5),2006,Used,"84,825 mi.","$11,450.00",Not specified
|
||||
21555,2276,2023 Chevrolet Tahoe RST,2023,New,Not available,"$67,794.00","MSRP $67,295"
|
||||
61420,743,2020 Lexus ES 300h Luxury,2020,Used,"27,649 mi.","$44,953.00",Not specified
|
||||
32580,3302,2014 Dodge Challenger R/T,2014,Used,"164,569 mi.","$12,995.00",Not specified
|
||||
112597,6845,2023 Volkswagen Atlas Cross Sport 3.6L V6 SE w/Technology,2023,New,Not available,"$40,423.00","$3,500 price drop"
|
||||
44739,1050,2023 Hyundai Tucson Limited,2023,New,Not available,"$39,510.00","MSRP $39,510"
|
||||
83743,2981,2018 Porsche Cayenne Platinum Edition,2018,Used,"37,491 mi.","$39,495.00",Not specified
|
||||
59550,7561,2021 INFINITI Q50 3.0t LUXE,2021,Used,"49,667 mi.","$24,995.00",Not specified
|
||||
39410,259,2022 Ford Explorer Timberline,2022,New,Not available,"$53,480.00",Not specified
|
||||
75383,4721,2022 Mercedes-Benz GLC 300 Base 4MATIC,2022,New,Not available,"$50,260.00","MSRP $50,260"
|
||||
35442,6164,2017 Dodge Challenger SXT,2017,Used,"56,467 mi.","$20,503.00",Not specified
|
||||
39144,9866,2015 Dodge Durango Limited,2015,Used,"99,683 mi.","$18,399.00","$1,600 price drop"
|
||||
103281,3357,2023 Toyota Tacoma SR5,2023,New,Not available,"$33,950.00","MSRP $33,950"
|
||||
7780,7780,2023 Acura MDX Advance,2023,New,Not available,"$65,890.00",Not specified
|
||||
34843,5565,2019 Dodge Challenger R/T Scat Pack,2019,Used,"38,565 mi.","$44,132.00","$2,368 price drop"
|
||||
44389,700,2021 Hyundai Palisade SEL,2021,Used,"11,036 mi.","$41,990.00",Not specified
|
||||
68336,7659,2022 Lexus LS 500,2022,New,Not available,"$106,445.00","MSRP $106,445"
|
||||
104750,4826,2019 Toyota Highlander LE Plus,2019,Used,"42,300 mi.","$28,500.00","$2,500 price drop"
|
||||
3991,3991,2019 Acura RDX Base,2019,Used,"45,362 mi.","$30,998.00","$1,000 price drop"
|
||||
82796,2034,2013 Porsche Cayenne Turbo,2013,Used,"73,086 mi.","$33,395.00",$604 price drop
|
||||
39509,358,2022 Ford F-350 XL,2022,New,Not available,"$66,790.00",Not specified
|
||||
57364,5375,2023 INFINITI QX60 SENSORY,2023,New,Not available,"$65,718.00","MSRP $63,825"
|
||||
70829,167,2022 Mercedes-Benz GLE 450 AWD 4MATIC,2022,New,Not available,"$74,260.00","MSRP $74,260"
|
||||
49749,6060,2016 Hyundai Genesis 3.8,2016,Used,"74,714 mi.","$23,000.00",Not specified
|
||||
81569,807,2022 Porsche 718 Boxster GTS,2022,Porsche Certified,586 mi.,"$113,991.00",Not specified
|
||||
3281,3281,2021 Acura TLX Technology,2021,Used,"9,604 mi.","$35,594.00",$619 price drop
|
||||
92193,1439,2016 Tesla Model S 75,2016,Used,"55,343 mi.","$34,995.00","$4,000 price drop"
|
||||
42775,3624,2022 Ford Escape SE,2022,New,Not available,"$34,690.00",Not specified
|
||||
87560,6798,2021 Porsche 718 Cayman GTS 4.0,2021,Used,"3,331 mi.","$115,000.00",Not specified
|
||||
69040,8363,2011 Lexus ES 350 Base,2011,Used,"68,332 mi.","$16,940.00",Not specified
|
||||
41276,2125,2022 Ford F-350 XL,2022,New,Not available,"$69,630.00",Not specified
|
||||
95380,4626,2020 Tesla Model X Long Range,2020,Used,"32,000 mi.","$79,995.00",Not specified
|
||||
99894,9140,2018 Tesla Model 3 Long Range,2018,Used,"14,344 mi.","$40,500.00",Not specified
|
||||
51927,8238,2021 Hyundai Palisade SE,2021,Used,"58,452 mi.","$31,990.00",Not specified
|
||||
9240,9240,2023 Acura Integra A-Spec,2023,New,Not available,"$35,095.00","MSRP $35,095"
|
||||
44883,1194,2021 Hyundai Palisade SEL,2021,Used,"32,524 mi.","$36,795.00",Not specified
|
||||
89424,8662,2021 Porsche Cayenne S,2021,Used,"18,463 mi.","$89,989.00",Not specified
|
||||
83920,3158,2021 Porsche Cayenne AWD,2021,Used,"17,242 mi.","$79,900.00",Not specified
|
||||
61322,645,2023 Lexus LX 600 Premium,2023,New,Not available,"$103,150.00","MSRP $103,150"
|
||||
110236,4484,2019 Volkswagen Beetle 2.0T S,2019,Used,"20,406 mi.","$21,921.00",Not specified
|
||||
88109,7347,2018 Porsche 718 Boxster,2018,Used,"16,141 mi.","$64,990.00",Not specified
|
||||
66076,5399,2014 Lexus LS 460 Base,2014,Used,"140,725 mi.","$21,975.00",Not specified
|
||||
65897,5220,2019 Lexus UX 200 F-SPORT,2019,Used,"34,362 mi.","$32,998.00",Not specified
|
||||
55324,3335,2017 INFINITI Q50 3.0T Premium,2017,Used,"72,174 mi.","$20,997.00",Not specified
|
||||
31470,2192,2022 Dodge Durango R/T,2022,New,Not available,"$64,930.00","MSRP $66,275"
|
||||
71177,515,2023 Mercedes-Benz GLE 450 AWD 4MATIC,2023,New,Not available,"$82,915.00","MSRP $82,915"
|
||||
50393,6704,2017 Hyundai Accent SE,2017,Used,"75,704 mi.","$12,388.00",Not specified
|
||||
45609,1920,2022 Hyundai Sonata SEL,2022,Used,821 mi.,"$27,011.00",Not specified
|
||||
32524,3246,2022 Dodge Challenger SRT Hellcat,2022,New,Not available,"$85,230.00","MSRP $85,980"
|
||||
113644,7892,2023 Volkswagen Atlas Cross Sport 2.0T SEL,2023,New,Not available,"$48,938.00","MSRP $48,938"
|
||||
28415,9136,2014 Chevrolet Silverado 1500 LT,2014,Used,"110,316 mi.","$21,660.00",Not specified
|
||||
12150,2754,2023 BMW 540 i,2023,New,Not available,"$68,520.00","MSRP $68,520"
|
||||
36372,7094,2013 Dodge Charger SE,2013,Used,"125,336 mi.","$9,995.00","$1,000 price drop"
|
||||
11948,2552,2018 BMW 320 i xDrive,2018,Used,"53,250 mi.","$23,998.00",Not specified
|
||||
76352,5690,2023 Mercedes-Benz Maybach S 580 4MATIC,2023,New,Not available,"$228,850.00","MSRP $228,850"
|
||||
91021,267,2021 Tesla Model Y Long Range,2021,Used,"19,859 mi.","$49,000.00","$5,000 price drop"
|
||||
58210,6221,2008 INFINITI G35 Journey,2008,Used,"160,591 mi.","$6,900.00",Not specified
|
||||
101534,1610,2022 Toyota Tundra Hybrid Platinum,2022,New,Not available,"$72,622.00",Not specified
|
||||
54403,2414,2023 INFINITI QX80 PREMIUM SELECT,2023,New,Not available,"$82,010.00","MSRP $84,510"
|
||||
90442,9680,2022 Porsche Cayenne,2022,Porsche Certified,"4,137 mi.","$203,995.00",Not specified
|
||||
72757,2095,2022 Mercedes-Benz S-Class S 580 4MATIC,2022,New,Not available,"$152,685.00","MSRP $152,685"
|
||||
83365,2603,2003 Porsche 911 Turbo,2003,Used,"28,658 mi.","$115,900.00",Not specified
|
||||
90060,9298,2019 Porsche Cayenne Base,2019,Used,"32,364 mi.","$54,494.00",Not specified
|
||||
53711,1722,2013 INFINITI G37 Journey,2013,Used,"129,963 mi.","$11,995.00",Not specified
|
||||
60355,8366,2023 INFINITI QX80 SENSORY,2023,New,Not available,"$87,955.00","MSRP $87,955"
|
||||
75909,5247,2023 Mercedes-Benz GLS 450 4MATIC,2023,New,Not available,"$93,645.00","MSRP $93,645"
|
||||
|
@@ -0,0 +1,5 @@
|
||||
customer_id,age,SSN,credit_score
|
||||
573291,26,266666666,500
|
||||
109382,42,422222222,600
|
||||
828400,35,355555555,700
|
||||
124013,61,611111111,800
|
||||
|
+5
@@ -0,0 +1,5 @@
|
||||
customer_id,city_code,state_code,country_code
|
||||
573291,1,49,2
|
||||
109382,2,40,2
|
||||
828400,3,31,2
|
||||
124013,4,5,2
|
||||
|
+5
@@ -0,0 +1,5 @@
|
||||
customer_id,city_code,state_code,country_code,email,name
|
||||
573291,1,49,2,john.lee@gmail.com,John Lee
|
||||
109382,2,40,2,olivequil@gmail.com,Olive Quil
|
||||
828400,3,31,2,liz.knee@gmail.com,Liz Knee
|
||||
124013,4,5,2,eileenbook@gmail.com,Eileen Book
|
||||
|
@@ -0,0 +1,5 @@
|
||||
customer_id,order_id,order_status,store_id
|
||||
573291,4132,1,303
|
||||
109382,5724,0,201
|
||||
828400,1942,0,431
|
||||
124013,6782,1,213
|
||||
|
Binary file not shown.
@@ -0,0 +1,2 @@
|
||||
{"source-ref":"s3://AWSDOC-EXAMPLE-BUCKET/example_image.jpg","species":"0","species-metadata":{"class-name": "dog","confidence": 0.00,"type": "groundtruth/image-classification","job-name": "identify-animal-species","human-annotated": "yes","creation-date": "2018-10-18T22:18:13.527256"}}
|
||||
{"source":"The food was delicious","mood":"1","mood-metadata":{"class-name": "positive","confidence": 0.8,"type": "groundtruth/text-classification","job-name": "label-sentiment","human-annotated": "yes","creation-date": "2020-10-18T22:18:13.527256"}}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 76 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 49 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 54 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 26 KiB |
+723
@@ -0,0 +1,723 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Amazon SageMaker Feature Store: Introduction to Feature Store"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This notebook demonstrates how to get started with Feature Store, create feature groups, and ingest data into them. These feature groups are stored in your Feature Store.\n",
|
||||
"\n",
|
||||
"Feature groups are resources that contain metadata for all data stored in your Feature Store. A feature group is a logical grouping of features, defined in the feature store to describe records. A feature group’s definition is composed of a list of feature definitions, a record identifier name, and configurations for its online and offline store. \n",
|
||||
"\n",
|
||||
"### Overview\n",
|
||||
"1. Set up\n",
|
||||
"2. Creating a feature group\n",
|
||||
"3. Ingest data into a feature group\n",
|
||||
"\n",
|
||||
"### Prerequisites\n",
|
||||
"This notebook uses sagemaker_core SDK and `Python 3 (Data Science)` kernel. This notebook works with Studio, Jupyter, and JupyterLab. \n",
|
||||
"\n",
|
||||
"#### Library dependencies:\n",
|
||||
"* `sagemaker_core`\n",
|
||||
"* `numpy`\n",
|
||||
"* `pandas`\n",
|
||||
"\n",
|
||||
"#### Role requirements:\n",
|
||||
"**IMPORTANT**: You must attach the following policies to your execution role:\n",
|
||||
"* `AmazonS3FullAccess`\n",
|
||||
"* `AmazonSageMakerFeatureStoreAccess`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Set up"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip uninstall sagemaker-core -y\n",
|
||||
"!pip install pip --upgrade --quiet\n",
|
||||
"!pip install sagemaker-core --upgrade"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import pandas as pd\n",
|
||||
"import numpy as np\n",
|
||||
"import io\n",
|
||||
"import sagemaker\n",
|
||||
"from sagemaker_core.helper.session_helper import get_execution_role, Session\n",
|
||||
"\n",
|
||||
"sagemaker_session = Session()\n",
|
||||
"REGION_NAME = sagemaker_session._region_name\n",
|
||||
"role = get_execution_role()\n",
|
||||
"s3_bucket_name = sagemaker.Session().default_bucket()\n",
|
||||
"prefix = \"sagemaker-featurestore-introduction\"\n",
|
||||
"default_bucket_prefix = sagemaker.Session().default_bucket_prefix\n",
|
||||
"\n",
|
||||
"# If a default bucket prefix is specified, append it to the s3 path\n",
|
||||
"if default_bucket_prefix:\n",
|
||||
" prefix = f\"{default_bucket_prefix}/{prefix}\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Inspect your data\n",
|
||||
"In this notebook example we ingest synthetic data. We read from `./data/feature_store_introduction_customer.csv` and `./data/feature_store_introduction_orders.csv`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customer_data = pd.read_csv(\"data/feature_store_introduction_customer.csv\")\n",
|
||||
"orders_data = pd.read_csv(\"data/feature_store_introduction_orders.csv\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customer_data.head()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"orders_data.head()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Below is an illustration on the steps the data goes through before it is ingested into a Feature Store. In this notebook, we illustrate the use-case where you have data from multiple sources and want to store them independently in a feature store. Our example considers data from a data warehouse (customer data), and data from a real-time streaming service (order data). "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Create a feature group\n",
|
||||
"\n",
|
||||
"We first start by creating feature group names for customer_data and orders_data. Following this, we create two Feature Groups, one for `customer_data` and another for `orders_data`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from time import gmtime, strftime, sleep\n",
|
||||
"\n",
|
||||
"customers_feature_group_name = \"customers-feature-group-\" + strftime(\"%d-%H-%M-%S\", gmtime())\n",
|
||||
"orders_feature_group_name = \"orders-feature-group-\" + strftime(\"%d-%H-%M-%S\", gmtime())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customers_feature_group_name"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Instantiate a FeatureGroup object for customers_data and orders_data. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.shapes import FeatureDefinition\n",
|
||||
"\n",
|
||||
"CustomerFeatureDefinitions = [\n",
|
||||
" FeatureDefinition(feature_name=\"customer_id\", feature_type=\"Integral\"),\n",
|
||||
" FeatureDefinition(feature_name=\"city_code\", feature_type=\"Integral\"),\n",
|
||||
" FeatureDefinition(feature_name=\"state_code\", feature_type=\"Integral\"),\n",
|
||||
" FeatureDefinition(feature_name=\"country_code\", feature_type=\"Integral\"),\n",
|
||||
" FeatureDefinition(feature_name=\"EventTime\", feature_type=\"Fractional\"),\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"OrderFeatureDefinitions = [\n",
|
||||
" FeatureDefinition(feature_name=\"customer_id\", feature_type=\"Integral\"),\n",
|
||||
" FeatureDefinition(feature_name=\"order_id\", feature_type=\"Integral\"),\n",
|
||||
" FeatureDefinition(feature_name=\"order_status\", feature_type=\"Integral\"),\n",
|
||||
" FeatureDefinition(feature_name=\"store_id\", feature_type=\"Integral\"),\n",
|
||||
" FeatureDefinition(feature_name=\"EventTime\", feature_type=\"Fractional\"),\n",
|
||||
"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import time\n",
|
||||
"\n",
|
||||
"current_time_sec = int(round(time.time()))\n",
|
||||
"\n",
|
||||
"record_identifier_feature_name = \"customer_id\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Append `EventTime` feature to your data frame. This parameter is required, and time stamps each data point."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customer_data[\"EventTime\"] = pd.Series([current_time_sec] * len(customer_data), dtype=\"float64\")\n",
|
||||
"orders_data[\"EventTime\"] = pd.Series([current_time_sec] * len(orders_data), dtype=\"float64\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Load feature definitions to your feature group. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Below we call create to create two feature groups, customers_feature_group and orders_feature_group respectively"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.shapes import OnlineStoreConfig, OfflineStoreConfig, S3StorageConfig\n",
|
||||
"from sagemaker_core.resources import FeatureGroup\n",
|
||||
"\n",
|
||||
"customers_feature_group = FeatureGroup.create(\n",
|
||||
" feature_group_name=customers_feature_group_name,\n",
|
||||
" record_identifier_feature_name=record_identifier_feature_name,\n",
|
||||
" event_time_feature_name=\"EventTime\",\n",
|
||||
" role_arn=role,\n",
|
||||
" online_store_config=OnlineStoreConfig(enable_online_store=True),\n",
|
||||
" feature_definitions=CustomerFeatureDefinitions,\n",
|
||||
" offline_store_config=OfflineStoreConfig(\n",
|
||||
" s3_storage_config=S3StorageConfig(s3_uri=f\"s3://{s3_bucket_name}/{prefix}\")\n",
|
||||
" ),\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.resources import FeatureGroup\n",
|
||||
"\n",
|
||||
"orders_feature_group = FeatureGroup.create(\n",
|
||||
" feature_group_name=orders_feature_group_name,\n",
|
||||
" record_identifier_feature_name=record_identifier_feature_name,\n",
|
||||
" event_time_feature_name=\"EventTime\",\n",
|
||||
" role_arn=role,\n",
|
||||
" online_store_config=OnlineStoreConfig(enable_online_store=True),\n",
|
||||
" feature_definitions=OrderFeatureDefinitions,\n",
|
||||
" offline_store_config=OfflineStoreConfig(\n",
|
||||
" s3_storage_config=S3StorageConfig(s3_uri=f\"s3://{s3_bucket_name}/{prefix}\")\n",
|
||||
" ),\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"To confirm that your FeatureGroup has been created we use `wait_for_status` functions to wait for the feature group to be created successfully."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def check_feature_group_status(feature_group):\n",
|
||||
" status = feature_group.wait_for_status(target_status=\"Created\")\n",
|
||||
" print(f\"FeatureGroup {feature_group.get_name()} successfully created.\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"check_feature_group_status(customers_feature_group)\n",
|
||||
"check_feature_group_status(orders_feature_group)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Add metadata to a feature\n",
|
||||
"\n",
|
||||
"We can add searchable metadata fields to FeatureGroup features by using the `FeatureMetadata` class. The currently supported metadata fields are `description` and `parameters`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.resources import FeatureMetadata\n",
|
||||
"from sagemaker_core.shapes import FeatureParameter\n",
|
||||
"\n",
|
||||
"customers_feature_metadata = FeatureMetadata(\n",
|
||||
" feature_group_name=customers_feature_group_name, feature_name=\"customer_id\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customers_feature_metadata.update(\n",
|
||||
" description=\"The ID of a customer. It is also used in orders_feature_group.\",\n",
|
||||
" parameter_additions=[FeatureParameter(key=\"idType\", value=\"primaryKey\")],\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"To view feature metadata, we can use `get` method to display that feature."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customers_feature_metadata.get(\n",
|
||||
" feature_group_name=customers_feature_group_name, feature_name=\"customer_id\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Feature metadata fields are searchable. We use `search` API to find features with metadata that matches some search criteria."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"sagemaker_session.boto_session.client(\"sagemaker\", region_name=region).search(\n",
|
||||
" Resource=\"FeatureMetadata\",\n",
|
||||
" SearchExpression={\n",
|
||||
" \"Filters\": [\n",
|
||||
" {\n",
|
||||
" \"Name\": \"FeatureGroupName\",\n",
|
||||
" \"Operator\": \"Contains\",\n",
|
||||
" \"Value\": \"customers-feature-group-\",\n",
|
||||
" },\n",
|
||||
" {\"Name\": \"Parameters.idType\", \"Operator\": \"Equals\", \"Value\": \"primaryKey\"},\n",
|
||||
" ]\n",
|
||||
" },\n",
|
||||
") # We use the boto client to search"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Ingest data into a feature group\n",
|
||||
"\n",
|
||||
"We can put data into the FeatureGroup by using the `PutRecord` API. It will take < 1 minute to ingest data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# converting all columns to integral 64 type for further processing\n",
|
||||
"customer_data[\"customer_id\"] = pd.to_numeric(customer_data[\"customer_id\"]).astype(\"Int64\")\n",
|
||||
"customer_data[\"city_code\"] = pd.to_numeric(customer_data[\"city_code\"]).astype(\"Int64\")\n",
|
||||
"customer_data[\"state_code\"] = pd.to_numeric(customer_data[\"state_code\"]).astype(\"Int64\")\n",
|
||||
"customer_data[\"country_code\"] = pd.to_numeric(customer_data[\"country_code\"]).astype(\"Int64\")\n",
|
||||
"\n",
|
||||
"orders_data[\"customer_id\"] = pd.to_numeric(orders_data[\"customer_id\"]).astype(\"Int64\")\n",
|
||||
"orders_data[\"order_id\"] = pd.to_numeric(orders_data[\"order_id\"]).astype(\"Int64\")\n",
|
||||
"orders_data[\"order_status\"] = pd.to_numeric(orders_data[\"order_status\"]).astype(\"Int64\")\n",
|
||||
"orders_data[\"store_id\"] = pd.to_numeric(orders_data[\"store_id\"]).astype(\"Int64\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Creating `IngestData` function to ingest all dataframe records using `PutRecord` API call."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.shapes import FeatureValue\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def IngestData(df, feature_group):\n",
|
||||
" try:\n",
|
||||
" feature_values_list = list()\n",
|
||||
" for index, row in df.iterrows():\n",
|
||||
" # Iterate through each column for the current row\n",
|
||||
" for column in df.columns:\n",
|
||||
" feature_values = FeatureValue(\n",
|
||||
" feature_name=str(column), value_as_string=str(row[column])\n",
|
||||
" )\n",
|
||||
" feature_values_list.append(feature_values)\n",
|
||||
" feature_group.put_record(record=feature_values_list)\n",
|
||||
" feature_values_list.clear()\n",
|
||||
" SuccessString = f\"The dataframe with {len(df)} rows has been ingested successfully for feature group {feature_group.get_name()}\"\n",
|
||||
" return SuccessString\n",
|
||||
" except Exception as e:\n",
|
||||
" # Handle any other exceptions\n",
|
||||
" print(f\"An unexpected error occurred: {e}\")\n",
|
||||
" return None"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"IngestData(customer_data, customers_feature_group)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"IngestData(orders_data, orders_feature_group)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Using an arbitrary customer record ID, 573291 we use `get_record` to check that the data has been ingested into the feature group."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customer_id = 573291\n",
|
||||
"sample_record = customers_feature_group.get_record(\n",
|
||||
" record_identifier_value_as_string=str(customer_id)\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"sample_record"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We use `batch_get_record` to check that all data has been ingested into two feature groups by providing customer IDs."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.shapes import BatchGetRecordIdentifier\n",
|
||||
"\n",
|
||||
"all_records_customers = customers_feature_group.batch_get_record(\n",
|
||||
" identifiers=[\n",
|
||||
" BatchGetRecordIdentifier(\n",
|
||||
" feature_group_name=customers_feature_group_name,\n",
|
||||
" record_identifiers_value_as_string=[\"573291\", \"109382\", \"828400\", \"124013\"],\n",
|
||||
" )\n",
|
||||
" ]\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"all_records_customers"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"all_records_orders = orders_feature_group.batch_get_record(\n",
|
||||
" identifiers=[\n",
|
||||
" BatchGetRecordIdentifier(\n",
|
||||
" feature_group_name=orders_feature_group_name,\n",
|
||||
" record_identifiers_value_as_string=[\"573291\", \"109382\", \"828400\", \"124013\"],\n",
|
||||
" )\n",
|
||||
" ]\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"all_records_orders"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Add features to a feature group\n",
|
||||
"\n",
|
||||
"If we want to update a FeatureGroup that has done the data ingestion, we can use the `Update` function and then re-ingest data by using the updated dataset."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customers_feature_group.update(\n",
|
||||
" feature_additions=[\n",
|
||||
" FeatureDefinition(feature_name=\"email\", feature_type=\"String\"),\n",
|
||||
" FeatureDefinition(feature_name=\"name\", feature_type=\"String\"),\n",
|
||||
" ]\n",
|
||||
")\n",
|
||||
"time.sleep(120) # waiting for 120 seconds for the update process to get completed"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Inspect the new dataset."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customer_data_updated = pd.read_csv(\"data/feature_store_introduction_customer_updated.csv\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customer_data_updated.head()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Append `EventTime` feature to your data frame again."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customer_data_updated[\"EventTime\"] = pd.Series(\n",
|
||||
" [current_time_sec] * len(customer_data), dtype=\"float64\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Ingest the new dataset."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"## need to see how to ingest data in new SDK\n",
|
||||
"IngestData(customer_data_updated, customers_feature_group)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Use `batch_get_record` again to check that all updated data has been ingested into `customers_feature_group` by providing customer IDs."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"updated_records_customers = customers_feature_group.batch_get_record(\n",
|
||||
" identifiers=[\n",
|
||||
" BatchGetRecordIdentifier(\n",
|
||||
" feature_group_name=customers_feature_group_name,\n",
|
||||
" record_identifiers_value_as_string=[\"573291\", \"109382\", \"828400\", \"124013\"],\n",
|
||||
" )\n",
|
||||
" ]\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"updated_records_customers"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Clean up\n",
|
||||
"Here we remove the Feature Groups we created. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customers_feature_group.delete()\n",
|
||||
"orders_feature_group.delete()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"instance_type": "ml.t3.medium",
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.14"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
@@ -0,0 +1,957 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"# Amazon SageMaker Using AWS Large Model Inference (LMI) Deep Learning Container with Speculative Decoding \n",
|
||||
"\n",
|
||||
"**Recommended Kernel(s):** You can run this notebook with any Amazon SageMaker Studio kernel. We recommend using the Data Science 3.0 kernel.\n",
|
||||
"\n",
|
||||
"This notebook demonstrates how to deploy [`meta-llama/Meta-Llama-3-8B`](https://huggingface.co/meta-llama/Meta-Llama-3-8B) HuggingFace model to a SageMaker Endpoint for text generation with speculative decoding enabled. In this example, the SageMaker-managed [LMI (Large Model Inference)](https://docs.djl.ai/docs/serving/serving/docs/large_model_inference.html) Docker image will serve as the inference image. LMI images feature a [DJL serving](https://github.com/deepjavalibrary/djl-serving) stack powered by the [Deep Java Library](https://djl.ai/). \n",
|
||||
"\n",
|
||||
"**What is speculative decoding?** In the context of large language model inference, speculative decoding, as introduced by [Y. Leviathan et al. (ICML 2023)](https://arxiv.org/abs/2211.17192), is a technique used to accelerate the decoding process of large and therefore slow LLMs for latency-critical applications. The key idea is to use a smaller, less powerful but faster model called the *draft model* to generate candidate tokens that get validated by the larger, more powerful but slower model called the *target model*. At each iteration, the draft model generates $K>1$ candidate tokens. Then, using a single forward pass of the larger *target model*, none, part, or all candidate tokens get accepted. The more aligned the selected draft model is with the target model, the better guesses it makes, resulting in a higher candidate token acceptance rate and potential for speed up.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## 1. Dependency Installation\n",
|
||||
"### 1.1. Python Dependencies & Imports\n",
|
||||
"This notebook requires the following Python dependencies:\n",
|
||||
"* AWS [`sagemaker_core`]()\n",
|
||||
"\n",
|
||||
"Let's install or upgrade these dependencies using the following command:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%pip install pip --upgrade --quiet\n",
|
||||
"%pip install sagemaker-core huggingface_hub --quiet"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sagemaker\n",
|
||||
"import json\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"from sagemaker_core.helper.session_helper import get_execution_role, Session\n",
|
||||
"import pathlib\n",
|
||||
"import huggingface_hub"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PROJECT_ROOT_DIR = pathlib.Path.cwd()\n",
|
||||
"\n",
|
||||
"SM_SESSION = Session()\n",
|
||||
"\n",
|
||||
"REGION = SM_SESSION._region_name\n",
|
||||
"\n",
|
||||
"SM_DEFAULT_EXECUTION_ROLE_ARN = get_execution_role()\n",
|
||||
"\n",
|
||||
"INSTANCE_TYPE = \"ml.g5.12xlarge\"\n",
|
||||
"\n",
|
||||
"# INSTANCE_TYPE = \"ml.p4d.24xlarge\"\n",
|
||||
"\n",
|
||||
"# See https://github.com/aws/deep-learning-containers/blob/master/available_images.md#large-model-inference-containers for lastest DLC.\n",
|
||||
"container_image_uri = (\n",
|
||||
" f\"763104351884.dkr.ecr.{REGION}.amazonaws.com/djl-inference:0.29.0-tensorrtllm0.11.0-cu124\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"default_bucket = sagemaker.Session().default_bucket()\n",
|
||||
"default_bucket_prefix = sagemaker.Session().default_bucket_prefix\n",
|
||||
"\n",
|
||||
"TARGET_MODEL = \"meta-llama/Meta-Llama-3-8B\"\n",
|
||||
"DRAFT_MODEL = \"sagemaker\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Download custom draft model"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"## 2. Deploy Speculative Decoding-Enabled Endpoint\n",
|
||||
"### 2.1. Endpoint Deployment"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We will configure what models to use and server startup parameters and via environment variables. \n",
|
||||
"\n",
|
||||
"* `OPTION_MODEL_ID` points to the HF Model ID or base S3 prefix of the target model artifacts\n",
|
||||
"* `OPTION_SPECULATIVE_DRAFT_MODEL` points to the HF Model ID, base S3 prefix of the draft model artifacts or in our case `sagemaker` which represents the SageMaker draft model. \n",
|
||||
"\n",
|
||||
"You can read about the other config paramters in LMI [documentation](https://github.com/deepjavalibrary/djl-serving/tree/master/serving/docs/lmi).\n",
|
||||
"\n",
|
||||
"Note you can replace the target and draft model for you own."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"environment = {\n",
|
||||
" \"HF_MODEL_ID\": TARGET_MODEL,\n",
|
||||
" \"OPTION_SPECULATIVE_DRAFT_MODEL\": DRAFT_MODEL,\n",
|
||||
" \"OPTION_GPU_MEMORY_UTILIZATION\": \"0.85\",\n",
|
||||
" \"HF_TOKEN\": \"<YourHFToken>\",\n",
|
||||
"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.shapes import ContainerDefinition, ProductionVariant\n",
|
||||
"from sagemaker_core.resources import Model, EndpointConfig, Endpoint\n",
|
||||
"from time import gmtime, strftime"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"container_defintion = ContainerDefinition(image=container_image_uri, environment=environment)\n",
|
||||
"container_defintion.environment = environment"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"model_name = f'speculative-decoding-hugging-face-{strftime(\"%H-%M-%S\", gmtime())}'\n",
|
||||
"\n",
|
||||
"model = Model.create(\n",
|
||||
" model_name=model_name,\n",
|
||||
" primary_container=container_defintion,\n",
|
||||
" execution_role_arn=SM_DEFAULT_EXECUTION_ROLE_ARN,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"tags": [],
|
||||
"vscode": {
|
||||
"languageId": "plaintext"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"Start-up of LLM inference containers can last longer than smaller models, mainly due to longer model downloading and loading times. Timeout values need to be increased accordingly from their default values. Each endpoint deployment takes a few minutes. We also set `routing_strategy` which would benefit us if we were to back our endpoint with multiple instances."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.shapes import ProductionVariantRoutingConfig\n",
|
||||
"\n",
|
||||
"routing_config = ProductionVariantRoutingConfig(routing_strategy=\"LEAST_OUTSTANDING_REQUESTS\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"endpoint_config = EndpointConfig.create(\n",
|
||||
" endpoint_config_name=model_name,\n",
|
||||
" production_variants=[\n",
|
||||
" ProductionVariant(\n",
|
||||
" variant_name=model_name,\n",
|
||||
" initial_instance_count=1,\n",
|
||||
" instance_type=INSTANCE_TYPE,\n",
|
||||
" model_name=model,\n",
|
||||
" container_startup_health_check_timeout_in_seconds=3600,\n",
|
||||
" model_data_download_timeout_in_seconds=3600,\n",
|
||||
" routing_config=routing_config,\n",
|
||||
" )\n",
|
||||
" ],\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"endpoint = Endpoint.create(\n",
|
||||
" endpoint_name=model_name,\n",
|
||||
" endpoint_config_name=endpoint_config, # Pass `EndpointConfig` object created above\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"This cells will block until the endpoint is deployed, which is necessary for the following steps."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"endpoint.wait_for_status(\"InService\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"### Endpoint invocation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"Let's invoke our endpoint and get a sample response."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response = endpoint.invoke(\n",
|
||||
" body=json.dumps(\n",
|
||||
" {\n",
|
||||
" \"inputs\": [\"What is the capital of France?\"],\n",
|
||||
" \"max_new_tokens\": 512,\n",
|
||||
" \"temperature\": 0.0,\n",
|
||||
" }\n",
|
||||
" ),\n",
|
||||
" content_type=\"application/json\",\n",
|
||||
")\n",
|
||||
"response[\"Body\"].read()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 3. Clean Up Endpoint"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Delete any sagemaker core resource objects created in this notebook\n",
|
||||
"def delete_all_sagemaker_resources():\n",
|
||||
" all_objects = list(locals().values()) + list(globals().values())\n",
|
||||
" deletable_objects = [\n",
|
||||
" obj\n",
|
||||
" for obj in all_objects\n",
|
||||
" if hasattr(obj, \"delete\") and obj.__class__.__module__ == \"sagemaker_core.main.resources\"\n",
|
||||
" ]\n",
|
||||
"\n",
|
||||
" for obj in deletable_objects:\n",
|
||||
" obj.delete()\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"delete_all_sagemaker_resources()"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"availableInstances": [
|
||||
{
|
||||
"_defaultOrder": 0,
|
||||
"_isFastLaunch": true,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 4,
|
||||
"name": "ml.t3.medium",
|
||||
"vcpuNum": 2
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 1,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 8,
|
||||
"name": "ml.t3.large",
|
||||
"vcpuNum": 2
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 2,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 16,
|
||||
"name": "ml.t3.xlarge",
|
||||
"vcpuNum": 4
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 3,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 32,
|
||||
"name": "ml.t3.2xlarge",
|
||||
"vcpuNum": 8
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 4,
|
||||
"_isFastLaunch": true,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 8,
|
||||
"name": "ml.m5.large",
|
||||
"vcpuNum": 2
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 5,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 16,
|
||||
"name": "ml.m5.xlarge",
|
||||
"vcpuNum": 4
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 6,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 32,
|
||||
"name": "ml.m5.2xlarge",
|
||||
"vcpuNum": 8
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 7,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 64,
|
||||
"name": "ml.m5.4xlarge",
|
||||
"vcpuNum": 16
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 8,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 128,
|
||||
"name": "ml.m5.8xlarge",
|
||||
"vcpuNum": 32
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 9,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 192,
|
||||
"name": "ml.m5.12xlarge",
|
||||
"vcpuNum": 48
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 10,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 256,
|
||||
"name": "ml.m5.16xlarge",
|
||||
"vcpuNum": 64
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 11,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 384,
|
||||
"name": "ml.m5.24xlarge",
|
||||
"vcpuNum": 96
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 12,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 8,
|
||||
"name": "ml.m5d.large",
|
||||
"vcpuNum": 2
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 13,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 16,
|
||||
"name": "ml.m5d.xlarge",
|
||||
"vcpuNum": 4
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 14,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 32,
|
||||
"name": "ml.m5d.2xlarge",
|
||||
"vcpuNum": 8
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 15,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 64,
|
||||
"name": "ml.m5d.4xlarge",
|
||||
"vcpuNum": 16
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 16,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 128,
|
||||
"name": "ml.m5d.8xlarge",
|
||||
"vcpuNum": 32
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 17,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 192,
|
||||
"name": "ml.m5d.12xlarge",
|
||||
"vcpuNum": 48
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 18,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 256,
|
||||
"name": "ml.m5d.16xlarge",
|
||||
"vcpuNum": 64
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 19,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 384,
|
||||
"name": "ml.m5d.24xlarge",
|
||||
"vcpuNum": 96
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 20,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": true,
|
||||
"memoryGiB": 0,
|
||||
"name": "ml.geospatial.interactive",
|
||||
"supportedImageNames": [
|
||||
"sagemaker-geospatial-v1-0"
|
||||
],
|
||||
"vcpuNum": 0
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 21,
|
||||
"_isFastLaunch": true,
|
||||
"category": "Compute optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 4,
|
||||
"name": "ml.c5.large",
|
||||
"vcpuNum": 2
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 22,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Compute optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 8,
|
||||
"name": "ml.c5.xlarge",
|
||||
"vcpuNum": 4
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 23,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Compute optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 16,
|
||||
"name": "ml.c5.2xlarge",
|
||||
"vcpuNum": 8
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 24,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Compute optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 32,
|
||||
"name": "ml.c5.4xlarge",
|
||||
"vcpuNum": 16
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 25,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Compute optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 72,
|
||||
"name": "ml.c5.9xlarge",
|
||||
"vcpuNum": 36
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 26,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Compute optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 96,
|
||||
"name": "ml.c5.12xlarge",
|
||||
"vcpuNum": 48
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 27,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Compute optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 144,
|
||||
"name": "ml.c5.18xlarge",
|
||||
"vcpuNum": 72
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 28,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Compute optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 192,
|
||||
"name": "ml.c5.24xlarge",
|
||||
"vcpuNum": 96
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 29,
|
||||
"_isFastLaunch": true,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 16,
|
||||
"name": "ml.g4dn.xlarge",
|
||||
"vcpuNum": 4
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 30,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 32,
|
||||
"name": "ml.g4dn.2xlarge",
|
||||
"vcpuNum": 8
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 31,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 64,
|
||||
"name": "ml.g4dn.4xlarge",
|
||||
"vcpuNum": 16
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 32,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 128,
|
||||
"name": "ml.g4dn.8xlarge",
|
||||
"vcpuNum": 32
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 33,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 4,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 192,
|
||||
"name": "ml.g4dn.12xlarge",
|
||||
"vcpuNum": 48
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 34,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 256,
|
||||
"name": "ml.g4dn.16xlarge",
|
||||
"vcpuNum": 64
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 35,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 61,
|
||||
"name": "ml.p3.2xlarge",
|
||||
"vcpuNum": 8
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 36,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 4,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 244,
|
||||
"name": "ml.p3.8xlarge",
|
||||
"vcpuNum": 32
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 37,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 8,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 488,
|
||||
"name": "ml.p3.16xlarge",
|
||||
"vcpuNum": 64
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 38,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 8,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 768,
|
||||
"name": "ml.p3dn.24xlarge",
|
||||
"vcpuNum": 96
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 39,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Memory Optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 16,
|
||||
"name": "ml.r5.large",
|
||||
"vcpuNum": 2
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 40,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Memory Optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 32,
|
||||
"name": "ml.r5.xlarge",
|
||||
"vcpuNum": 4
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 41,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Memory Optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 64,
|
||||
"name": "ml.r5.2xlarge",
|
||||
"vcpuNum": 8
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 42,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Memory Optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 128,
|
||||
"name": "ml.r5.4xlarge",
|
||||
"vcpuNum": 16
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 43,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Memory Optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 256,
|
||||
"name": "ml.r5.8xlarge",
|
||||
"vcpuNum": 32
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 44,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Memory Optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 384,
|
||||
"name": "ml.r5.12xlarge",
|
||||
"vcpuNum": 48
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 45,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Memory Optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 512,
|
||||
"name": "ml.r5.16xlarge",
|
||||
"vcpuNum": 64
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 46,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Memory Optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 768,
|
||||
"name": "ml.r5.24xlarge",
|
||||
"vcpuNum": 96
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 47,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 16,
|
||||
"name": "ml.g5.xlarge",
|
||||
"vcpuNum": 4
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 48,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 32,
|
||||
"name": "ml.g5.2xlarge",
|
||||
"vcpuNum": 8
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 49,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 64,
|
||||
"name": "ml.g5.4xlarge",
|
||||
"vcpuNum": 16
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 50,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 128,
|
||||
"name": "ml.g5.8xlarge",
|
||||
"vcpuNum": 32
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 51,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 256,
|
||||
"name": "ml.g5.16xlarge",
|
||||
"vcpuNum": 64
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 52,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 4,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 192,
|
||||
"name": "ml.g5.12xlarge",
|
||||
"vcpuNum": 48
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 53,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 4,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 384,
|
||||
"name": "ml.g5.24xlarge",
|
||||
"vcpuNum": 96
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 54,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 8,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 768,
|
||||
"name": "ml.g5.48xlarge",
|
||||
"vcpuNum": 192
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 55,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 8,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 1152,
|
||||
"name": "ml.p4d.24xlarge",
|
||||
"vcpuNum": 96
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 56,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 8,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 1152,
|
||||
"name": "ml.p4de.24xlarge",
|
||||
"vcpuNum": 96
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 57,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 32,
|
||||
"name": "ml.trn1.2xlarge",
|
||||
"vcpuNum": 8
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 58,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 512,
|
||||
"name": "ml.trn1.32xlarge",
|
||||
"vcpuNum": 128
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 59,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 512,
|
||||
"name": "ml.trn1n.32xlarge",
|
||||
"vcpuNum": 128
|
||||
}
|
||||
],
|
||||
"instance_type": "ml.t3.medium",
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.14"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
@@ -0,0 +1,946 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"# Amazon SageMaker Using AWS Large Model Inference (LMI) Deep Learning Container\n",
|
||||
"\n",
|
||||
"**Recommended Kernel(s):** You can run this notebook with any Amazon SageMaker Studio kernel. We recommend using the Data Science 3.0 kernel.\n",
|
||||
"\n",
|
||||
"This notebook demonstrates how to deploy a [`meta-llama/Meta-Llama-3-8B`](https://huggingface.co/meta-llama/Meta-Llama-3-8B) HuggingFace model to a SageMaker Endpoint for text generation. In this example, the SageMaker-managed [LMI (Large Model Inference)](https://docs.djl.ai/docs/serving/serving/docs/large_model_inference.html) Docker image will serve as the inference image. LMI images feature a [DJL serving](https://github.com/deepjavalibrary/djl-serving) stack powered by the [Deep Java Library](https://djl.ai/). \n",
|
||||
"\n",
|
||||
"## 1. Dependency Installation\n",
|
||||
"### 1.1. Python Dependencies & Imports\n",
|
||||
"This notebook requires the following Python dependencies:\n",
|
||||
"* AWS [`sagemaker_core`]()\n",
|
||||
"\n",
|
||||
"Let's install or upgrade these dependencies using the following command:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%pip install pip --upgrade --quiet\n",
|
||||
"%pip install sagemaker-core huggingface_hub --quiet"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"import os\n",
|
||||
"import sagemaker\n",
|
||||
"\n",
|
||||
"from sagemaker_core.helper.session_helper import get_execution_role, Session\n",
|
||||
"import pathlib\n",
|
||||
"import huggingface_hub"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PROJECT_ROOT_DIR = pathlib.Path.cwd()\n",
|
||||
"\n",
|
||||
"SM_SESSION = Session()\n",
|
||||
"\n",
|
||||
"REGION = SM_SESSION._region_name\n",
|
||||
"\n",
|
||||
"SM_DEFAULT_EXECUTION_ROLE_ARN = get_execution_role()\n",
|
||||
"\n",
|
||||
"INSTANCE_TYPE = \"ml.g5.12xlarge\"\n",
|
||||
"\n",
|
||||
"# See https://github.com/aws/deep-learning-containers/blob/master/available_images.md#large-model-inference-containers for lastest DLC.\n",
|
||||
"container_image_uri = (\n",
|
||||
" f\"763104351884.dkr.ecr.{REGION}.amazonaws.com/djl-inference:0.29.0-tensorrtllm0.11.0-cu124\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"default_bucket = sagemaker.Session().default_bucket()\n",
|
||||
"default_bucket_prefix = sagemaker.Session().default_bucket_prefix\n",
|
||||
"\n",
|
||||
"MODEL = \"meta-llama/Meta-Llama-3-8B\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Download custom draft model"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"## 2. Deploy Speculative Decoding-Enabled Endpoint\n",
|
||||
"### 2.1. Endpoint Deployment"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We will configure what models to use and server startup parameters and via environment variables. \n",
|
||||
"\n",
|
||||
"* `OPTION_MODEL_ID` points to the HF Model ID or base S3 prefix of the model artifacts\n",
|
||||
"You can read about the other config paramters in LMI [documentation](https://github.com/deepjavalibrary/djl-serving/tree/master/serving/docs/lmi)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"environment = {\n",
|
||||
" \"HF_MODEL_ID\": MODEL,\n",
|
||||
" \"OPTION_GPU_MEMORY_UTILIZATION\": \"0.85\",\n",
|
||||
" \"HF_TOKEN\": \"YourHFToken\",\n",
|
||||
"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.shapes import ContainerDefinition, ProductionVariant\n",
|
||||
"from sagemaker_core.resources import Model, EndpointConfig, Endpoint\n",
|
||||
"from time import gmtime, strftime"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"container_defintion = ContainerDefinition(image=container_image_uri, environment=environment)\n",
|
||||
"container_defintion.environment = environment"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"model_name = f'llama-3-8B-hugging-face-{strftime(\"%H-%M-%S\", gmtime())}'\n",
|
||||
"\n",
|
||||
"model = Model.create(\n",
|
||||
" model_name=model_name,\n",
|
||||
" primary_container=container_defintion,\n",
|
||||
" execution_role_arn=SM_DEFAULT_EXECUTION_ROLE_ARN,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"tags": [],
|
||||
"vscode": {
|
||||
"languageId": "plaintext"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"Start-up of LLM inference containers can last longer than smaller models, mainly due to longer model downloading and loading times. Timeout values need to be increased accordingly from their default values. Each endpoint deployment takes a few minutes. We also set `routing_strategy` which would benefit us if we were to back our endpoint with multiple instances."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.shapes import ProductionVariantRoutingConfig\n",
|
||||
"\n",
|
||||
"routing_config = ProductionVariantRoutingConfig(routing_strategy=\"LEAST_OUTSTANDING_REQUESTS\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"endpoint_config = EndpointConfig.create(\n",
|
||||
" endpoint_config_name=model_name,\n",
|
||||
" production_variants=[\n",
|
||||
" ProductionVariant(\n",
|
||||
" variant_name=model_name,\n",
|
||||
" initial_instance_count=1,\n",
|
||||
" instance_type=INSTANCE_TYPE,\n",
|
||||
" model_name=model,\n",
|
||||
" container_startup_health_check_timeout_in_seconds=3600,\n",
|
||||
" model_data_download_timeout_in_seconds=3600,\n",
|
||||
" routing_config=routing_config,\n",
|
||||
" )\n",
|
||||
" ],\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"endpoint = Endpoint.create(\n",
|
||||
" endpoint_name=model_name,\n",
|
||||
" endpoint_config_name=endpoint_config, # Pass `EndpointConfig` object created above\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"This cells will block until the endpoint is deployed, which is necessary for the following steps."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"endpoint.wait_for_status(\"InService\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"### Endpoint invocation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"Let's invoke our endpoint and get a sample response."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response = endpoint.invoke(\n",
|
||||
" body=json.dumps(\n",
|
||||
" {\n",
|
||||
" \"inputs\": [\"What is the capital of France?\"],\n",
|
||||
" \"max_new_tokens\": 512,\n",
|
||||
" \"temperature\": 0.0,\n",
|
||||
" }\n",
|
||||
" ),\n",
|
||||
" content_type=\"application/json\",\n",
|
||||
")\n",
|
||||
"response[\"Body\"].read()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 3. Clean Up Endpoint"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Delete any sagemaker core resource objects created in this notebook\n",
|
||||
"def delete_all_sagemaker_resources():\n",
|
||||
" all_objects = list(locals().values()) + list(globals().values())\n",
|
||||
" deletable_objects = [\n",
|
||||
" obj\n",
|
||||
" for obj in all_objects\n",
|
||||
" if hasattr(obj, \"delete\") and obj.__class__.__module__ == \"sagemaker_core.main.resources\"\n",
|
||||
" ]\n",
|
||||
"\n",
|
||||
" for obj in deletable_objects:\n",
|
||||
" obj.delete()\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"delete_all_sagemaker_resources()"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"availableInstances": [
|
||||
{
|
||||
"_defaultOrder": 0,
|
||||
"_isFastLaunch": true,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 4,
|
||||
"name": "ml.t3.medium",
|
||||
"vcpuNum": 2
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 1,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 8,
|
||||
"name": "ml.t3.large",
|
||||
"vcpuNum": 2
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 2,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 16,
|
||||
"name": "ml.t3.xlarge",
|
||||
"vcpuNum": 4
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 3,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 32,
|
||||
"name": "ml.t3.2xlarge",
|
||||
"vcpuNum": 8
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 4,
|
||||
"_isFastLaunch": true,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 8,
|
||||
"name": "ml.m5.large",
|
||||
"vcpuNum": 2
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 5,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 16,
|
||||
"name": "ml.m5.xlarge",
|
||||
"vcpuNum": 4
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 6,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 32,
|
||||
"name": "ml.m5.2xlarge",
|
||||
"vcpuNum": 8
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 7,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 64,
|
||||
"name": "ml.m5.4xlarge",
|
||||
"vcpuNum": 16
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 8,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 128,
|
||||
"name": "ml.m5.8xlarge",
|
||||
"vcpuNum": 32
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 9,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 192,
|
||||
"name": "ml.m5.12xlarge",
|
||||
"vcpuNum": 48
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 10,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 256,
|
||||
"name": "ml.m5.16xlarge",
|
||||
"vcpuNum": 64
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 11,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 384,
|
||||
"name": "ml.m5.24xlarge",
|
||||
"vcpuNum": 96
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 12,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 8,
|
||||
"name": "ml.m5d.large",
|
||||
"vcpuNum": 2
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 13,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 16,
|
||||
"name": "ml.m5d.xlarge",
|
||||
"vcpuNum": 4
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 14,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 32,
|
||||
"name": "ml.m5d.2xlarge",
|
||||
"vcpuNum": 8
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 15,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 64,
|
||||
"name": "ml.m5d.4xlarge",
|
||||
"vcpuNum": 16
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 16,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 128,
|
||||
"name": "ml.m5d.8xlarge",
|
||||
"vcpuNum": 32
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 17,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 192,
|
||||
"name": "ml.m5d.12xlarge",
|
||||
"vcpuNum": 48
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 18,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 256,
|
||||
"name": "ml.m5d.16xlarge",
|
||||
"vcpuNum": 64
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 19,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 384,
|
||||
"name": "ml.m5d.24xlarge",
|
||||
"vcpuNum": 96
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 20,
|
||||
"_isFastLaunch": false,
|
||||
"category": "General purpose",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": true,
|
||||
"memoryGiB": 0,
|
||||
"name": "ml.geospatial.interactive",
|
||||
"supportedImageNames": [
|
||||
"sagemaker-geospatial-v1-0"
|
||||
],
|
||||
"vcpuNum": 0
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 21,
|
||||
"_isFastLaunch": true,
|
||||
"category": "Compute optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 4,
|
||||
"name": "ml.c5.large",
|
||||
"vcpuNum": 2
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 22,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Compute optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 8,
|
||||
"name": "ml.c5.xlarge",
|
||||
"vcpuNum": 4
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 23,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Compute optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 16,
|
||||
"name": "ml.c5.2xlarge",
|
||||
"vcpuNum": 8
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 24,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Compute optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 32,
|
||||
"name": "ml.c5.4xlarge",
|
||||
"vcpuNum": 16
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 25,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Compute optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 72,
|
||||
"name": "ml.c5.9xlarge",
|
||||
"vcpuNum": 36
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 26,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Compute optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 96,
|
||||
"name": "ml.c5.12xlarge",
|
||||
"vcpuNum": 48
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 27,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Compute optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 144,
|
||||
"name": "ml.c5.18xlarge",
|
||||
"vcpuNum": 72
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 28,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Compute optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 192,
|
||||
"name": "ml.c5.24xlarge",
|
||||
"vcpuNum": 96
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 29,
|
||||
"_isFastLaunch": true,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 16,
|
||||
"name": "ml.g4dn.xlarge",
|
||||
"vcpuNum": 4
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 30,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 32,
|
||||
"name": "ml.g4dn.2xlarge",
|
||||
"vcpuNum": 8
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 31,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 64,
|
||||
"name": "ml.g4dn.4xlarge",
|
||||
"vcpuNum": 16
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 32,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 128,
|
||||
"name": "ml.g4dn.8xlarge",
|
||||
"vcpuNum": 32
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 33,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 4,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 192,
|
||||
"name": "ml.g4dn.12xlarge",
|
||||
"vcpuNum": 48
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 34,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 256,
|
||||
"name": "ml.g4dn.16xlarge",
|
||||
"vcpuNum": 64
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 35,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 61,
|
||||
"name": "ml.p3.2xlarge",
|
||||
"vcpuNum": 8
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 36,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 4,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 244,
|
||||
"name": "ml.p3.8xlarge",
|
||||
"vcpuNum": 32
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 37,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 8,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 488,
|
||||
"name": "ml.p3.16xlarge",
|
||||
"vcpuNum": 64
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 38,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 8,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 768,
|
||||
"name": "ml.p3dn.24xlarge",
|
||||
"vcpuNum": 96
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 39,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Memory Optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 16,
|
||||
"name": "ml.r5.large",
|
||||
"vcpuNum": 2
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 40,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Memory Optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 32,
|
||||
"name": "ml.r5.xlarge",
|
||||
"vcpuNum": 4
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 41,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Memory Optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 64,
|
||||
"name": "ml.r5.2xlarge",
|
||||
"vcpuNum": 8
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 42,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Memory Optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 128,
|
||||
"name": "ml.r5.4xlarge",
|
||||
"vcpuNum": 16
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 43,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Memory Optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 256,
|
||||
"name": "ml.r5.8xlarge",
|
||||
"vcpuNum": 32
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 44,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Memory Optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 384,
|
||||
"name": "ml.r5.12xlarge",
|
||||
"vcpuNum": 48
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 45,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Memory Optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 512,
|
||||
"name": "ml.r5.16xlarge",
|
||||
"vcpuNum": 64
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 46,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Memory Optimized",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 768,
|
||||
"name": "ml.r5.24xlarge",
|
||||
"vcpuNum": 96
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 47,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 16,
|
||||
"name": "ml.g5.xlarge",
|
||||
"vcpuNum": 4
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 48,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 32,
|
||||
"name": "ml.g5.2xlarge",
|
||||
"vcpuNum": 8
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 49,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 64,
|
||||
"name": "ml.g5.4xlarge",
|
||||
"vcpuNum": 16
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 50,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 128,
|
||||
"name": "ml.g5.8xlarge",
|
||||
"vcpuNum": 32
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 51,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 1,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 256,
|
||||
"name": "ml.g5.16xlarge",
|
||||
"vcpuNum": 64
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 52,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 4,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 192,
|
||||
"name": "ml.g5.12xlarge",
|
||||
"vcpuNum": 48
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 53,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 4,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 384,
|
||||
"name": "ml.g5.24xlarge",
|
||||
"vcpuNum": 96
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 54,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 8,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 768,
|
||||
"name": "ml.g5.48xlarge",
|
||||
"vcpuNum": 192
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 55,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 8,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 1152,
|
||||
"name": "ml.p4d.24xlarge",
|
||||
"vcpuNum": 96
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 56,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 8,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 1152,
|
||||
"name": "ml.p4de.24xlarge",
|
||||
"vcpuNum": 96
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 57,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 32,
|
||||
"name": "ml.trn1.2xlarge",
|
||||
"vcpuNum": 8
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 58,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 512,
|
||||
"name": "ml.trn1.32xlarge",
|
||||
"vcpuNum": 128
|
||||
},
|
||||
{
|
||||
"_defaultOrder": 59,
|
||||
"_isFastLaunch": false,
|
||||
"category": "Accelerated computing",
|
||||
"gpuNum": 0,
|
||||
"hideHardwareSpecs": false,
|
||||
"memoryGiB": 512,
|
||||
"name": "ml.trn1n.32xlarge",
|
||||
"vcpuNum": 128
|
||||
}
|
||||
],
|
||||
"instance_type": "ml.t3.medium",
|
||||
"kernelspec": {
|
||||
"display_name": "conda_pytorch_p310",
|
||||
"language": "python",
|
||||
"name": "conda_pytorch_p310"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.14"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
@@ -0,0 +1,901 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Distributed Data Processing using Apache Spark and SageMaker Processing\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n",
|
||||
"Apache Spark is a unified analytics engine for large-scale data processing. The Spark framework is often used within the context of machine learning workflows to run data transformation or feature engineering workloads at scale. Amazon SageMaker provides a set of prebuilt Docker images that include Apache Spark and other dependencies needed to run distributed data processing jobs on Amazon SageMaker. This example notebook demonstrates how to use the prebuilt Spark images on SageMaker Processing using the SageMaker Core Python SDK."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Runtime\n",
|
||||
"\n",
|
||||
"This notebook takes approximately 22 minutes to run."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Contents\n",
|
||||
"\n",
|
||||
"1. [Setup](#Setup)\n",
|
||||
"1. [Example 1: Running a basic PySpark application](#Example-1:-Running-a-basic-PySpark-application)\n",
|
||||
"1. [Example 2: Specify additional Python and jar file dependencies](#Example-2:-Specify-additional-Python-and-jar-file-dependencies)\n",
|
||||
"1. [Example 3: Run a Java/Scala Spark application](#Example-3:-Run-a-Java/Scala-Spark-application)\n",
|
||||
"1. [Example 4: Specifying additional Spark configuration](#Example-4:-Specifying-additional-Spark-configuration)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Setup"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Install the latest SageMaker Core Python SDK"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip uninstall sagemaker-core -y\n",
|
||||
"!pip install pip --upgrade --quiet\n",
|
||||
"!pip install sagemaker-core --upgrade"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"*Restart your notebook kernel after upgrading the SDK*"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Example 1: Running a basic PySpark application\n",
|
||||
"\n",
|
||||
"The first example is a basic Spark MLlib data processing script. This script will take a raw data set and do some transformations on it such as string indexing and one hot encoding.\n",
|
||||
"\n",
|
||||
"### Setup S3 bucket locations and roles\n",
|
||||
"\n",
|
||||
"First, setup some locations in the default SageMaker bucket to store the raw input datasets and the Spark job output. Here, you'll also define the role that will be used to run all SageMaker Processing jobs."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 93,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import logging\n",
|
||||
"import sagemaker\n",
|
||||
"from time import gmtime, strftime\n",
|
||||
"from sagemaker_core.helper.session_helper import get_execution_role, Session\n",
|
||||
"\n",
|
||||
"sagemaker_logger = logging.getLogger(\"sagemaker\")\n",
|
||||
"sagemaker_logger.setLevel(logging.INFO)\n",
|
||||
"sagemaker_logger.addHandler(logging.StreamHandler())\n",
|
||||
"\n",
|
||||
"sagemaker_session = Session()\n",
|
||||
"REGION_NAME = sagemaker_session._region_name\n",
|
||||
"role = get_execution_role()\n",
|
||||
"s3_bucket_name = sagemaker.Session().default_bucket()\n",
|
||||
"default_bucket_prefix = sagemaker.Session().default_bucket_prefix\n",
|
||||
"default_bucket_prefix_path = \"\"\n",
|
||||
"\n",
|
||||
"# If a default bucket prefix is specified, append it to the s3 path\n",
|
||||
"if default_bucket_prefix:\n",
|
||||
" default_bucket_prefix_path = f\"/{default_bucket_prefix}\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Next, you'll download the example dataset from a SageMaker staging bucket."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Fetch the dataset from the SageMaker bucket\n",
|
||||
"import boto3\n",
|
||||
"\n",
|
||||
"s3 = boto3.client(\"s3\")\n",
|
||||
"s3.download_file(\n",
|
||||
" f\"sagemaker-sample-files\", \"datasets/tabular/uci_abalone/abalone.csv\", \"./data/abalone.csv\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Write the PySpark script\n",
|
||||
"\n",
|
||||
"The source for a preprocessing script is in the cell below. The cell uses the `%%writefile` directive to save this file locally. This script does some basic feature engineering on a raw input dataset. In this example, the dataset is the [Abalone Data Set](https://archive.ics.uci.edu/ml/datasets/abalone) and the code below performs string indexing, one hot encoding, vector assembly, and combines them into a pipeline to perform these transformations in order. The script then does an 80-20 split to produce training and validation datasets as output."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%writefile ./code/preprocess.py\n",
|
||||
"from __future__ import print_function\n",
|
||||
"from __future__ import unicode_literals\n",
|
||||
"\n",
|
||||
"import argparse\n",
|
||||
"import csv\n",
|
||||
"import os\n",
|
||||
"import shutil\n",
|
||||
"import sys\n",
|
||||
"import time\n",
|
||||
"\n",
|
||||
"import pyspark\n",
|
||||
"from pyspark.sql import SparkSession\n",
|
||||
"from pyspark.ml import Pipeline\n",
|
||||
"from pyspark.ml.feature import (\n",
|
||||
" OneHotEncoder,\n",
|
||||
" StringIndexer,\n",
|
||||
" VectorAssembler,\n",
|
||||
" VectorIndexer,\n",
|
||||
")\n",
|
||||
"from pyspark.sql.functions import *\n",
|
||||
"from pyspark.sql.types import (\n",
|
||||
" DoubleType,\n",
|
||||
" StringType,\n",
|
||||
" StructField,\n",
|
||||
" StructType,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def csv_line(data):\n",
|
||||
" r = \",\".join(str(d) for d in data[1])\n",
|
||||
" return str(data[0]) + \",\" + r\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def main():\n",
|
||||
" parser = argparse.ArgumentParser(description=\"app inputs and outputs\")\n",
|
||||
" parser.add_argument(\"--s3_input_bucket\", type=str, help=\"s3 input bucket\")\n",
|
||||
" parser.add_argument(\"--s3_input_key_prefix\", type=str, help=\"s3 input key prefix\")\n",
|
||||
" parser.add_argument(\"--s3_output_bucket\", type=str, help=\"s3 output bucket\")\n",
|
||||
" parser.add_argument(\"--s3_output_key_prefix\", type=str, help=\"s3 output key prefix\")\n",
|
||||
" args = parser.parse_args()\n",
|
||||
"\n",
|
||||
" spark = SparkSession.builder.appName(\"PySparkApp\").getOrCreate()\n",
|
||||
"\n",
|
||||
" # This is needed to save RDDs which is the only way to write nested Dataframes into CSV format\n",
|
||||
" spark.sparkContext._jsc.hadoopConfiguration().set(\n",
|
||||
" \"mapred.output.committer.class\", \"org.apache.hadoop.mapred.FileOutputCommitter\"\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # Defining the schema corresponding to the input data. The input data does not contain the headers\n",
|
||||
" schema = StructType(\n",
|
||||
" [\n",
|
||||
" StructField(\"sex\", StringType(), True),\n",
|
||||
" StructField(\"length\", DoubleType(), True),\n",
|
||||
" StructField(\"diameter\", DoubleType(), True),\n",
|
||||
" StructField(\"height\", DoubleType(), True),\n",
|
||||
" StructField(\"whole_weight\", DoubleType(), True),\n",
|
||||
" StructField(\"shucked_weight\", DoubleType(), True),\n",
|
||||
" StructField(\"viscera_weight\", DoubleType(), True),\n",
|
||||
" StructField(\"shell_weight\", DoubleType(), True),\n",
|
||||
" StructField(\"rings\", DoubleType(), True),\n",
|
||||
" ]\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # Downloading the data from S3 into a Dataframe\n",
|
||||
" total_df = spark.read.csv(\n",
|
||||
" (\"s3://\" + os.path.join(args.s3_input_bucket, args.s3_input_key_prefix, \"abalone.csv\")),\n",
|
||||
" header=False,\n",
|
||||
" schema=schema,\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # StringIndexer on the sex column which has categorical value\n",
|
||||
" sex_indexer = StringIndexer(inputCol=\"sex\", outputCol=\"indexed_sex\")\n",
|
||||
"\n",
|
||||
" # one-hot-encoding is being performed on the string-indexed sex column (indexed_sex)\n",
|
||||
" sex_encoder = OneHotEncoder(inputCol=\"indexed_sex\", outputCol=\"sex_vec\")\n",
|
||||
"\n",
|
||||
" # vector-assembler will bring all the features to a 1D vector for us to save easily into CSV format\n",
|
||||
" assembler = VectorAssembler(\n",
|
||||
" inputCols=[\n",
|
||||
" \"sex_vec\",\n",
|
||||
" \"length\",\n",
|
||||
" \"diameter\",\n",
|
||||
" \"height\",\n",
|
||||
" \"whole_weight\",\n",
|
||||
" \"shucked_weight\",\n",
|
||||
" \"viscera_weight\",\n",
|
||||
" \"shell_weight\",\n",
|
||||
" ],\n",
|
||||
" outputCol=\"features\",\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # The pipeline is comprised of the steps added above\n",
|
||||
" pipeline = Pipeline(stages=[sex_indexer, sex_encoder, assembler])\n",
|
||||
"\n",
|
||||
" # This step trains the feature transformers\n",
|
||||
" model = pipeline.fit(total_df)\n",
|
||||
"\n",
|
||||
" # This step transforms the dataset with information obtained from the previous fit\n",
|
||||
" transformed_total_df = model.transform(total_df)\n",
|
||||
"\n",
|
||||
" # Split the overall dataset into 80-20 training and validation\n",
|
||||
" (train_df, validation_df) = transformed_total_df.randomSplit([0.8, 0.2])\n",
|
||||
"\n",
|
||||
" # Convert the train dataframe to RDD to save in CSV format and upload to S3\n",
|
||||
" train_rdd = train_df.rdd.map(lambda x: (x.rings, x.features))\n",
|
||||
" train_lines = train_rdd.map(csv_line)\n",
|
||||
" train_lines.saveAsTextFile(\n",
|
||||
" \"s3://\" + os.path.join(args.s3_output_bucket, args.s3_output_key_prefix, \"train\")\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # Convert the validation dataframe to RDD to save in CSV format and upload to S3\n",
|
||||
" validation_rdd = validation_df.rdd.map(lambda x: (x.rings, x.features))\n",
|
||||
" validation_lines = validation_rdd.map(csv_line)\n",
|
||||
" validation_lines.saveAsTextFile(\n",
|
||||
" \"s3://\" + os.path.join(args.s3_output_bucket, args.s3_output_key_prefix, \"validation\")\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"if __name__ == \"__main__\":\n",
|
||||
" main()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Run the SageMaker Processing Job\n",
|
||||
"\n",
|
||||
"Next, you'll use the `ProcessingJob` class to define a Spark job and run it using SageMaker Processing. A few things to note in the definition of the `ProcessingJob`:\n",
|
||||
"\n",
|
||||
"* This is a multi-node job with two m5.xlarge instances (which is specified via the `instance_count` and `instance_type` parameters)\n",
|
||||
"* Spark framework version 3.1 image is specified via the `image_uri` parameter\n",
|
||||
"* The PySpark script defined above is passed via via the `ProcessingInput` class\n",
|
||||
"* Command-line arguments to the PySpark script (such as the S3 input and output locations) are passed via the `arguments` parameter\n",
|
||||
"* Spark event logs will be offloaded to the S3 location in `spark_event_logs` folder.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.shapes import (\n",
|
||||
" ProcessingInput,\n",
|
||||
" ProcessingResources,\n",
|
||||
" AppSpecification,\n",
|
||||
" ProcessingS3Input,\n",
|
||||
" ProcessingOutputConfig,\n",
|
||||
")\n",
|
||||
"from sagemaker_core.shapes import (\n",
|
||||
" ProcessingResources,\n",
|
||||
" ProcessingClusterConfig,\n",
|
||||
" ProcessingOutput,\n",
|
||||
" ProcessingS3Output,\n",
|
||||
")\n",
|
||||
"from sagemaker_core.resources import ProcessingJob\n",
|
||||
"\n",
|
||||
"# Upload the raw input dataset to a unique S3 location\n",
|
||||
"timestamp_prefix = strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
|
||||
"prefix = \"sagemaker/spark-preprocess-demo/{}\".format(timestamp_prefix)\n",
|
||||
"\n",
|
||||
"# If a default bucket prefix is specified, append it to the s3 path\n",
|
||||
"if default_bucket_prefix:\n",
|
||||
" prefix = f\"{default_bucket_prefix}/{prefix}\"\n",
|
||||
"\n",
|
||||
"input_prefix_abalone = \"{}/input/raw/abalone\".format(prefix)\n",
|
||||
"input_preprocessed_prefix_abalone = \"{}/input/preprocessed/abalone\".format(prefix)\n",
|
||||
"\n",
|
||||
"base_job_name = \"sm-spark\"\n",
|
||||
"final_job_name = base_job_name + \"-\" + timestamp_prefix\n",
|
||||
"\n",
|
||||
"input_script_prefix = \"{}/input/code\".format(final_job_name)\n",
|
||||
"\n",
|
||||
"# uploading required data to S3 for reference\n",
|
||||
"# uploading abolone.csv to S3 bucket\n",
|
||||
"sagemaker_session.upload_data(\n",
|
||||
" path=\"./data/abalone.csv\", bucket=s3_bucket_name, key_prefix=input_prefix_abalone\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# uploading preprocess.py to S3 bucket\n",
|
||||
"sagemaker_session.upload_data(\n",
|
||||
" path=\"./code/preprocess.py\", bucket=s3_bucket_name, key_prefix=input_script_prefix\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# initializing ProcessingInputs,ProcessingResources,ProcessingOutputConfig and AppSpecification configurations\n",
|
||||
"processing_input = ProcessingInput(\n",
|
||||
" input_name=\"code\",\n",
|
||||
" s3_input=ProcessingS3Input(\n",
|
||||
" s3_uri=f\"s3://{s3_bucket_name}{default_bucket_prefix_path}/{final_job_name}/input/code/preprocess.py\",\n",
|
||||
" # s3_uri=\"s3://sagemaker-us-east-1-774297356213/sm-spark-2024-08-30-05-25-18-294/input/code/preprocess.py\",\n",
|
||||
" s3_data_type=\"S3Prefix\",\n",
|
||||
" local_path=\"/opt/ml/processing/input/code\",\n",
|
||||
" s3_input_mode=\"File\",\n",
|
||||
" ),\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"processing_output_config = ProcessingOutputConfig(\n",
|
||||
" outputs=[\n",
|
||||
" ProcessingOutput(\n",
|
||||
" output_name=\"output-1\",\n",
|
||||
" s3_output=ProcessingS3Output(\n",
|
||||
" s3_uri=f\"s3://{s3_bucket_name}/{prefix}/spark_event_logs\",\n",
|
||||
" local_path=\"/opt/ml/processing/spark-events/\",\n",
|
||||
" s3_upload_mode=\"Continuous\",\n",
|
||||
" ),\n",
|
||||
" )\n",
|
||||
" ]\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"processing_resources = ProcessingResources(\n",
|
||||
" cluster_config=ProcessingClusterConfig(\n",
|
||||
" instance_count=2, instance_type=\"ml.m5.xlarge\", volume_size_in_gb=30\n",
|
||||
" )\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"app_specification = AppSpecification(\n",
|
||||
" image_uri=\"173754725891.dkr.ecr.us-east-1.amazonaws.com/sagemaker-spark-processing:3.1-cpu\",\n",
|
||||
" container_entrypoint=[\n",
|
||||
" \"smspark-submit\",\n",
|
||||
" \"--local-spark-event-logs-dir\",\n",
|
||||
" \"/opt/ml/processing/spark-events/\",\n",
|
||||
" \"/opt/ml/processing/input/code/preprocess.py\",\n",
|
||||
" ],\n",
|
||||
" container_arguments=[\n",
|
||||
" \"--s3_input_bucket\",\n",
|
||||
" f\"{s3_bucket_name}\",\n",
|
||||
" \"--s3_input_key_prefix\",\n",
|
||||
" f\"{input_prefix_abalone}\",\n",
|
||||
" \"--s3_output_bucket\",\n",
|
||||
" f\"{s3_bucket_name}\",\n",
|
||||
" \"--s3_output_key_prefix\",\n",
|
||||
" f\"{input_preprocessed_prefix_abalone}\",\n",
|
||||
" ],\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Run the processing job\n",
|
||||
"processing_job_obj = ProcessingJob.create(\n",
|
||||
" processing_job_name=final_job_name,\n",
|
||||
" processing_resources=processing_resources,\n",
|
||||
" app_specification=app_specification,\n",
|
||||
" role_arn=role,\n",
|
||||
" processing_inputs=[processing_input],\n",
|
||||
" processing_output_config=processing_output_config,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"processing_job_obj.wait()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Validate Data Processing Results\n",
|
||||
"\n",
|
||||
"Next, validate the output of our data preprocessing job by looking at the first 5 rows of the output dataset."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"print(\"Top 5 rows from s3://{}/{}/train/\".format(s3_bucket_name, input_preprocessed_prefix_abalone))\n",
|
||||
"!aws s3 cp --quiet s3://$s3_bucket_name/$input_preprocessed_prefix_abalone/train/part-00000 - | head -n5"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Example 2: Specify additional Python and jar file dependencies\n",
|
||||
"\n",
|
||||
"The next example demonstrates a scenario where additional Python file dependencies are required by the PySpark script. You'll use a sample PySpark script that requires additional user-defined functions (UDFs) defined in a local module."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%writefile ./code/hello_py_spark_app.py\n",
|
||||
"import argparse\n",
|
||||
"import time\n",
|
||||
"\n",
|
||||
"# Import local module to test spark-submit--py-files dependencies\n",
|
||||
"import hello_py_spark_udfs as udfs\n",
|
||||
"from pyspark.sql import SparkSession, SQLContext\n",
|
||||
"from pyspark.sql.functions import udf\n",
|
||||
"from pyspark.sql.types import IntegerType\n",
|
||||
"import time\n",
|
||||
"\n",
|
||||
"if __name__ == \"__main__\":\n",
|
||||
" print(\"Hello World, this is PySpark!\")\n",
|
||||
"\n",
|
||||
" parser = argparse.ArgumentParser(description=\"inputs and outputs\")\n",
|
||||
" parser.add_argument(\"--input\", type=str, help=\"path to input data\")\n",
|
||||
" parser.add_argument(\"--output\", required=False, type=str, help=\"path to output data\")\n",
|
||||
" args = parser.parse_args()\n",
|
||||
" spark = SparkSession.builder.appName(\"SparkTestApp\").getOrCreate()\n",
|
||||
" sqlContext = SQLContext(spark.sparkContext)\n",
|
||||
"\n",
|
||||
" # Load test data set\n",
|
||||
" inputPath = args.input\n",
|
||||
" outputPath = args.output\n",
|
||||
" salesDF = spark.read.json(inputPath)\n",
|
||||
" salesDF.printSchema()\n",
|
||||
"\n",
|
||||
" salesDF.createOrReplaceTempView(\"sales\")\n",
|
||||
"\n",
|
||||
" # Define a UDF that doubles an integer column\n",
|
||||
" # The UDF function is imported from local module to test spark-submit--py-files dependencies\n",
|
||||
" double_udf_int = udf(udfs.double_x, IntegerType())\n",
|
||||
"\n",
|
||||
" # Save transformed data set to disk\n",
|
||||
" salesDF.select(\"date\", \"sale\", double_udf_int(\"sale\").alias(\"sale_double\")).write.json(\n",
|
||||
" outputPath\n",
|
||||
" )"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Creating `hello_py_spark_udfs.py` inside `code` folder"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%writefile ./code/hello_py_spark_udfs.py\n",
|
||||
"def double_x(x):\n",
|
||||
" return x + x"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Create a processing job with Python file dependencies\n",
|
||||
"\n",
|
||||
"Then, you'll create a processing job where the additional Python file dependencies are specified via the `py-files` input name in the `ProcessingInput` class."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.shapes import (\n",
|
||||
" ProcessingInput,\n",
|
||||
" ProcessingResources,\n",
|
||||
" AppSpecification,\n",
|
||||
" ProcessingS3Input,\n",
|
||||
" ProcessingOutputConfig,\n",
|
||||
")\n",
|
||||
"from sagemaker_core.shapes import (\n",
|
||||
" ProcessingResources,\n",
|
||||
" ProcessingClusterConfig,\n",
|
||||
" ProcessingOutput,\n",
|
||||
" ProcessingS3Output,\n",
|
||||
")\n",
|
||||
"from sagemaker_core.resources import ProcessingJob\n",
|
||||
"\n",
|
||||
"# Upload the raw input dataset to a unique S3 location\n",
|
||||
"timestamp_prefix = strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
|
||||
"prefix = \"sagemaker/spark-preprocess-demo/{}\".format(timestamp_prefix)\n",
|
||||
"\n",
|
||||
"# If a default bucket prefix is specified, append it to the s3 path\n",
|
||||
"if default_bucket_prefix:\n",
|
||||
" prefix = f\"{default_bucket_prefix}/{prefix}\"\n",
|
||||
"\n",
|
||||
"input_prefix_sales = \"{}/input/sales\".format(prefix)\n",
|
||||
"output_prefix_sales = \"{}/output/sales\".format(prefix)\n",
|
||||
"input_s3_uri = \"s3://{}/{}\".format(s3_bucket_name, input_prefix_sales)\n",
|
||||
"output_s3_uri = \"s3://{}/{}\".format(s3_bucket_name, output_prefix_sales)\n",
|
||||
"\n",
|
||||
"base_job_name = \"sm-spark-udfs\"\n",
|
||||
"final_job_name = base_job_name + \"-\" + timestamp_prefix\n",
|
||||
"\n",
|
||||
"input_script_prefix = \"{}/input/code\".format(final_job_name)\n",
|
||||
"input_pyfiles_prefix = \"{}/input/py-files\".format(final_job_name)\n",
|
||||
"\n",
|
||||
"# uploading required data to S3 for reference\n",
|
||||
"# uploading data.jsonl to S3\n",
|
||||
"sagemaker_session.upload_data(\n",
|
||||
" path=\"./data/data.jsonl\", bucket=s3_bucket_name, key_prefix=input_prefix_sales\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# uploading hello_py_spark_app.py to S3\n",
|
||||
"sagemaker_session.upload_data(\n",
|
||||
" path=\"./code/hello_py_spark_app.py\", bucket=s3_bucket_name, key_prefix=input_script_prefix\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# uploading hello_py_spark_udfs.py to S3\n",
|
||||
"sagemaker_session.upload_data(\n",
|
||||
" path=\"./code/hello_py_spark_udfs.py\", bucket=s3_bucket_name, key_prefix=input_pyfiles_prefix\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# initializing ProcessingInputs,ProcessingResources and AppSpecification configurations\n",
|
||||
"# providing processing script\n",
|
||||
"processing_input_code = ProcessingInput(\n",
|
||||
" input_name=\"code\",\n",
|
||||
" s3_input=ProcessingS3Input(\n",
|
||||
" s3_uri=f\"s3://{s3_bucket_name}{default_bucket_prefix_path}/{final_job_name}/input/code/hello_py_spark_app.py\",\n",
|
||||
" s3_data_type=\"S3Prefix\",\n",
|
||||
" local_path=\"/opt/ml/processing/input/code\",\n",
|
||||
" s3_input_mode=\"File\",\n",
|
||||
" ),\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# providing py files\n",
|
||||
"processing_input_pyfiles = ProcessingInput(\n",
|
||||
" input_name=\"py-files\",\n",
|
||||
" s3_input=ProcessingS3Input(\n",
|
||||
" s3_uri=f\"s3://{s3_bucket_name}{default_bucket_prefix_path}/{final_job_name}/input/py-files\",\n",
|
||||
" s3_data_type=\"S3Prefix\",\n",
|
||||
" local_path=\"/opt/ml/processing/input/py-files\",\n",
|
||||
" s3_input_mode=\"File\",\n",
|
||||
" ),\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# providing processing resources\n",
|
||||
"processing_resources = ProcessingResources(\n",
|
||||
" cluster_config=ProcessingClusterConfig(\n",
|
||||
" instance_count=2, instance_type=\"ml.m5.xlarge\", volume_size_in_gb=30\n",
|
||||
" )\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# providing app specification\n",
|
||||
"app_specification = AppSpecification(\n",
|
||||
" image_uri=\"173754725891.dkr.ecr.us-east-1.amazonaws.com/sagemaker-spark-processing:3.1-cpu\",\n",
|
||||
" container_entrypoint=[\n",
|
||||
" \"smspark-submit\",\n",
|
||||
" \"--py-files\",\n",
|
||||
" \"/opt/ml/processing/input/py-files\",\n",
|
||||
" \"/opt/ml/processing/input/code/hello_py_spark_app.py\",\n",
|
||||
" ],\n",
|
||||
" container_arguments=[\n",
|
||||
" \"--input\",\n",
|
||||
" f\"s3://{s3_bucket_name}/{input_prefix_sales}\",\n",
|
||||
" \"--output\",\n",
|
||||
" f\"s3://{s3_bucket_name}/{output_prefix_sales}\",\n",
|
||||
" ],\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Run the processing job\n",
|
||||
"processing_job_obj = ProcessingJob.create(\n",
|
||||
" processing_job_name=final_job_name,\n",
|
||||
" processing_resources=processing_resources,\n",
|
||||
" app_specification=app_specification,\n",
|
||||
" role_arn=role,\n",
|
||||
" processing_inputs=[processing_input_code, processing_input_pyfiles],\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"processing_job_obj.wait()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Validate Data Processing Results\n",
|
||||
"\n",
|
||||
"Next, validate the output of the Spark job by ensuring that the output URI contains the Spark `_SUCCESS` file along with the output json lines file."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"print(\"Output files in {}\".format(output_s3_uri))\n",
|
||||
"!aws s3 ls $output_s3_uri/"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Example 3: Run a Java/Scala Spark application\n",
|
||||
"\n",
|
||||
"In the next example, you'll take a Spark application jar (located in `./code/spark-test-app.jar`) that is already built and run it using SageMaker Processing."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.shapes import (\n",
|
||||
" ProcessingInput,\n",
|
||||
" ProcessingResources,\n",
|
||||
" AppSpecification,\n",
|
||||
" ProcessingS3Input,\n",
|
||||
" ProcessingOutputConfig,\n",
|
||||
")\n",
|
||||
"from sagemaker_core.shapes import (\n",
|
||||
" ProcessingResources,\n",
|
||||
" ProcessingClusterConfig,\n",
|
||||
" ProcessingOutput,\n",
|
||||
" ProcessingS3Output,\n",
|
||||
")\n",
|
||||
"from sagemaker_core.resources import ProcessingJob\n",
|
||||
"\n",
|
||||
"# Upload the raw input dataset to S3\n",
|
||||
"timestamp_prefix = strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
|
||||
"prefix = \"sagemaker/spark-preprocess-demo/{}\".format(timestamp_prefix)\n",
|
||||
"# If a default bucket prefix is specified, append it to the s3 path\n",
|
||||
"if default_bucket_prefix:\n",
|
||||
" prefix = f\"{default_bucket_prefix}/{prefix}\"\n",
|
||||
"\n",
|
||||
"input_prefix_sales = \"{}/input/sales\".format(prefix)\n",
|
||||
"output_prefix_sales = \"{}/output/sales\".format(prefix)\n",
|
||||
"input_s3_uri = \"s3://{}/{}\".format(s3_bucket_name, input_prefix_sales)\n",
|
||||
"output_s3_uri = \"s3://{}/{}\".format(s3_bucket_name, output_prefix_sales)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"base_job_name = \"sm-spark-java\"\n",
|
||||
"final_job_name = base_job_name + \"-\" + timestamp_prefix\n",
|
||||
"\n",
|
||||
"input_script_prefix = \"{}/input/code\".format(final_job_name)\n",
|
||||
"input_pyfiles_prefix = \"{}/input/py-files\".format(final_job_name)\n",
|
||||
"\n",
|
||||
"# uploading required data to S3 for reference\n",
|
||||
"sagemaker_session.upload_data(\n",
|
||||
" path=\"./data/data.jsonl\", bucket=s3_bucket_name, key_prefix=input_prefix_sales\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"sagemaker_session.upload_data(\n",
|
||||
" path=\"./code/spark-test-app.jar\", bucket=s3_bucket_name, key_prefix=input_script_prefix\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# initializing ProcessingInputs,ProcessingResources and AppSpecification configurations\n",
|
||||
"processing_input_code = ProcessingInput(\n",
|
||||
" input_name=\"code\",\n",
|
||||
" s3_input=ProcessingS3Input(\n",
|
||||
" s3_uri=f\"s3://{s3_bucket_name}{default_bucket_prefix_path}/{final_job_name}/input/code/spark-test-app.jar\",\n",
|
||||
" # s3_uri=\"s3://sagemaker-us-east-1-774297356213/sm-spark-2024-08-30-05-25-18-294/input/code/preprocess.py\",\n",
|
||||
" s3_data_type=\"S3Prefix\",\n",
|
||||
" local_path=\"/opt/ml/processing/input/code\",\n",
|
||||
" s3_input_mode=\"File\",\n",
|
||||
" ),\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"processing_resources = ProcessingResources(\n",
|
||||
" cluster_config=ProcessingClusterConfig(\n",
|
||||
" instance_count=2, instance_type=\"ml.m5.xlarge\", volume_size_in_gb=30\n",
|
||||
" )\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"app_specification = AppSpecification(\n",
|
||||
" image_uri=\"173754725891.dkr.ecr.us-east-1.amazonaws.com/sagemaker-spark-processing:3.1-cpu\",\n",
|
||||
" container_entrypoint=[\n",
|
||||
" \"smspark-submit\",\n",
|
||||
" \"--class\",\n",
|
||||
" \"com.amazonaws.sagemaker.spark.test.HelloJavaSparkApp\",\n",
|
||||
" \"/opt/ml/processing/input/code/spark-test-app.jar\",\n",
|
||||
" ],\n",
|
||||
" container_arguments=[\n",
|
||||
" \"--input\",\n",
|
||||
" f\"s3://{s3_bucket_name}/{input_prefix_sales}\",\n",
|
||||
" \"--output\",\n",
|
||||
" f\"s3://{s3_bucket_name}/{output_prefix_sales}\",\n",
|
||||
" ],\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Run the processing job\n",
|
||||
"processing_job_obj = ProcessingJob.create(\n",
|
||||
" processing_job_name=final_job_name,\n",
|
||||
" processing_resources=processing_resources,\n",
|
||||
" app_specification=app_specification,\n",
|
||||
" role_arn=role,\n",
|
||||
" processing_inputs=[processing_input_code, processing_input_pyfiles],\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# waiting for the processing job to be completed\n",
|
||||
"processing_job_obj.wait()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Example 4: Specifying additional Spark configuration\n",
|
||||
"\n",
|
||||
"Overriding Spark configuration is crucial for a number of tasks such as tuning your Spark application or configuring the Hive metastore. Using the SageMaker Python SDK, you can easily override Spark/Hive/Hadoop configuration.\n",
|
||||
"\n",
|
||||
"The next example demonstrates this by overriding Spark executor memory/cores.\n",
|
||||
"\n",
|
||||
"For more information on configuring your Spark application, see the EMR documentation on [Configuring Applications](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html)\n",
|
||||
"\n",
|
||||
"#### Creating configuration.json file for overriding Spark executor memory/cores "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%writefile ./code/configuration.json\n",
|
||||
"[{\"Classification\": \"spark-defaults\", \"Properties\": {\"spark.executor.memory\": \"2g\", \"spark.executor.cores\": \"1\"}}]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker_core.shapes import (\n",
|
||||
" ProcessingInput,\n",
|
||||
" ProcessingResources,\n",
|
||||
" AppSpecification,\n",
|
||||
" ProcessingS3Input,\n",
|
||||
" ProcessingOutputConfig,\n",
|
||||
")\n",
|
||||
"from sagemaker_core.shapes import (\n",
|
||||
" ProcessingResources,\n",
|
||||
" ProcessingClusterConfig,\n",
|
||||
" ProcessingOutput,\n",
|
||||
" ProcessingS3Output,\n",
|
||||
")\n",
|
||||
"from sagemaker_core.resources import ProcessingJob\n",
|
||||
"\n",
|
||||
"# Upload the raw input dataset to a unique S3 location\n",
|
||||
"timestamp_prefix = strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
|
||||
"prefix = \"sagemaker/spark-preprocess-demo/{}\".format(timestamp_prefix)\n",
|
||||
"\n",
|
||||
"# If a default bucket prefix is specified, append it to the s3 path\n",
|
||||
"if default_bucket_prefix:\n",
|
||||
" prefix = f\"{default_bucket_prefix}/{prefix}\"\n",
|
||||
"\n",
|
||||
"input_prefix_abalone = \"{}/input/raw/abalone\".format(prefix)\n",
|
||||
"input_preprocessed_prefix_abalone = \"{}/input/preprocessed/abalone\".format(prefix)\n",
|
||||
"\n",
|
||||
"# base job name\n",
|
||||
"base_job_name = \"sm-spark\"\n",
|
||||
"final_job_name = base_job_name + \"-\" + timestamp_prefix\n",
|
||||
"\n",
|
||||
"input_script_prefix = \"{}/input/code\".format(final_job_name)\n",
|
||||
"input_conf_prefix = \"{}/input/conf\".format(final_job_name)\n",
|
||||
"\n",
|
||||
"# uploading required data to S3 for reference\n",
|
||||
"sagemaker_session.upload_data(\n",
|
||||
" path=\"./data/abalone.csv\", bucket=s3_bucket_name, key_prefix=input_prefix_abalone\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"sagemaker_session.upload_data(\n",
|
||||
" path=\"./code/preprocess.py\", bucket=s3_bucket_name, key_prefix=input_script_prefix\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"sagemaker_session.upload_data(\n",
|
||||
" path=\"./code/configuration.json\", bucket=s3_bucket_name, key_prefix=input_conf_prefix\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# initializing ProcessingInputs,ProcessingResources and AppSpecification configurations\n",
|
||||
"processing_input_code = ProcessingInput(\n",
|
||||
" input_name=\"code\",\n",
|
||||
" s3_input=ProcessingS3Input(\n",
|
||||
" s3_uri=f\"s3://{s3_bucket_name}{default_bucket_prefix_path}/{final_job_name}/input/code/preprocess.py\",\n",
|
||||
" # s3_uri=\"s3://sagemaker-us-east-1-774297356213/sm-spark-2024-08-30-05-25-18-294/input/code/preprocess.py\",\n",
|
||||
" s3_data_type=\"S3Prefix\",\n",
|
||||
" local_path=\"/opt/ml/processing/input/code\",\n",
|
||||
" s3_input_mode=\"File\",\n",
|
||||
" ),\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"processing_input_conf = ProcessingInput(\n",
|
||||
" input_name=\"conf\",\n",
|
||||
" s3_input=ProcessingS3Input(\n",
|
||||
" s3_uri=f\"s3://{s3_bucket_name}{default_bucket_prefix_path}/{final_job_name}/input/conf/configuration.json\",\n",
|
||||
" # s3_uri=\"s3://sagemaker-us-east-1-774297356213/sm-spark-2024-08-30-05-25-18-294/input/code/preprocess.py\",\n",
|
||||
" s3_data_type=\"S3Prefix\",\n",
|
||||
" local_path=\"/opt/ml/processing/input/conf\",\n",
|
||||
" s3_input_mode=\"File\",\n",
|
||||
" ),\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"processing_resources = ProcessingResources(\n",
|
||||
" cluster_config=ProcessingClusterConfig(\n",
|
||||
" instance_count=2, instance_type=\"ml.m5.xlarge\", volume_size_in_gb=30\n",
|
||||
" )\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"app_specification = AppSpecification(\n",
|
||||
" image_uri=\"173754725891.dkr.ecr.us-east-1.amazonaws.com/sagemaker-spark-processing:3.1-cpu\",\n",
|
||||
" container_entrypoint=[\"smspark-submit\", \"/opt/ml/processing/input/code/preprocess.py\"],\n",
|
||||
" container_arguments=[\n",
|
||||
" \"--s3_input_bucket\",\n",
|
||||
" s3_bucket_name,\n",
|
||||
" \"--s3_input_key_prefix\",\n",
|
||||
" input_prefix_abalone,\n",
|
||||
" \"--s3_output_bucket\",\n",
|
||||
" s3_bucket_name,\n",
|
||||
" \"--s3_output_key_prefix\",\n",
|
||||
" input_preprocessed_prefix_abalone,\n",
|
||||
" ],\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Run the processing job\n",
|
||||
"processing_job_obj = ProcessingJob.create(\n",
|
||||
" processing_job_name=final_job_name,\n",
|
||||
" processing_resources=processing_resources,\n",
|
||||
" app_specification=app_specification,\n",
|
||||
" role_arn=role,\n",
|
||||
" processing_inputs=[processing_input_code, processing_input_conf],\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# waiting for the processing job to be completed\n",
|
||||
"processing_job_obj.wait()"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"instance_type": "ml.t3.medium",
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.14"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
@@ -0,0 +1,604 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# SageMakerCore Overview of Resource Level Abstractions - XGBoost Training Example"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"## Introductions\n",
|
||||
"SageMakerCore is a Python SDK designed as a lightweight layer over boto3, the AWS SDK for Python. It is built on the concept of resource level abstractions, where SageMaker Resources are represented as Python classes. This approach enables SageMakerCore to simplify the management of SageMaker Resources and provide a more object-oriented programming interface.\n",
|
||||
"\n",
|
||||
"### Resource Level Abstraction\n",
|
||||
"Resource Level Abstractions can be best understood by examining how the AWS TrainingJob APIs are transfromed into a TrainingJob Python class abstraction in SageMakerCore.\n",
|
||||
"\n",
|
||||
"For instance, an AWS TrainingJob has the following APIs:\n",
|
||||
"1. CreateTrainingJob\n",
|
||||
"2. DescribeTrainingJob\n",
|
||||
"3. UpdateTrainingJob\n",
|
||||
"4. StopTrainingJob\n",
|
||||
"5. ListTrainingJobs\n",
|
||||
"\n",
|
||||
"In SageMakerCore, these APIs are encapsulated within a TrainingJob class that exposes these operations as methods and attributes. The details of the TrainingJob class are below:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```python\n",
|
||||
"class TrainingJob(Base):\n",
|
||||
" # Class attributes are mapped to describe_training_job response\n",
|
||||
" training_job_name: str\n",
|
||||
" training_job_arn: Optional[str] = Unassigned()\n",
|
||||
" tuning_job_arn: Optional[str] = Unassigned()\n",
|
||||
" labeling_job_arn: Optional[str] = Unassigned()\n",
|
||||
" auto_ml_job_arn: Optional[str] = Unassigned()\n",
|
||||
" model_artifacts: Optional[ModelArtifacts] = Unassigned()\n",
|
||||
" training_job_status: Optional[str] = Unassigned()\n",
|
||||
" ...\n",
|
||||
"\n",
|
||||
" @classmethod\n",
|
||||
" def create(): # Calls `create_training_job`\n",
|
||||
"\n",
|
||||
" @classmethod\n",
|
||||
" def get(): # Calls `describe_training_job`\n",
|
||||
"\n",
|
||||
" @classmethod\n",
|
||||
" def get_all(): # Calls `list_training_job`\n",
|
||||
"\n",
|
||||
" \n",
|
||||
" def update(): # Calls `update_training_job`\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" def stop(): # Calls `stop_training_job`\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" def refresh(): # Calls `describe_training_job` and refreshes instance attributes\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" def wait(): # Calls `describe_training_job` and waits for TrainingJob to enter terminal state\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Comparing Boto3 and SageMakerCore SDKs\n",
|
||||
"\n",
|
||||
"In this notebook, we create an AWS TrainingJob to train an XGBoost Container. We will be using both Boto3 and the SageMakerCore SDKs with the goal of highlighting and comparing the differences in user experience for performing operations such as creating, updating, waiting, and listing AWS TrainingJobs."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Install Latest SageMakerCore\n",
|
||||
"All SageMakerCore beta distributions will be released to a private s3 bucket. After being allowlisted, run the cells below to install the latest version of SageMakerCore from `s3://sagemaker-core-beta-artifacts/sagemaker_core-latest.tar.gz`\n",
|
||||
"\n",
|
||||
"Ensure you are using a kernel with python version >=3.8"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Uninstall previous version of sagemaker_core and restart kernel\n",
|
||||
"!pip uninstall sagemaker-core -y"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Install the latest version of sagemaker_core\n",
|
||||
"\n",
|
||||
"!pip install sagemaker-core --upgrade"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Check the version of sagemaker_core\n",
|
||||
"!pip show -v sagemaker-core"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Install Additional Packages"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Install additional packages\n",
|
||||
"\n",
|
||||
"!pip install -U scikit-learn pandas boto3"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Setup\n",
|
||||
"\n",
|
||||
"Let's start by specifying:\n",
|
||||
"- AWS region.\n",
|
||||
"- The IAM role arn used to give learning and hosting access to your data. Ensure your enviornment has AWS Credentials configured.\n",
|
||||
"- The S3 bucket that you want to use for storing training and model data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sagemaker\n",
|
||||
"\n",
|
||||
"from sagemaker_core.helper.session_helper import Session, get_execution_role\n",
|
||||
"from rich import print\n",
|
||||
"\n",
|
||||
"# Get region, role, bucket\n",
|
||||
"\n",
|
||||
"sagemaker_session = Session()\n",
|
||||
"region = sagemaker_session.boto_region_name\n",
|
||||
"role = get_execution_role()\n",
|
||||
"bucket = sagemaker.Session().default_bucket()\n",
|
||||
"default_bucket_prefix = sagemaker.Session().default_bucket_prefix\n",
|
||||
"print(role)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Load and Prepare Dataset\n",
|
||||
"For this example, we will be using the IRIS data set from `sklearn.datasets` to train our XGBoost container."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sklearn.datasets import load_iris\n",
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"\n",
|
||||
"# Get IRIS Data\n",
|
||||
"\n",
|
||||
"iris = load_iris()\n",
|
||||
"iris_df = pd.DataFrame(iris.data, columns=iris.feature_names)\n",
|
||||
"iris_df[\"target\"] = iris.target"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# Prepare Data\n",
|
||||
"\n",
|
||||
"os.makedirs(\"./data\", exist_ok=True)\n",
|
||||
"\n",
|
||||
"iris_df = iris_df[[\"target\"] + [col for col in iris_df.columns if col != \"target\"]]\n",
|
||||
"\n",
|
||||
"train_data, test_data = train_test_split(iris_df, test_size=0.2, random_state=42)\n",
|
||||
"\n",
|
||||
"train_data.to_csv(\"./data/train.csv\", index=False, header=False)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Upload Data to S3\n",
|
||||
"In this step, we will upload the train and test data to the S3 bucket configured earlier using `sagemaker_session.default_bucket()`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Upload Data\n",
|
||||
"\n",
|
||||
"prefix = \"DEMO-scikit-iris\"\n",
|
||||
"\n",
|
||||
"# If a default bucket prefix is specified, append it to the s3 path\n",
|
||||
"if default_bucket_prefix:\n",
|
||||
" prefix = f\"{default_bucket_prefix}/{prefix}\"\n",
|
||||
"\n",
|
||||
"TRAIN_DATA = \"train.csv\"\n",
|
||||
"DATA_DIRECTORY = \"data\"\n",
|
||||
"\n",
|
||||
"train_input = sagemaker_session.upload_data(\n",
|
||||
" DATA_DIRECTORY, bucket=bucket, key_prefix=\"{}/{}\".format(prefix, DATA_DIRECTORY)\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"s3_input_path = \"s3://{}/{}/data/{}\".format(bucket, prefix, TRAIN_DATA)\n",
|
||||
"s3_output_path = \"s3://{}/{}/output\".format(bucket, prefix)\n",
|
||||
"\n",
|
||||
"print(s3_input_path)\n",
|
||||
"print(s3_output_path)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Fetch the XGBoost Image URI\n",
|
||||
"In this step, we will fetch the XGBoost Image URI we will use as an input parameter when creating an AWS TrainingJob"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Image name is hardcoded here\n",
|
||||
"# Image name can be programatically got by using sagemaker package and calling image_uris.retrieve\n",
|
||||
"# Since that is a high level abstraction that has multiple dependencies, the image URIs functionalities will live in sagemaker (V2)\n",
|
||||
"\n",
|
||||
"image = \"433757028032.dkr.ecr.us-west-2.amazonaws.com/xgboost:latest\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Create TrainingJob with Boto3\n",
|
||||
"With the necessary setup completed, we can now create an AWS TrainingJob. First we will begin by creating a TrainingJob with Boto3 to understand what the experience is like when interecting directly with low-level APIs through Boto3.\n",
|
||||
"\n",
|
||||
"When executing the following cells there are a few things to note about the experience with Boto3:\n",
|
||||
"1. Boto3 dynamically generates the API operation methods like `create_training_job`. When a client is instantiated, the methods are generated from the JSON service model description and are not statically coded into the boto3 library.\n",
|
||||
"2. Boto3 returns a JSON response. As a result, users must either be familiar with the structure of these responses or refer to the documentation to parse them correctly.\n",
|
||||
"3. Boto3 client methods expect keyword arguments. Similar to the experience with JSON response, users must be familiar with what keyword argumnets are expected or refer to the documentation to pass them correctly.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Create TrainingJob with Boto3\n",
|
||||
"\n",
|
||||
"import time\n",
|
||||
"import boto3\n",
|
||||
"\n",
|
||||
"client = boto3.client(\"sagemaker\")\n",
|
||||
"job_name_boto = \"xgboost-iris-\" + time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime())\n",
|
||||
"\n",
|
||||
"response = client.create_training_job(\n",
|
||||
" TrainingJobName=job_name_boto,\n",
|
||||
" HyperParameters={\n",
|
||||
" \"objective\": \"multi:softmax\",\n",
|
||||
" \"num_class\": \"3\",\n",
|
||||
" \"num_round\": \"10\",\n",
|
||||
" \"eval_metric\": \"merror\",\n",
|
||||
" },\n",
|
||||
" AlgorithmSpecification={\"TrainingImage\": image, \"TrainingInputMode\": \"File\"},\n",
|
||||
" RoleArn=role,\n",
|
||||
" InputDataConfig=[\n",
|
||||
" {\n",
|
||||
" \"ChannelName\": \"train\",\n",
|
||||
" \"ContentType\": \"csv\",\n",
|
||||
" \"DataSource\": {\n",
|
||||
" \"S3DataSource\": {\n",
|
||||
" \"S3DataType\": \"S3Prefix\",\n",
|
||||
" \"S3Uri\": s3_input_path,\n",
|
||||
" \"S3DataDistributionType\": \"FullyReplicated\",\n",
|
||||
" }\n",
|
||||
" },\n",
|
||||
" \"CompressionType\": \"None\",\n",
|
||||
" \"RecordWrapperType\": \"None\",\n",
|
||||
" }\n",
|
||||
" ],\n",
|
||||
" OutputDataConfig={\"S3OutputPath\": s3_output_path},\n",
|
||||
" ResourceConfig={\n",
|
||||
" \"InstanceType\": \"ml.m4.xlarge\",\n",
|
||||
" \"InstanceCount\": 1,\n",
|
||||
" \"VolumeSizeInGB\": 30,\n",
|
||||
" },\n",
|
||||
" StoppingCondition={\"MaxRuntimeInSeconds\": 600},\n",
|
||||
")\n",
|
||||
"print(response)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Wait for TrainingJob with Boto3\n",
|
||||
"When a user creates a TrainingJob it is often the case that they would wish to wait on the TrainingJob to complete. Below is an example of how a user wait on a TrainingJob using Boto3. Notebly, this requires creating some logic to poll the TrainingJob using `describe_training_job` until the `TrainingJobStatus` is `'Failed'`, `'Completed'`, or `'Stopped'`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Wait for TrainingJob with Boto3\n",
|
||||
"import time\n",
|
||||
"\n",
|
||||
"while True:\n",
|
||||
" response = client.describe_training_job(TrainingJobName=job_name_boto)\n",
|
||||
" status = response[\"TrainingJobStatus\"]\n",
|
||||
" if status in [\"Failed\", \"Completed\", \"Stopped\"]:\n",
|
||||
" if status == \"Failed\":\n",
|
||||
" print(response[\"FailureReason\"])\n",
|
||||
" break\n",
|
||||
" print(\"-\", end=\"\")\n",
|
||||
" time.sleep(5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Create TrainingJob with SageMakerCore\n",
|
||||
"In this step we will use SageMakerCore to create a TrainingJob to understand what experience the object-oriented resource level abstractions provide for users.\n",
|
||||
"\n",
|
||||
"When executing the following cells, there are a few things to note about the experience with SageMakerCore:\n",
|
||||
"1. SageMakerCore generates Python classes and methods from the service model JSON, similar to Boto3. However, this generation is done prior to a release, resulting in a statically coded interface in the library.\n",
|
||||
"2. SageMakerCore adopts an object-oriented approach, providing users with clear visibility of available methods and attributes through type hinting and IDE IntelliSense\n",
|
||||
"3. Instead of returning JSON responses like Boto3, SageMakerCore returns objects. This allows users to access response attributes directly from the returned object, eliminating the need to parse JSON or refer to the documentation for structure details."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Create TrainingJob with SageMakerCore\n",
|
||||
"\n",
|
||||
"import time\n",
|
||||
"from sagemaker_core.resources import (\n",
|
||||
" TrainingJob,\n",
|
||||
" AlgorithmSpecification,\n",
|
||||
" Channel,\n",
|
||||
" DataSource,\n",
|
||||
" S3DataSource,\n",
|
||||
" OutputDataConfig,\n",
|
||||
" ResourceConfig,\n",
|
||||
" StoppingCondition,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"job_name_v3 = \"xgboost-iris-\" + time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime())\n",
|
||||
"\n",
|
||||
"training_job = TrainingJob.create(\n",
|
||||
" training_job_name=job_name_v3,\n",
|
||||
" hyper_parameters={\n",
|
||||
" \"objective\": \"multi:softmax\",\n",
|
||||
" \"num_class\": \"3\",\n",
|
||||
" \"num_round\": \"10\",\n",
|
||||
" \"eval_metric\": \"merror\",\n",
|
||||
" },\n",
|
||||
" algorithm_specification=AlgorithmSpecification(\n",
|
||||
" training_image=image, training_input_mode=\"File\"\n",
|
||||
" ),\n",
|
||||
" role_arn=role,\n",
|
||||
" input_data_config=[\n",
|
||||
" Channel(\n",
|
||||
" channel_name=\"train\",\n",
|
||||
" content_type=\"csv\",\n",
|
||||
" compression_type=\"None\",\n",
|
||||
" record_wrapper_type=\"None\",\n",
|
||||
" data_source=DataSource(\n",
|
||||
" s3_data_source=S3DataSource(\n",
|
||||
" s3_data_type=\"S3Prefix\",\n",
|
||||
" s3_uri=s3_input_path,\n",
|
||||
" s3_data_distribution_type=\"FullyReplicated\",\n",
|
||||
" )\n",
|
||||
" ),\n",
|
||||
" )\n",
|
||||
" ],\n",
|
||||
" output_data_config=OutputDataConfig(s3_output_path=s3_output_path),\n",
|
||||
" resource_config=ResourceConfig(\n",
|
||||
" instance_type=\"ml.m4.xlarge\", instance_count=1, volume_size_in_gb=30\n",
|
||||
" ),\n",
|
||||
" stopping_condition=StoppingCondition(max_runtime_in_seconds=600),\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Wait for TrainingJob with SageMakerCore\n",
|
||||
"In SageMakerCore, the logic required to wait on a resource is abstracted away using a `wait()` method. As a result, a user can directly call the `wait()` method on a TrainingJob object instance like below. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Wait for TrainingJob with SageMakerCore\n",
|
||||
"\n",
|
||||
"training_job.wait()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## List TrainingJobs with Boto3\n",
|
||||
"When a user lists TrainingJobs, there are 2 main approaches provided by Boto3. \n",
|
||||
"\n",
|
||||
"1. The first is calling `list_training_jobs` directly and implementing some logic to handle the NextToken provided in the response to enable pagination.\n",
|
||||
"2. The second is by utilizing the Boto3 `get_paginator` method to get a paginator that encapsulates the NextToken and simplifies the logic required.\n",
|
||||
"\n",
|
||||
"Both approaches are shown below. Although the boto3 provided paginator simplifies the logic over using a NextToken, in both cases the user must understand the structure of the list responses or refer to the docs (ie, understand to access TrainingJobSummaries by doing `response[\"TrainingJobSummaries\"]`)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# List TrainingJobs with Boto3\n",
|
||||
"import datetime\n",
|
||||
"import boto3\n",
|
||||
"\n",
|
||||
"client = boto3.client(\"sagemaker\")\n",
|
||||
"\n",
|
||||
"creation_time_after = datetime.datetime.now() - datetime.timedelta(days=1)\n",
|
||||
"\n",
|
||||
"# List TrainingJobs with NextToken\n",
|
||||
"next_token = None\n",
|
||||
"while True:\n",
|
||||
" if next_token:\n",
|
||||
" response = client.list_training_jobs(\n",
|
||||
" CreationTimeAfter=creation_time_after, NextToken=next_token\n",
|
||||
" )\n",
|
||||
" else:\n",
|
||||
" response = client.list_training_jobs(CreationTimeAfter=creation_time_after)\n",
|
||||
"\n",
|
||||
" for job in response[\"TrainingJobSummaries\"]:\n",
|
||||
" print(job[\"TrainingJobName\"], job[\"TrainingJobStatus\"])\n",
|
||||
"\n",
|
||||
" next_token = response.get(\"NextToken\")\n",
|
||||
"\n",
|
||||
" if not next_token:\n",
|
||||
" break"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import datetime\n",
|
||||
"import boto3\n",
|
||||
"\n",
|
||||
"client = boto3.client(\"sagemaker\")\n",
|
||||
"creation_time_after = datetime.datetime.now() - datetime.timedelta(days=1)\n",
|
||||
"\n",
|
||||
"# List TrainingJobs with Boto3 Paginator\n",
|
||||
"paginator = client.get_paginator(\"list_training_jobs\")\n",
|
||||
"for response in paginator.paginate(CreationTimeAfter=creation_time_after):\n",
|
||||
" for job in response[\"TrainingJobSummaries\"]:\n",
|
||||
" print(job[\"TrainingJobName\"], job[\"TrainingJobStatus\"])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## List TrainingJobs with SageMakerCore\n",
|
||||
"\n",
|
||||
"In SageMakerCore, listing is done similar to the boto3 paginator approach but instead with a `ResourceIterator` which implements the python iterator protocol to instantiate and return resource objects only as they are accessed.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Below, is an example of how the `get_all()` method would be used to list TrainingJobs."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# List TrainingJobs with SageMakerCore\n",
|
||||
"import datetime\n",
|
||||
"from sagemaker_core.resources import TrainingJob\n",
|
||||
"\n",
|
||||
"creation_time_after = datetime.datetime.now() - datetime.timedelta(days=1)\n",
|
||||
"\n",
|
||||
"resource_iterator = TrainingJob.get_all(creation_time_after=creation_time_after)\n",
|
||||
"for job in resource_iterator:\n",
|
||||
" print(job.training_job_name, job.training_job_status)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Delete All SageMaker Resources\n",
|
||||
"The following code block will call the delete() method for any SageMaker Core Resources created during the execution of this notebook which were assigned to local or global variables. If you created any additional deleteable resources without assigning the returning object to a unique variable, you will need to delete the resource manually by doing something like:\n",
|
||||
"\n",
|
||||
"```python\n",
|
||||
"resource = Resource.get(\"resource-name\")\n",
|
||||
"resource.delete()\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Delete any sagemaker core resource objects created in this notebook\n",
|
||||
"def delete_all_sagemaker_resources():\n",
|
||||
" all_objects = list(locals().values()) + list(globals().values())\n",
|
||||
" deletable_objects = [\n",
|
||||
" obj\n",
|
||||
" for obj in all_objects\n",
|
||||
" if hasattr(obj, \"delete\") and obj.__class__.__module__ == \"sagemaker_core.main.resources\"\n",
|
||||
" ]\n",
|
||||
"\n",
|
||||
" for obj in deletable_objects:\n",
|
||||
" obj.delete()\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"delete_all_sagemaker_resources()"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.14"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
# Amazon SageMaker Examples
|
||||
|
||||
### End To End ML Lifecycle
|
||||
|
||||
These examples are a diverse collection of end-to-end notebooks that demonstrate how to build, train, and deploy machine learning models using Amazon SageMaker. These notebooks cover a wide range of machine learning tasks and use cases, providing you with a comprehensive understanding of the SageMaker workflow.
|
||||
|
||||
- [Customer Churn Prediction with Amazon SageMaker Autopilot](sm-autopilot_customer_churn.ipynb)
|
||||
- [Housing Price Prediction with Amazon SageMaker Autopilot](sm-autopilot_linear_regression_california_housing.ipynb)
|
||||
- [Time-Series Forecasting with Amazon SageMaker Autopilot](sm-sm-autopilot_time_series_forecasting.ipynb)
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,28 @@
|
||||
# Amazon SageMaker Examples
|
||||
|
||||
### Prepare Data
|
||||
|
||||
The example notebooks within this folder showcase Sagemaker's data preparation capabilities. Data preparation in machine learning refers to the process of collecting, preprocessing, and organizing raw data to make it suitable for analysis and modeling.
|
||||
|
||||
- [Data Wrangler Data Prep Widget - Example Notebook](sm-data_wrangler_data_prep_widget/sm-data_wrangler_data_prep_widget.ipynb)
|
||||
- [Amazon SageMaker Feature Store: Feature Processor Introduction](sm-feature_store_feature_processor/sm-feature_store_feature_processor.ipynb)
|
||||
- [Amazon SageMaker Feature Store: Ground Truth Classification labelling job output to Feature Store](sm-feature_store_ground_truth_classification_output_to_store/sm-feature_store_ground_truth_classification_output_to_store.ipynb)
|
||||
- [Amazon SageMaker Feature Store: Introduction to Feature Store](sm-feature_store_introduction/sm-feature_store_introduction.ipynb)
|
||||
- [Create an Active Learning Workflow using Amazon SageMaker Ground Truth](sm-ground_truth_active_learning_workflow_bring_your_own_model/sm-ground_truth_active_learning_workflow_bring_your_own_model.ipynb)
|
||||
- [Understanding Annotation Consolidation: A SageMaker Ground Truth Demonstration for Image Classification](sm-ground_truth_annotation_consolidation_image_classification/sm-ground_truth_annotation_consolidation_image_classification.ipynb)
|
||||
- [From Unlabeled Data to a Deployed Machine Learning Model: A SageMaker Ground Truth Demonstration for Object Detection](sm-ground_truth_object_detection_example/sm-ground_truth_object_detection_example.ipynb)
|
||||
- [Audit and Improve Video Annotation Quality Using Amazon SageMaker Ground Truth](sm-ground_truth_video_quality_metrics/sm-ground_truth_video_quality_metrics.ipynb)
|
||||
- [Amazon Augmented AI(A2I) Integrated with AWS Marketplace ML Models](sm-marketplace_augmented_ai_with_marketplace_ml_models/sm-marketplace_augmented_ai_with_marketplace_ml_models.ipynb)
|
||||
- [Feature transformation with Amazon SageMaker Processing and Dask](sm-processing_feature_transformation_with_dask/sm-processing_feature_transformation_with_dask.ipynb)
|
||||
- [Distributed Data Processing using Apache Spark and SageMaker Processing](sm-processing_spark_processing/sm-processing_spark_processing.ipynb)
|
||||
- [SageMaker PySpark PCA and K-Means Clustering MNIST Example](sm-spark_pca_kmeans/sm-spark_pca_kmeans.ipynb)
|
||||
- [Create a 3D Point Cloud Labeling Job with Amazon SageMaker Ground Truth](sm-ground_truth_3d_pointcloud_labeling.ipynb)
|
||||
- [Chaining using Ground Truth Streaming Labeling Jobs](sm-ground_truth_chained_streaming_labeling_job.ipynb)
|
||||
- [Create a Ground Truth Streaming Labeling Job](sm-ground_truth_create_streaming_labeling_job.ipynb)
|
||||
- [Labeling Adjustment Job Adaptation](sm-ground_truth_labeling_adjustment_job_adaptation.ipynb)
|
||||
- [Training Object Detection Models in SageMaker with Augmented Manifests](sm-ground_truth_object_detection_augmented_manifest_training.ipynb)
|
||||
- [Using a Pre-Trained Model for Cost Effective Data Labeling](sm-ground_truth_pretrained_model_labeling.ipynb)
|
||||
- [Improving Your LLMs with RLHF on SageMaker](sm-ground_truth_rlhf_llm_finetuning.ipynb)
|
||||
- [Identify Worker Labeling Efficiency using SageMaker GroundTruth](sm-ground_truth_text_classification_labeling_accuracy_analysis.ipynb)
|
||||
- [Get started with SageMaker Processing](sm-processing_introduction.ipynb)
|
||||
- [SageMaker PySpark K-Means Clustering MNIST Example](sm-spark_kmeans.ipynb)
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 24 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 7.6 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 178 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 68 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 18 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 169 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 20 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 105 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 155 KiB |
+435
@@ -0,0 +1,435 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "82a8f73f-c2f7-4e3a-87b7-47ae6fa896bf",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"## Data Wrangler Data Prep Widget - Example Notebook"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "a092b5af",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"---"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "6c07e63e-ef85-49db-8870-f357da0fc6df",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The [Data Wrangler data prep widget](https://aws.amazon.com/blogs/machine-learning/interactive-data-prep-widget-for-notebooks-powered-by-amazon-sagemaker-data-wrangler/) automatically generates key visualizations on top of a Pandas data frame to understand data distribution, detect data quality issues, and surface data insights such as outliers for each feature. It helps interact with the data and discover insights that may go unnoticed with ad hoc querying. It also recommends transformations to remediate, enables you to apply data transformations on the UI and automatically generate code in the notebook cells."
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "cb9b75d9-4b6f-47c3-913d-7312696d63f4",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"By `import sagemaker_datawrangler` we are enabling the interactive data preparation assistant widget for Pandas dataframe in Amazon SageMaker Studio Notebooks "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "2ac38f47-8006-4e6a-b3b6-65739db9c600",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import pandas as pd\n",
|
||||
"import boto3\n",
|
||||
"import io\n",
|
||||
"import sagemaker_datawrangler"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "45797202-2108-4d26-b326-e607dc797ad0",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"For our use case, we use modified version of the [Titanic dataset](https://www.openml.org/search?type=data&sort=runs&id=40945&status=active), a popular dataset in the ML community so you can get started with SageMaker Data Wrangler quickly. The original [dataset](https://www.openml.org/search?type=data&sort=runs&id=40945&status=active) was obtained from [OpenML](https://www.openml.org/), and modified to add synthetic data quality issues by Amazon for this demo. You can download the modified version of dataset from public S3 path `s3://sagemaker-example-files-prod-{region}/datasets/tabular/dirty-titanic/titanic-dirty-4.csv`\n",
|
||||
"\n",
|
||||
"Read the dataset with `pandas`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "e91c4aad-2c75-45d7-9d0f-13ad0cc0d7f4",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"s3 = boto3.client(\"s3\")\n",
|
||||
"obj = s3.get_object(\n",
|
||||
" Bucket=f\"sagemaker-example-files-prod-{boto3.session.Session().region_name}\",\n",
|
||||
" Key=\"datasets/tabular/dirty-titanic/titanic-dirty-4.csv\",\n",
|
||||
")\n",
|
||||
"df = pd.read_csv(io.BytesIO(obj[\"Body\"].read()))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "2fc1843c-596a-4707-8bfc-06c4be7785c7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"After the data is loaded in the Pandas data frame, you can view the data by just using `df` or `display(df)`. Along with listing the row, the data prep widget produces insights, visualizations, and advice on data quality. You don’t need to write any additional code to generate feature and target insights, distribution information, or rendering data quality checks. You can choose the data frame table’s header to view the statistical summary showing the data quality warnings, if any."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "35d220fd-96d0-44dd-b59c-2f6332b41437",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "235aafb0-c94d-47ce-92fc-60f6ac678d59",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"<img src=\"images/widget_default_view.png\" />"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "696b0b27-67e6-4933-9264-b1da7978ba2f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Each column shows a bar chart or histogram based on the data type. By default, the widget samples up to 10,000 observations for generating meaningful insights. It also provides the option to run the insight analysis on the entire dataset.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "f11153a6-1f49-41c6-a765-896dfd7b451f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"For categorical data, the widget generates the bar chart with all the categories. In the following screenshot, for example, the column \"sex\" identifies the categories on the data. You can hover over the bar (male in this case) to see the details of these categories, like the total number of rows with the value male and its distribution in the total visualized dataset (64.07% in this example). It also highlights the total percentage of missing values in a different color for categorical data. For quantitative data like the \"ticket\" column, it shows distribution along with the percentage of invalid values."
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "ad8a6c10-f3b1-43bb-a8d7-c3ea2b77e4c7",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"\n",
|
||||
"<img src=\"images/Categoricaldata.png\" width=\"325\"/> <img src=\"images/Categoricaldata2.png\" width=\"305\"/> <img src=\"images/Categoricaldata3.png\" width=\"400\"/> \n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "cc9f64fc-2cb3-46c0-aa24-dbb536c87033",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The Insights tab provides details with descriptions for each column. This section lists aggregated statistics, such as mode, number of uniques, ratios and counts for missing/invalid values, etc., as well as visualize data distribution with help of a histogram or a bar chart. In the following screenshots, you can check out the data insights and distribution information displayed with easily understandable visualizations generated for the selected column \"survived\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "dcad537f-d27f-421a-823e-025be0c7d1ed",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"<img src=\"images/insights.png\" width=\"325\"/> <img src=\"images/distribution.png\" width=\"300\"/> <img src=\"images/uniquevalues.png\" width=\"400\"/> "
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "bce3c589-dc0c-4ec9-8a3b-5498308d7f9a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The studio data prep widget highlights identified data quality issues with the warning sign in the header. Widget can identify the whole spectrum of data quality issues from basics (missing values, constant column, etc.) to more ML specific (target leakage, low predictive score features, etc.). Widget highlights the cells causing the data quality issue and reorganize the rows to put the problematic cells at the top. To remedy the data quality issue widget provides several transformers, applicable on a click of a button.\n",
|
||||
"\n",
|
||||
"To explore the data quality section, choose the column header, and in the side panel, choose the Data quality tab. You should see the following in your Studio environment."
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "6e0af1a7-25ef-4a5a-812c-d53b56b85793",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"![alt text](images/survived-dataquality.png \"Data Issue Warnings\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "671616c7-82e2-4d11-bc51-ed03ae65a3aa",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let’s look at the different options available on the Data quality tab. For this example, we choose the age column, which is detected as a quantitative column based on the data. As we can see in the following screenshot, this widget suggests different type of transformations that you could apply, including the most common actions, such as Replace with new value, Drop missing, Replace with median, or Replace with mean. You can choose any of those for your dataset based on the use case (the ML problem you’re trying to solve). It also gives you the Drop column option if you want to remove the feature altogether."
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "850a0bfd-cda6-4189-9927-2c10a4b819dc",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"![alt text](images/dataqualityoptions.png \"Set as a Target Column\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "970454df-1082-4089-84aa-9ac224f0db78",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"When you choose Apply and export code, the transform is applied to the deep copy of the data frame. After the transform is applied successfully, the data table is refreshed with the insights and visualizations. The transform code is generated after the existing cell in the notebook. You can run this exported code later on to apply the transformation on your datasets, and extend it as per your needs. You can customize the transformation by directly modifying the generated code. If we apply the Drop missing option in the Age column, the following transformation code is applied to the dataset, and code is also generated in a cell below the widget:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "31d3c92e-ed95-4df5-86fa-166df8f6dd65",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Pandas code generated by sagemaker_datawrangler\n",
|
||||
"output_df = df.copy(deep=True)\n",
|
||||
"\n",
|
||||
"# Code to Drop missing for column: age to resolve warning: Missing values\n",
|
||||
"output_df = output_df[output_df[\"age\"].notnull()]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "80cfc0c7-c8b8-4240-bbea-7d839c539944",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The following is generated code for example of a code snippet for Replace with new value:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cca125c2-e06b-4852-bfb0-3bf7ffbdf4fb",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Pandas code generated by sagemaker_datawrangler\n",
|
||||
"output_df = df.copy(deep=True)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Code to Replace with new value for column: age to resolve warning: Missing values\n",
|
||||
"generic_value = 0\n",
|
||||
"output_df[\"age\"] = output_df[\"age\"].fillna(generic_value)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "949f0ea3-0686-4bed-b938-fbb1f4fe7503",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The following is another example of a code snippet for Replace with median:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "4871e9df-539f-4e22-8406-6c8b699fd377",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Pandas code generated by sagemaker_datawrangler\n",
|
||||
"output_df = df.copy(deep=True)\n",
|
||||
"\n",
|
||||
"# Code to Replace with median for column: age to resolve warning: Missing values\n",
|
||||
"output_df[\"age\"] = output_df[\"age\"].fillna(output_df[\"age\"].median(skipna=True))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "3479bd85-4933-459a-8444-cd5a6816d532",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"You can view the results of the applied transform directly on the table. The widget automatically generates Pandas or PySpark code in the Notebook on a new cell. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "bf5da385-8980-41bf-b3ed-af8a95c2ea1d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now let’s look at the data prep widget’s target insight capability. Assume you want to use the survived feature to predict if a passenger will survive. Choose the survived column header. In the side panel, choose Select as target column. The ideal data distribution for the survived feature should have only two classes: yes (1) or no (0), which helps classify the Titanic crash survival chances. However, due to data inconsistencies in the chosen target column, the survived feature has 0, 1, ?, unknown, and yes."
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "c8ee1657-4585-449d-b68e-09017412d7dd",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The data prep widget lists the target column insights with recommendations and sample explanations to solve the issues with the target column data quality. It also automatically highlights the anomalous data in the column."
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "28de8131-e03e-4be2-ae88-dd67b681ab05",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"![alt text](images/dataquality.png \"Apply and export code\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "91a88f0f-69c7-4394-8818-a68ba873790f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We choose the recommended transform Drop rare target values, because there are fewer observations for the rare target values."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "7ed8446a-a851-4b21-a24d-2da3e6d93051",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Pandas code generated by sagemaker_datawrangler\n",
|
||||
"output_df = df.copy(deep=True)\n",
|
||||
"\n",
|
||||
"# Code to Drop rare target values for column: survived to resolve warning: Too few instances per class\n",
|
||||
"rare_target_labels_to_drop = [\"?\", \"unknown\", \"yes\"]\n",
|
||||
"output_df = output_df[~output_df[\"survived\"].isin(rare_target_labels_to_drop)]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "41cb54e0-2099-40cb-85c4-6b0c3b6be441",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"If you want to see a standard [pandas](https://pandas.pydata.org/) visualization in the notebook, you can choose View the Pandas table and toggle between the widget and the Pandas representation, as shown in the following screenshot."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "b1398e47-c7b2-4c1a-a735-fbb2a4c16055",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "e4ec4e34-ae87-4ff5-8c54-2a5b87d494cc",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"<img src=\"images/widget_pandas_view.png\" /> "
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "12fe8025",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Notebook CI Test Results\n",
|
||||
"\n",
|
||||
"This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"instance_type": "ml.t3.medium",
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (Data Science 3.0)",
|
||||
"language": "python",
|
||||
"name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
Id,Model,Year,Status,Mileage,Price,MSRP
|
||||
96864,6110,2019 Tesla Model 3 Standard Range,2019,Used,"50,412 mi.","$31,924.00",Not specified
|
||||
51128,7439,2016 Hyundai Sonata Sport,2016,Used,"97,237 mi.","$13,492.00",Not specified
|
||||
85327,4565,2021 Porsche Cayenne GTS,2021,Used,"23,375 mi.","$105,495.00",Not specified
|
||||
110510,4758,2020 Volkswagen Passat 2.0T SE,2020,Used,"46,249 mi.","$22,899.00",Not specified
|
||||
74498,3836,2023 Mercedes-Benz EQS 450+ EQS 450 4MATIC,2023,New,Not available,"$112,745.00",Not specified
|
||||
103711,3787,2023 Toyota Venza LE,2023,New,Not available,"$39,224.00",Not specified
|
||||
29375,97,2022 Dodge Challenger R/T Scat Pack,2022,New,Not available,"$62,985.00",Not specified
|
||||
41662,2511,2022 Ford F-250 XLT,2022,New,Not available,"$63,500.00","MSRP $64,500"
|
||||
60265,8276,2023 INFINITI QX50 Luxe,2023,New,Not available,"$49,100.00",Not specified
|
||||
2206,2206,2020 Acura RDX Technology Package,2020,Used,"24,712 mi.","$35,955.00",Not specified
|
||||
100010,86,2023 Toyota GR86 Premium,2023,New,Not available,"$35,683.00",$333 price drop
|
||||
54283,2294,2019 INFINITI QX50 ESSENTIAL,2019,Used,"34,986 mi.","$29,999.00",Not specified
|
||||
30829,1551,2022 Dodge Durango Citadel,2022,New,Not available,"$59,106.00",Not specified
|
||||
50828,7139,2023 Hyundai Santa Cruz 2.5T SEL Premium,2023,New,Not available,"$39,385.00","MSRP $39,385"
|
||||
85246,4484,2019 Porsche 911 GT3 RS,2019,Used,"4,241 mi.","$254,888.00",Not specified
|
||||
114257,8505,2018 Volkswagen Beetle 2.0T S,2018,Used,"46,926 mi.","$25,594.00",Not specified
|
||||
34452,5174,2008 Dodge Charger SXT,2008,Used,"133,467 mi.","$6,495.00",Not specified
|
||||
68825,8148,2011 Lexus IS-F Base,2011,Used,"36,800 mi.","$54,999.00",Not specified
|
||||
99988,64,2023 Toyota Tundra Platinum,2023,New,Not available,"$66,483.00",Not specified
|
||||
99824,9070,2023 Tesla Model 3 Base,2023,Used,"2,056 mi.","$45,888.00",Not specified
|
||||
103430,3506,2023 Toyota Camry XSE,2023,New,Not available,"$38,246.00",Not specified
|
||||
42818,3667,2017 Ford Focus ST Base,2017,Used,"64,421 mi.","$19,900.00",Not specified
|
||||
825,825,2020 Acura RDX Technology Package,2020,Used,"57,295 mi.","$31,881.00",Not specified
|
||||
98270,7516,2020 Tesla Model 3 Standard Range Plus,2020,Used,"8,817 mi.","$44,900.00",Not specified
|
||||
56852,4863,2019 INFINITI Q50 3.0t Signature Edition,2019,Used,"39,621 mi.","$27,500.00",Not specified
|
||||
108336,2584,2022 Volkswagen Atlas 3.6L SE w/Technology,2022,New,Not available,"$46,550.00",Not specified
|
||||
63813,3136,2021 Lexus RX 350L Premium,2021,Used,"16,560 mi.","$47,987.00","$2,000 price drop"
|
||||
100655,731,2023 Toyota Tundra Hybrid Platinum,2023,New,Not available,"$69,346.00",Not specified
|
||||
84924,4162,2014 Porsche 911 Carrera,2014,Used,"43,484 mi.","$67,950.00",$550 price drop
|
||||
113714,7962,2018 Volkswagen Atlas 3.6L SE,2018,Used,"62,614 mi.","$24,191.00",Not specified
|
||||
63736,3059,2018 Lexus RX 350L Premium,2018,Used,"34,918 mi.","$36,995.00",Not specified
|
||||
86932,6170,2012 Porsche Panamera 4,2012,Used,"42,986 mi.","$33,995.00",Not specified
|
||||
46008,2319,2023 Hyundai Palisade XRT,2023,New,Not available,"$42,470.00",Not specified
|
||||
96253,5499,2022 Tesla Model 3 Performance,2022,Used,"16,244 mi.","$49,999.00",Not specified
|
||||
6305,6305,2006 Acura TSX Base (A5),2006,Used,"84,825 mi.","$11,450.00",Not specified
|
||||
21555,2276,2023 Chevrolet Tahoe RST,2023,New,Not available,"$67,794.00","MSRP $67,295"
|
||||
61420,743,2020 Lexus ES 300h Luxury,2020,Used,"27,649 mi.","$44,953.00",Not specified
|
||||
32580,3302,2014 Dodge Challenger R/T,2014,Used,"164,569 mi.","$12,995.00",Not specified
|
||||
112597,6845,2023 Volkswagen Atlas Cross Sport 3.6L V6 SE w/Technology,2023,New,Not available,"$40,423.00","$3,500 price drop"
|
||||
44739,1050,2023 Hyundai Tucson Limited,2023,New,Not available,"$39,510.00","MSRP $39,510"
|
||||
83743,2981,2018 Porsche Cayenne Platinum Edition,2018,Used,"37,491 mi.","$39,495.00",Not specified
|
||||
59550,7561,2021 INFINITI Q50 3.0t LUXE,2021,Used,"49,667 mi.","$24,995.00",Not specified
|
||||
39410,259,2022 Ford Explorer Timberline,2022,New,Not available,"$53,480.00",Not specified
|
||||
75383,4721,2022 Mercedes-Benz GLC 300 Base 4MATIC,2022,New,Not available,"$50,260.00","MSRP $50,260"
|
||||
35442,6164,2017 Dodge Challenger SXT,2017,Used,"56,467 mi.","$20,503.00",Not specified
|
||||
39144,9866,2015 Dodge Durango Limited,2015,Used,"99,683 mi.","$18,399.00","$1,600 price drop"
|
||||
103281,3357,2023 Toyota Tacoma SR5,2023,New,Not available,"$33,950.00","MSRP $33,950"
|
||||
7780,7780,2023 Acura MDX Advance,2023,New,Not available,"$65,890.00",Not specified
|
||||
34843,5565,2019 Dodge Challenger R/T Scat Pack,2019,Used,"38,565 mi.","$44,132.00","$2,368 price drop"
|
||||
44389,700,2021 Hyundai Palisade SEL,2021,Used,"11,036 mi.","$41,990.00",Not specified
|
||||
68336,7659,2022 Lexus LS 500,2022,New,Not available,"$106,445.00","MSRP $106,445"
|
||||
104750,4826,2019 Toyota Highlander LE Plus,2019,Used,"42,300 mi.","$28,500.00","$2,500 price drop"
|
||||
3991,3991,2019 Acura RDX Base,2019,Used,"45,362 mi.","$30,998.00","$1,000 price drop"
|
||||
82796,2034,2013 Porsche Cayenne Turbo,2013,Used,"73,086 mi.","$33,395.00",$604 price drop
|
||||
39509,358,2022 Ford F-350 XL,2022,New,Not available,"$66,790.00",Not specified
|
||||
57364,5375,2023 INFINITI QX60 SENSORY,2023,New,Not available,"$65,718.00","MSRP $63,825"
|
||||
70829,167,2022 Mercedes-Benz GLE 450 AWD 4MATIC,2022,New,Not available,"$74,260.00","MSRP $74,260"
|
||||
49749,6060,2016 Hyundai Genesis 3.8,2016,Used,"74,714 mi.","$23,000.00",Not specified
|
||||
81569,807,2022 Porsche 718 Boxster GTS,2022,Porsche Certified,586 mi.,"$113,991.00",Not specified
|
||||
3281,3281,2021 Acura TLX Technology,2021,Used,"9,604 mi.","$35,594.00",$619 price drop
|
||||
92193,1439,2016 Tesla Model S 75,2016,Used,"55,343 mi.","$34,995.00","$4,000 price drop"
|
||||
42775,3624,2022 Ford Escape SE,2022,New,Not available,"$34,690.00",Not specified
|
||||
87560,6798,2021 Porsche 718 Cayman GTS 4.0,2021,Used,"3,331 mi.","$115,000.00",Not specified
|
||||
69040,8363,2011 Lexus ES 350 Base,2011,Used,"68,332 mi.","$16,940.00",Not specified
|
||||
41276,2125,2022 Ford F-350 XL,2022,New,Not available,"$69,630.00",Not specified
|
||||
95380,4626,2020 Tesla Model X Long Range,2020,Used,"32,000 mi.","$79,995.00",Not specified
|
||||
99894,9140,2018 Tesla Model 3 Long Range,2018,Used,"14,344 mi.","$40,500.00",Not specified
|
||||
51927,8238,2021 Hyundai Palisade SE,2021,Used,"58,452 mi.","$31,990.00",Not specified
|
||||
9240,9240,2023 Acura Integra A-Spec,2023,New,Not available,"$35,095.00","MSRP $35,095"
|
||||
44883,1194,2021 Hyundai Palisade SEL,2021,Used,"32,524 mi.","$36,795.00",Not specified
|
||||
89424,8662,2021 Porsche Cayenne S,2021,Used,"18,463 mi.","$89,989.00",Not specified
|
||||
83920,3158,2021 Porsche Cayenne AWD,2021,Used,"17,242 mi.","$79,900.00",Not specified
|
||||
61322,645,2023 Lexus LX 600 Premium,2023,New,Not available,"$103,150.00","MSRP $103,150"
|
||||
110236,4484,2019 Volkswagen Beetle 2.0T S,2019,Used,"20,406 mi.","$21,921.00",Not specified
|
||||
88109,7347,2018 Porsche 718 Boxster,2018,Used,"16,141 mi.","$64,990.00",Not specified
|
||||
66076,5399,2014 Lexus LS 460 Base,2014,Used,"140,725 mi.","$21,975.00",Not specified
|
||||
65897,5220,2019 Lexus UX 200 F-SPORT,2019,Used,"34,362 mi.","$32,998.00",Not specified
|
||||
55324,3335,2017 INFINITI Q50 3.0T Premium,2017,Used,"72,174 mi.","$20,997.00",Not specified
|
||||
31470,2192,2022 Dodge Durango R/T,2022,New,Not available,"$64,930.00","MSRP $66,275"
|
||||
71177,515,2023 Mercedes-Benz GLE 450 AWD 4MATIC,2023,New,Not available,"$82,915.00","MSRP $82,915"
|
||||
50393,6704,2017 Hyundai Accent SE,2017,Used,"75,704 mi.","$12,388.00",Not specified
|
||||
45609,1920,2022 Hyundai Sonata SEL,2022,Used,821 mi.,"$27,011.00",Not specified
|
||||
32524,3246,2022 Dodge Challenger SRT Hellcat,2022,New,Not available,"$85,230.00","MSRP $85,980"
|
||||
113644,7892,2023 Volkswagen Atlas Cross Sport 2.0T SEL,2023,New,Not available,"$48,938.00","MSRP $48,938"
|
||||
28415,9136,2014 Chevrolet Silverado 1500 LT,2014,Used,"110,316 mi.","$21,660.00",Not specified
|
||||
12150,2754,2023 BMW 540 i,2023,New,Not available,"$68,520.00","MSRP $68,520"
|
||||
36372,7094,2013 Dodge Charger SE,2013,Used,"125,336 mi.","$9,995.00","$1,000 price drop"
|
||||
11948,2552,2018 BMW 320 i xDrive,2018,Used,"53,250 mi.","$23,998.00",Not specified
|
||||
76352,5690,2023 Mercedes-Benz Maybach S 580 4MATIC,2023,New,Not available,"$228,850.00","MSRP $228,850"
|
||||
91021,267,2021 Tesla Model Y Long Range,2021,Used,"19,859 mi.","$49,000.00","$5,000 price drop"
|
||||
58210,6221,2008 INFINITI G35 Journey,2008,Used,"160,591 mi.","$6,900.00",Not specified
|
||||
101534,1610,2022 Toyota Tundra Hybrid Platinum,2022,New,Not available,"$72,622.00",Not specified
|
||||
54403,2414,2023 INFINITI QX80 PREMIUM SELECT,2023,New,Not available,"$82,010.00","MSRP $84,510"
|
||||
90442,9680,2022 Porsche Cayenne,2022,Porsche Certified,"4,137 mi.","$203,995.00",Not specified
|
||||
72757,2095,2022 Mercedes-Benz S-Class S 580 4MATIC,2022,New,Not available,"$152,685.00","MSRP $152,685"
|
||||
83365,2603,2003 Porsche 911 Turbo,2003,Used,"28,658 mi.","$115,900.00",Not specified
|
||||
90060,9298,2019 Porsche Cayenne Base,2019,Used,"32,364 mi.","$54,494.00",Not specified
|
||||
53711,1722,2013 INFINITI G37 Journey,2013,Used,"129,963 mi.","$11,995.00",Not specified
|
||||
60355,8366,2023 INFINITI QX80 SENSORY,2023,New,Not available,"$87,955.00","MSRP $87,955"
|
||||
75909,5247,2023 Mercedes-Benz GLS 450 4MATIC,2023,New,Not available,"$93,645.00","MSRP $93,645"
|
||||
|
+5
@@ -0,0 +1,5 @@
|
||||
customer_id,age,SSN,credit_score
|
||||
573291,26,266666666,500
|
||||
109382,42,422222222,600
|
||||
828400,35,355555555,700
|
||||
124013,61,611111111,800
|
||||
|
+5
@@ -0,0 +1,5 @@
|
||||
customer_id,city_code,state_code,country_code
|
||||
573291,1,49,2
|
||||
109382,2,40,2
|
||||
828400,3,31,2
|
||||
124013,4,5,2
|
||||
|
+5
@@ -0,0 +1,5 @@
|
||||
customer_id,city_code,state_code,country_code,email,name
|
||||
573291,1,49,2,john.lee@gmail.com,John Lee
|
||||
109382,2,40,2,olivequil@gmail.com,Olive Quil
|
||||
828400,3,31,2,liz.knee@gmail.com,Liz Knee
|
||||
124013,4,5,2,eileenbook@gmail.com,Eileen Book
|
||||
|
+5
@@ -0,0 +1,5 @@
|
||||
customer_id,order_id,order_status,store_id
|
||||
573291,4132,1,303
|
||||
109382,5724,0,201
|
||||
828400,1942,0,431
|
||||
124013,6782,1,213
|
||||
|
Binary file not shown.
@@ -0,0 +1,2 @@
|
||||
{"source-ref":"s3://AWSDOC-EXAMPLE-BUCKET/example_image.jpg","species":"0","species-metadata":{"class-name": "dog","confidence": 0.00,"type": "groundtruth/image-classification","job-name": "identify-animal-species","human-annotated": "yes","creation-date": "2018-10-18T22:18:13.527256"}}
|
||||
{"source":"The food was delicious","mood":"1","mood-metadata":{"class-name": "positive","confidence": 0.8,"type": "groundtruth/text-classification","job-name": "label-sentiment","human-annotated": "yes","creation-date": "2020-10-18T22:18:13.527256"}}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 76 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 49 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 54 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 26 KiB |
+781
@@ -0,0 +1,781 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "e2ac1559-3729-4cf3-acee-d4bb15c6f53d",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"# Amazon SageMaker Feature Store: Feature Processor Introduction"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "bfd7d612",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"---"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "c339cb18",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This notebook demonstrates how to get started with Feature Processor using SageMaker python SDK, create feature groups, perform batch transformation and ingest processed input data to feature groups.\n",
|
||||
"\n",
|
||||
"We first demonstrate how to use `@feature-processor` decorator to run the job locally and then show how to use `@remote` decorator to execute large batch transform and ingestion on SageMaker training job remotely. Besides, the SDK provides APIs to create scheduled pipelines based on transformation code.\n",
|
||||
"\n",
|
||||
"If you would like to learn more about Feature Processor, see documentation [Feature Processing](https://docs.aws.amazon.com/sagemaker/latest/dg/feature-store-feature-processing.html) for more info and examples."
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "a8b4ba90-e512-46bf-bfa9-541213021e86",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"## Setup For Notebook\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "e45c4dd7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Setup Runtime Environment\n",
|
||||
"\n",
|
||||
"First we create a new kernel to execute this notebook.\n",
|
||||
"1. Launch a new terminal in the current image (the '$_' icon at the top of this notebook).\n",
|
||||
"2. Execute the commands: \n",
|
||||
"```\n",
|
||||
"conda create --name feature-processing-py-3.9 python=3.9 -y\n",
|
||||
"conda activate feature-processing-py-3.9\n",
|
||||
"conda install ipykernel -y\n",
|
||||
"conda install openjdk -y\n",
|
||||
"```\n",
|
||||
"3. Return to this notebook and select the kernel with Image: 'Data Science' and Kernel: 'feature-processing-py-3.9'"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "a65db47d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Alternatively If you run this notebook on SageMaker Studio, you can execute the following cell to install runtime dependencies."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "efbd6006",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%capture\n",
|
||||
"\n",
|
||||
"!apt-get update\n",
|
||||
"!apt-get install openjdk-11-jdk -y\n",
|
||||
"%pip install ipykernel"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "f50b2d1f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"To get the Feature Processor module, we need to reinstall the SageMaker python SDK along with extra dependencies."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "7351b428",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%pip install 'sagemaker[feature-processor]<3.0' --force-reinstall"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "3b32941c-2df3-4014-a65f-133d19e43d46",
|
||||
"metadata": {
|
||||
"scrolled": true,
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"\"\"\"\n",
|
||||
"Restart the kernel.\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"from IPython.display import display_html\n",
|
||||
"\n",
|
||||
"display_html(\"<script>Jupyter.notebook.kernel.restart()</script>\", raw=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "ecd0598d-2b49-415c-9d23-c5caa9303323",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"\"\"\"\n",
|
||||
"Function to save the cell code in a file and execute the cell as well. This will be used later to create Lineage artifact for the code. \n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"from IPython.core.magic import register_cell_magic\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"@register_cell_magic\n",
|
||||
"def write_and_execute(line, cell):\n",
|
||||
" argz = line.split()\n",
|
||||
" file = argz[-1]\n",
|
||||
" mode = \"w\"\n",
|
||||
" if len(argz) == 2 and argz[0] == \"-a\":\n",
|
||||
" mode = \"a\"\n",
|
||||
" with open(file, mode) as f:\n",
|
||||
" f.write(cell)\n",
|
||||
" get_ipython().run_cell(cell)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "a303d7bc",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Create Feature Groups"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "f57390a2",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"First we start by creating two feature groups. One feature group is used for storing raw car sales dataset which is located in `data/car_data.csv`. We create another feature group to store aggregated feature values after feature processing, for example average value of `mileage`, `price` and `msrp`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cb7c6571-3f2f-49b5-a49a-3654ab076241",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"\"\"\"\n",
|
||||
"Set up feature groups.\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"import boto3, time\n",
|
||||
"import sagemaker\n",
|
||||
"from sagemaker.s3 import S3Uploader\n",
|
||||
"from sagemaker import get_execution_role\n",
|
||||
"import logging\n",
|
||||
"\n",
|
||||
"logger = logging.getLogger()\n",
|
||||
"logger.setLevel(logging.INFO)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"sagemaker_session = sagemaker.Session()\n",
|
||||
"sagemaker_client = boto3.client(\"sagemaker\")\n",
|
||||
"sagemaker_featurestore_runtime_client = boto3.client(\"sagemaker-featurestore-runtime\")\n",
|
||||
"\n",
|
||||
"aws_account_id = sagemaker_session.account_id()\n",
|
||||
"region = sagemaker_session.boto_region_name\n",
|
||||
"\n",
|
||||
"s3_bucket = sagemaker_session.default_bucket()\n",
|
||||
"default_bucket_prefix = sagemaker_session.default_bucket_prefix\n",
|
||||
"s3_prefix = \"feature-store/feature-processing\"\n",
|
||||
"\n",
|
||||
"# If a default bucket prefix is specified, append it to the s3 path\n",
|
||||
"if default_bucket_prefix:\n",
|
||||
" s3_prefix = f\"{default_bucket_prefix}/{s3_prefix}\"\n",
|
||||
"\n",
|
||||
"s3_data_prefix = f\"{s3_prefix}/data-sets\"\n",
|
||||
"s3_offline_store_prefix = f\"{s3_prefix}/offline-store\"\n",
|
||||
"offline_store_role = get_execution_role()\n",
|
||||
"\n",
|
||||
"\"\"\"\n",
|
||||
"Feature Group Definitions.\n",
|
||||
"\"\"\"\n",
|
||||
"from sagemaker.feature_store.feature_definition import FeatureDefinition, FeatureTypeEnum\n",
|
||||
"\n",
|
||||
"# S3 Data Source - Car Sales, and uploads to S3\n",
|
||||
"CAR_SALES_DATA_DIR = \"./data/car_data.csv\"\n",
|
||||
"RAW_CAR_SALES_S3_URI = S3Uploader.upload(CAR_SALES_DATA_DIR, f\"s3://{s3_bucket}/{s3_data_prefix}\")\n",
|
||||
"\n",
|
||||
"# Feature Group - Car Sales\n",
|
||||
"CAR_SALES_FG_NAME = \"car-data\"\n",
|
||||
"CAR_SALES_FG_ARN = f\"arn:aws:sagemaker:{region}:{aws_account_id}:feature-group/{CAR_SALES_FG_NAME}\"\n",
|
||||
"CAR_SALES_FG_ROLE_ARN = offline_store_role\n",
|
||||
"CAR_SALES_FG_OFFLINE_STORE_S3_URI = f\"s3://{s3_bucket}/{s3_offline_store_prefix}\"\n",
|
||||
"CAR_SALES_FG_FEATURE_DEFINITIONS = [\n",
|
||||
" FeatureDefinition(feature_name=\"id\", feature_type=FeatureTypeEnum.STRING),\n",
|
||||
" FeatureDefinition(feature_name=\"model\", feature_type=FeatureTypeEnum.STRING),\n",
|
||||
" FeatureDefinition(feature_name=\"model_year\", feature_type=FeatureTypeEnum.STRING),\n",
|
||||
" FeatureDefinition(feature_name=\"status\", feature_type=FeatureTypeEnum.STRING),\n",
|
||||
" FeatureDefinition(feature_name=\"mileage\", feature_type=FeatureTypeEnum.STRING),\n",
|
||||
" FeatureDefinition(feature_name=\"price\", feature_type=FeatureTypeEnum.STRING),\n",
|
||||
" FeatureDefinition(feature_name=\"msrp\", feature_type=FeatureTypeEnum.STRING),\n",
|
||||
" FeatureDefinition(feature_name=\"ingest_time\", feature_type=FeatureTypeEnum.FRACTIONAL),\n",
|
||||
"]\n",
|
||||
"CAR_SALES_FG_RECORD_IDENTIFIER_NAME = \"id\"\n",
|
||||
"CAR_SALES_FG_EVENT_TIME_FEATURE_NAME = \"ingest_time\"\n",
|
||||
"\n",
|
||||
"# Feature Group - Aggregated Car Sales\n",
|
||||
"AGG_CAR_SALES_FG_NAME = \"car-data-aggregated\"\n",
|
||||
"AGG_CAR_SALES_FG_ARN = (\n",
|
||||
" f\"arn:aws:sagemaker:{region}:{aws_account_id}:feature-group/{AGG_CAR_SALES_FG_NAME}\"\n",
|
||||
")\n",
|
||||
"AGG_CAR_SALES_FG_ROLE_ARN = offline_store_role\n",
|
||||
"AGG_CAR_SALES_FG_OFFLINE_STORE_S3_URI = f\"s3://{s3_bucket}/{s3_offline_store_prefix}\"\n",
|
||||
"AGG_CAR_SALES_FG_FEATURE_DEFINITIONS = [\n",
|
||||
" FeatureDefinition(feature_name=\"model_year_status\", feature_type=FeatureTypeEnum.STRING),\n",
|
||||
" FeatureDefinition(feature_name=\"avg_mileage\", feature_type=FeatureTypeEnum.STRING),\n",
|
||||
" FeatureDefinition(feature_name=\"max_mileage\", feature_type=FeatureTypeEnum.STRING),\n",
|
||||
" FeatureDefinition(feature_name=\"avg_price\", feature_type=FeatureTypeEnum.STRING),\n",
|
||||
" FeatureDefinition(feature_name=\"max_price\", feature_type=FeatureTypeEnum.STRING),\n",
|
||||
" FeatureDefinition(feature_name=\"avg_msrp\", feature_type=FeatureTypeEnum.STRING),\n",
|
||||
" FeatureDefinition(feature_name=\"max_msrp\", feature_type=FeatureTypeEnum.STRING),\n",
|
||||
" FeatureDefinition(feature_name=\"ingest_time\", feature_type=FeatureTypeEnum.FRACTIONAL),\n",
|
||||
"]\n",
|
||||
"AGG_CAR_SALES_FG_RECORD_IDENTIFIER_NAME = \"model_year_status\"\n",
|
||||
"AGG_CAR_SALES_FG_EVENT_TIME_FEATURE_NAME = \"ingest_time\"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\"\"\"\n",
|
||||
"Create the Feature Groups.\n",
|
||||
"\"\"\"\n",
|
||||
"from sagemaker.feature_store.feature_group import FeatureGroup\n",
|
||||
"\n",
|
||||
"# Create Feature Group - Car sale records.\n",
|
||||
"car_sales_fg = FeatureGroup(\n",
|
||||
" name=CAR_SALES_FG_NAME,\n",
|
||||
" feature_definitions=CAR_SALES_FG_FEATURE_DEFINITIONS,\n",
|
||||
" sagemaker_session=sagemaker_session,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" create_car_sales_fg_resp = car_sales_fg.create(\n",
|
||||
" record_identifier_name=CAR_SALES_FG_RECORD_IDENTIFIER_NAME,\n",
|
||||
" event_time_feature_name=CAR_SALES_FG_EVENT_TIME_FEATURE_NAME,\n",
|
||||
" s3_uri=CAR_SALES_FG_OFFLINE_STORE_S3_URI,\n",
|
||||
" enable_online_store=True,\n",
|
||||
" role_arn=CAR_SALES_FG_ROLE_ARN,\n",
|
||||
" )\n",
|
||||
" print(f\"Created feature group {create_car_sales_fg_resp}\")\n",
|
||||
"except Exception as e:\n",
|
||||
" if \"ResourceInUse\" in str(e):\n",
|
||||
" print(\"Feature Group already exists\")\n",
|
||||
" else:\n",
|
||||
" raise e\n",
|
||||
"\n",
|
||||
"# Create Feature Group - Aggregated car sales records.\n",
|
||||
"agg_car_sales_fg = FeatureGroup(\n",
|
||||
" name=AGG_CAR_SALES_FG_NAME,\n",
|
||||
" feature_definitions=AGG_CAR_SALES_FG_FEATURE_DEFINITIONS,\n",
|
||||
" sagemaker_session=sagemaker_session,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" create_agg_car_sales_fg_resp = agg_car_sales_fg.create(\n",
|
||||
" record_identifier_name=AGG_CAR_SALES_FG_RECORD_IDENTIFIER_NAME,\n",
|
||||
" event_time_feature_name=AGG_CAR_SALES_FG_EVENT_TIME_FEATURE_NAME,\n",
|
||||
" s3_uri=AGG_CAR_SALES_FG_OFFLINE_STORE_S3_URI,\n",
|
||||
" enable_online_store=True,\n",
|
||||
" role_arn=AGG_CAR_SALES_FG_ROLE_ARN,\n",
|
||||
" )\n",
|
||||
" print(f\"Created feature group {create_agg_car_sales_fg_resp}\")\n",
|
||||
" print(\"Sleeping for a bit, to let Feature Groups get ready.\")\n",
|
||||
" time.sleep(15)\n",
|
||||
"except Exception as e:\n",
|
||||
" if \"ResourceInUse\" in str(e):\n",
|
||||
" print(\"Feature Group already exists\")\n",
|
||||
" else:\n",
|
||||
" raise e"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "75d9c534-7b9d-40da-a99b-54aa8f927f8e",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"## `@feature_processor`\n",
|
||||
"\n",
|
||||
"The following example demonstrates how to use the @feature_processor decorator to load data from Amazon S3 to a SageMaker Feature Group. \n",
|
||||
"\n",
|
||||
"A `@feature_processor` decorated function automatically loads data from the configured inputs, applies the feature processing code and ingests the transformed data to a feature group."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "029580ed-bb28-4ae7-a937-ecfe123cee8e",
|
||||
"metadata": {
|
||||
"scrolled": true,
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%write_and_execute car-data-ingestion.py\n",
|
||||
"\n",
|
||||
"from sagemaker.feature_store.feature_processor import (\n",
|
||||
" feature_processor,\n",
|
||||
" FeatureGroupDataSource,\n",
|
||||
" CSVDataSource,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"@feature_processor(\n",
|
||||
" inputs=[CSVDataSource(RAW_CAR_SALES_S3_URI)],\n",
|
||||
" output=CAR_SALES_FG_ARN,\n",
|
||||
" target_stores=[\"OfflineStore\"],\n",
|
||||
")\n",
|
||||
"def transform(raw_s3_data_as_df):\n",
|
||||
" \"\"\"Load data from S3, perform basic feature engineering, store it in a Feature Group\"\"\"\n",
|
||||
" from pyspark.sql.functions import regexp_replace\n",
|
||||
" from pyspark.sql.functions import lit\n",
|
||||
" import time\n",
|
||||
"\n",
|
||||
" transformed_df = (\n",
|
||||
" raw_s3_data_as_df.withColumn(\"Price\", regexp_replace(\"Price\", \"\\$\", \"\"))\n",
|
||||
" # Rename Columns\n",
|
||||
" .withColumnRenamed(\"Id\", \"id\")\n",
|
||||
" .withColumnRenamed(\"Model\", \"model\")\n",
|
||||
" .withColumnRenamed(\"Year\", \"model_year\")\n",
|
||||
" .withColumnRenamed(\"Status\", \"status\")\n",
|
||||
" .withColumnRenamed(\"Mileage\", \"mileage\")\n",
|
||||
" .withColumnRenamed(\"Price\", \"price\")\n",
|
||||
" .withColumnRenamed(\"MSRP\", \"msrp\")\n",
|
||||
" # Add Event Time\n",
|
||||
" .withColumn(\"ingest_time\", lit(int(time.time())))\n",
|
||||
" # Remove punctuation and fluff; replace with NA\n",
|
||||
" .withColumn(\"mileage\", regexp_replace(\"mileage\", \"(,)|(mi\\.)\", \"\"))\n",
|
||||
" .withColumn(\"mileage\", regexp_replace(\"mileage\", \"Not available\", \"NA\"))\n",
|
||||
" .withColumn(\"price\", regexp_replace(\"price\", \",\", \"\"))\n",
|
||||
" .withColumn(\"msrp\", regexp_replace(\"msrp\", \"(^MSRP\\s\\\\$)|(,)\", \"\"))\n",
|
||||
" .withColumn(\"msrp\", regexp_replace(\"msrp\", \"Not specified\", \"NA\"))\n",
|
||||
" .withColumn(\"msrp\", regexp_replace(\"msrp\", \"\\\\$\\d+[a-zA-Z\\s]+\", \"NA\"))\n",
|
||||
" .withColumn(\"model\", regexp_replace(\"model\", \"^\\d\\d\\d\\d\\s\", \"\"))\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" transformed_df.show()\n",
|
||||
"\n",
|
||||
" return transformed_df\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Execute the FeatureProcessor and show the results.\n",
|
||||
"transform()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "23ef02b7-7b38-4c00-99fb-4caed9773321",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## `@feature_processor + @remote`\n",
|
||||
"\n",
|
||||
"The following example demonstrates how to run your feature processing code remotely.\n",
|
||||
"\n",
|
||||
"This is useful if you are working with large data sets that require hardware more powerful than locally available. You can decorate your code with the `@remote` decorator to run your local Python code as a single or multi-node distributed SageMaker training job. For more information on running your code as a SageMaker training job, see [Run your local code as a SageMaker training job](https://docs.aws.amazon.com/sagemaker/latest/dg/train-remote-decorator.html)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "d1f50d11",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"\"\"\"\n",
|
||||
"Create a requirements.txt and specify sagemaker as a remote job dependency.\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"sagemaker_version = sagemaker.__version__\n",
|
||||
"with open(\"requirements.txt\", \"w\") as file:\n",
|
||||
" file.write(f\"sagemaker=={sagemaker_version}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "907584ef-88fe-4a9f-9c99-299362845935",
|
||||
"metadata": {
|
||||
"scrolled": true,
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from pyspark.sql import DataFrame, SparkSession\n",
|
||||
"from sagemaker.remote_function import remote\n",
|
||||
"from sagemaker.feature_store.feature_processor import (\n",
|
||||
" feature_processor,\n",
|
||||
" CSVDataSource,\n",
|
||||
" FeatureGroupDataSource,\n",
|
||||
")\n",
|
||||
"from sagemaker.remote_function.spark_config import SparkConfig\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"@remote(\n",
|
||||
" spark_config=SparkConfig(),\n",
|
||||
" instance_type=\"ml.m5.2xlarge\",\n",
|
||||
" dependencies=\"./requirements.txt\",\n",
|
||||
" # keep_alive_period_in_seconds=900 # Requires an account limit increase to enable warm pooling.\n",
|
||||
")\n",
|
||||
"@feature_processor(\n",
|
||||
" inputs=[FeatureGroupDataSource(CAR_SALES_FG_ARN)],\n",
|
||||
" output=AGG_CAR_SALES_FG_ARN,\n",
|
||||
" target_stores=[\"OfflineStore\"],\n",
|
||||
")\n",
|
||||
"def aggregate(source_feature_group, spark):\n",
|
||||
" \"\"\"\n",
|
||||
" Aggregate the data using a SQL query and UDF.\n",
|
||||
" \"\"\"\n",
|
||||
" import time\n",
|
||||
" from pyspark.sql.types import StringType\n",
|
||||
" from pyspark.sql.functions import udf\n",
|
||||
"\n",
|
||||
" @udf(returnType=StringType())\n",
|
||||
" def custom_concat(*cols, delimeter: str = \"\"):\n",
|
||||
" return delimeter.join(cols)\n",
|
||||
"\n",
|
||||
" spark.udf.register(\"custom_concat\", custom_concat)\n",
|
||||
"\n",
|
||||
" # Execute SQL string.\n",
|
||||
" source_feature_group.createOrReplaceTempView(\"car_data\")\n",
|
||||
" aggregated_car_data = spark.sql(\n",
|
||||
" f\"\"\"\n",
|
||||
" SELECT \n",
|
||||
" custom_concat(model, \"_\", model_year, \"_\", status) as model_year_status,\n",
|
||||
" AVG(price) as avg_price,\n",
|
||||
" MAX(price) as max_price,\n",
|
||||
" AVG(mileage) as avg_mileage,\n",
|
||||
" MAX(mileage) as max_mileage,\n",
|
||||
" AVG(msrp) as avg_msrp,\n",
|
||||
" MAX(msrp) as max_msrp,\n",
|
||||
" \"{int(time.time())}\" as ingest_time\n",
|
||||
" FROM car_data\n",
|
||||
" GROUP BY model_year_status\n",
|
||||
" \"\"\"\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" aggregated_car_data.show()\n",
|
||||
"\n",
|
||||
" return aggregated_car_data\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Execute the aggregate\n",
|
||||
"aggregate()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "11e1a26a-35f1-4477-b71f-17c18c604ea7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## `to_pipeline and schedule`\n",
|
||||
"\n",
|
||||
"The following example demonstrates how to operationalize your feature processor by promoting it to a SageMaker Pipeline and configuring a schedule to execute it on a regular basis. This example uses the aggregate function defined above. Note, in order to create a pipeline, please make sure your method is annotated by both `@remote` and `@feature-processor` decorators."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "bfaf53b3-c394-47eb-b0be-6e51466d7b80",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"\"\"\"\n",
|
||||
"Upload the transformation_code.py saved earlier to S3, to track it in SageMaker ML Lineage.\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"from sagemaker.s3 import S3Uploader, s3_path_join\n",
|
||||
"\n",
|
||||
"car_data_s3_uri = s3_path_join(\n",
|
||||
" \"s3://\",\n",
|
||||
" sagemaker_session.default_bucket(),\n",
|
||||
" {default_bucket_prefix},\n",
|
||||
" \"transformation_code\",\n",
|
||||
" \"car-data-ingestion.py\",\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# If a default bucket prefix is specified, append it to the s3 path\n",
|
||||
"if default_bucket_prefix:\n",
|
||||
" car_data_s3_uri = s3_path_join(\n",
|
||||
" \"s3://\", sagemaker_session.default_bucket(), \"transformation_code\", \"car-data-ingestion.py\"\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"S3Uploader.upload(local_path=\"car-data-ingestion.py\", desired_s3_uri=car_data_s3_uri)\n",
|
||||
"\n",
|
||||
"print(car_data_s3_uri)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "0c3934b6",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"\"\"\"\n",
|
||||
"Annotate the transform method with @remote decorator so that we create Feature Processor Pipeline for it.\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"transform = remote(\n",
|
||||
" transform,\n",
|
||||
" spark_config=SparkConfig(),\n",
|
||||
" instance_type=\"ml.m5.2xlarge\",\n",
|
||||
" dependencies=\"./requirements.txt\",\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "83ef5ce7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In the following example, we will create and schedule the pipeline using `to_pipeline` and `schedule` method. If you want to test the job before scheduling, you can use `execute` to start only one execution.\n",
|
||||
"\n",
|
||||
"The SDK also provides two extra methods `describe` and `list_pipelines` for you to get insights about the pipeline info."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "4ca8cc26-259e-4fa9-accf-ba1b72914fad",
|
||||
"metadata": {
|
||||
"scrolled": true,
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sagemaker.feature_store.feature_processor as fp\n",
|
||||
"from sagemaker.feature_store.feature_processor import TransformationCode\n",
|
||||
"\n",
|
||||
"\"\"\"\n",
|
||||
"Create a Feature Processor Pipeline and start one execution.\n",
|
||||
"\"\"\"\n",
|
||||
"car_data_pipeline_name = f\"{CAR_SALES_FG_NAME}-ingestion-pipeline\"\n",
|
||||
"car_data_pipeline_arn = fp.to_pipeline(\n",
|
||||
" pipeline_name=car_data_pipeline_name,\n",
|
||||
" step=transform,\n",
|
||||
" transformation_code=TransformationCode(s3_uri=car_data_s3_uri),\n",
|
||||
")\n",
|
||||
"print(f\"Created SageMaker Pipeline: {car_data_pipeline_arn}.\")\n",
|
||||
"\n",
|
||||
"car_data_pipeline_execution_arn = fp.execute(pipeline_name=car_data_pipeline_name)\n",
|
||||
"print(f\"Started an execution with execution arn: {car_data_pipeline_execution_arn}\")\n",
|
||||
"\n",
|
||||
"fp.describe(pipeline_name=car_data_pipeline_name)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "fca016a1-65bb-4b4b-953e-b9ca55756e61",
|
||||
"metadata": {
|
||||
"scrolled": true,
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"\"\"\"\n",
|
||||
"Create a Feature Processor Pipeline and start one execution.\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"car_data_aggregated_pipeline_name = f\"{AGG_CAR_SALES_FG_NAME}-ingestion-pipeline\"\n",
|
||||
"car_data_aggregated_pipeline_arn = fp.to_pipeline(\n",
|
||||
" pipeline_name=car_data_aggregated_pipeline_name, step=aggregate\n",
|
||||
")\n",
|
||||
"print(f\"Created SageMaker Pipeline: {car_data_aggregated_pipeline_arn}.\")\n",
|
||||
"\n",
|
||||
"car_data_aggregated_pipeline_execution_arn = fp.execute(\n",
|
||||
" pipeline_name=car_data_aggregated_pipeline_name\n",
|
||||
")\n",
|
||||
"print(f\"Started an execution with execution arn: {car_data_aggregated_pipeline_execution_arn}\")\n",
|
||||
"\n",
|
||||
"\"\"\"\n",
|
||||
"Schedule the pipeline.\n",
|
||||
"\"\"\"\n",
|
||||
"fp.schedule(\n",
|
||||
" pipeline_name=car_data_aggregated_pipeline_name,\n",
|
||||
" schedule_expression=\"rate(24 hours)\",\n",
|
||||
" state=\"ENABLED\",\n",
|
||||
")\n",
|
||||
"print(f\"Created a schedule.\")\n",
|
||||
"\n",
|
||||
"fp.describe(pipeline_name=car_data_aggregated_pipeline_name)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "e2660dd0-31ff-4952-a4b3-66d9d1a5652a",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"\"\"\"\n",
|
||||
"Feature Processor Pipelines in this account.\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"fp.list_pipelines()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "be2d9751-e288-42db-b5fa-081939be66aa",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Explorating feature processing pipelines and ML Lineage.\n",
|
||||
"\n",
|
||||
"You can track scheduled SageMaker Pipelines with SageMaker Lineage in Amazon SageMaker Studio. This includes tracking scheduled executions, visualizing lineage to trace features back to their data sources, and viewing shared feature processing code all in one environment. \n",
|
||||
"\n",
|
||||
"Find the feature groups that were created in this notebook and view the Pipeline Executions and Lineage tabs.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "0e9af135",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Clean up Resources"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "9e30a53e",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Disable the scheduled pipeline\n",
|
||||
"fp.schedule(\n",
|
||||
" pipeline_name=car_data_aggregated_pipeline_name,\n",
|
||||
" schedule_expression=\"rate(24 hours)\",\n",
|
||||
" state=\"DISABLED\",\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(f\"Disabled the schedule.\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "fec416be",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Delete feature groups\n",
|
||||
"car_sales_fg.delete()\n",
|
||||
"agg_car_sales_fg.delete()\n",
|
||||
"\n",
|
||||
"print(f\"Feature groups are deleted.\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "6c1ebc50",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Notebook CI Test Results\n",
|
||||
"\n",
|
||||
"This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"instance_type": "ml.m5.2xlarge",
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (TensorFlow 2.10.0 Python 3.9 CPU Optimized)",
|
||||
"language": "python",
|
||||
"name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/tensorflow-2.10.1-cpu-py39-ubuntu20.04-sagemaker-v1.2"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.16"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
+101
@@ -0,0 +1,101 @@
|
||||
Id,Model,Year,Status,Mileage,Price,MSRP
|
||||
96864,6110,2019 Tesla Model 3 Standard Range,2019,Used,"50,412 mi.","$31,924.00",Not specified
|
||||
51128,7439,2016 Hyundai Sonata Sport,2016,Used,"97,237 mi.","$13,492.00",Not specified
|
||||
85327,4565,2021 Porsche Cayenne GTS,2021,Used,"23,375 mi.","$105,495.00",Not specified
|
||||
110510,4758,2020 Volkswagen Passat 2.0T SE,2020,Used,"46,249 mi.","$22,899.00",Not specified
|
||||
74498,3836,2023 Mercedes-Benz EQS 450+ EQS 450 4MATIC,2023,New,Not available,"$112,745.00",Not specified
|
||||
103711,3787,2023 Toyota Venza LE,2023,New,Not available,"$39,224.00",Not specified
|
||||
29375,97,2022 Dodge Challenger R/T Scat Pack,2022,New,Not available,"$62,985.00",Not specified
|
||||
41662,2511,2022 Ford F-250 XLT,2022,New,Not available,"$63,500.00","MSRP $64,500"
|
||||
60265,8276,2023 INFINITI QX50 Luxe,2023,New,Not available,"$49,100.00",Not specified
|
||||
2206,2206,2020 Acura RDX Technology Package,2020,Used,"24,712 mi.","$35,955.00",Not specified
|
||||
100010,86,2023 Toyota GR86 Premium,2023,New,Not available,"$35,683.00",$333 price drop
|
||||
54283,2294,2019 INFINITI QX50 ESSENTIAL,2019,Used,"34,986 mi.","$29,999.00",Not specified
|
||||
30829,1551,2022 Dodge Durango Citadel,2022,New,Not available,"$59,106.00",Not specified
|
||||
50828,7139,2023 Hyundai Santa Cruz 2.5T SEL Premium,2023,New,Not available,"$39,385.00","MSRP $39,385"
|
||||
85246,4484,2019 Porsche 911 GT3 RS,2019,Used,"4,241 mi.","$254,888.00",Not specified
|
||||
114257,8505,2018 Volkswagen Beetle 2.0T S,2018,Used,"46,926 mi.","$25,594.00",Not specified
|
||||
34452,5174,2008 Dodge Charger SXT,2008,Used,"133,467 mi.","$6,495.00",Not specified
|
||||
68825,8148,2011 Lexus IS-F Base,2011,Used,"36,800 mi.","$54,999.00",Not specified
|
||||
99988,64,2023 Toyota Tundra Platinum,2023,New,Not available,"$66,483.00",Not specified
|
||||
99824,9070,2023 Tesla Model 3 Base,2023,Used,"2,056 mi.","$45,888.00",Not specified
|
||||
103430,3506,2023 Toyota Camry XSE,2023,New,Not available,"$38,246.00",Not specified
|
||||
42818,3667,2017 Ford Focus ST Base,2017,Used,"64,421 mi.","$19,900.00",Not specified
|
||||
825,825,2020 Acura RDX Technology Package,2020,Used,"57,295 mi.","$31,881.00",Not specified
|
||||
98270,7516,2020 Tesla Model 3 Standard Range Plus,2020,Used,"8,817 mi.","$44,900.00",Not specified
|
||||
56852,4863,2019 INFINITI Q50 3.0t Signature Edition,2019,Used,"39,621 mi.","$27,500.00",Not specified
|
||||
108336,2584,2022 Volkswagen Atlas 3.6L SE w/Technology,2022,New,Not available,"$46,550.00",Not specified
|
||||
63813,3136,2021 Lexus RX 350L Premium,2021,Used,"16,560 mi.","$47,987.00","$2,000 price drop"
|
||||
100655,731,2023 Toyota Tundra Hybrid Platinum,2023,New,Not available,"$69,346.00",Not specified
|
||||
84924,4162,2014 Porsche 911 Carrera,2014,Used,"43,484 mi.","$67,950.00",$550 price drop
|
||||
113714,7962,2018 Volkswagen Atlas 3.6L SE,2018,Used,"62,614 mi.","$24,191.00",Not specified
|
||||
63736,3059,2018 Lexus RX 350L Premium,2018,Used,"34,918 mi.","$36,995.00",Not specified
|
||||
86932,6170,2012 Porsche Panamera 4,2012,Used,"42,986 mi.","$33,995.00",Not specified
|
||||
46008,2319,2023 Hyundai Palisade XRT,2023,New,Not available,"$42,470.00",Not specified
|
||||
96253,5499,2022 Tesla Model 3 Performance,2022,Used,"16,244 mi.","$49,999.00",Not specified
|
||||
6305,6305,2006 Acura TSX Base (A5),2006,Used,"84,825 mi.","$11,450.00",Not specified
|
||||
21555,2276,2023 Chevrolet Tahoe RST,2023,New,Not available,"$67,794.00","MSRP $67,295"
|
||||
61420,743,2020 Lexus ES 300h Luxury,2020,Used,"27,649 mi.","$44,953.00",Not specified
|
||||
32580,3302,2014 Dodge Challenger R/T,2014,Used,"164,569 mi.","$12,995.00",Not specified
|
||||
112597,6845,2023 Volkswagen Atlas Cross Sport 3.6L V6 SE w/Technology,2023,New,Not available,"$40,423.00","$3,500 price drop"
|
||||
44739,1050,2023 Hyundai Tucson Limited,2023,New,Not available,"$39,510.00","MSRP $39,510"
|
||||
83743,2981,2018 Porsche Cayenne Platinum Edition,2018,Used,"37,491 mi.","$39,495.00",Not specified
|
||||
59550,7561,2021 INFINITI Q50 3.0t LUXE,2021,Used,"49,667 mi.","$24,995.00",Not specified
|
||||
39410,259,2022 Ford Explorer Timberline,2022,New,Not available,"$53,480.00",Not specified
|
||||
75383,4721,2022 Mercedes-Benz GLC 300 Base 4MATIC,2022,New,Not available,"$50,260.00","MSRP $50,260"
|
||||
35442,6164,2017 Dodge Challenger SXT,2017,Used,"56,467 mi.","$20,503.00",Not specified
|
||||
39144,9866,2015 Dodge Durango Limited,2015,Used,"99,683 mi.","$18,399.00","$1,600 price drop"
|
||||
103281,3357,2023 Toyota Tacoma SR5,2023,New,Not available,"$33,950.00","MSRP $33,950"
|
||||
7780,7780,2023 Acura MDX Advance,2023,New,Not available,"$65,890.00",Not specified
|
||||
34843,5565,2019 Dodge Challenger R/T Scat Pack,2019,Used,"38,565 mi.","$44,132.00","$2,368 price drop"
|
||||
44389,700,2021 Hyundai Palisade SEL,2021,Used,"11,036 mi.","$41,990.00",Not specified
|
||||
68336,7659,2022 Lexus LS 500,2022,New,Not available,"$106,445.00","MSRP $106,445"
|
||||
104750,4826,2019 Toyota Highlander LE Plus,2019,Used,"42,300 mi.","$28,500.00","$2,500 price drop"
|
||||
3991,3991,2019 Acura RDX Base,2019,Used,"45,362 mi.","$30,998.00","$1,000 price drop"
|
||||
82796,2034,2013 Porsche Cayenne Turbo,2013,Used,"73,086 mi.","$33,395.00",$604 price drop
|
||||
39509,358,2022 Ford F-350 XL,2022,New,Not available,"$66,790.00",Not specified
|
||||
57364,5375,2023 INFINITI QX60 SENSORY,2023,New,Not available,"$65,718.00","MSRP $63,825"
|
||||
70829,167,2022 Mercedes-Benz GLE 450 AWD 4MATIC,2022,New,Not available,"$74,260.00","MSRP $74,260"
|
||||
49749,6060,2016 Hyundai Genesis 3.8,2016,Used,"74,714 mi.","$23,000.00",Not specified
|
||||
81569,807,2022 Porsche 718 Boxster GTS,2022,Porsche Certified,586 mi.,"$113,991.00",Not specified
|
||||
3281,3281,2021 Acura TLX Technology,2021,Used,"9,604 mi.","$35,594.00",$619 price drop
|
||||
92193,1439,2016 Tesla Model S 75,2016,Used,"55,343 mi.","$34,995.00","$4,000 price drop"
|
||||
42775,3624,2022 Ford Escape SE,2022,New,Not available,"$34,690.00",Not specified
|
||||
87560,6798,2021 Porsche 718 Cayman GTS 4.0,2021,Used,"3,331 mi.","$115,000.00",Not specified
|
||||
69040,8363,2011 Lexus ES 350 Base,2011,Used,"68,332 mi.","$16,940.00",Not specified
|
||||
41276,2125,2022 Ford F-350 XL,2022,New,Not available,"$69,630.00",Not specified
|
||||
95380,4626,2020 Tesla Model X Long Range,2020,Used,"32,000 mi.","$79,995.00",Not specified
|
||||
99894,9140,2018 Tesla Model 3 Long Range,2018,Used,"14,344 mi.","$40,500.00",Not specified
|
||||
51927,8238,2021 Hyundai Palisade SE,2021,Used,"58,452 mi.","$31,990.00",Not specified
|
||||
9240,9240,2023 Acura Integra A-Spec,2023,New,Not available,"$35,095.00","MSRP $35,095"
|
||||
44883,1194,2021 Hyundai Palisade SEL,2021,Used,"32,524 mi.","$36,795.00",Not specified
|
||||
89424,8662,2021 Porsche Cayenne S,2021,Used,"18,463 mi.","$89,989.00",Not specified
|
||||
83920,3158,2021 Porsche Cayenne AWD,2021,Used,"17,242 mi.","$79,900.00",Not specified
|
||||
61322,645,2023 Lexus LX 600 Premium,2023,New,Not available,"$103,150.00","MSRP $103,150"
|
||||
110236,4484,2019 Volkswagen Beetle 2.0T S,2019,Used,"20,406 mi.","$21,921.00",Not specified
|
||||
88109,7347,2018 Porsche 718 Boxster,2018,Used,"16,141 mi.","$64,990.00",Not specified
|
||||
66076,5399,2014 Lexus LS 460 Base,2014,Used,"140,725 mi.","$21,975.00",Not specified
|
||||
65897,5220,2019 Lexus UX 200 F-SPORT,2019,Used,"34,362 mi.","$32,998.00",Not specified
|
||||
55324,3335,2017 INFINITI Q50 3.0T Premium,2017,Used,"72,174 mi.","$20,997.00",Not specified
|
||||
31470,2192,2022 Dodge Durango R/T,2022,New,Not available,"$64,930.00","MSRP $66,275"
|
||||
71177,515,2023 Mercedes-Benz GLE 450 AWD 4MATIC,2023,New,Not available,"$82,915.00","MSRP $82,915"
|
||||
50393,6704,2017 Hyundai Accent SE,2017,Used,"75,704 mi.","$12,388.00",Not specified
|
||||
45609,1920,2022 Hyundai Sonata SEL,2022,Used,821 mi.,"$27,011.00",Not specified
|
||||
32524,3246,2022 Dodge Challenger SRT Hellcat,2022,New,Not available,"$85,230.00","MSRP $85,980"
|
||||
113644,7892,2023 Volkswagen Atlas Cross Sport 2.0T SEL,2023,New,Not available,"$48,938.00","MSRP $48,938"
|
||||
28415,9136,2014 Chevrolet Silverado 1500 LT,2014,Used,"110,316 mi.","$21,660.00",Not specified
|
||||
12150,2754,2023 BMW 540 i,2023,New,Not available,"$68,520.00","MSRP $68,520"
|
||||
36372,7094,2013 Dodge Charger SE,2013,Used,"125,336 mi.","$9,995.00","$1,000 price drop"
|
||||
11948,2552,2018 BMW 320 i xDrive,2018,Used,"53,250 mi.","$23,998.00",Not specified
|
||||
76352,5690,2023 Mercedes-Benz Maybach S 580 4MATIC,2023,New,Not available,"$228,850.00","MSRP $228,850"
|
||||
91021,267,2021 Tesla Model Y Long Range,2021,Used,"19,859 mi.","$49,000.00","$5,000 price drop"
|
||||
58210,6221,2008 INFINITI G35 Journey,2008,Used,"160,591 mi.","$6,900.00",Not specified
|
||||
101534,1610,2022 Toyota Tundra Hybrid Platinum,2022,New,Not available,"$72,622.00",Not specified
|
||||
54403,2414,2023 INFINITI QX80 PREMIUM SELECT,2023,New,Not available,"$82,010.00","MSRP $84,510"
|
||||
90442,9680,2022 Porsche Cayenne,2022,Porsche Certified,"4,137 mi.","$203,995.00",Not specified
|
||||
72757,2095,2022 Mercedes-Benz S-Class S 580 4MATIC,2022,New,Not available,"$152,685.00","MSRP $152,685"
|
||||
83365,2603,2003 Porsche 911 Turbo,2003,Used,"28,658 mi.","$115,900.00",Not specified
|
||||
90060,9298,2019 Porsche Cayenne Base,2019,Used,"32,364 mi.","$54,494.00",Not specified
|
||||
53711,1722,2013 INFINITI G37 Journey,2013,Used,"129,963 mi.","$11,995.00",Not specified
|
||||
60355,8366,2023 INFINITI QX80 SENSORY,2023,New,Not available,"$87,955.00","MSRP $87,955"
|
||||
75909,5247,2023 Mercedes-Benz GLS 450 4MATIC,2023,New,Not available,"$93,645.00","MSRP $93,645"
|
||||
|
+5
@@ -0,0 +1,5 @@
|
||||
customer_id,age,SSN,credit_score
|
||||
573291,26,266666666,500
|
||||
109382,42,422222222,600
|
||||
828400,35,355555555,700
|
||||
124013,61,611111111,800
|
||||
|
+5
@@ -0,0 +1,5 @@
|
||||
customer_id,city_code,state_code,country_code
|
||||
573291,1,49,2
|
||||
109382,2,40,2
|
||||
828400,3,31,2
|
||||
124013,4,5,2
|
||||
|
+5
@@ -0,0 +1,5 @@
|
||||
customer_id,city_code,state_code,country_code,email,name
|
||||
573291,1,49,2,john.lee@gmail.com,John Lee
|
||||
109382,2,40,2,olivequil@gmail.com,Olive Quil
|
||||
828400,3,31,2,liz.knee@gmail.com,Liz Knee
|
||||
124013,4,5,2,eileenbook@gmail.com,Eileen Book
|
||||
|
+5
@@ -0,0 +1,5 @@
|
||||
customer_id,order_id,order_status,store_id
|
||||
573291,4132,1,303
|
||||
109382,5724,0,201
|
||||
828400,1942,0,431
|
||||
124013,6782,1,213
|
||||
|
BIN
Binary file not shown.
+2
@@ -0,0 +1,2 @@
|
||||
{"source-ref":"s3://AWSDOC-EXAMPLE-BUCKET/example_image.jpg","species":"0","species-metadata":{"class-name": "dog","confidence": 0.00,"type": "groundtruth/image-classification","job-name": "identify-animal-species","human-annotated": "yes","creation-date": "2018-10-18T22:18:13.527256"}}
|
||||
{"source":"The food was delicious","mood":"1","mood-metadata":{"class-name": "positive","confidence": 0.8,"type": "groundtruth/text-classification","job-name": "label-sentiment","human-annotated": "yes","creation-date": "2020-10-18T22:18:13.527256"}}
|
||||
BIN
Binary file not shown.
|
After Width: | Height: | Size: 76 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 49 KiB |
+3
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 54 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 26 KiB |
+585
@@ -0,0 +1,585 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Amazon SageMaker Feature Store: Ground Truth Classification labelling job output to Feature Store"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"---"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This notebook demonstrates how to securely store the output of an image or text classification labelling job from [Amazon Ground Truth](https://docs.aws.amazon.com/sagemaker/latest/dg/sms.html) directly into Feature Store using a KMS key.\n",
|
||||
"\n",
|
||||
"This notebook starts by reading in the `output.manifest` file, which is the output file from your classification labeling job from Amazon SageMaker Ground Truth. You can substitute your own Amazon S3 bucket and path to a method we provide, which downloads the file to your current working directory. Then we prepare the manifest file for ingestion to an online or offline feature store. We use a [Key Management Service (KMS)](https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html) key for server-side encryption to ensure that your data is securely stored in your feature store.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"This notebook uses a KMS key for server side encryption for your Feature Store. For more information on server-side encryption, see [Feature Store: Encrypt Data in your Online or Offline Feature Store using KMS key](https://sagemaker-examples.readthedocs.io/en/latest/sagemaker-featurestore/feature_store_kms_key_encryption.html). \n",
|
||||
"\n",
|
||||
"To encrypt your data on the client side prior to ingestion, see [Amazon SageMaker Feature Store: Client-side Encryption using AWS Encryption SDK](https://sagemaker-examples.readthedocs.io/en/latest/sagemaker-featurestore/feature_store_client_side_encryption.html) for a demonstration. \n",
|
||||
"\n",
|
||||
"## Overview\n",
|
||||
"1. Set up.\n",
|
||||
"2. Prepare `output.manifest`for Feature Store. \n",
|
||||
"3. Create a feature group and ingest your data into it.\n",
|
||||
"\n",
|
||||
"## Prerequisites\n",
|
||||
"This notebook uses the Python SDK library for Feature Store, and the `Python 3 (Data Science)` kernel. To encrypt your data with KMS key for server side encryption, you will need to have an active KMS key. If you do not have a KMS key, then you can create one by following the [KMS Policy Template](https://sagemaker-examples.readthedocs.io/en/latest/sagemaker-featurestore/feature_store_kms_key_encryption.html#KMS-Policy-Template) steps, or you can visit the [KMS section in the console](https://console.aws.amazon.com/kms/home) and follow the prompts for creating a KMS key. This notebook is compatible with SageMaker Studio, Jupyter, and JupyterLab. \n",
|
||||
"\n",
|
||||
"## Library dependencies:\n",
|
||||
"* `sagemaker>=2.0.0`\n",
|
||||
"* `numpy`\n",
|
||||
"* `pandas`\n",
|
||||
"* `boto3`\n",
|
||||
"\n",
|
||||
"## Data\n",
|
||||
"This notebook uses a synthetic manifest file called `output.manifest` located in the data subfolder."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from time import gmtime, strftime\n",
|
||||
"from sagemaker.feature_store.feature_group import FeatureGroup\n",
|
||||
"\n",
|
||||
"import json\n",
|
||||
"import pandas as pd\n",
|
||||
"import sagemaker\n",
|
||||
"import time"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Set up"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"sagemaker_session = sagemaker.Session()\n",
|
||||
"s3_bucket_name = sagemaker_session.default_bucket() # This is the bucket for your offline store.\n",
|
||||
"default_bucket_prefix = sagemaker_session.default_bucket_prefix\n",
|
||||
"prefix = \"sagemaker-featurestore-demo\"\n",
|
||||
"\n",
|
||||
"# If a default bucket prefix is specified, append it to the s3 path\n",
|
||||
"if default_bucket_prefix:\n",
|
||||
" prefix = f\"{default_bucket_prefix}/{prefix}\"\n",
|
||||
"\n",
|
||||
"role = sagemaker.get_execution_role()\n",
|
||||
"region = sagemaker_session.boto_region_name"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Additional - Helper Method\n",
|
||||
"Below is a method that you can use to get your manifest file from your S3 bucket into your current working directory."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def download_file_from_s3(bucket, path, filename):\n",
|
||||
" \"\"\"\n",
|
||||
" Download filename to your current directory.\n",
|
||||
" Parameters:\n",
|
||||
" bucket: S3 bucket name\n",
|
||||
" path: path to file\n",
|
||||
" filename: the name of the file you are downloading\n",
|
||||
" Returns:\n",
|
||||
" None\n",
|
||||
" \"\"\"\n",
|
||||
" import os.path\n",
|
||||
"\n",
|
||||
" if not os.path.exists(filename):\n",
|
||||
" s3 = boto3.client(\"s3\")\n",
|
||||
" s3.download_file(Bucket=bucket, Key=path, Filename=filename)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Supply the path to your output.manifest file from your Ground Truth labelling job.\n",
|
||||
"# download_file_from_s3(public_s3_bucket_name, path='PATH', filename='output.manifest')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Prepare your manifest file for Feature Store. \n",
|
||||
"\n",
|
||||
"Below is a method that will parse your `output.manifest` file into a Panda's data frame for ingestion into your Feature Store. At this point, it is assumed that your manifest file is in your current working directory. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def create_dataframe_from_manifest(filename):\n",
|
||||
" \"\"\"\n",
|
||||
" Return a dataframe containing all information from your output.manifest file.\n",
|
||||
" Parameters:\n",
|
||||
" filename: path to your output.manifest file. This should be the\n",
|
||||
" output.manifest file from an AWS Ground Truth Classification labelling job.\n",
|
||||
" Returns:\n",
|
||||
" Data frame.\n",
|
||||
"\n",
|
||||
" Implementation details:\n",
|
||||
" i1 and i2: These are indices within dictionary d that we are looping through.\n",
|
||||
" k and j: k is a key of dictionary d which is also a dictionary, and j is a key of dictionary k.\n",
|
||||
" \"\"\"\n",
|
||||
" (\n",
|
||||
" item_name,\n",
|
||||
" classification,\n",
|
||||
" class_name_meta_data,\n",
|
||||
" confidence_meta_data,\n",
|
||||
" type_meta_data,\n",
|
||||
" job_name_meta_data,\n",
|
||||
" human_annotated_meta_data,\n",
|
||||
" creation_date,\n",
|
||||
" ) = ([] for _ in range(8))\n",
|
||||
"\n",
|
||||
" for entry in open(filename, \"r\"):\n",
|
||||
" d = json.loads(entry)\n",
|
||||
" for i1, k in enumerate(d.keys()):\n",
|
||||
" if i1 == 0:\n",
|
||||
" item_name.append(d[k])\n",
|
||||
" elif i1 == 1:\n",
|
||||
" classification.append(d[k])\n",
|
||||
" elif i1 == 2:\n",
|
||||
" for i2, j in enumerate(d[k].keys()):\n",
|
||||
" if i2 == 0:\n",
|
||||
" class_name_meta_data.append(d[k][j])\n",
|
||||
" elif i2 == 1:\n",
|
||||
" confidence_meta_data.append(d[k][j])\n",
|
||||
" elif i2 == 2:\n",
|
||||
" type_meta_data.append(d[k][j])\n",
|
||||
" elif i2 == 3:\n",
|
||||
" job_name_meta_data.append(d[k][j])\n",
|
||||
" elif i2 == 4:\n",
|
||||
" human_annotated_meta_data.append(d[k][j])\n",
|
||||
" elif i2 == 5:\n",
|
||||
" creation_date.append(d[k][j])\n",
|
||||
" return pd.DataFrame(\n",
|
||||
" {\n",
|
||||
" \"item_name\": item_name,\n",
|
||||
" \"classification\": classification,\n",
|
||||
" \"class_name_meta_data\": class_name_meta_data,\n",
|
||||
" \"confidence_meta_data\": confidence_meta_data,\n",
|
||||
" \"type_meta_data\": type_meta_data,\n",
|
||||
" \"job_name_meta_data\": job_name_meta_data,\n",
|
||||
" \"human_annotated_meta_data\": human_annotated_meta_data,\n",
|
||||
" \"creation_date\": creation_date,\n",
|
||||
" }\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# output.manifest is located in data/\n",
|
||||
"df = create_dataframe_from_manifest(\"data/output.manifest\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Preview the parsed manifest file as a data frame"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df.dtypes"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def cast_object_to_string(data_frame):\n",
|
||||
" \"\"\"\n",
|
||||
" Cast all columns of data_frame of type object to type string and return it.\n",
|
||||
" Parameters:\n",
|
||||
" data_frame: A pandas Dataframe\n",
|
||||
" Returns:\n",
|
||||
" Data frame\n",
|
||||
" \"\"\"\n",
|
||||
" for label in data_frame.columns:\n",
|
||||
" if data_frame.dtypes[label] == object:\n",
|
||||
" data_frame[label] = data_frame[label].astype(\"str\").astype(\"string\")\n",
|
||||
" return data_frame"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Cast columns of df of type object to string.\n",
|
||||
"df = cast_object_to_string(df)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Create Feature Group and Ingest data into it\n",
|
||||
"Below we start by appending the `EventTime` feature to your data to timestamp entries, then we load the feature definition, and instantiate the Feature Group object. Then lastly we ingest the data into your feature store."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"feature_group_name = \"ground-truth-classification-feature-group-\" + strftime(\n",
|
||||
" \"%d-%H-%M-%S\", gmtime()\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Instantiate a `FeatureGroup` object for your data. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"feature_group = FeatureGroup(name=feature_group_name, sagemaker_session=sagemaker_session)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"record_identifier_feature_name = \"item_name\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Append the `EventTime` feature to your data frame. This parameter is required, and time stamps each data point."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"current_time_sec = int(round(time.time()))\n",
|
||||
"\n",
|
||||
"event_time_feature_name = \"EventTime\"\n",
|
||||
"# append EventTime feature\n",
|
||||
"df[event_time_feature_name] = pd.Series([current_time_sec] * len(df), dtype=\"float64\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Load Feature Definition's of your data into your feature group."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"feature_group.load_feature_definitions(data_frame=df)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Create your feature group.\n",
|
||||
"\n",
|
||||
"**Important**: You will need to substitute your KMS Key ARN for `kms_key` for server side encryption. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"feature_group.create(\n",
|
||||
" s3_uri=f\"s3://{s3_bucket_name}/{prefix}\",\n",
|
||||
" record_identifier_name=record_identifier_feature_name,\n",
|
||||
" event_time_feature_name=\"EventTime\",\n",
|
||||
" role_arn=role,\n",
|
||||
" enable_online_store=False,\n",
|
||||
" offline_store_kms_key_id=kms_key,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"feature_group.describe()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Continually check your offline store until your data is available in it."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def check_feature_group_status(feature_group):\n",
|
||||
" \"\"\"\n",
|
||||
" Print when the feature group has been successfully created\n",
|
||||
" Parameters:\n",
|
||||
" feature_group: FeatureGroup\n",
|
||||
" Returns:\n",
|
||||
" None\n",
|
||||
" \"\"\"\n",
|
||||
" status = feature_group.describe().get(\"FeatureGroupStatus\")\n",
|
||||
" while status == \"Creating\":\n",
|
||||
" print(\"Waiting for Feature Group to be Created\")\n",
|
||||
" time.sleep(5)\n",
|
||||
" status = feature_group.describe().get(\"FeatureGroupStatus\")\n",
|
||||
" print(f\"FeatureGroup {feature_group.name} successfully created.\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"check_feature_group_status(feature_group)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Ingest your data into your feature group."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"feature_group.ingest(data_frame=df, max_workers=5, wait=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"time.sleep(30)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"s3_client = sagemaker_session.boto_session.client(\"s3\", region_name=region)\n",
|
||||
"\n",
|
||||
"feature_group_s3_uri = (\n",
|
||||
" feature_group.describe()\n",
|
||||
" .get(\"OfflineStoreConfig\")\n",
|
||||
" .get(\"S3StorageConfig\")\n",
|
||||
" .get(\"ResolvedOutputS3Uri\")\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"feature_group_s3_prefix = feature_group_s3_uri.replace(f\"s3://{s3_bucket_name}/\", \"\")\n",
|
||||
"\n",
|
||||
"# If a default bucket prefix is specified, append it to the s3 path\n",
|
||||
"if default_bucket_prefix:\n",
|
||||
" feature_group_s3_prefix = f\"{default_bucket_prefix}/{feature_group_s3_prefix}\"\n",
|
||||
"\n",
|
||||
"offline_store_contents = None\n",
|
||||
"while offline_store_contents is None:\n",
|
||||
" objects_in_bucket = s3_client.list_objects(\n",
|
||||
" Bucket=s3_bucket_name, Prefix=feature_group_s3_prefix\n",
|
||||
" )\n",
|
||||
" if \"Contents\" in objects_in_bucket and len(objects_in_bucket[\"Contents\"]) > 1:\n",
|
||||
" offline_store_contents = objects_in_bucket[\"Contents\"]\n",
|
||||
" else:\n",
|
||||
" print(\"Waiting for data in offline store...\\n\")\n",
|
||||
" time.sleep(60)\n",
|
||||
"\n",
|
||||
"print(\"Data available.\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Clean up resources\n",
|
||||
"Remove the Feature Group that was created. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"feature_group.delete()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Next steps\n",
|
||||
"\n",
|
||||
"In this notebook we covered how to securely store the output of an image or text classification labelling job from [Amazon Ground Truth](https://docs.aws.amazon.com/sagemaker/latest/dg/sms.html) directly into Feature Store using KMS key.\n",
|
||||
"\n",
|
||||
"To learn more about how server-side encryption is done with Feature Store, see [Feature Store: Encrypt Data in your Online or Offline Feature Store using KMS key](https://sagemaker-examples.readthedocs.io/en/latest/sagemaker-featurestore/feature_store_kms_key_encryption.html).\n",
|
||||
"\n",
|
||||
"To learn more about how to do client-side encryption to encrypt your image dataset prior to storing it in your feature store, see [Amazon SageMaker Feature Store: Client-side Encryption using AWS Encryption SDK](https://sagemaker-examples.readthedocs.io/en/latest/sagemaker-featurestore/feature_store_client_side_encryption.html). For more information on the AWS Encryption library, see [AWS Encryption SDK library](https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/introduction.html).\n",
|
||||
"\n",
|
||||
"For detailed information about Feature Store, see the [Developer Guide](https://docs.aws.amazon.com/sagemaker/latest/dg/feature-store.html).\n",
|
||||
"\n",
|
||||
"For a complete list of Feature Store notebooks, see [Feature Store notebook examples](https://docs.aws.amazon.com/sagemaker/latest/dg/feature-store-notebooks.html#feature-store-sample-notebooks)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Notebook CI Test Results\n",
|
||||
"\n",
|
||||
"This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"instance_type": "ml.t3.medium",
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (Data Science 3.0)",
|
||||
"language": "python",
|
||||
"name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
Id,Model,Year,Status,Mileage,Price,MSRP
|
||||
96864,6110,2019 Tesla Model 3 Standard Range,2019,Used,"50,412 mi.","$31,924.00",Not specified
|
||||
51128,7439,2016 Hyundai Sonata Sport,2016,Used,"97,237 mi.","$13,492.00",Not specified
|
||||
85327,4565,2021 Porsche Cayenne GTS,2021,Used,"23,375 mi.","$105,495.00",Not specified
|
||||
110510,4758,2020 Volkswagen Passat 2.0T SE,2020,Used,"46,249 mi.","$22,899.00",Not specified
|
||||
74498,3836,2023 Mercedes-Benz EQS 450+ EQS 450 4MATIC,2023,New,Not available,"$112,745.00",Not specified
|
||||
103711,3787,2023 Toyota Venza LE,2023,New,Not available,"$39,224.00",Not specified
|
||||
29375,97,2022 Dodge Challenger R/T Scat Pack,2022,New,Not available,"$62,985.00",Not specified
|
||||
41662,2511,2022 Ford F-250 XLT,2022,New,Not available,"$63,500.00","MSRP $64,500"
|
||||
60265,8276,2023 INFINITI QX50 Luxe,2023,New,Not available,"$49,100.00",Not specified
|
||||
2206,2206,2020 Acura RDX Technology Package,2020,Used,"24,712 mi.","$35,955.00",Not specified
|
||||
100010,86,2023 Toyota GR86 Premium,2023,New,Not available,"$35,683.00",$333 price drop
|
||||
54283,2294,2019 INFINITI QX50 ESSENTIAL,2019,Used,"34,986 mi.","$29,999.00",Not specified
|
||||
30829,1551,2022 Dodge Durango Citadel,2022,New,Not available,"$59,106.00",Not specified
|
||||
50828,7139,2023 Hyundai Santa Cruz 2.5T SEL Premium,2023,New,Not available,"$39,385.00","MSRP $39,385"
|
||||
85246,4484,2019 Porsche 911 GT3 RS,2019,Used,"4,241 mi.","$254,888.00",Not specified
|
||||
114257,8505,2018 Volkswagen Beetle 2.0T S,2018,Used,"46,926 mi.","$25,594.00",Not specified
|
||||
34452,5174,2008 Dodge Charger SXT,2008,Used,"133,467 mi.","$6,495.00",Not specified
|
||||
68825,8148,2011 Lexus IS-F Base,2011,Used,"36,800 mi.","$54,999.00",Not specified
|
||||
99988,64,2023 Toyota Tundra Platinum,2023,New,Not available,"$66,483.00",Not specified
|
||||
99824,9070,2023 Tesla Model 3 Base,2023,Used,"2,056 mi.","$45,888.00",Not specified
|
||||
103430,3506,2023 Toyota Camry XSE,2023,New,Not available,"$38,246.00",Not specified
|
||||
42818,3667,2017 Ford Focus ST Base,2017,Used,"64,421 mi.","$19,900.00",Not specified
|
||||
825,825,2020 Acura RDX Technology Package,2020,Used,"57,295 mi.","$31,881.00",Not specified
|
||||
98270,7516,2020 Tesla Model 3 Standard Range Plus,2020,Used,"8,817 mi.","$44,900.00",Not specified
|
||||
56852,4863,2019 INFINITI Q50 3.0t Signature Edition,2019,Used,"39,621 mi.","$27,500.00",Not specified
|
||||
108336,2584,2022 Volkswagen Atlas 3.6L SE w/Technology,2022,New,Not available,"$46,550.00",Not specified
|
||||
63813,3136,2021 Lexus RX 350L Premium,2021,Used,"16,560 mi.","$47,987.00","$2,000 price drop"
|
||||
100655,731,2023 Toyota Tundra Hybrid Platinum,2023,New,Not available,"$69,346.00",Not specified
|
||||
84924,4162,2014 Porsche 911 Carrera,2014,Used,"43,484 mi.","$67,950.00",$550 price drop
|
||||
113714,7962,2018 Volkswagen Atlas 3.6L SE,2018,Used,"62,614 mi.","$24,191.00",Not specified
|
||||
63736,3059,2018 Lexus RX 350L Premium,2018,Used,"34,918 mi.","$36,995.00",Not specified
|
||||
86932,6170,2012 Porsche Panamera 4,2012,Used,"42,986 mi.","$33,995.00",Not specified
|
||||
46008,2319,2023 Hyundai Palisade XRT,2023,New,Not available,"$42,470.00",Not specified
|
||||
96253,5499,2022 Tesla Model 3 Performance,2022,Used,"16,244 mi.","$49,999.00",Not specified
|
||||
6305,6305,2006 Acura TSX Base (A5),2006,Used,"84,825 mi.","$11,450.00",Not specified
|
||||
21555,2276,2023 Chevrolet Tahoe RST,2023,New,Not available,"$67,794.00","MSRP $67,295"
|
||||
61420,743,2020 Lexus ES 300h Luxury,2020,Used,"27,649 mi.","$44,953.00",Not specified
|
||||
32580,3302,2014 Dodge Challenger R/T,2014,Used,"164,569 mi.","$12,995.00",Not specified
|
||||
112597,6845,2023 Volkswagen Atlas Cross Sport 3.6L V6 SE w/Technology,2023,New,Not available,"$40,423.00","$3,500 price drop"
|
||||
44739,1050,2023 Hyundai Tucson Limited,2023,New,Not available,"$39,510.00","MSRP $39,510"
|
||||
83743,2981,2018 Porsche Cayenne Platinum Edition,2018,Used,"37,491 mi.","$39,495.00",Not specified
|
||||
59550,7561,2021 INFINITI Q50 3.0t LUXE,2021,Used,"49,667 mi.","$24,995.00",Not specified
|
||||
39410,259,2022 Ford Explorer Timberline,2022,New,Not available,"$53,480.00",Not specified
|
||||
75383,4721,2022 Mercedes-Benz GLC 300 Base 4MATIC,2022,New,Not available,"$50,260.00","MSRP $50,260"
|
||||
35442,6164,2017 Dodge Challenger SXT,2017,Used,"56,467 mi.","$20,503.00",Not specified
|
||||
39144,9866,2015 Dodge Durango Limited,2015,Used,"99,683 mi.","$18,399.00","$1,600 price drop"
|
||||
103281,3357,2023 Toyota Tacoma SR5,2023,New,Not available,"$33,950.00","MSRP $33,950"
|
||||
7780,7780,2023 Acura MDX Advance,2023,New,Not available,"$65,890.00",Not specified
|
||||
34843,5565,2019 Dodge Challenger R/T Scat Pack,2019,Used,"38,565 mi.","$44,132.00","$2,368 price drop"
|
||||
44389,700,2021 Hyundai Palisade SEL,2021,Used,"11,036 mi.","$41,990.00",Not specified
|
||||
68336,7659,2022 Lexus LS 500,2022,New,Not available,"$106,445.00","MSRP $106,445"
|
||||
104750,4826,2019 Toyota Highlander LE Plus,2019,Used,"42,300 mi.","$28,500.00","$2,500 price drop"
|
||||
3991,3991,2019 Acura RDX Base,2019,Used,"45,362 mi.","$30,998.00","$1,000 price drop"
|
||||
82796,2034,2013 Porsche Cayenne Turbo,2013,Used,"73,086 mi.","$33,395.00",$604 price drop
|
||||
39509,358,2022 Ford F-350 XL,2022,New,Not available,"$66,790.00",Not specified
|
||||
57364,5375,2023 INFINITI QX60 SENSORY,2023,New,Not available,"$65,718.00","MSRP $63,825"
|
||||
70829,167,2022 Mercedes-Benz GLE 450 AWD 4MATIC,2022,New,Not available,"$74,260.00","MSRP $74,260"
|
||||
49749,6060,2016 Hyundai Genesis 3.8,2016,Used,"74,714 mi.","$23,000.00",Not specified
|
||||
81569,807,2022 Porsche 718 Boxster GTS,2022,Porsche Certified,586 mi.,"$113,991.00",Not specified
|
||||
3281,3281,2021 Acura TLX Technology,2021,Used,"9,604 mi.","$35,594.00",$619 price drop
|
||||
92193,1439,2016 Tesla Model S 75,2016,Used,"55,343 mi.","$34,995.00","$4,000 price drop"
|
||||
42775,3624,2022 Ford Escape SE,2022,New,Not available,"$34,690.00",Not specified
|
||||
87560,6798,2021 Porsche 718 Cayman GTS 4.0,2021,Used,"3,331 mi.","$115,000.00",Not specified
|
||||
69040,8363,2011 Lexus ES 350 Base,2011,Used,"68,332 mi.","$16,940.00",Not specified
|
||||
41276,2125,2022 Ford F-350 XL,2022,New,Not available,"$69,630.00",Not specified
|
||||
95380,4626,2020 Tesla Model X Long Range,2020,Used,"32,000 mi.","$79,995.00",Not specified
|
||||
99894,9140,2018 Tesla Model 3 Long Range,2018,Used,"14,344 mi.","$40,500.00",Not specified
|
||||
51927,8238,2021 Hyundai Palisade SE,2021,Used,"58,452 mi.","$31,990.00",Not specified
|
||||
9240,9240,2023 Acura Integra A-Spec,2023,New,Not available,"$35,095.00","MSRP $35,095"
|
||||
44883,1194,2021 Hyundai Palisade SEL,2021,Used,"32,524 mi.","$36,795.00",Not specified
|
||||
89424,8662,2021 Porsche Cayenne S,2021,Used,"18,463 mi.","$89,989.00",Not specified
|
||||
83920,3158,2021 Porsche Cayenne AWD,2021,Used,"17,242 mi.","$79,900.00",Not specified
|
||||
61322,645,2023 Lexus LX 600 Premium,2023,New,Not available,"$103,150.00","MSRP $103,150"
|
||||
110236,4484,2019 Volkswagen Beetle 2.0T S,2019,Used,"20,406 mi.","$21,921.00",Not specified
|
||||
88109,7347,2018 Porsche 718 Boxster,2018,Used,"16,141 mi.","$64,990.00",Not specified
|
||||
66076,5399,2014 Lexus LS 460 Base,2014,Used,"140,725 mi.","$21,975.00",Not specified
|
||||
65897,5220,2019 Lexus UX 200 F-SPORT,2019,Used,"34,362 mi.","$32,998.00",Not specified
|
||||
55324,3335,2017 INFINITI Q50 3.0T Premium,2017,Used,"72,174 mi.","$20,997.00",Not specified
|
||||
31470,2192,2022 Dodge Durango R/T,2022,New,Not available,"$64,930.00","MSRP $66,275"
|
||||
71177,515,2023 Mercedes-Benz GLE 450 AWD 4MATIC,2023,New,Not available,"$82,915.00","MSRP $82,915"
|
||||
50393,6704,2017 Hyundai Accent SE,2017,Used,"75,704 mi.","$12,388.00",Not specified
|
||||
45609,1920,2022 Hyundai Sonata SEL,2022,Used,821 mi.,"$27,011.00",Not specified
|
||||
32524,3246,2022 Dodge Challenger SRT Hellcat,2022,New,Not available,"$85,230.00","MSRP $85,980"
|
||||
113644,7892,2023 Volkswagen Atlas Cross Sport 2.0T SEL,2023,New,Not available,"$48,938.00","MSRP $48,938"
|
||||
28415,9136,2014 Chevrolet Silverado 1500 LT,2014,Used,"110,316 mi.","$21,660.00",Not specified
|
||||
12150,2754,2023 BMW 540 i,2023,New,Not available,"$68,520.00","MSRP $68,520"
|
||||
36372,7094,2013 Dodge Charger SE,2013,Used,"125,336 mi.","$9,995.00","$1,000 price drop"
|
||||
11948,2552,2018 BMW 320 i xDrive,2018,Used,"53,250 mi.","$23,998.00",Not specified
|
||||
76352,5690,2023 Mercedes-Benz Maybach S 580 4MATIC,2023,New,Not available,"$228,850.00","MSRP $228,850"
|
||||
91021,267,2021 Tesla Model Y Long Range,2021,Used,"19,859 mi.","$49,000.00","$5,000 price drop"
|
||||
58210,6221,2008 INFINITI G35 Journey,2008,Used,"160,591 mi.","$6,900.00",Not specified
|
||||
101534,1610,2022 Toyota Tundra Hybrid Platinum,2022,New,Not available,"$72,622.00",Not specified
|
||||
54403,2414,2023 INFINITI QX80 PREMIUM SELECT,2023,New,Not available,"$82,010.00","MSRP $84,510"
|
||||
90442,9680,2022 Porsche Cayenne,2022,Porsche Certified,"4,137 mi.","$203,995.00",Not specified
|
||||
72757,2095,2022 Mercedes-Benz S-Class S 580 4MATIC,2022,New,Not available,"$152,685.00","MSRP $152,685"
|
||||
83365,2603,2003 Porsche 911 Turbo,2003,Used,"28,658 mi.","$115,900.00",Not specified
|
||||
90060,9298,2019 Porsche Cayenne Base,2019,Used,"32,364 mi.","$54,494.00",Not specified
|
||||
53711,1722,2013 INFINITI G37 Journey,2013,Used,"129,963 mi.","$11,995.00",Not specified
|
||||
60355,8366,2023 INFINITI QX80 SENSORY,2023,New,Not available,"$87,955.00","MSRP $87,955"
|
||||
75909,5247,2023 Mercedes-Benz GLS 450 4MATIC,2023,New,Not available,"$93,645.00","MSRP $93,645"
|
||||
|
@@ -0,0 +1,5 @@
|
||||
customer_id,age,SSN,credit_score
|
||||
573291,26,266666666,500
|
||||
109382,42,422222222,600
|
||||
828400,35,355555555,700
|
||||
124013,61,611111111,800
|
||||
|
+5
@@ -0,0 +1,5 @@
|
||||
customer_id,city_code,state_code,country_code
|
||||
573291,1,49,2
|
||||
109382,2,40,2
|
||||
828400,3,31,2
|
||||
124013,4,5,2
|
||||
|
+5
@@ -0,0 +1,5 @@
|
||||
customer_id,city_code,state_code,country_code,email,name
|
||||
573291,1,49,2,john.lee@gmail.com,John Lee
|
||||
109382,2,40,2,olivequil@gmail.com,Olive Quil
|
||||
828400,3,31,2,liz.knee@gmail.com,Liz Knee
|
||||
124013,4,5,2,eileenbook@gmail.com,Eileen Book
|
||||
|
@@ -0,0 +1,5 @@
|
||||
customer_id,order_id,order_status,store_id
|
||||
573291,4132,1,303
|
||||
109382,5724,0,201
|
||||
828400,1942,0,431
|
||||
124013,6782,1,213
|
||||
|
Binary file not shown.
@@ -0,0 +1,2 @@
|
||||
{"source-ref":"s3://AWSDOC-EXAMPLE-BUCKET/example_image.jpg","species":"0","species-metadata":{"class-name": "dog","confidence": 0.00,"type": "groundtruth/image-classification","job-name": "identify-animal-species","human-annotated": "yes","creation-date": "2018-10-18T22:18:13.527256"}}
|
||||
{"source":"The food was delicious","mood":"1","mood-metadata":{"class-name": "positive","confidence": 0.8,"type": "groundtruth/text-classification","job-name": "label-sentiment","human-annotated": "yes","creation-date": "2020-10-18T22:18:13.527256"}}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 76 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 49 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 54 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 26 KiB |
@@ -0,0 +1,772 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Amazon SageMaker Feature Store: Introduction to Feature Store"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"---"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This notebook demonstrates how to get started with Feature Store, create feature groups, and ingest data into them. These feature groups are stored in your Feature Store.\n",
|
||||
"\n",
|
||||
"Feature groups are resources that contain metadata for all data stored in your Feature Store. A feature group is a logical grouping of features, defined in the feature store to describe records. A feature group’s definition is composed of a list of feature definitions, a record identifier name, and configurations for its online and offline store. \n",
|
||||
"\n",
|
||||
"### Overview\n",
|
||||
"1. Set up\n",
|
||||
"2. Creating a feature group\n",
|
||||
"3. Ingest data into a feature group\n",
|
||||
"\n",
|
||||
"### Prerequisites\n",
|
||||
"This notebook uses both `boto3` and Python SDK libraries, and the `Python 3 (Data Science)` kernel. This notebook works with Studio, Jupyter, and JupyterLab. \n",
|
||||
"\n",
|
||||
"#### Library dependencies:\n",
|
||||
"* `sagemaker>=2.100.0`\n",
|
||||
"* `numpy`\n",
|
||||
"* `pandas`\n",
|
||||
"\n",
|
||||
"#### Role requirements:\n",
|
||||
"**IMPORTANT**: You must attach the following policies to your execution role:\n",
|
||||
"* `AmazonS3FullAccess`\n",
|
||||
"* `AmazonSageMakerFeatureStoreAccess`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Set up"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# SageMaker Python SDK version 2.100.0 is required\n",
|
||||
"# boto3 version 1.24.20 is required\n",
|
||||
"import sagemaker\n",
|
||||
"import boto3\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"!pip install \"sagemaker>=2.100.0,<3.0\"\n",
|
||||
"!pip install 'boto3>=1.24.20'"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import pandas as pd\n",
|
||||
"import numpy as np\n",
|
||||
"import io\n",
|
||||
"from sagemaker.session import Session\n",
|
||||
"from sagemaker import get_execution_role\n",
|
||||
"\n",
|
||||
"prefix = \"sagemaker-featurestore-introduction\"\n",
|
||||
"role = get_execution_role()\n",
|
||||
"\n",
|
||||
"sagemaker_session = sagemaker.Session()\n",
|
||||
"region = sagemaker_session.boto_region_name\n",
|
||||
"s3_bucket_name = sagemaker_session.default_bucket()\n",
|
||||
"default_bucket_prefix = sagemaker_session.default_bucket_prefix\n",
|
||||
"\n",
|
||||
"# If a default bucket prefix is specified, append it to the s3 path\n",
|
||||
"if default_bucket_prefix:\n",
|
||||
" prefix = f\"{default_bucket_prefix}/{prefix}\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Inspect your data\n",
|
||||
"In this notebook example we ingest synthetic data. We read from `./data/feature_store_introduction_customer.csv` and `./data/feature_store_introduction_orders.csv`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customer_data = pd.read_csv(\"data/feature_store_introduction_customer.csv\")\n",
|
||||
"orders_data = pd.read_csv(\"data/feature_store_introduction_orders.csv\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customer_data.head()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"orders_data.head()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Below is an illustration on the steps the data goes through before it is ingested into a Feature Store. In this notebook, we illustrate the use-case where you have data from multiple sources and want to store them independently in a feature store. Our example considers data from a data warehouse (customer data), and data from a real-time streaming service (order data). "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Create a feature group\n",
|
||||
"\n",
|
||||
"We first start by creating feature group names for customer_data and orders_data. Following this, we create two Feature Groups, one for `customer_data` and another for `orders_data`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from time import gmtime, strftime, sleep\n",
|
||||
"\n",
|
||||
"customers_feature_group_name = \"customers-feature-group-\" + strftime(\"%d-%H-%M-%S\", gmtime())\n",
|
||||
"orders_feature_group_name = \"orders-feature-group-\" + strftime(\"%d-%H-%M-%S\", gmtime())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Instantiate a FeatureGroup object for customers_data and orders_data. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker.feature_store.feature_group import FeatureGroup\n",
|
||||
"\n",
|
||||
"customers_feature_group = FeatureGroup(\n",
|
||||
" name=customers_feature_group_name, sagemaker_session=sagemaker_session\n",
|
||||
")\n",
|
||||
"orders_feature_group = FeatureGroup(\n",
|
||||
" name=orders_feature_group_name, sagemaker_session=sagemaker_session\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import time\n",
|
||||
"\n",
|
||||
"current_time_sec = int(round(time.time()))\n",
|
||||
"\n",
|
||||
"record_identifier_feature_name = \"customer_id\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Append `EventTime` feature to your data frame. This parameter is required, and time stamps each data point."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customer_data[\"EventTime\"] = pd.Series([current_time_sec] * len(customer_data), dtype=\"float64\")\n",
|
||||
"orders_data[\"EventTime\"] = pd.Series([current_time_sec] * len(orders_data), dtype=\"float64\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Load feature definitions to your feature group. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customers_feature_group.load_feature_definitions(data_frame=customer_data)\n",
|
||||
"orders_feature_group.load_feature_definitions(data_frame=orders_data)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Below we call create to create two feature groups, customers_feature_group and orders_feature_group respectively"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customers_feature_group.create(\n",
|
||||
" s3_uri=f\"s3://{s3_bucket_name}/{prefix}\",\n",
|
||||
" record_identifier_name=record_identifier_feature_name,\n",
|
||||
" event_time_feature_name=\"EventTime\",\n",
|
||||
" role_arn=role,\n",
|
||||
" enable_online_store=True,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"orders_feature_group.create(\n",
|
||||
" s3_uri=f\"s3://{s3_bucket_name}/{prefix}\",\n",
|
||||
" record_identifier_name=record_identifier_feature_name,\n",
|
||||
" event_time_feature_name=\"EventTime\",\n",
|
||||
" role_arn=role,\n",
|
||||
" enable_online_store=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"To confirm that your FeatureGroup has been created we use `DescribeFeatureGroup` and `ListFeatureGroups` APIs to display the created FeatureGroup."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customers_feature_group.describe()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"orders_feature_group.describe()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"sagemaker_session.boto_session.client(\n",
|
||||
" \"sagemaker\", region_name=region\n",
|
||||
").list_feature_groups() # We use the boto client to list FeatureGroups"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def check_feature_group_status(feature_group):\n",
|
||||
" status = feature_group.describe().get(\"FeatureGroupStatus\")\n",
|
||||
" while status == \"Creating\":\n",
|
||||
" print(\"Waiting for Feature Group to be Created\")\n",
|
||||
" time.sleep(5)\n",
|
||||
" status = feature_group.describe().get(\"FeatureGroupStatus\")\n",
|
||||
" print(f\"FeatureGroup {feature_group.name} successfully created.\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"check_feature_group_status(customers_feature_group)\n",
|
||||
"check_feature_group_status(orders_feature_group)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Add metadata to a feature\n",
|
||||
"\n",
|
||||
"We can add searchable metadata fields to FeatureGroup features by using the `UpdateFeatureMetadata` API. The currently supported metadata fields are `description` and `parameters`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker.feature_store.inputs import FeatureParameter\n",
|
||||
"\n",
|
||||
"customers_feature_group.update_feature_metadata(\n",
|
||||
" feature_name=\"customer_id\",\n",
|
||||
" description=\"The ID of a customer. It is also used in orders_feature_group.\",\n",
|
||||
" parameter_additions=[FeatureParameter(\"idType\", \"primaryKey\")],\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"To view feature metadata, we can use `DescribeFeatureMetadata` to display that feature."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customers_feature_group.describe_feature_metadata(feature_name=\"customer_id\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Feature metadata fields are searchable. We use `search` API to find features with metadata that matches some search criteria."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"sagemaker_session.boto_session.client(\"sagemaker\", region_name=region).search(\n",
|
||||
" Resource=\"FeatureMetadata\",\n",
|
||||
" SearchExpression={\n",
|
||||
" \"Filters\": [\n",
|
||||
" {\n",
|
||||
" \"Name\": \"FeatureGroupName\",\n",
|
||||
" \"Operator\": \"Contains\",\n",
|
||||
" \"Value\": \"customers-feature-group-\",\n",
|
||||
" },\n",
|
||||
" {\"Name\": \"Parameters.idType\", \"Operator\": \"Equals\", \"Value\": \"primaryKey\"},\n",
|
||||
" ]\n",
|
||||
" },\n",
|
||||
") # We use the boto client to search"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Ingest data into a feature group\n",
|
||||
"\n",
|
||||
"We can put data into the FeatureGroup by using the `PutRecord` API. It will take < 1 minute to ingest data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customers_feature_group.ingest(data_frame=customer_data, max_workers=3, wait=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"orders_feature_group.ingest(data_frame=orders_data, max_workers=3, wait=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Using an arbitrary customer record ID, 573291 we use `get_record` to check that the data has been ingested into the feature group."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customer_id = 573291\n",
|
||||
"sample_record = sagemaker_session.boto_session.client(\n",
|
||||
" \"sagemaker-featurestore-runtime\", region_name=region\n",
|
||||
").get_record(\n",
|
||||
" FeatureGroupName=customers_feature_group_name, RecordIdentifierValueAsString=str(customer_id)\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"sample_record"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We use `batch_get_record` to check that all data has been ingested into two feature groups by providing customer IDs."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"all_records = sagemaker_session.boto_session.client(\n",
|
||||
" \"sagemaker-featurestore-runtime\", region_name=region\n",
|
||||
").batch_get_record(\n",
|
||||
" Identifiers=[\n",
|
||||
" {\n",
|
||||
" \"FeatureGroupName\": customers_feature_group_name,\n",
|
||||
" \"RecordIdentifiersValueAsString\": [\"573291\", \"109382\", \"828400\", \"124013\"],\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"FeatureGroupName\": orders_feature_group_name,\n",
|
||||
" \"RecordIdentifiersValueAsString\": [\"573291\", \"109382\", \"828400\", \"124013\"],\n",
|
||||
" },\n",
|
||||
" ]\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"all_records"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Add features to a feature group\n",
|
||||
"\n",
|
||||
"If we want to update a FeatureGroup that has done the data ingestion, we can use the `UpdateFeatureGroup` API and then re-ingest data by using the updated dataset."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sagemaker.feature_store.feature_definition import StringFeatureDefinition\n",
|
||||
"\n",
|
||||
"customers_feature_group.update(\n",
|
||||
" feature_additions=[StringFeatureDefinition(\"email\"), StringFeatureDefinition(\"name\")]\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Verify the FeatureGroup has been updated successfully or not."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def check_last_update_status(feature_group):\n",
|
||||
" last_update_status = feature_group.describe().get(\"LastUpdateStatus\")[\"Status\"]\n",
|
||||
" while last_update_status == \"InProgress\":\n",
|
||||
" print(\"Waiting for FeatureGroup to be updated\")\n",
|
||||
" time.sleep(5)\n",
|
||||
" last_update_status = feature_group.describe().get(\"LastUpdateStatus\")\n",
|
||||
" if last_update_status == \"Successful\":\n",
|
||||
" print(f\"FeatureGroup {feature_group.name} successfully updated.\")\n",
|
||||
" else:\n",
|
||||
" print(\n",
|
||||
" f\"FeatureGroup {feature_group.name} updated failed. The LastUpdateStatus is\"\n",
|
||||
" + str(last_update_status)\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"check_last_update_status(customers_feature_group)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Inspect the new dataset."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customer_data_updated = pd.read_csv(\"data/feature_store_introduction_customer_updated.csv\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customer_data_updated.head()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Append `EventTime` feature to your data frame again."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customer_data_updated[\"EventTime\"] = pd.Series(\n",
|
||||
" [current_time_sec] * len(customer_data), dtype=\"float64\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Ingest the new dataset."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customers_feature_group.ingest(data_frame=customer_data_updated, max_workers=3, wait=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Use `batch_get_record` again to check that all updated data has been ingested into `customers_feature_group` by providing customer IDs."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"updated_customers_records = sagemaker_session.boto_session.client(\n",
|
||||
" \"sagemaker-featurestore-runtime\", region_name=region\n",
|
||||
").batch_get_record(\n",
|
||||
" Identifiers=[\n",
|
||||
" {\n",
|
||||
" \"FeatureGroupName\": customers_feature_group_name,\n",
|
||||
" \"RecordIdentifiersValueAsString\": [\"573291\", \"109382\", \"828400\", \"124013\"],\n",
|
||||
" }\n",
|
||||
" ]\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"updated_customers_records"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Clean up\n",
|
||||
"Here we remove the Feature Groups we created. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"customers_feature_group.delete()\n",
|
||||
"orders_feature_group.delete()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Next steps"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In this notebook you learned how to quickly get started with Feature Store and now know how to create feature groups, and ingest data into them.\n",
|
||||
"\n",
|
||||
"For an advanced example on how to use Feature Store for a Fraud Detection use-case, see [Fraud Detection with Feature Store](https://sagemaker-examples.readthedocs.io/en/latest/sagemaker-featurestore/sagemaker_featurestore_fraud_detection_python_sdk.html).\n",
|
||||
"\n",
|
||||
"For detailed information about Feature Store, see the [Developer Guide](https://docs.aws.amazon.com/sagemaker/latest/dg/feature-store.html).\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Programmers note"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In this notebook we used a variety of different API calls. Most of them are accessible through the Python SDK, however some only exist within `boto3`. You can invoke the Python SDK API calls directly on your Feature Store objects, whereas to invoke API calls that exist within `boto3`, you must first access a boto client through your boto and sagemaker sessions: e.g. `sagemaker_session.boto_session.client()`.\n",
|
||||
"\n",
|
||||
"Below we list API calls used in this notebook that exist within the Python SDK and ones that exist in `boto3` for your reference. \n",
|
||||
"\n",
|
||||
"#### Python SDK API Calls\n",
|
||||
"* `describe()`\n",
|
||||
"* `ingest()`\n",
|
||||
"* `delete()`\n",
|
||||
"* `create()`\n",
|
||||
"* `load_feature_definitions()`\n",
|
||||
"* `update()`\n",
|
||||
"* `update_feature_metadata()`\n",
|
||||
"* `describe_feature_metadata()`\n",
|
||||
"\n",
|
||||
"#### Boto3 API Calls\n",
|
||||
"* `list_feature_groups()`\n",
|
||||
"* `get_record()`\n",
|
||||
"* `batch_get_record()`\n",
|
||||
"* `search()`\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Notebook CI Test Results\n",
|
||||
"\n",
|
||||
"This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"instance_type": "ml.t3.medium",
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (Data Science 2.0)",
|
||||
"language": "python",
|
||||
"name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/sagemaker-data-science-38"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.13"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
@@ -0,0 +1,841 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Create a 3D Point Cloud Labeling Job with Amazon SageMaker Ground Truth\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"\n",
|
||||
"This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"---"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n",
|
||||
"\n",
|
||||
"This sample notebook takes you through an end-to-end workflow to demonstrate the functionality of SageMaker Ground Truth 3D point cloud built-in task types. \n",
|
||||
"\n",
|
||||
"### What is a Point Cloud\n",
|
||||
"\n",
|
||||
"A point cloud frame is defined as a collection of 3D points describing a 3D scene. Each point is described using three coordinates, x, y, and z. To add color and/or variations in point intensity to the point cloud, points may have additional attributes, such as i for intensity or values for the red (r), green (g), and blue (b) color channels (8-bit). All of the positional coordinates (x, y, z) are in meters. Point clouds are most commonly created from data that was collected by scanning the real world through various scanning methods, such as laser scanning and photogrammetry. Ground Truth currently also supports sensor fusion with video camera data. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"### 3D Point Cloud Built in Task Types\n",
|
||||
"\n",
|
||||
"You can use Ground Truth 3D point cloud labeling built-in task types to annotate 3D point cloud data. The following list briefly describes each task type. See [3D Point Cloud Task types](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-task-types.html) for more information.\n",
|
||||
"\n",
|
||||
"* 3D point cloud object detection – Use this task type when you want workers to indentify the location of and classify objects in a 3D point cloud by drawing 3D cuboids around objects. You can include one or more attributes for each class (label) you provide.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"* 3D point cloud object tracking – Use this task type when you want workers to track the trajectory of an object across a sequence of 3D point cloud frames. For example, you can use this task type to ask workers to track the movement of vehicles across a sequence of point cloud frames.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"* 3D point cloud semantic segmentation – Use this task type when you want workers to create a point-level semantic segmentation mask by painting objects in a 3D point cloud using different colors where each color is assigned to one of the classes you specify.\n",
|
||||
"\n",
|
||||
"You can use the Adjustment task types to verify and adjust annotations created for the three task types above.\n",
|
||||
"\n",
|
||||
"### Sensor Fusion\n",
|
||||
"\n",
|
||||
"One of the important features of this product is sensor fusion, which fuses the inputs of multiple sensors to provide labelers with a better understanding of the 3D scene. \n",
|
||||
"\n",
|
||||
"When you create a 3D point cloud labeling job, you can optionally provide camera data for sensor fusion. Ground Truth uses your camera data to include images in the worker UI. These images give workers more visual information about scenes in the 3D point cloud visualization and can be used to annotate (draw 3D cuboids or paint) objects. When a worker adjusts annotation around an object in either the 2D image or the 3D point cloud, the adjustments shows up in the other medium. This tutorial will demonstrate how you can include image data in your input manifest file for sensor fusion. \n",
|
||||
"\n",
|
||||
"## This Demo\n",
|
||||
"\n",
|
||||
"In this demo, you'll start by inspecting the input data and manifest files used to in the demo. Then, you will specifying resources needed to create a labeling job. In this step, you'll have the option to make yourself a worker on a private work team that you send the labeling job tasks to. This will allow you can preview and interact with the worker UI. Finally, you'll configure and send your CreateLabelingJob request.\n",
|
||||
"\n",
|
||||
"After your labeling job has been created, you can check your labeling job status. When the job is completed, you can view the output in Amazon S3. \n",
|
||||
"\n",
|
||||
"## Prerequisites\n",
|
||||
"\n",
|
||||
"To run this notebook, you can simply execute each cell in order. To understand what's happening, you'll need:\n",
|
||||
"\n",
|
||||
"* An S3 bucket you can write to -- please provide its name in `BUCKET`. The bucket must be in the same region as this SageMaker Notebook instance. You can also change the `EXP_NAME` to any valid S3 prefix. All the files related to this experiment will be stored in that prefix of your bucket. **Important: you must attach the CORS policy to this bucket. See the next section for more information**.\n",
|
||||
"* Familiarity with the [Ground Truth 3D Point Cloud Labeling Job](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud.html).\n",
|
||||
"* Familiarity with Python and [numpy](http://www.numpy.org/).\n",
|
||||
"* Basic familiarity with [AWS S3](https://docs.aws.amazon.com/s3/index.html).\n",
|
||||
"* Basic understanding of [AWS Sagemaker](https://aws.amazon.com/sagemaker/).\n",
|
||||
"* Basic familiarity with [AWS Command Line Interface (CLI)](https://aws.amazon.com/cli/) -- ideally, you should have it set up with credentials to access the AWS account you're running this notebook from.\n",
|
||||
"\n",
|
||||
"This notebook has only been tested on a SageMaker notebook instance. The runtimes given are approximate. We used an `ml.t2.medium` instance in our tests. However, you can likely run it on a local instance by first executing the cell below on SageMaker and then copying the `role` string to your local copy of the notebook."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### IMPORTANT: Attach CORS policy to your bucket\n",
|
||||
"\n",
|
||||
"You must attach the following CORS policy to your S3 bucket for the labeling task to render. To learn how to add a CORS policy to your S3 bucket, follow the instructions in [How do I add cross-domain resource sharing with CORS?](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/add-cors-configuration.html). Paste the following policy in the CORS configuration editor:\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n",
|
||||
"<CORSConfiguration xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\">\n",
|
||||
"<CORSRule>\n",
|
||||
" <AllowedOrigin>*</AllowedOrigin>\n",
|
||||
" <AllowedMethod>GET</AllowedMethod>\n",
|
||||
" <AllowedMethod>HEAD</AllowedMethod>\n",
|
||||
" <AllowedMethod>PUT</AllowedMethod>\n",
|
||||
" <MaxAgeSeconds>3000</MaxAgeSeconds>\n",
|
||||
" <ExposeHeader>Access-Control-Allow-Origin</ExposeHeader>\n",
|
||||
" <AllowedHeader>*</AllowedHeader>\n",
|
||||
"</CORSRule>\n",
|
||||
"<CORSRule>\n",
|
||||
" <AllowedOrigin>*</AllowedOrigin>\n",
|
||||
" <AllowedMethod>GET</AllowedMethod>\n",
|
||||
"</CORSRule>\n",
|
||||
"</CORSConfiguration>\n",
|
||||
"```\n",
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install boto3==1.14.8\n",
|
||||
"!pip install -U botocore"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import boto3\n",
|
||||
"import botocore\n",
|
||||
"import time\n",
|
||||
"import pprint\n",
|
||||
"import json\n",
|
||||
"import sagemaker\n",
|
||||
"from sagemaker import get_execution_role\n",
|
||||
"from datetime import datetime, timezone\n",
|
||||
"\n",
|
||||
"pp = pprint.PrettyPrinter(indent=4)\n",
|
||||
"\n",
|
||||
"sess = sagemaker.session.Session()\n",
|
||||
"role = sagemaker.get_execution_role()\n",
|
||||
"region = boto3.session.Session().region_name\n",
|
||||
"\n",
|
||||
"sagemaker_client = boto3.client(\"sagemaker\")\n",
|
||||
"s3 = boto3.client(\"s3\")\n",
|
||||
"iam = boto3.client(\"iam\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"### The following cell will set up your S3 bucket and execution role"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BUCKET = \"\"\n",
|
||||
"EXP_NAME = \"\" # Any valid S3 prefix."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Make sure the bucket is in the same region as this notebook.\n",
|
||||
"bucket_region = s3.head_bucket(Bucket=BUCKET)[\"ResponseMetadata\"][\"HTTPHeaders\"][\n",
|
||||
" \"x-amz-bucket-region\"\n",
|
||||
"]\n",
|
||||
"assert (\n",
|
||||
" bucket_region == region\n",
|
||||
"), \"Your S3 bucket {} and this notebook need to be in the same region.\".format(BUCKET)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Copy and modify files from the sample bucket\n",
|
||||
"\n",
|
||||
"The sample files for this demo are in a public bucket to provide you with the inputs to try this demo. In order for this demo to work, we will need to copy these files to the bucket you specified in `BUCKET` so that there are in a place where you have read/write access. Additionally, we will have to update the file paths that refer to our public demo bucket to the bucket you specified above."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!mkdir -p sample_files\n",
|
||||
"!aws s3 cp s3://aws-ml-blog/artifacts/gt-point-cloud-demos/manifests sample_files/manifests --quiet --recursive\n",
|
||||
"!aws s3 cp s3://aws-ml-blog/artifacts/gt-point-cloud-demos/label-category-config sample_files/label-category-config --quiet --recursive\n",
|
||||
"!aws s3 cp s3://aws-ml-blog/artifacts/gt-point-cloud-demos/output sample_files/output --quiet --recursive\n",
|
||||
"!aws s3 cp s3://aws-ml-blog/artifacts/gt-point-cloud-demos/sequences sample_files/sequences --quiet --recursive\n",
|
||||
"!aws s3 cp s3://aws-ml-blog/artifacts/gt-point-cloud-demos/frames/0.txt sample_files/frames/0.txt --quiet\n",
|
||||
"!aws s3 cp s3://aws-ml-blog/artifacts/gt-point-cloud-demos/frames/images/frame_0_camera_0.jpg sample_files/frames/images/frame_0_camera_0.jpg --quiet\n",
|
||||
"!aws s3 cp s3://aws-ml-blog/artifacts/gt-point-cloud-demos/frames/images/frame_1_camera_0.jpg sample_files/frames/images/frame_1_camera_0.jpg --quiet\n",
|
||||
"!aws s3 cp s3://aws-ml-blog/artifacts/gt-point-cloud-demos/frames/images/frame_2_camera_0.jpg sample_files/frames/images/frame_2_camera_0.jpg --quiet\n",
|
||||
"!aws s3 cp s3://aws-ml-blog/artifacts/gt-point-cloud-demos/frames/images/frame_3_camera_0.jpg sample_files/frames/images/frame_3_camera_0.jpg --quiet\n",
|
||||
"!aws s3 cp s3://aws-ml-blog/artifacts/gt-point-cloud-demos/frames/images/frame_4_camera_0.jpg sample_files/frames/images/frame_4_camera_0.jpg --quiet\n",
|
||||
"!aws s3 cp s3://aws-ml-blog/artifacts/gt-point-cloud-demos/frames/images/frame_5_camera_0.jpg sample_files/frames/images/frame_5_camera_0.jpg --quiet\n",
|
||||
"!aws s3 cp s3://aws-ml-blog/artifacts/gt-point-cloud-demos/frames/images/frame_6_camera_0.jpg sample_files/frames/images/frame_6_camera_0.jpg --quiet\n",
|
||||
"!aws s3 cp s3://aws-ml-blog/artifacts/gt-point-cloud-demos/frames/images/frame_7_camera_0.jpg sample_files/frames/images/frame_7_camera_0.jpg --quiet\n",
|
||||
"!aws s3 cp s3://aws-ml-blog/artifacts/gt-point-cloud-demos/frames/images/frame_8_camera_0.jpg sample_files/frames/images/frame_8_camera_0.jpg --quiet\n",
|
||||
"!aws s3 cp s3://aws-ml-blog/artifacts/gt-point-cloud-demos/frames/images/frame_9_camera_0.jpg sample_files/frames/images/frame_9_camera_0.jpg --quiet\n",
|
||||
"!find ./sample_files/ -type f -name \"*.json\" -print0 | xargs -0 sed -i -e \"s/aws-ml-blog/$1/g\"\n",
|
||||
"!aws s3 cp ./sample_files/ s3://$1/artifacts/gt-point-cloud-demos/ --quiet --recursive"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## The Dataset and Input Manifest Files"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The dataset and resources used in this notebook are located in the following Amazon S3 bucket. s3://aws-ml-blog/artifacts/gt-point-cloud-demos.\n",
|
||||
"\n",
|
||||
"This bucket contains our input manifest files and our input data: 3D point cloud frame files and images for sensor fusion. First, we'll inspect the input data and input manifest files. In the next section, you will select your labeling job type, and identify resources required to create a labeling job. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!aws s3 ls s3://$BUCKET/artifacts/gt-point-cloud-demos/"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Depending on the task type that you choose in the following section, you will use a **manifest with single-frame per task**, or *frame input manifest* or **manifest with multi-frame sequence per task**, or a *sequence input manifest*. To learn more about the types of 3D Point Cloud input manfiest files, see [3D Point Cloud Input Data](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-input-data.html).\n",
|
||||
"\n",
|
||||
"### Input Manifest File With Single Frame Per Task\n",
|
||||
"\n",
|
||||
"When you use a frame input manifest for 3D point cloud object detection and semantic segmentation task types, each line in the input manifest will identify the location of a single point cloud file in Amazon S3. When a task is created, workers will be asked to classify or add a segmentation mask to objects in that frame (depending on the task type). \n",
|
||||
"\n",
|
||||
"Let's look at the single-frame input manfiest. You'll see that this manifest file contains the location of a point cloud file in `source-ref`, as well as the pose of the vehicle used to collect the data (ego-vehicle), image pose information and other image data used for sensor fusion. See [Create a Point Cloud Frame Input Manifest File](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-single-frame-input-data.html) to learn more about these parameters. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!wget https://s3.amazonaws.com/aws-ml-blog/artifacts/gt-point-cloud-demos/manifests/SingleFrame-manifest.json -O SingleFrame-manifest.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"print(\"\\nThe single-frame input manifest file:\")\n",
|
||||
"with open(\"SingleFrame-manifest.json\", \"r\") as j:\n",
|
||||
" json_data = json.load(j)\n",
|
||||
" print(\"\\n\", json.dumps(json_data, indent=4, sort_keys=True))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The point cloud data in the file, `0.txt`, identified in the manfiest above is in ASCII format. Each line in the point cloud file contains information about a single point. The first three values are x, y, and z location coordinates, and the last element is the pixel intensity. To learn more about this raw data format, see [ASCII Format](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-raw-data-types.html#sms-point-cloud-raw-data-ascii-format)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!wget https://s3.amazonaws.com/aws-ml-blog/artifacts/gt-point-cloud-demos/frames/0.txt -O 0.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"frame = open(\"0.txt\")\n",
|
||||
"print(\"\\nA single line from the point cloud file with x, y, z and pixel intensity values: \\n\")\n",
|
||||
"frame.readline()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Input Manifest File With Multi-Frame Sequence Per Task\n",
|
||||
"\n",
|
||||
"When you chooose a sequence input manifest file, each line in the input manifest will point to a *sequence file* in Amazon S3. A sequence specifies a temporal series of point cloud frames. When a task is created using a sequence file, all point cloud frames in the sequence are sent to a worker to label. Workers can navigate back and forth between and annotate (with 3D cuboids) the sequence of frames to track the trajectory of objects across frames. \n",
|
||||
"\n",
|
||||
"Let's look at the sequence input manifest file. You'll see that this input manifest contains the location of a single sequence file. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!wget https://s3.amazonaws.com/aws-ml-blog/artifacts/gt-point-cloud-demos/manifests/OT-manifest-10-frame.json -O OT-manifest-10-frame.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"print(\"\\nThe multi-frame input manifest file:\")\n",
|
||||
"with open(\"OT-manifest-10-frame.json\", \"r\") as j:\n",
|
||||
" json_data = json.load(j)\n",
|
||||
" print(\"\\n\", json.dumps(json_data, indent=4, sort_keys=True))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's look at the sequence file, seq1.json. You will see that this single sequence file contains the location of 10 frames, as well as pose information on the vehicle (ego-vehicle) and camera. See [Create a Point Cloud Frame Sequence Input Manifest](http://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-multi-frame-input-data.html) to learn more about these parameters."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!wget https://s3.amazonaws.com/aws-ml-blog/artifacts/gt-point-cloud-demos/sequences/seq2.json -O seq1.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"with open(\"seq1.json\", \"r\") as j:\n",
|
||||
" json_data = json.load(j)\n",
|
||||
" print(\"\\nA single sequence file: \\n\\n\", json.dumps(json_data, indent=4, sort_keys=True))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Select a 3D Point Cloud Labeling Job Task Type"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In the following cell, select a [3D Point Cloud Task Type](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-task-types.html) by sepcifying a value for `task_type`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"## Choose from following:\n",
|
||||
"## 3DPointCloudObjectDetection\n",
|
||||
"## 3DPointCloudObjectTracking\n",
|
||||
"## 3DPointCloudSemanticSegmentation\n",
|
||||
"## Adjustment3DPointCloudObjectDetection\n",
|
||||
"## Adjustment3DPointCloudObjectTracking\n",
|
||||
"## Adjustment3DPointCloudSemanticSegmentation\n",
|
||||
"\n",
|
||||
"task_type = \"3DPointCloudObjectTracking\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Identify Resources for Labeling Job"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The following will be used to select the HumanTaskUiArn. When you create a 3D point cloud labeling job, Ground Truth provides the worker task UI. The following cell identifies the correct HumanTaskUiArn to use a worker UI that is specific to your task type. You can see examples of the worker UIs on the [3D Point Cloud Task Type](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-task-types.html) pages. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"## Set up human_task_ui_arn map\n",
|
||||
"\n",
|
||||
"human_task_ui_arn_map = {\n",
|
||||
" \"3DPointCloudObjectDetection\": f\"arn:aws:sagemaker:{region}:394669845002:human-task-ui/PointCloudObjectDetection\",\n",
|
||||
" \"3DPointCloudObjectTracking\": f\"arn:aws:sagemaker:{region}:394669845002:human-task-ui/PointCloudObjectTracking\",\n",
|
||||
" \"3DPointCloudSemanticSegmentation\": f\"arn:aws:sagemaker:{region}:394669845002:human-task-ui/PointCloudSemanticSegmentation\",\n",
|
||||
" \"Adjustment3DPointCloudObjectDetection\": f\"arn:aws:sagemaker:{region}:394669845002:human-task-ui/PointCloudObjectDetection\",\n",
|
||||
" \"Adjustment3DPointCloudObjectTracking\": f\"arn:aws:sagemaker:{region}:394669845002:human-task-ui/PointCloudObjectTracking\",\n",
|
||||
" \"Adjustment3DPointCloudSemanticSegmentation\": f\"arn:aws:sagemaker:{region}:394669845002:human-task-ui/PointCloudSemanticSegmentation\",\n",
|
||||
"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Input Data and Input Manifest File\n",
|
||||
"\n",
|
||||
"The following task types (and associated adjustment labeling jobs) require the following types of input manifest files. \n",
|
||||
"\n",
|
||||
"* 3D point cloud object detection – frame input manifest\n",
|
||||
"* 3D point cloud semantic segmentation – frame input manifest\n",
|
||||
"* 3D point cloud object tracking – sequence frame input manifest \n",
|
||||
"\n",
|
||||
"Run the following to identify an input manifest file for the task type you selected in the previous section. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"## Set up manifest_s3_uri_map, to be used to set up Input ManifestS3Uri\n",
|
||||
"\n",
|
||||
"manifest_s3_uri_map = {\n",
|
||||
" \"3DPointCloudObjectDetection\": f\"s3://{BUCKET}/artifacts/gt-point-cloud-demos/manifests/SingleFrame-manifest.json\",\n",
|
||||
" \"3DPointCloudObjectTracking\": f\"s3://{BUCKET}/artifacts/gt-point-cloud-demos/manifests/OT-manifest-10-frame.json\",\n",
|
||||
" \"3DPointCloudSemanticSegmentation\": f\"s3://{BUCKET}/artifacts/gt-point-cloud-demos/manifests/SS-manifest.json\",\n",
|
||||
" \"Adjustment3DPointCloudObjectDetection\": f\"s3://{BUCKET}/artifacts/gt-point-cloud-demos/manifests/OD-adjustment-manifest.json\",\n",
|
||||
" \"Adjustment3DPointCloudObjectTracking\": f\"s3://{BUCKET}/artifacts/gt-point-cloud-demos/manifests/OT-adjustment-manifest.json\",\n",
|
||||
" \"Adjustment3DPointCloudSemanticSegmentation\": f\"s3://{BUCKET}/artifacts/gt-point-cloud-demos/manifests/SS-audit-manifest-5-17.json\",\n",
|
||||
"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Label Category Configuration File\n",
|
||||
"\n",
|
||||
"Your label category configuration file is used to specify labels, or classes, for your labeling job.\n",
|
||||
"\n",
|
||||
"When you use the object detection or object tracking task types, you can also include [label category attributes](http://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-general-information.html#sms-point-cloud-worker-task-ui) in your label category configuration file. Workers can assign one or more attributes you provide to annotations to give more information about that object. For example, you may want to use the attribute *occluded* to have workers identify when an object is partially obstructed. \n",
|
||||
"\n",
|
||||
"Let's look at an example of the label category configuration file for an object detection or object tracking labeling job. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!wget https://s3.amazonaws.com/aws-ml-blog/artifacts/gt-point-cloud-demos/label-category-config/label-category.json -O label-category.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"with open(\"label-category.json\", \"r\") as j:\n",
|
||||
" json_data = json.load(j)\n",
|
||||
" print(\n",
|
||||
" \"\\nA label category configuration file: \\n\\n\",\n",
|
||||
" json.dumps(json_data, indent=4, sort_keys=True),\n",
|
||||
" )"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"To learn more about the label category configuration file, see [Create a Label Category Configuration File](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-point-cloud-label-category-config.html).\n",
|
||||
"\n",
|
||||
"Run the following cell to identify the labeling category configuration file."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"label_category_file_s3_uri_map = {\n",
|
||||
" \"3DPointCloudObjectDetection\": f\"s3://{BUCKET}/artifacts/gt-point-cloud-demos/label-category-config/label-category.json\",\n",
|
||||
" \"3DPointCloudObjectTracking\": f\"s3://{BUCKET}/artifacts/gt-point-cloud-demos/label-category-config/label-category.json\",\n",
|
||||
" \"3DPointCloudSemanticSegmentation\": f\"s3://{BUCKET}/artifacts/gt-point-cloud-demos/label-category-config/label-category.json\",\n",
|
||||
" \"Adjustment3DPointCloudObjectDetection\": f\"s3://{BUCKET}/artifacts/gt-point-cloud-demos/label-category-config/od-adjustment-label-categories-file.json\",\n",
|
||||
" \"Adjustment3DPointCloudObjectTracking\": f\"s3://{BUCKET}/artifacts/gt-point-cloud-demos/label-category-config/ot-adjustment-label-categories-file.json\",\n",
|
||||
" \"Adjustment3DPointCloudSemanticSegmentation\": f\"s3://{BUCKET}/artifacts/gt-point-cloud-demos/label-category-config/SS-audit-5-17-updated-manually-created-label-categories-file.json\",\n",
|
||||
"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# You can use this to identify your labeling job by appending these abbreviations to your lableing job name.\n",
|
||||
"name_abbreviation_map = {\n",
|
||||
" \"3DPointCloudObjectDetection\": \"OD\",\n",
|
||||
" \"3DPointCloudObjectTracking\": \"OT\",\n",
|
||||
" \"3DPointCloudSemanticSegmentation\": \"SS\",\n",
|
||||
" \"Adjustment3DPointCloudObjectDetection\": \"OD-ADJ\",\n",
|
||||
" \"Adjustment3DPointCloudObjectTracking\": \"OT-ADJ\",\n",
|
||||
" \"Adjustment3DPointCloudSemanticSegmentation\": \"SS-ADJ\",\n",
|
||||
"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Set up Human Task Configuration\n",
|
||||
"\n",
|
||||
"`HumanTaskConfig` is used to specify your work team, and configure your labeling job tasks. \n",
|
||||
"\n",
|
||||
"If you want to preview the worker task UI, create a private work team and add yourself as a worker. \n",
|
||||
"\n",
|
||||
"If you have already created a private workforce, follow the instructions in [Add or Remove Workers](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-workforce-management-private-console.html#add-remove-workers-sm) to add yourself to the work team you use to create a lableing job. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Create a Private Workforce and Add Yourself as a Worker\n",
|
||||
"\n",
|
||||
"To create and manage your private workforce, you can use the **Labeling workforces** page in the Amazon SageMaker console. When following the instructions below, you will have the option to create a private workforce by entering worker emails or importing a pre-existing workforce from an Amazon Cognito user pool. To import a workforce, see [Create a Private Workforce (Amazon Cognito Console)](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-workforce-create-private-cognito.html).\n",
|
||||
"\n",
|
||||
"To create a private workforce using worker emails:\n",
|
||||
"\n",
|
||||
"* Open the Amazon SageMaker console at https://console.aws.amazon.com/sagemaker/.\n",
|
||||
"\n",
|
||||
"* In the navigation pane, choose **Labeling workforces**.\n",
|
||||
"\n",
|
||||
"* Choose Private, then choose **Create private team**.\n",
|
||||
"\n",
|
||||
"* Choose **Invite new workers by email**.\n",
|
||||
"\n",
|
||||
"* Paste or type a list of up to 50 email addresses, separated by commas, into the email addresses box.\n",
|
||||
"\n",
|
||||
"* Enter an organization name and contact email.\n",
|
||||
"\n",
|
||||
"* Optionally choose an SNS topic to subscribe the team to so workers are notified by email when new Ground Truth labeling jobs become available. \n",
|
||||
"\n",
|
||||
"* Click the **Create private team** button.\n",
|
||||
"\n",
|
||||
"After you import your private workforce, refresh the page. On the Private workforce summary page, you'll see your work team ARN. Enter this ARN in the following cell. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"workteam_arn = \"\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Task Time Limits\n",
|
||||
"\n",
|
||||
"3D point cloud annotation jobs can take workers hours or days to complete. Workers will be able to start your labeling task, save their work as they go, and complete the task in multiple sittings. Ground Truth will also automatically save workers' annotations periodically as they work. \n",
|
||||
"\n",
|
||||
"When you configure your task, you can set the total amount of time that workers can work on each task when you create a labeling job using `TaskTimeLimitInSeconds`. The maximum time you can set for workers to work on tasks is 7 days. The default value is 3 days. \n",
|
||||
"\n",
|
||||
"If you set `TaskTimeLimitInSeconds` to be greater than 8 hours, you must set `MaxSessionDuration` for your IAM execution to at least 8 hours. To see how to update this value for your IAM role, see [Modifying a Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_manage_modify.html) in the IAM User Guide, choose your preferred method to modify the role, and then follow the steps in [Modifying a Role Maximum Session Duration](https://docs.aws.amazon.com/IAM/latest/UserGuide/roles-managingrole-editing-console.html#roles-modify_max-session-duration)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# See your execution role ARN. The role name is located at the end of the ARN.\n",
|
||||
"print(role)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"ac_arn_map = {\n",
|
||||
" \"us-west-2\": \"081040173940\",\n",
|
||||
" \"us-east-1\": \"432418664414\",\n",
|
||||
" \"us-east-2\": \"266458841044\",\n",
|
||||
" \"eu-west-1\": \"568282634449\",\n",
|
||||
" \"ap-northeast-1\": \"477331159723\",\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"prehuman_arn = \"arn:aws:lambda:{}:{}:function:PRE-{}\".format(region, ac_arn_map[region], task_type)\n",
|
||||
"acs_arn = \"arn:aws:lambda:{}:{}:function:ACS-{}\".format(region, ac_arn_map[region], task_type)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"## Set up Human Task Config\n",
|
||||
"\n",
|
||||
"## Modify the following\n",
|
||||
"task_description = \"add a task description here\"\n",
|
||||
"# example keywords\n",
|
||||
"task_keywords = [\"lidar\", \"pointcloud\"]\n",
|
||||
"# add a task title\n",
|
||||
"task_title = \"Add a Task Title Here - This is Displayed to Workers\"\n",
|
||||
"# add a job name to identify your labeling job\n",
|
||||
"job_name = \"add-job-name\"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"human_task_config = {\n",
|
||||
" \"AnnotationConsolidationConfig\": {\n",
|
||||
" \"AnnotationConsolidationLambdaArn\": acs_arn,\n",
|
||||
" },\n",
|
||||
" \"WorkteamArn\": workteam_arn,\n",
|
||||
" \"PreHumanTaskLambdaArn\": prehuman_arn,\n",
|
||||
" \"MaxConcurrentTaskCount\": 200, # 200 data objects (frames for OD and SS or sequences for OT) will be sent at a time to the workteam.\n",
|
||||
" \"NumberOfHumanWorkersPerDataObject\": 1, # One worker will work on each task\n",
|
||||
" \"TaskAvailabilityLifetimeInSeconds\": 18000, # Your workteam has 5 hours to complete all pending tasks.\n",
|
||||
" \"TaskDescription\": task_description,\n",
|
||||
" \"TaskKeywords\": task_keywords,\n",
|
||||
" \"TaskTimeLimitInSeconds\": 3600, # Each seq/frame must be labeled within 1 hour.\n",
|
||||
" \"TaskTitle\": task_title,\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"human_task_config[\"UiConfig\"] = {\"HumanTaskUiArn\": \"{}\".format(human_task_ui_arn_map[task_type])}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"print(json.dumps(human_task_config, indent=4, sort_keys=True))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Set up Create Labeling Request"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The following formats your labeling job request. For 3D point cloud object tracking and semantic segmentation task types, the `LabelAttributeName` must end in `-ref`. For other task types, the label attribute name may not end in `-ref`. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"## Set up Create Labeling Request\n",
|
||||
"\n",
|
||||
"labelAttributeName = job_name + \"-ref\"\n",
|
||||
"\n",
|
||||
"if (\n",
|
||||
" task_type == \"3DPointCloudObjectDetection\"\n",
|
||||
" or task_type == \"Adjustment3DPointCloudObjectDetection\"\n",
|
||||
"):\n",
|
||||
" labelAttributeName = job_name\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"ground_truth_request = {\n",
|
||||
" \"InputConfig\": {\n",
|
||||
" \"DataSource\": {\n",
|
||||
" \"S3DataSource\": {\n",
|
||||
" \"ManifestS3Uri\": \"{}\".format(manifest_s3_uri_map[task_type]),\n",
|
||||
" }\n",
|
||||
" },\n",
|
||||
" \"DataAttributes\": {\n",
|
||||
" \"ContentClassifiers\": [\"FreeOfPersonallyIdentifiableInformation\", \"FreeOfAdultContent\"]\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
" \"OutputConfig\": {\n",
|
||||
" \"S3OutputPath\": f\"s3://{BUCKET}/{EXP_NAME}/output/\",\n",
|
||||
" },\n",
|
||||
" \"HumanTaskConfig\": human_task_config,\n",
|
||||
" \"LabelingJobName\": job_name,\n",
|
||||
" \"RoleArn\": role,\n",
|
||||
" \"LabelAttributeName\": labelAttributeName,\n",
|
||||
" \"LabelCategoryConfigS3Uri\": label_category_file_s3_uri_map[task_type],\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"print(json.dumps(ground_truth_request, indent=4, sort_keys=True))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Call CreateLabelingJob"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"sagemaker_client.create_labeling_job(**ground_truth_request)\n",
|
||||
"print(f\"Labeling Job Name: {job_name}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Check Status of Labeling Job"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"## call describeLabelingJob\n",
|
||||
"describeLabelingJob = sagemaker_client.describe_labeling_job(LabelingJobName=job_name)\n",
|
||||
"print(describeLabelingJob)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Start Working on tasks\n",
|
||||
"\n",
|
||||
"When you add yourself to a private work team, you recieve an email invitation to access the worker portal. Use this invitation to sign in to the portal and view your 3D point cloud annotation tasks. Tasks may take up to 10 minutes to show up the worker portal. \n",
|
||||
"\n",
|
||||
"Once you are done working on the tasks, click **Submit**. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### View Output Data\n",
|
||||
"\n",
|
||||
"Once you have completed all of the tasks, you can view your output data in the S3 location you specified in `OutputConfig`. \n",
|
||||
"\n",
|
||||
"To read more about Ground Truth output data format for your task type, see [Output Data](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-data-output.html)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Notebook CI Test Results\n",
|
||||
"\n",
|
||||
"This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"instance_type": "ml.t3.medium",
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (Data Science 3.0)",
|
||||
"language": "python",
|
||||
"name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/sagemaker-data-science-310-v1"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
# Build an image that can do training and inference in SageMaker
|
||||
|
||||
FROM tensorflow/tensorflow:latest-py3
|
||||
RUN apt-get update && \
|
||||
apt-get install -y nginx
|
||||
|
||||
RUN pip install gevent gunicorn flask sagemaker-containers pandas s3fs sklearn
|
||||
|
||||
ENV PATH="/opt/program:${PATH}"
|
||||
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
|
||||
# Set up the program in the image
|
||||
COPY news-classifier /opt/program
|
||||
WORKDIR /opt/program
|
||||
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"input": "Why Exercise Alone May Not Be the Key to Weight Loss"
|
||||
}
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
#!/bin/bash
|
||||
|
||||
payload=$1
|
||||
content=${2:-text/csv}
|
||||
|
||||
curl -d @${payload} -H "Content-Type: ${content}" http://localhost:8080/invocations
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
#!/bin/sh
|
||||
|
||||
image=$1
|
||||
|
||||
docker run -v $(pwd)/test_dir:/opt/ml -p 8080:8080 --rm ${image} serve
|
||||
+1
@@ -0,0 +1 @@
|
||||
{}
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
#!/bin/sh
|
||||
|
||||
image=$1
|
||||
|
||||
mkdir -p test_dir/model
|
||||
mkdir -p test_dir/output
|
||||
|
||||
rm test_dir/model/*
|
||||
rm test_dir/output/*
|
||||
|
||||
docker run -v $(pwd)/test_dir:/opt/ml --rm ${image} train
|
||||
+43
@@ -0,0 +1,43 @@
|
||||
worker_processes 1;
|
||||
daemon off; # Prevent forking
|
||||
|
||||
|
||||
pid /tmp/nginx.pid;
|
||||
error_log /var/log/nginx/error.log;
|
||||
|
||||
events {
|
||||
# defaults
|
||||
}
|
||||
|
||||
http {
|
||||
include /etc/nginx/mime.types;
|
||||
default_type application/octet-stream;
|
||||
access_log /var/log/nginx/access.log combined;
|
||||
|
||||
upstream gunicorn {
|
||||
server unix:/tmp/gunicorn.sock;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 8080 deferred;
|
||||
client_max_body_size 6m;
|
||||
|
||||
keepalive_timeout 0;
|
||||
|
||||
location ~ ^/(ping|invocations) {
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header Host $http_host;
|
||||
# Setting all timeouts to 10 days.
|
||||
proxy_read_timeout 36000s;
|
||||
proxy_connect_timeout 36000s;
|
||||
proxy_send_timeout 36000s;
|
||||
|
||||
proxy_redirect off;
|
||||
proxy_pass http://gunicorn;
|
||||
}
|
||||
|
||||
location / {
|
||||
return 404 "{}";
|
||||
}
|
||||
}
|
||||
}
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
|
||||
pid /tmp/nginx.pid;
|
||||
error_log /var/log/nginx/error.log;
|
||||
|
||||
events {
|
||||
# defaults
|
||||
}
|
||||
|
||||
http {
|
||||
include /etc/nginx/mime.types;
|
||||
default_type application/octet-stream;
|
||||
access_log /var/log/nginx/access.log combined;
|
||||
|
||||
upstream gunicorn {
|
||||
server unix:/tmp/gunicorn.sock fail_timeout=3600;
|
||||
}
|
||||
|
||||
server {
|
||||
listen $NGINX_PORT deferred;
|
||||
client_max_body_size 20m;
|
||||
|
||||
keepalive_timeout 36000;
|
||||
proxy_read_timeout 36000s;
|
||||
proxy_connect_timeout 36000s;
|
||||
proxy_send_timeout 36000s;
|
||||
|
||||
location ~ ^/(ping|invocations) {
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header Host $http_host;
|
||||
proxy_ignore_client_abort on;
|
||||
proxy_redirect off;
|
||||
proxy_pass http://gunicorn;
|
||||
}
|
||||
|
||||
location / {
|
||||
return 404 "{}";
|
||||
}
|
||||
}
|
||||
}
|
||||
+144
@@ -0,0 +1,144 @@
|
||||
# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License").
|
||||
# You may not use this file except in compliance with the License.
|
||||
# A copy of the License is located at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# or in the "license" file accompanying this file. This file is distributed
|
||||
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
|
||||
# express or implied. See the License for the specific language governing
|
||||
# permissions and limitations under the License.
|
||||
#
|
||||
# This is the file that implements a flask server to do inferences. It's the file that you will modify to
|
||||
# implement the scoring for your own algorithm.
|
||||
|
||||
from __future__ import print_function
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import pickle
|
||||
import signal
|
||||
import sys
|
||||
import traceback
|
||||
from io import BytesIO, StringIO
|
||||
|
||||
import boto3
|
||||
import flask
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import tensorflow as tf
|
||||
from tensorflow.python.keras.preprocessing.sequence import pad_sequences
|
||||
from tensorflow.python.keras.preprocessing.text import Tokenizer
|
||||
|
||||
MAX_LEN = 100
|
||||
prefix = "/opt/ml/"
|
||||
model_path = os.environ.get("SM_MODEL_DIR", "/opt/ml/model")
|
||||
|
||||
# A singleton for holding the model. This simply loads the model and holds it.
|
||||
# It has a predict function that does a prediction based on the model and the input data.
|
||||
|
||||
|
||||
class ScoringService(object):
|
||||
model = None # Where we keep the model when it's loaded
|
||||
|
||||
def __init__(self):
|
||||
# This bucket should be updated based on the value in Part 2: Bring Your Own Model to an Active Learning Workflow
|
||||
# notebook after the preprocessing is done.
|
||||
tokenizer_bucket = "<Update tokenizer bucket here>"
|
||||
tokenizer_key = "sagemaker-byoal/tokenizer.pickle"
|
||||
pickle_file_name = tokenizer_key.split("/")[-1]
|
||||
boto3.resource("s3").Bucket(tokenizer_bucket).download_file(tokenizer_key, pickle_file_name)
|
||||
with open(pickle_file_name, "rb") as handle:
|
||||
self.tokenizer = pickle.load(handle)
|
||||
print("Successfully initialized tokenizer.")
|
||||
|
||||
def get_model(self):
|
||||
"""Get the model object for this instance, loading it if it's not already loaded."""
|
||||
|
||||
if self.model is None:
|
||||
self.model = tf.keras.models.load_model(
|
||||
os.path.join(model_path, "keras_news_classifier_model.h5")
|
||||
)
|
||||
print("Successfully loaded model.")
|
||||
|
||||
return self.model
|
||||
|
||||
def predict(self, input):
|
||||
"""For the input, do the predictions and return them.
|
||||
|
||||
Args:
|
||||
input (a single news headline): The data on which to do the predictions."""
|
||||
model = self.get_model()
|
||||
seq = self.tokenizer.texts_to_sequences([input])
|
||||
d = pad_sequences(seq, maxlen=MAX_LEN)
|
||||
prediction = model.predict(np.array(d))
|
||||
print("prediction received {}".format(prediction))
|
||||
|
||||
probs = np.array(prediction).flatten()
|
||||
descending_sorted_index = (-probs).argsort()
|
||||
return {
|
||||
"label": ["__label__{}".format(index) for index in descending_sorted_index],
|
||||
"prob": list(probs[descending_sorted_index].astype(float)),
|
||||
}
|
||||
|
||||
|
||||
# The flask app for serving predictions
|
||||
app = flask.Flask(__name__)
|
||||
app.logger.setLevel(logging.DEBUG)
|
||||
scoring_service = ScoringService()
|
||||
|
||||
|
||||
@app.route("/ping", methods=["GET"])
|
||||
def ping():
|
||||
"""Determine if the container is working and healthy. In this sample container, we declare
|
||||
it healthy if we can load the model successfully."""
|
||||
health = scoring_service.get_model() is not None # You can insert a health check here
|
||||
|
||||
status = 200 if health else 404
|
||||
return flask.Response(response="\n", status=status, mimetype="application/json")
|
||||
|
||||
|
||||
def _load_json_instance(instance):
|
||||
source = instance.get("source")
|
||||
if source is None:
|
||||
print(
|
||||
"Instance does not have source. Unexpected input to batch transform {}".format(instance)
|
||||
)
|
||||
return None
|
||||
return source.encode("utf-8").decode("utf-8")
|
||||
|
||||
|
||||
def _dump_jsonlines_entry(prediction):
|
||||
return (json.dumps(prediction, ensure_ascii=False, sort_keys=True) + "\n").encode("utf-8")
|
||||
|
||||
|
||||
@app.route("/invocations", methods=["POST"])
|
||||
def transformation():
|
||||
"""Do an inference on a single news headline."""
|
||||
data = None
|
||||
|
||||
if flask.request.content_type == "application/jsonlines":
|
||||
payload = flask.request.data
|
||||
if len(payload) == 0:
|
||||
return flask.Response(response="", status=204)
|
||||
|
||||
print("prediction input size in bytes:{} content:{}".format(len(payload), payload))
|
||||
|
||||
fr = StringIO(payload.decode("utf-8"))
|
||||
texts = [_load_json_instance(json.loads(line)) for line in iter(lambda: fr.readline(), "")]
|
||||
|
||||
predictions = [scoring_service.predict(text[0]) for text in texts if text is not None]
|
||||
|
||||
bio = BytesIO()
|
||||
for line in predictions:
|
||||
bio.write(_dump_jsonlines_entry(line))
|
||||
return flask.Response(response=bio.getvalue(), status=200, mimetype="application/jsonlines")
|
||||
else:
|
||||
return flask.Response(
|
||||
response="This predictor only supports application/jsonlines format",
|
||||
status=415,
|
||||
mimetype="text/plain",
|
||||
)
|
||||
+74
@@ -0,0 +1,74 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# This file implements the scoring service shell. You don't necessarily need to modify it for various
|
||||
# algorithms. It starts nginx and gunicorn with the correct configurations and then simply waits until
|
||||
# gunicorn exits.
|
||||
#
|
||||
# The flask server is specified to be the app object in wsgi.py
|
||||
#
|
||||
# We set the following parameters:
|
||||
#
|
||||
# Parameter Environment Variable Default Value
|
||||
# --------- -------------------- -------------
|
||||
# number of workers MODEL_SERVER_WORKERS the number of CPU cores
|
||||
# timeout MODEL_SERVER_TIMEOUT 60 seconds
|
||||
|
||||
from __future__ import print_function
|
||||
|
||||
import multiprocessing
|
||||
import os
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
cpu_count = multiprocessing.cpu_count()
|
||||
|
||||
model_server_timeout = 36000
|
||||
model_server_workers = int(os.environ.get('MODEL_SERVER_WORKERS', cpu_count)) * 2
|
||||
|
||||
def sigterm_handler(nginx_pid, gunicorn_pid):
|
||||
print("SIGTERM received.")
|
||||
try:
|
||||
os.kill(nginx_pid, signal.SIGQUIT)
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
os.kill(gunicorn_pid, signal.SIGTERM)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
sys.exit(0)
|
||||
|
||||
def start_server():
|
||||
print('Starting the inference server with {} workers. timeout = {}'.format(model_server_workers, model_server_timeout))
|
||||
|
||||
# link the log streams to stdout/err so they will be logged to the container logs
|
||||
subprocess.check_call(['ln', '-sf', '/dev/stdout', '/var/log/nginx/access.log'])
|
||||
subprocess.check_call(['ln', '-sf', '/dev/stderr', '/var/log/nginx/error.log'])
|
||||
|
||||
nginx = subprocess.Popen(['nginx', '-c', '/opt/program/nginx.conf'])
|
||||
gunicorn = subprocess.Popen(['gunicorn',
|
||||
'--timeout', str(model_server_timeout),
|
||||
'-k', 'gthread',
|
||||
'--threads', str(model_server_workers),
|
||||
'-b', 'unix:/tmp/gunicorn.sock',
|
||||
'-w', str(model_server_workers),
|
||||
'--log-level', 'debug',
|
||||
'wsgi:app'])
|
||||
|
||||
signal.signal(signal.SIGTERM, lambda a, b: sigterm_handler(nginx.pid, gunicorn.pid))
|
||||
|
||||
# If either subprocess exits, so do we.
|
||||
pids = set([nginx.pid, gunicorn.pid])
|
||||
while True:
|
||||
pid, _ = os.wait()
|
||||
if pid in pids:
|
||||
break
|
||||
|
||||
sigterm_handler(nginx.pid, gunicorn.pid)
|
||||
print('Inference server exiting')
|
||||
|
||||
# The main routine just invokes the start function.
|
||||
|
||||
if __name__ == '__main__':
|
||||
start_server()
|
||||
+151
@@ -0,0 +1,151 @@
|
||||
#!/usr/bin/env python
|
||||
# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License").
|
||||
# You may not use this file except in compliance with the License.
|
||||
# A copy of the License is located at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# or in the "license" file accompanying this file. This file is distributed
|
||||
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
|
||||
# express or implied. See the License for the specific language governing
|
||||
# permissions and limitations under the License.
|
||||
#
|
||||
# A sample training component that trains a keras text classification
|
||||
# model.
|
||||
|
||||
|
||||
from __future__ import print_function
|
||||
|
||||
import json
|
||||
import os
|
||||
import pickle
|
||||
import re
|
||||
import sys
|
||||
import traceback
|
||||
|
||||
import boto3
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import tensorflow as tf
|
||||
from sklearn.feature_extraction import stop_words
|
||||
from tensorflow.keras.losses import CategoricalCrossentropy
|
||||
from tensorflow.python.estimator.export.export import \
|
||||
build_raw_serving_input_receiver_fn
|
||||
from tensorflow.python.estimator.export.export_output import PredictOutput
|
||||
from tensorflow.python.keras.preprocessing.sequence import pad_sequences
|
||||
from tensorflow.python.keras.preprocessing.text import Tokenizer
|
||||
|
||||
# These are the paths to where SageMaker mounts interesting things in your container.
|
||||
|
||||
max_features=5000 #we set maximum number of words to 5000
|
||||
maxlen=100 #and maximum sequence length to 100
|
||||
embedding_dim = 50
|
||||
stop_words=stop_words.ENGLISH_STOP_WORDS
|
||||
|
||||
model_path = os.environ.get('SM_MODEL_DIR', '/opt/ml/model')
|
||||
input_path = os.environ.get('SM_CHANNEL_TRAIN', '/opt/ml/input/data')
|
||||
output_path = os.environ.get('SM_OUTPUT_DIR', '/opt/ml/output')
|
||||
|
||||
# This bucket should be updated based on the value in Part 2: Bring Your Own Model to an Active Learning Workflow
|
||||
# notebook after the preprocessing is done.
|
||||
tokenizer_bucket = '<Update tokenizer bucket here>'
|
||||
tokenizer_key = 'sagemaker-byoal/tokenizer.pickle'
|
||||
|
||||
# There is a minor path difference between the location of the input from notebook compared to the step function. This function looks for a file in both the paths.
|
||||
# Note - the hyperparameters and the validation file are ignored to keep this example simple.
|
||||
def get_training_file():
|
||||
train_file_paths = ['/opt/ml/input/data/train-manifest', '/opt/ml/input/data/training/train-manifest']
|
||||
for file in train_file_paths:
|
||||
if os.path.isfile(file):
|
||||
return file
|
||||
raise Exception("train-manifest not found in expected locations {}".format(",".join(train_file_paths)))
|
||||
|
||||
def get_keras_input(inp_file):
|
||||
tf_train=pd.DataFrame(columns=['TITLE','CATEGORY'])
|
||||
for line in inp_file:
|
||||
train_data=json.loads(line)
|
||||
single_train_input = {'CATEGORY': train_data['category'], 'TITLE':train_data['source']}
|
||||
tf_train=tf_train.append(single_train_input, ignore_index=True)
|
||||
tf_train["TITLE"]=tf_train["TITLE"].str.lower().replace('[^\w\s]','')
|
||||
tf_train["TITLE"]= tf_train["TITLE"].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop_words)]))
|
||||
tf_train.dropna(inplace=True)
|
||||
|
||||
cat=tf_train['CATEGORY'].astype("category").cat.categories
|
||||
tf_train['CATEGORY']=tf_train['CATEGORY'].astype("category").cat.codes
|
||||
y_train_int=tf_train['CATEGORY'].values
|
||||
# Convert labels to categorical one-hot encoding
|
||||
y_train = tf.keras.utils.to_categorical(y_train_int, num_classes=4)
|
||||
|
||||
pickle_file_name = tokenizer_key.split('/')[-1]
|
||||
boto3.resource('s3').Bucket(tokenizer_bucket).download_file(tokenizer_key, pickle_file_name)
|
||||
with open(pickle_file_name, 'rb') as handle:
|
||||
tok= pickle.load(handle)
|
||||
|
||||
tf_train=tok.texts_to_sequences(list(tf_train['TITLE'])) #this is how we create sequences
|
||||
X_train=tf.keras.preprocessing.sequence.pad_sequences(tf_train, maxlen=maxlen) #let's execute pad step
|
||||
vocab_size = len(tok.word_index) + 1
|
||||
|
||||
return X_train, y_train, vocab_size
|
||||
|
||||
def get_validation_data():
|
||||
validation_files = [ '/opt/ml/input/data/validation-manifest', '/opt/ml/input/data/validation/validation-manifest']
|
||||
for val_file in validation_files:
|
||||
if os.path.isfile(val_file):
|
||||
return get_keras_input(open(val_file, 'r'))[:2]
|
||||
raise Exception("validation-manifest not found in expected locations {}".format(",".join(validation_files)))
|
||||
|
||||
# The function to execute the training.
|
||||
def train():
|
||||
print('Starting the training with input_path {}'.format(input_path))
|
||||
try:
|
||||
|
||||
train_file=open(get_training_file(), 'r')
|
||||
X_train, y_train, vocab_size = get_keras_input(train_file)
|
||||
|
||||
model = tf.keras.models.Sequential([
|
||||
tf.keras.layers.Embedding(input_dim=vocab_size, #embedding input
|
||||
output_dim=embedding_dim,#embedding output
|
||||
input_length=maxlen), #maximum length of an input sequence
|
||||
|
||||
tf.keras.layers.GlobalMaxPool1D(), #Max pooling operation for temporal data
|
||||
|
||||
tf.keras.layers.Dropout(0.3), # Drop out to avoid overfitting
|
||||
|
||||
tf.keras.layers.Dense(4, activation=tf.nn.softmax) #ouput layer a Dense layer with 4 probabilities
|
||||
#we also define our final activation function which is the softmax function typical for multiclass
|
||||
#classification problems
|
||||
])
|
||||
|
||||
model.compile(optimizer=tf.keras.optimizers.Nadam(learning_rate=1e-3), \
|
||||
loss=CategoricalCrossentropy(label_smoothing=0.1), \
|
||||
metrics=['accuracy'])
|
||||
|
||||
# training loss is used instead of validation loss for stopping condition to increase
|
||||
# confidence in the predicted labels.
|
||||
early_stopping_cb = tf.keras.callbacks.EarlyStopping(monitor='loss',patience=2,restore_best_weights=True)
|
||||
checkpoint_cb = tf.keras.callbacks.ModelCheckpoint("keras_model.h5",save_best_only=True)
|
||||
history = model.fit(X_train, y_train, epochs=100, validation_data=get_validation_data(),
|
||||
callbacks=[checkpoint_cb, early_stopping_cb])
|
||||
|
||||
model_file_name = os.path.join(model_path,"keras_news_classifier_model.h5")
|
||||
model.save(model_file_name)
|
||||
|
||||
except Exception as e:
|
||||
# Write out an error file. This will be returned as the failureReason in the
|
||||
# DescribeTrainingJob result.
|
||||
trc = traceback.format_exc()
|
||||
with open(os.path.join(output_path, 'failure'), 'w') as s:
|
||||
s.write('Exception during training: ' + str(e) + '\n' + trc)
|
||||
# Printing this causes the exception to be in the training job logs, as well.
|
||||
print('Exception during training: ' + str(e) + '\n' + trc, file=sys.stderr)
|
||||
# A non-zero exit code causes the training job to be marked as Failed.
|
||||
sys.exit(255)
|
||||
|
||||
if __name__ == '__main__':
|
||||
train()
|
||||
|
||||
# A zero exit code causes the job to be marked a Succeeded.
|
||||
sys.exit(0)
|
||||
|
||||
+7
@@ -0,0 +1,7 @@
|
||||
import predictor as myapp
|
||||
|
||||
# This is just a simple wrapper for gunicorn to find your app.
|
||||
# If you want to change the algorithm file, simply change "predictor" above to the
|
||||
# new file.
|
||||
|
||||
app = myapp.app
|
||||
+1135
File diff suppressed because it is too large
Load Diff
+34
@@ -0,0 +1,34 @@
|
||||
import logging
|
||||
|
||||
from s3_helper import S3Ref, copy_with_query, create_ref_at_parent_key
|
||||
|
||||
logger = logging.getLogger()
|
||||
logger.setLevel(logging.INFO)
|
||||
|
||||
|
||||
def lambda_handler(event, context):
|
||||
"""
|
||||
This method selects 10% of the input manifest as validation and creates an s3 file containing the validation objects.
|
||||
"""
|
||||
label_attribute_name = event["LabelAttributeName"]
|
||||
meta_data = event["meta_data"]
|
||||
s3_input_uri = meta_data["IntermediateManifestS3Uri"]
|
||||
|
||||
input_total = int(meta_data["counts"]["input_total"])
|
||||
# 10% of the total input should be used for validation.
|
||||
validation_set_size = input_total // 10
|
||||
|
||||
source = S3Ref.from_uri(s3_input_uri)
|
||||
|
||||
validation_labeled_query = """select * from s3object[*] s where s."{}-metadata"."human-annotated" IN ('yes') LIMIT {}""".format(
|
||||
label_attribute_name, validation_set_size
|
||||
)
|
||||
dest = create_ref_at_parent_key(source, "validation_input.manifest")
|
||||
copy_with_query(source, dest, validation_labeled_query)
|
||||
logger.info(
|
||||
"Uploaded validation set of size {} to {}.".format(validation_set_size, dest.get_uri())
|
||||
)
|
||||
|
||||
meta_data["counts"]["validation"] = validation_set_size
|
||||
meta_data["ValidationS3Uri"] = dest.get_uri()
|
||||
return meta_data
|
||||
+85
@@ -0,0 +1,85 @@
|
||||
import random
|
||||
from datetime import datetime
|
||||
|
||||
AUTOANNOTATION_THRESHOLD = 0.50
|
||||
JOB_TYPE = "groundtruth/text-classification"
|
||||
|
||||
|
||||
class SimpleActiveLearning:
|
||||
def __init__(self, job_name, label_category_name, label_names, max_selections):
|
||||
self.job_name = job_name
|
||||
self.label_category_name = label_category_name
|
||||
self.label_names = label_names
|
||||
self.max_selections = max_selections
|
||||
|
||||
def compute_margin(self, probabilities, labels):
|
||||
"""
|
||||
compute the confidence and the best label given the probability distribution.
|
||||
"""
|
||||
max_probability = max(probabilities)
|
||||
max_prob_index = probabilities.index(max_probability)
|
||||
best_label = labels[max_prob_index]
|
||||
remaining_probs = [prob for i, prob in enumerate(probabilities) if i != max_prob_index]
|
||||
second_probability = max(remaining_probs, default=0.0)
|
||||
return max_probability - second_probability, best_label
|
||||
|
||||
def get_label_index(self, inference_label_output):
|
||||
"""
|
||||
inference_label_output is of the format "__label__0".
|
||||
This method gets an integer suffix from the end of the string.
|
||||
For this example, "__label__0" the function returns 0.
|
||||
"""
|
||||
return int(inference_label_output.split("_")[-1])
|
||||
|
||||
def make_metadata(self, margin, best_label):
|
||||
"""
|
||||
make required metadata to match the output label.
|
||||
"""
|
||||
return {
|
||||
"confidence": float(f"{margin: 1.2f}"),
|
||||
"job-name": self.job_name,
|
||||
"class-name": self.label_names[self.get_label_index(best_label)],
|
||||
"human-annotated": "no",
|
||||
"creation-date": datetime.utcnow().strftime("%Y-%m-%dT%H:%m:%S.%f"),
|
||||
"type": JOB_TYPE,
|
||||
}
|
||||
|
||||
def make_autoannotation(self, prediction, source, margin, best_label):
|
||||
"""
|
||||
generate the final output prediction with the label and confidence.
|
||||
"""
|
||||
return {
|
||||
"source": source["source"],
|
||||
"id": prediction["id"],
|
||||
f"{self.label_category_name}": self.get_label_index(best_label),
|
||||
f"{self.label_category_name}-metadata": self.make_metadata(margin, best_label),
|
||||
}
|
||||
|
||||
def autoannotate(self, predictions, sources):
|
||||
"""
|
||||
auto annotate all unlabeled data with confidence above AUTOANNOTATION_THRESHOLD.
|
||||
"""
|
||||
sources_by_id = {source["id"]: source for source in sources}
|
||||
autoannotations = []
|
||||
for prediction in predictions:
|
||||
probabilities = prediction["prob"]
|
||||
labels = prediction["label"]
|
||||
margin, best_label = self.compute_margin(probabilities, labels)
|
||||
if margin > AUTOANNOTATION_THRESHOLD:
|
||||
autoannotations.append(
|
||||
self.make_autoannotation(
|
||||
prediction, sources_by_id[prediction["id"]], margin, best_label
|
||||
)
|
||||
)
|
||||
|
||||
return autoannotations
|
||||
|
||||
def select_for_labeling(self, predictions, autoannotations):
|
||||
"""
|
||||
Select the next set of records to be labeled by humans.
|
||||
"""
|
||||
initial_ids = {prediction["id"] for prediction in predictions}
|
||||
autoannotation_ids = {autoannotation["id"] for autoannotation in autoannotations}
|
||||
remaining_ids = initial_ids - autoannotation_ids
|
||||
selections = random.sample(remaining_ids, min(self.max_selections, len(remaining_ids)))
|
||||
return selections
|
||||
+159
@@ -0,0 +1,159 @@
|
||||
import json
|
||||
import logging
|
||||
from io import StringIO
|
||||
|
||||
from ActiveLearning.helper import SimpleActiveLearning
|
||||
from s3_helper import S3Ref, create_ref_at_parent_key, download, download_with_query, upload
|
||||
from string_helper import generate_job_id_and_s3_path
|
||||
|
||||
logger = logging.getLogger()
|
||||
logger.setLevel(logging.INFO)
|
||||
|
||||
|
||||
def get_label_names_from_s3(labels_s3_uri):
|
||||
"""
|
||||
fetch the list of labels from a label s3 bucket.
|
||||
"""
|
||||
labels_source = S3Ref.from_uri(labels_s3_uri)
|
||||
labeled_query = """SELECT label FROM S3Object[*].labels[*].label"""
|
||||
result_inp = download_with_query(labels_source, labeled_query)
|
||||
label_names = []
|
||||
for line in result_inp:
|
||||
label_data = json.loads(line)
|
||||
label_names.append(label_data["label"])
|
||||
return label_names
|
||||
|
||||
|
||||
def get_sources(inference_input):
|
||||
"""
|
||||
Load inference input as a python list
|
||||
"""
|
||||
sources = []
|
||||
for line in inference_input:
|
||||
data = json.loads(line)
|
||||
sources.append(data)
|
||||
inference_input.seek(0)
|
||||
return sources
|
||||
|
||||
|
||||
def get_predictions(inference_output):
|
||||
"""
|
||||
Load inference output as a python list
|
||||
"""
|
||||
predictions = []
|
||||
for line in inference_output:
|
||||
data = json.loads(line)
|
||||
prediction = {}
|
||||
for key, value in data.items():
|
||||
if key != "SageMakerOutput":
|
||||
prediction[key] = value
|
||||
else:
|
||||
if not isinstance(value, dict):
|
||||
print("Error: Expected dictionary inside SageMakerOutput.")
|
||||
prediction.update(value)
|
||||
predictions.append(prediction)
|
||||
return predictions
|
||||
|
||||
|
||||
def collect_inference_inputs(s3_input_uri):
|
||||
"""
|
||||
collect information related to input to inference.
|
||||
"""
|
||||
inference_input_s3_ref = S3Ref.from_uri(s3_input_uri)
|
||||
inference_input = download(inference_input_s3_ref)
|
||||
sources = get_sources(inference_input)
|
||||
logger.info("Collected {} inference inputs.".format(len(sources)))
|
||||
return inference_input_s3_ref, inference_input, sources
|
||||
|
||||
|
||||
def collect_inference_outputs(inference_output_uri):
|
||||
"""
|
||||
collect information related to output of inference.
|
||||
"""
|
||||
sagemaker_output_file = "unlabeled.manifest.out"
|
||||
prediction_output_uri = inference_output_uri + sagemaker_output_file
|
||||
prediction_output_s3 = S3Ref.from_uri(prediction_output_uri)
|
||||
prediction_output = download(prediction_output_s3)
|
||||
predictions = get_predictions(prediction_output)
|
||||
logger.info("Collected {} inference outputs.".format(len(predictions)))
|
||||
return predictions
|
||||
|
||||
|
||||
def write_auto_annotations(simple_al, sources, predictions, inference_input_s3_ref):
|
||||
"""
|
||||
write auto annotations to s3
|
||||
"""
|
||||
logger.info("Generating auto annotations where confidence is high.")
|
||||
auto_annotation_stream = StringIO()
|
||||
auto_annotations = simple_al.autoannotate(predictions, sources)
|
||||
for auto_annotation in auto_annotations:
|
||||
auto_annotation_stream.write(json.dumps(auto_annotation) + "\n")
|
||||
|
||||
# Auto annotation.
|
||||
auto_dest = create_ref_at_parent_key(inference_input_s3_ref, "autoannotated.manifest")
|
||||
upload(auto_annotation_stream, auto_dest)
|
||||
logger.info("Uploaded autoannotations to {}.".format(auto_dest.get_uri()))
|
||||
return auto_dest.get_uri(), auto_annotations
|
||||
|
||||
|
||||
def write_selector_file(
|
||||
simple_al, sources, predictions, inference_input_s3_ref, inference_input, auto_annotations
|
||||
):
|
||||
"""
|
||||
write selector file to s3. This file is used to decide which records should be labeled by humans next.
|
||||
"""
|
||||
logger.info("Selecting input for next manual annotation")
|
||||
selection_data = StringIO()
|
||||
selections = simple_al.select_for_labeling(predictions, auto_annotations)
|
||||
selections_set = set(selections)
|
||||
for line in inference_input:
|
||||
data = json.loads(line)
|
||||
if data["id"] in selections_set:
|
||||
selection_data.write(json.dumps(data) + "\n")
|
||||
inference_input.seek(0)
|
||||
selection_dest = create_ref_at_parent_key(inference_input_s3_ref, "selection.manifest")
|
||||
upload(selection_data, selection_dest)
|
||||
logger.info("Uploaded selections to {}.".format(selection_dest.get_uri()))
|
||||
return selection_dest.get_uri(), selections
|
||||
|
||||
|
||||
def lambda_handler(event, context):
|
||||
"""
|
||||
This function generates auto annotatations and performs active learning.
|
||||
- auto annotations generates machine labels for confident examples.
|
||||
- active learning selects for examples to be labeled by humans next.
|
||||
"""
|
||||
labels_s3_uri = event["LabelCategoryConfigS3Uri"]
|
||||
job_name_prefix = event["LabelingJobNamePrefix"]
|
||||
job_name = "labeling-job/{}".format(job_name_prefix)
|
||||
label_attribute_name = event["LabelAttributeName"]
|
||||
meta_data = event["meta_data"]
|
||||
intermediate_folder_uri = meta_data["IntermediateFolderUri"]
|
||||
input_total = int(meta_data["counts"]["input_total"])
|
||||
# Select maximum of 10% of the input total for next round of manual labeling.
|
||||
max_selections = input_total // 10
|
||||
# Handle corner case where integer division can lead us to 0 selections.
|
||||
if max_selections == 0:
|
||||
max_selections = input_total
|
||||
|
||||
inference_input_s3_ref, inference_input, sources = collect_inference_inputs(
|
||||
meta_data["UnlabeledS3Uri"]
|
||||
)
|
||||
predictions = collect_inference_outputs(meta_data["transform_config"]["S3OutputPath"])
|
||||
label_names = get_label_names_from_s3(labels_s3_uri)
|
||||
logger.info("Collected {} label names.".format(len(label_names)))
|
||||
|
||||
simple_al = SimpleActiveLearning(job_name, label_attribute_name, label_names, max_selections)
|
||||
meta_data["autoannotations"], auto_annotations = write_auto_annotations(
|
||||
simple_al, sources, predictions, inference_input_s3_ref
|
||||
)
|
||||
meta_data["selections_s3_uri"], selections = write_selector_file(
|
||||
simple_al, sources, predictions, inference_input_s3_ref, inference_input, auto_annotations
|
||||
)
|
||||
(
|
||||
meta_data["selected_job_name"],
|
||||
meta_data["selected_job_output_uri"],
|
||||
) = generate_job_id_and_s3_path(job_name_prefix, intermediate_folder_uri)
|
||||
meta_data["counts"]["autoannotated"] = len(auto_annotations)
|
||||
meta_data["counts"]["selected"] = len(selections)
|
||||
return meta_data
|
||||
+61
@@ -0,0 +1,61 @@
|
||||
import json
|
||||
import logging
|
||||
from io import StringIO
|
||||
|
||||
from s3_helper import S3Ref, copy_with_query_and_transform, create_ref_at_parent_key
|
||||
|
||||
logger = logging.getLogger()
|
||||
logger.setLevel(logging.INFO)
|
||||
|
||||
|
||||
def augment_inference_input(inference_raw):
|
||||
"""
|
||||
The inference manifest needs to be augmented with a value 'k' so that blazing text
|
||||
produces all probabilities instead of just the top match.
|
||||
"""
|
||||
augmented_inference = StringIO()
|
||||
for line in inference_raw:
|
||||
infer_dict = json.loads(line)
|
||||
# Note: This number should ideally be equal to the number of classes.
|
||||
# But using a big number, produces the same result.
|
||||
infer_dict["k"] = 1000000
|
||||
augmented_inference.write(json.dumps(infer_dict) + "\n")
|
||||
logger.info("Augmented inference data by adding 'k' to each line.")
|
||||
return augmented_inference
|
||||
|
||||
|
||||
def create_tranform_config(training_config):
|
||||
"""
|
||||
Transform config specifies input parameters for the transform job.
|
||||
"""
|
||||
return {
|
||||
# We reuse the training job name for the model name and corresponding
|
||||
# transform job name.
|
||||
"TransformJobName": training_config["TrainingJobName"],
|
||||
"ModelName": training_config["TrainingJobName"],
|
||||
"S3OutputPath": training_config["S3OutputPath"],
|
||||
}
|
||||
|
||||
|
||||
def lambda_handler(event, context):
|
||||
"""
|
||||
This function generates auto annotations and performs active learning.
|
||||
"""
|
||||
label_attribute_name = event["LabelAttributeName"]
|
||||
meta_data = event["meta_data"]
|
||||
s3_input_uri = meta_data["IntermediateManifestS3Uri"]
|
||||
|
||||
transform_config = create_tranform_config(meta_data["training_config"])
|
||||
|
||||
source = S3Ref.from_uri(s3_input_uri)
|
||||
dest = S3Ref.from_uri(transform_config["S3OutputPath"] + "unlabeled.manifest")
|
||||
|
||||
logger.info("Creating inference output from unlabeled subset of input {}.".format(s3_input_uri))
|
||||
SQL_UNLABELED = """select * from s3object[*] s where s."{}" is missing """
|
||||
unlabeled_query = SQL_UNLABELED.format(label_attribute_name)
|
||||
copy_with_query_and_transform(source, dest, unlabeled_query, augment_inference_input)
|
||||
meta_data["UnlabeledS3Uri"] = dest.get_uri()
|
||||
logger.info("Uploaded unlabeled manifest for inference to {}.".format(dest.get_uri()))
|
||||
|
||||
meta_data["transform_config"] = transform_config
|
||||
return meta_data
|
||||
+143
@@ -0,0 +1,143 @@
|
||||
import json
|
||||
import logging
|
||||
from functools import partial
|
||||
from io import StringIO
|
||||
|
||||
from s3_helper import (
|
||||
S3Ref,
|
||||
copy_with_query_and_transform,
|
||||
create_ref_at_parent_key,
|
||||
download_with_query,
|
||||
)
|
||||
from string_helper import generate_job_id_and_s3_path
|
||||
|
||||
logger = logging.getLogger()
|
||||
logger.setLevel(logging.INFO)
|
||||
|
||||
|
||||
def remove_by_ids(s3_blacklist_uri, label_attribute_name, manifest_file):
|
||||
"""
|
||||
helper method to remove selected id in the given input file.
|
||||
This is used to create a training set which has no elements from the given validation set.
|
||||
"""
|
||||
logger.info("Remove validation set ids from training data.")
|
||||
blacklist = S3Ref.from_uri(s3_blacklist_uri)
|
||||
validation_id_query = """select s."id" from s3object[*] s where s."{}-metadata"."human-annotated" IN ('yes')""".format(
|
||||
label_attribute_name
|
||||
)
|
||||
validation_id_file = download_with_query(blacklist, validation_id_query)
|
||||
validation_ids = set()
|
||||
|
||||
for line in validation_id_file:
|
||||
data = json.loads(line)
|
||||
validation_ids.add(data["id"])
|
||||
|
||||
training_only_file = StringIO()
|
||||
training_set_size = 0
|
||||
for line in manifest_file:
|
||||
data = json.loads(line)
|
||||
if data["id"] not in validation_ids:
|
||||
training_set_size += 1
|
||||
training_only_file.write(json.dumps(data) + "\n")
|
||||
logger.info(
|
||||
"Remove ids complete. training set size = {} Validation set size = {}".format(
|
||||
training_set_size, len(validation_ids)
|
||||
)
|
||||
)
|
||||
return training_only_file
|
||||
|
||||
|
||||
class TrainingJobParameters:
|
||||
def __init__(self, event, training_folder_uri):
|
||||
self.event = event
|
||||
self.training_folder_uri = training_folder_uri
|
||||
|
||||
@property
|
||||
def attribute_names(self):
|
||||
"""
|
||||
attribute names to be parsed from the manifest file during training.
|
||||
"""
|
||||
label_attribute_name = self.event["LabelAttributeName"]
|
||||
input_mode = "source"
|
||||
return [input_mode, label_attribute_name]
|
||||
|
||||
@property
|
||||
def training_input(self):
|
||||
"""
|
||||
Generates the training input in an s3 location and returns the s3 uri.
|
||||
"""
|
||||
label_attribute_name = self.event["LabelAttributeName"]
|
||||
s3_input_uri = self.event["ManifestS3Uri"]
|
||||
meta_data = self.event["meta_data"]
|
||||
|
||||
source = S3Ref.from_uri(s3_input_uri)
|
||||
dest = S3Ref.from_uri(self.training_folder_uri + "training_input.manifest")
|
||||
logger.info("Creating training input at {} from human labeled data.".format(dest.get_uri()))
|
||||
removeValidationIds = partial(
|
||||
remove_by_ids, meta_data["ValidationS3Uri"], label_attribute_name
|
||||
)
|
||||
training_labeled_query = """select * from s3object[*] s where s."{}-metadata"."human-annotated" IN ('yes')""".format(
|
||||
label_attribute_name
|
||||
)
|
||||
copy_with_query_and_transform(source, dest, training_labeled_query, removeValidationIds)
|
||||
logger.info("Uploaded training input at {}.".format(dest.get_uri()))
|
||||
return dest.get_uri()
|
||||
|
||||
@property
|
||||
def resource_config(self):
|
||||
"""
|
||||
configure the instance where training will be run.
|
||||
"""
|
||||
return {"InstanceCount": 1, "InstanceType": "ml.c5.2xlarge", "VolumeSizeInGB": 60}
|
||||
|
||||
@property
|
||||
def algorithm_specification(self):
|
||||
"""
|
||||
configure the docker container uri for the training algorithm.
|
||||
"""
|
||||
return {
|
||||
# This assumes we are running in us-east-1 (IAD).
|
||||
# Refer to this doc to tweak this model if you run it in other regions.
|
||||
# https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html
|
||||
"TrainingImage": "811284229777.dkr.ecr.us-east-1.amazonaws.com/blazingtext:latest",
|
||||
"TrainingInputMode": "Pipe",
|
||||
}
|
||||
|
||||
@property
|
||||
def hyper_parameters(self):
|
||||
"""
|
||||
configure hyper parameters used for training.
|
||||
"""
|
||||
return {
|
||||
"early_stopping": "True",
|
||||
"epochs": "20",
|
||||
"learning_rate": "0.05",
|
||||
"min_count": "5",
|
||||
"min_epochs": "1",
|
||||
"mode": "supervised",
|
||||
"patience": "5",
|
||||
"vector_dim": "20",
|
||||
"word_ngrams": "2",
|
||||
}
|
||||
|
||||
|
||||
def lambda_handler(event, context):
|
||||
"""
|
||||
This function sets up all the input parameters required for the training job.
|
||||
"""
|
||||
training_job_name_prefix = event["LabelingJobNamePrefix"]
|
||||
intermediate_folder_uri = event["meta_data"]["IntermediateFolderUri"]
|
||||
training_job_name, training_folder_uri = generate_job_id_and_s3_path(
|
||||
training_job_name_prefix, intermediate_folder_uri
|
||||
)
|
||||
training_job_parameters = TrainingJobParameters(event, training_folder_uri)
|
||||
|
||||
return {
|
||||
"TrainingJobName": training_job_name,
|
||||
"trainS3Uri": training_job_parameters.training_input,
|
||||
"ResourceConfig": training_job_parameters.resource_config,
|
||||
"AlgorithmSpecification": training_job_parameters.algorithm_specification,
|
||||
"HyperParameters": training_job_parameters.hyper_parameters,
|
||||
"S3OutputPath": training_job_parameters.training_folder_uri,
|
||||
"AttributeNames": training_job_parameters.attribute_names,
|
||||
}
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
import json
|
||||
import logging
|
||||
from io import StringIO
|
||||
|
||||
from s3_helper import S3Ref, download, upload
|
||||
|
||||
logger = logging.getLogger()
|
||||
logger.setLevel(logging.INFO)
|
||||
|
||||
|
||||
def lambda_handler(event, context):
|
||||
"""
|
||||
This function adds a sequential id to each record in the input manifest.
|
||||
"""
|
||||
s3_input_uri = event["ManifestS3Uri"]
|
||||
s3_input = S3Ref.from_uri(s3_input_uri)
|
||||
|
||||
inp_file = download(s3_input)
|
||||
logger.info("Downloaded file from {} to {}".format(s3_input_uri, inp_file))
|
||||
|
||||
out_file = StringIO()
|
||||
total = 0
|
||||
for processed_id_count, line in enumerate(inp_file):
|
||||
data = json.loads(line)
|
||||
data["id"] = processed_id_count
|
||||
out_file.write(json.dumps(data) + "\n")
|
||||
total += 1
|
||||
logger.info("Added id field to {} records".format(total))
|
||||
|
||||
# Uploading back to the same location where we downloaded the file from.
|
||||
upload(out_file, s3_input)
|
||||
logger.info("Uploaded updated file from {} to {}".format(out_file, s3_input_uri))
|
||||
return event
|
||||
+57
@@ -0,0 +1,57 @@
|
||||
import logging
|
||||
|
||||
from s3_helper import S3Ref, copy, get_content_size
|
||||
|
||||
logger = logging.getLogger()
|
||||
logger.setLevel(logging.INFO)
|
||||
|
||||
|
||||
def validate_output_path(s3_output_uri):
|
||||
"""
|
||||
This method validates the input uri to make sure it ends with a "/" representing a s3 folder.
|
||||
"""
|
||||
# The output path should end with a "/" to indicate a folder intead of a file.
|
||||
if len(s3_output_uri) == 0 or not s3_output_uri.endswith("/"):
|
||||
raise Exception("S3OutputPath should end with '/'.")
|
||||
|
||||
|
||||
def lambda_handler(event, context):
|
||||
"""
|
||||
This function does a copy of the input manifest to the a location within the specified output path
|
||||
after performing the following validations
|
||||
1. The output uri is not empty and ends with a '/'.
|
||||
This condition throws a exception.
|
||||
2. The input refers to a manifest file of size <= 80 MB.
|
||||
This condition records a warning in the log and allows the code to proceed.
|
||||
"""
|
||||
logger.debug("event {} context {}".format(event, context))
|
||||
s3_input_uri = event["ManifestS3Uri"]
|
||||
s3_output_uri = event["S3OutputPath"]
|
||||
validate_output_path(s3_output_uri)
|
||||
|
||||
source = S3Ref.from_uri(s3_input_uri)
|
||||
# Add a warning if the input file is too big.
|
||||
# These limits are due to limited runtime and memory in lambda.
|
||||
size = get_content_size(source)
|
||||
SIZE_MESSAGE = """"This tutorial was not tested for inputs greater than 80 MB (approx 200,000 objects). You are using a %d MB input manifest file."""
|
||||
if size > 80 * 1024 * 1024:
|
||||
logger.warn(SIZE_MESSAGE, size / 1024 / 1024)
|
||||
|
||||
# Final output folder of the labeling job
|
||||
output_folder = S3Ref.from_uri(s3_output_uri)
|
||||
|
||||
# Create intermediate folder within the final output folder for saving
|
||||
# partially complete results.
|
||||
intermediate_folder_uri = s3_output_uri + "intermediate/"
|
||||
intermediate_file_uri = intermediate_folder_uri + "input.manifest"
|
||||
|
||||
# Copy original input to the intermediate s3 folder.
|
||||
dest = S3Ref.from_uri(intermediate_file_uri)
|
||||
logger.info("Copying s3 file from {} to {}".format(s3_input_uri, intermediate_file_uri))
|
||||
copy(source, dest)
|
||||
logger.info("Copied s3 file from {} to {}".format(s3_input_uri, intermediate_file_uri))
|
||||
|
||||
return {
|
||||
"IntermediateFolderUri": intermediate_folder_uri,
|
||||
"IntermediateManifestS3Uri": intermediate_file_uri,
|
||||
}
|
||||
+53
@@ -0,0 +1,53 @@
|
||||
import logging
|
||||
|
||||
from s3_helper import S3Ref, copy_with_query
|
||||
from string_helper import generate_job_id_and_s3_path
|
||||
|
||||
logger = logging.getLogger()
|
||||
logger.setLevel(logging.INFO)
|
||||
|
||||
|
||||
def get_unlabeled_subset_count(input_total, human_label_done_count):
|
||||
"""
|
||||
We want to label 20% of the dataset with humans.
|
||||
10% for training set and 10% for validation set.
|
||||
"""
|
||||
unlabeled_subset_count = int(input_total / 5) - human_label_done_count
|
||||
return unlabeled_subset_count
|
||||
|
||||
|
||||
def lambda_handler(event, context):
|
||||
"""
|
||||
Creates necessary input parameters for the first human labeling job so that after the job
|
||||
is complete 20% of the entire data is labelled.
|
||||
"""
|
||||
job_name_prefix = event["LabelingJobNamePrefix"]
|
||||
input_total = event["input_total"]
|
||||
human_label_done_count = event["human_label_done_count"]
|
||||
intermediate_folder_uri = event["IntermediateFolderUri"]
|
||||
label_attribute_name = event["LabelAttributeName"]
|
||||
s3_input_uri = event["ManifestS3Uri"]
|
||||
|
||||
unlabeled_subset_count = get_unlabeled_subset_count(input_total, human_label_done_count)
|
||||
|
||||
source = S3Ref.from_uri(s3_input_uri)
|
||||
dest = S3Ref.from_uri(intermediate_folder_uri + "human_input.manifest")
|
||||
unlabeled_query = """select * from s3object[*] s where s."{}" is missing LIMIT {}""".format(
|
||||
label_attribute_name, unlabeled_subset_count
|
||||
)
|
||||
copy_with_query(source, dest, unlabeled_query)
|
||||
human_input_s3_uri = dest.get_uri()
|
||||
logging.info(
|
||||
"Copied {} unlabled objects from {} to {}".format(
|
||||
unlabeled_subset_count, s3_input_uri, human_input_s3_uri
|
||||
)
|
||||
)
|
||||
labeling_job_name, labeling_job_output_uri = generate_job_id_and_s3_path(
|
||||
job_name_prefix, intermediate_folder_uri, "labeling-job"
|
||||
)
|
||||
|
||||
return {
|
||||
"human_input_s3_uri": human_input_s3_uri,
|
||||
"labeling_job_name": labeling_job_name,
|
||||
"labeling_job_output_uri": labeling_job_output_uri,
|
||||
}
|
||||
+54
@@ -0,0 +1,54 @@
|
||||
import logging
|
||||
from functools import partial
|
||||
|
||||
from s3_helper import S3Ref, get_count_with_query
|
||||
|
||||
logger = logging.getLogger()
|
||||
logger.setLevel(logging.INFO)
|
||||
|
||||
|
||||
def lambda_handler(event, context):
|
||||
"""
|
||||
This function returns the counts of the labeling job records
|
||||
- input_total : total records in the manifest.
|
||||
- human_label : total records labeled by human.
|
||||
- auto_label : total records auto labeled.
|
||||
- unlabeled : count of records not yet labeled.
|
||||
- human_label_percentage : percentage of records labeled by humans.
|
||||
"""
|
||||
label_attribute_name = event["LabelAttributeName"]
|
||||
meta_data = event["meta_data"]
|
||||
s3_input_uri = meta_data["IntermediateManifestS3Uri"]
|
||||
|
||||
source = S3Ref.from_uri(s3_input_uri)
|
||||
manifest_count = partial(get_count_with_query, source)
|
||||
logger.info("Getting counts from {}".format(s3_input_uri))
|
||||
total_query = "select count(*) from s3object s"
|
||||
human_labeled_query = """select count(*) from s3object[*] s where s."{}-metadata"."human-annotated" IN ('yes')""".format(
|
||||
label_attribute_name
|
||||
)
|
||||
auto_labeled_query = """select count(*) from s3object[*] s where s."{}-metadata"."human-annotated" IN ('no')""".format(
|
||||
label_attribute_name
|
||||
)
|
||||
|
||||
manifest_size = manifest_count(total_query)
|
||||
human_labeled_count = manifest_count(human_labeled_query)
|
||||
auto_labeled_count = manifest_count(auto_labeled_query)
|
||||
unlabeled_count = manifest_size - (auto_labeled_count + human_labeled_count)
|
||||
human_label_percentage = int(human_labeled_count * 100.0 / manifest_size)
|
||||
counts = {
|
||||
"input_total": manifest_size,
|
||||
"human_label": human_labeled_count,
|
||||
"auto_label": auto_labeled_count,
|
||||
"unlabeled": unlabeled_count,
|
||||
"human_label_percentage": human_label_percentage,
|
||||
}
|
||||
|
||||
# update the validation set count from previous iteration if present.
|
||||
if "counts" in meta_data and "validation" in meta_data["counts"]:
|
||||
counts["validation"] = meta_data["counts"]["validation"]
|
||||
else:
|
||||
counts["validation"] = 0
|
||||
|
||||
logger.info("Counts computed {} ".format(str(counts)))
|
||||
return counts
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
import json
|
||||
|
||||
|
||||
def lambda_handler(event, context):
|
||||
"""
|
||||
This function is used to update the meta_data values based on active learning logic output.
|
||||
"""
|
||||
output_str = event["active_learning_output"]
|
||||
output_json = json.loads(output_str)
|
||||
return output_json["meta_data"]
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user