microsoft--promptflow/docs/reference/tools-reference/contentsafety_text_tool.md

Content Safety (Text)

Azure Content Safety is a content moderation service developed by Microsoft that help users detect harmful content from different modalities and languages. This tool is a wrapper for the Azure Content Safety Text API, which allows you to detect text content and get moderation results. See the Azure Content Safety for more information.

Requirements

• For AzureML users, the tool is installed in default image, you can use the tool without extra installation.
• For local users, pip install promptflow-tools

Note

Content Safety (Text) tool is now incorporated into the latest promptflow-tools package. If you have previously installed the package promptflow-contentsafety, please uninstall it to avoid the duplication in your local tool list.

Prerequisites

• Create an Azure Content Safety resource.
• Add "Azure Content Safety" connection in prompt flow. Fill "API key" field with "Primary key" from "Keys and Endpoint" section of created resource.

Inputs

You can use the following parameters as inputs for this tool:

Name	Type	Description	Required
text	string	The text that need to be moderated.	Yes
hate_category	string	The moderation sensitivity for Hate category. You can choose from four options: disable, low_sensitivity, medium_sensitivity, or high_sensitivity. The disable option means no moderation for hate category. The other three options mean different degrees of strictness in filtering out hate content. The default option is medium_sensitivity.	Yes
sexual_category	string	The moderation sensitivity for Sexual category. You can choose from four options: disable, low_sensitivity, medium_sensitivity, or high_sensitivity. The disable option means no moderation for sexual category. The other three options mean different degrees of strictness in filtering out sexual content. The default option is medium_sensitivity.	Yes
self_harm_category	string	The moderation sensitivity for Self-harm category. You can choose from four options: disable, low_sensitivity, medium_sensitivity, or high_sensitivity. The disable option means no moderation for self-harm category. The other three options mean different degrees of strictness in filtering out self_harm content. The default option is medium_sensitivity.	Yes
violence_category	string	The moderation sensitivity for Violence category. You can choose from four options: disable, low_sensitivity, medium_sensitivity, or high_sensitivity. The disable option means no moderation for violence category. The other three options mean different degrees of strictness in filtering out violence content. The default option is medium_sensitivity.	Yes

For more information, please refer to Azure Content Safety

Outputs

The following is an example JSON format response returned by the tool:

Output

{
    "action_by_category": {
      "Hate": "Accept",
      "SelfHarm": "Accept",
      "Sexual": "Accept",
      "Violence": "Accept"
    },
    "suggested_action": "Accept"
  }

The action_by_category field gives you a binary value for each category: Accept or Reject. This value shows if the text meets the sensitivity level that you set in the request parameters for that category.

The suggested_action field gives you an overall recommendation based on the four categories. If any category has a Reject value, the suggested_action will be Reject as well.