{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"kernelspec": {
"name": "python3",
"display_name": "Python 3",
"language": "python"
},
"language_info": {
"name": "python",
"version": "3.7.6",
"mimetype": "text/x-python",
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"pygments_lexer": "ipython3",
"nbconvert_exporter": "python",
"file_extension": ".py"
},
"colab": {
"provenance": []
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "POWZoSJR6XzK"
},
"source": [
"# Model explainability\n",
"\n",
"Neural/transformers based approaches have recently made amazing advancements. But it is difficult to understand how models make decisions. This is especially important in sensitive areas where models are being used to drive critical decisions.\n",
"\n",
"This notebook will cover how to gain a level of understanding of complex natural language model outputs."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qa_PPKVX6XzN"
},
"source": [
"# Install dependencies\n",
"\n",
"Install `txtai` and all dependencies."
]
},
{
"cell_type": "code",
"metadata": {
"_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
"_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
"trusted": true,
"_kg_hide-output": true,
"id": "24q-1n5i6XzQ"
},
"source": [
"%%capture\n",
"!pip install git+https://github.com/neuml/txtai#egg=txtai[pipeline] shap"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"# Semantic Search\n",
"\n",
"The first example we'll cover is semantic search. Semantic search applications have an understanding of natural language and identify results that have the same meaning, not necessarily the same keywords. While this produces higher quality results, one advantage of keyword search is it's easy to understand why a result why selected. The keyword is there.\n",
"\n",
"Let's see if we can gain a better understanding of semantic search output. "
],
"metadata": {
"id": "snon4fqZbalQ"
}
},
{
"cell_type": "code",
"source": [
"from txtai.embeddings import Embeddings\n",
"\n",
"data = [\"US tops 5 million confirmed virus cases\",\n",
" \"Canada's last fully intact ice shelf has suddenly collapsed, forming a Manhattan-sized iceberg\",\n",
" \"Beijing mobilises invasion craft along coast as Taiwan tensions escalate\",\n",
" \"The National Park Service warns against sacrificing slower friends in a bear attack\",\n",
" \"Maine man wins $1M from $25 lottery ticket\",\n",
" \"Make huge profits without work, earn up to $100,000 a day\"]\n",
"\n",
"# Create embeddings index with content enabled. The default behavior is to only store indexed vectors.\n",
"embeddings = Embeddings({\"path\": \"sentence-transformers/nli-mpnet-base-v2\", \"content\": True})\n",
"\n",
"# Create an index for the list of text\n",
"embeddings.index([(uid, text, None) for uid, text in enumerate(data)])\n",
"\n",
"# Run a search\n",
"embeddings.explain(\"feel good story\", limit=1)"
],
"metadata": {
"id": "MnRR8pTzK8h4",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "956e405c-568b-44bc-b8ea-d4b6f3cea9b5"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"[{'id': '4',\n",
" 'score': 0.08329004049301147,\n",
" 'text': 'Maine man wins $1M from $25 lottery ticket',\n",
" 'tokens': [('Maine', 0.003297939896583557),\n",
" ('man', -0.03039500117301941),\n",
" ('wins', 0.03406312316656113),\n",
" ('$1M', -0.03121592104434967),\n",
" ('from', -0.02270638197660446),\n",
" ('$25', 0.012891143560409546),\n",
" ('lottery', -0.015372440218925476),\n",
" ('ticket', 0.007445111870765686)]}]"
]
},
"metadata": {},
"execution_count": 29
}
]
},
{
"cell_type": "markdown",
"source": [
"The `explain` method above ran an embeddings query like `search` but also analyzed each token to determine term importance. Looking at the results, it appears that `win` is the most important term. Let's visualize it."
],
"metadata": {
"id": "ZEkXurdobRKL"
}
},
{
"cell_type": "code",
"source": [
"from IPython.display import HTML\n",
"\n",
"def plot(query):\n",
" result = embeddings.explain(query, limit=1)[0]\n",
"\n",
" output = f\"{query} \"\n",
" spans = []\n",
" for token, score in result[\"tokens\"]:\n",
" color = None\n",
" if score >= 0.1:\n",
" color = \"#fdd835\"\n",
" elif score >= 0.075:\n",
" color = \"#ffeb3b\"\n",
" elif score >= 0.05:\n",
" color = \"#ffee58\"\n",
" elif score >= 0.02:\n",
" color = \"#fff59d\"\n",
"\n",
" spans.append((token, score, color))\n",
"\n",
" if result[\"score\"] >= 0.05 and not [color for _, _, color in spans if color]:\n",
" mscore = max([score for _, score, _ in spans])\n",
" spans = [(token, score, \"#fff59d\" if score == mscore else color) for token, score, color in spans]\n",
"\n",
" for token, _, color in spans:\n",
" if color:\n",
" output += f\"{token} \"\n",
" else:\n",
" output += f\"{token} \"\n",
"\n",
" return output\n",
"\n",
"HTML(plot(\"feel good story\"))"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 52
},
"id": "klJYGOrypXUL",
"outputId": "88959716-1dcc-4fee-d784-cb398aa87eb5"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
""
],
"text/html": [
"feel good story Maine man wins $1M from $25 lottery ticket "
]
},
"metadata": {},
"execution_count": 30
}
]
},
{
"cell_type": "markdown",
"source": [
"Let's try some more queries!"
],
"metadata": {
"id": "CCHqZffacPVh"
}
},
{
"cell_type": "code",
"source": [
"output = \"\"\n",
"for query in [\"feel good story\", \"climate change\", \"public health story\", \"war\", \"wildlife\", \"asia\", \"lucky\", \"dishonest junk\"]:\n",
" output += plot(query) + \"
\"\n",
"\n",
"HTML(output)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 434
},
"id": "OOc8OfKfcpPL",
"outputId": "75541eb6-f607-42b7-c2c8-43346fa7da8f"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
""
],
"text/html": [
"feel good story Maine man wins $1M from $25 lottery ticket
climate change Canada's last fully intact ice shelf hassuddenly collapsed, forming a Manhattan-sized iceberg
public health story US tops 5 million confirmedvirus cases
war Beijing mobilises invasioncraftalong coast as Taiwan tensions escalate
wildlife The National Park Service warns against sacrificing slower friends in a bear attack
asia Beijing mobilises invasion craft along coast as Taiwan tensions escalate
lucky Maine man wins $1M from $25 lottery ticket
dishonest junk Make huge profits without work, earn up to $100,000 a day
"
]
},
"metadata": {},
"execution_count": 31
}
]
},
{
"cell_type": "markdown",
"source": [
"There is also a batch method that can run bulk explainations more efficently. "
],
"metadata": {
"id": "WOa54eJ-c9nc"
}
},
{
"cell_type": "code",
"source": [
"queries = [\"feel good story\", \"climate change\", \"public health story\", \"war\", \"wildlife\", \"asia\", \"lucky\", \"dishonest junk\"]\n",
"results = embeddings.batchexplain(queries, limit=1)\n",
"\n",
"for x, result in enumerate(results):\n",
" print(result)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "viuEwL4scQjx",
"outputId": "cb65e678-b694-4fe6-eb94-dea821bda643"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"[{'id': '4', 'text': 'Maine man wins $1M from $25 lottery ticket', 'score': 0.08329004049301147, 'tokens': [('Maine', 0.003297939896583557), ('man', -0.03039500117301941), ('wins', 0.03406312316656113), ('$1M', -0.03121592104434967), ('from', -0.02270638197660446), ('$25', 0.012891143560409546), ('lottery', -0.015372440218925476), ('ticket', 0.007445111870765686)]}]\n",
"[{'id': '1', 'text': \"Canada's last fully intact ice shelf has suddenly collapsed, forming a Manhattan-sized iceberg\", 'score': 0.24478264153003693, 'tokens': [(\"Canada's\", -0.026454076170921326), ('last', 0.017057165503501892), ('fully', 0.007285907864570618), ('intact', -0.005608782172203064), ('ice', 0.009459629654884338), ('shelf', -0.029393181204795837), ('has', 0.0253918319940567), ('suddenly', 0.021642476320266724), ('collapsed,', -0.030680224299430847), ('forming', 0.01910528540611267), ('a', -0.00890059769153595), ('Manhattan-sized', -0.023612067103385925), ('iceberg', -0.009710296988487244)]}]\n",
"[{'id': '0', 'text': 'US tops 5 million confirmed virus cases', 'score': 0.1701308637857437, 'tokens': [('US', -0.02426217496395111), ('tops', -0.04896041750907898), ('5', -0.040287598967552185), ('million', -0.04737819731235504), ('confirmed', 0.02050541341304779), ('virus', 0.05511370301246643), ('cases', -0.029122650623321533)]}]\n",
"[{'id': '2', 'text': 'Beijing mobilises invasion craft along coast as Taiwan tensions escalate', 'score': 0.2714069187641144, 'tokens': [('Beijing', -0.040329575538635254), ('mobilises', -0.01986941695213318), ('invasion', 0.06464864313602448), ('craft', 0.044328778982162476), ('along', 0.021214008331298828), ('coast', -0.01738378405570984), ('as', -0.02182626724243164), ('Taiwan', -0.020671993494033813), ('tensions', -0.007258296012878418), ('escalate', -0.01663634181022644)]}]\n",
"[{'id': '3', 'text': 'The National Park Service warns against sacrificing slower friends in a bear attack', 'score': 0.28424495458602905, 'tokens': [('The', -0.022544533014297485), ('National', -0.005589812994003296), ('Park', 0.08145171403884888), ('Service', -0.016785144805908203), ('warns', -0.03266721963882446), ('against', -0.032368004322052), ('sacrificing', -0.04440906643867493), ('slower', 0.034766435623168945), ('friends', 0.0013159513473510742), ('in', -0.008420556783676147), ('a', 0.015498429536819458), ('bear', 0.08734165132045746), ('attack', -0.011731922626495361)]}]\n",
"[{'id': '2', 'text': 'Beijing mobilises invasion craft along coast as Taiwan tensions escalate', 'score': 0.24338798224925995, 'tokens': [('Beijing', -0.032770439982414246), ('mobilises', -0.04045189917087555), ('invasion', -0.0015233010053634644), ('craft', 0.017402753233909607), ('along', 0.004210904240608215), ('coast', 0.0028585344552993774), ('as', -0.0018710196018218994), ('Taiwan', 0.01866382360458374), ('tensions', -0.011064544320106506), ('escalate', -0.029331132769584656)]}]\n",
"[{'id': '4', 'text': 'Maine man wins $1M from $25 lottery ticket', 'score': 0.06539873033761978, 'tokens': [('Maine', 0.012625649571418762), ('man', -0.013015367090702057), ('wins', -0.022461198270320892), ('$1M', -0.041918568313121796), ('from', -0.02305116504430771), ('$25', -0.029282495379447937), ('lottery', 0.02279689908027649), ('ticket', -0.009147539734840393)]}]\n",
"[{'id': '5', 'text': 'Make huge profits without work, earn up to $100,000 a day', 'score': 0.033823199570178986, 'tokens': [('Make', 0.0013405345380306244), ('huge', 0.002276904881000519), ('profits', 0.02767787780612707), ('without', -0.007079385221004486), ('work,', -0.019851915538311005), ('earn', -0.026906955987215042), ('up', 0.00074811652302742), ('to', 0.007462538778781891), ('$100,000', -0.03565136343240738), ('a', -0.009965047240257263), ('day', -0.0021888017654418945)]}]\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"Of course, this method is supported through YAML-based applications and the API."
],
"metadata": {
"id": "WWH6l9nOdD-l"
}
},
{
"cell_type": "code",
"source": [
"from txtai.app import Application\n",
"\n",
"app = Application(\"\"\"\n",
"writable: true\n",
"embeddings:\n",
" path: sentence-transformers/nli-mpnet-base-v2\n",
" content: true\n",
"\"\"\")\n",
"\n",
"app.add([{\"id\": uid, \"text\": text} for uid, text in enumerate(data)])\n",
"app.index()\n",
"\n",
"app.explain(\"feel good story\", limit=1)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Nwrd3v6cdKR_",
"outputId": "da949ae7-f681-4197-9df2-d5e2cb304f24"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"[{'id': '4',\n",
" 'score': 0.08329004049301147,\n",
" 'text': 'Maine man wins $1M from $25 lottery ticket',\n",
" 'tokens': [('Maine', 0.003297939896583557),\n",
" ('man', -0.03039500117301941),\n",
" ('wins', 0.03406312316656113),\n",
" ('$1M', -0.03121592104434967),\n",
" ('from', -0.02270638197660446),\n",
" ('$25', 0.012891143560409546),\n",
" ('lottery', -0.015372440218925476),\n",
" ('ticket', 0.007445111870765686)]}]"
]
},
"metadata": {},
"execution_count": 33
}
]
},
{
"cell_type": "markdown",
"source": [
"# Pipeline models\n",
"\n",
"txtai pipelines are wrappers around Hugging Face pipelines with logic to easily integrate with txtai's workflow framework. Given that, we can use the [SHAP](https://github.com/slundberg/shap) library to explain predictions.\n",
"\n",
"Let's try a sentiment analysis example."
],
"metadata": {
"id": "RsRuLWYpdup_"
}
},
{
"cell_type": "code",
"source": [
"import shap\n",
"\n",
"from txtai.pipeline import Labels\n",
"\n",
"data = [\"Dodgers lose again, give up 3 HRs in a loss to the Giants\",\n",
" \"Massive dunk!!! they are now up by 15 with 2 minutes to go\"]\n",
"\n",
"labels = Labels(dynamic=False)\n",
"\n",
"# explain the model on two sample inputs\n",
"explainer = shap.Explainer(labels.pipeline) \n",
"shap_values = explainer(data)"
],
"metadata": {
"id": "SnoPSv1-fxvF"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"shap.plots.text(shap_values[0, :, \"NEGATIVE\"])"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 118
},
"id": "YW_qbzPKpRdL",
"outputId": "c21b3558-feed-4854-9444-954f87aa8422"
},
"execution_count": null,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
""
],
"text/html": [
"
"
]
},
"metadata": {}
}
]
},
{
"cell_type": "markdown",
"source": [
"The [SHAP documentation](https://shap.readthedocs.io/en/latest/text_examples.html) provides a great list of additional examples for translation, text generation, summarization, translation and question-answering.\n",
"\n",
"The SHAP library is pretty 🔥🔥🔥 Check it out for more!"
],
"metadata": {
"id": "8atTIz37iP9Q"
}
},
{
"cell_type": "markdown",
"source": [
"# Wrapping up\n",
"\n",
"This notebook briefly introduced model explainability. There is a lot of work in this area, expect a number of different methods to become available. Model explainability helps users gain a level of trust in model predictions. It also helps debug why a model is making a decision, which can potentially drive how to fine-tune a model to make better predictions. \n",
"\n",
"Keep an eye on this important area over the coming months!\n"
],
"metadata": {
"id": "YujdAMlGh0qT"
}
}
]
}