{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3", "language": "python" }, "language_info": { "name": "python", "version": "3.7.6", "mimetype": "text/x-python", "codemirror_mode": { "name": "ipython", "version": 3 }, "pygments_lexer": "ipython3", "nbconvert_exporter": "python", "file_extension": ".py" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": { "id": "POWZoSJR6XzK" }, "source": [ "# Model explainability\n", "\n", "Neural/transformers based approaches have recently made amazing advancements. But it is difficult to understand how models make decisions. This is especially important in sensitive areas where models are being used to drive critical decisions.\n", "\n", "This notebook will cover how to gain a level of understanding of complex natural language model outputs." ] }, { "cell_type": "markdown", "metadata": { "id": "qa_PPKVX6XzN" }, "source": [ "# Install dependencies\n", "\n", "Install `txtai` and all dependencies." ] }, { "cell_type": "code", "metadata": { "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5", "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19", "trusted": true, "_kg_hide-output": true, "id": "24q-1n5i6XzQ" }, "source": [ "%%capture\n", "!pip install git+https://github.com/neuml/txtai#egg=txtai[pipeline] shap" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "# Semantic Search\n", "\n", "The first example we'll cover is semantic search. Semantic search applications have an understanding of natural language and identify results that have the same meaning, not necessarily the same keywords. While this produces higher quality results, one advantage of keyword search is it's easy to understand why a result why selected. The keyword is there.\n", "\n", "Let's see if we can gain a better understanding of semantic search output. " ], "metadata": { "id": "snon4fqZbalQ" } }, { "cell_type": "code", "source": [ "from txtai.embeddings import Embeddings\n", "\n", "data = [\"US tops 5 million confirmed virus cases\",\n", " \"Canada's last fully intact ice shelf has suddenly collapsed, forming a Manhattan-sized iceberg\",\n", " \"Beijing mobilises invasion craft along coast as Taiwan tensions escalate\",\n", " \"The National Park Service warns against sacrificing slower friends in a bear attack\",\n", " \"Maine man wins $1M from $25 lottery ticket\",\n", " \"Make huge profits without work, earn up to $100,000 a day\"]\n", "\n", "# Create embeddings index with content enabled. The default behavior is to only store indexed vectors.\n", "embeddings = Embeddings({\"path\": \"sentence-transformers/nli-mpnet-base-v2\", \"content\": True})\n", "\n", "# Create an index for the list of text\n", "embeddings.index([(uid, text, None) for uid, text in enumerate(data)])\n", "\n", "# Run a search\n", "embeddings.explain(\"feel good story\", limit=1)" ], "metadata": { "id": "MnRR8pTzK8h4", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "956e405c-568b-44bc-b8ea-d4b6f3cea9b5" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "[{'id': '4',\n", " 'score': 0.08329004049301147,\n", " 'text': 'Maine man wins $1M from $25 lottery ticket',\n", " 'tokens': [('Maine', 0.003297939896583557),\n", " ('man', -0.03039500117301941),\n", " ('wins', 0.03406312316656113),\n", " ('$1M', -0.03121592104434967),\n", " ('from', -0.02270638197660446),\n", " ('$25', 0.012891143560409546),\n", " ('lottery', -0.015372440218925476),\n", " ('ticket', 0.007445111870765686)]}]" ] }, "metadata": {}, "execution_count": 29 } ] }, { "cell_type": "markdown", "source": [ "The `explain` method above ran an embeddings query like `search` but also analyzed each token to determine term importance. Looking at the results, it appears that `win` is the most important term. Let's visualize it." ], "metadata": { "id": "ZEkXurdobRKL" } }, { "cell_type": "code", "source": [ "from IPython.display import HTML\n", "\n", "def plot(query):\n", " result = embeddings.explain(query, limit=1)[0]\n", "\n", " output = f\"{query}
\"\n", " spans = []\n", " for token, score in result[\"tokens\"]:\n", " color = None\n", " if score >= 0.1:\n", " color = \"#fdd835\"\n", " elif score >= 0.075:\n", " color = \"#ffeb3b\"\n", " elif score >= 0.05:\n", " color = \"#ffee58\"\n", " elif score >= 0.02:\n", " color = \"#fff59d\"\n", "\n", " spans.append((token, score, color))\n", "\n", " if result[\"score\"] >= 0.05 and not [color for _, _, color in spans if color]:\n", " mscore = max([score for _, score, _ in spans])\n", " spans = [(token, score, \"#fff59d\" if score == mscore else color) for token, score, color in spans]\n", "\n", " for token, _, color in spans:\n", " if color:\n", " output += f\"{token} \"\n", " else:\n", " output += f\"{token} \"\n", "\n", " return output\n", "\n", "HTML(plot(\"feel good story\"))" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 52 }, "id": "klJYGOrypXUL", "outputId": "88959716-1dcc-4fee-d784-cb398aa87eb5" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ], "text/html": [ "feel good story
Maine man wins $1M from $25 lottery ticket " ] }, "metadata": {}, "execution_count": 30 } ] }, { "cell_type": "markdown", "source": [ "Let's try some more queries!" ], "metadata": { "id": "CCHqZffacPVh" } }, { "cell_type": "code", "source": [ "output = \"\"\n", "for query in [\"feel good story\", \"climate change\", \"public health story\", \"war\", \"wildlife\", \"asia\", \"lucky\", \"dishonest junk\"]:\n", " output += plot(query) + \"

\"\n", "\n", "HTML(output)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 434 }, "id": "OOc8OfKfcpPL", "outputId": "75541eb6-f607-42b7-c2c8-43346fa7da8f" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ], "text/html": [ "feel good story
Maine man wins $1M from $25 lottery ticket

climate change
Canada's last fully intact ice shelf has suddenly collapsed, forming a Manhattan-sized iceberg

public health story
US tops 5 million confirmed virus cases

war
Beijing mobilises invasion craft along coast as Taiwan tensions escalate

wildlife
The National Park Service warns against sacrificing slower friends in a bear attack

asia
Beijing mobilises invasion craft along coast as Taiwan tensions escalate

lucky
Maine man wins $1M from $25 lottery ticket

dishonest junk
Make huge profits without work, earn up to $100,000 a day

" ] }, "metadata": {}, "execution_count": 31 } ] }, { "cell_type": "markdown", "source": [ "There is also a batch method that can run bulk explainations more efficently. " ], "metadata": { "id": "WOa54eJ-c9nc" } }, { "cell_type": "code", "source": [ "queries = [\"feel good story\", \"climate change\", \"public health story\", \"war\", \"wildlife\", \"asia\", \"lucky\", \"dishonest junk\"]\n", "results = embeddings.batchexplain(queries, limit=1)\n", "\n", "for x, result in enumerate(results):\n", " print(result)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "viuEwL4scQjx", "outputId": "cb65e678-b694-4fe6-eb94-dea821bda643" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "[{'id': '4', 'text': 'Maine man wins $1M from $25 lottery ticket', 'score': 0.08329004049301147, 'tokens': [('Maine', 0.003297939896583557), ('man', -0.03039500117301941), ('wins', 0.03406312316656113), ('$1M', -0.03121592104434967), ('from', -0.02270638197660446), ('$25', 0.012891143560409546), ('lottery', -0.015372440218925476), ('ticket', 0.007445111870765686)]}]\n", "[{'id': '1', 'text': \"Canada's last fully intact ice shelf has suddenly collapsed, forming a Manhattan-sized iceberg\", 'score': 0.24478264153003693, 'tokens': [(\"Canada's\", -0.026454076170921326), ('last', 0.017057165503501892), ('fully', 0.007285907864570618), ('intact', -0.005608782172203064), ('ice', 0.009459629654884338), ('shelf', -0.029393181204795837), ('has', 0.0253918319940567), ('suddenly', 0.021642476320266724), ('collapsed,', -0.030680224299430847), ('forming', 0.01910528540611267), ('a', -0.00890059769153595), ('Manhattan-sized', -0.023612067103385925), ('iceberg', -0.009710296988487244)]}]\n", "[{'id': '0', 'text': 'US tops 5 million confirmed virus cases', 'score': 0.1701308637857437, 'tokens': [('US', -0.02426217496395111), ('tops', -0.04896041750907898), ('5', -0.040287598967552185), ('million', -0.04737819731235504), ('confirmed', 0.02050541341304779), ('virus', 0.05511370301246643), ('cases', -0.029122650623321533)]}]\n", "[{'id': '2', 'text': 'Beijing mobilises invasion craft along coast as Taiwan tensions escalate', 'score': 0.2714069187641144, 'tokens': [('Beijing', -0.040329575538635254), ('mobilises', -0.01986941695213318), ('invasion', 0.06464864313602448), ('craft', 0.044328778982162476), ('along', 0.021214008331298828), ('coast', -0.01738378405570984), ('as', -0.02182626724243164), ('Taiwan', -0.020671993494033813), ('tensions', -0.007258296012878418), ('escalate', -0.01663634181022644)]}]\n", "[{'id': '3', 'text': 'The National Park Service warns against sacrificing slower friends in a bear attack', 'score': 0.28424495458602905, 'tokens': [('The', -0.022544533014297485), ('National', -0.005589812994003296), ('Park', 0.08145171403884888), ('Service', -0.016785144805908203), ('warns', -0.03266721963882446), ('against', -0.032368004322052), ('sacrificing', -0.04440906643867493), ('slower', 0.034766435623168945), ('friends', 0.0013159513473510742), ('in', -0.008420556783676147), ('a', 0.015498429536819458), ('bear', 0.08734165132045746), ('attack', -0.011731922626495361)]}]\n", "[{'id': '2', 'text': 'Beijing mobilises invasion craft along coast as Taiwan tensions escalate', 'score': 0.24338798224925995, 'tokens': [('Beijing', -0.032770439982414246), ('mobilises', -0.04045189917087555), ('invasion', -0.0015233010053634644), ('craft', 0.017402753233909607), ('along', 0.004210904240608215), ('coast', 0.0028585344552993774), ('as', -0.0018710196018218994), ('Taiwan', 0.01866382360458374), ('tensions', -0.011064544320106506), ('escalate', -0.029331132769584656)]}]\n", "[{'id': '4', 'text': 'Maine man wins $1M from $25 lottery ticket', 'score': 0.06539873033761978, 'tokens': [('Maine', 0.012625649571418762), ('man', -0.013015367090702057), ('wins', -0.022461198270320892), ('$1M', -0.041918568313121796), ('from', -0.02305116504430771), ('$25', -0.029282495379447937), ('lottery', 0.02279689908027649), ('ticket', -0.009147539734840393)]}]\n", "[{'id': '5', 'text': 'Make huge profits without work, earn up to $100,000 a day', 'score': 0.033823199570178986, 'tokens': [('Make', 0.0013405345380306244), ('huge', 0.002276904881000519), ('profits', 0.02767787780612707), ('without', -0.007079385221004486), ('work,', -0.019851915538311005), ('earn', -0.026906955987215042), ('up', 0.00074811652302742), ('to', 0.007462538778781891), ('$100,000', -0.03565136343240738), ('a', -0.009965047240257263), ('day', -0.0021888017654418945)]}]\n" ] } ] }, { "cell_type": "markdown", "source": [ "Of course, this method is supported through YAML-based applications and the API." ], "metadata": { "id": "WWH6l9nOdD-l" } }, { "cell_type": "code", "source": [ "from txtai.app import Application\n", "\n", "app = Application(\"\"\"\n", "writable: true\n", "embeddings:\n", " path: sentence-transformers/nli-mpnet-base-v2\n", " content: true\n", "\"\"\")\n", "\n", "app.add([{\"id\": uid, \"text\": text} for uid, text in enumerate(data)])\n", "app.index()\n", "\n", "app.explain(\"feel good story\", limit=1)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Nwrd3v6cdKR_", "outputId": "da949ae7-f681-4197-9df2-d5e2cb304f24" }, "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "[{'id': '4',\n", " 'score': 0.08329004049301147,\n", " 'text': 'Maine man wins $1M from $25 lottery ticket',\n", " 'tokens': [('Maine', 0.003297939896583557),\n", " ('man', -0.03039500117301941),\n", " ('wins', 0.03406312316656113),\n", " ('$1M', -0.03121592104434967),\n", " ('from', -0.02270638197660446),\n", " ('$25', 0.012891143560409546),\n", " ('lottery', -0.015372440218925476),\n", " ('ticket', 0.007445111870765686)]}]" ] }, "metadata": {}, "execution_count": 33 } ] }, { "cell_type": "markdown", "source": [ "# Pipeline models\n", "\n", "txtai pipelines are wrappers around Hugging Face pipelines with logic to easily integrate with txtai's workflow framework. Given that, we can use the [SHAP](https://github.com/slundberg/shap) library to explain predictions.\n", "\n", "Let's try a sentiment analysis example." ], "metadata": { "id": "RsRuLWYpdup_" } }, { "cell_type": "code", "source": [ "import shap\n", "\n", "from txtai.pipeline import Labels\n", "\n", "data = [\"Dodgers lose again, give up 3 HRs in a loss to the Giants\",\n", " \"Massive dunk!!! they are now up by 15 with 2 minutes to go\"]\n", "\n", "labels = Labels(dynamic=False)\n", "\n", "# explain the model on two sample inputs\n", "explainer = shap.Explainer(labels.pipeline) \n", "shap_values = explainer(data)" ], "metadata": { "id": "SnoPSv1-fxvF" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "shap.plots.text(shap_values[0, :, \"NEGATIVE\"])" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 118 }, "id": "YW_qbzPKpRdL", "outputId": "c21b3558-feed-4854-9444-954f87aa8422" }, "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "text/plain": [ "" ], "text/html": [ "0.80.70.60.50.40.911.10.5677350.567735base value0.995090.99509fNEGATIVE(inputs)0.192 loss 0.143 lose 0.082 up 0.059 Dodgers 0.054 , 0.051 again 0.017 a 0.009 HR 0.007 s -0.108 Giants -0.021 give -0.02 the -0.018 3 -0.012 to -0.009 in -0.0
inputs
0.0
0.059
Dodgers
0.143
lose
0.051
again
0.054
,
-0.021
give
0.082
up
-0.018
3
0.009
HR
0.007
s
-0.009
in
0.017
a
0.192
loss
-0.012
to
-0.02
the
-0.108
Giants
-0.0
" ] }, "metadata": {} } ] }, { "cell_type": "code", "source": [ "shap.plots.text(shap_values[1, :, \"NEGATIVE\"])" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 118 }, "id": "SU2hHRjCpc_I", "outputId": "83e1ada5-de45-426a-ae64-dae5b66b8e6a" }, "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "text/plain": [ "" ], "text/html": [ "0.30-0.30.60.90.538360.53836base value00fNEGATIVE(inputs)0.199 Massive 0.171 minutes 0.045 ! 0.029 ! 0.025 by 0.024 ! 0.016 15 0.008 to 0.007 2 -0.392 k -0.184 up -0.123 go -0.121 dun -0.118 they -0.052 are -0.049 now -0.024 with
inputs
0.0
0.199
Massive
-0.121
dun
-0.392
k
0.045
!
0.024
!
0.029
!
-0.118
they
-0.052
are
-0.049
now
-0.184
up
0.025
by
0.016
15
-0.024
with
0.007
2
0.171
minutes
0.008
to
-0.123
go
0.0
" ] }, "metadata": {} } ] }, { "cell_type": "markdown", "source": [ "The [SHAP documentation](https://shap.readthedocs.io/en/latest/text_examples.html) provides a great list of additional examples for translation, text generation, summarization, translation and question-answering.\n", "\n", "The SHAP library is pretty 🔥🔥🔥 Check it out for more!" ], "metadata": { "id": "8atTIz37iP9Q" } }, { "cell_type": "markdown", "source": [ "# Wrapping up\n", "\n", "This notebook briefly introduced model explainability. There is a lot of work in this area, expect a number of different methods to become available. Model explainability helps users gain a level of trust in model predictions. It also helps debug why a model is making a decision, which can potentially drive how to fine-tune a model to make better predictions. \n", "\n", "Keep an eye on this important area over the coming months!\n" ], "metadata": { "id": "YujdAMlGh0qT" } } ] }