portkey-ai--gateway/cookbook/integrations/langchain.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"}},"cells":[{"cell_type":"markdown","source":["<h1 align=\"center\">\n","  <a href=\"https://portkey.ai\">\n","    <img width=\"300\" src=\"https://analyticsindiamag.com/wp-content/uploads/2023/08/Logo-on-white-background.png\" alt=\"portkey\">\n","  </a>\n","</h1>"],"metadata":{"id":"AZHkr3yrCER8"}},{"cell_type":"markdown","source":["[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1-EETdhw2RrOCrsmHZP6P7LzSDMsvsJeu?usp=sharing)"],"metadata":{"id":"jQJD4ga7CGfH"}},{"cell_type":"markdown","source":["# Portkey + Langchain\n","\n","[Portkey](https://app.portkey.ai/) is the Control Panel for AI apps. With it's popular AI Gateway and Observability Suite, hundreds of teams ship reliable, cost-efficient, and fast apps.\n","\n","Portkey brings production readiness to Langchain. With Portkey, you can\n","\n"," - Connect to 150+ models through a unified API,\n"," - View 42+ metrics & logs for all requests,\n"," - Enable semantic cache to reduce latency & costs,\n"," - Implement automatic retries & fallbacks for failed requests,\n"," - Add custom tags to requests for better tracking and analysis and more.\n"],"metadata":{"id":"ovHYC_Qyd8DK"}},{"cell_type":"markdown","source":["## Quickstart\n","\n","Since Portkey is fully compatible with the OpenAI signature, you can connect to the Portkey AI Gateway through the ChatOpenAI interface.\n","\n","- Set the `base_url` as `PORTKEY_GATEWAY_URL`\n","- Add `default_headers` to consume the headers needed by Portkey using the `createHeaders` helper method.\n","\n","To start, get your Portkey API key by signing up [here](https://app.portkey.ai/). (Click the profile icon on the bottom left, then click on \"Copy API Key\")"],"metadata":{"id":"2iY6qt5Ki76x"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"05aQljeSdaDs"},"outputs":[],"source":["!pip install -qU portkey-ai langchain-openai"]},{"cell_type":"markdown","source":["We can now connect to the Portkey AI Gateway by updating the `ChatOpenAI` model in Langchain"],"metadata":{"id":"JVzMZP6jjU-b"}},{"cell_type":"markdown","source":["### Using OpenAI models with Portkey + ChatOpenAI"],"metadata":{"id":"YT-pE3nQj3LO"}},{"cell_type":"code","source":["from langchain_openai import ChatOpenAI\n","from portkey_ai import createHeaders, PORTKEY_GATEWAY_URL\n","from google.colab import userdata\n","\n","portkey_headers = createHeaders(api_key= userdata.get(\"PORTKEY_API_KEY\"), ## Grab from https://app.portkey.ai/\n","                                provider=\"openai\"\n","                                )\n","\n","llm = ChatOpenAI(api_key= userdata.get(\"OPENAI_API_KEY\"),\n","                 base_url=PORTKEY_GATEWAY_URL,\n","                 default_headers=portkey_headers)\n","\n","llm.invoke(\"What is the meaning of life, universe and everything?\")"],"metadata":{"id":"4iLCfaRFe5bM"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["### Using Together AI models with Portkey + ChatOpenAI"],"metadata":{"id":"ANDJQw1zkAkJ"}},{"cell_type":"code","source":["from langchain_openai import ChatOpenAI\n","from portkey_ai import createHeaders, PORTKEY_GATEWAY_URL\n","from google.colab import userdata\n","\n","portkey_headers = createHeaders(api_key= userdata.get(\"PORTKEY_API_KEY\"), ## Grab from https://app.portkey.ai/\n","                                provider=\"together-ai\"\n","                                )\n","\n","llm = ChatOpenAI(model = \"meta-llama/Llama-3-8b-chat-hf\",\n","                 api_key= userdata.get(\"TOGETHER_API_KEY\"), ## Replace it with your provider key\n","                 base_url=PORTKEY_GATEWAY_URL,\n","                 default_headers=portkey_headers)\n","\n","llm.invoke(\"What is the meaning of life, universe and everything?\")"],"metadata":{"id":"6vnyI7gHgeQm"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["## Advanced Routing - Load Balancing, Fallbacks, Retries"],"metadata":{"id":"pQ52DJOYlU0z"}},{"cell_type":"markdown","source":["The Portkey AI Gateway brings capabilities like load-balancing, fallbacks, experimentation and canary testing to Langchain through a configuration-first approach.\n","\n","Let's take an example where we might want to split traffic between `llama-3-70b` and `gpt-3.5` 50:50 to test the two large models. The gateway configuration for this would look like the following:"],"metadata":{"id":"sKn-sbNDljdb"}},{"cell_type":"code","source":["config = {\n","    \"strategy\": {\n","         \"mode\": \"loadbalance\"\n","    },\n","    \"targets\": [{\n","        \"virtual_key\": \"gpt3-8070a6\", # OpenAI's virtual key\n","        \"override_params\": {\"model\": \"gpt-3.5-turbo\"},\n","        \"weight\": 0.5\n","    }, {\n","        \"virtual_key\": \"together-1c20e9\", # Together's virtual key\n","        \"override_params\": {\"model\": \"meta-llama/Llama-3-8b-chat-hf\"},\n","        \"weight\": 0.5\n","    }]\n","}"],"metadata":{"id":"g9VS1cvOFq88"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["from langchain_openai import ChatOpenAI\n","from portkey_ai import createHeaders, PORTKEY_GATEWAY_URL\n","from google.colab import userdata\n","\n","portkey_headers = createHeaders(\n","    api_key= userdata.get(\"PORTKEY_API_KEY\"),\n","    config=config\n",")\n","\n","llm = ChatOpenAI(api_key=\"X\", base_url=PORTKEY_GATEWAY_URL, default_headers=portkey_headers)\n","\n","llm.invoke(\"What is the meaning of life, universe and everything?\")"],"metadata":{"id":"WegP32PNl7jb"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"9hhYkPKbpDlr"},"execution_count":null,"outputs":[]}]}