wehub/portkey-ai--gateway
1
0
Fork 0
Code Issues Pull Requests Actions 1 Deployments Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
Check Markdown links / linkChecker (push) Waiting to run
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:38:56 +08:00
commit 3cd11ababe
766 changed files with 141100 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.dockerignore
+2
View File
@@ -0,0 +1,2 @@
.toml
docs
.git-blame-ignore-revs
+2
View File
@@ -0,0 +1,2 @@
# Initial prettier formatting
77f32e57b14bd3585169b7bc884ff9ddf38e8ea1
.gitattributes
+33
View File
@@ -0,0 +1,33 @@
# Declare files that will always have LF line endings on checkout.
*.sh text eol=lf
# Denote all files that are truly binary and should not be modified.
*.png binary
*.jpg binary
*.jpeg binary
*.gif binary
*.ico binary
*.mov binary
*.mp4 binary
*.mp3 binary
*.flv binary
*.fla binary
*.swf binary
*.gz binary
*.zip binary
*.7z binary
*.ttf binary
*.eot binary
*.woff binary
*.woff2 binary
# Ignore cookbook folder for language statistics
cookbook/** linguist-vendored
docs/** linguist-vendored
# Treat package-lock.json and yarn.lock as binary to avoid merge conflicts
package-lock.json binary
yarn.lock binary
# Treat .env files as binary to avoid accidental commits of sensitive information
*.env binary
.github/CODE_OF_CONDUCT.md
+132
View File
@@ -0,0 +1,132 @@
<p align="right">
<strong>English</strong>|<a href="https://github.com/Portkey-AI/gateway/blob/main/.github/cn.CODE_OF_CONDUCT.md">中文</a>
</p>
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, religion, or sexual identity
and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our
community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
* Focusing on what is best not just for us as individuals, but for the
overall community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or
advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email
address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.
Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
support@portkey.ai.
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series
of actions.
**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or
permanent ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within
the community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.0, available at
https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
Community Impact Guidelines were inspired by [Mozilla's code of conduct
enforcement ladder](https://github.com/mozilla/diversity).
[homepage]: https://www.contributor-covenant.org
For answers to common questions about this code of conduct, see the FAQ at
https://www.contributor-covenant.org/faq. Translations are available at
https://www.contributor-covenant.org/translations.
.github/CONTRIBUTING.cn.md
+40
View File
@@ -0,0 +1,40 @@
## 🎉 欢迎
你好，感谢你考虑为Portkey的AI网关做出贡献！无论你是在报告一个bug，建议一个功能，改进文档，还是编写代码，你的贡献对我们来说都是非常宝贵的。
## 🚀 快速开始
1. 在Github上Fork仓库。
2. 将你fork的仓库克隆到你的机器上。
```sh
$ git clone https://github.com/YOUR_USERNAME/gateway.git
```
## 🖋 贡献类型
1. 新的集成：为其他LLM供应商或总体供应商创建集成。
2. Bug修复
3. 增强
4. 文档
5. **Hacktoberfest** 提交！
## 🗓️ Hacktoberfest
在 [Hacktoberfest 月](https://hacktoberfest.com/)期间，从10月1日到31日，您被接受的 PR 将计入您的 Hacktoberfest 参与度！🚀
✅ 要获得接受，您的 PR 必须被合并、批准或贴上 `hacktoberfest-accepted` 标签。
🧐 记得遵守 [质量标准](https://hacktoberfest.digitalocean.com/resources/qualitystandards)以避免您的 PR 被标记为 `垃圾邮件` 或 `无效`。
## 🔄 提交 PR
1. 完成您的更改后，通过运行以下命令来格式化和 Lint 您的代码：
```sh
make format; make lint
```
2. 在提交 PR 时，请在标题前添加以下任何一个：
* `provider:` 添加新的 LLM 提供商。
* `integration:` 新的集成。
* `docs:` 新的指南、文档添加等。
* `improvement:` 改进或增强。
* `bug:` 修复错误。
* `hacktoberfest:` Hacktoberfest 贡献。
## 🤔 寻求帮助
遇到问题或有疑问？请毫不犹豫地在我们的 [Discord 社区](https://discord.com/invite/DD7vgKK299) 分享您的疑问或问题 - 这是获得支持和与其他贡献者联系的最快方式。
## 🚧 发布流程
我们尽快进行发布，以确保新功能和修复能快速地到达用户手中。我们遵循无缝的 CI/CD 流水线，以确保代码从开发到生产的平稳过渡。
## 🎊 您的 PR 被合并了！
所有成功的 PR 都会在我们的 [Discord](https://discord.com/invite/DD7vgKK299) 上庆祝，并在发布说明中提及，重大贡献将在我们的 [Twitter](https://twitter.com/PortkeyAI) 上突出显示。请继续关注，未来我们将为贡献者提供更多奖金和礼品！
.github/CONTRIBUTING.md
+45
View File
@@ -0,0 +1,45 @@
## 🎉 Welcome
Hello and thank you for considering contributing to Portkey's AI Gateway! Whether you're reporting a bug, suggesting a feature, improving documentation, or writing code, your contributions are invaluable to us.
## 🚀 Quick Start
1. Fork the repository on Github.
2. Clone your forked repository to your machine.
```sh
$ git clone https://github.com/YOUR_USERNAME/gateway.git
```
## 🖋 Types of Contributions
1. New integrations: Creating integrations for other LLM providers or vendors in general.
2. Bug fixes
3. Enhancements
4. Documentation
5. **Hacktoberfest** submissions!
## 🗓️ Hacktoberfest
During the [Hacktoberfest month](https://hacktoberfest.com/), running from October 1st to 31st, your accepted PR will count towards your Hacktoberfest participation! 🚀
✅ To gain acceptance, your PR must be merged, approved, or tagged with the `hacktoberfest-accepted` label.
🧐 Remember to adhere to the [quality standards](https://hacktoberfest.digitalocean.com/resources/qualitystandards) to avoid your PR being marked as `spam` or `invalid`.
## 🔄 Raising PRs
1. Once you are done with your changes, format and Lint your code by running:
```sh
make format; make lint
```
2. While raising your PRs, please prepend any of the following to your title:
* `provider:` for adding new LLM providers.
* `integration:` for new integrations.
* `docs`: for new cookbooks, doc additions, etc.
* `improvement:` for improvements or enhancements.
* `bug:` for bug fixes.
* `hacktoberfest:` for Hacktoberfest contributions
## 🤔 Getting Help
Facing issues or have questions? Don't hesitate to share your doubts or questions on our [Discord Community](https://discord.com/invite/DD7vgKK299) - this is the quickest way to get support and connect with other contributors.
## 🚧 Release Process
Releases are made as soon as possible to ensure that new features and fixes reach our users quickly. We follow a seamless CI/CD pipeline to ensure the smooth transition of code from development to production.
## 🎊 Your PR is Merged!
All successful PRs are celebrated on our [Discord](https://discord.com/invite/DD7vgKK299) and are mentioned in the release notes, and significant contributions are highlighted on our [Twitter](https://twitter.com/PortkeyAI). Stay tuned for more bounties and goodies for contributors in the near future!
.github/ISSUE_TEMPLATE/bug_report.yml
+29
View File
@@ -0,0 +1,29 @@
name: Bug Report
description: Report any issue with the project
labels: ["bug"]
body:
- type: textarea
id: what-happened
attributes:
label: What Happened?
validations:
required: true
- type: textarea
id: expected-behavior
attributes:
label: What Should Have Happened?
validations:
required: false
- type: textarea
id: code-snippet
attributes:
label: Relevant Code Snippet
validations:
required: false
- type: input
id: contact
attributes:
label: Your Twitter/LinkedIn
description: When the bug get fixed, we'd like to thank you publicly for reporting it.
validations:
required: false
.github/ISSUE_TEMPLATE/config.yml
+8
View File
@@ -0,0 +1,8 @@
blank_issues_enabled: true
contact_links:
- name: Discord for Support & Discussions
url: https://discord.com/invite/DD7vgKK299
about: Hang out with the community of LLM practitioners and resolve your issues fast
- name: Get on a Call
url: https://calendly.com/rohit-portkey/noam
about: Get a tailored demo for your use cases
.github/ISSUE_TEMPLATE/feature_request.yml
+25
View File
@@ -0,0 +1,25 @@
name: Feature Request
description: Suggest a new provider to integrate, new features, or something more
title: "[Feature] "
labels: ["enhancement"]
body:
- type: textarea
id: feature
attributes:
label: What Would You Like to See with the Gateway?
validations:
required: true
- type: textarea
id: context
attributes:
label: Context for your Request
description: Why you want this feature and how it beneits you.
validations:
required: false
- type: input
id: contact
attributes:
label: Your Twitter/LinkedIn
description: If we work on this request, we'd like to thank you publicly for suggesting it.
validations:
required: false
.github/README.cn.md
+230
View File
@@ -0,0 +1,230 @@
<div align="center">
<img src="/docs/images/gateway-border.png" width=350>
<p align="right">
<a href="../README.md">English</a> | <strong>中文</strong> | <a href="./README.jp.md">日本語</a>
</p>
# AI Gateway
### 通过一个快速友好的API链接超过100个大型语言模型。
[![License](https://img.shields.io/github/license/Ileriayo/markdown-badges)](./LICENSE)
[![Discord](https://img.shields.io/discord/1143393887742861333)](https://portkey.ai/community)
[![Twitter](https://img.shields.io/twitter/url/https/twitter/follow/portkeyai?style=social&label=Follow%20%40PortkeyAI)](https://twitter.com/portkeyai)
[![npm version](https://badge.fury.io/js/%40portkey-ai%2Fgateway.svg)](https://www.npmjs.com/package/@portkey-ai/gateway)
<!-- ![example workflow](https://github.com/github/docs/actions/workflows/main.yml/badge.svg) -->
</div>
<br><br>
[Portkey的AI网关](https://portkey.ai/features/ai-gateway) 是您的应用程序与托管的大型语言模型(LLMs)之间的接口。它通过统一的API简化了对OpenAI、Anthropic、Mistral、LLama2、Anyscale、Google Gemini等的API请求。
✅ 极速响应（快9.9倍），占用空间极小（安装后约45kb）<br>✅ 跨多个模型、提供商和密钥进行**负载均衡**<br>✅ 通过**备用方案**确保应用程序的稳定性<br>✅ 默认提供具有指数级备用方案的**自动重试**<br>✅ 根据需要插入中间件<br>✅ 经过超过**1000亿令牌**的实战测试<br> <br>
## 入门指南
### 安装
如果您熟悉Node.js和`npx`，您可以在本地运行您的私有AI网关。
```
npx @portkey-ai/gateway
```
<sup>
[ 其它部署选项 ]
&nbsp; <a href="https://portkey.wiki/gh-18"><img height="12" width="12" src="https://cfassets.portkey.ai/logo/dew-color.svg" /> Portkey Cloud 官方部署(推荐)</a>
&nbsp; <a href="../docs/installation-deployments.md#docker"><img height="12" width="12" src="https://cdn.simpleicons.org/docker/3776AB" /> Docker</a>
&nbsp; <a href="../docs/installation-deployments.md#nodejs-server"><img height="12" width="12" src="https://cdn.simpleicons.org/node.js/3776AB" /> Node.js</a>
&nbsp; <a href="../docs/installation-deployments.md#cloudflare-workers"><img height="12" width="12" src="https://cdn.simpleicons.org/cloudflare/3776AB" /> Cloudflare</a>
&nbsp; <a href="../docs/installation-deployments.md#replit"><img height="12" width="12" src="https://cdn.simpleicons.org/replit/3776AB" /> Replit</a>
&nbsp; <a href="../docs/installation-deployments.md"> 其它...</a>
</sup>
> 您的AI网关现在运行在 [http://localhost:8787](http://localhost:8787/) 🚀 <br>
### 使用方法
让我们尝试通过AI网关向OpenAI发起一个**聊天**请求：
```
bashCopy codecurl '127.0.0.1:8787/v1/chat/completions' \
-H 'x-portkey-provider: openai' \
-H "Authorization: Bearer $OPENAI_KEY" \
-H 'Content-Type: application/json' \
-d '{"messages": [{"role": "user","content": "Say this is test."}], "max_tokens": 20, "model": "gpt-4"}'
```
[支持的SDK完整列表](#supported-sdks)
<br>
## 支持的AI厂商
|| AI厂商 | 支持 | 流式 | 支持的端点 |
|---|---|---|---|--|
| <img src="docs/images/openai.png" width=25 />| OpenAI | ✅ |✅ | `/completions`, `/chat/completions`,`/embeddings`, `/assistants`, `/threads`, `/runs` |
| <img src="docs/images/azure.png" width=25>| Azure OpenAI | ✅ |✅ | `/completions`, `/chat/completions`,`/embeddings` |
| <img src="docs/images/anyscale.png" width=25>| Anyscale | ✅ | ✅ | `/chat/completions` |
| <img src="https://upload.wikimedia.org/wikipedia/commons/2/2d/Google-favicon-2015.png" width=25>| Google Gemini & Palm | ✅ |✅ | `/generateMessage`, `/generateText`, `/embedText` |
| <img src="docs/images/anthropic.png" width=25>| Anthropic | ✅ |✅ | `/messages`, `/complete` |
| <img src="docs/images/cohere.png" width=25>| Cohere | ✅ |✅ | `/generate`, `/embed`, `/rerank` |
| <img src="https://assets-global.website-files.com/64f6f2c0e3f4c5a91c1e823a/654693d569494912cfc0c0d4_favicon.svg" width=25>| Together AI | ✅ |✅ | `/chat/completions`, `/completions`, `/inference` |
| <img src="https://www.perplexity.ai/favicon.svg" width=25>| Perplexity | ✅ |✅ | `/chat/completions` |
| <img src="https://docs.mistral.ai/img/favicon.ico" width=25>| Mistral | ✅ |✅ | `/chat/completions`, `/embeddings` |
> [在这里查看支持的100多个模型的完整列表](https://portkey.ai/docs/welcome/what-is-portkey#ai-providers-supported)
<br />
## 特点
<table>
<tr>
<td>
<h4><a href="https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/universal-api">统一API签名</a></h4>
使用OpenAI的API签名连接100多个LLM。AI网关处理请求、响应和错误转换，因此您无需对代码进行任何更改。您可以使用OpenAI SDK本身连接到任何支持的LLM。
<br><br>
<img src="docs/images/openai.png" height=40 />&nbsp;&nbsp;&nbsp;<img src="docs/images/azure.png" height=40 />&nbsp;&nbsp;&nbsp;
<img src="docs/images/anyscale.png" height=40 />&nbsp;&nbsp;&nbsp;
<img src="https://upload.wikimedia.org/wikipedia/commons/2/2d/Google-favicon-2015.png" height=40 />&nbsp;&nbsp;&nbsp;<br><br>
<img src="docs/images/anthropic.png" height=40 />&nbsp;&nbsp;&nbsp;
<img src="docs/images/cohere.png" height=40 />&nbsp;&nbsp;&nbsp;
<img src="https://assets-global.website-files.com/64f6f2c0e3f4c5a91c1e823a/654693d569494912cfc0c0d4_favicon.svg" height=40 />&nbsp;&nbsp;&nbsp;<br><br>
<img src="https://www.perplexity.ai/favicon.svg" height=40 />&nbsp;&nbsp;&nbsp;
<img src="https://docs.mistral.ai/img/favicon.ico" height=40 />&nbsp;&nbsp;&nbsp;
<img src="https://1000logos.net/wp-content/uploads/2021/10/logo-Meta.png" height=40 />
<br><br>
</td>
<td>
<h4><a href="https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/fallbacks">备用方案</a></h4>
不要让失败阻止您。备用功能允许您按优先顺序指定语言模型API（LLMs）列表。如果主LLM无法响应或遇到错误，Portkey将自动备用到列表中的下一个LLM，确保您的应用程序的稳定性和可靠性。
<br><br>
<img src="https://framerusercontent.com/images/gmlOW8yeKP2pGuIsObM6gKLzeMI.png" height=200 />
</td>
</tr>
</table>
<table>
<tr>
<td>
<h4><a href="https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/automatic-retries">自动重试</a></h4>
临时问题不应该意味着手动重新运行。AI网关可以自动重试失败的请求多达5次。我们采用指数退避策略，间隔重试尝试以防止网络过载。
<br><br>
<img src="https://github.com/roh26it/Rubeus/assets/971978/8a6e653c-94b2-4ba7-95c7-93544ee476b1" height=200 />
</td>
<td>
<h4><a href="https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/load-balancing">负载均衡</a></h4>
根据自定义权重在多个API密钥或提供商之间有效分配负载。这确保了您的生成式AI应用程序的高可用性和最佳性能，防止任何单一LLM成为性能瓶颈。
<br><br>
<img src="https://framerusercontent.com/images/6EWuq3FWhqrPe3kKLqVspevi4.png" height=200 />
</td>
</tr>
</table>
<br>
## 配置 AI 网关
AI 网关支持[配置](https://portkey.ai/docs/api-reference/config-object)，以实现如**后备（fallbacks）**、**负载均衡（load balancing）**、**重试（retries）**等多样化的路由策略。
<br><br>
您可以在通过 `x-portkey-config` 头部进行 OpenAI 调用时使用这些配置
```js
// 使用 OpenAI JS SDK
const client = new OpenAI({
baseURL: "http://127.0.0.1:8787", // 网关 URL
defaultHeaders: {
'x-portkey-config': {.. 你的配置在这里 ..},
}
});
```
<br>
<details><summary>这里有一个示例配置，在回退到 Gemini Pro 之前会重试 OpenAI 请求 5 次</summary>
```js
{
"retry": { "count": 5 },
"strategy": { "mode": "fallback" },
"targets": [{
"provider": "openai",
"api_key": "sk-***"
},{
"provider": "google",
"api_key": "gt5***",
"override_params": {"model": "gemini-pro"}
}]
}
```
</details> <details> <summary>此配置将使得在 2 个 OpenAI 密钥之间实现等量的负载均衡</summary>
```js
{
"strategy": { "mode": "loadbalance" },
"targets": [{
"provider": "openai",
"api_key": "sk-***",
"weight": "0.5"
},{
"provider": "openai",
"api_key": "sk-***",
"weight": "0.5"
}
]
}
```
</details>
了解更多关于配置对象。
<br>
## 支持的SDKs
| 语言 | 支持的SDKs |
|---|---|
| Node.js / JS / TS | [Portkey SDK](https://www.npmjs.com/package/portkey-ai) <br> [OpenAI SDK](https://www.npmjs.com/package/openai) <br> [LangchainJS](https://www.npmjs.com/package/langchain) <br> [LlamaIndex.TS](https://www.npmjs.com/package/llamaindex) |
| Python | [Portkey SDK](https://pypi.org/project/portkey-ai/) <br> [OpenAI SDK](https://pypi.org/project/openai/) <br> [Langchain](https://pypi.org/project/langchain/) <br> [LlamaIndex](https://pypi.org/project/llama-index/) |
| Go | [go-openai](https://github.com/sashabaranov/go-openai) |
| Java | [openai-java](https://github.com/TheoKanning/openai-java) |
| Rust | [async-openai](https://docs.rs/async-openai/latest/async_openai/) |
| Ruby | [ruby-openai](https://github.com/alexrudall/ruby-openai) |
<br>
## 部署 AI 网关
[查看文档](docs/installation-deployments.md)了解如何在本地安装 AI 网关或者在流行的平台上部署它。
<br>
## 路线图
1. 支持更多的提供商。如果缺少某个提供商或 LLM 平台，请[提出功能请求](https://github.com/Portkey-AI/gateway/issues)。
2. 增强的负载均衡功能，以优化不同模型和提供商之间的资源使用。
3. 更加健壮的后备和重试策略，以进一步提高请求的可靠性。
4. 增加统一 API 签名的可定制性，以满足更多样化的使用案例。
[💬 在这里参与路线图讨论。](https://github.com/Portkey-AI/gateway/projects/)
<br>
## 贡献
最简单的贡献方式是选择任何带有 `good first issue` 标签的问题 💪。在[这里](./CONTRIBUTING.md)阅读贡献指南。
发现 Bug？[在这里提交](https://github.com/Portkey-AI/gateway/issues) | 有功能请求？[在这里提交](https://github.com/Portkey-AI/gateway/issues)
<br>
## 社区
加入我们不断增长的全球社区，寻求帮助，分享想法，讨论 AI。
- 查看我们的官方[博客](https://portkey.ai/blog)
- 在 [Discord](https://portkey.ai/community) 上与我们实时交流
- 在 [Twitter](https://twitter.com/PortkeyAI) 上关注我们
- 在 [LinkedIn](https://www.linkedin.com/company/portkey-ai/) 上与我们建立联系
- 阅读日文版文档 [日本語](./README.jp.md)
<!-- - 在 [YouTube](https://www.youtube.com/channel/UCZph50gLNXAh1DpmeX8sBdw) 上访问我们 --> <!-- - 加入我们的 [Dev 社区](https://dev.to/portkeyai) --> <!-- - 在 [Stack Overflow](https://stackoverflow.com/questions/tagged/portkey) 上查看标记为 #portkey 的问题 -->
![Rubeus Social Share (4)](https://github.com/Portkey-AI/gateway/assets/971978/89d6f0af-a95d-4402-b451-14764c40d03f)
.github/README.jp.md
+297
View File
@@ -0,0 +1,297 @@
<div align="center">
<p align="right">
<a href="../README.md">English</a> | <a href="./README.cn.md">中文</a> | <strong>日本語</strong>
</p>
# AIゲートウェイ
#### 1つの高速でフレンドリーなAPIで200以上のLLMに確実にルーティング
<img src="docs/images/demo.gif" width="650" alt="Gateway Demo"><br>
[![License](https://img.shields.io/github/license/Ileriayo/markdown-badges)](./LICENSE)
[![Discord](https://img.shields.io/discord/1143393887742861333)](https://portkey.ai/community)
[![Twitter](https://img.shields.io/twitter/url/https/twitter/follow/portkeyai?style=social&label=Follow%20%40PortkeyAI)](https://twitter.com/portkeyai)
[![npm version](https://badge.fury.io/js/%40portkey-ai%2Fgateway.svg)](https://www.npmjs.com/package/@portkey-ai/gateway)
[![Better Stack Badge](https://uptime.betterstack.com/status-badges/v1/monitor/q94g.svg)](https://status.portkey.ai/?utm_source=status_badge)
</div>
[AIゲートウェイ](https://portkey.ai/features/ai-gateway)は、250以上の言語、ビジョン、オーディオ、画像モデルへのリクエストを統一されたAPIで簡素化します。キャッシング、フォールバック、リトライ、タイムアウト、ロードバランシングをサポートし、最小の遅延でエッジデプロイが可能なプロダクション対応のゲートウェイです。
✅&nbsp; **超高速**（9.9倍速）で**小さなフットプリント**（ビルド後約100kb）<br>
✅&nbsp; 複数のモデル、プロバイダー、キー間で**ロードバランシング**<br>
✅&nbsp; **フォールバック**でアプリの信頼性を確保<br>
✅&nbsp; デフォルトで**自動リトライ**（指数関数的フォールバック）<br>
✅&nbsp; **リクエストタイムアウト**の設定が可能<br>
✅&nbsp; **マルチモーダル**でビジョン、TTS、STT、画像生成モデルをサポート<br>
✅&nbsp; 必要に応じてミドルウェアを**プラグイン**<br>
✅&nbsp; **480Bトークン**以上の実績<br>
✅&nbsp; **エンタープライズ対応**でセキュリティ、スケール、カスタムデプロイメントをサポート<br><br>
> [!TIP]
> ⭐️ **このリポジトリにスターを付ける**ことで、新しいプロバイダー統合や機能のGitHubリリース通知を受け取ることができます。
![star-2](https://github.com/user-attachments/assets/53597dce-6333-4ecc-a154-eb05532954e4)
<details>
<summary><kbd>スター履歴</kbd></summary>
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=portkey-ai%2Fgateway&theme=dark&type=Date">
<img width="100%" src="https://api.star-history.com/svg?repos=portkey-ai%2Fgateway&type=Date">
</picture>
</details>
<br>
## セットアップとインストール
AIゲートウェイを使用するには、**ホストされたAPI**を使用するか、**オープンソース**または**エンタープライズバージョン**を自分の環境にセルフホストします。
<br>
### 👉 portkey.aiでホストされたゲートウェイ（最速）
ホストされたAPIは、ジェネレーティブAIアプリケーションのためのAIゲートウェイをセットアップする最速の方法です。私たちは**毎日数十億のトークン**を処理しており、Postman、Haptik、Turing、MultiOn、SiteGPTなどの企業でプロダクションで使用されています。
<a href="https://app.portkey.ai/signup"><img src="https://portkey.ai/blog/content/images/2024/08/Get-API-Key--3-.png" height=50 alt="Get API Key" /></a><br>
<br>
### 👉 オープンソースバージョンのセルフホスト（[MITライセンス](https://github.com/Portkey-AI/gateway?tab=MIT-1-ov-file#readme)）
ローカルでAIゲートウェイを実行するには、ターミナルで以下のコマンドを実行します。（npxがインストールされている必要があります）または、[Cloudflare](https://github.com/Portkey-AI/gateway/blob/main/docs/installation-deployments.md#cloudflare-workers)、[Docker](https://github.com/Portkey-AI/gateway/blob/main/docs/installation-deployments.md#docker)、[Node.js](https://github.com/Portkey-AI/gateway/blob/main/docs/installation-deployments.md#nodejs-server)などのデプロイメントガイドを参照してください。
```bash
npx @portkey-ai/gateway
```
<sup>あなたのAIゲートウェイはhttp://localhost:8787で実行されています 🚀</sup>
<br>
### 👉 エンタープライズバージョンのセルフホスト
AIゲートウェイのエンタープライズバージョンは、**組織管理**、**ガバナンス**、**セキュリティ**などのエンタープライズ対応機能を提供します。オープンソース、ホスト、エンタープライズバージョンの比較は[こちら](https://docs.portkey.ai/docs/product/product-feature-comparison)をご覧ください。
エンタープライズデプロイメントアーキテクチャ、サポートされているプラットフォームについては、[**エンタープライズプライベートクラウドデプロイメント**](https://docs.portkey.ai/docs/product/enterprise-offering/private-cloud-deployments)をご覧ください。
<a href="https://portkey.sh/demo-22"><img src="https://portkey.ai/blog/content/images/2024/08/Get-API-Key--5-.png" height=50 alt="Book an enterprise AI gateway demo" /></a><br>
<br>
## AIゲートウェイを通じたリクエストの作成
### <img src="docs/images/openai.png" height=15 /> OpenAI API & SDKと互換性あり
AIゲートウェイはOpenAI API & SDKと互換性があり、200以上のLLMに信頼性のある呼び出しを拡張します。ゲートウェイを通じてOpenAIを使用するには、**クライアントを更新**してゲートウェイのURLとヘッダーを含め、通常通りリクエストを行います。AIゲートウェイは、OpenAI形式で書かれたリクエストを指定されたプロバイダーが期待するシグネチャに変換できます。[例を表示](https://docs.portkey.ai/docs/guides/getting-started/getting-started-with-ai-gateway)
<br><br>
### <img src="https://upload.wikimedia.org/wikipedia/commons/thumb/c/c3/Python-logo-notext.svg/1869px-Python-logo-notext.svg.png" height=15 /> Python SDKの使用 &nbsp;&nbsp;<a href="https://colab.research.google.com/drive/1hLvoq_VdGlJ_92sPPiwTznSra5Py0FuW?usp=sharing"><img src="https://colab.research.google.com/assets/colab-badge.svg"></a>
[Portkey Python SDK](https://github.com/Portkey-AI/portkey-python-sdk)は、OpenAI Python SDKのラッパーであり、他のすべてのプロバイダーに対する追加パラメータのサポートを提供します。**Pythonで構築している場合、これはゲートウェイに接続するための推奨ライブラリです**。
```bash
pip install -qU portkey-ai
```
<br>
### <img src="https://cdn-icons-png.flaticon.com/512/5968/5968322.png" height=15 /> Node.JS SDKの使用
[Portkey JS/TS SDK](https://www.npmjs.com/package/portkey-ai)は、OpenAI JS SDKのラッパーであり、他のすべてのプロバイダーに対する追加パラメータのサポートを提供します。**JSまたはTSで構築している場合、これはゲートウェイに接続するための推奨ライブラリです**。
```bash
npm install --save portkey-ai
```
<br>
### <img src="https://www.svgrepo.com/show/305922/curl.svg" height=15 /> REST APIの使用
AIゲートウェイは、すべての他のプロバイダーとモデルに対する追加パラメータのサポートを備えたOpenAI互換エンドポイントをサポートします。[APIリファレンスを表示](https://docs.portkey.ai/docs/api-reference/introduction)。
<br><br>
### その他の統合
| 言語 | サポートされているSDK |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| JS / TS | [LangchainJS](https://www.npmjs.com/package/langchain) <br> [LlamaIndex.TS](https://www.npmjs.com/package/llamaindex) |
| Python | <br> [Langchain](https://portkey.ai/docs/welcome/integration-guides/langchain-python) <br> [LlamaIndex](https://portkey.ai/docs/welcome/integration-guides/llama-index-python) |
| Go | [go-openai](https://github.com/sashabaranov/go-openai) |
| Java | [openai-java](https://github.com/TheoKanning/openai-java) |
| Rust | [async-openai](https://docs.rs/async-openai/latest/async_openai/) |
| Ruby | [ruby-openai](https://github.com/alexrudall/ruby-openai) |
<br>
## ゲートウェイクックブック
### トレンドのクックブック
- [Nvidia NIM](/cookbook/providers/nvidia.ipynb)のモデルをAIゲートウェイで使用する
- [CrewAIエージェント](/cookbook/monitoring-agents/CrewAI_with_Telemetry.ipynb)をPortkeyで監視する
- AIゲートウェイで[トップ10のLMSYSモデルを比較する](./use-cases/LMSYS%20Series/comparing-top10-LMSYS-models-with-Portkey.ipynb)
### 最新のクックブック
* [Nemotronを使用して合成データセットを作成する](/cookbook/use-cases/Nemotron_GPT_Finetuning_Portkey.ipynb)
* [PortkeyゲートウェイをVercelのAI SDKと使用する](/cookbook/integrations/vercel-ai.md)
* [PortkeyでLlamaエージェントを監視する](/cookbook/monitoring-agents/Llama_Agents_with_Telemetry.ipynb)
### [その他の例](https://github.com/Portkey-AI/gateway/tree/main/cookbook)
## サポートされているプロバイダー
[25以上のプロバイダー](https://portkey.ai/docs/welcome/integration-guides)と[6以上のフレームワーク](https://portkey.ai/docs/welcome/integration-guides)とのゲートウェイ統合を探索してください。
| | プロバイダー | サポート | ストリーム |
| -------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | ------- | ------ |
| <img src="docs/images/openai.png" width=35 /> | [OpenAI](https://portkey.ai/docs/welcome/integration-guides/openai) | ✅ | ✅ |
| <img src="docs/images/azure.png" width=35> | [Azure OpenAI](https://portkey.ai/docs/welcome/integration-guides/azure-openai) | ✅ | ✅ |
| <img src="docs/images/anyscale.png" width=35> | [Anyscale](https://portkey.ai/docs/welcome/integration-guides/anyscale-llama2-mistral-zephyr) | ✅ | ✅ |
| <img src="https://upload.wikimedia.org/wikipedia/commons/2/2d/Google-favicon-2015.png" width=35> | [Google Gemini & Palm](https://portkey.ai/docs/welcome/integration-guides/gemini) | ✅ | ✅ |
| <img src="docs/images/anthropic.png" width=35> | [Anthropic](https://portkey.ai/docs/welcome/integration-guides/anthropic) | ✅ | ✅ |
| <img src="docs/images/cohere.png" width=35> | [Cohere](https://portkey.ai/docs/welcome/integration-guides/cohere) | ✅ | ✅ |
| <img src="https://assets-global.website-files.com/64f6f2c0e3f4c5a91c1e823a/654693d569494912cfc0c0d4_favicon.svg" width=35> | [Together AI](https://portkey.ai/docs/welcome/integration-guides/together-ai) | ✅ | ✅ |
| <img src="https://www.perplexity.ai/favicon.svg" width=35> | [Perplexity](https://portkey.ai/docs/welcome/integration-guides/perplexity-ai) | ✅ | ✅ |
| <img src="https://docs.mistral.ai/img/favicon.ico" width=35> | [Mistral](https://portkey.ai/docs/welcome/integration-guides/mistral-ai) | ✅ | ✅ |
| <img src="https://docs.nomic.ai/img/nomic-logo.png" width=35> | [Nomic](https://portkey.ai/docs/welcome/integration-guides/nomic) | ✅ | ✅ |
| <img src="https://files.readme.io/d38a23e-small-studio-favicon.png" width=35> | [AI21](https://portkey.ai/docs/welcome/integration-guides) | ✅ | ✅ |
| <img src="https://platform.stability.ai/small-logo-purple.svg" width=35> | [Stability AI](https://portkey.ai/docs/welcome/integration-guides/stability-ai) | ✅ | ✅ |
| <img src="https://deepinfra.com/_next/static/media/logo.4a03fd3d.svg" width=35> | [DeepInfra](https://portkey.ai/docs/welcome/integration-guides) | ✅ | ✅ |
| <img src="https://ollama.com/public/ollama.png" width=35> | [Ollama](https://portkey.ai/docs/welcome/integration-guides/ollama) | ✅ | ✅ |
| <img src="https://novita.ai/favicon.ico" width=35> | Novita AI | ✅ | ✅ | `/chat/completions`, `/completions` |
> [サポートされている200以上のモデルの完全なリストを表示](https://portkey.ai/docs/welcome/what-is-portkey#ai-providers-supported)
<br>
<br>
## エージェント
ゲートウェイは、人気のあるエージェントフレームワークとシームレスに統合されます。[ドキュメントを読む](https://docs.portkey.ai/docs/welcome/agents)。
| フレームワーク | 200以上のLLMを呼び出す | 高度なルーティング | キャッシング | ロギングとトレース* | オブザーバビリティ* | プロンプト管理* |
|------------------------------|--------|-------------|---------|------|---------------|-------------------|
| [Autogen](https://docs.portkey.ai/docs/welcome/agents/autogen) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [CrewAI](https://docs.portkey.ai/docs/welcome/agents/crewai) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [LangChain](https://docs.portkey.ai/docs/welcome/agents/langchain-agents) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Phidata](https://docs.portkey.ai/docs/welcome/agents/phidata) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Llama Index](https://docs.portkey.ai/docs/welcome/agents/llama-agents) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Control Flow](https://docs.portkey.ai/docs/welcome/agents/control-flow) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [独自のエージェントを構築する](https://docs.portkey.ai/docs/welcome/agents/bring-your-own-agents) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
<br>
*ホストされたアプリでのみ利用可能です。詳細なドキュメントは[こちら](https://docs.portkey.ai/docs/welcome/agents)をご覧ください。
## 機能
<table width=100%>
<tr>
<td width="50%">
<strong><a href="https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/fallbacks">フォールバック</a></strong><br/>
失敗したリクエストに対して別のプロバイダーやモデルにフォールバックします。トリガーするエラーを指定できます。アプリケーションの信頼性を向上させます。
<br><br>
<img src="https://framerusercontent.com/images/gmlOW8yeKP2pGuIsObM6gKLzeMI.png" height=100 />
</td>
<td width="50%">
<strong><a href="https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/automatic-retries">自動リトライ</a></strong><br/>
失敗したリクエストを最大5回自動的にリトライします。指数関数的バックオフ戦略により、リトライ試行の間隔を空けてネットワークの過負荷を防ぎます。
<br><br>
<img src="https://github.com/roh26it/Rubeus/assets/971978/8a6e653c-94b2-4ba7-95c7-93544ee476b1" height=100 />
</td>
</tr>
</table>
<table width="100%">
<tr>
<td width="50%">
<strong><a href="https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/load-balancing">ロードバランシング</a></strong><br/>
複数のAPIキーやAIプロバイダー間でLLMリクエストを重み付けして分散させ、高可用性と最適なパフォーマンスを確保します。
<br><br>
<img src="https://framerusercontent.com/images/6EWuq3FWhqrPe3kKLqVspevi4.png" height=100 />
</td>
<td width="50%">
<strong><a href="https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/request-timeouts">リクエストタイムアウト</a></strong></br><br/>
応答しないLLMリクエストを自動的に終了させるために、詳細なリクエストタイムアウトを設定します。
<br><br>
<img src="https://github.com/vrushankportkey/gateway/assets/134934501/b23b98b2-6451-4747-8898-6847ad8baed4" height=100 />
</td>
</tr>
</table>
</table>
<table width="100%">
<tr>
<td width="50%">
<strong><a href="https://docs.portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/multimodal-capabilities">マルチモーダルLLMゲートウェイ</a></strong><br/>
ビジョン、オーディオ（テキストから音声、音声からテキスト）、画像生成モデルを複数のプロバイダーから呼び出すことができます — すべてOpenAIのシグネチャを使用して
<br><br>
<img src="https://docs.portkey.ai/~gitbook/image?url=https%3A%2F%2F2878743244-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252Fy3MCfQqftZOnHqSmVV5x%252Fuploads%252FOVuOxN4uFdBp1BdXX4E6%252Fmultimodal-icon.png%3Falt%3Dmedia%26token%3Db8b7bd49-0194-4d2f-89d4-c6633a872372&width=768&dpr=2&quality=100&sign=f51129a9&sv=1" height=100 />
</td>
<td width="50%">
<strong><a href="https://docs.portkey.ai/docs/product/guardrails">ガードレール</a></strong></br><br/>
指定されたチェックに従ってLLMの入力と出力をリアルタイムで検証します。独自のチェックを作成するか、20以上の事前構築されたガードレールから選択できます。
<br><br>
<img src="https://docs.portkey.ai/~gitbook/image?url=https%3A%2F%2F2878743244-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252Fy3MCfQqftZOnHqSmVV5x%252Fuploads%252FDFkhZpqtBfQMIW9BhVum%252Fguardrails-icon.png%3Falt%3Dmedia%26token%3D91cfe226-5ce9-44b3-a0e8-be9f3ae3917f&width=768&dpr=2&quality=100&sign=73608afc&sv=1" height=100 />
</td>
</tr>
</table>
**これらの機能は、`x-portkey-config`ヘッダーまたはSDKの`config`パラメータに追加されたゲートウェイ設定を通じて構成されます。**
以下は、上記の機能を示すサンプル設定JSONです。すべての機能はオプションです。
```json
{
"retry": { "attempts": 5 },
"request_timeout": 10000,
"strategy": { "mode": "fallback" }, // または 'loadbalance' など
"targets": [{
"provider": "openai",
"api_key": "sk-***"
},{
"strategy": {"mode": "loadbalance"}, // オプションのネスト
"targets": {...}
}]
}
```
次に、APIリクエストに設定を使用します。
### ゲートウェイ設定の使用
リクエストで設定オブジェクトを使用する方法については、[こちらのガイド](https://portkey.ai/docs/api-reference/config-object)をご覧ください。
<br>
## ゲートウェイエンタープライズバージョン
AIアプリを<ins>信頼性</ins>と<ins>将来の互換性</ins>を高め、完全な<ins>データセキュリティ</ins>と<ins>プライバシー</ins>を確保します。
✅&nbsp; セキュアなキー管理 - ロールベースのアクセス制御とトラッキングのため<br>
✅&nbsp; シンプルでセマンティックなキャッシング - 繰り返しのクエリを高速に提供し、コストを削減<br>
✅&nbsp; アクセス制御とインバウンドルール - 接続できるIPと地域を制御<br>
✅&nbsp; PII削除 - リクエストから自動的に機密データを削除し、意図しない露出を防止<br>
✅&nbsp; SOC2、ISO、HIPAA、GDPRコンプライアンス - ベストセキュリティプラクティスのため<br>
✅&nbsp; プロフェッショナルサポート - 機能の優先順位付けとともに<br>
[エンタープライズデプロイメントについての相談を予約する](https://portkey.sh/demo-22)
<br>
## 貢献
最も簡単な貢献方法は、`good first issue`タグの付いた問題を選ぶことです 💪。貢献ガイドラインは[こちら](/.github/CONTRIBUTING.md)をご覧ください。
バグ報告？[こちらで提出](https://github.com/Portkey-AI/gateway/issues) | 機能リクエスト？[こちらで提出](https://github.com/Portkey-AI/gateway/issues)
<br>
## コミュニティ
世界中の成長するコミュニティに参加して、AIに関するヘルプ、アイデア、ディスカッションを行いましょう。
- 公式[ブログ](https://portkey.ai/blog)を閲覧する
- [Discord](https://portkey.ai/community)でリアルタイムチャット
- [Twitter](https://twitter.com/PortkeyAI)でフォロー
- [LinkedIn](https://www.linkedin.com/company/portkey-ai/)で接続
- [日本語のドキュメント](./.github/README.jp.md)を読む
<!-- - [YouTube](https://www.youtube.com/channel/UCZph50gLNXAh1DpmeX8sBdw)で訪問 -->
<!-- - [Devコミュニティ](https://dev.to/portkeyai)に参加 -->
<!-- - [Stack Overflow](https://stackoverflow.com/questions/tagged/portkey)で#portkeyタグの質問を閲覧 -->
![Rubeus Social Share (4)](https://github.com/Portkey-AI/gateway/assets/971978/89d6f0af-a95d-4402-b451-14764c40d03f)
.github/SECURITY.md
+11
View File
@@ -0,0 +1,11 @@
# Security Policy
## Supported Versions
| Version | Supported |
| ------- | ------------------ |
| 1.x.x | :white_check_mark: |
## Reporting a Vulnerability
Please report any security vulnerabilities at support@portkey.ai
.github/SUPPORT.md
+6
View File
@@ -0,0 +1,6 @@
## How to file issues and get help
This project uses GitHub Issues to track bugs and feature requests. Please search the existing issues before filing new issues to avoid duplicates.
For new issues, file your bug or feature request as a new Issue.
For help and questions about using this project, please contact `support@portkey.ai`. Join the community discussions [here](https://discord.com/invite/DD7vgKK299).
.github/cn.CODE_OF_CONDUCT.md
+92
View File
@@ -0,0 +1,92 @@
<p align="right">
<a href=".github\CODE_OF_CONDUCT.md">English</a>|<strong>中文</strong>
</p>
# 贡献者公约行为准则
## 我们的承诺
我们作为成员、贡献者和领导者，承诺参与我们的
社区对每个人来说是一个无骚扰的体验，无论他们的年龄、体型、可见或不可见的残疾、种族、性特征、性别认同和表达、经验水平、教育、社会经济地位、国籍、个人外表、种族、宗教或性别认同和取向。
我们承诺以促进开放、欢迎、多元、包容和健康的社区的方式行动和互动。
## 我们的标准
对于有助于我们社区积极环境的行为举例包括：
* 对他人表现出同理心和善意
* 尊重不同的意见、观点和经验
* 优雅地给予并接受建设性反馈
* 承担责任并向受我们错误影响的人道歉，
并从经验中学习
* 关注的不仅是我们作为个体的最佳利益，而是整个社区的最佳利益
不可接受的行为举例包括：
* 使用性语言或图像，以及任何形式的性关注或
进步
* 恶意挑衅、侮辱性或贬损性评论，以及个人或政治攻击
* 公开或私下骚扰
* 在未经明确许可的情况下发布他人的私人信息，如实际地址或电子邮件地址
* 其他在专业环境中可能被合理视为不适当的行为
## 执行责任
社区领导者负责澄清和执行我们可接受行为的标准，并将采取适当和公平的纠正措施以响应他们认为不当、威胁性、冒犯性或有害的任何行为。
社区领导者有权利和责任移除、编辑或拒绝不符合本行为准则的评论、提交、代码、wiki编辑、问题以及其他贡献，并在适当时候沟通调节决策的原因。
## 范围
本行为准则适用于所有社区空间，并且当个人在公共空间正式代表社区时也适用。代表我们的社区的例子包括使用官方电子邮件地址、通过官方社交媒体账户发布，或者在在线或离线活动中担任指定代表。
## 执行
滥用、骚扰或以其他方式不可接受的行为可向负责执行的社区领导者报告到
support@portkey.ai。
所有投诉将进行及时和公正的审查和调查。
所有社区领导者都有义务尊重任何事件报告者的隐私和安全。
## 执行指南
社区领导者将根据以下社区影响指南确定他们认为违反本行为准则的任何行为的后果：
### 1. 更正
**社区影响**：使用不当的语言或其他在社区中被认为不专业或不受欢迎的行为。
**后果**：社区领导者将发出私人书面警告，提供违规行为的性质的清晰说明以及解释为什么行为不当。可能会要求公开道歉。
### 2. 警告
**社区影响**：通过单一事件或一系列行动的违规行为。
**后果**：警告连同持续行为的后果。不得与涉事人员进行互动，包括与执行行为准则的人进行未经请求的互动，在指定时间内。这包括避免在社区空间以及像社交媒体这样的外部渠道的互动。违反这些条款可能会导致临时或永久禁止。
### 3. 临时禁止
**社区影响**：严重违反社区标准，包括持续不当行为。
**后果**：在指定时间内禁止与社区进行任何形式的互动或公开沟通。在此期间，不允许与涉事人员进行公开或私下互动，包括与执行行为准则的人进行未经请求的互动。违反这些条款可能会导致永久禁止。
### 4. 永久禁止
**社区影响**：展现违反社区标准的行为模式，包括持续不当行为，骚扰个体，或对个体或群体的侵犯性或贬低性行为。
**后果**：永久禁止在社区内进行任何形式的公共互动。
## 归属
本行为准则改编自 [Contributor Covenant][homepage]，
2.0版本，可在
https://www.contributor-covenant.org/version/2/0/code_of_conduct.html 查看。
社区影响指南的灵感来自 [Mozilla 的行为准则执行阶梯](https://github.com/mozilla/diversity)。
[homepage]: https://www.contributor-covenant.org
有关本行为准则的常见问题解答，请参见
https://www.contributor-covenant.org/faq。翻译版本可在
https://www.contributor-covenant.org/translations 查看。
.github/pull_request_template.md
+14
View File
@@ -0,0 +1,14 @@
**Description:** (required)
- Detailed change 1
- Detailed change 2
**Tests Run/Test cases added:** (required)
- [ ] Description of test case
**Type of Change:**
<!-- Put an 'x' in the boxes that apply -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Documentation update
- [ ] Refactoring (no functional changes)
.github/workflows/check_code_formatting.yml
+25
View File
@@ -0,0 +1,25 @@
name: Check Prettier Formatting
on:
pull_request:
branches:
- main
jobs:
check-formatting:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Setup Node.js
uses: actions/setup-node@v2
with:
node-version: '20.x'
- name: Install dependencies
run: npm install
- name: Check formatting
run: npm run format:check
.github/workflows/docker_publish.yml
+44
View File
@@ -0,0 +1,44 @@
name: Publish Docker image
on:
release:
types: [published]
jobs:
push_to_registry:
name: Push Docker image to Docker Hub
runs-on: ubuntu-latest
steps:
- name: Check out the repo
uses: actions/checkout@v4
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ secrets.DOCKER_ORGANISATION }}/gateway
tags: |
type=raw,value=latest
type=pep440,pattern={{version}},value=${{ github.event.release.tag_name }}
- name: Build and push Docker image
uses: docker/build-push-action@v5
with:
context: .
file: ./Dockerfile
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
.github/workflows/link-checker.yml
+51
View File
@@ -0,0 +1,51 @@
name: Check Markdown links
on:
push:
paths:
- '**/*.md' # Only run when markdown files change
pull_request:
branches:
- main
schedule:
- cron: '0 0 * * 0' # Run weekly on Sundays
workflow_dispatch: # Allows manual triggering
jobs:
linkChecker:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Link Checker
uses: lycheeverse/lychee-action@v1.8.0
with:
args: --verbose --no-progress './**/*.md'
fail: true
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Create Issue If Failed
if: failure()
uses: actions/github-script@v6
with:
script: |
const title = '🔗 Broken links found in documentation';
const body = 'The link checker found broken links in the documentation. Please check the [workflow run](${process.env.GITHUB_SERVER_URL}/${process.env.GITHUB_REPOSITORY}/actions/runs/${process.env.GITHUB_RUN_ID}) for details.';
const existingIssues = await github.rest.issues.listForRepo({
owner: context.repo.owner,
repo: context.repo.repo,
labels: 'documentation,broken-links',
});
const issueExists = existingIssues.data.some(issue => issue.title === title);
if (!issueExists) {
await github.rest.issues.create({
owner: context.repo.owner,
repo: context.repo.repo,
title: title,
body: body,
labels: ['documentation', 'broken-links']
});
}
.github/workflows/npm_publish.yml
+21
View File
@@ -0,0 +1,21 @@
name: Publish to NPM
on:
release:
types: [published]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Setup Node
uses: actions/setup-node@v3
with:
node-version: '20.x'
registry-url: 'https://registry.npmjs.org'
- name: Install dependencies
run: npm ci
- name: Publish package on NPM
run: npm publish
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
.github/workflows/run_tests.yml
+75
View File
@@ -0,0 +1,75 @@
name: Gateway Tests
on:
issue_comment:
types: [created]
jobs:
gateway-tests:
if: ${{ github.event.issue.pull_request && contains(github.event.comment.body, 'run tests') }}
runs-on: ubuntu-latest
environment: production
steps:
- name: Checkout head
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Install dependencies
run: npm ci
- name: Build
run: npm run build
- name: Start gateway and run tests
id: run-tests
continue-on-error: true
run: |
npm run build/start-server.js &
echo "Waiting for gateway to start..."
while ! curl -s http://localhost:8787 > /dev/null; do
sleep 1
done
echo "Gateway is ready. Running tests..."
npm run test:gateway
- name: Update PR Check
uses: actions/github-script@v6
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
const { owner, repo } = context.repo;
const issue_number = context.issue.number;
try {
const { data: pull_request } = await github.rest.pulls.get({
owner,
repo,
pull_number: issue_number,
});
const sha = pull_request.head.sha;
await github.rest.checks.create({
owner,
repo,
name: 'Gateway Tests (Comment Triggered)',
head_sha: sha,
status: 'completed',
conclusion: '${{ steps.run-tests.outcome }}',
output: {
title: 'Gateway Test Results',
summary: 'Gateway tests have completed.',
text: 'These tests were triggered by a comment on the PR.'
},
});
console.log('Check run created successfully');
} catch (error) {
console.error('Error creating check run:', error);
core.setFailed(`Action failed with error: ${error}`);
}
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
COHERE_API_KEY: ${{ secrets.COHERE_API_KEY }}
.github/workflows/triage-label.yml
+26
View File
@@ -0,0 +1,26 @@
name: Auto Triage Label
on:
issues:
types: [opened]
jobs:
label_issues:
runs-on: ubuntu-latest
permissions:
issues: write
steps:
- uses: actions/checkout@v4.1.1
- name: Add Triage Label
uses: actions/github-script@v7.0.1
with:
script: |
github.rest.issues.addLabels({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
labels: ['triage']
})
github-token: ${{secrets.GITHUB_TOKEN}}
.gitignore
+149
View File
@@ -0,0 +1,149 @@
# Local configuration file
conf.json
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
lerna-debug.log*
.pnpm-debug.log*
# Diagnostic reports (https://nodejs.org/api/report.html)
report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json
# Runtime data
pids
*.pid
*.seed
*.pid.lock
# Directory for instrumented libs generated by jscoverage/JSCover
lib-cov
# Coverage directory used by tools like istanbul
coverage
*.lcov
# nyc test coverage
.nyc_output
# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
.grunt
# Bower dependency directory (https://bower.io/)
bower_components
# node-waf configuration
.lock-wscript
# Compiled binary addons (https://nodejs.org/api/addons.html)
build/Release
# Dependency directories
node_modules/
jspm_packages/
# Snowpack dependency directory (https://snowpack.dev/)
web_modules/
# TypeScript cache
*.tsbuildinfo
# Optional npm cache directory
.npm
# Optional eslint cache
.eslintcache
# Optional stylelint cache
.stylelintcache
# Microbundle cache
.rpt2_cache/
.rts2_cache_cjs/
.rts2_cache_es/
.rts2_cache_umd/
# Optional REPL history
.node_repl_history
# Output of 'npm pack'
*.tgz
# Yarn Integrity file
.yarn-integrity
# dotenv environment variable files
.env
.env.development.local
.env.test.local
.env.production.local
.env.local
# parcel-bundler cache (https://parceljs.org/)
.cache
.parcel-cache
# Next.js build output
.next
out
# Nuxt.js build / generate output
.nuxt
dist
# Gatsby files
.cache/
# Comment in the public line in if your project uses Gatsby and not Next.js
# https://nextjs.org/blog/next-9-1#public-directory-support
# public
# vuepress build output
.vuepress/dist
# vuepress v2.x temp and cache directory
.temp
.cache
# Docusaurus cache and generated files
.docusaurus
# Serverless directories
.serverless/
# FuseBox cache
.fusebox/
# DynamoDB Local files
.dynamodb/
# TernJS port file
.tern-port
# Stores VSCode versions used for testing VSCode extensions
.vscode-test
# yarn v2
.yarn/cache
.yarn/unplugged
.yarn/build-state.yml
.yarn/install-state.gz
.pnp.*
.dev.vars
# Rollup build dir
build
.DS_Store
# Wrangler temp directory
.wrangler
# IntelliJ Idea
.idea
plugins/**/.creds.json
plugins/**/creds.json
plugins/**/.parameters.json
src/handlers/tests/.creds.json
.cursor/
.husky/pre-commit
+1
View File
@@ -0,0 +1 @@
npm run format:check || (npm run format && return 1)
.husky/pre-push
+1
View File
@@ -0,0 +1 @@
npm run pre-push
.prettierignore
+6
View File
@@ -0,0 +1,6 @@
node_modules
package-lock.json
build
.wrangler
public
wrangler.toml
.prettierrc
+10
View File
@@ -0,0 +1,10 @@
{
"semi": true,
"tabWidth": 2,
"printWidth": 80,
"endOfLine": "lf",
"singleQuote": true,
"arrowParens": "always",
"bracketSpacing": true,
"trailingComma": "es5"
}
.vscode/launch.json
Vendored
+42
View File
@@ -0,0 +1,42 @@
{
/*
1. steps to run wrangler debugger (https://blog.cloudflare.com/debugging-cloudflare-workers/)
- run `npm run dev` in a terminal
- in debug and run tab, select 'Wrangler debug' and run
- add a breakpoint and verify
2. Steps tp run node debugger
- In debug tab select "Node debug" and run
- add a breakpoint and verify
*/
"version": "0.2.0",
"configurations": [
{
"name": "Wrangler debug",
"type": "node",
"request": "attach",
"port": 9229,
"cwd": "/",
"resolveSourceMapLocations": null,
"attachExistingChildren": false,
"autoAttachChildProcesses": false
},
{
"type": "node",
"request": "launch",
"name": "Node debug",
"runtimeExecutable": "tsx",
"runtimeArgs": ["${workspaceFolder}/build/src/start-server.js"],
"sourceMaps": true,
"outFiles": ["${workspaceFolder}/build/**/*.js"],
"sourceMapPathOverrides": {
"../src/*": "${workspaceFolder}/src/*"
},
"skipFiles": ["<node_internals>/**"],
"preLaunchTask": "npx tsc",
"console": "integratedTerminal",
"internalConsoleOptions": "neverOpen",
"postDebugTask": "cleanup build"
}
]
}
.vscode/tasks.json
Vendored
+36
View File
@@ -0,0 +1,36 @@
{
// See https://go.microsoft.com/fwlink/?LinkId=733558
// for the documentation about the tasks.json format
"version": "2.0.0",
"tasks": [
{
"type": "shell",
"label": "npx tsc",
"command": "npx",
"args": [
"tsc",
"--sourcemap",
"true",
"--outDir",
"${workspaceFolder}/build"
],
"problemMatcher": ["$tsc"],
"group": {
"kind": "build",
"isDefault": true
}
},
{
"type": "shell",
"label": "cleanup build",
"command": "rm",
"args": ["-rf", "${workspaceFolder}/build"],
"problemMatcher": [],
"group": "build",
"presentation": {
"reveal": "always",
"panel": "new"
}
}
]
}
CLAUDE.md
+95
View File
@@ -0,0 +1,95 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
This is the **Portkey AI Gateway** - a fast, reliable AI gateway that routes requests to 250+ LLMs with sub-1ms latency. It's built with Hono framework for TypeScript/JavaScript and can be deployed to multiple environments including Cloudflare Workers, Node.js servers, and Docker containers.
## Development Commands
### Core Development
- `npm run dev` - Start development server using Wrangler (Cloudflare Workers)
- `npm run dev:node` - Start development server using Node.js
- `npm run build` - Build the project for production
- `npm run build-plugins` - Build the plugin system
### Testing
- `npm run test:gateway` - Run tests for the main gateway code (src/)
- `npm run test:plugins` - Run tests for plugins
- `jest src/` - Run specific gateway tests
- `jest plugins/` - Run specific plugin tests
### Code Quality
- `npm run format` - Format code with Prettier
- `npm run format:check` - Check code formatting
- `npm run pretty` - Alternative format command
### Deployment
- `npm run deploy` - Deploy to Cloudflare Workers
- `npm run start:node` - Start production Node.js server
## Architecture
### Core Components
**Main Application (`src/index.ts`)**
- Hono-based HTTP server with middleware pipeline
- Handles multiple AI provider integrations
- Routes: `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, etc.
**Provider System (`src/providers/`)**
- Modular provider implementations (OpenAI, Anthropic, Azure, etc.)
- Each provider has standardized interface: `api.ts`, `chatComplete.ts`, `embed.ts`
- Provider configs define supported features and transformations
**Middleware Pipeline**
- `requestValidator` - Validates incoming requests
- `hooks` - Pre/post request hooks
- `memoryCache` - Response caching
- `logger` - Request/response logging
- `adminAuth` - Node-only local UI session auth for `/public/*` login endpoints and `/log/stream`
- `portkey` - Core Portkey-specific middleware for routing, guardrails, etc.
**Plugin System (`plugins/`)**
- Guardrail plugins for content filtering, PII detection, etc.
- Each plugin has `manifest.json` defining capabilities
- Plugins are built separately with `npm run build-plugins`
### Key Concepts
**Configs** - JSON configurations that define:
- Provider routing and fallbacks
- Load balancing strategies
- Guardrails and content filtering
- Caching and retry policies
**Handlers** - Route-specific request processors in `src/handlers/`
- Each AI API endpoint has dedicated handler
- Stream handling for real-time responses
- WebSocket support for realtime APIs
## File Structure
- `src/providers/` - AI provider integrations
- `src/handlers/` - API endpoint handlers
- `src/middlewares/` - Request/response middleware
- `plugins/` - Guardrail and validation plugins
- `cookbook/` - Example integrations and use cases
- `conf.json` - Runtime configuration
## Testing Strategy
Tests are organized by component:
- `src/tests/` - Core gateway functionality tests
- `src/handlers/__tests__/` - Handler-specific tests
- `plugins/*/**.test.ts` - Plugin tests
- Test timeout: 30 seconds (configured in jest.config.js)
## Configuration
The gateway uses `conf.json` for runtime configuration. Sample config available in `conf_sample.json`.
For local UI hardening, set `admin_token` in `conf.json`. The `/public` UI uses this token to establish an in-memory admin session, and `/log/stream` requires that session.
Key environment variables and configuration handled through Hono's adapter system for multi-environment deployment.
Dockerfile
+50
View File
@@ -0,0 +1,50 @@
# Use the official Node.js runtime as a parent image
FROM node:20-alpine AS build
# Set the working directory in the container
WORKDIR /app
# Copy package.json and package-lock.json to the working directory
COPY package*.json ./
COPY patches ./
# Upgrade system packages
RUN apk upgrade --no-cache
# Upgrade npm to version 10.9.2
RUN npm install -g npm@10.9.2
# Install app dependencies
RUN npm install
# Copy the rest of the application code
COPY . .
# Build the application and clean up
RUN npm run build \
&& rm -rf node_modules \
&& npm install --omit=dev
# Use the official Node.js runtime as a parent image
FROM node:20-alpine
# Upgrade system packages
RUN apk upgrade --no-cache
# Upgrade npm to version 10.9.2
RUN npm install -g npm@10.9.2
# Set the working directory in the container
WORKDIR /app
# Copy the build directory, node_modules, and package.json to the working directory
COPY --from=build /app/build /app/build
COPY --from=build /app/node_modules /app/node_modules
COPY --from=build /app/package.json /app/package.json
COPY --from=build /app/patches /app/patches
# Expose port 8787
EXPOSE 8787
ENTRYPOINT ["npm"]
CMD ["run", "start:node"]
LICENSE
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2024 Portkey, Inc
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
README.md
+338
View File
@@ -0,0 +1,338 @@
<p align="right">
<strong>English</strong> | <a href="./.github/README.cn.md">中文</a> | <a href="./.github/README.jp.md">日本語</a>
</p>
> [!IMPORTANT]
> :rocket: Gateway 2.0 (Pre-Release) Portkey's core enterprise gateway is merging into open-source with our 2.0 release. You can try the pre-release branch [here](https://github.com/portkey-ai/gateway/tree/2.0.0).
> Read more about what's next for Portkey in our [**Series A announcement**](https://portkey.wiki/rohit-a).
<div align="center">
🆕 **[Portkey Models](https://github.com/Portkey-AI/models)** - Open-source LLM pricing for 2,300+ models across 40+ providers. [Explore →](https://portkey.ai/models)
# AI Gateway
#### Route to 250+ LLMs with 1 fast & friendly API
<img src="https://cfassets.portkey.ai/sdk.gif" width="550px" alt="Portkey AI Gateway Demo showing LLM routing capabilities" style="margin-left:-35px">
[Docs](https://portkey.wiki/gh-1) | [Enterprise](https://portkey.wiki/gh-2) | [Hosted Gateway](https://portkey.wiki/gh-3) | [Changelog](https://portkey.wiki/gh-4) | [API Reference](https://portkey.wiki/gh-5)
[![License](https://img.shields.io/github/license/Ileriayo/markdown-badges)](./LICENSE)
[![Discord](https://img.shields.io/discord/1143393887742861333)](https://portkey.wiki/gh-6)
[![Twitter](https://img.shields.io/twitter/url/https/twitter/follow/portkeyai?style=social&label=Follow%20%40PortkeyAI)](https://portkey.wiki/gh-7)
[![npm version](https://badge.fury.io/js/%40portkey-ai%2Fgateway.svg)](https://portkey.wiki/gh-8)
[![Better Stack Badge](https://uptime.betterstack.com/status-badges/v1/monitor/q94g.svg)](https://portkey.wiki/gh-9)
<a href="https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/quickcreate?stackName=portkey-gateway&templateURL=https://portkey-gateway-ec2-quicklaunch.s3.us-east-1.amazonaws.com/portkey-gateway-ec2-quicklaunch.template.yaml"><img src="https://img.shields.io/badge/Deploy_to_EC2-232F3E?style=for-the-badge&logo=amazonwebservices&logoColor=white" alt="Deploy to AWS EC2" width="105"/></a> [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/Portkey-AI/gateway)
</div>
<br/>
The [**AI Gateway**](https://portkey.wiki/gh-10) is designed for fast, reliable & secure routing to 1600+ language, vision, audio, and image models. It is a lightweight, open-source, and enterprise-ready solution that allows you to integrate with any language model in under 2 minutes.
- [x] **Blazing fast** (<1ms latency) with a tiny footprint (122kb)
- [x] **Battle tested**, with over 10B tokens processed everyday
- [x] **Enterprise-ready** with enhanced security, scale, and custom deployments
<br>
#### What can you do with the AI Gateway?
- Integrate with any LLM in under 2 minutes - [Quickstart](#quickstart-2-mins)
- Prevent downtimes through **[automatic retries](https://portkey.wiki/gh-11)** and **[fallbacks](https://portkey.wiki/gh-12)**
- Scale AI apps with **[load balancing](https://portkey.wiki/gh-13)** and **[conditional routing](https://portkey.wiki/gh-14)**
- Protect your AI deployments with **[guardrails](https://portkey.wiki/gh-15)**
- Go beyond text with **[multi-modal capabilities](https://portkey.wiki/gh-16)**
- Explore **[agentic workflow](https://portkey.wiki/gh-17)** integrations
- Manage MCP servers with enterprise auth & observability using **[MCP Gateway](https://portkey.ai/docs/product/mcp-gateway)**
<br><br>
> [!TIP]
> Starring this repo helps more developers discover the AI Gateway 🙏🏻
>
> ![star-2](https://github.com/user-attachments/assets/53597dce-6333-4ecc-a154-eb05532954e4)
>
<br>
<br>
## Quickstart (2 mins)
### 1. Setup your AI Gateway
```bash
# Run the gateway locally (needs Node.js and npm)
npx @portkey-ai/gateway
```
> The Gateway is running on `http://localhost:8787/v1`
>
> The Gateway Console is running on `http://localhost:8787/public/`
<sup>
Deployment guides:
&nbsp; <a href="https://portkey.wiki/gh-18"><img height="12" width="12" src="https://cfassets.portkey.ai/logo/dew-color.svg" /> Portkey Cloud (Recommended)</a>
&nbsp; <a href="./docs/installation-deployments.md#docker"><img height="12" width="12" src="https://cdn.simpleicons.org/docker/3776AB" /> Docker</a>
&nbsp; <a href="./docs/installation-deployments.md#nodejs-server"><img height="12" width="12" src="https://cdn.simpleicons.org/node.js/3776AB" /> Node.js</a>
&nbsp; <a href="./docs/installation-deployments.md#cloudflare-workers"><img height="12" width="12" src="https://cdn.simpleicons.org/cloudflare/3776AB" /> Cloudflare</a>
&nbsp; <a href="./docs/installation-deployments.md#replit"><img height="12" width="12" src="https://cdn.simpleicons.org/replit/3776AB" /> Replit</a>
&nbsp; <a href="./docs/installation-deployments.md"> Others...</a>
</sup>
### 2. Make your first request
<!-- <details open>
<summary>Python Example</summary> -->
```python
# pip install -qU portkey-ai
from portkey_ai import Portkey
# OpenAI compatible client
client = Portkey(
provider="openai", # or 'anthropic', 'bedrock', 'groq', etc
Authorization="sk-***" # the provider API key
)
# Make a request through your AI Gateway
client.chat.completions.create(
messages=[{"role": "user", "content": "What's the weather like?"}],
model="gpt-4o-mini"
)
```
<sup>Supported Libraries:
&nbsp; [<img height="12" width="12" src="https://cdn.simpleicons.org/javascript/3776AB" /> JS](https://portkey.wiki/gh-19)
&nbsp; [<img height="12" width="12" src="https://cdn.simpleicons.org/python/3776AB" /> Python](https://portkey.wiki/gh-20)
&nbsp; [<img height="12" width="12" src="https://cdn.simpleicons.org/gnubash/3776AB" /> REST](https://portkey.sh/gh-84)
&nbsp; [<img height="12" width="12" src="https://cdn.simpleicons.org/openai/3776AB" /> OpenAI SDKs](https://portkey.wiki/gh-21)
&nbsp; [<img height="12" width="12" src="https://cdn.simpleicons.org/langchain/3776AB" /> Langchain](https://portkey.wiki/gh-22)
&nbsp; [LlamaIndex](https://portkey.wiki/gh-23)
&nbsp; [Autogen](https://portkey.wiki/gh-24)
&nbsp; [CrewAI](https://portkey.wiki/gh-25)
&nbsp; [More..](https://portkey.wiki/gh-26)
</sup>
On the Gateway Console (`http://localhost:8787/public/`) you can see all of your local logs in one place.
<img src="https://github.com/user-attachments/assets/362bc916-0fc9-43f1-a39e-4bd71aac4a3a" width="400" />
### 3. Routing & Guardrails
`Configs` in the LLM gateway allow you to create routing rules, add reliability and setup guardrails.
```python
config = {
"retry": {"attempts": 5},
"output_guardrails": [{
"default.contains": {"operator": "none", "words": ["Apple"]},
"deny": True
}]
}
# Attach the config to the client
client = client.with_options(config=config)
client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Reply randomly with Apple or Bat"}]
)
# This would always response with "Bat" as the guardrail denies all replies containing "Apple". The retry config would retry 5 times before giving up.
```
<div align="center">
<img src="https://portkey.ai/blog/content/images/size/w1600/2024/11/image-15.png" width=600 title="Request flow through Portkey's AI gateway with retries and guardrails" alt="Request flow through Portkey's AI gateway with retries and guardrails"/>
</div>
You can do a lot more stuff with configs in your AI gateway. [Jump to examples →](https://portkey.wiki/gh-27)
<br/>
### Enterprise Version (Private deployments)
<sup>
[<img height="12" width="12" src="https://cfassets.portkey.ai/amazon-logo.svg" /> AWS](https://portkey.wiki/gh-28)
&nbsp; [<img height="12" width="12" src="https://cfassets.portkey.ai/azure-logo.svg" /> Azure](https://portkey.wiki/gh-29)
&nbsp; [<img height="12" width="12" src="https://cdn.simpleicons.org/googlecloud/3776AB" /> GCP](https://portkey.wiki/gh-30)
&nbsp; [<img height="12" width="12" src="https://cdn.simpleicons.org/redhatopenshift/3776AB" /> OpenShift](https://portkey.wiki/gh-31)
&nbsp; [<img height="12" width="12" src="https://cdn.simpleicons.org/kubernetes/3776AB" /> Kubernetes](https://portkey.wiki/gh-85)
</sup>
The LLM Gateway's [enterprise version](https://portkey.wiki/gh-86) offers advanced capabilities for **org management**, **governance**, **security** and [more](https://portkey.wiki/gh-87) out of the box. [View Feature Comparison →](https://portkey.wiki/gh-32)
The enterprise deployment architecture for supported platforms is available here - [**Enterprise Private Cloud Deployments**](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture)
<a href="https://portkey.sh/demo-13"><img src="https://portkey.ai/blog/content/images/2024/08/Get-API-Key--5-.png" height=50 alt="Book an enterprise AI gateway demo" /></a><br/>
<br>
## MCP Gateway
[MCP Gateway](https://portkey.ai/docs/product/mcp-gateway) provides a centralized control plane for managing MCP (Model Context Protocol) servers across your organization.
- **Authentication** — Single auth layer at the gateway. Users authenticate once; your MCP servers receive verified requests
- **Access Control** — Control which teams and users can access which servers and tools. Revoke access instantly
- **Observability** — Every tool call logged with full context: who called what, parameters, response, latency
- **Identity Forwarding** — Forward user identity (email, team, roles) to MCP servers automatically
Works with Claude Desktop, Cursor, VS Code, and any MCP-compatible client. [Get started →](https://portkey.ai/docs/product/mcp-gateway/quickstart)
<br>
## Core Features
### Reliable Routing
- <a href="https://portkey.wiki/gh-37">**Fallbacks**</a>: Fallback to another provider or model on failed requests using the LLM gateway. You can specify the errors on which to trigger the fallback. Improves reliability of your application.
- <a href="https://portkey.wiki/gh-38">**Automatic Retries**</a>: Automatically retry failed requests up to 5 times. An exponential backoff strategy spaces out retry attempts to prevent network overload.
- <a href="https://portkey.wiki/gh-39">**Load Balancing**</a>: Distribute LLM requests across multiple API keys or AI providers with weights to ensure high availability and optimal performance.
- <a href="https://portkey.wiki/gh-40">**Request Timeouts**</a>: Manage unruly LLMs & latencies by setting up granular request timeouts, allowing automatic termination of requests that exceed a specified duration.
- <a href="https://portkey.wiki/gh-41">**Multi-modal LLM Gateway**</a>: Call vision, audio (text-to-speech & speech-to-text), and image generation models from multiple providers — all using the familiar OpenAI signature
- <a href="https://portkey.wiki/gh-42">**Realtime APIs**</a>: Call realtime APIs launched by OpenAI through the integrate websockets server.
### Security & Accuracy
- <a href="https://portkey.wiki/gh-88">**Guardrails**</a>: Verify your LLM inputs and outputs to adhere to your specified checks. Choose from the 40+ pre-built guardrails to ensure compliance with security and accuracy standards. You can <a href="https://portkey.wiki/gh-43">bring your own guardrails</a> or choose from our <a href="https://portkey.wiki/gh-44">many partners</a>.
- [**Secure Key Management**](https://portkey.wiki/gh-45): Use your own keys or generate virtual keys on the fly.
- [**Role-based access control**](https://portkey.wiki/gh-46): Granular access control for your users, workspaces and API keys.
- <a href="https://portkey.wiki/gh-47">**Compliance & Data Privacy**</a>: The AI gateway is SOC2, HIPAA, GDPR, and CCPA compliant.
### Cost Management
- [**Smart caching**](https://portkey.wiki/gh-48): Cache responses from LLMs to reduce costs and improve latency. Supports simple and semantic* caching.
- [**Usage analytics**](https://portkey.wiki/gh-49): Monitor and analyze your AI and LLM usage, including request volume, latency, costs and error rates.
- [**Provider optimization***](https://portkey.wiki/gh-89): Automatically switch to the most cost-effective provider based on usage patterns and pricing models.
### Collaboration & Workflows
- <a href="https://portkey.ai/docs/integrations/agents">**Agents Support**</a>: Seamlessly integrate with popular agent frameworks to build complex AI applications. The gateway seamlessly integrates with [Autogen](https://portkey.wiki/gh-50), [CrewAI](https://portkey.wiki/gh-51), [LangChain](https://portkey.wiki/gh-52), [LlamaIndex](https://portkey.wiki/gh-53), [Phidata](https://portkey.wiki/gh-54), [Control Flow](https://portkey.wiki/gh-55), and even [Custom Agents](https://portkey.wiki/gh-56).
- [**Prompt Template Management***](https://portkey.wiki/gh-57): Create, manage and version your prompt templates collaboratively through a universal prompt playground.
<br/><br/>
<sup>
*&nbsp;Available in hosted and enterprise versions
</sup>
<br>
## Portkey Models
Open-source LLM pricing database for 40+ providers - used by the Gateway for cost tracking.
[GitHub](https://github.com/Portkey-AI/models) | [Model Explorer](https://portkey.ai/models)
<br>
## Cookbooks
### ☄️ Trending
- Use models from [Nvidia NIM](/cookbook/providers/nvidia.ipynb) with AI Gateway
- Monitor [CrewAI Agents](/cookbook/monitoring-agents/CrewAI_with_Telemetry.ipynb) with Portkey!
- Comparing [Top 10 LMSYS Models](/cookbook/use-cases/LMSYS%20Series/comparing-top10-LMSYS-models-with-Portkey.ipynb) with AI Gateway.
### 🚨 Latest
* [Create Synthetic Datasets using Nemotron](/cookbook/use-cases/Nemotron_GPT_Finetuning_Portkey.ipynb)
* [Use the LLM Gateway with Vercel's AI SDK](/cookbook/integrations/vercel-ai.md)
* [Monitor Llama Agents with Portkey's LLM Gateway](/cookbook/monitoring-agents/Llama_Agents_with_Telemetry.ipynb)
[View all cookbooks →](https://portkey.wiki/gh-58)
<br/><br/>
## Supported Providers
Explore Gateway integrations with [45+ providers](https://portkey.wiki/gh-59) and [8+ agent frameworks](https://portkey.wiki/gh-90).
| | Provider | Support | Stream |
| -------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | ------- | ------ |
| <img src="docs/images/openai.png" width=35 /> | [OpenAI](https://portkey.wiki/gh-60) | ✅ | ✅ |
| <img src="docs/images/azure.png" width=35> | [Azure OpenAI](https://portkey.wiki/gh-61) | ✅ | ✅ |
| <img src="docs/images/anyscale.png" width=35> | [Anyscale](https://portkey.wiki/gh-62) | ✅ | ✅ |
| <img src="https://upload.wikimedia.org/wikipedia/commons/2/2d/Google-favicon-2015.png" width=35> | [Google Gemini](https://portkey.wiki/gh-63) | ✅ | ✅ |
| <img src="docs/images/anthropic.png" width=35> | [Anthropic](https://portkey.wiki/gh-64) | ✅ | ✅ |
| <img src="docs/images/cohere.png" width=35> | [Cohere](https://portkey.wiki/gh-65) | ✅ | ✅ |
| <img src="https://assets-global.website-files.com/64f6f2c0e3f4c5a91c1e823a/654693d569494912cfc0c0d4_favicon.svg" width=35> | [Together AI](https://portkey.wiki/gh-66) | ✅ | ✅ |
| <img src="https://www.perplexity.ai/favicon.svg" width=35> | [Perplexity](https://portkey.wiki/gh-67) | ✅ | ✅ |
| <img src="https://docs.mistral.ai/img/favicon.ico" width=35> | [Mistral](https://portkey.wiki/gh-68) | ✅ | ✅ |
| <img src="https://docs.nomic.ai/img/nomic-logo.png" width=35> | [Nomic](https://portkey.wiki/gh-69) | ✅ | ✅ |
| <img src="https://files.readme.io/d38a23e-small-studio-favicon.png" width=35> | [AI21](https://portkey.wiki/gh-91) | ✅ | ✅ |
| <img src="https://platform.stability.ai/small-logo-purple.svg" width=35> | [Stability AI](https://portkey.wiki/gh-71) | ✅ | ✅ |
| <img src="https://deepinfra.com/_next/static/media/logo.4a03fd3d.svg" width=35> | [DeepInfra](https://portkey.sh/gh-92) | ✅ | ✅ |
| <img src="https://ollama.com/public/ollama.png" width=35> | [Ollama](https://portkey.wiki/gh-72) | ✅ | ✅ |
| <img src="https://novita.ai/favicon.ico" width=35> | [Novita AI](https://portkey.wiki/gh-73) | ✅ | ✅ | `/chat/completions`, `/completions` |
> [View the complete list of 200+ supported models here](https://portkey.wiki/gh-74)
<br>
<br>
## Agents
Gateway seamlessly integrates with popular agent frameworks. [Read the documentation here](https://portkey.wiki/gh-75).
| Framework | Call 200+ LLMs | Advanced Routing | Caching | Logging & Tracing* | Observability* | Prompt Management* |
|------------------------------|--------|-------------|---------|------|---------------|-------------------|
| [Autogen](https://portkey.wiki/gh-93) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [CrewAI](https://portkey.wiki/gh-94) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [LangChain](https://portkey.wiki/gh-95) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Phidata](https://portkey.wiki/gh-96) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Llama Index](https://portkey.wiki/gh-97) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Control Flow](https://portkey.wiki/gh-98) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Build Your Own Agents](https://portkey.wiki/gh-99) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| <img src="https://io.net/favicon.ico" width=35> | [IO Intelligence](https://io.net/intelligence) | ✅ | ✅ |
<br>
*Available on the [hosted app](https://portkey.wiki/gh-76). For detailed documentation [click here](https://portkey.wiki/gh-100).
## Gateway Enterprise Version
Make your AI app more <ins>reliable</ins> and <ins>forward compatible</ins>, while ensuring complete <ins>data security</ins> and <ins>privacy</ins>.
✅&nbsp; Secure Key Management - for role-based access control and tracking <br>
✅&nbsp; Simple & Semantic Caching - to serve repeat queries faster & save costs <br>
✅&nbsp; Access Control & Inbound Rules - to control which IPs and Geos can connect to your deployments <br>
✅&nbsp; PII Redaction - to automatically remove sensitive data from your requests to prevent indavertent exposure <br>
✅&nbsp; SOC2, ISO, HIPAA, GDPR Compliances - for best security practices <br>
✅&nbsp; Professional Support - along with feature prioritization <br>
[Schedule a call to discuss enterprise deployments](https://portkey.sh/demo-13)
<br>
## Contributing
The easiest way to contribute is to pick an issue with the `good first issue` tag 💪. Read the contribution guidelines [here](/.github/CONTRIBUTING.md).
Bug Report? [File here](https://portkey.wiki/gh-78) | Feature Request? [File here](https://portkey.wiki/gh-78)
### Getting Started with the Community
Join our weekly AI Engineering Hours every Friday (8 AM PT) to:
- Meet other contributors and community members
- Learn advanced Gateway features and implementation patterns
- Share your experiences and get help
- Stay updated with the latest development priorities
[Join the next session →](https://portkey.wiki/gh-101) | [Meeting notes](https://portkey.wiki/gh-102)
<br>
## Community
Join our growing community around the world, for help, ideas, and discussions on AI.
- View our official [Blog](https://portkey.wiki/gh-78)
- Chat with us on [Discord](https://portkey.wiki/community)
- Follow us on [Twitter](https://portkey.wiki/gh-79)
- Connect with us on [LinkedIn](https://portkey.wiki/gh-80)
- Read the documentation in [Japanese](./.github/README.jp.md)
- Visit us on [YouTube](https://portkey.wiki/gh-103)
- Join our [Dev community](https://portkey.wiki/gh-82)
<!-- - Questions tagged #portkey on [Stack Overflow](https://stackoverflow.com/questions/tagged/portkey) -->
![Rubeus Social Share (4)](https://github.com/Portkey-AI/gateway/assets/971978/89d6f0af-a95d-4402-b451-14764c40d03f)
README.wehub.md
+7
View File
@@ -0,0 +1,7 @@
# WeHub 来源说明
- 原始项目：`Portkey-AI/gateway`
- 原始仓库：https://github.com/Portkey-AI/gateway
- 导入方式：上游默认分支的最新快照
- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
- 本文件仅用于记录来源，不代表 WeHub 是原项目作者
conf.example.json
+49
View File
@@ -0,0 +1,49 @@
{
"admin_token": "set-a-strong-local-admin-token",
"plugins_enabled": [
"default",
"portkey",
"aporia",
"sydelabs",
"pillar",
"patronus",
"pangea",
"promptsecurity",
"panw-prisma-airs",
"walledai"
],
"credentials": {
"portkey": {
"apiKey": "..."
}
},
"cache": false,
"integrations": [
{
"provider": "anthropic",
"slug": "dev_team_anthropic",
"credentials": {
"apiKey": "sk-ant-"
},
"rate_limits": [
{
"type": "requests",
"unit": "rph",
"value": 3
},
{
"type": "tokens",
"unit": "rph",
"value": 3000
}
],
"models": [
{
"slug": "claude-3-7-sonnet-20250219",
"status": "active",
"pricing_config": null
}
]
}
]
}
cookbook/README.md
Vendored
+64
View File
@@ -0,0 +1,64 @@
# Portkey Practitioners' Cookbooks
<img src="../docs/images/cookbook-header.png" height=300 alt="header" />
#### Strategies and examples for tackling the production challenges of LLMs with Portkey Gateway
[![Discord](https://img.shields.io/discord/1143393887742861333)](https://portkey.ai/community)
[![Twitter](https://img.shields.io/twitter/url/https/twitter/follow/portkeyai?style=social&label=Follow%20%40PortkeyAI)](https://twitter.com/portkeyai)
## Table of Contents
Please use the below table of contents to navigate through the cookbook.
### Latest Notebooks
- Use models from [Nvidia NIM](/cookbook/providers/nvidia.ipynb) with AI Gateway
- Monitor [Llama Agents](/cookbook/monitoring-agents/Llama_Agents_with_Telemetry.ipynb) with Portkey!
- Create [Synthetic Datasets](/cookbook/use-cases/Nemotron_GPT_Finetuning_Portkey.ipynb) using Nemotron-340B
- Comparing [Top 10 LMSYS Models](./use-cases/LMSYS%20Series/comparing-top10-LMSYS-models-with-Portkey.ipynb) with AI Gateway.
## getting-started
* [Gentle introduction to Portkey Gateway](./getting-started/gentle-introduction-to-portkey-gateway.ipynb)
* [Use Portkey cache to save LLM cost & time](./getting-started/enable-cache.md)
* [Retry automatically on LLM failures](./getting-started/automatic-retries-on-failures.md)
* [Image generation with Gateway](./getting-started/image-generation.ipynb)
* [Writing your first Gateway Config](./getting-started/writing-your-first-gateway-config.md)
* [Automatically Fallback from OpenAI to Azure](./getting-started/fallback-from-openai-to-azure.ipynb)
View the [official docs](https://portkey.ai/docs)
## providers
* [OpenAI](./providers/openai.ipynb)
* [Anthropic](./providers/anthropic.md)
* [Mistral](./providers/mistral.md)
* [Vercel AI](./integrations/vercel-ai.md)
* [Groq](./providers/groq.ipynb)
* [Mistral](./providers/mistral.ipynb)
* [Deepinfra](./providers/deepinfra.ipynb)
* [Segmind](./providers/segmind.ipynb)
* [Nvidia](./providers/nvidia.ipynb)
View the [full list of providers here](https://portkey.ai/docs/welcome/integration-guides).
## integrations
* [Langchain](./integrations/langchain.ipynb)
* [Llama Index](./integrations/llama-index.ipynb)
* [Instructor](./integrations/Instructor_with_Portkey.ipynb)
* [Phidata](./integrations/Phidata_with_Portkey.ipynb)
View the [full list of integrations here](https://portkey.ai/docs/welcome/integration-guides).
## monitoring-agents
* [Autogen](./monitoring-agents/Autogen_with_Telemetry.ipynb)
* [CrewAI](./monitoring-agents/CrewAI_with_Telemetry.ipynb)
* [Llama Agents](./monitoring-agents/Llama_Agents_with_Telemetry.ipynb)
* [ControlFlow](./monitoring-agents/ControlFlow_with_Telemetry.ipynb)
## contributing
This is a community-driven resource! We welcome ideas, improvements, quick fixes, and deeper contributions that help the community.
[Request Cookbooks here](https://github.com/portkey-ai/gateway/issues). Check out our contribution [guidelines here](../.github/CONTRIBUTING.md).
cookbook/getting-started/automatic-retries-on-failures.md
Vendored
+132
View File
@@ -0,0 +1,132 @@
# Automatically retry requests to LLMs
A sudden timeout or error could harm the user experience and hurt your service's reputation if your application relies on an LLM for a critical feature. To prevent this, it's crucial to have a reliable retry mechanism in place. This will ensure that users are not left frustrated and can depend on your service.
Retrying Requests to Large Langauge Models (LLMs) can significantly increase your Gen AI app's reliability.
It can help you handle cases such as:
1. Cases that timed out (no response from the model)
2. Cases that returned a transient error from the model
In this cookbook, you will learn to use Portkey to automatically retry the requests on specific response status codes and control the times you want to retry.
## 1. Import and Authenticate Portkey Client SDK
Portkey forwards your requests to your desired model and relays the response to your app. Portkey’s Client SDK is one of several ways to make those API calls through the AI gateway.
To install it, type the following in your NodeJS environment:
```sh
npm install portkey-ai
```
Import `Portkey` and instantiate it using the Portkey API Key
```js
const portkey = new Portkey({
apiKey: 'xxxxrk',
virtualKey: 'maixxx4d'
});
```
At this point, it’s essential to understand that you instantiate the `portkey` instance with `apiKey` and `virtualKey` parameters. You can find the arguments for both of them in your Portkey Dashboard.
Visit the reference to [obtain the Portkey API key](https://portkey.ai/docs/api-reference/authentication) and learn [how to create Virtual Keys](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/virtual-keys#creating-virtual-keys).
## 2. Gateway Configs to Automatically Retry
For the AI gateway to understand that you want to apply automatic retries to your requests, you must pass Gateway Configs in your request payload. Gateway Configs can be a JS Object or a JSON string.
A typical Gateway Config to automatically retry three times when you hit rate-limits:
```js
{
retry: {
attempts: 3,
on_status_codes: [429]
}
}
```
You created a `retry` object with `attempts` and `on_status_codes` keys. The value of `attempts` can be bumped up to `5` times to retry automatically, while `on_status_codes` is an optional key. By default, Portkey will attempt to retry on the status codes `[429, 500, 502, 503, 504]`.
Refer to the [101 on Gateway Configs](https://github.com/Portkey-AI/portkey-cookbook/blob/main/product/101-portkey-gateway-configs.md#a-reference-gateway-configs-from-the-ui) and [Automatic Retries](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/automatic-retries).
## 3. Make API calls using Portkey Client SDK
You are now ready to make an API call through Portkey. While there are several ways to make API calls, in this cookbook, let’s pass the gateway configuration during the chat completion call.
```js
let response = await portkey.chat.completions.create(
{
model: 'gpt-4',
messages: [
{
role: 'user',
content: 'What are 7 wonders of the world?'
}
]
},
{
config: JSON.stringify({
retry: {
attempts: 3,
on_status_codes: [429]
}
})
}
);
console.log(response.choices[0].message.content);
```
The Portkey SDK adds the configs in the HTTP headers to apply automatic retries to our requests. Broadly, the signature of the chat completion method:
```js
await portkey.chat.completions.create( modelParmeters [, gatewayConfigs])
```
## 4. View the Logs
Now that you successfully know how to make API calls through Portkey, it’s also helpful to learn about Logs. You can find all requests sent through Portkey on the **Dashboard** > **Logs** page.
![automatically-retry-requests-to-llms](../../docs/images/cookbooks/logs.png)
This page provides essential information such as time, cost, and response. Feel free to explore it!
Instead of using your own application-level looping or control structures to implement retries, you can use Portkey’s Gateway Configs to manage all of them.
<details>
<summary>See the full code</summary>
```js
import { Portkey } from 'portkey-ai';
const portkey = new Portkey({
apiKey: xxxx,
virtualKey: 'xaixxxxxxx2xx4d'
});
let response = await portkey.chat.completions.create(
{
model: 'gpt-4',
messages: [
{
role: 'user',
content: 'What are 7 wonders of the world?'
}
]
},
{
config: JSON.stringify({
retry: {
attempts: 3
}
})
}
);
console.log(response.choices[0].message.content);
```
</details>
cookbook/getting-started/enable-cache.md
Vendored
+219
View File
@@ -0,0 +1,219 @@
# Prevent unnecessary LLM requests with the Portkey Cache
If you have multiple users of your GenAI app triggering the same or similar queries to your models, fetching LLM response from the models can be slow and expensive. This is because it requires multiple round trips from your app to the model and you may end up paying for the duplicate queries.
To avoid such unnecessary LLM requests, you can use Portkey as your first line of defense. It is highly effective and can be made to work across the 100+ LLMs it supports by simply making changes to a few lines of code.
## How Portkey Cache Works
All requests that have caching enabled on them will serve the subsequent responses from the Portkey’s cache.
![prevent-unnecessary-llm-requests-with-the-portkey-cache](../../docs/images/cookbooks/cache-1.png)
Portkey offers two main ways of Caching techniques to enable on your requests — Simple and Semantic.
In short:
- Simple caching refers for identical input prompts to serve from cache.
- Semantic caching refers to an similarity threshold (uses cosine similarity) to serve from cache.
For detailed information, check out [this](https://portkey.ai/blog/reducing-llm-costs-and-latency-semantic-cache/) blog post.
## 1. Import and Authenticate Portkey Client SDK
You now have a brief mindmap of Portkey's approach to caching responses from LLMs.
Let's utilize the Portkey Client SDK to send chat completion requests and attach gateway configs, which in turn activate caching.
To install it, type the following in your NodeJS environment:
```sh
npm install portkey-ai
```
Instantiate Portkey instance
```js
const portkey = new Portkey({
apiKey: 'xxxxrk',
virtualKey: 'maixxx4d'
});
```
At this point, it’s essential to understand that you instantiate the `portkey` instance with `apiKey` and `virtualKey` parameters. You can find the arguments for both of them in your Portkey Dashboard.
Visit the reference to [obtain the Portkey API key](https://portkey.ai/docs/api-reference/authentication) and learn [how to create Virtual Keys](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/virtual-keys#creating-virtual-keys).
## 2. Use Gateway Configs to enable Caching
The AI gateway caches your requests and serves it respecting the gateway configs on the request headers. The configs are a simple JS object or JSON string that contains following key-value pairs.
The `mode` key specifies the desired strategy of caching you want for your app.
```js
// Simple Caching
"cache": { "mode": "simple" }
// Semantic Caching
"cache": { "mode": "semantic" }
```
Next up, attach these configs to the request using Portkey SDK. The SDK accepts an `config` parameter that can accept these configurations as an argument. To learn about more ways, refer to the [101 on Gateway Configs](https://github.com/Portkey-AI/portkey-cookbook/blob/main/product/101-portkey-gateway-configs.md#a-reference-gateway-configs-from-the-ui).
## 3. Make API calls, Serve from Cache
We are now ready to put what we’ve learned so far into action. We plan on making two requests to an OpenAI model (as an example) while one of them has simple caching activated on it, while other has semantic caching enabled.
```js
// Simple Cache
let simpleCacheResponse = await portkey.chat.completions.create(
{
model: 'gpt-4',
messages: [
{
role: 'user',
content: 'What are 7 wonders of the world?'
}
]
},
{
config: JSON.stringify({
cache: {
mode: 'simple'
}
})
}
);
console.log('Simple Cached Response:\n', simpleCacheResponse.choices[0].message.content);
```
Whereas for semantic caching,
```js
let semanticCacheResponse = await portkey.chat.completions.create(
{
model: 'gpt-4',
messages: [
{
role: 'user',
content: 'List the 5 senses of Human beings?'
}
]
},
{
config: JSON.stringify({
cache: {
mode: 'semantic'
}
})
}
);
console.log('\nSemantically Cached Response:\n', semanticCacheResponse.choices[0].message.content);
```
On the console:
```sh
Simple Cached Response:
1. The Great Wall of China
2. Petra, Jordan
3. Christ the Redeemer Statue, Brazil
4. Machu Picchu, Peru
5. The Chichen Itza Pyramid, Mexico
6. The Roman Colosseum, Italy
7. The Taj Mahal, India
Semantically Cached Response:
1. Sight (Vision)
2. Hearing (Auditory)
3. Taste (Gustatory)
4. Smell (Olfactory)
5. Touch (Tactile)
```
Try experimenting with rephrasing the prompts in the `messages` array and see if you notice any difference in the time it takes to receive a response or the quality of the response itself.
Can you refresh the cache on demand? Yes, you can!
Can you control how long the cache remains active? Absolutely!
Explore the [docs](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/cache-simple-and-semantic) on caching to know all the features available to control how you cache the LLM responses.
## 4. View Analytics and Logs
On the **Analytics** page, you can find Portkey's cache performance analytics under the Cache tab.
![Dasbhoard with Analytics](../../docs/images/cookbooks/cache-2.png)
The **Logs** page displays a list of LLM calls that served responses from cache. The corresponding icon is activated when the cache is hit.
![Dashboard with Logs](../../docs/images/cookbooks/cache-3.png)
## Next steps
By leveraging simple and semantic caching, you can avoid unnecessary LLM requests, reduce latency, and provide a better user experience. So go ahead and experiment with the Portkey Cache in your own projects – the benefits are just a few lines of code away!
Some suggestions to experiment:
- Try using the configs from the [Portkey UI](https://github.com/Portkey-AI/portkey-cookbook/blob/main/ai-gateway/101-portkey-gateway-configs.md#a-reference-gateway-configs-from-the-ui) as a reference.
- Implement caching when there are [multiple targets](https://github.com/Portkey-AI/portkey-cookbook/blob/main/ai-gateway/how-to-setup-fallback-from-openai-to-azure-openai.md#2-creating-fallback-configs) in your gateway configs. (Here’s a [clue](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/cache-simple-and-semantic#how-cache-works-with-configs))
<details>
<summary>See the full code</summary>
```js
import { Portkey } from 'portkey-ai';
const portkey = new Portkey({
apiKey: 'xxxxxk',
virtualKey: 'mxxxxxxxxd'
});
let simpleCacheResponse = await portkey.chat.completions.create(
{
model: 'gpt-4',
messages: [
{
role: 'user',
content: 'What are 7 wonders of the world?'
}
]
},
{
config: JSON.stringify({
cache: {
mode: 'simple'
}
})
}
);
console.log('Simple Cached Response:\n', simpleCacheResponse.choices[0].message.content);
let semanticCacheResponse = await portkey.chat.completions.create(
{
model: 'gpt-4',
messages: [
{
role: 'user',
content: 'List the 5 senses of Human beings?'
}
]
},
{
config: JSON.stringify({
cache: {
mode: 'semantic'
}
})
}
);
console.log('\nSemantically Cached Response:\n', semanticCacheResponse.choices[0].message.content);
```
</details>
cookbook/getting-started/fallback-from-openai-to-azure.ipynb
Vendored
+381
View File
@@ -0,0 +1,381 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "view-in-github"
},
"source": [
"<a href=\"https://colab.research.google.com/github/Portkey-AI/portkey-cookbook/blob/main/ai-gateway/how_to_setup_fallback_from_openai_to_azure_openai.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Pln7ewgvA3Gs"
},
"source": [
"## How to Setup Fallback from OpenAI to Azure OpenAI\n",
"\n",
"Let’s say you’ve built an LLM-based app and deployed it to production. It relies on OpenAI’s gpt-4 model. It’s [Mar 12, 2023](https://status.portkey.ai/incident/339664), and suddenly your users find errors with the functionality of the app — “It doesn’t work!”\n",
"\n",
"It turns out that in the logs, the app has encountered [503 errors](https://platform.openai.com/docs/guides/error-codes) due to overloaded requests on the server-side. What could you do? If you are in such a situation, we have an answer for you: Portkey Fallbacks.\n",
"\n",
"Portkey Fallbacks can automatically switch your app's requests from one LLM provider to another, ensuring reliability by allowing you to fallback among multiple LLMs. This is especially useful given the unpredictable nature of LLM APIs. With Portkey, you can switch to a different LLM provider, such as Azure, when needed, making your app Production-Ready.\n",
"\n",
"In this cookbook, we will learn how to implement a fallback mechanism in our apps that allows us to automatically switch the LLM provider from OpenAI to Azure OpenAI with just a few lines of code. Both providers have the exact same set of models, but they are deployed differently. Azure OpenAI comes with its own deployment mechanisms, which are generally considered to be more reliable.\n",
"\n",
"<span style=\"text-decoration:underline;\">Prerequisites:</span>\n",
"\n",
"1. You have the [Portkey API Key](https://portkey.ai/docs/api-reference/authentication#obtaining-your-api-key). [ [Sign Up](https://portkey.ai) ]\n",
"2. You stored OpenAI and Azure OpenAI API keys as [virtual keys](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/virtual-keys)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "leMVtjKrBce2"
},
"source": [
"## 1. Import the SDK and authenticate with Portkey\n",
"\n",
"We start by importing Portkey SDK into our NodeJS project using npm and authenticate by passing the Portkey API Key."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vOostjRiBfq_"
},
"outputs": [],
"source": [
"!pip install portkey-ai openai"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "b_985bXCCGGZ"
},
"outputs": [],
"source": [
"from portkey_ai import Portkey\n",
"from google.colab import userdata\n",
"\n",
"PORTKEYAI_API_KEY=userdata.get('PORTKEY_API_KEY')\n",
"OPENAI_VIRTUAL_KEY=userdata.get('OPENAI_VIRTUAL_KEY')\n",
"\n",
"portkey = Portkey(\n",
" api_key=PORTKEYAI_API_KEY,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cKbbE0t9DKV4"
},
"source": [
"## 2. Create Fallback Configs\n",
"\n",
"Next, we will create a configs object that influences the behavior of the request sent using Portkey.\n",
"\n",
"```json\n",
"{\n",
" strategy: {\n",
" mode: \"fallback\",\n",
" },\n",
" targets: [\n",
" {\n",
" virtual_key: OPENAI_VIRTUAL_KEY,\n",
" },\n",
" {\n",
" virtual_key: AZURE_OPENAI_VIRTUAL_KEY,\n",
" },\n",
" ],\n",
"}\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RFjHxpzUCbUJ"
},
"source": [
"This configuration instructs Portkey to use a \\_fallback \\_strategy with the requests. The \\_targets_array lists the virtual keys of LLMs in the order Portkey should fallback to an alternative.\n",
"\n",
"Most users find it way more cleaner to define the configs in the Portkey UI and reference the config ID in the code. [Try it out](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/configs#creating-configs).\n",
"\n",
"Add this configuration to the _portkey_ instance to apply the fallback behavior to all the requests."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"id": "t_-VBo6ZDX2v"
},
"outputs": [],
"source": [
"from portkey_ai import Portkey\n",
"from google.colab import userdata\n",
"import json\n",
"\n",
"PORTKEYAI_API_KEY=userdata.get('PORTKEY_API_KEY')\n",
"OPENAI_VIRTUAL_KEY=userdata.get('OPENAI_VIRTUAL_KEY')\n",
"AZURE_OPENAI_VIRTUAL_KEY=userdata.get('AZURE_OPENAI_VIRTUAL_KEY')\n",
"\n",
"config_data = {\n",
" 'strategy': {\n",
" 'mode': \"fallback\",\n",
" },\n",
" 'targets': [\n",
" {\n",
" 'virtual_key': OPENAI_VIRTUAL_KEY,\n",
" },\n",
" {\n",
" 'virtual_key': AZURE_OPENAI_VIRTUAL_KEY,\n",
" },\n",
" ],\n",
"}\n",
"\n",
"\n",
"portkey = Portkey(\n",
" api_key=PORTKEYAI_API_KEY,\n",
" virtual_key=OPENAI_VIRTUAL_KEY,\n",
" config=json.dumps(config_data)\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lBvXjINhEbxg"
},
"source": [
"Always reference the credentials from the environment variables to prevent exposure of any sensitive data. Portkey will automatically infer the LLM providers based on the passed virtual keys.\n",
"\n",
"> The Azure OpenAI virtual key only needs to be set up once, and it will then be accessible through Portkey in all subsequent API calls."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<details>\n",
"\n",
"<summary>Fallback Configs without virtual keys</summary>\n",
"\n",
"```json\n",
"{\n",
" \"strategy\": {\n",
" \"mode\": \"fallback\"\n",
" },\n",
" \"targets\": [\n",
" {\n",
" \"provider\": \"openai\",\n",
" \"api_key\": \"sk-xxxxxxxxpRT4xxxx5\"\n",
" },\n",
" {\n",
" \"provider\": \"azure-openai\",\n",
" \"api_key\": \"*******\"\n",
" }\n",
" ]\n",
"}\n",
"```\n",
"\n",
"</details>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fQ5JMtO3EeFu"
},
"source": [
"## 3. Make a request\n",
"\n",
"All the requests will hit OpenAI since Portkey proxies all those requests to the target(s) we already specified. Notice that the changes to the requests do not demand any code changes in the business logic implementation. Smooth!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "c8gYcOg3MOKq"
},
"outputs": [],
"source": [
"messages = [\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": \"What are the 7 wonders of the world?\"\n",
" }\n",
"]\n",
"\n",
"response = portkey.chat.completions.create(\n",
" messages = messages,\n",
" model = 'gpt-3.5-turbo'\n",
")\n",
"\n",
"print(response.choices[0].message.content) # Here is the plan"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gehAN1uiMd2e"
},
"source": [
"When OpenAI returns any 4xx or 5xx errors, Portkey will automatically switch to Azure OpenAI to ensure the same specified model is used."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DjDa1SXDMiGH"
},
"source": [
"## 4. View the Fallback Status in Logs\n",
"\n",
"Since all the requests go through Portkey, Portkey can log them for better observability of your app. You can find the specific requests by passing an _trace ID_. It can be any desired string name. In this case, `my-trace-id`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Lgg6X2VDMlZF"
},
"outputs": [],
"source": [
"response = portkey.with_options(trace_id=\"<my-trace-id>\").chat.completions.create(\n",
" messages = messages,\n",
" model = 'gpt-4'\n",
")\n",
"\n",
"print(response.choices[0].message.content)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "z_sY3uQbNH4W"
},
"source": [
"You can apply filter with Trace ID to list requests. Instances when the fallbacks are activated will highlight the fallback icon. The logs can be filtered by cost, tokens, status, config, trace id and so on.\n",
"\n",
"![](../../docs/images/cookbooks/fallback-trace-id.png)\n",
"\n",
"Learn more about [Logs](https://portkey.ai/docs/product/observability-modern-monitoring-for-llms/logs)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "R2KpXJfcO80T"
},
"source": [
"## 5. Advanced: Fallback on Specific Status Codes\n",
"\n",
"Portkey provides finer control over the when it should apply fallback strategy to your requests to LLMs. You can define the configuration to condition based on specific status codes returned by the LLM provider."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "xD1AYPf1PDGQ"
},
"outputs": [],
"source": [
"from portkey_ai import Portkey\n",
"from google.colab import userdata\n",
"import json\n",
"\n",
"PORTKEYAI_API_KEY=userdata.get('PORTKEY_API_KEY')\n",
"OPENAI_VIRTUAL_KEY=userdata.get('OPENAI_VIRTUAL_KEY')\n",
"AZURE_OPENAI_VIRTUAL_KEY=userdata.get('AZURE_OPENAI_VIRTUAL_KEY')\n",
"\n",
"config_data = {\n",
" 'strategy': {\n",
" 'mode': \"fallback\",\n",
" 'on_status_codes': [429]\n",
" },\n",
" 'targets': [\n",
" {\n",
" 'virtual_key': OPENAI_VIRTUAL_KEY,\n",
" },\n",
" {\n",
" 'virtual_key': AZURE_OPENAI_VIRTUAL_KEY,\n",
" },\n",
" ],\n",
"}\n",
"\n",
"\n",
"portkey = Portkey(\n",
" api_key=PORTKEYAI_API_KEY,\n",
" virtual_key=OPENAI_VIRTUAL_KEY,\n",
" config=json.dumps(config_data)\n",
")\n",
"\n",
"messages = [\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": \"What are the 7 wonders of the world?\"\n",
" }\n",
"]\n",
"\n",
"response = portkey.chat.completions.create(\n",
" messages = messages,\n",
" model = 'gpt-3.5-turbo'\n",
")\n",
"\n",
"print(response.choices[0].message.content) # Here is the plan"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "znyPLa3uPT8L"
},
"source": [
"In the above case for all the request that are acknowledged with the status code of 429 will fallback from OpenAI to Azure OpenAI."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "itZYQAdwPU0b"
},
"source": [
"## 6. Considerations\n",
"\n",
"That’s it; you can implement production-grade fallback mechanisms with just a few lines of code. While you are equipped with all the tools to implement fallbacks to your next GenAI app, here are few considerations:\n",
"\n",
"- The implementation of Fallback does not alter the quality of LLM outputs received by your app.\n",
"- Azure requires you to deploy specific models. Portkey will automatically trigger the chat completions endpoint using GPT4 if it is available instead of GPT3.5."
]
}
],
"metadata": {
"colab": {
"authorship_tag": "ABX9TyNTD3h7w6NbWJdNj5scDoZ7",
"include_colab_link": true,
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
cookbook/getting-started/fallback-from-stable-diffusion-to-dall-e.ipynb
Vendored
+307
View File
@@ -0,0 +1,307 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/github/Portkey-AI/portkey-cookbook/blob/main/ai-gateway/set-up-fallback-from-stable-diffusion-to-dall-e.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"id": "4586cc53",
"metadata": {
"id": "4586cc53"
},
"source": [
"# Set up Fallback from Stable Diffusion to Dall-E\n",
"\n",
"Generative AI models have revolutionized text generation and opened up new possibilities for developers. What next? A new category of image generation models.\n",
"\n",
"This cookbook introduces Portkey’s multimodal AI gateway, which helps you switch between multiple image generation models without any code changes — all with OpenAI SDK. You will learn to set up fallbacks from Stable Diffusion to Dall-E."
]
},
{
"cell_type": "markdown",
"id": "42ae2283",
"metadata": {
"id": "42ae2283"
},
"source": [
"## 1. Integrate Image Gen Models with Portkey \n",
"\n",
"Begin by storing API keys in the Portkey Vault.\n",
"\n",
"To save your OpenAI and StabilityAI keys in the Portkey Vault:\n",
"1. Go to **portkey.ai**\n",
"2. Click **Virtual Keys** and then **Create**\n",
" 1. Enter **Name** and **API Key**,\n",
" 2. Hit **Create**\n",
"3. Copy the virtual key from the **KEY** column\n",
"\n",
"We successfully have set up virtual keys!\n",
"\n",
"For more information, refer the [docs](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/virtual-keys).\n",
"\n",
"The multi-modal AI gateway will use these virtual keys in the future to apply a fallback mechanism to every request from your app."
]
},
{
"cell_type": "markdown",
"id": "5e2efe10",
"metadata": {
"id": "5e2efe10"
},
"source": [
"## 2. Making a call to Stability AI using OpenAI SDK\n",
"\n",
"With Portkey, you can call Stability AI models like SDXL right from inside the OpenAI SDK. Just change the `base_url` to Portkey Gateway and add `defaultHeaders` while instantiating your OpenAI client, and you're good to go\n",
"\n",
"Import the `openai` and `portkey_ai` libraries to send the requests, whereas the rest of the utility libraries will help decode the base64 response and print them onto Jupyter Notebook."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9c6ac821",
"metadata": {
"id": "9c6ac821"
},
"outputs": [],
"source": [
"from IPython.display import display\n",
"from PIL import Image\n",
"from openai import OpenAI\n",
"from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders\n",
"import requests, io, base64, json"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e6f4c81a",
"metadata": {
"id": "e6f4c81a"
},
"outputs": [],
"source": [
"PORTKEY_API_KEY=\"YOUR_PORTKEY_API_KEY_HERE\"\n",
"OPENAI_VIRTUAL_KEY=\"YOUR_OPENAI_VIRTUAL_KEY_HERE\"\n",
"CONFIG_ID=\"YOUR_CONFIG_ID_HERE\"\n",
"OPENAI_API_KEY=\"REDUNDANT\""
]
},
{
"cell_type": "markdown",
"id": "96aa498e",
"metadata": {
"id": "96aa498e"
},
"source": [
"Declare the arguments to pass to the parameters of OpenAI SDK and initialize a client instance."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6383332c",
"metadata": {
"id": "6383332c"
},
"outputs": [],
"source": [
"STABILITYAI_VIRTUAL_KEY=\"YOUR_STABILITYAI_VIRTUAL_KEY_HERE\"\n",
"\n",
"client = OpenAI(\n",
" api_key=\"REDUNDANT\",\n",
" base_url=PORTKEY_GATEWAY_URL,\n",
" default_headers=createHeaders(\n",
" provider=\"stabilityai\",\n",
" api_key=PORTKEY_API_KEY,\n",
" virtual_key=STABILITYAI_VIRTUAL_KEY,\n",
" )\n",
")"
]
},
{
"cell_type": "markdown",
"id": "75e47fca",
"metadata": {
"id": "75e47fca"
},
"source": [
"The `api_key` parameter is passed a random string since it’s redundant as the request will be handled through Portkey.\n",
"\n",
"To generate an image:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e15ab799",
"metadata": {
"scrolled": true,
"id": "e15ab799"
},
"outputs": [],
"source": [
"image = client.images.generate(\n",
" model=\"stable-diffusion-v1-6\",\n",
" prompt=\"Kraken in the milkyway galaxy\",\n",
" n=1,\n",
" size=\"1024x1024\",\n",
" response_format=\"b64_json\"\n",
")\n",
"\n",
"base64_image = image.data[0].b64_json\n",
"\n",
"image_data = base64.b64decode(base64_image)\n",
"\n",
"image = Image.open(io.BytesIO(image_data))\n",
"display(image)"
]
},
{
"cell_type": "markdown",
"id": "900e40aa",
"metadata": {
"id": "900e40aa"
},
"source": [
"The image you receive in the response is encoded in base64 format, which requires you to decode it before you can view it in the Jupyter Notebook. In addition, Portkey offers logging for observability. To find all the information for every request, simply check the requests on the **Dashboard > Logs**."
]
},
{
"cell_type": "markdown",
"id": "55edb172",
"metadata": {
"id": "55edb172"
},
"source": [
"## 3. Now, Setup a Fallback from SDXL to Dall-E\n",
"\n",
"Let’s learn how to enhance the reliability of your Stability AI requests by configuring automatic fallbacks to Dall-E in case of failures. You can use Gateway Configs on Portkey to implement this automated fallback logic. These configurations can be passed while creating your OpenAI client.\n",
"\n",
"From the Portkey Dashboard, open **Configs** and then click **Create**. In the config editor, write the JSON for Gateway Configs:\n",
"\n",
"```json\n",
"{\n",
" \"strategy\": {\n",
" \"mode\": \"fallback\"\n",
" },\n",
" \"targets\": [\n",
" {\n",
" \"virtual_key\": \"stability-ai-virtualkey\",\n",
" \"override_params\": {\n",
" \"model\": \"stable-diffusion-v1-6\"\n",
" }\n",
" },\n",
" {\n",
" \"virtual_key\": \"open-ai-virtual-key\",\n",
" \"override_params\": {\n",
" \"model\": \"dall-e-3\"\n",
" }\n",
" }\n",
" ]\n",
"}\n",
"```\n",
"\n",
"These configs tell the AI gateway to follow an `fallback` strategy, where the primary target to forward requests to is _Stability AI_ (automatically inferred from the virtual key) and then to _OpenAI_. The `override_params` let’s you override the default models for the provider. Finally, surprise surprise! — we also enabled caching with just one more key-value pair.\n",
"\n",
"Learn about [Gateway Configs](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/configs) and [Caching](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/cache-simple-and-semantic) from the docs.\n",
"\n",
"Hit **Save Config** on the top right corner and grab the **Config ID. **Next up, we are going to use the _Config ID _in our requests to activate fallback mechanism."
]
},
{
"cell_type": "markdown",
"id": "0681664d",
"metadata": {
"id": "0681664d"
},
"source": [
"## 4. Make a request with gateway configs \n",
"\n",
"Finally, the requests will be sent like we did with OpenAI SDK earlier, but with one specific difference - the `config` parameter. The request is sent through Portkey and uses saved gateway configs as references to activate the fallback mechanism."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e34d9797",
"metadata": {
"id": "e34d9797"
},
"outputs": [],
"source": [
"client = OpenAI(\n",
" api_key=OPENAI_API_KEY,\n",
" base_url=PORTKEY_GATEWAY_URL,\n",
" default_headers=createHeaders(\n",
" api_key=PORTKEY_API_KEY,\n",
" config=CONFIG_ID\n",
" )\n",
")\n",
"\n",
"image = client.images.generate(\n",
" model=\"stable-diffusion-v1-6\",\n",
" prompt=\"Harry Potter travelling the world using Portkey\",\n",
" n=1,\n",
" size=\"1024x1024\",\n",
" response_format=\"b64_json\"\n",
")\n",
"\n",
"base64_image = image.data[0].b64_json\n",
"\n",
"image_data = base64.b64decode(base64_image)\n",
"\n",
"image = Image.open(io.BytesIO(image_data))\n",
"display(image)"
]
},
{
"cell_type": "markdown",
"id": "1053e49e",
"metadata": {
"id": "1053e49e"
},
"source": [
"## Afterthoughts\n",
"\n",
"All the requests that go through Portkey will appear in the Logs page within the Portkey Dashboard. You can apply filters or even trace the specific set of requests. Check out [Request Tracing](https://portkey.ai/docs/product/observability-modern-monitoring-for-llms/traces). Simultaneously, a fallback icon is turned on for the log where the fallback is activated.\n",
"\n",
"Portkey supports multiple providers offering multimodal capabilities, such as OpenAI, Anthropic, and Stability AI, all accessible through a unified API interface following OpenAI Signature.\n",
"\n",
"For further exploration, why not [play with Vision capabilities](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/multimodal-capabilities/vision)?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.18"
},
"colab": {
"provenance": [],
"include_colab_link": true
}
},
"nbformat": 4,
"nbformat_minor": 5
}
cookbook/getting-started/gentle-introduction-to-portkey-gateway.ipynb
Vendored
+308
View File
@@ -0,0 +1,308 @@
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
}
},
"cells": [
{
"cell_type": "markdown",
"source": [
"<h1 align=\"center\">\n",
" <a href=\"https://portkey.ai\">\n",
" <img width=\"300\" src=\"https://analyticsindiamag.com/wp-content/uploads/2023/08/Logo-on-white-background.png\" alt=\"portkey\">\n",
" </a>\n",
"</h1>"
],
"metadata": {
"id": "APmF3kxYFiCY"
}
},
{
"cell_type": "markdown",
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1nQa-9EYcv9-O6VnwLATnVd9Q2wFtthOA?usp=sharing)"
],
"metadata": {
"id": "rAxc8aNDGMY2"
}
},
{
"cell_type": "markdown",
"source": [
"[Portkey](https://app.portkey.ai/) is the Control Panel for AI apps. With it's popular AI Gateway and Observability Suite, hundreds of teams ship reliable, cost-efficient, and fast apps.\n",
"\n",
"With Portkey, you can\n",
"\n",
" - Connect to 150+ models through a unified API,\n",
" - View 40+ metrics & logs for all requests,\n",
" - Enable semantic cache to reduce latency & costs,\n",
" - Implement automatic retries & fallbacks for failed requests,\n",
" - Add custom tags to requests for better tracking and analysis and more."
],
"metadata": {
"id": "L0q-knFpGUHN"
}
},
{
"cell_type": "markdown",
"source": [
"## Quickstart\n",
"\n",
"Since Portkey is fully compatible with the OpenAI signature, you can connect to the Portkey AI Gateway through OpenAI Client.\n",
"\n",
"- Set the `base_url` as `PORTKEY_GATEWAY_URL`\n",
"- Add `default_headers` to consume the headers needed by Portkey using the `createHeaders` helper method."
],
"metadata": {
"id": "tRvjIw-cGbef"
}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "fffx7Tc2ghTR",
"outputId": "b832e334-9770-4c7c-f7ea-dcba522986e8"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"\u001b[?25l \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/60.7 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m\u001b[90m━━━━━━\u001b[0m \u001b[32m51.2/60.7 kB\u001b[0m \u001b[31m1.9 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m60.7/60.7 kB\u001b[0m \u001b[31m1.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m262.9/262.9 kB\u001b[0m \u001b[31m11.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.6/75.6 kB\u001b[0m \u001b[31m5.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.5/12.5 MB\u001b[0m \u001b[31m54.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m53.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.9/77.9 kB\u001b[0m \u001b[31m6.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m5.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25h"
]
}
],
"source": [
"!pip install -qU portkey-ai openai"
]
},
{
"cell_type": "code",
"source": [
"from openai import OpenAI\n",
"from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders\n",
"from google.colab import userdata"
],
"metadata": {
"id": "QNRIgaAIk--q"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"## OpenAI"
],
"metadata": {
"id": "ptP4L78HlBUL"
}
},
{
"cell_type": "code",
"source": [
"client = OpenAI(\n",
" api_key=OPENAI_API_KEY,\n",
" base_url=PORTKEY_GATEWAY_URL,\n",
" default_headers=createHeaders(\n",
" provider=\"openai\",\n",
" api_key=PORTKEY_API_KEY\n",
" )\n",
")\n",
"\n",
"chat_complete = client.chat.completions.create(\n",
" model=\"gpt-4\",\n",
" messages=[{\"role\": \"user\",\n",
" \"content\": \"What's a fractal?\"}],\n",
")\n",
"\n",
"print(chat_complete.choices[0].message.content)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "I5YTh44Pgpqa",
"outputId": "1c763257-41ef-455a-fec6-2d9883316585"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"A fractal is a complex geometric shape that can be split into parts, each of which is a reduced-scale copy of the whole. Fractals are typically self-similar and independent of scale, meaning they look similar at any zoom level. They often appear in nature, in things like snowflakes, coastlines, and fern leaves. The term \"fractal\" was coined by mathematician Benoit Mandelbrot in 1975.\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"## Anthropic"
],
"metadata": {
"id": "FHTGygDilMGk"
}
},
{
"cell_type": "code",
"source": [
"from openai import OpenAI\n",
"from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders\n",
"\n",
"client = OpenAI(\n",
" api_key=userdata.get('ANTHROPIC_API_KEY')\n",
" base_url=PORTKEY_GATEWAY_URL,\n",
" default_headers=createHeaders(\n",
" provider=\"anthropic\",\n",
" api_key=PORTKEY_API_KEY\n",
" ),\n",
")\n",
"\n",
"response = client.chat.completions.create(\n",
" model=\"claude-3-opus-20240229\",\n",
" messages=[{\"role\": \"user\",\n",
" \"content\": \"What's a fractal?\"}],\n",
" max_tokens= 512\n",
")"
],
"metadata": {
"id": "5UaGvjbwYmj6"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"## Mistral AI"
],
"metadata": {
"id": "6hGepv90lP5T"
}
},
{
"cell_type": "code",
"source": [
"from openai import OpenAI\n",
"from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders\n",
"\n",
"client = OpenAI(\n",
" api_key=userdata.get('MISTRAL_API_KEY'),\n",
" base_url=PORTKEY_GATEWAY_URL,\n",
" default_headers=createHeaders(\n",
" provider=\"mistral-ai\",\n",
" api_key=PORTKEY_API_KEY\n",
" )\n",
")\n",
"\n",
"chat_complete = client.chat.completions.create(\n",
" model=\"mistral-medium\",\n",
" messages=[{\"role\": \"user\",\n",
" \"content\": \"What's a fractal?\"}],\n",
")\n",
"\n",
"print(chat_complete.choices[0].message.content)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ByWFpbVfW7Po",
"outputId": "b6274daf-0662-4e5c-808c-a239a653da8e"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"A fractal is a geometric shape or pattern that exhibits self-similarity at different scales. This means that the shape appears similar or identical when viewed at different levels of magnification. Fractals are often complex and intricate, and they can be generated mathematically using iterative algorithms. They are commonly found in nature, such as in the branching patterns of trees and the shapes of coastlines. Fractals have applications in various fields, including mathematics, physics, and computer graphics. Some famous examples of fractals include the Mandelbrot set and the Sierpinski triangle.\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"## Together AI"
],
"metadata": {
"id": "7o9Otqq2lSf8"
}
},
{
"cell_type": "code",
"source": [
"from openai import OpenAI\n",
"from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders\n",
"\n",
"client = OpenAI(\n",
" api_key=userdata.get('TOGETHER_API_KEY'),\n",
" base_url=PORTKEY_GATEWAY_URL,\n",
" default_headers=createHeaders(\n",
" provider=\"together-ai\",\n",
" api_key=PORTKEY_API_KEY\n",
" )\n",
")\n",
"\n",
"chat_complete = client.chat.completions.create(\n",
" model=\"meta-llama/Llama-2-70b-hf\",\n",
" messages=[{\"role\": \"user\",\n",
" \"content\": \"What's a fractal?\"}],\n",
")\n",
"\n",
"print(chat_complete.choices[0].message.content)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Yz7e9rokcCj0",
"outputId": "4305bf47-2c16-43c1-c1d4-40da7ce08e55"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"<|im_start|>user\n",
"A fractal is a never ending pattern. Fractals are infinitely complex patterns that are self-similar across different scales. They are created by repeating a simple process over and over in an ongoing feedback loop. Driven by recursion, fractals are images of dynamic systems – the pictures of Chaos. Geometrically, they exist in between our familiar dimensions. Fractal patterns are extremely familiar, since nature is full of fractals. For instance: trees, rivers, coastlines, mountains, clouds, seashells, hurricanes, etc\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"## Refer to our docs to integrate with other providers. [link](https://portkey.ai/docs/welcome/integration-guides)"
],
"metadata": {
"id": "doLNNsZyuEa6"
}
}
]
}
cookbook/getting-started/image-generation.ipynb
Vendored
+357
View File
File diff suppressed because one or more lines are too long
cookbook/getting-started/resilient-loadbalancing-with-failure-mitigating-fallbacks.md
Vendored
+265
View File
@@ -0,0 +1,265 @@
# Setting up resilient Load balancers with failure-mitigating Fallbacks
Companies often face challenges of scaling their services efficiently as the traffic to their applications grow - when you’re consuming APIs, the first point of failure is that if you hit the API too much, you can get rate limited. Loadbalancing is a proven way to scale usage horizontally without overburdening any one provider and thus staying within rate limits.
For your AI app, rate limits are even more stringent, and if you start hitting the providers’ rate limits, there’s nothing you can do except wait to cool down and try again. With Portkey, we help you solve this very easily.
This cookbook will teach you how to utilize Portkey to distribute traffic across multiple LLMs, ensuring that your loadbalancer is robust by setting up backups for requests. Additionally, you will learn how to load balance across OpenAI and Anthropic, leveraging the powerful Claude-3 models recently developed by Anthropic, with Azure serving as the fallback layer.
<span style="text-decoration:underline;">Prerequisites:</span>
You should have the [Portkey API Key](https://portkey.ai/docs/api-reference/authentication#obtaining-your-api-key). Please sign up to obtain it. Additionally, you should have stored the OpenAI, Azure OpenAI, and Anthropic details in the [Portkey vault](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/virtual-keys).
## 1. Import the SDK and authenticate Portkey
Start by installing the `portkey-ai` to your NodeJS project.
```sh
npm i --save portkey-ai
```
Once installed, you can import it and instantiate it with the API key to your Portkey account.
```js
import { Portkey } from 'portkey-ai';
const portkey = new Portkey({
apiKey: process.env['PORTKEYAI_API_KEY']
});
```
## 2. Create Configs: Loadbalance with Nested Fallbacks
Portkey acts as AI gateway to all of your requests to LLMs. It follows the OpenAI SDK signature in all of it’s methods and interfaces making it easy to use and switch. Here is an example of an chat completions requests through Portkey.
```js
const response = await portkey.chat.completions.create({
messages,
model: 'gpt-3.5-turbo'
});
```
The Portkey AI gateway can apply our desired behaviour to the requests to various LLMs. In a nutshell, our desired behaviour is the following:
![Flow diagram for loadbalancing and fallbacks](../../docs/images/cookbooks/resilient-loadbalance.png)
Lucky for us, all of this can implemented by passing a configs allowing us to express what behavior to apply to every request through the Portkey AI gateway.
```js
const config = {
strategy: {
mode: 'loadbalance'
},
targets: [
{
virtual_key: process.env['ANTHROPIC_VIRTUAL_KEY'],
weight: 0.5,
override_params: {
max_tokens: 200,
model: 'claude-3-opus-20240229'
}
},
{
strategy: {
mode: 'fallback'
},
targets: [
{
virtual_key: process.env['OPENAI_VIRTUAL_KEY']
},
{
virtual_key: process.env['AZURE_OPENAI_VIRTUAL_KEY']
}
],
weight: 0.5
}
]
};
const portkey = new Portkey({
apiKey: process.env['PORTKEYAI_API_KEY'],
config // pass configs as argument
});
```
We apply the `loadbalance` strategy across _Anthropic and OpenAI._ `weight` describes the traffic should be split into 50/50 among both the LLM providers while `override_params` will help us override the defaults.
Let’s take this a step further to apply a fallback mechanism for the requests from* OpenAI* to fallback to _Azure OpenAI_. This nested mechanism among the `targets` will ensure our app is reliable in the production in great confidence.
See the documentation for Portkey [Fallbacks](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/fallbacks) and [Loadbalancing](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/load-balancing).
## 3. Make a Request
Now that the `config` ‘s are concrete and are passed as arguments when instantiating the Portkey client instance, all subsequent will acquire desired behavior auto-magically — No additional changes to the codebase.
```js
const messages = [
{
role: 'system',
content: 'You are a very helpful assistant.'
},
{
role: 'user',
content: 'What are 7 wonders in the world?'
}
];
const response = await portkey.chat.completions.create({
messages,
model: 'gpt-3.5-turbo'
});
console.log(response.choices[0].message.content);
// The Seven Wonders of the Ancient World are:
```
Next, we will examine how to identify load-balanced requests or those that have been executed as fallbacks.
## 4. Trace the request from the logs
It can be challenging to identify particular requests from the thousands that are received every day, similar to trying to find a needle in a haystack. However, Portkey offers a solution by enabling us to attach a desired trace ID. Here `request-loadbalance-fallback`.
```js
const response = await portkey.chat.completions.create(
{
messages,
model: 'gpt-3.5-turbo'
},
{
traceID: 'request-loadbalance-fallback'
}
);
```
This trace ID can be used to filter requests from the Portkey Dashboard (>Logs) easily.
![logs-request-loadbalance-fallback](../../docs/images/cookbooks/resilient-loadbalance-1.png)
In addition to activating Loadbalance (icon), the logs provide essential observability information, including tokens, cost, and model.
Are the configs growing and becoming harder to manage in the code? [Try creating them from Portkey UI](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/configs#creating-configs) and reference the configs ID in your code. It will make it significantly easier to maintain.
## 5. Advanced: Canary Testing
Given there are new models coming every day and your app is in production — What is the best way to try the quality of those models? Canary Testing allows you to gradually roll out a change to a small subset of users before making it available to everyone.
Consider this scenario: You have been using OpenAI as your LLM provider for a while now, but are considering trying an open-source Llama model for your app through Anyscale.
```js
const config = {
strategy: {
mode: 'loadbalance'
},
targets: [
{
virtual_key: process.env['OPENAI_VIRTUAL_KEY'],
weight: 0.9
},
{
virtual_key: process.env['ANYSCALE_VIRTUAL_KEY'],
weight: 0.1,
override_params: {
model: 'meta-llama/Llama-2-70b-chat-hf'
}
}
]
};
const portkey = new Portkey({
apiKey: process.env['PORTKEYAI_API_KEY'],
config
});
const response = await portkey.chat.completions.create(
{
messages,
model: 'gpt-3.5-turbo'
},
{
traceID: 'canary-testing'
}
);
console.log(response.choices[0].message.content);
```
The `weight` , indication of traffic is split to have 10% of your user-base are served from Anyscale’s Llama models. Now, you are all set up to get feedback and observe the performance of your app and release increasingly to larger userbase.
## Considerations
You can implement production-grade Loadbalancing and nested fallback mechanisms with just a few lines of code. While you are equipped with all the tools for your next GenAI app, here are a few considerations:
- Every request has to adhere to the LLM provider’s requirements for it to be successful. For instance, `max_tokens` is required for Anthropic and not for OpenAI.
- While loadbalance helps reduce the load on one LLM - it is recommended to pair it with a Fallback strategy to ensure that your app stays reliable
- On Portkey, you can also pass the loadbalance weight as 0 - this will essentially stop routing requests to that target and you can amp it up when required
- Loadbalance has no target limits as such, so you can potentially add multiple account details from one provider and effectively multiply your available rate limits
- Loadbalance does not alter the outputs or the latency of the requests in any way
Happy Coding!
<details>
<summary>See the entire code</summary>
```js
import { Portkey } from 'portkey-ai';
const config = {
strategy: {
mode: 'loadbalance'
},
targets: [
{
virtual_key: process.env['ANTHROPIC_VIRTUAL_KEY'],
weight: 0.5,
override_params: {
max_tokens: 200,
model: 'claude-2.1'
}
},
{
strategy: {
mode: 'fallback'
},
targets: [
{
virtual_key: process.env['OPENAI_VIRTUAL_KEY']
},
{
virtual_key: process.env['AZURE_OPENAI_VIRTUAL_KEY']
}
],
weight: 0.5
}
]
};
const portkey = new Portkey({
apiKey: process.env['PORTKEYAI_API_KEY'],
config
});
const messages = [
{
role: 'system',
content: 'You are a very helpful assistant.'
},
{
role: 'user',
content: 'What are 7 wonders in the world?'
}
];
const response = await portkey.chat.completions.create(
{
messages,
model: 'gpt-3.5-turbo'
},
{
traceID: 'request-loadbalance-fallback'
}
);
console.log(response.choices[0].message.content);
```
</details>
cookbook/getting-started/writing-your-first-gateway-config.md
Vendored
+336
View File
@@ -0,0 +1,336 @@
# 101 on Portkey's Gateway Configs
You are likely familiar with how to make an API call to GPT4 for chat completions. However, did you know you can **set up** automatic retries for requests that might fail on OpenAI’s end using Portkey?
The Portkey AI gateway provides several useful features that you can use to enhance your requests. In this cookbook, we will start by making an API call to LLM and explore how Gateway Configs can be utilized to optimize these API calls.
## 1. API calls to LLMs with Portkey
Consider a typical API call to GPT4 to get chat completions using OpenAI SDK. It takes `messages` and `model` arguments to get us a response. If you have tried one before, the following code snippet should look familiar. That’s because Portkey Client SDK follows the same signature as OpenAI’s.
```js
import { Portkey } from 'portkey-ai';
const portkey = new Portkey({
apiKey: 'xxxxxxxtrk',
virtualKey: 'ma5xfxxxxx4x'
});
const messages = [
{
role: 'user',
content: `What are the 7 wonders of the world?`
}
];
const response = await portkey.chat.completions.create({
messages,
model: 'gpt-4'
});
console.log(response.choices[0].message.content);
```
Along with Portkey API Key ([get one](https://portkey.ai/docs/api-reference/authentication#obtaining-your-api-key)), you might’ve noticed a new parameter while instantiating the `portkey` variable — `virtualKey`. Portkey securely stores API keys of LLM providers in a vault and substitutes them at runtime in your requests. These unique identifiers to your API keys are called Virtual Keys. For more information, see the [docs](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/virtual-keys#creating-virtual-keys).
With basics out of our way, let’s jump into applying what we set out to do in the first place with the AI gateway — To automatically retry our request when we hit rate-limits (429 status codes).
## 2. Apply Gateway Configs
The AI gateway requires instructions to automatically retry requests. This involves providing Gateway Configs, which are essentially JSON objects that orchestrate the AI gateway. In our current scenario, we are targeting GPT4 with requests that have automatic retries on 429 status codes.
```json
{
"retry": {
"attempts": 3,
"on_status_codes": [429]
}
}
```
We now have our Gateway Configs sorted. But how do we instruct our AI gateway?
You guessed it, on the request headers. The next section will explore two ways to create and reference Gateway Configs.
### a. Reference Gateway Configs from the UI
Just as the title says — you create them on the UI and use an ID to have Portkey automatically apply in the request headers to instruct the AI gateway. UI builder features lint suggestions, makes it easy to reference (through config ID), eliminates manual management, and allows you to view version history.
To create Gateway Configs,
1. Go to **portkey.ai** and
2. Click on **Configs**
1. Select **Create**
2. Choose any name (such as request_retries)
Write the configs in the playground and click **Save Config**:
![Config Builder](../../docs/images/cookbooks/101-configs-1.png)
See the saved configs in the list along with the `ID`:
![Config ID](../../docs/images/cookbooks/101-configs-2.png)
Try it out now!
The Configs saved will appear as a row item on the Configs page. The `ID` is important as it is referenced in our calls through the AI gateway.
#### Portkey SDK
The Portkey SDK accepts the config parameter that takes the created config ID as it’s argument. To ensure all requests have automatic retries enabled on them, pass the config ID as argument when `portkey` is instantiated.
That’s right! One line of code, and all the request from your apps now inherit Gateway Configs and demonstrate automatic retries.
Let’s take a look at the code snippet:
```js
import { Portkey } from 'portkey-ai';
const portkey = new Portkey({
apiKey: 'xxxxxxrk',
virtualKey: 'xxxxx',
config: 'pc-xxxxx-edx21x' // Gateway Configs
});
const messages = [
{
role: 'user',
content: `What are the 7 wonders of the world?`
}
];
const response = await portkey.chat.completions.create({
messages,
model: 'gpt-4'
});
console.log(response.choices[0].message.content);
```
#### Axios
In the cases, where you are not able to use an SDK, you can pass the same configs as headers with the key `x-portkey-config` .
```js
const CONFIG_ID = 'pc-reques-edf21c';
const PORTKEY_API_KEY = 'xxxxxrk';
const OPENAI_API_KEY = 'sk-*******';
const response = await axios({
method: 'post',
url: 'https://api.portkey.ai/v1/chat/completions',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${OPENAI_API_KEY}`,
'x-portkey-api-key': PORTKEY_API_KEY,
'x-portkey-provider': 'openai',
'x-portkey-config': CONFIG_ID
},
data: data
});
console.log(response.data);
```
#### OpenAI SDK
Portkey can be used with OpenAI SDK.
To send a request with using OpenAI SDK client and apply gateway configs to the request pass a `baseURL` and necessary headers as follows:
```js
import OpenAI from 'openai'; // We're using the v4 SDK
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai';
const PORTKEY_API_KEY = 'xxxxxrk';
const CONFIG_ID = 'pc-reques-edf21c';
const messages = [
{
role: 'user',
content: `What are the 7 wonders of the world?`
}
];
const openai = new OpenAI({
apiKey: 'OPENAI_API_KEY', // When you pass the parameter `virtualKey`, this value is ignored.
baseURL: PORTKEY_GATEWAY_URL,
defaultHeaders: createHeaders({
provider: 'openai',
apiKey: PORTKEY_API_KEY,
virtualKey: 'open-ai-key-04ba3e', // OpenAI virtual key
config: CONFIG_ID
})
});
const chatCompletion = await openai.chat.completions.create({
messages,
model: 'gpt-4'
});
console.log(chatCompletion.choices[0].message.content);
```
The approach to declare the Gateway Configs in the UI and reference them in the code is recommended since it keeps the Configs atomic and decoupled from the business logic and can be upgraded to add more features. What if you want to enable caching for all your thousands of requests? Just update the Configs from the UI. No commits. No redeploys.
### b. Reference Gateway Configs in the Code
Depending on the dynamics of your app, you might want to construct the Gateway Configs at the runtime. All you need to do is to pass the Gateway Configs directly to the `config` parameter as an argument.
#### Portkey SDK
```js
import { Portkey } from 'portkey-ai';
const portkey = new Portkey({
apiKey: 'xxxxxxx',
virtualKey: 'maxxxxx8f4d',
config: JSON.stringify({
retry: {
attempts: 3,
on_status_codes: [429]
}
})
});
const messages = [
{
role: 'user',
content: `What are the 7 wonders of the world?`
}
];
const response = await portkey.chat.completions.create({
messages,
model: 'gpt-4'
});
console.log(response.choices[0].message.content);
```
#### Axios
```js
import axios from 'axios';
const CONFIG_ID = {
retry: {
attempts: 3,
on_status_codes: [429]
}
};
const PORTKEY_API_KEY = 'xxxxxxxx';
const OPENAI_API_KEY = 'sk-xxxxxxxxx';
const data = {
model: 'gpt-4',
messages: [
{
role: 'user',
content: 'What are 7 wonders of the world?'
}
]
};
const { data: response } = await axios({
method: 'post',
url: 'https://api.portkey.ai/v1/chat/completions',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${OPENAI_API_KEY}`,
'x-portkey-api-key': PORTKEY_API_KEY,
'x-portkey-provider': 'openai',
'x-portkey-config': JSON.stringify(CONFIG_ID)
},
data: data
});
console.log(response.choices[0].message.content);
```
#### OpenAI SDK
```js
import OpenAI from 'openai'; // We're using the v4 SDK
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai';
const PORTKEY_API_KEY = 'xxxxxrk';
const CONFIG_ID = 'pc-reques-edf21c';
const messages = [
{
role: 'user',
content: `What are the 7 wonders of the world?`
}
];
const openai = new OpenAI({
apiKey: 'OPENAI_API_KEY', // When you pass the parameter `virtualKey`, this value is ignored.
baseURL: PORTKEY_GATEWAY_URL,
defaultHeaders: createHeaders({
provider: 'openai',
apiKey: PORTKEY_API_KEY,
virtualKey: 'open-ai-key-04ba3e', // OpenAI virtual key
config: {
retry: {
attempts: 3,
on_status_codes: [429]
}
}
})
});
const chatCompletion = await openai.chat.completions.create({
messages,
model: 'gpt-4'
});
console.log(chatCompletion.choices[0].message.content);
```
Those are three ways to use Gateway Configs in your requests.
In the cases where you want to specifically add a config for a specific request instead of all, Portkey allows you to pass `config` argument as seperate objects right at the time of chat completions call instead of `Portkey({..})` instantiation.
```js
const response = await portkey.chat.completions.create(
{
messages,
model: 'gpt-4'
},
{
config: 'config_id' // or expanded Config Object
}
);
```
Applying retry super power to your requests is that easy!
## Next Steps: Dive into features of AI gateway
Great job on implementing the retry behavior for your LLM calls to OpenAI!
Gateway Configs is a tool that can help you manage fallbacks, request timeouts, load balancing, caching, and more. With Portkey's support for over 100+ LLMs, it is a powerful tool for managing complex use cases that involve multiple target configurations. A Gateway Config that encompasses such complexity may look like:
```
TARGET 1 (root):
OpenAI GPT4
Simple Cache
On 429:
TARGET 2 (loadbalance):
Anthropic Claude3
Semantic Cache
On 5XX
TARGET 3 (loadbalance):
Anyscale Mixtral 7B
On 4XX, 5XX
TARGET 4 (fallback):
Llama Models
Automatic Retries
Request Timeouts
```
For complete reference, refer to the _[Config Object](https://portkey.ai/docs/api-reference/config-object)_.
It's exciting to see all the AI gateway features available for your requests. Feel free to experiment and make the most of them. Keep up the great work!
cookbook/guardrails/Langchain Chatbot with PII Guardrails.ipynb
Vendored
+481
View File
@@ -0,0 +1,481 @@
{
"cells": [
{
"cell_type": "markdown",
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1YnDHZY6owEypgSgvJJgXPcntxOjy9GpS?usp=sharing)\n"
],
"metadata": {
"id": "L81TWFYhHM-K"
}
},
{
"cell_type": "markdown",
"source": [
"# Build a Chatbot with Guardrails: Using Langchain and Portkey to Enforce PII Detection and more\n",
"\n",
"In this tutorial, we'll create two versions of a customer support chatbot:\n",
"\n",
"1. A basic chatbot without PII protection\n",
"2. An enhanced chatbot using LangChain and Portkey guardrails to protect sensitive information\n",
"\n",
"By comparing these two implementations, we'll demonstrate the importance and effectiveness of using guardrails to prevent PII exposure in AI applications.\n",
"\n",
"\n",
"Portkey's **Guardrails** offer real-time enforcement of LLM behavior, with features including:\n",
"\n",
"- **Regex matching** and **JSON Schema validation**\n",
"- **Code detection**\n",
"- **Custom guardrail**s: Integrate your existing guardrail systems custom guardrail integration\n",
"-**LLM-based guardrails** (e.g., gibberish detection, prompt injection scanning)\n",
"\n",
"With **20+ deterministic guardrail**s and integrations with platforms like **Aporia, Pillar, Patronus AI and Prompt Security,** Portkey provides comprehensive AI safety solutions. Guardrails can be configured for inputs, outputs, or both, with actions ranging from request denial to alternative LLM fallbacks.\n",
"\n",
"For more details, visit the [Portkey Guardrails documentation](https://docs.portkey.ai/docs/product/guardrails/list-of-guardrail-checks/patronus-ai).\n",
"\n",
"\n",
"Let's build our guardrail-protected chatbot!"
],
"metadata": {
"id": "cWj7GyUW76uP"
}
},
{
"cell_type": "markdown",
"source": [
"# Chatbot Without Guardrails\n",
"\n",
"Before we implement PII protection with Portkey guardrails, let's create a basic chatbot using LangChain. This version will not have any PII detection or protection mechanisms.\n",
"\n",
"## Step 1: Setting up the environment\n",
"\n",
"First, let's install the necessary packages:"
],
"metadata": {
"id": "WYXKs8sQGy3H"
}
},
{
"cell_type": "code",
"source": [
"!pip install -qU langchain-openai langchain langchain_community"
],
"metadata": {
"id": "P-SvF2AKGyXU"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"#Step 2: Importing required libraries and setting up API keys"
],
"metadata": {
"id": "Etxbrc-CG42W"
}
},
{
"cell_type": "code",
"source": [
"import os\n",
"from typing import Dict\n",
"from langchain_openai import ChatOpenAI\n",
"from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder\n",
"from langchain_core.chat_history import BaseChatMessageHistory\n",
"from langchain_core.runnables.history import RunnableWithMessageHistory\n",
"from langchain_core.chat_history import InMemoryChatMessageHistory\n",
"\n",
"\n",
"model = ChatOpenAI(\n",
" api_key=\"YOUR_OPENAI_KEY\",\n",
" model=\"gpt-3.5-turbo\"\n",
")"
],
"metadata": {
"id": "sAf52HvBG6nS"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Create a chat prompt template\n",
"prompt = ChatPromptTemplate.from_messages([\n",
" (\"system\", \"You are a helpful assistant. Answer all questions to the best of your ability.\"),\n",
" MessagesPlaceholder(variable_name=\"messages\"),\n",
"])\n",
"\n",
"# Create a chain that combines the prompt and the model\n",
"chain = prompt | model\n",
"\n",
"# Create a dictionary to store chat histories for different sessions\n",
"store: Dict[str, BaseChatMessageHistory] = {}\n",
"\n",
"def get_session_history(session_id: str) -> BaseChatMessageHistory:\n",
" if session_id not in store:\n",
" store[session_id] = InMemoryChatMessageHistory()\n",
" return store[session_id]\n",
"\n",
"# Wrap the chain with message history\n",
"with_message_history = RunnableWithMessageHistory(\n",
" chain,\n",
" get_session_history,\n",
" input_messages_key=\"messages\",\n",
")\n",
"\n",
"# Function to chat with the bot\n",
"def chat_with_bot(session_id: str, user_input: str):\n",
" config = {\"configurable\": {\"session_id\": session_id}}\n",
" response1 = with_message_history.invoke(\n",
" {\"messages\": [{\"content\": user_input, \"type\": \"human\"}]},\n",
" config=config\n",
" )\n",
" return response1.content"
],
"metadata": {
"id": "_cA4eFpyG8st"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"Step 4: Running the chatbot"
],
"metadata": {
"id": "Cc3A6g9fHBlP"
}
},
{
"cell_type": "code",
"source": [
"# Example usage\n",
"if __name__ == \"__main__\":\n",
" session_id = \"abc123\"\n",
"\n",
" print(\"Chatbot Without Guardrails\")\n",
" print(\"Type 'exit', 'quit', or 'bye' to end the conversation.\")\n",
"\n",
" while True:\n",
" user_input1 = input(\"You: \")\n",
" if user_input1.lower() in ['exit', 'quit', 'bye']:\n",
" print(\"Bot: Goodbye!\")\n",
" break\n",
"\n",
" response1 = chat_with_bot(session_id, user_input1)\n",
" print(f\"Bot: {response1}\")"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "VrQjj6SFHAz_",
"outputId": "04876806-525a-4d4e-c10f-cb325b3efb1e"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Chatbot Without Guardrails\n",
"Type 'exit', 'quit', or 'bye' to end the conversation.\n",
"You: hey\n",
"Bot: Hello! How can I assist you today?\n",
"You: \"My name is Siddharth and my email id is xyz@gmail.com\n",
"Bot: Hello Siddharth! It's nice to meet you. How can I help you today?\n",
"You: what is my email\n",
"Bot: Your email is xyz@gmail.com.\n",
"You: exit\n",
"Bot: Goodbye!\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"# Build a Chatbot with Guardrails: Using Langchain and Portkey to Enforce PII Detection and more\n",
"\n",
"In this tutorial, we'll create two versions of a customer support chatbot:\n",
"\n",
"1. A basic chatbot without PII protection\n",
"2. An enhanced chatbot using LangChain and Portkey guardrails to protect sensitive information\n",
"\n",
"By comparing these two implementations, we'll demonstrate the importance and effectiveness of using guardrails to prevent PII exposure in AI applications.\n"
],
"metadata": {
"id": "4BJczP3_HFuN"
}
},
{
"cell_type": "markdown",
"source": [
"# Setting Up Portkey Guardrails\n",
"\n",
"Before building our chatbot, let's configure Portkey guardrails for PII detection:\n",
"\n",
"1. Sign Up for [portkey.ai](https://portkey.ai) (the dev account is **free** ferver) 🎊\n",
"\n",
"\n",
"2. Enable **Patronus AI** in the [Integrations page](https://app.portkey.ai/integrations) on Portkey App\n",
"\n",
"\n",
"3. Navigate to the [Guardrails dashboard](https://app.portkey.ai/guardrails)\n",
"\n",
"\n",
"4. Create a new guardrail, by selecting \"**Detect PII Guardrail**\" by Patronus AI\n",
"\n",
"\n",
"5. In the Actions tab, select \"**Deny the request if guardrail fails**\" from the Settings menu.\n",
"\n",
"6. Name your guardrail, create it, and then copy your Config ID for future use.\n",
"\n",
"\n",
"\n",
"This **Config ID** is crucial for integrating PII protection into our LangChain-Portkey chatbot. With this setup, we're ready to create a secure, PII-aware conversational AI.\n",
"\n",
"\n",
"<img src=\"https://raw.githubusercontent.com/siddharthsambharia-portkey/Portkey-Product-Images/main/Portkey_PII_Guardrails.png\" alt=\"O\" width=\"600\" />\n",
"\n",
"<img src=\"https://raw.githubusercontent.com/siddharthsambharia-portkey/Portkey-Product-Images/main/actions-gaurdrails.png\" alt=\"O\" width=\"600\" />\n",
"\n"
],
"metadata": {
"id": "RhXlcYb5NFqj"
}
},
{
"cell_type": "markdown",
"source": [
"# Step 1: Setting up the environment\n",
"\n",
"First, let's install the necessary packages:"
],
"metadata": {
"id": "_x0jcajG76AY"
}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "kXamAIlNgU7J",
"collapsed": true
},
"outputs": [],
"source": [
"!pip install -qU langchain-openai portkey-ai langchain langchain_community"
]
},
{
"cell_type": "markdown",
"source": [
"# Step 2: Importing required libraries and setting up API keys\n",
"\n",
"In this step, we'll import the necessary libraries and set up our API keys. We'll also create Portkey config object to add guardrails to protect against PII exposure. Get your Portkey API key from [here](https://app.portkey.ai/api-keys)."
],
"metadata": {
"id": "E943mJcZG7SX"
}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VZShDZzRr-yo",
"collapsed": true
},
"outputs": [],
"source": [
"import os\n",
"from typing import Dict\n",
"from langchain_openai import ChatOpenAI\n",
"from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder\n",
"from langchain_core.chat_history import BaseChatMessageHistory\n",
"from langchain_core.runnables.history import RunnableWithMessageHistory\n",
"from portkey_ai import createHeaders, PORTKEY_GATEWAY_URL\n",
"from langchain_core.chat_history import InMemoryChatMessageHistory\n",
"\n",
"\n",
"portkey_config={\n",
"\t\"after_request_hooks\": [\n",
"\t\t{\n",
"\t\t\t\"id\": \"pg-portke-416feb\" #YOUR_CONFIG_ID\n",
"\t\t}\n",
"\t]\n",
"}\n",
"\n",
"\n",
"llm = ChatOpenAI(\n",
" api_key=\"YOUR_OPENAI_API_KEY\",\n",
" base_url=PORTKEY_GATEWAY_URL,\n",
" model=\"gpt-3.5-turbo\",\n",
" default_headers=createHeaders(\n",
" provider=\"openai\",\n",
" api_key=\"Your_Portkey_API_key\",\n",
" config=portkey_config,\n",
" )\n",
")\n"
]
},
{
"cell_type": "markdown",
"source": [
"This code sets up our environment with the necessary configurations for Portkey guardrails. The portkey_config defines our guardrail settings, including retry attempts, caching, and hooks for request and response processing.\n",
"\n"
],
"metadata": {
"id": "IeIvEQiIG9y-"
}
},
{
"cell_type": "markdown",
"source": [
"# Step 3: Creating the chat prompt template and chain\n",
"\n",
"Now, let's set up our chat prompt template and create a chain that combines the prompt with our Portkey-protected model. This structure allows us to maintain a conversation history while applying guardrails to each interaction."
],
"metadata": {
"id": "TdMZ3Mx378rd"
}
},
{
"cell_type": "code",
"source": [
"\n",
"# Create a chat prompt template\n",
"prompt = ChatPromptTemplate.from_messages([\n",
" (\"system\", \"You are a helpful assistant. Answer all questions to the best of your ability. Reply with the user info given in the request\"),\n",
" MessagesPlaceholder(variable_name=\"messages\"),\n",
"])\n",
"\n",
"# Create a chain that combines the prompt and the model\n",
"chain1 = prompt | llm\n",
"\n",
"# Create a dictionary to store chat histories for different sessions\n",
"store: Dict[str, BaseChatMessageHistory] = {}\n",
"\n",
"def get_session_history_1(session_id: str) -> BaseChatMessageHistory:\n",
" if session_id not in store:\n",
" store[session_id] = InMemoryChatMessageHistory()\n",
" return store[session_id]\n",
"\n",
"# Wrap the chain with message history\n",
"with_message_history_1 = RunnableWithMessageHistory(\n",
" chain1,\n",
" get_session_history_1,\n",
" input_messages_key=\"messages\",\n",
")\n",
"\n",
"# Function to chat with the bot\n",
"def chat_with_bot1(session_id: str, user_input: str):\n",
" config = {\"configurable\": {\"session_id\": session_id}}\n",
" response = with_message_history_1.invoke(\n",
" {\"messages\": [{\"content\": user_input, \"type\": \"human\"}]},\n",
" config=config\n",
" )\n",
" return response.content\n",
"\n"
],
"metadata": {
"id": "oL9gBUBu7iLL"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"This code creates a chat prompt template, sets up a chain combining the prompt and the Portkey-protected model, and establishes a system for managing chat history. The chat_with_bot function handles individual interactions, applying our guardrails to each message.\n"
],
"metadata": {
"id": "BdxJenOI79hs"
}
},
{
"cell_type": "markdown",
"source": [
"# Step 4: Running the chatbot\n",
"Finally, we'll implement the main loop for our chatbot. This will allow users to interact with the bot while our Portkey guardrails work in the background to protect sensitive information.\n",
"\n",
"Try using this prompt to see guardrails in action- \"My name is Siddharth and my email id is xyz@gmail.com\""
],
"metadata": {
"id": "D02MN0oLHE2x"
}
},
{
"cell_type": "code",
"source": [
"# Example usage\n",
"if __name__ == \"__main__\":\n",
" session_id = \"testWithGuradrails\"\n",
"\n",
" while True:\n",
" user_input = input(\"You: \")\n",
" if user_input.lower() in ['exit', 'quit', 'bye']:\n",
" print(\"Bot: Goodbye!\")\n",
" break\n",
"\n",
" response = chat_with_bot1(session_id, user_input)\n",
" print(f\"Bot: {response}\")"
],
"metadata": {
"collapsed": true,
"id": "Z2w0DB0b7j6g"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"## Conclusion\n",
"\n",
"Here is how the traces of your request looks like on your Portkey dashboard-\n",
"\n",
"\n",
"<img src=\"https://raw.githubusercontent.com/siddharthsambharia-portkey/Portkey-Product-Images/main/PII-fail-traces-guardrails.png\" alt=\"O\" width=\"600\" />\n",
"\n",
"Portkey helps you connect to 250+ LLMs with just 2 lines of code. It helps you build robust aps with built in observability and guardrails.\n",
"\n",
"In this tutorial, we've created two versions of a chatbot:\n",
"\n",
"1. A basic chatbot without PII protection\n",
"2. An enhanced chatbot with Portkey guardrails\n",
"\n",
"\n",
"<img src=\"https://raw.githubusercontent.com/siddharthsambharia-portkey/Portkey-Product-Images/main/Portkey-Dashboard.png\" alt=\"O\" width=\"600\" />\n",
"\n",
"\n",
"\n",
"The first version demonstrates a simple implementation using LangChain, but it lacks any mechanism to detect or protect sensitive information. This could potentially lead to the exposure of PII in logs or responses.\n",
"\n",
"The second version, integrating Portkey with our LangChain-based chatbot, significantly enhances our AI application's capabilities, reliability, and security. By implementing PII detection guardrails, we've added a crucial layer of protection against the unintended exposure of sensitive information.\n",
"\n",
"By comparing these two implementations, we can clearly see the benefits of using Portkey guardrails in AI applications, especially when dealing with potentially sensitive user inputs."
],
"metadata": {
"id": "A3BOYCMwFoxF"
}
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
cookbook/integrations/Instructor_with_Portkey.ipynb
Vendored
+150
View File
@@ -0,0 +1,150 @@
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
}
},
"cells": [
{
"cell_type": "markdown",
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1jaemmsUi8TnrK6so6pvvDG6e666QIng2?usp=sharing)\n"
],
"metadata": {
"id": "03GPOrjb8FJ1"
}
},
{
"cell_type": "markdown",
"source": [
"# Get structured outputs from 100+ LLMs"
],
"metadata": {
"id": "QUvn05wUXBbA"
}
},
{
"cell_type": "markdown",
"source": [
"[**Instructor**](https://github.com/jxnl/instructor) is a Python library for getting structured outputs from LLMs. Built on top of Pydantic, it provides a simple, transparent, and user-friendly API to manage validation, retries, and streaming responses.\n",
"\n",
"<br>\n",
"\n",
"**Portkey** is an open source [**AI Gateway**](https://github.com/Portkey-AI/gateway) that helps you manage access to 250+ LLMs through a unified API while providing visibility into\n",
"\n",
"✅ cost \n",
"✅ performance \n",
"✅ accuracy metrics\n",
"\n",
"This notebook demonstrates how you can get structured outputs from 100s of LLMs using Portkey's AI Gateway."
],
"metadata": {
"id": "YhM8C8VDXG2Y"
}
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"id": "uQLdtnFhWbIE",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "b0e43ab5-fd2a-443b-a5e2-e10383289e9a"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.1/53.1 kB\u001b[0m \u001b[31m1.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m405.9/405.9 kB\u001b[0m \u001b[31m5.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m328.3/328.3 kB\u001b[0m \u001b[31m7.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m327.6/327.6 kB\u001b[0m \u001b[31m10.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.6/75.6 kB\u001b[0m \u001b[31m1.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.7/12.7 MB\u001b[0m \u001b[31m20.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.9/77.9 kB\u001b[0m \u001b[31m3.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m1.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25h"
]
}
],
"source": [
"!pip install -qU instructor portkey-ai openai jsonref"
]
},
{
"cell_type": "markdown",
"source": [
"### Structured Outputs for OpenAI models"
],
"metadata": {
"id": "80K8rNtgdvyy"
}
},
{
"cell_type": "code",
"source": [
"import instructor\n",
"from pydantic import BaseModel\n",
"from portkey_ai import Portkey\n",
"from openai import OpenAI\n",
"from google.colab import userdata\n",
"from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders\n",
"\n",
"portkey = OpenAI(\n",
" base_url=PORTKEY_GATEWAY_URL,\n",
" api_key = \"X\",\n",
" default_headers=createHeaders(\n",
" virtual_key= \"open-ai-key-fb040b\",\n",
" api_key=userdata.get('PORTKEY_API_KEY')\n",
"\n",
" )\n",
")\n",
"\n",
"class User(BaseModel):\n",
" name: str\n",
" age: int\n",
"\n",
"client = instructor.from_openai(portkey)\n",
"\n",
"user_info = client.chat.completions.create(\n",
" model=\"gpt-3.5-turbo\",\n",
" max_tokens=1024,\n",
" response_model=User,\n",
" messages=[{\"role\": \"user\", \"content\": \"John Doe is 30 years old.\"}],\n",
")\n",
"\n",
"print(user_info.name)\n",
"print(user_info.age)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "VxjMXaS1cgiY",
"outputId": "9506a983-aa2d-4c01-c4f4-4a4b7d07cc40"
},
"execution_count": 12,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"John Doe\n",
"30\n"
]
}
]
}
]
}
cookbook/integrations/Mistral.ipynb
Vendored
+1
View File
File diff suppressed because one or more lines are too long
cookbook/integrations/Phidata_with_ Perplexity.ipynb
Vendored
+326
View File
File diff suppressed because one or more lines are too long
cookbook/integrations/Phidata_with_Portkey.ipynb
Vendored
+476
View File
File diff suppressed because one or more lines are too long
cookbook/integrations/ReAct Agents Portkey.ipynb
Vendored
+402
View File
File diff suppressed because one or more lines are too long
cookbook/integrations/Sutra_with_Portkey.ipynb
Vendored
+435
View File
@@ -0,0 +1,435 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "title-section"
},
"source": [
"# SUTRA x Portkey: The Gateway Cookbook\n",
"\n",
"<img src=\"https://play-lh.googleusercontent.com/_O9p4Z4yucA2NLmZBu9mTJCuBwXeT9NcbtrDN6I8gKlkIPRySV0adOmbyipjSj9Gew\" width=\"150\">\n",
"\n",
"<img src=\"https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcRX8V9wZZ69LrmJjm8VmbSw_2FnBbUOtXDAAQ&s\" width=\"150\">\n",
"\n",
"\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/11FLZl0gScugIpXpC0M9tMDWn9C_osZ9A?usp=sharing)\n",
"\n",
"This notebook provides a simple, easy-to-follow guide for using Sutra-v2 models with Portkey's AI Gateway. We'll focus on the basics to get you up and running quickly."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kfUMivYLrMSe"
},
"source": [
"## Get Your API Keys\n",
"\n",
"Before you begin, make sure you have:\n",
"\n",
"1. A SUTRA API key (Get yours at [TWO AI's SUTRA API](https://app.portkey.ai/api-keys))\n",
"2. A Portkey API Key (Get yours at [PORTKEY API](https://www.two.ai/sutra/api))\n",
"2. Basic familiarity with Python and Jupyter notebooks\n",
"\n",
"This notebook is designed to run in Google Colab, so no local Python installation is required."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "setup-section"
},
"source": [
"## 1. Install Required Packages\n",
"\n",
"First, let's install the Portkey and OpenAI packages:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "install-packages",
"outputId": "0ecad99f-ee1a-4282-c2cc-b665655836d8"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[?25l \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/756.9 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K \u001b[91m━━━━━━━━━━━━\u001b[0m\u001b[90m╺\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m235.5/756.9 kB\u001b[0m \u001b[31m7.1 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m \u001b[32m747.5/756.9 kB\u001b[0m \u001b[31m13.4 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m756.9/756.9 kB\u001b[0m \u001b[31m7.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25h"
]
}
],
"source": [
"!pip install -q portkey-ai openai"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "api-keys"
},
"source": [
"## 2. Set Up Your API Keys\n",
"\n",
"You'll need both a Sutra API key and a Portkey API key. For security, we'll use environment variables:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "setup-keys"
},
"outputs": [],
"source": [
"import os\n",
"from google.colab import userdata\n",
"\n",
"# Set your API keys\n",
"sutra_api_key = userdata.get('SUTRA_API_KEY')\n",
"portkey_api_key = userdata.get('PORTKEY_API_KEY')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "initialize-portkey"
},
"source": [
"## 3. Initialize Portkey with Sutra-v2\n",
"\n",
"Now we'll set up Portkey to work with Sutra-v2 models:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "portkey-init"
},
"outputs": [],
"source": [
"from portkey_ai import Portkey\n",
"\n",
"\n",
"# Initialize Portkey client with Sutra as the provider\n",
"client = Portkey(\n",
" api_key=portkey_api_key,\n",
" provider=\"openai\", # Using openai provider for Sutra\n",
" base_url=\"https://api.two.ai/v2\", # Sutra API endpoint\n",
" Authorization=sutra_api_key # Sutra API key\n",
")\n",
"\n",
"print(\"Portkey client initialized with Sutra-v2!\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2x3bJTsL_fSV"
},
"outputs": [],
"source": [
"# Make a request through your AI Gateway\n",
"response = client.chat.completions.create(\n",
" messages=[{\"role\": \"user\", \"content\": \"Who is founder of sutra?\"}],\n",
" model=\"sutra-v2\"\n",
")\n",
"response.choices[0].message.content"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "helper-function"
},
"source": [
"## 4. Create a Simple Helper Function\n",
"\n",
"Let's create a simple function to generate text using Sutra-v2 models through Portkey:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "helper-code"
},
"outputs": [],
"source": [
"def ask_sutra(prompt, model=\"sutra-v2\", temperature=0.7, max_tokens=500):\n",
" \"\"\"Simple function to get responses from Sutra-v2 via Portkey\"\"\"\n",
" response = client.chat.completions.create(\n",
" model=model,\n",
" messages=[\n",
" {\"role\": \"user\", \"content\": prompt}\n",
" ],\n",
" temperature=temperature,\n",
" max_tokens=max_tokens\n",
" )\n",
" return response.choices[0].message.content"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "basic-example"
},
"source": [
"## 5. Try a Simple Example\n",
"\n",
"Let's test our setup with a basic prompt:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "test-basic"
},
"outputs": [],
"source": [
"# Test with a simple prompt\n",
"simple_prompt = \"Explain the importance of AI in modern healthcare in India.\"\n",
"response = ask_sutra(simple_prompt)\n",
"print(response)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "multilingual"
},
"source": [
"## 6. Try Multilingual Capabilities\n",
"\n",
"Sutra-v2 excels at Indian languages. Let's test it with a Hindi prompt:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "hindi-example"
},
"outputs": [],
"source": [
"# Hindi prompt\n",
"hindi_prompt = \"भारत में कृत्रिम बुद्धिमत्ता (AI) के महत्व के बारे में बताएं।\"\n",
"hindi_response = ask_sutra(hindi_prompt)\n",
"print(hindi_response)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "creative-writing"
},
"source": [
"## 7. Creative Writing Example\n",
"\n",
"Let's try a creative writing prompt:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "creative-example"
},
"outputs": [],
"source": [
"# Creative writing prompt\n",
"creative_prompt = \"Write a short poem about the beauty of the Himalayas.\"\n",
"creative_response = ask_sutra(creative_prompt)\n",
"print(creative_response)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "basic-portkey-features"
},
"source": [
"## 8. Using Basic Portkey Features\n",
"\n",
"Let's try a simple Portkey feature - automatic retries for reliability:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "retry-example"
},
"outputs": [],
"source": [
"# Configure retries\n",
"retry_config = {\n",
" \"retry\": {\n",
" \"attempts\": 3, # Retry up to 3 times\n",
" }\n",
"}\n",
"\n",
"# Create a client with retry configuration\n",
"retry_client = client.with_options(config=retry_config)\n",
"\n",
"# Function to use the retry-enabled client\n",
"def ask_with_retry(prompt, model=\"sutra-v2\"):\n",
" response = retry_client.chat.completions.create(\n",
" model=model,\n",
" messages=[{\"role\": \"user\", \"content\": prompt}],\n",
" max_tokens=500\n",
" )\n",
" return response.choices[0].message.content"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "C60fUQE3peLj"
},
"source": [
"### Test with a prompt\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-uge8fehpdPr"
},
"outputs": [],
"source": [
"# Test with a prompt\n",
"retry_prompt = \"What are the major festivals celebrated in different regions of India?\"\n",
"retry_response = ask_with_retry(retry_prompt)\n",
"print(retry_response)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "simple-caching"
},
"source": [
"## 9. Simple Caching for Better Performance\n",
"\n",
"Let's implement basic caching to improve response times for repeated queries:\n",
"\n",
"**Note :- Caching is only available on Portkey's hosted gateway and enterprise plans.**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cache-example"
},
"outputs": [],
"source": [
"# Configure simple caching\n",
"cache_config = {\n",
" \"cache\": {\n",
" \"mode\": \"simple\" \n",
" }\n",
"}\n",
"\n",
"# Create a client with caching\n",
"cached_client = client.with_options(config=cache_config)\n",
"\n",
"# Function to demonstrate caching\n",
"def test_simple_caching(prompt):\n",
" import time\n",
"\n",
" print(\"First request (cache miss):\")\n",
" start_time = time.time()\n",
" response1 = cached_client.chat.completions.create(\n",
" model=\"sutra-v2\",\n",
" messages=[{\"role\": \"user\", \"content\": prompt}],\n",
" max_tokens=500\n",
" )\n",
" time1 = time.time() - start_time\n",
" print(f\"Time taken: {time1:.2f} seconds\")\n",
" print(f\"Response: {response1.choices[0].message.content[:150]}...\\n\")\n",
"\n",
" print(\"Second request with same prompt (cache hit):\")\n",
" start_time = time.time()\n",
" response2 = cached_client.chat.completions.create(\n",
" model=\"sutra-v2\",\n",
" messages=[{\"role\": \"user\", \"content\": prompt}],\n",
" max_tokens=500\n",
" )\n",
" time2 = time.time() - start_time\n",
" print(f\"Time taken: {time2:.2f} seconds\")\n",
" print(f\"Response: {response2.choices[0].message.content[:150]}...\")\n",
"\n",
" if time2 < time1:\n",
" print(f\"\\nCaching improved response time by {time1/time2:.1f}x!\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LWBLtKMApYEL"
},
"source": [
"### Test caching\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "4-moy_aWpWzt"
},
"outputs": [],
"source": [
"# Test caching\n",
"cache_prompt = \"Explain the concept of artificial intelligence to a 10-year-old child.\"\n",
"test_simple_caching(cache_prompt)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "conclusion"
},
"source": [
"## 10. Conclusion\n",
"\n",
"In this simple guide, you've learned how to:\n",
"\n",
"1. Set up Portkey with Sutra-v2 models\n",
"2. Create a simple helper function for generating text\n",
"3. Test Sutra-v2's capabilities with different types of prompts\n",
"4. Use basic Portkey features like retries and caching\n",
"\n",
"This integration gives you the best of both worlds: Sutra-v2's powerful language capabilities (especially for Indian languages) and Portkey's reliability features.\n",
"\n"
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
cookbook/integrations/Tool_Use_with_Portkey.ipynb
Vendored
+507
View File
@@ -0,0 +1,507 @@
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
}
},
"cells": [
{
"cell_type": "markdown",
"source": [
"<h1 align=\"center\">\n",
" <a href=\"https://portkey.ai\">\n",
" <img width=\"300\" src=\"https://analyticsindiamag.com/wp-content/uploads/2023/08/Logo-on-white-background.png\" alt=\"portkey\">\n",
" </a>\n",
"</h1>"
],
"metadata": {
"id": "zld1izGSx2rl"
}
},
{
"cell_type": "markdown",
"source": [
"[Portkey](https://app.portkey.ai/) is the Control Panel for AI apps. With it's popular AI Gateway and Observability Suite, hundreds of teams ship reliable, cost-efficient, and fast apps.\n",
"\n",
"With Portkey, you can\n",
"\n",
"- Connect to 200+ models through a unified API,\n",
"- View 40+ metrics & logs for all requests\n",
"- Enable semantic cache to reduce latency & costs\n",
"- Implement automatic retries & fallbacks for failed requests"
],
"metadata": {
"id": "VfWXytp_1EVj"
}
},
{
"cell_type": "markdown",
"source": [
"This notebook demonstrates how to use Portkey for function calling with various LLM providers including OpenAI, Anthropic, and Google's Gemini."
],
"metadata": {
"id": "F1zYiehz1VAl"
}
},
{
"cell_type": "markdown",
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1TeqYiXhlfZvFhoOHWEiyBycfWu0arWwp?usp=sharing)"
],
"metadata": {
"id": "MosN-n4Wx4ZP"
}
},
{
"cell_type": "code",
"source": [
"!pip install -qU openai portkey-ai"
],
"metadata": {
"id": "lha5CiI-kvcu"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"tools = [\n",
" {\n",
" \"type\": \"function\",\n",
" \"function\": {\n",
" \"name\": \"get_current_weather\",\n",
" \"description\": \"Get the current weather in a given location\",\n",
" \"parameters\": {\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"location\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"The city and state, e.g. San Francisco, CA\",\n",
" },\n",
" \"unit\": {\"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"]},\n",
" },\n",
" \"required\": [\"location\"],\n",
" },\n",
" },\n",
" }\n",
" ]"
],
"metadata": {
"id": "_wviaUs5pi8B"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"restaurant_recommendations = [{\n",
" \"type\": \"function\",\n",
" \"function\": {\n",
" \"name\": \"get_restaurant_recommendations\",\n",
" \"description\": \"Get restaurant recommendations for a given location and cuisine type\",\n",
" \"parameters\": {\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"location\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"The city and state, e.g. New York, NY\"\n",
" },\n",
" \"cuisine\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"The type of cuisine, e.g. Italian, Chinese, Mexican\"\n",
" },\n",
" \"price_range\": {\n",
" \"type\": \"string\",\n",
" \"enum\": [\"$\", \"$$\", \"$$$\", \"$$$$\"],\n",
" \"description\": \"The price range of the restaurants\"\n",
" }\n",
" },\n",
" \"required\": [\"location\", \"cuisine\"]\n",
" }\n",
" }\n",
" },\n",
" ]"
],
"metadata": {
"id": "bLeShK0bTj_r"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"latest_ai_news = [{\n",
" \"type\": \"function\",\n",
" \"function\": {\n",
" \"name\": \"get_latest_ai_news\",\n",
" \"description\": \"Retrieve the latest news articles about artificial intelligence\",\n",
" \"parameters\": {\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"topic\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"Specific AI topic (optional, e.g., 'machine learning', 'neural networks', 'robotics')\"\n",
" },\n",
" \"timeframe\": {\n",
" \"type\": \"string\",\n",
" \"enum\": [\"today\", \"this_week\", \"this_month\"],\n",
" \"description\": \"Time period for the news articles\"\n",
" },\n",
" \"source_type\": {\n",
" \"type\": \"string\",\n",
" \"enum\": [\"all\", \"academic\", \"industry\", \"mainstream\"],\n",
" \"description\": \"Type of news sources to include\"\n",
" },\n",
" \"max_results\": {\n",
" \"type\": \"integer\",\n",
" \"description\": \"Maximum number of news articles to return (1-50)\",\n",
" \"minimum\": 1,\n",
" \"maximum\": 50\n",
" },\n",
" \"language\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"Preferred language for the articles (e.g., 'en' for English, 'es' for Spanish)\"\n",
" }\n",
" },\n",
" \"required\": [\"timeframe\"]\n",
" }\n",
" }\n",
"}]"
],
"metadata": {
"id": "kG-wPaBvXDH3"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Initialize Portkey client\n",
"from portkey_ai import Portkey\n",
"from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders\n",
"from google.colab import userdata\n",
"from tabulate import tabulate\n",
"\n",
"providers = [\n",
" (\"openai\", \"gpt-4o\"),\n",
" (\"openai\", \"gpt-4o-mini\"),\n",
" (\"anthropic\", \"claude-3-5-sonnet-20240620\"),\n",
" (\"anthropic\", \"claude-3-opus-20240229\"),\n",
" (\"anthropic\", \"claude-3-haiku-20240307\"),\n",
" (\"google\", \"gemini-1.5-flash-latest\"),\n",
" (\"google\", \"gemini-1.5-pro\"),\n",
" #(\"fireworks-ai\", \"accounts/yi-01-ai/models/yi-large\"),\n",
" (\"fireworks-ai\", \"accounts/fireworks/models/firefunction-v2\"),\n",
" # (\"fireworks-ai\", \"accounts/fireworks/models/codegemma-2b\"),\n",
" # (\"fireworks-ai\", \"accounts/fireworks/models/llama-v2-7b\"),\n",
" #(\"groq\", \"llama3-groq-70b-8192-tool-use-preview\"),\n",
"\n",
" # (\"together-ai\", \"meta-llama/Llama-2-70b-hf\"), #- Error code: 400 - {'error': {'message': 'together-ai error: google/gemma-2-27b-it is not supported for JSON mode/function calling\n",
"\n",
"\n",
" ]\n",
"\n",
"\n",
"portkey = Portkey(api_key=userdata.get('PORTKEY_API_KEY'))\n",
"\n",
"\n",
"def portkey_function_call(messages):\n",
" results = []\n",
"\n",
" for provider, model in providers:\n",
" config = {\n",
" \"provider\": provider.lower(),\n",
" \"api_key\": userdata.get(f'{provider.upper()}_API_KEY')\n",
" }\n",
"\n",
"\n",
" # Make the API call\n",
" response = portkey.with_options(config=config).chat.completions.create(\n",
" model=model,\n",
" messages=messages,\n",
" tools=latest_ai_news,\n",
" tool_choice=\"auto\",\n",
" max_tokens=512,\n",
" )\n",
"\n",
" response_message = response.choices[0].message\n",
" tool_call_response = str(response_message.tool_calls) if response_message.tool_calls else \"No tool calls\"\n",
"\n",
" results.append([model, provider, tool_call_response])\n",
"\n",
" # Print results using tabulate\n",
" print(tabulate(results, headers=[\"Model\", \"Provider\", \"Tool Call Response\"], tablefmt=\"grid\"))"
],
"metadata": {
"id": "PRj-hPyO6jVt"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"#Testing of Get Latest AI News Tool"
],
"metadata": {
"id": "2m_RKBxtYQFL"
}
},
{
"cell_type": "code",
"source": [
"# Test message\n",
"messages = [{\"role\": \"user\", \"content\": \"give me latest ai news of this week\"}]\n",
"\n",
"# Run function calling for all providers\n",
"portkey_function_call(messages)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ySeRNzxRXQ7a",
"outputId": "a353b7f3-babe-4d83-a459-09e4e9c932e2"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"+-------------------------------------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| Model | Provider | Tool Call Response |\n",
"+===========================================+==============+==========================================================================================================================================================================================================================================+\n",
"| gpt-4o | openai | [ChatCompletionMessageToolCall(id='call_9zk75e22eO5NVzq42vAFmawo', function=FunctionCall(arguments='{\"timeframe\":\"this_week\",\"max_results\":5}', name='get_latest_ai_news'), type='function')] |\n",
"+-------------------------------------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| gpt-4o-mini | openai | [ChatCompletionMessageToolCall(id='call_Geq0M1hYFgJmNmp6XbonxcmP', function=FunctionCall(arguments='{\"timeframe\":\"this_week\",\"max_results\":5}', name='get_latest_ai_news'), type='function')] |\n",
"+-------------------------------------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| claude-3-5-sonnet-20240620 | anthropic | [ChatCompletionMessageToolCall(id='toolu_01LmVxtbDtSaQXq27pX9zeCH', function=FunctionCall(arguments='{\"timeframe\":\"this_week\",\"max_results\":10,\"language\":\"en\"}', name='get_latest_ai_news'), type='function')] |\n",
"+-------------------------------------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| claude-3-opus-20240229 | anthropic | [ChatCompletionMessageToolCall(id='toolu_016DEvWVbfCYDNAvntbhCLE7', function=FunctionCall(arguments='{\"timeframe\":\"this_week\"}', name='get_latest_ai_news'), type='function')] |\n",
"+-------------------------------------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| claude-3-haiku-20240307 | anthropic | [ChatCompletionMessageToolCall(id='toolu_01UzArrKvobYz6Bo2CkG9AZW', function=FunctionCall(arguments='{\"timeframe\":\"this_week\",\"max_results\":10,\"source_type\":\"all\"}', name='get_latest_ai_news'), type='function')] |\n",
"+-------------------------------------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| gemini-1.5-flash-latest | google | [ChatCompletionMessageToolCall(id='portkey-69bbbb43-9bd8-442f-a9be-aa926b46458c', function=FunctionCall(arguments='{\"timeframe\":\"this_week\"}', name='get_latest_ai_news'), type='function')] |\n",
"+-------------------------------------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| gemini-1.5-pro | google | [ChatCompletionMessageToolCall(id='portkey-4424fd91-e7c1-4eed-ac7a-8be4b52e0325', function=FunctionCall(arguments='{\"timeframe\":\"this_week\"}', name='get_latest_ai_news'), type='function')] |\n",
"+-------------------------------------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| accounts/fireworks/models/firefunction-v2 | fireworks-ai | [ChatCompletionMessageToolCall(id='call_927xwUgfmPh5vRKqsUoEMVeK', function=FunctionCall(arguments='{\"timeframe\": \"this_week\", \"max_results\": 5, \"source_type\": \"all\", \"language\": \"en\"}', name='get_latest_ai_news'), type='function')] |\n",
"+-------------------------------------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"#Testing of restaurant_recommendations Tool"
],
"metadata": {
"id": "-3xJxlWxYZUL"
}
},
{
"cell_type": "code",
"source": [
"# Test message\n",
"messages = [{\"role\": \"user\", \"content\": \"Suggest latest and best restaurant of Aurangabad?\"}]\n",
"\n",
"# Run function calling for all providers\n",
"portkey_function_call(messages)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Rjmzq6yXUKGc",
"outputId": "aa9b0fce-cc6e-4588-e9df-abce595d8abc"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"+-------------------------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| Model | Provider | Tool Call Response |\n",
"+===========================================+==============+=================================================================================================================================================================================================================================================================+\n",
"| gpt-4o | openai | No tool calls |\n",
"+-------------------------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| gpt-4o-mini | openai | [ChatCompletionMessageToolCall(id='call_LuP8f1JvDdmKO5hOI5xoqsNs', function=FunctionCall(arguments='{\"topic\":\"restaurant\",\"timeframe\":\"this_month\",\"source_type\":\"mainstream\",\"max_results\":5,\"language\":\"en\"}', name='get_latest_ai_news'), type='function')] |\n",
"+-------------------------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| claude-3-5-sonnet-20240620 | anthropic | No tool calls |\n",
"+-------------------------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| claude-3-opus-20240229 | anthropic | No tool calls |\n",
"+-------------------------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| claude-3-haiku-20240307 | anthropic | [ChatCompletionMessageToolCall(id='toolu_018RjfwRm3xTPzWffnCKid35', function=FunctionCall(arguments='{\"language\":\"en\",\"max_results\":5,\"source_type\":\"all\",\"timeframe\":\"today\",\"topic\":\"aurangabad restaurants\"}', name='get_latest_ai_news'), type='function')] |\n",
"+-------------------------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| gemini-1.5-flash-latest | google | No tool calls |\n",
"+-------------------------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| gemini-1.5-pro | google | No tool calls |\n",
"+-------------------------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| accounts/fireworks/models/firefunction-v2 | fireworks-ai | No tool calls |\n",
"+-------------------------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"#Testing of Current Weather Featching Tool"
],
"metadata": {
"id": "60wn4j8DYjU_"
}
},
{
"cell_type": "code",
"source": [
"# Test message\n",
"messages = [{\"role\": \"user\", \"content\": \"How's the weather like in San Francisco?\"}]\n",
"\n",
"# Run function calling for all providers\n",
"portkey_function_call(messages)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "kAYKGo137cwP",
"outputId": "81fd7e38-2c0f-49d5-e1a7-8d2d45b38fc7"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"+-------------------------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| Model | Provider | Tool Call Response |\n",
"+===========================================+==============+===============================================================================================================================================================================================================================================================================================+\n",
"| gpt-4o | openai | [ChatCompletionMessageToolCall(id='call_39osRa58C5UZfcZl2IwTeO8b', function=FunctionCall(arguments='{\\n \"topic\": \"weather forecasting\",\\n \"timeframe\": \"today\",\\n \"source_type\": \"mainstream\",\\n \"max_results\": 1,\\n \"language\": \"en\"\\n}', name='get_latest_ai_news'), type='function')] |\n",
"+-------------------------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| gpt-4o-mini | openai | No tool calls |\n",
"+-------------------------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| claude-3-5-sonnet-20240620 | anthropic | No tool calls |\n",
"+-------------------------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| claude-3-opus-20240229 | anthropic | No tool calls |\n",
"+-------------------------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| claude-3-haiku-20240307 | anthropic | No tool calls |\n",
"+-------------------------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| gemini-1.5-flash-latest | google | No tool calls |\n",
"+-------------------------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| gemini-1.5-pro | google | No tool calls |\n",
"+-------------------------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| accounts/fireworks/models/firefunction-v2 | fireworks-ai | No tool calls |\n",
"+-------------------------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"# Test message\n",
"messages = [{\"role\": \"user\", \"content\": \"What's the weather like in San Francisco, Tokyo, and Paris?\"}]\n",
"\n",
"# Run function calling for all providers\n",
"portkey_function_call(messages)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ZrDT-TFv62HW",
"outputId": "f149c47e-d4c6-4e2b-db97-c8aab2760dcc"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"+-------------------------------------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| Model | Provider | Tool Call Response |\n",
"+===========================================+==============+========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================+\n",
"| gpt-4o | openai | [ChatCompletionMessageToolCall(id='call_jQM4yp3iVovjYBmBrdN74loM', function=FunctionCall(arguments='{\"location\": \"San Francisco\"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='call_w8ZYduT9EsP3NqJljv3T7Eyy', function=FunctionCall(arguments='{\"location\": \"Tokyo\"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='call_96F45BQiUOj0q4Gy8iKRGsZ7', function=FunctionCall(arguments='{\"location\": \"Paris\"}', name='get_current_weather'), type='function')] |\n",
"+-------------------------------------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| gpt-4o-mini | openai | [ChatCompletionMessageToolCall(id='call_FSiM3gfLKr108PsegkggycFw', function=FunctionCall(arguments='{\"topic\": \"weather\", \"timeframe\": \"today\", \"max_results\": 1, \"language\": \"en\"}', name='get_latest_ai_news'), type='function'), ChatCompletionMessageToolCall(id='call_8Ovrkm2tDdLaauoUaYVqmNYv', function=FunctionCall(arguments='{\"topic\": \"weather\", \"timeframe\": \"today\", \"max_results\": 1, \"language\": \"en\"}', name='get_latest_ai_news'), type='function'), ChatCompletionMessageToolCall(id='call_yKfJ6FFLY12JwiWzXen0LnLP', function=FunctionCall(arguments='{\"topic\": \"weather\", \"timeframe\": \"today\", \"max_results\": 1, \"language\": \"en\"}', name='get_latest_ai_news'), type='function')] |\n",
"+-------------------------------------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| claude-3-5-sonnet-20240620 | anthropic | No tool calls |\n",
"+-------------------------------------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| claude-3-opus-20240229 | anthropic | No tool calls |\n",
"+-------------------------------------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| claude-3-haiku-20240307 | anthropic | [ChatCompletionMessageToolCall(id='toolu_01EbJeB9eYvKG7ANQKkhS9bP', function=FunctionCall(arguments='{\"locations\":\"[\\\\\"San Francisco\\\\\", \\\\\"Tokyo\\\\\", \\\\\"Paris\\\\\"]\"}', name='get_weather'), type='function')] |\n",
"+-------------------------------------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| gemini-1.5-flash-latest | google | No tool calls |\n",
"+-------------------------------------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| gemini-1.5-pro | google | No tool calls |\n",
"+-------------------------------------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| accounts/fireworks/models/firefunction-v2 | fireworks-ai | [ChatCompletionMessageToolCall(id='call_B2EiSi4DGpkwTjZhRrBKYDXJ', function=FunctionCall(arguments='{\"topic\": \"weather\", \"timeframe\": \"today\", \"max_results\": 1, \"language\": \"en\"}', name='get_latest_ai_news'), type='function')] |\n",
"+-------------------------------------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"# Test message\n",
"messages = [{\"role\": \"user\", \"content\": \"What's the weather like in Mumbai ,Delhi and Banglore?\"}]\n",
"\n",
"# Run function calling for all providers\n",
"portkey_function_call(messages)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "jMv-2PMrVST6",
"outputId": "7a58e247-9a55-4d4e-e72e-98e8d1c469a6"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"+-------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| Model | Provider | Tool Call Response |\n",
"+===========================================+==============+=========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================+\n",
"| gpt-4o | openai | [ChatCompletionMessageToolCall(id='call_RDwPeeHLHI9c5MBOxXnRkUlQ', function=FunctionCall(arguments='{\"location\": \"Mumbai\"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='call_nP0eZZISJeyzXI1utBG6alLj', function=FunctionCall(arguments='{\"location\": \"Delhi\"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='call_pOnq4tp9sSSPAaiarGsvykNd', function=FunctionCall(arguments='{\"location\": \"Bangalore\"}', name='get_current_weather'), type='function')] |\n",
"+-------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| gpt-4o-mini | openai | [ChatCompletionMessageToolCall(id='call_vS2RKIJ6Acm3l1lz4HhwOXss', function=FunctionCall(arguments='{\"topic\": \"Mumbai\", \"timeframe\": \"today\", \"source_type\": \"all\", \"max_results\": 1, \"language\": \"en\"}', name='get_latest_ai_news'), type='function'), ChatCompletionMessageToolCall(id='call_i9gg5uKxQaq9XdBaPCcJnnAX', function=FunctionCall(arguments='{\"topic\": \"Delhi\", \"timeframe\": \"today\", \"source_type\": \"all\", \"max_results\": 1, \"language\": \"en\"}', name='get_latest_ai_news'), type='function'), ChatCompletionMessageToolCall(id='call_cRQyEpj3OHoxOuqWnVHDF3Ey', function=FunctionCall(arguments='{\"topic\": \"Bangalore\", \"timeframe\": \"today\", \"source_type\": \"all\", \"max_results\": 1, \"language\": \"en\"}', name='get_latest_ai_news'), type='function')] |\n",
"+-------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| claude-3-5-sonnet-20240620 | anthropic | No tool calls |\n",
"+-------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| claude-3-opus-20240229 | anthropic | No tool calls |\n",
"+-------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| claude-3-haiku-20240307 | anthropic | [ChatCompletionMessageToolCall(id='toolu_018NHHkq5s62Rb8G4bmeZYo1', function=FunctionCall(arguments='{\"language\":\"en\",\"max_results\":3,\"source_type\":\"all\",\"timeframe\":\"today\",\"topic\":\"weather forecast mumbai delhi bangalore\"}', name='get_latest_ai_news'), type='function')] |\n",
"+-------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| gemini-1.5-flash-latest | google | No tool calls |\n",
"+-------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| gemini-1.5-pro | google | No tool calls |\n",
"+-------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
"| accounts/fireworks/models/firefunction-v2 | fireworks-ai | [ChatCompletionMessageToolCall(id='call_IqRs42JAXIxof45YAcAMWLIB', function=FunctionCall(arguments='{\"topic\": \"none\", \"timeframe\": \"today\", \"source_type\": \"all\", \"max_results\": 3, \"language\": \"en\"}', name='get_latest_ai_news'), type='function')] |\n",
"+-------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n"
]
}
]
}
]
}
cookbook/integrations/anyscale.md
Vendored
+411
View File
@@ -0,0 +1,411 @@
# Portkey + Anyscale
Portkey helps bring Anyscale APIs to production with its abstractions for observability, fallbacks, caching, and more. Use the Anyscale API **through** Portkey for:
1. **Enhanced Logging**: Track API usage with detailed insights.
2. **Production Reliability**: Automated fallbacks, load balancing, and caching.
3. **Continuous Improvement**: Collect and apply user feedback.
4. **Enhanced Fine-Tuning**: Combine logs & user feedback for targetted fine-tuning.
### 1.1 Setup & Logging
1. Set `$ export OPENAI_API_KEY=ANYSCALE_API_KEY`
2. Obtain your [**Portkey API Key**](https://app.portkey.ai/).
3. Switch to **Portkey Gateway URL:** `https://api.portkey.ai/v1/proxy`
See full logs of requests (latency, cost, tokens)—and dig deeper into the data with their analytics suite.
```py
""" OPENAI PYTHON SDK """
import openai
PORTKEY_GATEWAY_URL = "https://api.portkey.ai/v1"
PORTKEY_HEADERS = {
'Authorization': 'Bearer ANYSCALE_KEY',
'Content-Type': 'application/json',
# **************************************
'x-portkey-api-key': 'PORTKEY_API_KEY', # Get from https://app.portkey.ai/,
'x-portkey-provider': 'anyscale' # Tell Portkey that the request is for Anyscale
# **************************************
}
client = openai.OpenAI(base_url=PORTKEY_GATEWAY_URL, default_headers=PORTKEY_HEADERS)
response = client.chat.completions.create(
model="mistralai/Mistral-7B-Instruct-v0.1",
messages=[{"role": "user", "content": "Say this is a test"}]
)
print(response.choices[0].message.content)
```
```javascript
""" OPENAI NODE SDK """
import OpenAI from 'openai';
const PORTKEY_GATEWAY_URL = "https://api.portkey.ai/v1"
const PORTKEY_HEADERS = {
'Authorization': 'Bearer ANYSCALE_KEY',
'Content-Type': 'application/json',
// **************************************
'x-portkey-api-key': 'PORTKEY_API_KEY', // Get from https://app.portkey.ai/,
'x-portkey-provider': 'anyscale' // Tell Portkey that the request is for Anyscale
// **************************************
}
const openai = new OpenAI({baseURL:PORTKEY_GATEWAY_URL, defaultHeaders:PORTKEY_HEADERS});
async function main() {
const chatCompletion = await openai.chat.completions.create({
messages: [{ role: 'user', content: 'Say this is a test' }],
model: 'mistralai/Mistral-7B-Instruct-v0.1',
});
console.log(chatCompletion.choices[0].message.content);
}
main();
```
```py
""" REQUESTS LIBRARY """
import requests
PORTKEY_GATEWAY_URL = "https://api.portkey.ai/v1/chat/completions"
PORTKEY_HEADERS = {
'Authorization': 'Bearer ANYSCALE_KEY',
'Content-Type': 'application/json',
# **************************************
'x-portkey-api-key': 'PORTKEY_API_KEY', # Get from https://app.portkey.ai/,
'x-portkey-provider': 'anyscale' # Tell Portkey that the request is for Anyscale
# **************************************
}
DATA = {
"messages": [{"role": "user", "content": "What happens when you mix red & yellow?"}],
"model": "mistralai/Mistral-7B-Instruct-v0.1"
}
response = requests.post(PORTKEY_GATEWAY_URL, headers=PORTKEY_HEADERS, json=DATA)
print(response.text)
```
```bash
""" CURL """
curl "https://api.portkey.ai/v1/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ANYSCALE_KEY" \
-H "x-portkey-api-key: PORTKEY_API_KEY" \
-H "x-portkey-provider: anyscale" \
-d '{
"model": "meta-llama/Llama-2-70b-chat-hf",
"messages": [{"role": "user", "content": "Say 'Test'."}]
}'
```
### 1.2. Enhanced Observability
- **Trace** requests with single id.
- **Append custom tags** for request segmenting & in-depth analysis.
Just add their relevant headers to your reuqest:
```py
""" OPENAI PYTHON SDK """
import json, openai
PORTKEY_GATEWAY_URL = "https://api.portkey.ai/v1"
TRACE_ID = 'anyscale_portkey_test'
METADATA = {
"_environment": "production",
"_user": "userid123",
"_organisation": "orgid123",
"_prompt": "summarisationPrompt"
}
PORTKEY_HEADERS = {
'Authorization': 'Bearer ANYSCALE_KEY',
'Content-Type': 'application/json',
'x-portkey-api-key': 'PORTKEY_API_KEY',
'x-portkey-provider': 'anyscale',
# **************************************
'x-portkey-trace-id': TRACE_ID, # Send the trace id
'x-portkey-metadata': json.dumps(METADATA) # Send the metadata
# **************************************
}
client = openai.OpenAI(base_url=PORTKEY_GATEWAY_URL, default_headers=PORTKEY_HEADERS)
response = client.chat.completions.create(
model="mistralai/Mistral-7B-Instruct-v0.1",
messages=[{"role": "user", "content": "Say this is a test"}]
)
print(response.choices[0].message.content)
```
```javascript
""" OPENAI NODE SDK """
import OpenAI from 'openai';
const PORTKEY_GATEWAY_URL = "https://api.portkey.ai/v1"
const TRACE_ID = 'anyscale_portkey_test'
const METADATA = {
"_environment": "production",
"_user": "userid123",
"_organisation": "orgid123",
"_prompt": "summarisationPrompt"
}
const PORTKEY_HEADERS = {
'Authorization': 'Bearer ANYSCALE_KEY',
'Content-Type': 'application/json',
'x-portkey-api-key': 'PORTKEY_API_KEY',
'x-portkey-provider': 'anyscale',
// **************************************
'x-portkey-trace-id': TRACE_ID, // Send the trace id
'x-portkey-metadata': JSON.stringify(METADATA) // Send the metadata
// **************************************
}
const openai = new OpenAI({baseURL:PORTKEY_GATEWAY_URL, defaultHeaders:PORTKEY_HEADERS});
async function main() {
const chatCompletion = await openai.chat.completions.create({
messages: [{ role: 'user', content: 'Say this is a test' }],
model: 'mistralai/Mistral-7B-Instruct-v0.1',
});
console.log(chatCompletion.choices[0].message.content);
}
main();
```
```py
""" REQUESTS LIBRARY """
import requests, json
PORTKEY_GATEWAY_URL = "https://api.portkey.ai/v1/chat/completions"
TRACE_ID = 'anyscale_portkey_test'
METADATA = {
"_environment": "production",
"_user": "userid123",
"_organisation": "orgid123",
"_prompt": "summarisationPrompt"
}
PORTKEY_HEADERS = {
'Authorization': 'Bearer ANYSCALE_KEY',
'Content-Type': 'application/json',
'x-portkey-api-key': 'PORTKEY_API_KEY',
'x-portkey-provider': 'anyscale',
# **************************************
'x-portkey-trace-id': TRACE_ID, # Send the trace id
'x-portkey-metadata': json.dumps(METADATA) # Send the metadata
# **************************************
}
DATA = {
"messages": [{"role": "user", "content": "What happens when you mix red & yellow?"}],
"model": "mistralai/Mistral-7B-Instruct-v0.1"
}
response = requests.post(PORTKEY_GATEWAY_URL, headers=PORTKEY_HEADERS, json=DATA)
print(response.text)
```
```bash
""" CURL """
curl "https://api.portkey.ai/v1/chat/completions" \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer ANYSCALE_KEY' \
-H 'x-portkey-api-key: PORTKEY_KEY' \
-H 'x-portkey-provider: anyscale' \
-H 'x-portkey-trace-id: TRACE_ID' \
-H 'x-portkey-metadata: {"_environment": "production","_user": "userid123","_organisation": "orgid123","_prompt": "summarisationPrompt"}' \
-d '{
"model": "meta-llama/Llama-2-70b-chat-hf",
"messages": [{"role": "user", "content": "Say 'Test'."}]
}'
```
Here’s how your logs will appear on your Portkey dashboard:
<img src="https://portkey.ai/blog/content/images/2023/11/logsgif.gif" />
### 2. Caching, Fallbacks, Load Balancing
- **Fallbacks**: Ensure your application remains functional even if a primary service fails.
- **Load Balancing**: Efficiently distribute incoming requests among multiple models.
- **Semantic Caching**: Reduce costs and latency by intelligently caching results.
Toggle these features by saving _Configs_ (from the Portkey dashboard > Configs tab).
If we want to enable semantic caching + fallback from Llama2 to Mistral, your Portkey config would look like this:
```json
{
"cache": { "mode": "semantic" },
"strategy": { "mode": "fallback" },
"targets": [
{
"provider": "anyscale",
"api_key": "...",
"override_params": { "model": "meta-llama/Llama-2-7b-chat-hf" }
},
{
"provider": "anyscale",
"api_key": "...",
"override_params": { "model": "mistralai/Mistral-7B-Instruct-v0.1" }
}
]
}
```
Now, just send the Config ID with `x-portkey-config` header:
```py
""" OPENAI PYTHON SDK """
import openai, json
PORTKEY_GATEWAY_URL = "https://api.portkey.ai/v1"
PORTKEY_HEADERS = {
'Content-Type': 'application/json',
'x-portkey-api-key': 'PORTKEY_API_KEY',
# **************************************
'x-portkey-config': 'CONFIG_ID'
# **************************************
}
client = openai.OpenAI(base_url=PORTKEY_GATEWAY_URL, default_headers=PORTKEY_HEADERS)
response = client.chat.completions.create(
model="mistralai/Mistral-7B-Instruct-v0.1",
messages=[{"role": "user", "content": "Say this is a test"}]
)
print(response.choices[0].message.content)
```
```javascript
""" OPENAI NODE SDK """
import OpenAI from 'openai';
const PORTKEY_GATEWAY_URL = "https://api.portkey.ai/v1"
const PORTKEY_HEADERS = {
'Content-Type': 'application/json',
'x-portkey-api-key': 'PORTKEY_API_KEY',
// **************************************
'x-portkey-config': 'CONFIG_ID'
// **************************************
}
const openai = new OpenAI({baseURL:PORTKEY_GATEWAY_URL, defaultHeaders:PORTKEY_HEADERS});
async function main() {
const chatCompletion = await openai.chat.completions.create({
messages: [{ role: 'user', content: 'Say this is a test' }],
model: 'mistralai/Mistral-7B-Instruct-v0.1',
});
console.log(chatCompletion.choices[0].message.content);
}
main();
```
```py
""" REQUESTS LIBRARY """
import requests, json
PORTKEY_GATEWAY_URL = "https://api.portkey.ai/v1/chat/completions"
PORTKEY_HEADERS = {
'Content-Type': 'application/json',
'x-portkey-api-key': 'PORTKEY_API_KEY',
# **************************************
'x-portkey-config': 'CONFIG_ID'
# **************************************
}
DATA = {"messages": [{"role": "user", "content": "What happens when you mix red & yellow?"}]}
response = requests.post(PORTKEY_GATEWAY_URL, headers=PORTKEY_HEADERS, json=DATA)
print(response.text)
```
```bash
""" CURL """
curl "https://api.portkey.ai/v1/chat/completions" \
-H "Content-Type: application/json" \
-H "x-portkey-api-key: PORTKEY_API_KEY" \
-H "x-portkey-config: CONFIG_ID" \
-d '{ "messages": [{"role": "user", "content": "Say 'Test'."}] }'
```
For more on Configs and other gateway feature like Load Balancing, [check out the docs.](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations)
### 3. Collect Feedback
Gather weighted feedback from users and improve your app:
```py
""" REQUESTS LIBRARY """
import requests
import json
PORTKEY_FEEDBACK_URL = "https://api.portkey.ai/v1/feedback" # Portkey Feedback Endpoint
PORTKEY_HEADERS = {
"x-portkey-api-key": "PORTKEY_API_KEY",
"Content-Type": "application/json",
}
DATA = {
"trace_id": "anyscale_portkey_test", # On Portkey, you can append feedback to a particular Trace ID
"value": 1,
"weight": 0.5
}
response = requests.post(PORTKEY_FEEDBACK_URL, headers=PORTKEY_HEADERS, data=json.dumps(DATA))
print(response.text)
```
```bash
""" CURL """
curl "https://api.portkey.ai/v1/feedback" \
-H "x-portkey-api-key: PORTKEY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"trace_id": "anyscale_portkey_test",
"value": 1,
"weight": 0.5
}'
```
### 4. Continuous Fine-Tuning
Once you start logging your requests and their feedback with Portkey, it becomes very easy to 1️) Curate & create data for fine-tuning, 2) Schedule fine-tuning jobs, and 3) Use the fine-tuned models!
Fine-tuning is currently enabled for select orgs - please request access on [Portkey Discord](https://discord.gg/sDk9JaNfK8) and we'll get back to you ASAP.
<img src="https://portkey.ai/blog/content/images/2023/11/fine-tune.gif" alt="header" width=600 />
#### Conclusion
Integrating Portkey with Anyscale helps you build resilient LLM apps from the get-go. With features like semantic caching, observability, load balancing, feedback, and fallbacks, you can ensure optimal performance and continuous improvement.
[Read full Portkey docs here.](https://portkey.ai/docs/) | [Reach out to the Portkey team.](https://discord.gg/sDk9JaNfK8)
cookbook/integrations/deepinfra.ipynb
Vendored
+1
View File
File diff suppressed because one or more lines are too long
cookbook/integrations/groq.ipynb
Vendored
+1
View File
File diff suppressed because one or more lines are too long
cookbook/integrations/langchain.ipynb
Vendored
+1
View File
File diff suppressed because one or more lines are too long
cookbook/integrations/mistral.md
Vendored
+210
View File
@@ -0,0 +1,210 @@
# Portkey + Mistral
Portkey helps bring Mistral's APIs to production with its observability suite & AI Gateway. Use the Mistral API **through** Portkey for:
1. **Enhanced Logging**: Track API usage with detailed insights and custom segmentation.
2. **Production Reliability**: Automated fallbacks, load balancing, retries, time outs, and caching.
3. **Continuous Improvement**: Collect and apply user feedback.
### 1.1 Setup & Logging
1. Obtain your [**Portkey API Key**](https://app.portkey.ai/).
2. Set `$ export PORTKEY_API_KEY=PORTKEY_API_KEY`
3. Set `$ export MISTRAL_API_KEY=MISTRAL_API_KEY`
4. `pip install portkey-ai` or `npm i portkey-ai`
```py
""" OPENAI PYTHON SDK """
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY",
# ************************************
provider="mistral-ai",
Authorization="Bearer MISTRAL_API_KEY"
# ************************************
)
response = portkey.chat.completions.create(
model="mistral-tiny",
messages = [{ "role": "user", "content": "c'est la vie" }]
)
```
```javascript
import Portkey from 'portkey-ai';
const portkey = new Portkey({
apiKey: "PORTKEY_API_KEY",
// ***********************************
provider: "mistral-ai",
Authorization: "Bearer MISTRAL_API_KEH"
// ***********************************
})
async function main(){
const response = await portkey.chat.completions.create({
model: "mistral-tiny",
messages: [{ role: 'user', content: "c'est la vie" }]
});
}
main()
```
### 1.2. Enhanced Observability
* **Trace** requests with single id.
* **Append custom tags** for request segmenting & in-depth analysis.
Just add their relevant headers to your request:
```py
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY",
provider="mistral-ai",
Authorization="Bearer MISTRAL_API_KEY"
)
response = portkey.with_options(
# ************************************
trace_id="ux5a7",
metadata={"user": "john_doe"}
# ************************************
).chat.completions.create(
model="mistral-tiny",
messages = [{ "role": "user", "content": "c'est la vie" }]
)
```
```javascript
import Portkey from 'portkey-ai';
const portkey = new Portkey({
apiKey: "PORTKEY_API_KEY",
provider: "mistral-ai",
Authorization: "Bearer MISTRAL_API_KEH"
})
async function main(){
const response = await portkey.chat.completions.create({
model: "mistral-tiny",
messages: [{ role: 'user', content: "c'est la vie" }]
},{
// ***********************************
traceID: "ux5a7",
metadata: {"user": "john_doe"}
});
}
main()
```
Here’s how your logs will appear on your Portkey dashboard:
<img src="https://portkey.ai/blog/content/images/2023/11/logsgif.gif" />
### 2. Caching, Fallbacks, Load Balancing
* **Fallbacks**: Ensure your application remains functional even if a primary service fails.
* **Load Balancing**: Efficiently distribute incoming requests among multiple models.
* **Semantic Caching**: Reduce costs and latency by intelligently caching results.
Toggle these features by saving _Configs_ (from the Portkey dashboard > Configs tab).
If we want to enable semantic caching + fallback from Mistral-Medium to Mistral-Tiny, your Portkey config would look like this:
```json
{
"cache": {"mode": "semantic"},
"strategy": {"mode": "fallback"},
"targets": [
{
"provider": "mistral-ai", "api_key": "...",
"override_params": {"model": "mistral-medium"}
},
{
"provider": "mistral-ai", "api_key": "...",
"override_params": {"model": "mistral-tiny"}
}
]
}
```
Now, just set the Config ID while instantiating Portkey:
```py
""" OPENAI PYTHON SDK """
from portkey_ai import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY",
# ************************************
config="pp-mistral-cache-xx"
# ************************************
)
response = portkey.chat.completions.create(
model="mistral-tiny",
messages = [{ "role": "user", "content": "c'est la vie" }]
)
```
```javascript
import Portkey from 'portkey-ai';
const portkey = new Portkey({
apiKey: "PORTKEY_API_KEY",
// ***********************************
config: "pp-mistral-cache-xx"
// ***********************************
})
async function main(){
const response = await portkey.chat.completions.create({
model: "mistral-tiny",
messages: [{ role: 'user', content: "c'est la vie" }]
});
}
main()
```
For more on Configs and other gateway feature like Load Balancing, [check out the docs.](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations)
### 3. Collect Feedback
Gather weighted feedback from users and improve your app:
```py
from portkey import Portkey
portkey = Portkey(
api_key="PORTKEY_API_KEY"
)
def send_feedback():
portkey.feedback.create(
'trace_id'= 'REQUEST_TRACE_ID',
'value'= 0 # For thumbs down
)
send_feedback()
```
```javascript
import Portkey from 'portkey-ai';
const portkey = new Portkey({
apiKey: "PORTKEY_API_KEY"
});
const sendFeedback = async () => {
await portkey.feedback.create({
traceID: "REQUEST_TRACE_ID",
value: 1 // For thumbs up
});
}
await sendFeedback();
```
#### Conclusion
Integrating Portkey with Mistral helps you build resilient LLM apps from the get-go. With features like semantic caching, observability, load balancing, feedback, and fallbacks, you can ensure optimal performance and continuous improvement.
[Read full Portkey docs here.](https://portkey.ai/docs/) | [Reach out to the Portkey team.](https://discord.gg/sDk9JaNfK8)
cookbook/integrations/mixtral-8x22b.ipynb
Vendored
+285
View File
File diff suppressed because one or more lines are too long
cookbook/integrations/ncompass.ipynb
Vendored
+244
View File
File diff suppressed because one or more lines are too long
cookbook/integrations/openai.ipynb
Vendored
+1
View File
File diff suppressed because one or more lines are too long
cookbook/integrations/segmind.ipynb
Vendored
+1
View File
File diff suppressed because one or more lines are too long
cookbook/integrations/vercel-ai.md
Vendored
+311
View File
@@ -0,0 +1,311 @@
## Portkey is the control panel for your Vercel AI app. It makes your LLM integrations prod-ready, reliable, fast, and cost-efficient.
Use Portkey with your Vercel app for:
1. Calling 100+ LLMs (open & closed)
2. Logging & analysing LLM usage
3. Caching responses
4. Automating fallbacks, retries, timeouts, and load balancing
5. Managing, versioning, and deploying prompts
6. Continuously improving app with user feedback
## Guide: Create a Portkey + OpenAI Chatbot
### 1. Create a NextJS app
Go ahead and create a Next.js application, and install `ai` and `portkey-ai` as dependencies.
```sh
pnpm dlx create-next-app my-ai-app
cd my-ai-app
pnpm install ai portkey-ai
```
### 2. Add Authentication keys to `.env`
1. Login to Portkey [here](https://app.portkey.ai/)
2. To integrate OpenAI with Portkey, add your OpenAI API key to Portkey’s Virtual Keys
3. This will give you a disposable key that you can use and rotate instead of directly using the OpenAI API key
4. Grab the Virtual key & your Portkey API key and add them to `.env` file:
```sh
# ".env"
PORTKEY_API_KEY="xxxxxxxxxx"
OPENAI_VIRTUAL_KEY="xxxxxxxxxx"
```
### 3. Create Route Handler
Create a Next.js Route Handler that utilizes the Edge Runtime to generate a chat completion. Stream back to Next.js.
For this example, create a route handler at `app/api/chat/route.ts` that calls GPT-4 and accepts a `POST` request with a messages array of strings:
```ts
// filename="app/api/chat/route.ts"
import { OpenAIStream, StreamingTextResponse } from 'ai';
import { Portkey } from 'portkey-ai';
// Create a Portkey API client
const portkey = new Portkey({
apiKey: process.env.PORTKEY_API_KEY,
virtualKey: process.env.OPENAI_VIRTUAL_KEY
});
// Set the runtime to edge for best performance
export const runtime = 'edge';
export async function POST(req: Request) {
const { messages } = await req.json();
// Call GPT-4
const response = await portkey.chat.completions.create({
model: 'gpt-4',
stream: true,
messages
});
// Convert the response into a friendly text-stream
const stream = OpenAIStream(response);
// Respond with the stream
return new StreamingTextResponse(stream);
}
```
The Vercel AI SDK provides <code>[OpenAIStream](https://sdk.vercel.ai/docs/api-reference/providers/openai-stream)</code> function that decodes the text tokens in the <code>response</code> and encodes them properly for simple consumption. The <code>[StreamingTextResponse](https://sdk.vercel.ai/docs/api-reference/streaming-text-response)</code> class utility extends the Node/Edge Runtime <code>Response</code> class with default headers.
Portkey follows the same signature as OpenAI SDK but extends it to work with **100+ LLMs**. Here, the chat completion call will be sent to the `gpt-4` model, and the response will be streamed to your Next.js app.
### 4. Switch from OpenAI to Anthropic
Portkey is powered by an [open-source, universal AI Gateway](https://github.com/portkey-ai/gateway) with which you can route to 100+ LLMs using the same, known OpenAI spec.
Let’s see how you can switch from GPT-4 to Claude-3-Opus by updating 2 lines of code (without breaking anything else).
1. Add your Anthropic API key or AWS Bedrock secrets to Portkey’s Virtual Keys
2. Update the virtual key while instantiating your Portkey client
3. Update the model name while making your `/chat/completions` call
Let’s see it in action:
```tsx filename="app/api/chat/route.ts"
// Set the runtime to edge for best performance
export const runtime = 'edge';
// Create a Portkey API client
const portkey = new Portkey({
apiKey: process.env.PORTKEY_API_KEY,
virtualKey: process.env.ANTHROPIC_VIRTUAL_KEY
});
export async function POST(req: Request) {
const { messages } = await req.json();
// Switch from GPT-4 to Claude-3-Opus
const response = await portkey.chat.completions.create({
model: 'claude-3-opus-20240229',
max_tokens: 512
stream: true,
messages
});
const stream = OpenAIStream(response);
return new StreamingTextResponse(stream);
}
```
### 5. Switch to Gemini 1.5
Similarly, you can just add your [Google AI Studio API key](https://aistudio.google.com/app/) to Portkey and call Gemini 1.5:
```tsx
const portkey = new Portkey({
apiKey: process.env.PORTKEY_API_KEY,
virtualKey: process.env.GOOGLE_VIRTUAL_KEY
});
const response = await portkey.chat.completions.create({
model: 'gemini-1.5-pro-latest,
stream: true,
messages
});
```
The same will follow for all the other providers like **Azure**, **Mistral**, **Anyscale**, **Together**, and more.
### 6. Wire up the UI
Let's create a Client component that will have a form to collect the prompt from the user and stream back the completion. The `useChat` hook will default use the `POST` Route Handler we created earlier (`/api/chat`). However, you can override this default value by passing an `api` prop to useChat(`{ api: '...'}`).
```tsx
//"app/page.tsx"
'use client';
import { useChat } from 'ai/react';
export default function Chat() {
const { messages, input, handleInputChange, handleSubmit } = useChat();
return (
<div className="flex flex-col w-full max-w-md py-24 mx-auto stretch">
{messages.map((m) => (
<div key={m.id} className="whitespace-pre-wrap">
{m.role === 'user' ? 'User: ' : 'AI: '}
{m.content}
</div>
))}
<form onSubmit={handleSubmit}>
<input
className="fixed bottom-0 w-full max-w-md p-2 mb-8 border border-gray-300 rounded shadow-xl"
value={input}
placeholder="Say something..."
onChange={handleInputChange}
/>
</form>
</div>
);
}
```
### 7. Log the Requests
Portkey logs all the requests you’re sending to help you debug errors, and get request-level + aggregate insights on costs, latency, errors, and more.
You can enhance the logging by tracing certain requests, passing custom metadata or user feedback.
![rolling logs and cachescreents](https://portkey.ai/blog/content/images/2024/04/logs.gif)
**Segmenting Requests with Metadata**
On Portkey, while making a `chat.completions` call, you can pass any `{"key":"value"}` pairs. Portkey segments the requests based on the metadata to give you granular insights.
```tsx
const response = await portkey.chat.completions.create(
{
model: 'gpt-4',
messages: [{ role: 'user', content: 'How do I optimise auditorium for maximum occupancy?' }]
},
{
metadata: {
user_name: 'john doe',
organization_name: 'acme'
}
}
);
```
Learn more about [tracing](https://portkey.ai/docs/product/observability-modern-monitoring-for-llms/traces) and [feedback](https://portkey.ai/docs/product/observability-modern-monitoring-for-llms/feedback).
## Guide: Handle OpenAI Failures
### 1. Solve 5xx, 4xx Errors
Portkey helps you automatically trigger a call to any other LLM/provider in case of primary failures.<br>
[Create](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/configs) a fallback logic with Portkey’s Gateway Config.
For example, for setting up a fallback from OpenAI to Anthropic, the Gateway Config would be:
```json
{
"strategy": { "mode": "fallback" },
"targets": [{ "virtual_key": "openai-virtual-key" }, { "virtual_key": "anthropic-virtual-key" }]
}
```
You can save this Config in Portkey app and get an associated Config ID that you can pass while instantiating your Portkey client:
### 2. Apply Config to the Route Handler
```tsx
const portkey = new Portkey({
apiKey: process.env.PORTKEY_API_KEY,
config: 'CONFIG_ID'
});
export async function POST(req: Request) {
const { messages } = await req.json();
const response = await portkey.chat.completions.create({
model: 'gpt-4',
stream: true,
messages
});
const stream = OpenAIStream(response);
return new StreamingTextResponse(stream);
}
```
### 3. Handle Rate Limit Errors
You can loadbalance your requests against multiple LLMs or accounts and prevent any one account from hitting rate limit thresholds.
For example, to route your requests between 1 OpenAI and 2 Azure OpenAI accounts:
```json
{
"strategy": { "mode": "loadbalance" },
"targets": [
{ "virtual_key": "openai-virtual-key", "weight": 1 },
{ "virtual_key": "azure-virtual-key-1", "weight": 1 },
{ "virtual_key": "azure-virtual-key-2", "weight": 1 }
]
}
```
Save this Config in the Portkey app and pass it while instantiating the Portkey client, just like we did above.
Portkey can also trigger [automatic retries](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/automatic-retries), set [request timeouts](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/request-timeouts), and more.
## Guide: Cache Semantically Similar Requests
Portkey can save LLM costs & reduce latencies 20x by storing responses for semantically similar queries and serving them from cache.
For Q&A use cases, cache hit rates go as high as 50%. To enable semantic caching, just set the `cache` `mode` to `semantic` in your Gateway Config:
```json
{
"cache": { "mode": "semantic" }
}
```
Same as above, you can save your cache Config in the Portkey app, and reference the Config ID while instantiating the Portkey client.
Moreover, you can set the `max-age` of the cache and force refresh a cache. See the [docs](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/cache-simple-and-semantic) for more information.
## Guide: Manage Prompts Separately
Storing prompt templates and instructions in code is messy. Using Portkey, you can create and manage all of your app’s prompts in a single place and directly hit our prompts API to get responses. Here’s more on [what Prompts on Portkey can do](https://portkey.ai/docs/product/prompt-library).
To create a Prompt Template,
1. From the Dashboard, Open **Prompts**
2. In the **Prompts** page, Click **Create**
3. Add your instructions, variables, and You can modify model parameters and click **Save**
![verel app prompt](https://raw.githubusercontent.com/Portkey-AI/portkey-cookbook/a358363e2102df4ff7657e56cc592f4e861f9d6c/integrations/images/vercel-portkey-guide/2-vercel-portkey-guide.png)
### Trigger the Prompt in the Route Handler
```js
const portkey = new Portkey({
apiKey: process.env.PORTKEY_API_KEY
});
export async function POST(req: Request) {
const { variable_content } = await req.json();
const response = await portkey.prompts.completions.create({
promptID: 'pp-vercel-app-f36a02',
variables: { variable_name: 'variable_content' }, // string
stream: true
});
const stream = OpenAIStream(response);
return new StreamingTextResponse(stream);
}
```
See [docs](https://portkey.ai/docs/api-reference/prompts/prompt-completion) for more information.
## Talk to the Developers
If you have any questions or issues, reach out to us on [Discord here](https://portkey.ai/community). On Discord, you will also meet many other practitioners who are putting their Vercel AI + Portkey app to production.
cookbook/integrations/vercel/app/api/chat/route.ts
Vendored
+12
View File
@@ -0,0 +1,12 @@
import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';
export async function POST(request: Request) {
const { messages } = await request.json();
const stream = await streamText({
model: openai('gpt-4o'),
system: 'You are a helpful assistant.',
messages,
});
return stream.toAIStreamResponse();
}
cookbook/integrations/vercel/app/examples/basic-chatbot/page.tsx
Vendored
+24
View File
@@ -0,0 +1,24 @@
'use client';
import { useChat } from 'ai/react';
export default function Chat() {
const { messages, input, handleInputChange, handleSubmit } = useChat();
return (
<div>
{messages.map((m) => (
<div key={m.id}>
{m.role === 'user' ? 'User: ' : 'AI: '}
{m.content}
</div>
))}
<form onSubmit={handleSubmit}>
<input
value={input}
placeholder="Say something..."
onChange={handleInputChange}
/>
</form>
</div>
);
}
cookbook/integrations/vercel/app/examples/generate-text/action.ts
Vendored
+22
View File
@@ -0,0 +1,22 @@
'use server';
import { generateText } from 'ai';
import { createPortkey } from '@portkey-ai/vercel-provider';
export const generateTextAction = async () => {
const llmClient = createPortkey({
apiKey: 'PORTKEY_API_KEY',
virtualKey: 'YOUR_OPENAI_VIRTUAL_KEY', //head over to https://app.portkey.ai to create Virtual key
//Portkey's config allows you to use- loadbalance, fallback, retires, timeouts, semantic caching, conditional routing, guardrails,etc. Head over to portkey docs to learn more
});
// Learn more at docs.portkey.ai
const result = await generateText({
model: llmClient.completionModel('gpt-3.5-turbo'), //choose model of choice
prompt: 'tell me a joke',
});
return result.text;
};
cookbook/integrations/vercel/app/examples/generate-text/conditional-routing.ts
Vendored
+56
View File
@@ -0,0 +1,56 @@
'use server';
import { generateText } from 'ai';
import { createPortkey } from '@portkey-ai/vercel-provider';
export const generateTextAction = async () => {
// Conditional routing config
const portkey_config = {
strategy: {
mode: 'conditional',
conditions: [
{
query: { 'metadata.user_plan': { $eq: 'paid' } },
then: 'anthropic-claude',
},
{
query: { 'metadata.user_plan': { $eq: 'free' } },
then: 'openai-gpt-4',
},
],
default: 'openai-gpt-4',
},
targets: [
{
name: 'anthropic-claude',
provider: 'anthropic',
api_key: 'YOUR_ANTHROPIC_API_KEY',
override_params: {
model: 'claude-3-5-sonnet-20240620',
},
},
{
name: 'openai-gpt-4',
provider: 'openai',
api_key: 'YOUR_OPENAI_API_KEY',
override_params: {
model: 'gpt-4o',
},
},
],
};
const llmClient = createPortkey({
apiKey: 'PORTKEY_API_KEY',
config: portkey_config,
//Portkey's config allows you to use- loadbalance, fallback, retires, timeouts, semantic caching, conditional routing, guardrails,etc. Head over to portkey docs to learn more
//we are using API keys inside config, that's why no virtual keys needed
});
const result = await generateText({
model: llmClient.completionModel('gpt-3.5-turbo'), //choose model of choice
prompt: 'tell me a joke',
});
return result.text;
};
cookbook/integrations/vercel/app/examples/generate-text/fallback.ts
Vendored
+45
View File
@@ -0,0 +1,45 @@
'use server';
import { generateText } from 'ai';
import { createPortkey } from '@portkey-ai/vercel-provider';
export const generateTextAction = async () => {
// Fallback config
const portkey_config = {
strategy: {
mode: 'fallback',
},
targets: [
{
provider: 'anthropic',
api_key: 'YOUR_ANTHROPIC_API_KEY',
override_params: {
model: 'claude-3-5-sonnet-20240620',
},
},
{
provider: 'openai',
api_key: 'YOUR_OPENAI_API_KEY',
override_params: {
model: 'gpt-4o',
},
},
],
};
const llmClient = createPortkey({
apiKey: 'PORTKEY_API_KEY',
config: portkey_config,
//Portkey's config allows you to use- loadbalance, fallback, retires, timeouts, semantic caching, conditional routing, guardrails,etc. Head over to portkey docs to learn more
//we are using API keys inside config, that's why no virtual keys needed
});
// Learn more at docs.portkey.ai
const result = await generateText({
model: llmClient.completionModel('gpt-3.5-turbo'), //choose model of choice
prompt: 'tell me a joke',
});
return result.text;
};
cookbook/integrations/vercel/app/examples/generate-text/guardrails.ts
Vendored
+42
View File
@@ -0,0 +1,42 @@
'use server';
import { generateText } from 'ai';
import { createPortkey } from '@portkey-ai/vercel-provider';
export const generateTextAction = async () => {
const portkey_config = {
retry: {
attempts: 3,
},
cache: {
mode: 'simple',
},
virtual_key: 'openai-xxx',
before_request_hooks: [
{
id: 'input-guardrail-id-xx',
},
],
after_request_hooks: [
{
id: 'output-guardrail-id-xx',
},
],
};
const llmClient = createPortkey({
apiKey: 'PORTKEY_API_KEY',
config: portkey_config,
//Portkey's config allows you to use- loadbalance, fallback, retires, timeouts, semantic caching, conditional routing, guardrails,etc. Head over to portkey docs to learn more
//we are using API keys inside config, that's why no virtual keys needed
});
// Learn more at docs.portkey.ai
const result = await generateText({
model: llmClient.completionModel('gpt-3.5-turbo'), //choose model of choice
prompt: 'tell me a joke',
});
return result.text;
};
cookbook/integrations/vercel/app/examples/generate-text/load-balance.ts
Vendored
+47
View File
@@ -0,0 +1,47 @@
'use server';
import { generateText } from 'ai';
import { createPortkey } from '@portkey-ai/vercel-provider';
export const generateTextAction = async () => {
// Fallback config
const portkey_config = {
strategy: {
mode: 'loadbalance',
},
targets: [
{
provider: 'anthropic',
api_key: 'YOUR_ANTHROPIC_API_KEY',
override_params: {
model: 'claude-3-5-sonnet-20240620',
},
weight: 0.25,
},
{
provider: 'openai',
api_key: 'YOUR_OPENAI_API_KEY',
override_params: {
model: 'gpt-4o',
},
weight: 0.75,
},
],
};
const llmClient = createPortkey({
apiKey: 'PORTKEY_API_KEY',
config: portkey_config,
//Portkey's config allows you to use- loadbalance, fallback, retires, timeouts, semantic caching, conditional routing, guardrails,etc. Head over to portkey docs to learn more
//we are using API keys inside config, that's why no virtual keys needed
});
// Learn more at docs.portkey.ai
const result = await generateText({
model: llmClient.completionModel('gpt-3.5-turbo'), //choose model of choice
prompt: 'tell me a joke',
});
return result.text;
};
cookbook/integrations/vercel/app/examples/generate-text/page.tsx
Vendored
+23
View File
@@ -0,0 +1,23 @@
'use client';
import { Button } from '@/components/ui/button';
import { generateTextAction } from './action';
import { useState } from 'react';
export default function Page() {
const [generation, setGeneration] = useState('');
return (
<div className="space-y-4">
<h1 className="text-xl font-semibold">Generate Text Example</h1>
<Button
onClick={async () => {
const result = await generateTextAction();
setGeneration(result);
}}
>
Tell me a joke
</Button>
<pre>{JSON.stringify(generation, null, 2)}</pre>
</div>
);
}
cookbook/integrations/vercel/app/examples/generate-ui-streamui/action.tsx
Vendored
+77
View File
@@ -0,0 +1,77 @@
'use server';
import { createAI, getMutableAIState, streamUI } from 'ai/rsc';
import { openai } from '@ai-sdk/openai';
import { ReactNode } from 'react';
import { z } from 'zod';
import { nanoid } from 'nanoid';
import { JokeComponent } from './joke-component';
import { generateObject } from 'ai';
import { jokeSchema } from './joke';
export interface ServerMessage {
role: 'user' | 'assistant';
content: string;
}
export interface ClientMessage {
id: string;
role: 'user' | 'assistant';
display: ReactNode;
}
export async function continueConversation(
input: string
): Promise<ClientMessage> {
'use server';
const history = getMutableAIState();
const result = await streamUI({
model: openai('gpt-4o'),
messages: [...history.get(), { role: 'user', content: input }],
text: ({ content, done }) => {
if (done) {
history.done((messages: ServerMessage[]) => [
...messages,
{ role: 'assistant', content },
]);
}
return <div>{content}</div>;
},
tools: {
tellAJoke: {
description: 'Tell a joke',
parameters: z.object({
location: z.string().describe('the users location'),
}),
generate: async function* ({ location }) {
yield <div>loading...</div>;
const joke = await generateObject({
model: openai('gpt-4o'),
schema: jokeSchema,
prompt:
'Generate a joke that incorporates the following location:' +
location,
});
return <JokeComponent joke={joke.object} />;
},
},
},
});
return {
id: nanoid(),
role: 'assistant',
display: result.value,
};
}
export const AI = createAI<ServerMessage[], ClientMessage[]>({
actions: {
continueConversation,
},
initialAIState: [],
initialUIState: [],
});
cookbook/integrations/vercel/app/examples/generate-ui-streamui/joke-component.tsx
Vendored
+21
View File
@@ -0,0 +1,21 @@
'use client';
import { useState } from 'react';
import { Button } from '@/components/ui/button';
import { Joke } from './joke';
export const JokeComponent = ({ joke }: { joke?: Joke }) => {
const [showPunchline, setShowPunchline] = useState(false);
return (
<div className="bg-neutral-100 p-4 rounded-md m-4 max-w-prose flex items-center justify-between">
<p>{showPunchline ? joke?.punchline : joke?.setup}</p>
<Button
onClick={() => setShowPunchline(true)}
disabled={showPunchline}
variant="outline"
>
Show Punchline!
</Button>
</div>
);
};
cookbook/integrations/vercel/app/examples/generate-ui-streamui/joke.ts
Vendored
+9
View File
@@ -0,0 +1,9 @@
import { DeepPartial } from 'ai';
import { z } from 'zod';
export const jokeSchema = z.object({
setup: z.string().describe('the setup of the joke'),
punchline: z.string().describe('the punchline of the joke'),
});
export type Joke = DeepPartial<typeof jokeSchema>;
cookbook/integrations/vercel/app/examples/generate-ui-streamui/page.tsx
Vendored
+51
View File
@@ -0,0 +1,51 @@
'use client';
import { useState } from 'react';
import { ClientMessage } from './action';
import { useActions, useUIState } from 'ai/rsc';
import { nanoid } from 'nanoid';
export default function Home() {
const [input, setInput] = useState<string>('');
const [conversation, setConversation] = useUIState();
const { continueConversation } = useActions();
return (
<div>
<div>
{conversation.map((message: ClientMessage) => (
<div key={message.id}>
{message.role}: {message.display}
</div>
))}
</div>
<form
onSubmit={async (e) => {
e.preventDefault();
setInput('');
setConversation((currentConversation: ClientMessage[]) => [
...currentConversation,
{ id: nanoid(), role: 'user', display: input },
]);
const message = await continueConversation(input);
setConversation((currentConversation: ClientMessage[]) => [
...currentConversation,
message,
]);
}}
>
<input
type="text"
value={input}
onChange={(event) => {
setInput(event.target.value);
}}
/>
<button>Send Message</button>
</form>
</div>
);
}
cookbook/integrations/vercel/app/examples/layout.tsx
Vendored
+3
View File
@@ -0,0 +1,3 @@
export default function Layout({ children }: { children: React.ReactNode }) {
return <main className="">{children}</main>;
}
cookbook/integrations/vercel/app/examples/stream-text/action.ts
Vendored
+20
View File
@@ -0,0 +1,20 @@
'use server';
import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';
import { createStreamableValue } from 'ai/rsc';
import { createPortkey } from '@portkey-ai/vercel-provider';
export const streamTextAction = async () => {
const llmClient = createPortkey({
apiKey: 'PORTKEY_API_KEY',
virtualKey: 'YOUR_OPENAI_VIRTUAL_KEY',
});
const result = await streamText({
model: llmClient.completionModel('gpt-3.5-turbo-instruct'),
temperature: 0.5,
prompt: 'Tell me a joke.',
});
return createStreamableValue(result.textStream).value;
};
cookbook/integrations/vercel/app/examples/stream-text/page.tsx
Vendored
+25
View File
@@ -0,0 +1,25 @@
'use client';
import { Button } from '@/components/ui/button';
import { streamTextAction } from './action';
import { useState } from 'react';
import { readStreamableValue } from 'ai/rsc';
export default function Page() {
const [generation, setGeneration] = useState('');
return (
<div className="space-y-4">
<h1 className="text-xl font-semibold">Stream Text Example</h1>
<Button
onClick={async () => {
const result = await streamTextAction();
for await (const delta of readStreamableValue(result))
setGeneration(delta ?? '');
}}
>
Tell me a joke
</Button>
<pre>{JSON.stringify(generation, null, 2)}</pre>
</div>
);
}
cookbook/integrations/vercel/app/examples/tools/basic/action.ts
Vendored
+40
View File
@@ -0,0 +1,40 @@
'use server';
import { openai } from '@ai-sdk/openai';
import { createPortkey } from '@portkey-ai/vercel-provider';
import { streamText, generateText } from 'ai';
import { createStreamableValue } from 'ai/rsc';
import { z } from 'zod';
export const generateTextAction = async (location: string) => {
const llmClient = createPortkey({
apiKey: 'PORTKEY_API_KEY',
virtualKey: 'YOUR_OPENAI_VIRTUAL_KEY',
});
('use server');
const { toolResults, toolCalls } = await generateText({
model: llmClient.completionModel('gpt-3.5-turbo-instruct'),
temperature: 0.8,
prompt: `You are a funny chatbot. users location: ${location}`,
tools: {
weather: {
description: "Get the weather for the user's location",
parameters: z.object({
location: z.string().describe("user's location"),
}),
execute: async ({ location }) => {
const temperature = Math.floor(Math.random() * 31); // call external api for {location}
return { temperature };
},
},
},
});
if (toolResults && toolCalls) {
const joke = await streamText({
model: openai('gpt-4o'),
prompt: `Tell me a joke that incorporates ${location} and it's current temperature (${toolResults[0].result.temperature})`,
});
return createStreamableValue(joke.textStream).value;
}
};
cookbook/integrations/vercel/app/examples/tools/basic/page.tsx
Vendored
+31
View File
@@ -0,0 +1,31 @@
'use client';
import { Button } from '@/components/ui/button';
import { generateTextAction } from './action';
import { useState } from 'react';
import { Input } from '@/components/ui/input';
import { readStreamableValue } from 'ai/rsc';
export default function Page() {
const [generation, setGeneration] = useState('');
return (
<div className="space-y-4">
<h1 className="text-xl font-semibold">Stream Text Tool Example</h1>
<form
action={async (data) => {
const location = data.get('location') as string;
const result = await generateTextAction(location);
if (result) {
for await (const delta of readStreamableValue(result)) {
setGeneration(delta ?? '');
}
}
}}
>
<Input name="location" required placeholder="San Francisco" />
<Button>Tell me a joke</Button>
</form>
<pre>{generation}</pre>
</div>
);
}
cookbook/integrations/vercel/app/favicon.ico
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 25 KiB

cookbook/integrations/vercel/app/globals.css
Vendored
+76
View File
@@ -0,0 +1,76 @@
@tailwind base;
@tailwind components;
@tailwind utilities;
@layer base {
:root {
--background: 0 0% 100%;
--foreground: 0 0% 3.9%;
--card: 0 0% 100%;
--card-foreground: 0 0% 3.9%;
--popover: 0 0% 100%;
--popover-foreground: 0 0% 3.9%;
--primary: 0 0% 9%;
--primary-foreground: 0 0% 98%;
--secondary: 0 0% 96.1%;
--secondary-foreground: 0 0% 9%;
--muted: 0 0% 96.1%;
--muted-foreground: 0 0% 45.1%;
--accent: 0 0% 96.1%;
--accent-foreground: 0 0% 9%;
--destructive: 0 84.2% 60.2%;
--destructive-foreground: 0 0% 98%;
--border: 0 0% 89.8%;
--input: 0 0% 89.8%;
--ring: 0 0% 3.9%;
--radius: 0.5rem;
}
.dark {
--background: 0 0% 3.9%;
--foreground: 0 0% 98%;
--card: 0 0% 3.9%;
--card-foreground: 0 0% 98%;
--popover: 0 0% 3.9%;
--popover-foreground: 0 0% 98%;
--primary: 0 0% 98%;
--primary-foreground: 0 0% 9%;
--secondary: 0 0% 14.9%;
--secondary-foreground: 0 0% 98%;
--muted: 0 0% 14.9%;
--muted-foreground: 0 0% 63.9%;
--accent: 0 0% 14.9%;
--accent-foreground: 0 0% 98%;
--destructive: 0 62.8% 30.6%;
--destructive-foreground: 0 0% 98%;
--border: 0 0% 14.9%;
--input: 0 0% 14.9%;
--ring: 0 0% 83.1%;
}
}
@layer base {
* {
@apply border-border;
}
body {
@apply bg-background text-foreground;
}
}
cookbook/integrations/vercel/app/layout.tsx
Vendored
+33
View File
@@ -0,0 +1,33 @@
import type { Metadata } from 'next';
import { Inter } from 'next/font/google';
import './globals.css';
import { AI } from './examples/generate-ui-streamui/action';
import { BackButton } from '@/components/back-button';
const inter = Inter({ subsets: ['latin'] });
export const metadata: Metadata = {
title: 'Create Next App',
description: 'Generated by create next app',
};
export default function RootLayout({
children,
}: Readonly<{
children: React.ReactNode;
}>) {
return (
<html lang="en">
<body className={inter.className}>
<div className="max-w-2xl p-8">
<AI>
<div className="mb-4">
<BackButton />
</div>
<div>{children}</div>
</AI>
</div>
</body>
</html>
);
}
cookbook/integrations/vercel/app/page.tsx
Vendored
+34
View File
@@ -0,0 +1,34 @@
import { Link } from '@/components/link';
export default function Page() {
return (
<main className="space-y-4">
<h1 className="text-xl font-semibold">
Vercel AI SDK Fundamentals with Portkey AI
</h1>
<p>
The following examples aim to showcase the fundamentals behind the
Vercel AI SDK. The examples have minimal loading states to remain as
simple as possible.
</p>
<p>
The prompt for the first 2 examples (stream/generate text) is `Tell me a
joke`.
</p>
<ul className="list-disc list-inside">
<li>
<Link href="/examples/generate-text">Generate Text</Link>
</li>
<li>
<Link href="/examples/stream-text">Stream Text</Link>
</li>
<li>
<Link href="/examples/tools/basic">Basic Tool</Link>
</li>
<li>
<Link href="/examples/basic-chatbot">Chatbot with `useChat`</Link>
</li>
</ul>
</main>
);
}
cookbook/integrations/vercel/components.json
Vendored
+17
View File
@@ -0,0 +1,17 @@
{
"$schema": "https://ui.shadcn.com/schema.json",
"style": "default",
"rsc": true,
"tsx": true,
"tailwind": {
"config": "tailwind.config.ts",
"css": "app/globals.css",
"baseColor": "neutral",
"cssVariables": true,
"prefix": ""
},
"aliases": {
"components": "@/components",
"utils": "@/lib/utils"
}
}
cookbook/integrations/vercel/components/back-button.tsx
Vendored
+9
View File
@@ -0,0 +1,9 @@
'use client';
import { Link } from './link';
import { usePathname } from 'next/navigation';
export const BackButton = () => {
const pathname = usePathname();
if (pathname == '/') return null;
return <Link href="/">Back</Link>;
};
cookbook/integrations/vercel/components/link.tsx
Vendored
+14
View File
@@ -0,0 +1,14 @@
import NextLink from 'next/link';
export const Link = ({
href,
children,
}: {
href: string;
children: React.ReactNode;
}) => {
return (
<NextLink className="hover:underline" href={href}>
{children}
</NextLink>
);
};
cookbook/integrations/vercel/components/ui/button.tsx
Vendored
+56
View File
@@ -0,0 +1,56 @@
import * as React from 'react';
import { Slot } from '@radix-ui/react-slot';
import { cva, type VariantProps } from 'class-variance-authority';
import { cn } from '@/lib/utils';
const buttonVariants = cva(
'inline-flex items-center justify-center whitespace-nowrap rounded-md text-sm font-medium ring-offset-background transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:pointer-events-none disabled:opacity-50',
{
variants: {
variant: {
default: 'bg-primary text-primary-foreground hover:bg-primary/90',
destructive:
'bg-destructive text-destructive-foreground hover:bg-destructive/90',
outline:
'border border-input bg-background hover:bg-accent hover:text-accent-foreground',
secondary:
'bg-secondary text-secondary-foreground hover:bg-secondary/80',
ghost: 'hover:bg-accent hover:text-accent-foreground',
link: 'text-primary underline-offset-4 hover:underline',
},
size: {
default: 'h-10 px-4 py-2',
sm: 'h-9 rounded-md px-3',
lg: 'h-11 rounded-md px-8',
icon: 'h-10 w-10',
},
},
defaultVariants: {
variant: 'default',
size: 'default',
},
}
);
export interface ButtonProps
extends React.ButtonHTMLAttributes<HTMLButtonElement>,
VariantProps<typeof buttonVariants> {
asChild?: boolean;
}
const Button = React.forwardRef<HTMLButtonElement, ButtonProps>(
({ className, variant, size, asChild = false, ...props }, ref) => {
const Comp = asChild ? Slot : 'button';
return (
<Comp
className={cn(buttonVariants({ variant, size, className }))}
ref={ref}
{...props}
/>
);
}
);
Button.displayName = 'Button';
export { Button, buttonVariants };
cookbook/integrations/vercel/components/ui/card.tsx
Vendored
+86
View File
@@ -0,0 +1,86 @@
import * as React from 'react';
import { cn } from '@/lib/utils';
const Card = React.forwardRef<
HTMLDivElement,
React.HTMLAttributes<HTMLDivElement>
>(({ className, ...props }, ref) => (
<div
ref={ref}
className={cn(
'rounded-lg border bg-card text-card-foreground shadow-sm',
className
)}
{...props}
/>
));
Card.displayName = 'Card';
const CardHeader = React.forwardRef<
HTMLDivElement,
React.HTMLAttributes<HTMLDivElement>
>(({ className, ...props }, ref) => (
<div
ref={ref}
className={cn('flex flex-col space-y-1.5 p-6', className)}
{...props}
/>
));
CardHeader.displayName = 'CardHeader';
const CardTitle = React.forwardRef<
HTMLParagraphElement,
React.HTMLAttributes<HTMLHeadingElement>
>(({ className, ...props }, ref) => (
<h3
ref={ref}
className={cn(
'text-2xl font-semibold leading-none tracking-tight',
className
)}
{...props}
/>
));
CardTitle.displayName = 'CardTitle';
const CardDescription = React.forwardRef<
HTMLParagraphElement,
React.HTMLAttributes<HTMLParagraphElement>
>(({ className, ...props }, ref) => (
<p
ref={ref}
className={cn('text-sm text-muted-foreground', className)}
{...props}
/>
));
CardDescription.displayName = 'CardDescription';
const CardContent = React.forwardRef<
HTMLDivElement,
React.HTMLAttributes<HTMLDivElement>
>(({ className, ...props }, ref) => (
<div ref={ref} className={cn('p-6 pt-0', className)} {...props} />
));
CardContent.displayName = 'CardContent';
const CardFooter = React.forwardRef<
HTMLDivElement,
React.HTMLAttributes<HTMLDivElement>
>(({ className, ...props }, ref) => (
<div
ref={ref}
className={cn('flex items-center p-6 pt-0', className)}
{...props}
/>
));
CardFooter.displayName = 'CardFooter';
export {
Card,
CardHeader,
CardFooter,
CardTitle,
CardDescription,
CardContent,
};
cookbook/integrations/vercel/components/ui/input.tsx
Vendored
+25
View File
@@ -0,0 +1,25 @@
import * as React from 'react';
import { cn } from '@/lib/utils';
export interface InputProps
extends React.InputHTMLAttributes<HTMLInputElement> {}
const Input = React.forwardRef<HTMLInputElement, InputProps>(
({ className, type, ...props }, ref) => {
return (
<input
type={type}
className={cn(
'flex h-10 w-full rounded-md border border-input bg-background px-3 py-2 text-sm ring-offset-background file:border-0 file:bg-transparent file:text-sm file:font-medium placeholder:text-muted-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:cursor-not-allowed disabled:opacity-50',
className
)}
ref={ref}
{...props}
/>
);
}
);
Input.displayName = 'Input';
export { Input };
cookbook/integrations/vercel/components/ui/label.tsx
Vendored
+26
View File
@@ -0,0 +1,26 @@
'use client';
import * as React from 'react';
import * as LabelPrimitive from '@radix-ui/react-label';
import { cva, type VariantProps } from 'class-variance-authority';
import { cn } from '@/lib/utils';
const labelVariants = cva(
'text-sm font-medium leading-none peer-disabled:cursor-not-allowed peer-disabled:opacity-70'
);
const Label = React.forwardRef<
React.ElementRef<typeof LabelPrimitive.Root>,
React.ComponentPropsWithoutRef<typeof LabelPrimitive.Root> &
VariantProps<typeof labelVariants>
>(({ className, ...props }, ref) => (
<LabelPrimitive.Root
ref={ref}
className={cn(labelVariants(), className)}
{...props}
/>
));
Label.displayName = LabelPrimitive.Root.displayName;
export { Label };
cookbook/integrations/vercel/core/generate-text.ts
Vendored
+16
View File
@@ -0,0 +1,16 @@
import { openai } from '@ai-sdk/openai';
import { generateText } from 'ai';
import dotenv from 'dotenv';
dotenv.config();
async function main() {
const result = await generateText({
model: openai('gpt-4o'),
prompt: 'Tell me a joke.',
});
console.log(result.text);
}
main().catch(console.error);
cookbook/integrations/vercel/core/stream-text.ts
Vendored
+18
View File
@@ -0,0 +1,18 @@
import { openai } from '@ai-sdk/openai';
import { streamText } from 'ai';
import dotenv from 'dotenv';
dotenv.config();
async function main() {
const result = await streamText({
model: openai('gpt-4o'),
prompt: 'Tell me a joke.',
});
for await (const textPart of result.textStream) {
process.stdout.write(textPart);
}
}
main().catch(console.error);
cookbook/integrations/vercel/core/tools.ts
Vendored
+40
View File
@@ -0,0 +1,40 @@
import { openai } from '@ai-sdk/openai';
import { generateText, streamText } from 'ai';
import dotenv from 'dotenv';
import { z } from 'zod';
dotenv.config();
async function main() {
const location = 'London';
const result = await generateText({
model: openai('gpt-4o'),
prompt: `You are a funny chatbot. users location: ${location}`,
tools: {
weather: {
description: "Get the weather for the user's location",
parameters: z.object({
location: z.string().describe("user's location"),
}),
execute: async ({ location }) => {
const temperature = Math.floor(Math.random() * 31); // call external api for {location}
return { temperature };
},
},
},
});
if (result.toolResults && result.toolCalls) {
const joke = await streamText({
model: openai('gpt-4o'),
prompt: `Tell me a joke that incorporates ${location}
and it's current temperature (${result.toolResults[0].result.temperature})`,
});
for await (const textPart of joke.textStream) {
process.stdout.write(textPart);
}
}
}
main().catch(console.error);
cookbook/integrations/vercel/lib/utils.ts
Vendored
+6
View File
@@ -0,0 +1,6 @@
import { type ClassValue, clsx } from 'clsx';
import { twMerge } from 'tailwind-merge';
export function cn(...inputs: ClassValue[]) {
return twMerge(clsx(inputs));
}
cookbook/integrations/vercel/next-env.d.ts
Vendored
+5
View File
@@ -0,0 +1,5 @@
/// <reference types="next" />
/// <reference types="next/image-types/global" />
// NOTE: This file should not be edited
// see https://nextjs.org/docs/app/building-your-application/configuring/typescript for more information.
cookbook/integrations/vercel/next.config.mjs
Vendored
+4
View File
@@ -0,0 +1,4 @@
/** @type {import('next').NextConfig} */
const nextConfig = {};
export default nextConfig;
cookbook/integrations/vercel/package-lock.json
Generated Vendored
+7999
View File
File diff suppressed because it is too large Load Diff
cookbook/integrations/vercel/package.json
Vendored
+42
View File
@@ -0,0 +1,42 @@
{
"name": "one",
"version": "0.1.0",
"private": true,
"scripts": {
"dev": "next dev --turbo",
"build": "next build",
"start": "next start",
"lint": "next lint"
},
"dependencies": {
"@ai-sdk/openai": "^0.0.66",
"@portkey-ai/vercel-provider": "^1.0.1",
"@radix-ui/react-label": "^2.1.8",
"@radix-ui/react-slot": "^1.2.4",
"ai": "^3.4.33",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"lucide-react": "^0.366.0",
"nanoid": "^5.1.6",
"next": "~14.2.35",
"react": "^18.3.1",
"react-dom": "^18.3.1",
"server-only": "^0.0.1",
"tailwind-merge": "^2.6.0",
"tailwindcss-animate": "^1.0.7",
"zod": "^3.25.76"
},
"devDependencies": {
"@types/node": "^20.19.27",
"@types/react": "^18.3.27",
"@types/react-dom": "^18.3.7",
"autoprefixer": "^10.4.23",
"dotenv": "^16.6.1",
"eslint": "^8.57.1",
"eslint-config-next": "14.1.4",
"postcss": "^8.5.6",
"tailwindcss": "^3.4.19",
"tsx": "^4.21.0",
"typescript": "^5.9.3"
}
}
cookbook/integrations/vercel/pnpm-lock.yaml
Generated Vendored
+4561
View File
File diff suppressed because it is too large Load Diff
cookbook/integrations/vercel/postcss.config.js
Vendored
+6
View File
@@ -0,0 +1,6 @@
module.exports = {
plugins: {
tailwindcss: {},
autoprefixer: {},
},
};
cookbook/integrations/vercel/public/next.svg
Vendored
+1
View File
@@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 394 80"><path fill="#000" d="M262 0h68.5v12.7h-27.2v66.6h-13.6V12.7H262V0ZM149 0v12.7H94v20.4h44.3v12.6H94v21h55v12.6H80.5V0h68.7zm34.3 0h-17.8l63.8 79.4h17.9l-32-39.7 32-39.6h-17.9l-23 28.6-23-28.6zm18.3 56.7-9-11-27.1 33.7h17.8l18.3-22.7z"/><path fill="#000" d="M81 79.3 17 0H0v79.3h13.6V17l50.2 62.3H81Zm252.6-.4c-1 0-1.8-.4-2.5-1s-1.1-1.6-1.1-2.6.3-1.8 1-2.5 1.6-1 2.6-1 1.8.3 2.5 1a3.4 3.4 0 0 1 .6 4.3 3.7 3.7 0 0 1-3 1.8zm23.2-33.5h6v23.3c0 2.1-.4 4-1.3 5.5a9.1 9.1 0 0 1-3.8 3.5c-1.6.8-3.5 1.3-5.7 1.3-2 0-3.7-.4-5.3-1s-2.8-1.8-3.7-3.2c-.9-1.3-1.4-3-1.4-5h6c.1.8.3 1.6.7 2.2s1 1.2 1.6 1.5c.7.4 1.5.5 2.4.5 1 0 1.8-.2 2.4-.6a4 4 0 0 0 1.6-1.8c.3-.8.5-1.8.5-3V45.5zm30.9 9.1a4.4 4.4 0 0 0-2-3.3 7.5 7.5 0 0 0-4.3-1.1c-1.3 0-2.4.2-3.3.5-.9.4-1.6 1-2 1.6a3.5 3.5 0 0 0-.3 4c.3.5.7.9 1.3 1.2l1.8 1 2 .5 3.2.8c1.3.3 2.5.7 3.7 1.2a13 13 0 0 1 3.2 1.8 8.1 8.1 0 0 1 3 6.5c0 2-.5 3.7-1.5 5.1a10 10 0 0 1-4.4 3.5c-1.8.8-4.1 1.2-6.8 1.2-2.6 0-4.9-.4-6.8-1.2-2-.8-3.4-2-4.5-3.5a10 10 0 0 1-1.7-5.6h6a5 5 0 0 0 3.5 4.6c1 .4 2.2.6 3.4.6 1.3 0 2.5-.2 3.5-.6 1-.4 1.8-1 2.4-1.7a4 4 0 0 0 .8-2.4c0-.9-.2-1.6-.7-2.2a11 11 0 0 0-2.1-1.4l-3.2-1-3.8-1c-2.8-.7-5-1.7-6.6-3.2a7.2 7.2 0 0 1-2.4-5.7 8 8 0 0 1 1.7-5 10 10 0 0 1 4.3-3.5c2-.8 4-1.2 6.4-1.2 2.3 0 4.4.4 6.2 1.2 1.8.8 3.2 2 4.3 3.4 1 1.4 1.5 3 1.5 5h-5.8z"/></svg>
Side by Side

After

Width:  |  Height:  |  Size: 1.3 KiB

Some files were not shown because too many files have changed in this diff Show More