diff --git a/README.en.md b/README.en.md new file mode 100644 index 0000000..199b744 --- /dev/null +++ b/README.en.md @@ -0,0 +1,338 @@ + +

+ English | 中文 | 日本語 +

+ +> [!IMPORTANT] +> :rocket: Gateway 2.0 (Pre-Release) Portkey's core enterprise gateway is merging into open-source with our 2.0 release. You can try the pre-release branch [here](https://github.com/portkey-ai/gateway/tree/2.0.0). +> Read more about what's next for Portkey in our [**Series A announcement**](https://portkey.wiki/rohit-a). + + +
+ +🆕 **[Portkey Models](https://github.com/Portkey-AI/models)** - Open-source LLM pricing for 2,300+ models across 40+ providers. [Explore →](https://portkey.ai/models) + + +# AI Gateway +#### Route to 250+ LLMs with 1 fast & friendly API + +Portkey AI Gateway Demo showing LLM routing capabilities + +[Docs](https://portkey.wiki/gh-1) | [Enterprise](https://portkey.wiki/gh-2) | [Hosted Gateway](https://portkey.wiki/gh-3) | [Changelog](https://portkey.wiki/gh-4) | [API Reference](https://portkey.wiki/gh-5) + + +[![License](https://img.shields.io/github/license/Ileriayo/markdown-badges)](./LICENSE) +[![Discord](https://img.shields.io/discord/1143393887742861333)](https://portkey.wiki/gh-6) +[![Twitter](https://img.shields.io/twitter/url/https/twitter/follow/portkeyai?style=social&label=Follow%20%40PortkeyAI)](https://portkey.wiki/gh-7) +[![npm version](https://badge.fury.io/js/%40portkey-ai%2Fgateway.svg)](https://portkey.wiki/gh-8) +[![Better Stack Badge](https://uptime.betterstack.com/status-badges/v1/monitor/q94g.svg)](https://portkey.wiki/gh-9) + +Deploy to AWS EC2 [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/Portkey-AI/gateway) +
+ +
+ +The [**AI Gateway**](https://portkey.wiki/gh-10) is designed for fast, reliable & secure routing to 1600+ language, vision, audio, and image models. It is a lightweight, open-source, and enterprise-ready solution that allows you to integrate with any language model in under 2 minutes. + +- [x] **Blazing fast** (<1ms latency) with a tiny footprint (122kb) +- [x] **Battle tested**, with over 10B tokens processed everyday +- [x] **Enterprise-ready** with enhanced security, scale, and custom deployments + +
+ +#### What can you do with the AI Gateway? +- Integrate with any LLM in under 2 minutes - [Quickstart](#quickstart-2-mins) +- Prevent downtimes through **[automatic retries](https://portkey.wiki/gh-11)** and **[fallbacks](https://portkey.wiki/gh-12)** +- Scale AI apps with **[load balancing](https://portkey.wiki/gh-13)** and **[conditional routing](https://portkey.wiki/gh-14)** +- Protect your AI deployments with **[guardrails](https://portkey.wiki/gh-15)** +- Go beyond text with **[multi-modal capabilities](https://portkey.wiki/gh-16)** +- Explore **[agentic workflow](https://portkey.wiki/gh-17)** integrations +- Manage MCP servers with enterprise auth & observability using **[MCP Gateway](https://portkey.ai/docs/product/mcp-gateway)** + +

+ +> [!TIP] +> Starring this repo helps more developers discover the AI Gateway 🙏🏻 +> +> ![star-2](https://github.com/user-attachments/assets/53597dce-6333-4ecc-a154-eb05532954e4) +> +
+ + +
+ +## Quickstart (2 mins) + +### 1. Setup your AI Gateway + +```bash +# Run the gateway locally (needs Node.js and npm) +npx @portkey-ai/gateway +``` +> The Gateway is running on `http://localhost:8787/v1` +> +> The Gateway Console is running on `http://localhost:8787/public/` + + +Deployment guides: +  Portkey Cloud (Recommended) +  Docker +  Node.js +  Cloudflare +  Replit +  Others... + + + +### 2. Make your first request + + +```python +# pip install -qU portkey-ai + +from portkey_ai import Portkey + +# OpenAI compatible client +client = Portkey( + provider="openai", # or 'anthropic', 'bedrock', 'groq', etc + Authorization="sk-***" # the provider API key +) + +# Make a request through your AI Gateway +client.chat.completions.create( + messages=[{"role": "user", "content": "What's the weather like?"}], + model="gpt-4o-mini" +) +``` + + + +Supported Libraries: +  [ JS](https://portkey.wiki/gh-19) +  [ Python](https://portkey.wiki/gh-20) +  [ REST](https://portkey.sh/gh-84) +  [ OpenAI SDKs](https://portkey.wiki/gh-21) +  [ Langchain](https://portkey.wiki/gh-22) +  [LlamaIndex](https://portkey.wiki/gh-23) +  [Autogen](https://portkey.wiki/gh-24) +  [CrewAI](https://portkey.wiki/gh-25) +  [More..](https://portkey.wiki/gh-26) + + +On the Gateway Console (`http://localhost:8787/public/`) you can see all of your local logs in one place. + + + + +### 3. Routing & Guardrails +`Configs` in the LLM gateway allow you to create routing rules, add reliability and setup guardrails. +```python +config = { + "retry": {"attempts": 5}, + + "output_guardrails": [{ + "default.contains": {"operator": "none", "words": ["Apple"]}, + "deny": True + }] +} + +# Attach the config to the client +client = client.with_options(config=config) + +client.chat.completions.create( + model="gpt-4o-mini", + messages=[{"role": "user", "content": "Reply randomly with Apple or Bat"}] +) + +# This would always response with "Bat" as the guardrail denies all replies containing "Apple". The retry config would retry 5 times before giving up. +``` +
+Request flow through Portkey's AI gateway with retries and guardrails +
+ +You can do a lot more stuff with configs in your AI gateway. [Jump to examples →](https://portkey.wiki/gh-27) + +
+ +### Enterprise Version (Private deployments) + + + +[ AWS](https://portkey.wiki/gh-28) +  [ Azure](https://portkey.wiki/gh-29) +  [ GCP](https://portkey.wiki/gh-30) +  [ OpenShift](https://portkey.wiki/gh-31) +  [ Kubernetes](https://portkey.wiki/gh-85) + + + +The LLM Gateway's [enterprise version](https://portkey.wiki/gh-86) offers advanced capabilities for **org management**, **governance**, **security** and [more](https://portkey.wiki/gh-87) out of the box. [View Feature Comparison →](https://portkey.wiki/gh-32) + +The enterprise deployment architecture for supported platforms is available here - [**Enterprise Private Cloud Deployments**](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture) + +Book an enterprise AI gateway demo
+ +
+ +## MCP Gateway + +[MCP Gateway](https://portkey.ai/docs/product/mcp-gateway) provides a centralized control plane for managing MCP (Model Context Protocol) servers across your organization. + +- **Authentication** — Single auth layer at the gateway. Users authenticate once; your MCP servers receive verified requests +- **Access Control** — Control which teams and users can access which servers and tools. Revoke access instantly +- **Observability** — Every tool call logged with full context: who called what, parameters, response, latency +- **Identity Forwarding** — Forward user identity (email, team, roles) to MCP servers automatically + +Works with Claude Desktop, Cursor, VS Code, and any MCP-compatible client. [Get started →](https://portkey.ai/docs/product/mcp-gateway/quickstart) + +
+ +## Core Features +### Reliable Routing +- **Fallbacks**: Fallback to another provider or model on failed requests using the LLM gateway. You can specify the errors on which to trigger the fallback. Improves reliability of your application. +- **Automatic Retries**: Automatically retry failed requests up to 5 times. An exponential backoff strategy spaces out retry attempts to prevent network overload. +- **Load Balancing**: Distribute LLM requests across multiple API keys or AI providers with weights to ensure high availability and optimal performance. +- **Request Timeouts**: Manage unruly LLMs & latencies by setting up granular request timeouts, allowing automatic termination of requests that exceed a specified duration. +- **Multi-modal LLM Gateway**: Call vision, audio (text-to-speech & speech-to-text), and image generation models from multiple providers — all using the familiar OpenAI signature +- **Realtime APIs**: Call realtime APIs launched by OpenAI through the integrate websockets server. + +### Security & Accuracy +- **Guardrails**: Verify your LLM inputs and outputs to adhere to your specified checks. Choose from the 40+ pre-built guardrails to ensure compliance with security and accuracy standards. You can bring your own guardrails or choose from our many partners. +- [**Secure Key Management**](https://portkey.wiki/gh-45): Use your own keys or generate virtual keys on the fly. +- [**Role-based access control**](https://portkey.wiki/gh-46): Granular access control for your users, workspaces and API keys. +- **Compliance & Data Privacy**: The AI gateway is SOC2, HIPAA, GDPR, and CCPA compliant. + +### Cost Management +- [**Smart caching**](https://portkey.wiki/gh-48): Cache responses from LLMs to reduce costs and improve latency. Supports simple and semantic* caching. +- [**Usage analytics**](https://portkey.wiki/gh-49): Monitor and analyze your AI and LLM usage, including request volume, latency, costs and error rates. +- [**Provider optimization***](https://portkey.wiki/gh-89): Automatically switch to the most cost-effective provider based on usage patterns and pricing models. + +### Collaboration & Workflows +- **Agents Support**: Seamlessly integrate with popular agent frameworks to build complex AI applications. The gateway seamlessly integrates with [Autogen](https://portkey.wiki/gh-50), [CrewAI](https://portkey.wiki/gh-51), [LangChain](https://portkey.wiki/gh-52), [LlamaIndex](https://portkey.wiki/gh-53), [Phidata](https://portkey.wiki/gh-54), [Control Flow](https://portkey.wiki/gh-55), and even [Custom Agents](https://portkey.wiki/gh-56). +- [**Prompt Template Management***](https://portkey.wiki/gh-57): Create, manage and version your prompt templates collaboratively through a universal prompt playground. +

+ + +* Available in hosted and enterprise versions + + +
+ +## Portkey Models +Open-source LLM pricing database for 40+ providers - used by the Gateway for cost tracking. + +[GitHub](https://github.com/Portkey-AI/models) | [Model Explorer](https://portkey.ai/models) + +
+ +## Cookbooks + +### ☄️ Trending +- Use models from [Nvidia NIM](/cookbook/providers/nvidia.ipynb) with AI Gateway +- Monitor [CrewAI Agents](/cookbook/monitoring-agents/CrewAI_with_Telemetry.ipynb) with Portkey! +- Comparing [Top 10 LMSYS Models](/cookbook/use-cases/LMSYS%20Series/comparing-top10-LMSYS-models-with-Portkey.ipynb) with AI Gateway. + +### 🚨 Latest +* [Create Synthetic Datasets using Nemotron](/cookbook/use-cases/Nemotron_GPT_Finetuning_Portkey.ipynb) +* [Use the LLM Gateway with Vercel's AI SDK](/cookbook/integrations/vercel-ai.md) +* [Monitor Llama Agents with Portkey's LLM Gateway](/cookbook/monitoring-agents/Llama_Agents_with_Telemetry.ipynb) + +[View all cookbooks →](https://portkey.wiki/gh-58) +

+ +## Supported Providers + +Explore Gateway integrations with [45+ providers](https://portkey.wiki/gh-59) and [8+ agent frameworks](https://portkey.wiki/gh-90). + +| | Provider | Support | Stream | +| -------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | ------- | ------ | +| | [OpenAI](https://portkey.wiki/gh-60) | ✅ | ✅ | +| | [Azure OpenAI](https://portkey.wiki/gh-61) | ✅ | ✅ | +| | [Anyscale](https://portkey.wiki/gh-62) | ✅ | ✅ | +| | [Google Gemini](https://portkey.wiki/gh-63) | ✅ | ✅ | +| | [Anthropic](https://portkey.wiki/gh-64) | ✅ | ✅ | +| | [Cohere](https://portkey.wiki/gh-65) | ✅ | ✅ | +| | [Together AI](https://portkey.wiki/gh-66) | ✅ | ✅ | +| | [Perplexity](https://portkey.wiki/gh-67) | ✅ | ✅ | +| | [Mistral](https://portkey.wiki/gh-68) | ✅ | ✅ | +| | [Nomic](https://portkey.wiki/gh-69) | ✅ | ✅ | +| | [AI21](https://portkey.wiki/gh-91) | ✅ | ✅ | +| | [Stability AI](https://portkey.wiki/gh-71) | ✅ | ✅ | +| | [DeepInfra](https://portkey.sh/gh-92) | ✅ | ✅ | +| | [Ollama](https://portkey.wiki/gh-72) | ✅ | ✅ | +| | [Novita AI](https://portkey.wiki/gh-73) | ✅ | ✅ | `/chat/completions`, `/completions` | + + +> [View the complete list of 200+ supported models here](https://portkey.wiki/gh-74) +
+ +
+ +## Agents +Gateway seamlessly integrates with popular agent frameworks. [Read the documentation here](https://portkey.wiki/gh-75). + + +| Framework | Call 200+ LLMs | Advanced Routing | Caching | Logging & Tracing* | Observability* | Prompt Management* | +|------------------------------|--------|-------------|---------|------|---------------|-------------------| +| [Autogen](https://portkey.wiki/gh-93) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| [CrewAI](https://portkey.wiki/gh-94) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| [LangChain](https://portkey.wiki/gh-95) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| [Phidata](https://portkey.wiki/gh-96) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| [Llama Index](https://portkey.wiki/gh-97) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| [Control Flow](https://portkey.wiki/gh-98) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| [Build Your Own Agents](https://portkey.wiki/gh-99) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| | [IO Intelligence](https://io.net/intelligence) | ✅ | ✅ | + +
+ +*Available on the [hosted app](https://portkey.wiki/gh-76). For detailed documentation [click here](https://portkey.wiki/gh-100). + + +## Gateway Enterprise Version +Make your AI app more reliable and forward compatible, while ensuring complete data security and privacy. + +✅  Secure Key Management - for role-based access control and tracking
+✅  Simple & Semantic Caching - to serve repeat queries faster & save costs
+✅  Access Control & Inbound Rules - to control which IPs and Geos can connect to your deployments
+✅  PII Redaction - to automatically remove sensitive data from your requests to prevent indavertent exposure
+✅  SOC2, ISO, HIPAA, GDPR Compliances - for best security practices
+✅  Professional Support - along with feature prioritization
+ +[Schedule a call to discuss enterprise deployments](https://portkey.sh/demo-13) + +
+ + +## Contributing + +The easiest way to contribute is to pick an issue with the `good first issue` tag 💪. Read the contribution guidelines [here](/.github/CONTRIBUTING.md). + +Bug Report? [File here](https://portkey.wiki/gh-78) | Feature Request? [File here](https://portkey.wiki/gh-78) + + +### Getting Started with the Community +Join our weekly AI Engineering Hours every Friday (8 AM PT) to: +- Meet other contributors and community members +- Learn advanced Gateway features and implementation patterns +- Share your experiences and get help +- Stay updated with the latest development priorities + +[Join the next session →](https://portkey.wiki/gh-101) | [Meeting notes](https://portkey.wiki/gh-102) + +
+ +## Community + +Join our growing community around the world, for help, ideas, and discussions on AI. + +- View our official [Blog](https://portkey.wiki/gh-78) +- Chat with us on [Discord](https://portkey.wiki/community) +- Follow us on [Twitter](https://portkey.wiki/gh-79) +- Connect with us on [LinkedIn](https://portkey.wiki/gh-80) +- Read the documentation in [Japanese](./.github/README.jp.md) +- Visit us on [YouTube](https://portkey.wiki/gh-103) +- Join our [Dev community](https://portkey.wiki/gh-82) + + +![Rubeus Social Share (4)](https://github.com/Portkey-AI/gateway/assets/971978/89d6f0af-a95d-4402-b451-14764c40d03f)