For a lot of teams, the first question about an AI gateway isn’t “what can it do” but “where does it run.” If you’re in a regulated industry, handling sensitive data, or simply allergic to sending every prompt and response through someone else’s cloud, a hosted SaaS gateway is a non-starter no matter how polished it is. You want something you can run inside your own infrastructure, inspect, and control.
The good news is that the open-source and self-hostable end of this market has gotten genuinely strong. An AI gateway is the control plane between your applications and your model providers, handling routing, reliability, security, and cost in one place, and several of the best now ship as software you can host yourself. Here are five worth shortlisting when self-hosting is a requirement, not a nice-to-have.
What “self-hostable” should actually mean
Before the list, a quick word on the trap. “Open source” and “self-hostable” are not the same thing, and neither guarantees the feature you need is in the free build. Some projects open-source a fast routing core but keep the governance, observability storage, and multi-team access control in a paid tier. So evaluate three things separately: whether the software runs in your own environment, whether the specific capabilities you need are in the version you can self-host, and whether data actually stays in your infrastructure end to end. With that lens, here’s how these five compare.
1. TrueFoundry — the enterprise control plane you run in your own cloud
TrueFoundry leads this list because it was built for exactly the constraint that pushes teams toward self-hosting: keeping data in your own environment without giving up enterprise capability. Its AI Gateway deploys inside your own cloud or VPC, so prompts and traces stay in your infrastructure region by region, while still giving you a single OpenAI-compatible interface to 1,000+ models, load balancing and fallbacks, a full guardrails suite, role-based access, budgets, and OpenTelemetry observability.
It also reaches further than a pure proxy. TrueFoundry governs MCP servers and agents through the same control plane, and its agent harness, TrueForge, is open source and vendor-neutral, so the pieces you’d want to inspect and extend are open while the platform itself runs where you need it. For teams that want the self-hosting posture of an open-source project but the depth and support of an enterprise platform, this is the combination that’s hard to find elsewhere. FloQast, for example, runs it entirely inside its own cloud to keep financial data resident across US, EU, and APAC regions, which you can read about in TrueFoundry’s AI Gateway documentation and case studies.
Best for: enterprises that need data residency and full governance while running the gateway themselves. Watch-out: it’s a rich platform, so plan the rollout rather than expecting a five-minute install.
2. Portkey — a fast, popular open-source routing core
Portkey open-sourced its gateway under an MIT license, and the core is a genuinely good piece of engineering: lightweight, very low added latency, routing to a large model catalog with integrated guardrails, and easy to run as a container you host yourself. If you want a proven open-source proxy to build on, it’s one of the most popular starting points for good reason.
The caveat is where the line falls between free and paid. The self-hosted build gives you the routing and reliability layer, but the observability storage and dashboards, multi-team role-based access, semantic caching at large scale, and enterprise support sit in the hosted or enterprise tier. So it self-hosts well as a router; a full governance stack usually means the paid plan.
Best for: teams that want a lean, self-hosted routing layer. Watch-out: confirm which governance features live in the open build before you standardize on it.
3. Envoy AI Gateway — the CNCF-backed open standard
Envoy AI Gateway extends the CNCF’s Envoy Gateway project with native support for AI traffic, and it reached a stable 1.0 in 2026. Created and maintained by Tetrate together with Bloomberg, it’s the first CNCF-backed open-source AI gateway, which matters if you care about vendor-neutral governance and a project that isn’t controlled by a single commercial vendor.
It offers an OpenAI-compatible interface across the major model providers, plus routing, failover, and OpenTelemetry GenAI tracing across chat, embeddings, and other endpoints. Bloomberg already runs it in production. The trade-off is that it’s infrastructure-grade software: powerful and standards-aligned, but it expects a team comfortable operating Envoy and Kubernetes, and higher-level conveniences are more DIY than in a managed platform.
Best for: platform teams that want a CNCF-standard, Envoy-based gateway they fully control. Watch-out: budget for the operational expertise Envoy and Kubernetes demand.
4. Higress — a high-scale, AI-native open-source gateway
Higress started inside Alibaba, was open-sourced under Apache-2.0, and is built on Istio and Envoy. It’s battle-tested at large scale, with production validation handling hundreds of thousands of requests per second and configuration changes that take effect in milliseconds rather than forcing disruptive reloads.
On the AI side, it supports mainstream model providers, can host MCP servers through its plugin mechanism so agents can call tools, and provides unified management for both LLM and MCP APIs, along with WAF and a range of authentication strategies. It’s an excellent choice if raw performance and a proven high-throughput core are your priorities. As with other infrastructure-first projects, the higher-level governance and product polish are lighter than a dedicated enterprise platform, and the community center of gravity may be less familiar to some Western teams.
Best for: teams that need a high-throughput, fully open-source AI-native gateway. Watch-out: you’ll assemble more of the governance layer yourself.
5. Gloo AI Gateway — Envoy-based and Kubernetes-native
Gloo AI Gateway, from Solo.io, builds AI traffic management on top of a mature Envoy- and Kubernetes-native gateway. It targets the specific needs of LLM traffic: semantic caching, token-based rate limiting, prompt enrichment, credential injection via Kubernetes secrets, and data-exfiltration protection to keep sensitive information from leaking. Solo.io donated its open-source gateway to the CNCF (as kgateway), so there’s a genuine open core.
The nuance mirrors the pattern on this list: some of the LLM-specific capabilities, such as token-based rate limiting, are typically part of the enterprise license rather than the free core. If you already run Kubernetes and want an Envoy-based gateway that treats AI traffic as a first-class citizen, it’s a strong fit as long as you map which features you need to the right tier.
Best for: Kubernetes-heavy teams wanting an Envoy-based gateway with AI traffic features. Watch-out: check which LLM features require the enterprise edition.
How to choose
If you want a pure open-source project to run and operate yourself, Envoy AI Gateway and Higress give you standards-aligned, high-performance cores, and Gloo is a strong pick when you’re already committed to Envoy on Kubernetes. Portkey is the lightest way to self-host a routing layer, provided you accept the tier boundaries. And if what you actually need is data staying in your own cloud with full enterprise governance and support behind it, TrueFoundry is the option that gives you the self-hosted posture without asking you to build the governance stack yourself.
The bottom line
Self-hosting an AI gateway used to mean giving up features for control. In 2026 that’s no longer true: you can get a CNCF-backed open standard, a high-throughput AI-native core, or a fully governed enterprise platform that runs inside your own environment. The right pick comes down to how much you want to operate yourself versus how much governance you want out of the box. For teams that want both control and depth, TrueFoundry is the one to beat.
