Skip to content

LLM gateways

You want your applications to call a language model without each one carrying a provider key, its own retry logic and its own idea of what a request costs. On AWS the closest thing is putting API Gateway in front of Bedrock so that callers hold your credential rather than Amazon's.

Here that is an ordinary workload in your project: a Deployment, a Service and a route, holding provider credentials server-side and speaking the OpenAI API to everything in front of it.

Prerequisites

The step-by-step guide and the console walkthrough cover the operations. The rest of this page explains how it fits together.

What changes when you add one

flowchart TB
    %% the invisible link stacks the two cases vertically instead of side by side,
    %% which keeps both readable at page width
    subgraph WITHOUT["without a gateway"]
        direction TB
        A1["app A<br/>provider key"] --> P1["provider"]
        A2["app B<br/>provider key"] --> P1
        A3["notebook<br/>provider key"] --> P1
    end
    subgraph WITH["with a gateway"]
        direction TB
        B1["app A<br/>virtual key"] --> GW["gateway<br/>holds the provider key"]
        B2["app B<br/>virtual key"] --> GW
        B3["notebook<br/>virtual key"] --> GW
        GW --> P2["provider"]
        GW --> SELF["your own models"]
    end
    WITHOUT ~~~ WITH

The provider key stops being something each caller has a copy of and becomes something only the gateway has. Everything that follows from this page is a consequence of that one move.

Provider key in each app Gateway
Revoking one caller rotate the key, redeploy everything delete one virtual key
Who spent what one bill, no attribution per-key token and cost accounting
Adding a second provider change every caller register it once, behind a model name
A runaway loop discovered on the invoice refused at the rate limit
Self-hosted and external different SDKs, different code the same call, a different model name

Your own gateway, or a shared one

There are two ways to get one. This guide covers running your own.

Your own gateway A shared gateway
Runs in your project, on your quota infrastructure someone else operates
You operate it - configuration, upgrades, availability nothing
Providers and models whatever you register what its operator offers
Prompts and request logs stay inside your project handled under the operator's policy
Access keys you issue a key issued to you
Cost shape the resources it runs on per token

Run your own when the prompts must not leave your project, or when you want the provider list, routing and log retention under your own control. A shared gateway is the option when you want a model endpoint without operating anything - GRN.CLOUD's own shared, usage-billed gateway is still being built, so everything below describes running your own.

The two kinds of key

This is the distinction the whole product rests on, and mixing them up is the mistake worth avoiding first.

Provider key Virtual key
Issued by the model provider your gateway
Held by the gateway only each app, teammate or agent
Count one per provider as many as you like
Revoking it breaks every caller breaks one caller
Carries a budget no yes

A provider key is a bearer credential for an account that can spend real money with no ceiling. A virtual key is one you mint, scope and throw away. Callers only ever get the second kind - see Add a user to the gateway.

Model names are an interface

Clients ask for a model by name. The gateway decides what that name resolves to, and it can change that answer without the caller knowing: a different provider, a cheaper tier, a self-hosted endpoint instead of an external one.

That indirection is most of the value, and it comes with the usual obligation of an interface: once callers depend on a name, changing what it means changes their behaviour, and removing it breaks them. Choose names once and treat them as API surface.

What is fixed after creation

Changeable
Which providers are registered yes
What a model name routes to yes
A virtual key's budget and rate limit yes
The gateway's own hostname no - callers have it configured
A model name callers already use not safely

Every model call in your estate goes through one workload

Every model call in your estate goes through one workload. That is what makes budgets and auditing possible, and it is also what makes an outage total. Run more than one replica, and decide what your applications do when the gateway is unreachable before you need the answer.