Skip to content

LLM gateway

The gateway is a single OpenAI-compatible endpoint that sits between your applications and every model behind them — self-hosted models on GRN.CLOUD GPUs, and external providers.

The gateway is powered by Bifrost and runs as an ordinary workload in your project: a Deployment, a Service, and a values file you keep in Git.

Without a gateway, every application carries its own provider keys, its own retry logic and its own idea of what a request costs. With one, those move to a single place your team can inspect, budget and audit.

What it handles

Concern What the gateway does
Keys Issues virtual keys per team, app or user; provider keys stay server-side
Budgets Per-key spend limits and rate limits, enforced before the request leaves
Routing Sends a request to the right model by name, tag or policy
Failover Falls back to a secondary provider or tier when the primary is unhealthy
Observability Per-request logging of model, tokens, latency and cost
Compatibility OpenAI-shaped API, so existing SDKs work unchanged

How it fits together

flowchart TB
    subgraph CALLERS["callers - each holds a virtual key"]
        direction LR
        APP1["app"]
        APP2["agent"]
        APP3["notebook"]
    end
    CALLERS --> GW["LLM gateway<br/>holds the provider keys"]
    GW --> SELF["your own models<br/>vLLM on GRN GPUs"]
    GW --> EXT["external providers"]
    GW -.->|budgets · request logs| ADMIN["platform team"]

Calling the gateway

Because the API is OpenAI-compatible, most applications need a base URL change and nothing else:

from openai import OpenAI

client = OpenAI(
    base_url="https://<your-gateway-host>/v1",
    api_key="<your-virtual-key>",          # issued by the gateway, not the provider
)

response = client.chat.completions.create(
    model="<model-name>",
    messages=[{"role": "user", "content": "Hello"}],
)

Registering a backend

Providers are declared in the values file. Each entry names the credential to use and the model names it answers for. A self-hosted engine is registered the same way an external vendor is:

values.yaml (excerpt)
bifrost:
  providerSecrets:                     # (1)!
    openai:
      envVar: OPENAI_API_KEY
      existingSecret: <provider-secret>
      key: openai-api-key
  providers:
    openai:
      keys:
      - name: openai-key
        value: env.OPENAI_API_KEY      # (2)!
        weight: 1
        models:
        - gpt-4o
        - gpt-4o-mini
    vllm:                              # (3)!
      keys:
      - name: <backend-name>
        value: ""
        weight: 1
        models:
        - <org>/<model>
        vllm_key_config:
          model_name: <org>/<model>
          url: http://<model-slug>.<your-project>.svc.cluster.local:8000

  1. Edit this. The Secret holding the upstream credential, created in the cluster and never committed. The gateway reads it as an environment variable.

  2. Leave as is. env. indirection is what keeps the key out of the values file. The literal credential never appears in Git.

  3. Edit this. A model you host yourself, registered as a provider like any other. The caller sees it as one more model name. See AI inference.

weight distributes traffic when several keys serve the same model name, which is also how failover between tiers is expressed.

Budgets, rate limits and virtual keys

The three are separate objects, referenced by id, so a limit can be written once and attached to many keys:

values.yaml (excerpt)
bifrost:
  governance:
    budgets:
    - id: standard-budget
      max_limit: 10                    # (1)!
      reset_duration: 1M
    rateLimits:
    - id: standard-rate
      request_max_limit: 100
      request_reset_duration: 1h
      token_max_limit: 100000
      token_reset_duration: 1d
    virtualKeys:
    - id: <key-id>
      name: <human-readable-name>
      description: <what this key is for>
      value: <the-secret-value>        # (2)!
      is_active: true
      budget_id: standard-budget
      rate_limit_id: standard-rate
      provider_configs:                # (3)!
      - provider: vllm
        allowed_models:
        - <org>/<model>
        weight: 1

  1. Tune this. Spend ceiling for the period in reset_duration. A budget caps the total; the rate limit caps the speed. A runaway loop needs the second one to stop quickly — the budget only notices once the money is gone.

  2. Edit this, and keep it out of Git. Read it from a secret store. A key committed to a repository is a key you have to rotate.

  3. Optional. Scoping a key to specific providers and models is what makes it safe to hand to CI or a batch job. Without it the key can reach every backend registered above.

What is fixed after creation

Changeable
Which providers are registered yes
What a model name routes to yes
A virtual key's budget and rate limit yes
The gateway's own hostname no; callers have it configured
A model name callers already use not safely

Connecting. The gateway serves the OpenAI API on port 8080:

What Where
In-cluster endpoint http://<gateway>.<your-project>.svc.cluster.local:8080/v1
Model list GET /v1/models
Chat completions POST /v1/chat/completions
Authentication Authorization: Bearer <your-virtual-key>

Deleting. Removing the gateway takes every model name with it, and every caller configured against its hostname fails at once. Move callers off the hostname before you remove the workload.

A 200 response does not prove the virtual key was checked

Key enforcement on inference requests is a configuration option, and it is not on by default in every deployment. A 200 proves the gateway is reachable, nothing more. Confirm that enforcement is enabled before treating a key as an access control.

Every model call in your estate goes through one workload

Centralising requests is what makes budgets and auditing possible, and it also means a gateway outage stops every model call at once. Run more than one replica, and decide what your applications do when the gateway is unreachable before you need the answer.

How to

Task Guide
Stand one up LLM gateways
Issue a scoped key with a budget Virtual keys
Put a backend behind a model name Model providers
Point a coding agent at it Coding agents