Skip to content

LLM gateway

The gateway is a single OpenAI-compatible endpoint that sits between your applications and every model behind them — self-hosted models on GRN.CLOUD GPUs, and external providers.

Without a gateway, every application carries its own provider keys, its own retry logic and its own idea of what a request costs. With one, those move to a single place your team can inspect, budget and audit.

What it handles

Concern What the gateway does
Keys Issues virtual keys per team, app or user; provider keys stay server-side
Budgets Per-key spend limits and rate limits, enforced before the request leaves
Routing Sends a request to the right model by name, tag or policy
Failover Falls back to a secondary provider or tier when the primary is unhealthy
Observability Per-request logging of model, tokens, latency and cost
Compatibility OpenAI-shaped API, so existing SDKs work unchanged

The client's view

Because the API is OpenAI-compatible, most applications need a base URL change and nothing else:

from openai import OpenAI

client = OpenAI(
    base_url="https://<your-gateway-host>/v1",
    api_key="<your-virtual-key>",          # issued by the gateway, not the provider
)

response = client.chat.completions.create(
    model="<model-name>",
    messages=[{"role": "user", "content": "Hello"}],
)

Where it sits

flowchart LR
    APP1["App"] --> GW
    APP2["Agent"] --> GW
    APP3["Notebook"] --> GW
    GW["LLM gateway"] --> SELF["Self-hosted<br/>vLLM on GRN GPUs"]
    GW --> P1["External provider A"]
    GW --> P2["External provider B"]
    GW -.->|budgets · audit| ADMIN["Platform team"]

What this section will cover

  • Gateway configuration and routing policy
  • Virtual keys, budgets and rate limits
  • Provider registration and failover tiers
  • Request logging and cost attribution

Scaffold status

This is a placeholder. Configuration reference, routing policy syntax and the admin API are written here once confirmed against the platform.