Skip to content

Subnets

You want an isolated network for a set of workloads - and, if it is public, one whose addresses are reachable from outside the cluster. On AWS that is a VPC subnet, public or private.

Here a subnet is three things created together: a project, a network attached to it, and admin access to that project for you. Workloads deployed into the project attach to the network automatically.

Prerequisites

  • An account allowed to create projects
  • A free address range - the subnets list shows every range already claimed

The step-by-step guide and the console walkthrough cover the operations. The rest of this page explains how it fits together.

Public or private

flowchart TB
    subgraph PUB["public subnet"]
        direction TB
        NS2["Namespace"]
        CUDN2["ClusterUserDefinedNetwork"]
        RA["RouteAdvertisements"]
        NS2 --- CUDN2
        W2["workloads"] --- CUDN2
        CUDN2 --- RA
        RA -->|BGP| FAB["network fabric"]
        FAB --> OUT2["reachable from outside"]
    end
    subgraph PRIV["private subnet"]
        direction TB
        NS1["Namespace"]
        CUDN1["ClusterUserDefinedNetwork"]
        NS1 --- CUDN1
        W1["workloads"] --- CUDN1
        CUDN1 -.->|not advertised| OUT1["outside the cluster"]
    end
Private Public
Objects Namespace + network Namespace + network + RouteAdvertisements
Reachable from workloads in the same subnet the internet too
Address range any free private range from the platform's public pool

The advertisement is the entire difference in routing. A private subnet creates no such object and nothing leaves the cluster.

The two also start with different firewalls. The project of a private subnet is created closed: a default-deny-ingress firewall plus a few that admit the project's own workloads, the platform and web traffic - see what a new project starts with. The project of a public subnet is created with no firewall at all, so a workload in it is reachable from outside on every port it listens on, from the moment it starts.

What a workload in a subnet can reach

A subnet is a network of its own, and that cuts both ways. Nothing on the cluster reaches into it uninvited, and a workload inside it does not reach the rest of the cluster either - the platform's own services included.

From a workload in a subnet Reachable
Workloads in the same subnet, by address or through a Service yes
The Kubernetes API and cluster DNS yes
The internet, and any address served from outside this cluster yes
A Service in another project no
A Route or a load balancer address served by this same cluster no
A workload in another subnet no

The last three fail silently: the connection times out, and no firewall you add changes it. This is a property of the network, not of the firewalls.

What that rules out today, for a project that is a subnet:

  • PostgreSQL backups. A PostgreSQL cluster runs and serves workloads in the same subnet, but it cannot write backups or its write-ahead log to the platform's S3 storage.
  • S3 buckets. The endpoint of an S3 bucket is a service of this cluster, so a workload in a subnet cannot open it.
  • Builds. A build in the project cannot push its image to the platform's registry. Images built elsewhere are pulled and run as usual.
  • Names. Cluster DNS answers with an address the subnet cannot reach - see Fixed address.

Put the workloads that depend on those services in a normal project, and keep a subnet for what needs an address range of its own or a public address.

Publishing one port of a private workload

A workload in a private subnet has no public address, and does not need one to serve the internet. A Service of type LoadBalancer in the private subnet's project gets a public address from the platform pool and forwards one port to the workload - the arrangement AWS calls a load balancer in a public subnet in front of an instance in a private one. For a VM the selector is the label the platform puts on the VM's pod:

apiVersion: v1
kind: Service
metadata:
  name: <vm>-ssh
  namespace: <subnet>
spec:
  type: LoadBalancer
  selector:
    vm.kubevirt.io/name: <vm>
  ports:
  - name: ssh
    port: 22
    targetPort: 22

The project's firewalls still apply to what arrives through the load balancer. The web ports pass the default firewalls; any other port, 22 in this example, needs a firewall of its own before it answers. See Load balancers for the address and its lifetime.

Why you create a project, not a network

The project, the network and your access to it have to agree with one another, and the project is the only one of the three you have the rights to create. So you create it with the range you want recorded on it, and the platform builds the network - and the advertisement, on a public subnet - from that. It is also why the console form creates a project of its own instead of using the one you have selected.

Public subnets are carved out of the platform's public pool with a prefix of /28 or /29. The pool is small and shared.

The platform takes the first two addresses

The first usable address becomes the gateway workloads route through, and the second the node management port. OVN reserves both itself and neither has a field - a /29 gives four usable addresses rather than six. A /30 would give none, which is why it is rejected.

An out-of-pool or overlapping range is rejected at admission

A cluster policy checks the range when you submit, and names the problem in the rejection. That costs you one attempt rather than leaving a half-built subnet behind - and retrying the same range fails the same way every time.

The range cannot change

A network is created with its range and workloads take addresses from it. Making a private subnet public means adding an advertisement; making a public one private means deleting it. The network itself is unchanged either way.

Delete the network, not the project

The project and the advertisement belong to the network, so deleting the network removes all three. See Delete.

A project deleted on its own never finishes

It holds a network attachment whose finalizer is only released when the network is deleted, and the network is not removed while the project is still terminating. The two wait on each other. Deleting the network releases a project already stuck this way.

Deleting the network deletes the workloads in its project.