diff --git a/docs/networking/kubeovn-pureunderlay.md b/docs/networking/kubeovn-pureunderlay.md index 4070edabf2..932619114c 100644 --- a/docs/networking/kubeovn-pureunderlay.md +++ b/docs/networking/kubeovn-pureunderlay.md @@ -95,11 +95,13 @@ When setting up a pure underlay, the overlay network must use the following conf For descriptions of settings that you must configure, see [Subnet Settings](./kubeovn-vpc.md#subnet-settings). -![](/img/subnetexternal.png) +![](/img/subnetdhcp.png) :::info important -When setting up a pure underlay, you must specify the name of the VLAN network that is mapped to the provider network. +When setting up a pure underlay network, specify the name of the VLAN network mapped to the provider network. + +If you are using this network as an external network for a VPC NAT gateway, ensure that DHCP is enabled, `dns_server` is set to a valid DNS server IP address (for example, `8.8.8.8`) and `natOutgoing` is set to `true`. This configuration ensures that virtual machines using these subnets can resolve DNS and route traffic through the default gateway. ::: @@ -108,3 +110,9 @@ When setting up a pure underlay, you must specify the name of the VLAN network t You can create virtual machines and attach them directly to a VLAN Network mapped to the configured provider network. These virtual machines can communicate with other virtual machines on the same subnet and route traffic directly to external networks via the physical infrastructure. For more information, see [Create a Virtual Machine](../vm/create-vm.md#how-to-create-a-vm). + +:::note + +The VLAN ID referenced in this guide is an example and can be replaced with any VLAN that fits your network design. When using a VPC NAT Gateway, the external network and external IP address must belong to the underlay network's designated VLAN to ensure proper connectivity. + +::: diff --git a/docs/networking/kubeovn-vpcnatgateway.md b/docs/networking/kubeovn-vpcnatgateway.md index 0d3a3bd080..4f1e9a23aa 100644 --- a/docs/networking/kubeovn-vpcnatgateway.md +++ b/docs/networking/kubeovn-vpcnatgateway.md @@ -33,560 +33,487 @@ In Harvester v1.6.x and v1.7.x, Outbound and Inbound connectivity for VMs works With the introduction of kubeovn as secondary CNI in Harvester v1.8.0, Outbound and Inbound connectivity for VMs works on subnets created on any custom VPCs using VPC NAT Gateway functionality. -### Example Configuration and Working +### Create a Tenant Network -#### Create a network attachment definition (tenant or internal network) +1. [Create an overlay network](./harvester-network.md#create-an-overlay-network). -``` -apiVersion: k8s.cni.cncf.io/v1 -kind: NetworkAttachmentDefinition -metadata: - labels: - network.harvesterhci.io/ready: "true" - network.harvesterhci.io/type: OverlayNetwork - name: vswitchinternal - namespace: default -spec: - config: '{"cniVersion":"0.3.1","name":"vswitchinternal","type":"kube-ovn","server_socket": - "/run/openvswitch/kube-ovn-daemon.sock", "provider": "vswitchinternal.default.ovn"}' -``` +1. [Create a custom VPC](./kubeovn-vpc.md#vpc-settings). -#### Create a network attachment definition (external network) +1. [Create a subnet](./kubeovn-vpc.md#subnet-settings) inside the VPC created in the previous step. -``` -kubectl get net-attach-def vswitchexternal1 -o yaml -apiVersion: k8s.cni.cncf.io/v1 -kind: NetworkAttachmentDefinition -metadata: - labels: - network.harvesterhci.io/ready: "true" - network.harvesterhci.io/type: OverlayNetwork - name: vswitchexternal - namespace: kube-system -spec: - config: '{"cniVersion":"0.3.1","name":"vswitchexternal","master": "eno50","type":"kube-ovn","server_socket": - "/run/openvswitch/kube-ovn-daemon.sock", "provider": "vswitchexternal.kube-system.ovn"}' + ![](/img/subnetinternal.png) -``` + :::info important -#### Create a custom vpc + When creating a subnet for a tenant network, ensure DHCP is enabled, `dns_server` is set to a valid DNS server IP address (for example, `8.8.8.8`), and `natOutgoing` is set to `true`. This configuration ensures that a default route is installed and DNS is resolved on virtual machines attached to these subnets. -``` -apiVersion: kubeovn.io/v1 -kind: Vpc -metadata: - name: custom-vpc -spec: {} + ::: -``` -#### create a subnet using the internal or tenant network in `custom-vpc` +The examples in this document use the following settings: -``` -apiVersion: kubeovn.io/v1 -kind: Subnet -metadata: - name: subnetinternal -spec: - cidrBlock: 172.20.10.0/24 - default: false - enableLb: true - excludeIps: - - 172.20.10.1 - gateway: 172.20.10.1 - gatewayNode: "" - natOutgoing: true - private: false - protocol: IPv4 - provider: vswitchinternal.default.ovn - vpc: custom-vpc +- **Overlay network**: `vswitchinternal` +- **VPC**: `commonvpc` +- **Subnet (in `commonvpc`)**: `subnetinternal` -``` -#### Create a subnet using the external network +### Create an External Network -``` -apiVersion: kubeovn.io/v1 -kind: Subnet -metadata: - name: subnetexternal -spec: - cidrBlock: 10.115.48.0/21 - default: false - enableLb: true - excludeIps: - - 10.115.55.254 - gateway: 10.115.55.254 - gatewayNode: "" - natOutgoing: true - private: false - protocol: IPv4 - provider: vswitchexternal.kube-system.ovn - vpc: custom-vpc + Follow [underlay network](./kubeovn-pureunderlay.md#underlay-configuration) to create an external network. -``` +### VPC NAT Gateway -#### Create the vpc nat gateway config +1. On the Harvester UI, go to **Overlay Networks > NAT & Internet > Gateways**. -``` -kind: VpcNatGateway -apiVersion: kubeovn.io/v1 -metadata: - annotations: - k8s.v1.cni.cncf.io/networks: default/vswitchinternal - name: gw1 -spec: - vpc: custom-vpc - subnet: subnetinternal - lanIp: 172.20.10.254 - externalSubnets: - - subnetexternal + ![](/img/vpc-nat-config.png) -``` -#### Verify if a new vpcnatgw statefulset and a pod created +1. Click **Create**. -``` -kubectl get statefulset -n kube-system vpc-nat-gw-gw1 -o yaml -apiVersion: apps/v1 -kind: StatefulSet -metadata: - creationTimestamp: "2026-03-06T22:33:47Z" - generation: 1 - labels: - app: vpc-nat-gw-gw1 - ovn.kubernetes.io/vpc-nat-gw: "true" - name: vpc-nat-gw-gw1 - namespace: kube-system - resourceVersion: "12139577" - uid: cb48e680-95e8-408d-be47-23499ad0ac5c -spec: - persistentVolumeClaimRetentionPolicy: - whenDeleted: Retain - whenScaled: Retain - podManagementPolicy: OrderedReady - replicas: 1 - revisionHistoryLimit: 10 - selector: - matchLabels: - app: vpc-nat-gw-gw1 - ovn.kubernetes.io/vpc-nat-gw: "true" - serviceName: "" - template: +1. Specify a unique name for the VPC NAT gateway. + +1. On the **Basic** tab, configure the following settings: + + - **Internal Tenant Network**: Name of the internal tenant network (for example, `vswitchinternal`) + + - **VPC** VPC of the internal tenant network subnet (for example, `commonvpc`) + + - **Subnet**: Subnet of the internal tenant network (for example, `subnetinternal`) + + - **LAN IP**: Gateway IP address of the internal tenant network + +1. On the **External Subnets** tab, configure the following settings: + + - **Subnet**: Subnet of the underlay external network (for example, `subnetexternal`) + +1. Click **Create**. + +#### Verification and Example Outputs + +1. Verify if a new vpcnatgw statefulset and a pod created + + ``` + $ kubectl get statefulset -n kube-system vpc-nat-gw-gw1 -o yaml + apiVersion: apps/v1 + kind: StatefulSet metadata: - annotations: - externalvswitch.kube-system.ovn.kubernetes.io/routes: '[{"dst":"0.0.0.0/0","gw":"10.115.55.254"}]' - internalvswitch.default.ovn.kubernetes.io/ip_address: 172.20.10.254 - internalvswitch.default.ovn.kubernetes.io/logical_switch: internalsubnet - internalvswitch.default.ovn.kubernetes.io/routes: '[{"dst":"10.55.0.0/16","gw":"172.20.10.1"},{"dst":"10.115.48.0/21","gw":"172.20.10.1"}]' - internalvswitch.default.ovn.kubernetes.io/vpc_nat_gw: gw1 - k8s.v1.cni.cncf.io/networks: default/internalvswitch, kube-system/externalvswitch - ovn.kubernetes.io/ip_address: 172.20.10.254 - ovn.kubernetes.io/logical_switch: internalsubnet - ovn.kubernetes.io/vpc_nat_gw: gw1 + creationTimestamp: "2026-06-05T22:42:17Z" + generation: 1 labels: app: vpc-nat-gw-gw1 ovn.kubernetes.io/vpc-nat-gw: "true" + name: vpc-nat-gw-gw1 + namespace: kube-system + resourceVersion: "18233827" + uid: ae7c9895-8a7f-46f6-b71d-be9f9b36b79a spec: - affinity: {} - containers: - - command: - - sleep - - infinity - env: - - name: GATEWAY_V4 - value: 10.115.55.254 - - name: GATEWAY_V6 - image: docker.io/kubeovn/vpc-nat-gateway:v1.15.0 - imagePullPolicy: IfNotPresent - lifecycle: - postStart: - exec: - command: - - sh - - -c - - sysctl -w net.ipv4.ip_forward=1 - name: vpc-nat-gw - resources: {} - securityContext: - allowPrivilegeEscalation: true - privileged: true - terminationMessagePath: /dev/termination-log - terminationMessagePolicy: File - dnsPolicy: ClusterFirst - restartPolicy: Always - schedulerName: default-scheduler - securityContext: {} - terminationGracePeriodSeconds: 0 - updateStrategy: - type: RollingUpdate -status: - availableReplicas: 1 - collisionCount: 0 - currentReplicas: 1 - currentRevision: vpc-nat-gw-gw1-98ff96d76 - observedGeneration: 1 - readyReplicas: 1 - replicas: 1 - updateRevision: vpc-nat-gw-gw1-98ff96d76 - updatedReplicas: 1 -``` -``` -kubectl describe pod vpc-nat-gw-gw1-0 -n kube-system -Name: vpc-nat-gw-gw1-0 -Namespace: kube-system -Priority: 0 -Service Account: default -Node: hp-46/10.115.14.70 -Start Time: Fri, 06 Mar 2026 22:33:47 +0000 -Labels: app=vpc-nat-gw-gw1 - apps.kubernetes.io/pod-index=0 - controller-revision-hash=vpc-nat-gw-gw1-98ff96d76 - ovn.kubernetes.io/vpc-nat-gw=true - statefulset.kubernetes.io/pod-name=vpc-nat-gw-gw1-0 -Annotations: cni.projectcalico.org/containerID: 3cf564b6124aae048a097bd9777dd3ac583b293357cb819a85146515e2ba1e52 - cni.projectcalico.org/podIP: 10.52.1.66/32 - cni.projectcalico.org/podIPs: 10.52.1.66/32 - externalvswitch.kube-system.ovn.kubernetes.io/allocated: true - externalvswitch.kube-system.ovn.kubernetes.io/cidr: 10.115.48.0/21 - externalvswitch.kube-system.ovn.kubernetes.io/gateway: 10.115.55.254 - externalvswitch.kube-system.ovn.kubernetes.io/ip_address: 10.115.48.1 - externalvswitch.kube-system.ovn.kubernetes.io/logical_router: commonvpc - externalvswitch.kube-system.ovn.kubernetes.io/logical_switch: externalsubnet - externalvswitch.kube-system.ovn.kubernetes.io/mac_address: fe:01:4a:61:43:6e - externalvswitch.kube-system.ovn.kubernetes.io/pod_nic_type: veth-pair - externalvswitch.kube-system.ovn.kubernetes.io/routed: true - externalvswitch.kube-system.ovn.kubernetes.io/routes: [{"dst":"0.0.0.0/0","gw":"10.115.55.254"}] - internalvswitch.default.ovn.kubernetes.io/allocated: true - internalvswitch.default.ovn.kubernetes.io/cidr: 172.20.10.0/24 - internalvswitch.default.ovn.kubernetes.io/gateway: 172.20.10.1 - internalvswitch.default.ovn.kubernetes.io/ip_address: 172.20.10.254 - internalvswitch.default.ovn.kubernetes.io/logical_router: commonvpc - internalvswitch.default.ovn.kubernetes.io/logical_switch: internalsubnet - internalvswitch.default.ovn.kubernetes.io/mac_address: d6:2d:c1:95:45:15 - internalvswitch.default.ovn.kubernetes.io/pod_nic_type: veth-pair - internalvswitch.default.ovn.kubernetes.io/routed: true - internalvswitch.default.ovn.kubernetes.io/routes: [{"dst":"10.55.0.0/16","gw":"172.20.10.1"},{"dst":"10.115.48.0/21","gw":"172.20.10.1"}] - internalvswitch.default.ovn.kubernetes.io/vpc_cidrs: ["172.20.10.0/24"] - internalvswitch.default.ovn.kubernetes.io/vpc_nat_gw: gw1 - k8s.v1.cni.cncf.io/network-status: - [{ - "name": "k8s-pod-network", - "interface": "eth0", - "ips": [ - "10.52.1.66" - ], - "mac": "be:86:12:cc:e3:30", - "default": true, - "dns": {} - },{ - "name": "default/internalvswitch", - "interface": "net1", - "ips": [ - "172.20.10.254" - ], - "mac": "d6:2d:c1:95:45:15", - "dns": {} - },{ - "name": "kube-system/externalvswitch", - "interface": "net2", - "ips": [ - "10.115.48.1" - ], - "mac": "fe:01:4a:61:43:6e", - "dns": {}, - "gateway": [ - "10.115.55.254" - ] - }] - k8s.v1.cni.cncf.io/networks: default/internalvswitch, kube-system/externalvswitch - ovn.kubernetes.io/ip_address: 172.20.10.254 - ovn.kubernetes.io/logical_switch: internalsubnet - ovn.kubernetes.io/vpc_nat_gw: gw1 - ovn.kubernetes.io/vpc_nat_gw_init: true -Status: Running -IP: 10.52.1.66 -IPs: - IP: 10.52.1.66 -Controlled By: StatefulSet/vpc-nat-gw-gw1 -Containers: - vpc-nat-gw: - Container ID: containerd://efcab898c60ad94ee5d79a598f96d5b0becf65c2ae917d6f02246b227be40eca - Image: docker.io/kubeovn/vpc-nat-gateway:v1.15.0 - Image ID: docker.io/kubeovn/vpc-nat-gateway@sha256:4c090dffaaf5c44593883bfa15a4524e860ac4d7c32555d98284aa8fa042ba6a - Port: - Host Port: - Command: - sleep - infinity - State: Running - Started: Fri, 06 Mar 2026 22:34:01 +0000 - Ready: True - Restart Count: 0 - Environment: - GATEWAY_V4: 10.115.55.254 - GATEWAY_V6: - Mounts: - /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qgb7d (ro) -Conditions: - Type Status - PodReadyToStartContainers True - Initialized True - Ready True - ContainersReady True - PodScheduled True -Volumes: - kube-api-access-qgb7d: - Type: Projected (a volume that contains injected data from multiple sources) - TokenExpirationSeconds: 3607 - ConfigMapName: kube-root-ca.crt - Optional: false - DownwardAPI: true -QoS Class: BestEffort -Node-Selectors: -Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s - node.kubernetes.io/unreachable:NoExecute op=Exists for 300s -Events: + persistentVolumeClaimRetentionPolicy: + whenDeleted: Retain + whenScaled: Retain + podManagementPolicy: OrderedReady + replicas: 1 + revisionHistoryLimit: 10 + selector: + matchLabels: + app: vpc-nat-gw-gw1 + ovn.kubernetes.io/vpc-nat-gw: "true" + serviceName: "" + template: + metadata: + annotations: + k8s.v1.cni.cncf.io/networks: default/vswitchinternal, kube-system/vswitchexternal + ovn.kubernetes.io/vpc_nat_gw: gw1 + vswitchexternal.kube-system.ovn.kubernetes.io/routes: '[{"dst":"0.0.0.0/0","gw":"10.115.55.254"}]' + vswitchinternal.default.ovn.kubernetes.io/ip_address: 172.20.10.1 + vswitchinternal.default.ovn.kubernetes.io/logical_switch: subnetinternal + vswitchinternal.default.ovn.kubernetes.io/routes: '[{"dst":"10.55.0.0/16","gw":"172.20.10.1"}]' + labels: + app: vpc-nat-gw-gw1 + ovn.kubernetes.io/vpc-nat-gw: "true" + spec: + affinity: {} + containers: + - command: + - sleep + - infinity + env: + - name: GATEWAY_V4 + value: 10.115.55.254 + - name: GATEWAY_V6 + image: docker.io/kubeovn/vpc-nat-gateway:v1.16.1 + imagePullPolicy: IfNotPresent + lifecycle: + postStart: + exec: + command: + - sh + - -c + - sysctl -w net.ipv4.ip_forward=1 + name: vpc-nat-gw + resources: {} + securityContext: + allowPrivilegeEscalation: true + privileged: true + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File + dnsPolicy: ClusterFirst + restartPolicy: Always + schedulerName: default-scheduler + securityContext: {} + terminationGracePeriodSeconds: 0 + updateStrategy: + type: RollingUpdate + status: + availableReplicas: 1 + collisionCount: 0 + currentReplicas: 1 + currentRevision: vpc-nat-gw-gw1-5659bc556b + observedGeneration: 1 + readyReplicas: 1 + replicas: 1 + updateRevision: vpc-nat-gw-gw1-5659bc556b + updatedReplicas: 1 + ``` + ``` + $ kubectl describe pod vpc-nat-gw-gw1-0 -n kube-system + Name: vpc-nat-gw-gw1-0 + Namespace: kube-system + Priority: 0 + Service Account: default + Node: hp-46/10.115.14.70 + Start Time: Fri, 05 Jun 2026 22:42:18 +0000 + Labels: app=vpc-nat-gw-gw1 + apps.kubernetes.io/pod-index=0 + controller-revision-hash=vpc-nat-gw-gw1-5659bc556b + ovn.kubernetes.io/vpc-nat-gw=true + statefulset.kubernetes.io/pod-name=vpc-nat-gw-gw1-0 + Annotations: cni.projectcalico.org/containerID: 1b133ea1f17aac64c3d70bde83e1244fe07eb5206a78ec7c76f6099a450e093f + cni.projectcalico.org/podIP: 10.52.1.30/32 + cni.projectcalico.org/podIPs: 10.52.1.30/32 + k8s.v1.cni.cncf.io/network-status: + [{ + "name": "k8s-pod-network", + "interface": "eth0", + "ips": [ + "10.52.1.30" + ], + "mac": "4a:26:76:cd:f3:fa", + "default": true, + "dns": {} + },{ + "name": "default/vswitchinternal", + "interface": "net1", + "ips": [ + "172.20.10.1" + ], + "mac": "22:02:d9:04:e7:bb", + "dns": {} + },{ + "name": "kube-system/vswitchexternal", + "interface": "net2", + "ips": [ + "10.115.48.6" + ], + "mac": "c6:1d:c3:e8:a5:76", + "dns": {}, + "gateway": [ + "10.115.55.254" + ] + }] + k8s.v1.cni.cncf.io/networks: default/vswitchinternal, kube-system/vswitchexternal + ovn.kubernetes.io/vpc_nat_gw: gw1 + ovn.kubernetes.io/vpc_nat_gw_init: true + vswitch2.default.ovn.kubernetes.io/vpc_cidrs: ["172.30.0.0/24"] + vswitchexternal.kube-system.ovn.kubernetes.io/allocated: true + vswitchexternal.kube-system.ovn.kubernetes.io/cidr: 10.115.48.0/21 + vswitchexternal.kube-system.ovn.kubernetes.io/gateway: 10.115.55.254 + vswitchexternal.kube-system.ovn.kubernetes.io/ip_address: 10.115.48.6 + vswitchexternal.kube-system.ovn.kubernetes.io/logical_switch: subnetexternal + vswitchexternal.kube-system.ovn.kubernetes.io/mac_address: c6:1d:c3:e8:a5:76 + vswitchexternal.kube-system.ovn.kubernetes.io/pod_nic_type: veth-pair + vswitchexternal.kube-system.ovn.kubernetes.io/provider_network: pn1 + vswitchexternal.kube-system.ovn.kubernetes.io/routed: true + vswitchexternal.kube-system.ovn.kubernetes.io/routes: [{"dst":"0.0.0.0/0","gw":"10.115.55.254"}] + vswitchexternal.kube-system.ovn.kubernetes.io/vlan_id: 2017 + vswitchinternal.default.ovn.kubernetes.io/allocated: true + vswitchinternal.default.ovn.kubernetes.io/cidr: 172.20.10.0/24 + vswitchinternal.default.ovn.kubernetes.io/gateway: 172.20.10.1 + vswitchinternal.default.ovn.kubernetes.io/ip_address: 172.20.10.1 + vswitchinternal.default.ovn.kubernetes.io/logical_router: commonvpc + vswitchinternal.default.ovn.kubernetes.io/logical_switch: subnetinternal + vswitchinternal.default.ovn.kubernetes.io/mac_address: 22:02:d9:04:e7:bb + vswitchinternal.default.ovn.kubernetes.io/pod_nic_type: veth-pair + vswitchinternal.default.ovn.kubernetes.io/routed: true + vswitchinternal.default.ovn.kubernetes.io/routes: [{"dst":"10.55.0.0/16","gw":"172.20.10.1"}] + vswitchinternal.default.ovn.kubernetes.io/vpc_cidrs: ["172.20.10.0/24"] + Status: Running + IP: 10.52.1.30 + IPs: + IP: 10.52.1.30 + Controlled By: StatefulSet/vpc-nat-gw-gw1 + Containers: + vpc-nat-gw: + Container ID: containerd://4e0d66777f1e56f6402e1106fba2feb9b4a6f0eaa467101a5f7b159ff9981c04 + Image: docker.io/kubeovn/vpc-nat-gateway:v1.16.1 + Image ID: docker.io/kubeovn/vpc-nat-gateway@sha256:59afff3ee0583378ae73ead77f3459745fef300d111f3a5e60ca555b19ca8e7b + Port: + Host Port: + Command: + sleep + infinity + State: Running + Started: Fri, 05 Jun 2026 22:42:29 +0000 + Ready: True + Restart Count: 0 + Environment: + GATEWAY_V4: 10.115.55.254 + GATEWAY_V6: + Mounts: + /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-97xmd (ro) + Conditions: + Type Status + PodReadyToStartContainers True + Initialized True + Ready True + ContainersReady True + PodScheduled True + Volumes: + kube-api-access-97xmd: + Type: Projected (a volume that contains injected data from multiple sources) + TokenExpirationSeconds: 3607 + ConfigMapName: kube-root-ca.crt + Optional: false + DownwardAPI: true + QoS Class: BestEffort + Node-Selectors: + Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s + node.kubernetes.io/unreachable:NoExecute op=Exists for 300s + Events: + ``` + +1. Check the vpc nat gw pod for interfaces (net1 attached to internal network, net2 attached to external network) + + ``` + $ kubectl exec -it vpc-nat-gw-gw1-0 -n kube-system -- /bin/bash + vpc-nat-gw-gw1-0:/kube-ovn# ip addr show + 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 + link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 + inet 127.0.0.1/8 scope host lo + valid_lft forever preferred_lft forever + inet6 ::1/128 scope host proto kernel_lo + valid_lft forever preferred_lft forever + 2: eth0@if9727: mtu 1450 qdisc noqueue state UP group default qlen 1000 + link/ether 4a:26:76:cd:f3:fa brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet 10.52.1.30/32 scope global eth0 + valid_lft forever preferred_lft forever + inet6 fe80::4826:76ff:fecd:f3fa/64 scope link proto kernel_ll + valid_lft forever preferred_lft forever + 9728: net1@if9729: mtu 1450 qdisc noqueue state UP group default + link/ether 22:02:d9:04:e7:bb brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet 172.20.10.1/24 brd 172.20.10.255 scope global net1 + valid_lft forever preferred_lft forever + inet6 fe80::2002:d9ff:fe04:e7bb/64 scope link proto kernel_ll + valid_lft forever preferred_lft forever + 9730: net2@if9731: mtu 1500 qdisc noqueue state UP group default + link/ether c6:1d:c3:e8:a5:76 brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet 10.115.48.6/21 brd 10.115.55.255 scope global net2 + valid_lft forever preferred_lft forever + inet 10.115.55.200/21 scope global secondary net2 + valid_lft forever preferred_lft forever + inet6 fe80::c41d:c3ff:fee8:a576/64 scope link proto kernel_ll + valid_lft forever preferred_lft forever + vpc-nat-gw-gw1-0:/kube-ovn# ip route show + default via 10.115.55.254 dev net2 onlink + 10.55.0.0/16 via 172.20.10.1 dev net1 + 10.115.48.0/21 dev net2 proto kernel scope link src 10.115.48.6 + 169.254.1.1 dev eth0 scope link + 172.20.10.0/24 dev net1 proto kernel scope link src 172.20.10.1 + 172.30.0.0/24 via 172.20.10.1 dev net1 + ``` + +1. Verify provider network bridge (br-pn1) and external subnet attached on the ovs (patch-localnet.subnetexternal-to-br-int) + + ``` + $ kubectl exec -it ovs-ovn-q92zk -n kube-system -- /bin/bash + Defaulted container "openvswitch" out of: openvswitch, hostpath-init (init) + nobody@hp-65:/kube-ovn$ ovs-vsctl show + 54ef5649-9fe6-4944-865b-30a591c95121 + Bridge br-pn1 + Port br-pn1 + Interface br-pn1 + type: internal + Port patch-localnet.subnetexternal-to-br-int + Interface patch-localnet.subnetexternal-to-br-int + type: patch + options: {peer=patch-br-int-to-localnet.subnetexternal} + Port eno50 + trunks: [0, 2017] + Interface eno50 + Bridge br-int + fail_mode: secure + datapath_type: system + Port br-int + Interface br-int + type: internal + Port "6159403e_net1_h" + Interface "6159403e_net1_h" + Port mirror0 + Interface mirror0 + type: internal + Port ovn0 + Interface ovn0 + type: internal + Port patch-br-int-to-localnet.subnetexternal + Interface patch-br-int-to-localnet.subnetexternal + type: patch + options: {peer=patch-localnet.subnetexternal-to-br-int} + Port "7e8b0_37a8eec_h" + Interface "7e8b0_37a8eec_h" + Port "6159403e_net2_h" + Interface "6159403e_net2_h" + ovs_version: "3.5.3" + ``` + +### SNAT for external connectivity from Overlay VMs + +1. Create EIP + + 1. On the Harvester UI, go to **Overlay Networks > NAT & Internet > External IPs**. + + ![](/img/eip.png) + + 1. Click **Create**. + + 1. Specify a unique name for the external IP. + + 1. On the **Basic** tab, configure the following settings: + + - **VPC NAT GW**: Name of the VPC NAT gateway (for example, `gw1`) + + - **External Subnet**: Subnet of the underlay external network (for example, `subnetexternal`) + + - **v4 IP**: Available IP address from the external subnet CIDR range to be used as a public IP for SNAT/DNAT + + 1. Click **Create**. + +1. Create an SNAT Rule + + 1. On the Harvester UI, go to **Overlay Networks > NAT & Internet > Source Rules**. + + ![](/img/snat.png) + + 1. Click **Create**. + + 1. Specify a unique name for the SNAT rule. + + 1. On the **Basic** tab, configure the following settings: + + - **EIP**: External IP (for example, `my-eip`) + + - **Internal CIDR**: CIDR of the internal tenant subnet + + 1. Click **Create**. -``` - -#### Check the vpc nat gw pod for interfaces (net1 attached to internal network, net2 attached to external network) +#### Verification and Example Outputs -``` -kubectl exec -it vpc-nat-gw-gw1-0 -n kube-system -- /bin/bash -vpc-nat-gw-gw1-0:/kube-ovn# ip addr show -1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 - link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 - inet 127.0.0.1/8 scope host lo - valid_lft forever preferred_lft forever - inet6 ::1/128 scope host proto kernel_lo - valid_lft forever preferred_lft forever -2: eth0@if4949: mtu 1450 qdisc noqueue state UP group default qlen 1000 - link/ether be:86:12:cc:e3:30 brd ff:ff:ff:ff:ff:ff link-netnsid 0 - inet 10.52.1.66/32 scope global eth0 - valid_lft forever preferred_lft forever - inet6 fe80::bc86:12ff:fecc:e330/64 scope link proto kernel_ll - valid_lft forever preferred_lft forever -4950: net1@if4951: mtu 1450 qdisc noqueue state UP group default - link/ether d6:2d:c1:95:45:15 brd ff:ff:ff:ff:ff:ff link-netnsid 0 - inet 172.20.10.254/24 brd 172.20.10.255 scope global net1 - valid_lft forever preferred_lft forever - inet6 fe80::d42d:c1ff:fe95:4515/64 scope link proto kernel_ll - valid_lft forever preferred_lft forever -4952: net2@if4953: mtu 1450 qdisc noqueue state UP group default - link/ether fe:01:4a:61:43:6e brd ff:ff:ff:ff:ff:ff link-netnsid 0 - inet 10.115.48.1/21 brd 10.115.55.255 scope global net2 - valid_lft forever preferred_lft forever - inet 10.115.55.200/21 scope global secondary net2 - valid_lft forever preferred_lft forever - inet6 fe80::fc01:4aff:fe61:436e/64 scope link proto kernel_ll - valid_lft forever preferred_lft forever -vpc-nat-gw-gw1-0:/kube-ovn# ip route show -default via 10.115.55.254 dev net2 -10.55.0.0/16 via 172.20.10.1 dev net1 -10.115.48.0/21 dev net2 proto kernel scope link src 10.115.48.1 -169.254.1.1 dev eth0 scope link -172.20.10.0/24 dev net1 proto kernel scope link src 172.20.10.254 - -``` +1. Verify the status of the external IP. -#### Create a provider network with vlan id 2017 and physical interface and a vlan network attached to the provider network. + ``` + $ kubectl get eip + NAME IP MAC NAT NATGWDP READY + my-eip 10.115.55.200 52:1b:4f:1d:14:ce dnat,snat gw1 true -``` -apiVersion: kubeovn.io/v1 -kind: ProviderNetwork -metadata: - name: pn1 -spec: - defaultInterface: eno50 -``` + ``` -``` -apiVersion: kubeovn.io/v1 -kind: Vlan -metadata: - name: vlan2017 -spec: - id: 2017 - provider: pn1 -``` +1. Verify that the SNAT rule was created in the iptables of the VPC NAT gateway pod. -#### Edit subnet subnetexternal to use vlan as vlan2017 (this will attach this subnet to the provider network underlay) + Example: + + ``` + vpc-nat-gw-gw1-0:/kube-ovn# iptables-legacy-save -t nat + # Generated by iptables-save v1.8.11 on Sat Mar 7 00:28:34 2026 + *nat + :PREROUTING ACCEPT [2949:373475] + :INPUT ACCEPT [107:31097] + :OUTPUT ACCEPT [0:0] + :POSTROUTING ACCEPT [1:60] + :DNAT_FILTER - [0:0] + :EXCLUSIVE_DNAT - [0:0] + :EXCLUSIVE_SNAT - [0:0] + :SHARED_DNAT - [0:0] + :SHARED_SNAT - [0:0] + :SNAT_FILTER - [0:0] + -A PREROUTING -j DNAT_FILTER + -A POSTROUTING -j SNAT_FILTER + -A SHARED_SNAT -s 172.20.10.0/24 -o net2 -j SNAT --to-source 10.115.55.200 --random-fully + -A SNAT_FILTER -j EXCLUSIVE_SNAT + -A SNAT_FILTER -j SHARED_SNAT + COMMIT + ``` + +### Create a Virtual Machine + +[Create a virtual machine](../vm/create-vm.md#how-to-create-a-vm) and attach it to the `vswitchinternal` overlay network. + +Ping from VM (inside guest os) to any IP address external to the cluster, such as 8.8.8.8 must be successful +The traffic from VM reaches net1 of vpc nat gw pod and with route installed egress out of net2 and hits the iptable rule for SNAT and translates 172.20.10.0/24 subnet ip to 10.115.55.200 for external connectivity. -``` -apiVersion: kubeovn.io/v1 -kind: Subnet -metadata: - name: subnetexternal -spec: - cidrBlock: 10.115.48.0/21 - default: false - enableLb: true - excludeIps: - - 10.115.55.254 - gateway: 10.115.55.254 - gatewayNode: "" - natOutgoing: true - private: false - protocol: IPv4 - provider: vswitchexternal.kube-system.ovn - vlan: vlan2017 - vpc: custom-vpc +![](/img/kubeovnSNATFlow.png) -``` -#### Verify provider network bridge (br-pn1) and external subnet attached on the ovs (patch-localnet.externalsubnet-to-br-int) -``` -kubectl exec -it ovs-ovn-q92zk -n kube-system -- /bin/bash -Defaulted container "openvswitch" out of: openvswitch, hostpath-init (init) -nobody@hp-65:/kube-ovn$ ovs-vsctl show -54ef5649-9fe6-4944-865b-30a591c95121 - Bridge br-pn1 - Port br-pn1 - Interface br-pn1 - type: internal - Port patch-localnet.externalsubnet-to-br-int - Interface patch-localnet.externalsubnet-to-br-int - type: patch - options: {peer=patch-br-int-to-localnet.externalsubnet} - Port eno50 - trunks: [0, 2017] - Interface eno50 - Bridge br-int - fail_mode: secure - datapath_type: system - Port br-int - Interface br-int - type: internal - Port "6159403e_net1_h" - Interface "6159403e_net1_h" - Port mirror0 - Interface mirror0 - type: internal - Port ovn0 - Interface ovn0 - type: internal - Port patch-br-int-to-localnet.externalsubnet - Interface patch-br-int-to-localnet.externalsubnet - type: patch - options: {peer=patch-localnet.externalsubnet-to-br-int} - Port "7e8b0_37a8eec_h" - Interface "7e8b0_37a8eec_h" - Port "6159403e_net2_h" - Interface "6159403e_net2_h" - ovs_version: "3.5.3" -``` +### DNAT for inbound access (external to overlay Virtual Machines) -#### SNAT for external connectivity from Overlay VMs +1. Create a DNAT Rule -##### Create EIP and SNAT resource + 1. On the Harvester UI, go to **Overlay Networks > NAT & Internet > Destination Rules**. -``` -apiVersion: kubeovn.io/v1 -kind: IptablesEIP -metadata: - name: my-eip -spec: - natGwDp: gw1 # Name of your VpcNatGateway - externalSubnet: subnetexternal - v4ip: 10.115.55.200 #random public ip from external subnet -``` + ![](/img/dnat.png) -``` -apiVersion: kubeovn.io/v1 -kind: IptablesSnatRule -metadata: - name: my-snat -spec: - eip: my-eip - internalCIDR: 172.20.10.0/24 #internal subnet CIDR -``` + 1. Click **Create**. -##### Verify EIP status -``` -kubectl get eip -NAME IP MAC NAT NATGWDP READY -my-eip 10.115.55.200 52:1b:4f:1d:14:ce dnat,snat gw1 true + 1. Specify a unique name for the DNAT rule. -``` + 1. On the **Basic** tab, configure the following settings: -##### Verify SNAT filter iptable rule created inside the VPC NAT gateway pod + - **EIP**: External IP assigned to the VPC NAT gateway (for example, `my-eip`) -``` -vpc-nat-gw-gw1-0:/kube-ovn# iptables-legacy-save -t nat -# Generated by iptables-save v1.8.11 on Sat Mar 7 00:28:34 2026 -*nat -:PREROUTING ACCEPT [2949:373475] -:INPUT ACCEPT [107:31097] -:OUTPUT ACCEPT [0:0] -:POSTROUTING ACCEPT [1:60] -:DNAT_FILTER - [0:0] -:EXCLUSIVE_DNAT - [0:0] -:EXCLUSIVE_SNAT - [0:0] -:SHARED_DNAT - [0:0] -:SHARED_SNAT - [0:0] -:SNAT_FILTER - [0:0] --A PREROUTING -j DNAT_FILTER --A POSTROUTING -j SNAT_FILTER --A SHARED_SNAT -s 172.20.10.0/24 -o net2 -j SNAT --to-source 10.115.55.200 --random-fully --A SNAT_FILTER -j EXCLUSIVE_SNAT --A SNAT_FILTER -j SHARED_SNAT -COMMIT + - **External Port**: Port on which incoming traffic is received from external clients -``` + - **Internal IP**: Destination IP address to which incoming traffic is forwarded (the IP address of the target overlay virtual machine) -##### Create a VM and attach it to the vswitchinternal overlay network and add the following default route in the guest os. - (172.20.10.254 is the ip addresses on net1 interface on vpc nat gw pod) -``` -ip route add default via 172.20.10.254 dev enp1s0 -``` - -Ping from VM (inside guest os) to 8.8.8.8 must be successful -The traffic from VM reaches net1 of vpc nat gw pod and with route installed egress out of net2 and hits the iptable rule for SNAT and translates 172.20.10.0/24 subnet ip to 10.115.55.200 for external connectivity. + - **Internal Port**: Destination port on the internal host to which incoming traffic is translated and forwarded -![](/img/kubeovnSNATFlow.png) + - **Protocol** `TCP` or `UDP` + 1. Click **Create**. -#### DNAT for inbound access (external to overlay VMs) +#### Verification and Example Outputs -##### Use the DNAT resource and use same EIP as SNAT +1. Verify that the DNAT iptables rule exists inside the VPC NAT gateway pod. -``` -kind: IptablesDnatRule -apiVersion: kubeovn.io/v1 -metadata: - name: dnat01 -spec: - eip: my-eip - externalPort: '8888' - internalIp: 172.20.10.2 #IP of the overlay VM - internalPort: '80' - protocol: tcp + Example: + ``` + vpc-nat-gw-gw1-0:/kube-ovn# iptables-legacy -t nat -L -n -v 2>/dev/null | grep -E "DNAT" + 3051 392K DNAT_FILTER all -- * * 0.0.0.0/0 0.0.0.0/0 + Chain DNAT_FILTER (1 references) + 3051 392K EXCLUSIVE_DNAT all -- * * 0.0.0.0/0 0.0.0.0/0 + 3051 392K SHARED_DNAT all -- * * 0.0.0.0/0 0.0.0.0/0 + Chain EXCLUSIVE_DNAT (1 references) + Chain SHARED_DNAT (1 references) + 1 60 DNAT tcp -- * * 0.0.0.0/0 10.115.55.200 tcp dpt:8888 to:172.20.10.2:80 -``` -##### Check the DNAT Filter inside the vpc NAT GW pod + ``` -``` -vpc-nat-gw-gw1-0:/kube-ovn# iptables-legacy -t nat -L -n -v 2>/dev/null | grep -E "DNAT" - 3051 392K DNAT_FILTER all -- * * 0.0.0.0/0 0.0.0.0/0 -Chain DNAT_FILTER (1 references) - 3051 392K EXCLUSIVE_DNAT all -- * * 0.0.0.0/0 0.0.0.0/0 - 3051 392K SHARED_DNAT all -- * * 0.0.0.0/0 0.0.0.0/0 -Chain EXCLUSIVE_DNAT (1 references) -Chain SHARED_DNAT (1 references) - 1 60 DNAT tcp -- * * 0.0.0.0/0 10.115.55.200 tcp dpt:8888 to:172.20.10.2:80 +### Create a Virtual Machine -``` +[Create a virtual machine](../vm/create-vm.md#how-to-create-a-vm) attached to the overlay network on `subnetinternal`, then install NGINX on the virtual machine to verify inbound connectivity. -##### Create a VM attached to overlay network on subnetinternal and Install nginx on VM to check inbound access ``` -sudo apt update -sudo apt install nginx -y -sudo systemctl status nginx +$ sudo apt update +$ sudo apt install nginx -y +$ sudo systemctl status nginx test nginx locally on the VM -curl http://127.0.0.1 +$ curl http://127.0.0.1 Now curl -k http://10.115.55.200:8888 from external must be successful ``` ![](/img/kubeovnDNATFlow.png) - - diff --git a/docs/rancher/guest-cluster-kubeovn-network.md b/docs/rancher/guest-cluster-kubeovn-network.md new file mode 100644 index 0000000000..7cb0c2db17 --- /dev/null +++ b/docs/rancher/guest-cluster-kubeovn-network.md @@ -0,0 +1,86 @@ +--- +sidebar_position: 11 +sidebar_label: Guest Cluster using KubeOVN Networks +title: "Guest Cluster using KubeOVN Networks" +keywords: + - Harvester + - Rancher + - KubeOVN +--- + + + + + +## Introduction + +Rancher provides centralized management for Kubernetes clusters across multiple environments. When integrated with Harvester, Rancher can provision guest Kubernetes clusters directly on virtual machine infrastructure managed by Harvester. + +Using the Harvester Node Driver, Rancher automatically creates virtual machines that serve as control plane and worker nodes. This simplifies cluster deployment and lifecycle management by providing a single interface for infrastructure and Kubernetes operations. + +While guest clusters can use traditional virtual machine networking, they can also leverage Kube-OVN overlay or underlay networking to provide a flexible and Kubernetes-native networking model for workloads. + +## Guest Cluster Deployment on Underlay Network + +Deploying guest clusters on pure underlay networks is functionally similar to using traditional virtual machine VLAN networks, but with the added advantages of Kube-OVN features such as network policies and tenant-isolated VPCs. + +For information about setting up the overlay infrastructure, see [Create an underlay network](../networking/kubeovn-pureunderlay.md#underlay-configuration). + +## Guest Cluster Deployment on Overlay Network + +Deploying guest clusters on overlay networks requires configuring a VPC NAT gateway to provide inbound and outbound connectivity to the virtual machines through DNAT and SNAT. + +The key architectural features are as follows: + +- Harvester provides the virtual machine infrastructure where the guest cluster control plane and worker nodes run. +- Rancher is responsible for provisioning and managing the lifecycle of the guest Kubernetes cluster. +- Kube-OVN provides the network connectivity for the VMs hosting the guest nodes through its overlay network between Harvester hosts. +- The guest cluster CNI (for example, Calico) manages pod networking, while pod traffic is transported over the underlying Kube-OVN-provided VM network. + +Kube-OVN overlay networking enables guest clusters to manage pod networking independently of the underlying Harvester infrastructure network. This provides the following operational benefits: + +- Simplified cluster deployment +- Reduced dependence on physical network configuration +- Dynamic cluster scaling without additional VLAN planning +- Consistent networking model across environments + +For information about setting up the overlay infrastructure, see [Create a VPC NAT Gateway](../networking/kubeovn-vpcnatgateway.md#kubeovn-as-secondary-cni). + +## Guest Cluster Provisioning + +### Importing Harvester Clusters into Rancher + +Harvester clusters are imported through Rancher's **Virtualization Management** interface. + +The following occur after a Harvester cluster is imported: + +- Rancher recognizes the cluster as a Harvester provider. +- Virtual machine resources become available through Rancher. +- Harvester can be selected as an infrastructure provider when creating downstream clusters. + +For detailed instructions, see [Importing Harvester Cluster](./virtualization-management.md#importing-harvester-cluster). + +### Provisioning a Guest Cluster + +Rancher uses the Harvester Node Driver to automatically provision the required virtual machines and bootstrap Kubernetes on them. + +For detailed instructions, see [Harvester Node Driver](./node/rke2-cluster.md). + +:::info important + +In the **Networks** section of the cluster creation screen, you must specify the correct underlay or overlay network. + +Also, For VMs provisioned as downstream cluster nodes, ensure external connectivity is available by enabling DHCP and dns_server is set to a valid DNS server IP address (for example, `8.8.8.8`) and natOutgoing=true on the subnet. This allows the VMs to obtain a default route and perform DNS resolution. + +::: + +![](/img/v1.9/rancher/gc-overlayvmnetwork.png) + +![](/img/v1.9/rancher/gc-overlayvm.png) + + +:::tip + +Since the guest cluster VM nodes over Kube-OVN overlay networks use private IP addresses, a route to the guest cluster VM node subnet must be configured on the Rancher VM with the Kube-OVN EIP (configured during the VPC NAT gateway EIP setup) as the next hop. This provides network reachability from the Rancher VM to the guest cluster VM nodes and enables SSH access through the Rancher UI for debugging purposes. + +::: diff --git a/static/img/dnat.png b/static/img/dnat.png new file mode 100644 index 0000000000..ab5982938a Binary files /dev/null and b/static/img/dnat.png differ diff --git a/static/img/eip.png b/static/img/eip.png new file mode 100644 index 0000000000..f4d2e2b17b Binary files /dev/null and b/static/img/eip.png differ diff --git a/static/img/snat.png b/static/img/snat.png new file mode 100644 index 0000000000..1a31c859ed Binary files /dev/null and b/static/img/snat.png differ diff --git a/static/img/subnetdhcp.png b/static/img/subnetdhcp.png new file mode 100644 index 0000000000..51a5cc7ec7 Binary files /dev/null and b/static/img/subnetdhcp.png differ diff --git a/static/img/subnetinternal.png b/static/img/subnetinternal.png new file mode 100644 index 0000000000..f83e9b2c2d Binary files /dev/null and b/static/img/subnetinternal.png differ diff --git a/static/img/v1.9/rancher/gc-overlayvm.png b/static/img/v1.9/rancher/gc-overlayvm.png new file mode 100644 index 0000000000..a854e3a1cd Binary files /dev/null and b/static/img/v1.9/rancher/gc-overlayvm.png differ diff --git a/static/img/v1.9/rancher/gc-overlayvmnetwork.png b/static/img/v1.9/rancher/gc-overlayvmnetwork.png new file mode 100644 index 0000000000..47f8c70988 Binary files /dev/null and b/static/img/v1.9/rancher/gc-overlayvmnetwork.png differ diff --git a/static/img/vpc-nat-config.png b/static/img/vpc-nat-config.png new file mode 100644 index 0000000000..b59c70490b Binary files /dev/null and b/static/img/vpc-nat-config.png differ diff --git a/static/img/vpc-nat-config2.png b/static/img/vpc-nat-config2.png new file mode 100644 index 0000000000..ca49d8dcc4 Binary files /dev/null and b/static/img/vpc-nat-config2.png differ