This example demonstrates how to integrate OpenPERouter with Multus to have a pod’s secondary interface connected to a layer 2 overlay, using SRv6 as the underlay.

Overview #

A layer 2 VNI is created, exposing a layer 2 domain on the host. On each node, a pod is created with a macvlan interface attached to that domain via a Linux bridge.

Example Setup #

The full example can be found in the project repository and can be deployed by running:

make docker-build demo-metallb-l3vpn-l2vni

The example configures both an L2 VNI and an L3VPN. The L2 VNI belongs to the L3VPN’s routing domain. Pods are running on two separate nodes and connected via the overlay. Additionally, the pods are able to reach the broader L3 domain via the L2 gateways.

Pod-to-Pod Connectivity #

The two pods are on the same L2 subnet. When a pod tries to reach the other, no routing is involved and the L2 traffic gets encapsulated in the corresponding L2 VNI.

Pod-to-External L3 Connectivity #

When a pod needs to reach an external host on the L3 domain, the traffic leaves via the pod’s secondary interface (added by the NetworkAttachmentDefinition) and is routed in the VRF of the OpenPERouter pod that the pod is connected to. The packet is subsequently forwarded via the SRv6 overlay.

Configuration #

OpenPERouter Configuration #

One L3VPN corresponding to the routing domain, one L2 VNI with the same VRF as the L3VPN:

apiVersion: network.openperouter.io/v1alpha1
kind: L3VPN
metadata:
  name: red
  namespace: openperouter-system
spec:
  hostSession:
    asn: 64514
    hostASN: 64515
    localCIDRs:
      - 192.169.10.0/24
  vrf: red
  rdAssignedNumber: 100
  exportRTs:
  - "64514:100"
  importRTs:
  - "64520:100"
---
apiVersion: network.openperouter.io/v1alpha1
kind: L2VNI
metadata:
  name: layer2
  namespace: openperouter-system
spec:
  vni: 110
  routingDomain:
    type: L3VPN
    l3vpn:
      name: red
  hostMaster:
    type: LinuxBridge
    linuxBridge:
      lifecycle: Managed
  gatewayIPs: ["192.170.1.1/24"]

Configuration Notes:

• gatewayIPs field: Allows each pod to have the same default gateway and be able to send traffic to the outer L3 domain
• hostMaster.linuxBridge.lifecycle: Managed instructs OpenPERouter to create a bridge local to the node that can be used to access the L2 domain

Underlay Configuration #

The underlay for this example includes both VXLAN and SRv6 tunnel endpoints, along with IS-IS and SRv6 configuration:

apiVersion: network.openperouter.io/v1alpha1
kind: Underlay
metadata:
  name: underlay
  namespace: openperouter-system
spec:
  tunnelEndpoint:
    cidrs:
    - 100.65.0.0/24
    - 2001:db8:1234:5678::/64
  asn: 64514
  neighbors:
  - address: 2001:db8:1234::1
    asn: 64520
    properties:
    - type: ebgpMultiHop
  - asn: 64512
    address: 192.168.11.2
  interfaces:
    - type: NetworkDevice
      networkDevice:
        interfaceName: toswitch1
  routerIDCIDR: 10.0.0.0/24
  isis:
    baseNet: "49.0001.0002.0003.0004.00"
    level: 1
  srv6:
    locator:
      basePrefix: "fd00:0:32::/48"
      format: "usid-f3216"

Network Attachment Definition #

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-conf
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "macvlan",
      "master": "br-hs-110",
      "mode": "bridge",
      "ipam": {
         "type": "static"
      }
    }'

A network attachment definition creating a macvlan interface with the bridge created by the OpenPERouter as master.

Workload Configuration #

Two pods, each running on a different node, with a Multus secondary network corresponding to the network attachment definition we just created:

apiVersion: v1
kind: Pod
metadata:
  name: second
  annotations:
    k8s.v1.cni.cncf.io/networks: '[{
      "name": "macvlan-conf",
      "namespace": "default",
      "ips": ["192.170.1.4/24"]
      }]'
spec:
  nodeSelector:
    kubernetes.io/hostname: pe-kind-worker
  containers:
    - name: agnhost
      image: k8s.gcr.io/e2e-test-images/agnhost:2.45
      command: ["/bin/sh", "-c", "ip r del default dev eth0 && /agnhost netexec --http-port=8090"]
      ports:
      - containerPort: 8090
        name: http
      securityContext:
        capabilities:
          add: ["NET_ADMIN"]

Validation #

Pod L2-to-L2 Connectivity #

We can check that the first pod is able to curl the second, and the client IP is the one corresponding to the secondary interface:

kubectl exec -it first -- curl 192.170.1.4:8090/clientip

Expected output:

192.170.1.3:44564

The traffic comes through the veth corresponding to the pod’s secondary interface and is sent out directly via the VXLAN interface and encapsulated.

Pod-to-External L3 Connectivity #

With 192.168.20.2 being the IP of the host on the red network:

kubectl exec -it first -- curl 192.170.20.2:8090/clientip

Expected output:

192.170.1.3:37144

The pod is able to reach a host on the layer 3 network and the IP is preserved.

The packet comes from the veth interface towards the bridge associated with VNI 110 (the default gateway), then is routed inside the layer 3 domain via the SRv6 overlay.