Multi-Pod Provisioning & Traffic Flow

Cisco ACI Multi-Pod deployment is automated in a lot of ways to offer easy provisioning with less user intervention.

Initially, the first pod (also known as ‘seed’ pod) should be already set up using the traditional ACI fabric bring-up procedure, and the ‘seed’ pod and the second pod should be physically connected to the IPN devices. Before Cisco ACI Multi-Pod provisioning process can start, you should set up the Cisco APIC and the IPN using the following these steps:

1. Configure access policies: Configure access policies for all the interfaces on the spine switches used to connect to the IPN. Define these policies as spine access policies. Use these policies to associate an Attached Entity Profile (AEP) for a Layer 3 domain that uses VLAN 4 (as a requirement) for the encapsulation for the subinterface. Define these subinterfaces in the same way as normal leaf access ports. The subinterfaces are used by the infrastructure L3Out interface that you define.

2. Define the Multi-Pod environment: For the Cisco ACI Multi-Pod setup, you should define the TEP address for the spine switches facing each other across the IPN. This IP address is used as anycast shared address across all spine switches in a pod. You should also define the Layer 3 interfaces between the spine interfaces and the IPN.

3. Configure the IPN. Configure the IPN devices with IP addresses on the interfaces facing the spine switches, and enable the OSPF routing protocol, MTU support, DHCP-relay, and PIM Bidir. The IPN devices create OSPF adjacencies with the spine switches and exchange the routes of the underlying IS-IS network part of VRF overlay-1. The configuration of the IPN defines the DHCP relay, which is critical for learning adjacencies because the DHCP frames forwarded across the IPN will reach the primary APIC in the first pod to get a DHCP address assignment from the TEP pool. Without DHCP relay in the IPN, zero-touch provisioning will not occur for Cisco ACI nodes deployed in the second pod.

4. Establish the interface access policies for the second pod: If you do not establish the access policies for the second pod, then the second pod cannot complete the process of joining the fabric. You can add the device to the fabric, but it does not complete the discovery process. Thus, the spine switch in the second pod has no way to talk to the original pod, since the OSPF adjacency cannot be established due to VLAN 4 requirement, and the OSPF interface profile and the external Layer 3 definition do not exist. You can reuse the access policies of the first pod as long as the spine interfaces you are using on both pods are the same. Hence, if the spine interfaces in both pods are the same and the ports in all the switches also are the same, then the only action you need to take is to add the spine switches to the switch profile that you define.

In the Cisco APIC user interface, you can use a wizard to add a pod to the Multi-Pod deployments, which helps you provision the necessary L3Outs on the spine switches connected to the IPN, MTU on all spine-to-IPN interfaces, OSPF configuration towards the IPN, Anycast TEP IP address, and so on. You can invoke this wizard using Fabric > Inventory > Quick Start > Add Pod and choose Add Pod from the work plane, as shown in this figure:

The Cisco ACI Multi-Pod discovery process follows this sequence:

Inter-Pods MP-BGP Control Plane

The COOP database information in each pod is shared via MP-BGP EVPN through IPN so that each pod knows which endpoint is learned in which pod. MP-BGP EVPN runs directly between spine switches in each pod. IPN devices will not participate in this BGP session, but it just provides TEP reachability to establish BGP sessions between each spine switch.