Intersite Network Overview

The ISN can be a generic Layer 3 infrastructure that interconnects different sites in the Cisco ACI MultiSite solution.

Since the ISN allows establishment of site-to-site VXLAN tunnels, it must support plain IP routing functionality. Therefore, you can use different solutions, from a simple single router device (or two for redundancy) as an ISN, to a complex network infrastructure spanning the world. In addition, starting from Cisco ACI Release 3.2(1), a back-to-back topology between spine switches in two sites is supported.

The spine interfaces are connected to the ISN devices through point-to-point routed subinterfaces with a fixed VLAN 4, which implies the need to define and support Layer 3 subinterfaces on both the spine switches and the directly connected ISN devices. It is hence critical to select ISN routers that allow the defining of multiple subinterfaces on the same device using the same VLAN tag 4 and still functioning as separate point-to-point Layer 3 links.

In addition, the ISN between sites must support these specific functionalities:

Increased maximum transmission unit (MTU) support to handle the VXLAN encapsulated traffic.
Open Shortest Path First (OSPF) support between spine switches and ISN routers.
QoS considerations for consistent QoS policy deployment across sites.

However, in Cisco ACI MultiSite deployments, the ISN does not need to support multicast to facilitate BUM traffic flows across sites. Instead, ACI uses headend replication for the flood traffic where the copy of the packet happens on the destination site.

Cisco Nexus EX platform or newer, since they must perform name-space translation function at line rate to avoid performance issues during intersite communication.

Increased MTU Support

The ISN must support increased MTU on its links to allow site-to-site VXLAN traffic, since the VXLAN data-plane traffic adds 50 bytes of overhead (54 bytes if the IEEE 802.1Q header of the original frame is preserved).

So, you have to configure the MTU in the ISN to the higher value of these two requirements, while following the generic recommendation to add 100 bytes above that value:

MTU of MP-BGP EVPN control-plane communication between spine nodes in different sites.
Maximum data-plane MTU of the frames that are generated by the endpoints that are connected to the fabric.

For example, if the endpoints support jumbo frames (9000 bytes) or if you use the default settings on spine nodes that generate 9000-byte packets for exchanging endpoint routing information, you should configure the ISN to support an MTU size of at least 9100 bytes.

OSPF Support

The ISN uses OSPF while establishing peering with the spine nodes deployed in each site. Therefore, the OSPF settings under the OSPF policy in the Cisco ACI domains should match the ISN side. Even though OSPF peering is required between the spine switches and the ISN devices, you are not limited to use OSPF across the entire ISN infrastructure. It can be a generic Layer 3 infrastructure interconnecting the sites (for example, an MPLS network or internet) that can use a separate routing protocol inside that Layer 3 network, as long as mutual redistribution is used so it advertises OSPF routes that are learned from spine nodes in each site.

QoS Considerations

When you deploy Cisco ACI MultiSite, you should ensure that traffic across sites is consistently prioritized (as it happens intrasite). Without any QoS policies configured at a site, the default DSCP value of the outer IP address of the VXLAN packet in the ISN is set to 0 (zero). Thus, you should configure a QoS DSCP marking policy on the spine nodes (CoS to DSCP) in each site to help ensure proper QoS treatment in the ISN.

The CoS-to-DSCP mappings are configured using the Cisco APIC in each site through Tenant > infra > Policies > Protocol > DSCP class-cos translation policy for L3 traffic, which modifies the default behavior.

Comment

You are will be the first.

Please login here to comment.

EMAIL SUPPORT

LOCATION

Intersite Network Overview