Multi-Site ISN Design

Multi-Site ISN Design

ISN is used to connect various APIC domains over Layer 3 Infrastructure. ISN requires IP routing in order to establish site-to-site VXLAN tunnel.

Spine Interface are connected to ISN devices, via point-to-point sub-interface, where Spine Interface would be 802.1q enabled and VLAN 4 is allowed on it.

Proper MTU configuration is also necessary on ISN network, which is based on following requirement.

If Endpoints is generating or configured to support jumbo frames, (9000 bytes) then ISN should be configured with minimum MTU of 9100 bytes.

If Endpoints are configured to support 1500 Bytes then then ISN should be configured with minimum MTU of 1600 bytes. But for MP-BGP control plane update, by default Spine node generates 9000 bytes packet for exchanging information , so in this case ISN must configured to Support 9000 bytes else control plane information across sites will be suppressed.

ISN & QOS deployment

Let’s understand how QOS works inside fabric. For that let’s understand the QOS class inside ACI fabric given in below table.

As per above table, below are some classification done, let’s understand this one by one.

• Six User-configurable classes of service for user data traffic received on leaf nodes from externally connected device like endpoints, routers, service nodes, etc.). By default all traffic received from those device is assigned to default level 3 class.
• Four reserved class of service used for traffic between APIC controller nodes, Control plane traffic generated by ACI fabric nodes (Leaf and Spine), SPAN and Traceroute traffic.

When a traffic flow inside ACI fabric, the traffic can easily be distinguished by QOS class based on values assigned to COS and DEI bits in the 802.1Q header of VXLAN encapsulated traffic.

But when this traffic is sent for intersite for communication between fabrics, this caused an issue.  Inside ISN, it is not possible that every packet will hold that 802.1Q header or that QOS values will be preserved.

Inorder to solve this issue, a specific mapping table is configured on each APIC mapping domain, to consistently map each QOS class to specific DSCP value. Doing so the mapping DSCP value is set on outer IP header of VXLAN encapsulated traffic and is then injected in to ISN.

End-to-End consistent QOS behavior across ACI sites.

Doing this, there are other two benefits, discussed as follows:

It allow admin to properly associate traffic received on remote ACI fabric,   to its proper QOS class based on specific DSCP values, carried in the packet . This will keep consistent QOS assignment across different ACI fabric domains.