ACI Multi-Site Architecture

ACI Multi-Site

Cisco ACI Multi-Site Architecture is used to connect multiple intersite APIC cluster domain with their associated Pods. This ACI Multi-Site design, interconnects separate regions, which is deployed as either single Pod or Multipod.

The reasons for Multi-Site design so to provide complete isolation of network and Tenant change-domain level across separate Cisco ACI networks.

Cisco ACI Multi-Site Architecture

Cisco Multi-Site Architecture interconnects separate APCI cluster domain, each representing different region, but all are part of same ACI Multi-Site domain.

Below figure describes overall architecture of Cisco ACI Multisite

The cisco ACI Multi-Site Architecture has following functional components, discussed below:

Cisco Multi-Site Orchestrator

This is the intersite policy Manager, which provides single pane of glass for below functions

• Centralized Policy Manager, so that all intersite policy can be pushed to different APIC domains.
• Monitor Health state and score of all interconnected sites
• Single Pane of Management

Intersite Control Plane

In Multi-Site control plane, End points reachability information is exchanged across sites over MP-BGP EVPN protocol.

With this MAC and IP address information is exchanged between sites to provide communication between them.

Spines of different fabric sites establish a MP-BGP EVPN session between them to develop control plane. 

Intersite Date Plane

in Multi-Site Site to Site VXLAN tunnel is used to provide Layer 2 and Layer 3 communication between endpoints.

Shadow EPG

When contracts between two different EPG( One at each site ) is defined at MSO , then it is pushed to different APIC domains, then in this case , a specific copies of each EPG ( called Shadow EPG ) are automatically created in each APIC domain. Using this, all policies are defined at one central level, and then can be locally instantiated in each site. Doing this all the security policies are properly enforced even each EPG is locally define and is not stretched across sites.

Let’s understand this with an example,

RED EPG is locally present in APIC Domain 1 whereas Green EPG is locally present in APIC Domain 2. However their shadow EPG are also present in those DC like, RED EPG (Shadow EPG ) is also present in APIC Domain 2 , whereas Green EPG ( Shadow EPG ) is also present in APIC Domain 1.

As we now know that, both APIC domain are independent to each other, so both RED and Green EPG will have different VNID and Class ID assigned to them. Due to which, translation of those values is required, in spines, because spines receives the data from remote site before injecting it to local site.

Below figure shows how, Name –space translation happens on receiving spines.

Whenever a policy is defined at MSO, saying RED EPG must communicate to Green EPG, MSO will instruct the APIC controllers to program proper translation rules in the local spines due to which policies are correctly applied on local leaf before sending traffic to destination node.

To achieve Name-space translation at line rate, it is recommended to use Cisco Nexus NX (newer Generation) of Spine switches in Multi-Site Deployment. The first generation of Spine Switch can co-exist with New Spine switch model , provided the new Generation Spine switch will be have connectivity to external IP network used for Intersite communication.

Cisco Multi-Site Orchestrator

Cisco MSO is used in ACI Multi-Site Environment, for provisioning, Health Monitoring, and managing different APIC domains.

Below are some Cisco ACI Multi-Site main functions.

• It creates and manages Multi-Site User or admins based on RBAC Model.
• It is used to add, delete, and modify Cisco ACI Sites
• It provides Health Monitor dashboard to monitor health, faults, and logs of ACI intersite policies, who are part of Cisco Multi-Site domain.
• Provision day 0 infrastructure , so that Spines at each site will peer and connect with each other , which further helps to achieve MP-BGP EVPN control plane setup , so that End Host reachability information ca be exchanged between sites.
• It help in creating new tenants and deploy them in all ACI connected Sites or subset of them.
• Define policy templates, which can be further associated or pushed to specific set of fabric.
• It helps in importing tenant policies from already deployed ACI fabric (brownfield) and deploy it to Greenfield deployment.

In well-defined MSO design, it has three nodes (Physical or Virtual), clustered together in active/active approach. The MSO cluster nodes communicates to each other via OOB Management Network of APIC clusters deployed at different sites.