Bridge Domain Configuration in ACI

Bridge Domain Configuration 

The basic bridge domain configuration options that should be considered when configuring a bridge domain behavior are as follows:

• Whether to use hardware proxy or flooding mode for Layer 2 Unknown Unicast packets

• Whether to enable or disable Address Resolution Protocol (ARP) flooding

• Whether to enable or disable unicast routing

• Whether or not to define one or more subnets under the bridge domain

Hardware proxy for Layer 2 unknown unicast traffic is the default option. If the destination MAC is not in the ingress leaf endpoint table, the packet is sent to the spine proxy. This forwarding behavior uses the COOP database on spine switches to forward unknown unicast traffic to the destination leaf without relying on flood-and-learn behavior, as long as the MAC address is known to the COOP database on spine switch.

With Layer 2 unknown unicast flooding, that is, if hardware proxy is not selected, the leaf endpoint table and spine COOP database are still populated with the MAC-to-VTEP information. However, the forwarding does not use the COOP database on spine switches. Layer 2 unknown unicast packets are flooded within the bridge domain.

If ARP flooding is enabled, ARP traffic will be flooded within the bridge domain as per regular ARP handling in traditional networks. If this option is disabled, the ingress leaf uses unicast to send the ARP traffic to the destination leaf or to spine-proxy (detailed scenarios are explained later in this section). Note that these options apply only if unicast routing is enabled on the bridge domain. If unicast routing is disabled, ARP traffic is always flooded within the bridge domain.

The Layer 3 Configurations tab allows you to configure the following basic parameters:

• Unicast Routing: If this setting is enabled and a subnet address is configured, the fabric provides the default gateway function within the bridge domain and routes the traffic. Enabling unicast routing also instructs the endpoint table on the leaf switches to learn the endpoint IP-to-TEP mapping for this bridge domain. The IP learning is not dependent upon having a subnet configured under the bridge domain.

• Subnet Address: This option configures the SVI IP addresses (default gateway) for the bridge domain. The options for a subnet under a bridge domain are as follows:

  1. Private to VRF: The subnet applies only within its tenant.

  2. Advertised externally: The subnet can be exported to a routed connection.

  3. Shared between VRFs: The subnet can be shared with and exported to multiple VRFs in the same tenant or across tenants as part of a shared service.

Unicast routing is enabled by default, and is required when you configure a default gateway for a bridge domain inside Cisco ACI fabric. If you configure the default gateway outside the fabric (for example, on a firewall), you should disable unicast routing and enable ARP flooding.

Unicast routing should be disabled to avoid unnecessary IP learning that may cause unexpected IP forwarding.

ACI Layer 2 Forwarding

Next, you will see how a Layer 2 or Layer 3 packet flows through fabric, how ARP is resolved, and how different bridge domain modes influence the data flow.

If the destination MAC address is known to an ingress leaf, the packet is forwarded to the local port if the endpoint is on the local leaf.

If the destination MAC address is known to an ingress leaf and the endpoint is not on the local leaf, the packet is forwarded directly to the remote leaf.

Look at an example where the traffic flows between two endpoints in the same EPG in the same Layer 2 bridge domain; MAC addresses of both endpoints are known. EP1 is known as local to leaf switch 1 and EP2 is known as remote endpoint to this switch. EP1 is known as remote endpoint to Leaf 2 and EP2 is known as local to Leaf 2. When traffic is going from EP1 to EP2, Leaf 1 will encapsulate the packet with the source address physical tunnel endpoint (PTEP) of Leaf 1 and destination address PTEP of Leaf 2. Leaf 2 will decapsulate the packet and send it out to the correct port to reach EP2. When traffic is going from EP2 to EP1, it is very similar. Leaf 2 encapsulates the packet with the source address of Leaf 2 PTEP and the destination address of Leaf 1.

If the destination MAC address is not known to an ingress leaf, then there are two options.

If the bridge domain is in Layer 2 Unknown Unicast Flood mode, the packet is flooded within the bridge domain. For example, Leaf 1 does not know where the MAC address of EP2 resides, so it will flood the packet to all endpoints in the bridge domain. When a packet is flooded within a BD, packets encapsulated in a multicast IP called Group IP Outer (GIPo) instead of TEP of leaf nodes. Each BD is assigned one GIPo internally for this purpose.

If a bridge domain is in Layer 2 Unknown Unicast hardware-proxy mode, the packet is forwarded to the spine proxy. Then again, there are two possible outcomes.

Leaf 1 will encapsulate the packet with an anycast MAC address of spine as a destination for spine-proxy. If COOP database on spine has information on MAC address of EP2, the spine forwards the packet to a remote leaf.If MAC address of an endpoint is unknown to spine COOP database (for example, silent host), the spine drops the packet.

ARP Fowarding in ACI 

When Unicast Routing is disabled in a bridge domain, regardless of ARP Flooding mode, ARP Request with broadcast MAC will be always flooded.

Look at the ARP resolution in Layer 3 mode. Unicast routing in a bridge domain is enabled and the subnet is defined under the bridge domain. At this point, an interesting question is “Where (on which leaf) is the pervasive gateway (Switch Virtual Interface [SVI]) configured?” SVI is configured on all leaf switches that have at least one endpoint that is a member of the EPG and bridge domain where a subnet is defined. So in case of ARP request for pervasive gateway of a subnet defined under the bridge domain, the ARP reply will come from that first leaf to which an endpoint is attached. In the case of vPC, one of the two leaf switches will send the ARP reply.

There are three possible scenarios of ARP to an endpoint in a bridge domain with unicast routing enabled.