EMAIL SUPPORT

dclessons@dclessons.com

LOCATION

US

ACI Fabric Discovery

ACI Fabric discovery  ( Preparing the Fabric Infrastructure )

This topic covers the discovery process for an ACI fabric in detail.

As soon as the ACI hardware installation is done, all Spines and Leafs are connected to each other and powered up and  once basic configuration of APIC is completed, Fabric discovery process starts working.

In the discovery process, ACI uses Intra-Fabric Messaging (IFM) process in which APIC and nodes exchange heartbeat massage. The process used by the APIC to push policy to the fabric leaf nodes is called as IFM Process.

ACI Fabric discovery completes in three stages.

  • In First stage the leaf node directly connected to the APIC is discovered.
  • In the second stage of discovery brings in the spines connected to that initial leaf where APIC was connection.
  • In third stage processes the discovery of the other leaf nodes and APICs in the cluster.

The Complete steps of IFM (Intra-Fabric Messaging) are as follows. After this all process is completed the fabric is ready for Production configuration.

  • Link Layer Discovery Protocol (LLDP) Neighbor Discovery
  • Tunnel End Point (TEP) IP address assignment to the node via DHCP 
  • Node software upgraded if necessary
  • ISIS adjacency mode
  • Certification Validation 
  • Start of DME Process on switches.
  • Tunnel Setup (iVxlan)
  • Policy Element IFM Setup

Node status may fluctuate between several states during the fabric registration process. The states are shown in the Fabric Node Vector table. The APIC CLI command to show the Fabric Node Vector table acidiag fnvread .

Following are the States and descriptions:

• Unknown – It states that Node discovered but no Node ID policy configured
• Undiscovered – It states that Node ID configured but not yet discovered
• Discovering – It states that Node discovered but IP not yet assigned
• Unsupported – It states that Node is not a supported model
• Disabled – when Node has been decommissioned, it will show Disabled
• Inactive – if you have No IP connectivity
• Active – When Node is active

ACI uses inter-fabric messaging (IFM) packets to communicate between the different nodes or between leaf and spine. These IFM packets are typically TCP packets, which are secured by 1024-bit SSL encryption, and the keys used for encryption are stored on secure storage. These keys are signed by Cisco Manufacturing Certificate Authority (CMCA). Any issues with IFM process can prevent fabric nodes communicating and from joining the fabric.

Below command shows the health of the registration process.

apic1# netstat -ant | grep :12
tcp 0 0 10.0.0.1:12151 0.0.0.0:* LISTEN
tcp 0 0 10.0.0.1:12215 0.0.0.0:* LISTEN
tcp 0 0 10.0.0.1:12471 0.0.0.0:* LISTEN
tcp 0 0 10.0.0.1:12279 0.0.0.0:* LISTEN
tcp 0 0 10.0.0.1:12567 10.0.248.29:49187 ESTABLISHED
tcp 0 0 10.0.0.1:12343 10.0.248.30:45965 ESTABLISHED
tcp 0 0 10.0.0.1:12343 10.0.248.31:47784 ESTABLISHED
tcp 0 0 10.0.0.1:12343 10.0.248.29:49942 ESTABLISHED
tcp 0 0 10.0.0.1:12343 10.0.248.30:42946 ESTABLISHED
tcp 0 0 10.0.0.1:50820 10.0.248.31:12439 ESTABLISHED
apic1# openssl s_client -state -connect 10.0.0.1:12151
CONNECTED(00000003)
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
depth=1 O = Cisco Systems, CN = Cisco Manufacturing CA
verify error:num=19:self signed certificate in certificate chain
verify return:0
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server certificate request A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client certificate A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL3 alert read:fatal:handshake failure
SSL_connect:failed in SSLv3 read server session ticket A
139682023904936:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1300:SSL alert number 40
139682023904936:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
0 s:/CN=serialNumber=PID:APIC-SERVER-L1 SN:TEP-1-1, CN=TEP-1-1
i:/O=Cisco Systems/CN=Cisco Manufacturing CA
1 s:/O=Cisco Systems/CN=Cisco Manufacturing CA
i:/O=Cisco Systems/CN=Cisco Manufacturing CA
---
Server certificate
-----BEGIN CERTIFICATE----- -----END CERTIFICATE-----
subject=/CN=serialNumber=PID:APIC-SERVER-L1 SN:TEP-1-1, CN=TEP-1-1
issuer=/O=Cisco Systems/CN=Cisco Manufacturing CA
---
No client certificate CA names sent
---
SSL handshake has read 2171 bytes and written 210 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : DHE-RSA-AES256-GCM-SHA384
Session-ID:
Session-ID-ctx:
Master-Key: 419BF5E19D0A02AA0D40BDF380E8E959A4F27371A87EFAD1B
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Compression: 1 (zlib compression)
Start Time: 1481059783
Timeout : 300 (sec)
Verify return code: 19 (self signed certificate in certificate chain)
---
apic1#
 

GENERAL FAQ

Cisco ACI Fabric Discovery is the automated process that integrates all fabric nodes into the ACI environment after the hardware is installed and powered on. Once the basic APIC configuration is completed, the discovery process begins and prepares the fabric for production deployment. This process uses secure communication between the APIC controllers, leaf switches, and spine switches.

Cisco ACI Fabric Discovery is performed in three phases. The first step is to identify the leaf switch connected to the APIC. Then the spine switches linked to the leaf are found. Then, third, all the leaf switches as well as APIC controllers within the cluster are identified. After this, the fabric is ready for configuration and deployment.

Intra-Fabric Messaging (IFM) is the primary communication mechanism used during Cisco ACI Fabric Discovery. It enables secure communication between the APIC and fabric nodes, including heartbeat exchanges and policy distribution. TEP IP addresses are assigned via DHCP during discovery, and IFM ensures proper policy synchronization and node registration. If IFM fails, nodes may not successfully join the fabric.

During Cisco ACI Fabric Discovery, a node may transition through several states while registering with the fabric. These states are visible in the Fabric Node Vector (FNV) table and help engineers understand the registration status of each device. The common node states include:

• Unknown – The node is discovered, but no Node ID policy has been configured.

• Undiscovered – The Node ID is configured, but the node has not yet been discovered.

• Discovering – The node is discovered, but the TEP IP address has not yet been assigned.

• Unsupported – The node model is not supported by the ACI fabric.

• Disabled – The node has been decommissioned.

• Inactive – The node does not have proper IP connectivity.

• Active – The node is fully registered and operational within the fabric.

Communication during Cisco ACI Fabric Discovery is secured using Intra-Fabric Messaging (IFM). IFM packets use encrypted TLS communication to ensure secure data exchange between the APIC controllers and fabric nodes. The encryption keys are stored securely and are signed by the Cisco Manufacturing Certificate Authority (CMCA). This security mechanism ensures that only trusted Cisco ACI devices can successfully join and communicate within the fabric, preventing unauthorized devices from participating in the network.

This page provides the foundational understanding required for Cisco ACI Fabric Discovery. However, the complete process, troubleshooting techniques, packet flow analysis, and real-world scenarios are covered in detail inside the course. You can access the full training through the available membership plans: Monthly ($100), Half-Yearly ($200), and Annual ($350).

Comment

  • PR

    can you add video as well.

    • Prathap
    • Aug 10, 2020

  • SA

    I learned how ACI fabric discovery in very easy language , superb course content

    • Sam
    • Dec 09, 2021

  • TH

    Hi, Thank you for this valuable course. I have a question regarding the Fabric verification. Actually the output of "show lldp nei" on leaf1 shows only APIC1 while the leaf1 has dual-homed connection to APIC-1 and -2. Please advise!!

    • TAOUFIK HAYAT
    • Jun 07, 2022

LEAVE A COMMENT

Please login here to comment.

TABLE OF CONTENTS

RECENT POSTS

Membership Plan

Monthly

$100/Monthly
Half Yearly

$200/6 Months
Half Yearly

$350/Year