NewValidation: CSCws92111

[cody-skidmore]

(use upvote 👍 for attentions)

Validation Type

[ ] - Fault

[ ] - Config

[X] - Bug

[ ] - Other

What needs to be validated

During an upgrade to 16.0.8f or later, primarily seen when upgrading to 16.1.4h, the spine in a multi-pod setup may take a powercycle discovery path (as seen with an SSD firmware upgrade). This powercycle discovery path leads to BGP global EVI being set to zero and L2RIB Sharding status under COOP to not have MPOD-CONFIGURED flag set. As a result, the spine will have a disconnect between BGP and COOP for endpoint exhcange.

Why it needs to be validated

Traffic loss across pods until the spine in this condition is clean reloaded

Additional context

Only applicable to mpod setups

[slater0013]

just went through this bug.... 👍

[wilsonbc93]

Proposed detection logic:

1. Multipod is enabled (at least two fabricSetupP MOs)
2. Target version is exactly 6.1(4h). No other versions are affected.
3. Spine is modular (N9500)
4. For each modular spine, check the eqptFlash MO for the current SSD revision. Compare the current SSD firmware version to expected firmware verison on the target ACI release. If they do not match, then an SSD upgrade is expected which is the trigger for the defect.

[asraf-khan]

@wilsonbc93 Currently i am working on exact steps for detection logic.

will add detection logic once done.

[asraf-khan]

1. check if Multipod is enabled (at least two fabricSetupP MOs) present.
2. Target version is exactly 6.1(4h). No other versions are affected.
3. Only applicable to modular Spine, get from fabricnode.
4. Admin access will check "bootscript" file missing & "bootstrap.xml" is present in /bootflash/ dir from admin SSH access on spine sup.

Exact steps as below ,

1. if setup is enabled with Multi-pod where count of fabricSetupP MO's >= 2

   icurl('class', 'fabricSetupP.json?query-target=self&rsp-subtree-include=count')
   

2. if tversion == 6.1(4h)

3. In fabricNode class ====> find role = spine & model = N9K-C9408 , N9K-C9504, N9K-C9508, N9K-C9516.

reference doc: https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/6x/release-notes/cisco-aci-nx-os-release-notes-1621.html#Supportedhardware

4. find if bootscript is present in bootflash else workaround is to delete bootstrap.xml file

    ls -l bootflash/ | grep boots
   

[jeestr4d]

created PR #366 , however getting error:

 ERROR tests/checks/mpod_spine_coop_sync_check/test_mpod_spine_coop_sync_check.py - FileNotFoundError: [Errno 2] No such file or directory: '/__w/ACI-Pre-Upgrade-Validation-Script/ACI-Pre-Upgrade-Validation-Script/tests/checks/mpod_spine_coop_sync_check/fabricNode-Neg.json'

[jeestr4d]

corrected file naming.