In this post I’m going to put a full config of a working vPC here without explaining basics of the vPC, which is widely available on the Internet.
To start, take a look at this topology, where we have two Nexus 9500 switches and two Nexus 5500 switches.

vPC Back-to-Back

Two Nexus 9k switches will be configured as vPC domain 9 and Nexus 5ks as vPC domain 5. There are more than 50 downstream access switches (c3850) in the actual topology, but for simplicity I just put one of them here. vPC domain 9 will be set as STP root bridge and also acts as a default gateway for some of VLANS.
Here are configurations:

Nexus 9k1 configuration:

feature vrrp
feature udld
feature interface-vlan
feature lacp
feature vpc
feature lldp
!
vrf context management
spanning-tree vlan 1-3967 priority 24576
!
vpc domain 9
  peer-switch
  peer-keepalive destination 192.168.1.2 source 192.168.1.1 interval 400 timeout 3
!
interface Vlan60
  description MANAGEMENT_VLAN
  no shutdown
  ip address 10.200.100.91/24
!
interface Vlan401
  no shutdown
  ip address 172.16.100.250/24
  vrrp 7
    priority 120
    address 172.16.100.1 
    no shutdown

interface Vlan405
  no shutdown
  ip address 172.28.100.250/24
  vrrp 3
    priority 120
    address 172.28.100.1 
    no shutdown

interface port-channel1
  description N9K_PEER_KEEPALIVE
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link

interface port-channel19
  description TO_FW
  switchport mode trunk
  vpc 19

interface port-channel29
  description TO_FW
  switchport mode trunk
  vpc 29

interface port-channel33
  description SERVER_1
  switchport access vlan 304
  vpc 33

interface port-channel34
  description SERVER_2
  switchport access vlan 306
  vpc 34

interface port-channel35
  description SERVER_3
  switchport mode trunk
  vpc 35

interface port-channel59
  description TRUNK_TO_N5K
  switchport mode trunk
  vpc 59
!
interface Ethernet1/33
  switchport access vlan 304
  channel-group 33 mode active

interface Ethernet1/34
  switchport access vlan 306
  channel-group 34 mode active

interface Ethernet1/35
  switchport mode trunk
  channel-group 35

interface Ethernet1/47
  switchport mode trunk
  channel-group 1 mode active

interface Ethernet1/49
  switchport mode trunk
  channel-group 19 mode active

interface Ethernet2/33
  switchport access vlan 304
  channel-group 33 mode active

interface Ethernet2/34
  switchport access vlan 306
  channel-group 34 mode active

interface Ethernet2/35
  switchport mode trunk
  channel-group 35

interface Ethernet2/45
  switchport mode trunk
  channel-group 59 mode active

interface Ethernet2/47
  switchport mode trunk
  channel-group 1 mode active

interface Ethernet2/49
  switchport mode trunk
  channel-group 29 mode active

interface mgmt0
  vrf member management
  ip address 192.168.1.1/24

Nexus 9k2 configuration:

feature vrrp
feature udld
feature interface-vlan
feature lacp
feature vpc
feature lldp
!
spanning-tree vlan 1-3967 priority 24576
!
vrf context management
vpc domain 9
  peer-switch
  peer-keepalive destination 192.168.1.1 source 192.168.1.2 interval 400 timeout 3
!
interface Vlan60
  description MANAGEMENT_VLAN
  no shutdown
  ip address 10.200.100.92/24
!
interface Vlan401
  no shutdown
  ip address 172.16.100.251/24
  vrrp 7
    address 172.16.100.1 
    no shutdown

interface Vlan405
  no shutdown
  ip address 172.28.100.251/24
  vrrp 3
    address 172.28.100.1 
    no shutdown

interface port-channel1
  description N9K_PEER_KEEPALIVE
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link

interface port-channel19
  description TO_FW
  switchport mode trunk
  vpc 19

interface port-channel29
  description TO_FW
  switchport mode trunk
  vpc 29

interface port-channel33
  description SERVER_1
  switchport access vlan 304
  vpc 33

interface port-channel34
  description SERVER_2
  switchport access vlan 306
  vpc 34

interface port-channel35
  description SERVER_3
  switchport mode trunk
  vpc 35

interface port-channel59
  description TRUNK_TO_N5K
  switchport mode trunk
  vpc 59

interface Ethernet1/33
  switchport access vlan 304
  channel-group 33 mode active

interface Ethernet1/34
  switchport access vlan 306
  channel-group 34 mode active

interface Ethernet1/35
  switchport mode trunk
  channel-group 35

interface Ethernet1/45
  switchport mode trunk
  channel-group 59 mode active

interface Ethernet1/47
  switchport mode trunk
  channel-group 1 mode passive

interface Ethernet1/49
  switchport mode trunk
  channel-group 29 mode active

interface Ethernet2/33
  switchport access vlan 304
  channel-group 33 mode active

interface Ethernet2/34
  switchport access vlan 306
  channel-group 34 mode active

interface Ethernet2/35
  switchport mode trunk
  channel-group 35

interface Ethernet2/45
  switchport mode trunk
  channel-group 59 mode active

interface Ethernet2/47
  switchport mode trunk
  channel-group 1 mode passive

interface Ethernet2/49
  switchport mode trunk
  channel-group 19 mode active

interface mgmt0
  vrf member management
  ip address 192.168.1.2/24

And the configuration for N5k switches; First N5k:

feature interface-vlan
feature lacp
feature vpc
feature lldp
!
vrf context management
  ip route 0.0.0.0/0 10.200.100.1
!
vpc domain 5
  peer-switch
  peer-keepalive destination 192.168.2.2 source 192.168.2.1 interval 1000 timeout 3

interface port-channel1
  description VPC_PEER_LINK
  switchport mode trunk
  spanning-tree port type network
  speed 10000
  vpc peer-link

interface port-channel59
  description TO_N9K
  switchport mode trunk
  speed 10000
  vpc 59

interface port-channel100
  description TO_3850_DOWNSTREAM_SW
  switchport mode trunk
  speed 10000
  vpc 100

interface Ethernet1/1
  switchport mode trunk
  channel-group 100 mode active

interface Ethernet1/15
  switchport mode trunk
  channel-group 1 mode passive

interface Ethernet1/16
  switchport mode trunk
  channel-group 59 mode active

interface Ethernet1/31
  switchport mode trunk
  channel-group 1 mode passive

interface Ethernet1/32
  switchport mode trunk
  channel-group 59 mode active

interface mgmt0
  vrf member management
  no ip redirects
  ip address 192.168.2.1/24
  ip address 10.200.100.13/24 secondary

Continue with second N5k:

feature interface-vlan
feature lacp
feature vpc
feature lldp
!
vrf context management
  ip route 0.0.0.0/0 10.200.100.1
!
vpc domain 5
  peer-switch
  peer-keepalive destination 192.168.2.1 source 192.168.2.2 interval 1000 timeout 3

interface port-channel1
  description VPC_PEER_LINK
  switchport mode trunk
  spanning-tree port type network
  speed 10000
  vpc peer-link

interface port-channel59
  description TO_N9K
  switchport mode trunk
  speed 10000
  vpc 59

interface port-channel100
  description TO_3850_DOWNSTREAM_SW
  switchport mode trunk
  speed 10000
  vpc 100

interface Ethernet1/1
  switchport mode trunk
  channel-group 100 mode active

interface Ethernet1/15
  switchport mode trunk
  channel-group 1 mode active

interface Ethernet1/16
  switchport mode trunk
  channel-group 59 mode active

interface Ethernet1/31
  switchport mode trunk
  channel-group 1 mode active

interface Ethernet1/32
  switchport mode trunk
  channel-group 59 mode active

interface mgmt0
  vrf member management
  no ip redirects
  ip address 192.168.2.2/24
  ip address 10.200.100.14/24 secondary

In this example, I’d assigned a local and unreachable IP address to mgmt port on Nexus 5k switches, because we expected to use another management vlan to remotely connect to these switches. But we didn’t managed to get a L3 daughter card for 5k switches, hence it got impossible to assign any IP address to any vlan interface on those switches. As an immediate workaround to this issue without breaking vPC or pushing switches to restart/restablish vPC, I assigned a secondary IP address to mgmt interface which was remotely reachable throughout the network and added a static route to a Management VRF” toward the default gateway (10.200.100.1).

vrf context management
  ip route 0.0.0.0/0 10.200.100.1

As you might have noticed, Nexus 9k switches have built-in L3 routing license and thus we could assign IP address to their management interface vlan60.
“peer-switch” command needs to be written only on the switches which are going to act as root L2 switches on the network. Nexus 9k devices will act as root bridges and thus we wrote that command only on the Nexus 9000s after lowering the STP Bridge Priority on those devices and left the default priority value intact on rest of the network.

If there were some other devices like load balancers on the network, which is probably the case in most networks and you configure your vPC device to act as default gateway for some or all of your vlans, then you must consider some important things which I’m going to briefly discuss here.
In this example, vlan401 and vlan405 on Nexus 9k devices are configured to be a default gateway for the respective vlans. For this we added VRRP on those SVIs on both 9ks, so the clients on those vlans need to use the VRRP virtual IP as their default gateway.

Vlan401 Actual IP addresses: 172.16.0.250,172.16.0.251
Vlan401 VRRP virtual IP address: 172.16.0.1
Vlan405 Actual IP addresses: 172.28.1.250,172.28.1.251
Vlan405 VRRP virtual IP address: 172.28.1.1

When a L3 packet received at the 9k devices from any of the vlans above, it is going to be route toward the actual destination and it’s L2 header being re-written as normal. But the issue with devices like load balancers is that those devices sometimes use interface vlan401 or interface vlan405 actual MAC address instead of the virtual VRRP address which is the same on both 9k devices. I don’t want to explain the mechanism of the issue here, because this post is going to be short 🙂 But to eliminate this kind of issues we need to change something either on Nexus devices or on load balancers.
On nexus 9ks, we can use “peer-gateway” command to solve the issue. Assuming you know the issue, take a look at the output of some commands on Nexus 9k devices “before” adding “peer-gateway”. For example, let’s extract MAC address of interface vlan401 on both 9ks:

N9K-1(config)# sh inter vlan 401 | inc address
  Hardware is EtherSVI, address is  80e0.1dcf.ed7f

N9K-2(config)# sh inter vlan 401 | inc address
  Hardware is EtherSVI, address is  80e0.1dcf.cc7f

I said that interface vlan401 on both switches acts as default gateway for clients on vlan 401 and thus will take part in routing packets off the vlan 401. Clients inside vlan 401 will “NOT” use these MAC addresses but instead will use VRRP virtual MAC address. Also 9ks will “NOT” use the actual MAC addresses of the interface vlan401 while routing packets sourced from vlan 401 toward any external network and use VRRP virtual MAC, as expected. But returning packets from any device like load balancers, will be sent toward “actual” MAC address of the interface vlan 401 on Nexus 9ks which is origin of the issue.
Let’s take a look at MAC address table contents on Nexus 9k devices, using any given vlan:
On 9k-1:

N9K-1# sh mac address-t vlan 309
Legend: 
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
*  309     0000.0000.0100   dynamic  0         F      F    Po29
+  309     0000.0000.0110   dynamic  0         F      F    Po19
*  309     001c.7f81.0901   dynamic  0         F      F    Po29
*  309     8c60.4f95.1b21   dynamic  0         F      F    Eth1/37
*  309     9c57.ad9a.a5f2   dynamic  0         F      F    Po20
+  309     8c60.4f95.1c41   dynamic  0         F      F    vPC Peer-Link
*  309     80e0.1dcf.cc7f   static   -         F      F    vPC Peer-Link(R)
G  309     80e0.1dcf.ed7f   static   -         F      F    sup-eth1(R)

On 9k-2:

N9K-2# sh mac address-t vlan 309
Legend: 
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
+  309     0000.0000.0100   dynamic  0         F      F    Po29
*  309     0000.0000.0110   dynamic  0         F      F    Po19
+  309     001c.7f81.0901   dynamic  0         F      F    Po29
*  309     8c60.4f95.1c41   dynamic  0         F      F    Eth1/37
+  309     9c57.ad9a.a5f2   dynamic  0         F      F    Po20
+  309     8c60.4f95.1b21   dynamic  0         F      F    vPC Peer-Link
G  309     80e0.1dcf.cc7f   static   -         F      F    sup-eth1(R)
*  309     80e0.1dcf.ed7f   static   -         F      F    vPC Peer-Link(R)

Find our “actual” MAC addresses of the interface vlan401 on the table. You can clearly see those MAC addresses in the output, but notice a tiny difference among them; Nexus 9k-1 has a “G” at the beginning of the line containing it’s “own” MAC address, but there is no “G” for the MAC address of the interface vlan401 of Nexus 9k-2. Reviewing the legends, you can see that “G” stands for “Gateway MAC”.
Each Nexus 9k acts as Gateway MAC for it’s owned MAC address but can reach the other MAC through the peer-link. With this situation, if any packets receives at the 9k-1 which actually destined to 9k-2 MAC address, needs to be forwarded to 9k-2 through peer-link, which will be dropped and make us a real trouble.
Adding “peer-gateway” command will make both nexus devices to forward any received packets toward either of these MAC addresses, without sending them through peer-link. In other words, if 9k-1 gets any packet destined to 80e0.1dcf.cc7f (which owned by 9k-2), will forward if to it’s actual L3 destination (or determined next-hop) by it’s own and “Will NOT” send it to 9k-2 via the peer-link.
These are outputs “after” adding “peer-gateway” command on 9ks:

On 9k-1:

vpc domain 9
  peer-gateway
!
!
N9K-1(config)# sh mac address-t vlan 309
Legend: 
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
*  309     0000.0000.0100   dynamic  0         F      F    Po29
+  309     0000.0000.0110   dynamic  0         F      F    Po19
*  309     001c.7f81.0901   dynamic  0         F      F    Po29
*  309     8c60.4f95.1b21   dynamic  0         F      F    Eth1/37
*  309     9c57.ad9a.a5f2   dynamic  0         F      F    Po20
+  309     8c60.4f95.1c41   dynamic  0         F      F    vPC Peer-Link
G  309     80e0.1dcf.cc7f   static   -         F      F    vPC Peer-Link(R)
G  309     80e0.1dcf.ed7f   static   -         F      F    sup-eth1(R)

On 9k-2:

vpc domain 9
  peer-gateway
!
!
N9K-2(config)# sh mac address-t vlan 309
Legend: 
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
+  309     0000.0000.0100   dynamic  0         F      F    Po29
*  309     0000.0000.0110   dynamic  0         F      F    Po19
+  309     001c.7f81.0901   dynamic  0         F      F    Po29
*  309     8c60.4f95.1c41   dynamic  0         F      F    Eth1/37
+  309     9c57.ad9a.a5f2   dynamic  0         F      F    Po20
+  309     8c60.4f95.1b21   dynamic  0         F      F    vPC Peer-Link
G  309     80e0.1dcf.cc7f   static   -         F      F    sup-eth1(R)
G  309     80e0.1dcf.ed7f   static   -         F      F    vPC Peer-Link(R)

Now you can see that “G” is present for both MAC addresses on both switches.
If you decide not to use this command on Nexus switches, you can eliminate this issue on load balancers and other devices with the same behavior, for example, by disabling “Auto-Last Hop” on F5 LTM device, or Packet Reflect feature on EMC SAN.

Cisco Nexus; vPC Back-to-Back
Tagged on: