In the 2nd part of the article I want to show inter-vxlan routing as well as connecting vxlan network to external L3 network.
Companies don’t have only a DC network; instead DC is just part of their network. So it is most probable that external clients want to connect to the DC network and all of applications and DB usually hosted by DC network.
Let’s take a look at our topology.

Cisco VxLAN Topology

If you had experiences with VRF, you would be familiar with route leakage between VRFs and what we need to do here is following the same procedure. Route leakage is done using Route Targets. By playing with RT values of VRFs we can affect routes being exported from or imported to VRFs.
Because we make switches decide RD and RT values automatically, we need to extract these values to be able to play with them. To do this, on N1 we can take a look at BGP EVPN table:

n1# sh bgp l2 evpn 99.1.1.2 vrf A
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 1.1.1.1:32769    (L2VNI 2)
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.5600.0013]:[32]:[99.1.1.2]/272, version 48
Paths: (1 available, best #1)
Flags: (0x000102) on xmit-list, is not in l2rib/evpn

  Advertised path-id 1
  Path type: local, path is valid, is best path
  AS-Path: NONE, path locally originated
    1.1.1.1 (metric 0) from 0.0.0.0 (1.1.1.1)
      Origin IGP, MED not set, localpref 100, weight 32768
      Received label 2 4
      Extcommunity: RT:2:2 RT:2:4 ENCAP:8 Router MAC:0050.5600.0008

  Path-id 1 advertised to peers:
    2.2.2.2

The RT values were listed as BGP Extcommunities: “RT:2:2 RT:2:4”
The 99.1.1.2 address belongs to VRF A and because RD and RT values are created per VRF, all of the networks inside a single VRF will have the same RD and RT values. So we can assume that the values above are for all of routes inside VRF A.
For VRF B we need to do the same:

n1# sh bgp l2 evpn 99.1.1.18 vrf B
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 1.1.1.1:32770    (L2VNI 3)
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.568f.4975]:[32]:[99.1.1.18]/272, version 1711
Paths: (1 available, best #1)
Flags: (0x000102) on xmit-list, is not in l2rib/evpn

  Advertised path-id 1
  Path type: local, path is valid, is best path
  AS-Path: NONE, path locally originated
    1.1.1.1 (metric 0) from 0.0.0.0 (1.1.1.1)
      Origin IGP, MED not set, localpref 100, weight 32768
      Received label 3 5
      Extcommunity: RT:2:3 RT:2:5 ENCAP:8 Router MAC:0050.5600.0008

  Path-id 1 advertised to peers:
    2.2.2.2 

Based on the output, RT values for VRF B are “RT:2:3 RT:2:5”.
The second RT values come from additional VLAN/VxLAN we’ve created. These additional VLANs/VxLANs are used as intermediate L3 structure with the main role of transporting packets between different VxLAN networks. The relative configuration regarding these two L3EVPN in my example looks like as follows:

vlan 4
  name customer_A_L3_routing_vxlan_vlan
  vn-segment 4
vlan 5
  name customer_B_L3_routing_vxlan_vlan
  vn-segment 5
!
interface Vlan4
  no shutdown
  vrf member A
  ip forward

interface Vlan5
  no shutdown
  vrf member B
  ip forward

Because these intermediate vlans only used as transport mediums, the vlan interfaces don’t need to have IP addresses assigned to them; Instead we use “ip forward”. Also we need to add these on NVE interface to tell switch that those VNIs are L3EVPN:

interface nve1
  member vni 4 associate-vrf
  member vni 5 associate-vrf

Returning back to the point, we should modify RT values on VRFs to make inter VRF routing possible. As we have obtained current RT values of our two VRFs, the rest will be simple:

vrf context A
  vni 4
  rd auto
  address-family ipv4 unicast
    route-target import 2:3 evpn
    route-target import 2:5 evpn
    route-target both auto
    route-target both auto evpn
vrf context B
  vni 5
  rd auto
  address-family ipv4 unicast
    route-target import 2:2 evpn
    route-target import 2:4 evpn
    route-target both auto
    route-target both auto evpn
!
!
evpn
  vni 2 l2
    rd auto
    route-target import auto
    route-target import 2:3
    route-target import 2:5
    route-target export auto
  vni 3 l2
    rd auto
    route-target import auto
    route-target import 2:2
    route-target import 2:4
    route-target export auto

At this point you may need to reset BGP connections (soft or hard) between Nexus devices. If everything has been configured correctly, we should have L3 connectivity between VRFs. To verify it is enough to use ping on PCs.

Intra-VxLAN connectivity test:

csr-1#ping 99.1.1.17
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 99.1.1.17, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 5/5/6 ms

Inter-VxLAN connectivity test:

csr-1#ping 99.1.1.4 
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 99.1.1.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 5/5/7 ms

Excellent. The second part of L3 connectivity is configuring access to external network 100.1.1.x. Nexus 1 has an interface inside VRF A which is connected to external network (in this case, Router 2). It has gotten some external EIGRP routes from R2:

N1# show run
interface Ethernet1/1
  no switchport
  vrf member A
  ip address 192.168.20.1/24
  ip router eigrp 1
  no shutdown
!
router eigrp 1
  address-family ipv4 unicast

Routing table on Nexus 1:

n1# sh ip route vrf A
IP Route Table for VRF "A"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%' in via output denotes VRF 

99.1.1.0/29, ubest/mbest: 1/0, attached
    *via 99.1.1.1, Vlan2, [0/0], 3d09h, direct
99.1.1.1/32, ubest/mbest: 1/0, attached
    *via 99.1.1.1, Vlan2, [0/0], 3d09h, local
99.1.1.2/32, ubest/mbest: 1/0, attached
    *via 99.1.1.2, Vlan2, [190/0], 3d08h, hmm
99.1.1.4/32, ubest/mbest: 1/0
    *via 3.3.3.3%default, [200/0], 2d08h, bgp-2, internal, tag 2 (evpn) segid: 4 tunnelid: 0x3030303 encap: VXLAN
 
99.1.1.17/32, ubest/mbest: 1/0
    *via 3.3.3.3%default, [200/0], 2d08h, bgp-2, internal, tag 2 (evpn) segid: 5 tunnelid: 0x3030303 encap: VXLAN
 
100.1.1.1/32, ubest/mbest: 1/0
    *via 192.168.20.2, Eth1/1, [90/130816], 13:04:44, eigrp-1, internal
100.1.1.2/32, ubest/mbest: 1/0
    *via 192.168.20.2, Eth1/1, [90/130816], 13:04:44, eigrp-1, internal
100.1.1.3/32, ubest/mbest: 1/0
    *via 192.168.20.2, Eth1/1, [90/130816], 13:04:44, eigrp-1, internal
150.1.1.1/32, ubest/mbest: 1/0
    *via 192.168.20.2, Eth1/1, [90/130816], 13:04:44, eigrp-1, internal
150.1.1.2/32, ubest/mbest: 1/0
    *via 192.168.20.2, Eth1/1, [90/130816], 13:04:44, eigrp-1, internal
192.168.20.0/24, ubest/mbest: 1/0, attached
    *via 192.168.20.1, Eth1/1, [0/0], 13:06:19, direct
192.168.20.1/32, ubest/mbest: 1/0, attached
    *via 192.168.20.1, Eth1/1, [0/0], 13:06:19, local

As seen, N1 has EIGRP routes toward some 100.1.1.x and 150.1.1.x networks inside its VRF A routing table and we should redistribute those into VxLAN EVPN.

ip access-list TACL
  10 permit ip any any
!
n1# sh route-map
route-map TMAP-100, permit, sequence 10 
  Match clauses:
    ip address (access-lists): TACL 
  Set clauses:
!
router bgp 2
  vrf A
    address-family ipv4 unicast
      redistribute eigrp 1 route-map TMAP-100

Redistribution is done into VRF A, so only those clients in that VRF will have access to external networks. Before verifying connectivity on PCs, let’s take a look at routing table on remote VTEP, Nexus 3:

n3# sh ip route vrf A
IP Route Table for VRF "A"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%' in via output denotes VRF 

99.1.1.0/29, ubest/mbest: 1/0, attached
    *via 99.1.1.1, Vlan2, [0/0], 2d08h, direct
99.1.1.1/32, ubest/mbest: 1/0, attached
    *via 99.1.1.1, Vlan2, [0/0], 2d08h, local
99.1.1.2/32, ubest/mbest: 1/0
    *via 1.1.1.1%default, [200/0], 1d09h, bgp-2, internal, tag 2 (evpn) segid: 4 tunnelid: 0x1010101 encap: VXLAN
 
99.1.1.4/32, ubest/mbest: 1/0, attached
    *via 99.1.1.4, Vlan2, [190/0], 2d08h, hmm
99.1.1.18/32, ubest/mbest: 1/0
    *via 1.1.1.1%default, [200/0], 1d09h, bgp-2, internal, tag 2 (evpn) segid: 5 tunnelid: 0x1010101 encap: VXLAN
 
100.1.1.1/32, ubest/mbest: 1/0
    *via 1.1.1.1%default, [200/130816], 13:09:05, bgp-2, internal, tag 2 (evpn) segid: 4 tunnelid: 0x1010101 encap: VXLAN
 
100.1.1.2/32, ubest/mbest: 1/0
    *via 1.1.1.1%default, [200/130816], 13:09:05, bgp-2, internal, tag 2 (evpn) segid: 4 tunnelid: 0x1010101 encap: VXLAN
 
100.1.1.3/32, ubest/mbest: 1/0
    *via 1.1.1.1%default, [200/130816], 13:09:05, bgp-2, internal, tag 2 (evpn) segid: 4 tunnelid: 0x1010101 encap: VXLAN
 
150.1.1.1/32, ubest/mbest: 1/0
    *via 1.1.1.1%default, [200/130816], 13:09:05, bgp-2, internal, tag 2 (evpn) segid: 4 tunnelid: 0x1010101 encap: VXLAN
 
150.1.1.2/32, ubest/mbest: 1/0
    *via 1.1.1.1%default, [200/130816], 13:09:05, bgp-2, internal, tag 2 (evpn) segid: 4 tunnelid: 0x1010101 encap: VXLAN

Great. External routes has been injected inside VRF A and has been sent to N3 via BGP. Final verification will be testing PCs:

n4# ping 100.1.1.1
PING 100.1.1.1 (100.1.1.1): 56 data bytes
64 bytes from 100.1.1.1: icmp_seq=0 ttl=252 time=8.085 ms
64 bytes from 100.1.1.1: icmp_seq=1 ttl=252 time=6.038 ms
64 bytes from 100.1.1.1: icmp_seq=2 ttl=252 time=5.899 ms
64 bytes from 100.1.1.1: icmp_seq=3 ttl=252 time=5.748 ms
64 bytes from 100.1.1.1: icmp_seq=4 ttl=252 time=5.863 ms

--- 100.1.1.1 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 5.748/6.326/8.085 ms
!
!
n4# ping 150.1.1.2
PING 150.1.1.2 (150.1.1.2): 56 data bytes
64 bytes from 150.1.1.2: icmp_seq=0 ttl=252 time=8.984 ms
64 bytes from 150.1.1.2: icmp_seq=1 ttl=252 time=1.456 ms
64 bytes from 150.1.1.2: icmp_seq=2 ttl=252 time=3.119 ms
64 bytes from 150.1.1.2: icmp_seq=3 ttl=252 time=5.002 ms
64 bytes from 150.1.1.2: icmp_seq=4 ttl=252 time=2.722 ms

--- 150.1.1.2 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 4.155/5.160/7.023 ms

Finally mission accomplished. This was a simple scenario of VxLAN, but it can be very complicated. for example you can assume using two router as BGP RR/Multicast RP to increase redundancy as well as deploying vPC on VTEPs and even using NAT to give access to Internet. If I had a time, I will post those scenarios too soon.

Deploying VxLAN with Cisco Nexus 9000v (Part 2)
Tagged on: