Scaling EVPN Multi-Site Overlays using Route-Servers

Cisco’s EVPN Multi-Site it’s a great technology that allows us to achieve massive scale of an EVPN network. With the latest release, the official scalability numbers give us something in the realm of over 12000 VTEPs (512 VTEPs per site x 25 sites).
I’m in no way suggesting that you would need such a big topology and you definitely should segment way sooner you reach the limit, but still…

The main configuration requirement for the Multi-Site overlay is to have a full mesh of eBGP peering between all border gateways.

This has scalability drawbacks as usual. Not only each leaf will have ever growing number of peers which will soon grow out of control, but maybe, worse is the fact that after one site is added, every other site must be touched too.

To avoid a full mesh, for iBGP topologies we would be using a Route Reflector, but with eBGP that’s obviously not an option. So, instead of a RR they way to scale eBGP peerings is to leverage a Route-Server.

A Route-Server provides route reflection capabilities and as such it must ensure that NLRIs attributes like the Next Hop and route-targets aren’t changed.
In Cisco’s EVPN implementation, the auto defined route-targets are based on ASN:VNI, and in order to be able to use this simplified config, the RS should also support the “rewrite-evpn-rt-asn” feature; if that’s not the case, then hard coded and consistent route-targets must be defined across the VTEPs in the network. Finally, the route-server doesn’t have to be in the data plane since it’s only a control plane node.

Unfortunately, for EVPN, there isn’t a “route-server-client” configuration nob yet 😦 , nor we can find a configuration example in the Cisco pages. Fortunately, knowing the requirements we can figure out how the config should look like.

NX-OS EVPN Route-Server Configuration

feature nv overlay
nv overlay evpn
feature bgp
!
route-map RETAIN-ORIGINAL-VTEP-NEXTHOP permit 10
set ip next-hop unchanged
!
router bgp 12345
log-neighbor-changes
address-family l2vpn evpn
retain route-target all
neighbor 1.1.1.0
remote-as 100
address-family l2vpn evpn
send-community
send-community extended
route-map RETAIN-ORIGINAL-VTEP-NEXTHOP out
rewrite-evpn-rt-asn
neighbor 1.1.1.3
remote-as 200
address-family l2vpn evpn
send-community
send-community extended
route-map RETAIN-ORIGINAL-VTEP-NEXTHOP out
rewrite-evpn-rt-asn

IOS-XE EVPN Route-Server Configuration

!
route-map RETAIN-ORIGINAL-VTEP-NEXTHOP permit 10
set ip next-hop unchanged
!
router bgp 12345
bgp log-neighbor-changes
no bgp default route-target filter
neighbor 1.1.1.0 remote-as 100
neighbor 1.1.1.0 disable-connected-check
neighbor 1.1.1.3 remote-as 200
neighbor 1.1.1.3 disable-connected-check
!
address-family l2vpn evpn
rewrite-evpn-rt-asn
neighbor 1.1.1.0 activate
neighbor 1.1.1.0 send-community both
neighbor 1.1.1.0 soft-reconfiguration inbound
neighbor 1.1.1.0 route-map RETAIN-ORIGINAL-VTEP-NEXTHOP out
neighbor 1.1.1.3 activate
neighbor 1.1.1.3 send-community both
neighbor 1.1.1.3 soft-reconfiguration inbound
neighbor 1.1.1.3 route-map RETAIN-ORIGINAL-VTEP-NEXTHOP out
exit-address-family
!

Just a couple of notes on the above config:

  1. The command “disable-connected-check” is required otherwise the router will reject received prefixes with “DENIED due to: non-connected MP_REACH NEXTHOP
  2. The command “next-hop-unchanged” has no effect in the address-family L2VPN EVPN (probably a bug). A route-map is necessary in order to achieve the same result.

IOS-XR EVPN Route-Server Configuration

!
route-policy ACCEPT-ALL
pass
end-policy
!
router bgp 12345
nsr
bgp router-id 1.1.1.1
bgp graceful-restart
!
address-family l2vpn evpn
retain route-target all
!
neighbor 1.1.1.0
remote-as 100
ignore-connected-check
!
address-family l2vpn evpn
send-community-ebgp
route-policy ACCEPT-ALL in
route-policy ACCEPT-ALL out
send-extended-community-ebgp
soft-reconfiguration inbound always
next-hop-unchanged
!
!
neighbor 1.1.1.3
remote-as 200
ignore-connected-check
!
address-family l2vpn evpn
send-community-ebgp
route-policy ACCEPT-ALL in
route-policy ACCEPT-ALL out
send-extended-community-ebgp
soft-reconfiguration inbound always
next-hop-unchanged
!
!
!

As for IOS-XE, the command “ignore-connected-check” is required.
Additionally, IOS-XR unfortunately doesn’t support “rewrite-evpn-rt-asn“.
This means that each VTEP will need to have manually configured the appropriate route-targets highly increasing configuration complexity.

Unless you have some automation backing up your EVPN deployment, probably isn’t a good idea to use IOS-XR as an EVPN Route-Server.

Do you have anything else to add? Then contact me, or leave a message below.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s