SONiC and White Box switches in the Enterprise DC! – Part 3

After discussing the architecture of our design during part 1, and the underlay configuration during part 2, today i’ll show how the overlay it’s configured and hopefully we will be able to draw our conclusions to the question: Are SONiC and White Box switches ready to be used in the enterprise DC?

Our two servers will be connected with LACP and trunk interfaces. 1 VLAN will be bridged (no SVI) and both servers will have an interface into such vlan so that layer 2 can be tested.
Other 2 vlans instead will each be configured on a different pair of switches together with an SVI so that Layer 3 symmetric IRB can be tested.

VRF Configuration

First of all, let’s create a VRF. This vrf requires an VLAN and a Layer 3 VNI for symmetric IRB to function. Configuration is really simple, but a small caveat must be overlooked, specifically every vrf must contain the prefix Vrf- in the name.

From a configuration point of view, we have to follow the usual steps:

  1. Create a VRF
  2. Create a Vlan and allow it to the peer-link port channel
  3. Create a SVI interface and assign it to the VRF itself
  4. Associate the VNI to the vlan, then map it as a L3 VNI
  5. Configure BGP’s AF (in FRR)
config vrf add Vrf-prod 
config vlan add 3800
config vlan member add 3800 PortChannel1
config interface vrf bind Vlan3800 Vrf-prod
config vxlan map add nve1 3800 1000000
config vrf add_vrf_vni_map Vrf-prod 1000000

vtysh
conf t
router bgp 65000 vrf Vrf-prod
bgp router-id 10.0.0.11
bgp log-neighbor-changes
bgp graceful-restart
bgp graceful-restart preserve-fw-state
!
address-family ipv4 unicast
redistribute connected
exit-address-family
!
address-family l2vpn evpn
advertise ipv4 unicast
exit-address-family
end
exit

Once done, we can easily verify the config

root@SONIC-Leaf301:/home/admin# show vlan brief
+-----------+---------------+--------------+----------------+-----------------------+
| VLAN ID | IP Address | Ports | Port Tagging | DHCP Helper Address |
+===========+===============+==============+================+=======================+
| 3800 | | PortChannel1 | tagged | |
+-----------+---------------+--------------+----------------+-----------------------+
| 3965 | 10.10.10.0/31 | PortChannel1 | tagged | |
+-----------+---------------+--------------+----------------+-----------------------+
root@SONIC-Leaf301:/home/admin# show vxlan vlanvnimap
+----------+---------+
| VLAN | VNI |
+==========+=========+
| Vlan3800 | 1000000 |
+----------+---------+
Total count : 1

root@SONIC-Leaf301:/home/admin# show vxlan vrfvnimap
+----------+---------+
| VRF | VNI |
+==========+=========+
| Vrf-prod | 1000000 |
+----------+---------+
Total count : 1

root@SONIC-Leaf301:/home/admin# show ip route vrf Vrf-prod
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued route, r - rejected route, # - not installed in hardware
VRF Vrf-prod:
C>* 1.1.1.0/31 is directly connected, Vlan1234, 00:00:06
B>* 1.1.1.2/31 [200/0] via 11.11.11.113, Vlan3800 onlink, 00:21:34
C>* 100.100.100.1/32 is directly connected, Loopback100, 00:07:20
B>* 100.100.100.2/32 [200/0] via 1.1.1.1, Vlan1234, 00:00:04
B>* 100.100.100.3/32 [200/0] via 11.11.11.113, Vlan3800 onlink, 00:21:34
B>* 100.100.100.4/32 [200/0] via 11.11.11.113, Vlan3800 onlink, 00:21:34

root@SONIC-Leaf301:/home/admin# show vxlan tunnel
+--------------+--------------+-------------------+--------------+
| SIP | DIP | Creation Source | OperStatus |
+==============+==============+===================+==============+
| 11.11.11.111 | 11.11.11.113 | EVPN | oper_up |
+--------------+--------------+-------------------+--------------+
Total count : 1

root@SONIC-Leaf301:/home/admin# ping 100.100.100.2 -I Vrf-prod
ping: Warning: source address might be selected on device other than Vrf-prod.
PING 100.100.100.2 (100.100.100.2) from 1.1.1.0 Vrf-prod: 56(84) bytes of data.
64 bytes from 100.100.100.2: icmp_seq=1 ttl=64 time=0.255 ms
64 bytes from 100.100.100.2: icmp_seq=2 ttl=64 time=0.239 ms
^C
--- 100.100.100.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1007ms
rtt min/avg/max/mdev = 0.239/0.247/0.255/0.008 ms
root@SONIC-Leaf301:/home/admin# ping 100.100.100.3 -I Vrf-prod
ping: Warning: source address might be selected on device other than Vrf-prod.
PING 100.100.100.3 (100.100.100.3) from 100.100.100.1 Vrf-prod: 56(84) bytes of data.
64 bytes from 100.100.100.3: icmp_seq=1 ttl=64 time=0.452 ms
64 bytes from 100.100.100.3: icmp_seq=2 ttl=64 time=0.301 ms
^C
--- 100.100.100.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1005ms
rtt min/avg/max/mdev = 0.301/0.376/0.452/0.077 ms
root@SONIC-Leaf301:/home/admin# ping 100.100.100.4 -I Vrf-prod
ping: Warning: source address might be selected on device other than Vrf-prod.
PING 100.100.100.4 (100.100.100.4) from 100.100.100.1 Vrf-prod: 56(84) bytes of data.
64 bytes from 100.100.100.4: icmp_seq=1 ttl=63 time=0.345 ms
64 bytes from 100.100.100.4: icmp_seq=2 ttl=63 time=0.279 ms
64 bytes from 100.100.100.4: icmp_seq=3 ttl=63 time=0.251 ms
^C
--- 100.100.100.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2045ms
rtt min/avg/max/mdev = 0.251/0.291/0.345/0.044 ms

I’ve also included some one unique Loopback on each leaf, and vrf-lite iBGP between the two MCLAG peers across the peer-link (the reason why this is necessary it’s left to the reader to figure out, at least for now 😉 ).
Connectivity between Loopbacks is also verified.

VLAN Configuration

It might have been overlooked, but while configuring a VRF, we already configured a vlan (vlan 3800). But let’s give it another try.

config vlan add 200
config vlan member add 200 PortChannel1
config vxlan map add nve1 200 1000200

The configuration of an SVI (when necessary) is also trivial, we just need to take care of enabling suppress-arp and to specify that the IP address is a Distributed Anycast Gateway (DAG):

config interface vrf bind Vlan200 Vrf-prod
config interface ip anycast-address add Vlan200 10.10.200.1/24
config neigh_suppress enable 200

In my case, i also want to configure DHCP relay from my server, and i can do that with a single line (can you tell why i need to enable option 82 sub-option link selection?):

config interface ip dhcp-relay add Vlan200 10.10.10.100 10.10.10.101 -src-intf=Loopback100 -link-select=enable

On top of the previously used show commands, other commands can be used to verify the config applied:

root@SONIC-Leaf301:/home/admin# show ip static-anycast-gateway 
Configured Anycast Gateway MAC address: 00:00:22:22:33:33
IPv4 Anycast Gateway MAC address: enable
Total number of gateway: 2
Total number of gateway admin UP: 2
Total number of gateway oper UP: 2
Interfaces Gateway Address Vrf Admin/Oper
------------ ----------------- -------- ------------
Vlan200 10.10.200.1/24 Vrf-prod up/up
Vlan500 10.10.10.1/24 Vrf-prod up/up

root@SONIC-Leaf301:/home/admin# show neigh-suppress all
+----------+----------------+---------------------+
| VLAN | STATUS | ASSOCIATED_NETDEV |
+==========+================+=====================+
| Vlan3800 | Not Configured | nve1-3800 |
+----------+----------------+---------------------+
| Vlan100 | Not Configured | nve1-100 |
+----------+----------------+---------------------+
| Vlan200 | Configured | nve1-200 |
+----------+----------------+---------------------+
| Vlan500 | Configured | nve1-500 |
+----------+----------------+---------------------+
Total count : 4

root@SONIC-Leaf301:/home/admin# show ip dhcp-relay brief
+------------------+-----------------------+
| Interface Name | DHCP Helper Address |
+==================+=======================+
| Vlan200 | 10.10.10.100 |
| | 10.10.10.101 |
+------------------+-----------------------+

SONIC-Leaf301# show bgp l2vpn evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
Number of L2 VNIs: 3
Number of L3 VNIs: 1
Flags: * - Kernel
VNI Type RD Import RT Export RT Tenant VRF
* 1000200 L2 10.0.0.11:200 65000:1000200 65000:1000200 Vrf-prod
* 1000500 L2 10.0.0.11:500 65000:1000500 65000:1000500 Vrf-prod
* 1000100 L2 10.0.0.11:100 65000:1000100 65000:1000100 default
* 1000000 L3 10.0.0.11:5096 65000:1000000 65000:1000000 Vrf-prod

We did quiet a big deal of configuration right now, but of course, we cannot see anything unless we configure the ports facing our hosts

Hosts port configuration

The switches i am working on, have a limitation where every 12 ports must have exactly the same speed. This is an issue of this specific switch, not a sonic problem, nonetheless we need to be aware of it.

root@SONIC-Leaf301:/home/admin# show portgroup                         
portgroup ports valid speeds
----------- ------------- ----------------
1 Ethernet0-11 25000,10000,1000
2 Ethernet12-23 25000,10000,1000
3 Ethernet24-35 25000,10000,1000
4 Ethernet36-47 25000,10000,1000

root@SONIC-Leaf301:/home/admin# config portgroup speed 1 10000
Config portgroup 1 speed 10000

Now it’s time to configure our MCLAG port-channel:

config portchannel add PortChannel100      
config portchannel member add PortChannel100 Ethernet9
config mclag member add 1 PortChannel100
config interface startup Ethernet9

to verify the config:

root@SONIC-Leaf301:/home/admin# show interfaces portchannel 
Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available, S - selected, D - deselected
No. Team Dev Protocol Ports
----- -------------- ----------- ---------------------------
1 PortChannel1 LACP(A)(Up) Ethernet52(S) Ethernet48(S)
100 PortChannel100 LACP(A)(Up) Ethernet9(S)
101 PortChannel101 LACP(A)(Up) Ethernet1(S)

admin@SONIC-Leaf301:~$ sonic-cli
SONIC-Leaf301# show mclag brief

Domain ID : 1
Role : active
Session Status : up
Peer Link Status : up
Source Address : 10.0.0.11
Peer Address : 10.0.0.12
Peer Link : PortChannel1
Keepalive Interval : 1 secs
Session Timeout : 30 secs
System Mac : 80:a2:35:81:dd:f0


Number of MLAG Interfaces:2
-----------------------------------------------------------
MLAG Interface Local/Remote Status
-----------------------------------------------------------
PortChannel101 up/up
PortChannel100 up/up

And to finish, we only need to add the vlans to the trunks:

config vlan member add 100 PortChannel100 
config vlan member add 200 PortChannel100
root@SONIC-Leaf301:/home/admin# show vlan brief 
+-----------+---------------+----------------+----------------+-----------------------+
| VLAN ID | IP Address | Ports | Port Tagging | DHCP Helper Address |
+===========+===============+================+================+=======================+
| 100 | | PortChannel1 | tagged | |
| | | PortChannel100 | tagged | |
+-----------+---------------+----------------+----------------+-----------------------+
| 200 | | PortChannel1 | tagged | 10.10.10.100 |
| | | PortChannel100 | tagged | 10.10.10.101 |
+-----------+---------------+----------------+----------------+-----------------------+
| 500 | | PortChannel1 | tagged | |
| | | PortChannel101 | tagged | |
+-----------+---------------+----------------+----------------+-----------------------+
| 1234 | 1.1.1.0/31 | PortChannel1 | tagged | |
+-----------+---------------+----------------+----------------+-----------------------+
| 3800 | | PortChannel1 | tagged | |
+-----------+---------------+----------------+----------------+-----------------------+
| 3965 | 10.10.10.0/31 | PortChannel1 | tagged | |
+-----------+---------------+----------------+----------------+-----------------------+

Now, if everything works as expect, i should be able to see remote mac addresses as well as /32 host routes in the routing table:

root@SONIC-Leaf301:/home/admin# show lldp table 
Capability codes: (R) Router, (B) Bridge, (O) Other
LocalPort RemoteDevice RemotePortID Capability RemotePortDescr
----------- -------------- ----------------- ------------ -------------------------
Ethernet1 csp-srv-02 90:e2:ba:f6:cd:6d O Interface 13 as enp7s0f1
Ethernet9 MKTK-SW01 bond1.200 R
Ethernet48 SONIC-Leaf302 80:a2:35:81:e3:f0 BR Ethernet48
Ethernet52 SONIC-Leaf302 80:a2:35:81:e3:f0 BR Ethernet52
Ethernet72 SONIC-Spine31 80:a2:35:f2:7f:94 BR Ethernet120
Ethernet76 SONIC-Spine32 80:a2:35:f2:80:c0 BR Ethernet120
eth0 c6500-vxlan Gi3/36 BR GigabitEthernet3/36
--------------------------------------------------
Total entries displayed: 7

root@SONIC-Leaf301:/home/admin# show mac
No. Vlan MacAddress Port Type
----- ------ ----------------- ----------------------- -------
1 100 B8:69:F4:99:D1:4A PortChannel100 Dynamic
2 100 B8:69:F4:99:D1:4C VxLAN DIP: 11.11.11.113 Dynamic
3 200 B8:69:F4:99:D1:4A PortChannel100 Dynamic
4 500 02:5C:1F:02:1F:11 VxLAN DIP: 11.11.11.113 Dynamic
5 500 02:5C:1F:02:20:10 PortChannel101 Dynamic
6 1234 80:A2:35:81:E3:F0 PortChannel1 Static
7 3965 80:A2:35:81:E3:F0 PortChannel1 Static
Total number of entries 7

Remote VTEP IP al invalid format
root@SONIC-Leaf301:/home/admin# show vxlan evpn_remote_mac all
+---------+-------------------+--------------+---------+---------+
| VLAN | MAC | RemoteVTEP | VNI | Type |
+=========+===================+==============+=========+=========+
| Vlan100 | b8:69:f4:99:d1:4c | 11.11.11.113 | 1000100 | dynamic |
+---------+-------------------+--------------+---------+---------+
| Vlan500 | 02:5c:1f:02:1f:11 | 11.11.11.113 | 1000500 | dynamic |
+---------+-------------------+--------------+---------+---------+
Total count : 2

root@SONIC-Leaf301:/home/admin# show ip route vrf Vrf-prod
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued route, r - rejected route, # - not installed in hardware
VRF Vrf-prod:
C>* 1.1.1.0/31 is directly connected, Vlan1234, 02:05:54
B>* 1.1.1.2/31 [200/0] via 11.11.11.113, Vlan3800 onlink, 02:27:22
C>* 10.10.10.0/24 is directly connected, sag500.256, 01:12:29
B>* 10.10.10.100/32 [200/0] via 11.11.11.113, Vlan3800 onlink, 02:27:22
B>* 10.10.30.0/24 [200/0] via 11.11.11.113, Vlan3800 onlink, 02:27:22
B>* 10.10.30.151/32 [200/0] via 11.11.11.113, Vlan3800 onlink, 02:27:22
C>* 10.10.200.0/24 is directly connected, sag200.256, 01:12:29
C>* 100.100.100.1/32 is directly connected, Loopback100, 02:13:08
B>* 100.100.100.2/32 [200/0] via 1.1.1.1, Vlan1234, 02:05:52
B>* 100.100.100.3/32 [200/0] via 11.11.11.113, Vlan3800 onlink, 02:27:22
B>* 100.100.100.4/32 [200/0] via 11.11.11.113, Vlan3800 onlink, 02:27:22

And of course, let’s not forget the real test, data-plane testing from the servers themselves.

DHCP:

[admin@MKTK-SW01] > /ip dhcp-client print 
Flags: X - disabled, I - invalid, D - dynamic
# INTERFACE USE-PEER-DNS ADD-DEFAULT-ROUTE STATUS ADDRESS
0 bond1.200 yes yes bound 10.10.200.149/24
1 bond2.300 yes yes bound 10.10.30.151/24

Bridging:

[admin@MKTK-SW01] > ping 10.10.100.20 routing-table=HOST1 interface=bond1.100
  SEQ HOST                                     SIZE TTL TIME  STATUS  
    0 10.10.100.20                               56  64 0ms  
    1 10.10.100.20                               56  64 0ms  
    2 10.10.100.20                               56  64 0ms  
    3 10.10.100.20                               56  64 0ms  
    4 10.10.100.20                               56  64 0ms  
    sent=5 received=5 packet-loss=0% min-rtt=0ms avg-rtt=0ms max-rtt=0ms 

[admin@MKTK-SW01] > tool traceroute 10.10.100.20 routing-table=HOST1 interface=bond1.100 
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS
 1 10.10.100.20                       0%    5   0.1ms     0.1     0.1     0.1       0

Routing:

[admin@MKTK-SW01] > ping 10.10.30.151 routing-table=HOST1 interface=bond1.200 
  SEQ HOST                                     SIZE TTL TIME  STATUS 
    0 10.10.30.151                               56  64 0ms  
    1 10.10.30.151                               56  64 0ms  
    2 10.10.30.151                               56  64 0ms  
    3 10.10.30.151                               56  64 0ms  
    4 10.10.30.151                               56  64 0ms  
    sent=5 received=5 packet-loss=0% min-rtt=0ms avg-rtt=0ms max-rtt=0ms 

[admin@MKTK-SW01] > tool traceroute 10.10.30.151 routing-table=HOST1 interface=bond1.200 
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS
 1 10.10.200.1                        0%    5   0.2ms     0.3     0.2     0.3       0
 2 1.1.1.3                            0%    5   0.2ms     0.2     0.2     0.3       0
 3 10.10.30.151                       0%    5   0.1ms     0.1     0.1     0.1       0

In Conclusion

With what we have seen so far, i really believe that SONiC is mature enough to cover most of the common DC network requirements. Notice that differently than other vendor’s solution that believe can do everything, including make you coffee or take you to the moon; SONiC is more specialised, does a few things and does them very well.
As long as what you need to do is supported by SONiC, then go ahead, it isn’t going to disappoint you.

An enterprise that considers to run SONiC should also understand the support model.
SONiC itself comes without support, and really, here we are looking at a typical open-source situation where you can choose to operate a software completely free of charge on your own, or you could pay a reputable company to provide you with a patched and supported software revision (a bit like Red Hat or SUSE Linux).

From an hardware standpoint, i think that the white boxes are mature enough. For example, Edge-Core’s AS7326-56X is basically identical to Juniper’s QFX 5120 (including port groups).
We are in the same world as your servers really. You can get your hardware from any vendor or you can find a trusted one like Dell. It’s up to you really..

In short then, what are the take away for “standard” enterprises?

  1. SONiC will work great if what you need to do fits the supported features
  2. White Box switches are comparable or identical to big vendor’s hardware
  3. You REALLY should be looking at someone to provide you with end to end support though. Maybe someone like Broadcom or other Service Providers to ensure you can get a single point of contact for all of your possible problems
  4. The knowledge gap can be scary at first, but it’s no longer a big obstacle. ACI for example was a nightmare and took me forever to learn and understand, SONiC on the other end was a piece of cake.
  5. Try and Experiment, open networking is so cheap that it costs almost nothing to bring up a small lab or even a production POC.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s