Tag Archives: open networking

SONiC and White Box switches in the Enterprise DC! – Part 3

Posted on by
Reply

After discussing the architecture of our design during part 1, and the underlay configuration during part 2, today i’ll show how the overlay it’s configured and hopefully we will be able to draw our conclusions to the question: Are SONiC and White Box switches ready to be used in the enterprise DC?

Our two servers will be connected with LACP and trunk interfaces. 1 VLAN will be bridged (no SVI) and both servers will have an interface into such vlan so that layer 2 can be tested.
Other 2 vlans instead will each be configured on a different pair of switches together with an SVI so that Layer 3 symmetric IRB can be tested.

VRF Configuration

First of all, let’s create a VRF. This vrf requires an VLAN and a Layer 3 VNI for symmetric IRB to function. Configuration is really simple, but a small caveat must be overlooked, specifically every vrf must contain the prefix Vrf- in the name.

From a configuration point of view, we have to follow the usual steps:

  1. Create a VRF
  2. Create a Vlan and allow it to the peer-link port channel
  3. Create a SVI interface and assign it to the VRF itself
  4. Associate the VNI to the vlan, then map it as a L3 VNI
  5. Configure BGP’s AF (in FRR)
config vrf add Vrf-prod 
config vlan add 3800 
config vlan member add 3800 PortChannel1 
config interface vrf bind Vlan3800 Vrf-prod 
config vxlan map add nve1 3800 1000000 
config vrf add_vrf_vni_map Vrf-prod 1000000

vtysh 
 conf t 
  router bgp 65000 vrf Vrf-prod 
   bgp router-id 10.0.0.11 
   bgp log-neighbor-changes 
   bgp graceful-restart 
   bgp graceful-restart preserve-fw-state 
   ! 
   address-family ipv4 unicast 
    redistribute connected 
   exit-address-family 
   ! 
   address-family l2vpn evpn 
    advertise ipv4 unicast 
   exit-address-family 
   end 
 exit

Once done, we can easily verify the config

root@SONIC-Leaf301:/home/admin# show vlan brief
+-----------+---------------+--------------+----------------+-----------------------+
|   VLAN ID | IP Address    | Ports        | Port Tagging   | DHCP Helper Address   |
+===========+===============+==============+================+=======================+
|      3800 |               | PortChannel1 | tagged         |                       |
+-----------+---------------+--------------+----------------+-----------------------+
|      3965 | 10.10.10.0/31 | PortChannel1 | tagged         |                       |
+-----------+---------------+--------------+----------------+-----------------------+
root@SONIC-Leaf301:/home/admin# show vxlan vlanvnimap 
+----------+---------+
| VLAN     |     VNI |
+==========+=========+
| Vlan3800 | 1000000 |
+----------+---------+
Total count : 1 

root@SONIC-Leaf301:/home/admin# show vxlan vrfvnimap 
+----------+---------+
| VRF      |     VNI |
+==========+=========+
| Vrf-prod | 1000000 |
+----------+---------+
Total count : 1 

root@SONIC-Leaf301:/home/admin# show ip route vrf Vrf-prod
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued route, r - rejected route, # - not installed in hardware
VRF Vrf-prod:
C>* 1.1.1.0/31 is directly connected, Vlan1234, 00:00:06
B>* 1.1.1.2/31 [200/0] via 11.11.11.113, Vlan3800 onlink, 00:21:34
C>* 100.100.100.1/32 is directly connected, Loopback100, 00:07:20
B>* 100.100.100.2/32 [200/0] via 1.1.1.1, Vlan1234, 00:00:04
B>* 100.100.100.3/32 [200/0] via 11.11.11.113, Vlan3800 onlink, 00:21:34
B>* 100.100.100.4/32 [200/0] via 11.11.11.113, Vlan3800 onlink, 00:21:34

root@SONIC-Leaf301:/home/admin# show vxlan tunnel
+--------------+--------------+-------------------+--------------+
| SIP | DIP | Creation Source | OperStatus |
+==============+==============+===================+==============+
| 11.11.11.111 | 11.11.11.113 | EVPN | oper_up |
+--------------+--------------+-------------------+--------------+
Total count : 1

root@SONIC-Leaf301:/home/admin# ping 100.100.100.2 -I Vrf-prod
ping: Warning: source address might be selected on device other than Vrf-prod.
PING 100.100.100.2 (100.100.100.2) from 1.1.1.0 Vrf-prod: 56(84) bytes of data.
64 bytes from 100.100.100.2: icmp_seq=1 ttl=64 time=0.255 ms
64 bytes from 100.100.100.2: icmp_seq=2 ttl=64 time=0.239 ms
^C
--- 100.100.100.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1007ms
rtt min/avg/max/mdev = 0.239/0.247/0.255/0.008 ms
root@SONIC-Leaf301:/home/admin# ping 100.100.100.3 -I Vrf-prod
ping: Warning: source address might be selected on device other than Vrf-prod.
PING 100.100.100.3 (100.100.100.3) from 100.100.100.1 Vrf-prod: 56(84) bytes of data.
64 bytes from 100.100.100.3: icmp_seq=1 ttl=64 time=0.452 ms
64 bytes from 100.100.100.3: icmp_seq=2 ttl=64 time=0.301 ms
^C
--- 100.100.100.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1005ms
rtt min/avg/max/mdev = 0.301/0.376/0.452/0.077 ms
root@SONIC-Leaf301:/home/admin# ping 100.100.100.4 -I Vrf-prod
ping: Warning: source address might be selected on device other than Vrf-prod.
PING 100.100.100.4 (100.100.100.4) from 100.100.100.1 Vrf-prod: 56(84) bytes of data.
64 bytes from 100.100.100.4: icmp_seq=1 ttl=63 time=0.345 ms
64 bytes from 100.100.100.4: icmp_seq=2 ttl=63 time=0.279 ms
64 bytes from 100.100.100.4: icmp_seq=3 ttl=63 time=0.251 ms
^C
--- 100.100.100.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2045ms
rtt min/avg/max/mdev = 0.251/0.291/0.345/0.044 ms

I’ve also included some one unique Loopback on each leaf, and vrf-lite iBGP between the two MCLAG peers across the peer-link (the reason why this is necessary it’s left to the reader to figure out, at least for now 😉 ).
Connectivity between Loopbacks is also verified.

VLAN Configuration

It might have been overlooked, but while configuring a VRF, we already configured a vlan (vlan 3800). But let’s give it another try.

config vlan add 200
config vlan member add 200 PortChannel1
config vxlan map add nve1 200 1000200

The configuration of an SVI (when necessary) is also trivial, we just need to take care of enabling suppress-arp and to specify that the IP address is a Distributed Anycast Gateway (DAG):

config interface vrf bind Vlan200 Vrf-prod
config interface ip anycast-address add Vlan200 10.10.200.1/24
config neigh_suppress enable 200

In my case, i also want to configure DHCP relay from my server, and i can do that with a single line (can you tell why i need to enable option 82 sub-option link selection?):

config interface ip dhcp-relay add Vlan200 10.10.10.100 10.10.10.101 -src-intf=Loopback100 -link-select=enable

On top of the previously used show commands, other commands can be used to verify the config applied:

root@SONIC-Leaf301:/home/admin# show ip static-anycast-gateway 
Configured Anycast Gateway MAC address: 00:00:22:22:33:33 
IPv4 Anycast Gateway MAC address: enable 
Total number of gateway: 2 
Total number of gateway admin UP: 2 
Total number of gateway oper UP: 2 
Interfaces    Gateway Address    Vrf       Admin/Oper 
------------  -----------------  --------  ------------ 
Vlan200       10.10.200.1/24     Vrf-prod  up/up 
Vlan500       10.10.10.1/24      Vrf-prod  up/up 

root@SONIC-Leaf301:/home/admin# show neigh-suppress all                                       
+----------+----------------+---------------------+ 
| VLAN     | STATUS         | ASSOCIATED_NETDEV   | 
+==========+================+=====================+ 
| Vlan3800 | Not Configured | nve1-3800           | 
+----------+----------------+---------------------+ 
| Vlan100  | Not Configured | nve1-100            | 
+----------+----------------+---------------------+ 
| Vlan200  | Configured     | nve1-200            | 
+----------+----------------+---------------------+ 
| Vlan500  | Configured     | nve1-500            | 
+----------+----------------+---------------------+ 
Total count : 4 

root@SONIC-Leaf301:/home/admin# show ip dhcp-relay brief 
+------------------+-----------------------+ 
| Interface Name   | DHCP Helper Address   | 
+==================+=======================+ 
| Vlan200          | 10.10.10.100          | 
|                  | 10.10.10.101          | 
+------------------+-----------------------+

SONIC-Leaf301# show bgp l2vpn evpn vni 
Advertise Gateway Macip: Disabled 
Advertise SVI Macip: Disabled 
Advertise All VNI flag: Enabled 
BUM flooding: Head-end replication 
Number of L2 VNIs: 3 
Number of L3 VNIs: 1 
Flags: * - Kernel 
  VNI        Type RD                    Import RT                 Export RT                 Tenant VRF                           
* 1000200    L2   10.0.0.11:200         65000:1000200             65000:1000200            Vrf-prod                             
* 1000500    L2   10.0.0.11:500         65000:1000500             65000:1000500            Vrf-prod                             
* 1000100    L2   10.0.0.11:100         65000:1000100             65000:1000100            default                              
* 1000000    L3   10.0.0.11:5096        65000:1000000             65000:1000000            Vrf-prod

We did quiet a big deal of configuration right now, but of course, we cannot see anything unless we configure the ports facing our hosts

Hosts port configuration

The switches i am working on, have a limitation where every 12 ports must have exactly the same speed. This is an issue of this specific switch, not a sonic problem, nonetheless we need to be aware of it.

root@SONIC-Leaf301:/home/admin# show portgroup                         
  portgroup          ports      valid speeds 
-----------  -------------  ---------------- 
          1   Ethernet0-11  25000,10000,1000 
          2  Ethernet12-23  25000,10000,1000 
          3  Ethernet24-35  25000,10000,1000 
          4  Ethernet36-47  25000,10000,1000 

root@SONIC-Leaf301:/home/admin# config portgroup speed 1 10000 
Config portgroup 1 speed 10000

Now it’s time to configure our MCLAG port-channel:

config portchannel add PortChannel100      
config portchannel member add PortChannel100 Ethernet9 
config mclag member add 1 PortChannel100 
config interface startup Ethernet9

to verify the config:

root@SONIC-Leaf301:/home/admin# show interfaces portchannel 
Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available, S - selected, D - deselected 
  No.  Team Dev        Protocol     Ports 
-----  --------------  -----------  --------------------------- 
    1  PortChannel1    LACP(A)(Up)  Ethernet52(S) Ethernet48(S) 
  100  PortChannel100  LACP(A)(Up)  Ethernet9(S) 
  101  PortChannel101  LACP(A)(Up)  Ethernet1(S) 

admin@SONIC-Leaf301:~$ sonic-cli 
SONIC-Leaf301# show mclag brief 
 
Domain ID            : 1 
Role                 : active 
Session Status       : up 
Peer Link Status     : up 
Source Address       : 10.0.0.11 
Peer Address         : 10.0.0.12 
Peer Link            : PortChannel1 
Keepalive Interval   : 1 secs 
Session Timeout      : 30 secs 
System Mac           : 80:a2:35:81:dd:f0 
 
 
Number of MLAG Interfaces:2 
----------------------------------------------------------- 
MLAG Interface       Local/Remote Status 
----------------------------------------------------------- 
PortChannel101           up/up 
PortChannel100           up/up

And to finish, we only need to add the vlans to the trunks:

config vlan member add 100 PortChannel100 
config vlan member add 200 PortChannel100
root@SONIC-Leaf301:/home/admin# show vlan brief 
+-----------+---------------+----------------+----------------+-----------------------+ 
|   VLAN ID | IP Address    | Ports          | Port Tagging   | DHCP Helper Address   | 
+===========+===============+================+================+=======================+ 
|       100 |               | PortChannel1   | tagged         |                       | 
|           |               | PortChannel100 | tagged         |                       | 
+-----------+---------------+----------------+----------------+-----------------------+ 
|       200 |               | PortChannel1   | tagged         | 10.10.10.100          | 
|           |               | PortChannel100 | tagged         | 10.10.10.101          | 
+-----------+---------------+----------------+----------------+-----------------------+ 
|       500 |               | PortChannel1   | tagged         |                       | 
|           |               | PortChannel101 | tagged         |                       | 
+-----------+---------------+----------------+----------------+-----------------------+ 
|      1234 | 1.1.1.0/31    | PortChannel1   | tagged         |                       | 
+-----------+---------------+----------------+----------------+-----------------------+ 
|      3800 |               | PortChannel1   | tagged         |                       | 
+-----------+---------------+----------------+----------------+-----------------------+ 
|      3965 | 10.10.10.0/31 | PortChannel1   | tagged         |                       | 
+-----------+---------------+----------------+----------------+-----------------------+

Now, if everything works as expect, i should be able to see remote mac addresses as well as /32 host routes in the routing table:

root@SONIC-Leaf301:/home/admin# show lldp table 
Capability codes: (R) Router, (B) Bridge, (O) Other
LocalPort    RemoteDevice    RemotePortID       Capability    RemotePortDescr
-----------  --------------  -----------------  ------------  -------------------------
Ethernet1    csp-srv-02      90:e2:ba:f6:cd:6d  O             Interface  13 as enp7s0f1
Ethernet9    MKTK-SW01       bond1.200          R
Ethernet48   SONIC-Leaf302   80:a2:35:81:e3:f0  BR            Ethernet48
Ethernet52   SONIC-Leaf302   80:a2:35:81:e3:f0  BR            Ethernet52
Ethernet72   SONIC-Spine31   80:a2:35:f2:7f:94  BR            Ethernet120
Ethernet76   SONIC-Spine32   80:a2:35:f2:80:c0  BR            Ethernet120
eth0         c6500-vxlan     Gi3/36             BR            GigabitEthernet3/36
--------------------------------------------------
Total entries displayed:  7

root@SONIC-Leaf301:/home/admin# show mac
  No.    Vlan  MacAddress         Port                     Type
-----  ------  -----------------  -----------------------  -------
    1     100  B8:69:F4:99:D1:4A  PortChannel100           Dynamic
    2     100  B8:69:F4:99:D1:4C  VxLAN DIP: 11.11.11.113  Dynamic
    3     200  B8:69:F4:99:D1:4A  PortChannel100           Dynamic
    4     500  02:5C:1F:02:1F:11  VxLAN DIP: 11.11.11.113  Dynamic
    5     500  02:5C:1F:02:20:10  PortChannel101           Dynamic
    6    1234  80:A2:35:81:E3:F0  PortChannel1             Static
    7    3965  80:A2:35:81:E3:F0  PortChannel1             Static
Total number of entries 7 

Remote VTEP IP al invalid format
root@SONIC-Leaf301:/home/admin# show vxlan evpn_remote_mac all
+---------+-------------------+--------------+---------+---------+
| VLAN    | MAC               | RemoteVTEP   |     VNI | Type    |
+=========+===================+==============+=========+=========+
| Vlan100 | b8:69:f4:99:d1:4c | 11.11.11.113 | 1000100 | dynamic |
+---------+-------------------+--------------+---------+---------+
| Vlan500 | 02:5c:1f:02:1f:11 | 11.11.11.113 | 1000500 | dynamic |
+---------+-------------------+--------------+---------+---------+
Total count : 2

root@SONIC-Leaf301:/home/admin# show ip route vrf Vrf-prod
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued route, r - rejected route, # - not installed in hardware
VRF Vrf-prod:
C>* 1.1.1.0/31 is directly connected, Vlan1234, 02:05:54
B>* 1.1.1.2/31 [200/0] via 11.11.11.113, Vlan3800 onlink, 02:27:22
C>* 10.10.10.0/24 is directly connected, sag500.256, 01:12:29
B>* 10.10.10.100/32 [200/0] via 11.11.11.113, Vlan3800 onlink, 02:27:22
B>* 10.10.30.0/24 [200/0] via 11.11.11.113, Vlan3800 onlink, 02:27:22
B>* 10.10.30.151/32 [200/0] via 11.11.11.113, Vlan3800 onlink, 02:27:22
C>* 10.10.200.0/24 is directly connected, sag200.256, 01:12:29
C>* 100.100.100.1/32 is directly connected, Loopback100, 02:13:08
B>* 100.100.100.2/32 [200/0] via 1.1.1.1, Vlan1234, 02:05:52
B>* 100.100.100.3/32 [200/0] via 11.11.11.113, Vlan3800 onlink, 02:27:22
B>* 100.100.100.4/32 [200/0] via 11.11.11.113, Vlan3800 onlink, 02:27:22

And of course, let’s not forget the real test, data-plane testing from the servers themselves.

DHCP:

[admin@MKTK-SW01] > /ip dhcp-client print 
Flags: X - disabled, I - invalid, D - dynamic 
 #   INTERFACE    USE-PEER-DNS ADD-DEFAULT-ROUTE STATUS        ADDRESS           
 0   bond1.200    yes          yes               bound         10.10.200.149/24  
 1   bond2.300    yes          yes               bound         10.10.30.151/24

Bridging:

[admin@MKTK-SW01] > ping 10.10.100.20 routing-table=HOST1 interface=bond1.100
  SEQ HOST                                     SIZE TTL TIME  STATUS  
    0 10.10.100.20                               56  64 0ms  
    1 10.10.100.20                               56  64 0ms  
    2 10.10.100.20                               56  64 0ms  
    3 10.10.100.20                               56  64 0ms  
    4 10.10.100.20                               56  64 0ms  
    sent=5 received=5 packet-loss=0% min-rtt=0ms avg-rtt=0ms max-rtt=0ms 

[admin@MKTK-SW01] > tool traceroute 10.10.100.20 routing-table=HOST1 interface=bond1.100 
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS
 1 10.10.100.20                       0%    5   0.1ms     0.1     0.1     0.1       0

Routing:

[admin@MKTK-SW01] > ping 10.10.30.151 routing-table=HOST1 interface=bond1.200 
  SEQ HOST                                     SIZE TTL TIME  STATUS 
    0 10.10.30.151                               56  64 0ms  
    1 10.10.30.151                               56  64 0ms  
    2 10.10.30.151                               56  64 0ms  
    3 10.10.30.151                               56  64 0ms  
    4 10.10.30.151                               56  64 0ms  
    sent=5 received=5 packet-loss=0% min-rtt=0ms avg-rtt=0ms max-rtt=0ms 

[admin@MKTK-SW01] > tool traceroute 10.10.30.151 routing-table=HOST1 interface=bond1.200 
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS
 1 10.10.200.1                        0%    5   0.2ms     0.3     0.2     0.3       0
 2 1.1.1.3                            0%    5   0.2ms     0.2     0.2     0.3       0
 3 10.10.30.151                       0%    5   0.1ms     0.1     0.1     0.1       0

In Conclusion…

With what we have seen so far, i really believe that SONiC is mature enough to cover most of the common DC network requirements. Notice that differently than other vendor’s solution that believe can do everything, including make you coffee or take you to the moon; SONiC is more specialised, does a few things and does them very well.
As long as what you need to do is supported by SONiC, then go ahead, it isn’t going to disappoint you.

An enterprise that considers to run SONiC should also understand the support model.
SONiC itself comes without support, and really, here we are looking at a typical open-source situation where you can choose to operate a software completely free of charge on your own, or you could pay a reputable company to provide you with a patched and supported software revision (a bit like Red Hat or SUSE Linux).

From an hardware standpoint, i think that the white boxes are mature enough. For example, Edge-Core’s AS7326-56X is basically identical to Juniper’s QFX 5120 (including port groups).
We are in the same world as your servers really. You can get your hardware from any vendor or you can find a trusted one like Dell. It’s up to you really..

In short then, what are the take away for “standard” enterprises?

  1. SONiC will work great if what you need to do fits the supported features
  2. White Box switches are comparable or identical to big vendor’s hardware
  3. You REALLY should be looking at someone to provide you with end to end support though. Maybe someone like Broadcom or other Service Providers to ensure you can get a single point of contact for all of your possible problems
  4. The knowledge gap can be scary at first, but it’s no longer a big obstacle. ACI for example was a nightmare and took me forever to learn and understand, SONiC on the other end was a piece of cake.
  5. Try and Experiment, open networking is so cheap that it costs almost nothing to bring up a small lab or even a production POC.

SONiC and White Box switches in the Enterprise DC! – Part 2

Posted on by
5

As discussed during our part 1, we are trying to configure a VXLAN-EVPN fabric using SONiC on white box switches in order to determine if Open Networking is ready to be deployed in most enterprise DCs.

As a small Recap, below is the topology we are trying to bring online:

Familiarise with the OS

The most interesting thing of SONiC is its architecture!
I’ll write a blog just about it because it’s a fascinating topic, but in short, every single process is living inside a dedicated container.

Linux SONIC-Leaf301 4.9.0-11-2-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64
You are on
  ____   ___  _   _ _  ____
 / ___| / _ \| \ | (_)/ ___|
 \___ \| | | |  \| | | |
  ___) | |_| | |\  | | |___
 |____/ \___/|_| \_|_|\____|

-- Software for Open Networking in the Cloud --

Unauthorized access and/or use are prohibited.
All access and/or use are subject to monitoring.

Help:    http://azure.github.io/SONiC/

Last login: Thu Apr 20 12:52:21 2017 from 192.168.0.31
admin@SONIC-Leaf301:~$ show version 

SONiC Software Version: SONiC-OS-3.0.1-Enterprise_Advanced
Product: Enterprise Advanced SONiC OS - Powered by Broadcom
Distribution: Debian 9.12
Kernel: 4.9.0-11-2-amd64
Build commit: aff85dcf1
Build date: Mon Apr 20 09:48:32 UTC 2020
Built by: sonicbld@sonic-lvn-csg-004

Platform: x86_64-accton_as7326_56x-r0
HwSKU: Accton-AS7326-56X
ASIC: broadcom
Serial Number: 732656X1916020
Uptime: 13:04:04 up 12 days, 29 min,  1 user,  load average: 2.14, 2.46, 2.42

Docker images:
REPOSITORY                       TAG                         IMAGE ID            SIZE
docker-sonic-telemetry           3.0.1-Enterprise_Advanced   15838a6dd8b6        397MB
docker-sonic-telemetry           latest                      15838a6dd8b6        397MB
docker-sonic-mgmt-framework      3.0.1-Enterprise_Advanced   c30542f39b9c        445MB
docker-sonic-mgmt-framework      latest                      c30542f39b9c        445MB
docker-swss-brcm-ent-advanced    3.0.1-Enterprise_Advanced   7db851611618        338MB
docker-swss-brcm-ent-advanced    latest                      7db851611618        338MB
docker-broadview                 3.0.1-Enterprise_Advanced   547641c0c886        330MB
docker-broadview                 latest                      547641c0c886        330MB
docker-nat                       3.0.1-Enterprise_Advanced   b819906b8bb6        320MB
docker-nat                       latest                      b819906b8bb6        320MB
docker-vrrp                      3.0.1-Enterprise_Advanced   2f0615d57ea4        333MB
docker-vrrp                      latest                      2f0615d57ea4        333MB
docker-teamd                     3.0.1-Enterprise_Advanced   3487700dc8d2        318MB
docker-teamd                     latest                      3487700dc8d2        318MB
docker-fpm-frr                   3.0.1-Enterprise_Advanced   bf6e7649147e        367MB
docker-fpm-frr                   latest                      bf6e7649147e        367MB
docker-iccpd                     3.0.1-Enterprise_Advanced   1c24858c993b        320MB
docker-iccpd                     latest                      1c24858c993b        320MB
docker-l2mcd                     3.0.1-Enterprise_Advanced   b0f6db69227b        319MB
docker-l2mcd                     latest                      b0f6db69227b        319MB
docker-stp                       3.0.1-Enterprise_Advanced   c812baaadda5        316MB
docker-stp                       latest                      c812baaadda5        316MB
docker-udld                      3.0.1-Enterprise_Advanced   66c2afbe849a        316MB
docker-udld                      latest                      66c2afbe849a        316MB
docker-sflow                     3.0.1-Enterprise_Advanced   9cf4e8a00ff9        318MB
docker-sflow                     latest                      9cf4e8a00ff9        318MB
docker-dhcp-relay                3.0.1-Enterprise_Advanced   5217cd436c40        326MB
docker-dhcp-relay                latest                      5217cd436c40        326MB
docker-syncd-brcm-ent-advanced   3.0.1-Enterprise_Advanced   800a3fc3af8b        439MB
docker-syncd-brcm-ent-advanced   latest                      800a3fc3af8b        439MB
docker-lldp-sv2                  3.0.1-Enterprise_Advanced   3a2e52d444f9        309MB
docker-lldp-sv2                  latest                      3a2e52d444f9        309MB
docker-snmp-sv2                  3.0.1-Enterprise_Advanced   d5a8e1d0ba7d        342MB
docker-snmp-sv2                  latest                      d5a8e1d0ba7d        342MB
docker-tam                       3.0.1-Enterprise_Advanced   272eabe18352        361MB
docker-tam                       latest                      272eabe18352        361MB
docker-pde                       3.0.1-Enterprise_Advanced   6ff2567c42b8        495MB
docker-pde                       latest                      6ff2567c42b8        495MB
docker-platform-monitor          3.0.1-Enterprise_Advanced   0b22d6abcd9a        367MB
docker-platform-monitor          latest                      0b22d6abcd9a        367MB
docker-router-advertiser         3.0.1-Enterprise_Advanced   9d201b15eae3        288MB
docker-router-advertiser         latest                      9d201b15eae3        288MB
docker-database                  3.0.1-Enterprise_Advanced   fb46e0661772        288MB
docker-database                  latest                      fb46e0661772        288MB

Below is the list of interfaces on my leaf. Notice how the naming of such interfaces can be confusing, specifically for the ones that can be channelised (like 40/100Gbps interfaces which support breakout). The primary channel is used as the interface number like with interface Ethernet48. In case an interface is then broken out, the other channels will be listed as Ethernet49, 50 and 51, making the next physical interface Ethernet52.
Interface Aliases are really interesting, unfortunately they currently act more like a description and even switching to “alias” as the default interface name, such alias is used in very few places making it pretty much useless as of now.

admin@SONIC-Leaf301:~$ show interfaces status 
  Interface            Lanes    Speed    MTU             Alias    Vlan    Oper    Admin             Type    Asym PFC
-----------  ---------------  -------  -----  ----------------  ------  ------  -------  ---------------  ----------
  Ethernet0                3      25G   9100   twentyFiveGigE1  routed    down     down   SFP/SFP+/SFP28         N/A
  Ethernet1                2      25G   9100   twentyFiveGigE2  routed    down     down   SFP/SFP+/SFP28         N/A
  Ethernet2                4      25G   9100   twentyFiveGigE3  routed    down     down              N/A         N/A
  Ethernet3                8      25G   9100   twentyFiveGigE4  routed    down     down              N/A         N/A
  Ethernet4                7      25G   9100   twentyFiveGigE5  routed    down     down              N/A         N/A
  Ethernet5                1      25G   9100   twentyFiveGigE6  routed    down     down              N/A         N/A
  Ethernet6                5      25G   9100   twentyFiveGigE7  routed    down     down              N/A         N/A
  Ethernet7               16      25G   9100   twentyFiveGigE8  routed    down     down              N/A         N/A
  Ethernet8                6      25G   9100   twentyFiveGigE9  routed    down     down              N/A         N/A
  Ethernet9               14      25G   9100  twentyFiveGigE10  routed    down     down   SFP/SFP+/SFP28         N/A
 Ethernet10               13      25G   9100  twentyFiveGigE11  routed    down     down              N/A         N/A
 Ethernet11               15      25G   9100  twentyFiveGigE12  routed    down     down              N/A         N/A
 Ethernet12               23      25G   9100  twentyFiveGigE13  routed    down     down              N/A         N/A
 Ethernet13               22      25G   9100  twentyFiveGigE14  routed    down     down              N/A         N/A
 Ethernet14               24      25G   9100  twentyFiveGigE15  routed    down     down              N/A         N/A
 Ethernet15               32      25G   9100  twentyFiveGigE16  routed    down     down              N/A         N/A
 Ethernet16               31      25G   9100  twentyFiveGigE17  routed    down     down              N/A         N/A
 Ethernet17               21      25G   9100  twentyFiveGigE18  routed    down     down              N/A         N/A
 Ethernet18               29      25G   9100  twentyFiveGigE19  routed    down     down              N/A         N/A
 Ethernet19               36      25G   9100  twentyFiveGigE20  routed    down     down              N/A         N/A
 Ethernet20               30      25G   9100  twentyFiveGigE21  routed    down     down              N/A         N/A
 Ethernet21               34      25G   9100  twentyFiveGigE22  routed    down     down              N/A         N/A
 Ethernet22               33      25G   9100  twentyFiveGigE23  routed    down     down              N/A         N/A
 Ethernet23               35      25G   9100  twentyFiveGigE24  routed    down     down              N/A         N/A
 Ethernet24               43      25G   9100  twentyFiveGigE25  routed    down     down              N/A         N/A
 Ethernet25               42      25G   9100  twentyFiveGigE26  routed    down     down              N/A         N/A
 Ethernet26               44      25G   9100  twentyFiveGigE27  routed    down     down              N/A         N/A
 Ethernet27               52      25G   9100  twentyFiveGigE28  routed    down     down              N/A         N/A
 Ethernet28               51      25G   9100  twentyFiveGigE29  routed    down     down              N/A         N/A
 Ethernet29               41      25G   9100  twentyFiveGigE30  routed    down     down              N/A         N/A
 Ethernet30               49      25G   9100  twentyFiveGigE31  routed    down     down              N/A         N/A
 Ethernet31               60      25G   9100  twentyFiveGigE32  routed    down     down              N/A         N/A
 Ethernet32               50      25G   9100  twentyFiveGigE33  routed    down     down              N/A         N/A
 Ethernet33               58      25G   9100  twentyFiveGigE34  routed    down     down              N/A         N/A
 Ethernet34               57      25G   9100  twentyFiveGigE35  routed    down     down              N/A         N/A
 Ethernet35               59      25G   9100  twentyFiveGigE36  routed    down     down              N/A         N/A
 Ethernet36               62      25G   9100  twentyFiveGigE37  routed    down     down              N/A         N/A
 Ethernet37               63      25G   9100  twentyFiveGigE38  routed    down     down              N/A         N/A
 Ethernet38               64      25G   9100  twentyFiveGigE39  routed    down     down              N/A         N/A
 Ethernet39               65      25G   9100  twentyFiveGigE40  routed    down     down              N/A         N/A
 Ethernet40               66      25G   9100  twentyFiveGigE41  routed    down     down              N/A         N/A
 Ethernet41               61      25G   9100  twentyFiveGigE42  routed    down     down              N/A         N/A
 Ethernet42               68      25G   9100  twentyFiveGigE43  routed    down     down              N/A         N/A
 Ethernet43               69      25G   9100  twentyFiveGigE44  routed    down     down              N/A         N/A
 Ethernet44               67      25G   9100  twentyFiveGigE45  routed    down     down              N/A         N/A
 Ethernet45               71      25G   9100  twentyFiveGigE46  routed    down     down              N/A         N/A
 Ethernet46               72      25G   9100  twentyFiveGigE47  routed    down     down              N/A         N/A
 Ethernet47               70      25G   9100  twentyFiveGigE48  routed    down     down              N/A         N/A
 Ethernet48      77,78,79,80     100G   9100     hundredGigE49  routed    down     down  QSFP28 or later         N/A
 Ethernet52      85,86,87,88     100G   9100     hundredGigE50  routed    down     down  QSFP28 or later         N/A
 Ethernet56      93,94,95,96     100G   9100     hundredGigE51  routed    down     down              N/A         N/A
 Ethernet60     97,98,99,100     100G   9100     hundredGigE52  routed    down     down              N/A         N/A
 Ethernet64  105,106,107,108     100G   9100     hundredGigE53  routed    down     down              N/A         N/A
 Ethernet68  113,114,115,116     100G   9100     hundredGigE54  routed    down     down              N/A         N/A
 Ethernet72  121,122,123,124     100G   9100     hundredGigE55  routed    down     down  QSFP28 or later         N/A
 Ethernet76  125,126,127,128     100G   9100     hundredGigE56  routed    down     down  QSFP28 or later         N/A
 Ethernet80              129      10G   9100     mgmtTenGigE57  routed    down     down              N/A         N/A
 Ethernet81              128      10G   9100     mgmtTenGigE58  routed    down     down              N/A         N/

Configuring the Underlay Routing

I’m a big fan of automation and configuration simplicity. I strongly believe that if i can automate with “notepad” using blind copy/paste i have good templates for fancier automation. For this reason i really think that the use of unnumbered interfaces are a great way to configure spine/leaf interfaces.

First step then, is to configure all fabric interfaces with a proper MTU and ip unnumbered as follows in the example below. Please notice that this post isn’t meant to be a full configuration tutorial.

config loopback add Loopback0
config interface ip add Loopback0 10.0.0.1/32
 
config interface ip unnumbered add Ethernet120 Loopback0
config interface mtu Ethernet120 9216
config interface startup Ethernet120
 
... Repeat for all interfaces facing a leaf ...
config save -y

A leaf switch would be configured exactly the same way, but I need also to add a second loopback interface to be used as VTEP source interface. As this loopback will act as an MC-LAG Anycast IP, both leafs in MC-LAG will have the same exact IP on their loopback 0

config loopback add Loopback0
config interface ip add Loopback0 10.0.0.11/32
config loopback add Loopback1
config interface ip add Loopback1 11.11.11.111/32

config interface ip unnumbered add Ethernet72 Loopback0
config interface mtu Ethernet72 9216
config interface description Ethernet72 "LINK_TO_SPINE_1"
config interface startup Ethernet72
  
config interface ip unnumbered add Ethernet76 Loopback0
config interface mtu Ethernet76 9216
config interface description Ethernet76 "LINK_TO_SPINE_2"
config interface startup Ethernet76
config save -y

At this point we need to configure OSPF between leafs and spines.
Unfortunately, advanced routing configs can only be applied inside the FRR container, so we need to switch to the FRR shell using the command “vtysh” first. From there on, really there is almost no difference with the well known cisco-like cli.

The biggest downside of this lack of integration is that FRR config needs to be saved separately from the rest of SONiC’s config, and we also need to tell SONiC to look at the routing config in a different place from the rest. To do that, we also need to apply the “config routing_config_mode split” command and most importantly, you need to reboot the box as the warning message will tell you. Failure to do so, will cause the switch to loose the FRR config in case of reload.

vtysh
 conf t
  !
  bfd
  !
  router ospf
   ospf router-id 10.0.0.11
   log-adjacency-changes
   auto-cost reference-bandwidth 100000
  !
  interface Ethernet72
   ip ospf area 0.0.0.1
   ip ospf bfd
   ip ospf network point-to-point
  !
  interface Ethernet76
   ip ospf area 0.0.0.1
   ip ospf bfd
   ip ospf network point-to-point
  !
  interface Loopback0
   ip ospf area 0.0.0.1
  !
  interface Loopback1
   ip ospf area 0.0.0.1
  end
 write memory
 exit
config routing_config_mode split
config save -y

Once everything is configured, from FRR we can check our routing:

SONIC-Leaf301# show ip ospf neighbor 

Neighbor ID     Pri State           Dead Time Address         Interface            RXmtL RqstL DBsmL
10.0.0.1          1 Full/DROther      33.775s 10.0.0.1        Ethernet72:10.0.0.11     0     0     0
10.0.0.2          1 Full/DROther      33.968s 10.0.0.2        Ethernet76:10.0.0.11     0     0     0

SONIC-Leaf301# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued route, r - rejected route, # - not installed in hardware

O>*  10.0.0.1/32 [110/11] via 10.0.0.1, Ethernet72 onlink, 00:03:08
O>*  10.0.0.2/32 [110/11] via 10.0.0.2, Ethernet76 onlink, 00:03:18
C *  10.0.0.11/32 is directly connected, Ethernet76, 00:05:08
C *  10.0.0.11/32 is directly connected, Ethernet72, 00:05:08
O    10.0.0.11/32 [110/10] via 0.0.0.0, Loopback0 onlink, 00:05:14
C>*  10.0.0.11/32 is directly connected, Loopback0, 00:05:15
O>*  10.0.0.12/32 [110/12] via 10.0.0.1, Ethernet72 onlink, 00:03:08
  *                        via 10.0.0.2, Ethernet76 onlink, 00:03:08
O>*  10.0.0.13/32 [110/12] via 10.0.0.1, Ethernet72 onlink, 00:03:08
  *                        via 10.0.0.2, Ethernet76 onlink, 00:03:08
O>*  10.0.0.14/32 [110/12] via 10.0.0.1, Ethernet72 onlink, 00:03:08
  *                        via 10.0.0.2, Ethernet76 onlink, 00:03:08
O>*  10.10.10.2/31 [110/12] via 10.0.0.1, Ethernet72 onlink, 00:03:08
  *                         via 10.0.0.2, Ethernet76 onlink, 00:03:08
O    11.11.11.111/32 [110/10] via 0.0.0.0, Loopback1 onlink, 00:05:14
C>*  11.11.11.111/32 is directly connected, Loopback1, 00:05:15
O>*  11.11.11.113/32 [110/12] via 10.0.0.1, Ethernet72 onlink, 00:03:08
  *                           via 10.0.0.2, Ethernet76 onlink, 00:03:08

SONIC-Leaf301# ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=0.243 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=0.186 ms
^C
--- 10.0.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1011ms
rtt min/avg/max/mdev = 0.186/0.214/0.243/0.032 ms
SONIC-Leaf301# ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=0.215 ms
^C
--- 10.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.215/0.215/0.215/0.000 ms

admin@SONIC-Leaf301:~$ traceroute 10.0.0.13
traceroute to 10.0.0.13 (10.0.0.13), 30 hops max, 60 byte packets
1 10.0.0.1 (10.0.0.1) 0.218 ms 10.0.0.2 (10.0.0.2) 0.178 ms 10.0.0.1 (10.0.0.1) 0.126 ms
2 10.0.0.13 (10.0.0.13) 0.439 ms 0.461 ms 0.468 ms

Now that every loopback is reachable (and we see also ECMP across the two spines) is time to configure MC-LAG between or leafs as well as underlay routing across the peer-link. This step can be done successfully only now because the MC-LAG peer has to be reachable via the fabric.

config portchannel add PortChannel1       
config interface mtu PortChannel1 9216

config interface mtu Ethernet48 9216
config interface description Ethernet48 "Peer-link"
config interface startup Ethernet48

config interface mtu Ethernet52 9216
config interface description Ethernet52 "Peer-link"
config interface startup Ethernet52

config portchannel member add PortChannel1 Ethernet48
config portchannel member add PortChannel1 Ethernet52
config mclag add 1 10.0.0.11 10.0.0.12 PortChannel1
 
config vlan add 3965
config vlan member add 3965 PortChannel1
config mclag unique-ip add Vlan3965
config interface ip add Vlan3965 10.10.10.0/31

vtysh
 conf t
  !
  interface Vlan3965
   ip ospf area 0.0.0.1
  end
 write memory
 exit
config save -y

Once done, we should see our additional OSPF peer and a working MC-LAG cluster

admin@SONIC-Leaf301:~$ vtysh 

Hello, this is FRRouting (version 7.2-sonic).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

SONIC-Leaf301# show ip ospf neighbor 

Neighbor ID     Pri State           Dead Time Address         Interface            RXmtL RqstL DBsmL
10.0.0.1          1 Full/DROther      30.645s 10.0.0.1        Ethernet72:10.0.0.11     0     0     0
10.0.0.2          1 Full/DROther      30.899s 10.0.0.2        Ethernet76:10.0.0.11     0     0     0
10.0.0.12         1 Full/DR           36.717s 10.10.10.1      Vlan3965:10.10.10.0      0     0     0

SONIC-Leaf301# exit

admin@SONIC-Leaf301:~$ sonic-cli 
SONIC-Leaf301# show mclag brief
 
Domain ID            : 1
Role                 : active
Session Status       : up
Peer Link Status     : up
Source Address       : 10.0.0.11
Peer Address         : 10.0.0.12
Peer Link            : PortChannel1
Keepalive Interval   : 1 secs
Session Timeout      : 30 secs
System Mac           : 80:a2:35:81:dd:f0
 
 
Number of MLAG Interfaces:0

Everything works as expected, but we also faced yet another SONiC problem. We needed to configure interfaces and their IP addresses, OSPF and MC-LAG, to do this, we needed access to 3 different configuration shells (Linux CLI, VTYSH and sonic-cli) either for configuration or to be able to run show commands and verify our configs.

Configuring BGP-EVPN control plane

Now it’s time to configure BGP. As per our architecture, I’ll be configuring iBGP with Route Reflectors sitting on the Spines. To do so i’ll need FRR shell.
The spine config will look something similar to this one:

vtysh
 conf t
  !
  router bgp 65000
   bgp router-id 10.0.0.1
   bgp log-neighbor-changes
   neighbor FABRIC peer-group
   neighbor FABRIC remote-as 65000
   neighbor FABRIC update-source Loopback0
   bgp listen range 10.0.0.0/24 peer-group FABRIC
  !
  address-family l2vpn evpn
   neighbor FABRIC activate
   neighbor FABRIC route-reflector-client
   advertise-all-vni
   exit-address-family
  end
 exit

And the leafs, like this:

vtysh
 conf t
  !
  router bgp 65000
   bgp router-id 10.0.0.11
   bgp log-neighbor-changes
   neighbor 10.0.0.1 remote-as 65000
   neighbor 10.0.0.1 update-source Loopback0
   neighbor 10.0.0.2 remote-as 65000
   neighbor 10.0.0.2 update-source Loopback0
  !
  address-family l2vpn evpn
   neighbor 10.0.0.1 activate
   neighbor 10.0.0.2 activate
   advertise-all-vni
   advertise ipv4 unicast
  exit-address-family
  end
 exit

Once done, I should be able to see all peering formed on my spines:

SONIC-Spine31# show bgp l2vpn evpn summary
BGP router identifier 10.0.0.1, local AS number 65000 vrf-id 0
BGP table version 0
RIB entries 16, using 3072 bytes of memory
Peers 4, using 82 KiB of memory
Peer groups 1, using 64 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
*10.0.0.11 4 65000 5 29 0 0 0 00:01:33 0
*10.0.0.12 4 65000 6 30 0 0 0 00:01:34 0
*10.0.0.13 4 65000 7 31 0 0 0 00:01:40 0
*10.0.0.14 4 65000 7 31 0 0 0 00:01:44 0
Total number of neighbors 4
- dynamic neighbor
4 dynamic neighbor(s), limit 100
Total number of neighbors established

At this point, the only missing piece is to configure the VTEP on the leaf switches as well as the anycast gateway’s mac-address; fortunately this is very simple and straight forward:

config vxlan add nve1 11.11.11.111
config vxlan evpn_nvo add nvo1 nve1
config ip anycast-mac-address add aa:aa:bb:bb:cc:cc
root@SONIC-Leaf301:/home/admin# show vxlan interface           
VTEP Information:

        VTEP Name : nve1, SIP  : 11.11.11.111
        NVO Name  : nvo1,  VTEP : nve1
        Source interface  : Loopback1

root@SONIC-Leaf301:/home/admin# show ip static-anycast-gateway 
Configured Anycast Gateway MAC address: aa:aa:bb:bb:cc:cc
IPv4 Anycast Gateway MAC address: enable

In short…

We configured a fully functional fabric providing underlay connectivity and EVPNcontrol plane as follows:

  1. A unique loopbacks on every switch
  2. Each physical interface between spine and leaf as an ip unnumbered interface
  3. OSPF area 1 within the fabric
  4. MC-LAG and underlay peering across the peer-link
  5. iBGP EVPN between leaf and spines with RR on the spines themeselves
  6. Each MC-LAG pair as a unique Virtual VTEP.

We also noticed how while the configuration isn’t complicated by any mean, the need to move between multiple shells just to apply or verify configs can be very confusing to the end user. To be fair though, the SONiC community is working on improving this part by working on delivering a single unified shell.

FRR config is always very familiar as it resembles Cisco’s IOS cli; on the other end the basic Sonic CLI can be a bit frustrating at times, especially due to the fact that it’s case sensitive making typos easy to occur.

In the next blog post we will look how to actually configure VXLANs and server facing interfaces… stay tuned!

SONiC and White Box switches in the Enterprise DC! – Part 1

Posted on by
8

In recent years two buzz words began to arise: open-networking and white box switches. Those two words go often hand-in-hand with each other. They are often promoted by big names like Facebook or Microsoft.
From the software side, SONiC is maybe the biggest player out there as it powers Microsoft Azure’s cloud, while from the hardware side, Accton has arguably been one of the most important vendors.

The truth though, at least in my opinion, is that while this innovation is great it is not ready to be embraced by everyone yet. Only companies willing to make this “leap of faith” can take advantage of all of this, but what about us poor mortals? Are SONiC and white boxes ready to be widely deployed? Well let’s give it a look!

We will be deploying a simple VXLAN-EVPN Fabric like in the picture below and we will be checking how difficult is to configure and troubleshoot the fabric, but also and most importantly if this common Enterprise design actually works.

The Hardware

For our spines we’ll be using Edge-Core’s AS7816-64X, powered by Broadcom’s Tomahawk II chipset. This switch is a 2RU lean spine providing 64x 40/100 Gbps QSF28 ports.

For the leafs, we’ll be using Edge-Core’s AS7326-56X, powered by Broadcom’s Trident III chipset. This switch is a 1RU TOR providing 48x 1/10/25 Gbps SFP28 and 8x 40/100 Gbps QSFP28 ports.

The Software

As for the software, we will be focusing on SONiC version 3.0.1.
This version introduces support to VXLAN-EVPN among many other things, that in my opinion, makes it ready for a more wide spread distribution.

The Architecture

Looking at SONiC’s features, we will try to implement the architecture below.
Some choices though, like the usage of a Virtual-VTEP as opposed of EVPN Multi-homing or ingress-replication for BUM traffic, are dictated only by SONiC support of such configurations.

SPINE/LEAF POD
│
├── ENDPOINT ACCESS
│   │
│   └── MCLAG with Virtual VTEP (all NLRI advertised with VIP as NH)
├── UNDERLAY
│   │
│   ├── Routing
│   │   │
│   │   └── OSPF
│   └── EVPN NLRI exchange
│       │
│       ├── iBGP
│       └── Route reflection
│           │
│           └── Spine, Fabric
└── OVERLAY
    │
    ├── L3 Gateway placement
    │   │
    │   └── Leafs
    ├── Distributed Anycast Gateway
    │   │
    │   └──  Same IP-MAC
    ├── Service Interface
    │   │
    │   └──  VLAN aware
    └── Host communication
        │
        ├── BUM traffic forwarding
        │   │
        │   └── Ingress Replication (EVPN Type 3)
        ├── Suppress ARP
        │
        └── Symmetric IRB
            │
            ├── Inter-Subnet
            │   │
            │   └── L3 VNI   
            └── Intra-Subnet
                │
                └── L2 VNI

I won’t explain why I’ve chosen OSPF+iBGP, this is a discussion for another time. It suffices to say that there is no reason to reinvent the wheel as this design worked perfectly for decades in the much more complex MPLS Service Provider’s space.

In short…

In this first post, I wanted to appeal to your curiosity, and set expectation right.
Accton switches powered by Boradcom chipset will be our white box switches while SONiC is our open source operating system.
In the next one instead, we will be implementing the above design looking at SONiC CLI and we’ll try to make it work.

Spoiler alert… It works but… well… details are a lot more interesting…