Nearly precisely a decade in the past I wrote that VXLAN isn’t an information middle interconnect know-how. That’s nonetheless true, however you may make it a bit higher with EVPN – on the very minimal you’ll get an ARP proxy and anycast gateway. Even this combo doesn’t tackle the opposite necessities I listed a decade in the past, however possibly I’m too demanding and adequate works properly sufficient.
Nonetheless, there may be one different bit that was lacking from most VXLAN implementations: LAN-to-WAN VXLAN-to-VXLAN bridging. Sounds bizarre? Supposedly an image is value a thousand phrases, so right here we go.
Most VXLAN-with-EVPN implementations can deal with a single unified bridging area – an ingress VTEP sends site visitors on to an egress VTEP.
Single VXLAN area stretched throughout a number of websites
That works properly in an information middle setting however would possibly lead to two challenges when used over WAN hyperlinks:
- You’re in all probability utilizing ingress replication (assuming you’re not an important fan of enabling large-scale IP multicast), which signifies that each ingress ToR change sends a separate copy of a flooded packet over the WAN hyperlink to each egress ToR change within the distant knowledge middle. Not precisely what you’d wish to see in your costly WAN hyperlink, proper?
- Switching ASICs assist a restricted variety of VXLAN neighbors (often 256) and a restricted variety of entries within the ingress replication record (often 128). You would possibly hit these limits when extending your VXLAN community throughout a number of websites
These challenges have an exquisite resolution: VXLAN-to-VXLAN bridging between LAN and WAN bridging domains on the WAN edge switches:
- WAN edge switches act as ultimate VXLAN VTEP for LAN and WAN friends. LAN friends don’t have to care about VTEPs in distant websites. WAN friends don’t have to care about native VTEPs.
- WAN edge switches obtain a single copy of a flooded packet (from LAN or WAN aspect) and flood it additional.
Hierarchical VXLAN with per-site bridging domains
For extra particulars, watch the superb Utilizing VXLAN and EVPN in Multi-Pod and Multi-Website Materials presentation by Lukas Krattiger, or learn the Multi-Area EVPN VXLAN doc on Arista’s site (warning: regwall).
There’s only a tiny little downside – the switching ASIC on the WAN edge units has to implement VXLAN-to-VXLAN bridging which incorporates:
- Break up-horizon forwarding: no matter is obtained from LAN friends shouldn’t be despatched to WAN friends and vice versa
- Break up-horizon flooding: no matter is obtained from LAN friends have to be flooded to WAN friends and vice versa.
- No dishonest with VXLAN VNI – identification of LAN and WAN friends have to be completed based mostly on supply IP addresses, not based mostly on completely different VNIs
For years, it seemed like the one ASIC able to doing VXLAN-to-VXLAN bridging was Cisco’s Cloud Scale ASIC… till Arista determined that’s an issue value fixing and found out easy methods to do it with Broadcom Jericho chipset. In line with the 2022 EANTC check report, the VXLAN-to-VXLAN stitching additionally works on Juniper QFX10K and Nokia 7750 SR-1.
Extra particulars
Thank You
Remi Locherer despatched me a pleasant electronic mail after the June 2022 design clinic saying “your info is a bit outdated” and included the hyperlink to 2022 EANTC check report and Arista documentation. I solemnly promise to reinforce these movies with I used to be fallacious callouts as soon as I get them again from the editor.
