The 1 Best Datacenter Network Design to Know

Datacenter Network Design Explained

Share This Post

In this article I want to prime you with datacenter network design, so that you know how a modern and scalable datacenter should be designed. I felt compelled to write this article because the first content you find online on datacenter network design is grossly outdated. In fact, even Cisco’s website presents you with a design that was fine in the 90s or early 2000s, but that is simply not good in a modern DC.

Instead, I use this article to explain why the spine-and-leaf architecture is the best datacenter network design, how it scales, and how it actually differs from the crappy multi-tiered architecture. Let’s dive into it.

Spine-and-Leaf Datacenter Network Design

How Spine-and-Leaf Architecture Works

The spine-and-leaf architecture is the golden standard of datacenter network design. This infrastructure is made up of L3-capable switches only, and each switch is assigned to a role: spine (also known as backbone), or leaf. You can have multiple switches in the leaf role, and multiple switches in the spine role.

Datacenter Network Design: Spine and Leaf
With the spine-and-leaf network design, you have the spines on top and the leaves at the bottom.

Explaining the two roles in this datacenter network design is simple:

  • A leaf switch connects hosts, such as servers and hypervisors, but also other types of hosts like an Internet connection
  • The spine switches provide connectivity between the leaf switches, they are designed to do one thing only: forward traffic fast

How many leaves and spines you have depends on how big your datacenter is. However, each spine that you add must connect to all leaves, and each leaf that you add should connect to all spines. You want to have at least two spine switches for redundancy, but you can have more to increase throughput.

The L2 domains are terminated on the leaves, which means there is no spanning tree running in this topology. Instead, all leaf-to-spine connection leverage an L3 routing protocol, such as OSPF or IS-IS. Furthermore, there is no spine-to-spine direct connection: spines connect only to leaves, and vice versa.

VLANs and Spine-and-Leaf Architecture

VLANs are an important concept in networking: they represent a virtual network. If two hosts or devices need to have the same IP address, or a similar IP address, they need to be in the same VLAN. You want a similar situation so that if a device fails for some reason, you can have another device take over its workload and perform in the exact same way, in the exact same VLAN.

The problem here is that VLANs need L2 propagation, and in this architecture we do not have that. A VLAN on a leaf switch cannot be propagated to another leaf switch because the L2 domain is truncated by the leaf switch itself. However, we want to have that VLAN on multiple leaf switches for the aforementioned redundancy reasons.

Instead of using only VLANs, this datacenter network design uses VXLANs, or VLANs that are virtualized to span across L3 boundaries. Normally, a VLAN is implemented by “tagging” each ethernet frame with the ID of the VLAN itself, and then this tagged frame is sent to the next switch that will understand it and place it in the proper VLAN. VXLANs working in a similar way, except they encapsulate the packets in an UDP segment which is then sent over the L3 network.

Of course, VXLANs are much more complex than normal VLANs, because an UDP segment must be sent to a specific switch/IP address, whereas a VLAN-tagged frame does not. To deliver a similar feature set that make VXLAN equivalent to VLANs we need to leverage multicast traffic in our spine network. Once VXLAN is configured, it will automatically send the right packets to the right switches.

With VXLAN, you can have much more than the traditional 4k VLANs, which is also great for scalability. Each VXLAN can be activated on any switch of the entire network, and this means having an extremely flexible network design.

Benefits of this Datacenter Network Design

Adopting a spine-and-leaf datacenter network design is good for you. First, it is extremely scalable and can start small. You can literally start with one leaf switch (a stack), and then add the spine and another leaf switch when you run out of capacity in the first switch.

This network design ensures consistent predictability of the path between any two hosts. You know that leaf-to-leaf communication will always pass through one spine. Performance, latency, and throughput will always be consistent, which is something highly desirable in a Datacenter. Because of this, any shared service will always connect to a leaf, just like a normal host. In legacy network designs, you have shared services connect to the core layer. Here, however, you don’t want to connect anything to the spine: only leaves. Your Internet link, load balancers, MPLS/WAN or any other shared service will go into a leaf, maybe a dedicated pair of leaves.

Because of the predictability in performance, you always avoid spine-to-spine physical connections or leaf-to-leaf physical connections. They would only take up some extra ports while adding no value nor improving performance.

Legacy Datacenter Network Design

Understand Multitiered Architecture

Now that we have introduced the spine-and-leaf datacenter network design, let’s spend a few words on the multitiered architecture. This is the architecture you will see in a campus network, not in a datacenter, and that is legacy and not recommended. It was the best practice in the 90s, and it is not something you want to do right now – not even in a campus to be honest.

Datacenter Network Design: Multitiered
With a multitiered datacenter, all connections up to the distribution are at L2, and the cores are connected directly with each other.

This architecture is simple. You have a pair of core switches, connected with each other, that connects downward to a set of distribution switches, each distribution switch serving a specific area of the datacenter. Then, each distribution switch connects downward to multiple access switches, that connect hosts. Communication between the core and distribution switches is L3 with routing protocols, while communication between access and distribution remains L2 with traditional VLANs.

If your datacenter is not as big, you can skip the distribution layer to have a “collapsed core”. In that case, your core will also act as distribution, and thus access switches will connect directly to it. As they do so, connection between the access and the core will be at L2, propagating the broadcast domain.

This means spanning tree run between switches at the access layer and switches at distribution or collapsed core. Spanning three ensures there are no loops in the L2 domain, which can cause the network to shut down. However, it is much slower to converge than a L3 routing protocol.

Why Multitiered Architecture is Bad

In this article on datacenter network design, I argued why the spine-and-leaf architecture is the best choices for any datacenter. But why the multitiered architecture is bad, and worse than the spine-and-leaf?

Most of that has to do with the fact that it relies on spanning-tree to work. Spanning tree is a slow-converging L2 protocol, and it cannot complete with the speed and reliability of routing protocols. Furthermore, routing protocol allows network engineers to influence path selection and balancing between links to a much better extent than spanning-tree does. In general, spanning tree is something you want to limit as much as possible if you want to have better performance in a DC.

The multitiered architecture uses traditional VLANs, and not VXLANs, which means it can have fewer VLANs (up to 4k) and that VLANs cannot span from a pod (a group of access switches and a distribution switch) to the next. This is not good if you want to create availability zones in your DC. Also, multitiered datacenter network design does not scale well, because of STP limitations.

Datacenter Network Design in Summary

In short, it does not really matter what your datacenter does or what is its purpose. The spine and leaf architecture is the one you need to use. It is the industry standard, and the one that brings more scalability to the table, a key concept for datacenter network design.

If you are interested in networking, we have a lot of articles on that. You should start with our free course here.

Picture of Alessandro Maggio

Alessandro Maggio

Project manager, critical-thinker, passionate about networking & coding. I believe that time is the most precious resource we have, and that technology can help us not to waste it. I founded with the same principle: I share what I learn so that you get value from it faster than I did.
Picture of Alessandro Maggio

Alessandro Maggio

Project manager, critical-thinker, passionate about networking & coding. I believe that time is the most precious resource we have, and that technology can help us not to waste it. I founded with the same principle: I share what I learn so that you get value from it faster than I did.

Join the Newsletter to Get Ahead

Revolutionary tips to get ahead with technology directly in your Inbox.

Alessandro Maggio