Why the network matters
A cluster of GPUs is only as useful as the network connecting them. Large training jobs exchange data between GPUs constantly, and a slow or congested fabric leaves expensive hardware waiting. For that reason, the network is a first-class part of the capacity decision, not an afterthought you sort out after the racks are installed.
InfiniBand in brief
InfiniBand has long been the default for large AI training clusters. It was designed for high bandwidth, low latency and efficient remote memory access, and the surrounding software and tooling are mature. Many reference architectures from GPU vendors assume it. The trade-offs are that it is a more specialized ecosystem, which can mean fewer vendors, different operational skills and, in some cases, higher cost.
Ethernet in brief
Ethernet is the common language of the data center, and it has been steadily adapted for AI workloads. Technologies that provide remote memory access over Ethernet, along with newer industry efforts to tune Ethernet for AI traffic, have made it a serious option, including at large scale. Its advantages are a broad ecosystem, familiar operations and often more flexible purchasing. Careful design matters, because an Ethernet fabric that is not built for AI traffic can underperform.
How to decide
- Workload. Large-scale training is the most sensitive to network performance. Many inference deployments are less demanding.
- Cluster size. The bigger the cluster, the more fabric design matters.
- Vendor guidance. Your hardware vendor's validated designs are a good starting point.
- Team skills. Choose a fabric your team, or your provider, can operate well.
- Provider support. What the facility offers today may decide the question for you.
What to ask a provider
Whichever fabric you choose, ask how it is built. Is the network non-blocking, or oversubscribed, and by how much? What is the topology? What link speeds are supported between nodes and between racks? Can you bring your own switches, or must you use theirs? How are cross-connects and external connectivity handled? Answers to these questions say far more than the single word InfiniBand or Ethernet on a listing.
Keep the option to change
Networking choices age quickly. Where possible, favor designs and contracts that allow upgrades, and confirm that the physical infrastructure, such as cable pathways and power for switching, can support the next generation.
Looking for InfiniBand-ready or Ethernet-based capacity? Tell us what you need.