The apparent reply can be Nvidia’s new GB200 programs, primarily one big 72-GPU server. However these value thousands and thousands, face excessive provide shortages, and aren’t accessible all over the place, the researchers famous. In the meantime, H100 and H200 programs are plentiful and comparatively low-cost.
The catch: operating giant fashions throughout a number of older programs has historically meant brutal efficiency penalties. “There are not any viable cross-provider options for LLM inference,” the analysis staff wrote, noting that current libraries both lack AWS assist totally or endure extreme efficiency degradation on Amazon’s {hardware}.
TransferEngine goals to alter that. “TransferEngine allows transportable point-to-point communication for contemporary LLM architectures, avoiding vendor lock-in whereas complementing collective libraries for cloud-native deployments,” the researchers wrote.
How TransferEngine works
TransferEngine acts as a common translator for GPU-to-GPU communication, in accordance with the paper. It creates a standard interface that works throughout completely different networking {hardware} by figuring out the core performance shared by numerous programs.
TransferEngine makes use of RDMA (Distant Direct Reminiscence Entry) expertise. This enables computer systems to switch information immediately between graphics playing cards with out involving the primary processor—consider it as a devoted categorical lane between chips.
Perplexity’s implementation achieved 400 gigabits per second throughput on each Nvidia ConnectX-7 and AWS EFA, matching current single-platform options. TransferEngine additionally helps utilizing a number of community playing cards per GPU, aggregating bandwidth for even quicker communication.
