FlashAttention-3 unleashes the power of H100 GPUs for LLMs

Consideration is a core element of the transformer structure utilized in massive language fashions (LLMs). However as LLMs develop bigger and deal with longer enter sequences, the computational value of consideration turns into a bottleneck. 

To deal with this problem, researchers from Colfax Analysis, Meta, Nvidia, Georgia Tech, Princeton College, and Collectively AI have launched FlashAttention-3, a brand new approach that considerably hastens consideration computation on Nvidia Hopper GPUs (H100 and H800).

FlashAttention-3 builds upon earlier work on FlashAttention and FlashAttention-2 and additional optimizes using assets on Nvidia Hopper GPUs to maximise efficiency and effectivity for LLM coaching and inference. 

The problem of consideration computation in LLMs

One of many key improvements of transformers is the eye mechanism, which allows the mannequin to compute the connection between totally different tokens in an enter sequence.

Whereas the eye mechanism may be very efficient, it is usually computationally costly. The price of consideration computation grows quadratically with the size of the enter sequence. As LLMs are scaled to deal with longer and longer enter sequences, the eye mechanism turns into a serious bottleneck. 

Moreover, trendy {hardware} accelerators resembling GPUs are optimized for matrix multiplication (matmul) operations, that are the constructing blocks of deep studying fashions. These accelerators even have computational items for different varieties of operations resembling exponentiation, however these items are tons of of instances slower than the matmul parts.

Consideration computations use a mixture of matrix multiplications and different particular capabilities that aren’t as optimized for GPUs.

For instance, the softmax operate, which is used to normalize the eye weights, is computationally dearer than matrix multiplication. Because of this, despite the fact that matrix multiplications account for a lot of the computations in consideration, the general computation can get slowed down by a small variety of particular capabilities.

One of many vital elements of optimizing consideration computation is to schedule the workloads in a manner that operations don’t get blocked by one another and make environment friendly use of various kinds of reminiscence parts. 

Making higher use of {hardware} assets

FlashAttention, launched in 2022, addressed the challenges of computing consideration by lowering the variety of reminiscence reads and writes between GPU excessive bandwidth reminiscence (HBM) and GPU on-chip static random access memory (SRAM) when doing consideration computation. As a substitute of computing the eye weights for all the sequence without delay, FlashAttention breaks down the computation into smaller chunks, known as “tiles,” that may be processed extra effectively on GPUs.

FlashAttention has been broadly adopted and has contributed to growing the context window of LLMs from a number of thousand tokens to tons of of 1000’s and even hundreds of thousands of tokens. 

Nevertheless, as {hardware} has improved, so have the probabilities of optimizing LLM computations. FlashAttention-2, launched in 2023, additional optimized using GPU assets, reaching as much as 70% of the declared most efficiency on Nvidia A100 GPUs. Nevertheless, the identical optimizations didn’t switch to the newer H100 GPUs. FlashAttention-2 solely used 35% of H100’s most capability.

FlashAttention-3

FlashAttention-3 takes benefit of latest options in Nvidia Hopper GPUs to maximise efficiency. These options allow increased throughput on matrix multiplication operations, sooner information switch throughout totally different reminiscence segments, and higher effectivity on low-precision operations.

FlashAttention-3 introduces a number of improvements to enhance the efficiency of consideration computation on H100 GPUs.

FlashAttention-3 schedules operations in a manner that maximizes the overlap between computation and the motion of knowledge between totally different reminiscence segments of the GPU. This reduces the time the GPU spends idle ready for information to be transferred. It additionally interleaves the matrix multiplication and softmax operations to scale back the potential of bottlenecks in computing consideration values.

FlashAttention-3 additionally makes use of a particular association of operations for sooner and extra correct computations of consideration in quantized models. Quantization is a well-liked approach that reduces the dimensions of fashions by utilizing low-bit numbers to retailer their weights. The tradeoff of quantization is the potential lack of accuracy. FlashAttention-3 addresses this drawback by fastidiously arranging the computations to reduce the affect of quantization on accuracy.

In line with the researchers, FlashAttention-3 achieves as much as 75% utilization of the H100 GPU’s most capabilities. This interprets to a 1.5–2x speedup in comparison with earlier variations of FlashAttention for each coaching and operating LLMs.

The advantages of FlashAttention-3

The sooner consideration computation supplied by FlashAttention-3 has a number of implications for LLM improvement and purposes.

Coaching LLMs is a computationally costly course of that may take weeks and even months. The quick consideration computation supplied by FlashAttention-3 can considerably cut back the time it takes to coach LLMs, which might allow researchers and builders to experiment with bigger fashions and datasets.

FlashAttention-3 may also assist prolong the context window of LLMs by enabling them to course of longer sequences extra effectively. This will unlock new purposes for LLMs in areas resembling long-form doc understanding and many-shot in-context studying.

And by utilizing a better share of GPU capability, FlashAttention-3 can cut back the variety of accelerators required to run LLMs and slash the price of operating fashions in manufacturing.

The researchers have open-sourced FlashAttention-3 below a permissive license and plan to combine it into in style deep studying libraries resembling PyTorch and Hugging Face Transformers. It will make it simpler for researchers and builders to make the most of the efficiency advantages of FlashAttention-3.
“Now we have seen that designing algorithms that make the most of the {hardware} they run on can deliver vital effectivity positive factors and unlock new mannequin capabilities resembling lengthy context,” the researchers wrote in a weblog submit revealed by Together AI. “We look ahead to future work on optimization for LLM inference, in addition to generalizing our strategies to different {hardware} architectures.”