A GPU Graphics Processing Unit. A processor designed to run many computations in parallel, used for tasks including training and running AI models. finishes a matrix multiplication and waits for the inputs to the next one. Its fastest units are ready, but another part of the program still needs to transform the results. Increasing matrix throughput does not remove that dependency.
FlashAttention-4 addresses this problem in the attention operation used by Transformers. Development continues with beta 33, published on September 30. The original paper dates to March 2026: this is an evolving software implementation, not an algorithm first announced in October. Beta 33, original paper.
Its significance goes beyond the version number. FA4 shows what happens when some accelerator units improve much faster than the resources preparing their work.
Between the matrix products the kernel must produce weights
Attention combines information from positions in a sequence. It first compares queries, Q, with keys, K. This matrix product produces scores. Softmax converts each row of scores into positive coefficients summing to one. A second matrix product uses those coefficients to combine the values, V.
That second product depends on softmax. Softmax must calculate exponentials and sums, then deliver usable coefficients. Tensor Cores, the specialized matrix engines, do not execute this entire chain on their own.
FlashAttention divides the work into blocks called tiles. It accumulates the output as tiles are processed, avoiding a write and subsequent read of the entire score matrix in external GPU memory. Reducing this traffic is the principle behind the original FlashAttention work. It does not eliminate the pairwise comparisons required by dense attention.
Blackwell shifts pressure onto other resources
In the team’s hardware analysis, BF16 matrix throughput per cycle per processing unit doubles from Hopper to B200, while exponential throughput stays the same. Shared-memory read throughput does not keep pace either. These resource ratios do not mean an entire kernel becomes twice as fast. Tri Dao’s analysis.
Consider a hypothetical tile needing 80 time units for matrix products, 30 for softmax and 20 for transfers. Sequential execution costs 130 units. Doubling only matrix throughput reduces the total to 90, a 1.44× program speedup.
With multiple tiles, operations can overlap. An ideal pipeline would run at the pace of its slowest stage. If matrix work falls to 20 units, the 30-unit softmax stage becomes the limit. This is an explanatory bound, not a benchmark: dependencies, buffer capacity and pipeline startup prevent an assumption of perfect overlap.
Keep tiles moving without overwriting live data
FA4 assigns different roles to groups of threads, the GPU’s execution lanes. Some prepare data, others calculate softmax, and others launch matrix operations. While one tile passes through softmax, another can occupy the matrix engines.
The inspected SM100 code reserves multiple regions for intermediate results. It uses Tensor Memory, internal storage associated with matrix execution on this architecture. Each region has a lifetime: a producer writes it, a consumer reads it, and only then may the storage be reused. A free execution unit cannot progress if its required buffer still holds data another operation needs. Version-pinned forward implementation.
The program can signal that an initial portion of the coefficients is ready before the remainder is finished. The consumer starts on that portion, with separate synchronization protecting the rest. This shortens the wait without permitting reads of incomplete data.
Keeping multiple tiles in flight consumes internal memory and registers, the small storage locations close to execution. More parallel work can eventually leave fewer resources for each group. Launching more threads alone does not resolve the tradeoff.
Move some exponential work to different units
FA4 also changes the work between matrix products. In the inspected softmax code, some exponentials can use an approximation built from ordinary multiplication and addition while others use the specialized instruction. That distributes work across different units. Softmax implementation, approximation functions.
The approximation adds instructions and temporary values. Its usefulness depends on contention for the specialized units and resources available elsewhere. Replacing every exponential this way would not automatically be optimal.
Tiled softmax also needs old and new contributions to share a consistent scale. Under a condition, the code avoids some rescaling operations. This reduces intermediate corrections while requiring coherent sums and an acceptable numerical range. Final normalization cannot repair a value that has already overflowed.
FP4 adds a numerical tradeoff
A separate September paper by Robert Hu explores four-bit computation. It extends the pipeline question: preparing and converting operands can consume part of the benefit of faster multiplication. It should not be attributed to the original FA4 author team or assumed to be integrated into the official beta. FP4 preprint.
Its variants have different measurement boundaries. A forward pass computes attention outputs; non-causal attention may use every sequence position. Speeding it up does not establish the same gain in text generation, where a causal mask restricts access to positions. Training experiments also need to track quality over time. A kernel’s best ratio is insufficient to predict a whole model’s speed.
Kernel gains still have to reach the application
The FA4 paper reports up to 1.3× over cuDNN 9.13 in BF16 on B200. That ceiling concerns the published attention tests. The consulted version contains an inconsistency: Appendix A.1 names B100 and March 2025, while the main text presents B200 and the paper dates to March 2026. We retain the result as an attributed claim without silently repairing its protocol. Paper and appendix.
A developer still needs to determine the fraction of total runtime occupied by this kernel. If attention accounts for 40% of runtime, doubling its speed reduces the total to 80%, a 1.25× overall speedup, assuming everything else stays unchanged. Other layers, communication and processing still take time.
FA4’s practical benefit therefore needs the intended model, sequence lengths and hardware. Verify the kernel actually selected by the runtime, then compare total duration and output quality. Hardware gains become useful when software can feed the fast units without moving all the waiting to the next stage.
Sources and method
Prepared on October 6, 2026, from the October 2 technical dossier, supplied code snapshots and targeted checks of the publications. LeCompute ran no GPU benchmarks. Pipeline timings and overall-speed examples are hypothetical calculations.
The FA4 paper and Tri Dao’s explanation establish the architectural problem. Beta 33 establishes the September 30 software milestone. Code discussion uses commit 843bf0b86bda1c92edc440ec58f9b5194609abac, not an assumed revision behind every historical measurement. Robert Hu’s FP4 paper is treated separately. SM100 observations are not transferred to RTX 50 GPUs.
