Article URL: https://blog.doubleword.ai/you-could-have-come-up-with-kimi-delta-attention Comments URL: https://news.ycombinator.com/item?id=49085909 Points: 253 # Comments: 107

A note on notation: this article defaults to bra-ket notation because (in my quantum-inspired opinion) it makes the shapes in this derivation very clear. The Math notation switch above rewrites every equation using conventional bold vectors and explicit transposes instead. In bra-ket mode, ∣q⟩\lvert q\rangle∣q⟩ is a column vector, ⟨k∣\langle k\rvert⟨k∣ is a row vector, ⟨k∣q⟩\langle k\rvert q\rangle⟨k∣q⟩ is a number, and ∣v⟩⟨k∣\lvert v\rangle\langle k\rvert∣v⟩⟨k∣ is a matrix. Vectors face right by default, while keys face left when written into the linear-attention state. We work with one causal attention head and real-valued vectors, assume DeltaNet’s keys are normalized, and let the state map from key space to value space. Modern linear attention variants are complex, and a upon first glance it is not so easy to see what they are designed to achieve. For reference here is the state update equation for Kimi Delta Attention (KDA): The reason they are so difficult to understand is that this is the latest in a family of linear attention variants that have been developed over the last few years and the complexity of them has inevitably ballooned such that from the outside the latest variants appear inaccessible. In this post we are going to walk through the DeltaNet family of linear attention variants, two of which are used by the latest Qwen and Kimi model families, and show how you might have arrived at the same equations by asserting simple things about your hidden state. Only after deriving KDA will we turn to the recurrent and chunkwise Triton programs that execute it. Every attention weight is a scalar. It measures the similarity between one key and one query, then softmax turns all of the scores for that query into a distribution. The output is a weighted sum of value vectors. Over a sequence of length TTT, there are T2T^2T2 key-query pairs. During autoregressive inference we can cache the keys and values instead of recomputing them, but the cache still grows with the sequence and every new query still has to inspect the entire history. The obstacle to rearranging this computation is the softmax. Its denominator depends jointly on the current query and every earlier key. So, for the moment, remove it.