Article URL: https://docs.pytorch.org/devlogs/compiler/2026-07-25-pytorch-a-reference-language/ Comments URL: https://news.ycombinator.com/item?id=49079454 Points: 18 # Comments: 1

A reference implementation is a simplified but complete version of a system that trades performance in return for clarity. We might then say a reference “language” is the fabric of APIs and conventions from which these implementations are cut. At first glance, PyTorch obviously is a reference language: it is, after all, commonly called the lingua franca of modern deep learning. But upon a closer look, there is confusion: Reference implementations usually aren’t deployed to production. But I do my training jobs with PyTorch! Everyone’s writing kernels with kernel DSLs. What is the role of PyTorch if it’s just gluing kernels together? AI coding will eventually mean that any stack can be rewritten from scratch; why does being written in PyTorch matter? So for me, recently, an unusually clarifying perspective has been to think of PyTorch as playing a dual role: as both the reference language and the implementation language. When the scale is not too large or the compiler is working well, the reference implementation can ship to production. But increasingly, I think it will be more and more natural to think of the reference implementation as a software artifact that stands apart from the actual production implementation, by which we can verify the correctness of the production implementation. One implementation to research in, one implementation to scale with, and one verifier to, in the darkness, bind them. The clearest demonstration of this is in the modern usage of kernel DSLs. The traditional, compiler-maximalist view argues that end users should write implementations of NN modules using a high level API (e.g., a Numpy/PyTorch-style API) which a compiler then determines how to compile into an optimized form. But for the most important operations like matrix multiplies and attention, it is not easy for compilers to guarantee peak performance; the proliferation of kernel DSLs has made it dramatically simpler for people to achieve optimal performance by explicitly spelling out tiling and data movement. Does this eliminate the high level API? Usually not: it’s pretty useful to have a reference implementation in plain PyTorch, and most kernel authors will maintain one in parallel with the optimized kernel, verifying correctness with numerical tests. In the same way kernel DSLs have changed how production implementations of operators can be written, I think coding agents change the way production implementations of train steps can be written. Traditionally, we think of autograd as a core part of PyTorch’s value proposition, because it guarantees you will get correct derivatives. However, at scale, the implicit backwards graph becomes an albatross around one’s neck: the majority of your compute is hidden away, with no opportunity to interact with it with normal debugging tools or apply fusions to it in the same way you can do it in eager forwards code. With a compiler, it is possible to modify the backwards graph with, e.g., a pattern match, but this is brittle and a less nice experience than just swapping a call from a reference implementation to a hand-written kernel. This is not a new observation: the now defunct Tangent library was built on the proposition that source-to-source automatic differentiation could be useful. The new recipe looks like this. Keep the traditional PyTorch autograd-friendly code as the reference implementation. Use LLMs to generate an explicit forward-backward version of the code, which can be optimized separately from the reference implementation. Unlike pattern matching, you never have to worry about your optimizations failing to apply. The cost is that the reference and the real implementation can diverge: we need a verifier that shows us they are equivalent. This verifier can be implemented simply with a bitwise equivalence test, or implemented as some sort of graph capture and structural equivalence, in the tradition of translation validation. To ensure the verifier works when one side has a fusion the other doesn’t, you only need to provide a reference implementation of the fusion (an inverse pattern match, if you will!).