Article URL: https://transformer-transformer.github.io/ Comments URL: https://news.ycombinator.com/item?id=49093232 Points: 24 # Comments: 0

Not all robots are created equal — but what if you could design one for a specific task? We propose Transformer Transformer, a unified model that does exactly this: hand it a manipulation demonstration, and it generates a complete robot — every link, joint, motor, and inertial property — optimized for that motion. We fabricated one such design for cloth flinging on an ALOHA2 bimanual platform; it reduced tracking error by 73% and max joint speed by 30% versus the original. Behind that result is a diffusion transformer trained on RoboTokens, a unified tokenization of robot embodiments, states, and actions. The same architecture spans embodiment spaces (wheeled bimanual, quadrupeds, humanoids) and use cases (embodiment generation, cross-embodiment control). Rather than overfitting to one reward function, it is a dynamics model whose reward-agnostic predictions are converted into reward-specific value predictions at inference time, then used to steer embodiment diffusion through a procedure we call Dynamics Self-Guidance. Experiments across three design spaces show zero-shot optimization of unseen rewards and trajectories, improving performance and runtime over an evolutionary baseline. A robot's embodiment decides what tasks it can do well. We pour effort into making robots smarter — bigger datasets, better algorithms, more capable policy architectures — but a bad embodiment can hamper even a great policy. You can collect all the tossing data in the world, but if your robot's shape is far from optimal, it might toss itself instead of the ball. So let's frame the problem concretely. Given as input a target end-effector motion and a reward function, we want to generate a complete robot embodiment for the task — by complete we mean every link, joint, motor, and inertial property, plus a controller that drives the whole thing. We call this motion-conditioned robot co-design. A randomly procedurally-generated robot attempting the target motion: it falls before it can finish 🙃 Our framework reframes robot co-design as a three-step process: you demonstrate the desired end-effector motion (e.g., from a UMI3 human demonstration), our model generates an optimized embodiment, and the same model validates the design by directly controlling it. The same network plays three roles — generator, critic, and cross-embodiment controller — by training jointly on one unified tokenization of robots. If we want one model to co-design any robot, the network needs a shared vocabulary that spans every embodiment. RoboTokens are that vocabulary: typed tokens for every link, joint, motor, state, and action — organized so a single sequence describes both a robot's embodiment (time-invariant) and its dynamics (time-varying). We tokenized 11 robots from the MuJoCo Menagerie10, spanning two orders of magnitude in mass (0.65 kg dexterous hand to 67.5 kg quadruped) and from 6 to 35 active joints. Each became a sequence of 28–101 RoboTokens — small enough to diffuse end-to-end.