Article URL: https://github.com/Saivineeth147/lora-speedrun Comments URL: https://news.ycombinator.com/item?id=48974325 Points: 105 # Comments: 20

How fast can you LoRA-fine-tune Qwen2.5-1.5B to ≥ 57% on GSM8K — on a single L40S? This is modded-nanogpt for fine-tuning: a frozen task, frozen hardware, and a public leaderboard of wall-clock records. Every record is independently re-run 3× with fresh seeds on identical hardware before it counts. Attempting and verifying are free: official timing runs on a Modal L40S sandbox, and Modal's free monthly compute credits cover full runs — so anyone can compete, and anyone can re-verify any record with one command. Current record: 6m 05s by @Saivineeth147 — Sequence packing + completion-only loss masking, 2 epochs. Same LoRA config as #0; ~2x faster at higher accuracy. Current record: 11m 08s by @Saivineeth147 — Track 2 baseline: plain LoRA r=16 on SQuAD, first 20k examples, 1 epoch, full-sequence loss. No tricks. Two frozen tracks, deliberately different model families and task types — so a technique only proves general by winning on both. Same hardware, caps, and verification everywhere. You control everything else: LoRA rank and placement, quantization, learning-rate schedules, sequence packing, data subset selection and ordering, custom kernels, when to stop. Train on 1,000 well-chosen examples for 90 seconds if you can make it clear the bar. LoRA/QLoRA is how most real-world fine-tuning actually happens, and the technique space exploded — DoRA, rsLoRA, PiSSA, LoRA+, NEFTune, Unsloth kernels, rank-adaptive methods — but there's no adversarial, apples-to-apples arena where these ideas race each other in public. Papers report numbers on different models, data, and hardware; nothing is comparable.