Article URL: https://github.com/david-g-3654/homebench Comments URL: https://news.ycombinator.com/item?id=49166308 Points: 3 # Comments: 0

Benchmark the local LLMs you already have — speed, memory, and quality — as a live terminal leaderboard. homebench is a single-command TUI that discovers the models installed in your local runner (Ollama, LM Studio, llama.cpp, vLLM, or any OpenAI-compatible server), runs a curated quality suite, measures tokens/sec, time-to-first-token, and memory footprint on your actual machine, and renders a live comparison leaderboard. There are great tools for one half of this problem, but nothing local-first that does both: homebench fills the gap: local-first, zero-config, UX-driven. Clone-and-run, point it at the models you already pulled, and get an at-a-glance answer to "which of my local models is actually good, and how fast is it on this laptop?" Auto-detection tries Ollama → LM Studio → llama.cpp → vLLM (the generic openai provider is explicit-only). Force one with --provider. Override host with --host or the env var above. The suite is small on purpose — enough tasks across categories to separate models, few enough that every model runs in a couple of minutes on a laptop. Each task is graded deterministically (exact numeric match, multiple-choice letter, substring, valid-JSON, regex). Temperature is 0 and a fixed seed is used for reproducibility. See homebench tasks for the list. The optional --judge MODEL flag turns on an LLM-as-judge (any local model) that scores open-ended tasks 1–5 against a reference answer. It's a signal, not an oracle. Benchmarking every model on the full suite takes a while on a laptop, so the defaults are tuned for a quick first look: