Article URL: https://github.com/gavamedia/deltafin Comments URL: https://news.ycombinator.com/item?id=49090233 Points: 26 # Comments: 11

Deltafin is a small research project that runs a Mixture-of-Experts model far larger than the machine it sits on. It is not fast — about 16 seconds per token on our M1 Max — but it is exact, reproducible, and it works on a 64 GB laptop. Newer chips and more RAM make it faster automatically. Every token reads 16 experts × 92 layers = 25.8 GB of expert data. From local disk that's about 4 seconds; over the network it's minutes. That single fact is the whole difference between the two columns. Run setup_k3.py with no flag and it picks --full when the disk allows, otherwise falls back to streaming and tells you exactly how much space you'd need to free. Streaming is a fine way to try Deltafin without committing 1.7 TB. Whenever you want the speed, one command finishes the job — no reinstall, no reconfiguration, and it picks up whatever is already cached: Deltafin prints a reminder at startup — both for the CLI and the API server — whenever it's still in streaming mode, showing how much of the pool is local and what finishing would cost. Halves per-token I/O for the non-expert weights, with no meaningful quality change in our checks. Takes a few minutes: Tokens print as they are generated, so you always see the text as it comes. Ctrl-C stops cleanly at any point and prints the result so far; --max-new N caps the length. One honest warning: K3 thinks before it answers, and at about a token per minute a full chat answer can take a while — watching it stream is part of the experience. Router selections are logged to router_trace.jsonl if you want to study K3's routing behaviour.