Hey HN, Henry & Roman here from Cactus. A small, on-device model is fast and private, but sometimes wrong, but frontier models are getting expensive pretty fast. So, we post-traine…

A small, on-device model is fast and private, but sometimes wrong. At Cactus we post-train models to know when they are wrong: we ship probes inside the checkpoint that score every answer with a confidence between 0 and 1, returned as structured data (never parsed out of the answer text). Answer on-device when confidence is high; you can re-route to a bigger model when it's low: We start the rollout with Gemma 4 E2B Hybrid, all builds live in the Cactus Hybrid collection on Hugging Face. Gemma 4 E2B hybrid, the smallest Gemma model, matches Gemini 3.1 Flash-Lite on most benchmarks by routing only 15–35% of queries to the Gemini 3.1 Flash-Lite and running the remnant itself. Load the model with an explicit .to(device), not device_map="auto": the probe scores generations outside the module forward() path, so weights that accelerate offloads (left on the meta device) crash the confidence read. llama.cpp is C++, so the probe is a patch you compile into the engine (see patches/llama.cpp/). Build the patched server once: Then serve and query it like any llama-server — the response carries a top-level confidence field: Gemma 4 E2B Hybrid AUROC measures how well the the separates wrong answers from right ones (higher = better, 0.5 is random, 1.0 is perfect): The strongest result: the probe was trained on zero audio data, yet achieves 0.79–0.88 AUROC on four audio benchmarks (two transcription, one audio MCQ, one out-of-domain transcription).