Article URL: https://www.fermionresearch.com/models/neutrino-8b/ Comments URL: https://news.ycombinator.com/item?id=49079490 Points: 42 # Comments: 5

Three models in a proprietary ternary-family format, served by one engine from one artifact on datacenter GPUs, Apple silicon, and desktop CPUs. The runtime built alongside the format: single-stream decode from one artifact on every backend, with a drafted path verified token for token against plain decode. The flagship of the Neutrino family: 36 decoder layers behind a coded ternary-family container that serves a datacenter GPU, a MacBook, and a desktop CPU from one artifact. Neutrino-1 8B is an 8.19B-parameter decoder-only transformer that ships as one 3.88 GB file. Every one of its 252 transformer linears is stored in a proprietary ternary-family weight format eight times smaller than fp16; the weights stay bit-packed at rest and are decoded inside the matrix kernels, so nothing in the decode path is stored as fp16 or fp32 weight material. Small weights change the serving economics. Single-stream decode is bound by how many bytes move per token, so a 3.88 GB working set decodes at rates a 16 GB fp16 artifact cannot reach on the same memory system, and the whole model fits beside its KV cache on an 8 GB GPU or a 16 GB laptop. The same container serves every platform below without conversion. A dense decoder-only transformer. Grouped-query attention holds the KV cache at a quarter of the query width, 144 KiB per token at fp16, so a 4k-token session costs 0.60 GB of cache beside the 3.88 GB of weights. Only the transformer linears carry the coded format. The two embedding tensors stay int8 because their rows are read one token at a time, not multiplied against the full activation stream, and the normalization weights are too small to be worth coding. A third of the file is vocabulary. Across the 6.95B coded weights, 62.63% sit at zero and the remainder splits 18.68% plus to 18.69% minus: sign-balanced to a hundredth of a point with no constraint asking for it. The balance is not uniform in depth. The gate and down feed-forward projections spike to 70 to 72% zeros in layers 1 through 3 while all four attention projections hold within about one point of 62% at every depth: the early feed-forward blocks shed weights the network does not need, and attention keeps a constant code density from layer 0 to layer 35.