Q-Omni: a model with no embedding table

Most language models spend a large slice of their parameters on two things that have nothing to do with language understanding: a giant lookup table mapping tokens to vectors, and weights precise enough to store sixteen bits of information each. Q-Omni is an experiment in refusing both — trained from scratch on a single consumer GPU.

Why spend nothing on the things you can compute for free

Ternary weights, trained that way from the first step

Q-Omni follows the BitNet b1.58 line of work: every attention and feed-forward matrix is quantized to {-1, 0, +1} on every forward pass during training, not compressed after the fact. Each row keeps one full-precision scale α (its mean absolute value); the forward pass computes clip(round(W/α), -1, 1) · α, with gradients flowing straight through the rounding step. The network only ever experiences the three-valued version of itself, so it never has to adapt to a distribution shift at deployment.

A vocabulary that costs nothing to store and nothing to train

Instead of a trained vocab_size × d_model embedding table, every token gets a fixed 512-bit code — a vector of ±1 values computed once, before training starts, from real co-occurrence statistics (PPMI-SVD, then thresholded to bits). Neither the input code nor the output code ever receives a gradient.

Arithmetic that is computed, not approximated

When Q-Omni needs to compute something exact, it writes a short structured span like <CALC>347*86<EQ>, and a deterministic logits processor intercepts that span, evaluates it exactly, and forces the correct digits into the output. The model's only job is recognizing when to reach for the circuit — fourteen deterministic sub-programs cover arithmetic, dates, weekdays, unit conversion, comparison, sorting, counting, and more. Across 300 held-out questions, it writes the correct span 99.7% of the time and the final answer is exact 98.7% of the time.

The architecture

Seven decoder blocks between a frozen input table and a frozen readout — everything inside stays ternary. The dots below trace one token's forward pass; watch it loop.

One vocabulary, three modalities

The same 512-bit fingerprint mechanism extends past text: 8,192 image codes (VQGAN) and 8,192 audio codes (four Mimi codebooks, frame-interleaved) live in the same id space as text — one model, one embedding mechanism, one readout head.

* after fine-tuning on the semantic Mimi codebook alone — see below.

Results, honestly

33 MB, and it runs in a browser tab

Ternary weights pack losslessly at 1.6 bits/weight; the frozen table packs at 1 bit/value. The whole model — architecture, weights, tokenizer, self-test — fits into one 33 MB file, a format we call QPACK1, down from 230–263 MB of fp32 safetensors.

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("q-project/Q-Omni-53M-Experimental")
model = AutoModelForCausalLM.from_pretrained(
    "q-project/Q-Omni-53M-Experimental", trust_remote_code=True)

ids = tok("Once upon a time,", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=60, do_sample=True, top_k=40)
print(tok.decode(out[0], skip_special_tokens=True))

What this adds up to

Q-Omni spends its parameters carefully: nothing on an embedding table, nothing on relearned arithmetic, and a measured, disclosed 18% quantization tax for the smallest possible footprint. The same frozen-vocabulary idea reaches across modalities, with one real bug found and fixed, and at least one — image generation — still open. Every number on this page comes from a checkpoint that was actually run, evaluated, and, where a claim could be checked twice, checked twice.