PADS·Explainer

Explainer: PADS from zero

No background assumed. If you know what a voice assistant is, you know enough to start. Technical words link to the glossary.

On this page

  1. Why voice assistants feel slow on a laptop
  2. What a language model actually does
  3. The shortcut: guessing early
  4. The pause: free time nobody uses
  5. Putting it together: what PADS does
  6. What we have actually proven
  7. What we have not proven

Why voice assistants feel slow on a laptop

Talk to a voice assistant on a powerful server and it answers quickly. Run the same kind of assistant on an ordinary laptop with no graphics card, and every thoughtful answer takes noticeably longer. The laptop is not broken — it is doing exactly what it was asked to do, just slowly. The question this project asks is: can we make it feel faster without changing the answers?

What a language model actually does

A language model does not write sentences. It predicts one word at a time. Each word is cut into pieces called tokens, each piece becomes a list of numbers, and those numbers pass through a stack of sixteen identical processing blocks called a transformer. At the end, every possible next word gets a probability, the winner is picked, and the whole pass runs again for the next word. One word, one full pass — that is why answers take time. On a laptop the pass is slow because moving data in memory, not arithmetic, is the bottleneck (see Technology for why).

Diagram: text becomes tokens, tokens pass through sixteen identical blocks, the output head scores every next word, and the chosen word loops back for the next pass.
One word costs one full pass through sixteen blocks. The dashed line after block eight is the shortcut the project is about — full diagrams on Architecture.

The shortcut: guessing early

Here is the key trick. If the model is trained the right way — a recipe called LayerSkip — then the first eight blocks already make a decent guess at the next word. So instead of always running all sixteen blocks, you can run eight, guess, and then run the remaining eight to check the guess. This is called self-speculative decoding: draft early, verify with the rest, on the same model.

The guarantee that makes this safe

If the check disagrees with the guess, the guess is thrown away and the checked answer is used. A thrown-away guess can never change the final answer. That guarantee is inherited from standard speculative decoding, and everything in this project depends on it.

Diagram: training with layer dropout and a shared output head; inference drafts at layer eight and verifies with layers nine to sixteen.
Training makes early layers accurate; inference drafts at layer 8 and verifies with layers 9–16 on the same saved computation.

The pause: free time nobody uses

When people talk, they pause briefly just before finishing a turn — typically a few hundred milliseconds of silence. Today's systems compute nothing during that silence: they wait for you to finish, then start thinking. We measured 29,297 real pauses and found the typical pause is about a second — roughly six times longer than one useful computing step. The pause is free wall-clock time, sitting unused.

Putting it together: what PADS does

PADS combines the shortcut with the pause. While you are still speaking, two small predictors watch your partial sentence: one estimates whether your turn is about to end, the other estimates whether your question will need deep thinking. Only when both are confident does the system start the deep computation inside the pause. Otherwise it stays shallow. When your turn actually ends, the early work is either confirmed — the answer is ready sooner — or discarded, which is always safe.

Diagram: data feeds training, training produces the model, evaluation judges it, and the inference loop runs it with a trigger built from two signals.
Every algorithm in the project and how it connects: data trains the model, evaluation judges it, and the inference loop runs it. Dashed arrows mean reuse.
Why a cautious gate

Being wrong costs a little wasted electricity; being right saves the user waiting. So the trigger fires only when both signals are confident — false alarms stay cheap and rare by design.

What we have actually proven

96.6%
of real pauses are long enough to fit one useful computing step (development host)
13.5 → 20.5%
early-exit success at half depth after our training, on data the model never trained on
+21.3 pp
improvement in the standard routing curve from training, at half the computation
0.859
accuracy of the lightweight dialogue classifier on real conversation data

All four are development-host measurements, not target-laptop results. Full provenance on Experiments.

What we have not proven