PADS·Explainer

Experiments

What was measured, what it showed, and what is still unmeasured. Every row cites the artifact it comes from; development-host results are labelled.

Provenance

All numbers below were measured on the development laptop (Intel Core Ultra 7 155H, 15 GB RAM, RTX 4070 Laptop), not on the Dell Latitude 5490 the project targets. They are methodology and early-signal evidence, not target-hardware results.

On this page

  1. Go/No-Go falsification suite
  2. Multi-exit training and the leakage correction
  3. Exit-policy comparison (Phase 2)
  4. Baselines (Phase 3, measured)
  5. Offline voice-loop PoC
  6. Not measured

Go/No-Go falsification suite

Seven experiments were pre-registered with explicit kill conditions before implementation. Six have been executed on the development host; the energy-tooling check needs root and is pending.

#TestResultKey numbers (development host)
1Bandwidth vs computePartial — 7B did not fit host RAM (fully swapped); 3B mitigation branch executed3B Q4_1: 5.3 → 14.2 tok/s from 1 to 8 threads, saturating past 5–6 threads
2Real pause-duration feasibilityPASSAMI corpus, 29,297 silence pauses: median 1,050 ms; 96.6% fit a 164 ms extrapolated 7B step, 100% fit the measured 3B 70 ms step
3Off-the-shelf acceptance rateKill signal fired at aggressive exitsGreedy, exit layers 4/6/8/10/12: 6.4 / 9.4 / 10.2 / 14.1 / 56.1%
4Thermal stabilityThrottling observed177 samples over ~35 min: max 103 °C, 16.9% of samples ≥ 95 °C
5RAM budget3B fits comfortably; 7B does not fit this host3B Q4_1 peak RSS 2.553 GB against a 13 GB budget
6Energy toolingBlocked on rootturbostat/powertop/RAPL present; energy_uj unreadable without root; USB power meter is the pre-planned fallback
7Literature scoop checkPartial overlap, not scoopedTwo systems require citation (arXiv:2606.13450, arXiv:2607.23204); 32/32 arXiv identifiers verified, none fabricated

Multi-exit training and the leakage correction

The off-the-shelf checkpoint's acceptance collapses at shallow exits, so we continued-train it with the LayerSkip recipe — layer dropout plus a multi-exit loss — using LoRA adapters.

Correction, kept visible

The first training run appended every AMI meeting, including the three the evaluation set is drawn from: 40/40 evaluation prompts appeared verbatim in the training corpus. The reported gain was inflated. The model was retrained with a held-out split (leak check 0/40) and all numbers below are the corrected ones. The contaminated figures were +22.1 pp at exit 8; the honest figure is +7.0 pp.

Exit layer (of 16)Baseline (N=40)Held-out trained (N=34 non-empty)Empties as failures (N=40)
49.16%3.84%3.26%
612.08%19.34%16.44%
813.48%20.46%17.39%
1016.55%29.97%25.48%
1253.95%57.89%49.20%

Acceptance = fraction of drafted tokens the full-depth model accepts, greedy decoding, 40 AMI conversational turns. The trained model emits empty generations on 6/40 prompts (baseline 0/40) — an unresolved quality regression. Gate verdict: PARTIAL (exit 8 clears the 20% target by 0.5 pp; the conservative view misses it).

Exit-policy comparison (Phase 2)

Two policies decide how deep to compute: token confidence (the standard per-token criterion) and the dialogue-act policy (a turn-level decision from the trained classifier). They are compared at matched compute — equal fractions of positions routed to full depth — by agreement with the full-depth oracle.

Compute budgetToken confidenceDialogue actDifference
25% deep55.2%49.4%−5.8 pp
50% deep76.0%67.1%−8.9 pp
75% deep91.6%84.5%−7.2 pp
Verdict

Negative result for the dialogue-act exit policy: it is behind token confidence in every point estimate at every matched budget, and never beats it — the safe core keeps token confidence, and the dialogue-act signal remains only a candidate pause-trigger input. Under the assumption-free uncertainty convention the fixed-budget deficit is not statistically resolved at n=40; the direction is consistent across all three conventions (sensitivity table in the run directory).

The safe-core win

The same experiment shows the trained model's token-confidence curve beats the base model's by +21.25 pp at 50% deep (95% CI 18.45–24.15) and +12.82 pp at 75% deep — a convention-robust result, and the strongest evidence so far that multi-exit training genuinely improves early-exit quality on CPU-bound models.

Baselines (Phase 3, measured)

The PRD requires four baselines reproduced locally rather than cited from other papers. Three are measured on the development host; the fourth — two-model speculative decoding — is blocked, with the reason recorded and no numbers invented.

Plain decoding (llama.cpp, 5 repeats per prompt)

ConfigurationHardwareTokens/sPeak RSSPrompt eval (derived)
Plain decoding, llama.cpp, LayerSkip 1B Q4_K_MCPU, 8 threads46.18 ± 0.181.196 GB3.63 ms/token (152.26 ms total)
Plain decoding, llama.cpp, LayerSkip 1B Q4_K_MGPU, 8 threads207.50 ± 1.231.105 GB0.43 ms/token (18.21 ms total)

200 measured runs per arm (40 AMI prompts × 5 repeats; 2 warm-up runs discarded), 128-token generations, 0 empty generations in both arms. Intervals are 95% CIs over per-prompt means (n=40; per-run bases ±0.13 CPU / ±0.78 GPU). Prompt-eval latency is derived as prompt_tokens × 1000/prompt_tps from the llama-cli summary line — not a first-token timestamp. GPU VRAM peak 1778 MiB (Δ +1048 over a 730 MiB baseline). Energy is null — RAPL needs root.

Blocked arm — no numbers

Two-model speculative decoding is BLOCKED: llama.cpp requires draft and target to share a vocabulary, and no valid same-vocab draft pair exists locally — the only GGUF target is llama-arch (models/gguf/layerskip-1b-Q4_K_M.gguf) while the only small draft candidates are qwen2-family. Inventing numbers for this arm would be fabrication, so none are given.

Plain cascading (0.5B → 1B, HF driver)

A small model (Qwen2.5-0.5B-Instruct) drafts every prompt; a large model (LayerSkip 1B, bf16) re-generates only escalated prompts. Escalation is driven by the small model's mean token log-probability against a threshold — an uncalibrated heuristic. Greedy decoding, 40 AMI prompts, one run per prompt (per-prompt and per-run CIs coincide).

ThresholdEscalatedMean wallTokens/s
−1.085/40 (12.5%)1.90 s ± 0.2172.78 ± 4.59
−0.8515/40 (37.5%)2.41 s ± 0.3261.94 ± 6.68
−0.7027/40 (67.5%)2.98 s ± 0.3148.22 ± 6.80

Expected escalation/latency trade-off: 12.5% → 67.5% escalation as the threshold tightens, wall 1.90 → 2.98 s. Output sanity 120/120 verbatim. Peak host RSS 4.23 GB; CUDA peaks 950 MiB (small) / 3316 MiB (large).

LayerSkip published configuration (off-the-shelf, no adapter)

The author's self-speculative recipe (draft at the exit layer, verify with the full model, 6 speculations, 128 steps max) on the unmerged checkpoint, timed by benchmark.py's in-process GPU timers. N=40 held-out prompts, 0 empties at every exit. Throughput here is descriptive only — no warm-up discard — so it is not speedup evidence.

Exit layer (of 16)AcceptanceTokens/s (descriptive)
49.16%29.96
612.08%28.60
813.48%26.72
1016.55%26.46
1253.95%50.31

Values are the log/JSON values from eval_baseline_exit{L}.log and metrics_extracted.json, which agree exactly; a stale prose line in acceptance_heldout.md (31.14/30.25/26.45/25.68/48.98 tok/s) does not match the raw logs and is not used.

Do not compare across arms

Cross-arm tok/s is not head-to-head: three different timers produced these numbers — llama.cpp subprocess timing with warm-up discarded (plain decoding), HF generate() walls with one run per prompt (cascading), and in-process benchmark.py timers with no warm-up discard (LayerSkip config). Additionally, a sibling track's llama-server held ~2384 MiB of VRAM during the cascade run, so part of the 1B model was CPU-offloaded and large-stage cascade latency is pessimistic versus exclusive-GPU use.

Offline voice-loop PoC

A working end-to-end voice loop runs fully offline on the development host CPU (src/poc/loop.py): wav → STT → dialogue-act classifier → trigger → LLM → TTS → wav. Single scripted turn, no GPU touched.

StageWall timeNotes (development host)
STT (faster-whisper tiny.en int8)1.576 s6.18 s of audio, beam 1
Dialogue-act classifier1.389 sIncludes bundle load and first sklearn import
Trigger (AND-gate)0.0 sPure Python, sub-millisecond
LLM (1B Q4_K_M, CPU, 64 tokens max)10.234 sDominates the loop
TTS (Piper sentence-level)1.781 s230-char response → 11.74 s of audio
Total14.981 sSum of stages; process wall 15.70 s, peak RSS 1.34 GB

The decision record is honest about what is measured and what is scripted: p_deep = 0.7401 (measured, clears the 0.65 threshold) and p_end = 0.85 (scripted input parameter, clears 0.75), giving trigger_deep.

What is stubbed

Scripted turn-taking: p_end is passed on the CLI — no audio VAP model exists yet. Synthetic audio: the 6.18 s input is Piper-synthesized, so STT accuracy is clean-synthetic-speech only. Single turn: one-shot batch STT, one trigger decision, no streaming, no false-trigger rate. Response quality is unevaluated — only that text flows end to end into spoken audio. Full notes: experiments/runs/poc-smoke/POC_NOTES.md.

Not measured