Experiments
What was measured, what it showed, and what is still unmeasured. Every row cites the artifact it comes from; development-host results are labelled.
All numbers below were measured on the development laptop (Intel Core Ultra 7 155H, 15 GB RAM, RTX 4070 Laptop), not on the Dell Latitude 5490 the project targets. They are methodology and early-signal evidence, not target-hardware results.
On this page
Go/No-Go falsification suite
Seven experiments were pre-registered with explicit kill conditions before implementation. Six have been executed on the development host; the energy-tooling check needs root and is pending.
| # | Test | Result | Key numbers (development host) |
|---|---|---|---|
| 1 | Bandwidth vs compute | Partial — 7B did not fit host RAM (fully swapped); 3B mitigation branch executed | 3B Q4_1: 5.3 → 14.2 tok/s from 1 to 8 threads, saturating past 5–6 threads |
| 2 | Real pause-duration feasibility | PASS | AMI corpus, 29,297 silence pauses: median 1,050 ms; 96.6% fit a 164 ms extrapolated 7B step, 100% fit the measured 3B 70 ms step |
| 3 | Off-the-shelf acceptance rate | Kill signal fired at aggressive exits | Greedy, exit layers 4/6/8/10/12: 6.4 / 9.4 / 10.2 / 14.1 / 56.1% |
| 4 | Thermal stability | Throttling observed | 177 samples over ~35 min: max 103 °C, 16.9% of samples ≥ 95 °C |
| 5 | RAM budget | 3B fits comfortably; 7B does not fit this host | 3B Q4_1 peak RSS 2.553 GB against a 13 GB budget |
| 6 | Energy tooling | Blocked on root | turbostat/powertop/RAPL present; energy_uj unreadable without root; USB power meter is the pre-planned fallback |
| 7 | Literature scoop check | Partial overlap, not scooped | Two systems require citation (arXiv:2606.13450, arXiv:2607.23204); 32/32 arXiv identifiers verified, none fabricated |
Multi-exit training and the leakage correction
The off-the-shelf checkpoint's acceptance collapses at shallow exits, so we continued-train it with the LayerSkip recipe — layer dropout plus a multi-exit loss — using LoRA adapters.
The first training run appended every AMI meeting, including the three the evaluation set is drawn from: 40/40 evaluation prompts appeared verbatim in the training corpus. The reported gain was inflated. The model was retrained with a held-out split (leak check 0/40) and all numbers below are the corrected ones. The contaminated figures were +22.1 pp at exit 8; the honest figure is +7.0 pp.
| Exit layer (of 16) | Baseline (N=40) | Held-out trained (N=34 non-empty) | Empties as failures (N=40) |
|---|---|---|---|
| 4 | 9.16% | 3.84% | 3.26% |
| 6 | 12.08% | 19.34% | 16.44% |
| 8 | 13.48% | 20.46% | 17.39% |
| 10 | 16.55% | 29.97% | 25.48% |
| 12 | 53.95% | 57.89% | 49.20% |
Acceptance = fraction of drafted tokens the full-depth model accepts, greedy decoding, 40 AMI conversational turns. The trained model emits empty generations on 6/40 prompts (baseline 0/40) — an unresolved quality regression. Gate verdict: PARTIAL (exit 8 clears the 20% target by 0.5 pp; the conservative view misses it).
Exit-policy comparison (Phase 2)
Two policies decide how deep to compute: token confidence (the standard per-token criterion) and the dialogue-act policy (a turn-level decision from the trained classifier). They are compared at matched compute — equal fractions of positions routed to full depth — by agreement with the full-depth oracle.
| Compute budget | Token confidence | Dialogue act | Difference |
|---|---|---|---|
| 25% deep | 55.2% | 49.4% | −5.8 pp |
| 50% deep | 76.0% | 67.1% | −8.9 pp |
| 75% deep | 91.6% | 84.5% | −7.2 pp |
Negative result for the dialogue-act exit policy: it is behind token confidence in every point estimate at every matched budget, and never beats it — the safe core keeps token confidence, and the dialogue-act signal remains only a candidate pause-trigger input. Under the assumption-free uncertainty convention the fixed-budget deficit is not statistically resolved at n=40; the direction is consistent across all three conventions (sensitivity table in the run directory).
The same experiment shows the trained model's token-confidence curve beats the base model's by +21.25 pp at 50% deep (95% CI 18.45–24.15) and +12.82 pp at 75% deep — a convention-robust result, and the strongest evidence so far that multi-exit training genuinely improves early-exit quality on CPU-bound models.
Baselines (Phase 3, measured)
The PRD requires four baselines reproduced locally rather than cited from other papers. Three are measured on the development host; the fourth — two-model speculative decoding — is blocked, with the reason recorded and no numbers invented.
Plain decoding (llama.cpp, 5 repeats per prompt)
| Configuration | Hardware | Tokens/s | Peak RSS | Prompt eval (derived) |
|---|---|---|---|---|
| Plain decoding, llama.cpp, LayerSkip 1B Q4_K_M | CPU, 8 threads | 46.18 ± 0.18 | 1.196 GB | 3.63 ms/token (152.26 ms total) |
| Plain decoding, llama.cpp, LayerSkip 1B Q4_K_M | GPU, 8 threads | 207.50 ± 1.23 | 1.105 GB | 0.43 ms/token (18.21 ms total) |
200 measured runs per arm (40 AMI prompts × 5 repeats; 2 warm-up runs discarded), 128-token generations, 0 empty generations in both arms. Intervals are 95% CIs over per-prompt means (n=40; per-run bases ±0.13 CPU / ±0.78 GPU). Prompt-eval latency is derived as prompt_tokens × 1000/prompt_tps from the llama-cli summary line — not a first-token timestamp. GPU VRAM peak 1778 MiB (Δ +1048 over a 730 MiB baseline). Energy is null — RAPL needs root.
Two-model speculative decoding is BLOCKED: llama.cpp requires draft and target to share a vocabulary, and no valid same-vocab draft pair exists locally — the only GGUF target is llama-arch (models/gguf/layerskip-1b-Q4_K_M.gguf) while the only small draft candidates are qwen2-family. Inventing numbers for this arm would be fabrication, so none are given.
Plain cascading (0.5B → 1B, HF driver)
A small model (Qwen2.5-0.5B-Instruct) drafts every prompt; a large model (LayerSkip 1B, bf16) re-generates only escalated prompts. Escalation is driven by the small model's mean token log-probability against a threshold — an uncalibrated heuristic. Greedy decoding, 40 AMI prompts, one run per prompt (per-prompt and per-run CIs coincide).
| Threshold | Escalated | Mean wall | Tokens/s |
|---|---|---|---|
| −1.08 | 5/40 (12.5%) | 1.90 s ± 0.21 | 72.78 ± 4.59 |
| −0.85 | 15/40 (37.5%) | 2.41 s ± 0.32 | 61.94 ± 6.68 |
| −0.70 | 27/40 (67.5%) | 2.98 s ± 0.31 | 48.22 ± 6.80 |
Expected escalation/latency trade-off: 12.5% → 67.5% escalation as the threshold tightens, wall 1.90 → 2.98 s. Output sanity 120/120 verbatim. Peak host RSS 4.23 GB; CUDA peaks 950 MiB (small) / 3316 MiB (large).
LayerSkip published configuration (off-the-shelf, no adapter)
The author's self-speculative recipe (draft at the exit layer, verify with the full model, 6 speculations, 128 steps max) on the unmerged checkpoint, timed by benchmark.py's in-process GPU timers. N=40 held-out prompts, 0 empties at every exit. Throughput here is descriptive only — no warm-up discard — so it is not speedup evidence.
| Exit layer (of 16) | Acceptance | Tokens/s (descriptive) |
|---|---|---|
| 4 | 9.16% | 29.96 |
| 6 | 12.08% | 28.60 |
| 8 | 13.48% | 26.72 |
| 10 | 16.55% | 26.46 |
| 12 | 53.95% | 50.31 |
Values are the log/JSON values from eval_baseline_exit{L}.log and metrics_extracted.json, which agree exactly; a stale prose line in acceptance_heldout.md (31.14/30.25/26.45/25.68/48.98 tok/s) does not match the raw logs and is not used.
Cross-arm tok/s is not head-to-head: three different timers produced these numbers — llama.cpp subprocess timing with warm-up discarded (plain decoding), HF generate() walls with one run per prompt (cascading), and in-process benchmark.py timers with no warm-up discard (LayerSkip config). Additionally, a sibling track's llama-server held ~2384 MiB of VRAM during the cascade run, so part of the 1B model was CPU-offloaded and large-stage cascade latency is pessimistic versus exclusive-GPU use.
Offline voice-loop PoC
A working end-to-end voice loop runs fully offline on the development host CPU (src/poc/loop.py): wav → STT → dialogue-act classifier → trigger → LLM → TTS → wav. Single scripted turn, no GPU touched.
| Stage | Wall time | Notes (development host) |
|---|---|---|
| STT (faster-whisper tiny.en int8) | 1.576 s | 6.18 s of audio, beam 1 |
| Dialogue-act classifier | 1.389 s | Includes bundle load and first sklearn import |
| Trigger (AND-gate) | 0.0 s | Pure Python, sub-millisecond |
| LLM (1B Q4_K_M, CPU, 64 tokens max) | 10.234 s | Dominates the loop |
| TTS (Piper sentence-level) | 1.781 s | 230-char response → 11.74 s of audio |
| Total | 14.981 s | Sum of stages; process wall 15.70 s, peak RSS 1.34 GB |
The decision record is honest about what is measured and what is scripted: p_deep = 0.7401 (measured, clears the 0.65 threshold) and p_end = 0.85 (scripted input parameter, clears 0.75), giving trigger_deep.
Scripted turn-taking: p_end is passed on the CLI — no audio VAP model exists yet. Synthetic audio: the 6.18 s input is Piper-synthesized, so STT accuracy is clean-synthetic-speech only. Single turn: one-shot batch STT, one trigger decision, no streaming, no false-trigger rate. Response quality is unevaluated — only that text flows end to end into spoken audio. Full notes: experiments/runs/poc-smoke/POC_NOTES.md.
Not measured
- Target hardware. No reported number comes from the Dell Latitude 5490 yet — that is Phase 5.
- Wall-clock speedup for PADS proper. The pipeline is a prototype without cache-aware depth switching; latency claims would be invalid, so none are made.
- Cross-arm tok/s is not head-to-head. Three different timers produced the Phase 3 numbers (llama.cpp subprocess timing, HF
generate()walls,benchmark.pyin-process timers), so no speedup claim compares across arms. - Cascade shared-GPU confound. A sibling process held ~2384 MiB of VRAM during the cascade run, forcing partial CPU offload — large-stage latency is pessimistic versus exclusive-GPU use.
- Generation-level quality (ROUGE-L / exact match) at matched compute — the Phase 2 comparison measures routing agreement with the oracle, not free-form generation quality.
- Energy and thermal on the target — energy tooling is blocked on root; the thermal run was a host substitute.
- Audio turn-taking. The current signal is a text-based proxy; VAP-style audio prediction is not implemented.
- The 15% empty-generation regression in the trained model is unresolved and is carried as a known quality issue.