Architecture: PADS end to end

Pause-Aware Depth Scheduling (PADS) runs one small model at two effective depths: a cheap shallow exit answers immediately, while the full network is started early — speculatively, inside the pause before a turn ends — so its cost is hidden instead of paid after the user stops speaking. This is self-speculative decoding applied to depth rather than to a separate draft model.

This page maps how it works: the runtime turn, the training recipe, the routing evaluation, the repository layout, and the measurement loop that produces the evidence. Structural constants (layer counts, thresholds, block sizes) are read from the code; executed numbers are labelled development host where applicable, with their evidence path, because the target hardware (a CPU-only Dell Latitude 5490) has not yet been run.

trigger + pipeline implemented and unit-tested audio turn-taking is an interface stub target-hardware runs pending

1 · System pipeline — PADS proper

The runtime is one conversational turn. A partial utterance arrives while the user is still speaking and fans out to two signals: the turn-taking predictor estimates p_end, the probability that the turn ends within roughly the next 300 ms, and the dialogue-act classifier estimates p_deep, the probability that this turn needs deep reasoning. The trigger policy (src/trigger_policy/policy.py:41) combines them into one decision: stay_shallow or trigger_deep.

The gate is an AND-gate with deliberately high thresholds — p_end ≥ 0.75 and p_deep ≥ 0.65 — and it is asymmetric on purpose: a false trigger spends compute that cannot be hidden, while a missed trigger only forfeits a speedup. The code comments say not to tune it for raw classification accuracy, and not to relax it to an OR-gate without re-deriving the cost asymmetry (policy.py:24-31, PRD §9).

Figure 1. The PADS turn. Two probability signals pass a conservative AND-gate; a fired trigger starts full-depth work inside the pause, and the end-of-turn check either adopts or discards that branch. The shallow path is always the fallback.

Once the true outcome is known, resolve() classifies the decision as correct_trigger, false_trigger, correct_shallow, or missed_trigger (policy.py:61). Latency composition then follows the cost asymmetry: a correct trigger hides min(full_depth, pause) of the speculative work behind the silence; a false trigger pays the shallow pass plus the work executed during the pause (branch_waste_ms); a missed trigger pays the full pass after the turn ends, with no waste (compose_ttft_ms, pads_pipeline.py:162). Every turn appends one JSONL record with p_end, p_deep, decision, pause, both TTFTs, the waste, and an explicit notes field marking mock records as non-reportable (pads_pipeline.py:147).

The correctness story is the one inherited from speculative decoding: a speculative branch is either adopted whole or thrown away, never blended, so the served output distribution matches non-speculative decoding. PRD §10 states this as a hard requirement ("output must be identical in distribution to non-speculative decoding"), and the discard edge in Figure 1 is what enforces it — a wrong branch costs time, never correctness.

Outcome vocabulary

correct_trigger — turn ended and depth was needed: the pause paid for the deep pass. false_trigger — one of the two did not hold: the branch is discarded and its executed work is the price. correct_shallow — shallow was the right answer. missed_trigger — deep was needed but the gate stayed shut: no speedup, no waste. These four labels, not accuracy, are what the policy is tuned to move (policy.py:61-75).

Limitation — no audio yet

Turn-taking exists as an integration contract only (src/turn_taking/README.md). There is no audio corpus or audio model in the repository, so the pipeline's StubTurnTaking deliberately raises instead of inventing a probability, and all executed runs inject p_end on the command line. The README also warns against shipping a fixed silence-threshold VAD as the final design.

Limitation — what the real mode verifies

The --real mode loads the LayerSkip checkpoint and times both passes with the model's own layers, but an offline benchmark cannot observe an actual end-of-turn, so a fired trigger is composed as if correct and branch_discarded stays false. The host demo therefore demonstrates the mechanism, not a verified speedup — and it ran on the development host GPU, not the Latitude 5490.

2 · Training pipeline — LayerSkip-recipe LoRA

PADS needs a checkpoint whose early exits are good enough to be worth routing through, so the base LayerSkip 1B is continued-trained with the layer-dropout plus shared-exit recipe (Elhoushi et al., LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding, ACL 2024, arXiv:2404.16710) using LoRA adapters rather than full fine-tuning.

Figure 2. Training recipe. Corpus construction holds out every meeting that appears in the evaluation set; each step samples a layer-dropout mask, runs a manual layer loop, and scores the primary exit plus the captured auxiliary exits.

transformer layers in the LayerSkip 1B base (2048-wide hidden state, 128,256-token vocabulary) — config.json

512

tokens per block; the held-out run built 4,575 blocks / 2,342,400 tokens (~0.66 passes) — train.log

11.27M

trainable LoRA parameters, 0.90% of the 1.247 B base — printed by the held-out run

fixed auxiliary exit depths {4, 6, 8, 10, 12}; exit layer 8 is the default shallow exit

The corpus joins DailyDialog's official train split with AMI meeting transcripts rebuilt from word-level timing annotations, one EOS-terminated line per utterance (src/training/ami_corpus.py). The held-out correction matters here: because all 40 evaluation prompts and responses occur verbatim in AMI meeting EN2001a, the training script gained --ami-exclude-meetings and the current recipe excludes the six meetings that contain an eval item. A direct count over the 687 AMI word files gives 80,489 utterances, and 76,616 after those exclusions; the run printed 4,575 blocks and 2,342,400 tokens from the held-out stream (experiments/runs/2026-09-22T18-22-08Z-heldout-retrain/train.log).

Each step samples a per-layer keep mask whose dropout rate ramps linearly from 0.0 to 0.5 with depth — a documented simplification: the mask is sampled per batch, not per sample. A manual forward loop then executes the kept layers, capturing hidden states at the requested depths only when they actually executed below the last executed layer, so no depth is double-counted (train_layerskip_lora.py:96-120). The multi-exit loss scores every captured depth through one shared norm and LM head: weight 1.0 for the primary exit at the last executed depth, 0.3 for each auxiliary exit, with auxiliary tokens strided and fp32 logits computed in fixed-size chunks so memory stays bounded (src/training/early_exit_loss.py).

Adapters checkpoint every 500 steps in the held-out run and at adapter-final; merge_lora.py then merges the adapter into the base so the plain-HuggingFace evaluation path can load it without PEFT. From there, src/training/eval_checkpoint_acceptance.sh measures the acceptance rate of the merged model.

Limitation — a small, simplified run

The recipe differs from the published one in three recorded ways: the layer-dropout mask is sampled per batch; the step loss (loss) is dropout-randomized and auxiliary-weighted while val_loss is full-depth with no dropout, so the two are not comparable; and the held-out run saw ~0.66 passes over a ~2.3 M-token corpus, which is small. All training to date ran on the development host GPU (RTX 4070 Laptop); the target laptop only ever appears as the deployment target.

3 · Evaluation pipeline — exit routing

Before trusting any trigger, the project measures routing quality in isolation: given shallow and full-depth predictions for the same positions, how often does a policy's choice match what full depth would have said? Figure 3 follows one prompt through src/eval/policy_eval.py. The eval set is 40 AMI turns (prompt/response records; 40 counted directly in experiments/runs/2026-09-21T19-10Z-test3-acceptance/conv_eval_ami.jsonl), and the held-out policy run covers 953 next-token positions at exit layer 8.

Figure 3. Evaluation pipeline (snake order: left→right, then right→left below). One forward pass yields both depths; policies only pick between the two logit sets, so no extra model calls are needed per threshold.

run_prompt() performs exactly one forward pass with hidden states enabled, slices the hidden state at the exit layer and at the final layer for all T−1 next-token positions, and applies the same norm and LM head to both (policy_eval.py:88). The two exit policies are pure decision functions with no model state: the token-confidence policy routes an individual position deep when the shallow top-1 probability falls below its threshold, while the dialogue-act policy routes the entire turn deep when the classifier's p_deep clears its threshold (src/eval/exit_policies.py).

Metrics are defined against the full-depth oracle, so the comparison is exact and needs no second model:

Metric	What it measures
`agreement`	Position-weighted fraction where the policy's chosen top-1 equals the full-depth oracle's top-1.
`shallow_agreement`	The same comparison with no routing at all — the always-shallow floor.
`deep_fraction`	Position-weighted fraction routed to full depth: the compute budget axis.
`n_positions`	953 next-token positions across 40 prompts in the held-out run at exit layer 8 (`policy_eval_heldout.json`).

Because policies and thresholds move along different curves, the comparison script re-sorts each policy family by deep fraction and linearly interpolates agreement at matched compute budgets of 25 / 50 / 75% — cells are only reported when a budget is actually bracketed, never extrapolated. Intervals come from a prompt-level cluster bootstrap: 2000 resamples draw 40 prompt indices with replacement (positions are never resampled), paired differences are computed inside each resample, and the seed is recorded (20260923) in make_comparison.py. The outputs are the per-run policy_eval_*.json files and the matched-compute table in comparison.md.

Limitation — routing, not speed

The evaluation deliberately reports no latency: the harness loads full hidden-state stacks instead of a KV cache, so any wall-clock figure would be invalid (policy_eval.py module docstring; the run manifest records wall_clock_latency_measured: false). Every executed number to date is from the development host (RTX 4070 Laptop GPU, reportable_as_target_hardware=false) — agreement measures routing quality, not inference speed on the target CPU.

4 · Component and repository map

The repository is organised by function, and each directory produces a specific artefact that a later stage consumes: decision code produces run inputs, runs produce manifests and metrics, and the site composes them into what you are reading. Figure 4 is the one-line map.

Figure 4. Repository map. Decision code (trigger, pipeline, policies) is hardware-agnostic and unit-tested; training and evaluation produce run directories; experiments/results/ and the run logs are the raw evidence behind every number.

Runtime path

A turn flows pads_pipeline.run_turn → policy.decide/resolve → compose_ttft_ms → one JSONL record. Nothing in this path needs a GPU: the policy operates purely on the two probabilities, which is why it is testable without the target laptop or real models (policy.py:5-8).

Evidence path

A measured claim flows a run directory (cmd.sh, manifest, metrics.jsonl, logs) → experiments/results/ verification logs → the experiments page. If a number cannot be traced back to a run directory or log, it does not belong on this site.

5 · Deployment and measurement map

Weekly work on this project is heavy jobs on a 15 GB laptop, so the measurement loop is as much part of the architecture as the model code. Figure 5 shows how a job is admitted, where its outputs land, and how progress is read back without touching the machine while it is busy.

Figure 5. Deployment and measurement map. Admission control protects the desktop; the run directory is the unit of evidence; the dashboard only reads.

tools/guard.sh is admission control plus a hard cap: it refuses to start when available memory is below a floor (2 GB by default, optionally a GPU-free-memory floor), then runs the job inside a systemd user scope with MemoryMax set, so a breach kills that job rather than the desktop. Each job writes into one timestamped run directory: the exact command (cmd.sh), a manifest capturing purpose, git SHA, seed, environment, and held-out state, a metrics.jsonl stream, and the raw logs. Progress is tracked as checkboxes in docs/superpowers/plans/*.md — the dashboard's "plan ledger" — and the read-only dashboard (tools/dashboard/server.py, bound to 127.0.0.1) combines machine vitals, the latest runs, and that ledger into one view.

Limitation — measurement gaps on the host

Energy readings are blocked: RAPL energy_uj requires root on the development host, so the Go/No-Go energy test is logged as BLOCKED-ON-ROOT. Sustained-load thermal behaviour is also unmeasured (only launch-time temperatures exist in the run artifacts). Both are target-hardware items; neither is inferred from the other.

6 · Known gaps

No audio turn-taking

p_end is injected on the command line in every executed run. The interface and data requirements are documented, but no predictor has been built, so the pause detector is a real architectural hole, not a placeholder constant.

No wall-clock latency

The evaluation reports routing quality only — agreement and deep fraction. There is no KV-cache-aware depth switching in the harness, so no speedup number exists to quote, and none is claimed.

No target-hardware results

Every executed training and evaluation number is from the development host (RTX 4070 Laptop). The Dell Latitude 5490 CPU-only benchmark — the actual research question — is still pending.

This page describes architecture and provenance. For what the runs actually found — including the corrected held-out acceptance numbers — see the Experiments page.