Technology Stack

PADS is a small original bet assembled from published parts. This page explains every technique and tool in the project — what it is, what the project actually configured, and why it is there — including the parts that produced negative results. Nothing here is aspirational: where the repository has an executed number, it is labelled development host (the RTX 4070 laptop used for training and dry runs) and distinguished from the target target hardware, a CPU-only Dell Latitude 5490 that has not yet been benchmarked.

mechanism implemented and unit-tested training + evaluation numbers are dev-host audio turn-taking is an interface stub target-hardware runs pending

1 · The inference problem: autoregressive decoding on a CPU

An autoregressive language model produces text one token at a time: each new token is conditioned on every token before it, so generating n tokens takes n sequential forward passes through the network. Each pass must read the model's weights and the KV cache of the conversation so far. On a CPU with no discrete GPU, this is a memory-bandwidth-bound workload: the arithmetic units are mostly waiting for bytes to arrive from DRAM, not crunching numbers. Adding threads helps only until the memory bus saturates, after which more threads add contention, not throughput. The project's first Go/No-Go test was built precisely to check that this regime is real, because PADS's entire premise — that a human pause can hide a meaningful chunk of decoding work — is weaker if the CPU has spare compute to burn anyway.

Development-host dry run, not target hardware: a 3B model at Q4_1 on the Intel Core Ultra 7 155H development laptop, one run per configuration, no confidence intervals, hybrid P/E cores (evidence: experiments/runs/2026-09-21T19-30Z-test1b-bandwidth-3b/results.csv). The intended 7–9B test on the Latitude 5490 is still outstanding. The plateau past five threads is directionally consistent with bandwidth saturation, which is what the project expects to characterise on the target.

Quantisation is the response to the memory problem: store the weights in fewer bits so fewer bytes move per token. GGUF is the container format llama.cpp uses — tensors plus tokenizer and metadata in one memory-mappable file. The project's intended default working precision is Q4_K_M, a k-quant scheme that keeps most tensors at roughly four bits with per-block scales and holds some tensors at higher precision, balancing size against accuracy; Q5, Q6, Q8 and F16 are reserved as calibration references. The build skill adds one non-negotiable rule: always quantise down from an F16 GGUF, never from an already-quantised file, because quantising quantised weights compounds precision loss. Both GGUF artifacts in the local model directory are Q4_K_M (models/gguf/layerskip-1b-Q4_K_M.gguf and models/gguf/Qwen2.5-7B-Instruct-Q4_K_M.gguf; weights are gitignored, so they live only on the development host).

Why no GPU? The research question is explicitly about the hardware most edge deployments actually have. The target is a Dell Latitude 5490: 8th-generation Intel CPU, Intel UHD 620 integrated graphics, 16 GB RAM, no discrete GPU. The development laptop's RTX 4070 is used only for training and dry runs; the project rule is that no inference, latency, or energy number may be reported as a target result unless it came off the Latitude. The repository enforces the distinction by tagging run manifests with reportable_as_target_hardware=false on host GPU runs.

2 · Self-speculative decoding

Classic speculative decoding uses two models: a small draft model proposes several tokens cheaply, and a large target model verifies them in a single forward pass. Because verification is parallel over the proposed tokens, accepted drafts yield multiple tokens per expensive pass. The catch is the KV cache: caches are tied to a model's own hidden dimension and layer count, so the draft model's cache cannot be handed to the target model. The draft model also has to be resident in memory, which is painful in a 16 GB budget, and its work is not reusable state.

Self-speculative decoding removes both problems by using one model at two depths. An early exit after layer k drafts tokens through the shared output head; the model's remaining layers then verify them. Both passes belong to the same network, so the KV cache populated by the shallow pass is reused verbatim by the deep continuation — same dimensions, same layers, no incompatibility and no second model's weights.

The mechanics, as implemented in the vendored LayerSkip reference generator: at each step the shallow exit generates --num_speculations draft tokens autoregressively; the deep layers then score the drafted sequence in one continuation pass; the longest prefix that matches the deep model's own tokens is accepted, and on a mismatch the deep model's token is emitted instead. The acceptance rate is accepted drafts divided by total drafted tokens, and it governs the speedup: high acceptance near full depth saves little because the deep pass runs anyway, while low acceptance at shallow exits wastes drafting work.

The correctness guarantee is inherited, not re-derived. Speculative decoding is constructed so that the final output distribution is identical to non-speculative decoding of the target model; with greedy verification the accepted token sequence is exactly the deep model's. The project treats this as a hard requirement (docs/03_SKILL.md §3.5): a discarded speculative branch must not change the served distribution, and a wrong trigger must therefore be cheap rather than dangerous.

The first acceptance measurements were the project's sharpest early warning. Running the off-the-shelf LayerSkip 1B unmodified on conversational prompts, greedy acceptance rose steeply with depth:

Development-host GPU, greedy sweep; with sampling at exit 8/16 the mean was 12.4% (evidence: experiments/runs/2026-09-21T19-10Z-test3-acceptance/). The kill signal fired at aggressive exits: half-depth acceptance was too low to matter, and the only usable exit was near full depth, where little is skipped. That result is what motivated fine-tuning rather than using the checkpoint off the shelf.

3 · LayerSkip

LayerSkip (Elhoushi et al., ACL 2024, arXiv:2404.16710) is the published recipe the project builds on. Its two ingredients are layer dropout during training — each training step randomly skips some decoder layers, with later layers dropped more often — and a shared early-exit loss: the model's single LM head is applied to the hidden state at the last executed layer, so the network learns to produce a usable next-token distribution at many depths, not only at full depth. At inference, any depth becomes a candidate exit layer.

In this project's 1B checkpoint (16 decoder layers), exit layer k means the first k decoder layers execute and the shared head reads the hidden state they produced. Exit layer 8 is 8 of 16 layers — half depth; exit layer 16 is the full network. The evaluated ladder is 4/6/8/10/12, i.e. 25%–75% depth. When a shallow exit is used for drafting and the result is verified, the deep continuation starts at layer k+1 from the same KV cache, so no prefix is recomputed.

Why adopt it rather than invent a mechanism? Because it is published, proven, and inherits the correctness guarantee of speculative decoding, and because it sidesteps the KV-cache incompatibility that would make a two-model design incoherent on a 16 GB device. The project's contribution is deliberately when the deep pass is triggered, not how depth extension is computed.

4 · Our training method

The base checkpoint is Meta's LayerSkip-recipe 1B; the project continues training it with LoRA rather than from scratch, per the build skill's explicit rule. A LoRA adapter is a pair of small low-rank matrices inserted beside the frozen weights; only those matrices train. The configuration used in every run to date is:

Layer dropout ramp. Each step samples a per-layer keep mask whose dropout rate increases linearly with depth, from 0.0 at the first layer to 0.5 at the last (train_layerskip_lora.py:63-69). The model therefore sees many different executed depths, including truncated tails where the last executed layer is well below 16. One documented simplification: the mask is sampled per batch, not per sample.

Multi-exit loss. After the manual layer loop, one shared norm and LM head score several depths at once. The primary exit is the last executed depth, weight 1.0; the auxiliary exits are the fixed depths {4, 6, 8, 10, 12}, each weighted 0.3 relative to the primary. Auxiliary exits score only every 4th token position (memory bound), while the primary exit always scores every position; logits are computed in fixed-size chunks in fp32 so activation memory stays bounded (src/training/early_exit_loss.py). This is the change made in response to the first training attempt: a 4,000-step single-exit LoRA run (loss applied only at the last executed layer) improved only exit 10 (+5.54 pp) and degraded exit 12 by 19.32 pp — a negative result recorded in experiments/go_no_go_results.md. The multi-exit objective was adopted to train the shallow exits directly.

Held-out split and why leakage mattered. The first multi-exit training run globbed all 687 AMI meetings with no exclusions — including EN2001a, the source of all 40 acceptance-eval items. The leak check found all 40 prompts and all 40 responses verbatim inside the training token stream, so the trained arm had been evaluated on data it trained on. The fix is source-level: ami_corpus.py::ami_text takes exclude_meetings, and the trainer takes --ami-exclude-meetings. The retrain excludes every meeting that verbatim-contains an eval item — EN2001a, EN2001d, EN2002c, ES2014c, IN1013 and IS1002c (the first is the named source; the rest contain one generic eval prompt or other item text) — shrinking the corpus from 80,489 to 76,616 AMI lines and from 4,712 to 4,575 blocks; after exclusion, 0/40 prompts and 0/40 responses remain (experiments/results/heldout_leak_check.txt). The contamination had inflated apparent acceptance by up to +15.15 pp at exit 8; the corrected numbers below supersede it.

What training produced. Held-out validation loss (full-depth, no dropout) fell from 4.8056 at step 10 to 3.3683 at step 1500 [MEASURED] in the retrain's metrics.jsonl. Acceptance on the same 40-item AMI eval set, same greedy decoding, same seed:

Development-host GPU; evidence experiments/runs/2026-09-22T18-22-08Z-heldout-retrain/acceptance_heldout.md. The trained arm produced 6 empty generations out of 40 at every exit (15.0%), which the benchmark skips before averaging; the conservative column charges each empty as acceptance 0. The baseline column is bit-identical to the pre-retrain baseline, so the comparison is like-for-like.

Two further results are recorded rather than hidden. First, generation quality: ROUGE-L on the same eval is 0.0429 for the trained adapter (N=34) versus 0.0368–0.0372 for the baseline (N=40) — above baseline on the non-empty denominator, roughly at parity when the empties are charged zero, but not like-for-like and with no confidence interval. Second, the exit-4 regression: acceptance at a quarter depth got worse, not better. And the 15% empty-generation rate is itself a quality regression relative to the baseline's 0/40 and remains unresolved.

5 · The dialogue-act classifier

PADS needs a second signal besides turn timing: whether the coming response needs deep reasoning at all. A dialogue act is the conversational function of an utterance (question, directive, statement, commitment). The project uses a deliberately lightweight CPU classifier for this because it must run concurrently with inference on the same laptop; a second transformer would compete for the same cores and memory that the pause is supposed to exploit.

Proxy labels. DailyDialog tags four acts, not the project's shallow/deep distinction, so the mapping is an explicit, documented assumption: question and directive → DEEP; inform and commissive → SHALLOW. The classifier's accuracy is therefore evidence about this proxy, not about the deployment task — the source file says exactly that.

Features and model. TF-IDF over word n-grams of size 1–2 (min_df=1, vocabulary 215,529) feeds a logistic regression (max_iter=1000, seed 42). Fitting took 4.3 s; the whole process peaked at 429,212 KB RSS and 13.1 s wall — small enough to coexist with generation.

Leakage handling. The official DailyDialog splits are used as published (no random re-split). Before fitting, any training utterance whose exact text also occurs in the test split is removed: 5,825 train utterances (1,804 distinct texts) were dropped, leaving 81,345; zero shared texts remain. This replaced an earlier synthetic dataset of 16 hand-written examples replicated 8× and split randomly — the same utterances appeared on both sides, so its 1.000 accuracy was leakage, not signal.

Development host, CPU-only; evidence experiments/results/dialogue_act_realdialog_metrics.json. A failure check confirms the script exits non-zero with a FATAL message when the dataset cannot be loaded — there is no synthetic fallback.

6 · Exit policies and the matched-compute comparison

An exit policy decides which positions are computed at full depth. The project compares two families, both pure decision functions with no model state (src/eval/exit_policies.py):

Matched compute. Comparing policies at their default thresholds would confound routing quality with how much compute each one buys. Instead the evaluation measures each policy curve — position-weighted agreement with the full-depth oracle versus deep fraction — and interpolates linearly at matched deep budgets of 25%, 50% and 75%. Targets outside a policy's measured range are reported as not bracketed, never extrapolated. Agreement is the fraction of next-token positions whose top-1 matches what full depth would have produced; it is a routing-quality measure, not latency. Intervals come from a prompt-level cluster bootstrap: the 40 prompts are the independent unit, resampled with replacement 2,000 times (seed 20260923), positions never resampled, paired differences computed within the same resample. The evaluation set has 953 next-token positions in total, cross-checked by an independent tokenizer pass that reproduced every prompt's count exactly.

Development-host GPU, exit layer 8 vs full depth, 40 prompts / 953 positions; references: always-shallow agreement 0.3274 (held-out) and 0.0955 (base); always-deep oracle agreement 1.0000 by definition. Evidence experiments/runs/2026-09-22T19-18-00Z-phase2-policy-eval-heldout/comparison.md. The base arm's minimum measured deep fraction is 0.3924, so a 25% budget for it is not reachable and is not extrapolated.

What the comparison found. The safe core is a positive result: at matched compute, the held-out-trained model's token-confidence policy beats the off-the-shelf base's by +21.25 pp at 50% deep (95% CI [18.45, 24.15]) and +12.82 pp at 75% ([10.96, 14.71]), with P(difference > 0) = 1.000. The dialogue-act policy is a negative result at the same budgets: it trails token confidence in every point estimate at every matched budget (by 5.78 / 8.85 / 7.15 pp), and on the base model it never wins. Whether the dialogue-act deficit is statistically established depends on the uncertainty convention; under the assumption-free fixed-bracket convention the deficit is not resolved at n=40, so the project does not cite it as established.

7 · Turn-taking and pauses

Turn-taking prediction estimates, from partial speech, whether the speaker is about to yield the floor. The project's eventual approach is Voice Activity Projection (VAP) or a comparable lightweight turn-taking model, both of which are documented to run in real time on CPU. That integration is not implemented: src/turn_taking/ is an interface stub (TurnTakingPredictor.update(audio_chunk) → P(turn ends within ~300 ms)) with no audio model behind it. What the pipeline demo used instead is a scripted text proxy: an injected p_end = 0.90, clearly labelled as such, so the AND-gate logic could be exercised end to end.

Pause durations. The timing premise was checked against real human silence, extracted from the AMI Meeting Corpus manual annotations (words XML, 171 meetings). A pause is defined as the maximal silence gap between consecutive same-speaker words not covered by any other speaker's speech, restricted to 0.1–5 s:

Evidence experiments/runs/2026-09-21T19-05Z-test2-pause-durations/ami_pause_stats.json. The 7B alternative was extrapolated at 2.33× parameters to 164.1 ms per step, which still fit 96.6% of pauses — but extrapolation is not measurement. AMI is meeting speech, not telephone dialogue; the corpus's own caveat says pause distributions differ from Switchboard/CallHome, so this is a proxy. The intended Switchboard/CallHome check remains open.

The mechanism demo. The real pipeline mode loads the trained dialogue-act classifier and the LayerSkip 1B, times a shallow pass and a deep continuation, and composes a perceived TTFT under an injected pause. One logged run: p_deep = 0.9685 from the real classifier, scripted p_end = 0.90, the AND-gate fired trigger_deep, shallow pass 9.2 ms and deep pass 9.3 ms (host GPU, mean of 3 trials; the logged deep-pass TTFT is 9.27 ms), simulated pause 0.8 s → post-confirmation TTFT 0.0 ms because the deep pass fit entirely inside the pause. This demonstrates the mechanism, not a speedup: the turn-taking signal is scripted, the pause is a command-line parameter rather than real speech timing, the branch-discard path cannot be verified offline (branch_discarded stays false), and it ran on the host GPU, not the target. Evidence: experiments/results/pads_pipeline_real_demo.jsonl.

Why AMI and DailyDialog? DailyDialog is an open, licensed conversational corpus with per-utterance act tags, used to train the classifier and to seed the continued-training corpus. AMI provides aligned word-level timing for pause statistics, meeting transcripts for the training corpus, and the 40-turn conversational eval set used for acceptance and routing. Neither is the target deployment domain; both are openly available stand-ins.

8 · Measurement methodology

Every benchmark run is captured as a structured record with the same fields: TTFT (time to first token), tokens/s, peak RSS, acceptance rate, and energy (or an explicit None with a note saying why). The harness never returns an unmeasured number: a non-zero exit, a timeout, or a missing timing line each raises an error carrying the captured output, rather than defaulting to zero or a guess.

Warm-up and discard-first-runs. The thermal Go/No-Go test logged 177 samples over roughly 35 minutes with a maximum temperature of 103 °C (16.9% of samples at or above 95 °C) and frequencies swinging between 1.3 and 4.3 GHz on hybrid cores — enough to make back-to-back runs non-reproducible. The permanent protocol is therefore a warm-up period before timed runs, discarding the first runs, a fixed CPU governor, and no competing background load; where that protocol was not applied (the acceptance runs' throughput means), the numbers are labelled descriptive and are not used as speedup evidence.

Confidence intervals. Two methods exist in the repository, and which one applies is recorded. The benchmark harness aggregates multiple runs with a normal-approximation 95% interval (1.96·σ/√n), with a code comment saying the approximation is rough at small N and must be noted in the methodology. The policy comparison uses the stronger prompt-level cluster bootstrap described in §6 — 2,000 resamples of the 40 prompts, 2.5th–97.5th percentiles — because positions within a prompt are not independent.

Oracle agreement and generation quality are separate. Acceptance and oracle agreement measure whether routing picks the same tokens the full-depth model would; they say nothing about whether the generated text is good. That is why ROUGE-L is tracked as a supplementary proxy, with its limitations stated: it is computed only on samples that update metrics, it is near the floor on short repetitive AMI turns, it has no confidence interval, and the empty-generation rate is a separate quality signal entirely.

9 · Tooling

llama.cpp is the inference engine for the CPU-only deployment. The build is CPU-only by design: CMake with native CPU optimisation, no CUDA or Metal backend, and the npm-based embedded web UI disabled (-DLLAMA_BUILD_UI=OFF), since only headless CLI inference is needed. Model conversion is two steps: convert_hf_to_gguf.py writes an F16 GGUF from the fine-tuned checkpoint, then llama-quantize produces the deployment precision (Q4_K_M) from that F16 file — quantising last, and never from an already-quantised source.

The training and evaluation stack is Python: PyTorch with bf16 weights for the manual layer loop; HuggingFace transformers for the model, tokenizer and causal-LM loading; peft for LoRA (LoraConfig, get_peft_model, adapter loading and merging); and datasets for DailyDialog (parquet revision) and AMI parsing. The LayerSkip reference repository is vendored under third_party/LayerSkip, and its self-speculative benchmark is launched with torchrun: --generation_strategy self_speculative --exit_layer L --num_speculations 6 --max_steps 128 --no_sample (greedy, seed 42).

Resource control. Heavy jobs run through the guard script (tools/guard.sh), which does two things: admission control (refuses to start if available memory is below a floor, or GPU memory below an optional floor, or warns on high GPU temperature) and enforcement via a systemd user scope (systemd-run --user --scope -p MemoryMax=<cap> -p MemorySwapMax=2G). A memory breach kills only that scope (exit 137), never the desktop — a direct response to this machine's documented history of OOM-killing itself. Every run lands in a timestamped directory containing the exact command (cmd.sh), a manifest with purpose, git SHA and environment, a metrics.jsonl stream, and the raw logs, so any number can be traced back to the command that produced it.

The dashboard (tools/dashboard/server.py) is a read-only local view bound to 127.0.0.1: CPU/memory/GPU vitals, the latest run manifests with their purpose strings, and an operator-maintained phase/ETA file. It exists to answer "what is running and is it healthy?" without touching the jobs.

10 · Statistical honesty rules used

The project keeps a written evidence convention, applied throughout this page and the repository. Every quantitative claim carries exactly one tag, inline, where the number appears:

The vacuous-pass fix. The RAM-budget Go/No-Go test originally had a pass condition that was a constant ((16 - 3) > 0), so it could never fail — a green check that proved nothing. It was fixed, and the kill-signal branch was then verified by monkeypatching the input to force the failure path. The same discipline appears in the parser tests, which ship a deliberate-red run showing the test failing when the input is broken, and in the classifier, which exits non-zero with a FATAL message when its dataset cannot load rather than falling back to synthetic data.

The contaminated run, corrected in place. The first multi-exit training run trained on every AMI meeting, including the source meeting of all 40 evaluation items; a leak check found 40/40 prompts and 40/40 responses verbatim in its training token stream. Rather than quietly dropping the affected numbers, the project documented the contamination, fixed the corpus at the source (exclude_meetings plus the --ami-exclude-meetings flag), retrained, verified 0/40 leakage, and published both the contaminated and corrected acceptance tables with the inflation quantified (up to +15.15 pp at exit 8). The earlier policy-evaluation trained curves, produced on the same contaminated model, are superseded by commit ae0d32e and are not used. When two documents disagree on a figure, the newer like-for-like measurement wins and the older one is marked superseded — the pattern used for the 20-sample test-3 acceptance reference and the 40-sample baseline.

Sources for the executed numbers on this page: experiments/go_no_go_results.md; run directories experiments/runs/2026-09-21T19-30Z-test1b-bandwidth-3b/, 2026-09-21T19-05Z-test2-pause-durations/, 2026-09-21T19-10Z-test3-acceptance/, 2026-09-21T18-46-49Z-test4-thermal/, 2026-09-21T18-46-49Z-test5-ram-budget/, 2026-09-22T15-35-01Z-layerskip-multiexit-training/, 2026-09-22T18-22-08Z-heldout-retrain/, 2026-09-22T19-18-00Z-phase2-policy-eval-heldout/; experiments/results/dialogue_act_realdialog_metrics.json, experiments/results/heldout_leak_check.txt, experiments/results/pads_pipeline_real_demo.jsonl; and the code under src/training/, src/eval/, src/dialogue_act/ and src/trigger_policy/. The manuscript draft is paper/PADS_manuscript.tex. Every measured number above is a development-host figure unless explicitly stated otherwise; structural constants (layer counts, adapter ranks, thresholds) are read from the code.