Every number below is from a real file on disk or the real, currently-running code — nothing estimated without saying so, nothing skipped. Read top to bottom; each answer only uses facts established in an earlier answer.
Two separate real systems, not one:
| System | What it measures | Bits covered per tensor, per pass | Uses the Hessian? |
|---|---|---|---|
| KL sensitivity sweep ( 06_measure_sensitivity_cascaded_V4.py) | How much the model's real output distribution shifts if this tensor is naively rounded | All of 4, 5, 6, 8, 16 together, in one pass (confirmed: every real checkpoint entry already stores a sensitivities dict with all five) | No — plain mx.quantize, no correction |
| YAQA correction ( yaqa_core.py, adaptive_safe_correction()) | How much real, curvature-weighted error is left after the best correction YAQA can make | Exactly one — whatever single bits value is passed in, confirmed at the only 3 real call sites in the codebase | Yes — this is the Hessian-based step |
So the naive KL sweep already tells you, cheaply, how bad naive rounding is at every candidate bit-width. What's missing is the equivalent for the corrected error — how bad is it after YAQA's real fix, at every candidate bit-width. That number currently only ever gets computed for the one bit-width the plan already committed to.
effective_rank(Hin_raw) and effective_rank(Hout_raw) take no bits argument at all. The Hessian (H_I, H_O) is a property of the tensor and the calibration data alone — it has no idea what bit-width you're about to round to. It is genuinely, fully reusable across every candidate bit-width for the same tensor.Not carelessness — a real, honest sequencing reason. To even know which bit-width to correct a tensor at, you need a plan first. The cheapest way to get a first plan is the naive KL sweep (it already tests all bit-widths at once, since no correction step is involved — just rounding and measuring). Once you have a plan, correcting each tensor at its one assigned bit-width is the minimum real work needed to ship a corrected model. Building the joint version — where YAQA's own corrected numbers feed back into the bit-width decision itself — is a bigger, second step, and it only becomes clearly worth building once you have evidence the first version's decisions are shaky. That evidence exists now: the cascade investigation earlier in this conversation found the isolated-KL-based plan disagreed with cascade-measured real risk on a real, named set of tensors, in both directions. That's the actual trigger for asking this question now, not something that should have been obvious on day one.
The full trunk correction (362 tensors, 41 real batches, one bit-width per tensor as assigned by the plan) — earliest batch log batch_0.log to latest batch_40.log:
27.56 real hours
(99,204 seconds, average 40.3 minutes per batch of 7-9 tensors.) Your memory of "very long, multiple days territory" was the right order of magnitude — the real number is 27.56 hours, not 48, but firmly in the same ballpark, not an order of magnitude off.
Inside one batch log, the structure is explicit: "24 real sequences, chunked 1 at a time... collecting real H_I/H_O for all 7 target tensors in this batch, ACCUMULATING" — the expensive part is 24 real forward+backward passes through the whole 27B model, done once, shared across every tensor in that batch. What follows immediately after (effective rank, block LDL, the correction itself) is matrix arithmetic on Hessians sized in the thousands, not full-model passes — structurally the cheap part, though no per-step timer exists in the log to give an exact split; that's stated as reasoning from the code's real structure, not a measured number, and flagged as such.
Two real reasons, both confirmed in code:
state[name]["hin"/"hout"]) and where correction consumes them. They live in memory for one batch, then MLX's allocator reclaims that memory once the next batch loads. There's nothing on disk to reuse.So this isn't "we were dumb and forgot to save it" in the sense of a careless oversight — persisting every tensor's full H_I/H_O matrix (thousands-by-thousands, per tensor, times 497 tensors) would itself be a real, large amount of disk space for data that, under the original one-bit-per-tensor design, was only ever going to be used once. It becomes worth keeping only once you're planning to reuse it across multiple bit-width candidates in the same pass — which is exactly the design below.
| Approach | What it does | Real cost |
|---|---|---|
| Done right — loop candidate bits inside the same pass | Compute H_I/H_O once per tensor (the expensive 24-chunk forward+backward pass), then, while still in memory, run adaptive_safe_correction() once per candidate bit-width before moving to the next tensor | ~27.6 hours plus a modest overhead for the extra correction/safety-gate loops (cheap matrix ops, not full-model passes) — a real but not multiplicative cost. I don't have an exact percentage for this overhead; it's not directly measured anywhere yet, and I'm not going to invent one. |
| Done the naive way — rerun the whole script per candidate bit-width | Run run_full_yaqa.sh six separate times, once each for bits ∈ {2,3,4,5,6,8} | 6 × 27.56h ≈ 165 hours ≈ 6.9 days — this is the number your memory was reaching for, and it's real only if built the wrong way |
[4,5,6,8,16], not [2,3,4,5,6,8]. Adding 2/3-bit means also handling cases where block_LDL/the safety gate may fail far more often at that precision (the real fallback-to-naive path already exists for exactly this). Worth deciding explicitly before building, not defaulting into.Right now the MILP/Pareto solver picks bit-widths by minimizing naive KL sensitivity subject to a bits-per-weight budget. YAQA's real, corrected error is computed only after that decision, at whichever bit got picked — so the solver is optimizing against a proxy for the thing that actually ships, not the thing itself. A multi-bit YAQA-corrected sensitivity checkpoint would let the solver instead minimize the real, corrected error directly — closing that proxy gap. Concretely, it would let you build a bit-allocation objective that asks "given YAQA will actually run on this, what's the true remaining error at each candidate bit-width" instead of "how bad is naive rounding at each candidate bit-width," which is a materially different, more honest question for the solver to answer.
This is genuinely open, and I'm not going to claim it's already answered just because it would be a satisfying conclusion. What is already real and verified, from earlier in this exact investigation: the simple, single-number hess_score (the aggregate effective-rank fraction, no correction involved) does not reliably track cascade-measured real risk — the correlation is weak and flips sign between isolated and cascaded measurement (−0.40 → +0.31 → +0.40), and on the 23 specific tensors the Hessian flagged as "safe" enough to unfloor, cascade measurement disagreed with 13 of them outright, pushing them to full 16-bit precision instead of freeing them to 4-bit.
A multi-bit, YAQA-corrected sensitivity map is a genuinely different, richer signal than that simple aggregate score — it's the real post-correction error, not a raw curvature summary. It is not proven to agree with cascade any better, and it's not proven to disagree either. That's exactly the kind of claim this project's own standard demands get tested, not assumed, once the data exists.
The honest version of this claim: if a multi-bit YAQA-corrected sensitivity map is built and it turns out to track real (cascaded) model behavior meaningfully better than naive KL does, then yes — it would mean the allocation decision itself should be driven by the corrected-error curve, not the naive one, and YAQA's role would expand from "fix the rounding after the plan is chosen" to "inform the plan in the first place." That's a real, coherent, buildable next step. It is not yet demonstrated — Question 8 is exactly the test that would demonstrate or refute it, and that test doesn't exist until the multi-bit checkpoint does.
| Multi-variable solver | Best-practice / structural-floor solver | |
|---|---|---|
| Inputs | Real YAQA-corrected error per candidate bit + cascade KL agreement + historical floor knowledge + bpw budget, jointly weighted | A fixed, human-decided list of layers that always stay BF16 regardless of score + bpw budget + Hessian data as a secondary check |
| Strength | Can in principle find allocations neither pure heuristic would, by trading off multiple real signals at once | Easy to audit, easy to explain, easy to override by hand; failure modes are visible (a floor either fires or doesn't) |
| Real risk | More free variables to get wrong; a badly-weighted joint objective can produce allocations that are hard to explain after the fact — exactly the kind of opacity this project's own documentation standard has been fighting against all session | Blunter — the late-attention-floor problem found earlier in this exact conversation (23 tensors floor-protected regardless of real score) is a direct real example of this approach's failure mode |
Neither is obviously correct in the abstract. Given everything verified in this document, the lower-risk real next step is: build the multi-bit YAQA-corrected checkpoint first (Question 6's "done right" cost, ~27.6h plus modest overhead), run the Question 8 correlation test for real, and let that real result — not a preference — decide which MILP design is worth building next.
A new flag on the existing correction script, not a new pipeline — reuses every real, already-proven mechanism (vjp Hessian collection, regularized_block_ldl, ldlq2hess_quantize, safety_gate), just loops the correction step over a candidate-bits list while the Hessian is still in memory, and writes a new checkpoint file instead of (or alongside) the corrected weights:
This is a real, scoped, reviewable change — it doesn't touch the existing single-bit correction path at all, only adds a new optional pass alongside it. I haven't built this yet; this is the design, ready to implement once you confirm the candidate-bits list (Question 6's open item) and whether this runs as a separate godmode pass or folded into the next real trunk build.