The Hessian, Fully Explained: Zero to Expert

Author: Hakim Ghelab, VegaLaboratories LTD

This document assumes you know nothing about this math. Every symbol is explained the first time it appears. Nothing is skipped for brevity. If a sentence uses a word from the glossary, that word was already defined before you got there.

0. Glossary (read this first, refer back anytime)

• Tensor: just a grid of numbers. A model “weight” is a tensor — think of it as a big spreadsheet of numbers the model multiplies your input by.
• Quantization: shrinking those numbers from a precise format (16-bit, lots of detail) to a coarser one (e.g. 5-bit, much less detail) to save memory. This necessarily loses some precision — the question this whole document is about is how much it hurts, and how to round in the way that hurts least.
• Calibration data: a small batch of real text you feed through the model, purely to observe how it behaves — not to train it, just to watch it.
• Forward pass: running real data through the model, layer by layer, to get an output.
• Backward pass / gradient: after a forward pass, you can compute “if this specific number inside the model had been slightly different, how much would the final output have changed?” That sensitivity number is called a gradient. A big gradient means “this really matters”; a near-zero gradient means “this barely matters.”
• Matrix: a tensor with exactly two dimensions (rows and columns) — a table of numbers.
• Matrix multiply (@): a specific way of combining two matrices into a new one. You don’t need the formula memorized — just know it’s how a layer actually processes its input, and it’s also how we combine sensitivity information below.
• Transpose (.T): flip a matrix’s rows and columns. A.T is A turned sideways.
• Outer product: multiplying a column of numbers by a row of numbers to produce a full matrix. v @ v.T (a column times its own transpose) always produces a very special kind of matrix — see PSD below. This is the single most important fact in this document.
• Eigenvalue: a number that describes how much a matrix “stretches” space along one specific direction. Every real, symmetric matrix has a full set of these — think of them as “how much does this matrix’s energy point along direction #1, #2, #3, …”. A large eigenvalue = the matrix cares a lot about that direction. A near-zero eigenvalue = the matrix basically ignores that direction.
• Positive semi-definite (PSD): a matrix where every single eigenvalue is zero or positive — never negative. This isn’t a coincidence for the matrices in this document; it’s a mathematical guarantee, explained in Section 5.
• Hessian (as used here — NOT the calculus-class “matrix of second derivatives”): this project’s Hessian is a curvature/sensitivity matrix built by accumulating real outer products (see above) across many calibration examples. It answers: “across all the real text I fed through this layer, which directions in this weight matrix actually mattered to the model’s output, and by how much?”
• Condition number: the ratio between a matrix’s largest and smallest eigenvalue. A small ratio = “well-conditioned” (the matrix treats all directions fairly evenly). A huge ratio = “ill-conditioned” (a few directions dominate completely, and math done on this matrix becomes numerically fragile — small errors get amplified).
• bf16 (bfloat16): a 16-bit number format. Fast and memory-cheap, but coarse — roughly 3 decimal digits of real precision.
• fp32 (float32): a 32-bit number format. Slower and uses more memory, but much more precise — roughly 7 decimal digits of real precision.
• Frobenius error: the plainest possible way to measure “how different are these two matrices” — square every difference, add them all up, take the square root. It does NOT know or care which differences matter more to the model; it treats every number equally.
• Weighted error: unlike Frobenius error, this measures difference through the lens of the Hessian — a difference in a direction the Hessian says matters a lot counts for much more than a difference in a direction the Hessian says barely matters.

1. The actual problem, in one sentence

Rounding every weight in a model to fewer bits is easy. Rounding them in a way that doesn’t wreck the model’s real behavior is the hard part — and the Hessian is the tool that tells the rounding process which mistakes are cheap and which are expensive.

2. A tiny worked example with real numbers — no abstract symbols, just arithmetic

Everything above was names and rules. Here is the actual math, with tiny made-up numbers, worked by hand, so every symbol becomes a real number you can check yourself.

Setup: imagine a tiny layer with just 2 input features and 2 output features (a real layer has thousands — this is small on purpose so every number is checkable by hand). We run 2 calibration examples through it.

Example 1: real input x[1] = [1, 2]. Real backward-pass sensitivity for this example: grad_output[1] = [0.5, 0.1].

Step: compute G[1] = grad_output[1].T @ x[1]. .T turns the row [0.5, 0.1] into a column. Multiplying a column by a row (this is the “outer product” from the glossary) means: multiply every entry of the column by every entry of the row, laid out in a grid:

G[1] = [0.5] @ [1, 2]  =  [0.5×1, 0.5×2]  =  [0.5, 1.0]
       [0.1]              [0.1×1, 0.1×2]     [0.1, 0.2]

Example 2: x[2] = [3, 1], grad_output[2] = [0.2, 0.4].

G[2] = [0.2] @ [3, 1]  =  [0.2×3, 0.2×1]  =  [0.6, 0.2]
       [0.4]              [0.4×3, 0.4×1]     [1.2, 0.4]

Step: build H_in by accumulating G.T @ G across both examples. First, G[1].T @ G[1] — multiply G[1]’s transpose by itself. Each entry of the result is a sum of products (this is what matrix multiply actually does, mechanically):

G[1].T @ G[1] = [0.26, 0.52]      G[2].T @ G[2] = [1.80, 0.60]
                [0.52, 1.04]                       [0.60, 0.20]

Add them together (this is the real “accumulate across every calibration example” step):

H_in = [0.26+1.80, 0.52+0.60]  =  [2.06, 1.12]
       [0.52+0.60, 1.04+0.20]     [1.12, 1.24]

Notice something for free, not by luck: the top-right number (1.12) equals the bottom-left number (1.12). This is called being “symmetric,” and it’s not a coincidence — G.T @ G is always symmetric, for any real G, guaranteed by the same structural fact that guarantees PSD (Section 5). This matters because eigenvalues of a non-symmetric matrix can come out as complicated numbers that don’t even fit on a normal number line — symmetry is what guarantees every eigenvalue is a plain, ordinary real number you can compare as “bigger” or “smaller” or “negative.”

Now check the real eigenvalues of this real H_in. For a 2×2 matrix, there’s a direct formula (you don’t need to memorize it, just see that it’s a real, mechanical calculation, not guesswork): add the two diagonal numbers (2.06 + 1.24 = 3.30 — this sum is called the “trace”), and separately compute (2.06 × 1.24) − (1.12 × 1.12) = 1.30 (this is called the “determinant”). Plugging into the real formula for a 2×2 matrix’s eigenvalues gives:

eigenvalue 1 ≈ 2.84
eigenvalue 2 ≈ 0.46

Both positive — exactly as guaranteed, not by chance. This tiny example has the exact same structure as the real 5120×5120 Hessians in production; the only difference is size. When the real pipeline computed a Hessian this same way but in bf16 instead of fp32, this is the exact calculation where precision loss crept in — enough of it, on an ill-conditioned real tensor, to flip what should always come out positive into something measurably negative.

3. The whole pipeline, visually, before the step-by-step text

4. Step by step: exactly what the code does, milestone by milestone

Milestone 1 — Get real calibration text flowing through the model

The code runs a handful of real sentences through the actual model, layer by layer, just like normal inference. Nothing is trained here. This is purely “watch the model work on real input.”

Milestone 2 — For ONE weight tensor at a time, capture two real things

Pick one specific weight matrix you’re about to quantize (say, one attention layer’s output projection). While the calibration text flows through:

• Capture x: the real input that arrived at this exact layer, for every calibration example.
• Capture grad_output: after a full forward pass, MLX can also tell you — going backward from the final output — exactly how sensitive that final output is to changes in THIS layer’s output, for every calibration example. This is the real gradient described in the glossary.

Both x and grad_output are real, measured matrices — not estimates, not assumptions.

Milestone 3 — Combine them into one real sensitivity matrix per example

For each calibration example b, the code computes:

G[b] = grad_output[b].T @ x[b]

In plain words: “take how much the final output cares about this layer’s output, and combine it with what actually came into this layer, for this one example.” The result, G[b], is a matrix the same shape as the weight tensor itself — it’s a real “sensitivity map” for this one specific calibration example.

Milestone 4 — Build the real Hessian by accumulating outer products across ALL examples

This is the step this whole investigation was actually about. For every example, the code computes:

H_in  = H_in  + G[b].T @ G[b]
H_out = H_out + G[b] @ G[b].T

Why two Hessians, not one? A weight matrix has two sides — rows and columns. H_in tracks sensitivity along the input side (which incoming directions matter), H_out tracks it along the output side (which outgoing directions matter). YAQA is a “two-sided” method specifically because it corrects using BOTH — that’s what “two-sided Hessian-weighted correction” means in every other document in this project. GPTQ, covered in Section 6, only ever builds the input-side one, which is exactly the structural difference that matters later.

Two things to understand about the accumulation step itself:

1. G[b].T @ G[b] and G[b] @ G[b].T are both outer-product-style constructions — a matrix multiplied by its own transpose. This is the single fact that guarantees H_in/H_out are PSD (Section 5) — as long as the arithmetic used to compute them is precise enough. This “as long as” clause is exactly where the real bug lived.
2. This is a running total, summed across every calibration example. H_in/H_out end up being real accumulated sensitivity matrices — “across ALL the real text we showed the model, here’s the total picture of which directions in this weight matrix actually mattered.”

Milestone 5 — Eigenvalues: what “checking the Hessian’s health” actually means

Once H_in/H_out are built, you can compute their eigenvalues (defined in the glossary). Concretely, for a matrix like this, an eigenvalue answers: “if I nudge the weight matrix along this specific direction, how much does the accumulated sensitivity respond?” A big eigenvalue = a direction the model is genuinely sensitive to. A tiny eigenvalue = a direction the model barely notices.

The critical fact: because H_in/H_out are built purely from outer products (G.T@G and G@G.T), real mathematics guarantees every eigenvalue must be zero or positive — never negative. This isn’t a rule of thumb; it’s provable: an outer product v @ v.T applied to any real vector v can never produce a negative “energy” value, no matter what v is. Summing many such outer products (across all calibration examples) can’t change that — you can’t add up a series of non-negative amounts and get something negative.

This is exactly why checking eigenvalues is a real diagnostic tool, not a guess. If you actually compute the eigenvalues of a real H_in/H_out and find a large negative number, that is not “the model is unusual” — it is mathematically impossible for a correctly-computed Hessian of this form. A large negative eigenvalue is proof, not a symptom, that the computation itself was corrupted somewhere before you ever got to looking at eigenvalues.

Milestone 6 — This is precisely how the BF16 bug was caught

Real numbers, from the actual investigation on tensor layers.14.linear_attn.in_proj_qkv:

• Computed in bf16 (the bug): H_in min eigenvalue = -5.618, H_out min eigenvalue = -7.534. Both impossible in exact math — real, measured proof of corruption, not noise.
• Computed in fp32 (the fix): min eigenvalues shrink to about -0.0003 / -0.0008 — six to seven orders of magnitude smaller. This tiny residual negativity is exactly what you’d expect from ordinary floating-point rounding noise, not real corruption. This is what “healthy” looks like: technically not perfectly zero, but small enough to be obviously just rounding dust.
• A completely independent float64 (even more precise) computation matched the fp32 result to 4 decimal places — proof the fp32 fix reached the actual correct answer, not just a different wrong one.

Milestone 7 — Condition number: why THIS tensor, specifically, was so vulnerable

Separately from “is it PSD,” you can ask “how lopsided is this Hessian” — the condition number (glossary). layers.14.linear_attn.in_proj_qkv had a real, measured ill-conditioned Hessian: a few directions dominated heavily over the rest. This matters because ill-conditioning amplifies whatever precision error already exists — a small bf16 rounding mistake gets stretched much larger by an ill-conditioned matrix than it would on a well-conditioned one. This is why THIS specific tensor showed a catastrophic 300x error while most other tensors, computed under the exact same bf16 bug, didn’t show nearly as dramatic an effect — they simply weren’t as ill-conditioned, so the same precision loss didn’t get amplified as much.

Both facts are true at once, and they don’t contradict: ill-conditioning explains WHY this tensor was so sensitive; bf16 precision loss is the actual mechanical cause of the corrupted numbers; fixing precision (not just adding a safety-net fallback) is what actually fixed it, and fixed it for every tensor, not just this one.

Milestone 8 — How the Hessian actually gets USED to round better

Once you have a clean, healthy H_in/H_out, the actual rounding step (block-LDL / Cholesky factorization — the real correction math) uses it to decide, weight by weight, in what order and by how much to adjust each rounding choice so that whatever error can’t be avoided gets pushed into directions the Hessian says the model barely notices, instead of directions it cares about a lot. This is the entire point of the method — naive rounding treats every weight equally; Hessian-weighted rounding treats them according to real, measured sensitivity.

Milestone 9 — How “did this actually help” gets measured

Two different numbers get compared, and they mean different things:

• Frobenius error (glossary): plain, unweighted “how different are the two weight matrices.” YAQA can legitimately make this WORSE while still being a real improvement — it’s deliberately trading raw numerical distance for less impact on the model’s real output.
• Weighted error: the real number that matters — how much does this rounding choice actually affect the model’s output, according to the real Hessian. This is what YAQA is actually trying to minimize, and it’s the number that should get better after a correct correction.

5. The concrete, no-jargon proof that PSD is guaranteed (for anyone who wants the “why”, not just the rule)

Take any single real number g. Squaring it, g × g, can never be negative — negative times negative is positive, positive times positive is positive, and zero times zero is zero. There’s no way to square a real number and get a negative result.

An outer product v @ v.T is the matrix version of exactly that same squaring operation, just done all at once for every pair of entries in v. Every single number inside the resulting matrix is built the same way “squaring” is — which is why the whole matrix inherits the same guarantee: it can be added together with other such matrices (as H_in/H_out do, across every calibration example) and the result can never develop a genuinely negative eigenvalue. It’s the same reason you can never add up a list of squared numbers and get a negative total.

6. lm_head / GPTQ: same concept, simpler formula, same real fix already applied

lm_head’s correction uses a real, different (but related) method — GPTQ, not YAQA. Its Hessian is simpler: one-sided only, H = x.T @ x (no separate output-side H_out, and no gradient/backward-pass step at all — just the raw input, squared against itself the same outer-product way as above).

Directly answering the question of whether the FP32 fix was actually ported here: yes, verified directly in the real code (06_lm_head_gptq.py, FP32Catcher class) — xf is cast to float32 BEFORE the outer product xf.T @ xf, the identical fix pattern used in YAQA. This was NOT missed or left half-done.

So why does lm_head still fall back to naive quantization? Not a precision bug — a real, structural limitation, confirmed by actual measurement (not assumed): GPTQ’s one-sided Hessian throws away the “how much does the model’s final output actually care” information that YAQA’s two-sided H_out captures. On this specific tensor’s real data, even after trying the default damping and three escalating fallback levels (up to 1000x stronger), GPTQ’s correction never beat plain naive rounding (frob_gptq=15.72 vs. frob_naive=10.89 at default settings — GPTQ was worse, and stayed worse at every damping level tried). The safety gate correctly detected this and fell back to naive — which, important to repeat, is still a real, valid, working quantized tensor, just without a correction that would have helped it.

7. The one-paragraph summary, if you read nothing else

The Hessian is a real, measured record of “which directions in this weight matrix actually mattered to the model’s output, across real calibration text” — built by adding up outer products, which mathematically guarantees every eigenvalue must be zero or positive. When bf16 precision was used instead of fp32 to build that record, the arithmetic became imprecise enough to produce impossible negative eigenvalues on one particularly lopsided (ill-conditioned) tensor — a real, provable, measurable form of data corruption, not a vague “something’s off.” Switching to fp32 fixed the arithmetic itself (proven correct by matching an independent float64 computation), which is why the fix works for every tensor, not just the one that happened to show visible symptoms.

© 2026 Hakim Ghelab, VegaLaboratories LTD. All rights reserved.