Why the same call returns different results on different machines.

The digits that move

A program computes the eigenvalues of a matrix on a workstation and again on a server, or on the same server after a library update. The two answers agree to fifteen digits and differ in the sixteenth. Every test still passes. Nobody can say which answer is right, because both are, to within the precision of the arithmetic.

In an experiment that is harmless. It stops being harmless the moment a result has to be reproduced by someone else, re-run to settle a dispute, or signed off and then shipped on hardware that was not in the lab. Then the question is not whether the sixteenth digit matters but why it moved at all, and the honest answer is that nothing in a tuned library promises it will not.

Why the order of additions matters

A double-precision number holds about sixteen significant digits. Every addition and multiplication rounds its result back to that width. So when three numbers are added, adding the first two and then the third can round differently from adding the last two first. For three numbers the difference is rarely visible. A dot product with a million terms, grouped one way on one machine and another way on another, will usually differ in the last bit, and an eigensolver is built out of millions of such dot products.

Neither grouping is wrong. Both are within the error bound the algorithm guarantees. The problem is only that there are two of them.

The three things that change the order

• The machine. A tuned library inspects the processor at run time and picks the kernel written for it: wider vector registers, a fused multiply-add instruction, a different cache blocking. Each kernel groups the arithmetic differently. A newer chip generation in the same rack can take a different path through the same library.
• The compiler. The same source compiled by GCC and by Clang, or by one compiler at two optimisation levels, can contract a multiply and an add into one rounded operation on one build and two on the other, or vectorise a loop in a different order.
• The library. A LAPACK call served by the reference implementation, by OpenBLAS and by Intel MKL is three different programs. Swapping the library underneath an application is an ordinary operations decision, and it silently changes the answer.

Running on several cores adds a fourth way the order can change, but the three above move the digits even on a single core, and they are the ones that move in real deployments: a new server, a toolchain upgrade, a different vendor's math library on the target.

Eigenvectors have a second problem

An eigenvector is only defined up to its sign: if v is an eigenvector, so is -v. When several eigenvalues coincide, any combination of their eigenvectors is also valid. Two correct libraries can therefore return visibly different eigenvectors for the same matrix without either rounding differently at all. This is a matter of convention rather than arithmetic, and the fix is a convention too: fix the sign of each vector by a rule before comparing. The fingerprints on our determinism page are taken after that canonicalisation, so they test the arithmetic and not the convention.

Reproducible modes, and where they stop

Tuned libraries offer opt-in reproducibility modes. They work by refusing to pick the fastest kernel and settling on one the whole machine family can run, which costs speed and holds only inside a boundary: one instruction-set family, one dispatch path. The mode exists because the library is not reproducible by default. It cannot carry an answer from an Intel server to an Arm board, because the two do not share a kernel to settle on.

What it takes to make the digits stop moving

Remove the choices. VVE EigenCore has one code path, no library underneath it to swap, and no kernel table to dispatch from. It computes in portable IEEE-754 double precision and asks the compiler not to fuse operations behind its back. The same source builds to the same arithmetic on x86_64 and aarch64, under GCC and under Clang, so there is nothing for the machine to change.

That is a design decision, not a tuning flag, and it was measured rather than asserted. Every record below is a SHA-256 fingerprint of the full result, compared across builds.