Paper Edits Attention With B-Spline Surfaces To Steer Model Output

The paper’s starting observation is a plain limit of standard transformer attention: a source token sends the identical value vector to every receiver, so the query can decide how much weight a token gets but not what gets pulled out of it. The author’s fix is to attach a learned two-dimensional B-spline surface to each value dimension, a small curved lookup table shaped by the query-key pair, initialised at zero so a freshly added model behaves exactly like the original until the surface is trained.

Applied to LLaMA 3.2 at 1B and 3B parameters, the surfaces cut WikiText-2 validation perplexity while adding a small fraction of extra parameters, and the paper reports that greedy-decoded output changed for a majority of a set of varied test prompts, with the change rate higher on the larger model and highest of all on prompts the author calls ambiguous or polysemous. The examples given include a corrected statement about the CAP theorem and a vague answer about quantum mechanics turned into a specific one naming position and momentum.

The second half of the paper treats the surfaces as a steering handle rather than just a quality fix. Inverting a layer’s coefficients changed output for nearly all of ten sampled prompts, and setting coefficients to negative one builds what the author calls an attention wall, blocking value flow through chosen dimensions. In one preliminary test, a layer-wide wall redirected a prompt asking for explosive-device instructions toward general educational content instead.

The numbers

The method adds a small parameter overhead, is evaluated on a set of 112 varied prompts, and produces a higher share of changed greedy outputs on the larger of the two tested models than on the smaller one, a gap the paper attributes to the larger model having more attention heads for the surfaces to attach to.

None of this is deployed anywhere yet. It is a single-author paper with experiments confined to two small open models, and the safety redirection result is described as preliminary rather than evaluated at scale. For any service whose whole product is a generated prose answer rather than a ranked list, a technique that edits specific factual errors after training, at this parameter cost and without retraining the whole model, is the kind of lever worth watching before it reaches a shipped system.