Qwen3.8 12 · QK Normalization and Scaling

What the result actually shows

The progress graph is flat because only the baseline completed evaluation. Experiments 2–10 all failed before returning a prediction-loss value; none could be accepted or meaningfully compared with the baseline.

This means the research did not establish that learned query/key scaling or attention-logit soft-capping is better, worse, or equal to the existing behavior. A failed evaluation is not evidence against the underlying hypothesis.

What was attempted

Most attempts added a learnable scale for each attention head after RMS-normalizing the query and key vectors. The goal was to retain normalization while allowing each head to adjust the strength of its attention scores. Several revisions tried to correct tensor-shape and broadcasting compatibility, but evaluation continued to exit with errors in the compiled attention path.

A second group applied a fixed soft cap of 30 to attention scores using tanh before softmax. This was intended to limit extreme scores and improve numerical stability. Those runs also failed during evaluation and produced no metric.

Because the failures were repeated rather than measured regressions, there are no rejected change families with valid loss results. There is likewise no evidence here about other permitted choices such as alternative normalization variants or fixed scaling values.

Why the search stalled

The immediate blocker was execution reliability, not model quality. The modified attention calculations repeatedly failed inside the compiled PyTorch/FlashAttention evaluation path. Candidate generation was also disrupted by repeated HTTP 404 responses because the requested local Qwen model was unavailable. After consecutive GPU evaluations failed without producing a metric, the run stopped to avoid an endless retry cycle.

Nine of the ten experiment records therefore consumed the experiment budget without yielding comparable measurements. Repeating either hypothesis again under the same conditions would not add useful evidence.

What to do next

Keep the baseline unchanged for now. Before resuming optimization, isolate and fix the evaluation failure with a minimal attention change, verifying tensor shapes, broadcasting, compilation, and compatibility with the fused attention kernel. That compatibility check should complete without using another measured experiment slot if the workflow permits it.

Once evaluation is reliable, rerun one hypothesis at a time—first either the per-head learned scale or the fixed soft cap—and compare its completed prediction loss directly with 1.248246. Until such a run finishes, the effect of query/key normalization and attention-logit scaling on this five-minute benchmark remains unknown.