Qwen3.8 18 · Auxiliary Prediction Objectives

What produced the measured gain

The graph shows that the meaningful numerical progress arrived late and came primarily from focal loss, not from the requested auxiliary objectives.

Replacing ordinary cross-entropy with focal loss using a focusing strength of 2 reduced prediction loss from 1.244497 to 1.160443. Making that focusing strength learnable, initially set to 2, improved it again to 1.035384. Initializing the learnable value at 3 produced the best recorded result, 1.019641.

In practical terms, focal loss reduced the influence of tokens the model already predicted confidently and concentrated more training pressure on difficult tokens. The parameter mattered: lowering the fixed strength from 2 to 1 was worse at 1.198029, whereas allowing it to adapt was substantially better. Starting the learnable version at 3 yielded a smaller additional improvement over the learnable-at-2 result.

There was also a modest softcap result. Allowing the model to learn its logit cap barely improved the baseline, reaching 1.247335. Starting the cap at 10 instead of 15 then reached 1.244497, but tighter values of 9 and 8 were slightly worse. This suggests a local optimum near 10 in this benchmark, although the gain was tiny compared with focal loss.

What did not help

The requested auxiliary-objective families were consistently unproductive in the completed runs.

A z-loss coefficient of 0.0001 produced 1.250016, and another z-loss attempt produced 1.249052—both worse than the applicable best result. A smaller z-loss attempt failed because the calculation referenced an undefined value, while the final z-loss experiment was stopped before a measurement. The evidence therefore does not establish that every sensible z-loss configuration is ineffective, only that no tested configuration delivered a strict improvement.

Shifted next-token auxiliary losses were worse whenever they completed: results included 1.251507, 1.545931, and 1.298306. Several related next-token and multi-token variants also failed during evaluation. The practical lesson is that simply shifting the targets and adding the resulting loss with weight 0.1 was neither reliable nor beneficial here.

Label smoothing also failed to provide evidence of improvement. One run did not complete, and the completed label-smoothed run reached 1.414856, substantially worse than the contemporary best.

Why the headline result cannot be accepted as requested

The winning sequence changed the model’s forward loss calculation, added model parameters, and altered the validation loss path. The softcap experiments likewise changed model behavior rather than adding only a training-time objective in the permitted location. These are material departures from the stated contract, not bookkeeping details.

The stopping condition was also not followed as described. The evidence contains 27 experiment records, including 19 with measured results, despite the requested stop after ten completed records. This does not invalidate the measurements themselves, but it prevents the run from being treated as a faithful execution of the specified research protocol.

Recommended conclusion

Treat 1.019641 as a promising out-of-scope focal-loss result, not as the answer to the constrained objective search. For a protocol-compliant conclusion, retain 1.247517 as the established baseline because the evidence contains no strict improvement that clearly preserves the required forward and validation behavior.

If the experiment is rerun, apply auxiliary terms only to the training loss outside the preserved model forward path, verify that evaluation still computes the original cross-entropy, and enforce the ten-completed-run limit. The existing evidence suggests deprioritizing the weight-0.1 shifted-target auxiliary loss and the tested z-loss settings; focal-style reweighting is the strongest lead only if the allowed search space is explicitly broadened.