Qwen3.8 14 · Weight Decay and Regularization
- Measurement
- Text prediction loss
- Started
- 1.2472
- Best
- 1.2472
The baseline remained the best result: 1.247235 prediction loss. None of the nine regularization runs produced a strict improvement, so no weight-decay change should be kept from this research.
- Started
- 1.2472
- Best
- 1.2472
What the experiments established
The graph shows a flat best-so-far result because every measured change after the baseline was rejected. Those runs all applied essentially the same hypothesis: extend the existing global weight decay to the token embedding and output projection layers, which had previously been excluded.
Their prediction losses ranged from 1.281307 to 1.282610. Because lower is better, even the strongest of these results was about 0.0341 worse than the baseline. The repeated measurements varied slightly, but all landed in the same clearly inferior region.
In practical terms, decaying these two parameter groups made the model predict validation data less effectively within the fixed five-minute RTX 3090 benchmark. The evidence therefore supports preserving their weight-decay exclusion.
The useful lesson from the repeated failures
Although nine post-baseline experiments were recorded, they did not test nine meaningfully different regularization ideas. They repeatedly evaluated the same embedding and output-layer decay choice. That repetition does provide evidence that its failure was consistent, but it does not broaden the conclusion to weight decay or regularization in general.
The result should be interpreted narrowly: applying the existing global decay setting to the token embedding and output projection was harmful under this benchmark. It does not show that every decay strength, parameter grouping, decoupled-decay variant, or norm penalty would be harmful.
What to keep and what remains unknown
Keep the baseline optimizer treatment, including the exclusion of the token embedding and output projection from weight decay. There is no supported replacement configuration from these ten completed records.
The requested wider search was not completed. The evidence contains no measured comparison of different decay strengths, alternative parameter exclusions, changed decoupled-decay behavior, or simple norm penalties. Those choices remain unknown and would require separate experiments, with one genuinely distinct hypothesis per run.