Qwen3.8 01 · Learning Rate and Schedule
- Measurement
- Text prediction loss
- Started
- 1.2755
- Best
- 1.2474
The best completed configuration reduced validation prediction loss from 1.275487 to 1.247420. The winning recipe was simple: keep warmup disabled, use a cosine-shaped final learning-rate decay, and raise the Muon matrix-parameter learning rate from 0.04 to 0.06. That is an absolute improvement of 0.028067, or about 2.2% relative to the baseline.
- Started
- 1.2755
- Best
- 1.2474
What actually improved the benchmark
The progress graph reflects three strict improvements.
First, replacing the linear warmdown with a cosine decay lowered the score from 1.275487 to 1.268126. This was a modest but repeatable-direction improvement: easing the learning rate down with a cosine curve during the final phase produced a better endpoint than the original linear decay.
The larger gain came from the learning rate applied to Muon-managed matrix parameters (MATRIX_LR). Increasing it from 0.04 to 0.05 reduced the score sharply to 1.250147. Raising it once more to 0.06 produced the best completed result, 1.247420.
In practical terms, this five-minute training regime benefited from more aggressive updates to the matrix parameters. The original rate was conservative enough to leave useful progress on the table, while the cosine warmdown helped finish training more smoothly.
What did not work
Adding linear warmup consistently hurt performance. Warmup over the first 5% of training scored 1.323303 and 1.341375 in two runs, while a 10% warmup scored 1.335175. All were substantially worse than the configurations without warmup. For this short benchmark, spending part of the limited training budget ramping up the learning rate appears counterproductive.
Lowering the matrix-parameter learning rate also moved decisively in the wrong direction. A rate of 0.03 scored 1.310508, and 0.02 deteriorated further to 1.373706. Together with the improvements at 0.05 and 0.06, these results give a clear local signal: within the tested range, higher—not lower—matrix learning rates were preferable.
Recommended configuration and remaining uncertainty
Use the cosine final decay, leave linear warmup disabled, and set the Muon matrix-parameter learning rate to 0.06. This is the best configuration supported by a completed measurement.
The natural next test is 0.07, but that evaluation was stopped before producing a result. Consequently, the evidence does not establish whether 0.06 is the optimum or merely the best tested value. Momentum timing, final learning-rate fraction, and alternative schedule shapes were also not measured in the completed experiments, so no conclusion can be drawn about them.