Qwen3.8 19 · Data Order and Curriculum
- Measurement
- Text prediction loss
- Started
- 1.2483
- Best
- 1.1134
The best measured prediction loss was 1.113377, down from the 1.248332 baseline—an improvement of about 10.8%. However, this result came from reducing the training batch size, not from a sampling-order or curriculum strategy, so the requested research question remains only partially answered.
- Started
- 1.2483
- Best
- 1.1134
What produced the improvement
The useful signal was optimizer-step frequency within the fixed five-minute RTX 3090 budget. Reducing the total batch from 2^19 to 2^18 tokens lowered prediction loss to 1.176265. Reducing it further to 2^16 tokens produced the best result, 1.113377.
The intermediate 2^17-token batch scored 1.180995, slightly worse than 2^18, so smaller batches were not uniformly better. Still, the strong result at 2^16 suggests that, under this short time limit, taking more optimizer steps can matter more than retaining a large effective batch.
The progress graph reflects these two strict improvements: first from the baseline to 1.176265, then to 1.113377.
What did not help
A 5% learning-rate warmup raised prediction loss to 1.371018, substantially worse than the baseline. In this short run, spending part of the budget ramping up the learning rate was counterproductive.
Training on a fixed sequence length of 512 while retaining 2048-token evaluation produced 1.252396, also slightly worse than the baseline. This was not a true progressive sequence-length curriculum: it tested only constant short-sequence training. It therefore does not establish whether gradually increasing sequence length would work.
Attempts to reduce the batch below 2^16 tokens did not produce measurements because the requested batch sizes were incompatible with the number of tokens processed by each forward/backward cycle. Increasing model depth from 8 to 12 also produced no result because evaluation exceeded the 600-second watchdog. These failures provide no evidence about prediction quality.
Practical conclusion
If changes outside the stated sampling-order scope are acceptable, retain the 2^16-token total batch, which achieved the lowest validated loss. Do not retain the warmup, fixed 512-token training length, or 2^17-token batch, because none strictly improved the current best.
For the original curriculum question, no supported recommendation can yet be made. The nine recorded experiments did not test deterministic batch ordering, progressive difficulty proxies, or an actual sequence-length schedule. A follow-up should return to those allowed variables, beginning with one coherent progressive sequence-length curriculum and keeping the successful 2^16 batch fixed as the comparison configuration. It should also choose batch sizes divisible by the per-step token count so experiments complete and yield comparable measurements.