Qwen3.8 08 · Batch and Accumulation

Why the smaller total batch won

The five-minute limit made optimizer-update frequency more valuable than processing more tokens before each update. Halving the total batch from 524,288 to 262,144 doubled the intended update frequency and delivered the only accepted improvement.

There was a clear limit to that benefit. Halving the total batch again to 131,072 produced losses of 1.188609 and 1.193304 in two runs. Both were worse than 1.181211. In practical terms, 262,144 was the useful middle ground: frequent enough updates to learn quickly within five minutes, but not so small that update quality deteriorated.

Larger batches moved in the wrong direction. Returning from 262,144 to 524,288 yielded 1.279783, while doubling the original total batch yielded 1.443623. The graph should therefore show one decisive improvement at 262,144 rather than a general rule that ever-smaller batches are better.

Device batch and accumulation

Increasing the device batch from 32 to 64 was intended to cut gradient accumulation from four steps to two and reduce overhead. Instead, the completed run reached 1.607232, the worst measured result in the search. An earlier attempt at the same change was stopped before producing a result.

Keep the device batch at 32 and retain the corresponding four accumulation steps. The evidence does not establish whether the larger device batch actually increased token throughput, only that its overall five-minute validation result was substantially worse. No out-of-memory result was recorded.

What to keep

Use a total batch of 262,144 tokens, a device batch of 32, and four gradient-accumulation steps. This is the only tested configuration that strictly beat the baseline.

The learning-rate changes did not rescue nearby configurations: increasing the embedding learning rate produced 1.193271, and increasing the matrix learning rate produced 1.187440. Those results reinforce the main lesson from this run: the meaningful gain came from balancing update frequency and batch quality, not from pushing throughput-related settings or compensating with a higher learning rate.

What remains unknown is whether values between 131,072 and 262,144, or between 262,144 and 524,288, could improve further. Actual tokens processed, optimizer-update counts, and throughput measurements were not included, so the mechanism is supported by the observed loss trend but cannot be quantified beyond the tested outcomes.