Qwen3.8 20 · Compile and Kernel Efficiency

What the benchmark established

The opening baseline used the runner-managed RTX 3090 batch profile and reached 1.245730 in the fixed five-minute evaluation. As the progress graph shows, none of the subsequent completed runs lowered that value.

This matters because the benchmark rewards useful training accomplished within a strict time budget, not model capacity or theoretical throughput in isolation. A change that makes each step more expensive can produce a worse final loss even if the model is larger or an individual kernel is faster.

Larger models consistently lost under the time limit

Increasing the transformer from eight to nine layers was measured several times, producing losses of 1.313253, 1.313461, 1.312003, and 1.313470. Increasing it to ten layers was worse still at 1.342928. Every result was above the 1.245730 baseline.

The useful lesson is not merely that a particular depth failed: adding capacity did not pay for its additional computation within five minutes. The small variation among repeated nine-layer runs did not change that conclusion.

These depth experiments also fell outside the requested implementation-only search space. They changed model architecture rather than testing an equivalent, kernel-friendlier implementation. Repeating them consumed much of the ten-record budget without answering the central compile-and-kernel-efficiency question.

Activation recomputation did not justify a larger batch

Combining gradient checkpointing with an increase in per-device batch size from 32 to 64 produced a loss of 1.355166, the worst completed result. Checkpointing saves activation memory by recomputing intermediate values during the backward pass; here, that extra work did not translate into better five-minute training progress.

Because checkpointing and batch size were changed together, the run cannot identify which change caused the regression or how much each contributed. A later repetition of the same combined idea was stopped before producing a measurement. The actionable result is therefore narrow: do not adopt this combined configuration on the evidence available. If revisited, checkpointing and batch size should be tested separately so that each experiment contains one coherent variable.

The compile-mode tests remain unresolved

Two relevant implementation-level hypotheses were attempted, but neither yielded a valid score:

• The maximum-autotuning mode without CUDA graphs ran kernel autotuning, but its output did not satisfy the evaluator’s required terminal-summary format. There is no loss measurement, so its effect is unknown.
• The reduced-overhead mode, intended to use CUDA graphs and reduce launch overhead, failed during backpropagation. The captured traceback is incomplete, so the precise failure and any possible performance benefit remain unknown.

These are failed measurements, not evidence that either compile mode is slower or less accurate. They should first be made evaluator-compatible and stable, then rerun unchanged against the baseline.

What to keep and what to test next

Keep the eight-layer baseline and its existing device batch size. No strict improvement supports changing them.

The next research budget should concentrate on untested, objective-preserving implementation changes: repair and isolate each compile mode; test fused operations one at a time; compare tensor layouts with identical model mathematics; and try equivalent kernel-friendly formulations. Record both a valid final loss and, where available, training progress or throughput so that a faster implementation can be distinguished from a merely different one.

The current evidence cannot determine whether compilation, fusion, layout changes, or equivalent kernel reformulations can improve the benchmark. Most of those requested areas were never successfully measured before the ten-record limit was reached.