Qwen3.8 06 · Attention Heads and GQA

What actually improved the result

The progress graph reflects a clear pattern: reducing key/value-head duplication became increasingly effective once the model used four query heads.

Starting from six query heads, switching from six to four key/value heads produced only a small improvement, from 1.250712 to 1.250518. Reducing the query-head count from six to four then reached 1.249585.

The substantial gains came from sharing key/value heads more aggressively:

• 4 query heads, 2 key/value heads: 1.238625
• 4 query heads, 1 key/value head: 1.235098

The final configuration is multi-query attention: all four query heads share one key head and one value head. It was the strongest configuration measured and should be retained for this benchmark.

Why fewer key/value heads helped

The experiments were motivated by the expectation that fewer key/value heads would reduce attention memory traffic and computation, allowing more useful training within the fixed time budget. The validation results are consistent with that explanation: each reduction from four to two and then one key/value head produced a strict improvement.

However, throughput and memory consumption were not recorded in the evidence. The benchmark establishes that the shared-head configurations achieved better final validation loss in five minutes; it does not quantify how much faster they ran or how much memory they saved.

The result is specific to this short, compute-constrained training regime. It does not establish that one key/value head has universally better modeling quality when training time or compute is held differently.

More query heads were counterproductive

Increasing the query-head count from six to eight worsened the loss to 1.275232, compared with the then-best 1.250518. In this time-limited benchmark, any extra capacity or finer attention partitioning failed to compensate for the associated cost.

The useful lesson is not simply “always use fewer heads.” It is that this model and hardware budget favored a compact four-query-head layout, followed by aggressive sharing of keys and values. Only one higher query-head count was tested, so the evidence does not map the full range of possible head counts.

Recommended configuration and remaining unknowns

Use 4 query heads and 1 key/value head as the new benchmark configuration. It delivered the lowest measured loss and improved strictly over every accepted predecessor.

No head-dimension experiment is present, so the best head dimension remains unknown. Likewise, no configurations below four query heads were evaluated, and the search stopped after reaching ten experiment records. A focused continuation should keep the winning four-to-one layout as its reference and test head dimension or nearby query-head counts one at a time while recording training throughput and peak memory alongside validation loss.

Several experiments changed model depth rather than attention-head structure. Increasing depth from eight layers to nine or ten produced much worse completed results, between 1.305881 and 1.368537; another nine-layer run was stopped before completion. These trials do not answer the requested head-structure question, but they do reinforce that adding capacity was unhelpful within the five-minute budget.