Layer re-merging: recombining open models by depth

Open checkpoints are not interchangeable parts. They are trained on different data, with different objectives, at different scales, and nothing guarantees that a layer taken from one means anything to another.

They are also, empirically, more recombinable than that description suggests. This note is about how far that goes and where it stops.

Re-merging is the technique behind most of the model stack at Redrob Labs: open models get decomposed layer by layer, conflicting weights are filtered out, and what remains is recombined by depth. Four model families have come out of that process, including the image model that placed third worldwide among open-source models on the Artificial Analysis board, with weights public under Apache 2.0.

Why bother

The motivation is not elegance. Training a competent multilingual model from scratch is expensive in a way that does not fit every budget, and the open ecosystem now contains a large number of checkpoints that are each good at something and mediocre at everything else.

There is a harder constraint underneath the budget one. The product this stack serves has to answer a question for a fraction of a cent, in twelve Indian languages, at a price an Indian consumer actually pays. That rules out renting a frontier model and it rules out pretraining our own. Re-merging is the third option: same quality from a smaller model, which pulls cost and latency down together.

If capability specialises by depth, and if depth-wise specialisation is stable enough across checkpoints, then recombining layers is a way to assemble a model with a capability profile you want without paying for the pretraining that produced each piece.

That is a large "if", stacked on another one.

What specialises by depth

The first question is whether the premise holds at all, and here the literature and our own probing agree reasonably well.

Early layers do work that is close to lexical and syntactic. They are strongly tied to the tokeniser and to the surface form of the language, and they are the layers where multilingual models differ most from monolingual ones.

Middle layers carry most of what looks like transferable representation. They are the least tied to the specific vocabulary and the most tied to structure and meaning.

Late layers are heavily shaped by the training objective and by whatever alignment or instruction tuning came last. They are the most opinionated and the least portable.

That pattern suggested the obvious experiment: keep the early layers of a model strong in the target languages, take middle layers from models strong in reasoning, and re-tune the late layers.

What actually happened

Tokeniser mismatch is fatal and comes first. Two checkpoints with different tokenisers cannot share early layers in any meaningful sense, because the embedding space is indexed differently. This is obvious in retrospect and cost us time anyway. Everything downstream assumes a shared vocabulary, and there is no clever alignment that repairs a mismatch cheaply.

Middle layers transfer better than expected, but not blindly. Splicing middle layers from a differently trained checkpoint of the same family and tokeniser produced a working model more often than not. Splicing across families produced noise, even where dimensions matched.

Conflicting weights have to be filtered, not averaged. Where two donors disagree strongly about a parameter, the average of them is frequently worse than either. Dropping the low-agreement mass outright beat interpolating it in every comparison we ran.

Residual scale drifts and has to be corrected. Layers from different checkpoints have different activation magnitudes. Naive concatenation produces a model whose residual stream grows or collapses through the spliced region. A per-layer rescaling fitted on a small calibration set fixed most of this, and skipping it made every other result uninterpretable.

The join needs re-tuning, the rest does not. Adaptation concentrated at the boundaries. Tuning a few layers either side of each splice recovered most of the loss, at a fraction of the cost of tuning the whole model.

The result that surprised us

The interesting finding was not that recombination worked. It was that a recombined model was better in the target languages than either parent, on inputs neither parent handled well.

Our reading is that the early layers of a language-strong checkpoint make code-mixed and non-Latin input legible to middle layers that were never trained to see it, and those middle layers were more general than their training distribution implied. The capability was latent and the input was the blocker.

We hold that interpretation loosely. It is consistent with what we measured and it is not the only explanation.

Where it stops

Three limits worth stating plainly.

Recombination cannot add a capability that is absent from every parent. It reorganises what exists.

Quality is bounded by the weakest link in the chain, and the weakest link is usually the early layers, because they are the ones you cannot swap freely.

Evaluation is the real constraint. A recombined model can look fine on everything you thought to measure and be badly broken on something you did not, and the failure surface is larger than for a conventionally trained model because the failure modes are less familiar.

Practical guidance

If you are trying this: match tokenisers before anything else, filter disagreement rather than averaging through it, calibrate residual scale before drawing any conclusions, tune at the joins rather than globally, and build the evaluation before the model rather than after it. The last one is the one everybody skips.

This work runs jointly with Seoul National University and Yonsei, funded by Korean government research grants.