Deciding which experts to drop

Every mixture-of-experts model has experts that barely earn their parameters. The tempting move is to rank experts by how often the router picks them and delete the bottom few. That ranking is wrong, and the way it is wrong is instructive.

Utilisation alone gets it backwards

Consider two experts. One takes 4% of routed tokens and is the only expert that handles a particular kind of input. The other takes 22% and is nearly identical to a sibling that takes 30%. Utilisation says drop the first. Capability says drop the second — its work is already covered, while the first is the only thing standing between you and a hole in the model.

So the pruner computes two numbers per expert, not one, and the plan comes from their interaction.

Utilisation, from a seeded router

The first number is share of routed tokens. Getting it honestly would mean running the model, which the browser cannot do. Instead the page runs a seeded simulation: synthetic token embeddings pass through the actual gate matrix read from the checkpoint, softmax, take top-k, and tally which experts win.

This is a simulation, and the page says so wherever the number appears. The gate weights are real — read from the file you loaded — but the tokens are synthetic, so the load distribution is representative of the router's geometry, not of your actual traffic. An expert that specialises in a domain absent from the synthetic distribution will look idle when it isn't.

The simulation is seeded, so the same checkpoint always produces the same board. That matters more than it sounds: an explainability tool whose numbers move between refreshes teaches you nothing.

Redundancy, from weight signatures

The second number asks a different question: if this expert vanished, is there another that does approximately the same thing?

Comparing two experts properly would mean comparing their full weight matrices — tens of millions of parameters each, which is neither loadable nor necessary. Instead each expert is reduced to a signature: a strided down-sample of its weights into a fixed-length vector, using the same striding machinery the visualiser uses for heatmaps. Then it's cosine similarity between every pair.

sim(i, j) = (sᵢ · sⱼ) / (‖sᵢ‖ ‖sⱼ‖)

Two experts trained to do the same job end up with weights pointing in similar directions, and this catches that at a cost of a few thousand floats per expert instead of tens of millions. It is a proxy — a coarse one — but it is a proxy that reliably surfaces genuine near-duplicate pairs, which is what the decision needs.

The keep-score

The two numbers combine into a single ranking: normalised utilisation, minus a penalty proportional to how similar an expert is to its most-used twin.

keep-score = normalised utilisation − redundancy penalty high use, no twin → keep, obviously high use, close twin → the pair is doing one job between them low use, no twin → the dangerous one: cheap but load-bearing low use, close twin → prune first

The asymmetry matters. An expert's penalty comes from its similarity to a more heavily used twin, so of a near-identical pair the busier one survives and the quieter one is a candidate. Penalise both symmetrically and you would happily delete both halves of a pair, which is how you lose a capability while congratulating yourself on the parameter savings.

The plan, and the number you should distrust

A slider sets aggressiveness; the page takes the lowest keep-scores up to that fraction and reports experts pruned, parameters cut, expert-weight cut, and an estimated capability retained.

That last figure is a heuristic, and it is labelled as one everywhere it appears. It is derived from the share of routed tokens whose preferred expert was removed, softened by how redundant that expert was — under the simulated gate, over synthetic tokens. It is a plausibility indicator, not an evaluation. Anyone who prunes a production model on the strength of a number computed in a browser tab deserves what happens next; validate against a real eval.

What the tool is genuinely good for is the shape of the answer: which experts cluster, whether the router's load is lopsided, and roughly how much of the model is duplicated effort. Those conclusions survive the approximations. The retention percentage does not.

The report

The download button generates a standalone HTML report — the ranking, the plan, the parameters, the caveats — assembled as a string in the page and handed to the browser as a Blob. Nothing is uploaded to produce it. You can hand the file to a colleague and it opens with no dependencies, which felt like the right shape for an artefact whose entire argument is "here is what I measured and here is what I inferred."