Attention, feed-forward networks and MoE

After decoding, look at the representation computation that precedes logits. Follow a position through attention, feed-forward transformations and expert routing. Weight residency and KV storage are covered next, so architectural structure and deployment capacity can be reasoned about separately.

How a Decoder Layer Updates Representations

This article uses the common structure of an autoregressive Transformer decoder as an example. Token IDs are first mapped into vectors; the model also needs to represent positional information. Positional information can be introduced via positional embeddings or position-related transformations on vectors in attention, and not all models simply add the two embeddings together.

A layer typically contains attention, a feed-forward network (FFN), residual connections, and normalization. The diagram below illustrates the pre-norm form for ease of understanding; original Transformers and subsequent models differ in normalization placement, activation functions, and positional mechanisms. Attention Is All You Need

Attention: How Current Representations Leverage Other Positions

For the input matrix X, linear projections yield Q=XWQ, K=XWK, V=XWV. The query at a specific position performs a dot product with keys from allowed positions, followed by scaling, masking, and softmax to obtain weights, which are then used to compute a weighted sum of values. A single head is often written as:

Attention(Q,K,V) = softmax(QKᵀ/√dₖ + M)V

The causal mask M prevents future positions from participating in the current position's attention, typically allowing access to the current position and all preceding positions. The representation at position i is used to predict subsequent tokens, so including the current position does not constitute peeking at unknown future information. Multi-head attention divides this computation into multiple groups and then merges the results; one cannot conclude that a specific head is responsible for fixed semantics or fully explain the model's judgment based on a single attention map.

Consider a masking example with three positions: if the original scores for the second position against all three positions are zero, the third position becomes inaccessible after masking, and the first two positions each receive a weight of 1/2. If the corresponding values are [2,0], [0,4], and [99,99], the result is [1,2], preventing large numbers from future positions from interfering. Only the third position is qualified to combine information from the first three positions.

FFN, Residuals, and Layer Depth

The FFN typically performs a linear transformation on the vector at each position, applies non-linear activation, and projects back to the model dimension; modern implementations often use gating mechanisms. "Per position" here means this sub-layer does not directly mix information across positions, but the input already carries information aggregated by previous attention layers. Attention and FFN jointly constitute representation computation across multiple layers; it is an oversimplification to divide them into "attention only looks, FFN only thinks."

Residual connections add the sub-layer's increment back to the original representation, providing direct paths for information and gradients; normalization helps control numerical scales. Increasing the number of layers changes the depth of sequential transformations, while increasing the number of experts primarily changes the set of selectable parameters; these two types of "scaling up" are not equivalent. The number of activated parameters is also not a measure of the number of inference steps within the model.

How does masking change the weighted result?

Dot products and scaling produce relative scores, masking determines eligibility, and softmax allocates weight. Keep q at zero to isolate causality, then change q to inspect contributions. Highlighted weights do not constitute a complete explanation of a model.

$${\alpha }_i={\mathrm{softmax}}_i(q{K}^{\top }/\sqrt{d_k}+M),o=\sum _i{\alpha }_i{v}_i$$

From Dense FFN to Sparse Experts

In a dense FFN layer, all positions use the same set of feed-forward weights. In a sparse MoE layer, multiple expert networks are prepared, and a router selects a subset based on the current position's representation. Other components, such as attention, still perform their own computations; the model is not simply cut into several mutually exclusive sub-models.

Suppose the router assigns weights [0.6, 0.3, 0.1] to three experts. If the layer selects the top-2 and normalizes within the selected set, the weights become [2/3, 1/3]. If the two selected experts output [3,0] and [0,6] respectively, the merged result is [2,2]. The third expert did not participate in the expert computation for this position, but it can be selected for the next position or a lower layer.

This is just one form of gating: top-k selection, weight normalization methods, capacity control, and which layers use MoE all depend on specific designs. Switch Transformers demonstrated top-1 sparse routing; Mixtral's publicly available architecture selects two feed-forward experts per token per layer. Switch Transformers, Mixtral of Experts

Experts Do Not Automatically Become Manually Defined Disciplines

"Expert" is simply the name of a sub-network; it does not imply that the system is pre-configured with math experts, legal experts, and code experts. Training may produce certain specializations, but actual routing is also influenced by representations, positions, and optimization objectives; one cannot treat a single selection as a reliable topic classification result.

Load imbalance during training can cause a few experts to receive a large number of tokens, while other experts are under-trained; hotspots and waiting times may also form during serving. Designs can use auxiliary losses, routing constraints, and other methods to improve this, but "absolute uniformity per batch" does not necessarily align with task requirements. Some architectures introduce always-active shared experts to handle common knowledge and reduce redundancy; this is a specific architectural choice, not a mandatory component for all MoE models. DeepSeekMoE

A Parameter Account: Separating Capacity from Computation

Assume the non-expert part of the model has 1B parameters, and all MoE layers collectively have 2B parameters per expert path, with 8 paths in total, selecting 2 paths per position. Then the total parameters are 1 + 8×2 = 17B. According to this simplified metric, the parameters involved per token are approximately 1 + 2×2 = 5B. If the number of paths increases to 16 while still selecting 2, the total becomes 33B, but the activated metric remains approximately 5B.

This example illustrates that candidate capacity can be increased without proportionally increasing expert computation, but it does not prove that computation remains completely unchanged: router overhead, communication, and load balancing will all change. Actual model parameter statistics also involve metrics for embeddings, output heads, and shared experts; one should read the model report rather than guessing the total using "number of experts × size of a certain dense model."

Selection and weighting are separate

Router scores are not final mixture coefficients: select top-k, then normalize the selected values. Raising E1 changes the mixture; increasing k also changes executed expert count. Fixed expert outputs isolate routing from expert computation.

Separately Verify Architectural Information and Task Capabilities

Tool-calling failures should first distinguish between model output, chat template, tool protocol parsing, missing context, and tool execution exceptions. The presence of function names in text form, without a valid calling structure, is insufficient evidence that MoE activated parameters are too few; the model may not have been trained on the corresponding protocol, or the inference engine may not be using the correct template.

There is no universal activation parameter threshold where "7B can stably perform single-step and 2B can stably perform multi-step." Training data, post-training methods, context, decoding configuration, task difficulty, and tool feedback all affect results. Using parameter scale as a filtering criterion is acceptable, but treating it as a guarantee of success lacks evidence.

Continue with：Inference, KV cache and memory。