Model Architecture: Dense vs. MoE

Almost all modern LLMs are Transformer decoders. MoE has become the most important architectural trade-off in recent years—allowing models to have "large parameters, but only use a small fraction per token," decoupling capacity from per-token compute.

Overview

Almost all modern LLMs are Transformer decoders. The differences lie mainly in scale and how that scale is utilized: more parameters generally mean greater capability, but inference cost, VRAM usage, and latency per token also increase. MoE (Mixture of Experts) is the most important architectural trade-off of the past few years—it allows models to be "large in parameters, but only use a small fraction per token," decoupling "capacity" from "per-token compute." Understanding the mechanisms of dense and MoE directly impacts three practical matters: local deployment selection and VRAM budget (see Local LLM Deployment), inference throughput, and interpreting cutting-edge model cards like Qwen-AgentWorld.

First, let's clarify the boundary: closed-source models (Claude, GPT, etc.) do not disclose their architectures—whether they are MoE, how many layers, or how many parameters, the official sources remain silent. This article discusses general mechanisms + verifiable facts about open-source models; for closed-source models, we can only rely on the capabilities reported by the Models API, not speculation about internal structures. Wherever Claude is mentioned below, we only discuss capability queries, not structure.

How Transformer Decoders Work

flowchart TD
    E["token embedding + positional encoding"] --> R["residual stream"]
    R --> L1["Layer 1"]
    subgraph L1["Each Layer (×N)"]
        direction TB
        A["self-attention<br/>(Q·K → weights, weighted V)"] --> F["FFN<br/>(up-projection → activation → down-projection)"]
    end
    L1 --> LN["... Layer N"]
    LN --> O["output logits<br/>(next token distribution)"]

Key points (decoder-only, i.e., GPT lineage):

• Residual stream: The output of each layer = input + incremental change from that layer. Information flows along this "main trunk," being read and written by each layer as it progresses.
• Self-attention: Each token projects into three sets of vectors: Q, K, and V. The weights for "what to attend to" are calculated using Q·K, followed by a weighted sum of V. Multi-head attention runs multiple sets of Q/K/V in parallel to capture different relationships. The causal mask ensures that a token can only look at previous tokens—this enables autoregressive generation. Attention has ~O(n²) cost and is the source of KV cache (see Context Engineering).
• FFN (Feed-Forward Network): Independently performs "up-projection → non-linearity → down-projection" (typically 4× up-projection) for each position. Most of the model's parameters are in the FFN—this is where MoE makes its move.

Dense vs. MoE: Mechanisms

• Dense: Every token passes through all FFN parameters. For a 70B dense model, every token utilizes all 70B parameters.
• MoE: Replaces the FFN in each layer with N experts + a router (gating network). The router calculates a distribution for each token indicating "which experts to use," activating only the top-k experts, while the rest do not participate in computation.

Router mechanism (using modern MoE as an example): For each token, the gating network outputs scores for N experts. After softmax, the top-k are selected, and the outputs of the selected experts are merged according to gating weights. Two engineering key points:

• Load balancing: Without constraints, the router would favor a few experts ("winner takes all"), leaving other experts under-trained. During training, a load balancing auxiliary loss is added to encourage uniform routing.
• Shared expert: Some designs (DeepSeekMoE, Qwen) keep a always-active shared expert to handle general capabilities, while the remaining routed experts handle specialized tasks—this is where descriptions like "8 routed + 1 shared" come from.

The 35B-A3B in the naming means this: 35B total parameters, but only ~3B activated per token.

Why MoE: Decoupling Capacity/Compute (and Its Costs)

In one sentence: MoE trades "all experts resident in VRAM" for "only computing a small part per token"—large capacity, low compute cost (high throughput), but does not save VRAM.

A VRAM/Compute Calculation (Local Perspective)

Taking the Qwen 35B-A3B resident on my machine as an example (Local LLM Deployment):

• VRAM: All 35B experts must reside. FP16 requires ~70GB, which won't fit in a 24GB card → quantization is mandatory: Q3_K_XL compresses weights to ~17GB, adding a 64k Q8_0 KV cache (~2.85GB), plus ~1.7GB for runtime computation buffers, totaling ~21.5GB, leaving ~3GB headroom for the desktop. VRAM depends on "total params + KV cache"; MoE saves nothing here.
• Compute: Only ~3B are activated per token, so throughput is close to a 3B dense model—paired with MTP draft heads,实测 ~111 t/s. Speed depends on "active params"; this is the MoE dividend.

This is why choosing MoE locally allows you to fit a high-capacity model into affordable VRAM and run it fast—the premise is using quantization to squeeze "total params" into the card. But the cost is that per-token inference depth is determined by active parameters (see next section). For tasks requiring multi-step reasoning chains, such as agent/tool calls, factor this cost into your selection.

MoE's Hidden Cost: Per-Token Inference Depth

Above, we only calculated VRAM and throughput, but there is a third dimension: per-token inference depth is determined by active parameters, not total parameters.

Recall the Transformer structure:

The attention layer is complete—the model knows which input tokens to attend to. But the FFN is responsible for processing and reasoning on the attended information, and MoE only activates a small portion of FFN parameters per token. A model with 35B total params / 3B active params has a "thinking depth" per inference step roughly equivalent to a 3B dense model. In other words: Attention determines "where to look," activated FFN determines "what was understood and what to do next"—MoE's eyes are not blind, but the number of brain cells working per step is only as many as the active parameters.

This has little impact on single-step tasks (classification, summarization, simple QA)—3B processing capability is sufficient. But for multi-step serial reasoning—typical of agent tool call chains:

Every step consumes FFN's inference budget. With only 3B active parameters, the model starts "forgetting" the intent of the first step by the second step. The manifestation is: it outputs <think> text, but when it should produce tool call tokens, its attention scatters, continuing to hallucinate text—thinking, but not acting.

Empirical Thresholds: Stable single-step tool calls require ~7–10B active parameters; stable multi-step agent loops require ~20B+ active parameters. This is why DeepSeek V3 (MoE, 671B total params but 37B active) has very stable tool calls—its active parameters are large enough. Under a 24GB VRAM constraint, prioritize dense models (e.g., Qwen 2.5 14B/32B, Qwen 3 32B) for agent tasks; their active parameters equal total parameters, making multi-step reasoning chains much more reliable than low-active MoE models of similar size.

Instances: From Local to Cutting-Edge

• Local Resident: Qwen 35B-A3B (35B total / 3B active) — standard MoE, see calculation above.
• Cutting-Edge Variant: Qwen-AgentWorld-35B-A3B is also 35B-A3B, but more aggressive: 256 experts, 8 routed + 1 shared activated per token; and it is a hybrid architecture—40 layers heavily use Gated DeltaNet (a variant of linear attention that reduces long-sequence attention's ~O(n²) to near-linear, saving context overhead), with only the last layer of each repeated block using Gated Attention (only 10 out of 40 layers), thereby supporting 256K context. It stacks "MoE capacity decoupling" and "linear attention long context" together to serve its world-model goal (simulating agent environments requires long context + strong capacity, see the cutting-edge section in Agent Loops and Tool Use).

These numbers come from verifiable open-source model cards. Do not apply them to Claude—official sources have not disclosed whether it is MoE, its parameter scale, or its attention variants.

Selection: Look at Capabilities, Don't Guess Structure

For application developers, the real question is not "is it MoE," but capabilities and cost. For closed-source models, use the Models API to check in real-time, don't guess:

m = client.models.retrieve("claude-opus-4-8")
m.max_input_tokens   # Context window
m.max_tokens         # Max output
m.capabilities       # image_input / thinking / effort / structured_outputs ...
# You can also filter by capabilities using client.models.list()

For open-source/local models, read the model card's total params / active params / layers / context / attention type, and combine this with VRAM budget (Local LLM Deployment's quantization and VRAM trade-offs) to make a selection.

Best Practices

• Check capabilities for closed-source models, don't guess structure. Use the Models API to get max_input_tokens/max_tokens/capabilities in real-time; don't speculate on whether it's MoE or how many parameters it has.
• Estimate MoE VRAM based on "total params + KV cache". Small active params do not mean small VRAM—all experts must reside; use quantization to squeeze total params into the card.
• MoE throughput depends on active params, not total params. 35B-A3B runs like a 3B model; this is the MoE dividend.
• Evaluate agent adaptability based on active params, not total params. Stable single-step tool calls need ~7–10B active; stable multi-step agents need ~20B+; under 24GB, prioritize dense or high-active MoE for agent tasks.
• Read the four-piece set of open-source model cards. Total params / active params / layers / attention type, combined with VRAM budget (Local Deployment) to determine the tier.

Trade-offs and Failure Modes

• Mistaking "small active params" for "small VRAM": MoE VRAM depends on total params, which will blow up the card → estimate VRAM based on total params + KV cache, use quantization to compress.
• Estimating MoE throughput with dense intuition: MoE is fast because active params are small, not because it's "small" → look at active params.
• Dispersed experts within a batch: Concurrent requests route to different experts, so actual compute utilization may be lower than theoretical → engineering issues with server-side scheduling/expert parallelism.
• Using MoE for agent tool calls but frequently breaking chains: Active parameters are too small (e.g., 3B), causing intent drift in multi-step reasoning, producing think text but not actual tool calls → choose dense or high-active MoE for agent scenarios, evaluating based on active parameters, not total parameters.
• Speculating on closed-source architecture: Unfounded → only use the capability fields from the Models API.

References

• Papers: "Attention Is All You Need" (Vaswani 2017), "Switch Transformers" (Fedus 2021), "DeepSeekMoE" (2024, shared expert), "GShard" (load balancing)
• Model Cards: Qwen-AgentWorld-35B-A3B
• Closed-Source Capability Query: Anthropic Models API (platform.claude.com)

Keywords: Transformer, decoder-only, residual stream, self-attention, QKV, multi-head, causal mask, FFN, KV cache, dense, MoE, Mixture of Experts, router, gating, top-k, shared expert, load balancing, total parameters, active parameters, 35B-A3B, throughput, Gated DeltaNet, linear attention, Qwen-AgentWorld, Models API, quantization, closed-source architecture