Architecture
Kimi K3.1 Pro Architecture Guide: 3T MoE, KDA Hybrid Attention, and 1M-Context Scaling
Deconstruct Kimi K3.1 Pro: how 69 Kernelized Delta Attention (KDA) and 24 Gated Sparse Attention (DSA) layers eliminate 1M-token KV-cache bottlenecks in 3T MoE models.
The 1M Context Wall: Why KV-Cache Explodes in Traditional Transformers
Support for a 1,000,000-token (1M) context window has transitioned from experimental curiosity to an operational requirement for enterprise agent workflows. However, deploying 1M context in production under standard Transformer architectures has long hit an insurmountable barrier: the memory footprint and bandwidth bottleneck of the Key-Value (KV) cache during autoregressive decoding.
In standard Multi-Head Attention (MHA) and Grouped-Query Attention (GQA), KV-cache grows linearly with sequence length. Generating each new token requires fetching hundreds of gigabytes of historical KV states from high-bandwidth memory (HBM) into on-chip SRAM, leaving Tensor Cores idle while waiting for memory transfers. While Multi-Head Latent Attention (MLA) compresses the KV dimension into a low-rank latent subspace, high concurrency over 1M tokens still causes severe throughput degradation.
| Attention Mechanism | KV-Cache Scaling | Computational Complexity | Memory Bandwidth Bound | Long-Context Retention |
|---|---|---|---|---|
| Standard Multi-Head Attention (MHA) | Linear O(L) footprint per token | Quadratic O(L^2) | Severe (Memory Bound) | Prone to softmax entropy dilution |
| Grouped-Query Attention (GQA) | Sub-linear O(L / G) footprint | Quadratic O(L^2) | High memory bound | Moderate retention over long sequences |
| Multi-Head Latent Attention (MLA) | Compressed latent O(L * d_latent) | Quadratic O(L^2) | Moderate memory bound | Solid retrieval within 128k - 256k |
| KDA + Gated DSA Hybrid (K3.1 Pro) | Constant O(1) in 69 KDA layers + Top-2048 in 24 DSA layers | Linear / Sub-quadratic | Minimal (Compute Bound achieved) | Zero-drift state tracking + pinpoint local retrieval |
With Kimi K3.1 Pro (~2.8 trillion total parameters, 104 billion active parameters per token), Moonshot AI discarded the full Gated MLA architecture of Kimi K3, shifting to an asymmetric hybrid topology: 69 Kernelized Delta Attention (KDA) layers paired with 24 Gated DeepSeek Sparse Attention (Gated DSA) layers. Architectural details can be explored on the Hugging Face model repository.
69 Layers of Kernelized Delta Attention (KDA): O(1) Memory and Error-Driven Updates
Out of 93 total transformer layers, Kimi K3.1 Pro assigns 69 layers to Kernelized Delta Attention (KDA). KDA bridges linear attention and state space models (SSMs). Traditional linear attention methods historically suffered from memory fading over long contexts because straightforward additive accumulation washes out fine-grained temporal distinctions.
KDA resolves this via two mathematical mechanics:
- Kernel Feature Mapping: Non-linear mapping functions φ(Q) and φ(K) project queries and keys into an orthogonal feature space, uncoupling softmax normalization and enabling associative matrix reordering.
- Delta Rule Error Correction: Instead of naively adding incoming key-value outer products to the state, KDA tracks a prediction error: Δ = Vt - St-1 φ(Kt). The state matrix S is updated only by the unexpected residual scaled by an adaptive learning rate βt.
The pseudocode below illustrates how KDA updates historical state in constant memory:
import torch
def kda_step(hidden_state, query, key, value, beta, phi):
"""
Conceptual representation of O(1) constant-memory state update in KDA.
hidden_state shape: (batch_size, dim_k, dim_v) - fixed regardless of sequence length.
"""
phi_k = phi(key) # Kernel feature projection
phi_q = phi(query)
# 1. Calculate prediction error against current persistent state (Delta Rule)
predicted_val = torch.matmul(hidden_state, phi_k)
residual_error = value - predicted_val
# 2. Update state matrix; memory footprint remains constant from token 10 to 1,000,000
state_delta = beta * torch.outer(residual_error, phi_k)
new_state = hidden_state + state_delta
# 3. Retrieve output embedding for the current step
step_output = torch.matmul(phi_q, new_state)
return step_output, new_stateBecause the state matrix dimension is independent of sequence position, the memory consumption of these 69 layers remains constant O(1) whether processing 1,000 tokens or 1,000,000 tokens.
24 Layers of Gated DSA: High-Resolution Local Retrieval Over 1 Million Tokens
While KDA maintains macro-level context with constant memory, purely linear mechanisms struggle with pinpoint associative lookups, such as resolving a variable definition across 80,000 lines of source code. Kimi K3.1 Pro addresses this by interspersing 24 layers of Gated DeepSeek Sparse Attention (Gated DSA) across key structural depths.
Gated DSA functions as a two-stage dynamic addressing engine:
- 32 Low-Dimensional Index Heads: Rather than performing dense dot-product attention over the full 7,168 hidden dimension, dedicated index heads (head dimension 128) evaluate keys on a low-dimensional manifold. This prunes away 99.8% of irrelevant context, routing only the Top-2,048 candidate keys to the full softmax computation.
- Input-Dependent Soft Gating: A gating vector acts as a dynamic signal-to-noise filter: O = Gate(X) ⊙ SparseAttention(Q, K, V). When executing local syntactic reasoning, the gate attenuates background long-range noise; when performing cross-document correlation, the gate opens to incorporate global sparse signals.
By bounding fine-grained attention to a strict Top-2,048 budget, computational cost remains capped even when ingesting massive multi-hour video feeds or sprawling codebases.
3T MoE Topology and Hardware Co-Design: 896 Experts, MXFP4 QAT, and Native Vision
The attention breakthrough operates in tight synergy with Moonshot AI's MoE routing strategy and quantization-aware training (QAT):
| Architectural Dimension | Specification | Engineering Role |
|---|---|---|
| Total Parameters | ~2.8 Trillion (3T Class) | Massive parametric capacity for cross-domain reasoning |
| Activated Parameters | 104 Billion per Token | Sparse computation preserving high inference throughput |
| Layer Topology | 93 Layers (1 Dense + 69 KDA + 24 DSA) | Asymmetric hybrid pipeline balancing memory and precision |
| MoE Configuration | 896 Total Experts (16 Routed + 2 Shared) | Fine-grained functional specialization across downstream tasks |
| Context Length | 1,048,576 Tokens (1M) | Ingests multi-module code repositories and longitudinal dossiers |
| Quantization Format | Native MXFP4 weights / MXFP8 activations | Trained with QAT to prevent post-hoc precision degradation |
| Vision Stack | MoonViT-V2 (401M Parameters) | Natively integrated vision encoder for text, images, and video |
Rather than applying post-training quantization (PTQ)—which frequently destabilizes attention scores in extreme sequence lengths—K3.1 Pro was trained end-to-end with micro-scaled 4-bit floating point (MXFP4) weights and MXFP8 activations. This allows the open weights to execute directly on modern tensor cores with minimal numerical drift, halving memory bandwidth demands compared to standard FP16 deployments.
Engineering Implications for Next-Generation Autonomous Agents
The emergence of Kimi K3.1 Pro signals a fundamental shift in how developers design autonomous coding and research agents:
- Moving Beyond Fragmented RAG: Chunking code repositories into arbitrary 500-token vectors frequently severs circular dependencies, interface implementations, and type hierarchies. With 1M context backed by KDA memory efficiency, agents can maintain entire multi-repository workspaces directly in working memory.
- Rigorous Benchmark Verification: Joint reinforcement learning across hundreds of downstream domains delivers measurable jumps in complex benchmarks: ~51.7% on SWE-Marathon v1.1 (long-horizon software engineering) and ~65.7% on Terminal-Bench 4.0 (autonomous command-line troubleshooting).
- The Future of Hybrid Attention: The bifurcation into linear macro-state retention (KDA) and bounded sparse retrieval (DSA) demonstrates that solving the 1M context barrier does not require brute-forcing quadratic attention. Modern AI engineering increasingly favors asymmetric, hardware-aware architectures that decouple memory capacity from computational cost.
When architecting 1M-token production workloads, engineers can review model rates on the pricing page, explore multi-model routing setup in the quickstart guide, or create an account at register to benchmark throughput directly.