Architecture

Kimi K3.1 Pro Architecture Guide: 3T MoE, KDA Hybrid Attention, and 1M-Context Scaling

Deconstruct Kimi K3.1 Pro: how 69 Kernelized Delta Attention (KDA) and 24 Gated Sparse Attention (DSA) layers eliminate 1M-token KV-cache bottlenecks in 3T MoE models.

The 1M Context Wall: Why KV-Cache Explodes in Traditional Transformers

Support for a 1,000,000-token (1M) context window has transitioned from experimental curiosity to an operational requirement for enterprise agent workflows. However, deploying 1M context in production under standard Transformer architectures has long hit an insurmountable barrier: the memory footprint and bandwidth bottleneck of the Key-Value (KV) cache during autoregressive decoding.

In standard Multi-Head Attention (MHA) and Grouped-Query Attention (GQA), KV-cache grows linearly with sequence length. Generating each new token requires fetching hundreds of gigabytes of historical KV states from high-bandwidth memory (HBM) into on-chip SRAM, leaving Tensor Cores idle while waiting for memory transfers. While Multi-Head Latent Attention (MLA) compresses the KV dimension into a low-rank latent subspace, high concurrency over 1M tokens still causes severe throughput degradation.

Attention Architectures at 1,000,000 Tokens
Attention MechanismKV-Cache ScalingComputational ComplexityMemory Bandwidth BoundLong-Context Retention
Standard Multi-Head Attention (MHA)Linear O(L) footprint per tokenQuadratic O(L^2)Severe (Memory Bound)Prone to softmax entropy dilution
Grouped-Query Attention (GQA)Sub-linear O(L / G) footprintQuadratic O(L^2)High memory boundModerate retention over long sequences
Multi-Head Latent Attention (MLA)Compressed latent O(L * d_latent)Quadratic O(L^2)Moderate memory boundSolid retrieval within 128k - 256k
KDA + Gated DSA Hybrid (K3.1 Pro)Constant O(1) in 69 KDA layers + Top-2048 in 24 DSA layersLinear / Sub-quadraticMinimal (Compute Bound achieved)Zero-drift state tracking + pinpoint local retrieval

With Kimi K3.1 Pro (~2.8 trillion total parameters, 104 billion active parameters per token), Moonshot AI discarded the full Gated MLA architecture of Kimi K3, shifting to an asymmetric hybrid topology: 69 Kernelized Delta Attention (KDA) layers paired with 24 Gated DeepSeek Sparse Attention (Gated DSA) layers. Architectural details can be explored on the Hugging Face model repository.

69 Layers of Kernelized Delta Attention (KDA): O(1) Memory and Error-Driven Updates

Out of 93 total transformer layers, Kimi K3.1 Pro assigns 69 layers to Kernelized Delta Attention (KDA). KDA bridges linear attention and state space models (SSMs). Traditional linear attention methods historically suffered from memory fading over long contexts because straightforward additive accumulation washes out fine-grained temporal distinctions.

KDA resolves this via two mathematical mechanics:

  • Kernel Feature Mapping: Non-linear mapping functions φ(Q) and φ(K) project queries and keys into an orthogonal feature space, uncoupling softmax normalization and enabling associative matrix reordering.
  • Delta Rule Error Correction: Instead of naively adding incoming key-value outer products to the state, KDA tracks a prediction error: Δ = Vt - St-1 φ(Kt). The state matrix S is updated only by the unexpected residual scaled by an adaptive learning rate βt.

The pseudocode below illustrates how KDA updates historical state in constant memory:

import torch

def kda_step(hidden_state, query, key, value, beta, phi):
    """
    Conceptual representation of O(1) constant-memory state update in KDA.
    hidden_state shape: (batch_size, dim_k, dim_v) - fixed regardless of sequence length.
    """
    phi_k = phi(key)      # Kernel feature projection
    phi_q = phi(query)

    # 1. Calculate prediction error against current persistent state (Delta Rule)
    predicted_val = torch.matmul(hidden_state, phi_k)
    residual_error = value - predicted_val

    # 2. Update state matrix; memory footprint remains constant from token 10 to 1,000,000
    state_delta = beta * torch.outer(residual_error, phi_k)
    new_state = hidden_state + state_delta

    # 3. Retrieve output embedding for the current step
    step_output = torch.matmul(phi_q, new_state)
    return step_output, new_state

Because the state matrix dimension is independent of sequence position, the memory consumption of these 69 layers remains constant O(1) whether processing 1,000 tokens or 1,000,000 tokens.

24 Layers of Gated DSA: High-Resolution Local Retrieval Over 1 Million Tokens

While KDA maintains macro-level context with constant memory, purely linear mechanisms struggle with pinpoint associative lookups, such as resolving a variable definition across 80,000 lines of source code. Kimi K3.1 Pro addresses this by interspersing 24 layers of Gated DeepSeek Sparse Attention (Gated DSA) across key structural depths.

Gated DSA functions as a two-stage dynamic addressing engine:

  • 32 Low-Dimensional Index Heads: Rather than performing dense dot-product attention over the full 7,168 hidden dimension, dedicated index heads (head dimension 128) evaluate keys on a low-dimensional manifold. This prunes away 99.8% of irrelevant context, routing only the Top-2,048 candidate keys to the full softmax computation.
  • Input-Dependent Soft Gating: A gating vector acts as a dynamic signal-to-noise filter: O = Gate(X) ⊙ SparseAttention(Q, K, V). When executing local syntactic reasoning, the gate attenuates background long-range noise; when performing cross-document correlation, the gate opens to incorporate global sparse signals.

By bounding fine-grained attention to a strict Top-2,048 budget, computational cost remains capped even when ingesting massive multi-hour video feeds or sprawling codebases.

3T MoE Topology and Hardware Co-Design: 896 Experts, MXFP4 QAT, and Native Vision

The attention breakthrough operates in tight synergy with Moonshot AI's MoE routing strategy and quantization-aware training (QAT):

Kimi K3.1 Pro Technical Specifications
Architectural DimensionSpecificationEngineering Role
Total Parameters~2.8 Trillion (3T Class)Massive parametric capacity for cross-domain reasoning
Activated Parameters104 Billion per TokenSparse computation preserving high inference throughput
Layer Topology93 Layers (1 Dense + 69 KDA + 24 DSA)Asymmetric hybrid pipeline balancing memory and precision
MoE Configuration896 Total Experts (16 Routed + 2 Shared)Fine-grained functional specialization across downstream tasks
Context Length1,048,576 Tokens (1M)Ingests multi-module code repositories and longitudinal dossiers
Quantization FormatNative MXFP4 weights / MXFP8 activationsTrained with QAT to prevent post-hoc precision degradation
Vision StackMoonViT-V2 (401M Parameters)Natively integrated vision encoder for text, images, and video

Rather than applying post-training quantization (PTQ)—which frequently destabilizes attention scores in extreme sequence lengths—K3.1 Pro was trained end-to-end with micro-scaled 4-bit floating point (MXFP4) weights and MXFP8 activations. This allows the open weights to execute directly on modern tensor cores with minimal numerical drift, halving memory bandwidth demands compared to standard FP16 deployments.

Engineering Implications for Next-Generation Autonomous Agents

The emergence of Kimi K3.1 Pro signals a fundamental shift in how developers design autonomous coding and research agents:

  1. Moving Beyond Fragmented RAG: Chunking code repositories into arbitrary 500-token vectors frequently severs circular dependencies, interface implementations, and type hierarchies. With 1M context backed by KDA memory efficiency, agents can maintain entire multi-repository workspaces directly in working memory.
  2. Rigorous Benchmark Verification: Joint reinforcement learning across hundreds of downstream domains delivers measurable jumps in complex benchmarks: ~51.7% on SWE-Marathon v1.1 (long-horizon software engineering) and ~65.7% on Terminal-Bench 4.0 (autonomous command-line troubleshooting).
  3. The Future of Hybrid Attention: The bifurcation into linear macro-state retention (KDA) and bounded sparse retrieval (DSA) demonstrates that solving the 1M context barrier does not require brute-forcing quadratic attention. Modern AI engineering increasingly favors asymmetric, hardware-aware architectures that decouple memory capacity from computational cost.

When architecting 1M-token production workloads, engineers can review model rates on the pricing page, explore multi-model routing setup in the quickstart guide, or create an account at register to benchmark throughput directly.