---

name: extreme-quantization-mastery description: "Advanced sub-8-bit quantization methodologies, including AWQ, GPTQ, and FP8 utilization on Hopper architectures."

Hardware Acceleration: Extreme Quantization

Mathematics of Sub-8-bit Quantization

Precision reduction targets $W$ (weights) and $A$ (activations) to compress memory footprint and accelerate compute-bound matrix multiplications.

• GPTQ (Post-Training Quantization): Utilizes inverse Hessian approximation to iteratively quantize weights, minimizing the mean squared error (MSE) of layer-wise outputs. Mathematically solves $\min_Q ||WX - QX||"_2^2$ via Cholesky decomposition.
• AWQ (Activation-aware Weight Quantization): Identifies a subset of salient weights based on activation magnitudes (quot;|X|"$). Applies optimal scaling to protect salient weights, exploiting the Pareto principle where <1% of weights dominate output variance.

Activation vs Weight Quantization

• Weight-Only (e.g., INT4/INT8): Alleviates memory bandwidth bottlenecks in memory-bound autoregressive decoding.
• Weight-Activation (W8A8): Accelerates compute-bound prefill phases by mapping GEMMs entirely into INT8 Tensor Cores.

FP8 on NVIDIA Hopper Architecture

Hopper introduces FP8 (E4M3 and E5M2 formats).

• E4M3: 4 exponent bits, 3 mantissa bits. Optimal for forward pass (Weights/Activations) due to higher precision.
• E5M2: 5 exponent bits, 2 mantissa bits. Offers expanded dynamic range, crucial for gradient propagation in backward passes. Transformer Engine seamlessly casts FP16/BF16 to FP8, leveraging hardware-native dot-products while mitigating overflow via delayed scaling.

Architecture Flow

%%{init: {"theme": "default", "flowchart": {"useMaxWidth": true}}}%%
flowchart TD
    A[FP16/BF16 Model] --> B{Quantization Target}
    
    B -->"|Weights Only| C[Memory-Bound Optimization]
    C --> D1[GPTQ: Hessian-based Inverse]
    C --> D2[AWQ: Activation-aware Scaling]
    D1 --> E[INT4/INT8 Weights]
    D2 --> E
    
    B -->|Weights & Activations| F[Compute-Bound Optimization]
    F --> G[Hopper FP8 Support]
    G --> H1[E4M3 Forward Pass]
    G --> H2[E5M2 Backward Pass]
    H1 --> I[FP8 Tensor Cores]
    H2 --> I