Background & Context§
The exponential growth of large language models (LLMs) has created a widening gap between model size and consumer-grade hardware capabilities. While the industry has largely relied on quantization, distillation, or pruning to compress models, these methods often sacrifice accuracy or require substantial engineering effort. AirLLM, an open-source project by Gavin Li, offers a radically different approach: instead of compressing the model, it leverages layer-wise loading to dramatically reduce memory footprint. This allows 70B models to run on a single 4GB GPU, and with the latest v3.0 update, even 671B DeepSeek-V3 and the 2.8-trillion-parameter Kimi K3 fit on modest hardware. This breakthrough democratizes access to state-of-the-art AI, enabling developers and researchers without high-end clusters to experiment with frontier models.
The News: What Happened Exactly§
AirLLM v3.0, released in June 2026, brings support for FP8 models and the latest architectures, but the headline feature is the ability to run DeepSeek-V3 (671B parameters) on approximately 12GB of VRAM and Qwen3-235B on just ~3GB. This is achieved through a combination of block-wise quantization and a unique inference strategy that loads only one layer at a time onto the GPU. The project now supports a wide range of models, including Qwen3, Llama 3.x/4, DeepSeek V2/V3, Phi-4, Gemma, and more, all through a single AutoModel class.
The key innovation is that AirLLM never holds the entire model in GPU memory. Instead, it decomposes the model into layers and streams them from disk as needed. The memory footprint is therefore proportional to the size of a single layer, not the total model size. For example, a 70B model with 80 layers would only need memory for one layer at a time, which is why it fits in 4GB. This technique is particularly effective for Mixture-of-Experts (MoE) models like DeepSeek-V3 and Kimi K3, as only the experts activated by a given token are loaded, rather than the entire layer.
The most recent update, dated July 2026, adds support for Kimi K3, the largest open-source model released to date with 2.8 trillion parameters. According to the project's documentation, Kimi K3 runs on a single card in just 3.72GB of VRAM, measured end-to-end on an RTX 6000 Ada. This is made possible by per-expert streaming, which loads only the experts that a token actually routes to. However, running K3 requires specific dependencies: pip install compressed-tensors flash-attn, a CUDA 12 build of torch (no prebuilt flash-attn wheel exists for CUDA 13), and transformers 4.56.x, as the model's remote code does not load on 5.x.
The code example below shows how simple it is to run massive models:
from airllm import AutoModel
model = AutoModel.from_pretrained("deepseek-ai/DeepSeek-V3") # 671B, runs in ~12GB
# model = AutoModel.from_pretrained("Qwen/Qwen3-235B-A22B") # 235B, runs in ~3GB
# model = AutoModel.from_pretrained("moonshotai/Kimi-K3") # 2.8T, runs in <4GBThe project has evolved significantly since its initial release in November 2023. Version 2.0 introduced model compression based on block-wise quantization, which speeds up inference by up to 3x with negligible accuracy loss. Unlike traditional quantization that quantizes both weights and activations, AirLLM's bottleneck is disk loading, so it only quantizes weights, making it easier to maintain accuracy. The compression can be applied simply by specifying compression='4bit' or '8bit' when loading the model.
Throughout 2024 and 2025, AirLLM added support for numerous models, including Llama 3.1 405B, Qwen2.5, and CPU inference. In December 2023, it introduced MacOS support, allowing 70B models to run on Apple Silicon. The project also added an AutoModel class that auto-detects model type, removing the need to specify the model class.
Historical Parallels & Similar Incidents§
AirLLM's approach echoes earlier attempts to make large models accessible on low-end hardware. One notable parallel is the llama.cpp project, which emerged in March 2023. LLaMA.cpp focused on running LLaMA models on CPU using 4-bit quantization (GGML/GGUF formats), enabling models like LLaMA-7B to run on a Raspberry Pi. The project gained immense popularity for its efficiency and portability, but it still required massive RAM for large models; for instance, LLaMA-65B needed around 40GB of RAM, far beyond consumer laptops.
Another parallel is DeepSpeed's ZeRO-Offload (2021), which offloaded model states to CPU and disk to train 100B+ models on a single GPU. While primarily for training, it demonstrated the feasibility of heterogeneous memory utilization. However, ZeRO-Offload was complex to configure and primarily targeted data scientists with enterprise hardware.
AirLLM differentiates itself by focusing on inference and simplicity, requiring only a single pip install airllm and a one-line API call. Unlike llama.cpp, which requires converting models to a specific format and often fails with unexpected architectures, AirLLM works out-of-the-box with most Hugging Face models. It also achieves more aggressive memory reduction by streaming layers, not just using quantization, which allows it to handle models that are orders of magnitude larger (e.g., 2.8T parameters in 4GB).
A recent precedent is KTransformers (2024), a project that optimizes MoE models by offloading experts to CPU and GPU, enabling 45B models on 24GB GPUs. However, AirLLM's per-expert streaming takes this further by loading only the experts needed for each token, reducing memory to the size of a single expert's weights. This is crucial for models like Kimi K3, which has over 1000 experts per layer; loading all of them would be impossible, but streaming them individually makes it feasible.
Lessons from these parallels include the importance of developer experience: llama.cpp's early versions were cumbersome, but AirLLM's AutoModel abstracts away complexity. Additionally, the success of community-driven projects underscores the demand for accessibility. AirLLM's sponsorship model and active GitHub community indicate a sustainable ecosystem, and its frequent updates reflect responsiveness to user needs, such as adding CPU inference and MacOS support.
Technical Innovations§
AirLLM's core innovation lies in layer-wise streaming combined with block-wise quantization. The layer-wise streaming decomposes the model into individual layers and loads them sequentially onto the GPU, processing activations as they arrive. This reduces VRAM usage to the size of a single layer's parameters plus activations. For MoE models, the system loads only the experts that are activated by the current token, further reducing memory to a fraction of the layer size.
Block-wise quantization, introduced in v2.0, compresses model weights in small blocks (e.g., 64 or 128 elements) and quantizes each block independently. This approach preserves accuracy better than whole-tensor quantization because it adapts to the distribution of weights in each block. The compression reduces disk load time by up to 3x, which is a critical bottleneck in AirLLM's architecture since layers are loaded from disk continuously.
AirLLM also employs prefetching (v2.5) to overlap disk loading with compute, achieving a 10% speed improvement. The v3.0 update adds FP8 support, which further reduces memory and accelerates inference on compatible hardware.
The project also supports safetensors and non-sharded models (v2.10.1), ensuring compatibility with a wide range of model repositories. For users with gated models, a hf_token parameter is provided.
The codebase is well-documented, with configuration options for layer shard saving paths and compression settings. The example notebooks and MacOS guide provide practical demonstrations. The project's FAQ addresses common issues like disk space errors during model splitting and tokenizer padding problems.
In summary, AirLLM v3.0 represents a significant leap in making frontier LLMs accessible to individual developers. By combining layer-wise streaming, block-wise quantization, and support for MoE architectures, it effectively eliminates VRAM as a barrier to entry, paving the way for more inclusive AI experimentation.