Can a standard notebook run the massive 744B GLM-5.2 model without a high-end GPU? The answer is a resounding yes, and it can even tackle the 2.8T Kimi K3 with just 32GB of RAM. The sensation on GitHub is Colibrì, an open-source project written in pure C with zero engine dependencies, which has rapidly amassed 32k Stars. It is a tiered #inference framework designed to run gargantuan models on consumer-grade hardware.
Colibrì leverages the architectural nature of MoE (Mixture of Experts) models. Take GLM-5.2 as an example: despite its total 744B parameters, only about 40B are active during the generation of any single token. Colibrì's logic is brutal yet brilliant: since the vast majority of experts are idle at any given step, there is no need to keep them constantly residing in precious and expensive high-speed RAM or VRAM.
Instead, Colibrì divides the model weights. Constant dense components like Attention and Embedding (~17B parameters, or 9.9GB in int4) permanently reside in RAM. Meanwhile, the massive pool of 19,456 routing experts (~370GB) is relegated to the NVMe SSD. Only when the Router activates a specific expert does Colibrì dynamically fetch it from disk. Developer JustVugg compares this to a weight-level JIT (Just-In-Time) compiler, scheduling data dynamically across SSD, RAM, and VRAM.
To overcome SSD-induced latency—which initially dragged generation speeds down to 0.05-0.1 token/s—Colibrì implements a sophisticated tiered memory system. It employs an LRU (Least Recently Used) cache to retain hot experts in RAM. Crucially, by exploiting the correlation between adjacent layers, Colibrì predicts the next layer's expert routing with an impressive 71.6% accuracy, overlapping SSD I/O fetching with active computation.
For high-end configurations, Colibrì even supports dual-SSD parallel reading. By duplicating the weights across two drives, users can double the read bandwidth and push the hardware to its absolute limits, significantly speeding up GPU-less inference.
[AgentUpdate Depth Analysis] For a long time, deploying sophisticated local AI Agents has been bottlenecked by hardware: running highly capable LLMs locally requires massive VRAM, which is unavailable on edge devices. Colibrì's breakthrough shifts the paradigm from VRAM capacity constraints to storage bandwidth constraints. By treating high-speed NVMe SSDs as dynamic virtual VRAM, it leverages #MoE's sparse activation to its absolute limit. This "Weight JIT" pathway lowers the barrier for running trillion-parameter models on edge consumer hardware. While current I/O speeds present a latency penalty, future hardware advances like PCIe 6.0/7.0 and next-gen SSD controllers will close this gap. This directly empowers local, privacy-first Multi-Agent systems, enabling complex reasoning and swarm intelligence to operate continuously on local hubs without relying on expensive cloud GPU clusters.
