WASTE: Running Trillion-Parameter Models on Consumer Hardware

The Impossible Made Possible

SQLite Cloud has released WASTE (Weight-Aware Streaming Tensor Engine), an open-source inference engine that achieves something previously thought impossible: running the complete, unmodified Kimi K3 model — a 2.78 trillion parameter mixture-of-experts model — on a consumer laptop with just 29 GB of RAM.

This isn’t a distilled, pruned, or reduced variant. It’s the full model, running at 0.49-0.54 tokens per second on a MacBook Pro. This represents a fundamental breakthrough in making frontier-scale AI accessible to individual developers and researchers.

The Problem: The Memory Wall

Kimi K3 is massive. At full precision, it weighs 1.42 terabytes. Even after conversion to an optimized format, it’s 982 gigabytes. Most consumer machines have 16-64 GB of RAM. The math doesn’t work.

Traditional approaches to this problem involve:

• Quantization: Reducing precision, which hurts quality
• Pruning: Removing parameters, which changes the model
• Distillation: Training a smaller model, which is expensive and time-consuming
• Cloud APIs: Paying per token, which gets expensive fast

WASTE takes a different approach: it recognizes that not all parameters need to be in memory at the same time.

The Key Insight: Mixture of Experts

Kimi K3 is a mixture-of-experts (MoE) model. This means that for each token, only a small fraction of the model’s parameters are actually used. Specifically, K3 activates about 4% of itself per token.

This is the crucial insight: if 96% of the model is idle at any given moment, it doesn’t need to be in RAM. It needs to be reachable in time.

WASTE keeps the model trunk (the parts that are always used) in memory and streams the experts (the parts that are sometimes used) directly from disk. The remaining RAM becomes a bounded expert cache that holds frequently-used experts.

The Architecture

The Container Format

WASTE converts models into a .waste container with a specific layout:

• A JSON manifest describing the model
• A resident trunk (the always-used parameters)
• One expert bank per layer
• Each expert record is 4 KiB-aligned with its gate, up and down matrices adjacent

This layout is crucial: routing to an expert costs exactly one pread() call — not three, not a seek per matrix. The arithmetic was never the bottleneck; I/O was.

Direct I/O and Cache Bypass

WASTE uses cache-bypass I/O (O_DIRECT on Linux, F_NOCACHE on macOS, FILE_FLAG_NO_BUFFERING on Windows). This is deliberate: with a 982 GB model and 64 GB of RAM, the kernel would cache everything, creating a false sense of performance. WASTE needs to measure real I/O performance.

Residual Vector Quantization

Experts are stored using residual vector quantization — three stages of 256-entry codebooks over 8-dimensional vectors, achieving 3.00 bits per weight. The matrix is never materialized; instead, for each token, the engine builds a table of partial dot products, after which every expert row is three table reads and two adds.

The trunk stays at 4 and 8 bits because the model was trained with quantization-aware training on experts only — it has no tolerance for a squeezed trunk.

Performance: The Numbers

On a MacBook Pro M5 Pro with 64 GB of RAM:

Kimi K3 (2.78T parameters, 982 GB container)

• Minimum RAM: 29.05 GB at 4K context
• Resident trunk: 27.28 GB
• Read per token: 17.0 GB
• Model load: 20 seconds
• Decode speed: 0.49–0.54 tokens per second
• Vision tower: 15.7 seconds for a 1024-patch image

Kimi-Linear (48B parameters, 19 GB container)

• Minimum RAM: 1.87 GB
• Decode speed: 10.7 tokens per second at 78% cache hit

The Memory Optimization Story

WASTE’s memory design is fascinating. The most predictive number in the entire project is that K3 touches 16 experts in each of 92 layers per token: 17.0 GB. Below this, an expert cached for one token is evicted before the next token asks for it, and the hit rate is zero.

Crossing this threshold buys about 8% of throughput (0.50 to 0.54 tok/s) because read-ahead already hides most of the I/O the cache would have saved.

But there’s a ceiling on the other side. At 52 GB on a 64 GB machine, the engine stops fitting, and the OS pages out the expert cache. A “hit” becomes a page fault, and throughput drops 20x.

The usable window is narrow: it opens at ~46 GB and closes by 52 GB. When WASTE picks a budget for itself, it steps down a whole working set at a time and takes the largest that fits under seven-eighths of RAM.

The I/O Reality

Where the time goes in decode on K3:

• MoE (all of it): 82.5%
  • Expert I/O: 53.5%
  • Expert matmul: 20.0%
• KDA layers: 14.5%
• MLA layers: 2.8%
• lm_head: 0.2%

The I/O already runs near the hardware limit — 17.0 GB per token at ~9.9 GB/s against the SSD’s measured 12.78 GB/s. It only gets cheaper by happening less often, which means cache, which means RAM.

Multimodal Support

WASTE includes a 401M ViT (Vision Transformer) with 27 layers and patch 14. Images are encoded into embeddings that are spliced into the token sequence. A 448×336 image becomes 192 image tokens; a 896×896 photo becomes 256 tokens.

The tower is loaded only when an image is present, and its weights (434 MB) come straight out of the expert cache budget.

Platform Support

WASTE builds and runs on:

• macOS arm64: Yes, with NEON SIMD
• Linux arm64: Yes, with NEON SIMD
• Linux x86_64: Yes, with AVX2 SIMD
• Windows x86_64: Yes, cross-compiled with MinGW-w64

SIMD is selected at runtime from CPUID, so a single x86 binary uses AVX-512 where it exists and AVX2 where it doesn’t.

The Conversion Process

Converting Kimi K3 takes about 4.7 hours with three processes on the M5 Pro (23.7 with pure PyTorch). The process is resumable — a layer whose bank is already written is skipped.

The download is the part that goes wrong. A 1.42 TB pull over hours will hit dropped connections and CDN 5xx errors. WASTE’s downloader resumes mid-file, retries with exponential backoff and jitter, and counts as done only when size matches Content-Length.

What’s Not There Yet

WASTE is honest about its limitations:

• Chat format support is limited to models whose format has been transcribed (K3 today)
• AVX-512 compiles but hasn’t executed (the test runner doesn’t have it)
• Windows builds and runs on one toolchain and one CPU
• The expert checksum is off by default (5% throughput cost)
• The trunk has no checksum at all

The Broader Implications

WASTE represents a shift in how we think about AI inference. Rather than asking “how do we make the model smaller?” it asks “how do we make the I/O faster and smarter?”

This approach opens up possibilities:

• Researchers can run frontier models locally without cloud APIs
• Developers can iterate on models without per-token costs
• Organizations can maintain data privacy by running models on-premises
• The economics of AI shift from “pay per token” to “pay once for hardware”

The Cost Analysis

At 0.5 tok/s and 42W sustained power consumption, the cost to run K3 is approximately $5 per million tokens (assuming 20¢/kWh electricity). This doesn’t include hardware amortization, but it’s still competitive with cloud APIs for heavy users.

Getting Started

WASTE is open source (Apache 2.0) and available on GitHub. For those wanting to try it without committing a terabyte of disk space, Kimi-Linear-48B runs in just 19 GB and achieves 10.7 tok/s.

Pre-converted containers are on their way to Hugging Face, which will eliminate the conversion step entirely.

Conclusion

WASTE is a remarkable achievement in systems engineering. By combining careful I/O optimization, residual vector quantization, and a deep understanding of mixture-of-experts models, the team has made frontier-scale AI accessible to individual developers.

This isn’t just a technical accomplishment — it’s a philosophical shift. The future of AI isn’t necessarily about making models smaller or slower. Sometimes it’s about making the systems that run them smarter.

For researchers, developers, and organizations that need frontier-scale AI without cloud dependencies, WASTE is a game-changer.

Visit the WASTE GitHub repository to learn more and get started.