🚨 UC Berkeley just open-sourced FreeToken.
It claims 2–4× faster local LLM inference than Ollama, and the wild part is the models it can run:
• Qwen3.6-35B on 8GB VRAM → 39.3 tok/s
• DeepSeek-V4-Flash 284B on 32GB VRAM → 22 tok/s
• GLM-5.2 753B on 96GB VRAM → 14.9 tok/s
How? These are Mixture-of-Experts models. A 35B model doesn't actually use all 35B parameters for every token.
FreeToken keeps the experts in system RAM and intelligently decides whether a missing expert should be sent to the GPU or computed on the CPU.
The good part is the best strategy depends on your exact machine. A 5090 desktop and an 8GB laptop may want completely opposite approaches.
It also checkpoints agent context, so coding agents don't repeatedly prefill thousands of unchanged tokens.
Open weights don't mean much if nobody can afford the hardware to run them. FreeToken is attacking that gap.
📄 Paper: https://arxiv.org/pdf/2608.16157
💻 Repo: https://github.com/FlashML-org/FreeToken
1
1August 31, 2026 90
