When the VRAM on a consumer-grade graphics card can't fit an entire massive MoE model, many people assume the only option is to use the cloud.FlashML-org/FreeToken By loading expert weights into the host memory and having the GPU cache only hotspot experts, the goal is to run state-of-the-art open-source MoE models interactively on desktop machines. Here’s the conclusion first:FreeToken is suitable for local setups where “the model is larger than the GPU memory and you need an OpenAI-compatible interface to connect to an agent”; if the model already fits into GPU memory, don’t expect it to completely replace llama.cpp or vLLM.. This article isAggregate Structure: Positioning, project card, suitability (yes/no), minimum installation requirements, at least one use case connected to Codex or a compatible endpoint, and a brief list of pros and cons. Versions and commands are subject to the official README or documentation available on the day you access this page.

Positioning: What Edge-Native MoE Services Solve—and What They Don’t
FreeToken bills itself as an edge-native MoE serving solution: it treats individual machines as a unified, elastic inference platform rather than “miniature data center GPUs.” The core idea is that MoE uses only a subset of experts at each step, with the expert models themselves residing in host RAM and an LRU cache on the GPU side.
Suitable for:NVIDIA consumer-grade / workstation graphics cards; need to run open-source MoE locally that "exceeds video memory capacity"; require an Anthropic/OpenAI-compatible API to integrate with Codex, Claude Code, and self-built agents.
Not suitable for:No NVIDIA / non-Linux x86_64 (subject to official requirements); the model fits easily into video memory, and you care more about the maximum throughput per request—in public benchmarks, llama.cpp and vLLM are often still faster when they fit; No one wants to deal with the CUDA toolkit, drivers, and initial JIT compilation.
![]()
The hardware range described in the paper and README file spans from 8GB laptop GPUs to single-card workstations; specific token-per-second (tok/s) figures are subject to the authors’ experiments and community reproductions. This article does not present any specific benchmark results as a guarantee for your machine.
Things to Do Tonight:List your GPU model, video memory, and system RAM. If you're low on RAM, don't select the largest model under "Expert: Use All System RAM" just yet.
Project Card
| 项 | Content |
|---|---|
| Warehouse | https://github.com/FlashML-org/FreeToken |
| Documentation / Installation | docs/install.md; For the desktop version, visit flashml.ai |
| Morphology | CLI / Engine + Optional Desktop App; OpenAI/Anthropic-Compatible API |
| License | Subject to the LICENSE in the repository (please verify again before installation) |
| Suitable for | Local Ultra-Large MoE, Agent-Compatible Endpoints, Want to Interact Even With Insufficient GPU Memory |
| Not suitable | Hardware not supported by the software; scenarios where the system has sufficient space and requires maximum single-card throughput |

Things to Do Tonight:Open the "LICENSE" file and the "Requirements" section in install.md (drivers, CUDA, Python), and mark "not met" as a blocker—don't wait until you're halfway through the installation to complain.
Use Case: Local `ft serve` + OpenAI-compatible endpoint connected to the coding agent

Use Case: Local service + OpenAI-compatible endpoint connected to the coding agent.According to the official CLI, after installation, use the `serve` command or its equivalent as described in the documentation to launch a compatible endpoint, and then connect Codex or other OpenAI-compatible clients to it. base_url Go ahead and run a read-only task like "List the current directory and summarize the README."
Acceptance Criteria: The client must be able to complete a full cycle of a tool call; after disconnecting from the service, the client must explicitly report a failure rather than silently returning a fabricated result. The model names and quantization formats are subject to the list supported by FreeToken on that day.
Note: The open-source model-of-everything (MoE) ecosystem also includes large models such as Mistral Large, which are available on Hugging Face—whether they have been included in the FreeToken support matrix depends on the repository’s description on that day; do not assume that “all HF MoEs can be run with a single click.”

Things to Do Tonight:Just set up Maoyan—get the service running → make curl compatible with chat/completions (or the official example) → then connect to Agent. If curl isn't working, don't connect to Codex just yet.
Quick List of Pros and Cons: When to Still Use llama.cpp / vLLM
| Instructions | When to Use It |
|---|---|
| FreeToken | MoE is larger than the video memory, and it needs to support desktop interactivity and be compatible with the API for connecting to the Agent. |
| llama.cpp | The model fits within your existing, mature GGUF workflow, and you want it to be simple and manageable. |
| vLLM, etc. | Parallel services, clusters, or graphics memory policies have been built around them |
| Pure Cloud API | Local hardware/operations costs exceed billing costs, or compliance permits out-of-region operations |
Advantages:Addresses the "insufficient video memory" issue; features an Agent-friendly interface; includes installation documentation and desktop options.
Disadvantages:Hardware and driver limitations; more capacity doesn’t necessarily mean faster performance; expert caching and PCIe have become new bottlenecks; requires someone to manage version updates.
Things to Do Tonight:Use the table above to map out a main path; before the main path starts to show signs of strain, do not run all three engines at the same time—"try them all."
Installation and Minimum Working Version (Can Be Done Tonight)
Follow the official installation instructions: Linux x86_64 + NVIDIA (see documentation for driver requirements) + Python ≥ 3.10. Recommended UV:
uv venv && source .venv/bin/activate uv pip install "freetoken[accel]" # Then, follow the README to pull or specify a model and start the compatible service (refer to the documentation for the day for specific subcommands) # Example: freetoken serve ... or the equivalent entry point provided in the documentation
CUDA kernels may be JIT-compiled the first time they are used; the CUDA toolkit specified in the documentation is required / nvcc. The desktop app is available at flashml.ai and is ideal for users who don't want to tackle the CLI right away—but if you're integrating with Codex, we still recommend setting up the CLI service address first.
Inspection Points:Installation successful → Service is listening → Compatibility request returned → Make a note on “How to stop the service/clear the cache.” If any step doesn’t match, stop there.
Things to Do Tonight:Write the actual successful commands into the local runbook (without including any keys), so you can reproduce them tomorrow even if you switch terminals.
Key Configuration Points and Troubleshooting (Week 1)
- First, verify the driver and CUDA versions, then install the Python packages;
- For expert caches and KV systems, memory allocation should follow the parameters specified in the documentation to avoid making arbitrary decisions;
- Set timeouts and retry limits on the agent side; when the TTFT increases, prioritize reducing the model size or the amount of context;
- Do not expose local compatibility endpoints to the public internet.
Troubleshooting System Crashes
- It can hold more, but it's slower:Switch back to llama.cpp/vLLM for comparison; the advantage of FreeToken is that “it can’t hold any more.”
- JIT / nvcc failure:Check if the toolkit is in the PATH; complete the setup as described in install.md.
- Agent Timeout:Check prefill and switch to an expert model; first reduce the context, then switch to a larger model.
- Incompatible field:Refer to the official API documentation; do not assume that 100% of the fields are available in the cloud.
Wrap-up: Insufficient video memory → FreeToken → Compatibility endpoint smoke test → Then move on to Codex. The tools are in the repository; your task is to run the smoke test tonight and finalize the hardware list for the first week. If `curl` isn’t working, don’t be quick to blame the Agent for being “too dumb.”


Comment List (0):
Load More Comments Loading...