flash-moe
An inference engine written in C and Metal that runs a very large 397 billion parameter language model on a laptop. It shows that big models can run on small consumer hardware.
Share on XNo license declared
Overview
Flash-MoE is an inference engine written in C, Objective-C and hand-tuned Metal shaders. It runs Qwen3.5-397B-A17B, a 397 billion parameter Mixture-of-Experts model, on a MacBook Pro with 48GB of memory. The 209GB model is streamed from the SSD through a custom Metal compute pipeline, with no Python and no frameworks.
Key features
- Runs a 397B parameter model on a 48GB MacBook Pro
- Streams the model from SSD through Metal compute
- Reported 4.36 tokens per second with 4-bit experts
- Supports tool calling in the 4-bit setup
Best for
Readers interested in how very large models can run on consumer Apple hardware. The 2-bit setting is faster but breaks JSON output and tool calling, so 4-bit is the production choice.
- Upstream
- danveloper/flash-moe
- Fork on GitHub
- Guo-astro/flash-moe
- Upstream stars
- 4.2k
- Category
- AI agents and LLM tools
- License
- No license declaredWithout a license, the author keeps all rights. Ask the upstream owner before reusing the code.
- Forked
- 2026-03-22
- Sync status
- In syncLast synced 2026-09-29
More in AI agents and LLM tools
A free desktop app where children in non-English-speaking countries practice English speaking and listening by talking with an AI companion. It can run locally and offline.
Forked 2026-09-28Last synced 2026-09-29License: MITAI agents and LLM toolsGitHub
An agent skill for Chinese patents. It helps find patentable points, write disclosure documents, explain patents in plain language, follow policy changes and assist with examination replies.
Forked 2026-09-28Last synced 2026-09-29License: MITAI agents and LLM toolsGitHub
A minimal Python library from Hugging Face for building AI agents that write and run code to solve tasks. It lets you set up capable agents in a few lines.
Forked 2026-09-28Last synced 2026-09-29License: Apache-2.0AI agents and LLM toolsGitHub
An open-source benchmark that measures how well AI agents handle realistic legal work. It provides tasks and evaluation for improving legal agents.
Forked 2026-09-28Last synced 2026-09-29License: MITAI agents and LLM toolsGitHub