flash-moe
An inference engine written in C and Metal that runs a very large 397 billion parameter language model on a laptop. It shows that big models can run on small consumer hardware.
Share on XNo license declared
Overview
Flash-MoE is an inference engine written in C, Objective-C and hand-tuned Metal shaders. It runs Qwen3.5-397B-A17B, a 397 billion parameter Mixture-of-Experts model, on a MacBook Pro with 48GB of memory. The 209GB model is streamed from the SSD through a custom Metal compute pipeline, with no Python and no frameworks.
Key features
- Runs a 397B parameter model on a 48GB MacBook Pro
- Streams the model from SSD through Metal compute
- Reported 4.36 tokens per second with 4-bit experts
- Supports tool calling in the 4-bit setup
Best for
Readers interested in how very large models can run on consumer Apple hardware. The 2-bit setting is faster but breaks JSON output and tool calling, so 4-bit is the production choice.
- Upstream
- danveloper/flash-moe
- Fork on GitHub
- Guo-astro/flash-moe
- Upstream stars
- 4.2k
- Category
- AI agents and LLM tools
- License
- No license declaredWithout a license, the author keeps all rights. Ask the upstream owner before reusing the code.
- Forked
- 2026-03-22
- Sync status
- In syncLast synced 2026-10-16
More in AI agents and LLM tools
Pi Agent is a local desktop app for using the Pi AI coding agent on macOS, Windows, and Linux. It helps you manage coding sessions, work with project files, and run saved prompts without setting up a terminal environment.
Forked 2026-10-15Last synced 2026-10-16License: MITAI agents and LLM toolsGitHub
A single-user AI assistant deployed in your Cloudflare account for chat, memory, tasks, notes, and scheduled reminders. It works without relying on a server or account run by the project developers.
Forked 2026-10-15Last synced 2026-10-16License: MITAI agents and LLM toolsGitHub
MiniMind is an open-source project for training a small language model from scratch and studying the LLM pipeline. It helps learners explore how models work through hands-on training and fine-tuning.
Forked 2026-10-14Last synced 2026-10-16License: Apache-2.0AI agents and LLM toolsGitHub
Reusable AI agent skills for turning articles into platform-specific social posts and recording visible OpenAI Responses API research output in a local report.
Forked 2026-10-14Last synced 2026-10-16License: MITAI agents and LLM toolsGitHub