shimmy
Shimmy runs GGUF language models locally through an OpenAI-compatible server. It lets existing AI tools use local GPU inference from a single Rust binary.
Pick a small issue you can reproduce, then fix it with your AI coding agent.
Share on XLicense: Apache-2.0
Overview
Shimmy is a local inference server for GGUF language models, designed to let existing tools that speak the OpenAI API run models on a user's own machine. It uses a single Rust binary and a WebGPU-based inference engine, without requiring a Python runtime or C++ toolchain. Users point compatible clients at the server, load a supported GGUF model, and generate responses locally. The documented hardware support includes discrete and integrated GPUs and Apple Silicon.
Key features
- Serves GGUF models through an OpenAI-compatible API.
- Uses a single Rust binary and a WebGPU inference engine.
- Runs without a Python runtime or C++ toolchain.
- Supports local inference on NVIDIA, AMD, Intel, integrated GPUs, and Apple Silicon.
- Supports extended context with YaRN RoPE scaling.
Best for
People who want to run supported GGUF models locally and connect them to tools that already use the OpenAI API. It is a fit when a Rust-based, WebGPU inference server without a Python or C++ setup is preferred.
- Upstream
- Michael-A-Kuykendall/shimmy
- Fork on GitHub
- Guo-astro/shimmy
- Upstream stars
- 6.0k
- Category
- AI agents and LLM tools
- Language
- Rust
- License
- Apache-2.0
- Forked
- 2026-10-05
- Sync status
- Diverged from upstreamLast synced 2026-10-29
More in AI agents and LLM tools
/last30days is an AI-agent skill for researching recent discussions across social platforms, video, news, markets, and the web. It brings scattered signals together in a synthesized brief for trend research and preparation.
Forked 2026-10-22Last synced 2026-11-08License: MITAI agents and LLM toolsGitHub
A portable C99 inference engine for running Kimi K3 on a CPU without a GPU or machine-learning framework. It streams the large checkpoint from disk to support inference with limited RAM, for batch text generation or text chat.
Forked 2026-10-21Last synced 2026-11-08License: Apache-2.0AI agents and LLM toolsGitHub
LayerFS gives AI agents writable workspaces over shared history, so they can work in parallel without copying the base. Use it to checkpoint, branch, rewind, and merge agent changes.
Forked 2026-10-21Last synced 2026-11-08License: MITAI agents and LLM toolsGitHub
Magnitude runs open models locally and tunes kernels for your hardware. Use it to connect local models to supported agents or an OpenAI-compatible client.
Forked 2026-10-19Last synced 2026-11-08License: Apache-2.0AI agents and LLM toolsGitHub