vlmrl
An experimental reinforcement learning gym for vision-language models, written in JAX. It lets you plug in environments and models and train them with PPO.
Pick a small issue you can reproduce, then fix it with your AI coding agent.
Share on XLicense: Apache-2.0
Overview
vlm-gym is an experimental reinforcement learning gym for vision-language models, written in JAX. The idea is to drop in any environment and any model and train with PPO. It ships with environments such as GeoGuessr, NLVR2 and captioning, and a reference Qwen3-VL-4B-Instruct model that can be converted from Hugging Face to JAX. The trainer, rollout engine and evaluation harness live in a core folder, and GeoGuessr training uses a staged curriculum.
Key features
- Pluggable vision environments: GeoGuessr, NLVR2, captioning
- PPO trainer for vision-language models
- Rollout engine and evaluation against a Hugging Face baseline
- Converts Qwen3-VL-4B-Instruct from Hugging Face to JAX
Best for
Suited to researchers experimenting with reinforcement learning on vision-language models in JAX. The README marks its status as experimental.
- Upstream
- sdan/vlm-gym
- Fork on GitHub
- Guo-astro/vlmrl
- Upstream stars
- 150
- Category
- AI agents and LLM tools
- Language
- Python
- License
- Apache-2.0
- Forked
- 2025-10-14
- Sync status
- In syncLast synced 2026-10-20
More in AI agents and LLM tools
Magnitude runs open models locally and tunes kernels for your hardware. Use it to connect local models to supported agents or an OpenAI-compatible client.
Forked 2026-10-19Last synced 2026-10-20License: Apache-2.0AI agents and LLM toolsGitHub
A collection of AI research skills and guides that helps researchers with idea evaluation, literature research, paper writing, and pre-submission review.
Forked 2026-10-18Last synced 2026-10-20Custom license (see repository)AI agents and LLM toolsGitHub
RLVE supports language model reinforcement learning with adaptive, verifiable environments for generating problems and evaluating model outputs.
Forked 2026-10-17Last synced 2026-10-20License: MITAI agents and LLM toolsGitHub
Provides modular Android development instructions for AI agents, helping them follow specialized workflows. Install individual skills or the full set with Android CLI.
Forked 2026-10-16Last synced 2026-10-20License: Apache-2.0AI agents and LLM toolsGitHub