Download
Diffusion for World Modeling is an experimental reinforcement learning system that trains intelligent agents inside a simulated environment generated by a diffusion-based world model. The project introduces the idea of using diffusion models, commonly used for image generation, to simulate the dynamics of an environment and predict future states based on previous observations and actions. Instead of interacting directly with a real environment, the reinforcement learning agent learns within a generative model that produces frames representing the environment. This approach allows training to occur in a simulated world that captures detailed visual dynamics while reducing the need for costly interactions with real environments. The system has been applied to tasks such as Atari game simulations and demonstrations involving complex environments like first-person shooter games.
Features
- Reinforcement learning agent trained inside a diffusion-based world model
- Simulation of environments using autoregressive generative modeling
- Support for experiments with Atari game environments and simulated gameplay
- Integration of diffusion models for predicting environment state transitions
- Configurable training pipelines using Hydra configuration management
- Interactive visualization tools for running agents or controlling environments
Project Samples
