We present GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction with a complex environment over long trajectories at high quality. GameNGen can interactively simulate the classic game DOOM at over 20 frames per second on a single TPU. Next frame prediction achieves a PSNR of 29.4, comparable to lossy JPEG compression. Human raters are only slightly better than random chance at distinguishing short clips of the game from clips of the simulation. GameNGen is trained in two phases: (1) an RL-agent learns to play the game and the training sessions are recorded, and (2) a diffusion model is trained to produce the next frame, conditioned on the sequence of past frames and actions. Conditioning augmentations enable stable auto-regressive generation over long trajectories.
Full Gameplay Videos Architecture
Data Collection via Agent Play
Since we cannot collect human gameplay at scale, as a first stage we train an automatic RL-agent to play the game, persisting its training episodes of actions and observations, which become the training data for our generative model.
Training the Generative Diffusion Model
We re-purpose a small diffusion model, Stable Diffusion v1.4, and condition it on a sequence of previous actions and observations (frames). To mitigate auto-regressive drift during inference, we corrupt context frames by adding Gaussian noise to encoded frames during training. This allows the network to correct information sampled in previous frames, and we found it to be critical for preserving visual stability over long time periods.
Latent Decoder Fine-Tuning
The pre-trained auto-encoder of Stable Diffusion v1.4, which compresses 8x8 pixel patches into 4 latent channels, results in meaningful artifacts when predicting game frames, which affect small details and particularly the bottom bar HUD. To leverage the pre-trained knowledge while improving image quality, we train just the decoder of the latent auto-encoder using an MSE loss computed against the target frame pixels.
BibTeX Acknowledgements
Image Description: A side-by-side comparison of the classic game DOOM and its simulation by GameNGen.
Remember these 3 key ideas for your startup:
Harnessing AI for Real-Time Interaction: The development of GameNGen showcases the potential of neural models to enable real-time interaction with complex environments. Startups can leverage similar AI technologies to create more interactive and responsive applications, enhancing user experience and engagement. For more on how AI can be used in project management, check out task automation.
Efficient Data Collection and Training: By using an RL-agent to play and record game sessions, GameNGen efficiently collects training data. This approach can be adopted by startups to gather large datasets without the need for extensive human labor, thus reducing costs and speeding up development. Learn more about how to brand yourself as a remote company.
Improving Visual Stability with Diffusion Models: The use of conditioning augmentations and fine-tuning latent decoders in GameNGen highlights the importance of maintaining visual stability over long periods. Startups can apply these techniques to ensure their AI-driven applications deliver consistent and high-quality visual outputs. For more insights, see the best productivity hacks to get your work done.
Edworking is the best and smartest decision for SMEs and startups to be more productive. Edworking is a FREE superapp of productivity that includes all you need for work powered by AI in the same superapp, connecting Task Management, Docs, Chat, Videocall, and File Management. Save money today by not paying for Slack, Trello, Dropbox, Zoom, and Notion.
For more details, see the original source.
