WorldGPT: A Sora-Inspired Video AI Agent as Rich World Models from Text
  and Image Inputs
Deshun Yang Luhui Hu Yu Tian Zihao Li Chris Kelly Bang Yang Cindy Yang Yuexian Zou
Several text-to-video diffusion models have demonstrated commendable
capabilities in synthesizing high-quality video content. However, it remains a
formidable challenge pertaining to maintaining temporal consistency and
ensuring action smoothness throughout the generated sequences. In this paper,
we present an innovative video generation AI agent that harnesses the power of
Sora-inspired multimodal learning to build skilled world models framework based
on textual prompts and accompanying images. The framework includes two parts:
prompt enhancer and full video translation. The first part employs the
capabilities of ChatGPT to meticulously distill and proactively construct
precise prompts for each subsequent step, thereby guaranteeing the utmost
accuracy in prompt communication and accurate execution in following model
operations. The second part employ compatible with existing advanced diffusion
techniques to expansively generate and refine the key frame at the conclusion
of a video. Then we can expertly harness the power of leading and trailing key
frames to craft videos with enhanced temporal consistency and action
smoothness. The experimental results confirm that our method has strong
effectiveness and novelty in constructing world models from text and image
inputs over the other methods.