Inferring Hybrid Neural Fluid Fields from Videos
Hong-Xing Yu Yang Zheng Yuan Gao Yitong Deng Bo Zhu Jiajun Wu
We study recovering fluid density and velocity from sparse multiview videos.
Existing neural dynamic reconstruction methods predominantly rely on optical
flows; therefore, they cannot accurately estimate the density and uncover the
underlying velocity due to the inherent visual ambiguities of fluid velocity,
as fluids are often shapeless and lack stable visual features. The challenge is
further pronounced by the turbulent nature of fluid flows, which calls for
properly designed fluid velocity representations. To address these challenges,
we propose hybrid neural fluid fields (HyFluid), a neural approach to jointly
infer fluid density and velocity fields. Specifically, to deal with visual
ambiguities of fluid velocity, we introduce a set of physics-based losses that
enforce inferring a physically plausible velocity field, which is
divergence-free and drives the transport of density. To deal with the turbulent
nature of fluid velocity, we design a hybrid neural velocity representation
that includes a base neural velocity field that captures most irrotational
energy and a vortex particle-based velocity that models residual turbulent
velocity. We show that our method enables recovering vortical flow details. Our
approach opens up possibilities for various learning and reconstruction
applications centered around 3D incompressible flow, including fluid
re-simulation and editing, future prediction, and neural dynamic scene
composition. Project website: https://kovenyu.com/HyFluid/