LITE: Modeling Environmental Ecosystems with Multimodal Large Language
  Models
Haoran Li Junqi Liu Zexian Wang Shiyuan Luo Xiaowei Jia Huaxiu Yao
The modeling of environmental ecosystems plays a pivotal role in the
sustainable management of our planet. Accurate prediction of key environmental
variables over space and time can aid in informed policy and decision-making,
thus improving people's livelihood. Recently, deep learning-based methods have
shown promise in modeling the spatial-temporal relationships for predicting
environmental variables. However, these approaches often fall short in handling
incomplete features and distribution shifts, which are commonly observed in
environmental data due to the substantial cost of data collection and
malfunctions in measuring instruments. To address these issues, we propose LITE
-- a multimodal large language model for environmental ecosystems modeling.
Specifically, LITE unifies different environmental variables by transforming
them into natural language descriptions and line graph images. Then, LITE
utilizes unified encoders to capture spatial-temporal dynamics and correlations
in different modalities. During this step, the incomplete features are imputed
by a sparse Mixture-of-Experts framework, and the distribution shift is handled
by incorporating multi-granularity information from past observations. Finally,
guided by domain instructions, a language model is employed to fuse the
multimodal representations for the prediction. Our experiments demonstrate that
LITE significantly enhances performance in environmental spatial-temporal
prediction across different domains compared to the best baseline, with a
41.25% reduction in prediction error. This justifies its effectiveness. Our
data and code are available at https://github.com/hrlics/LITE.