HyperLoRA for PDEs
Ritam Majumdar Vishal Jadhav Anirudh Deodhar Shirish Karande Lovekesh Vig Venkataramana Runkana
Physics-informed neural networks (PINNs) have been widely used to develop
neural surrogates for solutions of Partial Differential Equations. A drawback
of PINNs is that they have to be retrained with every change in
initial-boundary conditions and PDE coefficients. The Hypernetwork, a
model-based meta learning technique, takes in a parameterized task embedding as
input and predicts the weights of PINN as output. Predicting weights of a
neural network however, is a high-dimensional regression problem, and
hypernetworks perform sub-optimally while predicting parameters for large base
networks. To circumvent this issue, we use a low ranked adaptation (LoRA)
formulation to decompose every layer of the base network into low-ranked
tensors and use hypernetworks to predict the low-ranked tensors. Despite the
reduced dimensionality of the resulting weight-regression problem, LoRA-based
Hypernetworks violate the underlying physics of the given task. We demonstrate
that the generalization capabilities of LoRA-based hypernetworks drastically
improve when trained with an additional physics-informed loss component
(HyperPINN) to satisfy the governing differential equations. We observe that
LoRA-based HyperPINN training allows us to learn fast solutions for
parameterized PDEs like Burger's equation and Navier Stokes: Kovasznay flow,
while having an 8x reduction in prediction parameters on average without
compromising on accuracy when compared to all other baselines.