Fine-tuned vs. Prompt-tuned Supervised Representations: Which Better
  Account for Brain Language Representations?
Jingyuan Sun Marie-Francine Moens
To decipher the algorithm underlying the human brain's language
representation, previous work probed brain responses to language input with
pre-trained artificial neural network (ANN) models fine-tuned on NLU tasks.
However, full fine-tuning generally updates the entire parametric space and
distorts pre-trained features, cognitively inconsistent with the brain's robust
multi-task learning ability. Prompt-tuning, in contrast, protects pre-trained
weights and learns task-specific embeddings to fit a task. Could prompt-tuning
generate representations that better account for the brain's language
representations than fine-tuning? If so, what kind of NLU task leads a
pre-trained model to better decode the information represented in the human
brain? We investigate these questions by comparing prompt-tuned and fine-tuned
representations in neural decoding, that is predicting the linguistic stimulus
from the brain activities evoked by the stimulus. We find that on none of the
10 NLU tasks, full fine-tuning significantly outperforms prompt-tuning in
neural decoding, implicating that a more brain-consistent tuning method yields
representations that better correlate with brain data. Moreover, we identify
that tasks dealing with fine-grained concept meaning yield representations that
better decode brain activation patterns than other tasks, especially the
syntactic chunking task. This indicates that our brain encodes more
fine-grained concept information than shallow syntactic information when
representing languages.