Word Importance Explains How Prompts Affect Language Model Outputs
Stefan Hackmann Haniyeh Mahmoudian Mark Steadman Michael Schmidt
The emergence of large language models (LLMs) has revolutionized numerous
applications across industries. However, their "black box" nature often hinders
the understanding of how they make specific decisions, raising concerns about
their transparency, reliability, and ethical use. This study presents a method
to improve the explainability of LLMs by varying individual words in prompts to
uncover their statistical impact on the model outputs. This approach, inspired
by permutation importance for tabular data, masks each word in the system
prompt and evaluates its effect on the outputs based on the available text
scores aggregated over multiple user inputs. Unlike classical attention, word
importance measures the impact of prompt words on arbitrarily-defined text
scores, which enables decomposing the importance of words into the specific
measures of interest--including bias, reading level, verbosity, etc. This
procedure also enables measuring impact when attention weights are not
available. To test the fidelity of this approach, we explore the effect of
adding different suffixes to multiple different system prompts and comparing
subsequent generations with different large language models. Results show that
word importance scores are closely related to the expected suffix importances
for multiple scoring functions.