On Prompt Sensitivity of ChatGPT in Affective Computing
Mostafa M. Amin Björn W. Schuller
Recent studies have demonstrated the emerging capabilities of foundation
models like ChatGPT in several fields, including affective computing. However,
accessing these emerging capabilities is facilitated through prompt
engineering. Despite the existence of some prompting techniques, the field is
still rapidly evolving and many prompting ideas still require investigation. In
this work, we introduce a method to evaluate and investigate the sensitivity of
the performance of foundation models based on different prompts or generation
parameters. We perform our evaluation on ChatGPT within the scope of affective
computing on three major problems, namely sentiment analysis, toxicity
detection, and sarcasm detection. First, we carry out a sensitivity analysis on
pivotal parameters in auto-regressive text generation, specifically the
temperature parameter $T$ and the top-$p$ parameter in Nucleus sampling,
dictating how conservative or creative the model should be during generation.
Furthermore, we explore the efficacy of several prompting ideas, where we
explore how giving different incentives or structures affect the performance.
Our evaluation takes into consideration performance measures on the affective
computing tasks, and the effectiveness of the model to follow the stated
instructions, hence generating easy-to-parse responses to be smoothly used in
downstream applications.