Exploring the Efficacy of Large Language Models in Summarizing Mental
  Health Counseling Sessions: A Benchmark Study
Prottay Kumar Adhikary Aseem Srivastava Shivani Kumar Salam Michael Singh Puneet Manuja Jini K Gopinath Vijay Krishnan Swati Kedia Koushik Sinha Deb Tanmoy Chakraborty
Comprehensive summaries of sessions enable an effective continuity in mental
health counseling, facilitating informed therapy planning. Yet, manual
summarization presents a significant challenge, diverting experts' attention
from the core counseling process. This study evaluates the effectiveness of
state-of-the-art Large Language Models (LLMs) in selectively summarizing
various components of therapy sessions through aspect-based summarization,
aiming to benchmark their performance. We introduce MentalCLOUDS, a
counseling-component guided summarization dataset consisting of 191 counseling
sessions with summaries focused on three distinct counseling components (aka
counseling aspects). Additionally, we assess the capabilities of 11
state-of-the-art LLMs in addressing the task of component-guided summarization
in counseling. The generated summaries are evaluated quantitatively using
standard summarization metrics and verified qualitatively by mental health
professionals. Our findings demonstrate the superior performance of
task-specific LLMs such as MentalLlama, Mistral, and MentalBART in terms of
standard quantitative metrics such as Rouge-1, Rouge-2, Rouge-L, and BERTScore
across all aspects of counseling components. Further, expert evaluation reveals
that Mistral supersedes both MentalLlama and MentalBART based on six parameters
-- affective attitude, burden, ethicality, coherence, opportunity costs, and
perceived effectiveness. However, these models share the same weakness by
demonstrating a potential for improvement in the opportunity costs and
perceived effectiveness metrics.