BBScore: A Brownian Bridge Based Metric for Assessing Text Coherence
Zhecheng Sheng Tianhao Zhang Chen Jiang Dongyeop Kang
Measuring the coherence of text is a vital aspect of evaluating the quality
of written content. Recent advancements in neural coherence modeling have
demonstrated their efficacy in capturing entity coreference and discourse
relations, thereby enhancing coherence evaluation. However, many existing
methods heavily depend on static embeddings or focus narrowly on nearby
context, constraining their capacity to measure the overarching coherence of
long texts. In this paper, we posit that coherent texts inherently manifest a
sequential and cohesive interplay among sentences, effectively conveying the
central theme, purpose, or standpoint. To explore this abstract relationship,
we introduce the "BBScore," a novel reference-free metric grounded in Brownian
bridge theory for assessing text coherence. Our findings showcase that when
synergized with a simple additional classification component, this metric
attains a performance level comparable to state-of-the-art techniques on
standard artificial discrimination tasks. We also establish in downstream tasks
that this metric effectively differentiates between human-written documents and
text generated by large language models under a specific domain. Furthermore,
we illustrate the efficacy of this approach in detecting written styles
attributed to diverse large language models, underscoring its potential for
generalizability. In summary, we present a novel Brownian bridge coherence
metric capable of measuring both local and global text coherence, while
circumventing the need for end-to-end model training. This flexibility allows
for its application in various downstream tasks.