VIAssist: Adapting Multi-modal Large Language Models for Users with
  Visual Impairments
Bufang Yang Lixing He Kaiwei Liu Zhenyu Yan
Individuals with visual impairments, encompassing both partial and total
difficulties in visual perception, are referred to as visually impaired (VI)
people. An estimated 2.2 billion individuals worldwide are affected by visual
impairments. Recent advancements in multi-modal large language models (MLLMs)
have showcased their extraordinary capabilities across various domains. It is
desirable to help VI individuals with MLLMs' great capabilities of visual
understanding and reasoning. However, it is challenging for VI people to use
MLLMs due to the difficulties in capturing the desirable images to fulfill
their daily requests. For example, the target object is not fully or partially
placed in the image. This paper explores how to leverage MLLMs for VI
individuals to provide visual-question answers. VIAssist can identify undesired
images and provide detailed actions. Finally, VIAssist can provide reliable
answers to users' queries based on the images. Our results show that VIAssist
provides +0.21 and +0.31 higher BERTScore and ROUGE scores than the baseline,
respectively.