Dermacen Analytica: A Novel Methodology Integrating Multi-Modal Large
  Language Models with Machine Learning in tele-dermatology
Dimitrios P. Panagoulias Evridiki Tsoureli-Nikita Maria Virvou George A. Tsihrintzis
The rise of Artificial Intelligence creates great promise in the field of
medical discovery, diagnostics and patient management. However, the vast
complexity of all medical domains require a more complex approach that combines
machine learning algorithms, classifiers, segmentation algorithms and, lately,
large language models. In this paper, we describe, implement and assess an
Artificial Intelligence-empowered system and methodology aimed at assisting the
diagnosis process of skin lesions and other skin conditions within the field of
dermatology that aims to holistically address the diagnostic process in this
domain. The workflow integrates large language, transformer-based vision models
and sophisticated machine learning tools. This holistic approach achieves a
nuanced interpretation of dermatological conditions that simulates and
facilitates a dermatologist's workflow. We assess our proposed methodology
through a thorough cross-model validation technique embedded in an evaluation
pipeline that utilizes publicly available medical case studies of skin
conditions and relevant images. To quantitatively score the system performance,
advanced machine learning and natural language processing tools are employed
which focus on similarity comparison and natural language inference.
Additionally, we incorporate a human expert evaluation process based on a
structured checklist to further validate our results. We implemented the
proposed methodology in a system which achieved approximate (weighted) scores
of 0.87 for both contextual understanding and diagnostic accuracy,
demonstrating the efficacy of our approach in enhancing dermatological
analysis. The proposed methodology is expected to prove useful in the
development of next-generation tele-dermatology applications, enhancing remote
consultation capabilities and access to care, especially in underserved areas.