IITK at SemEval-2024 Task 2: Exploring the Capabilities of LLMs for Safe
  Biomedical Natural Language Inference for Clinical Trials
Shreyasi Mandal Ashutosh Modi
Large Language models (LLMs) have demonstrated state-of-the-art performance
in various natural language processing (NLP) tasks across multiple domains, yet
they are prone to shortcut learning and factual inconsistencies. This research
investigates LLMs' robustness, consistency, and faithful reasoning when
performing Natural Language Inference (NLI) on breast cancer Clinical Trial
Reports (CTRs) in the context of SemEval 2024 Task 2: Safe Biomedical Natural
Language Inference for Clinical Trials. We examine the reasoning capabilities
of LLMs and their adeptness at logical problem-solving. A comparative analysis
is conducted on pre-trained language models (PLMs), GPT-3.5, and Gemini Pro
under zero-shot settings using Retrieval-Augmented Generation (RAG) framework,
integrating various reasoning chains. The evaluation yields an F1 score of
0.69, consistency of 0.71, and a faithfulness score of 0.90 on the test
dataset.