On the Potential and Limitations of Few-Shot In-Context Learning to
  Generate Metamorphic Specifications for Tax Preparation Software
Dananjay Srinivas Rohan Das Saeid Tizpaz-Niari Ashutosh Trivedi Maria Leonor Pacheco
Due to the ever-increasing complexity of income tax laws in the United
States, the number of US taxpayers filing their taxes using tax preparation
software (henceforth, tax software) continues to increase. According to the
U.S. Internal Revenue Service (IRS), in FY22, nearly 50% of taxpayers filed
their individual income taxes using tax software. Given the legal consequences
of incorrectly filing taxes for the taxpayer, ensuring the correctness of tax
software is of paramount importance. Metamorphic testing has emerged as a
leading solution to test and debug legal-critical tax software due to the
absence of correctness requirements and trustworthy datasets. The key idea
behind metamorphic testing is to express the properties of a system in terms of
the relationship between one input and its slightly metamorphosed twinned
input. Extracting metamorphic properties from IRS tax publications is a tedious
and time-consuming process. As a response, this paper formulates the task of
generating metamorphic specifications as a translation task between properties
extracted from tax documents - expressed in natural language - to a contrastive
first-order logic form. We perform a systematic analysis on the potential and
limitations of in-context learning with Large Language Models(LLMs) for this
task, and outline a research agenda towards automating the generation of
metamorphic specifications for tax preparation software.