CleanAgent: Automating Data Standardization with LLM-based Agents
Danrui Qi Jiannan Wang
Data standardization is a crucial part in data science life cycle. While
tools like Pandas offer robust functionalities, their complexity and the manual
effort required for customizing code to diverse column types pose significant
challenges. Although large language models (LLMs) like ChatGPT have shown
promise in automating this process through natural language understanding and
code generation, it still demands expert-level programming knowledge and
continuous interaction for prompt refinement. To solve these challenges, our
key idea is to propose a Python library with declarative, unified APIs for
standardizing column types, simplifying the code generation of LLM with concise
API calls. We first propose Dataprep.Clean which is written as a component of
the Dataprep Library, offers a significant reduction in complexity by enabling
the standardization of specific column types with a single line of code. Then
we introduce the CleanAgent framework integrating Dataprep.Clean and LLM-based
agents to automate the data standardization process. With CleanAgent, data
scientists need only provide their requirements once, allowing for a
hands-free, automatic standardization process.