CodeCloak: A Method for Evaluating and Mitigating Code Leakage by LLM
  Code Assistants
Amit Finkman Eden Bar-Kochva Avishag Shapira Dudu Mimran Yuval Elovici Asaf Shabtai
LLM-based code assistants are becoming increasingly popular among developers.
These tools help developers improve their coding efficiency and reduce errors
by providing real-time suggestions based on the developer's codebase. While
beneficial, these tools might inadvertently expose the developer's proprietary
code to the code assistant service provider during the development process. In
this work, we propose two complementary methods to mitigate the risk of code
leakage when using LLM-based code assistants. The first is a technique for
reconstructing a developer's original codebase from code segments sent to the
code assistant service (i.e., prompts) during the development process, enabling
assessment and evaluation of the extent of code leakage to third parties (or
adversaries). The second is CodeCloak, a novel deep reinforcement learning
agent that manipulates the prompts before sending them to the code assistant
service. CodeCloak aims to achieve the following two contradictory goals: (i)
minimizing code leakage, while (ii) preserving relevant and useful suggestions
for the developer. Our evaluation, employing GitHub Copilot, StarCoder, and
CodeLlama LLM-based code assistants models, demonstrates the effectiveness of
our CodeCloak approach on a diverse set of code repositories of varying sizes,
as well as its transferability across different models. In addition, we
generate a realistic simulated coding environment to thoroughly analyze code
leakage risks and evaluate the effectiveness of our proposed mitigation
techniques under practical development scenarios.