Zero-Shot Detection of Machine-Generated Codes
Xianjun Yang Kexun Zhang Haifeng Chen Linda Petzold William Yang Wang Wei Cheng
This work proposes a training-free approach for the detection of
LLMs-generated codes, mitigating the risks associated with their indiscriminate
usage. To the best of our knowledge, our research is the first to investigate
zero-shot detection techniques applied to code generated by advanced black-box
LLMs like ChatGPT. Firstly, we find that existing training-based or zero-shot
text detectors are ineffective in detecting code, likely due to the unique
statistical properties found in code structures. We then modify the previous
zero-shot text detection method, DetectGPT (Mitchell et al., 2023) by utilizing
a surrogate white-box model to estimate the probability of the rightmost
tokens, allowing us to identify code snippets generated by language models.
Through extensive experiments conducted on the python codes of the CodeContest
and APPS dataset, our approach demonstrates its effectiveness by achieving
state-of-the-art detection results on text-davinci-003, GPT-3.5, and GPT-4
models. Moreover, our method exhibits robustness against revision attacks and
generalizes well to Java codes. We also find that the smaller code language
model like PolyCoder-160M performs as a universal code detector, outperforming
the billion-scale counterpart. The codes will be available at
https://github.com/ Xianjun-Yang/Code_detection.git