Searching by Code: a New SearchBySnippet Dataset and SnippeR Retrieval
  Model for Searching by Code Snippets
Ivan Sedykh Dmitry Abulkhanov Nikita Sorokin Sergey Nikolenko Valentin Malykh
Code search is an important task that has seen many developments in recent
years. However, previous attempts have mostly considered the problem of
searching for code by a text query. We argue that using a code snippet (and
possibly an associated traceback) as a query and looking for answers with
bugfixing instructions and code samples is a natural use case that is not
covered by existing approaches. Moreover, existing datasets use comments
extracted from code rather than full-text descriptions as text, making them
unsuitable for this use case. We present a new SearchBySnippet dataset
implementing the search-by-code use case based on StackOverflow data; it turns
out that in this setting, existing architectures fall short of the simplest
BM25 baseline even after fine-tuning. We present a new single encoder model
SnippeR that outperforms several strong baselines on the SearchBySnippet
dataset with a result of 0.451 Recall@10; we propose the SearchBySnippet
dataset and SnippeR as a new important benchmark for code search evaluation.