A comparative analysis of embedding models for patent similarity
Grazia Sveva Ascione Valerio Sterzi
This paper makes two contributions to the field of text-based patent
similarity. First, it compares the performance of different kinds of
patent-specific pretrained embedding models, namely static word embeddings
(such as word2vec and doc2vec models) and contextual word embeddings (such as
transformers based models), on the task of patent similarity calculation.
Second, it compares specifically the performance of Sentence Transformers
(SBERT) architectures with different training phases on the patent similarity
task. To assess the models' performance, we use information about patent
interferences, a phenomenon in which two or more patent claims belonging to
different patent applications are proven to be overlapping by patent examiners.
Therefore, we use these interferences cases as a proxy for maximum similarity
between two patents, treating them as ground-truth to evaluate the performance
of the different embedding models. Our results point out that, first, Patent
SBERT-adapt-ub, the domain adaptation of the pretrained Sentence Transformer
architecture proposed in this research, outperforms the current
state-of-the-art in patent similarity. Second, they show that, in some cases,
large static models performances are still comparable to contextual ones when
trained on extensive data; thus, we believe that the superiority in the
performance of contextual embeddings may not be related to the actual
architecture but rather to the way the training phase is performed.