PaECTER: Patent-level Representation Learning using Citation-informed Transformers
This work addresses the need for improved semantic similarity search in patents, particularly for prior art search by inventors and examiners, though it is incremental as it builds on existing pre-trained models.
The authors tackled the problem of generating numerical representations for patent documents by fine-tuning BERT for Patents with citation information, resulting in a model that outperforms state-of-the-art models in similarity tasks, such as predicting at least one most similar patent at an average rank of 1.32 against 25 irrelevant patents.
PaECTER is an open-source document-level encoder specific for patents. We fine-tune BERT for Patents with examiner-added citation information to generate numerical representations for patent documents. PaECTER performs better in similarity tasks than current state-of-the-art models used in the patent domain. More specifically, our model outperforms the patent specific pre-trained language model (BERT for Patents) and general-purpose text embedding models (e.g., E5, GTE, and BGE) on our patent citation prediction test dataset on different rank evaluation metrics. PaECTER predicts at least one most similar patent at a rank of 1.32 on average when compared against 25 irrelevant patents. Numerical representations generated by PaECTER from patent text can be used for downstream tasks such as classification, tracing knowledge flows, or semantic similarity search. Semantic similarity search is especially relevant in the context of prior art search for both inventors and patent examiners.