Domain-adaptation of spherical embeddings
This work addresses the challenge of adapting embeddings to specialized domains like chemistry, where jargon hinders general models, but it is incremental as it builds on existing spherical embedding methods.
The paper tackled the problem of domain adaptation for spherical embedding models, which suffer from non-convergence due to global rotations during training, by developing methods to counter this issue and proposing update strategies; the result showed that these strategies reduced the performance cost of domain adaptation to a level similar to Word2Vec.
Domain adaptation of embedding models, updating a generic embedding to the language of a specific domain, is a proven technique for domains that have insufficient data to train an effective model from scratch. Chemistry publications is one such domain, where scientific jargon and overloaded terminology inhibit the performance of a general language model. The recent spherical embedding model (JoSE) proposed in arXiv:1911.01196 jointly learns word and document embeddings during training on the multi-dimensional unit sphere, which performs well for document classification and word correlation tasks. But, we show a non-convergence caused by global rotations during its training prevents it from domain adaptation. In this work, we develop methods to counter the global rotation of the embedding space and propose strategies to update words and documents during domain specific training. Two new document classification data-sets are collated from general and chemistry scientific journals to compare the proposed update training strategies with benchmark models. We show that our strategies are able to reduce the performance cost of domain adaptation to a level similar to Word2Vec.