Emmanouil Stergiadis

Emmanouil Stergiadis, Satendra Kumar, Fedor Kovalev et al.

While NMT has achieved remarkable results in the last 5 years, production systems come with strict quality requirements in arbitrarily niche domains that are not always adequately covered by readily available parallel corpora. This is typically addressed by training domain specific models, using fine-tuning methods and some variation of back-translation on top of in-domain monolingual corpora. However, industrial practitioners can rarely afford to focus on a single domain. A far more typical scenario includes a set of closely related, yet succinctly different sub-domains. At Booking.com, we need to translate property descriptions, user reviews, as well as messages, (for example those sent between a customer and an agent or property manager). An editor might need to translate articles across a set of different topics. An e-commerce platform would typically need to translate both the description of each item and the user generated content related to them. To this end, we propose MDT: a novel method to simultaneously fine-tune on several sub-domains by passing multidimensional sentence-level information to the model during training and inference. We show that MDT achieves results competitive to N specialist models each fine-tuned on a single constituent domain, while effectively serving all N sub-domains, therefore cutting development and maintenance costs by the same factor. Besides BLEU (industry standard automatic evaluation metric known to only weakly correlate with human judgement) we also report rigorous human evaluation results for all models and sub-domains as well as specific examples that better contextualise the performance of each model in terms of adequacy and fluency. To facilitate further research, we plan to make the code available upon acceptance.

Arian Askari, Emmanouil Stergiadis, Ilya Gusev et al.

We present HotelMatch-LLM, a multimodal dense retrieval model for the travel domain that enables natural language property search, addressing the limitations of traditional travel search engines which require users to start with a destination and editing search parameters. HotelMatch-LLM features three key innovations: (1) Domain-specific multi-task optimization with three novel retrieval, visual, and language modeling objectives; (2) Asymmetrical dense retrieval architecture combining a small language model (SLM) for efficient online query processing and a large language model (LLM) for embedding hotel data; and (3) Extensive image processing to handle all property image galleries. Experiments on four diverse test sets show HotelMatch-LLM significantly outperforms state-of-the-art models, including VISTA and MARVEL. Specifically, on the test set -- main query type -- we achieve 0.681 for HotelMatch-LLM compared to 0.603 for the most effective baseline, MARVEL. Our analysis highlights the impact of our multi-task optimization, the generalizability of HotelMatch-LLM across LLM architectures, and its scalability for processing large image galleries.

Zeno Belligoli, Emmanouil Stergiadis, Eran Fainman et al.

Controlling the length of generated text can be crucial in various text-generation tasks, including summarization. Existing methods often require complex model alterations, limiting compatibility with pre-trained models. We address these limitations by developing a simple approach for controlling the length of automatic text summaries by increasing the importance of correctly predicting the EOS token in the cross-entropy loss computation. The proposed methodology is agnostic to architecture and decoding algorithms and orthogonal to other inference-time techniques to control the generation length. We tested it with encoder-decoder and modern GPT-style LLMs, and show that this method can control generation length, often without affecting the quality of the summary.

Emmanouil Stergiadis, Priyanka Agrawal, Oliver Squire

We propose an adaptation of the curriculum training framework, applicable to state-of-the-art meta learning techniques for few-shot classification. Curriculum-based training popularly attempts to mimic human learning by progressively increasing the training complexity to enable incremental concept learning. As the meta-learner's goal is learning how to learn from as few samples as possible, the exact number of those samples (i.e. the size of the support set) arises as a natural proxy of a given task's difficulty. We define a simple yet novel curriculum schedule that begins with a larger support size and progressively reduces it throughout training to eventually match the desired shot-size of the test setup. This proposed method boosts the learning efficiency as well as the generalization capability. Our experiments with the MAML algorithm on two few-shot image classification tasks show significant gains with the curriculum training framework. Ablation studies corroborate the independence of our proposed method from the model architecture as well as the meta-learning hyperparameters