Improving Clustering on Occupational Text Data through Dimensionality Reduction
This work addresses the need for automated mapping of occupational data to aid career transitions, but it appears incremental as it builds on existing BERT and clustering methods.
The study tackled the problem of mapping occupational definitions from different sources by proposing a pipeline using BERT-based techniques and clustering, and improved results with a specialized silhouette approach, achieving unspecified performance gains.
In this study, we focused on proposing an optimal clustering mechanism for the occupations defined in the well-known US-based occupational database, O*NET. Even though all occupations are defined according to well-conducted surveys in the US, their definitions can vary for different firms and countries. Hence, if one wants to expand the data that is already collected in O*NET for the occupations defined with different tasks, a map between the definitions will be a vital requirement. We proposed a pipeline using several BERT-based techniques with various clustering approaches to obtain such a map. We also examined the effect of dimensionality reduction approaches on several metrics used in measuring performance of clustering algorithms. Finally, we improved our results by using a specialized silhouette approach. This new clustering-based mapping approach with dimensionality reduction may help distinguish the occupations automatically, creating new paths for people wanting to change their careers.