Formal concept analysis for evaluating intrinsic dimension of a natural language
This work addresses the problem of dimensionality reduction in natural language processing for researchers, but it is incremental as it applies an existing method to new data.
The study tackled the problem of determining the intrinsic dimension of linguistic varieties for Bengali and Russian languages using formal concept analysis, finding that these dimensions are significantly lower than those used in popular neural network models.
Some results of a computational experiment for determining the intrinsic dimension of linguistic varieties for the Bengali and Russian languages are presented. At the same time, both sets of words and sets of bigrams in these languages were considered separately. The method used to solve this problem was based on formal concept analysis algorithms. It was found that the intrinsic dimensions of these languages are significantly less than the dimensions used in popular neural network models in natural language processing.