Abstract
This paper explores the use of techniques such as knowledge distillation, pruning, and architectural compression as an approach towards significantly reducing computational expenses associated with NLP models while maintaining their overall accuracy. However, average statistics do not necessarily reflect cases when compressed models fail. Previous studies conducted in the field of computer vision show that compression is likely to have a disparate effect on the examples that belong to the long tail of data distribution (Hooker et al., 2019). However, it is unclear whether this trend occurs in the case of named entity recognition (NER) tasks in low-resource morphologically rich languages.
For this purpose, the Kazakh Named Entity Recognition Dataset (KazNERD; Yeshpanov et al., 2022) is used to compare the performance of compressed models for the Kazakh NER task, starting from a baseline transformer model and finishing with a light-weight BiLSTM-CRF architecture, in terms of F1-score, number of parameters, inference time and memory usage. Based on the obtained benchmark results, two hypotheses are tested: (H1) compression has a disparate effect on rare entity classes, and (H2) entities that are fragmented into multiple subword tokens are more sensitive to compression. Hypothesis H2 incorporates previous findings related to compression bias (Hooker et al., 2019) and the role of subword fragmentation as a predictor of poor performance in underrepresented languages (Rust et al., 2021).
For 24 classes with sufficient representation, Hypothesis 1 (H1) is supported: class frequency is significantly negatively associated with compression sensitivity, both for the distilled model (r = −0.43, p = .034) and for the BiLSTM-CRF model (r = −0.58, p = .003). Hypothesis 2 (H2) is not supported both for the distilled model (r = 0.05, p = .82) and for the BiLSTM-CRF model (r = 0.18, p = .40). Frequency remains a significant predictor of compression sensitivity even when both predictors are considered simultaneously in a multiple regression, while fertility does not. The key contribution of this study consists in uncovering the Kazakh NER classes susceptible to model compression and demonstrating that frequency rather than subword fragmentation is the main factor behind their susceptibility.