approved
A Critical Reassessment of the Saerens-Latinne-Decaestecker Algorithm

We critically re-examine the Saerens-Latinne-Decaestecker (SLD) algorithm, a well-known method for estimating class prior probabilities (“priors”) and adjusting posterior probabilities (“posteriors”) in scenarios characterized by distribution shift, i.e., difference in the distribution of the priors between the training and the unlabelled documents. Given a machine-learned classifier and a set of unlabelled documents for which the classifier has returned posterior probabilities and estimates of the prior probabilities, SLD updates them both in an iterative, mutually recursive way, with the goal of making both more accurate; this is of key importance in downstream tasks such as single-label multiclass classification and cost-sensitive text classification. Since its publication, SLD has become the standard algorithm for improving the quality of the posteriors in the presence of distribution shift, and is still considered a top contender when we need to estimate the priors (a task that has become known as “quantification”). However, its real effectiveness in improving the quality of the posteriors has been questioned. We here present the results of systematic experiments conducted on a large, publicly available dataset, across multiple amounts of distribution shift and multiple learners. Our experiments show that SLD improves the quality of the posterior probabilities and of the estimates of the prior probabilities, but only when the number of classes in the classification scheme is very small and the classifier is calibrated. As the number of classes grows, or as we use non-calibrated classifiers, SLD converges more slowly (and often does not converge at all), performance degrades rapidly, and the impact of SLD on the quality of the prior estimates and of the posteriors becomes negative rather than positive.

Tags
Data and Resources
To access the resources you must log in
  • Link to Publication

    The resource: 'Link to Publication' is not accessible as guest user. You must login to access it!
Additional Info
Field Value
Creator Esuli, Andrea,
Creator Molinari, Alessio,
Creator Sebastiani, Fabrizio, fabrizio.sebastiani@isti.cnr.it
DOI https://doi.org/10.1145/3433164
Group Computational Epidemiology
Publisher ACM Transactions on Information Systems
Source ACM Transactions on Information Systems. December 2020, Article No. 19
Thematic Cluster Social Data [SD]
system:type JournalArticle
Management Info
Field Value
Author Joanna Wright
Maintainer Fabrizio Sebastiani
Version 1
Last Updated 19 July 2022, 15:51 (CEST)
Created 6 May 2021, 15:37 (CEST)