Bayesian analysis of rank data with covariates and heterogeneous rankers. (English) Zbl 07474195
Summary: Data in the form of ranking lists are frequently encountered, and combining ranking results from different sources can potentially generate a better ranking list and help understand behaviors of the rankers. Of interest here are the rank data under the following settings: (i) covariate information available for the ranked entities; (ii) rankers of varying qualities or having different opinions; and (iii) incomplete ranking lists for nonoverlapping subgroups. We review some key ideas built around the Thurstone model family by researchers in the past few decades and provide a unifying approach for Bayesian Analysis of Rank data with Covariates (BARC) and its extensions in handling heterogeneous rankers. With this Bayesian framework, we can study rankers’ varying quality, cluster rankers’ heterogeneous opinions, and measure the corresponding uncertainties. To enable an efficient Bayesian inference, we advocate a parameter-expanded Gibbs sampler to sample from the target posterior distribution. The posterior samples also result in a Bayesian aggregated ranking list, with credible intervals quantifying its uncertainty. We investigate and compare performances of the proposed methods and other rank aggregation methods in both simulation studies and two real-data examples.
MSC:
| 62-XX | Statistics |
Keywords:
Thurstone model; rank aggregation; heterogeneous rankers; infinite mixture model; parameter-expanded data augmentationReferences:
| [1] | Agresti, A. (1990). Categorical Data Analysis. Wiley Series in Probability and Mathematical Statistics: Applied Probability and Statistics. Wiley, New York. · Zbl 0716.62001 |
| [2] | Ailon, N. (2010). Aggregation of partial rankings, \(p\)-ratings and top-\(m\) lists. Algorithmica 57 284-300. · Zbl 1184.68627 · doi:10.1007/s00453-008-9211-1 |
| [3] | Albert, J. H. and Chib, S. (1993). Bayesian analysis of binary and polychotomous response data. J. Amer. Statist. Assoc. 88 669-679. · Zbl 0774.62031 |
| [4] | Allison, P. D. and Christakis, N. A. (1994). Logit models for sets of ranked items. Sociol. Method. 24 199-228. |
| [5] | Alvo, M. and Yu, P. L. H. (2014). Statistical Methods for Ranking Data. Frontiers in Probability and the Statistical Sciences. Springer, New York. · Zbl 1341.62001 · doi:10.1007/978-1-4939-1471-5 |
| [6] | Antoniak, C. E. (1974). Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. Ann. Statist. 2 1152-1174. · Zbl 0335.60034 |
| [7] | Azari Soufiani, H. (2014). Revisiting Random Utility Models. Ph.D. thesis, Harvard Univ. |
| [8] | Azari Soufiani, H., Chen, W., Parkes, D. C. and Xia, L. (2013). Generalized method-of-moments for rank aggregation. In Advances in Neural Information Processing Systems (C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani and K. Q. Weinberger, eds.) 26 2706-2714. Curran Associates, Red Hook, NY. |
| [9] | Badgeley, M. A., Chikina, M. D. and Sealfon, S. C. (2014). Hybrid Bayesian-rank integration approach improves the predictive power of genomic dataset aggregation. Bioinformatics 31 209-215. |
| [10] | Benter, W. (1994). Computer-based horse race handicapping and wagering systems: A report. In Efficiency of Racetrack Betting Markets (W. T. Ziemba, V. S. Lo and D. B. Haush, eds.) 183-198. Academic Press, London. |
| [11] | Bhowmik, A. and Ghosh, J. (2017). LETOR methods for unsupervised rank aggregation. In Proceedings of the 26th International Conference on World Wide Web 1331-1340. International World Wide Web Conferences Steering Committee. |
| [12] | Blackwell, D. and MacQueen, J. B. (1973). Ferguson distributions via Pólya urn schemes. Ann. Statist. 1 353-355. · Zbl 0276.62010 |
| [13] | Block, H. D. and Marschak, J. (1960). Random orderings and stochastic theories of responses. In Contributions to Probability and Statistics 97-132. Stanford Univ. Press, Stanford, CA. · Zbl 0096.12104 |
| [14] | Böckenholt, U. (1992). Thurstonian representation for partial ranking data. Br. J. Math. Stat. Psychol. 45 31-49. |
| [15] | Böckenholt, U. (1993). Applications of Thurstonian models to ranking data. In Probability Models and Statistical Analyses for Ranking Data (Amherst, MA, 1990) (M. A. M. A. and J. S. J. S., eds.). Lect. Notes Stat. 80 157-172. Springer, New York. · Zbl 0764.62097 · doi:10.1007/978-1-4612-2738-0_9 |
| [16] | Böckenholt, U. (2006). Thurstonian-based analyses: Past, present, and future utilities. Psychometrika 71 615-629. · Zbl 1306.62382 · doi:10.1007/s11336-006-1598-5 |
| [17] | Borda, J. C. (1781). Mémoire sur les élections au scrutin. |
| [18] | Bradley, R. A. and Terry, M. E. (1952). Rank analysis of incomplete block designs. I. The method of paired comparisons. Biometrika 39 324-345. · Zbl 0047.12903 · doi:10.2307/2334029 |
| [19] | Caron, F. and Doucet, A. (2012). Efficient Bayesian inference for generalized Bradley-Terry models. J. Comput. Graph. Statist. 21 174-196. · doi:10.1080/10618600.2012.638220 |
| [20] | Critchlow, D. E., Fligner, M. A. and Verducci, J. S. (1991). Probability models on rankings. J. Math. Psych. 35 294-318. · Zbl 0741.62024 · doi:10.1016/0022-2496(91)90050-4 |
| [21] | D’Ambrosio, A., Iorio, C., Staiano, M. and Siciliano, R. (2019). Median constrained bucket order rank aggregation. Comput. Statist. 34 787-802. · Zbl 1417.62338 · doi:10.1007/s00180-018-0858-z |
| [22] | Daniels, H. E. (1950). Rank correlation and population models. J. Roy. Statist. Soc. Ser. B 12 171-181. · Zbl 0040.22302 |
| [23] | David, H. A. (1963). The Method of Paired Comparisons, 12. Oxford University Press, Oxford. |
| [24] | Davidson, R. R. and Farquhar, P. H. (1976). A bibliography on the method of paired comparisons. Biometrics 32 241-252. · Zbl 0336.62067 |
| [25] | DeConde, R. P., Hawley, S., Falcon, S., Clegg, N., Knudsen, B. and Etzioni, R. (2006). Combining results of microarray experiments: A rank aggregation approach. Stat. Appl. Genet. Mol. Biol. 5 Art. 15, 25. · Zbl 1166.92306 · doi:10.2202/1544-6115.1204 |
| [26] | Deng, K., Han, S., Li, K. J. and Liu, J. S. (2014). Bayesian aggregation of order-based rank data. J. Amer. Statist. Assoc. 109 1023-1039. · Zbl 1368.62122 · doi:10.1080/01621459.2013.878660 |
| [27] | Diaconis, P. (1988). Group Representations in Probability and Statistics. Institute of Mathematical Statistics Lecture Notes—Monograph Series 11. IMS, Hayward, CA. · Zbl 0695.60012 |
| [28] | Dwork, C., Kumar, R., Naor, M. and Sivakumar, D. (2001). Rank aggregation methods for the web. In Proceedings of the 10th International Conference on World Wide Web 613-622. ACM, New York. |
| [29] | Escobar, M. D. and West, M. (1995). Bayesian density estimation and inference using mixtures. J. Amer. Statist. Assoc. 90 577-588. · Zbl 0826.62021 |
| [30] | Ferguson, T. S. (1983). Bayesian density estimation by mixtures of normal distributions. In Recent Advances in Statistics 287-302. Academic Press, New York. · Zbl 0557.62030 |
| [31] | Fong, D. K. H., Kim, S., Chen, Z. and DeSarbo, W. S. (2016). A Bayesian multinomial probit model for the analysis of panel choice data. Psychometrika 81 161-183. · Zbl 1342.62184 · doi:10.1007/s11336-014-9437-6 |
| [32] | Frühwirth-Schnatter, S. and Malsiner-Walli, G. (2019). From here to infinity: Sparse finite versus Dirichlet process mixtures in model-based clustering. Adv. Data Anal. Classif. 13 33-64. · Zbl 1474.62225 · doi:10.1007/s11634-018-0329-y |
| [33] | Gionis, A., Mannila, H., Puolamäki, K. and Ukkonen, A. (2006). Algorithms for discovering bucket orders from data. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 561-566. |
| [34] | Gormley, I. C. and Murphy, T. B. (2006). Analysis of Irish third-level college applications data. J. Roy. Statist. Soc. Ser. A 169 361-379. · doi:10.1111/j.1467-985X.2006.00412.x |
| [35] | Gormley, I. C. and Murphy, T. B. (2008a). Exploring voting blocs within the Irish electorate: A mixture modeling approach. J. Amer. Statist. Assoc. 103 1014-1027. · Zbl 1205.62198 · doi:10.1198/016214507000001049 |
| [36] | Gormley, I. C. and Murphy, T. B. (2008b). A mixture of experts model for rank data with applications in election studies. Ann. Appl. Stat. 2 1452-1477. · Zbl 1454.62498 · doi:10.1214/08-AOAS178 |
| [37] | Gormley, I. C. and Murphy, T. B. (2009). A grade of membership model for rank data. Bayesian Anal. 4 265-295. · Zbl 1330.62024 · doi:10.1214/09-BA410 |
| [38] | Gormley, I. C. and Murphy, T. B. (2010). Clustering ranked preference data using sociodemographic covariates. In Choice Modelling: The State-of-the-Art and the State-of-Practice: Proceedings from the Inaugural International Choice Modelling Conference 543-569. |
| [39] | Gray-Davies, T., Holmes, C. C. and Caron, F. (2016). Scalable Bayesian nonparametric regression via a Plackett-Luce model for conditional ranks. Electron. J. Stat. 10 1807-1828. · Zbl 1397.62148 · doi:10.1214/15-EJS1032 |
| [40] | Guiver, J. and Snelson, E. (2009). Bayesian inference for Plackett-Luce ranking models. In Proceedings of the 26th Annual International Conference on Machine Learning 377-384. ACM, New York. |
| [41] | Hastie, T. and Tibshirani, R. (1998). Classification by pairwise coupling. Ann. Statist. 26 451-471. · Zbl 0932.62071 · doi:10.1214/aos/1028144844 |
| [42] | Hausman, J. A. and Ruud, P. A. (1987). Specifying and testing econometric models for rank-ordered data. J. Econometrics 34 83-104. · Zbl 0646.62103 · doi:10.1016/0304-4076(87)90068-6 |
| [43] | Hoff, P. D. (2009). A First Course in Bayesian Statistical Methods. Springer Texts in Statistics. Springer, New York. · Zbl 1213.62044 · doi:10.1007/978-0-387-92407-6 |
| [44] | Huang, T.-K., Weng, R. C. and Lin, C.-J. (2006). Generalized Bradley-Terry models and multi-class probability estimates. J. Mach. Learn. Res. 7 85-115. · Zbl 1222.68219 |
| [45] | Hunter, D. R. (2004). MM algorithms for generalized Bradley-Terry models. Ann. Statist. 32 384-406. · Zbl 1105.62359 · doi:10.1214/aos/1079120141 |
| [46] | Johnson, V. E., Deaner, R. O. and van Schaik, C. P. (2002). Bayesian analysis of rank data with application to primate intelligence experiments. J. Amer. Statist. Assoc. 97 8-17. · Zbl 1073.62519 · doi:10.1198/016214502753479185 |
| [47] | Johnson, T. R. and Kuhn, K. M. (2013). Bayesian thurstonian models for ranking data using jags. Behavior Research Methods 45 857-872. |
| [48] | Kendall, M. G. (1938). A new measure of rank correlation. Biometrika 81-93. · Zbl 0019.13001 |
| [49] | Kenkre, S., Khan, A. and Pandit, V. (2011). On discovering bucket orders from preference data. In Proceedings of the 2011 SIAM International Conference on Data Mining 872-883. SIAM, Philadelphia. |
| [50] | Lee, M. D., Steyvers, M. and Miller, B. (2014). A cognitive model for aggregating people’s rankings. PLoS ONE 9 1-9. |
| [51] | Li, X., Wang, X. and Xiao, G. (2017). A comparative study of rank aggregation methods for partial and top ranked lists in genomic applications. Brief. Bioinform. 20 178-189. |
| [52] | Li, X., Yi, D. and Liu, J. S (2022). Supplement to “Bayesian Analysis of Rank Data with Covariates and Heterogeneous Rankers.” https://doi.org/10.1214/20-STS818SUPP · Zbl 07474195 |
| [53] | Li, X., Choudhary, P. K., Biswas, S. and Wang, X. (2018). A Bayesian latent variable approach to aggregation of partial and top-ranked lists in genomic studies. Stat. Med. 37 4266-4278. · doi:10.1002/sim.7920 |
| [54] | Li, H., Xu, M., Liu, J. S. and Fan, X. (2020). An extended mallows model for ranked data aggregation. J. Amer. Statist. Assoc. 115 730-746. · Zbl 1445.62040 · doi:10.1080/01621459.2019.1573733 |
| [55] | Lin, S. (2010). Rank aggregation methods. Wiley Interdiscip. Rev.: Comput. Stat. 2 555-570. |
| [56] | Lin, S. and Ding, J. (2009). Integration of ranked lists via cross entropy Monte Carlo with applications to mRNA and microRNA studies. Biometrics 65 9-18. · Zbl 1159.62077 · doi:10.1111/j.1541-0420.2008.01044.x |
| [57] | Liu, J. S. (1994). The collapsed Gibbs sampler in Bayesian computations with applications to a gene regulation problem. J. Amer. Statist. Assoc. 89 958-966. · Zbl 0804.62033 |
| [58] | Liu, T.-Y. (2009). Learning to rank for information retrieval. Foundations and Trends in Information Retrieval 3 225-331. |
| [59] | Liu, J. S. and Wu, Y. N. (1999). Parameter expansion for data augmentation. J. Amer. Statist. Assoc. 94 1264-1274. · Zbl 1069.62514 · doi:10.2307/2669940 |
| [60] | Liu, S.-Q., Shen, G., Bai, D., Zhou, H., Li, S., Chen, W.-J., Wang, D.-W., Li, W.-R., Geng, Z. et al. (2012). Consistency of the subjective evaluation of malocclusion severity by the Chinese orthodontic experts. Journal of Peking University. Health Sciences 44 98-102. |
| [61] | Liu, A., Zhao, Z., Liao, C., Lu, P. and Xia, L. (2019). Learning Plackett-Luce mixtures from partial preferences. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19). |
| [62] | Luce, R. D. (1959). Individual Choice Behavior: A Theoretical Analysis. Wiley, New York. · Zbl 0093.31708 |
| [63] | MacEachern, S. N. (1994). Estimating normal means with a conjugate style Dirichlet process prior. Comm. Statist. Simulation Comput. 23 727-741. · Zbl 0825.62053 · doi:10.1080/03610919408813196 |
| [64] | Mallows, C. L. (1957). Non-null ranking models. I. Biometrika 44 114-130. · Zbl 0087.34001 · doi:10.1093/biomet/44.1-2.114 |
| [65] | Marden, J. I. (1995). Analyzing and Modeling Rank Data. Monographs on Statistics and Applied Probability 64. CRC Press, London. · Zbl 0853.62006 |
| [66] | Maydeu-Olivares, A. (1999). Thurstonian modeling of ranking data via mean and covariance structure analysis. Psychometrika 64 325-340. · Zbl 1291.62230 · doi:10.1007/BF02294299 |
| [67] | McFadden, D. (1980). Econometric models for probabilistic choice among products. J. Bus. 53 S13-S29. |
| [68] | Meila, M. and Chen, H. (2012). Dirichlet process mixtures of generalized mallows models. Preprint. Available at arXiv:1203.3496. |
| [69] | Mosteller, F. (1951). Remarks on the method of paired comparisons: I. the least squares solution assuming equal standard deviations and equal correlations. Psychometrika 16 3-9. |
| [70] | Neal, R. M. (1992). Bayesian mixture modeling. In Maximum Entropy and Bayesian Methods 197-211. Springer, Berlin. · Zbl 0829.62033 |
| [71] | Neal, R. M. (2000). Markov chain sampling methods for Dirichlet process mixture models. J. Comput. Graph. Statist. 9 249-265. · doi:10.2307/1390653 |
| [72] | Plackett, R. L. (1975). The analysis of permutations. J. R. Stat. Soc. Ser. C. Appl. Stat. 24 193-202. · doi:10.2307/2346567 |
| [73] | Rand, W. M. (1971). Objective criteria for the evaluation of clustering methods. J. Amer. Statist. Assoc. 66 846-850. |
| [74] | Rao, P. V. and Kupper, L. L. (1967). Ties in paired-comparison experiments: A generalization of the Bradley-Terry model. J. Amer. Statist. Assoc. 62 194-204. |
| [75] | Richmond, S., Shaw, W. C., O’brien, K. D., Buchanan, I. B., Jones, R., Stephens, C. D., Roberts, C. T. and Andrews, M. (1992). The development of the par index (peer assessment rating): Reliability and validity. The European Journal of Orthodontics 14 125-139. |
| [76] | Schimek, M. G., Budinská, E., Kugler, K. G., Švendová, V., Ding, J. and Lin, S. (2015). TopKLists: A comprehensive \[\mathtt{R}\] package for statistical inference, stochastic aggregation, and visualization of multiple omics ranked lists. Stat. Appl. Genet. Mol. Biol. 14 311-316. · Zbl 1329.92008 · doi:10.1515/sagmb-2014-0093 |
| [77] | Song, G.-Y., Zhao, Z.-H., Ding, Y., Bai, Y.-X., Wang, L., He, H., Shen, G., Li, W.-R., Baumrind, S. et al. (2014). Reliability assessment and correlation analysis of evaluating orthodontic treatment outcome in Chinese patients. International Journal of Oral Science 6 50-55. |
| [78] | Song, G.-Y., Jiang, R.-P., Zhang, X.-Y., Liu, S.-Q., Yu, X.-N., Chen, Q., Weng, X.-R., Wu, W.-Z., Su, H. et al. and Research Group of Establishing Chinese Evaluation Standard of Orthodontic Treatment Outcome (2015). Validation of subjective and objective evaluation methods for orthodontic treatment outcome. Journal of Peking University. Health Sciences 47 90-97. |
| [79] | Stern, H. (1990). Models for distributions on permutations. J. Amer. Statist. Assoc. 85 558-564. |
| [80] | Švendová, V. and Schimek, M. G. (2017). A novel method for estimating the common signals for consensus across multiple ranked lists. Comput. Statist. Data Anal. 115 122-135. · Zbl 1466.62200 · doi:10.1016/j.csda.2017.05.010 |
| [81] | Tanner, M. A. and Wong, W. H. (1987). The calculation of posterior distributions by data augmentation. J. Amer. Statist. Assoc. 82 528-550. With discussion and with a reply by the authors. · Zbl 0619.62029 |
| [82] | Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review 34 273. |
| [83] | Van Erp, M. and Schomaker, L. (2000). Variants of the borda count method for combining ranked classifier hypotheses. In Proceedings of the 7th International Workshop on Frontiers in Handwriting Recognition. |
| [84] | Vitelli, V., SØrensen, Ø., Crispino, M., Frigessi, A. and Arjas, E. (2017). Probabilistic preference learning with the Mallows rank model. J. Mach. Learn. Res. 18 Paper No. 158, 49. · Zbl 1471.62268 |
| [85] | Walker, J. and Ben-Akiva, M. (2002). Generalized random utility model. Math. Social Sci. 43 303-343. · Zbl 1023.91013 · doi:10.1016/S0165-4896(02)00023-9 |
| [86] | Xia, L. and Conitzer, V. (2011). A maximum likelihood approach towards aggregating partial orders. In Twenty-Second International Joint Conference on Artificial Intelligence. |
| [87] | Yao, G. and Böckenholt, U. (1999). Bayesian estimation of thurstonian ranking models based on the Gibbs sampler. Br. J. Math. Stat. Psychol. 52 79-92. |
| [88] | Yu, P. L. H. (2000). Bayesian analysis of order-statistics models for ranking data. Psychometrika 65 281-299 · Zbl 1291.62265 · doi:10.1007/BF02296147 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
