Knowing Quality Helps Predict Alignment, but Does Not Replace It: Human-Score Transfer across Generators

Authors

  • Lucas Reed Author
  • Jeffrey Hayes Author
  • Ryan King Author

Abstract

Can human quality ratings stand in for text-alignment ratings on a different generator? Six leave-one-generator fits on AGIQA-3K map quality MOS to alignment MOS while retaining shared prompt content in training. Every held generator improves over a constant reference, but residual errors and signed biases remain. Equal-generator MSE falls from 1.545 to 0.431; its mean difference has a wide interval, whereas the MAE difference is negative under a small-cohort working summary. The experiment transfers a relationship between human scores, with no automatic image assessment or unseen-prompt claim.

Original publication: November 1, 2026.

References

  1. Gneiting, Tilmann; Raftery, Adrian E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association. https://doi.org/10.1198/016214506000001437.
  2. Ovadia, Yaniv, et al. “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift.” Advances in Neural Information Processing Systems, vol. 32, 2019. https://arxiv.org/abs/1906.02530.
  3. Cameron, A. Colin, and Douglas L. Miller. “A Practitioner’s Guide to Cluster-Robust Inference.” Journal of Human Resources, vol. 50, no. 2, 2015, pp. 317-372. https://doi.org/10.3368/jhr.50.2.317.
  4. Gebru, Timnit, et al. "Datasheets for datasets" Communications of the ACM, 2021. https://doi.org/10.1145/3458723.
  5. Lin, Lawrence I-Kuei. (1989). A Concordance Correlation Coefficient to Evaluate Reproducibility. Biometrics. https://doi.org/10.2307/2532051.
  6. White, Halbert. “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity.” Econometrica, vol. 48, no. 4, 1980, pp. 817-838. https://doi.org/10.2307/1912934.
  7. Roberts, David R., et al. “Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure.” Ecography, vol. 40, no. 8, 2017, pp. 913-929. https://doi.org/10.1111/ecog.02881.
  8. Varma, Sudhir and Richard Simon. “Bias in error estimation when using cross-validation for model selection.” BMC Bioinformatics, vol. 7, no. 1, 2006. https://doi.org/10.1186/1471-2105-7-91.
  9. Ke, Junjie, et al. "MUSIQ: Multi-scale Image Quality Transformer" arXiv preprint arXiv:2108.05997, 2021. https://arxiv.org/abs/2108.05997.
  10. Carroll, Raymond J., David Ruppert, Leonard A. Stefanski, and Ciprian M. Crainiceanu. “Measurement Error in Nonlinear Models.” 2006. https://doi.org/10.1201/9781420010138.
  11. Meng, Zijie. (2026). Decoupling Semantics from Distortions: Multi-Scale Two-Stream Vision-Language Alignment for AI-Generated Image Quality Assessment. arXiv:2606.16799. https://arxiv.org/abs/2606.16799.
  12. Fleiss, Joseph L.. (1971). Measuring nominal scale agreement among many raters.. Psychological Bulletin. https://doi.org/10.1037/h0031619.
  13. Yang, Sidi, et al. "MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment." arXiv preprint arXiv:2204.08958, 2022. https://arxiv.org/abs/2204.08958.
  14. Chai, T.; Draxler, R. R.. (2014). Root mean square error (RMSE) or mean absolute error (MAE)? - Arguments against avoiding RMSE in the literature. Geoscientific Model Development. https://doi.org/10.5194/gmd-7-1247-2014.
  15. Murray, Naila; Marchesotti, Luca; Perronnin, Florent. (2012). AVA: A large-scale database for aesthetic visual analysis. 2012 IEEE Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/cvpr.2012.6247954.
  16. Pearson, Karl. (1901). LIII. On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science. https://doi.org/10.1080/14786440109462720.
  17. Li, Chunyi, et al. "AGIQA-3K: An Open Database for AI-Generated Image Quality Assessment" arXiv preprint arXiv:2306.04717, 2023. https://arxiv.org/abs/2306.04717.
  18. Lin, Hanhe, et al. "KADID-10k: A Large-scale Artificially Distorted IQA Database" 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX), 2019. https://doi.org/10.1109/qomex.2019.8743252.
  19. Hodson, Timothy O.. (2022). Root-mean-square error (RMSE) or mean absolute error (MAE): when to use them or not. Geoscientific Model Development. https://doi.org/10.5194/gmd-15-5481-2022.
  20. Hoerl, Arthur E.; Kennard, Robert W.. (1970). Ridge Regression: Biased Estimation for Nonorthogonal Problems. Technometrics. https://doi.org/10.1080/00401706.1970.10488634.
  21. Talebi, Hossein and Peyman Milanfar. "NIMA: Neural Image Assessment" arXiv preprint arXiv:1709.05424, 2017. https://arxiv.org/abs/1709.05424.
  22. Ding, Keyan, et al. "Image Quality Assessment: Unifying Structure and Texture Similarity." arXiv preprint arXiv:2004.07728, 2020. https://arxiv.org/abs/2004.07728.
  23. Willmott, CJ; Matsuura, K. (2005). Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Climate Research. https://doi.org/10.3354/cr030079.
  24. Fang, Yuming; Zhu, Hanwei; Zeng, Yan; Ma, Kede; Wang, Zhou. (2020). Perceptual Quality Assessment of Smartphone Photography. CVPR 2020, 3677-3686. https://openaccess.thecvf.com/content_CVPR_2020/html/Fang_Perceptual_Quality_Assessment_of_Smartphone_Photography_CVPR_2020_paper.html.
  25. Stevens, S. S.. (1946). On the Theory of Scales of Measurement. Science. https://doi.org/10.1126/science.103.2684.677.
  26. Wilkinson, Mark D.; Dumontier, Michel; Aalbersberg, IJsbrand Jan; Appleton, Gabrielle; Axton, Myles; Baak, Arie; Blomberg, Niklas; Boiten, Jan-Willem; da Silva Santos, Luiz Bonino; Bourne, Philip E.; Bouwman, Jildau; Brookes, Anthony J.; Clark, Tim; Crosas, Mercè; Dillo, Ingrid; Dumon, Olivier; Edmunds, Scott; Evelo, Chris T.; Finkers, Richard; Gonzalez-Beltran, Alejandra; Gray, Alasdair J.G.; Groth, Paul; Goble, Carole; Grethe, Jeffrey S.; Heringa, Jaap; ’t Hoen, Peter A.C; Hooft, Rob; Kuhn, Tobias; Kok, Ruben; Kok, Joost; Lusher, Scott J.; Martone, Maryann E.; Mons, Albert; Packer, Abel L.; Persson, Bengt; Rocca-Serra, Philippe; Roos, Marco; van Schaik, Rene; Sansone, Susanna-Assunta; Schultes, Erik; Sengstag, Thierry; Slater, Ted; Strawn, George; Swertz, Morris A.; Thompson, Mark; van der Lei, Johan; van Mulligen, Erik; Velterop, Jan; Waagmeester, Andra; Wittenburg, Peter; Wolstencroft, Katherine; Zhao, Jun; Mons, Barend. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data. https://doi.org/10.1038/sdata.2016.18.
  27. Tipping, Michael E.; Bishop, Christopher M.. (1999). Probabilistic Principal Component Analysis. Journal of the Royal Statistical Society Series B: Statistical Methodology. https://doi.org/10.1111/1467-9868.00196.
  28. Wang, Jianyi, et al. "Exploring CLIP for Assessing the Look and Feel of Images." arXiv preprint arXiv:2207.12396, 2022. https://arxiv.org/abs/2207.12396.
  29. Kapoor, Sayash; Narayanan, Arvind. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns. https://doi.org/10.1016/j.patter.2023.100804.
  30. Bland, J. Martin, and Douglas G. Altman. "Statistical Methods for Assessing Agreement between Two Methods of Clinical Measurement." The Lancet, vol. 327, no. 8476, 1986, pp. 307-310. https://doi.org/10.1016/S0140-6736(86)90837-8.

Downloads

Published

2026-11-01