More Recorded Runs, More Weight: A Decomposition of a Pooled Atari Comparison

Authors

  • Julian Walker Author
  • Jonathan Mitchell Author
  • Gavin Bell Author

Abstract

Simply averaging every available run gives different games different weights, and those weights can differ between the compared conditions. In captured results for 55 Atari games, separately pooled BBF and no-reset means differed by 1.3800 normalized-score units. Giving each game equal weight reduced the gap to 1.0305. An exact symmetric decomposition attributed 0.2410 of the discrepancy to different game composition between conditions and 0.1086 to common reweighting of within-game gaps. Both aggregates favored BBF, but their magnitudes answered different questions. Reporting run counts alongside scores made that distinction recoverable.

Analysis of captured public benchmark results. The decomposition changes the population represented by an aggregate; it does not rerun an agent, identify a causal reset effect or evaluate a game renderer.

Original publication: November 1, 2026.

References

  1. Roberts, David R., et al. “Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure.” Ecography, vol. 40, no. 8, 2017, pp. 913-929. https://doi.org/10.1111/ecog.02881.
  2. Yu, Jiwen, et al. "GameFactory: Creating New Games with Generative Interactive Videos." arXiv preprint arXiv:2501.08325, 2025. https://arxiv.org/abs/2501.08325.
  3. Gebru, Timnit, et al. "Datasheets for datasets" Communications of the ACM, 2021. https://doi.org/10.1145/3458723.
  4. Bruce, Jake, et al. "Genie: Generative Interactive Environments" arXiv preprint arXiv:2402.15391, 2024. https://arxiv.org/abs/2402.15391.
  5. Meng, Zijie; Che, Jinming; Wei, Bingcai; Cao, Xixin. (2026). Make a game: A novel paradigm for interactive game rendering. ICASSP 2026, 1026-1030. https://doi.org/10.1109/ICASSP55912.2026.11461927.
  6. Towers, Mark; Kwiatkowski, Ariel; Terry, Jordan; Balis, John U.; De Cola, Gianluca; Deleu, Tristan; Goulão, Manuel; Kallinteris, Andreas; Krimmel, Markus; KG, Arjun; Perez-Vicente, Rodrigo; Pierré, Andrea; Schulhoff, Sander; Tai, Jun Jet; Tan, Hannah; Younis, Omar G.. (2024). Gymnasium: A Standard Interface for Reinforcement Learning Environments. arXiv:2407.17032. https://arxiv.org/abs/2407.17032.
  7. Elkan, Charles. “The Foundations of Cost-Sensitive Learning.” Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence, 2001, pp. 973-978. https://cseweb.ucsd.edu/~elkan/rescale.pdf.
  8. Kitagawa, Evelyn M.. (1955). Components of a Difference Between Two Rates*. Journal of the American Statistical Association. https://doi.org/10.1080/01621459.1955.10501299.
  9. Stevens, S. S.. (1946). On the Theory of Scales of Measurement. Science. https://doi.org/10.1126/science.103.2684.677.
  10. Fu, Justin, et al. "D4RL: Datasets for Deep Data-Driven Reinforcement Learning." arXiv preprint arXiv:2004.07219, 2020. https://arxiv.org/abs/2004.07219.
  11. Solon, Gary; Haider, Steven J.; Wooldridge, Jeffrey M.. (2015). What Are We Weighting For?. Journal of Human Resources. https://doi.org/10.3368/jhr.50.2.301.
  12. Hampel, Frank R.. (1974). The Influence Curve and its Role in Robust Estimation. Journal of the American Statistical Association. https://doi.org/10.1080/01621459.1974.10482962.
  13. Wasserstein, Ronald L.; Lazar, Nicole A.. (2016). The ASA Statement on p -Values: Context, Process, and Purpose. The American Statistician. https://doi.org/10.1080/00031305.2016.1154108.
  14. Ha, David and Jürgen Schmidhuber. "World Models" arXiv preprint arXiv:1803.10122, 2018. https://arxiv.org/abs/1803.10122.
  15. Ye, Weirui, et al. "Mastering Atari Games with Limited Data." arXiv preprint arXiv:2111.00210, 2021. https://arxiv.org/abs/2111.00210.
  16. Varma, Sudhir and Richard Simon. “Bias in error estimation when using cross-validation for model selection.” BMC Bioinformatics, vol. 7, no. 1, 2006. https://doi.org/10.1186/1471-2105-7-91.
  17. Henderson, Peter; Islam, Riashat; Bachman, Philip; Pineau, Joelle; Precup, Doina; Meger, David. (2017). Deep Reinforcement Learning that Matters. arXiv:1709.06560. https://arxiv.org/abs/1709.06560.
  18. Tassa, Yuval, et al. "DeepMind Control Suite." arXiv preprint arXiv:1801.00690, 2018. https://arxiv.org/abs/1801.00690.
  19. Benjamini, Yoav and Yosef Hochberg. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society Series B: Statistical Methodology, vol. 57, no. 1, 1995, pp. 289-300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x.
  20. Alonso, Eloi, et al. "Diffusion for World Modeling: Visual Details Matter in Atari" arXiv preprint arXiv:2405.12399, 2024. https://arxiv.org/abs/2405.12399.
  21. Huang, Shengyi, et al. "CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms." arXiv preprint arXiv:2111.08819, 2021. https://arxiv.org/abs/2111.08819.
  22. Chen, Lili, et al. "Decision Transformer: Reinforcement Learning via Sequence Modeling." arXiv preprint arXiv:2106.01345, 2021. https://arxiv.org/abs/2106.01345.
  23. Wilkinson, Mark D.; Dumontier, Michel; Aalbersberg, IJsbrand Jan; Appleton, Gabrielle; Axton, Myles; Baak, Arie; Blomberg, Niklas; Boiten, Jan-Willem; da Silva Santos, Luiz Bonino; Bourne, Philip E.; Bouwman, Jildau; Brookes, Anthony J.; Clark, Tim; Crosas, Mercè; Dillo, Ingrid; Dumon, Olivier; Edmunds, Scott; Evelo, Chris T.; Finkers, Richard; Gonzalez-Beltran, Alejandra; Gray, Alasdair J.G.; Groth, Paul; Goble, Carole; Grethe, Jeffrey S.; Heringa, Jaap; ’t Hoen, Peter A.C; Hooft, Rob; Kuhn, Tobias; Kok, Ruben; Kok, Joost; Lusher, Scott J.; Martone, Maryann E.; Mons, Albert; Packer, Abel L.; Persson, Bengt; Rocca-Serra, Philippe; Roos, Marco; van Schaik, Rene; Sansone, Susanna-Assunta; Schultes, Erik; Sengstag, Thierry; Slater, Ted; Strawn, George; Swertz, Morris A.; Thompson, Mark; van der Lei, Johan; van Mulligen, Erik; Velterop, Jan; Waagmeester, Andra; Wittenburg, Peter; Wolstencroft, Katherine; Zhao, Jun; Mons, Barend. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data. https://doi.org/10.1038/sdata.2016.18.
  24. White, Halbert. “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity.” Econometrica, vol. 48, no. 4, 1980, pp. 817-838. https://doi.org/10.2307/1912934.
  25. Agarwal, Rishabh, et al. "Deep Reinforcement Learning at the Edge of the Statistical Precipice" arXiv preprint arXiv:2108.13264, 2021. https://arxiv.org/abs/2108.13264.
  26. Schwarzer, Max; Obando-Ceron, Johan; Courville, Aaron; Bellemare, Marc; Agarwal, Rishabh; Castro, Pablo Samuel. (2023). Bigger, Better, Faster: Human-level Atari with human-level efficiency. arXiv:2305.19452. https://arxiv.org/abs/2305.19452.
  27. Bellemare, Marc G., et al. "The Arcade Learning Environment: An Evaluation Platform for General Agents" arXiv preprint arXiv:1207.4708, 2012. https://arxiv.org/abs/1207.4708.
  28. Gneiting, Tilmann; Raftery, Adrian E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association. https://doi.org/10.1198/016214506000001437.
  29. Hafner, Danijar, et al. "Mastering Diverse Domains through World Models." arXiv preprint arXiv:2301.04104, 2023. https://arxiv.org/abs/2301.04104.
  30. Cook, R. Dennis. (1977). Detection of Influential Observation in Linear Regression. Technometrics. https://doi.org/10.1080/00401706.1977.10489493.

Downloads

Published

2026-11-01