Publication Cohorts and Current Severity Labels in Reviewed Python Package Advisories

Authors

  • Jeremy Franklin Author
  • Robert Bailey Author
  • Matthew Fisher Author

Abstract

Does a change in the severity mix of published advisories survive adjustment for which packages are represented? In 2,948 eligible PyPI advisory records, the current HIGH-or-CRITICAL share was 52.96% for publications in 2022-2023 and 49.96% in 2024-2025. Among 131 packages represented in both windows, a package-adjusted contrast was larger, at −12.89 percentage points. That finding concerns current labels attached to different publication cohorts. Many records were subsequently modified, so it cannot establish a historical reduction in software risk or the information available to a tool-selection system at the time.

Retrospective description of a captured public database. Current advisory content is grouped by publication date; it is not reconstructed historical knowledge or a forecast of security incidents.

Original publication: November 4, 2026.

References

  1. Ioannidis, John P. A.. (2005). Why Most Published Research Findings Are False. PLoS Medicine. https://doi.org/10.1371/journal.pmed.0020124.
  2. Gneiting, Tilmann; Raftery, Adrian E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association. https://doi.org/10.1198/016214506000001437.
  3. Li, Qinfei; Dong, Xiaoxuan; Zhang, Jin; Yu, Dexu; Deng, Wenhao; Fu, Junchen; Li, Youhua; Du, Hanwen; Li, Chunxiao. (2026). Risk-Aware Reranking for Agentic Tool Retrieval. arXiv:2608.22751. https://arxiv.org/abs/2608.22751.
  4. Stuart, Elizabeth A. "Matching Methods for Causal Inference: A Review and a Look Forward." Statistical Science, vol. 25, no. 1, 2010, pp. 1-21. https://doi.org/10.1214/09-sts313.
  5. White, Halbert. “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity.” Econometrica, vol. 48, no. 4, 1980, pp. 817-838. https://doi.org/10.2307/1912934.
  6. GitHub. (2026). GitHub Advisory database. Official documentation snapshot accessed 7 November 2026. https://docs.github.com/en/code-security/concepts/vulnerability-reporting-and-management/github-advisory-database.
  7. Debenedetti, Edoardo; Zhang, Jie; Balunović, Mislav; Beurer-Kellner, Luca; Fischer, Marc; Tramèr, Florian. (2024). AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. arXiv:2406.13352. https://arxiv.org/abs/2406.13352.
  8. OpenSSF. (2026). Open Source Vulnerability format, schema version 1.9.2. Official documentation snapshot accessed 7 November 2026. https://ossf.github.io/osv-schema/.
  9. Zhan, Qiusi; Liang, Zhixiang; Ying, Zifan; Kang, Daniel. (2024). InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents. arXiv:2403.02691. https://arxiv.org/abs/2403.02691.
  10. Cameron, A. Colin, and Douglas L. Miller. “A Practitioner’s Guide to Cluster-Robust Inference.” Journal of Human Resources, vol. 50, no. 2, 2015, pp. 317-372. https://doi.org/10.3368/jhr.50.2.317.
  11. Ovadia, Yaniv, et al. “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift.” Advances in Neural Information Processing Systems, vol. 32, 2019. https://arxiv.org/abs/1906.02530.
  12. Fawcett, Tom. (2006). An introduction to ROC analysis. Pattern Recognition Letters. https://doi.org/10.1016/j.patrec.2005.10.010.
  13. Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. “On Calibration of Modern Neural Networks.” Proceedings of the 34th International Conference on Machine Learning, PMLR, vol. 70, 2017, pp. 1321-1330. https://proceedings.mlr.press/v70/guo17a.html.
  14. Li, Minghao; Zhao, Yingxiu; Yu, Bowen; Song, Feifan; Li, Hangyu; Yu, Haiyang; Li, Zhoujun; Huang, Fei; Li, Yongbin. (2023). API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs. arXiv:2304.08244. https://arxiv.org/abs/2304.08244.
  15. OSV Project. (2026). OSV: Frequently Asked Questions. Official documentation snapshot accessed 7 November 2026. https://google.github.io/osv.dev/faq/.
  16. Greshake, Kai; Abdelnabi, Sahar; Mishra, Shailesh; Endres, Christoph; Holz, Thorsten; Fritz, Mario. (2023). Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv:2302.12173. https://arxiv.org/abs/2302.12173.
  17. Yao, Shunyu; Zhao, Jeffrey; Yu, Dian; Du, Nan; Shafran, Izhak; Narasimhan, Karthik; Cao, Yuan. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629. https://arxiv.org/abs/2210.03629.
  18. Roberts, David R., et al. “Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure.” Ecography, vol. 40, no. 8, 2017, pp. 913-929. https://doi.org/10.1111/ecog.02881.
  19. Little, Roderick J. A., and Donald B. Rubin. Statistical Analysis with Missing Data. Wiley, 2002. https://doi.org/10.1002/9781119013563.
  20. Saito, Takaya and Marc Rehmsmeier. “The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets.” PLOS ONE, vol. 10, no. 3, 2015, pp. e0118432. https://doi.org/10.1371/journal.pone.0118432.
  21. Bird, Christian; Rigby, Peter C.; Barr, Earl T.; Hamilton, David J.; German, Daniel M.; Devanbu, Premkumar. (2009). The promises and perils of mining git. https://doi.org/10.1109/MSR.2009.5069475.
  22. Gebru, Timnit, et al. "Datasheets for datasets" Communications of the ACM, 2021. https://doi.org/10.1145/3458723.
  23. Greenland, Sander, Judea Pearl, and James M. Robins. “Causal Diagrams for Epidemiologic Research.” Epidemiology, vol. 10, no. 1, 1999, pp. 37-48. https://doi.org/10.1097/00001648-199901000-00008.
  24. Python Packaging Authority. (2026). Names and normalization. Official documentation snapshot accessed 7 November 2026. https://packaging.python.org/en/latest/specifications/name-normalization/.
  25. Lu, Jiarui; Holleis, Thomas; Zhang, Yizhe; Aumayer, Bernhard; Nan, Feng; Bai, Felix; Ma, Shuang; Ma, Shen; Li, Mengyu; Yin, Guoli; Wang, Zirui; Pang, Ruoming. (2024). ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities. arXiv:2408.04682. https://arxiv.org/abs/2408.04682.
  26. Herzig, Kim; Just, Sascha; Zeller, Andreas. (2013). It’s not a bug, it’s a feature: How misclassification impacts bug prediction. https://doi.org/10.1109/ICSE.2013.6606585.
  27. Wilkinson, Mark D.; Dumontier, Michel; Aalbersberg, IJsbrand Jan; Appleton, Gabrielle; Axton, Myles; Baak, Arie; Blomberg, Niklas; Boiten, Jan-Willem; da Silva Santos, Luiz Bonino; Bourne, Philip E.; Bouwman, Jildau; Brookes, Anthony J.; Clark, Tim; Crosas, Mercè; Dillo, Ingrid; Dumon, Olivier; Edmunds, Scott; Evelo, Chris T.; Finkers, Richard; Gonzalez-Beltran, Alejandra; Gray, Alasdair J.G.; Groth, Paul; Goble, Carole; Grethe, Jeffrey S.; Heringa, Jaap; ’t Hoen, Peter A.C; Hooft, Rob; Kuhn, Tobias; Kok, Ruben; Kok, Joost; Lusher, Scott J.; Martone, Maryann E.; Mons, Albert; Packer, Abel L.; Persson, Bengt; Rocca-Serra, Philippe; Roos, Marco; van Schaik, Rene; Sansone, Susanna-Assunta; Schultes, Erik; Sengstag, Thierry; Slater, Ted; Strawn, George; Swertz, Morris A.; Thompson, Mark; van der Lei, Johan; van Mulligen, Erik; Velterop, Jan; Waagmeester, Andra; Wittenburg, Peter; Wolstencroft, Katherine; Zhao, Jun; Mons, Barend. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data. https://doi.org/10.1038/sdata.2016.18.
  28. Wasserstein, Ronald L.; Lazar, Nicole A.. (2016). The ASA Statement on p -Values: Context, Process, and Purpose. The American Statistician. https://doi.org/10.1080/00031305.2016.1154108.
  29. Ruan, Yangjun; Dong, Honghua; Wang, Andrew; Pitis, Silviu; Zhou, Yongchao; Ba, Jimmy; Dubois, Yann; Maddison, Chris J.; Hashimoto, Tatsunori. (2023). Identifying the Risks of LM Agents with an LM-Emulated Sandbox. arXiv:2309.15817. https://arxiv.org/abs/2309.15817.
  30. Carroll, Raymond J., David Ruppert, Leonard A. Stefanski, and Ciprian M. Crainiceanu. “Measurement Error in Nonlinear Models.” 2006. https://doi.org/10.1201/9781420010138.

Downloads

Published

2026-11-04