Merging Abilities when Offline Values Have Different Meanings
Abstract
Cross-modality model merging and heterogeneous value alignment combine information at different levels. Evaluation should separately retain grounded perceptual outcomes and compatible transition-based value definitions. An aggregate answer score cannot establish policy value, while value alignment cannot verify visual grounding. A useful common budget requires independently checked task-specific consequences. This focused synthesis develops an empirical comparison that preserves the terminal action, observation process, and total verification cost. Its proposed tests are prospective; no new simulation run, model-training result, or intervention effect is reported.
References
Z. Qiao, J. Lyu, C. Bai, P. Wang, S. Gao, and S. Qiu, “Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasets,” Proceedings of the 2026 International Conference on Machine Learning, 2026.
S. Chen, J. Zhang, T. Zhu, W. Liu, S. Gao, M. Xiong, M. Li, and J. He, “Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging,” Proceedings of the 42nd International Conference on Machine Learning, 2025.
Li, Yuanhao, Mingshan Liu, Hongbo Wang, Yiding Zhang, Yifei Ma, and Wei Tan. “DRAFT-RL: Multi-Agent Chain-of-Draft Reasoning for Reinforcement Learning-Enhanced LLMs.” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 35, 2026, pp. 29530-29537. https://doi.org/10.1609/aaai.v40i35.40195.
Levine, Sergey, Aviral Kumar, George Tucker, and Justin Fu. “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems.” arXiv, 2020, arXiv:2005.01643. https://arxiv.org/abs/2005.01643.
Kumar, Aviral, Aurick Zhou, George Tucker, and Sergey Levine. “Conservative Q-Learning for Offline Reinforcement Learning.” Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 1179–1191. https://arxiv.org/abs/2006.04779.
Lewis, Patrick, et al. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 9459–9474. https://arxiv.org/abs/2005.11401.
Geifman, Yonatan, and Ran El-Yaniv. “Selective Classification for Deep Neural Networks.” Advances in Neural Information Processing Systems, vol. 30, 2017. https://arxiv.org/abs/1705.08500.
Ovadia, Yaniv, et al. “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift.” Advances in Neural Information Processing Systems, vol. 32, 2019. https://arxiv.org/abs/1906.02530.
Bertsimas, Dimitris, and Melvyn Sim. “The Price of Robustness.” Operations Research, vol. 52, no. 1, 2004, pp. 35–53. https://doi.org/10.1287/opre.1030.0065.
Rasmussen, Carl Edward, and Christopher K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006. https://gaussianprocess.org/gpml/.
Hu, Edward J., et al. “LoRA: Low-Rank Adaptation of Large Language Models.” International Conference on Learning Representations, 2022. https://openreview.net/forum?id=nZeVKeeFYf9.
Published
Versions
- Version 2
- Version 1