mHC: Manifold-Constrained Hyper-Connections
Xie Z , Wei Y , Cao H ,et al.mHC: Manifold-Constrained Hyper-Connections[J]. 2025.
DeepSeek
mHC：流形约束超连接
Abstract
Recently, studies exemplified by Hyper-Connections (HC) have extended the ubiquitous residual connection paradigm established over the past decade by expanding the residual stream width and diversifying connectivity patterns. While yielding substantial performance gains, this diversification fundamentally compromises the identity mapping property intrinsic to the residual connection, which causes severe training instability and restricted scalability, and additionally incurs notable memory access overhead. To address these challenges, we propose Manifold-Constrained Hyper-Connections (mHC), a general framework that projects the residual connection space of HC onto a specific manifold to restore the identity mapping property, while incorporating rigorous infrastructure optimization to ensure efficiency. Empirical experiments demonstrate that mHC is effective for training at scale, offering tangible performance improvements and superior scalability. We anticipate that mHC, as a flexible and practical extension of HC, will contribute to a deeper understanding of topological architecture design and suggest promising directions for the evolution of foundational models.
近期，以超连接网络（HC）为代表的研究通过扩展残差流的宽度并丰富连接模式，发展了近十年来广泛应用的残差连接范式。尽管这种方法带来了显著的性能提升，但其多样化的连接方式从根本上削弱了残差连接固有的恒等映射特性，导致严重的训练不稳定性与可扩展性受限，同时产生了显著的内存访问开销。为解决这些问题，我们提出流形约束超连接网络（mHC）——一个通用框架，通过将HC的残差连接空间投影至特定流形以恢复恒等映射特性，并结合严格的基础设施优化以确保效率。实验证明，mHC能有效支持大规模训练，在提升性能的同时展现出卓越的可扩展性。我们预期，mHC作为HC框架的灵活实用拓展，将深化对拓扑架构设计的理解，并为基础模型的演进提供值得关注的研究方向。