DeepSeek mHC now official

DeepSeek mHC (Manifold-Constrained Hyper-Connections) is now official. This is the company’s new training method, which analysts say could change how large AI models are scaled.

DeepSeek’s mHC method modifies residual connections to create multiple information streams between layers. In doing so, it also simultaneously mathematically constrains how they mix, keeping signals stable even in deep networks.

As per DeepSeek’s tests on 3B, 9B, and 27B-parameter models, mHC delivered lower loss and better benchmark performance compared to unconstrained hyper-connections. Further, it did so avoiding training instability that typically appears as layers stack.

However, it comes with added training costs of about 6-7 percent. DeepSeek argues that this is negligible at large scale. Analysts from Counterpoint Research, HKUST, and Omdia describe the work as a breakthrough for transformer-based LLMs.

They’re expecting rival labs to develop similar architectures, with the paper’s release fueling speculation that mHC will underpin DeepSeek’s next-gen model. This should translate into the long-rumored R2 model or a future V4.