Extended-stability Runge–Kutta solvers for Neural ODE: Faster, more stable training with predictable compute

Neural ordinary differential equations provide a principled continuous depth formulation, yet their practical use is often limited by slow, numerically fragile, and computationally expensive training caused by the weaknesses of standard ODEs solvers. We introduce extended stability Runge Kutta methods, which are explicit fixed step solvers designed to remain stable at much larger step sizes than classical schemes such as RK4 or adaptive methods such as Dormand Prince. By enlarging the stability region of the real axis, these solvers cross the integration horizon with far fewer steps, providing deterministic computation, greatly reduced function evaluation cost, and smoother optimization dynamics. Across CIFAR-10, CIFAR-100, and Tiny-ImageNet, extended stability solvers match the final accuracy of higher-order methods, usually within one to two percent, while producing significantly more stable gradients. On CIFAR 10, they reduce gradient clipping rates to between zero and twenty-five percent depending on integration horizon, compared with sixty to one hundred percent for standard solvers, and they maintain reduced spectral amplification in moderately stiff regimes, while preserving coherent gradient flow in more extreme stiffness settings. The 15-stage variant achieves substantial reductions in wall-clock time relative to Dormand–Prince 5 and improved computational efficiency compared with ResNet-20 while requiring significantly fewer function evaluations, with comparable floating point operations to RK4.They also allow successful optimization in regimes where classical explicit solvers become unstable or diverge. Overall, the results show that the geometry of the stability region, rather than formal order or adaptivity, is the key factor that governs gradient flow conditioning under a fixed integration horizon. Higher-order methods can reach similar accuracy but suffer from unstable and oscillatory gradients. Extended stability methods maintain coherent gradient flow while providing substantial practical speedups, making stability region geometry an effective and simple tool for accelerating and stabilizing the training of neural ordinary differential equation models.