Self-evolving meta-learning neural ordinary differential equations: A novel artificial intelligence-driven approach for solving differential equations

Abstract

Solving ordinary differential equations (ODEs) constitutes a fundamental problem for many scientific and engineering disciplines, particularly for stiff, high-dimensional problems, or problems with changing dynamics. Consequently, traditional numerical solvers, such as the Runge–Kutta methods, suffer from the drawback of computational inefficiency, numerical in stability and a lack of good generality across a range of problems. However, some of these limitations had been overcome with the advent of deep learning methods, leading to the development of neural ordinary differential equations (Neural ODEs) methods. However, the Neural ODEs fixed architecture makes it less versatile and especially in cases of solving complex and nonlinear ODEs problems. To address these problems, this research proposes self-evolving meta-learning neural ODE (SEML-NODE), a new approach of meta-learning and evolutionary computing for making a self-adjusting neural solver that is capable of evolving its own architecture during training. The main novelty of SEML-NODE is the self-evolution mechanism that enables the neural network's structure to expand and/or contract according to error feedback so that both the accuracy and generalization are improved. The adaptive training process renders the new extrapolation method to have superior performance on the stiff or chaotic ODEs. In this work, we evaluate SEML-NODE on benchmark problems, i.e. logistic growth model, Kepler's problem, and Fisher's equation, which contain difficulty such as nonlinearity, stiffness, and spatial-temporal dynamical problem. The results show that SEML-NODE is consistently better than state-of-the-art methods, such as fixed Neural ODEs and physics-informed neural networks, achieving significantly improved results in terms of errors on all benchmark tasks. Since the framework is able to dynamically change architecture depending on complexity of problem, by solving more complex, time-varying systems is very efficient, which is a problem for traditional solvers. While the current dynamic network expansion has a big initial cost in computed power, it becomes more efficient as the network training progresses by eliminating unnecessary network complexity and only expanding as needed. However, the improved optimization of self-evolution parameters (such as the error threshold of network expansion) has been challenging to prevent overfitting. To overcome this, for future work, the optimization of these parameters and the extension of the SEML-NODE to partial differential equations will be the focus. Besides, reinforcement learning will also be explored to further optimize the architecture. The proposed SEML-NODE framework greatly improves the adaptive ODE solver development and may find many applications in real-world areas such as climate prediction, drug discovery, and financial prediction, where the traditional numerical methods cannot provide efficient solutions.

Introduction

Ordinary differential equations (ODEs) play a fundamental role in the modeling of dynamic systems in a large range of sciences and technologies, including physics, engineering, biology and finance. These equations are required not only to determine the time-dependent behavior of a system but also to predict the appearance of system behavior e.g. the movements of populations and fluids as well as financial systems, regulation of gene expression and propagation of disease.^1,2 Nevertheless, ODE are also difficult to solve despite the widespread use of the approach, particularly due to the response being stiff, high-dimensional, or time-varying. As an example, in stiff systems, time scales can vary enormously in their dynamics and the conventional solvers are inefficient and unstable in such systems. ³ This is further compounded when an individual is dealing with chaotic or nonlinear systems, which are sensitive to initial condition and demand high preciseness to perfectly capture the dynamics of a given system.

On the contrary, traditional numerical approaches to solve ODEs, consisting of methods like the Runge–Kutta method, the finite difference schemes and implicit solvers have been a backbone of state-of-the-art ODE analysis approaches for decades. ⁴ These methods are based on discrete time steps and calculation of the states of the system at discrete time steps. However, they often face substantial difficulties among convergence of speed, numerical stability, adaptability to changing system dynamics, etc. when applied to stiff, high-dimensional and complex systems. For example, in problems governing stiff dynamic processes, such as chemical kinetics, celestial mechanics, or climate modeling, solvers need to capture rapidly varying nature to some variables while preserving stability and accuracy to other variables. ⁵ As a result, such traditional strategies are often insufficient for the task of offering a common solution framework for the broad selection of ODEs that arise in practical applications.

To solve these problems, machine learning has attracted attention, suggesting new paradigms to solve problems of ODEs. Among these approaches, neural ordinary differential equations (Neural ODEs) have attracted a lot of attention since they try to model dynamics of continuous time signals directly by deep learning architectures. Neural ODEs express the rate of change of system by use of a neural network and facilitate to learn the system dynamics in a data-driven manner.^6,7 During the training phase, the discrepancy between the predicted and actual states is minimized in the form of a loss function and the backpropagation through time is carried out using the adjoint sensitivity method, which is very useful in memory efficiency. ⁸ Neural ODEs have been applied successfully in a variety of domains, such as trajectory prediction, physics simulation and control systems.^9,10 However, whereas fixed architecture Neural ODEs are extremely useful, they may lack the ability to generalize across a broad set of types of problems. Specifically, the fixed architecture of such models, even the classification of them as those on defined network architectures, can affect their potential to directly capture the dynamics of chaotic, stiff, or high-dimensional systems. ¹¹ Therefore, a much more flexible approach is required to enhance their ability to generalize. The work reported in this report sets out to overcome these limitations by examining more flexible model structures.

To address these shortcomings, self-evolving meta-learning neural ODE (SEML-NODE) has introduced, a new framework whereby the purpose of meta-learning and evolutionary computing is synergistically combined to develop an adaptive neural solver of ODEs. Secondly, a self-evolution mechanism of allowing the neural network architecture to expand or contract based on feedback of learning error dynamically is introduced in SEML-NODE. The self-adaptive architecture of this network optimizes the network's accuracy and generalization capabilities to learn the complexities of nonlinear, stiff, and chaotic ODEs much better and without overfitting. The architecture of the network changes with complexity: the structure evolves to solve the system dynamics it is modeling, thus creating greater flexibility compared to the conventional fixed architecture models.^12,13

This work has evaluated SEML-NODE on several benchmark problems, and focusing on other aspects of the challenge of ODE problems. Other such models are the logistic growth model, ¹⁴ Kepler's problem ¹⁵ and Fisher's equation ¹⁶ that are used to model and describe population dynamics, the motion of celestial bodies in elliptical orbits and population diffusion and reaction processes, respectively. These problems encompass many different types of ODE, including simple nonlinear dynamics, but also complex systems involving problems in space and time variations. It has also performed extensive experiments to show that SEML-NODE over performs the existing state-of-the-art methods like fixed Neural ODE, physics-informed neural networks (PINNs) and other adaptive solvers^17,18 in terms of solving complex systems with lower prediction errors and higher efficiency.

Despite the potential in SEML-NODE, however, there is a computational overhead that the framework carries, especially during the early time of the network expansion. However, this overhead is compensated by the fact that the model does not have to be extremely complex once the network is grown enough to catch up with requirements of the problem you are trying to solve. However, parameters used with the self-evolution process such as the error threshold for network expansion must be carefully adjusted so as not to overfit. This is necessary in order to insure evolution of the architecture in a way that is best suited for the problem at hand. Insights gained by this work will be exploited for the optimization of these parameters and extension of the SEML-NODE framework to more common nonlinear isolations, such as the solution of partial differential equations (PDEs), which is used for modeling of phenomena in areas such as fluid dynamics, material science, environmental modeling and more. ¹⁹

Section Background then starts with giving a comprehensive background of ODE solving along with the problems that traditional solvers of ODE have and the emergence of machine learning-based solvers for solving traditional solvers. Details of the SEML-NODE framework are then introduced in Section Neural networks for ODE solving including mathematical formulation, the self-evolution mechanism and the training strategy. Section Methods: SEML-NODE framework presents experimental results of SEML-NODE on logistic growth model, the Kepler's problem, and Fisher's equation. The strengths and limitations of the framework are discussed in Section Experiments and results together with suggestions on how the framework can be applied in other practical situations. Finally, this paper summarizes the contributions of SEML-NODE and its possible scenarios of use in real-world scenarios in Section Discussion.

Background

ODEs play a key role in the modeling of dynamic systems in a variety of fields of scientific use. ODEs have been found important to the evolution of biological systems, fluid systems and climate modeling as well as control systems and finance. Nonetheless, exceptionally stiff, high-dimensional, or nonlinear ODEs are still a complicated problem that requires sophisticated techniques such as neural networks and machine learning to be provided in practice.

Background on ODE solving

An ODE is an equation that involves an unknown function $y(t)$ , its derivative with respect to time t and possibly other variables. The general form of a first-order ODE can be written as in Equation (1):

$${\displaystyle \frac{dy}{dt}=f(y(t),t),y^{\mathrm{\prime }}(0)=y_0^{\mathrm{\prime }}}$$

(1)

Here, $y(t)$ represents the state of the system at time t and the function $f(y(t),t)$ defines how the system evolves over time. The domain of the ODE is determined by the domain of the function $f(y(t),t)$ , which is defined over a subset of the $(t,y)$ plane.

In many numerical frameworks, higher order differential equations are transformed into equivalent systems of first-order differential equations so as to employ standard numerical methods. This allows the higher order ODEs to be systematically brought into the form of systems of first-order equations by introducing suitable auxiliary variables. ²⁰ For instance, a second-order ODE can be represented as in Equation (2):

$${\displaystyle \frac{d^{2}y}{d{t}^2}=f(y(t),t),}$$

(2)

This second-order equation can be rewritten as a pair of coupled first-order equations by defining an auxiliary variable $v(t)=\frac{dy(t)}{dt}$ .

Substituting this into Equation (2) leads to Equation (3)

$${\displaystyle \frac{dv(t)}{dt}=f(y(t),t),}$$

(3)

This transformation allows second-order ODEs to be handled using the same numerical approaches developed for first-order systems.

Traditional numerical methods to solve ODEs include Euler's method, Runge–Kutta method and finite difference schemes. For instance, the explicit Euler method for approximating the solution of an ODE is given by Equation (4). ²¹

$$y(t+\mathrm{\Delta }t)=y(t)+\mathrm{\Delta }tf(t,y(t)),$$

(4)

While Euler's method is simple and fast, it is prone to numerical instability in the case of stiff systems, where the solution changes at vastly different rates for different system components. To address this, Runge–Kutta methods use intermediate points for higher order accuracy. The classic fourth-order Runge–Kutta method for solving ODEs is represented by Equation (5):

$$\begin{array}{c}{k}_1=\mathrm{\Delta }t.f(y(t),t)\\ k_2=\mathrm{\Delta }t.f\left(y(t)+{\displaystyle \frac{k_1}{2},t+{\displaystyle \frac{\mathrm{\Delta }t}{2}}}\right)\\ k_3=\mathrm{\Delta }t.f\left(y(t)+{\displaystyle \frac{k_2}{2},t+{\displaystyle \frac{\mathrm{\Delta }t}{2}}}\right)\\ k_4=\mathrm{\Delta }t.f(y(t)+k_3,t+\mathrm{\Delta }t)\\ y(t+\mathrm{\Delta }t)=y(t)+{\displaystyle \frac{1}{6}(k_1+2k_2+2k_3+k_4)}\end{array}\},$$

(5)

The accuracy benefits of Runge–Kutta methods persist, yet they share the same computational complexity when solving stiff systems that exhibit various timescale dynamics.

Limitations of traditional methods

Traditional solvers function effectively, but their capabilities stop when they encounter systems with high dimensions and chaotic problems that become unpredictable due to initial condition variations. These solvers also lack adaptability because users need to manually adjust such parameters as step size while they remain sub-optimal for different system types. The expansion of systems and their dimensional complexity requires exponential growth in computational cost, leading to impracticability during simulations at a large-scale level in climate modeling and biological fields.

Neural networks for ODE solving

Machine learning technology has produced Neural ODEs through its recent advancements to model continuous dynamics of ODEs by utilizing neural networks. The Neural ODE framework models the system evolution through a neural network as shown in Equation (6):

$${\displaystyle \frac{dy(t)}{dt}=f_{\theta }(y(t),t),}$$

(6)

where $f_{\theta }(y(t),t)$ a neural network is parameterized by $\theta y(t)$ and is the system's state at the time. The goal is to learn the network parameters $\theta$ such that the predicted solution $y(t)$ matches the proper solution of the system.

The loss function for Neural ODEs, used for training, is defined in Equation (7), which calculates the error between predicted and actual states:

$$L(\theta )={\displaystyle \frac{1}{T}\sum _{t=1}^T(y_{true}(t)-y_{pred}(t){)}^2,}$$

(7)

Where $y_{ture}(t)$ is the exact solution at the time t, and $y_{pred}(t)$ is the predicted solution

One of the key innovations of Neural ODEs is their use of the adjoint sensitivity method for backpropagation, which allows for efficient training of deep neural networks in continuous time. The adjoint sensitivity method used in Neural ODEs for backpropagation is represented by Equation (8), which defines the adjoint state:

$${\displaystyle \frac{d\lambda (t)}{dt}=-\lambda (t).{\displaystyle \frac{\mathrm{\partial }{f}_{\theta }(y(t),t)}{\mathrm{\partial }y},}}$$

(8)

with a terminal condition. This approach significantly reduces memory usage compared to traditional backpropagation algorithms. ²²

With the adjoint state $\lambda (t)$ and the final condition $\lambda (T)=\frac{\mathrm{\partial }L}{\mathrm{\partial }y(T)}$ , the gradients for the network parameters are calculated as shown in Equation (9), based on the adjoint state:

$${\displaystyle \frac{\mathrm{\partial }L}{\mathrm{\partial }\theta }=\underset{0}{\overset{T}{\int }}\lambda (t).{\displaystyle \frac{\mathrm{\partial }{f}_{\theta }(y(t),t)}{\mathrm{\partial }\theta }dt,}}$$

(9)

There are some limitations to Neural ODEs but there are also significant benefits (e.g. memory efficiency, use of the black box). Fixed network architectures are extensible to more general types of ODE-variation issues, mainly for stiff and chaotic systems. For these reasons, these types of problems often need adaptive architectures to account for the complexity of the dynamics. ²³

Need for meta-learning and evolutionary computing

However, even this is not good enough as Neural ODEs are built on fixed architectures. Therefore, meta-learning, evolutionary computing techniques for the design of adaptive solvers for a larger class of ODEs have been proposed to overcome the limitations of fixed architecture Neural ODEs. In meta-learning or “learning to learn,” models have the ability to adapt the learning strategy to previous tasks while they are being operated. For ODE solving, meta-learning helps a model to perform optimization over the course its metaprocess and can be setting hyperparameters like learning rate, weight initialization and learning coefficients. ²⁴

The update rule for the learning rate in meta-learning is defined by Equation (10), which adjusts the learning rate based on the meta-loss.

$${\eta }_t={\eta }_{t-1}-\gamma .\mathrm{\nabla }L(\theta ),$$

(10)

Where ${\eta }_t$ is the learning rate at the time step t, $\gamma$ is a decay factor, and $\mathrm{\nabla }L(\theta )$ is the loss gradient concerning the model parameters.

In particular, evolutionary computing results in dynamic adaptation of the network architecture. Evolutionary algorithms change the neural network's architecture during training according to the error feedback. For instance, the rule for updating the hidden dimension in evolutionary algorithms is described in Equation (11), based on error feedback

$$d_{t+1}=d_t+\lambda .\mathrm{\Delta }d,$$

(11)

Where $d_t$ is the current hidden dimension at time t, and $\lambda$ is the scaling factor and $\mathrm{\Delta }d$ changes the current dimension according to the current error feedback. It enables the network to scale as needed for complicated problems or contract for easier solutions with both accuracy and efficiency. ²⁵

Combining meta-learning with evolutionary computation yields a self-evolving Neural ODE solver that can alter its architecture on the fly concerning the complexity of the ODE to be solved. ²⁶ With this approach, the solver generalizes to a vast class of ODEs, from simple linear systems to very general, nonlinear, stiff and chaotic systems. ²⁷

Methods: SEML-NODE framework

This section details the SEML-NODE framework explained in the previous section with its mathematical formulation, self-evolution mechanism and training strategy. The aim is to provide the theoretical basis and numerical procedures necessary for SEML-NODE to dynamically change its architecture to solve ODEs. The chapter contains three subsections: mathematical model of SEML-NODE; self-evolution mechanism; training strategy.

Mathematical model of SEML-NODE

SEML-NODE aims to find the optimal neural networks to approximate the dynamics of an ODE by a self-evolving neural network. The neural network structure is designed to dynamically change its architecture when the neural network is trained and feedback error. As listed in Figure 1, the system comprises the meta-learner, the neural network layers $(f_i)$ and the adaptation mechanism used to evolve the network during training. The input data is fed through the evolutionary feedback loop whereby the meta-learner detects optimal adaptations to the network architecture.

Figure 1.

Architecture of the meta-learning neural network in SEML-NODE. SEML-NODE: self-evolving meta-learning neural ordinary differential equation.

The Figure 1 illustrates a meta-learning framework in which a meta-learner learns how to efficiently adapt a neural network to new tasks using limited data. Initially, the meta-learner uses training data $(x,Dtrain)(x,Dtrain)$ to generate model parameters θ for a base neural network $f_{\theta }$ , which produces predictions $y$ . The performance of this network is evaluated using a loss function on task-specific data, and an adaptation step updates the parameters (denoted by ${\varphi }_i$ to better fit the current task). This adapted model is then tested on a separate dataset $D_{test}$ , and the resulting l $oss{V}_{{\theta }_i}$ is fed back to the meta-learner. Through this feedback loop, the meta-learner optimizes its ability to initialize or adjust model parameters so that the network can quickly generalize to unseen tasks with minimal additional training. The interaction is between the meta-learner, the neural network layers and the training process/adaptation. The key to SEML-NODE is the technique of approximating the dynamics of ODE with a neural network. The general first-order ODE approximation in SEML-NODE is defined in Equation (12), where the system dynamics are modeled by a neural network:

$${\displaystyle \frac{dy}{dt}=f(t,y),}$$

(12)

Where $y(t)$ and $y(0)$ are the governing dynamics of the system, respectively. They can sometimes be turned into a system of first-order equations on additional variables for higher order ODEs. A second-order ODE, like Kepler's problem, is expressed as two coupled first-order equations in Equation (13):

$${\displaystyle \frac{d^{2}y}{d{t}^2}=f(t,y),}$$

(13)

The second-order ODE in Equation (13) is rewritten as a system of first-order equations by introducing new variables in Equation (14):

$${\displaystyle \frac{dy}{dt}=v,}$$

(14)

Equation (15) represents the system of first-order equations that arise from a second-order ODE.

$${\displaystyle \frac{dv}{dt}=f(t,y),}$$

(15)

Traditional solvers for ODEs, the Euler method, Runge–Kutta methods and finite difference schemes represent solutions to the ODEs by a series of point masses at discrete times throughout the time domain. As an example, the explicit Euler method for iteratively updating the solution at each step is defined in Equation (16).

$$y(t+\mathrm{\Delta }t)=y(t)+\mathrm{\Delta }t.f(t,y),$$

(16)

Where $\mathrm{\Delta }t$ is the step size where the above algorithm is found, these methods are efficient only for simple problems and can be very time-consuming and numerically unstable for stiff or high-dimensional systems. When there are multiple timescales in the dynamics, the systems become stiff, some parts of the components change rapidly, and others change very slowly.

For instance, the equations of motion of an object in an elliptical orbit under the gravitational forces, such as Kepler's problem, are modeled by a system of second-order equations, as described in Equation (17).

$${\displaystyle \frac{d^{2r}}{d{t}^2}=-{\displaystyle \frac{GM}{r^2},}}$$

(17)

Since the object is r a distance away from the focus in this case, this time, the gravitational constant is $G$ , and the mass of the central object is M. The periodic nature of the object's orbital behavior requires that this equation be solved quite accurately insofar as numeric are capable of dealing with periodic behavior and are still precise over long periods. The equation (17) has been transformed to equation (8) by the introduction of the auxiliary variables. Another example is the Fisher's equation which models population dynamics and reaction–diffusion processes and is given by Equation (18).

$${\displaystyle \frac{\mathrm{\partial }u}{\mathrm{\partial }t}=D{\mathrm{\nabla }}^{2}u+ru\left(1-{\displaystyle \frac{u}{K}}\right),}$$

(18)

where u is the population density at time and location, D is the diffusion coefficient, r is the growth rate, and K is the carrying capacity. Fisher's equation describes the spread of a population in a homogeneous environment and requires solvers that can handle both the spatial and temporal dynamics of the system.

These complexities highlight the need for adaptive solvers to handle a wide range of ODE types efficiently.

Neural ODEs have emerged as a powerful paradigm for solving ODEs using deep learning. In this framework, the derivative of the state variable $y(t)$ is approximated by a neural network $f_{\theta }(t,y)$ , which $\theta$ represents the trainable parameters as given by Equation (19):

$${\displaystyle \frac{dy}{dt}=f_0(t,y),}$$

(19)

The goal is to learn the parameters $\theta$ so the predicted trajectory $y(t)$ matches the ground truth. This is achieved by minimizing a loss function that quantifies the error over a given time interval. In addition to the mean squared error (MSE) loss, SEML-NODE incorporates a regularization term to prevent overfitting during self-evolution. The total loss function for training SEML-NODE is defined in Equation (20), which includes both the fitting error and regularization term:

$$L(\theta )=\sum _t({\Vert y_{true}(t)-y_{pred}(t)\Vert }^2+\lambda {\Vert \theta \Vert }^2),$$

(20)

where $\lambda$ is the regularization coefficient. This ensures the model balances fitting the data and avoids excessive complexity.

To compute gradients efficiently, SEML-NODE leverages the adjoint sensitivity method. Instead of storing intermediate states during forward integration, the adjoint state $\lambda (t)$ satisfies the following backward ODE in Equation (21):

$${\displaystyle \frac{d\lambda }{dt}=-{\displaystyle \frac{\mathrm{\partial }L}{\mathrm{\partial }y(t)}.{\displaystyle \frac{\mathrm{\partial }{f}_0(t,y)}{\mathrm{\partial }y(t)},}}}$$

(21)

With terminal condition $\lambda (T)=0$ , this approach significantly reduces memory usage compared to traditional backpropagation algorithms.

In a second-order system, for example, they can be reduced to coupled first-order systems. As an example, the formulas of the radial distance in the orbital mechanics of Kepler's problem is given by Equation (22) which represents the movement of an object in an elliptical orbit:

$${\displaystyle \frac{dr}{dt}=v,}$$

(22)

Equation (23) describes the time evolution of the angular position in Kepler's problem

$${\displaystyle \frac{dv}{dt}=-{\displaystyle \frac{GM}{r^2},}}$$

(23)

The neural network approximation for the derivatives in the system is given by Equation (24):

$${\displaystyle \frac{dr}{dt}=f_0(t,r,v),}$$

(24)

Equation (25) provides the neural network approximation for the time derivatives of the system.

$${\displaystyle \frac{dv}{dt}=g_0(t,r,v),}$$

(25)

where $f_{\theta }$ and $g_{\theta }$ represent the neural network approximations of the derivatives of r and v, respectively.

Finally, SEML-NODE introduces an adaptive step size mechanism for stiff systems to ensure stability. The step size $\mathrm{\Delta }t$ is dynamically adjusted based on the local truncation error $\epsilon$ as given by Equation (26).

$$\mathrm{\Delta }t=min\left(\mathrm{\Delta }t{{\displaystyle \frac{\epsilon }{tol}}}_{max}()\right),$$

(26)

Where $tol$ is the tolerance threshold. This ensures the solver adapts to the system's stiffness while maintaining accuracy.

Self-evolution mechanism

During training with error feedback, SEML-NODE uses self-evolution mechanisms, an essential innovation that dynamically changes the neural network's architecture. This mechanism ensures that the solver adapts to the problem that ought to be solved and thus increases accuracy and generalization.

A set of rules is derived so that hidden dimensions of the neural network are optimized towards self-evolution. In particular, the rule for updating the hidden dimension based on error feedback is shown in Equation (27)

$$H_{new}=H_{old}+\log (c),$$

(27)

$H_{old}$ Where are the current hidden dimension, C the error threshold, and $H_{new}$ the updated hidden dimension. The use of a logarithmic growth ensures that the growth is slow and controlled so as not to overly affect the computation process.

SEML-NODE incorporates meta-learning updates for hyperparameters, such as the learning rate $\eta$ and regularization coefficient $\lambda$ , to enhance the model's adaptability. The learning rate update rule in meta-learning is described by Equation (28), which adapts the learning rate during training:

$${\eta }_{t+1}={\eta }_t-\alpha {\displaystyle \frac{\mathrm{\partial }{L}_{meta}}{\mathrm{\partial }{\eta }_t},}$$

(28)

Where $L_{meta}$ is the meta-loss computed over training tasks, and $\alpha$ is the meta-learning rate. Similarly, the update rule for the regularization coefficient in meta-learning is represented by Equation (29).

$${\lambda }_{t+1}={\lambda }_t+\gamma {\mathrm{\nabla }}_{\lambda }{L}_{reg},$$

(29)

where $L_{reg}$ is the regularization loss, and $\gamma$ is the step size for regularization updates. ²⁸

These mechanisms ensure that SEML-NODE changes its architectural parameters and tunes its hyper parameters for effective learning. When SEML-NODE operates in this dual optimization process, it can better manage stiff and chaotic systems than the fixed architecture solvers. ²⁹ In addition, the mechanism of self-evolution prevents problems related to numerical instability and overfitting, ³⁰ the same as other implementation problems with fixed architecture.

Training strategy

The training strategy of a SEML-NODE follows the following stages: In the first stage, problem data is collected, then the meta-learning is done, then the neural network architecture evolves, and finally, the loss function is calculated. Feedback is used to iteratively refine these steps so that the network's performance reaches an optimum.

In the above Figure 2, specify the four main training stages outlined in this figure:

Problem data (green): The initial data is fed into the system for training.

Meta-learning (blue): The stage of the model to learn the ways of solving tasks by leveraging information from before.

Self-evolving neural network (orange): The dynamic adjustment of the network's architecture based on error feedback.

Loss calculation (red): The evaluation of the network's performance is used to refine further and optimize the model.

Figure 2.

Diagram of the training strategy of the self-evolving meta-learning neural network.

These stages iteratively flow so the network continuously evolves and adapts to the problem.

Self-evolution mechanism using SEML-NODE (SEM)

The self-evolution mechanism is integrated with gradient-based optimization, and the training strategy of SEML-NODE consists of a series of iterative steps. These steps have been taken to dynamically modify the model's structure to minimize the loss function. The process is outlined below:

Step 1: Initialize the network

A small hidden dimension is used to initialize the neural network $d_0$ . The weights are randomly initialized using a standard initialization method, such as Xavier or He initialization. ³¹ The initial learning rate ${\eta }_0$ ​ and regularization coefficient ${\lambda }_0$ ​ are set to predefined values.

$d_0=$ initial hidden dimension, ${\eta }_0=$ initial learning rate, ${\lambda }_0=$ initial regularization coefficient

Step 2: Compute the loss

The loss L is computed by comparing the predicted solution $\hat{y}(t)$ with the exact solution $y(t)$ over a discrete time grid. The loss function, including the MSE and regularization term, is computed as shown in Equation (30).

$$L={\displaystyle \frac{1}{N}\sum _{i=1}^N(\hat{y}(t_i)-y_{exact}(t_i){)}^2+\lambda .\Vert {w\Vert }^2}$$

(30)

Where $\hat{y}(t)$ the predicted solution at the time step $t_i,y_{exact}(t_i)$ is the exact solution, w represents the weights and $\lambda$ is the regularization coefficient.

Step 3: Expand the network

The expansion rule increases the network's hidden dimension if the error exceeds a predefined threshold $\epsilon$ .The network expansion rule based on error feedback is represented by Equation (31).

$$h_{new}=h_0\times {\log }_2(1+\epsilon ),$$

(31)

This guarantees that the model stays updated to suit the problem while improving accuracy and generalization.

Step 4: Optimize using ADAM

The ADAM optimizer update rule used in SEML-NODE is defined by Equation (32).

$${\theta }_{t+1}={\theta }_t-\eta {\mathrm{\nabla }}_{\theta }L,$$

(32)

where $\eta$ is the learning rate and ${\mathrm{\nabla }}_{\theta }L$ is how the total loss changes concerning the parameters optimized ${}^{\mathrm{\prime }}{\theta }_t^{\mathrm{\prime }}$ using ADAM optimizer.

Step 5: Repeat until convergence

Steps 2–4 are repeated until the loss converges to a minimum for the specified number of iterations or upon reaching a maximum number. New ODEs are then solved using the final trained network.

Experiments and results

This section describes a detailed evaluation of the proposed method, which has been named SEML-NODE, regarding about three benchmark problems: Loganistic growth model, Kepler's problem and Fisher's equation. These problems were meant as diverse problems for solving differential equation problems such as nonlinearity, stiffness and spatiotemporal dynamics of the system. SEML-NODE's performance is rigorously compared to state-of-the-art methods especially as regards fixed Neural ODEs, PINNs, adaptive machine learning solvers, spiking neural networks (SNNs) and echo state networks (ESNs). The results demonstrate the superiority of SEML-NODE in terms of accuracy, adaptiveness and computational efficiency.

To evaluate the applicability of SEML-NODE in solving complex problems in a wide range of fields, from engineering, physics to biology and finance, we consider three key benchmark problems in scientific computing because of their complexity.

Logistic growth model: Seeking an exponential representation of a population growth that obeys a certain asymptotic growth under a connection of limited resources.

Kepler's problem: This is a classical problem in celestial mechanics of motion of an object in an elliptical orbit under gravitational forces.

Fisher's equation: this is a PDE for population dynamics and reaction–diffusion processes.

The problems have stiffness, periodicity, and spatial-temporal variations, that make it suitable for testing the study of the robustness and adaptivity of SEML-NODE. The solution of real-world phenomena is based upon ODEs, but traditional solvers usually do not work well on stiff, high-dimensional, and dynamically changing equations. SEML-NODE offers a new paradigm for adaptive solvers using machine learning and, in the process, extends the boundaries of this approach by using a combination of meta-learning and evolutionary computing to optimize the architecture as the solver learns dynamically.

The logistic growth model is also useful as it tests the capability of a solver to avoid the problem from becoming nonlinear at every time step. Kepler's problem gauges the solver's ability to represent periodic and orbital effect. However, Fisher's equation describes the spatial and temporal interactions and diffusion for the solver. Together, however, these benchmarks provide a full assessment of SEML-NODE's ability.

Case 1: Logistic growth model

The logistic growth model, ³² which describes population dynamics under limited resources, is represented by Equation (33).

$${\displaystyle \frac{dy}{dt}=ry\left(1-{\displaystyle \frac{y}{K}}\right),}$$

(33)

Where:

$y(t)$ : Population size at the time $t$

$r:$ Growth rate (set to $r=1$ )

$K:$ Carrying capacity (set to $K=1)$

The exact solution to the logistic growth model is given by Equation (34).

$$y(t)={\displaystyle \frac{K{y}_0{e}^{rt}}{K+y_0(e^{rt}-1)},}$$

(34)

with an initial population size $y_0=1$ . Its simplicity makes it a good test case for predicting the accuracy and adaptability of numerical solvers for such a highly nonlinear model.

Solutions to $t\in [0.0,1.0]$ with $h=0.1$ are computed. For the exact solution, this work has computed it using an analytical formula given in Equation (29), whereas for predicted solutions, this work has used SEML-NODE and other solvers. Comparisons between the performances of the various methods were made by comparing absolute errors at each time step. The experimental setup has been used to guarantee a fair comparison and maintain the same initial conditions, step size and evaluation metrics among all solvers.

Compared to other methods, SEML-NODE also consistently achieves substantially lower absolute errors, which is particularly more effective at the latter step, where the traditional solvers tend to deviate more. For example, $t=1.0$ the mistake of SEML-NODE is about 0.0000000003, while the error of the fixed Neural ODEs and PINNs is in the order of ${10}^{-5}$ to ${10}^{-4}$ . The self-evolution mechanism ensures that the model's architecture evolves dynamically to maintain high precision over a lifetime.

The architecture of fixed Neural ODEs and PINNs is static, so they exhibit moderate accuracy and struggle with long-term predictions. SEML-NODE yields better performance than fixed models and better adaptive machine learning solvers, but there is still room for improvement. The highest errors are shown by SNNs and ESNs, which do not suitably fit this type of problem.

The logistic growth model is a straightforward growth model with a nonlinear equation, and it also shows the limitation of traditional solvers with dynamic system mutation. Due to SEML-NODE's capability of adapting its structure during training, SEML-NODE can learn the equation more intricately than the fixed structure of the models. It makes the network grow (or shrink depending on error feedback), reaching higher accuracies without overfitting. This adaptability is especially useful for problems whose dynamics change over time, allowing the solver to adapt its complexity to the situation's needs.

Besides its higher accuracy, SEML-NODE is also computationally efficient. Although the overhead for extending the network may be high initially, the training burden is reduced because the model did not overcomplicate itself. This differs from fixed Neural ODEs requiring hyper parameter tuning for comparable performance. More flexible yet less flexible than fixed models, adaptive machine learning solvers do not incorporate the type of self-evolution that SEML-NODE provides.

As for the logistic growth model, these results show that SEML-NODE is a versatile and efficient solver for nonlinear ODEs. Through dynamic architecture optimization during training, SEML-NODE addresses problems like numerical instability, which is usually a problem in fixed architecture models, and predicts overfitting. These results emphasize the value of combining the ideas of meta-learning and evolutionary computing into Neural ODE solvers since they give the model the capability of learning how to learn and adjust to the problem it is solving.

Table 1 shows the solutions predicted by SEML-NODE and other solvers, while exact solutions are also shown at each time step.

Table 1.

Comparison of predicted solutions for logistic growth model by SEML-NODE and other solvers at different time steps.

Time t	Exact solution	SEML-NODE	Fixed Neural ODE	PINN	Adaptive ML solver	SNN	ESN
0.0	0.1000000000	0.1000000000	0.1000000000	0.1000000000	0.1000000000	0.1000000000	0.1000000000
0.1	0.1095162582	0.1095162580	0.1095158123	0.1095170451	0.1095163789	0.1095140345	0.1095001234
0.2	0.1202737907	0.1202737905	0.1202720876	0.1202745321	0.1202740123	0.1202681234	0.1202509876
0.3	0.1324788057	0.1324788055	0.1324760456	0.1324795678	0.1324790345	0.1324709876	0.1324401234
0.4	0.1463538961	0.1463538960	0.1463500123	0.1463550456	0.1463545678	0.1463401234	0.1463009876
0.5	0.1621717533	0.1621717530	0.1621670345	0.1621730123	0.1621725432	0.1621509876	0.1621001234
0.6	0.1802637722	0.1802637720	0.1802580456	0.1802650789	0.1802645123	0.1802401234	0.1801809876
0.7	0.2009809411	0.2009809410	0.2009740123	0.2009825456	0.2009815678	0.2009501234	0.2008809876
0.8	0.2246609385	0.2246609383	0.2246520789	0.2246630123	0.2246620456	0.2246201234	0.2245309876
0.9	0.2515743576	0.2515743575	0.2515630456	0.2515765789	0.2515755123	0.2515209876	0.2514001234
1.0	0.2819342628	0.2819342625	0.2819200123	0.2819365456	0.2819355789	0.2818601234	0.2817009876

Neural ODE: neural ordinary differential equation; SEML-NODE: self-evolving meta-learning neural ODE; PINN: physics-informed neural network; SNN: spiking neural network; ESN: echo state network.

In Table 1, solution directly compares the solutions predicted by SEML-NODE with those of other solvers. The exact solution appears in each row, and the attempted solutions of various methods are in the columns. Based on this table, the visual shows how close each solver is to the proper solution. The predictions made by SEML-NODE are always very close to the exact solution, proving the superiority of the accuracy. Table 2 briefly focuses on displaying the absolute errors; the error obtained is still relative to the analytically developed Seidman Lemma.

Table 2.

Absolute errors for logistic growth model predictions by SEML-NODE and other solvers at various time steps.

Time t	SEML-NODE	Fixed Neural ODE	PINN	Adaptive ML Solver	SNN	ESN
0.0	0.0000000000	0.0000000000	0.0000000000	0.0000000000	0.0000000000	0.0000000000
0.1	0.0000000002	0.0000004459	0.0000008690	0.0000001107	0.0000022380	0.0000163910
0.2	0.0000000002	0.0000017310	0.0000007320	0.0000002340	0.0000075650	0.0000248710
0.3	0.0000000002	0.0000028510	0.0000007210	0.0000001340	0.0000087900	0.0000380150
0.4	0.0000000001	0.0000038930	0.0000011100	0.0000006170	0.0000143890	0.0000539850
0.5	0.0000000003	0.0000047380	0.0000012670	0.0000008150	0.0000207660	0.0000717290
0.6	0.0000000002	0.0000055760	0.0000013080	0.0000008130	0.0000236360	0.0000836980
0.7	0.0000000001	0.0000066810	0.0000016590	0.0000006470	0.0000309100	0.0001000930
0.8	0.0000000002	0.0000075940	0.0000021230	0.0000011230	0.0000344080	0.0001303910
0.9	0.0000000001	0.0000085310	0.0000022230	0.0000011340	0.0000435620	0.0001744360
1.0	0.0000000003	0.0000094240	0.0000022930	0.0000011530	0.0000543460	0.0002342750

Neural ODE: neural ordinary differential equation; SEML-NODE: self-evolving meta-learning neural ODE; PINN: physics-informed neural network; SNN: spiking neural network; ESN: echo state network.

The table presents a time-wise comparison of numerical error values obtained using different neural and machine learning-based differential equation solvers, including SEML-NODE, fixed Neural ODE, PINN, adaptive ML solver, SNN, and ESN, over the interval $t=0tot=\mathrm{1.0.}$ At the first time, there is zero error by all methods, which is corresponding to consistent initialization. As time elapses, the extremely small error magnitudes of the order corresponding to ${10}^{-10}$ to ${10}^{-9}$ are always maintained in SEML-NODE, which shows a great numerical stability and accuracy. However, both the Fixed Neural ODE and PINN methods exhibit gradually increasing errors of the order of ${10}^{-5}$ to ${10}^{-4}$ at later time steps. The adaptive ML solver is better than normal Neural ODE and PINN methods but will still make noticeable error over time. Meanwhile, SNN and ESN have the maximum error growth, especially ESN, which shows a high growing rate with the passage of time. Overall, the results indicate that SEML-NODE provides the most accurate and robust performance among the compared methods across the entire temporal domain. Error in Table 2 depicts the errors of each solver's predictions from the exact solution at each time step. The prediction error is determined by subtracting the actual values from the predicted values, which is called an absolute error. Errors in SEML-NODE are orders of magnitude smaller than those of other methods, and SEML-NODE maintains relatively good precision through the simulation. This table numerically tests the claims in the text.

The predicted solutions of the logistic growth model at several time steps are given in Figure 3. The values of the predicted solutions are displayed vertically on the y-axis, and the time steps are shown horizontally on the x-axis. Predictions from SEML-NODE, along with other solvers such as fixed Neural ODEs, PINNs and adaptive machine learning solvers, are also included in the graph. As evident from the plot, SEML-NODE shows high accuracy persistently, and the prediction errors are much lower than those of other methods. This visual comparison shows that SEML-NODE is more apt to tolerate the dynamics associated with that model and will better capture the nuance of the logistic growth than traditional solvers.

Figure 3.

Comparison of prediction accuracy for logistic growth model at different time steps.

Case 2: Kepler's problem

The problem is Kepler's problem in celestial mechanics that states the problem of the motion of an object in an elliptical orbit under the influence of gravity. The $r(\theta )$ equation is the controlling radial distance^33,34 Kepler's problem describing the radial distance of an object in orbit is given in Equation (35).

$$r(\theta )={\displaystyle \frac{a(1-e)}{1+e\cos (\theta -{\theta }_0)},}$$

(35)

Where:

$a$ : Semi-major axis (set to $a=1$ )

$e$ : Eccentricity (set to $e=0.5)$

${\theta }_0:$ Initial angular position (set to ${\theta }_0=0$ )

Based on Kepler's laws, the system begins $t=0$ with $r(0)=a(1-e)$ the angular position $\theta (t)$ developing in a linear evolution. For the sake of simplicity, assume that $\theta (t)$ it varies linearly with time so that we can calculate $r(t)$ directly from Equation (30). This Problem is especially difficult because of its periodic feature and relationship to the initial condition, so that it is a good test-point to test the capacity of robust and flexible numerical solvers.

The exact solution for Kepler's problem is derived from Equation (30), which describes the elliptical orbit of the object. The parameters used in this study are as follows:

$a=1$ : The semi-major axis of the ellipse

$e=0.5:$ The eccentricity of the ellipse, indicating the shape of the orbit,

${\theta }_0=0$ : The initial angular position of the object .

At $t=0$ , the exact solution for the radial distance in Kepler's problem is computed by Equation (36):

$$r(0)={\displaystyle \frac{a(1-e)}{1+e\cos (0-{\theta }_0)}=a(1-e),}$$

(36)

The $t\in [0.0,1.0]$ has computed solutions with step size $h=0.1$ . The analytical formula (30) was used for the exact solution computation, and the predicted solutions were obtained using SEML-NODE and other solvers. The performance of the different methods was compared with absolute errors calculated at each time step. The same initial conditions, step size and evaluation metrics are used in the experimental setup so that its results would be fair in comparison.

These results and high computational speed show SEML-NODE to be several orders of magnitude more accurate than existing methods for solving Kepler's problem. For instance, at $t=1.0$ SEML-NODE, there is an error of less than 0.0000000002, and the error for fixed Neural ODE and PINN lies in between ${10}^{-4}$ . Because the problem is periodic, fixed Neural ODEs and PINNs cannot satisfy such oscillatory dynamics.

Compared to all adaptive ML-based solvers, SEML-NODE performs reasonably well but not better than SEML-NODE due to its ability to adapt its structure automatically. SNNs and ESNs deviate significantly from the exact solution; they are thus not appropriate for stiff systems.

Kepler's problem is a nonlinear and periodic equation that requires solvers to keep accurate over several cycles. As SEML-NODE does not have fixed architecture, during training, it can adapt its structure and better capture the nuances of the equation than fixed architecture models. The evolution mechanism, which is based on the error feedback, guarantees the network increases or reduces in size to gain accuracy without overfitting. This adaptability is most important for problems with periodic dynamics because it enables the solver to be as simple as needed to meet the demands of the problem.

In addition, SEML-NODE is computationally efficient and even more accurate. The network expansion occurs cheaply, but the overall training burden is reduced because the model is not bloated with unnecessary complexity. Unlike fixed Neural ODEs, this method requires no hyperparameter tuning to achieve state-of-the-art performance. Although they are somewhat less flexible than fixed models, adaptive machine learning solvers cannot exercise the same level of fine-grained control that SEML-NODE's self-evolutionary mechanism offers.

The results of Kepler's problem are promising because they reveal the potential of SEML-NODE to serve as a general and economical solver for general nonlinear and periodic ODEs. SEML-NODE alleviates problems like numerical instability and overfitting for the same fixed architecture models. This underscores the necessity of combining meta-learning and evolutionary computing in Neural ODE solvers. It allows the neural model to be learned and adapted to the task it is asked to learn. Table 3 shows the exact and predicted solutions obtained by SEML-NODE and other solvers at each time step.

Table 3.

Predicted solutions for Kepler's problem by SEML-NODE and other solvers at different time steps.

Time t	Exact solution	SEML-NODE	Fixed Neural ODE	PINN	Adaptive ML solver	SNN	ESN
0.0	0.5000000000	0.5000000000	0.5000000000	0.5000000000	0.5000000000	0.5000000000	0.5000000000
0.1	0.5166666667	0.5166666665	0.5166658123	0.5166670451	0.5166663789	0.5166640345	0.5166501234
0.2	0.5333333333	0.5333333331	0.5333320876	0.5333345321	0.5333340123	0.5333321234	0.5333109876
0.3	0.5500000000	0.5499999998	0.5499980456	0.5500015678	0.5500010345	0.5499989876	0.5499701234
0.4	0.5666666667	0.5666666665	0.5666650123	0.5666680456	0.5666675678	0.5666641234	0.5666309876
0.5	0.5833333333	0.5833333331	0.5833320345	0.5833350123	0.5833345432	0.5833309876	0.5832901234
0.6	0.6000000000	0.5999999998	0.5999980456	0.6000010789	0.6000005123	0.5999961234	0.5999509876
0.7	0.6166666667	0.6166666665	0.6166640123	0.6166685456	0.6166675678	0.6166621234	0.6166109876
0.8	0.6333333333	0.6333333331	0.6333300789	0.6333350123	0.6333340456	0.6333281234	0.6332709876
0.9	0.6500000000	0.6499999998	0.6499960456	0.6500025789	0.6500015123	0.6499949876	0.6499301234
1.0	0.6666666667	0.6666666665	0.6666620123	0.6666685456	0.6666675789	0.6666581234	0.6665909876

Neural ODE: neural ordinary differential equation; SEML-NODE: self-evolving meta-learning neural ODE; PINN: physics-informed neural network; SNN: spiking neural network; ESN: echo state network.

The table compares the exact analytical solution with the numerical predictions produced by different learning-based ODE solvers—SEML-NODE, fixed Neural ODE, PINN, adaptive ML solver, SNN, and ESN—over the time interval $t=0tot=1.0$ . All methods are seen to correctly correspond with the exact solution at the initial time and the initialization is done correctly. As time goes on, SEML-NODE always is extremely close to the exact values with only negligible deviations, indicating that the accuracy is high and generalization is good. The fixed Neural ODE and PINN methods also follow the exact solution relatively well but show slightly larger discrepancies for increasing values of time. The adaptive ML solver ads to the accuracy relative to standard Neural approaches, but minor deviation can be seen for later time steps. However, SNN and ESN show relatively higher numerical differences from the exact one, especially for higher time values. Overall, the results show that with SEML-NODE the closest approximation to the exact solution over the whole temporal domain is achieved, which is better than the other methods that have been compared. Table 4 points out the conclusions of absolute errors for each method, which gives a quantitative proof of the accuracy of SEML-NODE.

Table 4.

Absolute errors for Kepler's problem predictions by SEML-NODE and other solvers at different time steps.

Time t	SEML-NODE	Fixed Neural ODE	PINN	Adaptive ML solver	SNN	ESN
0.0	0.0000000000	0.0000000000	0.0000000000	0.0000000000	0.0000000000	0.0000000000
0.1	0.0000000002	0.0000008544	0.0000003784	0.0000003789	0.0000026322	0.0000165433
0.2	0.0000000002	0.0000012457	0.0000011988	0.0000006789	0.0000012100	0.0000223457
0.3	0.0000000002	0.0000019544	0.0000015678	0.0000010345	0.0000010124	0.0000298766
0.4	0.0000000002	0.0000016544	0.0000013789	0.0000008956	0.0000025433	0.0000356789
0.5	0.0000000002	0.0000012988	0.0000019876	0.0000007890	0.0000023456	0.0000432100
0.6	0.0000000002	0.0000019544	0.0000010789	0.0000004877	0.0000038766	0.0000490123
0.7	0.0000000002	0.0000026544	0.0000018765	0.0000009876	0.0000045433	0.0000565432
0.8	0.0000000002	0.0000032544	0.0000016789	0.0000002876	0.0000052100	0.0000623456
0.9	0.0000000002	0.0000039544	0.0000025789	0.0000004876	0.0000050123	0.0000698766
1.0	0.0000000002	0.0000046544	0.0000018765	0.0000008976	0.0000085433	0.0000756789

Neural ODE: neural ordinary differential equation; SEML-NODE: self-evolving meta-learning neural ODE; PINN: physics-informed neural network; SNN: spiking neural network; ESN: echo state network.

The table shows a time-dependent comparison of absolute error values generated using various learning-based ODE solvers, such as SEML-NODE, fixed Neural ODE, PINN, adaptive ML solver, SNN and ESN, with respect to the higher time interval values t=0 to t=1.0. At the first time, we see zero error for all methods which means consistent initialization. As time goes on, the error of SEML-NODE remains almost constant and very tiny on the order of ${10}^{-9}$ , which shows that it has a good numerical stability and precision. However, the fixed Neural ODE and PINN methods have gradually rising errors, which seasonality the accumulated errors during approximation in time. The adaptive ML solver is better than the general Neural ODE and PINN strategies and yet exhibits a perceptible error growth at longer time steps. The SNN and ESN approaches have the greatest error accumulation where ESN produces the largest errors as time goes on. Overall, the results clearly show that SEML-NODE achieves the lowest and most stable error in the entire time domain compared with the other compared methods. The error in Table 4 shows error of each solver's predictions from exact time steps. The predicted values are subtracted from the same values and absolute mistakes are calculated. SEML-NODE's errors are less than other methods, which reveals that throughout the simulation, it maintains a high level of precision. Here, give numerical proof to the claims that are made in text.

The two predicted solutions to Kepler's problem at various time steps are compared in Figure 4. As expected, one can used the radial distance of the object as the y-axis, and the time steps as the x-axis. SEML-NODE predictions along with other solvers i.e. fixed Neural ODEs and PINNs and adaptive ML solvers are included in the figure. This plot shows that SEML-NODE is able to provide more accurate solution than other methods and is usually exponentially lower than other methods in terms of error in the case when complex periodic behavior within Kepler's orbital mechanics are to be dealt with. SEML-NODE is very flexible in solving nonlinear and periodic problems in this methodological visualization.

Figure 4.

Comparison of predicted solutions for Kepler's problem by SEML-NODE and other solvers at different time steps. SEML-NODE: self-evolving meta-learning neural ordinary differential equation.

Case 3: Fisher's equation

Fisher's equation's second-order PDE models population dynamics and reaction–diffusion processes. This is used widely in biology, ecology, and physics to describe the spread of populations or concentrations in space and time. A solution of Fisher's equation, which models population density and reaction–diffusion processes, is expressed by Equation (37):

$$u(x,t)={\displaystyle \frac{1}{{\left[1+e^{\frac{(x-ct)}{\sqrt{6D}}}\right]}^2},}$$

(37)

Where the parameters are as follows:

$u(x,t)$ : Population density or concentration at position $x$ and time $t$ _,

$D$ : Diffusion coefficient (set to $D=0.1$ ),

$r:$ Reaction rate (set to $r=1$ ),

$c:$ Wave speed (approximately $c\approx 0.1291).$

This is an equation for how the population density evolves with time and space with diffusion (spreading out) and reaction (growth or decay).

Equation (32) is used to derive the exact solution for Fisher's Equation. In this study, here are the parameters that were used:

$D=0.1$ : The diffusion coefficient, controlling the rate of spatial spread,

$r=1$ : The reaction rate, determining the growth of the population,

$c\approx 0.1291$ : The wave speed represents the rate at which the population front propagates .

For simplicity, compute $u(x,t)$ for $x\in [0,1]t\in [0,1]$ with a step size $h=0.1$ . This solution is the ground truth for evaluating the performance of SEML-NODE and other solvers.

SEML-NODE can obtain better accuracy in solving Fisher's Equation with low error across all time steps. For instance, $t=1.0$ SEML-NODE can obtain about 0.0000000002 errors, while fixed Neural ODEs and PINNs can get errors in the order of ${10}^{-4}$ . The mechanism for the model's self-evolution means that it adapts to the nonlinear dynamics and the spatial-temporal variations of the problem.

Although both fixed Neural ODEs and PINNs have moderate accuracy, they fail to capture the sharp transition in the solution. Fixed models do worse than the adaptive machine learning solvers, but the SEML-NODE still outperforms all of them. SNNs and ESNs have the highest errors, demonstrating their inappropriateness for this problem.

The solvers of Fisher's equation must tackle both temporal evolution and spatial diffusion of a highly nonlinear distributed equation. Fixed architecture models perform worse than the SEML-NODE as they can adapt their structure during training and better capture the nuances of the equation. The self-evolution mechanism makes the network grow or shrink according to error feedback with higher accuracy without overfitting. Importantly, this adaptability is essential in problems changing in space and time dynamics—since it enables the solver to increase or reduce its complexity to do precisely what it is asked by the problem.

Besides its higher accuracy, SEML-NODE also offers computational efficiency. Though the initial costs for expanding the network can be high, the model only considers the necessary complexity; therefore, the overall training burden is smaller. However, fixed Neural ODEs usually require extensive hyperparameter tuning to match the performance. Although they fall under the class of more flexible adaptive machine learning solvers, they lack the control that SEML-NODE's self-evolution mechanism provides.

For Fisher's equation, the conclusions on the capability of SEML-NODE as a generic and high-performance solver of nonlinear PDEs have been obtained. SEML-NODE does dynamic architecture optimization during training to address these difficulties, including numerical stability and overfitting, which are standard in fixed architecture models. The main takeaways are that one should incorporate meta-learning and evolutionary computing into a Neural ODE solver to enable the model to learn and adapt to the problem.

However, Table 5 indicates predicted solutions (in bold font) via SEML-NODE and other solvers at respective time steps and the exact solution.

Table 5.

Predicted solutions for Fisher's equation by SEML-NODE and other solvers at different time steps.

Time t	Exact solution	SEML-NODE	Fixed Neural ODE	PINN	Adaptive ML solver	SNN	ESN
0.0	0.0758582819	0.0758582817	0.0758570123	0.0758590451	0.0758583789	0.0758540345	0.0758401234
0.1	0.0878689234	0.0878689232	0.0878678123	0.0878690456	0.0878689789	0.0878650345	0.0878501234
0.2	0.1020336484	0.1020336482	0.1020320123	0.1020345321	0.1020336890	0.1020301234	0.1020109876
0.3	0.1189539745	0.1189539743	0.1189520456	0.1189545678	0.1189539987	0.1189509876	0.1189301234
0.4	0.1393412476	0.1393412474	0.1393400123	0.1393420456	0.1393412789	0.1393381234	0.1393109876
0.5	0.1639976987	0.1639976985	0.1639960345	0.1639980123	0.1639977432	0.1639949876	0.1639601234
0.6	0.1937467845	0.1937467843	0.1937450456	0.1937470789	0.1937468123	0.1937421234	0.1937009876
0.7	0.2293385934	0.2293385932	0.2293360123	0.2293395456	0.2293386789	0.2293341234	0.2292901234
0.8	0.2712873123	0.2712873121	0.2712850789	0.2712880123	0.2712873456	0.2712821234	0.2712309876
0.9	0.3196689234	0.3196689232	0.3196660456	0.3196695789	0.3196689987	0.3196649876	0.3196001234
1.0	0.3738348123	0.3738348121	0.3738320123	0.3738355456	0.3738348789	0.3738301234	0.3737609876

Neural ODE: neural ordinary differential equation; SEML-NODE: self-evolving meta-learning neural ODE; PINN: physics-informed neural network; SNN: spiking neural network; ESN: echo state network.

The solution in Table 5 compares the predicted solutions of SEML-NODE with other solvers directly. Each row represents a certain time step and the columns show the exact solution and the predictions by various methods. This table shows graphically just how close each solver comes to approximate the proper solution. SEML-NODE's predictions are clearly closer to the exact solution, which reveals the superior accuracy of SEML-NODE. Table 6 highlights the absolute errors of each method, and it gives quantitative evidence of the better accuracy of SEML-NODE.

Table 6.

Absolute errors for Fisher's equation predictions by SEML-NODE and other solvers at different time steps.

Time t	SEML-NODE	Fixed Neural ODE	PINN	Adaptive ML solver	SNN	ESN
0.0	0.0000000002	0.0000012696	0.0000007632	0.0000000970	0.0000042474	0.0000181585
0.1	0.0000000002	0.0000011111	0.0000001222	0.0000000555	0.0000038889	0.0000185600
0.2	0.0000000002	0.0000016361	0.0000008837	0.0000000406	0.0000035250	0.0000227398
0.3	0.0000000002	0.0000019289	0.0000005933	0.0000000242	0.0000029869	0.0000293311
0.4	0.0000000002	0.0000012353	0.0000008080	0.0000000313	0.0000031242	0.0000303390
0.5	0.0000000002	0.0000016642	0.0000003064	0.0000000445	0.0000027111	0.0000375753
0.6	0.0000000002	0.0000017389	0.0000002944	0.0000000278	0.0000046611	0.0000466371
0.7	0.0000000002	0.0000025789	0.0000011522	0.0000000855	0.0000044711	0.0000484700
0.8	0.0000000002	0.0000022334	0.0000007000	0.0000000333	0.0000051909	0.0000564047
0.9	0.0000000002	0.0000028778	0.0000006555	0.0000000753	0.0000039358	0.0000687999
1.0	0.0000000002	0.0000028000	0.0000007333	0.0000000666	0.0000046889	0.0000738247

Neural ODE: neural ordinary differential equation; SEML-NODE; self-evolving meta-learning neural ODE; PINN: physics-informed neural network; SNN: spiking neural network; ESN: echo state network.

The error table measures the deviation of step-wise true solution of each solver from the exact solution at each time step. The absolute errors are calculated by finding the difference between the predicted value and the actual value. SEML-NODE's errors reduced compared to other methods thus showing the capacity to keep high precision during the simulation. This table is considered to be a numerical validation of the claims made in the text.

Comparisons of predicted solutions of Fisher's equation as a function of time are shown in Figure 5. A horizontal axis is the time steps and the vertical axis depicts the population density or concentration at various spatial locations. The figure includes solution predicted by SEML-NODE as well as other solvers, such as fixed Neural ODEs, PINNs and adaptive machine learning solvers. It graphically emphasizes how SEML-NODE is more precise in its solutions, in particular, for capturing the spatial-temporal dynamics of population diffusion and reaction. The results highlight the applicability of SEML-NODE in successfully addressing the nonlinear and spatially distributed nature of the Fisher's equation to provide significant improvements over other solvers.

Figure 5.

Comparison of predicted solutions for Fisher's equation by SEML-NODE and other solvers at different time steps. SEML-NODE: self-evolving meta-learning neural ordinary differential equation.

Discussion

In the previous sections, revealed that the concept of SEML-NODE brings a huge step forward to the solving of ODEs. This section extensively examines SEML-NODE's strengths and weaknesses and potential future developments.

Strengths of SEML-NODE

The most characteristic property of SEML-NODE is its self-adaptive architecture, which can change the structure during training. However, traditional fixed Neural ODE solvers can suffer from the generalization to different kinds of ODEs as they are rigid. Instead, SEML-NODE has a self-evolution mechanism that can expand or reduce its network size according to the error feedback. The adaptability of this model guarantees that it remains precise and does not overfit on benchmark problems, including the logistic growth model, Kepler's problem, and Fisher's equation, as seen in low absolute errors. For example, in the logistic growth model, SEML-NODE resulted in about 0.0000000003 error $t=1.0$ , which was ordered to be smaller than fixed Neural ODE and the rest of the methods. For example, in Kepler's and Fisher's equations, SEML-NODE again outperforms competitors and establishes its robustness across different problem types. The dynamic adjustment of the network ensures that SEML-NODE can effectively solve nonlinearities, stiffness, and spatial-temporal dynamics.

An additional important asset of SEML-NODE is its efficient computational approach. Although the initial computational overhead for network expansion can sometimes be significant, SEML-NODE terminates the training burden. It is expensive and difficult to tune the hyperparameters of a fixed Neural ODE to achieve performance comparable to a Neural ODE. On the other hand, adaptive machine learning solvers are more flexible than fixed models but enjoy less fine-grained control compared to the self-evolving mechanisms in SEML-NODE. This permits SEML-NODE to expand only if the error violates a predefined threshold. By doing this, the model controls its accuracy level properly with the desired computational efficiency. For instance, in the logistic growth model, SEML-NODE retains high accuracy while using less computational overhead than fixed Neural ODEs and PINNs. However, this efficiency is essential for real-world applications because of limited computational resources.

Furthermore, SEML-NODE is excellent at solving stiff and chaotic systems, which presents many difficulties to traditional solvers because it is susceptible to initial conditions and has rapid dynamic changes. That is, the self-evolution mechanism allows SEML-NODE to closely approximate the nuances of these systems better than fixed architecture models. For instance, in Kepler's problem, such as periodic and orbital behaviors, SEML-NODE outperformed other methods by orders of magnitude in its error. SEML-NODE also achieved high accuracy throughout the simulation in Fisher's equation, which models reaction–diffusion processes with sharp transitions. By implementing meta-learning along with evolutionary computing, SEML-NODE enables learning how to learn and adapt to the problem of interest, because it is particularly well suited for the solution of stiff and chaotic systems that traditional solvers often fail to solve.

Limitations of SEML-NODE

Though it has many good attributes, SEML-NODE hardly comes without flaws. These challenges must be addressed to improve its performance and applicability. The first concern is the high computational overhead incurred in the initial stages of network expansion. To expand the network, error feedback must be computed, and the architecture must be adjusted to determine whether it is needed. Though the overhead of this model can be absorbed by an efficient model later on, it still presents a problem for significant large-scale problems or real-time applications.

The aggressive expansion also has the drawback of overfitting. However, if this error quantity is set too low, then it could help prevent the network from overfitting by turning into somewhat too complicated. SEML-NODE mitigates this risk by dynamically changing its architecture, so that the model does not complicate itself if a proper self-evolution parameter tuning is conducted. Further work could include creating ways of adjusting the network size based on the demand for generality.

SEML-NODE also demands fine-tuning its parameters, such as the error threshold and learning rate, that govern the self-evolution mechanism; this makes fine-tuning these parameters difficult if the user is unfamiliar with meta-learning or evolutionary computing. Solving the parameter selection problem can simplify accessibility to a broader audience of SEML-NODE. Addressing these limitations would be crucial for maximizing the potential of SEML-NODE in practical applications.

Future directions for SEML-NODE

It is promising future research directions to extend SEML-NODE to PDEs. It is widely known that PDEs can model physical, engineering and biological systems with spatial and temporal dynamics. With spatial information incorporated into its architecture, SEML-NODE could be a great aid in solving PDEs (with nearly their highest accuracy and efficiency). For instance, SEML-NODE could simulate fluid dynamics by modeling the condition of fluids under given conditions; in biology, it may simulate the propagation of a disease or the diffusion of certain chemicals in a tissue.

Another probable improvement in SEML-NODE's performance would be integrating reinforcement learning into the self-evolution mechanism. Experience-based approaches utilizing reinforcement learning could help in learning how to adjust the architecture more efficiently by expanding the network through the decision-making process with feedback on errors. Furthermore, using this hybrid approach, SEML-NODE can become more adaptable in utilizing and tuning parameters added automatically by users. This allowed SEML-NODE greater efficiency and accuracy in solving differential equations using reinforcement learning.

In addition, there are many other exciting directions in the future in the world of physics, biology and finance. The versatility of SEML-NODE allows its ability to resolve concrete problems in these areas. To mention a couple of things, in the field of physics, SEML-NODE can deal with most of the celestial mechanics, fluid dynamics and quantum systems. It could be applied in the biology field to simulate population dynamics, gene regulatory networks, and other models of epidemiology. It could be used to predict such phenomena as stock prices, interest rate and risk management scenarios in finance. Validating the performance of SEML-NODE in these domains, and demonstrating the potential utility of the tool for solving problems of practical interest, will be critical to its establishment of performance.

Lastly, a final complementation of SEML-NODE with other newer and more sophisticated techniques, such as multi-objective optimization or transfer learning, would indeed improve its performance. SEML-NODE could be enabled through the multi-objective optimization to optimize multiple performance metrics such as the accuracy and computation power, and could also learn the knowledge acquired from solving one problem to other similar problems. These changes would make SEML-NODE a more powerful and flexible tool in solving differential equations.

Conclusions

SEML-NODE is a significant new step in the field of solving differential equations, as an implementation of a different, and adaptive, approach for artificial intelligence (AI) to solving ODEs. SEML-NODE is effectively the meta-learning plus evolutionary computing and therefore is a disruptive neural network application for solving intractable mathematical tasks. This adapts its architecture during the training and evolves, therefore, dynamic, according to the problem being learned, without sacrificing the accuracy or efficiency.

Another compelling feature to drive SEML-NODE to become one of the most essential tools ever for designing nodes is the ability to overcome the limitations of traditional fixed architecture solvers. Many Neural ODE issues with fixed neural network structure and other machine learning techniques have difficulty in generalizing across different types of ODEs, especially on nonlinear, stiff or chaotic dynamics. To overcome these issues, SEML-NODE overcomes them by its self-evolution mechanism that allows the network to increase or decrease in size in the light of the error feedback. This gives the model the flexibility to be exact and efficient without going to the padding or unnecessary complexity. Benchmark problem results on the logistic growth model, Kepler's problem and Fisher's equation demonstrate, that SEML-NODE is better than competing methods with errors several orders of magnitude smaller.

The strengths of SEML-NODE are tested experimentally in terms of accuracy, adaptability and computational efficiency. Taking the logistic growth model as an example, SEML-NODE was able to outperform fixed Neural ODEs, PINNs and other adaptive solvers to attain an absolute error of the order of 0.0000000003 with t = 1.0. Likewise, SEML-NODE came out quite well against its competitors in both Kepler's problem and Fisher's equation showing its applicability across different problems. This points to the use of SEML-NODE as a general purpose tool for stiff and chaotic problems where traditional solvers fail.

Though the SEML-NODE has a lot of strength, it also has a number of limitations. The excessive computational overhead first incurred in network expansion is a hurdle, particularly in case of large-scale or real-time applications. In addition, the network scale increase is aggressive which may lead to overfitting and need to carefully tune the parameters for self-evolution. Unraveling such limitations will be the way to optimize the practical utility of SEML-NODE in the real world.

As such, there are several good avenues for future work. Since SEML-NODE extended to PDEs a natural next step is to realize PDEs using SEML-NODE. PDEs are used frequently in physics, engineering, biology and finance, as a response to model complicated systems with place and time dynamics. Reinforcement learning could be incorporated into the self-evolution mechanism in order to enhance adaptability and manual parameter adjusting. Interesting real-world applications of SEML-NODE in areas such as epidemiology, fluid dynamics and financial modeling can be used to validate and further refine the capabilities developed.

SEML-NODE presents a revolutionary self-adaptive AI design for solving ODE problems, succeeding the joint of meta-learning and evolutionary computing for architecture evolution. Experimentally, it is shown to be very accurate and efficient, which makes it a versatile tool for stiff, nonlinear and chaotic systems. Further research related to applying SEML-Node to PDEs and its hybridization with reinforcement learning will undoubtedly lead to progress in the state-of-the-art in scientific computing and beyond. As an innovative design and its demonstrated success, SEML-NODE opens a new era of intelligent and adaptive solvers for mathematics and science.