Skip to content
Back to search results

Neural ODE

Introduced by
Chen, T. Q. et al.
Year
2018
Type
Continuous-depth neural network (ODE-parameterised dynamics)
Font size

What does it do?

A Neural ODE, introduced by Chen and colleagues in 2018, models a hidden state as the continuous solution of an ordinary differential equation whose dynamics are parameterised by a neural network.

It generalises the limiting case of residual connections, making it well suited to irregularly spaced time series and physics-based modelling.

The intuition behind the method

A deep residual network adds a small learned increment to its hidden state at each layer. If you imagine shrinking those layers until they become infinitesimally close, the discrete stack of updates turns into a smooth curve, and the network describes the rate of change of the hidden state rather than a fixed number of steps. Neural ODEs take this continuous limit literally: instead of stacking layers, they learn the velocity of the state and let an ODE solver integrate it forward in time, so the state can be evaluated at any moment, not just at fixed layer boundaries.

History

Neural ODEs were introduced by Chen, Rubanova, Bettencourt and Duvenaud at NeurIPS 2018, framing the continuous limit of residual networks as an ordinary differential equation solved by a numerical integrator with the memory-efficient adjoint method. The following year, Rubanova and colleagues extended the idea to Latent ODEs for irregularly sampled time series, demonstrating that the continuous-time formulation handles unevenly spaced observations more naturally than discrete recurrent models.

When to use it

Use a Neural ODE for prediction or forecasting on continuous-time data — particularly irregularly sampled time series and longitudinal trajectories — where a smooth, continuously evolving dynamic system is a plausible model. It works on continuous variables and does not require normally distributed data, but it expects a reasonably long series (about 100 observations as a floor, and noticeably more for stable training) and an ODE solver using the adjoint method. On small datasets it cannot learn the continuous dynamics well; with fewer than around 500 observations a tree ensemble such as Random Forest is safer, and below about 100 a solver-based model is not meaningful at all.

Limitations and considerations

  • Needs a fair amount of data: with fewer than about 500 observations it cannot learn the dynamics reliably.
  • Below roughly 100 observations a solver-based model is essentially meaningless.
  • Training relies on an ODE solver and the adjoint method, adding numerical and engineering complexity.
  • Stiff or rapidly changing dynamics can make the solver slow and the training unstable.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Handles irregularly and unevenly sampled time series naturally, since the state is defined at any time point.
  • Memory-efficient training: the adjoint method keeps memory constant regardless of solver depth.
  • A principled fit for physics-based and continuous dynamical-system modelling.
  • Continuous-depth formulation generalises residual networks and lets accuracy be traded against solver cost.

Applications

  • Forecasting and prediction on irregularly sampled or unevenly spaced time series.
  • Modelling continuous dynamical and physics-based systems.
  • Longitudinal trajectory modelling where observations arrive at arbitrary times.
  • Continuous-depth alternatives to deep residual networks where memory is constrained.

How it works

How it works

1

Rather than a fixed sequence of layers, a neural network f_theta defines the instantaneous rate of change of the hidden state as a function of the current state and time. This turns network depth into continuous time: the model specifies how the state evolves, not how many discrete transformations to apply.

Formula
\frac{dh(t)}{dt} = f_\theta(h(t), t)
2

To obtain the state at a later time, a numerical ODE solver integrates the learned dynamics forward from the initial state. Because the solution is defined for any time point, the model naturally handles observations that arrive at irregular, unevenly spaced moments without forcing them onto a fixed grid.

Formula
h(t_1) = h(t_0) + \int_{t_0}^{t_1} f_\theta(h(t), t)\, dt
3

Training requires gradients of the loss with respect to the network parameters. The adjoint sensitivity method computes these by solving a second ODE backwards in time, which keeps memory cost low and constant regardless of how many solver steps the forward integration took.

Common pitfalls

  • Applying it to a very short series where there is too little signal to identify a continuous dynamic.
  • Ignoring solver tolerance settings, so accuracy and runtime are silently mismatched.
  • Forcing irregular timestamps onto a regular grid and discarding the method's main advantage.
  • Skipping the adjoint method and exhausting memory by storing every solver step.

Frequently asked questions

How is a Neural ODE different from a residual network?

A residual network adds a discrete increment at each of a fixed number of layers. A Neural ODE takes the limit where those increments become infinitesimal, so a neural network defines the continuous rate of change of the state and an ODE solver integrates it. Depth becomes continuous time rather than a fixed layer count.

Why use the adjoint method for training?

The adjoint sensitivity method computes gradients by solving an auxiliary ODE backwards in time. This keeps memory cost low and constant regardless of how many steps the forward solver took, which is the main reason it is recommended for training Neural ODEs.

When should I avoid a Neural ODE?

On small datasets. With fewer than about 500 observations the model cannot learn the continuous dynamics reliably, and below roughly 100 a solver-based model is not meaningful. In those cases a tree ensemble such as Random Forest or XGBoost is a safer choice.

What kind of data does it suit best?

Continuous-valued time series and longitudinal data, especially when observations are irregularly or unevenly spaced. Because the state is defined at any time point, the method does not require resampling the data onto a fixed grid.

Related methods

Sources

  1. Chen, T. Q., Rubanova, Y., Bettencourt, J. & Duvenaud, D. (2018). Neural Ordinary Differential Equations. Advances in Neural Information Processing Systems (NeurIPS).

    Open publication reference
    Copied.
  2. Rubanova, Y., Chen, T. Q. & Duvenaud, D. (2019). Latent ODEs for Irregularly-Sampled Time Series. Advances in Neural Information Processing Systems (NeurIPS).

    Open publication reference
    Copied.