Skip to content
Back to search results

Dilated CNN

Introduced by
van den Oord, A. et al.; Bai, S., Kolter, J.Z. & Koltun, V.
Year
2016
Type
Deep learning (dilated 1D convolutional network)
Font size

What does it do?

A Dilated CNN is a one-dimensional convolutional network whose receptive field grows exponentially with depth, letting it model long-range structure in time series and audio signals.

WaveNet (van den Oord et al., 2016) and the Temporal Convolutional Network of Bai, Kolter and Koltun (2018) are the prominent members of this family.

The intuition behind the method

Ordinary convolutions only look a few steps back, so reaching far into the past would need a very deep stack. A dilated convolution instead skips over inputs with a gap that doubles layer by layer, so each added layer roughly doubles how far back the network can see. A handful of layers can therefore cover thousands of time steps while keeping the parameter count modest.

History

Dilated convolutions for sequences were popularised by WaveNet (van den Oord et al., 2016), which used a stack of causal dilated convolutions to generate raw audio. Bai, Kolter and Koltun (2018) generalised the idea into the Temporal Convolutional Network and showed empirically that such convolutional architectures often match or outperform recurrent networks on a broad range of sequence-modelling tasks.

When to use it

Use a Dilated CNN for forecasting or prediction on continuous, sequential time-series data when long-range dependencies matter and you have at least a few hundred observations. It does not assume normally distributed data, but it needs sequential data, benefits strongly from a GPU, and the receptive-field size should be matched to the task. Below about 500 observations it tends to overfit, and below roughly 100 a deep convolutional model is not worth training.

Limitations and considerations

  • Needs a fairly large training set — overfits when n is below about 500.
  • A GPU is effectively required for practical training.
  • The receptive-field size must be tuned to the task; too small a field misses long dependencies.
  • Deep convolutional sequence training is not meaningful on very short series (n below about 100).

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Receptive field grows exponentially with depth, capturing long-range dependencies with few layers.
  • Convolutions parallelise well, training faster than recurrent networks on long sequences.
  • No normality assumption; works directly on continuous sequential signals.
  • Proven on demanding tasks, from WaveNet raw-audio generation to general TCN sequence modelling.

Applications

  • Time-series forecasting where long-range dependencies matter.
  • Raw-audio generation and modelling, as in WaveNet.
  • General sequence modelling as an alternative to recurrent networks (TCN).
  • Signal prediction tasks on long continuous sequences.

How it works

How it works

1

Convolutional filters are applied with a dilation rate that grows (typically 1, 2, 4, 8, ...) from one layer to the next. Each filter still has only a few weights, but the gaps between the inputs it touches widen with depth, so the model captures both short-term detail and long-term dependencies without exploding in size.

2

Because the dilation doubles at each layer, the receptive field expands exponentially with depth: a stack of L layers with kernel size k can attend to on the order of a thousand past steps with only a handful of layers. The receptive-field size must be tuned to the horizon the task actually requires.

Formula
r = 1 + (k-1)\sum_{l} d_l
3

Stacked dilated convolutions feed a final layer that emits the prediction for each position — the next value in a forecast or the label for a sequence step. The same architecture underlies WaveNet's raw-audio generation and the TCN's general sequence modelling.

Common pitfalls

  • Setting the receptive field too small so long-range dependencies are never seen.
  • Training on a short series where the model overfits or learning is meaningless.
  • Expecting usable results without a GPU on long sequences.
  • Ignoring that with very little data a simpler model such as a tree ensemble is the safer choice.

Frequently asked questions

Why dilation instead of just a deeper network?

Dilation lets the receptive field grow exponentially rather than linearly with depth, so a few layers can reach thousands of past steps while keeping the parameter count and computation modest.

How does it compare to recurrent networks?

Bai, Kolter and Koltun (2018) found that convolutional architectures like the TCN often match or beat recurrent networks on sequence tasks, and they train faster because convolutions parallelise across time.

How much data do I need?

Plan for at least a few hundred sequential observations. Below about 500 the model tends to overfit, and below roughly 100 training a deep convolutional model is not worthwhile — a tree-based model is safer.

Do I need a GPU?

A GPU is strongly recommended; without one, training on long sequences becomes impractically slow.

Related methods

Sources

  1. van den Oord, A. et al. (2016). WaveNet: A Generative Model for Raw Audio. arXiv.

    Open publication reference
    Copied.
  2. Bai, S., Kolter, J.Z. & Koltun, V. (2018). An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling. arXiv:1803.01271.

    Open publication reference
    Copied.