Skip to content
Back to search results

Graph Attention Network

Introduced by
Veličković, P. et al.
Year
2018
Type
Graph neural network (attention-based)
Font size

What does it do?

The Graph Attention Network (GAT), introduced by Veličković and colleagues in 2018, is a graph neural network variant that learns how much importance to assign to each neighbouring node through a self-attention mechanism.

On heterogeneous neighbourhoods and relational classification it produces results superior to graph convolutional networks (GCN).

The intuition behind the method

When data lives on a graph, each node is described not only by its own features but also by who its neighbours are. Earlier graph networks treated every neighbour equally when blending their information together. GAT instead lets each node learn which neighbours matter most for the task and weights their contributions accordingly, so a node draws more from the connections that are actually informative and less from the ones that are not.

History

Graph Attention Networks were introduced by Veličković and colleagues at ICLR 2018, bringing self-attention to graph neural networks so that the importance of each neighbour is learned rather than fixed. A 2022 follow-up by Brody and colleagues, How Attentive are Graph Attention Networks?, examined the expressiveness of the original attention formulation and proposed a refinement, deepening understanding of how GAT actually attends to its neighbours.

When to use it

Use GAT when your data already has a graph structure with computed node features and you need node classification or relational learning, typically with at least around 200 nodes and ideally several hundred or more. It does not assume normally distributed data and works with continuous and categorical node features. A GPU is recommended. Below roughly 500 nodes the attention weights cannot be learned reliably and overfitting becomes a risk; with very small graphs (under about 100 nodes) classical machine-learning methods are the safer choice.

Limitations and considerations

  • Requires an existing graph structure and pre-computed node features.
  • A GPU is recommended; training is computationally demanding.
  • On small graphs (fewer than about 500 nodes) the attention weights are not learned reliably and the model overfits.
  • With very small networks (under about 100 nodes) the model becomes meaningless and classical ML is preferable.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Learns neighbour importance through self-attention rather than treating all neighbours equally.
  • Outperforms graph convolutional networks (GCN) on heterogeneous neighbourhoods and relational classification.
  • Assumption-light: no normality requirement, and it handles continuous and categorical node features.
  • Multiple attention heads stabilise learning on graph-structured data.

Applications

  • Node classification on graph-structured data.
  • Relational learning across connected entities.
  • Modelling heterogeneous neighbourhoods where GCN underperforms.
  • Tasks where the relative importance of connections must be learned.

How it works

How it works

1

Each node and its features are first transformed by a shared weight matrix. For every connected pair of nodes the model then computes an attention score that measures how relevant one neighbour is to the other, capturing the idea that not all connections carry equal weight.

Formula
e_{ij} = a(W h_i, W h_j)
2

The raw attention scores for all of a node's neighbours are passed through a softmax so they sum to one. This turns them into comparable importance weights, telling the node what fraction of its updated representation should come from each neighbour.

Formula
\alpha_{ij} = \operatorname{softmax}_j(e_{ij})
3

Each node builds its new representation as the attention-weighted sum of its neighbours' transformed features, passed through a nonlinearity. Using several attention heads in parallel stabilises this process, and stacking layers lets information flow across larger neighbourhoods, which is what gives GAT its edge on heterogeneous and relational graphs.

Formula
h_i' = \sigma\!\left(\sum_{j \in \mathcal{N}_i} \alpha_{ij} W h_j\right)

Common pitfalls

  • Applying GAT to a small graph where attention weights cannot be learned reliably, inviting overfitting.
  • Forgetting that a usable graph structure must exist before the model can be applied.
  • Computing or supplying node features inconsistently, which the attention mechanism depends on.
  • Training without a GPU and underestimating the computational cost.

Frequently asked questions

How is GAT different from a graph convolutional network (GCN)?

A GCN combines a node's neighbours with weights that depend only on the graph structure, treating connections in a fixed way. GAT instead learns attention weights so each node decides which neighbours matter most, which is why it tends to do better on heterogeneous neighbourhoods and relational classification.

How much data does GAT need?

It is intended for graphs with at least around 200 nodes. Below roughly 500 nodes the attention weights are not learned reliably and overfitting becomes a real risk, and with very small networks (under about 100 nodes) classical machine-learning methods are the better choice.

Do I need a GPU to use GAT?

A GPU is recommended. As a deep learning model trained over graph neighbourhoods with multiple attention heads, GAT is computationally demanding, and a GPU makes training practical.

What do I need before applying GAT?

You need an existing graph structure and node features that have already been computed, since the attention mechanism operates on the features of connected nodes.

Related methods

Sources

  1. Veličković, P. et al. (2018). Graph Attention Networks. ICLR.

    Open publication reference
    Copied.
  2. Brody, S. et al. (2022). How Attentive are Graph Attention Networks? ICLR.

    Open publication reference
    Copied.