How can we create general-purpose graph foundation models?
(by Dmitry Eremeev)
For a long time, we believed that general-purpose graph foundation models were impossible to create. Indeed, graphs are used to represent data across many different domains, and thus graph machine learning must handle tasks on extremely diverse datasets, such as social, information, transportation, and co-purchasing networks, or models of various physical, biological, or engineering systems. Given the vast differences in structure, features, and labels among these datasets, it seemed unlikely that a single model could achieve robust cross-domain generalization and perform well on all of them.
However, we noticed that tabular machine learning faces a similar challenge of working with diverse datasets containing different features and labels. And yet, this field has recently witnessed the emergence of first successful foundation models such as TabPFNv2, which are based on the prior-data fitted networks (PFNs) paradigm. Thus, we have decided to try to bring their success to the graph domain.
Our first attempt, G2T-FM, was relatively straightforward. We manually injected graph information into node features by computing structural and positional encodings, along with neighborhood-aggregated features. We then applied tabular foundation models (TabPFNv2 and LimiX) to these enriched features. Even this simple approach delivered impressive results. G2T-FM not only strongly outperforms previous graph foundation models on the GraphLand benchmark and classic datasets, but also often outperforms architecturally-improved and carefully tuned GNNs trained from scratch.
Building on this, our next step was to create GraphPFN – the first graph foundation model in the PFN framework. Moving beyond manual feature engineering of the previous approach, we first integrated message passing modules into the LimiX model so that it could learn graph-based dependencies directly, and then continually pretrained it on 4,000,000 synthetic graph datasets sampled from our specially designed attributed graph prior. The obtained model can perform node property prediction on graph datasets in a single forward pass via in-context learning and produces strong results, substantially outperforming both G2T-FM and classic GNNs on several datasets.
There remains much work to be done, including scaling to larger graphs, improving model architectures and designing better graph priors for synthetic dataset generation. However, we are now convinced that building general-purpose graph foundation models is indeed possible, and a prior-data fitted network approach is a promising path towards this goal.
For more details, check out our papers:
Turning Tabular Foundation Models into Graph Foundation Models
GraphPFN: A Prior-Data Fitted Graph Foundation Model
