Anna
Shalova
TU Eindhoven
Porous medium is the message: variational analysis of toy transformers
Abstract.
We study an aggregation PDE with competing attractive and repulsive forces on a high-dimensional sphere. In particular, we consider the limit of localized repulsion with a constant attraction term. We prove convergence of solutions of such a system to solutions of the aggregation-diffusion equation with the porous-medium-type diffusion. The proof combines variational techniques with the elements of harmonic analysis on a sphere. Specifically, we characterize the square root of the convolution operator in terms of the spherical harmonics, which allows us to overcome difficulties arising due to the convolution on a sphere being non-commutative. The system under study naturally arises as an extension of the toy model of transformers introduced by Geshkovski et al. (2024) and provides some insights on the driving mechanisms behind the modern language models. This is a joint work with Mark Peletier.