On the Role of Attention Masks and LayerNorm in Transformers
Publications
Preprints
Residual Connections and Normalization Can Provably Prevent Oversmoothing in GNNs
What Are We Optimizing For? A Human-centric Evaluation of Deep Learning-based Movie Recommenders
2023
Demystifying Oversmoothing in Attention-Based Graph Neural Networks
NeurIPS 2023 (Spotlight) / Paper
LOG 2023 (Oral)/ Talk (starting at 4:47:40) / DeepMath 2023 (Oral) / Talk (starting at 6:20:30)An Non-Asymptotic Analysis of Oversmoothing in Graph Neural Networks
ICLR 2023 / Paper
Blog: Oversmoothing in GNNs: why does it happen so fast? (and do popular solutions such as residual connections or normalization really work?)2022
2020
Spectra of Quantum Graphs via a Scattering Matrix Approach
Senior Thesis / Paper