Teaching Transformers Modular Arithmetic at Scale
The work introduces novel techniques to help ML models learn modular addition. These techniques—varying the diversity of training data, using an angular embedding for model inputs and outputs, and introducing a regularized loss function—enable ML models to add hundreds of elements mod a large $q$ with high accuracy, a significant improvement over prior work.
Modular addition: given $N$ elements in $Z_q$, compute their sum modulo $q$.
October 22, 2024 567 6
