Learning Kolmogorov-Arnold Neural Activation Functions by Infinite-Dimensional Optimization

Inspired by the Kolmogorov-Arnold Representation Theorem, Kolmogorov-Arnold Networks (KAN) have recently reshaped the landscape of functional representation in the context of training univariate activation functions, thus achieving remarkable performance gains over traditional Multi-Layer Perceptron (MLPs). However, the parametrization of the activation functions in KANs hinders their scalability and usage in machine-learning applications. In this article, we propose a novel infinite-dimensional optimization framework to learn the activation functions. Our game-changing approach enables the achievement of boundedness, interpretability, and, seamless compatibility with training by backpropagation are all preserved for the network, while also significantly reducing the number of parameters required to represent each edge. Through functional representation examples, our results reveal superior accuracy in tasks such as functional representation, positioning our method as a transformative leap forward in neural network design and optimization.