In this paper, we derive explicit equations governing the cumulative biases and weights in Deep Learning with ReLU activation function. These equations are based on gradient descent for the Euclidean loss in the input layer, under the assumption that the weights are, in a precise sense, adapted to the coordinate system distinguished by the activations.
We show that gradient descent corresponds to a dynamical process in the input layer, whereby clusters of data are progressively reduced in complexity ('truncated') at an exponential rate that increases with the number of data points that have already been truncated.
We provide a detailed discussion of several types of solutions to the gradient flow equations. A main motivation for this work is to shed light on the interpretability question in supervised learning.
Blogger's Review: This paper provides an important mathematical framework in deep learning, revealing the profound connection between dynamic changes in data complexity and model training. This research not only lays the foundation for theoretical understanding but also points the way for practical model optimization, particularly through the idea of truncation strategies to enhance model efficiency, which deserves further exploration in real-world applications.