The architecture of deep feedforward neural networks is ubiquitous in deep learning, either as a whole system or as a subnetwork of other architectures. Thus, its mechanism is a key ingredient of the black box of neural networks. Based on the simplest two-layer ReLU network, this paper systematically studies the mechanism of deep feedforward ReLU networks with multiple hidden layers and successfully explains the training solution obtained by the back-propagation algorithm. The concept of a path, especially in terms of the relationships between paths, plays a central role in uncovering the mystery of the black box. It is shown that a unit of a deep ReLU network can form a piecewise linear manifold to divide the input space, instead of a hyperplane of the two-layer case. How to efficiently use the hidden-layer units to produce both linear functions and partitions of the input space is also a central problem. The principles of a two-layer ReLU network can be generalized to the deeper case to a large extent, such as multiple strict partial orders and continuity restriction. The combination of the basic and simple principles proposed can yield complicated instantiations including the training solutions, and in this sense, the black box of deep feedforward ReLU networks is revealed.
Blogger's Review: This paper provides profound insights into the structure and mechanisms of deep feedforward ReLU networks, particularly with the introduction of the path concept, which offers a new perspective on understanding the network's complexity. This has significant theoretical implications and practical applications for researching more efficient deep learning models. Overall, the authors successfully combine theory with practice, laying a foundation for future studies.