Deep neural networks often contain substantial hidden-state redundancy, but most compression methods operate directly on weights, neurons, or quantized representations without explicitly characterizing the dynamical role of internal states. This paper proposes a controllability-observability framework for empirical state-order reduction of deep neural networks. By viewing a trained network as a depth-indexed nonlinear dynamical system, we construct data-driven reachability, observability, and balanced Gramians from hidden-state snapshots and output Jacobians. The resulting A/B/C tests estimate layer-wise reachable, observable, and jointly reachable-observable ranks. These ranks are then used not only as diagnostic measures of hidden-state redundancy but also as actual compressed layer widths for realized reduced networks.
Experiments on MNIST and CIFAR-10 compare the proposed balanced realization against projection-based reduction, unstructured pruning, structured pruning, low-rank SVD, dynamic INT8 quantization, and linear baselines. On MNIST, a four-layer SiLU DNN is reduced from state order 1024 to 277, giving 72.95% state compression and 73.48% parameter compression while maintaining 95.45% accuracy compared with 96.60% for the full model. On CIFAR-10, a larger SiLU DNN is reduced from state order 4608 to 1339, achieving 70.94% state compression and 83.09% parameter compression, while preserving accuracy from 54.45% to 54.44% and reducing CUDA inference latency by approximately 3X. The results demonstrate that balanced reachable-observable ranks provide a principled empirical minimal-realization criterion for designing compact neural architectures with little or no loss in accuracy.
Blogger's Review: The proposed controllability-observability framework offers a fresh perspective on neural network compression, effectively reducing hidden states while balancing compression and accuracy. This work holds significant theoretical and practical value, and future research could explore its application potential in more complex network architectures.