Abstract
We introduce a novel method for both partial and full optimization of the connections in deep differentiable logic gate networks (LGNs) and lookup table networks (LUTNs). Our training method utilizes a probability distribution over a set of connections per gate/lookup table (LUT) input pin, selecting the connection with the highest merit, all whilst the optimal gate types or LUT-entries are learned in parallel.
We show that the connection-optimized LGNs outperform standard fixed-connection LGNs on the Yin-Yang, MNIST Handwritten Digits, and Fashion-MNIST benchmarks, while requiring only a fraction of the number of logic gates. We achieve 98.92% on the MNIST dataset with two layers of 8000 gates. With only one layer of 8000 gates, we obtain 98.45%, demonstrating that our method requires almost 50 times fewer gates compared to fixed-connection LGNs.
Training stability up to ten layers has been ensured by employing a high learning rate, straight-through estimators, and trimming constant-output gate types. Additionally, we present a LUT neuron description that enables stable training with backpropagation, tested up to 6-layer deep networks. The model requires four times fewer trainable parameters and still achieves a higher accuracy compared to the fixed-connection LGN training algorithm. Our connection-training algorithm also works well for the LUTNs, achieving an accuracy of 98.88% for two layers of 2000 6-input LUTs.
Blogger's Review: The proposed optimization method in this paper shows great potential in the realm of logic gate networks and lookup table networks, significantly enhancing accuracy while reducing computational resource demands. It demonstrates the synergy between deep learning and logic optimization, warranting further exploration and validation in practical applications.