Deep learning has achieved remarkable success in various domains including time series analysis, computer vision, and natural language processing. However, the high computational and memory demands of state-of-the-art architectures pose challenges for deployment in resource-limited environments. Knowledge Distillation (KD) addresses this by transferring knowledge from a large teacher model to a smaller, more efficient student model while maintaining competitive performance.
This work investigates the effectiveness of KD for Time Series Classification (TSC) across three architectures: the classical Fully Convolutional Network (FCN), the convolutional Inception model, and the transformer-based ConvTran model. We evaluate our approach on the UCR Archive, the largest benchmark repository of time series datasets, by modifying architectural components such as convolutional filters, Inception modules, and attention heads across the three architectures.
Our results consistently show that KD most effectively benefits student models of intermediate complexity across all three architectures. The distilled FCN student reduces parameters by a factor of 38, the distilled Inception student achieves nearly the same performance as the teacher with 42% fewer parameters, and the distilled ConvTran student with 2 attention heads shows the most significant improvement through distillation. To encourage further research and reproducibility, we provide our implementation at GitHub.
Blogger's Review: This paper significantly enhances the efficiency of time series classification models through knowledge distillation, particularly in resource-constrained scenarios, showcasing the flexibility and adaptability of deep learning models, which is worth noting for researchers in related fields.