Abstract
Data-free knowledge distillation transfers knowledge encoded in a teacher model to a student model without access to the original training data. Prior work like Contrastive Abductive Knowledge Extraction (CAKE) achieves this by synthesizing samples near the teacher's decision boundary. This study investigates whether this boundary-seeking principle extends to autoencoder distillation through experiments on the MNIST dataset.
To enable direct comparison, we reformulate continuous reconstruction as a dense, per-feature classification task, allowing the decoder to output categorical logits. We demonstrate that boundary-seeking objectives are fundamentally ill-posed in bottlenecked generative architectures. CAKE operates on a single, instance-level objective, while a decoder acts as an array of tightly coupled, feature-level classifiers constrained by a shared low-dimensional bottleneck. Independently sampling contrastive targets for these coupled outputs violates the geometry of the learned latent manifold, resulting in severe gradient conflicts rather than informative boundary samples. Manifold-aware synthesis bypasses these conflicts entirely, establishing an effective baseline for data-free generative distillation.
Blogger's Review: This paper reveals the limitations of boundary-seeking methods in bottlenecked generative models, emphasizing the significance of manifold-aware synthesis. It provides new insights for future research in data-free generative distillation, where resolving gradient conflicts will be crucial.