In teacher‑student systems a teacher network generates labels and a student network learns to implement the same function. Prior work often assumes the teacher’s parameters are random Gaussian, overlooking structural differences. We define learnability as the success rate of converging to the global minimum, a function of over‑parameterization, learning algorithm, student initialization and teacher geometry. By constructing two teacher distributions—an easy one that maximizes node dissimilarity and a hard one that minimizes it—we observe markedly different success rates across many settings and activation functions. To explain the gap we analyze the loss landscape of small networks that contain two kinds of suboptimal local minima: out‑of‑bounds (OOB) minima at the edge of the data distribution and interior minima inside it. Assuming infinite data and a fast readout layer, the loss can be analytically reduced to two dimensions, revealing that the attraction region of interior minima varies with teacher structure. Empirically, in larger networks maximally dissimilar teachers produce more interior minima, while minimally dissimilar teachers produce more OOB minima. Guided by this analysis, increasing the learning rate of the readout layer while decreasing the learning rate of inner biases markedly improves success rates. These findings narrow the gap between abstract teacher‑student studies and the structured functions encountered in practice.
Review