Span‑level masked language modeling (MLM) has been shown to outperform the original token‑wise MLM because entities, phrases and their dependencies are essential for language understanding. Prior work mainly models the span length using discrete distributions while assuming masked spans are uniformly positioned, thus ignoring positional dependencies. POSPAN introduces a general framework that combines a span length distribution with a position constraint distribution, enabling diverse position‑constrained span masking strategies and unifying all existing span‑level masking methods. Experiments on several NLU benchmark datasets demonstrate that the position constraint consistently enhances span‑level masking; the best POSPAN setting outperforms both length‑only baselines and vanilla MLM. Theoretical analysis reveals that position constraints increase coverage and information density of masked tokens, explaining POSPAN’s effectiveness and confirming its rationality and necessity.
Review