Decentralized social platforms introduce multi‑layered challenges and opportunities for computational mental‑health research, spanning data access, moderation, labeling, and model deployment. This paper presents a validation‑aware weak‑supervision system for building suicidal ideation (SI) and broader mental‑health (MH) disclosure benchmarks on Bluesky, a platform built on the AT Protocol. The pipeline integrates public firehose collection, task‑specific lexicon filtering, Llama‑3‑8B‑assisted binary annotation, human‑adjudicated validation subsets, and transformer‑based model benchmarking. Using this workflow we assembled two corpora containing 8,346 SI‑labeled posts and 9,988 MH‑labeled posts. Evaluation shows that performance hinges on both task definition and validation protocol: BERT+LSTM achieves the highest SI stratified cross‑validation F1, RoBERTa attains the strongest SI holdout F1, and DistilRoBERTa yields the best MH cross‑validation F1. Human validation uncovers distinct weak‑label failure modes—SI labels are dominated by false negatives, while MH labels are dominated by false positives. These findings demonstrate that decentralized social media can support reproducible mental‑health benchmarking, provided system design, label provenance, validation strategy, and deployment constraints are jointly considered.
Review: The framework offers a comprehensive recipe that couples large‑model‑assisted labeling with human validation on a distributed platform, delivering a reproducible and auditable benchmark pipeline for future mental‑health research.