Preference labels are used to align model training, yet the judgments behind them are hard to inspect or verify. This paper investigates how preference construction can be audited when intermediate decisions are recorded, and which parts remain uncertain. We introduce Symbolic Preference Labeling: a pretrained language model classifies predefined attributes of prompts and responses, explicit rules assign response verdicts based on those attributes, and a fixed verdict ranking determines the final preference label. Each preference label has a recorded derivation, meaning the response verdicts and label can be recomputed from stored attribute assignments, rules, and the ranking without rerunning the model. Experiments on 200 human‑annotated prompts show the model produces valid structured outputs for 192 prompts; all eight parsing failures occur in the self‑harm subset marked as harmful. Using 465 preference pairs, we train a policy model with direct preference optimization across five random seeds. The mean benign refusal rate is 40.82% on 1,319 benign prompts resembling unsafe ones, and the mean toxic refusal rate is 84.98% on 655 toxic prompts. The study separates reproducibility of preference labels from attribute correctness. Recorded derivations make preference construction inspectable, while attribute correctness and semantic safety require separate validation.
Review