We introduce TasteBench, a pairwise good‑vs‑bad benchmark covering visual, brand, book, and tech items. It uses a frozen rubric and expert‑only gold labels (panel n=3, primary gate agreement 0.60, weak engagement labels for training/analysis only). To ensure the benchmark rewards genuine craft judgment rather than test‑gaming, we devise a control protocol that separates craft assessment from gaming behavior.\ \ Applying the protocol to a Round‑1 pilot (primary_human_expert; three biological human experts; n=250 pairs) surfaces three validity threats:\
- Position bias: the preferred item appears in slot A with probability 0.956, so an always‑choose‑A policy scores 0.956 (fair chance 0.5).\
- Lexical leakage: a filename token cue yields a score of 0.912.\
- Difficulty ceiling: human agreement in craft domains is near‑perfect (book/brand/tech = 1.0). Even under A/B randomization (eval_assignment_seed7; gold_side_A≈0.47), all five VLM cards reach 1.0 on craft, with only the visual dimension separating (≈0.84‑0.91). A post‑hoc hard subset (n=107) still leaves craft at 1.0.\ \ The protocol’s core components are expert‑only gold, a weak‑label firewall, side randomization during construction, filename scrubbing, and hard filters. For the two construction artifacts (position and leakage), we provide measurable and correctable fixes, releasing a reference evaluation harness and a side‑randomization tool. The tool reduces the position skew from 0.916 to roughly 0.495/0.505.\ \ The result is a calibrated view of what the benchmark currently measures: position bias and leakage are fixable, while the craft‑domain ceiling reflects a pair‑difficulty limit that Round‑2 must address. We do not evaluate engagement, era/circle conditioning, or calibration here, and we make no claim of a general taste SOTA.\ \ Review