ExecRetrieval is a new benchmark that targets functional correctness in code‑embedding retrieval. It comprises 939 Python tasks, each paired with an execution‑verified canonical implementation and up to four buggy distractors created by a single‑edit mechanical mutation. The distractors remain almost identical to the canonical code but introduce functional errors, providing controlled counterfactuals inside the retrieval pool.
We evaluate 23 dense embedding configurations and BM25 in the provider‑native setting, applying paired McNemar tests and query‑level bootstrap intervals. The best system reaches exec@10 = 1.00 but only exec@1 = 0.331; rank‑1 misses are buggy variants 91.5%–99.4% of the time, and the canonical score falls below at least one distractor in 67%–78% of queries. The full dataset, execution oracle, embedding matrices, environment snapshot, and pairwise statistical tests are released.
Review