NeFut Logo NeFut
Admin Login

[CS.AI] mimeo: Compiling Public Expert Corpora into Agent Skills and Testing Transfer

Published at: 2026-09-02 22:00 Last updated: 2026-09-03 02:56
#AI #Machine Learning #Open Source

mimeo is an open‑source utility that automatically discovers a public expert's work, verifies each extracted quotation against a cached source, and writes a file that an agent can load. Across eight logged builds the system made an average of 38 model calls per build, and the verification step rejected 13.2% of the quotations. We evaluated four expert files using a single coding‑agent harness.

For knowledge access, mimeo answered all 20 obscure, quotation‑heavy questions, whereas the closed‑book condition never exceeded 10 correct answers. A BM25 keyword search over the same pages yielded 15‑17 correct responses, leaving a gap that this sample cannot resolve.

Regarding grounding, personas generated from model memory misstated a documented position on 1‑4 of the 20 answers under every grader, while both the plain agent and mimeo never made such errors. Personas were easy to spot on short open prompts; adding task material reduced identification by 18‑23 points. mimeo was no more identifiable than a from‑memory profile.

Judgment transfer remained unresolved because both tests hit a ceiling: every condition discovered 94‑97% of planted problems in engineering tasks and scored 94‑100% on 16 new application scenarios. An AI‑judged “sounds like the expert” score changed with the judge—two of four judges preferred answers based on a model’s stereotype, while the other two saw no difference—highlighting the risk of relying on a single AI judge.

Overall, the evidence supports mimeo as a compact, inspectable reference on a person, but not as a demonstrated transfer of that person’s judgment. The toolkit and expert profiles are available at https://github.com/K-Dense-AI/mimeo

Review

Original Source: https://arxiv.org/abs/2609.00453

[h] Back to Home