Abstract
Product data scientists often ask LLM-based agents to assist with recurring execution tasks such as cleaning data, writing SQL, choosing statistical tests, and formatting results. Reusable skill files aim to package guidance for a task family to avoid prompting from scratch. Expert-written skills can encode high-quality guidance, but writing and maintaining them across many data-science task families creates a manual bottleneck. We investigate whether LLM-generated skills provide a useful low-curation alternative: do they improve performance over the task prompt alone?
We test this across four lifecycle stages: data preparation, data extraction, statistical analysis, and reporting, using one generated skill per stage. Our findings indicate no reliable improvement from fully generated skills over No-Skill prompting. We then explore whether any part of the skill is useful by ablating different skill components. The main ablation covers 56 tasks, nine model configurations, and three providers, yielding 7,560 runs. Compared with task-only prompting, neither the full generated skill nor any ablated variant significantly improves performance; all p-values are at least 0.396, and the total spread across variants is only 1.2 pp. A supplemental token-matched control adds 1,512 runs, showing that Full skills perform similarly to task-irrelevant skill-formatted content. These results caution against using a single LLM-generated skill per data-science workflow as a default prompting strategy.
Blogger's Review: This study indicates that while LLM-generated skills may seem promising, they do not significantly enhance performance in data science tasks. It serves as a reminder to exercise caution when relying on AI tools, especially in complex workflows, and to avoid dependence on a single generated skill.