Abstract
Document-based LLM systems often flatten a PDF before guardrails inspect it. That step can discard evidence that an instruction was never visible to the user. We introduce CrackedPDFs, a controlled benchmark for hidden prompt injection in PDFs. The benchmark contains 29,322 generated PDFs from 4,983 base documents, including 9,774 injected files and 19,548 benign or matched-confounder files.
We evaluate PromptGuard and a rule baseline, as well as structural-only learned models and a sanitized hybrid detector. The evaluation uses held-out provenance splits and paired benign-confounder controls, along with label-shuffle checks and shortcut audits.
On a 2,919-document held-out test set, the hybrid detector achieves 0.960 F1, with ROC-AUC at 0.998 and PR-AUC at 0.997. It ranks injected files above matched benign confounders in 95.9% of 973 pairs. However, PromptGuard shows low recall when given extracted text only. Structural-only learned models perform weakly under paired controls. A text-only TF-IDF model reaches perfect held-out scores but fails shortcut audits. These results indicate that document-aware hybrid detection is useful under controlled paired evaluation, but do not demonstrate broad real-world robustness or reliable cross-family generalization.
Blogger's Review: CrackedPDFs offers a significant benchmark for prompt injection in PDFs, highlighting potential flaws in current LLM systems under protective mechanisms. While the hybrid detector performs exceptionally well in controlled settings, its reliability in real-world applications remains to be validated.