IntLawNER is the first named entity recognition dataset and benchmark for international law texts. It covers International Court of Justice decisions, UN Security Council resolutions and European Court of Human Rights judgments, containing 2,987 gold‑annotated sentences and 8,094 entity spans across seven institution‑specific types.
We built it with a cost‑effective hybrid algorithm‑human pipeline: candidate sentences are retrieved, filtered by large language models (LLMs), and finally reviewed by humans. This reduces 468 k source sentences to a compact annotation set, with 89.6% of gold spans unchanged from the silver layer. Comparing silver to gold reveals that a high boundary‑matched Cohen κ of 0.964 masks true performance—macro‑F1 drops to 0.753 when missing entities, boundary errors and label corrections are accounted for.
Benchmark results show that zero‑shot span‑based GLiNER collapses on entity types defined by institutional function rather than surface form (micro‑F1 0.243), and fine‑tuned transformers also struggle with rare labels. Supplying a few carefully chosen few‑shot examples that illustrate label contrasts improves every LLM, with Claude Opus reaching the best 0.873 micro‑F1. IntLawNER is released as a benchmark and reusable resource for extracting references in international legal texts.
Review