LGMT is an open‑source tool designed to expose logical reasoning inconsistencies in large language models (LLMs). Conventional static benchmarks evaluate each query in isolation, so they miss cases where a model gives contradictory answers to semantically equivalent formulations. LGMT tackles this by employing metamorphic relations grounded in first‑order logic. The workflow is straightforward:
- Provide a source test case in premise‑conclusion form.
- Generate logically equivalent follow‑up cases according to built‑in or user‑defined metamorphic relations.
- Query the model on both cases and check whether the entailment predictions agree; no ground‑truth labels are required.
The tool ships with 20 pre‑defined relations and allows custom ones, which are verified for logical equivalence by the Z3 theorem prover before use.
LGMT can be run in two modes:
- CLI: command‑line interface for batch processing.
- Web Studio: interactive web UI for exploratory testing. Both modes support an offline mock mode that does not need any API key.
When evaluated on six widely used LLMs (e.g., GPT‑4, Claude, Llama‑2) over a bundled benchmark of 3,091 metamorphic groups, LGMT uncovered inconsistency rates ranging from 20.84% to 29.44%, highlighting substantial reliability gaps in current models.
The source code is publicly available on GitHub (https://github.com/MaxwelsDonc/lgmt-tool) and a demonstration video is hosted on YouTube (https://youtu.be/gou4opptu1U).
Review: LGMT offers a systematic and extensible approach to assess LLM logical consistency, especially valuable in label‑free settings, enabling researchers and engineers to pinpoint reasoning flaws early in the development cycle.