ContraGAND: Auditing and Repairing Gender Ambiguity Failures in LLMs with Neurosymbolic Contrastive Data Augmentation

EMNLP 2026

A contrastive corpus of 4,852 gender-ambiguous sentences with minimal-pair variants, used to audit six LLMs and to fine-tune 4B students past a 31B teacher.

4,852 gender-ambiguous sentences, each with a masculine and a feminine minimal pair, Table 1

A symbolic router, an LLM proposer, an LLM judge and symbolic validators with a repair loop, Figure 1

Every model drops when it must find the referent itself, most of all the small ones, Tables 4 and 5

Fine-tuning on the contrastive variants lifts two 4B students past the 31B teacher, Table 8

Gender-ambiguous sentences are where translation systems quietly pick a gender. ContraGAND is a contrastive corpus of 4,852 natural gender-ambiguous English sentences, each minimally paired with a masculine and a feminine variant generated by a neurosymbolic pipeline and audited by two annotators. We use it to evaluate gender ambiguity detection in six open-weight LLMs under two framings: a closed task where the referent is given, which current models solve, and an open task where the model must find the referent itself, which opens a gap that grows as models shrink. Two failure modes dominate, a binding failure where a cross-clause cue is not linked back to its referent, and a format failure where small models never produce a parseable answer. Both are over-prediction of the ambiguous class rather than stereotype-driven errors. Prompting and decoding fixes do not close the gap, but fine-tuning on the contrastive variants does, and distillation from a 31B teacher lets a 4B student match and pass it.

PaperACL Anthology CodeGitHub DatasetHugging Face Model, closed taskHugging Face Model, open taskHugging Face DemoHugging Face Spaces

Research theme: Fairness in language models