Quality-Control (QC). Each QC item is a known-answer (gold) task with a few labels deliberately
pre-filled wrong (planted errors) and the rest correct — indistinguishable from a normal task:
- seen — total (entity × label) answers the annotator gave across their QC items.
- corrected / planted — how many planted (deliberately-wrong) labels were shown, and how many they fixed.
- correction — corrected ÷ planted: share of planted errors they caught (higher = better).
- false corr. — of labels that were already correct, how often they changed them to something wrong (lower = better).
- obvious false — same, but only for labels marked obviously correct (breaking one is a strong bad signal).
- accuracy — share of all their answers that matched the gold.
- acceptance — how often they left the pre-filled value unchanged.
A good annotator: high correction + high accuracy + low false-correction.
A rubber-stamper / cheater: low correction + high false-correction.
LLM pre-annotation. How the 3 AI models' suggestions fared against the humans:
- shown — number of (annotation × model × entity × label) instances where an AI suggestion
was shown and answered (counted per model ×3, so it is large by design).
- change rate — how often humans changed the AI's suggested label.
- chg=final — of those changes, how many matched the final merged consensus (human correctly overrode the AI).
- chg=llm — of those changes, how many the merge kept the AI's value (the AI was right / the change didn't win).