Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge

2025 · Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics · Creative Commons Attribution 4.0 International (CC BY 4.0)

Lin Shi; Chiyu Ma; Wenhua Liang; Xingjian Diao; Weicheng Ma; Soroush Vosoughi. (2025). Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge. Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics.

DOI: 10.18653/v1/2025.ijcnlp-long.18

Original publication: https://doi.org/10.18653/v1/2025.ijcnlp-long.18

Licensed under Creative Commons Attribution 4.0 International (CC BY 4.0).

This copy is redistributed under the terms of the licence above. No changes have been made to the content.

Read at the publisher → Download PDF Open PDF in a new tab

This is a copy redistributed under Creative Commons Attribution 4.0 International (CC BY 4.0). The publisher's copy at the link above is the version of record.

Read here