LexGenius

LexGenius: An Expert-Level Benchmark for Large Language Models in
Chinese Legal General Intelligence

Wenjin Liu^1,2,* Haoran Luo^2,* Xin Feng¹ Xiang Ji¹ Lijuan Zhou^1,†
Rui Mao² Jiapu Wang³ Shirui Pan⁴ Erik Cambria²

¹Hainan University
²Nanyang Technological University
³Nanjing University of Science and Technology
⁴Griffith University

wenjinliu23@outlook.com, haoran.luo@ieee.org

Abstract

Legal general intelligence (GI) refers to artificial intelligence (AI) that encompasses legal understanding, reasoning, and decision-making, simulating the expertise of legal experts across domains. However, existing benchmarks are result-oriented and fail to systematically evaluate the legal intelligence of large language models (LLMs), hindering the development of legal GI.

To address this, we propose LexGenius, an expert-level Chinese legal benchmark for evaluating legal GI in LLMs. It follows a Dimension–Task–Ability framework, covering seven dimensions, eleven tasks, and twenty abilities. We use recent legal cases and exam questions to create multiple-choice questions, combining manual and LLM reviews to reduce data leakage risks and ensure accuracy and reliability through multiple rounds of verification.

We evaluate twelve state-of-the-art LLMs on LexGenius and conduct an in-depth analysis. Our findings reveal significant disparities across legal intelligence abilities, with even the strongest LLMs still lagging behind human legal professionals. We believe LexGenius can serve as a comprehensive benchmark for assessing legal intelligence abilities in LLMs and contribute to advancing legal GI development. Our project is available at https://github.com/QwenQKing/LexGenius.

BibTeX

@misc{liu2025lexgeniusbenchmark, title = {LexGenius: An Expert-Level Benchmark for Large Language Models in Chinese Legal General Intelligence}, author = {Wenjin Liu and Haoran Luo and Xin Feng and Xiang Ji and Lijuan Zhou and Rui Mao and Jiapu Wang and Shirui Pan and Erik Cambria}, year = {2025}, eprint = {2512.04578}, archivePrefix = {arXiv}, primaryClass = {cs.CL}, url = {https://arxiv.org/abs/2512.04578} }

LexGenius: An Expert-Level Benchmark for Large Language Models in
Chinese Legal General Intelligence

Comparison of LLMs and human legal experts across seven legal intelligence dimensions, showing that humans consistently outperform LLMs in all evaluated areas.

The LexGenius three-level framework, consisting of seven legal dimensions, eleven task types, and twenty fine-grained abilities for evaluating legal intelligence in LLMs.

The MCQ construction workflow of LexGenius, combining LLM-assisted generation with multi-stage human evaluation across data collection, rule construction, preliminary screening, and final selection.

Data distribution of LexGenius across Chinese laws and the hierarchical structure of dimensions, tasks, and abilities, showing balanced yet diverse coverage for evaluating legal intelligence in LLMs.

Comparison of twelve state-of-the-art LLMs with human legal experts across seven core dimensions of legal intelligence, revealing significant performance gaps.

Performance comparison of twelve LLMs and human experts across eleven legal tasks, revealing a substantial performance gap. DeepSeek-R1 and DeepSeek-V3 achieve the highest scores, with Task 3 being the most challenging for all models.

Average ranking and mean score ranking of twelve state-of-the-art LLMs across twenty legal intelligence abilities, highlighting notable performance differences.

Performance of twelve LLMs across nine legal language indicators, showing notable gaps from the human baseline. Even the top-performing models, DeepSeek-R1 and DeepSeek-V3, fall short of human proficiency in mastering legal language.

Correlation analysis of legal intelligence across abilities, tasks, and dimensions in LexGenius, based on the performance patterns of twelve LLMs.

Abstract

Paper

BibTeX