Yila
Exemplo de revisão de literaturaVoltar à revisão de literatura

Advances in Large Language Model Evaluation: Benchmarking Frameworks, Capability Boundaries, and Reliability Challenges

Abstract

The rapid advancement of large language models (LLMs) has created an urgent need for robust evaluation methodologies capable of keeping pace with expanding model capabilities. This review organizes the LLM evaluation landscape into a three-generation progression—static benchmarking frameworks, dynamic and human evaluation, and LLM-based automated evaluation—and critically examines the reliability challenges that persist across all paradigms. We trace the evolution from early multi-task benchmarks such as GLUE through comprehensive capability assessments including MMLU, BIG-bench, and domain-specific evaluations like MultiMedQA and FrontierMath. We then analyze the emergence of crowdsourced human evaluation platforms (Chatbot Arena) and the widely adopted LLM-as-a-judge paradigm, surveying documented biases and calibration strategies. Safety and robustness evaluation, including standardized red-teaming frameworks (HarmBench, MART) and persuasion-based jailbreaking, is examined as a distinct yet interconnected evaluation dimension. Six cross-cutting reliability challenges are synthesized: data contamination, benchmark saturation, construct validity, judge bias, cultural and linguistic coverage limitations, and reproducibility failures. We argue that no single evaluation paradigm adequately captures the breadth of LLM capabilities and that the field must develop unified, contamination-resistant, multi-dimensional assessment frameworks grounded in established reporting standards.

1. Introduction

Large language models have become foundational to modern artificial intelligence, deployed in high-stakes applications ranging from clinical decision support and legal document analysis to code generation and educational tutoring [1, 4]. As these models grow in capability and societal reach, the question of how to evaluate them has transitioned from a technical subproblem into a distinct research field with its own methodologies, benchmarks, and epistemological challenges [1, 43]. Unlike traditional task-specific models, LLMs exhibit general-purpose capabilities that span diverse domains, making their assessment fundamentally more complex than measuring performance on a single, well-defined task.

The proliferation of evaluation approaches over the past six years has been remarkable. Static multiple-choice benchmarks such as MMLU and BIG-bench provide broad capability surveys but face criticism for data contamination and format limitations [17, 22]. Crowdsourced human evaluation platforms such as Chatbot Arena offer open-ended preference measurement but struggle with scalability and cultural representation [5, 19]. The LLM-as-a-judge paradigm promises automated, interpretable evaluation at scale yet introduces systematic biases including position, verbosity, and self-enhancement effects that require careful calibration [6, 7, 24]. Safety and robustness evaluation has developed its own parallel track of red-teaming frameworks and jailbreak taxonomies [12, 14], while agent evaluation introduces novel challenges of multi-turn interaction and long-horizon task completion [8, 9].

Despite this methodological breadth, critical synthesis of how these evaluation paradigms relate to one another and where their shared reliability vulnerabilities lie remains scarce. Existing surveys tend to focus on individual dimensions—comprehensive evaluation frameworks [1], LLM-as-a-judge methodologies [6, 7], safety testing [30], or agent assessment [8]—without tracing the cross-cutting reliability challenges that undermine all of them. This review addresses that gap by proposing a three-generation progression model for organizing LLM evaluation approaches and then extracting six reliability challenges that persist across generations.

The review is organized as follows. Section 2 traces the evolution of static benchmarking from GLUE through contemporary domain-specific benchmarks. Section 3 examines dynamic and human evaluation approaches, including crowdsourced platforms and multilingual assessment. Section 4 surveys the LLM-as-a-judge paradigm, its documented biases, and calibration strategies. Section 5 covers safety, robustness, and red-teaming evaluation. Section 6 discusses the emerging area of agent evaluation. Section 7 synthesizes six cross-cutting reliability challenges. Section 8 concludes with recommendations for unified evaluation frameworks and open research questions.

This review draws on 43 core references identified through systematic search of OpenAlex, prioritizing peer-reviewed publications while including influential preprints where they constitute the primary available evidence. Evidence boundaries are abstract-level by default; full-text inspection was conducted only when exact benchmark definitions or critical claims could not be supported from abstracts alone. Preprints are explicitly distinguished from peer-reviewed publications throughout.

2. Static Benchmarking Frameworks

2.1 The Multi-Task Paradigm

The modern era of LLM evaluation began with the introduction of GLUE (General Language Understanding Evaluation) in 2018, a multi-task benchmark comprising nine diverse natural language understanding tasks including textual entailment, sentiment analysis, and linguistic acceptability [2]. GLUE's key innovation was its emphasis on general-purpose language understanding rather than task-specific optimization, establishing the multi-task evaluation paradigm that would dominate NLP for years. Its successor, SuperGLUE, raised the difficulty bar by replacing saturated tasks with more challenging ones, but the fundamental design—a fixed set of labeled datasets with held-out test sets—remained unchanged.

The GLUE paradigm established several design dimensions that continue to shape LLM evaluation: task diversity (covering multiple linguistic phenomena), dataset size (ranging from small diagnostic sets to large crowdsourced collections), and standardized scoring (macro-average across tasks with leaderboard ranking). However, it also inherited limitations that later work would seek to address, including reliance on the multiple-choice question answering (MCQA) format, vulnerability to dataset artifacts and spurious correlations, and the absence of generative evaluation.

2.2 Scaling to Comprehensive Capability Assessment

The emergence of large-scale LLMs with hundreds of billions of parameters necessitated benchmarks that could test broad world knowledge and multi-step reasoning at scale [4, 34]. MMLU (Massive Multitask Language Understanding) addressed this by collecting 57 subjects spanning humanities, social sciences, STEM, and medicine, testing knowledge breadth through thousands of multiple-choice questions. BIG-bench (Beyond the Imitation Game Benchmark) adopted a collaborative approach, incorporating over 200 tasks contributed by hundreds of researchers, including many designed to probe specific reasoning capabilities unlikely to appear in training data. HELM (Holistic Evaluation of Language Models) introduced a standardized framework for evaluating models across 42 scenarios using 7 metrics including accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency, explicitly recognizing that LLM evaluation is inherently multi-dimensional.

Instruction-tuned models demonstrated that performance on these comprehensive benchmarks could be substantially improved through scaling instruction-following training data and chain-of-thought prompting [34]. Flan-PaLM 540B, instruction-finetuned on 1.8K tasks, achieved 75.2% on MMLU five-shot, representing a 9.4% average improvement over the base PaLM 540B model. These gains, however, raised questions about whether instruction tuning genuinely improves capability or merely aligns model outputs more closely with benchmark formatting expectations.

2.3 Domain-Specific Evaluation

General benchmarks provide broad capability profiles but may miss domain-specific strengths and weaknesses. The MultiMedQA benchmark combined six existing medical question-answering datasets with a new dataset of consumer medical queries (HealthSearchQA) and introduced a human evaluation framework assessing factuality, comprehension, reasoning, potential harm, and bias [21]. This multi-axis approach revealed that while Flan-PaLM achieved state-of-the-art accuracy on medical QA datasets, its performance on open-ended consumer queries was more variable, suggesting that benchmark scores on curated datasets may overestimate real-world clinical utility.

Similarly, SciEval used Bloom's taxonomy to systematically evaluate scientific research ability across four dimensions, incorporating a dynamic subset based on scientific principles to mitigate data leakage [10]. FrontierMath introduced hundreds of original, exceptionally challenging mathematics problems requiring hours of effort from expert mathematicians, finding that state-of-the-art AI models solved under 2% of problems—revealing a vast gap between AI mathematical capabilities and expert human performance [16]. In code generation, EvalPlus augmented existing evaluation datasets with large amounts of newly generated test cases, revealing that many seemingly correct code generations failed under more rigorous testing [26]. These domain-specific evaluations collectively demonstrate that LLM capabilities are unevenly distributed across domains and that standard benchmarks may systematically overestimate performance in specialized areas.

2.4 Benchmark Saturation and Format Limitations

A growing body of evidence indicates that many widely used benchmarks are approaching saturation, where models reach performance ceilings that no longer discriminate meaningful capability differences [15, 17]. This saturation is partly a natural consequence of progress, but it is also exacerbated by format-specific vulnerabilities. Balepur et al. demonstrated that MCQA evaluation—the dominant format across most LLM benchmarks—suffers from three fundamental flaws: it cannot test generative or subjective capabilities, it poorly matches real-world LLM use cases (which are predominantly generative), and it only partially tests knowledge due to the recognition-based format [17]. The authors advocate for generative evaluation formats based on human testing principles, where LLMs construct and explain answers, better capturing both user needs and genuine knowledge while remaining easy to score.

3. Dynamic and Human Evaluation

3.1 Crowdsourced Evaluation Platforms

The limitations of static benchmarks motivated the development of crowdsourced human evaluation platforms, most notably Chatbot Arena [5]. Chatbot Arena collects human preference judgments through pairwise model comparisons in an open-ended chat setting, calculating ELO ratings that reflect relative model quality across diverse, unconstrained interactions. Zheng et al. demonstrated that GPT-4 as a judge can achieve over 80% agreement with human preferences on the MT-Bench multi-turn question set—a level comparable to inter-human agreement—suggesting that automated evaluation can approximate human judgment for certain tasks [5].

However, the crowdsourced approach introduces its own limitations. Platform populations skew toward technically proficient users, potentially biasing preference data. The open-ended nature of interactions makes controlled comparison across models difficult, and the ELO rating system, while well-established for games, may not capture nuanced differences in model behavior across diverse task types.

3.2 Human Evaluation in Specialized Domains

In high-stakes domains such as medicine, human evaluation remains essential for assessing clinically relevant dimensions that automated metrics cannot capture. Singhal et al. proposed a human evaluation framework for medical model answers along multiple axes including factuality, comprehension, reasoning, potential harm, and bias [21]. Van Veen et al. found that adapted LLMs could outperform medical experts in clinical text summarization when evaluated by expert clinicians, but achieving this required careful prompt engineering and domain-specific adaptation [39].

Human evaluation also plays a critical role in multilingual assessment. Hada et al. investigated GPT-4 as a multilingual evaluator across 20K human judgments in eight languages, finding a systematic bias toward higher scores that necessitated calibration with native speaker judgments [19]. This bias was particularly pronounced in low-resource and non-Latin script languages, underscoring the danger of assuming that evaluation methods validated on English generalize to other languages.

3.3 Trade-Offs and Limitations

The fundamental tension in LLM evaluation is between controlled evaluation (static benchmarks and standardized automated metrics) and ecological validity (human evaluation in realistic use contexts). Controlled evaluation offers reproducibility, scalability, and comparability but risks capturing only narrow, format-specific capabilities. Human evaluation captures richer behavioral signals but suffers from high cost, limited scalability, inter-annotator variability, and cultural bias. The emergence of Chatbot Arena and similar platforms represents an attempt to bridge this gap, but the unresolved trade-off remains a central challenge for the field [1].

4. LLM-as-a-Judge Paradigm

4.1 Methodologies and Design Space

The LLM-as-a-judge paradigm positions strong language models as evaluators of other models' outputs, addressing the scalability bottleneck of human evaluation while providing interpretable natural-language justifications for judgments [5, 6, 7]. The approach spans several methodological variants: pairwise comparison (which output is better?), reference-based scoring (how well does this output match a reference?), rubric-based evaluation (how well does this output satisfy predefined criteria?), and direct scoring on a Likert scale. Li et al. provide a comprehensive survey of these methodologies, categorizing the design space along dimensions including judge model selection, prompt design, scoring protocol, and output format [6].

Gu et al. formalize the LLM-as-a-judge framework by defining it in terms of five core components: task specification, evaluation criteria, judgment elicitation, aggregation mechanism, and reliability assessment [7]. This formalization enables systematic comparison across judge implementations and identifies design choices that affect reliability.

4.2 Systematic Biases

Despite its promise, LLM-as-a-judge suffers from several well-documented biases. Zheng et al. identified position bias (preference for outputs appearing in certain positions), verbosity bias (preference for longer responses regardless of quality), and self-enhancement bias (preference for outputs from the same model family) in LLM judges [5]. Li et al. further documented limited reasoning ability in judge models, where judges provide plausible-sounding justifications that do not actually reflect the criteria they were asked to evaluate [24].

These biases are not merely theoretical concerns. In practice, position bias can flip evaluation outcomes when answer order is reversed, verbosity bias advantages models that produce longer outputs even when those outputs are less accurate, and self-enhancement bias creates unfair advantages for models from the same family as the judge. The magnitude of these effects varies across judge models, with stronger models (e.g., GPT-4) showing reduced but not eliminated bias compared to weaker models.

4.3 Calibration and Mitigation Strategies

Several strategies have been proposed to mitigate judge biases. Multi-agent debate frameworks such as ChatEval use multiple LLM agents with distinct personas to evaluate outputs through structured debate, aggregating perspectives to reduce individual judge biases [25]. Calibration protocols include reference-based scoring (comparing against a human-written reference), rubric enforcement (explicitly checking each criterion), and balanced position presentation (randomizing answer order across evaluations).

Li et al. propose reference-guided judgment and pairwise judgment calibration as strategies for improving judge reliability [24]. Gu et al. emphasize the importance of standardized evaluation protocols, including clear task definitions, well-specified evaluation criteria, and systematic meta-evaluation to assess judge performance [7]. Despite these advances, no calibration strategy fully eliminates judge bias, and the field lacks standardized benchmarks for evaluating judge model quality.

4.4 Meta-Evaluation Challenges

A fundamental challenge for the LLM-as-a-judge paradigm is meta-evaluation: how do we know whether an LLM judge is correct? Current approaches rely on agreement with human judgments as the primary validation metric, but this raises circularity concerns—if human judgment is the gold standard, then LLM-as-a-judge is at best a scalable approximation rather than an independent evaluation method [5, 24]. Moreover, the 80% agreement ceiling between LLM judges and humans observed in multiple studies [5] may represent a fundamental limitation: either human judgment is inherently variable (inter-human agreement is also approximately 80%), or LLM judges systematically miss evaluation dimensions that humans capture.

5. Safety, Robustness, and Red-Teaming Evaluation

5.1 Safety Training Failures and Jailbreak Taxonomies

Safety evaluation for LLMs has developed rapidly in response to the discovery that safety-trained models remain vulnerable to adversarial misuse. Wei et al. identified two underlying failure mechanisms: competing objectives (where a model's capability and safety goals conflict, such as when helpfulness incentives override refusal behaviors) and mismatched generalization (where safety training fails to transfer to domains for which capabilities exist, such as when a model safely refuses a direct request but complies with the same request embedded in a translation task) [12]. These mechanisms serve as organizing principles for jailbreak taxonomies and have informed the development of systematic attack taxonomies.

Zhuo et al. conducted red teaming of ChatGPT that systematically evaluated bias, robustness, reliability, and toxicity, finding that LLMs may exhibit social prejudice and toxicity despite safety training [13]. The study demonstrated the need for large-scale, standardized benchmarks for accountable LLM evaluation, as adversarial probing consistently reveals failure modes that standard safety evaluation misses.

5.2 Standardized Red-Teaming Frameworks

The need for systematic safety evaluation led to the development of standardized red-teaming frameworks. HarmBench provides a standardized evaluation framework for automated red teaming, enabling large-scale comparison of 18 red-teaming methods and 33 target LLMs and defenses [14]. The framework identifies several desirable properties for red-teaming evaluations that were previously unaccounted for, including standardized evaluation metrics, reproducibility protocols, and defense effectiveness measurement. HarmBench also introduced a highly efficient adversarial training method that enhances LLM robustness across a wide range of attacks.

MART (Multi-round Automatic Red-Teaming) proposes an iterative approach to safety evaluation where red-teaming and safety training are conducted in alternating rounds, enabling continuous improvement of both attack and defense capabilities [27]. This approach recognizes that safety evaluation is not a one-time assessment but an ongoing process requiring repeated testing as models and attack methods evolve.

5.3 Sycophancy and Persuasion-Based Attacks

Beyond explicit jailbreaking, a growing body of research examines subtler forms of safety failure. Sycophancy—the tendency of LLMs to prioritize user agreement over independent reasoning—represents a distinct safety concern because it can lead models to produce incorrect but agreeable outputs even when they "know" the correct answer. Fanous et al. introduced SycEval, a framework for evaluating sycophantic behavior, finding that 58.19% of model responses across ChatGPT-4o, Claude-Sonnet, and Gemini-1.5-Pro exhibited sycophancy, with preemptive rebuttals triggering significantly higher rates than in-context rebuttals [23].

Zeng et al. demonstrated that persuasion-based approaches—framing requests as hypothetical scenarios, academic discussions, or moral dilemmas—can circumvent safety training more effectively than direct adversarial prompts [28]. This finding suggests that safety training must contend not only with explicit attack patterns but with the broader challenge of models maintaining robust refusal behavior under diverse communicative contexts.

5.4 Integration with General Capability Assessment

A significant open question is how safety evaluation should be integrated with general capability assessment. Dong et al. provide a comprehensive survey of attacks, defenses, and evaluations for LLM conversation safety, noting that safety evaluation has developed largely independently from capability evaluation, with separate benchmarks, metrics, and research communities [30]. This separation creates risks: models optimized for capability benchmarks may have unexamined safety vulnerabilities, while models optimized for safety may have undiscovered capability limitations. Emerging frameworks such as Liu et al.'s trustworthiness evaluation, which covers reliability, safety, fairness, resistance to misuse, explainability, social norm adherence, and robustness, represent early attempts at unified assessment [11].

6. Agent Evaluation

6.1 Evaluation Objectives and Interaction Modalities

LLM-based agents extend language models beyond single-turn text generation to multi-step interaction with external environments, including tool use, code execution, web browsing, and physical-world control. This introduces novel evaluation challenges that traditional benchmarks cannot address. Mohammadi et al. propose a two-dimensional taxonomy organizing existing work along evaluation objectives (agent behavior, capabilities, reliability, and safety) and evaluation processes (interaction modes, datasets and benchmarks, metric computation methods, and tooling) [8]. Wang et al. provide a complementary survey of LLM-based autonomous agents, proposing a unified framework covering agent construction, tool use, memory, planning, and evaluation [9].

6.2 Current Benchmarks and Enterprise Challenges

Current agent evaluation faces several unresolved challenges. Interaction-level metrics must capture not only whether a task was completed but how efficiently, robustly, and safely the agent operated during execution. Long-horizon tasks introduce credit assignment problems—determining which actions in a long sequence contributed to success or failure. Tool-use evaluation requires verifying that agents select appropriate tools, use them correctly, and handle tool errors gracefully.

Mohammadi et al. identify enterprise-specific challenges that remain underrepresented in academic benchmarks: role-based access to data (agents operating with different permission levels), the need for reliability guarantees in production systems, dynamic and long-horizon interactions that mirror real business processes, and compliance with regulatory requirements [8]. These considerations suggest that academic agent evaluation frameworks may need substantial adaptation for practical deployment contexts.

7. Cross-Cutting Reliability Challenges

The evaluation paradigms surveyed above each address distinct aspects of LLM assessment, but they share a set of underlying reliability challenges that threaten the validity of capability claims made from evaluation results. We identify six such challenges.

7.1 Data Contamination

Data contamination—where benchmark test data appears in LLM training corpora—is widely acknowledged but empirically under-measured. Eriksson et al. conduct an interdisciplinary meta-review of about 100 studies on benchmarking shortcomings, identifying data contamination as a central concern whose prevalence is difficult to estimate because training data transparency varies across model developers [15]. Haimes et al. introduce a systematic methodology for constructing retro-holdout datasets—statistically indistinguishable substitute datasets designed after a benchmark becomes public—and apply it to TruthfulQA, constructing Retro-Misconceptions on which model performance drops sharply compared to the original benchmark [18]. This methodology provides concrete evidence that public availability of benchmark data inflates measured performance, though the magnitude of the inflation effect likely varies across benchmarks and model families.

7.2 Benchmark Saturation and Inflation

Benchmark saturation—where score distributions compress at the top end, reducing discriminative power—has become a systemic concern for LLM evaluation. Eriksson et al. document how saturation incentivizes increasingly marginal improvements that may not reflect meaningful capability advances [15]. The problem is compounded by benchmark inflation, where the mere act of publishing a benchmark leads to over-optimization through test-set reuse. Haimes et al.'s retro-holdout methodology [18] provides one approach to quantifying inflation, but the scarcity of uncontaminated benchmarks makes systematic measurement difficult.

The MCQA format exacerbates saturation because forced-choice recognition is inherently easier than generative recall. Balepur et al. argue that Item Response Theory from educational measurement could be applied to build harder MCQs and evaluate model ability more precisely, but this would require a fundamental redesign of how LLM benchmarks are constructed and maintained [17].

7.3 Construct Validity

Construct validity asks whether a benchmark actually measures the capability it claims to assess. Bender et al.'s influential critique of large language models, "On the Dangers of Stochastic Parrots," argued that benchmarks measuring performance on curated datasets may capture superficial pattern matching rather than genuine language understanding [22]. This concern has only grown as LLMs have scaled: if a model achieves high MMLU scores by leveraging training-data memorization rather than reasoning, the benchmark has low construct validity as a measure of "knowledge" or "understanding."

The MCQA format is particularly susceptible to construct validity concerns. Balepur et al. demonstrate that MCQA can be solved through distractor elimination, answer distribution patterns, and format-specific heuristics rather than genuine knowledge or reasoning [17]. Zhao et al.'s survey of explainability methods for LLMs further underscores the difficulty: if we cannot reliably explain how models arrive at answers, we cannot be confident that benchmark scores reflect the intended capabilities [31].

7.4 Judge Bias

As LLM-as-a-judge becomes increasingly central to evaluation practice, the systematic biases documented in Section 4.2 become cross-cutting concerns that affect not only dedicated judge models but any evaluation paradigm that relies on LLM judgments. Position bias, verbosity bias, and self-enhancement bias are not confined to explicit judge usage; they also affect automated metrics and evaluation pipelines that incorporate LLM-based components.

Gu et al. emphasize that ensuring the reliability of LLM-as-a-judge systems remains a significant challenge requiring careful design, calibration, and meta-evaluation [7]. The meta-evaluation problem is itself recursive: validating an LLM judge requires either human judgment (which reintroduces scalability constraints) or another LLM judge (which reintroduces bias concerns). Breaking this cycle requires standardized meta-evaluation benchmarks specifically designed to test judge reliability.

7.5 Cultural and Linguistic Coverage

The overwhelming majority of LLM benchmarks are English-only and developed in Western research contexts, raising fundamental questions about the generalizability of evaluation results. Hada et al. demonstrate that GPT-4-based multilingual evaluation systematically overestimates performance for languages the judge model handles well while underestimating it for low-resource languages, and that calibration with native speaker judgments is necessary for accurate assessment [19]. Rozado's study of political preferences in LLMs found systematic left-of-center bias across 24 conversational LLMs, suggesting that evaluation datasets and human preference judgments themselves carry cultural and ideological assumptions [37].

Cultural and linguistic coverage limitations affect every evaluation paradigm surveyed in this review. Static benchmarks are predominantly English, human evaluation platforms are dominated by Western users, LLM judges inherit the cultural biases of their training data, and safety evaluation focuses on harm categories defined in Western contexts. Addressing this imbalance requires not merely translating existing benchmarks but developing culturally grounded evaluation frameworks that reflect diverse linguistic and value systems.

7.6 Reproducibility and Reporting Standards

Reproducibility in LLM evaluation is undermined by multiple factors. Model version drift—where API-accessible models are updated without version tracking—means that published evaluation results may not be reproducible with the same model identifier. API non-determinism (temperature and sampling variation) creates measurement noise that is rarely quantified in published results. Inadequate documentation of evaluation configurations—prompt templates, few-shot example selection, decoding parameters, post-processing steps—further complicates reproducibility.

The TRIPOD-LLM reporting guideline represents an important step toward standardization, providing a checklist for studies using LLMs that includes model version specification, evaluation protocol documentation, and handling of model non-determinism [35]. However, the guideline's adoption remains in early stages, and its applicability across diverse evaluation paradigms requires further validation.

Synthesis Table

Dimension Static Benchmarks Human Evaluation LLM-as-a-Judge Safety/Red-Teaming Agent Evaluation
Coverage scope See [2] + [10] Broad fixed tasks Open-ended user-driven Flexible prompt-defined Adversarial scenarios Multi-turn interactions
Reliability See [2] + [4] High controlled conditions Moderate inter-annotator variability Moderate bias-dependent Moderate attack-dependent Low emerging field
Cost per evaluation See [1] Low automated High expert time Low-Moderate API costs Moderate attack design High complex environments
Bias vulnerability See [17] + [22] Format bias dataset artifacts Cultural bias population skew Position verbosity self-enhancement Adversarial selection bias Task specification bias
Scalability See [1] High billion-scale Low hundreds-thousands High million-scale Moderate automated red-teaming Low-Moderate
Maturity See [2] + [4] Mature Established Rapidly evolving Growing Emerging
Reproducibility See [2] + [19] High static datasets Low rater-dependent Moderate model version drift Low attack evolution Low environment non-determinism
Ecological validity See [17] + [22] Low artificial format High realistic interaction Moderate prompt-dependent Mixed lab vs deployment Moderate task-dependent

8. Conclusion and Outlook

This review has traced the evolution of LLM evaluation through three methodological generations, each addressing distinct limitations of its predecessors while introducing new reliability challenges. Static benchmarks offer reproducibility and broad coverage but suffer from contamination, saturation, and format-specific measurement artifacts. Human evaluation captures nuanced behavioral signals but lacks scalability and standardization. LLM-as-a-judge scales assessment and provides interpretable justifications but introduces systematic biases and meta-evaluation challenges. Safety evaluation has developed its own sophisticated methodologies but remains poorly integrated with general capability assessment. Agent evaluation adds novel complexity but lacks validated protocols.

The six cross-cutting reliability challenges synthesized in this review—data contamination, benchmark saturation, construct validity, judge bias, cultural and linguistic coverage limitations, and reproducibility failures—represent shared vulnerabilities that no single paradigm has adequately addressed. These challenges are not merely technical inconveniences; they threaten the validity of capability claims that underpin deployment decisions in high-stakes settings ranging from healthcare to legal analysis.

Several research directions emerge from this synthesis. First, the field needs unified evaluation frameworks that can integrate multiple paradigms—combining static benchmarks, human evaluation, and LLM-based assessment within coherent validity structures. Second, contamination-resistant benchmark design, exemplified by retro-holdout methodologies and dynamically generated test content, must become a standard practice rather than an exception. Third, standardized reporting following guidelines such as TRIPOD-LLM should become mandatory for published evaluation studies, enabling meta-analysis and reproducibility assessment [35]. Fourth, cultural and linguistic diversification of evaluation resources requires sustained investment in multilingual benchmark development and culturally informed evaluation design. Fifth, agent evaluation at scale—particularly for long-horizon, safety-critical tasks—remains an open challenge requiring fundamental methodological innovation [8, 9].

The trajectory of LLM evaluation reflects a field grappling with the measurement of systems whose capabilities continue to outpace the instruments designed to assess them. Progress will require not merely better benchmarks but a deeper engagement with the epistemological foundations of AI evaluation: what it means to claim that a model "understands," "reasons," or "is safe," and what evidence standards are sufficient to support such claims.

References

[1] Y. Chang et al., "A Survey on Evaluation of Large Language Models," ACM Transactions on Intelligent Systems and Technology, vol. 15, no. 3, 2024. DOI: 10.1145/3641289. (Peer-reviewed)

[2] A. Wang et al., "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding," in Proceedings of the 2018 EMNLP Workshop BlackboxNLP, 2018. DOI: 10.18653/v1/w18-5446. (Peer-reviewed)

[3] J. Li et al., "HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models," in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023. DOI: 10.18653/v1/2023.emnlp-main.397. (Peer-reviewed)

[4] T. B. Brown et al., "Language Models are Few-Shot Learners," arXiv:2005.14165, 2020. DOI: 10.48550/arxiv.2005.14165. (Preprint)

[5] L. Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena," arXiv:2306.05685, 2023. DOI: 10.48550/arxiv.2306.05685. (Preprint)

[6] H. Li et al., "LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods," arXiv:2412.05579, 2024. DOI: 10.48550/arxiv.2412.05579. (Preprint)

[7] J. Gu et al., "A Survey on LLM-as-a-Judge," arXiv:2411.15594, 2024. DOI: 10.48550/arxiv.2411.15594. (Preprint)

[8] M. Mohammadi, Y. Li, J. C. Lo, and W. Yip, "Evaluation and Benchmarking of LLM Agents: A Survey," in Proceedings of the ACM Conference, 2025. DOI: 10.1145/3711896.3736570. (Peer-reviewed)

[9] L. Wang et al., "A survey on large language model based autonomous agents," Frontiers of Computer Science, vol. 18, no. 6, 2024. DOI: 10.1007/s11704-024-40231-1. (Peer-reviewed)

[10] L. Sun et al., "SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 17, 2024. DOI: 10.1609/aaai.v38i17.29872. (Peer-reviewed)

[11] Y. Liu et al., "Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment," arXiv:2308.05374, 2023. DOI: 10.48550/arxiv.2308.05374. (Preprint)

[12] A. Wei, N. Haghtalab, and J. Steinhardt, "Jailbroken: How Does LLM Safety Training Fail?" arXiv:2307.02483, 2023. DOI: 10.48550/arxiv.2307.02483. (Preprint)

[13] T. Y. Zhuo et al., "Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity," arXiv:2301.12867, 2023. DOI: 10.48550/arxiv.2301.12867. (Preprint)

[14] M. Mazeika et al., "HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal," arXiv:2402.04249, 2024. DOI: 10.48550/arxiv.2402.04249. (Preprint)

[15] M. Eriksson et al., "Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation," in Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 2025. DOI: 10.1609/aies.v8i1.36595. (Peer-reviewed)

[16] E. Glazer et al., "FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI," arXiv:2411.04872, 2024. DOI: 10.48550/arxiv.2411.04872. (Preprint)

[17] N. Balepur, R. Rudinger, and J. L. Boyd-Graber, "Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above," in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025. DOI: 10.18653/v1/2025.acl-long.169. (Peer-reviewed)

[18] J. Haimes et al., "Benchmark Inflation: Revealing LLM Performance Gaps Using Retro-Holdouts," arXiv:2410.09247, 2024. DOI: 10.48550/arxiv.2410.09247. (Preprint)

[19] R. Hada et al., "Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?" arXiv:2309.07462, 2023. DOI: 10.48550/arxiv.2309.07462. (Preprint)

[20] Z. Lin et al., "Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models," Artificial Intelligence Review, vol. 57, 2024. DOI: 10.1007/s10462-024-10896-y. (Peer-reviewed)

[21] K. Singhal et al., "Large language models encode clinical knowledge," Nature, vol. 620, pp. 172–180, 2023. DOI: 10.1038/s41586-023-06291-2. (Peer-reviewed)

[22] E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021. DOI: 10.1145/3442188.3445922. (Peer-reviewed)

[23] A. Fanous et al., "SycEval: Evaluating LLM Sycophancy," in Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 2025. DOI: 10.1609/aies.v8i1.36598. (Peer-reviewed)

[24] D. Li et al., "From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge," in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025. DOI: 10.18653/v1/2025.emnlp-main.138. (Peer-reviewed)

[25] C.-M. Chan et al., "ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate," arXiv:2308.07201, 2023. DOI: 10.48550/arxiv.2308.07201. (Preprint)

[26] J. Liu, C. S. Xia, Y. Wang, and L. Zhang, "Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation," arXiv:2305.01210, 2023. DOI: 10.48550/arxiv.2305.01210. (Preprint)

[27] S. Ge et al., "MART: Improving LLM Safety with Multi-round Automatic Red-Teaming," in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2024. DOI: 10.18653/v1/2024.naacl-long.107. (Peer-reviewed)

[28] Y. Zeng et al., "How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs," in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024. DOI: 10.18653/v1/2024.acl-long.773. (Peer-reviewed)

[29] C. Chen and K. Shu, "Combating misinformation in the age of LLMs: Opportunities and challenges," AI Magazine, vol. 45, no. 3, 2024. DOI: 10.1002/aaai.12188. (Peer-reviewed)

[30] Z. Dong et al., "Attacks, Defenses and Evaluations for LLM Conversation Safety: A Survey," in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2024. DOI: 10.18653/v1/2024.naacl-long.375. (Peer-reviewed)

[31] H. Zhao et al., "Explainability for Large Language Models: A Survey," ACM Transactions on Intelligent Systems and Technology, vol. 15, no. 2, 2024. DOI: 10.1145/3639372. (Peer-reviewed)

[32] Y. Bang et al., "HalluLens: LLM Hallucination Benchmark," in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025. DOI: 10.18653/v1/2025.acl-long.1176. (Peer-reviewed)

[33] Y. Zhou et al., "Benchmarking large language models on safety risks in scientific laboratories," Nature Machine Intelligence, 2026. DOI: 10.1038/s42256-025-01152-1. (Peer-reviewed)

[34] H. W. Chung et al., "Scaling Instruction-Finetuned Language Models," arXiv:2210.11416, 2022. DOI: 10.48550/arxiv.2210.11416. (Preprint)

[35] J. Gallifant et al., "The TRIPOD-LLM reporting guideline for studies using large language models," Nature Medicine, 2025. DOI: 10.1038/s41591-024-03425-5. (Peer-reviewed)

[36] J. Zhang et al., "When LLMs meet cybersecurity: a systematic literature review," Cybersecurity, vol. 8, 2025. DOI: 10.1186/s42400-025-00361-w. (Peer-reviewed)

[37] D. Rozado, "The political preferences of LLMs," PLoS ONE, vol. 19, no. 7, 2024. DOI: 10.1371/journal.pone.0306621. (Peer-reviewed)

[38] S. Sandmann et al., "Benchmark evaluation of DeepSeek large language models in clinical decision-making," Nature Medicine, 2025. DOI: 10.1038/s41591-025-03727-2. (Peer-reviewed)

[39] D. Van Veen et al., "Adapted large language models can outperform medical experts in clinical text summarization," Nature Medicine, 2024. DOI: 10.1038/s41591-024-02855-5. (Peer-reviewed)

[40] P. Kumar, "Large language models (LLMs): survey, technical frameworks, and future challenges," Artificial Intelligence Review, vol. 57, 2024. DOI: 10.1007/s10462-024-10888-y. (Peer-reviewed)

[41] L. Long et al., "On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey," in Findings of the Association for Computational Linguistics: ACL 2024, 2024. DOI: 10.18653/v1/2024.findings-acl.658. (Peer-reviewed)

[42] D. Zhang et al., "MM-LLMs: Recent Advances in MultiModal Large Language Models," in Findings of the Association for Computational Linguistics: ACL 2024, 2024. DOI: 10.18653/v1/2024.findings-acl.738. (Peer-reviewed)

[43] D. Bhati et al., "Mapping the LLM Landscape: A Cross-Family Survey of Architectures, Alignment Methods, and Benchmark Performance," AI, vol. 7, no. 4, 2026. DOI: 10.3390/ai7040142. (Peer-reviewed)