G-Eval: How LLM-as-a-Judge Scoring Actually Works

G-Eval scores generated text using a large language model as the judge, through chain-of-thought prompting and a form-filling format. It needs no reference answer, which is why teams reach for it when no gold-standard output exists to compare against. It reports a Spearman correlation of 0.514 with human ratings on summarisation.

Last updated: August 2026 By Atul Jain

That number is the reason G-Eval spread, and it is also the most misread figure in evaluation tooling. It is a moderate correlation, not a solved problem, and the two mechanisms the method is famous for turn out to contribute very little of it.

This explainer covers what G-Eval does, what its published results actually say, where the method breaks, and when a simpler check is the better choice.

What is G-Eval?

G-Eval was introduced in G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment, published at EMNLP 2023 by Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu and Chenguang Zhu at Microsoft Cognitive Services Research.

It is a reference-free metric. Older automatic metrics such as BLEU and ROUGE score an output by its overlap with a reference answer, which means someone has to write the reference first, and a correct answer phrased differently scores badly. The paper's own framing is that these conventional reference-based metrics "have been shown to have relatively low correlation with human judgments, especially for tasks that require creativity and diversity."

G-Eval removes the reference. You supply a task description and an evaluation criterion in plain language, and the judge model produces a score. That makes it task-agnostic: the same machinery evaluates coherence, groundedness, tone or safety depending only on the criterion you write.

How does G-Eval work?

Three steps.

1. Generate the evaluation steps. You write a criterion in natural language, for example "Coherence (1-5): the collective quality of all sentences." The model expands that into a numbered list of evaluation steps through chain-of-thought prompting, rather than being handed a rubric by a human.

2. Fill in the form. The generated steps, the criterion, the input context and the output being judged all go into one prompt, which ends with a scoring form the model completes. This is the "form-filling paradigm."

3. Weight the score by token probability. Here the paper solves a real problem. Asking a model for a 1-5 score produces clustering, because one digit dominates the distribution, and models emit integers even when asked for decimals, which "leads to many ties in evaluation scores which do not capture the subtle difference between generated texts."

The fix is to take the probability the model assigns to each possible score and compute a weighted sum, so a judge that is 60% confident in 4 and 40% confident in 3 returns 3.6 rather than 4. That produces continuous scores instead of a pile of 3s.

There is a practical catch worth knowing before you build on this. At the time of the paper, GPT-4 did not expose token probabilities at all, so the authors sampled the model 20 times at temperature 1 to estimate them. The elegant scoring mathematics required an API feature the flagship model did not provide, and the workaround multiplied the cost of every judgment by twenty.

How well does G-Eval correlate with human judgment?

The headline figure is the average across four dimensions on the SummEval benchmark, using GPT-4 as the judge. Here is the full row, alongside the baselines the paper compares against.

Summary-level Spearman correlations on SummEval, from Table 1 of the paper.

Two things are worth reading off that table that the surrounding commentary usually skips.

First, 0.514 is moderate agreement, not agreement. It was the best result available at the time, and it still means the judge and the human ranked outputs differently a great deal of the time. Treat a G-Eval score as a signal that moves with quality, not as a verdict.

Second, UniEval reached 0.474 without a frontier model at all. A purpose-built evaluator got within four points of GPT-4 for a fraction of the inference cost. If your criterion is a standard one, a smaller specialised model may be the better trade.

On the Topical-Chat dialogue benchmark the gap over older metrics is wider. G-Eval with GPT-4 averaged 0.575 Spearman against 0.262 for BERTScore, 0.189 for BLEU-4 and 0.243 for ROUGE-L. Notably, G-Eval with GPT-3.5 averaged 0.574 on the same benchmark, effectively tied with the GPT-4 version, and scored higher on engagingness and groundedness. The bigger judge is not uniformly the better judge.

What the ablations show about chain-of-thought and token probabilities

This is the part of the paper that most secondhand explanations leave incomplete, and it is the most useful part.

The paper ablates both signature mechanisms:

Removing chain-of-thought costs 0.014. Removing probability weighting costs 0.012. The two mechanisms G-Eval is named for, and which most explanations of it lead with, account for a combined couple of points of correlation.

The overwhelming majority of the result comes from using GPT-4 rather than GPT-3.5 as the judge: 0.401 to 0.514, a gain of 0.113, roughly eight times either mechanism's contribution.

The practical reading is direct. If you are choosing where to spend effort on an LLM judge, judge model capability and criterion quality dominate. Elaborate chain-of-thought scaffolding and logprob arithmetic are refinements, and if your provider does not expose token probabilities you are giving up about 0.012 of correlation rather than the core of the method.

Where G-Eval breaks

Three failure modes, all documented.

It prefers machine-written text. The paper's own analysis is unambiguous. Comparing human-written summaries against GPT-3.5 summaries, "G-Eval-4 always gives higher scores to GPT-3.5 summaries than human-written summaries, even when human judges prefer human-written summaries." The abstract raises this directly as a concern about LLM-based evaluators carrying "a bias towards the LLM-generated texts."

This matters most when the judge and the system under test share a family. Using GPT-4 to grade GPT-4 output builds a preference for the model's own style into your metric, and that preference points away from human judgment rather than toward it.

It has position bias in comparisons. Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge evaluated 15 LLM judges across MTBench and DevBench, covering 22 tasks and roughly 40 solution-generating models, producing over 150,000 evaluation instances. The authors report that position bias is not due to random chance, varies significantly across judges and tasks, and is strongly affected by the quality gap between the solutions compared. Judges are least reliable when two candidates are close in quality, which is the comparison you usually need. How exploitable that is has been shown directly: Large Language Models are not Fair Evaluators, published at ACL 2024, reports that "the quality ranking of candidate responses can be easily hacked by simply altering their order of appearance in the context," and that under such a reordering "Vicuna-13B could beat ChatGPT on 66 over 80 tested queries with ChatGPT as an evaluator." If reordering can flip 66 of 80 verdicts, an uncalibrated pairwise score is not a measurement. Run pairwise comparisons in both orders.

Scores drift when anything upstream changes. The score depends on the judge model, the judge prompt and the generated evaluation steps. Change any one without versioning it and a score movement becomes uninterpretable, because you cannot tell whether the application got worse or the ruler changed. Pin all three.

When to use G-Eval, and when not to

Reach for it when no reference answer exists and the quality you care about is genuinely subjective: coherence, tone, helpfulness, whether an answer is grounded in retrieved context.

Do not reach for it when a deterministic check would settle the question. Schema validity, the presence of a required disclosure, a numeric tolerance, a latency budget: these are all free, instant and reproducible in code. A model-graded check costs an inference per case and introduces a component that itself has bias. Teams routinely spend a judge call on something a regex answers, and then spend a second one debugging why the judge is inconsistent.

The general trade-offs of the judge pattern, beyond G-Eval specifically, are covered in LLM as a Judge: How It Works, Pros, Cons, and Best Practices. For grader types that do not need a judge at all, OpenAI's evals documentation describes deterministic options such as string_check, though note that page carries a deprecation notice: read on 14 August 2026, the platform goes read-only on 31 October 2026 and shuts down on 30 November 2026.

Running G-Eval on conversational and voice agents

G-Eval was built and measured on single-output tasks: one summary, one dialogue response. A production conversational agent is neither. It is a multi-turn exchange where the failure often lies in the sequence rather than any single utterance, and on a voice agent it also carries speech recognition, turn-taking and a telephony leg that a text judge never sees.

Cekura applies criterion-based scoring across the whole conversation rather than a single response. Cekura runs configurable evaluators per turn and per call, so a criterion written the way a G-Eval criterion is written can be scored against a full simulated conversation instead of an isolated output. Cekura documents the underlying metric set in A Developer's Guide to Voice AI Evaluation Metrics, and the regression workflow that replays real production calls against a new model version in Test New Model Versions with Real Production Calls Using Cekura.

The repeat discipline matters more here than in text evaluation, because conversational failures are intermittent. Cekura's benchmarks tested six voice orchestration platforms with 59 evaluators across four categories, running each scenario three times and scoring on pass^3, so a scenario counts as passed only when all three runs pass. Retell scored 96.6%, Vapi 94.9%, Pipecat 89.8%, LiveKit 84.7%, Synthflow 81.4% and ElevenLabs 76.3%.

Two caveats travel with those figures wherever they appear. Pass^3 is deliberately harsher than a single-run pass rate, so the two are not comparable. And the language model was held constant at gpt-4.1 at temperature 0 across all six. Speech recognition was pinned to Deepgram nova-3 on Vapi, Synthflow, LiveKit and Pipecat only, because Retell exposes only a coarse mode and ElevenLabs forces its own Scribe, so recognition is an uncontrolled variable on those two. Most of the spread therefore describes orchestration behaviour rather than model choice.

The connection to G-Eval is the sampling logic. G-Eval sampled 20 times to estimate a score distribution because one judgment was too noisy to trust. Requiring three consecutive passes is the same instinct applied to a whole conversation.

Frequently asked questions

What is G-Eval?

G-Eval is a reference-free evaluation framework that scores generated text using a large language model as the judge, combining chain-of-thought generation of evaluation steps with a form-filling prompt and probability-weighted scoring. It was published at EMNLP 2023 and needs no gold-standard answer to compare against.

What does the G-Eval score of 0.514 mean?

It is the average summary-level Spearman correlation between G-Eval with GPT-4 and human ratings across four SummEval dimensions: coherence 0.582, consistency 0.507, fluency 0.455 and relevance 0.547. It was the strongest published result at the time, and it describes moderate agreement with human judgment rather than a match.

Is G-Eval the same as LLM-as-a-judge?

No. LLM-as-a-judge is the general pattern of using one model to score another's output. G-Eval is one specific implementation of that pattern, defined by its auto-generated evaluation steps and its probability-weighted scoring.

Does G-Eval need token log probabilities?

It is designed around them, but it degrades gently without them. The paper's ablation shows removing probability weighting moves the average correlation from 0.514 to 0.502. GPT-4 did not expose token probabilities when the paper was written, so the authors sampled the model 20 times at temperature 1 to estimate the distribution instead.

What is the biggest weakness of G-Eval?

Its preference for machine-written text. The paper reports that G-Eval with GPT-4 always scored GPT-3.5 summaries above human-written ones, even when human judges preferred the human-written summaries. Using a judge from the same model family as the system under test amplifies this.

Do the Cekura benchmark numbers apply to text evaluation?

Not directly. The pass^3 rates from 96.6% for Retell down to 76.3% for ElevenLabs measure voice orchestration platforms with the language model held constant at gpt-4.1 at temperature 0 across all six, though speech recognition was pinned on only four of them, so they describe orchestration behaviour rather than model quality, and pass^3 is harsher than a single-run pass rate. The transferable idea is requiring repeated passes before a result counts.

Cekura scores voice and chat agents against criteria you define, across whole conversations rather than single outputs. Book a demo to see it run against your own agent.