Chapter 7 — Using a Model to Score Model Output

Course: Build and Evaluate RAG Applications

Prerequisites: You can read and run a short Python script that loops over a list and counts matches. You know what an evaluation case, a grader, a pass rate and a regression are: a case is an input with an expected result and a rule that decides pass or fail, a grader is the procedure that applies the rule, and a regression is a case that passed before a change and fails after it. The notes restate each of these in one line before using it.

What the reader can do after this chapter:

• Write a rubric for a model grader that asks yes/no questions and returns a single parseable verdict, and say which cases need a model grader rather than code.
• Measure a model grader against labels made by people, using raw agreement and a chance-corrected agreement score, and list the cases where the grader passed something a person failed.
• Detect position bias in a pairwise grader by running each pair in both orders, and decide from the result whether the grader may be trusted without a person checking it.

Core question this chapter answers: If a second model scores your feature’s output, how do you find out whether its scores can be trusted before you let them decide whether a change ships?

Connections:

• Builds on: the evaluation suite, where each case has a grader and a written gate rule reads the results; this chapter takes the cases that code cannot grade.
• Leads into: Chapter 8, next in this course, which applies the same measure-against-people habit to one specific failure: statements in an output that nothing in the supplied material supports.

Your answers are scored

This chapter is assessed. The quiz is marked against a key held separately from your materials. Questions 1–8 are multiple choice and are marked by comparing your letter with the key. Questions 9–10 ask for a short answer in an exact format, and each one is marked by a rule printed with the question.

• Nothing is judged by reading it for style. An answer either matches the rule or it does not.
• All three project tasks are scored from an evidence file of key: value lines, plus one rubric file whose structure is checked by a script. Every accepted value, or the rule that decides it, follows from what the task prints.

Build and Evaluate RAG Applications — Chapter 7: Using a Model to Score Model Output

Before you start

Nothing to install for the reading. For the project you need Python 3 and nothing beyond its standard library. You do not need an account with any model provider or an API key: every judge verdict the project uses is printed in the project file.

Keep one question in mind while you read: if a model says “this reply is good”, what do I know about whether it is right?

Summary

A model grader, often called an LLM-as-a-judge, is a second model that reads an output and a written rubric and returns a verdict or a score. It exists for the cases code cannot grade, such as whether a reply acknowledges a customer’s problem before stating policy. It is fast and scales, but it is a measuring instrument with known faults: it can favour whichever answer is listed first, favour the longer answer, and favour answers written by a model like itself. Because of those faults a judge is never trusted on its own say-so. It is checked against labels made by people, using agreement that is corrected for chance, and a pairwise judge is checked by swapping the order. Only a judge that passes those checks is allowed to decide whether a change ships.

Key concepts

Term	Definition
Model grader (LLM-as-a-judge)	A model asked to grade an output against a rubric, returning a verdict such as pass or fail, or a score.
Rubric	The written criteria the grader applies. A good one is specific and asks questions that can be answered yes or no.
Single-answer grading	The grader sees one output on its own and scores it.
Pairwise comparison	The grader sees two outputs, A and B, and says which is better by stated criteria.
Reference-guided grading	The grader is also given a reference (“gold”) answer to compare the output against.
Position bias	A tendency to prefer an answer because of where it is listed, not because of its quality.
Verbosity bias	A tendency to prefer the longer answer.
Self-enhancement bias	A tendency of a model to prefer answers written by itself or a similar model.
Human label	The pass or fail a person gives a case. In this chapter it is the standard the judge is measured against.
Agreement	The share of cases where the judge’s verdict equals the human label, from 0 to 1.
Cohen’s kappa	An agreement score corrected for chance: (observed agreement − chance agreement) ÷ (1 − chance agreement). 1 is perfect agreement, 0 means observed agreement equals chance agreement, and a negative value means observed agreement is below what chance would give.
False pass	A case the judge passed and a person failed.

Explanations

The failure, before the explanation

A team builds a reply drafter for a support desk. Code can check some things: no internal ticket number appears, the reply is under 120 words. It cannot check the thing that matters most to customers: does the reply acknowledge the problem before it states the policy? So the team asks a model: “Is this a good reply? Answer 1 to 10.” The scores come back averaging 8.4, the new prompt averages 8.6, and the change ships.

A week later a support lead reads forty replies by hand and finds that a third open with the policy and never mention the customer’s problem. The judge had scored them 8 and 9. It had never been told what “good” meant, and nobody had ever compared its scores with what a person would say. The instrument was untested, and its output was treated as a measurement.

What a model grader is for

Start with an example. Three kinds of grader can be ranked by cost: code, then a model, then a person. One provider lists LLM-graded questions beside multiple-choice, string-match and code-graded ones as ways to “allow for automated grading” (Anthropic, Define success criteria and build evaluations, retrieved 2026-10-04), and another says to validate a judge “against your human labels before optimizing for cost or latency” (OpenAI, Evaluation best practices, retrieved 2026-10-04). That second half is this whole unit.

A second way to see it: a model grader is a junior reviewer you hired to read ten thousand replies. It is quick and never tired, but before you let it approve anything you read a sample of what it approved and compare.

• Use it when the property is real but cannot be written as a string or pattern rule: tone, whether a summary keeps the main point, whether an answer addresses the question.
• Do not use it when code can see the defect. A forbidden pattern, a length limit and a valid label are checked by code, which is cheaper and does not drift.
• What changes in your system: the grader becomes a component with its own prompt, its own cost and its own failure modes, and its prompt needs the same version control as the feature.
• Versus the alternative: a person is more reliable per case but slow; a model grader is the option when the volume makes a person impractical, provided it has been checked against one.

Three ways to ask the judge

A support-desk example, three framings of the same question:

• Single-answer grading: “Here is one reply. Does it acknowledge the problem? Answer pass or fail.”
• Pairwise comparison: “Here are replies A and B. Which acknowledges the problem better?”
• Reference-guided grading: “Here is a reply and a model reply a person wrote. Does the reply contain what the model reply contains?”

One provider lists all three and adds: “Use pairwise comparison or pass/fail for more reliability” (OpenAI, Evaluation best practices, retrieved 2026-10-04). A second way to picture the choice: a 1-to-10 score is a thermometer with no markings, so two raters read it differently; a pass/fail question or a head-to-head comparison is a checkbox, and checkboxes are more repeatable.

• Use pairwise when you are choosing between two versions of a prompt. Use single-answer pass/fail when you are running a gate, because the gate needs a yes or no per case.
• Practical consequence: pairwise results can only say which of two is better; they cannot say that both are bad. A gate needs an absolute verdict.

Writing the rubric

An example of a weak rubric: “Is this a good reply? 1 to 10”. An example of a stronger one:

You are grading a customer-support reply. Answer each question yes or no.

1. Does the first sentence name the customer’s problem?
2. Is the policy stated after that, not before it?
3. Does the reply avoid promising anything the policy does not say? Give one sentence of reasoning, then on a new line write VERDICT: pass if all three answers are yes, otherwise VERDICT: fail.

Two published instructions behind this: ask for clear, detailed rubrics (“Ensure eval rubrics are clear and detailed”, OpenAI, Evaluation best practices), and ask for a constrained output: one provider’s example grader prompts end with “Output only the number” (Anthropic, Define success criteria and build evaluations, retrieved 2026-10-04). The same OpenAI page recommends to “Add reasoning and chain-of-thought as reasoning before scoring improves eval performance”, so the stronger rubric above asks for a sentence of reasoning first and then a fixed final line your code can parse.

A second way to see it: a rubric is a test script for a person who cannot ask you questions. Anything you leave to the reader’s taste is a place where two readers, or two runs, will differ.

• Applied use: parse only the final VERDICT: line in code, so the grader’s reasoning cannot break your pipeline.
• Selection: use yes/no sub-questions when you can name the criteria. If you cannot name them, you have not yet decided what good means, and a more elaborate judge will not fix that.

The three biases, each shown before it is named

Position. Suppose you give a pairwise judge replies A and B, then the same two with the order swapped. If it picks “the first one” both times, it chose by slot, not by quality. This is position bias. Second picture: a quiz where the correct answer is always listed first teaches test-takers nothing about the subject.

Length. Suppose reply B is the same advice as A with two extra sentences of filler, and the judge prefers B. This is verbosity bias. OpenAI’s page lists “Position bias (response order), verbosity bias (preferring longer responses)” and advises: “Control for response lengths as LLMs bias towards longer responses” (retrieved 2026-10-04).

Self-enhancement. Suppose the model that wrote the replies also judges them and rates its own style higher. The research that named these three biases lists “position, verbosity, and self-enhancement biases, as well as limited reasoning ability” (Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, arXiv 2306.05685, retrieved 2026-10-04). One provider’s practical advice: “Generally best practice to use a different model to evaluate than the model used to generate the evaluated output” (Anthropic, Define success criteria and build evaluations, retrieved 2026-10-04).

Bias	How to see it	What to do
Position	Run each pair in both orders; count verdicts that change when only the order changes	Run both orders and count a result only when both agree
Verbosity	The winner is systematically the longer reply	Compare replies of similar length, or add a length rule to the rubric
Self-enhancement	The judge favours its own model’s replies	Judge with a different model than the one that wrote the output

A caution: these are tendencies reported for judges in the research and the provider guidance cited here. They do not say every judge shows every bias to the same degree. That is why you measure your own judge on your own cases instead of assuming either way.

Calibrating against people

Here is the measurement. Take a set of cases, have a person label each pass or fail, and run the judge on the same cases.

Case	Human	Judge	Same?
a	pass	pass	yes
b	fail	pass	no — false pass
c	fail	fail	yes
d	pass	fail	no

Raw agreement is the share of matching rows: 2 of 4 = 0.500.

Raw agreement can mislead. A second picture: if 90 of 100 replies are good, a judge that stamps “pass” on everything agrees with people 90 times and has learned nothing. Cohen’s kappa removes the part of agreement that chance would give you: the scikit-learn documentation describes it as “a score that expresses the level of agreement between two annotators on a classification problem”, with the formula (p_o − p_e) / (1 − p_e), where p_o is the observed agreement and p_e the agreement expected by chance, and “0.0 indicates no agreement beyond what would be expected by chance” (retrieved 2026-10-04). The same page gives the range as −1.0 to 1.0. In the formula, kappa is 0 when po equals pe, and negative when po is below pe. For two verdicts:

1. Observed agreement po = the share of matching rows.
2. Chance agreement pe = (human pass share × judge pass share) + (human fail share × judge fail share).
3. kappa = (po − pe) ÷ (1 − pe).

For the four-row table, human pass share is 0.5 and judge pass share is 0.5, so pe = 0.25 + 0.25 = 0.50 and kappa = (0.50 − 0.50) ÷ 0.50 = 0.000. Half the rows matched, which is the agreement that two coin-flippers with those pass shares would be expected to reach. For the “stamps pass on everything” judge, pe equals po, and kappa is 0.

• Applied use: report agreement, kappa and the list of false passes together. A false pass is the expensive error: it lets a bad output through the gate.
• Practical consequence: write the trust rule before you measure, for example “use the judge unattended only if agreement is at least 0.80 and kappa is at least 0.60”. A threshold chosen after you see the number is not a check. The thresholds in this chapter are the chapter’s own illustration, not a published standard.
• Selection: if the judge fails the rule, the fix is the rubric or the judge, then measure again on freshly labelled cases, because a rubric edited to fix the cases it failed will look better on exactly those cases. Lowering the threshold or reading fewer cases is not a fix.

What does the research say about numbers? The research cited above (Zheng et al.) reports that “strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans” (retrieved 2026-10-04). That is a result for that study’s judges and tasks. For your judge on your cases the number is whatever you measure. OpenAI’s guidance is to validate the judge “against your human labels before optimizing for cost or latency”, and to scale up once the judge “is faster, cheaper, and consistently agrees with human annotations” (retrieved 2026-10-04).

Three situations and what they mean for your design

1. A team replaces a person who read a sample of replies on a regular schedule with a judge model, and the reported score jumps from 7.1 to 8.8 with no change to the feature.
2. A comparison of two prompts using a pairwise judge says version two wins 70 of 100 pairs. Every pair was shown with version two second.
3. A team uses the same model for the feature and the judge, and the judge’s scores track the feature’s cost per call more than they track customer complaints.

What you see	What it means	What to do instead
A jump in score with no change to the feature (1)	The instrument changed, not the product	Re-measure the judge against fresh human labels; keep a fixed labelled set to re-run
Always the same slot (2)	The result may be position bias	Swap the order for every pair and count only results that agree
Judge and feature are one model (3)	Self-enhancement is a risk, and cost is not quality	Judge with a different model, and compare to people

Flashcards

Say the answer aloud before revealing it. Speaking it is what exposes the gaps that reading past a written answer hides.

When do you use a model grader instead of code?

When the property is real but no string or pattern rule can check it, such as tone or whether an answer addresses the question. If code can see the defect, code is cheaper and steadier.

Name the three ways to ask a judge.

Single-answer grading (one output scored alone), pairwise comparison (two outputs, which is better) and reference-guided grading (the output compared with a reference answer).

Why prefer pass/fail questions to a 1-to-10 score?

A pass/fail question on a named criterion is more repeatable. The provider guidance cited in the notes recommends pairwise comparison or pass/fail for reliability.

What is position bias and how do you test for it?

Preferring an answer by where it is listed. Run each pair in both orders and count the pairs where the verdict changes when only the order changed.

What is verbosity bias and what is one response to it?

A tendency to prefer the longer answer. Compare replies of similar length, or add a length rule to the rubric, and check whether winners are systematically longer.

Why use a different model to judge than the one that wrote the output?

A model can favour its own style (self-enhancement), and the provider guidance cited says that using a different model is the general best practice.

Why is 90% agreement not enough on its own?

If most cases are pass, a judge that always says pass agrees that often. Kappa removes the agreement that chance gives, and the false-pass list shows which bad outputs got through.

How is Cohen's kappa computed from two verdict columns?

Find observed agreement, find chance agreement from each column’s pass share, then (observed − chance) ÷ (1 − chance). It is 0 when observed agreement equals chance agreement, and negative when observed agreement is below chance agreement (the numerator is then negative).

A judge's agreement with people is 0.75 and the rule says 0.80. What do you do?

Do not use it unattended. Look at the false passes, tighten the rubric or change the judge, and measure again on freshly labelled cases.

Why have the judge give reasoning before the verdict?

The guidance cited says reasoning before scoring improves eval performance. Parse only a fixed final line, so the reasoning cannot break your code.

Model answer

The question: “You want a model to grade your assistant’s replies in the pipeline. How do you know you can trust it?” A strong answer hits five beats, in this order:

1. Rubric — specific yes/no criteria and a fixed final verdict line.
2. Biases — position, length and self-preference named, each with its check.
3. Labels — a set of cases labelled by people, with the judge run on the same cases.
4. Agreement — raw agreement, chance-corrected agreement and the false passes.
5. Threshold — a written trust rule, set beforehand, and a re-check when the judge or rubric changes.

“First I’d write the rubric as yes/no questions with reasoning first and a fixed final line, so the verdict is parseable. I’d know it has three typical biases: it may favour the first answer, the longer one, or its own model’s style, so pairwise runs go in both orders, I compare similar-length replies, and I judge with a different model than the one that wrote the output. Then I’d have a person label a set of cases, run the judge on the same ones, and report agreement, kappa and the list of false passes, because agreement alone can be high when most cases pass. I’d set the trust rule before looking, say at least 0.80 agreement and 0.60 kappa, and if the judge misses it I fix the rubric and measure again on fresh labelled cases. When the prompt or judge model changes I re-run a labelled set.”

Missing a beat is a failure state. Without a rubric the judge applies its own taste. Without naming biases you will not look for them. Without labels there is nothing to compare with. Without agreement the labels are not used. Without a threshold any number can be called good enough.

Why this matters

• Gating changes on quality that code cannot see. A judge that passes the checks lets tone and relevance be part of the gate.
• Choosing between prompts. A pairwise judge run in both orders gives a result that is not just a position artefact.
• Cost pressure. “Replace the reviewers with the judge” is a decision to make on measured agreement, not on a hunch.
• Audits. A customer or a regulator asking how scores were produced is answered with the rubric, the labelled set and the agreement figures.

After this chapter

The project for this chapter is Calibrate a Judge (project.md). You will measure a judge against human labels, test a pairwise judge for position bias, and write a rubric file whose structure a script checks.

Chapter 8, next in this course, narrows the question to one failure: statements in an output that the supplied material does not support, and how to catch them before a user does.

Hands-on Project — Calibrate a Judge

Objective

Measure a model grader against human labels with an agreement score corrected for chance, test a pairwise grader for position bias, and write the rubric a model grader would use, so that you hold a judge-checking routine you can re-run whenever a judge changes.

What this project needs: Python 3 with its standard library only. No task needs an account with a model provider, an API key, an internet connection or any payment. Every judge verdict the tasks use is printed below. All three tasks are required, and each can be done on its own, in any order.

Your evidence file. Keep one plain-text file, chapter7-evidence.txt. Each task tells you which lines to add. Every line has the form key: value, one per line, with the key in lower case exactly as printed. Write what your code printed, not what you expected it to print.

Task 1 — Measure a judge against people

Twenty replies were labelled pass or fail by a person, and then graded by a model grader. Each row is (case id, person's label, judge's verdict):

LABELS = [
    ("r01", "pass", "pass"), ("r02", "pass", "pass"), ("r03", "fail", "pass"), ("r04", "pass", "pass"),
    ("r05", "fail", "fail"), ("r06", "pass", "fail"), ("r07", "fail", "fail"), ("r08", "pass", "pass"),
    ("r09", "fail", "pass"), ("r10", "pass", "pass"), ("r11", "fail", "fail"), ("r12", "pass", "pass"),
    ("r13", "fail", "fail"), ("r14", "pass", "pass"), ("r15", "fail", "pass"), ("r16", "pass", "pass"),
    ("r17", "fail", "fail"), ("r18", "pass", "pass"), ("r19", "fail", "fail"), ("r20", "pass", "fail"),
]

1. Compute the agreement: the number of rows where the two verdicts match, divided by 20.
2. Compute Cohen’s kappa, an agreement score corrected for chance (formula from the scikit-learn documentation, cohen_kappa_score): po is the agreement; pe = (share of person passes × share of judge passes) + (share of person fails × share of judge fails); kappa = (po − pe) ÷ (1 − pe).
3. List the false passes: rows where the judge said pass and the person said fail.
4. Apply this trust rule, written before measuring: the judge may be used unattended only if agreement is at least 0.800 and kappa is at least 0.600. Otherwise it may not. Print trust or do_not_trust.

Stop there: one table, four evidence values. You do not need a statistics library.

Add to your evidence file:

t1_agreement: <3 decimal places>
t1_kappa: <3 decimal places>
t1_false_pass_ids: <case ids, case-id order, comma-separated, no spaces>
t1_decision: <trust or do_not_trust>

Output: a short script, and four evidence lines.

Task 2 — Test a pairwise judge for position bias

A pairwise judge compared two candidate replies, A and B, in twelve pairs. Each pair was judged twice. In order 1, A was listed first; in order 2, B was listed first. Each row is (pair id, candidate chosen in order 1, candidate chosen in order 2); the chosen candidate is "A" or "B":

PAIRS = [
    ("p01", "A", "A"), ("p02", "B", "B"), ("p03", "A", "B"), ("p04", "A", "A"),
    ("p05", "A", "B"), ("p06", "B", "B"), ("p07", "A", "A"), ("p08", "A", "B"),
    ("p09", "B", "B"), ("p10", "A", "A"), ("p11", "A", "B"), ("p12", "A", "A"),
]

1. A pair is consistent when the same candidate is chosen in both orders, and flipped when different candidates are chosen. Count the consistent pairs and list the flipped pair ids.
2. A flipped pair is a first-slot flip when the judge chose the candidate listed first both times: that is "A" in order 1 and "B" in order 2. Count them.
3. The flip rate is the number of flipped pairs divided by 12.
4. Rule: if the flip rate is above 0.200, the judge’s pairwise results are unreliable; otherwise reliable.

Add to your evidence file:

t2_consistent: <number of consistent pairs>
t2_flipped_ids: <flipped pair ids, pair-id order, comma-separated, no spaces>
t2_first_slot_flips: <number of first-slot flips>
t2_flip_rate: <3 decimal places>
t2_decision: <reliable or unreliable>

Output: a short script, and five evidence lines.

Task 3 — Write a rubric

A support reply must acknowledge the customer’s problem before it states the policy. Write the rubric a model grader would use for that check, in a file my-rubric.txt. The file has exactly these lines, in this form:

criterion-1: <a yes/no question about the customer's problem, at least 5 words, ending with ?>
criterion-2: <a different yes/no question about the policy, at least 5 words, ending with ?>
criterion-3: <a third, different yes/no question, at least 5 words, ending with ?>
verdict-line: VERDICT: pass or VERDICT: fail
rule: pass only if all three answers are yes

Each criterion must begin with one of the words does, is, are, did, do, has, have, can or will (any letter case), which is what makes it a yes/no question. criterion-1 must contain the word problem, criterion-2 must contain the word policy and criterion-3 must contain the word before, so that the third question asks about the order of the two. Copy the verdict-line and rule lines exactly as printed. Stop at three criteria. Whether your questions are good is your own call: the file is marked for these printed rules, and a rubric is shown to be good only by measuring a judge that uses it, as Task 1 does for a different judge.

Add to your evidence file:

t3_criteria_count: <number of criterion lines in my-rubric.txt>
t3_criterion_1_words: <number of whitespace-separated words in your criterion-1 question>

Output: my-rubric.txt, and two evidence lines.

What carries forward

Keep my-rubric.txt and chapter7-evidence.txt. Chapter 8, next in this course, asks the same kind of question about one specific failure: it takes a model’s answer and a source passage, and asks which claims in the answer the passage supports, and what your design should do with the ones it does not.