# How to evaluate AI feature output quality on Mac: a practical guide for product managers

> Evaluate AI feature output quality with a fixed test set, a short rubric and spot checks. Steps, costs and tools for product managers on Mac.

URL: https://www.lazur.app/for/product-managers/problems/evaluate-ai-feature-output-quality
Updated: 2026-11-09

To evaluate AI feature output quality, you need the same inputs, the same scoring rules and a record of every run. Most teams start with a few demo prompts and a gut feeling. That works until a model update or a prompt edit quietly makes answers worse. This guide shows how to build a small test set, grade it and keep the results honest on your Mac.

## Quick answer

- Build a fixed test set from real user inputs, not from demo prompts.
- Score each output against a short rubric with one clear line per criterion.
- Use a different model as the grader, then read a sample of its scores yourself.
- Log every failure, and rerun the same set after each prompt or model change.

## What measuring AI output quality costs you

- **Rework you do not see.** Each bad output that reaches a colleague costs a fix. BetterUp estimates about two hours per incident, based on its own survey. The time spreads across reviewers, so it rarely shows up in a sprint report. Count it before you argue about model quality.
- **Money per person.** BetterUp puts workslop at $186 per employee per month. Its estimate for a 10,000-person company is about $9 million a year. These are self-reported figures from one vendor, so use them to frame the conversation, not to set a budget.
- **Customer trust.** One confident wrong answer can end a user's trust in the whole feature. Users rarely report a near miss. They just stop using the feature. You see that drop in usage before you see a complaint.
- **Slower releases.** When nobody can say whether a prompt change is safe, teams stop changing prompts. Or they ship changes and check them by hand. Both slow the roadmap. A saved test set turns that debate into a short check.
- **Decisions made on impressions.** Teams that judge AI features by impression often pick the wrong fix. A demo that looks sharp can hide a failure on a common input type. The METR result shows how far a feeling can drift from the measured result.

## Why it happens

1. **No single right answer.** A normal feature returns the same result for the same input. An AI feature varies. A test that passes once tells you little. Without a fixed set and a fixed rubric, you are comparing anecdotes.
2. **Demo prompts hide failures.** Demos are chosen because they work. They are clean, short and familiar to the team. Real users write messy, long and odd inputs. The gap between those two sets is where quality problems live.
3. **Feelings run high.** The METR trial found developers believed they were faster while measured timing showed slower work. People judge their own tools generously. A product manager is no different when the feature is theirs.
4. **Fluent text earns trust.** Clear, confident writing looks correct. KPMG's global press release reports 66% saying many people rely on AI output without evaluating it. Reviewers skim fluent text and miss the invented detail.
5. **Reviewers score differently.** Without a written rubric, one reviewer cares about tone and another about facts. Their scores then measure their preferences. Disagreement looks like noise, but it is usually a missing rule.
6. **Ownership falls between teams.** Engineering runs the model, design shapes the flow and product owns the outcome. Evaluation can fall between those jobs. Someone has to own the test set, the rubric and the release check.

## How to fix it

- **Freeze a baseline before you change anything.** Run the current version on your full input set and save the results with the date. This is your baseline. Do not edit the prompt, model or inputs until it is saved. Every later comparison depends on it.
- **Write must-pass cases as a release gate.** Pick the cases where a wrong answer is unacceptable, such as a required refusal, a legal disclaimer or a fact users act on. Mark them must-pass. A failure on one blocks the release, even when the average looks fine.
- **Grade with a different model, then sample.** Use a model other than the one that wrote the output. Read a sample of its passes and fails each week. If you disagree often, change the rubric line that caused it, then rerun the set.
- **Split long outputs into parts.** Score sections separately. Give numbers, names, dates and links their own line in the rubric. Those are the details reviewers skim past, and the details users check first.
- **Turn each failure into a tracked issue.** Create one Linear issue per failure, with the input and output attached. Label issues by feature area so patterns show up. Group the failures by label each Friday to see which area needs work first.
- **Calibrate reviewers in one session.** Put two or three reviewers on a call and have them score the same outputs. Compare scores line by line. Rewrite any rubric line that caused an argument. Repeat the session whenever a new reviewer joins.
- **Keep the rubric and decisions on one page.** Store the rubric, the baseline file link and a decision log in one Notion page. A dated log of what changed and why stops the same debate from returning every sprint.
- **Redact sensitive inputs before testing.** If inputs include client, patient or customer data, follow your employer's rules first. Use redacted or made-up copies for tests when those rules require it. Confirm the approach with your security or legal team before you start.

A normal feature has a right answer. A save button either saves the file or it does not. An AI feature can give a different answer to each input, and sometimes a different answer to the same input twice. A quick try in a demo rarely tells you how the feature behaves. You need many inputs, one scoring method, and a record you can compare over time.

![Team members looking at a laptop screen during a product review of AI results](https://images.lazur.app/content-images/problem/evaluate-ai-feature-output-quality/evaluate-ai-feature-output-quality-1-9cc08d.jpg)

## What bad AI output costs your team

Bad output costs time before it costs anything else. BetterUp surveyed 1,150 full-time U.S. desk workers in September 2025. It found that 40% had received workslop in the last month. Workslop is AI-generated work that looks finished but does not move the task forward. The survey is self-reported, so read it as a signal, not a precise measure.

BetterUp puts the average fix at two hours per incident. It estimates $186 per employee per month, or about $9 million a year for a 10,000-person company. Those figures come from one vendor's research. Your team may see more or less. The more useful number is how many hours your own reviewers spend fixing AI output each week.

- **40%** of U.S. desk workers got workslop, AI output that looks finished but does not move work forward, last month (self-reported) ([source](https://www.betterup.com/workslop))
- **$186** per employee per month lost to workslop, according to BetterUp's estimate ([source](https://www.betterup.com/workslop))
- **56%** of employees in KPMG's global study say they made mistakes at work because of AI ([source](https://kpmg.com/xx/en/media/press-releases/2025/04/trust-of-ai-remains-a-critical-challenge.html))
- **66%** say many people rely on AI output without evaluating its accuracy, per KPMG's global study ([source](https://kpmg.com/xx/en/media/press-releases/2025/04/trust-of-ai-remains-a-critical-challenge.html))

| Where the cost shows up | Who pays it | Early warning sign |
| --- | --- | --- |
| Rework on drafts that look finished | The reviewer, then the author | Reviewers start rewriting whole sections |
| Wrong facts reach customers | Support and the account owner | Tickets that quote the AI feature |
| Trust drops in the feature | The product team | Users stop using it after one bad answer |
| Slow releases | Engineering and QA | Every prompt change needs a long manual check |

_Where bad AI output usually costs money, and how to spot it early._

Put a rough number on it before you argue about model quality. Ask three questions about last week. How many AI drafts did reviewers rewrite? How many tickets quoted a wrong AI answer? How long did the last prompt change take to check? The answers give you a cost you can defend in a planning meeting.

## Why AI output quality slips past reviewers

The fastest way to fool yourself is to trust how a tool feels. In a July 2025 randomized trial, METR tracked 16 experienced open-source developers on 246 real issues. With AI tools allowed, the developers took 19% longer to finish. Before the study, they expected a 24% speedup. Afterward, they still believed AI had made them 20% faster.

That gap is the lesson for product managers. A feeling that the feature is good is a hypothesis, not data. METR also warns that these results used early-2025 tools and do not generalize to most software work. Treat the study as a warning about self-assessment, not as a fact about your product.

Reviewers miss errors for a similar reason. KPMG's 2025 global study surveyed more than 48,000 people across 47 countries. In its press release, 66% said many people rely on AI output without evaluating its accuracy. Another 56% said they had made mistakes at work because of AI. These are self-reports, and regional pages show slightly different numbers.

## Step 1: build a test set from real inputs

A test set is a list of inputs your feature should handle, plus notes on what a good answer looks like. Pull the inputs from real use. Support tickets, search logs, sales call notes and your own past requests all work. Demo prompts are too clean. They show the feature working on the examples you picked, which is not the same as working on your users' messy text.

![Product team meeting to evaluate AI feature output quality around a shared screen](https://images.lazur.app/content-images/problem/evaluate-ai-feature-output-quality/evaluate-ai-feature-output-quality-2-14291b.jpg)

Start with a few dozen inputs. Include easy cases, hard cases and awkward ones. Add long inputs, inputs with typos, inputs that mix two languages, and requests the feature should refuse. Put each input in its own row of a Google Sheets file, with columns for the input, the output and each score.

- Very short inputs, such as one or two words
- Very long inputs close to the length limit
- Typos, slang and text pasted with stray formatting
- Requests outside the feature's scope
- Inputs with names, account numbers or other sensitive details

Freeze the set once it is ready. If you swap inputs after you see results, the scores stop meaning anything. Add new failures as you find them, and keep the old cases too. Old cases show you when a fix in one area breaks another. You can filter the sheet by category with [the Google Sheets filter steps](https://www.lazur.app/how-to/add-a-filter-to-a-range-in-google-sheets).

> **Check data rules first:** If your feature handles client, patient or customer records, follow your employer's rules before you paste any of it into an AI tool. Use redacted or made-up inputs when your rules require it. Ask your security or legal team when you are unsure.

## Step 2: write a rubric before you read outputs

A rubric is a short list of criteria. Each criterion has a clear way to score it. Write the rubric before you read any outputs. If you write it afterward, you will grade toward what you already like. Keep it short enough that a reviewer can apply it in a minute or two per output.

Take a feature that summarizes support tickets for an internal triage team. A useful rubric might ask these questions. Does the summary include every customer request? Does it add any fact that is not in the ticket? Does it use the right product names? Is it short enough to scan quickly? Score each line on a 5-point scale, where 1 is unusable and 5 needs no edits.

Define the low end too. Write one sentence that says what a 1 looks like for each criterion. Without that, two reviewers can score the same output differently and both feel right. Score a handful of outputs together first, then fix any line that starts an argument.

> **Split facts from style:** A summary can read well and still invent a deadline. Give facts their own line in the rubric. A fluent answer then cannot hide a wrong one.

## Step 3: use a second model as the grader

Reading every output by hand does not scale. Many teams ask a language model to grade outputs against the rubric instead. Researchers call this LLM-as-a-judge. In the MT-bench and Chatbot Arena paper, strong judges such as GPT-4 reached over 80% agreement with human preferences. That matched the agreement between humans.

The same paper lists known weaknesses. Judges can favor the answer shown first, favor longer answers, and favor output from their own model family. They also reason imperfectly on some tasks. The benchmark tested chat answers, not product summaries, so the 80% figure does not carry over directly to your feature.

Anthropic's evaluation docs recommend using a different model from the one that generated the output, when you use an LLM for grading. If your feature runs on one model, grade with another. Then read a sample of the grader's scores yourself every week. Sampling is how you catch a grader that has drifted or has a blind spot.

> **Do not trust an unchecked grader:** A grader can sound confident and still be wrong. Read the outputs it marked as passes, not only the failures. If longer answers always win, fix the rubric before you trust the next run.

## How to run an AI evaluation with Promptfoo on Mac

Promptfoo is a command-line tool for running prompts against test cases. Its getting-started guide documents an install with Homebrew. Its config file, promptfooconfig.yaml, lists your prompts, inputs and checks. The steps below follow that guide. Check the current page before you run them, because commands and labels change.

1. Open Terminal with ⌘ Space, type Terminal, and press Return.
2. Run brew install promptfoo, then run promptfoo init --example getting-started.
3. Open promptfooconfig.yaml in a text editor. Replace the example prompt and inputs with your feature's.
4. Add an llm-rubric check for each criterion, written as a plain instruction, such as: The summary must not add facts absent from the ticket.
5. Run promptfoo eval, then run promptfoo view to open the results in your browser.

The llm-rubric check is a model-graded test of free-form instructions. The Promptfoo page uses the example 'Do not apologize.' Your rubric lines work the same way. Keep one criterion per check, so a failure tells you what went wrong. A check that covers four things at once tells you almost nothing.

Save each results file with the date in its name, and keep the first run as your baseline. A saved baseline lets you compare the feature before and after a prompt change. That comparison matters more than any single score.

## Side-by-side checks in Anthropic's Console

If your feature runs on Claude, Anthropic's Console has an Evaluate feature for prompts. The July 2024 launch post describes a Generate Test Case button, CSV import and side-by-side comparison of prompts. Subject matter experts score responses on a 5-point scale. Labels may have changed since then, so match the post's steps to what your Console shows.

Side-by-side comparison suits product managers well. Show two prompt versions to a domain expert and ask which answer is better. Record the choice and the reason. Then sort the reasons by how often they come up. The most common reason usually points to the real problem.

## Mistakes that make AI quality scores meaningless

- Testing only the demo prompts, which were picked because they work.
- Running each input once, then treating one answer as the feature's behavior.
- Changing the prompt, the model and the rubric in the same week.
- Letting the generating model grade its own output.
- Scoring without a baseline, so every number looks like progress or failure.
- Reporting an average and hiding the failures users will notice.
- Reading the summary score but never opening a failed output.

The last mistake is the most common. A feature can average well and still fail on one customer type. Sort failures by input category. Keep a short list of must-pass cases, such as a refusal that must happen or a fact that must be right. One failed must-pass case should block the release, whatever the average says.

## Where Lazur fits in your evaluation work

Lazur is a voice assistant for the Mac. Hold ⌥ ⌘ to ask about anything on your screen, and Lazur answers out loud. Ask where a control is, and Lazur draws a circle and a label around it. Ask how to do something, and Lazur guides you one click at a time, waiting for each click.

Lazur does not run Promptfoo, click buttons in the Console or grade outputs for you. It does not operate your spreadsheet. Use it to explain a rubric line, draft test inputs in your product's voice, or dictate reviewer notes. Hold fn to dictate clean text at the cursor in any app. You still choose the criteria and read the failures.

Research jobs run in Lazur's own Chrome tabs, through the Lazur for Chrome extension. They return a summary and a report. That helps when you want to see how other teams grade AI output. Check each claim against its original source before it goes into a spec.

Lazur listens and looks at the screen only while you hold the shortcut. Audio and the screenshot go to AI providers to answer, and they are not stored. Content is never used to train AI models. Still, follow your employer's rules before you hold the shortcut over client or patient data.

## Edge cases: long outputs, unstable answers and small teams

![Laptop and notebook on a desk with a spreadsheet of test results](https://images.lazur.app/content-images/problem/evaluate-ai-feature-output-quality/evaluate-ai-feature-output-quality-3-9e4962.jpg)

Long outputs, such as reports or generated code, need more than one score. Split them into sections and grade each section. Check the parts that matter most, such as numbers, names and links. A reviewer who skims a long answer can miss a made-up figure in the middle.

Some outputs change from run to run. If the feature samples its answers, run each input more than once. An input that passes one run and fails the next is a finding. Record it as unstable, not as a pass. Unstable cases often point to a prompt that leaves too much open.

Small teams can start with one reviewer. Ask a second person to score a sample each week. When two people disagree, write down the reason and update the rubric. That habit turns disagreement into a better test.

## What to do next

Pick the feature that causes the most rework this month. Build the input set first, then the rubric, then the baseline. Do not change the prompt until the baseline is saved. Every later score depends on that first file meaning something.

## With Lazur

Lazur is a voice assistant for the Mac that lives in the notch. Hold ⌥ ⌘ and ask; it sees your screen, answers out loud, points at buttons and walks you through apps. Hold fn to dictate. Free for 3 days, then $18/month: https://www.lazur.app/download

- “Where do I set the score column in this sheet?”: Lazur circles the score column header and labels it. You then type each 1 to 5 score into the cells yourself. (Points at it)
- “Explain what llm-rubric does in this config”: Lazur explains the check in plain words and points to the line it belongs on, based on what is on your screen.
- “Where is the run button in this view?”: Lazur draws a circle and a label around the control. You press it yourself.
- “Walk me through adding a filter to the score column”: Lazur guides you one click at a time through Data > Create a filter in Google Sheets, waiting for each click.
- “Draft a dozen test inputs for a ticket summary”: Lazur drafts realistic inputs in your product's voice. You review each one before it goes into the set.
- “Dictate these reviewer notes into the doc”: Hold fn and speak. Lazur types clean text at the cursor in any app.
- “Find how other teams score AI output quality”: Lazur runs research in its own Chrome tabs and returns a summary and a report. Check each source before you rely on it.

## A plan you can start this week

- **Monday: Pick one feature and one user job.** Choose the feature that causes the most rework or support tickets. Write the job it does for users in one sentence. Leave the other features for later.
- **Tuesday: Pull real inputs into a sheet.** Export a week of real inputs from tickets or logs. Add the edge cases, then freeze the list. Redact anything your employer's rules do not allow in tools.
- **Wednesday: Write the rubric and save a baseline.** Draft a short rubric with a few criteria. Write one line that describes a score of 1 and one for a score of 5. Run the current version on every input and save the results file.
- **Thursday: Grade with a second model and sample.** Set up the grader in Promptfoo or in the Console. Read a sample of its passes and fails yourself. Note each place you disagree and why, then update the rubric line.
- **Friday: Log failures and agree on the gate.** Turn each failure into a tracked issue with the input attached. Mark the must-pass cases. Agree with engineering that a must-pass failure blocks the next release.

## FAQ

### Can I evaluate AI feature output quality without writing code?

Yes. A Google Sheets file with inputs, outputs, rubric scores and a date column is enough to start. Use Anthropic's Console Evaluate feature or a reviewer panel for side-by-side checks. Code helps when you want to rerun many tests quickly, but it is not required for a first baseline.

### How many test inputs do I need?

A few dozen real inputs is enough to start. Include the awkward cases, not only typical ones. Add each new failure to the set as you find it. Coverage of what users actually do matters more than the raw count.

### Can the same model grade its own output?

Avoid it. Anthropic's evaluation docs recommend a different model for LLM-based grading. Even then, read a sample of the grader's scores yourself. A grader can favor longer answers or the answer it saw first.

### How often should I rerun the test set?

Rerun it after every prompt change, model change or data change. Rerun it before each release too. Keep the results files so you can compare runs over time and spot regressions early.

### Does Lazur score AI outputs for me?

No. Lazur can explain a rubric line, point at controls, draft test inputs and dictate notes. It does not run Promptfoo, click buttons in other apps or grade outputs. You still set the criteria and review the failures.

### Is the METR result about product managers?

No. The study looked at experienced open-source developers using early-2025 tools. METR says the results do not generalize to most software work. Use it as a reminder that self-reports about speed and quality can be wrong.

## Sources

1. [Workslop: The Hidden Cost of AI-Generated Busywork](https://www.betterup.com/workslop), BetterUp, checked 2026-11-09
2. [KPMG global study press release: trust of AI remains a critical challenge (April 2025)](https://kpmg.com/xx/en/media/press-releases/2025/04/trust-of-ai-remains-a-critical-challenge.html), KPMG International, checked 2026-11-09
3. [Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/), METR, checked 2026-11-09
4. [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685), arXiv, checked 2026-11-09
5. [Promptfoo Getting Started](https://www.promptfoo.dev/docs/getting-started/), Promptfoo, checked 2026-11-09
6. [Using the Evaluation Tool](https://platform.claude.com/docs/en/test-and-evaluate/eval-tool), Anthropic, checked 2026-11-09
7. [Evaluate prompts in the developer console](https://claude.com/blog/evaluate-prompts), Anthropic, checked 2026-11-09