07 / AI at Work · Field briefing
Good enough needs a test
Good enough needs a test.
A convincing demo answers one question: can the system produce an attractive result on that example? A useful evaluation asks whether it meets the requirements across representative inputs, including difficult cases. The evaluation should describe the work you intend to do, rather than borrow a leaderboard as a substitute.
What evidence should make us trust a workflow?
An explanatory schematic—not a live system or measured performance.
KAF World Digital · Prepared 4 October 2026 · First editorial edition. Worked examples are invented for teaching.
Try the example02 / Count the failures
Count the failures
Define acceptance before comparing models or prompts. An invoice extraction task might require the correct total, currency and due date, valid output structure and explicit treatment of missing values. A summary may need all stated decisions, no invented owners and a source reference for each action. Style and factual support deserve separate judgments.
Build a case set from authorised or synthetic material. Include ordinary inputs, edge cases, ambiguous material and deliberate gaps. Preserve expected outputs or a reviewer rubric. Keep development examples separate from a held-out check so repeated tuning does not turn the test into a memorised target.
In an invented ten-case exercise, seven outputs meet the strict rules. An eighth has supported facts but a different structure; it passes only when the formatting rule permits that variation. Two outputs invent commitments and fail under both rules. These counts describe this exercise, not future accuracy. Count failures by type and consequence as well as reporting a percentage.
Use the simplest suitable grading method. Code can check required fields, calculations or exact answers. Humans can review meaning, omissions and practical usability. A model-based judge can help with scale, but needs its own calibration against human judgments and can inherit biases or miss subtle failures.
Repeat enough runs to see instability when outputs vary. Record the model identifier, prompt version, retrieval settings and tool configuration. Comparing two runs while changing all those variables makes the result hard to interpret.
Measure review time and cost alongside quality. A high-scoring draft that requires excessive checking may not meet the workflow's purpose. Re-evaluate after meaningful model, data or configuration changes; a previous pass applies to the system that was tested.
Evidence for the technical mechanisms: Anthropic · Building Evals, Microsoft · Retrieval Augmented Generation. The worked examples and practical framework are KAF editorial recommendations.
03 / Worked example
Try the acceptance rules
Separate a convincing demo from evidence about a workflow. This interactive lesson uses prepared synthetic examples and local calculations.
| Cases | Observed output | Result |
|---|---|---|
| 01–07 | Complete, supported, required structure | Pass |
| 08 | Supported facts, different structure | Fail — format |
| 09–10 | Invented commitment or owner | Fail — unsupported |
Use the companion worksheet and the worked example. All assumptions are illustrative.
No account, model API or external service is connected. Inputs are not submitted or stored by this page.
04 / Go deeper
Useful words. Important details.
- Held-out case
- An example kept separate from development to check performance on unseen material.
- Rubric
- Explicit conditions a reviewer uses to grade an output.
- Omission
- Important information missing from an answer, even if what remains is correct.
- Calibration
- Checking that a scoring method agrees with reliable judgments on the task.
An observed rate needs a denominator
Report which cases were attempted, accepted and excluded. Small synthetic sets explain a method; they do not establish population accuracy or the probability of rare failures.
A judge needs evaluation too
Code graders can be precise for structural or exact-answer checks. Model graders need calibration against trusted judgments, especially when evaluating another model’s plausible language.
Change one thing when comparing
Keep prompt, dataset, retrieval, tool permissions and settings recorded. Repeated runs reveal variability. Re-check the system when an important component changes.
The technical notes qualify the explanation; they are not product-specific assurance or a certification.
05 / Evidence & scope
Inspect the source.
Keep the context.
Sources were accessed on 4 October 2026. These technical and security references support the mechanisms described; they do not validate our fictional examples or establish comparative product performance.
- Anthropic · Building Evals
Task-specific evaluations and code, human and model-based grading. Provider examples are not independent comparative evidence.
- Microsoft · Retrieval Augmented Generation
Retrieval, document chunking, search and evaluation of grounded answers.
- OWASP · Misinformation, 2026
Unsupported outputs, stale context, omissions and claim-check-act patterns. A risk framework, not a research accuracy benchmark.
What this briefing does—and does not—establish
The worked examples, suggested questions and worksheets are original KAF teaching material. Numbers are explicitly illustrative. No customer data, observed adoption results, market-size forecast or live provider pricing is presented.
Check the exact product, release, configuration and policy before using a workflow. Source documents can change; this is a dated educational briefing.
Keep a useful copy.
Read the full briefing offline, or use the companion worksheet to apply it.