Engineering note / Public teaching example
A plausible answer still needs evidence.
A small exercise in separating valid structure from supported claims.
The exercise
The public exercise is an original recreated example. It is not an Oracle document or evidence that every topic was delivered there.
Prompt: Given only the supplied source, return a summary and a source identifier. If the source does not establish the answer, return insufficient_evidence.
Failure case: A response can match the expected JSON shape while citing a source that does not exist.
Review criteria
- Validate the response shape at the API boundary.
- Check that the cited identifier exists in the supplied source set.
- Keep missing evidence unresolved rather than inventing a citation.
- Use human review to assess whether the source actually supports the summary.
A useful review asks two separate questions: does the reference exist, and does it support the statement? The first can be checked deterministically; the second still requires assessing meaning.
What this demonstrates
Define failure behavior before asking an agent to execute. A missing source should remain visible as missing evidence, so the next step is a review or a better input.
An original public exercise using isolated synthetic fixtures. No model calls or employer code. The gate checks shape and source existence; a human must assess whether the source supports the answer.
Try the original fixture: download the example and run node evidence-gate.mjs.
Four deterministic fixtures produced the expected outcomes: existing source → ready for review; invented citation → missing source; explicit missing evidence → needs review; malformed response → invalid shape.
Download the runnable exercise ↓See executable acceptance checks in Evalgate ↗