Skip to content

GH-300: prove the tests can reject a bad suggestion

Allow 45–60 minutes. This implements the local fallback for Lab 6 in the GH-300 guide. The starter directory contains a contract, two prompts, an intentionally broken candidate, a reference, and a second mutation. Python 3.10 or later is sufficient.

Establish the oracle first

An order subtotal is a nonnegative integer number of cents. Shipping costs 499 cents below 5,000 cents and is free at or above 5,000 cents. Reject booleans, floats, strings, nulls, collections and negative integers with ValueError. Tax, discounts, currency conversion and complete checkout-system behavior are outside scope.

Write these expectations before reading implementation files:

Input Expected
0 or 4999 499
5000 or 5001 0
A very large positive integer 0
-1 ValueError
True, False, 50.0, "5000", None, [], {} ValueError

test_contract.py runs seven tests, with invalid types checked as subtests. Its expected values come from this written requirement. Agreement between a suggestion and tests derived from that suggestion would not establish the correct boundary.

Run the deliberate failures

From the repository root:

python labs/gh-300/test_contract.py --implementation starter
python labs/gh-300/test_contract.py --implementation reference
python labs/gh-300/test_contract.py --implementation boolean_mutant
python labs/gh-300/compare.py

Run these commands individually if your shell stops on a nonzero exit. With the original files:

  • Starter: exit 1, exactly the threshold test fails: 499 != 0 at 5000.
  • Reference: exit 0, all seven tests pass.
  • Boolean mutation: exit 1, the invalid-type test fails for both True and False. Python's isinstance(value, int) accepts booleans; this contract rejects them.
  • Comparison: exit 0 only when each candidate has its expected result and there are no unrelated errors. It prints the actual results, including failure output. An import error does not count as a successful mutation check.

Work in a copy when repairing starter.py. After the fix, run its contract command again and expect all seven tests to pass. The comparison harness describes the original intentionally broken fixtures: it should reject a repaired starter because the expected deliberate failure is gone. Keep original fixtures available for reproducibility; do not alter the oracle to make a broken function pass.

Add Copilot when a supported session is available

CONTEXT.md contains the exact requirement, relevant paths, expected results, verification command, and two prompts. It is a manually supplied context packet, not an enforced policy or a special instruction file that every client automatically loads.

  1. Copy the lab into a disposable workspace. Keep the reference and mutation out of the model's selected context for the comparison.
  2. In one fresh session, use the deliberately vague prompt. Save the actual prompt, selected context, suggested diff, output and review effort.
  3. Restore the broken starter in that disposable copy. In another fresh session, supply the context packet and its more precise prompt. Save the same evidence.
  4. Inspect both diffs and run the unchanged contract against each result. Explain which context was useful and which assertions caught defects. Do not claim a general productivity improvement from this one task.
  5. Record the actual Copilot client/version/model, relevant settings and date. Instructions can guide generation; they do not authenticate a tool, enforce a merge rule, or establish that a test ran.

No Copilot session, organization policy, content exclusion, agent/MCP tool, or required status check was tested in this local run. Those other guide labs need their own supported environment and evidence.

Recorded validation and review

September 28, 2026: the reference passed seven tests. The comparison also ran seven tests against each of the three prepared candidates, detecting the one threshold failure and two boolean subtest failures exactly as expected. These are overlapping runs of the same contract, not 28 distinct requirements. All candidates and tests were AI-assisted local examples; none is represented as observed Copilot output.

The execution receipt records runtime, commands, actual output and file hashes. Complete review-record.json for actual Copilot sessions and the independent review packet; both remain pending.