AI-103: evaluate a maintenance assistant locally

Allow 45–60 minutes. Use this starter for the retrieval and action-contract portions of Lab 8 in the AI-103 guide. Download or clone the starter files. Python 3.10 or later is sufficient; the exercise calls no Azure service.

Predict the retrieval results

Inspect retrieval-cases.json before maintenance.py. Labels identify relevant and authorized documents independently of the retrieved set. Document IDs are synthetic; SECRET-DOC contains no secret. Hand-calculate these outcomes:

Case	Precision	Recall	Decision to investigate
D1, D3, D4 returned; D1, D2 relevant	1/3	1/2	D2 is missing; unrelated evidence dilutes context
D1, D1, D2 returned	1	1	Set-based scoring counts each ID once
No results; D1 relevant	0	0	Abstain and investigate retrieval
No relevant answer exists in the corpus	0	Undefined (null)	Abstain; do not call recall 100%
D1 and a restricted document returned	1/2	1	Access breach blocks use despite perfect recall

The program records unauthorized IDs before filtering, so a leak cannot disappear from the evaluation. eligible_context and abstain use known relevance labels in this offline exercise. They are oracle outputs for the fixture, not a production relevance classifier or a grounded-answer guarantee. In a real application, authorization belongs in the data boundary; labels used for retrospective scoring are generally unavailable at query time.

python labs/ai-103/maintenance.py
python -m unittest discover -s labs/ai-103 -v

The first command prints all five results and a timeout/reconciliation demonstration. The suite runs 13 tests, including the five retrieval cases as subtests. Expected result: OK.

Follow the write across the boundary

The trusted test request concerns asset P-104. Creating a proposal makes no destination record. An authorized supervisor must approve its exact operation ID, asset and description. Caller access is checked separately. Try these changes in a copy, predicting each result first:

1. Submit before approval: reject with no write.
2. Approve a proposal, then change its description to close an order: reject the changed action.
3. Substitute P-140 for P-104: reject even if that proposal has approval. The expected asset must come from independently validated request context, not a second model-supplied field.
4. Use a caller without asset access: reject despite approval.
5. Replay the same approved operation: return the existing record; exactly one destination record exists.
6. Lose the response after the write: the client initially has an unknown outcome. Look up the original operation, observe its record, and replay only that operation. Do not invent a new key to bypass uncertainty.
7. Reuse an existing operation ID for different, newly approved content: reject the collision and preserve the original record.

WorkOrders is an in-memory, single-process model with trusted identity strings. It does not authenticate people, persist data across restarts, enforce concurrent uniqueness, expire/revoke approvals, or implement a real service's retry contract. These omissions are discussion points for the architecture review, not guarantees the learner can transfer to Azure.

Service validation packet

Once a disposable environment and spending limit are available, implement a separate adapter using the guide's current Azure references and pinned dependencies. Keep this fixture as the independent expected-outcome set. Record:

• the actual Foundry project, agent/model/API versions, dataset version, and supported region;
• effective caller, retrieval and tool identities, their permissions, and authorization failure evidence;
• retrieved IDs and access decisions before generation, final claims and citations, and missing-evidence behavior;
• tool arguments, approval binding, run/operation IDs, timeout observations and authoritative destination counts;
• actual service latency/cost, relevant traces, and cleanup of lab-owned resources.

The integrated guide exercise also covers a spoken negation, an unreadable equipment plate, and inconsistent invoice arithmetic. Those speech, image and document paths remain unexecuted here. Preserve “do not close” in the interpreted intent, abstain on an unreadable ID, and flag the arithmetic discrepancy without rewriting source evidence. Record actual media, extraction/transcription results and reviewer judgments when those services are tested.

Recorded validation and review

September 28, 2026: all 13 local tests passed. The timeout demonstration reconciled to one existing record; the restricted-document fixture reported a breach even with recall 1. Execution receipt records commands, outputs, runtime and file hashes. Azure execution and independent practitioner review are pending. Use review-record.json and the review process for that evidence.