Before a commercial team can promise anything involving a client's data, someone has to read that client's privacy policy and work out whether the proposed use is actually permitted. It is slow, it needs legal literacy, and the answer has to be defensible months later, which means an opinion is not enough. It needs evidence.
- Graded each proposed use case against the policy through a citations API, so every verdict points back at the passage it came from
- Split the amber status in two, ambiguous versus silent, because a policy that never mentions a use case is a different risk from one that mentions it unclearly
- Enforced the distinction in code: no citations means the verdict cannot be green
- Made runs, verdicts, citations and overrides insert-only, so each assessment is an immutable snapshot rather than a mutable record
- Added advisory second opinions from two other model families, and a machine API so other automations can trigger assessments
A live internal tool, signed in through corporate SSO, where anyone can add a client and get a defensible, cited grid of verdicts with PDF export. A full assessment runs for under $2 by prompt-caching the policy document across per-use-case calls.
- Next.js · Vercel
- Supabase (Postgres, RLS)
- Anthropic citations API
- OpenRouter (advisory models)
Designing a system that refuses to guess
The temptation with a grading tool is to always return an answer. That is exactly the failure mode worth engineering against, because the expensive mistake here is not an uncertain verdict. It is a confident wrong one. So the interesting decisions were all about what the system is not permitted to conclude.
Splitting amber was the first. A policy that addresses a use case ambiguously and one that never addresses it at all look similar in a summary and are completely different in a negotiation, so they became distinct statuses. The second was making that structural rather than advisory: the grounding layer returns citations, and if a verdict arrives with none, the code will not let it be green regardless of how confident the model sounded. Silence never reads as permission.
Grounding it this way had a cost worth noting. The citations API is incompatible with structured outputs, so you cannot simply ask for clean JSON. The pipeline parses lightly-structured text and takes the citations from the API's own objects instead of trusting the model to format them. Slightly more work, and a much stronger guarantee: the evidence comes from the platform, not the prose.
The bug that graded a document made of navigation menus
One client's policy page was fully JavaScript-rendered. The URL fetcher retrieved it, got 491 characters of navigation chrome, and stored that as a perfectly valid snapshot. No error, no flag. An entire sector brief was then researched and written on top of it.
The fix that mattered was not the file-upload fallback, though that shipped too. It was raising the floor on what counts as a successful fetch and making the failure loud, so the fetcher now refuses thin content and points at the alternative rather than quietly succeeding. Then the bad snapshot was invalidated and everything derived from it regenerated. A silent success is worse than an error, because nothing downstream has any reason to doubt it, and in a system whose whole value proposition is evidence, that is the one failure you cannot tolerate.