The rules, the test sets and the answer keys come from QC practice, and every AI result is measured against them.
Rules first
What counts as a flaw, what blocks a reduced test, what the tool must refuse to do. Written as a spec before any code.
Known-answer test sets
Synthetic records with known flaws and known outcomes. The answer key is fixed in advance, so the model never grades itself.
Measured against the key
Every AI result on this page is scored against an answer key fixed before the run.
Each tool is labelled by how much it decides on its own: a script is fixed code, a workflow is fixed steps with AI at some of them, an agent chooses its next step.
01EL-01 · script
Compendial Change Tracker
RecordEL-01
KindScript, no AI
DataPublic USP-NF pages
OutputExcel report of new items
USP-NF changes arrive as notices of intent to revise, revision bulletins and errata. Somebody in the lab has to spot the new ones and check whether any touch the lab’s products. This tool does the spotting: it lists only what is new since the last check, so the product check starts from a short list.
What it does
It reads the three public USP-NF update pages, compares every entry against a saved baseline, and writes an Excel report of what is new. It is deterministic Python with no model in it, because nothing here needs judgment: it is a list comparison.
fixed codeHow a run works.
12752
Notices of intent to reviseGeneral announcementsPublication announcementsGeneral chapter prospectuses
Sample output: July 2026 run
General Announcement
Call for Data: Request for Stakeholder Information Related to Pharmacopeial Forum (PF) 52(2)
26-Jun-2026
General Announcement
Proposal Status Reporting on USPNF.com
26-Jun-2026
Notice of Intent to Revise
Bemotrizinol
09-Jun-2026
Notice of Intent to Revise
Alpha Lipoic Acid
29-May-2026
Notice of Intent to Revise
Erythromycin
29-May-2026
Notice of Intent to Revise
Methyl Salicylate
29-May-2026
+ 20 more items in the same run
The first 6 of the 26 new items, copied from the run file and set as a table here. That run had no new revision bulletins or errata.
02EL-02 · agent
Deviation Review Agent
RecordEL-02
KindAgent + verifier + human gate
Data12 synthetic deviations
Human gateRequired
An AI second reviewer for deviation investigations: every finding quoted from the record, every finding challenged, and a human sign-off that can’t be skipped.
What it does
Reviewer agent reads one deviation investigation, the way a QA second reviewer would. It can read one file in an empty folder and nothing else, and every finding must quote the passage it rests on.
Verifier agent gets the record and the reviewer’s claims, never its reasoning, and must keep, weaken or drop each one with its own evidence.
Human sign-off: a QA reviewer accepts, rejects or edits every finding and must write a reason. A hook on the closing agent refuses to write the record unless every reason is present and appears word for word.
agenthumangateThe record cannot close until the human step is complete.
93-second narrated walkthrough. Synthetic records throughout.
What was measured
12 synthetic records for a fictional company, 26 planted flaws (15 serious), 3 clean records including one trap. Scored against a fixed answer key.
Single runs on a small synthetic test set, so indicative only. Matching was done by an AI judge whose decisions are on file. The false-alarm trap (a late closure with an approved extension) passed in every configuration. The clean record that drew false alarms was an environmental-monitoring excursion.
Sample finding with human sign-off: DEV-2026-008
DEV-2026-008 · finding 1RC_UNSUPPORTEDprimary
“The investigation concludes that the operator was distracted during the wet-massing step and did not stop the mixer at 8 min, then recorded the intended time rather than the actual time in the batch record.”
Operator distraction is asserted with no supporting evidence, such as an interview about what distracted the operator, an observation, or a PLC event log. […]
accepthuman sign-off
“No evidence for distraction shown. No proof that the button is working correctly”
Tamara Levine · 12 Oct 2026, 10:01 UTC
One real finding from the kept configuration, with the quoted passage it rests on and the signed reason, copied from the run files. The record is synthetic.
Close record DEV-2026-008
DENIED record not written
finding 4: reason missing or shorter than 10 characters
Close record DEV-2026-008, after sign-off
ALLOWED record written
4 of 4 findings signed; 4 human reasons found verbatim
03EL-03 · workflow
Reduced-Testing Writer
RecordEL-03
KindWorkflow with AI steps
Data21 synthetic evidence packs
Halts onMissing evidence
Full monograph testing of every incoming raw-material batch costs real lab time. A justified reduced or ID-only scope for a qualified supplier saves it, but the justification means pulling specs, three batches of in-house and vendor results, investigation history and vendor status from several systems. So in many labs it simply waits.
What it does
Given one material’s evidence pack, it proposes FULL or REDUCED for every test, classifies the material as ID-only, partially reduced or full testing, and writes a report for a QA reviewer in which every sentence points to its source. If evidence is missing or contradictory, it writes no report at all, only a missing-evidence notice.
The design decision
AI reads what code cannot, proposes what a human must approve, and writes what code has already decided.
Every decision (matching, unit conversion, spec tightness, within-spec checks, eligibility, classification, halting) is tested Python on rules a reviewer can read and edit. AI is used in five places only: reading vendor certificates of analysis in different layouts, proposing name mappings for a human to approve, drafting a method-equivalence rationale for a human to decide, drafting the report text, and checking that text against the evidence. Identity and description are never reduced. A vendor or in-house result out of spec stops the whole abbreviation.
AI step (5)Python rulehumanhaltFixed order, wired by code. The AI never takes a decision.
27/27
Whole packs with AI in the loop: 9 packs, 3 repeats each, every run matched the answer key.
What was tested
Result
Note
Deterministic rules
110 tests pass
No AI in these
Whole packs, with AI in the loop
27 of 27 runs
9 packs, 3 repeats each, matched the answer key
Reading synthetic CoAs
480 of 480 fields
12 CoAs in 3 generated layouts (easy case)
Reading real public CoAs
48 of 48 (larger model) 45 of 48 (smaller model)
5 public vendor CoAs, not republished here; labels by a QC expert
Claim checker on planted false claims
71 of 72 caught
Wrongly flagged 5 of 72 true claims
Missing or contradictory evidence
halt packs
Halted correctly, before any AI step was called
Sample report, raw output: pack P01 (synthetic)
SYNTHETIC DATA. PROTOTYPE ONLY. NOT VALIDATED. NOT FIT FOR USE.
Material classification is FULLY. M-class 2 tests are proposed FULL and 8 REDUCED, so the testing mode is ID_ONLY. M-counts, M-mode Next periodic review date is 2027-03-10. M-review
2. Per-test proposed scope
Test
In-house
Vendor name as printed
Mapping
Scope
T01 Description
VISUAL
Appearance
alias
FULLT01-scope
T02 Identification A (IR)
IR
Identification (IR)
proposed-approved
FULLT02-scope
T03 Assay (anhydrous basis)
TITRATION
Assay (anhydrous basis)
exact
REDUCEDT03-scope
T04 pH (5% w/v solution)
POTENTIOMETRY
pH, 5% solution
proposed-approved
REDUCEDT04-scope
T05 Water (KF)
KF
Water by KF
proposed-approved
REDUCEDT05-scope
+ 5 more tests, all REDUCED; then the three-batch comparison
FULLY means fully reduced: every test except identity and description is reduced, so the testing mode is ID_ONLY. Run without the model: the decisions are the same Python, and the wording is the fixed template, not the AI draft. Pointers like T01-scope tie each line to its source.
Missing evidence notice (no report written)
SYNTHETIC DATA. PROTOTYPE ONLY. NOT VALIDATED.
Halt code: MISSING_COA
Detail: no CoA for in-house batch NB-2603 (vendor batch AL-5503)
The writer halts rather than write around missing or
contradictory evidence. Supply the missing item and rerun.
Questions
Who builds the tools?
The rules, test sets, answer keys and every design decision come from hands-on experience in pharma QC and QA. The code is written with AI and tested against them, and every output is reviewed by a human.
Is any of this validated?
No. These are prototypes on synthetic data, not validated for GMP use. Any lab use would need validation under the lab’s own CSV process.
Can we try one?
Yes. Send a short description of the problem and the pilot is scoped together: which tool, which data, and what a useful result would look like.
Why not just use an AI chat like Copilot, ChatGPT or Gemini?
For drafting text, often you should. These tools are about the parts a chat assistant does not give you: fixed rules that a QA reviewer can audit, refusal when evidence is missing, and a record of what the human decided and why.
Contact
Have a lab problem like these, or want to try one of the tools on your own data? Send a few lines about it.