Notes on evaluating AI for scholarly writing

How to read these notes

Method

How The Clerk's code works today, with when it entered the code.

Development observation

Something seen while building The Clerk: a single instance, not a tested result.

Validated finding

A result from a designed study. None are reported on this site.

These notes were last checked against the code on 23 October 2026.

A model’s output is a claim to check

• Method Content checks in code. When a revision is proposed, a deterministic check compares it with the original for citations, numbers, p-values, years, hypothesis labels, table and figure references, DOIs and quotations. A revision that changed protected content cannot be applied in one click. In the code since September 2026.
• Method Quotes checked against the source. When Deep Research reads a journal's author guidelines, its official links go through the normal download, and every quote is verified against the downloaded pages before it is used. Conventions it observes in recent papers are kept as advice, never as requirements. In the code since September 2026.
• Method Papers confirmed before they are cited. In Find gaps in this journal, every paper in the report must be one The Clerk gathered or one OpenAlex confirms; others are removed and listed. Each gap's strength (stated and checked, stated, or inferred) is set in code, not by the model. In the code since September 2026.
• Method Style reviews say what they are. AI Tell-Signs is a style-risk review of writing patterns. It always states that it is not proof that a text was AI-generated. In the code since September 2026.

Detector scores are signals

A detector score is a signal for your own review, never proof of how a text was written.

Method The Clerk asks for Pangram 4 explicitly, the model Pangram's own web app runs, and Detector check shows scores for human review and offers no rewrite. In the code since September 2026.

Development observations

Development observation Recorded in the code in September 2026. Detectors can disagree with one another. During development, Pangram 3.3.2 read one paragraph as human (medium confidence) where Pangram 4 read the same paragraph as AI-generated (high confidence). One paragraph is an observation, not a finding; it is one reason Detector check shows scores for human review and offers no rewrite.