Prickles
Prickles
Case file AI6
AIAI Collaboration

Self-Review Pass

Critique your own draft before you hand it back.

The model can draft. The model also has to grade what it drafted before handing back: against the brief, against the spec, against the tests. Pure introspection is not enough.

ByAdam LewisPublished3 May 2026Revised13 Oct 2026Reading13 minVersionv1.2ConfidenceHigh
§0b

Opinion

For years I've told juniors to read their own pull requests before requesting review. The intern's instinct is to type, hit publish, and watch for feedback. The senior engineer's instinct is to draft, then read the diff like a stranger, then fix what was missed, then request review. The agent's instinct is the intern's, only faster: a thousand lines typed, done declared, the human waiting to find the bug. The fix is to give the agent the senior engineer's habit, in a form the protocol can enforce.

The peer-reviewed cluster is the strongest in the AI pillar. Reflexion (Shinn et al., NeurIPS 2023): adding a self-evaluation loop pushes HumanEval pass@1 from 80% to 91%.1 Self-Refine (Madaan et al., 2023): the same finding, the same shape, an LLM provides feedback on its own output and refines iteratively.2 CRITIC (Gou et al., ICLR 2024): tool-interactive self-critique, where the model validates its output against external tools before committing.3 Constitutional AI (Bai et al., Anthropic 2022): the model learns to critique and revise against a constitution rather than against human labels of harmful outputs.4 Anthropic's Building Effective Agents formalises the production pattern as evaluator-optimizer: one call generates, another evaluates, the loop iterates until the artefact passes grading.5

The qualifier is essential. Huang et al. (ICLR 2024) showed that pure introspective self-correction can decrease accuracy: the model's confidence in the revision is uncorrelated with the revision's correctness when no external grader is involved.6 The implication is sharper than “ask the agent to check itself”: the self-review must run against external evidence. Tests, the persistent brief (AI2), the verifiable spec (AI4), types, current docs (AI5): the grader has to be something the model's confidence cannot edit.

The closing rule is editorial. The author edits last. The model can draft prose, plans, schemas, code. The brand voice and the editorial responsibility do not delegate. AI text on Prickles is always edited by a human before it goes out. The same discipline applies in the codebase: the human reads the diff, the human approves the merge, the human takes responsibility for the change. The self-review pass is not a substitute for the human review; it is the work the agent does so the human review does not have to find the obvious bugs.

Copy a note and link

Grab this short comment and drop it into a PR comment or an LLM chat to prompt the right change.

Draft first, then critique what you drafted, then fix it. The agent grades its own work against the brief, the spec and the tests before handing back. The human grades the result. Pure introspection isn't enough: ground the self-review in external evidence (Reflexion, Self-Refine, CRITIC, Constitutional AI; Huang 2024 caveat).

/tenet/self-review-pass/AI6
Copy for paste
§0c

AI eyes only

Rule: draft it, then critique the draft, then fix it. Run an external grader before handing back.

Reject: handing back work that has not been re-checked. Reject: using the model's own sense that “this looks right” as the grader. Reject: silently swallowing a failing check.

Generate: after every change, run the external graders (spec, type-check, lint, build) in that order. Fix any failure before handing back. Name any remaining uncertainty in the handover.

Diagnostic: the handover must cite the grader output. If the grader did not run, the work is not ready.

§0d

Why?

  • Backed by the strongest evidence cluster in the AI pillar: three peer-reviewed papers (Reflexion, Self-Refine, CRITIC) plus Anthropic's Constitutional AI and evaluator-optimizer. Five lines of evidence converging on one finding.
  • Filters the diff before it reaches the human reviewer. Obvious bugs are gone before review starts. The human review focuses on judgement rather than typo-hunting.
  • Absorbs the Huang 2024 qualifier rather than ignoring it. The self-review is grounded in external evidence (tests, brief, types), not in the model's sense that the new draft “looks better.”
  • Compounds with AI1 The Intern Pattern. The self-review is the AI side of the review leg. The human review is the human side. Both gates close before merge.
  • Compounds with AI4 Verifiable Specs. The runnable spec is the grader the self-review runs against. The spec exists for exactly this loop to close.
  • The author edits last. AI prose, plans, schemas, code: all drafts. The brand voice and the editorial responsibility do not delegate. The human takes the final pass.
  • Runs at machine speed. The agent's self-review takes seconds rather than minutes and the cost falls on the agent rather than the human. Cheap insurance against the agent's signature failure mode.
The receipts
Origins, quoted passages, evidence, the strongest counter-argument and the reply.
§1

Origins

The pre-agent ancestor is the senior-engineer habit of reading your own diff before posting it. Code-review folklore has carried this for decades. The move from junior to senior is partly the move from “I typed it. Done” to “I typed it. Let me read it again like a stranger.” Addy Osmani presents the post-agent restatement of the same idea: “treat AI-generated code as a helpful draft that must be verified ... never commit code you can't explain.”7

Anthropic's 2022 Constitutional AI paper opened the formal academic line.4 The model is trained to critique and revise its own outputs against a constitution. The constitution is the external grader. The principle generalised quickly: Reflexion (Shinn et al., NeurIPS 2023) showed verbal self-reflection on task feedback produces large benchmark gains;1 Self-Refine (Madaan et al., 2023) showed iterative self-feedback works cross-domain;2 CRITIC (Gou et al., ICLR 2024) showed tool-interactive self-critique improves results on free-form question answering, mathematical program synthesis and toxicity reduction.3

Huang et al. (2024) is the qualifier the row is built around.6 Pure introspective self-correction without external grounding can decrease accuracy. The model's confidence in its revision is not a reliable signal of the revision's correctness. The implication for the row is sharp: the self-review must ground in external evidence (tests, the persistent brief, types, the runnable spec) not in the model's sense that the new draft “looks better.” The finding strengthens the rule rather than weakening it.

Anthropic's Building Effective Agents (December 2024) shipped the production pattern.5 The evaluator-optimizer workflow names the cycle: one call generates, another evaluates, the loop iterates until the artefact passes grading. The pattern is the agent analogue of pair programming where one engineer types and the other catches bugs: except the typing engineer is also the catching engineer, on a separate pass, against an external rubric.

§2

Quotes

Reflexion agents verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials.

Shinn et al. · Reflexion (NeurIPS 2023)

Motivated by how humans refine their written text, we introduce Self-Refine, an approach for improving initial outputs from LLMs through iterative feedback and refinement.

Madaan et al. · Self-Refine (NeurIPS 2023)

In the evaluator-optimizer workflow, one LLM call generates a response while another provides evaluation and feedback in a loop. … This workflow is particularly effective when we have clear evaluation criteria, and when iterative refinement provides measurable value.

Anthropic · Building Effective Agents (December 2024)

In the context of reasoning, our research indicates that LLMs struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction.

Huang et al. · Large Language Models Cannot Self-Correct Reasoning Yet (ICLR 2024)
§3

Evidence

16 external sources, ranked by author authority. The first five are the canon; expand to see the rest, including the qualifiers and the named opposers. Each links out to its primary source.

  1. 01
    Reflexion: Language Agents with Verbal Reinforcement LearningSupports
    Noah Shinn et al. · 2023
    NeurIPS paper. HumanEval pass@1 91% with Reflexion vs 80% prior SOTA. The single most-cited paper for the self-review pattern at the agent layer.
  2. 02
    Self-Refine: Iterative Refinement with Self-FeedbackSupports
    Aman Madaan et al. · 2023
    Cross-domain confirmation of the Reflexion finding. The same LLM provides feedback on its own output and refines iteratively across math, code, and creative tasks.
  3. 03
    CRITIC: Large Language Models Can Self-Correct with Tool-Interactive CritiquingSupports
    Zhibin Gou et al. · 2024
    ICLR paper. Tool-interactive self-critique. The external-grader requirement the row depends on. Tools are the grader, not introspection.
  4. 04
    Constitutional AI: Harmlessness from AI FeedbackSupports
    Yuntao Bai et al. (Anthropic) · 2022
    Anthropic 2022. The model learns to critique and revise against a constitution. Earliest of the agent-self-critique cluster and the source of the idea.
  5. 05
    Building Effective AgentsSupports
    Anthropic · 2024
    Names the evaluator-optimizer pattern as the production workflow for the row. Direct architectural source.
  1. 06
    Large Language Models Cannot Self-Correct Reasoning YetQualifies
    Jie Huang et al. · 2024
    ICLR paper. Pure introspection without external grounding can decrease accuracy. The load-bearing qualifier. The rule is built around this finding.
  2. 07
    Code Review in the Age of AISupports
    Addy Osmani · 2026
    Practitioner essay. Treat AI-generated code as a helpful draft that must be verified. Never commit code you cannot explain. The editorial responsibility clause.
  3. 08
    How to Write a Good Spec for AI AgentsSupports
    Addy Osmani · 2026
    Verification criteria as a first-class element of the spec. The grader the self-review runs against starts with the spec.
  4. 09
    Effective Context Engineering for AI AgentsSupports
    Anthropic · 2025
    The grader has to be in the agent's surface. Tests, type-check output, the persistent brief: the self-review only works when the grader is something the agent can see.
  5. 10
    Building Agents with the Claude Agent SDKSupports
    Anthropic · 2025
    Subagent-based evaluator-optimizer at production scale. Useful as a worked example of the row in shipped systems.
  6. 11
    How I Use LLMs to Help Me Write CodeSupports
    Simon Willison · 2025
    “The one thing you absolutely cannot outsource to the machine is testing that the code actually works.” The human side of the second pass. The rule's editorial responsibility clause.
  7. 12
    Sleeper Agents: Training Deceptive LLMsQualifies
    Hubinger et al. · 2024
    Background safety case. Confident, fluent generations can be wrong in ways the generator cannot detect. Reinforces the case for an external grader rather than introspection.
  8. 13
    Tree of Thoughts: Deliberate Problem Solving with Large Language ModelsSupports
    Yao et al. · 2023
    NeurIPS paper. Branching self-evaluation extends the Reflexion / Self-Refine finding to structured search. Another angle on the same evidence.
  9. 14
    Chain-of-Thought prompting and structured prompting best practicesSupports
    Anthropic · 2025
    First-party prompting guidance includes a self-critique step (`<thinking>` tags, then refinement). The row's pattern is part of Anthropic's house style.
  10. 15
    AutoGen: agent reflection patternsSupports
    Microsoft Research · 2023 onwards
    Cross-framework instantiation. Reflection is a first-class abstraction in AutoGen. The row is the convergence point, not vendor-specific.
  11. 16
    Cline AI-editable rulesQualifies
    Cline · 2024
    Worked example of an AI-editable agent-rules file. Useful as a cautionary tale: agent-edited self-review prompts need the same external grading the row demands of code.

Sixteen sources. The supports are the strongest peer-reviewed cluster on the site: Reflexion (Shinn), Self-Refine (Madaan), CRITIC (Gou), Constitutional AI (Bai), and Anthropic's evaluator-optimizer pattern. The qualifier further down is load-bearing: pure introspection without external grounding fails. The rule is built around that finding, not against it. The opposing voice (review fatigue) gets the reply.

§4

Examples

Viewing: TypeScript.
Avoid

Before single-shot. Agent drafts, releases, waits for the human to find the bug.

Filewrite-rescue-hedgehog.ts
async function writeRescueHedgehog(spec: Spec): Promise<string> {
  const draft = await agent.generate(spec);
  return draft;
}
Prefer

After draft, critique, refine. The author edits last; the agent self-edits first.

Filewrite-rescue-hedgehog.ts
async function writeRescueHedgehog(spec: Spec): Promise<string> {
  const draft = await agent.generate(spec);

  const critique = await agent.critique(draft, spec,
    "find every off-by-one and every assumption that contradicts the spec");

  const refined = await agent.refine(draft, critique);

  await runTests(refined);
  return refined;
}
§4b

Enforcement

Viewing: TypeScript.

Apply these rules in dangerfile.ts. The full enforcement across every tenet lives on the implementation page.

RuleToolCatches
tsc --noEmit (grader 1)tscimaginary signatures, fields that do not exist and the wrong arity. The first external grader for the self-review.
vitest run (grader 2)Vitestthe verifiable spec from AI4. The test suite grades the work, so the self-review must report a green run.
ESLint (grader 3)ESLintrule violations the agent introduced and didn't notice. The lint output is part of the grader package.
Danger PR body sectionDanger JSPRs missing the Self-Review section, or missing the external-grader naming inside it. Without the section, the loop never closed.
dangerfile.tsconfiguration snippet
import { danger, fail, warn, message } from 'danger';

const body = (danger.github.pr.body ?? '').toLowerCase();
const hasSelfReview = /## self[- ]review\b/.test(body);
const hasGrader = /spec passes|tests pass|type-check passes|lint passes/.test(body);

if (!hasSelfReview) {
  fail('AI6: PR body must include a "## Self-Review" section.');
}
if (!hasGrader) {
  fail('AI6: self-review must name the external grader (spec / tests / type-check / lint).');
}
message('AI6: human review follows the agent self-review. Both gates close before merge.');
§4c

AI rules

File.cursor/rules/ai6-self-review-pass.mdc
---
alwaysApply: true
---

## Prickles AI6: Self-Review Pass

Draft, then critique what you drafted, then fix it, before you hand anything back. The model grades its own draft, the loop that Reflexion, Self-Refine, CRITIC and Anthropic's evaluator-optimizer share.

Pure introspection isn't enough: Huang et al. 2024. Ground the self-review in tests, docs, types and the persistent brief. Introspection without external grounding can decrease accuracy.

Every change opens its own self-PR. The agent reviews against the brief, the verifiable spec and the test runner before declaring done. The human reviews after.

The author edits last. AI prose, AI plans, AI schemas: all drafts. The final edit is human. The brand voice and the editorial responsibility do not delegate.

Repo layout, CI, and ESLint wiring for these paths live on /implementation, not repeated on every tenet.

§5

Counter-argument

Counter

The honest steelman is review fatigue. Two reviews per change (the agent's self-review then the human's) doubles the friction without doubling the value, and teams under deadline pressure will rationalise the self-review away as ceremony. The Cognition / Devin school argues for the opposite extreme: full autonomy with retroactive review at the merge gate, on the bet that the agent's self-grading approaches a human's quickly enough that the upfront pass becomes overhead. If a future self-review is good enough, the human pass may be the last review the rule needs.

§6

Counter-argument retort

Reply

The review-fatigue argument concedes the rule for the present and asks for it to retire as models improve. Two responses. Both empirical.

First, the cost is overstated. The agent's self-review is fast (seconds rather than minutes) and the cost falls on the agent rather than the human. The human review that follows is faster precisely because the obvious bugs have already been caught. The self-review filters the diff so the human review can focus on judgement rather than typo-hunting. The two passes catch different things. The cited studies measure accuracy, not cost: CRITIC found tool-grounded critique improves results, and Huang et al. found self-correction without external feedback can make them worse.

Second, the optimistic future does not retire the rule. It changes who runs the second pass. If the agent's self-review approaches human-quality, then the human review can become a sampling rather than a comprehensive reading: closer to a pull-request audit than a pull-request approval. Anthropic's evaluator-optimizer pattern already anticipates this; in production agent systems the “evaluator” is sometimes a separate model rather than a human. But until that future arrives reliably, the human remains the second grader and the agent's self-review is the cheapest way to make the human's grading job easier.

The Devin-flavoured full-autonomy bet does not survive the Huang qualifier. Without the external grader, the agent's confidence in its own work is uncorrelated with the work's correctness. Long-running autonomous agents that grade their own output against their own intuition produce fluent confusion at machine speed. The loop only works when the grader is something the model's confidence cannot edit. Pair this with AI1 The Intern Pattern: the four moves and the self-review reinforce each other. The agent's self-review is the AI side of the review leg.

§7

Notes

  1. [1]Noah Shinn et al.: Reflexion: Language Agents with Verbal Reinforcement Learning (NeurIPS 2023). HumanEval pass@1 91% with Reflexion vs 80% prior SOTA. Reflexive self-evaluation produces large, repeatable benchmark gains.
  2. [2]Aman Madaan et al.: Self-Refine: Iterative Refinement with Self-Feedback (NeurIPS 2023). The same LLM generates output, provides feedback, and refines iteratively. Cross-domain confirmation of the Reflexion finding.
  3. [3]Zhibin Gou et al.: CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (ICLR 2024). The model validates its output by interacting with appropriate tools: the external grader the rule depends on.
Disagree? Found a hole in the argument? Take issue with this tenet →
Last revised: 2026-10-13