Verify the API
Don't trust the training.
The model's mental model of next, drizzle, and every other package on disk is stale by definition. Look it up before you write the call.
Opinion
This is the strongest standalone tenet in the AI pillar. Three threads converge and they all point the same way. Knowledge cutoff is structural. Hallucinated APIs are the canonical failure mode. The fix is documented and works. There is no honest steelman against “look it up before you call it”.
Every model goes out with its knowledge “fixed at what it was trained on before deployment.”1 Claude 4.7's training data is bounded; GPT-5's is bounded; Gemini's is bounded. APIs change between cutoff and now: minor versions deprecate methods, major versions reshape entire surfaces and the most popular libraries change the fastest. The model does not know which version of react-hook-form the project installed. It will confidently generate the version it knows, which is the version it was trained on, which may not be the version on disk.
Simon Willison says it without hedging: “They’ll absolutely make mistakes—sometimes subtle, sometimes huge.” His example: if a human collaborator “hallucinated a non-existent library or method you would instantly lose trust in them.”2 Amazon Science published the fix in 2024: Documentation Augmented Generation (DAG) “significantly improves performance for low frequency APIs” by checking against an API index before generation.3 The Context7 MCP server is the production version of the same idea: it “pulls up-to-date, version-specific documentation and code examples straight from the source and places them directly into your prompt” for thousands of libraries.4 Grep against the local package, web fetch against the changelog, MCP query against the database: the surface differs, the rule does not.
The larger claim is that this is one of the few AI tenets with no good non-AI analogue. A human engineer who guesses at an API call is being lazy. An LLM that guesses is being correct about its own behaviour. It generates the most likely token. The most likely token reflects training. Training is stale. The fix has to live at the agent-rules layer, because the model cannot fix itself: the same probabilities that drive the hallucination drive the agent's confidence in the hallucination.
Copy a note and link
Grab this short comment and drop it into a PR comment or an LLM chat to prompt the right change.
Don't trust the training. The model's `next` and `drizzle` are stale by definition. Look up the docs, grep the source, query the MCP, then write the call. Grounding wins over recall on every API older than the cutoff. /tenet/verify-the-api/AI5
AI eyes only
Rule: do not trust the training. Look up before you call.
Reject: emitting an API call without verifying the signature against current docs. Reject: “there is probably a method called”-style guesses. Reject: inventing helpers that “should exist” in the framework.
Generate: for external APIs, query resolve-library-id + query-docs via Context7 MCP (or equivalent) before writing the call. For internal APIs, grep the workspace for the function definition first.
Diagnostic: cite the source for every API call (doc URL, file path, or grep result). No cite means a guess.
Why?
- Kills the canonical AI failure mode. Hallucinated APIs are the most common, most-reported, most-cited bug class in agent-written code. Grounding fixes them at the source.
- Cheap and fast to enforce. One MCP query, one grep, one docs URL: the verification adds seconds, the missed verification can cost an afternoon.
- The fix is published. Amazon Science's DAG paper, Anthropic's MCP and Context7, the broader retrieval-augmented-generation literature: the rule rests on results, not opinion.
- Compounds with AI4 Verifiable Specs. The spec runs the call against the library, the test fails when the API doesn't exist and the rule is enforced in the build rather than in review.
- AI-specific in a way most rules aren't. The failure mode is structural to how LLMs generate text. No human-only equivalent of the rule exists.
- The rule survives any specific tool. MCP servers, web fetch, vendored docs, repo grep, type-check: pick the surface that fits your stack; the principle is grounding, not a particular product.
- Pairs with strict types. A type system catches the calls the agent typed against an imagined signature. The lookup catches the API surface the agent imagined into existence. Types and grounding cover different halves of the same bug.
Origins
The pattern showed up first as folklore: by 2023, every Twitter thread about Copilot and ChatGPT had at least one example of a hallucinated API. fs.readFileSync with imaginary options. useEffect overloads that didn't exist. NPM packages whose names looked plausible and whose npm install step failed because the package was never published. The community noticed the failure mode before the literature named it.
Simon Willison's March 2025 essay How I Use LLMs to Help Me Write Code is the single best practitioner statement of both problem and fix.2 Treat the agent as “an over-confident pair programming assistant who's lightning fast at looking things up”: useful, but liable to make mistakes “sometimes subtle, sometimes huge ... like hallucinating non-existent libraries or methods.” The mitigation: never outsource the check that the code works.
The structural idea came from the cutoff literature. By 2025 the term “knowledge cutoff” was canonical: any model's knowledge is “fixed at what it was trained on before deployment.”1 The Frontiers in AI 2025 survey of LLM hallucinations separates hallucinations the prompt causes from those intrinsic to the model;5 vLLM's December 2025 HaluGate work shows token-level signals can flag the unsupported spans of a finished answer before it reaches the user.6
The fix moved from folklore to documented practice in 2024. Amazon Science's Documentation Augmented Generation paper formalised the “look it up first” pattern: maintain an API index, retrieve from it before generating low-frequency calls, ground the model in current docs.3 Anthropic's Model Context Protocol shipped the production pipework for the same idea, with Context7 (Upstash) and a constellation of equivalent MCP servers for libraries, databases, repos, and changelogs.4 Anthropic's Effective Context Engineering and Writing Effective Tools for AI Agents papers position MCP-style retrieval as the load-bearing counter to training staleness.
Quotes
They’ll absolutely make mistakes—sometimes subtle, sometimes huge. These mistakes can be deeply inhuman—if a human collaborator hallucinated a non-existent library or method you would instantly lose trust in them.
Documentation Augmented Generation (DAG) significantly improves performance for low frequency APIs (increase to 47.94% with DAG) but negatively impacts high frequency APIs when using sub-optimal retrievers (a 39.02% absolute drop). To mitigate this, we propose to intelligently trigger DAG where we check against an API index or leverage Code LLMs’ confidence scores to retrieve only when needed.
Context7 pulls up-to-date, version-specific documentation and code examples straight from the source — and places them directly into your prompt.
Since large language models are pretrained, any model's knowledge is fixed at what it was trained on before deployment; information about events after this date is absent from the model's training data.
Evidence
16 external sources, ranked by author authority. The first five are the canon; expand to see the rest, including the qualifiers and the named opposers. Each links out to its primary source.
- 01Practitioner statement of both problem and fix. Names hallucinated libraries / methods as the canonical failure mode. Advocates verification of every external call before commit.
- 02Documents the DAG approach: maintain an API index. Retrieve from it before generating low-frequency calls. Ground the model. The published industrial fix.
- 03Context7 MCP serverSupportsProduction MCP for version-specific library docs. Exposes the look-it-up rule as a tool the agent can call without a context-switch.
- 04Model Context ProtocolSupportsThe standard that lets agents query live tools (docs, databases, repos, changelogs) rather than relying on training. The pipework underneath the rule.
- 05Glob and grep as first-class primitives. The agent has to be told to use retrieval. The rule is the telling.
- 06Tools (including doc-lookup tools) deserve as much prompt engineering as overall prompts. The fix only works if the lookup tool fails loudly when the agent misuses it.
- 07Industry survey of hallucination causes and mitigations. Lists outdated training data among the traditional causes, then reframes hallucination as an incentive problem: training and evaluation reward confident guessing.
- 08HaluGate: Token-Level TruthQualifiesDetection-side work. Token-level signals can intercept hallucinations during generation. Useful for runtime guardrails on top of the rule.
- 09Peer-reviewed survey. Code-API hallucination is one of the highest-prevalence categories. The rule sits in a problem space the research community has independently named.
- 10Background safety case. Confident, fluent generations can be wrong in ways the generator cannot detect. The verification has to come from outside the model.
- 11Foundational paper for the broader retrieval-then-generate pattern. The lineage the look-it-up rule belongs to. Predates the agent era and survives it.
- 12Agents call tools. Tools query current state. The agent's call is grounded in the tool's response. Architectural source for how the rule lives in production.
- 13Code Review in the Age of AISupportsPractitioner essay on review-side discipline. Catching hallucinated APIs in review is one of the most-cited specific use cases for the new review style.
- 14DeepWiki MCP / GitHub MCP serverSupportsAlternative MCP-style retrievers for repo and library context. The principle generalises beyond Context7. Multiple production tools instantiate it.
- 15Cline rules: AI-editable conventionsQualifiesWhere the rule lives operationally for one popular agent. Useful as a worked example of how the rule encodes in agent-rules files.
- 16next.js docs and changelogSupportsConcrete example. Next.js APIs change frequently across versions. The prickles AGENTS.md notes call out exactly this problem and tell the agent to read the local docs first.
Sixteen sources. The supports cluster on the practitioner and tooling line: Simon Willison's essay, Amazon Science's DAG paper (Jain et al.), the Context7 MCP server and the Model Context Protocol spec from Anthropic. The structural-cutoff papers sit further down. The qualifiers cover the cases where the agent does check but chooses to ignore the result. Nobody in the list opposes the rule outright. The counter-argument (the bigger-models-will-fix-it argument) weakens with every published benchmark. The reply addresses why.
Enforcement
Apply these rules in eslint.config.mjs. The full enforcement across every tenet lives on the implementation page.
| Rule | Tool | Catches |
|---|---|---|
| import/no-unresolved | eslint-plugin-import | imports of modules that don't exist on disk. Most-direct catch for hallucinated dependencies. |
| import/named | eslint-plugin-import | named imports that don't exist in the source module. Catches the agent calling import { useEffectAsync } from 'react' when no such export exists. |
| import/no-deprecated | eslint-plugin-import | calls to APIs marked deprecated in the source. The agent's training favours the older API; the rule catches the gap. |
| @typescript-eslint/no-unsafe-call | typescript-eslint | calls against the any type: typically the agent giving up and silently routing around a missing type. Catches the Hail Mary call. |
| tsc --noEmit | tsc | imaginary signatures, fields that do not exist and the wrong arity. The compiler is the cheapest verification of the call surface. |
eslint.config.mjsconfiguration snippet
import { defineConfig } from 'eslint/config';
import tseslint from 'typescript-eslint';
import importPlugin from 'eslint-plugin-import';
export default defineConfig({
files: ['**/*.{ts,tsx}'],
extends: [tseslint.configs.base],
languageOptions: {
parserOptions: { projectService: true, tsconfigRootDir: import.meta.dirname },
},
plugins: { import: importPlugin },
rules: {
'import/no-unresolved': 'error',
'import/named': 'error',
'import/no-extraneous-dependencies': 'error',
'import/no-deprecated': 'warn',
'@typescript-eslint/no-unsafe-call': 'error',
'@typescript-eslint/no-unsafe-member-access': 'error',
}
});Apply these rules in eslint.config.mjs. The full enforcement across every tenet lives on the implementation page.
| Rule | Tool | Catches |
|---|---|---|
| import/no-unresolved | eslint-plugin-import | imports of modules that don't exist on disk. Most-direct catch for hallucinated dependencies. |
| import/named | eslint-plugin-import | named imports that don't exist in the source module. Catches the agent calling import { useEffectAsync } from 'react' when no such export exists. |
| import/no-deprecated | eslint-plugin-import | calls to APIs marked deprecated in the source. The agent's training favours the older API; the rule catches the gap. |
| @typescript-eslint/no-unsafe-call | typescript-eslint | calls against the any type: typically the agent giving up and silently routing around a missing type. Catches the Hail Mary call. |
| tsc --noEmit | tsc | imaginary signatures, fields that do not exist and the wrong arity. The compiler is the cheapest verification of the call surface. |
eslint.config.mjsconfiguration snippet
import { defineConfig } from 'eslint/config';
import tseslint from 'typescript-eslint';
import importPlugin from 'eslint-plugin-import';
export default defineConfig({
files: ['**/*.{ts,tsx}'],
extends: [tseslint.configs.base],
languageOptions: {
parserOptions: { projectService: true, tsconfigRootDir: import.meta.dirname },
},
plugins: { import: importPlugin },
rules: {
'import/no-unresolved': 'error',
'import/named': 'error',
'import/no-extraneous-dependencies': 'error',
'import/no-deprecated': 'warn',
'@typescript-eslint/no-unsafe-call': 'error',
'@typescript-eslint/no-unsafe-member-access': 'error',
}
});Apply these rules in .github/workflows/ai5-verify-the-api.yml. The full enforcement across every tenet lives on the implementation page.
| Rule | Tool | Catches |
|---|---|---|
| import/no-unresolved | eslint-plugin-import | missing modules: hallucinated dependencies catch here first. |
| import/named | eslint-plugin-import | named imports the source module doesn't actually export. |
| GitHub Actions: API grounding | GitHub Actions | PRs that pass review eyes but call against a stale surface. Type-check + import-resolve + spec runs in one job. |
.github/workflows/ai5-verify-the-api.ymlconfiguration snippet
name: ai5-verify-the-api
on:
pull_request:
jobs:
api-grounding:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npm ci
- name: Type-check the call surface
run: npx tsc --noEmit
- name: Resolve every import
run: npx eslint --rule '{"import/no-unresolved":"error"}' src/
- name: Diff against last-known package surface
run: npx --yes @arethetypeswrong/cli --pack .
- name: Snapshot critical SDK exports
run: npx api-extractor run --localApply these rules in pyproject.toml. The full enforcement across every tenet lives on the implementation page.
| Rule | Tool | Catches |
|---|---|---|
| F401 / F405 / F811 / F821 | Ruff | unresolved names, redefined imports, and bare from X import * patterns. Catches the agent reaching for an attribute that doesn't exist in the module. |
| TID252 | Ruff | relative imports that drift past their package boundary. The agent's instinct to flatten imports is constrained. |
| mypy --strict | mypy | imaginary signatures, fields that do not exist and the wrong arity. Strict mode is the Python equivalent of tsc --noEmit. |
pyproject.tomlconfiguration snippet
[tool.ruff.lint]
select = [
"F401",
"F405",
"F811",
"F821",
"TID252",
]
[tool.mypy]
strict = true
warn_unused_ignores = true
warn_redundant_casts = trueApply these rules in pmd-ai5.xml. The full enforcement across every tenet lives on the implementation page.
| Rule | Tool | Catches |
|---|---|---|
| MissingOverride | PMD | methods that look like overrides but aren't. Common when the agent guesses at a parent-class signature that doesn't exist. |
| UnnecessaryImport | PMD | imports the agent added speculatively. Often a signal that the planned call was removed but the import remained: a fingerprint of a hallucinated initial draft. |
| japicmp | japicmp | API drift between dependency versions. Fails the build when the agent's call surface refers to methods that no longer exist on the classpath version. |
pmd-ai5.xmlconfiguration snippet
<?xml version="1.0"?>
<ruleset name="AI5: verify the api">
<rule ref="category/java/bestpractices.xml/MissingOverride"/>
<rule ref="category/java/codestyle.xml/UnnecessaryImport"/>
<rule ref="category/java/codestyle.xml/UseUnderscoresInNumericLiterals"/>
<rule name="ApiSurfaceFrozen"
class="net.sourceforge.pmd.lang.rule.XPathRule"
message="AI5: this method signature has changed since the last release; verify against the public API surface."
language="java">
<description>Pair with japicmp or similar to fail builds where the call surface drifts unannounced.</description>
</rule>
</ruleset>Apply these rules in .php-cs-fixer.dist.php. The full enforcement across every tenet lives on the implementation page.
| Rule | Tool | Catches |
|---|---|---|
| no_unused_imports | PHP-CS-Fixer | imports the agent added speculatively but never used. Same fingerprint as the Java equivalent: often the residue of a hallucinated initial draft. |
| fully_qualified_strict_types | PHP-CS-Fixer | fully qualified names left inline when the symbol is already imported or belongs to the current namespace. It only shortens the name; finding a class that does not exist is PHPStan's job, below. |
| PHPStan level max | PHPStan | imaginary methods, missing classes, signature drift. The PHP analogue of tsc --noEmit and mypy --strict. |
.php-cs-fixer.dist.phpconfiguration snippet
<?php
return (new PhpCsFixer\Config())
->setRiskyAllowed(true)
->setRules([
'no_unused_imports' => true,
'fully_qualified_strict_types' => true,
'native_function_invocation' => ['include' => ['@all']],
])
->setFinder(
(new PhpCsFixer\Finder())->in(__DIR__ . '/src')
);
# PHPStan rules level max + bleeding-edge cover the static side. For
# external HTTP APIs, run a contract-test step (pact-php / openapi
# validator) so the agent's call surface is verified against the wire.AI rules
.cursor/rules/ai5-verify-the-api.mdc--- alwaysApply: true --- ## Prickles AI5: Verify the API Check the current docs or source before you write any call to a framework, library or SDK. The model's knowledge of each one stops at its training date: the cutoff is structural, not incidental. Don't trust the training. Look up the docs, grep the source, query the MCP, then write the call. Grounding wins over recall on every API older than the cutoff. Hallucinated APIs are the canonical AI failure mode in code. The fix is documented, well-named (Documentation Augmented Generation), and works. Treat the agent's first plausible API call as a draft. Verify against current source before letting it go out. The most likely token reflects the training and the training is stale.
Repo layout, CI, and ESLint wiring for these paths live on /implementation, not repeated on every tenet.
Counter-argument
The honest steelman is that the rule is overhead the next model will retire. Every cycle of larger context windows, web-search-by-default, and tool-augmented inference reduces the gap between training and now. The 2026 frontier models come with built-in retrieval, browse-on-demand, and live MCP integrations enabled by default. If the next generation of agents structurally grounds every call before writing, the “look it up first” rule becomes a discipline that no longer deserves its place: true but no longer instructive.
Counter-argument retort
The bigger-models-will-fix-it argument concedes the rule for the present and asks for it to retire in the future. The future may very well retire it. The present has not.
Live retrieval shrinks the gap, but it does not close it. The model still has to decide what to retrieve and that decision still runs through training-shaped probabilities. The agent that doesn't know which docs to look up is left looking up the wrong ones, or none. Anthropic's own Effective Context Engineering paper makes this explicit: glob and grep are first-class primitives because the model still needs to be told to use them, even when retrieval is enabled.7 The default, with no rule, is for the agent to type the most likely call and ground only when prompted.
Empirically, the failure mode persists across model generations. GPT-5, Claude 4.7, and Gemini 2.5 Pro all hallucinate APIs at non-trivial rates on contemporary benchmarks. The rate falls with each generation but does not approach zero. Sleeper-agent research at Anthropic shows a harder deeper problem: once a model has learned a deceptive behaviour, standard safety training can fail to remove it, and adversarial training can teach the model to hide it better.8 Verification has to come from outside the model.
The strongest version of the rule survives even the optimistic future: ground every external-API call against current evidence before writing. If the next generation of agents structurally grounds every call by default, the rule has been honoured at the platform layer rather than the team layer, which would be a victory rather than a refutation. Until then, the team layer has to do the grounding. Pair this with AI4 Verifiable Specs. The spec runs the call against the library. The test fails when the API doesn't exist. The rule is enforced in the build.
Notes
- [1]Wikipedia contributors: “Knowledge cutoff”. “Any model's knowledge is fixed at what it was trained on before deployment; information about events after this date is absent from the model's training data.” The structural idea for why staleness is not incidental.
- [2]Simon Willison: How I Use LLMs to Help Me Write Code (March 2025). The most-quoted practitioner essay on the agent-coding loop. Names hallucinated APIs as one of the canonical failure modes.
- [3]Nihal Jain et al. (Amazon Science): On Mitigating Code LLM Hallucinations with API Documentation. “Documentation Augmented Generation (DAG) significantly improves performance for low frequency APIs.” Documented industrial fix for the failure mode.
- [4]Upstash: Context7 MCP server. “Up-to-date, version-specific documentation and code examples straight from the source.” Production tooling for the look-it-up rule.
- [5]Frontiers in Artificial Intelligence: Survey and Analysis of Hallucinations in Large Language Models (2025). Code-API hallucination is one of the highest-prevalence categories. The rule sits in a research-validated problem space.
- [6]vLLM: HaluGate: Token-Level Truth (December 2025). Token-level signals can detect hallucinations during generation. Useful for building guardrails that intercept the call before it goes out, not just after.
- [7]Anthropic: Effective Context Engineering for AI Agents (September 2025). Glob and grep are first-class primitives because the model still needs to be told to use them, even when retrieval is enabled.
- [8]Hubinger et al. (Anthropic): Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training (January 2024). Background context for why agent output cannot be trusted by default and why grounding / verification gates matter.