Prickles
Prickles
Case file TA2
TATooling

Duplication Detection

Eyeballs miss copy-paste.

Two functions with different signatures can do exactly the same thing, and human eyes will miss it every time. Run a tool. Run several. Detect duplication of logic, not just text.

ByAdam LewisPublished3 May 2026Revised13 Oct 2026Reading12 minVersionv1.2ConfidenceHigh
§0b

Opinion

I've done enough code review on enough teams to know that the duplication a human catches is the duplication that lives in the same file or the same pull request. The duplication that is released, that lives in production for two years, that breaks when one of two parallel implementations gets patched and the other does not: that duplication never shows up in review. It is too far apart in the file tree, too different in the variable names, too disguised by a wrapping abstraction. The eye does not catch it. The token-counter does.

The literature on this is decades old and it makes a distinction Adam keeps reaching for: the four clone types.1 Type-1 is identical code with whitespace differences. Type-2 is the same code with renamed identifiers. Type-3 is the same loop with one extra log line, or one reordered branch. Type-4 is the case I started with: two functions, completely different syntax, that compute the same answer. The first three are detectable with deterministic tools and have been since the 1990s. The fourth is the hardest, and trained detectors did well on it before the LLM studies: TreeCen scored an F1 of 0.99 on BigCloneBench's weak Type-3/Type-4 pairs in 2022. Since 2023, LLMs can read both functions and sometimes notice the equivalence.2

The mistake the existing TA2 is correcting is the assumption that jscpd green means duplication-free.3 jscpd is a Rabin-Karp token tool. It catches Type-1 reliably, Type-2 only through its opt-in ignoreIdentifiers flag, Type-3 only through its opt-in maxGapLines and similarity passes, and is silent on Type-4 by construction. The honest version of this tenet is “run jscpd in CI” and “run a structural pass periodically” and “ask the agent during review whether anything in this change duplicates anything in the codebase.” Three different tools, three different clone types, three different cadences. The cost of any one of them is small. The cost of merging a near-duplicate that lives for two years is enormous.

The cross-cut is F3 Don't Repeat Yourself. Detection is not extraction. The tool catches the pattern. The team decides whether to extract. Sandi Metz's wrong-abstraction warning is the same warning a Type-2 jscpd hit deserves: two functions that look the same today might diverge tomorrow and the abstraction that pretends they belong together can be more expensive than the duplication.4

Copy a note and link

Grab this short comment and drop it into a PR comment or an LLM chat to prompt the right change.

Eyeballs miss copy-paste. Run jscpd in CI for Type-1 and Type-2 clones; Semgrep or ast-grep weekly for Type-3. An LLM review pass on the change set for Type-4. Detect duplication of logic, not just text.

/tenet/duplication-detection/TA2
Copy for paste
§0c

AI eyes only

Rule: jscpd catches Type 1, and Types 2 and 3 only with its opt-in passes. The AI catches Type 4 (semantic).

Reject: releasing with jscpd over the threshold. Reject: dismissing a flagged duplicate without a reason. Reject: declaring duplication-free without an LLM pass for semantic equivalents.

Generate: run jscpd in CI as a build gate. After every change, ask the AI reviewer “is anything in this change semantically duplicating anything else in the codebase?” Surface and resolve.

Diagnostic: every duplicate the tooling flags is either deduplicated or justified inline. Type-4 surfaces only via the LLM pass. The absence of jscpd hits is not the absence of duplication.

§0d

Why?

  • Catches the duplicate the eye misses. Cross-file, cross-module, cross-team near-duplicates are the class of duplication review never finds and a token tool finds in seconds.
  • Stratified by clone type. jscpd for Type-1 and Type-2 in CI; Semgrep or ast-grep for Type-3 on a schedule; LLM-assisted detection for Type-4 in review. Three tools, each with its own cadence and cost profile.
  • Detection is not extraction. The tool flags the duplicate. The team decides whether to extract per F3 DRY. Detection is asymmetric in a way extraction is not.
  • Reaches the semantic clone class current LLMs can detect and deterministic tools cannot. Two functions with different signatures computing the same answer is the case the canon explicitly delegates to AI tooling.
  • Ratchets down. .jscpd.json with a threshold of zero on greenfield codebases; directory-scoped overrides for legacy zones with a paydown plan. The threshold only ever moves toward zero.
  • Cheap on the hot path. The Type-1 and Type-2 pass costs single-digit seconds in CI. The Type-3 weekly pass costs minutes. Only the Type-4 LLM pass costs real money, and only on the change set, not the repo.
  • Honest about its perimeter. Each tool comes with a stated detection ceiling. The tenet does not pretend a clean run means duplication-free.
The receipts
Origins, quoted passages, evidence, the strongest counter-argument and the reply.
§1

Origins

Code-clone research as a formal discipline begins with Brenda Baker's 1995 paper On Finding Duplication and Near-Duplication in Large Software Systems, which used suffix-tree algorithms to find textually similar regions in a million lines of system code.7 Baker's idea (that duplication is a code-quality property worth measuring with tooling, not just intuiting at review) established the field.

Ira Baxter and his collaborators introduced AST-based clone detection in 1998, which moved the work from token sequences to abstract syntax trees and made Type-2 detection (renamed identifiers) reliable.8 A decade later Lingxiao Jiang, Ghassan Misherghi, Zhendong Su and Stephane Glondu published DECKARD at ICSE 2007, which used AST-vector clustering to push into Type-3 territory at scale.9

The four-clone-types taxonomy crystallised through a series of survey papers in the mid-2000s: Roy and Cordy's 2007 technical report A Survey on Software Clone Detection Research is the canonical reference, and every modern clone-detection paper still cites its Section 7.2 taxonomy.1 The same taxonomy underwrites the International Workshop on Software Clones (IWSC) that has been the forum for the field for two decades.

The arrival of LLMs that can do Type-4 detection is recent. The clearest empirical case is in the ICPC 2024 paper Investigating the Efficacy of LLMs for Code Clone Detection5 and the separate arXiv study Towards Understanding the Capability of LLMs on Code Clone Detection.2 Neither reports reliable detection. Dou et al. found GPT-4 and GPT-3.5 beat the non-LLM detectors they tested at recalling BigCloneBench's Type-4 pairs, yet with a plain prompt they caught only 15% and 9% of them, and results varied by language. Khajezade and colleagues' preliminary study gave ChatGPT an F1 of 0.877 on Java-Ruby pairs and 0.878 on Java-Java, and found that the prompt and the difficulty of the problem change the result.

Production tooling for the first three clone types has been mature for a decade. jscpd, PMD CPD and SonarQube CPD all use Rabin-Karp token matching for Type-1 and Type-2 with a small slice of Type-3.3 Semgrep and ast-grep are not replacements for DECKARD. DECKARD clusters similar subtrees, so it finds clones nobody knew were there. Semgrep and ast-grep match a pattern someone has already written: metavariables abstract identifiers, and Semgrep's ... and ast-grep's $$$ skip inserted statements, so once a team has found a duplicated shape, one rule catches its renamed and lightly edited copies. The question this tenet answers is which tool runs on which cadence. The larger question of whether to extract is a different debate handled by F3 DRY.

§2

Quotes

This paper describes how a program called dup can be used to locate instances of duplication or near-duplication in a software system. Dup reports both textually identical sections of code and sections that are the same textually except for systematic substitution of one set of variable names and constants for another.

Brenda S. Baker, On Finding Duplication and Near-Duplication in Large Software Systems (1995)

Two code fragments can be similar based on the similarity of their program text or they can be similar in their functionalities without being textually similar. … These types of clones not only define an increasing level of subtlety from Type I through Type IV but also the analytical complexity and sophistication in detecting such clones increases from Type I through Type IV with Type IV being the highest.

Chanchal K. Roy & James R. Cordy, A Survey on Software Clone Detection Research (2007)

Once an abstraction is proved wrong the best strategy is to re-introduce duplication and let it show you what’s right.

Sandi Metz, The Wrong Abstraction (2016)

By building a mono-lingual and cross-lingual CCD dataset derived from CodeNet, we first investigated two different prompts using ChatGPT to detect Type-4 code clones in Java-Java and Java-Ruby pairs in a zero-shot setting. … ChatGPT surpasses the baselines in cross-language CCD attaining an F1-score of 0.877 and achieves comparable performance to fully fine-tuned models for mono-lingual CCD, with an F1-score of 0.878.

ICPC 2024, Investigating the Efficacy of LLMs for Code Clone Detection
§3

Evidence

18 external sources, ranked by author authority. The first five are the canon; expand to see the rest, including the qualifiers and the named opposers. Each links out to its primary source.

  1. 01
    On Finding Duplication and Near-Duplication in Large Software SystemsSupports
    Brenda S. Baker · 1995
    The progenitor: clone detection as a tooling discipline. Suffix-tree algorithm for textual duplication that established the field.
  2. 02
    Clone Detection Using Abstract Syntax TreesSupports
    Ira D. Baxter et al. · 1998
    ICSM 1998. Moved clone detection from token sequences to ASTs. Established Type-2 detection (renamed identifiers) as a reliable tooling target.
  3. 03
    DECKARD: Scalable and Accurate Tree-based Detection of Code ClonesSupports
    Lingxiao Jiang et al. · 2007
    ICSE 2007. AST-vector clustering that pushed into Type-3. Later deterministic tools (SourcererCC, NiCad, CCAligner) outperform it on BigCloneBench.
  4. 04
    A Survey on Software Clone Detection ResearchSupports
    Chanchal K. Roy & James R. Cordy · 2007
    The canonical four-clone-types taxonomy. Type-1, Type-2, Type-3, Type-4: the categorisation every modern clone-detection paper still cites.
  5. 05
    Comparison and Evaluation of Clone Detection ToolsQualifies
    Stefan Bellon et al. · 2007
    IEEE TSE. The empirical comparison: every detection family has a different recall/precision profile. No single tool dominates. Underwrites the multi-tool cadence model.
  1. 06
    Towards Understanding the Capability of LLMs on Code Clone DetectionSupports
    Shihan Dou et al. · 2023
    Empirical study of ten LLMs against eight non-LLM detectors. GPT-4 and GPT-3.5 beat the non-LLM detectors on Type-4 recall, yet with a plain prompt caught only 15% and 9% of Type-4 pairs, and results vary by language.
  2. 07
    Investigating the Efficacy of LLMs for Code Clone DetectionSupports
    Mohamad Khajezade et al. · 2024
    ICPC 2024. A preliminary exploratory study of ChatGPT on Type-4 clones in Java-Java and Java-Ruby pairs: F1 0.878 and 0.877, beating the baselines on the cross-language pairs, with results that depend on the prompt and the difficulty of the problem.
  3. 08
    Assessing the Code Clone Detection Capability of LLMsSupports
    Zixian Zhang & Takfarinas Saber · 2024
    Tests GPT-3.5 and GPT-4 on BigCloneBench and GPTCloneBench and finds both weak on the most complex Type-4 clones. A qualifier on the Type-4 result, not a confirmation.
  4. 09
    TreeCen: Building a Tree-Graph for Semantic Clone DetectionQualifies
    Yutao Hu et al. · 2022
    ASE 2022. A trained detector: AST centrality vectors fed to a decision-tree classifier, F1 0.99 on BigCloneBench's weak Type-3/Type-4 pairs. Evidence that learned models reached Type-4 before LLMs, not that pure determinism can.
  5. 10
    jscpdSupports
    Andrey Kucherenko · 2024
    Token-based, Rabin-Karp algorithm. Reliable on Type-1 by default. Type-2 and Type-3 only through opt-in passes (ignoreIdentifiers, maxGapLines, similarity). Silent on Type-4. The right default for the CI gate stage.
  6. 11
    PMD CPD documentationSupports
    PMD maintainers · 2024
    Token-based clone detector with language-aware tokenisers. Same Type-1/2 ceiling as jscpd. The Java-side CI referent.
  7. 12
    SonarQube CPD: Manage Duplicated CodeSupports
    SonarSource · 2012
    Statement-based detection tuned to suppress false positives like repeated import blocks. Reaches into Type-2 and partial Type-3.
  8. 13
    Semgrep AST detection documentationSupports
    Semgrep maintainers · 2024
    AST-based pattern matching with metavariables. The right tool for the Type-3 weekly pass. Pattern syntax abstracts identifiers and reaches structural duplication.
  9. 14
    ast-grepSupports
    Herrington Darkholme · 2024
    Tree-sitter-based structural search and rewriting. Same shape as Semgrep. Pseudo-code patterns with metavariables. Strong on Types 2 and 3.
  10. 15
    GitHub Copilot Duplication Detection FilterSupports
    GitHub · 2024
    Production-grade lexeme-based filter checking 65-lexeme suggestions against a public-code corpus. Proof that LLM-adjacent stacks can do find-code-like-this at scale.
  11. 16
    The Wrong AbstractionQualifies
    Sandi Metz · 2016
    Duplication is far cheaper than the wrong abstraction. The reply: this is a warning about extraction, not detection; TA2 is the latter.
  12. 17
    The DRY Principle and Incidental DuplicationQualifies
    Anthony Sciamanna · 2018
    The case that not all duplication is meaningful. The steelman for treating detector hits as conversation starters, not refactor obligations.
  13. 18
    Refactoring 2nd ed., Duplicated CodeSupports
    Martin Fowler · 2018
    The principle this tenet enforces. Duplicated Code, listed second after Mysterious Name in the smell catalogue. Pairs each instance with the Extract Function refactoring that retires it.

Eighteen sources. The supports include the canonical clone-detection literature (Baker's 1995 paper, Baxter et al. on AST clones, Jiang's DECKARD, Roy & Cordy's 2007 survey) and the tools' own documentation. The qualifiers are Bellon et al.'s tool comparison, TreeCen's learned Type-4 detection from before the LLMs, Metz on the wrong abstraction and Sciamanna on incidental duplication. None of the eighteen opposes the tenet.

§4b

Enforcement

Viewing: TypeScript.

Apply these rules in .jscpd.json. The full enforcement across every tenet lives on the implementation page.

RuleToolCatches
jscpdjscpdType-1 (identical) clones via Rabin-Karp token matching. Type-2 (renamed) only when ignoreIdentifiers is set, and the config below does not set it. The CI gate.
ast-grep structuralast-grepType-2 reliably and meaningful Type-3 via tree-sitter pattern matching with metavariables. Weekly scheduled pass.
Semgrep duplication patternsSemgrepType-2 and Type-3 via AST pattern matching with metavariables. Originally a security tool; the duplication use is adjacent.
SonarQube CPDSonarQube CPDtoken-based detection for TypeScript: at least 100 duplicated tokens across at least 10 lines (only Java is statement-based). Differences in indentation and string literals are ignored.
Copilot duplication filterGitHub Copilot duplication filter65-lexeme suggestion checks against a public-code corpus. Production-grade lexeme-based filter at scale.
LLM Type-4 review passagent rule in CLAUDE.md / .cursor/rulesType-4 (semantically equivalent, syntactically different) clones in the change set; the review prompt asks the model to identify functions in the diff that duplicate logic elsewhere.
.jscpd.jsonconfiguration snippet
{
  "threshold": 0,
  "reporters": ["html", "console", "markdown"],
  "ignore": [
    "**/*.spec.ts",
    "**/*.test.ts",
    "**/node_modules/**",
    "**/dist/**",
    "**/.next/**"
  ],
  "absolute": true,
  "gitignore": true,
  "format": ["typescript", "javascript", "tsx", "jsx"],
  "minTokens": 50,
  "minLines": 5,
  "mode": "strict"
}
§4c

AI rules

File.cursor/rules/ta2-duplication-detection.mdc
---
alwaysApply: true
---

## Prickles TA2: Duplication Detection

Run jscpd in CI for every commit. Threshold zero on greenfield code. Directory-scoped overrides for legacy zones with a paydown plan.

Run a structural pass with Semgrep or ast-grep on a weekly schedule. The AST tools reach Type-2 reliably and meaningful Type-3.

Run an LLM duplication-review pass on the change set during AI review. The model catches Type-4 (semantically equivalent, syntactically different) clones that no deterministic tool can.

Detection is not extraction. The tool flags the duplicate. The engineer extracts on the second occurrence (F3 DRY). Don't wait for three.

Repo layout, CI, and ESLint wiring for these paths live on /implementation, not repeated on every tenet.

§5

Counter-argument

Counter

The honest steelman is Sandi Metz's.4 Detection without judgement is exactly the failure mode that creates the wrong abstraction: jscpd sees twenty shared tokens, the team rushes to extract a parameter, two months later one caller needs a different shape and the abstraction has to be torn down. The duplication was cheaper than the abstraction it was eliminating. Anthony Sciamanna's work on incidental duplication makes the same point: not every shared shape is meaningful.6

§6

Counter-argument retort

Reply

Metz's wrong-abstraction warning4 is doing something, but it is a warning about extraction rather than about detection. The tenet is the latter. The team still applies judgement at the moment of decision. Jscpd does not file the pull request. The cost of the detection pass is small (one CI step, ten seconds, no human attention required) and the cost of releasing a near-duplicate that lives for two years is enormous. The detection is asymmetric in the way Metz's extraction is not.

Sciamanna's incidental-duplication objection6 is correct. His own test settles it: “are we looking at syntax duplication or knowledge duplication?” Code that looks the same but does a different job is not duplication, and F3 leaves it alone. Code that repeats the same knowledge is, and F3 extracts it on the second occurrence, not at Sciamanna's third strike. The detection tool flags the shape and the human reads the call sites. If the shape is incidental the team excludes that block with an ignorePattern in .jscpd.json and moves on. The exclusion is editable, audit-able, and reviewable in exactly the way an inline eslint-disable is not.

The harder steelman is the cost-of-Type-4 objection: the LLM pass costs more than the token pass and the false-positive rate is higher. The reply is the cadence. Type-1 and Type-2 run on every commit. The cost is a few seconds and the recall is high. Type-3 runs on a weekly schedule against the whole repo. The cost is a few minutes and the recall is medium. Type-4 runs as part of the AI review pass on the change set rather than the whole repo: AI6 Self-Review Pass is the right surface for it. Three different cadences, three different cost profiles, three different rates of false positive. Detecting duplication of logic rather than text needs all three cadences, each matched to what its own tool catches.

§7

Notes

  1. [1]Chanchal K. Roy & James R. Cordy: A Survey on Software Clone Detection Research, Queen's University Technical Report 541 (2007). The canonical four-clone-types taxonomy: Type-1 (identical), Type-2 (renamed), Type-3 (with edits), Type-4 (semantically equivalent, syntactically different). Cited by every modern clone-detection paper.
  2. [2]Shihan Dou et al.: Towards Understanding the Capability of LLMs on Code Clone Detection (arXiv 2308.01191, 2023). Empirical study showing GPT-4 and GPT-3.5 beat the non-LLM detectors it tested on Type-4 recall, though with a plain prompt they found only 15% and 9% of Type-4 pairs, and results vary by language.
  3. [3]jscpd maintainers: jscpd, the JavaScript copy-paste detector. Token-based, Rabin-Karp algorithm. Reliable on Type-1 by default. Type-2 only with the opt-in ignoreIdentifiers/ignoreLiterals flags (the mild/weak modes only drop whitespace and comments). Type-3 only through the opt-in maxGapLines or similarity passes. Silent on Type-4. The right default for a CI gate.
Disagree? Found a hole in the argument? Take issue with this tenet →
Last revised: 2026-10-13