About Services Tools Blog Contact us

Catalogue Duplicate Content Check

Scores each product description against other pages on the web and lists the closest matches for an editor to review.

  • Web Scraping & Crawling
  • Footwear retail
  • Pipeline
  • Productised
  • Hybrid
  • 2025

The problem

A retailer cannot tell which product descriptions were copied from other sites. Copied text can hurt search ranking and can bring a complaint from the original publisher. Checking by hand means pasting each style into a search engine and reading the results.

What it does

The pipeline imports the product list used for shopping adverts and scores each description against other pages. It writes a review list with each score and the web addresses of the matches. An editor rewrites the worst copies first. Matches on the retailer's own site are dropped so pages are not flagged for matching themselves.

Outcomes

  • Copied descriptions are ranked by score.
  • Editors see the page a description may be copied from.
  • The retailer's own pages are not counted as duplicates.

Simplified mockup

Interactive preview with sample data. Nothing is sent to a server.

Description review

Styles
8
Rewrite first
3
Own site only
1
Under 0.40
3
Scores under 0.40 stay collapsed.
StyleScoreMatchesState
0.91
2 Rewrite
0.74
1 Review
0.62
2 Review
Salt Court
0.55
1 Own site
0.48
1 Open
Meadow Mary Jane
0.33
1 Collapsed
Canal Slip-on
0.22
0 Collapsed
Beacon Sandal
0.11
0 Collapsed
Ash Lane Chelsea · closest 0.91 Own site dropped
northpier.example/ash-lane-chelsea 0.91 fellandco.example/boots/ash-lane 0.88

Ours · shop.example/ash-lane-chelsea

The Ash Lane Chelsea is a leather upper with a stitched apron toe and a stacked heel that holds its shape through the day. Elastic side gussets and a leather sole.

Theirs · northpier.example/ash-lane-chelsea

This chelsea is a leather upper with a stitched apron toe and a stacked heel that holds its shape through the day. Offered in black and tan.

Rewrite queue

Queue rewrite on the review tab sends the top style here. Scores under 0.40 never enter the queue.

Review
Ash Lane Chelsea
0.91 · two outside pages
North Pier Trainer
0.74 · one outside page
Queued
Mill House Loafer
0.62 · waiting on copy
Done
Rewrite brief
Ash Lane Chelsea: replace the highlighted apron-toe sentence. Do not keep the stacked heel line.
North Pier Trainer: still open. The overlap is the wet-pavement outsole line.
Salt Court: not a duplicate. The only hit is this shop, so Hide own site drops it.

How it works

  1. Import the shopping advert product list into a table of products.
  2. Search the web for pages that closely match each description.
  3. Score the overlap and keep the matching web addresses.
  4. Drop matches on the retailer's own site.
  5. Show a review list sorted by score.

Pipeline

flowchart LR
  A[Shopping feed] --> B[Descriptions]
  B --> C[Web matches]
  C --> D[Score]
  D --> E[Drop own site]
  E --> F[Review list]