About Services Tools Blog Contact us

Website Text Collector

Saves the readable text of every page on one website, from a start address or a sitemap. If the live site blocks collection, it falls back to the Wayback Machine or Archive.today.

  • Web Scraping & Crawling
  • Cross-sector
  • Pipeline
  • Productised
  • Local/private
  • 2025

The problem

Content that already sits on a site still has to be copied by hand before anyone can search it. Automatic collection often hits a security check page, so nothing is saved locally to read. A collector that follows every link also wanders across the web, so the start address and the depth need limits.

What it does

You give a start address and a link depth from zero to three. The tool fetches pages on that website only, extracts the readable text and can add the pages to a searchable local store. A sitemap mode follows the site's page list instead. When the live site shows a security check, the Wayback Machine or Archive.today can supply the page. A demo mode writes sample files without using the internet.

Outcomes

  • A site becomes a folder of readable text without hand copying.
  • Collection stays on the one website and within the chosen depth.
  • A security check page falls back to a Wayback Machine or Archive.today copy.
  • Demo mode shows the file writing works without using the internet.

Simplified mockup

Interactive preview with sample data. Nothing is sent to a server.

Same-domain crawl

0123

Follow links from the start address. Stay on notes.internal.

Walk sitemap.xml. Depth is ignored in sitemap mode.

Pause between pages: 1 second
This run
Demo mode is on. Crawl writes the sample corpus.
Archive fallback
Off until you ask. Replaces the challenge row only.

home.txt

Live
URL: https://notes.internal/fitting
Fetched: 10 Oct 2026, 09:14
Source: live
Words: 92

Measure a child’s foot at the end of the day, standing, with weight on both feet. Leave about a thumb’s width beyond the longest toe. Length and width are separate. Write both down before anyone looks at a shoe.

A wide fitting is E or wider. A narrower fitting sits under E. These notes stay on this host. A link that leaves notes.internal is ignored, and the crawl does not follow it.

Same header shape on measure.txt, wide.txt, narrow.txt and autumn.txt. trade-portal.txt is the challenge until the archive fallback runs.

Local index

Nothing from this crawl is searchable until you add it. The query box filters the six titles.

Add this crawl
Index is empty for this host.

How it works

  1. Take a start address and a link depth from 0 to 3, or a sitemap.
  2. Fetch pages that stay on the same website, with a pause between requests.
  3. If the live response is a security check, try the Wayback Machine or Archive.today for that address.
  4. Extract the readable text and write one file per page, with the address at the top.
  5. Optionally add the texts to the local searchable store and report the counts.

Pipeline

flowchart LR
  A[Start or sitemap] --> B[Same domain]
  B --> C[Live fetch]
  C --> D[Archive fallback]
  C --> E[Readable text]
  D --> E
  E --> F[Local index]