About Services Tools Blog Contact us

Capturing Pages That Build in the Browser

Opens a page in a real browser and returns the finished content after its scripts have run, for pages where a simple download comes back empty.

  • Web Scraping & Crawling
  • Cross-sector
  • Agent skill
  • Productised
  • Local/private
  • 2026

The problem

Product and directory pages that build in the browser never appear in a simple download. The download comes back nearly empty, so catalogues and monitors miss the text they were sent to collect. The failure looks like a short page, so nobody sees that the page never finished loading.

What it does

You give it a page address, a time limit and an optional wait mode. When a simple download is too small, or a full render is requested, the page is opened in a real browser through Playwright and the finished HTML is returned. If the browser fails, the caller gets a warning instead of a silent empty result. This is a setup and usage guide, and it pairs with a second method for sites that block requests before any script runs.

Outcomes

  • Script-built pages give back the text a visitor would see.
  • A timeout is explicit, so a hung page does not stall the batch.
  • A browser failure comes back as a warning instead of an empty success.

Simplified mockup

Interactive preview with sample data. Nothing is sent to a server.

Headless page capture

TLS blocks are handled by the page-fetch tool first. This desk runs only when the plain body is a shell.

Plain fetch Empty shell
Body400 charactersno product nodes
<!DOCTYPE html>
<html lang="en-GB">
<head>
<meta charset="utf-8" />
<title>Loading the catalogue</title>
<!-- client scripts have not run. -->
<link rel="stylesheet"
 href="/assets/app.css">
</head>
<body>
<div id="app" data-page="product">
</div>
<noscript>Drawn by script.
A plain fetch sees only
this empty shell now.</noscript>
<script src="/assets/catalogue.js"
 defer></script>
</body>
</html>
After scripts
Plain body is a shell. Render to fill this side.

Timeout preset

Wait for selector #product-title. Stop at 5 seconds and keep the plain body.

Plain fetch kept Empty shell
catalogue.internal/p/size-chart400 characters
<!DOCTYPE html>
<html lang="en-GB">
<head>
<meta charset="utf-8" />
<title>Loading the catalogue</title>
<!-- client scripts have not run. -->
<link rel="stylesheet"
 href="/assets/app.css">
</head>
<body>
<div id="app" data-page="product">
</div>
<noscript>Drawn by script.
A plain fetch sees only
this empty shell now.</noscript>
<script src="/assets/catalogue.js"
 defer></script>
</body>
</html>
Browser
Preset waits five seconds, then warns.

Capture queue

Today · 4 jobs
Latest renders
PageWaitResultCharacters
/p/ash-court-trainernetworkidle · 20sRendered18,406
/shops/countynetworkidle · 20sRendered9,240
/trade/portalnetworkidle · 20sBrowser failed400
/p/size-chartselector · 5sBrowser timed out400

A failed browser keeps the plain body and raises a warning. The batch moves on.

How it works

  1. Download the page the simple way and check how much came back.
  2. If the page is nearly empty, or a full render was requested, open it in a real browser.
  3. Wait for the chosen signal or the time limit.
  4. Return the finished page content.
  5. If the browser fails, return a warning and keep the simple download for inspection.

Pipeline

flowchart LR
  A[URL] --> B[Plain fetch]
  B --> C{Shell}
  C --> D[Playwright]
  D --> E[Rendered HTML]
  D --> F[Warning]