Duc-Tam Nguyen
e6afa91e09
kage renders every page in headless Chrome, snapshots the final DOM, strips all JavaScript, and localises CSS, images, and fonts so a site can be browsed offline as a plain folder of files. The engine is split into small packages: urlx deterministic URL to local-path mapping and scope rules sanitize remove scripts, on* handlers, and javascript: URLs asset rewrite HTML and CSS references, download assets browser headless Chrome pool over the DevTools protocol robots robots.txt matcher clone the orchestrator: a polite resumable breadth-first crawl The cli package wires a cobra and fang command surface with two commands, clone and serve. Every pure package has table tests; the browser and clone packages add Chrome-driven end-to-end tests that skip when no browser is present or under -short. CI runs gofmt, vet, build, race tests, golangci-lint, govulncheck, and a tidy check on Linux and macOS. A goreleaser config fans one tag out to archives, deb/rpm/apk, a Chromium-bundled GHCR image, and the package managers. A tago docs site builds to Pages and Cloudflare.
2.3 KiB
title, description, weight
| title | description | weight |
|---|---|---|
| Introduction | Why kage renders before it saves, and what it means to strip the JavaScript out of a clone. | 10 |
A normal website is not a document; it is a program. The HTML the server sends is often a near-empty shell, and the page you actually see is assembled in your browser by JavaScript: fetching data, building the DOM, wiring up handlers. That is why "Save As" so often fails. You get the shell, not the page, and whatever you do get still runs trackers and phones home when you open it.
kage treats a clone as three steps in order.
1. Render
Every page is loaded in a real headless Chrome through the DevTools protocol. kage navigates to the URL, waits for the network to go quiet, optionally scrolls to trigger lazy-loaded images, and then serialises the final DOM, the markup that exists after the page's JavaScript has finished building it. This is the same thing you would see if you opened the page and chose "Inspect".
2. Strip
From that captured DOM, kage removes everything executable:
- every
<script>tag, inline or external; - every
on*event handler attribute (onclick,onload, and the rest); - every
javascript:URL; <meta http-equiv="refresh">redirects and dead resource hints like<link rel="preload" as="script">.
What remains is inert. The saved page makes no network calls, runs no code, and tracks nothing.
3. Localise
A page with no working CSS or images is not much of a clone, so kage keeps the
parts that define how it looks. It downloads every stylesheet, image, font, and
media file, rewrites the references in the HTML and inside the CSS (url() and
@import) to relative local paths, and rewrites in-scope page links to point at
the other saved pages. The mirror is fully self-contained: you can move the
folder anywhere, open it with no network, and click around.
The shape of a clone
kage crawls breadth-first from a seed URL, staying within the seed's host (and
optionally its subdomains). It is polite by default: it honours robots.txt and
seeds itself from sitemap.xml. Output lands in kage-out/<host>/, with pages
as <path>/index.html and assets under a reserved _kage/ directory alongside
the crawl state that powers --resume.
Next: install kage.