Duc-Tam Nguyen
e6afa91e09
kage renders every page in headless Chrome, snapshots the final DOM, strips all JavaScript, and localises CSS, images, and fonts so a site can be browsed offline as a plain folder of files. The engine is split into small packages: urlx deterministic URL to local-path mapping and scope rules sanitize remove scripts, on* handlers, and javascript: URLs asset rewrite HTML and CSS references, download assets browser headless Chrome pool over the DevTools protocol robots robots.txt matcher clone the orchestrator: a polite resumable breadth-first crawl The cli package wires a cobra and fang command surface with two commands, clone and serve. Every pure package has table tests; the browser and clone packages add Chrome-driven end-to-end tests that skip when no browser is present or under -short. CI runs gofmt, vet, build, race tests, golangci-lint, govulncheck, and a tidy check on Linux and macOS. A goreleaser config fans one tag out to archives, deb/rpm/apk, a Chromium-bundled GHCR image, and the package managers. A tago docs site builds to Pages and Cloudflare.
54 lines
2.3 KiB
Markdown
54 lines
2.3 KiB
Markdown
---
|
|
title: "Introduction"
|
|
description: "Why kage renders before it saves, and what it means to strip the JavaScript out of a clone."
|
|
weight: 10
|
|
---
|
|
|
|
A normal website is not a document; it is a program. The HTML the server sends
|
|
is often a near-empty shell, and the page you actually see is assembled in your
|
|
browser by JavaScript: fetching data, building the DOM, wiring up handlers. That
|
|
is why "Save As" so often fails. You get the shell, not the page, and whatever
|
|
you do get still runs trackers and phones home when you open it.
|
|
|
|
kage treats a clone as three steps in order.
|
|
|
|
## 1. Render
|
|
|
|
Every page is loaded in a real headless Chrome through the DevTools protocol.
|
|
kage navigates to the URL, waits for the network to go quiet, optionally scrolls
|
|
to trigger lazy-loaded images, and then serialises the **final** DOM, the markup
|
|
that exists after the page's JavaScript has finished building it. This is the
|
|
same thing you would see if you opened the page and chose "Inspect".
|
|
|
|
## 2. Strip
|
|
|
|
From that captured DOM, kage removes everything executable:
|
|
|
|
- every `<script>` tag, inline or external;
|
|
- every `on*` event handler attribute (`onclick`, `onload`, and the rest);
|
|
- every `javascript:` URL;
|
|
- `<meta http-equiv="refresh">` redirects and dead resource hints like
|
|
`<link rel="preload" as="script">`.
|
|
|
|
What remains is inert. The saved page makes no network calls, runs no code, and
|
|
tracks nothing.
|
|
|
|
## 3. Localise
|
|
|
|
A page with no working CSS or images is not much of a clone, so kage keeps the
|
|
parts that define how it looks. It downloads every stylesheet, image, font, and
|
|
media file, rewrites the references in the HTML and inside the CSS (`url()` and
|
|
`@import`) to relative local paths, and rewrites in-scope page links to point at
|
|
the other saved pages. The mirror is fully self-contained: you can move the
|
|
folder anywhere, open it with no network, and click around.
|
|
|
|
## The shape of a clone
|
|
|
|
kage crawls breadth-first from a seed URL, staying within the seed's host (and
|
|
optionally its subdomains). It is polite by default: it honours `robots.txt` and
|
|
seeds itself from `sitemap.xml`. Output lands in `kage-out/<host>/`, with pages
|
|
as `<path>/index.html` and assets under a reserved `_kage/` directory alongside
|
|
the crawl state that powers `--resume`.
|
|
|
|
Next: [install kage](/getting-started/installation/).
|