Duc-Tam Nguyen
e6afa91e09
kage renders every page in headless Chrome, snapshots the final DOM, strips all JavaScript, and localises CSS, images, and fonts so a site can be browsed offline as a plain folder of files. The engine is split into small packages: urlx deterministic URL to local-path mapping and scope rules sanitize remove scripts, on* handlers, and javascript: URLs asset rewrite HTML and CSS references, download assets browser headless Chrome pool over the DevTools protocol robots robots.txt matcher clone the orchestrator: a polite resumable breadth-first crawl The cli package wires a cobra and fang command surface with two commands, clone and serve. Every pure package has table tests; the browser and clone packages add Chrome-driven end-to-end tests that skip when no browser is present or under -short. CI runs gofmt, vet, build, race tests, golangci-lint, govulncheck, and a tidy check on Linux and macOS. A goreleaser config fans one tag out to archives, deb/rpm/apk, a Chromium-bundled GHCR image, and the package managers. A tago docs site builds to Pages and Cloudflare.
2.2 KiB
2.2 KiB
title, description, weight
| title | description | weight |
|---|---|---|
| Configuration | Environment variables kage reads, and the layout of a cloned mirror on disk. | 20 |
kage is configured almost entirely through command-line flags (see the CLI reference). It reads a couple of environment variables for locating the browser.
Environment variables
| Variable | Meaning |
|---|---|
KAGE_CHROME |
Path to the Chrome/Chromium binary. Takes precedence over autodetection. Equivalent to --chrome. |
CHROME_BIN |
Fallback Chrome path, read if KAGE_CHROME is unset. |
If neither is set and no system Chrome is found in the usual install locations, kage's launcher can download a private copy of Chromium on first use.
Output layout
A clone of example.com lands under kage-out/example.com/:
kage-out/example.com/
├── index.html # the home page (/), scripts stripped
├── about/index.html # /about
├── blog/
│ ├── index.html # /blog
│ └── a-post/index.html # /blog/a-post
├── _kage/ # reserved directory
│ ├── example.com/
│ │ ├── site.css # localised stylesheet, url() rewritten
│ │ ├── logo.png
│ │ └── fonts/body.woff2
│ ├── cdn.example.com/ # assets from other hosts, by host
│ └── state.json # visited set, for --resume
└── ...
Key points:
- Pages become directories. A page at
/aboutis written asabout/index.html, so a link to/aboutresolves to a real file when served. - Assets live under the reserved directory. Everything kage downloads, CSS,
images, fonts, media, goes under
_kage/<asset-host>/, mirroring the path it had on its origin. Cross-origin assets are grouped by their own host. - Query strings are folded into the filename. An asset like
style.css?v=3is saved with a short hash suffix so two versions never collide. - State lives in the mirror.
_kage/state.jsonrecords every page written, which is what makes--resumeable to skip completed work. Rename the reserved directory with--reservedif_kagewould clash with a real path on the site.