Add the clone engine, CLI, tests, CI, and docs

kage renders every page in headless Chrome, snapshots the final DOM, strips all JavaScript, and localises CSS, images, and fonts so a site can be browsed offline as a plain folder of files. The engine is split into small packages: urlx deterministic URL to local-path mapping and scope rules sanitize remove scripts, on* handlers, and javascript: URLs asset rewrite HTML and CSS references, download assets browser headless Chrome pool over the DevTools protocol robots robots.txt matcher clone the orchestrator: a polite resumable breadth-first crawl The cli package wires a cobra and fang command surface with two commands, clone and serve. Every pure package has table tests; the browser and clone packages add Chrome-driven end-to-end tests that skip when no browser is present or under -short. CI runs gofmt, vet, build, race tests, golangci-lint, govulncheck, and a tidy check on Linux and macOS. A goreleaser config fans one tag out to archives, deb/rpm/apk, a Chromium-bundled GHCR image, and the package managers. A tago docs site builds to Pages and Cloudflare.
2026-06-14 18:22:25 +07:00
parent f7c5a9a1c6
commit e6afa91e09
48 changed files with 4697 additions and 0 deletions
@@ -0,0 +1,54 @@
+---
+title: "Configuration"
+description: "Environment variables kage reads, and the layout of a cloned mirror on disk."
+weight: 20
+---
+
+kage is configured almost entirely through command-line flags (see the
+[CLI reference](/reference/cli/)). It reads a couple of environment variables for
+locating the browser.
+
+## Environment variables
+
+| Variable | Meaning |
+|----------|---------|
+| `KAGE_CHROME` | Path to the Chrome/Chromium binary. Takes precedence over autodetection. Equivalent to `--chrome`. |
+| `CHROME_BIN` | Fallback Chrome path, read if `KAGE_CHROME` is unset. |
+
+If neither is set and no system Chrome is found in the usual install locations,
+kage's launcher can download a private copy of Chromium on first use.
+
+## Output layout
+
+A clone of `example.com` lands under `kage-out/example.com/`:
+
+```
+kage-out/example.com/
+├── index.html                  # the home page (/), scripts stripped
+├── about/index.html            # /about
+├── blog/
+│   ├── index.html              # /blog
+│   └── a-post/index.html       # /blog/a-post
+├── _kage/                      # reserved directory
+│   ├── example.com/
+│   │   ├── site.css            # localised stylesheet, url() rewritten
+│   │   ├── logo.png
+│   │   └── fonts/body.woff2
+│   ├── cdn.example.com/        # assets from other hosts, by host
+│   └── state.json              # visited set, for --resume
+└── ...
+```
+
+Key points:
+
+- **Pages become directories.** A page at `/about` is written as
+  `about/index.html`, so a link to `/about` resolves to a real file when served.
+- **Assets live under the reserved directory.** Everything kage downloads, CSS,
+  images, fonts, media, goes under `_kage/<asset-host>/`, mirroring the path it
+  had on its origin. Cross-origin assets are grouped by their own host.
+- **Query strings are folded into the filename.** An asset like
+  `style.css?v=3` is saved with a short hash suffix so two versions never
+  collide.
+- **State lives in the mirror.** `_kage/state.json` records every page written,
+  which is what makes `--resume` able to skip completed work. Rename the reserved
+  directory with `--reserved` if `_kage` would clash with a real path on the site.