Add the clone engine, CLI, tests, CI, and docs

kage renders every page in headless Chrome, snapshots the final DOM, strips all JavaScript, and localises CSS, images, and fonts so a site can be browsed offline as a plain folder of files. The engine is split into small packages: urlx deterministic URL to local-path mapping and scope rules sanitize remove scripts, on* handlers, and javascript: URLs asset rewrite HTML and CSS references, download assets browser headless Chrome pool over the DevTools protocol robots robots.txt matcher clone the orchestrator: a polite resumable breadth-first crawl The cli package wires a cobra and fang command surface with two commands, clone and serve. Every pure package has table tests; the browser and clone packages add Chrome-driven end-to-end tests that skip when no browser is present or under -short. CI runs gofmt, vet, build, race tests, golangci-lint, govulncheck, and a tidy check on Linux and macOS. A goreleaser config fans one tag out to archives, deb/rpm/apk, a Chromium-bundled GHCR image, and the package managers. A tago docs site builds to Pages and Cloudflare.
2026-06-14 18:22:25 +07:00
parent f7c5a9a1c6
commit e6afa91e09
48 changed files with 4697 additions and 0 deletions
@@ -0,0 +1,11 @@
+---
+title: "Reference"
+linkTitle: "Reference"
+description: "The complete kage surface: every command, every flag, and every environment variable."
+weight: 30
+featured: true
+---
+
+The exhaustive reference. The [CLI](/reference/cli/) page lists every command and
+flag; the [configuration](/reference/configuration/) page covers environment
+variables and the output layout on disk.
@@ -0,0 +1,85 @@
+---
+title: "CLI reference"
+description: "Every kage command and flag."
+weight: 10
+---
+
+```
+kage [command] [flags]
+```
+
+Two commands: `clone` fetches a site into an offline folder, `serve` previews
+one. Run `kage <command> --help` for the canonical, up-to-date list.
+
+## kage clone
+
+```
+kage clone <url> [flags]
+```
+
+Renders each page in headless Chrome, strips all JavaScript, localises CSS,
+images, and fonts, and writes a browsable mirror to `<out>/<host>/`.
+
+### Output
+
+| Flag | Default | Meaning |
+|------|---------|---------|
+| `-o, --out` | `kage-out` | Output root; the mirror lands in `<out>/<host>/` |
+| `--reserved` | `_kage` | Reserved directory name for assets and crawl state |
+| `-f, --force` | `false` | Delete any existing mirror for the host before crawling |
+| `--no-resume` | `false` | Do not read or write resume state |
+
+### Scope
+
+| Flag | Default | Meaning |
+|------|---------|---------|
+| `-p, --max-pages` | `0` | Stop after N pages (0 = unlimited) |
+| `-d, --max-depth` | `0` | Link-follow depth cap (0 = unlimited) |
+| `--scope-prefix` | | Only crawl pages whose path starts with this prefix |
+| `--subdomains` | `false` | Treat subdomains of the seed host as in scope |
+| `--exclude` | | Path prefixes to skip (repeatable) |
+| `--traversal` | `bfs` | Frontier order: `bfs` or `dfs` |
+
+### Politeness
+
+| Flag | Default | Meaning |
+|------|---------|---------|
+| `--no-robots` | `false` | Ignore `robots.txt` |
+| `--no-sitemap` | `false` | Do not seed URLs from `sitemap.xml` |
+| `--user-agent` | Chrome UA | User-Agent for asset and robots fetches |
+
+### Rendering
+
+| Flag | Default | Meaning |
+|------|---------|---------|
+| `--scroll` | `false` | Auto-scroll each page to trigger lazy loading |
+| `--settle` | `1.5s` | Network-idle quiet period before snapshotting the DOM |
+| `--render-timeout` | `30s` | Hard cap per page render |
+| `--headful` | `false` | Run Chrome with a visible window (debugging) |
+| `--chrome` | | Path to the Chrome/Chromium binary |
+| `--control-url` | | Attach to an existing Chrome DevTools endpoint |
+| `--keep-noscript` | `false` | Unwrap `<noscript>` content instead of dropping it |
+
+### Concurrency and limits
+
+| Flag | Default | Meaning |
+|------|---------|---------|
+| `--workers` | `4` | Concurrent page render workers |
+| `--asset-workers` | `8` | Concurrent asset download workers |
+| `--browser-pages` | `4` | Chrome page-pool size |
+| `--max-asset-mb` | `25` | Skip assets larger than N MB |
+| `--timeout` | `30s` | Per-request timeout |
+| `-q, --quiet` | `false` | Suppress per-page progress lines |
+
+## kage serve
+
+```
+kage serve [dir] [flags]
+```
+
+Runs a local static file server over a cloned folder. With no `dir`, serves the
+current directory.
+
+| Flag | Default | Meaning |
+|------|---------|---------|
+| `-a, --addr` | `127.0.0.1:8800` | Address to listen on |
@@ -0,0 +1,54 @@
+---
+title: "Configuration"
+description: "Environment variables kage reads, and the layout of a cloned mirror on disk."
+weight: 20
+---
+
+kage is configured almost entirely through command-line flags (see the
+[CLI reference](/reference/cli/)). It reads a couple of environment variables for
+locating the browser.
+
+## Environment variables
+
+| Variable | Meaning |
+|----------|---------|
+| `KAGE_CHROME` | Path to the Chrome/Chromium binary. Takes precedence over autodetection. Equivalent to `--chrome`. |
+| `CHROME_BIN` | Fallback Chrome path, read if `KAGE_CHROME` is unset. |
+
+If neither is set and no system Chrome is found in the usual install locations,
+kage's launcher can download a private copy of Chromium on first use.
+
+## Output layout
+
+A clone of `example.com` lands under `kage-out/example.com/`:
+
+```
+kage-out/example.com/
+├── index.html                  # the home page (/), scripts stripped
+├── about/index.html            # /about
+├── blog/
+│   ├── index.html              # /blog
+│   └── a-post/index.html       # /blog/a-post
+├── _kage/                      # reserved directory
+│   ├── example.com/
+│   │   ├── site.css            # localised stylesheet, url() rewritten
+│   │   ├── logo.png
+│   │   └── fonts/body.woff2
+│   ├── cdn.example.com/        # assets from other hosts, by host
+│   └── state.json              # visited set, for --resume
+└── ...
+```
+
+Key points:
+
+- **Pages become directories.** A page at `/about` is written as
+  `about/index.html`, so a link to `/about` resolves to a real file when served.
+- **Assets live under the reserved directory.** Everything kage downloads, CSS,
+  images, fonts, media, goes under `_kage/<asset-host>/`, mirroring the path it
+  had on its origin. Cross-origin assets are grouped by their own host.
+- **Query strings are folded into the filename.** An asset like
+  `style.css?v=3` is saved with a short hash suffix so two versions never
+  collide.
+- **State lives in the mirror.** `_kage/state.json` records every page written,
+  which is what makes `--resume` able to skip completed work. Rename the reserved
+  directory with `--reserved` if `_kage` would clash with a real path on the site.