Add the clone engine, CLI, tests, CI, and docs

kage renders every page in headless Chrome, snapshots the final DOM, strips all JavaScript, and localises CSS, images, and fonts so a site can be browsed offline as a plain folder of files. The engine is split into small packages: urlx deterministic URL to local-path mapping and scope rules sanitize remove scripts, on* handlers, and javascript: URLs asset rewrite HTML and CSS references, download assets browser headless Chrome pool over the DevTools protocol robots robots.txt matcher clone the orchestrator: a polite resumable breadth-first crawl The cli package wires a cobra and fang command surface with two commands, clone and serve. Every pure package has table tests; the browser and clone packages add Chrome-driven end-to-end tests that skip when no browser is present or under -short. CI runs gofmt, vet, build, race tests, golangci-lint, govulncheck, and a tidy check on Linux and macOS. A goreleaser config fans one tag out to archives, deb/rpm/apk, a Chromium-bundled GHCR image, and the package managers. A tago docs site builds to Pages and Cloudflare.
2026-06-14 18:22:25 +07:00
parent f7c5a9a1c6
commit e6afa91e09
48 changed files with 4697 additions and 0 deletions
@@ -0,0 +1,11 @@
+---
+title: "Getting started"
+linkTitle: "Getting started"
+description: "Install kage and clone your first site into a browsable offline folder in under a minute."
+weight: 10
+featured: true
+---
+
+Three short pages: how kage thinks about cloning a site (render, strip, localise),
+how to install the binary and point it at a browser, and a guided first run that
+ends with a real offline mirror you can click through.
@@ -0,0 +1,63 @@
+---
+title: "Installation"
+description: "Install kage from Go, Homebrew, a release archive, a Linux package, or the container image, and point it at a browser."
+weight: 20
+---
+
+kage is a single binary. Pick whichever channel suits you.
+
+## Go
+
+```bash
+go install github.com/tamnd/kage/cmd/kage@latest
+```
+
+## Homebrew
+
+```bash
+brew install tamnd/tap/kage
+```
+
+## Release archives and Linux packages
+
+Every [release](https://github.com/tamnd/kage/releases) attaches `tar.gz`
+archives (and a `.zip` for Windows) for Linux, macOS, Windows, and FreeBSD, plus
+`.deb`, `.rpm`, and `.apk` packages and a `checksums.txt` with a cosign
+signature. Download the one for your platform, extract `kage`, and put it on your
+`PATH`.
+
+```bash
+# Debian/Ubuntu
+sudo dpkg -i kage_*_linux_amd64.deb
+
+# Fedora/RHEL
+sudo rpm -i kage_*_linux_amd64.rpm
+```
+
+## Container
+
+The image bundles Chromium, so it needs nothing else:
+
+```bash
+docker run -v "$PWD/out:/out" ghcr.io/tamnd/kage clone example.com
+```
+
+The mirror lands in `./out/example.com/` on your host.
+
+## You need a browser
+
+kage drives a real Chrome to render pages. Outside the container image, it needs
+Chrome or Chromium available on the machine. It looks for a system install
+automatically (Google Chrome on macOS and Windows, `google-chrome`/`chromium` on
+Linux). To use a specific binary:
+
+```bash
+kage clone example.com --chrome /path/to/chromium
+# or
+export KAGE_CHROME=/path/to/chromium
+```
+
+If no browser is found, kage's launcher can download a private copy of Chromium
+on first use.
+
+Next: [the quick start](/getting-started/quick-start/).
@@ -0,0 +1,53 @@
+---
+title: "Introduction"
+description: "Why kage renders before it saves, and what it means to strip the JavaScript out of a clone."
+weight: 10
+---
+
+A normal website is not a document; it is a program. The HTML the server sends
+is often a near-empty shell, and the page you actually see is assembled in your
+browser by JavaScript: fetching data, building the DOM, wiring up handlers. That
+is why "Save As" so often fails. You get the shell, not the page, and whatever
+you do get still runs trackers and phones home when you open it.
+
+kage treats a clone as three steps in order.
+
+## 1. Render
+
+Every page is loaded in a real headless Chrome through the DevTools protocol.
+kage navigates to the URL, waits for the network to go quiet, optionally scrolls
+to trigger lazy-loaded images, and then serialises the **final** DOM, the markup
+that exists after the page's JavaScript has finished building it. This is the
+same thing you would see if you opened the page and chose "Inspect".
+
+## 2. Strip
+
+From that captured DOM, kage removes everything executable:
+
+- every `<script>` tag, inline or external;
+- every `on*` event handler attribute (`onclick`, `onload`, and the rest);
+- every `javascript:` URL;
+- `<meta http-equiv="refresh">` redirects and dead resource hints like
+  `<link rel="preload" as="script">`.
+
+What remains is inert. The saved page makes no network calls, runs no code, and
+tracks nothing.
+
+## 3. Localise
+
+A page with no working CSS or images is not much of a clone, so kage keeps the
+parts that define how it looks. It downloads every stylesheet, image, font, and
+media file, rewrites the references in the HTML and inside the CSS (`url()` and
+`@import`) to relative local paths, and rewrites in-scope page links to point at
+the other saved pages. The mirror is fully self-contained: you can move the
+folder anywhere, open it with no network, and click around.
+
+## The shape of a clone
+
+kage crawls breadth-first from a seed URL, staying within the seed's host (and
+optionally its subdomains). It is polite by default: it honours `robots.txt` and
+seeds itself from `sitemap.xml`. Output lands in `kage-out/<host>/`, with pages
+as `<path>/index.html` and assets under a reserved `_kage/` directory alongside
+the crawl state that powers `--resume`.
+
+Next: [install kage](/getting-started/installation/).
@@ -0,0 +1,68 @@
+---
+title: "Quick start"
+description: "From an empty terminal to a self-contained offline mirror you can click through."
+weight: 30
+---
+
+This walks the core loop: clone a small site, look at what landed on disk, and
+serve it back so links and assets resolve the way they would on a real host.
+
+## 1. Clone a site
+
+```bash
+kage clone example.com
+```
+
+kage launches headless Chrome, renders the home page, strips its scripts, and
+follows in-scope links breadth-first. A live counter shows pages, assets, and
+errors as it goes; the final summary tells you where the mirror landed.
+
+```
+kage cloning https://example.com
+done kage-out/example.com
+  pages 12   assets 38
+  open kage serve kage-out/example.com
+```
+
+## 2. Look at what landed
+
+```bash
+ls kage-out/example.com
+```
+
+```
+index.html        # the home page, scripts stripped
+about/index.html  # /about
+_kage/            # localised assets and crawl state
+```
+
+Open `index.html` directly in a browser and it renders offline, with no network.
+Grep it and you will find no `<script>`, no `onclick`, no `javascript:`.
+
+## 3. Serve it back
+
+Opening files directly works, but some sites use root-relative links. `kage
+serve` runs a local static server so everything resolves exactly as it would
+live:
+
+```bash
+kage serve kage-out/example.com
+# open http://127.0.0.1:8800
+```
+
+## 4. Scope a bigger crawl
+
+For a large site, bound the crawl so it does not run away:
+
+```bash
+# Just the docs section, three levels deep, at most 200 pages
+kage clone example.com --scope-prefix /docs --max-depth 3 --max-pages 200
+```
+
+If you stop a run with Ctrl-C, kage saves its state. Run the same command again
+and it resumes, skipping the pages it already wrote.
+
+## Where to go next
+
+- The [guides](/guides/) cover scoping, serving, and resuming in depth.
+- The [CLI reference](/reference/cli/) lists every flag.