Duc-Tam Nguyen
e6afa91e09
kage renders every page in headless Chrome, snapshots the final DOM, strips all JavaScript, and localises CSS, images, and fonts so a site can be browsed offline as a plain folder of files. The engine is split into small packages: urlx deterministic URL to local-path mapping and scope rules sanitize remove scripts, on* handlers, and javascript: URLs asset rewrite HTML and CSS references, download assets browser headless Chrome pool over the DevTools protocol robots robots.txt matcher clone the orchestrator: a polite resumable breadth-first crawl The cli package wires a cobra and fang command surface with two commands, clone and serve. Every pure package has table tests; the browser and clone packages add Chrome-driven end-to-end tests that skip when no browser is present or under -short. CI runs gofmt, vet, build, race tests, golangci-lint, govulncheck, and a tidy check on Linux and macOS. A goreleaser config fans one tag out to archives, deb/rpm/apk, a Chromium-bundled GHCR image, and the package managers. A tago docs site builds to Pages and Cloudflare.
1.4 KiB
title, description, weight
| title | description | weight |
|---|---|---|
| Resuming a run | Pick up an interrupted clone where it left off, and start fresh when you want to. | 30 |
Cloning a large site can take a while, and runs get interrupted: you press Ctrl-C, your laptop sleeps, the network drops. kage is built to pick up where it left off.
How resume works
As it writes each page, kage records it in a small state file inside the mirror,
at <host>/_kage/state.json. When a run ends, for any reason, that file holds
the set of pages already written. Resume is on by default: the next time you
run the same clone, kage loads the state and skips every page it already wrote,
re-crawling only what is left.
kage clone example.com
# ... press Ctrl-C partway through ...
# interrupted; resume state saved (rerun to continue)
kage clone example.com
# resume: 137 pages already done
Ctrl-C is a clean stop: kage cancels in-flight renders, flushes the state file, and exits. You will not lose the pages already written.
Start fresh
To ignore any previous run and rebuild the mirror from scratch, delete the
existing host folder first with --force:
kage clone example.com --force
This removes kage-out/example.com/ before crawling, so nothing from a prior run
carries over.
To run without reading or writing any resume state at all, for a strictly
one-shot clone, use --no-resume:
kage clone example.com --no-resume