frostbyte_neo/kage
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

Feature paulgraham in the README and docs, split ZIM and binary packing

Browse Source 
Rewrite the README around a real example, mirroring paulgraham.com for
offline reading, and split packing into two clean sections: a single ZIM
file (with what ZIM is and how to read it back through Kiwix) and a
self-contained binary. Re-record the demo gif against paulgraham.com and
add a screenshot of the native window serving the essays offline. Carry the
same framing into the docs intro pages and the packing guide, and cut the
v0.1.1 release notes.
This commit is contained in:
Duc-Tam Nguyen
2026-06-15 00:01:01 +07:00
parent 3af26ae0e5
commit 5b4b523419
9 changed files with 208 additions and 270 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+4 -1
View File
@@ -6,6 +6,8 @@ All notable changes to kage are recorded here. The format follows
## [Unreleased]
## [0.1.1] - 2026-06-14
### Added
- `kage pack <mirror-dir>` packs a cloned folder into one distributable file.
@@ -63,5 +65,6 @@ can browse offline, with every script stripped out.
a multi-arch container image on GHCR (Chromium bundled), checksums, SBOMs, and
a cosign signature, all cut from one version tag by GoReleaser.
[Unreleased]: https://github.com/tamnd/kage/compare/v0.1.0...HEAD
[Unreleased]: https://github.com/tamnd/kage/compare/v0.1.1...HEAD
[0.1.1]: https://github.com/tamnd/kage/compare/v0.1.0...v0.1.1
[0.1.0]: https://github.com/tamnd/kage/releases/tag/v0.1.0
README.md
+105 -107
View File
@@ -6,25 +6,17 @@
[![Go Report Card](https://goreportcard.com/badge/github.com/tamnd/kage)](https://goreportcard.com/report/github.com/tamnd/kage)
[![License](https://img.shields.io/github/license/tamnd/kage)](./LICENSE)
**kage** (影, "shadow") clones a website into a self-contained folder you can
browse offline, with all the JavaScript stripped out. It renders every page in
headless Chrome, snapshots the final rendered DOM, removes every script and
event handler, then downloads the CSS, images, and fonts and rewrites them to
local paths. The result looks like the live site but runs no code.
**kage** (影, "shadow") clones a website into a folder you can browse offline, with every script stripped out. It opens each page in real headless Chrome, waits for the page to settle, snapshots the DOM a human would have seen, then deletes all the JavaScript and pulls the CSS, images, and fonts down to local paths. What lands on disk looks like the live site and runs no code.
[Install](#install) • [Commands](#commands) • [Clone](#clone) • [Pack](#pack-it-into-one-file) • [Native viewer](#a-native-window-not-a-browser-tab) • [How it works](#how-it-works)
[Install](#install) • [Quick start](#quick-start) • [Commands](#commands) • [Clone](#clone) • [Pack](#pack-it-into-one-file) • [Native window](#a-real-window-not-a-browser-tab) • [How it works](#how-it-works)
![kage cloning a site, packing it into one file, and serving it back offline](docs/static/demo.gif)
![kage cloning paulgraham.com, packing it into one file, and serving it back offline](docs/static/demo.gif)
Saving a page with "Save As" gives you a copy that still phones home, still runs
analytics, and often renders blank because the markup is built by JavaScript at
runtime. kage takes the opposite approach: it drives a real browser, waits for
the page to settle, captures the DOM a human would have seen, and then strips
every script out of it. What lands on disk is inert. No tracking, no network
calls, no surprises, just a folder of `.html` files you can open straight from
disk or pack into a single file to hand to someone.
You already know the problem. You hit "Save As" on a page you want to keep, and six months later you open it to find a blank screen, a spinner that never stops, or a copy that still tries to phone home to an analytics server that no longer exists. The page was never really yours. It was a thin client for someone else's JavaScript.
Full reference and guides live at [kage.tamnd.com](https://kage.tamnd.com).
kage takes the other road. It drives a real browser, lets the page finish doing whatever it does, grabs the finished result, and then rips every script out of it. No tracking, no network calls, no surprises. Just `.html` files you can open straight off disk, hand to a friend, or pack into a single file and forget about for a decade.
Full docs and guides live at **[kage.tamnd.com](https://kage.tamnd.com)**.
## Install
@@ -32,23 +24,44 @@ Full reference and guides live at [kage.tamnd.com](https://kage.tamnd.com).
go install github.com/tamnd/kage/cmd/kage@latest
```
Or grab a prebuilt archive, `.deb`/`.rpm`/`.apk` package, or checksum from the
[releases](https://github.com/tamnd/kage/releases), or run the container image,
which bundles Chromium:
Prefer a prebuilt binary? Grab an archive, a `.deb`/`.rpm`/`.apk`, or a checksum from [releases](https://github.com/tamnd/kage/releases). Or skip installing Chrome yourself and use the container image, which bundles Chromium:
```bash
docker run --rm -v "$PWD/out:/out" ghcr.io/tamnd/kage clone example.com
docker run --rm -v "$PWD/out:/out" ghcr.io/tamnd/kage clone paulgraham.com
```
kage drives a real browser, so it needs Chrome or Chromium on the host. It finds
a system install automatically; point it at a specific binary with `--chrome` or
the `KAGE_CHROME` environment variable. The container image needs nothing extra.
kage drives a real browser, so it needs Chrome or Chromium on the host. It finds a system install on its own; point it somewhere specific with `--chrome` or the `KAGE_CHROME` environment variable. The container needs nothing extra.
Shell completion is built in: `kage completion bash|zsh|fish|powershell`.
Shell completion ships in the box: `kage completion bash|zsh|fish|powershell`.
## Quick start
Let's mirror Paul Graham's essays so you can read them on a plane, on a laptop with no wifi, or in the year 2050 after the site has finally changed its design:
```bash
# 1. Clone the site into $HOME/data/kage/paulgraham.com/
kage clone paulgraham.com
# 2. Read it back offline in your browser
kage serve $HOME/data/kage/paulgraham.com
# open http://127.0.0.1:8800
```
That's the whole loop. Every essay, every image, every stylesheet, frozen on your disk and runnable with zero network. The next two steps are optional but nice: collapse the whole thing into one file, and pop it open in its own window.
```bash
# 3. Squeeze the mirror into a single shareable file
kage pack paulgraham.com # -> paulgraham.com.zim
kage open paulgraham.com.zim
# 4. Or into one executable that *is* the site
kage pack paulgraham.com --format binary -o paulgraham
./paulgraham # serves itself, needs nothing installed
```
## Commands
| Command | Does |
| Command | What it does |
| --- | --- |
| `kage clone <url>` | render a site in headless Chrome and write a browsable, script-free mirror |
| `kage serve [dir]` | preview a cloned folder over a local HTTP server |
@@ -58,112 +71,106 @@ Shell completion is built in: `kage completion bash|zsh|fish|powershell`.
## Clone
```bash
# Clone a whole site into $HOME/data/kage/<host>/
kage clone https://example.com
# The whole site, into $HOME/data/kage/<host>/
kage clone https://paulgraham.com
# Limit the crawl
kage clone example.com --max-pages 200 --max-depth 3
# Just the first 50 pages, two links deep, for a quick taste
kage clone paulgraham.com --max-pages 50 --max-depth 2
# Only a section of the site
kage clone example.com --scope-prefix /docs
# Only one section of a bigger site
kage clone go.dev --scope-prefix /doc
# Include subdomains, and trigger lazy-loaded images by scrolling
# Pull in subdomains too, and scroll each page to trip lazy-loaded images
kage clone example.com --subdomains --scroll
# Re-render every page in place to pull in changed content
kage clone example.com --refresh
# Come back next month and re-render in place to catch new essays
kage clone paulgraham.com --refresh
```
A clone is a polite breadth-first crawl. It honours `robots.txt`, seeds itself
from `sitemap.xml`, and scopes to the seed host unless you widen it. It is also
idempotent: each page is keyed by the file it writes, so the same URL reached
over http and https, with or without a trailing slash, is fetched once.
Re-running resumes where it left off; Ctrl-C saves state on the way out.
`--refresh` re-renders in place, `--force` wipes and starts clean.
A clone is a polite, breadth-first crawl. It reads `robots.txt`, seeds itself from `sitemap.xml`, and stays on the seed host unless you tell it otherwise. It is also stubbornly idempotent: each page is keyed by the file it writes, so the same essay reached over http and https, with or without a trailing slash, gets fetched exactly once. Hit Ctrl-C and it saves its place on the way out; run it again and it picks up where it stopped. `--refresh` re-renders in place, `--force` wipes the host and starts clean.
Common flags:
The flags you'll actually reach for:
| Flag | Default | Meaning |
|------|---------|---------|
| `-o, --out` | `$HOME/data/kage` | Output root; the mirror lands in `<out>/<host>/` |
| `-p, --max-pages` | `0` | Stop after N pages (0 = unlimited) |
| `-d, --max-depth` | `0` | Link-follow depth cap (0 = unlimited) |
| `--scope-prefix` | | Only crawl pages whose path starts with this prefix |
| `-p, --max-pages` | `0` | Stop after N pages (0 = no limit) |
| `-d, --max-depth` | `0` | How many links deep to follow (0 = no limit) |
| `--scope-prefix` | | Only crawl paths starting with this prefix |
| `--subdomains` | `false` | Treat subdomains of the seed host as in scope |
| `--exclude` | | Path prefixes to skip (repeatable) |
| `--scroll` | `false` | Auto-scroll each page to trigger lazy loading |
| `--workers` | `4` | Concurrent page render workers |
| `--no-robots` | `false` | Ignore `robots.txt` |
| `--workers` | `4` | How many pages to render at once |
| `--no-robots` | `false` | Ignore `robots.txt` (be nice) |
| `-f, --force` | `false` | Delete any existing mirror for the host first |
| `--chrome` | | Path to the Chrome/Chromium binary |
Run `kage clone --help` for the full set, including the render-timing,
concurrency, and asset-size controls.
`kage clone --help` has the rest, including render-timing, concurrency, and asset-size knobs.
### Serve
`kage serve` runs a local static file server over a cloned folder so links and
assets resolve the way they would on a real host:
`kage serve` runs a tiny static file server over a cloned folder so links and assets resolve the way they would on a real host:
```bash
kage serve $HOME/data/kage/example.com
kage serve $HOME/data/kage/paulgraham.com
# open http://127.0.0.1:8800
```
## Pack it into one file
A clone is a folder, which is easy to browse but awkward to move: copying
thousands of small files is slow, and a directory is less tidy to hand over than
a single file. `kage pack` collapses a mirror into one artifact.
A mirror is a folder, which is great for browsing and lousy for moving around. Copying thousands of little files is slow, and "here, have this directory" is a clumsy thing to hand someone. `kage pack` collapses the whole mirror into one artifact, and you choose the shape: an open ZIM archive, or a single executable that *is* the site.
### A single ZIM file
```bash
# An open ZIM archive, the single-file format Kiwix uses
kage pack example.com
kage open example.com.zim
# A single executable that *is* the site
kage pack example.com --format binary
./example
kage pack paulgraham.com # -> paulgraham.com.zim
kage open paulgraham.com.zim
```
The default is a [ZIM](https://wiki.openzim.org/wiki/ZIM_file_format) archive:
the whole mirror in one file, text zstd-compressed and media stored as-is, that
`kage open` or any ZIM reader can browse. `--format binary` appends that archive
to a copy of kage and produces a single executable that serves the site offline
when run, so the recipient needs nothing installed, not even kage.
ZIM is an open file format built for exactly this: a whole website (or a whole Wikipedia) squeezed into one compressed, indexed, read-only file. kage writes the entire mirror into it, text zstd-compressed and media stored as-is. It is the format behind [Kiwix](https://kiwix.org), the offline-content project people use to carry Wikipedia, Stack Overflow, and Project Gutenberg onto boats, into classrooms with no internet, and onto a phone for a long flight. Because the format is a documented standard and not a kage invention, a `paulgraham.com.zim` you make today will still open in any ZIM reader years from now.
Packing is deterministic: the same mirror produces a byte-identical file, with
the archive UUID derived from the content rather than randomised, so a pack is
safe to checksum and cache. A bare host name resolves against the default output
directory, so `kage pack example.com` works right after `kage clone example.com`.
The appended archive is platform-independent; only the base executable carries
the architecture. Point `--base` at a kage built for another OS to make a viewer
for that platform from your own machine:
So you are not locked into kage. `kage open` is the quickest way back in, but the very same file works across the wider Kiwix ecosystem:
```bash
# From macOS, build a Windows viewer
kage pack example.com --format binary --base kage-windows-amd64.exe # -> example.exe
kage open paulgraham.com.zim # read it back with kage
kiwix-serve paulgraham.com.zim # or serve it with Kiwix at http://localhost
```
## A native window, not a browser tab
You can also double-click the file in the [Kiwix desktop app](https://kiwix.org/en/applications/), or load it on [Kiwix for Android or iOS](https://kiwix.org/en/applications/) to read your mirror on your phone. One caveat: kage writes a structurally valid archive with the standard metadata, but it does not build the full-text search index that Kiwix's own packs ship with, so browsing and clicking work everywhere while in-reader search is limited.
By default a packed binary opens the system browser, which means a tab with an
address bar alongside your others. Build kage with the `webview` tag and it
opens the site in its own window instead, backed by the operating system's
WebView (WKWebView on macOS, WebView2 on Windows, WebKitGTK on Linux), so a
packed binary feels like a standalone app:
Packing is deterministic. The same mirror always produces a byte-identical file, with the archive UUID derived from the content instead of randomized, so a pack is safe to checksum and cache. A bare host name resolves against the default output directory, which is why `kage pack paulgraham.com` just works right after `kage clone paulgraham.com`.
### A self-contained binary
`--format binary` glues the archive onto a copy of kage and hands you a single executable that serves the site offline when you run it. Whoever you send it to needs nothing installed: not kage, not a ZIM reader, nothing.
```bash
kage pack paulgraham.com --format binary -o paulgraham
./paulgraham
```
The appended archive is platform-independent; only the base executable carries the architecture. By default kage appends to itself, so you get a viewer for the machine you ran it on. Point `--base` at a kage built for another OS to produce a viewer for that platform from your own machine:
```bash
# Sitting on a Mac, build a Windows viewer
kage pack paulgraham.com --format binary --base kage-windows-amd64.exe # -> paulgraham.exe
```
The trade is size. The binary carries a whole kage, so it weighs around 13 MiB plus the site no matter how small the mirror is. When you only need the content, the ZIM is far leaner.
## A real window, not a browser tab
By default a packed binary opens your system browser, which means the site shows up as yet another tab, address bar and all, next to the 47 you already have open. Build kage with the `webview` tag and it opens the site in its own window instead, backed by the operating system's WebView (WKWebView on macOS, WebView2 on Windows, WebKitGTK on Linux). Paul Graham's essays, offline, in something that looks and feels like a real app:
![paulgraham.com served offline in a native kage window](docs/static/webview.png)
```bash
make build-webview # or: CGO_ENABLED=1 go build -tags webview ./cmd/kage
kage pack example.com --format binary --base bin/kage
./example # opens a window, no browser
kage pack paulgraham.com --format binary --base bin/kage -o paulgraham
./paulgraham # opens a window, no browser in sight
```
This build needs cgo and links the platform WebView, so it stays opt-in. The
default build is pure Go (`CGO_ENABLED=0`) and the prebuilt release binaries
open the browser, which keeps the cross-compiled release pipeline simple.
`kage open` honours the same tag.
This build needs cgo and links the platform WebView, so it stays opt-in. The default build is pure Go (`CGO_ENABLED=0`) and the prebuilt release binaries open the browser, which keeps the cross-compiled release simple. `kage open` honours the same tag, so built with `-tags webview` it shows a ZIM in a native window too.
## How it works
@@ -172,25 +179,20 @@ seed URL ─▶ headless Chrome ─▶ final DOM ─▶ strip JS ─▶ localise
(render) (snapshot) (sanitize) (rewrite links)
```
Pages are rendered by a pool of Chrome tabs; assets are fetched over plain HTTP
by a separate worker pool. Every URL maps deterministically to a local path, so
links can be rewritten before the asset they point at has finished downloading.
Output layout:
A pool of Chrome tabs renders pages; a separate pool fetches assets over plain HTTP. Every URL maps deterministically to a local path, so links get rewritten before the asset they point at has even finished downloading. The output looks like this:
```
example.com/
├── index.html # the home page, scripts stripped
├── about/index.html # /about
paulgraham.com/
├── index.html # the home page, scripts stripped
├── greatwork.html # /greatwork.html, an essay
├── _kage/ # reserved: assets and crawl state
│ ├── example.com/site.css # localised stylesheet (url() rewritten)
│ ├── example.com/logo.png
│ ├── paulgraham.com/site.css # localised stylesheet (url() rewritten)
│ ├── paulgraham.com/pg.png
│ └── state.json # visited set, for resuming
└── ...
```
The same model underlies `pack`: the mirror's links are already mirror-relative
paths, and those map one-to-one onto the archive's content entries, so a click
in a served page hits the right entry with no rewriting.
`pack` rides on the same idea: the mirror's links are already mirror-relative paths, and those map one-to-one onto the archive's content entries, so a click in a served page hits the right entry with no rewriting at all.
## Building from source
@@ -203,7 +205,7 @@ make test # full suite, including the Chrome-driven end-to-end tests
make test-short # skip the tests that launch a browser
```
The repository is laid out by concern:
The repo is split by concern:
```
cmd/kage/ thin main: pins the main thread, then hands off to cli.Execute
@@ -221,18 +223,14 @@ docs/ the tago documentation site
## Releasing
Push a version tag and GitHub Actions runs GoReleaser, which builds the
archives, the `.deb`/`.rpm`/`.apk` packages, a multi-arch GHCR image with
Chromium bundled, checksums, SBOMs, and a cosign signature:
Push a version tag and GitHub Actions runs GoReleaser, which builds the archives, the `.deb`/`.rpm`/`.apk` packages, a multi-arch GHCR image with Chromium bundled, checksums, SBOMs, and a cosign signature:
```bash
git tag v0.1.0
git tag v0.1.1
git push --tags
```
The image tag carries no `v` prefix (`ghcr.io/tamnd/kage:0.1.0`). The Homebrew
and Scoop steps self-disable until their tokens exist, so the first release
works with no extra secrets.
The image tag carries no `v` prefix (`ghcr.io/tamnd/kage:0.1.1`). The Homebrew and Scoop steps self-disable until their tokens exist, so the first release works with no extra secrets.
## License
docs/content/_index.md
+17 -24
View File
@@ -7,39 +7,32 @@ heroPrimaryURL: "/getting-started/quick-start/"
heroPrimaryText: "Get started"
---
Saving a page with "Save As" gives you a copy that still phones home, still runs
analytics, and often renders blank because the markup is built by JavaScript at
runtime. kage (影, "shadow") takes the opposite approach: it drives a real
browser, captures the page the way a human would have seen it, then makes it
inert.
Saving a page with "Save As" gives you a copy that still phones home, still runs analytics, and often renders blank because the markup is built by JavaScript at runtime. kage (影, "shadow") takes the opposite approach: it drives a real browser, captures the page the way a human would have seen it, then makes it inert.
Say you want Paul Graham's essays on a laptop with no wifi. One command mirrors the site; a second serves it back offline:
```bash
kage clone example.com
kage serve $HOME/data/kage/example.com
kage clone paulgraham.com
kage serve $HOME/data/kage/paulgraham.com
```
![kage cloning a site, packing it into one file, and serving it back offline](/demo.gif)
![kage cloning paulgraham.com, packing it into one file, and serving it back offline](/demo.gif)
## What it does
- **Renders first, saves second.** Each page goes through real headless Chrome,
so a page whose content is assembled by JavaScript is captured fully, not as
an empty shell.
- **Strips every script.** Once the DOM is captured, kage removes all `<script>`
tags, every `on*` event handler, and any `javascript:` URL. The saved page
makes no network calls and runs no code.
- **Keeps the layout.** Stylesheets, images, fonts, and media are downloaded and
rewritten to relative local paths, so the offline copy looks like the original.
- **Stays browsable.** In-scope links are rewritten to point at the other saved
pages, so you can click around the mirror exactly as you would the live site.
- **Renders first, saves second.** Each page goes through real headless Chrome, so a page whose content is assembled by JavaScript is captured fully, not as an empty shell.
- **Strips every script.** Once the DOM is captured, kage removes all `<script>` tags, every `on*` event handler, and any `javascript:` URL. The saved page makes no network calls and runs no code.
- **Keeps the layout.** Stylesheets, images, fonts, and media are downloaded and rewritten to relative local paths, so the offline copy looks like the original.
- **Stays browsable.** In-scope links are rewritten to point at the other saved pages, so you can click around the mirror exactly as you would the live site.
- **Packs into one file.** Collapse a mirror into a single [ZIM archive](/guides/packing-a-mirror/), the open format Kiwix uses, or a self-contained binary that serves the site when run.
Build kage with the `webview` tag and a packed binary opens in its own window instead of a browser tab, so an offline mirror feels like a real app:
![paulgraham.com served offline in a native kage window](/webview.png)
## Where to go next
- New here? Start with the [introduction](/getting-started/introduction/), then
the [quick start](/getting-started/quick-start/).
- New here? Start with the [introduction](/getting-started/introduction/), then the [quick start](/getting-started/quick-start/).
- Want to install it? See [installation](/getting-started/installation/).
- Looking for a specific task? The [guides](/guides/) cover scoping a crawl,
serving a mirror, resuming an interrupted run, and
[packing a mirror](/guides/packing-a-mirror/) into one file or a self-contained
viewer.
- Looking for a specific task? The [guides](/guides/) cover scoping a crawl, serving a mirror, resuming an interrupted run, and [packing a mirror](/guides/packing-a-mirror/) into one file or a self-contained viewer.
- Need every flag? The [CLI reference](/reference/cli/) is the full surface.
docs/content/getting-started/introduction.md
+19 -25
View File
@@ -4,21 +4,20 @@ description: "Why kage renders before it saves, and what it means to strip the J
weight: 10
---
A normal website is not a document; it is a program. The HTML the server sends
is often a near-empty shell, and the page you actually see is assembled in your
browser by JavaScript: fetching data, building the DOM, wiring up handlers. That
is why "Save As" so often fails. You get the shell, not the page, and whatever
you do get still runs trackers and phones home when you open it.
A normal website is not a document; it is a program. The HTML the server sends is often a near-empty shell, and the page you actually see is assembled in your browser by JavaScript: fetching data, building the DOM, wiring up handlers. That is why "Save As" so often fails. You get the shell, not the page, and whatever you do get still runs trackers and phones home when you open it.
Say you want to keep Paul Graham's essays. Hand the site to "Save As" and you get a brittle copy that may still call out to scripts that no longer exist. Hand it to kage and you get the essays as they look in a browser, frozen and inert:
```bash
kage clone paulgraham.com
kage serve $HOME/data/kage/paulgraham.com
```
kage treats a clone as three steps in order.
## 1. Render
Every page is loaded in a real headless Chrome through the DevTools protocol.
kage navigates to the URL, waits for the network to go quiet, optionally scrolls
to trigger lazy-loaded images, and then serialises the **final** DOM, the markup
that exists after the page's JavaScript has finished building it. This is the
same thing you would see if you opened the page and chose "Inspect".
Every page is loaded in a real headless Chrome through the DevTools protocol. kage navigates to the URL, waits for the network to go quiet, optionally scrolls to trigger lazy-loaded images, and then serialises the **final** DOM, the markup that exists after the page's JavaScript has finished building it. This is the same thing you would see if you opened the page and chose "Inspect".
## 2. Strip
@@ -27,27 +26,22 @@ From that captured DOM, kage removes everything executable:
- every `<script>` tag, inline or external;
- every `on*` event handler attribute (`onclick`, `onload`, and the rest);
- every `javascript:` URL;
- `<meta http-equiv="refresh">` redirects and dead resource hints like
`<link rel="preload" as="script">`.
- `<meta http-equiv="refresh">` redirects and dead resource hints like `<link rel="preload" as="script">`.
What remains is inert. The saved page makes no network calls, runs no code, and
tracks nothing.
What remains is inert. The saved page makes no network calls, runs no code, and tracks nothing.
## 3. Localise
A page with no working CSS or images is not much of a clone, so kage keeps the
parts that define how it looks. It downloads every stylesheet, image, font, and
media file, rewrites the references in the HTML and inside the CSS (`url()` and
`@import`) to relative local paths, and rewrites in-scope page links to point at
the other saved pages. The mirror is fully self-contained: you can move the
folder anywhere, open it with no network, and click around.
A page with no working CSS or images is not much of a clone, so kage keeps the parts that define how it looks. It downloads every stylesheet, image, font, and media file, rewrites the references in the HTML and inside the CSS (`url()` and `@import`) to relative local paths, and rewrites in-scope page links to point at the other saved pages. The mirror is fully self-contained: you can move the folder anywhere, open it with no network, and click around.
## The shape of a clone
kage crawls breadth-first from a seed URL, staying within the seed's host (and
optionally its subdomains). It is polite by default: it honours `robots.txt` and
seeds itself from `sitemap.xml`. Output lands in `$HOME/data/kage/<host>/`, with pages
as `<path>/index.html` and assets under a reserved `_kage/` directory alongside
the crawl state that powers `--resume`.
kage crawls breadth-first from a seed URL, staying within the seed's host (and optionally its subdomains). It is polite by default: it honours `robots.txt` and seeds itself from `sitemap.xml`. Output lands in `$HOME/data/kage/paulgraham.com/`, with pages as `<path>/index.html` and assets under a reserved `_kage/` directory alongside the crawl state that powers resuming.
## Then what?
A folder is the starting point, not the end. Once you have a mirror you can [pack it](/guides/packing-a-mirror/) into a single ZIM file, the open offline-archive format Kiwix uses, so the whole site travels as one file that any ZIM reader can open. Or build kage with the `webview` tag and a packed binary opens the site in its own native window instead of a browser tab:
![paulgraham.com served offline in a native kage window](/webview.png)
Next: [install kage](/getting-started/installation/).
docs/content/guides/packing-a-mirror.md
+46 -76
View File
@@ -4,139 +4,109 @@ description: "Turn a cloned folder into one ZIM file or a self-contained offline
weight: 30
---
A clone is a folder of files, which is easy to browse but awkward to move around:
copying thousands of small files is slow, and handing someone a directory is less
tidy than handing them one file. `kage pack` collapses a mirror into a single
distributable artifact, either an open ZIM archive or a self-contained executable
that serves the site offline when run.
A clone is a folder of files, which is easy to browse but awkward to move around: copying thousands of small files is slow, and handing someone a directory is less tidy than handing them one file. `kage pack` collapses a mirror into a single distributable artifact, and you choose the shape: an open ZIM archive, or a self-contained executable that serves the site offline when run.
## Pack to a ZIM file
ZIM is the open, single-file offline-archive format Kiwix uses. `kage pack`
writes one from a cloned host directory:
The two examples below assume you have already cloned a site, for instance Paul Graham's essays:
```bash
kage pack $HOME/data/kage/example.com
kage clone paulgraham.com
```
## A single ZIM file
ZIM is the open, single-file offline-archive format Kiwix uses. `kage pack` writes one from a cloned host directory:
```bash
kage pack paulgraham.com
```
```
packed example.com.zim
packed paulgraham.com.zim
size 4.2 MiB
open kage open example.com.zim
open kage open paulgraham.com.zim
```
The whole mirror, pages and assets, lives in that one file. Text is zstd
compressed; already-compressed media (images, fonts, video) is stored as-is.
Packing the same mirror twice produces a byte-identical file, so a ZIM is safe to
checksum, diff, and cache.
The whole mirror, pages and assets, lives in that one file. Text is zstd compressed; already-compressed media (images, fonts, video) is stored as-is. Packing the same mirror twice produces a byte-identical file, so a ZIM is safe to checksum, diff, and cache.
If you cloned with the default output directory, you can pass a bare host name and
kage finds the mirror for you:
If you cloned with the default output directory, you can pass a bare host name and kage finds the mirror for you. That is why `kage pack paulgraham.com` works straight after `kage clone paulgraham.com`; pass a full path if your mirror lives somewhere else.
### What ZIM is, and using it with Kiwix
ZIM is built for exactly this job: a whole website (or a whole Wikipedia) squeezed into one compressed, indexed, read-only file. It is the format behind [Kiwix](https://kiwix.org), the offline-content project people use to carry Wikipedia, Stack Overflow, and Project Gutenberg onto boats, into classrooms with no internet, and onto a phone for a long flight. Because the format is a documented standard rather than a kage invention, a `paulgraham.com.zim` you make today still opens in any ZIM reader years from now.
So you are not locked into kage. `kage open` is the read side and the quickest way back in, but the same file works across the wider Kiwix ecosystem:
```bash
kage clone example.com
kage pack example.com
kage open paulgraham.com.zim # read it back with kage
kiwix-serve paulgraham.com.zim # or serve it with Kiwix at http://localhost
```
### Read it back
You can also double-click the file in the [Kiwix desktop app](https://kiwix.org/en/applications/), or load it on Kiwix for Android or iOS to read your mirror on your phone. One caveat: kage writes a structurally valid archive with the standard metadata, but it does not write the full-text search index that Kiwix's own packs ship with, so browsing works everywhere while in-reader search is limited.
`kage open` is the read side: it serves a ZIM over a local HTTP server and opens
your browser, the same way `kage serve` does for a folder.
## A self-contained binary
`--format binary` appends the ZIM to a copy of kage, producing one executable that *is* the site. Run it and it serves the mirror on a free local port and opens your browser; it ignores its arguments, because the binary is the site, not the kage CLI.
```bash
kage open example.com.zim
kage pack paulgraham.com --format binary -o paulgraham
```
```
kage open example.com.zim
open http://127.0.0.1:8800
press Ctrl-C to stop
```
Any other ZIM reader (Kiwix among them) can also open the file. kage writes a
structurally valid archive with the standard metadata; full-text search indexes
are not written, so browsing works everywhere but in-reader search is limited.
## Pack to a self-contained binary
`--format binary` appends the ZIM to a copy of kage, producing one executable
that *is* the site. Run it and it serves the mirror on a free local port and
opens your browser; it ignores its arguments, because the binary is the site, not
the kage CLI.
```bash
kage pack $HOME/data/kage/example.com --format binary
```
```
packed example
packed paulgraham
size 21.9 MiB
run ./example to view the site offline
run ./paulgraham to view the site offline
```
```bash
./example
./paulgraham
```
```
serving offline site at http://127.0.0.1:52431 (Ctrl-C to stop)
```
The binary carries a full kage, so it is tens of megabytes regardless of site
size; the trade is that the recipient needs nothing installed, not even kage.
The binary carries a full kage, so it is tens of megabytes regardless of site size; the trade is that the recipient needs nothing installed, not even kage, not even a ZIM reader.
### A native window instead of a browser
By default the viewer opens the system browser, which means a tab with an address
bar and your other tabs alongside. Build kage with the `webview` tag and it opens
the site in its own native window instead, backed by the operating system's
WebView (WKWebView on macOS, WebView2 on Windows, WebKitGTK on Linux), so a
packed binary feels like a standalone app:
By default the viewer opens the system browser, which means a tab with an address bar and your other tabs alongside. Build kage with the `webview` tag and it opens the site in its own native window instead, backed by the operating system's WebView (WKWebView on macOS, WebView2 on Windows, WebKitGTK on Linux), so a packed binary feels like a standalone app:
```bash
CGO_ENABLED=1 go build -tags webview -o kage ./cmd/kage
kage pack $HOME/data/kage/example.com --format binary --base kage
./example # opens a window, no browser
kage pack paulgraham.com --format binary --base kage -o paulgraham
./paulgraham # opens a window, no browser
```
The window title comes from the archive's title. This build needs cgo and links
the platform WebView, so it is opt-in and kept out of the default
`CGO_ENABLED=0` release; the prebuilt binaries open the browser. `kage open` honours
the same tag: built with `-tags webview` it shows the ZIM in a native window too.
![paulgraham.com served offline in a native kage window](/webview.png)
The window title comes from the archive's title. This build needs cgo and links the platform WebView, so it is opt-in and kept out of the default `CGO_ENABLED=0` release; the prebuilt binaries open the browser. `kage open` honours the same tag: built with `-tags webview` it shows the ZIM in a native window too.
### Build a viewer for another platform
The appended archive is platform-independent; only the base executable carries
the architecture. Point `--base` at a kage binary built for another OS (download
one from a kage release) to produce a viewer for that platform from your own
machine:
The appended archive is platform-independent; only the base executable carries the architecture. Point `--base` at a kage binary built for another OS (download one from a kage release) to produce a viewer for that platform from your own machine:
```bash
# From macOS, build a Windows viewer
kage pack $HOME/data/kage/example.com --format binary --base kage-windows-amd64.exe
# -> example.exe
kage pack paulgraham.com --format binary --base kage-windows-amd64.exe
# -> paulgraham.exe
```
### macOS note
A binary you built or downloaded may be quarantined by Gatekeeper on first run.
kage prints the exact command to clear it:
A binary you built or downloaded may be quarantined by Gatekeeper on first run. kage prints the exact command to clear it:
```bash
xattr -d com.apple.quarantine ./example
xattr -d com.apple.quarantine ./paulgraham
```
## Metadata and options
```bash
kage pack $HOME/data/kage/example.com \
--title "Example, offline" \
kage pack paulgraham.com \
--title "Paul Graham, offline" \
--description "A snapshot taken for archival" \
--language eng \
--date 2026-06-14
```
`--title` defaults to the main page's `<title>`, then the host name. `--date`
defaults to today; pass a fixed value for a fully reproducible file. `--no-compress`
stores every cluster raw, which packs fastest and lets a reader without zstd open
the result. `-o/--out` overrides the output path for either format.
`--title` defaults to the main page's `<title>`, then the host name. `--date` defaults to today; pass a fixed value for a fully reproducible file. `--no-compress` stores every cluster raw, which packs fastest and lets a reader without zstd open the result. `-o/--out` overrides the output path for either format.
docs/content/reference/release-notes.md
+11 -31
View File
@@ -4,43 +4,23 @@ description: "What changed in each kage release."
weight: 40
---
The authoritative, commit-level history lives in
[`CHANGELOG.md`](https://github.com/tamnd/kage/blob/main/CHANGELOG.md) and on the
[releases page](https://github.com/tamnd/kage/releases). This page summarises
each version.
The authoritative, commit-level history lives in [`CHANGELOG.md`](https://github.com/tamnd/kage/blob/main/CHANGELOG.md) and on the [releases page](https://github.com/tamnd/kage/releases). This page summarises each version.
## Unreleased
## v0.1.1
Packing, so a clone can travel as one file instead of a folder.
- **`kage pack <mirror-dir>`** collapses a mirror into a single distributable
file. `--format zim` (the default) writes an open ZIM archive, the same format
Kiwix uses; `--format binary` appends that archive to a copy of kage to make a
self-contained executable that serves the site offline when run. Packing is
deterministic, so the same mirror produces a byte-identical file.
- **`kage open <file.zim>`** serves a packed ZIM back over a local HTTP server,
the read side of `kage pack --format zim`.
- **An optional native-window viewer.** Built with `-tags webview`, `kage open`
and a packed binary show the site in a real window backed by the operating
system's WebView instead of a browser tab. The default build stays pure Go and
opens the browser, so the release pipeline is unchanged.
- **A pure-Go `zim` package** that reads and writes the ZIM format: a fixed
header, MIME and pointer lists, zstd or stored clusters, redirects, and a
trailing MD5.
- **`kage pack <mirror-dir>`** collapses a mirror into a single distributable file. `--format zim` (the default) writes an open ZIM archive, the same format [Kiwix](https://kiwix.org) uses, so the file opens in any ZIM reader and not just kage. `--format binary` appends that archive to a copy of kage to make a self-contained executable that serves the site offline when run. Packing is deterministic, so the same mirror produces a byte-identical file.
- **`kage open <file.zim>`** serves a packed ZIM back over a local HTTP server, the read side of `kage pack --format zim`.
- **An optional native-window viewer.** Built with `-tags webview`, `kage open` and a packed binary show the site in a real window backed by the operating system's WebView instead of a browser tab. The default build stays pure Go and opens the browser, so the release pipeline is unchanged.
- **A pure-Go `zim` package** that reads and writes the ZIM format: a fixed header, MIME and pointer lists, zstd or stored clusters, redirects, and a trailing MD5.
## v0.1.0
The first release. kage clones a live website into a self-contained folder you
can browse offline, with every script stripped out.
The first release. kage clones a live website into a self-contained folder you can browse offline, with every script stripped out.
- **`kage clone <url>`** renders each page in headless Chrome, strips all
JavaScript, and localises CSS, images, and fonts to relative paths.
- **`kage clone <url>`** renders each page in headless Chrome, strips all JavaScript, and localises CSS, images, and fonts to relative paths.
- **`kage serve [dir]`** previews a cloned folder over a local file server.
- **Idempotent and resumable.** Each page is keyed by the file it writes, so a
page reached over http and https, or as `/index.html` versus `/`, is fetched
once. Re-running resumes; `--refresh` re-renders in place; `--force` starts
clean.
- **Polite by default.** Honours `robots.txt`, seeds from `sitemap.xml`, scopes
to the seed host, and runs three parallel worker tiers.
- **Packaged everywhere.** Archives, `.deb`/`.rpm`/`.apk`, a multi-arch GHCR
image with Chromium bundled, checksums, SBOMs, and a cosign signature.
- **Idempotent and resumable.** Each page is keyed by the file it writes, so a page reached over http and https, or as `/index.html` versus `/`, is fetched once. Re-running resumes; `--refresh` re-renders in place; `--force` starts clean.
- **Polite by default.** Honours `robots.txt`, seeds from `sitemap.xml`, scopes to the seed host, and runs three parallel worker tiers.
- **Packaged everywhere.** Archives, `.deb`/`.rpm`/`.apk`, a multi-arch GHCR image with Chromium bundled, checksums, SBOMs, and a cosign signature.
docs/demo/kage.tape
+6 -6
View File
@@ -6,27 +6,27 @@
# kage must be on PATH inside the recording shell, and Chrome available.
Hide
Type "export PS1='$ ' && cd $(mktemp -d) && clear"
Type "export PS1='$ ' PATH=/tmp/kagebin:$PATH && cd $(mktemp -d) && clear"
Enter
Show
Sleep 600ms
Type "kage clone example.com --out ."
Type "kage clone paulgraham.com --max-pages 3 --out ."
Sleep 700ms
Enter
Sleep 4.5s
Sleep 9s
Type "kage pack example.com"
Type "kage pack ./paulgraham.com"
Sleep 700ms
Enter
Sleep 2.5s
Type "kage pack example.com --format binary -o example"
Type "kage pack ./paulgraham.com --format binary -o paulgraham"
Sleep 700ms
Enter
Sleep 3s
Type "kage open example.com.zim --open=false"
Type "kage open paulgraham.com.zim --open=false"
Sleep 700ms
Enter
Sleep 3s
docs/static/demo.gif
Vendored
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 169 KiB

After

Width:  |  Height:  |  Size: 345 KiB

docs/static/webview.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 630 KiB