A developer.apple.com crawl came back at 19 GB, 18 of it assets, most of
that the site's own videos, .dmg/.pkg installers, and PDF manuals pulled
from a couple dozen hosts including unrelated third parties. None of it
helps read the docs offline. This changes what kage localizes by default.
Bulk media, installers, archives, and PDFs are left pointing at their
live URL instead of downloaded. The decision is made from the URL alone,
so the rewritten HTML simply keeps the remote link. --keep-media restores
the old behavior and --skip-ext adds more extensions to leave remote.
Assets are localized only from the seed's registrable domain by default,
so www.apple.com and images.apple.com still come along but a separate
brand domain or an off-topic third party does not. --all-asset-hosts goes
back to downloading from any host.
The size cap was also truncating instead of skipping: it wrapped the body
in a LimitReader, so an over-cap file was saved as exactly the first N MB
of itself, a corrupt fragment that would never play or run. On the apple
crawl that was around a gigabyte of half-downloaded WWDC videos. kage now
checks the response size and leaves an over-cap asset out of the mirror.
kage parquet export turns a packed archive into a columnar Parquet table,
one row per entry, and kage parquet import rebuilds the archive from it. The
table follows the open-index/open-markdown field names (doc_id, url, host,
crawl_date, content_length, text_length, text) so a kage export sits next to
other web-crawl datasets on Hugging Face and reads straight into DuckDB or
pandas. Alongside those columns it keeps the raw content bytes and the ZIM
structure (namespace, redirect target), so the round trip is lossless: a ZIM
exported and reimported is byte-identical.
doc_id is a deterministic UUID v5 of the page URL. HTML pages also get a
derived plain-text column for full-text search and training use; it plays no
part in the round trip, which rebuilds pages from the stored content.
Compressing clusters with zstd is the slow part of packing a large
mirror. When a mirror is re-packed after a small change, most clusters
are byte-identical and there is no reason to compress them again.
pack --incremental keeps a content-addressed cache of compressed
clusters in a sidecar next to the output. On the next pack, a cluster
whose uncompressed bytes match the cache is served from it instead of
being recompressed; only new or changed clusters go through zstd. A
cache hit returns exactly what a fresh compression would, so the archive
stays deterministic and byte-identical to a cold pack.
The zim writer gains a settable cluster compressor so the cache can hook
in without changing the format. The cache only writes back the clusters
it touched this run, so clusters that left the mirror drop out and it
cannot grow without bound.
Two related changes for crawling faceted sites, where one path spawns
thousands of ?q=.../?page=... URLs that all render the same page.
The progress line was counting every written file, so the page number ran
far ahead of the site's real size. It now shows distinct URL paths as
"pages" and folds the query-string permutations into a separate "variants"
count, so the live counter tracks real pages and is easy to read.
Pages with identical bytes are now stored once. The first page with a given
content is written normally; a later page with the same bytes becomes a hard
link to it, so duplicate content never takes disk twice. Links still resolve
because each variant keeps its own path. When hard links are unsupported the
bytes are written, so correctness never depends on the link. The summary
reports how many pages were deduped.
Wrapping a packed viewer in a .app or .AppImage was its own --format app
value, parallel to zim and binary. But an app is really just the binary
format with a bundle around it, so a separate format meant duplicating the
base/icon handling and made the three formats feel like an awkward choice.
Turn it into a --app flag that builds on the binary format. It composes
with --base (including a webview base) and --icon, while --format stays
zim or binary. The bundle builders are unchanged; only the CLI surface
moves.
The new format dispatches on the base's target OS: a .app on macOS, an .AppDir
(plus an .AppImage when appimagetool is present) on Linux, and a friendly redirect
to --format binary on Windows, where the .exe is already the app. The icon comes
from the mirror's favicon by default and can be overridden with --icon.
kage launched Chrome with --no-sandbox unconditionally, which turns off the
browser's main security boundary for every run, including ordinary desktop
use where the sandbox works fine. Since kage renders pages from the open web,
a renderer exploit could then reach the host. Reported in #10.
Keep the sandbox on by default and drop it only where it genuinely cannot
initialize: inside a container, or when running as root (Chrome refuses to
start a sandbox as root). Containers are detected from IN_DOCKER or the
/.dockerenv marker, and there kage also sets --disable-dev-shm-usage because
the default 64 MB /dev/shm is too small for the renderer on large pages.
Whenever the sandbox is dropped kage says so on stderr, so it is never silent.
Thanks to Dimitrios Prasakis for the report and to the commenter on Hacker
News who suggested the IN_DOCKER opt-in.
Packing with --base pointed at a kage built for another OS used to guess the
target from the file name: a base ending in .exe meant Windows, anything else
meant the host. That misfired in both directions. A Windows base without the
.exe suffix produced a viewer with no extension that will not run on Windows,
and an --out name that dropped .exe made the run hint print a macOS quarantine
note for a Windows file.
Sniff the base's executable header (ELF, PE, Mach-O) instead, so the target OS
comes from the bytes rather than the name. A Windows viewer now always gets a
.exe suffix, and the run hint names the real target and only mentions Gatekeeper
for actual macOS viewers.
errcheck flagged six naked defer Close() calls in the pack code and its
tests. Wrap them in the same defer func() { _ = x.Close() }() form the rest
of the tree already uses.
A packed binary opened the system browser, so it felt like a tab, not
an app. Build with -tags webview (cgo) and the viewer instead opens the
site in its own window backed by the OS WebView: WKWebView on macOS,
WebView2 on Windows, WebKitGTK on Linux.
The viewer package picks an implementation at build time. The default
file opens the browser and keeps the build pure Go, so CGO_ENABLED=0 and
the release pipeline are untouched. The webview file links the platform
WebView and runs its event loop on the main goroutine, which main now
pins with LockOSThread before anything else, since macOS requires UI on
the initial thread. Both kage open and the embedded viewer serve over
HTTP in a goroutine and hand the URL to the viewer, then tear the server
down when the window closes or Ctrl-C cancels.
The window title comes from the archive's M/Title. OpenInBrowser moves
out of pack into the viewer package, its only caller.
pack packs a mirror to a zim file or a runnable viewer, accepting a bare
host that it resolves against the default output directory. open serves a
zim over http like serve does for a folder. Execute checks for an appended
archive first, so a packed kage runs as an offline viewer on an ephemeral
port and ignores its arguments.
Crawling keyed off the raw URL, so the same page reached over http and
https, or as /index.html versus /, was a different frontier entry that
nonetheless wrote to the same file. A clone of paulgraham.com did 948
render passes for 474 files. Key pages and assets by the local path they
write instead, and collapse a directory-index document to its directory,
so each page is fetched exactly once.
Add --refresh to re-render a mirror in place (re-fetch every page, keep
the directory, overwrite) and make --no-resume truly stateless by not
persisting state.json. The default remains a resumable, idempotent crawl
that skips work already on disk.
kage renders every page in headless Chrome, snapshots the final
DOM, strips all JavaScript, and localises CSS, images, and fonts
so a site can be browsed offline as a plain folder of files.
The engine is split into small packages:
urlx deterministic URL to local-path mapping and scope rules
sanitize remove scripts, on* handlers, and javascript: URLs
asset rewrite HTML and CSS references, download assets
browser headless Chrome pool over the DevTools protocol
robots robots.txt matcher
clone the orchestrator: a polite resumable breadth-first crawl
The cli package wires a cobra and fang command surface with two
commands, clone and serve. Every pure package has table tests; the
browser and clone packages add Chrome-driven end-to-end tests that
skip when no browser is present or under -short.
CI runs gofmt, vet, build, race tests, golangci-lint, govulncheck,
and a tidy check on Linux and macOS. A goreleaser config fans one
tag out to archives, deb/rpm/apk, a Chromium-bundled GHCR image,
and the package managers. A tago docs site builds to Pages and
Cloudflare.
Xirui