kage parquet export turns a packed archive into a columnar Parquet table,
one row per entry, and kage parquet import rebuilds the archive from it. The
table follows the open-index/open-markdown field names (doc_id, url, host,
crawl_date, content_length, text_length, text) so a kage export sits next to
other web-crawl datasets on Hugging Face and reads straight into DuckDB or
pandas. Alongside those columns it keeps the raw content bytes and the ZIM
structure (namespace, redirect target), so the round trip is lossless: a ZIM
exported and reimported is byte-identical.
doc_id is a deterministic UUID v5 of the page URL. HTML pages also get a
derived plain-text column for full-text search and training use; it plays no
part in the round trip, which rebuilds pages from the stored content.
A packed binary opened the system browser, so it felt like a tab, not
an app. Build with -tags webview (cgo) and the viewer instead opens the
site in its own window backed by the OS WebView: WKWebView on macOS,
WebView2 on Windows, WebKitGTK on Linux.
The viewer package picks an implementation at build time. The default
file opens the browser and keeps the build pure Go, so CGO_ENABLED=0 and
the release pipeline are untouched. The webview file links the platform
WebView and runs its event loop on the main goroutine, which main now
pins with LockOSThread before anything else, since macOS requires UI on
the initial thread. Both kage open and the embedded viewer serve over
HTTP in a goroutine and hand the URL to the viewer, then tear the server
down when the window closes or Ctrl-C cancels.
The window title comes from the archive's M/Title. OpenInBrowser moves
out of pack into the viewer package, its only caller.
pack packs a mirror to a zim file or a runnable viewer, accepting a bare
host that it resolves against the default output directory. open serves a
zim over http like serve does for a folder. Execute checks for an appended
archive first, so a packed kage runs as an offline viewer on an ephemeral
port and ignores its arguments.
kage renders every page in headless Chrome, snapshots the final
DOM, strips all JavaScript, and localises CSS, images, and fonts
so a site can be browsed offline as a plain folder of files.
The engine is split into small packages:
urlx deterministic URL to local-path mapping and scope rules
sanitize remove scripts, on* handlers, and javascript: URLs
asset rewrite HTML and CSS references, download assets
browser headless Chrome pool over the DevTools protocol
robots robots.txt matcher
clone the orchestrator: a polite resumable breadth-first crawl
The cli package wires a cobra and fang command surface with two
commands, clone and serve. Every pure package has table tests; the
browser and clone packages add Chrome-driven end-to-end tests that
skip when no browser is present or under -short.
CI runs gofmt, vet, build, race tests, golangci-lint, govulncheck,
and a tidy check on Linux and macOS. A goreleaser config fans one
tag out to archives, deb/rpm/apk, a Chromium-bundled GHCR image,
and the package managers. A tago docs site builds to Pages and
Cloudflare.
Tam Nguyen Duc