Wrapping a packed viewer in a .app or .AppImage was its own --format app
value, parallel to zim and binary. But an app is really just the binary
format with a bundle around it, so a separate format meant duplicating the
base/icon handling and made the three formats feel like an awkward choice.
Turn it into a --app flag that builds on the binary format. It composes
with --base (including a webview base) and --icon, while --format stays
zim or binary. The bundle builders are unchanged; only the CLI surface
moves.
README and the packing guide gain a double-click app section covering the macOS
.app, the Linux .AppDir/.AppImage, favicon icons, and the windows-gui base. The
changelog and docs release notes record the new format under Unreleased.
A second goreleaser build links a Windows binary for the GUI subsystem
(-H windowsgui) and ships it as kage_<version>_windows-gui_<arch>.zip, scoped so
the package managers still install the console build. Packing a viewer onto this
base gives a double-click .exe with no console behind it. A CI job cross-compiles
the windowsgui link so it cannot rot.
The new format dispatches on the base's target OS: a .app on macOS, an .AppDir
(plus an .AppImage when appimagetool is present) on Linux, and a friendly redirect
to --format binary on Windows, where the .exe is already the app. The icon comes
from the mirror's favicon by default and can be overridden with --icon.
Share the base++zim++trailer assembly behind a small helper, then add two bundle
builders on top: BuildApp writes a macOS .app (Info.plist, the viewer under
Contents/MacOS, an .icns in Resources), and BuildAppDir writes an AppImage-style
.AppDir (AppRun, a Terminal=false .desktop launcher, a PNG icon). Both keep the
trailer at the tail so Embedded still finds the archive at runtime.
A pure-Go .icns writer (PNG-embedded entries from 16 to 1024) and a finder that
digs the site's favicon out of a cloned mirror, preferring a large apple-touch
icon and unwrapping a PNG-based .ico. These feed the bundle icon for the upcoming
app formats. Adds golang.org/x/image for high-quality resampling.
Move the Unreleased entries into a 0.1.2 section in the changelog and
summarise the release on the docs release-notes page: the Chrome sandbox
now stays on by default, asset downloads retry on a transient failure,
crawl errors report a clear reason and provenance, and the container
image runs again.
Two failures stopped a docker run from producing anything. Chrome
aborted on launch with 'chrome_crashpad_handler: --database is
required', because its crash reporter cannot start in a minimal
container, so disable the crash reporter on the container launch path.
kage never uploads Chrome crash dumps, so nothing is lost.
The image also created the kage user without a home directory, so HOME
was an unwritable /home/kage. kage writes its default output and resume
state under $HOME/data/kage and Chrome puts its profile and crash
database under HOME too, so both failed with a permission error and the
mounted /out volume captured nothing. Point HOME at the /out volume so
all of it lands somewhere writable that the mount picks up.
The GitHub Ubuntu runner disables unprivileged user namespaces with
AppArmor, so Chrome's sandbox cannot initialize and the new secure
default (sandbox on) makes Chrome refuse to start. IN_DOCKER is the
documented escape hatch for that case, so set it on the Ubuntu leg of
the test job. It also exercises the container code path. macOS does not
need the flag, so it stays empty there.
The crawl printed asset failures as "asset error <url>: status 403 for
<url>", repeating the URL and saying nothing about which page wanted the
file or whether the failure was worth worrying about. The final summary then
collapsed everything into a single error count.
Give failures a classified reason (HTTP 403 Forbidden, timed out, ...), name
the page that referenced the asset, and list what went wrong in the summary
instead of only counting it. Failures are collected during the run and capped
so a broken site cannot grow the list without bound.
Retry transient failures (403/429, 5xx, network blips) with a short backoff.
Bot-protection in front of a site often rejects the first request of a burst
but serves a retry fine, which is exactly what cost us stylesheets on a busy
crawl. Permanent failures (404, 401, ...) are not retried.
kage launched Chrome with --no-sandbox unconditionally, which turns off the
browser's main security boundary for every run, including ordinary desktop
use where the sandbox works fine. Since kage renders pages from the open web,
a renderer exploit could then reach the host. Reported in #10.
Keep the sandbox on by default and drop it only where it genuinely cannot
initialize: inside a container, or when running as root (Chrome refuses to
start a sandbox as root). Containers are detected from IN_DOCKER or the
/.dockerenv marker, and there kage also sets --disable-dev-shm-usage because
the default 64 MB /dev/shm is too small for the renderer on large pages.
Whenever the sandbox is dropped kage says so on stderr, so it is never silent.
Thanks to Dimitrios Prasakis for the report and to the commenter on Hacker
News who suggested the IN_DOCKER opt-in.
The gofmt gate listed packages by hand and missed pack, zim, and viewer, so a
formatting slip in the newer code would sail through. Check the whole module
instead. Add a macOS job that compiles the -tags webview viewer, the cgo path
the pure-Go CI never builds; the viewer code is identical across platforms, so
one compile guards it. Also note the new base-OS detection in the docs.
Packing with --base pointed at a kage built for another OS used to guess the
target from the file name: a base ending in .exe meant Windows, anything else
meant the host. That misfired in both directions. A Windows base without the
.exe suffix produced a viewer with no extension that will not run on Windows,
and an --out name that dropped .exe made the run hint print a macOS quarantine
note for a Windows file.
Sniff the base's executable header (ELF, PE, Mach-O) instead, so the target OS
comes from the bytes rather than the name. A Windows viewer now always gets a
.exe suffix, and the run hint names the real target and only mentions Gatekeeper
for actual macOS viewers.
errcheck flagged six naked defer Close() calls in the pack code and its
tests. Wrap them in the same defer func() { _ = x.Close() }() form the rest
of the tree already uses.
Rewrite the README around a real example, mirroring paulgraham.com for
offline reading, and split packing into two clean sections: a single ZIM
file (with what ZIM is and how to read it back through Kiwix) and a
self-contained binary. Re-record the demo gif against paulgraham.com and
add a screenshot of the native window serving the essays offline. Carry the
same framing into the docs intro pages and the packing guide, and cut the
v0.1.1 release notes.
Rework the README into the house style: badges, a one-line pitch, an
anchor nav, a commands table, and dedicated sections for clone, pack, and
the native viewer. Every flag and default is checked against the current
binary so the docs match what kage actually does.
Add a demo recorded with ascii-gif. The tape clones example.com, packs it
to a ZIM and to a self-contained binary, and serves it back offline, so
the whole loop reads in one frame. It sits at the top of the README and on
the docs home.
While reviewing the docs, fix the output path everywhere: the default is
$HOME/data/kage, not the kage-out the pages claimed, including a few
fabricated 'done kage-out/...' lines. Document pack, open, and the native
viewer in the release notes.
A packed binary opened the system browser, so it felt like a tab, not
an app. Build with -tags webview (cgo) and the viewer instead opens the
site in its own window backed by the OS WebView: WKWebView on macOS,
WebView2 on Windows, WebKitGTK on Linux.
The viewer package picks an implementation at build time. The default
file opens the browser and keeps the build pure Go, so CGO_ENABLED=0 and
the release pipeline are untouched. The webview file links the platform
WebView and runs its event loop on the main goroutine, which main now
pins with LockOSThread before anything else, since macOS requires UI on
the initial thread. Both kage open and the embedded viewer serve over
HTTP in a goroutine and hand the URL to the viewer, then tear the server
down when the window closes or Ctrl-C cancels.
The window title comes from the archive's M/Title. OpenInBrowser moves
out of pack into the viewer package, its only caller.
pack packs a mirror to a zim file or a runnable viewer, accepting a bare
host that it resolves against the default output directory. open serves a
zim over http like serve does for a folder. Execute checks for an appended
archive first, so a packed kage runs as an offline viewer on an ephemeral
port and ignores its arguments.
BuildZIM walks a cloned host directory, turns every file into a content
entry with a MIME inferred from its extension, picks a main page, and adds
the standard metadata plus a mainPage redirect. state.json is skipped.
BuildBinary appends the archive to a copy of kage behind a KAGEPCK1 trailer,
and Embedded detects that trailer at startup so the binary serves itself.
Handler maps / to the main page and /path to a content entry, the same
handler the embedded viewer uses.
ZIM is the open single-file archive format Kiwix uses for offline content:
a fixed header, a MIME list, URL/title/cluster pointer lists, directory
entries, zstd-compressed or stored clusters, and a trailing MD5. The writer
lays out a mirror in two passes (assign positions, then emit bytes) and
derives the UUID from the content so packing is deterministic. The reader
random-accesses entries by namespace and url, follows redirects, and reads
xz clusters too so archives from other tooling open.
Cross-checked against an independent reader (gozim): header, MIME list,
namespaces, urls, dirents, and a non-last cluster's blob all read back
byte-for-byte.
Crawling keyed off the raw URL, so the same page reached over http and
https, or as /index.html versus /, was a different frontier entry that
nonetheless wrote to the same file. A clone of paulgraham.com did 948
render passes for 474 files. Key pages and assets by the local path they
write instead, and collapse a directory-index document to its directory,
so each page is fetched exactly once.
Add --refresh to re-render a mirror in place (re-fetch every page, keep
the directory, overwrite) and make --no-resume truly stateless by not
persisting state.json. The default remains a resumable, idempotent crawl
that skips work already on disk.
kage renders every page in headless Chrome, snapshots the final
DOM, strips all JavaScript, and localises CSS, images, and fonts
so a site can be browsed offline as a plain folder of files.
The engine is split into small packages:
urlx deterministic URL to local-path mapping and scope rules
sanitize remove scripts, on* handlers, and javascript: URLs
asset rewrite HTML and CSS references, download assets
browser headless Chrome pool over the DevTools protocol
robots robots.txt matcher
clone the orchestrator: a polite resumable breadth-first crawl
The cli package wires a cobra and fang command surface with two
commands, clone and serve. Every pure package has table tests; the
browser and clone packages add Chrome-driven end-to-end tests that
skip when no browser is present or under -short.
CI runs gofmt, vet, build, race tests, golangci-lint, govulncheck,
and a tidy check on Linux and macOS. A goreleaser config fans one
tag out to archives, deb/rpm/apk, a Chromium-bundled GHCR image,
and the package managers. A tago docs site builds to Pages and
Cloudflare.
Duc-Tam Nguyen