When a page's JavaScript builds deeply nested object graphs, Chrome's
DevTools Protocol returns error -32000 'Object reference chain is too
long' during WaitLoad. The page has still loaded its HTML — the error is
about Chrome's internal object tracking, not the document itself.
This change detects this specific error and proceeds with rendering
instead of failing the entire page, so sites with complex JS still get
cloned successfully (issue #36).
Signed-off-by: Gautam Kumar <gautamkumarofficial@users.noreply.github.com>
An extensionless link is queued as a page, so the page worker navigated to
it in headless Chrome. When such a link served a binary, a zip or a CSV,
Chrome saved the file to the user's Downloads folder, a surprise side effect
of a clone (issue #32).
Deny Chrome-initiated downloads browser-wide, since kage fetches every asset
through its own downloader and never needs the browser to write a file. Then
watch the main document's response, and when it is not HTML, return a typed
ErrNotHTML so the page worker reroutes the URL to the asset downloader, where
the existing size and media policy decides whether to localise it or leave it
on the live web.
Verified against the two URLs from the issue, a zip and a CSV: both land
under the mirror's reserved tree and nothing is written to Downloads.
A page that declared its charset only in the HTTP Content-Type header,
with no meta charset in the markup, lost that signal once kage saved it
as a file. A reader serving the bytes without a charset then fell back to
its locale encoding and garbled every curly quote, dash, and nbsp.
kage writes UTF-8, so it now inserts a meta charset utf-8 at the top of
head when the page does not already declare one. Verified on the blog
post from the report: the saved HTML now carries the meta and renders the
quotes correctly.
Move the Unreleased fix into a dated 0.3.1 section, update the compare
links, and add a v0.3.1 summary to the docs release-notes page. This
patch redirects / to the main page so a mirror whose entry point is a
nested page keeps its CSS and images when served.
The served main page broke its own CSS and images when the entry point
was a nested page. kage saves each page's asset links as mirror-relative
paths (../_kage/...) computed for that page's own location, but the
handler served the main page's bytes directly at /, so the browser
resolved those relative URLs against / and 404ed every one of them. A
developer.apple.com/documentation mirror landed at / with no styles.
Redirect / to the main page's canonical content path, the way the
archive's W/mainPage redirect already does, so the browser navigates to
the page's real URL and resolves its relative assets correctly. Kiwix
was unaffected because it follows that redirect itself.
Move the Unreleased entries into a dated 0.3.0 section, update the
compare links, and add a v0.3.0 summary to the docs release-notes page.
This release leaves bulk and off-domain assets remote, skips over-cap
downloads instead of truncating them, adds parquet export and import,
incremental packing, and identical-page dedup.
A developer.apple.com crawl came back at 19 GB, 18 of it assets, most of
that the site's own videos, .dmg/.pkg installers, and PDF manuals pulled
from a couple dozen hosts including unrelated third parties. None of it
helps read the docs offline. This changes what kage localizes by default.
Bulk media, installers, archives, and PDFs are left pointing at their
live URL instead of downloaded. The decision is made from the URL alone,
so the rewritten HTML simply keeps the remote link. --keep-media restores
the old behavior and --skip-ext adds more extensions to leave remote.
Assets are localized only from the seed's registrable domain by default,
so www.apple.com and images.apple.com still come along but a separate
brand domain or an off-topic third party does not. --all-asset-hosts goes
back to downloading from any host.
The size cap was also truncating instead of skipping: it wrapped the body
in a LimitReader, so an over-cap file was saved as exactly the first N MB
of itself, a corrupt fragment that would never play or run. On the apple
crawl that was around a gigabyte of half-downloaded WWDC videos. kage now
checks the response size and leaves an over-cap asset out of the mirror.
A site busts a stylesheet's cache with a date string, so a page links to
styles/main.css?Thursday, 26-Feb-2026 16:26:41 UTC. A browser encodes the
spaces before requesting, but kage parsed the href and passed RawQuery through
verbatim, so the request line carried literal spaces and the server answered
400 Bad Request. On a developer.apple.com crawl this was the bulk of the
download errors.
Re-encode the query on the canonical URL: percent-encode any byte a query may
not carry (space, control, non-ASCII) while leaving valid sub-delims and
existing %XX escapes alone. Doing it in canonical fixes the fetch and keeps the
on-disk key in step with the request.
kage parquet export turns a packed archive into a columnar Parquet table,
one row per entry, and kage parquet import rebuilds the archive from it. The
table follows the open-index/open-markdown field names (doc_id, url, host,
crawl_date, content_length, text_length, text) so a kage export sits next to
other web-crawl datasets on Hugging Face and reads straight into DuckDB or
pandas. Alongside those columns it keeps the raw content bytes and the ZIM
structure (namespace, redirect target), so the round trip is lossless: a ZIM
exported and reimported is byte-identical.
doc_id is a deterministic UUID v5 of the page URL. HTML pages also get a
derived plain-text column for full-text search and training use; it plays no
part in the round trip, which rebuilds pages from the stored content.
Compressing clusters with zstd is the slow part of packing a large
mirror. When a mirror is re-packed after a small change, most clusters
are byte-identical and there is no reason to compress them again.
pack --incremental keeps a content-addressed cache of compressed
clusters in a sidecar next to the output. On the next pack, a cluster
whose uncompressed bytes match the cache is served from it instead of
being recompressed; only new or changed clusters go through zstd. A
cache hit returns exactly what a fresh compression would, so the archive
stays deterministic and byte-identical to a cold pack.
The zim writer gains a settable cluster compressor so the cache can hook
in without changing the format. The cache only writes back the clusters
it touched this run, so clusters that left the mirror drop out and it
cannot grow without bound.
Two related changes for crawling faceted sites, where one path spawns
thousands of ?q=.../?page=... URLs that all render the same page.
The progress line was counting every written file, so the page number ran
far ahead of the site's real size. It now shows distinct URL paths as
"pages" and folds the query-string permutations into a separate "variants"
count, so the live counter tracks real pages and is easy to read.
Pages with identical bytes are now stored once. The first page with a given
content is written normally; a later page with the same bytes becomes a hard
link to it, so duplicate content never takes disk twice. Links still resolve
because each variant keeps its own path. When hard links are unsupported the
bytes are written, so correctness never depends on the link. The summary
reports how many pages were deduped.
The sanitizer walked only element nodes, so a script hidden in a
downlevel IE conditional comment slipped through. golang.org/x/net/html
parses <!--[if lt IE 9]><script src="..."></script><![endif]--> as a
single comment node whose data holds the raw markup, so the element walk
never sees the <script> and it rendered straight back out, a live-CDN
script reference left sitting in a page kage promises is inert.
Plenty of older docs sites (clojure.org, cordova.apache.org, and the
async library docs among them) still ship html5shiv, respond.js, or
placeholders.js this way.
Drop conditional comments in the walk. The downlevel-hidden form is one
comment and goes whole; the downlevel-revealed form keeps its content,
which lives in sibling nodes, and loses only the two markers.
A <link rel> attribute is a space-separated token list, but kage matched
the whole string against its set of downloadable rels. So a tag like
<link rel="preload stylesheet" href="/assets/style.css"> matched neither
"preload" nor "stylesheet" and its stylesheet was left on the live web,
absolute and undownloaded.
VitePress ships exactly this form, and it is the only stylesheet on the
page, so a cloned Vue docs site (and any other VitePress site) rendered
completely unstyled offline.
Tokenize the rel and match if any single token is a known asset rel, the
way the HTML spec reads it. This also subsumes the old combined
"shortcut icon" entry, since the "icon" token alone now matches.
Apple's developer.apple.com ships a classic BMP/DIB favicon.ico rather
than a PNG one, so the icon discovery that feeds the ZIM book icon found
nothing and the archive shipped without an Illustrator_48x48@1 image.
Decode the .ico container directly: read the directory, pick the largest
entry, and decode it either as the PNG a modern favicon embeds or as the
BITMAPINFOHEADER bitmap older ones still carry. The BMP path handles
32/24/8/4/1 bpp, the stacked AND transparency mask, and the all-zero
alpha case where a 32-bpp icon leans on the mask for its shape.
* Add mandatory ZIM metadata for zimcheck
ZIM archives were missing two pieces of metadata that the spec and
zimcheck treat as mandatory: a Description and the Illustrator_48x48@1
favicon Kiwix shows as the book icon. A Name was missing too.
Every archive now writes a Name and a Description, defaulting the
description to a host-derived line when --description is not given. When
the mirror has a usable icon, the favicon is rescaled to a 48x48 PNG and
stored as Illustrator_48x48@1 with an image/png MIME, reusing the icon
discovery and square-fit scaling the app packer already uses.
AddMetadataBytes is added to the zim writer so a binary metadata value
can carry its own MIME instead of being forced to text/plain.
Verified by reading the output back through the libzim engine: all
mandatory keys are present and the illustrator decodes as a 48x48 PNG.
* Update docs for ZIM metadata and current flags
Document the new mandatory metadata in the packing guide and the Kiwix
compatibility note, and default --description in the CLI reference.
While in the reference, bring it back in line with the code: add the
--app and --icon pack flags (shipped in v0.2.0 but never documented),
drop the --max-asset-mb clone flag that does not exist, and fix a stale
--resume mention in the configuration layout.
Add the v0.2.1 release notes and cut the changelog entry.
Give the double-click app work its own 0.2.0 section in the changelog,
above the released 0.1.2, and update the compare links. Summarise it on
the docs release-notes page: kage pack --app wraps the viewer in a
desktop app, the release ships a GUI-subsystem Windows base, and packing
detects the base binary's target OS from its executable header.
Wrapping a packed viewer in a .app or .AppImage was its own --format app
value, parallel to zim and binary. But an app is really just the binary
format with a bundle around it, so a separate format meant duplicating the
base/icon handling and made the three formats feel like an awkward choice.
Turn it into a --app flag that builds on the binary format. It composes
with --base (including a webview base) and --icon, while --format stays
zim or binary. The bundle builders are unchanged; only the CLI surface
moves.
README and the packing guide gain a double-click app section covering the macOS
.app, the Linux .AppDir/.AppImage, favicon icons, and the windows-gui base. The
changelog and docs release notes record the new format under Unreleased.
A second goreleaser build links a Windows binary for the GUI subsystem
(-H windowsgui) and ships it as kage_<version>_windows-gui_<arch>.zip, scoped so
the package managers still install the console build. Packing a viewer onto this
base gives a double-click .exe with no console behind it. A CI job cross-compiles
the windowsgui link so it cannot rot.
The new format dispatches on the base's target OS: a .app on macOS, an .AppDir
(plus an .AppImage when appimagetool is present) on Linux, and a friendly redirect
to --format binary on Windows, where the .exe is already the app. The icon comes
from the mirror's favicon by default and can be overridden with --icon.
Share the base++zim++trailer assembly behind a small helper, then add two bundle
builders on top: BuildApp writes a macOS .app (Info.plist, the viewer under
Contents/MacOS, an .icns in Resources), and BuildAppDir writes an AppImage-style
.AppDir (AppRun, a Terminal=false .desktop launcher, a PNG icon). Both keep the
trailer at the tail so Embedded still finds the archive at runtime.
A pure-Go .icns writer (PNG-embedded entries from 16 to 1024) and a finder that
digs the site's favicon out of a cloned mirror, preferring a large apple-touch
icon and unwrapping a PNG-based .ico. These feed the bundle icon for the upcoming
app formats. Adds golang.org/x/image for high-quality resampling.
Move the Unreleased entries into a 0.1.2 section in the changelog and
summarise the release on the docs release-notes page: the Chrome sandbox
now stays on by default, asset downloads retry on a transient failure,
crawl errors report a clear reason and provenance, and the container
image runs again.
Two failures stopped a docker run from producing anything. Chrome
aborted on launch with 'chrome_crashpad_handler: --database is
required', because its crash reporter cannot start in a minimal
container, so disable the crash reporter on the container launch path.
kage never uploads Chrome crash dumps, so nothing is lost.
The image also created the kage user without a home directory, so HOME
was an unwritable /home/kage. kage writes its default output and resume
state under $HOME/data/kage and Chrome puts its profile and crash
database under HOME too, so both failed with a permission error and the
mounted /out volume captured nothing. Point HOME at the /out volume so
all of it lands somewhere writable that the mount picks up.
The GitHub Ubuntu runner disables unprivileged user namespaces with
AppArmor, so Chrome's sandbox cannot initialize and the new secure
default (sandbox on) makes Chrome refuse to start. IN_DOCKER is the
documented escape hatch for that case, so set it on the Ubuntu leg of
the test job. It also exercises the container code path. macOS does not
need the flag, so it stays empty there.
The crawl printed asset failures as "asset error <url>: status 403 for
<url>", repeating the URL and saying nothing about which page wanted the
file or whether the failure was worth worrying about. The final summary then
collapsed everything into a single error count.
Give failures a classified reason (HTTP 403 Forbidden, timed out, ...), name
the page that referenced the asset, and list what went wrong in the summary
instead of only counting it. Failures are collected during the run and capped
so a broken site cannot grow the list without bound.
Retry transient failures (403/429, 5xx, network blips) with a short backoff.
Bot-protection in front of a site often rejects the first request of a burst
but serves a retry fine, which is exactly what cost us stylesheets on a busy
crawl. Permanent failures (404, 401, ...) are not retried.
kage launched Chrome with --no-sandbox unconditionally, which turns off the
browser's main security boundary for every run, including ordinary desktop
use where the sandbox works fine. Since kage renders pages from the open web,
a renderer exploit could then reach the host. Reported in #10.
Keep the sandbox on by default and drop it only where it genuinely cannot
initialize: inside a container, or when running as root (Chrome refuses to
start a sandbox as root). Containers are detected from IN_DOCKER or the
/.dockerenv marker, and there kage also sets --disable-dev-shm-usage because
the default 64 MB /dev/shm is too small for the renderer on large pages.
Whenever the sandbox is dropped kage says so on stderr, so it is never silent.
Thanks to Dimitrios Prasakis for the report and to the commenter on Hacker
News who suggested the IN_DOCKER opt-in.
The gofmt gate listed packages by hand and missed pack, zim, and viewer, so a
formatting slip in the newer code would sail through. Check the whole module
instead. Add a macOS job that compiles the -tags webview viewer, the cgo path
the pure-Go CI never builds; the viewer code is identical across platforms, so
one compile guards it. Also note the new base-OS detection in the docs.
Packing with --base pointed at a kage built for another OS used to guess the
target from the file name: a base ending in .exe meant Windows, anything else
meant the host. That misfired in both directions. A Windows base without the
.exe suffix produced a viewer with no extension that will not run on Windows,
and an --out name that dropped .exe made the run hint print a macOS quarantine
note for a Windows file.
Sniff the base's executable header (ELF, PE, Mach-O) instead, so the target OS
comes from the bytes rather than the name. A Windows viewer now always gets a
.exe suffix, and the run hint names the real target and only mentions Gatekeeper
for actual macOS viewers.
errcheck flagged six naked defer Close() calls in the pack code and its
tests. Wrap them in the same defer func() { _ = x.Close() }() form the rest
of the tree already uses.
Rewrite the README around a real example, mirroring paulgraham.com for
offline reading, and split packing into two clean sections: a single ZIM
file (with what ZIM is and how to read it back through Kiwix) and a
self-contained binary. Re-record the demo gif against paulgraham.com and
add a screenshot of the native window serving the essays offline. Carry the
same framing into the docs intro pages and the packing guide, and cut the
v0.1.1 release notes.
Rework the README into the house style: badges, a one-line pitch, an
anchor nav, a commands table, and dedicated sections for clone, pack, and
the native viewer. Every flag and default is checked against the current
binary so the docs match what kage actually does.
Add a demo recorded with ascii-gif. The tape clones example.com, packs it
to a ZIM and to a self-contained binary, and serves it back offline, so
the whole loop reads in one frame. It sits at the top of the README and on
the docs home.
While reviewing the docs, fix the output path everywhere: the default is
$HOME/data/kage, not the kage-out the pages claimed, including a few
fabricated 'done kage-out/...' lines. Document pack, open, and the native
viewer in the release notes.
A packed binary opened the system browser, so it felt like a tab, not
an app. Build with -tags webview (cgo) and the viewer instead opens the
site in its own window backed by the OS WebView: WKWebView on macOS,
WebView2 on Windows, WebKitGTK on Linux.
The viewer package picks an implementation at build time. The default
file opens the browser and keeps the build pure Go, so CGO_ENABLED=0 and
the release pipeline are untouched. The webview file links the platform
WebView and runs its event loop on the main goroutine, which main now
pins with LockOSThread before anything else, since macOS requires UI on
the initial thread. Both kage open and the embedded viewer serve over
HTTP in a goroutine and hand the URL to the viewer, then tear the server
down when the window closes or Ctrl-C cancels.
The window title comes from the archive's M/Title. OpenInBrowser moves
out of pack into the viewer package, its only caller.
Gautam Kumar