* Add mandatory ZIM metadata for zimcheck
ZIM archives were missing two pieces of metadata that the spec and
zimcheck treat as mandatory: a Description and the Illustrator_48x48@1
favicon Kiwix shows as the book icon. A Name was missing too.
Every archive now writes a Name and a Description, defaulting the
description to a host-derived line when --description is not given. When
the mirror has a usable icon, the favicon is rescaled to a 48x48 PNG and
stored as Illustrator_48x48@1 with an image/png MIME, reusing the icon
discovery and square-fit scaling the app packer already uses.
AddMetadataBytes is added to the zim writer so a binary metadata value
can carry its own MIME instead of being forced to text/plain.
Verified by reading the output back through the libzim engine: all
mandatory keys are present and the illustrator decodes as a 48x48 PNG.
* Update docs for ZIM metadata and current flags
Document the new mandatory metadata in the packing guide and the Kiwix
compatibility note, and default --description in the CLI reference.
While in the reference, bring it back in line with the code: add the
--app and --icon pack flags (shipped in v0.2.0 but never documented),
drop the --max-asset-mb clone flag that does not exist, and fix a stale
--resume mention in the configuration layout.
Add the v0.2.1 release notes and cut the changelog entry.
Rework the README into the house style: badges, a one-line pitch, an
anchor nav, a commands table, and dedicated sections for clone, pack, and
the native viewer. Every flag and default is checked against the current
binary so the docs match what kage actually does.
Add a demo recorded with ascii-gif. The tape clones example.com, packs it
to a ZIM and to a self-contained binary, and serves it back offline, so
the whole loop reads in one frame. It sits at the top of the README and on
the docs home.
While reviewing the docs, fix the output path everywhere: the default is
$HOME/data/kage, not the kage-out the pages claimed, including a few
fabricated 'done kage-out/...' lines. Document pack, open, and the native
viewer in the release notes.
Crawling keyed off the raw URL, so the same page reached over http and
https, or as /index.html versus /, was a different frontier entry that
nonetheless wrote to the same file. A clone of paulgraham.com did 948
render passes for 474 files. Key pages and assets by the local path they
write instead, and collapse a directory-index document to its directory,
so each page is fetched exactly once.
Add --refresh to re-render a mirror in place (re-fetch every page, keep
the directory, overwrite) and make --no-resume truly stateless by not
persisting state.json. The default remains a resumable, idempotent crawl
that skips work already on disk.
kage renders every page in headless Chrome, snapshots the final
DOM, strips all JavaScript, and localises CSS, images, and fonts
so a site can be browsed offline as a plain folder of files.
The engine is split into small packages:
urlx deterministic URL to local-path mapping and scope rules
sanitize remove scripts, on* handlers, and javascript: URLs
asset rewrite HTML and CSS references, download assets
browser headless Chrome pool over the DevTools protocol
robots robots.txt matcher
clone the orchestrator: a polite resumable breadth-first crawl
The cli package wires a cobra and fang command surface with two
commands, clone and serve. Every pure package has table tests; the
browser and clone packages add Chrome-driven end-to-end tests that
skip when no browser is present or under -short.
CI runs gofmt, vet, build, race tests, golangci-lint, govulncheck,
and a tidy check on Linux and macOS. A goreleaser config fans one
tag out to archives, deb/rpm/apk, a Chromium-bundled GHCR image,
and the package managers. A tago docs site builds to Pages and
Cloudflare.
Tam Nguyen Duc