Compressing clusters with zstd is the slow part of packing a large
mirror. When a mirror is re-packed after a small change, most clusters
are byte-identical and there is no reason to compress them again.
pack --incremental keeps a content-addressed cache of compressed
clusters in a sidecar next to the output. On the next pack, a cluster
whose uncompressed bytes match the cache is served from it instead of
being recompressed; only new or changed clusters go through zstd. A
cache hit returns exactly what a fresh compression would, so the archive
stays deterministic and byte-identical to a cold pack.
The zim writer gains a settable cluster compressor so the cache can hook
in without changing the format. The cache only writes back the clusters
it touched this run, so clusters that left the mirror drop out and it
cannot grow without bound.
* Add mandatory ZIM metadata for zimcheck
ZIM archives were missing two pieces of metadata that the spec and
zimcheck treat as mandatory: a Description and the Illustrator_48x48@1
favicon Kiwix shows as the book icon. A Name was missing too.
Every archive now writes a Name and a Description, defaulting the
description to a host-derived line when --description is not given. When
the mirror has a usable icon, the favicon is rescaled to a 48x48 PNG and
stored as Illustrator_48x48@1 with an image/png MIME, reusing the icon
discovery and square-fit scaling the app packer already uses.
AddMetadataBytes is added to the zim writer so a binary metadata value
can carry its own MIME instead of being forced to text/plain.
Verified by reading the output back through the libzim engine: all
mandatory keys are present and the illustrator decodes as a 48x48 PNG.
* Update docs for ZIM metadata and current flags
Document the new mandatory metadata in the packing guide and the Kiwix
compatibility note, and default --description in the CLI reference.
While in the reference, bring it back in line with the code: add the
--app and --icon pack flags (shipped in v0.2.0 but never documented),
drop the --max-asset-mb clone flag that does not exist, and fix a stale
--resume mention in the configuration layout.
Add the v0.2.1 release notes and cut the changelog entry.
errcheck flagged six naked defer Close() calls in the pack code and its
tests. Wrap them in the same defer func() { _ = x.Close() }() form the rest
of the tree already uses.
BuildZIM walks a cloned host directory, turns every file into a content
entry with a MIME inferred from its extension, picks a main page, and adds
the standard metadata plus a mainPage redirect. state.json is skipped.
BuildBinary appends the archive to a copy of kage behind a KAGEPCK1 trailer,
and Embedded detects that trailer at startup so the binary serves itself.
Handler maps / to the main page and /path to a content entry, the same
handler the embedded viewer uses.
Tam Nguyen Duc