kage parquet export turns a packed archive into a columnar Parquet table,
one row per entry, and kage parquet import rebuilds the archive from it. The
table follows the open-index/open-markdown field names (doc_id, url, host,
crawl_date, content_length, text_length, text) so a kage export sits next to
other web-crawl datasets on Hugging Face and reads straight into DuckDB or
pandas. Alongside those columns it keeps the raw content bytes and the ZIM
structure (namespace, redirect target), so the round trip is lossless: a ZIM
exported and reimported is byte-identical.
doc_id is a deterministic UUID v5 of the page URL. HTML pages also get a
derived plain-text column for full-text search and training use; it plays no
part in the round trip, which rebuilds pages from the stored content.
Compressing clusters with zstd is the slow part of packing a large
mirror. When a mirror is re-packed after a small change, most clusters
are byte-identical and there is no reason to compress them again.
pack --incremental keeps a content-addressed cache of compressed
clusters in a sidecar next to the output. On the next pack, a cluster
whose uncompressed bytes match the cache is served from it instead of
being recompressed; only new or changed clusters go through zstd. A
cache hit returns exactly what a fresh compression would, so the archive
stays deterministic and byte-identical to a cold pack.
The zim writer gains a settable cluster compressor so the cache can hook
in without changing the format. The cache only writes back the clusters
it touched this run, so clusters that left the mirror drop out and it
cannot grow without bound.
* Add mandatory ZIM metadata for zimcheck
ZIM archives were missing two pieces of metadata that the spec and
zimcheck treat as mandatory: a Description and the Illustrator_48x48@1
favicon Kiwix shows as the book icon. A Name was missing too.
Every archive now writes a Name and a Description, defaulting the
description to a host-derived line when --description is not given. When
the mirror has a usable icon, the favicon is rescaled to a 48x48 PNG and
stored as Illustrator_48x48@1 with an image/png MIME, reusing the icon
discovery and square-fit scaling the app packer already uses.
AddMetadataBytes is added to the zim writer so a binary metadata value
can carry its own MIME instead of being forced to text/plain.
Verified by reading the output back through the libzim engine: all
mandatory keys are present and the illustrator decodes as a 48x48 PNG.
* Update docs for ZIM metadata and current flags
Document the new mandatory metadata in the packing guide and the Kiwix
compatibility note, and default --description in the CLI reference.
While in the reference, bring it back in line with the code: add the
--app and --icon pack flags (shipped in v0.2.0 but never documented),
drop the --max-asset-mb clone flag that does not exist, and fix a stale
--resume mention in the configuration layout.
Add the v0.2.1 release notes and cut the changelog entry.
ZIM is the open single-file archive format Kiwix uses for offline content:
a fixed header, a MIME list, URL/title/cluster pointer lists, directory
entries, zstd-compressed or stored clusters, and a trailing MD5. The writer
lays out a mirror in two passes (assign positions, then emit bytes) and
derives the UUID from the content so packing is deterministic. The reader
random-accesses entries by namespace and url, follows redirects, and reads
xz clusters too so archives from other tooling open.
Cross-checked against an independent reader (gozim): header, MIME list,
namespaces, urls, dirents, and a non-last cluster's blob all read back
byte-for-byte.
Tam Nguyen Duc