kage parquet export turns a packed archive into a columnar Parquet table,
one row per entry, and kage parquet import rebuilds the archive from it. The
table follows the open-index/open-markdown field names (doc_id, url, host,
crawl_date, content_length, text_length, text) so a kage export sits next to
other web-crawl datasets on Hugging Face and reads straight into DuckDB or
pandas. Alongside those columns it keeps the raw content bytes and the ZIM
structure (namespace, redirect target), so the round trip is lossless: a ZIM
exported and reimported is byte-identical.
doc_id is a deterministic UUID v5 of the page URL. HTML pages also get a
derived plain-text column for full-text search and training use; it plays no
part in the round trip, which rebuilds pages from the stored content.
ZIM is the open single-file archive format Kiwix uses for offline content:
a fixed header, a MIME list, URL/title/cluster pointer lists, directory
entries, zstd-compressed or stored clusters, and a trailing MD5. The writer
lays out a mirror in two passes (assign positions, then emit bytes) and
derives the UUID from the content so packing is deterministic. The reader
random-accesses entries by namespace and url, follows redirects, and reads
xz clusters too so archives from other tooling open.
Cross-checked against an independent reader (gozim): header, MIME list,
namespaces, urls, dirents, and a non-last cluster's blob all read back
byte-for-byte.
Tam Nguyen Duc