Without a directory the scoop manifest landed at the repository root,
while the seeded manifests and the path scoop reads live under bucket/.
A scoop install would keep resolving the stale root-shadowed manifest.
Set directory: bucket so the release overwrites bucket/<tool>.json, the
same place the manifest is served from.
GoReleaser validates that a repository token template is exactly
{{ .Env.VAR_NAME }} and rejects any other interpolation, so the
index .Env form aborted the publish with 'expected {{ .Env.VAR_NAME }}
only'. The token is only rendered when an upload actually happens, and
in CI the secret is always set, so the bare form is safe here.
skip_upload keeps the index .Env form: it is not subject to the token
rule and index degrades to an empty string when the secret is unset, so
an unconfigured fork still skips the upload instead of erroring.
The Homebrew cask and Scoop manifest stanzas gated their token and
skip_upload on envOrDefault, which is a sprig function GoReleaser does
not define, so a real release aborted with 'function envOrDefault not
defined' the moment it reached the publish phase. goreleaser check does
not evaluate templates, so the gate slipped through until a tagged run.
Switch to index .Env "NAME", the idiom GoReleaser documents for an
optional variable: it yields the token when the secret is set (push the
update) and an empty string when it is not (skip_upload stays true).
cosign 3.x flips the new bundle format on by default, which ignores the
--output-signature and --output-certificate flags our signs block passes
to sign-blob and then aborts trying to write a bundle to an empty path.
The last few releases failed at the signing step for exactly this reason.
Pin cosign-installer to v2.6.3 so the release keeps producing the
checksums.txt.sig and checksums.txt.pem pair, and so the signing tool
stops floating to a latest that can break the pipeline without warning.
Documents the signed apt and dnf repository alongside Homebrew and Scoop, and
fires a repository_dispatch on release so the Linux repo rebuilds with the new
packages. The step is skipped when the dispatch token is unset.
The README never mentioned the package managers and the docs covered Homebrew
but not Scoop, even though both ship from every release. Add the Homebrew
(macOS) and Scoop (Windows) channels to both. Fix the Linux package globs:
GoReleaser names them kage_<ver>_amd64.deb and kage-<ver>-1.x86_64.rpm, not
kage_*_linux_amd64.{deb,rpm}, so the documented commands matched nothing.
Three issues surfaced when viewing the ZIM in Kiwix iOS:
1. Blank column at top: hiding img[usemap] left its containing <td> as an
empty box. Now td:has(>img[usemap]) hides the entire nav column.
2. Content clipped on the right: the inner content table had width="435"
as an HTML attribute. Added [width]{width:auto!important} to cancel all
fixed HTML attribute widths on any element (tables, tds, imgs).
3. Spacer column: the 26px <td> holding a 1x1 transparent GIF kept its
allocated space. td:has(>img[src*="trans_1x1"]:only-child) hides it.
Also: overflow-x:hidden on body, box-sizing:border-box globally, and
img{max-width:100%;height:auto} so any inline images stay within column.
Cloning a 1990s/2000s site like paulgraham.com and opening it in Kiwix
on a phone produces microscopic text: the pages use <font size="2">,
table layouts, and no viewport declaration, so the mobile browser shrinks
everything to desktop scale and then the font-size attribute makes it
smaller still.
kage clone --mobile injects two things into every saved page:
- <meta name="viewport" content="width=device-width, initial-scale=1">
so the browser stops shrinking the page
- a <style> block that lifts the base font to 18px, inherits it through
<font> elements (overriding the in-HTML size/face attributes), caps the
content width at 720px, loosens line height to 1.7, and hides
image-map nav elements whose source GIFs 404 offline
The style block goes last in <head> to win specificity ties, and
ensureViewport skips pages that already carry a viewport meta.
Two tests cover the happy path and the no-duplicate case.
A packed page entry took its URL path as its title, so a ZIM reader's
search box suggested filenames like 5founders.html instead of readable
titles. The page's <title> was only read for the archive-level M/Title.
Read each HTML page's <title> at pack time and store it on the entry,
collapsing wrapped whitespace to one line and falling back to the path
when a page has none. A reader's suggestion search walks the title
pointer list, which is sorted by title, so typing now offers entries
like Five Founders and Female Founders. The per-page title also lands in
the title column of a parquet export.
Document the search story in the packing guide: title suggestions in any
reader, and full-text search of page bodies through parquet and DuckDB.
A Xapian full-text index stays out on purpose, since Xapian is GPL and
kage is MIT.
When a page's JavaScript builds deeply nested object graphs, Chrome's
DevTools Protocol returns error -32000 'Object reference chain is too
long' during WaitLoad. The page has still loaded its HTML — the error is
about Chrome's internal object tracking, not the document itself.
This change detects this specific error and proceeds with rendering
instead of failing the entire page, so sites with complex JS still get
cloned successfully (issue #36).
Signed-off-by: Gautam Kumar <gautamkumarofficial@users.noreply.github.com>
An extensionless link is queued as a page, so the page worker navigated to
it in headless Chrome. When such a link served a binary, a zip or a CSV,
Chrome saved the file to the user's Downloads folder, a surprise side effect
of a clone (issue #32).
Deny Chrome-initiated downloads browser-wide, since kage fetches every asset
through its own downloader and never needs the browser to write a file. Then
watch the main document's response, and when it is not HTML, return a typed
ErrNotHTML so the page worker reroutes the URL to the asset downloader, where
the existing size and media policy decides whether to localise it or leave it
on the live web.
Verified against the two URLs from the issue, a zip and a CSV: both land
under the mirror's reserved tree and nothing is written to Downloads.
A page that declared its charset only in the HTTP Content-Type header,
with no meta charset in the markup, lost that signal once kage saved it
as a file. A reader serving the bytes without a charset then fell back to
its locale encoding and garbled every curly quote, dash, and nbsp.
kage writes UTF-8, so it now inserts a meta charset utf-8 at the top of
head when the page does not already declare one. Verified on the blog
post from the report: the saved HTML now carries the meta and renders the
quotes correctly.
Move the Unreleased fix into a dated 0.3.1 section, update the compare
links, and add a v0.3.1 summary to the docs release-notes page. This
patch redirects / to the main page so a mirror whose entry point is a
nested page keeps its CSS and images when served.
The served main page broke its own CSS and images when the entry point
was a nested page. kage saves each page's asset links as mirror-relative
paths (../_kage/...) computed for that page's own location, but the
handler served the main page's bytes directly at /, so the browser
resolved those relative URLs against / and 404ed every one of them. A
developer.apple.com/documentation mirror landed at / with no styles.
Redirect / to the main page's canonical content path, the way the
archive's W/mainPage redirect already does, so the browser navigates to
the page's real URL and resolves its relative assets correctly. Kiwix
was unaffected because it follows that redirect itself.
Move the Unreleased entries into a dated 0.3.0 section, update the
compare links, and add a v0.3.0 summary to the docs release-notes page.
This release leaves bulk and off-domain assets remote, skips over-cap
downloads instead of truncating them, adds parquet export and import,
incremental packing, and identical-page dedup.
A developer.apple.com crawl came back at 19 GB, 18 of it assets, most of
that the site's own videos, .dmg/.pkg installers, and PDF manuals pulled
from a couple dozen hosts including unrelated third parties. None of it
helps read the docs offline. This changes what kage localizes by default.
Bulk media, installers, archives, and PDFs are left pointing at their
live URL instead of downloaded. The decision is made from the URL alone,
so the rewritten HTML simply keeps the remote link. --keep-media restores
the old behavior and --skip-ext adds more extensions to leave remote.
Assets are localized only from the seed's registrable domain by default,
so www.apple.com and images.apple.com still come along but a separate
brand domain or an off-topic third party does not. --all-asset-hosts goes
back to downloading from any host.
The size cap was also truncating instead of skipping: it wrapped the body
in a LimitReader, so an over-cap file was saved as exactly the first N MB
of itself, a corrupt fragment that would never play or run. On the apple
crawl that was around a gigabyte of half-downloaded WWDC videos. kage now
checks the response size and leaves an over-cap asset out of the mirror.
A site busts a stylesheet's cache with a date string, so a page links to
styles/main.css?Thursday, 26-Feb-2026 16:26:41 UTC. A browser encodes the
spaces before requesting, but kage parsed the href and passed RawQuery through
verbatim, so the request line carried literal spaces and the server answered
400 Bad Request. On a developer.apple.com crawl this was the bulk of the
download errors.
Re-encode the query on the canonical URL: percent-encode any byte a query may
not carry (space, control, non-ASCII) while leaving valid sub-delims and
existing %XX escapes alone. Doing it in canonical fixes the fetch and keeps the
on-disk key in step with the request.
kage parquet export turns a packed archive into a columnar Parquet table,
one row per entry, and kage parquet import rebuilds the archive from it. The
table follows the open-index/open-markdown field names (doc_id, url, host,
crawl_date, content_length, text_length, text) so a kage export sits next to
other web-crawl datasets on Hugging Face and reads straight into DuckDB or
pandas. Alongside those columns it keeps the raw content bytes and the ZIM
structure (namespace, redirect target), so the round trip is lossless: a ZIM
exported and reimported is byte-identical.
doc_id is a deterministic UUID v5 of the page URL. HTML pages also get a
derived plain-text column for full-text search and training use; it plays no
part in the round trip, which rebuilds pages from the stored content.
Compressing clusters with zstd is the slow part of packing a large
mirror. When a mirror is re-packed after a small change, most clusters
are byte-identical and there is no reason to compress them again.
pack --incremental keeps a content-addressed cache of compressed
clusters in a sidecar next to the output. On the next pack, a cluster
whose uncompressed bytes match the cache is served from it instead of
being recompressed; only new or changed clusters go through zstd. A
cache hit returns exactly what a fresh compression would, so the archive
stays deterministic and byte-identical to a cold pack.
The zim writer gains a settable cluster compressor so the cache can hook
in without changing the format. The cache only writes back the clusters
it touched this run, so clusters that left the mirror drop out and it
cannot grow without bound.
Two related changes for crawling faceted sites, where one path spawns
thousands of ?q=.../?page=... URLs that all render the same page.
The progress line was counting every written file, so the page number ran
far ahead of the site's real size. It now shows distinct URL paths as
"pages" and folds the query-string permutations into a separate "variants"
count, so the live counter tracks real pages and is easy to read.
Pages with identical bytes are now stored once. The first page with a given
content is written normally; a later page with the same bytes becomes a hard
link to it, so duplicate content never takes disk twice. Links still resolve
because each variant keeps its own path. When hard links are unsupported the
bytes are written, so correctness never depends on the link. The summary
reports how many pages were deduped.
The sanitizer walked only element nodes, so a script hidden in a
downlevel IE conditional comment slipped through. golang.org/x/net/html
parses <!--[if lt IE 9]><script src="..."></script><![endif]--> as a
single comment node whose data holds the raw markup, so the element walk
never sees the <script> and it rendered straight back out, a live-CDN
script reference left sitting in a page kage promises is inert.
Plenty of older docs sites (clojure.org, cordova.apache.org, and the
async library docs among them) still ship html5shiv, respond.js, or
placeholders.js this way.
Drop conditional comments in the walk. The downlevel-hidden form is one
comment and goes whole; the downlevel-revealed form keeps its content,
which lives in sibling nodes, and loses only the two markers.
Valid-Systems