* Drop the embedded leakless binary that trips antivirus (#68)
go-rod's launcher imports github.com/ysmood/leakless, which base64/gzip
embeds a prebuilt leakless.exe for every target. On Windows that helper is
linked straight into kage.exe, and Windows Defender flags its signature and
quarantines a fresh scoop install before kage ever runs.
kage already launches Chrome with leakless disabled (browser/leakless.go),
so the guard was never doing anything, only adding the flagged bytes. This
adds an API-compatible stub for the package under third_party/leakless with
no embedded binary and points a replace directive at it. The Windows build
loses about 1.28 MB of packed executable and no longer carries the payload
that antivirus reacts to. Support() returns false, so go-rod skips the
leakless path even if a caller re-enabled it.
* Document the leakless antivirus fix for v0.3.9
Add the release-notes and changelog entries for the leakless helper removal
(#68), so the docs site and CHANGELOG explain why the Windows build shrank and
what the earlier virus warning was.
* Drop the now-unused leakless module hash from go.sum
The replace points leakless at a local directory, so go mod tidy no longer
needs the upstream module's checksum. Keeps the tidy CI check green.
The tap distributes kage as a Cask (prebuilt binary), not a Formula,
so brew needs --cask to find it. Without the flag brew errors out with
"No available formula or cask with the name kage".
Documents the signed apt and dnf repository alongside Homebrew and Scoop, and
fires a repository_dispatch on release so the Linux repo rebuilds with the new
packages. The step is skipped when the dispatch token is unset.
The README never mentioned the package managers and the docs covered Homebrew
but not Scoop, even though both ship from every release. Add the Homebrew
(macOS) and Scoop (Windows) channels to both. Fix the Linux package globs:
GoReleaser names them kage_<ver>_amd64.deb and kage-<ver>-1.x86_64.rpm, not
kage_*_linux_amd64.{deb,rpm}, so the documented commands matched nothing.
A packed page entry took its URL path as its title, so a ZIM reader's
search box suggested filenames like 5founders.html instead of readable
titles. The page's <title> was only read for the archive-level M/Title.
Read each HTML page's <title> at pack time and store it on the entry,
collapsing wrapped whitespace to one line and falling back to the path
when a page has none. A reader's suggestion search walks the title
pointer list, which is sorted by title, so typing now offers entries
like Five Founders and Female Founders. The per-page title also lands in
the title column of a parquet export.
Document the search story in the packing guide: title suggestions in any
reader, and full-text search of page bodies through parquet and DuckDB.
A Xapian full-text index stays out on purpose, since Xapian is GPL and
kage is MIT.
Move the Unreleased fix into a dated 0.3.1 section, update the compare
links, and add a v0.3.1 summary to the docs release-notes page. This
patch redirects / to the main page so a mirror whose entry point is a
nested page keeps its CSS and images when served.
Move the Unreleased entries into a dated 0.3.0 section, update the
compare links, and add a v0.3.0 summary to the docs release-notes page.
This release leaves bulk and off-domain assets remote, skips over-cap
downloads instead of truncating them, adds parquet export and import,
incremental packing, and identical-page dedup.
* Add mandatory ZIM metadata for zimcheck
ZIM archives were missing two pieces of metadata that the spec and
zimcheck treat as mandatory: a Description and the Illustrator_48x48@1
favicon Kiwix shows as the book icon. A Name was missing too.
Every archive now writes a Name and a Description, defaulting the
description to a host-derived line when --description is not given. When
the mirror has a usable icon, the favicon is rescaled to a 48x48 PNG and
stored as Illustrator_48x48@1 with an image/png MIME, reusing the icon
discovery and square-fit scaling the app packer already uses.
AddMetadataBytes is added to the zim writer so a binary metadata value
can carry its own MIME instead of being forced to text/plain.
Verified by reading the output back through the libzim engine: all
mandatory keys are present and the illustrator decodes as a 48x48 PNG.
* Update docs for ZIM metadata and current flags
Document the new mandatory metadata in the packing guide and the Kiwix
compatibility note, and default --description in the CLI reference.
While in the reference, bring it back in line with the code: add the
--app and --icon pack flags (shipped in v0.2.0 but never documented),
drop the --max-asset-mb clone flag that does not exist, and fix a stale
--resume mention in the configuration layout.
Add the v0.2.1 release notes and cut the changelog entry.
Give the double-click app work its own 0.2.0 section in the changelog,
above the released 0.1.2, and update the compare links. Summarise it on
the docs release-notes page: kage pack --app wraps the viewer in a
desktop app, the release ships a GUI-subsystem Windows base, and packing
detects the base binary's target OS from its executable header.
Wrapping a packed viewer in a .app or .AppImage was its own --format app
value, parallel to zim and binary. But an app is really just the binary
format with a bundle around it, so a separate format meant duplicating the
base/icon handling and made the three formats feel like an awkward choice.
Turn it into a --app flag that builds on the binary format. It composes
with --base (including a webview base) and --icon, while --format stays
zim or binary. The bundle builders are unchanged; only the CLI surface
moves.
README and the packing guide gain a double-click app section covering the macOS
.app, the Linux .AppDir/.AppImage, favicon icons, and the windows-gui base. The
changelog and docs release notes record the new format under Unreleased.
Move the Unreleased entries into a 0.1.2 section in the changelog and
summarise the release on the docs release-notes page: the Chrome sandbox
now stays on by default, asset downloads retry on a transient failure,
crawl errors report a clear reason and provenance, and the container
image runs again.
The gofmt gate listed packages by hand and missed pack, zim, and viewer, so a
formatting slip in the newer code would sail through. Check the whole module
instead. Add a macOS job that compiles the -tags webview viewer, the cgo path
the pure-Go CI never builds; the viewer code is identical across platforms, so
one compile guards it. Also note the new base-OS detection in the docs.
Rewrite the README around a real example, mirroring paulgraham.com for
offline reading, and split packing into two clean sections: a single ZIM
file (with what ZIM is and how to read it back through Kiwix) and a
self-contained binary. Re-record the demo gif against paulgraham.com and
add a screenshot of the native window serving the essays offline. Carry the
same framing into the docs intro pages and the packing guide, and cut the
v0.1.1 release notes.
Rework the README into the house style: badges, a one-line pitch, an
anchor nav, a commands table, and dedicated sections for clone, pack, and
the native viewer. Every flag and default is checked against the current
binary so the docs match what kage actually does.
Add a demo recorded with ascii-gif. The tape clones example.com, packs it
to a ZIM and to a self-contained binary, and serves it back offline, so
the whole loop reads in one frame. It sits at the top of the README and on
the docs home.
While reviewing the docs, fix the output path everywhere: the default is
$HOME/data/kage, not the kage-out the pages claimed, including a few
fabricated 'done kage-out/...' lines. Document pack, open, and the native
viewer in the release notes.
A packed binary opened the system browser, so it felt like a tab, not
an app. Build with -tags webview (cgo) and the viewer instead opens the
site in its own window backed by the OS WebView: WKWebView on macOS,
WebView2 on Windows, WebKitGTK on Linux.
The viewer package picks an implementation at build time. The default
file opens the browser and keeps the build pure Go, so CGO_ENABLED=0 and
the release pipeline are untouched. The webview file links the platform
WebView and runs its event loop on the main goroutine, which main now
pins with LockOSThread before anything else, since macOS requires UI on
the initial thread. Both kage open and the embedded viewer serve over
HTTP in a goroutine and hand the URL to the viewer, then tear the server
down when the window closes or Ctrl-C cancels.
The window title comes from the archive's M/Title. OpenInBrowser moves
out of pack into the viewer package, its only caller.
Crawling keyed off the raw URL, so the same page reached over http and
https, or as /index.html versus /, was a different frontier entry that
nonetheless wrote to the same file. A clone of paulgraham.com did 948
render passes for 474 files. Key pages and assets by the local path they
write instead, and collapse a directory-index document to its directory,
so each page is fetched exactly once.
Add --refresh to re-render a mirror in place (re-fetch every page, keep
the directory, overwrite) and make --no-resume truly stateless by not
persisting state.json. The default remains a resumable, idempotent crawl
that skips work already on disk.
kage renders every page in headless Chrome, snapshots the final
DOM, strips all JavaScript, and localises CSS, images, and fonts
so a site can be browsed offline as a plain folder of files.
The engine is split into small packages:
urlx deterministic URL to local-path mapping and scope rules
sanitize remove scripts, on* handlers, and javascript: URLs
asset rewrite HTML and CSS references, download assets
browser headless Chrome pool over the DevTools protocol
robots robots.txt matcher
clone the orchestrator: a polite resumable breadth-first crawl
The cli package wires a cobra and fang command surface with two
commands, clone and serve. Every pure package has table tests; the
browser and clone packages add Chrome-driven end-to-end tests that
skip when no browser is present or under -short.
CI runs gofmt, vet, build, race tests, golangci-lint, govulncheck,
and a tidy check on Linux and macOS. A goreleaser config fans one
tag out to archives, deb/rpm/apk, a Chromium-bundled GHCR image,
and the package managers. A tago docs site builds to Pages and
Cloudflare.
Tam Nguyen Duc