Duc-Tam Nguyen
d59c85afc8
The crawl printed asset failures as "asset error <url>: status 403 for <url>", repeating the URL and saying nothing about which page wanted the file or whether the failure was worth worrying about. The final summary then collapsed everything into a single error count. Give failures a classified reason (HTTP 403 Forbidden, timed out, ...), name the page that referenced the asset, and list what went wrong in the summary instead of only counting it. Failures are collected during the run and capped so a broken site cannot grow the list without bound. Retry transient failures (403/429, 5xx, network blips) with a short backoff. Bot-protection in front of a site often rejects the first request of a burst but serves a retry fine, which is exactly what cost us stylesheets on a busy crawl. Permanent failures (404, 401, ...) are not retried.
4.9 KiB
4.9 KiB
Changelog
All notable changes to kage are recorded here. The format follows Keep a Changelog, and the project aims to follow Semantic Versioning.
Unreleased
Security
- Chrome now keeps its sandbox on by default. It was previously launched with
--no-sandboxunconditionally, which removed Chrome's main line of defense when rendering pages from the open web (reported in #10). The sandbox is now dropped only where it genuinely cannot run: inside a container, or when running as root, and the choice is logged so it is never silent.
Added
- Container-aware Chrome flags. kage detects a container from the
IN_DOCKERenvironment variable or a/.dockerenvmarker and, only there, drops the sandbox and adds--disable-dev-shm-usage(the default 64 MB/dev/shmis too small for Chrome on large pages). Outside a container the faster shared memory is left in place. - Asset downloads retry on a transient failure (a 403/429, a 5xx, or a network blip) with a short backoff, recovering files that bot-protection rejects on the first request of a burst. Permanent failures (404, 401, ...) are not retried.
Changed
- Clearer crawl error reporting. Each failure is logged with a classified reason (
HTTP 403 Forbidden,timed out, ...), the URL, and the page that referenced it, and the end-of-run summary lists what went wrong instead of printing only a count.
0.1.1 - 2026-06-14
Added
kage pack <mirror-dir>packs a cloned folder into one distributable file.--format zim(the default) writes an open ZIM archive, the same single-file format Kiwix uses;--format binaryappends that archive to a copy of kage to produce a self-contained executable that serves the site offline when run. Flags cover the output path, metadata (--title,--description,--language,--date), a--basebinary for cross-platform viewers, and--no-compress.kage open <file.zim>serves a packed ZIM over a local HTTP server and opens your browser, the read side ofkage pack --format zim.- An optional native-window viewer. Built with
-tags webview(which needs cgo),kage openand a packed binary present the offline site in a real window backed by the operating system's WebView (WKWebView, WebView2, WebKitGTK) instead of a browser tab, so a packed kage feels like a standalone app. The default build stays pure Go (CGO_ENABLED=0) and falls back to the system browser, so the release pipeline is unchanged. - A pure-Go
zimpackage that writes and reads the ZIM format: a fixed header, MIME and pointer lists, zstd-compressed (or stored) clusters, redirects, and a trailing MD5. It reads xz clusters so archives from other tooling open, and writes zstd or stored only. Packing is deterministic: the same mirror produces a byte-identical archive, with the UUID derived from the content rather than randomised.
0.1.0 - 2026-06-14
The first release. kage clones a live website into a self-contained folder you can browse offline, with every script stripped out.
Added
kage clone <url>renders each page in headless Chrome, snapshots the final DOM, removes every<script>,on*handler, andjavascript:URL, and downloads the CSS, images, fonts, and media, rewriting them to local paths.kage serve [dir]runs a local static file server over a cloned folder so the mirror's links and assets resolve the way they would on a real host.- Deterministic URL-to-path mapping: pages become
<slug>/index.htmldirectories, assets live under the reserved_kage/<host>/tree, and query strings fold into a short hash suffix so versioned URLs never collide. - Three concurrency tiers run in parallel: page-render workers (
--workers), asset-download workers (--asset-workers), and a Chrome page pool (--browser-pages). - A polite crawl by default: honours
robots.txt, seeds fromsitemap.xml, and scopes to the seed host.--scope-prefix,--max-depth,--max-pages,--subdomains, and--excludeshape the frontier. - Idempotent, resumable crawling. Each page is keyed by the file it writes, so
the same URL reached over http and https, with or without a trailing slash,
or as
/index.htmlversus/, is fetched exactly once. A re-run resumes from_kage/state.json;--refreshre-renders a mirror in place to pull in changed content;--forcewipes and starts clean;--no-resumeruns stateless. - Defaults to a per-user data directory (
$HOME/data/kage), overridable with-o/--out. - Cross-platform distribution: prebuilt archives,
.deb/.rpm/.apkpackages, a multi-arch container image on GHCR (Chromium bundled), checksums, SBOMs, and a cosign signature, all cut from one version tag by GoReleaser.