Compare commits
9 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| e126968c65 | |||
| 5b4b523419 | |||
| 3af26ae0e5 | |||
| 5b7f7d9f31 | |||
| 26dabd03bf | |||
| 42f57491c0 | |||
| fafa3dfa51 | |||
| ffdb4ca969 | |||
| b59f782165 |
+28
-1
@@ -6,6 +6,32 @@ All notable changes to kage are recorded here. The format follows
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.1.1] - 2026-06-14
|
||||
|
||||
### Added
|
||||
|
||||
- `kage pack <mirror-dir>` packs a cloned folder into one distributable file.
|
||||
`--format zim` (the default) writes an open ZIM archive, the same single-file
|
||||
format Kiwix uses; `--format binary` appends that archive to a copy of kage to
|
||||
produce a self-contained executable that serves the site offline when run.
|
||||
Flags cover the output path, metadata (`--title`, `--description`,
|
||||
`--language`, `--date`), a `--base` binary for cross-platform viewers, and
|
||||
`--no-compress`.
|
||||
- `kage open <file.zim>` serves a packed ZIM over a local HTTP server and opens
|
||||
your browser, the read side of `kage pack --format zim`.
|
||||
- An optional native-window viewer. Built with `-tags webview` (which needs
|
||||
cgo), `kage open` and a packed binary present the offline site in a real
|
||||
window backed by the operating system's WebView (WKWebView, WebView2,
|
||||
WebKitGTK) instead of a browser tab, so a packed kage feels like a standalone
|
||||
app. The default build stays pure Go (`CGO_ENABLED=0`) and falls back to the
|
||||
system browser, so the release pipeline is unchanged.
|
||||
- A pure-Go `zim` package that writes and reads the ZIM format: a fixed header,
|
||||
MIME and pointer lists, zstd-compressed (or stored) clusters, redirects, and a
|
||||
trailing MD5. It reads xz clusters so archives from other tooling open, and
|
||||
writes zstd or stored only. Packing is deterministic: the same mirror produces
|
||||
a byte-identical archive, with the UUID derived from the content rather than
|
||||
randomised.
|
||||
|
||||
## [0.1.0] - 2026-06-14
|
||||
|
||||
The first release. kage clones a live website into a self-contained folder you
|
||||
@@ -39,5 +65,6 @@ can browse offline, with every script stripped out.
|
||||
a multi-arch container image on GHCR (Chromium bundled), checksums, SBOMs, and
|
||||
a cosign signature, all cut from one version tag by GoReleaser.
|
||||
|
||||
[Unreleased]: https://github.com/tamnd/kage/compare/v0.1.0...HEAD
|
||||
[Unreleased]: https://github.com/tamnd/kage/compare/v0.1.1...HEAD
|
||||
[0.1.1]: https://github.com/tamnd/kage/compare/v0.1.0...v0.1.1
|
||||
[0.1.0]: https://github.com/tamnd/kage/releases/tag/v0.1.0
|
||||
|
||||
@@ -10,11 +10,17 @@ LDFLAGS := -s -w \
|
||||
|
||||
export CGO_ENABLED := 0
|
||||
|
||||
.PHONY: build install test test-short vet tidy clean run
|
||||
.PHONY: build build-webview install test test-short vet tidy clean run
|
||||
|
||||
build:
|
||||
go build -ldflags "$(LDFLAGS)" -o bin/$(BIN) $(PKG)
|
||||
|
||||
# A native-window viewer: opens packed sites in their own OS WebView window
|
||||
# instead of the browser. Needs cgo, so it is built separately from the default
|
||||
# pure-Go binary and the release pipeline.
|
||||
build-webview:
|
||||
CGO_ENABLED=1 go build -tags webview -ldflags "$(LDFLAGS)" -o bin/$(BIN) $(PKG)
|
||||
|
||||
install:
|
||||
go install -ldflags "$(LDFLAGS)" $(PKG)
|
||||
|
||||
|
||||
@@ -1,116 +1,177 @@
|
||||
# kage
|
||||
|
||||
**kage** (影, "shadow") clones a website into a self-contained folder you can
|
||||
browse offline, with all the JavaScript stripped out. It renders every page in
|
||||
headless Chrome, snapshots the final rendered DOM, removes every script and
|
||||
event handler, and downloads the CSS, images, and fonts and rewrites them to
|
||||
local paths. The result looks like the live site but runs no code: a plain
|
||||
folder of `.html` files you can open straight from disk.
|
||||
[](https://github.com/tamnd/kage/actions/workflows/ci.yml)
|
||||
[](https://github.com/tamnd/kage/releases/latest)
|
||||
[](https://pkg.go.dev/github.com/tamnd/kage)
|
||||
[](https://goreportcard.com/report/github.com/tamnd/kage)
|
||||
[](./LICENSE)
|
||||
|
||||
```bash
|
||||
kage clone example.com
|
||||
kage serve kage-out/example.com
|
||||
```
|
||||
**kage** (影, "shadow") clones a website into a folder you can browse offline, with every script stripped out. It opens each page in real headless Chrome, waits for the page to settle, snapshots the DOM a human would have seen, then deletes all the JavaScript and pulls the CSS, images, and fonts down to local paths. What lands on disk looks like the live site and runs no code.
|
||||
|
||||
## Why
|
||||
[Install](#install) • [Quick start](#quick-start) • [Commands](#commands) • [Clone](#clone) • [Pack](#pack-it-into-one-file) • [Native window](#a-real-window-not-a-browser-tab) • [How it works](#how-it-works)
|
||||
|
||||
Saving a page with "Save As" gives you a copy that still phones home, still runs
|
||||
analytics, and often renders blank because the markup is built by JavaScript at
|
||||
runtime. kage takes the opposite approach:
|
||||

|
||||
|
||||
- **Render first, save second.** Each page goes through real headless Chrome, so
|
||||
a page whose content is assembled by JavaScript is captured the way a human
|
||||
would have seen it, not as an empty shell.
|
||||
- **Strip every script.** Once the DOM is captured, kage removes all `<script>`
|
||||
tags, every `on*` event handler, and any `javascript:` URL. The saved page is
|
||||
inert: no tracking, no network calls, no surprises.
|
||||
- **Keep the layout.** Stylesheets, images, fonts, and media are downloaded and
|
||||
rewritten to relative local paths, so the offline copy looks like the original.
|
||||
- **Stay browsable.** In-scope links are rewritten to point at the other saved
|
||||
pages, so you can click around the mirror exactly as you would the live site.
|
||||
You already know the problem. You hit "Save As" on a page you want to keep, and six months later you open it to find a blank screen, a spinner that never stops, or a copy that still tries to phone home to an analytics server that no longer exists. The page was never really yours. It was a thin client for someone else's JavaScript.
|
||||
|
||||
kage takes the other road. It drives a real browser, lets the page finish doing whatever it does, grabs the finished result, and then rips every script out of it. No tracking, no network calls, no surprises. Just `.html` files you can open straight off disk, hand to a friend, or pack into a single file and forget about for a decade.
|
||||
|
||||
Full docs and guides live at **[kage.tamnd.com](https://kage.tamnd.com)**.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
# Go
|
||||
go install github.com/tamnd/kage/cmd/kage@latest
|
||||
|
||||
# Homebrew (once the tap is published)
|
||||
brew install tamnd/tap/kage
|
||||
|
||||
# Container (Chromium bundled)
|
||||
docker run -v "$PWD/out:/out" ghcr.io/tamnd/kage clone example.com
|
||||
```
|
||||
|
||||
Prebuilt archives, `.deb`/`.rpm`/`.apk` packages, and a multi-arch image are
|
||||
attached to each [release](https://github.com/tamnd/kage/releases).
|
||||
|
||||
kage drives a real browser, so it needs Chrome or Chromium available. It finds a
|
||||
system install automatically; point it at a specific binary with `--chrome` or
|
||||
the `KAGE_CHROME` environment variable. The container image bundles Chromium.
|
||||
|
||||
## Usage
|
||||
Prefer a prebuilt binary? Grab an archive, a `.deb`/`.rpm`/`.apk`, or a checksum from [releases](https://github.com/tamnd/kage/releases). Or skip installing Chrome yourself and use the container image, which bundles Chromium:
|
||||
|
||||
```bash
|
||||
kage clone <url> [flags]
|
||||
kage serve [dir] [flags]
|
||||
docker run --rm -v "$PWD/out:/out" ghcr.io/tamnd/kage clone paulgraham.com
|
||||
```
|
||||
|
||||
### Clone
|
||||
kage drives a real browser, so it needs Chrome or Chromium on the host. It finds a system install on its own; point it somewhere specific with `--chrome` or the `KAGE_CHROME` environment variable. The container needs nothing extra.
|
||||
|
||||
Shell completion ships in the box: `kage completion bash|zsh|fish|powershell`.
|
||||
|
||||
## Quick start
|
||||
|
||||
Let's mirror Paul Graham's essays so you can read them on a plane, on a laptop with no wifi, or in the year 2050 after the site has finally changed its design:
|
||||
|
||||
```bash
|
||||
# Clone a whole site into kage-out/<host>/
|
||||
kage clone https://example.com
|
||||
# 1. Clone the site into $HOME/data/kage/paulgraham.com/
|
||||
kage clone paulgraham.com
|
||||
|
||||
# Limit the crawl
|
||||
kage clone example.com --max-pages 200 --max-depth 3
|
||||
# 2. Read it back offline in your browser
|
||||
kage serve $HOME/data/kage/paulgraham.com
|
||||
# open http://127.0.0.1:8800
|
||||
```
|
||||
|
||||
# Only a section of the site
|
||||
kage clone example.com --scope-prefix /docs
|
||||
That's the whole loop. Every essay, every image, every stylesheet, frozen on your disk and runnable with zero network. The next two steps are optional but nice: collapse the whole thing into one file, and pop it open in its own window.
|
||||
|
||||
# Include subdomains, and trigger lazy-loaded images by scrolling
|
||||
```bash
|
||||
# 3. Squeeze the mirror into a single shareable file
|
||||
kage pack paulgraham.com # -> paulgraham.com.zim
|
||||
kage open paulgraham.com.zim
|
||||
|
||||
# 4. Or into one executable that *is* the site
|
||||
kage pack paulgraham.com --format binary -o paulgraham
|
||||
./paulgraham # serves itself, needs nothing installed
|
||||
```
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | What it does |
|
||||
| --- | --- |
|
||||
| `kage clone <url>` | render a site in headless Chrome and write a browsable, script-free mirror |
|
||||
| `kage serve [dir]` | preview a cloned folder over a local HTTP server |
|
||||
| `kage pack <mirror-dir>` | collapse a mirror into one ZIM archive, or a self-contained viewer binary |
|
||||
| `kage open <file.zim>` | serve a packed ZIM back for offline reading |
|
||||
|
||||
## Clone
|
||||
|
||||
```bash
|
||||
# The whole site, into $HOME/data/kage/<host>/
|
||||
kage clone https://paulgraham.com
|
||||
|
||||
# Just the first 50 pages, two links deep, for a quick taste
|
||||
kage clone paulgraham.com --max-pages 50 --max-depth 2
|
||||
|
||||
# Only one section of a bigger site
|
||||
kage clone go.dev --scope-prefix /doc
|
||||
|
||||
# Pull in subdomains too, and scroll each page to trip lazy-loaded images
|
||||
kage clone example.com --subdomains --scroll
|
||||
|
||||
# Resume an interrupted run (on by default; Ctrl-C saves state)
|
||||
kage clone example.com
|
||||
|
||||
# Re-render every page in place to pull in changed content
|
||||
kage clone example.com --refresh
|
||||
# Come back next month and re-render in place to catch new essays
|
||||
kage clone paulgraham.com --refresh
|
||||
```
|
||||
|
||||
A clone is idempotent: each page is keyed by the file it writes, so the same URL
|
||||
reached over http and https, with or without a trailing slash, is fetched once.
|
||||
Re-running resumes where it left off; `--refresh` re-renders in place, `--force`
|
||||
wipes and starts clean.
|
||||
A clone is a polite, breadth-first crawl. It reads `robots.txt`, seeds itself from `sitemap.xml`, and stays on the seed host unless you tell it otherwise. It is also stubbornly idempotent: each page is keyed by the file it writes, so the same essay reached over http and https, with or without a trailing slash, gets fetched exactly once. Hit Ctrl-C and it saves its place on the way out; run it again and it picks up where it stopped. `--refresh` re-renders in place, `--force` wipes the host and starts clean.
|
||||
|
||||
Common flags:
|
||||
The flags you'll actually reach for:
|
||||
|
||||
| Flag | Default | Meaning |
|
||||
|------|---------|---------|
|
||||
| `-o, --out` | `$HOME/data/kage` | Output root; the mirror lands in `<out>/<host>/` |
|
||||
| `-p, --max-pages` | `0` | Stop after N pages (0 = unlimited) |
|
||||
| `-d, --max-depth` | `0` | Link-follow depth cap (0 = unlimited) |
|
||||
| `--scope-prefix` | | Only crawl pages whose path starts with this prefix |
|
||||
| `-p, --max-pages` | `0` | Stop after N pages (0 = no limit) |
|
||||
| `-d, --max-depth` | `0` | How many links deep to follow (0 = no limit) |
|
||||
| `--scope-prefix` | | Only crawl paths starting with this prefix |
|
||||
| `--subdomains` | `false` | Treat subdomains of the seed host as in scope |
|
||||
| `--exclude` | | Path prefixes to skip (repeatable) |
|
||||
| `--scroll` | `false` | Auto-scroll each page to trigger lazy loading |
|
||||
| `--workers` | `4` | Concurrent page render workers |
|
||||
| `--no-robots` | `false` | Ignore `robots.txt` (be polite) |
|
||||
| `--workers` | `4` | How many pages to render at once |
|
||||
| `--no-robots` | `false` | Ignore `robots.txt` (be nice) |
|
||||
| `-f, --force` | `false` | Delete any existing mirror for the host first |
|
||||
| `--chrome` | | Path to the Chrome/Chromium binary |
|
||||
|
||||
Run `kage clone --help` for the full list.
|
||||
`kage clone --help` has the rest, including render-timing, concurrency, and asset-size knobs.
|
||||
|
||||
### Serve
|
||||
|
||||
`kage serve` runs a local static file server over a cloned folder so links and
|
||||
assets resolve the way they would on a real host:
|
||||
`kage serve` runs a tiny static file server over a cloned folder so links and assets resolve the way they would on a real host:
|
||||
|
||||
```bash
|
||||
kage serve kage-out/example.com
|
||||
kage serve $HOME/data/kage/paulgraham.com
|
||||
# open http://127.0.0.1:8800
|
||||
```
|
||||
|
||||
## Pack it into one file
|
||||
|
||||
A mirror is a folder, which is great for browsing and lousy for moving around. Copying thousands of little files is slow, and "here, have this directory" is a clumsy thing to hand someone. `kage pack` collapses the whole mirror into one artifact, and you choose the shape: an open ZIM archive, or a single executable that *is* the site.
|
||||
|
||||
### A single ZIM file
|
||||
|
||||
```bash
|
||||
kage pack paulgraham.com # -> paulgraham.com.zim
|
||||
kage open paulgraham.com.zim
|
||||
```
|
||||
|
||||
ZIM is an open file format built for exactly this: a whole website (or a whole Wikipedia) squeezed into one compressed, indexed, read-only file. kage writes the entire mirror into it, text zstd-compressed and media stored as-is. It is the format behind [Kiwix](https://kiwix.org), the offline-content project people use to carry Wikipedia, Stack Overflow, and Project Gutenberg onto boats, into classrooms with no internet, and onto a phone for a long flight. Because the format is a documented standard and not a kage invention, a `paulgraham.com.zim` you make today will still open in any ZIM reader years from now.
|
||||
|
||||
So you are not locked into kage. `kage open` is the quickest way back in, but the very same file works across the wider Kiwix ecosystem:
|
||||
|
||||
```bash
|
||||
kage open paulgraham.com.zim # read it back with kage
|
||||
kiwix-serve paulgraham.com.zim # or serve it with Kiwix at http://localhost
|
||||
```
|
||||
|
||||
You can also double-click the file in the [Kiwix desktop app](https://kiwix.org/en/applications/), or load it on [Kiwix for Android or iOS](https://kiwix.org/en/applications/) to read your mirror on your phone. One caveat: kage writes a structurally valid archive with the standard metadata, but it does not build the full-text search index that Kiwix's own packs ship with, so browsing and clicking work everywhere while in-reader search is limited.
|
||||
|
||||
Packing is deterministic. The same mirror always produces a byte-identical file, with the archive UUID derived from the content instead of randomized, so a pack is safe to checksum and cache. A bare host name resolves against the default output directory, which is why `kage pack paulgraham.com` just works right after `kage clone paulgraham.com`.
|
||||
|
||||
### A self-contained binary
|
||||
|
||||
`--format binary` glues the archive onto a copy of kage and hands you a single executable that serves the site offline when you run it. Whoever you send it to needs nothing installed: not kage, not a ZIM reader, nothing.
|
||||
|
||||
```bash
|
||||
kage pack paulgraham.com --format binary -o paulgraham
|
||||
./paulgraham
|
||||
```
|
||||
|
||||
The appended archive is platform-independent; only the base executable carries the architecture. By default kage appends to itself, so you get a viewer for the machine you ran it on. Point `--base` at a kage built for another OS to produce a viewer for that platform from your own machine:
|
||||
|
||||
```bash
|
||||
# Sitting on a Mac, build a Windows viewer
|
||||
kage pack paulgraham.com --format binary --base kage-windows-amd64.exe # -> paulgraham.exe
|
||||
```
|
||||
|
||||
The trade is size. The binary carries a whole kage, so it weighs around 13 MiB plus the site no matter how small the mirror is. When you only need the content, the ZIM is far leaner.
|
||||
|
||||
## A real window, not a browser tab
|
||||
|
||||
By default a packed binary opens your system browser, which means the site shows up as yet another tab, address bar and all, next to the 47 you already have open. Build kage with the `webview` tag and it opens the site in its own window instead, backed by the operating system's WebView (WKWebView on macOS, WebView2 on Windows, WebKitGTK on Linux). Paul Graham's essays, offline, in something that looks and feels like a real app:
|
||||
|
||||

|
||||
|
||||
```bash
|
||||
make build-webview # or: CGO_ENABLED=1 go build -tags webview ./cmd/kage
|
||||
kage pack paulgraham.com --format binary --base bin/kage -o paulgraham
|
||||
./paulgraham # opens a window, no browser in sight
|
||||
```
|
||||
|
||||
This build needs cgo and links the platform WebView, so it stays opt-in. The default build is pure Go (`CGO_ENABLED=0`) and the prebuilt release binaries open the browser, which keeps the cross-compiled release simple. `kage open` honours the same tag, so built with `-tags webview` it shows a ZIM in a native window too.
|
||||
|
||||
## How it works
|
||||
|
||||
```
|
||||
@@ -118,33 +179,59 @@ seed URL ─▶ headless Chrome ─▶ final DOM ─▶ strip JS ─▶ localise
|
||||
(render) (snapshot) (sanitize) (rewrite links)
|
||||
```
|
||||
|
||||
A clone is a polite breadth-first crawl. Pages are rendered by a pool of Chrome
|
||||
tabs; assets are fetched over plain HTTP by a separate worker pool. Every URL
|
||||
maps deterministically to a local path, so links can be rewritten before the
|
||||
asset they point at has even finished downloading. The crawl honours
|
||||
`robots.txt` and seeds itself from `sitemap.xml` by default. Output layout:
|
||||
A pool of Chrome tabs renders pages; a separate pool fetches assets over plain HTTP. Every URL maps deterministically to a local path, so links get rewritten before the asset they point at has even finished downloading. The output looks like this:
|
||||
|
||||
```
|
||||
kage-out/example.com/
|
||||
├── index.html # the home page, scripts stripped
|
||||
├── about/index.html # /about
|
||||
paulgraham.com/
|
||||
├── index.html # the home page, scripts stripped
|
||||
├── greatwork.html # /greatwork.html, an essay
|
||||
├── _kage/ # reserved: assets and crawl state
|
||||
│ ├── example.com/site.css # localised stylesheet (url() rewritten)
|
||||
│ ├── example.com/logo.png
|
||||
│ └── state.json # visited set, for --resume
|
||||
│ ├── paulgraham.com/site.css # localised stylesheet (url() rewritten)
|
||||
│ ├── paulgraham.com/pg.png
|
||||
│ └── state.json # visited set, for resuming
|
||||
└── ...
|
||||
```
|
||||
|
||||
`pack` rides on the same idea: the mirror's links are already mirror-relative paths, and those map one-to-one onto the archive's content entries, so a click in a served page hits the right entry with no rewriting at all.
|
||||
|
||||
## Building from source
|
||||
|
||||
```bash
|
||||
git clone https://github.com/tamnd/kage
|
||||
cd kage
|
||||
make build # -> bin/kage
|
||||
make test # full suite, including Chrome-driven end-to-end tests
|
||||
make build # -> bin/kage (pure Go, opens the browser)
|
||||
make build-webview # -> bin/kage with the native-window viewer (needs cgo)
|
||||
make test # full suite, including the Chrome-driven end-to-end tests
|
||||
make test-short # skip the tests that launch a browser
|
||||
```
|
||||
|
||||
The repo is split by concern:
|
||||
|
||||
```
|
||||
cmd/kage/ thin main: pins the main thread, then hands off to cli.Execute
|
||||
cli/ the cobra command tree and flag wiring
|
||||
clone/ the crawl: frontier, render workers, asset workers, resume state
|
||||
browser/ headless Chrome control and DOM snapshotting
|
||||
sanitize/ strip scripts, handlers, and javascript: URLs from the DOM
|
||||
asset/ download and localise CSS, images, and fonts
|
||||
urlx/ the deterministic URL-to-path mapping
|
||||
zim/ a pure-Go ZIM reader and writer
|
||||
pack/ mirror to ZIM or self-contained binary, and the offline HTTP handler
|
||||
viewer/ present a served site: system browser, or native window (webview tag)
|
||||
docs/ the tago documentation site
|
||||
```
|
||||
|
||||
## Releasing
|
||||
|
||||
Push a version tag and GitHub Actions runs GoReleaser, which builds the archives, the `.deb`/`.rpm`/`.apk` packages, a multi-arch GHCR image with Chromium bundled, checksums, SBOMs, and a cosign signature:
|
||||
|
||||
```bash
|
||||
git tag v0.1.1
|
||||
git push --tags
|
||||
```
|
||||
|
||||
The image tag carries no `v` prefix (`ghcr.io/tamnd/kage:0.1.1`). The Homebrew and Scoop steps self-disable until their tokens exist, so the first release works with no extra secrets.
|
||||
|
||||
## License
|
||||
|
||||
MIT. See [LICENSE](LICENSE).
|
||||
|
||||
+66
@@ -0,0 +1,66 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"github.com/tamnd/kage/pack"
|
||||
"github.com/tamnd/kage/viewer"
|
||||
"github.com/tamnd/kage/zim"
|
||||
)
|
||||
|
||||
func newOpenCmd() *cobra.Command {
|
||||
var addr string
|
||||
var openBrowser bool
|
||||
cmd := &cobra.Command{
|
||||
Use: "open <file.zim>",
|
||||
Short: "Serve a ZIM archive in your browser for offline reading",
|
||||
Long: "open serves a packed ZIM file over a local HTTP server so you can browse the\n" +
|
||||
"site exactly as it was cloned. It is the read side of kage pack --format zim.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
return runOpen(cmd.Context(), args[0], addr, openBrowser)
|
||||
},
|
||||
}
|
||||
cmd.Flags().StringVarP(&addr, "addr", "a", "127.0.0.1:8800", "address to listen on")
|
||||
cmd.Flags().BoolVar(&openBrowser, "open", true, "open the default browser")
|
||||
return cmd
|
||||
}
|
||||
|
||||
func runOpen(ctx context.Context, path, addr string, openBrowser bool) error {
|
||||
r, err := zim.Open(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("cannot open %q: %w", path, err)
|
||||
}
|
||||
defer func() { _ = r.Close() }()
|
||||
|
||||
ln, err := net.Listen("tcp", addr)
|
||||
if err != nil {
|
||||
return fmt.Errorf("cannot listen on %s: %w", addr, err)
|
||||
}
|
||||
url := "http://" + ln.Addr().String()
|
||||
|
||||
fmt.Fprintln(os.Stderr, styleTitle.Render("kage open")+" "+styleDim.Render(path))
|
||||
fmt.Fprintln(os.Stderr, " open "+styleAccent.Render(url))
|
||||
if viewer.Native {
|
||||
fmt.Fprintln(os.Stderr, styleDim.Render(" close the window to stop"))
|
||||
} else {
|
||||
fmt.Fprintln(os.Stderr, styleDim.Render(" press Ctrl-C to stop"))
|
||||
}
|
||||
|
||||
srv := &http.Server{Handler: pack.Handler(r)}
|
||||
srvErr := make(chan error, 1)
|
||||
go func() { srvErr <- srv.Serve(ln) }()
|
||||
|
||||
_ = viewer.Show(ctx, viewer.Options{Title: archiveTitle(r), URL: url, Browser: openBrowser})
|
||||
_ = srv.Close()
|
||||
if err := <-srvErr; err != nil && err != http.ErrServerClosed {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
+168
@@ -0,0 +1,168 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"github.com/tamnd/kage/clone"
|
||||
"github.com/tamnd/kage/pack"
|
||||
)
|
||||
|
||||
// packFlags holds the parsed flags for one invocation of kage pack.
|
||||
type packFlags struct {
|
||||
format string
|
||||
out string
|
||||
base string
|
||||
noCompress bool
|
||||
title string
|
||||
description string
|
||||
language string
|
||||
date string
|
||||
}
|
||||
|
||||
func newPackCmd() *cobra.Command {
|
||||
f := &packFlags{}
|
||||
cmd := &cobra.Command{
|
||||
Use: "pack <mirror-dir>",
|
||||
Short: "Pack a cloned mirror into a ZIM file or a self-contained viewer",
|
||||
Long: "pack turns a cloned folder into one distributable file. With --format zim\n" +
|
||||
"it writes an open ZIM archive (the format Kiwix uses) that kage open or any\n" +
|
||||
"ZIM reader can browse. With --format binary it appends that archive to a copy\n" +
|
||||
"of kage, producing a single executable that serves the site offline when run.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
return runPack(args[0], f)
|
||||
},
|
||||
}
|
||||
fs := cmd.Flags()
|
||||
fs.StringVar(&f.format, "format", "zim", "output format: zim or binary")
|
||||
fs.StringVarP(&f.out, "out", "o", "", "output path (default per format)")
|
||||
fs.StringVar(&f.base, "base", "", "base kage binary for --format binary (default this kage)")
|
||||
fs.BoolVar(&f.noCompress, "no-compress", false, "store every cluster raw, no zstd")
|
||||
fs.StringVar(&f.title, "title", "", "archive title (default the main page's <title>)")
|
||||
fs.StringVar(&f.description, "description", "", "archive description")
|
||||
fs.StringVar(&f.language, "language", "eng", "archive language code")
|
||||
fs.StringVar(&f.date, "date", time.Now().UTC().Format("2006-01-02"), "archive date (YYYY-MM-DD)")
|
||||
return cmd
|
||||
}
|
||||
|
||||
func runPack(mirrorArg string, f *packFlags) error {
|
||||
dir := resolveMirror(mirrorArg)
|
||||
zopts := pack.ZIMOptions{
|
||||
Out: f.out,
|
||||
NoCompress: f.noCompress,
|
||||
Title: f.title,
|
||||
Description: f.description,
|
||||
Language: f.language,
|
||||
Date: f.date,
|
||||
Version: Version,
|
||||
}
|
||||
|
||||
switch f.format {
|
||||
case "zim":
|
||||
out, size, err := pack.BuildZIM(dir, zopts)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
printPackResult(out, size)
|
||||
fmt.Fprintf(os.Stderr, " open %s\n", styleAccent.Render("kage open "+out))
|
||||
return nil
|
||||
|
||||
case "binary":
|
||||
zbytes, err := pack.BuildZIMBytes(dir, zopts)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
out := f.out
|
||||
if out == "" {
|
||||
out = defaultBinaryName(dir, f.base)
|
||||
}
|
||||
path, size, err := pack.BuildBinary(zbytes, pack.BinaryOptions{Out: out, Base: f.base})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
printPackResult(path, size)
|
||||
printRunHint(path)
|
||||
return nil
|
||||
|
||||
default:
|
||||
return fmt.Errorf("unknown --format %q (want zim or binary)", f.format)
|
||||
}
|
||||
}
|
||||
|
||||
// resolveMirror accepts either a path to a mirror dir or a bare host. A bare
|
||||
// host that is not a directory in the working dir is resolved against the
|
||||
// default out dir, so "kage pack paulgraham.com" works right after a clone.
|
||||
func resolveMirror(arg string) string {
|
||||
if info, err := os.Stat(arg); err == nil && info.IsDir() {
|
||||
return arg
|
||||
}
|
||||
candidate := filepath.Join(clone.DefaultOutDir(), arg)
|
||||
if info, err := os.Stat(candidate); err == nil && info.IsDir() {
|
||||
return candidate
|
||||
}
|
||||
return arg
|
||||
}
|
||||
|
||||
// defaultBinaryName derives a clean program name from the mirror's host: strip a
|
||||
// trailing dot-suffix (paulgraham.com -> paulgraham), and append .exe when the
|
||||
// target is Windows. The target is the running OS unless --base names a binary
|
||||
// that looks like it was built for Windows.
|
||||
func defaultBinaryName(dir, base string) string {
|
||||
host := filepath.Base(dir)
|
||||
name := host
|
||||
if i := strings.IndexByte(host, '.'); i > 0 {
|
||||
name = host[:i]
|
||||
}
|
||||
if windowsTarget(base) {
|
||||
name += ".exe"
|
||||
}
|
||||
return name
|
||||
}
|
||||
|
||||
func windowsTarget(base string) bool {
|
||||
if base == "" {
|
||||
return runtime.GOOS == "windows"
|
||||
}
|
||||
return strings.HasSuffix(strings.ToLower(base), ".exe")
|
||||
}
|
||||
|
||||
func printPackResult(path string, size int64) {
|
||||
fmt.Fprintln(os.Stderr, styleOK.Render("packed")+" "+styleTitle.Render(path))
|
||||
fmt.Fprintf(os.Stderr, " %s %s\n", styleAccent.Render("size"), humanBytes(size))
|
||||
}
|
||||
|
||||
func printRunHint(path string) {
|
||||
rel := path
|
||||
if !strings.ContainsAny(path, "/\\") {
|
||||
rel = "./" + path
|
||||
}
|
||||
if windowsTarget(path) && runtime.GOOS != "windows" {
|
||||
fmt.Fprintf(os.Stderr, " this is a Windows viewer; run %s on Windows\n", styleAccent.Render(filepath.Base(path)))
|
||||
return
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, " run %s to view the site offline\n", styleAccent.Render(rel))
|
||||
if runtime.GOOS == "darwin" {
|
||||
fmt.Fprintln(os.Stderr, styleDim.Render(" (macOS may quarantine it: xattr -d com.apple.quarantine "+rel+")"))
|
||||
}
|
||||
}
|
||||
|
||||
// humanBytes renders a byte count in B, KiB, MiB, or GiB.
|
||||
func humanBytes(n int64) string {
|
||||
const unit = 1024
|
||||
if n < unit {
|
||||
return fmt.Sprintf("%d B", n)
|
||||
}
|
||||
div, exp := int64(unit), 0
|
||||
for x := n / unit; x >= unit; x /= unit {
|
||||
div *= unit
|
||||
exp++
|
||||
}
|
||||
return fmt.Sprintf("%.1f %ciB", float64(n)/float64(div), "KMGTPE"[exp])
|
||||
}
|
||||
+63
@@ -7,15 +7,29 @@ package cli
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
|
||||
"github.com/charmbracelet/fang"
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"github.com/tamnd/kage/pack"
|
||||
"github.com/tamnd/kage/viewer"
|
||||
"github.com/tamnd/kage/zim"
|
||||
)
|
||||
|
||||
// Execute builds the root command and runs it through fang. main passes the
|
||||
// signal-aware context so Ctrl-C cancels the in-flight clone and flushes resume
|
||||
// state. It returns the process exit code.
|
||||
func Execute(ctx context.Context) int {
|
||||
// A kage binary with a ZIM appended runs as an offline viewer for that site,
|
||||
// ignoring its arguments. A normal build has no trailer and falls through.
|
||||
if ra, size, ok := pack.Embedded(); ok {
|
||||
return runEmbeddedViewer(ctx, ra, size)
|
||||
}
|
||||
|
||||
root := newRoot()
|
||||
opts := []fang.Option{
|
||||
fang.WithVersion(Version),
|
||||
@@ -41,5 +55,54 @@ func newRoot() *cobra.Command {
|
||||
}
|
||||
root.AddCommand(newCloneCmd())
|
||||
root.AddCommand(newServeCmd())
|
||||
root.AddCommand(newPackCmd())
|
||||
root.AddCommand(newOpenCmd())
|
||||
return root
|
||||
}
|
||||
|
||||
// runEmbeddedViewer serves the ZIM appended to this executable on an ephemeral
|
||||
// local port and shows it: a native window in the webview build, the system
|
||||
// browser otherwise. It runs until the viewer closes or the context is
|
||||
// cancelled (Ctrl-C) and ignores all command-line arguments, because a packed
|
||||
// binary is the site, not the kage CLI.
|
||||
func runEmbeddedViewer(ctx context.Context, ra io.ReaderAt, size int64) int {
|
||||
r, err := zim.NewReader(ra, size)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "kage: corrupt embedded archive:", err)
|
||||
return 1
|
||||
}
|
||||
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "kage: cannot start viewer:", err)
|
||||
return 1
|
||||
}
|
||||
url := "http://" + ln.Addr().String()
|
||||
if viewer.Native {
|
||||
fmt.Fprintln(os.Stderr, "opening offline site (close the window to stop)")
|
||||
} else {
|
||||
fmt.Fprintln(os.Stderr, "serving offline site at "+url+" (Ctrl-C to stop)")
|
||||
}
|
||||
|
||||
srv := &http.Server{Handler: pack.Handler(r)}
|
||||
srvErr := make(chan error, 1)
|
||||
go func() { srvErr <- srv.Serve(ln) }()
|
||||
|
||||
// Show blocks until the window closes (native) or ctx is cancelled (browser);
|
||||
// either way, tear the server down afterwards.
|
||||
_ = viewer.Show(ctx, viewer.Options{Title: archiveTitle(r), URL: url, Browser: true})
|
||||
_ = srv.Close()
|
||||
if err := <-srvErr; err != nil && err != http.ErrServerClosed {
|
||||
fmt.Fprintln(os.Stderr, "kage:", err)
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// archiveTitle returns the archive's M/Title metadata for use as a window
|
||||
// title, falling back to the empty string (viewer defaults it to "kage").
|
||||
func archiveTitle(r *zim.Reader) string {
|
||||
if b, err := r.Get(zim.NamespaceMetadata, "Title"); err == nil {
|
||||
return string(b.Data)
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
@@ -9,9 +9,15 @@ import (
|
||||
"os/signal"
|
||||
|
||||
"github.com/tamnd/kage/cli"
|
||||
"github.com/tamnd/kage/viewer"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// Pin the main goroutine to the process's initial OS thread before anything
|
||||
// else. In the webview build the native window must be driven from that
|
||||
// thread; in the default build this is a harmless no-op.
|
||||
viewer.LockMainThread()
|
||||
|
||||
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
|
||||
defer stop()
|
||||
os.Exit(cli.Execute(ctx))
|
||||
|
||||
+18
-21
@@ -7,35 +7,32 @@ heroPrimaryURL: "/getting-started/quick-start/"
|
||||
heroPrimaryText: "Get started"
|
||||
---
|
||||
|
||||
Saving a page with "Save As" gives you a copy that still phones home, still runs
|
||||
analytics, and often renders blank because the markup is built by JavaScript at
|
||||
runtime. kage (影, "shadow") takes the opposite approach: it drives a real
|
||||
browser, captures the page the way a human would have seen it, then makes it
|
||||
inert.
|
||||
Saving a page with "Save As" gives you a copy that still phones home, still runs analytics, and often renders blank because the markup is built by JavaScript at runtime. kage (影, "shadow") takes the opposite approach: it drives a real browser, captures the page the way a human would have seen it, then makes it inert.
|
||||
|
||||
Say you want Paul Graham's essays on a laptop with no wifi. One command mirrors the site; a second serves it back offline:
|
||||
|
||||
```bash
|
||||
kage clone example.com
|
||||
kage serve kage-out/example.com
|
||||
kage clone paulgraham.com
|
||||
kage serve $HOME/data/kage/paulgraham.com
|
||||
```
|
||||
|
||||

|
||||
|
||||
## What it does
|
||||
|
||||
- **Renders first, saves second.** Each page goes through real headless Chrome,
|
||||
so a page whose content is assembled by JavaScript is captured fully, not as
|
||||
an empty shell.
|
||||
- **Strips every script.** Once the DOM is captured, kage removes all `<script>`
|
||||
tags, every `on*` event handler, and any `javascript:` URL. The saved page
|
||||
makes no network calls and runs no code.
|
||||
- **Keeps the layout.** Stylesheets, images, fonts, and media are downloaded and
|
||||
rewritten to relative local paths, so the offline copy looks like the original.
|
||||
- **Stays browsable.** In-scope links are rewritten to point at the other saved
|
||||
pages, so you can click around the mirror exactly as you would the live site.
|
||||
- **Renders first, saves second.** Each page goes through real headless Chrome, so a page whose content is assembled by JavaScript is captured fully, not as an empty shell.
|
||||
- **Strips every script.** Once the DOM is captured, kage removes all `<script>` tags, every `on*` event handler, and any `javascript:` URL. The saved page makes no network calls and runs no code.
|
||||
- **Keeps the layout.** Stylesheets, images, fonts, and media are downloaded and rewritten to relative local paths, so the offline copy looks like the original.
|
||||
- **Stays browsable.** In-scope links are rewritten to point at the other saved pages, so you can click around the mirror exactly as you would the live site.
|
||||
- **Packs into one file.** Collapse a mirror into a single [ZIM archive](/guides/packing-a-mirror/), the open format Kiwix uses, or a self-contained binary that serves the site when run.
|
||||
|
||||
Build kage with the `webview` tag and a packed binary opens in its own window instead of a browser tab, so an offline mirror feels like a real app:
|
||||
|
||||

|
||||
|
||||
## Where to go next
|
||||
|
||||
- New here? Start with the [introduction](/getting-started/introduction/), then
|
||||
the [quick start](/getting-started/quick-start/).
|
||||
- New here? Start with the [introduction](/getting-started/introduction/), then the [quick start](/getting-started/quick-start/).
|
||||
- Want to install it? See [installation](/getting-started/installation/).
|
||||
- Looking for a specific task? The [guides](/guides/) cover scoping a crawl,
|
||||
serving a mirror, and resuming an interrupted run.
|
||||
- Looking for a specific task? The [guides](/guides/) cover scoping a crawl, serving a mirror, resuming an interrupted run, and [packing a mirror](/guides/packing-a-mirror/) into one file or a self-contained viewer.
|
||||
- Need every flag? The [CLI reference](/reference/cli/) is the full surface.
|
||||
|
||||
@@ -4,21 +4,20 @@ description: "Why kage renders before it saves, and what it means to strip the J
|
||||
weight: 10
|
||||
---
|
||||
|
||||
A normal website is not a document; it is a program. The HTML the server sends
|
||||
is often a near-empty shell, and the page you actually see is assembled in your
|
||||
browser by JavaScript: fetching data, building the DOM, wiring up handlers. That
|
||||
is why "Save As" so often fails. You get the shell, not the page, and whatever
|
||||
you do get still runs trackers and phones home when you open it.
|
||||
A normal website is not a document; it is a program. The HTML the server sends is often a near-empty shell, and the page you actually see is assembled in your browser by JavaScript: fetching data, building the DOM, wiring up handlers. That is why "Save As" so often fails. You get the shell, not the page, and whatever you do get still runs trackers and phones home when you open it.
|
||||
|
||||
Say you want to keep Paul Graham's essays. Hand the site to "Save As" and you get a brittle copy that may still call out to scripts that no longer exist. Hand it to kage and you get the essays as they look in a browser, frozen and inert:
|
||||
|
||||
```bash
|
||||
kage clone paulgraham.com
|
||||
kage serve $HOME/data/kage/paulgraham.com
|
||||
```
|
||||
|
||||
kage treats a clone as three steps in order.
|
||||
|
||||
## 1. Render
|
||||
|
||||
Every page is loaded in a real headless Chrome through the DevTools protocol.
|
||||
kage navigates to the URL, waits for the network to go quiet, optionally scrolls
|
||||
to trigger lazy-loaded images, and then serialises the **final** DOM, the markup
|
||||
that exists after the page's JavaScript has finished building it. This is the
|
||||
same thing you would see if you opened the page and chose "Inspect".
|
||||
Every page is loaded in a real headless Chrome through the DevTools protocol. kage navigates to the URL, waits for the network to go quiet, optionally scrolls to trigger lazy-loaded images, and then serialises the **final** DOM, the markup that exists after the page's JavaScript has finished building it. This is the same thing you would see if you opened the page and chose "Inspect".
|
||||
|
||||
## 2. Strip
|
||||
|
||||
@@ -27,27 +26,22 @@ From that captured DOM, kage removes everything executable:
|
||||
- every `<script>` tag, inline or external;
|
||||
- every `on*` event handler attribute (`onclick`, `onload`, and the rest);
|
||||
- every `javascript:` URL;
|
||||
- `<meta http-equiv="refresh">` redirects and dead resource hints like
|
||||
`<link rel="preload" as="script">`.
|
||||
- `<meta http-equiv="refresh">` redirects and dead resource hints like `<link rel="preload" as="script">`.
|
||||
|
||||
What remains is inert. The saved page makes no network calls, runs no code, and
|
||||
tracks nothing.
|
||||
What remains is inert. The saved page makes no network calls, runs no code, and tracks nothing.
|
||||
|
||||
## 3. Localise
|
||||
|
||||
A page with no working CSS or images is not much of a clone, so kage keeps the
|
||||
parts that define how it looks. It downloads every stylesheet, image, font, and
|
||||
media file, rewrites the references in the HTML and inside the CSS (`url()` and
|
||||
`@import`) to relative local paths, and rewrites in-scope page links to point at
|
||||
the other saved pages. The mirror is fully self-contained: you can move the
|
||||
folder anywhere, open it with no network, and click around.
|
||||
A page with no working CSS or images is not much of a clone, so kage keeps the parts that define how it looks. It downloads every stylesheet, image, font, and media file, rewrites the references in the HTML and inside the CSS (`url()` and `@import`) to relative local paths, and rewrites in-scope page links to point at the other saved pages. The mirror is fully self-contained: you can move the folder anywhere, open it with no network, and click around.
|
||||
|
||||
## The shape of a clone
|
||||
|
||||
kage crawls breadth-first from a seed URL, staying within the seed's host (and
|
||||
optionally its subdomains). It is polite by default: it honours `robots.txt` and
|
||||
seeds itself from `sitemap.xml`. Output lands in `kage-out/<host>/`, with pages
|
||||
as `<path>/index.html` and assets under a reserved `_kage/` directory alongside
|
||||
the crawl state that powers `--resume`.
|
||||
kage crawls breadth-first from a seed URL, staying within the seed's host (and optionally its subdomains). It is polite by default: it honours `robots.txt` and seeds itself from `sitemap.xml`. Output lands in `$HOME/data/kage/paulgraham.com/`, with pages as `<path>/index.html` and assets under a reserved `_kage/` directory alongside the crawl state that powers resuming.
|
||||
|
||||
## Then what?
|
||||
|
||||
A folder is the starting point, not the end. Once you have a mirror you can [pack it](/guides/packing-a-mirror/) into a single ZIM file, the open offline-archive format Kiwix uses, so the whole site travels as one file that any ZIM reader can open. Or build kage with the `webview` tag and a packed binary opens the site in its own native window instead of a browser tab:
|
||||
|
||||

|
||||
|
||||
Next: [install kage](/getting-started/installation/).
|
||||
|
||||
@@ -19,15 +19,15 @@ errors as it goes; the final summary tells you where the mirror landed.
|
||||
|
||||
```
|
||||
kage cloning https://example.com
|
||||
done kage-out/example.com
|
||||
done $HOME/data/kage/example.com
|
||||
pages 12 assets 38
|
||||
open kage serve kage-out/example.com
|
||||
open kage serve $HOME/data/kage/example.com
|
||||
```
|
||||
|
||||
## 2. Look at what landed
|
||||
|
||||
```bash
|
||||
ls kage-out/example.com
|
||||
ls $HOME/data/kage/example.com
|
||||
```
|
||||
|
||||
```
|
||||
@@ -46,7 +46,7 @@ serve` runs a local static server so everything resolves exactly as it would
|
||||
live:
|
||||
|
||||
```bash
|
||||
kage serve kage-out/example.com
|
||||
kage serve $HOME/data/kage/example.com
|
||||
# open http://127.0.0.1:8800
|
||||
```
|
||||
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
---
|
||||
title: "Packing a mirror"
|
||||
description: "Turn a cloned folder into one ZIM file or a self-contained offline viewer with kage pack."
|
||||
weight: 30
|
||||
---
|
||||
|
||||
A clone is a folder of files, which is easy to browse but awkward to move around: copying thousands of small files is slow, and handing someone a directory is less tidy than handing them one file. `kage pack` collapses a mirror into a single distributable artifact, and you choose the shape: an open ZIM archive, or a self-contained executable that serves the site offline when run.
|
||||
|
||||
The two examples below assume you have already cloned a site, for instance Paul Graham's essays:
|
||||
|
||||
```bash
|
||||
kage clone paulgraham.com
|
||||
```
|
||||
|
||||
## A single ZIM file
|
||||
|
||||
ZIM is the open, single-file offline-archive format Kiwix uses. `kage pack` writes one from a cloned host directory:
|
||||
|
||||
```bash
|
||||
kage pack paulgraham.com
|
||||
```
|
||||
|
||||
```
|
||||
packed paulgraham.com.zim
|
||||
size 4.2 MiB
|
||||
open kage open paulgraham.com.zim
|
||||
```
|
||||
|
||||
The whole mirror, pages and assets, lives in that one file. Text is zstd compressed; already-compressed media (images, fonts, video) is stored as-is. Packing the same mirror twice produces a byte-identical file, so a ZIM is safe to checksum, diff, and cache.
|
||||
|
||||
If you cloned with the default output directory, you can pass a bare host name and kage finds the mirror for you. That is why `kage pack paulgraham.com` works straight after `kage clone paulgraham.com`; pass a full path if your mirror lives somewhere else.
|
||||
|
||||
### What ZIM is, and using it with Kiwix
|
||||
|
||||
ZIM is built for exactly this job: a whole website (or a whole Wikipedia) squeezed into one compressed, indexed, read-only file. It is the format behind [Kiwix](https://kiwix.org), the offline-content project people use to carry Wikipedia, Stack Overflow, and Project Gutenberg onto boats, into classrooms with no internet, and onto a phone for a long flight. Because the format is a documented standard rather than a kage invention, a `paulgraham.com.zim` you make today still opens in any ZIM reader years from now.
|
||||
|
||||
So you are not locked into kage. `kage open` is the read side and the quickest way back in, but the same file works across the wider Kiwix ecosystem:
|
||||
|
||||
```bash
|
||||
kage open paulgraham.com.zim # read it back with kage
|
||||
kiwix-serve paulgraham.com.zim # or serve it with Kiwix at http://localhost
|
||||
```
|
||||
|
||||
You can also double-click the file in the [Kiwix desktop app](https://kiwix.org/en/applications/), or load it on Kiwix for Android or iOS to read your mirror on your phone. One caveat: kage writes a structurally valid archive with the standard metadata, but it does not write the full-text search index that Kiwix's own packs ship with, so browsing works everywhere while in-reader search is limited.
|
||||
|
||||
## A self-contained binary
|
||||
|
||||
`--format binary` appends the ZIM to a copy of kage, producing one executable that *is* the site. Run it and it serves the mirror on a free local port and opens your browser; it ignores its arguments, because the binary is the site, not the kage CLI.
|
||||
|
||||
```bash
|
||||
kage pack paulgraham.com --format binary -o paulgraham
|
||||
```
|
||||
|
||||
```
|
||||
packed paulgraham
|
||||
size 21.9 MiB
|
||||
run ./paulgraham to view the site offline
|
||||
```
|
||||
|
||||
```bash
|
||||
./paulgraham
|
||||
```
|
||||
|
||||
```
|
||||
serving offline site at http://127.0.0.1:52431 (Ctrl-C to stop)
|
||||
```
|
||||
|
||||
The binary carries a full kage, so it is tens of megabytes regardless of site size; the trade is that the recipient needs nothing installed, not even kage, not even a ZIM reader.
|
||||
|
||||
### A native window instead of a browser
|
||||
|
||||
By default the viewer opens the system browser, which means a tab with an address bar and your other tabs alongside. Build kage with the `webview` tag and it opens the site in its own native window instead, backed by the operating system's WebView (WKWebView on macOS, WebView2 on Windows, WebKitGTK on Linux), so a packed binary feels like a standalone app:
|
||||
|
||||
```bash
|
||||
CGO_ENABLED=1 go build -tags webview -o kage ./cmd/kage
|
||||
kage pack paulgraham.com --format binary --base kage -o paulgraham
|
||||
./paulgraham # opens a window, no browser
|
||||
```
|
||||
|
||||

|
||||
|
||||
The window title comes from the archive's title. This build needs cgo and links the platform WebView, so it is opt-in and kept out of the default `CGO_ENABLED=0` release; the prebuilt binaries open the browser. `kage open` honours the same tag: built with `-tags webview` it shows the ZIM in a native window too.
|
||||
|
||||
### Build a viewer for another platform
|
||||
|
||||
The appended archive is platform-independent; only the base executable carries the architecture. Point `--base` at a kage binary built for another OS (download one from a kage release) to produce a viewer for that platform from your own machine:
|
||||
|
||||
```bash
|
||||
# From macOS, build a Windows viewer
|
||||
kage pack paulgraham.com --format binary --base kage-windows-amd64.exe
|
||||
# -> paulgraham.exe
|
||||
```
|
||||
|
||||
### macOS note
|
||||
|
||||
A binary you built or downloaded may be quarantined by Gatekeeper on first run. kage prints the exact command to clear it:
|
||||
|
||||
```bash
|
||||
xattr -d com.apple.quarantine ./paulgraham
|
||||
```
|
||||
|
||||
## Metadata and options
|
||||
|
||||
```bash
|
||||
kage pack paulgraham.com \
|
||||
--title "Paul Graham, offline" \
|
||||
--description "A snapshot taken for archival" \
|
||||
--language eng \
|
||||
--date 2026-06-14
|
||||
```
|
||||
|
||||
`--title` defaults to the main page's `<title>`, then the host name. `--date` defaults to today; pass a fixed value for a fully reproducible file. `--no-compress` stores every cluster raw, which packs fastest and lets a reader without zstd open the result. `-o/--out` overrides the output path for either format.
|
||||
@@ -37,7 +37,7 @@ existing host folder first with `--force`:
|
||||
kage clone example.com --force
|
||||
```
|
||||
|
||||
This removes `kage-out/example.com/` before crawling, so nothing from a prior run
|
||||
This removes `$HOME/data/kage/example.com/` before crawling, so nothing from a prior run
|
||||
carries over.
|
||||
|
||||
To run without reading or writing any resume state at all, for a strictly
|
||||
|
||||
@@ -12,11 +12,11 @@ when served from the root of a host. `kage serve` gives you that root.
|
||||
## Serve a clone
|
||||
|
||||
```bash
|
||||
kage serve kage-out/example.com
|
||||
kage serve $HOME/data/kage/example.com
|
||||
```
|
||||
|
||||
```
|
||||
kage serve /…/kage-out/example.com
|
||||
kage serve $HOME/data/kage/example.com
|
||||
open http://127.0.0.1:8800
|
||||
press Ctrl-C to stop
|
||||
```
|
||||
@@ -31,10 +31,10 @@ By default kage serves on `127.0.0.1:8800`. Change it with `--addr`:
|
||||
|
||||
```bash
|
||||
# A different port
|
||||
kage serve kage-out/example.com --addr 127.0.0.1:9000
|
||||
kage serve $HOME/data/kage/example.com --addr 127.0.0.1:9000
|
||||
|
||||
# Reachable from other machines on your network (be deliberate about this)
|
||||
kage serve kage-out/example.com --addr 0.0.0.0:8800
|
||||
kage serve $HOME/data/kage/example.com --addr 0.0.0.0:8800
|
||||
```
|
||||
|
||||
## Serve the current directory
|
||||
@@ -43,6 +43,6 @@ With no argument, `kage serve` serves the current directory, which is handy from
|
||||
inside an output folder:
|
||||
|
||||
```bash
|
||||
cd kage-out/example.com
|
||||
cd $HOME/data/kage/example.com
|
||||
kage serve
|
||||
```
|
||||
|
||||
@@ -8,8 +8,9 @@ weight: 10
|
||||
kage [command] [flags]
|
||||
```
|
||||
|
||||
Two commands: `clone` fetches a site into an offline folder, `serve` previews
|
||||
one. Run `kage <command> --help` for the canonical, up-to-date list.
|
||||
Four commands: `clone` fetches a site into an offline folder, `serve` previews
|
||||
one, `pack` collapses a mirror into a single file, and `open` serves a packed
|
||||
file. Run `kage <command> --help` for the canonical, up-to-date list.
|
||||
|
||||
## kage clone
|
||||
|
||||
@@ -84,3 +85,43 @@ current directory.
|
||||
| Flag | Default | Meaning |
|
||||
|------|---------|---------|
|
||||
| `-a, --addr` | `127.0.0.1:8800` | Address to listen on |
|
||||
|
||||
## kage pack
|
||||
|
||||
```
|
||||
kage pack <mirror-dir> [flags]
|
||||
```
|
||||
|
||||
Packs a cloned mirror into one distributable file: an open ZIM archive, or a
|
||||
self-contained executable that serves the site offline when run. A bare host name
|
||||
is resolved against the default output directory, so `kage pack example.com`
|
||||
works right after `kage clone example.com`.
|
||||
|
||||
| Flag | Default | Meaning |
|
||||
|------|---------|---------|
|
||||
| `--format` | `zim` | Output format: `zim` or `binary` |
|
||||
| `-o, --out` | per format | Output path; `<host>.zim` for zim, `<host>` (or `<host>.exe`) for binary |
|
||||
| `--base` | this kage | Base kage binary to append to (`--format binary`); point at another platform's binary to build a viewer for it |
|
||||
| `--no-compress` | `false` | Store every cluster raw, no zstd |
|
||||
| `--title` | main page `<title>` | Archive title |
|
||||
| `--description` | | Archive description |
|
||||
| `--language` | `eng` | Archive language code |
|
||||
| `--date` | today | Archive date (`YYYY-MM-DD`); pass a fixed value for a reproducible file |
|
||||
|
||||
## kage open
|
||||
|
||||
```
|
||||
kage open <file.zim> [flags]
|
||||
```
|
||||
|
||||
Serves a packed ZIM over a local HTTP server for offline reading, the read side
|
||||
of `kage pack --format zim`.
|
||||
|
||||
| Flag | Default | Meaning |
|
||||
|------|---------|---------|
|
||||
| `-a, --addr` | `127.0.0.1:8800` | Address to listen on |
|
||||
| `--open` | `true` | Open the default browser (`--open=false` to skip) |
|
||||
|
||||
Built with `-tags webview` (which needs cgo), `kage open` shows the archive in a
|
||||
native window instead of the browser, and `--open` no longer applies. The default
|
||||
`CGO_ENABLED=0` build uses the browser.
|
||||
|
||||
@@ -24,7 +24,7 @@ A clone of `example.com` lands under `$HOME/data/kage/example.com/` (override th
|
||||
root with `-o/--out`):
|
||||
|
||||
```
|
||||
kage-out/example.com/
|
||||
$HOME/data/kage/example.com/
|
||||
├── index.html # the home page (/), scripts stripped
|
||||
├── about/index.html # /about
|
||||
├── blog/
|
||||
|
||||
@@ -4,24 +4,23 @@ description: "What changed in each kage release."
|
||||
weight: 40
|
||||
---
|
||||
|
||||
The authoritative, commit-level history lives in
|
||||
[`CHANGELOG.md`](https://github.com/tamnd/kage/blob/main/CHANGELOG.md) and on the
|
||||
[releases page](https://github.com/tamnd/kage/releases). This page summarises
|
||||
each version.
|
||||
The authoritative, commit-level history lives in [`CHANGELOG.md`](https://github.com/tamnd/kage/blob/main/CHANGELOG.md) and on the [releases page](https://github.com/tamnd/kage/releases). This page summarises each version.
|
||||
|
||||
## v0.1.1
|
||||
|
||||
Packing, so a clone can travel as one file instead of a folder.
|
||||
|
||||
- **`kage pack <mirror-dir>`** collapses a mirror into a single distributable file. `--format zim` (the default) writes an open ZIM archive, the same format [Kiwix](https://kiwix.org) uses, so the file opens in any ZIM reader and not just kage. `--format binary` appends that archive to a copy of kage to make a self-contained executable that serves the site offline when run. Packing is deterministic, so the same mirror produces a byte-identical file.
|
||||
- **`kage open <file.zim>`** serves a packed ZIM back over a local HTTP server, the read side of `kage pack --format zim`.
|
||||
- **An optional native-window viewer.** Built with `-tags webview`, `kage open` and a packed binary show the site in a real window backed by the operating system's WebView instead of a browser tab. The default build stays pure Go and opens the browser, so the release pipeline is unchanged.
|
||||
- **A pure-Go `zim` package** that reads and writes the ZIM format: a fixed header, MIME and pointer lists, zstd or stored clusters, redirects, and a trailing MD5.
|
||||
|
||||
## v0.1.0
|
||||
|
||||
The first release. kage clones a live website into a self-contained folder you
|
||||
can browse offline, with every script stripped out.
|
||||
The first release. kage clones a live website into a self-contained folder you can browse offline, with every script stripped out.
|
||||
|
||||
- **`kage clone <url>`** renders each page in headless Chrome, strips all
|
||||
JavaScript, and localises CSS, images, and fonts to relative paths.
|
||||
- **`kage clone <url>`** renders each page in headless Chrome, strips all JavaScript, and localises CSS, images, and fonts to relative paths.
|
||||
- **`kage serve [dir]`** previews a cloned folder over a local file server.
|
||||
- **Idempotent and resumable.** Each page is keyed by the file it writes, so a
|
||||
page reached over http and https, or as `/index.html` versus `/`, is fetched
|
||||
once. Re-running resumes; `--refresh` re-renders in place; `--force` starts
|
||||
clean.
|
||||
- **Polite by default.** Honours `robots.txt`, seeds from `sitemap.xml`, scopes
|
||||
to the seed host, and runs three parallel worker tiers.
|
||||
- **Packaged everywhere.** Archives, `.deb`/`.rpm`/`.apk`, a multi-arch GHCR
|
||||
image with Chromium bundled, checksums, SBOMs, and a cosign signature.
|
||||
- **Idempotent and resumable.** Each page is keyed by the file it writes, so a page reached over http and https, or as `/index.html` versus `/`, is fetched once. Re-running resumes; `--refresh` re-renders in place; `--force` starts clean.
|
||||
- **Polite by default.** Honours `robots.txt`, seeds from `sitemap.xml`, scopes to the seed host, and runs three parallel worker tiers.
|
||||
- **Packaged everywhere.** Archives, `.deb`/`.rpm`/`.apk`, a multi-arch GHCR image with Chromium bundled, checksums, SBOMs, and a cosign signature.
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
# Demo tape for kage. Rendered with ascii-gif (github.com/tamnd/ascii-gif),
|
||||
# which supplies the window chrome and theme; this file is just the action.
|
||||
#
|
||||
# ascii-gif render docs/demo/kage.tape -o docs/static/demo.gif
|
||||
#
|
||||
# kage must be on PATH inside the recording shell, and Chrome available.
|
||||
|
||||
Hide
|
||||
Type "export PS1='$ ' PATH=/tmp/kagebin:$PATH && cd $(mktemp -d) && clear"
|
||||
Enter
|
||||
Show
|
||||
|
||||
Sleep 600ms
|
||||
Type "kage clone paulgraham.com --max-pages 3 --out ."
|
||||
Sleep 700ms
|
||||
Enter
|
||||
Sleep 9s
|
||||
|
||||
Type "kage pack ./paulgraham.com"
|
||||
Sleep 700ms
|
||||
Enter
|
||||
Sleep 2.5s
|
||||
|
||||
Type "kage pack ./paulgraham.com --format binary -o paulgraham"
|
||||
Sleep 700ms
|
||||
Enter
|
||||
Sleep 3s
|
||||
|
||||
Type "kage open paulgraham.com.zim --open=false"
|
||||
Sleep 700ms
|
||||
Enter
|
||||
Sleep 3s
|
||||
Ctrl+C
|
||||
Sleep 1.2s
|
||||
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 345 KiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 630 KiB |
@@ -7,7 +7,9 @@ require (
|
||||
github.com/charmbracelet/fang v1.0.0
|
||||
github.com/go-rod/rod v0.116.2
|
||||
github.com/go-rod/stealth v0.4.9
|
||||
github.com/klauspost/compress v1.18.6
|
||||
github.com/spf13/cobra v1.10.2
|
||||
github.com/webview/webview_go v0.0.0-20240831120633-6173450d4dd6
|
||||
golang.org/x/net v0.56.0
|
||||
)
|
||||
|
||||
|
||||
@@ -36,6 +36,8 @@ github.com/go-rod/stealth v0.4.9 h1:X2PmQk4DUF2wzw6GOsWjW/glb8K5ebnftbEvLh7MlZ4=
|
||||
github.com/go-rod/stealth v0.4.9/go.mod h1:eAzyvw8c0iAd5nJJsSWeh0fQ5z94vCIfdi1hUmYDimc=
|
||||
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
|
||||
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
|
||||
github.com/klauspost/compress v1.18.6 h1:2jupLlAwFm95+YDR+NwD2MEfFO9d4z4Prjl1XXDjuao=
|
||||
github.com/klauspost/compress v1.18.6/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
||||
github.com/lucasb-eyer/go-colorful v1.3.0 h1:2/yBRLdWBZKrf7gB40FoiKfAWYQ0lqNcbuQwVHXptag=
|
||||
github.com/lucasb-eyer/go-colorful v1.3.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
|
||||
github.com/mattn/go-runewidth v0.0.19 h1:v++JhqYnZuu5jSKrk9RbgF5v4CGUjqRfBm05byFGLdw=
|
||||
@@ -61,6 +63,8 @@ github.com/spf13/pflag v1.0.9 h1:9exaQaMOCwffKiiiYk6/BndUBv+iRViNW+4lEMi0PvY=
|
||||
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
||||
github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
|
||||
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
|
||||
github.com/webview/webview_go v0.0.0-20240831120633-6173450d4dd6 h1:VQpB2SpK88C6B5lPHTuSZKb2Qee1QWwiFlC5CKY4AW0=
|
||||
github.com/webview/webview_go v0.0.0-20240831120633-6173450d4dd6/go.mod h1:yE65LFCeWf4kyWD5re+h4XNvOHJEXOCOuJZ4v8l5sgk=
|
||||
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavMF/ppJZNG9ZpyihvCd0w101no=
|
||||
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM=
|
||||
github.com/ysmood/fetchup v0.2.3 h1:ulX+SonA0Vma5zUFXtv52Kzip/xe7aj4vqT5AJwQ+ZQ=
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
package pack
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/binary"
|
||||
"fmt"
|
||||
"os"
|
||||
)
|
||||
|
||||
const (
|
||||
// trailerMagic brackets the appended-archive trailer at both ends, so a stray
|
||||
// copy of it inside the base binary cannot be mistaken for a real trailer.
|
||||
trailerMagic = "KAGEPCK1"
|
||||
// trailerLen is magic + uint64 archive length + magic again.
|
||||
trailerLen = len(trailerMagic) + 8 + len(trailerMagic)
|
||||
)
|
||||
|
||||
// BinaryOptions controls how a self-contained viewer is assembled.
|
||||
type BinaryOptions struct {
|
||||
Out string // output path
|
||||
Base string // base kage binary; default os.Executable()
|
||||
}
|
||||
|
||||
// BuildBinary writes baseExe ++ zimBytes ++ trailer to opts.Out and marks it
|
||||
// executable. The base must be a kage binary, since the viewer behaviour lives
|
||||
// in kage's own startup hook (see Embedded); appending a ZIM to an arbitrary
|
||||
// executable would only produce a broken file. It returns the output path and
|
||||
// the total byte size.
|
||||
func BuildBinary(zimBytes []byte, opts BinaryOptions) (string, int64, error) {
|
||||
base := opts.Base
|
||||
if base == "" {
|
||||
exe, err := os.Executable()
|
||||
if err != nil {
|
||||
return "", 0, fmt.Errorf("pack: locate base binary: %w", err)
|
||||
}
|
||||
base = exe
|
||||
}
|
||||
if opts.Out == "" {
|
||||
return "", 0, fmt.Errorf("pack: BuildBinary requires an output path")
|
||||
}
|
||||
|
||||
baseBytes, err := os.ReadFile(base)
|
||||
if err != nil {
|
||||
return "", 0, fmt.Errorf("pack: read base binary %q: %w", base, err)
|
||||
}
|
||||
|
||||
var tr bytes.Buffer
|
||||
tr.WriteString(trailerMagic)
|
||||
_ = binary.Write(&tr, binary.LittleEndian, uint64(len(zimBytes)))
|
||||
tr.WriteString(trailerMagic)
|
||||
|
||||
f, err := os.Create(opts.Out)
|
||||
if err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
for _, chunk := range [][]byte{baseBytes, zimBytes, tr.Bytes()} {
|
||||
if _, err := f.Write(chunk); err != nil {
|
||||
_ = f.Close()
|
||||
return opts.Out, 0, err
|
||||
}
|
||||
}
|
||||
if err := f.Close(); err != nil {
|
||||
return opts.Out, 0, err
|
||||
}
|
||||
if err := os.Chmod(opts.Out, 0o755); err != nil {
|
||||
return opts.Out, 0, err
|
||||
}
|
||||
return opts.Out, int64(len(baseBytes) + len(zimBytes) + trailerLen), nil
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
package pack
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"io"
|
||||
"os"
|
||||
)
|
||||
|
||||
// Embedded inspects the running executable for an appended ZIM archive. If the
|
||||
// KAGEPCK1 trailer is present, it returns a ReaderAt bounded to the archive, its
|
||||
// size, and ok=true; the file handle stays open for the life of the process so
|
||||
// the viewer can serve from it. A normal kage build has no trailer, so the cost
|
||||
// to every ordinary invocation is one Open plus a 24-byte ReadAt.
|
||||
func Embedded() (ra io.ReaderAt, size int64, ok bool) {
|
||||
exe, err := os.Executable()
|
||||
if err != nil {
|
||||
return nil, 0, false
|
||||
}
|
||||
f, err := os.Open(exe)
|
||||
if err != nil {
|
||||
return nil, 0, false
|
||||
}
|
||||
info, err := f.Stat()
|
||||
if err != nil {
|
||||
_ = f.Close()
|
||||
return nil, 0, false
|
||||
}
|
||||
total := info.Size()
|
||||
if total < int64(trailerLen) {
|
||||
_ = f.Close()
|
||||
return nil, 0, false
|
||||
}
|
||||
|
||||
tr := make([]byte, trailerLen)
|
||||
if _, err := f.ReadAt(tr, total-int64(trailerLen)); err != nil {
|
||||
_ = f.Close()
|
||||
return nil, 0, false
|
||||
}
|
||||
if string(tr[:8]) != trailerMagic || string(tr[trailerLen-8:]) != trailerMagic {
|
||||
_ = f.Close()
|
||||
return nil, 0, false
|
||||
}
|
||||
zlen := int64(binary.LittleEndian.Uint64(tr[8:16]))
|
||||
start := total - int64(trailerLen) - zlen
|
||||
if zlen <= 0 || start < 0 {
|
||||
_ = f.Close()
|
||||
return nil, 0, false
|
||||
}
|
||||
return io.NewSectionReader(f, start, zlen), zlen, true
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
package pack
|
||||
|
||||
import (
|
||||
"path"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// mimeByExt maps a lower-case file extension (with the dot) to the MIME type
|
||||
// kage records for it. Inference is by extension only, never by sniffing the
|
||||
// bytes, so the same input always yields the same output. Anything not listed
|
||||
// falls back to application/octet-stream and is stored uncompressed.
|
||||
var mimeByExt = map[string]string{
|
||||
".html": "text/html",
|
||||
".htm": "text/html",
|
||||
".css": "text/css",
|
||||
".js": "text/javascript",
|
||||
".mjs": "text/javascript",
|
||||
".json": "application/json",
|
||||
".xml": "application/xml",
|
||||
".svg": "image/svg+xml",
|
||||
".txt": "text/plain",
|
||||
".png": "image/png",
|
||||
".jpg": "image/jpeg",
|
||||
".jpeg": "image/jpeg",
|
||||
".gif": "image/gif",
|
||||
".webp": "image/webp",
|
||||
".avif": "image/avif",
|
||||
".ico": "image/x-icon",
|
||||
".woff2": "font/woff2",
|
||||
".woff": "font/woff",
|
||||
".ttf": "font/ttf",
|
||||
".otf": "font/otf",
|
||||
".eot": "application/vnd.ms-fontobject",
|
||||
".mp4": "video/mp4",
|
||||
".m4v": "video/mp4",
|
||||
".webm": "video/webm",
|
||||
".mp3": "audio/mpeg",
|
||||
".ogg": "audio/ogg",
|
||||
".pdf": "application/pdf",
|
||||
".zip": "application/zip",
|
||||
".wasm": "application/wasm",
|
||||
}
|
||||
|
||||
// MimeForExt returns the MIME type for a path's extension, defaulting to
|
||||
// application/octet-stream when the extension is unknown or absent.
|
||||
func MimeForExt(p string) string {
|
||||
ext := strings.ToLower(path.Ext(p))
|
||||
if m, ok := mimeByExt[ext]; ok {
|
||||
return m
|
||||
}
|
||||
return "application/octet-stream"
|
||||
}
|
||||
@@ -0,0 +1,253 @@
|
||||
package pack
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/binary"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/tamnd/kage/urlx"
|
||||
"github.com/tamnd/kage/zim"
|
||||
)
|
||||
|
||||
// writeMirror lays down a small kage-style mirror under a temp dir and returns
|
||||
// the host dir.
|
||||
func writeMirror(t *testing.T) string {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
host := filepath.Join(root, "example.com")
|
||||
files := map[string]string{
|
||||
"index.html": "<!doctype html><title>Example Home</title><h1>Hi</h1>",
|
||||
"about/index.html": "<!doctype html><title>About</title><h1>About</h1>",
|
||||
"_kage/example.com/x/logo.png": "\x89PNGfake",
|
||||
"_kage/state.json": `{"visited":[]}`, // must be skipped
|
||||
}
|
||||
for rel, body := range files {
|
||||
p := filepath.Join(host, filepath.FromSlash(rel))
|
||||
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(p, []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return host
|
||||
}
|
||||
|
||||
func TestMimeForExt(t *testing.T) {
|
||||
cases := map[string]string{
|
||||
"a/b/index.html": "text/html",
|
||||
"style.CSS": "text/css",
|
||||
"data.json": "application/json",
|
||||
"icon.svg": "image/svg+xml",
|
||||
"logo.png": "image/png",
|
||||
"photo.JPEG": "image/jpeg",
|
||||
"font.woff2": "font/woff2",
|
||||
"clip.mp4": "video/mp4",
|
||||
"doc.pdf": "application/pdf",
|
||||
"mystery": "application/octet-stream",
|
||||
"archive.tar.zst": "application/octet-stream",
|
||||
}
|
||||
for in, want := range cases {
|
||||
if got := MimeForExt(in); got != want {
|
||||
t.Errorf("MimeForExt(%q) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildZIMRoundTrip(t *testing.T) {
|
||||
host := writeMirror(t)
|
||||
out := filepath.Join(t.TempDir(), "example.zim")
|
||||
path, size, err := BuildZIM(host, ZIMOptions{Out: out, Date: "2026-06-14", Version: "test"})
|
||||
if err != nil {
|
||||
t.Fatalf("BuildZIM: %v", err)
|
||||
}
|
||||
if path != out {
|
||||
t.Errorf("path = %q, want %q", path, out)
|
||||
}
|
||||
fi, err := os.Stat(out)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if fi.Size() != size {
|
||||
t.Errorf("reported size %d, file is %d", size, fi.Size())
|
||||
}
|
||||
|
||||
r, err := zim.Open(out)
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
defer func() { _ = r.Close() }()
|
||||
|
||||
// Main page is the root index.
|
||||
mp, err := r.MainPage()
|
||||
if err != nil {
|
||||
t.Fatalf("MainPage: %v", err)
|
||||
}
|
||||
if !bytes.Contains(mp.Data, []byte("Example Home")) {
|
||||
t.Errorf("main page wrong: %.40q", mp.Data)
|
||||
}
|
||||
if mp.MimeType != "text/html" {
|
||||
t.Errorf("main page mime = %q", mp.MimeType)
|
||||
}
|
||||
|
||||
// Binary asset round-trips byte-for-byte.
|
||||
logo, err := r.Get(zim.NamespaceContent, "_kage/example.com/x/logo.png")
|
||||
if err != nil {
|
||||
t.Fatalf("Get logo: %v", err)
|
||||
}
|
||||
if string(logo.Data) != "\x89PNGfake" {
|
||||
t.Errorf("logo bytes wrong: %q", logo.Data)
|
||||
}
|
||||
|
||||
// Title metadata comes from the main page's <title>.
|
||||
title, err := r.Get(zim.NamespaceMetadata, "Title")
|
||||
if err != nil || string(title.Data) != "Example Home" {
|
||||
t.Errorf("M/Title = %q, %v", title.Data, err)
|
||||
}
|
||||
|
||||
// state.json was skipped.
|
||||
if _, err := r.Get(zim.NamespaceContent, urlx.DefaultReserved+"/state.json"); err == nil {
|
||||
t.Error("state.json should not be packed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildZIMDeterministic(t *testing.T) {
|
||||
host := writeMirror(t)
|
||||
dir := t.TempDir()
|
||||
a, _, err := BuildZIM(host, ZIMOptions{Out: filepath.Join(dir, "a.zim"), Date: "2026-06-14"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _, err := BuildZIM(host, ZIMOptions{Out: filepath.Join(dir, "b.zim"), Date: "2026-06-14"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ba, _ := os.ReadFile(a)
|
||||
bb, _ := os.ReadFile(b)
|
||||
if !bytes.Equal(ba, bb) {
|
||||
t.Error("same mirror produced different archives")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPickMainPage(t *testing.T) {
|
||||
cases := []struct {
|
||||
in []string
|
||||
want string
|
||||
}{
|
||||
{[]string{"a/index.html", "index.html", "b.html"}, "index.html"},
|
||||
{[]string{"z/deep/p.html", "top.html", "a/p.html"}, "top.html"},
|
||||
{[]string{"b/x.html", "a/x.html"}, "a/x.html"}, // same depth, lexical
|
||||
{nil, ""},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := pickMainPage(c.in); got != c.want {
|
||||
t.Errorf("pickMainPage(%v) = %q, want %q", c.in, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestBinaryTrailerRoundTrip exercises the BuildBinary append contract and the
|
||||
// trailer it leaves, without depending on os.Executable: it appends a ZIM to a
|
||||
// fake base, reads the trailer back the way Embedded does, and serves the
|
||||
// recovered archive.
|
||||
func TestBinaryTrailerRoundTrip(t *testing.T) {
|
||||
host := writeMirror(t)
|
||||
zbytes, err := BuildZIMBytes(host, ZIMOptions{Date: "2026-06-14"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
dir := t.TempDir()
|
||||
base := filepath.Join(dir, "fakekage")
|
||||
baseBytes := bytes.Repeat([]byte("BASE"), 64) // stand-in for a kage binary
|
||||
if err := os.WriteFile(base, baseBytes, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
out := filepath.Join(dir, "viewer")
|
||||
_, total, err := BuildBinary(zbytes, BinaryOptions{Out: out, Base: base})
|
||||
if err != nil {
|
||||
t.Fatalf("BuildBinary: %v", err)
|
||||
}
|
||||
if total != int64(len(baseBytes)+len(zbytes)+trailerLen) {
|
||||
t.Errorf("total %d, want %d", total, len(baseBytes)+len(zbytes)+trailerLen)
|
||||
}
|
||||
|
||||
f, err := os.Open(out)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer func() { _ = f.Close() }()
|
||||
fi, _ := f.Stat()
|
||||
end := fi.Size()
|
||||
|
||||
tr := make([]byte, trailerLen)
|
||||
if _, err := f.ReadAt(tr, end-int64(trailerLen)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(tr[:8]) != trailerMagic || string(tr[trailerLen-8:]) != trailerMagic {
|
||||
t.Fatal("trailer magic missing")
|
||||
}
|
||||
zlen := int64(binary.LittleEndian.Uint64(tr[8:16]))
|
||||
if zlen != int64(len(zbytes)) {
|
||||
t.Errorf("trailer length %d, want %d", zlen, len(zbytes))
|
||||
}
|
||||
start := end - int64(trailerLen) - zlen
|
||||
if start != int64(len(baseBytes)) {
|
||||
t.Errorf("archive start %d, want %d", start, len(baseBytes))
|
||||
}
|
||||
|
||||
sec := io.NewSectionReader(f, start, zlen)
|
||||
r, err := zim.NewReader(sec, zlen)
|
||||
if err != nil {
|
||||
t.Fatalf("reopen appended zim: %v", err)
|
||||
}
|
||||
mp, err := r.MainPage()
|
||||
if err != nil || !bytes.Contains(mp.Data, []byte("Example Home")) {
|
||||
t.Errorf("recovered main page wrong: %.40q (%v)", mp.Data, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandler(t *testing.T) {
|
||||
host := writeMirror(t)
|
||||
out := filepath.Join(t.TempDir(), "h.zim")
|
||||
if _, _, err := BuildZIM(host, ZIMOptions{Out: out, Date: "2026-06-14"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r, err := zim.Open(out)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer func() { _ = r.Close() }()
|
||||
|
||||
srv := httptest.NewServer(Handler(r))
|
||||
defer srv.Close()
|
||||
|
||||
get := func(p string) (int, string) {
|
||||
resp, err := http.Get(srv.URL + p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
return resp.StatusCode, string(b)
|
||||
}
|
||||
|
||||
if code, body := get("/"); code != 200 || !bytes.Contains([]byte(body), []byte("Example Home")) {
|
||||
t.Errorf("GET / = %d %.30q", code, body)
|
||||
}
|
||||
if code, _ := get("/about/index.html"); code != 200 {
|
||||
t.Errorf("GET /about/index.html = %d", code)
|
||||
}
|
||||
if code, _ := get("/" + urlx.DefaultReserved + "/state.json"); code != 404 {
|
||||
t.Errorf("GET state.json = %d, want 404", code)
|
||||
}
|
||||
if code, _ := get("/missing.html"); code != 404 {
|
||||
t.Errorf("GET missing = %d, want 404", code)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
package pack
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/tamnd/kage/zim"
|
||||
)
|
||||
|
||||
// Handler serves a ZIM archive over HTTP. "/" maps to the archive's main page;
|
||||
// "/a/b.png" maps to the C/a/b.png content entry. Because the saved HTML's links
|
||||
// are mirror-relative paths, and those are exactly the C urls, a click in a
|
||||
// served page hits the right entry with no rewriting. A miss is a plain 404.
|
||||
func Handler(r *zim.Reader) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
|
||||
p := strings.TrimPrefix(req.URL.Path, "/")
|
||||
if p == "" {
|
||||
blob, err := r.MainPage()
|
||||
if err != nil {
|
||||
http.NotFound(w, req)
|
||||
return
|
||||
}
|
||||
serveBlob(w, blob)
|
||||
return
|
||||
}
|
||||
blob, err := r.Get(zim.NamespaceContent, p)
|
||||
if errors.Is(err, zim.ErrNotFound) {
|
||||
http.NotFound(w, req)
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
serveBlob(w, blob)
|
||||
})
|
||||
}
|
||||
|
||||
func serveBlob(w http.ResponseWriter, b zim.Blob) {
|
||||
if b.MimeType != "" {
|
||||
w.Header().Set("Content-Type", b.MimeType)
|
||||
}
|
||||
_, _ = w.Write(b.Data)
|
||||
}
|
||||
+244
@@ -0,0 +1,244 @@
|
||||
// Package pack turns a kage mirror on disk into a distributable artifact: a ZIM
|
||||
// archive, or a self-contained executable that serves the mirror offline. It is
|
||||
// the only pack-side package that touches the filesystem and the running
|
||||
// executable; the byte-level format work lives in the zim package.
|
||||
package pack
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"golang.org/x/net/html"
|
||||
|
||||
"github.com/tamnd/kage/urlx"
|
||||
"github.com/tamnd/kage/zim"
|
||||
)
|
||||
|
||||
// ZIMOptions controls how a mirror is packed into a ZIM archive. Date is passed
|
||||
// in from the CLI boundary rather than read from the clock, so the zim and pack
|
||||
// packages stay pure and packing the same mirror twice is byte-identical.
|
||||
type ZIMOptions struct {
|
||||
Out string // output path (default <mirror-base>.zim)
|
||||
NoCompress bool // store every cluster raw (code 1)
|
||||
Title string // overrides M/Title
|
||||
Description string // M/Description
|
||||
Language string // M/Language (default "eng")
|
||||
Date string // M/Date, e.g. "2026-06-14"
|
||||
Version string // kage version, recorded as M/Scraper
|
||||
}
|
||||
|
||||
// BuildZIM walks mirrorDir, turns every file into a C/ content entry, infers the
|
||||
// MIME from the extension, picks a main page, adds M/ metadata and a W/mainPage
|
||||
// redirect, and writes a .zim to opts.Out. It returns the output path and the
|
||||
// number of bytes written.
|
||||
func BuildZIM(mirrorDir string, opts ZIMOptions) (string, int64, error) {
|
||||
w, err := buildWriter(mirrorDir, opts)
|
||||
if err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
out := opts.Out
|
||||
if out == "" {
|
||||
out = filepath.Base(mirrorDir) + ".zim"
|
||||
}
|
||||
f, err := os.Create(out)
|
||||
if err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
bw := bufio.NewWriter(f)
|
||||
n, err := w.WriteTo(bw)
|
||||
if err != nil {
|
||||
_ = f.Close()
|
||||
return out, n, err
|
||||
}
|
||||
if err := bw.Flush(); err != nil {
|
||||
_ = f.Close()
|
||||
return out, n, err
|
||||
}
|
||||
return out, n, f.Close()
|
||||
}
|
||||
|
||||
// BuildZIMBytes is the buffer-returning sibling of BuildZIM: it runs the same
|
||||
// walk and returns the archive in memory, which the binary path appends to a
|
||||
// base executable without writing the ZIM to disk first.
|
||||
func BuildZIMBytes(mirrorDir string, opts ZIMOptions) ([]byte, error) {
|
||||
w, err := buildWriter(mirrorDir, opts)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var buf bytes.Buffer
|
||||
if _, err := w.WriteTo(&buf); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return buf.Bytes(), nil
|
||||
}
|
||||
|
||||
// buildWriter does the shared work of both BuildZIM and BuildZIMBytes: it loads
|
||||
// every file under mirrorDir into a zim.Writer with metadata and a main page.
|
||||
func buildWriter(mirrorDir string, opts ZIMOptions) (*zim.Writer, error) {
|
||||
info, err := os.Stat(mirrorDir)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !info.IsDir() {
|
||||
return nil, fmt.Errorf("pack: %q is not a directory", mirrorDir)
|
||||
}
|
||||
|
||||
w := zim.NewWriter()
|
||||
if opts.NoCompress {
|
||||
w.SetNoCompress(true)
|
||||
}
|
||||
|
||||
skip := urlx.DefaultReserved + "/state.json"
|
||||
var htmlPages []string
|
||||
counts := map[string]int{}
|
||||
|
||||
walkErr := filepath.WalkDir(mirrorDir, func(p string, d fs.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if d.IsDir() {
|
||||
return nil
|
||||
}
|
||||
rel := slashRel(mirrorDir, p)
|
||||
if rel == skip {
|
||||
return nil
|
||||
}
|
||||
data, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
mime := MimeForExt(rel)
|
||||
if mime == "text/html" {
|
||||
htmlPages = append(htmlPages, rel)
|
||||
}
|
||||
counts[mime]++
|
||||
w.AddContent(zim.NamespaceContent, rel, "", mime, data)
|
||||
return nil
|
||||
})
|
||||
if walkErr != nil {
|
||||
return nil, walkErr
|
||||
}
|
||||
|
||||
main := pickMainPage(htmlPages)
|
||||
if main != "" {
|
||||
w.SetMainPage(zim.NamespaceContent, main)
|
||||
w.AddRedirect(zim.NamespaceWellKnown, "mainPage", "", zim.NamespaceContent, main)
|
||||
}
|
||||
|
||||
host := filepath.Base(mirrorDir)
|
||||
title := firstNonEmpty(opts.Title, htmlTitleOf(mirrorDir, main), host)
|
||||
w.AddMetadata("Title", title)
|
||||
w.AddMetadata("Language", firstNonEmpty(opts.Language, "eng"))
|
||||
if opts.Description != "" {
|
||||
w.AddMetadata("Description", opts.Description)
|
||||
}
|
||||
w.AddMetadata("Creator", "kage")
|
||||
w.AddMetadata("Publisher", "kage")
|
||||
if opts.Date != "" {
|
||||
w.AddMetadata("Date", opts.Date)
|
||||
}
|
||||
w.AddMetadata("Scraper", strings.TrimSpace("kage "+opts.Version))
|
||||
w.AddMetadata("Source", host)
|
||||
w.AddMetadata("Counter", counterString(counts))
|
||||
return w, nil
|
||||
}
|
||||
|
||||
// pickMainPage chooses the archive's entry point: the root index if present,
|
||||
// else the shallowest HTML page, ties broken lexicographically for determinism.
|
||||
// It returns "" when the mirror has no HTML at all.
|
||||
func pickMainPage(htmlPages []string) string {
|
||||
for _, p := range htmlPages {
|
||||
if p == "index.html" {
|
||||
return p
|
||||
}
|
||||
}
|
||||
sorted := append([]string(nil), htmlPages...)
|
||||
sort.Slice(sorted, func(i, j int) bool {
|
||||
di, dj := strings.Count(sorted[i], "/"), strings.Count(sorted[j], "/")
|
||||
if di != dj {
|
||||
return di < dj
|
||||
}
|
||||
return sorted[i] < sorted[j]
|
||||
})
|
||||
if len(sorted) > 0 {
|
||||
return sorted[0]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// htmlTitleOf reads the main page off disk and returns its <title>, or "" if
|
||||
// there is no main page or no title.
|
||||
func htmlTitleOf(mirrorDir, mainURL string) string {
|
||||
if mainURL == "" {
|
||||
return ""
|
||||
}
|
||||
f, err := os.Open(filepath.Join(mirrorDir, filepath.FromSlash(mainURL)))
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
defer func() { _ = f.Close() }()
|
||||
doc, err := html.Parse(f)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(findTitle(doc))
|
||||
}
|
||||
|
||||
// findTitle returns the text of the first <title> element in depth-first order.
|
||||
func findTitle(n *html.Node) string {
|
||||
if n.Type == html.ElementNode && n.Data == "title" {
|
||||
var b strings.Builder
|
||||
for c := n.FirstChild; c != nil; c = c.NextSibling {
|
||||
if c.Type == html.TextNode {
|
||||
b.WriteString(c.Data)
|
||||
}
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
for c := n.FirstChild; c != nil; c = c.NextSibling {
|
||||
if t := findTitle(c); t != "" {
|
||||
return t
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// counterString renders the M/Counter value Kiwix uses for stats: a
|
||||
// semicolon-separated list of mime=count pairs, sorted for determinism.
|
||||
func counterString(counts map[string]int) string {
|
||||
mimes := make([]string, 0, len(counts))
|
||||
for m := range counts {
|
||||
mimes = append(mimes, m)
|
||||
}
|
||||
sort.Strings(mimes)
|
||||
parts := make([]string, len(mimes))
|
||||
for i, m := range mimes {
|
||||
parts[i] = fmt.Sprintf("%s=%d", m, counts[m])
|
||||
}
|
||||
return strings.Join(parts, ";")
|
||||
}
|
||||
|
||||
// slashRel returns p relative to root using forward slashes, the form ZIM urls
|
||||
// take regardless of the host filesystem separator.
|
||||
func slashRel(root, p string) string {
|
||||
rel, err := filepath.Rel(root, p)
|
||||
if err != nil {
|
||||
rel = p
|
||||
}
|
||||
return filepath.ToSlash(rel)
|
||||
}
|
||||
|
||||
func firstNonEmpty(vals ...string) string {
|
||||
for _, v := range vals {
|
||||
if v != "" {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
//go:build !webview
|
||||
|
||||
package viewer
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os/exec"
|
||||
"runtime"
|
||||
)
|
||||
|
||||
// Native is false in the default pure-Go build: there is no native window, so
|
||||
// the viewer hands the URL to the system browser.
|
||||
const Native = false
|
||||
|
||||
// LockMainThread is a no-op without a native UI to pin to the main thread.
|
||||
func LockMainThread() {}
|
||||
|
||||
// Show opens the system browser at o.URL when o.Browser is set, then blocks
|
||||
// until the context is cancelled (Ctrl-C), leaving the caller's HTTP server up
|
||||
// in the meantime. Launching the browser is best-effort; a failure is ignored
|
||||
// because the URL has already been printed for the user to open by hand.
|
||||
func Show(ctx context.Context, o Options) error {
|
||||
if o.Browser {
|
||||
_ = openInBrowser(o.URL)
|
||||
}
|
||||
<-ctx.Done()
|
||||
return nil
|
||||
}
|
||||
|
||||
func openInBrowser(url string) error {
|
||||
var cmd *exec.Cmd
|
||||
switch runtime.GOOS {
|
||||
case "darwin":
|
||||
cmd = exec.Command("open", url)
|
||||
case "windows":
|
||||
cmd = exec.Command("rundll32", "url.dll,FileProtocolHandler", url)
|
||||
default:
|
||||
cmd = exec.Command("xdg-open", url)
|
||||
}
|
||||
return cmd.Start()
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
//go:build !webview
|
||||
|
||||
package viewer
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestNativeIsFalseInDefaultBuild(t *testing.T) {
|
||||
if Native {
|
||||
t.Fatal("Native should be false without the webview build tag")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLockMainThreadIsNoop(t *testing.T) {
|
||||
// Must not panic; there is no native UI to pin to.
|
||||
LockMainThread()
|
||||
}
|
||||
|
||||
func TestShowReturnsWhenContextCancelled(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
done := make(chan error, 1)
|
||||
// Browser:false so no system browser is launched during the test.
|
||||
go func() { done <- Show(ctx, Options{URL: "http://127.0.0.1:0", Browser: false}) }()
|
||||
|
||||
cancel()
|
||||
select {
|
||||
case err := <-done:
|
||||
if err != nil {
|
||||
t.Fatalf("Show returned error: %v", err)
|
||||
}
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("Show did not return after context cancellation")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
// Package viewer presents a served site to the user. It has two
|
||||
// implementations chosen at build time: by default (pure Go, CGO_ENABLED=0) it
|
||||
// opens the system browser, and with the "webview" build tag (which needs cgo)
|
||||
// it opens a native window backed by the operating system's WebView, so a
|
||||
// packed kage binary feels like a standalone app rather than a browser tab.
|
||||
//
|
||||
// Both builds expose the same three symbols: Native, LockMainThread, and Show.
|
||||
// The caller starts an HTTP server, then calls Show on the main goroutine; Show
|
||||
// blocks until the window is closed (native) or the context is cancelled
|
||||
// (browser), at which point the caller shuts the server down.
|
||||
package viewer
|
||||
|
||||
// Options configures a viewer window.
|
||||
type Options struct {
|
||||
Title string // window title; the archive's M/Title, falling back to "kage"
|
||||
URL string // local URL the server is listening on
|
||||
// Browser, in the default build, opens the system browser. The native build
|
||||
// ignores it and always shows its own window.
|
||||
Browser bool
|
||||
}
|
||||
|
||||
// Native reports whether this build opens a native window (webview tag) or
|
||||
// falls back to the system browser. Show and LockMainThread are defined in the
|
||||
// per-build files browser.go and webview.go.
|
||||
@@ -0,0 +1,53 @@
|
||||
//go:build webview
|
||||
|
||||
package viewer
|
||||
|
||||
import (
|
||||
"context"
|
||||
"runtime"
|
||||
|
||||
webview "github.com/webview/webview_go"
|
||||
)
|
||||
|
||||
// Native is true in the webview build: Show opens a real window backed by the
|
||||
// operating system's WebView (WKWebView on macOS, WebView2 on Windows,
|
||||
// WebKitGTK on Linux), so a packed kage feels like a standalone app.
|
||||
//
|
||||
// This build needs cgo and links the platform WebView, so it is opt-in
|
||||
// (-tags webview) and kept out of the default CGO_ENABLED=0 release pipeline.
|
||||
const Native = true
|
||||
|
||||
// LockMainThread pins the calling goroutine to its OS thread. main calls it
|
||||
// first thing, while the main goroutine is still on the process's initial
|
||||
// thread, because the macOS WebView must be driven from that thread.
|
||||
func LockMainThread() { runtime.LockOSThread() }
|
||||
|
||||
// Show opens a native window pointed at o.URL and runs the UI event loop on the
|
||||
// calling (main) goroutine, blocking until the window is closed. A cancelled
|
||||
// context terminates the loop too, so Ctrl-C still shuts the viewer down. The
|
||||
// o.Browser flag is ignored: the whole point of this build is the native window.
|
||||
func Show(ctx context.Context, o Options) error {
|
||||
w := webview.New(false)
|
||||
defer w.Destroy()
|
||||
|
||||
title := o.Title
|
||||
if title == "" {
|
||||
title = "kage"
|
||||
}
|
||||
w.SetTitle(title)
|
||||
w.SetSize(1024, 768, webview.HintNone)
|
||||
w.Navigate(o.URL)
|
||||
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
w.Dispatch(func() { w.Terminate() })
|
||||
case <-done:
|
||||
}
|
||||
}()
|
||||
|
||||
w.Run()
|
||||
close(done)
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
package zim
|
||||
|
||||
import (
|
||||
"sync"
|
||||
|
||||
"github.com/klauspost/compress/zstd"
|
||||
)
|
||||
|
||||
// A single shared zstd codec. Both EncodeAll and DecodeAll are safe for
|
||||
// concurrent use, so one encoder and one decoder serve the whole process.
|
||||
var (
|
||||
zstdOnce sync.Once
|
||||
zstdEnc *zstd.Encoder
|
||||
zstdDec *zstd.Decoder
|
||||
)
|
||||
|
||||
func initZstd() {
|
||||
zstdOnce.Do(func() {
|
||||
zstdEnc, _ = zstd.NewWriter(nil, zstd.WithEncoderLevel(zstd.SpeedBetterCompression))
|
||||
zstdDec, _ = zstd.NewReader(nil)
|
||||
})
|
||||
}
|
||||
|
||||
func zstdEncode(p []byte) []byte {
|
||||
initZstd()
|
||||
return zstdEnc.EncodeAll(p, nil)
|
||||
}
|
||||
|
||||
func zstdDecode(p []byte) ([]byte, error) {
|
||||
initZstd()
|
||||
return zstdDec.DecodeAll(p, nil)
|
||||
}
|
||||
|
||||
// isTextMime reports whether content of this MIME type is worth compressing.
|
||||
// Already-compressed media (images, fonts, audio, video, archives) is stored
|
||||
// uncompressed so we do not burn CPU inflating it by a few bytes.
|
||||
func isTextMime(mime string) bool {
|
||||
switch mime {
|
||||
case "application/json", "application/xml", "application/javascript",
|
||||
"application/x-javascript":
|
||||
return true
|
||||
}
|
||||
if len(mime) >= 5 && mime[:5] == "text/" {
|
||||
return true
|
||||
}
|
||||
// Any structured-XML type: application/rss+xml, image/svg+xml, ...
|
||||
return len(mime) >= 4 && mime[len(mime)-4:] == "+xml"
|
||||
}
|
||||
+115
@@ -0,0 +1,115 @@
|
||||
// Package zim reads and writes the ZIM offline-archive format, the open
|
||||
// single-file container that Kiwix uses to ship offline content. kage uses it
|
||||
// to pack a cloned mirror into one indexable, compressed file that a reader can
|
||||
// random-access without unpacking.
|
||||
//
|
||||
// The package is pure: no network, no clock, no global state beyond a lazily
|
||||
// built zstd codec. A ZIM file is laid out as a fixed header, a MIME-type list,
|
||||
// three pointer lists (URL, title, cluster), a run of directory entries, a run
|
||||
// of clusters that hold the content, and a trailing MD5. Every cross-reference
|
||||
// is an absolute file position recorded in the header, so the writer assigns
|
||||
// positions in one pass and emits bytes in a second. All integers are
|
||||
// little-endian.
|
||||
//
|
||||
// We write the new namespace scheme (minor version 1): all content lives under
|
||||
// the single 'C' namespace, metadata under 'M', and a 'W/mainPage' redirect
|
||||
// points at the entry point. Reading handles redirects and both offset widths.
|
||||
package zim
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// Magic is the ZIM header magic number, the first four bytes of every file.
|
||||
const Magic uint32 = 0x44D495A // 72173914
|
||||
|
||||
const (
|
||||
majorVersion uint16 = 5
|
||||
minorVersion uint16 = 1 // single 'C' content namespace
|
||||
headerLen = 80
|
||||
)
|
||||
|
||||
// Namespaces in the new (minor version 1) scheme.
|
||||
const (
|
||||
NamespaceContent byte = 'C' // pages and assets
|
||||
NamespaceMetadata byte = 'M' // M/Title, M/Date, ...
|
||||
NamespaceWellKnown byte = 'W' // W/mainPage redirect
|
||||
)
|
||||
|
||||
// Compression codes carried in the low nibble of a cluster's info byte.
|
||||
const (
|
||||
compNone uint8 = 1 // stored, no compression
|
||||
compXZ uint8 = 4 // xz / LZMA2 (read-only support)
|
||||
compZstd uint8 = 5 // zstd (what we write for text)
|
||||
|
||||
extendedFlag uint8 = 0x10 // bit 4: cluster offsets are uint64, not uint32
|
||||
)
|
||||
|
||||
// Sentinels stored in a directory entry's mimetype field to mark non-content
|
||||
// entries. A redirect reuses the cluster slot to hold its target's URL index.
|
||||
const (
|
||||
redirectEntry uint16 = 0xffff
|
||||
linkTargetEntry uint16 = 0xfffe
|
||||
deletedEntry uint16 = 0xfffd
|
||||
)
|
||||
|
||||
// noMainPage is the mainPage/layoutPage value meaning "none".
|
||||
const noMainPage uint32 = 0xffffffff
|
||||
|
||||
// header is the 80-byte ZIM header.
|
||||
type header struct {
|
||||
uuid [16]byte
|
||||
articleCount uint32
|
||||
clusterCount uint32
|
||||
urlPtrPos uint64
|
||||
titlePtrPos uint64
|
||||
clusterPtrPos uint64
|
||||
mimeListPos uint64
|
||||
mainPage uint32
|
||||
layoutPage uint32
|
||||
checksumPos uint64
|
||||
}
|
||||
|
||||
// marshal encodes the header to its 80 wire bytes.
|
||||
func (h header) marshal() []byte {
|
||||
b := make([]byte, headerLen)
|
||||
le := binary.LittleEndian
|
||||
le.PutUint32(b[0:], Magic)
|
||||
le.PutUint16(b[4:], majorVersion)
|
||||
le.PutUint16(b[6:], minorVersion)
|
||||
copy(b[8:24], h.uuid[:])
|
||||
le.PutUint32(b[24:], h.articleCount)
|
||||
le.PutUint32(b[28:], h.clusterCount)
|
||||
le.PutUint64(b[32:], h.urlPtrPos)
|
||||
le.PutUint64(b[40:], h.titlePtrPos)
|
||||
le.PutUint64(b[48:], h.clusterPtrPos)
|
||||
le.PutUint64(b[56:], h.mimeListPos)
|
||||
le.PutUint32(b[64:], h.mainPage)
|
||||
le.PutUint32(b[68:], h.layoutPage)
|
||||
le.PutUint64(b[72:], h.checksumPos)
|
||||
return b
|
||||
}
|
||||
|
||||
// parseHeader decodes and validates an 80-byte header.
|
||||
func parseHeader(b []byte) (header, error) {
|
||||
var h header
|
||||
if len(b) < headerLen {
|
||||
return h, fmt.Errorf("zim: short header: %d bytes", len(b))
|
||||
}
|
||||
le := binary.LittleEndian
|
||||
if le.Uint32(b[0:]) != Magic {
|
||||
return h, fmt.Errorf("zim: bad magic, not a ZIM file")
|
||||
}
|
||||
copy(h.uuid[:], b[8:24])
|
||||
h.articleCount = le.Uint32(b[24:])
|
||||
h.clusterCount = le.Uint32(b[28:])
|
||||
h.urlPtrPos = le.Uint64(b[32:])
|
||||
h.titlePtrPos = le.Uint64(b[40:])
|
||||
h.clusterPtrPos = le.Uint64(b[48:])
|
||||
h.mimeListPos = le.Uint64(b[56:])
|
||||
h.mainPage = le.Uint32(b[64:])
|
||||
h.layoutPage = le.Uint32(b[68:])
|
||||
h.checksumPos = le.Uint64(b[72:])
|
||||
return h, nil
|
||||
}
|
||||
+339
@@ -0,0 +1,339 @@
|
||||
package zim
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/binary"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// ErrNotFound is returned by Get when no entry matches the namespace and url.
|
||||
// Callers (such as the HTTP handler) test for it with errors.Is to map a miss
|
||||
// to a 404.
|
||||
var ErrNotFound = errors.New("zim: not found")
|
||||
|
||||
// Reader provides random access to a ZIM file's entries. Open one with Open or
|
||||
// NewReader, then look entries up by namespace and url, or fetch the main page.
|
||||
// Decompressed clusters are cached so repeated reads from one cluster are cheap.
|
||||
type Reader struct {
|
||||
ra io.ReaderAt
|
||||
closer io.Closer
|
||||
size int64
|
||||
hdr header
|
||||
mimes []string
|
||||
|
||||
mu sync.Mutex
|
||||
cache map[uint32][]byte // cluster index -> decompressed data section
|
||||
cacheExtended map[uint32]bool // cluster index -> uint64-offset cluster
|
||||
}
|
||||
|
||||
// Blob is the result of a lookup: the resolved entry's bytes and metadata.
|
||||
type Blob struct {
|
||||
Namespace byte
|
||||
URL string
|
||||
Title string
|
||||
MimeType string
|
||||
Data []byte
|
||||
}
|
||||
|
||||
// Open opens a ZIM file on disk. Close the returned reader when done.
|
||||
func Open(path string) (*Reader, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
fi, err := f.Stat()
|
||||
if err != nil {
|
||||
_ = f.Close()
|
||||
return nil, err
|
||||
}
|
||||
r, err := NewReader(f, fi.Size())
|
||||
if err != nil {
|
||||
_ = f.Close()
|
||||
return nil, err
|
||||
}
|
||||
r.closer = f
|
||||
return r, nil
|
||||
}
|
||||
|
||||
// NewReader reads the header and MIME list from ra, which must hold size bytes.
|
||||
func NewReader(ra io.ReaderAt, size int64) (*Reader, error) {
|
||||
r := &Reader{ra: ra, size: size, cache: map[uint32][]byte{}}
|
||||
hb, err := r.at(0, headerLen)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("zim: read header: %w", err)
|
||||
}
|
||||
r.hdr, err = parseHeader(hb)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if r.hdr.mimeListPos > r.hdr.urlPtrPos || r.hdr.urlPtrPos > uint64(size) {
|
||||
return nil, fmt.Errorf("zim: inconsistent header offsets")
|
||||
}
|
||||
mb, err := r.at(r.hdr.mimeListPos, int(r.hdr.urlPtrPos-r.hdr.mimeListPos))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("zim: read mime list: %w", err)
|
||||
}
|
||||
for _, part := range bytes.Split(mb, []byte{0}) {
|
||||
if len(part) == 0 {
|
||||
break
|
||||
}
|
||||
r.mimes = append(r.mimes, string(part))
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
|
||||
// Close releases the underlying file, if Open created one.
|
||||
func (r *Reader) Close() error {
|
||||
if r.closer != nil {
|
||||
return r.closer.Close()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Count returns the number of directory entries.
|
||||
func (r *Reader) Count() uint32 { return r.hdr.articleCount }
|
||||
|
||||
// MimeTypes returns the archive's MIME-type list.
|
||||
func (r *Reader) MimeTypes() []string { return r.mimes }
|
||||
|
||||
// MainPage returns the archive's entry point, or an error if none is set.
|
||||
func (r *Reader) MainPage() (Blob, error) {
|
||||
if r.hdr.mainPage == noMainPage {
|
||||
return Blob{}, fmt.Errorf("zim: no main page")
|
||||
}
|
||||
return r.blobAtIndex(r.hdr.mainPage, 0)
|
||||
}
|
||||
|
||||
// Get resolves the entry at (namespace, url), following one or more redirects.
|
||||
func (r *Reader) Get(namespace byte, url string) (Blob, error) {
|
||||
target := key(namespace, url)
|
||||
lo, hi := uint32(0), r.hdr.articleCount
|
||||
for lo < hi {
|
||||
mid := lo + (hi-lo)/2
|
||||
d, err := r.direntAtIndex(mid)
|
||||
if err != nil {
|
||||
return Blob{}, err
|
||||
}
|
||||
switch k := key(d.namespace, d.url); {
|
||||
case k < target:
|
||||
lo = mid + 1
|
||||
case k > target:
|
||||
hi = mid
|
||||
default:
|
||||
return r.blobAtIndex(mid, 0)
|
||||
}
|
||||
}
|
||||
return Blob{}, fmt.Errorf("%w: %c/%s", ErrNotFound, namespace, url)
|
||||
}
|
||||
|
||||
const maxRedirectHops = 16
|
||||
|
||||
func (r *Reader) blobAtIndex(idx uint32, hop int) (Blob, error) {
|
||||
if hop > maxRedirectHops {
|
||||
return Blob{}, fmt.Errorf("zim: redirect loop")
|
||||
}
|
||||
d, err := r.direntAtIndex(idx)
|
||||
if err != nil {
|
||||
return Blob{}, err
|
||||
}
|
||||
if d.redirect {
|
||||
return r.blobAtIndex(d.targetIndex, hop+1)
|
||||
}
|
||||
data, err := r.blobData(d.cluster, d.blob)
|
||||
if err != nil {
|
||||
return Blob{}, err
|
||||
}
|
||||
mime := ""
|
||||
if int(d.mimeIdx) < len(r.mimes) {
|
||||
mime = r.mimes[d.mimeIdx]
|
||||
}
|
||||
return Blob{Namespace: d.namespace, URL: d.url, Title: d.title, MimeType: mime, Data: data}, nil
|
||||
}
|
||||
|
||||
type dirent struct {
|
||||
mimeIdx uint16
|
||||
namespace byte
|
||||
url, title string
|
||||
cluster uint32
|
||||
blob uint32
|
||||
redirect bool
|
||||
targetIndex uint32
|
||||
}
|
||||
|
||||
func (r *Reader) direntAtIndex(idx uint32) (dirent, error) {
|
||||
pb, err := r.at(r.hdr.urlPtrPos+8*uint64(idx), 8)
|
||||
if err != nil {
|
||||
return dirent{}, err
|
||||
}
|
||||
return r.direntAt(binary.LittleEndian.Uint64(pb))
|
||||
}
|
||||
|
||||
func (r *Reader) direntAt(off uint64) (dirent, error) {
|
||||
// Read a window large enough for the fixed head plus url and title; grow if
|
||||
// either string is not terminated within it.
|
||||
window := 512
|
||||
for {
|
||||
b, err := r.at(off, window)
|
||||
if err != nil && len(b) == 0 {
|
||||
return dirent{}, err
|
||||
}
|
||||
var d dirent
|
||||
le := binary.LittleEndian
|
||||
d.mimeIdx = le.Uint16(b[0:])
|
||||
d.namespace = b[3]
|
||||
var p int
|
||||
if d.mimeIdx == redirectEntry {
|
||||
d.redirect = true
|
||||
d.targetIndex = le.Uint32(b[8:])
|
||||
p = 12
|
||||
} else {
|
||||
d.cluster = le.Uint32(b[8:])
|
||||
d.blob = le.Uint32(b[12:])
|
||||
p = 16
|
||||
}
|
||||
url, n1, ok := readCString(b, p)
|
||||
if !ok {
|
||||
if window >= 1<<20 || off+uint64(window) >= uint64(r.size) {
|
||||
return dirent{}, fmt.Errorf("zim: unterminated url at %d", off)
|
||||
}
|
||||
window *= 4
|
||||
continue
|
||||
}
|
||||
title, _, ok := readCString(b, n1)
|
||||
if !ok {
|
||||
if window >= 1<<20 || off+uint64(window) >= uint64(r.size) {
|
||||
return dirent{}, fmt.Errorf("zim: unterminated title at %d", off)
|
||||
}
|
||||
window *= 4
|
||||
continue
|
||||
}
|
||||
d.url, d.title = url, title
|
||||
return d, nil
|
||||
}
|
||||
}
|
||||
|
||||
// blobData returns one blob's bytes, decompressing and caching its cluster.
|
||||
func (r *Reader) blobData(cluster, blob uint32) ([]byte, error) {
|
||||
data, extended, err := r.clusterData(cluster)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
w := uint32(4)
|
||||
if extended {
|
||||
w = 8
|
||||
}
|
||||
need := int((blob + 2) * w)
|
||||
if need > len(data) {
|
||||
return nil, fmt.Errorf("zim: blob %d out of range in cluster %d", blob, cluster)
|
||||
}
|
||||
o0 := readUint(data[blob*w:], w)
|
||||
o1 := readUint(data[(blob+1)*w:], w)
|
||||
if o0 > o1 || int(o1) > len(data) {
|
||||
return nil, fmt.Errorf("zim: bad blob offsets in cluster %d", cluster)
|
||||
}
|
||||
out := make([]byte, o1-o0)
|
||||
copy(out, data[o0:o1])
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (r *Reader) clusterData(cluster uint32) (data []byte, extended bool, err error) {
|
||||
r.mu.Lock()
|
||||
if c, ok := r.cache[cluster]; ok {
|
||||
r.mu.Unlock()
|
||||
// extended-ness is recoverable from the info byte, but the cache stores
|
||||
// already-decoded data whose offsets we re-read with the recorded width.
|
||||
return c, r.cacheExtended[cluster], nil
|
||||
}
|
||||
r.mu.Unlock()
|
||||
|
||||
start, err := r.clusterOffset(cluster)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
end := r.hdr.checksumPos
|
||||
if cluster+1 < r.hdr.clusterCount {
|
||||
if end, err = r.clusterOffset(cluster + 1); err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
}
|
||||
if start >= end || end > uint64(r.size) {
|
||||
return nil, false, fmt.Errorf("zim: bad cluster bounds for %d", cluster)
|
||||
}
|
||||
raw, err := r.at(start, int(end-start))
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
info := raw[0]
|
||||
comp := info & 0x0f
|
||||
extended = info&extendedFlag != 0
|
||||
body := raw[1:]
|
||||
switch comp {
|
||||
case compNone:
|
||||
data = body
|
||||
case compZstd:
|
||||
if data, err = zstdDecode(body); err != nil {
|
||||
return nil, false, fmt.Errorf("zim: zstd cluster %d: %w", cluster, err)
|
||||
}
|
||||
case compXZ:
|
||||
return nil, false, fmt.Errorf("zim: xz clusters are not supported for reading")
|
||||
default:
|
||||
return nil, false, fmt.Errorf("zim: unknown compression %d in cluster %d", comp, cluster)
|
||||
}
|
||||
r.mu.Lock()
|
||||
r.cache[cluster] = data
|
||||
if r.cacheExtended == nil {
|
||||
r.cacheExtended = map[uint32]bool{}
|
||||
}
|
||||
r.cacheExtended[cluster] = extended
|
||||
r.mu.Unlock()
|
||||
return data, extended, nil
|
||||
}
|
||||
|
||||
func (r *Reader) clusterOffset(cluster uint32) (uint64, error) {
|
||||
b, err := r.at(r.hdr.clusterPtrPos+8*uint64(cluster), 8)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return binary.LittleEndian.Uint64(b), nil
|
||||
}
|
||||
|
||||
// at reads n bytes at off, clamped to the file size.
|
||||
func (r *Reader) at(off uint64, n int) ([]byte, error) {
|
||||
if n < 0 {
|
||||
return nil, fmt.Errorf("zim: negative read length")
|
||||
}
|
||||
if off > uint64(r.size) {
|
||||
return nil, io.EOF
|
||||
}
|
||||
if off+uint64(n) > uint64(r.size) {
|
||||
n = int(uint64(r.size) - off)
|
||||
}
|
||||
b := make([]byte, n)
|
||||
if n == 0 {
|
||||
return b, nil
|
||||
}
|
||||
_, err := r.ra.ReadAt(b, int64(off))
|
||||
return b, err
|
||||
}
|
||||
|
||||
func readCString(b []byte, start int) (string, int, bool) {
|
||||
if start > len(b) {
|
||||
return "", start, false
|
||||
}
|
||||
i := bytes.IndexByte(b[start:], 0)
|
||||
if i < 0 {
|
||||
return "", start, false
|
||||
}
|
||||
return string(b[start : start+i]), start + i + 1, true
|
||||
}
|
||||
|
||||
func readUint(b []byte, width uint32) uint32 {
|
||||
if width == 8 {
|
||||
return uint32(binary.LittleEndian.Uint64(b))
|
||||
}
|
||||
return binary.LittleEndian.Uint32(b)
|
||||
}
|
||||
+392
@@ -0,0 +1,392 @@
|
||||
package zim
|
||||
|
||||
import (
|
||||
"crypto/md5"
|
||||
"encoding/binary"
|
||||
"fmt"
|
||||
"io"
|
||||
"sort"
|
||||
)
|
||||
|
||||
// maxClusterContent caps how much blob content accumulates in one cluster
|
||||
// before a new one is started, balancing compression ratio against the cost of
|
||||
// decompressing a whole cluster to read one small blob.
|
||||
const maxClusterContent = 2 << 20 // 2 MiB
|
||||
|
||||
// Writer accumulates entries and serialises them as a ZIM file. Build it with
|
||||
// NewWriter, add content/redirects/metadata, optionally set a main page, then
|
||||
// call WriteTo. The writer holds entries in memory; a kage mirror comfortably
|
||||
// fits, and packing is a one-shot batch job.
|
||||
type Writer struct {
|
||||
entries []*entry
|
||||
byKey map[string]*entry
|
||||
mainKey string
|
||||
noCompress bool
|
||||
}
|
||||
|
||||
type entry struct {
|
||||
namespace byte
|
||||
url string
|
||||
title string
|
||||
mime string
|
||||
data []byte
|
||||
|
||||
redirect bool
|
||||
targetKey string // "<ns><url>" of the redirect target
|
||||
|
||||
// assigned during planning
|
||||
mimeIdx uint16
|
||||
cluster uint32
|
||||
blob uint32
|
||||
targetIndex uint32
|
||||
urlIndex uint32
|
||||
position uint64
|
||||
}
|
||||
|
||||
func key(ns byte, url string) string { return string(ns) + url }
|
||||
|
||||
// NewWriter returns an empty Writer.
|
||||
func NewWriter() *Writer {
|
||||
return &Writer{byKey: map[string]*entry{}}
|
||||
}
|
||||
|
||||
// SetNoCompress stores every cluster uncompressed. Useful when the input is
|
||||
// already compressed or when a reader without zstd must open the file.
|
||||
func (w *Writer) SetNoCompress(v bool) { w.noCompress = v }
|
||||
|
||||
// AddContent adds a content entry. A later add with the same namespace and url
|
||||
// replaces the earlier one. An empty title defaults to the url.
|
||||
func (w *Writer) AddContent(namespace byte, url, title, mime string, data []byte) {
|
||||
if title == "" {
|
||||
title = url
|
||||
}
|
||||
if mime == "" {
|
||||
mime = "application/octet-stream"
|
||||
}
|
||||
w.put(&entry{namespace: namespace, url: url, title: title, mime: mime, data: data})
|
||||
}
|
||||
|
||||
// AddMetadata adds an 'M' namespace text entry, e.g. AddMetadata("Title", "...").
|
||||
func (w *Writer) AddMetadata(name, value string) {
|
||||
w.put(&entry{namespace: NamespaceMetadata, url: name, title: name, mime: "text/plain", data: []byte(value)})
|
||||
}
|
||||
|
||||
// AddRedirect adds a redirect from (namespace,url) to (targetNamespace,targetURL).
|
||||
func (w *Writer) AddRedirect(namespace byte, url, title string, targetNamespace byte, targetURL string) {
|
||||
if title == "" {
|
||||
title = url
|
||||
}
|
||||
w.put(&entry{namespace: namespace, url: url, title: title, redirect: true, targetKey: key(targetNamespace, targetURL)})
|
||||
}
|
||||
|
||||
// SetMainPage marks an entry as the archive's entry point.
|
||||
func (w *Writer) SetMainPage(namespace byte, url string) { w.mainKey = key(namespace, url) }
|
||||
|
||||
func (w *Writer) put(e *entry) {
|
||||
k := key(e.namespace, e.url)
|
||||
if old, ok := w.byKey[k]; ok {
|
||||
*old = *e // replace in place, keep slice order
|
||||
return
|
||||
}
|
||||
w.byKey[k] = e
|
||||
w.entries = append(w.entries, e)
|
||||
}
|
||||
|
||||
// plan holds the prebuilt sections of the file, ready to emit in order.
|
||||
type plan struct {
|
||||
hdr header
|
||||
mimeList []byte
|
||||
urlPtrs []byte
|
||||
titlePtrs []byte
|
||||
clusterPtrs []byte
|
||||
dirents [][]byte // URL order
|
||||
clusters [][]byte
|
||||
}
|
||||
|
||||
// WriteTo serialises the archive to out and returns the number of bytes written.
|
||||
func (w *Writer) WriteTo(out io.Writer) (int64, error) {
|
||||
p, err := w.buildPlan()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
sum := md5.New()
|
||||
mw := io.MultiWriter(out, sum)
|
||||
var n int64
|
||||
write := func(b []byte) error {
|
||||
m, err := mw.Write(b)
|
||||
n += int64(m)
|
||||
return err
|
||||
}
|
||||
for _, section := range append([][]byte{
|
||||
p.hdr.marshal(), p.mimeList, p.urlPtrs, p.titlePtrs, p.clusterPtrs,
|
||||
}, append(p.dirents, p.clusters...)...) {
|
||||
if err := write(section); err != nil {
|
||||
return n, err
|
||||
}
|
||||
}
|
||||
// The MD5 covers everything before it and is not itself hashed.
|
||||
m, err := out.Write(sum.Sum(nil))
|
||||
n += int64(m)
|
||||
return n, err
|
||||
}
|
||||
|
||||
func (w *Writer) buildPlan() (plan, error) {
|
||||
var p plan
|
||||
// 1. URL order: sort by <namespace><url>, assign indices.
|
||||
ents := make([]*entry, len(w.entries))
|
||||
copy(ents, w.entries)
|
||||
sort.Slice(ents, func(i, j int) bool {
|
||||
return key(ents[i].namespace, ents[i].url) < key(ents[j].namespace, ents[j].url)
|
||||
})
|
||||
index := make(map[string]uint32, len(ents))
|
||||
for i, e := range ents {
|
||||
e.urlIndex = uint32(i)
|
||||
index[key(e.namespace, e.url)] = uint32(i)
|
||||
}
|
||||
|
||||
// 2. Resolve redirect targets.
|
||||
for _, e := range ents {
|
||||
if !e.redirect {
|
||||
continue
|
||||
}
|
||||
ti, ok := index[e.targetKey]
|
||||
if !ok {
|
||||
return p, fmt.Errorf("zim: redirect %q points at missing target %q", key(e.namespace, e.url), e.targetKey)
|
||||
}
|
||||
e.targetIndex = ti
|
||||
}
|
||||
|
||||
// 3. MIME list (first-seen order over content entries).
|
||||
var mimes []string
|
||||
mimeIndex := map[string]uint16{}
|
||||
for _, e := range ents {
|
||||
if e.redirect {
|
||||
continue
|
||||
}
|
||||
if _, ok := mimeIndex[e.mime]; !ok {
|
||||
mimeIndex[e.mime] = uint16(len(mimes))
|
||||
mimes = append(mimes, e.mime)
|
||||
}
|
||||
e.mimeIdx = mimeIndex[e.mime]
|
||||
}
|
||||
p.mimeList = encodeMimeList(mimes)
|
||||
|
||||
// 4. Cluster packing: split text vs binary, cap each cluster, assign blobs.
|
||||
clusters := w.packClusters(ents)
|
||||
p.clusters = make([][]byte, len(clusters))
|
||||
for i, c := range clusters {
|
||||
p.clusters[i] = c.encode(w.noCompress)
|
||||
}
|
||||
|
||||
// 5. Directory entry bytes (URL order).
|
||||
p.dirents = make([][]byte, len(ents))
|
||||
for i, e := range ents {
|
||||
p.dirents[i] = e.encodeDirent()
|
||||
}
|
||||
|
||||
// 6. Layout: assign absolute positions.
|
||||
count := uint32(len(ents))
|
||||
pos := uint64(headerLen)
|
||||
mimeListPos := pos
|
||||
pos += uint64(len(p.mimeList))
|
||||
urlPtrPos := pos
|
||||
pos += 8 * uint64(count)
|
||||
titlePtrPos := pos
|
||||
pos += 4 * uint64(count)
|
||||
clusterPtrPos := pos
|
||||
pos += 8 * uint64(len(p.clusters))
|
||||
for i, e := range ents {
|
||||
e.position = pos
|
||||
pos += uint64(len(p.dirents[i]))
|
||||
}
|
||||
clusterPos := make([]uint64, len(p.clusters))
|
||||
for i := range p.clusters {
|
||||
clusterPos[i] = pos
|
||||
pos += uint64(len(p.clusters[i]))
|
||||
}
|
||||
checksumPos := pos
|
||||
|
||||
// 7. Pointer lists.
|
||||
p.urlPtrs = make([]byte, 8*count)
|
||||
for i, e := range ents {
|
||||
binary.LittleEndian.PutUint64(p.urlPtrs[8*i:], e.position)
|
||||
}
|
||||
p.clusterPtrs = make([]byte, 8*len(clusterPos))
|
||||
for i, cp := range clusterPos {
|
||||
binary.LittleEndian.PutUint64(p.clusterPtrs[8*i:], cp)
|
||||
}
|
||||
p.titlePtrs = encodeTitlePtrs(ents)
|
||||
|
||||
// 8. Header.
|
||||
p.hdr = header{
|
||||
uuid: deriveUUID(ents),
|
||||
articleCount: count,
|
||||
clusterCount: uint32(len(p.clusters)),
|
||||
urlPtrPos: urlPtrPos,
|
||||
titlePtrPos: titlePtrPos,
|
||||
clusterPtrPos: clusterPtrPos,
|
||||
mimeListPos: mimeListPos,
|
||||
mainPage: noMainPage,
|
||||
layoutPage: noMainPage,
|
||||
checksumPos: checksumPos,
|
||||
}
|
||||
if w.mainKey != "" {
|
||||
if mi, ok := index[w.mainKey]; ok {
|
||||
p.hdr.mainPage = mi
|
||||
}
|
||||
}
|
||||
return p, nil
|
||||
}
|
||||
|
||||
// clusterBuf accumulates blobs destined for one cluster.
|
||||
type clusterBuf struct {
|
||||
comp uint8
|
||||
blobs [][]byte
|
||||
size int
|
||||
}
|
||||
|
||||
func (w *Writer) packClusters(ents []*entry) []*clusterBuf {
|
||||
var clusters []*clusterBuf
|
||||
var curText, curBin *clusterBuf
|
||||
|
||||
closeIf := func(c **clusterBuf) {
|
||||
if *c != nil && (*c).size >= maxClusterContent {
|
||||
*c = nil
|
||||
}
|
||||
}
|
||||
add := func(cur **clusterBuf, comp uint8, e *entry) {
|
||||
if *cur == nil {
|
||||
*cur = &clusterBuf{comp: comp}
|
||||
clusters = append(clusters, *cur)
|
||||
}
|
||||
c := *cur
|
||||
e.cluster = uint32(indexOf(clusters, c))
|
||||
e.blob = uint32(len(c.blobs))
|
||||
c.blobs = append(c.blobs, e.data)
|
||||
c.size += len(e.data)
|
||||
}
|
||||
|
||||
for _, e := range ents {
|
||||
if e.redirect {
|
||||
continue
|
||||
}
|
||||
if isTextMime(e.mime) {
|
||||
add(&curText, compZstd, e)
|
||||
closeIf(&curText)
|
||||
} else {
|
||||
add(&curBin, compNone, e)
|
||||
closeIf(&curBin)
|
||||
}
|
||||
}
|
||||
return clusters
|
||||
}
|
||||
|
||||
func indexOf(cs []*clusterBuf, c *clusterBuf) int {
|
||||
for i := range cs {
|
||||
if cs[i] == c {
|
||||
return i
|
||||
}
|
||||
}
|
||||
return -1
|
||||
}
|
||||
|
||||
// encode renders a cluster: an info byte followed by the (optionally zstd)
|
||||
// data section, which is an offset table of (N+1) uint32 values then the N
|
||||
// concatenated blobs. Offsets are relative to the start of the data section.
|
||||
func (c *clusterBuf) encode(noCompress bool) []byte {
|
||||
n := len(c.blobs)
|
||||
tableLen := 4 * (n + 1)
|
||||
total := tableLen
|
||||
for _, b := range c.blobs {
|
||||
total += len(b)
|
||||
}
|
||||
data := make([]byte, tableLen, total)
|
||||
off := uint32(tableLen)
|
||||
binary.LittleEndian.PutUint32(data[0:], off)
|
||||
for i, b := range c.blobs {
|
||||
off += uint32(len(b))
|
||||
binary.LittleEndian.PutUint32(data[4*(i+1):], off)
|
||||
}
|
||||
for _, b := range c.blobs {
|
||||
data = append(data, b...)
|
||||
}
|
||||
|
||||
comp := c.comp
|
||||
if noCompress {
|
||||
comp = compNone
|
||||
}
|
||||
payload := data
|
||||
if comp == compZstd {
|
||||
payload = zstdEncode(data)
|
||||
} else {
|
||||
comp = compNone
|
||||
}
|
||||
out := make([]byte, 0, len(payload)+1)
|
||||
out = append(out, comp) // non-extended: bit 4 clear, uint32 offsets
|
||||
return append(out, payload...)
|
||||
}
|
||||
|
||||
func (e *entry) encodeDirent() []byte {
|
||||
le := binary.LittleEndian
|
||||
var head []byte
|
||||
if e.redirect {
|
||||
head = make([]byte, 12)
|
||||
le.PutUint16(head[0:], redirectEntry)
|
||||
head[3] = e.namespace
|
||||
le.PutUint32(head[8:], e.targetIndex)
|
||||
} else {
|
||||
head = make([]byte, 16)
|
||||
le.PutUint16(head[0:], e.mimeIdx)
|
||||
head[3] = e.namespace
|
||||
le.PutUint32(head[8:], e.cluster)
|
||||
le.PutUint32(head[12:], e.blob)
|
||||
}
|
||||
out := append(head, e.url...)
|
||||
out = append(out, 0)
|
||||
out = append(out, e.title...)
|
||||
return append(out, 0)
|
||||
}
|
||||
|
||||
func encodeMimeList(mimes []string) []byte {
|
||||
var b []byte
|
||||
for _, m := range mimes {
|
||||
b = append(b, m...)
|
||||
b = append(b, 0)
|
||||
}
|
||||
return append(b, 0) // terminating empty string
|
||||
}
|
||||
|
||||
func encodeTitlePtrs(ents []*entry) []byte {
|
||||
order := make([]*entry, len(ents))
|
||||
copy(order, ents)
|
||||
sort.Slice(order, func(i, j int) bool {
|
||||
ti := string(order[i].namespace) + order[i].title
|
||||
tj := string(order[j].namespace) + order[j].title
|
||||
if ti != tj {
|
||||
return ti < tj
|
||||
}
|
||||
return order[i].urlIndex < order[j].urlIndex
|
||||
})
|
||||
b := make([]byte, 4*len(order))
|
||||
for i, e := range order {
|
||||
binary.LittleEndian.PutUint32(b[4*i:], e.urlIndex)
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// deriveUUID makes the file deterministic: identical input yields an identical
|
||||
// archive. It hashes every entry's key and content, so repacking the same
|
||||
// mirror is idempotent and diffable.
|
||||
func deriveUUID(ents []*entry) [16]byte {
|
||||
h := md5.New()
|
||||
var n [8]byte
|
||||
for _, e := range ents {
|
||||
h.Write([]byte(key(e.namespace, e.url)))
|
||||
binary.LittleEndian.PutUint64(n[:], uint64(len(e.data)))
|
||||
h.Write(n[:])
|
||||
h.Write(e.data)
|
||||
}
|
||||
var u [16]byte
|
||||
copy(u[:], h.Sum(nil))
|
||||
return u
|
||||
}
|
||||
+136
@@ -0,0 +1,136 @@
|
||||
package zim
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/md5"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// buildSample writes a small archive exercising text, binary, metadata, a
|
||||
// redirect, and a main page, and returns its bytes.
|
||||
func buildSample(t *testing.T, noCompress bool) []byte {
|
||||
t.Helper()
|
||||
w := NewWriter()
|
||||
w.SetNoCompress(noCompress)
|
||||
w.AddContent(NamespaceContent, "index.html", "Home", "text/html",
|
||||
[]byte("<h1>Home</h1>"+strings.Repeat(" word", 500)))
|
||||
w.AddContent(NamespaceContent, "about/index.html", "About", "text/html",
|
||||
[]byte("<h1>About</h1>"))
|
||||
w.AddContent(NamespaceContent, "_kage/h/logo.png", "", "image/png",
|
||||
[]byte{0x89, 'P', 'N', 'G', 0, 1, 2, 3, 4, 5})
|
||||
w.AddMetadata("Title", "Sample")
|
||||
w.AddMetadata("Language", "eng")
|
||||
w.AddRedirect(NamespaceWellKnown, "mainPage", "Main", NamespaceContent, "index.html")
|
||||
w.SetMainPage(NamespaceContent, "index.html")
|
||||
|
||||
var buf bytes.Buffer
|
||||
n, err := w.WriteTo(&buf)
|
||||
if err != nil {
|
||||
t.Fatalf("WriteTo: %v", err)
|
||||
}
|
||||
if int(n) != buf.Len() {
|
||||
t.Fatalf("WriteTo reported %d bytes, buffer has %d", n, buf.Len())
|
||||
}
|
||||
return buf.Bytes()
|
||||
}
|
||||
|
||||
func TestRoundTrip(t *testing.T) {
|
||||
for _, noCompress := range []bool{false, true} {
|
||||
data := buildSample(t, noCompress)
|
||||
r, err := NewReader(bytes.NewReader(data), int64(len(data)))
|
||||
if err != nil {
|
||||
t.Fatalf("NewReader (noCompress=%v): %v", noCompress, err)
|
||||
}
|
||||
|
||||
// Content round-trips with the right mime.
|
||||
home, err := r.Get(NamespaceContent, "index.html")
|
||||
if err != nil {
|
||||
t.Fatalf("Get home: %v", err)
|
||||
}
|
||||
if !strings.HasPrefix(string(home.Data), "<h1>Home</h1>") {
|
||||
t.Errorf("home content wrong: %.20q", home.Data)
|
||||
}
|
||||
if home.MimeType != "text/html" {
|
||||
t.Errorf("home mime = %q", home.MimeType)
|
||||
}
|
||||
|
||||
// Binary blob survives byte-for-byte.
|
||||
logo, err := r.Get(NamespaceContent, "_kage/h/logo.png")
|
||||
if err != nil {
|
||||
t.Fatalf("Get logo: %v", err)
|
||||
}
|
||||
if !bytes.Equal(logo.Data, []byte{0x89, 'P', 'N', 'G', 0, 1, 2, 3, 4, 5}) {
|
||||
t.Errorf("logo bytes wrong: %v", logo.Data)
|
||||
}
|
||||
if logo.MimeType != "image/png" {
|
||||
t.Errorf("logo mime = %q", logo.MimeType)
|
||||
}
|
||||
|
||||
// Metadata.
|
||||
meta, err := r.Get(NamespaceMetadata, "Title")
|
||||
if err != nil || string(meta.Data) != "Sample" {
|
||||
t.Errorf("metadata Title = %q, %v", meta.Data, err)
|
||||
}
|
||||
|
||||
// Redirect resolves to the target's content.
|
||||
red, err := r.Get(NamespaceWellKnown, "mainPage")
|
||||
if err != nil {
|
||||
t.Fatalf("Get redirect: %v", err)
|
||||
}
|
||||
if !strings.HasPrefix(string(red.Data), "<h1>Home</h1>") {
|
||||
t.Errorf("redirect did not resolve to home: %.20q", red.Data)
|
||||
}
|
||||
|
||||
// Main page.
|
||||
mp, err := r.MainPage()
|
||||
if err != nil {
|
||||
t.Fatalf("MainPage: %v", err)
|
||||
}
|
||||
if !strings.HasPrefix(string(mp.Data), "<h1>Home</h1>") {
|
||||
t.Errorf("main page wrong: %.20q", mp.Data)
|
||||
}
|
||||
|
||||
// Misses error.
|
||||
if _, err := r.Get(NamespaceContent, "nope.html"); err == nil {
|
||||
t.Error("expected miss to error")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestChecksum(t *testing.T) {
|
||||
data := buildSample(t, false)
|
||||
if len(data) < 16 {
|
||||
t.Fatal("archive too short")
|
||||
}
|
||||
body, sum := data[:len(data)-16], data[len(data)-16:]
|
||||
want := md5.Sum(body)
|
||||
if !bytes.Equal(sum, want[:]) {
|
||||
t.Errorf("trailing MD5 does not match body hash")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeterministic(t *testing.T) {
|
||||
a := buildSample(t, false)
|
||||
b := buildSample(t, false)
|
||||
if !bytes.Equal(a, b) {
|
||||
t.Error("same input produced different archives; packing is not deterministic")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMagicAndHeader(t *testing.T) {
|
||||
data := buildSample(t, false)
|
||||
h, err := parseHeader(data[:headerLen])
|
||||
if err != nil {
|
||||
t.Fatalf("parseHeader: %v", err)
|
||||
}
|
||||
if h.checksumPos != uint64(len(data)-16) {
|
||||
t.Errorf("checksumPos = %d, want %d", h.checksumPos, len(data)-16)
|
||||
}
|
||||
if h.articleCount != 6 {
|
||||
t.Errorf("articleCount = %d, want 6", h.articleCount)
|
||||
}
|
||||
if h.mainPage == noMainPage {
|
||||
t.Error("main page not set in header")
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user