Compare commits

...

19 Commits

Author SHA1 Message Date
Tam Nguyen Duc 24b5bff396 Merge pull request #4 from tamnd/feat/pack-app
Pack double-click apps: kage pack --app (v0.2.0)
2026-06-15 12:55:07 +07:00
Duc-Tam Nguyen 7cee00c331 Cut the v0.2.0 release notes
Give the double-click app work its own 0.2.0 section in the changelog,
above the released 0.1.2, and update the compare links. Summarise it on
the docs release-notes page: kage pack --app wraps the viewer in a
desktop app, the release ships a GUI-subsystem Windows base, and packing
detects the base binary's target OS from its executable header.
2026-06-15 12:50:46 +07:00
Duc-Tam Nguyen b5f32b7b2b Make the desktop app a --app flag instead of a format
Wrapping a packed viewer in a .app or .AppImage was its own --format app
value, parallel to zim and binary. But an app is really just the binary
format with a bundle around it, so a separate format meant duplicating the
base/icon handling and made the three formats feel like an awkward choice.

Turn it into a --app flag that builds on the binary format. It composes
with --base (including a webview base) and --icon, while --format stays
zim or binary. The bundle builders are unchanged; only the CLI surface
moves.
2026-06-15 12:49:39 +07:00
Duc-Tam Nguyen 8b8331c435 Document double-click app bundles and the Windows GUI base
README and the packing guide gain a double-click app section covering the macOS
.app, the Linux .AppDir/.AppImage, favicon icons, and the windows-gui base. The
changelog and docs release notes record the new format under Unreleased.
2026-06-15 12:49:39 +07:00
Duc-Tam Nguyen 05a87960d1 Ship a GUI-subsystem Windows base and cover it in CI
A second goreleaser build links a Windows binary for the GUI subsystem
(-H windowsgui) and ships it as kage_<version>_windows-gui_<arch>.zip, scoped so
the package managers still install the console build. Packing a viewer onto this
base gives a double-click .exe with no console behind it. A CI job cross-compiles
the windowsgui link so it cannot rot.
2026-06-15 12:48:52 +07:00
Duc-Tam Nguyen a40da25b8c Add kage pack --format app for a double-click viewer
The new format dispatches on the base's target OS: a .app on macOS, an .AppDir
(plus an .AppImage when appimagetool is present) on Linux, and a friendly redirect
to --format binary on Windows, where the .exe is already the app. The icon comes
from the mirror's favicon by default and can be overridden with --icon.
2026-06-15 12:48:52 +07:00
Duc-Tam Nguyen e3d3c48ce0 Build double-click app bundles for macOS and Linux
Share the base++zim++trailer assembly behind a small helper, then add two bundle
builders on top: BuildApp writes a macOS .app (Info.plist, the viewer under
Contents/MacOS, an .icns in Resources), and BuildAppDir writes an AppImage-style
.AppDir (AppRun, a Terminal=false .desktop launcher, a PNG icon). Both keep the
trailer at the tail so Embedded still finds the archive at runtime.
2026-06-15 12:48:52 +07:00
Duc-Tam Nguyen 09543d1e11 Add an ICNS encoder and favicon discovery for app icons
A pure-Go .icns writer (PNG-embedded entries from 16 to 1024) and a finder that
digs the site's favicon out of a cloned mirror, preferring a large apple-touch
icon and unwrapping a PNG-based .ico. These feed the bundle icon for the upcoming
app formats. Adds golang.org/x/image for high-quality resampling.
2026-06-15 12:48:52 +07:00
Tam Nguyen Duc 0cffea568a Merge pull request #13 from tamnd/release/v0.1.2
Cut the v0.1.2 release notes
2026-06-15 12:47:49 +07:00
Duc-Tam Nguyen b5fb185b86 Cut the v0.1.2 release notes
Move the Unreleased entries into a 0.1.2 section in the changelog and
summarise the release on the docs release-notes page: the Chrome sandbox
now stays on by default, asset downloads retry on a transient failure,
crawl errors report a clear reason and provenance, and the container
image runs again.
2026-06-15 12:45:48 +07:00
Tam Nguyen Duc d19433e412 Merge pull request #12 from tamnd/harden/browser-sandbox-and-errors
Keep the Chrome sandbox on by default, report crawl errors clearly, and fix the container image
2026-06-15 12:35:32 +07:00
Duc-Tam Nguyen ebe66ab535 Make the container image actually clone (issue #7)
Two failures stopped a docker run from producing anything. Chrome
aborted on launch with 'chrome_crashpad_handler: --database is
required', because its crash reporter cannot start in a minimal
container, so disable the crash reporter on the container launch path.
kage never uploads Chrome crash dumps, so nothing is lost.

The image also created the kage user without a home directory, so HOME
was an unwritable /home/kage. kage writes its default output and resume
state under $HOME/data/kage and Chrome puts its profile and crash
database under HOME too, so both failed with a permission error and the
mounted /out volume captured nothing. Point HOME at the /out volume so
all of it lands somewhere writable that the mount picks up.
2026-06-15 12:33:19 +07:00
Duc-Tam Nguyen d59b7e1dff Set IN_DOCKER on the Ubuntu CI test job
The GitHub Ubuntu runner disables unprivileged user namespaces with
AppArmor, so Chrome's sandbox cannot initialize and the new secure
default (sandbox on) makes Chrome refuse to start. IN_DOCKER is the
documented escape hatch for that case, so set it on the Ubuntu leg of
the test job. It also exercises the container code path. macOS does not
need the flag, so it stays empty there.
2026-06-15 12:30:07 +07:00
Duc-Tam Nguyen d59c85afc8 Report crawl errors clearly and retry transient ones
The crawl printed asset failures as "asset error <url>: status 403 for
<url>", repeating the URL and saying nothing about which page wanted the
file or whether the failure was worth worrying about. The final summary then
collapsed everything into a single error count.

Give failures a classified reason (HTTP 403 Forbidden, timed out, ...), name
the page that referenced the asset, and list what went wrong in the summary
instead of only counting it. Failures are collected during the run and capped
so a broken site cannot grow the list without bound.

Retry transient failures (403/429, 5xx, network blips) with a short backoff.
Bot-protection in front of a site often rejects the first request of a burst
but serves a retry fine, which is exactly what cost us stylesheets on a busy
crawl. Permanent failures (404, 401, ...) are not retried.
2026-06-15 12:25:05 +07:00
Duc-Tam Nguyen dab6c11ea8 Keep the Chrome sandbox on by default
kage launched Chrome with --no-sandbox unconditionally, which turns off the
browser's main security boundary for every run, including ordinary desktop
use where the sandbox works fine. Since kage renders pages from the open web,
a renderer exploit could then reach the host. Reported in #10.

Keep the sandbox on by default and drop it only where it genuinely cannot
initialize: inside a container, or when running as root (Chrome refuses to
start a sandbox as root). Containers are detected from IN_DOCKER or the
/.dockerenv marker, and there kage also sets --disable-dev-shm-usage because
the default 64 MB /dev/shm is too small for the renderer on large pages.
Whenever the sandbox is dropped kage says so on stderr, so it is never silent.

Thanks to Dimitrios Prasakis for the report and to the commenter on Hacker
News who suggested the IN_DOCKER opt-in.
2026-06-15 12:24:55 +07:00
Tam Nguyen Duc 01e75b87ec Merge pull request #3 from tamnd/feat/pack-cross-os
Make cross-platform packing robust and cover the webview build in CI
2026-06-15 00:24:11 +07:00
Duc-Tam Nguyen 5d8057473b Cover all packages in gofmt and compile the webview build in CI
The gofmt gate listed packages by hand and missed pack, zim, and viewer, so a
formatting slip in the newer code would sail through. Check the whole module
instead. Add a macOS job that compiles the -tags webview viewer, the cgo path
the pure-Go CI never builds; the viewer code is identical across platforms, so
one compile guards it. Also note the new base-OS detection in the docs.
2026-06-15 00:17:02 +07:00
Duc-Tam Nguyen d81b90b38c Detect the base binary's OS when packing a cross-platform viewer
Packing with --base pointed at a kage built for another OS used to guess the
target from the file name: a base ending in .exe meant Windows, anything else
meant the host. That misfired in both directions. A Windows base without the
.exe suffix produced a viewer with no extension that will not run on Windows,
and an --out name that dropped .exe made the run hint print a macOS quarantine
note for a Windows file.

Sniff the base's executable header (ELF, PE, Mach-O) instead, so the target OS
comes from the bytes rather than the name. A Windows viewer now always gets a
.exe suffix, and the run hint names the real target and only mentions Gatekeeper
for actual macOS viewers.
2026-06-15 00:15:16 +07:00
Tam Nguyen Duc c4895d5b92 Merge pull request #2 from tamnd/feat/pack
Add kage pack and kage open: a mirror packed into one file
2026-06-15 00:05:15 +07:00
29 changed files with 2082 additions and 66 deletions
+44 -1
View File
@@ -38,7 +38,7 @@ jobs:
id: chrome
- name: gofmt
run: |
unformatted=$(gofmt -s -l asset browser cli clone cmd robots sanitize urlx)
unformatted=$(gofmt -s -l .)
if [ -n "$unformatted" ]; then
echo "These files need gofmt -s -w:"
echo "$unformatted"
@@ -48,9 +48,15 @@ jobs:
run: go vet ./...
- name: build
run: go build ./...
# The GitHub Ubuntu runner disables unprivileged user namespaces (AppArmor),
# so Chrome's sandbox cannot initialize there and the secure default would
# make it refuse to start. IN_DOCKER is kage's documented escape hatch for
# exactly that case, and setting it here also exercises the container path.
# macOS does not need it, so it stays empty on that leg.
- name: test
env:
KAGE_CHROME: ${{ steps.chrome.outputs.chrome-path }}
IN_DOCKER: ${{ matrix.os == 'ubuntu-latest' && '1' || '' }}
run: go test -race -count=1 -coverprofile=coverage.out ./...
- name: coverage summary
if: matrix.os == 'ubuntu-latest'
@@ -101,3 +107,40 @@ jobs:
run: |
go mod tidy
git diff --exit-code -- go.mod go.sum
# Compile the optional native-window viewer (-tags webview, cgo) so that path
# keeps building. The default CI build is pure Go and never touches it. The
# viewer code is the same Go on every OS, only the system WebView library
# differs, so a macOS compile (WebKit ships in the SDK) catches our
# regressions without the WebKitGTK version juggling Linux runners need. It is
# build-only: actually opening a window needs a display.
webview:
runs-on: macos-latest
steps:
- uses: actions/checkout@v5
- uses: actions/setup-go@v6
with:
go-version-file: go.mod
check-latest: true
cache: true
- name: build webview viewer
run: CGO_ENABLED=1 go build -tags webview ./cmd/kage
# Cross-compile the GUI-subsystem Windows base the release ships for
# double-click viewers (kage pack --base). A change that breaks the
# -H windowsgui link is caught here instead of at release time. Pure Go, so it
# cross-compiles from Linux with no extra toolchain.
windows-gui:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/setup-go@v6
with:
go-version-file: go.mod
check-latest: true
cache: true
- name: build windowsgui base
env:
GOOS: windows
CGO_ENABLED: "0"
run: go build -ldflags "-H=windowsgui" -o kage-windowsgui.exe ./cmd/kage
+48 -3
View File
@@ -43,9 +43,35 @@ builds:
- freebsd_amd64
- freebsd_arm64
# A second Windows build linked for the GUI subsystem (-H windowsgui). It is
# the same pure-Go kage, but a viewer packed onto it shows only its window when
# double-clicked, with no console flashing behind it. Users point `kage pack
# --base` at this to build a clean double-click Windows app. The default build
# above stays console-attached so the CLI still prints clone progress.
- id: kage-gui
binary: kage
main: ./cmd/kage
env:
- CGO_ENABLED=0
flags:
- -trimpath
ldflags:
- -s -w
- -H=windowsgui
- -X github.com/tamnd/kage/cli.Version={{ .Version }}
- -X github.com/tamnd/kage/cli.Commit={{ .ShortCommit }}
- -X github.com/tamnd/kage/cli.Date={{ .CommitDate }}
mod_timestamp: "{{ .CommitTimestamp }}"
targets:
- windows_amd64
- windows_arm64
archives:
# tar.gz everywhere except a zip on Windows.
# tar.gz everywhere except a zip on Windows. Scoped to the console build so
# this is the archive every package manager installs.
- id: default
ids:
- kage
name_template: "kage_{{ .Version }}_{{ .Os }}_{{ .Arch }}{{ with .Arm }}v{{ . }}{{ end }}"
format_overrides:
- goos: windows
@@ -54,6 +80,18 @@ archives:
- LICENSE
- README.md
# A separate zip for the GUI-subsystem Windows binary, named so it cannot be
# confused with the console build. It is a base to pack against, not something
# to run directly, so no package manager points at it.
- id: windows-gui
ids:
- kage-gui
name_template: "kage_{{ .Version }}_windows-gui_{{ .Arch }}"
formats: [zip]
files:
- LICENSE
- README.md
nfpms:
# One nfpm definition emits the deb, rpm, and apk for every Linux build. kage
# is a user command, not a daemon, so there is no unit file and no
@@ -61,6 +99,8 @@ nfpms:
# dependency the user supplies (or the container image bundles).
- id: linux-packages
package_name: kage
ids:
- kage
file_name_template: "{{ .ConventionalFileName }}"
vendor: tamnd
homepage: https://github.com/tamnd/kage
@@ -106,6 +146,8 @@ homebrew_casks:
# HOMEBREW_TAP_GITHUB_TOKEN (a PAT with write to tamnd/homebrew-tap) is set,
# so a tokenless release still writes the cask into dist for inspection.
- name: kage
ids:
- default
repository:
owner: tamnd
name: homebrew-tap
@@ -119,8 +161,11 @@ homebrew_casks:
email: tamnd87@gmail.com
scoops:
# Scoop manifest for Windows, pushed to the bucket repository.
- repository:
# Scoop manifest for Windows, pushed to the bucket repository. It installs the
# console build (the CLI), not the GUI base.
- ids:
- default
repository:
owner: tamnd
name: scoop-bucket
token: '{{ envOrDefault "SCOOP_BUCKET_GITHUB_TOKEN" "" }}'
+47 -1
View File
@@ -6,6 +6,50 @@ All notable changes to kage are recorded here. The format follows
## [Unreleased]
## [0.2.0] - 2026-06-15
### Added
- `kage pack --app` wraps the packed viewer in a double-click desktop app with
the site's favicon as the icon. The flag builds on the binary format, so it
composes with `--base` (including a `webview` base) and `--icon`. On macOS it
writes a `.app` bundle (`Info.plist`, the viewer under `Contents/MacOS`, and an
`.icns` generated from the favicon); on Linux, with a Linux `--base`, it writes
an AppImage-style `.AppDir` and folds it into a single `.AppImage` when
`appimagetool` is installed. The icon is found in the mirror automatically
(preferring a large `apple-touch-icon.png`, then `favicon.png` or a PNG-based
`favicon.ico`) and can be overridden with `--icon`.
- The release now ships a GUI-subsystem Windows base,
`kage_<version>_windows-gui_<arch>.zip`. Packing a viewer onto it with
`--format binary --base` produces a `.exe` that opens with no console window
behind it, the Windows equivalent of the `.app` double-click experience.
### Changed
- Cross-platform packing detects the base binary's target OS from its executable
header (ELF, PE, or Mach-O) rather than its file name, so a Windows viewer
always gets a `.exe` suffix and the run hint names the right platform even when
the base is named without one.
## [0.1.2] - 2026-06-15
### Security
- Chrome now keeps its sandbox on by default. It was previously launched with `--no-sandbox` unconditionally, which removed Chrome's main line of defense when rendering pages from the open web (reported in #10). The sandbox is now dropped only where it genuinely cannot run: inside a container, or when running as root, and the choice is logged so it is never silent.
### Added
- Container-aware Chrome flags. kage detects a container from the `IN_DOCKER` environment variable or a `/.dockerenv` marker and, only there, drops the sandbox and adds `--disable-dev-shm-usage` (the default 64 MB `/dev/shm` is too small for Chrome on large pages). Outside a container the faster shared memory is left in place.
- Asset downloads retry on a transient failure (a 403/429, a 5xx, or a network blip) with a short backoff, recovering files that bot-protection rejects on the first request of a burst. Permanent failures (404, 401, ...) are not retried.
### Changed
- Clearer crawl error reporting. Each failure is logged with a classified reason (`HTTP 403 Forbidden`, `timed out`, ...), the URL, and the page that referenced it, and the end-of-run summary lists what went wrong instead of printing only a count.
### Fixed
- The container image now runs. Chrome aborted on launch with `chrome_crashpad_handler: --database is required`, so kage disables Chrome's crash reporter inside a container, and the `kage` user now has a writable home (the mounted `/out` volume) so the default output, resume state, and Chrome's profile no longer fail with a permission error (issue #7).
## [0.1.1] - 2026-06-14
### Added
@@ -65,6 +109,8 @@ can browse offline, with every script stripped out.
a multi-arch container image on GHCR (Chromium bundled), checksums, SBOMs, and
a cosign signature, all cut from one version tag by GoReleaser.
[Unreleased]: https://github.com/tamnd/kage/compare/v0.1.1...HEAD
[Unreleased]: https://github.com/tamnd/kage/compare/v0.2.0...HEAD
[0.2.0]: https://github.com/tamnd/kage/compare/v0.1.2...v0.2.0
[0.1.2]: https://github.com/tamnd/kage/compare/v0.1.1...v0.1.2
[0.1.1]: https://github.com/tamnd/kage/compare/v0.1.0...v0.1.1
[0.1.0]: https://github.com/tamnd/kage/releases/tag/v0.1.0
+8 -1
View File
@@ -29,7 +29,14 @@ WORKDIR /out
# Point kage at the bundled Chromium and write mirrors under /out by default:
#
# docker run -v "$PWD/out:/out" ghcr.io/tamnd/kage clone example.com
ENV KAGE_CHROME=/usr/bin/chromium-browser
#
# The kage user has no home directory of its own, so HOME points at the mounted
# /out volume. That keeps two things writable: kage's default output and resume
# state (it lands under $HOME/data/kage), and Chrome's profile and crash
# database. Without this both fail with a permission error in the container
# (issue #7), and the mounted volume captures nothing.
ENV KAGE_CHROME=/usr/bin/chromium-browser \
HOME=/out
VOLUME ["/out"]
+29 -3
View File
@@ -8,7 +8,7 @@
**kage** (影, "shadow") clones a website into a folder you can browse offline, with every script stripped out. It opens each page in real headless Chrome, waits for the page to settle, snapshots the DOM a human would have seen, then deletes all the JavaScript and pulls the CSS, images, and fonts down to local paths. What lands on disk looks like the live site and runs no code.
[Install](#install) • [Quick start](#quick-start) • [Commands](#commands) • [Clone](#clone) • [Pack](#pack-it-into-one-file) • [Native window](#a-real-window-not-a-browser-tab) • [How it works](#how-it-works)
[Install](#install) • [Quick start](#quick-start) • [Commands](#commands) • [Clone](#clone) • [Pack](#pack-it-into-one-file) • [Double-click app](#a-double-click-app) • [Native window](#a-real-window-not-a-browser-tab) • [How it works](#how-it-works)
![kage cloning paulgraham.com, packing it into one file, and serving it back offline](docs/static/demo.gif)
@@ -65,7 +65,7 @@ kage pack paulgraham.com --format binary -o paulgraham
| --- | --- |
| `kage clone <url>` | render a site in headless Chrome and write a browsable, script-free mirror |
| `kage serve [dir]` | preview a cloned folder over a local HTTP server |
| `kage pack <mirror-dir>` | collapse a mirror into one ZIM archive, or a self-contained viewer binary |
| `kage pack <mirror-dir>` | collapse a mirror into one ZIM archive, a self-contained viewer binary, or a double-click app |
| `kage open <file.zim>` | serve a packed ZIM back for offline reading |
## Clone
@@ -149,7 +149,7 @@ kage pack paulgraham.com --format binary -o paulgraham
./paulgraham
```
The appended archive is platform-independent; only the base executable carries the architecture. By default kage appends to itself, so you get a viewer for the machine you ran it on. Point `--base` at a kage built for another OS to produce a viewer for that platform from your own machine:
The appended archive is platform-independent; only the base executable carries the architecture. By default kage appends to itself, so you get a viewer for the machine you ran it on. Point `--base` at a kage built for another OS (grab one from a [release](https://github.com/tamnd/kage/releases); every platform ships one) to produce a viewer for that platform from your own machine. kage reads the base's executable header to figure out the target, so a Windows viewer automatically gets a `.exe` name:
```bash
# Sitting on a Mac, build a Windows viewer
@@ -158,6 +158,32 @@ kage pack paulgraham.com --format binary --base kage-windows-amd64.exe # -> pa
The trade is size. The binary carries a whole kage, so it weighs around 13 MiB plus the site no matter how small the mirror is. When you only need the content, the ZIM is far leaner.
### A double-click app
A bare binary is great from a terminal, but double-click it in a file manager and the experience is rough: macOS opens a Terminal window behind the site, and on Windows a console flashes up next to it. Add `--app` and kage wraps the same viewer in a proper desktop app so a double-click just opens the site, no terminal, with the mirror's own favicon as the icon.
On macOS you get a real `.app` bundle:
```bash
kage pack paulgraham.com --app # -> paulgraham.app
open paulgraham.app # or double-click it in Finder
```
On Linux, point `--base` at a Linux kage and you get an [AppImage](https://appimage.org)-style `.AppDir` with a `.desktop` launcher (`Terminal=false`, so no console). If [`appimagetool`](https://github.com/AppImage/appimagetool) is installed, kage folds it into a single double-clickable `.AppImage` for you:
```bash
kage pack paulgraham.com --app --base kage-linux-amd64 # -> paulgraham.AppDir (+ .AppImage)
```
kage finds the icon by digging the favicon out of the mirror (it prefers a large `apple-touch-icon.png` and falls back to `favicon.ico`); pass `--icon some.png` to override it. Pair `--app` with a `webview` base (below) and the double-click opens a native window instead of the browser, which is the full "it's an app" effect.
Windows needs no bundle, because there a single `.exe` already is the app. The catch is the console window. The release ships a `kage_<version>_windows-gui_<arch>.zip` whose binary is linked for the GUI subsystem, so a viewer packed onto it opens with no console behind it:
```bash
# Build a console-free Windows viewer (from any OS)
kage pack paulgraham.com --format binary --base kage-windows-gui-amd64.exe # -> paulgraham.exe
```
## A real window, not a browser tab
By default a packed binary opens your system browser, which means the site shows up as yet another tab, address bar and all, next to the 47 you already have open. Build kage with the `webview` tag and it opens the site in its own window instead, backed by the operating system's WebView (WKWebView on macOS, WebView2 on Windows, WebKitGTK on Linux). Paul Graham's essays, offline, in something that looks and feels like a real app:
+81 -1
View File
@@ -2,6 +2,7 @@ package asset
import (
"context"
"errors"
"fmt"
"io"
"net/http"
@@ -18,6 +19,7 @@ type Downloader struct {
Client *http.Client
UserAgent string
MaxBytes int64 // per-asset cap; 0 = unlimited
Retries int // extra attempts for a transient failure (0 = try once)
}
// NewDownloader builds a Downloader with a sane client and the given timeout.
@@ -26,6 +28,10 @@ func NewDownloader(userAgent string, timeout time.Duration, maxBytes int64) *Dow
Client: &http.Client{Timeout: timeout},
UserAgent: userAgent,
MaxBytes: maxBytes,
// A few sites (and the bot-protection in front of them) reject the first
// request of a burst with a 403 or 429 but serve a retry fine, so give
// transient failures a couple of extra tries before giving up.
Retries: 3,
}
}
@@ -36,9 +42,55 @@ type Result struct {
IsCSS bool
}
// StatusError reports a non-2xx HTTP response. It carries the code so callers
// can render a clear message ("HTTP 403 Forbidden") and decide whether a retry
// is worthwhile, without the URL baked in (the caller already has it).
type StatusError struct {
Code int
}
func (e *StatusError) Error() string {
if t := http.StatusText(e.Code); t != "" {
return fmt.Sprintf("HTTP %d %s", e.Code, t)
}
return fmt.Sprintf("HTTP %d", e.Code)
}
// Get fetches u, sending referer as the Referer header. It reads at most
// MaxBytes and reports whether the body is CSS (so the caller can rewrite it).
// A transient failure (a 403/429/5xx or a network blip) is retried with a short
// backoff up to Retries times.
func (d *Downloader) Get(ctx context.Context, u *url.URL, referer string) (*Result, error) {
attempts := d.Retries + 1
if attempts < 1 {
attempts = 1
}
var lastErr error
for i := 0; i < attempts; i++ {
if err := ctx.Err(); err != nil {
return nil, err
}
if i > 0 {
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(backoff(i)):
}
}
res, err := d.try(ctx, u, referer)
if err == nil {
return res, nil
}
lastErr = err
if !transient(err) {
break
}
}
return nil, lastErr
}
// try performs a single fetch attempt.
func (d *Downloader) try(ctx context.Context, u *url.URL, referer string) (*Result, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u.String(), nil)
if err != nil {
return nil, err
@@ -55,7 +107,7 @@ func (d *Downloader) Get(ctx context.Context, u *url.URL, referer string) (*Resu
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("status %d for %s", resp.StatusCode, u)
return nil, &StatusError{Code: resp.StatusCode}
}
var r io.Reader = resp.Body
if d.MaxBytes > 0 {
@@ -73,6 +125,34 @@ func (d *Downloader) Get(ctx context.Context, u *url.URL, referer string) (*Resu
}, nil
}
// backoff returns the pause before retry attempt i (1-based): 500ms, 1s, 2s, …
func backoff(i int) time.Duration {
d := 500 * time.Millisecond << (i - 1)
if max := 5 * time.Second; d > max {
d = max
}
return d
}
// transient reports whether an error is worth retrying. Bot-protection statuses
// (403/429), request-timeout and too-early (408/425), and 5xx server errors are
// transient; other 4xx (404, 401, 410, …) are permanent. A network error is
// retried, but a cancelled or expired context is not.
func transient(err error) bool {
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
return false
}
var se *StatusError
if errors.As(err, &se) {
switch se.Code {
case http.StatusForbidden, http.StatusRequestTimeout, http.StatusTooEarly, http.StatusTooManyRequests:
return true
}
return se.Code >= 500
}
return true
}
// isCSS reports whether a response is a stylesheet, by content-type or by a
// .css path when the server sends no useful type.
func isCSS(contentType string, u *url.URL) bool {
+112
View File
@@ -0,0 +1,112 @@
package asset
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"net/url"
"sync/atomic"
"testing"
"time"
)
func TestStatusErrorMessage(t *testing.T) {
cases := map[int]string{
403: "HTTP 403 Forbidden",
404: "HTTP 404 Not Found",
999: "HTTP 999",
}
for code, want := range cases {
if got := (&StatusError{Code: code}).Error(); got != want {
t.Errorf("StatusError{%d} = %q; want %q", code, got, want)
}
}
}
func TestGetRetriesTransientThenSucceeds(t *testing.T) {
var hits int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// 403 on the first try (like bot-protection), then serve the file.
if atomic.AddInt32(&hits, 1) == 1 {
w.WriteHeader(http.StatusForbidden)
return
}
w.Header().Set("Content-Type", "text/css")
_, _ = w.Write([]byte("body{}"))
}))
defer srv.Close()
d := NewDownloader("kage-test", 5*time.Second, 0)
u, _ := url.Parse(srv.URL + "/style.css")
res, err := d.Get(context.Background(), u, "")
if err != nil {
t.Fatalf("Get after retry: %v", err)
}
if !res.IsCSS || string(res.Body) != "body{}" {
t.Errorf("unexpected result: css=%v body=%q", res.IsCSS, res.Body)
}
if hits < 2 {
t.Errorf("expected a retry; server saw %d hits", hits)
}
}
func TestGetDoesNotRetryPermanent(t *testing.T) {
var hits int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
atomic.AddInt32(&hits, 1)
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
d := NewDownloader("kage-test", 5*time.Second, 0)
u, _ := url.Parse(srv.URL + "/missing.png")
_, err := d.Get(context.Background(), u, "")
var se *StatusError
if !errors.As(err, &se) || se.Code != 404 {
t.Fatalf("got %v; want StatusError 404", err)
}
if hits != 1 {
t.Errorf("404 should not be retried; server saw %d hits", hits)
}
}
func TestGetGivesUpAfterRetries(t *testing.T) {
var hits int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
atomic.AddInt32(&hits, 1)
w.WriteHeader(http.StatusTooManyRequests)
}))
defer srv.Close()
d := NewDownloader("kage-test", 5*time.Second, 0)
d.Retries = 2
u, _ := url.Parse(srv.URL + "/rate.css")
_, err := d.Get(context.Background(), u, "")
var se *StatusError
if !errors.As(err, &se) || se.Code != 429 {
t.Fatalf("got %v; want StatusError 429", err)
}
if hits != 3 { // 1 try + 2 retries
t.Errorf("expected 3 attempts, server saw %d", hits)
}
}
func TestTransientClassification(t *testing.T) {
transientCodes := []int{403, 408, 425, 429, 500, 502, 503}
for _, c := range transientCodes {
if !transient(&StatusError{Code: c}) {
t.Errorf("status %d should be transient", c)
}
}
for _, c := range []int{400, 401, 404, 410} {
if transient(&StatusError{Code: c}) {
t.Errorf("status %d should be permanent", c)
}
}
if transient(context.Canceled) {
t.Error("context.Canceled should not be transient")
}
}
+98 -2
View File
@@ -10,6 +10,7 @@ import (
"fmt"
"os"
"runtime"
"strings"
"sync"
"time"
@@ -130,9 +131,30 @@ func (p *Pool) getBrowser() (*rod.Browser, error) {
l := launcher.New().
Headless(p.opts.Headless).
Set("disable-blink-features", "AutomationControlled").
Set("disable-dev-shm-usage", "").
Set("no-sandbox", "").
Set("disable-gpu", "")
// Chrome's sandbox is the main line of defense when rendering pages from
// the open web, so kage keeps it on by default (issue #10). It is dropped
// only where it genuinely cannot initialize: inside a container, or when
// running as root, where Chrome otherwise refuses to start. The decision
// is logged so it is never silent.
if off, reason := disableSandbox(); off {
l = l.Set("no-sandbox", "")
warnSandboxDisabled(reason)
}
// In a container, the default /dev/shm is only 64 MB, too small for
// Chrome's renderer on large pages, so steer it to a temp file instead.
// Outside a container /dev/shm is roomy and faster, so leave it alone.
// Chrome's crashpad handler also aborts with "--database is required" in a
// minimal container, which fails the whole launch (issue #7), so turn the
// crash reporter off there. kage never uploads Chrome crash dumps anyway.
if inContainer() {
l = l.Set("disable-dev-shm-usage", "").
Set("disable-crash-reporter", "").
Set("disable-breakpad", "")
}
if bin := p.chromeBin(); bin != "" {
l = l.Bin(bin)
}
@@ -225,6 +247,80 @@ func systemChromeCandidates() []string {
}
}
// disableSandbox decides whether Chrome should launch without its sandbox, with
// a short reason for the log. The secure default is to keep the sandbox on; it
// is dropped only where it cannot run: inside a container, or when running as
// root (Chrome refuses to start a sandbox as root).
func disableSandbox() (off bool, reason string) {
if inContainer() {
return true, "container"
}
if isRoot() {
return true, "root"
}
return false, ""
}
// warnSandboxDisabled prints why the sandbox was turned off, so dropping a
// security boundary is always visible rather than silent.
func warnSandboxDisabled(reason string) {
switch reason {
case "container":
fmt.Fprintln(os.Stderr, "kage: container detected, Chrome sandbox disabled")
case "root":
fmt.Fprintln(os.Stderr, "kage: running as root, Chrome sandbox disabled (run as a non-root user to keep it on)")
}
}
// inContainer reports whether kage is running inside a container, where Chrome
// needs container-specific flags. It honors IN_DOCKER (set it in your image)
// and the /.dockerenv marker that Docker writes into every container.
//
// Keeping the sandbox on by default and dropping it only here was prompted by
// Dimitrios Prasakis (issue #10); the IN_DOCKER opt-in was suggested on Hacker
// News (https://news.ycombinator.com/item?id=48534865). Thanks to both.
func inContainer() bool {
if envTrue("IN_DOCKER") {
return true
}
if _, err := os.Stat("/.dockerenv"); err == nil {
return true
}
return false
}
// isRoot reports whether the process runs as the superuser. On Windows
// os.Geteuid returns -1, so this is false there.
func isRoot() bool {
return os.Geteuid() == 0
}
// envTrue reports whether the named environment variable is set to a truthy
// value.
func envTrue(name string) bool {
v, ok := envBool(name)
return ok && v
}
// envBool parses a boolean-ish environment variable. It returns ok=false when
// the variable is unset or empty. "1", "true", "yes", "on" are true and "0",
// "false", "no", "off" are false (case-insensitive); any other non-empty value
// counts as true, so IN_DOCKER=docker reads as set.
func envBool(name string) (val, ok bool) {
s := strings.TrimSpace(os.Getenv(name))
if s == "" {
return false, false
}
switch strings.ToLower(s) {
case "1", "true", "yes", "on":
return true, true
case "0", "false", "no", "off":
return false, true
default:
return true, true
}
}
// settle waits for the network to go quiet for d, recovering from any rod
// panic and capping the wait so a chatty page can never hang the worker.
func settle(page *rod.Page, d time.Duration) {
+65
View File
@@ -4,6 +4,7 @@ import (
"context"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"time"
@@ -17,6 +18,70 @@ func TestLookChromeReadsEnv(t *testing.T) {
}
}
func TestEnvBool(t *testing.T) {
cases := []struct {
in string
set bool
wantVal bool
wantOk bool
}{
{"", false, false, false},
{"1", true, true, true},
{"true", true, true, true},
{"TRUE", true, true, true},
{"yes", true, true, true},
{"on", true, true, true},
{"0", true, false, true},
{"false", true, false, true},
{"off", true, false, true},
{"no", true, false, true},
{"docker", true, true, true}, // any other non-empty value is true
{" true ", true, true, true}, // trimmed
}
for _, c := range cases {
if c.set {
t.Setenv("KAGE_TEST_BOOL", c.in)
} else {
_ = os.Unsetenv("KAGE_TEST_BOOL")
}
val, ok := envBool("KAGE_TEST_BOOL")
if val != c.wantVal || ok != c.wantOk {
t.Errorf("envBool(%q) = (%v, %v); want (%v, %v)", c.in, val, ok, c.wantVal, c.wantOk)
}
}
}
func TestDisableSandboxDefaultKeepsItOn(t *testing.T) {
// Not in a container and not root, the sandbox stays on. (When the test
// itself runs as root, e.g. some CI containers, "root" is the honest
// reason; accept that rather than asserting a false negative.)
t.Setenv("IN_DOCKER", "")
off, reason := disableSandbox()
if isRoot() || inContainer() {
if !off {
t.Errorf("disableSandbox() = false as root/container; want true")
}
return
}
if off {
t.Errorf("disableSandbox() = true (%q) on a normal host; want sandbox kept on", reason)
}
}
func TestInContainerHonorsEnv(t *testing.T) {
t.Setenv("IN_DOCKER", "1")
if !inContainer() {
t.Errorf("inContainer() = false with IN_DOCKER=1; want true")
}
}
func TestDisableSandboxContainer(t *testing.T) {
t.Setenv("IN_DOCKER", "true")
if off, reason := disableSandbox(); !off || reason != "container" {
t.Errorf("in container: got (%v, %q); want (true, container)", off, reason)
}
}
func TestRenderCapturesFinalDOM(t *testing.T) {
if testing.Short() {
t.Skip("render test drives Chrome; skipped under -short")
+18
View File
@@ -186,9 +186,27 @@ func printSummary(res clone.Result) {
styleAccent.Render("assets"), res.Assets)
if res.PageErrors+res.AssetErrors > 0 {
fmt.Fprintf(os.Stderr, " %s %d\n", styleErr.Render("errors"), res.PageErrors+res.AssetErrors)
printFailures(res)
}
if res.Skipped > 0 {
fmt.Fprintf(os.Stderr, " %s %d\n", styleWarn.Render("skipped"), res.Skipped)
}
fmt.Fprintf(os.Stderr, " open %s\n", styleAccent.Render("kage serve "+res.OutDir))
}
// printFailures lists what went wrong, grouped reason and URL, so the error
// count is actionable instead of opaque. The list is capped during the crawl;
// when it overflows, say how many more there were.
func printFailures(res clone.Result) {
total := res.PageErrors + res.AssetErrors
for _, f := range res.Failures {
line := fmt.Sprintf(" %s %s", styleErr.Render(f.Reason), f.URL)
fmt.Fprintln(os.Stderr, line)
if f.Referer != "" {
fmt.Fprintln(os.Stderr, styleDim.Render(" referenced by "+f.Referer))
}
}
if more := total - int64(len(res.Failures)); more > 0 {
fmt.Fprintln(os.Stderr, styleDim.Render(fmt.Sprintf(" ... and %d more", more)))
}
}
+246 -22
View File
@@ -2,7 +2,9 @@ package cli
import (
"fmt"
"image"
"os"
"os/exec"
"path/filepath"
"runtime"
"strings"
@@ -19,6 +21,8 @@ type packFlags struct {
format string
out string
base string
app bool
icon string
noCompress bool
title string
description string
@@ -34,7 +38,9 @@ func newPackCmd() *cobra.Command {
Long: "pack turns a cloned folder into one distributable file. With --format zim\n" +
"it writes an open ZIM archive (the format Kiwix uses) that kage open or any\n" +
"ZIM reader can browse. With --format binary it appends that archive to a copy\n" +
"of kage, producing a single executable that serves the site offline when run.",
"of kage, producing a single executable that serves the site offline when run.\n" +
"Add --app to wrap that executable in a double-click desktop app (a .app bundle\n" +
"on macOS, an AppImage-style .AppDir on Linux) with the site's favicon as the icon.",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
return runPack(args[0], f)
@@ -43,7 +49,9 @@ func newPackCmd() *cobra.Command {
fs := cmd.Flags()
fs.StringVar(&f.format, "format", "zim", "output format: zim or binary")
fs.StringVarP(&f.out, "out", "o", "", "output path (default per format)")
fs.StringVar(&f.base, "base", "", "base kage binary for --format binary (default this kage)")
fs.StringVar(&f.base, "base", "", "base kage binary for the viewer (default this kage)")
fs.BoolVar(&f.app, "app", false, "wrap the viewer in a double-click desktop app (.app on macOS, .AppImage/.AppDir on Linux)")
fs.StringVar(&f.icon, "icon", "", "icon file for --app (default the site's favicon)")
fs.BoolVar(&f.noCompress, "no-compress", false, "store every cluster raw, no zstd")
fs.StringVar(&f.title, "title", "", "archive title (default the main page's <title>)")
fs.StringVar(&f.description, "description", "", "archive description")
@@ -64,6 +72,12 @@ func runPack(mirrorArg string, f *packFlags) error {
Version: Version,
}
// --app wraps the packed viewer in a desktop bundle. It builds on the binary
// format, so it owns the flow rather than being one more --format value.
if f.app {
return runPackApp(dir, f, zopts)
}
switch f.format {
case "zim":
out, size, err := pack.BuildZIM(dir, zopts)
@@ -79,16 +93,22 @@ func runPack(mirrorArg string, f *packFlags) error {
if err != nil {
return err
}
target := resolveTargetOS(f.base)
out := f.out
if out == "" {
out = defaultBinaryName(dir, f.base)
out = defaultBinaryName(dir)
}
// A Windows viewer must end in .exe to run, whether the name came from
// --out or the default, so make sure it does.
if target == "windows" && !strings.HasSuffix(strings.ToLower(out), ".exe") {
out += ".exe"
}
path, size, err := pack.BuildBinary(zbytes, pack.BinaryOptions{Out: out, Base: f.base})
if err != nil {
return err
}
printPackResult(path, size)
printRunHint(path)
printRunHint(path, target)
return nil
default:
@@ -96,6 +116,188 @@ func runPack(mirrorArg string, f *packFlags) error {
}
}
// runPackApp builds a double-clickable desktop app around the packed viewer,
// shaped for whichever OS the base targets: a .app bundle on macOS, an
// AppImage-style .AppDir on Linux. Windows needs no bundle (the .exe is the
// app), so it is redirected to --format binary with a GUI base.
func runPackApp(dir string, f *packFlags, zopts pack.ZIMOptions) error {
target := resolveTargetOS(f.base)
switch target {
case "windows":
return fmt.Errorf("a Windows app is just the .exe, with no bundle to build: use --format binary and a GUI base (kage built with -ldflags -H=windowsgui)")
case "":
if f.base != "" {
return fmt.Errorf("--app could not tell which OS %q is for; pass a macOS or Linux kage as --base", f.base)
}
// No base and an unknown runtime: fall through with the host's GOOS.
target = runtime.GOOS
}
prog := defaultBinaryName(dir)
name := f.title
if name == "" {
name = prog
}
icon, iconSrc, err := resolveIcon(dir, f.icon)
if err != nil {
return err
}
zbytes, err := pack.BuildZIMBytes(dir, zopts)
if err != nil {
return err
}
switch target {
case "darwin":
return packMacApp(zbytes, dir, f, prog, name, icon, iconSrc)
case "linux":
return packLinuxApp(zbytes, dir, f, prog, name, icon, iconSrc)
default:
return fmt.Errorf("--app supports macOS and Linux bases; %s is not one of them", osLabel(target))
}
}
// packMacApp writes the .app bundle and prints how to launch it.
func packMacApp(zbytes []byte, dir string, f *packFlags, prog, name string, icon image.Image, iconSrc string) error {
out := f.out
if out == "" {
out = prog + ".app"
} else if !strings.HasSuffix(strings.ToLower(out), ".app") {
out += ".app"
}
path, size, err := pack.BuildApp(zbytes, pack.AppOptions{
Out: out,
Base: f.base,
Name: name,
ExecName: prog,
Identifier: bundleID(prog),
Version: appVersion(),
Icon: icon,
})
if err != nil {
return err
}
printPackResult(path, size)
printIconLine(iconSrc)
fmt.Fprintf(os.Stderr, " double-click %s to open the site offline\n", styleAccent.Render(filepath.Base(path)))
if f.base == "" {
fmt.Fprintln(os.Stderr, styleDim.Render(" (built around this kage; pass --base a webview build to open a native window instead of the browser)"))
}
fmt.Fprintln(os.Stderr, styleDim.Render(" (macOS may quarantine it: xattr -dr com.apple.quarantine "+path+")"))
return nil
}
// packLinuxApp writes the .AppDir and, when appimagetool is installed, folds it
// into a single double-clickable .AppImage.
func packLinuxApp(zbytes []byte, dir string, f *packFlags, prog, name string, icon image.Image, iconSrc string) error {
out := f.out
if out == "" {
out = prog + ".AppDir"
} else if !strings.HasSuffix(out, ".AppDir") {
out += ".AppDir"
}
path, size, hasIcon, err := pack.BuildAppDir(zbytes, pack.LinuxAppOptions{
Out: out,
Base: f.base,
Name: name,
ExecName: prog,
Comment: f.description,
Version: appVersion(),
Icon: icon,
})
if err != nil {
return err
}
printPackResult(path, size)
printIconLine(iconSrc)
// appimagetool turns the directory into one portable file. It needs an icon,
// so only attempt it when the mirror gave us one.
if hasIcon {
if img, ok := tryAppImage(path, prog); ok {
fmt.Fprintf(os.Stderr, " built %s\n", styleTitle.Render(img))
fmt.Fprintf(os.Stderr, " double-click %s to open the site offline\n", styleAccent.Render(filepath.Base(img)))
return nil
}
}
fmt.Fprintf(os.Stderr, " run %s to open the site offline\n", styleAccent.Render("./"+filepath.Join(filepath.Base(path), "AppRun")))
fmt.Fprintln(os.Stderr, styleDim.Render(" (install appimagetool to fold this .AppDir into one double-clickable .AppImage)"))
return nil
}
// tryAppImage runs appimagetool over the AppDir if it is installed, returning
// the .AppImage path on success. A missing tool or a build failure is not fatal:
// the caller falls back to the AppDir.
func tryAppImage(appDir, prog string) (string, bool) {
tool, err := exec.LookPath("appimagetool")
if err != nil {
return "", false
}
out := prog + ".AppImage"
cmd := exec.Command(tool, appDir, out)
// appimagetool reads the target arch from the AppRun ELF; suppress its noisy
// progress so kage's own output stays clean, but surface a real failure.
if err := cmd.Run(); err != nil {
return "", false
}
if _, err := os.Stat(out); err != nil {
return "", false
}
return out, true
}
func printIconLine(iconSrc string) {
if iconSrc != "" {
fmt.Fprintf(os.Stderr, " icon %s\n", styleDim.Render(iconSrc))
}
}
// resolveIcon picks the bundle icon: an explicit --icon path if given (an error
// there is fatal, since the user asked for that file), otherwise the site's
// favicon discovered in the mirror. A mirror with no usable icon is fine; the
// bundle just ships without a custom one.
func resolveIcon(dir, iconFlag string) (img image.Image, src string, err error) {
if iconFlag != "" {
img, err = pack.DecodeIcon(iconFlag)
if err != nil {
return nil, "", err
}
return img, iconFlag, nil
}
if img, src, ok := pack.FindIcon(dir); ok {
return img, src, nil
}
return nil, "", nil
}
// bundleID builds a reverse-DNS CFBundleIdentifier from the program name,
// keeping only characters Apple allows in an identifier.
func bundleID(prog string) string {
var b strings.Builder
for _, r := range strings.ToLower(prog) {
switch {
case r >= 'a' && r <= 'z', r >= '0' && r <= '9', r == '-':
b.WriteRune(r)
default:
b.WriteRune('-')
}
}
id := strings.Trim(b.String(), "-")
if id == "" {
id = "app"
}
return "com.kage." + id
}
// appVersion uses kage's own version for the bundle, falling back to 1.0 for a
// dev build whose version is the default "dev".
func appVersion() string {
if Version == "" || Version == "dev" {
return "1.0"
}
return strings.TrimPrefix(Version, "v")
}
// resolveMirror accepts either a path to a mirror dir or a bare host. A bare
// host that is not a directory in the working dir is resolved against the
// default out dir, so "kage pack paulgraham.com" works right after a clone.
@@ -110,27 +312,32 @@ func resolveMirror(arg string) string {
return arg
}
// defaultBinaryName derives a clean program name from the mirror's host: strip a
// trailing dot-suffix (paulgraham.com -> paulgraham), and append .exe when the
// target is Windows. The target is the running OS unless --base names a binary
// that looks like it was built for Windows.
func defaultBinaryName(dir, base string) string {
// defaultBinaryName derives a clean program name from the mirror's host by
// stripping a trailing dot-suffix (paulgraham.com -> paulgraham). The caller
// appends .exe for Windows targets.
func defaultBinaryName(dir string) string {
host := filepath.Base(dir)
name := host
if i := strings.IndexByte(host, '.'); i > 0 {
name = host[:i]
return host[:i]
}
if windowsTarget(base) {
name += ".exe"
}
return name
return host
}
func windowsTarget(base string) bool {
// resolveTargetOS reports which OS the packed viewer will run on. With no
// --base it is this kage's OS; with one, we sniff the base's executable header
// so detection does not hinge on the file being named ".exe". If the header is
// unrecognised we fall back to that name heuristic.
func resolveTargetOS(base string) string {
if base == "" {
return runtime.GOOS == "windows"
return runtime.GOOS
}
return strings.HasSuffix(strings.ToLower(base), ".exe")
if os := pack.SniffOS(base); os != "" {
return os
}
if strings.HasSuffix(strings.ToLower(base), ".exe") {
return "windows"
}
return ""
}
func printPackResult(path string, size int64) {
@@ -138,21 +345,38 @@ func printPackResult(path string, size int64) {
fmt.Fprintf(os.Stderr, " %s %s\n", styleAccent.Render("size"), humanBytes(size))
}
func printRunHint(path string) {
func printRunHint(path, target string) {
rel := path
if !strings.ContainsAny(path, "/\\") {
rel = "./" + path
}
if windowsTarget(path) && runtime.GOOS != "windows" {
fmt.Fprintf(os.Stderr, " this is a Windows viewer; run %s on Windows\n", styleAccent.Render(filepath.Base(path)))
// A viewer built for another OS cannot run here, so say where it goes
// instead of printing a run command that would not work.
if target != "" && target != runtime.GOOS {
fmt.Fprintf(os.Stderr, " this is a %s viewer; copy %s to that machine to run it\n",
osLabel(target), styleAccent.Render(filepath.Base(path)))
return
}
fmt.Fprintf(os.Stderr, " run %s to view the site offline\n", styleAccent.Render(rel))
if runtime.GOOS == "darwin" {
if target == "darwin" {
fmt.Fprintln(os.Stderr, styleDim.Render(" (macOS may quarantine it: xattr -d com.apple.quarantine "+rel+")"))
}
}
// osLabel turns a GOOS value into a friendly name for the run hint.
func osLabel(goos string) string {
switch goos {
case "windows":
return "Windows"
case "darwin":
return "macOS"
case "linux":
return "Linux"
default:
return goos
}
}
// humanBytes renders a byte count in B, KiB, MiB, or GiB.
func humanBytes(n int64) string {
const unit = 1024
+50 -12
View File
@@ -2,6 +2,7 @@ package clone
import (
"context"
"errors"
"fmt"
"io"
"net/http"
@@ -165,7 +166,7 @@ func (c *Cloner) Run(ctx context.Context) (Result, error) {
}
}
res := Result{Progress: c.stats.snapshot(), OutDir: c.outRoot}
res := Result{Progress: c.stats.snapshot(), OutDir: c.outRoot, Failures: c.stats.recordedFailures()}
if ctx.Err() != nil {
return res, ctx.Err()
}
@@ -236,15 +237,13 @@ func (c *Cloner) processPage(ctx context.Context, j pageItem) {
res, err := c.pool.Render(ctx, j.u.String())
if err != nil {
c.stats.pageErrors.Add(1)
c.logf("page error %s: %v", j.u, err)
c.failPage(j.u.String(), fmt.Errorf("render: %w", err))
return
}
root, err := html.Parse(strings.NewReader(res.HTML))
if err != nil {
c.stats.pageErrors.Add(1)
c.logf("parse error %s: %v", j.u, err)
c.failPage(j.u.String(), fmt.Errorf("parse: %w", err))
return
}
@@ -275,12 +274,11 @@ func (c *Cloner) processPage(ctx context.Context, j pageItem) {
var buf strings.Builder
if err := html.Render(&buf, root); err != nil {
c.stats.pageErrors.Add(1)
c.failPage(j.u.String(), fmt.Errorf("render html: %w", err))
return
}
if err := c.writeFile(localFile, []byte(buf.String())); err != nil {
c.stats.pageErrors.Add(1)
c.logf("write error %s: %v", localFile, err)
c.failPage(j.u.String(), fmt.Errorf("write %s: %w", localFile, err))
return
}
c.front.markVisited(key)
@@ -295,8 +293,7 @@ func (c *Cloner) processAsset(ctx context.Context, j assetItem) {
}
res, err := c.dl.Get(ctx, j.u, j.referer)
if err != nil {
c.stats.assetErrors.Add(1)
c.logf("asset error %s: %v", j.u, err)
c.failAsset(j.u.String(), j.referer, err)
return
}
@@ -312,13 +309,54 @@ func (c *Cloner) processAsset(ctx context.Context, j assetItem) {
body = asset.RewriteCSS(body, j.u, cssSink)
}
if err := c.writeFile(localFile, body); err != nil {
c.stats.assetErrors.Add(1)
c.logf("write error %s: %v", localFile, err)
c.failAsset(j.u.String(), j.referer, fmt.Errorf("write %s: %w", localFile, err))
return
}
c.stats.assets.Add(1)
}
// failAsset records and logs a failed asset, naming the page that referenced it
// so a 403 or 404 is traceable back to where it came from. The reason is
// classified (HTTP status, timeout, or other) for a readable line.
func (c *Cloner) failAsset(u, referer string, err error) {
c.stats.assetErrors.Add(1)
reason := classifyError(err)
c.stats.recordFailure(Failure{Kind: "asset", URL: u, Referer: referer, Reason: reason})
if referer != "" {
c.logf("asset error: %s\n %s\n referenced by %s", reason, u, referer)
} else {
c.logf("asset error: %s\n %s", reason, u)
}
}
// failPage records and logs a failed page.
func (c *Cloner) failPage(u string, err error) {
c.stats.pageErrors.Add(1)
reason := classifyError(err)
c.stats.recordFailure(Failure{Kind: "page", URL: u, Reason: reason})
c.logf("page error: %s\n %s", reason, u)
}
// classifyError turns an error into a short, human-readable reason for the log
// and the final report: an HTTP status with its name, a timeout, a cancellation,
// or the underlying message otherwise.
func classifyError(err error) string {
if err == nil {
return ""
}
var se *asset.StatusError
if errors.As(err, &se) {
return se.Error()
}
switch {
case errors.Is(err, context.DeadlineExceeded):
return "timed out"
case errors.Is(err, context.Canceled):
return "cancelled"
}
return err.Error()
}
// enqueuePage offers a page URL to the frontier, honouring the visited set, the
// depth cap, and the page budget. It reports whether the page was newly queued.
func (c *Cloner) enqueuePage(ctx context.Context, u *url.URL, depth int) bool {
+39 -1
View File
@@ -1,6 +1,14 @@
package clone
import "sync/atomic"
import (
"sync"
"sync/atomic"
)
// maxRecordedFailures caps how many individual failures Run keeps for the final
// report, so a huge broken site cannot grow the slice without bound. The error
// counters still count every failure.
const maxRecordedFailures = 100
// stats are the live counters of a run, read by the CLI's progress ticker.
type stats struct {
@@ -9,6 +17,34 @@ type stats struct {
pageErrors atomic.Int64
assetErrors atomic.Int64
skipped atomic.Int64 // robots-disallowed or out of budget
muFail sync.Mutex
failures []Failure
}
// Failure is one thing that went wrong, kept for the end-of-run report so the
// errors are visible as a list rather than only as a count.
type Failure struct {
Kind string // "page" or "asset"
URL string
Referer string // the page that referenced it, when known
Reason string // e.g. "HTTP 403 Forbidden"
}
func (s *stats) recordFailure(f Failure) {
s.muFail.Lock()
if len(s.failures) < maxRecordedFailures {
s.failures = append(s.failures, f)
}
s.muFail.Unlock()
}
func (s *stats) recordedFailures() []Failure {
s.muFail.Lock()
defer s.muFail.Unlock()
out := make([]Failure, len(s.failures))
copy(out, s.failures)
return out
}
// Progress is a snapshot of a run for display.
@@ -34,4 +70,6 @@ func (s *stats) snapshot() Progress {
type Result struct {
Progress
OutDir string
// Failures is a capped sample of what went wrong, for the final report.
Failures []Failure
}
+1 -1
View File
@@ -40,7 +40,7 @@ kage crawls breadth-first from a seed URL, staying within the seed's host (and o
## Then what?
A folder is the starting point, not the end. Once you have a mirror you can [pack it](/guides/packing-a-mirror/) into a single ZIM file, the open offline-archive format Kiwix uses, so the whole site travels as one file that any ZIM reader can open. Or build kage with the `webview` tag and a packed binary opens the site in its own native window instead of a browser tab:
A folder is the starting point, not the end. Once you have a mirror you can [pack it](/guides/packing-a-mirror/) into a single ZIM file, the open offline-archive format Kiwix uses, so the whole site travels as one file that any ZIM reader can open. You can also pack it into a self-contained binary, or a double-click desktop app with the site's favicon as its icon. Or build kage with the `webview` tag and a packed binary opens the site in its own native window instead of a browser tab:
![paulgraham.com served offline in a native kage window](/webview.png)
+41 -1
View File
@@ -83,7 +83,7 @@ The window title comes from the archive's title. This build needs cgo and links
### Build a viewer for another platform
The appended archive is platform-independent; only the base executable carries the architecture. Point `--base` at a kage binary built for another OS (download one from a kage release) to produce a viewer for that platform from your own machine:
The appended archive is platform-independent; only the base executable carries the architecture. Point `--base` at a kage binary built for another OS (download one from a kage release; every platform ships one) to produce a viewer for that platform from your own machine. kage reads the base's executable header to detect the target OS, so a Windows viewer automatically gets a `.exe` name and the run hint names the right platform:
```bash
# From macOS, build a Windows viewer
@@ -99,6 +99,46 @@ A binary you built or downloaded may be quarantined by Gatekeeper on first run.
xattr -d com.apple.quarantine ./paulgraham
```
## A double-click app
The self-contained binary is perfect from a terminal, but double-clicking it in a file manager is less tidy: on macOS Finder opens a Terminal window behind the site, and on Windows a console flashes alongside it. Add `--app` and kage wraps the same viewer in a real desktop app, so a double-click just opens the mirror with no terminal in sight, using the site's own favicon as the icon.
On macOS it writes a standard `.app` bundle:
```bash
kage pack paulgraham.com --app
```
```
packed paulgraham.app
size 13.5 MiB
icon paulgraham.com/apple-touch-icon.png
double-click paulgraham.app to open the site offline
```
The bundle holds the packed viewer under `Contents/MacOS`, an `Info.plist` describing the app, and the icon converted to `Contents/Resources/icon.icns`. Double-click it in Finder, or run `open paulgraham.app`, and the site comes up with no console attached.
On Linux, point `--base` at a Linux kage and you get an [AppImage](https://appimage.org)-style `.AppDir`: the viewer as `AppRun`, a `.desktop` launcher with `Terminal=false`, and the icon as a PNG. When [`appimagetool`](https://github.com/AppImage/appimagetool) is on your `PATH`, kage runs it for you and turns the directory into one double-clickable `.AppImage`; otherwise it leaves the `.AppDir` ready for any AppImage tool.
```bash
kage pack paulgraham.com --app --base kage-linux-amd64 # -> paulgraham.AppDir (+ .AppImage)
```
kage picks the icon by digging through the mirror for the site's favicon. It prefers a large `apple-touch-icon.png` and falls back to `favicon.png` or a PNG-based `favicon.ico`; if a site only ships a legacy BMP `.ico` the bundle is built without a custom icon rather than with a mangled one. Override the choice with `--icon path/to/image.png`.
For the full "it's an app" effect, pair `--app` with a `webview` base so the double-click opens a native window instead of the system browser:
```bash
make build-webview
kage pack paulgraham.com --app --base bin/kage
```
Windows needs no bundle, because there a single `.exe` already is the app. What it needs is to lose the console window. A normal build is console-attached (handy for the CLI, since that is where clone progress prints), so the release ships a second Windows binary linked for the GUI subsystem in `kage_<version>_windows-gui_<arch>.zip`. Pack a viewer onto that base and double-clicking the result opens the site with no console behind it:
```bash
kage pack paulgraham.com --format binary --base kage-windows-gui-amd64.exe # -> paulgraham.exe
```
## Metadata and options
```bash
+17
View File
@@ -6,6 +6,23 @@ weight: 40
The authoritative, commit-level history lives in [`CHANGELOG.md`](https://github.com/tamnd/kage/blob/main/CHANGELOG.md) and on the [releases page](https://github.com/tamnd/kage/releases). This page summarises each version.
## v0.2.0
Double-click apps, so a packed mirror opens like a real desktop app instead of a terminal program.
- **`kage pack --app`** wraps the viewer in a double-click app with the site's favicon as its icon. The flag builds on the binary format, so it composes with `--base` (including a `webview` base) and `--icon`. On macOS that is a `.app` bundle; on Linux, with a Linux `--base`, an [AppImage](https://appimage.org)-style `.AppDir` that becomes a single `.AppImage` when `appimagetool` is installed. The icon is pulled from the mirror automatically, or set with `--icon`.
- **A GUI-subsystem Windows base** ships in the release as `kage_<version>_windows-gui_<arch>.zip`. Pack a viewer onto it with `--format binary --base` and the resulting `.exe` opens with no console window behind it.
- **Smarter cross-platform packing.** kage reads the base binary's executable header to detect its target OS, so a Windows viewer always gets a `.exe` name and the right run hint, regardless of how the base file is named.
## v0.1.2
A security fix for how kage launches Chrome, clearer crawl errors, and a container image that actually runs.
- **Chrome keeps its sandbox on by default.** Earlier versions launched Chrome with `--no-sandbox` on every run, which switched off the browser's main security boundary even on an ordinary desktop where the sandbox works fine ([#10](https://github.com/tamnd/kage/issues/10)). The sandbox now stays on, and is dropped only where it genuinely cannot start: inside a container (detected from `IN_DOCKER` or `/.dockerenv`) or when running as root. Whenever it is dropped, kage says so on stderr, so the choice is never silent.
- **Transient asset failures retry.** A download that hits a 403/429, a 5xx, or a network blip is retried with a short backoff, which recovers files that bot-protection rejects on the first request of a burst. Permanent failures like a 404 are not retried.
- **Clearer crawl errors.** Each failure now logs a classified reason (`HTTP 403 Forbidden`, `timed out`, ...), the URL, and the page that referenced it, and the end-of-run summary lists what went wrong instead of printing only a count.
- **The container image runs.** Chrome aborted in the image with `chrome_crashpad_handler: --database is required`, so the crash reporter is now disabled inside a container, and the `kage` user has a writable home (the mounted `/out` volume) so output, resume state, and Chrome's profile no longer fail with a permission error ([#7](https://github.com/tamnd/kage/issues/7)).
## v0.1.1
Packing, so a clone can travel as one file instead of a folder.
+1
View File
@@ -10,6 +10,7 @@ require (
github.com/klauspost/compress v1.18.6
github.com/spf13/cobra v1.10.2
github.com/webview/webview_go v0.0.0-20240831120633-6173450d4dd6
golang.org/x/image v0.42.0
golang.org/x/net v0.56.0
)
+2
View File
@@ -87,6 +87,8 @@ github.com/ysmood/leakless v0.9.0/go.mod h1:R8iAXPRaG97QJwqxs74RdwzcRHT1SWCGTNqY
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
golang.org/x/exp v0.0.0-20231006140011-7918f672742d h1:jtJma62tbqLibJ5sFQz8bKtEM8rJBtfilJ2qTU199MI=
golang.org/x/exp v0.0.0-20231006140011-7918f672742d/go.mod h1:ldy0pHrwJyGW56pPQzzkH36rKxoZW1tw7ZJpeKx+hdo=
golang.org/x/image v0.42.0 h1:1gSs6ehNWXLbkHBIPcWztk3D/6aIA/8hauiAYtlodVY=
golang.org/x/image v0.42.0/go.mod h1:rrpelvGFt+kLPAjPM4HeWPgrl0FtafueU//e5N0qk/Q=
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
+161
View File
@@ -0,0 +1,161 @@
package pack
import (
"encoding/xml"
"fmt"
"image"
"os"
"path/filepath"
"strings"
)
// AppOptions controls how a macOS .app bundle is assembled around a packed
// viewer.
type AppOptions struct {
Out string // path to the .app directory
Base string // base kage binary (must be a macOS build); default os.Executable()
Name string // display name shown in Finder and the Dock
ExecName string // file name of the executable inside Contents/MacOS
Identifier string // CFBundleIdentifier, e.g. com.kage.paulgraham
Version string // CFBundleShortVersionString; default 1.0
Icon image.Image // optional; written as Resources/icon.icns
}
// BuildApp writes a double-clickable macOS application bundle that serves the
// packed site. Finder runs Contents/MacOS/<ExecName> with no terminal attached,
// which is the whole point: the bare appended binary opens a Terminal window
// when double-clicked, a .app does not. The executable is the same
// base++zim++trailer image BuildBinary produces, so Embedded finds the archive
// at runtime exactly as it does for a plain viewer.
//
// It returns the bundle path and the size of the executable inside it (the part
// that dominates, since the plist and icon are tiny).
func BuildApp(zimBytes []byte, opts AppOptions) (string, int64, error) {
if opts.Out == "" {
return "", 0, fmt.Errorf("pack: BuildApp requires an output path")
}
if filepath.Ext(opts.Out) != ".app" {
return "", 0, fmt.Errorf("pack: app bundle path must end in .app, got %q", opts.Out)
}
base := opts.Base
if base == "" {
exe, err := os.Executable()
if err != nil {
return "", 0, fmt.Errorf("pack: locate base binary: %w", err)
}
base = exe
}
baseBytes, err := os.ReadFile(base)
if err != nil {
return "", 0, fmt.Errorf("pack: read base binary %q: %w", base, err)
}
execName := opts.ExecName
if execName == "" {
execName = "kage"
}
name := opts.Name
if name == "" {
name = execName
}
version := opts.Version
if version == "" {
version = "1.0"
}
// Start from a clean bundle so a rebuild never leaves a stale icon or
// executable behind. The path is known to end in .app from the guard above.
if err := os.RemoveAll(opts.Out); err != nil {
return "", 0, err
}
contents := filepath.Join(opts.Out, "Contents")
macOS := filepath.Join(contents, "MacOS")
resources := filepath.Join(contents, "Resources")
if err := os.MkdirAll(macOS, 0o755); err != nil {
return "", 0, err
}
hasIcon := opts.Icon != nil
if hasIcon {
if err := os.MkdirAll(resources, 0o755); err != nil {
return "", 0, err
}
icns, err := EncodeICNS(opts.Icon)
if err != nil {
return "", 0, err
}
if err := os.WriteFile(filepath.Join(resources, "icon.icns"), icns, 0o644); err != nil {
return "", 0, err
}
}
plist := infoPlist(infoPlistData{
Name: name,
ExecName: execName,
Identifier: opts.Identifier,
Version: version,
HasIcon: hasIcon,
})
if err := os.WriteFile(filepath.Join(contents, "Info.plist"), []byte(plist), 0o644); err != nil {
return "", 0, err
}
// PkgInfo is the eight-byte legacy type/creator stamp; APPL???? is the
// generic application value every modern bundle uses.
if err := os.WriteFile(filepath.Join(contents, "PkgInfo"), []byte("APPL????"), 0o644); err != nil {
return "", 0, err
}
exe := assemble(baseBytes, zimBytes)
if err := os.WriteFile(filepath.Join(macOS, execName), exe, 0o755); err != nil {
return "", 0, err
}
return opts.Out, int64(len(exe)), nil
}
type infoPlistData struct {
Name string
ExecName string
Identifier string
Version string
HasIcon bool
}
// infoPlist renders the bundle's Info.plist. Values are XML-escaped so a site
// title with an ampersand or angle bracket cannot corrupt the file.
func infoPlist(d infoPlistData) string {
pairs := [][2]string{
{"CFBundleName", d.Name},
{"CFBundleDisplayName", d.Name},
{"CFBundleExecutable", d.ExecName},
{"CFBundleIdentifier", d.Identifier},
{"CFBundleInfoDictionaryVersion", "6.0"},
{"CFBundlePackageType", "APPL"},
{"CFBundleShortVersionString", d.Version},
{"CFBundleVersion", d.Version},
{"LSMinimumSystemVersion", "10.13"},
}
if d.HasIcon {
pairs = append(pairs, [2]string{"CFBundleIconFile", "icon"})
}
var b strings.Builder
b.WriteString(xml.Header)
b.WriteString(`<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">` + "\n")
b.WriteString(`<plist version="1.0">` + "\n<dict>\n")
for _, kv := range pairs {
b.WriteString("\t<key>" + esc(kv[0]) + "</key>\n")
b.WriteString("\t<string>" + esc(kv[1]) + "</string>\n")
}
// NSHighResolutionCapable is a boolean, not a string, so the icon and text
// render sharp on Retina displays.
b.WriteString("\t<key>NSHighResolutionCapable</key>\n\t<true/>\n")
b.WriteString("</dict>\n</plist>\n")
return b.String()
}
func esc(s string) string {
var b strings.Builder
_ = xml.EscapeText(&b, []byte(s))
return b.String()
}
+131
View File
@@ -0,0 +1,131 @@
package pack
import (
"encoding/binary"
"image/color"
"os"
"path/filepath"
"runtime"
"strings"
"testing"
)
func TestBuildApp(t *testing.T) {
dir := t.TempDir()
base := filepath.Join(dir, "kage-darwin")
baseBytes := []byte("\xcf\xfa\xed\xfeMACHO-BASE-BYTES")
if err := os.WriteFile(base, baseBytes, 0o755); err != nil {
t.Fatal(err)
}
zim := []byte("ZIM-ARCHIVE-PAYLOAD")
out := filepath.Join(dir, "Paulgraham.app")
path, size, err := BuildApp(zim, AppOptions{
Out: out,
Base: base,
Name: "Paul Graham",
ExecName: "paulgraham",
Identifier: "com.kage.paulgraham",
Version: "1.0",
Icon: solid(48, 48, color.RGBA{A: 0xff}),
})
if err != nil {
t.Fatal(err)
}
if path != out {
t.Errorf("path = %q, want %q", path, out)
}
// The bundle skeleton.
exe := filepath.Join(out, "Contents", "MacOS", "paulgraham")
for _, f := range []string{
filepath.Join(out, "Contents", "Info.plist"),
filepath.Join(out, "Contents", "PkgInfo"),
filepath.Join(out, "Contents", "Resources", "icon.icns"),
exe,
} {
if _, err := os.Stat(f); err != nil {
t.Errorf("missing bundle file %s: %v", f, err)
}
}
// The executable is base++zim++trailer, and size reports its length.
exeBytes, err := os.ReadFile(exe)
if err != nil {
t.Fatal(err)
}
if int64(len(exeBytes)) != size {
t.Errorf("size = %d, want %d", size, len(exeBytes))
}
if !strings.HasPrefix(string(exeBytes), string(baseBytes)) {
t.Error("executable does not start with the base bytes")
}
tr := exeBytes[len(exeBytes)-trailerLen:]
if string(tr[:8]) != trailerMagic || string(tr[trailerLen-8:]) != trailerMagic {
t.Error("trailer magic missing from the executable")
}
if got := binary.LittleEndian.Uint64(tr[8:16]); got != uint64(len(zim)) {
t.Errorf("trailer records zim length %d, want %d", got, len(zim))
}
// On a unix host the executable bit must be set so Finder can launch it.
if runtime.GOOS != "windows" {
info, err := os.Stat(exe)
if err != nil {
t.Fatal(err)
}
if info.Mode().Perm()&0o111 == 0 {
t.Errorf("executable mode = %v, want the execute bit set", info.Mode().Perm())
}
}
// Info.plist carries the identity and points at the icon.
plist, err := os.ReadFile(filepath.Join(out, "Contents", "Info.plist"))
if err != nil {
t.Fatal(err)
}
for _, want := range []string{
"<string>paulgraham</string>", // CFBundleExecutable
"<string>Paul Graham</string>", // CFBundleName / DisplayName
"<string>com.kage.paulgraham</string>", // CFBundleIdentifier
"<key>CFBundleIconFile</key>",
"<string>icon</string>",
"NSHighResolutionCapable",
} {
if !strings.Contains(string(plist), want) {
t.Errorf("Info.plist missing %q", want)
}
}
}
func TestBuildAppNoIcon(t *testing.T) {
dir := t.TempDir()
base := filepath.Join(dir, "kage")
if err := os.WriteFile(base, []byte("\xcf\xfa\xed\xfeBASE"), 0o755); err != nil {
t.Fatal(err)
}
out := filepath.Join(dir, "Site.app")
if _, _, err := BuildApp([]byte("ZIM"), AppOptions{Out: out, Base: base, ExecName: "site"}); err != nil {
t.Fatal(err)
}
// No icon means no Resources dir and no icon key in the plist.
if _, err := os.Stat(filepath.Join(out, "Contents", "Resources")); !os.IsNotExist(err) {
t.Errorf("Resources dir should be absent without an icon, got err=%v", err)
}
plist, err := os.ReadFile(filepath.Join(out, "Contents", "Info.plist"))
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(plist), "CFBundleIconFile") {
t.Error("plist names an icon file but none was provided")
}
}
func TestBuildAppRejectsBadOut(t *testing.T) {
if _, _, err := BuildApp([]byte("ZIM"), AppOptions{Out: "noapp", Base: "x"}); err == nil {
t.Error("BuildApp should reject an output path without a .app extension")
}
if _, _, err := BuildApp([]byte("ZIM"), AppOptions{Out: ""}); err == nil {
t.Error("BuildApp should reject an empty output path")
}
}
+156
View File
@@ -0,0 +1,156 @@
package pack
import (
"bytes"
"fmt"
"image"
"image/png"
"os"
"path/filepath"
"strings"
)
// LinuxAppOptions controls how a Linux application directory is assembled around
// a packed viewer.
type LinuxAppOptions struct {
Out string // path to the .AppDir directory
Base string // base kage binary (must be a Linux build); default os.Executable()
Name string // display name shown in menus
ExecName string // base name for the .desktop and icon files
Comment string // optional one-line description for the launcher
Version string // version string recorded in the .desktop
Icon image.Image // optional; written as the launcher icon
}
// BuildAppDir writes an AppImage-style application directory. The layout follows
// the AppDir convention so `appimagetool` can fold it into a single
// double-clickable .AppImage, but it is useful on its own: AppRun is the packed
// viewer, the .desktop file launches it with Terminal=false (no console), and
// the icon gives it a face in the file manager and menus.
//
// It returns the AppDir path, the size of the executable inside it, and whether
// an icon was written (the caller needs that to decide if an .AppImage can be
// built, since AppImage requires one).
func BuildAppDir(zimBytes []byte, opts LinuxAppOptions) (path string, size int64, hasIcon bool, err error) {
if opts.Out == "" {
return "", 0, false, fmt.Errorf("pack: BuildAppDir requires an output path")
}
if !strings.HasSuffix(opts.Out, ".AppDir") {
return "", 0, false, fmt.Errorf("pack: app dir path must end in .AppDir, got %q", opts.Out)
}
base := opts.Base
if base == "" {
exe, e := os.Executable()
if e != nil {
return "", 0, false, fmt.Errorf("pack: locate base binary: %w", e)
}
base = exe
}
baseBytes, err := os.ReadFile(base)
if err != nil {
return "", 0, false, fmt.Errorf("pack: read base binary %q: %w", base, err)
}
execName := opts.ExecName
if execName == "" {
execName = "kage"
}
name := opts.Name
if name == "" {
name = execName
}
if err := os.RemoveAll(opts.Out); err != nil {
return "", 0, false, err
}
if err := os.MkdirAll(opts.Out, 0o755); err != nil {
return "", 0, false, err
}
// AppRun is the AppImage entrypoint; pointing it straight at the packed
// binary keeps the directory to a single executable with no wrapper script.
exe := assemble(baseBytes, zimBytes)
if err := os.WriteFile(filepath.Join(opts.Out, "AppRun"), exe, 0o755); err != nil {
return "", 0, false, err
}
hasIcon = opts.Icon != nil
if hasIcon {
pngBytes, err := encodeIconPNG(opts.Icon, 256)
if err != nil {
return "", 0, false, err
}
// The icon lives at the root under the name the .desktop references, and
// .DirIcon is the AppImage convention for the directory's own icon.
if err := os.WriteFile(filepath.Join(opts.Out, execName+".png"), pngBytes, 0o644); err != nil {
return "", 0, false, err
}
if err := os.WriteFile(filepath.Join(opts.Out, ".DirIcon"), pngBytes, 0o644); err != nil {
return "", 0, false, err
}
}
desktop := desktopEntry(desktopData{
Name: name,
ExecName: execName,
Comment: opts.Comment,
Version: opts.Version,
HasIcon: hasIcon,
})
if err := os.WriteFile(filepath.Join(opts.Out, execName+".desktop"), []byte(desktop), 0o644); err != nil {
return "", 0, false, err
}
return opts.Out, int64(len(exe)), hasIcon, nil
}
type desktopData struct {
Name string
ExecName string
Comment string
Version string
HasIcon bool
}
// desktopEntry renders a freedesktop .desktop launcher. Terminal=false is the
// line that keeps a console from opening, the Linux echo of the macOS .app.
func desktopEntry(d desktopData) string {
var b strings.Builder
b.WriteString("[Desktop Entry]\n")
b.WriteString("Type=Application\n")
b.WriteString("Name=" + desktopValue(d.Name) + "\n")
if d.Comment != "" {
b.WriteString("Comment=" + desktopValue(d.Comment) + "\n")
}
// AppRun is on PATH inside a running AppImage, so Exec names it directly.
b.WriteString("Exec=AppRun\n")
if d.HasIcon {
b.WriteString("Icon=" + d.ExecName + "\n")
}
b.WriteString("Categories=Network;Utility;\n")
b.WriteString("Terminal=false\n")
if d.Version != "" {
b.WriteString("X-AppImage-Version=" + desktopValue(d.Version) + "\n")
}
return b.String()
}
// desktopValue strips the characters that would break a .desktop key=value line
// (newlines and the leading-space/escape pitfalls), which is enough for a name
// or comment drawn from a page title.
func desktopValue(s string) string {
s = strings.ReplaceAll(s, "\n", " ")
s = strings.ReplaceAll(s, "\r", " ")
return strings.TrimSpace(s)
}
// encodeIconPNG scales img to a px-by-px square and encodes it as PNG, the icon
// format the freedesktop world expects.
func encodeIconPNG(img image.Image, px int) ([]byte, error) {
var buf bytes.Buffer
if err := png.Encode(&buf, scaleSquare(img, px)); err != nil {
return nil, fmt.Errorf("pack: encode icon png: %w", err)
}
return buf.Bytes(), nil
}
+125
View File
@@ -0,0 +1,125 @@
package pack
import (
"image/color"
"image/png"
"os"
"path/filepath"
"strings"
"testing"
)
func TestBuildAppDir(t *testing.T) {
dir := t.TempDir()
base := filepath.Join(dir, "kage-linux")
baseBytes := []byte("\x7fELF-LINUX-BASE")
if err := os.WriteFile(base, baseBytes, 0o755); err != nil {
t.Fatal(err)
}
out := filepath.Join(dir, "paulgraham.AppDir")
path, size, hasIcon, err := BuildAppDir([]byte("ZIM-PAYLOAD"), LinuxAppOptions{
Out: out,
Base: base,
Name: "Paul Graham",
ExecName: "paulgraham",
Comment: "Offline mirror",
Version: "1.0",
Icon: solid(64, 64, color.RGBA{B: 0xff, A: 0xff}),
})
if err != nil {
t.Fatal(err)
}
if path != out || !hasIcon {
t.Fatalf("path=%q hasIcon=%v", path, hasIcon)
}
apprun := filepath.Join(out, "AppRun")
for _, f := range []string{
apprun,
filepath.Join(out, "paulgraham.desktop"),
filepath.Join(out, "paulgraham.png"),
filepath.Join(out, ".DirIcon"),
} {
if _, err := os.Stat(f); err != nil {
t.Errorf("missing %s: %v", f, err)
}
}
exe, err := os.ReadFile(apprun)
if err != nil {
t.Fatal(err)
}
if int64(len(exe)) != size {
t.Errorf("size = %d, want %d", size, len(exe))
}
if !strings.HasPrefix(string(exe), string(baseBytes)) {
t.Error("AppRun does not start with the base bytes")
}
tr := exe[len(exe)-trailerLen:]
if string(tr[:8]) != trailerMagic {
t.Error("AppRun missing the trailer")
}
if info, _ := os.Stat(apprun); info.Mode().Perm()&0o111 == 0 {
t.Error("AppRun is not executable")
}
// The icon is a real 256px PNG.
f, err := os.Open(filepath.Join(out, "paulgraham.png"))
if err != nil {
t.Fatal(err)
}
defer func() { _ = f.Close() }()
img, err := png.Decode(f)
if err != nil {
t.Fatalf("icon is not a PNG: %v", err)
}
if img.Bounds().Dx() != 256 {
t.Errorf("icon width = %d, want 256", img.Bounds().Dx())
}
desktop, err := os.ReadFile(filepath.Join(out, "paulgraham.desktop"))
if err != nil {
t.Fatal(err)
}
for _, want := range []string{
"Name=Paul Graham",
"Exec=AppRun",
"Icon=paulgraham",
"Terminal=false",
"Comment=Offline mirror",
} {
if !strings.Contains(string(desktop), want) {
t.Errorf(".desktop missing %q", want)
}
}
}
func TestBuildAppDirNoIcon(t *testing.T) {
dir := t.TempDir()
base := filepath.Join(dir, "kage")
if err := os.WriteFile(base, []byte("\x7fELF"), 0o755); err != nil {
t.Fatal(err)
}
out := filepath.Join(dir, "site.AppDir")
_, _, hasIcon, err := BuildAppDir([]byte("ZIM"), LinuxAppOptions{Out: out, Base: base, ExecName: "site"})
if err != nil {
t.Fatal(err)
}
if hasIcon {
t.Error("hasIcon should be false without an icon")
}
if _, err := os.Stat(filepath.Join(out, "site.png")); !os.IsNotExist(err) {
t.Errorf("icon should be absent, got err=%v", err)
}
desktop, _ := os.ReadFile(filepath.Join(out, "site.desktop"))
if strings.Contains(string(desktop), "Icon=") {
t.Error(".desktop names an icon but none was written")
}
}
func TestBuildAppDirRejectsBadOut(t *testing.T) {
if _, _, _, err := BuildAppDir([]byte("Z"), LinuxAppOptions{Out: "site", Base: "x"}); err == nil {
t.Error("BuildAppDir should reject a path without .AppDir")
}
}
+22 -17
View File
@@ -44,26 +44,31 @@ func BuildBinary(zimBytes []byte, opts BinaryOptions) (string, int64, error) {
return "", 0, fmt.Errorf("pack: read base binary %q: %w", base, err)
}
payload := assemble(baseBytes, zimBytes)
if err := os.WriteFile(opts.Out, payload, 0o755); err != nil {
return opts.Out, 0, err
}
// WriteFile honours the mode only when it creates the file; chmod makes an
// overwrite executable too.
if err := os.Chmod(opts.Out, 0o755); err != nil {
return opts.Out, 0, err
}
return opts.Out, int64(len(payload)), nil
}
// assemble builds the self-contained viewer image: the base executable, then the
// ZIM archive, then the KAGEPCK1 trailer that records the archive length. ELF,
// PE, and Mach-O loaders all ignore trailing bytes, so the result still runs on
// its target OS while Embedded finds the archive at the tail.
func assemble(baseBytes, zimBytes []byte) []byte {
var tr bytes.Buffer
tr.WriteString(trailerMagic)
_ = binary.Write(&tr, binary.LittleEndian, uint64(len(zimBytes)))
tr.WriteString(trailerMagic)
f, err := os.Create(opts.Out)
if err != nil {
return "", 0, err
}
for _, chunk := range [][]byte{baseBytes, zimBytes, tr.Bytes()} {
if _, err := f.Write(chunk); err != nil {
_ = f.Close()
return opts.Out, 0, err
}
}
if err := f.Close(); err != nil {
return opts.Out, 0, err
}
if err := os.Chmod(opts.Out, 0o755); err != nil {
return opts.Out, 0, err
}
return opts.Out, int64(len(baseBytes) + len(zimBytes) + trailerLen), nil
out := make([]byte, 0, len(baseBytes)+len(zimBytes)+tr.Len())
out = append(out, baseBytes...)
out = append(out, zimBytes...)
out = append(out, tr.Bytes()...)
return out
}
+92
View File
@@ -0,0 +1,92 @@
package pack
import (
"bytes"
"encoding/binary"
"fmt"
"image"
"image/png"
xdraw "golang.org/x/image/draw"
)
// An .icns file is a tiny container: the magic "icns", a uint32 big-endian total
// length, then a run of entries. Each entry is a four-byte OSType, a uint32
// big-endian length covering the 8-byte entry header plus the payload, and the
// payload itself. Since OS X 10.7 the payload may be a PNG, which lets us avoid
// the old packed-RGBA formats entirely and just store one PNG per size.
//
// We emit the retina-era PNG OSTypes. macOS picks whichever size it needs for
// the Dock, Finder, and Cmd-Tab, so covering 16 through 1024 keeps the icon
// crisp everywhere without shipping a huge file.
var icnsSizes = []struct {
osType string
px int
}{
{"icp4", 16},
{"icp5", 32},
{"icp6", 64},
{"ic07", 128},
{"ic08", 256},
{"ic09", 512},
{"ic10", 1024},
}
// EncodeICNS renders img into a macOS .icns at every standard size. The source
// is scaled to each size with Catmull-Rom resampling, which keeps a small
// favicon from turning to mush when it is enlarged for the Dock. It returns an
// error only if img is empty or a PNG fails to encode.
func EncodeICNS(img image.Image) ([]byte, error) {
if img == nil || img.Bounds().Empty() {
return nil, fmt.Errorf("pack: icns source image is empty")
}
var body bytes.Buffer
for _, s := range icnsSizes {
scaled := scaleSquare(img, s.px)
var pngBuf bytes.Buffer
if err := png.Encode(&pngBuf, scaled); err != nil {
return nil, fmt.Errorf("pack: encode %s icon: %w", s.osType, err)
}
body.WriteString(s.osType)
writeU32BE(&body, uint32(8+pngBuf.Len()))
body.Write(pngBuf.Bytes())
}
var out bytes.Buffer
out.WriteString("icns")
writeU32BE(&out, uint32(8+body.Len()))
out.Write(body.Bytes())
return out.Bytes(), nil
}
// scaleSquare returns img resampled to a px-by-px RGBA image. A non-square
// source is fitted into the square and centred, so a wide or tall favicon is not
// stretched.
func scaleSquare(img image.Image, px int) image.Image {
dst := image.NewRGBA(image.Rect(0, 0, px, px))
src := img.Bounds()
sw, sh := src.Dx(), src.Dy()
// Scale to fit, preserving aspect ratio.
scale := float64(px) / float64(sw)
if float64(sh)*scale > float64(px) {
scale = float64(px) / float64(sh)
}
dw, dh := int(float64(sw)*scale), int(float64(sh)*scale)
if dw < 1 {
dw = 1
}
if dh < 1 {
dh = 1
}
off := image.Pt((px-dw)/2, (px-dh)/2)
rect := image.Rectangle{Min: off, Max: off.Add(image.Pt(dw, dh))}
xdraw.CatmullRom.Scale(dst, rect, img, src, xdraw.Over, nil)
return dst
}
func writeU32BE(w *bytes.Buffer, v uint32) {
var b [4]byte
binary.BigEndian.PutUint32(b[:], v)
w.Write(b[:])
}
+73
View File
@@ -0,0 +1,73 @@
package pack
import (
"bytes"
"encoding/binary"
"image"
"image/color"
"image/png"
"testing"
)
// solid returns a w-by-h image filled with one colour, enough to exercise the
// encoder without depending on any fixture file.
func solid(w, h int, c color.Color) image.Image {
img := image.NewRGBA(image.Rect(0, 0, w, h))
for y := 0; y < h; y++ {
for x := 0; x < w; x++ {
img.Set(x, y, c)
}
}
return img
}
func TestEncodeICNS(t *testing.T) {
data, err := EncodeICNS(solid(48, 48, color.RGBA{R: 0x33, G: 0x66, B: 0x99, A: 0xff}))
if err != nil {
t.Fatal(err)
}
if string(data[:4]) != "icns" {
t.Fatalf("magic = %q, want icns", data[:4])
}
if got := binary.BigEndian.Uint32(data[4:8]); int(got) != len(data) {
t.Fatalf("header length = %d, want %d", got, len(data))
}
// Walk the entries and confirm each declared OSType is present and its PNG
// decodes to the size it claims.
seen := map[string]int{}
off := 8
for off < len(data) {
osType := string(data[off : off+4])
size := int(binary.BigEndian.Uint32(data[off+4 : off+8]))
if size < 8 || off+size > len(data) {
t.Fatalf("entry %s has bogus length %d at offset %d", osType, size, off)
}
img, err := png.Decode(bytes.NewReader(data[off+8 : off+size]))
if err != nil {
t.Fatalf("entry %s payload is not a PNG: %v", osType, err)
}
seen[osType] = img.Bounds().Dx()
off += size
}
for _, s := range icnsSizes {
px, ok := seen[s.osType]
if !ok {
t.Errorf("missing OSType %s", s.osType)
continue
}
if px != s.px {
t.Errorf("OSType %s is %dpx, want %d", s.osType, px, s.px)
}
}
}
func TestEncodeICNSEmpty(t *testing.T) {
if _, err := EncodeICNS(nil); err == nil {
t.Fatal("EncodeICNS(nil) should error")
}
if _, err := EncodeICNS(image.NewRGBA(image.Rect(0, 0, 0, 0))); err == nil {
t.Fatal("EncodeICNS(empty) should error")
}
}
+143
View File
@@ -0,0 +1,143 @@
package pack
import (
"bytes"
"encoding/binary"
"fmt"
"image"
"io/fs"
"os"
"path/filepath"
"strings"
// Register the decoders image.Decode dispatches to. A favicon is almost
// always one of these; SVG and legacy BMP-in-ICO are handled (or skipped)
// explicitly below.
_ "image/gif"
_ "image/jpeg"
_ "image/png"
)
// iconNames ranks the file names sites use for their icon, best first. A large
// PNG (an apple-touch icon, typically 180px) makes a far better app icon than a
// 16px favicon.ico, so we prefer those even though .ico is the classic name.
var iconNames = []string{
"apple-touch-icon-precomposed.png",
"apple-touch-icon.png",
"icon.png",
"favicon.png",
"favicon.ico",
}
// FindIcon looks through a cloned mirror for the site's icon and decodes it. It
// returns the image, the path it came from (for a friendly log line), and
// ok=false when nothing usable is found, in which case the caller just builds a
// bundle with the default icon. Discovery never fails the pack.
func FindIcon(mirrorDir string) (image.Image, string, bool) {
for _, name := range iconNames {
for _, p := range globIcon(mirrorDir, name) {
if img, err := DecodeIcon(p); err == nil {
return img, p, true
}
}
}
return nil, "", false
}
// globIcon returns every file under dir whose base name equals name, nearest the
// root first. Clones store assets under rewritten paths, so the icon may sit a
// few directories deep rather than at the mirror root.
func globIcon(dir, name string) []string {
var hits []string
_ = filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
if err != nil || d.IsDir() {
return nil
}
if strings.EqualFold(d.Name(), name) {
hits = append(hits, p)
}
return nil
})
// Shallower paths (fewer separators) are likelier to be the real site icon.
for i := 1; i < len(hits); i++ {
for j := i; j > 0 && depth(hits[j]) < depth(hits[j-1]); j-- {
hits[j], hits[j-1] = hits[j-1], hits[j]
}
}
return hits
}
func depth(p string) int { return strings.Count(p, string(filepath.Separator)) }
// DecodeIcon reads an icon file into an image. It handles the stdlib raster
// formats (PNG, JPEG, GIF) directly and unwraps a PNG stored inside a .ico
// container, which is how modern sites ship a high-resolution favicon.ico. A
// legacy BMP-only .ico returns an error rather than a mangled image, so the
// caller falls back to the default icon.
func DecodeIcon(path string) (image.Image, error) {
data, err := os.ReadFile(path)
if err != nil {
return nil, err
}
if isICO(data) {
png, err := largestPNGInICO(data)
if err != nil {
return nil, err
}
data = png
}
img, _, err := image.Decode(bytes.NewReader(data))
if err != nil {
return nil, fmt.Errorf("pack: decode icon %q: %w", path, err)
}
return img, nil
}
// isICO reports whether data begins with an ICONDIR header: reserved 0, type 1
// (icon), and a non-zero image count.
func isICO(data []byte) bool {
return len(data) >= 6 &&
data[0] == 0 && data[1] == 0 &&
data[2] == 1 && data[3] == 0 &&
(uint16(data[4])|uint16(data[5])<<8) > 0
}
var pngMagic = []byte{0x89, 'P', 'N', 'G', '\r', '\n', 0x1a, '\n'}
// largestPNGInICO scans an .ico directory for PNG-encoded entries and returns
// the bytes of the largest one. It ignores BMP entries; if none of the entries
// are PNG it returns an error.
func largestPNGInICO(data []byte) ([]byte, error) {
count := int(binary.LittleEndian.Uint16(data[4:6]))
var best []byte
var bestArea int
for i := 0; i < count; i++ {
e := 6 + i*16
if e+16 > len(data) {
break
}
w, h := int(data[e]), int(data[e+1])
if w == 0 {
w = 256
}
if h == 0 {
h = 256
}
size := int(binary.LittleEndian.Uint32(data[e+8 : e+12]))
off := int(binary.LittleEndian.Uint32(data[e+12 : e+16]))
if off < 0 || size <= 0 || off+size > len(data) {
continue
}
chunk := data[off : off+size]
if !bytes.HasPrefix(chunk, pngMagic) {
continue // a BMP/DIB entry; skip it
}
if w*h > bestArea {
bestArea, best = w*h, chunk
}
}
if best == nil {
return nil, fmt.Errorf("pack: .ico holds no PNG entry")
}
return best, nil
}
+135
View File
@@ -0,0 +1,135 @@
package pack
import (
"bytes"
"encoding/binary"
"image/color"
"image/png"
"os"
"path/filepath"
"testing"
)
// pngBytes encodes a solid square as PNG, used to seed icon fixtures.
func pngBytes(t *testing.T, px int) []byte {
t.Helper()
var buf bytes.Buffer
if err := png.Encode(&buf, solid(px, px, color.RGBA{R: 0x10, G: 0x80, B: 0x40, A: 0xff})); err != nil {
t.Fatal(err)
}
return buf.Bytes()
}
// leWrite appends v to w in little-endian, swallowing the error a bytes.Buffer
// never returns. It keeps the .ico fixtures readable without an errcheck flag on
// every field.
func leWrite(w *bytes.Buffer, v any) { _ = binary.Write(w, binary.LittleEndian, v) }
// icoWithPNG wraps PNG payloads in a minimal .ico container so the ICO path is
// exercised without a binary fixture on disk.
func icoWithPNG(pngs [][]byte) []byte {
var dir, body bytes.Buffer
put := leWrite
put(&dir, uint16(0)) // reserved
put(&dir, uint16(1)) // type: icon
put(&dir, uint16(len(pngs))) // count
offset := 6 + len(pngs)*16
for i, p := range pngs {
// width/height bytes: 0 means 256; use a distinct small size per entry.
dim := byte(16 * (i + 1))
dir.WriteByte(dim) // width
dir.WriteByte(dim) // height
dir.WriteByte(0) // colours
dir.WriteByte(0) // reserved
put(&dir, uint16(1)) // planes
put(&dir, uint16(32)) // bit count
put(&dir, uint32(len(p))) // bytes in resource
put(&dir, uint32(offset)) // offset
offset += len(p)
body.Write(p)
}
return append(dir.Bytes(), body.Bytes()...)
}
func TestDecodeIconPNG(t *testing.T) {
p := filepath.Join(t.TempDir(), "favicon.png")
if err := os.WriteFile(p, pngBytes(t, 64), 0o644); err != nil {
t.Fatal(err)
}
img, err := DecodeIcon(p)
if err != nil {
t.Fatal(err)
}
if img.Bounds().Dx() != 64 {
t.Errorf("decoded width = %d, want 64", img.Bounds().Dx())
}
}
func TestDecodeIconICOWithPNG(t *testing.T) {
// Two PNG entries; the decoder should pick the larger (second, 32px square).
ico := icoWithPNG([][]byte{pngBytes(t, 16), pngBytes(t, 32)})
p := filepath.Join(t.TempDir(), "favicon.ico")
if err := os.WriteFile(p, ico, 0o644); err != nil {
t.Fatal(err)
}
img, err := DecodeIcon(p)
if err != nil {
t.Fatal(err)
}
if img.Bounds().Dx() != 32 {
t.Errorf("decoded width = %d, want the larger 32px entry", img.Bounds().Dx())
}
}
func TestDecodeIconBMPOnlyICOFails(t *testing.T) {
// A one-entry .ico whose payload is not a PNG (here just filler) must error
// so the caller falls back to the default icon.
var ico bytes.Buffer
leWrite(&ico, uint16(0))
leWrite(&ico, uint16(1))
leWrite(&ico, uint16(1))
ico.Write([]byte{32, 32, 0, 0})
leWrite(&ico, uint16(1))
leWrite(&ico, uint16(32))
leWrite(&ico, uint32(4))
leWrite(&ico, uint32(22))
ico.Write([]byte{0x42, 0x4d, 0x00, 0x00}) // "BM" DIB filler
p := filepath.Join(t.TempDir(), "favicon.ico")
if err := os.WriteFile(p, ico.Bytes(), 0o644); err != nil {
t.Fatal(err)
}
if _, err := DecodeIcon(p); err == nil {
t.Fatal("BMP-only .ico should fail to decode")
}
}
func TestFindIconPrefersAppleTouch(t *testing.T) {
dir := t.TempDir()
// A tiny favicon at the root and a bigger apple-touch icon a level down.
if err := os.WriteFile(filepath.Join(dir, "favicon.png"), pngBytes(t, 16), 0o644); err != nil {
t.Fatal(err)
}
sub := filepath.Join(dir, "assets")
if err := os.MkdirAll(sub, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(sub, "apple-touch-icon.png"), pngBytes(t, 180), 0o644); err != nil {
t.Fatal(err)
}
img, src, ok := FindIcon(dir)
if !ok {
t.Fatal("FindIcon found nothing")
}
if filepath.Base(src) != "apple-touch-icon.png" {
t.Errorf("picked %s, want apple-touch-icon.png", filepath.Base(src))
}
if img.Bounds().Dx() != 180 {
t.Errorf("width = %d, want 180", img.Bounds().Dx())
}
}
func TestFindIconNone(t *testing.T) {
if _, _, ok := FindIcon(t.TempDir()); ok {
t.Fatal("FindIcon should report nothing in an empty dir")
}
}
+57
View File
@@ -0,0 +1,57 @@
package pack
import "os"
// Executable-format magic numbers, enough to tell the three desktop targets
// apart by their first bytes. We only need the family (windows, darwin, linux),
// not the architecture, so the smallest distinguishing prefix is plenty.
var (
magicELF = []byte{0x7f, 'E', 'L', 'F'} // Linux, FreeBSD, and other ELF systems
magicPE = []byte{'M', 'Z'} // Windows PE/COFF (DOS stub header)
machOLE64 = []byte{0xcf, 0xfa, 0xed, 0xfe} // Mach-O 64-bit, little-endian (amd64, arm64)
machOLE32 = []byte{0xce, 0xfa, 0xed, 0xfe} // Mach-O 32-bit, little-endian
machOBE64 = []byte{0xfe, 0xed, 0xfa, 0xcf} // Mach-O 64-bit, big-endian
machOBE32 = []byte{0xfe, 0xed, 0xfa, 0xce} // Mach-O 32-bit, big-endian
machOFat = []byte{0xca, 0xfe, 0xba, 0xbe} // Mach-O universal (fat) binary
)
// SniffOS reads the first bytes of an executable and returns the GOOS family it
// was built for: "windows", "darwin", "linux", or "" when the bytes match none
// of them. It is how pack decides whether a cross-built viewer needs a .exe
// suffix and which run hint to print, without trusting the base's file name.
func SniffOS(path string) string {
f, err := os.Open(path)
if err != nil {
return ""
}
defer func() { _ = f.Close() }()
head := make([]byte, 4)
if _, err := f.ReadAt(head, 0); err != nil {
return ""
}
switch {
case hasPrefix(head, magicPE):
return "windows"
case hasPrefix(head, magicELF):
return "linux"
case hasPrefix(head, machOLE64), hasPrefix(head, machOLE32),
hasPrefix(head, machOBE64), hasPrefix(head, machOBE32),
hasPrefix(head, machOFat):
return "darwin"
default:
return ""
}
}
func hasPrefix(b, prefix []byte) bool {
if len(b) < len(prefix) {
return false
}
for i := range prefix {
if b[i] != prefix[i] {
return false
}
}
return true
}
+40
View File
@@ -0,0 +1,40 @@
package pack
import (
"os"
"path/filepath"
"testing"
)
func TestSniffOS(t *testing.T) {
cases := []struct {
name string
head []byte
want string
}{
{"elf", []byte{0x7f, 'E', 'L', 'F', 0x02, 0x01}, "linux"},
{"pe", []byte{'M', 'Z', 0x90, 0x00}, "windows"},
{"macho-le64", []byte{0xcf, 0xfa, 0xed, 0xfe}, "darwin"},
{"macho-le32", []byte{0xce, 0xfa, 0xed, 0xfe}, "darwin"},
{"macho-be64", []byte{0xfe, 0xed, 0xfa, 0xcf}, "darwin"},
{"macho-fat", []byte{0xca, 0xfe, 0xba, 0xbe}, "darwin"},
{"unknown", []byte{0x00, 0x01, 0x02, 0x03}, ""},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
p := filepath.Join(t.TempDir(), tc.name)
if err := os.WriteFile(p, tc.head, 0o644); err != nil {
t.Fatal(err)
}
if got := SniffOS(p); got != tc.want {
t.Errorf("SniffOS(%s) = %q, want %q", tc.name, got, tc.want)
}
})
}
}
func TestSniffOSMissingFile(t *testing.T) {
if got := SniffOS(filepath.Join(t.TempDir(), "nope")); got != "" {
t.Errorf("SniffOS(missing) = %q, want empty", got)
}
}