frostbyte_neo/kage
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

Add the clone engine, CLI, tests, CI, and docs

Browse Source 
kage renders every page in headless Chrome, snapshots the final
DOM, strips all JavaScript, and localises CSS, images, and fonts
so a site can be browsed offline as a plain folder of files.

The engine is split into small packages:

  urlx      deterministic URL to local-path mapping and scope rules
  sanitize  remove scripts, on* handlers, and javascript: URLs
  asset     rewrite HTML and CSS references, download assets
  browser   headless Chrome pool over the DevTools protocol
  robots    robots.txt matcher
  clone     the orchestrator: a polite resumable breadth-first crawl

The cli package wires a cobra and fang command surface with two
commands, clone and serve. Every pure package has table tests; the
browser and clone packages add Chrome-driven end-to-end tests that
skip when no browser is present or under -short.

CI runs gofmt, vet, build, race tests, golangci-lint, govulncheck,
and a tidy check on Linux and macOS. A goreleaser config fans one
tag out to archives, deb/rpm/apk, a Chromium-bundled GHCR image,
and the package managers. A tago docs site builds to Pages and
Cloudflare.
This commit is contained in:
Duc-Tam Nguyen
2026-06-14 18:22:25 +07:00
parent f7c5a9a1c6
commit e6afa91e09
48 changed files with 4697 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/content/reference/cli.md
+85
View File
@@ -0,0 +1,85 @@
---
title: "CLI reference"
description: "Every kage command and flag."
weight: 10
---
```
kage [command] [flags]
```
Two commands: `clone` fetches a site into an offline folder, `serve` previews
one. Run `kage <command> --help` for the canonical, up-to-date list.
## kage clone
```
kage clone <url> [flags]
```
Renders each page in headless Chrome, strips all JavaScript, localises CSS,
images, and fonts, and writes a browsable mirror to `<out>/<host>/`.
### Output
| Flag | Default | Meaning |
|------|---------|---------|
| `-o, --out` | `kage-out` | Output root; the mirror lands in `<out>/<host>/` |
| `--reserved` | `_kage` | Reserved directory name for assets and crawl state |
| `-f, --force` | `false` | Delete any existing mirror for the host before crawling |
| `--no-resume` | `false` | Do not read or write resume state |
### Scope
| Flag | Default | Meaning |
|------|---------|---------|
| `-p, --max-pages` | `0` | Stop after N pages (0 = unlimited) |
| `-d, --max-depth` | `0` | Link-follow depth cap (0 = unlimited) |
| `--scope-prefix` | | Only crawl pages whose path starts with this prefix |
| `--subdomains` | `false` | Treat subdomains of the seed host as in scope |
| `--exclude` | | Path prefixes to skip (repeatable) |
| `--traversal` | `bfs` | Frontier order: `bfs` or `dfs` |
### Politeness
| Flag | Default | Meaning |
|------|---------|---------|
| `--no-robots` | `false` | Ignore `robots.txt` |
| `--no-sitemap` | `false` | Do not seed URLs from `sitemap.xml` |
| `--user-agent` | Chrome UA | User-Agent for asset and robots fetches |
### Rendering
| Flag | Default | Meaning |
|------|---------|---------|
| `--scroll` | `false` | Auto-scroll each page to trigger lazy loading |
| `--settle` | `1.5s` | Network-idle quiet period before snapshotting the DOM |
| `--render-timeout` | `30s` | Hard cap per page render |
| `--headful` | `false` | Run Chrome with a visible window (debugging) |
| `--chrome` | | Path to the Chrome/Chromium binary |
| `--control-url` | | Attach to an existing Chrome DevTools endpoint |
| `--keep-noscript` | `false` | Unwrap `<noscript>` content instead of dropping it |
### Concurrency and limits
| Flag | Default | Meaning |
|------|---------|---------|
| `--workers` | `4` | Concurrent page render workers |
| `--asset-workers` | `8` | Concurrent asset download workers |
| `--browser-pages` | `4` | Chrome page-pool size |
| `--max-asset-mb` | `25` | Skip assets larger than N MB |
| `--timeout` | `30s` | Per-request timeout |
| `-q, --quiet` | `false` | Suppress per-page progress lines |
## kage serve
```
kage serve [dir] [flags]
```
Runs a local static file server over a cloned folder. With no `dir`, serves the
current directory.
| Flag | Default | Meaning |
|------|---------|---------|
| `-a, --addr` | `127.0.0.1:8800` | Address to listen on |