frostbyte_neo/kage
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

Dedup pages and assets by output path, add --refresh

Browse Source 
Crawling keyed off the raw URL, so the same page reached over http and
https, or as /index.html versus /, was a different frontier entry that
nonetheless wrote to the same file. A clone of paulgraham.com did 948
render passes for 474 files. Key pages and assets by the local path they
write instead, and collapse a directory-index document to its directory,
so each page is fetched exactly once.

Add --refresh to re-render a mirror in place (re-fetch every page, keep
the directory, overwrite) and make --no-resume truly stateless by not
persisting state.json. The default remains a resumable, idempotent crawl
that skips work already on disk.
This commit is contained in:
Duc-Tam Nguyen
2026-06-14 19:06:28 +07:00
parent ea70aa1342
commit a871265cbe
9 changed files with 176 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/content/reference/cli.md
+2 -1
View File
@@ -24,9 +24,10 @@ images, and fonts, and writes a browsable mirror to `<out>/<host>/`.
| Flag | Default | Meaning |
|------|---------|---------|
| `-o, --out` | `kage-out` | Output root; the mirror lands in `<out>/<host>/` |
| `-o, --out` | `$HOME/data/kage` | Output root; the mirror lands in `<out>/<host>/` |
| `--reserved` | `_kage` | Reserved directory name for assets and crawl state |
| `-f, --force` | `false` | Delete any existing mirror for the host before crawling |
| `--refresh` | `false` | Re-render every page in place to pull in changed content |
| `--no-resume` | `false` | Do not read or write resume state |
### Scope
docs/content/reference/configuration.md
+17 -2
View File
@@ -20,7 +20,8 @@ kage's launcher can download a private copy of Chromium on first use.
## Output layout
A clone of `example.com` lands under `kage-out/example.com/`:
A clone of `example.com` lands under `$HOME/data/kage/example.com/` (override the
root with `-o/--out`):
```
kage-out/example.com/
@@ -50,5 +51,19 @@ Key points:
`style.css?v=3` is saved with a short hash suffix so two versions never
collide.
- **State lives in the mirror.** `_kage/state.json` records every page written,
which is what makes `--resume` able to skip completed work. Rename the reserved
which is what lets a repeated run skip completed work. Rename the reserved
directory with `--reserved` if `_kage` would clash with a real path on the site.
## Resume, refresh, and re-crawl
A clone is idempotent: every page is keyed by the file it writes, so the same
page reached over `http` and `https`, with or without a trailing slash, or as
`/index.html` versus `/`, is fetched exactly once. Re-running picks the work back
up rather than starting over.
| You want to… | Use | What happens |
|--------------|-----|--------------|
| Continue an interrupted crawl | *(default)* | Loads `state.json`, skips pages already written, fetches only what is missing |
| Pull in content that changed on the site | `--refresh` | Keeps the mirror, re-renders every page in place, overwrites with the new DOM |
| Start completely clean | `--force` | Deletes the host's mirror, then crawls from scratch |
| Run once and leave no trace | `--no-resume` | Skips nothing, writes no `state.json` |