frostbyte_neo/kage
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

Dedup pages and assets by output path, add --refresh

Browse Source 
Crawling keyed off the raw URL, so the same page reached over http and
https, or as /index.html versus /, was a different frontier entry that
nonetheless wrote to the same file. A clone of paulgraham.com did 948
render passes for 474 files. Key pages and assets by the local path they
write instead, and collapse a directory-index document to its directory,
so each page is fetched exactly once.

Add --refresh to re-render a mirror in place (re-fetch every page, keep
the directory, overwrite) and make --no-resume truly stateless by not
persisting state.json. The default remains a resumable, idempotent crawl
that skips work already on disk.
This commit is contained in:
Duc-Tam Nguyen
2026-06-14 19:06:28 +07:00
parent ea70aa1342
commit a871265cbe
9 changed files with 176 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/content/reference/cli.md
+2 -1
View File
@@ -24,9 +24,10 @@ images, and fonts, and writes a browsable mirror to `<out>/<host>/`.
| Flag | Default | Meaning |
|------|---------|---------|
| `-o, --out` | `kage-out` | Output root; the mirror lands in `<out>/<host>/` |
| `-o, --out` | `$HOME/data/kage` | Output root; the mirror lands in `<out>/<host>/` |
| `--reserved` | `_kage` | Reserved directory name for assets and crawl state |
| `-f, --force` | `false` | Delete any existing mirror for the host before crawling |
| `--refresh` | `false` | Re-render every page in place to pull in changed content |
| `--no-resume` | `false` | Do not read or write resume state |
### Scope