Crawling keyed off the raw URL, so the same page reached over http and
https, or as /index.html versus /, was a different frontier entry that
nonetheless wrote to the same file. A clone of paulgraham.com did 948
render passes for 474 files. Key pages and assets by the local path they
write instead, and collapse a directory-index document to its directory,
so each page is fetched exactly once.
Add --refresh to re-render a mirror in place (re-fetch every page, keep
the directory, overwrite) and make --no-resume truly stateless by not
persisting state.json. The default remains a resumable, idempotent crawl
that skips work already on disk.
Duc-Tam Nguyen
a871265cbe