Two related changes for crawling faceted sites, where one path spawns
thousands of ?q=.../?page=... URLs that all render the same page.
The progress line was counting every written file, so the page number ran
far ahead of the site's real size. It now shows distinct URL paths as
"pages" and folds the query-string permutations into a separate "variants"
count, so the live counter tracks real pages and is easy to read.
Pages with identical bytes are now stored once. The first page with a given
content is written normally; a later page with the same bytes becomes a hard
link to it, so duplicate content never takes disk twice. Links still resolve
because each variant keeps its own path. When hard links are unsupported the
bytes are written, so correctness never depends on the link. The summary
reports how many pages were deduped.
Tam Nguyen Duc
7dc3621ced