A developer.apple.com crawl came back at 19 GB, 18 of it assets, most of
that the site's own videos, .dmg/.pkg installers, and PDF manuals pulled
from a couple dozen hosts including unrelated third parties. None of it
helps read the docs offline. This changes what kage localizes by default.
Bulk media, installers, archives, and PDFs are left pointing at their
live URL instead of downloaded. The decision is made from the URL alone,
so the rewritten HTML simply keeps the remote link. --keep-media restores
the old behavior and --skip-ext adds more extensions to leave remote.
Assets are localized only from the seed's registrable domain by default,
so www.apple.com and images.apple.com still come along but a separate
brand domain or an off-topic third party does not. --all-asset-hosts goes
back to downloading from any host.
The size cap was also truncating instead of skipping: it wrapped the body
in a LimitReader, so an over-cap file was saved as exactly the first N MB
of itself, a corrupt fragment that would never play or run. On the apple
crawl that was around a gigabyte of half-downloaded WWDC videos. kage now
checks the response size and leaves an over-cap asset out of the mirror.
Duc-Tam Nguyen
249761f29f