feat: 1. add a crawl delay function to honor the Crawl-delay directive parsed from robots.txt during clone. (#57)

2. add --craw-delay flag to specify/override robots directive.
2026-06-22 21:16:54 +12:00
parent 28d08ade88
commit 8bc9c8bea1
8 changed files with 96 additions and 3 deletions
@@ -47,6 +47,7 @@ images, and fonts, and writes a browsable mirror to `<out>/<host>/`.
 | Flag | Default | Meaning |
 |------|---------|---------|
 | `--no-robots` | `false` | Ignore `robots.txt` |
+| `--crawl-delay` | `0s` | Override robots.txt `Crawl-delay` between page starts (0 = use robots.txt) |
 | `--no-sitemap` | `false` | Do not seed URLs from `sitemap.xml` |
 | `--user-agent` | Chrome UA | User-Agent for asset and robots fetches |