Xirui 8bc9c8bea1 feat: 1. add a crawl delay function to honor the Crawl-delay directive parsed from robots.txt during clone. (#57) ...

2. add --craw-delay flag to specify/override robots directive.

2026-06-22 16:16:54 +07:00

..

cloner_test.go

feat: 1. add a crawl delay function to honor the Crawl-delay directive parsed from robots.txt during clone. (#57)

2026-06-22 16:16:54 +07:00

cloner.go

feat: 1. add a crawl delay function to honor the Crawl-delay directive parsed from robots.txt during clone. (#57)

2026-06-22 16:16:54 +07:00

config.go

feat: 1. add a crawl delay function to honor the Crawl-delay directive parsed from robots.txt during clone. (#57)

2026-06-22 16:16:54 +07:00

dedup_test.go

Count real pages and store identical pages once (#20)

2026-06-15 19:07:49 +07:00

frontier_test.go

Add the clone engine, CLI, tests, CI, and docs

2026-06-14 18:22:25 +07:00

frontier.go

Add the clone engine, CLI, tests, CI, and docs

2026-06-14 18:22:25 +07:00

sitemap.go

Add the clone engine, CLI, tests, CI, and docs

2026-06-14 18:22:25 +07:00

stats.go

Leave bulk and off-domain assets remote, skip over-cap downloads

2026-06-15 20:27:47 +07:00

wantasset_test.go

Leave bulk and off-domain assets remote, skip over-cap downloads

2026-06-15 20:27:47 +07:00