Duc-Tam Nguyen
e6afa91e09
kage renders every page in headless Chrome, snapshots the final DOM, strips all JavaScript, and localises CSS, images, and fonts so a site can be browsed offline as a plain folder of files. The engine is split into small packages: urlx deterministic URL to local-path mapping and scope rules sanitize remove scripts, on* handlers, and javascript: URLs asset rewrite HTML and CSS references, download assets browser headless Chrome pool over the DevTools protocol robots robots.txt matcher clone the orchestrator: a polite resumable breadth-first crawl The cli package wires a cobra and fang command surface with two commands, clone and serve. Every pure package has table tests; the browser and clone packages add Chrome-driven end-to-end tests that skip when no browser is present or under -short. CI runs gofmt, vet, build, race tests, golangci-lint, govulncheck, and a tidy check on Linux and macOS. A goreleaser config fans one tag out to archives, deb/rpm/apk, a Chromium-bundled GHCR image, and the package managers. A tago docs site builds to Pages and Cloudflare.
74 lines
1.9 KiB
Markdown
74 lines
1.9 KiB
Markdown
---
|
|
title: "Scoping a crawl"
|
|
description: "Keep a clone inside the lines with depth, page, prefix, subdomain, and exclude controls."
|
|
weight: 10
|
|
---
|
|
|
|
By default kage crawls every in-scope page it can reach from the seed, staying on
|
|
the seed's host. On a large site that can be a lot of pages. These flags bound
|
|
the crawl.
|
|
|
|
## Limit by count and depth
|
|
|
|
```bash
|
|
# Stop after 200 pages
|
|
kage clone example.com --max-pages 200
|
|
|
|
# Only follow links three hops from the seed
|
|
kage clone example.com --max-depth 3
|
|
```
|
|
|
|
`--max-depth 0` (the default) means unlimited depth; `--max-pages 0` means
|
|
unlimited pages. Combine them to put a hard ceiling on a run.
|
|
|
|
## Limit by path
|
|
|
|
To clone just one section of a site, restrict the crawl to a path prefix:
|
|
|
|
```bash
|
|
kage clone example.com --scope-prefix /docs
|
|
```
|
|
|
|
Only pages whose path starts with `/docs` are followed. Assets are still fetched
|
|
from wherever the page references them, so the section renders correctly.
|
|
|
|
To skip parts of a site, exclude path prefixes (repeatable):
|
|
|
|
```bash
|
|
kage clone example.com --exclude /archive --exclude /tags
|
|
```
|
|
|
|
## Subdomains
|
|
|
|
By default a clone stays on the exact seed host. To treat subdomains of the seed
|
|
as in scope, add `--subdomains`:
|
|
|
|
```bash
|
|
kage clone example.com --subdomains
|
|
```
|
|
|
|
Now `blog.example.com` and `docs.example.com` are crawled too, each landing
|
|
under its own host directory inside the mirror.
|
|
|
|
## Politeness
|
|
|
|
kage honours `robots.txt` by default and seeds itself from `sitemap.xml`. If you
|
|
are cloning a site you control, or you have a reason to ignore the robots rules,
|
|
you can turn them off, but do so responsibly:
|
|
|
|
```bash
|
|
kage clone example.com --no-robots --no-sitemap
|
|
```
|
|
|
|
## Lazy-loaded media
|
|
|
|
Sites that load images as you scroll will only have their above-the-fold images
|
|
captured unless you tell kage to scroll each page:
|
|
|
|
```bash
|
|
kage clone example.com --scroll
|
|
```
|
|
|
|
This makes each render a little slower but captures media that only loads on
|
|
view.
|