Duc-Tam Nguyen
e6afa91e09
kage renders every page in headless Chrome, snapshots the final DOM, strips all JavaScript, and localises CSS, images, and fonts so a site can be browsed offline as a plain folder of files. The engine is split into small packages: urlx deterministic URL to local-path mapping and scope rules sanitize remove scripts, on* handlers, and javascript: URLs asset rewrite HTML and CSS references, download assets browser headless Chrome pool over the DevTools protocol robots robots.txt matcher clone the orchestrator: a polite resumable breadth-first crawl The cli package wires a cobra and fang command surface with two commands, clone and serve. Every pure package has table tests; the browser and clone packages add Chrome-driven end-to-end tests that skip when no browser is present or under -short. CI runs gofmt, vet, build, race tests, golangci-lint, govulncheck, and a tidy check on Linux and macOS. A goreleaser config fans one tag out to archives, deb/rpm/apk, a Chromium-bundled GHCR image, and the package managers. A tago docs site builds to Pages and Cloudflare.
38 lines
800 B
Go
38 lines
800 B
Go
package clone
|
|
|
|
import "sync/atomic"
|
|
|
|
// stats are the live counters of a run, read by the CLI's progress ticker.
|
|
type stats struct {
|
|
pages atomic.Int64
|
|
assets atomic.Int64
|
|
pageErrors atomic.Int64
|
|
assetErrors atomic.Int64
|
|
skipped atomic.Int64 // robots-disallowed or out of budget
|
|
}
|
|
|
|
// Progress is a snapshot of a run for display.
|
|
type Progress struct {
|
|
Pages int64
|
|
Assets int64
|
|
PageErrors int64
|
|
AssetErrors int64
|
|
Skipped int64
|
|
}
|
|
|
|
func (s *stats) snapshot() Progress {
|
|
return Progress{
|
|
Pages: s.pages.Load(),
|
|
Assets: s.assets.Load(),
|
|
PageErrors: s.pageErrors.Load(),
|
|
AssetErrors: s.assetErrors.Load(),
|
|
Skipped: s.skipped.Load(),
|
|
}
|
|
}
|
|
|
|
// Result is the final outcome returned by Run.
|
|
type Result struct {
|
|
Progress
|
|
OutDir string
|
|
}
|