frostbyte_neo/kage
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

Report crawl errors clearly and retry transient ones

Browse Source 
The crawl printed asset failures as "asset error <url>: status 403 for
<url>", repeating the URL and saying nothing about which page wanted the
file or whether the failure was worth worrying about. The final summary then
collapsed everything into a single error count.

Give failures a classified reason (HTTP 403 Forbidden, timed out, ...), name
the page that referenced the asset, and list what went wrong in the summary
instead of only counting it. Failures are collected during the run and capped
so a broken site cannot grow the list without bound.

Retry transient failures (403/429, 5xx, network blips) with a short backoff.
Bot-protection in front of a site often rejects the first request of a burst
but serves a retry fine, which is exactly what cost us stylesheets on a busy
crawl. Permanent failures (404, 401, ...) are not retried.
This commit is contained in:
Duc-Tam Nguyen
2026-06-15 12:25:05 +07:00
parent dab6c11ea8
commit d59c85afc8
1 changed files with 4 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+4 -19
View File
@@ -8,31 +8,16 @@ All notable changes to kage are recorded here. The format follows
### Security
- Chrome now keeps its sandbox on by default. It was previously launched with
`--no-sandbox` unconditionally, which removed Chrome's main line of defense
when rendering pages from the open web (reported in #10). The sandbox is now
dropped only where it genuinely cannot run: inside a container, or when
running as root. `KAGE_NO_SANDBOX=1` forces it off and `KAGE_NO_SANDBOX=0`
forces it on, and the choice is logged so it is never silent.
- Chrome now keeps its sandbox on by default. It was previously launched with `--no-sandbox` unconditionally, which removed Chrome's main line of defense when rendering pages from the open web (reported in #10). The sandbox is now dropped only where it genuinely cannot run: inside a container, or when running as root, and the choice is logged so it is never silent.
### Added
- Container-aware Chrome flags. kage detects a container from the `IN_DOCKER`
environment variable or a `/.dockerenv` marker and, only there, drops the
sandbox and adds `--disable-dev-shm-usage` (the default 64 MB `/dev/shm` is
too small for Chrome on large pages). Outside a container the faster shared
memory is left in place.
- Asset downloads retry on a transient failure (a 403/429, a 5xx, or a network
blip) with a short backoff, recovering files that bot-protection rejects on
the first request of a burst. Permanent failures (404, 401, …) are not
retried.
- Container-aware Chrome flags. kage detects a container from the `IN_DOCKER` environment variable or a `/.dockerenv` marker and, only there, drops the sandbox and adds `--disable-dev-shm-usage` (the default 64 MB `/dev/shm` is too small for Chrome on large pages). Outside a container the faster shared memory is left in place.
- Asset downloads retry on a transient failure (a 403/429, a 5xx, or a network blip) with a short backoff, recovering files that bot-protection rejects on the first request of a burst. Permanent failures (404, 401, ...) are not retried.
### Changed
- Clearer crawl error reporting. Each failure is logged with a classified
reason (`HTTP 403 Forbidden`, `timed out`, …), the URL, and the page that
referenced it, and the end-of-run summary lists what went wrong instead of
printing only a count.
- Clearer crawl error reporting. Each failure is logged with a classified reason (`HTTP 403 Forbidden`, `timed out`, ...), the URL, and the page that referenced it, and the end-of-run summary lists what went wrong instead of printing only a count.
## [0.1.1] - 2026-06-14