Merge pull request #40 from tamnd/release-0.3.4

Cut the v0.3.4 release notes

CHANGELOG.md

@@ -6,6 +6,25 @@ All notable changes to kage are recorded here. The format follows
 
 ## [Unreleased]
 
+## [0.3.4] - 2026-06-17
+
+### Fixed
+
+- `kage serve` now stops on Ctrl-C instead of ignoring it.
+  The preview server was started with a blocking call that never watched for an interrupt, so the only way to stop it was to kill the process.
+  kage now shuts the server down gracefully on an interrupt or a `SIGTERM`, with a short timeout before it forces the listener closed, so a preview exits cleanly. Thanks to Xirui Wang (#35) and Kaidi Zhao (#38).
+- A page whose JavaScript builds a deeply nested object graph no longer fails to clone.
+  Chrome's DevTools Protocol returns "Object reference chain is too long" while loading such a page, but the HTML has already loaded and the error is only about Chrome's internal object tracking, not the document.
+  kage now recognises that specific error and finishes rendering the page instead of dropping it (reported in #36). Thanks to Gautam Kumar (#39).
+
+## [0.3.3] - 2026-06-16
+
+### Fixed
+
+- Chrome no longer downloads a file to your Downloads folder when a crawl follows a link that turns out to be a binary (reported in #32).
+  An extensionless link is queued as a page, so the page worker navigated to it in Chrome, and a link that served a zip or a CSV made Chrome save the file to `~/Downloads`, a surprise side effect of a clone.
+  kage now denies Chrome-initiated downloads browser-wide, since every asset is fetched through kage's own downloader, and detects a navigation whose response is not HTML and reroutes that URL to the asset downloader, where the size and media policy decides whether to localise it or leave it on the live web.
+
 ## [0.3.2] - 2026-06-16
 
 ### Fixed
@@ -205,7 +224,9 @@ can browse offline, with every script stripped out.
   a multi-arch container image on GHCR (Chromium bundled), checksums, SBOMs, and
   a cosign signature, all cut from one version tag by GoReleaser.
 
-[Unreleased]: https://github.com/tamnd/kage/compare/v0.3.2...HEAD
+[Unreleased]: https://github.com/tamnd/kage/compare/v0.3.4...HEAD
+[0.3.4]: https://github.com/tamnd/kage/compare/v0.3.3...v0.3.4
+[0.3.3]: https://github.com/tamnd/kage/compare/v0.3.2...v0.3.3
 [0.3.2]: https://github.com/tamnd/kage/compare/v0.3.1...v0.3.2
 [0.3.1]: https://github.com/tamnd/kage/compare/v0.3.0...v0.3.1
 [0.3.0]: https://github.com/tamnd/kage/compare/v0.2.1...v0.3.0

browser/pool.go

@@ -67,6 +67,21 @@ type RenderResult struct {
 	Title    string
 }
 
+// ErrNotHTML reports that a URL kage tried to render as a page is not HTML: the
+// server returned some other content type (a zip, a CSV, a PDF, a bare image).
+// Such a URL reaches the page worker when its link carried no file extension to
+// classify it by. The caller reroutes it to the asset downloader, where the
+// asset policy decides whether to localise or leave it remote, instead of saving
+// an empty or broken page or letting Chrome download it (issue #32).
+type ErrNotHTML struct {
+	URL         string
+	ContentType string
+}
+
+func (e *ErrNotHTML) Error() string {
+	return fmt.Sprintf("not HTML (%s): %s", e.ContentType, e.URL)
+}
+
 // Render navigates to rawURL, lets it settle, and returns the final rendered
 // HTML. It acquires a page slot from the pool and releases it when done.
 func (p *Pool) Render(ctx context.Context, rawURL string) (RenderResult, error) {
@@ -90,11 +105,33 @@ func (p *Pool) Render(ctx context.Context, rawURL string) (RenderResult, error)
 
 	page = page.Context(ctx).Timeout(p.opts.RenderTimeout)
 
-	if err := page.Navigate(rawURL); err != nil {
-		return RenderResult{}, fmt.Errorf("navigate %s: %w", rawURL, err)
+	// Watch the main document's response so a navigation that turns out to be a
+	// non-HTML resource (a zip, a CSV, a bare image) is caught and handed back for
+	// the asset downloader, rather than rendered as a broken page or, with downloads
+	// denied, left as an aborted navigation (issue #32). The content type arrives in
+	// the response headers whether Chrome renders the body or aborts it as a denied
+	// download, so this catches both.
+	mainContentType := watchMainDocument(page)
+
+	navErr := page.Navigate(rawURL)
+	// A denied download aborts the navigation, so inspect the captured content type
+	// before treating a navigation error as a failure. waitFor gives the response
+	// event a brief moment to be processed; for an HTML page it returns at once.
+	if ct := waitFor(ctx, mainContentType, 2*time.Second); ct != "" && !isHTML(ct) {
+		return RenderResult{}, &ErrNotHTML{URL: rawURL, ContentType: ct}
+	}
+	if navErr != nil {
+		return RenderResult{}, fmt.Errorf("navigate %s: %w", rawURL, navErr)
 	}
 	if err := page.WaitLoad(); err != nil {
-		return RenderResult{}, fmt.Errorf("wait load %s: %w", rawURL, err)
+		// Chrome's DevTools Protocol may return "Object reference chain is too
+		// long" when a page's JavaScript builds deeply nested object graphs.
+		// The page has still loaded its HTML — the error is only about Chrome's
+		// internal object tracking, not about the document. Log the warning and
+		// continue rendering rather than failing the entire page (issue #36).
+		if !isObjRefChainError(err) {
+			return RenderResult{}, fmt.Errorf("wait load %s: %w", rawURL, err)
+		}
 	}
 	settle(page, p.opts.Settle)
 	if p.opts.Scroll {
@@ -169,6 +206,21 @@ func (p *Pool) getBrowser() (*rod.Browser, error) {
 	if err := b.Connect(); err != nil {
 		return nil, fmt.Errorf("connect Chrome: %w", err)
 	}
+
+	// kage never wants Chrome to write a file to disk. Every asset is fetched
+	// through kage's own downloader, which applies the size and media policy, so a
+	// Chrome-initiated download is only ever an accident: navigating an <a> link
+	// that turns out to be a binary (a zip, an installer, a CSV) makes Chrome save
+	// it to the user's Downloads folder, a surprise side effect of a crawl
+	// (issue #32). Denying downloads browser-wide stops that. The navigation is
+	// aborted instead, and Render's non-HTML detection reroutes the URL through the
+	// asset downloader, where the asset policy decides its fate. This is
+	// best-effort: if the call is unsupported, the non-HTML detection still keeps
+	// the binary out of the saved mirror.
+	_ = proto.BrowserSetDownloadBehavior{
+		Behavior: proto.BrowserSetDownloadBehaviorBehaviorDeny,
+	}.Call(b)
+
 	p.browser = b
 	return b, nil
 }
@@ -321,6 +373,83 @@ func envBool(name string) (val, ok bool) {
 	}
 }
 
+// watchMainDocument subscribes to network responses and returns an accessor for
+// the main document's content type. The first Document-type response is the main
+// frame's navigation; later Document responses are sub-frames (iframes), whose
+// type kage does not police, so only the first is kept. The accessor is safe to
+// call from another goroutine. Any setup error leaves the accessor returning "",
+// which the caller reads as "unknown, render normally".
+func watchMainDocument(page *rod.Page) func() string {
+	var (
+		mu sync.Mutex
+		ct string
+	)
+	if err := (proto.NetworkEnable{}).Call(page); err != nil {
+		return func() string { return "" }
+	}
+	wait := page.EachEvent(func(e *proto.NetworkResponseReceived) {
+		if e.Type != proto.NetworkResourceTypeDocument || e.Response == nil {
+			return
+		}
+		mu.Lock()
+		if ct == "" {
+			ct = e.Response.MIMEType
+		}
+		mu.Unlock()
+	})
+	// EachEvent's wait blocks until the page context ends, draining events as they
+	// arrive; run it for the page's lifetime. The deferred page.Close in Render
+	// cancels the context and unblocks it.
+	go wait()
+	return func() string {
+		mu.Lock()
+		defer mu.Unlock()
+		return ct
+	}
+}
+
+// waitFor polls get until it returns a non-empty value, the deadline passes, or
+// the context is cancelled, then returns whatever it last saw. It exists because
+// the network response is processed on another goroutine, so the value may not be
+// set the instant Navigate returns; an HTML page sets it within a few
+// milliseconds, while a never-arriving response simply waits out the deadline.
+func waitFor(ctx context.Context, get func() string, deadline time.Duration) string {
+	const step = 20 * time.Millisecond
+	for waited := time.Duration(0); waited < deadline; waited += step {
+		if v := get(); v != "" {
+			return v
+		}
+		select {
+		case <-ctx.Done():
+			return get()
+		case <-time.After(step):
+		}
+	}
+	return get()
+}
+
+// isHTML reports whether a document content type is one kage renders and saves as
+// a page. HTML and XHTML qualify; an empty type is treated as HTML so an unlabelled
+// response still renders. Anything else (a zip, a CSV, a PDF, a bare image or
+// JSON) is an asset that reached the page worker because its link carried no file
+// extension to classify it by.
+func isHTML(contentType string) bool {
+	mt := strings.ToLower(strings.TrimSpace(contentType))
+	if i := strings.IndexByte(mt, ';'); i >= 0 {
+		mt = strings.TrimSpace(mt[:i])
+	}
+	return mt == "" || mt == "text/html" || mt == "application/xhtml+xml"
+}
+
+// isObjRefChainError reports whether err is the Chrome DevTools Protocol error
+// "Object reference chain is too long" (code -32000). This surfaces when a
+// page's JavaScript builds deeply nested object graphs. The page has still
+// loaded — Chrome's internal state tracking hit a limit, not the document
+// itself (issue #36).
+func isObjRefChainError(err error) bool {
+	return err != nil && strings.Contains(err.Error(), "Object reference chain is too long")
+}
+
 // settle waits for the network to go quiet for d, recovering from any rod
 // panic and capping the wait so a chatty page can never hang the worker.
 func settle(page *rod.Page, d time.Duration) {

browser/pool_test.go

@@ -2,6 +2,7 @@ package browser
 
 import (
 	"context"
+	"errors"
 	"net/http"
 	"net/http/httptest"
 	"os"
@@ -115,3 +116,86 @@ func TestRenderCapturesFinalDOM(t *testing.T) {
 		t.Errorf("render did not capture the JS-built DOM:\n%s", res.HTML)
 	}
 }
+
+func TestIsHTML(t *testing.T) {
+	cases := []struct {
+		ct   string
+		want bool
+	}{
+		{"text/html", true},
+		{"text/html; charset=utf-8", true},
+		{"TEXT/HTML", true},
+		{"  text/html ", true},
+		{"application/xhtml+xml", true},
+		{"", true}, // unknown: render rather than misclassify
+		{"application/zip", false},
+		{"text/csv", false},
+		{"application/pdf", false},
+		{"image/png", false},
+		{"application/json", false},
+		{"application/octet-stream", false},
+	}
+	for _, c := range cases {
+		if got := isHTML(c.ct); got != c.want {
+			t.Errorf("isHTML(%q) = %v, want %v", c.ct, got, c.want)
+		}
+	}
+}
+
+func TestRenderRoutesNonHTML(t *testing.T) {
+	if testing.Short() {
+		t.Skip("render test drives Chrome; skipped under -short")
+	}
+	if _, ok := LookChrome(); !ok {
+		t.Skip("no Chrome/Chromium found; skipping render test")
+	}
+
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch r.URL.Path {
+		case "/page":
+			w.Header().Set("Content-Type", "text/html; charset=utf-8")
+			_, _ = w.Write([]byte(`<!doctype html><html><body><p>a real page</p></body></html>`))
+		case "/file.zip", "/download":
+			// A binary served with no useful extension on the path, the shape that
+			// makes Chrome download to ~/Downloads when navigated to (issue #32).
+			w.Header().Set("Content-Type", "application/zip")
+			_, _ = w.Write([]byte("PK\x03\x04 not really a zip"))
+		case "/data":
+			w.Header().Set("Content-Type", "text/csv")
+			_, _ = w.Write([]byte("a,b\n1,2\n"))
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	defer srv.Close()
+
+	p := New(Options{Headless: true, Workers: 1, Settle: 300 * time.Millisecond, RenderTimeout: 20 * time.Second})
+	defer func() { _ = p.Close() }()
+
+	ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
+	defer cancel()
+
+	// A real HTML page renders as before.
+	if res, err := p.Render(ctx, srv.URL+"/page"); err != nil {
+		t.Errorf("render HTML page: %v", err)
+	} else if !strings.Contains(res.HTML, "a real page") {
+		t.Errorf("HTML page did not render:\n%s", res.HTML)
+	}
+
+	// Non-HTML navigation targets come back as *ErrNotHTML so the caller can route
+	// them to the asset downloader instead of saving a broken page or downloading.
+	for _, tc := range []struct{ path, wantCT string }{
+		{"/download", "application/zip"},
+		{"/data", "text/csv"},
+	} {
+		_, err := p.Render(ctx, srv.URL+tc.path)
+		var notHTML *ErrNotHTML
+		if !errors.As(err, &notHTML) {
+			t.Errorf("Render(%s) error = %v, want *ErrNotHTML", tc.path, err)
+			continue
+		}
+		if !strings.Contains(notHTML.ContentType, tc.wantCT) {
+			t.Errorf("Render(%s) content type = %q, want %q", tc.path, notHTML.ContentType, tc.wantCT)
+		}
+	}
+}

cli/serve.go

@@ -1,11 +1,13 @@
 package cli
 
 import (
+	"context"
 	"fmt"
 	"net"
 	"net/http"
 	"os"
 	"path/filepath"
+	"time"
 
 	"github.com/spf13/cobra"
 )
@@ -24,14 +26,14 @@ func newServeCmd() *cobra.Command {
 			if len(args) == 1 {
 				dir = args[0]
 			}
-			return runServe(dir, addr)
+			return runServe(cmd.Context(), dir, addr)
 		},
 	}
 	cmd.Flags().StringVarP(&addr, "addr", "a", "127.0.0.1:8800", "address to listen on")
 	return cmd
 }
 
-func runServe(dir, addr string) error {
+func runServe(ctx context.Context, dir, addr string) error {
 	info, err := os.Stat(dir)
 	if err != nil {
 		return fmt.Errorf("cannot serve %q: %w", dir, err)
@@ -50,5 +52,24 @@ func runServe(dir, addr string) error {
 	fmt.Fprintln(os.Stderr, styleTitle.Render("kage serve")+" "+styleDim.Render(abs))
 	fmt.Fprintln(os.Stderr, "  open "+styleAccent.Render("http://"+ln.Addr().String()))
 	fmt.Fprintln(os.Stderr, styleDim.Render("  press Ctrl-C to stop"))
-	return srv.Serve(ln)
+
+	srvErr := make(chan error, 1)
+	go func() { srvErr <- srv.Serve(ln) }()
+
+	select {
+	case err := <-srvErr:
+		if err != nil && err != http.ErrServerClosed {
+			return err
+		}
+	case <-ctx.Done():
+		shutdownCtx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
+		defer cancel()
+		if err := srv.Shutdown(shutdownCtx); err != nil {
+			_ = srv.Close()
+		}
+		if err := <-srvErr; err != nil && err != http.ErrServerClosed {
+			return err
+		}
+	}
+	return nil
 }

clone/cloner.go

@@ -255,6 +255,22 @@ func (c *Cloner) processPage(ctx context.Context, j pageItem) {
 
 	res, err := c.pool.Render(ctx, j.u.String())
 	if err != nil {
+		var notHTML *browser.ErrNotHTML
+		if errors.As(err, &notHTML) {
+			// The URL is not a page but a file (a zip, a CSV, a bare image) that
+			// reached the page worker through an extensionless link. Hand it to the
+			// asset downloader, where the size and media policy decides whether to
+			// localise it or leave it remote, rather than saving a broken page or
+			// letting Chrome download it to the user's Downloads folder (issue #32).
+			c.front.markVisited(key)
+			if c.wantAsset(j.u) {
+				c.enqueueAsset(ctx, j.u, "")
+				c.logf("not a page, fetching as asset (%s): %s", notHTML.ContentType, j.u.String())
+			} else {
+				c.logf("not a page, left on the live web (%s): %s", notHTML.ContentType, j.u.String())
+			}
+			return
+		}
 		c.failPage(j.u.String(), fmt.Errorf("render: %w", err))
 		return
 	}

clone/cloner_test.go

@@ -261,6 +261,79 @@ func TestCloneRefreshReRenders(t *testing.T) {
 	}
 }
 
+// TestCloneRoutesNonHTMLToAsset guards issue #32: an extensionless link that
+// turns out to be a file (a zip) is classified as a page up front, but once the
+// page worker sees it is not HTML it must be handed to the asset downloader, not
+// saved as a broken page nor downloaded by Chrome to ~/Downloads.
+func TestCloneRoutesNonHTMLToAsset(t *testing.T) {
+	if testing.Short() {
+		t.Skip("clone test drives Chrome; skipped under -short")
+	}
+	if _, ok := browser.LookChrome(); !ok {
+		t.Skip("no Chrome/Chromium found; skipping clone test")
+	}
+
+	mux := http.NewServeMux()
+	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
+		if r.URL.Path != "/" {
+			http.NotFound(w, r)
+			return
+		}
+		w.Header().Set("Content-Type", "text/html; charset=utf-8")
+		// The link has no extension, so it is queued as a page; the server then
+		// answers it with a zip.
+		_, _ = w.Write([]byte(`<!doctype html><html><body>
+<h1>Home</h1><a href="/download">grab the bundle</a></body></html>`))
+	})
+	mux.HandleFunc("/download", func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "application/zip")
+		_, _ = w.Write([]byte("PK\x03\x04 pretend bundle"))
+	})
+	srv := httptest.NewServer(mux)
+	defer srv.Close()
+
+	seed, err := urlx.ParseSeed(srv.URL)
+	if err != nil {
+		t.Fatalf("parse seed: %v", err)
+	}
+
+	out := t.TempDir()
+	cfg := DefaultConfig()
+	cfg.OutDir = out
+	cfg.Settle = 300 * time.Millisecond
+	cfg.RenderTimeout = 20 * time.Second
+	cfg.Timeout = 10 * time.Second
+
+	ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
+	defer cancel()
+
+	res, err := New(seed, cfg, t.Logf).Run(ctx)
+	if err != nil {
+		t.Fatalf("run: %v", err)
+	}
+
+	root := res.OutDir
+	// The home page is a real page and is written.
+	if !fileExists(filepath.Join(root, "index.html")) {
+		t.Error("home page was not written")
+	}
+	// The zip is NOT saved as a page: no download/index.html exists.
+	if fileExists(filepath.Join(root, "download", "index.html")) {
+		t.Error("non-HTML target was saved as a page")
+	}
+	// The zip is fetched as an asset under the reserved tree instead.
+	if res.Assets < 1 {
+		t.Errorf("expected the zip to be fetched as an asset, assets=%d", res.Assets)
+	}
+	assetDir := filepath.Join(root, cfg.Reserved)
+	if !anyFileUnder(t, assetDir, "download") {
+		t.Error("the zip was not downloaded into the reserved asset tree")
+	}
+	if res.PageErrors != 0 {
+		t.Errorf("a rerouted non-HTML target must not count as a page error, got %d", res.PageErrors)
+	}
+}
+
 func readFile(t *testing.T, path string) string {
 	t.Helper()
 	b, err := os.ReadFile(path)

cmd/kage/main.go

@@ -7,6 +7,7 @@ import (
 	"context"
 	"os"
 	"os/signal"
+	"syscall"
 
 	"github.com/tamnd/kage/cli"
 	"github.com/tamnd/kage/viewer"
@@ -18,7 +19,13 @@ func main() {
 	// thread; in the default build this is a harmless no-op.
 	viewer.LockMainThread()
 
-	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
+	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
 	defer stop()
+
+	go func() {
+		<-ctx.Done()
+		stop()
+	}()
+
 	os.Exit(cli.Execute(ctx))
 }

docs/content/reference/release-notes.md

@@ -6,6 +6,19 @@ weight: 40
 
 The authoritative, commit-level history lives in [`CHANGELOG.md`](https://github.com/tamnd/kage/blob/main/CHANGELOG.md) and on the [releases page](https://github.com/tamnd/kage/releases). This page summarises each version.
 
+## v0.3.4
+
+Two community fixes: a clean stop for `kage serve`, and pages with heavy JavaScript that used to be dropped.
+
+- **`kage serve` stops on Ctrl-C.** The preview server was started with a blocking call that never watched for an interrupt, so stopping it meant killing the process. kage now shuts the server down gracefully on an interrupt or a `SIGTERM`, with a short timeout before forcing the listener closed. Thanks to Xirui Wang ([#35](https://github.com/tamnd/kage/pull/35)) and Kaidi Zhao ([#38](https://github.com/tamnd/kage/pull/38)).
+- **Pages with deeply nested JavaScript still clone.** Chrome's DevTools Protocol returns "Object reference chain is too long" while loading a page whose script builds a deeply nested object graph, but the page's HTML has already loaded and the error is only about Chrome's internal object tracking. kage now recognises that error and finishes rendering instead of dropping the page ([#36](https://github.com/tamnd/kage/issues/36)). Thanks to Gautam Kumar ([#39](https://github.com/tamnd/kage/pull/39)).
+
+## v0.3.3
+
+A fix for Chrome saving a file to your Downloads folder mid-crawl.
+
+- **A crawl never writes to your Downloads folder.** A link with no file extension is queued as a page, so the page worker opened it in Chrome. When such a link served a binary, a zip or a CSV, Chrome saved the file to `~/Downloads`, a surprise side effect of running a clone ([#32](https://github.com/tamnd/kage/issues/32)). kage now denies Chrome-initiated downloads outright, since every asset is fetched through kage's own downloader and the browser never needs to write a file. As a second layer, kage detects a navigation whose response is not HTML and reroutes that URL to the asset downloader, where the size and media policy decides whether to localise it or leave it on the live web, instead of saving a broken page.
+
 ## v0.3.2
 
 A fix for garbled text on pages that did not carry a charset of their own.