Add a meta charset to saved pages so text does not mojibake

A page that declared its charset only in the HTTP Content-Type header, with no meta charset in the markup, lost that signal once kage saved it as a file. A reader serving the bytes without a charset then fell back to its locale encoding and garbled every curly quote, dash, and nbsp. kage writes UTF-8, so it now inserts a meta charset utf-8 at the top of head when the page does not already declare one. Verified on the blog post from the report: the saved HTML now carries the meta and renders the quotes correctly.
Merge pull request #28 from tamnd/release-0.3.1
2026-06-16 00:15:27 +07:00 · 2026-06-15 23:14:41 +07:00 · 2026-06-15 23:12:08 +07:00 · 2026-06-15 22:49:08 +07:00 · 2026-06-15 22:46:42 +07:00
6 changed files with 176 additions and 4 deletions
@@ -6,6 +6,29 @@ All notable changes to kage are recorded here. The format follows

 ## [Unreleased]

+### Fixed
+
+- Saved pages now declare their character encoding, so text no longer mojibakes in a reader.
+  kage writes every page as UTF-8, but a source that set its charset only in the HTTP `Content-Type` header, with no `<meta charset>` in the markup, lost that signal once the page became a standalone file.
+  A reader serving the bytes without a charset then fell back to its locale encoding and turned every curly quote, dash, and non-breaking space into mojibake (reported in #16 and #29).
+  kage now inserts a `<meta charset="utf-8">` at the top of `<head>` when the page does not already declare one, so the page is self-describing in any reader.
+
+## [0.3.1] - 2026-06-15
+
+### Fixed
+
+- A served mirror whose entry point is a nested page no longer loses its CSS and
+  images when opened at the root. kage saves each page's asset links as
+  mirror-relative paths (`../_kage/...`) computed for that page's own location,
+  but the viewer answered `/` with the main page's bytes in place, so the browser
+  resolved those relative URLs against `/` and missed every one. A
+  `developer.apple.com/documentation` mirror, whose main page is
+  `developer.apple.com/documentation/index.html`, landed at `/` completely
+  unstyled. kage now redirects `/` to the main page's canonical content path, the
+  way the archive's `W/mainPage` redirect already does, so the browser resolves
+  the page's relative assets correctly. Kiwix was unaffected because it follows
+  that redirect itself.
+
 ## [0.3.0] - 2026-06-15

 ### Added
@@ -180,7 +203,8 @@ can browse offline, with every script stripped out.
  a multi-arch container image on GHCR (Chromium bundled), checksums, SBOMs, and
  a cosign signature, all cut from one version tag by GoReleaser.

-[Unreleased]: https://github.com/tamnd/kage/compare/v0.3.0...HEAD
+[Unreleased]: https://github.com/tamnd/kage/compare/v0.3.1...HEAD
+[0.3.1]: https://github.com/tamnd/kage/compare/v0.3.0...v0.3.1
 [0.3.0]: https://github.com/tamnd/kage/compare/v0.2.1...v0.3.0
 [0.2.1]: https://github.com/tamnd/kage/compare/v0.2.0...v0.2.1
 [0.2.0]: https://github.com/tamnd/kage/compare/v0.1.2...v0.2.0
@@ -6,6 +6,12 @@ weight: 40

 The authoritative, commit-level history lives in [`CHANGELOG.md`](https://github.com/tamnd/kage/blob/main/CHANGELOG.md) and on the [releases page](https://github.com/tamnd/kage/releases). This page summarises each version.

+## v0.3.1
+
+A fix for broken styling when a packed mirror's home page is a nested page.
+
+- **`/` redirects to the main page instead of serving it in place.** A page's saved asset links are mirror-relative (`../_kage/...`), computed for that page's own location. The viewer was answering `/` with the main page's bytes directly, so the browser resolved those links against `/` and 404ed the page's CSS and images. A `developer.apple.com/documentation` mirror opened at `/` came up completely unstyled. kage now redirects `/` to the main page's canonical path, the way the archive's `W/mainPage` redirect does, so relative assets resolve correctly. Kiwix already followed that redirect, so it was never affected.
+
 ## v0.3.0

 Leaner mirrors, and a way to publish one as a dataset. A clone now keeps the assets that make a site readable offline and leaves the bulk downloads on the live web, and a packed archive converts to a columnar table that drops straight into dataset tooling.
@@ -326,6 +326,24 @@ func TestHandler(t *testing.T) {
 	if code, body := get("/"); code != 200 || !bytes.Contains([]byte(body), []byte("Example Home")) {
 		t.Errorf("GET / = %d %.30q", code, body)
 	}
+
+	// "/" must redirect to the main page's canonical content path, not serve its
+	// bytes in place, so the page's mirror-relative asset URLs resolve against the
+	// right base instead of 404ing.
+	noFollow := &http.Client{CheckRedirect: func(*http.Request, []*http.Request) error {
+		return http.ErrUseLastResponse
+	}}
+	resp, err := noFollow.Get(srv.URL + "/")
+	if err != nil {
+		t.Fatal(err)
+	}
+	_ = resp.Body.Close()
+	if resp.StatusCode != http.StatusFound {
+		t.Errorf("GET / status = %d, want 302", resp.StatusCode)
+	}
+	if loc := resp.Header.Get("Location"); loc != "/index.html" {
+		t.Errorf("GET / Location = %q, want %q", loc, "/index.html")
+	}
 	if code, _ := get("/about/index.html"); code != 200 {
 		t.Errorf("GET /about/index.html = %d", code)
 	}
@@ -8,14 +8,24 @@ import (
 	"github.com/tamnd/kage/zim"
 )

-// Handler serves a ZIM archive over HTTP. "/" maps to the archive's main page;
-// "/a/b.png" maps to the C/a/b.png content entry. Because the saved HTML's links
-// are mirror-relative paths, and those are exactly the C urls, a click in a
+// Handler serves a ZIM archive over HTTP. "/" redirects to the archive's main
+// page; "/a/b.png" maps to the C/a/b.png content entry. Because the saved HTML's
+// links are mirror-relative paths, and those are exactly the C urls, a click in a
 // served page hits the right entry with no rewriting. A miss is a plain 404.
 func Handler(r *zim.Reader) http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
 		p := strings.TrimPrefix(req.URL.Path, "/")
 		if p == "" {
+			// The main page's saved HTML carries mirror-relative asset URLs
+			// (../_kage/...) computed for its own nested location, so serving its
+			// bytes at "/" would resolve them against the wrong base and 404 the
+			// page's CSS and images. Redirect to the page's canonical content path
+			// instead, the way the archive's W/mainPage redirect does, so the
+			// browser resolves those relative URLs correctly.
+			if ns, url, ok := r.MainPageRef(); ok && ns == zim.NamespaceContent {
+				http.Redirect(w, req, "/"+url, http.StatusFound)
+				return
+			}
 			blob, err := r.MainPage()
 			if err != nil {
 				http.NotFound(w, req)
@@ -41,6 +41,7 @@ type Report struct {
 	MetaRefreshRemoved  int
 	DeadLinksRemoved    int
 	CondCommentsRemoved int
+	CharsetAdded        bool
 }

 // jsURLAttrs are attributes whose value may be a javascript: URL.
@@ -70,6 +71,7 @@ func Strip(doc []byte, opts Options) ([]byte, Report, error) {
 func CleanTree(root *html.Node, opts Options) Report {
 	var rep Report
 	clean(root, opts, &rep)
+	rep.CharsetAdded = ensureCharset(root)
 	if opts.Banner != "" {
 		insertBanner(root, opts.Banner)
 	}
@@ -227,6 +229,68 @@ func unwrapNoscript(parent, ns *html.Node) {
 	parent.RemoveChild(ns)
 }

+// ensureCharset guarantees the document declares UTF-8, inserting a
+// <meta charset="utf-8"> at the top of <head> when none is present, and reports
+// whether it added one. kage renders every saved page as UTF-8, but a source
+// that set its charset only in the HTTP Content-Type header, with no <meta>
+// charset in the markup, loses that signal once the page is a standalone file.
+// A reader then serving the bytes without a charset falls back to its locale
+// encoding and mojibakes every multibyte character (curly quotes, dashes, a
+// non-breaking space). Declaring the charset in the markup makes the page
+// self-describing in any reader, kage's own viewer and Kiwix alike.
+func ensureCharset(root *html.Node) bool {
+	head := findElement(root, atom.Head)
+	if head == nil {
+		return false
+	}
+	if hasCharsetMeta(head) {
+		return false
+	}
+	meta := &html.Node{
+		Type:     html.ElementNode,
+		Data:     "meta",
+		DataAtom: atom.Meta,
+		Attr:     []html.Attribute{{Key: "charset", Val: "utf-8"}},
+	}
+	// The declaration must precede any content for a reader to honour it, so it
+	// goes first in <head>.
+	head.InsertBefore(meta, head.FirstChild)
+	return true
+}
+
+// hasCharsetMeta reports whether head already declares a character encoding,
+// either as <meta charset="..."> or the older <meta http-equiv="Content-Type"
+// content="...; charset=...">.
+func hasCharsetMeta(head *html.Node) bool {
+	for c := head.FirstChild; c != nil; c = c.NextSibling {
+		if c.Type != html.ElementNode || c.DataAtom != atom.Meta {
+			continue
+		}
+		if attr(c, "charset") != "" {
+			return true
+		}
+		if strings.EqualFold(attr(c, "http-equiv"), "content-type") &&
+			strings.Contains(strings.ToLower(attr(c, "content")), "charset=") {
+			return true
+		}
+	}
+	return false
+}
+
+// findElement returns the first element node of the given atom in document
+// order, or nil if none exists.
+func findElement(n *html.Node, a atom.Atom) *html.Node {
+	if n.Type == html.ElementNode && n.DataAtom == a {
+		return n
+	}
+	for c := n.FirstChild; c != nil; c = c.NextSibling {
+		if found := findElement(c, a); found != nil {
+			return found
+		}
+	}
+	return nil
+}
+
 // insertBanner prepends an HTML comment to the document.
 func insertBanner(root *html.Node, text string) {
 	c := &html.Node{Type: html.CommentNode, Data: " " + text + " "}
@@ -166,3 +166,53 @@ func TestKeepMetaRefreshPlain(t *testing.T) {
 		t.Error("JS-target meta refresh must be removed regardless")
 	}
 }
+
+func TestCharsetAddedWhenMissing(t *testing.T) {
+	// A page whose source declared its charset only in the HTTP header has no
+	// <meta charset>. The saved file must gain one so a reader does not fall back
+	// to its locale encoding and mojibake the UTF-8 text.
+	in := `<html><head><title>Quotes</title></head><body><p>` +
+		"“curly” — café</p></body></html>"
+	out, rep, err := Strip([]byte(in), Options{})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !rep.CharsetAdded {
+		t.Error("CharsetAdded = false, want true")
+	}
+	s := string(out)
+	if !strings.Contains(strings.ToLower(s), `<meta charset="utf-8"/>`) {
+		t.Errorf("expected an injected meta charset:\n%s", s)
+	}
+	// It must sit at the very start of <head>, before any content.
+	headIdx := strings.Index(s, "<head>")
+	metaIdx := strings.Index(strings.ToLower(s), "<meta charset")
+	titleIdx := strings.Index(s, "<title>")
+	if headIdx >= metaIdx || metaIdx >= titleIdx {
+		t.Errorf("meta charset must come first in head (head=%d meta=%d title=%d)", headIdx, metaIdx, titleIdx)
+	}
+	// The original bytes are preserved as UTF-8.
+	if !strings.Contains(s, "café") {
+		t.Error("UTF-8 content should be preserved")
+	}
+}
+
+func TestCharsetNotDuplicated(t *testing.T) {
+	// A page that already declares a charset, in either form, is left alone.
+	cases := []string{
+		`<html><head><meta charset="utf-8"><title>x</title></head><body></body></html>`,
+		`<html><head><meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"><title>x</title></head><body></body></html>`,
+	}
+	for _, in := range cases {
+		out, rep, err := Strip([]byte(in), Options{})
+		if err != nil {
+			t.Fatal(err)
+		}
+		if rep.CharsetAdded {
+			t.Errorf("CharsetAdded = true for a page that already declares one:\n%s", in)
+		}
+		if n := strings.Count(strings.ToLower(string(out)), "charset"); n != 1 {
+			t.Errorf("charset count = %d, want 1:\n%s", n, out)
+		}
+	}
+}
Author	SHA1	Message	Date
Duc-Tam Nguyen	16c9fcbcfe	Add a meta charset to saved pages so text does not mojibake A page that declared its charset only in the HTTP Content-Type header, with no meta charset in the markup, lost that signal once kage saved it as a file. A reader serving the bytes without a charset then fell back to its locale encoding and garbled every curly quote, dash, and nbsp. kage writes UTF-8, so it now inserts a meta charset utf-8 at the top of head when the page does not already declare one. Verified on the blog post from the report: the saved HTML now carries the meta and renders the quotes correctly.	2026-06-16 00:15:27 +07:00
Tam Nguyen Duc	2d8a64ca6f	Merge pull request #28 from tamnd/release-0.3.1 Cut the v0.3.1 release notes	2026-06-15 23:14:41 +07:00
Duc-Tam Nguyen	a96233cf73	Cut the v0.3.1 release notes Move the Unreleased fix into a dated 0.3.1 section, update the compare links, and add a v0.3.1 summary to the docs release-notes page. This patch redirects / to the main page so a mirror whose entry point is a nested page keeps its CSS and images when served.	2026-06-15 23:12:08 +07:00
Tam Nguyen Duc	2064ef918a	Merge pull request #27 from tamnd/fix-mainpage-redirect Redirect / to the main page instead of serving it in place	2026-06-15 22:49:08 +07:00
Duc-Tam Nguyen	9eabd93098	Redirect / to the main page instead of serving it in place The served main page broke its own CSS and images when the entry point was a nested page. kage saves each page's asset links as mirror-relative paths (../_kage/...) computed for that page's own location, but the handler served the main page's bytes directly at /, so the browser resolved those relative URLs against / and 404ed every one of them. A developer.apple.com/documentation mirror landed at / with no styles. Redirect / to the main page's canonical content path, the way the archive's W/mainPage redirect already does, so the browser navigates to the page's real URL and resolves its relative assets correctly. Kiwix was unaffected because it follows that redirect itself.	2026-06-15 22:46:42 +07:00