Merge pull request #43 from tamnd/feat/mobile-readable

feat: add --mobile flag for readable archives of legacy sites
2026-06-19 15:14:44 +07:00 · 2026-06-19 15:14:19 +07:00 · 2026-06-17 18:23:21 +07:00 · 2026-06-17 18:12:46 +07:00
9 changed files with 253 additions and 16 deletions
@@ -6,6 +6,24 @@ All notable changes to kage are recorded here. The format follows

 ## [Unreleased]

+## [0.3.5] - 2026-06-19
+
+### Changed
+
+- Each saved page is now stored in a packed ZIM under its own `<title>` instead of its URL path, so a ZIM reader's search box suggests pages by their readable title.
+  Typing into Kiwix's search now offers "Five Founders" or "Female Founders" rather than a filename, because the title pointer list a reader's suggestion search walks carries the real page titles.
+  A page with no `<title>` still falls back to its path, and the per-page title also flows into the `title` column of `kage parquet export`.
+
+### Added
+
+- `kage clone --mobile` makes legacy "font-era" sites readable on a phone.
+  Sites from the 1990s and early 2000s — paulgraham.com is a good example — embed typography directly in the HTML with `<font size="2" face="verdana">`, table-based layouts, and no viewport declaration.
+  A mobile browser receiving that markup without a viewport meta shrinks everything to desktop scale, and the `<font size="2">` instruction then makes the already-small text microscopic.
+  Passing `--mobile` injects two things into every saved page before it is written: a `<meta name="viewport" content="width=device-width, initial-scale=1">` tag so the browser stops shrinking, and a small `<style>` block that lifts the base font size to 18 px, inherits that size through `<font>` elements, caps the content width at 720 px, loosens line height to 1.7, and hides image-map navigation elements (usually a GIF served from an external CDN that 404s offline anyway).
+  The override is deliberately last in `<head>` so it wins specificity ties, and it does not touch pages that already carry a viewport and readable type sizes.
+- The packing guide now documents how search works on a kage archive: title suggestions in any ZIM reader, and full-text search of page bodies through `kage parquet export` and DuckDB.
+  A Xapian full-text index is deliberately not written, since Xapian is GPL and kage is MIT.
+
 ## [0.3.4] - 2026-06-17

 ### Fixed
@@ -39,8 +39,9 @@ type cloneFlags struct {
 	noRobots      bool
 	noSitemap     bool
 	headful       bool
-	keepNoscript  bool
-	chromeBin     string
+	keepNoscript    bool
+	mobileReadable  bool
+	chromeBin       string
 	controlURL    string
 	noResume      bool
 	refresh       bool
@@ -86,6 +87,7 @@ func newCloneCmd() *cobra.Command {
 	fs.BoolVar(&f.noSitemap, "no-sitemap", false, "do not seed URLs from sitemap.xml")
 	fs.BoolVar(&f.headful, "headful", false, "run Chrome with a visible window (debugging)")
 	fs.BoolVar(&f.keepNoscript, "keep-noscript", false, "unwrap <noscript> content instead of dropping it")
+	fs.BoolVar(&f.mobileReadable, "mobile", false, "inject viewport and CSS overrides so legacy sites read comfortably on a phone")
 	fs.StringVar(&f.chromeBin, "chrome", "", "path to the Chrome/Chromium binary")
 	fs.StringVar(&f.controlURL, "control-url", "", "attach to an existing Chrome DevTools endpoint")
 	fs.BoolVar(&f.noResume, "no-resume", false, "do not reuse or write resume state")
@@ -140,6 +142,7 @@ func runClone(ctx context.Context, arg string, f *cloneFlags) error {
 	cfg.FollowSitemap = !f.noSitemap
 	cfg.Headless = !f.headful
 	cfg.KeepNoscript = f.keepNoscript
+	cfg.MobileReadable = f.mobileReadable
 	cfg.ChromeBin = f.chromeBin
 	cfg.ControlURL = f.controlURL
 	cfg.Resume = !f.noResume
@@ -305,8 +305,9 @@ func (c *Cloner) processPage(ctx context.Context, j pageItem) {

 	asset.RewriteHTML(root, j.u, sink)
 	sanitize.CleanTree(root, sanitize.Options{
-		KeepNoscript: c.cfg.KeepNoscript,
-		Banner:       "cloned by kage from " + j.u.String(),
+		KeepNoscript:   c.cfg.KeepNoscript,
+		MobileReadable: c.cfg.MobileReadable,
+		Banner:         "cloned by kage from " + j.u.String(),
 	})

 	var buf strings.Builder
@@ -56,12 +56,13 @@ type Config struct {
 	ScopePrefix       string
 	ExcludePaths      []string

-	RespectRobots bool
-	FollowSitemap bool
-	Headless      bool
-	KeepNoscript  bool
-	ChromeBin     string
-	ControlURL    string
+	RespectRobots  bool
+	FollowSitemap  bool
+	Headless       bool
+	KeepNoscript   bool
+	MobileReadable bool
+	ChromeBin      string
+	ControlURL     string

 	// Resume loads the prior run's visited set and skips pages already written,
 	// so an interrupted or repeated clone picks up where it left off instead of
@@ -41,7 +41,18 @@ kage open paulgraham.com.zim            # read it back with kage
 kiwix-serve paulgraham.com.zim          # or serve it with Kiwix at http://localhost
 ```

-You can also double-click the file in the [Kiwix desktop app](https://kiwix.org/en/applications/), or load it on Kiwix for Android or iOS to read your mirror on your phone. kage writes the metadata the format and `zimcheck` treat as mandatory, including the title, description, and the favicon Kiwix shows as the book icon in its library, so the archive shows up properly rather than as an untitled, iconless entry. One caveat: kage does not write the full-text search index that Kiwix's own packs ship with, so browsing works everywhere while in-reader search is limited.
+You can also double-click the file in the [Kiwix desktop app](https://kiwix.org/en/applications/), or load it on Kiwix for Android or iOS to read your mirror on your phone. kage writes the metadata the format and `zimcheck` treat as mandatory, including the title, description, and the favicon Kiwix shows as the book icon in its library, so the archive shows up properly rather than as an untitled, iconless entry.
+
+### Searching a packed mirror
+
+Each page is stored under its own real `<title>`, so the search box in Kiwix (and any other ZIM reader) suggests pages by their readable title: type `five` into a paulgraham.com archive and it offers "Five Founders" and "Female Founders", not a filename. This title search works in every reader with no extra index.
+
+What kage does not write is a Xapian full-text index, the separate search database that lets Kiwix match words inside a page's body. Xapian is GPL and kage is MIT, so linking it in would change kage's license; rather than do that, kage leaves full-text search to its own columnar export. `kage parquet export mirror.zim` writes one row per page with a `text` column, which [DuckDB](https://duckdb.org) searches with its `fts` extension or a plain `ILIKE`, fully offline and ranked:
+
+```bash
+kage parquet export paulgraham.com.zim -o paulgraham.parquet
+duckdb -c "SELECT url FROM 'paulgraham.parquet' WHERE text ILIKE '%schlep blindness%'"
+```

 ## A self-contained binary

@@ -354,3 +354,80 @@ func TestHandler(t *testing.T) {
 		t.Errorf("GET missing = %d, want 404", code)
 	}
 }
+
+// titleOf returns the stored entry title for a content url, scanning entries in
+// url order since the reader exposes titles only through EntryAt.
+func titleOf(t *testing.T, r *zim.Reader, url string) string {
+	t.Helper()
+	for i := uint32(0); i < r.Count(); i++ {
+		e, err := r.EntryAt(i)
+		if err != nil {
+			continue
+		}
+		if e.Namespace == zim.NamespaceContent && e.URL == url {
+			return e.Title
+		}
+	}
+	t.Fatalf("no content entry for %q", url)
+	return ""
+}
+
+// TestBuildZIMPageTitles checks that each HTML page entry carries its own
+// <title> (collapsed to one line), that a page with no title falls back to its
+// url, and that a non-HTML asset keeps the url as its title.
+func TestBuildZIMPageTitles(t *testing.T) {
+	root := t.TempDir()
+	host := filepath.Join(root, "example.com")
+	files := map[string]string{
+		"index.html":       "<!doctype html><title>Home</title><h1>Hi</h1>",
+		"essay/index.html": "<!doctype html><title>\n  A Long\n  Title  </title><p>x</p>",
+		"bare/index.html":  "<!doctype html><h1>No title here</h1>",
+		"logo.png":         "\x89PNGfake",
+	}
+	for rel, body := range files {
+		p := filepath.Join(host, filepath.FromSlash(rel))
+		if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
+			t.Fatal(err)
+		}
+		if err := os.WriteFile(p, []byte(body), 0o644); err != nil {
+			t.Fatal(err)
+		}
+	}
+
+	out := filepath.Join(t.TempDir(), "titles.zim")
+	if _, _, err := BuildZIM(host, ZIMOptions{Out: out, Date: "2026-06-14"}); err != nil {
+		t.Fatalf("BuildZIM: %v", err)
+	}
+	r, err := zim.Open(out)
+	if err != nil {
+		t.Fatalf("Open: %v", err)
+	}
+	defer func() { _ = r.Close() }()
+
+	cases := map[string]string{
+		"index.html":       "Home",
+		"essay/index.html": "A Long Title", // newlines and runs collapsed to single spaces
+		"bare/index.html":  "bare/index.html",
+		"logo.png":         "logo.png",
+	}
+	for url, want := range cases {
+		if got := titleOf(t, r, url); got != want {
+			t.Errorf("title of %q = %q, want %q", url, got, want)
+		}
+	}
+}
+
+func TestCollapseSpaces(t *testing.T) {
+	cases := map[string]string{
+		"  hello  world ":      "hello world",
+		"line\n\tone\r\n  two": "line one two",
+		"":                     "",
+		"   ":                  "",
+		"single":               "single",
+	}
+	for in, want := range cases {
+		if got := collapseSpaces(in); got != want {
+			t.Errorf("collapseSpaces(%q) = %q, want %q", in, got, want)
+		}
+	}
+}
@@ -170,11 +170,16 @@ func buildWriter(mirrorDir string, opts ZIMOptions) (*zim.Writer, *clusterCache,
 			return err
 		}
 		mime := MimeForExt(rel)
+		title := ""
 		if mime == "text/html" {
 			htmlPages = append(htmlPages, rel)
+			// Store the page's real <title> on its entry, not the URL path, so a
+			// ZIM reader's search box and suggestions match the readable title.
+			// An empty result leaves the writer to fall back to the url.
+			title = htmlTitleOfBytes(data)
 		}
 		counts[mime]++
-		w.AddContent(zim.NamespaceContent, rel, "", mime, data)
+		w.AddContent(zim.NamespaceContent, rel, title, mime, data)
 		return nil
 	})
 	if walkErr != nil {
@@ -240,16 +245,30 @@ func htmlTitleOf(mirrorDir, mainURL string) string {
 	if mainURL == "" {
 		return ""
 	}
-	f, err := os.Open(filepath.Join(mirrorDir, filepath.FromSlash(mainURL)))
+	data, err := os.ReadFile(filepath.Join(mirrorDir, filepath.FromSlash(mainURL)))
 	if err != nil {
 		return ""
 	}
-	defer func() { _ = f.Close() }()
-	doc, err := html.Parse(f)
+	return htmlTitleOfBytes(data)
+}
+
+// htmlTitleOfBytes returns the first <title> in the given HTML with surrounding
+// and internal whitespace collapsed, or "" if the bytes do not parse or carry
+// no title. Page entries use it so a ZIM reader shows the page's real title
+// instead of its URL path.
+func htmlTitleOfBytes(data []byte) string {
+	doc, err := html.Parse(bytes.NewReader(data))
 	if err != nil {
 		return ""
 	}
-	return strings.TrimSpace(findTitle(doc))
+	return collapseSpaces(findTitle(doc))
+}
+
+// collapseSpaces trims s and replaces every run of whitespace (including the
+// newlines a wrapped <title> can carry) with a single space, so a title reads
+// as one clean line in a reader's library and search results.
+func collapseSpaces(s string) string {
+	return strings.Join(strings.Fields(s), " ")
 }

 // findTitle returns the text of the first <title> element in depth-first order.
@@ -29,6 +29,14 @@ type Options struct {
 	// Banner, when non-empty, is inserted as an HTML comment at the top of the
 	// document.
 	Banner string
+	// MobileReadable injects a viewport meta tag and a small CSS block that
+	// makes legacy, font-era sites readable on mobile. It is intended for
+	// archives of 1990s/2000s sites that use <font size="2">, table layouts,
+	// and no viewport declaration — all of which render as microscopic text on
+	// a phone. The injected CSS overrides font sizes, loosens line height, caps
+	// the content width, and hides image-map navigation elements that are
+	// useless offline.
+	MobileReadable bool
 }

 // Report counts what was removed, for the run summary and for tests.
@@ -72,6 +80,10 @@ func CleanTree(root *html.Node, opts Options) Report {
 	var rep Report
 	clean(root, opts, &rep)
 	rep.CharsetAdded = ensureCharset(root)
+	if opts.MobileReadable {
+		ensureViewport(root)
+		injectMobileCSS(root)
+	}
 	if opts.Banner != "" {
 		insertBanner(root, opts.Banner)
 	}
@@ -291,6 +303,67 @@ func findElement(n *html.Node, a atom.Atom) *html.Node {
 	return nil
 }

+// mobileCSS is injected when MobileReadable is set. It targets legacy "font
+// era" HTML that renders as microscopic text on mobile:
+//   - :root font-size 18 px — baseline all em/rem sizes upward
+//   - body — centre, cap width, add padding, loosen line height
+//   - font element — override the in-HTML size/face attributes that sites like
+//     paulgraham.com embed directly in the markup (e.g. <font size="2">)
+//   - table/td — prevent overflow; add minimal cell breathing room
+//   - img[usemap], map — image-map navigation is useless offline (the image
+//     itself usually 404s from an external CDN); hide both the image and the map
+const mobileCSS = `body{max-width:720px;margin:0 auto;padding:.75em 1em;line-height:1.7;font-family:Georgia,"Times New Roman",serif}` +
+	`:root{font-size:18px}` +
+	`font{font-size:1rem!important;font-family:inherit!important;color:inherit!important}` +
+	`table{max-width:100%!important;word-break:break-word}` +
+	`td,th{padding:.25em!important}` +
+	`img[usemap],map{display:none!important}`
+
+// ensureViewport inserts <meta name="viewport" content="width=device-width,
+// initial-scale=1"> at the top of <head> when the document does not already
+// carry one. Without it a mobile browser shrinks the page to fit the screen
+// at desktop scale, making text unreadably small regardless of CSS font sizes.
+func ensureViewport(root *html.Node) {
+	head := findElement(root, atom.Head)
+	if head == nil {
+		return
+	}
+	// Check whether a viewport meta already exists.
+	for c := head.FirstChild; c != nil; c = c.NextSibling {
+		if c.Type == html.ElementNode && c.DataAtom == atom.Meta &&
+			strings.EqualFold(attr(c, "name"), "viewport") {
+			return
+		}
+	}
+	meta := &html.Node{
+		Type:     html.ElementNode,
+		Data:     "meta",
+		DataAtom: atom.Meta,
+		Attr: []html.Attribute{
+			{Key: "name", Val: "viewport"},
+			{Key: "content", Val: "width=device-width, initial-scale=1"},
+		},
+	}
+	head.InsertBefore(meta, head.FirstChild)
+}
+
+// injectMobileCSS appends a <style> block containing mobileCSS to <head>.
+// It goes at the end of <head> so it wins specificity ties over any existing
+// inline styles the page already carries.
+func injectMobileCSS(root *html.Node) {
+	head := findElement(root, atom.Head)
+	if head == nil {
+		return
+	}
+	style := &html.Node{
+		Type:     html.ElementNode,
+		Data:     "style",
+		DataAtom: atom.Style,
+	}
+	style.AppendChild(&html.Node{Type: html.TextNode, Data: mobileCSS})
+	head.AppendChild(style)
+}
+
 // insertBanner prepends an HTML comment to the document.
 func insertBanner(root *html.Node, text string) {
 	c := &html.Node{Type: html.CommentNode, Data: " " + text + " "}
@@ -197,6 +197,40 @@ func TestCharsetAddedWhenMissing(t *testing.T) {
 	}
 }

+func TestMobileReadableInjectsViewportAndCSS(t *testing.T) {
+	// A paulgraham.com-style page: no viewport, tiny <font size="2"> markup.
+	in := `<html><head><title>Essay</title></head>` +
+		`<body><font size="2" face="verdana"><p>Hello world.</p></font></body></html>`
+	out, _, err := Strip([]byte(in), Options{MobileReadable: true})
+	if err != nil {
+		t.Fatal(err)
+	}
+	s := string(out)
+	if !strings.Contains(s, `name="viewport"`) {
+		t.Error("viewport meta not injected")
+	}
+	if !strings.Contains(s, "width=device-width") {
+		t.Error("viewport content wrong")
+	}
+	if !strings.Contains(s, "font-size:18px") {
+		t.Error("mobile CSS not injected")
+	}
+	if !strings.Contains(s, "font{font-size:1rem") {
+		t.Error("font override not in mobile CSS")
+	}
+}
+
+func TestMobileReadableSkipsExistingViewport(t *testing.T) {
+	in := `<html><head><meta name="viewport" content="width=device-width"><title>x</title></head><body></body></html>`
+	out, _, err := Strip([]byte(in), Options{MobileReadable: true})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if n := strings.Count(string(out), `name="viewport"`); n != 1 {
+		t.Errorf("viewport injected when one already existed (count %d)", n)
+	}
+}
+
 func TestCharsetNotDuplicated(t *testing.T) {
 	// A page that already declares a charset, in either form, is left alone.
 	cases := []string{
Author	SHA1	Message	Date
Tam Nguyen Duc	2c15f79ff5	Merge pull request #43 from tamnd/feat/mobile-readable feat: add --mobile flag for readable archives of legacy sites	2026-06-19 15:14:44 +07:00
Duc-Tam Nguyen	060b4b6449	feat: add --mobile flag for readable archives of legacy sites Cloning a 1990s/2000s site like paulgraham.com and opening it in Kiwix on a phone produces microscopic text: the pages use <font size="2">, table layouts, and no viewport declaration, so the mobile browser shrinks everything to desktop scale and then the font-size attribute makes it smaller still. kage clone --mobile injects two things into every saved page: - <meta name="viewport" content="width=device-width, initial-scale=1"> so the browser stops shrinking the page - a <style> block that lifts the base font to 18px, inherits it through <font> elements (overriding the in-HTML size/face attributes), caps the content width at 720px, loosens line height to 1.7, and hides image-map nav elements whose source GIFs 404 offline The style block goes last in <head> to win specificity ties, and ensureViewport skips pages that already carry a viewport meta. Two tests cover the happy path and the no-duplicate case.	2026-06-19 15:14:19 +07:00
Tam Nguyen Duc	3117c4c55c	Merge pull request #41 from tamnd/feat/zim-search-index Store each page under its real title in a packed ZIM	2026-06-17 18:23:21 +07:00
Duc-Tam Nguyen	6655a434b9	Store each page under its real title in a packed ZIM A packed page entry took its URL path as its title, so a ZIM reader's search box suggested filenames like 5founders.html instead of readable titles. The page's <title> was only read for the archive-level M/Title. Read each HTML page's <title> at pack time and store it on the entry, collapsing wrapped whitespace to one line and falling back to the path when a page has none. A reader's suggestion search walks the title pointer list, which is sorted by title, so typing now offers entries like Five Founders and Female Founders. The per-page title also lands in the title column of a parquet export. Document the search story in the packing guide: title suggestions in any reader, and full-text search of page bodies through parquet and DuckDB. A Xapian full-text index stays out on purpose, since Xapian is GPL and kage is MIT.	2026-06-17 18:12:46 +07:00