Compare commits

...

8 Commits

Author SHA1 Message Date
Tam Nguyen Duc c8fc6faaa3 Merge pull request #45 from tamnd/release-0.3.6
Cut the v0.3.6 release notes
2026-06-19 16:14:53 +07:00
Duc-Tam Nguyen c5b39d6816 Cut the v0.3.6 release notes 2026-06-19 16:14:39 +07:00
Tam Nguyen Duc 057937938d Merge pull request #44 from tamnd/fix/mobile-css-layout
fix: correct mobile layout for paulgraham.com-style table pages
2026-06-19 15:55:51 +07:00
Duc-Tam Nguyen 4e920c57ec fix: correct mobile layout for paulgraham.com-style table pages
Three issues surfaced when viewing the ZIM in Kiwix iOS:

1. Blank column at top: hiding img[usemap] left its containing <td> as an
   empty box. Now td:has(>img[usemap]) hides the entire nav column.

2. Content clipped on the right: the inner content table had width="435"
   as an HTML attribute. Added [width]{width:auto!important} to cancel all
   fixed HTML attribute widths on any element (tables, tds, imgs).

3. Spacer column: the 26px <td> holding a 1x1 transparent GIF kept its
   allocated space. td:has(>img[src*="trans_1x1"]:only-child) hides it.

Also: overflow-x:hidden on body, box-sizing:border-box globally, and
img{max-width:100%;height:auto} so any inline images stay within column.
2026-06-19 15:55:28 +07:00
Tam Nguyen Duc 2c15f79ff5 Merge pull request #43 from tamnd/feat/mobile-readable
feat: add --mobile flag for readable archives of legacy sites
2026-06-19 15:14:44 +07:00
Duc-Tam Nguyen 060b4b6449 feat: add --mobile flag for readable archives of legacy sites
Cloning a 1990s/2000s site like paulgraham.com and opening it in Kiwix
on a phone produces microscopic text: the pages use <font size="2">,
table layouts, and no viewport declaration, so the mobile browser shrinks
everything to desktop scale and then the font-size attribute makes it
smaller still.

kage clone --mobile injects two things into every saved page:

- <meta name="viewport" content="width=device-width, initial-scale=1">
  so the browser stops shrinking the page
- a <style> block that lifts the base font to 18px, inherits it through
  <font> elements (overriding the in-HTML size/face attributes), caps the
  content width at 720px, loosens line height to 1.7, and hides
  image-map nav elements whose source GIFs 404 offline

The style block goes last in <head> to win specificity ties, and
ensureViewport skips pages that already carry a viewport meta.

Two tests cover the happy path and the no-duplicate case.
2026-06-19 15:14:19 +07:00
Tam Nguyen Duc 3117c4c55c Merge pull request #41 from tamnd/feat/zim-search-index
Store each page under its real title in a packed ZIM
2026-06-17 18:23:21 +07:00
Duc-Tam Nguyen 6655a434b9 Store each page under its real title in a packed ZIM
A packed page entry took its URL path as its title, so a ZIM reader's
search box suggested filenames like 5founders.html instead of readable
titles. The page's <title> was only read for the archive-level M/Title.

Read each HTML page's <title> at pack time and store it on the entry,
collapsing wrapped whitespace to one line and falling back to the path
when a page has none. A reader's suggestion search walks the title
pointer list, which is sorted by title, so typing now offers entries
like Five Founders and Female Founders. The per-page title also lands in
the title column of a parquet export.

Document the search story in the packing guide: title suggestions in any
reader, and full-text search of page bodies through parquet and DuckDB.
A Xapian full-text index stays out on purpose, since Xapian is GPL and
kage is MIT.
2026-06-17 18:12:46 +07:00
9 changed files with 316 additions and 16 deletions
+31
View File
@@ -6,6 +6,37 @@ All notable changes to kage are recorded here. The format follows
## [Unreleased]
## [0.3.6] - 2026-06-19
### Fixed
- `kage clone --mobile` now produces a correct layout when the source page uses table-based navigation with image maps, as paulgraham.com does.
Three rendering problems appeared in Kiwix iOS after the 0.3.5 release: a tall blank box at the top of every page, content clipped on the right edge, and a narrow spacer column occupying screen space.
The blank box came from hiding only the `<img usemap>` element while leaving its `<td>` container in place; the cell collapsed to empty but kept its full height.
`td:has(>img[usemap])` now hides the entire nav column rather than just the image inside it.
The right-side clip came from `width="435"` set directly as an HTML attribute on the inner content table; the earlier `max-width` CSS rule does not override HTML attributes.
A new `[width]{width:auto!important;max-width:100%!important}` rule cancels every fixed HTML width attribute on any element — tables, cells, and images alike — so the content column stretches to fill the phone screen.
The spacer column (a 26 px `<td>` holding a 1×1 transparent GIF) kept its allocated width for the same reason; `td:has(>img[src*="trans_1x1"]:only-child)` hides it alongside the nav column.
Two additional rules round out the fix: `*{box-sizing:border-box}` prevents padding from pushing content past the viewport edge, and `img{max-width:100%;height:auto}` keeps any inline photos within their column.
## [0.3.5] - 2026-06-19
### Changed
- Each saved page is now stored in a packed ZIM under its own `<title>` instead of its URL path, so a ZIM reader's search box suggests pages by their readable title.
Typing into Kiwix's search now offers "Five Founders" or "Female Founders" rather than a filename, because the title pointer list a reader's suggestion search walks carries the real page titles.
A page with no `<title>` still falls back to its path, and the per-page title also flows into the `title` column of `kage parquet export`.
### Added
- `kage clone --mobile` makes legacy "font-era" sites readable on a phone.
Sites from the 1990s and early 2000s — paulgraham.com is a good example — embed typography directly in the HTML with `<font size="2" face="verdana">`, table-based layouts, and no viewport declaration.
A mobile browser receiving that markup without a viewport meta shrinks everything to desktop scale, and the `<font size="2">` instruction then makes the already-small text microscopic.
Passing `--mobile` injects two things into every saved page before it is written: a `<meta name="viewport" content="width=device-width, initial-scale=1">` tag so the browser stops shrinking, and a small `<style>` block that lifts the base font size to 18 px, inherits that size through `<font>` elements, caps the content width at 720 px, loosens line height to 1.7, and hides image-map navigation elements (usually a GIF served from an external CDN that 404s offline anyway).
The override is deliberately last in `<head>` so it wins specificity ties, and it does not touch pages that already carry a viewport and readable type sizes.
- The packing guide now documents how search works on a kage archive: title suggestions in any ZIM reader, and full-text search of page bodies through `kage parquet export` and DuckDB.
A Xapian full-text index is deliberately not written, since Xapian is GPL and kage is MIT.
## [0.3.4] - 2026-06-17
### Fixed
+5 -2
View File
@@ -39,8 +39,9 @@ type cloneFlags struct {
noRobots bool
noSitemap bool
headful bool
keepNoscript bool
chromeBin string
keepNoscript bool
mobileReadable bool
chromeBin string
controlURL string
noResume bool
refresh bool
@@ -86,6 +87,7 @@ func newCloneCmd() *cobra.Command {
fs.BoolVar(&f.noSitemap, "no-sitemap", false, "do not seed URLs from sitemap.xml")
fs.BoolVar(&f.headful, "headful", false, "run Chrome with a visible window (debugging)")
fs.BoolVar(&f.keepNoscript, "keep-noscript", false, "unwrap <noscript> content instead of dropping it")
fs.BoolVar(&f.mobileReadable, "mobile", false, "inject viewport and CSS overrides so legacy sites read comfortably on a phone")
fs.StringVar(&f.chromeBin, "chrome", "", "path to the Chrome/Chromium binary")
fs.StringVar(&f.controlURL, "control-url", "", "attach to an existing Chrome DevTools endpoint")
fs.BoolVar(&f.noResume, "no-resume", false, "do not reuse or write resume state")
@@ -140,6 +142,7 @@ func runClone(ctx context.Context, arg string, f *cloneFlags) error {
cfg.FollowSitemap = !f.noSitemap
cfg.Headless = !f.headful
cfg.KeepNoscript = f.keepNoscript
cfg.MobileReadable = f.mobileReadable
cfg.ChromeBin = f.chromeBin
cfg.ControlURL = f.controlURL
cfg.Resume = !f.noResume
+3 -2
View File
@@ -305,8 +305,9 @@ func (c *Cloner) processPage(ctx context.Context, j pageItem) {
asset.RewriteHTML(root, j.u, sink)
sanitize.CleanTree(root, sanitize.Options{
KeepNoscript: c.cfg.KeepNoscript,
Banner: "cloned by kage from " + j.u.String(),
KeepNoscript: c.cfg.KeepNoscript,
MobileReadable: c.cfg.MobileReadable,
Banner: "cloned by kage from " + j.u.String(),
})
var buf strings.Builder
+7 -6
View File
@@ -56,12 +56,13 @@ type Config struct {
ScopePrefix string
ExcludePaths []string
RespectRobots bool
FollowSitemap bool
Headless bool
KeepNoscript bool
ChromeBin string
ControlURL string
RespectRobots bool
FollowSitemap bool
Headless bool
KeepNoscript bool
MobileReadable bool
ChromeBin string
ControlURL string
// Resume loads the prior run's visited set and skips pages already written,
// so an interrupted or repeated clone picks up where it left off instead of
+12 -1
View File
@@ -41,7 +41,18 @@ kage open paulgraham.com.zim # read it back with kage
kiwix-serve paulgraham.com.zim # or serve it with Kiwix at http://localhost
```
You can also double-click the file in the [Kiwix desktop app](https://kiwix.org/en/applications/), or load it on Kiwix for Android or iOS to read your mirror on your phone. kage writes the metadata the format and `zimcheck` treat as mandatory, including the title, description, and the favicon Kiwix shows as the book icon in its library, so the archive shows up properly rather than as an untitled, iconless entry. One caveat: kage does not write the full-text search index that Kiwix's own packs ship with, so browsing works everywhere while in-reader search is limited.
You can also double-click the file in the [Kiwix desktop app](https://kiwix.org/en/applications/), or load it on Kiwix for Android or iOS to read your mirror on your phone. kage writes the metadata the format and `zimcheck` treat as mandatory, including the title, description, and the favicon Kiwix shows as the book icon in its library, so the archive shows up properly rather than as an untitled, iconless entry.
### Searching a packed mirror
Each page is stored under its own real `<title>`, so the search box in Kiwix (and any other ZIM reader) suggests pages by their readable title: type `five` into a paulgraham.com archive and it offers "Five Founders" and "Female Founders", not a filename. This title search works in every reader with no extra index.
What kage does not write is a Xapian full-text index, the separate search database that lets Kiwix match words inside a page's body. Xapian is GPL and kage is MIT, so linking it in would change kage's license; rather than do that, kage leaves full-text search to its own columnar export. `kage parquet export mirror.zim` writes one row per page with a `text` column, which [DuckDB](https://duckdb.org) searches with its `fts` extension or a plain `ILIKE`, fully offline and ranked:
```bash
kage parquet export paulgraham.com.zim -o paulgraham.parquet
duckdb -c "SELECT url FROM 'paulgraham.parquet' WHERE text ILIKE '%schlep blindness%'"
```
## A self-contained binary
+77
View File
@@ -354,3 +354,80 @@ func TestHandler(t *testing.T) {
t.Errorf("GET missing = %d, want 404", code)
}
}
// titleOf returns the stored entry title for a content url, scanning entries in
// url order since the reader exposes titles only through EntryAt.
func titleOf(t *testing.T, r *zim.Reader, url string) string {
t.Helper()
for i := uint32(0); i < r.Count(); i++ {
e, err := r.EntryAt(i)
if err != nil {
continue
}
if e.Namespace == zim.NamespaceContent && e.URL == url {
return e.Title
}
}
t.Fatalf("no content entry for %q", url)
return ""
}
// TestBuildZIMPageTitles checks that each HTML page entry carries its own
// <title> (collapsed to one line), that a page with no title falls back to its
// url, and that a non-HTML asset keeps the url as its title.
func TestBuildZIMPageTitles(t *testing.T) {
root := t.TempDir()
host := filepath.Join(root, "example.com")
files := map[string]string{
"index.html": "<!doctype html><title>Home</title><h1>Hi</h1>",
"essay/index.html": "<!doctype html><title>\n A Long\n Title </title><p>x</p>",
"bare/index.html": "<!doctype html><h1>No title here</h1>",
"logo.png": "\x89PNGfake",
}
for rel, body := range files {
p := filepath.Join(host, filepath.FromSlash(rel))
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
out := filepath.Join(t.TempDir(), "titles.zim")
if _, _, err := BuildZIM(host, ZIMOptions{Out: out, Date: "2026-06-14"}); err != nil {
t.Fatalf("BuildZIM: %v", err)
}
r, err := zim.Open(out)
if err != nil {
t.Fatalf("Open: %v", err)
}
defer func() { _ = r.Close() }()
cases := map[string]string{
"index.html": "Home",
"essay/index.html": "A Long Title", // newlines and runs collapsed to single spaces
"bare/index.html": "bare/index.html",
"logo.png": "logo.png",
}
for url, want := range cases {
if got := titleOf(t, r, url); got != want {
t.Errorf("title of %q = %q, want %q", url, got, want)
}
}
}
func TestCollapseSpaces(t *testing.T) {
cases := map[string]string{
" hello world ": "hello world",
"line\n\tone\r\n two": "line one two",
"": "",
" ": "",
"single": "single",
}
for in, want := range cases {
if got := collapseSpaces(in); got != want {
t.Errorf("collapseSpaces(%q) = %q, want %q", in, got, want)
}
}
}
+24 -5
View File
@@ -170,11 +170,16 @@ func buildWriter(mirrorDir string, opts ZIMOptions) (*zim.Writer, *clusterCache,
return err
}
mime := MimeForExt(rel)
title := ""
if mime == "text/html" {
htmlPages = append(htmlPages, rel)
// Store the page's real <title> on its entry, not the URL path, so a
// ZIM reader's search box and suggestions match the readable title.
// An empty result leaves the writer to fall back to the url.
title = htmlTitleOfBytes(data)
}
counts[mime]++
w.AddContent(zim.NamespaceContent, rel, "", mime, data)
w.AddContent(zim.NamespaceContent, rel, title, mime, data)
return nil
})
if walkErr != nil {
@@ -240,16 +245,30 @@ func htmlTitleOf(mirrorDir, mainURL string) string {
if mainURL == "" {
return ""
}
f, err := os.Open(filepath.Join(mirrorDir, filepath.FromSlash(mainURL)))
data, err := os.ReadFile(filepath.Join(mirrorDir, filepath.FromSlash(mainURL)))
if err != nil {
return ""
}
defer func() { _ = f.Close() }()
doc, err := html.Parse(f)
return htmlTitleOfBytes(data)
}
// htmlTitleOfBytes returns the first <title> in the given HTML with surrounding
// and internal whitespace collapsed, or "" if the bytes do not parse or carry
// no title. Page entries use it so a ZIM reader shows the page's real title
// instead of its URL path.
func htmlTitleOfBytes(data []byte) string {
doc, err := html.Parse(bytes.NewReader(data))
if err != nil {
return ""
}
return strings.TrimSpace(findTitle(doc))
return collapseSpaces(findTitle(doc))
}
// collapseSpaces trims s and replaces every run of whitespace (including the
// newlines a wrapped <title> can carry) with a single space, so a title reads
// as one clean line in a reader's library and search results.
func collapseSpaces(s string) string {
return strings.Join(strings.Fields(s), " ")
}
// findTitle returns the text of the first <title> element in depth-first order.
+89
View File
@@ -29,6 +29,14 @@ type Options struct {
// Banner, when non-empty, is inserted as an HTML comment at the top of the
// document.
Banner string
// MobileReadable injects a viewport meta tag and a small CSS block that
// makes legacy, font-era sites readable on mobile. It is intended for
// archives of 1990s/2000s sites that use <font size="2">, table layouts,
// and no viewport declaration — all of which render as microscopic text on
// a phone. The injected CSS overrides font sizes, loosens line height, caps
// the content width, and hides image-map navigation elements that are
// useless offline.
MobileReadable bool
}
// Report counts what was removed, for the run summary and for tests.
@@ -72,6 +80,10 @@ func CleanTree(root *html.Node, opts Options) Report {
var rep Report
clean(root, opts, &rep)
rep.CharsetAdded = ensureCharset(root)
if opts.MobileReadable {
ensureViewport(root)
injectMobileCSS(root)
}
if opts.Banner != "" {
insertBanner(root, opts.Banner)
}
@@ -291,6 +303,83 @@ func findElement(n *html.Node, a atom.Atom) *html.Node {
return nil
}
// mobileCSS is injected when MobileReadable is set. It rewrites font-era
// HTML for comfortable reading on a phone. Key rules:
//
// - box-sizing:border-box — makes padding predictable in a layout built with
// HTML width attributes, so our padding doesn't cause overflow.
// - body — no fixed max-width here; let the table rules handle width instead.
// overflow-x:hidden catches any stray overflow without a scrollbar.
// - font element — overrides in-HTML size/face attributes (e.g. <font size="2">).
// - [width],[height] — cancels all HTML attribute widths/heights on every
// element (tables, tds, imgs, etc.) so fixed-pixel columns become fluid.
// - table — fluid, auto layout, no horizontal scroll.
// - td — auto width so a three-column table (nav | spacer | content) collapses
// to one usable column once the nav td is hidden.
// - img — responsive: never wider than its container.
// - td:has(>img[usemap]) — hides the entire nav column td, not just the image
// inside it; hiding only the img left a tall empty white box.
// - td:has(>img[src*="trans_1x1"]) — hides the 26 px spacer column td (a 1×1
// transparent GIF whose only job was spacing in the original table layout).
const mobileCSS = `*{box-sizing:border-box}` +
`:root{font-size:18px}` +
`body{margin:0;padding:.75em 1em;line-height:1.7;font-family:Georgia,"Times New Roman",serif;overflow-x:hidden}` +
`font{font-size:1rem!important;font-family:inherit!important;color:inherit!important}` +
`[width]{width:auto!important;max-width:100%!important}` +
`[height]{height:auto!important}` +
`table{width:100%!important;max-width:100%!important;table-layout:auto!important;border-collapse:collapse!important;word-break:break-word}` +
`td,th{width:auto!important;max-width:100%!important;padding:.35em .5em!important;vertical-align:top!important;overflow-wrap:break-word}` +
`img{max-width:100%!important;height:auto!important}` +
`img[usemap],map{display:none!important}` +
`td:has(>img[usemap]),td:has(>map){display:none!important}` +
`img[src*="trans_1x1"],img[src*="spacer"],img[height="1"],img[width="1"]{display:none!important}` +
`td:has(>img[src*="trans_1x1"]:only-child),td:has(>img[height="1"]:only-child){display:none!important}`
// ensureViewport inserts <meta name="viewport" content="width=device-width,
// initial-scale=1"> at the top of <head> when the document does not already
// carry one. Without it a mobile browser shrinks the page to fit the screen
// at desktop scale, making text unreadably small regardless of CSS font sizes.
func ensureViewport(root *html.Node) {
head := findElement(root, atom.Head)
if head == nil {
return
}
// Check whether a viewport meta already exists.
for c := head.FirstChild; c != nil; c = c.NextSibling {
if c.Type == html.ElementNode && c.DataAtom == atom.Meta &&
strings.EqualFold(attr(c, "name"), "viewport") {
return
}
}
meta := &html.Node{
Type: html.ElementNode,
Data: "meta",
DataAtom: atom.Meta,
Attr: []html.Attribute{
{Key: "name", Val: "viewport"},
{Key: "content", Val: "width=device-width, initial-scale=1"},
},
}
head.InsertBefore(meta, head.FirstChild)
}
// injectMobileCSS appends a <style> block containing mobileCSS to <head>.
// It goes at the end of <head> so it wins specificity ties over any existing
// inline styles the page already carries.
func injectMobileCSS(root *html.Node) {
head := findElement(root, atom.Head)
if head == nil {
return
}
style := &html.Node{
Type: html.ElementNode,
Data: "style",
DataAtom: atom.Style,
}
style.AppendChild(&html.Node{Type: html.TextNode, Data: mobileCSS})
head.AppendChild(style)
}
// insertBanner prepends an HTML comment to the document.
func insertBanner(root *html.Node, text string) {
c := &html.Node{Type: html.CommentNode, Data: " " + text + " "}
+68
View File
@@ -197,6 +197,74 @@ func TestCharsetAddedWhenMissing(t *testing.T) {
}
}
func TestMobileReadableInjectsViewportAndCSS(t *testing.T) {
// A paulgraham.com-style page: no viewport, tiny <font size="2"> markup.
in := `<html><head><title>Essay</title></head>` +
`<body><font size="2" face="verdana"><p>Hello world.</p></font></body></html>`
out, _, err := Strip([]byte(in), Options{MobileReadable: true})
if err != nil {
t.Fatal(err)
}
s := string(out)
if !strings.Contains(s, `name="viewport"`) {
t.Error("viewport meta not injected")
}
if !strings.Contains(s, "width=device-width") {
t.Error("viewport content wrong")
}
if !strings.Contains(s, "font-size:18px") {
t.Error("mobile CSS not injected")
}
if !strings.Contains(s, "font{font-size:1rem") {
t.Error("font override not in mobile CSS")
}
// Fluid table rules must be present.
if !strings.Contains(s, "table{width:100%") {
t.Error("fluid table rule missing")
}
// [width] override must be present so fixed HTML width attributes are cancelled.
if !strings.Contains(s, "[width]{width:auto") {
t.Error("[width] override missing")
}
}
func TestMobileReadableHidesNavColumnTd(t *testing.T) {
// paulgraham.com wraps its image-map nav in a <td>. Hiding only the img
// leaves a tall empty box; the CSS must also target the containing td.
in := `<html><head><title>T</title></head><body>` +
`<table><tr>` +
`<td><map name="nav"><area shape="rect" coords="0,0,67,21" href="index.html"></map>` +
`<img src="nav.gif" width="69" height="357" usemap="#nav"/></td>` +
`<td><img src="https://cdn.example.com/trans_1x1.gif" height="1" width="26"/></td>` +
`<td><p>Essay text.</p></td>` +
`</tr></table>` +
`</body></html>`
out, _, err := Strip([]byte(in), Options{MobileReadable: true})
if err != nil {
t.Fatal(err)
}
s := string(out)
// td:has(>img[usemap]) rule must be present so the whole nav column is hidden.
if !strings.Contains(s, "td:has(>img[usemap])") {
t.Error("td:has(>img[usemap]) rule missing from mobile CSS")
}
// td:has(>img[src*="trans_1x1"]) rule must be present for spacer column.
if !strings.Contains(s, `td:has(>img[src*="trans_1x1"]`) {
t.Error(`td:has(>img[src*="trans_1x1"]) spacer-column rule missing from mobile CSS`)
}
}
func TestMobileReadableSkipsExistingViewport(t *testing.T) {
in := `<html><head><meta name="viewport" content="width=device-width"><title>x</title></head><body></body></html>`
out, _, err := Strip([]byte(in), Options{MobileReadable: true})
if err != nil {
t.Fatal(err)
}
if n := strings.Count(string(out), `name="viewport"`); n != 1 {
t.Errorf("viewport injected when one already existed (count %d)", n)
}
}
func TestCharsetNotDuplicated(t *testing.T) {
// A page that already declares a charset, in either form, is left alone.
cases := []string{