Tam Nguyen Duc
0849557725
kage parquet export turns a packed archive into a columnar Parquet table, one row per entry, and kage parquet import rebuilds the archive from it. The table follows the open-index/open-markdown field names (doc_id, url, host, crawl_date, content_length, text_length, text) so a kage export sits next to other web-crawl datasets on Hugging Face and reads straight into DuckDB or pandas. Alongside those columns it keeps the raw content bytes and the ZIM structure (namespace, redirect target), so the round trip is lossless: a ZIM exported and reimported is byte-identical. doc_id is a deterministic UUID v5 of the page URL. HTML pages also get a derived plain-text column for full-text search and training use; it plays no part in the round trip, which rebuilds pages from the stored content.
56 lines
2.2 KiB
Modula-2
56 lines
2.2 KiB
Modula-2
module github.com/tamnd/kage
|
|
|
|
go 1.26.4
|
|
|
|
require (
|
|
charm.land/lipgloss/v2 v2.0.0-beta.3.0.20251106193318-19329a3e8410
|
|
github.com/charmbracelet/fang v1.0.0
|
|
github.com/go-rod/rod v0.116.2
|
|
github.com/go-rod/stealth v0.4.9
|
|
github.com/google/uuid v1.6.0
|
|
github.com/klauspost/compress v1.18.6
|
|
github.com/parquet-go/parquet-go v0.30.1
|
|
github.com/spf13/cobra v1.10.2
|
|
github.com/webview/webview_go v0.0.0-20240831120633-6173450d4dd6
|
|
golang.org/x/image v0.42.0
|
|
golang.org/x/net v0.56.0
|
|
)
|
|
|
|
require (
|
|
github.com/andybalholm/brotli v1.1.1 // indirect
|
|
github.com/charmbracelet/colorprofile v0.3.3 // indirect
|
|
github.com/charmbracelet/ultraviolet v0.0.0-20251106190538-99ea45596692 // indirect
|
|
github.com/charmbracelet/x/ansi v0.11.0 // indirect
|
|
github.com/charmbracelet/x/exp/charmtone v0.0.0-20250603201427-c31516f43444 // indirect
|
|
github.com/charmbracelet/x/term v0.2.2 // indirect
|
|
github.com/charmbracelet/x/termios v0.1.1 // indirect
|
|
github.com/charmbracelet/x/windows v0.2.2 // indirect
|
|
github.com/clipperhouse/displaywidth v0.4.1 // indirect
|
|
github.com/clipperhouse/stringish v0.1.1 // indirect
|
|
github.com/clipperhouse/uax29/v2 v2.3.0 // indirect
|
|
github.com/inconshreveable/mousetrap v1.1.0 // indirect
|
|
github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
|
|
github.com/mattn/go-runewidth v0.0.19 // indirect
|
|
github.com/muesli/cancelreader v0.2.2 // indirect
|
|
github.com/muesli/mango v0.1.0 // indirect
|
|
github.com/muesli/mango-cobra v1.2.0 // indirect
|
|
github.com/muesli/mango-pflag v0.1.0 // indirect
|
|
github.com/muesli/roff v0.1.0 // indirect
|
|
github.com/parquet-go/bitpack v1.0.0 // indirect
|
|
github.com/parquet-go/jsonlite v1.0.0 // indirect
|
|
github.com/pierrec/lz4/v4 v4.1.21 // indirect
|
|
github.com/rivo/uniseg v0.4.7 // indirect
|
|
github.com/spf13/pflag v1.0.9 // indirect
|
|
github.com/twpayne/go-geom v1.6.1 // indirect
|
|
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
|
|
github.com/ysmood/fetchup v0.2.3 // indirect
|
|
github.com/ysmood/goob v0.4.0 // indirect
|
|
github.com/ysmood/got v0.40.0 // indirect
|
|
github.com/ysmood/gson v0.7.3 // indirect
|
|
github.com/ysmood/leakless v0.9.0 // indirect
|
|
golang.org/x/sync v0.21.0 // indirect
|
|
golang.org/x/sys v0.46.0 // indirect
|
|
golang.org/x/text v0.38.0 // indirect
|
|
google.golang.org/protobuf v1.34.2 // indirect
|
|
)
|