wehub-resource-sync
129 lines
6.9 KiB
Markdown
129 lines
6.9 KiB
Markdown
# tree-sitter-cobol.wasm — provenance & rebuild
|
|
|
|
`src/extraction/wasm/tree-sitter-cobol.wasm` is built from
|
|
[yutaro-sakamoto/tree-sitter-cobol](https://github.com/yutaro-sakamoto/tree-sitter-cobol)
|
|
at commit `e99dbdc3d800d5fa2796476efd60af91f6b43d93` with the patch in
|
|
`tree-sitter-cobol.patch` applied (grammar.js + src/scanner.c; everything else
|
|
is regenerated by `tree-sitter generate`).
|
|
|
|
## What the patch adds
|
|
|
|
The upstream grammar (COBOL85, derived from opensource-cobol, NIST-tested)
|
|
parses batch COBOL well but chokes on the constructs that dominate real
|
|
mainframe and GnuCOBOL code:
|
|
|
|
1. **`EXEC ... END-EXEC` blocks** (CICS / SQL / DLI). Upstream has no EXEC
|
|
support at all — the tokens get absorbed into the preceding statement until
|
|
one breaks the parse, cascading hundreds of lines into one ERROR. The patch
|
|
adds an `EXEC_BLOCK` external-scanner token that consumes the whole block,
|
|
surfaced as a single `exec_statement` node (valid as a procedure statement
|
|
and as a data-division entry, for `EXEC SQL INCLUDE`/`DECLARE`).
|
|
2. **`NOT=`** without a space (IBM COBOL accepts it).
|
|
3. **`FD <name>.` followed by `COPY`** for the record layout — upstream
|
|
required a literal record description after every FD (also fixed for
|
|
LINKAGE SECTION).
|
|
4. **`COPY ... REPLACING ==pseudo-text== BY ==pseudo-text==`** with multiple
|
|
replacement pairs. (Upstream's `replacing_clause` never consumed the
|
|
REPLACING keyword and allowed only one WORD/string pair.)
|
|
5. **Single-quote string continuation lines** (hyphen in the indicator
|
|
column). Upstream's scanner handled continuation only for double-quoted
|
|
strings; the patch generalizes the quote character and handles doubled-
|
|
quote escapes (`'DON''T'`).
|
|
6. **`FUNCTION <intrinsic>(refmod)`** — upstream's generic FUNCTION fallback
|
|
took arguments but not a reference-modification suffix
|
|
(`FUNCTION CURRENT-DATE(1:4)`; the dedicated intrinsic tokens like
|
|
`CURRENT-DATE-FUNC` only match opensource-cobol's *preprocessed* names).
|
|
7. **Standalone copybook entry point** — a `copybook_fragment` start
|
|
alternative so `.cpy` files (data records or procedure paragraphs, no
|
|
IDENTIFICATION DIVISION) parse without a wrapper. A file is either
|
|
programs or one fragment, keeping program suffixes unambiguous.
|
|
8. **Wide mode for free-format source** — the extractor converts free-format
|
|
COBOL to fixed by left-padding 7 columns and plants a `CGWIDE` sentinel in
|
|
the first line's sequence area; the scanner then relaxes the column-72
|
|
right margin (free-format lines routinely exceed it). One byte of scanner
|
|
state, carried through serialize/deserialize. Fixed-format files are
|
|
untouched.
|
|
9. **COBOL-2002 / GnuCOBOL surface**: `BINARY-LONG-LONG`, `FLOAT-LONG`,
|
|
`FLOAT-SHORT` usages; `PROGRAM-ID. X IS RECURSIVE.`; relational `WHEN`
|
|
objects (`WHEN > 0`); abbreviated combined relations
|
|
(`WHEN X < 0 OR > Y`); bitwise operators (`B-AND`, `B-OR`, `B-XOR`,
|
|
`B-NOT`, `B-SHIFT-L/-LC`, `B-SHIFT-R/-RC`); the `FREE` statement;
|
|
`VALUE <constant-name>`; `PIC X(CONSTANT-NAME)`; `>>` compiler
|
|
directives as comments; `ENTRY 'literal' USING ...` (IMS batch
|
|
alternate entry points); empty `DATE-COMPILED.` / `DATE-WRITTEN.` /
|
|
`SECURITY.` headers.
|
|
10. **`CALL ... GIVING`** — upstream *intended* to support it but a misnested
|
|
`field()` call swallowed the GIVING alternative entirely.
|
|
|
|
## Measured parse health (at vendoring time)
|
|
|
|
| Corpus | Clean parses |
|
|
|---|---|
|
|
| AWS CardDemo, all programs incl. DB2/IMS/MQ variants (44 `.cbl`) | 43/44 — upstream: 9/31 on the base set alone. The one residual (a period-less `EXEC SQL INCLUDE` between paragraphs) is repaired by the extractor's preParse, which terminates the single-line form with a period. |
|
|
| AWS CardDemo copybooks (29 `.cpy`) | 28/29 — upstream: 0/29 (the failure is a `COPY REPLACING` template containing `(TESTVAR1)` placeholders, not valid COBOL pre-substitution) |
|
|
| NIST COBOL85 suite (382 programs) | 373/382 — unchanged from upstream |
|
|
| CobolCraft (17 free-format GnuCOBOL programs, via wide mode) | 17/17 — upstream: unparseable (free format) |
|
|
| Upstream corpus tests | 1 pre-existing failure (`comment`), no new failures |
|
|
|
|
|
|
## Rebuild
|
|
|
|
```bash
|
|
git clone https://github.com/yutaro-sakamoto/tree-sitter-cobol
|
|
cd tree-sitter-cobol
|
|
git checkout e99dbdc3d800d5fa2796476efd60af91f6b43d93
|
|
git apply path/to/tree-sitter-cobol.patch
|
|
# tree-sitter 0.24.x needs a tree-sitter.json; grammar name must stay COBOL
|
|
# (the C symbols are tree_sitter_COBOL*):
|
|
cat > tree-sitter.json <<'JSON'
|
|
{
|
|
"grammars": [
|
|
{ "name": "COBOL", "camelcase": "COBOL", "scope": "source.cobol",
|
|
"path": ".", "file-types": ["cbl", "cob", "cpy"] }
|
|
],
|
|
"metadata": { "version": "0.1.1", "license": "MIT",
|
|
"description": "COBOL grammar for tree-sitter",
|
|
"links": { "repository": "https://github.com/yutaro-sakamoto/tree-sitter-cobol" } }
|
|
}
|
|
JSON
|
|
npm install tree-sitter-cli@0.24.5
|
|
npx tree-sitter generate
|
|
npx tree-sitter build --wasm -o tree-sitter-cobol.wasm # needs emscripten or Docker
|
|
```
|
|
|
|
The patches are written to be upstreamable — each is independent and comes
|
|
with the failing construct documented above.
|
|
|
|
## Upstreaming
|
|
|
|
Sent as [yutaro-sakamoto/tree-sitter-cobol#41](https://github.com/yutaro-sakamoto/tree-sitter-cobol/pull/41)
|
|
(branch `real-world-cobol-sources` on the colbymchenry fork). If upstream
|
|
merges it, the vendored wasm can track upstream releases instead of this
|
|
patch. Until then, `git apply tree-sitter-cobol.patch` on upstream commit
|
|
`e99dbdc3` reproduces the fork exactly. The PR body as sent:
|
|
|
|
> **Parse real-world CICS/DB2 and GnuCOBOL sources**
|
|
>
|
|
> This adds the constructs that block the grammar on production COBOL, found
|
|
> while integrating it into a code-indexing tool. Measured on public corpora:
|
|
> AWS CardDemo goes from 9/31 to 43/44 clean parses, CobolCraft (free-format
|
|
> GnuCOBOL) from 0 to 17/17, NIST COBOL85 unchanged at 373/382, and the
|
|
> existing corpus tests keep their single pre-existing failure (`comment`).
|
|
>
|
|
> - `EXEC ... END-EXEC` blocks as an external-scanner token (`exec_statement`)
|
|
> - Fixed-format single-quote string continuation + doubled-quote escapes
|
|
> - `COPY ... REPLACING ==pseudo-text==` with multiple pairs
|
|
> - `FD`/`LINKAGE SECTION` record descriptions supplied via `COPY`
|
|
> - `NOT=`, `CALL ... GIVING` (a misnested `field()` dropped it), `FREE`,
|
|
> `ENTRY`, `PROGRAM-ID ... IS RECURSIVE`, empty `DATE-COMPILED.` headers
|
|
> - `FUNCTION <name>(refmod)`, `VALUE <constant>`, `PIC X(CONSTANT)`
|
|
> - Relational and abbreviated-combined `WHEN` objects, bitwise `B-*` ops,
|
|
> `>>` directives-as-comments, COBOL-2002 usages (`BINARY-LONG-LONG`, ...)
|
|
> - A `copybook_fragment` entry point so standalone `.cpy` files parse
|
|
> - An opt-in wide mode (sentinel in the first line's sequence area) that a
|
|
> free-format preprocessor can use to relax the column-72 margin
|
|
>
|
|
> Happy to split any of these out or adjust naming/style. Each item is
|
|
> independent; `tree-sitter test` passes minus the one pre-existing failure.
|
|
|