colbymchenry--codegraph/docs/grammars/tree-sitter-cobol.md

tree-sitter-cobol.wasm — provenance & rebuild

src/extraction/wasm/tree-sitter-cobol.wasm is built from yutaro-sakamoto/tree-sitter-cobol at commit e99dbdc3d800d5fa2796476efd60af91f6b43d93 with the patch in tree-sitter-cobol.patch applied (grammar.js + src/scanner.c; everything else is regenerated by tree-sitter generate).

What the patch adds

The upstream grammar (COBOL85, derived from opensource-cobol, NIST-tested) parses batch COBOL well but chokes on the constructs that dominate real mainframe and GnuCOBOL code:

1. EXEC ... END-EXEC blocks (CICS / SQL / DLI). Upstream has no EXEC support at all — the tokens get absorbed into the preceding statement until one breaks the parse, cascading hundreds of lines into one ERROR. The patch adds an EXEC_BLOCK external-scanner token that consumes the whole block, surfaced as a single exec_statement node (valid as a procedure statement and as a data-division entry, for EXEC SQL INCLUDE/DECLARE).
2. NOT= without a space (IBM COBOL accepts it).
3. FD <name>. followed by COPY for the record layout — upstream required a literal record description after every FD (also fixed for LINKAGE SECTION).
4. COPY ... REPLACING ==pseudo-text== BY ==pseudo-text== with multiple replacement pairs. (Upstream's replacing_clause never consumed the REPLACING keyword and allowed only one WORD/string pair.)
5. Single-quote string continuation lines (hyphen in the indicator column). Upstream's scanner handled continuation only for double-quoted strings; the patch generalizes the quote character and handles doubled- quote escapes ('DON''T').
6. FUNCTION <intrinsic>(refmod) — upstream's generic FUNCTION fallback took arguments but not a reference-modification suffix (FUNCTION CURRENT-DATE(1:4); the dedicated intrinsic tokens like CURRENT-DATE-FUNC only match opensource-cobol's preprocessed names).
7. Standalone copybook entry point — a copybook_fragment start alternative so .cpy files (data records or procedure paragraphs, no IDENTIFICATION DIVISION) parse without a wrapper. A file is either programs or one fragment, keeping program suffixes unambiguous.
8. Wide mode for free-format source — the extractor converts free-format COBOL to fixed by left-padding 7 columns and plants a CGWIDE sentinel in the first line's sequence area; the scanner then relaxes the column-72 right margin (free-format lines routinely exceed it). One byte of scanner state, carried through serialize/deserialize. Fixed-format files are untouched.
9. COBOL-2002 / GnuCOBOL surface: BINARY-LONG-LONG, FLOAT-LONG, FLOAT-SHORT usages; PROGRAM-ID. X IS RECURSIVE.; relational WHEN objects (WHEN > 0); abbreviated combined relations (WHEN X < 0 OR > Y); bitwise operators (B-AND, B-OR, B-XOR, B-NOT, B-SHIFT-L/-LC, B-SHIFT-R/-RC); the FREE statement; VALUE <constant-name>; PIC X(CONSTANT-NAME); >> compiler directives as comments; ENTRY 'literal' USING ... (IMS batch alternate entry points); empty DATE-COMPILED. / DATE-WRITTEN. / SECURITY. headers.
10. CALL ... GIVING — upstream intended to support it but a misnested field() call swallowed the GIVING alternative entirely.

Measured parse health (at vendoring time)

Corpus	Clean parses
AWS CardDemo, all programs incl. DB2/IMS/MQ variants (44 .cbl)	43/44 — upstream: 9/31 on the base set alone. The one residual (a period-less EXEC SQL INCLUDE between paragraphs) is repaired by the extractor's preParse, which terminates the single-line form with a period.
AWS CardDemo copybooks (29 .cpy)	28/29 — upstream: 0/29 (the failure is a COPY REPLACING template containing (TESTVAR1) placeholders, not valid COBOL pre-substitution)
NIST COBOL85 suite (382 programs)	373/382 — unchanged from upstream
CobolCraft (17 free-format GnuCOBOL programs, via wide mode)	17/17 — upstream: unparseable (free format)
Upstream corpus tests	1 pre-existing failure (comment), no new failures

Rebuild

git clone https://github.com/yutaro-sakamoto/tree-sitter-cobol
cd tree-sitter-cobol
git checkout e99dbdc3d800d5fa2796476efd60af91f6b43d93
git apply path/to/tree-sitter-cobol.patch
# tree-sitter 0.24.x needs a tree-sitter.json; grammar name must stay COBOL
# (the C symbols are tree_sitter_COBOL*):
cat > tree-sitter.json <<'JSON'
{
  "grammars": [
    { "name": "COBOL", "camelcase": "COBOL", "scope": "source.cobol",
      "path": ".", "file-types": ["cbl", "cob", "cpy"] }
  ],
  "metadata": { "version": "0.1.1", "license": "MIT",
    "description": "COBOL grammar for tree-sitter",
    "links": { "repository": "https://github.com/yutaro-sakamoto/tree-sitter-cobol" } }
}
JSON
npm install tree-sitter-cli@0.24.5
npx tree-sitter generate
npx tree-sitter build --wasm -o tree-sitter-cobol.wasm   # needs emscripten or Docker

The patches are written to be upstreamable — each is independent and comes with the failing construct documented above.

Upstreaming

Sent as yutaro-sakamoto/tree-sitter-cobol#41 (branch real-world-cobol-sources on the colbymchenry fork). If upstream merges it, the vendored wasm can track upstream releases instead of this patch. Until then, git apply tree-sitter-cobol.patch on upstream commit e99dbdc3 reproduces the fork exactly. The PR body as sent:

Parse real-world CICS/DB2 and GnuCOBOL sources

This adds the constructs that block the grammar on production COBOL, found while integrating it into a code-indexing tool. Measured on public corpora: AWS CardDemo goes from 9/31 to 43/44 clean parses, CobolCraft (free-format GnuCOBOL) from 0 to 17/17, NIST COBOL85 unchanged at 373/382, and the existing corpus tests keep their single pre-existing failure (comment).

• EXEC ... END-EXEC blocks as an external-scanner token (exec_statement)
• Fixed-format single-quote string continuation + doubled-quote escapes
• COPY ... REPLACING ==pseudo-text== with multiple pairs
• FD/LINKAGE SECTION record descriptions supplied via COPY
• NOT=, CALL ... GIVING (a misnested field() dropped it), FREE, ENTRY, PROGRAM-ID ... IS RECURSIVE, empty DATE-COMPILED. headers
• FUNCTION <name>(refmod), VALUE <constant>, PIC X(CONSTANT)
• Relational and abbreviated-combined WHEN objects, bitwise B-* ops, >> directives-as-comments, COBOL-2002 usages (BINARY-LONG-LONG, ...)
• A copybook_fragment entry point so standalone .cpy files parse
• An opt-in wide mode (sentinel in the first line's sequence area) that a free-format preprocessor can use to relax the column-72 margin

Happy to split any of these out or adjust naming/style. Each item is independent; tree-sitter test passes minus the one pre-existing failure.