# tree-sitter-cobol.wasm — provenance & rebuild

`src/extraction/wasm/tree-sitter-cobol.wasm` is built from
[yutaro-sakamoto/tree-sitter-cobol](https://github.com/yutaro-sakamoto/tree-sitter-cobol)
at commit `e99dbdc3d800d5fa2796476efd60af91f6b43d93` with the patch in
`tree-sitter-cobol.patch` applied (grammar.js + src/scanner.c; everything else
is regenerated by `tree-sitter generate`).

## What the patch adds

The upstream grammar (COBOL85, derived from opensource-cobol, NIST-tested)
parses batch COBOL well but chokes on the constructs that dominate real
mainframe and GnuCOBOL code:

1. **`EXEC ... END-EXEC` blocks** (CICS / SQL / DLI). Upstream has no EXEC
   support at all — the tokens get absorbed into the preceding statement until
   one breaks the parse, cascading hundreds of lines into one ERROR. The patch
   adds an `EXEC_BLOCK` external-scanner token that consumes the whole block,
   surfaced as a single `exec_statement` node (valid as a procedure statement
   and as a data-division entry, for `EXEC SQL INCLUDE`/`DECLARE`).
2. **`NOT=`** without a space (IBM COBOL accepts it).
3. **`FD <name>.` followed by `COPY`** for the record layout — upstream
   required a literal record description after every FD (also fixed for
   LINKAGE SECTION).
4. **`COPY ... REPLACING ==pseudo-text== BY ==pseudo-text==`** with multiple
   replacement pairs. (Upstream's `replacing_clause` never consumed the
   REPLACING keyword and allowed only one WORD/string pair.)
5. **Single-quote string continuation lines** (hyphen in the indicator
   column). Upstream's scanner handled continuation only for double-quoted
   strings; the patch generalizes the quote character and handles doubled-
   quote escapes (`'DON''T'`).
6. **`FUNCTION <intrinsic>(refmod)`** — upstream's generic FUNCTION fallback
   took arguments but not a reference-modification suffix
   (`FUNCTION CURRENT-DATE(1:4)`; the dedicated intrinsic tokens like
   `CURRENT-DATE-FUNC` only match opensource-cobol's *preprocessed* names).
7. **Standalone copybook entry point** — a `copybook_fragment` start
   alternative so `.cpy` files (data records or procedure paragraphs, no
   IDENTIFICATION DIVISION) parse without a wrapper. A file is either
   programs or one fragment, keeping program suffixes unambiguous.
8. **Wide mode for free-format source** — the extractor converts free-format
   COBOL to fixed by left-padding 7 columns and plants a `CGWIDE` sentinel in
   the first line's sequence area; the scanner then relaxes the column-72
   right margin (free-format lines routinely exceed it). One byte of scanner
   state, carried through serialize/deserialize. Fixed-format files are
   untouched.
9. **COBOL-2002 / GnuCOBOL surface**: `BINARY-LONG-LONG`, `FLOAT-LONG`,
   `FLOAT-SHORT` usages; `PROGRAM-ID. X IS RECURSIVE.`; relational `WHEN`
   objects (`WHEN > 0`); abbreviated combined relations
   (`WHEN X < 0 OR > Y`); bitwise operators (`B-AND`, `B-OR`, `B-XOR`,
   `B-NOT`, `B-SHIFT-L/-LC`, `B-SHIFT-R/-RC`); the `FREE` statement;
   `VALUE <constant-name>`; `PIC X(CONSTANT-NAME)`; `>>` compiler
   directives as comments; `ENTRY 'literal' USING ...` (IMS batch
   alternate entry points); empty `DATE-COMPILED.` / `DATE-WRITTEN.` /
   `SECURITY.` headers.
10. **`CALL ... GIVING`** — upstream *intended* to support it but a misnested
    `field()` call swallowed the GIVING alternative entirely.

## Measured parse health (at vendoring time)

| Corpus | Clean parses |
|---|---|
| AWS CardDemo, all programs incl. DB2/IMS/MQ variants (44 `.cbl`) | 43/44 — upstream: 9/31 on the base set alone. The one residual (a period-less `EXEC SQL INCLUDE` between paragraphs) is repaired by the extractor's preParse, which terminates the single-line form with a period. |
| AWS CardDemo copybooks (29 `.cpy`) | 28/29 — upstream: 0/29 (the failure is a `COPY REPLACING` template containing `(TESTVAR1)` placeholders, not valid COBOL pre-substitution) |
| NIST COBOL85 suite (382 programs) | 373/382 — unchanged from upstream |
| CobolCraft (17 free-format GnuCOBOL programs, via wide mode) | 17/17 — upstream: unparseable (free format) |
| Upstream corpus tests | 1 pre-existing failure (`comment`), no new failures |


## Rebuild

```bash
git clone https://github.com/yutaro-sakamoto/tree-sitter-cobol
cd tree-sitter-cobol
git checkout e99dbdc3d800d5fa2796476efd60af91f6b43d93
git apply path/to/tree-sitter-cobol.patch
# tree-sitter 0.24.x needs a tree-sitter.json; grammar name must stay COBOL
# (the C symbols are tree_sitter_COBOL*):
cat > tree-sitter.json <<'JSON'
{
  "grammars": [
    { "name": "COBOL", "camelcase": "COBOL", "scope": "source.cobol",
      "path": ".", "file-types": ["cbl", "cob", "cpy"] }
  ],
  "metadata": { "version": "0.1.1", "license": "MIT",
    "description": "COBOL grammar for tree-sitter",
    "links": { "repository": "https://github.com/yutaro-sakamoto/tree-sitter-cobol" } }
}
JSON
npm install tree-sitter-cli@0.24.5
npx tree-sitter generate
npx tree-sitter build --wasm -o tree-sitter-cobol.wasm   # needs emscripten or Docker
```

The patches are written to be upstreamable — each is independent and comes
with the failing construct documented above.

## Upstreaming

Sent as [yutaro-sakamoto/tree-sitter-cobol#41](https://github.com/yutaro-sakamoto/tree-sitter-cobol/pull/41)
(branch `real-world-cobol-sources` on the colbymchenry fork). If upstream
merges it, the vendored wasm can track upstream releases instead of this
patch. Until then, `git apply tree-sitter-cobol.patch` on upstream commit
`e99dbdc3` reproduces the fork exactly. The PR body as sent:

> **Parse real-world CICS/DB2 and GnuCOBOL sources**
>
> This adds the constructs that block the grammar on production COBOL, found
> while integrating it into a code-indexing tool. Measured on public corpora:
> AWS CardDemo goes from 9/31 to 43/44 clean parses, CobolCraft (free-format
> GnuCOBOL) from 0 to 17/17, NIST COBOL85 unchanged at 373/382, and the
> existing corpus tests keep their single pre-existing failure (`comment`).
>
> - `EXEC ... END-EXEC` blocks as an external-scanner token (`exec_statement`)
> - Fixed-format single-quote string continuation + doubled-quote escapes
> - `COPY ... REPLACING ==pseudo-text==` with multiple pairs
> - `FD`/`LINKAGE SECTION` record descriptions supplied via `COPY`
> - `NOT=`, `CALL ... GIVING` (a misnested `field()` dropped it), `FREE`,
>   `ENTRY`, `PROGRAM-ID ... IS RECURSIVE`, empty `DATE-COMPILED.` headers
> - `FUNCTION <name>(refmod)`, `VALUE <constant>`, `PIC X(CONSTANT)`
> - Relational and abbreviated-combined `WHEN` objects, bitwise `B-*` ops,
>   `>>` directives-as-comments, COBOL-2002 usages (`BINARY-LONG-LONG`, ...)
> - A `copybook_fragment` entry point so standalone `.cpy` files parse
> - An opt-in wide mode (sentinel in the first line's sequence area) that a
>   free-format preprocessor can use to relax the column-72 margin
>
> Happy to split any of these out or adjust naming/style. Each item is
> independent; `tree-sitter test` passes minus the one pre-existing failure.