wehub/colbymchenry--codegraph
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
Deploy site to GitHub Pages / build (push) Failing after 1s
Details
Deploy site to GitHub Pages / deploy (push) Has been skipped
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:02:56 +08:00
commit 5f98960d22
495 changed files with 149129 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/grammars/tree-sitter-cobol.md
+128
View File
@@ -0,0 +1,128 @@
# tree-sitter-cobol.wasm — provenance & rebuild
`src/extraction/wasm/tree-sitter-cobol.wasm` is built from
[yutaro-sakamoto/tree-sitter-cobol](https://github.com/yutaro-sakamoto/tree-sitter-cobol)
at commit `e99dbdc3d800d5fa2796476efd60af91f6b43d93` with the patch in
`tree-sitter-cobol.patch` applied (grammar.js + src/scanner.c; everything else
is regenerated by `tree-sitter generate`).
## What the patch adds
The upstream grammar (COBOL85, derived from opensource-cobol, NIST-tested)
parses batch COBOL well but chokes on the constructs that dominate real
mainframe and GnuCOBOL code:
1. **`EXEC ... END-EXEC` blocks** (CICS / SQL / DLI). Upstream has no EXEC
support at all — the tokens get absorbed into the preceding statement until
one breaks the parse, cascading hundreds of lines into one ERROR. The patch
adds an `EXEC_BLOCK` external-scanner token that consumes the whole block,
surfaced as a single `exec_statement` node (valid as a procedure statement
and as a data-division entry, for `EXEC SQL INCLUDE`/`DECLARE`).
2. **`NOT=`** without a space (IBM COBOL accepts it).
3. **`FD <name>.` followed by `COPY`** for the record layout — upstream
required a literal record description after every FD (also fixed for
LINKAGE SECTION).
4. **`COPY ... REPLACING ==pseudo-text== BY ==pseudo-text==`** with multiple
replacement pairs. (Upstream's `replacing_clause` never consumed the
REPLACING keyword and allowed only one WORD/string pair.)
5. **Single-quote string continuation lines** (hyphen in the indicator
column). Upstream's scanner handled continuation only for double-quoted
strings; the patch generalizes the quote character and handles doubled-
quote escapes (`'DON''T'`).
6. **`FUNCTION <intrinsic>(refmod)`** — upstream's generic FUNCTION fallback
took arguments but not a reference-modification suffix
(`FUNCTION CURRENT-DATE(1:4)`; the dedicated intrinsic tokens like
`CURRENT-DATE-FUNC` only match opensource-cobol's *preprocessed* names).
7. **Standalone copybook entry point** — a `copybook_fragment` start
alternative so `.cpy` files (data records or procedure paragraphs, no
IDENTIFICATION DIVISION) parse without a wrapper. A file is either
programs or one fragment, keeping program suffixes unambiguous.
8. **Wide mode for free-format source** — the extractor converts free-format
COBOL to fixed by left-padding 7 columns and plants a `CGWIDE` sentinel in
the first line's sequence area; the scanner then relaxes the column-72
right margin (free-format lines routinely exceed it). One byte of scanner
state, carried through serialize/deserialize. Fixed-format files are
untouched.
9. **COBOL-2002 / GnuCOBOL surface**: `BINARY-LONG-LONG`, `FLOAT-LONG`,
`FLOAT-SHORT` usages; `PROGRAM-ID. X IS RECURSIVE.`; relational `WHEN`
objects (`WHEN > 0`); abbreviated combined relations
(`WHEN X < 0 OR > Y`); bitwise operators (`B-AND`, `B-OR`, `B-XOR`,
`B-NOT`, `B-SHIFT-L/-LC`, `B-SHIFT-R/-RC`); the `FREE` statement;
`VALUE <constant-name>`; `PIC X(CONSTANT-NAME)`; `>>` compiler
directives as comments; `ENTRY 'literal' USING ...` (IMS batch
alternate entry points); empty `DATE-COMPILED.` / `DATE-WRITTEN.` /
`SECURITY.` headers.
10. **`CALL ... GIVING`** — upstream *intended* to support it but a misnested
`field()` call swallowed the GIVING alternative entirely.
## Measured parse health (at vendoring time)
| Corpus | Clean parses |
|---|---|
| AWS CardDemo, all programs incl. DB2/IMS/MQ variants (44 `.cbl`) | 43/44 — upstream: 9/31 on the base set alone. The one residual (a period-less `EXEC SQL INCLUDE` between paragraphs) is repaired by the extractor's preParse, which terminates the single-line form with a period. |
| AWS CardDemo copybooks (29 `.cpy`) | 28/29 — upstream: 0/29 (the failure is a `COPY REPLACING` template containing `(TESTVAR1)` placeholders, not valid COBOL pre-substitution) |
| NIST COBOL85 suite (382 programs) | 373/382 — unchanged from upstream |
| CobolCraft (17 free-format GnuCOBOL programs, via wide mode) | 17/17 — upstream: unparseable (free format) |
| Upstream corpus tests | 1 pre-existing failure (`comment`), no new failures |
## Rebuild
```bash
git clone https://github.com/yutaro-sakamoto/tree-sitter-cobol
cd tree-sitter-cobol
git checkout e99dbdc3d800d5fa2796476efd60af91f6b43d93
git apply path/to/tree-sitter-cobol.patch
# tree-sitter 0.24.x needs a tree-sitter.json; grammar name must stay COBOL
# (the C symbols are tree_sitter_COBOL*):
cat > tree-sitter.json <<'JSON'
{
"grammars": [
{ "name": "COBOL", "camelcase": "COBOL", "scope": "source.cobol",
"path": ".", "file-types": ["cbl", "cob", "cpy"] }
],
"metadata": { "version": "0.1.1", "license": "MIT",
"description": "COBOL grammar for tree-sitter",
"links": { "repository": "https://github.com/yutaro-sakamoto/tree-sitter-cobol" } }
}
JSON
npm install tree-sitter-cli@0.24.5
npx tree-sitter generate
npx tree-sitter build --wasm -o tree-sitter-cobol.wasm # needs emscripten or Docker
```
The patches are written to be upstreamable — each is independent and comes
with the failing construct documented above.
## Upstreaming
Sent as [yutaro-sakamoto/tree-sitter-cobol#41](https://github.com/yutaro-sakamoto/tree-sitter-cobol/pull/41)
(branch `real-world-cobol-sources` on the colbymchenry fork). If upstream
merges it, the vendored wasm can track upstream releases instead of this
patch. Until then, `git apply tree-sitter-cobol.patch` on upstream commit
`e99dbdc3` reproduces the fork exactly. The PR body as sent:
> **Parse real-world CICS/DB2 and GnuCOBOL sources**
>
> This adds the constructs that block the grammar on production COBOL, found
> while integrating it into a code-indexing tool. Measured on public corpora:
> AWS CardDemo goes from 9/31 to 43/44 clean parses, CobolCraft (free-format
> GnuCOBOL) from 0 to 17/17, NIST COBOL85 unchanged at 373/382, and the
> existing corpus tests keep their single pre-existing failure (`comment`).
>
> - `EXEC ... END-EXEC` blocks as an external-scanner token (`exec_statement`)
> - Fixed-format single-quote string continuation + doubled-quote escapes
> - `COPY ... REPLACING ==pseudo-text==` with multiple pairs
> - `FD`/`LINKAGE SECTION` record descriptions supplied via `COPY`
> - `NOT=`, `CALL ... GIVING` (a misnested `field()` dropped it), `FREE`,
> `ENTRY`, `PROGRAM-ID ... IS RECURSIVE`, empty `DATE-COMPILED.` headers
> - `FUNCTION <name>(refmod)`, `VALUE <constant>`, `PIC X(CONSTANT)`
> - Relational and abbreviated-combined `WHEN` objects, bitwise `B-*` ops,
> `>>` directives-as-comments, COBOL-2002 usages (`BINARY-LONG-LONG`, ...)
> - A `copybook_fragment` entry point so standalone `.cpy` files parse
> - An opt-in wide mode (sentinel in the first line's sequence area) that a
> free-format preprocessor can use to relax the column-72 margin
>
> Happy to split any of these out or adjust naming/style. Each item is
> independent; `tree-sitter test` passes minus the one pre-existing failure.
docs/grammars/tree-sitter-cobol.patch
+513
View File
@@ -0,0 +1,513 @@
diff --git a/grammar.js b/grammar.js
index 6d48b47..e700a18 100644
--- a/grammar.js
+++ b/grammar.js
@@ -19,6 +19,7 @@ module.exports = grammar({
$._LINE_COMMENT,
$.comment_entry,
$._multiline_string,
+ $._exec_block,
],
extras: $ => [
@@ -29,20 +30,30 @@ module.exports = grammar({
$._LINE_COMMENT_ALIAS,
$.copy_statement,
$.comment,
+ $.directive,
],
rules: {
- start: $ => repeat(
- choice(
- $.program_definition,
- //optional($.function_definition) //todo
- )
+ start: $ => optional(choice(
+ repeat1($.program_definition),
+ $.copybook_fragment,
+ //optional($.function_definition) //todo
+ )),
+
+ // A standalone copybook (.cpy): data descriptions or procedure code
+ // without a program header.
+ copybook_fragment: $ => choice(
+ $.record_description_list,
+ $._procedure_division_contenet
),
_LINE_COMMENT_ALIAS: $ => alias($._LINE_COMMENT, $.comment),
comment: $ => /\*>[^\n]*/,
+ // GnuCOBOL / COBOL-2002 compiler directives: >>SOURCE, >>IF, >>DEFINE, ...
+ directive: $ => />>[^\n]*/,
+
program_definition: $ => prec.right(seq(
$.identification_division,
optional($.environment_division),
@@ -60,7 +71,8 @@ module.exports = grammar({
optional(choice(
$.as_literal,
$.is_initial,
- $.is_common)),
+ $.is_common,
+ $.is_recursive)),
'.')),
repeat(choice(
$.author_section,
@@ -90,6 +102,11 @@ module.exports = grammar({
$._COMMON
),
+ is_recursive: $ => seq(
+ optional($._IS),
+ $._RECURSIVE
+ ),
+
author_section: $ => seq(
$._AUTHOR, '.',
field('comment', repeat1($.comment_entry)),
@@ -102,17 +119,17 @@ module.exports = grammar({
date_written_section: $ => seq(
$._DATE_WRITTEN, '.',
- field('comment', repeat1($.comment_entry)),
+ field('comment', repeat($.comment_entry)),
),
date_compiled_section: $ => seq(
$._DATE_COMPILED, '.',
- field('comment', repeat1($.comment_entry)),
+ field('comment', repeat($.comment_entry)),
),
security_section: $ => seq(
$._SECURITY, '.',
- field('comment', repeat1($.comment_entry)),
+ field('comment', repeat($.comment_entry)),
),
function_definition: $ => seq(
@@ -734,7 +751,7 @@ module.exports = grammar({
file_description: $ => seq(
$.file_type,
$.file_description_entry,
- $.record_description_list
+ optional($.record_description_list)
),
file_type: $ => choice(
@@ -870,12 +887,12 @@ module.exports = grammar({
),
record_description_list: $ => seq(
- repeat1(seq($.data_description, repeat1('.')))
+ repeat1(choice(seq($.data_description, repeat1('.')), prec(1, seq($.exec_statement, optional('.')))))
),
working_storage_section: $ => seq(
$._WORKING_STORAGE, $._SECTION, '.',
- repeat(seq($.data_description, repeat1('.')))
+ repeat(choice(seq($.data_description, repeat1('.')), seq($.exec_statement, optional('.'))))
),
data_description: $ => choice(
@@ -1070,7 +1087,7 @@ module.exports = grammar({
$.picture_edit,
),
- picture_x: $ => /([xX](\([0-9]+\))?)+/,
+ picture_x: $ => /([xX](\(([0-9]+|[a-zA-Z][a-zA-Z0-9-]*)\))?)+/,
picture_n: $ => /([nN](\([0-9]+\))?)+/,
@@ -1119,6 +1136,11 @@ module.exports = grammar({
seq($.BINARY_SHORT, $.SIGNED),
seq($.BINARY_SHORT, $.UNSIGNED),
$.BINARY_SHORT,
+ seq($.BINARY_LONG_LONG, $.SIGNED),
+ seq($.BINARY_LONG_LONG, $.UNSIGNED),
+ $.BINARY_LONG_LONG,
+ $.FLOAT_LONG,
+ $.FLOAT_SHORT,
seq($.BINARY_LONG, $.SIGNED),
seq($.BINARY_LONG, $.UNSIGNED),
$.BINARY_LONG,
@@ -1197,7 +1219,7 @@ module.exports = grammar({
)),
value_item: $ => seq(
- $._literal,
+ choice($._literal, $.qualified_word),
optional(seq(
$.THRU,
$._literal
@@ -1227,7 +1249,7 @@ module.exports = grammar({
linkage_section: $ => seq(
$._LINKAGE, $._SECTION, '.',
- $.record_description_list
+ optional($.record_description_list)
),
report_section: $ => /report_section/,
@@ -1357,6 +1379,19 @@ module.exports = grammar({
'.'
),
+ exec_statement: $ => $._exec_block,
+
+ free_statement: $ => prec.right(seq(
+ $._FREE,
+ repeat1($._target_x)
+ )),
+
+ entry_statement: $ => prec.right(seq(
+ $._ENTRY,
+ field('name', $.string),
+ optional(seq($._USING, repeat1($._call_param)))
+ )),
+
end_program: $ => prec(1, seq(
$._END_PROGRAM,
$.program_name,
@@ -1364,6 +1399,9 @@ module.exports = grammar({
)),
_statement: $ => choice(
+ $.exec_statement,
+ $.entry_statement,
+ $.free_statement,
$.accept_statement,
$.add_statement,
$.allocate_statement,
@@ -1465,12 +1503,19 @@ module.exports = grammar({
),
replacing_clause: $ => seq(
+ $._REPLACING,
+ repeat1($.replacing_pair)
+ ),
+
+ replacing_pair: $ => seq(
field('leading_or_trailing', optional(choice($.LEADING, $.TRAILING))),
- field('x', choice($.WORD, $.string)),
- optional($._BY),
- field('by', choice($.WORD, $.string)),
+ field('x', choice($.pseudo_text, $.WORD, $.string)),
+ $._BY,
+ field('by', choice($.pseudo_text, $.WORD, $.string)),
),
+ pseudo_text: $ => /==([^=\n]|=[^=\n])*==/,
+
start_statement: $ => seq(
$._START,
field('file_name', $.WORD),
@@ -1670,9 +1715,9 @@ module.exports = grammar({
repeat1($._call_param)
))),
optional(choice(
- field('returning', seq($._RETURNING, $._identifier),
- field('giving', seq($._GIVING, $._identifier)),
- )))
+ field('returning', seq($._RETURNING, $._identifier)),
+ field('giving', seq($._GIVING, $._identifier))
+ ))
),
_call_param: $ => choice(
@@ -1902,6 +1947,7 @@ module.exports = grammar({
_evaluate_object: $ => choice(
$.expr,
+ prec.right(seq(choice('=', '>', '<', '>=', '<=', $._NOT_EQUAL), $.expr)),
$.ANY,
$.TRUE,
$.FALSE
@@ -1951,6 +1997,10 @@ module.exports = grammar({
expr: $ => prec.left(choice(
seq($.NOT, $.expr),
seq($.expr, choice($.AND, $.OR), $.expr),
+ // Abbreviated combined relation: `X < 0 OR > Y` — the subject of the
+ // second comparison is implied. (The AND_LT/OR_GT combined tokens the
+ // grammar carries for this never win against keyword lexing.)
+ seq($.expr, choice($.AND, $.OR), $._comparator, $._expr_calc),
$._expr_bool,
seq("(", $.expr, ")")
)),
@@ -1965,6 +2015,11 @@ module.exports = grammar({
)),
_expr_calc_binary: $ => choice(
+ prec.left(1, seq($._expr_calc, $.B_AND, $._expr_calc)),
+ prec.left(1, seq($._expr_calc, $.B_OR, $._expr_calc)),
+ prec.left(1, seq($._expr_calc, $.B_XOR, $._expr_calc)),
+ prec.left(1, seq($._expr_calc, $.B_SHIFT_L, $._expr_calc)),
+ prec.left(1, seq($._expr_calc, $.B_SHIFT_R, $._expr_calc)),
prec.left(1, seq($._expr_calc, '+', $._expr_calc)),
prec.left(1, seq($._expr_calc, '-', $._expr_calc)),
prec.left(2, seq($._expr_calc, '**', $._expr_calc)),
@@ -1974,6 +2029,7 @@ module.exports = grammar({
),
_expr_calc_unary: $ => prec(4, choice(
+ seq($.B_NOT, $._expr_calc),
seq('+', $._expr_calc),
seq('-', $._expr_calc),
seq('^', $._expr_calc),
@@ -2753,7 +2809,7 @@ module.exports = grammar({
seq($.TRIM_FUNCTION, '(', $._trim_args, ')', optional($.func_refmod)),
seq($.NUMVALC_FUNC, '(', $._numvalc_args, ')'),
seq($.LOCALE_DT_FUNC, '(', $._locale_dt_args, ')', optional($.func_refmod)),
- seq($.WORD, optional($.func_args)),
+ seq($.WORD, optional($.func_args), optional($.func_refmod)),
))),
func_refmod: $ => choice(
@@ -2868,6 +2924,16 @@ module.exports = grammar({
_BINARY_CHAR: $ => /[bB][iI][nN][aA][rR][yY]-[cC][hH][aA][rR]/,
_BINARY_DOUBLE: $ => /[bB][iI][nN][aA][rR][yY]-[dD][oO][uU][bB][lL][eE]/,
_BINARY_LONG: $ => /[bB][iI][nN][aA][rR][yY]-[lL][oO][nN][gG]/,
+ _BINARY_LONG_LONG: $ => /[bB][iI][nN][aA][rR][yY]-[lL][oO][nN][gG]-[lL][oO][nN][gG]/,
+ _FREE: $ => /[fF][rR][eE][eE]/,
+ B_AND: $ => /[bB]-[aA][nN][dD]/,
+ B_OR: $ => /[bB]-[oO][rR]/,
+ B_XOR: $ => /[bB]-[xX][oO][rR]/,
+ B_NOT: $ => /[bB]-[nN][oO][tT]/,
+ B_SHIFT_L: $ => /[bB]-[sS][hH][iI][fF][tT]-[lL][cC]?/,
+ B_SHIFT_R: $ => /[bB]-[sS][hH][iI][fF][tT]-[rR][cC]?/,
+ _FLOAT_LONG: $ => /[fF][lL][oO][aA][tT]-[lL][oO][nN][gG]/,
+ _FLOAT_SHORT: $ => /[fF][lL][oO][aA][tT]-[sS][hH][oO][rR][tT]/,
_BINARY_SHORT: $ => /[bB][iI][nN][aA][rR][yY]-[sS][hH][oO][rR][tT]/,
_BLANK: $ => /[bB][lL][aA][nN][kK]/,
_BLANK_LINE: $ => /[bB][lL][aA][nN][kK]-[lL][iI][nN][eE]/,
@@ -3335,6 +3401,9 @@ module.exports = grammar({
BINARY_CHAR: $ => $._BINARY_CHAR,
BINARY_DOUBLE: $ => $._BINARY_DOUBLE,
BINARY_LONG: $ => $._BINARY_LONG,
+ BINARY_LONG_LONG: $ => $._BINARY_LONG_LONG,
+ FLOAT_LONG: $ => $._FLOAT_LONG,
+ FLOAT_SHORT: $ => $._FLOAT_SHORT,
BINARY_SHORT: $ => $._BINARY_SHORT,
//BLANK: $ => $._BLANK,
BLANK_LINE: $ => $._BLANK_LINE,
@@ -3754,7 +3823,7 @@ module.exports = grammar({
COMPUTATIONAL: $ => $._COMPUTATIONAL,
_COMPUTATIONAL: $ => /[cC][oO][mM][pP][uU][tT][aA][tT][iI][oO][nN][aA][lL]/,
- _NOT_EQUAL: $ => /(!=)|([nN][oO][tT][ \t]+(([eE][qQ][uU][aA][lL])|=))/,
+ _NOT_EQUAL: $ => /(!=)|([nN][oO][tT]([ \t]+[eE][qQ][uU][aA][lL]|[ \t]*=))/,
_NOT_LESS: $ => /([nN][oO][tT][ \t]+(<|[lL][eE][sS][sS]))/,
_NOT_GREATER: $ => /([nN][oO][tT][ \t]+(>|[gG][rR][eE][aA][tT][eE][rR]))/,
diff --git a/src/scanner.c b/src/scanner.c
index 6c1ae90..b69cf53 100644
--- a/src/scanner.c
+++ b/src/scanner.c
@@ -1,4 +1,5 @@
#include <tree_sitter/parser.h>
+#include <stdlib.h>
#include <wctype.h>
enum TokenType {
@@ -8,10 +9,21 @@ enum TokenType {
LINE_COMMENT,
COMMENT_ENTRY,
multiline_string,
+ EXEC_BLOCK,
};
+// Wide mode: a preprocessor that converts free-format COBOL to fixed format
+// by left-padding every line 7 columns writes the sentinel "CGWIDE" into the
+// sequence area (columns 1-6) of the FIRST line. Free-format lines routinely
+// run past column 72, so when the sentinel is seen the fixed-format right
+// margin (column 73+ = ignored identification area) is pushed out of reach.
+// The one-byte flag is scanner state, carried through serialize/deserialize.
+#define CG_FIXED_WIDTH 72
+#define CG_WIDE_WIDTH 4096
+static const char CG_WIDE_SENTINEL[6] = {'C', 'G', 'W', 'I', 'D', 'E'};
+
void *tree_sitter_COBOL_external_scanner_create() {
- return NULL;
+ return calloc(1, 1);
}
static bool is_white_space(int c) {
@@ -31,7 +43,7 @@ char* any_content_keyword[] = {
"procedure division",
};
-static bool start_with_word( TSLexer *lexer, char *words[], int number_of_words) {
+static bool start_with_word( TSLexer *lexer, char *words[], int number_of_words, int width) {
while(lexer->lookahead == ' ' || lexer->lookahead == '\t') {
lexer->advance(lexer, true);
}
@@ -45,7 +57,7 @@ static bool start_with_word( TSLexer *lexer, char *words[], int number_of_words)
while(true) {
// At the end of the line
- if(lexer->get_column(lexer) > 71 || lexer->lookahead == '\n' || lexer->lookahead == 0) {
+ if(lexer->get_column(lexer) > width - 1 || lexer->lookahead == '\n' || lexer->lookahead == 0) {
return false;
}
@@ -58,7 +70,7 @@ static bool start_with_word( TSLexer *lexer, char *words[], int number_of_words)
}
if(all_match_failed) {
- for(; lexer->get_column(lexer) < 71 && lexer->lookahead != '\n' && lexer->lookahead != 0;
+ for(; lexer->get_column(lexer) < width - 1 && lexer->lookahead != '\n' && lexer->lookahead != 0;
lexer->advance(lexer, true)) {
}
return false;
@@ -94,6 +106,9 @@ bool tree_sitter_COBOL_external_scanner_scan(void *payload, TSLexer *lexer,
return false;
}
+ char *wide = (char *)payload;
+ const int width = (wide && *wide) ? CG_WIDE_WIDTH : CG_FIXED_WIDTH;
+
if(valid_symbols[WHITE_SPACES]) {
if(is_white_space(lexer->lookahead)) {
while(is_white_space(lexer->lookahead)) {
@@ -106,9 +121,20 @@ bool tree_sitter_COBOL_external_scanner_scan(void *payload, TSLexer *lexer,
}
if(valid_symbols[LINE_PREFIX_COMMENT] && lexer->get_column(lexer) <= 5) {
+ // The sequence area is ignored content — but the free-format
+ // preprocessor plants the CGWIDE sentinel here on the first line.
+ int match = 0;
while(lexer->get_column(lexer) <= 5) {
+ if(match >= 0 && match < 6 && lexer->lookahead == CG_WIDE_SENTINEL[match]) {
+ match++;
+ } else {
+ match = -1;
+ }
lexer->advance(lexer, true);
}
+ if(match == 6 && wide) {
+ *wide = 1;
+ }
lexer->result_symbol = LINE_PREFIX_COMMENT;
lexer->mark_end(lexer);
return true;
@@ -132,7 +158,7 @@ bool tree_sitter_COBOL_external_scanner_scan(void *payload, TSLexer *lexer,
}
if(valid_symbols[LINE_SUFFIX_COMMENT]) {
- if(lexer->get_column(lexer) >= 72) {
+ if(lexer->get_column(lexer) >= width) {
while(lexer->lookahead != '\n' && lexer->lookahead != 0) {
lexer->advance(lexer, true);
}
@@ -143,7 +169,7 @@ bool tree_sitter_COBOL_external_scanner_scan(void *payload, TSLexer *lexer,
}
if(valid_symbols[COMMENT_ENTRY]) {
- if(!start_with_word(lexer, any_content_keyword, number_of_comment_entry_keywords)) {
+ if(!start_with_word(lexer, any_content_keyword, number_of_comment_entry_keywords, width)) {
lexer->mark_end(lexer);
lexer->result_symbol = COMMENT_ENTRY;
return true;
@@ -152,18 +178,66 @@ bool tree_sitter_COBOL_external_scanner_scan(void *payload, TSLexer *lexer,
}
}
+ if(valid_symbols[EXEC_BLOCK]) {
+ // EXEC (CICS|SQL|DLI|...) ... END-EXEC embedded block. Match the word
+ // EXEC (case-insensitive) followed by whitespace, then consume through
+ // the next END-EXEC. On any mismatch return false so the internal
+ // lexer re-reads the same characters as an ordinary WORD.
+ if(lexer->lookahead == 'e' || lexer->lookahead == 'E') {
+ const char *kw = "exec";
+ int ki = 0;
+ while(ki < 4 && (lexer->lookahead == towupper(kw[ki]) || lexer->lookahead == towlower(kw[ki]))) {
+ lexer->advance(lexer, false);
+ ki++;
+ }
+ if(ki == 4 && (lexer->lookahead == ' ' || lexer->lookahead == '\t' ||
+ lexer->lookahead == '\n' || lexer->lookahead == '\r')) {
+ char ring[8] = {0,0,0,0,0,0,0,0};
+ while(lexer->lookahead != 0) {
+ for(int i = 0; i < 7; ++i) ring[i] = ring[i+1];
+ ring[7] = (char)towlower(lexer->lookahead);
+ lexer->advance(lexer, false);
+ if(ring[0]=='e' && ring[1]=='n' && ring[2]=='d' && ring[3]=='-' &&
+ ring[4]=='e' && ring[5]=='x' && ring[6]=='e' && ring[7]=='c') {
+ lexer->result_symbol = EXEC_BLOCK;
+ lexer->mark_end(lexer);
+ return true;
+ }
+ }
+ }
+ return false;
+ }
+ }
+
if(valid_symbols[multiline_string]) {
+ int quote = lexer->lookahead;
+ if(quote != '"' && quote != '\'') {
+ return false;
+ }
while(true) {
- if(lexer->lookahead != '"') {
+ if(lexer->lookahead != quote) {
return false;
}
lexer->advance(lexer, false);
- while(lexer->lookahead != '"' && lexer->lookahead != 0 && lexer->get_column(lexer) < 72) {
+ bool closed = false;
+ while(true) {
+ while(lexer->lookahead != quote && lexer->lookahead != 0 && lexer->get_column(lexer) < width) {
+ lexer->advance(lexer, false);
+ }
+ if(lexer->lookahead != quote) {
+ break;
+ }
lexer->advance(lexer, false);
+ if(lexer->lookahead == quote) {
+ // doubled quote = escaped quote inside the literal
+ lexer->advance(lexer, false);
+ continue;
+ }
+ closed = true;
+ break;
}
- if(lexer->lookahead == '"') {
+ if(closed) {
lexer->result_symbol = multiline_string;
- lexer->advance(lexer, false);
lexer->mark_end(lexer);
return true;
}
@@ -187,7 +261,7 @@ bool tree_sitter_COBOL_external_scanner_scan(void *payload, TSLexer *lexer,
}
lexer->advance(lexer, true);
- while(lexer->lookahead == ' ' && lexer->get_column(lexer) < 72) {
+ while(lexer->lookahead == ' ' && lexer->get_column(lexer) < width) {
lexer->advance(lexer, true);
}
}
@@ -197,11 +271,19 @@ bool tree_sitter_COBOL_external_scanner_scan(void *payload, TSLexer *lexer,
}
unsigned tree_sitter_COBOL_external_scanner_serialize(void *payload, char *buffer) {
+ if(payload && buffer) {
+ buffer[0] = *(char *)payload;
+ return 1;
+ }
return 0;
}
void tree_sitter_COBOL_external_scanner_deserialize(void *payload, const char *buffer, unsigned length) {
+ if(payload) {
+ *(char *)payload = (buffer && length >= 1) ? buffer[0] : 0;
+ }
}
void tree_sitter_COBOL_external_scanner_destroy(void *payload) {
+ free(payload);
}
docs/grammars/tree-sitter-vbnet.md
+171
View File
@@ -0,0 +1,171 @@
# tree-sitter-vbnet.wasm — provenance & rebuild
`src/extraction/wasm/tree-sitter-vbnet.wasm` is built from
[govindbanura/tree-sitter-vbnet](https://github.com/govindbanura/tree-sitter-vbnet)
(MIT) at commit `538b7087bf80e86004531b392fe1186379c0a2b5` with the patch in
`tree-sitter-vbnet.patch` applied. The patch carries two files: `grammar.js`
(edits) and `src/scanner.c` (a new external scanner; upstream has none). The
upstream repo checks in no generated `src/`, so everything else is produced by
`tree-sitter generate`.
Alternatives considered: `CodeAnt-AI/tree-sitter-vb-dotnet` (22★) has **no
license file** and its git history stopped in July 2025 — unusable for
vendoring; `gabriel-gubert/tree-sitter-vbnet` is a 470-line VBScript-flavored
toy. The Roslyn-based approach (PR #627) was withdrawn by its author in favor
of tree-sitter — a Roslyn sidecar would add a .NET runtime dependency to a
local-first npm tool.
## What the patch adds
Upstream parses textbook VB.NET but fails on the constructs that dominate real
codebases (measured: 3–18% of files parsed clean across PolicyPlus, CompactGUI,
and staxrip before patching). Each item below was found by parse-error census
on those repos plus SCrawler and PCL:
1. **Generic type arguments in dotted names** — `System.Collections.Generic.
Dictionary(Of K, V)`, `Implements IRepository(Of Invoice)`, and
method-level `Implements I(Of T).Member` (generic segments were only
accepted unqualified). Open generic types (`GetType(LoaderTask(Of ,))`)
parse too.
2. **Interpolated strings** `$"… {expr[,align][:fmt]} …"` with `""`/`{{`/`}}`
escapes — including multi-line bodies and content pieces that begin with an
apostrophe: the pieces carry lexical precedence 101 (above `comment`'s 100)
because the comment **extra** otherwise fires *inside* the string rule and
eats the rest of the line, closing quote included.
3. **Date/time literals** `#1/15/2020#` — previously lexed as a preprocessor
directive that swallowed to end-of-line. Directives are now constrained to
`#` + letter (`#If`, `#Region`, …), which real directives always satisfy.
4. **VB 14 multi-line string literals** (a `"…"` literal may span lines since
VS 2015) and single-token `string_literal`/`character_literal` (`"["c`) —
the old multi-token form let extras interleave mid-string.
5. **Numeric literal forms** — hex/octal/binary (`&HFF`, `&O777`, `&B1010`),
digit separators (`1_000`), type characters (`6.0!`, `50.0#`, `1.5@`,
`123&`, `7%`), and lowercase `f/r/d` suffixes. WinForms `.Designer.vb`
files are full of `6.0!`.
6. **Identifier type characters and Unicode identifiers** — `Dim i% = 0`,
`Dim r$ = …` (classic VB style, pervasive in SCrawler) and full Unicode
identifiers (`CrashReason.Java虚拟机参数有误` — PCL is written in Chinese).
The identifier token is now `[\p{L}\p{Nl}_][\p{L}\p{Nl}\p{Nd}\p{Mn}\p{Pc}]*
[%$&!#@]?` with the `u` regex flag. **The `u` flag requires
tree-sitter-cli ≥ 0.25** — 0.24.x silently drops the `\p{…}` classes.
7. **`As New T(args)` initializer clauses** — `as_clause` embeds a full
`object_creation_expression` for the `As New` form, so `Dim x As New
StringBuilder` / `Property P As New List(Of String)` produce instantiation
nodes. `Dim x? = expr` nullable declarators parse as well.
8. **Statement separators and single-line forms** — `:` as a statement
terminator and block opener (`Class X : Inherits Y`, `Case 1 : Return "X"`),
single-line `If … Then stmt Else stmt` (via terminator-less inline statement
variants, aliased to the normal statement node names), inline `RaiseEvent`,
and optional `Then` on block `If` and `ElseIf` (legal VB, used in staxrip).
9. **Multi-line lambdas** — `Sub(…) … End Sub` / `Function(…) … End Function`
bodies (upstream had a statement-block body with no `End` closer, so every
block lambda broke its surrounding argument list), `Async`/`Iterator`
lambda modifiers, `ByVal`/`ByRef` lambda parameters, and single-line
`Sub() If cond Then …` statement bodies.
10. **Member declarations** — `Declare [Auto|Ansi|Unicode] Sub/Function … Lib
"dll" [Alias "…"]` P/Invoke declarations, `Custom Event … AddHandler/
RemoveHandler/RaiseEvent … End Event`, stacked attribute lines above one
member, property `= initializer` before `Implements`, type-less
auto-properties, and **`MustOverride` body-less methods and properties**:
`MustOverride` lexes as a dedicated token (removed from the
`member_modifier` alternation) that only `abstract_method_declaration` /
`abstract_property_declaration` accept, making the body-less parse
deterministic. (A GLR body-less alternative on `method_declaration` was
tried first and measurably poisoned error recovery — 100%→60% clean on
PolicyPlus — before being replaced with the token split.)
11. **Expressions** — VB 15 tuple literals `(a, b)`, array literals
`{1, 2, 3}` (plus nested `{{k, v}, …}` dictionary groups, replacing the
ambiguous upstream `dictionary_initializer`), omitted argument slots
(`f(a,, b)` — Optional parameters passed positionally), `TypeOf x IsNot T`,
generic method calls without parens (`items.OfType(Of Panel)`),
null-conditional indexing `x?(0)`, and `Global.`-qualified type names.
12. **LINQ queries** — query expressions no longer require a trailing
`Select`/`Group` clause, `Aggregate`-led queries, and
`Distinct`/`Skip`/`Take` clauses.
### External scanner (`src/scanner.c`, new)
Two constructs are not LR(1)-parseable with tree-sitter's newline-as-extra
treatment; both get external tokens:
- **`QUERY_CLAUSE_CONTINUATION`** — multi-line LINQ (`From x In xs` ↵
`Where …`). At a clause boundary the newline alone cannot distinguish
"query continues on the next line" from "statement ends here". The scanner
looks past the newline run at the next word and emits the continuation
token only when it is a query-clause keyword (with a `Select Case`
guard), so the decision is made by the lexer instead of the LR table.
- **`XML_LITERAL`** — whole VB XML literals (`<Tags><Tag/></Tags>`) consumed
as one opaque token: element nesting, attributes, comments, CDATA,
processing instructions, and **nested** `<%= … %>` embedded expressions
(the staxrip `WriteTagfile` shape). Valid only where a literal can begin an
expression, so a relational `<` (which always *follows* an expression)
never collides. The scanner never skips a leading newline (it must remain
available as a statement terminator).
The scanner is stateless (serialize/deserialize are no-ops).
The `_eof` hack upstream (a literal-`$` token) cannot match a real
end-of-file, so files whose last line has no trailing newline would end with a
MISSING-newline error; the extractor's `preParse` appends a trailing newline
instead of patching that in the grammar.
## Measured parse health (at vendoring time)
| Corpus | Clean parses |
|---|---|
| Fleex255/PolicyPlus (94 `.vb`) | 94/94 (100%) — upstream: 3/94 |
| IridiumIO/CompactGUI (66) | 66/66 (100%) — upstream: 12/66 |
| staxrip/staxrip (145) | 138/145 (95.2%) — upstream: 22/145 |
| AAndyProgram/SCrawler (320) | 279/320 (87.2%) |
| Meloong-Git/PCL (112, Chinese identifiers) | 98/112 (87.5%) |
Known remaining gap (localized ERROR regions, deliberately unpatched):
- **Column-0 GoTo labels** (`Recheck:` at the start of a line inside indented
code — the classic VB label style, used heavily in PCL). The `word:`
keyword-extraction token interacts badly with a newline immediately followed
by a word at column 0, consuming the newline and dropping the previous
statement's terminator. Removing `word:` fixes labels but reintroduces
keyword-prefix identifier bugs corpus-wide (measured: staxrip 95%→28%), so
`word:` stays and column-0 labels keep a localized error; indented labels
parse fine. Worth an upstream tree-sitter investigation eventually.
## Rebuild
```bash
git clone https://github.com/govindbanura/tree-sitter-vbnet
cd tree-sitter-vbnet
git checkout 538b7087bf80e86004531b392fe1186379c0a2b5
git apply path/to/tree-sitter-vbnet.patch # patches grammar.js, adds src/scanner.c
# tree-sitter needs a tree-sitter.json (upstream ships none); grammar name is
# `vbnet` (C symbols tree_sitter_vbnet*):
cat > tree-sitter.json <<'JSON'
{
"grammars": [
{ "name": "vbnet", "camelcase": "Vbnet", "scope": "source.vbnet",
"path": ".", "file-types": ["vb"] }
],
"metadata": { "version": "0.1.0", "license": "MIT",
"description": "VB.NET grammar for tree-sitter",
"links": { "repository": "https://github.com/govindbanura/tree-sitter-vbnet" } }
}
JSON
npm install tree-sitter-cli@0.25.10 # ≥0.25 REQUIRED: the /u regex flag (Unicode
# identifiers) is dropped silently by 0.24.x
npx tree-sitter generate # src/scanner.c from the patch is picked up
npx tree-sitter build --wasm -o tree-sitter-vbnet.wasm # needs emscripten or Docker
```
Upstream's checked-in `test/corpus` expectations predate its own grammar.js
(every corpus test fails at the pinned commit, before any patching), so the
five-repo parse-health sweep above — plus 16 construct repros and the
`__tests__/extraction.test.ts` VB.NET block — is the regression baseline.
## Upstreaming
Not yet sent. The patch is one large, coherent "parse real-world VB.NET"
change; if upstream shows signs of life it can be offered as a PR the same way
the COBOL patch was ([tree-sitter-cobol#41](https://github.com/yutaro-sakamoto/tree-sitter-cobol/pull/41)),
with the corpus numbers above as the motivation. Until then,
`git apply tree-sitter-vbnet.patch` on upstream commit `538b708` reproduces
the vendored grammar exactly.
docs/grammars/tree-sitter-vbnet.patch
+1293
View File
File diff suppressed because it is too large Load Diff