tree-sitter-vbnet.wasm — provenance & rebuild

src/extraction/wasm/tree-sitter-vbnet.wasm is built from govindbanura/tree-sitter-vbnet (MIT) at commit 538b7087bf80e86004531b392fe1186379c0a2b5 with the patch in tree-sitter-vbnet.patch applied. The patch carries two files: grammar.js (edits) and src/scanner.c (a new external scanner; upstream has none). The upstream repo checks in no generated src/, so everything else is produced by tree-sitter generate.

Alternatives considered: CodeAnt-AI/tree-sitter-vb-dotnet (22★) has no license file and its git history stopped in July 2025 — unusable for vendoring; gabriel-gubert/tree-sitter-vbnet is a 470-line VBScript-flavored toy. The Roslyn-based approach (PR #627) was withdrawn by its author in favor of tree-sitter — a Roslyn sidecar would add a .NET runtime dependency to a local-first npm tool.

What the patch adds

Upstream parses textbook VB.NET but fails on the constructs that dominate real codebases (measured: 3–18% of files parsed clean across PolicyPlus, CompactGUI, and staxrip before patching). Each item below was found by parse-error census on those repos plus SCrawler and PCL:

Generic type arguments in dotted names — System.Collections.Generic. Dictionary(Of K, V), Implements IRepository(Of Invoice), and method-level Implements I(Of T).Member (generic segments were only accepted unqualified). Open generic types (GetType(LoaderTask(Of ,))) parse too.
Interpolated strings $"… {expr[,align][:fmt]} …" with ""/{{/}} escapes — including multi-line bodies and content pieces that begin with an apostrophe: the pieces carry lexical precedence 101 (above comment's 100) because the comment extra otherwise fires inside the string rule and eats the rest of the line, closing quote included.
Date/time literals #1/15/2020# — previously lexed as a preprocessor directive that swallowed to end-of-line. Directives are now constrained to # + letter (#If, #Region, …), which real directives always satisfy.
VB 14 multi-line string literals (a "…" literal may span lines since VS 2015) and single-token string_literal/character_literal ("["c) — the old multi-token form let extras interleave mid-string.
Numeric literal forms — hex/octal/binary (&HFF, &O777, &B1010), digit separators (1_000), type characters (6.0!, 50.0#, 1.5@, 123&, 7%), and lowercase f/r/d suffixes. WinForms .Designer.vb files are full of 6.0!.
Identifier type characters and Unicode identifiers — Dim i% = 0, Dim r$ = … (classic VB style, pervasive in SCrawler) and full Unicode identifiers (CrashReason.Java虚拟机参数有误 — PCL is written in Chinese). The identifier token is now [\p{L}\p{Nl}_][\p{L}\p{Nl}\p{Nd}\p{Mn}\p{Pc}]* [%$&!#@]? with the u regex flag. The u flag requires tree-sitter-cli ≥ 0.25 — 0.24.x silently drops the \p{…} classes.
As New T(args) initializer clauses — as_clause embeds a full object_creation_expression for the As New form, so Dim x As New StringBuilder / Property P As New List(Of String) produce instantiation nodes. Dim x? = expr nullable declarators parse as well.
Statement separators and single-line forms — : as a statement terminator and block opener (Class X : Inherits Y, Case 1 : Return "X"), single-line If … Then stmt Else stmt (via terminator-less inline statement variants, aliased to the normal statement node names), inline RaiseEvent, and optional Then on block If and ElseIf (legal VB, used in staxrip).
Multi-line lambdas — Sub(…) … End Sub / Function(…) … End Function bodies (upstream had a statement-block body with no End closer, so every block lambda broke its surrounding argument list), Async/Iterator lambda modifiers, ByVal/ByRef lambda parameters, and single-line Sub() If cond Then … statement bodies.
Member declarations — Declare [Auto|Ansi|Unicode] Sub/Function … Lib "dll" [Alias "…"] P/Invoke declarations, Custom Event … AddHandler/ RemoveHandler/RaiseEvent … End Event, stacked attribute lines above one member, property = initializer before Implements, type-less auto-properties, and MustOverride body-less methods and properties: MustOverride lexes as a dedicated token (removed from the member_modifier alternation) that only abstract_method_declaration / abstract_property_declaration accept, making the body-less parse deterministic. (A GLR body-less alternative on method_declaration was tried first and measurably poisoned error recovery — 100%→60% clean on PolicyPlus — before being replaced with the token split.)
Expressions — VB 15 tuple literals (a, b), array literals {1, 2, 3} (plus nested {{k, v}, …} dictionary groups, replacing the ambiguous upstream dictionary_initializer), omitted argument slots (f(a,, b) — Optional parameters passed positionally), TypeOf x IsNot T, generic method calls without parens (items.OfType(Of Panel)), null-conditional indexing x?(0), and Global.-qualified type names.
LINQ queries — query expressions no longer require a trailing Select/Group clause, Aggregate-led queries, and Distinct/Skip/Take clauses.

External scanner (`src/scanner.c`, new)

Two constructs are not LR(1)-parseable with tree-sitter's newline-as-extra treatment; both get external tokens:

QUERY_CLAUSE_CONTINUATION — multi-line LINQ (From x In xs ↵ Where …). At a clause boundary the newline alone cannot distinguish "query continues on the next line" from "statement ends here". The scanner looks past the newline run at the next word and emits the continuation token only when it is a query-clause keyword (with a Select Case guard), so the decision is made by the lexer instead of the LR table.
XML_LITERAL — whole VB XML literals (<Tags><Tag/></Tags>) consumed as one opaque token: element nesting, attributes, comments, CDATA, processing instructions, and nested <%= … %> embedded expressions (the staxrip WriteTagfile shape). Valid only where a literal can begin an expression, so a relational < (which always follows an expression) never collides. The scanner never skips a leading newline (it must remain available as a statement terminator).

The scanner is stateless (serialize/deserialize are no-ops).

The _eof hack upstream (a literal-$ token) cannot match a real end-of-file, so files whose last line has no trailing newline would end with a MISSING-newline error; the extractor's preParse appends a trailing newline instead of patching that in the grammar.

Measured parse health (at vendoring time)

Corpus	Clean parses
Fleex255/PolicyPlus (94 `.vb`)	94/94 (100%) — upstream: 3/94
IridiumIO/CompactGUI (66)	66/66 (100%) — upstream: 12/66
staxrip/staxrip (145)	138/145 (95.2%) — upstream: 22/145
AAndyProgram/SCrawler (320)	279/320 (87.2%)
Meloong-Git/PCL (112, Chinese identifiers)	98/112 (87.5%)

Known remaining gap (localized ERROR regions, deliberately unpatched):

Column-0 GoTo labels (Recheck: at the start of a line inside indented code — the classic VB label style, used heavily in PCL). The word: keyword-extraction token interacts badly with a newline immediately followed by a word at column 0, consuming the newline and dropping the previous statement's terminator. Removing word: fixes labels but reintroduces keyword-prefix identifier bugs corpus-wide (measured: staxrip 95%→28%), so word: stays and column-0 labels keep a localized error; indented labels parse fine. Worth an upstream tree-sitter investigation eventually.

Rebuild

git clone https://github.com/govindbanura/tree-sitter-vbnet
cd tree-sitter-vbnet
git checkout 538b7087bf80e86004531b392fe1186379c0a2b5
git apply path/to/tree-sitter-vbnet.patch   # patches grammar.js, adds src/scanner.c
# tree-sitter needs a tree-sitter.json (upstream ships none); grammar name is
# `vbnet` (C symbols tree_sitter_vbnet*):
cat > tree-sitter.json <<'JSON'
{
  "grammars": [
    { "name": "vbnet", "camelcase": "Vbnet", "scope": "source.vbnet",
      "path": ".", "file-types": ["vb"] }
  ],
  "metadata": { "version": "0.1.0", "license": "MIT",
    "description": "VB.NET grammar for tree-sitter",
    "links": { "repository": "https://github.com/govindbanura/tree-sitter-vbnet" } }
}
JSON
npm install tree-sitter-cli@0.25.10   # ≥0.25 REQUIRED: the /u regex flag (Unicode
                                      # identifiers) is dropped silently by 0.24.x
npx tree-sitter generate              # src/scanner.c from the patch is picked up
npx tree-sitter build --wasm -o tree-sitter-vbnet.wasm   # needs emscripten or Docker

Upstream's checked-in test/corpus expectations predate its own grammar.js (every corpus test fails at the pinned commit, before any patching), so the five-repo parse-health sweep above — plus 16 construct repros and the __tests__/extraction.test.ts VB.NET block — is the regression baseline.

Upstreaming

Not yet sent. The patch is one large, coherent "parse real-world VB.NET" change; if upstream shows signs of life it can be offered as a PR the same way the COBOL patch was (tree-sitter-cobol#41), with the corpus numbers above as the motivation. Until then, git apply tree-sitter-vbnet.patch on upstream commit 538b708 reproduces the vendored grammar exactly.

9.6 KiB Raw Permalink Blame History Unescape Escape