wehub-resource-sync
172 lines
9.6 KiB
Markdown
172 lines
9.6 KiB
Markdown
# tree-sitter-vbnet.wasm — provenance & rebuild
|
||
|
||
`src/extraction/wasm/tree-sitter-vbnet.wasm` is built from
|
||
[govindbanura/tree-sitter-vbnet](https://github.com/govindbanura/tree-sitter-vbnet)
|
||
(MIT) at commit `538b7087bf80e86004531b392fe1186379c0a2b5` with the patch in
|
||
`tree-sitter-vbnet.patch` applied. The patch carries two files: `grammar.js`
|
||
(edits) and `src/scanner.c` (a new external scanner; upstream has none). The
|
||
upstream repo checks in no generated `src/`, so everything else is produced by
|
||
`tree-sitter generate`.
|
||
|
||
Alternatives considered: `CodeAnt-AI/tree-sitter-vb-dotnet` (22★) has **no
|
||
license file** and its git history stopped in July 2025 — unusable for
|
||
vendoring; `gabriel-gubert/tree-sitter-vbnet` is a 470-line VBScript-flavored
|
||
toy. The Roslyn-based approach (PR #627) was withdrawn by its author in favor
|
||
of tree-sitter — a Roslyn sidecar would add a .NET runtime dependency to a
|
||
local-first npm tool.
|
||
|
||
## What the patch adds
|
||
|
||
Upstream parses textbook VB.NET but fails on the constructs that dominate real
|
||
codebases (measured: 3–18% of files parsed clean across PolicyPlus, CompactGUI,
|
||
and staxrip before patching). Each item below was found by parse-error census
|
||
on those repos plus SCrawler and PCL:
|
||
|
||
1. **Generic type arguments in dotted names** — `System.Collections.Generic.
|
||
Dictionary(Of K, V)`, `Implements IRepository(Of Invoice)`, and
|
||
method-level `Implements I(Of T).Member` (generic segments were only
|
||
accepted unqualified). Open generic types (`GetType(LoaderTask(Of ,))`)
|
||
parse too.
|
||
2. **Interpolated strings** `$"… {expr[,align][:fmt]} …"` with `""`/`{{`/`}}`
|
||
escapes — including multi-line bodies and content pieces that begin with an
|
||
apostrophe: the pieces carry lexical precedence 101 (above `comment`'s 100)
|
||
because the comment **extra** otherwise fires *inside* the string rule and
|
||
eats the rest of the line, closing quote included.
|
||
3. **Date/time literals** `#1/15/2020#` — previously lexed as a preprocessor
|
||
directive that swallowed to end-of-line. Directives are now constrained to
|
||
`#` + letter (`#If`, `#Region`, …), which real directives always satisfy.
|
||
4. **VB 14 multi-line string literals** (a `"…"` literal may span lines since
|
||
VS 2015) and single-token `string_literal`/`character_literal` (`"["c`) —
|
||
the old multi-token form let extras interleave mid-string.
|
||
5. **Numeric literal forms** — hex/octal/binary (`&HFF`, `&O777`, `&B1010`),
|
||
digit separators (`1_000`), type characters (`6.0!`, `50.0#`, `1.5@`,
|
||
`123&`, `7%`), and lowercase `f/r/d` suffixes. WinForms `.Designer.vb`
|
||
files are full of `6.0!`.
|
||
6. **Identifier type characters and Unicode identifiers** — `Dim i% = 0`,
|
||
`Dim r$ = …` (classic VB style, pervasive in SCrawler) and full Unicode
|
||
identifiers (`CrashReason.Java虚拟机参数有误` — PCL is written in Chinese).
|
||
The identifier token is now `[\p{L}\p{Nl}_][\p{L}\p{Nl}\p{Nd}\p{Mn}\p{Pc}]*
|
||
[%$&!#@]?` with the `u` regex flag. **The `u` flag requires
|
||
tree-sitter-cli ≥ 0.25** — 0.24.x silently drops the `\p{…}` classes.
|
||
7. **`As New T(args)` initializer clauses** — `as_clause` embeds a full
|
||
`object_creation_expression` for the `As New` form, so `Dim x As New
|
||
StringBuilder` / `Property P As New List(Of String)` produce instantiation
|
||
nodes. `Dim x? = expr` nullable declarators parse as well.
|
||
8. **Statement separators and single-line forms** — `:` as a statement
|
||
terminator and block opener (`Class X : Inherits Y`, `Case 1 : Return "X"`),
|
||
single-line `If … Then stmt Else stmt` (via terminator-less inline statement
|
||
variants, aliased to the normal statement node names), inline `RaiseEvent`,
|
||
and optional `Then` on block `If` and `ElseIf` (legal VB, used in staxrip).
|
||
9. **Multi-line lambdas** — `Sub(…) … End Sub` / `Function(…) … End Function`
|
||
bodies (upstream had a statement-block body with no `End` closer, so every
|
||
block lambda broke its surrounding argument list), `Async`/`Iterator`
|
||
lambda modifiers, `ByVal`/`ByRef` lambda parameters, and single-line
|
||
`Sub() If cond Then …` statement bodies.
|
||
10. **Member declarations** — `Declare [Auto|Ansi|Unicode] Sub/Function … Lib
|
||
"dll" [Alias "…"]` P/Invoke declarations, `Custom Event … AddHandler/
|
||
RemoveHandler/RaiseEvent … End Event`, stacked attribute lines above one
|
||
member, property `= initializer` before `Implements`, type-less
|
||
auto-properties, and **`MustOverride` body-less methods and properties**:
|
||
`MustOverride` lexes as a dedicated token (removed from the
|
||
`member_modifier` alternation) that only `abstract_method_declaration` /
|
||
`abstract_property_declaration` accept, making the body-less parse
|
||
deterministic. (A GLR body-less alternative on `method_declaration` was
|
||
tried first and measurably poisoned error recovery — 100%→60% clean on
|
||
PolicyPlus — before being replaced with the token split.)
|
||
11. **Expressions** — VB 15 tuple literals `(a, b)`, array literals
|
||
`{1, 2, 3}` (plus nested `{{k, v}, …}` dictionary groups, replacing the
|
||
ambiguous upstream `dictionary_initializer`), omitted argument slots
|
||
(`f(a,, b)` — Optional parameters passed positionally), `TypeOf x IsNot T`,
|
||
generic method calls without parens (`items.OfType(Of Panel)`),
|
||
null-conditional indexing `x?(0)`, and `Global.`-qualified type names.
|
||
12. **LINQ queries** — query expressions no longer require a trailing
|
||
`Select`/`Group` clause, `Aggregate`-led queries, and
|
||
`Distinct`/`Skip`/`Take` clauses.
|
||
|
||
### External scanner (`src/scanner.c`, new)
|
||
|
||
Two constructs are not LR(1)-parseable with tree-sitter's newline-as-extra
|
||
treatment; both get external tokens:
|
||
|
||
- **`QUERY_CLAUSE_CONTINUATION`** — multi-line LINQ (`From x In xs` ↵
|
||
`Where …`). At a clause boundary the newline alone cannot distinguish
|
||
"query continues on the next line" from "statement ends here". The scanner
|
||
looks past the newline run at the next word and emits the continuation
|
||
token only when it is a query-clause keyword (with a `Select Case`
|
||
guard), so the decision is made by the lexer instead of the LR table.
|
||
- **`XML_LITERAL`** — whole VB XML literals (`<Tags><Tag/></Tags>`) consumed
|
||
as one opaque token: element nesting, attributes, comments, CDATA,
|
||
processing instructions, and **nested** `<%= … %>` embedded expressions
|
||
(the staxrip `WriteTagfile` shape). Valid only where a literal can begin an
|
||
expression, so a relational `<` (which always *follows* an expression)
|
||
never collides. The scanner never skips a leading newline (it must remain
|
||
available as a statement terminator).
|
||
|
||
The scanner is stateless (serialize/deserialize are no-ops).
|
||
|
||
The `_eof` hack upstream (a literal-`$` token) cannot match a real
|
||
end-of-file, so files whose last line has no trailing newline would end with a
|
||
MISSING-newline error; the extractor's `preParse` appends a trailing newline
|
||
instead of patching that in the grammar.
|
||
|
||
## Measured parse health (at vendoring time)
|
||
|
||
| Corpus | Clean parses |
|
||
|---|---|
|
||
| Fleex255/PolicyPlus (94 `.vb`) | 94/94 (100%) — upstream: 3/94 |
|
||
| IridiumIO/CompactGUI (66) | 66/66 (100%) — upstream: 12/66 |
|
||
| staxrip/staxrip (145) | 138/145 (95.2%) — upstream: 22/145 |
|
||
| AAndyProgram/SCrawler (320) | 279/320 (87.2%) |
|
||
| Meloong-Git/PCL (112, Chinese identifiers) | 98/112 (87.5%) |
|
||
|
||
Known remaining gap (localized ERROR regions, deliberately unpatched):
|
||
|
||
- **Column-0 GoTo labels** (`Recheck:` at the start of a line inside indented
|
||
code — the classic VB label style, used heavily in PCL). The `word:`
|
||
keyword-extraction token interacts badly with a newline immediately followed
|
||
by a word at column 0, consuming the newline and dropping the previous
|
||
statement's terminator. Removing `word:` fixes labels but reintroduces
|
||
keyword-prefix identifier bugs corpus-wide (measured: staxrip 95%→28%), so
|
||
`word:` stays and column-0 labels keep a localized error; indented labels
|
||
parse fine. Worth an upstream tree-sitter investigation eventually.
|
||
|
||
## Rebuild
|
||
|
||
```bash
|
||
git clone https://github.com/govindbanura/tree-sitter-vbnet
|
||
cd tree-sitter-vbnet
|
||
git checkout 538b7087bf80e86004531b392fe1186379c0a2b5
|
||
git apply path/to/tree-sitter-vbnet.patch # patches grammar.js, adds src/scanner.c
|
||
# tree-sitter needs a tree-sitter.json (upstream ships none); grammar name is
|
||
# `vbnet` (C symbols tree_sitter_vbnet*):
|
||
cat > tree-sitter.json <<'JSON'
|
||
{
|
||
"grammars": [
|
||
{ "name": "vbnet", "camelcase": "Vbnet", "scope": "source.vbnet",
|
||
"path": ".", "file-types": ["vb"] }
|
||
],
|
||
"metadata": { "version": "0.1.0", "license": "MIT",
|
||
"description": "VB.NET grammar for tree-sitter",
|
||
"links": { "repository": "https://github.com/govindbanura/tree-sitter-vbnet" } }
|
||
}
|
||
JSON
|
||
npm install tree-sitter-cli@0.25.10 # ≥0.25 REQUIRED: the /u regex flag (Unicode
|
||
# identifiers) is dropped silently by 0.24.x
|
||
npx tree-sitter generate # src/scanner.c from the patch is picked up
|
||
npx tree-sitter build --wasm -o tree-sitter-vbnet.wasm # needs emscripten or Docker
|
||
```
|
||
|
||
Upstream's checked-in `test/corpus` expectations predate its own grammar.js
|
||
(every corpus test fails at the pinned commit, before any patching), so the
|
||
five-repo parse-health sweep above — plus 16 construct repros and the
|
||
`__tests__/extraction.test.ts` VB.NET block — is the regression baseline.
|
||
|
||
## Upstreaming
|
||
|
||
Not yet sent. The patch is one large, coherent "parse real-world VB.NET"
|
||
change; if upstream shows signs of life it can be offered as a PR the same way
|
||
the COBOL patch was ([tree-sitter-cobol#41](https://github.com/yutaro-sakamoto/tree-sitter-cobol/pull/41)),
|
||
with the corpus numbers above as the motivation. Until then,
|
||
`git apply tree-sitter-vbnet.patch` on upstream commit `538b708` reproduces
|
||
the vendored grammar exactly.
|