chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,171 @@
|
||||
# tree-sitter-vbnet.wasm — provenance & rebuild
|
||||
|
||||
`src/extraction/wasm/tree-sitter-vbnet.wasm` is built from
|
||||
[govindbanura/tree-sitter-vbnet](https://github.com/govindbanura/tree-sitter-vbnet)
|
||||
(MIT) at commit `538b7087bf80e86004531b392fe1186379c0a2b5` with the patch in
|
||||
`tree-sitter-vbnet.patch` applied. The patch carries two files: `grammar.js`
|
||||
(edits) and `src/scanner.c` (a new external scanner; upstream has none). The
|
||||
upstream repo checks in no generated `src/`, so everything else is produced by
|
||||
`tree-sitter generate`.
|
||||
|
||||
Alternatives considered: `CodeAnt-AI/tree-sitter-vb-dotnet` (22★) has **no
|
||||
license file** and its git history stopped in July 2025 — unusable for
|
||||
vendoring; `gabriel-gubert/tree-sitter-vbnet` is a 470-line VBScript-flavored
|
||||
toy. The Roslyn-based approach (PR #627) was withdrawn by its author in favor
|
||||
of tree-sitter — a Roslyn sidecar would add a .NET runtime dependency to a
|
||||
local-first npm tool.
|
||||
|
||||
## What the patch adds
|
||||
|
||||
Upstream parses textbook VB.NET but fails on the constructs that dominate real
|
||||
codebases (measured: 3–18% of files parsed clean across PolicyPlus, CompactGUI,
|
||||
and staxrip before patching). Each item below was found by parse-error census
|
||||
on those repos plus SCrawler and PCL:
|
||||
|
||||
1. **Generic type arguments in dotted names** — `System.Collections.Generic.
|
||||
Dictionary(Of K, V)`, `Implements IRepository(Of Invoice)`, and
|
||||
method-level `Implements I(Of T).Member` (generic segments were only
|
||||
accepted unqualified). Open generic types (`GetType(LoaderTask(Of ,))`)
|
||||
parse too.
|
||||
2. **Interpolated strings** `$"… {expr[,align][:fmt]} …"` with `""`/`{{`/`}}`
|
||||
escapes — including multi-line bodies and content pieces that begin with an
|
||||
apostrophe: the pieces carry lexical precedence 101 (above `comment`'s 100)
|
||||
because the comment **extra** otherwise fires *inside* the string rule and
|
||||
eats the rest of the line, closing quote included.
|
||||
3. **Date/time literals** `#1/15/2020#` — previously lexed as a preprocessor
|
||||
directive that swallowed to end-of-line. Directives are now constrained to
|
||||
`#` + letter (`#If`, `#Region`, …), which real directives always satisfy.
|
||||
4. **VB 14 multi-line string literals** (a `"…"` literal may span lines since
|
||||
VS 2015) and single-token `string_literal`/`character_literal` (`"["c`) —
|
||||
the old multi-token form let extras interleave mid-string.
|
||||
5. **Numeric literal forms** — hex/octal/binary (`&HFF`, `&O777`, `&B1010`),
|
||||
digit separators (`1_000`), type characters (`6.0!`, `50.0#`, `1.5@`,
|
||||
`123&`, `7%`), and lowercase `f/r/d` suffixes. WinForms `.Designer.vb`
|
||||
files are full of `6.0!`.
|
||||
6. **Identifier type characters and Unicode identifiers** — `Dim i% = 0`,
|
||||
`Dim r$ = …` (classic VB style, pervasive in SCrawler) and full Unicode
|
||||
identifiers (`CrashReason.Java虚拟机参数有误` — PCL is written in Chinese).
|
||||
The identifier token is now `[\p{L}\p{Nl}_][\p{L}\p{Nl}\p{Nd}\p{Mn}\p{Pc}]*
|
||||
[%$&!#@]?` with the `u` regex flag. **The `u` flag requires
|
||||
tree-sitter-cli ≥ 0.25** — 0.24.x silently drops the `\p{…}` classes.
|
||||
7. **`As New T(args)` initializer clauses** — `as_clause` embeds a full
|
||||
`object_creation_expression` for the `As New` form, so `Dim x As New
|
||||
StringBuilder` / `Property P As New List(Of String)` produce instantiation
|
||||
nodes. `Dim x? = expr` nullable declarators parse as well.
|
||||
8. **Statement separators and single-line forms** — `:` as a statement
|
||||
terminator and block opener (`Class X : Inherits Y`, `Case 1 : Return "X"`),
|
||||
single-line `If … Then stmt Else stmt` (via terminator-less inline statement
|
||||
variants, aliased to the normal statement node names), inline `RaiseEvent`,
|
||||
and optional `Then` on block `If` and `ElseIf` (legal VB, used in staxrip).
|
||||
9. **Multi-line lambdas** — `Sub(…) … End Sub` / `Function(…) … End Function`
|
||||
bodies (upstream had a statement-block body with no `End` closer, so every
|
||||
block lambda broke its surrounding argument list), `Async`/`Iterator`
|
||||
lambda modifiers, `ByVal`/`ByRef` lambda parameters, and single-line
|
||||
`Sub() If cond Then …` statement bodies.
|
||||
10. **Member declarations** — `Declare [Auto|Ansi|Unicode] Sub/Function … Lib
|
||||
"dll" [Alias "…"]` P/Invoke declarations, `Custom Event … AddHandler/
|
||||
RemoveHandler/RaiseEvent … End Event`, stacked attribute lines above one
|
||||
member, property `= initializer` before `Implements`, type-less
|
||||
auto-properties, and **`MustOverride` body-less methods and properties**:
|
||||
`MustOverride` lexes as a dedicated token (removed from the
|
||||
`member_modifier` alternation) that only `abstract_method_declaration` /
|
||||
`abstract_property_declaration` accept, making the body-less parse
|
||||
deterministic. (A GLR body-less alternative on `method_declaration` was
|
||||
tried first and measurably poisoned error recovery — 100%→60% clean on
|
||||
PolicyPlus — before being replaced with the token split.)
|
||||
11. **Expressions** — VB 15 tuple literals `(a, b)`, array literals
|
||||
`{1, 2, 3}` (plus nested `{{k, v}, …}` dictionary groups, replacing the
|
||||
ambiguous upstream `dictionary_initializer`), omitted argument slots
|
||||
(`f(a,, b)` — Optional parameters passed positionally), `TypeOf x IsNot T`,
|
||||
generic method calls without parens (`items.OfType(Of Panel)`),
|
||||
null-conditional indexing `x?(0)`, and `Global.`-qualified type names.
|
||||
12. **LINQ queries** — query expressions no longer require a trailing
|
||||
`Select`/`Group` clause, `Aggregate`-led queries, and
|
||||
`Distinct`/`Skip`/`Take` clauses.
|
||||
|
||||
### External scanner (`src/scanner.c`, new)
|
||||
|
||||
Two constructs are not LR(1)-parseable with tree-sitter's newline-as-extra
|
||||
treatment; both get external tokens:
|
||||
|
||||
- **`QUERY_CLAUSE_CONTINUATION`** — multi-line LINQ (`From x In xs` ↵
|
||||
`Where …`). At a clause boundary the newline alone cannot distinguish
|
||||
"query continues on the next line" from "statement ends here". The scanner
|
||||
looks past the newline run at the next word and emits the continuation
|
||||
token only when it is a query-clause keyword (with a `Select Case`
|
||||
guard), so the decision is made by the lexer instead of the LR table.
|
||||
- **`XML_LITERAL`** — whole VB XML literals (`<Tags><Tag/></Tags>`) consumed
|
||||
as one opaque token: element nesting, attributes, comments, CDATA,
|
||||
processing instructions, and **nested** `<%= … %>` embedded expressions
|
||||
(the staxrip `WriteTagfile` shape). Valid only where a literal can begin an
|
||||
expression, so a relational `<` (which always *follows* an expression)
|
||||
never collides. The scanner never skips a leading newline (it must remain
|
||||
available as a statement terminator).
|
||||
|
||||
The scanner is stateless (serialize/deserialize are no-ops).
|
||||
|
||||
The `_eof` hack upstream (a literal-`$` token) cannot match a real
|
||||
end-of-file, so files whose last line has no trailing newline would end with a
|
||||
MISSING-newline error; the extractor's `preParse` appends a trailing newline
|
||||
instead of patching that in the grammar.
|
||||
|
||||
## Measured parse health (at vendoring time)
|
||||
|
||||
| Corpus | Clean parses |
|
||||
|---|---|
|
||||
| Fleex255/PolicyPlus (94 `.vb`) | 94/94 (100%) — upstream: 3/94 |
|
||||
| IridiumIO/CompactGUI (66) | 66/66 (100%) — upstream: 12/66 |
|
||||
| staxrip/staxrip (145) | 138/145 (95.2%) — upstream: 22/145 |
|
||||
| AAndyProgram/SCrawler (320) | 279/320 (87.2%) |
|
||||
| Meloong-Git/PCL (112, Chinese identifiers) | 98/112 (87.5%) |
|
||||
|
||||
Known remaining gap (localized ERROR regions, deliberately unpatched):
|
||||
|
||||
- **Column-0 GoTo labels** (`Recheck:` at the start of a line inside indented
|
||||
code — the classic VB label style, used heavily in PCL). The `word:`
|
||||
keyword-extraction token interacts badly with a newline immediately followed
|
||||
by a word at column 0, consuming the newline and dropping the previous
|
||||
statement's terminator. Removing `word:` fixes labels but reintroduces
|
||||
keyword-prefix identifier bugs corpus-wide (measured: staxrip 95%→28%), so
|
||||
`word:` stays and column-0 labels keep a localized error; indented labels
|
||||
parse fine. Worth an upstream tree-sitter investigation eventually.
|
||||
|
||||
## Rebuild
|
||||
|
||||
```bash
|
||||
git clone https://github.com/govindbanura/tree-sitter-vbnet
|
||||
cd tree-sitter-vbnet
|
||||
git checkout 538b7087bf80e86004531b392fe1186379c0a2b5
|
||||
git apply path/to/tree-sitter-vbnet.patch # patches grammar.js, adds src/scanner.c
|
||||
# tree-sitter needs a tree-sitter.json (upstream ships none); grammar name is
|
||||
# `vbnet` (C symbols tree_sitter_vbnet*):
|
||||
cat > tree-sitter.json <<'JSON'
|
||||
{
|
||||
"grammars": [
|
||||
{ "name": "vbnet", "camelcase": "Vbnet", "scope": "source.vbnet",
|
||||
"path": ".", "file-types": ["vb"] }
|
||||
],
|
||||
"metadata": { "version": "0.1.0", "license": "MIT",
|
||||
"description": "VB.NET grammar for tree-sitter",
|
||||
"links": { "repository": "https://github.com/govindbanura/tree-sitter-vbnet" } }
|
||||
}
|
||||
JSON
|
||||
npm install tree-sitter-cli@0.25.10 # ≥0.25 REQUIRED: the /u regex flag (Unicode
|
||||
# identifiers) is dropped silently by 0.24.x
|
||||
npx tree-sitter generate # src/scanner.c from the patch is picked up
|
||||
npx tree-sitter build --wasm -o tree-sitter-vbnet.wasm # needs emscripten or Docker
|
||||
```
|
||||
|
||||
Upstream's checked-in `test/corpus` expectations predate its own grammar.js
|
||||
(every corpus test fails at the pinned commit, before any patching), so the
|
||||
five-repo parse-health sweep above — plus 16 construct repros and the
|
||||
`__tests__/extraction.test.ts` VB.NET block — is the regression baseline.
|
||||
|
||||
## Upstreaming
|
||||
|
||||
Not yet sent. The patch is one large, coherent "parse real-world VB.NET"
|
||||
change; if upstream shows signs of life it can be offered as a PR the same way
|
||||
the COBOL patch was ([tree-sitter-cobol#41](https://github.com/yutaro-sakamoto/tree-sitter-cobol/pull/41)),
|
||||
with the corpus numbers above as the motivation. Until then,
|
||||
`git apply tree-sitter-vbnet.patch` on upstream commit `538b708` reproduces
|
||||
the vendored grammar exactly.
|
||||
Reference in New Issue
Block a user