wehub-resource-sync
145 lines
4.3 KiB
Markdown
145 lines
4.3 KiB
Markdown
# Contributing
|
|
|
|
DwarfStar4 changes should be tested against the failure mode they can realistically
|
|
affect. The project has two regression tracks: correctness and speed. Please
|
|
include the commands you ran, the machine/backend, the model quant, and any
|
|
notable failures in the PR or commit notes.
|
|
|
|
Do not send PRs affecting one or more inference backends without checking if the
|
|
resulting code is still correct and fast. The only acceptable regression speed
|
|
is when an important correctness bug is fixed and it requires some speed penalty.
|
|
|
|
## Correctness Regression Tests
|
|
|
|
Build the default backend first:
|
|
|
|
```sh
|
|
make clean
|
|
make
|
|
```
|
|
|
|
The C test runner is `ds4_test`. Running it without arguments is equivalent to
|
|
`--all`:
|
|
|
|
```sh
|
|
make test
|
|
```
|
|
|
|
Useful narrower checks:
|
|
|
|
```sh
|
|
./ds4_test --server
|
|
./ds4_test --logprob-vectors
|
|
./ds4_test --long-context
|
|
./ds4_test --tool-call-quality
|
|
./ds4_test --metal-kernels
|
|
```
|
|
|
|
What they cover:
|
|
|
|
- `--server`: request parsing, chat rendering, streaming, tool-call parsing,
|
|
thinking controls, KV disk-cache bookkeeping, and other server-side logic.
|
|
This is the best quick check for API and prompt-rendering changes.
|
|
- `--logprob-vectors`: compares local token bytes and top-logprob slices against
|
|
official DeepSeek V4 Flash continuation vectors. This catches tokenizer,
|
|
template, attention, and logits regressions.
|
|
- `--long-context`: runs a long-context story fact-recall regression from
|
|
`tests/long_context_story_prompt.txt`. The model must retrieve spelled-out
|
|
person-number assignments from a long prose prompt and return `Name=number`
|
|
lines that the test parses.
|
|
- `--tool-call-quality`: exercises actual model behavior for DSML tool-call
|
|
emission in both fast and exact paths.
|
|
- `--metal-kernels`: isolated Metal kernel numeric checks.
|
|
|
|
The runner defaults to `ds4flash.gguf`. Override paths when needed:
|
|
|
|
```sh
|
|
DS4_TEST_MODEL=/path/to/model.gguf ./ds4_test --logprob-vectors
|
|
DS4_TEST_VECTOR_FILE=/path/to/official.vec ./ds4_test --logprob-vectors
|
|
DS4_TEST_LONG_PROMPT=/path/to/prompt.txt ./ds4_test --long-context
|
|
```
|
|
|
|
For CUDA-specific changes, test on a CUDA machine:
|
|
|
|
```sh
|
|
make
|
|
make cuda-regression
|
|
```
|
|
|
|
For CPU portability, at least verify that the CPU target still builds:
|
|
|
|
```sh
|
|
make cpu
|
|
```
|
|
|
|
The CPU backend is a reference/debug path, not the production performance
|
|
target. Remember that executing the CPU path on Metal can crash the system
|
|
because of a kernel bug in macOS.
|
|
|
|
## Quality Checks For Quantization Changes
|
|
|
|
For GGUF or quantization work, use the official-continuation scorer in
|
|
`gguf-tools/quality-testing`. The test compares how much probability a local
|
|
GGUF assigns to official DeepSeek V4 Flash continuations, token by token.
|
|
|
|
Build the scorer:
|
|
|
|
```sh
|
|
make -C gguf-tools quality-score
|
|
```
|
|
|
|
Then score old and new GGUFs against the same manifest and compare:
|
|
|
|
```sh
|
|
gguf-tools/quality-testing/score_official OLD.gguf \
|
|
gguf-tools/quality-testing/data/manifest.tsv /tmp/old.tsv 4096
|
|
|
|
gguf-tools/quality-testing/score_official NEW.gguf \
|
|
gguf-tools/quality-testing/data/manifest.tsv /tmp/new.tsv 4096
|
|
|
|
python3 gguf-tools/quality-testing/compare_scores.py /tmp/old.tsv /tmp/new.tsv
|
|
```
|
|
|
|
Lower `avg_nll` is better. See
|
|
`gguf-tools/quality-testing/README.md` for collecting or refreshing official
|
|
continuations.
|
|
|
|
## Speed Regression Tests
|
|
|
|
Use `ds4-bench` for throughput regressions. It reports instantaneous prefill and
|
|
generation speed at context frontiers, not one whole-run average. Prefill is
|
|
incremental: each row measures only the newly processed suffix since the
|
|
previous frontier.
|
|
|
|
Default linear sweep:
|
|
|
|
```sh
|
|
./ds4-bench \
|
|
-m ds4flash.gguf \
|
|
--prompt-file speed-bench/promessi_sposi.txt \
|
|
--ctx-start 2048 \
|
|
--ctx-max 65536 \
|
|
--step-incr 2048 \
|
|
--gen-tokens 128 \
|
|
--csv /tmp/ds4-speed.csv
|
|
```
|
|
|
|
Use the same machine, backend, model file, context sweep, power/thermal state,
|
|
and background load when comparing two commits. For backend work, run at least
|
|
one before/after CSV and compare both `prefill_tps` and `gen_tps`. Generation is
|
|
greedy and skips EOS so each frontier gets the same number of generated tokens.
|
|
|
|
To generate a graph for a CSV:
|
|
|
|
```sh
|
|
python3 speed-bench/plot_speed.py /tmp/ds4-speed.csv --title "Machine t/s"
|
|
```
|
|
|
|
## Reporting sessions bugs
|
|
|
|
For debugging a failing generation, keep the trace:
|
|
|
|
```sh
|
|
./ds4-server --trace /tmp/ds4-trace.txt ...
|
|
```
|