chore: import upstream snapshot with attribution
This commit is contained in:
+144
@@ -0,0 +1,144 @@
|
||||
# Contributing
|
||||
|
||||
DwarfStar4 changes should be tested against the failure mode they can realistically
|
||||
affect. The project has two regression tracks: correctness and speed. Please
|
||||
include the commands you ran, the machine/backend, the model quant, and any
|
||||
notable failures in the PR or commit notes.
|
||||
|
||||
Do not send PRs affecting one or more inference backends without checking if the
|
||||
resulting code is still correct and fast. The only acceptable regression speed
|
||||
is when an important correctness bug is fixed and it requires some speed penalty.
|
||||
|
||||
## Correctness Regression Tests
|
||||
|
||||
Build the default backend first:
|
||||
|
||||
```sh
|
||||
make clean
|
||||
make
|
||||
```
|
||||
|
||||
The C test runner is `ds4_test`. Running it without arguments is equivalent to
|
||||
`--all`:
|
||||
|
||||
```sh
|
||||
make test
|
||||
```
|
||||
|
||||
Useful narrower checks:
|
||||
|
||||
```sh
|
||||
./ds4_test --server
|
||||
./ds4_test --logprob-vectors
|
||||
./ds4_test --long-context
|
||||
./ds4_test --tool-call-quality
|
||||
./ds4_test --metal-kernels
|
||||
```
|
||||
|
||||
What they cover:
|
||||
|
||||
- `--server`: request parsing, chat rendering, streaming, tool-call parsing,
|
||||
thinking controls, KV disk-cache bookkeeping, and other server-side logic.
|
||||
This is the best quick check for API and prompt-rendering changes.
|
||||
- `--logprob-vectors`: compares local token bytes and top-logprob slices against
|
||||
official DeepSeek V4 Flash continuation vectors. This catches tokenizer,
|
||||
template, attention, and logits regressions.
|
||||
- `--long-context`: runs a long-context story fact-recall regression from
|
||||
`tests/long_context_story_prompt.txt`. The model must retrieve spelled-out
|
||||
person-number assignments from a long prose prompt and return `Name=number`
|
||||
lines that the test parses.
|
||||
- `--tool-call-quality`: exercises actual model behavior for DSML tool-call
|
||||
emission in both fast and exact paths.
|
||||
- `--metal-kernels`: isolated Metal kernel numeric checks.
|
||||
|
||||
The runner defaults to `ds4flash.gguf`. Override paths when needed:
|
||||
|
||||
```sh
|
||||
DS4_TEST_MODEL=/path/to/model.gguf ./ds4_test --logprob-vectors
|
||||
DS4_TEST_VECTOR_FILE=/path/to/official.vec ./ds4_test --logprob-vectors
|
||||
DS4_TEST_LONG_PROMPT=/path/to/prompt.txt ./ds4_test --long-context
|
||||
```
|
||||
|
||||
For CUDA-specific changes, test on a CUDA machine:
|
||||
|
||||
```sh
|
||||
make
|
||||
make cuda-regression
|
||||
```
|
||||
|
||||
For CPU portability, at least verify that the CPU target still builds:
|
||||
|
||||
```sh
|
||||
make cpu
|
||||
```
|
||||
|
||||
The CPU backend is a reference/debug path, not the production performance
|
||||
target. Remember that executing the CPU path on Metal can crash the system
|
||||
because of a kernel bug in macOS.
|
||||
|
||||
## Quality Checks For Quantization Changes
|
||||
|
||||
For GGUF or quantization work, use the official-continuation scorer in
|
||||
`gguf-tools/quality-testing`. The test compares how much probability a local
|
||||
GGUF assigns to official DeepSeek V4 Flash continuations, token by token.
|
||||
|
||||
Build the scorer:
|
||||
|
||||
```sh
|
||||
make -C gguf-tools quality-score
|
||||
```
|
||||
|
||||
Then score old and new GGUFs against the same manifest and compare:
|
||||
|
||||
```sh
|
||||
gguf-tools/quality-testing/score_official OLD.gguf \
|
||||
gguf-tools/quality-testing/data/manifest.tsv /tmp/old.tsv 4096
|
||||
|
||||
gguf-tools/quality-testing/score_official NEW.gguf \
|
||||
gguf-tools/quality-testing/data/manifest.tsv /tmp/new.tsv 4096
|
||||
|
||||
python3 gguf-tools/quality-testing/compare_scores.py /tmp/old.tsv /tmp/new.tsv
|
||||
```
|
||||
|
||||
Lower `avg_nll` is better. See
|
||||
`gguf-tools/quality-testing/README.md` for collecting or refreshing official
|
||||
continuations.
|
||||
|
||||
## Speed Regression Tests
|
||||
|
||||
Use `ds4-bench` for throughput regressions. It reports instantaneous prefill and
|
||||
generation speed at context frontiers, not one whole-run average. Prefill is
|
||||
incremental: each row measures only the newly processed suffix since the
|
||||
previous frontier.
|
||||
|
||||
Default linear sweep:
|
||||
|
||||
```sh
|
||||
./ds4-bench \
|
||||
-m ds4flash.gguf \
|
||||
--prompt-file speed-bench/promessi_sposi.txt \
|
||||
--ctx-start 2048 \
|
||||
--ctx-max 65536 \
|
||||
--step-incr 2048 \
|
||||
--gen-tokens 128 \
|
||||
--csv /tmp/ds4-speed.csv
|
||||
```
|
||||
|
||||
Use the same machine, backend, model file, context sweep, power/thermal state,
|
||||
and background load when comparing two commits. For backend work, run at least
|
||||
one before/after CSV and compare both `prefill_tps` and `gen_tps`. Generation is
|
||||
greedy and skips EOS so each frontier gets the same number of generated tokens.
|
||||
|
||||
To generate a graph for a CSV:
|
||||
|
||||
```sh
|
||||
python3 speed-bench/plot_speed.py /tmp/ds4-speed.csv --title "Machine t/s"
|
||||
```
|
||||
|
||||
## Reporting sessions bugs
|
||||
|
||||
For debugging a failing generation, keep the trace:
|
||||
|
||||
```sh
|
||||
./ds4-server --trace /tmp/ds4-trace.txt ...
|
||||
```
|
||||
Reference in New Issue
Block a user