wehub-resource-sync
65 lines
2.2 KiB
Markdown
65 lines
2.2 KiB
Markdown
# DeepSeek V4 Flash Test Vectors
|
|
|
|
These vectors were captured from the official DeepSeek V4 Flash API using
|
|
`deepseek-v4-flash`, greedy decoding, thinking disabled, and
|
|
`top_logprobs=20`. The hosted API does not expose full logits, so these files
|
|
store the best logprob slice the API provides.
|
|
|
|
Files:
|
|
|
|
- `prompts/*.txt`: exact user prompts.
|
|
- `official/*.official.json`: official API continuations and top-logprobs.
|
|
- `official.vec`: compact C-test fixture generated from the official JSON.
|
|
- `local-golden.vec`: local top-k/logit fixture captured from a known-sane DS4
|
|
Flash run. It is used to catch substantial backend drift that can keep the
|
|
same greedy token while damaging the logits distribution.
|
|
|
|
Regenerate official vectors:
|
|
|
|
```sh
|
|
DEEPSEEK_API_KEY=... ./tests/test-vectors/fetch_official_vectors.py
|
|
```
|
|
|
|
Running the fetcher without `--only` also regenerates `official.vec`.
|
|
|
|
The C runner consumes `official.vec` directly:
|
|
|
|
```sh
|
|
./ds4_test --logprob-vectors
|
|
```
|
|
|
|
It also consumes the local golden fixture:
|
|
|
|
```sh
|
|
./ds4_test --local-golden-vectors
|
|
```
|
|
|
|
The Metal SSD-streaming cache-pressure repro for issue #384 is a focused
|
|
variant of the official-vector check. It forces a 16GiB routed-expert cache and
|
|
runs only the `short_code_completion` case that exposes wrong logits when
|
|
layer-batched decode reuses expert-cache buffers before the command buffer has
|
|
completed:
|
|
|
|
```sh
|
|
DS4_TEST_MODEL=gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf \
|
|
./ds4_test --metal-ssd-streaming-cache-pressure
|
|
```
|
|
|
|
The runner opens the normal non-quality path with accelerator-specific fast
|
|
routes disabled and pins `DS4_METAL_PREFILL_CHUNK=2048` for this strict
|
|
official-vector check.
|
|
|
|
`official.vec` is intentionally trivial to parse from C: each case points to a
|
|
prompt file and each expected token is hex-encoded by bytes. The official JSON
|
|
files remain in the tree so the compact fixture can be audited against the raw
|
|
API response.
|
|
|
|
To inspect a local top-logprob dump manually:
|
|
|
|
```sh
|
|
./ds4 --metal --nothink -sys "" --temp 0 -n 4 --ctx 16384 \
|
|
--prompt-file tests/test-vectors/prompts/long_code_audit.txt \
|
|
--dump-logprobs /tmp/long_code_audit.ds4.json \
|
|
--logprobs-top-k 20
|
|
```
|