antirez--ds4/tests/test-vectors/README.md

DeepSeek V4 Flash Test Vectors

These vectors were captured from the official DeepSeek V4 Flash API using deepseek-v4-flash, greedy decoding, thinking disabled, and top_logprobs=20. The hosted API does not expose full logits, so these files store the best logprob slice the API provides.

Files:

• prompts/*.txt: exact user prompts.
• official/*.official.json: official API continuations and top-logprobs.
• official.vec: compact C-test fixture generated from the official JSON.
• local-golden.vec: local top-k/logit fixture captured from a known-sane DS4 Flash run. It is used to catch substantial backend drift that can keep the same greedy token while damaging the logits distribution.

Regenerate official vectors:

DEEPSEEK_API_KEY=... ./tests/test-vectors/fetch_official_vectors.py

Running the fetcher without --only also regenerates official.vec.

The C runner consumes official.vec directly:

./ds4_test --logprob-vectors

It also consumes the local golden fixture:

./ds4_test --local-golden-vectors

The Metal SSD-streaming cache-pressure repro for issue #384 is a focused variant of the official-vector check. It forces a 16GiB routed-expert cache and runs only the short_code_completion case that exposes wrong logits when layer-batched decode reuses expert-cache buffers before the command buffer has completed:

DS4_TEST_MODEL=gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf \
  ./ds4_test --metal-ssd-streaming-cache-pressure

The runner opens the normal non-quality path with accelerator-specific fast routes disabled and pins DS4_METAL_PREFILL_CHUNK=2048 for this strict official-vector check.

official.vec is intentionally trivial to parse from C: each case points to a prompt file and each expected token is hex-encoded by bytes. The official JSON files remain in the tree so the compact fixture can be audited against the raw API response.

To inspect a local top-logprob dump manually:

./ds4 --metal --nothink -sys "" --temp 0 -n 4 --ctx 16384 \
  --prompt-file tests/test-vectors/prompts/long_code_audit.txt \
  --dump-logprobs /tmp/long_code_audit.ds4.json \
  --logprobs-top-k 20