Files
ggml-org--llama.cpp/tests/peg-parser/simple-tokenize.h
T
wehub-resource-sync 09a3d3ab17
Copilot Setup Steps / copilot-setup-steps (push) Failing after 2s
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
chore: import upstream snapshot with attribution
2026-07-13 11:57:56 +08:00

7 lines
114 B
C++

#pragma once
#include <string>
#include <vector>
std::vector<std::string> simple_tokenize(const std::string &);