{ "

Basic english tokenizer

\n

We use character level tokenizer in this experiment. You can switch by setting,

\n_^_0_^_

in the configurations dictionary when starting the experiment.

\n": "

\u57fa\u7840\u82f1\u8bed\u5206\u8bcd\u5668

\n

\u6211\u4eec\u5728\u8fd9\u4e2a\u5b9e\u9a8c\u4e2d\u4f7f\u7528\u89d2\u8272\u7b49\u7ea7\u5206\u8bcd\u5668\u3002\u4f60\u53ef\u4ee5\u901a\u8fc7\u8bbe\u7f6e\u8fdb\u884c\u5207\u6362\uff0c

\n_^_0_^_

\u5f00\u59cb\u5b9e\u9a8c\u65f6\u5728\u914d\u7f6e\u5b57\u5178\u4e2d\u3002

\n", "

Character level tokenizer

\n": "

\u89d2\u8272\u7b49\u7ea7\u5206\u8bcd\u5668

\n", "

\n

Tokenizer Configurations

\n": "

\n

\u5206\u8bcd\u5668\u914d\u7f6e

\n", "

Character level tokenizer configuration

\n": "

\u89d2\u8272\u7ea7\u522b\u5206\u8bcd\u5668\u914d\u7f6e

\n", "tokenizer.py": "tokenizer.py" }