{ "

Basic english tokenizer

\n

We use character level tokenizer in this experiment. You can switch by setting,

\n_^_0_^_

in the configurations dictionary when starting the experiment.

\n": "

\u30d9\u30fc\u30b7\u30c3\u30af\u30fb\u30a4\u30f3\u30b0\u30ea\u30c3\u30b7\u30e5\u30fb\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc

\n

\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001\u30ad\u30e3\u30e9\u30af\u30bf\u30fc\u30ec\u30d9\u30eb\u306e\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u3092\u4f7f\u7528\u3057\u307e\u3059\u3002\u8a2d\u5b9a\u3067\u5207\u308a\u66ff\u3048\u308b\u3053\u3068\u304c\u3067\u304d\u307e\u3059\u304c\u3001

\n_^_0_^_

\u5b9f\u9a13\u3092\u958b\u59cb\u3059\u308b\u3068\u304d\u306b\u69cb\u6210\u8f9e\u66f8\u306b\u3042\u308a\u307e\u3059\u3002

\n", "

Character level tokenizer

\n": "

\u30ad\u30e3\u30e9\u30af\u30bf\u30fc\u30ec\u30d9\u30eb\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc

\n", "

\n

Tokenizer Configurations

\n": "

\n

\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u8a2d\u5b9a

\n", "

Character level tokenizer configuration

\n": "

\u30ad\u30e3\u30e9\u30af\u30bf\u30fc\u30ec\u30d9\u30eb\u306e\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u8a2d\u5b9a

\n", "tokenizer.py": "tokenizer.py" }