hankcs--hanlp/docs/api/hanlp/pretrained/tok.md

jupytext, kernelspec

jupytext	kernelspec
formats text_representation ipynb,md:myst extension format_name format_version jupytext_version .md myst 0.8 1.4.2	display_name language name Python 3 python python3

tok

Tokenization is a way of separating a sentence into smaller units called tokens. In lexical analysis, tokens usually refer to words.

```{hint}
Tokenize multiple sentences at once for faster speed! 
```

```{seealso}
See [this tutorial](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_demo/hanlp_demo/zh/demo_custom_dict.py) for custom dictionary.
```

To tokenize raw sentences:

:tags: [output_scroll]
import hanlp

tok = hanlp.load(hanlp.pretrained.tok.COARSE_ELECTRA_SMALL_ZH)
tok(['商品和服务。', '晓美焰来到北京立方庭参观自然语义科技公司'])

All the pre-trained tokenizers and their details are listed below.

.. automodule:: hanlp.pretrained.tok
    :members: