wehub-resource-sync
134 lines
5.1 KiB
Markdown
134 lines
5.1 KiB
Markdown
---
|
|
jupytext:
|
|
formats: ipynb,md:myst
|
|
text_representation:
|
|
extension: .md
|
|
format_name: myst
|
|
format_version: '0.8'
|
|
jupytext_version: 1.4.2
|
|
kernelspec:
|
|
display_name: Python 3
|
|
language: python
|
|
name: python3
|
|
---
|
|
|
|
# Tutorial
|
|
|
|
Natural Language Processing is an exciting field consisting of many closely related tasks like lexical analysis
|
|
and parsing. Each task involves many datasets and models, all requiring a high degree of expertise.
|
|
Things become even more complex when dealing with multilingual text, as there's simply no datasets for some
|
|
low-resource languages. However, with HanLP 2.1, core NLP tasks have been made easy to access and efficient in
|
|
production environments. In this tutorial, we'll walk through the APIs in HanLP step by step.
|
|
|
|
HanLP offers out-of-the-box RESTful API and native Python API which share very similar interfaces
|
|
while they are designed for different scenes.
|
|
|
|
```{code-cell} ipython3
|
|
:tags: [remove_cell]
|
|
|
|
import hanlp_common.constant
|
|
|
|
hanlp_common.constant.IPYTHON = False # Avoid pretty_print prints html which doesn't play well with this theme
|
|
```
|
|
|
|
## RESTful API
|
|
|
|
RESTful API is an endpoint where you send your documents to then get the parsed annotations back.
|
|
We are hosting a **non-commercial** API service and you are welcome to [apply for an auth key](https://bbs.hankcs.com/t/apply-for-free-hanlp-restful-apis/3178).
|
|
An auth key is a password which gives you access to our API and protects our server from being abused.
|
|
Once obtained such an auth key, you can parse your document with our RESTful client which can be installed via:
|
|
|
|
````{margin} **Non-Commercial**
|
|
```{seealso}
|
|
Our models and RESTful APIs are under the [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) licence.
|
|
```
|
|
````
|
|
|
|
````{margin} **Zero-Shot Learning**
|
|
```{note}
|
|
Although UD covers 104 languages, OntoNotes (NER, CON, SRL) covers only English, Chinese and Arabic.
|
|
So NER/CON/SRL of languages other than the 3 are considered as Zero-Shot and their accuracies can be very low.
|
|
```
|
|
````
|
|
|
|
```bash
|
|
pip install hanlp_restful
|
|
```
|
|
|
|
```{eval-rst}
|
|
Then initiate a :class:`~hanlp_restful.HanLPClient` with your auth key and send a document to have it parsed.
|
|
```
|
|
|
|
```{code-cell} ipython3
|
|
:tags: [output_scroll]
|
|
from hanlp_restful import HanLPClient
|
|
# Fill in your auth, set language='zh' to use Chinese models
|
|
HanLP = HanLPClient('https://hanlp.hankcs.com/api', auth=None, language='mul')
|
|
doc = HanLP('In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environments. ' \
|
|
'2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。' \
|
|
'2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。')
|
|
print(doc)
|
|
```
|
|
````{margin} **But what do these annotations mean?**
|
|
```{seealso}
|
|
See our [data format](data_format) and [annotations](annotations/index) for details.
|
|
```
|
|
````
|
|
|
|
|
|
## Visualization
|
|
|
|
```{eval-rst}
|
|
The returned :class:`~hanlp_common.document.Document` has a handy method :meth:`~hanlp_common.document.Document.pretty_print`
|
|
which offers visualization in any mono-width text environment.
|
|
```
|
|
|
|
````{margin} **Non-ASCII**
|
|
```{note}
|
|
Non-ASCII text might be skewed in terminals but in Jupyter Notebook it will align correctly.
|
|
You can also use our [live demo](https://hanlp.hankcs.com/).
|
|
```
|
|
````
|
|
|
|
````{margin} **Non-Projective**
|
|
```{note}
|
|
Non-projective dependency trees cannot be visualized and won't be printed out at this moment.
|
|
```
|
|
````
|
|
|
|
```{code-cell} ipython3
|
|
doc.pretty_print()
|
|
```
|
|
|
|
## Native API
|
|
|
|
### Multi-Task Learning
|
|
|
|
If you want to run our models locally or you want to implement your own RESTful server,
|
|
you can [install the native API](https://hanlp.hankcs.com/docs/install.html#install-native-package)
|
|
and call it just like the RESTful one.
|
|
|
|
````{margin} **Sentences Required**
|
|
```{seealso}
|
|
As MTL doesn't predict sentence boundaries, inputs have to be split beforehand.
|
|
See our [data format](data_format) for details.
|
|
```
|
|
````
|
|
|
|
```{code-cell} ipython3
|
|
:tags: [output_scroll]
|
|
import hanlp
|
|
HanLP = hanlp.load(hanlp.pretrained.mtl.UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_XLMR_BASE)
|
|
print(HanLP(['In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environments.',
|
|
'2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。',
|
|
'2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。']))
|
|
```
|
|
|
|
Due to the fact that the service provider is very likely running a different model or having different settings, the
|
|
RESTful and native results might be slightly different.
|
|
|
|
To process Chinese or Japanese, HanLP provides mono-lingual models in each language which significantly outperform the multi-lingual model. See [docs](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/mtl.html) for the list of models.
|
|
|
|
### Single-Task Learning
|
|
|
|
HanLP also provides a full spectrum of single-task learning models for core NLP tasks including tagging and parsing. Please refer to the documentations of [`pretrained`](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/index.html) models for details. |