chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,40 @@
|
||||
---
|
||||
name: 🐛发现一个bug
|
||||
about: 需提交版本号、触发代码、错误日志
|
||||
title: ''
|
||||
labels: bug
|
||||
assignees: hankcs
|
||||
|
||||
---
|
||||
|
||||
<!--
|
||||
感谢找出bug，请认真填写下表：
|
||||
-->
|
||||
|
||||
**Describe the bug**
|
||||
A clear and concise description of what the bug is.
|
||||
|
||||
**Code to reproduce the issue**
|
||||
Provide a reproducible test case that is the bare minimum necessary to generate the problem.
|
||||
|
||||
```python
|
||||
```
|
||||
|
||||
**Describe the current behavior**
|
||||
A clear and concise description of what happened.
|
||||
|
||||
**Expected behavior**
|
||||
A clear and concise description of what you expected to happen.
|
||||
|
||||
**System information**
|
||||
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
|
||||
- Python version:
|
||||
- HanLP version:
|
||||
|
||||
**Other info / logs**
|
||||
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
|
||||
|
||||
* [ ] I've completed this form and searched the web for solutions.
|
||||
<!-- ⬆️此处务必勾选，否则你的issue会被机器人自动删除！ -->
|
||||
<!-- ⬆️此处务必勾选，否则你的issue会被机器人自动删除！ -->
|
||||
<!-- ⬆️此处务必勾选，否则你的issue会被机器人自动删除！ -->
|
||||
Executable
+5
@@ -0,0 +1,5 @@
|
||||
blank_issues_enabled: false
|
||||
contact_links:
|
||||
- name: ⁉️ 提问求助请上论坛
|
||||
url: https://bbs.hankcs.com/
|
||||
about: 欢迎前往蝴蝶效应论坛求助
|
||||
@@ -0,0 +1,36 @@
|
||||
---
|
||||
name: 🚀新功能请愿
|
||||
about: 建议增加一个新功能
|
||||
title: ''
|
||||
labels: feature request
|
||||
assignees: hankcs
|
||||
|
||||
---
|
||||
|
||||
<!--
|
||||
提问请上论坛，不要发这里！
|
||||
提问请上论坛，不要发这里！
|
||||
提问请上论坛，不要发这里！
|
||||
|
||||
以下必填，否则直接关闭。
|
||||
-->
|
||||
|
||||
**Describe the feature and the current behavior/state.**
|
||||
|
||||
**Will this change the current api? How?**
|
||||
|
||||
**Who will benefit with this feature?**
|
||||
|
||||
**Are you willing to contribute it (Yes/No):**
|
||||
|
||||
**System information**
|
||||
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
|
||||
- Python version:
|
||||
- HanLP version:
|
||||
|
||||
**Any other info**
|
||||
|
||||
* [ ] I've carefully completed this form.
|
||||
<!-- 发表前先搜索，此处一定要勾选！ -->
|
||||
<!-- 发表前先搜索，此处一定要勾选！ -->
|
||||
<!-- 发表前先搜索，此处一定要勾选！ -->
|
||||
@@ -0,0 +1,38 @@
|
||||
<!--
|
||||
Thank you for being interested in contributing to HanLP! You are awesome ✨.
|
||||
⚠️Changes must be made on dev branch.
|
||||
-->
|
||||
|
||||
# Title of Your Pull Request
|
||||
|
||||
## Description
|
||||
|
||||
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
|
||||
|
||||
Fixes # (issue)
|
||||
|
||||
## Type of Change
|
||||
|
||||
Please check any relevant options and delete the rest.
|
||||
|
||||
- [ ] Bug fix (non-breaking change which fixes an issue)
|
||||
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
|
||||
- [ ] New feature (non-breaking change which adds functionality)
|
||||
- [ ] This change requires a documentation update
|
||||
|
||||
## How Has This Been Tested?
|
||||
|
||||
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
|
||||
|
||||
## Checklist
|
||||
|
||||
Check all items that apply.
|
||||
|
||||
- [ ] ⚠️Changes **must** be made on `dev` branch instead of `master`
|
||||
- [ ] I have added tests that prove my fix is effective or that my feature works
|
||||
- [ ] New and existing unit tests pass locally with my changes
|
||||
- [ ] My code follows the style guidelines of this project
|
||||
- [ ] I have commented my code, particularly in hard-to-understand areas
|
||||
- [ ] I have made corresponding changes to the documentation
|
||||
- [ ] My changes generate no new warnings
|
||||
- [ ] I have checked my code and corrected any misspellings
|
||||
@@ -0,0 +1,86 @@
|
||||
name: Unit Tests
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ "**" ]
|
||||
pull_request:
|
||||
branches: [ "**" ]
|
||||
|
||||
jobs:
|
||||
build:
|
||||
|
||||
runs-on: ${{ matrix.os }}
|
||||
env:
|
||||
HANLP_HOME: ${{ github.workspace }}/data
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
os: [ ubuntu-latest, macos-latest, windows-latest ]
|
||||
python-version: [ 3.6, 3.7, 3.8, 3.9, '3.10' ]
|
||||
exclude:
|
||||
# GHA doesn't list 3.6 for ubuntu-22.04
|
||||
- os: ubuntu-latest
|
||||
python-version: "3.6"
|
||||
|
||||
# MacOS 14.4.1 for arm64 doesn't support Python < 3.8
|
||||
- os: macos-latest
|
||||
python-version: "3.6"
|
||||
- os: macos-latest
|
||||
python-version: "3.7"
|
||||
|
||||
include:
|
||||
# GHA doesn't list 3.6 for ubuntu-22
|
||||
- os: ubuntu-20.04
|
||||
python-version: "3.6"
|
||||
|
||||
# MacOS 13 required for Python < 3.8
|
||||
- os: macos-13
|
||||
python-version: "3.6"
|
||||
- os: macos-13
|
||||
python-version: "3.7"
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
|
||||
- name: Set up Python ${{ matrix.python-version }}
|
||||
uses: actions/setup-python@v3
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
|
||||
- name: Install dependencies
|
||||
shell: bash
|
||||
run: |
|
||||
python -m pip install -e plugins/hanlp_trie
|
||||
python -m pip install -e plugins/hanlp_common
|
||||
python -m pip install -e .
|
||||
python -m pip install pytest
|
||||
|
||||
- name: Cache data
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: ${{ env.HANLP_HOME }}
|
||||
key: hanlp-data
|
||||
|
||||
- name: Test with pytest
|
||||
shell: bash
|
||||
run: |
|
||||
pytest tests
|
||||
pytest plugins/hanlp_trie/tests
|
||||
deploy:
|
||||
needs: build
|
||||
if: github.event_name == 'push' && github.ref == 'refs/heads/master'
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install setuptools wheel twine
|
||||
- name: Deploy to PyPI
|
||||
run: |
|
||||
python setup.py sdist bdist_wheel
|
||||
python -m twine upload dist/*
|
||||
env:
|
||||
TWINE_USERNAME: __token__
|
||||
TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
|
||||
TWINE_REPOSITORY: pypi
|
||||
+293
@@ -0,0 +1,293 @@
|
||||
# Created by .ignore support plugin (hsz.mobi)
|
||||
### Python template
|
||||
# Byte-compiled / optimized / DLL files
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
|
||||
# C extensions
|
||||
*.so
|
||||
|
||||
# Distribution / packaging
|
||||
.Python
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
lib/
|
||||
lib64/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
wheels/
|
||||
pip-wheel-metadata/
|
||||
share/python-wheels/
|
||||
*.egg-info/
|
||||
.installed.cfg
|
||||
*.egg
|
||||
MANIFEST
|
||||
|
||||
# PyInstaller
|
||||
# Usually these files are written by a python script from a template
|
||||
# before PyInstaller builds the exe, so as to inject date/other infos into it.
|
||||
*.manifest
|
||||
*.spec
|
||||
|
||||
# Installer logs
|
||||
pip-log.txt
|
||||
pip-delete-this-directory.txt
|
||||
|
||||
# Unit test / coverage reports
|
||||
htmlcov/
|
||||
.tox/
|
||||
.nox/
|
||||
.coverage
|
||||
.coverage.*
|
||||
.cache
|
||||
nosetests.xml
|
||||
coverage.xml
|
||||
*.cover
|
||||
.hypothesis/
|
||||
.pytest_cache/
|
||||
|
||||
# Translations
|
||||
*.mo
|
||||
*.pot
|
||||
|
||||
# Django stuff:
|
||||
*.log
|
||||
local_settings.py
|
||||
db.sqlite3
|
||||
db.sqlite3-journal
|
||||
|
||||
# Flask stuff:
|
||||
instance/
|
||||
.webassets-cache
|
||||
|
||||
# Scrapy stuff:
|
||||
.scrapy
|
||||
|
||||
# Sphinx documentation
|
||||
docs/_build/
|
||||
|
||||
# PyBuilder
|
||||
target/
|
||||
|
||||
# Jupyter Notebook
|
||||
.ipynb_checkpoints
|
||||
|
||||
# IPython
|
||||
profile_default/
|
||||
ipython_config.py
|
||||
|
||||
# pyenv
|
||||
.python-version
|
||||
|
||||
# pipenv
|
||||
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
|
||||
# However, in case of collaboration, if having platform-specific dependencies or dependencies
|
||||
# having no cross-platform support, pipenv may install dependencies that don't work, or not
|
||||
# install all needed dependencies.
|
||||
#Pipfile.lock
|
||||
|
||||
# celery beat schedule file
|
||||
celerybeat-schedule
|
||||
|
||||
# SageMath parsed files
|
||||
*.sage.py
|
||||
|
||||
# Environments
|
||||
.env
|
||||
.venv
|
||||
env/
|
||||
venv/
|
||||
ENV/
|
||||
env.bak/
|
||||
venv.bak/
|
||||
|
||||
# Spyder project settings
|
||||
.spyderproject
|
||||
.spyproject
|
||||
|
||||
# Rope project settings
|
||||
.ropeproject
|
||||
|
||||
# mkdocs documentation
|
||||
/site
|
||||
|
||||
# mypy
|
||||
.mypy_cache/
|
||||
.dmypy.json
|
||||
dmypy.json
|
||||
|
||||
# Pyre type checker
|
||||
.pyre/
|
||||
|
||||
### Java template
|
||||
# Compiled class file
|
||||
*.class
|
||||
|
||||
# Log file
|
||||
|
||||
# BlueJ files
|
||||
*.ctxt
|
||||
|
||||
# Mobile Tools for Java (J2ME)
|
||||
.mtj.tmp/
|
||||
|
||||
# Package Files #
|
||||
*.jar
|
||||
*.war
|
||||
*.nar
|
||||
*.ear
|
||||
*.zip
|
||||
*.tar.gz
|
||||
*.rar
|
||||
|
||||
# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml
|
||||
hs_err_pid*
|
||||
|
||||
### Eclipse template
|
||||
.metadata
|
||||
bin/
|
||||
tmp/
|
||||
*.tmp
|
||||
*.bak
|
||||
*.swp
|
||||
*~.nib
|
||||
local.properties
|
||||
.settings/
|
||||
.loadpath
|
||||
.recommenders
|
||||
|
||||
# External tool builders
|
||||
.externalToolBuilders/
|
||||
|
||||
# Locally stored "Eclipse launch configurations"
|
||||
*.launch
|
||||
|
||||
# PyDev specific (Python IDE for Eclipse)
|
||||
*.pydevproject
|
||||
|
||||
# CDT-specific (C/C++ Development Tooling)
|
||||
.cproject
|
||||
|
||||
# CDT- autotools
|
||||
.autotools
|
||||
|
||||
# Java annotation processor (APT)
|
||||
.factorypath
|
||||
|
||||
# PDT-specific (PHP Development Tools)
|
||||
.buildpath
|
||||
|
||||
# sbteclipse plugin
|
||||
.target
|
||||
|
||||
# Tern plugin
|
||||
.tern-project
|
||||
|
||||
# TeXlipse plugin
|
||||
.texlipse
|
||||
|
||||
# STS (Spring Tool Suite)
|
||||
.springBeans
|
||||
|
||||
# Code Recommenders
|
||||
.recommenders/
|
||||
|
||||
# Annotation Processing
|
||||
.apt_generated/
|
||||
|
||||
# Scala IDE specific (Scala & Java development for Eclipse)
|
||||
.cache-main
|
||||
.scala_dependencies
|
||||
.worksheet
|
||||
|
||||
### VisualStudioCode template
|
||||
.vscode/*
|
||||
!.vscode/settings.json
|
||||
!.vscode/tasks.json
|
||||
!.vscode/launch.json
|
||||
!.vscode/extensions.json
|
||||
|
||||
### JetBrains template
|
||||
# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio and WebStorm
|
||||
# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839
|
||||
|
||||
# User-specific stuff
|
||||
.idea/**/workspace.xml
|
||||
.idea/**/tasks.xml
|
||||
.idea/**/usage.statistics.xml
|
||||
.idea/**/dictionaries
|
||||
.idea/**/shelf
|
||||
|
||||
# Generated files
|
||||
.idea/**/contentModel.xml
|
||||
|
||||
# Sensitive or high-churn files
|
||||
.idea/**/dataSources/
|
||||
.idea/**/dataSources.ids
|
||||
.idea/**/dataSources.local.xml
|
||||
.idea/**/sqlDataSources.xml
|
||||
.idea/**/dynamic.xml
|
||||
.idea/**/uiDesigner.xml
|
||||
.idea/**/dbnavigator.xml
|
||||
|
||||
# Gradle
|
||||
.idea/**/gradle.xml
|
||||
.idea/**/libraries
|
||||
|
||||
# Gradle and Maven with auto-import
|
||||
# When using Gradle or Maven with auto-import, you should exclude module files,
|
||||
# since they will be recreated, and may cause churn. Uncomment if using
|
||||
# auto-import.
|
||||
# .idea/modules.xml
|
||||
# .idea/*.iml
|
||||
# .idea/modules
|
||||
# *.iml
|
||||
# *.ipr
|
||||
|
||||
# CMake
|
||||
cmake-build-*/
|
||||
|
||||
# Mongo Explorer plugin
|
||||
.idea/**/mongoSettings.xml
|
||||
|
||||
# File-based project format
|
||||
*.iws
|
||||
|
||||
# IntelliJ
|
||||
out/
|
||||
|
||||
# mpeltonen/sbt-idea plugin
|
||||
.idea_modules/
|
||||
|
||||
# JIRA plugin
|
||||
atlassian-ide-plugin.xml
|
||||
|
||||
# Cursive Clojure plugin
|
||||
.idea/replstate.xml
|
||||
|
||||
# Crashlytics plugin (for Android Studio and IntelliJ)
|
||||
com_crashlytics_export_strings.xml
|
||||
crashlytics.properties
|
||||
crashlytics-build.properties
|
||||
fabric.properties
|
||||
|
||||
# Editor-based Rest HanLPClient
|
||||
.idea/httpRequests
|
||||
|
||||
# Android studio 3.1+ serialized cache file
|
||||
.idea/caches/build_file_checksums.ser
|
||||
.idea
|
||||
*.iml
|
||||
data
|
||||
.vscode
|
||||
*.pkl
|
||||
*.pdf
|
||||
_static/
|
||||
_build/
|
||||
_templates/
|
||||
@@ -0,0 +1,40 @@
|
||||
cff-version: 1.2.0
|
||||
message: "If you use this software, please cite it as below."
|
||||
authors:
|
||||
- family-names: He
|
||||
given-names: Han
|
||||
orcid: "https://orcid.org/0009-0005-1778-917X"
|
||||
title: "HanLP: Han Language Processing"
|
||||
version: 2.1
|
||||
date-released: 2015-05-27
|
||||
url: "https://github.com/hankcs/HanLP"
|
||||
preferred-citation:
|
||||
type: conference-paper
|
||||
authors:
|
||||
- family-names: He
|
||||
given-names: Han
|
||||
- family-names: Choi
|
||||
given-names: Jinho D.
|
||||
title: "The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders"
|
||||
editors:
|
||||
- family-names: Moens
|
||||
given-names: Marie-Francine
|
||||
- family-names: Huang
|
||||
given-names: Xuanjing
|
||||
- family-names: Specia
|
||||
given-names: Lucia
|
||||
- family-names: Yih
|
||||
given-names: Scott Wen-tau
|
||||
year: 2021
|
||||
month: 11
|
||||
date-released: 2021-11
|
||||
conference:
|
||||
name: "2021 Conference on Empirical Methods in Natural Language Processing"
|
||||
place: "Online and Punta Cana, Dominican Republic"
|
||||
url: "https://aclanthology.org/2021.emnlp-main.451"
|
||||
doi: "10.18653/v1/2021.emnlp-main.451"
|
||||
url: "https://aclanthology.org/2021.emnlp-main.451"
|
||||
publisher: "Association for Computational Linguistics"
|
||||
booktitle: "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing"
|
||||
location: "Online and Punta Cana, Dominican Republic"
|
||||
pages: "5555-5577"
|
||||
@@ -0,0 +1,201 @@
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
@@ -0,0 +1,343 @@
|
||||
<h2 align="center">HanLP: Han Language Processing</h2>
|
||||
|
||||
<div align="center">
|
||||
<a href="https://github.com/hankcs/HanLP/actions/workflows/unit-tests.yml">
|
||||
<img alt="Unit Tests" src="https://github.com/hankcs/hanlp/actions/workflows/unit-tests.yml/badge.svg?branch=master">
|
||||
</a>
|
||||
<a href="https://pypi.org/project/hanlp/">
|
||||
<img alt="PyPI Version" src="https://img.shields.io/pypi/v/hanlp?color=blue">
|
||||
</a>
|
||||
<a href="https://pypi.org/project/hanlp/">
|
||||
<img alt="Python Versions" src="https://img.shields.io/pypi/pyversions/hanlp?colorB=blue">
|
||||
</a>
|
||||
<a href="https://pepy.tech/project/hanlp">
|
||||
<img alt="Downloads" src="https://static.pepy.tech/badge/hanlp">
|
||||
</a>
|
||||
<a href="https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Ftutorial.ipynb">
|
||||
<img alt="在线运行" src="https://mybinder.org/badge_logo.svg">
|
||||
</a>
|
||||
</div>
|
||||
<h4 align="center">
|
||||
<a href="https://github.com/hankcs/HanLP/tree/master">English</a> |
|
||||
<a href="https://github.com/hankcs/HanLP/tree/doc-ja">日本語</a> |
|
||||
<a href="https://hanlp.hankcs.com/docs/">文档</a> |
|
||||
<a href="https://bbs.hankcs.com/t/topic/3940">论文</a> |
|
||||
<a href="https://bbs.hankcs.com/">论坛</a> |
|
||||
<a href="https://github.com/wangedison/hanlp-jupyterlab-docker">docker</a> |
|
||||
<a href="https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Ftutorial.ipynb">▶️在线运行</a>
|
||||
</h4>
|
||||
|
||||
|
||||
|
||||
面向生产环境的多语种自然语言处理工具包，基于PyTorch和TensorFlow 2.x双引擎，目标是普及落地最前沿的NLP技术。HanLP具备功能完善、精度准确、性能高效、语料时新、架构清晰、可自定义的特点。
|
||||
|
||||
[](https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Ftutorial.ipynb)
|
||||
|
||||
借助世界上最大的多语种语料库，HanLP2.1支持包括简繁中英日俄法德在内的[130种语言](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/mtl.html#hanlp.pretrained.mtl.UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_MMINILMV2L6)上的10种联合任务以及多种单任务。HanLP预训练了十几种任务上的数十个模型并且正在持续迭代语料库与模型：
|
||||
|
||||
<div align="center">
|
||||
|
||||
| 功能 | RESTful | 多任务 | 单任务 | 模型 | 标注标准 |
|
||||
| ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ |
|
||||
| [分词](https://hanlp.hankcs.com/demos/tok.html) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/tok_restful.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/tok_mtl.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/tok_stl.ipynb) | [tok](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/tok.html) | [粗分](https://hanlp.hankcs.com/docs/annotations/tok/msr.html)、[细分](https://hanlp.hankcs.com/docs/annotations/tok/ctb.html) |
|
||||
| [词性标注](https://hanlp.hankcs.com/demos/pos.html) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/pos_restful.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/pos_mtl.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/pos_stl.ipynb) | [pos](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/pos.html) | [CTB](https://hanlp.hankcs.com/docs/annotations/pos/ctb.html)、[PKU](https://hanlp.hankcs.com/docs/annotations/pos/pku.html)、[863](https://hanlp.hankcs.com/docs/annotations/pos/863.html) |
|
||||
| [命名实体识别](https://hanlp.hankcs.com/demos/ner.html) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/ner_restful.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/ner_mtl.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/ner_stl.ipynb) | [ner](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/ner.html) | [PKU](https://hanlp.hankcs.com/docs/annotations/ner/pku.html)、[MSRA](https://hanlp.hankcs.com/docs/annotations/ner/msra.html)、[OntoNotes](https://hanlp.hankcs.com/docs/annotations/ner/ontonotes.html) |
|
||||
| [依存句法分析](https://hanlp.hankcs.com/demos/dep.html) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/dep_restful.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/dep_mtl.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/dep_stl.ipynb) | [dep](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/dep.html) | [SD](https://hanlp.hankcs.com/docs/annotations/dep/sd_zh.html)、[UD](https://hanlp.hankcs.com/docs/annotations/dep/ud.html#chinese)、[PMT](https://hanlp.hankcs.com/docs/annotations/dep/pmt.html) |
|
||||
| [成分句法分析](https://hanlp.hankcs.com/demos/con.html) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/con_restful.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/con_mtl.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/con_stl.ipynb) | [con](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/constituency.html) | [Chinese Tree Bank](https://hanlp.hankcs.com/docs/annotations/constituency/ctb.html) |
|
||||
| [语义依存分析](https://hanlp.hankcs.com/demos/sdp.html) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/sdp_restful.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/sdp_mtl.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/sdp_stl.ipynb) | [sdp](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/sdp.html) | [CSDP](https://hanlp.hankcs.com/docs/annotations/sdp/semeval16.html#) |
|
||||
| [语义角色标注](https://hanlp.hankcs.com/demos/srl.html) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/srl_restful.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/srl_mtl.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/srl_stl.ipynb) | [srl](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/srl.html) | [Chinese Proposition Bank](https://hanlp.hankcs.com/docs/annotations/srl/cpb.html) |
|
||||
| [抽象意义表示](https://hanlp.hankcs.com/demos/amr.html) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/amr_restful.ipynb) | 暂无 | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/amr_stl.ipynb) | [amr](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/amr.html) | [CAMR](https://www.hankcs.com/nlp/corpus/introduction-to-chinese-abstract-meaning-representation.html) |
|
||||
| [指代消解](https://hanlp.hankcs.com/demos/cor.html) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/cor_restful.ipynb) | 暂无 | 暂无 | 暂无 | OntoNotes |
|
||||
| [语义文本相似度](https://hanlp.hankcs.com/demos/sts.html) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/sts_restful.ipynb) | 暂无 | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/sts_stl.ipynb) | [sts](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/sts.html) | 暂无 |
|
||||
| [文本风格转换](https://hanlp.hankcs.com/demos/tst.html) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/tst_restful.ipynb) | 暂无 | 暂无 | 暂无 | 暂无 |
|
||||
| [关键词短语提取](https://hanlp.hankcs.com/demos/keyphrase.html) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/keyphrase_restful.ipynb) | 暂无 | 暂无 | 暂无 | 暂无 |
|
||||
| [抽取式自动摘要](https://hanlp.hankcs.com/demos/exsum.html) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/extractive_summarization_restful.ipynb) | 暂无 | 暂无 | 暂无 | 暂无 |
|
||||
| [生成式自动摘要](https://hanlp.hankcs.com/demos/absum.html) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/abstractive_summarization_restful.ipynb) | 暂无 | 暂无 | 暂无 | 暂无 |
|
||||
| [文本语法纠错](https://hanlp.hankcs.com/demos/gec.html) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/gec_restful.ipynb) | 暂无 | 暂无 | 暂无 | 暂无 |
|
||||
| [文本分类](https://hanlp.hankcs.com/demos/classification.html) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/classification_restful.ipynb) | 暂无 | 暂无 | 暂无 | 暂无 |
|
||||
| [情感分析](https://hanlp.hankcs.com/demos/sentiment.html) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/sentiment_restful.ipynb) | 暂无 | 暂无 | 暂无 | `[-1,+1]` |
|
||||
| [语种检测](https://hanlp.hankcs.com/demos/classification.html) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/lid_restful.ipynb) | 暂无 | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/lid_stl.ipynb) | 暂无 | [ISO 639-1编码](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) |
|
||||
|
||||
</div>
|
||||
|
||||
- 词干提取、词法语法特征提取请参考[英文教程](https://hanlp.hankcs.com/docs/tutorial.html)；[词向量](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/word2vec.html)和[完形填空](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/mlm.html)请参考相应文档。
|
||||
- 简繁转换、拼音、新词发现、文本聚类请参考[1.x教程](https://github.com/hankcs/HanLP/tree/1.x)。
|
||||
|
||||
量体裁衣，HanLP提供**RESTful**和**native**两种API，分别面向轻量级和海量级两种场景。无论何种API何种语言，HanLP接口在语义上保持一致，在代码上坚持开源。如果您在研究中使用了HanLP，请引用我们的[EMNLP论文](https://aclanthology.org/2021.emnlp-main.451/)。
|
||||
|
||||
### 轻量级RESTful API
|
||||
|
||||
仅数KB，适合敏捷开发、移动APP等场景。简单易用，无需GPU配环境，秒速安装。语料更多、模型更大、精度更高，**强烈推荐**。服务器GPU算力有限，匿名用户配额较少，[建议申请**免费公益**API秘钥`auth`](https://bbs.hanlp.com/t/hanlp2-1-restful-api/53)。
|
||||
|
||||
#### Python
|
||||
|
||||
```shell
|
||||
pip install hanlp_restful
|
||||
```
|
||||
|
||||
创建客户端，填入服务器地址和秘钥：
|
||||
|
||||
```python
|
||||
from hanlp_restful import HanLPClient
|
||||
HanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh') # auth不填则匿名，zh中文，mul多语种
|
||||
```
|
||||
|
||||
#### Golang
|
||||
|
||||
安装 `go get -u github.com/hankcs/gohanlp@main` ，创建客户端，填入服务器地址和秘钥：
|
||||
|
||||
```go
|
||||
HanLP := hanlp.HanLPClient(hanlp.WithAuth(""),hanlp.WithLanguage("zh")) // auth不填则匿名，zh中文，mul多语种
|
||||
```
|
||||
|
||||
#### Java
|
||||
|
||||
在`pom.xml`中添加依赖：
|
||||
|
||||
```xml
|
||||
<dependency>
|
||||
<groupId>com.hankcs.hanlp.restful</groupId>
|
||||
<artifactId>hanlp-restful</artifactId>
|
||||
<version>0.0.12</version>
|
||||
</dependency>
|
||||
```
|
||||
|
||||
创建客户端，填入服务器地址和秘钥：
|
||||
|
||||
```java
|
||||
HanLPClient HanLP = new HanLPClient("https://www.hanlp.com/api", null, "zh"); // auth不填则匿名，zh中文，mul多语种
|
||||
```
|
||||
|
||||
#### 快速上手
|
||||
|
||||
无论何种开发语言，调用`parse`接口，传入一篇文章，得到HanLP精准的分析结果。
|
||||
|
||||
```java
|
||||
HanLP.parse("2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。阿婆主来到北京立方庭参观自然语义科技公司。")
|
||||
```
|
||||
|
||||
更多功能包括语义相似度、风格转换、指代消解等，请参考[文档](https://hanlp.hankcs.com/docs/api/restful.html)和[测试用例](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_restful/tests/test_client.py)。
|
||||
|
||||
### 海量级native API
|
||||
|
||||
依赖PyTorch、TensorFlow等深度学习技术，适合**专业**NLP工程师、研究者以及本地海量数据场景。要求Python 3.6至3.10，支持Windows，推荐*nix。可以在CPU上运行，推荐GPU/TPU。安装PyTorch版：
|
||||
|
||||
```bash
|
||||
pip install hanlp
|
||||
```
|
||||
|
||||
- HanLP每次发布都通过了Linux、macOS和Windows上Python3.6至3.10的[单元测试](https://github.com/hankcs/HanLP/actions?query=branch%3Amaster)，不存在安装问题。
|
||||
|
||||
HanLP发布的模型分为多任务和单任务两种，多任务速度快省显存，单任务精度高更灵活。
|
||||
|
||||
#### 多任务模型
|
||||
|
||||
HanLP的工作流程为加载模型然后将其当作函数调用，例如下列联合多任务模型：
|
||||
|
||||
```python
|
||||
import hanlp
|
||||
HanLP = hanlp.load(hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH) # 世界最大中文语料库
|
||||
HanLP(['2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', '阿婆主来到北京立方庭参观自然语义科技公司。'])
|
||||
```
|
||||
|
||||
Native API的输入单位为句子，需使用[多语种分句模型](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_demo/hanlp_demo/sent_split.py)或[基于规则的分句函数](https://github.com/hankcs/HanLP/blob/master/hanlp/utils/rules.py#L19)先行分句。RESTful和native两种API的语义设计完全一致，用户可以无缝互换。简洁的接口也支持灵活的参数，常用的技巧有：
|
||||
|
||||
- 灵活的`tasks`任务调度，任务越少，速度越快，详见[教程](https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Ftutorial.ipynb)。在内存有限的场景下，用户还可以[删除不需要的任务](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_demo/hanlp_demo/zh/demo_del_tasks.py)达到模型瘦身的效果。
|
||||
- 高效的trie树自定义词典，以及强制、合并、校正3种规则，请参考[demo](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/tok_mtl.ipynb)和[文档](https://hanlp.hankcs.com/docs/api/hanlp/components/tokenizers/transformer.html)。规则系统的效果将无缝应用到后续统计模型，从而快速适应新领域。
|
||||
|
||||
#### 单任务模型
|
||||
|
||||
根据我们的[最新研究](https://aclanthology.org/2021.emnlp-main.451)，多任务学习的优势在于速度和显存，然而精度往往不如单任务模型。所以，HanLP预训练了许多单任务模型并设计了优雅的[流水线模式](https://hanlp.hankcs.com/docs/api/hanlp/components/pipeline.html#hanlp.components.pipeline.Pipeline)将其组装起来。
|
||||
|
||||
```python
|
||||
import hanlp
|
||||
HanLP = hanlp.pipeline() \
|
||||
.append(hanlp.utils.rules.split_sentence, output_key='sentences') \
|
||||
.append(hanlp.load('FINE_ELECTRA_SMALL_ZH'), output_key='tok') \
|
||||
.append(hanlp.load('CTB9_POS_ELECTRA_SMALL'), output_key='pos') \
|
||||
.append(hanlp.load('MSRA_NER_ELECTRA_SMALL_ZH'), output_key='ner', input_key='tok') \
|
||||
.append(hanlp.load('CTB9_DEP_ELECTRA_SMALL', conll=0), output_key='dep', input_key='tok')\
|
||||
.append(hanlp.load('CTB9_CON_ELECTRA_SMALL'), output_key='con', input_key='tok')
|
||||
HanLP('2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。阿婆主来到北京立方庭参观自然语义科技公司。')
|
||||
```
|
||||
|
||||
更多功能，请参考[demo](https://github.com/hankcs/HanLP/tree/doc-zh/plugins/hanlp_demo/hanlp_demo/zh)和[文档](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/index.html)了解更多模型与用法。
|
||||
|
||||
### 输出格式
|
||||
|
||||
无论何种API何种开发语言何种自然语言，HanLP的输出统一为`json`格式兼容`dict`的[`Document`](https://hanlp.hankcs.com/docs/api/common/document.html):
|
||||
|
||||
```json
|
||||
{
|
||||
"tok/fine": [
|
||||
["2021年", "HanLPv2.1", "为", "生产", "环境", "带来", "次", "世代", "最", "先进", "的", "多", "语种", "NLP", "技术", "。"],
|
||||
["阿婆主", "来到", "北京", "立方庭", "参观", "自然", "语义", "科技", "公司", "。"]
|
||||
],
|
||||
"tok/coarse": [
|
||||
["2021年", "HanLPv2.1", "为", "生产", "环境", "带来", "次世代", "最", "先进", "的", "多语种", "NLP", "技术", "。"],
|
||||
["阿婆主", "来到", "北京立方庭", "参观", "自然语义科技公司", "。"]
|
||||
],
|
||||
"pos/ctb": [
|
||||
["NT", "NR", "P", "NN", "NN", "VV", "JJ", "NN", "AD", "JJ", "DEG", "CD", "NN", "NR", "NN", "PU"],
|
||||
["NN", "VV", "NR", "NR", "VV", "NN", "NN", "NN", "NN", "PU"]
|
||||
],
|
||||
"pos/pku": [
|
||||
["t", "nx", "p", "vn", "n", "v", "b", "n", "d", "a", "u", "a", "n", "nx", "n", "w"],
|
||||
["n", "v", "ns", "ns", "v", "n", "n", "n", "n", "w"]
|
||||
],
|
||||
"pos/863": [
|
||||
["nt", "w", "p", "v", "n", "v", "a", "nt", "d", "a", "u", "a", "n", "ws", "n", "w"],
|
||||
["n", "v", "ns", "n", "v", "n", "n", "n", "n", "w"]
|
||||
],
|
||||
"ner/pku": [
|
||||
[],
|
||||
[["北京立方庭", "ns", 2, 4], ["自然语义科技公司", "nt", 5, 9]]
|
||||
],
|
||||
"ner/msra": [
|
||||
[["2021年", "DATE", 0, 1], ["HanLPv2.1", "ORGANIZATION", 1, 2]],
|
||||
[["北京", "LOCATION", 2, 3], ["立方庭", "LOCATION", 3, 4], ["自然语义科技公司", "ORGANIZATION", 5, 9]]
|
||||
],
|
||||
"ner/ontonotes": [
|
||||
[["2021年", "DATE", 0, 1], ["HanLPv2.1", "ORG", 1, 2]],
|
||||
[["北京立方庭", "FAC", 2, 4], ["自然语义科技公司", "ORG", 5, 9]]
|
||||
],
|
||||
"srl": [
|
||||
[[["2021年", "ARGM-TMP", 0, 1], ["HanLPv2.1", "ARG0", 1, 2], ["为生产环境", "ARG2", 2, 5], ["带来", "PRED", 5, 6], ["次世代最先进的多语种NLP技术", "ARG1", 6, 15]], [["最", "ARGM-ADV", 8, 9], ["先进", "PRED", 9, 10], ["技术", "ARG0", 14, 15]]],
|
||||
[[["阿婆主", "ARG0", 0, 1], ["来到", "PRED", 1, 2], ["北京立方庭", "ARG1", 2, 4]], [["阿婆主", "ARG0", 0, 1], ["参观", "PRED", 4, 5], ["自然语义科技公司", "ARG1", 5, 9]]]
|
||||
],
|
||||
"dep": [
|
||||
[[6, "tmod"], [6, "nsubj"], [6, "prep"], [5, "nn"], [3, "pobj"], [0, "root"], [8, "amod"], [15, "nn"], [10, "advmod"], [15, "rcmod"], [10, "assm"], [13, "nummod"], [15, "nn"], [15, "nn"], [6, "dobj"], [6, "punct"]],
|
||||
[[2, "nsubj"], [0, "root"], [4, "nn"], [2, "dobj"], [2, "conj"], [9, "nn"], [9, "nn"], [9, "nn"], [5, "dobj"], [2, "punct"]]
|
||||
],
|
||||
"sdp": [
|
||||
[[[6, "Time"]], [[6, "Exp"]], [[5, "mPrep"]], [[5, "Desc"]], [[6, "Datv"]], [[13, "dDesc"]], [[0, "Root"], [8, "Desc"], [13, "Desc"]], [[15, "Time"]], [[10, "mDegr"]], [[15, "Desc"]], [[10, "mAux"]], [[8, "Quan"], [13, "Quan"]], [[15, "Desc"]], [[15, "Nmod"]], [[6, "Pat"]], [[6, "mPunc"]]],
|
||||
[[[2, "Agt"], [5, "Agt"]], [[0, "Root"]], [[4, "Loc"]], [[2, "Lfin"]], [[2, "ePurp"]], [[8, "Nmod"]], [[9, "Nmod"]], [[9, "Nmod"]], [[5, "Datv"]], [[5, "mPunc"]]]
|
||||
],
|
||||
"con": [
|
||||
["TOP", [["IP", [["NP", [["NT", ["2021年"]]]], ["NP", [["NR", ["HanLPv2.1"]]]], ["VP", [["PP", [["P", ["为"]], ["NP", [["NN", ["生产"]], ["NN", ["环境"]]]]]], ["VP", [["VV", ["带来"]], ["NP", [["ADJP", [["NP", [["ADJP", [["JJ", ["次"]]]], ["NP", [["NN", ["世代"]]]]]], ["ADVP", [["AD", ["最"]]]], ["VP", [["JJ", ["先进"]]]]]], ["DEG", ["的"]], ["NP", [["QP", [["CD", ["多"]]]], ["NP", [["NN", ["语种"]]]]]], ["NP", [["NR", ["NLP"]], ["NN", ["技术"]]]]]]]]]], ["PU", ["。"]]]]]],
|
||||
["TOP", [["IP", [["NP", [["NN", ["阿婆主"]]]], ["VP", [["VP", [["VV", ["来到"]], ["NP", [["NR", ["北京"]], ["NR", ["立方庭"]]]]]], ["VP", [["VV", ["参观"]], ["NP", [["NN", ["自然"]], ["NN", ["语义"]], ["NN", ["科技"]], ["NN", ["公司"]]]]]]]], ["PU", ["。"]]]]]]
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
特别地，Python RESTful和native API支持基于等宽字体的[可视化](https://hanlp.hankcs.com/docs/tutorial.html#visualization)，能够直接将语言学结构在控制台内可视化出来：
|
||||
|
||||
```python
|
||||
HanLP(['2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', '阿婆主来到北京立方庭参观自然语义科技公司。']).pretty_print()
|
||||
|
||||
Dep Tree Token Relati PoS Tok NER Type Tok SRL PA1 Tok SRL PA2 Tok PoS 3 4 5 6 7 8 9
|
||||
──────────── ───────── ────── ─── ───────── ──────────────── ───────── ──────────── ───────── ──────────── ───────── ─────────────────────────────────────────────────────────
|
||||
┌─────────► 2021年 tmod NT 2021年 ───►DATE 2021年 ───►ARGM-TMP 2021年 2021年 NT ───────────────────────────────────────────►NP ───┐
|
||||
│┌────────► HanLPv2.1 nsubj NR HanLPv2.1 ───►ORGANIZATION HanLPv2.1 ───►ARG0 HanLPv2.1 HanLPv2.1 NR ───────────────────────────────────────────►NP────┤
|
||||
││┌─►┌───── 为 prep P 为 为 ◄─┐ 为 为 P ───────────┐ │
|
||||
│││ │ ┌─► 生产 nn NN 生产 生产 ├►ARG2 生产 生产 NN ──┐ ├────────────────────────►PP ───┐ │
|
||||
│││ └─►└── 环境 pobj NN 环境 环境 ◄─┘ 环境 环境 NN ──┴►NP ───┘ │ │
|
||||
┌┼┴┴──────── 带来 root VV 带来 带来 ╟──►PRED 带来 带来 VV ──────────────────────────────────┐ │ │
|
||||
││ ┌─► 次 amod JJ 次 次 ◄─┐ 次 次 JJ ───►ADJP──┐ │ ├►VP────┤
|
||||
││ ┌───►└── 世代 nn NN 世代 世代 │ 世代 世代 NN ───►NP ───┴►NP ───┐ │ │ │
|
||||
││ │ ┌─► 最 advmod AD 最 最 │ 最 ───►ARGM-ADV 最 AD ───────────►ADVP──┼►ADJP──┐ ├►VP ───┘ ├►IP
|
||||
││ │┌──►├── 先进 rcmod JJ 先进 先进 │ 先进 ╟──►PRED 先进 JJ ───────────►VP ───┘ │ │ │
|
||||
││ ││ └─► 的 assm DEG 的 的 ├►ARG1 的 的 DEG──────────────────────────┤ │ │
|
||||
││ ││ ┌─► 多 nummod CD 多 多 │ 多 多 CD ───►QP ───┐ ├►NP ───┘ │
|
||||
││ ││┌─►└── 语种 nn NN 语种 语种 │ 语种 语种 NN ───►NP ───┴────────►NP────┤ │
|
||||
││ │││ ┌─► NLP nn NR NLP NLP │ NLP NLP NR ──┐ │ │
|
||||
│└─►└┴┴──┴── 技术 dobj NN 技术 技术 ◄─┘ 技术 ───►ARG0 技术 NN ──┴────────────────►NP ───┘ │
|
||||
└──────────► 。 punct PU 。 。 。 。 PU ──────────────────────────────────────────────────┘
|
||||
|
||||
Dep Tree Tok Relat Po Tok NER Type Tok SRL PA1 Tok SRL PA2 Tok Po 3 4 5 6
|
||||
──────────── ─── ───── ── ─── ──────────────── ─── ──────── ─── ──────── ─── ────────────────────────────────
|
||||
┌─► 阿婆主 nsubj NN 阿婆主 阿婆主 ───►ARG0 阿婆主 ───►ARG0 阿婆主 NN───────────────────►NP ───┐
|
||||
┌┬────┬──┴── 来到 root VV 来到 来到 ╟──►PRED 来到 来到 VV──────────┐ │
|
||||
││ │ ┌─► 北京 nn NR 北京 ───►LOCATION 北京 ◄─┐ 北京 北京 NR──┐ ├►VP ───┐ │
|
||||
││ └─►└── 立方庭 dobj NR 立方庭 ───►LOCATION 立方庭 ◄─┴►ARG1 立方庭 立方庭 NR──┴►NP ───┘ │ │
|
||||
│└─►┌─────── 参观 conj VV 参观 参观 参观 ╟──►PRED 参观 VV──────────┐ ├►VP────┤
|
||||
│ │ ┌───► 自然 nn NN 自然 ◄─┐ 自然 自然 ◄─┐ 自然 NN──┐ │ │ ├►IP
|
||||
│ │ │┌──► 语义 nn NN 语义 │ 语义 语义 │ 语义 NN │ ├►VP ───┘ │
|
||||
│ │ ││┌─► 科技 nn NN 科技 ├►ORGANIZATION 科技 科技 ├►ARG1 科技 NN ├►NP ───┘ │
|
||||
│ └─►└┴┴── 公司 dobj NN 公司 ◄─┘ 公司 公司 ◄─┘ 公司 NN──┘ │
|
||||
└──────────► 。 punct PU 。 。 。 。 PU──────────────────────────┘
|
||||
```
|
||||
|
||||
关于标注集含义，请参考[《语言学标注规范》](https://hanlp.hankcs.com/docs/annotations/index.html)及[《格式规范》](https://hanlp.hankcs.com/docs/data_format.html)。我们购买、标注或采用了世界上量级最大、种类最多的语料库用于联合多语种多任务学习，所以HanLP的标注集也是覆盖面最广的。
|
||||
|
||||
## 训练你自己的领域模型
|
||||
|
||||
写深度学习模型一点都不难，难的是复现较高的准确率。下列[代码](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_demo/hanlp_demo/zh/train_sota_bert_pku.py)展示了如何在sighan2005 PKU语料库上花6分钟训练一个超越学术界state-of-the-art的中文分词模型。
|
||||
|
||||
```python
|
||||
tokenizer = TransformerTaggingTokenizer()
|
||||
save_dir = 'data/model/cws/sighan2005_pku_bert_base_96.73'
|
||||
tokenizer.fit(
|
||||
SIGHAN2005_PKU_TRAIN_ALL,
|
||||
SIGHAN2005_PKU_TEST, # Conventionally, no devset is used. See Tian et al. (2020).
|
||||
save_dir,
|
||||
'bert-base-chinese',
|
||||
max_seq_len=300,
|
||||
char_level=True,
|
||||
hard_constraint=True,
|
||||
sampler_builder=SortingSamplerBuilder(batch_size=32),
|
||||
epochs=3,
|
||||
adam_epsilon=1e-6,
|
||||
warmup_steps=0.1,
|
||||
weight_decay=0.01,
|
||||
word_dropout=0.1,
|
||||
seed=1660853059,
|
||||
)
|
||||
tokenizer.evaluate(SIGHAN2005_PKU_TEST, save_dir)
|
||||
```
|
||||
|
||||
其中，由于指定了随机数种子，结果一定是`96.73`。不同于那些虚假宣传的学术论文或商业项目，HanLP保证所有结果可复现。如果你有任何质疑，我们将当作最高优先级的致命性bug第一时间排查问题。
|
||||
|
||||
请参考[demo](https://github.com/hankcs/HanLP/tree/master/plugins/hanlp_demo/hanlp_demo/zh/train)了解更多训练脚本。
|
||||
|
||||
## 性能
|
||||
|
||||
<table><thead><tr><th rowspan="2">lang</th><th rowspan="2">corpora</th><th rowspan="2">model</th><th colspan="2">tok</th><th colspan="4">pos</th><th colspan="3">ner</th><th rowspan="2">dep</th><th rowspan="2">con</th><th rowspan="2">srl</th><th colspan="4">sdp</th><th rowspan="2">lem</th><th rowspan="2">fea</th><th rowspan="2">amr</th></tr><tr><th>fine</th><th>coarse</th><th>ctb</th><th>pku</th><th>863</th><th>ud</th><th>pku</th><th>msra</th><th>ontonotes</th><th>SemEval16</th><th>DM</th><th>PAS</th><th>PSD</th></tr></thead><tbody><tr><td rowspan="2">mul</td><td rowspan="2">UD2.7<br>OntoNotes5</td><td>small</td><td>98.62</td><td>-</td><td>-</td><td>-</td><td>-</td><td>93.23</td><td>-</td><td>-</td><td>74.42</td><td>79.10</td><td>76.85</td><td>70.63</td><td>-</td><td>91.19</td><td>93.67</td><td>85.34</td><td>87.71</td><td>84.51</td><td>-</td></tr><tr><td>base</td><td>98.97</td><td>-</td><td>-</td><td>-</td><td>-</td><td>90.32</td><td>-</td><td>-</td><td>80.32</td><td>78.74</td><td>71.23</td><td>73.63</td><td>-</td><td>92.60</td><td>96.04</td><td>81.19</td><td>85.08</td><td>82.13</td><td>-</td></tr><tr><td rowspan="5">zh</td><td rowspan="2">open</td><td>small</td><td>97.25</td><td>-</td><td>96.66</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>95.00</td><td>84.57</td><td>87.62</td><td>73.40</td><td>84.57</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>base</td><td>97.50</td><td>-</td><td>97.07</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>96.04</td><td>87.11</td><td>89.84</td><td>77.78</td><td>87.11</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td rowspan="3">close</td><td>small</td><td>96.70</td><td>95.93</td><td>96.87</td><td>97.56</td><td>95.05</td><td>-</td><td>96.22</td><td>95.74</td><td>76.79</td><td>84.44</td><td>88.13</td><td>75.81</td><td>74.28</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>base</td><td>97.52</td><td>96.44</td><td>96.99</td><td>97.59</td><td>95.29</td><td>-</td><td>96.48</td><td>95.72</td><td>77.77</td><td>85.29</td><td>88.57</td><td>76.52</td><td>73.76</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>ernie</td><td>96.95</td><td>97.29</td><td>96.76</td><td>97.64</td><td>95.22</td><td>-</td><td>97.31</td><td>96.47</td><td>77.95</td><td>85.67</td><td>89.17</td><td>78.51</td><td>74.10</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr></tbody></table>
|
||||
|
||||
- 根据我们的[最新研究](https://aclanthology.org/2021.emnlp-main.451)，单任务学习的性能往往优于多任务学习。在乎精度甚于速度的话，建议使用[单任务模型](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/index.html)。
|
||||
|
||||
HanLP采用的数据预处理与拆分比例与流行方法未必相同，比如HanLP采用了[完整版的MSRA命名实体识别语料](https://bbs.hankcs.com/t/topic/3033)，而非大众使用的阉割版；HanLP使用了语法覆盖更广的[Stanford Dependencies标准](https://hanlp.hankcs.com/docs/annotations/dep/sd_zh.html)，而非学术界沿用的Zhang and Clark (2008)标准；HanLP提出了[均匀分割CTB的方法](https://bbs.hankcs.com/t/topic/3024)，而不采用学术界不均匀且遗漏了51个黄金文件的方法。HanLP开源了[一整套语料预处理脚本与相应语料库](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_demo/hanlp_demo/zh/train/open_small.py)，力图推动中文NLP的透明化。
|
||||
|
||||
总之，HanLP只做我们认为正确、先进的事情，而不一定是流行、权威的事情。
|
||||
|
||||
## 引用
|
||||
|
||||
如果你在研究中使用了HanLP，请按如下格式引用：
|
||||
|
||||
```bibtex
|
||||
@inproceedings{he-choi-2021-stem,
|
||||
title = "The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders",
|
||||
author = "He, Han and Choi, Jinho D.",
|
||||
booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
|
||||
month = nov,
|
||||
year = "2021",
|
||||
address = "Online and Punta Cana, Dominican Republic",
|
||||
publisher = "Association for Computational Linguistics",
|
||||
url = "https://aclanthology.org/2021.emnlp-main.451",
|
||||
pages = "5555--5577",
|
||||
abstract = "Multi-task learning with transformer encoders (MTL) has emerged as a powerful technique to improve performance on closely-related tasks for both accuracy and efficiency while a question still remains whether or not it would perform as well on tasks that are distinct in nature. We first present MTL results on five NLP tasks, POS, NER, DEP, CON, and SRL, and depict its deficiency over single-task learning. We then conduct an extensive pruning analysis to show that a certain set of attention heads get claimed by most tasks during MTL, who interfere with one another to fine-tune those heads for their own objectives. Based on this finding, we propose the Stem Cell Hypothesis to reveal the existence of attention heads naturally talented for many tasks that cannot be jointly trained to create adequate embeddings for all of those tasks. Finally, we design novel parameter-free probes to justify our hypothesis and demonstrate how attention heads are transformed across the five tasks during MTL through label analysis.",
|
||||
}
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
### 源代码
|
||||
|
||||
HanLP源代码的授权协议为 **Apache License 2.0**，可免费用做商业用途。请在产品说明中附加HanLP的链接和授权协议。HanLP受版权法保护，侵权必究。
|
||||
|
||||
##### 自然语义（青岛）科技有限公司
|
||||
|
||||
HanLP从v1.7版起独立运作，由自然语义（青岛）科技有限公司作为项目主体，主导后续版本的开发，并拥有后续版本的版权。
|
||||
|
||||
##### 上海林原公司
|
||||
|
||||
HanLP 早期得到了上海林原公司的大力支持，并拥有1.28及前序版本的版权，相关版本也曾在上海林原公司网站发布。
|
||||
|
||||
### 预训练模型
|
||||
|
||||
机器学习模型的授权在法律上没有定论，但本着尊重开源语料库原始授权的精神，如不特别说明，HanLP的多语种模型授权沿用[CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)，中文模型授权为仅供研究与教学使用。
|
||||
|
||||
## References
|
||||
|
||||
https://hanlp.hankcs.com/docs/references.html
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
# WeHub 来源说明
|
||||
|
||||
- 原始项目：`hankcs/HanLP`
|
||||
- 原始仓库：https://github.com/hankcs/HanLP
|
||||
- 导入方式：上游默认分支的最新快照
|
||||
- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
|
||||
- 本文件仅用于记录来源，不代表 WeHub 是原项目作者
|
||||
@@ -0,0 +1,20 @@
|
||||
# Minimal makefile for Sphinx documentation
|
||||
#
|
||||
|
||||
# You can set these variables from the command line, and also
|
||||
# from the environment for the first two.
|
||||
SPHINXOPTS ?=
|
||||
SPHINXBUILD ?= sphinx-build
|
||||
SOURCEDIR = .
|
||||
BUILDDIR = _build
|
||||
|
||||
# Put it first so that "make" without argument is like "make help".
|
||||
help:
|
||||
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
|
||||
|
||||
.PHONY: help Makefile
|
||||
|
||||
# Catch-all target: route all unknown targets to Sphinx using the new
|
||||
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
|
||||
%: Makefile
|
||||
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
|
||||
@@ -0,0 +1,54 @@
|
||||
<!--
|
||||
# ========================================================================
|
||||
# Copyright 2020 hankcs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be included in all
|
||||
# copies or substantial portions of the Software.
|
||||
# ========================================================================
|
||||
-->
|
||||
|
||||
# Chinese Tree Bank
|
||||
|
||||
See also [The Bracketing Guidelines for the Penn Chinese Treebank (3.0)](https://repository.upenn.edu/cgi/viewcontent.cgi?article=1040&context=ircs_reports).
|
||||
|
||||
| Tag | Definition | 定义 | 例子 |
|
||||
|------|----------------------------------------------|----------------------------------------------------|-------------------|
|
||||
| ADJP | adjective phrase | 形容词短语，以形容词为中心词 | 不完全、大型 |
|
||||
| ADVP | adverbial phrase headed by AD (adverb) | 副词短语，以副词为中心词 | 非常、很 |
|
||||
| CLP | classifier phrase | 由量词构成的短语 | 系列、大批 |
|
||||
| CP | clause headed by C (complementizer) | 从句，通过带补语（如“的”、“吗”等） | 张三喜欢李四吗？ |
|
||||
| DNP | phrase formed by ‘‘XP + DEG’’ | 结构为XP + DEG(的)的短语，其中XP可以是ADJP、DP、QP、PP等等，用于修饰名词短语。 | 大型的、前几年的、五年的、在上海的 |
|
||||
| DP | determiner phrase | 限定词短语，通常由限定词和数量词构成 | 这三个、任何 |
|
||||
| DVP | phrase formed by ‘‘XP + DEV’’ | 结构为XP+地的短评，用于修饰动词短语VP | 心情失落地、大批地 |
|
||||
| FRAG | fragment | 片段 | (完） |
|
||||
| INTJ | interjection | 插话，感叹语 | 哈哈、切 |
|
||||
| IP | simple clause headed by I (INFL) | 简单子句或句子，通常不带补语（如“的”、“吗”等） | 张三喜欢李四。 |
|
||||
| LCP | phrase formed by ‘‘XP + LC’’ | 用于表本地点+方位词（LC)的短语 | 生活中、田野上 |
|
||||
| LST | list marker | 列表短语，包括标点符号 | 一. |
|
||||
| MSP | some particles | 其他小品词 | 所、而、来、去 |
|
||||
| NN | common noun | 名词 | HanLP、技术 |
|
||||
| NP | noun phrase | 名词短语，中心词通常为名词 | 美好生活、经济水平 |
|
||||
| PP | preposition phrase | 介词短语，中心词通常为介词 | 在北京、据报道 |
|
||||
| PRN | parenthetical | 插入语 | ，（张三说)， |
|
||||
| QP | quantifier phrase | 量词短语 | 三个、五百辆 |
|
||||
| TOP | root node | 根节点 | 根节点 |
|
||||
| UCP | unidentical coordination phrase | 不对称的并列短语，指并列词两侧的短语类型不致 | (养老、医疗）保险 |
|
||||
| VCD | coordinated verb compound | 复合动词 | 出版发行 |
|
||||
| VCP | verb compounds formed by VV + VC | VV + VC形式的动词短语 | 看作是 |
|
||||
| VNV | verb compounds formed by A-not-A or A-one-A | V不V形式的动词短语 | 能不能、信不信 |
|
||||
| VP | verb phrase | 动词短语，中心词通常为动词 | 完成任务、努力工作 |
|
||||
| VPT | potential form V-de-R or V-bu-R | V不R、V得R形式的动词短语 | 打不赢、打得过 |
|
||||
| VRD | verb resultative compound | 动补结构短语 | 研制成功、降下来 |
|
||||
| VSB | verb compounds formed by a modifier + a head | 修饰语+中心词构成的动词短语 | 拿来支付、仰头望去 |
|
||||
@@ -0,0 +1,17 @@
|
||||
# Constituency Parsing
|
||||
|
||||
## Chinese
|
||||
```{toctree}
|
||||
ctb
|
||||
```
|
||||
|
||||
## English
|
||||
```{toctree}
|
||||
ptb
|
||||
```
|
||||
|
||||
## Japanese
|
||||
```{toctree}
|
||||
npcmj
|
||||
```
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
<!--
|
||||
# ========================================================================
|
||||
# Copyright 2020 hankcs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be included in all
|
||||
# copies or substantial portions of the Software.
|
||||
# ========================================================================
|
||||
-->
|
||||
|
||||
# NPCMJ
|
||||
|
||||
| Tag | Description |
|
||||
|-----------------|-----------------------------------------|
|
||||
| ADVP | adverb phrase |
|
||||
| ADVP-CMPL | complement adverb phrase |
|
||||
| ADVP-MSR | measurement adverb phrase |
|
||||
| ADVP-PRD | predicate adverb phrase |
|
||||
| ADVP-TMP | temporal adverb phrase |
|
||||
| CONJP | conjunction phrase |
|
||||
| CP-EXL | exclamative |
|
||||
| CP-IMP | imperative |
|
||||
| CP-FINAL | projection for sentence final particle |
|
||||
| CP-QUE | question (direct or indirect) |
|
||||
| CP-QUE-ADV | question used adverbially |
|
||||
| CP-QUE-OB1 | question used as object |
|
||||
| CP-QUE-PRD | question used as a nominal predicate |
|
||||
| CP-QUE-SBJ | question used as subject |
|
||||
| CP-THT | complementizer clause |
|
||||
| CP-THT-ADV | complementizer clause used adverbially |
|
||||
| CP-THT-OB1 | complementizer clause used as object |
|
||||
| CP-THT-PRD | complementizer clause used as predicate |
|
||||
| CP-THT-PRP | purposive complementizer clause |
|
||||
| CP-THT-SBJ | complementizer clause used as subject |
|
||||
| FRAG | fragment |
|
||||
| FS | false start |
|
||||
| INTJP | interjection phrase |
|
||||
| IP-ADV | adverbial clause |
|
||||
| IP-ADV-CONJ | coordinated clause |
|
||||
| IP-ADV-PRD | adverbial clause used as predicate |
|
||||
| IP-ADV-SCON | subordinate clause |
|
||||
| IP-ADV-SCON-CND | |
|
||||
| conditional | clause |
|
||||
| IP-EMB | gapless noun-modifying clause |
|
||||
| IP-IMP | imperative clause |
|
||||
| IP-MAT | matrix clause |
|
||||
| IP-NMZ | nominalized clause |
|
||||
| IP-NMZ-PRD | nominalized clause used as predicate |
|
||||
| IP-REL | relative clause |
|
||||
| IP-SMC | small clause |
|
||||
| IP-SMC-CNT | small clause in continuative form |
|
||||
| IP-SMC-OB1 | small clause used as object |
|
||||
| IP-SMC-SBJ | small clause used as subject |
|
||||
| IP-SUB | clause under CP* layer |
|
||||
| multi-sentence | multiple sentence |
|
||||
| NML | intermediate nominal layer |
|
||||
| NP | noun phrase |
|
||||
| NP-ADV | adverbial noun phrase |
|
||||
| NP-CZZ | causee noun phrase |
|
||||
| NP-DOB1 | derived primary object noun phrase |
|
||||
| NP-DSBJ | derived subject noun phrase |
|
||||
| NP-LGS | logical subject noun phrase |
|
||||
| NP-LOC | locational noun phrase |
|
||||
| NP-MSR | measure noun phrase |
|
||||
| NP-OB1 | primary object noun phrase |
|
||||
| NP-OB2 | secondary object noun phrase |
|
||||
| NP-POS | possessive noun phrase |
|
||||
| NP-PRD | predicate noun phrase |
|
||||
| NP-SBJ | subject noun phrase |
|
||||
| NP-SBJ2 | secondary subject noun phrase |
|
||||
| NP-TMP | temporal noun phrase |
|
||||
| NP-TPC | topic noun phrase |
|
||||
| NP-VOC | vocative noun phrase |
|
||||
| NUMCLP | numeral-classifier phrase |
|
||||
| PNLP | prenominal phrase |
|
||||
| PP | particle phrase |
|
||||
| PP-ADV | adverbial particle phrase |
|
||||
| PP-CMPL | complement particle phrase |
|
||||
| PP-CONJ | coordination particle phrase |
|
||||
| PP-CZZ | causee particle phrase |
|
||||
| PP-DOB1 | derived primary object particle phrase |
|
||||
| PP-DSBJ | derived subject particle phrase |
|
||||
| PP-LGS | logical subject particle phrase |
|
||||
| PP-LOC | locational particle phrase |
|
||||
| PP-MSR | measure particle phrase |
|
||||
| PP-OB1 | primary object particle phrase |
|
||||
| PP-OB2 | secondary object particle phrase |
|
||||
| PP-PRD | predicate particle phrase |
|
||||
| PP-PRP | purpositive particle phrase |
|
||||
| PP-SBJ | subject particle phrase |
|
||||
| PP-SBJ2 | secondary subject particle phrase |
|
||||
| PP-SCON | subordination particle phrase |
|
||||
| PP-SCON-CND | conditional particle phrase |
|
||||
| PP-TMP | temporal particle phrase |
|
||||
| PP-TPC | topic particle phrase |
|
||||
| PP-VOC | vocative particle phrase |
|
||||
| PRN | parenthetical |
|
||||
@@ -0,0 +1,53 @@
|
||||
<!--
|
||||
# ========================================================================
|
||||
# Copyright 2020 hankcs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be included in all
|
||||
# copies or substantial portions of the Software.
|
||||
# ========================================================================
|
||||
-->
|
||||
|
||||
# Penn Treebank
|
||||
|
||||
| Tag | Description |
|
||||
|--------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| ADJP | Adjective Phrase. |
|
||||
| ADVP | Adverb Phrase. |
|
||||
| CONJP | Conjunction Phrase. |
|
||||
| FRAG | Fragment. |
|
||||
| INTJ | Interjection. Corresponds approximately to the part-of-speech tag UH. |
|
||||
| LST | List marker. Includes surrounding punctuation. |
|
||||
| NAC | Not a Constituent; used to show the scope of certain prenominal modifiers within an NP. |
|
||||
| NP | Noun Phrase. |
|
||||
| NX | - Used within certain complex NPs to mark the head of the NP. Corresponds very roughly to N-bar level but used quite differently. |
|
||||
| PP | Prepositional Phrase. |
|
||||
| PRN | Parenthetical |
|
||||
| PRT | Particle. Category for words that should be tagged RP. |
|
||||
| QP | Quantifier Phrase (i.e. complex measure/amount phrase); used within NP. |
|
||||
| ROOT | No description |
|
||||
| RRC | Reduced Relative Clause. |
|
||||
| S | conjunction or a wh-word and that does not exhibit subject-verb inversion. |
|
||||
| SBAR | Clause introduced by a (possibly empty) subordinating conjunction. |
|
||||
| SBARQ | - Direct question introduced by a wh-word or a wh-phrase. Indirect questions and relative clauses should be bracketed as SBAR, not SBARQ. |
|
||||
| SINV | - Inverted declarative sentence, i.e. one in which the subject follows the tensed verb or modal. |
|
||||
| SQ | Inverted yes/no question, or main clause of a wh-question, following the wh-phrase in SBARQ. |
|
||||
| UCP | Unlike Coordinated Phrase. |
|
||||
| VP | Verb Phrase. |
|
||||
| WHADJP | Wh-adjective Phrase. Adjectival phrase containing a wh-adverb, as in how hot. |
|
||||
| WHADVP | - Wh-adverb Phrase. Introduces a clause with an NP gap. May be null (containing the 0 complementizer) or lexical, containing a wh-adverb such as how or why. |
|
||||
| WHNP | - Wh-noun Phrase. Introduces a clause with an NP gap. May be null (containing the 0 complementizer) or lexical, containing some wh-word, e.g. who, which book, whose daughter, none of which, or how many leopards. |
|
||||
| WHPP | - Wh-prepositional Phrase. Prepositional phrase containing a wh-noun phrase (such as of which or by whose authority) that either introduces a PP gap or is contained by a WHNP. |
|
||||
| X | - Unknown, uncertain, or unbracketable. X is often used for bracketing typos and in bracketing the…the-constructions. |
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
# Dependency Parsing
|
||||
|
||||
## Chinese
|
||||
|
||||
```{toctree}
|
||||
sd_zh
|
||||
pmt
|
||||
```
|
||||
|
||||
## English
|
||||
|
||||
```{toctree}
|
||||
sd_en
|
||||
```
|
||||
|
||||
## Multilingual
|
||||
|
||||
```{toctree}
|
||||
ud
|
||||
```
|
||||
@@ -0,0 +1,64 @@
|
||||
<!--
|
||||
# ========================================================================
|
||||
# Copyright 2020 hankcs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be included in all
|
||||
# copies or substantial portions of the Software.
|
||||
# ========================================================================
|
||||
-->
|
||||
|
||||
# PKU Multi-view Chinese Treebank
|
||||
|
||||
|
||||
```{eval-rst}
|
||||
|
||||
See also :cite:`qiu-etal-2014-multi`.
|
||||
|
||||
```
|
||||
|
||||
| Tag | Description | 依存关系 |
|
||||
| ---- | ------------------------------------------- | -------------- |
|
||||
| ACT | action object | 行为宾语 |
|
||||
| ADV | adverbial | 状语 |
|
||||
| APP | appositive element | 同位 |
|
||||
| ATT | attribute | 定语 |
|
||||
| CMP | complement | 补语 |
|
||||
| COO | other coordination element | 一般并列 |
|
||||
| COS | share-right-child coordination element | 共享并列 |
|
||||
| DE | de (modifier of 的(special function word)) | 的字 |
|
||||
| DEI | dei (modifier of 得(special function word)) | 得字 |
|
||||
| DI | di (modifier of 地(special function word)) | 地字 |
|
||||
| FOC | focus | 强调 |
|
||||
| HED | root of a sentence | 核心 |
|
||||
| IC | independent clause | 小句 |
|
||||
| IOB | indirect object | 间接宾语 |
|
||||
| IS | independent structure | 独立结构 |
|
||||
| ISC | non-shared independent structure | 并列式独立结构 |
|
||||
| LAD | left additive | 前附加 |
|
||||
| MT | modality and time | 时体 |
|
||||
| NUM | number | 数字 |
|
||||
| POB | propositional object | 介宾 |
|
||||
| PUN | punctuation | 标点 |
|
||||
| PUS | cross-clause punctuation | 跨句标点 |
|
||||
| QUC | post-positional quantity | 数量补语 |
|
||||
| QUCC | non-shared post-positional quantity | 非共享数量补语 |
|
||||
| QUN | quantity | 数量 |
|
||||
| RAD | right additive | 后附加 |
|
||||
| RADC | non-shared right additive | 非共享后附加 |
|
||||
| RED | reduplicate element | 重叠 |
|
||||
| SBV | subject | 主语 |
|
||||
| TPC | topic | 话题 |
|
||||
| VOB | direct object | 宾语 |
|
||||
| VV | serial verb construction | 连动 |
|
||||
@@ -0,0 +1,88 @@
|
||||
<!--
|
||||
# ========================================================================
|
||||
# Copyright 2020 hankcs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be included in all
|
||||
# copies or substantial portions of the Software.
|
||||
# ========================================================================
|
||||
-->
|
||||
|
||||
# Stanford Dependencies English
|
||||
|
||||
See also [Stanford typed dependencies manual](https://nlp.stanford.edu/software/dependencies_manual.pdf).
|
||||
|
||||
| Tag | Description |
|
||||
|------------|-----------------------------------|
|
||||
| abbrev | abbreviation modifier |
|
||||
| acomp | adjectival complement |
|
||||
| advcl | adverbial clause modifier |
|
||||
| advmod | adverbial modifier |
|
||||
| agent | agent |
|
||||
| amod | adjectival modifier |
|
||||
| appos | appositional modifier |
|
||||
| arg | argument |
|
||||
| attr | attributive |
|
||||
| aux | auxiliary |
|
||||
| auxpass | passive auxiliary |
|
||||
| cc | coordination |
|
||||
| ccomp | clausal complement |
|
||||
| comp | complement |
|
||||
| complm | complementizer |
|
||||
| conj | conjunct |
|
||||
| cop | copula |
|
||||
| csubj | clausal subject |
|
||||
| csubjpass | clausal passive subject |
|
||||
| dep | dependent |
|
||||
| det | determiner |
|
||||
| discourse | discourse element |
|
||||
| dobj | direct object |
|
||||
| expl | expletive |
|
||||
| goeswith | goes with |
|
||||
| iobj | indirect object |
|
||||
| mark | marker |
|
||||
| mod | modifier |
|
||||
| mwe | multi-word expression |
|
||||
| neg | negation modifier |
|
||||
| nn | noun compound modifier |
|
||||
| npadvmod | noun phrase as adverbial modifier |
|
||||
| nsubj | nominal subject |
|
||||
| nsubjpass | passive nominal subject |
|
||||
| num | numeric modifier |
|
||||
| number | element of compound number |
|
||||
| obj | object |
|
||||
| parataxis | parataxis |
|
||||
| pcomp | prepositional complement |
|
||||
| pobj | object of a preposition |
|
||||
| poss | possession modifier |
|
||||
| possessive | possessive modifier |
|
||||
| preconj | preconjunct |
|
||||
| pred | predicate |
|
||||
| predet | predeterminer |
|
||||
| prep | prepositional modifier |
|
||||
| prepc | prepositional clausal modifier |
|
||||
| prt | phrasal verb particle |
|
||||
| punct | punctuation |
|
||||
| purpcl | purpose clause modifier |
|
||||
| quantmod | quantifier phrase modifier |
|
||||
| rcmod | relative clause modifier |
|
||||
| ref | referent |
|
||||
| rel | relative |
|
||||
| root | root |
|
||||
| sdep | semantic dependent |
|
||||
| subj | subject |
|
||||
| tmod | temporal modifier |
|
||||
| vmod | verb modifier |
|
||||
| xcomp | open clausal complement |
|
||||
| xsubj | controlling subject |
|
||||
@@ -0,0 +1,77 @@
|
||||
<!--
|
||||
# ========================================================================
|
||||
# Copyright 2020 hankcs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be included in all
|
||||
# copies or substantial portions of the Software.
|
||||
# ========================================================================
|
||||
-->
|
||||
|
||||
# Stanford Dependencies Chinese
|
||||
|
||||
|
||||
```{eval-rst}
|
||||
|
||||
See also :cite:`chang-etal-2009-discriminative`.
|
||||
|
||||
```
|
||||
|
||||
|Tag|Description|中文简称|例句|依存弧|
|
||||
| ---- | ---- | ---- | ---- | ---- |
|
||||
|nn|noun compound modifier|复合名词修饰|服务中心|nn(中心，服务）|
|
||||
|punct|punctuation|标点符号|海关统计表明，|punct(表明，，)|
|
||||
|nsubj|nominal subject|名词性主语|梅花盛开|nsubj (盛开，梅花）|
|
||||
|conj|conjunct (links two conjuncts)|连接性状语|设备和原材料|conj(原材料，设备）|
|
||||
|dobj|direct object|直接宾语|浦东颁布了七十一件文件|dobj(颁布，文件）|
|
||||
|advmod|adverbial modifier|副词性状语|部门先送上文件|advmod(送上，先）|
|
||||
|prep|prepositional modifier|介词性修饰语|在实践中逐步完善|prep(完善，在）|
|
||||
|nummod|number modifier|数词修饰语|七十一件文件|nummod(件，七十一）|
|
||||
|amod|adjectival modifier|形容词修饰语|跨世纪工程|amod(工程，跨世纪）|
|
||||
|pobj|prepositional object|介词性宾语|根据有关规定|pobj (根据，规定）|
|
||||
|rcmod|relative clause modifier|关系从句修饰语|不曾遇到过的情况|rcmod(情况，遇到）|
|
||||
|cpm|complementizer|补语|开发浦东的经济活动|cpm(开发，的）|
|
||||
|assm|associative marker|关联标记|企业的商品|assm(企业，的）|
|
||||
|assmod|associative modifier|关联修饰|企业的商品|assmod(商品，企业）|
|
||||
|cc|coordinating conjunction|并列关系|设备和原材料|cc(原材料，和）|
|
||||
|clf|classifier modifier|类别修饰|七十一件文件|clf(文件，件）|
|
||||
|ccomp|clausal complement|从句补充|银行决定先取得信用评级|ccomp(决定，取得）|
|
||||
|det|determiner|限定语|这些经济活动|det(活动，这些）|
|
||||
|lobj|localizer object|范围宾语|近年来|lobj(来，近年）|
|
||||
|range|dative object that is a quantifier phrase|数量词间接宾语|成交药品一亿多元|range(成交，元）|
|
||||
|asp|aspect marker|时态标记|发挥了作用|asp(发挥，了）|
|
||||
|tmod|temporal modifier|时间修饰语|以前不曾遇到过|tmod(遇到，以前）|
|
||||
|plmod|localizer modifier of a preposition|介词性地点修饰|在这片热土上|plmod(在，上）|
|
||||
|attr|attributive|属性|贸易额为二百亿美元|attr(为，美元）|
|
||||
|mmod|modal verb modifier|情态动词|利益能得到保障|mmod(得到，能）|
|
||||
|loc|localizer|位置补语|占九成以上|loc(占，以上）|
|
||||
|top|topic|主题|建筑是主要活动|top(是，建筑）|
|
||||
|pccomp|clausal complement of a preposition|介词补语|据有关部门介绍|pccomp(据，介绍）|
|
||||
|etc|etc modifier|省略关系|科技、文教等领域|etc(文教，等）|
|
||||
|lccomp|clausal complement of a localizer|位置补语|中国对外开放中升起的明星|lccomp(中，开放）|
|
||||
|ordmod|ordinal number modifier|量词修饰|第七个机构|ordmod(个，第七）|
|
||||
|xsubj|controlling subject|控制主语|银行决定先取得信用评级|xsubj (取得，银行）|
|
||||
|neg|negative modifier|否定修饰|以前不曾遇到过|neg(遇到，不）|
|
||||
|rcomp|resultative complement|结果补语|研究成功|rcomp(研究，成功）|
|
||||
|comod|coordinated verb compound modifier|并列联合动词|颁布实行|comod(颁布，实行）|
|
||||
|vmod|verb modifier|动词修饰|其在支持外商企业方面的作用|vmod(方面，支持）|
|
||||
|prtmod|particles such as 所，以，来，而|小品词|在产业化所取得的成就|prtmod(取得，所）|
|
||||
|ba|“ba” construction|把字关系|把注意力转向市场|ba(转向，把）|
|
||||
|dvpm|manner DE(地）modifier|地字修饰|有效地防止流失|dvpm(有效，地）|
|
||||
|dvpmod|a "XP+DEV", phrase that modifies VP|地字动词短语|有效地防止流失|dvpmod(防止，有效）|
|
||||
|prnmod|parenthetical modifier|插入词修饰|八五期间（1990-1995 )|pmmod(期间，1995)|
|
||||
|cop|copular|系动词|原是自给自足的经济|cop(自给自足，是）|
|
||||
|pass|passive marker|被动标记|被认定为高技术产业|pass(认定，被）|
|
||||
|nsubjpass|nominal passive subject|被动名词主语|镍被称作现代工业的维生素|nsubjpass(称作，镍）|
|
||||
|dep|dependent|其他依赖关系|新华社北京二月十二日电|dep(电，新华社）|
|
||||
@@ -0,0 +1,123 @@
|
||||
<!--
|
||||
# ========================================================================
|
||||
# Copyright 2020 hankcs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be included in all
|
||||
# copies or substantial portions of the Software.
|
||||
# ========================================================================
|
||||
-->
|
||||
|
||||
# Universal Dependencies
|
||||
|
||||
## Cross-Linguistic
|
||||
|
||||
See also [Universal Dependencies](https://universaldependencies.org/docs/u/dep/index.html).
|
||||
|
||||
| Tag | Description |
|
||||
|------------|----------------------------------------------|
|
||||
| acl | clausal modifier of noun (adjectival clause) |
|
||||
| advcl | adverbial clause modifier |
|
||||
| advmod | adverbial modifier |
|
||||
| amod | adjectival modifier |
|
||||
| appos | appositional modifier |
|
||||
| aux | auxiliary |
|
||||
| auxpass | passive auxiliary |
|
||||
| case | case marking |
|
||||
| cc | coordinating conjunction |
|
||||
| ccomp | clausal complement |
|
||||
| compound | compound |
|
||||
| conj | conjunct |
|
||||
| cop | copula |
|
||||
| csubj | clausal subject |
|
||||
| csubjpass | clausal passive subject |
|
||||
| dep | unspecified dependency |
|
||||
| det | determiner |
|
||||
| discourse | discourse element |
|
||||
| dislocated | dislocated elements |
|
||||
| dobj | direct object |
|
||||
| expl | expletive |
|
||||
| foreign | foreign words |
|
||||
| goeswith | goes with |
|
||||
| iobj | indirect object |
|
||||
| list | list |
|
||||
| mark | marker |
|
||||
| mwe | multi-word expression |
|
||||
| name | name |
|
||||
| neg | negation modifier |
|
||||
| nmod | nominal modifier |
|
||||
| nsubj | nominal subject |
|
||||
| nsubjpass | passive nominal subject |
|
||||
| nummod | numeric modifier |
|
||||
| parataxis | parataxis |
|
||||
| punct | punctuation |
|
||||
| remnant | remnant in ellipsis |
|
||||
| reparandum | overridden disfluency |
|
||||
| root | root |
|
||||
| vocative | vocative |
|
||||
| xcomp | open clausal complement |
|
||||
|
||||
|
||||
## Localization
|
||||
|
||||
### Chinese
|
||||
|
||||
| Tag | 简称 | 例句 |
|
||||
| :--------------- |---------:| -----------------------------------------------------------: |
|
||||
| acl | 形容词子句 |  |
|
||||
| advcl:loc | 状语从句修饰语 |  |
|
||||
| advmod | 状语 |  |
|
||||
| advmod:dvp | 状语:地 |  |
|
||||
| advmod:loc | 状语:限定 |  |
|
||||
| advmod:rcomp | 状语:因果 |  |
|
||||
| amod | 形容 |  |
|
||||
| amod:ordmod | 形容:数量 |  |
|
||||
| appos | 同位 |  |
|
||||
| aux:asp | 助语:时态 |  |
|
||||
| aux:ba | 助语:把 |  |
|
||||
| aux:modal | 助语:情态 |  |
|
||||
| aux:prtmod | 助语:分词 |  |
|
||||
| auxpass | 被动 |  |
|
||||
| case | 条件 |  |
|
||||
| cc | 并列连词 |  |
|
||||
| ccomp | 从句补语 |  |
|
||||
| compound:nn | 复合名词 |  |
|
||||
| compound:vc | 复合动词 |  |
|
||||
| conj | 连接 |  |
|
||||
| cop | 系动 |  |
|
||||
| csubj | 从句主语 |  |
|
||||
| dep | 未定义 |  |
|
||||
| det | 限定 |  |
|
||||
| discourse | 语气 |  |
|
||||
| dobj | 直接宾语 |  |
|
||||
| etc | 省略 |  |
|
||||
| mark | 标记 |  |
|
||||
| mark:clf | 标记:量词 |  |
|
||||
| name | 名称 |  |
|
||||
| neg | 否定 |  |
|
||||
| nmod | 名词修饰 |  |
|
||||
| nmod:assmod | 名词修饰:关联 |  |
|
||||
| nmod:poss | 名词修饰:所有格 |  |
|
||||
| nmod:prep | 名词修饰:介词 |  |
|
||||
| nmod:range | 名词修饰:范围 |  |
|
||||
| nmod:tmod | 名词修饰:时间 |  |
|
||||
| nmod:topic | 名词修饰:主题 |  |
|
||||
| nsubj | 名词主语 |  |
|
||||
| nsubj:xsubj | 名词主语: 补语 |  |
|
||||
| nsubjpass | 被动态主语 |  |
|
||||
| nummod | 数量 |  |
|
||||
| parataxis:prnmod | 并列 |  |
|
||||
| punct | 标点符号 |  |
|
||||
| root | 根 |  |
|
||||
| xcomp | 从句补语 |  |
|
||||
@@ -0,0 +1,13 @@
|
||||
# Annotations
|
||||
|
||||
|
||||
```{toctree}
|
||||
tok/index
|
||||
pos/index
|
||||
ner/index
|
||||
dep/index
|
||||
sdp/index
|
||||
srl/index
|
||||
constituency/index
|
||||
```
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
# Named Entity Recognition
|
||||
|
||||
## Chinese
|
||||
|
||||
```{toctree}
|
||||
pku
|
||||
msra
|
||||
```
|
||||
|
||||
## Multilingual
|
||||
|
||||
```{toctree}
|
||||
ontonotes
|
||||
```
|
||||
@@ -0,0 +1,56 @@
|
||||
<!--
|
||||
# ========================================================================
|
||||
# Copyright 2020 hankcs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be included in all
|
||||
# copies or substantial portions of the Software.
|
||||
# ========================================================================
|
||||
-->
|
||||
|
||||
# msra
|
||||
|
||||
| Category | Subcategory | Tag-set of Format-1 | Tag-set of Format-2 |
|
||||
|----------|----------------|---------------------|---------------------|
|
||||
| NAMEX | Person | P | PERSON |
|
||||
| | Location | L | LOCATION |
|
||||
| | Organization | 〇 | ORGANIZATION |
|
||||
| TIMEX | Date | dat | DATE |
|
||||
| | Duration | dur | DURATION |
|
||||
| | Time | tim | TIME |
|
||||
| NUMEX | Percent | per | PERCENT |
|
||||
| | Money | mon | MONEY |
|
||||
| | Frequency | fre | FREQUENCY |
|
||||
| | Integer | int | INTEGER |
|
||||
| | Fraction | fra | FRACTION |
|
||||
| | Decimal | dec | DECIMAL |
|
||||
| | Ordinal | ord | ORDINAL |
|
||||
| | Rate | rat | RATE |
|
||||
| MEASUREX | Age | age | AGE |
|
||||
| | Weight | wei | WEIGHT |
|
||||
| | Length | len | LENGTH |
|
||||
| | Temperature | tem | TEMPERATURE |
|
||||
| | Angle | ang | ANGLE |
|
||||
| | Area | are | AREA |
|
||||
| | Capacity | cap | CAPACITY |
|
||||
| | Speed | spe | SPEED |
|
||||
| | Acceleration | acc | ACCELERATION |
|
||||
| | Other measures | mea | MEASURE |
|
||||
| ADDREX | Email | ema | EMAIL |
|
||||
| | Phone | pho | PHONE |
|
||||
| | Fax | fax | FAX |
|
||||
| | Telex | tel | TELEX |
|
||||
| | WWW | WWW | WWW |
|
||||
| | Postalcode | pos | POSTALCODE |
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
<!--
|
||||
# ========================================================================
|
||||
# Copyright 2020 hankcs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be included in all
|
||||
# copies or substantial portions of the Software.
|
||||
# ========================================================================
|
||||
-->
|
||||
|
||||
# ontonotes
|
||||
|
||||
| TAG | Description |
|
||||
|--------------|------------------------------------------------------|
|
||||
| PERSON | People, including fictional |
|
||||
| NORP | Nationalities or religious or political groups |
|
||||
| FACILITY | Buildings, airports, highways, bridges, etc. |
|
||||
| ORGANIZATION | Companies, agencies, institutions, etc. |
|
||||
| GPE | Countries, cities, states |
|
||||
| LOCATION | Non-GPE locations, mountain ranges, bodies of water |
|
||||
| PRODUCT | Vehicles, weapons, foods, etc. (Not services) |
|
||||
| EVENT | Named hurricanes, battles, wars, sports events, etc. |
|
||||
| WORK OF ART | Titles of books, songs, etc. |
|
||||
| LAW | Named documents made into laws |
|
||||
| DATE | Absolute or relative dates or periods |
|
||||
| TIME | Times smaller than a day |
|
||||
| PERCENT | Percentage |
|
||||
| MONEY | Monetary values, including unit |
|
||||
| QUANTITY | Measurements, as of weight or distance |
|
||||
| ORDINAL | “first”, “second” |
|
||||
| CARDINAL | Numerals that do not fall under another type |
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
<!--
|
||||
# ========================================================================
|
||||
# Copyright 2020 hankcs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be included in all
|
||||
# copies or substantial portions of the Software.
|
||||
# ========================================================================
|
||||
-->
|
||||
|
||||
# pku
|
||||
|
||||
| 序号 | 词性 | 名称 | 帮助记忆的诠释 | 例子及注解 |
|
||||
| ---- | ---- | -------- | ------------------------------------------------------ | ------------------------------------------------------------ |
|
||||
| 1 | nr | 人名 | 名词代码n和“人(ren)”的声母并在一起。 | 1. 汉族人及与汉族起名方式相同的非汉族人的姓和名单独切分，并分别标注为nr。张/nr 仁伟/nr， 欧阳/nr 修/nr， 阮/nr 志雄/nr， 朴/nr 贞爱/nr汉族人除有单姓和复姓外，还有双姓，即有的女子出嫁后，在原来的姓上加上丈夫的姓。如：陈方安生。这种情况切分、标注为：陈/nr 方/nr 安生/nr；唐姜氏，切分、标注为：唐/nr 姜氏/nr。2. 姓名后的职务、职称或称呼要分开。江/nr 主席/n， 小平/nr 同志/n， 江/nr 总书记/n，张/nr 教授/n， 王/nr 部长/n， 陈/nr 老总/n， 李/nr 大娘/n， 刘/nr 阿姨/n， 龙/nr 姑姑/n3. 对人的简称、尊称等若为两个字，则合为一个切分单位，并标以nr。老张/nr， 大李/nr， 小郝/nr， 郭老/nr， 陈总/nr4. 明显带排行的亲属称谓要切分开，分不清楚的则不切开。三/m 哥/n， 大婶/n， 大/a 女儿/n， 大哥/n， 小弟/n， 老爸/n5. 一些著名作者的或不易区分姓和名的笔名通常作为一个切分单位。鲁迅/nr， 茅盾/nr， 巴金/nr， 三毛/nr， 琼瑶/nr， 白桦/nr6. 外国人或少数民族的译名（包括日本人的姓名）不予切分，标注为nr。克林顿/nr， 叶利钦/nr， 才旦卓玛/nr， 小林多喜二/nr， 北研二/nr，华盛顿/nr， 爱因斯坦/nr有些西方人的姓名中有小圆点，也不分开。卡尔·马克思/nr |
|
||||
| 2 | ns | 地名 | 名词代码n和处所词代码s并在一起。 | 安徽/ns，深圳/ns，杭州/ns，拉萨/ns，哈尔滨/ns， 呼和浩特/ns， 乌鲁木齐/ns，长江/ns，黄海/ns，太平洋/ns， 泰山/ns， 华山/ns，亚洲/ns， 海南岛/ns，太湖/ns，白洋淀/ns， 俄罗斯/ns，哈萨克斯坦/ns，彼得堡/ns， 伏尔加格勒/ns 1. 国名不论长短，作为一个切分单位。中国/ns， 中华人民共和国/ns， 日本国/ns， 美利坚合众国/ns， 美国/ns2. 地名后有“省”、“市”、“县”、“区”、“乡”、“镇”、“村”、“旗”、“州”、“都”、“府”、“道”等单字的行政区划名称时，不切分开，作为一个切分单位。四川省/ns， 天津市/ns，景德镇/ns沙市市/ns， 牡丹江市/ns，正定县/ns，海淀区/ns， 通州区/ns，东升乡/ns， 双桥镇/ns 南化村/ns，华盛顿州/ns，俄亥俄州/ns，东京都/ns， 大阪府/ns，北海道/ns， 长野县/ns，开封府/ns，宣城县/ns3. 地名后的行政区划有两个以上的汉字，则将地名同行政区划名称切开，不过要将地名同行政区划名称用方括号括起来，并标以短语NS。[芜湖/ns 专区/n] NS，[宣城/ns 地区/n]ns，[内蒙古/ns 自治区/n]NS，[深圳/ns 特区/n]NS， [厦门/ns 经济/n 特区/n]NS， [香港/ns 特别/a 行政区/n]NS，[香港/ns 特区/n]NS， [华盛顿/ns 特区/n]NS，4. 地名后有表示地形地貌的一个字的普通名词，如“江、河、山、洋、海、岛、峰、湖”等，不予切分。鸭绿江/ns，亚马逊河/ns， 喜马拉雅山/ns， 珠穆朗玛峰/ns，地中海/ns，大西洋/ns，洞庭湖/ns， 塞普路斯岛/ns 5. 地名后接的表示地形地貌的普通名词若有两个以上汉字，则应切开。然后将地名同该普通名词标成短语NS。[台湾/ns 海峡/n]NS，[华北/ns 平原/n]NS，[帕米尔/ns 高原/n]NS， [南沙/ns 群岛/n]NS，[京东/ns 大/a 峡谷/n]NS [横断/b 山脉/n]NS6．地名后有表示自然区划的一个字的普通名词，如“ 街，路，道，巷，里，町，庄，村，弄，堡”等，不予切分。 中关村/ns，长安街/ns，学院路/ns， 景德镇/ns， 吴家堡/ns， 庞各庄/ns， 三元里/ns，彼得堡/ns， 北菜市巷/ns， 7．地名后接的表示自然区划的普通名词若有两个以上汉字，则应切开。然后将地名同自然区划名词标成短语NS。[米市/ns 大街/n]NS， [蒋家/nz 胡同/n]NS ， [陶然亭/ns 公园/n]NS ， 8． 大小地名相连时的标注方式为：北京市/ns 海淀区/ns 海淀镇/ns [南/f 大街/n]NS [蒋家/nz 胡同/n]NS 24/m 号/q ， |
|
||||
| 3 | nt | 机构团体 | “团”的声母为t，名词代码n和t并在一起。 | （参见2。短语标记说明--NT）联合国/nt，中共中央/nt，国务院/nt， 北京大学/nt1．大多数团体、机构、组织的专有名称一般是短语型的，较长，且含有地名或人名等专名，再组合，标注为短语NT。[中国/ns 计算机/n 学会/n]NT， [香港/ns 钟表业/n 总会/n]NT， [烟台/ns 大学/n]NT， [香港/ns 理工大学/n]NT， [华东/ns 理工大学/n]NT，[合肥/ns 师范/n 学院/n]NT， [北京/ns 图书馆/n]NT， [富士通/nz 株式会社/n]NT， [香山/ns 植物园/n]NT， [安娜/nz 美容院/n]NT，[上海/ns 手表/n 厂/n]NT， [永和/nz 烧饼铺/n]NT，[北京/ns 国安/nz 队/n]NT，2. 对于在国际或中国范围内的知名的唯一的团体、机构、组织的名称即使前面没有专名，也标为nt或NT。联合国/nt，国务院/nt，外交部/nt， 财政部/nt，教育部/nt， 国防部/nt，[世界/n 贸易/n 组织/n]NT， [国家/n 教育/vn 委员会/n]NT，[信息/n 产业/n 部/n]NT，[全国/n 信息/n 技术/n 标准化/vn 委员会/n]NT，[全国/n 总/b 工会/n]NT，[全国/n 人民/n 代表/n 大会/n]NT，美国的“国务院”，其他国家的“外交部、财政部、教育部”，必须在其所属国的国名之后出现时，才联合标注为NT。[美国/ns 国务院/n]NT，[法国/ns 外交部/n]NT，[美/j 国会/n]NT，日本有些政府机构名称很特别，无论是否出现在“日本”国名之后都标为nt。[日本/ns 外务省/nt]NT，[日/j 通产省/nt]NT通产省/nt 3. 前后相连有上下位关系的团体机构组织名称的处理方式如下:[联合国/nt 教科文/j 组织/n]NT， [中国/ns 银行/n 北京/ns 分行/n]NT，[河北省/ns 正定县/ns 西平乐乡/ns 南化村/ns 党支部/n]NT， 当下位名称含有专名（如“北京/ns 分行/n”、“南化村/ns 党支部/n”、“昌平/ns 分校/n”）时，也可脱离前面的上位名称单独标注为NT。[中国/ns 银行/n]NT [北京/ns 分行/n]NT，北京大学/nt [昌平/ns 分校/n]NT，4. 团体、机构、组织名称中用圆括号加注简称时:[宝山/ns 钢铁/n （/w 宝钢/j ）/w 总/b 公司/n]NT，[宝山/ns 钢铁/n 总/b 公司/n]NT，（/w 宝钢/j ）/w |
|
||||
@@ -0,0 +1,59 @@
|
||||
<!--
|
||||
# ========================================================================
|
||||
# Copyright 2020 hankcs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be included in all
|
||||
# copies or substantial portions of the Software.
|
||||
# ========================================================================
|
||||
-->
|
||||
|
||||
# 863
|
||||
|
||||
| 词性 | 名称 | 说明 | 例子 |
|
||||
| :-- | -----: | ---------------------------: | -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|
||||
| a | 形容词 | 取英语形容词adjective的第1个字母 | [重要/a 步伐/n]NP ，美丽/a ，看似/v 抽象/a ， |
|
||||
| c | 连词 | 取英语连词conjunction的第1个字母。 | 合作/vn 与/c 伙伴/n |
|
||||
| d | 副词 | 取adverb的第2个字母，因其第1个字母已用于形容词。 | 进一步/d 发展/v ， |
|
||||
| e | 叹词 | 取英语叹词exclamation的第1个字母。 | 啊/e ，/w 那/r 金灿灿/z 的/u 麦穗/n ， |
|
||||
| f | 方位词 | 取汉字“方”。 | 军人/n 的/u 眼睛/n 里/f 不/d 是/v 没有/v 风景/n ， |
|
||||
| g | 语素字 | | |
|
||||
| h | 前接成分 | 取英语head的第1个字母。 | 许多/m 非/h 主角/n 人物/n ，办事处/n 的/u “/w 准/h 政府/n ”/w 功能/n 不断/d 加强/v ， |
|
||||
| i | 成语 | 取英语成语idiom的第1个字母。 | 一言一行/i ，义无反顾/i ， |
|
||||
| j | 简称略语 | 取汉字“简”的声母。 | [德/j 外长/n]NP ，文教/j ， |
|
||||
| k | 后接成分 | 后接成分。 | 少年儿童/l 朋友/n 们/k ，身体/n 健康/a 者/k ， |
|
||||
| m | 数词 | 取英语numeral的第3个字母，n，u已有他用。 | 1．数量词组应切分为数词和量词。 三/m 个/q， 10/m 公斤/q， 一/m 盒/q 点心/n ，但少数数量词已是词典的登录单位，则不再切分。 一个/m ， 一些/m ，2. 基数、序数、小数、分数、百分数一律不予切分，为一个切分单位，标注为 m 。一百二十三/m，20万/m， 123.54/m， 一个/m， 第一/m， 第三十五/m， 20%/m， 三分之二/m， 千分之三十/m， 几十/m 人/n， 十几万/m 元/q， 第一百零一/m 个/q ，3. 约数，前加副词、形容词或后加“来、多、左右”等助数词的应予分开。约/d 一百/m 多/m 万/m，仅/d 一百/m 个/q， 四十/m 来/m 个/q，二十/m 余/m 只/q， 十几/m 个/q，三十/m 左右/m ，两个数词相连的及“成百”、“上千”等则不予切分。五六/m 年/q， 七八/m 天/q，十七八/m 岁/q， 成百/m 学生/n，上千/m 人/n， 4．表序关系的“数＋名”结构，应予切分。二/m 连/n ， 三/m 部/n ， |
|
||||
| mq | 数量词 | | |
|
||||
| n | 名词 | 取英语名词noun的第1个字母。 | （参见 动词--v）岗位/n ， 城市/n ， 机会/n ，她/r 是/v 责任/n 编辑/n ， |
|
||||
| nd | 方位名词 | 方位名词（nd），表示位置的相对方向 | 上 下 左 右 前 后 里 外 中 东 西 南 北前边 左面 里头 中间 外部 |
|
||||
| nh | 人名 | 人名（nh），表示人的名称的专有名词 | 华罗庚 阿凡提 诸葛亮 司马相如 松赞干布 卡尔·马克思 |
|
||||
| nhf | 姓 | | |
|
||||
| nhs | 名 | | |
|
||||
| ni | 机构名 | 机构名（ni），表示团体、组织、机构名称的专有名词 | 联合国 教育部 北京大学 中国科学院 |
|
||||
| nl | 处所名词 | 处所名词（nl），表示处所 | 空中 高处 隔壁 门口 附近 边疆 一旁 野外 |
|
||||
| ns | 地名 | 地名（ns），表示地理区域名称的专有名词 | 亚洲 大西洋 地中海 阿尔卑斯山 加拿大中国 北京 浙江 景德镇 呼和浩特 中关村 |
|
||||
| nt | 时间名词 | 时间名词（nt），包括一般所说的时量词 | 年 月 日 分 秒现在 过去 昨天 去年 将来 宋朝 星期一 |
|
||||
| nz | 其他专有名词 | 其他专有名词（nz） | 五粮液 宫爆鸡丁 桑塔纳 |
|
||||
| o | 拟声词 | 取英语拟声词onomatopoeia的第1个字母。 | 哈哈/o 一/m 笑/v ，装载机/n 隆隆/o 推进/v ， |
|
||||
| p | 介词 | 取英语介词prepositional的第1个字母。 | 对/p 子孙后代/n 负责/v ，以/p 煤/n 养/v 农/Ng ，为/p 治理/v 荒山/n 服务/v ， 把/p 青年/n 推/v 上/v 了/u 领导/vn 岗位/n ， |
|
||||
| q | 量词 | 取英语quantity的第1个字母。 | （参见数词m）首/m 批/q ，一/m 年/q ， |
|
||||
| r | 代词 | 取英语代词pronoun的第2个字母，因p已用于介词。 | 单音节代词“本”、“每”、“各”、“诸”后接单音节名词时，和后接的单音节名词合为代词；当后接双音节名词时，应予切分。本报/r， 每人/r， 本社/r， 本/r 地区/n， 各/r 部门/n |
|
||||
| u | 助词 | 取英语助词auxiliary。 | [[俄罗斯/ns 和/c 北约/j]NP-BL 之间/f [战略/n 伙伴/n 关系/n]NP 的/u 建立/vn]NP 填平/v 了/u [[欧洲/ns 安全/a 政治/n]NP 的/u 鸿沟/n]NP |
|
||||
| v | 动词 | 取英语动词verb的第一个字母。 | （参见 名词--n）[[[欧盟/j 扩大/v]S 的/u [历史性/n 决定/n]NP]NP 和/c [北约/j 开放/v]S]NP-BL [为/p [创建/v [一/m 种/q 新/a 的/u 欧洲/ns 安全/a 格局/n]NP]VP-SBI]PP-MD [奠定/v 了/u 基础/n]V-SBI ，， |
|
||||
| vd | 趋向动词 | 趋向动词（vd），表示趋向 | （走）上 （趴）下 （进）来 （回）去（跑）上来 （掉）下去 （提）起来 （扔）过去 |
|
||||
| vl | 联系动词 | 联系动词（vl），表示关系的判断 | 是 |
|
||||
| vu | 能愿动词 | 能愿动词（vu），表示可能、意愿 | 能够 能 应该 可以 可能 情愿 愿意 要 |
|
||||
| w | 标点符号 | | ”/w ：/w |
|
||||
| ws | 非汉字字符串 | 非汉字字符串（ws），如： | HanLP office windows |
|
||||
| x | 非语素字 | 非语素字只是一个符号，字母x通常用于代表未知数、符号。 |
|
||||
@@ -0,0 +1,64 @@
|
||||
<!--
|
||||
# ========================================================================
|
||||
# Copyright 2020 hankcs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be included in all
|
||||
# copies or substantial portions of the Software.
|
||||
# ========================================================================
|
||||
-->
|
||||
|
||||
# ctb
|
||||
|
||||
See also [The Part-Of-Speech Tagging Guidelines for the Penn Chinese Treebank (3.0)](https://repository.upenn.edu/cgi/viewcontent.cgi?article=1039&context=ircs_reports).
|
||||
|
||||
| Tag | Description | Chinese | Chinese Description | Examples |
|
||||
|-----|------------------------------------------------------|---------|---------------------------------------------------------|------------------------|
|
||||
| AD | adverb | 副词 | 副词 | 仍然、很、大大、约 |
|
||||
| AS | aspect marker | 动态助词 | 助词 | 了、着、过 |
|
||||
| BA | `bǎ` in ba-construction | 把字句 | 当“把”、“将”出现在结构“NP0 + BA + NP1+VP”时的词性 | 把、将 |
|
||||
| CC | coordinating conjunction | 并列连接词 | 并列连词 | 与、和、或者、还是 |
|
||||
| CD | cardinal number | 概数词 | 数词或表达数量的词 | 一百、好些、若干 |
|
||||
| CS | subordinating conjunction | 从属连词 | 从属连词 | 如果、那么、就 |
|
||||
| DEC | `de` as a complementizer or a nominalizer | 补语成分“的” | 当“的”或“之”作补语标记或名词化标记时的词性，其结构为：S/VP DEC {NP}，如，喜欢旅游的大学生 | 的、之 |
|
||||
| DEG | `de` as a genitive marker and an associative marker | 属格“的” | 当“的”或“之”作所有格时的词性，其结构为:NP/PP/JJ/DT DEG {NP}， 如，他的车、经济的发展 | 的、之 |
|
||||
| DER | resultative `de`, `de` in V-de const and V-de-R | 表结果的“得” | 当“得”出现在结构“V-得-R”时的词性，如，他跑得很快 | 得 |
|
||||
| DEV | manner `de`, `de` before VP | 表方式的“地” | 当“地”出现在结构“X-地-VP”时的词性，如，高兴地说 | 地 |
|
||||
| DT | determiner | 限定词 | 代冠词，通常用来修饰名词 | 这、那、该、每、各 |
|
||||
| ETC | for words like "etc." | 表示省略 | “等”、“等等”的词性 | 等、等等 |
|
||||
| EM | emoji | 表情符 | 表情符、或称颜文字 | ：） |
|
||||
| FW | foreign words | 外来语 | 外来词 | 卡拉、A型 |
|
||||
| IC | incomplete component | 不完整成分 | 不完整成分，尤指ASR导致的错误 | 好*xin*、那个*ba* |
|
||||
| IJ | interjection | 句首感叹词 | 感叹词，通常出现在句子首部 | 啊 |
|
||||
| JJ | other noun-modifier | 其他名词修饰语 | 形容词 | 共同、新 |
|
||||
| LB | `bèi` in long bei-const | 长句式表被动 | 当“被”、“叫”、“给”出现在结构“NP0 + LB + NP1+ VP”结构时 的词性，如，他被我训了一顿 | 被、叫、给 |
|
||||
| LC | localizer | 方位词 | 方位词以及表示范围的限定词 | 前、旁、到、在内、以来、为止 |
|
||||
| M | measure word | 量词 | 量词 | 个、群、公里 |
|
||||
| MSP | other particle | 其他小品词 | 其他虚词，包括“所”、“以”、“来”和“而”等出现在VP前的词 | 所、以、来、而 |
|
||||
| NN | common noun | 其他名词 | 除专有名词和时间名词外的所有名词 | 桌子、生活、经济 |
|
||||
| NOI | noise that characters are written in the wrong order | 噪声 | 汉字顺序颠倒产生的噪声 | 事/NOI 类/NOI 各/NOI 故/NOI |
|
||||
| NR | proper noun | 专有名词 | 专有名词，通常表示地名、人名、机构名等 | 北京、乔丹、微软 |
|
||||
| NT | temporal noun | 时间名词 | 表示时间概念的名词 | 一月、汉朝、当今 |
|
||||
| OD | ordinal number | 序数词 | 序列词 | 第一百 |
|
||||
| ON | onomatopoeia | 象声词 | 象声词 | 哗哗、呼、咯吱 |
|
||||
| P | preposition e.g., "from" and "to" | 介词 | 介词 | 从、对、根据 |
|
||||
| PN | pronoun | 代词 | 代词，通常用来指代名词 | 我、这些、其、自己 |
|
||||
| PU | punctuation | 标点符号 | 标点符号 | ?、。、； |
|
||||
| SB | `bèi` in short bei-const | 短句式表被动 | 当“被”、“给”出现在NP0 +SB+ VP结果时的词性，如，他被训了 一顿 | 被、叫 |
|
||||
| SP | sentence final particle | 句末助词 | 经常出现在句尾的词 | 吧、呢、啊、啊 |
|
||||
| URL | web address | 网址 | 网址 | www.hankcs.com |
|
||||
| VA | predicative adjective | 表语形容词 | 可以接在“很”后面的形容词谓语 | 雪白、厉害 |
|
||||
| VC | copula, be words | 系动词 | 系动词，表示“是”或“非”概念的动词 | 是、为、非 |
|
||||
| VE | `yǒu` as the main verb | 动词有无 | 表示“有”或“无”概念的动词 | 有、没有、无 |
|
||||
| VV | other verb | 其他动词 | 其他普通动词，包括情态词、控制动词、动作动词、心理动词等等 | 可能、要、走、喜欢 |
|
||||
@@ -0,0 +1,22 @@
|
||||
# Part-of-Speech Tagging
|
||||
|
||||
## Chinese
|
||||
```{toctree}
|
||||
ctb
|
||||
pku
|
||||
863
|
||||
```
|
||||
|
||||
## Japanese
|
||||
```{toctree}
|
||||
npcmj
|
||||
```
|
||||
|
||||
## Multilingual
|
||||
|
||||
```{toctree}
|
||||
ud
|
||||
```
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
<!--
|
||||
# ========================================================================
|
||||
# Copyright 2020 hankcs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be included in all
|
||||
# copies or substantial portions of the Software.
|
||||
# ========================================================================
|
||||
-->
|
||||
|
||||
# NPCMJ
|
||||
|
||||
|
||||
| Tag | Description |
|
||||
|-----------|-----------------------------------|
|
||||
| ADJI | イ-adjective |
|
||||
| ADJI-MD | modal イ-adjective |
|
||||
| ADJN | ナ-adjective |
|
||||
| ADJN-MD | modal ナ-adjective |
|
||||
| ADV | adverb |
|
||||
| AX | auxiliary verb (including copula) |
|
||||
| AXD | auxiliary verb, past tense |
|
||||
| CL | classifier |
|
||||
| CONJ | coordinating conjunction |
|
||||
| D | determiner |
|
||||
| FN | formal noun |
|
||||
| FW | foreign word |
|
||||
| INTJ | interjection |
|
||||
| MD | modal element |
|
||||
| N | noun |
|
||||
| N-MENTION | mentioned expression |
|
||||
| NEG | negation |
|
||||
| NPR | proper noun |
|
||||
| NUM | numeral |
|
||||
| P-COMP | complementizer particle |
|
||||
| P-CONN | conjunctional particle |
|
||||
| P-FINAL | final particle |
|
||||
| P-INTJ | interjectional particle |
|
||||
| P-OPTR | toritate particle |
|
||||
| P-ROLE | role particle |
|
||||
| PASS | direct passive |
|
||||
| PASS2 | indirect passive |
|
||||
| PNL | prenominal |
|
||||
| PRO | pronoun |
|
||||
| PU | punctuation |
|
||||
| PUL | left bracket |
|
||||
| PUR | right bracket |
|
||||
| Q | quantifier |
|
||||
| QUOT | quote |
|
||||
| SYM | symbol |
|
||||
| VB | verb (or verb stem) |
|
||||
| VB0 | light verb |
|
||||
| VB2 | secondary verb |
|
||||
| WADV | indeterminate adverb |
|
||||
| WD | indeterminate determiner |
|
||||
| WNUM | indeterminate numeral |
|
||||
| WPRO | indeterminate pronoun |
|
||||
@@ -0,0 +1,68 @@
|
||||
<!--
|
||||
# ========================================================================
|
||||
# Copyright 2020 hankcs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be included in all
|
||||
# copies or substantial portions of the Software.
|
||||
# ========================================================================
|
||||
-->
|
||||
|
||||
# pku
|
||||
|
||||
| 序号 | 词性 | 名称 | 帮助记忆的诠释 | 例子及注解 |
|
||||
| ---- | ---- | -------- | ------------------------------------------------------ | ------------------------------------------------------------ |
|
||||
| 1 | Ag | 形语素 | 形容词性语素。形容词代码为a，语素代码ｇ前面置以A。 | 绿色/n 似/d 锦/Ag ， |
|
||||
| 2 | a | 形容词 | 取英语形容词adjective的第1个字母 | [重要/a 步伐/n]NP ，美丽/a ，看似/v 抽象/a ， |
|
||||
| 3 | ad | 副形词 | 直接作状语的形容词。形容词代码a和副词代码d并在一起。 | [积极/ad 谋求/v]V-ZZ ，幻象/n 易/ad 逝/Vg ， |
|
||||
| 4 | an | 名形词 | 具有名词功能的形容词。形容词代码a和名词代码n并在一起。 | [外交/n 和/c 安全/an]NP-BL ， |
|
||||
| 5 | Bg | 区别语素 | 区别词性语素。区别词代码为b，语素代码ｇ前面置以B。 | 赤/Ag 橙/Bg 黄/a 绿/a 青/a 蓝/a 紫/a ， |
|
||||
| 6 | b | 区别词 | 取汉字“别”的声母。 | 女/b 司机/n， 金/b 手镯/n， 慢性/b 胃炎/n， 古/b 钱币/n， 副/b 主任/n， 总/b 公司/n 单音节区别词和单音节名词或名语素组合，作为一个词，并标以名词词性n。 |
|
||||
| 7 | c | 连词 | 取英语连词conjunction的第1个字母。 | 合作/vn 与/c 伙伴/n |
|
||||
| 8 | Dg | 副语素 | 副词性语素。副词代码为d，语素代码ｇ前面置以D。 | 了解/v 甚/Dg 深/a ，煞/Dg 是/v 喜人/a ， |
|
||||
| 9 | d | 副词 | 取adverb的第2个字母，因其第1个字母已用于形容词。 | 进一步/d 发展/v ， |
|
||||
| 10 | e | 叹词 | 取英语叹词exclamation的第1个字母。 | 啊/e ，/w 那/r 金灿灿/z 的/u 麦穗/n ， |
|
||||
| 11 | f | 方位词 | 取汉字“方”。 | 军人/n 的/u 眼睛/n 里/f 不/d 是/v 没有/v 风景/n ， |
|
||||
| 12 | h | 前接成分 | 取英语head的第1个字母。 | 许多/m 非/h 主角/n 人物/n ，办事处/n 的/u “/w 准/h 政府/n ”/w 功能/n 不断/d 加强/v ， |
|
||||
| 13 | i | 成语 | 取英语成语idiom的第1个字母。 | 一言一行/i ，义无反顾/i ， |
|
||||
| 14 | j | 简称略语 | 取汉字“简”的声母。 | [德/j 外长/n]NP ，文教/j ， |
|
||||
| 15 | k | 后接成分 | 后接成分。 | 少年儿童/l 朋友/n 们/k ，身体/n 健康/a 者/k ， |
|
||||
| 16 | l | 习用语 | 习用语尚未成为成语，有点“临时性”，取“临”的声母。 | 少年儿童/l 朋友/n 们/k ，落到实处/l ， |
|
||||
| 17 | Mg | 数语素 | 数词性语素。数词代码为m，语素代码ｇ前面置以M。 | 甲/Mg 减下/v 的/u 人/n 让/v 乙/Mg 背上/v ，凡/d “/w 寅/Mg 年/n ”/w 中/f 出生/v 的/u 人/n 生肖/n 都/d 属/v 虎/n ， |
|
||||
| 18 | m | 数词 | 取英语numeral的第3个字母，n，u已有他用。 | 1．数量词组应切分为数词和量词。 三/m 个/q， 10/m 公斤/q， 一/m 盒/q 点心/n ，但少数数量词已是词典的登录单位，则不再切分。 一个/m ， 一些/m ，2. 基数、序数、小数、分数、百分数一律不予切分，为一个切分单位，标注为 m 。一百二十三/m，20万/m， 123.54/m， 一个/m， 第一/m， 第三十五/m， 20%/m， 三分之二/m， 千分之三十/m， 几十/m 人/n， 十几万/m 元/q， 第一百零一/m 个/q ，3. 约数，前加副词、形容词或后加“来、多、左右”等助数词的应予分开。约/d 一百/m 多/m 万/m，仅/d 一百/m 个/q， 四十/m 来/m 个/q，二十/m 余/m 只/q， 十几/m 个/q，三十/m 左右/m ，两个数词相连的及“成百”、“上千”等则不予切分。五六/m 年/q， 七八/m 天/q，十七八/m 岁/q， 成百/m 学生/n，上千/m 人/n， 4．表序关系的“数＋名”结构，应予切分。二/m 连/n ， 三/m 部/n ， |
|
||||
| 19 | Ng | 名语素 | 名词性语素。名词代码为n，语素代码ｇ前面置以N。 | 出/v 过/u 两/m 天/q 差/Ng， 理/v 了/u 一/m 次/q 发/Ng， |
|
||||
| 20 | n | 名词 | 取英语名词noun的第1个字母。 | （参见 动词--v）岗位/n ， 城市/n ， 机会/n ，她/r 是/v 责任/n 编辑/n ， |
|
||||
| 21 | nr | 人名 | 名词代码n和“人(ren)”的声母并在一起。 | 1. 汉族人及与汉族起名方式相同的非汉族人的姓和名单独切分，并分别标注为nr。张/nr 仁伟/nr， 欧阳/nr 修/nr， 阮/nr 志雄/nr， 朴/nr 贞爱/nr汉族人除有单姓和复姓外，还有双姓，即有的女子出嫁后，在原来的姓上加上丈夫的姓。如：陈方安生。这种情况切分、标注为：陈/nr 方/nr 安生/nr；唐姜氏，切分、标注为：唐/nr 姜氏/nr。2. 姓名后的职务、职称或称呼要分开。江/nr 主席/n， 小平/nr 同志/n， 江/nr 总书记/n，张/nr 教授/n， 王/nr 部长/n， 陈/nr 老总/n， 李/nr 大娘/n， 刘/nr 阿姨/n， 龙/nr 姑姑/n3. 对人的简称、尊称等若为两个字，则合为一个切分单位，并标以nr。老张/nr， 大李/nr， 小郝/nr， 郭老/nr， 陈总/nr4. 明显带排行的亲属称谓要切分开，分不清楚的则不切开。三/m 哥/n， 大婶/n， 大/a 女儿/n， 大哥/n， 小弟/n， 老爸/n5. 一些著名作者的或不易区分姓和名的笔名通常作为一个切分单位。鲁迅/nr， 茅盾/nr， 巴金/nr， 三毛/nr， 琼瑶/nr， 白桦/nr6. 外国人或少数民族的译名（包括日本人的姓名）不予切分，标注为nr。克林顿/nr， 叶利钦/nr， 才旦卓玛/nr， 小林多喜二/nr， 北研二/nr，华盛顿/nr， 爱因斯坦/nr有些西方人的姓名中有小圆点，也不分开。卡尔·马克思/nr |
|
||||
| 22 | ns | 地名 | 名词代码n和处所词代码s并在一起。 | 安徽/ns，深圳/ns，杭州/ns，拉萨/ns，哈尔滨/ns， 呼和浩特/ns， 乌鲁木齐/ns，长江/ns，黄海/ns，太平洋/ns， 泰山/ns， 华山/ns，亚洲/ns， 海南岛/ns，太湖/ns，白洋淀/ns， 俄罗斯/ns，哈萨克斯坦/ns，彼得堡/ns， 伏尔加格勒/ns 1. 国名不论长短，作为一个切分单位。中国/ns， 中华人民共和国/ns， 日本国/ns， 美利坚合众国/ns， 美国/ns2. 地名后有“省”、“市”、“县”、“区”、“乡”、“镇”、“村”、“旗”、“州”、“都”、“府”、“道”等单字的行政区划名称时，不切分开，作为一个切分单位。四川省/ns， 天津市/ns，景德镇/ns沙市市/ns， 牡丹江市/ns，正定县/ns，海淀区/ns， 通州区/ns，东升乡/ns， 双桥镇/ns 南化村/ns，华盛顿州/ns，俄亥俄州/ns，东京都/ns， 大阪府/ns，北海道/ns， 长野县/ns，开封府/ns，宣城县/ns3. 地名后的行政区划有两个以上的汉字，则将地名同行政区划名称切开，不过要将地名同行政区划名称用方括号括起来，并标以短语NS。[芜湖/ns 专区/n] NS，[宣城/ns 地区/n]ns，[内蒙古/ns 自治区/n]NS，[深圳/ns 特区/n]NS， [厦门/ns 经济/n 特区/n]NS， [香港/ns 特别/a 行政区/n]NS，[香港/ns 特区/n]NS， [华盛顿/ns 特区/n]NS，4. 地名后有表示地形地貌的一个字的普通名词，如“江、河、山、洋、海、岛、峰、湖”等，不予切分。鸭绿江/ns，亚马逊河/ns， 喜马拉雅山/ns， 珠穆朗玛峰/ns，地中海/ns，大西洋/ns，洞庭湖/ns， 塞普路斯岛/ns 5. 地名后接的表示地形地貌的普通名词若有两个以上汉字，则应切开。然后将地名同该普通名词标成短语NS。[台湾/ns 海峡/n]NS，[华北/ns 平原/n]NS，[帕米尔/ns 高原/n]NS， [南沙/ns 群岛/n]NS，[京东/ns 大/a 峡谷/n]NS [横断/b 山脉/n]NS6．地名后有表示自然区划的一个字的普通名词，如“ 街，路，道，巷，里，町，庄，村，弄，堡”等，不予切分。 中关村/ns，长安街/ns，学院路/ns， 景德镇/ns， 吴家堡/ns， 庞各庄/ns， 三元里/ns，彼得堡/ns， 北菜市巷/ns， 7．地名后接的表示自然区划的普通名词若有两个以上汉字，则应切开。然后将地名同自然区划名词标成短语NS。[米市/ns 大街/n]NS， [蒋家/nz 胡同/n]NS ， [陶然亭/ns 公园/n]NS ， 8． 大小地名相连时的标注方式为：北京市/ns 海淀区/ns 海淀镇/ns [南/f 大街/n]NS [蒋家/nz 胡同/n]NS 24/m 号/q ， |
|
||||
| 23 | nt | 机构团体 | “团”的声母为t，名词代码n和t并在一起。 | （参见2。短语标记说明--NT）联合国/nt，中共中央/nt，国务院/nt， 北京大学/nt1．大多数团体、机构、组织的专有名称一般是短语型的，较长，且含有地名或人名等专名，再组合，标注为短语NT。[中国/ns 计算机/n 学会/n]NT， [香港/ns 钟表业/n 总会/n]NT， [烟台/ns 大学/n]NT， [香港/ns 理工大学/n]NT， [华东/ns 理工大学/n]NT，[合肥/ns 师范/n 学院/n]NT， [北京/ns 图书馆/n]NT， [富士通/nz 株式会社/n]NT， [香山/ns 植物园/n]NT， [安娜/nz 美容院/n]NT，[上海/ns 手表/n 厂/n]NT， [永和/nz 烧饼铺/n]NT，[北京/ns 国安/nz 队/n]NT，2. 对于在国际或中国范围内的知名的唯一的团体、机构、组织的名称即使前面没有专名，也标为nt或NT。联合国/nt，国务院/nt，外交部/nt， 财政部/nt，教育部/nt， 国防部/nt，[世界/n 贸易/n 组织/n]NT， [国家/n 教育/vn 委员会/n]NT，[信息/n 产业/n 部/n]NT，[全国/n 信息/n 技术/n 标准化/vn 委员会/n]NT，[全国/n 总/b 工会/n]NT，[全国/n 人民/n 代表/n 大会/n]NT，美国的“国务院”，其他国家的“外交部、财政部、教育部”，必须在其所属国的国名之后出现时，才联合标注为NT。[美国/ns 国务院/n]NT，[法国/ns 外交部/n]NT，[美/j 国会/n]NT，日本有些政府机构名称很特别，无论是否出现在“日本”国名之后都标为nt。[日本/ns 外务省/nt]NT，[日/j 通产省/nt]NT通产省/nt 3. 前后相连有上下位关系的团体机构组织名称的处理方式如下:[联合国/nt 教科文/j 组织/n]NT， [中国/ns 银行/n 北京/ns 分行/n]NT，[河北省/ns 正定县/ns 西平乐乡/ns 南化村/ns 党支部/n]NT， 当下位名称含有专名（如“北京/ns 分行/n”、“南化村/ns 党支部/n”、“昌平/ns 分校/n”）时，也可脱离前面的上位名称单独标注为NT。[中国/ns 银行/n]NT [北京/ns 分行/n]NT，北京大学/nt [昌平/ns 分校/n]NT，4. 团体、机构、组织名称中用圆括号加注简称时:[宝山/ns 钢铁/n （/w 宝钢/j ）/w 总/b 公司/n]NT，[宝山/ns 钢铁/n 总/b 公司/n]NT，（/w 宝钢/j ）/w |
|
||||
| 24 | nx | 外文字符 | 外文字符。 | A/nx 公司/n ，B/nx 先生/n ，X/nx 君/Ng ，24/m K/nx 镀金/n ，C/nx 是/v 光速/n ，Windows98/nx ，PentiumIV/nx ，I LOVE THIS GAME/nx ，HanLP/nx |
|
||||
| 25 | nz | 其他专名 | “专”的声母的第1个字母为z，名词代码n和z并在一起。 | （参见2。短语标记说明--NZ）除人名、国名、地名、团体、机构、组织以外的其他专有名词都标以nz。满族/nz，俄罗斯族/nz，汉语/nz，罗马利亚语/nz， 捷克语/nz，中文/nz， 英文/nz， 满人/nz， 哈萨克人/nz， 诺贝尔奖/nz， 茅盾奖/nz， 1.包含专有名称（或简称）的交通线，标以nz；短语型的，标为NZ。津浦路/nz， 石太线/nz， [京/j 九/j 铁路/n]NZ， [京/j 津/j 高速/b 公路/n]NZ， 2. 历史上重要事件、运动等专有名称一般是短语型的，按短语型专有名称处理，标以NZ。[卢沟桥/ns 事件/n]NZ， [西安/ns 事变/n]NZ，[五四/t 运动/n]NZ， [明治/nz 维新/n]NZ，[甲午/t 战争/n]NZ，3.专有名称后接多音节的名词，如“语言”、“文学”、“文化”、“方式”、“精神”等，失去专指性，则应分开。欧洲/ns 语言/n， 法国/ns 文学/n， 西方/ns 文化/n， 贝多芬/nr 交响乐/n， 雷锋/nr 精神/n， 美国/ns 方式/n，日本/ns 料理/n， 宋朝/t 古董/n 4. 商标（包括专名及后接的“牌”、“型”等）是专指的，标以nz，但其后所接的商品仍标以普通名词n。康师傅/nr 方便面/n， 中华牌/nz 香烟/n， 牡丹III型/nz 电视机/n， 联想/nz 电脑/n， 鳄鱼/nz 衬衣/n， 耐克/nz 鞋/n5. 以序号命名的名称一般不认为是专有名称。2/m 号/q 国道/n ，十一/m 届/q 三中全会/j如果前面有专名，合起来作为短语型专名。[中国/ns 101/m 国道/n]NZ， [中共/j 十一/m 届/q 三中全会/j]NZ，6. 书、报、杂志、文档、报告、协议、合同等的名称通常有书名号加以标识，不作为专有名词。由于这些名字往往较长，名字本身按常规处理。《/w 宁波/ns 日报/n 》/w ，《/w 鲁迅/nr 全集/n 》/w，中华/nz 读书/vn 报/n， 杜甫/nr 诗选/n，少数书名、报刊名等专有名称，则不切分。红楼梦/nz， 人民日报/nz，儒林外史/nz 7. 当有些专名无法分辨它们是人名还是地名或机构名时，暂标以nz。[巴黎/ns 贝尔希/nz 体育馆/n]NT，其中“贝尔希”只好暂标为nz。 |
|
||||
| 26 | o | 拟声词 | 取英语拟声词onomatopoeia的第1个字母。 | 哈哈/o 一/m 笑/v ，装载机/n 隆隆/o 推进/v ， |
|
||||
| 27 | p | 介词 | 取英语介词prepositional的第1个字母。 | 对/p 子孙后代/n 负责/v ，以/p 煤/n 养/v 农/Ng ，为/p 治理/v 荒山/n 服务/v ， 把/p 青年/n 推/v 上/v 了/u 领导/vn 岗位/n ， |
|
||||
| 28 | q | 量词 | 取英语quantity的第1个字母。 | （参见数词m）首/m 批/q ，一/m 年/q ， |
|
||||
| 29 | Rg | 代语素 | 代词性语素。代词代码为r，在语素的代码g前面置以R。 | 读者/n 就/d 是/v 这/r 两/m 棵/q 小树/n 扎根/v 于/p 斯/Rg 、/w 成长/v 于/p 斯/Rg 的/u 肥田/n 沃土/n ， |
|
||||
| 30 | r | 代词 | 取英语代词pronoun的第2个字母，因p已用于介词。 | 单音节代词“本”、“每”、“各”、“诸”后接单音节名词时，和后接的单音节名词合为代词；当后接双音节名词时，应予切分。本报/r， 每人/r， 本社/r， 本/r 地区/n， 各/r 部门/n |
|
||||
| 31 | s | 处所词 | 取英语space的第1个字母。 | 家里/s 的/u 电脑/n 都/d 联通/v 了/u 国际/n 互联网/n ，西部/s 交通/n 咽喉/n ， |
|
||||
| 32 | Tg | 时语素 | 时间词性语素。时间词代码为t，在语素的代码g前面置以T。 | ３日/t 晚/Tg 在/p 总统府/n 发表/v 声明/n ，尊重/v 现/Tg 执政/vn 当局/n 的/u 权威/n ， |
|
||||
| 33 | t | 时间词 | 取英语time的第1个字母。 | 1. 年月日时分秒，按年、月、日、时、分、秒切分，标注为t 。1997年/t 3月/t 19日/t 下午/t 2时/t 18分/t若数字后无表示时间的“年、月、日、时、分、秒”等的标为数词m。1998/m 中文/n 信息/n 处理/vn 国际/n 会议/n 2. 历史朝代的名称虽然有专有名词的性质，仍标注为t。西周/t， 秦朝/t， 东汉/t， 南北朝/t， 清代/t“牛年、虎年”等一律不予切分，标注为：牛年/t， 虎年/t， 甲午年/t， 甲午/t 战争/n， 庚子/t 赔款/n， 戊戌/t 变法/n |
|
||||
| 34 | u | 助词 | 取英语助词auxiliary。 | [[俄罗斯/ns 和/c 北约/j]NP-BL 之间/f [战略/n 伙伴/n 关系/n]NP 的/u 建立/vn]NP 填平/v 了/u [[欧洲/ns 安全/a 政治/n]NP 的/u 鸿沟/n]NP |
|
||||
| 35 | Vg | 动语素 | 动词性语素。动词代码为v。在语素的代码g前面置以V。 | 洗/v 了/u 一个/m 舒舒服服/z 的/u 澡/Vg |
|
||||
| 36 | v | 动词 | 取英语动词verb的第一个字母。 | （参见 名词--n）[[[欧盟/j 扩大/v]S 的/u [历史性/n 决定/n]NP]NP 和/c [北约/j 开放/v]S]NP-BL [为/p [创建/v [一/m 种/q 新/a 的/u 欧洲/ns 安全/a 格局/n]NP]VP-SBI]PP-MD [奠定/v 了/u 基础/n]V-SBI ，， |
|
||||
| 37 | vd | 副动词 | 直接作状语的动词。动词和副词的代码并在一起。 | 形势/n 会/v 持续/vd 好转/v ，认为/v 是/v 电话局/n 收/v 错/vd 了/u 费/n ， |
|
||||
| 38 | vn | 名动词 | 指具有名词功能的动词。动词和名词的代码并在一起。 | 引起/v 人们/n 的/u 关注/vn 和/c 思考/vn ，收费/vn 电话/n 的/u 号码/n ， |
|
||||
| 39 | w | 标点符号 | | ”/w ：/w |
|
||||
| 40 | x | 非语素字 | 非语素字只是一个符号，字母x通常用于代表未知数、符号。 | |
|
||||
| 41 | Yg | 语气语素 | 语气词性语素。语气词代码为y。在语素的代码g前面置以Y。 | 唯/d 大力/d 者/k 能/v 致/v 之/u 耳/Yg |
|
||||
| 42 | y | 语气词 | 取汉字“语”的声母。 | 会/v 泄露/v 用户/n 隐私/n 吗/y ，又/d 何在/v 呢/y ？ |
|
||||
| 43 | z | 状态词 | 取汉字“状”的声母的前一个字母。 | 取得/v 扎扎实实/z 的/u 突破性/n 进展/vn ，四季/n 常青/z 的/u 热带/n 树木/n ，短短/z 几/m 年/q 间， |
|
||||
@@ -0,0 +1,44 @@
|
||||
<!--
|
||||
# ========================================================================
|
||||
# Copyright 2020 hankcs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be included in all
|
||||
# copies or substantial portions of the Software.
|
||||
# ========================================================================
|
||||
-->
|
||||
|
||||
# Universal Dependencies
|
||||
|
||||
See also [Universal Dependencies](https://universaldependencies.org/u/pos/).
|
||||
|
||||
| Tag | Description |
|
||||
|------------|----------------------------------------------|
|
||||
| ADJ | adjective |
|
||||
| ADP | adposition |
|
||||
| ADV | adverb |
|
||||
| AUX | auxiliary |
|
||||
| CCONJ | coordinating conjunction |
|
||||
| DET | determiner |
|
||||
| INTJ | interjection |
|
||||
| NOUN | noun |
|
||||
| NUM | numeral |
|
||||
| PART | particle |
|
||||
| PRON | pronoun |
|
||||
| PROPN | proper noun |
|
||||
| PUNCT | punctuation |
|
||||
| SCONJ | subordinating conjunction |
|
||||
| SYM | symbol |
|
||||
| VERB | verb |
|
||||
| X | other |
|
||||
@@ -0,0 +1,3 @@
|
||||
# The reduction of Minimal Recursion Semantics
|
||||
|
||||
Please refer to [Minimal Recursion Semantics An Introduction](https://www.cl.cam.ac.uk/~aac10/papers/mrs.pdf).
|
||||
@@ -0,0 +1,16 @@
|
||||
# Semantic Dependency Parsing
|
||||
|
||||
## Chinese
|
||||
|
||||
```{toctree}
|
||||
semeval16
|
||||
```
|
||||
|
||||
## English
|
||||
|
||||
```{toctree}
|
||||
dm
|
||||
pas
|
||||
psd
|
||||
```
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
# Predicate-Argument Structures
|
||||
|
||||
Please refer to [Probabilistic disambiguation models for wide-coverage HPSG parsing](https://www.aclweb.org/anthology/P05-1011.pdf).
|
||||
@@ -0,0 +1,3 @@
|
||||
# Prague Czech-English Dependency Treebank
|
||||
|
||||
Please refer to [Prague Czech-English Dependency Treebank](http://ufal.mff.cuni.cz/pcedt2.0/en/index.html).
|
||||
@@ -0,0 +1,166 @@
|
||||
<!--
|
||||
# ========================================================================
|
||||
# Copyright 2020 hankcs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be included in all
|
||||
# copies or substantial portions of the Software.
|
||||
# ========================================================================
|
||||
-->
|
||||
|
||||
# SemEval2016
|
||||
|
||||
## CSDP
|
||||
|
||||
SemEval2016 adopts the CSDP guideline listed as follows.
|
||||
|
||||
### 语义关系标注标签集
|
||||
|
||||
| 分类 | | | |
|
||||
| ------------ | ------------ | --------------- | ------------------------------------------------------------ |
|
||||
| 语义周边角色 | 主体角色 | 施事AGT； | 施事Agt；感事Aft |
|
||||
| | | 当事EXP； | 当事Exp；领事Poss |
|
||||
| | 客体角色 | 受事PAT； | 受事Pat |
|
||||
| | | 客事CONT； | 客事Cont；成事Prod；结局Cons |
|
||||
| | | 涉事DATV； | 涉事Datv；比较Comp；源事Orig |
|
||||
| | | 系事LINK； | 类事Clas；属事Belg |
|
||||
| | 情境角色 | 工具TOOL； | 工具Tool |
|
||||
| | | 材料MATL； | 材料Matl |
|
||||
| | | 方式MANN； | 方式Mann；依据Accd |
|
||||
| | | 范围SCO； | 范围Sco |
|
||||
| | | 缘由REAS； | 缘故Reas；意图Int |
|
||||
| | | 时间TIME； | 时间Time；时间起点Tini；时间终点Tfin；时段Tdur；时距Trang |
|
||||
| | | 空间LOC； | 空间Loc；原处所Lini；终处所Lfin；通过处所Lthru；趋向Dir |
|
||||
| | | 度量MEAS； | 数量Quan；起始量Nini；终止量Nfin；数量短语Qp；频率Freq；顺序Seq；变化量Nvar |
|
||||
| | | 状态STAT； | 状态Stat；起始状态Sini；终止状态Sfin；历经状态Sproc |
|
||||
| | | 修饰FEAT； | 描写Desc；宿主Host；名词修饰语Nmod；时间修饰语Tmod |
|
||||
| 语义结构关系 | 反关系 | 反施事rAGT； | 反施事rAgt；反感事rAft |
|
||||
| | | 反当事rEXP。 | 反当事rExp；反领事rPoss |
|
||||
| | | 反受事rPAT； | 反受事rPat |
|
||||
| | | 反客事rCONT； | 反客事rCont；反成事rProd；反结局rCons |
|
||||
| | | 反涉事rDATV； | 反涉事rDatv；反比较rComp；反源事rOrig |
|
||||
| | | 反系事rLINK。 | 反类事rClas；反属事rBelg |
|
||||
| | | 反工具rTOOL； | 反工具rTool |
|
||||
| | | 反材料rMATL； | 反材料rMatl |
|
||||
| | | 反方式RMANN； | 反方式rMann；反依据rAccd |
|
||||
| | | 反范围rSCO； | 反范围rSco |
|
||||
| | | 反缘由rREAS； | 反缘故rReas；反意图rInt |
|
||||
| | | 反时间rTIME； | 反时间rTime；反时间起点rTini；反时间终点rTfin；反时段rTdur；反时距rTrang |
|
||||
| | | 反空间rLOC； | 反空间rLoc；反原处所rLini；反终处所rLfin；反通过处所rLthru；反趋向rDir |
|
||||
| | | 反度量rMEAS； | 反数量rQuan；反起始量rNini；反终止量rNfin；反数量短语rQp；反频率rFreq；反顺序rSeq；反变化量rNvar |
|
||||
| | | 反状态rSTAT； | 反状态rStat；反起始状态rSini；反终止状态rSfin；反历经状态rSproc |
|
||||
| | | 反修饰rFEAT； | 反描写rDesc；反宿主rHost; 反名词修饰语rNmod; 反时间修饰语rTmod |
|
||||
| | 嵌套事件关系 | 嵌套施事dAGT； | 嵌套施事dAgt；嵌套感事dAft |
|
||||
| | | 嵌套当事dEXP。 | 嵌套当事dExp；嵌套领事dPoss |
|
||||
| | | 嵌套受事dPAT； | 嵌套受事dPat |
|
||||
| | | 嵌套客事dCONT； | 嵌套客事dCont；嵌套成事dProd；嵌套结局dCons |
|
||||
| | | 嵌套涉事dDATV； | 嵌套涉事dDatv；嵌套比较dComp；嵌套源事dOrig |
|
||||
| | | 嵌套系事dLINK。 | 嵌套类事dClas；嵌套属事dBelg |
|
||||
| | | 嵌套工具dTOOL； | 嵌套工具dTool |
|
||||
| | | 嵌套材料dMATL； | 嵌套材料dMatl |
|
||||
| | | 嵌套方式dMANN； | 嵌套方式dMann；嵌套依据dAccd |
|
||||
| | | 嵌套范围dSCO； | 嵌套范围dSco |
|
||||
| | | 嵌套缘由dREAS； | 嵌套缘故dReas；嵌套意图dInt |
|
||||
| | | 嵌套时间dTIME； | 嵌套时间dTime；嵌套时间起点dTini；嵌套时间终点dTfin；嵌套时段dTdur；嵌套时距dTrang |
|
||||
| | | 嵌套空间dLOC； | 嵌套空间dLoc；嵌套原处所dLini；嵌套终处所dLfin；嵌套通过处所dLthru；嵌套趋向dDir |
|
||||
| | | 嵌套度量dMEAS； | 嵌套数量dQuan；嵌套起始量dNini；嵌套终止量dNfin；嵌套数量短语dQp；嵌套频率dFreq；嵌套顺序dSeq；嵌套变化量dNvar |
|
||||
| | | 嵌套状态dSTAT； | 嵌套状态dStat；嵌套起始状态dSini；嵌套终止状态dSfin；嵌套历经状态dSproc |
|
||||
| | | 嵌套修饰dFEAT； | 嵌套描写dDesc；嵌套宿主dHost; 嵌套名词修饰语dNmod; 嵌套时间修饰语dTmod |
|
||||
| | 事件关系 | 并列关系eCOO； | 并列eCoo；等同eEqu；分叙eRect；选择eSelt;割舍eAban；选取ePref；总括eSum |
|
||||
| | | 先行关系ePREC； | 先行ePrec；原因eCau；条件eCond；假设eSupp；手段eMetd；让步eConc |
|
||||
| | | 后继关系eSUCC； | 后继eSucc；递进eProg；转折 eAdvt；目的ePurp；结果eResu；推论eInf |
|
||||
| 语义依附标记 | 标点标记 | 标点标记mPUNC； | 标点标记mPunc |
|
||||
| | 依附标记 | 否定标记mNEG； | 否定标记mNeg |
|
||||
| | | 关系标记mRELA； | 连词标记mConj；介词标记mPrep |
|
||||
| | | 依附标记mDEPD； | 语气标记mTone；时间标记mTime;范围标记mRang；情态标记mMod； 频率标记mFreq；程度标记mDegr；趋向标记mDir；的字标记mAux； 多数标记mMaj；插入语标记mPars；离合标记mSepa；实词虚化标记mVain 重复标记mRept |
|
||||
|
||||
## SemEval2016
|
||||
|
||||
The following table is a subset of CSDP but offers some examples to illustrate the idea.
|
||||
|
||||
| 关系类型 | Tag | Description | Example |
|
||||
|--------|---------------|--------------------|-----------------------------|
|
||||
| 施事关系 | Agt | Agent | 我送她一束花 (我 <– 送) |
|
||||
| 当事关系 | Exp | Experiencer | 我跑得快 (跑 –> 我) |
|
||||
| 感事关系 | Aft | Affection | 我思念家乡 (思念 –> 我) |
|
||||
| 领事关系 | Poss | Possessor | 他有一本好读 (他 <– 有) |
|
||||
| 受事关系 | Pat | Patient | 他打了小明 (打 –> 小明) |
|
||||
| 客事关系 | Cont | Content | 他听到鞭炮声 (听 –> 鞭炮声) |
|
||||
| 成事关系 | Prod | Product | 他写了本小说 (写 –> 小说) |
|
||||
| 源事关系 | Orig | Origin | 我军缴获敌人四辆坦克 (缴获 –> 坦克) |
|
||||
| 涉事关系 | Datv | Dative | 他告诉我个秘密 ( 告诉 –> 我 ) |
|
||||
| 比较角色 | Comp | Comitative | 他成绩比我好 (他 –> 我) |
|
||||
| 属事角色 | Belg | Belongings | 老赵有俩女儿 (老赵 <– 有) |
|
||||
| 类事角色 | Clas | Classification | 他是中学生 (是 –> 中学生) |
|
||||
| 依据角色 | Accd | According | 本庭依法宣判 (依法 <– 宣判) |
|
||||
| 缘故角色 | Reas | Reason | 他在愁女儿婚事 (愁 –> 婚事) |
|
||||
| 意图角色 | Int | Intention | 为了金牌他拼命努力 (金牌 <– 努力) |
|
||||
| 结局角色 | Cons | Consequence | 他跑了满头大汗 (跑 –> 满头大汗) |
|
||||
| 方式角色 | Mann | Manner | 球慢慢滚进空门 (慢慢 <– 滚) |
|
||||
| 工具角色 | Tool | Tool | 她用砂锅熬粥 (砂锅 <– 熬粥) |
|
||||
| 材料角色 | Malt | Material | 她用小米熬粥 (小米 <– 熬粥) |
|
||||
| 时间角色 | Time | Time | 唐朝有个李白 (唐朝 <– 有) |
|
||||
| 空间角色 | Loc | Location | 这房子朝南 (朝 –> 南) |
|
||||
| 历程角色 | Proc | Process | 火车正在过长江大桥 (过 –> 大桥) |
|
||||
| 趋向角色 | Dir | Direction | 部队奔向南方 (奔 –> 南) |
|
||||
| 范围角色 | Sco | Scope | 产品应该比质量 (比 –> 质量) |
|
||||
| 数量角色 | Quan | Quantity | 一年有365天 (有 –> 天) |
|
||||
| 数量数组 | Qp | Quantity-phrase | 三本书 (三 –> 本) |
|
||||
| 频率角色 | Freq | Frequency | 他每天看书 (每天 <– 看) |
|
||||
| 顺序角色 | Seq | Sequence | 他跑第一 (跑 –> 第一) |
|
||||
| 描写角色 | Desc(Feat) | Description | 他长得胖 (长 –> 胖) |
|
||||
| 宿主角色 | Host | Host | 住房面积 (住房 <– 面积) |
|
||||
| 名字修饰角色 | Nmod | Name-modifier | 果戈里大街 (果戈里 <– 大街) |
|
||||
| 时间修饰角色 | Tmod | Time-modifier | 星期一上午 (星期一 <– 上午) |
|
||||
| 反角色 | r + main role | | 打篮球的小姑娘 (打篮球 <– 姑娘) |
|
||||
| 嵌套角色 | d + main role | | 爷爷看见孙子在跑 (看见 –> 跑) |
|
||||
| 并列关系 | eCoo | event Coordination | 我喜欢唱歌和跳舞 (唱歌 –> 跳舞) |
|
||||
| 选择关系 | eSelt | event Selection | 您是喝茶还是喝咖啡 (茶 –> 咖啡) |
|
||||
| 等同关系 | eEqu | event Equivalent | 他们三个人一起走 (他们 –> 三个人) |
|
||||
| 先行关系 | ePrec | event Precedent | 首先，先 |
|
||||
| 顺承关系 | eSucc | event Successor | 随后，然后 |
|
||||
| 递进关系 | eProg | event Progression | 况且，并且 |
|
||||
| 转折关系 | eAdvt | event adversative | 却，然而 |
|
||||
| 原因关系 | eCau | event Cause | 因为，既然 |
|
||||
| 结果关系 | eResu | event Result | 因此，以致 |
|
||||
| 推论关系 | eInf | event Inference | 才，则 |
|
||||
| 条件关系 | eCond | event Condition | 只要，除非 |
|
||||
| 假设关系 | eSupp | event Supposition | 如果，要是 |
|
||||
| 让步关系 | eConc | event Concession | 纵使，哪怕 |
|
||||
| 手段关系 | eMetd | event Method | |
|
||||
| 目的关系 | ePurp | event Purpose | 为了，以便 |
|
||||
| 割舍关系 | eAban | event Abandonment | 与其，也不 |
|
||||
| 选取关系 | ePref | event Preference | 不如，宁愿 |
|
||||
| 总括关系 | eSum | event Summary | 总而言之 |
|
||||
| 分叙关系 | eRect | event Recount | 例如，比方说 |
|
||||
| 连词标记 | mConj | Conjunction | 和，或 |
|
||||
| 的字标记 | mAux | Auxiliary | 的，地，得 |
|
||||
| 介词标记 | mPrep | Preposition | 把，被 |
|
||||
| 语气标记 | mTone | Tone | 吗，呢 |
|
||||
| 时间标记 | mTime | Time | 才，曾经 |
|
||||
| 范围标记 | mRang | Range | 都，到处 |
|
||||
| 程度标记 | mDegr | Degree | 很，稍微 |
|
||||
| 频率标记 | mFreq | Frequency Marker | 再，常常 |
|
||||
| 趋向标记 | mDir | Direction Marker | 上去，下来 |
|
||||
| 插入语标记 | mPars | Parenthesis Marker | 总的来说，众所周知 |
|
||||
| 否定标记 | mNeg | Negation Marker | 不，没，未 |
|
||||
| 情态标记 | mMod | Modal Marker | 幸亏，会，能 |
|
||||
| 标点标记 | mPunc | Punctuation Marker | ，。！ |
|
||||
| 重复标记 | mPept | Repetition Marker | 走啊走 (走 –> 走) |
|
||||
| 多数标记 | mMaj | Majority Marker | 们，等 |
|
||||
| 实词虚化标记 | mVain | Vain Marker | |
|
||||
| 离合标记 | mSepa | Seperation Marker | 吃了个饭 (吃 –> 饭) 洗了个澡 (洗 –> 澡) |
|
||||
| 根节点 | Root | Root | 全句核心节点 |
|
||||
|
||||
See also [SemEval-2016 Task 9](https://www.hankcs.com/nlp/sdp-corpus.html) and [CSDP](https://csdp-doc.readthedocs.io/zh_CN/latest/%E9%99%84%E5%BD%95/).
|
||||
@@ -0,0 +1,62 @@
|
||||
<!--
|
||||
# ========================================================================
|
||||
# Copyright 2020 hankcs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be included in all
|
||||
# copies or substantial portions of the Software.
|
||||
# ========================================================================
|
||||
-->
|
||||
|
||||
# Chinese Proposition Bank
|
||||
|
||||
| | 标签 | 角色 | 例子 |
|
||||
|------|----------|-------|-------------------------|
|
||||
| 中心角色 | ARG0 | 施事者 | (ARG0中国政府)提供援助 |
|
||||
| | ARG1 | 受事者 | 中国政府提供(ARG1援助) |
|
||||
| | ARG2 | 依谓词而定 | 失业率控制(ARG2在百分之十内) |
|
||||
| | ARG3 | 依谓词而定 | (ARG3从城市)扩大到农村 |
|
||||
| | ARG4 | 依谓词而定 | 提高(ARG4百分之二十) |
|
||||
| 附属角色 | ARGM-ADV | 状语 | (ARGM-ADV共同)承担 |
|
||||
| | ARGM-BNF | 受益者 | (ARGM-BNF为其他国家)进行融资 |
|
||||
| | ARGM-CND | 条件 | (ARGM-CND如果成功)，他就留下 |
|
||||
| | ARGM-DIR | 方向 | (ARGM-DIR向和平)迈出一大步 |
|
||||
| | ARGM-EXT | 范围 | 在北京逗留(ARGM-EXT两天) |
|
||||
| | ARGM-FRQ | 频率 | 每半年执行(ARGM-FRQ一次) |
|
||||
| | ARGM-LOC | 地点、位置 | (ARGM-LOC在机场)被捕获 |
|
||||
| | ARGM-MNR | 方式 | (ARGM-MNR以中英文)发行 |
|
||||
| | ARGM-PRP | 目的或原因 | (ARGM-PRP由于危机)而破产 |
|
||||
| | ARGM-TMP | 时间 | 公司(ARGM-TMP去年)成立 |
|
||||
| | ARGM-TPC | 主题 | (ARGM-TPC稳定政策)，核心是... |
|
||||
| | ARGM-DIS | 话语标记 | (ARGM-DIS因此)，他感到不公 |
|
||||
| | ARGM-CRD | 并列论元 | (ARGM-CRD与台湾)非正式接触 |
|
||||
| | ARGM-PRD | 次谓词 | 指控廉政公署五人(ARGM-PRD接受贿赂) |
|
||||
|
||||
|
||||
```{note}
|
||||
Although ARG0 and ARG1 share general definitions across all predicates, word sense disambiguation is required to find
|
||||
the coresponding definition of semantic roles. Given the word sense of `变化`, say `变化-2`,
|
||||
[its second frameset](http://verbs.colorado.edu/chinese/cpb/html_frames/0183-bian-hua.html) can
|
||||
be found which defines the following 2 arguments:
|
||||
|
||||
1. ARG0: agent/cause
|
||||
2. ARG1: entity arg0 changes
|
||||
|
||||
These definitions are different from that of frameset `变化-1`:
|
||||
|
||||
1. ARG0: entity undergoing change
|
||||
|
||||
Sometimes, the number of arguments and definitions can vary a lot across framesets.
|
||||
In summary, word sense disambiguation is essential if SRL is to be used to best effect in practical applications
|
||||
```
|
||||
@@ -0,0 +1,12 @@
|
||||
# Semantic Role Labeling
|
||||
|
||||
## Chinese
|
||||
```{toctree}
|
||||
cpb
|
||||
```
|
||||
|
||||
## English
|
||||
```{toctree}
|
||||
propbank
|
||||
```
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
<!--
|
||||
# ========================================================================
|
||||
# Copyright 2020 hankcs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be included in all
|
||||
# copies or substantial portions of the Software.
|
||||
# ========================================================================
|
||||
-->
|
||||
|
||||
# English PropBank
|
||||
|
||||
| Role | Description |
|
||||
|------|----------------------------------------|
|
||||
| ARG0 | agent |
|
||||
| ARG1 | patient |
|
||||
| ARG2 | instrument, benefactive, attribute |
|
||||
| ARG3 | starting point, benefactive, attribute |
|
||||
| ARG4 | ending point |
|
||||
| ARGM | modifier |
|
||||
| COM | Comitative |
|
||||
| LOC | Locative |
|
||||
| DIR | Directional |
|
||||
| GOL | Goal |
|
||||
| MNR | Manner |
|
||||
| TMP | Temporal |
|
||||
| EXT | Extent |
|
||||
| REC | Reciprocals |
|
||||
| PRD | Secondary Predication |
|
||||
| PRP | Purpose |
|
||||
| CAU | Cause |
|
||||
| DIS | Discourse |
|
||||
| ADV | Adverbials |
|
||||
| ADJ | Adjectival |
|
||||
| MOD | Modal |
|
||||
| NEG | Negation |
|
||||
| DSP | Direct Speech |
|
||||
| LVB | Light Verb |
|
||||
| CXN | Construction |
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,7 @@
|
||||
# Tokenization
|
||||
|
||||
## Chinese
|
||||
```{toctree}
|
||||
ctb
|
||||
msr
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,11 @@
|
||||
.. _api/configurable:
|
||||
|
||||
configurable
|
||||
====================
|
||||
|
||||
|
||||
.. autoclass:: hanlp_common.configurable.Configurable
|
||||
:members:
|
||||
|
||||
.. autoclass:: hanlp_common.configurable.AutoConfigurable
|
||||
:members:
|
||||
@@ -0,0 +1,14 @@
|
||||
.. _api/conll:
|
||||
|
||||
conll
|
||||
====================
|
||||
|
||||
|
||||
.. autoclass:: hanlp_common.conll.CoNLLWord
|
||||
:members:
|
||||
|
||||
.. autoclass:: hanlp_common.conll.CoNLLUWord
|
||||
:members:
|
||||
|
||||
.. autoclass:: hanlp_common.conll.CoNLLSentence
|
||||
:members:
|
||||
@@ -0,0 +1,6 @@
|
||||
constant
|
||||
====================
|
||||
|
||||
|
||||
.. automodule:: hanlp_common.constant
|
||||
:members:
|
||||
@@ -0,0 +1,9 @@
|
||||
.. _api/document:
|
||||
|
||||
document
|
||||
====================
|
||||
|
||||
.. currentmodule:: hanlp_common
|
||||
|
||||
.. autoclass:: hanlp_common.document.Document
|
||||
:members:
|
||||
@@ -0,0 +1,11 @@
|
||||
# hanlp_common
|
||||
|
||||
Common APIs shared between `hanlp` and `restful`.
|
||||
|
||||
```{toctree}
|
||||
document
|
||||
conll
|
||||
configurable
|
||||
constant
|
||||
```
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
component
|
||||
=================
|
||||
|
||||
.. currentmodule:: hanlp.common
|
||||
|
||||
.. autoclass:: hanlp.common.component.Component
|
||||
:members:
|
||||
@@ -0,0 +1,64 @@
|
||||
# dataset
|
||||
|
||||
This module provides base definition for datasets, dataloaders and samplers.
|
||||
|
||||
## datasets
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.common
|
||||
|
||||
.. autoclass:: hanlp.common.dataset.Transformable
|
||||
:members:
|
||||
|
||||
.. autoclass:: hanlp.common.dataset.TransformableDataset
|
||||
:members:
|
||||
:special-members:
|
||||
:exclude-members: __init__, __repr__
|
||||
```
|
||||
|
||||
## dataloaders
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.common
|
||||
|
||||
.. autoclass:: hanlp.common.dataset.PadSequenceDataLoader
|
||||
:members:
|
||||
:special-members:
|
||||
:exclude-members: __init__, __repr__
|
||||
|
||||
.. autoclass:: hanlp.common.dataset.PrefetchDataLoader
|
||||
:members:
|
||||
:special-members:
|
||||
:exclude-members: __init__, __repr__
|
||||
```
|
||||
|
||||
## samplers
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.common
|
||||
|
||||
.. autoclass:: hanlp.common.dataset.BucketSampler
|
||||
:members:
|
||||
|
||||
.. autoclass:: hanlp.common.dataset.KMeansSampler
|
||||
:members:
|
||||
|
||||
.. autoclass:: hanlp.common.dataset.SortingSampler
|
||||
:members:
|
||||
```
|
||||
|
||||
## sampler builders
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.common
|
||||
|
||||
.. autoclass:: hanlp.common.dataset.SamplerBuilder
|
||||
:members:
|
||||
|
||||
.. autoclass:: hanlp.common.dataset.SortingSamplerBuilder
|
||||
:members:
|
||||
|
||||
.. autoclass:: hanlp.common.dataset.KMeansSamplerBuilder
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,13 @@
|
||||
# common
|
||||
|
||||
Common base classes.
|
||||
|
||||
```{toctree}
|
||||
structure
|
||||
vocab
|
||||
transform
|
||||
dataset
|
||||
component
|
||||
torch_component
|
||||
```
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
# structure
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.common
|
||||
|
||||
.. autoclass:: hanlp.common.structure.ConfigTracker
|
||||
:members:
|
||||
|
||||
.. autoclass:: hanlp.common.structure.History
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,9 @@
|
||||
# torch_component
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.common.torch_component
|
||||
|
||||
.. autoclass:: hanlp.common.torch_component.TorchComponent
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,9 @@
|
||||
# transform
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.common
|
||||
|
||||
.. autoclass:: hanlp.common.transform.VocabDict
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,11 @@
|
||||
# vocab
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.common
|
||||
|
||||
.. autoclass:: hanlp.common.transform.Vocab
|
||||
:members:
|
||||
:special-members:
|
||||
:exclude-members: __init__, __repr__, __call__, __str__
|
||||
|
||||
```
|
||||
@@ -0,0 +1,9 @@
|
||||
# classifiers
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.classifiers
|
||||
|
||||
.. autoclass:: hanlp.components.classifiers.transformer_classifier.TransformerClassifier
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,9 @@
|
||||
# eos
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.eos
|
||||
|
||||
.. autoclass:: hanlp.components.eos.ngram.NgramSentenceBoundaryDetector
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,18 @@
|
||||
# components
|
||||
|
||||
NLP components.
|
||||
|
||||
```{toctree}
|
||||
mtl/index
|
||||
classifiers
|
||||
eos
|
||||
tokenizers/index
|
||||
lemmatizer
|
||||
taggers/index
|
||||
ner/index
|
||||
parsers/index
|
||||
srl/index
|
||||
pipeline
|
||||
sts
|
||||
```
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
# lemmatizer
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.lemmatizer
|
||||
|
||||
.. autoclass:: TransformerLemmatizer
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,9 @@
|
||||
# mtl
|
||||
|
||||
Multi-Task Learning (MTL) framework.
|
||||
|
||||
```{toctree}
|
||||
mtl
|
||||
tasks/index
|
||||
```
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
# MultiTaskLearning
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.mtl
|
||||
|
||||
.. autoclass:: hanlp.components.mtl.multi_task_learning.MultiTaskLearning
|
||||
:members:
|
||||
:special-members:
|
||||
:exclude-members: __init__, __repr__
|
||||
|
||||
```
|
||||
@@ -0,0 +1,12 @@
|
||||
# con
|
||||
|
||||
Constituency parsing.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.mtl
|
||||
|
||||
.. autoclass:: hanlp.components.mtl.tasks.constituency.CRFConstituencyParsing
|
||||
:members:
|
||||
:exclude-members: execute_training_loop, fit_dataloader
|
||||
|
||||
```
|
||||
@@ -0,0 +1,12 @@
|
||||
# dep
|
||||
|
||||
Dependency parsing.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.mtl
|
||||
|
||||
.. autoclass:: hanlp.components.mtl.tasks.dep.BiaffineDependencyParsing
|
||||
:members:
|
||||
:exclude-members: execute_training_loop, fit_dataloader
|
||||
|
||||
```
|
||||
@@ -0,0 +1,17 @@
|
||||
# tasks
|
||||
|
||||
Multi-Task Learning (MTL) tasks.
|
||||
|
||||
```{toctree}
|
||||
task
|
||||
constituency
|
||||
dep
|
||||
sdp
|
||||
ud
|
||||
lem
|
||||
pos
|
||||
tok
|
||||
ner/index
|
||||
srl/index
|
||||
```
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
# lem
|
||||
|
||||
Lemmatization.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.mtl
|
||||
|
||||
.. autoclass:: hanlp.components.mtl.tasks.lem.TransformerLemmatization
|
||||
:members:
|
||||
:exclude-members: execute_training_loop, fit_dataloader
|
||||
|
||||
```
|
||||
@@ -0,0 +1,12 @@
|
||||
# biaffine_ner
|
||||
|
||||
Biaffine Named Entity Recognition.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.mtl
|
||||
|
||||
.. autoclass:: hanlp.components.mtl.tasks.ner.biaffine_ner.BiaffineNamedEntityRecognition
|
||||
:members:
|
||||
:exclude-members: execute_training_loop, fit_dataloader
|
||||
|
||||
```
|
||||
@@ -0,0 +1,9 @@
|
||||
# ner
|
||||
|
||||
Named Entity Recognition.
|
||||
|
||||
```{toctree}
|
||||
tag_ner
|
||||
biaffine_ner
|
||||
```
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
# tag_ner
|
||||
|
||||
Tagging based Named Entity Recognition.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.mtl
|
||||
|
||||
.. autoclass:: hanlp.components.mtl.tasks.ner.tag_ner.TaggingNamedEntityRecognition
|
||||
:members:
|
||||
:exclude-members: execute_training_loop, fit_dataloader
|
||||
|
||||
```
|
||||
@@ -0,0 +1,12 @@
|
||||
# pos
|
||||
|
||||
Part-of-speech tagging.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.mtl
|
||||
|
||||
.. autoclass:: hanlp.components.mtl.tasks.pos.TransformerTagging
|
||||
:members:
|
||||
:exclude-members: execute_training_loop, fit_dataloader
|
||||
|
||||
```
|
||||
@@ -0,0 +1,12 @@
|
||||
# sdp
|
||||
|
||||
Semantic Dependency Parsing.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.mtl
|
||||
|
||||
.. autoclass:: hanlp.components.mtl.tasks.sdp.BiaffineSemanticDependencyParsing
|
||||
:members:
|
||||
:exclude-members: execute_training_loop, fit_dataloader
|
||||
|
||||
```
|
||||
@@ -0,0 +1,12 @@
|
||||
# bio_srl
|
||||
|
||||
BIO Tagging based Semantic Role Labeling.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.mtl
|
||||
|
||||
.. autoclass:: hanlp.components.mtl.tasks.srl.bio_srl.SpanBIOSemanticRoleLabeling
|
||||
:members:
|
||||
:exclude-members: execute_training_loop, fit_dataloader
|
||||
|
||||
```
|
||||
@@ -0,0 +1,9 @@
|
||||
# srl
|
||||
|
||||
Semantic Role Labeling.
|
||||
|
||||
```{toctree}
|
||||
bio_srl
|
||||
rank_srl
|
||||
```
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
# rank_srl
|
||||
|
||||
Span Ranking Semantic Role Labeling.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.mtl
|
||||
|
||||
.. autoclass:: hanlp.components.mtl.tasks.srl.rank_srl.SpanRankingSemanticRoleLabeling
|
||||
:members:
|
||||
:exclude-members: execute_training_loop, fit_dataloader
|
||||
|
||||
```
|
||||
@@ -0,0 +1,10 @@
|
||||
# Task
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.mtl
|
||||
|
||||
.. autoclass:: hanlp.components.mtl.tasks.Task
|
||||
:members:
|
||||
:exclude-members: execute_training_loop, fit_dataloader
|
||||
|
||||
```
|
||||
@@ -0,0 +1,12 @@
|
||||
# tok
|
||||
|
||||
Tokenization.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.mtl
|
||||
|
||||
.. autoclass:: hanlp.components.mtl.tasks.tok.tag_tok.TaggingTokenization
|
||||
:members:
|
||||
:exclude-members: execute_training_loop, fit_dataloader
|
||||
|
||||
```
|
||||
@@ -0,0 +1,12 @@
|
||||
# ud
|
||||
|
||||
Universal Dependencies Parsing (lemmatization, features, PoS tagging and dependency parsing).
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.mtl
|
||||
|
||||
.. autoclass:: hanlp.components.mtl.tasks.ud.UniversalDependenciesParsing
|
||||
:members:
|
||||
:exclude-members: execute_training_loop, fit_dataloader
|
||||
|
||||
```
|
||||
@@ -0,0 +1,11 @@
|
||||
# biaffine_ner
|
||||
|
||||
Biaffine Named Entity Recognition.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.ner.transformer_ner
|
||||
|
||||
.. autoclass:: hanlp.components.ner.biaffine_ner.biaffine_ner.BiaffineNamedEntityRecognizer
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,10 @@
|
||||
# ner
|
||||
|
||||
Named Entity Recognition.
|
||||
|
||||
```{toctree}
|
||||
transformer_ner
|
||||
rnn_ner
|
||||
biaffine_ner
|
||||
```
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
# rnn_ner
|
||||
|
||||
Tagging based Named Entity Recognition.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.ner.rnn_ner
|
||||
|
||||
.. autoclass:: hanlp.components.ner.rnn_ner.RNNNamedEntityRecognizer
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,11 @@
|
||||
# transformer_ner
|
||||
|
||||
Tagging based Named Entity Recognition.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.ner.transformer_ner
|
||||
|
||||
.. autoclass:: hanlp.components.ner.transformer_ner.TransformerNamedEntityRecognizer
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,11 @@
|
||||
# biaffine_dep
|
||||
|
||||
Biaffine dependency parser.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components
|
||||
|
||||
.. autoclass:: hanlp.components.parsers.biaffine.biaffine_dep.BiaffineDependencyParser
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,11 @@
|
||||
# biaffine_sdp
|
||||
|
||||
Biaffine dependency parser.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components
|
||||
|
||||
.. autoclass:: hanlp.components.parsers.biaffine.biaffine_sdp.BiaffineSemanticDependencyParser
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,11 @@
|
||||
# crf_constituency_parser
|
||||
|
||||
Biaffine dependency parser.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components
|
||||
|
||||
.. autoclass:: hanlp.components.parsers.constituency.crf_constituency_parser.CRFConstituencyParser
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,11 @@
|
||||
# parsers
|
||||
|
||||
Parsers.
|
||||
|
||||
```{toctree}
|
||||
biaffine_dep
|
||||
biaffine_sdp
|
||||
ud_parser
|
||||
crf_constituency_parser
|
||||
```
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
# ud_parser
|
||||
|
||||
Universal Dependencies Parsing (lemmatization, features, PoS tagging and dependency parsing).
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components
|
||||
|
||||
.. autoclass:: hanlp.components.parsers.ud.ud_parser.UniversalDependenciesParser
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,13 @@
|
||||
# pipeline
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.pipeline
|
||||
|
||||
.. autoclass:: hanlp.components.pipeline.Pipe
|
||||
:members:
|
||||
|
||||
.. autoclass:: hanlp.components.pipeline.Pipeline
|
||||
:members:
|
||||
|
||||
```
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
# srl
|
||||
|
||||
Semantic Role Labelers.
|
||||
|
||||
```{toctree}
|
||||
span_rank
|
||||
span_bio
|
||||
```
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
# span_bio
|
||||
|
||||
Span BIO tagging based SRL.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.srl.span_bio.span_bio
|
||||
|
||||
.. autoclass:: SpanBIOSemanticRoleLabeler
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,11 @@
|
||||
# span_rank
|
||||
|
||||
Span Rank based SRL.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.srl.span_rank.span_rank
|
||||
|
||||
.. autoclass:: SpanRankingSemanticRoleLabeler
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,10 @@
|
||||
# sts
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.sts
|
||||
|
||||
.. autoclass:: hanlp.components.sts.transformer_sts.TransformerSemanticTextualSimilarity
|
||||
:members:
|
||||
|
||||
```
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
# taggers
|
||||
|
||||
Taggers.
|
||||
|
||||
```{toctree}
|
||||
transformer_tagger
|
||||
rnn_tagger
|
||||
```
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
# rnn_tagger
|
||||
|
||||
RNN based tagger.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components
|
||||
|
||||
.. autoclass:: hanlp.components.taggers.rnn_tagger.RNNTagger
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,11 @@
|
||||
# transformer_tagger
|
||||
|
||||
Transformer based tagger.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components
|
||||
|
||||
.. autoclass:: hanlp.components.taggers.transformers.transformer_tagger.TransformerTagger
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,9 @@
|
||||
# tokenizers
|
||||
|
||||
Tokenizers.
|
||||
|
||||
```{toctree}
|
||||
transformer
|
||||
multi_criteria
|
||||
```
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
# multi_criteria
|
||||
|
||||
Transformer based Multi-Criteria Word tokenizer.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.tokenizers.multi_criteria_cws_transformer
|
||||
|
||||
.. autoclass:: hanlp.components.tokenizers.multi_criteria_cws_transformer.MultiCriteriaTransformerTaggingTokenizer
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,11 @@
|
||||
# transformer
|
||||
|
||||
Transformer based tokenizer.
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.components.tokenizers.transformer
|
||||
|
||||
.. autoclass:: hanlp.components.tokenizers.transformer.TransformerTaggingTokenizer
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,8 @@
|
||||
# constituency_dataset
|
||||
|
||||
```{eval-rst}
|
||||
|
||||
.. autoclass:: hanlp.datasets.parsing.loaders.constituency_dataset.ConstituencyDataset
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,9 @@
|
||||
# con
|
||||
|
||||
Constituency parsing datasets.
|
||||
|
||||
```{toctree}
|
||||
constituency_dataset
|
||||
resources
|
||||
```
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
# resources
|
||||
|
||||
## Chinese Treebank
|
||||
|
||||
|
||||
### CTB8
|
||||
|
||||
|
||||
|
||||
````{margin} **Discussion**
|
||||
```{seealso}
|
||||
About our data split on [our forum](https://bbs.hankcs.com/t/topic/3024).
|
||||
```
|
||||
````
|
||||
|
||||
```{eval-rst}
|
||||
|
||||
|
||||
.. autodata:: hanlp.datasets.parsing.ctb8.CTB8_BRACKET_LINE_NOEC_TRAIN
|
||||
.. autodata:: hanlp.datasets.parsing.ctb8.CTB8_BRACKET_LINE_NOEC_DEV
|
||||
.. autodata:: hanlp.datasets.parsing.ctb8.CTB8_BRACKET_LINE_NOEC_TEST
|
||||
|
||||
```
|
||||
|
||||
### CTB9
|
||||
|
||||
````{margin} **Discussion**
|
||||
```{seealso}
|
||||
About our data split on [our forum](https://bbs.hankcs.com/t/topic/3024).
|
||||
```
|
||||
````
|
||||
|
||||
```{eval-rst}
|
||||
|
||||
|
||||
.. autodata:: hanlp.datasets.parsing.ctb9.CTB9_BRACKET_LINE_NOEC_TRAIN
|
||||
.. autodata:: hanlp.datasets.parsing.ctb9.CTB9_BRACKET_LINE_NOEC_DEV
|
||||
.. autodata:: hanlp.datasets.parsing.ctb9.CTB9_BRACKET_LINE_NOEC_TEST
|
||||
|
||||
```
|
||||
|
||||
## English Treebank
|
||||
|
||||
### PTB
|
||||
|
||||
```{eval-rst}
|
||||
|
||||
.. autodata:: hanlp.datasets.parsing.ptb.PTB_TRAIN
|
||||
.. autodata:: hanlp.datasets.parsing.ptb.PTB_DEV
|
||||
.. autodata:: hanlp.datasets.parsing.ptb.PTB_TEST
|
||||
|
||||
```
|
||||
@@ -0,0 +1,10 @@
|
||||
# conll
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: hanlp.datasets.parsing.loaders.conll_dataset
|
||||
|
||||
|
||||
.. autoclass:: CoNLLParsingDataset
|
||||
:members:
|
||||
|
||||
```
|
||||
@@ -0,0 +1,9 @@
|
||||
# dep
|
||||
|
||||
Dependency parsing datasets.
|
||||
|
||||
```{toctree}
|
||||
conll_dataset
|
||||
resources
|
||||
```
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user