chore: import upstream snapshot with attribution
tests / Test (macos-latest, 3.14) (push) Has been cancelled
tests / Test (ubuntu-latest, 3.10) (push) Has been cancelled
tests / Test (ubuntu-latest, 3.11) (push) Has been cancelled
tests / Test (windows-latest, 3.13) (push) Has been cancelled
tests / Test (windows-latest, 3.14) (push) Has been cancelled
tests / Validate (push) Has been cancelled
tests / Test (macos-latest, 3.10) (push) Has been cancelled
tests / Test (macos-latest, 3.11) (push) Has been cancelled
tests / Test (macos-latest, 3.12) (push) Has been cancelled
tests / Test (macos-latest, 3.13) (push) Has been cancelled
tests / Test (ubuntu-latest, 3.12) (push) Has been cancelled
tests / Test (ubuntu-latest, 3.13) (push) Has been cancelled
tests / Test (ubuntu-latest, 3.14) (push) Has been cancelled
tests / Test (windows-latest, 3.10) (push) Has been cancelled
tests / Test (windows-latest, 3.11) (push) Has been cancelled
tests / Test (windows-latest, 3.12) (push) Has been cancelled
universe validation / Validate (push) Has been cancelled
tests / Test (macos-latest, 3.14) (push) Has been cancelled
tests / Test (ubuntu-latest, 3.10) (push) Has been cancelled
tests / Test (ubuntu-latest, 3.11) (push) Has been cancelled
tests / Test (windows-latest, 3.13) (push) Has been cancelled
tests / Test (windows-latest, 3.14) (push) Has been cancelled
tests / Validate (push) Has been cancelled
tests / Test (macos-latest, 3.10) (push) Has been cancelled
tests / Test (macos-latest, 3.11) (push) Has been cancelled
tests / Test (macos-latest, 3.12) (push) Has been cancelled
tests / Test (macos-latest, 3.13) (push) Has been cancelled
tests / Test (ubuntu-latest, 3.12) (push) Has been cancelled
tests / Test (ubuntu-latest, 3.13) (push) Has been cancelled
tests / Test (ubuntu-latest, 3.14) (push) Has been cancelled
tests / Test (windows-latest, 3.10) (push) Has been cancelled
tests / Test (windows-latest, 3.11) (push) Has been cancelled
tests / Test (windows-latest, 3.12) (push) Has been cancelled
universe validation / Validate (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,90 @@
|
||||
The central data structures in spaCy are the [`Language`](/api/language) class,
|
||||
the [`Vocab`](/api/vocab) and the [`Doc`](/api/doc) object. The `Language` class
|
||||
is used to process a text and turn it into a `Doc` object. It's typically stored
|
||||
as a variable called `nlp`. The `Doc` object owns the **sequence of tokens** and
|
||||
all their annotations. By centralizing strings, word vectors and lexical
|
||||
attributes in the `Vocab`, we avoid storing multiple copies of this data. This
|
||||
saves memory, and ensures there's a **single source of truth**.
|
||||
|
||||
Text annotations are also designed to allow a single source of truth: the `Doc`
|
||||
object owns the data, and [`Span`](/api/span) and [`Token`](/api/token) are
|
||||
**views that point into it**. The `Doc` object is constructed by the
|
||||
[`Tokenizer`](/api/tokenizer), and then **modified in place** by the components
|
||||
of the pipeline. The `Language` object coordinates these components. It takes
|
||||
raw text and sends it through the pipeline, returning an **annotated document**.
|
||||
It also orchestrates training and serialization.
|
||||
|
||||
|
||||
|
||||
### Container objects {id="architecture-containers"}
|
||||
|
||||
| Name | Description |
|
||||
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| [`Doc`](/api/doc) | A container for accessing linguistic annotations. |
|
||||
| [`DocBin`](/api/docbin) | A collection of `Doc` objects for efficient binary serialization. Also used for [training data](/api/data-formats#binary-training). |
|
||||
| [`Example`](/api/example) | A collection of training annotations, containing two `Doc` objects: the reference data and the predictions. |
|
||||
| [`Language`](/api/language) | Processing class that turns text into `Doc` objects. Different languages implement their own subclasses of it. The variable is typically called `nlp`. |
|
||||
| [`Lexeme`](/api/lexeme) | An entry in the vocabulary. It's a word type with no context, as opposed to a word token. It therefore has no part-of-speech tag, dependency parse etc. |
|
||||
| [`Span`](/api/span) | A slice from a `Doc` object. |
|
||||
| [`SpanGroup`](/api/spangroup) | A named collection of spans belonging to a `Doc`. |
|
||||
| [`Token`](/api/token) | An individual token — i.e. a word, punctuation symbol, whitespace, etc. |
|
||||
|
||||
### Processing pipeline {id="architecture-pipeline"}
|
||||
|
||||
The processing pipeline consists of one or more **pipeline components** that are
|
||||
called on the `Doc` in order. The tokenizer runs before the components. Pipeline
|
||||
components can be added using [`Language.add_pipe`](/api/language#add_pipe).
|
||||
They can contain a statistical model and trained weights, or only make
|
||||
rule-based modifications to the `Doc`. spaCy provides a range of built-in
|
||||
components for different language processing tasks and also allows adding
|
||||
[custom components](/usage/processing-pipelines#custom-components).
|
||||
|
||||
|
||||
|
||||
| Name | Description |
|
||||
| ----------------------------------------------- | ------------------------------------------------------------------------------------------- |
|
||||
| [`AttributeRuler`](/api/attributeruler) | Set token attributes using matcher rules. |
|
||||
| [`DependencyParser`](/api/dependencyparser) | Predict syntactic dependencies. |
|
||||
| [`EditTreeLemmatizer`](/api/edittreelemmatizer) | Predict base forms of words. |
|
||||
| [`EntityLinker`](/api/entitylinker) | Disambiguate named entities to nodes in a knowledge base. |
|
||||
| [`EntityRecognizer`](/api/entityrecognizer) | Predict named entities, e.g. persons or products. |
|
||||
| [`EntityRuler`](/api/entityruler) | Add entity spans to the `Doc` using token-based rules or exact phrase matches. |
|
||||
| [`Lemmatizer`](/api/lemmatizer) | Determine the base forms of words using rules and lookups. |
|
||||
| [`Morphologizer`](/api/morphologizer) | Predict morphological features and coarse-grained part-of-speech tags. |
|
||||
| [`SentenceRecognizer`](/api/sentencerecognizer) | Predict sentence boundaries. |
|
||||
| [`Sentencizer`](/api/sentencizer) | Implement rule-based sentence boundary detection that doesn't require the dependency parse. |
|
||||
| [`Tagger`](/api/tagger) | Predict part-of-speech tags. |
|
||||
| [`TextCategorizer`](/api/textcategorizer) | Predict categories or labels over the whole document. |
|
||||
| [`Tok2Vec`](/api/tok2vec) | Apply a "token-to-vector" model and set its outputs. |
|
||||
| [`Tokenizer`](/api/tokenizer) | Segment raw text and create `Doc` objects from the words. |
|
||||
| [`TrainablePipe`](/api/pipe) | Class that all trainable pipeline components inherit from. |
|
||||
| [`Transformer`](/api/transformer) | Use a transformer model and set its outputs. |
|
||||
| [Other functions](/api/pipeline-functions) | Automatically apply something to the `Doc`, e.g. to merge spans of tokens. |
|
||||
|
||||
### Matchers {id="architecture-matchers"}
|
||||
|
||||
Matchers help you find and extract information from [`Doc`](/api/doc) objects
|
||||
based on match patterns describing the sequences you're looking for. A matcher
|
||||
operates on a `Doc` and gives you access to the matched tokens **in context**.
|
||||
|
||||
| Name | Description |
|
||||
| --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| [`DependencyMatcher`](/api/dependencymatcher) | Match sequences of tokens based on dependency trees using [Semgrex operators](https://nlp.stanford.edu/nlp/javadoc/javanlp/edu/stanford/nlp/semgraph/semgrex/SemgrexPattern.html). |
|
||||
| [`Matcher`](/api/matcher) | Match sequences of tokens, based on pattern rules, similar to regular expressions. |
|
||||
| [`PhraseMatcher`](/api/phrasematcher) | Match sequences of tokens based on phrases. |
|
||||
|
||||
### Other classes {id="architecture-other"}
|
||||
|
||||
| Name | Description |
|
||||
| ------------------------------------------------ | -------------------------------------------------------------------------------------------------- |
|
||||
| [`Corpus`](/api/corpus) | Class for managing annotated corpora for training and evaluation data. |
|
||||
| [`KnowledgeBase`](/api/kb) | Abstract base class for storage and retrieval of data for entity linking. |
|
||||
| [`InMemoryLookupKB`](/api/inmemorylookupkb) | Implementation of `KnowledgeBase` storing all data in memory. |
|
||||
| [`Candidate`](/api/kb#candidate) | Object associating a textual mention with a specific entity contained in a `KnowledgeBase`. |
|
||||
| [`Lookups`](/api/lookups) | Container for convenient access to large lookup tables and dictionaries. |
|
||||
| [`MorphAnalysis`](/api/morphology#morphanalysis) | A morphological analysis. |
|
||||
| [`Morphology`](/api/morphology) | Store morphological analyses and map them to and from hash values. |
|
||||
| [`Scorer`](/api/scorer) | Compute evaluation scores. |
|
||||
| [`StringStore`](/api/stringstore) | Map strings to and from hash values. |
|
||||
| [`Vectors`](/api/vectors) | Container class for vector data keyed by string. |
|
||||
| [`Vocab`](/api/vocab) | The shared vocabulary that stores strings and gives you access to [`Lexeme`](/api/lexeme) objects. |
|
||||
@@ -0,0 +1,32 @@
|
||||
Every language is different – and usually full of **exceptions and special
|
||||
cases**, especially amongst the most common words. Some of these exceptions are
|
||||
shared across languages, while others are **entirely specific** – usually so
|
||||
specific that they need to be hard-coded. The
|
||||
[`lang`](%%GITHUB_SPACY/spacy/lang) module contains all language-specific data,
|
||||
organized in simple Python files. This makes the data easy to update and extend.
|
||||
|
||||
The **shared language data** in the directory root includes rules that can be
|
||||
generalized across languages – for example, rules for basic punctuation, emoji,
|
||||
emoticons and single-letter abbreviations. The **individual language data** in a
|
||||
submodule contains rules that are only relevant to a particular language. It
|
||||
also takes care of putting together all components and creating the
|
||||
[`Language`](/api/language) subclass – for example, `English` or `German`. The
|
||||
values are defined in the [`Language.Defaults`](/api/language#defaults).
|
||||
|
||||
> ```python
|
||||
> from spacy.lang.en import English
|
||||
> from spacy.lang.de import German
|
||||
>
|
||||
> nlp_en = English() # Includes English data
|
||||
> nlp_de = German() # Includes German data
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| --------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Stop words**<br />[`stop_words.py`](%%GITHUB_SPACY/spacy/lang/en/stop_words.py) | List of most common words of a language that are often useful to filter out, for example "and" or "I". Matching tokens will return `True` for `is_stop`. |
|
||||
| **Tokenizer exceptions**<br />[`tokenizer_exceptions.py`](%%GITHUB_SPACY/spacy/lang/de/tokenizer_exceptions.py) | Special-case rules for the tokenizer, for example, contractions like "can't" and abbreviations with punctuation, like "U.K.". |
|
||||
| **Punctuation rules**<br />[`punctuation.py`](%%GITHUB_SPACY/spacy/lang/punctuation.py) | Regular expressions for splitting tokens, e.g. on punctuation or special characters like emoji. Includes rules for prefixes, suffixes and infixes. |
|
||||
| **Character classes**<br />[`char_classes.py`](%%GITHUB_SPACY/spacy/lang/char_classes.py) | Character classes to be used in regular expressions, for example, Latin characters, quotes, hyphens or icons. |
|
||||
| **Lexical attributes**<br />[`lex_attrs.py`](%%GITHUB_SPACY/spacy/lang/en/lex_attrs.py) | Custom functions for setting lexical attributes on tokens, e.g. `like_num`, which includes language-specific words like "ten" or "hundred". |
|
||||
| **Syntax iterators**<br />[`syntax_iterators.py`](%%GITHUB_SPACY/spacy/lang/en/syntax_iterators.py) | Functions that compute views of a `Doc` object based on its syntax. At the moment, only used for [noun chunks](/usage/linguistic-features#noun-chunks). |
|
||||
| **Lemmatizer**<br />[`lemmatizer.py`](%%GITHUB_SPACY/spacy/lang/fr/lemmatizer.py) [`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data) | Custom lemmatizer implementation and lemmatization tables. |
|
||||
@@ -0,0 +1,36 @@
|
||||
A named entity is a "real-world object" that's assigned a name – for example, a
|
||||
person, a country, a product or a book title. spaCy can **recognize various
|
||||
types of named entities in a document, by asking the model for a prediction**.
|
||||
Because models are statistical and strongly depend on the examples they were
|
||||
trained on, this doesn't always work _perfectly_ and might need some tuning
|
||||
later, depending on your use case.
|
||||
|
||||
Named entities are available as the `ents` property of a `Doc`:
|
||||
|
||||
```python {executable="true"}
|
||||
import spacy
|
||||
|
||||
nlp = spacy.load("en_core_web_sm")
|
||||
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")
|
||||
|
||||
for ent in doc.ents:
|
||||
print(ent.text, ent.start_char, ent.end_char, ent.label_)
|
||||
```
|
||||
|
||||
> - **Text:** The original entity text.
|
||||
> - **Start:** Index of start of entity in the `Doc`.
|
||||
> - **End:** Index of end of entity in the `Doc`.
|
||||
> - **Label:** Entity label, i.e. type.
|
||||
|
||||
| Text | Start | End | Label | Description |
|
||||
| ----------- | :---: | :-: | ------- | ---------------------------------------------------- |
|
||||
| Apple | 0 | 5 | `ORG` | Companies, agencies, institutions. |
|
||||
| U.K. | 27 | 31 | `GPE` | Geopolitical entity, i.e. countries, cities, states. |
|
||||
| \$1 billion | 44 | 54 | `MONEY` | Monetary values, including unit. |
|
||||
|
||||
Using spaCy's built-in [displaCy visualizer](/usage/visualizers), here's what
|
||||
our example sentence and its named entities look like:
|
||||
|
||||
<Standalone height={120}>
|
||||
<div style={{lineHeight: 2.5, fontFamily: "-apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'", fontSize: 18}}><mark style={{ background: '#7aecec', padding: '0.45em 0.6em', margin: '0 0.25em', lineHeight: 1, borderRadius: '0.35em'}}>Apple <span style={{ fontSize: '0.8em', fontWeight: 'bold', lineHeight: 1, borderRadius: '0.35em', marginLeft: '0.5rem'}}>ORG</span></mark> is looking at buying <mark style={{ background: '#feca74', padding: '0.45em 0.6em', margin: '0 0.25em', lineHeight: 1, borderRadius: '0.35em'}}>U.K. <span style={{ fontSize: '0.8em', fontWeight: 'bold', lineHeight: 1, borderRadius: '0.35em', marginLeft: '0.5rem'}}>GPE</span></mark> startup for <mark style={{ background: '#e4e7d2', padding: '0.45em 0.6em', margin: '0 0.25em', lineHeight: 1, borderRadius: '0.35em'}}>$1 billion <span style={{ fontSize: '0.8em', fontWeight: 'bold', lineHeight: 1, borderRadius: '0.35em', marginLeft: '0.5rem'}}>MONEY</span></mark></div>
|
||||
</Standalone>
|
||||
@@ -0,0 +1,77 @@
|
||||
When you call `nlp` on a text, spaCy first tokenizes the text to produce a `Doc`
|
||||
object. The `Doc` is then processed in several different steps – this is also
|
||||
referred to as the **processing pipeline**. The pipeline used by the
|
||||
[trained pipelines](/models) typically include a tagger, a lemmatizer, a parser
|
||||
and an entity recognizer. Each pipeline component returns the processed `Doc`,
|
||||
which is then passed on to the next component.
|
||||
|
||||
|
||||
|
||||
> - **Name**: ID of the pipeline component.
|
||||
> - **Component:** spaCy's implementation of the component.
|
||||
> - **Creates:** Objects, attributes and properties modified and set by the
|
||||
> component.
|
||||
|
||||
| Name | Component | Creates | Description |
|
||||
| --------------------- | ------------------------------------------------------------------ | --------------------------------------------------------- | ------------------------------------------------ |
|
||||
| **tokenizer** | [`Tokenizer`](/api/tokenizer) | `Doc` | Segment text into tokens. |
|
||||
| _processing pipeline_ | | |
|
||||
| **tagger** | [`Tagger`](/api/tagger) | `Token.tag` | Assign part-of-speech tags. |
|
||||
| **parser** | [`DependencyParser`](/api/dependencyparser) | `Token.head`, `Token.dep`, `Doc.sents`, `Doc.noun_chunks` | Assign dependency labels. |
|
||||
| **ner** | [`EntityRecognizer`](/api/entityrecognizer) | `Doc.ents`, `Token.ent_iob`, `Token.ent_type` | Detect and label named entities. |
|
||||
| **lemmatizer** | [`Lemmatizer`](/api/lemmatizer) | `Token.lemma` | Assign base forms. |
|
||||
| **textcat** | [`TextCategorizer`](/api/textcategorizer) | `Doc.cats` | Assign document labels. |
|
||||
| **custom** | [custom components](/usage/processing-pipelines#custom-components) | `Doc._.xxx`, `Token._.xxx`, `Span._.xxx` | Assign custom attributes, methods or properties. |
|
||||
|
||||
The capabilities of a processing pipeline always depend on the components, their
|
||||
models and how they were trained. For example, a pipeline for named entity
|
||||
recognition needs to include a trained named entity recognizer component with a
|
||||
statistical model and weights that enable it to **make predictions** of entity
|
||||
labels. This is why each pipeline specifies its components and their settings in
|
||||
the [config](/usage/training#config):
|
||||
|
||||
```ini
|
||||
[nlp]
|
||||
pipeline = ["tok2vec", "tagger", "parser", "ner"]
|
||||
```
|
||||
|
||||
<Accordion title="Does the order of pipeline components matter?" id="pipeline-components-order">
|
||||
|
||||
The statistical components like the tagger or parser are typically independent
|
||||
and don't share any data between each other. For example, the named entity
|
||||
recognizer doesn't use any features set by the tagger and parser, and so on.
|
||||
This means that you can swap them, or remove single components from the pipeline
|
||||
without affecting the others. However, components may share a "token-to-vector"
|
||||
component like [`Tok2Vec`](/api/tok2vec) or [`Transformer`](/api/transformer).
|
||||
You can read more about this in the docs on
|
||||
[embedding layers](/usage/embeddings-transformers#embedding-layers).
|
||||
|
||||
Custom components may also depend on annotations set by other components. For
|
||||
example, a custom lemmatizer may need the part-of-speech tags assigned, so it'll
|
||||
only work if it's added after the tagger. The parser will respect pre-defined
|
||||
sentence boundaries, so if a previous component in the pipeline sets them, its
|
||||
dependency predictions may be different. Similarly, it matters if you add the
|
||||
[`EntityRuler`](/api/entityruler) before or after the statistical entity
|
||||
recognizer: if it's added before, the entity recognizer will take the existing
|
||||
entities into account when making predictions. The
|
||||
[`EntityLinker`](/api/entitylinker), which resolves named entities to knowledge
|
||||
base IDs, should be preceded by a pipeline component that recognizes entities
|
||||
such as the [`EntityRecognizer`](/api/entityrecognizer).
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Why is the tokenizer special?" id="pipeline-components-tokenizer">
|
||||
|
||||
The tokenizer is a "special" component and isn't part of the regular pipeline.
|
||||
It also doesn't show up in `nlp.pipe_names`. The reason is that there can only
|
||||
really be one tokenizer, and while all other pipeline components take a `Doc`
|
||||
and return it, the tokenizer takes a **string of text** and turns it into a
|
||||
`Doc`. You can still customize the tokenizer, though. `nlp.tokenizer` is
|
||||
writable, so you can either create your own
|
||||
[`Tokenizer` class from scratch](/usage/linguistic-features#native-tokenizers),
|
||||
or even replace it with an
|
||||
[entirely custom function](/usage/linguistic-features#custom-tokenizer).
|
||||
|
||||
</Accordion>
|
||||
|
||||
---
|
||||
@@ -0,0 +1,62 @@
|
||||
After tokenization, spaCy can **parse** and **tag** a given `Doc`. This is where
|
||||
the trained pipeline and its statistical models come in, which enable spaCy to
|
||||
**make predictions** of which tag or label most likely applies in this context.
|
||||
A trained component includes binary data that is produced by showing a system
|
||||
enough examples for it to make predictions that generalize across the language –
|
||||
for example, a word following "the" in English is most likely a noun.
|
||||
|
||||
Linguistic annotations are available as
|
||||
[`Token` attributes](/api/token#attributes). Like many NLP libraries, spaCy
|
||||
**encodes all strings to hash values** to reduce memory usage and improve
|
||||
efficiency. So to get the readable string representation of an attribute, we
|
||||
need to add an underscore `_` to its name:
|
||||
|
||||
```python {executable="true"}
|
||||
import spacy
|
||||
|
||||
nlp = spacy.load("en_core_web_sm")
|
||||
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")
|
||||
|
||||
for token in doc:
|
||||
print(token.text, token.lemma_, token.pos_, token.tag_, token.dep_,
|
||||
token.shape_, token.is_alpha, token.is_stop)
|
||||
```
|
||||
|
||||
> - **Text:** The original word text.
|
||||
> - **Lemma:** The base form of the word.
|
||||
> - **POS:** The simple [UPOS](https://universaldependencies.org/u/pos/)
|
||||
> part-of-speech tag.
|
||||
> - **Tag:** The detailed part-of-speech tag.
|
||||
> - **Dep:** Syntactic dependency, i.e. the relation between tokens.
|
||||
> - **Shape:** The word shape – capitalization, punctuation, digits.
|
||||
> - **is alpha:** Is the token an alpha character?
|
||||
> - **is stop:** Is the token part of a stop list, i.e. the most common words of
|
||||
> the language?
|
||||
|
||||
| Text | Lemma | POS | Tag | Dep | Shape | alpha | stop |
|
||||
| ------- | ------- | ------- | ----- | ---------- | ------- | ------- | ------- |
|
||||
| Apple | apple | `PROPN` | `NNP` | `nsubj` | `Xxxxx` | `True` | `False` |
|
||||
| is | be | `AUX` | `VBZ` | `aux` | `xx` | `True` | `True` |
|
||||
| looking | look | `VERB` | `VBG` | `ROOT` | `xxxx` | `True` | `False` |
|
||||
| at | at | `ADP` | `IN` | `prep` | `xx` | `True` | `True` |
|
||||
| buying | buy | `VERB` | `VBG` | `pcomp` | `xxxx` | `True` | `False` |
|
||||
| U.K. | u.k. | `PROPN` | `NNP` | `compound` | `X.X.` | `False` | `False` |
|
||||
| startup | startup | `NOUN` | `NN` | `dobj` | `xxxx` | `True` | `False` |
|
||||
| for | for | `ADP` | `IN` | `prep` | `xxx` | `True` | `True` |
|
||||
| \$ | \$ | `SYM` | `$` | `quantmod` | `$` | `False` | `False` |
|
||||
| 1 | 1 | `NUM` | `CD` | `compound` | `d` | `False` | `False` |
|
||||
| billion | billion | `NUM` | `CD` | `pobj` | `xxxx` | `True` | `False` |
|
||||
|
||||
> #### Tip: Understanding tags and labels
|
||||
>
|
||||
> Most of the tags and labels look pretty abstract, and they vary between
|
||||
> languages. `spacy.explain` will show you a short description – for example,
|
||||
> `spacy.explain("VBZ")` returns "verb, 3rd person singular present".
|
||||
|
||||
Using spaCy's built-in [displaCy visualizer](/usage/visualizers), here's what
|
||||
our example sentence and its dependencies look like:
|
||||
|
||||
<ImageScrollable
|
||||
src="/images/displacy-long.svg"
|
||||
width={1975}
|
||||
/>
|
||||
@@ -0,0 +1,29 @@
|
||||
If you've been modifying the pipeline, vocabulary, vectors and entities, or made
|
||||
updates to the component models, you'll eventually want to **save your
|
||||
progress** – for example, everything that's in your `nlp` object. This means
|
||||
you'll have to translate its contents and structure into a format that can be
|
||||
saved, like a file or a byte string. This process is called serialization. spaCy
|
||||
comes with **built-in serialization methods** and supports the
|
||||
[Pickle protocol](https://www.diveinto.org/python3/serializing.html#dump).
|
||||
|
||||
> #### What's pickle?
|
||||
>
|
||||
> Pickle is Python's built-in object persistence system. It lets you transfer
|
||||
> arbitrary Python objects between processes. This is usually used to load an
|
||||
> object to and from disk, but it's also used for distributed computing, e.g.
|
||||
> with
|
||||
> [PySpark](https://spark.apache.org/docs/0.9.0/python-programming-guide.html)
|
||||
> or [Dask](https://dask.org). When you unpickle an object, you're agreeing to
|
||||
> execute whatever code it contains. It's like calling `eval()` on a string – so
|
||||
> don't unpickle objects from untrusted sources.
|
||||
|
||||
All container classes, i.e. [`Language`](/api/language) (`nlp`),
|
||||
[`Doc`](/api/doc), [`Vocab`](/api/vocab) and [`StringStore`](/api/stringstore)
|
||||
have the following methods available:
|
||||
|
||||
| Method | Returns | Example |
|
||||
| ------------ | ------- | ------------------------ |
|
||||
| `to_bytes` | bytes | `data = nlp.to_bytes()` |
|
||||
| `from_bytes` | object | `nlp.from_bytes(data)` |
|
||||
| `to_disk` | - | `nlp.to_disk("/path")` |
|
||||
| `from_disk` | object | `nlp.from_disk("/path")` |
|
||||
@@ -0,0 +1,49 @@
|
||||
During processing, spaCy first **tokenizes** the text, i.e. segments it into
|
||||
words, punctuation and so on. This is done by applying rules specific to each
|
||||
language. For example, punctuation at the end of a sentence should be split off
|
||||
– whereas "U.K." should remain one token. Each `Doc` consists of individual
|
||||
tokens, and we can iterate over them:
|
||||
|
||||
```python {executable="true"}
|
||||
import spacy
|
||||
|
||||
nlp = spacy.load("en_core_web_sm")
|
||||
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")
|
||||
for token in doc:
|
||||
print(token.text)
|
||||
```
|
||||
|
||||
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|
||||
| :---: | :-: | :-----: | :-: | :----: | :--: | :-----: | :-: | :-: | :-: | :-----: |
|
||||
| Apple | is | looking | at | buying | U.K. | startup | for | \$ | 1 | billion |
|
||||
|
||||
First, the raw text is split on whitespace characters, similar to
|
||||
`text.split(' ')`. Then, the tokenizer processes the text from left to right. On
|
||||
each substring, it performs two checks:
|
||||
|
||||
1. **Does the substring match a tokenizer exception rule?** For example, "don't"
|
||||
does not contain whitespace, but should be split into two tokens, "do" and
|
||||
"n't", while "U.K." should always remain one token.
|
||||
|
||||
2. **Can a prefix, suffix or infix be split off?** For example punctuation like
|
||||
commas, periods, hyphens or quotes.
|
||||
|
||||
If there's a match, the rule is applied and the tokenizer continues its loop,
|
||||
starting with the newly split substrings. This way, spaCy can split **complex,
|
||||
nested tokens** like combinations of abbreviations and multiple punctuation
|
||||
marks.
|
||||
|
||||
> - **Tokenizer exception:** Special-case rule to split a string into several
|
||||
> tokens or prevent a token from being split when punctuation rules are
|
||||
> applied.
|
||||
> - **Prefix:** Character(s) at the beginning, e.g. `$`, `(`, `“`, `¿`.
|
||||
> - **Suffix:** Character(s) at the end, e.g. `km`, `)`, `”`, `!`.
|
||||
> - **Infix:** Character(s) in between, e.g. `-`, `--`, `/`, `…`.
|
||||
|
||||
|
||||
|
||||
While punctuation rules are usually pretty general, tokenizer exceptions
|
||||
strongly depend on the specifics of the individual language. This is why each
|
||||
[available language](/usage/models#languages) has its own subclass, like
|
||||
`English` or `German`, that loads in lists of hard-coded data and exception
|
||||
rules.
|
||||
@@ -0,0 +1,41 @@
|
||||
spaCy's tagger, parser, text categorizer and many other components are powered
|
||||
by **statistical models**. Every "decision" these components make – for example,
|
||||
which part-of-speech tag to assign, or whether a word is a named entity – is a
|
||||
**prediction** based on the model's current **weight values**. The weight values
|
||||
are estimated based on examples the model has seen during **training**. To train
|
||||
a model, you first need training data – examples of text, and the labels you
|
||||
want the model to predict. This could be a part-of-speech tag, a named entity or
|
||||
any other information.
|
||||
|
||||
Training is an iterative process in which the model's predictions are compared
|
||||
against the reference annotations in order to estimate the **gradient of the
|
||||
loss**. The gradient of the loss is then used to calculate the gradient of the
|
||||
weights through [backpropagation](https://thinc.ai/docs/backprop101). The
|
||||
gradients indicate how the weight values should be changed so that the model's
|
||||
predictions become more similar to the reference labels over time.
|
||||
|
||||
> - **Training data:** Examples and their annotations.
|
||||
> - **Text:** The input text the model should predict a label for.
|
||||
> - **Label:** The label the model should predict.
|
||||
> - **Gradient:** The direction and rate of change for a numeric value.
|
||||
> Minimising the gradient of the weights should result in predictions that are
|
||||
> closer to the reference labels on the training data.
|
||||
|
||||
|
||||
|
||||
When training a model, we don't just want it to memorize our examples – we want
|
||||
it to come up with a theory that can be **generalized across unseen data**.
|
||||
After all, we don't just want the model to learn that this one instance of
|
||||
"Amazon" right here is a company – we want it to learn that "Amazon", in
|
||||
contexts _like this_, is most likely a company. That's why the training data
|
||||
should always be representative of the data we want to process. A model trained
|
||||
on Wikipedia, where sentences in the first person are extremely rare, will
|
||||
likely perform badly on Twitter. Similarly, a model trained on romantic novels
|
||||
will likely perform badly on legal text.
|
||||
|
||||
This also means that in order to know how the model is performing, and whether
|
||||
it's learning the right things, you don't only need **training data** – you'll
|
||||
also need **evaluation data**. If you only test the model with the data it was
|
||||
trained on, you'll have no idea how well it's generalizing. If you want to train
|
||||
a model from scratch, you usually need at least a few hundred examples for both
|
||||
training and evaluation.
|
||||
@@ -0,0 +1,150 @@
|
||||
Similarity is determined by comparing **word vectors** or "word embeddings",
|
||||
multi-dimensional meaning representations of a word. Word vectors can be
|
||||
generated using an algorithm like
|
||||
[word2vec](https://en.wikipedia.org/wiki/Word2vec) and usually look like this:
|
||||
|
||||
```python {title="banana.vector"}
|
||||
array([2.02280000e-01, -7.66180009e-02, 3.70319992e-01,
|
||||
3.28450017e-02, -4.19569999e-01, 7.20689967e-02,
|
||||
-3.74760002e-01, 5.74599989e-02, -1.24009997e-02,
|
||||
5.29489994e-01, -5.23800015e-01, -1.97710007e-01,
|
||||
-3.41470003e-01, 5.33169985e-01, -2.53309999e-02,
|
||||
1.73800007e-01, 1.67720005e-01, 8.39839995e-01,
|
||||
5.51070012e-02, 1.05470002e-01, 3.78719985e-01,
|
||||
2.42750004e-01, 1.47449998e-02, 5.59509993e-01,
|
||||
1.25210002e-01, -6.75960004e-01, 3.58420014e-01,
|
||||
# ... and so on ...
|
||||
3.66849989e-01, 2.52470002e-03, -6.40089989e-01,
|
||||
-2.97650009e-01, 7.89430022e-01, 3.31680000e-01,
|
||||
-1.19659996e+00, -4.71559986e-02, 5.31750023e-01], dtype=float32)
|
||||
```
|
||||
|
||||
<Infobox title="Important note" variant="warning">
|
||||
|
||||
To make them compact and fast, spaCy's small [pipeline packages](/models) (all
|
||||
packages that end in `sm`) **don't ship with word vectors**, and only include
|
||||
context-sensitive **tensors**. This means you can still use the `similarity()`
|
||||
methods to compare documents, spans and tokens – but the result won't be as
|
||||
good, and individual tokens won't have any vectors assigned. So in order to use
|
||||
_real_ word vectors, you need to download a larger pipeline package:
|
||||
|
||||
```diff
|
||||
- python -m spacy download en_core_web_sm
|
||||
+ python -m spacy download en_core_web_lg
|
||||
```
|
||||
|
||||
</Infobox>
|
||||
|
||||
Pipeline packages that come with built-in word vectors make them available as
|
||||
the [`Token.vector`](/api/token#vector) attribute.
|
||||
[`Doc.vector`](/api/doc#vector) and [`Span.vector`](/api/span#vector) will
|
||||
default to an average of their token vectors. You can also check if a token has
|
||||
a vector assigned, and get the L2 norm, which can be used to normalize vectors.
|
||||
|
||||
```python {executable="true"}
|
||||
import spacy
|
||||
|
||||
nlp = spacy.load("en_core_web_md")
|
||||
tokens = nlp("dog cat banana afskfsd")
|
||||
|
||||
for token in tokens:
|
||||
print(token.text, token.has_vector, token.vector_norm, token.is_oov)
|
||||
```
|
||||
|
||||
> - **Text**: The original token text.
|
||||
> - **has vector**: Does the token have a vector representation?
|
||||
> - **Vector norm**: The L2 norm of the token's vector (the square root of the
|
||||
> sum of the values squared)
|
||||
> - **OOV**: Out-of-vocabulary
|
||||
|
||||
The words "dog", "cat" and "banana" are all pretty common in English, so they're
|
||||
part of the pipeline's vocabulary, and come with a vector. The word "afskfsd" on
|
||||
the other hand is a lot less common and out-of-vocabulary – so its vector
|
||||
representation consists of 300 dimensions of `0`, which means it's practically
|
||||
nonexistent. If your application will benefit from a **large vocabulary** with
|
||||
more vectors, you should consider using one of the larger pipeline packages or
|
||||
loading in a full vector package, for example,
|
||||
[`en_core_web_lg`](/models/en#en_core_web_lg), which includes **685k unique
|
||||
vectors**.
|
||||
|
||||
spaCy is able to compare two objects, and make a prediction of **how similar
|
||||
they are**. Predicting similarity is useful for building recommendation systems
|
||||
or flagging duplicates. For example, you can suggest a user content that's
|
||||
similar to what they're currently looking at, or label a support ticket as a
|
||||
duplicate if it's very similar to an already existing one.
|
||||
|
||||
Each [`Doc`](/api/doc), [`Span`](/api/span), [`Token`](/api/token) and
|
||||
[`Lexeme`](/api/lexeme) comes with a [`.similarity`](/api/token#similarity)
|
||||
method that lets you compare it with another object, and determine the
|
||||
similarity. Of course similarity is always subjective – whether two words, spans
|
||||
or documents are similar really depends on how you're looking at it. spaCy's
|
||||
similarity implementation usually assumes a pretty general-purpose definition of
|
||||
similarity.
|
||||
|
||||
> #### 📝 Things to try
|
||||
>
|
||||
> 1. Compare two different tokens and try to find the two most _dissimilar_
|
||||
> tokens in the texts with the lowest similarity score (according to the
|
||||
> vectors).
|
||||
> 2. Compare the similarity of two [`Lexeme`](/api/lexeme) objects, entries in
|
||||
> the vocabulary. You can get a lexeme via the `.lex` attribute of a token.
|
||||
> You should see that the similarity results are identical to the token
|
||||
> similarity.
|
||||
|
||||
```python {executable="true"}
|
||||
import spacy
|
||||
|
||||
nlp = spacy.load("en_core_web_md") # make sure to use larger package!
|
||||
doc1 = nlp("I like salty fries and hamburgers.")
|
||||
doc2 = nlp("Fast food tastes very good.")
|
||||
|
||||
# Similarity of two documents
|
||||
print(doc1, "<->", doc2, doc1.similarity(doc2))
|
||||
# Similarity of tokens and spans
|
||||
french_fries = doc1[2:4]
|
||||
burgers = doc1[5]
|
||||
print(french_fries, "<->", burgers, french_fries.similarity(burgers))
|
||||
```
|
||||
|
||||
### What to expect from similarity results {id="similarity-expectations"}
|
||||
|
||||
Computing similarity scores can be helpful in many situations, but it's also
|
||||
important to maintain **realistic expectations** about what information it can
|
||||
provide. Words can be related to each other in many ways, so a single
|
||||
"similarity" score will always be a **mix of different signals**, and vectors
|
||||
trained on different data can produce very different results that may not be
|
||||
useful for your purpose. Here are some important considerations to keep in mind:
|
||||
|
||||
- There's no objective definition of similarity. Whether "I like burgers" and "I
|
||||
like pasta" is similar **depends on your application**. Both talk about food
|
||||
preferences, which makes them very similar – but if you're analyzing mentions
|
||||
of food, those sentences are pretty dissimilar, because they talk about very
|
||||
different foods.
|
||||
- The similarity of [`Doc`](/api/doc) and [`Span`](/api/span) objects defaults
|
||||
to the **average** of the token vectors. This means that the vector for "fast
|
||||
food" is the average of the vectors for "fast" and "food", which isn't
|
||||
necessarily representative of the phrase "fast food".
|
||||
- Vector averaging means that the vector of multiple tokens is **insensitive to
|
||||
the order** of the words. Two documents expressing the same meaning with
|
||||
dissimilar wording will return a lower similarity score than two documents
|
||||
that happen to contain the same words while expressing different meanings.
|
||||
|
||||
<Infobox title="Tip: Check out sense2vec" emoji="💡">
|
||||
|
||||
<Image
|
||||
src="/images/sense2vec.jpg"
|
||||
href="https://github.com/explosion/sense2vec"
|
||||
alt="sense2vec Screenshot"
|
||||
/>
|
||||
|
||||
[`sense2vec`](https://github.com/explosion/sense2vec) is a library developed by
|
||||
us that builds on top of spaCy and lets you train and query more interesting and
|
||||
detailed word vectors. It combines noun phrases like "fast food" or "fair game"
|
||||
and includes the part-of-speech tags and entity labels. The library also
|
||||
includes annotation recipes for our annotation tool [Prodigy](https://prodi.gy)
|
||||
that let you evaluate vectors and create terminology lists. For more details,
|
||||
check out [our blog post](https://explosion.ai/blog/sense2vec-reloaded). To
|
||||
explore the semantic similarities across all Reddit comments of 2015 and 2019,
|
||||
see the [interactive demo](https://explosion.ai/demos/sense2vec).
|
||||
|
||||
</Infobox>
|
||||
@@ -0,0 +1,40 @@
|
||||
<figure>
|
||||
|
||||
| Pipeline | Parser | Tagger | NER |
|
||||
| ---------------------------------------------------------- | -----: | -----: | ---: |
|
||||
| [`en_core_web_trf`](/models/en#en_core_web_trf) (spaCy v3) | 95.1 | 97.8 | 89.8 |
|
||||
| [`en_core_web_lg`](/models/en#en_core_web_lg) (spaCy v3) | 92.0 | 97.4 | 85.5 |
|
||||
| `en_core_web_lg` (spaCy v2) | 91.9 | 97.2 | 85.5 |
|
||||
|
||||
<figcaption className="caption">
|
||||
|
||||
**Full pipeline accuracy** on the
|
||||
[OntoNotes 5.0](https://catalog.ldc.upenn.edu/LDC2013T19) corpus (reported on
|
||||
the development set).
|
||||
|
||||
</figcaption>
|
||||
|
||||
</figure>
|
||||
|
||||
<figure>
|
||||
|
||||
| Named Entity Recognition System | OntoNotes | CoNLL '03 |
|
||||
| -------------------------------- | --------: | --------: |
|
||||
| spaCy RoBERTa (2020) | 89.8 | 91.6 |
|
||||
| Stanza (StanfordNLP)<sup>1</sup> | 88.8 | 92.1 |
|
||||
| Flair<sup>2</sup> | 89.7 | 93.1 |
|
||||
|
||||
<figcaption className="caption">
|
||||
|
||||
**Named entity recognition accuracy** on the
|
||||
[OntoNotes 5.0](https://catalog.ldc.upenn.edu/LDC2013T19) and
|
||||
[CoNLL-2003](https://www.aclweb.org/anthology/W03-0419.pdf) corpora. See
|
||||
[NLP-progress](http://nlpprogress.com/english/named_entity_recognition.html) for
|
||||
more results. Project template:
|
||||
[`benchmarks/ner_conll03`](%%GITHUB_PROJECTS/benchmarks/ner_conll03). **1.**
|
||||
[Qi et al. (2020)](https://arxiv.org/pdf/2003.07082.pdf). **2.**
|
||||
[Akbik et al. (2018)](https://www.aclweb.org/anthology/C18-1139/).
|
||||
|
||||
</figcaption>
|
||||
|
||||
</figure>
|
||||
@@ -0,0 +1,987 @@
|
||||
---
|
||||
title: Embeddings, Transformers and Transfer Learning
|
||||
teaser: Using transformer embeddings like BERT in spaCy
|
||||
menu:
|
||||
- ['Embedding Layers', 'embedding-layers']
|
||||
- ['Transformers', 'transformers']
|
||||
- ['Static Vectors', 'static-vectors']
|
||||
- ['Pretraining', 'pretraining']
|
||||
next: /usage/training
|
||||
---
|
||||
|
||||
spaCy supports a number of **transfer and multi-task learning** workflows that
|
||||
can often help improve your pipeline's efficiency or accuracy. Transfer learning
|
||||
refers to techniques such as word vector tables and language model pretraining.
|
||||
These techniques can be used to import knowledge from raw text into your
|
||||
pipeline, so that your models are able to generalize better from your annotated
|
||||
examples.
|
||||
|
||||
You can convert **word vectors** from popular tools like
|
||||
[FastText](https://fasttext.cc) and [Gensim](https://radimrehurek.com/gensim),
|
||||
or you can load in any pretrained **transformer model** if you install
|
||||
[`spacy-transformers`](https://github.com/explosion/spacy-transformers). You can
|
||||
also do your own language model pretraining via the
|
||||
[`spacy pretrain`](/api/cli#pretrain) command. You can even **share** your
|
||||
transformer or other contextual embedding model across multiple components,
|
||||
which can make long pipelines several times more efficient. To use transfer
|
||||
learning, you'll need at least a few annotated examples for what you're trying
|
||||
to predict. Otherwise, you could try using a "one-shot learning" approach using
|
||||
[vectors and similarity](/usage/linguistic-features#vectors-similarity).
|
||||
|
||||
<Accordion title="What’s the difference between word vectors and language models?" id="vectors-vs-language-models">
|
||||
|
||||
[Transformers](#transformers) are large and powerful neural networks that give
|
||||
you better accuracy, but are harder to deploy in production, as they require a
|
||||
GPU to run effectively. [Word vectors](#word-vectors) are a slightly older
|
||||
technique that can give your models a smaller improvement in accuracy, and can
|
||||
also provide some additional capabilities.
|
||||
|
||||
The key difference between word-vectors and contextual language models such as
|
||||
transformers is that word vectors model **lexical types**, rather than _tokens_.
|
||||
If you have a list of terms with no context around them, a transformer model
|
||||
like BERT can't really help you. BERT is designed to understand language **in
|
||||
context**, which isn't what you have. A word vectors table will be a much better
|
||||
fit for your task. However, if you do have words in context – whole sentences or
|
||||
paragraphs of running text – word vectors will only provide a very rough
|
||||
approximation of what the text is about.
|
||||
|
||||
Word vectors are also very computationally efficient, as they map a word to a
|
||||
vector with a single indexing operation. Word vectors are therefore useful as a
|
||||
way to **improve the accuracy** of neural network models, especially models that
|
||||
are small or have received little or no pretraining. In spaCy, word vector
|
||||
tables are only used as **static features**. spaCy does not backpropagate
|
||||
gradients to the pretrained word vectors table. The static vectors table is
|
||||
usually used in combination with a smaller table of learned task-specific
|
||||
embeddings.
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="When should I add word vectors to my model?">
|
||||
|
||||
Word vectors are not compatible with most [transformer models](#transformers),
|
||||
but if you're training another type of NLP network, it's almost always worth
|
||||
adding word vectors to your model. As well as improving your final accuracy,
|
||||
word vectors often make experiments more consistent, as the accuracy you reach
|
||||
will be less sensitive to how the network is randomly initialized. High variance
|
||||
due to random chance can slow down your progress significantly, as you need to
|
||||
run many experiments to filter the signal from the noise.
|
||||
|
||||
Word vector features need to be enabled prior to training, and the same word
|
||||
vectors table will need to be available at runtime as well. You cannot add word
|
||||
vector features once the model has already been trained, and you usually cannot
|
||||
replace one word vectors table with another without causing a significant loss
|
||||
of performance.
|
||||
|
||||
</Accordion>
|
||||
|
||||
## Shared embedding layers {id="embedding-layers"}
|
||||
|
||||
spaCy lets you share a single transformer or other token-to-vector ("tok2vec")
|
||||
embedding layer between multiple components. You can even update the shared
|
||||
layer, performing **multi-task learning**. Reusing the tok2vec layer between
|
||||
components can make your pipeline run a lot faster and result in much smaller
|
||||
models. However, it can make the pipeline less modular and make it more
|
||||
difficult to swap components or retrain parts of the pipeline. Multi-task
|
||||
learning can affect your accuracy (either positively or negatively), and may
|
||||
require some retuning of your hyper-parameters.
|
||||
|
||||
|
||||
|
||||
| Shared | Independent |
|
||||
| ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
|
||||
| ✅ **smaller:** models only need to include a single copy of the embeddings | ❌ **larger:** models need to include the embeddings for each component |
|
||||
| ✅ **faster:** embed the documents once for your whole pipeline | ❌ **slower:** rerun the embedding for each component |
|
||||
| ❌ **less composable:** all components require the same embedding component in the pipeline | ✅ **modular:** components can be moved and swapped freely |
|
||||
|
||||
You can share a single transformer or other tok2vec model between multiple
|
||||
components by adding a [`Transformer`](/api/transformer) or
|
||||
[`Tok2Vec`](/api/tok2vec) component near the start of your pipeline. Components
|
||||
later in the pipeline can "connect" to it by including a **listener layer** like
|
||||
[Tok2VecListener](/api/architectures#Tok2VecListener) within their model.
|
||||
|
||||
|
||||
|
||||
At the beginning of training, the [`Tok2Vec`](/api/tok2vec) component will grab
|
||||
a reference to the relevant listener layers in the rest of your pipeline. When
|
||||
it processes a batch of documents, it will pass forward its predictions to the
|
||||
listeners, allowing the listeners to **reuse the predictions** when they are
|
||||
eventually called. A similar mechanism is used to pass gradients from the
|
||||
listeners back to the model. The [`Transformer`](/api/transformer) component and
|
||||
[TransformerListener](/api/architectures#TransformerListener) layer do the same
|
||||
thing for transformer models, but the `Transformer` component will also save the
|
||||
transformer outputs to the
|
||||
[`Doc._.trf_data`](/api/transformer#custom_attributes) extension attribute,
|
||||
giving you access to them after the pipeline has finished running.
|
||||
|
||||
### Example: Shared vs. independent config {id="embedding-layers-config"}
|
||||
|
||||
The [config system](/usage/training#config) lets you express model configuration
|
||||
for both shared and independent embedding layers. The shared setup uses a single
|
||||
[`Tok2Vec`](/api/tok2vec) component with the
|
||||
[Tok2Vec](/api/architectures#Tok2Vec) architecture. All other components, like
|
||||
the entity recognizer, use a
|
||||
[Tok2VecListener](/api/architectures#Tok2VecListener) layer as their model's
|
||||
`tok2vec` argument, which connects to the `tok2vec` component model.
|
||||
|
||||
```ini {title="Shared",highlight="1-2,4-5,19-20"}
|
||||
[components.tok2vec]
|
||||
factory = "tok2vec"
|
||||
|
||||
[components.tok2vec.model]
|
||||
@architectures = "spacy.Tok2Vec.v2"
|
||||
|
||||
[components.tok2vec.model.embed]
|
||||
@architectures = "spacy.MultiHashEmbed.v2"
|
||||
|
||||
[components.tok2vec.model.encode]
|
||||
@architectures = "spacy.MaxoutWindowEncoder.v2"
|
||||
|
||||
[components.ner]
|
||||
factory = "ner"
|
||||
|
||||
[components.ner.model]
|
||||
@architectures = "spacy.TransitionBasedParser.v1"
|
||||
|
||||
[components.ner.model.tok2vec]
|
||||
@architectures = "spacy.Tok2VecListener.v1"
|
||||
```
|
||||
|
||||
In the independent setup, the entity recognizer component defines its own
|
||||
[Tok2Vec](/api/architectures#Tok2Vec) instance. Other components will do the
|
||||
same. This makes them fully independent and doesn't require an upstream
|
||||
[`Tok2Vec`](/api/tok2vec) component to be present in the pipeline.
|
||||
|
||||
```ini {title="Independent", highlight="7-8"}
|
||||
[components.ner]
|
||||
factory = "ner"
|
||||
|
||||
[components.ner.model]
|
||||
@architectures = "spacy.TransitionBasedParser.v1"
|
||||
|
||||
[components.ner.model.tok2vec]
|
||||
@architectures = "spacy.Tok2Vec.v2"
|
||||
|
||||
[components.ner.model.tok2vec.embed]
|
||||
@architectures = "spacy.MultiHashEmbed.v2"
|
||||
|
||||
[components.ner.model.tok2vec.encode]
|
||||
@architectures = "spacy.MaxoutWindowEncoder.v2"
|
||||
```
|
||||
|
||||
{/* TODO: Once rehearsal is tested, mention it here. */}
|
||||
|
||||
## Using transformer models {id="transformers"}
|
||||
|
||||
Transformers are a family of neural network architectures that compute **dense,
|
||||
context-sensitive representations** for the tokens in your documents. Downstream
|
||||
models in your pipeline can then use these representations as input features to
|
||||
**improve their predictions**. You can connect multiple components to a single
|
||||
transformer model, with any or all of those components giving feedback to the
|
||||
transformer to fine-tune it to your tasks. spaCy's transformer support
|
||||
interoperates with [PyTorch](https://pytorch.org) and the
|
||||
[HuggingFace `transformers`](https://huggingface.co/transformers/) library,
|
||||
giving you access to thousands of pretrained models for your pipelines. There
|
||||
are many [great guides](http://jalammar.github.io/illustrated-transformer/) to
|
||||
transformer models, but for practical purposes, you can simply think of them as
|
||||
drop-in replacements that let you achieve **higher accuracy** in exchange for
|
||||
**higher training and runtime costs**.
|
||||
|
||||
### Setup and installation {id="transformers-installation"}
|
||||
|
||||
> #### System requirements
|
||||
>
|
||||
> We recommend an NVIDIA **GPU** with at least **10GB of memory** in order to
|
||||
> work with transformer models. Make sure your GPU drivers are up to date and
|
||||
> you have **CUDA v9+** installed.
|
||||
|
||||
> The exact requirements will depend on the transformer model. Training a
|
||||
> transformer-based model without a GPU will be too slow for most practical
|
||||
> purposes.
|
||||
>
|
||||
> Provisioning a new machine will require about **5GB** of data to be
|
||||
> downloaded: 3GB CUDA runtime, 800MB PyTorch, 400MB CuPy, 500MB weights, 200MB
|
||||
> spaCy and dependencies.
|
||||
|
||||
Once you have CUDA installed, we recommend installing PyTorch following the
|
||||
[PyTorch installation guidelines](https://pytorch.org/get-started/locally/) for
|
||||
your package manager and CUDA version. If you skip this step, pip will install
|
||||
PyTorch as a dependency below, but it may not find the best version for your
|
||||
setup.
|
||||
|
||||
```bash {title="Example: Install PyTorch 1.11.0 for CUDA 11.3 with pip"}
|
||||
# See: https://pytorch.org/get-started/locally/
|
||||
$ pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
|
||||
```
|
||||
|
||||
Next, install spaCy with the extras for your CUDA version and transformers. The
|
||||
CUDA extra (e.g., `cuda102`, `cuda113`) installs the correct version of
|
||||
[`cupy`](https://docs.cupy.dev/en/stable/install.html#installing-cupy), which is
|
||||
just like `numpy`, but for GPU. You may also need to set the `CUDA_PATH`
|
||||
environment variable if your CUDA runtime is installed in a non-standard
|
||||
location. Putting it all together, if you had installed CUDA 11.3 in
|
||||
`/opt/nvidia/cuda`, you would run:
|
||||
|
||||
```bash {title="Installation with CUDA"}
|
||||
$ export CUDA_PATH="/opt/nvidia/cuda"
|
||||
$ pip install -U %%SPACY_PKG_NAME[cuda113,transformers]%%SPACY_PKG_FLAGS
|
||||
```
|
||||
|
||||
For [`transformers`](https://huggingface.co/transformers/) v4.0.0+ and models
|
||||
that require [`SentencePiece`](https://github.com/google/sentencepiece) (e.g.,
|
||||
ALBERT, CamemBERT, XLNet, Marian, and T5), install the additional dependencies
|
||||
with:
|
||||
|
||||
```bash {title="Install sentencepiece"}
|
||||
$ pip install transformers[sentencepiece]
|
||||
```
|
||||
|
||||
### Runtime usage {id="transformers-runtime"}
|
||||
|
||||
Transformer models can be used as **drop-in replacements** for other types of
|
||||
neural networks, so your spaCy pipeline can include them in a way that's
|
||||
completely invisible to the user. Users will download, load and use the model in
|
||||
the standard way, like any other spaCy pipeline. Instead of using the
|
||||
transformers as subnetworks directly, you can also use them via the
|
||||
[`Transformer`](/api/transformer) pipeline component.
|
||||
|
||||
|
||||
|
||||
The `Transformer` component sets the
|
||||
[`Doc._.trf_data`](/api/transformer#custom_attributes) extension attribute,
|
||||
which lets you access the transformers outputs at runtime. The trained
|
||||
transformer-based [pipelines](/models) provided by spaCy end on `_trf`, e.g.
|
||||
[`en_core_web_trf`](/models/en#en_core_web_trf).
|
||||
|
||||
```bash
|
||||
$ python -m spacy download en_core_web_trf
|
||||
```
|
||||
|
||||
```python {title="Example"}
|
||||
import spacy
|
||||
from thinc.api import set_gpu_allocator, require_gpu
|
||||
|
||||
# Use the GPU, with memory allocations directed via PyTorch.
|
||||
# This prevents out-of-memory errors that would otherwise occur from competing
|
||||
# memory pools.
|
||||
set_gpu_allocator("pytorch")
|
||||
require_gpu(0)
|
||||
|
||||
nlp = spacy.load("en_core_web_trf")
|
||||
for doc in nlp.pipe(["some text", "some other text"]):
|
||||
# For spaCy v3.0-v3.6, trf pipelines use spacy-transformers and the transformer output in doc._.trf_data is a TransformerData object.
|
||||
try:
|
||||
tokvecs = doc._.trf_data.tensors[-1]
|
||||
# For spaCy v3.7+, trf pipelines use spacy-curated-transformers and doc._.trf_data is a DocTransformerOutput object.
|
||||
except AttributeError as e:
|
||||
tokvecs = doc._.trf_data.last_hidden_layer_state # or doc._.trf_data.all_outputs[-1]
|
||||
```
|
||||
|
||||
You can also customize how the [`Transformer`](/api/transformer) component sets
|
||||
annotations onto the [`Doc`](/api/doc) by specifying a custom
|
||||
`set_extra_annotations` function. This callback will be called with the raw
|
||||
input and output data for the whole batch, along with the batch of `Doc`
|
||||
objects, allowing you to implement whatever you need. The annotation setter is
|
||||
called with a batch of [`Doc`](/api/doc) objects and a
|
||||
[`FullTransformerBatch`](/api/transformer#fulltransformerbatch) containing the
|
||||
transformers data for the batch.
|
||||
|
||||
```python
|
||||
def custom_annotation_setter(docs, trf_data):
|
||||
doc_data = list(trf_data.doc_data)
|
||||
for doc, data in zip(docs, doc_data):
|
||||
doc._.custom_attr = data
|
||||
|
||||
nlp = spacy.load("en_core_web_trf")
|
||||
nlp.get_pipe("transformer").set_extra_annotations = custom_annotation_setter
|
||||
doc = nlp("This is a text")
|
||||
assert isinstance(doc._.custom_attr, TransformerData)
|
||||
print(doc._.custom_attr.tensors)
|
||||
```
|
||||
|
||||
### Training usage {id="transformers-training"}
|
||||
|
||||
The recommended workflow for training is to use spaCy's
|
||||
[config system](/usage/training#config), usually via the
|
||||
[`spacy train`](/api/cli#train) command. The training config defines all
|
||||
component settings and hyperparameters in one place and lets you describe a tree
|
||||
of objects by referring to creation functions, including functions you register
|
||||
yourself. For details on how to get started with training your own model, check
|
||||
out the [training quickstart](/usage/training#quickstart).
|
||||
|
||||
{/* TODO: <Project id="pipelines/transformers"> */}
|
||||
|
||||
{/* The easiest way to get started is to clone a transformers-based project */}
|
||||
{/* template. Swap in your data, edit the settings and hyperparameters and train, */}
|
||||
{/* evaluate, package and visualize your model. */}
|
||||
|
||||
{/* </Project> */}
|
||||
|
||||
The `[components]` section in the [`config.cfg`](/api/data-formats#config)
|
||||
describes the pipeline components and the settings used to construct them,
|
||||
including their model implementation. Here's a config snippet for the
|
||||
[`Transformer`](/api/transformer) component, along with matching Python code. In
|
||||
this case, the `[components.transformer]` block describes the `transformer`
|
||||
component:
|
||||
|
||||
> #### Python equivalent
|
||||
>
|
||||
> ```python
|
||||
> from spacy_transformers import Transformer, TransformerModel
|
||||
> from spacy_transformers.annotation_setters import null_annotation_setter
|
||||
> from spacy_transformers.span_getters import get_doc_spans
|
||||
>
|
||||
> trf = Transformer(
|
||||
> nlp.vocab,
|
||||
> TransformerModel(
|
||||
> "bert-base-cased",
|
||||
> get_spans=get_doc_spans,
|
||||
> tokenizer_config={"use_fast": True},
|
||||
> ),
|
||||
> set_extra_annotations=null_annotation_setter,
|
||||
> max_batch_items=4096,
|
||||
> )
|
||||
> ```
|
||||
|
||||
```ini {title="config.cfg",excerpt="true"}
|
||||
[components.transformer]
|
||||
factory = "transformer"
|
||||
max_batch_items = 4096
|
||||
|
||||
[components.transformer.model]
|
||||
@architectures = "spacy-transformers.TransformerModel.v3"
|
||||
name = "bert-base-cased"
|
||||
tokenizer_config = {"use_fast": true}
|
||||
|
||||
[components.transformer.model.get_spans]
|
||||
@span_getters = "spacy-transformers.doc_spans.v1"
|
||||
|
||||
[components.transformer.set_extra_annotations]
|
||||
@annotation_setters = "spacy-transformers.null_annotation_setter.v1"
|
||||
|
||||
```
|
||||
|
||||
The `[components.transformer.model]` block describes the `model` argument passed
|
||||
to the transformer component. It's a Thinc
|
||||
[`Model`](https://thinc.ai/docs/api-model) object that will be passed into the
|
||||
component. Here, it references the function
|
||||
[spacy-transformers.TransformerModel.v3](/api/architectures#TransformerModel)
|
||||
registered in the [`architectures` registry](/api/top-level#registry). If a key
|
||||
in a block starts with `@`, it's **resolved to a function** and all other
|
||||
settings are passed to the function as arguments. In this case, `name`,
|
||||
`tokenizer_config` and `get_spans`.
|
||||
|
||||
`get_spans` is a function that takes a batch of `Doc` objects and returns lists
|
||||
of potentially overlapping `Span` objects to process by the transformer. Several
|
||||
[built-in functions](/api/transformer#span_getters) are available – for example,
|
||||
to process the whole document or individual sentences. When the config is
|
||||
resolved, the function is created and passed into the model as an argument.
|
||||
|
||||
The `name` value is the name of any [HuggingFace model](huggingface-models),
|
||||
which will be downloaded automatically the first time it's used. You can also
|
||||
use a local file path. For full details, see the
|
||||
[`TransformerModel` docs](/api/architectures#TransformerModel).
|
||||
|
||||
[huggingface-models]:
|
||||
https://huggingface.co/models?library=pytorch&sort=downloads
|
||||
|
||||
A wide variety of PyTorch models are supported, but some might not work. If a
|
||||
model doesn't seem to work feel free to open an
|
||||
[issue](https://github.com/explosion/spacy/issues). Additionally note that
|
||||
Transformers loaded in spaCy can only be used for tensors, and pretrained
|
||||
task-specific heads or text generation features cannot be used as part of the
|
||||
`transformer` pipeline component.
|
||||
|
||||
<Infobox variant="warning">
|
||||
|
||||
Remember that the `config.cfg` used for training should contain **no missing
|
||||
values** and requires all settings to be defined. You don't want any hidden
|
||||
defaults creeping in and changing your results! spaCy will tell you if settings
|
||||
are missing, and you can run
|
||||
[`spacy init fill-config`](/api/cli#init-fill-config) to automatically fill in
|
||||
all defaults.
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Customizing the settings {id="transformers-training-custom-settings"}
|
||||
|
||||
To change any of the settings, you can edit the `config.cfg` and re-run the
|
||||
training. To change any of the functions, like the span getter, you can replace
|
||||
the name of the referenced function – e.g.
|
||||
`@span_getters = "spacy-transformers.sent_spans.v1"` to process sentences. You
|
||||
can also register your own functions using the
|
||||
[`span_getters` registry](/api/top-level#registry). For instance, the following
|
||||
custom function returns [`Span`](/api/span) objects following sentence
|
||||
boundaries, unless a sentence succeeds a certain amount of tokens, in which case
|
||||
subsentences of at most `max_length` tokens are returned.
|
||||
|
||||
> #### config.cfg
|
||||
>
|
||||
> ```ini
|
||||
> [components.transformer.model.get_spans]
|
||||
> @span_getters = "custom_sent_spans"
|
||||
> max_length = 25
|
||||
> ```
|
||||
|
||||
```python {title="code.py"}
|
||||
import spacy_transformers
|
||||
|
||||
@spacy_transformers.registry.span_getters("custom_sent_spans")
|
||||
def configure_custom_sent_spans(max_length: int):
|
||||
def get_custom_sent_spans(docs):
|
||||
spans = []
|
||||
for doc in docs:
|
||||
spans.append([])
|
||||
for sent in doc.sents:
|
||||
start = 0
|
||||
end = max_length
|
||||
while end <= len(sent):
|
||||
spans[-1].append(sent[start:end])
|
||||
start += max_length
|
||||
end += max_length
|
||||
if start < len(sent):
|
||||
spans[-1].append(sent[start:len(sent)])
|
||||
return spans
|
||||
|
||||
return get_custom_sent_spans
|
||||
```
|
||||
|
||||
To resolve the config during training, spaCy needs to know about your custom
|
||||
function. You can make it available via the `--code` argument that can point to
|
||||
a Python file. For more details on training with custom code, see the
|
||||
[training documentation](/usage/training#custom-functions).
|
||||
|
||||
```bash
|
||||
python -m spacy train ./config.cfg --code ./code.py
|
||||
```
|
||||
|
||||
### Customizing the model implementations {id="training-custom-model"}
|
||||
|
||||
The [`Transformer`](/api/transformer) component expects a Thinc
|
||||
[`Model`](https://thinc.ai/docs/api-model) object to be passed in as its `model`
|
||||
argument. You're not limited to the implementation provided by
|
||||
`spacy-transformers` – the only requirement is that your registered function
|
||||
must return an object of type ~~Model[List[Doc], FullTransformerBatch]~~: that
|
||||
is, a Thinc model that takes a list of [`Doc`](/api/doc) objects, and returns a
|
||||
[`FullTransformerBatch`](/api/transformer#fulltransformerbatch) object with the
|
||||
transformer data.
|
||||
|
||||
The same idea applies to task models that power the **downstream components**.
|
||||
Most of spaCy's built-in model creation functions support a `tok2vec` argument,
|
||||
which should be a Thinc layer of type ~~Model[List[Doc], List[Floats2d]]~~. This
|
||||
is where we'll plug in our transformer model, using the
|
||||
[TransformerListener](/api/architectures#TransformerListener) layer, which
|
||||
sneakily delegates to the `Transformer` pipeline component.
|
||||
|
||||
```ini {title="config.cfg (excerpt)",highlight="12"}
|
||||
[components.ner]
|
||||
factory = "ner"
|
||||
|
||||
[nlp.pipeline.ner.model]
|
||||
@architectures = "spacy.TransitionBasedParser.v1"
|
||||
state_type = "ner"
|
||||
extra_state_tokens = false
|
||||
hidden_width = 128
|
||||
maxout_pieces = 3
|
||||
use_upper = false
|
||||
|
||||
[nlp.pipeline.ner.model.tok2vec]
|
||||
@architectures = "spacy-transformers.TransformerListener.v1"
|
||||
grad_factor = 1.0
|
||||
|
||||
[nlp.pipeline.ner.model.tok2vec.pooling]
|
||||
@layers = "reduce_mean.v1"
|
||||
```
|
||||
|
||||
The [TransformerListener](/api/architectures#TransformerListener) layer expects
|
||||
a [pooling layer](https://thinc.ai/docs/api-layers#reduction-ops) as the
|
||||
argument `pooling`, which needs to be of type ~~Model[Ragged, Floats2d]~~. This
|
||||
layer determines how the vector for each spaCy token will be computed from the
|
||||
zero or more source rows the token is aligned against. Here we use the
|
||||
[`reduce_mean`](https://thinc.ai/docs/api-layers#reduce_mean) layer, which
|
||||
averages the wordpiece rows. We could instead use
|
||||
[`reduce_max`](https://thinc.ai/docs/api-layers#reduce_max), or a custom
|
||||
function you write yourself.
|
||||
|
||||
You can have multiple components all listening to the same transformer model,
|
||||
and all passing gradients back to it. By default, all of the gradients will be
|
||||
**equally weighted**. You can control this with the `grad_factor` setting, which
|
||||
lets you reweight the gradients from the different listeners. For instance,
|
||||
setting `grad_factor = 0` would disable gradients from one of the listeners,
|
||||
while `grad_factor = 2.0` would multiply them by 2. This is similar to having a
|
||||
custom learning rate for each component. Instead of a constant, you can also
|
||||
provide a schedule, allowing you to freeze the shared parameters at the start of
|
||||
training.
|
||||
|
||||
## Static vectors {id="static-vectors"}
|
||||
|
||||
If your pipeline includes a **word vectors table**, you'll be able to use the
|
||||
`.similarity()` method on the [`Doc`](/api/doc), [`Span`](/api/span),
|
||||
[`Token`](/api/token) and [`Lexeme`](/api/lexeme) objects. You'll also be able
|
||||
to access the vectors using the `.vector` attribute, or you can look up one or
|
||||
more vectors directly using the [`Vocab`](/api/vocab) object. Pipelines with
|
||||
word vectors can also **use the vectors as features** for the statistical
|
||||
models, which can **improve the accuracy** of your components.
|
||||
|
||||
Word vectors in spaCy are "static" in the sense that they are not learned
|
||||
parameters of the statistical models, and spaCy itself does not feature any
|
||||
algorithms for learning word vector tables. You can train a word vectors table
|
||||
using tools such as [floret](https://github.com/explosion/floret),
|
||||
[Gensim](https://radimrehurek.com/gensim/), [FastText](https://fasttext.cc/) or
|
||||
[GloVe](https://nlp.stanford.edu/projects/glove/), or download existing
|
||||
pretrained vectors. The [`init vectors`](/api/cli#init-vectors) command lets you
|
||||
convert vectors for use with spaCy and will give you a directory you can load or
|
||||
refer to in your [training configs](/usage/training#config).
|
||||
|
||||
<Infobox title="Word vectors and similarity" emoji="📖">
|
||||
|
||||
For more details on loading word vectors into spaCy, using them for similarity
|
||||
and improving word vector coverage by truncating and pruning the vectors, see
|
||||
the usage guide on
|
||||
[word vectors and similarity](/usage/linguistic-features#vectors-similarity).
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Using word vectors in your models {id="word-vectors-models"}
|
||||
|
||||
Many neural network models are able to use word vector tables as additional
|
||||
features, which sometimes results in significant improvements in accuracy.
|
||||
spaCy's built-in embedding layer,
|
||||
[MultiHashEmbed](/api/architectures#MultiHashEmbed), can be configured to use
|
||||
word vector tables using the `include_static_vectors` flag.
|
||||
|
||||
```ini
|
||||
[tagger.model.tok2vec.embed]
|
||||
@architectures = "spacy.MultiHashEmbed.v2"
|
||||
width = 128
|
||||
attrs = ["LOWER","PREFIX","SUFFIX","SHAPE"]
|
||||
rows = [5000,2500,2500,2500]
|
||||
include_static_vectors = true
|
||||
```
|
||||
|
||||
<Infobox title="How it works" emoji="💡">
|
||||
|
||||
The configuration system will look up the string `"spacy.MultiHashEmbed.v2"` in
|
||||
the `architectures` [registry](/api/top-level#registry), and call the returned
|
||||
object with the rest of the arguments from the block. This will result in a call
|
||||
to the
|
||||
[`MultiHashEmbed`](https://github.com/explosion/spacy/tree/develop/spacy/ml/models/tok2vec.py)
|
||||
function, which will return a [Thinc](https://thinc.ai) model object with the
|
||||
type signature ~~Model[List[Doc], List[Floats2d]]~~. Because the embedding layer
|
||||
takes a list of `Doc` objects as input, it does not need to store a copy of the
|
||||
vectors table. The vectors will be retrieved from the `Doc` objects that are
|
||||
passed in, via the `doc.vocab.vectors` attribute. This part of the process is
|
||||
handled by the [StaticVectors](/api/architectures#StaticVectors) layer.
|
||||
|
||||
</Infobox>
|
||||
|
||||
#### Creating a custom embedding layer {id="custom-embedding-layer"}
|
||||
|
||||
The [MultiHashEmbed](/api/architectures#StaticVectors) layer is spaCy's
|
||||
recommended strategy for constructing initial word representations for your
|
||||
neural network models, but you can also implement your own. You can register any
|
||||
function to a string name, and then reference that function within your config
|
||||
(see the [training docs](/usage/training) for more details). To try this out,
|
||||
you can save the following little example to a new Python file:
|
||||
|
||||
```python
|
||||
from spacy.ml.staticvectors import StaticVectors
|
||||
from spacy.util import registry
|
||||
|
||||
print("I was imported!")
|
||||
|
||||
@registry.architectures("my_example.MyEmbedding.v1")
|
||||
def MyEmbedding(output_width: int) -> Model[List[Doc], List[Floats2d]]:
|
||||
print("I was called!")
|
||||
return StaticVectors(nO=output_width)
|
||||
```
|
||||
|
||||
If you pass the path to your file to the [`spacy train`](/api/cli#train) command
|
||||
using the `--code` argument, your file will be imported, which means the
|
||||
decorator registering the function will be run. Your function is now on equal
|
||||
footing with any of spaCy's built-ins, so you can drop it in instead of any
|
||||
other model with the same input and output signature. For instance, you could
|
||||
use it in the tagger model as follows:
|
||||
|
||||
```ini
|
||||
[tagger.model.tok2vec.embed]
|
||||
@architectures = "my_example.MyEmbedding.v1"
|
||||
output_width = 128
|
||||
```
|
||||
|
||||
Now that you have a custom function wired into the network, you can start
|
||||
implementing the logic you're interested in. For example, let's say you want to
|
||||
try a relatively simple embedding strategy that makes use of static word
|
||||
vectors, but combines them via summation with a smaller table of learned
|
||||
embeddings.
|
||||
|
||||
```python
|
||||
from thinc.api import add, chain, remap_ids, Embed
|
||||
from spacy.ml.staticvectors import StaticVectors
|
||||
from spacy.ml.featureextractor import FeatureExtractor
|
||||
from spacy.util import registry
|
||||
|
||||
@registry.architectures("my_example.MyEmbedding.v1")
|
||||
def MyCustomVectors(
|
||||
output_width: int,
|
||||
vector_width: int,
|
||||
embed_rows: int,
|
||||
key2row: Dict[int, int]
|
||||
) -> Model[List[Doc], List[Floats2d]]:
|
||||
return add(
|
||||
StaticVectors(nO=output_width),
|
||||
chain(
|
||||
FeatureExtractor(["ORTH"]),
|
||||
remap_ids(key2row),
|
||||
Embed(nO=output_width, nV=embed_rows)
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
#### Creating a custom vectors implementation {id="custom-vectors",version="3.7"}
|
||||
|
||||
You can specify a custom registered vectors class under `[nlp.vectors]` in order
|
||||
to use static vectors in formats other than the ones supported by
|
||||
[`Vectors`](/api/vectors). Extend the abstract [`BaseVectors`](/api/basevectors)
|
||||
class to implement your custom vectors.
|
||||
|
||||
As an example, the following `BPEmbVectors` class implements support for
|
||||
[BPEmb subword embeddings](https://bpemb.h-its.org/):
|
||||
|
||||
```python
|
||||
# requires: pip install bpemb
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
from typing import Callable, Optional, cast
|
||||
|
||||
from bpemb import BPEmb
|
||||
from thinc.api import Ops, get_current_ops
|
||||
from thinc.backends import get_array_ops
|
||||
from thinc.types import Floats2d
|
||||
|
||||
from spacy.strings import StringStore
|
||||
from spacy.util import registry
|
||||
from spacy.vectors import BaseVectors
|
||||
from spacy.vocab import Vocab
|
||||
|
||||
|
||||
class BPEmbVectors(BaseVectors):
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
strings: Optional[StringStore] = None,
|
||||
lang: Optional[str] = None,
|
||||
vs: Optional[int] = None,
|
||||
dim: Optional[int] = None,
|
||||
cache_dir: Optional[Path] = None,
|
||||
encode_extra_options: Optional[str] = None,
|
||||
model_file: Optional[Path] = None,
|
||||
emb_file: Optional[Path] = None,
|
||||
):
|
||||
kwargs = {}
|
||||
if lang is not None:
|
||||
kwargs["lang"] = lang
|
||||
if vs is not None:
|
||||
kwargs["vs"] = vs
|
||||
if dim is not None:
|
||||
kwargs["dim"] = dim
|
||||
if cache_dir is not None:
|
||||
kwargs["cache_dir"] = cache_dir
|
||||
if encode_extra_options is not None:
|
||||
kwargs["encode_extra_options"] = encode_extra_options
|
||||
if model_file is not None:
|
||||
kwargs["model_file"] = model_file
|
||||
if emb_file is not None:
|
||||
kwargs["emb_file"] = emb_file
|
||||
self.bpemb = BPEmb(**kwargs)
|
||||
self.strings = strings
|
||||
self.name = repr(self.bpemb)
|
||||
self.n_keys = -1
|
||||
self.mode = "BPEmb"
|
||||
self.to_ops(get_current_ops())
|
||||
|
||||
def __contains__(self, key):
|
||||
return True
|
||||
|
||||
def is_full(self):
|
||||
return True
|
||||
|
||||
def add(self, key, *, vector=None, row=None):
|
||||
warnings.warn(
|
||||
(
|
||||
"Skipping BPEmbVectors.add: the bpemb vector table cannot be "
|
||||
"modified. Vectors are calculated from bytepieces."
|
||||
)
|
||||
)
|
||||
return -1
|
||||
|
||||
def __getitem__(self, key):
|
||||
return self.get_batch([key])[0]
|
||||
|
||||
def get_batch(self, keys):
|
||||
keys = [self.strings.as_string(key) for key in keys]
|
||||
bp_ids = self.bpemb.encode_ids(keys)
|
||||
ops = get_array_ops(self.bpemb.emb.vectors)
|
||||
indices = ops.asarray(ops.xp.hstack(bp_ids), dtype="int32")
|
||||
lengths = ops.asarray([len(x) for x in bp_ids], dtype="int32")
|
||||
vecs = ops.reduce_mean(cast(Floats2d, self.bpemb.emb.vectors[indices]), lengths)
|
||||
return vecs
|
||||
|
||||
@property
|
||||
def shape(self):
|
||||
return self.bpemb.vectors.shape
|
||||
|
||||
def __len__(self):
|
||||
return self.shape[0]
|
||||
|
||||
@property
|
||||
def vectors_length(self):
|
||||
return self.shape[1]
|
||||
|
||||
@property
|
||||
def size(self):
|
||||
return self.bpemb.vectors.size
|
||||
|
||||
def to_ops(self, ops: Ops):
|
||||
self.bpemb.emb.vectors = ops.asarray(self.bpemb.emb.vectors)
|
||||
|
||||
|
||||
@registry.vectors("BPEmbVectors.v1")
|
||||
def create_bpemb_vectors(
|
||||
lang: Optional[str] = "multi",
|
||||
vs: Optional[int] = None,
|
||||
dim: Optional[int] = None,
|
||||
cache_dir: Optional[Path] = None,
|
||||
encode_extra_options: Optional[str] = None,
|
||||
model_file: Optional[Path] = None,
|
||||
emb_file: Optional[Path] = None,
|
||||
) -> Callable[[Vocab], BPEmbVectors]:
|
||||
def bpemb_vectors_factory(vocab: Vocab) -> BPEmbVectors:
|
||||
return BPEmbVectors(
|
||||
strings=vocab.strings,
|
||||
lang=lang,
|
||||
vs=vs,
|
||||
dim=dim,
|
||||
cache_dir=cache_dir,
|
||||
encode_extra_options=encode_extra_options,
|
||||
model_file=model_file,
|
||||
emb_file=emb_file,
|
||||
)
|
||||
|
||||
return bpemb_vectors_factory
|
||||
```
|
||||
|
||||
<Infobox variant="warning">
|
||||
|
||||
Note that the serialization methods are not implemented, so the embeddings are
|
||||
loaded from your local cache or downloaded by `BPEmb` each time the pipeline is
|
||||
loaded.
|
||||
|
||||
</Infobox>
|
||||
|
||||
To use this in your pipeline, specify this registered function under
|
||||
`[nlp.vectors]` in your config:
|
||||
|
||||
```ini
|
||||
[nlp.vectors]
|
||||
@vectors = "BPEmbVectors.v1"
|
||||
lang = "en"
|
||||
```
|
||||
|
||||
Or specify it when creating a blank pipeline:
|
||||
|
||||
```python
|
||||
nlp = spacy.blank("en", config={"nlp.vectors": {"@vectors": "BPEmbVectors.v1", "lang": "en"}})
|
||||
```
|
||||
|
||||
Remember to include this code with `--code` when using
|
||||
[`spacy train`](/api/cli#train) and [`spacy package`](/api/cli#package).
|
||||
|
||||
## Pretraining {id="pretraining"}
|
||||
|
||||
The [`spacy pretrain`](/api/cli#pretrain) command lets you initialize your
|
||||
models with **information from raw text**. Without pretraining, the models for
|
||||
your components will usually be initialized randomly. The idea behind
|
||||
pretraining is simple: random probably isn't optimal, so if we have some text to
|
||||
learn from, we can probably find a way to get the model off to a better start.
|
||||
|
||||
Pretraining uses the same [`config.cfg`](/usage/training#config) file as the
|
||||
regular training, which helps keep the settings and hyperparameters consistent.
|
||||
The additional `[pretraining]` section has several configuration subsections
|
||||
that are familiar from the training block: the `[pretraining.batcher]`,
|
||||
`[pretraining.optimizer]` and `[pretraining.corpus]` all work the same way and
|
||||
expect the same types of objects, although for pretraining your corpus does not
|
||||
need to have any annotations, so you will often use a different reader, such as
|
||||
the [`JsonlCorpus`](/api/top-level#jsonlcorpus).
|
||||
|
||||
> #### Raw text format
|
||||
>
|
||||
> The raw text can be provided in spaCy's
|
||||
> [binary `.spacy` format](/api/data-formats#training) consisting of serialized
|
||||
> `Doc` objects or as a JSONL (newline-delimited JSON) with a key `"text"` per
|
||||
> entry. This allows the data to be read in line by line, while also allowing
|
||||
> you to include newlines in the texts.
|
||||
>
|
||||
> ```json
|
||||
> {"text": "Can I ask where you work now and what you do, and if you enjoy it?"}
|
||||
> {"text": "They may just pull out of the Seattle market completely, at least until they have autonomous vehicles."}
|
||||
> ```
|
||||
>
|
||||
> You can also use your own custom corpus loader instead.
|
||||
|
||||
You can add a `[pretraining]` block to your config by setting the
|
||||
`--pretraining` flag on [`init config`](/api/cli#init-config) or
|
||||
[`init fill-config`](/api/cli#init-fill-config):
|
||||
|
||||
```bash
|
||||
$ python -m spacy init fill-config config.cfg config_pretrain.cfg --pretraining
|
||||
```
|
||||
|
||||
You can then run [`spacy pretrain`](/api/cli#pretrain) with the updated config
|
||||
and pass in optional config overrides, like the path to the raw text file:
|
||||
|
||||
```bash
|
||||
$ python -m spacy pretrain config_pretrain.cfg ./output --paths.raw_text text.jsonl
|
||||
```
|
||||
|
||||
The following defaults are used for the `[pretraining]` block and merged into
|
||||
your existing config when you run [`init config`](/api/cli#init-config) or
|
||||
[`init fill-config`](/api/cli#init-fill-config) with `--pretraining`. If needed,
|
||||
you can [configure](#pretraining-configure) the settings and hyperparameters or
|
||||
change the [objective](#pretraining-objectives).
|
||||
|
||||
```ini
|
||||
%%GITHUB_SPACY/spacy/default_config_pretraining.cfg
|
||||
```
|
||||
|
||||
### How pretraining works {id="pretraining-details"}
|
||||
|
||||
The impact of [`spacy pretrain`](/api/cli#pretrain) varies, but it will usually
|
||||
be worth trying if you're **not using a transformer** model and you have
|
||||
**relatively little training data** (for instance, fewer than 5,000 sentences).
|
||||
A good rule of thumb is that pretraining will generally give you a similar
|
||||
accuracy improvement to using word vectors in your model. If word vectors have
|
||||
given you a 10% error reduction, pretraining with spaCy might give you another
|
||||
10%, for a 20% error reduction in total.
|
||||
|
||||
The [`spacy pretrain`](/api/cli#pretrain) command will take a **specific
|
||||
subnetwork** within one of your components, and add additional layers to build a
|
||||
network for a temporary task that forces the model to learn something about
|
||||
sentence structure and word cooccurrence statistics.
|
||||
|
||||
Pretraining produces a **binary weights file** that can be loaded back in at the
|
||||
start of training, using the configuration option `initialize.init_tok2vec`. The
|
||||
weights file specifies an initial set of weights. Training then proceeds as
|
||||
normal.
|
||||
|
||||
You can only pretrain one subnetwork from your pipeline at a time, and the
|
||||
subnetwork must be typed ~~Model[List[Doc], List[Floats2d]]~~ (i.e. it has to be
|
||||
a "tok2vec" layer). The most common workflow is to use the
|
||||
[`Tok2Vec`](/api/tok2vec) component to create a shared token-to-vector layer for
|
||||
several components of your pipeline, and apply pretraining to its whole model.
|
||||
|
||||
#### Configuring the pretraining {id="pretraining-configure"}
|
||||
|
||||
The [`spacy pretrain`](/api/cli#pretrain) command is configured using the
|
||||
`[pretraining]` section of your [config file](/usage/training#config). The
|
||||
`component` and `layer` settings tell spaCy how to **find the subnetwork** to
|
||||
pretrain. The `layer` setting should be either the empty string (to use the
|
||||
whole model), or a
|
||||
[node reference](https://thinc.ai/docs/usage-models#model-state). Most of
|
||||
spaCy's built-in model architectures have a reference named `"tok2vec"` that
|
||||
will refer to the right layer.
|
||||
|
||||
```ini {title="config.cfg"}
|
||||
# 1. Use the whole model of the "tok2vec" component
|
||||
[pretraining]
|
||||
component = "tok2vec"
|
||||
layer = ""
|
||||
|
||||
# 2. Pretrain the "tok2vec" node of the "textcat" component
|
||||
[pretraining]
|
||||
component = "textcat"
|
||||
layer = "tok2vec"
|
||||
```
|
||||
|
||||
#### Connecting pretraining to training {id="pretraining-training"}
|
||||
|
||||
To benefit from pretraining, your training step needs to know to initialize its
|
||||
`tok2vec` component with the weights learned from the pretraining step. You do
|
||||
this by setting `initialize.init_tok2vec` to the filename of the `.bin` file
|
||||
that you want to use from pretraining.
|
||||
|
||||
A pretraining step that runs for 5 epochs with an output path of `pretrain/`, as
|
||||
an example, produces `pretrain/model0.bin` through `pretrain/model4.bin` plus a
|
||||
copy of the last iteration as `pretrain/model-last.bin`. Additionally, you can
|
||||
configure `n_save_epoch` to tell pretraining in which epoch interval it should
|
||||
save the current training progress. To use the final output to initialize your
|
||||
`tok2vec` layer, you could fill in this value in your config file:
|
||||
|
||||
```ini {title="config.cfg"}
|
||||
|
||||
[paths]
|
||||
init_tok2vec = "pretrain/model-last.bin"
|
||||
|
||||
[initialize]
|
||||
init_tok2vec = ${paths.init_tok2vec}
|
||||
```
|
||||
|
||||
<Infobox variant="warning">
|
||||
|
||||
The outputs of `spacy pretrain` are not the same data format as the pre-packaged
|
||||
static word vectors that would go into
|
||||
[`initialize.vectors`](/api/data-formats#config-initialize). The pretraining
|
||||
output consists of the weights that the `tok2vec` component should start with in
|
||||
an existing pipeline, so it goes in `initialize.init_tok2vec`.
|
||||
|
||||
</Infobox>
|
||||
|
||||
#### Pretraining objectives {id="pretraining-objectives"}
|
||||
|
||||
> ```ini
|
||||
> ### Characters objective
|
||||
> [pretraining.objective]
|
||||
> @architectures = "spacy.PretrainCharacters.v1"
|
||||
> maxout_pieces = 3
|
||||
> hidden_size = 300
|
||||
> n_characters = 4
|
||||
> ```
|
||||
>
|
||||
> ```ini
|
||||
> ### Vectors objective
|
||||
> [pretraining.objective]
|
||||
> @architectures = "spacy.PretrainVectors.v1"
|
||||
> maxout_pieces = 3
|
||||
> hidden_size = 300
|
||||
> loss = "cosine"
|
||||
> ```
|
||||
|
||||
Two pretraining objectives are available, both of which are variants of the
|
||||
cloze task [Devlin et al. (2018)](https://arxiv.org/abs/1810.04805) introduced
|
||||
for BERT. The objective can be defined and configured via the
|
||||
`[pretraining.objective]` config block.
|
||||
|
||||
- [`PretrainCharacters`](/api/architectures#pretrain_chars): The `"characters"`
|
||||
objective asks the model to predict some number of leading and trailing UTF-8
|
||||
bytes for the words. For instance, setting `n_characters = 2`, the model will
|
||||
try to predict the first two and last two characters of the word.
|
||||
|
||||
- [`PretrainVectors`](/api/architectures#pretrain_vectors): The `"vectors"`
|
||||
objective asks the model to predict the word's vector, from a static
|
||||
embeddings table. This requires a word vectors model to be trained and loaded.
|
||||
The vectors objective can optimize either a cosine or an L2 loss. We've
|
||||
generally found cosine loss to perform better.
|
||||
|
||||
These pretraining objectives use a trick that we term **language modelling with
|
||||
approximate outputs (LMAO)**. The motivation for the trick is that predicting an
|
||||
exact word ID introduces a lot of incidental complexity. You need a large output
|
||||
layer, and even then, the vocabulary is too large, which motivates tokenization
|
||||
schemes that do not align to actual word boundaries. At the end of training, the
|
||||
output layer will be thrown away regardless: we just want a task that forces the
|
||||
network to model something about word cooccurrence statistics. Predicting
|
||||
leading and trailing characters does that more than adequately, as the exact
|
||||
word sequence could be recovered with high accuracy if the initial and trailing
|
||||
characters are predicted accurately. With the vectors objective, the pretraining
|
||||
uses the embedding space learned by an algorithm such as
|
||||
[GloVe](https://nlp.stanford.edu/projects/glove/) or
|
||||
[Word2vec](https://code.google.com/archive/p/word2vec/), allowing the model to
|
||||
focus on the contextual modelling we actual care about.
|
||||
@@ -0,0 +1,116 @@
|
||||
---
|
||||
title: Facts & Figures
|
||||
teaser: The hard numbers for spaCy and how it compares to other tools
|
||||
next: /usage/spacy-101
|
||||
menu:
|
||||
- ['Feature Comparison', 'comparison']
|
||||
- ['Benchmarks', 'benchmarks']
|
||||
# TODO: - ['Citing spaCy', 'citation']
|
||||
---
|
||||
|
||||
## Comparison {id="comparison",hidden="true"}
|
||||
|
||||
spaCy is a **free, open-source library** for advanced **Natural Language
|
||||
Processing** (NLP) in Python. It's designed specifically for **production use**
|
||||
and helps you build applications that process and "understand" large volumes of
|
||||
text. It can be used to build information extraction or natural language
|
||||
understanding systems.
|
||||
|
||||
### Feature overview {id="comparison-features"}
|
||||
|
||||
<Features />
|
||||
|
||||
### When should I use spaCy? {id="comparison-usage"}
|
||||
|
||||
- ✅ **I'm a beginner and just getting started with NLP.** – spaCy makes it easy
|
||||
to get started and comes with extensive documentation, including a
|
||||
beginner-friendly [101 guide](/usage/spacy-101), a free interactive
|
||||
[online course](https://course.spacy.io) and a range of
|
||||
[video tutorials](https://www.youtube.com/c/ExplosionAI).
|
||||
- ✅ **I want to build an end-to-end production application.** – spaCy is
|
||||
specifically designed for production use and lets you build and train powerful
|
||||
NLP pipelines and package them for easy deployment.
|
||||
- ✅ **I want my application to be efficient on GPU _and_ CPU.** – While spaCy
|
||||
lets you train modern NLP models that are best run on GPU, it also offers
|
||||
CPU-optimized pipelines, which are less accurate but much cheaper to run.
|
||||
- ✅ **I want to try out different neural network architectures for NLP.** –
|
||||
spaCy lets you customize and swap out the model architectures powering its
|
||||
components, and implement your own using a framework like PyTorch or
|
||||
TensorFlow. The declarative configuration system makes it easy to mix and
|
||||
match functions and keep track of your hyperparameters to make sure your
|
||||
experiments are reproducible.
|
||||
- ❌ **I want to build a language generation application.** – spaCy's focus is
|
||||
natural language _processing_ and extracting information from large volumes of
|
||||
text. While you can use it to help you re-write existing text, it doesn't
|
||||
include any specific functionality for language generation tasks.
|
||||
- ❌ **I want to research machine learning algorithms.** spaCy is built on the
|
||||
latest research, but it's not a research library. If your goal is to write
|
||||
papers and run benchmarks, spaCy is probably not a good choice. However, you
|
||||
can use it to make the results of your research easily available for others to
|
||||
use, e.g. via a custom spaCy component.
|
||||
|
||||
## Benchmarks {id="benchmarks"}
|
||||
|
||||
spaCy v3.0 introduces transformer-based pipelines that bring spaCy's accuracy
|
||||
right up to **current state-of-the-art**. You can also use a CPU-optimized
|
||||
pipeline, which is less accurate but much cheaper to run.
|
||||
|
||||
{/* TODO: update benchmarks and intro */}
|
||||
|
||||
> #### Evaluation details
|
||||
>
|
||||
> - **OntoNotes 5.0:** spaCy's English models are trained on this corpus, as
|
||||
> it's several times larger than other English treebanks. However, most
|
||||
> systems do not report accuracies on it.
|
||||
> - **Penn Treebank:** The "classic" parsing evaluation for research. However,
|
||||
> it's quite far removed from actual usage: it uses sentences with
|
||||
> gold-standard segmentation and tokenization, from a pretty specific type of
|
||||
> text (articles from a single newspaper, 1984-1989).
|
||||
|
||||
<Benchmarks />
|
||||
|
||||
<figure>
|
||||
|
||||
| Dependency Parsing System | UAS | LAS |
|
||||
| ------------------------------------------------------------------------------ | ---: | ---: |
|
||||
| spaCy RoBERTa (2020) | 95.1 | 93.7 |
|
||||
| [Mrini et al.](https://khalilmrini.github.io/Label_Attention_Layer.pdf) (2019) | 97.4 | 96.3 |
|
||||
| [Zhou and Zhao](https://www.aclweb.org/anthology/P19-1230/) (2019) | 97.2 | 95.7 |
|
||||
|
||||
<figcaption className="caption">
|
||||
|
||||
**Dependency parsing accuracy** on the Penn Treebank. See
|
||||
[NLP-progress](http://nlpprogress.com/english/dependency_parsing.html) for more
|
||||
results. Project template:
|
||||
[`benchmarks/parsing_penn_treebank`](%%GITHUB_PROJECTS/benchmarks/parsing_penn_treebank).
|
||||
|
||||
</figcaption>
|
||||
|
||||
</figure>
|
||||
|
||||
### Speed comparison {id="benchmarks-speed"}
|
||||
|
||||
We compare the speed of different NLP libraries, measured in words per second
|
||||
(WPS) - higher is better. The evaluation was performed on 10,000 Reddit
|
||||
comments.
|
||||
|
||||
<figure>
|
||||
|
||||
| Library | Pipeline | WPS CPU <Help>words per second on CPU, higher is better</Help> | WPS GPU <Help>words per second on GPU, higher is better</Help> |
|
||||
| ------- | ----------------------------------------------- | -------------------------------------------------------------: | -------------------------------------------------------------: |
|
||||
| spaCy | [`en_core_web_lg`](/models/en#en_core_web_lg) | 10,014 | 14,954 |
|
||||
| spaCy | [`en_core_web_trf`](/models/en#en_core_web_trf) | 684 | 3,768 |
|
||||
| Stanza | `en_ewt` | 878 | 2,180 |
|
||||
| Flair | `pos`(`-fast`) & `ner`(`-fast`) | 323 | 1,184 |
|
||||
| UDPipe | `english-ewt-ud-2.5` | 1,101 | _n/a_ |
|
||||
|
||||
<figcaption className="caption">
|
||||
|
||||
**End-to-end processing speed** on raw unannotated text. Project template:
|
||||
[`benchmarks/speed`](%%GITHUB_PROJECTS/benchmarks/speed).
|
||||
|
||||
</figcaption>
|
||||
|
||||
</figure>
|
||||
|
||||
{/* TODO: ## Citing spaCy {id="citation"} */}
|
||||
@@ -0,0 +1,470 @@
|
||||
---
|
||||
title: Install spaCy
|
||||
next: /usage/models
|
||||
menu:
|
||||
- ['Quickstart', 'quickstart']
|
||||
- ['Instructions', 'installation']
|
||||
- ['Troubleshooting', 'troubleshooting']
|
||||
- ['Changelog', 'changelog']
|
||||
---
|
||||
|
||||
## Quickstart {hidden="true"}
|
||||
|
||||
> #### 📖 Looking for the old docs?
|
||||
>
|
||||
> To help you make the transition from v2.x to v3.0, we've uploaded the old
|
||||
> website to [**v2.spacy.io**](https://v2.spacy.io/docs). To see what's changed
|
||||
> and how to migrate, see the [v3.0 guide](/usage/v3).
|
||||
|
||||
<QuickstartInstall id="quickstart" />
|
||||
|
||||
## Installation instructions {id="installation"}
|
||||
|
||||
spaCy is compatible with **64-bit CPython 3.7+** and runs on **Unix/Linux**,
|
||||
**macOS/OS X** and **Windows**. The latest spaCy releases are available over
|
||||
[pip](https://pypi.python.org/pypi/spacy) and
|
||||
[conda](https://anaconda.org/conda-forge/spacy).
|
||||
|
||||
### pip {id="pip"}
|
||||
|
||||
Using pip, spaCy releases are available as source packages and binary wheels.
|
||||
Before you install spaCy and its dependencies, make sure that your `pip`,
|
||||
`setuptools` and `wheel` are up to date.
|
||||
|
||||
> #### Download pipelines
|
||||
>
|
||||
> After installation you typically want to download a trained pipeline. For more
|
||||
> info and available packages, see the [models directory](/models).
|
||||
>
|
||||
> ```bash
|
||||
> $ python -m spacy download en_core_web_sm
|
||||
>
|
||||
> >>> import spacy
|
||||
> >>> nlp = spacy.load("en_core_web_sm")
|
||||
> ```
|
||||
|
||||
```bash
|
||||
$ pip install -U pip setuptools wheel
|
||||
$ pip install -U %%SPACY_PKG_NAME%%SPACY_PKG_FLAGS
|
||||
```
|
||||
|
||||
When using pip it is generally recommended to install packages in a virtual
|
||||
environment to avoid modifying system state:
|
||||
|
||||
```bash
|
||||
$ python -m venv .env
|
||||
$ source .env/bin/activate
|
||||
$ pip install -U pip setuptools wheel
|
||||
$ pip install -U %%SPACY_PKG_NAME%%SPACY_PKG_FLAGS
|
||||
```
|
||||
|
||||
spaCy also lets you install extra dependencies by specifying the following
|
||||
keywords in brackets, e.g. `spacy[ja]` or `spacy[lookups,transformers]` (with
|
||||
multiple comma-separated extras). See the `[options.extras_require]` section in
|
||||
spaCy's [`setup.cfg`](%%GITHUB_SPACY/setup.cfg) for details on what's included.
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```bash
|
||||
> $ pip install %%SPACY_PKG_NAME[lookups,transformers]%%SPACY_PKG_FLAGS
|
||||
> ```
|
||||
|
||||
| Name | Description |
|
||||
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `lookups` | Install [`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data) for data tables for lemmatization and lexeme normalization. The data is serialized with trained pipelines, so you only need this package if you want to train your own models. |
|
||||
| `transformers` | Install [`spacy-transformers`](https://github.com/explosion/spacy-transformers). The package will be installed automatically when you install a transformer-based pipeline. |
|
||||
| `cuda`, ... | Install spaCy with GPU support provided by [CuPy](https://cupy.chainer.org) for your given CUDA version. See the GPU [installation instructions](#gpu) for details and options. |
|
||||
| `apple` | Install [`thinc-apple-ops`](https://github.com/explosion/thinc-apple-ops) to improve performance on an Apple M1. |
|
||||
| `ja`, `ko`, `th` | Install additional dependencies required for tokenization for the [languages](/usage/models#languages). |
|
||||
|
||||
### conda {id="conda"}
|
||||
|
||||
Thanks to our great community, we've been able to re-add conda support. You can
|
||||
also install spaCy via `conda-forge`:
|
||||
|
||||
```bash
|
||||
$ conda install -c conda-forge spacy
|
||||
```
|
||||
|
||||
For the feedstock including the build recipe and configuration, check out
|
||||
[this repository](https://github.com/conda-forge/spacy-feedstock). Note that we
|
||||
currently don't publish any [pre-releases](#changelog-pre) on conda.
|
||||
|
||||
### Upgrading spaCy {id="upgrading"}
|
||||
|
||||
> #### Upgrading from v2 to v3
|
||||
>
|
||||
> Although we've tried to keep breaking changes to a minimum, upgrading from
|
||||
> spaCy v2.x to v3.x may still require some changes to your code base. For
|
||||
> details see the sections on [backwards incompatibilities](/usage/v3#incompat)
|
||||
> and [migrating](/usage/v3#migrating). Also remember to download the new
|
||||
> trained pipelines, and retrain your own pipelines.
|
||||
|
||||
When updating to a newer version of spaCy, it's generally recommended to start
|
||||
with a clean virtual environment. If you're upgrading to a new major version,
|
||||
make sure you have the latest **compatible trained pipelines** installed, and
|
||||
that there are no old and incompatible packages left over in your environment,
|
||||
as this can often lead to unexpected results and errors. If you've trained your
|
||||
own models, keep in mind that your train and runtime inputs must match. This
|
||||
means you'll have to **retrain your pipelines** with the new version.
|
||||
|
||||
spaCy also provides a [`validate`](/api/cli#validate) command, which lets you
|
||||
verify that all installed pipeline packages are compatible with your spaCy
|
||||
version. If incompatible packages are found, tips and installation instructions
|
||||
are printed. It's recommended to run the command with `python -m` to make sure
|
||||
you're executing the correct version of spaCy.
|
||||
|
||||
```bash
|
||||
$ pip install -U %%SPACY_PKG_NAME%%SPACY_PKG_FLAGS
|
||||
$ python -m spacy validate
|
||||
```
|
||||
|
||||
### Run spaCy with GPU {id="gpu",version="2.0.14"}
|
||||
|
||||
As of v2.0, spaCy comes with neural network models that are implemented in our
|
||||
machine learning library, [Thinc](https://thinc.ai). For GPU support, we've been
|
||||
grateful to use the work of Chainer's [CuPy](https://cupy.chainer.org) module,
|
||||
which provides a numpy-compatible interface for GPU arrays.
|
||||
|
||||
spaCy can be installed for a CUDA-compatible GPU by specifying `spacy[cuda]`,
|
||||
`spacy[cuda102]`, `spacy[cuda112]`, `spacy[cuda113]`, etc. If you know your CUDA
|
||||
version, using the more explicit specifier allows CuPy to be installed via
|
||||
wheel, saving some compilation time. The specifiers should install
|
||||
[`cupy`](https://cupy.chainer.org).
|
||||
|
||||
```bash
|
||||
$ pip install -U %%SPACY_PKG_NAME[cuda113]%%SPACY_PKG_FLAGS
|
||||
```
|
||||
|
||||
Once you have a GPU-enabled installation, the best way to activate it is to call
|
||||
[`spacy.prefer_gpu`](/api/top-level#spacy.prefer_gpu) or
|
||||
[`spacy.require_gpu()`](/api/top-level#spacy.require_gpu) somewhere in your
|
||||
script before any pipelines have been loaded. `require_gpu` will raise an error
|
||||
if no GPU is available.
|
||||
|
||||
```python
|
||||
import spacy
|
||||
|
||||
spacy.prefer_gpu()
|
||||
nlp = spacy.load("en_core_web_sm")
|
||||
```
|
||||
|
||||
### Compile from source {id="source"}
|
||||
|
||||
The other way to install spaCy is to clone its
|
||||
[GitHub repository](https://github.com/explosion/spaCy) and build it from
|
||||
source. That is the common way if you want to make changes to the code base.
|
||||
You'll need to make sure that you have a development environment consisting of a
|
||||
Python distribution including header files, a compiler,
|
||||
[pip](https://pip.pypa.io/en/stable/) and [git](https://git-scm.com) installed.
|
||||
The compiler part is the trickiest. How to do that depends on your system. See
|
||||
notes on [Ubuntu](#source-ubuntu), [macOS / OS X](#source-osx) and
|
||||
[Windows](#source-windows) for details.
|
||||
|
||||
```bash
|
||||
$ python -m pip install -U pip setuptools wheel # install/update build tools
|
||||
$ git clone https://github.com/explosion/spaCy # clone spaCy
|
||||
$ cd spaCy # navigate into dir
|
||||
$ python -m venv .env # create environment in .env
|
||||
$ source .env/bin/activate # activate virtual env
|
||||
$ pip install -r requirements.txt # install requirements
|
||||
$ pip install --no-build-isolation --editable . # compile and install spaCy
|
||||
```
|
||||
|
||||
To install with extras:
|
||||
|
||||
```bash
|
||||
$ pip install --no-build-isolation --editable .[lookups,cuda102]
|
||||
```
|
||||
|
||||
How to install compilers and related build tools:
|
||||
|
||||
- <strong id="source-ubuntu">Ubuntu:</strong> Install system-level dependencies via
|
||||
`apt-get`: `sudo apt-get install build-essential python-dev git`
|
||||
- <strong id="source-osx">macOS / OS X:</strong> Install a recent version of [XCode](https://developer.apple.com/xcode/),
|
||||
including the so-called "Command Line Tools". macOS and OS X ship with Python and
|
||||
Git preinstalled.
|
||||
- <strong id="source-windows">Windows:</strong> Install a version of the [Visual
|
||||
C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/) or
|
||||
[Visual Studio Express](https://www.visualstudio.com/vs/visual-studio-express/)
|
||||
that matches the version that was used to compile your Python interpreter.
|
||||
|
||||
#### Using build constraints when compiling from source
|
||||
|
||||
If you install spaCy from source or with `pip` for platforms where there are not
|
||||
binary wheels on PyPI, you may need to use build constraints if any package in
|
||||
your environment requires an older version of `numpy`.
|
||||
|
||||
If `numpy` gets downgraded from the most recent release at any point after
|
||||
you've compiled `spacy`, you might see an error that looks like this:
|
||||
|
||||
```
|
||||
numpy.ndarray size changed, may indicate binary incompatibility.
|
||||
```
|
||||
|
||||
To fix this, create a new virtual environment and install `spacy` and all of its
|
||||
dependencies using build constraints.
|
||||
[Build constraints](https://pip.pypa.io/en/stable/user_guide/#constraints-files)
|
||||
specify an older version of `numpy` that is only used while compiling `spacy`,
|
||||
and then your runtime environment can use any newer version of `numpy` and still
|
||||
be compatible. In addition, use `--no-cache-dir` to ignore any previously cached
|
||||
wheels so that all relevant packages are recompiled from scratch:
|
||||
|
||||
```shell
|
||||
PIP_CONSTRAINT=https://raw.githubusercontent.com/explosion/spacy/master/build-constraints.txt \
|
||||
pip install spacy --no-cache-dir
|
||||
```
|
||||
|
||||
Our build constraints currently specify the oldest supported `numpy` available
|
||||
on PyPI for `x86_64` and `aarch64`. Depending on your platform and environment,
|
||||
you may want to customize the specific versions of `numpy`. For other platforms,
|
||||
you can have a look at SciPy's
|
||||
[`oldest-supported-numpy`](https://github.com/scipy/oldest-supported-numpy/blob/main/setup.cfg)
|
||||
package to see what the oldest recommended versions of `numpy` are.
|
||||
|
||||
(_Warning_: don't use `pip install -c constraints.txt` instead of
|
||||
`PIP_CONSTRAINT`, since this isn't applied to the isolated build environments.)
|
||||
|
||||
#### Additional options for developers {id="source-developers"}
|
||||
|
||||
Some additional options may be useful for spaCy developers who are editing the
|
||||
source code and recompiling frequently.
|
||||
|
||||
- Install in editable mode. Changes to `.py` files will be reflected as soon as
|
||||
the files are saved, but edits to Cython files (`.pxd`, `.pyx`) will require
|
||||
the `pip install` command below to be run again. Before installing in editable
|
||||
mode, be sure you have removed any previous installs with
|
||||
`pip uninstall spacy`, which you may need to run multiple times to remove all
|
||||
traces of earlier installs.
|
||||
|
||||
```bash
|
||||
$ pip install -r requirements.txt
|
||||
$ pip install --no-build-isolation --editable .
|
||||
```
|
||||
|
||||
- Build in parallel. Starting in v3.4.0, you can specify the number of build
|
||||
jobs with the environment variable `SPACY_NUM_BUILD_JOBS`:
|
||||
|
||||
```bash
|
||||
$ pip install -r requirements.txt
|
||||
$ SPACY_NUM_BUILD_JOBS=4 pip install --no-build-isolation --editable .
|
||||
```
|
||||
|
||||
- For editable mode and parallel builds with `python setup.py` instead of `pip`
|
||||
(no longer recommended):
|
||||
|
||||
```bash
|
||||
$ pip install -r requirements.txt
|
||||
$ python setup.py build_ext --inplace -j 4
|
||||
$ python setup.py develop
|
||||
```
|
||||
|
||||
#### Visual Studio Code extension
|
||||
|
||||
|
||||
|
||||
The [spaCy VSCode Extension](https://github.com/explosion/spacy-vscode) provides
|
||||
additional tooling and features for working with spaCy's config files. Version
|
||||
1.0.0 includes hover descriptions for registry functions, variables, and section
|
||||
names within the config as an installable extension.
|
||||
|
||||
1. Install a supported version of Python on your system (`>=3.7`)
|
||||
2. Install the
|
||||
[Python Extension for Visual Studio Code](https://code.visualstudio.com/docs/python/python-tutorial)
|
||||
3. Create a
|
||||
[virtual python environment](https://docs.python.org/3/library/venv.html)
|
||||
4. Install all python requirements (`spaCy >= 3.4.0` & `pygls >= 1.0.0`)
|
||||
5. Install
|
||||
[spaCy extension for Visual Studio Code](https://marketplace.visualstudio.com/items?itemName=Explosion.spacy-extension)
|
||||
6. Select your python environment
|
||||
7. You are ready to work with `.cfg` files in spaCy!
|
||||
|
||||
### Building an executable {id="executable"}
|
||||
|
||||
The spaCy repository includes a [`Makefile`](%%GITHUB_SPACY/Makefile) that
|
||||
builds an executable zip file using [`pex`](https://github.com/pantsbuild/pex)
|
||||
(**P**ython **Ex**ecutable). The executable includes spaCy and all its package
|
||||
dependencies and only requires the system Python at runtime. Building an
|
||||
executable `.pex` file is often the most convenient way to deploy spaCy, as it
|
||||
lets you separate the build from the deployment process.
|
||||
|
||||
> #### Usage
|
||||
>
|
||||
> To use a `.pex` file, just replace `python` with the path to the file when you
|
||||
> execute your code or CLI commands. This is equivalent to running Python in a
|
||||
> virtual environment with spaCy installed.
|
||||
>
|
||||
> ```bash
|
||||
> $ ./spacy.pex my_script.py
|
||||
> $ ./spacy.pex -m spacy info
|
||||
> ```
|
||||
|
||||
```bash
|
||||
$ git clone https://github.com/explosion/spaCy
|
||||
$ cd spaCy
|
||||
$ make
|
||||
```
|
||||
|
||||
You can configure the build process with the following environment variables:
|
||||
|
||||
| Variable | Description |
|
||||
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `SPACY_EXTRAS` | Additional Python packages to install alongside spaCy with optional version specifications. Should be a string that can be passed to `pip install`. See [`Makefile`](%%GITHUB_SPACY/Makefile) for defaults. |
|
||||
| `PYVER` | The Python version to build against. This version needs to be available on your build and runtime machines. Defaults to `3.8`. |
|
||||
| `WHEELHOUSE` | Directory to store the wheel files during compilation. Defaults to `./wheelhouse`. |
|
||||
|
||||
### Run tests {id="run-tests"}
|
||||
|
||||
spaCy comes with an [extensive test suite](%%GITHUB_SPACY/spacy/tests). In order
|
||||
to run the tests, you'll usually want to clone the [repository](%%GITHUB_SPACY)
|
||||
and [build spaCy from source](#source). This will also install the required
|
||||
development dependencies and test utilities defined in the `requirements.txt`.
|
||||
|
||||
Alternatively, you can find out where spaCy is installed and run `pytest` on
|
||||
that directory. Don't forget to also install the test utilities via spaCy's
|
||||
[`requirements.txt`](%%GITHUB_SPACY/requirements.txt):
|
||||
|
||||
```bash
|
||||
$ python -c "import os; import spacy; print(os.path.dirname(spacy.__file__))"
|
||||
$ pip install -r path/to/requirements.txt
|
||||
$ python -m pytest --pyargs %%SPACY_PKG_NAME
|
||||
```
|
||||
|
||||
Calling `pytest` on the spaCy directory will run only the basic tests. The flag
|
||||
`--slow` is optional and enables additional tests that take longer.
|
||||
|
||||
```bash
|
||||
$ python -m pip install -U pytest # update pytest
|
||||
$ python -m pytest --pyargs %%SPACY_PKG_NAME # basic tests
|
||||
$ python -m pytest --pyargs %%SPACY_PKG_NAME --slow # basic and slow tests
|
||||
```
|
||||
|
||||
## Troubleshooting guide {id="troubleshooting"}
|
||||
|
||||
This section collects some of the most common errors you may come across when
|
||||
installing, loading and using spaCy, as well as their solutions. Also see the
|
||||
[Discussions FAQ Thread](https://github.com/explosion/spaCy/discussions/8226),
|
||||
which is updated more frequently and covers more transitory issues.
|
||||
|
||||
> #### Help us improve this guide
|
||||
>
|
||||
> Did you come across a problem like the ones listed here and want to share the
|
||||
> solution? You can find the "Suggest edits" button at the bottom of this page
|
||||
> that points you to the source. We always appreciate
|
||||
> [pull requests](https://github.com/explosion/spaCy/pulls)!
|
||||
|
||||
<Accordion title="No compatible model found" id="compatible-model">
|
||||
|
||||
```
|
||||
No compatible package found for [lang] (spaCy vX.X.X).
|
||||
```
|
||||
|
||||
This usually means that the trained pipeline you're trying to download does not
|
||||
exist, or isn't available for your version of spaCy. Check the
|
||||
[compatibility table](https://github.com/explosion/spacy-models/tree/master/compatibility.json)
|
||||
to see which packages are available for your spaCy version. If you're using an
|
||||
old version, consider upgrading to the latest release. Note that while spaCy
|
||||
supports tokenization for [a variety of languages](/usage/models#languages), not
|
||||
all of them come with trained pipelines. To only use the tokenizer, import the
|
||||
language's `Language` class instead, for example
|
||||
`from spacy.lang.fr import French`.
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Import error: No module named spacy" id="import-error">
|
||||
|
||||
```
|
||||
Import Error: No module named spacy
|
||||
```
|
||||
|
||||
This error means that the spaCy module can't be located on your system, or in
|
||||
your environment. Make sure you have spaCy installed. If you're using a virtual
|
||||
environment, make sure it's activated and check that spaCy is installed in that
|
||||
environment – otherwise, you're trying to load a system installation. You can
|
||||
also run `which python` to find out where your Python executable is located.
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Import error: No module named [name]" id="import-error-models">
|
||||
|
||||
```
|
||||
ImportError: No module named 'en_core_web_sm'
|
||||
```
|
||||
|
||||
As of spaCy v1.7, all trained pipelines can be installed as Python packages.
|
||||
This means that they'll become importable modules of your application. If this
|
||||
fails, it's usually a sign that the package is not installed in the current
|
||||
environment. Run `pip list` or `pip freeze` to check which pipeline packages you
|
||||
have installed, and install the [correct package](/models) if necessary. If
|
||||
you're importing a package manually at the top of a file, make sure to use the
|
||||
full name of the package.
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Command not found: spacy" id="command-not-found">
|
||||
|
||||
```
|
||||
command not found: spacy
|
||||
```
|
||||
|
||||
This error may occur when running the `spacy` command from the command line.
|
||||
spaCy does not currently add an entry to your `PATH` environment variable, as
|
||||
this can lead to unexpected results, especially when using a virtual
|
||||
environment. Instead, spaCy adds an auto-alias that maps `spacy` to
|
||||
`python -m spacy`. If this is not working as expected, run the command with
|
||||
`python -m`, yourself – for example `python -m spacy download en_core_web_sm`.
|
||||
For more info on this, see the [`download`](/api/cli#download) command.
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="'module' object has no attribute 'load'" id="module-load">
|
||||
|
||||
```
|
||||
AttributeError: 'module' object has no attribute 'load'
|
||||
```
|
||||
|
||||
While this could technically have many causes, including spaCy being broken, the
|
||||
most likely one is that your script's file or directory name is "shadowing" the
|
||||
module – e.g. your file is called `spacy.py`, or a directory you're importing
|
||||
from is called `spacy`. So, when using spaCy, never call anything else `spacy`.
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="NER model doesn't recognize other entities anymore after training" id="catastrophic-forgetting">
|
||||
|
||||
If your training data only contained new entities and you didn't mix in any
|
||||
examples the model previously recognized, it can cause the model to "forget"
|
||||
what it had previously learned. This is also referred to as the "catastrophic
|
||||
forgetting problem". A solution is to pre-label some text, and mix it with the
|
||||
new text in your updates. You can also do this by running spaCy over some text,
|
||||
extracting a bunch of entities the model previously recognized correctly, and
|
||||
adding them to your training examples.
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Unhashable type: 'list'" id="unhashable-list">
|
||||
|
||||
```
|
||||
TypeError: unhashable type: 'list'
|
||||
```
|
||||
|
||||
If you're training models, writing them to disk, and versioning them with git,
|
||||
you might encounter this error when trying to load them in a Windows
|
||||
environment. This happens because a default install of Git for Windows is
|
||||
configured to automatically convert Unix-style end-of-line characters (LF) to
|
||||
Windows-style ones (CRLF) during file checkout (and the reverse when
|
||||
committing). While that's mostly fine for text files, a trained model written to
|
||||
disk has some binary files that should not go through this conversion. When they
|
||||
do, you get the error above. You can fix it by either changing your
|
||||
[`core.autocrlf`](https://git-scm.com/book/en/v2/Customizing-Git-Git-Configuration)
|
||||
setting to `"false"`, or by committing a
|
||||
[`.gitattributes`](https://git-scm.com/docs/gitattributes) file to your
|
||||
repository to tell Git on which files or folders it shouldn't do LF-to-CRLF
|
||||
conversion, with an entry like `path/to/spacy/model/** -text`. After you've done
|
||||
either of these, clone your repository again.
|
||||
|
||||
</Accordion>
|
||||
|
||||
## Changelog {id="changelog"}
|
||||
|
||||
<Changelog />
|
||||
@@ -0,0 +1,534 @@
|
||||
---
|
||||
title: Large Language Models
|
||||
teaser: Integrating LLMs into structured NLP pipelines
|
||||
menu:
|
||||
- ['Motivation', 'motivation']
|
||||
- ['Install', 'install']
|
||||
- ['Usage', 'usage']
|
||||
- ['Logging', 'logging']
|
||||
- ['API', 'api']
|
||||
- ['Tasks', 'tasks']
|
||||
- ['Models', 'models']
|
||||
---
|
||||
|
||||
[The spacy-llm package](https://github.com/explosion/spacy-llm) integrates Large
|
||||
Language Models (LLMs) into spaCy pipelines, featuring a modular system for
|
||||
**fast prototyping** and **prompting**, and turning unstructured responses into
|
||||
**robust outputs** for various NLP tasks, **no training data** required.
|
||||
|
||||
- Serializable `llm` **component** to integrate prompts into your pipeline
|
||||
- **Modular functions** to define the [**task**](#tasks) (prompting and parsing)
|
||||
and [**model**](#models) (model to use)
|
||||
- Support for **hosted APIs** and self-hosted **open-source models**
|
||||
- Integration with [`LangChain`](https://github.com/hwchase17/langchain)
|
||||
- Access to
|
||||
**[OpenAI API](https://platform.openai.com/docs/api-reference/introduction)**,
|
||||
including GPT-4 and various GPT-3 models
|
||||
- Built-in support for various **open-source** models hosted on
|
||||
[Hugging Face](https://huggingface.co/)
|
||||
- Usage examples for standard NLP tasks such as **Named Entity Recognition** and
|
||||
**Text Classification**
|
||||
- Easy implementation of **your own functions** via the
|
||||
[registry](/api/top-level#registry) for custom prompting, parsing and model
|
||||
integrations
|
||||
|
||||
## Motivation {id="motivation"}
|
||||
|
||||
Large Language Models (LLMs) feature powerful natural language understanding
|
||||
capabilities. With only a few (and sometimes no) examples, an LLM can be
|
||||
prompted to perform custom NLP tasks such as text categorization, named entity
|
||||
recognition, coreference resolution, information extraction and more.
|
||||
|
||||
Supervised learning is much worse than LLM prompting for prototyping, but for
|
||||
many tasks it's much better for production. A transformer model that runs
|
||||
comfortably on a single GPU is extremely powerful, and it's likely to be a
|
||||
better choice for any task for which you have a well-defined output. You train
|
||||
the model with anything from a few hundred to a few thousand labelled examples,
|
||||
and it will learn to do exactly that. Efficiency, reliability and control are
|
||||
all better with supervised learning, and accuracy will generally be higher than
|
||||
LLM prompting as well.
|
||||
|
||||
`spacy-llm` lets you have **the best of both worlds**. You can quickly
|
||||
initialize a pipeline with components powered by LLM prompts, and freely mix in
|
||||
components powered by other approaches. As your project progresses, you can look
|
||||
at replacing some or all of the LLM-powered components as you require.
|
||||
|
||||
Of course, there can be components in your system for which the power of an LLM
|
||||
is fully justified. If you want a system that can synthesize information from
|
||||
multiple documents in subtle ways and generate a nuanced summary for you, bigger
|
||||
is better. However, even if your production system needs an LLM for some of the
|
||||
task, that doesn't mean you need an LLM for all of it. Maybe you want to use a
|
||||
cheap text classification model to help you find the texts to summarize, or
|
||||
maybe you want to add a rule-based system to sanity check the output of the
|
||||
summary. These before-and-after tasks are much easier with a mature and
|
||||
well-thought-out library, which is exactly what spaCy provides.
|
||||
|
||||
## Install {id="install"}
|
||||
|
||||
`spacy-llm` will be installed automatically in future spaCy versions. For now,
|
||||
you can run the following in the same virtual environment where you already have
|
||||
`spacy` [installed](/usage).
|
||||
|
||||
> ⚠️ This package is still experimental and it is possible that changes made to
|
||||
> the interface will be breaking in minor version updates.
|
||||
|
||||
```bash
|
||||
python -m pip install spacy-llm
|
||||
```
|
||||
|
||||
## Usage {id="usage"}
|
||||
|
||||
The task and the model have to be supplied to the `llm` pipeline component using
|
||||
the [config system](/api/data-formats#config). This package provides various
|
||||
built-in functionality, as detailed in the [API](#-api) documentation.
|
||||
|
||||
### Example 1: Add a text classifier using a GPT-3 model from OpenAI {id="example-1"}
|
||||
|
||||
Create a new API key from openai.com or fetch an existing one, and ensure the
|
||||
keys are set as environmental variables. For more background information, see
|
||||
the [OpenAI](/api/large-language-models#gpt-3-5) section.
|
||||
|
||||
Create a config file `config.cfg` containing at least the following (or see the
|
||||
full example
|
||||
[here](https://github.com/explosion/spacy-llm/tree/main/usage_examples/textcat_openai)):
|
||||
|
||||
```ini
|
||||
[nlp]
|
||||
lang = "en"
|
||||
pipeline = ["llm"]
|
||||
|
||||
[components]
|
||||
|
||||
[components.llm]
|
||||
factory = "llm"
|
||||
|
||||
[components.llm.task]
|
||||
@llm_tasks = "spacy.TextCat.v2"
|
||||
labels = ["COMPLIMENT", "INSULT"]
|
||||
|
||||
[components.llm.model]
|
||||
@llm_models = "spacy.GPT-3-5.v1"
|
||||
config = {"temperature": 0.0}
|
||||
```
|
||||
|
||||
Now run:
|
||||
|
||||
```python
|
||||
from spacy_llm.util import assemble
|
||||
|
||||
nlp = assemble("config.cfg")
|
||||
doc = nlp("You look gorgeous!")
|
||||
print(doc.cats)
|
||||
```
|
||||
|
||||
### Example 2: Add NER using an open-source model through Hugging Face {id="example-2"}
|
||||
|
||||
To run this example, ensure that you have a GPU enabled, and `transformers`,
|
||||
`torch` and CUDA installed. For more background information, see the
|
||||
[DollyHF](/api/large-language-models#dolly) section.
|
||||
|
||||
Create a config file `config.cfg` containing at least the following (or see the
|
||||
full example
|
||||
[here](https://github.com/explosion/spacy-llm/tree/main/usage_examples/ner_dolly)):
|
||||
|
||||
```ini
|
||||
[nlp]
|
||||
lang = "en"
|
||||
pipeline = ["llm"]
|
||||
|
||||
[components]
|
||||
|
||||
[components.llm]
|
||||
factory = "llm"
|
||||
|
||||
[components.llm.task]
|
||||
@llm_tasks = "spacy.NER.v3"
|
||||
labels = ["PERSON", "ORGANISATION", "LOCATION"]
|
||||
|
||||
[components.llm.model]
|
||||
@llm_models = "spacy.Dolly.v1"
|
||||
# For better performance, use dolly-v2-12b instead
|
||||
name = "dolly-v2-3b"
|
||||
```
|
||||
|
||||
Now run:
|
||||
|
||||
```python
|
||||
from spacy_llm.util import assemble
|
||||
|
||||
nlp = assemble("config.cfg")
|
||||
doc = nlp("Jack and Jill rode up the hill in Les Deux Alpes")
|
||||
print([(ent.text, ent.label_) for ent in doc.ents])
|
||||
```
|
||||
|
||||
Note that Hugging Face will download the `"databricks/dolly-v2-3b"` model the
|
||||
first time you use it. You can
|
||||
[define the cached directory](https://huggingface.co/docs/huggingface_hub/main/en/guides/manage-cache)
|
||||
by setting the environmental variable `HF_HOME`. Also, you can upgrade the model
|
||||
to be `"databricks/dolly-v2-12b"` for better performance.
|
||||
|
||||
### Example 3: Create the component directly in Python {id="example-3"}
|
||||
|
||||
The `llm` component behaves as any other component does, and there are
|
||||
[task-specific components](/api/large-language-models#config) defined to help
|
||||
you hit the ground running with a reasonable built-in task implementation.
|
||||
|
||||
```python
|
||||
import spacy
|
||||
|
||||
nlp = spacy.blank("en")
|
||||
llm_ner = nlp.add_pipe("llm_ner")
|
||||
llm_ner.add_label("PERSON")
|
||||
llm_ner.add_label("LOCATION")
|
||||
nlp.initialize()
|
||||
doc = nlp("Jack and Jill rode up the hill in Les Deux Alpes")
|
||||
print([(ent.text, ent.label_) for ent in doc.ents])
|
||||
```
|
||||
|
||||
Note that for efficient usage of resources, typically you would use
|
||||
[`nlp.pipe(docs)`](/api/language#pipe) with a batch, instead of calling
|
||||
`nlp(doc)` with a single document.
|
||||
|
||||
### Example 4: Implement your own custom task {id="example-4"}
|
||||
|
||||
To write a [`task`](#tasks), you need to implement two functions:
|
||||
`generate_prompts` that takes a list of [`Doc`](/api/doc) objects and transforms
|
||||
them into a list of prompts, and `parse_responses` that transforms the LLM
|
||||
outputs into annotations on the [`Doc`](/api/doc), e.g. entity spans, text
|
||||
categories and more.
|
||||
|
||||
To register your custom task, decorate a factory function using the
|
||||
`spacy_llm.registry.llm_tasks` decorator with a custom name that you can refer
|
||||
to in your config.
|
||||
|
||||
> 📖 For more details, see the
|
||||
> [**usage example on writing your own task**](https://github.com/explosion/spacy-llm/tree/main/usage_examples#writing-your-own-task)
|
||||
|
||||
```python
|
||||
from typing import Iterable, List
|
||||
from spacy.tokens import Doc
|
||||
from spacy_llm.registry import registry
|
||||
from spacy_llm.util import split_labels
|
||||
|
||||
|
||||
@registry.llm_tasks("my_namespace.MyTask.v1")
|
||||
def make_my_task(labels: str, my_other_config_val: float) -> "MyTask":
|
||||
labels_list = split_labels(labels)
|
||||
return MyTask(labels=labels_list, my_other_config_val=my_other_config_val)
|
||||
|
||||
|
||||
class MyTask:
|
||||
def __init__(self, labels: List[str], my_other_config_val: float):
|
||||
...
|
||||
|
||||
def generate_prompts(self, docs: Iterable[Doc]) -> Iterable[str]:
|
||||
...
|
||||
|
||||
def parse_responses(
|
||||
self, docs: Iterable[Doc], responses: Iterable[str]
|
||||
) -> Iterable[Doc]:
|
||||
...
|
||||
```
|
||||
|
||||
```ini
|
||||
# config.cfg (excerpt)
|
||||
[components.llm.task]
|
||||
@llm_tasks = "my_namespace.MyTask.v1"
|
||||
labels = LABEL1,LABEL2,LABEL3
|
||||
my_other_config_val = 0.3
|
||||
```
|
||||
|
||||
## Logging {id="logging"}
|
||||
|
||||
spacy-llm has a built-in logger that can log the prompt sent to the LLM as well
|
||||
as its raw response. This logger uses the debug level and by default has a
|
||||
`logging.NullHandler()` configured.
|
||||
|
||||
In order to use this logger, you can setup a simple handler like this:
|
||||
|
||||
```python
|
||||
import logging
|
||||
import spacy_llm
|
||||
|
||||
|
||||
spacy_llm.logger.addHandler(logging.StreamHandler())
|
||||
spacy_llm.logger.setLevel(logging.DEBUG)
|
||||
```
|
||||
|
||||
> NOTE: Any `logging` handler will work here so you probably want to use some
|
||||
> sort of rotating `FileHandler` as the generated prompts can be quite long,
|
||||
> especially for tasks with few-shot examples.
|
||||
|
||||
Then when using the pipeline you'll be able to view the prompt and response.
|
||||
|
||||
E.g. with the config and code from [Example 1](#example-1) above:
|
||||
|
||||
```python
|
||||
from spacy_llm.util import assemble
|
||||
|
||||
|
||||
nlp = assemble("config.cfg")
|
||||
doc = nlp("You look gorgeous!")
|
||||
print(doc.cats)
|
||||
```
|
||||
|
||||
You will see `logging` output similar to:
|
||||
|
||||
```
|
||||
Generated prompt for doc: You look gorgeous!
|
||||
|
||||
You are an expert Text Classification system. Your task is to accept Text as input
|
||||
and provide a category for the text based on the predefined labels.
|
||||
|
||||
Classify the text below to any of the following labels: COMPLIMENT, INSULT
|
||||
The task is non-exclusive, so you can provide more than one label as long as
|
||||
they're comma-delimited. For example: Label1, Label2, Label3.
|
||||
Do not put any other text in your answer, only one or more of the provided labels with nothing before or after.
|
||||
If the text cannot be classified into any of the provided labels, answer `==NONE==`.
|
||||
|
||||
Here is the text that needs classification
|
||||
|
||||
|
||||
Text:
|
||||
'''
|
||||
You look gorgeous!
|
||||
'''
|
||||
|
||||
Model response for doc: You look gorgeous!
|
||||
COMPLIMENT
|
||||
```
|
||||
|
||||
`print(doc.cats)` to standard output should look like:
|
||||
|
||||
```
|
||||
{'COMPLIMENT': 1.0, 'INSULT': 0.0}
|
||||
```
|
||||
|
||||
## API {id="api"}
|
||||
|
||||
`spacy-llm` exposes an `llm` factory with
|
||||
[configurable settings](/api/large-language-models#config).
|
||||
|
||||
An `llm` component is defined by two main settings:
|
||||
|
||||
- A [**task**](#tasks), defining the prompt to send to the LLM as well as the
|
||||
functionality to parse the resulting response back into structured fields on
|
||||
the [Doc](/api/doc) objects.
|
||||
- A [**model**](#models) defining the model to use and how to connect to it.
|
||||
Note that `spacy-llm` supports both access to external APIs (such as OpenAI)
|
||||
as well as access to self-hosted open-source LLMs (such as using Dolly through
|
||||
Hugging Face).
|
||||
|
||||
Moreover, `spacy-llm` exposes a customizable [**caching**](#cache) functionality
|
||||
to avoid running the same document through an LLM service (be it local or
|
||||
through a REST API) more than once.
|
||||
|
||||
Finally, you can choose to save a stringified version of LLM prompts/responses
|
||||
within the `Doc.user_data["llm_io"]` attribute by setting `save_io` to `True`.
|
||||
`Doc.user_data["llm_io"]` is a dictionary containing one entry for every LLM
|
||||
component within the `nlp` pipeline. Each entry is itself a dictionary, with two
|
||||
keys: `prompt` and `response`.
|
||||
|
||||
A note on `validate_types`: by default, `spacy-llm` checks whether the
|
||||
signatures of the `model` and `task` callables are consistent with each other
|
||||
and emits a warning if they don't. `validate_types` can be set to `False` if you
|
||||
want to disable this behavior.
|
||||
|
||||
### Tasks {id="tasks"}
|
||||
|
||||
A _task_ defines an NLP problem or question, that will be sent to the LLM via a
|
||||
prompt. Further, the task defines how to parse the LLM's responses back into
|
||||
structured information. All tasks are registered in the `llm_tasks` registry.
|
||||
|
||||
Practically speaking, a task should adhere to the `Protocol` named `LLMTask`
|
||||
defined in
|
||||
[`ty.py`](https://github.com/explosion/spacy-llm/blob/main/spacy_llm/ty.py). It
|
||||
needs to define a `generate_prompts` function and a `parse_responses` function.
|
||||
|
||||
Tasks may support prompt sharding (for more info see the API docs on
|
||||
[sharding](/api/large-language-models#task-sharding) and
|
||||
[non-sharding](/api/large-language-models#task-nonsharding) tasks). The function
|
||||
signatures for `generate_prompts` and `parse_responses` depend on whether they
|
||||
do.
|
||||
|
||||
For tasks **not supporting** sharding:
|
||||
|
||||
| Task | Description | |
|
||||
| --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | --- |
|
||||
| [`task.generate_prompts`](/api/large-language-models#task-nonsharding-generate-prompts) | Takes a collection of documents, and returns a collection of prompts, which can be of type `Any`. |
|
||||
| [`task.parse_responses`](/api/large-language-models#task-nonsharding-parse-responses) | Takes a collection of LLM responses and the original documents, parses the responses into structured information, and sets the annotations on the documents. |
|
||||
|
||||
For tasks **supporting** sharding:
|
||||
|
||||
| Task | Description | |
|
||||
| ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --- |
|
||||
| [`task.generate_prompts`](/api/large-language-models#task-sharding-generate-prompts) | Takes a collection of documents, and returns a collection of collections of prompt shards, which can be of type `Any`. |
|
||||
| [`task.parse_responses`](/api/large-language-models#task-sharding-parse-responses) | Takes a collection of collections of LLM responses (one per prompt shard) and the original documents, parses the responses into structured information, sets the annotations on the doc shards, and merges those doc shards back into a single doc instance. |
|
||||
|
||||
Moreover, the task may define an optional [`scorer` method](/api/scorer#score).
|
||||
It should accept an iterable of `Example` objects as input and return a score
|
||||
dictionary. If the `scorer` method is defined, `spacy-llm` will call it to
|
||||
evaluate the component.
|
||||
|
||||
| Component | Description |
|
||||
| ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
|
||||
| [`spacy.EntityLinker.v1`](/api/large-language-models#el-v1) | The entity linking task prompts the model to link all entities in a given text to entries in a knowledge base. |
|
||||
| [`spacy.Summarization.v1`](/api/large-language-models#summarization-v1) | The summarization task prompts the model for a concise summary of the provided text. |
|
||||
| [`spacy.NER.v3`](/api/large-language-models#ner-v3) | Implements Chain-of-Thought reasoning for NER extraction - obtains higher accuracy than v1 or v2. |
|
||||
| [`spacy.NER.v2`](/api/large-language-models#ner-v2) | Builds on v1 and additionally supports defining the provided labels with explicit descriptions. |
|
||||
| [`spacy.NER.v1`](/api/large-language-models#ner-v1) | The original version of the built-in NER task supports both zero-shot and few-shot prompting. |
|
||||
| [`spacy.SpanCat.v3`](/api/large-language-models#spancat-v3) | Adaptation of the v3 NER task to support overlapping entities and store its annotations in `doc.spans`. |
|
||||
| [`spacy.SpanCat.v2`](/api/large-language-models#spancat-v2) | Adaptation of the v2 NER task to support overlapping entities and store its annotations in `doc.spans`. |
|
||||
| [`spacy.SpanCat.v1`](/api/large-language-models#spancat-v1) | Adaptation of the v1 NER task to support overlapping entities and store its annotations in `doc.spans`. |
|
||||
| [`spacy.REL.v1`](/api/large-language-models#rel-v1) | Relation Extraction task supporting both zero-shot and few-shot prompting. |
|
||||
| [`spacy.TextCat.v3`](/api/large-language-models#textcat-v3) | Version 3 builds on v2 and allows setting definitions of labels. |
|
||||
| [`spacy.TextCat.v2`](/api/large-language-models#textcat-v2) | Version 2 builds on v1 and includes an improved prompt template. |
|
||||
| [`spacy.TextCat.v1`](/api/large-language-models#textcat-v1) | Version 1 of the built-in TextCat task supports both zero-shot and few-shot prompting. |
|
||||
| [`spacy.Lemma.v1`](/api/large-language-models#lemma-v1) | Lemmatizes the provided text and updates the `lemma_` attribute of the tokens accordingly. |
|
||||
| [`spacy.Raw.v1`](/api/large-language-models#raw-v1) | Executes raw doc content as prompt to LLM. |
|
||||
| [`spacy.Sentiment.v1`](/api/large-language-models#sentiment-v1) | Performs sentiment analysis on provided texts. |
|
||||
| [`spacy.Translation.v1`](/api/large-language-models#translation-v1) | Translates doc content into the specified target language. |
|
||||
| [`spacy.NoOp.v1`](/api/large-language-models#noop-v1) | This task is only useful for testing - it tells the LLM to do nothing, and does not set any fields on the `docs`. |
|
||||
|
||||
#### Providing examples for few-shot prompts {id="few-shot-prompts"}
|
||||
|
||||
All built-in tasks support few-shot prompts, i. e. including examples in a
|
||||
prompt. Examples can be supplied in two ways: (1) as a separate file containing
|
||||
only examples or (2) by initializing `llm` with a `get_examples()` callback
|
||||
(like any other pipeline component).
|
||||
|
||||
##### (1) Few-shot example file
|
||||
|
||||
A file containing examples for few-shot prompting can be configured like this:
|
||||
|
||||
```ini
|
||||
[components.llm.task]
|
||||
@llm_tasks = "spacy.NER.v2"
|
||||
labels = PERSON,ORGANISATION,LOCATION
|
||||
[components.llm.task.examples]
|
||||
@misc = "spacy.FewShotReader.v1"
|
||||
path = "ner_examples.yml"
|
||||
```
|
||||
|
||||
The supplied file has to conform to the format expected by the required task
|
||||
(see the task documentation further down).
|
||||
|
||||
##### (2) Initializing the `llm` component with a `get_examples()` callback
|
||||
|
||||
Alternatively, you can initialize your `nlp` pipeline by providing a
|
||||
`get_examples` callback for [`nlp.initialize`](/api/language#initialize) and
|
||||
setting `n_prompt_examples` to a positive number to automatically fetch a few
|
||||
examples for few-shot learning. Set `n_prompt_examples` to `-1` to use all
|
||||
examples as part of the few-shot learning prompt.
|
||||
|
||||
```ini
|
||||
[initialize.components.llm]
|
||||
n_prompt_examples = 3
|
||||
```
|
||||
|
||||
### Model {id="models"}
|
||||
|
||||
A _model_ defines which LLM model to query, and how to query it. It can be a
|
||||
simple function taking a collection of prompts (consistent with the output type
|
||||
of `task.generate_prompts()`) and returning a collection of responses
|
||||
(consistent with the expected input of `parse_responses`). Generally speaking,
|
||||
it's a function of type `Callable[[Iterable[Any]], Iterable[Any]]`, but specific
|
||||
implementations can have other signatures, like
|
||||
`Callable[[Iterable[str]], Iterable[str]]`.
|
||||
|
||||
All built-in models are registered in `llm_models`. If no model is specified,
|
||||
the repo currently connects to the `OpenAI` API by default using REST, and
|
||||
accesses the `"gpt-3.5-turbo"` model.
|
||||
|
||||
Currently three different approaches to use LLMs are supported:
|
||||
|
||||
1. `spacy-llm`s native REST interface. This is the default for all hosted models
|
||||
(e. g. OpenAI, Cohere, Anthropic, ...).
|
||||
2. A HuggingFace integration that allows to run a limited set of HF models
|
||||
locally.
|
||||
3. A LangChain integration that allows to run any model supported by LangChain
|
||||
(hosted or locally).
|
||||
|
||||
Approaches 1. and 2 are the default for hosted model and local models,
|
||||
respectively. Alternatively you can use LangChain to access hosted or local
|
||||
models by specifying one of the models registered with the `langchain.` prefix.
|
||||
|
||||
<Infobox>
|
||||
_Why LangChain if there are also are native REST and HuggingFace interfaces? When should I use what?_
|
||||
|
||||
Third-party libraries like `langchain` focus on prompt management, integration
|
||||
of many different LLM APIs, and other related features such as conversational
|
||||
memory or agents. `spacy-llm` on the other hand emphasizes features we consider
|
||||
useful in the context of NLP pipelines utilizing LLMs to process documents
|
||||
(mostly) independent from each other. It makes sense that the feature sets of
|
||||
such third-party libraries and `spacy-llm` aren't identical - and users might
|
||||
want to take advantage of features not available in `spacy-llm`.
|
||||
|
||||
The advantage of implementing our own REST and HuggingFace integrations is that
|
||||
we can ensure a larger degree of stability and robustness, as we can guarantee
|
||||
backwards-compatibility and more smoothly integrated error handling.
|
||||
|
||||
If however there are features or APIs not natively covered by `spacy-llm`, it's
|
||||
trivial to utilize LangChain to cover this - and easy to customize the prompting
|
||||
mechanism, if so required.
|
||||
|
||||
</Infobox>
|
||||
|
||||
<Infobox variant="warning">
|
||||
Note that when using hosted services, you have to ensure that the [proper API
|
||||
keys](/api/large-language-models#api-keys) are set as environment variables as described by the corresponding
|
||||
provider's documentation.
|
||||
|
||||
</Infobox>
|
||||
|
||||
| Model | Description |
|
||||
| ----------------------------------------------------------------------- | ---------------------------------------------- |
|
||||
| [`spacy.GPT-4.v2`](/api/large-language-models#models-rest) | OpenAI’s `gpt-4` model family. |
|
||||
| [`spacy.GPT-3-5.v2`](/api/large-language-models#models-rest) | OpenAI’s `gpt-3-5` model family. |
|
||||
| [`spacy.Text-Davinci.v2`](/api/large-language-models#models-rest) | OpenAI’s `text-davinci` model family. |
|
||||
| [`spacy.Code-Davinci.v2`](/api/large-language-models#models-rest) | OpenAI’s `code-davinci` model family. |
|
||||
| [`spacy.Text-Curie.v2`](/api/large-language-models#models-rest) | OpenAI’s `text-curie` model family. |
|
||||
| [`spacy.Text-Babbage.v2`](/api/large-language-models#models-rest) | OpenAI’s `text-babbage` model family. |
|
||||
| [`spacy.Text-Ada.v2`](/api/large-language-models#models-rest) | OpenAI’s `text-ada` model family. |
|
||||
| [`spacy.Davinci.v2`](/api/large-language-models#models-rest) | OpenAI’s `davinci` model family. |
|
||||
| [`spacy.Curie.v2`](/api/large-language-models#models-rest) | OpenAI’s `curie` model family. |
|
||||
| [`spacy.Babbage.v2`](/api/large-language-models#models-rest) | OpenAI’s `babbage` model family. |
|
||||
| [`spacy.Ada.v2`](/api/large-language-models#models-rest) | OpenAI’s `ada` model family. |
|
||||
| [`spacy.Azure.v1`](/api/large-language-models#models-rest) | Azure's OpenAI models. |
|
||||
| [`spacy.Command.v1`](/api/large-language-models#models-rest) | Cohere’s `command` model family. |
|
||||
| [`spacy.Claude-2.v1`](/api/large-language-models#models-rest) | Anthropic’s `claude-2` model family. |
|
||||
| [`spacy.Claude-1.v1`](/api/large-language-models#models-rest) | Anthropic’s `claude-1` model family. |
|
||||
| [`spacy.Claude-instant-1.v1`](/api/large-language-models#models-rest) | Anthropic’s `claude-instant-1` model family. |
|
||||
| [`spacy.Claude-instant-1-1.v1`](/api/large-language-models#models-rest) | Anthropic’s `claude-instant-1.1` model family. |
|
||||
| [`spacy.Claude-1-0.v1`](/api/large-language-models#models-rest) | Anthropic’s `claude-1.0` model family. |
|
||||
| [`spacy.Claude-1-2.v1`](/api/large-language-models#models-rest) | Anthropic’s `claude-1.2` model family. |
|
||||
| [`spacy.Claude-1-3.v1`](/api/large-language-models#models-rest) | Anthropic’s `claude-1.3` model family. |
|
||||
| [`spacy.PaLM.v1`](/api/large-language-models#models-rest) | Google’s `PaLM` model family. |
|
||||
| [`spacy.Dolly.v1`](/api/large-language-models#models-hf) | Dolly models through HuggingFace. |
|
||||
| [`spacy.Falcon.v1`](/api/large-language-models#models-hf) | Falcon models through HuggingFace. |
|
||||
| [`spacy.Mistral.v1`](/api/large-language-models#models-hf) | Mistral models through HuggingFace. |
|
||||
| [`spacy.Llama2.v1`](/api/large-language-models#models-hf) | Llama2 models through HuggingFace. |
|
||||
| [`spacy.StableLM.v1`](/api/large-language-models#models-hf) | StableLM models through HuggingFace. |
|
||||
| [`spacy.OpenLLaMA.v1`](/api/large-language-models#models-hf) | OpenLLaMA models through HuggingFace. |
|
||||
| [LangChain models](/api/large-language-models#langchain-models) | LangChain models for API retrieval. |
|
||||
|
||||
Note that the chat models variants of Llama 2 are currently not supported. This
|
||||
is because they need a particular prompting setup and don't add any discernible
|
||||
benefits in the use case of `spacy-llm` (i. e. no interactive chat) compared to
|
||||
the completion model variants.
|
||||
|
||||
### Cache {id="cache"}
|
||||
|
||||
Interacting with LLMs, either through an external API or a local instance, is
|
||||
costly. Since developing an NLP pipeline generally means a lot of exploration
|
||||
and prototyping, `spacy-llm` implements a built-in
|
||||
[cache](/api/large-language-models#cache) to avoid reprocessing the same
|
||||
documents at each run that keeps batches of documents stored on disk.
|
||||
|
||||
### Various functions {id="various-functions"}
|
||||
|
||||
| Function | Description |
|
||||
| ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| [`spacy.FewShotReader.v1`](/api/large-language-models#fewshotreader-v1) | This function is registered in spaCy's `misc` registry, and reads in examples from a `.yml`, `.yaml`, `.json` or `.jsonl` file. It uses [`srsly`](https://github.com/explosion/srsly) to read in these files and parses them depending on the file extension. |
|
||||
| [`spacy.FileReader.v1`](/api/large-language-models#filereader-v1) | This function is registered in spaCy's `misc` registry, and reads a file provided to the `path` to return a `str` representation of its contents. This function is typically used to read [Jinja](https://jinja.palletsprojects.com/en/3.1.x/) files containing the prompt template. |
|
||||
| [Normalizer functions](/api/large-language-models#normalizer-functions) | These functions provide simple normalizations for string comparisons, e.g. between a list of specified labels and a label given in the raw text of the LLM response. |
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,131 @@
|
||||
---
|
||||
title: Memory Management
|
||||
teaser: Managing Memory for persistent services
|
||||
version: 3.8
|
||||
menu:
|
||||
- ['Memory Zones', 'memoryzones']
|
||||
- ['Clearing Doc attributes', 'doc-attrs']
|
||||
---
|
||||
|
||||
spaCy maintains a few internal caches that improve speed,
|
||||
but cause memory to increase slightly over time. If you're
|
||||
running a batch process that you don't need to be long-lived,
|
||||
the increase in memory usage generally isn't a problem.
|
||||
However, if you're running spaCy inside a web service, you'll
|
||||
often want spaCy's memory usage to stay consistent. Transformer
|
||||
models can also run into memory problems sometimes, especially when
|
||||
used on a GPU.
|
||||
|
||||
## Memory zones {id="memoryzones"}
|
||||
|
||||
You can tell spaCy to free data from its internal caches (especially the
|
||||
[`Vocab`](/api/vocab)) using the [`Language.memory_zone`](/api/language#memory_zone) context manager. Enter
|
||||
the contextmanager and process your text within it, and spaCy will
|
||||
**reset its internal caches** (freeing up the associated memory) at the
|
||||
end of the block. spaCy objects created inside the memory zone must
|
||||
not be accessed once the memory zone is finished.
|
||||
|
||||
```python
|
||||
### Using memory zones
|
||||
from collections import Counter
|
||||
|
||||
def count_words(nlp, texts):
|
||||
counts = Counter()
|
||||
with nlp.memory_zone():
|
||||
for doc in nlp.pipe(texts):
|
||||
for token in doc:
|
||||
counts[token.text] += 1
|
||||
return counts
|
||||
```
|
||||
|
||||
<Infobox title="Important note" variant="warning">
|
||||
|
||||
Exiting the memory-zone invalidates all `Doc`, `Token`, `Span` and `Lexeme`
|
||||
objects that were created within it. If you access these objects
|
||||
after the memory zone exits, you may encounter a segmentation fault
|
||||
due to invalid memory access.
|
||||
|
||||
</Infobox>
|
||||
|
||||
spaCy needs the memory zone contextmanager because the processing pipeline
|
||||
can't keep track of which [`Doc`](/api/doc) objects are referring to data in the shared
|
||||
[`Vocab`](/api/vocab) cache. For instance, when spaCy encounters a new word, a new [`Lexeme`](/api/lexeme)
|
||||
entry is stored in the `Vocab`, and the `Doc` object points to this shared
|
||||
data. When the `Doc` goes out of scope, the `Vocab` has no way of knowing that
|
||||
this `Lexeme` is no longer in use.
|
||||
|
||||
The memory zone solves this problem by
|
||||
allowing you to tell the processing pipeline that all data created
|
||||
between two points is no longer in use. It is up to the you to honor
|
||||
this agreement. If you access objects that are supposed to no longer be in
|
||||
use, you may encounter a segmentation fault due to invalid memory access.
|
||||
|
||||
A common use case for memory zones will be **within a web service**. The processing
|
||||
pipeline can be loaded once, either as a context variable or a global, and each
|
||||
request can be handled within a memory zone:
|
||||
|
||||
```python
|
||||
### Memory zones with FastAPI {highlight="10,23"}
|
||||
from fastapi import FastAPI, APIRouter, Depends, Request
|
||||
import spacy
|
||||
from spacy.language import Language
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def make_app():
|
||||
app = FastAPI()
|
||||
app.state.NLP = spacy.load("en_core_web_sm")
|
||||
app.include_router(router)
|
||||
return app
|
||||
|
||||
|
||||
def get_nlp(request: Request) -> Language:
|
||||
return request.app.state.NLP
|
||||
|
||||
|
||||
@router.post("/parse")
|
||||
def parse_texts(
|
||||
*, text_batch: list[str], nlp: Language = Depends(get_nlp)
|
||||
) -> list[dict]:
|
||||
with nlp.memory_zone():
|
||||
# Put the spaCy call within a separate function, so we can't
|
||||
# leak the Doc objects outside the scope of the memory zone.
|
||||
output = _process_text(nlp, text_batch)
|
||||
return output
|
||||
|
||||
|
||||
def _process_text(nlp: Language, texts: list[str]) -> list[dict]:
|
||||
# Call spaCy, and transform the output into our own data
|
||||
# structures. This function is called from inside a memory
|
||||
# zone, so must not return the spaCy objects.
|
||||
docs = list(nlp.pipe(texts))
|
||||
return [
|
||||
{
|
||||
"tokens": [{"text": t.text} for t in doc],
|
||||
"entities": [
|
||||
{"start": e.start, "end": e.end, "label": e.label_} for e in doc.ents
|
||||
],
|
||||
}
|
||||
for doc in docs
|
||||
]
|
||||
|
||||
|
||||
app = make_app()
|
||||
```
|
||||
|
||||
## Clearing transformer tensors and other Doc attributes {id="doc-attrs"}
|
||||
|
||||
The [`Transformer`](/api/transformer) and [`Tok2Vec`](/api/tok2vec) components set intermediate values onto the `Doc`
|
||||
object during parsing. This can cause GPU memory to be exhausted if many `Doc`
|
||||
objects are kept in memory together.
|
||||
|
||||
To resolve this, you can add the [`doc_cleaner`](/api/pipeline-functions#doc_cleaner) component to your pipeline. By default
|
||||
this will clean up the [`Doc._.trf_data`](/api/transformer#custom_attributes) extension attribute and the [`Doc.tensor`](/api/doc#attributes) attribute.
|
||||
You can have it clean up other intermediate extension attributes you use in custom
|
||||
pipeline components as well.
|
||||
|
||||
```python
|
||||
### Adding the doc_cleaner
|
||||
nlp.add_pipe("doc_cleaner", config={"attrs": {"tensor": None}})
|
||||
```
|
||||
@@ -0,0 +1,570 @@
|
||||
---
|
||||
title: Models & Languages
|
||||
next: usage/facts-figures
|
||||
menu:
|
||||
- ['Quickstart', 'quickstart']
|
||||
- ['Language Support', 'languages']
|
||||
- ['Installation & Usage', 'download']
|
||||
- ['Production Use', 'production']
|
||||
---
|
||||
|
||||
spaCy's trained pipelines can be installed as **Python packages**. This means
|
||||
that they're a component of your application, just like any other module.
|
||||
They're versioned and can be defined as a dependency in your `requirements.txt`.
|
||||
Trained pipelines can be installed from a download URL or a local directory,
|
||||
manually or via [pip](https://pypi.python.org/pypi/pip). Their data can be
|
||||
located anywhere on your file system.
|
||||
|
||||
> #### Important note
|
||||
>
|
||||
> If you're upgrading to spaCy v3.x, you need to **download the new pipeline
|
||||
> packages**. If you've trained your own pipelines, you need to **retrain** them
|
||||
> after updating spaCy.
|
||||
|
||||
## Quickstart {hidden="true"}
|
||||
|
||||
<QuickstartModels
|
||||
title="Quickstart"
|
||||
id="quickstart"
|
||||
description="Install a default trained pipeline package, get the code to load it from within spaCy and an example to test it. For more options, see the section on available packages below."
|
||||
/>
|
||||
|
||||
### Usage note
|
||||
|
||||
> If lemmatization rules are available for your language, make sure to install
|
||||
> spaCy with the `lookups` option, or install
|
||||
> [`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data)
|
||||
> separately in the same environment:
|
||||
>
|
||||
> ```bash
|
||||
> $ pip install -U %%SPACY_PKG_NAME[lookups]%%SPACY_PKG_FLAGS
|
||||
> ```
|
||||
|
||||
If a trained pipeline is available for a language, you can download it using the
|
||||
[`spacy download`](/api/cli#download) command as shown above. In order to use
|
||||
languages that don't yet come with a trained pipeline, you have to import them
|
||||
directly, or use [`spacy.blank`](/api/top-level#spacy.blank):
|
||||
|
||||
```python
|
||||
from spacy.lang.yo import Yoruba
|
||||
nlp = Yoruba() # use directly
|
||||
nlp = spacy.blank("yo") # blank instance
|
||||
```
|
||||
|
||||
A blank pipeline is typically just a tokenizer. You might want to create a blank
|
||||
pipeline when you only need a tokenizer, when you want to add more components
|
||||
from scratch, or for testing purposes. Initializing the language object directly
|
||||
yields the same result as generating it using `spacy.blank()`. In both cases the
|
||||
default configuration for the chosen language is loaded, and no pretrained
|
||||
components will be available.
|
||||
|
||||
## Language support {id="languages"}
|
||||
|
||||
spaCy currently provides support for the following languages. You can help by
|
||||
improving the existing [language data](/usage/linguistic-features#language-data)
|
||||
and extending the tokenization patterns.
|
||||
[See here](https://github.com/explosion/spaCy/issues/3056) for details on how to
|
||||
contribute to development. Also see the
|
||||
[training documentation](/usage/training) for how to train your own pipelines on
|
||||
your data.
|
||||
|
||||
<Languages />
|
||||
|
||||
### Multi-language support {id="multi-language",version="2"}
|
||||
|
||||
> ```python
|
||||
> # Standard import
|
||||
> from spacy.lang.xx import MultiLanguage
|
||||
> nlp = MultiLanguage()
|
||||
>
|
||||
> # With lazy-loading
|
||||
> nlp = spacy.blank("xx")
|
||||
> ```
|
||||
|
||||
spaCy also supports pipelines trained on more than one language. This is
|
||||
especially useful for named entity recognition. The language ID used for
|
||||
multi-language or language-neutral pipelines is `xx`. The language class, a
|
||||
generic subclass containing only the base language data, can be found in
|
||||
[`lang/xx`](%%GITHUB_SPACY/spacy/lang/xx).
|
||||
|
||||
To train a pipeline using the neutral multi-language class, you can set
|
||||
`lang = "xx"` in your [training config](/usage/training#config). You can also
|
||||
\import the `MultiLanguage` class directly, or call
|
||||
[`spacy.blank("xx")`](/api/top-level#spacy.blank) for lazy-loading.
|
||||
|
||||
### Chinese language support {id="chinese",version="2.3"}
|
||||
|
||||
The Chinese language class supports three word segmentation options, `char`,
|
||||
`jieba` and `pkuseg`.
|
||||
|
||||
> #### Manual setup
|
||||
>
|
||||
> ```python
|
||||
> from spacy.lang.zh import Chinese
|
||||
>
|
||||
> # Character segmentation (default)
|
||||
> nlp = Chinese()
|
||||
> # Jieba
|
||||
> cfg = {"segmenter": "jieba"}
|
||||
> nlp = Chinese.from_config({"nlp": {"tokenizer": cfg}})
|
||||
> # PKUSeg with "mixed" model provided by pkuseg
|
||||
> cfg = {"segmenter": "pkuseg"}
|
||||
> nlp = Chinese.from_config({"nlp": {"tokenizer": cfg}})
|
||||
> nlp.tokenizer.initialize(pkuseg_model="mixed")
|
||||
> ```
|
||||
|
||||
```ini {title="config.cfg"}
|
||||
[nlp.tokenizer]
|
||||
@tokenizers = "spacy.zh.ChineseTokenizer"
|
||||
segmenter = "char"
|
||||
```
|
||||
|
||||
| Segmenter | Description |
|
||||
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `char` | **Character segmentation:** Character segmentation is the default segmentation option. It's enabled when you create a new `Chinese` language class or call `spacy.blank("zh")`. |
|
||||
| `jieba` | **Jieba:** to use [Jieba](https://github.com/fxsjy/jieba) for word segmentation, you can set the option `segmenter` to `"jieba"`. |
|
||||
| `pkuseg` | **PKUSeg**: As of spaCy v2.3.0, support for [PKUSeg](https://github.com/explosion/spacy-pkuseg) has been added to support better segmentation for Chinese OntoNotes and the provided [Chinese pipelines](/models/zh). Enable PKUSeg by setting tokenizer option `segmenter` to `"pkuseg"`. |
|
||||
|
||||
<Infobox title="Changed in v3.0" variant="warning">
|
||||
|
||||
In v3.0, the default word segmenter has switched from Jieba to character
|
||||
segmentation. Because the `pkuseg` segmenter depends on a model that can be
|
||||
loaded from a file, the model is loaded on
|
||||
[initialization](/usage/training#config-lifecycle) (typically before training).
|
||||
This ensures that your packaged Chinese model doesn't depend on a local path at
|
||||
runtime.
|
||||
|
||||
</Infobox>
|
||||
|
||||
<Accordion title="Details on spaCy's Chinese API">
|
||||
|
||||
The `initialize` method for the Chinese tokenizer class supports the following
|
||||
config settings for loading `pkuseg` models:
|
||||
|
||||
| Name | Description |
|
||||
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `pkuseg_model` | Name of a model provided by `spacy-pkuseg` or the path to a local model directory. ~~str~~ |
|
||||
| `pkuseg_user_dict` | Optional path to a file with one word per line which overrides the default `pkuseg` user dictionary. Defaults to `"default"`, the default provided dictionary. ~~str~~ |
|
||||
|
||||
The initialization settings are typically provided in the
|
||||
[training config](/usage/training#config) and the data is loaded in before
|
||||
training and serialized with the model. This allows you to load the data from a
|
||||
local path and save out your pipeline and config, without requiring the same
|
||||
local path at runtime. See the usage guide on the
|
||||
[config lifecycle](/usage/training#config-lifecycle) for more background on
|
||||
this.
|
||||
|
||||
```ini {title="config.cfg"}
|
||||
[initialize]
|
||||
|
||||
[initialize.tokenizer]
|
||||
pkuseg_model = "/path/to/model"
|
||||
pkuseg_user_dict = "default"
|
||||
```
|
||||
|
||||
You can also initialize the tokenizer for a blank language class by calling its
|
||||
`initialize` method:
|
||||
|
||||
```python {title="Examples"}
|
||||
# Initialize the pkuseg tokenizer
|
||||
cfg = {"segmenter": "pkuseg"}
|
||||
nlp = Chinese.from_config({"nlp": {"tokenizer": cfg}})
|
||||
|
||||
# Load spaCy's OntoNotes model
|
||||
nlp.tokenizer.initialize(pkuseg_model="spacy_ontonotes")
|
||||
|
||||
# Load pkuseg's "news" model
|
||||
nlp.tokenizer.initialize(pkuseg_model="news")
|
||||
|
||||
# Load local model
|
||||
nlp.tokenizer.initialize(pkuseg_model="/path/to/pkuseg_model")
|
||||
|
||||
# Override the user directory
|
||||
nlp.tokenizer.initialize(pkuseg_model="spacy_ontonotes", pkuseg_user_dict="/path/to/user_dict")
|
||||
```
|
||||
|
||||
You can also modify the user dictionary on-the-fly:
|
||||
|
||||
```python
|
||||
# Append words to user dict
|
||||
nlp.tokenizer.pkuseg_update_user_dict(["中国", "ABC"])
|
||||
|
||||
# Remove all words from user dict and replace with new words
|
||||
nlp.tokenizer.pkuseg_update_user_dict(["中国"], reset=True)
|
||||
|
||||
# Remove all words from user dict
|
||||
nlp.tokenizer.pkuseg_update_user_dict([], reset=True)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Details on trained and custom Chinese pipelines" spaced>
|
||||
|
||||
The [Chinese pipelines](/models/zh) provided by spaCy include a custom `pkuseg`
|
||||
model trained only on
|
||||
[Chinese OntoNotes 5.0](https://catalog.ldc.upenn.edu/LDC2013T19), since the
|
||||
models provided by `pkuseg` include data restricted to research use. For
|
||||
research use, `pkuseg` provides models for several different domains (`"mixed"`
|
||||
(equivalent to `"default"` from `pkuseg` packages), `"news"` `"web"`,
|
||||
`"medicine"`, `"tourism"`) and for other uses, `pkuseg` provides a simple
|
||||
[training API](https://github.com/explosion/spacy-pkuseg/blob/master/readme/readme_english.md#usage):
|
||||
|
||||
```python
|
||||
import spacy_pkuseg as pkuseg
|
||||
from spacy.lang.zh import Chinese
|
||||
|
||||
# Train pkuseg model
|
||||
pkuseg.train("train.utf8", "test.utf8", "/path/to/pkuseg_model")
|
||||
|
||||
# Load pkuseg model in spaCy Chinese tokenizer
|
||||
cfg = {"segmenter": "pkuseg"}
|
||||
nlp = Chinese.from_config({"nlp": {"tokenizer": cfg}})
|
||||
nlp.tokenizer.initialize(pkuseg_model="/path/to/pkuseg_model")
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### Japanese language support {id="japanese",version="2.3"}
|
||||
|
||||
> #### Manual setup
|
||||
>
|
||||
> ```python
|
||||
> from spacy.lang.ja import Japanese
|
||||
>
|
||||
> # Load SudachiPy with split mode A (default)
|
||||
> nlp = Japanese()
|
||||
> # Load SudachiPy with split mode B
|
||||
> cfg = {"split_mode": "B"}
|
||||
> nlp = Japanese.from_config({"nlp": {"tokenizer": cfg}})
|
||||
> ```
|
||||
|
||||
The Japanese language class uses
|
||||
[SudachiPy](https://github.com/WorksApplications/SudachiPy) for word
|
||||
segmentation and part-of-speech tagging. The default Japanese language class and
|
||||
the provided Japanese pipelines use SudachiPy split mode `A`. The tokenizer
|
||||
config can be used to configure the split mode to `A`, `B` or `C`.
|
||||
|
||||
```ini {title="config.cfg"}
|
||||
[nlp.tokenizer]
|
||||
@tokenizers = "spacy.ja.JapaneseTokenizer"
|
||||
split_mode = "A"
|
||||
```
|
||||
|
||||
Extra information, such as reading, inflection form, and the SudachiPy
|
||||
normalized form, is available in `Token.morph`. For `B` or `C` split modes,
|
||||
subtokens are stored in `Doc.user_data["sub_tokens"]`.
|
||||
|
||||
<Infobox variant="warning">
|
||||
|
||||
If you run into errors related to `sudachipy`, which is currently under active
|
||||
development, we suggest downgrading to `sudachipy==0.4.9`, which is the version
|
||||
used for training the current [Japanese pipelines](/models/ja).
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Korean language support {id="korean"}
|
||||
|
||||
> #### mecab-ko tokenizer
|
||||
>
|
||||
> ```python
|
||||
> nlp = spacy.blank("ko")
|
||||
> ```
|
||||
|
||||
The default MeCab-based Korean tokenizer requires:
|
||||
|
||||
- [mecab-ko](https://bitbucket.org/eunjeon/mecab-ko/src/master/README.md)
|
||||
- [mecab-ko-dic](https://bitbucket.org/eunjeon/mecab-ko-dic)
|
||||
- [natto-py](https://github.com/buruzaemon/natto-py)
|
||||
|
||||
For some Korean datasets and tasks, the
|
||||
[rule-based tokenizer](/usage/linguistic-features#tokenization) is better-suited
|
||||
than MeCab. To configure a Korean pipeline with the rule-based tokenizer:
|
||||
|
||||
> #### Rule-based tokenizer
|
||||
>
|
||||
> ```python
|
||||
> config = {"nlp": {"tokenizer": {"@tokenizers": "spacy.Tokenizer.v1"}}}
|
||||
> nlp = spacy.blank("ko", config=config)
|
||||
> ```
|
||||
|
||||
```ini {title="config.cfg"}
|
||||
[nlp]
|
||||
lang = "ko"
|
||||
tokenizer = {"@tokenizers" = "spacy.Tokenizer.v1"}
|
||||
```
|
||||
|
||||
<Infobox>
|
||||
|
||||
The [Korean trained pipelines](/models/ko) use the rule-based tokenizer, so no
|
||||
additional dependencies are required.
|
||||
|
||||
</Infobox>
|
||||
|
||||
## Installing and using trained pipelines {id="download"}
|
||||
|
||||
The easiest way to download a trained pipeline is via spaCy's
|
||||
[`download`](/api/cli#download) command. It takes care of finding the
|
||||
best-matching package compatible with your spaCy installation.
|
||||
|
||||
> #### Important note for v3.0
|
||||
>
|
||||
> Note that as of spaCy v3.0, shortcut links like `en` that create (potentially
|
||||
> brittle) symlinks in your spaCy installation are **deprecated**. To download
|
||||
> and load an installed pipeline package, use its full name:
|
||||
>
|
||||
> ```diff
|
||||
> - python -m spacy download en
|
||||
> + python -m spacy download en_core_web_sm
|
||||
> ```
|
||||
>
|
||||
> ```diff
|
||||
> - nlp = spacy.load("en")
|
||||
> + nlp = spacy.load("en_core_web_sm")
|
||||
> ```
|
||||
|
||||
```bash
|
||||
# Download best-matching version of a package for your spaCy installation
|
||||
$ python -m spacy download en_core_web_sm
|
||||
|
||||
# Download exact package version
|
||||
$ python -m spacy download en_core_web_sm-3.0.0 --direct
|
||||
```
|
||||
|
||||
The download command will [install the package](/usage/models#download-pip) via
|
||||
pip and place the package in your `site-packages` directory.
|
||||
|
||||
```bash
|
||||
$ pip install -U %%SPACY_PKG_NAME%%SPACY_PKG_FLAGS
|
||||
$ python -m spacy download en_core_web_sm
|
||||
```
|
||||
|
||||
```python
|
||||
import spacy
|
||||
nlp = spacy.load("en_core_web_sm")
|
||||
doc = nlp("This is a sentence.")
|
||||
```
|
||||
|
||||
If you're in a **Jupyter notebook** or similar environment, you can use the `!`
|
||||
prefix to
|
||||
[execute commands](https://ipython.org/ipython-doc/3/interactive/tutorial.html#system-shell-commands).
|
||||
Make sure to **restart your kernel** or runtime after installation (just like
|
||||
you would when installing other Python packages) to make sure that the installed
|
||||
pipeline package can be found.
|
||||
|
||||
```bash
|
||||
!python -m spacy download en_core_web_sm
|
||||
```
|
||||
|
||||
### Installation via pip {id="download-pip"}
|
||||
|
||||
To download a trained pipeline directly using
|
||||
[pip](https://pypi.python.org/pypi/pip), point `pip install` to the URL or local
|
||||
path of the wheel file or archive. Installing the wheel is usually more
|
||||
efficient.
|
||||
|
||||
> #### Pipeline Package URLs {id="pipeline-urls"}
|
||||
>
|
||||
> Pretrained pipeline distributions are hosted on
|
||||
> [Github Releases](https://github.com/explosion/spacy-models/releases), and you
|
||||
> can find download links there, as well as on the model page. You can also get
|
||||
> URLs directly from the command line by using `spacy info` with the `--url`
|
||||
> flag, which may be useful for automation.
|
||||
>
|
||||
> ```bash
|
||||
> spacy info en_core_web_sm --url
|
||||
> ```
|
||||
>
|
||||
> This command will print the URL for the latest version of a pipeline
|
||||
> compatible with the version of spaCy you're using. Note that in order to look
|
||||
> up the compatibility information an internet connection is required.
|
||||
|
||||
```bash
|
||||
# With external URL
|
||||
$ pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.0.0/en_core_web_sm-3.0.0-py3-none-any.whl
|
||||
$ pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.0.0/en_core_web_sm-3.0.0.tar.gz
|
||||
|
||||
# Using spacy info to get the external URL
|
||||
$ pip install $(spacy info en_core_web_sm --url)
|
||||
|
||||
# With local file
|
||||
$ pip install /Users/you/en_core_web_sm-3.0.0-py3-none-any.whl
|
||||
$ pip install /Users/you/en_core_web_sm-3.0.0.tar.gz
|
||||
```
|
||||
|
||||
By default, this will install the pipeline package into your `site-packages`
|
||||
directory. You can then use `spacy.load` to load it via its package name or
|
||||
[import it](#usage-import) explicitly as a module. If you need to download
|
||||
pipeline packages as part of an automated process, we recommend using pip with a
|
||||
direct link, instead of relying on spaCy's [`download`](/api/cli#download)
|
||||
command.
|
||||
|
||||
You can also add the direct download link to your application's
|
||||
`requirements.txt`. For more details, see the section on
|
||||
[working with pipeline packages in production](#production).
|
||||
|
||||
### Manual download and installation {id="download-manual"}
|
||||
|
||||
In some cases, you might prefer downloading the data manually, for example to
|
||||
place it into a custom directory. You can download the package via your browser
|
||||
from the [latest releases](https://github.com/explosion/spacy-models/releases),
|
||||
or configure your own download script using the URL of the archive file. The
|
||||
archive consists of a package directory that contains another directory with the
|
||||
pipeline data.
|
||||
|
||||
```yaml {title="Directory structure",highlight="6"}
|
||||
└── en_core_web_md-3.0.0.tar.gz # downloaded archive
|
||||
├── setup.py # setup file for pip installation
|
||||
├── meta.json # copy of pipeline meta
|
||||
└── en_core_web_md # 📦 pipeline package
|
||||
├── __init__.py # init for pip installation
|
||||
└── en_core_web_md-3.0.0 # pipeline data
|
||||
├── config.cfg # pipeline config
|
||||
├── meta.json # pipeline meta
|
||||
└── ... # directories with component data
|
||||
```
|
||||
|
||||
You can place the **pipeline package directory** anywhere on your local file
|
||||
system.
|
||||
|
||||
### Installation from Python {id="download-python"}
|
||||
|
||||
Since the [`spacy download`](/api/cli#download) command installs the pipeline as
|
||||
a **Python package**, we always recommend running it from the command line, just
|
||||
like you install other Python packages with `pip install`. However, if you need
|
||||
to, or if you want to integrate the download process into another CLI command,
|
||||
you can also import and call the `download` function used by the CLI via Python.
|
||||
|
||||
<Infobox variant="warning">
|
||||
|
||||
Keep in mind that the `download` command installs a Python package into your
|
||||
environment. In order for it to be found after installation, you will need to
|
||||
**restart or reload** your Python process so that new packages are recognized.
|
||||
|
||||
</Infobox>
|
||||
|
||||
```python
|
||||
import spacy
|
||||
spacy.cli.download("en_core_web_sm")
|
||||
```
|
||||
|
||||
### Using trained pipelines with spaCy {id="usage"}
|
||||
|
||||
To load a pipeline package, use [`spacy.load`](/api/top-level#spacy.load) with
|
||||
the package name or a path to the data directory:
|
||||
|
||||
> #### Important note for v3.0
|
||||
>
|
||||
> Note that as of spaCy v3.0, shortcut links like `en` that create (potentially
|
||||
> brittle) symlinks in your spaCy installation are **deprecated**. To download
|
||||
> and load an installed pipeline package, use its full name:
|
||||
>
|
||||
> ```diff
|
||||
> - python -m spacy download en
|
||||
> + python -m spacy download en_core_web_sm
|
||||
> ```
|
||||
|
||||
```python
|
||||
import spacy
|
||||
nlp = spacy.load("en_core_web_sm") # load package "en_core_web_sm"
|
||||
nlp = spacy.load("/path/to/en_core_web_sm") # load package from a directory
|
||||
|
||||
doc = nlp("This is a sentence.")
|
||||
```
|
||||
|
||||
<Infobox title="Tip: Preview model info" emoji="💡">
|
||||
|
||||
You can use the [`info`](/api/cli#info) command or
|
||||
[`spacy.info()`](/api/top-level#spacy.info) method to print a pipeline package's
|
||||
meta data before loading it. Each `Language` object with a loaded pipeline also
|
||||
exposes the pipeline's meta data as the attribute `meta`. For example,
|
||||
`nlp.meta['version']` will return the package version.
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Importing pipeline packages as modules {id="usage-import"}
|
||||
|
||||
If you've installed a trained pipeline via [`spacy download`](/api/cli#download)
|
||||
or directly via pip, you can also `import` it and then call its `load()` method
|
||||
with no arguments:
|
||||
|
||||
```python {executable="true"}
|
||||
import en_core_web_sm
|
||||
|
||||
nlp = en_core_web_sm.load()
|
||||
doc = nlp("This is a sentence.")
|
||||
```
|
||||
|
||||
How you choose to load your trained pipelines ultimately depends on personal
|
||||
preference. However, **for larger code bases**, we usually recommend native
|
||||
imports, as this will make it easier to integrate pipeline packages with your
|
||||
existing build process, continuous integration workflow and testing framework.
|
||||
It'll also prevent you from ever trying to load a package that is not installed,
|
||||
as your code will raise an `ImportError` immediately, instead of failing
|
||||
somewhere down the line when calling `spacy.load()`. For more details, see the
|
||||
section on [working with pipeline packages in production](#production).
|
||||
|
||||
## Using trained pipelines in production {id="production"}
|
||||
|
||||
If your application depends on one or more trained pipeline packages, you'll
|
||||
usually want to integrate them into your continuous integration workflow and
|
||||
build process. While spaCy provides a range of useful helpers for downloading
|
||||
and loading pipeline packages, the underlying functionality is entirely based on
|
||||
native Python packaging. This allows your application to handle a spaCy pipeline
|
||||
like any other package dependency.
|
||||
|
||||
### Downloading and requiring package dependencies {id="models-download"}
|
||||
|
||||
spaCy's built-in [`download`](/api/cli#download) command is mostly intended as a
|
||||
convenient, interactive wrapper. It performs compatibility checks and prints
|
||||
detailed error messages and warnings. However, if you're downloading pipeline
|
||||
packages as part of an automated build process, this only adds an unnecessary
|
||||
layer of complexity. If you know which packages your application needs, you
|
||||
should be specifying them directly.
|
||||
|
||||
Because pipeline packages are valid Python packages, you can add them to your
|
||||
application's `requirements.txt`. If you're running your own internal PyPi
|
||||
installation, you can upload the pipeline packages there. pip's
|
||||
[requirements file format](https://pip.pypa.io/en/latest/reference/requirements-file-format/)
|
||||
supports both package names to download via a PyPi server, as well as
|
||||
[direct URLs](#pipeline-urls). For instance, you can specify the
|
||||
`en_core_web_sm` model for spaCy 3.7.x as follows:
|
||||
|
||||
```text {title="requirements.txt"}
|
||||
spacy>=3.0.0,<4.0.0
|
||||
en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl
|
||||
```
|
||||
|
||||
See the [list of models](https://spacy.io/models) for model download links for
|
||||
the current spaCy version.
|
||||
|
||||
All pipeline packages are versioned and specify their spaCy dependency. This
|
||||
ensures cross-compatibility and lets you specify exact version requirements for
|
||||
each pipeline. If you've [trained](/usage/training) your own pipeline, you can
|
||||
use the [`spacy package`](/api/cli#package) command to generate the required
|
||||
meta data and turn it into a loadable package.
|
||||
|
||||
### Loading and testing pipeline packages {id="models-loading"}
|
||||
|
||||
Pipeline packages are regular Python packages, so you can also import them as a
|
||||
package using Python's native `import` syntax, and then call the `load` method
|
||||
to load the data and return an `nlp` object:
|
||||
|
||||
```python
|
||||
import en_core_web_sm
|
||||
nlp = en_core_web_sm.load()
|
||||
```
|
||||
|
||||
In general, this approach is recommended for larger code bases, as it's more
|
||||
"native", and doesn't rely on spaCy's loader to resolve string names to
|
||||
packages. If a package can't be imported, Python will raise an `ImportError`
|
||||
immediately. And if a package is imported but not used, any linter will catch
|
||||
that.
|
||||
|
||||
Similarly, it'll give you more flexibility when writing tests that require
|
||||
loading pipelines. For example, instead of writing your own `try` and `except`
|
||||
logic around spaCy's loader, you can use
|
||||
[pytest](http://pytest.readthedocs.io/en/latest/)'s
|
||||
[`importorskip()`](https://docs.pytest.org/en/latest/builtin.html#_pytest.outcomes.importorskip)
|
||||
method to only run a test if a specific pipeline package or version is
|
||||
installed. Each pipeline package exposes a `__version__` attribute which you can
|
||||
also use to perform your own version compatibility checks before loading it.
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,777 @@
|
||||
---
|
||||
title: Saving and Loading
|
||||
menu:
|
||||
- ['Basics', 'basics']
|
||||
- ['Serializing Docs', 'docs']
|
||||
- ['Serialization Methods', 'serialization-methods']
|
||||
- ['Entry Points', 'entry-points']
|
||||
- ['Trained Pipelines', 'models']
|
||||
---
|
||||
|
||||
## Basics {id="basics",hidden="true"}
|
||||
|
||||
<Serialization101 />
|
||||
|
||||
### Serializing the pipeline {id="pipeline"}
|
||||
|
||||
When serializing the pipeline, keep in mind that this will only save out the
|
||||
**binary data for the individual components** to allow spaCy to restore them –
|
||||
not the entire objects. This is a good thing, because it makes serialization
|
||||
safe. But it also means that you have to take care of storing the config, which
|
||||
contains the pipeline configuration and all the relevant settings.
|
||||
|
||||
> #### Saving the meta and config
|
||||
>
|
||||
> The [`nlp.meta`](/api/language#meta) attribute is a JSON-serializable
|
||||
> dictionary and contains all pipeline meta information like the author and
|
||||
> license information. The [`nlp.config`](/api/language#config) attribute is a
|
||||
> dictionary containing the training configuration, pipeline component factories
|
||||
> and other settings. It is saved out with a pipeline as the `config.cfg`.
|
||||
|
||||
```python {title="Serialize"}
|
||||
config = nlp.config
|
||||
bytes_data = nlp.to_bytes()
|
||||
```
|
||||
|
||||
```python {title="Deserialize"}
|
||||
lang_cls = spacy.util.get_lang_class(config["nlp"]["lang"])
|
||||
nlp = lang_cls.from_config(config)
|
||||
nlp.from_bytes(bytes_data)
|
||||
```
|
||||
|
||||
This is also how spaCy does it under the hood when loading a pipeline: it loads
|
||||
the `config.cfg` containing the language and pipeline information, initializes
|
||||
the language class, creates and adds the pipeline components based on the config
|
||||
and _then_ loads in the binary data. You can read more about this process
|
||||
[here](/usage/processing-pipelines#pipelines).
|
||||
|
||||
## Serializing Doc objects efficiently {id="docs",version="2.2"}
|
||||
|
||||
If you're working with lots of data, you'll probably need to pass analyses
|
||||
between machines, either to use something like [Dask](https://dask.org) or
|
||||
[Spark](https://spark.apache.org), or even just to save out work to disk. Often
|
||||
it's sufficient to use the [`Doc.to_array`](/api/doc#to_array) functionality for
|
||||
this, and just serialize the numpy arrays – but other times you want a more
|
||||
general way to save and restore `Doc` objects.
|
||||
|
||||
The [`DocBin`](/api/docbin) class makes it easy to serialize and deserialize a
|
||||
collection of `Doc` objects together, and is much more efficient than calling
|
||||
[`Doc.to_bytes`](/api/doc#to_bytes) on each individual `Doc` object. You can
|
||||
also control what data gets saved, and you can merge pallets together for easy
|
||||
map/reduce-style processing.
|
||||
|
||||
```python {highlight="4,8,9,13,14"}
|
||||
import spacy
|
||||
from spacy.tokens import DocBin
|
||||
|
||||
doc_bin = DocBin(attrs=["LEMMA", "ENT_IOB", "ENT_TYPE"], store_user_data=True)
|
||||
texts = ["Some text", "Lots of texts...", "..."]
|
||||
nlp = spacy.load("en_core_web_sm")
|
||||
for doc in nlp.pipe(texts):
|
||||
doc_bin.add(doc)
|
||||
bytes_data = doc_bin.to_bytes()
|
||||
|
||||
# Deserialize later, e.g. in a new process
|
||||
nlp = spacy.blank("en")
|
||||
doc_bin = DocBin().from_bytes(bytes_data)
|
||||
docs = list(doc_bin.get_docs(nlp.vocab))
|
||||
```
|
||||
|
||||
If `store_user_data` is set to `True`, the `Doc.user_data` will be serialized as
|
||||
well, which includes the values of
|
||||
[extension attributes](/usage/processing-pipelines#custom-components-attributes)
|
||||
(if they're serializable with msgpack).
|
||||
|
||||
<Infobox title="Important note on serializing extension attributes" variant="warning">
|
||||
|
||||
Including the `Doc.user_data` and extension attributes will only serialize the
|
||||
**values** of the attributes. To restore the values and access them via the
|
||||
`doc._.` property, you need to register the global attribute on the `Doc` again.
|
||||
|
||||
```python
|
||||
docs = list(doc_bin.get_docs(nlp.vocab))
|
||||
Doc.set_extension("my_custom_attr", default=None)
|
||||
print([doc._.my_custom_attr for doc in docs])
|
||||
```
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Using Pickle {id="pickle"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> doc = nlp("This is a text.")
|
||||
> data = pickle.dumps(doc)
|
||||
> ```
|
||||
|
||||
When pickling spaCy's objects like the [`Doc`](/api/doc) or the
|
||||
[`EntityRecognizer`](/api/entityrecognizer), keep in mind that they all require
|
||||
the shared [`Vocab`](/api/vocab) (which includes the string to hash mappings,
|
||||
label schemes and optional vectors). This means that their pickled
|
||||
representations can become very large, especially if you have word vectors
|
||||
loaded, because it won't only include the object itself, but also the entire
|
||||
shared vocab it depends on.
|
||||
|
||||
If you need to pickle multiple objects, try to pickle them **together** instead
|
||||
of separately. For instance, instead of pickling all pipeline components, pickle
|
||||
the entire pipeline once. And instead of pickling several `Doc` objects
|
||||
separately, pickle a list of `Doc` objects. Since they all share a reference to
|
||||
the _same_ `Vocab` object, it will only be included once.
|
||||
|
||||
```python {title="Pickling objects with shared data",highlight="8-9"}
|
||||
doc1 = nlp("Hello world")
|
||||
doc2 = nlp("This is a test")
|
||||
|
||||
doc1_data = pickle.dumps(doc1)
|
||||
doc2_data = pickle.dumps(doc2)
|
||||
print(len(doc1_data) + len(doc2_data)) # 6636116 😞
|
||||
|
||||
doc_data = pickle.dumps([doc1, doc2])
|
||||
print(len(doc_data)) # 3319761 😃
|
||||
```
|
||||
|
||||
<Infobox title="Pickling spans and tokens" variant="warning">
|
||||
|
||||
Pickling `Token` and `Span` objects isn't supported. They're only views of the
|
||||
`Doc` and can't exist on their own. Pickling them would always mean pulling in
|
||||
the parent document and its vocabulary, which has practically no advantage over
|
||||
pickling the parent `Doc`.
|
||||
|
||||
```diff
|
||||
- data = pickle.dumps(doc[10:20])
|
||||
+ data = pickle.dumps(doc)
|
||||
```
|
||||
|
||||
If you really only need a span – for example, a particular sentence – you can
|
||||
use [`Span.as_doc`](/api/span#as_doc) to make a copy of it and convert it to a
|
||||
`Doc` object. However, note that this will not let you recover contextual
|
||||
information from _outside_ the span.
|
||||
|
||||
```diff
|
||||
+ span_doc = doc[10:20].as_doc()
|
||||
data = pickle.dumps(span_doc)
|
||||
```
|
||||
|
||||
</Infobox>
|
||||
|
||||
## Implementing serialization methods {id="serialization-methods"}
|
||||
|
||||
When you call [`nlp.to_disk`](/api/language#to_disk),
|
||||
[`nlp.from_disk`](/api/language#from_disk) or load a pipeline package, spaCy
|
||||
will iterate over the components in the pipeline, check if they expose a
|
||||
`to_disk` or `from_disk` method and if so, call it with the path to the pipeline
|
||||
directory plus the string name of the component. For example, if you're calling
|
||||
`nlp.to_disk("/path")`, the data for the named entity recognizer will be saved
|
||||
in `/path/ner`.
|
||||
|
||||
If you're using custom pipeline components that depend on external data – for
|
||||
example, model weights or terminology lists – you can take advantage of spaCy's
|
||||
built-in component serialization by making your custom component expose its own
|
||||
`to_disk` and `from_disk` or `to_bytes` and `from_bytes` methods. When an `nlp`
|
||||
object with the component in its pipeline is saved or loaded, the component will
|
||||
then be able to serialize and deserialize itself.
|
||||
|
||||
<Infobox title="Custom components and data" emoji="📖">
|
||||
|
||||
For more details on how to work with pipeline components that depend on data
|
||||
resources and manage data loading and initialization at training and runtime,
|
||||
see the usage guide on initializing and serializing
|
||||
[component data](/usage/processing-pipelines#component-data).
|
||||
|
||||
</Infobox>
|
||||
|
||||
The following example shows a custom component that keeps arbitrary
|
||||
JSON-serializable data, allows the user to add to that data and saves and loads
|
||||
the data to and from a JSON file.
|
||||
|
||||
> #### Real-world example
|
||||
>
|
||||
> To see custom serialization methods in action, check out the new
|
||||
> [`EntityRuler`](/api/entityruler) component and its
|
||||
> [source](%%GITHUB_SPACY/spacy/pipeline/entityruler.py). Patterns added to the
|
||||
> component will be saved to a `.jsonl` file if the pipeline is serialized to
|
||||
> disk, and to a bytestring if the pipeline is serialized to bytes. This allows
|
||||
> saving out a pipeline with a rule-based entity recognizer and including all
|
||||
> rules _with_ the component data.
|
||||
|
||||
```python {highlight="16-23,25-30"}
|
||||
import json
|
||||
from spacy import Language
|
||||
from spacy.util import ensure_path
|
||||
|
||||
@Language.factory("my_component")
|
||||
class CustomComponent:
|
||||
def __init__(self, nlp: Language, name: str = "my_component"):
|
||||
self.name = name
|
||||
self.data = []
|
||||
|
||||
def __call__(self, doc):
|
||||
# Do something to the doc here
|
||||
return doc
|
||||
|
||||
def add(self, data):
|
||||
# Add something to the component's data
|
||||
self.data.append(data)
|
||||
|
||||
def to_disk(self, path, exclude=tuple()):
|
||||
# This will receive the directory path + /my_component
|
||||
path = ensure_path(path)
|
||||
if not path.exists():
|
||||
path.mkdir()
|
||||
data_path = path / "data.json"
|
||||
with data_path.open("w", encoding="utf8") as f:
|
||||
f.write(json.dumps(self.data))
|
||||
|
||||
def from_disk(self, path, exclude=tuple()):
|
||||
# This will receive the directory path + /my_component
|
||||
data_path = path / "data.json"
|
||||
with data_path.open("r", encoding="utf8") as f:
|
||||
self.data = json.load(f)
|
||||
return self
|
||||
```
|
||||
|
||||
After adding the component to the pipeline and adding some data to it, we can
|
||||
serialize the `nlp` object to a directory, which will call the custom
|
||||
component's `to_disk` method.
|
||||
|
||||
```python {highlight="2-4"}
|
||||
nlp = spacy.load("en_core_web_sm")
|
||||
my_component = nlp.add_pipe("my_component")
|
||||
my_component.add({"hello": "world"})
|
||||
nlp.to_disk("/path/to/pipeline")
|
||||
```
|
||||
|
||||
The contents of the directory would then look like this.
|
||||
`CustomComponent.to_disk` converted the data to a JSON string and saved it to a
|
||||
file `data.json` in its subdirectory:
|
||||
|
||||
```yaml {title="Directory structure",highlight="2-3"}
|
||||
└── /path/to/pipeline
|
||||
├── my_component # data serialized by "my_component"
|
||||
│ └── data.json
|
||||
├── ner # data for "ner" component
|
||||
├── parser # data for "parser" component
|
||||
├── tagger # data for "tagger" component
|
||||
├── vocab # pipeline vocabulary
|
||||
├── meta.json # pipeline meta.json
|
||||
├── config.cfg # pipeline config
|
||||
└── tokenizer # tokenization rules
|
||||
```
|
||||
|
||||
When you load the data back in, spaCy will call the custom component's
|
||||
`from_disk` method with the given file path, and the component can then load the
|
||||
contents of `data.json`, convert them to a Python object and restore the
|
||||
component state. The same works for other types of data, of course – for
|
||||
instance, you could add a
|
||||
[wrapper for a model](/usage/layers-architectures#frameworks) trained with a
|
||||
different library like TensorFlow or PyTorch and make spaCy load its weights
|
||||
automatically when you load the pipeline package.
|
||||
|
||||
<Infobox title="Important note on loading custom components" variant="warning">
|
||||
|
||||
When you load back a pipeline with custom components, make sure that the
|
||||
components are **available** and that the
|
||||
[`@Language.component`](/api/language#component) or
|
||||
[`@Language.factory`](/api/language#factory) decorators are executed _before_
|
||||
your pipeline is loaded back. Otherwise, spaCy won't know how to resolve the
|
||||
string name of a component factory like `"my_component"` back to a function. For
|
||||
more details, see the documentation on
|
||||
[adding factories](/usage/processing-pipelines#custom-components-factories) or
|
||||
use [entry points](#entry-points) to make your extension package expose your
|
||||
custom components to spaCy automatically.
|
||||
|
||||
</Infobox>
|
||||
|
||||
{/* ## Initializing components with data {id="initialization",version="3"} */}
|
||||
|
||||
## Using entry points {id="entry-points",version="2.1"}
|
||||
|
||||
Entry points let you expose parts of a Python package you write to other Python
|
||||
packages. This lets one application easily customize the behavior of another, by
|
||||
exposing an entry point in its `setup.py`. For a quick and fun intro to entry
|
||||
points in Python, check out
|
||||
[this excellent blog post](https://amir.rachum.com/blog/2017/07/28/python-entry-points/).
|
||||
spaCy can load custom functions from several different entry points to add
|
||||
pipeline component factories, language classes and other settings. To make spaCy
|
||||
use your entry points, your package needs to expose them and it needs to be
|
||||
installed in the same environment – that's it.
|
||||
|
||||
| Entry point | Description |
|
||||
| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| [`spacy_factories`](#entry-points-components) | Group of entry points for pipeline component factories, keyed by component name. Can be used to expose custom components defined by another package. |
|
||||
| [`spacy_languages`](#entry-points-languages) | Group of entry points for custom [`Language` subclasses](/usage/linguistic-features#language-data), keyed by language shortcut. |
|
||||
| `spacy_lookups` | Group of entry points for custom [`Lookups`](/api/lookups), including lemmatizer data. Used by spaCy's [`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data) package. |
|
||||
| [`spacy_displacy_colors`](#entry-points-displacy) | Group of entry points of custom label colors for the [displaCy visualizer](/usage/visualizers#ent). The key name doesn't matter, but it should point to a dict of labels and color values. Useful for custom models that predict different entity types. |
|
||||
|
||||
### Loading probability tables into existing models
|
||||
|
||||
You can load a probability table from
|
||||
[spacy-lookups-data](https://github.com/explosion/spacy-lookups-data) into an
|
||||
existing spaCy model like `en_core_web_sm`.
|
||||
|
||||
```python
|
||||
# Requirements: pip install spacy-lookups-data
|
||||
import spacy
|
||||
from spacy.lookups import load_lookups
|
||||
nlp = spacy.load("en_core_web_sm")
|
||||
lookups = load_lookups("en", ["lexeme_prob"])
|
||||
nlp.vocab.lookups.add_table("lexeme_prob", lookups.get_table("lexeme_prob"))
|
||||
```
|
||||
|
||||
When training a model from scratch you can also specify probability tables in
|
||||
the `config.cfg`.
|
||||
|
||||
```ini {title="config.cfg (excerpt)"}
|
||||
[initialize.lookups]
|
||||
@misc = "spacy.LookupsDataLoader.v1"
|
||||
lang = ${nlp.lang}
|
||||
tables = ["lexeme_prob"]
|
||||
```
|
||||
|
||||
### Custom components via entry points {id="entry-points-components"}
|
||||
|
||||
When you load a pipeline, spaCy will generally use its `config.cfg` to set up
|
||||
the language class and construct the pipeline. The pipeline is specified as a
|
||||
list of strings, e.g. `pipeline = ["tagger", "parser", "ner"]`. For each of
|
||||
those strings, spaCy will call `nlp.add_pipe` and look up the name in all
|
||||
factories defined by the decorators
|
||||
[`@Language.component`](/api/language#component) and
|
||||
[`@Language.factory`](/api/language#factory). This means that you have to import
|
||||
your custom components _before_ loading the pipeline.
|
||||
|
||||
Using entry points, pipeline packages and extension packages can define their
|
||||
own `"spacy_factories"`, which will be loaded automatically in the background
|
||||
when the `Language` class is initialized. So if a user has your package
|
||||
installed, they'll be able to use your components – even if they **don't import
|
||||
them**!
|
||||
|
||||
To stick with the theme of
|
||||
[this entry points blog post](https://amir.rachum.com/blog/2017/07/28/python-entry-points/),
|
||||
consider the following custom spaCy
|
||||
[pipeline component](/usage/processing-pipelines#custom-components) that prints
|
||||
a snake when it's called:
|
||||
|
||||
> #### Package directory structure
|
||||
>
|
||||
> ```yaml
|
||||
> ├── snek.py # the extension code
|
||||
> └── setup.py # setup file for pip installation
|
||||
> ```
|
||||
|
||||
```python {title="snek.py"}
|
||||
from spacy.language import Language
|
||||
|
||||
snek = """
|
||||
--..,_ _,.--.
|
||||
`'.'. .'`__ o `;__. {text}
|
||||
'.'. .'.'` '---'` `
|
||||
'.`'--....--'`.'
|
||||
`'--....--'`
|
||||
"""
|
||||
|
||||
@Language.component("snek")
|
||||
def snek_component(doc):
|
||||
print(snek.format(text=doc.text))
|
||||
return doc
|
||||
```
|
||||
|
||||
Since it's a very complex and sophisticated module, you want to split it off
|
||||
into its own package so you can version it and upload it to PyPi. You also want
|
||||
your custom package to be able to define `pipeline = ["snek"]` in its
|
||||
`config.cfg`. For that, you need to be able to tell spaCy where to find the
|
||||
component `"snek"`. If you don't do this, spaCy will raise an error when you try
|
||||
to load the pipeline because there's no built-in `"snek"` component. To add an
|
||||
entry to the factories, you can now expose it in your `setup.py` via the
|
||||
`entry_points` dictionary:
|
||||
|
||||
> #### Entry point syntax
|
||||
>
|
||||
> Python entry points for a group are formatted as a **list of strings**, with
|
||||
> each string following the syntax of `name = module:object`. In this example,
|
||||
> the created entry point is named `snek` and points to the function
|
||||
> `snek_component` in the module `snek`, i.e. `snek.py`.
|
||||
|
||||
```python {title="setup.py",highlight="5-7"}
|
||||
from setuptools import setup
|
||||
|
||||
setup(
|
||||
name="snek",
|
||||
entry_points={
|
||||
"spacy_factories": ["snek = snek:snek_component"]
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
The same package can expose multiple entry points, by the way. To make them
|
||||
available to spaCy, all you need to do is install the package in your
|
||||
environment:
|
||||
|
||||
```bash
|
||||
$ python -m pip install .
|
||||
```
|
||||
|
||||
spaCy is now able to create the pipeline component `"snek"` – even though you
|
||||
never imported `snek_component`. When you save the
|
||||
[`nlp.config`](/api/language#config) to disk, it includes an entry for your
|
||||
`"snek"` component and any pipeline you train with this config will include the
|
||||
component and know how to load it – if your `snek` package is installed.
|
||||
|
||||
> #### config.cfg (excerpt)
|
||||
>
|
||||
> ```diff
|
||||
> [nlp]
|
||||
> lang = "en"
|
||||
> + pipeline = ["snek"]
|
||||
>
|
||||
> [components]
|
||||
>
|
||||
> + [components.snek]
|
||||
> + factory = "snek"
|
||||
> ```
|
||||
|
||||
```
|
||||
>>> from spacy.lang.en import English
|
||||
>>> nlp = English()
|
||||
>>> nlp.add_pipe("snek") # this now works! 🐍🎉
|
||||
>>> doc = nlp("I am snek")
|
||||
--..,_ _,.--.
|
||||
`'.'. .'`__ o `;__. I am snek
|
||||
'.'. .'.'` '---'` `
|
||||
'.`'--....--'`.'
|
||||
`'--....--'`
|
||||
```
|
||||
|
||||
Instead of making your snek component a simple
|
||||
[stateless component](/usage/processing-pipelines#custom-components-simple), you
|
||||
could also make it a
|
||||
[factory](/usage/processing-pipelines#custom-components-factories) that takes
|
||||
settings. Your users can then pass in an optional `config` when they add your
|
||||
component to the pipeline and customize its appearance – for example, the
|
||||
`snek_style`.
|
||||
|
||||
> #### config.cfg (excerpt)
|
||||
>
|
||||
> ```diff
|
||||
> [components.snek]
|
||||
> factory = "snek"
|
||||
> + snek_style = "basic"
|
||||
> ```
|
||||
|
||||
```python
|
||||
SNEKS = {"basic": snek, "cute": cute_snek} # collection of sneks
|
||||
|
||||
@Language.factory("snek", default_config={"snek_style": "basic"})
|
||||
class SnekFactory:
|
||||
def __init__(self, nlp: Language, name: str, snek_style: str):
|
||||
self.nlp = nlp
|
||||
self.snek_style = snek_style
|
||||
self.snek = SNEKS[self.snek_style]
|
||||
|
||||
def __call__(self, doc):
|
||||
print(self.snek)
|
||||
return doc
|
||||
```
|
||||
|
||||
```diff {title="setup.py"}
|
||||
entry_points={
|
||||
- "spacy_factories": ["snek = snek:snek_component"]
|
||||
+ "spacy_factories": ["snek = snek:SnekFactory"]
|
||||
}
|
||||
```
|
||||
|
||||
The factory can also implement other pipeline component methods like `to_disk`
|
||||
and `from_disk` for serialization, or even `update` to make the component
|
||||
trainable. If a component exposes a `from_disk` method and is included in a
|
||||
pipeline, spaCy will call it on load. This lets you ship custom data with your
|
||||
pipeline package. When you save out a pipeline using `nlp.to_disk` and the
|
||||
component exposes a `to_disk` method, it will be called with the disk path.
|
||||
|
||||
```python
|
||||
from spacy.util import ensure_path
|
||||
|
||||
def to_disk(self, path, exclude=tuple()):
|
||||
path = ensure_path(path)
|
||||
if not path.exists():
|
||||
path.mkdir()
|
||||
snek_path = path / "snek.txt"
|
||||
with snek_path.open("w", encoding="utf8") as snek_file:
|
||||
snek_file.write(self.snek)
|
||||
|
||||
def from_disk(self, path, exclude=tuple()):
|
||||
snek_path = path / "snek.txt"
|
||||
with snek_path.open("r", encoding="utf8") as snek_file:
|
||||
self.snek = snek_file.read()
|
||||
return self
|
||||
```
|
||||
|
||||
The above example will serialize the current snake in a `snek.txt` in the data
|
||||
directory. When a pipeline using the `snek` component is loaded, it will open
|
||||
the `snek.txt` and make it available to the component.
|
||||
|
||||
### Custom language classes via entry points {id="entry-points-languages"}
|
||||
|
||||
To stay with the theme of the previous example and
|
||||
[this blog post on entry points](https://amir.rachum.com/blog/2017/07/28/python-entry-points/),
|
||||
let's imagine you wanted to implement your own `SnekLanguage` class for your
|
||||
custom pipeline – but you don't necessarily want to modify spaCy's code to add a
|
||||
language. In your package, you could then implement the following
|
||||
[custom language subclass](/usage/linguistic-features#language-subclass):
|
||||
|
||||
```python {title="snek.py"}
|
||||
from spacy.language import Language
|
||||
|
||||
class SnekDefaults(Language.Defaults):
|
||||
stop_words = set(["sss", "hiss"])
|
||||
|
||||
class SnekLanguage(Language):
|
||||
lang = "snk"
|
||||
Defaults = SnekDefaults
|
||||
```
|
||||
|
||||
Alongside the `spacy_factories`, there's also an entry point option for
|
||||
`spacy_languages`, which maps language codes to language-specific `Language`
|
||||
subclasses:
|
||||
|
||||
```diff {title="setup.py"}
|
||||
from setuptools import setup
|
||||
|
||||
setup(
|
||||
name="snek",
|
||||
entry_points={
|
||||
"spacy_factories": ["snek = snek:SnekFactory"],
|
||||
+ "spacy_languages": ["snk = snek:SnekLanguage"]
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
In spaCy, you can then load the custom `snk` language and it will be resolved to
|
||||
`SnekLanguage` via the custom entry point. This is especially relevant for
|
||||
pipeline packages you [train](/usage/training), which could then specify
|
||||
`lang = snk` in their `config.cfg` without spaCy raising an error because the
|
||||
language is not available in the core library.
|
||||
|
||||
### Custom displaCy colors via entry points {id="entry-points-displacy",version="2.2"}
|
||||
|
||||
If you're training a named entity recognition model for a custom domain, you may
|
||||
end up training different labels that don't have pre-defined colors in the
|
||||
[`displacy` visualizer](/usage/visualizers#ent). The `spacy_displacy_colors`
|
||||
entry point lets you define a dictionary of entity labels mapped to their color
|
||||
values. It's added to the pre-defined colors and can also overwrite existing
|
||||
values.
|
||||
|
||||
> #### Domain-specific NER labels
|
||||
>
|
||||
> Good examples of pipelines with domain-specific label schemes are
|
||||
> [scispaCy](/universe/project/scispacy) and
|
||||
> [Blackstone](/universe/project/blackstone).
|
||||
|
||||
```python {title="snek.py"}
|
||||
displacy_colors = {"SNEK": "#3dff74", "HUMAN": "#cfc5ff"}
|
||||
```
|
||||
|
||||
Given the above colors, the entry point can be defined as follows. Entry points
|
||||
need to have a name, so we use the key `colors`. However, the name doesn't
|
||||
matter and whatever is defined in the entry point group will be used.
|
||||
|
||||
```diff {title="setup.py"}
|
||||
from setuptools import setup
|
||||
|
||||
setup(
|
||||
name="snek",
|
||||
entry_points={
|
||||
+ "spacy_displacy_colors": ["colors = snek:displacy_colors"]
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
After installing the package, the custom colors will be used when visualizing
|
||||
text with `displacy`. Whenever the label `SNEK` is assigned, it will be
|
||||
displayed in `#3dff74`.
|
||||
|
||||
<Standalone height={100}>
|
||||
<div style={{lineHeight: 2.5, fontFamily: "-apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'", fontSize: 18}}>🌱🌿 <mark style={{ background: '#3dff74', padding: '0.45em 0.6em', margin: '0 0.25em', lineHeight: 1, borderRadius: '0.35em'}}>🐍 <span style={{ fontSize: '0.8em', fontWeight: 'bold', lineHeight: 1, borderRadius: '0.35em', marginLeft: '0.5rem'}}>SNEK</span></mark> ____ 🌳🌲 ____ <mark style={{ background: '#cfc5ff', padding: '0.45em 0.6em', margin: '0 0.25em', lineHeight: 1, borderRadius: '0.35em'}}>👨🌾 <span style={{ fontSize: '0.8em', fontWeight: 'bold', lineHeight: 1, borderRadius: '0.35em', marginLeft: '0.5rem'}}>HUMAN</span></mark> 🏘️</div>
|
||||
</Standalone>
|
||||
|
||||
## Saving, loading and distributing trained pipelines {id="models"}
|
||||
|
||||
After training your pipeline, you'll usually want to save its state, and load it
|
||||
back later. You can do this with the [`Language.to_disk`](/api/language#to_disk)
|
||||
method:
|
||||
|
||||
```python
|
||||
nlp.to_disk("./en_example_pipeline")
|
||||
```
|
||||
|
||||
The directory will be created if it doesn't exist, and the whole pipeline data,
|
||||
meta and configuration will be written out. To make the pipeline more convenient
|
||||
to deploy, we recommend wrapping it as a [Python package](/api/cli#package).
|
||||
|
||||
<Accordion title="What’s the difference between the config.cfg and meta.json?" spaced id="models-meta-vs-config" spaced>
|
||||
|
||||
When you save a pipeline in spaCy v3.0+, two files will be exported: a
|
||||
[`config.cfg`](/api/data-formats#config) based on
|
||||
[`nlp.config`](/api/language#config) and a [`meta.json`](/api/data-formats#meta)
|
||||
based on [`nlp.meta`](/api/language#meta).
|
||||
|
||||
- **config**: Configuration used to create the current `nlp` object, its
|
||||
pipeline components and models, as well as training settings and
|
||||
hyperparameters. Can include references to registered functions like
|
||||
[pipeline components](/usage/processing-pipelines#custom-components) or
|
||||
[model architectures](/api/architectures). Given a config, spaCy is able
|
||||
reconstruct the whole tree of objects and the `nlp` object. An exported config
|
||||
can also be used to [train a pipeline](/usage/training#config) with the same
|
||||
settings.
|
||||
- **meta**: Meta information about the pipeline and the Python package, such as
|
||||
the author information, license, version, data sources and label scheme. This
|
||||
is mostly used for documentation purposes and for packaging pipelines. It has
|
||||
no impact on the functionality of the `nlp` object.
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Project id="pipelines/tagger_parser_ud">
|
||||
|
||||
The easiest way to get started with an end-to-end workflow is to clone a
|
||||
[project template](/usage/projects) and run it – for example, this template that
|
||||
lets you train a **part-of-speech tagger** and **dependency parser** on a
|
||||
Universal Dependencies treebank and generates an installable Python package.
|
||||
|
||||
</Project>
|
||||
|
||||
### Generating a pipeline package {id="models-generating"}
|
||||
|
||||
<Infobox title="Important note" variant="warning">
|
||||
|
||||
Pipeline packages are typically **not suitable** for the public
|
||||
[pypi.python.org](https://pypi.python.org) directory, which is not designed for
|
||||
binary data and files over 50 MB. However, if your company is running an
|
||||
**internal installation** of PyPi, publishing your pipeline packages on there
|
||||
can be a convenient way to share them with your team.
|
||||
|
||||
</Infobox>
|
||||
|
||||
spaCy comes with a handy CLI command that will create all required files, and
|
||||
walk you through generating the meta data. You can also create the
|
||||
[`meta.json`](/api/data-formats#meta) manually and place it in the data
|
||||
directory, or supply a path to it using the `--meta` flag. For more info on
|
||||
this, see the [`package`](/api/cli#package) docs.
|
||||
|
||||
> #### meta.json (example)
|
||||
>
|
||||
> ```json
|
||||
> {
|
||||
> "name": "example_pipeline",
|
||||
> "lang": "en",
|
||||
> "version": "1.0.0",
|
||||
> "spacy_version": ">=2.0.0,<3.0.0",
|
||||
> "description": "Example pipeline for spaCy",
|
||||
> "author": "You",
|
||||
> "email": "you@example.com",
|
||||
> "license": "CC BY-SA 3.0"
|
||||
> }
|
||||
> ```
|
||||
|
||||
```bash
|
||||
$ python -m spacy package ./en_example_pipeline ./packages
|
||||
```
|
||||
|
||||
This command will create a pipeline package directory and will run
|
||||
`python -m build` in that directory to create a binary `.whl` file or
|
||||
`.tar.gz` archive of your package that can be installed using `pip install`.
|
||||
Installing the binary wheel is usually more efficient.
|
||||
|
||||
```yaml {title="Directory structure"}
|
||||
└── /
|
||||
├── MANIFEST.in # to include meta.json
|
||||
├── meta.json # pipeline meta data
|
||||
├── setup.py # setup file for pip installation
|
||||
├── en_example_pipeline # pipeline directory
|
||||
│ ├── __init__.py # init for pip installation
|
||||
│ └── en_example_pipeline-1.0.0 # pipeline data
|
||||
│ ├── config.cfg # pipeline config
|
||||
│ ├── meta.json # pipeline meta
|
||||
│ └── ... # directories with component data
|
||||
└── dist
|
||||
└── en_example_pipeline-1.0.0.tar.gz # installable package
|
||||
```
|
||||
|
||||
You can also find templates for all files in the
|
||||
[`cli/package.py` source](https://github.com/explosion/spacy/tree/master/spacy/cli/package.py).
|
||||
If you're creating the package manually, keep in mind that the directories need
|
||||
to be named according to the naming conventions of `lang_name` and
|
||||
`lang_name-version`.
|
||||
|
||||
### Including custom functions and components {id="models-custom"}
|
||||
|
||||
If your pipeline includes
|
||||
[custom components](/usage/processing-pipelines#custom-components), model
|
||||
architectures or other [code](/usage/training#custom-code), those functions need
|
||||
to be registered **before** your pipeline is loaded. Otherwise, spaCy won't know
|
||||
how to create the objects referenced in the config. If you're loading your own
|
||||
pipeline in Python, you can make custom components available just by importing
|
||||
the code that defines them before calling
|
||||
[`spacy.load`](/api/top-level#spacy.load). This is also how the `--code`
|
||||
argument to CLI commands works.
|
||||
|
||||
With the [`spacy package`](/api/cli#package) command, you can provide one or
|
||||
more paths to Python files containing custom registered functions using the
|
||||
`--code` argument.
|
||||
|
||||
> #### \_\_init\_\_.py (excerpt)
|
||||
>
|
||||
> ```python
|
||||
> from . import functions
|
||||
>
|
||||
> def load(**overrides):
|
||||
> ...
|
||||
> ```
|
||||
|
||||
```bash
|
||||
$ python -m spacy package ./en_example_pipeline ./packages --code functions.py
|
||||
```
|
||||
|
||||
The Python files will be copied over into the root of the package, and the
|
||||
package's `__init__.py` will import them as modules. This ensures that functions
|
||||
are registered when the pipeline is imported, e.g. when you call `spacy.load`. A
|
||||
simple import is all that's needed to make registered functions available.
|
||||
|
||||
Make sure to include **all Python files** that are referenced in your custom
|
||||
code, including modules imported by others. If your custom code depends on
|
||||
**external packages**, make sure they're listed in the list of `"requirements"`
|
||||
in your [`meta.json`](/api/data-formats#meta). For the majority of use cases,
|
||||
registered functions should provide you with all customizations you need, from
|
||||
custom components to custom model architectures and lifecycle hooks. However, if
|
||||
you do want to customize the setup in more detail, you can edit the package's
|
||||
`__init__.py` and the package's `load` function that's called by
|
||||
[`spacy.load`](/api/top-level#spacy.load).
|
||||
|
||||
<Infobox variant="warning" title="Important note on making manual edits">
|
||||
|
||||
While it's no problem to edit the package code or meta information, avoid making
|
||||
edits to the `config.cfg` **after** training, as this can easily lead to data
|
||||
incompatibility. For instance, changing an architecture or hyperparameter can
|
||||
mean that the trained weights are now incompatible. If you want to make
|
||||
adjustments, you can do so before training. Otherwise, you should always trust
|
||||
spaCy to export the current state of its `nlp` objects via
|
||||
[`nlp.config`](/api/language#config).
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Loading a custom pipeline package {id="loading"}
|
||||
|
||||
To load a pipeline from a data directory, you can use
|
||||
[`spacy.load()`](/api/top-level#spacy.load) with the local path. This will look
|
||||
for a `config.cfg` in the directory and use the `lang` and `pipeline` settings
|
||||
to initialize a `Language` class with a processing pipeline and load in the
|
||||
model data.
|
||||
|
||||
```python
|
||||
nlp = spacy.load("/path/to/pipeline")
|
||||
```
|
||||
|
||||
If you want to **load only the binary data**, you'll have to create a `Language`
|
||||
class and call [`from_disk`](/api/language#from_disk) instead.
|
||||
|
||||
```python
|
||||
nlp = spacy.blank("en").from_disk("/path/to/data")
|
||||
```
|
||||
@@ -0,0 +1,584 @@
|
||||
---
|
||||
title: 'spaCy 101: Everything you need to know'
|
||||
teaser: The most important concepts, explained in simple terms
|
||||
menu:
|
||||
- ["What's spaCy?", 'whats-spacy']
|
||||
- ['Features', 'features']
|
||||
- ['Linguistic Annotations', 'annotations']
|
||||
- ['Pipelines', 'pipelines']
|
||||
- ['Architecture', 'architecture']
|
||||
- ['Vocab', 'vocab']
|
||||
- ['Serialization', 'serialization']
|
||||
- ['Training', 'training']
|
||||
- ['Language Data', 'language-data']
|
||||
- ['Community & FAQ', 'community-faq']
|
||||
---
|
||||
|
||||
Whether you're new to spaCy, or just want to brush up on some NLP basics and
|
||||
implementation details – this page should have you covered. Each section will
|
||||
explain one of spaCy's features in simple terms and with examples or
|
||||
illustrations. Some sections will also reappear across the usage guides as a
|
||||
quick introduction.
|
||||
|
||||
> #### Help us improve the docs
|
||||
>
|
||||
> Did you spot a mistake or come across explanations that are unclear? We always
|
||||
> appreciate improvement
|
||||
> [suggestions](https://github.com/explosion/spaCy/issues) or
|
||||
> [pull requests](https://github.com/explosion/spaCy/pulls). You can find a
|
||||
> "Suggest edits" link at the bottom of each page that points you to the source.
|
||||
|
||||
<Infobox title="Take the free interactive course">
|
||||
|
||||
<Image
|
||||
src="/images/course.jpg"
|
||||
href="https://course.spacy.io"
|
||||
alt="Advanced NLP with spaCy"
|
||||
/>
|
||||
|
||||
In this course you'll learn how to use spaCy to build advanced natural language
|
||||
understanding systems, using both rule-based and machine learning approaches. It
|
||||
includes 55 exercises featuring interactive coding practice, multiple-choice
|
||||
questions and slide decks.
|
||||
|
||||
<Button to="https://course.spacy.io" variant="primary">
|
||||
{'Start the course'}
|
||||
</Button>
|
||||
|
||||
</Infobox>
|
||||
|
||||
## What's spaCy? {id="whats-spacy"}
|
||||
|
||||
<Grid cols={2}>
|
||||
|
||||
<div>
|
||||
|
||||
spaCy is a **free, open-source library** for advanced **Natural Language
|
||||
Processing** (NLP) in Python.
|
||||
|
||||
If you're working with a lot of text, you'll eventually want to know more about
|
||||
it. For example, what's it about? What do the words mean in context? Who is
|
||||
doing what to whom? What companies and products are mentioned? Which texts are
|
||||
similar to each other?
|
||||
|
||||
spaCy is designed specifically for **production use** and helps you build
|
||||
applications that process and "understand" large volumes of text. It can be used
|
||||
to build **information extraction** or **natural language understanding**
|
||||
systems, or to pre-process text for **deep learning**.
|
||||
|
||||
</div>
|
||||
|
||||
<Infobox title="Table of contents" id="toc">
|
||||
|
||||
- [Features](#features)
|
||||
- [Linguistic annotations](#annotations)
|
||||
- [Tokenization](#annotations-token)
|
||||
- [POS tags and dependencies](#annotations-pos-deps)
|
||||
- [Named entities](#annotations-ner)
|
||||
- [Word vectors and similarity](#vectors-similarity)
|
||||
- [Pipelines](#pipelines)
|
||||
- [Library architecture](#architecture)
|
||||
- [Vocab, hashes and lexemes](#vocab)
|
||||
- [Serialization](#serialization)
|
||||
- [Training](#training)
|
||||
- [Language data](#language-data)
|
||||
- [Community & FAQ](#community)
|
||||
|
||||
</Infobox>
|
||||
|
||||
</Grid>
|
||||
|
||||
### What spaCy isn't {id="what-spacy-isnt"}
|
||||
|
||||
- ❌ **spaCy is not a platform or "an API"**. Unlike a platform, spaCy does not
|
||||
provide a software as a service, or a web application. It's an open-source
|
||||
library designed to help you build NLP applications, not a consumable service.
|
||||
- ❌ **spaCy is not an out-of-the-box chat bot engine**. While spaCy can be used
|
||||
to power conversational applications, it's not designed specifically for chat
|
||||
bots, and only provides the underlying text processing capabilities.
|
||||
- ❌**spaCy is not research software**. It's built on the latest research, but
|
||||
it's designed to get things done. This leads to fairly different design
|
||||
decisions than [NLTK](https://github.com/nltk/nltk) or
|
||||
[CoreNLP](https://stanfordnlp.github.io/CoreNLP/), which were created as
|
||||
platforms for teaching and research. The main difference is that spaCy is
|
||||
integrated and opinionated. spaCy tries to avoid asking the user to choose
|
||||
between multiple algorithms that deliver equivalent functionality. Keeping the
|
||||
menu small lets spaCy deliver generally better performance and developer
|
||||
experience.
|
||||
- ❌ **spaCy is not a company**. It's an open-source library. Our company
|
||||
publishing spaCy and other software is called
|
||||
[Explosion](https://explosion.ai).
|
||||
|
||||
## Features {id="features"}
|
||||
|
||||
In the documentation, you'll come across mentions of spaCy's features and
|
||||
capabilities. Some of them refer to linguistic concepts, while others are
|
||||
related to more general machine learning functionality.
|
||||
|
||||
| Name | Description |
|
||||
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
|
||||
| **Tokenization** | Segmenting text into words, punctuations marks etc. |
|
||||
| **Part-of-speech** (POS) **Tagging** | Assigning word types to tokens, like verb or noun. |
|
||||
| **Dependency Parsing** | Assigning syntactic dependency labels, describing the relations between individual tokens, like subject or object. |
|
||||
| **Lemmatization** | Assigning the base forms of words. For example, the lemma of "was" is "be", and the lemma of "rats" is "rat". |
|
||||
| **Sentence Boundary Detection** (SBD) | Finding and segmenting individual sentences. |
|
||||
| **Named Entity Recognition** (NER) | Labelling named "real-world" objects, like persons, companies or locations. |
|
||||
| **Entity Linking** (EL) | Disambiguating textual entities to unique identifiers in a knowledge base. |
|
||||
| **Similarity** | Comparing words, text spans and documents and how similar they are to each other. |
|
||||
| **Text Classification** | Assigning categories or labels to a whole document, or parts of a document. |
|
||||
| **Rule-based Matching** | Finding sequences of tokens based on their texts and linguistic annotations, similar to regular expressions. |
|
||||
| **Training** | Updating and improving a statistical model's predictions. |
|
||||
| **Serialization** | Saving objects to files or byte strings. |
|
||||
|
||||
### Statistical models {id="statistical-models"}
|
||||
|
||||
While some of spaCy's features work independently, others require
|
||||
[trained pipelines](/models) to be loaded, which enable spaCy to **predict**
|
||||
linguistic annotations – for example, whether a word is a verb or a noun. A
|
||||
trained pipeline can consist of multiple components that use a statistical model
|
||||
trained on labeled data. spaCy currently offers trained pipelines for a variety
|
||||
of languages, which can be installed as individual Python modules. Pipeline
|
||||
packages can differ in size, speed, memory usage, accuracy and the data they
|
||||
include. The package you choose always depends on your use case and the texts
|
||||
you're working with. For a general-purpose use case, the small, default packages
|
||||
are always a good start. They typically include the following components:
|
||||
|
||||
- **Binary weights** for the part-of-speech tagger, dependency parser and named
|
||||
entity recognizer to predict those annotations in context.
|
||||
- **Lexical entries** in the vocabulary, i.e. words and their
|
||||
context-independent attributes like the shape or spelling.
|
||||
- **Data files** like lemmatization rules and lookup tables.
|
||||
- **Word vectors**, i.e. multi-dimensional meaning representations of words that
|
||||
let you determine how similar they are to each other.
|
||||
- **Configuration** options, like the language and processing pipeline settings
|
||||
and model implementations to use, to put spaCy in the correct state when you
|
||||
load the pipeline.
|
||||
|
||||
## Linguistic annotations {id="annotations"}
|
||||
|
||||
spaCy provides a variety of linguistic annotations to give you **insights into a
|
||||
text's grammatical structure**. This includes the word types, like the parts of
|
||||
speech, and how the words are related to each other. For example, if you're
|
||||
analyzing text, it makes a huge difference whether a noun is the subject of a
|
||||
sentence, or the object – or whether "google" is used as a verb, or refers to
|
||||
the website or company in a specific context.
|
||||
|
||||
> #### Loading pipelines
|
||||
>
|
||||
> ```bash
|
||||
> $ python -m spacy download en_core_web_sm
|
||||
>
|
||||
> >>> import spacy
|
||||
> >>> nlp = spacy.load("en_core_web_sm")
|
||||
> ```
|
||||
|
||||
Once you've [downloaded and installed](/usage/models) a trained pipeline, you
|
||||
can load it via [`spacy.load`](/api/top-level#spacy.load). This will return a
|
||||
`Language` object containing all components and data needed to process text. We
|
||||
usually call it `nlp`. Calling the `nlp` object on a string of text will return
|
||||
a processed `Doc`:
|
||||
|
||||
```python {executable="true"}
|
||||
import spacy
|
||||
|
||||
nlp = spacy.load("en_core_web_sm")
|
||||
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")
|
||||
for token in doc:
|
||||
print(token.text, token.pos_, token.dep_)
|
||||
```
|
||||
|
||||
Even though a `Doc` is processed – e.g. split into individual words and
|
||||
annotated – it still holds **all information of the original text**, like
|
||||
whitespace characters. You can always get the offset of a token into the
|
||||
original string, or reconstruct the original by joining the tokens and their
|
||||
trailing whitespace. This way, you'll never lose any information when processing
|
||||
text with spaCy.
|
||||
|
||||
### Tokenization {id="annotations-token"}
|
||||
|
||||
<Tokenization101 />
|
||||
|
||||
<Infobox title="Tokenization rules" emoji="📖">
|
||||
|
||||
To learn more about how spaCy's tokenization rules work in detail, how to
|
||||
**customize and replace** the default tokenizer and how to **add
|
||||
language-specific data**, see the usage guides on
|
||||
[language data](/usage/linguistic-features#language-data) and
|
||||
[customizing the tokenizer](/usage/linguistic-features#tokenization).
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Part-of-speech tags and dependencies {id="annotations-pos-deps",model="parser"}
|
||||
|
||||
<PosDeps101 />
|
||||
|
||||
<Infobox title="Part-of-speech tagging and morphology" emoji="📖">
|
||||
|
||||
To learn more about **part-of-speech tagging** and rule-based morphology, and
|
||||
how to **navigate and use the parse tree** effectively, see the usage guides on
|
||||
[part-of-speech tagging](/usage/linguistic-features#pos-tagging) and
|
||||
[using the dependency parse](/usage/linguistic-features#dependency-parse).
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Named Entities {id="annotations-ner",model="ner"}
|
||||
|
||||
<NER101 />
|
||||
|
||||
<Infobox title="Named Entity Recognition" emoji="📖">
|
||||
|
||||
To learn more about entity recognition in spaCy, how to **add your own
|
||||
entities** to a document and how to **train and update** the entity predictions
|
||||
of a model, see the usage guides on
|
||||
[named entity recognition](/usage/linguistic-features#named-entities) and
|
||||
[training pipelines](/usage/training).
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Word vectors and similarity {id="vectors-similarity",model="vectors"}
|
||||
|
||||
<Vectors101 />
|
||||
|
||||
<Infobox title="Word vectors" emoji="📖">
|
||||
|
||||
To learn more about word vectors, how to **customize them** and how to load
|
||||
**your own vectors** into spaCy, see the usage guide on
|
||||
[using word vectors and semantic similarities](/usage/linguistic-features#vectors-similarity).
|
||||
|
||||
</Infobox>
|
||||
|
||||
## Pipelines {id="pipelines"}
|
||||
|
||||
<Pipelines101 />
|
||||
|
||||
<Infobox title="Processing pipelines" emoji="📖">
|
||||
|
||||
To learn more about **how processing pipelines work** in detail, how to enable
|
||||
and disable their components, and how to **create your own**, see the usage
|
||||
guide on [language processing pipelines](/usage/processing-pipelines).
|
||||
|
||||
</Infobox>
|
||||
|
||||
## Architecture {id="architecture"}
|
||||
|
||||
<Architecture101 />
|
||||
|
||||
## Vocab, hashes and lexemes {id="vocab"}
|
||||
|
||||
Whenever possible, spaCy tries to store data in a vocabulary, the
|
||||
[`Vocab`](/api/vocab), that will be **shared by multiple documents**. To save
|
||||
memory, spaCy also encodes all strings to **hash values** – in this case for
|
||||
example, "coffee" has the hash `3197928453018144401`. Entity labels like "ORG"
|
||||
and part-of-speech tags like "VERB" are also encoded. Internally, spaCy only
|
||||
"speaks" in hash values.
|
||||
|
||||
> - **Token**: A word, punctuation mark etc. _in context_, including its
|
||||
> attributes, tags and dependencies.
|
||||
> - **Lexeme**: A "word type" with no context. Includes the word shape and
|
||||
> flags, e.g. if it's lowercase, a digit or punctuation.
|
||||
> - **Doc**: A processed container of tokens in context.
|
||||
> - **Vocab**: The collection of lexemes.
|
||||
> - **StringStore**: The dictionary mapping hash values to strings, for example
|
||||
> `3197928453018144401` → "coffee".
|
||||
|
||||
|
||||
|
||||
If you process lots of documents containing the word "coffee" in all kinds of
|
||||
different contexts, storing the exact string "coffee" every time would take up
|
||||
way too much space. So instead, spaCy hashes the string and stores it in the
|
||||
[`StringStore`](/api/stringstore). You can think of the `StringStore` as a
|
||||
**lookup table that works in both directions** – you can look up a string to get
|
||||
its hash, or a hash to get its string:
|
||||
|
||||
```python {executable="true"}
|
||||
import spacy
|
||||
|
||||
nlp = spacy.load("en_core_web_sm")
|
||||
doc = nlp("I love coffee")
|
||||
print(doc.vocab.strings["coffee"]) # 3197928453018144401
|
||||
print(doc.vocab.strings[3197928453018144401]) # 'coffee'
|
||||
```
|
||||
|
||||
Now that all strings are encoded, the entries in the vocabulary **don't need to
|
||||
include the word text** themselves. Instead, they can look it up in the
|
||||
`StringStore` via its hash value. Each entry in the vocabulary, also called
|
||||
[`Lexeme`](/api/lexeme), contains the **context-independent** information about
|
||||
a word. For example, no matter if "love" is used as a verb or a noun in some
|
||||
context, its spelling and whether it consists of alphabetic characters won't
|
||||
ever change. Its hash value will also always be the same.
|
||||
|
||||
```python {executable="true"}
|
||||
import spacy
|
||||
|
||||
nlp = spacy.load("en_core_web_sm")
|
||||
doc = nlp("I love coffee")
|
||||
for word in doc:
|
||||
lexeme = doc.vocab[word.text]
|
||||
print(lexeme.text, lexeme.orth, lexeme.shape_, lexeme.prefix_, lexeme.suffix_,
|
||||
lexeme.is_alpha, lexeme.is_digit, lexeme.is_title, lexeme.lang_)
|
||||
```
|
||||
|
||||
> - **Text**: The original text of the lexeme.
|
||||
> - **Orth**: The hash value of the lexeme.
|
||||
> - **Shape**: The abstract word shape of the lexeme.
|
||||
> - **Prefix**: By default, the first letter of the word string.
|
||||
> - **Suffix**: By default, the last three letters of the word string.
|
||||
> - **is alpha**: Does the lexeme consist of alphabetic characters?
|
||||
> - **is digit**: Does the lexeme consist of digits?
|
||||
|
||||
| Text | Orth | Shape | Prefix | Suffix | is_alpha | is_digit |
|
||||
| ------ | --------------------- | ------ | ------ | ------ | -------- | -------- |
|
||||
| I | `4690420944186131903` | `X` | I | I | `True` | `False` |
|
||||
| love | `3702023516439754181` | `xxxx` | l | ove | `True` | `False` |
|
||||
| coffee | `3197928453018144401` | `xxxx` | c | fee | `True` | `False` |
|
||||
|
||||
The mapping of words to hashes doesn't depend on any state. To make sure each
|
||||
value is unique, spaCy uses a
|
||||
[hash function](https://en.wikipedia.org/wiki/Hash_function) to calculate the
|
||||
hash **based on the word string**. This also means that the hash for "coffee"
|
||||
will always be the same, no matter which pipeline you're using or how you've
|
||||
configured spaCy.
|
||||
|
||||
However, hashes **cannot be reversed** and there's no way to resolve
|
||||
`3197928453018144401` back to "coffee". All spaCy can do is look it up in the
|
||||
vocabulary. That's why you always need to make sure all objects you create have
|
||||
access to the same vocabulary. If they don't, spaCy might not be able to find
|
||||
the strings it needs.
|
||||
|
||||
```python {executable="true"}
|
||||
import spacy
|
||||
from spacy.tokens import Doc
|
||||
from spacy.vocab import Vocab
|
||||
|
||||
nlp = spacy.load("en_core_web_sm")
|
||||
doc = nlp("I love coffee") # Original Doc
|
||||
print(doc.vocab.strings["coffee"]) # 3197928453018144401
|
||||
print(doc.vocab.strings[3197928453018144401]) # 'coffee' 👍
|
||||
|
||||
empty_doc = Doc(Vocab()) # New Doc with empty Vocab
|
||||
# empty_doc.vocab.strings[3197928453018144401] will raise an error :(
|
||||
|
||||
empty_doc.vocab.strings.add("coffee") # Add "coffee" and generate hash
|
||||
print(empty_doc.vocab.strings[3197928453018144401]) # 'coffee' 👍
|
||||
|
||||
new_doc = Doc(doc.vocab) # Create new doc with first doc's vocab
|
||||
print(new_doc.vocab.strings[3197928453018144401]) # 'coffee' 👍
|
||||
```
|
||||
|
||||
If the vocabulary doesn't contain a string for `3197928453018144401`, spaCy will
|
||||
raise an error. You can re-add "coffee" manually, but this only works if you
|
||||
actually _know_ that the document contains that word. To prevent this problem,
|
||||
spaCy will also export the `Vocab` when you save a `Doc` or `nlp` object. This
|
||||
will give you the object and its encoded annotations, plus the "key" to decode
|
||||
it.
|
||||
|
||||
## Serialization {id="serialization"}
|
||||
|
||||
<Serialization101 />
|
||||
|
||||
<Infobox title="Saving and loading" emoji="📖">
|
||||
|
||||
To learn more about how to **save and load your own pipelines**, see the usage
|
||||
guide on [saving and loading](/usage/saving-loading#models).
|
||||
|
||||
</Infobox>
|
||||
|
||||
## Training {id="training"}
|
||||
|
||||
<Training101 />
|
||||
|
||||
<Infobox title="Training pipelines and models" emoji="📖">
|
||||
|
||||
To learn more about **training and updating** pipelines, how to create training
|
||||
data and how to improve spaCy's named models, see the usage guides on
|
||||
[training](/usage/training).
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Training config and lifecycle {id="training-config"}
|
||||
|
||||
Training config files include all **settings and hyperparameters** for training
|
||||
your pipeline. Instead of providing lots of arguments on the command line, you
|
||||
only need to pass your `config.cfg` file to [`spacy train`](/api/cli#train).
|
||||
This also makes it easy to integrate custom models and architectures, written in
|
||||
your framework of choice. A pipeline's `config.cfg` is considered the "single
|
||||
source of truth", both at **training** and **runtime**.
|
||||
|
||||
> ```ini
|
||||
> ### config.cfg (excerpt)
|
||||
> [training]
|
||||
> accumulate_gradient = 3
|
||||
>
|
||||
> [training.optimizer]
|
||||
> @optimizers = "Adam.v1"
|
||||
>
|
||||
> [training.optimizer.learn_rate]
|
||||
> @schedules = "warmup_linear.v1"
|
||||
> warmup_steps = 250
|
||||
> total_steps = 20000
|
||||
> initial_rate = 0.01
|
||||
> ```
|
||||
|
||||
|
||||
|
||||
<Infobox title="Training configuration system" emoji="📖">
|
||||
|
||||
For more details on spaCy's **configuration system** and how to use it to
|
||||
customize your pipeline components, component models, training settings and
|
||||
hyperparameters, see the [training config](/usage/training#config) usage guide.
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Trainable components {id="training-components"}
|
||||
|
||||
spaCy's [`Pipe`](/api/pipe) class helps you implement your own trainable
|
||||
components that have their own model instance, make predictions over `Doc`
|
||||
objects and can be updated using [`spacy train`](/api/cli#train). This lets you
|
||||
plug fully custom machine learning components into your pipeline that can be
|
||||
configured via a single training config.
|
||||
|
||||
> #### config.cfg (excerpt)
|
||||
>
|
||||
> ```ini
|
||||
> [components.my_component]
|
||||
> factory = "my_component"
|
||||
>
|
||||
> [components.my_component.model]
|
||||
> @architectures = "my_model.v1"
|
||||
> width = 128
|
||||
> ```
|
||||
|
||||
|
||||
|
||||
<Infobox title="Custom trainable components" emoji="📖">
|
||||
|
||||
To learn more about how to implement your own **model architectures** and use
|
||||
them to power custom **trainable components**, see the usage guides on the
|
||||
[trainable component API](/usage/processing-pipelines#trainable-components) and
|
||||
implementing [layers and architectures](/usage/layers-architectures#components)
|
||||
for trainable components.
|
||||
|
||||
</Infobox>
|
||||
|
||||
## Language data {id="language-data"}
|
||||
|
||||
<LanguageData101 />
|
||||
|
||||
## Community & FAQ {id="community-faq"}
|
||||
|
||||
We're very happy to see the spaCy community grow and include a mix of people
|
||||
from all kinds of different backgrounds – computational linguistics, data
|
||||
science, deep learning, research and more. If you'd like to get involved, below
|
||||
are some answers to the most important questions and resources for further
|
||||
reading.
|
||||
|
||||
### Help, my code isn't working! {id="faq-help-code"}
|
||||
|
||||
Bugs suck, and we're doing our best to continuously improve the tests and fix
|
||||
bugs as soon as possible. Before you submit an issue, do a quick search and
|
||||
check if the problem has already been reported. If you're having installation or
|
||||
loading problems, make sure to also check out the
|
||||
[troubleshooting guide](/usage/#troubleshooting). Help with spaCy is available
|
||||
via the following platforms:
|
||||
|
||||
> #### How do I know if something is a bug?
|
||||
>
|
||||
> Of course, it's always hard to know for sure, so don't worry – we're not going
|
||||
> to be mad if a bug report turns out to be a typo in your code. As a simple
|
||||
> rule, any C-level error without a Python traceback, like a **segmentation
|
||||
> fault** or **memory error**, is **always** a spaCy bug.
|
||||
>
|
||||
> Because models are statistical, their performance will never be _perfect_.
|
||||
> However, if you come across **patterns that might indicate an underlying
|
||||
> issue**, please do file a report. Similarly, we also care about behaviors that
|
||||
> **contradict our docs**.
|
||||
|
||||
- [Stack Overflow](https://stackoverflow.com/questions/tagged/spacy): **Usage
|
||||
questions** and everything related to problems with your specific code. The
|
||||
Stack Overflow community is much larger than ours, so if your problem can be
|
||||
solved by others, you'll receive help much quicker.
|
||||
- [GitHub discussions](https://github.com/explosion/spaCy/discussions):
|
||||
**General discussion**, **project ideas** and **usage questions**. Meet other
|
||||
community members to get help with a specific code implementation, discuss
|
||||
ideas for new projects/plugins, support more languages, and share best
|
||||
practices.
|
||||
- [GitHub issue tracker](https://github.com/explosion/spaCy/issues): **Bug
|
||||
reports** and **improvement suggestions**, i.e. everything that's likely
|
||||
spaCy's fault. This also includes problems with the trained pipelines beyond
|
||||
statistical imprecisions, like patterns that point to a bug.
|
||||
|
||||
<Infobox title="Important note" variant="warning">
|
||||
|
||||
Please understand that we won't be able to provide individual support via email.
|
||||
We also believe that help is much more valuable if it's shared publicly, so that
|
||||
**more people can benefit from it**. If you come across an issue and you think
|
||||
you might be able to help, consider posting a quick update with your solution.
|
||||
No matter how simple, it can easily save someone a lot of time and headache –
|
||||
and the next time you need help, they might repay the favor.
|
||||
|
||||
</Infobox>
|
||||
|
||||
### How can I contribute to spaCy? {id="faq-contributing"}
|
||||
|
||||
You don't have to be an NLP expert or Python pro to contribute, and we're happy
|
||||
to help you get started. If you're new to spaCy, a good place to start is the
|
||||
[`help wanted (easy)` label](https://github.com/explosion/spaCy/issues?q=is%3Aissue+is%3Aopen+label%3A"help+wanted+%28easy%29")
|
||||
on GitHub, which we use to tag bugs and feature requests that are easy and
|
||||
self-contained. We also appreciate contributions to the docs – whether it's
|
||||
fixing a typo, improving an example or adding additional explanations. You'll
|
||||
find a "Suggest edits" link at the bottom of each page that points you to the
|
||||
source.
|
||||
|
||||
Another way of getting involved is to help us improve the
|
||||
[language data](/usage/linguistic-features#language-data) – especially if you
|
||||
happen to speak one of the languages currently in
|
||||
[alpha support](/usage/models#languages). Even adding simple tokenizer
|
||||
exceptions, stop words or lemmatizer data can make a big difference. It will
|
||||
also make it easier for us to provide a trained pipeline for the language in the
|
||||
future. Submitting a test that documents a bug or performance issue, or covers
|
||||
functionality that's especially important for your application is also very
|
||||
helpful. This way, you'll also make sure we never accidentally introduce
|
||||
regressions to the parts of the library that you care about the most.
|
||||
|
||||
**For more details on the types of contributions we're looking for, the code
|
||||
conventions and other useful tips, make sure to check out the
|
||||
[contributing guidelines](%%GITHUB_SPACY/CONTRIBUTING.md).**
|
||||
|
||||
<Infobox title="Code of Conduct" variant="warning">
|
||||
|
||||
spaCy adheres to the
|
||||
[Contributor Covenant Code of Conduct](http://contributor-covenant.org/version/1/4/).
|
||||
By participating, you are expected to uphold this code.
|
||||
|
||||
</Infobox>
|
||||
|
||||
### I've built something cool with spaCy – how can I get the word out? {id="faq-project-with-spacy"}
|
||||
|
||||
First, congrats – we'd love to check it out! If you think your project would be a good fit for the
|
||||
[spaCy Universe](/universe), **feel free to submit it!** Tutorials are also
|
||||
incredibly valuable to other users and a great way to get exposure. So we
|
||||
strongly encourage **writing up your experiences**, or sharing your code and
|
||||
some tips and tricks on your blog. Since our website is open-source, you can add
|
||||
your project or tutorial by making a pull request on GitHub.
|
||||
|
||||
If you would like to use the spaCy logo on your site, please get in touch and
|
||||
ask us first. However, if you want to show support and tell others that your
|
||||
project is using spaCy, you can grab one of our **spaCy badges** here:
|
||||
|
||||
<img
|
||||
src={`https://img.shields.io/badge/built%20with-spaCy-09a3d5.svg`}
|
||||
alt="Built with spaCy"
|
||||
/>
|
||||
|
||||
```markdown
|
||||
[](https://spacy.io)
|
||||
```
|
||||
|
||||
<img
|
||||
src={`https://img.shields.io/badge/made%20with%20❤%20and-spaCy-09a3d5.svg`}
|
||||
alt="Made with love and spaCy"
|
||||
/>
|
||||
|
||||
```markdown
|
||||
[](https://spacy.io)
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,309 @@
|
||||
---
|
||||
title: What's New in v2.1
|
||||
teaser: New features, backwards incompatibilities and migration guide
|
||||
menu:
|
||||
- ['New Features', 'features']
|
||||
- ['Backwards Incompatibilities', 'incompat']
|
||||
---
|
||||
|
||||
## New Features {id="features",hidden="true"}
|
||||
|
||||
spaCy v2.1 has focussed primarily on stability and performance, solidifying the
|
||||
design changes introduced in [v2.0](/usage/v2). As well as smaller models,
|
||||
faster runtime, and many bug fixes, v2.1 also introduces experimental support
|
||||
for some exciting new NLP innovations. For the full changelog, see the
|
||||
[release notes on GitHub](https://github.com/explosion/spaCy/releases/tag/v2.1.0).
|
||||
For more details and a behind-the-scenes look at the new release,
|
||||
[see our blog post](https://explosion.ai/blog/spacy-v2-1).
|
||||
|
||||
### BERT/ULMFit/Elmo-style pre-training {id="pretraining",tag="experimental"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```bash
|
||||
> $ python -m spacy pretrain ./raw_text.jsonl
|
||||
> en_core_web_lg ./pretrained-model
|
||||
> ```
|
||||
|
||||
spaCy v2.1 introduces a new CLI command, `spacy pretrain`, that can make your
|
||||
models much more accurate. It's especially useful when you have **limited
|
||||
training data**. The `spacy pretrain` command lets you use transfer learning to
|
||||
initialize your models with information from raw text, using a language model
|
||||
objective similar to the one used in Google's BERT system. We've taken
|
||||
particular care to ensure that pretraining works well even with spaCy's small
|
||||
default architecture sizes, so you don't have to compromise on efficiency to use
|
||||
it.
|
||||
|
||||
<Infobox>
|
||||
|
||||
**API:** [`spacy pretrain`](/api/cli#pretrain) **Usage: **
|
||||
[Improving accuracy with transfer learning](/usage/training#transfer-learning)
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Extended match pattern API {id="matcher-api"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> # Matches "love cats" or "likes flowers"
|
||||
> pattern1 = [{"LEMMA": {"IN": ["like", "love"]}}, {"POS": "NOUN"}]
|
||||
> # Matches tokens of length >= 10
|
||||
> pattern2 = [{"LENGTH": {">=": 10}}]
|
||||
> # Matches custom attribute with regex
|
||||
> pattern3 = [{"_": {"country": {"REGEX": "^([Uu](\\.?|nited) ?[Ss](\\.?|tates)"}}}]
|
||||
> ```
|
||||
|
||||
Instead of mapping to a single value, token patterns can now also map to a
|
||||
**dictionary of properties**. For example, to specify that the value of a lemma
|
||||
should be part of a list of values, or to set a minimum character length. It now
|
||||
also supports a `REGEX` property, as well as set membership via `IN` and
|
||||
`NOT_IN`, custom extension attributes via `_` and rich comparison for numeric
|
||||
values.
|
||||
|
||||
<Infobox>
|
||||
|
||||
**API:** [`Matcher`](/api/matcher) **Usage: **
|
||||
[Extended pattern syntax and attributes](/usage/rule-based-matching#adding-patterns-attributes-extended),
|
||||
[Regular expressions](/usage/rule-based-matching#regex)
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Easy rule-based entity recognition {id="entity-ruler"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> from spacy.pipeline import EntityRuler
|
||||
> ruler = EntityRuler(nlp)
|
||||
> ruler.add_patterns([{"label": "ORG", "pattern": "Apple"}])
|
||||
> nlp.add_pipe(ruler, before="ner")
|
||||
> ```
|
||||
|
||||
The `EntityRuler` is an exciting new component that lets you add named entities
|
||||
based on pattern dictionaries, and makes it easy to combine rule-based and
|
||||
statistical named entity recognition for even more powerful models. Entity rules
|
||||
can be phrase patterns for exact string matches, or token patterns for full
|
||||
flexibility.
|
||||
|
||||
<Infobox>
|
||||
|
||||
**API:** [`EntityRuler`](/api/entityruler) **Usage: **
|
||||
[Rule-based entity recognition](/usage/rule-based-matching#entityruler)
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Phrase matching with other attributes {id="phrasematcher"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> matcher = PhraseMatcher(nlp.vocab, attr="POS")
|
||||
> matcher.add("PATTERN", None, nlp("I love cats"))
|
||||
> doc = nlp("You like dogs")
|
||||
> matches = matcher(doc)
|
||||
> ```
|
||||
|
||||
By default, the `PhraseMatcher` will match on the verbatim token text, e.g.
|
||||
`Token.text`. By setting the `attr` argument on initialization, you can change
|
||||
**which token attribute the matcher should use** when comparing the phrase
|
||||
pattern to the matched `Doc`. For example, `LOWER` for case-insensitive matches
|
||||
or `POS` for finding sequences of the same part-of-speech tags.
|
||||
|
||||
<Infobox>
|
||||
|
||||
**API:** [`PhraseMatcher`](/api/phrasematcher) **Usage: **
|
||||
[Matching on other token attributes](/usage/rule-based-matching#phrasematcher-attrs)
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Retokenizer for merging and splitting {id="retokenizer"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> doc = nlp("I like David Bowie")
|
||||
> with doc.retokenize() as retokenizer:
|
||||
> attrs = {"LEMMA": "David Bowie"}
|
||||
> retokenizer.merge(doc[2:4], attrs=attrs)
|
||||
> ```
|
||||
|
||||
The new `Doc.retokenize` context manager allows merging spans of multiple tokens
|
||||
into one single token, and splitting single tokens into multiple tokens.
|
||||
Modifications to the `Doc`'s tokenization are stored, and then made all at once
|
||||
when the context manager exits. This is much more efficient, and less
|
||||
error-prone. `Doc.merge` and `Span.merge` still work, but they're considered
|
||||
deprecated.
|
||||
|
||||
<Infobox>
|
||||
|
||||
**API:** [`Doc.retokenize`](/api/doc#retokenize),
|
||||
[`Retokenizer.merge`](/api/doc#retokenizer.merge),
|
||||
[`Retokenizer.split`](/api/doc#retokenizer.split)<br />**Usage:
|
||||
**[Merging and splitting](/usage/linguistic-features#retokenization)
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Components and languages via entry points {id="entry-points"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> from setuptools import setup
|
||||
> setup(
|
||||
> name="custom_extension_package",
|
||||
> entry_points={
|
||||
> "spacy_factories": ["your_component = component:ComponentFactory"]
|
||||
> "spacy_languages": ["xyz = language:XYZLanguage"]
|
||||
> }
|
||||
> )
|
||||
> ```
|
||||
|
||||
Using entry points, model packages and extension packages can now define their
|
||||
own `"spacy_factories"` and `"spacy_languages"`, which will be added to the
|
||||
built-in factories and languages. If a package in the same environment exposes
|
||||
spaCy entry points, all of this happens automatically and no further user action
|
||||
is required.
|
||||
|
||||
<Infobox>
|
||||
|
||||
**Usage:** [Using entry points](/usage/saving-loading#entry-points)
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Improved documentation {id="docs"}
|
||||
|
||||
Although it looks pretty much the same, we've rebuilt the entire documentation
|
||||
using [Gatsby](https://www.gatsbyjs.org/) and [MDX](https://mdxjs.com/). It's
|
||||
now an even faster progressive web app and allows us to write all content
|
||||
entirely **in Markdown**, without having to compromise on easy-to-use custom UI
|
||||
components. We're hoping that the Markdown source will make it even easier to
|
||||
contribute to the documentation. For more details, check out the
|
||||
[styleguide](/styleguide) and
|
||||
[source](https://github.com/explosion/spacy/tree/v2.x/website). While converting
|
||||
the pages to Markdown, we've also fixed a bunch of typos, improved the existing
|
||||
pages and added some new content:
|
||||
|
||||
- **Usage Guide:** [Rule-based Matching](/usage/rule-based-matching)<br/>How to
|
||||
use the `Matcher`, `PhraseMatcher` and the new `EntityRuler`, and write
|
||||
powerful components to combine statistical models and rules.
|
||||
- **Usage Guide:** [Saving and Loading](/usage/saving-loading)<br/>Everything
|
||||
you need to know about serialization, and how to save and load pipeline
|
||||
components, package your spaCy models as Python modules and use entry points.
|
||||
- **Usage Guide: **
|
||||
[Merging and Splitting](/usage/linguistic-features#retokenization)<br />How to
|
||||
retokenize a `Doc` using the new `retokenize` context manager and merge spans
|
||||
into single tokens and split single tokens into multiple.
|
||||
- **Universe:** [Videos](/universe/category/videos) and
|
||||
[Podcasts](/universe/category/podcasts)
|
||||
- **API:** [`EntityRuler`](/api/entityruler)
|
||||
- **API:** [`Sentencizer`](/api/sentencizer)
|
||||
- **API:** [Pipeline functions](/api/pipeline-functions)
|
||||
|
||||
## Backwards incompatibilities {id="incompat"}
|
||||
|
||||
<Infobox title="Important note on models" variant="warning">
|
||||
|
||||
If you've been training **your own models**, you'll need to **retrain** them
|
||||
with the new version. Also don't forget to upgrade all models to the latest
|
||||
versions. Models for v2.0.x aren't compatible with models for v2.1.x. To check
|
||||
if all of your models are up to date, you can run the
|
||||
[`spacy validate`](/api/cli#validate) command.
|
||||
|
||||
</Infobox>
|
||||
|
||||
- Due to difficulties linking our new
|
||||
[`blis`](https://github.com/explosion/cython-blis) for faster
|
||||
platform-independent matrix multiplication, this release currently **doesn't
|
||||
work on Python 2.7 on Windows**. We expect this to be corrected in the future.
|
||||
|
||||
- While the [`Matcher`](/api/matcher) API is fully backwards compatible, its
|
||||
algorithm has changed to fix a number of bugs and performance issues. This
|
||||
means that the `Matcher` in v2.1.x may produce different results compared to
|
||||
the `Matcher` in v2.0.x.
|
||||
|
||||
- The deprecated [`Doc.merge`](/api/doc#merge) and
|
||||
[`Span.merge`](/api/span#merge) methods still work, but you may notice that
|
||||
they now run slower when merging many objects in a row. That's because the
|
||||
merging engine was rewritten to be more reliable and to support more efficient
|
||||
merging **in bulk**. To take advantage of this, you should rewrite your logic
|
||||
to use the [`Doc.retokenize`](/api/doc#retokenize) context manager and perform
|
||||
as many merges as possible together in the `with` block.
|
||||
|
||||
```diff
|
||||
- doc[1:5].merge()
|
||||
- doc[6:8].merge()
|
||||
+ with doc.retokenize() as retokenizer:
|
||||
+ retokenizer.merge(doc[1:5])
|
||||
+ retokenizer.merge(doc[6:8])
|
||||
```
|
||||
|
||||
- The serialization methods `to_disk`, `from_disk`, `to_bytes` and `from_bytes`
|
||||
now support a single `exclude` argument to provide a list of string names to
|
||||
exclude. The docs have been updated to list the available serialization fields
|
||||
for each class. The `disable` argument on the [`Language`](/api/language)
|
||||
serialization methods has been renamed to `exclude` for consistency.
|
||||
|
||||
```diff
|
||||
- nlp.to_disk("/path", disable=["parser", "ner"])
|
||||
+ nlp.to_disk("/path", exclude=["parser", "ner"])
|
||||
- data = nlp.tokenizer.to_bytes(vocab=False)
|
||||
+ data = nlp.tokenizer.to_bytes(exclude=["vocab"])
|
||||
```
|
||||
|
||||
- The .pos value for several common English words has changed, due to
|
||||
corrections to long-standing mistakes in the English tag map (see
|
||||
[issue #593](https://github.com/explosion/spaCy/issues/593) and
|
||||
[issue #3311](https://github.com/explosion/spaCy/issues/3311) for details).
|
||||
|
||||
- For better compatibility with the Universal Dependencies data, the lemmatizer
|
||||
now preserves capitalization, e.g. for proper nouns. See
|
||||
[issue #3256](https://github.com/explosion/spaCy/issues/3256) for details.
|
||||
|
||||
- The built-in rule-based sentence boundary detector is now only called
|
||||
`"sentencizer"` – the name `"sbd"` is deprecated.
|
||||
|
||||
```diff
|
||||
- sentence_splitter = nlp.create_pipe("sbd")
|
||||
+ sentence_splitter = nlp.create_pipe("sentencizer")
|
||||
```
|
||||
|
||||
- The `is_sent_start` attribute of the first token in a `Doc` now correctly
|
||||
defaults to `True`. It previously defaulted to `None`.
|
||||
|
||||
- The keyword argument `n_threads` on the `.pipe` methods is now deprecated, as
|
||||
the v2.x models cannot release the global interpreter lock. (Future versions
|
||||
may introduce a `n_process` argument for parallel inference via
|
||||
multiprocessing.)
|
||||
|
||||
- The `Doc.print_tree` method is now deprecated. If you need a custom nested
|
||||
JSON representation of a `Doc` object, you might want to write your own helper
|
||||
function. For a simple and consistent JSON representation of the `Doc` object
|
||||
and its annotations, you can now use the [`Doc.to_json`](/api/doc#to_json)
|
||||
method. Going forward, this method will output the same format as the JSON
|
||||
training data expected by [`spacy train`](/api/cli#train).
|
||||
|
||||
- The [`spacy train`](/api/cli#train) command now lets you specify a
|
||||
comma-separated list of pipeline component names, instead of separate flags
|
||||
like `--no-parser` to disable components. This is more flexible and also
|
||||
handles custom components out-of-the-box.
|
||||
|
||||
```diff
|
||||
- $ spacy train en /output train_data.json dev_data.json --no-parser
|
||||
+ $ spacy train en /output train_data.json dev_data.json --pipeline tagger,ner
|
||||
```
|
||||
|
||||
- The [`spacy init-model`](/api/cli#init-model) command now uses a `--jsonl-loc`
|
||||
argument to pass in a a newline-delimited JSON (JSONL) file containing one
|
||||
lexical entry per line instead of a separate `--freqs-loc` and
|
||||
`--clusters-loc`.
|
||||
|
||||
```diff
|
||||
- $ spacy init-model en ./model --freqs-loc ./freqs.txt --clusters-loc ./clusters.txt
|
||||
+ $ spacy init-model en ./model --jsonl-loc ./vocab.jsonl
|
||||
```
|
||||
|
||||
- Also note that some of the model licenses have changed:
|
||||
[`it_core_news_sm`](/models/it#it_core_news_sm) is now correctly licensed
|
||||
under CC BY-NC-SA 3.0, and all [English](/models/en) and [German](/models/de)
|
||||
models are now published under the MIT license.
|
||||
@@ -0,0 +1,455 @@
|
||||
---
|
||||
title: What's New in v2.2
|
||||
teaser: New features, backwards incompatibilities and migration guide
|
||||
menu:
|
||||
- ['New Features', 'features']
|
||||
- ['Backwards Incompatibilities', 'incompat']
|
||||
- ['Migrating from v2.1', 'migrating']
|
||||
---
|
||||
|
||||
## New Features {id="features",hidden="true"}
|
||||
|
||||
spaCy v2.2 features improved statistical models, new pretrained models for
|
||||
Norwegian and Lithuanian, better Dutch NER, as well as a new mechanism for
|
||||
storing language data that makes the installation about **5-10× smaller**
|
||||
on disk. We've also added a new class to efficiently **serialize annotations**,
|
||||
an improved and **10× faster** phrase matching engine, built-in scoring
|
||||
and **CLI training for text classification**, a new command to analyze and
|
||||
**debug training data**, data augmentation during training and more. For the
|
||||
full changelog, see the
|
||||
[release notes on GitHub](https://github.com/explosion/spaCy/releases/tag/v2.2.0).
|
||||
|
||||
For more details and a behind-the-scenes look at the new release,
|
||||
[see our blog post](https://explosion.ai/blog/spacy-v2-2).
|
||||
|
||||
### Better pretrained models and more languages {id="models"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```bash
|
||||
> python -m spacy download nl_core_news_sm
|
||||
> python -m spacy download nb_core_news_sm
|
||||
> python -m spacy download lt_core_news_sm
|
||||
> ```
|
||||
|
||||
The new version also features new and re-trained models for all languages and
|
||||
resolves a number of data bugs. The [Dutch model](/models/nl) has been retrained
|
||||
with a new and custom-labelled NER corpus using the same extended label scheme
|
||||
as the English models. It should now produce significantly better NER results
|
||||
overall. We've also added new core models for [Norwegian](/models/nb) (MIT) and
|
||||
[Lithuanian](/models/lt) (CC BY-SA).
|
||||
|
||||
<Infobox>
|
||||
|
||||
**Usage:** [Models directory](/models) **Benchmarks: **
|
||||
[Release notes](https://github.com/explosion/spaCy/releases/tag/v2.2.0)
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Text classification scores and CLI training {id="train-textcat-cli"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```bash
|
||||
> $ python -m spacy train en /output /train /dev \\
|
||||
> --pipeline textcat --textcat-arch simple_cnn \\
|
||||
> --textcat-multilabel
|
||||
> ```
|
||||
|
||||
When training your models using the `spacy train` command, you can now also
|
||||
include text categories in the JSON-formatted training data. The `Scorer` and
|
||||
`nlp.evaluate` now report the text classification scores, calculated as the
|
||||
F-score on positive label for binary exclusive tasks, the macro-averaged F-score
|
||||
for 3+ exclusive labels or the macro-averaged AUC ROC score for multilabel
|
||||
classification.
|
||||
|
||||
<Infobox>
|
||||
|
||||
**API:** [`spacy train`](/api/cli#train), [`Scorer`](/api/scorer),
|
||||
[`Language.evaluate`](/api/language#evaluate)
|
||||
|
||||
</Infobox>
|
||||
|
||||
### New DocBin class to efficiently serialize Doc collections
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> from spacy.tokens import DocBin
|
||||
> doc_bin = DocBin(attrs=["LEMMA", "ENT_IOB", "ENT_TYPE"], store_user_data=True)
|
||||
> for doc in nlp.pipe(texts):
|
||||
> doc_bin.add(doc)
|
||||
> bytes_data = doc_bin.to_bytes()
|
||||
> # Deserialize later, e.g. in a new process
|
||||
> nlp = spacy.blank("en")
|
||||
> doc_bin = DocBin().from_bytes(bytes_data)
|
||||
> docs = list(doc_bin.get_docs(nlp.vocab))
|
||||
> ```
|
||||
|
||||
If you're working with lots of data, you'll probably need to pass analyses
|
||||
between machines, either to use something like [Dask](https://dask.org) or
|
||||
[Spark](https://spark.apache.org), or even just to save out work to disk. Often
|
||||
it's sufficient to use the `Doc.to_array` functionality for this, and just
|
||||
serialize the numpy arrays – but other times you want a more general way to save
|
||||
and restore `Doc` objects.
|
||||
|
||||
The new `DocBin` class makes it easy to serialize and deserialize a collection
|
||||
of `Doc` objects together, and is much more efficient than calling
|
||||
`Doc.to_bytes` on each individual `Doc` object. You can also control what data
|
||||
gets saved, and you can merge pallets together for easy map/reduce-style
|
||||
processing.
|
||||
|
||||
<Infobox>
|
||||
|
||||
**API:** [`DocBin`](/api/docbin) **Usage: **
|
||||
[Serializing Doc objects](/usage/saving-loading#docs)
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Serializable lookup tables and smaller installation {id="lookups"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> data = {"foo": "bar"}
|
||||
> nlp.vocab.lookups.add_table("my_dict", data)
|
||||
>
|
||||
> def custom_component(doc):
|
||||
> table = doc.vocab.lookups.get_table("my_dict")
|
||||
> print(table.get("foo")) # look something up
|
||||
> return doc
|
||||
> ```
|
||||
|
||||
The new `Lookups` API lets you add large dictionaries and lookup tables to the
|
||||
`Vocab` and access them from the tokenizer or custom components and extension
|
||||
attributes. Internally, the tables use Bloom filters for efficient lookup
|
||||
checks. They're also fully serializable out-of-the-box. All large data resources
|
||||
like lemmatization tables have been moved to a separate package,
|
||||
[`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data) that can
|
||||
be installed alongside the core library. This allowed us to make the spaCy
|
||||
installation **5-10× smaller on disk** (depending on your platform).
|
||||
[Pretrained models](/models) now include their data files, so you only need to
|
||||
install the lookups if you want to build blank models or use lemmatization with
|
||||
languages that don't yet ship with pretrained models.
|
||||
|
||||
<Infobox>
|
||||
|
||||
**API:** [`Lookups`](/api/lookups),
|
||||
[`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data) **Usage:
|
||||
** [Adding languages: Lemmatizer](/usage/adding-languages#lemmatizer)
|
||||
|
||||
</Infobox>
|
||||
|
||||
### CLI command to debug and validate training data {id="debug-data"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```bash
|
||||
> $ python -m spacy debug-data en train.json dev.json
|
||||
> ```
|
||||
|
||||
The new `debug-data` command lets you analyze and validate your training and
|
||||
development data, get useful stats, and find problems like invalid entity
|
||||
annotations, cyclic dependencies, low data labels and more. If you're training a
|
||||
model with `spacy train` and the results seem surprising or confusing,
|
||||
`debug-data` may help you track down the problems and improve your training
|
||||
data.
|
||||
|
||||
<Accordion title="Example output">
|
||||
|
||||
```
|
||||
=========================== Data format validation ===========================
|
||||
✔ Corpus is loadable
|
||||
|
||||
=============================== Training stats ===============================
|
||||
Training pipeline: tagger, parser, ner
|
||||
Starting with blank model 'en'
|
||||
18127 training docs
|
||||
2939 evaluation docs
|
||||
⚠ 34 training examples also in evaluation data
|
||||
|
||||
============================== Vocab & Vectors ==============================
|
||||
ℹ 2083156 total words in the data (56962 unique)
|
||||
⚠ 13020 misaligned tokens in the training data
|
||||
⚠ 2423 misaligned tokens in the dev data
|
||||
10 most common words: 'the' (98429), ',' (91756), '.' (87073), 'to' (50058),
|
||||
'of' (49559), 'and' (44416), 'a' (34010), 'in' (31424), 'that' (22792), 'is'
|
||||
(18952)
|
||||
ℹ No word vectors present in the model
|
||||
|
||||
========================== Named Entity Recognition ==========================
|
||||
ℹ 18 new labels, 0 existing labels
|
||||
528978 missing values (tokens with '-' label)
|
||||
New: 'ORG' (23860), 'PERSON' (21395), 'GPE' (21193), 'DATE' (18080), 'CARDINAL'
|
||||
(10490), 'NORP' (9033), 'MONEY' (5164), 'PERCENT' (3761), 'ORDINAL' (2122),
|
||||
'LOC' (2113), 'TIME' (1616), 'WORK_OF_ART' (1229), 'QUANTITY' (1150), 'FAC'
|
||||
(1134), 'EVENT' (974), 'PRODUCT' (935), 'LAW' (444), 'LANGUAGE' (338)
|
||||
✔ Good amount of examples for all labels
|
||||
✔ Examples without occurrences available for all labels
|
||||
✔ No entities consisting of or starting/ending with whitespace
|
||||
|
||||
=========================== Part-of-speech Tagging ===========================
|
||||
ℹ 49 labels in data (57 labels in tag map)
|
||||
'NN' (266331), 'IN' (227365), 'DT' (185600), 'NNP' (164404), 'JJ' (119830),
|
||||
'NNS' (110957), '.' (101482), ',' (92476), 'RB' (90090), 'PRP' (90081), 'VB'
|
||||
(74538), 'VBD' (68199), 'CC' (62862), 'VBZ' (50712), 'VBP' (43420), 'VBN'
|
||||
(42193), 'CD' (40326), 'VBG' (34764), 'TO' (31085), 'MD' (25863), 'PRP$'
|
||||
(23335), 'HYPH' (13833), 'POS' (13427), 'UH' (13322), 'WP' (10423), 'WDT'
|
||||
(9850), 'RP' (8230), 'WRB' (8201), ':' (8168), '''' (7392), '``' (6984), 'NNPS'
|
||||
(5817), 'JJR' (5689), '$' (3710), 'EX' (3465), 'JJS' (3118), 'RBR' (2872),
|
||||
'-RRB-' (2825), '-LRB-' (2788), 'PDT' (2078), 'XX' (1316), 'RBS' (1142), 'FW'
|
||||
(794), 'NFP' (557), 'SYM' (440), 'WP$' (294), 'LS' (293), 'ADD' (191), 'AFX'
|
||||
(24)
|
||||
✔ All labels present in tag map for language 'en'
|
||||
|
||||
============================= Dependency Parsing =============================
|
||||
ℹ Found 111703 sentences with an average length of 18.6 words.
|
||||
ℹ Found 2251 nonprojective train sentences
|
||||
ℹ Found 303 nonprojective dev sentences
|
||||
ℹ 47 labels in train data
|
||||
ℹ 211 labels in projectivized train data
|
||||
'punct' (236796), 'prep' (188853), 'pobj' (182533), 'det' (172674), 'nsubj'
|
||||
(169481), 'compound' (116142), 'ROOT' (111697), 'amod' (107945), 'dobj' (93540),
|
||||
'aux' (86802), 'advmod' (86197), 'cc' (62679), 'conj' (59575), 'poss' (36449),
|
||||
'ccomp' (36343), 'advcl' (29017), 'mark' (27990), 'nummod' (24582), 'relcl'
|
||||
(21359), 'xcomp' (21081), 'attr' (18347), 'npadvmod' (17740), 'acomp' (17204),
|
||||
'auxpass' (15639), 'appos' (15368), 'neg' (15266), 'nsubjpass' (13922), 'case'
|
||||
(13408), 'acl' (12574), 'pcomp' (10340), 'nmod' (9736), 'intj' (9285), 'prt'
|
||||
(8196), 'quantmod' (7403), 'dep' (4300), 'dative' (4091), 'agent' (3908), 'expl'
|
||||
(3456), 'parataxis' (3099), 'oprd' (2326), 'predet' (1946), 'csubj' (1494),
|
||||
'subtok' (1147), 'preconj' (692), 'meta' (469), 'csubjpass' (64), 'iobj' (1)
|
||||
⚠ Low number of examples for label 'iobj' (1)
|
||||
⚠ Low number of examples for 130 labels in the projectivized dependency
|
||||
trees used for training. You may want to projectivize labels such as punct
|
||||
before training in order to improve parser performance.
|
||||
⚠ Projectivized labels with low numbers of examples: appos||attr: 12
|
||||
advmod||dobj: 13 prep||ccomp: 12 nsubjpass||ccomp: 15 pcomp||prep: 14
|
||||
amod||dobj: 9 attr||xcomp: 14 nmod||nsubj: 17 prep||advcl: 2 prep||prep: 5
|
||||
nsubj||conj: 12 advcl||advmod: 18 ccomp||advmod: 11 ccomp||pcomp: 5 acl||pobj:
|
||||
10 npadvmod||acomp: 7 dobj||pcomp: 14 nsubjpass||pcomp: 1 nmod||pobj: 8
|
||||
amod||attr: 6 nmod||dobj: 12 aux||conj: 1 neg||conj: 1 dative||xcomp: 11
|
||||
pobj||dative: 3 xcomp||acomp: 19 advcl||pobj: 2 nsubj||advcl: 2 csubj||ccomp: 1
|
||||
advcl||acl: 1 relcl||nmod: 2 dobj||advcl: 10 advmod||advcl: 3 nmod||nsubjpass: 6
|
||||
amod||pobj: 5 cc||neg: 1 attr||ccomp: 16 advcl||xcomp: 3 nmod||attr: 4
|
||||
advcl||nsubjpass: 5 advcl||ccomp: 4 ccomp||conj: 1 punct||acl: 1 meta||acl: 1
|
||||
parataxis||acl: 1 prep||acl: 1 amod||nsubj: 7 ccomp||ccomp: 3 acomp||xcomp: 5
|
||||
dobj||acl: 5 prep||oprd: 6 advmod||acl: 2 dative||advcl: 1 pobj||agent: 5
|
||||
xcomp||amod: 1 dep||advcl: 1 prep||amod: 8 relcl||compound: 1 advcl||csubj: 3
|
||||
npadvmod||conj: 2 npadvmod||xcomp: 4 advmod||nsubj: 3 ccomp||amod: 7
|
||||
advcl||conj: 1 nmod||conj: 2 advmod||nsubjpass: 2 dep||xcomp: 2 appos||ccomp: 1
|
||||
advmod||dep: 1 advmod||advmod: 5 aux||xcomp: 8 dep||advmod: 1 dative||ccomp: 2
|
||||
prep||dep: 1 conj||conj: 1 dep||ccomp: 4 cc||ROOT: 1 prep||ROOT: 1 nsubj||pcomp:
|
||||
3 advmod||prep: 2 relcl||dative: 1 acl||conj: 1 advcl||attr: 4 prep||npadvmod: 1
|
||||
nsubjpass||xcomp: 1 neg||advmod: 1 xcomp||oprd: 1 advcl||advcl: 1 dobj||dep: 3
|
||||
nsubjpass||parataxis: 1 attr||pcomp: 1 ccomp||parataxis: 1 advmod||attr: 1
|
||||
nmod||oprd: 1 appos||nmod: 2 advmod||relcl: 1 appos||npadvmod: 1 appos||conj: 1
|
||||
prep||expl: 1 nsubjpass||conj: 1 punct||pobj: 1 cc||pobj: 1 conj||pobj: 1
|
||||
punct||conj: 1 ccomp||dep: 1 oprd||xcomp: 3 ccomp||xcomp: 1 ccomp||nsubj: 1
|
||||
nmod||dep: 1 xcomp||ccomp: 1 acomp||advcl: 1 intj||advmod: 1 advmod||acomp: 2
|
||||
relcl||oprd: 1 advmod||prt: 1 advmod||pobj: 1 appos||nummod: 1 relcl||npadvmod:
|
||||
3 mark||advcl: 1 aux||ccomp: 1 amod||nsubjpass: 1 npadvmod||advmod: 1 conj||dep:
|
||||
1 nummod||pobj: 1 amod||npadvmod: 1 intj||pobj: 1 nummod||npadvmod: 1
|
||||
xcomp||xcomp: 1 aux||dep: 1 advcl||relcl: 1
|
||||
⚠ The following labels were found only in the train data: xcomp||amod,
|
||||
advcl||relcl, prep||nsubjpass, acl||nsubj, nsubjpass||conj, xcomp||oprd,
|
||||
advmod||conj, advmod||advmod, iobj, advmod||nsubjpass, dobj||conj, ccomp||amod,
|
||||
meta||acl, xcomp||xcomp, prep||attr, prep||ccomp, advcl||acomp, acl||dobj,
|
||||
advcl||advcl, pobj||agent, prep||advcl, nsubjpass||xcomp, prep||dep,
|
||||
acomp||xcomp, aux||ccomp, ccomp||dep, conj||dep, relcl||compound,
|
||||
nsubjpass||ccomp, nmod||dobj, advmod||advcl, advmod||acl, dobj||advcl,
|
||||
dative||xcomp, prep||nsubj, ccomp||ccomp, nsubj||ccomp, xcomp||acomp,
|
||||
prep||acomp, dep||advmod, acl||pobj, appos||dobj, npadvmod||acomp, cc||ROOT,
|
||||
relcl||nsubj, nmod||pobj, acl||nsubjpass, ccomp||advmod, pcomp||prep,
|
||||
amod||dobj, advmod||attr, advcl||csubj, appos||attr, dobj||pcomp, prep||ROOT,
|
||||
relcl||pobj, advmod||pobj, amod||nsubj, ccomp||xcomp, prep||oprd,
|
||||
npadvmod||advmod, appos||nummod, advcl||pobj, neg||advmod, acl||attr,
|
||||
appos||nsubjpass, csubj||ccomp, amod||nsubjpass, intj||pobj, dep||advcl,
|
||||
cc||neg, xcomp||ccomp, dative||ccomp, nmod||oprd, pobj||dative, prep||dobj,
|
||||
dep||ccomp, relcl||attr, ccomp||nsubj, advcl||xcomp, nmod||dep, advcl||advmod,
|
||||
ccomp||conj, pobj||prep, advmod||acomp, advmod||relcl, attr||pcomp,
|
||||
ccomp||parataxis, oprd||xcomp, intj||advmod, nmod||nsubjpass, prep||npadvmod,
|
||||
parataxis||acl, prep||pobj, advcl||dobj, amod||pobj, prep||acl, conj||pobj,
|
||||
advmod||dep, punct||pobj, ccomp||acomp, acomp||advcl, nummod||npadvmod,
|
||||
dobj||dep, npadvmod||xcomp, advcl||conj, relcl||npadvmod, punct||acl,
|
||||
relcl||dobj, dobj||xcomp, nsubjpass||parataxis, dative||advcl, relcl||nmod,
|
||||
advcl||ccomp, appos||npadvmod, ccomp||pcomp, prep||amod, mark||advcl,
|
||||
prep||advmod, prep||xcomp, appos||nsubj, attr||ccomp, advmod||prt, dobj||ccomp,
|
||||
aux||conj, advcl||nsubj, conj||conj, advmod||ccomp, advcl||nsubjpass,
|
||||
attr||xcomp, nmod||conj, npadvmod||conj, relcl||dative, prep||expl,
|
||||
nsubjpass||pcomp, advmod||xcomp, advmod||dobj, appos||pobj, nsubj||conj,
|
||||
relcl||nsubjpass, advcl||attr, appos||ccomp, advmod||prep, prep||conj,
|
||||
nmod||attr, punct||conj, neg||conj, dep||xcomp, aux||xcomp, dobj||acl,
|
||||
nummod||pobj, amod||npadvmod, nsubj||pcomp, advcl||acl, appos||nmod,
|
||||
relcl||oprd, prep||prep, cc||pobj, nmod||nsubj, amod||attr, aux||dep,
|
||||
appos||conj, advmod||nsubj, nsubj||advcl, acl||conj
|
||||
To train a parser, your data should include at least 20 instances of each label.
|
||||
⚠ Multiple root labels (ROOT, nsubj, aux, npadvmod, prep) found in
|
||||
training data. spaCy's parser uses a single root label ROOT so this distinction
|
||||
will not be available.
|
||||
|
||||
================================== Summary ==================================
|
||||
✔ 5 checks passed
|
||||
⚠ 8 warnings
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Infobox>
|
||||
|
||||
**API:** [`spacy debug-data`](/api/cli#debug-data)
|
||||
|
||||
</Infobox>
|
||||
|
||||
## Backwards incompatibilities {id="incompat"}
|
||||
|
||||
<Infobox title="Important note on models" variant="warning">
|
||||
|
||||
If you've been training **your own models**, you'll need to **retrain** them
|
||||
with the new version. Also don't forget to upgrade all models to the latest
|
||||
versions. Models for v2.0 or v2.1 aren't compatible with models for v2.2. To
|
||||
check if all of your models are up to date, you can run the
|
||||
[`spacy validate`](/api/cli#validate) command.
|
||||
|
||||
</Infobox>
|
||||
|
||||
> #### Install with lookups data
|
||||
>
|
||||
> ```bash
|
||||
> $ pip install spacy[lookups]
|
||||
> ```
|
||||
>
|
||||
> You can also install
|
||||
> [`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data)
|
||||
> directly.
|
||||
|
||||
- The lemmatization tables have been moved to their own package,
|
||||
[`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data), which
|
||||
is not installed by default. If you're using pretrained models, **nothing
|
||||
changes**, because the tables are now included in the model packages. If you
|
||||
want to use the lemmatizer for other languages that don't yet have pretrained
|
||||
models (e.g. Turkish or Croatian) or start off with a blank model that
|
||||
contains lookup data (e.g. `spacy.blank("en")`), you'll need to **explicitly
|
||||
install spaCy plus data** via `pip install spacy[lookups]`.
|
||||
- Lemmatization tables (rules, exceptions, index and lookups) are now part of
|
||||
the `Vocab` and serialized with it. This means that serialized objects (`nlp`,
|
||||
pipeline components, vocab) will now include additional data, and models
|
||||
written to disk will include additional files.
|
||||
- The [`Lemmatizer`](/api/lemmatizer) class is now initialized with an instance
|
||||
of [`Lookups`](/api/lookups) containing the rules and tables, instead of dicts
|
||||
as separate arguments. This makes it easier to share data tables and modify
|
||||
them at runtime. This is mostly internals, but if you've been implementing a
|
||||
custom `Lemmatizer`, you'll need to update your code.
|
||||
- The [Dutch model](/models/nl) has been trained on a new NER corpus (custom
|
||||
labelled UD instead of WikiNER), so their predictions may be very different
|
||||
compared to the previous version. The results should be significantly better
|
||||
and more generalizable, though.
|
||||
- The [`spacy download`](/api/cli#download) command does **not** set the
|
||||
`--no-deps` pip argument anymore by default, meaning that model package
|
||||
dependencies (if available) will now be also downloaded and installed. If
|
||||
spaCy (which is also a model dependency) is not installed in the current
|
||||
environment, e.g. if a user has built from source, `--no-deps` is added back
|
||||
automatically to prevent spaCy from being downloaded and installed again from
|
||||
pip.
|
||||
- The built-in
|
||||
[`biluo_tags_from_offsets`](/api/top-level#biluo_tags_from_offsets) converter
|
||||
is now stricter and will raise an error if entities are overlapping (instead
|
||||
of silently skipping them). If your data contains invalid entity annotations,
|
||||
make sure to clean it and resolve conflicts. You can now also use the new
|
||||
`debug-data` command to find problems in your data.
|
||||
- Pipeline components can now overwrite IOB tags of tokens that are not yet part
|
||||
of an entity. Once a token has an `ent_iob` value set, it won't be reset to an
|
||||
"unset" state and will always have at least `O` assigned. `list(doc.ents)` now
|
||||
actually keeps the annotations on the token level consistent, instead of
|
||||
resetting `O` to an empty string.
|
||||
- The default punctuation in the [`Sentencizer`](/api/sentencizer) has been
|
||||
extended and now includes more characters common in various languages. This
|
||||
also means that the results it produces may change, depending on your text. If
|
||||
you want the previous behavior with limited characters, set
|
||||
`punct_chars=[".", "!", "?"]` on initialization.
|
||||
- The [`PhraseMatcher`](/api/phrasematcher) algorithm was rewritten from scratch
|
||||
and it's now 10× faster. The rewrite also resolved a few subtle bugs
|
||||
with very large terminology lists. So if you were matching large lists, you
|
||||
may see slightly different results – however, the results should now be fully
|
||||
correct. See [this PR](https://github.com/explosion/spaCy/pull/4309) for more
|
||||
details.
|
||||
- The `Serbian` language class (introduced in v2.1.8) incorrectly used the
|
||||
language code `rs` instead of `sr`. This has now been fixed, so `Serbian` is
|
||||
now available via `spacy.lang.sr`.
|
||||
- The `"sources"` in the `meta.json` have changed from a list of strings to a
|
||||
list of dicts. This is mostly internals, but if your code used
|
||||
`nlp.meta["sources"]`, you might have to update it.
|
||||
|
||||
### Migrating from spaCy 2.1 {id="migrating"}
|
||||
|
||||
#### Lemmatization data and lookup tables
|
||||
|
||||
If you application needs lemmatization for [languages](/usage/models#languages)
|
||||
with only tokenizers, you now need to install that data explicitly via
|
||||
`pip install spacy[lookups]` or `pip install spacy-lookups-data`. No additional
|
||||
setup is required – the package just needs to be installed in the same
|
||||
environment as spaCy.
|
||||
|
||||
```python {highlight="3-4"}
|
||||
nlp = Turkish()
|
||||
doc = nlp("Bu bir cümledir.")
|
||||
# 🚨 This now requires the lookups data to be installed explicitly
|
||||
print([token.lemma_ for token in doc])
|
||||
```
|
||||
|
||||
The same applies to blank models that you want to update and train – for
|
||||
instance, you might use [`spacy.blank`](/api/top-level#spacy.blank) to create a
|
||||
blank English model and then train your own part-of-speech tagger on top. If you
|
||||
don't explicitly install the lookups data, that `nlp` object won't have any
|
||||
lemmatization rules available. spaCy will now show you a warning when you train
|
||||
a new part-of-speech tagger and the vocab has no lookups available.
|
||||
|
||||
#### Lemmatizer initialization
|
||||
|
||||
This is mainly internals and should hopefully not affect your code. But if
|
||||
you've been creating custom [`Lemmatizers`](/api/lemmatizer), you'll need to
|
||||
update how they're initialized and pass in an instance of
|
||||
[`Lookups`](/api/lookups) with the (optional) tables `lemma_index`, `lemma_exc`,
|
||||
`lemma_rules` and `lemma_lookup`.
|
||||
|
||||
```diff
|
||||
from spacy.lemmatizer import Lemmatizer
|
||||
+ from spacy.lookups import Lookups
|
||||
|
||||
lemma_index = {"verb": ("cope", "cop")}
|
||||
lemma_exc = {"verb": {"coping": ("cope",)}}
|
||||
lemma_rules = {"verb": [["ing", ""]]}
|
||||
- lemmatizer = Lemmatizer(lemma_index, lemma_exc, lemma_rules)
|
||||
+ lookups = Lookups()
|
||||
+ lookups.add_table("lemma_index", lemma_index)
|
||||
+ lookups.add_table("lemma_exc", lemma_exc)
|
||||
+ lookups.add_table("lemma_rules", lemma_rules)
|
||||
+ lemmatizer = Lemmatizer(lookups)
|
||||
```
|
||||
|
||||
#### Converting entity offsets to BILUO tags
|
||||
|
||||
If you've been using the
|
||||
[`biluo_tags_from_offsets`](/api/top-level#biluo_tags_from_offsets) helper to
|
||||
convert character offsets into token-based BILUO tags, you may now see an error
|
||||
if the offsets contain overlapping tokens and make it impossible to create a
|
||||
valid BILUO sequence. This is helpful, because it lets you spot potential
|
||||
problems in your data that can lead to inconsistent results later on. But it
|
||||
also means that you need to adjust and clean up the offsets before converting
|
||||
them:
|
||||
|
||||
```diff
|
||||
doc = nlp("I live in Berlin Kreuzberg")
|
||||
- entities = [(10, 26, "LOC"), (10, 16, "GPE"), (17, 26, "LOC")]
|
||||
+ entities = [(10, 16, "GPE"), (17, 26, "LOC")]
|
||||
tags = get_biluo_tags_from_offsets(doc, entities)
|
||||
```
|
||||
|
||||
#### Serbian language data
|
||||
|
||||
If you've been working with `Serbian` (introduced in v2.1.8), you'll need to
|
||||
change the language code from `rs` to the correct `sr`:
|
||||
|
||||
```diff
|
||||
- from spacy.lang.rs import Serbian
|
||||
+ from spacy.lang.sr import Serbian
|
||||
```
|
||||
@@ -0,0 +1,344 @@
|
||||
---
|
||||
title: What's New in v2.3
|
||||
teaser: New features, backwards incompatibilities and migration guide
|
||||
menu:
|
||||
- ['New Features', 'features']
|
||||
- ['Backwards Incompatibilities', 'incompat']
|
||||
- ['Migrating from v2.2', 'migrating']
|
||||
---
|
||||
|
||||
## New Features {id="features",hidden="true"}
|
||||
|
||||
spaCy v2.3 features new pretrained models for five languages, word vectors for
|
||||
all language models, and decreased model size and loading times for models with
|
||||
vectors. We've added pretrained models for **Chinese, Danish, Japanese, Polish
|
||||
and Romanian** and updated the training data and vectors for most languages.
|
||||
Model packages with vectors are about **2×** smaller on disk and load
|
||||
**2-4×** faster. For the full changelog, see the
|
||||
[release notes on GitHub](https://github.com/explosion/spaCy/releases/tag/v2.3.0).
|
||||
For more details and a behind-the-scenes look at the new release,
|
||||
[see our blog post](https://explosion.ai/blog/spacy-v2-3).
|
||||
|
||||
### Expanded model families with vectors {id="models"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```bash
|
||||
> python -m spacy download da_core_news_sm
|
||||
> python -m spacy download ja_core_news_sm
|
||||
> python -m spacy download pl_core_news_sm
|
||||
> python -m spacy download ro_core_news_sm
|
||||
> python -m spacy download zh_core_web_sm
|
||||
> ```
|
||||
|
||||
With new model families for Chinese, Danish, Polish, Romanian and Chinese plus
|
||||
`md` and `lg` models with word vectors for all languages, this release provides
|
||||
a total of 46 model packages. For models trained using
|
||||
[Universal Dependencies](https://universaldependencies.org) corpora, the
|
||||
training data has been updated to UD v2.5 (v2.6 for Japanese, v2.3 for Polish)
|
||||
and Dutch has been extended to include both UD Dutch Alpino and LassySmall.
|
||||
|
||||
<Infobox>
|
||||
|
||||
**Models:** [Models directory](/models) **Benchmarks: **
|
||||
[Release notes](https://github.com/explosion/spaCy/releases/tag/v2.3.0)
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Chinese {id="chinese"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> from spacy.lang.zh import Chinese
|
||||
>
|
||||
> # Load with "default" model provided by pkuseg
|
||||
> cfg = {"pkuseg_model": "default", "require_pkuseg": True}
|
||||
> nlp = Chinese(meta={"tokenizer": {"config": cfg}})
|
||||
>
|
||||
> # Append words to user dict
|
||||
> nlp.tokenizer.pkuseg_update_user_dict(["中国", "ABC"])
|
||||
> ```
|
||||
|
||||
This release adds support for
|
||||
[`pkuseg`](https://github.com/lancopku/pkuseg-python) for word segmentation and
|
||||
the new Chinese models ship with a custom pkuseg model trained on OntoNotes. The
|
||||
Chinese tokenizer can be initialized with both `pkuseg` and custom models and
|
||||
the `pkuseg` user dictionary is easy to customize. Note that
|
||||
[`pkuseg`](https://github.com/lancopku/pkuseg-python) doesn't yet ship with
|
||||
pre-compiled wheels for Python 3.8. See the
|
||||
[usage documentation](/usage/models#chinese) for details on how to install it on
|
||||
Python 3.8.
|
||||
|
||||
<Infobox>
|
||||
|
||||
**Models:** [Chinese models](/models/zh) **Usage: **
|
||||
[Chinese tokenizer usage](/usage/models#chinese)
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Japanese {id="japanese"}
|
||||
|
||||
The updated Japanese language class switches to
|
||||
[`SudachiPy`](https://github.com/WorksApplications/SudachiPy) for word
|
||||
segmentation and part-of-speech tagging. Using `SudachiPy` greatly simplifies
|
||||
installing spaCy for Japanese, which is now possible with a single command:
|
||||
`pip install spacy[ja]`.
|
||||
|
||||
<Infobox>
|
||||
|
||||
**Models:** [Japanese models](/models/ja) **Usage:**
|
||||
[Japanese tokenizer usage](/usage/models#japanese)
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Small CLI updates
|
||||
|
||||
- [`spacy debug-data`](/api/cli#debug-data) provides the coverage of the vectors
|
||||
in a base model with `spacy debug-data lang train dev -b base_model`
|
||||
- [`spacy evaluate`](/api/cli#evaluate) supports `blank:lg` (e.g.
|
||||
`spacy evaluate blank:en dev.json`) to evaluate the tokenization accuracy
|
||||
without loading a model
|
||||
- [`spacy train`](/api/cli#train) on GPU restricts the CPU timing evaluation to
|
||||
the first iteration
|
||||
|
||||
## Backwards incompatibilities {id="incompat"}
|
||||
|
||||
<Infobox title="Important note on models" variant="warning">
|
||||
|
||||
If you've been training **your own models**, you'll need to **retrain** them
|
||||
with the new version. Also don't forget to upgrade all models to the latest
|
||||
versions. Models for earlier v2 releases (v2.0, v2.1, v2.2) aren't compatible
|
||||
with models for v2.3. To check if all of your models are up to date, you can run
|
||||
the [`spacy validate`](/api/cli#validate) command.
|
||||
|
||||
</Infobox>
|
||||
|
||||
> #### Install with lookups data
|
||||
>
|
||||
> ```bash
|
||||
> $ pip install spacy[lookups]
|
||||
> ```
|
||||
>
|
||||
> You can also install
|
||||
> [`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data)
|
||||
> directly.
|
||||
|
||||
- If you're training new models, you'll want to install the package
|
||||
[`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data), which
|
||||
now includes both the lemmatization tables (as in v2.2) and the normalization
|
||||
tables (new in v2.3). If you're using pretrained models, **nothing changes**,
|
||||
because the relevant tables are included in the model packages.
|
||||
- Due to the updated Universal Dependencies training data, the fine-grained
|
||||
part-of-speech tags will change for many provided language models. The
|
||||
coarse-grained part-of-speech tagset remains the same, but the mapping from
|
||||
particular fine-grained to coarse-grained tags may show minor differences.
|
||||
- For French, Italian, Portuguese and Spanish, the fine-grained part-of-speech
|
||||
tagsets contain new merged tags related to contracted forms, such as `ADP_DET`
|
||||
for French `"au"`, which maps to UPOS `ADP` based on the head `"à"`. This
|
||||
increases the accuracy of the models by improving the alignment between
|
||||
spaCy's tokenization and Universal Dependencies multi-word tokens used for
|
||||
contractions.
|
||||
|
||||
### Migrating from spaCy 2.2 {id="migrating"}
|
||||
|
||||
#### Tokenizer settings
|
||||
|
||||
In spaCy v2.2.2-v2.2.4, there was a change to the precedence of `token_match`
|
||||
that gave prefixes and suffixes priority over `token_match`, which caused
|
||||
problems for many custom tokenizer configurations. This has been reverted in
|
||||
v2.3 so that `token_match` has priority over prefixes and suffixes as in v2.2.1
|
||||
and earlier versions.
|
||||
|
||||
A new tokenizer setting `url_match` has been introduced in v2.3.0 to handle
|
||||
cases like URLs where the tokenizer should remove prefixes and suffixes (e.g., a
|
||||
comma at the end of a URL) before applying the match. See the full
|
||||
[tokenizer documentation](/usage/linguistic-features#tokenization) and try out
|
||||
[`nlp.tokenizer.explain()`](/usage/linguistic-features#tokenizer-debug) when
|
||||
debugging your tokenizer configuration.
|
||||
|
||||
#### Warnings configuration
|
||||
|
||||
spaCy's custom warnings have been replaced with native Python
|
||||
[`warnings`](https://docs.python.org/3/library/warnings.html). Instead of
|
||||
setting `SPACY_WARNING_IGNORE`, use the
|
||||
[`warnings` filters](https://docs.python.org/3/library/warnings.html#the-warnings-filter)
|
||||
to manage warnings.
|
||||
|
||||
```diff
|
||||
import spacy
|
||||
+ import warnings
|
||||
|
||||
- spacy.errors.SPACY_WARNING_IGNORE.append('W007')
|
||||
+ warnings.filterwarnings("ignore", message=r"\\[W007\\]", category=UserWarning)
|
||||
```
|
||||
|
||||
#### Normalization tables
|
||||
|
||||
The normalization tables have moved from the language data in
|
||||
[`spacy/lang`](https://github.com/explosion/spacy/tree/v2.x/spacy/lang) to the
|
||||
package [`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data).
|
||||
If you're adding data for a new language, the normalization table should be
|
||||
added to `spacy-lookups-data`. See
|
||||
[adding norm exceptions](/usage/adding-languages#norm-exceptions).
|
||||
|
||||
#### No preloaded vocab for models with vectors
|
||||
|
||||
To reduce the initial loading time, the lexemes in `nlp.vocab` are no longer
|
||||
loaded on initialization for models with vectors. As you process texts, the
|
||||
lexemes will be added to the vocab automatically, just as in small models
|
||||
without vectors.
|
||||
|
||||
To see the number of unique vectors and number of words with vectors, see
|
||||
`nlp.meta['vectors']`, for example for `en_core_web_md` there are `20000` unique
|
||||
vectors and `684830` words with vectors:
|
||||
|
||||
```python
|
||||
{
|
||||
'width': 300,
|
||||
'vectors': 20000,
|
||||
'keys': 684830,
|
||||
'name': 'en_core_web_md.vectors'
|
||||
}
|
||||
```
|
||||
|
||||
If required, for instance if you are working directly with word vectors rather
|
||||
than processing texts, you can load all lexemes for words with vectors at once:
|
||||
|
||||
```python
|
||||
for orth in nlp.vocab.vectors:
|
||||
_ = nlp.vocab[orth]
|
||||
```
|
||||
|
||||
If your workflow previously iterated over `nlp.vocab`, a similar alternative is
|
||||
to iterate over words with vectors instead:
|
||||
|
||||
```diff
|
||||
- lexemes = [w for w in nlp.vocab]
|
||||
+ lexemes = [nlp.vocab[orth] for orth in nlp.vocab.vectors]
|
||||
```
|
||||
|
||||
Be aware that the set of preloaded lexemes in a v2.2 model is not equivalent to
|
||||
the set of words with vectors. For English, v2.2 `md/lg` models have 1.3M
|
||||
provided lexemes but only 685K words with vectors. The vectors have been updated
|
||||
for most languages in v2.2, but the English models contain the same vectors for
|
||||
both v2.2 and v2.3.
|
||||
|
||||
#### Lexeme.is_oov and Token.is_oov
|
||||
|
||||
<Infobox title="Important note" variant="warning">
|
||||
|
||||
Due to a bug, the values for `is_oov` are reversed in v2.3.0, but this will be
|
||||
fixed in the next patch release v2.3.1.
|
||||
|
||||
</Infobox>
|
||||
|
||||
In v2.3, `Lexeme.is_oov` and `Token.is_oov` are `True` if the lexeme does not
|
||||
have a word vector. This is equivalent to `token.orth not in nlp.vocab.vectors`.
|
||||
|
||||
Previously in v2.2, `is_oov` corresponded to whether a lexeme had stored
|
||||
probability and cluster features. The probability and cluster features are no
|
||||
longer included in the provided medium and large models (see the next section).
|
||||
|
||||
#### Probability and cluster features
|
||||
|
||||
> #### Load and save extra prob lookups table
|
||||
>
|
||||
> ```python
|
||||
> from spacy.lang.en import English
|
||||
> nlp = English()
|
||||
> doc = nlp("the")
|
||||
> print(doc[0].prob) # lazily loads extra prob table
|
||||
> nlp.to_disk("/path/to/model") # includes prob table
|
||||
> ```
|
||||
|
||||
The `Token.prob` and `Token.cluster` features, which are no longer used by the
|
||||
core pipeline components as of spaCy v2, are no longer provided in the
|
||||
pretrained models to reduce the model size. To keep these features available for
|
||||
users relying on them, the `prob` and `cluster` features for the most frequent
|
||||
1M tokens have been moved to
|
||||
[`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data) as
|
||||
`extra` features for the relevant languages (English, German, Greek and
|
||||
Spanish).
|
||||
|
||||
The extra tables are loaded lazily, so if you have `spacy-lookups-data`
|
||||
installed and your code accesses `Token.prob`, the full table is loaded into the
|
||||
model vocab, which will take a few seconds on initial loading. When you save
|
||||
this model after loading the `prob` table, the full `prob` table will be saved
|
||||
as part of the model vocab.
|
||||
|
||||
To load the probability table into a provided model, first make sure you have
|
||||
`spacy-lookups-data` installed. To load the table, remove the empty provided
|
||||
`lexeme_prob` table and then access `Lexeme.prob` for any word to load the table
|
||||
from `spacy-lookups-data`:
|
||||
|
||||
```diff
|
||||
+ # prerequisite: pip install spacy-lookups-data
|
||||
import spacy
|
||||
|
||||
nlp = spacy.load("en_core_web_md")
|
||||
|
||||
# remove the empty placeholder prob table
|
||||
+ if nlp.vocab.lookups_extra.has_table("lexeme_prob"):
|
||||
+ nlp.vocab.lookups_extra.remove_table("lexeme_prob")
|
||||
|
||||
# access any `.prob` to load the full table into the model
|
||||
assert nlp.vocab["a"].prob == -3.9297883511
|
||||
|
||||
# if desired, save this model with the probability table included
|
||||
nlp.to_disk("/path/to/model")
|
||||
```
|
||||
|
||||
If you'd like to include custom `cluster`, `prob`, or `sentiment` tables as part
|
||||
of a new model, add the data to
|
||||
[`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data) under
|
||||
the entry point `lg_extra`, e.g. `en_extra` for English. Alternatively, you can
|
||||
initialize your [`Vocab`](/api/vocab) with the `lookups_extra` argument with a
|
||||
[`Lookups`](/api/lookups) object that includes the tables `lexeme_cluster`,
|
||||
`lexeme_prob`, `lexeme_sentiment` or `lexeme_settings`. `lexeme_settings` is
|
||||
currently only used to provide a custom `oov_prob`. See examples in the
|
||||
[`data` directory](https://github.com/explosion/spacy-lookups-data/tree/master/spacy_lookups_data/data)
|
||||
in `spacy-lookups-data`.
|
||||
|
||||
#### Initializing new models without extra lookups tables
|
||||
|
||||
When you initialize a new model with [`spacy init-model`](/api/cli#init-model),
|
||||
the `prob` table from `spacy-lookups-data` may be loaded as part of the
|
||||
initialization. If you'd like to omit this extra data as in spaCy's provided
|
||||
v2.3 models, use the new flag `--omit-extra-lookups`.
|
||||
|
||||
#### Tag maps in provided models vs. blank models
|
||||
|
||||
The tag maps in the provided models may differ from the tag maps in the spaCy
|
||||
library. You can access the tag map in a loaded model under
|
||||
`nlp.vocab.morphology.tag_map`.
|
||||
|
||||
The tag map from `spacy.lang.lg.tag_map` is still used when a blank model is
|
||||
initialized. If you want to provide an alternate tag map, update
|
||||
`nlp.vocab.morphology.tag_map` after initializing the model or if you're using
|
||||
the [train CLI](/api/cli#train), you can use the new `--tag-map-path` option to
|
||||
provide in the tag map as a JSON dict.
|
||||
|
||||
If you want to export a tag map from a provided model for use with the train
|
||||
CLI, you can save it as a JSON dict. To only use string keys as required by JSON
|
||||
and to make it easier to read and edit, any internal integer IDs need to be
|
||||
converted back to strings:
|
||||
|
||||
```python
|
||||
import spacy
|
||||
import srsly
|
||||
|
||||
nlp = spacy.load("en_core_web_sm")
|
||||
tag_map = {}
|
||||
|
||||
# convert any integer IDs to strings for JSON
|
||||
for tag, morph in nlp.vocab.morphology.tag_map.items():
|
||||
tag_map[tag] = {}
|
||||
for feat, val in morph.items():
|
||||
feat = nlp.vocab.strings.as_string(feat)
|
||||
if not isinstance(val, bool):
|
||||
val = nlp.vocab.strings.as_string(val)
|
||||
tag_map[tag][feat] = val
|
||||
|
||||
srsly.write_json("tag_map.json", tag_map)
|
||||
```
|
||||
@@ -0,0 +1,613 @@
|
||||
---
|
||||
title: What's New in v2.0
|
||||
teaser: New features, backwards incompatibilities and migration guide
|
||||
menu:
|
||||
- ['Summary', 'summary']
|
||||
- ['New Features', 'features']
|
||||
- ['Backwards Incompatibilities', 'incompat']
|
||||
- ['Migrating from v1.x', 'migrating']
|
||||
---
|
||||
|
||||
We're very excited to finally introduce spaCy v2.0! On this page, you'll find a
|
||||
summary of the new features, information on the backwards incompatibilities,
|
||||
including a handy overview of what's been renamed or deprecated. To help you
|
||||
make the most of v2.0, we also **re-wrote almost all of the usage guides and API
|
||||
docs**, and added more [real-world examples](/usage/examples). If you're new to
|
||||
spaCy, or just want to brush up on some NLP basics and the details of the
|
||||
library, check out the [spaCy 101 guide](/usage/spacy-101) that explains the
|
||||
most important concepts with examples and illustrations.
|
||||
|
||||
## Summary {id="summary"}
|
||||
|
||||
<Grid cols={2}>
|
||||
|
||||
<div>
|
||||
|
||||
This release features entirely new **deep learning-powered models** for spaCy's
|
||||
tagger, parser and entity recognizer. The new models are **10× smaller**, **20%
|
||||
more accurate** and **even cheaper to run** than the previous generation.
|
||||
|
||||
We've also made several usability improvements that are particularly helpful for
|
||||
**production deployments**. spaCy v2 now fully supports the Pickle protocol,
|
||||
making it easy to use spaCy with [Apache Spark](https://spark.apache.org/). The
|
||||
string-to-integer mapping is **no longer stateful**, making it easy to reconcile
|
||||
annotations made in different processes. Models are smaller and use less memory,
|
||||
and the APIs for serialization are now much more consistent. Custom pipeline
|
||||
components let you modify the `Doc` at any stage in the pipeline. You can now
|
||||
also add your own custom attributes, properties and methods to the `Doc`,
|
||||
`Token` and `Span`.
|
||||
|
||||
</div>
|
||||
|
||||
<Infobox title="Table of Contents" id="toc">
|
||||
|
||||
- [Summary](#summary)
|
||||
- [New features](#features)
|
||||
- [Neural network models](#features-models)
|
||||
- [Improved processing pipelines](#features-pipelines)
|
||||
- [Text classification](#features-text-classification)
|
||||
- [Hash values as IDs](#features-hash-ids)
|
||||
- [Improved word vectors support](#features-vectors)
|
||||
- [Saving, loading and serialization](#features-serializer)
|
||||
- [displaCy visualizer](#features-displacy)
|
||||
- [Language data and lazy loading](#features-language)
|
||||
- [Revised matcher API and phrase matcher](#features-matcher)
|
||||
- [Backwards incompatibilities](#incompat)
|
||||
- [Migrating from spaCy v1.x](#migrating)
|
||||
|
||||
</Infobox>
|
||||
|
||||
</Grid>
|
||||
|
||||
The main usability improvements you'll notice in spaCy v2.0 are around
|
||||
**defining, training and loading your own models** and components. The new
|
||||
neural network models make it much easier to train a model from scratch, or
|
||||
update an existing model with a few examples. In v1.x, the statistical models
|
||||
depended on the state of the `Vocab`. If you taught the model a new word, you
|
||||
would have to save and load a lot of data — otherwise the model wouldn't
|
||||
correctly recall the features of your new example. That's no longer the case.
|
||||
|
||||
Due to some clever use of hashing, the statistical models **never change size**,
|
||||
even as they learn new vocabulary items. The whole pipeline is also now fully
|
||||
differentiable. Even if you don't have explicitly annotated data, you can update
|
||||
spaCy using all the **latest deep learning tricks** like adversarial training,
|
||||
noise contrastive estimation or reinforcement learning.
|
||||
|
||||
## New features {id="features"}
|
||||
|
||||
This section contains an overview of the most important **new features and
|
||||
improvements**. The [API docs](/api) include additional deprecation notes.
|
||||
|
||||
### Convolutional neural network models {id="features-models"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```bash
|
||||
> python -m spacy download en_core_web_sm
|
||||
> python -m spacy download de_core_news_sm
|
||||
> python -m spacy download xx_ent_wiki_sm
|
||||
> ```
|
||||
|
||||
spaCy v2.0 features new neural models for tagging, parsing and entity
|
||||
recognition. The models have been designed and implemented from scratch
|
||||
specifically for spaCy, to give you an unmatched balance of speed, size and
|
||||
accuracy. The new models are **10× smaller**, **20% more accurate**, and **even
|
||||
cheaper to run** than the previous generation.
|
||||
|
||||
spaCy v2.0's new neural network models bring significant improvements in
|
||||
accuracy, especially for English Named Entity Recognition. The new
|
||||
[`en_core_web_lg`](/models/en#en_core_web_lg) model makes about **25% fewer
|
||||
mistakes** than the corresponding v1.x model and is within **1% of the current
|
||||
state-of-the-art**
|
||||
([Strubell et al., 2017](https://arxiv.org/pdf/1702.02098.pdf)). The v2.0 models
|
||||
are also cheaper to run at scale, as they require **under 1 GB of memory** per
|
||||
process.
|
||||
|
||||
<Infobox>
|
||||
|
||||
**Usage:** [Models directory](/models)
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Improved processing pipelines {id="features-pipelines"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> # Set custom attributes
|
||||
> Doc.set_extension("my_attr", default=False)
|
||||
> Token.set_extension("my_attr", getter=my_token_getter)
|
||||
> assert doc._.my_attr, token._.my_attr
|
||||
>
|
||||
> # Add components to the pipeline
|
||||
> my_component = lambda doc: doc
|
||||
> nlp.add_pipe(my_component)
|
||||
> ```
|
||||
|
||||
It's now much easier to **customize the pipeline** with your own components:
|
||||
functions that receive a `Doc` object, modify and return it. Extensions let you
|
||||
write any **attributes, properties and methods** to the `Doc`, `Token` and
|
||||
`Span`. You can add data, implement new features, integrate other libraries with
|
||||
spaCy or plug in your own machine learning models.
|
||||
|
||||
|
||||
|
||||
<Infobox>
|
||||
|
||||
**API:** [`Language`](/api/language),
|
||||
[`Doc.set_extension`](/api/doc#set_extension),
|
||||
[`Span.set_extension`](/api/span#set_extension),
|
||||
[`Token.set_extension`](/api/token#set_extension) **Usage:**
|
||||
[Processing pipelines](/usage/processing-pipelines) **Code:**
|
||||
[Pipeline examples](/usage/examples#section-pipeline)
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Text classification {id="features-text-classification"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> textcat = nlp.create_pipe("textcat")
|
||||
> nlp.add_pipe(textcat, last=True)
|
||||
> nlp.begin_training()
|
||||
> for itn in range(100):
|
||||
> for doc, gold in train_data:
|
||||
> nlp.update([doc], [gold])
|
||||
> doc = nlp("This is a text.")
|
||||
> print(doc.cats)
|
||||
> ```
|
||||
|
||||
spaCy v2.0 lets you add text categorization models to spaCy pipelines. The model
|
||||
supports classification with multiple, non-mutually exclusive labels – so
|
||||
multiple labels can apply at once. You can change the model architecture rather
|
||||
easily, but by default, the `TextCategorizer` class uses a convolutional neural
|
||||
network to assign position-sensitive vectors to each word in the document.
|
||||
|
||||
<Infobox>
|
||||
|
||||
**API:** [`TextCategorizer`](/api/textcategorizer),
|
||||
[`Doc.cats`](/api/doc#attributes), `GoldParse.cats` **Usage:**
|
||||
[Training a text classification model](/usage/training#textcat)
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Hash values instead of integer IDs {id="features-hash-ids"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> doc = nlp("I love coffee")
|
||||
> assert doc.vocab.strings["coffee"] == 3197928453018144401
|
||||
> assert doc.vocab.strings[3197928453018144401] == "coffee"
|
||||
>
|
||||
> beer_hash = doc.vocab.strings.add("beer")
|
||||
> assert doc.vocab.strings["beer"] == beer_hash
|
||||
> assert doc.vocab.strings[beer_hash] == "beer"
|
||||
> ```
|
||||
|
||||
The [`StringStore`](/api/stringstore) now resolves all strings to hash values
|
||||
instead of integer IDs. This means that the string-to-int mapping **no longer
|
||||
depends on the vocabulary state**, making a lot of workflows much simpler,
|
||||
especially during training. Unlike integer IDs in spaCy v1.x, hash values will
|
||||
**always match** – even across models. Strings can now be added explicitly using
|
||||
the new [`Stringstore.add`](/api/stringstore#add) method. A token's hash is
|
||||
available via `token.orth`.
|
||||
|
||||
<Infobox>
|
||||
|
||||
**API:** [`StringStore`](/api/stringstore) **Usage:**
|
||||
[Vocab, hashes and lexemes 101](/usage/spacy-101#vocab)
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Improved word vectors support {id="features-vectors"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> for word, vector in vector_data:
|
||||
> nlp.vocab.set_vector(word, vector)
|
||||
> nlp.vocab.vectors.from_glove("/path/to/vectors")
|
||||
> # Keep 10000 unique vectors and remap the rest
|
||||
> nlp.vocab.prune_vectors(10000)
|
||||
> nlp.to_disk("/model")
|
||||
> ```
|
||||
|
||||
The new [`Vectors`](/api/vectors) class helps the `Vocab` manage the vectors
|
||||
assigned to strings, and lets you assign vectors individually, or
|
||||
[load in GloVe vectors](/usage/linguistic-features#adding-vectors) from a
|
||||
directory. To help you strike a good balance between coverage and memory usage,
|
||||
the `Vectors` class lets you map **multiple keys** to the **same row** of the
|
||||
table. If you're using the [`spacy init-model`](/api/cli#init-model) command to
|
||||
create a vocabulary, pruning the vectors will be taken care of automatically if
|
||||
you set the `--prune-vectors` flag. Otherwise, you can use the new
|
||||
[`Vocab.prune_vectors`](/api/vocab#prune_vectors).
|
||||
|
||||
<Infobox>
|
||||
|
||||
**API:** [`Vectors`](/api/vectors), [`Vocab`](/api/vocab) **Usage:**
|
||||
[Word vectors and semantic similarity](/usage/vectors-similarity)
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Saving, loading and serialization {id="features-serializer"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> nlp = spacy.load("en") # shortcut link
|
||||
> nlp = spacy.load("en_core_web_sm") # package
|
||||
> nlp = spacy.load("/path/to/en") # unicode path
|
||||
> nlp = spacy.load(Path("/path/to/en")) # pathlib Path
|
||||
>
|
||||
> nlp.to_disk("/path/to/nlp")
|
||||
> nlp = English().from_disk("/path/to/nlp")
|
||||
> ```
|
||||
|
||||
spaCy's serialization API has been made consistent across classes and objects.
|
||||
All container classes, i.e. `Language`, `Doc`, `Vocab` and `StringStore` now
|
||||
have a `to_bytes()`, `from_bytes()`, `to_disk()` and `from_disk()` method that
|
||||
supports the Pickle protocol.
|
||||
|
||||
The improved `spacy.load` makes loading models easier and more transparent. You
|
||||
can load a model by supplying its shortcut link, the name of an installed
|
||||
[model package](/models) or a path. The `Language` class to initialize will be
|
||||
determined based on the model's settings. For a blank language, you can import
|
||||
the class directly, e.g. `from spacy.lang.en import English` or use
|
||||
[`spacy.blank()`](/api/top-level#spacy.blank).
|
||||
|
||||
<Infobox>
|
||||
|
||||
**API:** [`spacy.load`](/api/top-level#spacy.load),
|
||||
[`Language.to_disk`](/api/language#to_disk) **Usage:**
|
||||
[Models](/usage/models#usage),
|
||||
[Saving and loading](/usage/saving-loading#models)
|
||||
|
||||
</Infobox>
|
||||
|
||||
### displaCy visualizer with Jupyter support {id="features-displacy"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> from spacy import displacy
|
||||
> doc = nlp("This is a sentence about Facebook.")
|
||||
> displacy.serve(doc, style="dep") # run the web server
|
||||
> html = displacy.render(doc, style="ent") # generate HTML
|
||||
> ```
|
||||
|
||||
Our popular dependency and named entity visualizers are now an official part of
|
||||
the spaCy library. displaCy can run a simple web server, or generate raw HTML
|
||||
markup or SVG files to be exported. You can pass in one or more docs, and
|
||||
customize the style. displaCy also auto-detects whether you're running
|
||||
[Jupyter](https://jupyter.org) and will render the visualizations in your
|
||||
notebook.
|
||||
|
||||
<Infobox>
|
||||
|
||||
**API:** [`displacy`](/api/top-level#displacy) **Usage:**
|
||||
[Visualizing spaCy](/usage/visualizers)
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Improved language data and lazy loading {id="features-language"}
|
||||
|
||||
Language-specific data now lives in its own submodule, `spacy.lang`. Languages
|
||||
are lazy-loaded, i.e. only loaded when you import a `Language` class, or load a
|
||||
model that initializes one. This allows languages to contain more custom data,
|
||||
e.g. lemmatizer lookup tables, or complex regular expressions. The language data
|
||||
has also been tidied up and simplified. spaCy now also supports simple
|
||||
lookup-based lemmatization – and **many new languages**!
|
||||
|
||||
<Infobox>
|
||||
|
||||
**API:** [`Language`](/api/language) **Code:**
|
||||
[`spacy/lang`](https://github.com/explosion/spacy/tree/v2.x/spacy/lang)
|
||||
**Usage:** [Adding languages](/usage/adding-languages)
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Revised matcher API and phrase matcher {id="features-matcher"}
|
||||
|
||||
> #### Example
|
||||
>
|
||||
> ```python
|
||||
> from spacy.matcher import Matcher, PhraseMatcher
|
||||
>
|
||||
> matcher = Matcher(nlp.vocab)
|
||||
> matcher.add('HEARTS', None, [{"ORTH": "❤️", "OP": '+'}])
|
||||
>
|
||||
> phrasematcher = PhraseMatcher(nlp.vocab)
|
||||
> phrasematcher.add("OBAMA", None, nlp("Barack Obama"))
|
||||
> ```
|
||||
|
||||
Patterns can now be added to the matcher by calling
|
||||
[`matcher.add()`](/api/matcher#add) with a match ID, an optional callback
|
||||
function to be invoked on each match, and one or more patterns. This allows you
|
||||
to write powerful, pattern-specific logic using only one matcher. For example,
|
||||
you might only want to merge some entity types, and set custom flags for other
|
||||
matched patterns. The new [`PhraseMatcher`](/api/phrasematcher) lets you
|
||||
efficiently match very large terminology lists using `Doc` objects as match
|
||||
patterns.
|
||||
|
||||
<Infobox>
|
||||
|
||||
**API:** [`Matcher`](/api/matcher), [`PhraseMatcher`](/api/phrasematcher)
|
||||
**Usage:** [Rule-based matching](/usage/rule-based-matching)
|
||||
|
||||
</Infobox>
|
||||
|
||||
## Backwards incompatibilities {id="incompat"}
|
||||
|
||||
The following modules, classes and methods have changed between v1.x and v2.0.
|
||||
|
||||
| Old | New |
|
||||
| ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `spacy.download.en`, `spacy.download.de` | [`cli.download`](/api/cli#download) |
|
||||
| `spacy.en` etc. | `spacy.lang.en` etc. |
|
||||
| `spacy.en.word_sets` | `spacy.lang.en.stop_words` |
|
||||
| `spacy.orth` | `spacy.lang.xx.lex_attrs` |
|
||||
| `spacy.syntax.iterators` | `spacy.lang.xx.syntax_iterators` |
|
||||
| `spacy.tagger.Tagger` | `spacy.pipeline.Tagger` |
|
||||
| `spacy.cli.model` | [`spacy.cli.vocab`](/api/cli#vocab) |
|
||||
| `Language.save_to_directory` | [`Language.to_disk`](/api/language#to_disk) |
|
||||
| `Language.end_training` | [`Language.begin_training`](/api/language#begin_training) |
|
||||
| `Language.create_make_doc` | [`Language.tokenizer`](/api/language#attributes) |
|
||||
| `Vocab.resize_vectors` | [`Vectors.resize`](/api/vectors#resize) |
|
||||
| `Vocab.load` `Vocab.load_lexemes` | [`Vocab.from_disk`](/api/vocab#from_disk) [`Vocab.from_bytes`](/api/vocab#from_bytes) |
|
||||
| `Vocab.dump` | [`Vocab.to_disk`](/api/vocab#to_disk) [`Vocab.to_bytes`](/api/vocab#to_bytes) |
|
||||
| `Vocab.load_vectors` `Vocab.load_vectors_from_bin_loc` | [`Vectors.from_disk`](/api/vectors#from_disk) [`Vectors.from_bytes`](/api/vectors#from_bytes) [`Vectors.from_glove`](/api/vectors#from_glove) |
|
||||
| `Vocab.dump_vectors` | [`Vectors.to_disk`](/api/vectors#to_disk) [`Vectors.to_bytes`](/api/vectors#to_bytes) |
|
||||
| `StringStore.load` | [`StringStore.from_disk`](/api/stringstore#from_disk) [`StringStore.from_bytes`](/api/stringstore#from_bytes) |
|
||||
| `StringStore.dump` | [`StringStore.to_disk`](/api/stringstore#to_disk) [`StringStore.to_bytes`](/api/stringstore#to_bytes) |
|
||||
| `Tokenizer.load` | [`Tokenizer.from_disk`](/api/tokenizer#from_disk) [`Tokenizer.from_bytes`](/api/tokenizer#from_bytes) |
|
||||
| `Tagger.load` | [`Tagger.from_disk`](/api/tagger#from_disk) [`Tagger.from_bytes`](/api/tagger#from_bytes) |
|
||||
| `Tagger.tag_names` | `Tagger.labels` |
|
||||
| `DependencyParser.load` | [`DependencyParser.from_disk`](/api/dependencyparser#from_disk) [`DependencyParser.from_bytes`](/api/dependencyparser#from_bytes) |
|
||||
| `EntityRecognizer.load` | [`EntityRecognizer.from_disk`](/api/entityrecognizer#from_disk) [`EntityRecognizer.from_bytes`](/api/entityrecognizer#from_bytes) |
|
||||
| `Matcher.load` | - |
|
||||
| `Matcher.add_pattern` `Matcher.add_entity` | [`Matcher.add`](/api/matcher#add) [`PhraseMatcher.add`](/api/phrasematcher#add) |
|
||||
| `Matcher.get_entity` | [`Matcher.get`](/api/matcher#get) |
|
||||
| `Matcher.has_entity` | [`Matcher.has_key`](/api/matcher#has_key) |
|
||||
| `Doc.read_bytes` | [`Doc.to_bytes`](/api/doc#to_bytes) [`Doc.from_bytes`](/api/doc#from_bytes) [`Doc.to_disk`](/api/doc#to_disk) [`Doc.from_disk`](/api/doc#from_disk) |
|
||||
| `Token.is_ancestor_of` | [`Token.is_ancestor`](/api/token#is_ancestor) |
|
||||
|
||||
### Deprecated {id="deprecated"}
|
||||
|
||||
The following methods are deprecated. They can still be used, but should be
|
||||
replaced.
|
||||
|
||||
| Old | New |
|
||||
| ---------------------------- | ----------------------------------------------- |
|
||||
| `Tokenizer.tokens_from_list` | [`Doc`](/api/doc) |
|
||||
| `Span.sent_start` | [`Span.is_sent_start`](/api/span#is_sent_start) |
|
||||
|
||||
## Migrating from spaCy 1.x {id="migrating"}
|
||||
|
||||
Because we'e made so many architectural changes to the library, we've tried to
|
||||
**keep breaking changes to a minimum**. A lot of projects follow the philosophy
|
||||
that if you're going to break anything, you may as well break everything. We
|
||||
think migration is easier if there's a logic to what has changed. We've
|
||||
therefore followed a policy of avoiding breaking changes to the `Doc`, `Span`
|
||||
and `Token` objects. This way, you can focus on only migrating the code that
|
||||
does training, loading and serialization — in other words, code that works with
|
||||
the `nlp` object directly. Code that uses the annotations should continue to
|
||||
work.
|
||||
|
||||
<Infobox title="Important note" variant="warning">
|
||||
|
||||
If you've trained your own models, keep in mind that your train and runtime
|
||||
inputs must match. This means you'll have to **retrain your models** with spaCy
|
||||
v2.0.
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Document processing {id="migrating-document-processing"}
|
||||
|
||||
The [`Language.pipe`](/api/language#pipe) method allows spaCy to batch
|
||||
documents, which brings a **significant performance advantage** in v2.0. The new
|
||||
neural networks introduce some overhead per batch, so if you're processing a
|
||||
number of documents in a row, you should use `nlp.pipe` and process the texts as
|
||||
a stream.
|
||||
|
||||
```diff
|
||||
- docs = (nlp(text) for text in texts)
|
||||
|
||||
+ docs = nlp.pipe(texts)
|
||||
```
|
||||
|
||||
To make usage easier, there's now a boolean `as_tuples` keyword argument, that
|
||||
lets you pass in an iterator of `(text, context)` pairs, so you can get back an
|
||||
iterator of `(doc, context)` tuples.
|
||||
|
||||
### Saving, loading and serialization {id="migrating-saving-loading"}
|
||||
|
||||
Double-check all calls to `spacy.load()` and make sure they don't use the `path`
|
||||
keyword argument. If you're only loading in binary data and not a model package
|
||||
that can construct its own `Language` class and pipeline, you should now use the
|
||||
[`Language.from_disk`](/api/language#from_disk) method.
|
||||
|
||||
```diff
|
||||
- nlp = spacy.load("en", path="/model")
|
||||
|
||||
+ nlp = spacy.load("/model")
|
||||
+ nlp = spacy.blank("en").from_disk("/model/data")
|
||||
```
|
||||
|
||||
Review all other code that writes state to disk or bytes. All containers, now
|
||||
share the same, consistent API for saving and loading. Replace saving with
|
||||
`to_disk()` or `to_bytes()`, and loading with `from_disk()` and `from_bytes()`.
|
||||
|
||||
```diff
|
||||
- nlp.save_to_directory("/model")
|
||||
- nlp.vocab.dump("/vocab")
|
||||
|
||||
+ nlp.to_disk("/model")
|
||||
+ nlp.vocab.to_disk("/vocab")
|
||||
```
|
||||
|
||||
If you've trained models with input from v1.x, you'll need to **retrain them**
|
||||
with spaCy v2.0. All previous models will not be compatible with the new
|
||||
version.
|
||||
|
||||
### Processing pipelines and language data {id="migrating-languages"}
|
||||
|
||||
If you're importing language data or `Language` classes, make sure to change
|
||||
your import statements to import from `spacy.lang`. If you've added your own
|
||||
custom language, it needs to be moved to `spacy/lang/xx` and adjusted
|
||||
accordingly.
|
||||
|
||||
```diff
|
||||
- from spacy.en import English
|
||||
|
||||
+ from spacy.lang.en import English
|
||||
```
|
||||
|
||||
If you've been using custom pipeline components, check out the new guide on
|
||||
[processing pipelines](/usage/processing-pipelines). Pipeline components are now
|
||||
`(name, func)` tuples. Appending them to the pipeline still works – but the
|
||||
[`add_pipe`](/api/language#add_pipe) method now makes this much more convenient.
|
||||
Methods for removing, renaming, replacing and retrieving components have been
|
||||
added as well. Components can now be disabled by passing a list of their names
|
||||
to the `disable` keyword argument on load, or by using
|
||||
[`disable_pipes`](/api/language#disable_pipes) as a method or context manager:
|
||||
|
||||
```diff
|
||||
- nlp = spacy.load("en_core_web_sm", tagger=False, entity=False)
|
||||
- doc = nlp("I don't want parsed", parse=False)
|
||||
|
||||
+ nlp = spacy.load("en_core_web_sm", disable=["tagger", "ner"])
|
||||
+ with nlp.disable_pipes("parser"):
|
||||
+ doc = nlp("I don't want parsed")
|
||||
```
|
||||
|
||||
To add spaCy's built-in pipeline components to your pipeline, you can still
|
||||
import and instantiate them directly – but it's more convenient to use the new
|
||||
[`create_pipe`](/api/language#create_pipe) method with the component name, i.e.
|
||||
`'tagger'`, `'parser'`, `'ner'` or `'textcat'`.
|
||||
|
||||
```diff
|
||||
- from spacy.pipeline import Tagger
|
||||
- tagger = Tagger(nlp.vocab)
|
||||
- nlp.pipeline.insert(0, tagger)
|
||||
|
||||
+ tagger = nlp.create_pipe("tagger")
|
||||
+ nlp.add_pipe(tagger, first=True)
|
||||
```
|
||||
|
||||
### Training {id="migrating-training"}
|
||||
|
||||
All built-in pipeline components are now subclasses of [`Pipe`](/api/pipe),
|
||||
fully trainable and serializable, and follow the same API. Instead of updating
|
||||
the model and telling spaCy when to _stop_, you can now explicitly call
|
||||
[`begin_training`](/api/language#begin_training), which returns an optimizer you
|
||||
can pass into the [`update`](/api/language#update) function. While `update`
|
||||
still accepts sequences of `Doc` and `GoldParse` objects, you can now also pass
|
||||
in a list of strings and dictionaries describing the annotations. We call this
|
||||
the ["simple training style"](/usage/training#training-simple-style). This is
|
||||
also the recommended usage, as it removes one layer of abstraction from the
|
||||
training.
|
||||
|
||||
```diff
|
||||
- for itn in range(1000):
|
||||
- for text, entities in train_data:
|
||||
- doc = Doc(text)
|
||||
- gold = GoldParse(doc, entities=entities)
|
||||
- nlp.update(doc, gold)
|
||||
- nlp.end_training()
|
||||
- nlp.save_to_directory("/model")
|
||||
|
||||
+ nlp.begin_training()
|
||||
+ for itn in range(1000):
|
||||
+ for texts, annotations in train_data:
|
||||
+ nlp.update(texts, annotations)
|
||||
+ nlp.to_disk("/model")
|
||||
```
|
||||
|
||||
### Attaching custom data to the Doc {id="migrating-doc"}
|
||||
|
||||
Previously, you had to create a new container in order to attach custom data to
|
||||
a `Doc` object. This often required converting the `Doc` objects to and from
|
||||
arrays. In spaCy v2.0, you can set your own attributes, properties and methods
|
||||
on the `Doc`, `Token` and `Span` via
|
||||
[custom extensions](/usage/processing-pipelines#custom-components-attributes).
|
||||
This means that your application can – and should – only pass around `Doc`
|
||||
objects and refer to them as the single source of truth.
|
||||
|
||||
```diff
|
||||
- doc = nlp("This is a regular doc")
|
||||
- doc_array = doc.to_array(["ORTH", "POS"])
|
||||
- doc_with_meta = {"doc_array": doc_array, "meta": get_doc_meta(doc_array)}
|
||||
|
||||
+ Doc.set_extension("meta", getter=get_doc_meta)
|
||||
+ doc_with_meta = nlp(u'This is a doc with meta data')
|
||||
+ meta = doc._.meta
|
||||
```
|
||||
|
||||
If you wrap your extension attributes in a
|
||||
[custom pipeline component](/usage/processing-pipelines#custom-components), they
|
||||
will be assigned automatically when you call `nlp` on a text. If your
|
||||
application assigns custom data to spaCy's container objects, or includes other
|
||||
utilities that interact with the pipeline, consider moving this logic into its
|
||||
own extension module.
|
||||
|
||||
```diff
|
||||
- doc = nlp("Doc with a standard pipeline")
|
||||
- meta = get_meta(doc)
|
||||
|
||||
+ nlp.add_pipe(meta_component)
|
||||
+ doc = nlp("Doc with a custom pipeline that assigns meta")
|
||||
+ meta = doc._.meta
|
||||
```
|
||||
|
||||
### Strings and hash values {id="migrating-strings"}
|
||||
|
||||
The change from integer IDs to hash values may not actually affect your code
|
||||
very much. However, if you're adding strings to the vocab manually, you now need
|
||||
to call [`StringStore.add`](/api/stringstore#add) explicitly. You can also now
|
||||
be sure that the string-to-hash mapping will always match across vocabularies.
|
||||
|
||||
```diff
|
||||
- nlp.vocab.strings["coffee"] # 3672
|
||||
- other_nlp.vocab.strings["coffee"] # 40259
|
||||
|
||||
+ nlp.vocab.strings.add("coffee")
|
||||
+ nlp.vocab.strings["coffee"] # 3197928453018144401
|
||||
+ other_nlp.vocab.strings["coffee"] # 3197928453018144401
|
||||
```
|
||||
|
||||
### Adding patterns and callbacks to the matcher {id="migrating-matcher"}
|
||||
|
||||
If you're using the matcher, you can now add patterns in one step. This should
|
||||
be easy to update – simply merge the ID, callback and patterns into one call to
|
||||
[`Matcher.add()`](/api/matcher#add). The matcher now also supports string keys,
|
||||
which saves you an extra import. If you've been using **acceptor functions**,
|
||||
you'll need to move this logic into the
|
||||
[`on_match` callbacks](/usage/linguistic-features#on_match). The callback
|
||||
function is invoked on every match and will give you access to the doc, the
|
||||
index of the current match and all total matches. This lets you both accept or
|
||||
reject the match, and define the actions to be triggered.
|
||||
|
||||
```diff
|
||||
- matcher.add_entity("GoogleNow", on_match=merge_phrases)
|
||||
- matcher.add_pattern("GoogleNow", [{ORTH: "Google"}, {ORTH: "Now"}])
|
||||
|
||||
+ matcher.add("GoogleNow", merge_phrases, [{"ORTH": "Google"}, {"ORTH": "Now"}])
|
||||
```
|
||||
|
||||
If you need to match large terminology lists, you can now also use the
|
||||
[`PhraseMatcher`](/api/phrasematcher), which accepts `Doc` objects as match
|
||||
patterns and is more efficient than the regular, rule-based matcher.
|
||||
|
||||
```diff
|
||||
- matcher = Matcher(nlp.vocab)
|
||||
- matcher.add_entity("PRODUCT")
|
||||
- for text in large_terminology_list
|
||||
- matcher.add_pattern("PRODUCT", [{ORTH: text}])
|
||||
|
||||
+ from spacy.matcher import PhraseMatcher
|
||||
+ matcher = PhraseMatcher(nlp.vocab)
|
||||
+ patterns = [nlp.make_doc(text) for text in large_terminology_list]
|
||||
+ matcher.add("PRODUCT", None, *patterns)
|
||||
```
|
||||
@@ -0,0 +1,322 @@
|
||||
---
|
||||
title: What's New in v3.1
|
||||
teaser: New features and how to upgrade
|
||||
menu:
|
||||
- ['New Features', 'features']
|
||||
- ['Upgrading Notes', 'upgrading']
|
||||
---
|
||||
|
||||
## New Features {id="features",hidden="true"}
|
||||
|
||||
It's been great to see the adoption of the new spaCy v3, which introduced
|
||||
[transformer-based](/usage/embeddings-transformers) pipelines, a new
|
||||
[config and training system](/usage/training) for reproducible experiments,
|
||||
[projects](/usage/projects) for end-to-end workflows, and many
|
||||
[other features](/usage/v3). Version 3.1 adds more on top of it, including the
|
||||
ability to use predicted annotations during training, a new `SpanCategorizer`
|
||||
component for predicting arbitrary and potentially overlapping spans, support
|
||||
for partial incorrect annotations in the entity recognizer, new trained
|
||||
pipelines for Catalan and Danish, as well as many bug fixes and improvements.
|
||||
|
||||
### Using predicted annotations during training {id="predicted-annotations-training"}
|
||||
|
||||
By default, components are updated in isolation during training, which means
|
||||
that they don't see the predictions of any earlier components in the pipeline.
|
||||
The new
|
||||
[`[training.annotating_components]`](/usage/training#annotating-components)
|
||||
config setting lets you specify pipeline components that should set annotations
|
||||
on the predicted docs during training. This makes it easy to use the predictions
|
||||
of a previous component in the pipeline as features for a subsequent component,
|
||||
e.g. the dependency labels in the tagger:
|
||||
|
||||
```ini {title="config.cfg (excerpt)",highlight="7,12"}
|
||||
[nlp]
|
||||
pipeline = ["parser", "tagger"]
|
||||
|
||||
[components.tagger.model.tok2vec.embed]
|
||||
@architectures = "spacy.MultiHashEmbed.v1"
|
||||
width = ${components.tagger.model.tok2vec.encode.width}
|
||||
attrs = ["NORM","DEP"]
|
||||
rows = [5000,2500]
|
||||
include_static_vectors = false
|
||||
|
||||
[training]
|
||||
annotating_components = ["parser"]
|
||||
```
|
||||
|
||||
<Project id="pipelines/tagger_parser_predicted_annotations">
|
||||
|
||||
This project shows how to use the `token.dep` attribute predicted by the parser
|
||||
as a feature for a subsequent tagger component in the pipeline.
|
||||
|
||||
</Project>
|
||||
|
||||
### SpanCategorizer for predicting arbitrary and overlapping spans {id="spancategorizer",tag="experimental"}
|
||||
|
||||
A common task in applied NLP is extracting spans of texts from documents,
|
||||
including longer phrases or nested expressions. Named entity recognition isn't
|
||||
the right tool for this problem, since an entity recognizer typically predicts
|
||||
single token-based tags that are very sensitive to boundaries. This is effective
|
||||
for proper nouns and self-contained expressions, but less useful for other types
|
||||
of phrases or overlapping spans. The new
|
||||
[`SpanCategorizer`](/api/spancategorizer) component and
|
||||
[SpanCategorizer](/api/architectures#spancategorizer) architecture let you label
|
||||
arbitrary and potentially overlapping spans of texts. A span categorizer
|
||||
consists of two parts: a [suggester function](/api/spancategorizer#suggesters)
|
||||
that proposes candidate spans, which may or may not overlap, and a labeler model
|
||||
that predicts zero or more labels for each candidate. The predicted spans are
|
||||
available via the [`Doc.spans`](/api/doc#spans) container.
|
||||
|
||||
<Project id="experimental/ner_spancat">
|
||||
|
||||
This project trains a span categorizer for Indonesian NER.
|
||||
|
||||
</Project>
|
||||
|
||||
<Infobox title="Tip: Create data with Prodigy's new span annotation UI">
|
||||
|
||||
<Image
|
||||
src="/images/prodigy_spans-manual.jpg"
|
||||
href="https://support.prodi.gy/t/3861"
|
||||
alt="Prodigy: example of the new manual spans UI"
|
||||
/>
|
||||
|
||||
The upcoming version of our annotation tool [Prodigy](https://prodi.gy)
|
||||
(currently available as a [pre-release](https://support.prodi.gy/t/3861) for all
|
||||
users) features a [new workflow and UI](https://support.prodi.gy/t/3861) for
|
||||
annotating overlapping and nested spans. You can use it to create training data
|
||||
for spaCy's `SpanCategorizer` component.
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Update the entity recognizer with partial incorrect annotations {id="negative-samples"}
|
||||
|
||||
> #### config.cfg (excerpt)
|
||||
>
|
||||
> ```ini
|
||||
> [components.ner]
|
||||
> factory = "ner"
|
||||
> incorrect_spans_key = "incorrect_spans"
|
||||
> moves = null
|
||||
> update_with_oracle_cut_size = 100
|
||||
> ```
|
||||
|
||||
The [`EntityRecognizer`](/api/entityrecognizer) can now be updated with known
|
||||
incorrect annotations, which lets you take advantage of partial and sparse data.
|
||||
For example, you'll be able to use the information that certain spans of text
|
||||
are definitely **not** `PERSON` entities, without having to provide the complete
|
||||
gold-standard annotations for the given example. The incorrect span annotations
|
||||
can be added via the [`Doc.spans`](/api/doc#spans) in the training data under
|
||||
the key defined as [`incorrect_spans_key`](/api/entityrecognizer#init) in the
|
||||
component config.
|
||||
|
||||
```python
|
||||
train_doc = nlp.make_doc("Barack Obama was born in Hawaii.")
|
||||
# The doc.spans key can be defined in the config
|
||||
train_doc.spans["incorrect_spans"] = [
|
||||
Span(doc, 0, 2, label="ORG"),
|
||||
Span(doc, 5, 6, label="PRODUCT")
|
||||
]
|
||||
```
|
||||
|
||||
{/* TODO: more details and/or example project? */}
|
||||
|
||||
### New pipeline packages for Catalan and Danish {id="pipeline-packages"}
|
||||
|
||||
spaCy v3.1 adds 5 new pipeline packages, including a new core family for Catalan
|
||||
and a new transformer-based pipeline for Danish using the
|
||||
[`danish-bert-botxo`](http://huggingface.co/Maltehb/danish-bert-botxo) weights.
|
||||
See the [models directory](/models) for an overview of all available trained
|
||||
pipelines and the [training guide](/usage/training) for details on how to train
|
||||
your own.
|
||||
|
||||
> Thanks to Carlos Rodríguez Penagos and the
|
||||
> [Barcelona Supercomputing Center](https://temu.bsc.es/) for their
|
||||
> contributions for Catalan and to Kenneth Enevoldsen for Danish. For additional
|
||||
> Danish pipelines, check out [DaCy](https://github.com/KennethEnevoldsen/DaCy).
|
||||
|
||||
| Package | Language | UPOS | Parser LAS | NER F |
|
||||
| ------------------------------------------------- | -------- | ---: | ---------: | ----: |
|
||||
| [`ca_core_news_sm`](/models/ca#ca_core_news_sm) | Catalan | 98.2 | 87.4 | 79.8 |
|
||||
| [`ca_core_news_md`](/models/ca#ca_core_news_md) | Catalan | 98.3 | 88.2 | 84.0 |
|
||||
| [`ca_core_news_lg`](/models/ca#ca_core_news_lg) | Catalan | 98.5 | 88.4 | 84.2 |
|
||||
| [`ca_core_news_trf`](/models/ca#ca_core_news_trf) | Catalan | 98.9 | 93.0 | 91.2 |
|
||||
| [`da_core_news_trf`](/models/da#da_core_news_trf) | Danish | 98.0 | 85.0 | 82.9 |
|
||||
|
||||
### Resizable text classification architectures {id="resizable-textcat"}
|
||||
|
||||
Previously, the [`TextCategorizer`](/api/textcategorizer) architectures could
|
||||
not be resized, meaning that you couldn't add new labels to an already trained
|
||||
model. In spaCy v3.1, the [TextCatCNN](/api/architectures#TextCatCNN) and
|
||||
[TextCatBOW](/api/architectures#TextCatBOW) architectures are now resizable,
|
||||
while ensuring that the predictions for the old labels remain the same.
|
||||
|
||||
### CLI command to assemble pipeline from config {id="assemble"}
|
||||
|
||||
The [`spacy assemble`](/api/cli#assemble) command lets you assemble a pipeline
|
||||
from a config file without additional training. It can be especially useful for
|
||||
creating a blank pipeline with a custom tokenizer, rule-based components or word
|
||||
vectors.
|
||||
|
||||
```bash
|
||||
$ python -m spacy assemble config.cfg ./output
|
||||
```
|
||||
|
||||
### Pretty pipeline package READMEs {id="package-readme"}
|
||||
|
||||
The [`spacy package`](/api/cli#package) command now auto-generates a pretty
|
||||
`README.md` based on the pipeline information defined in the `meta.json`. This
|
||||
includes a table with a general overview, as well as the label scheme and
|
||||
accuracy figures, if available. For an example, see the
|
||||
[model releases](https://github.com/explosion/spacy-models/releases).
|
||||
|
||||
### Support for streaming large or infinite corpora {id="streaming-corpora"}
|
||||
|
||||
> #### config.cfg (excerpt)
|
||||
>
|
||||
> ```ini
|
||||
> [training]
|
||||
> max_epochs = -1
|
||||
> ```
|
||||
|
||||
The training process now supports streaming large or infinite corpora
|
||||
out-of-the-box, which can be controlled via the
|
||||
[`[training.max_epochs]`](/api/data-formats#training) config setting. Setting it
|
||||
to `-1` means that the train corpus should be streamed rather than loaded into
|
||||
memory with no shuffling within the training loop. For details on how to
|
||||
implement a custom corpus loader, e.g. to stream in data from a remote storage,
|
||||
see the usage guide on
|
||||
[custom data reading](/usage/training#custom-code-readers-batchers).
|
||||
|
||||
When streaming a corpus, only the first 100 examples will be used for
|
||||
[initialization](/usage/training#config-lifecycle). This is no problem if you're
|
||||
training a component like the text classifier with data that specifies all
|
||||
available labels in every example. If necessary, you can use the
|
||||
[`init labels`](/api/cli#init-labels) command to pre-generate the labels for
|
||||
your components using a representative sample so the model can be initialized
|
||||
correctly before training.
|
||||
|
||||
### New lemmatizers for Catalan and Italian {id="pos-lemmatizers"}
|
||||
|
||||
The trained pipelines for [Catalan](/models/ca) and [Italian](/models/it) now
|
||||
include lemmatizers that use the predicted part-of-speech tags as part of the
|
||||
lookup lemmatization for higher lemmatization accuracy. If you're training your
|
||||
own pipelines for these languages and you want to include a lemmatizer, make
|
||||
sure you have the
|
||||
[`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data) package
|
||||
installed, which provides the relevant tables.
|
||||
|
||||
### Upload your pipelines to the Hugging Face Hub {id="huggingface-hub"}
|
||||
|
||||
The [Hugging Face Hub](https://huggingface.co/) lets you upload models and share
|
||||
them with others, and it now supports spaCy pipelines out-of-the-box. The new
|
||||
[`spacy-huggingface-hub`](https://github.com/explosion/spacy-huggingface-hub)
|
||||
package automatically adds the `huggingface-hub` command to your `spacy` CLI. It
|
||||
lets you upload any pipelines packaged with [`spacy package`](/api/cli#package)
|
||||
and `--build wheel` and takes care of auto-generating all required meta
|
||||
information.
|
||||
|
||||
After uploading, you'll get a live URL for your model page that includes all
|
||||
details, files and interactive visualizers, as well as a direct URL to the wheel
|
||||
file that you can install via `pip install`. For examples, check out the
|
||||
[spaCy pipelines](https://huggingface.co/spacy) we've uploaded.
|
||||
|
||||
```bash
|
||||
$ pip install spacy-huggingface-hub
|
||||
$ huggingface-cli login
|
||||
$ python -m spacy package ./en_ner_fashion ./output --build wheel
|
||||
$ cd ./output/en_ner_fashion-0.0.0/dist
|
||||
$ python -m spacy huggingface-hub push en_ner_fashion-0.0.0-py3-none-any.whl
|
||||
```
|
||||
|
||||
You can also integrate the upload command into your
|
||||
[project template](/usage/projects#huggingface_hub) to automatically upload your
|
||||
packaged pipelines after training.
|
||||
|
||||
<Project id="integrations/huggingface_hub">
|
||||
|
||||
Get started with uploading your models to the Hugging Face hub using our project
|
||||
template. It trains a simple pipeline, packages it and uploads it if the
|
||||
packaged model has changed. This makes it easy to deploy your models end-to-end.
|
||||
|
||||
</Project>
|
||||
|
||||
## Notes about upgrading from v3.0 {id="upgrading"}
|
||||
|
||||
### Pipeline package version compatibility {id="version-compat"}
|
||||
|
||||
> #### Using legacy implementations
|
||||
>
|
||||
> In spaCy v3, you'll still be able to load and reference legacy implementations
|
||||
> via [`spacy-legacy`](https://github.com/explosion/spacy-legacy), even if the
|
||||
> components or architectures change and newer versions are available in the
|
||||
> core library.
|
||||
|
||||
When you're loading a pipeline package trained with spaCy v3.0, you will see a
|
||||
warning telling you that the pipeline may be incompatible. This doesn't
|
||||
necessarily have to be true, but we recommend running your pipelines against
|
||||
your test suite or evaluation data to make sure there are no unexpected results.
|
||||
If you're using one of the [trained pipelines](/models) we provide, you should
|
||||
run [`spacy download`](/api/cli#download) to update to the latest version. To
|
||||
see an overview of all installed packages and their compatibility, you can run
|
||||
[`spacy validate`](/api/cli#validate).
|
||||
|
||||
If you've trained your own custom pipeline and you've confirmed that it's still
|
||||
working as expected, you can update the spaCy version requirements in the
|
||||
[`meta.json`](/api/data-formats#meta):
|
||||
|
||||
```diff
|
||||
- "spacy_version": ">=3.0.0,<3.1.0",
|
||||
+ "spacy_version": ">=3.0.0,<3.2.0",
|
||||
```
|
||||
|
||||
### Updating v3.0 configs
|
||||
|
||||
To update a config from spaCy v3.0 with the new v3.1 settings, run
|
||||
[`init fill-config`](/api/cli#init-fill-config):
|
||||
|
||||
```bash
|
||||
python -m spacy init fill-config config-v3.0.cfg config-v3.1.cfg
|
||||
```
|
||||
|
||||
In many cases (`spacy train`, `spacy.load()`), the new defaults will be filled
|
||||
in automatically, but you'll need to fill in the new settings to run
|
||||
[`debug config`](/api/cli#debug) and [`debug data`](/api/cli#debug-data).
|
||||
|
||||
### Sourcing pipeline components with vectors {id="source-vectors"}
|
||||
|
||||
If you're sourcing a pipeline component that requires static vectors (for
|
||||
example, a tagger or parser from an `md` or `lg` pretrained pipeline), be sure
|
||||
to include the source model's vectors in the setting `[initialize.vectors]`. In
|
||||
spaCy v3.0, a bug allowed vectors to be loaded implicitly through `source`,
|
||||
however in v3.1 this setting must be provided explicitly as
|
||||
`[initialize.vectors]`:
|
||||
|
||||
```ini {title="config.cfg (excerpt)"}
|
||||
[components.ner]
|
||||
source = "en_core_web_md"
|
||||
|
||||
[initialize]
|
||||
vectors = "en_core_web_md"
|
||||
```
|
||||
|
||||
<Infobox title="Important note" variant="warning">
|
||||
|
||||
Each pipeline can only store one set of static vectors, so it's not possible to
|
||||
assemble a pipeline with components that were trained on different static
|
||||
vectors.
|
||||
|
||||
</Infobox>
|
||||
|
||||
[`spacy train`](/api/cli#train) and [`spacy assemble`](/api/cli#assemble) will
|
||||
provide warnings if the source and target pipelines don't contain the same
|
||||
vectors. If you are sourcing a rule-based component like an entity ruler or
|
||||
lemmatizer that does not use the vectors as a model feature, then this warning
|
||||
can be safely ignored.
|
||||
|
||||
### Warnings {id="warnings"}
|
||||
|
||||
Logger warnings have been converted to Python warnings. Use
|
||||
[`warnings.filterwarnings`](https://docs.python.org/3/library/warnings.html#warnings.filterwarnings)
|
||||
or the new helper method `spacy.errors.filter_warning(action, error_msg='')` to
|
||||
manage warnings.
|
||||
@@ -0,0 +1,242 @@
|
||||
---
|
||||
title: What's New in v3.2
|
||||
teaser: New features and how to upgrade
|
||||
menu:
|
||||
- ['New Features', 'features']
|
||||
- ['Upgrading Notes', 'upgrading']
|
||||
---
|
||||
|
||||
## New Features {id="features",hidden="true"}
|
||||
|
||||
spaCy v3.2 adds support for [`floret`](https://github.com/explosion/floret)
|
||||
vectors, makes custom `Doc` creation and scoring easier, and includes many bug
|
||||
fixes and improvements. For the trained pipelines, there's a new transformer
|
||||
pipeline for Japanese and the Universal Dependencies training data has been
|
||||
updated across the board to the most recent release.
|
||||
|
||||
<Infobox title="Improve performance for spaCy on Apple M1 with AppleOps" variant="warning" emoji="📣">
|
||||
|
||||
spaCy is now up to **8 × faster on M1 Macs** by calling into Apple's
|
||||
native Accelerate library for matrix multiplication. For more details, see
|
||||
[`thinc-apple-ops`](https://github.com/explosion/thinc-apple-ops).
|
||||
|
||||
```bash
|
||||
$ pip install spacy[apple]
|
||||
```
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Registered scoring functions {id="registered-scoring-functions"}
|
||||
|
||||
To customize the scoring, you can specify a scoring function for each component
|
||||
in your config from the new [`scorers` registry](/api/top-level#registry):
|
||||
|
||||
```ini {title="config.cfg (excerpt)",highlight="3"}
|
||||
[components.tagger]
|
||||
factory = "tagger"
|
||||
scorer = {"@scorers":"spacy.tagger_scorer.v1"}
|
||||
```
|
||||
|
||||
### Overwrite settings {id="overwrite"}
|
||||
|
||||
Most pipeline components now include an `overwrite` setting in the config that
|
||||
determines whether existing annotation in the `Doc` is preserved or overwritten:
|
||||
|
||||
```ini {title="config.cfg (excerpt)",highlight="3"}
|
||||
[components.tagger]
|
||||
factory = "tagger"
|
||||
overwrite = false
|
||||
```
|
||||
|
||||
### Doc input for pipelines {id="doc-input"}
|
||||
|
||||
[`nlp`](/api/language#call) and [`nlp.pipe`](/api/language#pipe) accept
|
||||
[`Doc`](/api/doc) input, skipping the tokenizer if a `Doc` is provided instead
|
||||
of a string. This makes it easier to create a `Doc` with custom tokenization or
|
||||
to set custom extensions before processing:
|
||||
|
||||
```python
|
||||
doc = nlp.make_doc("This is text 500.")
|
||||
doc._.text_id = 500
|
||||
doc = nlp(doc)
|
||||
```
|
||||
|
||||
### Support for floret vectors {id="vectors"}
|
||||
|
||||
We recently published [`floret`](https://github.com/explosion/floret), an
|
||||
extended version of [fastText](https://fasttext.cc) that combines fastText's
|
||||
subwords with Bloom embeddings for compact, full-coverage vectors. The use of
|
||||
subwords means that there are no OOV words and due to Bloom embeddings, the
|
||||
vector table can be kept very small at \<100K entries. Bloom embeddings are
|
||||
already used by [HashEmbed](https://thinc.ai/docs/api-layers#hashembed) in
|
||||
[tok2vec](/api/architectures#tok2vec-arch) for compact spaCy models.
|
||||
|
||||
For easy integration, floret includes a
|
||||
[Python wrapper](https://github.com/explosion/floret/blob/main/python/README.md):
|
||||
|
||||
```bash
|
||||
$ pip install floret
|
||||
```
|
||||
|
||||
A demo project shows how to train and import floret vectors:
|
||||
|
||||
<Project id="pipelines/floret_vectors_demo">
|
||||
|
||||
Train toy English floret vectors and import them into a spaCy pipeline.
|
||||
|
||||
</Project>
|
||||
|
||||
Two additional demo projects compare standard fastText vectors with floret
|
||||
vectors for full spaCy pipelines. For agglutinative languages like Finnish or
|
||||
Korean, there are large improvements in performance due to the use of subwords
|
||||
(no OOV words!), with a vector table containing merely 50K entries.
|
||||
|
||||
<Project id="pipelines/floret_fi_core_demo">
|
||||
|
||||
Finnish UD+NER vector and pipeline training, comparing standard fasttext vs.
|
||||
floret vectors.
|
||||
|
||||
For the default project settings with 1M (2.6G) tokenized training texts and 50K
|
||||
300-dim vectors, ~300K keys for the standard vectors:
|
||||
|
||||
| Vectors | TAG | POS | DEP UAS | DEP LAS | NER F |
|
||||
| -------------------------------------------- | -------: | -------: | -------: | -------: | -------: |
|
||||
| none | 93.3 | 92.3 | 79.7 | 72.8 | 61.0 |
|
||||
| standard (pruned: 50K vectors for 300K keys) | 95.9 | 94.7 | 83.3 | 77.9 | 68.5 |
|
||||
| standard (unpruned: 300K vectors/keys) | 96.0 | 95.0 | **83.8** | 78.4 | 69.1 |
|
||||
| floret (minn 4, maxn 5; 50K vectors, no OOV) | **96.6** | **95.5** | 83.5 | **78.5** | **70.9** |
|
||||
|
||||
</Project>
|
||||
|
||||
<Project id="pipelines/floret_ko_ud_demo">
|
||||
|
||||
Korean UD vector and pipeline training, comparing standard fasttext vs. floret
|
||||
vectors.
|
||||
|
||||
For the default project settings with 1M (3.3G) tokenized training texts and 50K
|
||||
300-dim vectors, ~800K keys for the standard vectors:
|
||||
|
||||
| Vectors | TAG | POS | DEP UAS | DEP LAS |
|
||||
| -------------------------------------------- | -------: | -------: | -------: | -------: |
|
||||
| none | 72.5 | 85.0 | 73.2 | 64.3 |
|
||||
| standard (pruned: 50K vectors for 800K keys) | 77.9 | 89.4 | 78.8 | 72.8 |
|
||||
| standard (unpruned: 800K vectors/keys) | 79.0 | 90.2 | 79.2 | 73.9 |
|
||||
| floret (minn 2, maxn 3; 50K vectors, no OOV) | **82.5** | **93.8** | **83.0** | **80.1** |
|
||||
|
||||
</Project>
|
||||
|
||||
### Updates for spacy-transformers v1.1 {id="spacy-transformers"}
|
||||
|
||||
[`spacy-transformers`](https://github.com/explosion/spacy-transformers) v1.1 has
|
||||
been refactored to improve serialization and support of inline transformer
|
||||
components and replacing listeners. In addition, the transformer model output is
|
||||
provided as
|
||||
[`ModelOutput`](https://huggingface.co/transformers/main_classes/output.html?highlight=modeloutput#transformers.file_utils.ModelOutput)
|
||||
instead of tuples in
|
||||
`TransformerData.model_output and FullTransformerBatch.model_output.` For
|
||||
backwards compatibility, the tuple format remains available under
|
||||
`TransformerData.tensors` and `FullTransformerBatch.tensors`. See more details
|
||||
in the [transformer API docs](/api/architectures#TransformerModel).
|
||||
|
||||
`spacy-transformers` v1.1 also adds support for `transformer_config` settings
|
||||
such as `output_attentions`. Additional output is stored under
|
||||
`TransformerData.model_output`. More details are in the
|
||||
[TransformerModel docs](/api/architectures#TransformerModel). The training speed
|
||||
has been improved by streamlining allocations for tokenizer output and there is
|
||||
new support for [mixed-precision training](/api/architectures#TransformerModel).
|
||||
|
||||
### New transformer package for Japanese {id="pipeline-packages"}
|
||||
|
||||
spaCy v3.2 adds a new transformer pipeline package for Japanese
|
||||
[`ja_core_news_trf`](/models/ja#ja_core_news_trf), which uses the `basic`
|
||||
pretokenizer instead of `mecab` to limit the number of dependencies required for
|
||||
the pipeline. Thanks to Hiroshi Matsuda and the spaCy Japanese community for
|
||||
their contributions!
|
||||
|
||||
### Pipeline and language updates {id="pipeline-updates"}
|
||||
|
||||
- All Universal Dependencies training data has been updated to v2.8.
|
||||
- The Catalan data, tokenizer and lemmatizer have been updated, thanks to Carlos
|
||||
Rodriguez, Carme Armentano and the Barcelona Supercomputing Center!
|
||||
- The transformer pipelines are trained using spacy-transformers v1.1, with
|
||||
improved IO and more options for
|
||||
[model config and output](/api/architectures#TransformerModel).
|
||||
- Trailing whitespace has been added as a `tok2vec` feature, improving the
|
||||
performance for many components, especially fine-grained tagging and sentence
|
||||
segmentation.
|
||||
- The English attribute ruler patterns have been overhauled to improve
|
||||
`Token.pos` and `Token.morph`.
|
||||
|
||||
spaCy v3.2 also features a new Irish lemmatizer, support for `noun_chunks` in
|
||||
Portuguese, improved `noun_chunks` for Spanish and additional updates for
|
||||
Bulgarian, Catalan, Sinhala, Tagalog, Tigrinya and Vietnamese.
|
||||
|
||||
## Notes about upgrading from v3.1 {id="upgrading"}
|
||||
|
||||
### Pipeline package version compatibility {id="version-compat"}
|
||||
|
||||
> #### Using legacy implementations
|
||||
>
|
||||
> In spaCy v3, you'll still be able to load and reference legacy implementations
|
||||
> via [`spacy-legacy`](https://github.com/explosion/spacy-legacy), even if the
|
||||
> components or architectures change and newer versions are available in the
|
||||
> core library.
|
||||
|
||||
When you're loading a pipeline package trained with spaCy v3.0 or v3.1, you will
|
||||
see a warning telling you that the pipeline may be incompatible. This doesn't
|
||||
necessarily have to be true, but we recommend running your pipelines against
|
||||
your test suite or evaluation data to make sure there are no unexpected results.
|
||||
If you're using one of the [trained pipelines](/models) we provide, you should
|
||||
run [`spacy download`](/api/cli#download) to update to the latest version. To
|
||||
see an overview of all installed packages and their compatibility, you can run
|
||||
[`spacy validate`](/api/cli#validate).
|
||||
|
||||
If you've trained your own custom pipeline and you've confirmed that it's still
|
||||
working as expected, you can update the spaCy version requirements in the
|
||||
[`meta.json`](/api/data-formats#meta):
|
||||
|
||||
```diff
|
||||
- "spacy_version": ">=3.1.0,<3.2.0",
|
||||
+ "spacy_version": ">=3.2.0,<3.3.0",
|
||||
```
|
||||
|
||||
### Updating v3.1 configs
|
||||
|
||||
To update a config from spaCy v3.1 with the new v3.2 settings, run
|
||||
[`init fill-config`](/api/cli#init-fill-config):
|
||||
|
||||
```bash
|
||||
$ python -m spacy init fill-config config-v3.1.cfg config-v3.2.cfg
|
||||
```
|
||||
|
||||
In many cases ([`spacy train`](/api/cli#train),
|
||||
[`spacy.load`](/api/top-level#spacy.load)), the new defaults will be filled in
|
||||
automatically, but you'll need to fill in the new settings to run
|
||||
[`debug config`](/api/cli#debug) and [`debug data`](/api/cli#debug-data).
|
||||
|
||||
## Notes about upgrading from spacy-transformers v1.0 {id="upgrading-transformers"}
|
||||
|
||||
When you're loading a transformer pipeline package trained with
|
||||
[`spacy-transformers`](https://github.com/explosion/spacy-transformers) v1.0
|
||||
after upgrading to `spacy-transformers` v1.1, you'll see a warning telling you
|
||||
that the pipeline may be incompatible. `spacy-transformers` v1.1 should be able
|
||||
to import v1.0 `transformer` components into the new internal format with no
|
||||
change in performance, but here we'd also recommend running your test suite to
|
||||
verify that the pipeline still performs as expected.
|
||||
|
||||
If you save your pipeline with [`nlp.to_disk`](/api/language#to_disk), it will
|
||||
be saved in the new v1.1 format and should be fully compatible with
|
||||
`spacy-transformers` v1.1. Once you've confirmed the performance, you can update
|
||||
the requirements in [`meta.json`](/api/data-formats#meta):
|
||||
|
||||
```diff
|
||||
"requirements": [
|
||||
- "spacy-transformers>=1.0.3,<1.1.0"
|
||||
+ "spacy-transformers>=1.1.2,<1.2.0"
|
||||
]
|
||||
```
|
||||
|
||||
If you're using one of the [trained pipelines](/models) we provide, you should
|
||||
run [`spacy download`](/api/cli#download) to update to the latest version. To
|
||||
see an overview of all installed packages and their compatibility, you can run
|
||||
[`spacy validate`](/api/cli#validate).
|
||||
@@ -0,0 +1,247 @@
|
||||
---
|
||||
title: What's New in v3.3
|
||||
teaser: New features and how to upgrade
|
||||
menu:
|
||||
- ['New Features', 'features']
|
||||
- ['Upgrading Notes', 'upgrading']
|
||||
---
|
||||
|
||||
## New features {id="features",hidden="true"}
|
||||
|
||||
spaCy v3.3 improves the speed of core pipeline components, adds a new trainable
|
||||
lemmatizer, and introduces trained pipelines for Finnish, Korean and Swedish.
|
||||
|
||||
### Speed improvements {id="speed"}
|
||||
|
||||
v3.3 includes a slew of speed improvements:
|
||||
|
||||
- Speed up parser and NER by using constant-time head lookups.
|
||||
- Support unnormalized softmax probabilities in `spacy.Tagger.v2` to speed up
|
||||
inference for tagger, morphologizer, senter and trainable lemmatizer.
|
||||
- Speed up parser projectivization functions.
|
||||
- Replace `Ragged` with faster `AlignmentArray` in `Example` for training.
|
||||
- Improve `Matcher` speed.
|
||||
- Improve serialization speed for empty `Doc.spans`.
|
||||
|
||||
For longer texts, the trained pipeline speeds improve **15%** or more in
|
||||
prediction. We benchmarked `en_core_web_md` (same components as in v3.2) and
|
||||
`de_core_news_md` (with the new trainable lemmatizer) across a range of text
|
||||
sizes on Linux (Intel Xeon W-2265) and OS X (M1) to compare spaCy v3.2 vs. v3.3:
|
||||
|
||||
**Intel Xeon W-2265**
|
||||
|
||||
| Model | Avg. Words/Doc | v3.2 Words/Sec | v3.3 Words/Sec | Diff |
|
||||
| :----------------------------------------------- | -------------: | -------------: | -------------: | -----: |
|
||||
| [`en_core_web_md`](/models/en#en_core_web_md) | 100 | 17292 | 17441 | 0.86% |
|
||||
| (=same components) | 1000 | 15408 | 16024 | 4.00% |
|
||||
| | 10000 | 12798 | 15346 | 19.91% |
|
||||
| [`de_core_news_md`](/models/de/#de_core_news_md) | 100 | 20221 | 19321 | -4.45% |
|
||||
| (+v3.3 trainable lemmatizer) | 1000 | 17480 | 17345 | -0.77% |
|
||||
| | 10000 | 14513 | 17036 | 17.38% |
|
||||
|
||||
**Apple M1**
|
||||
|
||||
| Model | Avg. Words/Doc | v3.2 Words/Sec | v3.3 Words/Sec | Diff |
|
||||
| ------------------------------------------------ | -------------: | -------------: | -------------: | -----: |
|
||||
| [`en_core_web_md`](/models/en#en_core_web_md) | 100 | 18272 | 18408 | 0.74% |
|
||||
| (=same components) | 1000 | 18794 | 19248 | 2.42% |
|
||||
| | 10000 | 15144 | 17513 | 15.64% |
|
||||
| [`de_core_news_md`](/models/de/#de_core_news_md) | 100 | 19227 | 19591 | 1.89% |
|
||||
| (+v3.3 trainable lemmatizer) | 1000 | 20047 | 20628 | 2.90% |
|
||||
| | 10000 | 15921 | 18546 | 16.49% |
|
||||
|
||||
### Trainable lemmatizer {id="trainable-lemmatizer"}
|
||||
|
||||
The new [trainable lemmatizer](/api/edittreelemmatizer) component uses
|
||||
[edit trees](https://explosion.ai/blog/edit-tree-lemmatizer) to transform tokens
|
||||
into lemmas. Try out the trainable lemmatizer with the
|
||||
[training quickstart](/usage/training#quickstart)!
|
||||
|
||||
### displaCy support for overlapping spans and arcs {id="displacy"}
|
||||
|
||||
displaCy now supports overlapping spans with a new
|
||||
[`span`](/usage/visualizers#span) style and multiple arcs with different labels
|
||||
between the same tokens for [`dep`](/usage/visualizers#dep) visualizations.
|
||||
|
||||
Overlapping spans can be visualized for any spans key in `doc.spans`:
|
||||
|
||||
```python
|
||||
import spacy
|
||||
from spacy import displacy
|
||||
from spacy.tokens import Span
|
||||
|
||||
nlp = spacy.blank("en")
|
||||
text = "Welcome to the Bank of China."
|
||||
doc = nlp(text)
|
||||
doc.spans["custom"] = [Span(doc, 3, 6, "ORG"), Span(doc, 5, 6, "GPE")]
|
||||
displacy.serve(doc, style="span", options={"spans_key": "custom"})
|
||||
```
|
||||
|
||||
<Standalone height={100}>
|
||||
<div style={{ lineHeight: 2.5, direction: 'ltr', fontFamily: "-apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'", fontSize: 18 }}>Welcome to the <span style={{ fontWeight: 'bold', display: 'inline-block', position: 'relative'}}>Bank<span style={{ background: '#7aecec', top: 40, height: 4, left: -1, width: 'calc(100% + 2px)', position: 'absolute' }}></span><span style={{ background: '#7aecec', top: 40, height: 4, borderTopLeftRadius: 3, borderBottomLeftRadius: 3, left: -1, width: 'calc(100% + 2px)', position: 'absolute' }}><span style={{ background: '#7aecec', color: '#000', top: '-0.5em', padding: '2px 3px', position: 'absolute', fontSize: '0.6em', fontWeight: 'bold', lineHeight: 1, borderRadius: 3 }}>ORG</span></span></span> <span style={{ fontWeight: 'bold', display: 'inline-block', position: 'relative'}}>of <span style={{ background: '#7aecec', top: 40, height: 4, left: -1, width: 'calc(100% + 2px)', position: 'absolute' }}></span></span> <span style={{ fontWeight: 'bold', display: 'inline-block', position: 'relative'}}>China<span style={{ background: '#7aecec', top: 40, height: 4, left: -1, width: 'calc(100% + 2px)', position: 'absolute' }}></span><span style={{ background: '#feca74', top: 57, height: 4, left: -1, width: 'calc(100% + 2px)', position: 'absolute' }}></span><span style={{ background: '#feca74', top: 57, height: 4, borderTopLeftRadius: 3, borderBottomLeftRadius: 3, left: -1, width: 'calc(100% + 2px)', position: 'absolute' }}><span style={{ background: '#feca74', color: '#000', top: '-0.5em', padding: '2px 3px', position: 'absolute', fontSize: '0.6em', fontWeight: 'bold', lineHeight: 1, borderRadius: 3 }}>GPE</span></span></span>.</div>
|
||||
</Standalone>
|
||||
|
||||
## Additional features and improvements
|
||||
|
||||
- Config comparisons with [`spacy debug diff-config`](/api/cli#debug-diff).
|
||||
- Span suggester debugging with
|
||||
[`SpanCategorizer.set_candidates`](/api/spancategorizer#set_candidates).
|
||||
- Big endian support with
|
||||
[`thinc-bigendian-ops`](https://github.com/andrewsi-z/thinc-bigendian-ops) and
|
||||
updates to make `floret`, `murmurhash`, Thinc and spaCy endian neutral.
|
||||
- Initial support for Lower Sorbian and Upper Sorbian.
|
||||
- Language updates for English, French, Italian, Japanese, Korean, Norwegian,
|
||||
Russian, Slovenian, Spanish, Turkish, Ukrainian and Vietnamese.
|
||||
- New noun chunks for Finnish.
|
||||
|
||||
## Trained pipelines {id="pipelines"}
|
||||
|
||||
### New trained pipelines {id="new-pipelines"}
|
||||
|
||||
v3.3 introduces new CPU/CNN pipelines for Finnish, Korean and Swedish, which use
|
||||
the new trainable lemmatizer and
|
||||
[floret vectors](https://github.com/explosion/floret). Due to the use
|
||||
[Bloom embeddings](https://explosion.ai/blog/bloom-embeddings) and subwords, the
|
||||
pipelines have compact vectors with no out-of-vocabulary words.
|
||||
|
||||
| Package | Language | UPOS | Parser LAS | NER F |
|
||||
| ----------------------------------------------- | -------- | ---: | ---------: | ----: |
|
||||
| [`fi_core_news_sm`](/models/fi#fi_core_news_sm) | Finnish | 92.5 | 71.9 | 75.9 |
|
||||
| [`fi_core_news_md`](/models/fi#fi_core_news_md) | Finnish | 95.9 | 78.6 | 80.6 |
|
||||
| [`fi_core_news_lg`](/models/fi#fi_core_news_lg) | Finnish | 96.2 | 79.4 | 82.4 |
|
||||
| [`ko_core_news_sm`](/models/ko#ko_core_news_sm) | Korean | 86.1 | 65.6 | 71.3 |
|
||||
| [`ko_core_news_md`](/models/ko#ko_core_news_md) | Korean | 94.7 | 80.9 | 83.1 |
|
||||
| [`ko_core_news_lg`](/models/ko#ko_core_news_lg) | Korean | 94.7 | 81.3 | 85.3 |
|
||||
| [`sv_core_news_sm`](/models/sv#sv_core_news_sm) | Swedish | 95.0 | 75.9 | 74.7 |
|
||||
| [`sv_core_news_md`](/models/sv#sv_core_news_md) | Swedish | 96.3 | 78.5 | 79.3 |
|
||||
| [`sv_core_news_lg`](/models/sv#sv_core_news_lg) | Swedish | 96.3 | 79.1 | 81.1 |
|
||||
|
||||
### Pipeline updates {id="pipeline-updates"}
|
||||
|
||||
The following languages switch from lookup or rule-based lemmatizers to the new
|
||||
trainable lemmatizer: Danish, Dutch, German, Greek, Italian, Lithuanian,
|
||||
Norwegian, Polish, Portuguese and Romanian. The overall lemmatizer accuracy
|
||||
improves for all of these pipelines, but be aware that the types of errors may
|
||||
look quite different from the lookup-based lemmatizers. If you'd prefer to
|
||||
continue using the previous lemmatizer, you can
|
||||
[switch from the trainable lemmatizer to a non-trainable lemmatizer](/models#design-modify).
|
||||
|
||||
<figure>
|
||||
|
||||
| Model | v3.2 Lemma Acc | v3.3 Lemma Acc |
|
||||
| ----------------------------------------------- | -------------: | -------------: |
|
||||
| [`da_core_news_md`](/models/da#da_core_news_md) | 84.9 | 94.8 |
|
||||
| [`de_core_news_md`](/models/de#de_core_news_md) | 73.4 | 97.7 |
|
||||
| [`el_core_news_md`](/models/el#el_core_news_md) | 56.5 | 88.9 |
|
||||
| [`fi_core_news_md`](/models/fi#fi_core_news_md) | - | 86.2 |
|
||||
| [`it_core_news_md`](/models/it#it_core_news_md) | 86.6 | 97.2 |
|
||||
| [`ko_core_news_md`](/models/ko#ko_core_news_md) | - | 90.0 |
|
||||
| [`lt_core_news_md`](/models/lt#lt_core_news_md) | 71.1 | 84.8 |
|
||||
| [`nb_core_news_md`](/models/nb#nb_core_news_md) | 76.7 | 97.1 |
|
||||
| [`nl_core_news_md`](/models/nl#nl_core_news_md) | 81.5 | 94.0 |
|
||||
| [`pl_core_news_md`](/models/pl#pl_core_news_md) | 87.1 | 93.7 |
|
||||
| [`pt_core_news_md`](/models/pt#pt_core_news_md) | 76.7 | 96.9 |
|
||||
| [`ro_core_news_md`](/models/ro#ro_core_news_md) | 81.8 | 95.5 |
|
||||
| [`sv_core_news_md`](/models/sv#sv_core_news_md) | - | 95.5 |
|
||||
|
||||
</figure>
|
||||
|
||||
In addition, the vectors in the English pipelines are deduplicated to improve
|
||||
the pruned vectors in the `md` models and reduce the `lg` model size.
|
||||
|
||||
## Notes about upgrading from v3.2 {id="upgrading"}
|
||||
|
||||
### Span comparisons
|
||||
|
||||
Span comparisons involving ordering (`<`, `<=`, `>`, `>=`) now take all span
|
||||
attributes into account (start, end, label, and KB ID) so spans may be sorted in
|
||||
a slightly different order.
|
||||
|
||||
### Whitespace annotation
|
||||
|
||||
During training, annotation on whitespace tokens is handled in the same way as
|
||||
annotation on non-whitespace tokens in order to allow custom whitespace
|
||||
annotation.
|
||||
|
||||
### Doc.from_docs
|
||||
|
||||
[`Doc.from_docs`](/api/doc#from_docs) now includes `Doc.tensor` by default and
|
||||
supports excludes with an `exclude` argument in the same format as
|
||||
`Doc.to_bytes`. The supported exclude fields are `spans`, `tensor` and
|
||||
`user_data`.
|
||||
|
||||
Docs including `Doc.tensor` may be quite a bit larger in RAM, so to exclude
|
||||
`Doc.tensor` as in v3.2:
|
||||
|
||||
```diff
|
||||
-merged_doc = Doc.from_docs(docs)
|
||||
+merged_doc = Doc.from_docs(docs, exclude=["tensor"])
|
||||
```
|
||||
|
||||
### Using trained pipelines with floret vectors
|
||||
|
||||
If you're running a new trained pipeline for Finnish, Korean or Swedish on new
|
||||
texts and working with `Doc` objects, you shouldn't notice any difference with
|
||||
floret vectors vs. default vectors.
|
||||
|
||||
If you use vectors for similarity comparisons, there are a few differences,
|
||||
mainly because a floret pipeline doesn't include any kind of frequency-based
|
||||
word list similar to the list of in-vocabulary vector keys with default vectors.
|
||||
|
||||
- If your workflow iterates over the vector keys, you should use an external
|
||||
word list instead:
|
||||
|
||||
```diff
|
||||
- lexemes = [nlp.vocab[orth] for orth in nlp.vocab.vectors]
|
||||
+ lexemes = [nlp.vocab[word] for word in external_word_list]
|
||||
```
|
||||
|
||||
- `Vectors.most_similar` is not supported because there's no fixed list of
|
||||
vectors to compare your vectors to.
|
||||
|
||||
### Pipeline package version compatibility {id="version-compat"}
|
||||
|
||||
> #### Using legacy implementations
|
||||
>
|
||||
> In spaCy v3, you'll still be able to load and reference legacy implementations
|
||||
> via [`spacy-legacy`](https://github.com/explosion/spacy-legacy), even if the
|
||||
> components or architectures change and newer versions are available in the
|
||||
> core library.
|
||||
|
||||
When you're loading a pipeline package trained with an earlier version of spaCy
|
||||
v3, you will see a warning telling you that the pipeline may be incompatible.
|
||||
This doesn't necessarily have to be true, but we recommend running your
|
||||
pipelines against your test suite or evaluation data to make sure there are no
|
||||
unexpected results.
|
||||
|
||||
If you're using one of the [trained pipelines](/models) we provide, you should
|
||||
run [`spacy download`](/api/cli#download) to update to the latest version. To
|
||||
see an overview of all installed packages and their compatibility, you can run
|
||||
[`spacy validate`](/api/cli#validate).
|
||||
|
||||
If you've trained your own custom pipeline and you've confirmed that it's still
|
||||
working as expected, you can update the spaCy version requirements in the
|
||||
[`meta.json`](/api/data-formats#meta):
|
||||
|
||||
```diff
|
||||
- "spacy_version": ">=3.2.0,<3.3.0",
|
||||
+ "spacy_version": ">=3.2.0,<3.4.0",
|
||||
```
|
||||
|
||||
### Updating v3.2 configs
|
||||
|
||||
To update a config from spaCy v3.2 with the new v3.3 settings, run
|
||||
[`init fill-config`](/api/cli#init-fill-config):
|
||||
|
||||
```bash
|
||||
$ python -m spacy init fill-config config-v3.2.cfg config-v3.3.cfg
|
||||
```
|
||||
|
||||
In many cases ([`spacy train`](/api/cli#train),
|
||||
[`spacy.load`](/api/top-level#spacy.load)), the new defaults will be filled in
|
||||
automatically, but you'll need to fill in the new settings to run
|
||||
[`debug config`](/api/cli#debug) and [`debug data`](/api/cli#debug-data).
|
||||
|
||||
To see the speed improvements for the
|
||||
[`Tagger` architecture](/api/architectures#Tagger), edit your config to switch
|
||||
from `spacy.Tagger.v1` to `spacy.Tagger.v2` and then run `init fill-config`.
|
||||
@@ -0,0 +1,143 @@
|
||||
---
|
||||
title: What's New in v3.4
|
||||
teaser: New features and how to upgrade
|
||||
menu:
|
||||
- ['New Features', 'features']
|
||||
- ['Upgrading Notes', 'upgrading']
|
||||
---
|
||||
|
||||
## New features {id="features",hidden="true"}
|
||||
|
||||
spaCy v3.4 brings typing and speed improvements along with new vectors for
|
||||
English CNN pipelines and new trained pipelines for Croatian. This release also
|
||||
includes prebuilt linux aarch64 wheels for all spaCy dependencies distributed by
|
||||
Explosion.
|
||||
|
||||
### Typing improvements {id="typing"}
|
||||
|
||||
spaCy v3.4 supports pydantic v1.9 and mypy 0.950+ through extensive updates to
|
||||
types in Thinc v8.1.
|
||||
|
||||
### Speed improvements {id="speed"}
|
||||
|
||||
- For the parser, use C `saxpy`/`sgemm` provided by the `Ops` implementation in
|
||||
order to use Accelerate through `thinc-apple-ops`.
|
||||
- Improved speed of vector lookups.
|
||||
- Improved speed for `Example.get_aligned_parse` and `Example.get_aligned`.
|
||||
|
||||
## Additional features and improvements
|
||||
|
||||
- Min/max `{n,m}` operator for `Matcher` patterns.
|
||||
- Language updates:
|
||||
- Improve tokenization for Cyrillic combining diacritics.
|
||||
- Improve English tokenizer exceptions for contractions with
|
||||
this/that/these/those.
|
||||
- Updated `spacy project clone` to try both `main` and `master` branches by
|
||||
default.
|
||||
- Added confidence threshold for named entity linker.
|
||||
- Improved handling of Typer optional default values for `init_config_cli`.
|
||||
- Added cycle detection in parser projectivization methods.
|
||||
- Added counts for NER labels in `debug data`.
|
||||
- Support for adding NVTX ranges to `TrainablePipe` components.
|
||||
- Support env variable `SPACY_NUM_BUILD_JOBS` to specify the number of build
|
||||
jobs to run in parallel with `pip`.
|
||||
|
||||
## Trained pipelines {id="pipelines"}
|
||||
|
||||
### New trained pipelines {id="new-pipelines"}
|
||||
|
||||
v3.4 introduces new CPU/CNN pipelines for Croatian, which use the trainable
|
||||
lemmatizer and [floret vectors](https://github.com/explosion/floret). Due to the
|
||||
use of [Bloom embeddings](https://explosion.ai/blog/bloom-embeddings) and
|
||||
subwords, the pipelines have compact vectors with no out-of-vocabulary words.
|
||||
|
||||
| Package | UPOS | Parser LAS | NER F |
|
||||
| ----------------------------------------------- | ---: | ---------: | ----: |
|
||||
| [`hr_core_news_sm`](/models/hr#hr_core_news_sm) | 96.6 | 77.5 | 76.1 |
|
||||
| [`hr_core_news_md`](/models/hr#hr_core_news_md) | 97.3 | 80.1 | 81.8 |
|
||||
| [`hr_core_news_lg`](/models/hr#hr_core_news_lg) | 97.5 | 80.4 | 83.0 |
|
||||
|
||||
### Pipeline updates {id="pipeline-updates"}
|
||||
|
||||
All CNN pipelines have been extended with whitespace augmentation.
|
||||
|
||||
The English CNN pipelines have new word vectors:
|
||||
|
||||
| Package | Model Version | TAG | Parser LAS | NER F |
|
||||
| --------------------------------------------- | ------------- | ---: | ---------: | ----: |
|
||||
| [`en_core_web_md`](/models/en#en_core_web_md) | v3.3.0 | 97.3 | 90.1 | 84.6 |
|
||||
| [`en_core_web_md`](/models/en#en_core_web_md) | v3.4.0 | 97.2 | 90.3 | 85.5 |
|
||||
| [`en_core_web_lg`](/models/en#en_core_web_lg) | v3.3.0 | 97.4 | 90.1 | 85.3 |
|
||||
| [`en_core_web_lg`](/models/en#en_core_web_lg) | v3.4.0 | 97.3 | 90.2 | 85.6 |
|
||||
|
||||
## Notes about upgrading from v3.3 {id="upgrading"}
|
||||
|
||||
### Doc.has_vector
|
||||
|
||||
`Doc.has_vector` now matches `Token.has_vector` and `Span.has_vector`: it
|
||||
returns `True` if at least one token in the doc has a vector rather than
|
||||
checking only whether the vocab contains vectors.
|
||||
|
||||
### Using trained pipelines with floret vectors
|
||||
|
||||
If you're using a trained pipeline for Croatian, Finnish, Korean or Swedish with
|
||||
new texts and working with `Doc` objects, you shouldn't notice any difference
|
||||
between floret vectors and default vectors.
|
||||
|
||||
If you use vectors for similarity comparisons, there are a few differences,
|
||||
mainly because a floret pipeline doesn't include any kind of frequency-based
|
||||
word list similar to the list of in-vocabulary vector keys with default vectors.
|
||||
|
||||
- If your workflow iterates over the vector keys, you should use an external
|
||||
word list instead:
|
||||
|
||||
```diff
|
||||
- lexemes = [nlp.vocab[orth] for orth in nlp.vocab.vectors]
|
||||
+ lexemes = [nlp.vocab[word] for word in external_word_list]
|
||||
```
|
||||
|
||||
- `Vectors.most_similar` is not supported because there's no fixed list of
|
||||
vectors to compare your vectors to.
|
||||
|
||||
### Pipeline package version compatibility {id="version-compat"}
|
||||
|
||||
> #### Using legacy implementations
|
||||
>
|
||||
> In spaCy v3, you'll still be able to load and reference legacy implementations
|
||||
> via [`spacy-legacy`](https://github.com/explosion/spacy-legacy), even if the
|
||||
> components or architectures change and newer versions are available in the
|
||||
> core library.
|
||||
|
||||
When you're loading a pipeline package trained with an earlier version of spaCy
|
||||
v3, you will see a warning telling you that the pipeline may be incompatible.
|
||||
This doesn't necessarily have to be true, but we recommend running your
|
||||
pipelines against your test suite or evaluation data to make sure there are no
|
||||
unexpected results.
|
||||
|
||||
If you're using one of the [trained pipelines](/models) we provide, you should
|
||||
run [`spacy download`](/api/cli#download) to update to the latest version. To
|
||||
see an overview of all installed packages and their compatibility, you can run
|
||||
[`spacy validate`](/api/cli#validate).
|
||||
|
||||
If you've trained your own custom pipeline and you've confirmed that it's still
|
||||
working as expected, you can update the spaCy version requirements in the
|
||||
[`meta.json`](/api/data-formats#meta):
|
||||
|
||||
```diff
|
||||
- "spacy_version": ">=3.3.0,<3.4.0",
|
||||
+ "spacy_version": ">=3.3.0,<3.5.0",
|
||||
```
|
||||
|
||||
### Updating v3.3 configs
|
||||
|
||||
To update a config from spaCy v3.3 with the new v3.4 settings, run
|
||||
[`init fill-config`](/api/cli#init-fill-config):
|
||||
|
||||
```bash
|
||||
$ python -m spacy init fill-config config-v3.3.cfg config-v3.4.cfg
|
||||
```
|
||||
|
||||
In many cases ([`spacy train`](/api/cli#train),
|
||||
[`spacy.load`](/api/top-level#spacy.load)), the new defaults will be filled in
|
||||
automatically, but you'll need to fill in the new settings to run
|
||||
[`debug config`](/api/cli#debug) and [`debug data`](/api/cli#debug-data).
|
||||
@@ -0,0 +1,230 @@
|
||||
---
|
||||
title: What's New in v3.5
|
||||
teaser: New features and how to upgrade
|
||||
menu:
|
||||
- ['New Features', 'features']
|
||||
- ['Upgrading Notes', 'upgrading']
|
||||
---
|
||||
|
||||
## New features {id="features",hidden="true"}
|
||||
|
||||
spaCy v3.5 introduces three new CLI commands, `apply`, `benchmark` and
|
||||
`find-threshold`, adds fuzzy matching, provides improvements to our entity
|
||||
linking functionality, and includes a range of language updates and bug fixes.
|
||||
|
||||
### New CLI commands {id="cli"}
|
||||
|
||||
#### apply CLI
|
||||
|
||||
The [`apply` CLI](/api/cli#apply) can be used to apply a pipeline to one or more
|
||||
`.txt`, `.jsonl` or `.spacy` input files, saving the annotated docs in a single
|
||||
`.spacy` file.
|
||||
|
||||
```bash
|
||||
$ spacy apply en_core_web_sm my_texts/ output.spacy
|
||||
```
|
||||
|
||||
#### benchmark CLI
|
||||
|
||||
The [`benchmark` CLI](/api/cli#benchmark) has been added to extend the existing
|
||||
`evaluate` functionality with a wider range of profiling subcommands.
|
||||
|
||||
The `benchmark accuracy` CLI is introduced as an alias for `evaluate`. The new
|
||||
`benchmark speed` CLI performs warmup rounds before measuring the speed in words
|
||||
per second on batches of randomly shuffled documents from the provided data.
|
||||
|
||||
```bash
|
||||
$ spacy benchmark speed my_pipeline data.spacy
|
||||
```
|
||||
|
||||
The output is the mean performance using batches (`nlp.pipe`) with a 95%
|
||||
confidence interval, e.g., profiling `en_core_web_sm` on CPU:
|
||||
|
||||
```none
|
||||
Outliers: 2.0%, extreme outliers: 0.0%
|
||||
Mean: 18904.1 words/s (95% CI: -256.9 +244.1)
|
||||
```
|
||||
|
||||
#### find-threshold CLI
|
||||
|
||||
The [`find-threshold` CLI](/api/cli#find-threshold) runs a series of trials
|
||||
across threshold values from `0.0` to `1.0` and identifies the best threshold
|
||||
for the provided score metric.
|
||||
|
||||
The following command runs 20 trials for the `spancat` component in
|
||||
`my_pipeline`, recording the `spans_sc_f` score for each value of the threshold
|
||||
`[components.spancat.threshold]` from `0.0` to `1.0`:
|
||||
|
||||
```bash
|
||||
$ spacy find-threshold my_pipeline data.spacy spancat threshold spans_sc_f --n_trials 20
|
||||
```
|
||||
|
||||
The `find-threshold` CLI can be used with `textcat_multilabel`, `spancat` and
|
||||
custom components with thresholds that are applied while predicting or scoring.
|
||||
|
||||
### Fuzzy matching {id="fuzzy"}
|
||||
|
||||
New `FUZZY` operators support [fuzzy matching](/usage/rule-based-matching#fuzzy)
|
||||
with the `Matcher`. By default, the `FUZZY` operator allows a Levenshtein edit
|
||||
distance of 2 and up to 30% of the pattern string length. `FUZZY1`..`FUZZY9` can
|
||||
be used to specify the exact number of allowed edits.
|
||||
|
||||
```python
|
||||
# Match lowercase with fuzzy matching (allows up to 3 edits)
|
||||
pattern = [{"LOWER": {"FUZZY": "definitely"}}]
|
||||
|
||||
# Match custom attribute values with fuzzy matching (allows up to 3 edits)
|
||||
pattern = [{"_": {"country": {"FUZZY": "Kyrgyzstan"}}}]
|
||||
|
||||
# Match with exact Levenshtein edit distance limits (allows up to 4 edits)
|
||||
pattern = [{"_": {"country": {"FUZZY4": "Kyrgyzstan"}}}]
|
||||
```
|
||||
|
||||
Note that `FUZZY` uses Levenshtein edit distance rather than Damerau-Levenshtein
|
||||
edit distance, so a transposition like `teh` for `the` counts as two edits, one
|
||||
insertion and one deletion.
|
||||
|
||||
If you'd prefer an alternate fuzzy matching algorithm, you can provide your own
|
||||
custom method to the `Matcher` or as a config option for an entity ruler and
|
||||
span ruler.
|
||||
|
||||
### FUZZY and REGEX with lists {id="fuzzy-regex-lists"}
|
||||
|
||||
The `FUZZY` and `REGEX` operators are also now supported for lists with `IN` and
|
||||
`NOT_IN`:
|
||||
|
||||
```python
|
||||
pattern = [{"TEXT": {"FUZZY": {"IN": ["awesome", "cool", "wonderful"]}}}]
|
||||
pattern = [{"TEXT": {"REGEX": {"NOT_IN": ["^awe(some)?$", "^wonder(ful)?"]}}}]
|
||||
```
|
||||
|
||||
### Entity linking generalization {id="el"}
|
||||
|
||||
The knowledge base used for entity linking is now easier to customize and has a
|
||||
new default implementation [`InMemoryLookupKB`](/api/inmemorylookupkb).
|
||||
|
||||
### Additional features and improvements {id="additional-features-and-improvements"}
|
||||
|
||||
- Language updates:
|
||||
- Extended support for Slovenian
|
||||
- Fixed lookup fallback for French and Catalan lemmatizers
|
||||
- Switch Russian and Ukrainian lemmatizers to `pymorphy3`
|
||||
- Support for editorial punctuation in Ancient Greek
|
||||
- Update to Russian tokenizer exceptions
|
||||
- Small fix for Dutch stop words
|
||||
- Allow up to `typer` v0.7.x, `mypy` 0.990 and `typing_extensions` v4.4.x.
|
||||
- New `spacy.ConsoleLogger.v3` with expanded progress
|
||||
[tracking](/api/top-level#ConsoleLogger).
|
||||
- Improved scoring behavior for `textcat` with `spacy.textcat_scorer.v2` and
|
||||
`spacy.textcat_multilabel_scorer.v2`.
|
||||
- Updates so that downstream components can train properly on a frozen `tok2vec`
|
||||
or `transformer` layer.
|
||||
- Allow interpolation of variables in directory names in projects.
|
||||
- Support for local file system [remotes](/usage/projects#remote) for projects.
|
||||
- Improve UX around `displacy.serve` when the default port is in use.
|
||||
- Optional `before_update` callback that is invoked at the start of each
|
||||
[training step](/api/data-formats#config-training).
|
||||
- Improve performance of `SpanGroup` and fix typing issues for `SpanGroup` and
|
||||
`Span` objects.
|
||||
- Patch a
|
||||
[security vulnerability](https://github.com/advisories/GHSA-gw9q-c7gh-j9vm) in
|
||||
extracting tar files.
|
||||
- Add equality definition for `Vectors`.
|
||||
- Ensure `Vocab.to_disk` respects the exclude setting for `lookups` and
|
||||
`vectors`.
|
||||
- Correctly handle missing annotations in the edit tree lemmatizer.
|
||||
|
||||
### Trained pipeline updates {id="pipelines"}
|
||||
|
||||
- The CNN pipelines add `IS_SPACE` as a `tok2vec` feature for `tagger` and
|
||||
`morphologizer` components to improve tagging of non-whitespace vs. whitespace
|
||||
tokens.
|
||||
- The transformer pipelines require `spacy-transformers` v1.2, which uses the
|
||||
exact alignment from `tokenizers` for fast tokenizers instead of the heuristic
|
||||
alignment from `spacy-alignments`. For all trained pipelines except
|
||||
`ja_core_news_trf`, the alignments between spaCy tokens and transformer tokens
|
||||
may be slightly different. More details about the `spacy-transformers` changes
|
||||
in the
|
||||
[v1.2.0 release notes](https://github.com/explosion/spacy-transformers/releases/tag/v1.2.0).
|
||||
|
||||
## Notes about upgrading from v3.4 {id="upgrading"}
|
||||
|
||||
### Validation of textcat values {id="textcat-validation"}
|
||||
|
||||
An error is now raised when unsupported values are given as input to train a
|
||||
`textcat` or `textcat_multilabel` model - ensure that values are `0.0` or `1.0`
|
||||
as explained in the [docs](/api/textcategorizer#assigned-attributes).
|
||||
|
||||
### Using the default knowledge base
|
||||
|
||||
As `KnowledgeBase` is now an abstract class, you should call the constructor of
|
||||
the new `InMemoryLookupKB` instead when you want to use spaCy's default KB
|
||||
implementation:
|
||||
|
||||
```diff
|
||||
- kb = KnowledgeBase()
|
||||
+ kb = InMemoryLookupKB()
|
||||
```
|
||||
|
||||
If you've written a custom KB that inherits from `KnowledgeBase`, you'll need to
|
||||
implement its abstract methods, or alternatively inherit from `InMemoryLookupKB`
|
||||
instead.
|
||||
|
||||
### Updated scorers for tokenization and textcat {id="scores"}
|
||||
|
||||
We fixed a bug that inflated the `token_acc` scores in v3.0-v3.4. The reported
|
||||
`token_acc` will drop from v3.4 to v3.5, but if `token_p/r/f` stay the same,
|
||||
your tokenization performance has not changed from v3.4.
|
||||
|
||||
For new `textcat` or `textcat_multilabel` configs, the new default `v2` scorers:
|
||||
|
||||
- ignore `threshold` for `textcat`, so the reported `cats_p/r/f` may increase
|
||||
slightly in v3.5 even though the underlying predictions are unchanged
|
||||
- report the performance of only the **final** `textcat` or `textcat_multilabel`
|
||||
component in the pipeline by default
|
||||
- allow custom scorers to be used to score multiple `textcat` and
|
||||
`textcat_multilabel` components with `Scorer.score_cats` by restricting the
|
||||
evaluation to the component's provided labels
|
||||
|
||||
### Pipeline package version compatibility {id="version-compat"}
|
||||
|
||||
> #### Using legacy implementations
|
||||
>
|
||||
> In spaCy v3, you'll still be able to load and reference legacy implementations
|
||||
> via [`spacy-legacy`](https://github.com/explosion/spacy-legacy), even if the
|
||||
> components or architectures change and newer versions are available in the
|
||||
> core library.
|
||||
|
||||
When you're loading a pipeline package trained with an earlier version of spaCy
|
||||
v3, you will see a warning telling you that the pipeline may be incompatible.
|
||||
This doesn't necessarily have to be true, but we recommend running your
|
||||
pipelines against your test suite or evaluation data to make sure there are no
|
||||
unexpected results.
|
||||
|
||||
If you're using one of the [trained pipelines](/models) we provide, you should
|
||||
run [`spacy download`](/api/cli#download) to update to the latest version. To
|
||||
see an overview of all installed packages and their compatibility, you can run
|
||||
[`spacy validate`](/api/cli#validate).
|
||||
|
||||
If you've trained your own custom pipeline and you've confirmed that it's still
|
||||
working as expected, you can update the spaCy version requirements in the
|
||||
[`meta.json`](/api/data-formats#meta):
|
||||
|
||||
```diff
|
||||
- "spacy_version": ">=3.4.0,<3.5.0",
|
||||
+ "spacy_version": ">=3.4.0,<3.6.0",
|
||||
```
|
||||
|
||||
### Updating v3.4 configs
|
||||
|
||||
To update a config from spaCy v3.4 with the new v3.5 settings, run
|
||||
[`init fill-config`](/api/cli#init-fill-config):
|
||||
|
||||
```cli
|
||||
$ python -m spacy init fill-config config-v3.4.cfg config-v3.5.cfg
|
||||
```
|
||||
|
||||
In many cases ([`spacy train`](/api/cli#train),
|
||||
[`spacy.load`](/api/top-level#spacy.load)), the new defaults will be filled in
|
||||
automatically, but you'll need to fill in the new settings to run
|
||||
[`debug config`](/api/cli#debug) and [`debug data`](/api/cli#debug-data).
|
||||
@@ -0,0 +1,143 @@
|
||||
---
|
||||
title: What's New in v3.6
|
||||
teaser: New features and how to upgrade
|
||||
menu:
|
||||
- ['New Features', 'features']
|
||||
- ['Upgrading Notes', 'upgrading']
|
||||
---
|
||||
|
||||
## New features {id="features",hidden="true"}
|
||||
|
||||
spaCy v3.6 adds the new [`SpanFinder`](/api/spanfinder) component to the core
|
||||
spaCy library and new trained pipelines for Slovenian.
|
||||
|
||||
### SpanFinder {id="spanfinder"}
|
||||
|
||||
The [`SpanFinder`](/api/spanfinder) component identifies potentially
|
||||
overlapping, unlabeled spans by identifying span start and end tokens. It is
|
||||
intended for use in combination with a component like
|
||||
[`SpanCategorizer`](/api/spancategorizer) that may further filter or label the
|
||||
spans. See our
|
||||
[Spancat blog post](https://explosion.ai/blog/spancat#span-finder) for a more
|
||||
detailed introduction to the span finder.
|
||||
|
||||
To train a pipeline with `span_finder` + `spancat`, remember to add
|
||||
`span_finder` (and its `tok2vec` or `transformer` if required) to
|
||||
`[training.annotating_components]` so that the `spancat` component can be
|
||||
trained directly from its predictions:
|
||||
|
||||
```ini
|
||||
[nlp]
|
||||
pipeline = ["tok2vec","span_finder","spancat"]
|
||||
|
||||
[training]
|
||||
annotating_components = ["tok2vec","span_finder"]
|
||||
```
|
||||
|
||||
In practice it can be helpful to initially train the `span_finder` separately
|
||||
before [sourcing](/usage/processing-pipelines#sourced-components) it (along with
|
||||
its `tok2vec`) into the `spancat` pipeline for further training. Otherwise the
|
||||
memory usage can spike for `spancat` in the first few training steps if the
|
||||
`span_finder` makes a large number of predictions.
|
||||
|
||||
### Additional features and improvements {id="additional-features-and-improvements"}
|
||||
|
||||
- Language updates:
|
||||
- Add initial support for Malay.
|
||||
- Update Latin defaults to support noun chunks, update lexical/tokenizer
|
||||
settings and add example sentences.
|
||||
- Support `spancat_singlelabel` in `spacy debug data` CLI.
|
||||
- Add `doc.spans` rendering to `spacy evaluate` CLI displaCy output.
|
||||
- Support custom token/lexeme attribute for vectors.
|
||||
- Add option to return scores separately keyed by component name with
|
||||
`spacy evaluate --per-component`, `Language.evaluate(per_component=True)` and
|
||||
`Scorer.score(per_component=True)`. This is useful when the pipeline contains
|
||||
more than one of the same component like `textcat` that may have overlapping
|
||||
scores keys.
|
||||
- Typing updates for `PhraseMatcher` and `SpanGroup`.
|
||||
|
||||
## Trained pipelines {id="pipelines"}
|
||||
|
||||
### New trained pipelines {id="new-pipelines"}
|
||||
|
||||
v3.6 introduces new pipelines for Slovenian, which use the trainable lemmatizer
|
||||
and [floret vectors](https://github.com/explosion/floret).
|
||||
|
||||
| Package | UPOS | Parser LAS | NER F |
|
||||
| ------------------------------------------------- | ---: | ---------: | ----: |
|
||||
| [`sl_core_news_sm`](/models/sl#sl_core_news_sm) | 96.9 | 82.1 | 62.9 |
|
||||
| [`sl_core_news_md`](/models/sl#sl_core_news_md) | 97.6 | 84.3 | 73.5 |
|
||||
| [`sl_core_news_lg`](/models/sl#sl_core_news_lg) | 97.7 | 84.3 | 79.0 |
|
||||
| [`sl_core_news_trf`](/models/sl#sl_core_news_trf) | 99.0 | 91.7 | 90.0 |
|
||||
|
||||
### Pipeline updates {id="pipeline-updates"}
|
||||
|
||||
The English pipelines have been updated to improve handling of contractions with
|
||||
various apostrophes and to lemmatize "get" as a passive auxiliary.
|
||||
|
||||
The Danish pipeline `da_core_news_trf` has been updated to use
|
||||
[`vesteinn/DanskBERT`](https://huggingface.co/vesteinn/DanskBERT) with
|
||||
performance improvements across the board.
|
||||
|
||||
## Notes about upgrading from v3.5 {id="upgrading"}
|
||||
|
||||
### SpanGroup spans are now required to be from the same doc {id="spangroup-spans"}
|
||||
|
||||
When initializing a `SpanGroup`, there is a new check to verify that all added
|
||||
spans refer to the current doc. Without this check, it was possible to run into
|
||||
string store or other errors.
|
||||
|
||||
One place this may crop up is when creating `Example` objects for training with
|
||||
custom spans:
|
||||
|
||||
```diff
|
||||
doc = Doc(nlp.vocab, words=tokens) # predicted doc
|
||||
example = Example.from_dict(doc, {"ner": iob_tags})
|
||||
# use the reference doc when creating reference spans
|
||||
- span = Span(doc, 0, 5, "ORG")
|
||||
+ span = Span(example.reference, 0, 5, "ORG")
|
||||
example.reference.spans[spans_key] = [span]
|
||||
```
|
||||
|
||||
### Pipeline package version compatibility {id="version-compat"}
|
||||
|
||||
> #### Using legacy implementations
|
||||
>
|
||||
> In spaCy v3, you'll still be able to load and reference legacy implementations
|
||||
> via [`spacy-legacy`](https://github.com/explosion/spacy-legacy), even if the
|
||||
> components or architectures change and newer versions are available in the
|
||||
> core library.
|
||||
|
||||
When you're loading a pipeline package trained with an earlier version of spaCy
|
||||
v3, you will see a warning telling you that the pipeline may be incompatible.
|
||||
This doesn't necessarily have to be true, but we recommend running your
|
||||
pipelines against your test suite or evaluation data to make sure there are no
|
||||
unexpected results.
|
||||
|
||||
If you're using one of the [trained pipelines](/models) we provide, you should
|
||||
run [`spacy download`](/api/cli#download) to update to the latest version. To
|
||||
see an overview of all installed packages and their compatibility, you can run
|
||||
[`spacy validate`](/api/cli#validate).
|
||||
|
||||
If you've trained your own custom pipeline and you've confirmed that it's still
|
||||
working as expected, you can update the spaCy version requirements in the
|
||||
[`meta.json`](/api/data-formats#meta):
|
||||
|
||||
```diff
|
||||
- "spacy_version": ">=3.5.0,<3.6.0",
|
||||
+ "spacy_version": ">=3.5.0,<3.7.0",
|
||||
```
|
||||
|
||||
### Updating v3.5 configs
|
||||
|
||||
To update a config from spaCy v3.5 with the new v3.6 settings, run
|
||||
[`init fill-config`](/api/cli#init-fill-config):
|
||||
|
||||
```cli
|
||||
$ python -m spacy init fill-config config-v3.5.cfg config-v3.6.cfg
|
||||
```
|
||||
|
||||
In many cases ([`spacy train`](/api/cli#train),
|
||||
[`spacy.load`](/api/top-level#spacy.load)), the new defaults will be filled in
|
||||
automatically, but you'll need to fill in the new settings to run
|
||||
[`debug config`](/api/cli#debug) and [`debug data`](/api/cli#debug-data).
|
||||
@@ -0,0 +1,140 @@
|
||||
---
|
||||
title: What's New in v3.7
|
||||
teaser: New features and how to upgrade
|
||||
menu:
|
||||
- ['New Features', 'features']
|
||||
- ['Upgrading Notes', 'upgrading']
|
||||
---
|
||||
|
||||
## New features {id="features",hidden="true"}
|
||||
|
||||
spaCy v3.7 adds support for Python 3.12, introduces the new standalone library
|
||||
[Weasel](https://github.com/explosion/weasel) for project workflows, and updates
|
||||
the transformer-based trained pipelines to use our new
|
||||
[Curated Transformers](https://github.com/explosion/curated-transformers)
|
||||
library.
|
||||
|
||||
This release drops support for Python 3.6.
|
||||
|
||||
### Weasel {id="weasel"}
|
||||
|
||||
The [spaCy projects](/usage/projects) functionality has been moved into a new
|
||||
standalone library [Weasel](https://github.com/explosion/weasel). This brings
|
||||
minor changes to spaCy-specific settings in spaCy projects (see
|
||||
[upgrading](#upgrading) below), but also makes it possible to use the same
|
||||
workflow functionality outside of spaCy.
|
||||
|
||||
All `spacy project` commands should run as before, just now they're using Weasel
|
||||
under the hood.
|
||||
|
||||
<Infobox title="Remote storage for Python 3.12" variant="warning">
|
||||
|
||||
Remote storage for spaCy projects is not yet supported for Python 3.12. Use
|
||||
Python 3.11 or earlier for remote storage.
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Registered vectors {id="custom-vectors"}
|
||||
|
||||
You can specify a custom registered vectors class under `[nlp.vectors]` in order
|
||||
to use static vectors in formats other than the ones supported by
|
||||
[`Vectors`](/api/vectors). To implement your custom vectors, extend the abstract
|
||||
class [`BaseVectors`](/api/basevectors). See an example using
|
||||
[BPEmb subword embeddings](/usage/embeddings-transformers#custom-vectors).
|
||||
|
||||
### Additional features and improvements {id="additional-features-and-improvements"}
|
||||
|
||||
- Add support for Python 3.12.
|
||||
- Extend to Thinc v8.2.
|
||||
- Extend `transformers` extra to `spacy-transformers` v1.3.
|
||||
- Add `--spans-key` option for CLI evaluation with `spacy benchmark accuracy`.
|
||||
- Load the CLI module lazily for `spacy.info`.
|
||||
- Add type stubs for for `spacy.training.example`.
|
||||
- Warn for unsupported pattern keys in dependency matcher.
|
||||
- `Language.replace_listeners`: Pass the replaced listener and the `tok2vec`
|
||||
pipe to the callback in order to support `spacy-curated-transformers`.
|
||||
- Always use `tqdm` with `disable=None` in order to disable output in
|
||||
non-interactive environments.
|
||||
- Language updates:
|
||||
- Add left and right pointing angle brackets as punctuation to ancient Greek.
|
||||
- Update example sentences for Turkish.
|
||||
- Package setup updates:
|
||||
- Update NumPy build constraints for NumPy 1.25+. For Python 3.9+, it is no
|
||||
longer necessary to set build constraints while building binary wheels.
|
||||
- Refactor Cython profiling in order to disable profiling for Python 3.12 in
|
||||
the package setup, since Cython does not currently support profiling for
|
||||
Python 3.12.
|
||||
|
||||
## Trained pipelines {id="pipelines"}
|
||||
|
||||
### Pipeline updates {id="pipeline-updates"}
|
||||
|
||||
The transformer-based `trf` pipelines have been updated to use our new
|
||||
[Curated Transformers](https://github.com/explosion/curated-transformers)
|
||||
library using the Thinc model wrappers and pipeline component from
|
||||
[spaCy Curated Transformers](https://github.com/explosion/spacy-curated-transformers).
|
||||
|
||||
## Notes about upgrading from v3.6 {id="upgrading"}
|
||||
|
||||
This release drops support for Python 3.6, drops mypy checks for Python 3.7 and
|
||||
removes the `ray` extra. In addition there are several minor changes for spaCy
|
||||
projects described in the following section.
|
||||
|
||||
### Backwards incompatibilities for spaCy Projects {id="upgrading-projects"}
|
||||
|
||||
`spacy project` has a few backwards incompatibilities due to the transition to
|
||||
the standalone library [Weasel](https://github.com/explosion/weasel), which is
|
||||
not as tightly coupled to spaCy. Weasel produces warnings when it detects older
|
||||
spaCy-specific settings in your environment or project config.
|
||||
|
||||
- Support for the `spacy_version` configuration key has been dropped.
|
||||
- Support for the `check_requirements` configuration key has been dropped due to
|
||||
the deprecation of `pkg_resources`.
|
||||
- The `SPACY_CONFIG_OVERRIDES` environment variable is no longer checked. You
|
||||
can set configuration overrides using `WEASEL_CONFIG_OVERRIDES`.
|
||||
- Support for `SPACY_PROJECT_USE_GIT_VERSION` environment variable has been
|
||||
dropped.
|
||||
- Error codes are now Weasel-specific and do not follow spaCy error codes.
|
||||
|
||||
### Pipeline package version compatibility {id="version-compat"}
|
||||
|
||||
> #### Using legacy implementations
|
||||
>
|
||||
> In spaCy v3, you'll still be able to load and reference legacy implementations
|
||||
> via [`spacy-legacy`](https://github.com/explosion/spacy-legacy), even if the
|
||||
> components or architectures change and newer versions are available in the
|
||||
> core library.
|
||||
|
||||
When you're loading a pipeline package trained with an earlier version of spaCy
|
||||
v3, you will see a warning telling you that the pipeline may be incompatible.
|
||||
This doesn't necessarily have to be true, but we recommend running your
|
||||
pipelines against your test suite or evaluation data to make sure there are no
|
||||
unexpected results.
|
||||
|
||||
If you're using one of the [trained pipelines](/models) we provide, you should
|
||||
run [`spacy download`](/api/cli#download) to update to the latest version. To
|
||||
see an overview of all installed packages and their compatibility, you can run
|
||||
[`spacy validate`](/api/cli#validate).
|
||||
|
||||
If you've trained your own custom pipeline and you've confirmed that it's still
|
||||
working as expected, you can update the spaCy version requirements in the
|
||||
[`meta.json`](/api/data-formats#meta):
|
||||
|
||||
```diff
|
||||
- "spacy_version": ">=3.6.0,<3.7.0",
|
||||
+ "spacy_version": ">=3.6.0,<3.8.0",
|
||||
```
|
||||
|
||||
### Updating v3.6 configs
|
||||
|
||||
To update a config from spaCy v3.6 with the new v3.7 settings, run
|
||||
[`init fill-config`](/api/cli#init-fill-config):
|
||||
|
||||
```cli
|
||||
$ python -m spacy init fill-config config-v3.6.cfg config-v3.7.cfg
|
||||
```
|
||||
|
||||
In many cases ([`spacy train`](/api/cli#train),
|
||||
[`spacy.load`](/api/top-level#spacy.load)), the new defaults will be filled in
|
||||
automatically, but you'll need to fill in the new settings to run
|
||||
[`debug config`](/api/cli#debug) and [`debug data`](/api/cli#debug-data).
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,440 @@
|
||||
---
|
||||
title: Visualizers
|
||||
teaser: Visualize dependencies and entities in your browser or in a notebook
|
||||
version: 2
|
||||
menu:
|
||||
- ['Dependencies', 'dep']
|
||||
- ['Named Entities', 'ent']
|
||||
- ['Spans', 'span']
|
||||
- ['Jupyter Notebooks', 'jupyter']
|
||||
- ['Rendering HTML', 'html']
|
||||
- ['Web app usage', 'webapp']
|
||||
---
|
||||
|
||||
Visualizing a dependency parse or named entities in a text is not only a fun NLP
|
||||
demo – it can also be incredibly helpful in speeding up development and
|
||||
debugging your code and training process. That's why our popular visualizers,
|
||||
[displaCy](https://explosion.ai/demos/displacy) and
|
||||
[displaCy <sup>ENT</sup>](https://explosion.ai/demos/displacy-ent) are also an
|
||||
official part of the core library. If you're running a
|
||||
[Jupyter](https://jupyter.org) notebook, displaCy will detect this and return
|
||||
the markup in a format [ready to be rendered and exported](#jupyter).
|
||||
|
||||
The quickest way to visualize `Doc` is to use
|
||||
[`displacy.serve`](/api/top-level#displacy.serve). This will spin up a simple
|
||||
web server and let you view the result straight from your browser. displaCy can
|
||||
either take a single `Doc` or a list of `Doc` objects as its first argument.
|
||||
This lets you construct them however you like – using any pipeline or
|
||||
modifications you like. If you're using [Streamlit](https://streamlit.io), check
|
||||
out the [`spacy-streamlit`](https://github.com/explosion/spacy-streamlit)
|
||||
package that helps you integrate spaCy visualizations into your apps!
|
||||
|
||||
## Visualizing the dependency parse {id="dep"}
|
||||
|
||||
The dependency visualizer, `dep`, shows part-of-speech tags and syntactic
|
||||
dependencies.
|
||||
|
||||
```python {title="Dependency example"}
|
||||
import spacy
|
||||
from spacy import displacy
|
||||
|
||||
nlp = spacy.load("en_core_web_sm")
|
||||
doc = nlp("This is a sentence.")
|
||||
displacy.serve(doc, style="dep")
|
||||
```
|
||||
|
||||
|
||||
|
||||
The argument `options` lets you specify a dictionary of settings to customize
|
||||
the layout, for example:
|
||||
|
||||
<Infobox title="Important note" variant="warning">
|
||||
|
||||
There's currently a known issue with the `compact` mode for sentences with short
|
||||
arrows and long dependency labels, that causes labels longer than the arrow to
|
||||
wrap. So if you come across this problem, especially when using custom labels,
|
||||
you'll have to increase the `distance` setting in the `options` to allow longer
|
||||
arcs.
|
||||
|
||||
Moreover, you might need to modify the `offset_x` argument depending on the shape
|
||||
of your document. Otherwise, the left part of the document may overflow beyond the
|
||||
container's border.
|
||||
|
||||
</Infobox>
|
||||
|
||||
| Argument | Description |
|
||||
| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `compact` | "Compact mode" with square arrows that takes up less space. Defaults to `False`. ~~bool~~ |
|
||||
| `color` | Text color. Can be provided in any CSS legal format as a string e.g.: `"#00ff00"`, `"rgb(0, 255, 0)"`, `"hsl(120, 100%, 50%)"` and `"green"` all correspond to the color green (without transparency). Defaults to `"#000000"`. ~~str~~ |
|
||||
| `bg` | Background color. Can be provided in any CSS legal format as a string e.g.: `"#00ff00"`, `"rgb(0, 255, 0)"`, `"hsl(120, 100%, 50%)"` and `"green"` all correspond to the color green (without transparency). Defaults to `"#ffffff"`. ~~str~~ |
|
||||
| `font` | Font name or font family for all text. Defaults to `"Arial"`. ~~str~~ |
|
||||
| `offset_x` | Spacing on left side of the SVG in px. You might need to tweak this setting for long texts. Defaults to `50`. ~~int~~ |
|
||||
|
||||
For a list of all available options, see the
|
||||
[`displacy` API documentation](/api/top-level#displacy_options).
|
||||
|
||||
> #### Options example
|
||||
>
|
||||
> ```python
|
||||
> options = {"compact": True, "bg": "#09a3d5",
|
||||
> "color": "white", "font": "Source Sans Pro"}
|
||||
> displacy.serve(doc, style="dep", options=options)
|
||||
> ```
|
||||
|
||||
|
||||
|
||||
### Visualizing long texts {id="dep-long-text",version="2.0.12"}
|
||||
|
||||
Long texts can become difficult to read when displayed in one row, so it's often
|
||||
better to visualize them sentence-by-sentence instead. As of v2.0.12, `displacy`
|
||||
supports rendering both [`Doc`](/api/doc) and [`Span`](/api/span) objects, as
|
||||
well as lists of `Doc`s or `Span`s. Instead of passing the full `Doc` to
|
||||
`displacy.serve`, you can also pass in a list `doc.sents`. This will create one
|
||||
visualization for each sentence.
|
||||
|
||||
```python
|
||||
import spacy
|
||||
from spacy import displacy
|
||||
|
||||
nlp = spacy.load("en_core_web_sm")
|
||||
text = """In ancient Rome, some neighbors live in three adjacent houses. In the center is the house of Senex, who lives there with wife Domina, son Hero, and several slaves, including head slave Hysterium and the musical's main character Pseudolus. A slave belonging to Hero, Pseudolus wishes to buy, win, or steal his freedom. One of the neighboring houses is owned by Marcus Lycus, who is a buyer and seller of beautiful women; the other belongs to the ancient Erronius, who is abroad searching for his long-lost children (stolen in infancy by pirates). One day, Senex and Domina go on a trip and leave Pseudolus in charge of Hero. Hero confides in Pseudolus that he is in love with the lovely Philia, one of the courtesans in the House of Lycus (albeit still a virgin)."""
|
||||
doc = nlp(text)
|
||||
sentence_spans = list(doc.sents)
|
||||
displacy.serve(sentence_spans, style="dep")
|
||||
```
|
||||
|
||||
## Visualizing the entity recognizer {id="ent"}
|
||||
|
||||
The entity visualizer, `ent`, highlights named entities and their labels in a
|
||||
text.
|
||||
|
||||
```python {title="Named Entity example"}
|
||||
import spacy
|
||||
from spacy import displacy
|
||||
|
||||
text = "When Sebastian Thrun started working on self-driving cars at Google in 2007, few people outside of the company took him seriously."
|
||||
|
||||
nlp = spacy.load("en_core_web_sm")
|
||||
doc = nlp(text)
|
||||
displacy.serve(doc, style="ent")
|
||||
```
|
||||
|
||||
<Standalone height={180}>
|
||||
<div style={{lineHeight: 2.5, fontFamily: "-apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'", fontSize: 18}}>When <mark style={{ background: '#aa9cfc', padding: '0.45em 0.6em', margin: '0 0.25em', lineHeight: 1, borderRadius: '0.35em'}}>Sebastian Thrun <span style={{ fontSize: '0.8em', fontWeight: 'bold', lineHeight: 1, borderRadius: '0.35em', marginLeft: '0.5rem'}}>PERSON</span></mark> started working on self-driving cars at <mark style={{ background: '#7aecec', padding: '0.45em 0.6em', margin: '0 0.25em', lineHeight: 1, borderRadius: '0.35em'}}>Google <span style={{ fontSize: '0.8em', fontWeight: 'bold', lineHeight: 1, borderRadius: '0.35em', marginLeft: '0.5rem'}}>ORG</span></mark> in <mark style={{ background: '#bfe1d9', padding: '0.45em 0.6em', margin: '0 0.25em', lineHeight: 1, borderRadius: '0.35em'}}>2007 <span style={{ fontSize: '0.8em', fontWeight: 'bold', lineHeight: 1, borderRadius: '0.35em', marginLeft: '0.5rem'}}>DATE</span></mark>, few people outside of the company took him seriously.</div>
|
||||
</Standalone>
|
||||
|
||||
The entity visualizer lets you customize the following `options`:
|
||||
|
||||
| Argument | Description |
|
||||
| -------- | ------------------------------------------------------------------------------------------------------------- |
|
||||
| `ents` | Entity types to highlight (`None` for all types). Defaults to `None`. ~~Optional[List[str]]~~ |
|
||||
| `colors` | Color overrides. Entity types should be mapped to color names or values. Defaults to `{}`. ~~Dict[str, str]~~ |
|
||||
|
||||
If you specify a list of `ents`, only those entity types will be rendered – for
|
||||
example, you can choose to display `PERSON` entities. Internally, the visualizer
|
||||
knows nothing about available entity types and will render whichever spans and
|
||||
labels it receives. This makes it especially easy to work with custom entity
|
||||
types. By default, displaCy comes with colors for all entity types used by
|
||||
[trained spaCy pipelines](/models). If you're using custom entity types, you can
|
||||
use the `colors` setting to add your own colors for them.
|
||||
|
||||
> #### Options example
|
||||
>
|
||||
> ```python
|
||||
> colors = {"ORG": "linear-gradient(90deg, #aa9cfc, #fc9ce7)"}
|
||||
> options = {"ents": ["ORG"], "colors": colors}
|
||||
> displacy.serve(doc, style="ent", options=options)
|
||||
> ```
|
||||
|
||||
<Standalone height={225}>
|
||||
<div style={{lineHeight: 2.5, fontFamily: "-apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'", fontSize: 18}}>But <mark style={{ background: 'linear-gradient(90deg, #aa9cfc, #fc9ce7)', padding: '0.45em 0.6em', margin: '0 0.25em', lineHeight: 1, borderRadius: '0.35em'}}>Google <span style={{ fontSize: '0.8em', fontWeight: 'bold', lineHeight: 1, borderRadius: '0.35em', marginLeft: '0.5rem'}}>ORG</span></mark> is starting from behind. The company made a late push into hardware, and <mark style={{ background: 'linear-gradient(90deg, #aa9cfc, #fc9ce7)', padding: '0.45em 0.6em', margin: '0 0.25em', lineHeight: 1, borderRadius: '0.35em'}}>Apple <span style={{ fontSize: '0.8em', fontWeight: 'bold', lineHeight: 1, borderRadius: '0.35em', marginLeft: '0.5rem'}}>ORG</span></mark>’s Siri, available on iPhones, and <mark style={{ background: 'linear-gradient(90deg, #aa9cfc, #fc9ce7)', padding: '0.45em 0.6em', margin: '0 0.25em', lineHeight: 1, borderRadius: '0.35em'}}>Amazon <span style={{ fontSize: '0.8em', fontWeight: 'bold', lineHeight: 1, borderRadius: '0.35em', marginLeft: '0.5rem'}}>ORG</span></mark>’s Alexa software, which runs on its Echo and Dot devices, have clear leads in consumer adoption.</div>
|
||||
</Standalone>
|
||||
|
||||
The above example uses a little trick: Since the background color values are
|
||||
added as the `background` style attribute, you can use any
|
||||
[valid background value](https://tympanus.net/codrops/css_reference/background/)
|
||||
or shorthand – including gradients and even images!
|
||||
|
||||
### Adding titles to documents {id="ent-titles"}
|
||||
|
||||
Rendering several large documents on one page can easily become confusing. To
|
||||
add a headline to each visualization, you can add a `title` to its `user_data`.
|
||||
User data is never touched or modified by spaCy.
|
||||
|
||||
```python
|
||||
doc = nlp("This is a sentence about Google.")
|
||||
doc.user_data["title"] = "This is a title"
|
||||
displacy.serve(doc, style="ent")
|
||||
```
|
||||
|
||||
This feature is especially handy if you're using displaCy to compare performance
|
||||
at different stages of a process, e.g. during training. Here you could use the
|
||||
title for a brief description of the text example and the number of iterations.
|
||||
|
||||
## Visualizing spans {id="span"}
|
||||
|
||||
The span visualizer, `span`, highlights overlapping spans in a text.
|
||||
|
||||
```python {title="Span example"}
|
||||
import spacy
|
||||
from spacy import displacy
|
||||
from spacy.tokens import Span
|
||||
|
||||
text = "Welcome to the Bank of China."
|
||||
|
||||
nlp = spacy.blank("en")
|
||||
doc = nlp(text)
|
||||
|
||||
doc.spans["sc"] = [
|
||||
Span(doc, 3, 6, "ORG"),
|
||||
Span(doc, 5, 6, "GPE"),
|
||||
]
|
||||
|
||||
displacy.serve(doc, style="span")
|
||||
```
|
||||
|
||||
<Standalone height={100}>
|
||||
<div style={{ lineHeight: 2.5, direction: 'ltr', fontFamily: "-apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'", fontSize: 18 }}>Welcome to the <span style={{ fontWeight: 'bold', display: 'inline-block', position: 'relative'}}>Bank<span style={{ background: '#7aecec', top: 40, height: 4, left: -1, width: 'calc(100% + 2px)', position: 'absolute' }}></span><span style={{ background: '#7aecec', top: 40, height: 4, borderTopLeftRadius: 3, borderBottomLeftRadius: 3, left: -1, width: 'calc(100% + 2px)', position: 'absolute' }}><span style={{ background: '#7aecec', color: '#000', top: '-0.5em', padding: '2px 3px', position: 'absolute', fontSize: '0.6em', fontWeight: 'bold', lineHeight: 1, borderRadius: 3 }}>ORG</span></span></span> <span style={{ fontWeight: 'bold', display: 'inline-block', position: 'relative'}}>of <span style={{ background: '#7aecec', top: 40, height: 4, left: -1, width: 'calc(100% + 2px)', position: 'absolute' }}></span></span> <span style={{ fontWeight: 'bold', display: 'inline-block', position: 'relative'}}>China<span style={{ background: '#7aecec', top: 40, height: 4, left: -1, width: 'calc(100% + 2px)', position: 'absolute' }}></span><span style={{ background: '#feca74', top: 57, height: 4, left: -1, width: 'calc(100% + 2px)', position: 'absolute' }}></span><span style={{ background: '#feca74', top: 57, height: 4, borderTopLeftRadius: 3, borderBottomLeftRadius: 3, left: -1, width: 'calc(100% + 2px)', position: 'absolute' }}><span style={{ background: '#feca74', color: '#000', top: '-0.5em', padding: '2px 3px', position: 'absolute', fontSize: '0.6em', fontWeight: 'bold', lineHeight: 1, borderRadius: 3 }}>GPE</span></span></span>.</div>
|
||||
</Standalone>
|
||||
|
||||
The span visualizer lets you customize the following `options`:
|
||||
|
||||
| Argument | Description |
|
||||
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `spans_key` | Which spans key to render spans from. Default is `"sc"`. ~~str~~ |
|
||||
| `templates` | Dictionary containing the keys `"span"`, `"slice"`, and `"start"`. These dictate how the overall span, a span slice, and the starting token will be rendered. ~~Optional[Dict[str, str]~~ |
|
||||
| `kb_url_template` | Optional template to construct the KB url for the entity to link to. Expects a python f-string format with single field to fill in ~~Optional[str]~~ |
|
||||
| `colors` | Color overrides. Entity types should be mapped to color names or values. ~~Dict[str, str]~~ |
|
||||
|
||||
Because spans can be stored across different keys in `doc.spans`, you need to
|
||||
specify which one displaCy should use with `spans_key` (`sc` is the default).
|
||||
|
||||
> #### Options example
|
||||
>
|
||||
> ```python
|
||||
> doc.spans["custom"] = [Span(doc, 3, 6, "BANK")]
|
||||
> options = {"spans_key": "custom"}
|
||||
> displacy.serve(doc, style="span", options=options)
|
||||
> ```
|
||||
|
||||
<Standalone height={100}>
|
||||
<div style={{ lineHeight: 2.5, direction: 'ltr', fontFamily: "-apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'", fontSize: 18 }}>Welcome to the <span style={{ fontWeight: 'bold', display: 'inline-block', position: 'relative'}}>Bank<span style={{ background: '#ddd', top: 40, height: 4, left: -1, width: 'calc(100% + 2px)', position: 'absolute' }}></span><span style={{ background: '#ddd', top: 40, height: 4, borderTopLeftRadius: 3, borderBottomLeftRadius: 3, left: -1, width: 'calc(100% + 2px)', position: 'absolute' }}><span style={{ background: '#ddd', color: '#000', top: '-0.5em', padding: '2px 3px', position: 'absolute', fontSize: '0.6em', fontWeight: 'bold', lineHeight: 1, borderRadius: 3 }}>BANK</span></span></span> <span style={{ fontWeight: 'bold', display: 'inline-block', position: 'relative'}}>of <span style={{ background: '#ddd', top: 40, height: 4, left: -1, width: 'calc(100% + 2px)', position: 'absolute' }}></span></span> <span style={{ fontWeight: 'bold', display: 'inline-block', position: 'relative'}}>China<span style={{ background: '#ddd', top: 40, height: 4, left: -1, width: 'calc(100% + 2px)', position: 'absolute' }}></span></span>.</div>
|
||||
</Standalone>
|
||||
|
||||
## Using displaCy in Jupyter notebooks {id="jupyter"}
|
||||
|
||||
displaCy is able to detect whether you're working in a
|
||||
[Jupyter](https://jupyter.org) notebook, and will return markup that can be
|
||||
rendered in a cell straight away. When you export your notebook, the
|
||||
visualizations will be included as HTML.
|
||||
|
||||
```python {title="Jupyter example"}
|
||||
# Don't forget to install a trained pipeline, e.g.: python -m spacy download en
|
||||
|
||||
# In[1]:
|
||||
import spacy
|
||||
from spacy import displacy
|
||||
|
||||
# In[2]:
|
||||
doc = nlp("Rats are various medium-sized, long-tailed rodents.")
|
||||
displacy.render(doc, style="dep")
|
||||
|
||||
# In[3]:
|
||||
doc2 = nlp(LONG_NEWS_ARTICLE)
|
||||
displacy.render(doc2, style="ent")
|
||||
```
|
||||
|
||||
<Infobox variant="warning" title="Important note">
|
||||
|
||||
To explicitly enable or disable "Jupyter mode", you can use the `jupyter`
|
||||
keyword argument – e.g. to return raw HTML in a notebook, or to force Jupyter
|
||||
rendering if auto-detection fails.
|
||||
|
||||
</Infobox>
|
||||
|
||||
|
||||
|
||||
Internally, displaCy imports `display` and `HTML` from `IPython.display`
|
||||
and returns a Jupyter HTML object. If you were doing it manually, it'd look like
|
||||
this:
|
||||
|
||||
```python
|
||||
from IPython.display import display, HTML
|
||||
|
||||
html = displacy.render(doc, style="dep")
|
||||
display(HTML(html))
|
||||
```
|
||||
|
||||
## Rendering HTML {id="html"}
|
||||
|
||||
If you don't need the web server and just want to generate the markup – for
|
||||
example, to export it to a file or serve it in a custom way – you can use
|
||||
[`displacy.render`](/api/top-level#displacy.render). It works the same way, but
|
||||
returns a string containing the markup.
|
||||
|
||||
```python {title="Example"}
|
||||
import spacy
|
||||
from spacy import displacy
|
||||
|
||||
nlp = spacy.load("en_core_web_sm")
|
||||
doc1 = nlp("This is a sentence.")
|
||||
doc2 = nlp("This is another sentence.")
|
||||
html = displacy.render([doc1, doc2], style="dep", page=True)
|
||||
```
|
||||
|
||||
`page=True` renders the markup wrapped as a full HTML page. For minified and
|
||||
more compact HTML markup, you can set `minify=True`. If you're rendering a
|
||||
dependency parse, you can also export it as an `.svg` file.
|
||||
|
||||
> #### What's SVG?
|
||||
>
|
||||
> Unlike other image formats, the SVG (Scalable Vector Graphics) uses XML markup
|
||||
> that's easy to manipulate
|
||||
> [using CSS](https://www.smashingmagazine.com/2014/11/styling-and-animating-svgs-with-css/)
|
||||
> or
|
||||
> [JavaScript](https://css-tricks.com/smil-is-dead-long-live-smil-a-guide-to-alternatives-to-smil-features/).
|
||||
> Essentially, SVG lets you design with code, which makes it a perfect fit for
|
||||
> visualizing dependency trees. SVGs can be embedded online in an `<img>` tag,
|
||||
> or inlined in an HTML document. They're also pretty easy to
|
||||
> [convert](https://convertio.co/image-converter/).
|
||||
|
||||
```python
|
||||
svg = displacy.render(doc, style="dep")
|
||||
output_path = Path("/images/sentence.svg")
|
||||
output_path.open("w", encoding="utf-8").write(svg)
|
||||
```
|
||||
|
||||
<Infobox title="Important note" variant="warning">
|
||||
|
||||
Since each visualization is generated as a separate SVG, exporting `.svg` files
|
||||
only works if you're rendering **one single doc** at a time. (This makes sense –
|
||||
after all, each visualization should be a standalone graphic.) So instead of
|
||||
rendering all `Doc`s at once, loop over them and export them separately.
|
||||
|
||||
</Infobox>
|
||||
|
||||
### Example: Export SVG graphics of dependency parses {id="examples-export-svg"}
|
||||
|
||||
```python {title="Example"}
|
||||
import spacy
|
||||
from spacy import displacy
|
||||
from pathlib import Path
|
||||
|
||||
nlp = spacy.load("en_core_web_sm")
|
||||
sentences = ["This is an example.", "This is another one."]
|
||||
for sent in sentences:
|
||||
doc = nlp(sent)
|
||||
svg = displacy.render(doc, style="dep", jupyter=False)
|
||||
file_name = '-'.join([w.text for w in doc if not w.is_punct]) + ".svg"
|
||||
output_path = Path("/images/" + file_name)
|
||||
output_path.open("w", encoding="utf-8").write(svg)
|
||||
```
|
||||
|
||||
The above code will generate the dependency visualizations as two files,
|
||||
`This-is-an-example.svg` and `This-is-another-one.svg`.
|
||||
|
||||
### Rendering data manually {id="manual-usage"}
|
||||
|
||||
You can also use displaCy to manually render data. This can be useful if you
|
||||
want to visualize output from other libraries, like [NLTK](http://www.nltk.org)
|
||||
or
|
||||
[SyntaxNet](https://github.com/tensorflow/models/tree/master/research/syntaxnet).
|
||||
If you set `manual=True` on either `render()` or `serve()`, you can pass in data
|
||||
in displaCy's format as a dictionary (instead of `Doc` objects). There are
|
||||
helper functions for converting `Doc` objects to
|
||||
[displaCy's format](/api/top-level#displacy_structures) for use with
|
||||
`manual=True`: [`displacy.parse_deps`](/api/top-level#displacy.parse_deps),
|
||||
[`displacy.parse_ents`](/api/top-level#displacy.parse_ents), and
|
||||
[`displacy.parse_spans`](/api/top-level#displacy.parse_spans).
|
||||
|
||||
> #### Example with parse function
|
||||
>
|
||||
> ```python
|
||||
> doc = nlp("But Google is starting from behind.")
|
||||
> ex = displacy.parse_ents(doc)
|
||||
> html = displacy.render(ex, style="ent", manual=True)
|
||||
> ```
|
||||
|
||||
> #### Example with raw data
|
||||
>
|
||||
> ```python
|
||||
> ex = [{"text": "But Google is starting from behind.",
|
||||
> "ents": [{"start": 4, "end": 10, "label": "ORG"}],
|
||||
> "title": None}]
|
||||
> html = displacy.render(ex, style="ent", manual=True)
|
||||
> ```
|
||||
|
||||
```python {title="DEP input"}
|
||||
{
|
||||
"words": [
|
||||
{"text": "This", "tag": "DT"},
|
||||
{"text": "is", "tag": "VBZ"},
|
||||
{"text": "a", "tag": "DT"},
|
||||
{"text": "sentence", "tag": "NN"}
|
||||
],
|
||||
"arcs": [
|
||||
{"start": 0, "end": 1, "label": "nsubj", "dir": "left"},
|
||||
{"start": 2, "end": 3, "label": "det", "dir": "left"},
|
||||
{"start": 1, "end": 3, "label": "attr", "dir": "right"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
```python {title="ENT input"}
|
||||
{
|
||||
"text": "But Google is starting from behind.",
|
||||
"ents": [{"start": 4, "end": 10, "label": "ORG"}],
|
||||
"title": None
|
||||
}
|
||||
```
|
||||
|
||||
```python {title="ENT input with knowledge base links"}
|
||||
{
|
||||
"text": "But Google is starting from behind.",
|
||||
"ents": [{"start": 4, "end": 10, "label": "ORG", "kb_id": "Q95", "kb_url": "https://www.wikidata.org/entity/Q95"}],
|
||||
"title": None
|
||||
}
|
||||
```
|
||||
|
||||
```python {title="SPANS input"}
|
||||
{
|
||||
"text": "Welcome to the Bank of China.",
|
||||
"spans": [
|
||||
{"start_token": 3, "end_token": 6, "label": "ORG"},
|
||||
{"start_token": 5, "end_token": 6, "label": "GPE"},
|
||||
],
|
||||
"tokens": ["Welcome", "to", "the", "Bank", "of", "China", "."],
|
||||
}
|
||||
```
|
||||
|
||||
## Using displaCy in a web application {id="webapp"}
|
||||
|
||||
If you want to use the visualizers as part of a web application, for example to
|
||||
create something like our [online demo](https://explosion.ai/demos/displacy),
|
||||
it's not recommended to only wrap and serve the displaCy renderer. Instead, you
|
||||
should only rely on the server to perform spaCy's processing capabilities, and
|
||||
use a client-side implementation like
|
||||
[`displaCy.js`](https://github.com/explosion/displacy) to render the
|
||||
JSON-formatted output.
|
||||
|
||||
> #### Why not return the HTML by the server?
|
||||
>
|
||||
> It's certainly possible to just have your server return the markup. But
|
||||
> outputting raw, unsanitized HTML is risky and makes your app vulnerable to
|
||||
> [cross-site scripting](https://en.wikipedia.org/wiki/Cross-site_scripting)
|
||||
> (XSS). All your user needs to do is find a way to make spaCy return text like
|
||||
> `<script src="malicious-code.js"></script>`, which is pretty easy in NER mode.
|
||||
> Instead of relying on the server to render and sanitize HTML, you can do this
|
||||
> on the client in JavaScript. displaCy.js creates the markup as DOM nodes and
|
||||
> will never insert raw HTML.
|
||||
|
||||
<Grid cols={2}>
|
||||
|
||||
Alternatively, if you're using [Streamlit](https://streamlit.io), check out the
|
||||
[`spacy-streamlit`](https://github.com/explosion/spacy-streamlit) package that
|
||||
helps you integrate spaCy visualizations into your apps. It includes a full
|
||||
embedded visualizer, as well as individual components.
|
||||
|
||||
|
||||
|
||||
</Grid>
|
||||
Reference in New Issue
Block a user