wehub-resource-sync
7.5 KiB
Chapter 10: Text Feature Engineering
The chapter establishes the baseline methods that made large-scale financial text analysis possible: lexicons, bag-of-words, and TF-IDF. It matters because it shows both why these methods remain useful and why they are not enough for modern trading use cases: they are fast, interpretable, and domain-adaptable, but they cannot represent context, synonymy, negation, or changing meaning across uses.
Financial text features have moved from dictionaries and bag-of-words toward contextual Transformer representations such as BERT and FinBERT.
Learning Objectives
- Distinguish lexical features, static embeddings, sequential models, and Transformers in terms of the information each representation preserves and loses
- Explain how Transformer self-attention produces contextual embeddings and why this resolves key limitations of earlier NLP methods, including polysemy and long-range dependence
- Apply a practical financial NLP workflow that combines pre-trained checkpoints, domain adaptation when needed, and task fine-tuning for classification or extraction tasks
- Design text-derived features such as sentiment, narrative surprise, or structured event signals using point-in-time-safe timestamps, model cutoffs, and aggregation rules
- Evaluate text-derived signals using horizon-aware diagnostics, coverage-aware analysis, and event-time alignment rather than benchmark accuracy alone
- Use token-level attribution and related diagnostics to audit, debug, and stress-test NLP features before deployment
Sections
10.1 Lexical and Statistical Models
This section establishes the baseline methods that made large-scale financial text analysis possible: lexicons, bag-of-words, and TF-IDF. It matters because it shows both why these methods remain useful and why they are not enough for modern trading use cases: they are fast, interpretable, and domain-adaptable, but they cannot represent context, synonymy, negation, or changing meaning across uses.
10.2 Static Embeddings
This section explains the first major leap beyond counting words: learning dense semantic representations from co-occurrence. It matters because it introduces the core intuition behind modern representation learning and shows that the same logic can extend beyond text, as in asset embeddings learned from portfolio holdings. At the same time, it makes clear why static embeddings are still only an intermediate step: one vector per word is not enough for finance, where context changes meaning constantly.
10.3 Sequential Models
This section gives readers the missing bridge between static embeddings and Transformers. It matters because it explains what RNNs and LSTMs solved, what they could not solve, and why the field moved on. The reader should care because this is the architectural turning point: once long-range dependence, sequential computation, and scaling become bottlenecks, Transformer-style attention stops being a technical curiosity and becomes the practical default.
10.4 Transformers
This section is the chapter's conceptual center of gravity on model architecture. It explains self-attention, multi-head attention, positional encoding, BERT-style encoders, finance-specific checkpoints, and the practical consequences of domain adaptation, fine-tuning, and model choice. Readers should care because this is where the chapter moves from general NLP history to the modern tools that actually power financial text classification, embedding generation, and representation-based feature engineering.
10.5 The Modern Feature Extraction Workflow
This is the chapter's real practical core. It explains that a strong text model is not yet a tradable signal unless timestamps, entity resolution, revisions, aggregation rules, model training cutoffs, and evaluation protocols are all made point-in-time safe. It also connects representation models to real feature families such as sentiment, narrative surprise, topic exposure, structured event extraction, and interpretable diagnostics. Readers should care because this section is what turns NLP from a benchmark exercise into a research and production workflow suitable for systematic trading.
Running the Notebooks
# From the repository root
uv run python 10_text_feature_engineering/<notebook>.py
# Test mode (reduced data via Papermill)
uv run pytest tests/test_notebooks.py -v -k "10_text_feature_engineering"
Docker image split (chapter-specific)
Three notebooks require the ml4t-py312 image because gensim has no Python 3.14 wheel:
docker compose --profile py312 run --rm py312 \
python 10_text_feature_engineering/01_word2vec_training.py
| Notebook | Image |
|---|---|
01_word2vec_training |
ml4t-py312 (gensim Word2Vec) |
02_asset_embeddings |
ml4t-py312 (gensim Word2Vec) |
03_sentiment_evolution |
ml4t-py312 (gensim GloVe loader) |
04_bert_finetuning |
ml4t-gpu (PyTorch GPU recommended) |
05_financial_ner_finetuning |
ml4t-gpu (PyTorch GPU recommended) |
06_finbert_cross_dataset |
ml4t-gpu (PyTorch GPU recommended) |
07_news_return_signals |
ml4t-gpu (PyTorch GPU recommended) |
08_text_feature_evaluation |
ml4t (CPU-only IC + quintile diagnostics) |
09_filing_text_signals |
ml4t-gpu (PyTorch GPU recommended) |
Runtime callouts
02_asset_embeddings: ~5–6 min (gensim skip-gram on ~500 13F portfolios; CPU-bound).
09_filing_text_signals: ~7 min on GPU (FinBERT sentence-level scoring across S&P-500 MD&A filings, GPU recommended).
References
- Rajeev Bhargava et al. (2023). Quantifying Narratives and Their Impact on Financial Markets. The Journal of Portfolio Management.
- Jacob Devlin et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Association for Computational Linguistics.
- Allen Huang et al. (2020). FinBERT—A Deep Learning Approach to Extracting Textual Information. SSRN Electronic Journal.
- Tim Loughran and Bill Mcdonald (2011). When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and 10-Ks. The Journal of Finance.
- Tim Loughran and Bill McDonald (2020). Textual Analysis in Finance. Annual Review of Financial Economics.
- Scott M Lundberg et al. (2017). A Unified Approach to Interpreting Model Predictions. Curran Associates, Inc..
- Tomas Mikolov et al. (2013). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
- Nils Reimers and Iryna Gurevych (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. arXiv:1908.10084 [cs].
- Ashish Vaswani et al. (2017). Attention Is All You Need. arXiv:1706.03762 [cs].
- Qianqian Xie et al. (2024). Finben: A holistic financial benchmark for large language models. Advances in Neural Information Processing Systems.