NIRT was a research collaboration where we built deep learning systems that turned unstructured text into stock forecasts.
Highlights
- Enterprise AI: Built a deep learning system for Nationale-Nederlanden's investment department to forecast stock movements from unstructured text
- NLP pipeline: Ingested and parsed large volumes of noisy earnings call transcripts and quarterlies, extracting relevant signal from unstructured data
- Hierarchical Attention Network: Implemented a HAN model (PyTorch) to learn relationships between text corpora and historic stock signals, finding significant correlations
- Engineering-first: Delivered a production-quality codebase (not a notebook), built with clean architecture and proper engineering practices
- Research collaboration: Short-lived but successful partnership between the Vrije Universiteit Amsterdam and Nationale-Nederlanden
NIRT was a short-lived collaboration between the Vrije Universiteit of Amsterdam and Nationale-Nederlanden (a Dutch insurance company, bank, and investment firm). A former professor of ours with good connections saw an opportunity and put us forward for the project. We knew AI, we knew how to build systems, and we had some application development experience. So he set us up to build a system that could help predict stock movements.
The problem
We didn’t predict the value of a stock. We predicted the delta (the increase or decrease). The data we had was a combination of supervised learned patterns, earnings call transcripts, quarterlies, and a lot of unstructured text data. This was future forecasting from language.
Ingesting the data
We’re not talking about a few documents. We’re talking about a large amount of large unstructured data. We needed to learn how to parse that information, work with noisy unstructured data, and extract relevant signal. This was the first real data engineering challenge I had faced in an AI context.
The model
From Earnings Calls to Stock Forecasts
How the system turns unstructured text into predictions
Sentiment-bearing words in earnings calls feed into a Hierarchical Attention Network to forecast stock movements.
Research revealed that we could use a Hierarchical Attention Network (HAN), a model similar in spirit to transformers. The HAN processes text at both word and sentence level using attention mechanisms, allowing it to learn the relationship between these text corpora and historic stock signals. We found significant correlations in the system.
What made it different
This was a research project with the intention of embedding it in their forecasting. But we treated it like an engineering project. We built a proper system rather than a simple data science notebook, with readable code and real engineering practices. When we delivered it, they wanted us to continue, but we felt our calling was elsewhere. In reality, this was the first time we built an AI system strong enough to be used in a large company, using state-of-the-art deep learning with promising results they could embed. We considered it a very successful project.