FinText: A Specialised Financial LLM Repository

🚀 **Stage 1 Release** 🚀

We are thrilled to introduce a specialised suite of 68 large language models (LLMs), meticulously designed for the accounting and finance. The FinText models have been pre-trained on high quality, domain-specific historical data, addressing challenges such as look-ahead bias and information leakage. These models are crafted to elevate the accuracy and depth of financial research and analysis.

💡 Key Features:

• Domain-Specific Training: FinText utilises diverse financial datasets including news articles, regulatory filings, transcripts, IP records, key information, board information, speeches (ECB, FED), and major Wikipedia articles.
• Time-Period Specific Models: Separate models are pre-trained for each year from 2007 to 2023, ensuring the utmost precision and historical relevance.
• RoBERTa Architecture: The suite includes both a base model with 125 million parameters and a smaller variant with 51 million parameters.
• Two distinct pre-training durations: We also introduce a series of models to explore the impact of futher pre-training.
• Accessibility: The models are pre-trained using BF16, but are released in FP32 format to ensure they are accessible to a broader community, including those without high-end GPUs.
• Sustainability: The entire electricity used was fully traceable and sourced exclusively from renewable energy.

For further details on this and citation, please refer to the paper, which is accessible from here.

Stay tuned for upcoming updates and new features for FinText. We expect to launch stages 2 and 3 within next months. 🎉

This project is supported by several key resources and institutions. We would like to acknowledge the invaluable assistance provided by Research IT and the use of the Computational Shared Facility at The University of Manchester. The project also benefited from the resources of the N8 Centre of Excellence in Computationally Intensive Research (N8 CIR), supported by the N8 research partnership and the Engineering and Physical Sciences Research Council (EPSRC) under Grant No. EP/T022167/1. The N8 CIR is coordinated by the Universities of Durham, Manchester, and York. Additionally, we are grateful for the financial support provided by Digital Futures at The University of Manchester, the Alan Turing Institute, and the Alliance Manchester Business School (AMBS). We are grateful to AMBS and the Oxford-Man Institute of Quantitative Finance for providing internal server access, which was essential for completing this project successfully.

Developed by:

Alliance Manchester Business School