From Index to Equity: Pre-Training Transformers for Stock Return Prediction
Incremental improvement in stock return prediction for quantitative finance practitioners, with modest gains over existing methods.
Pre-training a transformer on the TSX index improved intra-day return direction prediction for individual stocks, reducing binary cross-entropy loss from 0.69 to 0.64. The fine-tuned transformer regression model achieved lower mean squared error than LSTM and XGBoost, though ensemble and XGBoost yielded higher average daily returns.
This research aims to leverage machine learning to improve stock price prediction and support informed investment decisions related to buying, selling, and holding assets. Specifically, this work investigates transformer-based models for stock prediction and examines the impact of pre-training strategies on forecasting performance. A transformer model was first pre-trained on the Toronto Stock Exchange Index (TSX) to predict intra-day return direction and subsequently fine-tuned on individual TSX stocks. The model was further adapted for return-value regression tasks. Performance was benchmarked against Long Short-Term Memory (LSTM) and XGBoost models. Pre-training on the market index improved the binary cross-entropy loss for individual stock prediction from 0.69 to 0.64. The fine-tuned transformer regression model achieved lower mean squared error than the benchmark models, although the ensemble and XGBoost models achieved higher average daily returns. In addition, a practical application was developed to deliver real-time stock predictions for trading support. Future work will focus on increasing transformer model capacity, incorporating broader global technical indicators, and filtering out stocks with low predictability.